2024-02-22T23:59:13.304 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-22T23:59:13.304 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --name yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394 --description rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} -- /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394/orig.config.yaml 2024-02-22T23:59:13.742 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-22T23:59:13.829 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394 branch: pacific-release description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} email: null first_in_suite: false job_id: '7571394' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(MON_DOWN\) - \(OSDMAP_FLAGS\) - flag\(s\) set - \(CACHE_POOL_NO_HIT_SET\) - \(PG_ - \(OSD_ - \(POOL_ - \(CEPHADM_STRAY_DAEMON\) - PG_ - degraded - backfill - mons down - OSD_ - is down - acting - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) sha1: eb66ed921e744301e6863be6353618d63967ea59 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: eb66ed921e744301e6863be6353618d63967ea59 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-ksirivad-pacific-release-whitelist sha1: 4f758cd8e8854d1f126da86e254076852bf059a9 owner: scheduled_yuriw@teuthology priority: 61 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 264 sha1: eb66ed921e744301e6863be6353618d63967ea59 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-ksirivad-pacific-release-whitelist suite_path: /home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa suite_relpath: qa suite_repo: https://github.com/kamoltat/ceph.git suite_sha1: 4f758cd8e8854d1f126da86e254076852bf059a9 targets: smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG18gcYhtOJmt3HHvenammzLLFirbbVECGK7ZCuWfAeEmu+6oX0B/nLyGnYIE1yKHtRGRYHHNYKNZ0yaR+SYmrw= smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzOyZei/1wpgCXnjKU9xo9p1MMg970O+tjaEwG0Bi5A16K5vGrdD/3ohhWIj8h2INcrdRW5/wN/mxE9yVeEBYk= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-22_21:39:39 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-22T23:59:13.829 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa; will attempt to use it 2024-02-22T23:59:13.845 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_kamoltat_ceph_4f758cd8e8854d1f126da86e254076852bf059a9/qa/tasks 2024-02-22T23:59:13.845 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-22T23:59:13.878 INFO:teuthology.task.internal:Checking packages... 2024-02-22T23:59:13.903 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'eb66ed921e744301e6863be6353618d63967ea59' 2024-02-22T23:59:13.903 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-22T23:59:13.903 INFO:teuthology.packaging:ref: None 2024-02-22T23:59:13.903 INFO:teuthology.packaging:tag: None 2024-02-22T23:59:13.903 INFO:teuthology.packaging:branch: pacific-release 2024-02-22T23:59:13.903 INFO:teuthology.packaging:sha1: eb66ed921e744301e6863be6353618d63967ea59 2024-02-22T23:59:13.903 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-22T23:59:14.064 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-550.geb66ed92 2024-02-22T23:59:14.066 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-22T23:59:14.076 INFO:teuthology.task.internal:no buildpackages task found 2024-02-22T23:59:14.076 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-22T23:59:14.163 INFO:teuthology.task.internal:Saving configuration 2024-02-22T23:59:14.177 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-22T23:59:14.337 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-22T23:59:14.365 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 23:53:49.978859', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG18gcYhtOJmt3HHvenammzLLFirbbVECGK7ZCuWfAeEmu+6oX0B/nLyGnYIE1yKHtRGRYHHNYKNZ0yaR+SYmrw='} 2024-02-22T23:59:14.390 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 23:53:49.980546', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzOyZei/1wpgCXnjKU9xo9p1MMg970O+tjaEwG0Bi5A16K5vGrdD/3ohhWIj8h2INcrdRW5/wN/mxE9yVeEBYk='} 2024-02-22T23:59:14.390 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-22T23:59:14.523 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-22T23:59:14.523 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-22T23:59:14.523 INFO:teuthology.run_tasks:Running task console_log... 2024-02-22T23:59:14.630 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1f879b50d0>, signals=[15]) 2024-02-22T23:59:14.631 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-22T23:59:14.636 INFO:teuthology.task.internal:Opening connections... 2024-02-22T23:59:14.637 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2024-02-22T23:59:14.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-22T23:59:14.713 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi171.front.sepia.ceph.com 2024-02-22T23:59:14.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-22T23:59:14.784 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-22T23:59:14.790 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-02-22T23:59:14.813 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-02-22T23:59:14.813 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-02-22T23:59:14.875 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-02-22T23:59:14.875 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="8" 2024-02-22T23:59:14.875 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-02-22T23:59:14.875 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-02-22T23:59:14.875 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="8" 2024-02-22T23:59:14.876 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el8" 2024-02-22T23:59:14.876 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-22T23:59:14.876 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-02-22T23:59:14.876 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-22T23:59:14.876 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-02-22T23:59:14.876 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-22T23:59:14.876 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-22T23:59:14.876 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-22T23:59:14.877 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-02-22T23:59:14.905 DEBUG:teuthology.orchestra.run.smithi171:> uname -m 2024-02-22T23:59:14.928 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2024-02-22T23:59:14.928 DEBUG:teuthology.orchestra.run.smithi171:> cat /etc/os-release 2024-02-22T23:59:14.990 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2024-02-22T23:59:14.990 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="8" 2024-02-22T23:59:14.991 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2024-02-22T23:59:14.991 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2024-02-22T23:59:14.991 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="8" 2024-02-22T23:59:14.991 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el8" 2024-02-22T23:59:14.991 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-22T23:59:14.991 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2024-02-22T23:59:14.991 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-22T23:59:14.991 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2024-02-22T23:59:14.992 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-22T23:59:14.992 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-22T23:59:14.992 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-22T23:59:14.992 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2024-02-22T23:59:15.014 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-22T23:59:15.023 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-22T23:59:15.059 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-22T23:59:15.059 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2024-02-22T23:59:15.062 DEBUG:teuthology.orchestra.run.smithi171:> test '!' -e /home/ubuntu/cephtest 2024-02-22T23:59:15.082 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-22T23:59:15.088 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-22T23:59:15.088 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2024-02-22T23:59:15.123 DEBUG:teuthology.orchestra.run.smithi171:> test -z $(ls -A /var/lib/ceph) 2024-02-22T23:59:15.151 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-22T23:59:15.208 INFO:teuthology.run_tasks:Running task kernel... 2024-02-22T23:59:15.223 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-22T23:59:15.223 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-22T23:59:15.223 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-22T23:59:15.224 DEBUG:teuthology.orchestra.run.smithi171:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-22T23:59:15.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-22T23:59:15.244 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-02-22T23:59:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-22T23:59:15.245 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-02-22T23:59:15.305 INFO:teuthology.orchestra.run.smithi087.stdout:4.18.0-489.el8.x86_64 2024-02-22T23:59:15.305 INFO:teuthology.task.kernel:Running kernel on smithi087: 4.18.0-489.el8.x86_64 2024-02-22T23:59:15.305 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2024-02-22T23:59:15.306 INFO:teuthology.orchestra.run.smithi171.stdout:4.18.0-489.el8.x86_64 2024-02-22T23:59:15.307 INFO:teuthology.task.kernel:Running kernel on smithi171: 4.18.0-489.el8.x86_64 2024-02-22T23:59:15.307 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum install -y kernel 2024-02-22T23:59:20.899 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - AppStream 9.0 MB/s | 27 MB 00:03 2024-02-22T23:59:21.365 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - AppStream 7.8 MB/s | 27 MB 00:03 2024-02-22T23:59:26.716 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-02-22T23:59:26.884 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - BaseOS 16 MB/s | 10 MB 00:00 2024-02-22T23:59:28.616 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - Extras 65 kB/s | 18 kB 00:00 2024-02-22T23:59:28.685 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras 88 kB/s | 18 kB 00:00 2024-02-22T23:59:28.950 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - Extras common packages 31 kB/s | 6.9 kB 00:00 2024-02-22T23:59:29.144 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-02-22T23:59:29.193 INFO:teuthology.orchestra.run.smithi171.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-22T23:59:29.379 INFO:teuthology.orchestra.run.smithi087.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-02-22T23:59:29.938 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2024-02-22T23:59:30.663 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-02-22T23:59:33.744 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 378 kB/s | 24 kB 00:00 2024-02-22T23:59:34.413 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 430 kB/s | 24 kB 00:00 2024-02-22T23:59:36.741 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-22T23:59:36.741 INFO:teuthology.orchestra.run.smithi171.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-22T23:59:36.855 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-22T23:59:36.855 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout: kernel x86_64 4.18.0-540.el8 baseos 10 M 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-core x86_64 4.18.0-540.el8 baseos 44 M 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules x86_64 4.18.0-540.el8 baseos 36 M 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout:Install 3 Packages 2024-02-22T23:59:36.856 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-22T23:59:36.857 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 90 M 2024-02-22T23:59:36.857 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 97 M 2024-02-22T23:59:36.857 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-22T23:59:37.368 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-22T23:59:37.369 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-22T23:59:37.478 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-22T23:59:37.478 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-22T23:59:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 4.18.0-540.el8 baseos 10 M 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 4.18.0-540.el8 baseos 44 M 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 4.18.0-540.el8 baseos 36 M 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout:Install 3 Packages 2024-02-22T23:59:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-22T23:59:37.480 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 90 M 2024-02-22T23:59:37.480 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 97 M 2024-02-22T23:59:37.480 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-22T23:59:37.687 INFO:teuthology.orchestra.run.smithi171.stdout:(1/3): kernel-4.18.0-540.el8.x86_64.rpm 16 MB/s | 10 MB 00:00 2024-02-22T23:59:38.690 INFO:teuthology.orchestra.run.smithi171.stdout:(2/3): kernel-core-4.18.0-540.el8.x86_64.rpm 26 MB/s | 44 MB 00:01 2024-02-22T23:59:39.040 INFO:teuthology.orchestra.run.smithi087.stdout:(1/3): kernel-4.18.0-540.el8.x86_64.rpm 7.5 MB/s | 10 MB 00:01 2024-02-22T23:59:39.458 INFO:teuthology.orchestra.run.smithi087.stdout:(2/3): kernel-modules-4.18.0-540.el8.x86_64.rpm 20 MB/s | 36 MB 00:01 2024-02-22T23:59:39.828 INFO:teuthology.orchestra.run.smithi171.stdout:(3/3): kernel-modules-4.18.0-540.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2024-02-22T23:59:39.828 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-22T23:59:39.828 INFO:teuthology.orchestra.run.smithi171.stdout:Total 30 MB/s | 90 MB 00:02 2024-02-22T23:59:39.957 INFO:teuthology.orchestra.run.smithi087.stdout:(3/3): kernel-core-4.18.0-540.el8.x86_64.rpm 19 MB/s | 44 MB 00:02 2024-02-22T23:59:39.957 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-22T23:59:39.957 INFO:teuthology.orchestra.run.smithi087.stdout:Total 36 MB/s | 90 MB 00:02 2024-02-22T23:59:40.513 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-22T23:59:40.640 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-22T23:59:40.933 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-22T23:59:40.933 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-22T23:59:41.062 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-22T23:59:41.062 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-22T23:59:42.187 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-22T23:59:42.210 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-22T23:59:42.294 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-22T23:59:42.322 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-22T23:59:45.003 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-22T23:59:45.106 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-22T23:59:45.500 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-core-4.18.0-540.el8.x86_64 1/3 2024-02-22T23:59:45.569 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-4.18.0-540.el8.x86_64 1/3 2024-02-22T23:59:49.086 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-4.18.0-540.el8.x86_64 1/3 2024-02-22T23:59:49.103 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-540.el8.x86_64 1/3 2024-02-22T23:59:49.469 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-4.18.0-540.el8.x86_64 2/3 2024-02-22T23:59:49.474 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-modules-4.18.0-540.el8.x86_64 2/3 2024-02-22T23:59:52.903 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-4.18.0-540.el8.x86_64 2/3 2024-02-22T23:59:52.920 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-4.18.0-540.el8.x86_64 2/3 2024-02-22T23:59:53.174 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : kernel-4.18.0-540.el8.x86_64 3/3 2024-02-22T23:59:53.218 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:15.085 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-core-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:15.108 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-modules-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:15.458 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: kernel-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:15.458 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-4.18.0-540.el8.x86_64 1/3 2024-02-23T00:00:15.458 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-core-4.18.0-540.el8.x86_64 2/3 2024-02-23T00:00:15.768 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:15.768 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-4.18.0-540.el8.x86_64 1/3 2024-02-23T00:00:15.769 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-4.18.0-540.el8.x86_64 2/3 2024-02-23T00:00:15.954 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : kernel-modules-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:15.955 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:00:15.955 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:00:15.955 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-4.18.0-540.el8.x86_64 kernel-core-4.18.0-540.el8.x86_64 2024-02-23T00:00:15.955 INFO:teuthology.orchestra.run.smithi171.stdout: kernel-modules-4.18.0-540.el8.x86_64 2024-02-23T00:00:15.955 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:00:15.955 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:00:16.103 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-02-23T00:00:16.262 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-4.18.0-540.el8.x86_64 3/3 2024-02-23T00:00:16.263 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:00:16.263 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:00:16.263 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-4.18.0-540.el8.x86_64 kernel-core-4.18.0-540.el8.x86_64 2024-02-23T00:00:16.263 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-4.18.0-540.el8.x86_64 2024-02-23T00:00:16.263 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:00:16.263 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:00:16.318 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-4.18.0-540.el8.x86_64 2024-02-23T00:00:16.319 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-540.el8.x86_64 2024-02-23T00:00:16.319 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-540.el8.x86_64 2024-02-23T00:00:16.320 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-23T00:00:16.320 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-23T00:00:16.320 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-23T00:00:16.320 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi171.front.sepia.ceph.com, path=None, version=distro) 2024-02-23T00:00:16.321 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q kernel | sort -rV | head -n 1 2024-02-23T00:00:16.425 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-02-23T00:00:16.533 INFO:teuthology.orchestra.run.smithi171.stdout:kernel-4.18.0-540.el8.x86_64 2024-02-23T00:00:16.533 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-540.el8.x86_64 2024-02-23T00:00:16.533 DEBUG:teuthology.orchestra.run.smithi171:> sudo rpm -qi grub2-tools 2024-02-23T00:00:16.567 INFO:teuthology.orchestra.run.smithi171.stdout:Name : grub2-tools 2024-02-23T00:00:16.567 INFO:teuthology.orchestra.run.smithi171.stdout:Epoch : 1 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Version : 2.02 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Release : 129.el8 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Architecture: x86_64 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Group : System Environment/Base 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Size : 9265586 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:License : GPLv3+ 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-23T00:00:16.568 INFO:teuthology.orchestra.run.smithi171.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:Relocations : (not relocatable) 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:Packager : CentOS BuildSystem 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:Vendor : CentOS 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:Summary : Support tools for GRUB. 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:Description : 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:hardware devices. 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:00:16.569 INFO:teuthology.orchestra.run.smithi171.stdout:This subpackage provides tools for support of all platforms. 2024-02-23T00:00:16.570 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-23T00:00:16.571 INFO:teuthology.task.kernel:Updating grub on smithi171 to boot 4.18.0-540.el8.x86_64 2024-02-23T00:00:16.571 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-23T00:00:16.654 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-4.18.0-540.el8.x86_64 2024-02-23T00:00:16.654 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-540.el8.x86_64 2024-02-23T00:00:16.654 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-540.el8.x86_64 2024-02-23T00:00:16.654 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-23T00:00:16.654 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-23T00:00:16.655 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-23T00:00:16.655 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2024-02-23T00:00:16.655 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2024-02-23T00:00:16.866 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-4.18.0-540.el8.x86_64 2024-02-23T00:00:16.866 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-540.el8.x86_64 2024-02-23T00:00:16.866 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.02 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 129.el8 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Group : System Environment/Base 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 9265586 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-23T00:00:16.900 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:Relocations : (not relocatable) 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : CentOS BuildSystem 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:00:16.901 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2024-02-23T00:00:16.902 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-23T00:00:16.902 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 4.18.0-540.el8.x86_64 2024-02-23T00:00:16.903 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-23T00:00:18.153 INFO:teuthology.orchestra.run.smithi171.stderr:Generating grub configuration file ... 2024-02-23T00:00:18.304 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2024-02-23T00:00:19.999 INFO:teuthology.orchestra.run.smithi171.stderr:done 2024-02-23T00:00:20.001 DEBUG:teuthology.orchestra.run.smithi171:> mktemp 2024-02-23T00:00:20.041 INFO:teuthology.orchestra.run.smithi171.stdout:/tmp/tmp.RzeU8T3ban 2024-02-23T00:00:20.041 DEBUG:teuthology.orchestra.run.smithi171:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RzeU8T3ban 2024-02-23T00:00:20.114 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0666 /tmp/tmp.RzeU8T3ban 2024-02-23T00:00:20.280 DEBUG:teuthology.orchestra.remote:smithi171:/tmp/tmp.RzeU8T3ban is 6KB 2024-02-23T00:00:20.330 INFO:teuthology.orchestra.run.smithi087.stderr:done 2024-02-23T00:00:20.332 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2024-02-23T00:00:20.340 DEBUG:teuthology.orchestra.run.smithi171:> rm -fr /tmp/tmp.RzeU8T3ban 2024-02-23T00:00:20.357 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.I3ExDAALts 2024-02-23T00:00:20.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.I3ExDAALts 2024-02-23T00:00:20.359 DEBUG:teuthology.orchestra.run.smithi171:> sudo /bin/ls /boot/loader/entries || true 2024-02-23T00:00:20.429 INFO:teuthology.orchestra.run.smithi171.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-23T00:00:20.429 INFO:teuthology.orchestra.run.smithi171.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-23T00:00:20.429 INFO:teuthology.orchestra.run.smithi171.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-23T00:00:20.429 INFO:teuthology.orchestra.run.smithi171.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-540.el8.x86_64.conf 2024-02-23T00:00:20.430 DEBUG:teuthology.orchestra.run.smithi171:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-540.el8.x86_64 2024-02-23T00:00:20.432 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.I3ExDAALts 2024-02-23T00:00:20.574 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.I3ExDAALts is 6KB 2024-02-23T00:00:20.581 DEBUG:teuthology.orchestra.run.smithi171:> sudo shutdown -r now 2024-02-23T00:00:20.599 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.I3ExDAALts 2024-02-23T00:00:20.617 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2024-02-23T00:00:20.684 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-23T00:00:20.684 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-23T00:00:20.684 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-23T00:00:20.684 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-540.el8.x86_64.conf 2024-02-23T00:00:20.685 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-540.el8.x86_64 2024-02-23T00:00:20.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2024-02-23T00:00:50.612 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-23T00:00:50.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-02-23T00:00:50.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:00:50.864 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-23T00:00:50.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-23T00:00:50.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:01:09.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-02-23T00:01:09.181 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-02-23T00:01:18.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-02-23T00:01:18.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:01:18.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-23T00:01:18.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:01:21.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.171 2024-02-23T00:01:21.244 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-02-23T00:01:33.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-02-23T00:01:33.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:01:33.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-23T00:01:33.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:02:05.647 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-02-23T00:02:06.223 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-23T00:02:06.224 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-540.el8.x86_64"... 2024-02-23T00:02:06.224 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2024-02-23T00:02:06.243 INFO:teuthology.orchestra.run.smithi087.stdout:4.18.0-540.el8.x86_64 2024-02-23T00:02:06.243 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-540.el8.x86_64 vs 4.18.0-540.el8.x86_64 2024-02-23T00:02:06.243 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-23T00:02:06.243 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-23T00:02:07.244 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-23T00:02:07.244 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-23T00:02:07.305 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2024-02-23T00:02:07.317 DEBUG:teuthology.parallel:result is None 2024-02-23T00:02:33.258 DEBUG:teuthology.orchestra.remote:timed out 2024-02-23T00:02:48.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-02-23T00:02:48.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:02:48.565 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-02-23T00:02:48.873 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-02-23T00:02:48.874 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-540.el8.x86_64"... 2024-02-23T00:02:48.874 DEBUG:teuthology.orchestra.run.smithi171:> uname -r 2024-02-23T00:02:48.894 INFO:teuthology.orchestra.run.smithi171.stdout:4.18.0-540.el8.x86_64 2024-02-23T00:02:48.894 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-540.el8.x86_64 vs 4.18.0-540.el8.x86_64 2024-02-23T00:02:48.895 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-23T00:02:48.895 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-23T00:02:49.896 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-23T00:02:49.896 DEBUG:teuthology.orchestra.run.smithi171:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-23T00:02:49.958 INFO:teuthology.orchestra.run.smithi171.stdout:ttyS1 2024-02-23T00:02:49.974 DEBUG:teuthology.parallel:result is None 2024-02-23T00:02:49.974 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-23T00:02:49.995 INFO:teuthology.task.internal:Creating test directory... 2024-02-23T00:02:49.995 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-23T00:02:49.998 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-23T00:02:50.018 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-23T00:02:50.059 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-23T00:02:50.110 INFO:teuthology.task.internal:Creating archive directory... 2024-02-23T00:02:50.110 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-23T00:02:50.113 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-23T00:02:50.174 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-23T00:02:50.184 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-23T00:02:50.184 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-23T00:02:50.189 DEBUG:teuthology.orchestra.run.smithi171:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-23T00:02:50.225 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-23T00:02:50.225 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-23T00:02:50.235 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-23T00:02:50.236 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-23T00:02:50.237 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-23T00:02:50.281 INFO:teuthology.task.internal:Configuring sudo... 2024-02-23T00:02:50.281 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-23T00:02:50.284 DEBUG:teuthology.orchestra.run.smithi171:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-23T00:02:50.314 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-23T00:02:50.327 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-23T00:02:50.328 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-23T00:02:50.355 DEBUG:teuthology.orchestra.run.smithi171:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-23T00:02:50.373 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-23T00:02:50.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-23T00:02:50.516 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-23T00:02:50.602 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-23T00:02:50.644 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:02:50.644 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-23T00:02:50.711 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-23T00:02:50.754 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-23T00:02:50.808 DEBUG:teuthology.orchestra.run.smithi171:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-23T00:02:50.897 DEBUG:teuthology.orchestra.run.smithi171:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-23T00:02:50.935 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:02:50.935 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-23T00:02:51.002 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2024-02-23T00:02:51.004 DEBUG:teuthology.orchestra.run.smithi171:> sudo service rsyslog restart 2024-02-23T00:02:51.074 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-23T00:02:51.098 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-23T00:02:51.453 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-23T00:02:51.487 INFO:teuthology.task.internal:Starting timer... 2024-02-23T00:02:51.487 INFO:teuthology.run_tasks:Running task pcp... 2024-02-23T00:02:51.586 INFO:teuthology.run_tasks:Running task selinux... 2024-02-23T00:02:51.688 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-23T00:02:51.688 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2024-02-23T00:02:51.766 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2024-02-23T00:02:51.768 DEBUG:teuthology.orchestra.run.smithi171:> sudo service auditd rotate 2024-02-23T00:02:51.828 INFO:teuthology.orchestra.run.smithi171.stdout:Rotating logs: 2024-02-23T00:02:51.829 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-23T00:02:51.829 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2024-02-23T00:02:51.892 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2024-02-23T00:02:51.893 DEBUG:teuthology.orchestra.run.smithi171:> /usr/sbin/getenforce 2024-02-23T00:02:51.914 INFO:teuthology.orchestra.run.smithi171.stdout:Permissive 2024-02-23T00:02:51.915 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi087.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi171.front.sepia.ceph.com': 'permissive'} 2024-02-23T00:02:51.915 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-23T00:02:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:02:51.943 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-23T00:02:51.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:02:51.983 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-23T00:02:51.983 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2024-02-23T00:02:52.028 DEBUG:teuthology.orchestra.run.smithi171:> sudo /usr/sbin/setenforce permissive 2024-02-23T00:02:52.056 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-23T00:02:52.179 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-23T00:02:52.202 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-23T00:02:52.204 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi087.front.sepia.ceph.com,smithi171.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-23T00:13:03.514 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi087.front.sepia.ceph.com'), Remote(name='ubuntu@smithi171.front.sepia.ceph.com')] 2024-02-23T00:13:03.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-23T00:13:03.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:13:03.642 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-02-23T00:13:03.715 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-02-23T00:13:03.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-02-23T00:13:03.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi171.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-23T00:13:03.811 DEBUG:teuthology.orchestra.run.smithi171:> true 2024-02-23T00:13:03.890 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi171.front.sepia.ceph.com' 2024-02-23T00:13:03.891 INFO:teuthology.run_tasks:Running task clock... 2024-02-23T00:13:03.906 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-23T00:13:03.906 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-23T00:13:03.907 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-23T00:13:03.909 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-23T00:13:03.909 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-23T00:13:03.944 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-23T00:13:03.960 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-23T00:13:03.968 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-23T00:13:03.983 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-23T00:13:04.027 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2024-02-23T00:13:04.039 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2024-02-23T00:13:04.047 INFO:teuthology.orchestra.run.smithi171.stderr:sudo: ntpd: command not found 2024-02-23T00:13:04.053 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-23T00:13:04.059 INFO:teuthology.orchestra.run.smithi171.stdout:506 Cannot talk to daemon 2024-02-23T00:13:04.067 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-23T00:13:04.074 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-23T00:13:04.088 INFO:teuthology.orchestra.run.smithi171.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-23T00:13:04.156 INFO:teuthology.orchestra.run.smithi171.stderr:bash: ntpq: command not found 2024-02-23T00:13:04.156 INFO:teuthology.orchestra.run.smithi087.stderr:bash: ntpq: command not found 2024-02-23T00:13:04.161 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-23T00:13:04.161 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-02-23T00:13:04.161 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:04.161 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:04.162 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:04.162 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:04.162 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-23T00:13:04.162 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-02-23T00:13:04.162 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:04.162 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:04.162 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:04.163 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:13:04.163 INFO:teuthology.run_tasks:Running task pexec... 2024-02-23T00:13:04.181 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-23T00:13:04.182 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi087.front.sepia.ceph.com 2024-02-23T00:13:04.182 DEBUG:teuthology.orchestra.run.smithi087:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-23T00:13:04.182 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi171.front.sepia.ceph.com 2024-02-23T00:13:04.182 DEBUG:teuthology.orchestra.run.smithi171:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-23T00:13:04.986 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:21 ago on Fri 23 Feb 2024 12:11:43 AM UTC. 2024-02-23T00:13:04.998 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:22 ago on Fri 23 Feb 2024 12:11:42 AM UTC. 2024-02-23T00:13:07.028 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:13:07.028 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:13:07.028 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-23T00:13:07.028 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:13:07.028 INFO:teuthology.orchestra.run.smithi087.stdout:Resetting modules: 2024-02-23T00:13:07.028 INFO:teuthology.orchestra.run.smithi087.stdout: container-tools 2024-02-23T00:13:07.029 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:13:07.029 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:13:07.029 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:13:07.029 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:13:07.058 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:13:07.059 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:13:07.060 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-02-23T00:13:07.060 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:13:07.060 INFO:teuthology.orchestra.run.smithi171.stdout:Resetting modules: 2024-02-23T00:13:07.060 INFO:teuthology.orchestra.run.smithi171.stdout: container-tools 2024-02-23T00:13:07.061 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:13:07.061 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:13:07.061 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:13:07.061 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:13:07.272 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:13:07.311 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:13:08.136 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:24 ago on Fri 23 Feb 2024 12:11:43 AM UTC. 2024-02-23T00:13:08.161 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:25 ago on Fri 23 Feb 2024 12:11:42 AM UTC. 2024-02-23T00:13:10.199 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:13:10.217 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout:Installing group/module packages: 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout: crun x86_64 1.14.1-1.module_el8+951+32019cde appstream 257 k 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-23T00:13:10.235 INFO:teuthology.orchestra.run.smithi087.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout:Installing weak dependencies: 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout:Installing module profiles: 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout: container-tools/common 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout:Enabling module streams: 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout: container-tools rhel8 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:13:10.236 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:13:10.237 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:13:10.237 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-02-23T00:13:10.237 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:13:10.237 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 27 M 2024-02-23T00:13:10.237 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 91 M 2024-02-23T00:13:10.237 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:13:10.253 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:13:10.253 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:13:10.253 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:13:10.253 INFO:teuthology.orchestra.run.smithi171.stdout:Installing group/module packages: 2024-02-23T00:13:10.253 INFO:teuthology.orchestra.run.smithi171.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-23T00:13:10.253 INFO:teuthology.orchestra.run.smithi171.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: crun x86_64 1.14.1-1.module_el8+951+32019cde appstream 257 k 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout:Installing weak dependencies: 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout:Installing module profiles: 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: container-tools/common 2024-02-23T00:13:10.254 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:13:10.255 INFO:teuthology.orchestra.run.smithi171.stdout:Enabling module streams: 2024-02-23T00:13:10.255 INFO:teuthology.orchestra.run.smithi171.stdout: container-tools rhel8 2024-02-23T00:13:10.255 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:13:10.255 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:13:10.255 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:13:10.255 INFO:teuthology.orchestra.run.smithi171.stdout:Install 13 Packages 2024-02-23T00:13:10.255 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:13:10.255 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 27 M 2024-02-23T00:13:10.256 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 91 M 2024-02-23T00:13:10.256 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:13:10.484 INFO:teuthology.orchestra.run.smithi171.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 3.6 MB/s | 688 kB 00:00 2024-02-23T00:13:10.551 INFO:teuthology.orchestra.run.smithi171.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 4.0 MB/s | 1.0 MB 00:00 2024-02-23T00:13:10.568 INFO:teuthology.orchestra.run.smithi171.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 451 kB/s | 38 kB 00:00 2024-02-23T00:13:10.643 INFO:teuthology.orchestra.run.smithi171.stdout:(4/13): crun-1.14.1-1.module_el8+951+32019cde.x 2.7 MB/s | 257 kB 00:00 2024-02-23T00:13:10.702 INFO:teuthology.orchestra.run.smithi171.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 3.0 MB/s | 180 kB 00:00 2024-02-23T00:13:10.752 INFO:teuthology.orchestra.run.smithi171.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 494 kB/s | 25 kB 00:00 2024-02-23T00:13:10.792 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 1.8 MB/s | 688 kB 00:00 2024-02-23T00:13:10.802 INFO:teuthology.orchestra.run.smithi171.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.8 MB/s | 94 kB 00:00 2024-02-23T00:13:10.850 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 651 kB/s | 38 kB 00:00 2024-02-23T00:13:10.909 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.1 MB/s | 1.0 MB 00:00 2024-02-23T00:13:10.911 INFO:teuthology.orchestra.run.smithi171.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 12 MB/s | 4.1 MB 00:00 2024-02-23T00:13:11.034 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): crun-1.14.1-1.module_el8+951+32019cde.x 1.4 MB/s | 257 kB 00:00 2024-02-23T00:13:11.151 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.5 MB/s | 180 kB 00:00 2024-02-23T00:13:11.187 INFO:teuthology.orchestra.run.smithi171.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 11 MB/s | 9.7 MB 00:00 2024-02-23T00:13:11.201 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 497 kB/s | 25 kB 00:00 2024-02-23T00:13:11.237 INFO:teuthology.orchestra.run.smithi171.stdout:(10/13): toolbox-0.0.99.5-1.module_el8+847+7863 7.8 MB/s | 2.5 MB 00:00 2024-02-23T00:13:11.260 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-02-23T00:13:11.287 INFO:teuthology.orchestra.run.smithi171.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 821 kB/s | 41 kB 00:00 2024-02-23T00:13:11.313 INFO:teuthology.orchestra.run.smithi171.stdout:(12/13): udica-0.2.6-20.module_el8+804+f131391c 700 kB/s | 53 kB 00:00 2024-02-23T00:13:11.802 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 4.6 MB/s | 4.1 MB 00:00 2024-02-23T00:13:11.822 INFO:teuthology.orchestra.run.smithi171.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 8.6 MB/s | 8.8 MB 00:01 2024-02-23T00:13:11.822 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:13:11.822 INFO:teuthology.orchestra.run.smithi171.stdout:Total 18 MB/s | 27 MB 00:01 2024-02-23T00:13:12.090 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:13:12.102 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 8.4 MB/s | 2.5 MB 00:00 2024-02-23T00:13:12.111 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:13:12.111 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:13:12.144 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 1.2 MB/s | 53 kB 00:00 2024-02-23T00:13:12.186 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 987 kB/s | 41 kB 00:00 2024-02-23T00:13:12.345 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): buildah-1.33.5-1.module_el8+885+7da147 5.0 MB/s | 9.7 MB 00:01 2024-02-23T00:13:12.364 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:13:12.367 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:13:12.570 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 6.7 MB/s | 8.8 MB 00:01 2024-02-23T00:13:12.571 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:13:12.571 INFO:teuthology.orchestra.run.smithi087.stdout:Total 12 MB/s | 27 MB 00:02 2024-02-23T00:13:12.842 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:13:12.862 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:13:12.862 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:13:13.058 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:13:13.126 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:13:13.129 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:13:13.895 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:13:14.008 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-23T00:13:14.219 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-23T00:13:14.403 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-23T00:13:14.587 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-23T00:13:14.835 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-23T00:13:14.858 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-23T00:13:15.055 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-23T00:13:15.242 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-23T00:13:15.398 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-23T00:13:15.425 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-23T00:13:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-23T00:13:16.237 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-23T00:13:16.360 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-02-23T00:13:16.564 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-23T00:13:16.785 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : crun-1.14.1-1.module_el8+951+32019cde.x86_64 9/13 2024-02-23T00:13:17.236 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-02-23T00:13:17.255 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-23T00:13:17.466 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-23T00:13:17.467 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-23T00:13:17.696 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : crun-1.14.1-1.module_el8+951+32019cde.x86_64 9/13 2024-02-23T00:13:17.697 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-23T00:13:17.926 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-23T00:13:18.125 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-23T00:13:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-23T00:13:18.372 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-23T00:13:18.372 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-23T00:13:18.372 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : crun-1.14.1-1.module_el8+951+32019cde.x86_64 5/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-23T00:13:18.373 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-23T00:13:18.570 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-23T00:13:18.741 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-23T00:13:19.025 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-23T00:13:19.025 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:13:19.025 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: crun-1.14.1-1.module_el8+951+32019cde.x86_64 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-23T00:13:19.026 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:13:19.027 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:13:19.193 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-23T00:13:19.193 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-23T00:13:19.193 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-23T00:13:19.193 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-23T00:13:19.193 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-23T00:13:19.193 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : crun-1.14.1-1.module_el8+951+32019cde.x86_64 5/13 2024-02-23T00:13:19.194 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-02-23T00:13:19.194 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-23T00:13:19.194 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-23T00:13:19.194 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-23T00:13:19.194 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-23T00:13:19.194 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-23T00:13:19.194 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-23T00:13:19.328 DEBUG:teuthology.parallel:result is None 2024-02-23T00:13:19.943 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-23T00:13:19.943 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:13:19.943 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: crun-1.14.1-1.module_el8+951+32019cde.x86_64 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-23T00:13:19.944 INFO:teuthology.orchestra.run.smithi087.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-23T00:13:19.945 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:13:19.945 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:13:20.197 DEBUG:teuthology.parallel:result is None 2024-02-23T00:13:20.197 INFO:teuthology.run_tasks:Running task install... 2024-02-23T00:13:20.207 DEBUG:teuthology.task.install:project ceph 2024-02-23T00:13:20.208 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59'}} 2024-02-23T00:13:20.208 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59'} 2024-02-23T00:13:20.208 INFO:teuthology.task.install:Using flavor: default 2024-02-23T00:13:20.214 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-23T00:13:20.214 INFO:teuthology.task.install:extra packages: [] 2024-02-23T00:13:20.214 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59', 'tag': None, 'wait_for_package': False} 2024-02-23T00:13:20.215 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:13:20.217 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59', 'tag': None, 'wait_for_package': False} 2024-02-23T00:13:20.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:13:20.455 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/pacific/eb66ed921e744301e6863be6353618d63967ea59/centos/8/flavors/default/ 2024-02-23T00:13:20.455 INFO:teuthology.task.install.rpm:Package version is 16.2.14-550.geb66ed92 2024-02-23T00:13:20.472 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/pacific/eb66ed921e744301e6863be6353618d63967ea59/centos/8/flavors/default/ 2024-02-23T00:13:20.472 INFO:teuthology.task.install.rpm:Package version is 16.2.14-550.geb66ed92 2024-02-23T00:13:20.577 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/pacific/eb66ed921e744301e6863be6353618d63967ea59/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/pacific/eb66ed921e744301e6863be6353618d63967ea59/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/pacific/eb66ed921e744301e6863be6353618d63967ea59/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-23T00:13:20.577 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:13:20.578 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-23T00:13:20.588 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/pacific/eb66ed921e744301e6863be6353618d63967ea59/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/pacific/eb66ed921e744301e6863be6353618d63967ea59/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/pacific/eb66ed921e744301e6863be6353618d63967ea59/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-23T00:13:20.588 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:13:20.588 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-23T00:13:20.605 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-23T00:13:20.605 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/eb66ed921e744301e6863be6353618d63967ea59/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-23T00:13:20.621 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-23T00:13:20.621 DEBUG:teuthology.orchestra.run.smithi171:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/eb66ed921e744301e6863be6353618d63967ea59/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-23T00:13:20.672 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-23T00:13:20.692 DEBUG:teuthology.orchestra.run.smithi171:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-23T00:13:20.749 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-23T00:13:20.773 DEBUG:teuthology.orchestra.run.smithi171:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-23T00:13:20.815 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2024-02-23T00:13:20.818 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2024-02-23T00:13:20.840 INFO:teuthology.orchestra.run.smithi171.stdout:check_obsoletes = 1 2024-02-23T00:13:20.841 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum clean all 2024-02-23T00:13:21.084 INFO:teuthology.orchestra.run.smithi087.stdout:76 files removed 2024-02-23T00:13:21.106 INFO:teuthology.orchestra.run.smithi171.stdout:76 files removed 2024-02-23T00:13:21.110 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2024-02-23T00:13:21.134 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-radosgw 2024-02-23T00:13:21.886 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 204 kB/s | 76 kB 00:00 2024-02-23T00:13:21.894 INFO:teuthology.orchestra.run.smithi171.stdout:ceph packages for x86_64 217 kB/s | 76 kB 00:00 2024-02-23T00:13:22.153 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 67 kB/s | 15 kB 00:00 2024-02-23T00:13:22.262 INFO:teuthology.orchestra.run.smithi171.stdout:ceph noarch packages 47 kB/s | 15 kB 00:00 2024-02-23T00:13:22.445 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 5.5 kB/s | 1.4 kB 00:00 2024-02-23T00:13:22.613 INFO:teuthology.orchestra.run.smithi171.stdout:ceph source packages 4.5 kB/s | 1.4 kB 00:00 2024-02-23T00:13:25.371 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - AppStream 9.5 MB/s | 27 MB 00:02 2024-02-23T00:13:25.656 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - AppStream 9.1 MB/s | 27 MB 00:03 2024-02-23T00:13:31.064 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-02-23T00:13:31.356 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-02-23T00:13:32.881 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras 81 kB/s | 18 kB 00:00 2024-02-23T00:13:33.165 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - Extras 96 kB/s | 18 kB 00:00 2024-02-23T00:13:33.206 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras common packages 32 kB/s | 6.9 kB 00:00 2024-02-23T00:13:33.616 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-02-23T00:17:55.216 INFO:teuthology.orchestra.run.smithi171.stdout:CentOS Stream 8 - PowerTools 19 kB/s | 4.9 MB 04:21 2024-02-23T00:17:55.246 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - PowerTools 19 kB/s | 4.9 MB 04:21 2024-02-23T00:17:56.250 INFO:teuthology.orchestra.run.smithi171.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2024-02-23T00:17:56.297 INFO:teuthology.orchestra.run.smithi087.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-02-23T00:17:57.524 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-02-23T00:18:01.331 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 259 kB/s | 24 kB 00:00 2024-02-23T00:18:03.336 INFO:teuthology.orchestra.run.smithi171.stdout:Extra Packages for Enterprise Linux 2.3 MB/s | 16 MB 00:06 2024-02-23T00:18:04.926 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:18:05.732 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:18:05.732 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:18:05.732 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:18:05.732 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:16.2.14-550.geb66ed92.el8 ceph 12 M 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 3.7 M 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 4.2 M 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:16.2.14-550.geb66ed92.el8 ceph 6.1 M 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:16.2.14-550.geb66ed92.el8 ceph 24 M 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:16.2.14-550.geb66ed92.el8 ceph 26 k 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-23T00:18:05.733 INFO:teuthology.orchestra.run.smithi087.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 825 k 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 494 k 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 3.8 M 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:16.2.14-550.geb66ed92.el8 ceph 45 k 2024-02-23T00:18:05.734 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:16.2.14-550.geb66ed92.el8 ceph 99 k 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:16.2.14-550.geb66ed92.el8 ceph 215 k 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:16.2.14-550.geb66ed92.el8 ceph 389 k 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:16.2.14-550.geb66ed92.el8 ceph 369 k 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:16.2.14-550.geb66ed92.el8 ceph 114 k 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout:Install 21 Packages 2024-02-23T00:18:05.735 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2024-02-23T00:18:05.736 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:05.736 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 58 M 2024-02-23T00:18:05.736 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:18:06.690 INFO:teuthology.orchestra.run.smithi087.stdout:(1/23): ceph-base-16.2.14-550.geb66ed92.el8.x86 4.2 MB/s | 6.1 MB 00:01 2024-02-23T00:18:06.775 INFO:teuthology.orchestra.run.smithi087.stdout:(2/23): ceph-selinux-16.2.14-550.geb66ed92.el8. 308 kB/s | 26 kB 00:00 2024-02-23T00:18:07.132 INFO:teuthology.orchestra.run.smithi087.stdout:(3/23): ceph-common-16.2.14-550.geb66ed92.el8.x 13 MB/s | 24 MB 00:01 2024-02-23T00:18:07.174 INFO:teuthology.orchestra.run.smithi087.stdout:(4/23): libcephfs2-16.2.14-550.geb66ed92.el8.x8 2.0 MB/s | 825 kB 00:00 2024-02-23T00:18:07.175 INFO:teuthology.orchestra.run.smithi171.stdout:lab-extras 491 kB/s | 24 kB 00:00 2024-02-23T00:18:07.199 INFO:teuthology.orchestra.run.smithi087.stdout:(5/23): libradosstriper1-16.2.14-550.geb66ed92. 7.2 MB/s | 494 kB 00:00 2024-02-23T00:18:07.241 INFO:teuthology.orchestra.run.smithi087.stdout:(6/23): python3-ceph-argparse-16.2.14-550.geb66 1.1 MB/s | 45 kB 00:00 2024-02-23T00:18:07.383 INFO:teuthology.orchestra.run.smithi087.stdout:(7/23): ceph-radosgw-16.2.14-550.geb66ed92.el8. 5.4 MB/s | 12 MB 00:02 2024-02-23T00:18:07.409 INFO:teuthology.orchestra.run.smithi087.stdout:(8/23): python3-ceph-common-16.2.14-550.geb66ed 591 kB/s | 99 kB 00:00 2024-02-23T00:18:07.451 INFO:teuthology.orchestra.run.smithi087.stdout:(9/23): python3-cephfs-16.2.14-550.geb66ed92.el 3.1 MB/s | 215 kB 00:00 2024-02-23T00:18:07.476 INFO:teuthology.orchestra.run.smithi087.stdout:(10/23): python3-rados-16.2.14-550.geb66ed92.el 5.7 MB/s | 389 kB 00:00 2024-02-23T00:18:07.535 INFO:teuthology.orchestra.run.smithi087.stdout:(11/23): python3-rgw-16.2.14-550.geb66ed92.el8. 1.9 MB/s | 114 kB 00:00 2024-02-23T00:18:07.560 INFO:teuthology.orchestra.run.smithi087.stdout:(12/23): python3-rbd-16.2.14-550.geb66ed92.el8. 3.3 MB/s | 369 kB 00:00 2024-02-23T00:18:07.710 INFO:teuthology.orchestra.run.smithi087.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.7 MB/s | 259 kB 00:00 2024-02-23T00:18:07.769 INFO:teuthology.orchestra.run.smithi087.stdout:(14/23): librgw2-16.2.14-550.geb66ed92.el8.x86_ 6.4 MB/s | 3.8 MB 00:00 2024-02-23T00:18:07.794 INFO:teuthology.orchestra.run.smithi087.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 526 kB/s | 44 kB 00:00 2024-02-23T00:18:07.819 INFO:teuthology.orchestra.run.smithi087.stdout:(16/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.3 MB/s | 677 kB 00:00 2024-02-23T00:18:07.945 INFO:teuthology.orchestra.run.smithi087.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 1.4 MB/s | 181 kB 00:00 2024-02-23T00:18:07.970 INFO:teuthology.orchestra.run.smithi087.stdout:(18/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 234 kB/s | 47 kB 00:00 2024-02-23T00:18:07.995 INFO:teuthology.orchestra.run.smithi087.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2024-02-23T00:18:08.021 INFO:teuthology.orchestra.run.smithi087.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2024-02-23T00:18:08.079 INFO:teuthology.orchestra.run.smithi087.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 691 kB/s | 75 kB 00:00 2024-02-23T00:18:08.324 INFO:teuthology.orchestra.run.smithi087.stdout:(22/23): librbd1-16.2.14-550.geb66ed92.el8.x86_ 14 MB/s | 4.2 MB 00:00 2024-02-23T00:18:08.530 INFO:teuthology.orchestra.run.smithi087.stdout:(23/23): librados2-16.2.14-550.geb66ed92.el8.x8 7.0 MB/s | 3.7 MB 00:00 2024-02-23T00:18:08.532 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:18:08.532 INFO:teuthology.orchestra.run.smithi087.stdout:Total 16 MB/s | 58 MB 00:03 2024-02-23T00:18:08.574 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:18:08.622 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:18:08.622 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:18:08.961 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:18:08.964 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:18:10.008 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:18:10.201 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-23T00:18:10.412 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-23T00:18:10.739 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:18:10.741 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:18:10.741 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:18:10.741 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:18:10.741 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:18:10.741 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw x86_64 2:16.2.14-550.geb66ed92.el8 ceph 12 M 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrading: 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: librados2 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 3.7 M 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 4.2 M 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base x86_64 2:16.2.14-550.geb66ed92.el8 ceph 6.1 M 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common x86_64 2:16.2.14-550.geb66ed92.el8 ceph 24 M 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux x86_64 2:16.2.14-550.geb66ed92.el8 ceph 26 k 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 825 k 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-23T00:18:10.742 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 494 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2 x86_64 2:16.2.14-550.geb66ed92.el8 ceph 3.8 M 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse x86_64 2:16.2.14-550.geb66ed92.el8 ceph 45 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common x86_64 2:16.2.14-550.geb66ed92.el8 ceph 99 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs x86_64 2:16.2.14-550.geb66ed92.el8 ceph 215 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados x86_64 2:16.2.14-550.geb66ed92.el8 ceph 389 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd x86_64 2:16.2.14-550.geb66ed92.el8 ceph 369 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw x86_64 2:16.2.14-550.geb66ed92.el8 ceph 114 k 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:18:10.743 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:18:10.744 INFO:teuthology.orchestra.run.smithi171.stdout:Install 21 Packages 2024-02-23T00:18:10.744 INFO:teuthology.orchestra.run.smithi171.stdout:Upgrade 2 Packages 2024-02-23T00:18:10.744 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:18:10.744 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 58 M 2024-02-23T00:18:10.744 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:18:10.804 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-23T00:18:10.995 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:16.2.14-550.geb66ed92.el8.x86_64 3/25 2024-02-23T00:18:11.087 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.14-550.geb66ed92.el8.x86_64 3/25 2024-02-23T00:18:11.368 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:16.2.14-550.geb66ed92.el8.x86_64 4/25 2024-02-23T00:18:11.670 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 5/25 2024-02-23T00:18:11.718 INFO:teuthology.orchestra.run.smithi171.stdout:(1/23): ceph-base-16.2.14-550.geb66ed92.el8.x86 8.0 MB/s | 6.1 MB 00:00 2024-02-23T00:18:11.785 INFO:teuthology.orchestra.run.smithi171.stdout:(2/23): ceph-selinux-16.2.14-550.geb66ed92.el8. 384 kB/s | 26 kB 00:00 2024-02-23T00:18:11.910 INFO:teuthology.orchestra.run.smithi171.stdout:(3/23): libcephfs2-16.2.14-550.geb66ed92.el8.x8 6.4 MB/s | 825 kB 00:00 2024-02-23T00:18:11.994 INFO:teuthology.orchestra.run.smithi171.stdout:(4/23): libradosstriper1-16.2.14-550.geb66ed92. 5.8 MB/s | 494 kB 00:00 2024-02-23T00:18:12.077 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 5/25 2024-02-23T00:18:12.128 INFO:teuthology.orchestra.run.smithi171.stdout:(5/23): ceph-radosgw-16.2.14-550.geb66ed92.el8. 9.9 MB/s | 12 MB 00:01 2024-02-23T00:18:12.178 INFO:teuthology.orchestra.run.smithi171.stdout:(6/23): python3-ceph-argparse-16.2.14-550.geb66 902 kB/s | 45 kB 00:00 2024-02-23T00:18:12.237 INFO:teuthology.orchestra.run.smithi171.stdout:(7/23): python3-ceph-common-16.2.14-550.geb66ed 1.7 MB/s | 99 kB 00:00 2024-02-23T00:18:12.253 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 6/25 2024-02-23T00:18:12.300 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 6/25 2024-02-23T00:18:12.329 INFO:teuthology.orchestra.run.smithi171.stdout:(8/23): librgw2-16.2.14-550.geb66ed92.el8.x86_6 11 MB/s | 3.8 MB 00:00 2024-02-23T00:18:12.354 INFO:teuthology.orchestra.run.smithi171.stdout:(9/23): python3-cephfs-16.2.14-550.geb66ed92.el 1.8 MB/s | 215 kB 00:00 2024-02-23T00:18:12.389 INFO:teuthology.orchestra.run.smithi171.stdout:(10/23): python3-rados-16.2.14-550.geb66ed92.el 6.4 MB/s | 389 kB 00:00 2024-02-23T00:18:12.439 INFO:teuthology.orchestra.run.smithi171.stdout:(11/23): python3-rbd-16.2.14-550.geb66ed92.el8. 4.3 MB/s | 369 kB 00:00 2024-02-23T00:18:12.464 INFO:teuthology.orchestra.run.smithi171.stdout:(12/23): python3-rgw-16.2.14-550.geb66ed92.el8. 1.5 MB/s | 114 kB 00:00 2024-02-23T00:18:12.562 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-23T00:18:12.623 INFO:teuthology.orchestra.run.smithi171.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.6 MB/s | 259 kB 00:00 2024-02-23T00:18:12.658 INFO:teuthology.orchestra.run.smithi171.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 3.0 MB/s | 677 kB 00:00 2024-02-23T00:18:12.674 INFO:teuthology.orchestra.run.smithi171.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 867 kB/s | 44 kB 00:00 2024-02-23T00:18:12.833 INFO:teuthology.orchestra.run.smithi171.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 268 kB/s | 47 kB 00:00 2024-02-23T00:18:13.056 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-23T00:18:13.066 INFO:teuthology.orchestra.run.smithi171.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 781 kB/s | 306 kB 00:00 2024-02-23T00:18:13.133 INFO:teuthology.orchestra.run.smithi171.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 880 kB/s | 59 kB 00:00 2024-02-23T00:18:13.175 INFO:teuthology.orchestra.run.smithi171.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 528 kB/s | 181 kB 00:00 2024-02-23T00:18:13.203 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:16.2.14-550.geb66ed92.el8.x86_64 9/25 2024-02-23T00:18:13.209 INFO:teuthology.orchestra.run.smithi171.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 995 kB/s | 75 kB 00:00 2024-02-23T00:18:13.305 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:16.2.14-550.geb66ed92.el8.x86_64 9/25 2024-02-23T00:18:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-23T00:18:13.568 INFO:teuthology.orchestra.run.smithi171.stdout:(21/23): librados2-16.2.14-550.geb66ed92.el8.x8 9.6 MB/s | 3.7 MB 00:00 2024-02-23T00:18:13.635 INFO:teuthology.orchestra.run.smithi171.stdout:(22/23): librbd1-16.2.14-550.geb66ed92.el8.x86_ 9.8 MB/s | 4.2 MB 00:00 2024-02-23T00:18:13.666 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:16.2.14-550.geb66ed92.el8. 11/25 2024-02-23T00:18:13.865 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:16.2.14-550.geb66ed92.el8.x86_64 12/25 2024-02-23T00:18:14.063 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:16.2.14-550.geb66ed92.el8.x86_64 13/25 2024-02-23T00:18:14.311 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:16.2.14-550.geb66ed92.el8.x86_64 14/25 2024-02-23T00:18:14.469 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:16.2.14-550.geb66ed92.el8.x86_6 15/25 2024-02-23T00:18:14.569 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:16.2.14-550.geb66ed92.el8.x86_6 15/25 2024-02-23T00:18:14.768 INFO:teuthology.orchestra.run.smithi171.stdout:(23/23): ceph-common-16.2.14-550.geb66ed92.el8. 6.4 MB/s | 24 MB 00:03 2024-02-23T00:18:14.769 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-23T00:18:14.773 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:18:14.773 INFO:teuthology.orchestra.run.smithi171.stdout:Total 14 MB/s | 58 MB 00:04 2024-02-23T00:18:14.815 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:18:14.862 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:18:14.862 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:18:14.961 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-23T00:18:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-23T00:18:15.214 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:18:15.216 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:18:15.387 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:16.2.14-550.geb66ed92.el8.x8 19/25 2024-02-23T00:18:16.201 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:18:16.420 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-23T00:18:16.569 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-23T00:18:16.972 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-23T00:18:17.127 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librados2-2:16.2.14-550.geb66ed92.el8.x86_64 3/25 2024-02-23T00:18:17.211 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-2:16.2.14-550.geb66ed92.el8.x86_64 3/25 2024-02-23T00:18:17.467 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rados-2:16.2.14-550.geb66ed92.el8.x86_64 4/25 2024-02-23T00:18:17.685 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 5/25 2024-02-23T00:18:17.908 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 20/25 2024-02-23T00:18:18.078 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 5/25 2024-02-23T00:18:18.145 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 20/25 2024-02-23T00:18:18.243 INFO:teuthology.orchestra.run.smithi171.stdout: Upgrading : librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 6/25 2024-02-23T00:18:18.292 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 6/25 2024-02-23T00:18:18.478 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-23T00:18:18.852 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 20/25 2024-02-23T00:18:18.967 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-23T00:18:19.110 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librgw2-2:16.2.14-550.geb66ed92.el8.x86_64 9/25 2024-02-23T00:18:19.157 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 21/25 2024-02-23T00:18:19.190 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librgw2-2:16.2.14-550.geb66ed92.el8.x86_64 9/25 2024-02-23T00:18:19.266 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 21/25 2024-02-23T00:18:19.374 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-23T00:18:19.377 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:16.2.14-550.geb66ed92.el8.x86_64 22/25 2024-02-23T00:18:19.540 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-argparse-2:16.2.14-550.geb66ed92.el8. 11/25 2024-02-23T00:18:19.739 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cephfs-2:16.2.14-550.geb66ed92.el8.x86_64 12/25 2024-02-23T00:18:19.953 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rgw-2:16.2.14-550.geb66ed92.el8.x86_64 13/25 2024-02-23T00:18:20.201 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rbd-2:16.2.14-550.geb66ed92.el8.x86_64 14/25 2024-02-23T00:18:20.351 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libradosstriper1-2:16.2.14-550.geb66ed92.el8.x86_6 15/25 2024-02-23T00:18:20.405 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libradosstriper1-2:16.2.14-550.geb66ed92.el8.x86_6 15/25 2024-02-23T00:18:20.611 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-23T00:18:20.793 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-23T00:18:20.990 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-23T00:18:21.206 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-ceph-common-2:16.2.14-550.geb66ed92.el8.x8 19/25 2024-02-23T00:18:23.708 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 20/25 2024-02-23T00:18:23.935 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 20/25 2024-02-23T00:18:24.634 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 20/25 2024-02-23T00:18:24.925 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 21/25 2024-02-23T00:18:25.013 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 21/25 2024-02-23T00:18:25.159 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-selinux-2:16.2.14-550.geb66ed92.el8.x86_64 22/25 2024-02-23T00:18:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:16.2.14-550.geb66ed92.el8.x86_64 22/25 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /dev 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /run 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /dev 2024-02-23T00:18:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:39.779 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:16.2.14-550.geb66ed92.el8.x86_64 23/25 2024-02-23T00:18:39.858 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-550.geb66ed92.el8.x86_64 23/25 2024-02-23T00:18:39.858 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-23T00:18:40.118 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-23T00:18:40.118 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-23T00:18:40.836 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-23T00:18:40.836 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 1/25 2024-02-23T00:18:40.836 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 2/25 2024-02-23T00:18:40.836 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:16.2.14-550.geb66ed92.el8.x86_64 3/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:16.2.14-550.geb66ed92.el8.x86_64 4/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 5/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:16.2.14-550.geb66ed92.el8.x86_6 6/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:16.2.14-550.geb66ed92.el8.x86_64 7/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:16.2.14-550.geb66ed92.el8. 8/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:16.2.14-550.geb66ed92.el8.x8 9/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:16.2.14-550.geb66ed92.el8.x86_64 10/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:16.2.14-550.geb66ed92.el8.x86_64 11/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:16.2.14-550.geb66ed92.el8.x86_64 12/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:16.2.14-550.geb66ed92.el8.x86_64 13/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-23T00:18:40.837 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-23T00:18:40.838 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-23T00:18:40.838 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-23T00:18:40.838 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-23T00:18:40.838 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-23T00:18:40.838 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-23T00:18:40.838 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.14-550.geb66ed92.el8.x86_64 22/25 2024-02-23T00:18:40.838 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-23T00:18:40.838 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 24/25 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-23T00:18:41.709 INFO:teuthology.orchestra.run.smithi087.stdout: leveldb-1.22-1.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.710 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:41.711 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:41.711 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:18:41.988 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2024-02-23T00:18:42.743 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:41 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:18:44.309 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:18:44.310 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-selinux-2:16.2.14-550.geb66ed92.el8.x86_64 22/25 2024-02-23T00:18:44.310 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /sys 2024-02-23T00:18:44.310 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /proc 2024-02-23T00:18:44.310 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /dev 2024-02-23T00:18:44.310 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /run 2024-02-23T00:18:44.311 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /mnt 2024-02-23T00:18:44.311 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /var/tmp 2024-02-23T00:18:44.311 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /home 2024-02-23T00:18:44.311 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /tmp 2024-02-23T00:18:44.311 INFO:teuthology.orchestra.run.smithi171.stdout:skipping the directory /dev 2024-02-23T00:18:44.311 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:18:44.312 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:18:44.312 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:18:44.312 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:18:44.312 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:18:44.312 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:16.2.14-550.geb66ed92.el8 ceph 52 M 2024-02-23T00:18:44.312 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:16.2.14-550.geb66ed92.el8 ceph 174 k 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 53 M 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 197 M 2024-02-23T00:18:44.313 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:18:44.515 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 1.2 MB/s | 203 kB 00:00 2024-02-23T00:18:44.557 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): libcephsqlite-16.2.14-550.geb66ed92.el8. 858 kB/s | 174 kB 00:00 2024-02-23T00:18:44.582 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.7 MB/s | 188 kB 00:00 2024-02-23T00:18:44.624 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-02-23T00:18:44.716 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2024-02-23T00:18:45.497 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-radosgw-2:16.2.14-550.geb66ed92.el8.x86_64 23/25 2024-02-23T00:18:45.579 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-550.geb66ed92.el8.x86_64 23/25 2024-02-23T00:18:45.580 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-23T00:18:45.808 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-23T00:18:45.808 INFO:teuthology.orchestra.run.smithi171.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-23T00:18:46.473 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-23T00:18:46.473 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 1/25 2024-02-23T00:18:46.473 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 2/25 2024-02-23T00:18:46.473 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-radosgw-2:16.2.14-550.geb66ed92.el8.x86_64 3/25 2024-02-23T00:18:46.473 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-selinux-2:16.2.14-550.geb66ed92.el8.x86_64 4/25 2024-02-23T00:18:46.474 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 5/25 2024-02-23T00:18:46.474 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libradosstriper1-2:16.2.14-550.geb66ed92.el8.x86_6 6/25 2024-02-23T00:18:46.474 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librgw2-2:16.2.14-550.geb66ed92.el8.x86_64 7/25 2024-02-23T00:18:46.477 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-argparse-2:16.2.14-550.geb66ed92.el8. 8/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-ceph-common-2:16.2.14-550.geb66ed92.el8.x8 9/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cephfs-2:16.2.14-550.geb66ed92.el8.x86_64 10/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rados-2:16.2.14-550.geb66ed92.el8.x86_64 11/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rbd-2:16.2.14-550.geb66ed92.el8.x86_64 12/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rgw-2:16.2.14-550.geb66ed92.el8.x86_64 13/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-23T00:18:46.478 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-23T00:18:46.479 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-23T00:18:46.479 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-23T00:18:46.479 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-23T00:18:46.479 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-2:16.2.14-550.geb66ed92.el8.x86_64 22/25 2024-02-23T00:18:46.479 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-23T00:18:46.479 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 24/25 2024-02-23T00:18:46.824 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-test-16.2.14-550.geb66ed92.el8.x86_ 21 MB/s | 52 MB 00:02 2024-02-23T00:18:46.826 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:18:46.827 INFO:teuthology.orchestra.run.smithi087.stdout:Total 21 MB/s | 53 MB 00:02 2024-02-23T00:18:46.868 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:18:46.888 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:18:46.889 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:18:47.202 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:18:47.205 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:18:47.300 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-23T00:18:47.300 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:18:47.300 INFO:teuthology.orchestra.run.smithi171.stdout:Upgraded: 2024-02-23T00:18:47.300 INFO:teuthology.orchestra.run.smithi171.stdout: librados2-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.300 INFO:teuthology.orchestra.run.smithi171.stdout: librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-radosgw-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-selinux-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: leveldb-1.22-1.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: libradosstriper1-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: librgw2-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.301 INFO:teuthology.orchestra.run.smithi171.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-argparse-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout: python3-ceph-common-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cephfs-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rados-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rbd-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rgw-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:18:47.302 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:18:47.569 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-test 2024-02-23T00:18:47.726 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:18:47.942 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-23T00:18:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-23T00:18:48.290 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-23T00:18:48.325 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:41 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:18:48.371 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-23T00:18:48.604 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-23T00:18:48.782 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:16.2.14-550.geb66ed92.el8.x86_64 5/6 2024-02-23T00:18:49.891 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:18:49.892 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:18:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:18:49.893 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:18:49.893 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:18:49.893 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test x86_64 2:16.2.14-550.geb66ed92.el8 ceph 52 M 2024-02-23T00:18:49.893 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-23T00:18:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-23T00:18:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite x86_64 2:16.2.14-550.geb66ed92.el8 ceph 174 k 2024-02-23T00:18:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-23T00:18:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-23T00:18:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-23T00:18:49.894 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:18:49.895 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:18:49.895 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:18:49.895 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2024-02-23T00:18:49.895 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:18:49.896 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 53 M 2024-02-23T00:18:49.896 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 197 M 2024-02-23T00:18:49.896 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:18:50.092 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 1.3 MB/s | 203 kB 00:00 2024-02-23T00:18:50.125 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): libcephsqlite-16.2.14-550.geb66ed92.el8. 915 kB/s | 174 kB 00:00 2024-02-23T00:18:50.151 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-02-23T00:18:50.192 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-02-23T00:18:50.293 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2024-02-23T00:18:52.577 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): ceph-test-16.2.14-550.geb66ed92.el8.x86_ 20 MB/s | 52 MB 00:02 2024-02-23T00:18:52.866 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:18:52.866 INFO:teuthology.orchestra.run.smithi171.stdout:Total 20 MB/s | 53 MB 00:02 2024-02-23T00:18:52.867 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:18:52.867 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:18:52.867 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:18:52.949 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:18:52.960 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:18:53.399 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:16.2.14-550.geb66ed92.el8.x86_64 5/6 2024-02-23T00:18:53.437 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:18:53.671 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:16.2.14-550.geb66ed92.el8.x86_64 6/6 2024-02-23T00:18:53.675 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-23T00:18:53.865 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-23T00:18:53.963 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:16.2.14-550.geb66ed92.el8.x86_64 6/6 2024-02-23T00:18:53.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:16.2.14-550.geb66ed92.el8.x86_64 1/6 2024-02-23T00:18:53.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:16.2.14-550.geb66ed92.el8.x86_64 2/6 2024-02-23T00:18:53.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-23T00:18:53.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-23T00:18:53.964 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-23T00:18:53.996 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-23T00:18:54.082 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-23T00:18:54.261 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-23T00:18:54.421 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephsqlite-2:16.2.14-550.geb66ed92.el8.x86_64 5/6 2024-02-23T00:18:54.457 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-23T00:18:54.457 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:54.457 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:18:54.457 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:54.457 INFO:teuthology.orchestra.run.smithi087.stdout: jq-1.6-8.el8.x86_64 2024-02-23T00:18:54.457 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:54.457 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-23T00:18:54.458 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-23T00:18:54.458 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-23T00:18:54.458 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:54.458 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:18:54.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2024-02-23T00:18:55.447 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:54 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:18:57.011 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:18:57.014 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:16.2.14-550.geb66ed92.el8 ceph 6.4 k 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:16.2.14-550.geb66ed92.el8 ceph 2.4 M 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:16.2.14-550.geb66ed92.el8 ceph 1.6 M 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 245 k 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:16.2.14-550.geb66ed92.el8 ceph 5.2 M 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:16.2.14-550.geb66ed92.el8 ceph 19 M 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-23T00:18:57.015 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-23T00:18:57.016 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout:Install 32 Packages 2024-02-23T00:18:57.017 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:18:57.018 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 32 M 2024-02-23T00:18:57.018 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 104 M 2024-02-23T00:18:57.018 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:18:57.425 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-16.2.14-550.geb66ed92.el8.x86_64.r 52 kB/s | 6.4 kB 00:00 2024-02-23T00:18:57.633 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-mgr-16.2.14-550.geb66ed92.el8.x86_ 4.9 MB/s | 1.6 MB 00:00 2024-02-23T00:18:57.700 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): ceph-mds-16.2.14-550.geb66ed92.el8.x86_ 6.1 MB/s | 2.4 MB 00:00 2024-02-23T00:18:57.759 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): ceph-mgr-modules-core-16.2.14-550.geb66 4.1 MB/s | 245 kB 00:00 2024-02-23T00:18:57.909 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.0 MB/s | 157 kB 00:00 2024-02-23T00:18:57.959 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 780 kB/s | 39 kB 00:00 2024-02-23T00:18:58.027 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2024-02-23T00:18:58.069 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2024-02-23T00:18:58.136 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 6.7 MB/s | 457 kB 00:00 2024-02-23T00:18:58.328 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): ceph-mon-16.2.14-550.geb66ed92.el8.x86 5.8 MB/s | 5.2 MB 00:00 2024-02-23T00:18:58.378 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 249 kB 00:00 2024-02-23T00:18:58.528 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 728 kB/s | 109 kB 00:00 2024-02-23T00:18:58.703 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): ceph-osd-16.2.14-550.geb66ed92.el8.x86 17 MB/s | 19 MB 00:01 2024-02-23T00:18:58.862 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.2 MB/s | 641 kB 00:00 2024-02-23T00:18:59.014 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephsqlite-2:16.2.14-550.geb66ed92.el8.x86_64 5/6 2024-02-23T00:18:59.046 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 84 kB/s | 44 kB 00:00 2024-02-23T00:18:59.196 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 376 kB/s | 185 kB 00:00 2024-02-23T00:18:59.260 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-test-2:16.2.14-550.geb66ed92.el8.x86_64 6/6 2024-02-23T00:18:59.330 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2024-02-23T00:18:59.363 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 345 kB/s | 173 kB 00:00 2024-02-23T00:18:59.388 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2024-02-23T00:18:59.439 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2024-02-23T00:18:59.489 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 387 kB/s | 49 kB 00:00 2024-02-23T00:18:59.515 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 468 kB/s | 59 kB 00:00 2024-02-23T00:18:59.533 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-test-2:16.2.14-550.geb66ed92.el8.x86_64 6/6 2024-02-23T00:18:59.533 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-test-2:16.2.14-550.geb66ed92.el8.x86_64 1/6 2024-02-23T00:18:59.533 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephsqlite-2:16.2.14-550.geb66ed92.el8.x86_64 2/6 2024-02-23T00:18:59.534 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-23T00:18:59.534 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-23T00:18:59.534 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-23T00:18:59.635 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 107 kB/s | 16 kB 00:00 2024-02-23T00:18:59.665 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 124 kB/s | 19 kB 00:00 2024-02-23T00:18:59.690 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2024-02-23T00:18:59.740 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 229 kB/s | 24 kB 00:00 2024-02-23T00:18:59.782 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 250 kB/s | 29 kB 00:00 2024-02-23T00:18:59.808 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2024-02-23T00:18:59.916 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 789 kB/s | 85 kB 00:00 2024-02-23T00:18:59.941 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-02-23T00:18:59.952 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-23T00:18:59.952 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:18:59.952 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:18:59.952 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-test-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:59.952 INFO:teuthology.orchestra.run.smithi171.stdout: jq-1.6-8.el8.x86_64 2024-02-23T00:18:59.952 INFO:teuthology.orchestra.run.smithi171.stdout: libcephsqlite-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:18:59.953 INFO:teuthology.orchestra.run.smithi171.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-23T00:18:59.953 INFO:teuthology.orchestra.run.smithi171.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-23T00:18:59.953 INFO:teuthology.orchestra.run.smithi171.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-23T00:18:59.953 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:18:59.953 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:18:59.967 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.1 MB/s | 255 kB 00:00 2024-02-23T00:19:00.017 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-02-23T00:19:00.019 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:00.019 INFO:teuthology.orchestra.run.smithi087.stdout:Total 11 MB/s | 32 MB 00:03 2024-02-23T00:19:00.100 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:19:00.139 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:19:00.139 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:19:00.203 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph 2024-02-23T00:19:00.462 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:19:00.465 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:19:00.971 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:00:53 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:19:01.610 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:19:01.860 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-23T00:19:02.063 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-23T00:19:02.355 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-23T00:19:02.537 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:19:02.540 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:02.540 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:19:02.540 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:02.540 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:19:02.540 INFO:teuthology.orchestra.run.smithi171.stdout: ceph x86_64 2:16.2.14-550.geb66ed92.el8 ceph 6.4 k 2024-02-23T00:19:02.540 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-23T00:19:02.540 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds x86_64 2:16.2.14-550.geb66ed92.el8 ceph 2.4 M 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr x86_64 2:16.2.14-550.geb66ed92.el8 ceph 1.6 M 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 245 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon x86_64 2:16.2.14-550.geb66ed92.el8 ceph 5.2 M 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd x86_64 2:16.2.14-550.geb66ed92.el8 ceph 19 M 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-23T00:19:02.541 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-23T00:19:02.542 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-23T00:19:02.543 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-23T00:19:02.543 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:02.543 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:19:02.543 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:02.543 INFO:teuthology.orchestra.run.smithi171.stdout:Install 32 Packages 2024-02-23T00:19:02.543 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:02.543 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 32 M 2024-02-23T00:19:02.544 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 104 M 2024-02-23T00:19:02.544 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:19:02.578 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-23T00:19:02.750 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-23T00:19:02.972 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-23T00:19:03.141 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-23T00:19:03.360 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-23T00:19:03.577 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-23T00:19:03.840 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-23T00:19:04.069 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-23T00:19:04.276 INFO:teuthology.orchestra.run.smithi171.stdout:(1/32): ceph-16.2.14-550.geb66ed92.el8.x86_64.r 32 kB/s | 6.4 kB 00:00 2024-02-23T00:19:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-23T00:19:04.560 INFO:teuthology.orchestra.run.smithi171.stdout:(2/32): ceph-mgr-16.2.14-550.geb66ed92.el8.x86_ 3.4 MB/s | 1.6 MB 00:00 2024-02-23T00:19:04.610 INFO:teuthology.orchestra.run.smithi171.stdout:(3/32): ceph-mds-16.2.14-550.geb66ed92.el8.x86_ 4.6 MB/s | 2.4 MB 00:00 2024-02-23T00:19:04.694 INFO:teuthology.orchestra.run.smithi171.stdout:(4/32): ceph-mgr-modules-core-16.2.14-550.geb66 2.9 MB/s | 245 kB 00:00 2024-02-23T00:19:04.701 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-23T00:19:04.836 INFO:teuthology.orchestra.run.smithi171.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-02-23T00:19:04.878 INFO:teuthology.orchestra.run.smithi171.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 932 kB/s | 39 kB 00:00 2024-02-23T00:19:04.920 INFO:teuthology.orchestra.run.smithi171.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2024-02-23T00:19:04.949 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-23T00:19:04.962 INFO:teuthology.orchestra.run.smithi171.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2024-02-23T00:19:05.032 INFO:teuthology.orchestra.run.smithi171.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 6.7 MB/s | 457 kB 00:00 2024-02-23T00:19:05.162 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-23T00:19:05.230 INFO:teuthology.orchestra.run.smithi171.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.2 MB/s | 249 kB 00:00 2024-02-23T00:19:05.314 INFO:teuthology.orchestra.run.smithi171.stdout:(11/32): ceph-mon-16.2.14-550.geb66ed92.el8.x86 5.0 MB/s | 5.2 MB 00:01 2024-02-23T00:19:05.453 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-23T00:19:05.656 INFO:teuthology.orchestra.run.smithi171.stdout:(12/32): ceph-osd-16.2.14-550.geb66ed92.el8.x86 17 MB/s | 19 MB 00:01 2024-02-23T00:19:05.672 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-23T00:19:05.681 INFO:teuthology.orchestra.run.smithi171.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 297 kB/s | 109 kB 00:00 2024-02-23T00:19:05.807 INFO:teuthology.orchestra.run.smithi171.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.1 MB/s | 641 kB 00:00 2024-02-23T00:19:05.932 INFO:teuthology.orchestra.run.smithi171.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 158 kB/s | 44 kB 00:00 2024-02-23T00:19:05.988 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-23T00:19:06.048 INFO:teuthology.orchestra.run.smithi171.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 504 kB/s | 185 kB 00:00 2024-02-23T00:19:06.132 INFO:teuthology.orchestra.run.smithi171.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 133 kB/s | 11 kB 00:00 2024-02-23T00:19:06.166 INFO:teuthology.orchestra.run.smithi171.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 481 kB/s | 173 kB 00:00 2024-02-23T00:19:06.200 INFO:teuthology.orchestra.run.smithi171.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-02-23T00:19:06.235 INFO:teuthology.orchestra.run.smithi171.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 172 kB/s | 18 kB 00:00 2024-02-23T00:19:06.243 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-23T00:19:06.259 INFO:teuthology.orchestra.run.smithi171.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 526 kB/s | 49 kB 00:00 2024-02-23T00:19:06.276 INFO:teuthology.orchestra.run.smithi171.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 774 kB/s | 59 kB 00:00 2024-02-23T00:19:06.335 INFO:teuthology.orchestra.run.smithi171.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 206 kB/s | 16 kB 00:00 2024-02-23T00:19:06.352 INFO:teuthology.orchestra.run.smithi171.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-02-23T00:19:06.394 INFO:teuthology.orchestra.run.smithi171.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2024-02-23T00:19:06.411 INFO:teuthology.orchestra.run.smithi171.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 315 kB/s | 24 kB 00:00 2024-02-23T00:19:06.423 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-23T00:19:06.432 INFO:teuthology.orchestra.run.smithi171.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 385 kB/s | 29 kB 00:00 2024-02-23T00:19:06.470 INFO:teuthology.orchestra.run.smithi171.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 363 kB/s | 27 kB 00:00 2024-02-23T00:19:06.528 INFO:teuthology.orchestra.run.smithi171.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2024-02-23T00:19:06.562 INFO:teuthology.orchestra.run.smithi171.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 927 kB/s | 85 kB 00:00 2024-02-23T00:19:06.587 INFO:teuthology.orchestra.run.smithi171.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.4 MB/s | 255 kB 00:00 2024-02-23T00:19:06.613 INFO:teuthology.orchestra.run.smithi171.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 273 kB/s | 23 kB 00:00 2024-02-23T00:19:06.613 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-23T00:19:06.615 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:06.615 INFO:teuthology.orchestra.run.smithi171.stdout:Total 7.8 MB/s | 32 MB 00:04 2024-02-23T00:19:06.697 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:19:06.737 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:19:06.737 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:19:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-23T00:19:07.058 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:19:07.060 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:19:07.091 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-23T00:19:07.315 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-23T00:19:07.625 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-23T00:19:07.965 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-23T00:19:08.154 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:19:08.328 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-550.geb66ed92.el8. 27/32 2024-02-23T00:19:08.365 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-23T00:19:08.542 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 28/32 2024-02-23T00:19:08.585 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-23T00:19:08.870 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-23T00:19:09.092 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-23T00:19:09.281 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-23T00:19:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-23T00:19:09.655 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-23T00:19:09.833 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-23T00:19:10.034 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-23T00:19:10.056 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 28/32 2024-02-23T00:19:10.237 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:16.2.14-550.geb66ed92.el8.x86_64 29/32 2024-02-23T00:19:10.287 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-23T00:19:10.501 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-23T00:19:10.679 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:16.2.14-550.geb66ed92.el8.x86_64 29/32 2024-02-23T00:19:10.774 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-23T00:19:10.887 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:16.2.14-550.geb66ed92.el8.x86_64 30/32 2024-02-23T00:19:11.116 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:16.2.14-550.geb66ed92.el8.x86_64 30/32 2024-02-23T00:19:11.163 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-23T00:19:11.296 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:16.2.14-550.geb66ed92.el8.x86_64 31/32 2024-02-23T00:19:11.313 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:16.2.14-550.geb66ed92.el8.x86_64 31/32 2024-02-23T00:19:11.445 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:16.2.14-550.geb66ed92.el8.x86_64 32/32 2024-02-23T00:19:11.445 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-23T00:19:11.634 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-23T00:19:11.842 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:16.2.14-550.geb66ed92.el8.x86_64 32/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:16.2.14-550.geb66ed92.el8.x86_64 1/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:16.2.14-550.geb66ed92.el8.x86_64 2/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 3/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:16.2.14-550.geb66ed92.el8.x86_64 4/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:16.2.14-550.geb66ed92.el8.x86_64 5/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-550.geb66ed92.el8. 6/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-23T00:19:11.843 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-23T00:19:11.844 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-23T00:19:11.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-23T00:19:11.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-23T00:19:11.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-23T00:19:11.845 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-23T00:19:11.869 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-23T00:19:12.111 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-23T00:19:12.421 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-23T00:19:12.665 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-23T00:19:12.791 INFO:teuthology.orchestra.run.smithi087.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-2.6-1.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-23T00:19:12.792 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:12.793 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:19:12.870 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-23T00:19:13.025 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2024-02-23T00:19:13.073 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-23T00:19:13.319 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-23T00:19:13.547 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-23T00:19:13.754 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-23T00:19:13.784 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:12 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:19:14.094 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-23T00:19:14.387 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-23T00:19:14.911 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-550.geb66ed92.el8. 27/32 2024-02-23T00:19:15.084 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 28/32 2024-02-23T00:19:15.312 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:19:15.359 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:19:15.360 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-23T00:19:15.360 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:19:15.478 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2024-02-23T00:19:16.226 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:15 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:19:16.598 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 28/32 2024-02-23T00:19:16.809 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-osd-2:16.2.14-550.geb66ed92.el8.x86_64 29/32 2024-02-23T00:19:17.252 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-osd-2:16.2.14-550.geb66ed92.el8.x86_64 29/32 2024-02-23T00:19:17.459 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mon-2:16.2.14-550.geb66ed92.el8.x86_64 30/32 2024-02-23T00:19:17.683 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mon-2:16.2.14-550.geb66ed92.el8.x86_64 30/32 2024-02-23T00:19:17.769 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 85 k 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-23T00:19:17.770 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:17.771 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 85 k 2024-02-23T00:19:17.771 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 352 k 2024-02-23T00:19:17.771 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:19:17.868 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mds-2:16.2.14-550.geb66ed92.el8.x86_64 31/32 2024-02-23T00:19:17.885 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mds-2:16.2.14-550.geb66ed92.el8.x86_64 31/32 2024-02-23T00:19:18.030 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-16.2.14-550.geb66ed92.el8.noarch.rpm 330 kB/s | 85 kB 00:00 2024-02-23T00:19:18.030 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:18.030 INFO:teuthology.orchestra.run.smithi087.stdout:Total 329 kB/s | 85 kB 00:00 2024-02-23T00:19:18.030 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:19:18.033 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-2:16.2.14-550.geb66ed92.el8.x86_64 32/32 2024-02-23T00:19:18.037 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:19:18.037 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:19:18.039 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:19:18.042 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:19:18.090 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:19:18.443 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-2:16.2.14-550.geb66ed92.el8.x86_64 32/32 2024-02-23T00:19:18.443 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-2:16.2.14-550.geb66ed92.el8.x86_64 1/32 2024-02-23T00:19:18.443 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mds-2:16.2.14-550.geb66ed92.el8.x86_64 2/32 2024-02-23T00:19:18.443 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 3/32 2024-02-23T00:19:18.443 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mon-2:16.2.14-550.geb66ed92.el8.x86_64 4/32 2024-02-23T00:19:18.443 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-osd-2:16.2.14-550.geb66ed92.el8.x86_64 5/32 2024-02-23T00:19:18.443 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-550.geb66ed92.el8. 6/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-23T00:19:18.444 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-23T00:19:18.445 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-23T00:19:18.445 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-23T00:19:18.445 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-23T00:19:18.445 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-23T00:19:18.445 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-23T00:19:18.446 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-23T00:19:18.446 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-23T00:19:18.446 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-23T00:19:18.446 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-23T00:19:18.446 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-23T00:19:18.446 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-23T00:19:18.631 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/1 2024-02-23T00:19:18.795 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/1 2024-02-23T00:19:19.025 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/1 2024-02-23T00:19:19.250 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-23T00:19:19.250 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:19.250 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mds-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-modules-core-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mon-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-osd-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-23T00:19:19.251 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-portend-2.6-1.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-23T00:19:19.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-23T00:19:19.253 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-23T00:19:19.253 INFO:teuthology.orchestra.run.smithi171.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-23T00:19:19.253 INFO:teuthology.orchestra.run.smithi171.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-23T00:19:19.253 INFO:teuthology.orchestra.run.smithi171.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-23T00:19:19.253 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:19.253 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:19:19.427 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/1 2024-02-23T00:19:19.427 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:19.428 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:19:19.428 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:19.428 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:19.428 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:19:19.462 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-base 2024-02-23T00:19:19.654 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2024-02-23T00:19:20.226 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:13 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:19:20.415 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:19 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:19:21.727 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-base-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:19:21.773 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:19:21.774 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-02-23T00:19:21.774 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:19:21.871 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install cephadm 2024-02-23T00:19:21.966 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:19:21.967 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:21.967 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-02-23T00:19:21.967 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-550.geb66ed92.el8 ceph 161 k 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 161 k 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 470 k 2024-02-23T00:19:21.968 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:19:22.167 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-16.2.14-550.geb66ed 811 kB/s | 161 kB 00:00 2024-02-23T00:19:22.167 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:22.167 INFO:teuthology.orchestra.run.smithi087.stdout:Total 806 kB/s | 161 kB 00:00 2024-02-23T00:19:22.168 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:19:22.178 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:19:22.178 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:19:22.225 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:19:22.228 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:19:22.344 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:19:22.544 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-550.geb66ed92. 1/1 2024-02-23T00:19:22.628 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:15 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:19:22.950 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-550.geb66ed92. 1/1 2024-02-23T00:19:23.369 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-550.geb66ed92. 1/1 2024-02-23T00:19:23.369 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:23.369 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:19:23.369 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:23.369 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:23.370 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:19:23.619 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2024-02-23T00:19:24.203 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:19:24.203 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:24.203 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:19:24.203 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:24.203 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:19:24.203 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 85 k 2024-02-23T00:19:24.203 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:24.203 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:19:24.204 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:24.204 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-02-23T00:19:24.204 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:24.204 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 85 k 2024-02-23T00:19:24.204 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 352 k 2024-02-23T00:19:24.204 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:19:24.374 INFO:teuthology.orchestra.run.smithi171.stdout:cephadm-16.2.14-550.geb66ed92.el8.noarch.rpm 503 kB/s | 85 kB 00:00 2024-02-23T00:19:24.375 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:24.375 INFO:teuthology.orchestra.run.smithi171.stdout:Total 499 kB/s | 85 kB 00:00 2024-02-23T00:19:24.375 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:19:24.376 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:23 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:19:24.382 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:19:24.382 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:19:24.384 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:19:24.387 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:19:24.433 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:19:24.904 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/1 2024-02-23T00:19:25.078 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/1 2024-02-23T00:19:25.312 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/1 2024-02-23T00:19:25.904 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:19:25.950 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:19:25.952 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-23T00:19:25.952 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:19:26.053 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2024-02-23T00:19:26.264 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/1 2024-02-23T00:19:26.265 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:26.265 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:19:26.265 INFO:teuthology.orchestra.run.smithi171.stdout: cephadm-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:26.265 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:26.265 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:19:26.498 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-immutable-object-cache 2024-02-23T00:19:26.801 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:25 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:19:27.267 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:20 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:19:28.374 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:19:28.375 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 3.1 M 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 24 k 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 15 k 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:19:28.376 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:28.377 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-02-23T00:19:28.377 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:28.377 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.4 M 2024-02-23T00:19:28.377 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 22 M 2024-02-23T00:19:28.377 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:19:28.736 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): ceph-grafana-dashboards-16.2.14-550.geb6 174 kB/s | 24 kB 00:00 2024-02-23T00:19:28.752 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): ceph-prometheus-alerts-16.2.14-550.geb66 95 kB/s | 15 kB 00:00 2024-02-23T00:19:28.841 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-550.geb66ed92.el8 ceph 161 k 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-02-23T00:19:28.842 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:28.843 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 161 k 2024-02-23T00:19:28.843 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 470 k 2024-02-23T00:19:28.843 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:19:28.919 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 202 kB/s | 34 kB 00:00 2024-02-23T00:19:28.947 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 202 kB/s | 43 kB 00:00 2024-02-23T00:19:29.151 INFO:teuthology.orchestra.run.smithi171.stdout:ceph-immutable-object-cache-16.2.14-550.geb66ed 522 kB/s | 161 kB 00:00 2024-02-23T00:19:29.152 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:29.152 INFO:teuthology.orchestra.run.smithi171.stdout:Total 520 kB/s | 161 kB 00:00 2024-02-23T00:19:29.152 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:19:29.163 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:19:29.164 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:19:29.205 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 688 kB/s | 196 kB 00:00 2024-02-23T00:19:29.210 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:19:29.213 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:19:29.288 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-mgr-dashboard-16.2.14-550.geb66ed92 4.5 MB/s | 3.1 MB 00:00 2024-02-23T00:19:29.292 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:29.292 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.7 MB/s | 3.4 MB 00:00 2024-02-23T00:19:29.302 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:19:29.312 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:19:29.312 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:19:29.348 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:19:29.448 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:19:29.451 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:19:29.524 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-550.geb66ed92. 1/1 2024-02-23T00:19:29.748 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:19:29.944 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-550.geb66ed92. 1/1 2024-02-23T00:19:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-23T00:19:30.210 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-23T00:19:30.310 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-550.geb66ed92. 1/1 2024-02-23T00:19:30.311 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:30.311 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:19:30.311 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-immutable-object-cache-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:19:30.311 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:30.311 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:19:30.437 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-23T00:19:30.543 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr 2024-02-23T00:19:30.557 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-550.geb66ed92.el8.n 4/6 2024-02-23T00:19:31.308 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:24 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:19:31.622 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-550.geb66ed92.el8. 5/6 2024-02-23T00:19:31.847 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-550.geb66ed92.el8.noarc 6/6 2024-02-23T00:19:32.171 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-550.geb66ed92.el8.noarc 6/6 2024-02-23T00:19:32.171 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-550.geb66ed92.el8. 1/6 2024-02-23T00:19:32.171 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-550.geb66ed92.el8.noarc 2/6 2024-02-23T00:19:32.171 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-550.geb66ed92.el8.n 3/6 2024-02-23T00:19:32.171 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-23T00:19:32.171 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-23T00:19:32.563 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-23T00:19:32.563 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:32.563 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:19:32.563 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:32.564 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:32.564 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:32.564 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-23T00:19:32.564 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-23T00:19:32.564 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-23T00:19:32.564 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:32.564 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:19:32.686 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-23T00:19:32.827 INFO:teuthology.orchestra.run.smithi171.stdout:Package ceph-mgr-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:19:32.876 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:19:32.877 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-02-23T00:19:32.877 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:19:32.981 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-dashboard 2024-02-23T00:19:33.432 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:32 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:19:33.733 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:26 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:19:34.996 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:19:34.997 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-02-23T00:19:34.998 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:19:34.998 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-02-23T00:19:34.998 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:19:34.998 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 8.6 M 2024-02-23T00:19:34.998 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-23T00:19:34.998 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-23T00:19:34.998 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout:============================================================================================ 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout:Install 8 Packages 2024-02-23T00:19:34.999 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:35.000 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 37 M 2024-02-23T00:19:35.000 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 210 M 2024-02-23T00:19:35.000 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:19:35.309 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:19:35.310 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 3.1 M 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 24 k 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 15 k 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-23T00:19:35.311 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:35.312 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:19:35.312 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:35.312 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2024-02-23T00:19:35.312 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:35.312 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 3.4 M 2024-02-23T00:19:35.313 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 22 M 2024-02-23T00:19:35.313 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:19:35.559 INFO:teuthology.orchestra.run.smithi087.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-02-23T00:19:35.667 INFO:teuthology.orchestra.run.smithi087.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2024-02-23T00:19:35.718 INFO:teuthology.orchestra.run.smithi087.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2024-02-23T00:19:35.766 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): ceph-grafana-dashboards-16.2.14-550.geb6 183 kB/s | 24 kB 00:00 2024-02-23T00:19:35.801 INFO:teuthology.orchestra.run.smithi087.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 15 MB/s | 3.7 MB 00:00 2024-02-23T00:19:35.899 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): ceph-prometheus-alerts-16.2.14-550.geb66 56 kB/s | 15 kB 00:00 2024-02-23T00:19:35.952 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 232 kB/s | 43 kB 00:00 2024-02-23T00:19:36.037 INFO:teuthology.orchestra.run.smithi087.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.7 MB/s | 645 kB 00:00 2024-02-23T00:19:36.092 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 175 kB/s | 34 kB 00:00 2024-02-23T00:19:36.129 INFO:teuthology.orchestra.run.smithi087.stdout:(6/8): ceph-mgr-diskprediction-local-16.2.14-55 9.5 MB/s | 8.6 MB 00:00 2024-02-23T00:19:36.226 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 725 kB/s | 196 kB 00:00 2024-02-23T00:19:36.270 INFO:teuthology.orchestra.run.smithi087.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 26 MB/s | 14 MB 00:00 2024-02-23T00:19:36.312 INFO:teuthology.orchestra.run.smithi087.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 626 kB/s | 172 kB 00:00 2024-02-23T00:19:36.313 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:36.313 INFO:teuthology.orchestra.run.smithi087.stdout:Total 28 MB/s | 37 MB 00:01 2024-02-23T00:19:36.542 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:19:36.561 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:19:36.561 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:19:36.843 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): ceph-mgr-dashboard-16.2.14-550.geb66ed92 2.6 MB/s | 3.1 MB 00:01 2024-02-23T00:19:36.846 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:36.846 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.2 MB/s | 3.4 MB 00:01 2024-02-23T00:19:36.857 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:19:36.868 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:19:36.868 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:19:36.887 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:19:36.890 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:19:37.033 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:19:37.036 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:19:37.326 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:19:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:19:37.536 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-23T00:19:37.567 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-23T00:19:37.737 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-23T00:19:37.798 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-23T00:19:37.900 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-23T00:19:38.008 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-23T00:19:38.137 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-550.geb66ed92.el8.n 4/6 2024-02-23T00:19:38.476 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-23T00:19:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:19:39.238 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-550.geb66ed92.el8. 5/6 2024-02-23T00:19:39.246 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:19:39.433 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-23T00:19:39.537 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-550.geb66ed92.el8.noarc 6/6 2024-02-23T00:19:39.818 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-550.geb66ed92.el8.noarc 6/6 2024-02-23T00:19:39.819 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-550.geb66ed92.el8. 1/6 2024-02-23T00:19:39.819 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-550.geb66ed92.el8.noarc 2/6 2024-02-23T00:19:39.819 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-550.geb66ed92.el8.n 3/6 2024-02-23T00:19:39.819 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-23T00:19:39.819 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-23T00:19:40.127 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-23T00:19:40.195 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-23T00:19:40.195 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:40.195 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:19:40.195 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-grafana-dashboards-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:40.195 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-dashboard-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:40.195 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-prometheus-alerts-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:40.195 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-23T00:19:40.196 INFO:teuthology.orchestra.run.smithi171.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-23T00:19:40.196 INFO:teuthology.orchestra.run.smithi171.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-23T00:19:40.196 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:40.196 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:19:40.330 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-23T00:19:40.520 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-23T00:19:40.742 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-23T00:19:41.087 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:34 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:19:42.505 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-23T00:19:42.645 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 8.6 M 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-23T00:19:42.646 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-23T00:19:42.647 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-23T00:19:42.647 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-23T00:19:42.647 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-23T00:19:42.647 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:42.647 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:19:42.647 INFO:teuthology.orchestra.run.smithi171.stdout:============================================================================================ 2024-02-23T00:19:42.647 INFO:teuthology.orchestra.run.smithi171.stdout:Install 8 Packages 2024-02-23T00:19:42.647 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:42.648 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 37 M 2024-02-23T00:19:42.648 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 210 M 2024-02-23T00:19:42.648 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:19:43.386 INFO:teuthology.orchestra.run.smithi171.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-02-23T00:19:43.553 INFO:teuthology.orchestra.run.smithi171.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 22 MB/s | 3.7 MB 00:00 2024-02-23T00:19:43.603 INFO:teuthology.orchestra.run.smithi171.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2024-02-23T00:19:43.696 INFO:teuthology.orchestra.run.smithi171.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.3 MB/s | 4.9 MB 00:00 2024-02-23T00:19:43.813 INFO:teuthology.orchestra.run.smithi171.stdout:(5/8): ceph-mgr-diskprediction-local-16.2.14-55 11 MB/s | 8.6 MB 00:00 2024-02-23T00:19:43.835 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-23T00:19:44.008 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-550.geb66ed9 8/8 2024-02-23T00:19:44.155 INFO:teuthology.orchestra.run.smithi171.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 26 MB/s | 14 MB 00:00 2024-02-23T00:19:44.180 INFO:teuthology.orchestra.run.smithi171.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 470 kB/s | 172 kB 00:00 2024-02-23T00:19:44.297 INFO:teuthology.orchestra.run.smithi171.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 645 kB 00:00 2024-02-23T00:19:44.298 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:44.298 INFO:teuthology.orchestra.run.smithi171.stdout:Total 22 MB/s | 37 MB 00:01 2024-02-23T00:19:44.323 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-550.geb66ed9 8/8 2024-02-23T00:19:44.323 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-550.geb66ed9 1/8 2024-02-23T00:19:44.323 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-23T00:19:44.323 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:19:44.323 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-23T00:19:44.323 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-23T00:19:44.324 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-23T00:19:44.324 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-23T00:19:44.530 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:19:44.548 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:19:44.549 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:19:44.769 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-23T00:19:44.769 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:44.769 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:44.770 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:19:44.898 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2024-02-23T00:19:44.901 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:19:44.904 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:19:45.466 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:19:45.621 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-23T00:19:45.644 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:44 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:19:45.759 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-23T00:19:45.904 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-23T00:19:46.486 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-23T00:19:46.646 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:19:47.226 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:19:47.228 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:47.228 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:19:47.228 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:47.228 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:19:47.228 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 27 k 2024-02-23T00:19:47.228 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-23T00:19:47.228 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:19:47.229 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:47.230 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2024-02-23T00:19:47.230 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:47.230 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.9 M 2024-02-23T00:19:47.230 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 24 M 2024-02-23T00:19:47.230 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:19:47.268 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:19:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-23T00:19:47.544 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 201 kB/s | 20 kB 00:00 2024-02-23T00:19:47.560 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 229 kB/s | 27 kB 00:00 2024-02-23T00:19:47.644 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-02-23T00:19:47.669 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.0 MB/s | 110 kB 00:00 2024-02-23T00:19:47.695 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): ceph-mgr-rook-16.2.14-550.geb66ed92.el8 109 kB/s | 27 kB 00:00 2024-02-23T00:19:47.837 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 255 kB/s | 43 kB 00:00 2024-02-23T00:19:47.854 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 740 kB/s | 155 kB 00:00 2024-02-23T00:19:47.879 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 179 kB/s | 33 kB 00:00 2024-02-23T00:19:48.054 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 75 kB/s | 16 kB 00:00 2024-02-23T00:19:48.096 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-23T00:19:48.096 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 335 kB/s | 81 kB 00:00 2024-02-23T00:19:48.172 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 816 kB/s | 61 kB 00:00 2024-02-23T00:19:48.197 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 452 kB/s | 64 kB 00:00 2024-02-23T00:19:48.230 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.3 MB/s | 1.2 MB 00:00 2024-02-23T00:19:48.231 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:48.231 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.9 MB/s | 1.9 MB 00:01 2024-02-23T00:19:48.281 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:19:48.296 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:19:48.296 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:19:48.382 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:19:48.385 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:19:49.023 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:19:49.055 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-23T00:19:49.263 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-23T00:19:49.287 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-23T00:19:49.460 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-23T00:19:49.695 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-23T00:19:49.931 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-23T00:19:50.189 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-23T00:19:50.424 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-23T00:19:50.710 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-23T00:19:50.956 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-23T00:19:51.060 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-23T00:19:52.080 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-23T00:19:52.330 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-23T00:19:52.354 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-23T00:19:52.545 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-550.geb66ed9 8/8 2024-02-23T00:19:52.556 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-23T00:19:52.714 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-23T00:19:52.865 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-550.geb66ed9 8/8 2024-02-23T00:19:52.865 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-550.geb66ed9 1/8 2024-02-23T00:19:52.866 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-23T00:19:52.866 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-23T00:19:52.866 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-23T00:19:52.866 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-23T00:19:52.866 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-23T00:19:52.866 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-23T00:19:52.911 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:16.2.14-550.geb66ed92.el8.noarch 13/13 2024-02-23T00:19:53.180 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-550.geb66ed92.el8.noarch 13/13 2024-02-23T00:19:53.180 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:16.2.14-550.geb66ed92.el8.noarch 1/13 2024-02-23T00:19:53.180 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-23T00:19:53.180 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-23T00:19:53.180 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-23T00:19:53.180 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-23T00:19:53.180 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-23T00:19:53.180 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-23T00:19:53.180 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-23T00:19:53.181 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-23T00:19:53.181 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-23T00:19:53.181 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-23T00:19:53.181 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-23T00:19:53.251 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-23T00:19:53.251 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:53.251 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:19:53.251 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-diskprediction-local-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:53.251 INFO:teuthology.orchestra.run.smithi171.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-23T00:19:53.251 INFO:teuthology.orchestra.run.smithi171.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-23T00:19:53.252 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-23T00:19:53.252 INFO:teuthology.orchestra.run.smithi171.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-23T00:19:53.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-23T00:19:53.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-23T00:19:53.252 INFO:teuthology.orchestra.run.smithi171.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-23T00:19:53.252 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:53.252 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:19:53.398 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-rook 2024-02-23T00:19:53.675 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-23T00:19:53.675 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:53.675 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:19:53.675 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:19:53.675 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-23T00:19:53.675 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-23T00:19:53.675 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-23T00:19:53.675 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:53.676 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:19:53.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2024-02-23T00:19:54.164 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:47 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:19:54.547 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:53 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:19:55.717 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:19:55.718 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:55.718 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:19:55.718 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 27 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-23T00:19:55.719 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-23T00:19:55.720 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:55.720 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:19:55.720 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:19:55.720 INFO:teuthology.orchestra.run.smithi171.stdout:Install 13 Packages 2024-02-23T00:19:55.720 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:19:55.720 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 1.9 M 2024-02-23T00:19:55.721 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 24 M 2024-02-23T00:19:55.721 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:19:56.077 INFO:teuthology.orchestra.run.smithi171.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 198 kB/s | 20 kB 00:00 2024-02-23T00:19:56.098 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:19:56.099 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 116 k 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:19:56.100 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:19:56.101 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2024-02-23T00:19:56.101 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:19:56.101 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 5.9 M 2024-02-23T00:19:56.101 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 24 M 2024-02-23T00:19:56.101 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:19:56.103 INFO:teuthology.orchestra.run.smithi171.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 211 kB/s | 27 kB 00:00 2024-02-23T00:19:56.170 INFO:teuthology.orchestra.run.smithi171.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.3 MB/s | 126 kB 00:00 2024-02-23T00:19:56.195 INFO:teuthology.orchestra.run.smithi171.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2024-02-23T00:19:56.221 INFO:teuthology.orchestra.run.smithi171.stdout:(5/13): ceph-mgr-rook-16.2.14-550.geb66ed92.el8 112 kB/s | 27 kB 00:00 2024-02-23T00:19:56.321 INFO:teuthology.orchestra.run.smithi171.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 340 kB/s | 43 kB 00:00 2024-02-23T00:19:56.438 INFO:teuthology.orchestra.run.smithi171.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 578 kB/s | 155 kB 00:00 2024-02-23T00:19:56.472 INFO:teuthology.orchestra.run.smithi171.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 131 kB/s | 33 kB 00:00 2024-02-23T00:19:56.488 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 3.2 MB/s | 578 kB 00:00 2024-02-23T00:19:56.556 INFO:teuthology.orchestra.run.smithi171.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 70 kB/s | 16 kB 00:00 2024-02-23T00:19:56.596 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): ceph-mgr-cephadm-16.2.14-550.geb66ed92.e 406 kB/s | 116 kB 00:00 2024-02-23T00:19:56.621 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 159 kB/s | 21 kB 00:00 2024-02-23T00:19:56.706 INFO:teuthology.orchestra.run.smithi171.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 430 kB/s | 64 kB 00:00 2024-02-23T00:19:56.747 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.3 MB/s | 355 kB 00:00 2024-02-23T00:19:56.748 INFO:teuthology.orchestra.run.smithi171.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 263 kB/s | 81 kB 00:00 2024-02-23T00:19:56.763 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 344 kB/s | 49 kB 00:00 2024-02-23T00:19:56.798 INFO:teuthology.orchestra.run.smithi171.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 3.6 MB/s | 1.2 MB 00:00 2024-02-23T00:19:56.824 INFO:teuthology.orchestra.run.smithi171.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 521 kB/s | 61 kB 00:00 2024-02-23T00:19:56.827 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:56.827 INFO:teuthology.orchestra.run.smithi171.stdout:Total 1.7 MB/s | 1.9 MB 00:01 2024-02-23T00:19:56.880 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:19:56.896 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:19:56.896 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:19:56.982 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:19:56.985 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:19:57.014 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.8 MB/s | 4.8 MB 00:00 2024-02-23T00:19:57.014 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:19:57.014 INFO:teuthology.orchestra.run.smithi087.stdout:Total 6.4 MB/s | 5.9 MB 00:00 2024-02-23T00:19:57.065 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:19:57.074 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:19:57.074 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:19:57.182 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:19:57.185 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:19:57.472 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:19:57.555 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:19:57.731 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-23T00:19:57.780 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-23T00:19:57.977 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-23T00:19:58.046 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-23T00:19:58.204 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-23T00:19:58.417 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-23T00:19:58.641 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-23T00:19:58.848 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-23T00:19:59.062 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-23T00:19:59.105 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-23T00:19:59.348 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-23T00:19:59.404 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-23T00:19:59.683 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-23T00:19:59.932 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-550.geb66ed92.el8.noarch 6/6 2024-02-23T00:20:00.218 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-550.geb66ed92.el8.noarch 6/6 2024-02-23T00:20:00.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/6 2024-02-23T00:20:00.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-23T00:20:00.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-23T00:20:00.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-23T00:20:00.218 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-23T00:20:00.397 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-23T00:20:00.635 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-23T00:20:00.635 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:00.635 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:20:00.635 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:20:00.635 INFO:teuthology.orchestra.run.smithi087.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-23T00:20:00.635 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-23T00:20:00.635 INFO:teuthology.orchestra.run.smithi087.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-23T00:20:00.635 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-02-23T00:20:00.635 INFO:teuthology.orchestra.run.smithi087.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-23T00:20:00.636 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:00.636 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:00.642 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-23T00:20:00.813 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2024-02-23T00:20:00.859 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-23T00:20:01.051 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-23T00:20:01.242 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-rook-2:16.2.14-550.geb66ed92.el8.noarch 13/13 2024-02-23T00:20:01.529 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-550.geb66ed92.el8.noarch 13/13 2024-02-23T00:20:01.529 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-rook-2:16.2.14-550.geb66ed92.el8.noarch 1/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-23T00:20:01.530 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-23T00:20:01.568 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:00 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:01.969 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-rook-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-23T00:20:01.970 INFO:teuthology.orchestra.run.smithi171.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-23T00:20:01.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-23T00:20:01.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-23T00:20:01.971 INFO:teuthology.orchestra.run.smithi171.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-23T00:20:01.971 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:01.971 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:02.087 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-mgr-cephadm 2024-02-23T00:20:02.840 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:01:55 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:03.109 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:03.109 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:16.2.14-550.geb66ed92.el8 ceph 911 k 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2024-02-23T00:20:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:03.111 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 994 k 2024-02-23T00:20:03.111 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.8 M 2024-02-23T00:20:03.111 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:20:04.421 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:04.422 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:04.422 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:20:04.422 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:04.422 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm noarch 2:16.2.14-550.geb66ed92.el8 ceph-noarch 116 k 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout:Install 6 Packages 2024-02-23T00:20:04.423 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:04.424 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 5.9 M 2024-02-23T00:20:04.424 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 24 M 2024-02-23T00:20:04.424 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:20:04.533 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 377 kB/s | 83 kB 00:00 2024-02-23T00:20:04.566 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-16.2.14-550.geb66ed92.el8.x86_ 3.5 MB/s | 911 kB 00:00 2024-02-23T00:20:04.567 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:04.567 INFO:teuthology.orchestra.run.smithi087.stdout:Total 682 kB/s | 994 kB 00:01 2024-02-23T00:20:04.579 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:20:04.601 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:20:04.601 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:20:04.693 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:20:04.697 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:20:04.869 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:20:05.145 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-23T00:20:05.380 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 2/2 2024-02-23T00:20:05.436 INFO:teuthology.orchestra.run.smithi171.stdout:(1/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 2.9 MB/s | 578 kB 00:00 2024-02-23T00:20:05.478 INFO:teuthology.orchestra.run.smithi171.stdout:(2/6): ceph-mgr-cephadm-16.2.14-550.geb66ed92.e 494 kB/s | 116 kB 00:00 2024-02-23T00:20:05.620 INFO:teuthology.orchestra.run.smithi171.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2024-02-23T00:20:05.696 INFO:teuthology.orchestra.run.smithi171.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 82 kB/s | 21 kB 00:00 2024-02-23T00:20:05.772 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 2/2 2024-02-23T00:20:05.772 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 1/2 2024-02-23T00:20:05.787 INFO:teuthology.orchestra.run.smithi171.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.1 MB/s | 355 kB 00:00 2024-02-23T00:20:05.829 INFO:teuthology.orchestra.run.smithi171.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 234 kB/s | 49 kB 00:00 2024-02-23T00:20:05.830 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:05.831 INFO:teuthology.orchestra.run.smithi171.stdout:Total 4.2 MB/s | 5.9 MB 00:01 2024-02-23T00:20:05.884 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:20:05.895 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:20:05.895 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:20:05.996 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:20:05.999 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:20:06.164 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-23T00:20:06.165 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:06.165 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:20:06.165 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-23T00:20:06.165 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:06.165 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:06.276 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:20:06.380 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2024-02-23T00:20:06.533 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-23T00:20:06.853 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-23T00:20:07.145 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:06 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:07.808 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-23T00:20:08.146 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-23T00:20:08.392 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-23T00:20:08.567 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-550.geb66ed92.el8.noarch 6/6 2024-02-23T00:20:08.726 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:08.726 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:16.2.14-550.geb66ed92.el8 ceph 131 k 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 131 k 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 432 k 2024-02-23T00:20:08.727 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:20:08.851 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-550.geb66ed92.el8.noarch 6/6 2024-02-23T00:20:08.851 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-550.geb66ed92.el8.noarch 1/6 2024-02-23T00:20:08.851 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-23T00:20:08.851 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-23T00:20:08.851 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-23T00:20:08.852 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-23T00:20:09.144 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-16.2.14-550.geb66ed92.el8.x86_64 314 kB/s | 131 kB 00:00 2024-02-23T00:20:09.145 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:09.145 INFO:teuthology.orchestra.run.smithi087.stdout:Total 313 kB/s | 131 kB 00:00 2024-02-23T00:20:09.145 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:20:09.151 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:20:09.151 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:20:09.202 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:20:09.205 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:20:09.232 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-mgr-cephadm-2:16.2.14-550.geb66ed92.el8.noarch 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:09.233 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:09.319 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:20:09.369 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install ceph-fuse 2024-02-23T00:20:09.545 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:09.829 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:10.136 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:03 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:10.231 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:10.232 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:20:10.232 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:10.232 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:10.232 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:10.451 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2024-02-23T00:20:11.212 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:10 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:11.702 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repository Size 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse x86_64 2:16.2.14-550.geb66ed92.el8 ceph 911 k 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout:Installing dependencies: 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:20:11.703 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:11.704 INFO:teuthology.orchestra.run.smithi171.stdout:Install 2 Packages 2024-02-23T00:20:11.704 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:11.704 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 994 k 2024-02-23T00:20:11.704 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 2.8 M 2024-02-23T00:20:11.704 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:20:12.128 INFO:teuthology.orchestra.run.smithi171.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 372 kB/s | 83 kB 00:00 2024-02-23T00:20:12.161 INFO:teuthology.orchestra.run.smithi171.stdout:(2/2): ceph-fuse-16.2.14-550.geb66ed92.el8.x86_ 3.5 MB/s | 911 kB 00:00 2024-02-23T00:20:12.161 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:12.162 INFO:teuthology.orchestra.run.smithi171.stdout:Total 2.1 MB/s | 994 kB 00:00 2024-02-23T00:20:12.172 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:20:12.192 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:20:12.193 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:20:12.281 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:20:12.284 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:20:12.446 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:20:12.718 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-23T00:20:12.724 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:12.772 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:12.774 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-23T00:20:12.774 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:12.941 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : ceph-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 2/2 2024-02-23T00:20:13.073 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2024-02-23T00:20:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: ceph-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 2/2 2024-02-23T00:20:13.329 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : ceph-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 1/2 2024-02-23T00:20:13.731 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-23T00:20:13.731 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:13.732 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:20:13.732 INFO:teuthology.orchestra.run.smithi171.stdout: ceph-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-23T00:20:13.732 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:13.732 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:13.833 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:12 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:13.950 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados-devel 2024-02-23T00:20:14.725 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:07 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:15.407 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:15.408 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:15.408 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-02-23T00:20:15.408 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:15.408 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:20:15.408 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:16.2.14-550.geb66ed92.el8 ceph 25 k 2024-02-23T00:20:15.408 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:15.409 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:20:15.409 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:15.409 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-23T00:20:15.409 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:15.409 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 25 k 2024-02-23T00:20:15.409 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 107 k 2024-02-23T00:20:15.409 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:20:15.559 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-16.2.14-550.geb66ed92.el8.x86_6 170 kB/s | 25 kB 00:00 2024-02-23T00:20:15.559 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:15.560 INFO:teuthology.orchestra.run.smithi087.stdout:Total 169 kB/s | 25 kB 00:00 2024-02-23T00:20:15.560 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:20:15.563 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:20:15.563 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:20:15.565 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:20:15.569 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:20:15.644 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:20:15.842 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:16.111 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:16.276 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:16.277 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:16.277 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-02-23T00:20:16.277 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:16.277 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:20:16.277 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel x86_64 2:16.2.14-550.geb66ed92.el8 ceph 131 k 2024-02-23T00:20:16.277 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:16.277 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:20:16.277 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:16.278 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-02-23T00:20:16.278 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:16.278 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 131 k 2024-02-23T00:20:16.278 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 432 k 2024-02-23T00:20:16.278 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:20:16.468 INFO:teuthology.orchestra.run.smithi171.stdout:librados-devel-16.2.14-550.geb66ed92.el8.x86_64 692 kB/s | 131 kB 00:00 2024-02-23T00:20:16.468 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:16.469 INFO:teuthology.orchestra.run.smithi171.stdout:Total 687 kB/s | 131 kB 00:00 2024-02-23T00:20:16.469 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:20:16.474 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:20:16.474 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:20:16.494 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:16.494 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:16.494 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:20:16.494 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:16.494 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:16.494 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:16.523 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:20:16.526 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:20:16.605 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2024-02-23T00:20:16.647 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:20:16.864 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : librados-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:17.156 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: librados-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:17.353 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:16 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : librados-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:17.541 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:20:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: librados-devel-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:17.541 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:17.541 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:17.753 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs2 2024-02-23T00:20:18.521 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:11 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:18.859 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:18.905 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:18.907 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-23T00:20:18.907 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:19.005 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2024-02-23T00:20:19.757 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:18 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:20.046 INFO:teuthology.orchestra.run.smithi171.stdout:Package libcephfs2-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:20.091 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:20.093 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-02-23T00:20:20.093 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:20.359 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install libcephfs-devel 2024-02-23T00:20:21.118 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:14 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:21.271 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:21.317 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:21.319 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-23T00:20:21.319 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:21.422 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2024-02-23T00:20:22.175 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:21 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:22.678 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:22.679 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel x86_64 2:16.2.14-550.geb66ed92.el8 ceph 25 k 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 25 k 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 107 k 2024-02-23T00:20:22.680 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:20:22.829 INFO:teuthology.orchestra.run.smithi171.stdout:libcephfs-devel-16.2.14-550.geb66ed92.el8.x86_6 172 kB/s | 25 kB 00:00 2024-02-23T00:20:22.829 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:22.830 INFO:teuthology.orchestra.run.smithi171.stdout:Total 170 kB/s | 25 kB 00:00 2024-02-23T00:20:22.830 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:20:22.833 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:20:22.833 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:20:22.835 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:20:22.838 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:20:22.889 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:20:23.121 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : libcephfs-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:23.684 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:23.720 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : libcephfs-devel-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:23.721 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:23.721 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:20:23.721 INFO:teuthology.orchestra.run.smithi171.stdout: libcephfs-devel-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:23.721 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:23.721 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:23.729 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:23.730 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-23T00:20:23.731 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:23.842 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librados2 2024-02-23T00:20:23.895 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2024-02-23T00:20:24.591 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:17 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:24.650 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:23 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:26.115 INFO:teuthology.orchestra.run.smithi171.stdout:Package librados2-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:26.144 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:26.161 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:26.163 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-02-23T00:20:26.163 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:26.191 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:26.192 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-23T00:20:26.192 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:26.278 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install librbd1 2024-02-23T00:20:26.290 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2024-02-23T00:20:27.044 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:26 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:27.045 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:20 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:28.556 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:28.564 INFO:teuthology.orchestra.run.smithi171.stdout:Package librbd1-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:28.602 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:28.603 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-23T00:20:28.603 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:28.610 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:28.611 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-02-23T00:20:28.611 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:28.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2024-02-23T00:20:28.712 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rados 2024-02-23T00:20:29.449 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:28 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:29.465 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:22 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:30.961 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:30.968 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rados-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:31.008 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:31.009 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2024-02-23T00:20:31.009 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:31.014 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:31.016 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-02-23T00:20:31.016 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:31.110 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2024-02-23T00:20:31.118 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rgw 2024-02-23T00:20:31.857 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:30 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:31.874 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:24 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:33.397 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rgw-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:33.406 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:16.2.14-550.geb66ed92.el8 ceph 88 k 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 88 k 2024-02-23T00:20:33.408 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 240 k 2024-02-23T00:20:33.409 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:20:33.446 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:33.447 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-02-23T00:20:33.447 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:33.574 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-cephfs 2024-02-23T00:20:33.674 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-16.2.14-550.geb66ed92.el8.x86_64.rpm 334 kB/s | 88 kB 00:00 2024-02-23T00:20:33.674 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:33.674 INFO:teuthology.orchestra.run.smithi087.stdout:Total 332 kB/s | 88 kB 00:00 2024-02-23T00:20:33.675 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:20:33.684 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:20:33.684 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:20:33.733 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:20:33.736 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:20:33.846 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:20:34.058 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:34.331 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:27 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:34.352 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:34.736 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:34.737 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:34.737 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:20:34.737 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:34.737 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:34.737 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:34.979 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2024-02-23T00:20:35.739 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:34 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:35.839 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-cephfs-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:35.885 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:35.886 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-02-23T00:20:35.886 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:36.003 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install python3-rbd 2024-02-23T00:20:36.761 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:29 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:37.311 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:16.2.14-550.geb66ed92.el8 ceph 3.7 M 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-23T00:20:37.312 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:37.313 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.7 M 2024-02-23T00:20:37.313 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 15 M 2024-02-23T00:20:37.313 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:20:37.859 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-16.2.14-550.geb66ed92.el8.x86_64.rpm 6.7 MB/s | 3.7 MB 00:00 2024-02-23T00:20:37.860 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:37.860 INFO:teuthology.orchestra.run.smithi087.stdout:Total 6.7 MB/s | 3.7 MB 00:00 2024-02-23T00:20:37.861 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:20:37.871 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:20:37.872 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:20:37.934 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:20:37.937 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:20:38.286 INFO:teuthology.orchestra.run.smithi171.stdout:Package python3-rbd-2:16.2.14-550.geb66ed92.el8.x86_64 is already installed. 2024-02-23T00:20:38.333 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:38.334 INFO:teuthology.orchestra.run.smithi171.stdout:Nothing to do. 2024-02-23T00:20:38.334 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:38.374 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:20:38.440 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-fuse 2024-02-23T00:20:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:39.009 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:39.199 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:32 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:39.380 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:39.380 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:39.380 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:20:39.380 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:39.381 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:39.381 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:39.631 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2024-02-23T00:20:40.392 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:39 ago on Fri 23 Feb 2024 12:18:01 AM UTC. 2024-02-23T00:20:40.753 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse x86_64 2:16.2.14-550.geb66ed92.el8 ceph 88 k 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:40.754 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 88 k 2024-02-23T00:20:40.755 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 240 k 2024-02-23T00:20:40.755 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:20:41.013 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-fuse-16.2.14-550.geb66ed92.el8.x86_64.rpm 343 kB/s | 88 kB 00:00 2024-02-23T00:20:41.013 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:41.013 INFO:teuthology.orchestra.run.smithi171.stdout:Total 341 kB/s | 88 kB 00:00 2024-02-23T00:20:41.014 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:20:41.023 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:20:41.023 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:20:41.074 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:20:41.077 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:20:41.191 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:20:41.386 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:41.671 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:41.952 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:16.2.14-550.geb66ed92.el8 ceph 175 k 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:41.953 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 175 k 2024-02-23T00:20:41.954 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 523 k 2024-02-23T00:20:41.954 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2024-02-23T00:20:42.038 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:42.038 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:42.038 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:20:42.038 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-fuse-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:42.038 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:42.038 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:42.254 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-16.2.14-550.geb66ed92.el8.x86_64.rpm 584 kB/s | 175 kB 00:00 2024-02-23T00:20:42.254 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:42.254 INFO:teuthology.orchestra.run.smithi087.stdout:Total 581 kB/s | 175 kB 00:00 2024-02-23T00:20:42.255 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2024-02-23T00:20:42.265 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2024-02-23T00:20:42.265 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2024-02-23T00:20:42.271 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-mirror 2024-02-23T00:20:42.316 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2024-02-23T00:20:42.319 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2024-02-23T00:20:42.436 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2024-02-23T00:20:42.629 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:43.037 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:36 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:43.326 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:43.326 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:43.326 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2024-02-23T00:20:43.326 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:43.326 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:20:43.326 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2024-02-23T00:20:43.623 DEBUG:teuthology.parallel:result is None 2024-02-23T00:20:44.604 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:44.605 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:44.605 INFO:teuthology.orchestra.run.smithi171.stdout: Package Arch Version Repo Size 2024-02-23T00:20:44.605 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:44.605 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:20:44.605 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror x86_64 2:16.2.14-550.geb66ed92.el8 ceph 3.7 M 2024-02-23T00:20:44.605 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:44.606 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:20:44.606 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:44.606 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-02-23T00:20:44.606 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:44.606 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 3.7 M 2024-02-23T00:20:44.606 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 15 M 2024-02-23T00:20:44.606 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:20:45.151 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-mirror-16.2.14-550.geb66ed92.el8.x86_64.rpm 6.7 MB/s | 3.7 MB 00:00 2024-02-23T00:20:45.152 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:45.152 INFO:teuthology.orchestra.run.smithi171.stdout:Total 6.7 MB/s | 3.7 MB 00:00 2024-02-23T00:20:45.153 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:20:45.164 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:20:45.164 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:20:45.227 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:20:45.230 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:20:45.669 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:20:45.870 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-mirror-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:46.286 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-mirror-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:46.661 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-mirror-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:46.661 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:46.661 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:20:46.661 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-mirror-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:46.661 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:46.661 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:46.860 DEBUG:teuthology.orchestra.run.smithi171:> sudo yum -y install rbd-nbd 2024-02-23T00:20:47.624 INFO:teuthology.orchestra.run.smithi171.stdout:Last metadata expiration check: 0:02:40 ago on Fri 23 Feb 2024 12:18:07 AM UTC. 2024-02-23T00:20:49.197 INFO:teuthology.orchestra.run.smithi171.stdout:Dependencies resolved. 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout: Package Architecture Version Repository Size 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout:Installing: 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd x86_64 2:16.2.14-550.geb66ed92.el8 ceph 175 k 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction Summary 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout:================================================================================ 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout:Install 1 Package 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout:Total download size: 175 k 2024-02-23T00:20:49.198 INFO:teuthology.orchestra.run.smithi171.stdout:Installed size: 523 k 2024-02-23T00:20:49.199 INFO:teuthology.orchestra.run.smithi171.stdout:Downloading Packages: 2024-02-23T00:20:49.391 INFO:teuthology.orchestra.run.smithi171.stdout:rbd-nbd-16.2.14-550.geb66ed92.el8.x86_64.rpm 911 kB/s | 175 kB 00:00 2024-02-23T00:20:49.391 INFO:teuthology.orchestra.run.smithi171.stdout:-------------------------------------------------------------------------------- 2024-02-23T00:20:49.392 INFO:teuthology.orchestra.run.smithi171.stdout:Total 906 kB/s | 175 kB 00:00 2024-02-23T00:20:49.392 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction check 2024-02-23T00:20:49.402 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction check succeeded. 2024-02-23T00:20:49.402 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction test 2024-02-23T00:20:49.452 INFO:teuthology.orchestra.run.smithi171.stdout:Transaction test succeeded. 2024-02-23T00:20:49.455 INFO:teuthology.orchestra.run.smithi171.stdout:Running transaction 2024-02-23T00:20:49.564 INFO:teuthology.orchestra.run.smithi171.stdout: Preparing : 1/1 2024-02-23T00:20:49.808 INFO:teuthology.orchestra.run.smithi171.stdout: Installing : rbd-nbd-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:50.100 INFO:teuthology.orchestra.run.smithi171.stdout: Running scriptlet: rbd-nbd-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:50.474 INFO:teuthology.orchestra.run.smithi171.stdout: Verifying : rbd-nbd-2:16.2.14-550.geb66ed92.el8.x86_64 1/1 2024-02-23T00:20:50.475 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:50.475 INFO:teuthology.orchestra.run.smithi171.stdout:Installed: 2024-02-23T00:20:50.475 INFO:teuthology.orchestra.run.smithi171.stdout: rbd-nbd-2:16.2.14-550.geb66ed92.el8.x86_64 2024-02-23T00:20:50.475 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:20:50.475 INFO:teuthology.orchestra.run.smithi171.stdout:Complete! 2024-02-23T00:20:50.682 DEBUG:teuthology.parallel:result is None 2024-02-23T00:20:50.683 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:20:50.833 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-23T00:20:50.860 INFO:teuthology.orchestra.run.smithi087.stdout:16.2.14-550.geb66ed92.el8 2024-02-23T00:20:50.861 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-550.geb66ed92.el8 2024-02-23T00:20:50.861 INFO:teuthology.task.install:The correct ceph version 16.2.14-550.geb66ed92 is installed. 2024-02-23T00:20:50.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:20:51.096 DEBUG:teuthology.orchestra.run.smithi171:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-23T00:20:51.118 INFO:teuthology.orchestra.run.smithi171.stdout:16.2.14-550.geb66ed92.el8 2024-02-23T00:20:51.118 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-550.geb66ed92.el8 2024-02-23T00:20:51.118 INFO:teuthology.task.install:The correct ceph version 16.2.14-550.geb66ed92 is installed. 2024-02-23T00:20:51.120 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-23T00:20:51.121 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:20:51.121 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-23T00:20:51.154 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:20:51.154 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-23T00:20:51.185 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-23T00:20:51.185 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:20:51.185 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2024-02-23T00:20:51.225 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-23T00:20:51.297 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:20:51.298 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/daemon-helper 2024-02-23T00:20:51.324 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-23T00:20:51.392 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-23T00:20:51.392 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:20:51.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-23T00:20:51.421 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-23T00:20:51.490 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:20:51.490 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-23T00:20:51.517 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-23T00:20:51.583 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-23T00:20:51.584 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:20:51.584 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2024-02-23T00:20:51.612 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-23T00:20:51.678 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:20:51.679 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/usr/bin/stdin-killer 2024-02-23T00:20:51.705 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-23T00:20:51.773 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-23T00:20:51.854 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', 'flag\\(s\\) set', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(PG_', '\\(OSD_', '\\(POOL_', '\\(CEPHADM_STRAY_DAEMON\\)', 'PG_', 'degraded', 'backfill', 'mons down', 'OSD_', 'is down', 'acting', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59', 'cephadm_mode': 'root'} 2024-02-23T00:20:51.854 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 2024-02-23T00:20:51.854 INFO:tasks.cephadm:Cluster fsid is 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:20:51.854 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-23T00:20:51.855 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.87', 'mon.c': '[v2:172.21.15.87:3301,v1:172.21.15.87:6790]', 'mon.b': '172.21.15.171'} 2024-02-23T00:20:51.855 INFO:tasks.cephadm:First mon is mon.a on smithi087 2024-02-23T00:20:51.855 INFO:tasks.cephadm:First mgr is y 2024-02-23T00:20:51.855 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-23T00:20:51.855 DEBUG:teuthology.orchestra.run.smithi087:> sudo hostname $(hostname -s) 2024-02-23T00:20:51.883 DEBUG:teuthology.orchestra.run.smithi171:> sudo hostname $(hostname -s) 2024-02-23T00:20:51.910 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref eb66ed921e744301e6863be6353618d63967ea59)... 2024-02-23T00:20:51.911 DEBUG:teuthology.orchestra.run.smithi087:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/eb66ed921e744301e6863be6353618d63967ea59/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-23T00:20:52.178 INFO:teuthology.orchestra.run.smithi087.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 23 00:20 /home/ubuntu/cephtest/cephadm 2024-02-23T00:20:52.179 DEBUG:teuthology.orchestra.run.smithi171:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/eb66ed921e744301e6863be6353618d63967ea59/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-23T00:20:52.356 INFO:teuthology.orchestra.run.smithi171.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 23 00:20 /home/ubuntu/cephtest/cephadm 2024-02-23T00:20:52.356 DEBUG:teuthology.orchestra.run.smithi087:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-23T00:20:52.375 DEBUG:teuthology.orchestra.run.smithi171:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-23T00:20:52.431 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph 2024-02-23T00:20:52.458 DEBUG:teuthology.orchestra.run.smithi171:> sudo mkdir -p /etc/ceph 2024-02-23T00:20:52.486 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /etc/ceph 2024-02-23T00:20:52.525 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 777 /etc/ceph 2024-02-23T00:20:52.552 INFO:tasks.cephadm:Writing seed config... 2024-02-23T00:20:52.553 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-23T00:20:52.554 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-23T00:20:52.555 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-23T00:20:52.556 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:20:52.556 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-23T00:20:52.583 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 66db207c-d1e1-11ee-95c0-87774f69a715 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-23T00:20:52.584 DEBUG:teuthology.orchestra.run.smithi087:mon.a> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.a.service 2024-02-23T00:20:52.626 DEBUG:teuthology.orchestra.run.smithi087:mgr.y> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mgr.y.service 2024-02-23T00:20:52.652 INFO:journalctl@ceph.mon.a.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:20:52.653 INFO:tasks.cephadm:Bootstrapping... 2024-02-23T00:20:52.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 -v bootstrap --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.87 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:20:52.678 INFO:journalctl@ceph.mgr.y.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:20:52.934 INFO:teuthology.orchestra.run.smithi087.stderr:-------------------------------------------------------------------------------- 2024-02-23T00:20:52.982 INFO:teuthology.orchestra.run.smithi087.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59', '-v', 'bootstrap', '--fsid', '66db207c-d1e1-11ee-95c0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.87', '--skip-admin-label'] 2024-02-23T00:20:53.937 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: 4.9.3-dev 2024-02-23T00:20:53.947 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying podman|docker is present... 2024-02-23T00:20:53.970 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: 4.9.3-dev 2024-02-23T00:20:53.983 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying lvm2 is present... 2024-02-23T00:20:53.983 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying time synchronization is in place... 2024-02-23T00:20:53.990 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-23T00:20:53.997 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-23T00:20:54.003 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: enabled 2024-02-23T00:20:54.009 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: active 2024-02-23T00:20:54.010 INFO:teuthology.orchestra.run.smithi087.stderr:Unit chronyd.service is enabled and running 2024-02-23T00:20:54.010 INFO:teuthology.orchestra.run.smithi087.stderr:Repeating the final host check... 2024-02-23T00:20:54.030 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: 4.9.3-dev 2024-02-23T00:20:54.043 INFO:teuthology.orchestra.run.smithi087.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-23T00:20:54.043 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl is present 2024-02-23T00:20:54.044 INFO:teuthology.orchestra.run.smithi087.stderr:lvcreate is present 2024-02-23T00:20:54.049 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-23T00:20:54.054 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-23T00:20:54.060 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: enabled 2024-02-23T00:20:54.066 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: active 2024-02-23T00:20:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:Unit chronyd.service is enabled and running 2024-02-23T00:20:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:Host looks OK 2024-02-23T00:20:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:Cluster fsid: 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:20:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:Acquiring lock 140296366991848 on /run/cephadm/66db207c-d1e1-11ee-95c0-87774f69a715.lock 2024-02-23T00:20:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:Lock 140296366991848 acquired on /run/cephadm/66db207c-d1e1-11ee-95c0-87774f69a715.lock 2024-02-23T00:20:54.068 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying IP 172.21.15.87 port 3300 ... 2024-02-23T00:20:54.068 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying IP 172.21.15.87 port 6789 ... 2024-02-23T00:20:54.068 INFO:teuthology.orchestra.run.smithi087.stderr:Base mon IP(s) is [172.21.15.87:3300, 172.21.15.87:6789], mon addrv is [v2:172.21.15.87:3300,v1:172.21.15.87:6789] 2024-02-23T00:20:54.071 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.87 metric 100 2024-02-23T00:20:54.071 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.87 metric 100 2024-02-23T00:20:54.074 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-23T00:20:54.074 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-23T00:20:54.074 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-02-23T00:20:54.077 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-23T00:20:54.077 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-23T00:20:54.077 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-23T00:20:54.077 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-23T00:20:54.077 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d5f5/64 scope link 2024-02-23T00:20:54.077 INFO:teuthology.orchestra.run.smithi087.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-23T00:20:54.078 INFO:teuthology.orchestra.run.smithi087.stderr:Mon IP `172.21.15.87` is in CIDR network `172.21.0.0/20` 2024-02-23T00:20:54.078 INFO:teuthology.orchestra.run.smithi087.stderr:Mon IP `172.21.15.87` is in CIDR network `172.21.0.0/20` 2024-02-23T00:20:54.078 INFO:teuthology.orchestra.run.smithi087.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-23T00:20:54.079 INFO:teuthology.orchestra.run.smithi087.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-23T00:20:54.079 INFO:teuthology.orchestra.run.smithi087.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59... 2024-02-23T00:20:54.184 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59... 2024-02-23T00:20:54.374 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Getting image source signatures 2024-02-23T00:20:54.379 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Copying blob sha256:6a725db67f9d8a3d1854323a2987a594424da05f10d8a333e41377f3b137fc12 2024-02-23T00:20:54.379 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-23T00:21:19.395 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Copying config sha256:7403d18623076b313def53d9a4ae24b05fb9f62358c436a81c22d7e96d9d9550 2024-02-23T00:21:19.569 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: Writing manifest to image destination 2024-02-23T00:21:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:/bin/podman: 7403d18623076b313def53d9a4ae24b05fb9f62358c436a81c22d7e96d9d9550 2024-02-23T00:21:20.575 INFO:teuthology.orchestra.run.smithi087.stderr:ceph: ceph version 16.2.14-550-geb66ed92 (eb66ed921e744301e6863be6353618d63967ea59) pacific (stable) 2024-02-23T00:21:21.086 INFO:teuthology.orchestra.run.smithi087.stderr:Ceph version: ceph version 16.2.14-550-geb66ed92 (eb66ed921e744301e6863be6353618d63967ea59) pacific (stable) 2024-02-23T00:21:21.087 INFO:teuthology.orchestra.run.smithi087.stderr:Extracting ceph user uid/gid from container image... 2024-02-23T00:21:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:stat: 167 167 2024-02-23T00:21:22.238 INFO:teuthology.orchestra.run.smithi087.stderr:Creating initial keys... 2024-02-23T00:21:22.754 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-authtool: AQAC5ddlExQvKBAADOMKiMplenRcDI2vhlnbSQ== 2024-02-23T00:21:23.952 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-authtool: AQAD5ddl96xwMhAA0+Wjkbscjh1CrqzPQ+ZlFA== 2024-02-23T00:21:25.092 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-authtool: AQAE5ddlh7gMOBAAGhbQdwMr/LeWBLVJyxkwcQ== 2024-02-23T00:21:25.641 INFO:teuthology.orchestra.run.smithi087.stderr:Creating initial monmap... 2024-02-23T00:21:26.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-23T00:21:26.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:26.132 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-23T00:21:26.707 INFO:teuthology.orchestra.run.smithi087.stderr:monmaptool for a [v2:172.21.15.87:3300,v1:172.21.15.87:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-23T00:21:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: set fsid to 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-23T00:21:26.708 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:21:26.708 INFO:teuthology.orchestra.run.smithi087.stderr:Creating mon... 2024-02-23T00:21:27.261 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.144+0000 7f00ee2c1880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-23T00:21:27.261 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.144+0000 7f00ee2c1880 1 imported monmap: 2024-02-23T00:21:27.261 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-23T00:21:27.261 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: fsid 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:27.261 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: last_changed 2024-02-23T00:21:26.056087+0000 2024-02-23T00:21:27.261 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: created 2024-02-23T00:21:26.056087+0000 2024-02-23T00:21:27.261 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-23T00:21:27.261 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-23T00:21:27.261 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.144+0000 7f00ee2c1880 0 /usr/bin/ceph-mon: set fsid to 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: RocksDB version: 6.8.1 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Compile date Feb 20 2024 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: DB SUMMARY 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.error_if_exists: 0 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.create_if_missing: 1 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.paranoid_checks: 1 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.env: 0x56274df58080 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.fs: Posix File System 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.info_log: 0x56274f8c5a00 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.statistics: (nil) 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.use_fsync: 0 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_log_file_size: 0 2024-02-23T00:21:27.263 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.allow_fallocate: 1 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.use_direct_reads: 0 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.db_log_dir: 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-23T00:21:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_subcompactions: 1 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_background_flushes: -1 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.write_buffer_manager: 0x56274f8cb920 2024-02-23T00:21:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.unordered_write: 0 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-23T00:21:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.row_cache: None 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.wal_filter: None 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.preserve_deletes: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.two_write_queues: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.atomic_flush: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.log_readahead_size: 0 2024-02-23T00:21:27.267 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_background_jobs: 2 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_background_compactions: -1 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.max_open_files: -1 2024-02-23T00:21:27.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Compression algorithms supported: 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: kZSTD supported: 0 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: kXpressCompression supported: 0 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: kLZ4Compression supported: 1 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: kBZip2Compression supported: 0 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: kZlibCompression supported: 1 2024-02-23T00:21:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: kSnappyCompression supported: 1 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.145+0000 7f00ee2c1880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.merge_operator: 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_filter: None 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-23T00:21:27.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56274f7dbd18) 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-23T00:21:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_cache: 0x56274f810d10 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-23T00:21:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-23T00:21:27.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compression: NoCompression 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.num_levels: 7 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-23T00:21:27.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-23T00:21:27.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-23T00:21:27.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-23T00:21:27.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.table_properties_collectors: 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.inplace_update_support: 0 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.bloom_locality: 0 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.max_successive_merges: 0 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-23T00:21:27.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.ttl: 2592000 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.146+0000 7f00ee2c1880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.147+0000 7f00ee2c1880 4 rocksdb: DB pointer 0x56274f8d9800 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.147+0000 7f00d7134700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.147+0000 7f00d7134700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-23T00:21:27.279 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:21:27.280 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:21:27.281 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:21:27.282 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.148+0000 7f00ee2c1880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.148+0000 7f00ee2c1880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-23T00:21:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph-mon: debug 2024-02-23T00:21:27.148+0000 7f00ee2c1880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-02-23T00:21:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:create mon.a on 2024-02-23T00:21:27.794 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-23T00:21:27.885 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-23T00:21:27.976 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-66db207c-d1e1-11ee-95c0-87774f69a715.target → /etc/systemd/system/ceph-66db207c-d1e1-11ee-95c0-87774f69a715.target. 2024-02-23T00:21:27.977 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-66db207c-d1e1-11ee-95c0-87774f69a715.target → /etc/systemd/system/ceph-66db207c-d1e1-11ee-95c0-87774f69a715.target. 2024-02-23T00:21:28.173 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Failed to reset failed state of unit ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.a.service: Unit ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.a.service not loaded. 2024-02-23T00:21:28.179 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/ceph-66db207c-d1e1-11ee-95c0-87774f69a715.target.wants/ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-66db207c-d1e1-11ee-95c0-87774f69a715@.service. 2024-02-23T00:21:28.924 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: disabled 2024-02-23T00:21:28.932 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-23T00:21:28.932 INFO:teuthology.orchestra.run.smithi087.stderr:firewalld.service is not enabled 2024-02-23T00:21:28.932 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to enable service . firewalld.service is not available 2024-02-23T00:21:28.933 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mon to start... 2024-02-23T00:21:28.933 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mon... 2024-02-23T00:21:29.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:28 smithi087 bash[37783]: c0a89bc606dc573dbf86e8f7e04cb428a92cab9718e4f7bb5b7970018ced392e 2024-02-23T00:21:29.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:28 smithi087 systemd[1]: Started Ceph mon.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:21:29.399 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.393+0000 7f0284918700 1 Processor -- start 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.393+0000 7f0284918700 1 -- start start 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.393+0000 7f0284918700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f02800fec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.393+0000 7f0284918700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02800ff1d0 con 0x7f0280070c80 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.394+0000 7f027e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f02800fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.394+0000 7f027e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f02800fec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36008/0 (socket says 172.21.15.87:36008) 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.394+0000 7f027e59c700 1 -- 172.21.15.87:0/3077969921 learned_addr learned my addr 172.21.15.87:0/3077969921 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.394+0000 7f027e59c700 1 -- 172.21.15.87:0/3077969921 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02800ff310 con 0x7f0280070c80 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.394+0000 7f027e59c700 1 --2- 172.21.15.87:0/3077969921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f02800fec50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0270009730 tx=0x7f02700090f0).ready entity=mon.0 client_cookie=1ed17cbf9b64b0c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.394+0000 7f027d59a700 1 -- 172.21.15.87:0/3077969921 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f027000b580 con 0x7f0280070c80 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.394+0000 7f027d59a700 1 -- 172.21.15.87:0/3077969921 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027000b6e0 con 0x7f0280070c80 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.394+0000 7f027d59a700 1 -- 172.21.15.87:0/3077969921 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f027000b990 con 0x7f0280070c80 2024-02-23T00:21:29.400 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.395+0000 7f0284918700 1 -- 172.21.15.87:0/3077969921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 msgr2=0x7f02800fec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.395+0000 7f0284918700 1 --2- 172.21.15.87:0/3077969921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f02800fec50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f0270009730 tx=0x7f02700090f0).stop 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.395+0000 7f0284918700 1 -- 172.21.15.87:0/3077969921 shutdown_connections 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.395+0000 7f0284918700 1 --2- 172.21.15.87:0/3077969921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f02800fec50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.395+0000 7f0284918700 1 -- 172.21.15.87:0/3077969921 >> 172.21.15.87:0/3077969921 conn(0x7f02800f6670 msgr2=0x7f02800f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.395+0000 7f0284918700 1 -- 172.21.15.87:0/3077969921 shutdown_connections 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.395+0000 7f0284918700 1 -- 172.21.15.87:0/3077969921 wait complete. 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.395+0000 7f0284918700 1 librados: starting msgr at 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.395+0000 7f0284918700 1 librados: starting objecter 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f0284918700 1 Processor -- start 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f0284918700 1 -- start start 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f0284918700 1 librados: setting wanted keys 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f0284918700 1 librados: calling monclient init 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f0284918700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f028010a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:29.401 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f0284918700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028010a920 con 0x7f0280070c80 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f027e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f028010a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f027e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f028010a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36014/0 (socket says 172.21.15.87:36014) 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f027e59c700 1 -- 172.21.15.87:0/108707831 learned_addr learned my addr 172.21.15.87:0/108707831 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f027e59c700 1 -- 172.21.15.87:0/108707831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0270009370 con 0x7f0280070c80 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f027e59c700 1 --2- 172.21.15.87:0/108707831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f028010a430 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0270009130 tx=0x7f0270017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f026f7fe700 10 client.?.objecter ms_handle_connect 0x7f0280070c80 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f026f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.396+0000 7f026f7fe700 1 -- 172.21.15.87:0/108707831 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0270011cd0 con 0x7f0280070c80 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f026f7fe700 1 -- 172.21.15.87:0/108707831 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0270011e30 con 0x7f0280070c80 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f026f7fe700 1 -- 172.21.15.87:0/108707831 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f027001d570 con 0x7f0280070c80 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f0284918700 1 -- 172.21.15.87:0/108707831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028010ab20 con 0x7f0280070c80 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f0284918700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f0284918700 1 -- 172.21.15.87:0/108707831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028010afb0 con 0x7f0280070c80 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f0284918700 1 librados: init done 2024-02-23T00:21:29.402 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f026f7fe700 1 -- 172.21.15.87:0/108707831 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f027001da00 con 0x7f0280070c80 2024-02-23T00:21:29.403 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f026f7fe700 1 -- 172.21.15.87:0/108707831 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0270026720 con 0x7f0280070c80 2024-02-23T00:21:29.403 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f026f7fe700 10 client.4098.objecter ms_dispatch 0x7f0280001550 osd_map(1..1 src has 1..1) v4 2024-02-23T00:21:29.403 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f026f7fe700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-02-23T00:21:29.403 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f026f7fe700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-02-23T00:21:29.403 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.397+0000 7f026f7fe700 20 client.4098.objecter dump_active .. 0 homeless 2024-02-23T00:21:29.403 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.398+0000 7f0284918700 1 -- 172.21.15.87:0/108707831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0280046ce0 con 0x7f0280070c80 2024-02-23T00:21:29.403 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.401+0000 7f026f7fe700 1 -- 172.21.15.87:0/108707831 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f0270021030 con 0x7f0280070c80 2024-02-23T00:21:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.467+0000 7f0284918700 1 -- 172.21.15.87:0/108707831 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f0280103ea0 con 0x7f0280070c80 2024-02-23T00:21:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.468+0000 7f026f7fe700 1 -- 172.21.15.87:0/108707831 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f0270026960 con 0x7f0280070c80 2024-02-23T00:21:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: cluster: 2024-02-23T00:21:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: id: 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-23T00:21:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: services: 2024-02-23T00:21:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.609637s) 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: data: 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: pgs: 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.468+0000 7f0284918700 10 librados: watch_flush enter 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.468+0000 7f0284918700 10 librados: watch_flush exit 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.468+0000 7f0284918700 20 client.4098.objecter shutdown clearing up homeless session... 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.468+0000 7f0284918700 10 client.4098.objecter successfully canceled tick 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.469+0000 7f0284918700 1 -- 172.21.15.87:0/108707831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 msgr2=0x7f028010a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.469+0000 7f0284918700 1 --2- 172.21.15.87:0/108707831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f028010a430 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0270009130 tx=0x7f0270017000).stop 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.469+0000 7f0284918700 1 -- 172.21.15.87:0/108707831 shutdown_connections 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.469+0000 7f0284918700 1 --2- 172.21.15.87:0/108707831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0280070c80 0x7f028010a430 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.469+0000 7f0284918700 1 -- 172.21.15.87:0/108707831 >> 172.21.15.87:0/108707831 conn(0x7f02800f6670 msgr2=0x7f02800f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.469+0000 7f0284918700 1 -- 172.21.15.87:0/108707831 shutdown_connections 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.469+0000 7f0284918700 1 -- 172.21.15.87:0/108707831 wait complete. 2024-02-23T00:21:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:29.469+0000 7f0284918700 1 librados: shutdown 2024-02-23T00:21:30.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: cluster 2024-02-23T00:21:28.859720+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-23T00:21:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: cluster 2024-02-23T00:21:28.860199+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} 2024-02-23T00:21:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: cluster 2024-02-23T00:21:28.863581+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-02-23T00:21:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: cluster 2024-02-23T00:21:28.867460+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-23T00:21:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: cluster 2024-02-23T00:21:28.867855+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-23T00:21:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: audit 2024-02-23T00:21:29.469308+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.87:0/108707831' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-23T00:21:30.722 INFO:teuthology.orchestra.run.smithi087.stderr:mon is available 2024-02-23T00:21:30.722 INFO:teuthology.orchestra.run.smithi087.stderr:Assimilating anything we can from ceph.conf... 2024-02-23T00:21:31.236 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.232+0000 7f3a43f47700 1 Processor -- start 2024-02-23T00:21:31.236 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.232+0000 7f3a43f47700 1 -- start start 2024-02-23T00:21:31.236 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.232+0000 7f3a43f47700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.232+0000 7f3a43f47700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0ff190 con 0x7f3a3c070cc0 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.232+0000 7f3a41ce3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.232+0000 7f3a41ce3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36024/0 (socket says 172.21.15.87:36024) 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.232+0000 7f3a41ce3700 1 -- 172.21.15.87:0/2798252158 learned_addr learned my addr 172.21.15.87:0/2798252158 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.232+0000 7f3a41ce3700 1 -- 172.21.15.87:0/2798252158 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c0ff2d0 con 0x7f3a3c070cc0 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.233+0000 7f3a41ce3700 1 --2- 172.21.15.87:0/2798252158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fec10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f3a38009730 tx=0x7f3a380090f0).ready entity=mon.0 client_cookie=77bbbc8aadc1643e server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.233+0000 7f3a40ce1700 1 -- 172.21.15.87:0/2798252158 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a3800b580 con 0x7f3a3c070cc0 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.233+0000 7f3a40ce1700 1 -- 172.21.15.87:0/2798252158 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3800b6e0 con 0x7f3a3c070cc0 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.233+0000 7f3a40ce1700 1 -- 172.21.15.87:0/2798252158 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a3800b990 con 0x7f3a3c070cc0 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.233+0000 7f3a43f47700 1 -- 172.21.15.87:0/2798252158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 msgr2=0x7f3a3c0fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.233+0000 7f3a43f47700 1 --2- 172.21.15.87:0/2798252158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fec10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f3a38009730 tx=0x7f3a380090f0).stop 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.233+0000 7f3a43f47700 1 -- 172.21.15.87:0/2798252158 shutdown_connections 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.233+0000 7f3a43f47700 1 --2- 172.21.15.87:0/2798252158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fec10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:31.237 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.233+0000 7f3a43f47700 1 -- 172.21.15.87:0/2798252158 >> 172.21.15.87:0/2798252158 conn(0x7f3a3c0f6670 msgr2=0x7f3a3c0f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:31.238 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 -- 172.21.15.87:0/2798252158 shutdown_connections 2024-02-23T00:21:31.238 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 -- 172.21.15.87:0/2798252158 wait complete. 2024-02-23T00:21:31.238 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 librados: starting msgr at 2024-02-23T00:21:31.238 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 librados: starting objecter 2024-02-23T00:21:31.238 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 Processor -- start 2024-02-23T00:21:31.238 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 -- start start 2024-02-23T00:21:31.238 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 librados: setting wanted keys 2024-02-23T00:21:31.238 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 librados: calling monclient init 2024-02-23T00:21:31.241 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:31.241 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a43f47700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0fab10 con 0x7f3a3c070cc0 2024-02-23T00:21:31.241 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.234+0000 7f3a41ce3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:31.241 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a41ce3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36032/0 (socket says 172.21.15.87:36032) 2024-02-23T00:21:31.241 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a41ce3700 1 -- 172.21.15.87:0/202845277 learned_addr learned my addr 172.21.15.87:0/202845277 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:31.241 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a41ce3700 1 -- 172.21.15.87:0/202845277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38009370 con 0x7f3a3c070cc0 2024-02-23T00:21:31.241 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a41ce3700 1 --2- 172.21.15.87:0/202845277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fc690 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f3a38009130 tx=0x7f3a38017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:31.241 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 10 client.?.objecter ms_handle_connect 0x7f3a3c070cc0 2024-02-23T00:21:31.241 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 1 -- 172.21.15.87:0/202845277 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a38011cd0 con 0x7f3a3c070cc0 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 1 -- 172.21.15.87:0/202845277 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a38011e30 con 0x7f3a3c070cc0 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a43f47700 1 -- 172.21.15.87:0/202845277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3c0facb0 con 0x7f3a3c070cc0 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 1 -- 172.21.15.87:0/202845277 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3a3801d570 con 0x7f3a3c070cc0 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a43f47700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a43f47700 1 -- 172.21.15.87:0/202845277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3c0fb140 con 0x7f3a3c070cc0 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a43f47700 1 librados: init done 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 1 -- 172.21.15.87:0/202845277 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f3a3801da00 con 0x7f3a3c070cc0 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 1 -- 172.21.15.87:0/202845277 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3a38026710 con 0x7f3a3c070cc0 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 10 client.4100.objecter ms_dispatch 0x7f3a3c001550 osd_map(1..1 src has 1..1) v4 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.235+0000 7f3a32ffd700 20 client.4100.objecter dump_active .. 0 homeless 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.237+0000 7f3a43f47700 1 -- 172.21.15.87:0/202845277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3c05a310 con 0x7f3a3c070cc0 2024-02-23T00:21:31.242 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.239+0000 7f3a32ffd700 1 -- 172.21.15.87:0/202845277 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f3a38021030 con 0x7f3a3c070cc0 2024-02-23T00:21:31.305 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.303+0000 7f3a43f47700 1 -- 172.21.15.87:0/202845277 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3a3c0ff640 con 0x7f3a3c070cc0 2024-02-23T00:21:31.309 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.306+0000 7f3a32ffd700 1 -- 172.21.15.87:0/202845277 <== mon.0 v2:172.21.15.87:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a380198a0 con 0x7f3a3c070cc0 2024-02-23T00:21:31.309 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.306+0000 7f3a32ffd700 1 -- 172.21.15.87:0/202845277 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+363 (secure 0 0 0) 0x7f3a380269f0 con 0x7f3a3c070cc0 2024-02-23T00:21:31.309 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: [global] 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: fsid = 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.87:3300,v1:172.21.15.87:6789] 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: [mgr] 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-23T00:21:31.310 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.312 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: [osd] 2024-02-23T00:21:31.312 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-23T00:21:31.312 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 10 librados: watch_flush enter 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 10 librados: watch_flush exit 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 20 client.4100.objecter shutdown clearing up homeless session... 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 10 client.4100.objecter successfully canceled tick 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 1 -- 172.21.15.87:0/202845277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 msgr2=0x7f3a3c0fc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 1 --2- 172.21.15.87:0/202845277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fc690 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f3a38009130 tx=0x7f3a38017000).stop 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 1 -- 172.21.15.87:0/202845277 shutdown_connections 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 1 --2- 172.21.15.87:0/202845277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a3c070cc0 0x7f3a3c0fc690 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 1 -- 172.21.15.87:0/202845277 >> 172.21.15.87:0/202845277 conn(0x7f3a3c0f6670 msgr2=0x7f3a3c0f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 1 -- 172.21.15.87:0/202845277 shutdown_connections 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 1 -- 172.21.15.87:0/202845277 wait complete. 2024-02-23T00:21:31.313 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:31.307+0000 7f3a43f47700 1 librados: shutdown 2024-02-23T00:21:31.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: audit 2024-02-23T00:21:31.304977+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.87:0/202845277' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-23T00:21:31.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: audit 2024-02-23T00:21:31.307914+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.87:0/202845277' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-23T00:21:31.853 INFO:teuthology.orchestra.run.smithi087.stderr:Generating new minimal ceph.conf... 2024-02-23T00:21:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.365+0000 7f4527b37700 1 Processor -- start 2024-02-23T00:21:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.365+0000 7f4527b37700 1 -- start start 2024-02-23T00:21:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.366+0000 7f4527b37700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f45200fec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.366+0000 7f4527b37700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45200ff1d0 con 0x7f4520070c80 2024-02-23T00:21:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.366+0000 7f45258d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f45200fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:32.373 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.366+0000 7f45258d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f45200fec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58630/0 (socket says 172.21.15.87:58630) 2024-02-23T00:21:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.366+0000 7f45258d3700 1 -- 172.21.15.87:0/677150671 learned_addr learned my addr 172.21.15.87:0/677150671 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.366+0000 7f45258d3700 1 -- 172.21.15.87:0/677150671 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45200ff310 con 0x7f4520070c80 2024-02-23T00:21:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.366+0000 7f45258d3700 1 --2- 172.21.15.87:0/677150671 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f45200fec50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f4510009730 tx=0x7f45100090f0).ready entity=mon.0 client_cookie=67059313a1e1300e server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f45248d1700 1 -- 172.21.15.87:0/677150671 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f451000b580 con 0x7f4520070c80 2024-02-23T00:21:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f45248d1700 1 -- 172.21.15.87:0/677150671 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f451000bb90 con 0x7f4520070c80 2024-02-23T00:21:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f45248d1700 1 -- 172.21.15.87:0/677150671 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4510011c30 con 0x7f4520070c80 2024-02-23T00:21:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f4527b37700 1 -- 172.21.15.87:0/677150671 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 msgr2=0x7f45200fec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f4527b37700 1 --2- 172.21.15.87:0/677150671 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f45200fec50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f4510009730 tx=0x7f45100090f0).stop 2024-02-23T00:21:32.374 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f4527b37700 1 -- 172.21.15.87:0/677150671 shutdown_connections 2024-02-23T00:21:32.375 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f4527b37700 1 --2- 172.21.15.87:0/677150671 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f45200fec50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:32.375 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f4527b37700 1 -- 172.21.15.87:0/677150671 >> 172.21.15.87:0/677150671 conn(0x7f45200f6670 msgr2=0x7f45200f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:32.375 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f4527b37700 1 -- 172.21.15.87:0/677150671 shutdown_connections 2024-02-23T00:21:32.375 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.367+0000 7f4527b37700 1 -- 172.21.15.87:0/677150671 wait complete. 2024-02-23T00:21:32.375 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f4527b37700 1 librados: starting msgr at 2024-02-23T00:21:32.375 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f4527b37700 1 librados: starting objecter 2024-02-23T00:21:32.375 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f4527b37700 1 Processor -- start 2024-02-23T00:21:32.376 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f4527b37700 1 -- start start 2024-02-23T00:21:32.376 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f4527b37700 1 librados: setting wanted keys 2024-02-23T00:21:32.376 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f4527b37700 1 librados: calling monclient init 2024-02-23T00:21:32.376 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f4527b37700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f452010ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:32.376 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f4527b37700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452010b270 con 0x7f4520070c80 2024-02-23T00:21:32.376 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f45258d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f452010ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:32.376 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f45258d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f452010ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58644/0 (socket says 172.21.15.87:58644) 2024-02-23T00:21:32.376 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f45258d3700 1 -- 172.21.15.87:0/766787896 learned_addr learned my addr 172.21.15.87:0/766787896 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:32.377 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.368+0000 7f45258d3700 1 -- 172.21.15.87:0/766787896 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4510009370 con 0x7f4520070c80 2024-02-23T00:21:32.377 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f45258d3700 1 --2- 172.21.15.87:0/766787896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f452010ad80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4510006940 tx=0x7f45100090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:32.377 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 10 client.?.objecter ms_handle_connect 0x7f4520070c80 2024-02-23T00:21:32.377 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:32.377 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 1 -- 172.21.15.87:0/766787896 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f451001a430 con 0x7f4520070c80 2024-02-23T00:21:32.377 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 1 -- 172.21.15.87:0/766787896 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f451001aa40 con 0x7f4520070c80 2024-02-23T00:21:32.377 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4527b37700 1 -- 172.21.15.87:0/766787896 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452010b470 con 0x7f4520070c80 2024-02-23T00:21:32.378 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 1 -- 172.21.15.87:0/766787896 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4510023b00 con 0x7f4520070c80 2024-02-23T00:21:32.378 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4527b37700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:32.378 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4527b37700 1 -- 172.21.15.87:0/766787896 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452010c080 con 0x7f4520070c80 2024-02-23T00:21:32.378 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4527b37700 1 librados: init done 2024-02-23T00:21:32.378 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 1 -- 172.21.15.87:0/766787896 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f451002c650 con 0x7f4520070c80 2024-02-23T00:21:32.378 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 1 -- 172.21.15.87:0/766787896 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4510022770 con 0x7f4520070c80 2024-02-23T00:21:32.378 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 10 client.4102.objecter ms_dispatch 0x7f4520001550 osd_map(1..1 src has 1..1) v4 2024-02-23T00:21:32.378 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-02-23T00:21:32.379 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-02-23T00:21:32.379 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.369+0000 7f4516ffd700 20 client.4102.objecter dump_active .. 0 homeless 2024-02-23T00:21:32.379 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.370+0000 7f4527b37700 1 -- 172.21.15.87:0/766787896 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452010b900 con 0x7f4520070c80 2024-02-23T00:21:32.379 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.372+0000 7f4516ffd700 1 -- 172.21.15.87:0/766787896 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f452010b900 con 0x7f4520070c80 2024-02-23T00:21:32.438 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.436+0000 7f4527b37700 1 -- 172.21.15.87:0/766787896 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f452005a310 con 0x7f4520070c80 2024-02-23T00:21:32.438 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.436+0000 7f4516ffd700 1 -- 172.21.15.87:0/766787896 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f451001e030 con 0x7f4520070c80 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 10 librados: watch_flush enter 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 10 librados: watch_flush exit 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 20 client.4102.objecter shutdown clearing up homeless session... 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 10 client.4102.objecter successfully canceled tick 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 1 -- 172.21.15.87:0/766787896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 msgr2=0x7f452010ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 1 --2- 172.21.15.87:0/766787896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f452010ad80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4510006940 tx=0x7f45100090f0).stop 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 1 -- 172.21.15.87:0/766787896 shutdown_connections 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 1 --2- 172.21.15.87:0/766787896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4520070c80 0x7f452010ad80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 1 -- 172.21.15.87:0/766787896 >> 172.21.15.87:0/766787896 conn(0x7f45200f6670 msgr2=0x7f45200f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 1 -- 172.21.15.87:0/766787896 shutdown_connections 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 1 -- 172.21.15.87:0/766787896 wait complete. 2024-02-23T00:21:32.440 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:32.437+0000 7f4527b37700 1 librados: shutdown 2024-02-23T00:21:32.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: audit 2024-02-23T00:21:32.438030+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.87:0/766787896' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:21:33.110 INFO:teuthology.orchestra.run.smithi087.stderr:Restarting the monitor... 2024-02-23T00:21:33.367 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:33 smithi087 systemd[1]: Stopping Ceph mon.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:21:33.368 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:33 smithi087 bash[38120]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon.a" found: no such container 2024-02-23T00:21:33.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: debug 2024-02-23T00:21:33.365+0000 7f7977a68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-23T00:21:33.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[37797]: debug 2024-02-23T00:21:33.365+0000 7f7977a68700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-23T00:21:34.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:33 smithi087 bash[38160]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a 2024-02-23T00:21:34.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.a.service: Succeeded. 2024-02-23T00:21:34.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 systemd[1]: Stopped Ceph mon.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:21:34.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 systemd[1]: Starting Ceph mon.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:21:34.827 INFO:teuthology.orchestra.run.smithi087.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-23T00:21:35.064 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.764+0000 7fe8f8aa0880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-23T00:21:35.064 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.764+0000 7fe8f8aa0880 0 ceph version 16.2.14-550-geb66ed92 (eb66ed921e744301e6863be6353618d63967ea59) pacific (stable), process ceph-mon, pid 7 2024-02-23T00:21:35.064 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.764+0000 7fe8f8aa0880 0 pidfile_write: ignore empty --pid-file 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 0 load: jerasure load: lrc load: isa 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: RocksDB version: 6.8.1 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Compile date Feb 20 2024 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: DB SUMMARY 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: CURRENT file: CURRENT 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-23T00:21:35.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 94571 ; 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.error_if_exists: 0 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.create_if_missing: 0 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.paranoid_checks: 1 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.env: 0x55888cc58080 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.fs: Posix File System 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.info_log: 0x55888dfd0260 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.statistics: (nil) 2024-02-23T00:21:35.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.use_fsync: 0 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_log_file_size: 0 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.allow_fallocate: 1 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.use_direct_reads: 0 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-23T00:21:35.067 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.db_log_dir: 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_subcompactions: 1 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_background_flushes: -1 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.write_buffer_manager: 0x55888ecc23c0 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-23T00:21:35.068 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-23T00:21:35.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-23T00:21:35.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-23T00:21:35.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-23T00:21:35.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-23T00:21:35.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-23T00:21:35.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-23T00:21:35.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.unordered_write: 0 2024-02-23T00:21:35.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-23T00:21:35.069 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.row_cache: None 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.wal_filter: None 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.preserve_deletes: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.two_write_queues: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.atomic_flush: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.log_readahead_size: 0 2024-02-23T00:21:35.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-23T00:21:35.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_background_jobs: 2 2024-02-23T00:21:35.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_background_compactions: -1 2024-02-23T00:21:35.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-23T00:21:35.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-23T00:21:35.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-23T00:21:35.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-23T00:21:35.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-23T00:21:35.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-23T00:21:35.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_open_files: -1 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Compression algorithms supported: 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: kZSTD supported: 0 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: kXpressCompression supported: 0 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: kLZ4Compression supported: 1 2024-02-23T00:21:35.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: kBZip2Compression supported: 0 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: kZlibCompression supported: 1 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: kSnappyCompression supported: 1 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.merge_operator: 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_filter: None 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55888df75d10) 2024-02-23T00:21:35.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cache_index_and_filter_blocks: 1 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: pin_top_level_index_and_filter: 1 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: index_type: 0 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: data_block_index_type: 0 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: index_shortening: 1 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: data_block_hash_table_util_ratio: 0.750000 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: hash_index_allow_collision: 1 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: checksum: 1 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: no_block_cache: 0 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: block_cache: 0x55888dfaaf10 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: block_cache_name: BinnedLRUCache 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: block_cache_options: 2024-02-23T00:21:35.074 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: capacity : 536870912 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: num_shard_bits : 4 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: strict_capacity_limit : 0 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: high_pri_pool_ratio: 0.000 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: block_cache_compressed: (nil) 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: persistent_cache: (nil) 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: block_size: 4096 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: block_size_deviation: 10 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: block_restart_interval: 16 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: index_block_restart_interval: 1 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: metadata_block_size: 4096 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: partition_filters: 0 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: use_delta_encoding: 1 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: whole_key_filtering: 1 2024-02-23T00:21:35.075 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: verify_compression: 0 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: read_amp_bytes_per_bit: 0 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: format_version: 2 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: enable_index_compression: 1 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: block_align: 0 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compression: NoCompression 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.num_levels: 7 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-23T00:21:35.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-23T00:21:35.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-23T00:21:35.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.769+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-23T00:21:35.079 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-23T00:21:35.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-23T00:21:35.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.table_properties_collectors: 2024-02-23T00:21:35.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.inplace_update_support: 0 2024-02-23T00:21:35.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-23T00:21:35.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.bloom_locality: 0 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.max_successive_merges: 0 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.ttl: 2592000 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-23T00:21:35.081 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647694771920, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.770+0000 7fe8f8aa0880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.771+0000 7fe8f8aa0880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647694772764, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 90411, "table_properties": {"data_size": 88513, "index_size": 375, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 11592, "raw_average_key_size": 49, "raw_value_size": 82152, "raw_average_value_size": 348, "num_data_blocks": 12, "num_entries": 236, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708647694, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.771+0000 7fe8f8aa0880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.771+0000 7fe8f8aa0880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647694773130, "job": 1, "event": "recovery_finished"} 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.772+0000 7fe8f8aa0880 4 rocksdb: DB pointer 0x55888e073800 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.772+0000 7fe8df892700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-23T00:21:35.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.772+0000 7fe8df892700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ** DB Stats ** 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ** Compaction Stats [default] ** 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: L0 2/0 89.95 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.3 0.00 0.00 1 0.001 0 0 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Sum 2/0 89.95 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.3 0.00 0.00 1 0.001 0 0 2024-02-23T00:21:35.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.3 0.00 0.00 1 0.001 0 0 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ** Compaction Stats [default] ** 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 152.3 0.00 0.00 1 0.001 0 0 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Cumulative compaction: 0.00 GB write, 33.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:21:35.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Interval compaction: 0.00 GB write, 33.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ** Compaction Stats [default] ** 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: L0 2/0 89.95 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.3 0.00 0.00 1 0.001 0 0 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Sum 2/0 89.95 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.3 0.00 0.00 1 0.001 0 0 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ** Compaction Stats [default] ** 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:21:35.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 152.3 0.00 0.00 1 0.001 0 0 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Cumulative compaction: 0.00 GB write, 32.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.773+0000 7fe8f8aa0880 0 starting mon.a rank 0 at public addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] at bind addrs [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.774+0000 7fe8f8aa0880 1 mon.a@-1(???) e1 preinit fsid 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.774+0000 7fe8f8aa0880 0 mon.a@-1(???).mds e1 new map 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.774+0000 7fe8f8aa0880 0 mon.a@-1(???).mds e1 print_map 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: e1 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: legacy client fscid: -1 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: No filesystems configured 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.774+0000 7fe8f8aa0880 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.774+0000 7fe8f8aa0880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:21:35.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.774+0000 7fe8f8aa0880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:21:35.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.774+0000 7fe8f8aa0880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:21:35.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:32.438030+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.87:0/766787896' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:21:35.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:21:34.774+0000 7fe8f8aa0880 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-23T00:21:35.089 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 bash[38284]: 90359c0f70a04bfb1336d0c04c8dda2bd1882992ff22d54a9735c98e8ad2dd44 2024-02-23T00:21:35.089 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 systemd[1]: Started Ceph mon.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:21:35.089 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:34.791227+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-23T00:21:35.089 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:34.791284+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]} removed_ranks: {} 2024-02-23T00:21:35.089 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:34.791648+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-02-23T00:21:35.089 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:34.791693+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-23T00:21:35.089 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:34.792126+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.884+0000 7f06d5983700 1 Processor -- start 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.884+0000 7f06d5983700 1 -- start start 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.885+0000 7f06d5983700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d00fb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.885+0000 7f06d5983700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d00fb8e0 con 0x7f06d00faf40 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.885+0000 7f06ceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d00fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.885+0000 7f06ceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d00fb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58652/0 (socket says 172.21.15.87:58652) 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.885+0000 7f06ceffd700 1 -- 172.21.15.87:0/3084868068 learned_addr learned my addr 172.21.15.87:0/3084868068 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.885+0000 7f06ceffd700 1 -- 172.21.15.87:0/3084868068 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d00fba20 con 0x7f06d00faf40 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06ceffd700 1 --2- 172.21.15.87:0/3084868068 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d00fb360 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f06c0009e50 tx=0x7f06c0009840).ready entity=mon.0 client_cookie=3d2f17cd48cbcbf6 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06cdffb700 1 -- 172.21.15.87:0/3084868068 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06c000e070 con 0x7f06d00faf40 2024-02-23T00:21:36.529 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06cdffb700 1 -- 172.21.15.87:0/3084868068 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06c0013420 con 0x7f06d00faf40 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06cdffb700 1 -- 172.21.15.87:0/3084868068 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06c0007400 con 0x7f06d00faf40 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06d5983700 1 -- 172.21.15.87:0/3084868068 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 msgr2=0x7f06d00fb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06d5983700 1 --2- 172.21.15.87:0/3084868068 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d00fb360 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f06c0009e50 tx=0x7f06c0009840).stop 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06d5983700 1 -- 172.21.15.87:0/3084868068 shutdown_connections 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06d5983700 1 --2- 172.21.15.87:0/3084868068 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d00fb360 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06d5983700 1 -- 172.21.15.87:0/3084868068 >> 172.21.15.87:0/3084868068 conn(0x7f06d00f6660 msgr2=0x7f06d00f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06d5983700 1 -- 172.21.15.87:0/3084868068 shutdown_connections 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.886+0000 7f06d5983700 1 -- 172.21.15.87:0/3084868068 wait complete. 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06d5983700 1 librados: starting msgr at 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06d5983700 1 librados: starting objecter 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06d5983700 1 Processor -- start 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06d5983700 1 -- start start 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06d5983700 1 librados: setting wanted keys 2024-02-23T00:21:36.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06d5983700 1 librados: calling monclient init 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06d5983700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d010ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06d5983700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d010b270 con 0x7f06d00faf40 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06ceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d010ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06ceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d010ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58656/0 (socket says 172.21.15.87:58656) 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.887+0000 7f06ceffd700 1 -- 172.21.15.87:0/3066366952 learned_addr learned my addr 172.21.15.87:0/3066366952 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06ceffd700 1 -- 172.21.15.87:0/3066366952 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c0009ae0 con 0x7f06d00faf40 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06ceffd700 1 --2- 172.21.15.87:0/3066366952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d010ad80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f06c0013aa0 tx=0x7f06c0009840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 10 client.?.objecter ms_handle_connect 0x7f06d00faf40 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 1 -- 172.21.15.87:0/3066366952 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06c000e040 con 0x7f06d00faf40 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06d5983700 1 -- 172.21.15.87:0/3066366952 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d010b470 con 0x7f06d00faf40 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 1 -- 172.21.15.87:0/3066366952 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06c000bc70 con 0x7f06d00faf40 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 1 -- 172.21.15.87:0/3066366952 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06c001c830 con 0x7f06d00faf40 2024-02-23T00:21:36.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06d5983700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06d5983700 1 -- 172.21.15.87:0/3066366952 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d010c080 con 0x7f06d00faf40 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06d5983700 1 librados: init done 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 1 -- 172.21.15.87:0/3066366952 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f06c0003040 con 0x7f06d00faf40 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 1 -- 172.21.15.87:0/3066366952 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f06c0025840 con 0x7f06d00faf40 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 10 client.14098.objecter ms_dispatch 0x7f06d0001550 osd_map(1..1 src has 1..1) v4 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.888+0000 7f06c7fff700 20 client.14098.objecter dump_active .. 0 homeless 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.889+0000 7f06d5983700 1 -- 172.21.15.87:0/3066366952 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d010b900 con 0x7f06d00faf40 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.891+0000 7f06c7fff700 1 -- 172.21.15.87:0/3066366952 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f06d010b900 con 0x7f06d00faf40 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.955+0000 7f06d5983700 1 -- 172.21.15.87:0/3066366952 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f06d0046ce0 con 0x7f06d00faf40 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.957+0000 7f06c7fff700 1 -- 172.21.15.87:0/3066366952 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f06c0018040 con 0x7f06d00faf40 2024-02-23T00:21:36.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.957+0000 7f06d5983700 10 librados: watch_flush enter 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 10 librados: watch_flush exit 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 20 client.14098.objecter shutdown clearing up homeless session... 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 10 client.14098.objecter successfully canceled tick 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 1 -- 172.21.15.87:0/3066366952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 msgr2=0x7f06d010ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 1 --2- 172.21.15.87:0/3066366952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d010ad80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f06c0013aa0 tx=0x7f06c0009840).stop 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 1 -- 172.21.15.87:0/3066366952 shutdown_connections 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 1 --2- 172.21.15.87:0/3066366952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f06d00faf40 0x7f06d010ad80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 1 -- 172.21.15.87:0/3066366952 >> 172.21.15.87:0/3066366952 conn(0x7f06d00f6660 msgr2=0x7f06d0070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 1 -- 172.21.15.87:0/3066366952 shutdown_connections 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 1 -- 172.21.15.87:0/3066366952 wait complete. 2024-02-23T00:21:36.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:35.958+0000 7f06d5983700 1 librados: shutdown 2024-02-23T00:21:37.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:35.958583+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.87:0/3066366952' entity='client.admin' 2024-02-23T00:21:38.337 INFO:teuthology.orchestra.run.smithi087.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-23T00:21:38.338 INFO:teuthology.orchestra.run.smithi087.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:21:38.338 INFO:teuthology.orchestra.run.smithi087.stderr:Creating mgr... 2024-02-23T00:21:38.338 INFO:teuthology.orchestra.run.smithi087.stderr:Verifying port 9283 ... 2024-02-23T00:21:38.467 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Failed to reset failed state of unit ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mgr.y.service: Unit ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mgr.y.service not loaded. 2024-02-23T00:21:38.473 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: Created symlink /etc/systemd/system/ceph-66db207c-d1e1-11ee-95c0-87774f69a715.target.wants/ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-66db207c-d1e1-11ee-95c0-87774f69a715@.service. 2024-02-23T00:21:38.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:38 smithi087 systemd[1]: Starting Ceph mgr.y for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:21:39.225 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: disabled 2024-02-23T00:21:39.234 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-23T00:21:39.234 INFO:teuthology.orchestra.run.smithi087.stderr:firewalld.service is not enabled 2024-02-23T00:21:39.234 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to enable service . firewalld.service is not available 2024-02-23T00:21:39.242 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: disabled 2024-02-23T00:21:39.249 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-23T00:21:39.250 INFO:teuthology.orchestra.run.smithi087.stderr:firewalld.service is not enabled 2024-02-23T00:21:39.250 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-23T00:21:39.250 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mgr to start... 2024-02-23T00:21:39.250 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mgr... 2024-02-23T00:21:39.526 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:39 smithi087 bash[38559]: 80338b79c5160d3ea40489321ebb45160509e7b105168b68df8eaa2c8ae2619b 2024-02-23T00:21:39.526 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:39 smithi087 systemd[1]: Started Ceph mgr.y for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:21:39.526 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:39.258+0000 7fae43407680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-23T00:21:39.526 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:39.342+0000 7fae43407680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-23T00:21:39.832 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:39.716+0000 7fae43407680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.859+0000 7f1aef8c1700 1 Processor -- start 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.859+0000 7f1aef8c1700 1 -- start start 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.859+0000 7f1aef8c1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae80fec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.859+0000 7f1aef8c1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80ff1d0 con 0x7f1ae8070c80 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.859+0000 7f1aed65d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae80fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.859+0000 7f1aed65d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae80fec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58684/0 (socket says 172.21.15.87:58684) 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.859+0000 7f1aed65d700 1 -- 172.21.15.87:0/3893053519 learned_addr learned my addr 172.21.15.87:0/3893053519 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.860+0000 7f1aed65d700 1 -- 172.21.15.87:0/3893053519 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae80ff310 con 0x7f1ae8070c80 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.860+0000 7f1aed65d700 1 --2- 172.21.15.87:0/3893053519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae80fec50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1ae4009730 tx=0x7f1ae40090f0).ready entity=mon.0 client_cookie=6c5dfe0b6c2e0fe server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:39.891 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.860+0000 7f1adffff700 1 -- 172.21.15.87:0/3893053519 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ae400b580 con 0x7f1ae8070c80 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.860+0000 7f1adffff700 1 -- 172.21.15.87:0/3893053519 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ae400bb90 con 0x7f1ae8070c80 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.860+0000 7f1adffff700 1 -- 172.21.15.87:0/3893053519 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ae4011c30 con 0x7f1ae8070c80 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.860+0000 7f1aef8c1700 1 -- 172.21.15.87:0/3893053519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 msgr2=0x7f1ae80fec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.860+0000 7f1aef8c1700 1 --2- 172.21.15.87:0/3893053519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae80fec50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f1ae4009730 tx=0x7f1ae40090f0).stop 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 -- 172.21.15.87:0/3893053519 shutdown_connections 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 --2- 172.21.15.87:0/3893053519 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae80fec50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 -- 172.21.15.87:0/3893053519 >> 172.21.15.87:0/3893053519 conn(0x7f1ae80f6670 msgr2=0x7f1ae80f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 -- 172.21.15.87:0/3893053519 shutdown_connections 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 -- 172.21.15.87:0/3893053519 wait complete. 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 librados: starting msgr at 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 librados: starting objecter 2024-02-23T00:21:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 Processor -- start 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 -- start start 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 librados: setting wanted keys 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.861+0000 7f1aef8c1700 1 librados: calling monclient init 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aef8c1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae810ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aef8c1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae810b270 con 0x7f1ae8070c80 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aed65d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae810ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aed65d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae810ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58692/0 (socket says 172.21.15.87:58692) 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aed65d700 1 -- 172.21.15.87:0/910200814 learned_addr learned my addr 172.21.15.87:0/910200814 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aed65d700 1 -- 172.21.15.87:0/910200814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae4009370 con 0x7f1ae8070c80 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aed65d700 1 --2- 172.21.15.87:0/910200814 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae810ad80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1ae4006940 tx=0x7f1ae40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1ade7fc700 10 client.?.objecter ms_handle_connect 0x7f1ae8070c80 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1ade7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1ade7fc700 1 -- 172.21.15.87:0/910200814 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ae401a430 con 0x7f1ae8070c80 2024-02-23T00:21:39.893 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1ade7fc700 1 -- 172.21.15.87:0/910200814 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ae401aa40 con 0x7f1ae8070c80 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aef8c1700 1 -- 172.21.15.87:0/910200814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae810b470 con 0x7f1ae8070c80 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1ade7fc700 1 -- 172.21.15.87:0/910200814 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ae4023b90 con 0x7f1ae8070c80 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aef8c1700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aef8c1700 1 -- 172.21.15.87:0/910200814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae810c080 con 0x7f1ae8070c80 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.862+0000 7f1aef8c1700 1 librados: init done 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.863+0000 7f1aef8c1700 1 -- 172.21.15.87:0/910200814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae810b900 con 0x7f1ae8070c80 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.863+0000 7f1ade7fc700 1 -- 172.21.15.87:0/910200814 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f1ae401a590 con 0x7f1ae8070c80 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.863+0000 7f1ade7fc700 1 -- 172.21.15.87:0/910200814 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1ae402cca0 con 0x7f1ae8070c80 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.863+0000 7f1ade7fc700 10 client.14102.objecter ms_dispatch 0x7f1ae8001550 osd_map(1..1 src has 1..1) v4 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.863+0000 7f1ade7fc700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.863+0000 7f1ade7fc700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.863+0000 7f1ade7fc700 20 client.14102.objecter dump_active .. 0 homeless 2024-02-23T00:21:39.894 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.865+0000 7f1ade7fc700 1 -- 172.21.15.87:0/910200814 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f1ae810b900 con 0x7f1ae8070c80 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.934+0000 7f1aef8c1700 1 -- 172.21.15.87:0/910200814 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1ae805a310 con 0x7f1ae8070c80 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.934+0000 7f1ade7fc700 1 -- 172.21.15.87:0/910200814 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (secure 0 0 0) 0x7f1ae401e030 con 0x7f1ae8070c80 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "health": { 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "checks": {}, 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mutes": [] 2024-02-23T00:21:39.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum": [ 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 0 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "a" 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "monmap": { 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:39.944 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osdmap": { 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgmap": { 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-23T00:21:39.945 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsmap": { 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": false, 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modules": [ 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "iostat", 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "nfs", 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "restful" 2024-02-23T00:21:39.946 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "servicemap": { 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modified": "2024-02-23T00:21:28.861877+0000", 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 10 librados: watch_flush enter 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 10 librados: watch_flush exit 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 20 client.14102.objecter shutdown clearing up homeless session... 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 10 client.14102.objecter successfully canceled tick 2024-02-23T00:21:39.947 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 1 -- 172.21.15.87:0/910200814 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 msgr2=0x7f1ae810ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:39.948 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 1 --2- 172.21.15.87:0/910200814 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae810ad80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1ae4006940 tx=0x7f1ae40090f0).stop 2024-02-23T00:21:39.948 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 1 -- 172.21.15.87:0/910200814 shutdown_connections 2024-02-23T00:21:39.948 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 1 --2- 172.21.15.87:0/910200814 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ae8070c80 0x7f1ae810ad80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:39.948 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 1 -- 172.21.15.87:0/910200814 >> 172.21.15.87:0/910200814 conn(0x7f1ae80f6670 msgr2=0x7f1ae80f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:39.948 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 1 -- 172.21.15.87:0/910200814 shutdown_connections 2024-02-23T00:21:39.948 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 1 -- 172.21.15.87:0/910200814 wait complete. 2024-02-23T00:21:39.948 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:39.935+0000 7f1aef8c1700 1 librados: shutdown 2024-02-23T00:21:40.327 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:39.935636+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.87:0/910200814' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-23T00:21:40.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:40.477+0000 7fae43407680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-23T00:21:40.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:40.560+0000 7fae43407680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-23T00:21:40.605 INFO:teuthology.orchestra.run.smithi087.stderr:mgr not available, waiting (1/15)... 2024-02-23T00:21:41.082 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:40.771+0000 7fae43407680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-23T00:21:41.989 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:41.697+0000 7fae43407680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-23T00:21:41.989 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:41.822+0000 7fae43407680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:21:41.989 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:41.899+0000 7fae43407680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-23T00:21:41.989 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:41.987+0000 7fae43407680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-23T00:21:42.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:42.179+0000 7fae43407680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-23T00:21:42.832 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:42.452+0000 7fae43407680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-23T00:21:42.832 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:42.548+0000 7fae43407680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-23T00:21:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fd6cf0700 1 Processor -- start 2024-02-23T00:21:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fd6cf0700 1 -- start start 2024-02-23T00:21:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fd6cf0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd00fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fd6cf0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd00ff190 con 0x7f3fd0070cc0 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fd4a8c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd00fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fd4a8c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd00fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53952/0 (socket says 172.21.15.87:53952) 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fd4a8c700 1 -- 172.21.15.87:0/3337249013 learned_addr learned my addr 172.21.15.87:0/3337249013 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fd4a8c700 1 -- 172.21.15.87:0/3337249013 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd00ff2d0 con 0x7f3fd0070cc0 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fd4a8c700 1 --2- 172.21.15.87:0/3337249013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd00fec10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3fc4009730 tx=0x7f3fc40090f0).ready entity=mon.0 client_cookie=55f5f910b78f10b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.188+0000 7f3fcf7fe700 1 -- 172.21.15.87:0/3337249013 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fc400b580 con 0x7f3fd0070cc0 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.189+0000 7f3fcf7fe700 1 -- 172.21.15.87:0/3337249013 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fc400bb90 con 0x7f3fd0070cc0 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.189+0000 7f3fcf7fe700 1 -- 172.21.15.87:0/3337249013 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fc4011c30 con 0x7f3fd0070cc0 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.189+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3337249013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 msgr2=0x7f3fd00fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.189+0000 7f3fd6cf0700 1 --2- 172.21.15.87:0/3337249013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd00fec10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3fc4009730 tx=0x7f3fc40090f0).stop 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.189+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3337249013 shutdown_connections 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.189+0000 7f3fd6cf0700 1 --2- 172.21.15.87:0/3337249013 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd00fec10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.189+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3337249013 >> 172.21.15.87:0/3337249013 conn(0x7f3fd00f6670 msgr2=0x7f3fd00f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.189+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3337249013 shutdown_connections 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.189+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3337249013 wait complete. 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd6cf0700 1 librados: starting msgr at 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd6cf0700 1 librados: starting objecter 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd6cf0700 1 Processor -- start 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd6cf0700 1 -- start start 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd6cf0700 1 librados: setting wanted keys 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd6cf0700 1 librados: calling monclient init 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd6cf0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd010ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd6cf0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd010b270 con 0x7f3fd0070cc0 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd4a8c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd010ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd4a8c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd010ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53962/0 (socket says 172.21.15.87:53962) 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd4a8c700 1 -- 172.21.15.87:0/3057338922 learned_addr learned my addr 172.21.15.87:0/3057338922 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd4a8c700 1 -- 172.21.15.87:0/3057338922 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc4009370 con 0x7f3fd0070cc0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fd4a8c700 1 --2- 172.21.15.87:0/3057338922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd010ad80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f3fc4006940 tx=0x7f3fc40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fcdffb700 10 client.?.objecter ms_handle_connect 0x7f3fd0070cc0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.190+0000 7f3fcdffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fcdffb700 1 -- 172.21.15.87:0/3057338922 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fc401a430 con 0x7f3fd0070cc0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3057338922 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd010b470 con 0x7f3fd0070cc0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fcdffb700 1 -- 172.21.15.87:0/3057338922 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fc401aa40 con 0x7f3fd0070cc0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fcdffb700 1 -- 172.21.15.87:0/3057338922 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3fc4023b00 con 0x7f3fd0070cc0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fd6cf0700 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3057338922 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd010c080 con 0x7f3fd0070cc0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fd6cf0700 1 librados: init done 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fcdffb700 1 -- 172.21.15.87:0/3057338922 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f3fc402c650 con 0x7f3fd0070cc0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fcdffb700 1 -- 172.21.15.87:0/3057338922 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3fc4022730 con 0x7f3fd0070cc0 2024-02-23T00:21:43.247 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fcdffb700 10 client.14104.objecter ms_dispatch 0x7f3fd0001550 osd_map(1..1 src has 1..1) v4 2024-02-23T00:21:43.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fcdffb700 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-02-23T00:21:43.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fcdffb700 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-02-23T00:21:43.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.191+0000 7f3fcdffb700 20 client.14104.objecter dump_active .. 0 homeless 2024-02-23T00:21:43.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.192+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3057338922 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd010b900 con 0x7f3fd0070cc0 2024-02-23T00:21:43.248 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.195+0000 7f3fcdffb700 1 -- 172.21.15.87:0/3057338922 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f3fd010b900 con 0x7f3fd0070cc0 2024-02-23T00:21:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.264+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3057338922 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3fd005a310 con 0x7f3fd0070cc0 2024-02-23T00:21:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.265+0000 7f3fcdffb700 1 -- 172.21.15.87:0/3057338922 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (secure 0 0 0) 0x7f3fd005a310 con 0x7f3fd0070cc0 2024-02-23T00:21:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-23T00:21:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:21:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "health": { 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "checks": {}, 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mutes": [] 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum": [ 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 0 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "a" 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "monmap": { 2024-02-23T00:21:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osdmap": { 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgmap": { 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-23T00:21:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsmap": { 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-23T00:21:43.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": false, 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modules": [ 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "iostat", 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "nfs", 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "restful" 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "servicemap": { 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modified": "2024-02-23T00:21:28.861877+0000", 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-23T00:21:43.277 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.265+0000 7f3fd6cf0700 10 librados: watch_flush enter 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.265+0000 7f3fd6cf0700 10 librados: watch_flush exit 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.265+0000 7f3fd6cf0700 20 client.14104.objecter shutdown clearing up homeless session... 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.265+0000 7f3fd6cf0700 10 client.14104.objecter successfully canceled tick 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.266+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3057338922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 msgr2=0x7f3fd010ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.266+0000 7f3fd6cf0700 1 --2- 172.21.15.87:0/3057338922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd010ad80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f3fc4006940 tx=0x7f3fc40090f0).stop 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.266+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3057338922 shutdown_connections 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.266+0000 7f3fd6cf0700 1 --2- 172.21.15.87:0/3057338922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3fd0070cc0 0x7f3fd010ad80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.266+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3057338922 >> 172.21.15.87:0/3057338922 conn(0x7f3fd00f6670 msgr2=0x7f3fd00f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.266+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3057338922 shutdown_connections 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.266+0000 7f3fd6cf0700 1 -- 172.21.15.87:0/3057338922 wait complete. 2024-02-23T00:21:43.278 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:43.266+0000 7f3fd6cf0700 1 librados: shutdown 2024-02-23T00:21:43.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:43.266185+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.87:0/3057338922' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-23T00:21:43.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:43.288+0000 7fae43407680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-23T00:21:43.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:43.382+0000 7fae43407680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-23T00:21:43.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:43.472+0000 7fae43407680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-23T00:21:43.910 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:43.645+0000 7fae43407680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-23T00:21:44.062 INFO:teuthology.orchestra.run.smithi087.stderr:mgr not available, waiting (2/15)... 2024-02-23T00:21:44.175 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:43.908+0000 7fae43407680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-23T00:21:44.175 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:44.039+0000 7fae43407680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-23T00:21:44.175 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:44.173+0000 7fae43407680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:21:44.425 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:44.342+0000 7fae43407680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-23T00:21:44.832 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:44.423+0000 7fae43407680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-23T00:21:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:44.426301+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2024-02-23T00:21:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:44.428762+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00256019s) 2024-02-23T00:21:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.429565+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:21:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.429746+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:21:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.429910+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:21:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.430083+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:21:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.430296+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-23T00:21:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:44.443674+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2024-02-23T00:21:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.452126+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' 2024-02-23T00:21:44.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.457307+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:21:44.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.460015+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-23T00:21:44.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.462956+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' 2024-02-23T00:21:44.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:44.465245+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.87:0/3831807392' entity='mgr.y' 2024-02-23T00:21:46.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:45.432333+0000 mon.a (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.00613s) 2024-02-23T00:21:46.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.611+0000 7fa5c1ddc700 1 Processor -- start 2024-02-23T00:21:46.618 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.611+0000 7fa5c1ddc700 1 -- start start 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.611+0000 7fa5c1ddc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.611+0000 7fa5c1ddc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0ff210 con 0x7fa5bc070c40 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.611+0000 7fa5bb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.611+0000 7fa5bb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54026/0 (socket says 172.21.15.87:54026) 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.611+0000 7fa5bb7fe700 1 -- 172.21.15.87:0/1761681587 learned_addr learned my addr 172.21.15.87:0/1761681587 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.612+0000 7fa5bb7fe700 1 -- 172.21.15.87:0/1761681587 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0ff350 con 0x7fa5bc070c40 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.612+0000 7fa5bb7fe700 1 --2- 172.21.15.87:0/1761681587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fec90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa5a4009730 tx=0x7fa5a40090f0).ready entity=mon.0 client_cookie=f5818438be1d3171 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.612+0000 7fa5ba7fc700 1 -- 172.21.15.87:0/1761681587 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5a400b580 con 0x7fa5bc070c40 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.612+0000 7fa5ba7fc700 1 -- 172.21.15.87:0/1761681587 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5a400bb90 con 0x7fa5bc070c40 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.612+0000 7fa5ba7fc700 1 -- 172.21.15.87:0/1761681587 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5a4011c30 con 0x7fa5bc070c40 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.612+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/1761681587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 msgr2=0x7fa5bc0fec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.612+0000 7fa5c1ddc700 1 --2- 172.21.15.87:0/1761681587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fec90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa5a4009730 tx=0x7fa5a40090f0).stop 2024-02-23T00:21:46.619 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/1761681587 shutdown_connections 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 --2- 172.21.15.87:0/1761681587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fec90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/1761681587 >> 172.21.15.87:0/1761681587 conn(0x7fa5bc0f6670 msgr2=0x7fa5bc0f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/1761681587 shutdown_connections 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/1761681587 wait complete. 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 librados: starting msgr at 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 librados: starting objecter 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 Processor -- start 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 -- start start 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 librados: setting wanted keys 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.613+0000 7fa5c1ddc700 1 librados: calling monclient init 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5c1ddc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5c1ddc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0fab90 con 0x7fa5bc070c40 2024-02-23T00:21:46.620 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5bb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5bb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54028/0 (socket says 172.21.15.87:54028) 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5bb7fe700 1 -- 172.21.15.87:0/4044520222 learned_addr learned my addr 172.21.15.87:0/4044520222 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5bb7fe700 1 -- 172.21.15.87:0/4044520222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a4009370 con 0x7fa5bc070c40 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5bb7fe700 1 --2- 172.21.15.87:0/4044520222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fc710 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa5a4006940 tx=0x7fa5a40090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5b8ff9700 10 client.?.objecter ms_handle_connect 0x7fa5bc070c40 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5b8ff9700 1 -- 172.21.15.87:0/4044520222 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5a401a430 con 0x7fa5bc070c40 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5b8ff9700 1 -- 172.21.15.87:0/4044520222 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5a401aa40 con 0x7fa5bc070c40 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5b8ff9700 1 -- 172.21.15.87:0/4044520222 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5a4023b90 con 0x7fa5bc070c40 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5bc0fad30 con 0x7fa5bc070c40 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5c1ddc700 10 client.14110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc0fb1c0 con 0x7fa5bc070c40 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.614+0000 7fa5c1ddc700 1 librados: init done 2024-02-23T00:21:46.621 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.615+0000 7fa5b8ff9700 1 -- 172.21.15.87:0/4044520222 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 3) v1 ==== 38465+0+0 (secure 0 0 0) 0x7fa5a4023410 con 0x7fa5bc070c40 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5b8ff9700 1 --2- 172.21.15.87:0/4044520222 >> v2:172.21.15.87:6800/1685406414 conn(0x7fa5a8033e80 0x7fa5a8036340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc05a310 con 0x7fa5bc070c40 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5b8ff9700 1 -- 172.21.15.87:0/4044520222 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa5a40448c0 con 0x7fa5bc070c40 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5b8ff9700 10 client.14110.objecter ms_dispatch 0x7fa5bc001550 osd_map(1..1 src has 1..1) v4 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5b8ff9700 3 client.14110.objecter handle_osd_map got epochs [1,1] > 0 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5b8ff9700 3 client.14110.objecter handle_osd_map decoding full epoch 1 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5b8ff9700 20 client.14110.objecter dump_active .. 0 homeless 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5baffd700 1 --2- 172.21.15.87:0/4044520222 >> v2:172.21.15.87:6800/1685406414 conn(0x7fa5a8033e80 0x7fa5a8036340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5baffd700 1 --2- 172.21.15.87:0/4044520222 >> v2:172.21.15.87:6800/1685406414 conn(0x7fa5a8033e80 0x7fa5a8036340 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fa5ac009780 tx=0x7fa5ac006cc0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:46.622 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.616+0000 7fa5b8ff9700 10 client.14110.objecter ms_handle_connect 0x7fa5a8033e80 2024-02-23T00:21:46.623 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.620+0000 7fa5b8ff9700 1 -- 172.21.15.87:0/4044520222 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa5a401e020 con 0x7fa5bc070c40 2024-02-23T00:21:46.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.844+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa5bc046ce0 con 0x7fa5bc070c40 2024-02-23T00:21:46.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.845+0000 7fa5b8ff9700 1 -- 172.21.15.87:0/4044520222 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (secure 0 0 0) 0x7fa5a4019570 con 0x7fa5bc070c40 2024-02-23T00:21:46.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-23T00:21:46.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:21:46.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "health": { 2024-02-23T00:21:46.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-23T00:21:46.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "checks": {}, 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mutes": [] 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum": [ 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 0 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "a" 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "monmap": { 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-23T00:21:46.850 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:46.851 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osdmap": { 2024-02-23T00:21:46.851 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:46.851 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-23T00:21:46.851 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-23T00:21:46.851 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgmap": { 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-23T00:21:46.856 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "fsmap": { 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": true, 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modules": [ 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "iostat", 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "nfs", 2024-02-23T00:21:46.857 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "restful" 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ], 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "servicemap": { 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "modified": "2024-02-23T00:21:28.861877+0000", 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "services": {} 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: }, 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 10 librados: watch_flush enter 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 10 librados: watch_flush exit 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 20 client.14110.objecter shutdown clearing up homeless session... 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 10 client.14110.objecter successfully canceled tick 2024-02-23T00:21:46.858 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 >> v2:172.21.15.87:6800/1685406414 conn(0x7fa5a8033e80 msgr2=0x7fa5a8036340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 --2- 172.21.15.87:0/4044520222 >> v2:172.21.15.87:6800/1685406414 conn(0x7fa5a8033e80 0x7fa5a8036340 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fa5ac009780 tx=0x7fa5ac006cc0).stop 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 msgr2=0x7fa5bc0fc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 --2- 172.21.15.87:0/4044520222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fc710 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa5a4006940 tx=0x7fa5a40090f0).stop 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 shutdown_connections 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 --2- 172.21.15.87:0/4044520222 >> v2:172.21.15.87:6800/1685406414 conn(0x7fa5a8033e80 0x7fa5a8036340 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 --2- 172.21.15.87:0/4044520222 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa5bc070c40 0x7fa5bc0fc710 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 >> 172.21.15.87:0/4044520222 conn(0x7fa5bc0f6670 msgr2=0x7fa5bc0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 shutdown_connections 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 -- 172.21.15.87:0/4044520222 wait complete. 2024-02-23T00:21:46.859 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:46.848+0000 7fa5c1ddc700 1 librados: shutdown 2024-02-23T00:21:47.444 INFO:teuthology.orchestra.run.smithi087.stderr:mgr is available 2024-02-23T00:21:47.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:46.846284+0000 mon.a (mon.0) 23 : audit [DBG] from='client.? 172.21.15.87:0/4044520222' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.965+0000 7fece49d7700 1 Processor -- start 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.965+0000 7fece49d7700 1 -- start start 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.965+0000 7fece49d7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece00fec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.965+0000 7fece49d7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece00ff1d0 con 0x7fece0070c80 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.965+0000 7fecde59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece00fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.965+0000 7fecde59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece00fec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54038/0 (socket says 172.21.15.87:54038) 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.965+0000 7fecde59c700 1 -- 172.21.15.87:0/2694291670 learned_addr learned my addr 172.21.15.87:0/2694291670 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.965+0000 7fecde59c700 1 -- 172.21.15.87:0/2694291670 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece00ff310 con 0x7fece0070c80 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fecde59c700 1 --2- 172.21.15.87:0/2694291670 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece00fec50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fecc8009730 tx=0x7fecc80090f0).ready entity=mon.0 client_cookie=87606a887864fb5d server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:47.969 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fecdd59a700 1 -- 172.21.15.87:0/2694291670 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fecc800b580 con 0x7fece0070c80 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fecdd59a700 1 -- 172.21.15.87:0/2694291670 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecc800bb90 con 0x7fece0070c80 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fecdd59a700 1 -- 172.21.15.87:0/2694291670 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fecc8011c30 con 0x7fece0070c80 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fece49d7700 1 -- 172.21.15.87:0/2694291670 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 msgr2=0x7fece00fec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fece49d7700 1 --2- 172.21.15.87:0/2694291670 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece00fec50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fecc8009730 tx=0x7fecc80090f0).stop 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fece49d7700 1 -- 172.21.15.87:0/2694291670 shutdown_connections 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fece49d7700 1 --2- 172.21.15.87:0/2694291670 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece00fec50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fece49d7700 1 -- 172.21.15.87:0/2694291670 >> 172.21.15.87:0/2694291670 conn(0x7fece00f6670 msgr2=0x7fece00f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fece49d7700 1 -- 172.21.15.87:0/2694291670 shutdown_connections 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.966+0000 7fece49d7700 1 -- 172.21.15.87:0/2694291670 wait complete. 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.967+0000 7fece49d7700 1 librados: starting msgr at 2024-02-23T00:21:47.970 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.967+0000 7fece49d7700 1 librados: starting objecter 2024-02-23T00:21:47.972 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.967+0000 7fece49d7700 1 Processor -- start 2024-02-23T00:21:47.972 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.967+0000 7fece49d7700 1 -- start start 2024-02-23T00:21:47.972 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.967+0000 7fece49d7700 1 librados: setting wanted keys 2024-02-23T00:21:47.972 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.967+0000 7fece49d7700 1 librados: calling monclient init 2024-02-23T00:21:47.972 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fece49d7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece010ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:47.972 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fece49d7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece010b270 con 0x7fece0070c80 2024-02-23T00:21:47.972 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecde59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece010ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecde59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece010ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54040/0 (socket says 172.21.15.87:54040) 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecde59c700 1 -- 172.21.15.87:0/1435881332 learned_addr learned my addr 172.21.15.87:0/1435881332 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecde59c700 1 -- 172.21.15.87:0/1435881332 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8009370 con 0x7fece0070c80 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecde59c700 1 --2- 172.21.15.87:0/1435881332 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece010ad80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fecc8006940 tx=0x7fecc80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecd77fe700 10 client.?.objecter ms_handle_connect 0x7fece0070c80 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecd77fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecd77fe700 1 -- 172.21.15.87:0/1435881332 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fecc801a430 con 0x7fece0070c80 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecd77fe700 1 -- 172.21.15.87:0/1435881332 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecc801aa40 con 0x7fece0070c80 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fecd77fe700 1 -- 172.21.15.87:0/1435881332 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fecc8023b00 con 0x7fece0070c80 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece010b470 con 0x7fece0070c80 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fece49d7700 10 client.14112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece010c080 con 0x7fece0070c80 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.968+0000 7fece49d7700 1 librados: init done 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.969+0000 7fecd77fe700 1 -- 172.21.15.87:0/1435881332 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 4) v1 ==== 38571+0+0 (secure 0 0 0) 0x7fecc802c3f0 con 0x7fece0070c80 2024-02-23T00:21:47.973 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.969+0000 7fecd77fe700 1 --2- 172.21.15.87:0/1435881332 >> v2:172.21.15.87:6800/1685406414 conn(0x7feccc033f90 0x7feccc036450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:47.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.969+0000 7fecd77fe700 1 -- 172.21.15.87:0/1435881332 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fecc8045030 con 0x7fece0070c80 2024-02-23T00:21:47.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.969+0000 7fecd77fe700 10 client.14112.objecter ms_dispatch 0x7fece0001550 osd_map(1..1 src has 1..1) v4 2024-02-23T00:21:47.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.969+0000 7fecd77fe700 3 client.14112.objecter handle_osd_map got epochs [1,1] > 0 2024-02-23T00:21:47.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.969+0000 7fecd77fe700 3 client.14112.objecter handle_osd_map decoding full epoch 1 2024-02-23T00:21:47.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.969+0000 7fecd77fe700 20 client.14112.objecter dump_active .. 0 homeless 2024-02-23T00:21:47.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.969+0000 7fecddd9b700 1 --2- 172.21.15.87:0/1435881332 >> v2:172.21.15.87:6800/1685406414 conn(0x7feccc033f90 0x7feccc036450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:47.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.970+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece010b900 con 0x7fece0070c80 2024-02-23T00:21:47.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.970+0000 7fecddd9b700 1 --2- 172.21.15.87:0/1435881332 >> v2:172.21.15.87:6800/1685406414 conn(0x7feccc033f90 0x7feccc036450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fecd0009780 tx=0x7fecd0006cc0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:47.974 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.970+0000 7fecd77fe700 10 client.14112.objecter ms_handle_connect 0x7feccc033f90 2024-02-23T00:21:47.976 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:47.974+0000 7fecd77fe700 1 -- 172.21.15.87:0/1435881332 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fece010b900 con 0x7fece0070c80 2024-02-23T00:21:48.138 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.136+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fece010b900 con 0x7fece0070c80 2024-02-23T00:21:48.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.137+0000 7fecd77fe700 1 -- 172.21.15.87:0/1435881332 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+307 (secure 0 0 0) 0x7fece010b900 con 0x7fece0070c80 2024-02-23T00:21:48.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: [global] 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: fsid = 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: [mgr] 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: [osd] 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-23T00:21:48.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-23T00:21:48.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 10 librados: watch_flush enter 2024-02-23T00:21:48.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 10 librados: watch_flush exit 2024-02-23T00:21:48.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 20 client.14112.objecter shutdown clearing up homeless session... 2024-02-23T00:21:48.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 10 client.14112.objecter successfully canceled tick 2024-02-23T00:21:48.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 >> v2:172.21.15.87:6800/1685406414 conn(0x7feccc033f90 msgr2=0x7feccc036450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:48.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 --2- 172.21.15.87:0/1435881332 >> v2:172.21.15.87:6800/1685406414 conn(0x7feccc033f90 0x7feccc036450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fecd0009780 tx=0x7fecd0006cc0).stop 2024-02-23T00:21:48.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 msgr2=0x7fece010ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:48.143 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 --2- 172.21.15.87:0/1435881332 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece010ad80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fecc8006940 tx=0x7fecc80090f0).stop 2024-02-23T00:21:48.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 shutdown_connections 2024-02-23T00:21:48.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 --2- 172.21.15.87:0/1435881332 >> v2:172.21.15.87:6800/1685406414 conn(0x7feccc033f90 0x7feccc036450 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:48.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 --2- 172.21.15.87:0/1435881332 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fece0070c80 0x7fece010ad80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:48.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 >> 172.21.15.87:0/1435881332 conn(0x7fece00f6670 msgr2=0x7fece00f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:48.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 shutdown_connections 2024-02-23T00:21:48.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 -- 172.21.15.87:0/1435881332 wait complete. 2024-02-23T00:21:48.144 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:48.139+0000 7fece49d7700 1 librados: shutdown 2024-02-23T00:21:48.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:47.434848+0000 mon.a (mon.0) 24 : cluster [DBG] mgrmap e4: y(active, since 3s) 2024-02-23T00:21:48.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:48.138383+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.87:0/1435881332' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-23T00:21:50.080 INFO:teuthology.orchestra.run.smithi087.stderr:Enabling cephadm module... 2024-02-23T00:21:53.110 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:52.777126+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.87:0/4051787159' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-23T00:21:53.111 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ignoring --setuser ceph since I am not root 2024-02-23T00:21:53.111 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ignoring --setgroup ceph since I am not root 2024-02-23T00:21:53.111 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:53.029+0000 7f2fda8c2680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-23T00:21:53.111 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:53.109+0000 7f2fda8c2680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-23T00:21:53.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.563+0000 7f2c2b0f1700 1 Processor -- start 2024-02-23T00:21:53.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.563+0000 7f2c2b0f1700 1 -- start start 2024-02-23T00:21:53.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.563+0000 7f2c2b0f1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c240fda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:53.530 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.563+0000 7f2c2b0f1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c240fe010 con 0x7f2c240fd670 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.563+0000 7f2c28e8d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c240fda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.563+0000 7f2c28e8d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c240fda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44900/0 (socket says 172.21.15.87:44900) 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.563+0000 7f2c28e8d700 1 -- 172.21.15.87:0/2828122973 learned_addr learned my addr 172.21.15.87:0/2828122973 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.564+0000 7f2c28e8d700 1 -- 172.21.15.87:0/2828122973 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c240fe150 con 0x7f2c240fd670 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.564+0000 7f2c28e8d700 1 --2- 172.21.15.87:0/2828122973 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c240fda90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f2c18009730 tx=0x7f2c180090f0).ready entity=mon.0 client_cookie=f71860bf6691c573 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.564+0000 7f2c237fe700 1 -- 172.21.15.87:0/2828122973 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c1800b580 con 0x7f2c240fd670 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.564+0000 7f2c237fe700 1 -- 172.21.15.87:0/2828122973 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c1800bb90 con 0x7f2c240fd670 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.564+0000 7f2c237fe700 1 -- 172.21.15.87:0/2828122973 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c18011c30 con 0x7f2c240fd670 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.564+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/2828122973 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 msgr2=0x7f2c240fda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.564+0000 7f2c2b0f1700 1 --2- 172.21.15.87:0/2828122973 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c240fda90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f2c18009730 tx=0x7f2c180090f0).stop 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.564+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/2828122973 shutdown_connections 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.564+0000 7f2c2b0f1700 1 --2- 172.21.15.87:0/2828122973 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c240fda90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/2828122973 >> 172.21.15.87:0/2828122973 conn(0x7f2c240f5df0 msgr2=0x7f2c240f8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:53.531 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/2828122973 shutdown_connections 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/2828122973 wait complete. 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 librados: starting msgr at 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 librados: starting objecter 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 Processor -- start 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 -- start start 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 librados: setting wanted keys 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 librados: calling monclient init 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c2410ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c2b0f1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2410b240 con 0x7f2c240fd670 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c28e8d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c2410ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c28e8d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c2410ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44908/0 (socket says 172.21.15.87:44908) 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.565+0000 7f2c28e8d700 1 -- 172.21.15.87:0/4051787159 learned_addr learned my addr 172.21.15.87:0/4051787159 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c28e8d700 1 -- 172.21.15.87:0/4051787159 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c18009370 con 0x7f2c240fd670 2024-02-23T00:21:53.532 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c28e8d700 1 --2- 172.21.15.87:0/4051787159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c2410ad50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f2c18010600 tx=0x7f2c180090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c21ffb700 10 client.?.objecter ms_handle_connect 0x7f2c240fd670 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c21ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c21ffb700 1 -- 172.21.15.87:0/4051787159 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c18011690 con 0x7f2c240fd670 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c21ffb700 1 -- 172.21.15.87:0/4051787159 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c18011a70 con 0x7f2c240fd670 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c21ffb700 1 -- 172.21.15.87:0/4051787159 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c1801ab80 con 0x7f2c240fd670 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2410b440 con 0x7f2c240fd670 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c2b0f1700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2410bff0 con 0x7f2c240fd670 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.566+0000 7f2c2b0f1700 1 librados: init done 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c21ffb700 1 -- 172.21.15.87:0/4051787159 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 4) v1 ==== 38571+0+0 (secure 0 0 0) 0x7f2c18010b50 con 0x7f2c240fd670 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c21ffb700 1 --2- 172.21.15.87:0/4051787159 >> v2:172.21.15.87:6800/1685406414 conn(0x7f2c14034030 0x7f2c140364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2405a310 con 0x7f2c240fd670 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c21ffb700 1 -- 172.21.15.87:0/4051787159 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2c18044830 con 0x7f2c240fd670 2024-02-23T00:21:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c21ffb700 10 client.14114.objecter ms_dispatch 0x7f2c24001550 osd_map(1..1 src has 1..1) v4 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c21ffb700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c21ffb700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c21ffb700 20 client.14114.objecter dump_active .. 0 homeless 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c23fff700 1 --2- 172.21.15.87:0/4051787159 >> v2:172.21.15.87:6800/1685406414 conn(0x7f2c14034030 0x7f2c140364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c21ffb700 10 client.14114.objecter ms_handle_connect 0x7f2c14034030 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.567+0000 7f2c23fff700 1 --2- 172.21.15.87:0/4051787159 >> v2:172.21.15.87:6800/1685406414 conn(0x7f2c14034030 0x7f2c140364f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2c10009780 tx=0x7f2c10006cc0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.572+0000 7f2c21ffb700 1 -- 172.21.15.87:0/4051787159 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2c1801e080 con 0x7f2c240fd670 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.776+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2c240fac30 con 0x7f2c240fd670 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.828+0000 7f2c21ffb700 1 -- 172.21.15.87:0/4051787159 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 5) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f2c18043ac0 con 0x7f2c240fd670 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.829+0000 7f2c21ffb700 1 -- 172.21.15.87:0/4051787159 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f2c18018b40 con 0x7f2c240fd670 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 10 librados: watch_flush enter 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 10 librados: watch_flush exit 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 20 client.14114.objecter shutdown clearing up homeless session... 2024-02-23T00:21:53.534 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 10 client.14114.objecter successfully canceled tick 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 >> v2:172.21.15.87:6800/1685406414 conn(0x7f2c14034030 msgr2=0x7f2c140364f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 1 --2- 172.21.15.87:0/4051787159 >> v2:172.21.15.87:6800/1685406414 conn(0x7f2c14034030 0x7f2c140364f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2c10009780 tx=0x7f2c10006cc0).stop 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 msgr2=0x7f2c2410ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 1 --2- 172.21.15.87:0/4051787159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c2410ad50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f2c18010600 tx=0x7f2c180090f0).stop 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 shutdown_connections 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 1 --2- 172.21.15.87:0/4051787159 >> v2:172.21.15.87:6800/1685406414 conn(0x7f2c14034030 0x7f2c140364f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 1 --2- 172.21.15.87:0/4051787159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2c240fd670 0x7f2c2410ad50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.831+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 >> 172.21.15.87:0/4051787159 conn(0x7f2c240f5df0 msgr2=0x7f2c240f75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.832+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 shutdown_connections 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.832+0000 7f2c2b0f1700 1 -- 172.21.15.87:0/4051787159 wait complete. 2024-02-23T00:21:53.535 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:52.832+0000 7f2c2b0f1700 1 librados: shutdown 2024-02-23T00:21:53.832 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:53.471+0000 7f2fda8c2680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-23T00:21:54.220 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:52.830193+0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.87:0/4051787159' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-23T00:21:54.220 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:52.830247+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 8s) 2024-02-23T00:21:54.510 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:54.218+0000 7f2fda8c2680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-23T00:21:54.510 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:54.300+0000 7f2fda8c2680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-23T00:21:55.736 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:54.509+0000 7f2fda8c2680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-23T00:21:55.737 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:55.437+0000 7f2fda8c2680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-23T00:21:55.737 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:55.567+0000 7f2fda8c2680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:21:55.737 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:55.646+0000 7f2fda8c2680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-23T00:21:55.737 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:55.735+0000 7f2fda8c2680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-23T00:21:56.082 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:55.930+0000 7f2fda8c2680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-23T00:21:56.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:56.207+0000 7f2fda8c2680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-23T00:21:56.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:56.303+0000 7f2fda8c2680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-23T00:21:57.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:57.018+0000 7f2fda8c2680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-23T00:21:57.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:57.098+0000 7f2fda8c2680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-23T00:21:57.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:57.188+0000 7f2fda8c2680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-23T00:21:57.615 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:57.358+0000 7f2fda8c2680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-23T00:21:57.866 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:57.614+0000 7f2fda8c2680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-23T00:21:57.866 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:57.745+0000 7f2fda8c2680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-23T00:21:58.134 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:57.881+0000 7f2fda8c2680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:21:58.134 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:58.054+0000 7f2fda8c2680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-23T00:21:58.459 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.453+0000 7f68992cd700 1 Processor -- start 2024-02-23T00:21:58.459 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.453+0000 7f68992cd700 1 -- start start 2024-02-23T00:21:58.459 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.453+0000 7f68992cd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f68940fc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.453+0000 7f68992cd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68940fcf40 con 0x7f68940fc5a0 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.453+0000 7f6892ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f68940fc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.453+0000 7f6892ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f68940fc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44982/0 (socket says 172.21.15.87:44982) 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.453+0000 7f6892ffd700 1 -- 172.21.15.87:0/2919249843 learned_addr learned my addr 172.21.15.87:0/2919249843 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.453+0000 7f6892ffd700 1 -- 172.21.15.87:0/2919249843 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68940fd080 con 0x7f68940fc5a0 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f6892ffd700 1 --2- 172.21.15.87:0/2919249843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f68940fc9c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6884009730 tx=0x7f68840090f0).ready entity=mon.0 client_cookie=4a7dd259eb042f06 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f6891ffb700 1 -- 172.21.15.87:0/2919249843 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f688400b580 con 0x7f68940fc5a0 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f6891ffb700 1 -- 172.21.15.87:0/2919249843 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f688400bb90 con 0x7f68940fc5a0 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f6891ffb700 1 -- 172.21.15.87:0/2919249843 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6884011c30 con 0x7f68940fc5a0 2024-02-23T00:21:58.460 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f68992cd700 1 -- 172.21.15.87:0/2919249843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 msgr2=0x7f68940fc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f68992cd700 1 --2- 172.21.15.87:0/2919249843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f68940fc9c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6884009730 tx=0x7f68840090f0).stop 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f68992cd700 1 -- 172.21.15.87:0/2919249843 shutdown_connections 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f68992cd700 1 --2- 172.21.15.87:0/2919249843 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f68940fc9c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f68992cd700 1 -- 172.21.15.87:0/2919249843 >> 172.21.15.87:0/2919249843 conn(0x7f68940f5e30 msgr2=0x7f68940f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f68992cd700 1 -- 172.21.15.87:0/2919249843 shutdown_connections 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.454+0000 7f68992cd700 1 -- 172.21.15.87:0/2919249843 wait complete. 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f68992cd700 1 librados: starting msgr at 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f68992cd700 1 librados: starting objecter 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f68992cd700 1 Processor -- start 2024-02-23T00:21:58.461 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f68992cd700 1 -- start start 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f68992cd700 1 librados: setting wanted keys 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f68992cd700 1 librados: calling monclient init 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f68992cd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f689410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f68992cd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689410d3b0 con 0x7f68940fc5a0 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f6892ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f689410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f6892ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f689410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:44996/0 (socket says 172.21.15.87:44996) 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.455+0000 7f6892ffd700 1 -- 172.21.15.87:0/307031305 learned_addr learned my addr 172.21.15.87:0/307031305 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f6892ffd700 1 -- 172.21.15.87:0/307031305 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6884009370 con 0x7f68940fc5a0 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f6892ffd700 1 --2- 172.21.15.87:0/307031305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f689410cec0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f68840105d0 tx=0x7f68840090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f687bfff700 10 client.?.objecter ms_handle_connect 0x7f68940fc5a0 2024-02-23T00:21:58.462 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f687bfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f687bfff700 1 -- 172.21.15.87:0/307031305 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6884011690 con 0x7f68940fc5a0 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f687bfff700 1 -- 172.21.15.87:0/307031305 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6884011a70 con 0x7f68940fc5a0 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f68992cd700 1 -- 172.21.15.87:0/307031305 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689410d5b0 con 0x7f68940fc5a0 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f687bfff700 1 -- 172.21.15.87:0/307031305 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f688401ab70 con 0x7f68940fc5a0 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f68992cd700 10 client.14122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f68992cd700 1 -- 172.21.15.87:0/307031305 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689410da40 con 0x7f68940fc5a0 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.456+0000 7f68992cd700 1 librados: init done 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.457+0000 7f687bfff700 1 -- 172.21.15.87:0/307031305 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 6) v1 ==== 38441+0+0 (secure 0 0 0) 0x7f68840223b0 con 0x7f68940fc5a0 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.457+0000 7f687bfff700 1 -- 172.21.15.87:0/307031305 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f6884044620 con 0x7f68940fc5a0 2024-02-23T00:21:58.463 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.457+0000 7f687bfff700 10 client.14122.objecter ms_dispatch 0x7f6894001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:21:58.464 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.457+0000 7f687bfff700 3 client.14122.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:21:58.464 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.457+0000 7f687bfff700 3 client.14122.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:21:58.464 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.457+0000 7f68992cd700 1 -- 172.21.15.87:0/307031305 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6894046ce0 con 0x7f68940fc5a0 2024-02-23T00:21:58.464 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.457+0000 7f687bfff700 20 client.14122.objecter dump_active .. 0 homeless 2024-02-23T00:21:58.464 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.462+0000 7f687bfff700 1 -- 172.21.15.87:0/307031305 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f688401e070 con 0x7f68940fc5a0 2024-02-23T00:21:58.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:21:58.143+0000 7f2fda8c2680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:58.144557+0000 mon.a (mon.0) 29 : cluster [INF] Active manager daemon y restarted 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:58.144764+0000 mon.a (mon.0) 30 : cluster [INF] Activating manager daemon y 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:58.147422+0000 mon.a (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:58.147515+0000 mon.a (mon.0) 32 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00282715s) 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.148535+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.149064+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.149640+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.149780+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.149914+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:58.163605+0000 mon.a (mon.0) 38 : cluster [INF] Manager daemon y is now available 2024-02-23T00:21:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.176455+0000 mon.a (mon.0) 39 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:21:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.664+0000 7f68992cd700 1 -- 172.21.15.87:0/307031305 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f6894106ba0 con 0x7f68940fc5a0 2024-02-23T00:21:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.665+0000 7f687bfff700 1 -- 172.21.15.87:0/307031305 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v6) v1 ==== 56+0+89 (secure 0 0 0) 0x7f6884011e60 con 0x7f68940fc5a0 2024-02-23T00:21:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-23T00:21:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 6, 2024-02-23T00:21:58.667 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": false, 2024-02-23T00:21:58.667 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-23T00:21:58.667 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-23T00:21:58.667 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-23T00:21:58.668 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 10 librados: watch_flush enter 2024-02-23T00:21:58.668 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 10 librados: watch_flush exit 2024-02-23T00:21:58.668 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 20 client.14122.objecter shutdown clearing up homeless session... 2024-02-23T00:21:58.668 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 10 client.14122.objecter successfully canceled tick 2024-02-23T00:21:58.668 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 1 -- 172.21.15.87:0/307031305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 msgr2=0x7f689410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:58.669 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 1 --2- 172.21.15.87:0/307031305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f689410cec0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f68840105d0 tx=0x7f68840090f0).stop 2024-02-23T00:21:58.669 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 1 -- 172.21.15.87:0/307031305 shutdown_connections 2024-02-23T00:21:58.669 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 1 --2- 172.21.15.87:0/307031305 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68940fc5a0 0x7f689410cec0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:58.669 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 1 -- 172.21.15.87:0/307031305 >> 172.21.15.87:0/307031305 conn(0x7f68940f5e30 msgr2=0x7f68940f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:58.669 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 1 -- 172.21.15.87:0/307031305 shutdown_connections 2024-02-23T00:21:58.669 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.666+0000 7f68992cd700 1 -- 172.21.15.87:0/307031305 wait complete. 2024-02-23T00:21:58.669 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:58.667+0000 7f68992cd700 1 librados: shutdown 2024-02-23T00:21:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for the mgr to restart... 2024-02-23T00:21:59.321 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mgr epoch 6... 2024-02-23T00:21:59.323 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:21:58.173826+0000 mgr.y (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-02-23T00:21:59.323 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.263237+0000 mon.a (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:21:59.323 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.264270+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:21:59.323 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.265405+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:21:59.323 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.267020+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:21:59.323 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.280929+0000 mon.a (mon.0) 44 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:21:59.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.283631+0000 mon.a (mon.0) 45 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-23T00:21:59.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:58.665344+0000 mon.a (mon.0) 46 : audit [DBG] from='client.? 172.21.15.87:0/307031305' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-23T00:21:59.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:21:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:21:59.149547+0000 mon.a (mon.0) 47 : cluster [DBG] mgrmap e7: y(active, since 1.00485s) 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.872+0000 7f66e1d6e700 1 Processor -- start 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.872+0000 7f66e1d6e700 1 -- start start 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.872+0000 7f66e1d6e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc0fb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.872+0000 7f66e1d6e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc0fb8f0 con 0x7f66dc0faf50 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.872+0000 7f66db7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc0fb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.872+0000 7f66db7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc0fb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45000/0 (socket says 172.21.15.87:45000) 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.872+0000 7f66db7fe700 1 -- 172.21.15.87:0/2784848327 learned_addr learned my addr 172.21.15.87:0/2784848327 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66db7fe700 1 -- 172.21.15.87:0/2784848327 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc0fba30 con 0x7f66dc0faf50 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66db7fe700 1 --2- 172.21.15.87:0/2784848327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc0fb370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f66d0009730 tx=0x7f66d00090f0).ready entity=mon.0 client_cookie=c48e4b4321a86479 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66da7fc700 1 -- 172.21.15.87:0/2784848327 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66d000b580 con 0x7f66dc0faf50 2024-02-23T00:21:59.879 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66da7fc700 1 -- 172.21.15.87:0/2784848327 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66d000bb90 con 0x7f66dc0faf50 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66da7fc700 1 -- 172.21.15.87:0/2784848327 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66d0011c30 con 0x7f66dc0faf50 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66e1d6e700 1 -- 172.21.15.87:0/2784848327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 msgr2=0x7f66dc0fb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66e1d6e700 1 --2- 172.21.15.87:0/2784848327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc0fb370 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f66d0009730 tx=0x7f66d00090f0).stop 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66e1d6e700 1 -- 172.21.15.87:0/2784848327 shutdown_connections 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66e1d6e700 1 --2- 172.21.15.87:0/2784848327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc0fb370 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.873+0000 7f66e1d6e700 1 -- 172.21.15.87:0/2784848327 >> 172.21.15.87:0/2784848327 conn(0x7f66dc0f6670 msgr2=0x7f66dc0f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.874+0000 7f66e1d6e700 1 -- 172.21.15.87:0/2784848327 shutdown_connections 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.874+0000 7f66e1d6e700 1 -- 172.21.15.87:0/2784848327 wait complete. 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.874+0000 7f66e1d6e700 1 librados: starting msgr at 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.874+0000 7f66e1d6e700 1 librados: starting objecter 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66e1d6e700 1 Processor -- start 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66e1d6e700 1 -- start start 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66e1d6e700 1 librados: setting wanted keys 2024-02-23T00:21:59.880 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66e1d6e700 1 librados: calling monclient init 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66e1d6e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc10da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66e1d6e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc070c40 con 0x7f66dc0faf50 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66db7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc10da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66db7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc10da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45004/0 (socket says 172.21.15.87:45004) 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66db7fe700 1 -- 172.21.15.87:0/686916424 learned_addr learned my addr 172.21.15.87:0/686916424 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.875+0000 7f66db7fe700 1 -- 172.21.15.87:0/686916424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d0009370 con 0x7f66dc0faf50 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66db7fe700 1 --2- 172.21.15.87:0/686916424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc10da20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f66d0000c00 tx=0x7f66d00090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66d8ff9700 10 client.?.objecter ms_handle_connect 0x7f66dc0faf50 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66d8ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66d8ff9700 1 -- 172.21.15.87:0/686916424 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66d000b9a0 con 0x7f66dc0faf50 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66d8ff9700 1 -- 172.21.15.87:0/686916424 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66d001aaf0 con 0x7f66dc0faf50 2024-02-23T00:21:59.881 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66e1d6e700 1 -- 172.21.15.87:0/686916424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc070e40 con 0x7f66dc0faf50 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66d8ff9700 1 -- 172.21.15.87:0/686916424 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66d0023ac0 con 0x7f66dc0faf50 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66e1d6e700 10 client.14124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66e1d6e700 1 -- 172.21.15.87:0/686916424 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc10e0d0 con 0x7f66dc0faf50 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.876+0000 7f66e1d6e700 1 librados: init done 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66d8ff9700 1 -- 172.21.15.87:0/686916424 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 7) v1 ==== 38476+0+0 (secure 0 0 0) 0x7f66d002c3f0 con 0x7f66dc0faf50 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66d8ff9700 1 --2- 172.21.15.87:0/686916424 >> v2:172.21.15.87:6800/2138166229 conn(0x7f66c4033eb0 0x7f66c4036370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66d8ff9700 1 -- 172.21.15.87:0/686916424 --> v2:172.21.15.87:6800/2138166229 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66c40369f0 con 0x7f66c4033eb0 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66d8ff9700 1 -- 172.21.15.87:0/686916424 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f66d00446d0 con 0x7f66dc0faf50 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66d8ff9700 10 client.14124.objecter ms_dispatch 0x7f66dc001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66d8ff9700 3 client.14124.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66d8ff9700 3 client.14124.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66d8ff9700 20 client.14124.objecter dump_active .. 0 homeless 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66daffd700 1 --2- 172.21.15.87:0/686916424 >> v2:172.21.15.87:6800/2138166229 conn(0x7f66c4033eb0 0x7f66c4036370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66d8ff9700 10 client.14124.objecter ms_handle_connect 0x7f66c4033eb0 2024-02-23T00:21:59.882 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.877+0000 7f66daffd700 1 --2- 172.21.15.87:0/686916424 >> v2:172.21.15.87:6800/2138166229 conn(0x7f66c4033eb0 0x7f66c4036370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f66cc009780 tx=0x7f66cc006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:21:59.883 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.878+0000 7f66d8ff9700 1 -- 172.21.15.87:0/686916424 <== mgr.14116 v2:172.21.15.87:6800/2138166229 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f66c40369f0 con 0x7f66c4033eb0 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.883+0000 7f66e1d6e700 1 -- 172.21.15.87:0/686916424 --> v2:172.21.15.87:6800/2138166229 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f66dc05a310 con 0x7f66c4033eb0 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.883+0000 7f66d8ff9700 1 -- 172.21.15.87:0/686916424 <== mgr.14116 v2:172.21.15.87:6800/2138166229 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f66dc05a310 con 0x7f66c4033eb0 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "initialized": true 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 10 librados: watch_flush enter 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 10 librados: watch_flush exit 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 20 client.14124.objecter shutdown clearing up homeless session... 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 10 client.14124.objecter successfully canceled tick 2024-02-23T00:21:59.886 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 -- 172.21.15.87:0/686916424 >> v2:172.21.15.87:6800/2138166229 conn(0x7f66c4033eb0 msgr2=0x7f66c4036370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 --2- 172.21.15.87:0/686916424 >> v2:172.21.15.87:6800/2138166229 conn(0x7f66c4033eb0 0x7f66c4036370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f66cc009780 tx=0x7f66cc006cc0).stop 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 -- 172.21.15.87:0/686916424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 msgr2=0x7f66dc10da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 --2- 172.21.15.87:0/686916424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc10da20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f66d0000c00 tx=0x7f66d00090f0).stop 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 -- 172.21.15.87:0/686916424 shutdown_connections 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 --2- 172.21.15.87:0/686916424 >> v2:172.21.15.87:6800/2138166229 conn(0x7f66c4033eb0 0x7f66c4036370 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 --2- 172.21.15.87:0/686916424 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66dc0faf50 0x7f66dc10da20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 -- 172.21.15.87:0/686916424 >> 172.21.15.87:0/686916424 conn(0x7f66dc0f6670 msgr2=0x7f66dc0f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 -- 172.21.15.87:0/686916424 shutdown_connections 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 -- 172.21.15.87:0/686916424 wait complete. 2024-02-23T00:21:59.887 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:21:59.884+0000 7f66e1d6e700 1 librados: shutdown 2024-02-23T00:22:00.511 INFO:teuthology.orchestra.run.smithi087.stderr:mgr epoch 6 is available 2024-02-23T00:22:00.511 INFO:teuthology.orchestra.run.smithi087.stderr:Setting orchestrator backend to cephadm... 2024-02-23T00:22:00.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:59.878273+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-23T00:22:00.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:21:59.884104+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-23T00:22:01.100 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.071+0000 7f71a1cdc700 1 Processor -- start 2024-02-23T00:22:01.100 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.071+0000 7f71a1cdc700 1 -- start start 2024-02-23T00:22:01.100 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f71a1cdc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c0fb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:01.100 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f71a1cdc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c0fb8e0 con 0x7f719c0faf40 2024-02-23T00:22:01.100 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f719b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c0fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:01.100 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f719b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c0fb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45020/0 (socket says 172.21.15.87:45020) 2024-02-23T00:22:01.100 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f719b7fe700 1 -- 172.21.15.87:0/315217755 learned_addr learned my addr 172.21.15.87:0/315217755 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f719b7fe700 1 -- 172.21.15.87:0/315217755 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c0fba20 con 0x7f719c0faf40 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f719b7fe700 1 --2- 172.21.15.87:0/315217755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c0fb360 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f7190009730 tx=0x7f71900090f0).ready entity=mon.0 client_cookie=a6ec69c03a7689e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f719a7fc700 1 -- 172.21.15.87:0/315217755 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f719000b580 con 0x7f719c0faf40 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f719a7fc700 1 -- 172.21.15.87:0/315217755 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f719000bb90 con 0x7f719c0faf40 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.072+0000 7f719a7fc700 1 -- 172.21.15.87:0/315217755 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7190011c30 con 0x7f719c0faf40 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.073+0000 7f71a1cdc700 1 -- 172.21.15.87:0/315217755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 msgr2=0x7f719c0fb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.073+0000 7f71a1cdc700 1 --2- 172.21.15.87:0/315217755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c0fb360 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f7190009730 tx=0x7f71900090f0).stop 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.073+0000 7f71a1cdc700 1 -- 172.21.15.87:0/315217755 shutdown_connections 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.073+0000 7f71a1cdc700 1 --2- 172.21.15.87:0/315217755 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c0fb360 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.073+0000 7f71a1cdc700 1 -- 172.21.15.87:0/315217755 >> 172.21.15.87:0/315217755 conn(0x7f719c0f6660 msgr2=0x7f719c0f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.073+0000 7f71a1cdc700 1 -- 172.21.15.87:0/315217755 shutdown_connections 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.073+0000 7f71a1cdc700 1 -- 172.21.15.87:0/315217755 wait complete. 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.073+0000 7f71a1cdc700 1 librados: starting msgr at 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.073+0000 7f71a1cdc700 1 librados: starting objecter 2024-02-23T00:22:01.101 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f71a1cdc700 1 Processor -- start 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f71a1cdc700 1 -- start start 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f71a1cdc700 1 librados: setting wanted keys 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f71a1cdc700 1 librados: calling monclient init 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f71a1cdc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c109bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f71a1cdc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c10a0a0 con 0x7f719c0faf40 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f719b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c109bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f719b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c109bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45026/0 (socket says 172.21.15.87:45026) 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f719b7fe700 1 -- 172.21.15.87:0/3233038483 learned_addr learned my addr 172.21.15.87:0/3233038483 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.074+0000 7f719b7fe700 1 -- 172.21.15.87:0/3233038483 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7190009370 con 0x7f719c0faf40 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f719b7fe700 1 --2- 172.21.15.87:0/3233038483 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c109bb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7190000c00 tx=0x7f71900090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f7198ff9700 10 client.?.objecter ms_handle_connect 0x7f719c0faf40 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f7198ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f7198ff9700 1 -- 172.21.15.87:0/3233038483 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f719000b9a0 con 0x7f719c0faf40 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f7198ff9700 1 -- 172.21.15.87:0/3233038483 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f719001aaf0 con 0x7f719c0faf40 2024-02-23T00:22:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719c10a2a0 con 0x7f719c0faf40 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f7198ff9700 1 -- 172.21.15.87:0/3233038483 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7190023ac0 con 0x7f719c0faf40 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f71a1cdc700 10 client.14126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c10a730 con 0x7f719c0faf40 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.075+0000 7f71a1cdc700 1 librados: init done 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f7198ff9700 1 -- 172.21.15.87:0/3233038483 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f719002c3f0 con 0x7f719c0faf40 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719c046ce0 con 0x7f719c0faf40 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f7198ff9700 1 --2- 172.21.15.87:0/3233038483 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7184033fd0 0x7f7184036490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f7198ff9700 1 -- 172.21.15.87:0/3233038483 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f7190044bf0 con 0x7f719c0faf40 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f7198ff9700 10 client.14126.objecter ms_dispatch 0x7f719c001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f7198ff9700 3 client.14126.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f7198ff9700 3 client.14126.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f7198ff9700 20 client.14126.objecter dump_active .. 0 homeless 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f719affd700 1 --2- 172.21.15.87:0/3233038483 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7184033fd0 0x7f7184036490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:01.103 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f7198ff9700 10 client.14126.objecter ms_handle_connect 0x7f7184033fd0 2024-02-23T00:22:01.104 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.076+0000 7f719affd700 1 --2- 172.21.15.87:0/3233038483 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7184033fd0 0x7f7184036490 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f718c009780 tx=0x7f718c006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:01.104 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.080+0000 7f7198ff9700 1 -- 172.21.15.87:0/3233038483 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7190021320 con 0x7f719c0faf40 2024-02-23T00:22:01.262 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.260+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 --> v2:172.21.15.87:6800/2138166229 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f719c103900 con 0x7f7184033fd0 2024-02-23T00:22:01.266 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.265+0000 7f7198ff9700 1 -- 172.21.15.87:0/3233038483 <== mgr.14116 v2:172.21.15.87:6800/2138166229 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f719c103900 con 0x7f7184033fd0 2024-02-23T00:22:01.268 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 10 librados: watch_flush enter 2024-02-23T00:22:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 10 librados: watch_flush exit 2024-02-23T00:22:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 20 client.14126.objecter shutdown clearing up homeless session... 2024-02-23T00:22:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 10 client.14126.objecter successfully canceled tick 2024-02-23T00:22:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7184033fd0 msgr2=0x7f7184036490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 --2- 172.21.15.87:0/3233038483 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7184033fd0 0x7f7184036490 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f718c009780 tx=0x7f718c006cc0).stop 2024-02-23T00:22:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 msgr2=0x7f719c109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 --2- 172.21.15.87:0/3233038483 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c109bb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f7190000c00 tx=0x7f71900090f0).stop 2024-02-23T00:22:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 shutdown_connections 2024-02-23T00:22:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 --2- 172.21.15.87:0/3233038483 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7184033fd0 0x7f7184036490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 --2- 172.21.15.87:0/3233038483 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f719c0faf40 0x7f719c109bb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 >> 172.21.15.87:0/3233038483 conn(0x7f719c0f6660 msgr2=0x7f719c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 shutdown_connections 2024-02-23T00:22:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.267+0000 7f71a1cdc700 1 -- 172.21.15.87:0/3233038483 wait complete. 2024-02-23T00:22:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:01.268+0000 7f71a1cdc700 1 librados: shutdown 2024-02-23T00:22:01.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:00.265389+0000 mon.a (mon.0) 48 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-02-23T00:22:01.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:01.265639+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:22:02.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.460+0000 7fe466d18700 1 Processor -- start 2024-02-23T00:22:02.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.460+0000 7fe466d18700 1 -- start start 2024-02-23T00:22:02.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.460+0000 7fe466d18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe4600fec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:02.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.460+0000 7fe466d18700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4600ff1d0 con 0x7fe460070c80 2024-02-23T00:22:02.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.460+0000 7fe464ab4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe4600fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:02.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.460+0000 7fe464ab4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe4600fec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55612/0 (socket says 172.21.15.87:55612) 2024-02-23T00:22:02.466 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.460+0000 7fe464ab4700 1 -- 172.21.15.87:0/1182829282 learned_addr learned my addr 172.21.15.87:0/1182829282 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.461+0000 7fe464ab4700 1 -- 172.21.15.87:0/1182829282 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4600ff310 con 0x7fe460070c80 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.461+0000 7fe464ab4700 1 --2- 172.21.15.87:0/1182829282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe4600fec50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe454009730 tx=0x7fe4540090f0).ready entity=mon.0 client_cookie=8290acb6c635a1b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.461+0000 7fe45f7fe700 1 -- 172.21.15.87:0/1182829282 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe45400b580 con 0x7fe460070c80 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.461+0000 7fe45f7fe700 1 -- 172.21.15.87:0/1182829282 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe45400bb90 con 0x7fe460070c80 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.461+0000 7fe45f7fe700 1 -- 172.21.15.87:0/1182829282 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe454011c30 con 0x7fe460070c80 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 -- 172.21.15.87:0/1182829282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 msgr2=0x7fe4600fec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 --2- 172.21.15.87:0/1182829282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe4600fec50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe454009730 tx=0x7fe4540090f0).stop 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 -- 172.21.15.87:0/1182829282 shutdown_connections 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 --2- 172.21.15.87:0/1182829282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe4600fec50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 -- 172.21.15.87:0/1182829282 >> 172.21.15.87:0/1182829282 conn(0x7fe4600f6670 msgr2=0x7fe4600f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 -- 172.21.15.87:0/1182829282 shutdown_connections 2024-02-23T00:22:02.467 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 -- 172.21.15.87:0/1182829282 wait complete. 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 librados: starting msgr at 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 librados: starting objecter 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 Processor -- start 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 -- start start 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 librados: setting wanted keys 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 librados: calling monclient init 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe46010ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.462+0000 7fe466d18700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46010b270 con 0x7fe460070c80 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe464ab4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe46010ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe464ab4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe46010ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55624/0 (socket says 172.21.15.87:55624) 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe464ab4700 1 -- 172.21.15.87:0/2515297347 learned_addr learned my addr 172.21.15.87:0/2515297347 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe464ab4700 1 -- 172.21.15.87:0/2515297347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe454009370 con 0x7fe460070c80 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe464ab4700 1 --2- 172.21.15.87:0/2515297347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe46010ad80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe454006940 tx=0x7fe4540090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe45dffb700 10 client.?.objecter ms_handle_connect 0x7fe460070c80 2024-02-23T00:22:02.468 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe45dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe45dffb700 1 -- 172.21.15.87:0/2515297347 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe45401a430 con 0x7fe460070c80 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe45dffb700 1 -- 172.21.15.87:0/2515297347 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe45401aa40 con 0x7fe460070c80 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe45dffb700 1 -- 172.21.15.87:0/2515297347 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe454023b90 con 0x7fe460070c80 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46010b470 con 0x7fe460070c80 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe466d18700 10 client.14128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46010c080 con 0x7fe460070c80 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.463+0000 7fe466d18700 1 librados: init done 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.464+0000 7fe45dffb700 1 -- 172.21.15.87:0/2515297347 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7fe454023410 con 0x7fe460070c80 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.464+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46010b900 con 0x7fe460070c80 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.464+0000 7fe45dffb700 1 --2- 172.21.15.87:0/2515297347 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe450034030 0x7fe4500364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.464+0000 7fe45dffb700 1 -- 172.21.15.87:0/2515297347 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fe454044740 con 0x7fe460070c80 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.464+0000 7fe45dffb700 10 client.14128.objecter ms_dispatch 0x7fe460001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.464+0000 7fe45dffb700 3 client.14128.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:02.469 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.464+0000 7fe45dffb700 3 client.14128.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.464+0000 7fe45dffb700 20 client.14128.objecter dump_active .. 0 homeless 2024-02-23T00:22:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.464+0000 7fe45ffff700 1 --2- 172.21.15.87:0/2515297347 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe450034030 0x7fe4500364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.465+0000 7fe45dffb700 10 client.14128.objecter ms_handle_connect 0x7fe450034030 2024-02-23T00:22:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.465+0000 7fe45ffff700 1 --2- 172.21.15.87:0/2515297347 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe450034030 0x7fe4500364f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe44c009780 tx=0x7fe44c006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:02.470 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.469+0000 7fe45dffb700 1 -- 172.21.15.87:0/2515297347 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe46010b900 con 0x7fe460070c80 2024-02-23T00:22:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:01.261614+0000 mgr.y (mgr.14116) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:01.290642+0000 mon.a (mon.0) 50 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:02.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.633+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 --> v2:172.21.15.87:6800/2138166229 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fe46010b900 con 0x7fe450034030 2024-02-23T00:22:02.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.634+0000 7fe45dffb700 1 -- 172.21.15.87:0/2515297347 <== mgr.14116 v2:172.21.15.87:6800/2138166229 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fe46010b900 con 0x7fe450034030 2024-02-23T00:22:02.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: value unchanged 2024-02-23T00:22:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 10 librados: watch_flush enter 2024-02-23T00:22:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 10 librados: watch_flush exit 2024-02-23T00:22:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 20 client.14128.objecter shutdown clearing up homeless session... 2024-02-23T00:22:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 10 client.14128.objecter successfully canceled tick 2024-02-23T00:22:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe450034030 msgr2=0x7fe4500364f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:02.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 1 --2- 172.21.15.87:0/2515297347 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe450034030 0x7fe4500364f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe44c009780 tx=0x7fe44c006cc0).stop 2024-02-23T00:22:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 msgr2=0x7fe46010ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 1 --2- 172.21.15.87:0/2515297347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe46010ad80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe454006940 tx=0x7fe4540090f0).stop 2024-02-23T00:22:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 shutdown_connections 2024-02-23T00:22:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 1 --2- 172.21.15.87:0/2515297347 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe450034030 0x7fe4500364f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 1 --2- 172.21.15.87:0/2515297347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe460070c80 0x7fe46010ad80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.635+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 >> 172.21.15.87:0/2515297347 conn(0x7fe4600f6670 msgr2=0x7fe4600f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.636+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 shutdown_connections 2024-02-23T00:22:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.636+0000 7fe466d18700 1 -- 172.21.15.87:0/2515297347 wait complete. 2024-02-23T00:22:02.639 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:02.636+0000 7fe466d18700 1 librados: shutdown 2024-02-23T00:22:03.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:02.634203+0000 mgr.y (mgr.14116) 5 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:03.683 INFO:teuthology.orchestra.run.smithi087.stderr:Generating ssh key... 2024-02-23T00:22:04.264 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.263+0000 7ffbc4a13700 1 Processor -- start 2024-02-23T00:22:04.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.263+0000 7ffbc4a13700 1 -- start start 2024-02-23T00:22:04.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbc4a13700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc00fb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:04.270 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbc4a13700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc00fb8f0 con 0x7ffbc00faf50 2024-02-23T00:22:04.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbbe59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc00fb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:04.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbbe59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc00fb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55640/0 (socket says 172.21.15.87:55640) 2024-02-23T00:22:04.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbbe59c700 1 -- 172.21.15.87:0/4038490035 learned_addr learned my addr 172.21.15.87:0/4038490035 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:04.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbbe59c700 1 -- 172.21.15.87:0/4038490035 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc00fba30 con 0x7ffbc00faf50 2024-02-23T00:22:04.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbbe59c700 1 --2- 172.21.15.87:0/4038490035 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc00fb370 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7ffba8009730 tx=0x7ffba80090f0).ready entity=mon.0 client_cookie=d8c73d851e1687f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:04.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbbd59a700 1 -- 172.21.15.87:0/4038490035 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffba800b580 con 0x7ffbc00faf50 2024-02-23T00:22:04.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbbd59a700 1 -- 172.21.15.87:0/4038490035 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffba800bb90 con 0x7ffbc00faf50 2024-02-23T00:22:04.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.264+0000 7ffbbd59a700 1 -- 172.21.15.87:0/4038490035 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffba8011c30 con 0x7ffbc00faf50 2024-02-23T00:22:04.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.265+0000 7ffbc4a13700 1 -- 172.21.15.87:0/4038490035 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 msgr2=0x7ffbc00fb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:04.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.265+0000 7ffbc4a13700 1 --2- 172.21.15.87:0/4038490035 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc00fb370 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7ffba8009730 tx=0x7ffba80090f0).stop 2024-02-23T00:22:04.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.265+0000 7ffbc4a13700 1 -- 172.21.15.87:0/4038490035 shutdown_connections 2024-02-23T00:22:04.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.265+0000 7ffbc4a13700 1 --2- 172.21.15.87:0/4038490035 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc00fb370 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:04.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.265+0000 7ffbc4a13700 1 -- 172.21.15.87:0/4038490035 >> 172.21.15.87:0/4038490035 conn(0x7ffbc00f6670 msgr2=0x7ffbc00f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:04.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.265+0000 7ffbc4a13700 1 -- 172.21.15.87:0/4038490035 shutdown_connections 2024-02-23T00:22:04.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.265+0000 7ffbc4a13700 1 -- 172.21.15.87:0/4038490035 wait complete. 2024-02-23T00:22:04.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbc4a13700 1 librados: starting msgr at 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbc4a13700 1 librados: starting objecter 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbc4a13700 1 Processor -- start 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbc4a13700 1 -- start start 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbc4a13700 1 librados: setting wanted keys 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbc4a13700 1 librados: calling monclient init 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbc4a13700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc010da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbc4a13700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0070c40 con 0x7ffbc00faf50 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbbe59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc010da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbbe59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc010da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55654/0 (socket says 172.21.15.87:55654) 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbbe59c700 1 -- 172.21.15.87:0/3840492677 learned_addr learned my addr 172.21.15.87:0/3840492677 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:04.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.266+0000 7ffbbe59c700 1 -- 172.21.15.87:0/3840492677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8009370 con 0x7ffbc00faf50 2024-02-23T00:22:04.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbbe59c700 1 --2- 172.21.15.87:0/3840492677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc010da20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ffba800bf60 tx=0x7ffba80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:04.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbb77fe700 10 client.?.objecter ms_handle_connect 0x7ffbc00faf50 2024-02-23T00:22:04.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbb77fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:04.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbb77fe700 1 -- 172.21.15.87:0/3840492677 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffba8011bf0 con 0x7ffbc00faf50 2024-02-23T00:22:04.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbb77fe700 1 -- 172.21.15.87:0/3840492677 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffba801aaf0 con 0x7ffbc00faf50 2024-02-23T00:22:04.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc0070e40 con 0x7ffbc00faf50 2024-02-23T00:22:04.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbb77fe700 1 -- 172.21.15.87:0/3840492677 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffba80229c0 con 0x7ffbc00faf50 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbc4a13700 10 client.14130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc010e0d0 con 0x7ffbc00faf50 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.267+0000 7ffbc4a13700 1 librados: init done 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbb77fe700 1 -- 172.21.15.87:0/3840492677 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7ffba8022b20 con 0x7ffbc00faf50 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbb77fe700 1 --2- 172.21.15.87:0/3840492677 >> v2:172.21.15.87:6800/2138166229 conn(0x7ffbac034030 0x7ffbac0364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbb77fe700 1 -- 172.21.15.87:0/3840492677 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7ffba80447c0 con 0x7ffbc00faf50 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbb77fe700 10 client.14130.objecter ms_dispatch 0x7ffbc0001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbb77fe700 3 client.14130.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbb77fe700 3 client.14130.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:04.275 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbb77fe700 20 client.14130.objecter dump_active .. 0 homeless 2024-02-23T00:22:04.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc005a310 con 0x7ffbc00faf50 2024-02-23T00:22:04.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbbdd9b700 1 --2- 172.21.15.87:0/3840492677 >> v2:172.21.15.87:6800/2138166229 conn(0x7ffbac034030 0x7ffbac0364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:04.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbb77fe700 10 client.14130.objecter ms_handle_connect 0x7ffbac034030 2024-02-23T00:22:04.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.268+0000 7ffbbdd9b700 1 --2- 172.21.15.87:0/3840492677 >> v2:172.21.15.87:6800/2138166229 conn(0x7ffbac034030 0x7ffbac0364f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ffbb0009780 tx=0x7ffbb0006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:04.276 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.272+0000 7ffbb77fe700 1 -- 172.21.15.87:0/3840492677 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ffba801e070 con 0x7ffbc00faf50 2024-02-23T00:22:04.437 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.435+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 --> v2:172.21.15.87:6800/2138166229 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbc0059120 con 0x7ffbac034030 2024-02-23T00:22:04.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.567+0000 7ffbb77fe700 1 -- 172.21.15.87:0/3840492677 <== mgr.14116 v2:172.21.15.87:6800/2138166229 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffbc0059120 con 0x7ffbac034030 2024-02-23T00:22:04.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.568+0000 7ffbc4a13700 10 librados: watch_flush enter 2024-02-23T00:22:04.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.568+0000 7ffbc4a13700 10 librados: watch_flush exit 2024-02-23T00:22:04.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.568+0000 7ffbc4a13700 20 client.14130.objecter shutdown clearing up homeless session... 2024-02-23T00:22:04.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.568+0000 7ffbc4a13700 10 client.14130.objecter successfully canceled tick 2024-02-23T00:22:04.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 >> v2:172.21.15.87:6800/2138166229 conn(0x7ffbac034030 msgr2=0x7ffbac0364f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 --2- 172.21.15.87:0/3840492677 >> v2:172.21.15.87:6800/2138166229 conn(0x7ffbac034030 0x7ffbac0364f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ffbb0009780 tx=0x7ffbb0006cc0).stop 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 msgr2=0x7ffbc010da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 --2- 172.21.15.87:0/3840492677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc010da20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ffba800bf60 tx=0x7ffba80090f0).stop 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 shutdown_connections 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 --2- 172.21.15.87:0/3840492677 >> v2:172.21.15.87:6800/2138166229 conn(0x7ffbac034030 0x7ffbac0364f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 --2- 172.21.15.87:0/3840492677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffbc00faf50 0x7ffbc010da20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 >> 172.21.15.87:0/3840492677 conn(0x7ffbc00f6670 msgr2=0x7ffbc00f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 shutdown_connections 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 -- 172.21.15.87:0/3840492677 wait complete. 2024-02-23T00:22:04.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:04.569+0000 7ffbc4a13700 1 librados: shutdown 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: Generating public/private rsa key pair. 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: Your identification has been saved in /tmp/tmpohtll1m9/key. 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: Your public key has been saved in /tmp/tmpohtll1m9/key.pub. 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: The key fingerprint is: 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: SHA256:l2Al1ag2ebSFquf2RtILwaPF/lyZtg61FENobnOZJ/U ceph-66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: The key's randomart image is: 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: +---[RSA 3072]----+ 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: | ..o=. | 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: | o*.o . | 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: | ooB oo+ . | 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: | .%.*.=o. E| 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: | *SBooo= | 2024-02-23T00:22:04.833 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: | o =.oo=. | 2024-02-23T00:22:04.834 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: | o *.+.. | 2024-02-23T00:22:04.834 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: | o =.. | 2024-02-23T00:22:04.834 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: | . o... | 2024-02-23T00:22:04.834 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: +----[SHA256]-----+ 2024-02-23T00:22:05.769 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.764+0000 7f7288082700 1 Processor -- start 2024-02-23T00:22:05.769 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.764+0000 7f7288082700 1 -- start start 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.764+0000 7f7288082700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f72800fec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.764+0000 7f7288082700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72800ff1f0 con 0x7f7280070c60 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.764+0000 7f7285e1e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f72800fec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.764+0000 7f7285e1e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f72800fec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55660/0 (socket says 172.21.15.87:55660) 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.764+0000 7f7285e1e700 1 -- 172.21.15.87:0/387989951 learned_addr learned my addr 172.21.15.87:0/387989951 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7285e1e700 1 -- 172.21.15.87:0/387989951 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72800ff330 con 0x7f7280070c60 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7285e1e700 1 --2- 172.21.15.87:0/387989951 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f72800fec70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f727c009730 tx=0x7f727c0090f0).ready entity=mon.0 client_cookie=10c39506edcbfbdf server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7284e1c700 1 -- 172.21.15.87:0/387989951 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f727c00b580 con 0x7f7280070c60 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7284e1c700 1 -- 172.21.15.87:0/387989951 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f727c00bb90 con 0x7f7280070c60 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7284e1c700 1 -- 172.21.15.87:0/387989951 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f727c011c30 con 0x7f7280070c60 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7288082700 1 -- 172.21.15.87:0/387989951 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 msgr2=0x7f72800fec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7288082700 1 --2- 172.21.15.87:0/387989951 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f72800fec70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f727c009730 tx=0x7f727c0090f0).stop 2024-02-23T00:22:05.770 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7288082700 1 -- 172.21.15.87:0/387989951 shutdown_connections 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7288082700 1 --2- 172.21.15.87:0/387989951 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f72800fec70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.765+0000 7f7288082700 1 -- 172.21.15.87:0/387989951 >> 172.21.15.87:0/387989951 conn(0x7f72800f6670 msgr2=0x7f72800f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.766+0000 7f7288082700 1 -- 172.21.15.87:0/387989951 shutdown_connections 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.766+0000 7f7288082700 1 -- 172.21.15.87:0/387989951 wait complete. 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.766+0000 7f7288082700 1 librados: starting msgr at 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.766+0000 7f7288082700 1 librados: starting objecter 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.766+0000 7f7288082700 1 Processor -- start 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.766+0000 7f7288082700 1 -- start start 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.766+0000 7f7288082700 1 librados: setting wanted keys 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.766+0000 7f7288082700 1 librados: calling monclient init 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7288082700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f728010ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7288082700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728010b230 con 0x7f7280070c60 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7285e1e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f728010ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7285e1e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f728010ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55668/0 (socket says 172.21.15.87:55668) 2024-02-23T00:22:05.771 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7285e1e700 1 -- 172.21.15.87:0/2433833965 learned_addr learned my addr 172.21.15.87:0/2433833965 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:05.772 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7285e1e700 1 -- 172.21.15.87:0/2433833965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c009370 con 0x7f7280070c60 2024-02-23T00:22:05.772 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7285e1e700 1 --2- 172.21.15.87:0/2433833965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f728010ad40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f727c000c00 tx=0x7f727c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7276ffd700 10 client.?.objecter ms_handle_connect 0x7f7280070c60 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7276ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7276ffd700 1 -- 172.21.15.87:0/2433833965 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f727c01a430 con 0x7f7280070c60 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7276ffd700 1 -- 172.21.15.87:0/2433833965 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f727c01aa40 con 0x7f7280070c60 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7276ffd700 1 -- 172.21.15.87:0/2433833965 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f727c023b00 con 0x7f7280070c60 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728010b430 con 0x7f7280070c60 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7288082700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728010c040 con 0x7f7280070c60 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.767+0000 7f7288082700 1 librados: init done 2024-02-23T00:22:05.774 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.768+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728010b8c0 con 0x7f7280070c60 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.768+0000 7f7276ffd700 1 -- 172.21.15.87:0/2433833965 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f727c02c3f0 con 0x7f7280070c60 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.768+0000 7f7276ffd700 1 --2- 172.21.15.87:0/2433833965 >> v2:172.21.15.87:6800/2138166229 conn(0x7f726c034030 0x7f726c0364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.768+0000 7f7276ffd700 1 -- 172.21.15.87:0/2433833965 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f727c044d50 con 0x7f7280070c60 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.768+0000 7f7276ffd700 10 client.14132.objecter ms_dispatch 0x7f7280001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.768+0000 7f7276ffd700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.768+0000 7f7276ffd700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.768+0000 7f7276ffd700 20 client.14132.objecter dump_active .. 0 homeless 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.768+0000 7f728561d700 1 --2- 172.21.15.87:0/2433833965 >> v2:172.21.15.87:6800/2138166229 conn(0x7f726c034030 0x7f726c0364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.769+0000 7f7276ffd700 10 client.14132.objecter ms_handle_connect 0x7f726c034030 2024-02-23T00:22:05.775 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.769+0000 7f728561d700 1 --2- 172.21.15.87:0/2433833965 >> v2:172.21.15.87:6800/2138166229 conn(0x7f726c034030 0x7f726c0364f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7270009780 tx=0x7f7270006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:05.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.773+0000 7f7276ffd700 1 -- 172.21.15.87:0/2433833965 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f728010b8c0 con 0x7f7280070c60 2024-02-23T00:22:05.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:04.436786+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:05.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:22:04.436981+0000 mgr.y (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-02-23T00:22:05.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:04.565003+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:22:05.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:04.566737+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:22:05.939 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.937+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 --> v2:172.21.15.87:6800/2138166229 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7280100320 con 0x7f726c034030 2024-02-23T00:22:05.940 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.937+0000 7f7276ffd700 1 -- 172.21.15.87:0/2433833965 <== mgr.14116 v2:172.21.15.87:6800/2138166229 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7280100320 con 0x7f726c034030 2024-02-23T00:22:05.940 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4xb4ehDsa1R++8TY7iX5ijpvAcLZRUn9xhWWhRsIU/yRO19ZT8nJ94q6Y7uQNAtyRM2jy6cFHpaXKyMv0Jg5Qx3WVDBTtrpBDX8/VssUEZ4SyAOxXCKXGrpMQ6XTwBMXHWb4qUhAMKF6GObfJpufyuZF0+M41w4WvXkpM8DqYf6VGWdHLGdxOD7R+5ilGbRdBhdgoJD1T1b2E6MI1waKsbwk6yYYZUyPMPhrEJG6KhHC6AOiFEz8WkIFpaiH4nzXR6LaW5oGsIeVnnaYp1K3X/TV3Qo0w+rV0PMNAkMkuUG7XEbd2Tu8Dt8YjUld6AjQWnvcsOeAwmaw1FXbpVtmyLdH3RCkDzhAlAYG1GyjK/rF7LzC+4rVU8JZy4Mt43pVUt0vjyi8P2ZwaA2Ba2IWw8LIlKAW3R6Sz0HQql2jApM9wY350SvOaG/8KyCoXikueYZUmDQpOzBHvhlkDS3r9ytps8fyIJPRPjwNwiGMI7jiF3P73Vw0SM6/BGkRBKt0= ceph-66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 10 librados: watch_flush enter 2024-02-23T00:22:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 10 librados: watch_flush exit 2024-02-23T00:22:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 20 client.14132.objecter shutdown clearing up homeless session... 2024-02-23T00:22:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 10 client.14132.objecter successfully canceled tick 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 >> v2:172.21.15.87:6800/2138166229 conn(0x7f726c034030 msgr2=0x7f726c0364f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 --2- 172.21.15.87:0/2433833965 >> v2:172.21.15.87:6800/2138166229 conn(0x7f726c034030 0x7f726c0364f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7270009780 tx=0x7f7270006cc0).stop 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 msgr2=0x7f728010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 --2- 172.21.15.87:0/2433833965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f728010ad40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f727c000c00 tx=0x7f727c0090f0).stop 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 shutdown_connections 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 --2- 172.21.15.87:0/2433833965 >> v2:172.21.15.87:6800/2138166229 conn(0x7f726c034030 0x7f726c0364f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 --2- 172.21.15.87:0/2433833965 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7280070c60 0x7f728010ad40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 >> 172.21.15.87:0/2433833965 conn(0x7f72800f6670 msgr2=0x7f72800f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 shutdown_connections 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 -- 172.21.15.87:0/2433833965 wait complete. 2024-02-23T00:22:05.943 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:05.939+0000 7f7288082700 1 librados: shutdown 2024-02-23T00:22:06.582 INFO:teuthology.orchestra.run.smithi087.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-23T00:22:06.582 INFO:teuthology.orchestra.run.smithi087.stderr:Adding key to root@localhost authorized_keys... 2024-02-23T00:22:06.582 INFO:teuthology.orchestra.run.smithi087.stderr:Adding host smithi087... 2024-02-23T00:22:07.134 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.130+0000 7f9e61963700 1 Processor -- start 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.130+0000 7f9e61963700 1 -- start start 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.130+0000 7f9e61963700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c0fec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.130+0000 7f9e61963700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c0ff180 con 0x7f9e5c070cc0 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.131+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c0fec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.131+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c0fec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55676/0 (socket says 172.21.15.87:55676) 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.131+0000 7f9e5affd700 1 -- 172.21.15.87:0/1033885667 learned_addr learned my addr 172.21.15.87:0/1033885667 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.131+0000 7f9e5affd700 1 -- 172.21.15.87:0/1033885667 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0ff2c0 con 0x7f9e5c070cc0 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.131+0000 7f9e5affd700 1 --2- 172.21.15.87:0/1033885667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c0fec00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9e50009730 tx=0x7f9e500090f0).ready entity=mon.0 client_cookie=b71c48aae75f4b72 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.131+0000 7f9e59ffb700 1 -- 172.21.15.87:0/1033885667 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e5000b580 con 0x7f9e5c070cc0 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.131+0000 7f9e59ffb700 1 -- 172.21.15.87:0/1033885667 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e5000bb90 con 0x7f9e5c070cc0 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.131+0000 7f9e59ffb700 1 -- 172.21.15.87:0/1033885667 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e50011c30 con 0x7f9e5c070cc0 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 -- 172.21.15.87:0/1033885667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 msgr2=0x7f9e5c0fec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 --2- 172.21.15.87:0/1033885667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c0fec00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9e50009730 tx=0x7f9e500090f0).stop 2024-02-23T00:22:07.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 -- 172.21.15.87:0/1033885667 shutdown_connections 2024-02-23T00:22:07.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 --2- 172.21.15.87:0/1033885667 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c0fec00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:07.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 -- 172.21.15.87:0/1033885667 >> 172.21.15.87:0/1033885667 conn(0x7f9e5c0f6660 msgr2=0x7f9e5c0f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:07.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 -- 172.21.15.87:0/1033885667 shutdown_connections 2024-02-23T00:22:07.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 -- 172.21.15.87:0/1033885667 wait complete. 2024-02-23T00:22:07.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 librados: starting msgr at 2024-02-23T00:22:07.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 librados: starting objecter 2024-02-23T00:22:07.136 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.132+0000 7f9e61963700 1 Processor -- start 2024-02-23T00:22:07.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e61963700 1 -- start start 2024-02-23T00:22:07.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e61963700 1 librados: setting wanted keys 2024-02-23T00:22:07.139 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e61963700 1 librados: calling monclient init 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e61963700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c10ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e61963700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c10b240 con 0x7f9e5c070cc0 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c10ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e5affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c10ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55684/0 (socket says 172.21.15.87:55684) 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e5affd700 1 -- 172.21.15.87:0/461071802 learned_addr learned my addr 172.21.15.87:0/461071802 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e5affd700 1 -- 172.21.15.87:0/461071802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e50009370 con 0x7f9e5c070cc0 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e5affd700 1 --2- 172.21.15.87:0/461071802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c10ad50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9e50006940 tx=0x7f9e500090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e43fff700 10 client.?.objecter ms_handle_connect 0x7f9e5c070cc0 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e43fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e43fff700 1 -- 172.21.15.87:0/461071802 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e5001a430 con 0x7f9e5c070cc0 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c10b440 con 0x7f9e5c070cc0 2024-02-23T00:22:07.140 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e43fff700 1 -- 172.21.15.87:0/461071802 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e5001aa40 con 0x7f9e5c070cc0 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e43fff700 1 -- 172.21.15.87:0/461071802 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e50023b90 con 0x7f9e5c070cc0 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e61963700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c10bff0 con 0x7f9e5c070cc0 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.133+0000 7f9e61963700 1 librados: init done 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.134+0000 7f9e43fff700 1 -- 172.21.15.87:0/461071802 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f9e50023cf0 con 0x7f9e5c070cc0 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.134+0000 7f9e43fff700 1 --2- 172.21.15.87:0/461071802 >> v2:172.21.15.87:6800/2138166229 conn(0x7f9e44033fe0 0x7f9e440364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.134+0000 7f9e43fff700 1 -- 172.21.15.87:0/461071802 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f9e50045770 con 0x7f9e5c070cc0 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.134+0000 7f9e43fff700 10 client.14134.objecter ms_dispatch 0x7f9e5c001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.134+0000 7f9e43fff700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.134+0000 7f9e43fff700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.134+0000 7f9e43fff700 20 client.14134.objecter dump_active .. 0 homeless 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.134+0000 7f9e5a7fc700 1 --2- 172.21.15.87:0/461071802 >> v2:172.21.15.87:6800/2138166229 conn(0x7f9e44033fe0 0x7f9e440364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.134+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c05a310 con 0x7f9e5c070cc0 2024-02-23T00:22:07.141 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.135+0000 7f9e5a7fc700 1 --2- 172.21.15.87:0/461071802 >> v2:172.21.15.87:6800/2138166229 conn(0x7f9e44033fe0 0x7f9e440364a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9e4c009780 tx=0x7f9e4c006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:07.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.135+0000 7f9e43fff700 10 client.14134.objecter ms_handle_connect 0x7f9e44033fe0 2024-02-23T00:22:07.142 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.139+0000 7f9e43fff700 1 -- 172.21.15.87:0/461071802 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9e5001e070 con 0x7f9e5c070cc0 2024-02-23T00:22:07.307 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.305+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 --> v2:172.21.15.87:6800/2138166229 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi087", "addr": "172.21.15.87", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e5c10b8d0 con 0x7f9e44033fe0 2024-02-23T00:22:07.329 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:05.939088+0000 mgr.y (mgr.14116) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:07.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [79B blob data] 2024-02-23T00:22:07.917 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Added host 'smithi087' with addr '172.21.15.87' 2024-02-23T00:22:07.918 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.914+0000 7f9e43fff700 1 -- 172.21.15.87:0/461071802 <== mgr.14116 v2:172.21.15.87:6800/2138166229 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f9e5c10b8d0 con 0x7f9e44033fe0 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.916+0000 7f9e61963700 10 librados: watch_flush enter 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.916+0000 7f9e61963700 10 librados: watch_flush exit 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.916+0000 7f9e61963700 20 client.14134.objecter shutdown clearing up homeless session... 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.916+0000 7f9e61963700 10 client.14134.objecter successfully canceled tick 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 >> v2:172.21.15.87:6800/2138166229 conn(0x7f9e44033fe0 msgr2=0x7f9e440364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 --2- 172.21.15.87:0/461071802 >> v2:172.21.15.87:6800/2138166229 conn(0x7f9e44033fe0 0x7f9e440364a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9e4c009780 tx=0x7f9e4c006cc0).stop 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 msgr2=0x7f9e5c10ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 --2- 172.21.15.87:0/461071802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c10ad50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9e50006940 tx=0x7f9e500090f0).stop 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 shutdown_connections 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 --2- 172.21.15.87:0/461071802 >> v2:172.21.15.87:6800/2138166229 conn(0x7f9e44033fe0 0x7f9e440364a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:07.920 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 --2- 172.21.15.87:0/461071802 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e5c070cc0 0x7f9e5c10ad50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:07.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 >> 172.21.15.87:0/461071802 conn(0x7f9e5c0f6660 msgr2=0x7f9e5c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:07.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 shutdown_connections 2024-02-23T00:22:07.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 -- 172.21.15.87:0/461071802 wait complete. 2024-02-23T00:22:07.921 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:07.917+0000 7f9e61963700 1 librados: shutdown 2024-02-23T00:22:08.606 INFO:teuthology.orchestra.run.smithi087.stderr:Deploying unmanaged mon service... 2024-02-23T00:22:09.204 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.159+0000 7f16256c7700 1 Processor -- start 2024-02-23T00:22:09.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.159+0000 7f16256c7700 1 -- start start 2024-02-23T00:22:09.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f16256c7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f16200fb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:09.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f16256c7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16200fb8e0 con 0x7f16200faf40 2024-02-23T00:22:09.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f161effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f16200fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:09.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f161effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f16200fb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55686/0 (socket says 172.21.15.87:55686) 2024-02-23T00:22:09.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f161effd700 1 -- 172.21.15.87:0/1871648929 learned_addr learned my addr 172.21.15.87:0/1871648929 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:09.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f161effd700 1 -- 172.21.15.87:0/1871648929 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16200fba20 con 0x7f16200faf40 2024-02-23T00:22:09.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f161effd700 1 --2- 172.21.15.87:0/1871648929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f16200fb360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f1614009730 tx=0x7f16140090f0).ready entity=mon.0 client_cookie=3cfe0ae3a10293c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:09.205 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f161dffb700 1 -- 172.21.15.87:0/1871648929 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f161400b580 con 0x7f16200faf40 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f161dffb700 1 -- 172.21.15.87:0/1871648929 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f161400bb90 con 0x7f16200faf40 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.160+0000 7f161dffb700 1 -- 172.21.15.87:0/1871648929 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1614011c30 con 0x7f16200faf40 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 -- 172.21.15.87:0/1871648929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 msgr2=0x7f16200fb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 --2- 172.21.15.87:0/1871648929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f16200fb360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f1614009730 tx=0x7f16140090f0).stop 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 -- 172.21.15.87:0/1871648929 shutdown_connections 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 --2- 172.21.15.87:0/1871648929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f16200fb360 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 -- 172.21.15.87:0/1871648929 >> 172.21.15.87:0/1871648929 conn(0x7f16200f6660 msgr2=0x7f16200f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 -- 172.21.15.87:0/1871648929 shutdown_connections 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 -- 172.21.15.87:0/1871648929 wait complete. 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 librados: starting msgr at 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 librados: starting objecter 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.161+0000 7f16256c7700 1 Processor -- start 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f16256c7700 1 -- start start 2024-02-23T00:22:09.206 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f16256c7700 1 librados: setting wanted keys 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f16256c7700 1 librados: calling monclient init 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f16256c7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f162010d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f16256c7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1620070c40 con 0x7f16200faf40 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f161effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f162010d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f161effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f162010d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55702/0 (socket says 172.21.15.87:55702) 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f161effd700 1 -- 172.21.15.87:0/1120606855 learned_addr learned my addr 172.21.15.87:0/1120606855 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f161effd700 1 -- 172.21.15.87:0/1120606855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1614009370 con 0x7f16200faf40 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f161effd700 1 --2- 172.21.15.87:0/1120606855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f162010d9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f1614000c00 tx=0x7f16140090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f1607fff700 10 client.?.objecter ms_handle_connect 0x7f16200faf40 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f1607fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f1607fff700 1 -- 172.21.15.87:0/1120606855 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f161400b9a0 con 0x7f16200faf40 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f1607fff700 1 -- 172.21.15.87:0/1120606855 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f161401aaf0 con 0x7f16200faf40 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1620070e40 con 0x7f16200faf40 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f1607fff700 1 -- 172.21.15.87:0/1120606855 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1614023ac0 con 0x7f16200faf40 2024-02-23T00:22:09.207 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f16256c7700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162010e080 con 0x7f16200faf40 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.162+0000 7f16256c7700 1 librados: init done 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.163+0000 7f1607fff700 1 -- 172.21.15.87:0/1120606855 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f1614023d30 con 0x7f16200faf40 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.163+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f162005a310 con 0x7f16200faf40 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.163+0000 7f1607fff700 1 --2- 172.21.15.87:0/1120606855 >> v2:172.21.15.87:6800/2138166229 conn(0x7f1608034020 0x7f16080364e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.163+0000 7f1607fff700 1 -- 172.21.15.87:0/1120606855 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f1614044be0 con 0x7f16200faf40 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.163+0000 7f1607fff700 10 client.14136.objecter ms_dispatch 0x7f1620001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.163+0000 7f1607fff700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.163+0000 7f1607fff700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.163+0000 7f1607fff700 20 client.14136.objecter dump_active .. 0 homeless 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.163+0000 7f161e7fc700 1 --2- 172.21.15.87:0/1120606855 >> v2:172.21.15.87:6800/2138166229 conn(0x7f1608034020 0x7f16080364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.164+0000 7f1607fff700 10 client.14136.objecter ms_handle_connect 0x7f1608034020 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.164+0000 7f161e7fc700 1 --2- 172.21.15.87:0/1120606855 >> v2:172.21.15.87:6800/2138166229 conn(0x7f1608034020 0x7f16080364e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f1610009780 tx=0x7f1610006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:09.208 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.168+0000 7f1607fff700 1 -- 172.21.15.87:0/1120606855 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f161402b9a0 con 0x7f16200faf40 2024-02-23T00:22:09.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:07.307497+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi087", "addr": "172.21.15.87", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:09.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:07.915354+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:22:09.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:22:07.915638+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Added host smithi087 2024-02-23T00:22:09.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:07.919663+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:09.344 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.342+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 --> v2:172.21.15.87:6800/2138166229 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1620103d60 con 0x7f1608034020 2024-02-23T00:22:09.349 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.347+0000 7f1607fff700 1 -- 172.21.15.87:0/1120606855 <== mgr.14116 v2:172.21.15.87:6800/2138166229 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1620103d60 con 0x7f1608034020 2024-02-23T00:22:09.349 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-23T00:22:09.352 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 10 librados: watch_flush enter 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 10 librados: watch_flush exit 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 20 client.14136.objecter shutdown clearing up homeless session... 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 10 client.14136.objecter successfully canceled tick 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 >> v2:172.21.15.87:6800/2138166229 conn(0x7f1608034020 msgr2=0x7f16080364e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 --2- 172.21.15.87:0/1120606855 >> v2:172.21.15.87:6800/2138166229 conn(0x7f1608034020 0x7f16080364e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f1610009780 tx=0x7f1610006cc0).stop 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 msgr2=0x7f162010d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 --2- 172.21.15.87:0/1120606855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f162010d9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f1614000c00 tx=0x7f16140090f0).stop 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 shutdown_connections 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 --2- 172.21.15.87:0/1120606855 >> v2:172.21.15.87:6800/2138166229 conn(0x7f1608034020 0x7f16080364e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 --2- 172.21.15.87:0/1120606855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f16200faf40 0x7f162010d9d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 >> 172.21.15.87:0/1120606855 conn(0x7f16200f6660 msgr2=0x7f16200f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 shutdown_connections 2024-02-23T00:22:09.354 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.349+0000 7f16256c7700 1 -- 172.21.15.87:0/1120606855 wait complete. 2024-02-23T00:22:09.354 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:09.350+0000 7f16256c7700 1 librados: shutdown 2024-02-23T00:22:10.301 INFO:teuthology.orchestra.run.smithi087.stderr:Deploying unmanaged mgr service... 2024-02-23T00:22:10.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:09.344759+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:10.565 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:22:09.345879+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-23T00:22:10.565 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:09.348851+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:22:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.977+0000 7f0014017700 1 Processor -- start 2024-02-23T00:22:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.977+0000 7f0014017700 1 -- start start 2024-02-23T00:22:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.977+0000 7f0014017700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c0fece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.977+0000 7f0014017700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c0ff1d0 con 0x7f000c070be0 2024-02-23T00:22:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.977+0000 7f0011db3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c0fece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.977+0000 7f0011db3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c0fece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55712/0 (socket says 172.21.15.87:55712) 2024-02-23T00:22:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.977+0000 7f0011db3700 1 -- 172.21.15.87:0/1741896378 learned_addr learned my addr 172.21.15.87:0/1741896378 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:10.984 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0011db3700 1 -- 172.21.15.87:0/1741896378 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c0ff310 con 0x7f000c070be0 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0011db3700 1 --2- 172.21.15.87:0/1741896378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c0fece0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0008009730 tx=0x7f00080090f0).ready entity=mon.0 client_cookie=2c63f3e07ab1113a server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0010db1700 1 -- 172.21.15.87:0/1741896378 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f000800b580 con 0x7f000c070be0 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0010db1700 1 -- 172.21.15.87:0/1741896378 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f000800bb90 con 0x7f000c070be0 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0010db1700 1 -- 172.21.15.87:0/1741896378 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0008011c30 con 0x7f000c070be0 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0014017700 1 -- 172.21.15.87:0/1741896378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 msgr2=0x7f000c0fece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0014017700 1 --2- 172.21.15.87:0/1741896378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c0fece0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0008009730 tx=0x7f00080090f0).stop 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0014017700 1 -- 172.21.15.87:0/1741896378 shutdown_connections 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0014017700 1 --2- 172.21.15.87:0/1741896378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c0fece0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.978+0000 7f0014017700 1 -- 172.21.15.87:0/1741896378 >> 172.21.15.87:0/1741896378 conn(0x7f000c0f6660 msgr2=0x7f000c0f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 -- 172.21.15.87:0/1741896378 shutdown_connections 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 -- 172.21.15.87:0/1741896378 wait complete. 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 librados: starting msgr at 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 librados: starting objecter 2024-02-23T00:22:10.985 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 Processor -- start 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 -- start start 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 librados: setting wanted keys 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 librados: calling monclient init 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c10ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0014017700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c10b270 con 0x7f000c070be0 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0011db3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c10ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0011db3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c10ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55720/0 (socket says 172.21.15.87:55720) 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.979+0000 7f0011db3700 1 -- 172.21.15.87:0/897306179 learned_addr learned my addr 172.21.15.87:0/897306179 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0011db3700 1 -- 172.21.15.87:0/897306179 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0008009370 con 0x7f000c070be0 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0011db3700 1 --2- 172.21.15.87:0/897306179 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c10ad80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0008006940 tx=0x7f00080090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0002ffd700 10 client.?.objecter ms_handle_connect 0x7f000c070be0 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0002ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0002ffd700 1 -- 172.21.15.87:0/897306179 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f000801a430 con 0x7f000c070be0 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0002ffd700 1 -- 172.21.15.87:0/897306179 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f000801aa40 con 0x7f000c070be0 2024-02-23T00:22:10.986 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0002ffd700 1 -- 172.21.15.87:0/897306179 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0008023ab0 con 0x7f000c070be0 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000c10b470 con 0x7f000c070be0 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0014017700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c10c080 con 0x7f000c070be0 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.980+0000 7f0014017700 1 librados: init done 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.981+0000 7f0002ffd700 1 -- 172.21.15.87:0/897306179 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f000801abb0 con 0x7f000c070be0 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.981+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c10b900 con 0x7f000c070be0 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.981+0000 7f0002ffd700 1 --2- 172.21.15.87:0/897306179 >> v2:172.21.15.87:6800/2138166229 conn(0x7efff8034020 0x7efff80364e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.981+0000 7f0002ffd700 1 -- 172.21.15.87:0/897306179 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f0008044800 con 0x7f000c070be0 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.981+0000 7f0002ffd700 10 client.14138.objecter ms_dispatch 0x7f000c001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.981+0000 7f0002ffd700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.981+0000 7f0002ffd700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.981+0000 7f0002ffd700 20 client.14138.objecter dump_active .. 0 homeless 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.981+0000 7f00115b2700 1 --2- 172.21.15.87:0/897306179 >> v2:172.21.15.87:6800/2138166229 conn(0x7efff8034020 0x7efff80364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:10.987 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.982+0000 7f00115b2700 1 --2- 172.21.15.87:0/897306179 >> v2:172.21.15.87:6800/2138166229 conn(0x7efff8034020 0x7efff80364e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efffc009780 tx=0x7efffc006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:10.988 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.982+0000 7f0002ffd700 10 client.14138.objecter ms_handle_connect 0x7efff8034020 2024-02-23T00:22:10.988 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:10.986+0000 7f0002ffd700 1 -- 172.21.15.87:0/897306179 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f000c10b900 con 0x7f000c070be0 2024-02-23T00:22:11.164 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.161+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 --> v2:172.21.15.87:6800/2138166229 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f000c100300 con 0x7efff8034020 2024-02-23T00:22:11.169 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.166+0000 7f0002ffd700 1 -- 172.21.15.87:0/897306179 <== mgr.14116 v2:172.21.15.87:6800/2138166229 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f000c100300 con 0x7efff8034020 2024-02-23T00:22:11.169 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-23T00:22:11.171 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 10 librados: watch_flush enter 2024-02-23T00:22:11.171 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 10 librados: watch_flush exit 2024-02-23T00:22:11.171 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 20 client.14138.objecter shutdown clearing up homeless session... 2024-02-23T00:22:11.171 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 10 client.14138.objecter successfully canceled tick 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 >> v2:172.21.15.87:6800/2138166229 conn(0x7efff8034020 msgr2=0x7efff80364e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 --2- 172.21.15.87:0/897306179 >> v2:172.21.15.87:6800/2138166229 conn(0x7efff8034020 0x7efff80364e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efffc009780 tx=0x7efffc006cc0).stop 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 msgr2=0x7f000c10ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 --2- 172.21.15.87:0/897306179 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c10ad80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0008006940 tx=0x7f00080090f0).stop 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 shutdown_connections 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 --2- 172.21.15.87:0/897306179 >> v2:172.21.15.87:6800/2138166229 conn(0x7efff8034020 0x7efff80364e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 --2- 172.21.15.87:0/897306179 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f000c070be0 0x7f000c10ad80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 >> 172.21.15.87:0/897306179 conn(0x7f000c0f6660 msgr2=0x7f000c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 shutdown_connections 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.168+0000 7f0014017700 1 -- 172.21.15.87:0/897306179 wait complete. 2024-02-23T00:22:11.172 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:11.169+0000 7f0014017700 1 librados: shutdown 2024-02-23T00:22:12.437 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:11.164006+0000 mgr.y (mgr.14116) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:12.437 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:22:11.165133+0000 mgr.y (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-23T00:22:12.437 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:11.167792+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:22:12.437 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:11.573615+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:22:12.437 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:11.806546+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:22:12.772 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.769+0000 7f2ca5e6a700 1 Processor -- start 2024-02-23T00:22:12.772 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.769+0000 7f2ca5e6a700 1 -- start start 2024-02-23T00:22:12.772 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.769+0000 7f2ca5e6a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca00faad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:12.773 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.769+0000 7f2ca5e6a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca00fb050 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.773 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.770+0000 7f2c9f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca00faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:12.773 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.770+0000 7f2c9f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca00faad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48616/0 (socket says 172.21.15.87:48616) 2024-02-23T00:22:12.773 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.770+0000 7f2c9f7fe700 1 -- 172.21.15.87:0/2585959821 learned_addr learned my addr 172.21.15.87:0/2585959821 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:12.773 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.770+0000 7f2c9f7fe700 1 -- 172.21.15.87:0/2585959821 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca00fb190 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.773 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.770+0000 7f2c9f7fe700 1 --2- 172.21.15.87:0/2585959821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca00faad0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f2c9800af60 tx=0x7f2c9800a950).ready entity=mon.0 client_cookie=f98c6cc97e4660db server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:12.773 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.770+0000 7f2c9e7fc700 1 -- 172.21.15.87:0/2585959821 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c9800f070 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.773 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.770+0000 7f2c9e7fc700 1 -- 172.21.15.87:0/2585959821 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c9800a2e0 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.773 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.770+0000 7f2c9e7fc700 1 -- 172.21.15.87:0/2585959821 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c98014420 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.771+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/2585959821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 msgr2=0x7f2ca00faad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.771+0000 7f2ca5e6a700 1 --2- 172.21.15.87:0/2585959821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca00faad0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f2c9800af60 tx=0x7f2c9800a950).stop 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.771+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/2585959821 shutdown_connections 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.771+0000 7f2ca5e6a700 1 --2- 172.21.15.87:0/2585959821 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca00faad0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.771+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/2585959821 >> 172.21.15.87:0/2585959821 conn(0x7f2ca00f5e30 msgr2=0x7f2ca00f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.771+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/2585959821 shutdown_connections 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.771+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/2585959821 wait complete. 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2ca5e6a700 1 librados: starting msgr at 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2ca5e6a700 1 librados: starting objecter 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2ca5e6a700 1 Processor -- start 2024-02-23T00:22:12.776 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2ca5e6a700 1 -- start start 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2ca5e6a700 1 librados: setting wanted keys 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2ca5e6a700 1 librados: calling monclient init 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2ca5e6a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca0108a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2ca5e6a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca0108f80 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2c9f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca0108a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2c9f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca0108a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48626/0 (socket says 172.21.15.87:48626) 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.772+0000 7f2c9f7fe700 1 -- 172.21.15.87:0/791909993 learned_addr learned my addr 172.21.15.87:0/791909993 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2c9f7fe700 1 -- 172.21.15.87:0/791909993 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9800abf0 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2c9f7fe700 1 --2- 172.21.15.87:0/791909993 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca0108a90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f2c9800afd0 tx=0x7f2c9800a950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2c9cff9700 10 client.?.objecter ms_handle_connect 0x7f2ca00fa6b0 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2c9cff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2c9cff9700 1 -- 172.21.15.87:0/791909993 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c9800f070 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca0109180 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2c9cff9700 1 -- 172.21.15.87:0/791909993 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c98008e90 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2ca5e6a700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2c9cff9700 1 -- 172.21.15.87:0/791909993 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c9801dab0 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca01095b0 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.778 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.773+0000 7f2ca5e6a700 1 librados: init done 2024-02-23T00:22:12.779 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.774+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca0046ce0 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.779 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.775+0000 7f2c9cff9700 1 -- 172.21.15.87:0/791909993 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f2c98003040 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.779 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.775+0000 7f2c9cff9700 1 --2- 172.21.15.87:0/791909993 >> v2:172.21.15.87:6800/2138166229 conn(0x7f2c88036210 0x7f2c880386d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:12.779 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.775+0000 7f2c9cff9700 1 -- 172.21.15.87:0/791909993 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f2c98044bb0 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.779 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.775+0000 7f2c9cff9700 10 client.14140.objecter ms_dispatch 0x7f2ca0001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:12.779 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.775+0000 7f2c9cff9700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:12.779 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.775+0000 7f2c9cff9700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:12.780 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.775+0000 7f2c9cff9700 20 client.14140.objecter dump_active .. 0 homeless 2024-02-23T00:22:12.780 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.775+0000 7f2c9effd700 1 --2- 172.21.15.87:0/791909993 >> v2:172.21.15.87:6800/2138166229 conn(0x7f2c88036210 0x7f2c880386d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:12.780 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.776+0000 7f2c9effd700 1 --2- 172.21.15.87:0/791909993 >> v2:172.21.15.87:6800/2138166229 conn(0x7f2c88036210 0x7f2c880386d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f2c94009780 tx=0x7f2c94006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:12.780 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.776+0000 7f2c9cff9700 10 client.14140.objecter ms_handle_connect 0x7f2c88036210 2024-02-23T00:22:12.781 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.779+0000 7f2c9cff9700 1 -- 172.21.15.87:0/791909993 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2c980082c0 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.956 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.954+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f2ca00fbe40 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.959 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.957+0000 7f2c9cff9700 1 -- 172.21.15.87:0/791909993 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f2c98025b50 con 0x7f2ca00fa6b0 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 10 librados: watch_flush enter 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 10 librados: watch_flush exit 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 20 client.14140.objecter shutdown clearing up homeless session... 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 10 client.14140.objecter successfully canceled tick 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 >> v2:172.21.15.87:6800/2138166229 conn(0x7f2c88036210 msgr2=0x7f2c880386d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 --2- 172.21.15.87:0/791909993 >> v2:172.21.15.87:6800/2138166229 conn(0x7f2c88036210 0x7f2c880386d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f2c94009780 tx=0x7f2c94006cc0).stop 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 msgr2=0x7f2ca0108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 --2- 172.21.15.87:0/791909993 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca0108a90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f2c9800afd0 tx=0x7f2c9800a950).stop 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 shutdown_connections 2024-02-23T00:22:12.962 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 --2- 172.21.15.87:0/791909993 >> v2:172.21.15.87:6800/2138166229 conn(0x7f2c88036210 0x7f2c880386d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:12.963 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 --2- 172.21.15.87:0/791909993 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2ca00fa6b0 0x7f2ca0108a90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:12.963 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 >> 172.21.15.87:0/791909993 conn(0x7f2ca00f5e30 msgr2=0x7f2ca00fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:12.963 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 shutdown_connections 2024-02-23T00:22:12.963 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 -- 172.21.15.87:0/791909993 wait complete. 2024-02-23T00:22:12.963 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:12.959+0000 7f2ca5e6a700 1 librados: shutdown 2024-02-23T00:22:14.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:12.958571+0000 mon.a (mon.0) 59 : audit [INF] from='client.? 172.21.15.87:0/791909993' entity='client.admin' 2024-02-23T00:22:14.567 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.564+0000 7fe570007700 1 Processor -- start 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.564+0000 7fe570007700 1 -- start start 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.564+0000 7fe570007700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe568071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.564+0000 7fe570007700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680730c0 con 0x7fe568072ca0 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.565+0000 7fe56dda3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe568071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.565+0000 7fe56dda3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe568071240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48642/0 (socket says 172.21.15.87:48642) 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.565+0000 7fe56dda3700 1 -- 172.21.15.87:0/2007185221 learned_addr learned my addr 172.21.15.87:0/2007185221 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.565+0000 7fe56dda3700 1 -- 172.21.15.87:0/2007185221 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe568071730 con 0x7fe568072ca0 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.565+0000 7fe56dda3700 1 --2- 172.21.15.87:0/2007185221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe568071240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe564009730 tx=0x7fe5640090f0).ready entity=mon.0 client_cookie=1019edd76c3458f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.565+0000 7fe56cda1700 1 -- 172.21.15.87:0/2007185221 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe56400b580 con 0x7fe568072ca0 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.565+0000 7fe56cda1700 1 -- 172.21.15.87:0/2007185221 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe56400bbb0 con 0x7fe568072ca0 2024-02-23T00:22:14.568 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.565+0000 7fe56cda1700 1 -- 172.21.15.87:0/2007185221 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe564011bc0 con 0x7fe568072ca0 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.566+0000 7fe570007700 1 -- 172.21.15.87:0/2007185221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 msgr2=0x7fe568071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.566+0000 7fe570007700 1 --2- 172.21.15.87:0/2007185221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe568071240 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe564009730 tx=0x7fe5640090f0).stop 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.566+0000 7fe570007700 1 -- 172.21.15.87:0/2007185221 shutdown_connections 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.566+0000 7fe570007700 1 --2- 172.21.15.87:0/2007185221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe568071240 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.566+0000 7fe570007700 1 -- 172.21.15.87:0/2007185221 >> 172.21.15.87:0/2007185221 conn(0x7fe5680f7800 msgr2=0x7fe5680f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 -- 172.21.15.87:0/2007185221 shutdown_connections 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 -- 172.21.15.87:0/2007185221 wait complete. 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 librados: starting msgr at 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 librados: starting objecter 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 Processor -- start 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 -- start start 2024-02-23T00:22:14.571 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 librados: setting wanted keys 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 librados: calling monclient init 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe56810cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.567+0000 7fe570007700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56810d420 con 0x7fe568072ca0 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe56dda3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe56810cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe56dda3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe56810cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48658/0 (socket says 172.21.15.87:48658) 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe56dda3700 1 -- 172.21.15.87:0/2282261336 learned_addr learned my addr 172.21.15.87:0/2282261336 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe56dda3700 1 -- 172.21.15.87:0/2282261336 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe564009370 con 0x7fe568072ca0 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe56dda3700 1 --2- 172.21.15.87:0/2282261336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe56810cf30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fe564000c00 tx=0x7fe5640090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe55effd700 10 client.?.objecter ms_handle_connect 0x7fe568072ca0 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe55effd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe55effd700 1 -- 172.21.15.87:0/2282261336 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe56401a640 con 0x7fe568072ca0 2024-02-23T00:22:14.572 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56810d620 con 0x7fe568072ca0 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe55effd700 1 -- 172.21.15.87:0/2282261336 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe56401ac70 con 0x7fe568072ca0 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe55effd700 1 -- 172.21.15.87:0/2282261336 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe564023b80 con 0x7fe568072ca0 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe570007700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56810dab0 con 0x7fe568072ca0 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.568+0000 7fe570007700 1 librados: init done 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.569+0000 7fe55effd700 1 -- 172.21.15.87:0/2282261336 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7fe56402c3f0 con 0x7fe568072ca0 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.569+0000 7fe55effd700 1 --2- 172.21.15.87:0/2282261336 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe554033fd0 0x7fe554036490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.569+0000 7fe55effd700 1 -- 172.21.15.87:0/2282261336 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7fe564045830 con 0x7fe568072ca0 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.569+0000 7fe55effd700 10 client.14142.objecter ms_dispatch 0x7fe568001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.569+0000 7fe55effd700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.569+0000 7fe55effd700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:14.573 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.569+0000 7fe55effd700 20 client.14142.objecter dump_active .. 0 homeless 2024-02-23T00:22:14.574 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.569+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe568046ce0 con 0x7fe568072ca0 2024-02-23T00:22:14.574 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.569+0000 7fe56d5a2700 1 --2- 172.21.15.87:0/2282261336 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe554033fd0 0x7fe554036490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:14.574 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.570+0000 7fe55effd700 10 client.14142.objecter ms_handle_connect 0x7fe554033fd0 2024-02-23T00:22:14.574 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.570+0000 7fe56d5a2700 1 --2- 172.21.15.87:0/2282261336 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe554033fd0 0x7fe554036490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe558009780 tx=0x7fe558006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:14.579 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.578+0000 7fe55effd700 1 -- 172.21.15.87:0/2282261336 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe56401e070 con 0x7fe568072ca0 2024-02-23T00:22:14.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.758+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fe5681023b0 con 0x7fe568072ca0 2024-02-23T00:22:14.762 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.761+0000 7fe55effd700 1 -- 172.21.15.87:0/2282261336 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fe564023e40 con 0x7fe568072ca0 2024-02-23T00:22:14.766 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 10 librados: watch_flush enter 2024-02-23T00:22:14.766 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 10 librados: watch_flush exit 2024-02-23T00:22:14.767 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 20 client.14142.objecter shutdown clearing up homeless session... 2024-02-23T00:22:14.767 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 10 client.14142.objecter successfully canceled tick 2024-02-23T00:22:14.767 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe554033fd0 msgr2=0x7fe554036490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:14.767 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 --2- 172.21.15.87:0/2282261336 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe554033fd0 0x7fe554036490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe558009780 tx=0x7fe558006cc0).stop 2024-02-23T00:22:14.767 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 msgr2=0x7fe56810cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:14.767 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 --2- 172.21.15.87:0/2282261336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe56810cf30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fe564000c00 tx=0x7fe5640090f0).stop 2024-02-23T00:22:14.767 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 shutdown_connections 2024-02-23T00:22:14.767 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 --2- 172.21.15.87:0/2282261336 >> v2:172.21.15.87:6800/2138166229 conn(0x7fe554033fd0 0x7fe554036490 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0x7fe558009780 tx=0x7fe558006cc0).stop 2024-02-23T00:22:14.767 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 --2- 172.21.15.87:0/2282261336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe568072ca0 0x7fe56810cf30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:14.768 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 >> 172.21.15.87:0/2282261336 conn(0x7fe5680f7800 msgr2=0x7fe5680f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:14.768 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 shutdown_connections 2024-02-23T00:22:14.768 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 -- 172.21.15.87:0/2282261336 wait complete. 2024-02-23T00:22:14.768 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:14.763+0000 7fe570007700 1 librados: shutdown 2024-02-23T00:22:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:14.379010+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.87:0/2864463936' entity='mgr.y' 2024-02-23T00:22:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:14.762437+0000 mon.a (mon.0) 61 : audit [INF] from='client.? 172.21.15.87:0/2282261336' entity='client.admin' 2024-02-23T00:22:15.896 INFO:teuthology.orchestra.run.smithi087.stderr:Enabling the dashboard module... 2024-02-23T00:22:16.634 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.429+0000 7f7d9186b700 1 Processor -- start 2024-02-23T00:22:16.634 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.429+0000 7f7d9186b700 1 -- start start 2024-02-23T00:22:16.634 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.429+0000 7f7d9186b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:16.634 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.429+0000 7f7d9186b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c0730c0 con 0x7f7d8c072ca0 2024-02-23T00:22:16.634 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.429+0000 7f7d8affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:16.634 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.429+0000 7f7d8affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c071240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48668/0 (socket says 172.21.15.87:48668) 2024-02-23T00:22:16.634 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.429+0000 7f7d8affd700 1 -- 172.21.15.87:0/3842594633 learned_addr learned my addr 172.21.15.87:0/3842594633 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.430+0000 7f7d8affd700 1 -- 172.21.15.87:0/3842594633 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8c071730 con 0x7f7d8c072ca0 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.430+0000 7f7d8affd700 1 --2- 172.21.15.87:0/3842594633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c071240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7d80009730 tx=0x7f7d800090f0).ready entity=mon.0 client_cookie=8e348190797cc2ae server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.430+0000 7f7d89ffb700 1 -- 172.21.15.87:0/3842594633 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d8000b580 con 0x7f7d8c072ca0 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.430+0000 7f7d89ffb700 1 -- 172.21.15.87:0/3842594633 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d8000bbb0 con 0x7f7d8c072ca0 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.430+0000 7f7d89ffb700 1 -- 172.21.15.87:0/3842594633 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d80011bc0 con 0x7f7d8c072ca0 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.430+0000 7f7d9186b700 1 -- 172.21.15.87:0/3842594633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 msgr2=0x7f7d8c071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.430+0000 7f7d9186b700 1 --2- 172.21.15.87:0/3842594633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c071240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7d80009730 tx=0x7f7d800090f0).stop 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 -- 172.21.15.87:0/3842594633 shutdown_connections 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 --2- 172.21.15.87:0/3842594633 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c071240 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 -- 172.21.15.87:0/3842594633 >> 172.21.15.87:0/3842594633 conn(0x7f7d8c0f7800 msgr2=0x7f7d8c0f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 -- 172.21.15.87:0/3842594633 shutdown_connections 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 -- 172.21.15.87:0/3842594633 wait complete. 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 librados: starting msgr at 2024-02-23T00:22:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 librados: starting objecter 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 Processor -- start 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 -- start start 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 librados: setting wanted keys 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 librados: calling monclient init 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c10cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.431+0000 7f7d9186b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c10d420 con 0x7f7d8c072ca0 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d8affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c10cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d8affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c10cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48670/0 (socket says 172.21.15.87:48670) 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d8affd700 1 -- 172.21.15.87:0/962108876 learned_addr learned my addr 172.21.15.87:0/962108876 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d8affd700 1 -- 172.21.15.87:0/962108876 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d80009370 con 0x7f7d8c072ca0 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d8affd700 1 --2- 172.21.15.87:0/962108876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c10cf30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f7d80011a00 tx=0x7f7d800090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d73fff700 10 client.?.objecter ms_handle_connect 0x7f7d8c072ca0 2024-02-23T00:22:16.636 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d73fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d73fff700 1 -- 172.21.15.87:0/962108876 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d8000be70 con 0x7f7d8c072ca0 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8c10d620 con 0x7f7d8c072ca0 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d73fff700 1 -- 172.21.15.87:0/962108876 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d8001ab10 con 0x7f7d8c072ca0 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d73fff700 1 -- 172.21.15.87:0/962108876 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d80023b20 con 0x7f7d8c072ca0 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d9186b700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8c10dab0 con 0x7f7d8c072ca0 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.432+0000 7f7d9186b700 1 librados: init done 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d73fff700 1 -- 172.21.15.87:0/962108876 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 8) v1 ==== 38582+0+0 (secure 0 0 0) 0x7f7d8002c3f0 con 0x7f7d8c072ca0 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d73fff700 1 --2- 172.21.15.87:0/962108876 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7d74033fe0 0x7f7d740364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8c046ce0 con 0x7f7d8c072ca0 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d73fff700 1 -- 172.21.15.87:0/962108876 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f7d800448d0 con 0x7f7d8c072ca0 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d73fff700 10 client.14144.objecter ms_dispatch 0x7f7d8c001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d73fff700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:16.637 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d73fff700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:16.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d73fff700 20 client.14144.objecter dump_active .. 0 homeless 2024-02-23T00:22:16.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d8a7fc700 1 --2- 172.21.15.87:0/962108876 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7d74033fe0 0x7f7d740364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:16.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d73fff700 10 client.14144.objecter ms_handle_connect 0x7f7d74033fe0 2024-02-23T00:22:16.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.433+0000 7f7d8a7fc700 1 --2- 172.21.15.87:0/962108876 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7d74033fe0 0x7f7d740364a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7d7c009780 tx=0x7f7d7c006cc0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:16.638 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.438+0000 7f7d73fff700 1 -- 172.21.15.87:0/962108876 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7d8001e030 con 0x7f7d8c072ca0 2024-02-23T00:22:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.644+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f7d8c1023b0 con 0x7f7d8c072ca0 2024-02-23T00:22:16.698 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.696+0000 7f7d73fff700 1 -- 172.21.15.87:0/962108876 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 9) v1 ==== 38595+0+0 (secure 0 0 0) 0x7f7d80023410 con 0x7f7d8c072ca0 2024-02-23T00:22:16.700 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.698+0000 7f7d73fff700 1 -- 172.21.15.87:0/962108876 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f7d800219c0 con 0x7f7d8c072ca0 2024-02-23T00:22:16.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 10 librados: watch_flush enter 2024-02-23T00:22:16.704 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 10 librados: watch_flush exit 2024-02-23T00:22:16.705 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 20 client.14144.objecter shutdown clearing up homeless session... 2024-02-23T00:22:16.705 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 10 client.14144.objecter successfully canceled tick 2024-02-23T00:22:16.705 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7d74033fe0 msgr2=0x7f7d740364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:16.705 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 1 --2- 172.21.15.87:0/962108876 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7d74033fe0 0x7f7d740364a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7d7c009780 tx=0x7f7d7c006cc0).stop 2024-02-23T00:22:16.705 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 msgr2=0x7f7d8c10cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:16.705 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 1 --2- 172.21.15.87:0/962108876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c10cf30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f7d80011a00 tx=0x7f7d800090f0).stop 2024-02-23T00:22:16.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 shutdown_connections 2024-02-23T00:22:16.706 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 1 --2- 172.21.15.87:0/962108876 >> v2:172.21.15.87:6800/2138166229 conn(0x7f7d74033fe0 0x7f7d740364a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:16.707 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 1 --2- 172.21.15.87:0/962108876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7d8c072ca0 0x7f7d8c10cf30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:16.707 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 >> 172.21.15.87:0/962108876 conn(0x7f7d8c0f7800 msgr2=0x7f7d8c0f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:16.707 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.700+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 shutdown_connections 2024-02-23T00:22:16.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.701+0000 7f7d9186b700 1 -- 172.21.15.87:0/962108876 wait complete. 2024-02-23T00:22:16.708 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:16.701+0000 7f7d9186b700 1 librados: shutdown 2024-02-23T00:22:17.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:16.646264+0000 mon.a (mon.0) 62 : audit [INF] from='client.? 172.21.15.87:0/962108876' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-23T00:22:17.066 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ignoring --setuser ceph since I am not root 2024-02-23T00:22:17.066 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ignoring --setgroup ceph since I am not root 2024-02-23T00:22:17.066 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:16.854+0000 7fe4cc161680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-23T00:22:17.066 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:16.934+0000 7fe4cc161680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-23T00:22:17.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:17.297+0000 7fe4cc161680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-23T00:22:18.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.022+0000 7f5bd5854700 1 Processor -- start 2024-02-23T00:22:18.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.022+0000 7f5bd5854700 1 -- start start 2024-02-23T00:22:18.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.022+0000 7f5bd5854700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd006a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:18.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.022+0000 7f5bd5854700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd006ab80 con 0x7f5bd006a1e0 2024-02-23T00:22:18.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.022+0000 7f5bceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd006a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:18.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.022+0000 7f5bceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd006a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48706/0 (socket says 172.21.15.87:48706) 2024-02-23T00:22:18.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.022+0000 7f5bceffd700 1 -- 172.21.15.87:0/203730159 learned_addr learned my addr 172.21.15.87:0/203730159 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:18.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.023+0000 7f5bceffd700 1 -- 172.21.15.87:0/203730159 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd006acc0 con 0x7f5bd006a1e0 2024-02-23T00:22:18.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.023+0000 7f5bceffd700 1 --2- 172.21.15.87:0/203730159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd006a600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5bc0009fd0 tx=0x7f5bc0009a30).ready entity=mon.0 client_cookie=e90b83044161e77 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.023+0000 7f5bce7fc700 1 -- 172.21.15.87:0/203730159 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bc000e070 con 0x7f5bd006a1e0 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.023+0000 7f5bce7fc700 1 -- 172.21.15.87:0/203730159 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bc0007da0 con 0x7f5bd006a1e0 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.023+0000 7f5bce7fc700 1 -- 172.21.15.87:0/203730159 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bc0013420 con 0x7f5bd006a1e0 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.024+0000 7f5bd5854700 1 -- 172.21.15.87:0/203730159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 msgr2=0x7f5bd006a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.024+0000 7f5bd5854700 1 --2- 172.21.15.87:0/203730159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd006a600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5bc0009fd0 tx=0x7f5bc0009a30).stop 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.024+0000 7f5bd5854700 1 -- 172.21.15.87:0/203730159 shutdown_connections 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.024+0000 7f5bd5854700 1 --2- 172.21.15.87:0/203730159 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd006a600 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.024+0000 7f5bd5854700 1 -- 172.21.15.87:0/203730159 >> 172.21.15.87:0/203730159 conn(0x7f5bd0063ef0 msgr2=0x7f5bd0066330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.024+0000 7f5bd5854700 1 -- 172.21.15.87:0/203730159 shutdown_connections 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.024+0000 7f5bd5854700 1 -- 172.21.15.87:0/203730159 wait complete. 2024-02-23T00:22:18.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.025+0000 7f5bd5854700 1 librados: starting msgr at 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.025+0000 7f5bd5854700 1 librados: starting objecter 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.025+0000 7f5bd5854700 1 Processor -- start 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.025+0000 7f5bd5854700 1 -- start start 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.025+0000 7f5bd5854700 1 librados: setting wanted keys 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.025+0000 7f5bd5854700 1 librados: calling monclient init 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.025+0000 7f5bd5854700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd0107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.025+0000 7f5bd5854700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd0107840 con 0x7f5bd006a1e0 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.025+0000 7f5bceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd0107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd0107350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48722/0 (socket says 172.21.15.87:48722) 2024-02-23T00:22:18.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bceffd700 1 -- 172.21.15.87:0/3184388859 learned_addr learned my addr 172.21.15.87:0/3184388859 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:18.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bceffd700 1 -- 172.21.15.87:0/3184388859 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc0009c90 con 0x7f5bd006a1e0 2024-02-23T00:22:18.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bceffd700 1 --2- 172.21.15.87:0/3184388859 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd0107350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5bc0007f50 tx=0x7f5bc0009a30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:18.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bccff9700 10 client.?.objecter ms_handle_connect 0x7f5bd006a1e0 2024-02-23T00:22:18.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bccff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:18.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bccff9700 1 -- 172.21.15.87:0/3184388859 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bc000e030 con 0x7f5bd006a1e0 2024-02-23T00:22:18.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd0107a40 con 0x7f5bd006a1e0 2024-02-23T00:22:18.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bd5854700 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:18.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd0107df0 con 0x7f5bd006a1e0 2024-02-23T00:22:18.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bd5854700 1 librados: init done 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bccff9700 1 -- 172.21.15.87:0/3184388859 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bc0009280 con 0x7f5bd006a1e0 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.026+0000 7f5bccff9700 1 -- 172.21.15.87:0/3184388859 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bc001b720 con 0x7f5bd006a1e0 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.027+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb0004f50 con 0x7f5bd006a1e0 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.028+0000 7f5bccff9700 1 -- 172.21.15.87:0/3184388859 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 9) v1 ==== 38595+0+0 (secure 0 0 0) 0x7f5bc001b8e0 con 0x7f5bd006a1e0 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.028+0000 7f5bccff9700 1 --2- 172.21.15.87:0/3184388859 >> v2:172.21.15.87:6800/2138166229 conn(0x7f5bbc0340c0 0x7f5bbc036580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.028+0000 7f5bccff9700 1 -- 172.21.15.87:0/3184388859 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f5bc00445c0 con 0x7f5bd006a1e0 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.028+0000 7f5bccff9700 10 client.14148.objecter ms_dispatch 0x7f5bd0001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.028+0000 7f5bccff9700 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.028+0000 7f5bccff9700 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:18.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.028+0000 7f5bccff9700 20 client.14148.objecter dump_active .. 0 homeless 2024-02-23T00:22:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.028+0000 7f5bc7fff700 1 -- 172.21.15.87:0/3184388859 >> v2:172.21.15.87:6800/2138166229 conn(0x7f5bbc0340c0 msgr2=0x7f5bbc036580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/2138166229 2024-02-23T00:22:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.028+0000 7f5bc7fff700 1 --2- 172.21.15.87:0/3184388859 >> v2:172.21.15.87:6800/2138166229 conn(0x7f5bbc0340c0 0x7f5bbc036580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:22:18.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.036+0000 7f5bccff9700 1 -- 172.21.15.87:0/3184388859 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5bc0011d40 con 0x7f5bd006a1e0 2024-02-23T00:22:18.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:16.699557+0000 mon.a (mon.0) 63 : audit [INF] from='client.? 172.21.15.87:0/962108876' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-23T00:22:18.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:16.699599+0000 mon.a (mon.0) 64 : cluster [DBG] mgrmap e9: y(active, since 18s) 2024-02-23T00:22:18.231 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.229+0000 7f5bc7fff700 1 -- 172.21.15.87:0/3184388859 >> v2:172.21.15.87:6800/2138166229 conn(0x7f5bbc0340c0 msgr2=0x7f5bbc036580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/2138166229 2024-02-23T00:22:18.231 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.229+0000 7f5bc7fff700 1 --2- 172.21.15.87:0/3184388859 >> v2:172.21.15.87:6800/2138166229 conn(0x7f5bbc0340c0 0x7f5bbc036580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-23T00:22:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.268+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f5bb0005a10 con 0x7f5bd006a1e0 2024-02-23T00:22:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.268+0000 7f5bccff9700 1 -- 172.21.15.87:0/3184388859 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f5bc0011b60 con 0x7f5bd006a1e0 2024-02-23T00:22:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-23T00:22:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-23T00:22:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "available": true, 2024-02-23T00:22:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-23T00:22:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-23T00:22:18.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-23T00:22:18.272 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 10 librados: watch_flush enter 2024-02-23T00:22:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 10 librados: watch_flush exit 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 20 client.14148.objecter shutdown clearing up homeless session... 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 10 client.14148.objecter successfully canceled tick 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 >> v2:172.21.15.87:6800/2138166229 conn(0x7f5bbc0340c0 msgr2=0x7f5bbc036580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 1 --2- 172.21.15.87:0/3184388859 >> v2:172.21.15.87:6800/2138166229 conn(0x7f5bbc0340c0 0x7f5bbc036580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 msgr2=0x7f5bd0107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 1 --2- 172.21.15.87:0/3184388859 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd0107350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5bc0007f50 tx=0x7f5bc0009a30).stop 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 shutdown_connections 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 1 --2- 172.21.15.87:0/3184388859 >> v2:172.21.15.87:6800/2138166229 conn(0x7f5bbc0340c0 0x7f5bbc036580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 1 --2- 172.21.15.87:0/3184388859 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5bd006a1e0 0x7f5bd0107350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.270+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 >> 172.21.15.87:0/3184388859 conn(0x7f5bd0063ef0 msgr2=0x7f5bd0066260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.271+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 shutdown_connections 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.271+0000 7f5bd5854700 1 -- 172.21.15.87:0/3184388859 wait complete. 2024-02-23T00:22:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:18.271+0000 7f5bd5854700 1 librados: shutdown 2024-02-23T00:22:18.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:18.070+0000 7fe4cc161680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-23T00:22:18.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:18.154+0000 7fe4cc161680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-23T00:22:18.665 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:18.375+0000 7fe4cc161680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-23T00:22:18.968 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for the mgr to restart... 2024-02-23T00:22:18.968 INFO:teuthology.orchestra.run.smithi087.stderr:Waiting for mgr epoch 9... 2024-02-23T00:22:18.970 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:18.270139+0000 mon.a (mon.0) 65 : audit [DBG] from='client.? 172.21.15.87:0/3184388859' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-23T00:22:19.543 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:19.330+0000 7fe4cc161680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-23T00:22:19.543 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:19.459+0000 7fe4cc161680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.585+0000 7f310c12f700 1 Processor -- start 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.585+0000 7f310c12f700 1 -- start start 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.585+0000 7f310c12f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f31040fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.585+0000 7f310c12f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31040ff190 con 0x7f3104070cc0 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.585+0000 7f3109ecb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f31040fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.585+0000 7f3109ecb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f31040fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48738/0 (socket says 172.21.15.87:48738) 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.585+0000 7f3109ecb700 1 -- 172.21.15.87:0/1620497728 learned_addr learned my addr 172.21.15.87:0/1620497728 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.585+0000 7f3109ecb700 1 -- 172.21.15.87:0/1620497728 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31040ff2d0 con 0x7f3104070cc0 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f3109ecb700 1 --2- 172.21.15.87:0/1620497728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f31040fec10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f3100009730 tx=0x7f31000090f0).ready entity=mon.0 client_cookie=9ed49d40b5f02641 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f3108ec9700 1 -- 172.21.15.87:0/1620497728 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f310000b580 con 0x7f3104070cc0 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f3108ec9700 1 -- 172.21.15.87:0/1620497728 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f310000bbb0 con 0x7f3104070cc0 2024-02-23T00:22:19.590 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f3108ec9700 1 -- 172.21.15.87:0/1620497728 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3100011bc0 con 0x7f3104070cc0 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f310c12f700 1 -- 172.21.15.87:0/1620497728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 msgr2=0x7f31040fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f310c12f700 1 --2- 172.21.15.87:0/1620497728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f31040fec10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f3100009730 tx=0x7f31000090f0).stop 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f310c12f700 1 -- 172.21.15.87:0/1620497728 shutdown_connections 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f310c12f700 1 --2- 172.21.15.87:0/1620497728 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f31040fec10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f310c12f700 1 -- 172.21.15.87:0/1620497728 >> 172.21.15.87:0/1620497728 conn(0x7f31040f6670 msgr2=0x7f31040f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f310c12f700 1 -- 172.21.15.87:0/1620497728 shutdown_connections 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.586+0000 7f310c12f700 1 -- 172.21.15.87:0/1620497728 wait complete. 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.587+0000 7f310c12f700 1 librados: starting msgr at 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.587+0000 7f310c12f700 1 librados: starting objecter 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.587+0000 7f310c12f700 1 Processor -- start 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.587+0000 7f310c12f700 1 -- start start 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.587+0000 7f310c12f700 1 librados: setting wanted keys 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.587+0000 7f310c12f700 1 librados: calling monclient init 2024-02-23T00:22:19.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.587+0000 7f310c12f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f310410ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:19.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f310c12f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310410b270 con 0x7f3104070cc0 2024-02-23T00:22:19.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f3109ecb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f310410ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:19.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f3109ecb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f310410ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48752/0 (socket says 172.21.15.87:48752) 2024-02-23T00:22:19.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f3109ecb700 1 -- 172.21.15.87:0/3398722291 learned_addr learned my addr 172.21.15.87:0/3398722291 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:19.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f3109ecb700 1 -- 172.21.15.87:0/3398722291 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3100009370 con 0x7f3104070cc0 2024-02-23T00:22:19.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f3109ecb700 1 --2- 172.21.15.87:0/3398722291 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f310410ad80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f3100012fd0 tx=0x7f31000090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:19.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f30faffd700 10 client.?.objecter ms_handle_connect 0x7f3104070cc0 2024-02-23T00:22:19.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f30faffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:19.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f310001a640 con 0x7f3104070cc0 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f310c12f700 1 -- 172.21.15.87:0/3398722291 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310410b470 con 0x7f3104070cc0 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f310001ac70 con 0x7f3104070cc0 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3100023c00 con 0x7f3104070cc0 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f310c12f700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f310c12f700 1 -- 172.21.15.87:0/3398722291 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310410c0b0 con 0x7f3104070cc0 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.588+0000 7f310c12f700 1 librados: init done 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 9) v1 ==== 38595+0+0 (secure 0 0 0) 0x7f310001a7a0 con 0x7f3104070cc0 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f30faffd700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 0x7f30f00364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 --> v2:172.21.15.87:6800/2138166229 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30f0036b70 con 0x7f30f0034030 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (secure 0 0 0) 0x7f3100044970 con 0x7f3104070cc0 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f31096ca700 1 -- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 msgr2=0x7f30f00364f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/2138166229 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f30faffd700 10 client.14150.objecter ms_dispatch 0x7f3104001550 osd_map(2..2 src has 1..2) v4 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f30faffd700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f30faffd700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-02-23T00:22:19.594 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f31096ca700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 0x7f30f00364f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:22:19.595 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.589+0000 7f30faffd700 20 client.14150.objecter dump_active .. 0 homeless 2024-02-23T00:22:19.792 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.790+0000 7f31096ca700 1 -- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 msgr2=0x7f30f00364f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/2138166229 2024-02-23T00:22:19.792 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:19.790+0000 7f31096ca700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 0x7f30f00364f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-23T00:22:19.832 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:19.541+0000 7fe4cc161680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-23T00:22:19.832 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:19.636+0000 7fe4cc161680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-23T00:22:20.108 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:19.830+0000 7fe4cc161680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-23T00:22:20.193 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:20.191+0000 7f31096ca700 1 -- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 msgr2=0x7f30f00364f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/2138166229 2024-02-23T00:22:20.193 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:20.191+0000 7f31096ca700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 0x7f30f00364f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-23T00:22:20.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:20.106+0000 7fe4cc161680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-23T00:22:20.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:20.203+0000 7fe4cc161680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-23T00:22:21.004 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:20.992+0000 7f31096ca700 1 -- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 msgr2=0x7f30f00364f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/2138166229 2024-02-23T00:22:21.004 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:20.992+0000 7f31096ca700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 0x7f30f00364f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-23T00:22:21.259 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:20.912+0000 7fe4cc161680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-23T00:22:21.259 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:20.993+0000 7fe4cc161680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-23T00:22:21.259 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:21.085+0000 7fe4cc161680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-23T00:22:21.779 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:21.257+0000 7fe4cc161680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-23T00:22:21.780 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:21.511+0000 7fe4cc161680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-23T00:22:21.780 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:21.640+0000 7fe4cc161680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-23T00:22:22.034 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:21.777+0000 7fe4cc161680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:22:22.034 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:21.949+0000 7fe4cc161680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-23T00:22:22.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.047+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mgrmap(e 10) v1 ==== 38454+0+0 (secure 0 0 0) 0x7f310002c3f0 con 0x7f3104070cc0 2024-02-23T00:22:22.049 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.047+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 msgr2=0x7f30f00364f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:22:22.050 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:22.047+0000 7f30faffd700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 0x7f30f00364f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:22:22.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:22:22.032+0000 7fe4cc161680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-23T00:22:22.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [79B blob data] 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:22.047059+0000 mon.a (mon.0) 66 : cluster [INF] Active manager daemon y restarted 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:22.047271+0000 mon.a (mon.0) 67 : cluster [INF] Activating manager daemon y 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:22.050045+0000 mon.a (mon.0) 68 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:22.050172+0000 mon.a (mon.0) 69 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00298453s) 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:22.050944+0000 mon.a (mon.0) 70 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:22.051558+0000 mon.a (mon.0) 71 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:22.052282+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:22.052488+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:22.052622+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:22:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:22.054442+0000 mon.a (mon.0) 75 : cluster [INF] Manager daemon y is now available 2024-02-23T00:22:23.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.048+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 11) v1 ==== 38489+0+0 (secure 0 0 0) 0x7f310002bc50 con 0x7f3104070cc0 2024-02-23T00:22:23.051 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.048+0000 7f30faffd700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/1121018082 conn(0x7f30f0037bd0 0x7f30f0039fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:23.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.048+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 --> v2:172.21.15.87:6800/1121018082 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30f0036b70 con 0x7f30f0037bd0 2024-02-23T00:22:23.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.049+0000 7f31096ca700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/1121018082 conn(0x7f30f0037bd0 0x7f30f0039fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:23.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.049+0000 7f30faffd700 10 client.14150.objecter ms_handle_connect 0x7f30f0037bd0 2024-02-23T00:22:23.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.049+0000 7f31096ca700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/1121018082 conn(0x7f30f0037bd0 0x7f30f0039fc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f30f4003850 tx=0x7f30f4006e60).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:23.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.050+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f30f0036b70 con 0x7f30f0037bd0 2024-02-23T00:22:23.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 -- 172.21.15.87:0/3398722291 --> v2:172.21.15.87:6800/1121018082 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f310410b900 con 0x7f30f0037bd0 2024-02-23T00:22:23.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f30faffd700 1 -- 172.21.15.87:0/3398722291 <== mgr.14146 v2:172.21.15.87:6800/1121018082 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f310410b900 con 0x7f30f0037bd0 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: { 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: "initialized": true 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: } 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 10 librados: watch_flush enter 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 10 librados: watch_flush exit 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 20 client.14150.objecter shutdown clearing up homeless session... 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 10 client.14150.objecter successfully canceled tick 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 -- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/1121018082 conn(0x7f30f0037bd0 msgr2=0x7f30f0039fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:23.060 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/1121018082 conn(0x7f30f0037bd0 0x7f30f0039fc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f30f4003850 tx=0x7f30f4006e60).stop 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 -- 172.21.15.87:0/3398722291 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 msgr2=0x7f310410ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 --2- 172.21.15.87:0/3398722291 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f310410ad80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f3100012fd0 tx=0x7f31000090f0).stop 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 -- 172.21.15.87:0/3398722291 shutdown_connections 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/1121018082 conn(0x7f30f0037bd0 0x7f30f0039fc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 --2- 172.21.15.87:0/3398722291 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3104070cc0 0x7f310410ad80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 --2- 172.21.15.87:0/3398722291 >> v2:172.21.15.87:6800/2138166229 conn(0x7f30f0034030 0x7f30f00364f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 -- 172.21.15.87:0/3398722291 >> 172.21.15.87:0/3398722291 conn(0x7f31040f6670 msgr2=0x7f31040f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 -- 172.21.15.87:0/3398722291 shutdown_connections 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 -- 172.21.15.87:0/3398722291 wait complete. 2024-02-23T00:22:23.061 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:23.056+0000 7f310c12f700 1 librados: shutdown 2024-02-23T00:22:23.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:22.147790+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:23.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:22.149360+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:23.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:22.172737+0000 mon.a (mon.0) 78 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:22:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:22.175691+0000 mon.a (mon.0) 79 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-23T00:22:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:23.051976+0000 mon.a (mon.0) 80 : cluster [DBG] mgrmap e11: y(active, since 1.00479s) 2024-02-23T00:22:24.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:23.051363+0000 mgr.y (mgr.14146) 1 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-23T00:22:24.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:23.057561+0000 mgr.y (mgr.14146) 2 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-23T00:22:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:mgr epoch 9 is available 2024-02-23T00:22:24.772 INFO:teuthology.orchestra.run.smithi087.stderr:Generating a dashboard self-signed certificate... 2024-02-23T00:22:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.316+0000 7f858a688700 1 Processor -- start 2024-02-23T00:22:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.316+0000 7f858a688700 1 -- start start 2024-02-23T00:22:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.316+0000 7f858a688700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.316+0000 7f858a688700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85840fad20 con 0x7f85840fa410 2024-02-23T00:22:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.317+0000 7f8583fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.317+0000 7f8583fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34322/0 (socket says 172.21.15.87:34322) 2024-02-23T00:22:25.324 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.317+0000 7f8583fff700 1 -- 172.21.15.87:0/1787571726 learned_addr learned my addr 172.21.15.87:0/1787571726 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.317+0000 7f8583fff700 1 -- 172.21.15.87:0/1787571726 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85840fae60 con 0x7f85840fa410 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.317+0000 7f8583fff700 1 --2- 172.21.15.87:0/1787571726 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fa830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f8574009730 tx=0x7f85740090f0).ready entity=mon.0 client_cookie=c38b7daae7a3ebda server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.317+0000 7f85837fe700 1 -- 172.21.15.87:0/1787571726 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f857400b580 con 0x7f85840fa410 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.317+0000 7f85837fe700 1 -- 172.21.15.87:0/1787571726 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f857400bbb0 con 0x7f85840fa410 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.317+0000 7f85837fe700 1 -- 172.21.15.87:0/1787571726 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8574011bc0 con 0x7f85840fa410 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.318+0000 7f858a688700 1 -- 172.21.15.87:0/1787571726 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 msgr2=0x7f85840fa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.318+0000 7f858a688700 1 --2- 172.21.15.87:0/1787571726 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fa830 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f8574009730 tx=0x7f85740090f0).stop 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.318+0000 7f858a688700 1 -- 172.21.15.87:0/1787571726 shutdown_connections 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.318+0000 7f858a688700 1 --2- 172.21.15.87:0/1787571726 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fa830 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.318+0000 7f858a688700 1 -- 172.21.15.87:0/1787571726 >> 172.21.15.87:0/1787571726 conn(0x7f85840f5f10 msgr2=0x7f85840f8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.318+0000 7f858a688700 1 -- 172.21.15.87:0/1787571726 shutdown_connections 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.318+0000 7f858a688700 1 -- 172.21.15.87:0/1787571726 wait complete. 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.318+0000 7f858a688700 1 librados: starting msgr at 2024-02-23T00:22:25.325 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.318+0000 7f858a688700 1 librados: starting objecter 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f858a688700 1 Processor -- start 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f858a688700 1 -- start start 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f858a688700 1 librados: setting wanted keys 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f858a688700 1 librados: calling monclient init 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f858a688700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f858a688700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85840fb480 con 0x7f85840fa410 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f8583fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f8583fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34332/0 (socket says 172.21.15.87:34332) 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f8583fff700 1 -- 172.21.15.87:0/3445328010 learned_addr learned my addr 172.21.15.87:0/3445328010 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f8583fff700 1 -- 172.21.15.87:0/3445328010 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8574009370 con 0x7f85840fa410 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f8583fff700 1 --2- 172.21.15.87:0/3445328010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fd000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8574000c00 tx=0x7f85740090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.319+0000 7f8581ffb700 10 client.?.objecter ms_handle_connect 0x7f85840fa410 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.320+0000 7f8581ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:25.326 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.320+0000 7f8581ffb700 1 -- 172.21.15.87:0/3445328010 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f857401a640 con 0x7f85840fa410 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.320+0000 7f8581ffb700 1 -- 172.21.15.87:0/3445328010 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f857401ac70 con 0x7f85840fa410 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.320+0000 7f8581ffb700 1 -- 172.21.15.87:0/3445328010 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8574023b80 con 0x7f85840fa410 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.320+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85840fb620 con 0x7f85840fa410 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.320+0000 7f858a688700 10 client.14156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.320+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85840fbab0 con 0x7f85840fa410 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.320+0000 7f858a688700 1 librados: init done 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.321+0000 7f8581ffb700 1 -- 172.21.15.87:0/3445328010 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) v1 ==== 38595+0+0 (secure 0 0 0) 0x7f857401a7a0 con 0x7f85840fa410 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.321+0000 7f8581ffb700 1 --2- 172.21.15.87:0/3445328010 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8570034070 0x7f8570036530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.321+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858405a310 con 0x7f85840fa410 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.321+0000 7f8581ffb700 1 -- 172.21.15.87:0/3445328010 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f8574044d70 con 0x7f85840fa410 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.321+0000 7f8581ffb700 10 client.14156.objecter ms_dispatch 0x7f8584001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.321+0000 7f8581ffb700 3 client.14156.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:22:25.327 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.321+0000 7f8581ffb700 3 client.14156.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:22:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.321+0000 7f8581ffb700 20 client.14156.objecter dump_active .. 0 homeless 2024-02-23T00:22:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.321+0000 7f857bfff700 1 --2- 172.21.15.87:0/3445328010 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8570034070 0x7f8570036530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.322+0000 7f8581ffb700 10 client.14156.objecter ms_handle_connect 0x7f8570034070 2024-02-23T00:22:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.322+0000 7f857bfff700 1 --2- 172.21.15.87:0/3445328010 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8570034070 0x7f8570036530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f856c009780 tx=0x7f856c006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.326+0000 7f8581ffb700 1 -- 172.21.15.87:0/3445328010 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f85740223c0 con 0x7f85840fa410 2024-02-23T00:22:25.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:24.111108+0000 mon.a (mon.0) 81 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-02-23T00:22:25.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:24.133809+0000 mon.a (mon.0) 82 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:25.498 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.496+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f8584104990 con 0x7f8570034070 2024-02-23T00:22:25.595 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.593+0000 7f8581ffb700 1 -- 172.21.15.87:0/3445328010 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f8584104990 con 0x7f8570034070 2024-02-23T00:22:25.595 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-23T00:22:25.599 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 10 librados: watch_flush enter 2024-02-23T00:22:25.599 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 10 librados: watch_flush exit 2024-02-23T00:22:25.599 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 20 client.14156.objecter shutdown clearing up homeless session... 2024-02-23T00:22:25.599 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 10 client.14156.objecter successfully canceled tick 2024-02-23T00:22:25.599 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8570034070 msgr2=0x7f8570036530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:25.599 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 --2- 172.21.15.87:0/3445328010 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8570034070 0x7f8570036530 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f856c009780 tx=0x7f856c006cc0).stop 2024-02-23T00:22:25.599 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 msgr2=0x7f85840fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:25.600 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 --2- 172.21.15.87:0/3445328010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fd000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8574000c00 tx=0x7f85740090f0).stop 2024-02-23T00:22:25.600 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 shutdown_connections 2024-02-23T00:22:25.600 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 --2- 172.21.15.87:0/3445328010 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8570034070 0x7f8570036530 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:25.600 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 --2- 172.21.15.87:0/3445328010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85840fa410 0x7f85840fd000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:25.600 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 >> 172.21.15.87:0/3445328010 conn(0x7f85840f5f10 msgr2=0x7f85840f7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:25.600 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 shutdown_connections 2024-02-23T00:22:25.600 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 -- 172.21.15.87:0/3445328010 wait complete. 2024-02-23T00:22:25.600 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:25.596+0000 7f858a688700 1 librados: shutdown 2024-02-23T00:22:26.742 INFO:teuthology.orchestra.run.smithi087.stderr:Creating initial admin user... 2024-02-23T00:22:27.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:25.497935+0000 mgr.y (mgr.14146) 3 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:27.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:25.591816+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:27.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:25.594243+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:27.845 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.830+0000 7fb0207b0700 1 Processor -- start 2024-02-23T00:22:27.845 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.830+0000 7fb0207b0700 1 -- start start 2024-02-23T00:22:27.845 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.830+0000 7fb0207b0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb0180fb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:27.845 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.830+0000 7fb0207b0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180fb8f0 con 0x7fb0180faf50 2024-02-23T00:22:27.845 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.831+0000 7fb01e54c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb0180fb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:27.845 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.831+0000 7fb01e54c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb0180fb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34342/0 (socket says 172.21.15.87:34342) 2024-02-23T00:22:27.845 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.831+0000 7fb01e54c700 1 -- 172.21.15.87:0/1776874056 learned_addr learned my addr 172.21.15.87:0/1776874056 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.831+0000 7fb01e54c700 1 -- 172.21.15.87:0/1776874056 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180fba30 con 0x7fb0180faf50 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.831+0000 7fb01e54c700 1 --2- 172.21.15.87:0/1776874056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb0180fb370 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb008009730 tx=0x7fb0080090f0).ready entity=mon.0 client_cookie=75dd5b81d7d1bab9 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.831+0000 7fb01d54a700 1 -- 172.21.15.87:0/1776874056 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb00800b580 con 0x7fb0180faf50 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.831+0000 7fb01d54a700 1 -- 172.21.15.87:0/1776874056 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb00800bbb0 con 0x7fb0180faf50 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.831+0000 7fb01d54a700 1 -- 172.21.15.87:0/1776874056 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb008011bc0 con 0x7fb0180faf50 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 -- 172.21.15.87:0/1776874056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 msgr2=0x7fb0180fb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 --2- 172.21.15.87:0/1776874056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb0180fb370 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb008009730 tx=0x7fb0080090f0).stop 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 -- 172.21.15.87:0/1776874056 shutdown_connections 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 --2- 172.21.15.87:0/1776874056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb0180fb370 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 -- 172.21.15.87:0/1776874056 >> 172.21.15.87:0/1776874056 conn(0x7fb0180f6670 msgr2=0x7fb0180f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 -- 172.21.15.87:0/1776874056 shutdown_connections 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 -- 172.21.15.87:0/1776874056 wait complete. 2024-02-23T00:22:27.846 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 librados: starting msgr at 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 librados: starting objecter 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 Processor -- start 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 -- start start 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 librados: setting wanted keys 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 librados: calling monclient init 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb01810da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.832+0000 7fb0207b0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb018070c60 con 0x7fb0180faf50 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb01e54c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb01810da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb01e54c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb01810da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34358/0 (socket says 172.21.15.87:34358) 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb01e54c700 1 -- 172.21.15.87:0/1730547347 learned_addr learned my addr 172.21.15.87:0/1730547347 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb01e54c700 1 -- 172.21.15.87:0/1730547347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008009370 con 0x7fb0180faf50 2024-02-23T00:22:27.847 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb01e54c700 1 --2- 172.21.15.87:0/1730547347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb01810da50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb008000c00 tx=0x7fb0080090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb00f7fe700 10 client.?.objecter ms_handle_connect 0x7fb0180faf50 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb00f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb00f7fe700 1 -- 172.21.15.87:0/1730547347 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb00801a640 con 0x7fb0180faf50 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb00f7fe700 1 -- 172.21.15.87:0/1730547347 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb00801ac70 con 0x7fb0180faf50 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb018070e60 con 0x7fb0180faf50 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb00f7fe700 1 -- 172.21.15.87:0/1730547347 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb008023b80 con 0x7fb0180faf50 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb0207b0700 10 client.14158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01810e100 con 0x7fb0180faf50 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.833+0000 7fb0207b0700 1 librados: init done 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb00f7fe700 1 -- 172.21.15.87:0/1730547347 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 12) v1 ==== 38595+0+0 (secure 0 0 0) 0x7fb00802c3f0 con 0x7fb0180faf50 2024-02-23T00:22:27.848 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb00f7fe700 1 --2- 172.21.15.87:0/1730547347 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb004034070 0x7fb004036530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb00f7fe700 1 -- 172.21.15.87:0/1730547347 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fb0080450a0 con 0x7fb0180faf50 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb00f7fe700 10 client.14158.objecter ms_dispatch 0x7fb018001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb00f7fe700 3 client.14158.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb00f7fe700 3 client.14158.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb00f7fe700 20 client.14158.objecter dump_active .. 0 homeless 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb01dd4b700 1 --2- 172.21.15.87:0/1730547347 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb004034070 0x7fb004036530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01805a310 con 0x7fb0180faf50 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb01dd4b700 1 --2- 172.21.15.87:0/1730547347 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb004034070 0x7fb004036530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb014009780 tx=0x7fb014006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.834+0000 7fb00f7fe700 10 client.14158.objecter ms_handle_connect 0x7fb004034070 2024-02-23T00:22:27.849 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:27.839+0000 7fb00f7fe700 1 -- 172.21.15.87:0/1730547347 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb00801e070 con 0x7fb0180faf50 2024-02-23T00:22:28.020 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.018+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb018046ce0 con 0x7fb004034070 2024-02-23T00:22:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.122+0000 7fb00f7fe700 1 -- 172.21.15.87:0/1730547347 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7fb00802c6f0 con 0x7fb0180faf50 2024-02-23T00:22:28.240 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.238+0000 7fb00f7fe700 1 -- 172.21.15.87:0/1730547347 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb018046ce0 con 0x7fb004034070 2024-02-23T00:22:28.240 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$CMdBbRdiI2Q9FNNDcxX8pu.Xe63X82fP/JUIjBvJqNlqGcqkO1VPC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708647748, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-23T00:22:28.243 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 10 librados: watch_flush enter 2024-02-23T00:22:28.243 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 10 librados: watch_flush exit 2024-02-23T00:22:28.243 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 20 client.14158.objecter shutdown clearing up homeless session... 2024-02-23T00:22:28.243 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 10 client.14158.objecter successfully canceled tick 2024-02-23T00:22:28.243 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb004034070 msgr2=0x7fb004036530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:28.243 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 --2- 172.21.15.87:0/1730547347 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb004034070 0x7fb004036530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb014009780 tx=0x7fb014006cc0).stop 2024-02-23T00:22:28.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 msgr2=0x7fb01810da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:28.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 --2- 172.21.15.87:0/1730547347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb01810da50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fb008000c00 tx=0x7fb0080090f0).stop 2024-02-23T00:22:28.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 shutdown_connections 2024-02-23T00:22:28.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 --2- 172.21.15.87:0/1730547347 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb004034070 0x7fb004036530 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:28.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 --2- 172.21.15.87:0/1730547347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0180faf50 0x7fb01810da50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:28.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 >> 172.21.15.87:0/1730547347 conn(0x7fb0180f6670 msgr2=0x7fb0180f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:28.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 shutdown_connections 2024-02-23T00:22:28.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 -- 172.21.15.87:0/1730547347 wait complete. 2024-02-23T00:22:28.244 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:28.240+0000 7fb0207b0700 1 librados: shutdown 2024-02-23T00:22:29.020 INFO:teuthology.orchestra.run.smithi087.stderr:Fetching dashboard port number... 2024-02-23T00:22:29.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:28.020863+0000 mgr.y (mgr.14146) 4 : audit [DBG] from='client.14158 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:22:29.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:28.125010+0000 mon.a (mon.0) 85 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-02-23T00:22:29.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:28.239416+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.582+0000 7fa7ddac7700 1 Processor -- start 2024-02-23T00:22:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.582+0000 7fa7ddac7700 1 -- start start 2024-02-23T00:22:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.582+0000 7fa7ddac7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d80fec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:29.587 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.582+0000 7fa7ddac7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d80ff180 con 0x7fa7d8070cc0 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.582+0000 7fa7d77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d80fec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.582+0000 7fa7d77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d80fec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34374/0 (socket says 172.21.15.87:34374) 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.582+0000 7fa7d77fe700 1 -- 172.21.15.87:0/4147074569 learned_addr learned my addr 172.21.15.87:0/4147074569 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.582+0000 7fa7d77fe700 1 -- 172.21.15.87:0/4147074569 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d80ff2c0 con 0x7fa7d8070cc0 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.582+0000 7fa7d77fe700 1 --2- 172.21.15.87:0/4147074569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d80fec00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa7c8009730 tx=0x7fa7c80090f0).ready entity=mon.0 client_cookie=5e94ef498d837e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.583+0000 7fa7d67fc700 1 -- 172.21.15.87:0/4147074569 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7c800b580 con 0x7fa7d8070cc0 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.583+0000 7fa7d67fc700 1 -- 172.21.15.87:0/4147074569 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7c800bbb0 con 0x7fa7d8070cc0 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.583+0000 7fa7d67fc700 1 -- 172.21.15.87:0/4147074569 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7c8011bc0 con 0x7fa7d8070cc0 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.583+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4147074569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 msgr2=0x7fa7d80fec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.583+0000 7fa7ddac7700 1 --2- 172.21.15.87:0/4147074569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d80fec00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa7c8009730 tx=0x7fa7c80090f0).stop 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.584+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4147074569 shutdown_connections 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.584+0000 7fa7ddac7700 1 --2- 172.21.15.87:0/4147074569 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d80fec00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:29.588 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.584+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4147074569 >> 172.21.15.87:0/4147074569 conn(0x7fa7d80f6660 msgr2=0x7fa7d80f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.584+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4147074569 shutdown_connections 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.584+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4147074569 wait complete. 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.584+0000 7fa7ddac7700 1 librados: starting msgr at 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.584+0000 7fa7ddac7700 1 librados: starting objecter 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.584+0000 7fa7ddac7700 1 Processor -- start 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7ddac7700 1 -- start start 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7ddac7700 1 librados: setting wanted keys 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7ddac7700 1 librados: calling monclient init 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7ddac7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d810ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7ddac7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d810b230 con 0x7fa7d8070cc0 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7d77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d810ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7d77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d810ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34382/0 (socket says 172.21.15.87:34382) 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7d77fe700 1 -- 172.21.15.87:0/4174958496 learned_addr learned my addr 172.21.15.87:0/4174958496 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:29.589 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7d77fe700 1 -- 172.21.15.87:0/4174958496 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c8009370 con 0x7fa7d8070cc0 2024-02-23T00:22:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7d77fe700 1 --2- 172.21.15.87:0/4174958496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d810ad40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa7c8012fd0 tx=0x7fa7c80090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7d4f74700 10 client.?.objecter ms_handle_connect 0x7fa7d8070cc0 2024-02-23T00:22:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7d4f74700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.585+0000 7fa7d4f74700 1 -- 172.21.15.87:0/4174958496 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7c801a640 con 0x7fa7d8070cc0 2024-02-23T00:22:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.586+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d810b430 con 0x7fa7d8070cc0 2024-02-23T00:22:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.586+0000 7fa7d4f74700 1 -- 172.21.15.87:0/4174958496 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7c801ac70 con 0x7fa7d8070cc0 2024-02-23T00:22:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.586+0000 7fa7d4f74700 1 -- 172.21.15.87:0/4174958496 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7c8023c00 con 0x7fa7d8070cc0 2024-02-23T00:22:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.586+0000 7fa7ddac7700 10 client.14160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.586+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d810c040 con 0x7fa7d8070cc0 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.586+0000 7fa7ddac7700 1 librados: init done 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.586+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d810b8c0 con 0x7fa7d8070cc0 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d4f74700 1 -- 172.21.15.87:0/4174958496 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7fa7c801a7a0 con 0x7fa7d8070cc0 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d4f74700 1 --2- 172.21.15.87:0/4174958496 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa7c00340d0 0x7fa7c0036590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d4f74700 1 -- 172.21.15.87:0/4174958496 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fa7c8044de0 con 0x7fa7d8070cc0 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d4f74700 10 client.14160.objecter ms_dispatch 0x7fa7d8001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d4f74700 3 client.14160.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d4f74700 3 client.14160.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d4f74700 20 client.14160.objecter dump_active .. 0 homeless 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d6ffd700 1 --2- 172.21.15.87:0/4174958496 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa7c00340d0 0x7fa7c0036590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d4f74700 10 client.14160.objecter ms_handle_connect 0x7fa7c00340d0 2024-02-23T00:22:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.587+0000 7fa7d6ffd700 1 --2- 172.21.15.87:0/4174958496 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa7c00340d0 0x7fa7c0036590 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa7cc009780 tx=0x7fa7cc006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:29.593 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.591+0000 7fa7d4f74700 1 -- 172.21.15.87:0/4174958496 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa7d810b8c0 con 0x7fa7d8070cc0 2024-02-23T00:22:29.756 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.754+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fa7d810b8c0 con 0x7fa7d8070cc0 2024-02-23T00:22:29.756 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.754+0000 7fa7d4f74700 1 -- 172.21.15.87:0/4174958496 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fa7c801e070 con 0x7fa7d8070cc0 2024-02-23T00:22:29.756 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 8443 2024-02-23T00:22:29.759 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 10 librados: watch_flush enter 2024-02-23T00:22:29.759 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 10 librados: watch_flush exit 2024-02-23T00:22:29.759 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 20 client.14160.objecter shutdown clearing up homeless session... 2024-02-23T00:22:29.759 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 10 client.14160.objecter successfully canceled tick 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa7c00340d0 msgr2=0x7fa7c0036590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 --2- 172.21.15.87:0/4174958496 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa7c00340d0 0x7fa7c0036590 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa7cc009780 tx=0x7fa7cc006cc0).stop 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 msgr2=0x7fa7d810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 --2- 172.21.15.87:0/4174958496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d810ad40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa7c8012fd0 tx=0x7fa7c80090f0).stop 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 shutdown_connections 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 --2- 172.21.15.87:0/4174958496 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa7c00340d0 0x7fa7c0036590 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 --2- 172.21.15.87:0/4174958496 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa7d8070cc0 0x7fa7d810ad40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 >> 172.21.15.87:0/4174958496 conn(0x7fa7d80f6660 msgr2=0x7fa7d80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 shutdown_connections 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 -- 172.21.15.87:0/4174958496 wait complete. 2024-02-23T00:22:29.760 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:29.756+0000 7fa7ddac7700 1 librados: shutdown 2024-02-23T00:22:30.448 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:29.755943+0000 mon.a (mon.0) 87 : audit [DBG] from='client.? 172.21.15.87:0/4174958496' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-23T00:22:30.454 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: disabled 2024-02-23T00:22:30.462 INFO:teuthology.orchestra.run.smithi087.stderr:systemctl: inactive 2024-02-23T00:22:30.462 INFO:teuthology.orchestra.run.smithi087.stderr:firewalld.service is not enabled 2024-02-23T00:22:30.463 INFO:teuthology.orchestra.run.smithi087.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-23T00:22:30.466 INFO:teuthology.orchestra.run.smithi087.stderr:Ceph Dashboard is now available at: 2024-02-23T00:22:30.466 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:30.466 INFO:teuthology.orchestra.run.smithi087.stderr: URL: https://smithi087.front.sepia.ceph.com:8443/ 2024-02-23T00:22:30.466 INFO:teuthology.orchestra.run.smithi087.stderr: User: admin 2024-02-23T00:22:30.466 INFO:teuthology.orchestra.run.smithi087.stderr: Password: gy3v853j8x 2024-02-23T00:22:30.466 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:30.466 INFO:teuthology.orchestra.run.smithi087.stderr:Enabling autotune for osd_memory_target 2024-02-23T00:22:31.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.045+0000 7f0dda575700 1 Processor -- start 2024-02-23T00:22:31.052 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.045+0000 7f0dda575700 1 -- start start 2024-02-23T00:22:31.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.045+0000 7f0dda575700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd40fb360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:31.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.045+0000 7f0dda575700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40fb8e0 con 0x7f0dd40faf40 2024-02-23T00:22:31.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.045+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd40fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:31.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.045+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd40fb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34394/0 (socket says 172.21.15.87:34394) 2024-02-23T00:22:31.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.045+0000 7f0dd3fff700 1 -- 172.21.15.87:0/2406234202 learned_addr learned my addr 172.21.15.87:0/2406234202 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:31.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dd3fff700 1 -- 172.21.15.87:0/2406234202 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd40fba20 con 0x7f0dd40faf40 2024-02-23T00:22:31.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dd3fff700 1 --2- 172.21.15.87:0/2406234202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd40fb360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f0dc400b9d0 tx=0x7f0dc400b3b0).ready entity=mon.0 client_cookie=58336f1bd0bbd9d server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:31.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dd2ffd700 1 -- 172.21.15.87:0/2406234202 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc400da70 con 0x7f0dd40faf40 2024-02-23T00:22:31.053 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dd2ffd700 1 -- 172.21.15.87:0/2406234202 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0dc400dbd0 con 0x7f0dd40faf40 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dd2ffd700 1 -- 172.21.15.87:0/2406234202 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc400f410 con 0x7f0dd40faf40 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dda575700 1 -- 172.21.15.87:0/2406234202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 msgr2=0x7f0dd40fb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dda575700 1 --2- 172.21.15.87:0/2406234202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd40fb360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f0dc400b9d0 tx=0x7f0dc400b3b0).stop 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dda575700 1 -- 172.21.15.87:0/2406234202 shutdown_connections 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dda575700 1 --2- 172.21.15.87:0/2406234202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd40fb360 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dda575700 1 -- 172.21.15.87:0/2406234202 >> 172.21.15.87:0/2406234202 conn(0x7f0dd40f6660 msgr2=0x7f0dd40f8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.046+0000 7f0dda575700 1 -- 172.21.15.87:0/2406234202 shutdown_connections 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.047+0000 7f0dda575700 1 -- 172.21.15.87:0/2406234202 wait complete. 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.047+0000 7f0dda575700 1 librados: starting msgr at 2024-02-23T00:22:31.054 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.047+0000 7f0dda575700 1 librados: starting objecter 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.047+0000 7f0dda575700 1 Processor -- start 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.047+0000 7f0dda575700 1 -- start start 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.047+0000 7f0dda575700 1 librados: setting wanted keys 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.047+0000 7f0dda575700 1 librados: calling monclient init 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.047+0000 7f0dda575700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd410cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.047+0000 7f0dda575700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd410d3e0 con 0x7f0dd40faf40 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd410cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd410cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34408/0 (socket says 172.21.15.87:34408) 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd3fff700 1 -- 172.21.15.87:0/82770853 learned_addr learned my addr 172.21.15.87:0/82770853 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:31.055 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd3fff700 1 -- 172.21.15.87:0/82770853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc400b610 con 0x7f0dd40faf40 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd3fff700 1 --2- 172.21.15.87:0/82770853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd410cef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f0dc400f830 tx=0x7f0dc400b3b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd17fa700 10 client.?.objecter ms_handle_connect 0x7f0dd40faf40 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd17fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd17fa700 1 -- 172.21.15.87:0/82770853 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc4010900 con 0x7f0dd40faf40 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd410d5e0 con 0x7f0dd40faf40 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd17fa700 1 -- 172.21.15.87:0/82770853 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0dc4010bd0 con 0x7f0dd40faf40 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dd17fa700 1 -- 172.21.15.87:0/82770853 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dc4022d10 con 0x7f0dd40faf40 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dda575700 10 client.14162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd410da70 con 0x7f0dd40faf40 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.048+0000 7f0dda575700 1 librados: init done 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.049+0000 7f0dd17fa700 1 -- 172.21.15.87:0/82770853 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7f0dc4019450 con 0x7f0dd40faf40 2024-02-23T00:22:31.056 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.049+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd405a310 con 0x7f0dd40faf40 2024-02-23T00:22:31.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.049+0000 7f0dd17fa700 1 --2- 172.21.15.87:0/82770853 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0dbc0340d0 0x7f0dbc036590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:31.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.049+0000 7f0dd17fa700 1 -- 172.21.15.87:0/82770853 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f0dc4045de0 con 0x7f0dd40faf40 2024-02-23T00:22:31.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.049+0000 7f0dd17fa700 10 client.14162.objecter ms_dispatch 0x7f0dd4001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:22:31.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.049+0000 7f0dd17fa700 3 client.14162.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:22:31.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.049+0000 7f0dd17fa700 3 client.14162.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:22:31.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.049+0000 7f0dd17fa700 20 client.14162.objecter dump_active .. 0 homeless 2024-02-23T00:22:31.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.049+0000 7f0dd37fe700 1 --2- 172.21.15.87:0/82770853 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0dbc0340d0 0x7f0dbc036590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:31.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.050+0000 7f0dd17fa700 10 client.14162.objecter ms_handle_connect 0x7f0dbc0340d0 2024-02-23T00:22:31.057 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.050+0000 7f0dd37fe700 1 --2- 172.21.15.87:0/82770853 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0dbc0340d0 0x7f0dbc036590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0dc8009780 tx=0x7f0dc8006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:31.058 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.054+0000 7f0dd17fa700 1 -- 172.21.15.87:0/82770853 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0dc40197a0 con 0x7f0dd40faf40 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.216+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f0dd41024d0 con 0x7f0dd40faf40 2024-02-23T00:22:31.218 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.216+0000 7f0dd17fa700 1 -- 172.21.15.87:0/82770853 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f0dc4018bf0 con 0x7f0dd40faf40 2024-02-23T00:22:31.221 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 10 librados: watch_flush enter 2024-02-23T00:22:31.221 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 10 librados: watch_flush exit 2024-02-23T00:22:31.221 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 20 client.14162.objecter shutdown clearing up homeless session... 2024-02-23T00:22:31.221 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 10 client.14162.objecter successfully canceled tick 2024-02-23T00:22:31.221 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0dbc0340d0 msgr2=0x7f0dbc036590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:31.221 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 --2- 172.21.15.87:0/82770853 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0dbc0340d0 0x7f0dbc036590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0dc8009780 tx=0x7f0dc8006cc0).stop 2024-02-23T00:22:31.221 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 msgr2=0x7f0dd410cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:31.221 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 --2- 172.21.15.87:0/82770853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd410cef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f0dc400f830 tx=0x7f0dc400b3b0).stop 2024-02-23T00:22:31.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 shutdown_connections 2024-02-23T00:22:31.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 --2- 172.21.15.87:0/82770853 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0dbc0340d0 0x7f0dbc036590 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:31.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 --2- 172.21.15.87:0/82770853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0dd40faf40 0x7f0dd410cef0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:31.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 >> 172.21.15.87:0/82770853 conn(0x7f0dd40f6660 msgr2=0x7f0dd40ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:31.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 shutdown_connections 2024-02-23T00:22:31.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 -- 172.21.15.87:0/82770853 wait complete. 2024-02-23T00:22:31.222 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:31.218+0000 7f0dda575700 1 librados: shutdown 2024-02-23T00:22:32.485 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.478+0000 7f567c4bf700 1 Processor -- start 2024-02-23T00:22:32.485 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.478+0000 7f567c4bf700 1 -- start start 2024-02-23T00:22:32.485 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.478+0000 7f567c4bf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f56740fc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.478+0000 7f567c4bf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56740fce60 con 0x7f56740fc4c0 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.478+0000 7f567a25b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f56740fc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.478+0000 7f567a25b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f56740fc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39698/0 (socket says 172.21.15.87:39698) 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.478+0000 7f567a25b700 1 -- 172.21.15.87:0/244858848 learned_addr learned my addr 172.21.15.87:0/244858848 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.479+0000 7f567a25b700 1 -- 172.21.15.87:0/244858848 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56740fcfa0 con 0x7f56740fc4c0 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.479+0000 7f567a25b700 1 --2- 172.21.15.87:0/244858848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f56740fc8e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f5670009730 tx=0x7f56700090f0).ready entity=mon.0 client_cookie=f87d5ddffb4bdc1 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.479+0000 7f5679259700 1 -- 172.21.15.87:0/244858848 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f567000b580 con 0x7f56740fc4c0 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.479+0000 7f5679259700 1 -- 172.21.15.87:0/244858848 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f567000bbb0 con 0x7f56740fc4c0 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.479+0000 7f5679259700 1 -- 172.21.15.87:0/244858848 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5670011bc0 con 0x7f56740fc4c0 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 -- 172.21.15.87:0/244858848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 msgr2=0x7f56740fc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 --2- 172.21.15.87:0/244858848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f56740fc8e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f5670009730 tx=0x7f56700090f0).stop 2024-02-23T00:22:32.486 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 -- 172.21.15.87:0/244858848 shutdown_connections 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 --2- 172.21.15.87:0/244858848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f56740fc8e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 -- 172.21.15.87:0/244858848 >> 172.21.15.87:0/244858848 conn(0x7f56740f5df0 msgr2=0x7f56740f8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 -- 172.21.15.87:0/244858848 shutdown_connections 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 -- 172.21.15.87:0/244858848 wait complete. 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 librados: starting msgr at 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 librados: starting objecter 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.480+0000 7f567c4bf700 1 Processor -- start 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567c4bf700 1 -- start start 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567c4bf700 1 librados: setting wanted keys 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567c4bf700 1 librados: calling monclient init 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567c4bf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f567410d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567c4bf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56740fae40 con 0x7f56740fc4c0 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567a25b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f567410d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567a25b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f567410d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39714/0 (socket says 172.21.15.87:39714) 2024-02-23T00:22:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567a25b700 1 -- 172.21.15.87:0/2484359750 learned_addr learned my addr 172.21.15.87:0/2484359750 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567a25b700 1 -- 172.21.15.87:0/2484359750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5670009370 con 0x7f56740fc4c0 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567a25b700 1 --2- 172.21.15.87:0/2484359750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f567410d9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f567000bf80 tx=0x7f56700090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f566b7fe700 10 client.?.objecter ms_handle_connect 0x7f56740fc4c0 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f566b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f566b7fe700 1 -- 172.21.15.87:0/2484359750 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f567001a640 con 0x7f56740fc4c0 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f566b7fe700 1 -- 172.21.15.87:0/2484359750 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f567001ac70 con 0x7f56740fc4c0 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f566b7fe700 1 -- 172.21.15.87:0/2484359750 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5670023c30 con 0x7f56740fc4c0 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56740fb040 con 0x7f56740fc4c0 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567c4bf700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567410e080 con 0x7f56740fc4c0 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.481+0000 7f567c4bf700 1 librados: init done 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.482+0000 7f566b7fe700 1 -- 172.21.15.87:0/2484359750 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7f567001a7a0 con 0x7f56740fc4c0 2024-02-23T00:22:32.488 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.482+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5674046ce0 con 0x7f56740fc4c0 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.482+0000 7f566b7fe700 1 --2- 172.21.15.87:0/2484359750 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5660034110 0x7f56600365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.483+0000 7f566b7fe700 1 -- 172.21.15.87:0/2484359750 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f5670044e90 con 0x7f56740fc4c0 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.483+0000 7f566b7fe700 10 client.14164.objecter ms_dispatch 0x7f5674001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.483+0000 7f566b7fe700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.483+0000 7f566b7fe700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.483+0000 7f566b7fe700 20 client.14164.objecter dump_active .. 0 homeless 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.483+0000 7f5679a5a700 1 --2- 172.21.15.87:0/2484359750 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5660034110 0x7f56600365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.483+0000 7f566b7fe700 10 client.14164.objecter ms_handle_connect 0x7f5660034110 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.483+0000 7f5679a5a700 1 --2- 172.21.15.87:0/2484359750 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5660034110 0x7f56600365d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5664009780 tx=0x7f5664006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:32.489 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.487+0000 7f566b7fe700 1 -- 172.21.15.87:0/2484359750 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5670023670 con 0x7f56740fc4c0 2024-02-23T00:22:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.741+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f56740fbea0 con 0x7f56740fc4c0 2024-02-23T00:22:32.745 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.743+0000 7f566b7fe700 1 -- 172.21.15.87:0/2484359750 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v24)=0 set mgr/dashboard/cluster/status v24) v1 ==== 153+0+0 (secure 0 0 0) 0x7f567001e020 con 0x7f56740fc4c0 2024-02-23T00:22:32.750 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 10 librados: watch_flush enter 2024-02-23T00:22:32.750 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 10 librados: watch_flush exit 2024-02-23T00:22:32.750 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 20 client.14164.objecter shutdown clearing up homeless session... 2024-02-23T00:22:32.750 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 10 client.14164.objecter successfully canceled tick 2024-02-23T00:22:32.750 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5660034110 msgr2=0x7f56600365d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:32.751 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 --2- 172.21.15.87:0/2484359750 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5660034110 0x7f56600365d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5664009780 tx=0x7f5664006cc0).stop 2024-02-23T00:22:32.751 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 msgr2=0x7f567410d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:32.751 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 --2- 172.21.15.87:0/2484359750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f567410d9d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f567000bf80 tx=0x7f56700090f0).stop 2024-02-23T00:22:32.751 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 shutdown_connections 2024-02-23T00:22:32.751 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 --2- 172.21.15.87:0/2484359750 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5660034110 0x7f56600365d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:32.751 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 --2- 172.21.15.87:0/2484359750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f56740fc4c0 0x7f567410d9d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:32.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 >> 172.21.15.87:0/2484359750 conn(0x7f56740f5df0 msgr2=0x7f56740f75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:32.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 shutdown_connections 2024-02-23T00:22:32.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 -- 172.21.15.87:0/2484359750 wait complete. 2024-02-23T00:22:32.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: 2024-02-23T00:22:32.745+0000 7f567c4bf700 1 librados: shutdown 2024-02-23T00:22:32.752 INFO:teuthology.orchestra.run.smithi087.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-23T00:22:33.467 INFO:teuthology.orchestra.run.smithi087.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr:Or, if you are only running a single cluster on this host: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: ceph telemetry on 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr:For more information see: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:33.468 INFO:teuthology.orchestra.run.smithi087.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-02-23T00:22:33.469 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:22:33.469 INFO:teuthology.orchestra.run.smithi087.stderr:Bootstrap complete. 2024-02-23T00:22:33.508 INFO:tasks.cephadm:Fetching config... 2024-02-23T00:22:33.508 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:22:33.508 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-23T00:22:33.528 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-23T00:22:33.528 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:22:33.528 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-23T00:22:33.587 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-23T00:22:33.587 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:22:33.587 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-23T00:22:33.655 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-23T00:22:33.655 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:22:33.655 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-23T00:22:33.713 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-23T00:22:33.714 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4xb4ehDsa1R++8TY7iX5ijpvAcLZRUn9xhWWhRsIU/yRO19ZT8nJ94q6Y7uQNAtyRM2jy6cFHpaXKyMv0Jg5Qx3WVDBTtrpBDX8/VssUEZ4SyAOxXCKXGrpMQ6XTwBMXHWb4qUhAMKF6GObfJpufyuZF0+M41w4WvXkpM8DqYf6VGWdHLGdxOD7R+5ilGbRdBhdgoJD1T1b2E6MI1waKsbwk6yYYZUyPMPhrEJG6KhHC6AOiFEz8WkIFpaiH4nzXR6LaW5oGsIeVnnaYp1K3X/TV3Qo0w+rV0PMNAkMkuUG7XEbd2Tu8Dt8YjUld6AjQWnvcsOeAwmaw1FXbpVtmyLdH3RCkDzhAlAYG1GyjK/rF7LzC+4rVU8JZy4Mt43pVUt0vjyi8P2ZwaA2Ba2IWw8LIlKAW3R6Sz0HQql2jApM9wY350SvOaG/8KyCoXikueYZUmDQpOzBHvhlkDS3r9ytps8fyIJPRPjwNwiGMI7jiF3P73Vw0SM6/BGkRBKt0= ceph-66db207c-d1e1-11ee-95c0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-23T00:22:33.796 INFO:teuthology.orchestra.run.smithi087.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4xb4ehDsa1R++8TY7iX5ijpvAcLZRUn9xhWWhRsIU/yRO19ZT8nJ94q6Y7uQNAtyRM2jy6cFHpaXKyMv0Jg5Qx3WVDBTtrpBDX8/VssUEZ4SyAOxXCKXGrpMQ6XTwBMXHWb4qUhAMKF6GObfJpufyuZF0+M41w4WvXkpM8DqYf6VGWdHLGdxOD7R+5ilGbRdBhdgoJD1T1b2E6MI1waKsbwk6yYYZUyPMPhrEJG6KhHC6AOiFEz8WkIFpaiH4nzXR6LaW5oGsIeVnnaYp1K3X/TV3Qo0w+rV0PMNAkMkuUG7XEbd2Tu8Dt8YjUld6AjQWnvcsOeAwmaw1FXbpVtmyLdH3RCkDzhAlAYG1GyjK/rF7LzC+4rVU8JZy4Mt43pVUt0vjyi8P2ZwaA2Ba2IWw8LIlKAW3R6Sz0HQql2jApM9wY350SvOaG/8KyCoXikueYZUmDQpOzBHvhlkDS3r9ytps8fyIJPRPjwNwiGMI7jiF3P73Vw0SM6/BGkRBKt0= ceph-66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:33.807 DEBUG:teuthology.orchestra.run.smithi171:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4xb4ehDsa1R++8TY7iX5ijpvAcLZRUn9xhWWhRsIU/yRO19ZT8nJ94q6Y7uQNAtyRM2jy6cFHpaXKyMv0Jg5Qx3WVDBTtrpBDX8/VssUEZ4SyAOxXCKXGrpMQ6XTwBMXHWb4qUhAMKF6GObfJpufyuZF0+M41w4WvXkpM8DqYf6VGWdHLGdxOD7R+5ilGbRdBhdgoJD1T1b2E6MI1waKsbwk6yYYZUyPMPhrEJG6KhHC6AOiFEz8WkIFpaiH4nzXR6LaW5oGsIeVnnaYp1K3X/TV3Qo0w+rV0PMNAkMkuUG7XEbd2Tu8Dt8YjUld6AjQWnvcsOeAwmaw1FXbpVtmyLdH3RCkDzhAlAYG1GyjK/rF7LzC+4rVU8JZy4Mt43pVUt0vjyi8P2ZwaA2Ba2IWw8LIlKAW3R6Sz0HQql2jApM9wY350SvOaG/8KyCoXikueYZUmDQpOzBHvhlkDS3r9ytps8fyIJPRPjwNwiGMI7jiF3P73Vw0SM6/BGkRBKt0= ceph-66db207c-d1e1-11ee-95c0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-23T00:22:33.851 INFO:teuthology.orchestra.run.smithi171.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4xb4ehDsa1R++8TY7iX5ijpvAcLZRUn9xhWWhRsIU/yRO19ZT8nJ94q6Y7uQNAtyRM2jy6cFHpaXKyMv0Jg5Qx3WVDBTtrpBDX8/VssUEZ4SyAOxXCKXGrpMQ6XTwBMXHWb4qUhAMKF6GObfJpufyuZF0+M41w4WvXkpM8DqYf6VGWdHLGdxOD7R+5ilGbRdBhdgoJD1T1b2E6MI1waKsbwk6yYYZUyPMPhrEJG6KhHC6AOiFEz8WkIFpaiH4nzXR6LaW5oGsIeVnnaYp1K3X/TV3Qo0w+rV0PMNAkMkuUG7XEbd2Tu8Dt8YjUld6AjQWnvcsOeAwmaw1FXbpVtmyLdH3RCkDzhAlAYG1GyjK/rF7LzC+4rVU8JZy4Mt43pVUt0vjyi8P2ZwaA2Ba2IWw8LIlKAW3R6Sz0HQql2jApM9wY350SvOaG/8KyCoXikueYZUmDQpOzBHvhlkDS3r9ytps8fyIJPRPjwNwiGMI7jiF3P73Vw0SM6/BGkRBKt0= ceph-66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:22:33.863 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-23T00:22:34.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:32.745040+0000 mon.a (mon.0) 88 : audit [INF] from='client.? 172.21.15.87:0/2484359750' entity='client.admin' 2024-02-23T00:22:36.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.873+0000 7f55a702b700 1 -- 172.21.15.87:0/3952808145 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 msgr2=0x7f55a00f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:36.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.873+0000 7f55a702b700 1 --2- 172.21.15.87:0/3952808145 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 0x7f55a00f68d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f5594004770 tx=0x7f5594009cb0).stop 2024-02-23T00:22:36.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.873+0000 7f55a702b700 1 -- 172.21.15.87:0/3952808145 shutdown_connections 2024-02-23T00:22:36.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.873+0000 7f55a702b700 1 --2- 172.21.15.87:0/3952808145 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 0x7f55a00f68d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:36.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.873+0000 7f55a702b700 1 -- 172.21.15.87:0/3952808145 >> 172.21.15.87:0/3952808145 conn(0x7f55a00f1c10 msgr2=0x7f55a00f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.873+0000 7f55a702b700 1 -- 172.21.15.87:0/3952808145 shutdown_connections 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.873+0000 7f55a702b700 1 -- 172.21.15.87:0/3952808145 wait complete. 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a702b700 1 librados: starting msgr at 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a702b700 1 librados: starting objecter 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a702b700 1 Processor -- start 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a702b700 1 -- start start 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a702b700 1 librados: setting wanted keys 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a702b700 1 librados: calling monclient init 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a702b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 0x7f55a00f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a702b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a00f2590 con 0x7f55a00f64b0 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a4dc7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 0x7f55a00f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:36.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a4dc7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 0x7f55a00f4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39728/0 (socket says 172.21.15.87:39728) 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.874+0000 7f55a4dc7700 1 -- 172.21.15.87:0/3578176075 learned_addr learned my addr 172.21.15.87:0/3578176075 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f55a4dc7700 1 -- 172.21.15.87:0/3578176075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5594005040 con 0x7f55a00f64b0 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f55a4dc7700 1 --2- 172.21.15.87:0/3578176075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 0x7f55a00f4110 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5594013560 tx=0x7f5594009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f559dffb700 10 client.?.objecter ms_handle_connect 0x7f55a00f64b0 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f559dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f559dffb700 1 -- 172.21.15.87:0/3578176075 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f559400e030 con 0x7f55a00f64b0 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f559dffb700 1 -- 172.21.15.87:0/3578176075 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5594009710 con 0x7f55a00f64b0 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a00f2730 con 0x7f55a00f64b0 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f559dffb700 1 -- 172.21.15.87:0/3578176075 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5594018070 con 0x7f55a00f64b0 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f55a702b700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:36.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a00f2bc0 con 0x7f55a00f64b0 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.875+0000 7f55a702b700 1 librados: init done 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559dffb700 1 -- 172.21.15.87:0/3578176075 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7f559401c450 con 0x7f55a00f64b0 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559dffb700 1 --2- 172.21.15.87:0/3578176075 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5590033f60 0x7f5590036420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559dffb700 1 -- 172.21.15.87:0/3578176075 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f5594044280 con 0x7f55a00f64b0 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559dffb700 10 client.14166.objecter ms_dispatch 0x7f55a0001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559dffb700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559dffb700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a005a310 con 0x7f55a00f64b0 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559dffb700 20 client.14166.objecter dump_active .. 0 homeless 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559ffff700 1 --2- 172.21.15.87:0/3578176075 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5590033f60 0x7f5590036420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:36.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559dffb700 10 client.14166.objecter ms_handle_connect 0x7f5590033f60 2024-02-23T00:22:36.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.876+0000 7f559ffff700 1 --2- 172.21.15.87:0/3578176075 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5590033f60 0x7f5590036420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f558c009780 tx=0x7f558c006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:36.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:36.881+0000 7f559dffb700 1 -- 172.21.15.87:0/3578176075 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5594027440 con 0x7f55a00f64b0 2024-02-23T00:22:37.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.047+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f55a00fb970 con 0x7f55a00f64b0 2024-02-23T00:22:37.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.050+0000 7f559dffb700 1 -- 172.21.15.87:0/3578176075 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f559401bb60 con 0x7f55a00f64b0 2024-02-23T00:22:37.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.052+0000 7f55a702b700 10 librados: watch_flush enter 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.052+0000 7f55a702b700 10 librados: watch_flush exit 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.052+0000 7f55a702b700 20 client.14166.objecter shutdown clearing up homeless session... 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.052+0000 7f55a702b700 10 client.14166.objecter successfully canceled tick 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5590033f60 msgr2=0x7f5590036420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 --2- 172.21.15.87:0/3578176075 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5590033f60 0x7f5590036420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f558c009780 tx=0x7f558c006cc0).stop 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 msgr2=0x7f55a00f4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 --2- 172.21.15.87:0/3578176075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 0x7f55a00f4110 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5594013560 tx=0x7f5594009cb0).stop 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 shutdown_connections 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 --2- 172.21.15.87:0/3578176075 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5590033f60 0x7f5590036420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 --2- 172.21.15.87:0/3578176075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55a00f64b0 0x7f55a00f4110 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 >> 172.21.15.87:0/3578176075 conn(0x7f55a00f1c10 msgr2=0x7f55a0060d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 shutdown_connections 2024-02-23T00:22:37.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 -- 172.21.15.87:0/3578176075 wait complete. 2024-02-23T00:22:37.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:37.053+0000 7f55a702b700 1 librados: shutdown 2024-02-23T00:22:38.066 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-23T00:22:38.066 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-23T00:22:38.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:37.050709+0000 mon.a (mon.0) 89 : audit [INF] from='client.? 172.21.15.87:0/3578176075' entity='client.admin' 2024-02-23T00:22:40.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 -- 172.21.15.87:0/3673440702 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 msgr2=0x7f39ec0fa1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:40.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 --2- 172.21.15.87:0/3673440702 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 0x7f39ec0fa1f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f39e8004770 tx=0x7f39e8009cb0).stop 2024-02-23T00:22:40.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 -- 172.21.15.87:0/3673440702 shutdown_connections 2024-02-23T00:22:40.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 --2- 172.21.15.87:0/3673440702 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 0x7f39ec0fa1f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:40.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 -- 172.21.15.87:0/3673440702 >> 172.21.15.87:0/3673440702 conn(0x7f39ec0f1c10 msgr2=0x7f39ec0f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:40.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 -- 172.21.15.87:0/3673440702 shutdown_connections 2024-02-23T00:22:40.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 -- 172.21.15.87:0/3673440702 wait complete. 2024-02-23T00:22:40.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 librados: starting msgr at 2024-02-23T00:22:40.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 librados: starting objecter 2024-02-23T00:22:40.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 Processor -- start 2024-02-23T00:22:40.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 -- start start 2024-02-23T00:22:40.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 librados: setting wanted keys 2024-02-23T00:22:40.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.345+0000 7f39f3e53700 1 librados: calling monclient init 2024-02-23T00:22:40.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f3e53700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 0x7f39ec0f7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:40.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f3e53700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39ec0f60f0 con 0x7f39ec061190 2024-02-23T00:22:40.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f1bef700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 0x7f39ec0f7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:40.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f1bef700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 0x7f39ec0f7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39736/0 (socket says 172.21.15.87:39736) 2024-02-23T00:22:40.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f1bef700 1 -- 172.21.15.87:0/1905678066 learned_addr learned my addr 172.21.15.87:0/1905678066 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:22:40.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f1bef700 1 -- 172.21.15.87:0/1905678066 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e8005040 con 0x7f39ec061190 2024-02-23T00:22:40.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f1bef700 1 --2- 172.21.15.87:0/1905678066 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 0x7f39ec0f7c70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f39e8013a00 tx=0x7f39e8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:40.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39e2ffd700 10 client.?.objecter ms_handle_connect 0x7f39ec061190 2024-02-23T00:22:40.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39e2ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:22:40.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39e2ffd700 1 -- 172.21.15.87:0/1905678066 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e800e030 con 0x7f39ec061190 2024-02-23T00:22:40.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39e2ffd700 1 -- 172.21.15.87:0/1905678066 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39e8013630 con 0x7f39ec061190 2024-02-23T00:22:40.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ec0f6290 con 0x7f39ec061190 2024-02-23T00:22:40.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39e2ffd700 1 -- 172.21.15.87:0/1905678066 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e8018070 con 0x7f39ec061190 2024-02-23T00:22:40.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f3e53700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:22:40.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ec0f6720 con 0x7f39ec061190 2024-02-23T00:22:40.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.346+0000 7f39f3e53700 1 librados: init done 2024-02-23T00:22:40.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.347+0000 7f39e2ffd700 1 -- 172.21.15.87:0/1905678066 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7f39e801d460 con 0x7f39ec061190 2024-02-23T00:22:40.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.347+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ec05a310 con 0x7f39ec061190 2024-02-23T00:22:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.347+0000 7f39e2ffd700 1 --2- 172.21.15.87:0/1905678066 >> v2:172.21.15.87:6800/1121018082 conn(0x7f39d8034030 0x7f39d80364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:22:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.347+0000 7f39e2ffd700 1 -- 172.21.15.87:0/1905678066 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f39e8043c50 con 0x7f39ec061190 2024-02-23T00:22:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.347+0000 7f39e2ffd700 10 client.14168.objecter ms_dispatch 0x7f39ec001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:22:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.347+0000 7f39e2ffd700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:22:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.347+0000 7f39e2ffd700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:22:40.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.347+0000 7f39e2ffd700 20 client.14168.objecter dump_active .. 0 homeless 2024-02-23T00:22:40.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.347+0000 7f39f13ee700 1 --2- 172.21.15.87:0/1905678066 >> v2:172.21.15.87:6800/1121018082 conn(0x7f39d8034030 0x7f39d80364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:22:40.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.348+0000 7f39e2ffd700 10 client.14168.objecter ms_handle_connect 0x7f39d8034030 2024-02-23T00:22:40.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.348+0000 7f39f13ee700 1 --2- 172.21.15.87:0/1905678066 >> v2:172.21.15.87:6800/1121018082 conn(0x7f39d8034030 0x7f39d80364f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f39dc009780 tx=0x7f39dc006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:22:40.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.352+0000 7f39e2ffd700 1 -- 172.21.15.87:0/1905678066 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f39e801d710 con 0x7f39ec061190 2024-02-23T00:22:40.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.517+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f39ec0f2be0 con 0x7f39d8034030 2024-02-23T00:22:40.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.523+0000 7f39e2ffd700 1 -- 172.21.15.87:0/1905678066 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f39ec0f2be0 con 0x7f39d8034030 2024-02-23T00:22:40.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.524+0000 7f39f3e53700 10 librados: watch_flush enter 2024-02-23T00:22:40.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.524+0000 7f39f3e53700 10 librados: watch_flush exit 2024-02-23T00:22:40.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.524+0000 7f39f3e53700 20 client.14168.objecter shutdown clearing up homeless session... 2024-02-23T00:22:40.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.524+0000 7f39f3e53700 10 client.14168.objecter successfully canceled tick 2024-02-23T00:22:40.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.524+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 >> v2:172.21.15.87:6800/1121018082 conn(0x7f39d8034030 msgr2=0x7f39d80364f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:40.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 --2- 172.21.15.87:0/1905678066 >> v2:172.21.15.87:6800/1121018082 conn(0x7f39d8034030 0x7f39d80364f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f39dc009780 tx=0x7f39dc006cc0).stop 2024-02-23T00:22:40.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 msgr2=0x7f39ec0f7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:22:40.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 --2- 172.21.15.87:0/1905678066 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 0x7f39ec0f7c70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f39e8013a00 tx=0x7f39e8009cb0).stop 2024-02-23T00:22:40.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 shutdown_connections 2024-02-23T00:22:40.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 --2- 172.21.15.87:0/1905678066 >> v2:172.21.15.87:6800/1121018082 conn(0x7f39d8034030 0x7f39d80364f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:40.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 --2- 172.21.15.87:0/1905678066 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39ec061190 0x7f39ec0f7c70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:22:40.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 >> 172.21.15.87:0/1905678066 conn(0x7f39ec0f1c10 msgr2=0x7f39ec0f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:22:40.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 shutdown_connections 2024-02-23T00:22:40.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 -- 172.21.15.87:0/1905678066 wait complete. 2024-02-23T00:22:40.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:22:40.525+0000 7f39f3e53700 1 librados: shutdown 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:39.892425+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.188519+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi087", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.191613+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.192826+0000 mon.a (mon.0) 93 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.198237+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.523317+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.563224+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.564557+0000 mon.a (mon.0) 97 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.565076+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:22:40.566048+0000 mgr.y (mgr.14146) 6 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.conf 2024-02-23T00:22:41.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:22:40.592800+0000 mgr.y (mgr.14146) 7 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:22:41.200 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.636384+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:41.200 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:22:40.641577+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:22:41.219 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi171 2024-02-23T00:22:41.220 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:22:41.220 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.conf 2024-02-23T00:22:41.241 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:22:41.242 DEBUG:teuthology.orchestra.run.smithi171:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:22:41.304 INFO:tasks.cephadm:Adding host smithi171 to orchestrator... 2024-02-23T00:22:41.304 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch host add smithi171 2024-02-23T00:22:43.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:42.053774+0000 mgr.y (mgr.14146) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:22:45.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:44.054127+0000 mgr.y (mgr.14146) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:22:47.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:46.054523+0000 mgr.y (mgr.14146) 10 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:22:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:48.054840+0000 mgr.y (mgr.14146) 11 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:22:51.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:50.055113+0000 mgr.y (mgr.14146) 12 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:22:53.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:52.055379+0000 mgr.y (mgr.14146) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:22:55.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:54.055708+0000 mgr.y (mgr.14146) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:22:57.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:56.056094+0000 mgr.y (mgr.14146) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:22:59.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:22:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:22:58.056315+0000 mgr.y (mgr.14146) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:00.056635+0000 mgr.y (mgr.14146) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:03.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:02.056998+0000 mgr.y (mgr.14146) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:05.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:04.057269+0000 mgr.y (mgr.14146) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:07.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:06.057626+0000 mgr.y (mgr.14146) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:09.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:08.057904+0000 mgr.y (mgr.14146) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:10.772 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.768+0000 7f37935b5700 1 -- 172.21.15.171:0/1917152409 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 msgr2=0x7f378c0f79e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.768+0000 7f37935b5700 1 --2- 172.21.15.171:0/1917152409 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 0x7f378c0f79e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f3788004770 tx=0x7f3788009cb0).stop 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.768+0000 7f37935b5700 1 -- 172.21.15.171:0/1917152409 shutdown_connections 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.768+0000 7f37935b5700 1 --2- 172.21.15.171:0/1917152409 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 0x7f378c0f79e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.768+0000 7f37935b5700 1 -- 172.21.15.171:0/1917152409 >> 172.21.15.171:0/1917152409 conn(0x7f378c0f2d40 msgr2=0x7f378c0f5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.768+0000 7f37935b5700 1 -- 172.21.15.171:0/1917152409 shutdown_connections 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.768+0000 7f37935b5700 1 -- 172.21.15.171:0/1917152409 wait complete. 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f37935b5700 1 librados: starting msgr at 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f37935b5700 1 librados: starting objecter 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f37935b5700 1 Processor -- start 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f37935b5700 1 -- start start 2024-02-23T00:23:10.773 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f37935b5700 1 librados: setting wanted keys 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f37935b5700 1 librados: calling monclient init 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f37935b5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 0x7f378c06cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f37935b5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c06b030 con 0x7f378c0f75c0 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f3791351700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 0x7f378c06cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f3791351700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 0x7f378c06cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.171:47648/0 (socket says 172.21.15.171:47648) 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f3791351700 1 -- 172.21.15.171:0/929029672 learned_addr learned my addr 172.21.15.171:0/929029672 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f3791351700 1 -- 172.21.15.171:0/929029672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3788005040 con 0x7f378c0f75c0 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f3791351700 1 --2- 172.21.15.171:0/929029672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 0x7f378c06cbb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f3788013560 tx=0x7f3788009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.769+0000 7f37827fc700 10 client.?.objecter ms_handle_connect 0x7f378c0f75c0 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.770+0000 7f37827fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.770+0000 7f37827fc700 1 -- 172.21.15.171:0/929029672 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f378800e030 con 0x7f378c0f75c0 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.770+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378c06b1d0 con 0x7f378c0f75c0 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.770+0000 7f37827fc700 1 -- 172.21.15.171:0/929029672 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3788009710 con 0x7f378c0f75c0 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.770+0000 7f37827fc700 1 -- 172.21.15.171:0/929029672 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3788018070 con 0x7f378c0f75c0 2024-02-23T00:23:10.774 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.770+0000 7f37935b5700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.770+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378c06b660 con 0x7f378c0f75c0 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.770+0000 7f37935b5700 1 librados: init done 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f37827fc700 1 -- 172.21.15.171:0/929029672 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7f378801d420 con 0x7f378c0f75c0 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378c05a310 con 0x7f378c0f75c0 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f37827fc700 1 --2- 172.21.15.171:0/929029672 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3778033fb0 0x7f3778036470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f37827fc700 1 -- 172.21.15.171:0/929029672 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f37880441e0 con 0x7f378c0f75c0 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f37827fc700 10 client.14170.objecter ms_dispatch 0x7f378c001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f37827fc700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f37827fc700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f37827fc700 20 client.14170.objecter dump_active .. 0 homeless 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f3790b50700 1 --2- 172.21.15.171:0/929029672 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3778033fb0 0x7f3778036470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f37827fc700 10 client.14170.objecter ms_handle_connect 0x7f3778033fb0 2024-02-23T00:23:10.775 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.771+0000 7f3790b50700 1 --2- 172.21.15.171:0/929029672 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3778033fb0 0x7f3778036470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f377c009780 tx=0x7f377c006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:10.777 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.776+0000 7f37827fc700 1 -- 172.21.15.171:0/929029672 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3788025b80 con 0x7f378c0f75c0 2024-02-23T00:23:10.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:10.940+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}) v1 -- 0x7f378c0f3cd0 con 0x7f3778033fb0 2024-02-23T00:23:11.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:10.058282+0000 mgr.y (mgr.14146) 22 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:11.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:23:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [90B blob data] 2024-02-23T00:23:11.571 INFO:teuthology.orchestra.run.smithi171.stdout:Added host 'smithi171' with addr '172.21.15.171' 2024-02-23T00:23:11.571 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.569+0000 7f37827fc700 1 -- 172.21.15.171:0/929029672 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f378c0f3cd0 con 0x7f3778033fb0 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 10 librados: watch_flush enter 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 10 librados: watch_flush exit 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 20 client.14170.objecter shutdown clearing up homeless session... 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 10 client.14170.objecter successfully canceled tick 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3778033fb0 msgr2=0x7f3778036470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 --2- 172.21.15.171:0/929029672 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3778033fb0 0x7f3778036470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f377c009780 tx=0x7f377c006cc0).stop 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 msgr2=0x7f378c06cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 --2- 172.21.15.171:0/929029672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 0x7f378c06cbb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f3788013560 tx=0x7f3788009cb0).stop 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 shutdown_connections 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 --2- 172.21.15.171:0/929029672 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3778033fb0 0x7f3778036470 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:11.573 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 --2- 172.21.15.171:0/929029672 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0f75c0 0x7f378c06cbb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:11.574 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 >> 172.21.15.171:0/929029672 conn(0x7f378c0f2d40 msgr2=0x7f378c0f35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:11.574 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 shutdown_connections 2024-02-23T00:23:11.574 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 -- 172.21.15.171:0/929029672 wait complete. 2024-02-23T00:23:11.574 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:11.571+0000 7f37935b5700 1 librados: shutdown 2024-02-23T00:23:12.082 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:23:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [80B blob data] 2024-02-23T00:23:12.205 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch host ls --format=json 2024-02-23T00:23:12.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:10.942072+0000 mgr.y (mgr.14146) 23 : audit [DBG] from='client.14170 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi171", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:12.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:11.568963+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:11.569382+0000 mgr.y (mgr.14146) 24 : cephadm [INF] Added host smithi171 2024-02-23T00:23:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:11.584764+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:12.262571+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:13.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:12.058643+0000 mgr.y (mgr.14146) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:13.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:13.500425+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:13.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.939+0000 7fa969f91700 1 -- 172.21.15.171:0/44174660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 msgr2=0x7fa9640f68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:13.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.939+0000 7fa969f91700 1 --2- 172.21.15.171:0/44174660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 0x7fa9640f68d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa954004770 tx=0x7fa954009cb0).stop 2024-02-23T00:23:13.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.939+0000 7fa969f91700 1 -- 172.21.15.171:0/44174660 shutdown_connections 2024-02-23T00:23:13.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.939+0000 7fa969f91700 1 --2- 172.21.15.171:0/44174660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 0x7fa9640f68d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:13.940 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.939+0000 7fa969f91700 1 -- 172.21.15.171:0/44174660 >> 172.21.15.171:0/44174660 conn(0x7fa9640f1c10 msgr2=0x7fa9640f4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:13.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.939+0000 7fa969f91700 1 -- 172.21.15.171:0/44174660 shutdown_connections 2024-02-23T00:23:13.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.939+0000 7fa969f91700 1 -- 172.21.15.171:0/44174660 wait complete. 2024-02-23T00:23:13.941 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.939+0000 7fa969f91700 1 librados: starting msgr at 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.939+0000 7fa969f91700 1 librados: starting objecter 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.940+0000 7fa969f91700 1 Processor -- start 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.940+0000 7fa969f91700 1 -- start start 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.940+0000 7fa969f91700 1 librados: setting wanted keys 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.940+0000 7fa969f91700 1 librados: calling monclient init 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.940+0000 7fa969f91700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 0x7fa9640f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.940+0000 7fa969f91700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9640f2590 con 0x7fa9640f64b0 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.940+0000 7fa9637fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 0x7fa9640f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.941+0000 7fa9637fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 0x7fa9640f4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.171:57664/0 (socket says 172.21.15.171:57664) 2024-02-23T00:23:13.942 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.941+0000 7fa9637fe700 1 -- 172.21.15.171:0/3234727521 learned_addr learned my addr 172.21.15.171:0/3234727521 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:23:13.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.941+0000 7fa9637fe700 1 -- 172.21.15.171:0/3234727521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa954005040 con 0x7fa9640f64b0 2024-02-23T00:23:13.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.941+0000 7fa9637fe700 1 --2- 172.21.15.171:0/3234727521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 0x7fa9640f4110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa954013560 tx=0x7fa954009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:13.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.941+0000 7fa960ff9700 10 client.?.objecter ms_handle_connect 0x7fa9640f64b0 2024-02-23T00:23:13.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.942+0000 7fa960ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:13.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.942+0000 7fa960ff9700 1 -- 172.21.15.171:0/3234727521 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa95400e030 con 0x7fa9640f64b0 2024-02-23T00:23:13.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.942+0000 7fa960ff9700 1 -- 172.21.15.171:0/3234727521 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa954009710 con 0x7fa9640f64b0 2024-02-23T00:23:13.943 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.942+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9640f2730 con 0x7fa9640f64b0 2024-02-23T00:23:13.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.942+0000 7fa960ff9700 1 -- 172.21.15.171:0/3234727521 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa954018070 con 0x7fa9640f64b0 2024-02-23T00:23:13.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.942+0000 7fa969f91700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:13.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.942+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9640f2bc0 con 0x7fa9640f64b0 2024-02-23T00:23:13.944 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.942+0000 7fa969f91700 1 librados: init done 2024-02-23T00:23:13.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.943+0000 7fa960ff9700 1 -- 172.21.15.171:0/3234727521 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7fa95401d420 con 0x7fa9640f64b0 2024-02-23T00:23:13.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.943+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96405a310 con 0x7fa9640f64b0 2024-02-23T00:23:13.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.943+0000 7fa960ff9700 1 --2- 172.21.15.171:0/3234727521 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa94c033f60 0x7fa94c036420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:13.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.943+0000 7fa960ff9700 1 -- 172.21.15.171:0/3234727521 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fa9540442e0 con 0x7fa9640f64b0 2024-02-23T00:23:13.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.943+0000 7fa960ff9700 10 client.14172.objecter ms_dispatch 0x7fa964001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:23:13.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.943+0000 7fa960ff9700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:23:13.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.943+0000 7fa960ff9700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:23:13.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.943+0000 7fa960ff9700 20 client.14172.objecter dump_active .. 0 homeless 2024-02-23T00:23:13.945 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.944+0000 7fa962ffd700 1 --2- 172.21.15.171:0/3234727521 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa94c033f60 0x7fa94c036420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:13.946 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.944+0000 7fa960ff9700 10 client.14172.objecter ms_handle_connect 0x7fa94c033f60 2024-02-23T00:23:13.946 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.944+0000 7fa962ffd700 1 --2- 172.21.15.171:0/3234727521 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa94c033f60 0x7fa94c036420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa958009780 tx=0x7fa958006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:13.949 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:13.948+0000 7fa960ff9700 1 -- 172.21.15.171:0/3234727521 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa954025b80 con 0x7fa9640f64b0 2024-02-23T00:23:14.123 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.122+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa9640f2e00 con 0x7fa94c033f60 2024-02-23T00:23:14.125 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.124+0000 7fa960ff9700 1 -- 172.21.15.171:0/3234727521 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fa9640f2e00 con 0x7fa94c033f60 2024-02-23T00:23:14.125 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:23:14.125 INFO:teuthology.orchestra.run.smithi171.stdout:[{"addr": "172.21.15.87", "hostname": "smithi087", "labels": [], "status": ""}, {"addr": "172.21.15.171", "hostname": "smithi171", "labels": [], "status": ""}] 2024-02-23T00:23:14.127 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 10 librados: watch_flush enter 2024-02-23T00:23:14.127 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 10 librados: watch_flush exit 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 20 client.14172.objecter shutdown clearing up homeless session... 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 10 client.14172.objecter successfully canceled tick 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa94c033f60 msgr2=0x7fa94c036420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 --2- 172.21.15.171:0/3234727521 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa94c033f60 0x7fa94c036420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa958009780 tx=0x7fa958006cc0).stop 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 msgr2=0x7fa9640f4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 --2- 172.21.15.171:0/3234727521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 0x7fa9640f4110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa954013560 tx=0x7fa954009cb0).stop 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 shutdown_connections 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 --2- 172.21.15.171:0/3234727521 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa94c033f60 0x7fa94c036420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 --2- 172.21.15.171:0/3234727521 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9640f64b0 0x7fa9640f4110 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 >> 172.21.15.171:0/3234727521 conn(0x7fa9640f1c10 msgr2=0x7fa964060d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 shutdown_connections 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 -- 172.21.15.171:0/3234727521 wait complete. 2024-02-23T00:23:14.128 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:14.126+0000 7fa969f91700 1 librados: shutdown 2024-02-23T00:23:15.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:14.058964+0000 mgr.y (mgr.14146) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:15.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:14.124187+0000 mgr.y (mgr.14146) 27 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:23:15.894 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-02-23T00:23:15.894 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd crush tunables jewel 2024-02-23T00:23:17.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.664+0000 7f00cf694700 1 -- 172.21.15.87:0/789706446 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 msgr2=0x7f00c80fc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:17.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.664+0000 7f00cf694700 1 --2- 172.21.15.87:0/789706446 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 0x7f00c80fc910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f00b8004770 tx=0x7f00b8009cb0).stop 2024-02-23T00:23:17.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.664+0000 7f00cf694700 1 -- 172.21.15.87:0/789706446 shutdown_connections 2024-02-23T00:23:17.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.664+0000 7f00cf694700 1 --2- 172.21.15.87:0/789706446 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 0x7f00c80fc910 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:17.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.664+0000 7f00cf694700 1 -- 172.21.15.87:0/789706446 >> 172.21.15.87:0/789706446 conn(0x7f00c80f2100 msgr2=0x7f00c80f4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:17.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.664+0000 7f00cf694700 1 -- 172.21.15.87:0/789706446 shutdown_connections 2024-02-23T00:23:17.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.664+0000 7f00cf694700 1 -- 172.21.15.87:0/789706446 wait complete. 2024-02-23T00:23:17.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cf694700 1 librados: starting msgr at 2024-02-23T00:23:17.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cf694700 1 librados: starting objecter 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cf694700 1 Processor -- start 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cf694700 1 -- start start 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cf694700 1 librados: setting wanted keys 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cf694700 1 librados: calling monclient init 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cf694700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 0x7f00c810d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cf694700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c810d520 con 0x7f00c80fc530 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cd430700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 0x7f00c810d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cd430700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 0x7f00c810d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33888/0 (socket says 172.21.15.87:33888) 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cd430700 1 -- 172.21.15.87:0/3894932516 learned_addr learned my addr 172.21.15.87:0/3894932516 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cd430700 1 -- 172.21.15.87:0/3894932516 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b8005040 con 0x7f00c80fc530 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00cd430700 1 --2- 172.21.15.87:0/3894932516 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 0x7f00c810d030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f00b8009b10 tx=0x7f00b8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00be7fc700 10 client.?.objecter ms_handle_connect 0x7f00c80fc530 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00be7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:17.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.665+0000 7f00be7fc700 1 -- 172.21.15.87:0/3894932516 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00b800e030 con 0x7f00c80fc530 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.666+0000 7f00be7fc700 1 -- 172.21.15.87:0/3894932516 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00b8009430 con 0x7f00c80fc530 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.666+0000 7f00be7fc700 1 -- 172.21.15.87:0/3894932516 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00b8018070 con 0x7f00c80fc530 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.666+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c810d720 con 0x7f00c80fc530 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.666+0000 7f00cf694700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.666+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c810db30 con 0x7f00c80fc530 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.666+0000 7f00cf694700 1 librados: init done 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.666+0000 7f00be7fc700 1 -- 172.21.15.87:0/3894932516 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7f00b801d420 con 0x7f00c80fc530 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.666+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c805fe40 con 0x7f00c80fc530 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.666+0000 7f00be7fc700 1 --2- 172.21.15.87:0/3894932516 >> v2:172.21.15.87:6800/1121018082 conn(0x7f00b4034000 0x7f00b40364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.667+0000 7f00be7fc700 1 -- 172.21.15.87:0/3894932516 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f00b8044260 con 0x7f00c80fc530 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.667+0000 7f00be7fc700 10 client.14174.objecter ms_dispatch 0x7f00c8001550 osd_map(3..3 src has 1..3) v4 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.667+0000 7f00be7fc700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.667+0000 7f00be7fc700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.667+0000 7f00be7fc700 20 client.14174.objecter dump_active .. 0 homeless 2024-02-23T00:23:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.667+0000 7f00ccc2f700 1 --2- 172.21.15.87:0/3894932516 >> v2:172.21.15.87:6800/1121018082 conn(0x7f00b4034000 0x7f00b40364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:17.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.667+0000 7f00be7fc700 10 client.14174.objecter ms_handle_connect 0x7f00b4034000 2024-02-23T00:23:17.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.667+0000 7f00ccc2f700 1 --2- 172.21.15.87:0/3894932516 >> v2:172.21.15.87:6800/1121018082 conn(0x7f00b4034000 0x7f00b40364c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f00c4009780 tx=0x7f00c4006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:17.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.672+0000 7f00be7fc700 1 -- 172.21.15.87:0/3894932516 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f00b8027440 con 0x7f00c80fc530 2024-02-23T00:23:17.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:16.059239+0000 mgr.y (mgr.14146) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:17.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:17.494831+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:17.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:17.837+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) v1 -- 0x7f00c8046ce0 con 0x7f00c80fc530 2024-02-23T00:23:18.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.586+0000 7f00be7fc700 1 -- 172.21.15.87:0/3894932516 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) v1 ==== 120+0+0 (secure 0 0 0) 0x7f00b8025b80 con 0x7f00c80fc530 2024-02-23T00:23:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 10 librados: watch_flush enter 2024-02-23T00:23:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 10 librados: watch_flush exit 2024-02-23T00:23:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 20 client.14174.objecter shutdown clearing up homeless session... 2024-02-23T00:23:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 10 client.14174.objecter successfully canceled tick 2024-02-23T00:23:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 >> v2:172.21.15.87:6800/1121018082 conn(0x7f00b4034000 msgr2=0x7f00b40364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 --2- 172.21.15.87:0/3894932516 >> v2:172.21.15.87:6800/1121018082 conn(0x7f00b4034000 0x7f00b40364c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f00c4009780 tx=0x7f00c4006cc0).stop 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 msgr2=0x7f00c810d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 --2- 172.21.15.87:0/3894932516 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 0x7f00c810d030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f00b8009b10 tx=0x7f00b8009cb0).stop 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 shutdown_connections 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 --2- 172.21.15.87:0/3894932516 >> v2:172.21.15.87:6800/1121018082 conn(0x7f00b4034000 0x7f00b40364c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 --2- 172.21.15.87:0/3894932516 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f00c80fc530 0x7f00c810d030 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 >> 172.21.15.87:0/3894932516 conn(0x7f00c80f2100 msgr2=0x7f00c8060610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 shutdown_connections 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 -- 172.21.15.87:0/3894932516 wait complete. 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:18.588+0000 7f00cf694700 1 librados: shutdown 2024-02-23T00:23:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to jewel 2024-02-23T00:23:18.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:17.839283+0000 mon.a (mon.0) 106 : audit [INF] from='client.? 172.21.15.87:0/3894932516' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-02-23T00:23:18.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:17.954526+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:18.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:17.955150+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi171", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:23:18.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:17.955884+0000 mon.a (mon.0) 109 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:18.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:17.956463+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:18.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:18.013087+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:18.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:18.015213+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:19.200 INFO:tasks.cephadm:Adding mon.a on smithi087 2024-02-23T00:23:19.200 INFO:tasks.cephadm:Adding mon.c on smithi087 2024-02-23T00:23:19.200 INFO:tasks.cephadm:Adding mon.b on smithi171 2024-02-23T00:23:19.200 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch apply mon '3;smithi087:172.21.15.87=a;smithi087:[v2:172.21.15.87:3301,v1:172.21.15.87:6790]=c;smithi171:172.21.15.171=b' 2024-02-23T00:23:20.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:17.957408+0000 mgr.y (mgr.14146) 29 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2024-02-23T00:23:20.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:17.977628+0000 mgr.y (mgr.14146) 30 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:23:20.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:18.059642+0000 mgr.y (mgr.14146) 31 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:20.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:18.586586+0000 mon.a (mon.0) 113 : audit [INF] from='client.? 172.21.15.87:0/3894932516' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-02-23T00:23:20.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:18.586666+0000 mon.a (mon.0) 114 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-23T00:23:20.958 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.955+0000 7fd6e8439700 1 -- 172.21.15.171:0/1534377510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 msgr2=0x7fd6e00f8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.955+0000 7fd6e8439700 1 --2- 172.21.15.171:0/1534377510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 0x7fd6e00f8a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd6d0004770 tx=0x7fd6d0009cb0).stop 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.955+0000 7fd6e8439700 1 -- 172.21.15.171:0/1534377510 shutdown_connections 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.955+0000 7fd6e8439700 1 --2- 172.21.15.171:0/1534377510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 0x7fd6e00f8a70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.955+0000 7fd6e8439700 1 -- 172.21.15.171:0/1534377510 >> 172.21.15.171:0/1534377510 conn(0x7fd6e006c440 msgr2=0x7fd6e006c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.955+0000 7fd6e8439700 1 -- 172.21.15.171:0/1534377510 shutdown_connections 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.955+0000 7fd6e8439700 1 -- 172.21.15.171:0/1534377510 wait complete. 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.956+0000 7fd6e8439700 1 librados: starting msgr at 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.956+0000 7fd6e8439700 1 librados: starting objecter 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.956+0000 7fd6e8439700 1 Processor -- start 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.956+0000 7fd6e8439700 1 -- start start 2024-02-23T00:23:20.959 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.956+0000 7fd6e8439700 1 librados: setting wanted keys 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.956+0000 7fd6e8439700 1 librados: calling monclient init 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.956+0000 7fd6e8439700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 0x7fd6e010d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.956+0000 7fd6e8439700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e010d600 con 0x7fd6e00f8690 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6e61d5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 0x7fd6e010d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6e61d5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 0x7fd6e010d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.171:57694/0 (socket says 172.21.15.171:57694) 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6e61d5700 1 -- 172.21.15.171:0/2794203020 learned_addr learned my addr 172.21.15.171:0/2794203020 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6e61d5700 1 -- 172.21.15.171:0/2794203020 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0005040 con 0x7fd6e00f8690 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6e61d5700 1 --2- 172.21.15.171:0/2794203020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 0x7fd6e010d110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd6d0009b10 tx=0x7fd6d0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6d77fe700 10 client.?.objecter ms_handle_connect 0x7fd6e00f8690 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6d77fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6d77fe700 1 -- 172.21.15.171:0/2794203020 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6d000e030 con 0x7fd6e00f8690 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e010d800 con 0x7fd6e00f8690 2024-02-23T00:23:20.960 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6d77fe700 1 -- 172.21.15.171:0/2794203020 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6d0009430 con 0x7fd6e00f8690 2024-02-23T00:23:20.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6d77fe700 1 -- 172.21.15.171:0/2794203020 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6d0018070 con 0x7fd6e00f8690 2024-02-23T00:23:20.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6e8439700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:20.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e010dc10 con 0x7fd6e00f8690 2024-02-23T00:23:20.961 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.957+0000 7fd6e8439700 1 librados: init done 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.958+0000 7fd6d77fe700 1 -- 172.21.15.171:0/2794203020 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7fd6d0027440 con 0x7fd6e00f8690 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.958+0000 7fd6d77fe700 1 --2- 172.21.15.171:0/2794203020 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd6cc034050 0x7fd6cc036510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.958+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e005f1f0 con 0x7fd6e00f8690 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.958+0000 7fd6d77fe700 1 -- 172.21.15.171:0/2794203020 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fd6d0044450 con 0x7fd6e00f8690 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.958+0000 7fd6d77fe700 10 client.14176.objecter ms_dispatch 0x7fd6e0001550 osd_map(4..4 src has 1..4) v4 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.958+0000 7fd6d77fe700 3 client.14176.objecter handle_osd_map got epochs [4,4] > 0 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.958+0000 7fd6d77fe700 3 client.14176.objecter handle_osd_map decoding full epoch 4 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.958+0000 7fd6d77fe700 20 client.14176.objecter dump_active .. 0 homeless 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.958+0000 7fd6e59d4700 1 --2- 172.21.15.171:0/2794203020 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd6cc034050 0x7fd6cc036510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:20.962 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.959+0000 7fd6d77fe700 10 client.14176.objecter ms_handle_connect 0x7fd6cc034050 2024-02-23T00:23:20.963 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.959+0000 7fd6e59d4700 1 --2- 172.21.15.171:0/2794203020 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd6cc034050 0x7fd6cc036510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd6dc009780 tx=0x7fd6dc006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:20.964 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:20.963+0000 7fd6d77fe700 1 -- 172.21.15.171:0/2794203020 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd6d00263e0 con 0x7fd6e00f8690 2024-02-23T00:23:21.134 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.133+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi087:172.21.15.87=a;smithi087:[v2:172.21.15.87:3301,v1:172.21.15.87:6790]=c;smithi171:172.21.15.171=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6e0046ce0 con 0x7fd6cc034050 2024-02-23T00:23:21.141 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.140+0000 7fd6d77fe700 1 -- 172.21.15.171:0/2794203020 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd6e0046ce0 con 0x7fd6cc034050 2024-02-23T00:23:21.141 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mon update... 2024-02-23T00:23:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 10 librados: watch_flush enter 2024-02-23T00:23:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 10 librados: watch_flush exit 2024-02-23T00:23:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 20 client.14176.objecter shutdown clearing up homeless session... 2024-02-23T00:23:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 10 client.14176.objecter successfully canceled tick 2024-02-23T00:23:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd6cc034050 msgr2=0x7fd6cc036510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 --2- 172.21.15.171:0/2794203020 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd6cc034050 0x7fd6cc036510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd6dc009780 tx=0x7fd6dc006cc0).stop 2024-02-23T00:23:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 msgr2=0x7fd6e010d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 --2- 172.21.15.171:0/2794203020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 0x7fd6e010d110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd6d0009b10 tx=0x7fd6d0009cb0).stop 2024-02-23T00:23:21.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 shutdown_connections 2024-02-23T00:23:21.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 --2- 172.21.15.171:0/2794203020 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd6cc034050 0x7fd6cc036510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:21.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 --2- 172.21.15.171:0/2794203020 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6e00f8690 0x7fd6e010d110 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:21.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 >> 172.21.15.171:0/2794203020 conn(0x7fd6e006c440 msgr2=0x7fd6e00fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:21.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 shutdown_connections 2024-02-23T00:23:21.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 -- 172.21.15.171:0/2794203020 wait complete. 2024-02-23T00:23:21.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:21.141+0000 7fd6e8439700 1 librados: shutdown 2024-02-23T00:23:21.751 DEBUG:teuthology.orchestra.run.smithi087:mon.c> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.c.service 2024-02-23T00:23:21.754 DEBUG:teuthology.orchestra.run.smithi171:mon.b> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.b.service 2024-02-23T00:23:21.757 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-23T00:23:21.757 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:23:21.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:23:21.790 INFO:journalctl@ceph.mon.c.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:23:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:20.059922+0000 mgr.y (mgr.14146) 32 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:21.140062+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:21.187886+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:21.189478+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:21.190235+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:21.193615+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:21.195276+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:22.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:21.196008+0000 mon.a (mon.0) 121 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:23.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:21.135062+0000 mgr.y (mgr.14146) 33 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi087:172.21.15.87=a;smithi087:[v2:172.21.15.87:3301,v1:172.21.15.87:6790]=c;smithi171:172.21.15.171=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:23.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:21.137302+0000 mgr.y (mgr.14146) 34 : cephadm [INF] Saving service mon spec with placement smithi087:172.21.15.87=a;smithi087:[v2:172.21.15.87:3301,v1:172.21.15.87:6790]=c;smithi171:172.21.15.171=b;count:3 2024-02-23T00:23:23.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:21.196619+0000 mgr.y (mgr.14146) 35 : cephadm [INF] Deploying daemon mon.b on smithi171 2024-02-23T00:23:23.691 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.689+0000 7fad3afda700 1 -- 172.21.15.171:0/3909610735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 msgr2=0x7fad340fda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.689+0000 7fad3afda700 1 --2- 172.21.15.171:0/3909610735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 0x7fad340fda00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fad28004770 tx=0x7fad28009cb0).stop 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.689+0000 7fad3afda700 1 -- 172.21.15.171:0/3909610735 shutdown_connections 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.689+0000 7fad3afda700 1 --2- 172.21.15.171:0/3909610735 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 0x7fad340fda00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.689+0000 7fad3afda700 1 -- 172.21.15.171:0/3909610735 >> 172.21.15.171:0/3909610735 conn(0x7fad340f3220 msgr2=0x7fad340f5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 -- 172.21.15.171:0/3909610735 shutdown_connections 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 -- 172.21.15.171:0/3909610735 wait complete. 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 librados: starting msgr at 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 librados: starting objecter 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 Processor -- start 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 -- start start 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 librados: setting wanted keys 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 librados: calling monclient init 2024-02-23T00:23:23.692 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 0x7fad3406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:23.693 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.690+0000 7fad3afda700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3406b5c0 con 0x7fad340fd620 2024-02-23T00:23:23.694 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad38d76700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 0x7fad3406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:23.694 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad38d76700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 0x7fad3406b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.171:54054/0 (socket says 172.21.15.171:54054) 2024-02-23T00:23:23.694 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad38d76700 1 -- 172.21.15.171:0/4267373244 learned_addr learned my addr 172.21.15.171:0/4267373244 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad38d76700 1 -- 172.21.15.171:0/4267373244 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad28005040 con 0x7fad340fd620 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad38d76700 1 --2- 172.21.15.171:0/4267373244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 0x7fad3406b0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fad28009b10 tx=0x7fad28009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad31ffb700 10 client.?.objecter ms_handle_connect 0x7fad340fd620 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad31ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad31ffb700 1 -- 172.21.15.171:0/4267373244 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad2800e030 con 0x7fad340fd620 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3406b7c0 con 0x7fad340fd620 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad31ffb700 1 -- 172.21.15.171:0/4267373244 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad28009430 con 0x7fad340fd620 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.691+0000 7fad31ffb700 1 -- 172.21.15.171:0/4267373244 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad28018070 con 0x7fad340fd620 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.692+0000 7fad3afda700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.692+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3406bc50 con 0x7fad340fd620 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.692+0000 7fad3afda700 1 librados: init done 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.692+0000 7fad31ffb700 1 -- 172.21.15.171:0/4267373244 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7fad2801d420 con 0x7fad340fd620 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.692+0000 7fad31ffb700 1 --2- 172.21.15.171:0/4267373244 >> v2:172.21.15.87:6800/1121018082 conn(0x7fad24034000 0x7fad240364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.692+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3405f150 con 0x7fad340fd620 2024-02-23T00:23:23.695 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.693+0000 7fad31ffb700 1 -- 172.21.15.171:0/4267373244 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fad28044240 con 0x7fad340fd620 2024-02-23T00:23:23.696 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.693+0000 7fad31ffb700 10 client.14178.objecter ms_dispatch 0x7fad34001550 osd_map(4..4 src has 1..4) v4 2024-02-23T00:23:23.696 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.693+0000 7fad31ffb700 3 client.14178.objecter handle_osd_map got epochs [4,4] > 0 2024-02-23T00:23:23.696 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.693+0000 7fad31ffb700 3 client.14178.objecter handle_osd_map decoding full epoch 4 2024-02-23T00:23:23.696 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.693+0000 7fad31ffb700 20 client.14178.objecter dump_active .. 0 homeless 2024-02-23T00:23:23.696 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.693+0000 7fad33fff700 1 --2- 172.21.15.171:0/4267373244 >> v2:172.21.15.87:6800/1121018082 conn(0x7fad24034000 0x7fad240364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:23.696 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.693+0000 7fad31ffb700 10 client.14178.objecter ms_handle_connect 0x7fad24034000 2024-02-23T00:23:23.696 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.693+0000 7fad33fff700 1 --2- 172.21.15.171:0/4267373244 >> v2:172.21.15.87:6800/1121018082 conn(0x7fad24034000 0x7fad240364c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fad20009780 tx=0x7fad20006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:23.699 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.698+0000 7fad31ffb700 1 -- 172.21.15.171:0/4267373244 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fad28023e80 con 0x7fad340fd620 2024-02-23T00:23:23.927 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.925+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad3405a310 con 0x7fad340fd620 2024-02-23T00:23:23.927 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.926+0000 7fad31ffb700 1 -- 172.21.15.171:0/4267373244 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+723 (secure 0 0 0) 0x7fad28023e80 con 0x7fad340fd620 2024-02-23T00:23:23.927 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:23:23.928 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":1,"fsid":"66db207c-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:21:26.056087Z","created":"2024-02-23T00:21:26.056087Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-23T00:23:23.929 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 10 librados: watch_flush enter 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 10 librados: watch_flush exit 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 20 client.14178.objecter shutdown clearing up homeless session... 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 10 client.14178.objecter successfully canceled tick 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 >> v2:172.21.15.87:6800/1121018082 conn(0x7fad24034000 msgr2=0x7fad240364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 --2- 172.21.15.171:0/4267373244 >> v2:172.21.15.87:6800/1121018082 conn(0x7fad24034000 0x7fad240364c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fad20009780 tx=0x7fad20006cc0).stop 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 msgr2=0x7fad3406b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 --2- 172.21.15.171:0/4267373244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 0x7fad3406b0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fad28009b10 tx=0x7fad28009cb0).stop 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 shutdown_connections 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 --2- 172.21.15.171:0/4267373244 >> v2:172.21.15.87:6800/1121018082 conn(0x7fad24034000 0x7fad240364c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 --2- 172.21.15.171:0/4267373244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fad340fd620 0x7fad3406b0d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 >> 172.21.15.171:0/4267373244 conn(0x7fad340f3220 msgr2=0x7fad340fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 shutdown_connections 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 -- 172.21.15.171:0/4267373244 wait complete. 2024-02-23T00:23:23.930 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:23.928+0000 7fad3afda700 1 librados: shutdown 2024-02-23T00:23:23.931 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 1 2024-02-23T00:23:24.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:22.060185+0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:25.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:23.927017+0000 mon.a (mon.0) 122 : audit [DBG] from='client.? 172.21.15.171:0/4267373244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:23:25.452 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-23T00:23:25.453 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:23:26.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:24.060555+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:26 smithi171 systemd[1]: Starting Ceph mon.b for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:23:28.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:26.060900+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:29.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.360+0000 7f2fb7317700 1 -- 172.21.15.171:0/2018940129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 msgr2=0x7f2fb006b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:29.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.360+0000 7f2fb7317700 1 --2- 172.21.15.171:0/2018940129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 0x7f2fb006b4f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f2fac004770 tx=0x7f2fac009cb0).stop 2024-02-23T00:23:29.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.360+0000 7f2fb7317700 1 -- 172.21.15.171:0/2018940129 shutdown_connections 2024-02-23T00:23:29.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.360+0000 7f2fb7317700 1 --2- 172.21.15.171:0/2018940129 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 0x7f2fb006b4f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:29.421 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.360+0000 7f2fb7317700 1 -- 172.21.15.171:0/2018940129 >> 172.21.15.171:0/2018940129 conn(0x7f2fb00f3220 msgr2=0x7f2fb00f5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.360+0000 7f2fb7317700 1 -- 172.21.15.171:0/2018940129 shutdown_connections 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.360+0000 7f2fb7317700 1 -- 172.21.15.171:0/2018940129 wait complete. 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb7317700 1 librados: starting msgr at 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb7317700 1 librados: starting objecter 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb7317700 1 Processor -- start 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb7317700 1 -- start start 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb7317700 1 librados: setting wanted keys 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb7317700 1 librados: calling monclient init 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb7317700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 0x7f2fb010f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:29.422 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb7317700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb010f7b0 con 0x7f2fb006b110 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb50b3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 0x7f2fb010f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb50b3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 0x7f2fb010f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.171:54072/0 (socket says 172.21.15.171:54072) 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.361+0000 7f2fb50b3700 1 -- 172.21.15.171:0/2907410271 learned_addr learned my addr 172.21.15.171:0/2907410271 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fb50b3700 1 -- 172.21.15.171:0/2907410271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac005040 con 0x7f2fb006b110 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fb50b3700 1 --2- 172.21.15.171:0/2907410271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 0x7f2fb010f2c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f2fac009b10 tx=0x7f2fac009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fa67fc700 10 client.?.objecter ms_handle_connect 0x7f2fb006b110 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fa67fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fa67fc700 1 -- 172.21.15.171:0/2907410271 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fac00e030 con 0x7f2fb006b110 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fa67fc700 1 -- 172.21.15.171:0/2907410271 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fac009430 con 0x7f2fb006b110 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb010f9b0 con 0x7f2fb006b110 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fa67fc700 1 -- 172.21.15.171:0/2907410271 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fac018070 con 0x7f2fb006b110 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fb7317700 10 client.14180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb010fdc0 con 0x7f2fb006b110 2024-02-23T00:23:29.423 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.362+0000 7f2fb7317700 1 librados: init done 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.363+0000 7f2fa67fc700 1 -- 172.21.15.171:0/2907410271 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7f2fac027440 con 0x7f2fb006b110 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.363+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb005f150 con 0x7f2fb006b110 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.363+0000 7f2fa67fc700 1 --2- 172.21.15.171:0/2907410271 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2f9c034000 0x7f2f9c0364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.363+0000 7f2fa67fc700 1 -- 172.21.15.171:0/2907410271 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f2fac044540 con 0x7f2fb006b110 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.363+0000 7f2fa67fc700 10 client.14180.objecter ms_dispatch 0x7f2fb0001550 osd_map(4..4 src has 1..4) v4 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.363+0000 7f2fa67fc700 3 client.14180.objecter handle_osd_map got epochs [4,4] > 0 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.363+0000 7f2fa67fc700 3 client.14180.objecter handle_osd_map decoding full epoch 4 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.363+0000 7f2fa67fc700 20 client.14180.objecter dump_active .. 0 homeless 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.363+0000 7f2fb48b2700 1 --2- 172.21.15.171:0/2907410271 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2f9c034000 0x7f2f9c0364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.364+0000 7f2fa67fc700 10 client.14180.objecter ms_handle_connect 0x7f2f9c034000 2024-02-23T00:23:29.424 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.364+0000 7f2fb48b2700 1 --2- 172.21.15.171:0/2907410271 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2f9c034000 0x7f2f9c0364c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f2fa0009780 tx=0x7f2fa0006cc0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:29.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.368+0000 7f2fa67fc700 1 -- 172.21.15.171:0/2907410271 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2fac01b960 con 0x7f2fb006b110 2024-02-23T00:23:29.425 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.405+0000 7f2fa67fc700 1 -- 172.21.15.171:0/2907410271 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f2fac01b960 con 0x7f2fb006b110 2024-02-23T00:23:29.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:29.605+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fb0046ce0 con 0x7f2fb006b110 2024-02-23T00:23:29.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.370+0000 7f0bce423880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.370+0000 7f0bce423880 0 ceph version 16.2.14-550-geb66ed92 (eb66ed921e744301e6863be6353618d63967ea59) pacific (stable), process ceph-mon, pid 7 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.371+0000 7f0bce423880 0 pidfile_write: ignore empty --pid-file 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.375+0000 7f0bce423880 0 load: jerasure load: lrc load: isa 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: RocksDB version: 6.8.1 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Compile date Feb 20 2024 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: DB SUMMARY 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: CURRENT file: CURRENT 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 511 ; 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.error_if_exists: 0 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.create_if_missing: 0 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.paranoid_checks: 1 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.env: 0x55cd3fd6d080 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.fs: Posix File System 2024-02-23T00:23:29.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.info_log: 0x55cd4103c260 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.statistics: (nil) 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.use_fsync: 0 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_log_file_size: 0 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.allow_fallocate: 1 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.use_direct_reads: 0 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-23T00:23:29.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.db_log_dir: 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_subcompactions: 1 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_background_flushes: -1 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-23T00:23:29.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.write_buffer_manager: 0x55cd41d2e3c0 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.unordered_write: 0 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-23T00:23:29.699 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.row_cache: None 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.wal_filter: None 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.preserve_deletes: 0 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.two_write_queues: 0 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.atomic_flush: 0 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-23T00:23:29.700 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.log_readahead_size: 0 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_background_jobs: 2 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_background_compactions: -1 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-23T00:23:29.701 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_open_files: -1 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Compression algorithms supported: 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: kZSTD supported: 0 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: kXpressCompression supported: 0 2024-02-23T00:23:29.702 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: kLZ4Compression supported: 1 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: kBZip2Compression supported: 0 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: kZlibCompression supported: 1 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: kSnappyCompression supported: 1 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.merge_operator: 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_filter: None 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-23T00:23:29.703 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cd40fe1d10) 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cache_index_and_filter_blocks: 1 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: pin_top_level_index_and_filter: 1 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: index_type: 0 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: data_block_index_type: 0 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: index_shortening: 1 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: data_block_hash_table_util_ratio: 0.750000 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: hash_index_allow_collision: 1 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: checksum: 1 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: no_block_cache: 0 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: block_cache: 0x55cd41016f10 2024-02-23T00:23:29.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: block_cache_name: BinnedLRUCache 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: block_cache_options: 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: capacity : 536870912 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: num_shard_bits : 4 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: strict_capacity_limit : 0 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: high_pri_pool_ratio: 0.000 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: block_cache_compressed: (nil) 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: persistent_cache: (nil) 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: block_size: 4096 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: block_size_deviation: 10 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: block_restart_interval: 16 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: index_block_restart_interval: 1 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: metadata_block_size: 4096 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: partition_filters: 0 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: use_delta_encoding: 1 2024-02-23T00:23:29.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: whole_key_filtering: 1 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: verify_compression: 0 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: read_amp_bytes_per_bit: 0 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: format_version: 2 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: enable_index_compression: 1 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: block_align: 0 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compression: NoCompression 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.num_levels: 7 2024-02-23T00:23:29.706 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:23:29.707 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-23T00:23:29.708 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-23T00:23:29.709 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.table_properties_collectors: 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.inplace_update_support: 0 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.bloom_locality: 0 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.max_successive_merges: 0 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-23T00:23:29.710 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.ttl: 2592000 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.376+0000 7f0bce423880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.377+0000 7f0bce423880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647809377672, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.377+0000 7f0bce423880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.377+0000 7f0bce423880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.377+0000 7f0bce423880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647809378139, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708647809, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-23T00:23:29.711 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.377+0000 7f0bce423880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.377+0000 7f0bce423880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647809378513, "job": 1, "event": "recovery_finished"} 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.378+0000 7f0bce423880 4 rocksdb: DB pointer 0x55cd410df800 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.378+0000 7f0bb5215700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.378+0000 7f0bb5215700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ** DB Stats ** 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:23:29.712 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-23T00:23:29.713 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-23T00:23:29.713 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:23:29.713 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.713 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ** Compaction Stats [default] ** 2024-02-23T00:23:29.713 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:23:29.713 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:23:29.713 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:29.713 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:29.713 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ** Compaction Stats [default] ** 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:23:29.714 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ** Compaction Stats [default] ** 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ** Compaction Stats [default] ** 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:23:29.715 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.716 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.378+0000 7f0bce423880 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.378+0000 7f0bce423880 0 using public_addr v2:172.21.15.171:0/0 -> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.379+0000 7f0bce423880 0 starting mon.b rank -1 at public addrs [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] at bind addrs [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.380+0000 7f0bce423880 1 mon.b@-1(???) e0 preinit fsid 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.399+0000 7f0bb7a1a700 0 mon.b@-1(synchronizing).mds e1 new map 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.399+0000 7f0bb7a1a700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: e1 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: legacy client fscid: -1 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: No filesystems configured 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.399+0000 7f0bb7a1a700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-23T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.399+0000 7f0bb3211700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.399+0000 7f0bbd225700 4 rocksdb: (Original Log Time 2024/02/23-00:23:29.400578) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.399+0000 7f0bbd225700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bbd225700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647809400760, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 931, "num_deletes": 15, "total_data_size": 3847694, "memory_usage": 3864824, "flush_reason": "Manual Compaction"} 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bbd225700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:23:29.719 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:23:29.719 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:26.060900+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:29.719 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:23:29.400+0000 7f0bb7a1a700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-23T00:23:29.719 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 bash[39549]: ea598a016bef8d654210fbd4c26db0a0cd4fd0ab9576c503bed9ddbcc9475031 2024-02-23T00:23:29.719 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:29 smithi171 systemd[1]: Started Ceph mon.b for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:23:34.364 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.363+0000 7f2fb6b16700 10 client.14180.objecter tick 2024-02-23T00:23:34.457 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.456+0000 7f2fa67fc700 1 -- 172.21.15.171:0/2907410271 <== mon.0 v2:172.21.15.87:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+993 (secure 0 0 0) 0x7f2fac01b370 con 0x7f2fb006b110 2024-02-23T00:23:34.458 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:23:34.458 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":2,"fsid":"66db207c-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:23:29.405303Z","created":"2024-02-23T00:21:26.056087Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:3300","nonce":0},{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-23T00:23:34.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 10 librados: watch_flush enter 2024-02-23T00:23:34.460 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 10 librados: watch_flush exit 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 20 client.14180.objecter shutdown clearing up homeless session... 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 10 client.14180.objecter successfully canceled tick 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2f9c034000 msgr2=0x7f2f9c0364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 --2- 172.21.15.171:0/2907410271 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2f9c034000 0x7f2f9c0364c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f2fa0009780 tx=0x7f2fa0006cc0).stop 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 msgr2=0x7f2fb010f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 --2- 172.21.15.171:0/2907410271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 0x7f2fb010f2c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f2fac009b10 tx=0x7f2fac009cb0).stop 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 shutdown_connections 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 --2- 172.21.15.171:0/2907410271 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2f9c034000 0x7f2f9c0364c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 --2- 172.21.15.171:0/2907410271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2fb006b110 0x7f2fb010f2c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 >> 172.21.15.171:0/2907410271 conn(0x7f2fb00f3220 msgr2=0x7f2fb0101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 shutdown_connections 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 -- 172.21.15.171:0/2907410271 wait complete. 2024-02-23T00:23:34.461 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:34.459+0000 7f2fb7317700 1 librados: shutdown 2024-02-23T00:23:34.462 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 2 2024-02-23T00:23:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:29.407908+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:23:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:29.408109+0000 mon.a (mon.0) 125 : cluster [INF] mon.a calling monitor election 2024-02-23T00:23:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:29.424249+0000 mon.a (mon.0) 126 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:29.606563+0000 mon.a (mon.0) 127 : audit [DBG] from='client.? 172.21.15.171:0/2907410271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:23:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:30.061597+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:30.405630+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:31.405164+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:31.406424+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:32.061919+0000 mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:32.405320+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:33.405364+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:34.405603+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:34.438191+0000 mon.a (mon.0) 133 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:34.443365+0000 mon.a (mon.0) 134 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0],b=[v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0]} removed_ranks: {} 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:34.455941+0000 mon.a (mon.0) 135 : cluster [DBG] fsmap 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:34.455991+0000 mon.a (mon.0) 136 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:34.456194+0000 mon.a (mon.0) 137 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:34.456328+0000 mon.a (mon.0) 138 : cluster [INF] overall HEALTH_OK 2024-02-23T00:23:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:34.460070+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:34.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:34.461126+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:34.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:34.461857+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:29.407908+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:23:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:29.408109+0000 mon.a (mon.0) 125 : cluster [INF] mon.a calling monitor election 2024-02-23T00:23:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:29.424249+0000 mon.a (mon.0) 126 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:29.606563+0000 mon.a (mon.0) 127 : audit [DBG] from='client.? 172.21.15.171:0/2907410271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:23:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:30.061597+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:30.405630+0000 mon.a (mon.0) 128 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:31.405164+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:31.406424+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:32.061919+0000 mgr.y (mgr.14146) 41 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:32.405320+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:33.405364+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:34.405603+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:34.438191+0000 mon.a (mon.0) 133 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:34.443365+0000 mon.a (mon.0) 134 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0],b=[v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0]} removed_ranks: {} 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:34.455941+0000 mon.a (mon.0) 135 : cluster [DBG] fsmap 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:34.455991+0000 mon.a (mon.0) 136 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:34.456194+0000 mon.a (mon.0) 137 : cluster [DBG] mgrmap e13: y(active, since 72s) 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:34.456328+0000 mon.a (mon.0) 138 : cluster [INF] overall HEALTH_OK 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:34.460070+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:34.461126+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:34.461857+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:35.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:34.062232+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:35.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:34.462489+0000 mgr.y (mgr.14146) 43 : cephadm [INF] Deploying daemon mon.c on smithi087 2024-02-23T00:23:35.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:35.406002+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:35.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:34.062232+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:35.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:34.462489+0000 mgr.y (mgr.14146) 43 : cephadm [INF] Deploying daemon mon.c on smithi087 2024-02-23T00:23:35.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:35.406002+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:37.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 systemd[1]: Starting Ceph mon.c for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:23:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:36.062461+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:38.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:36.062461+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:38.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.691+0000 7f8a4bc91880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-23T00:23:38.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.691+0000 7f8a4bc91880 0 ceph version 16.2.14-550-geb66ed92 (eb66ed921e744301e6863be6353618d63967ea59) pacific (stable), process ceph-mon, pid 7 2024-02-23T00:23:38.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.691+0000 7f8a4bc91880 0 pidfile_write: ignore empty --pid-file 2024-02-23T00:23:38.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.695+0000 7f8a4bc91880 0 load: jerasure load: lrc load: isa 2024-02-23T00:23:38.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: RocksDB version: 6.8.1 2024-02-23T00:23:38.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-23T00:23:38.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Compile date Feb 20 2024 2024-02-23T00:23:38.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: DB SUMMARY 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: CURRENT file: CURRENT 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000003.log size: 636 ; 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.error_if_exists: 0 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.create_if_missing: 0 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.paranoid_checks: 1 2024-02-23T00:23:38.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.env: 0x558d8dd5e080 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.fs: Posix File System 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.info_log: 0x558d8f7a0260 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.statistics: (nil) 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.use_fsync: 0 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_log_file_size: 0 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.allow_fallocate: 1 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.use_direct_reads: 0 2024-02-23T00:23:38.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.db_log_dir: 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_subcompactions: 1 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_background_flushes: -1 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-23T00:23:38.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.write_buffer_manager: 0x558d90492390 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.unordered_write: 0 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-23T00:23:38.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.row_cache: None 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.wal_filter: None 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.preserve_deletes: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.two_write_queues: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.atomic_flush: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.log_readahead_size: 0 2024-02-23T00:23:38.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_background_jobs: 2 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_background_compactions: -1 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.max_open_files: -1 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-23T00:23:38.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Compression algorithms supported: 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: kZSTD supported: 0 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: kXpressCompression supported: 0 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: kLZ4Compression supported: 1 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: kBZip2Compression supported: 0 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: kZlibCompression supported: 1 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: kSnappyCompression supported: 1 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.696+0000 7f8a4bc91880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2024-02-23T00:23:38.092 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.merge_operator: 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_filter: None 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558d8f745d10) 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cache_index_and_filter_blocks: 1 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: pin_top_level_index_and_filter: 1 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: index_type: 0 2024-02-23T00:23:38.093 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: data_block_index_type: 0 2024-02-23T00:23:38.094 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: index_shortening: 1 2024-02-23T00:23:38.094 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: data_block_hash_table_util_ratio: 0.750000 2024-02-23T00:23:38.094 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: hash_index_allow_collision: 1 2024-02-23T00:23:38.094 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: checksum: 1 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: no_block_cache: 0 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: block_cache: 0x558d8f77af10 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: block_cache_name: BinnedLRUCache 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: block_cache_options: 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: capacity : 536870912 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: num_shard_bits : 4 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: strict_capacity_limit : 0 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: high_pri_pool_ratio: 0.000 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: block_cache_compressed: (nil) 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: persistent_cache: (nil) 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: block_size: 4096 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: block_size_deviation: 10 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: block_restart_interval: 16 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: index_block_restart_interval: 1 2024-02-23T00:23:38.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: metadata_block_size: 4096 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: partition_filters: 0 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: use_delta_encoding: 1 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: whole_key_filtering: 1 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: verify_compression: 0 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: read_amp_bytes_per_bit: 0 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: format_version: 2 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: enable_index_compression: 1 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: block_align: 0 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compression: NoCompression 2024-02-23T00:23:38.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.num_levels: 7 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-23T00:23:38.097 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-23T00:23:38.098 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-23T00:23:38.099 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.table_properties_collectors: 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.inplace_update_support: 0 2024-02-23T00:23:38.100 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.bloom_locality: 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.max_successive_merges: 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.ttl: 2592000 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.101 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647817698974, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.697+0000 7f8a4bc91880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.698+0000 7f8a4bc91880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647817699399, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1562, "table_properties": {"data_size": 648, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708647817, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.698+0000 7f8a4bc91880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.698+0000 7f8a4bc91880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647817699769, "job": 1, "event": "recovery_finished"} 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.699+0000 7f8a4bc91880 4 rocksdb: DB pointer 0x558d8f843800 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.699+0000 7f8a32a83700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.699+0000 7f8a32a83700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ** DB Stats ** 2024-02-23T00:23:38.102 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ** Compaction Stats [default] ** 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:38.103 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ** Compaction Stats [default] ** 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:23:38.104 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ** Compaction Stats [default] ** 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ** Compaction Stats [default] ** 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-23T00:23:38.105 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: AddFile(Total Files): cumulative 0, interval 0 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: AddFile(Keys): cumulative 0, interval 0 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: ** File Read Latency Histogram By Level [default] ** 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.106 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.699+0000 7f8a4bc91880 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.699+0000 7f8a4bc91880 0 using public_addrv [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.699+0000 7f8a4bc91880 0 starting mon.c rank -1 at public addrs [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] at bind addrs [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.700+0000 7f8a4bc91880 1 mon.c@-1(???) e0 preinit fsid 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.716+0000 7f8a35288700 0 mon.c@-1(synchronizing).mds e1 new map 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.716+0000 7f8a35288700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: e1 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: legacy client fscid: -1 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: No filesystems configured 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.716+0000 7f8a35288700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-23T00:23:38.107 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.716+0000 7f8a30a7f700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.716+0000 7f8a3aa93700 4 rocksdb: (Original Log Time 2024/02/23-00:23:37.718155) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.716+0000 7f8a3aa93700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.716+0000 7f8a3aa93700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708647817718206, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 963, "num_deletes": 15, "total_data_size": 4011752, "memory_usage": 4029656, "flush_reason": "Manual Compaction"} 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.716+0000 7f8a3aa93700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.716+0000 7f8a35288700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.717+0000 7f8a35288700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.717+0000 7f8a35288700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.717+0000 7f8a35288700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.717+0000 7f8a35288700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.717+0000 7f8a35288700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.717+0000 7f8a35288700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:23:38.108 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.717+0000 7f8a35288700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:23:38.109 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.717+0000 7f8a35288700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-23T00:23:38.109 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:36.062461+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:38.109 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:23:37.717+0000 7f8a35288700 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-23T00:23:38.109 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 bash[44808]: 335fb3a114f3cfb4977648f01e0df395c35dcaeafa4e905510f8ad6b3aaeaa80 2024-02-23T00:23:38.109 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:37 smithi087 systemd[1]: Started Ceph mon.c for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:23:40.169 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-23T00:23:40.170 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph mon dump -f json 2024-02-23T00:23:43.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:37.726659+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:37.726814+0000 mon.a (mon.0) 145 : cluster [INF] mon.a calling monitor election 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:37.730492+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:37.740179+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:37.740288+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:38.062783+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:38.721687+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:39.721607+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:39.723503+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:40.063090+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:40.722008+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:41.722033+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:42.063382+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:42.722104+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:42.754186+0000 mon.a (mon.0) 153 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:42.759763+0000 mon.a (mon.0) 154 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0],b=[v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0],c=[v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0]} removed_ranks: {} 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:42.771972+0000 mon.a (mon.0) 155 : cluster [DBG] fsmap 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:37.726659+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:37.726814+0000 mon.a (mon.0) 145 : cluster [INF] mon.a calling monitor election 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:37.730492+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-23T00:23:43.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:37.740179+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:37.740288+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:38.062783+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:38.721687+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:39.721607+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:39.723503+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:40.063090+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:40.722008+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:41.722033+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:42.063382+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:42.722104+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:42.754186+0000 mon.a (mon.0) 153 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:42.759763+0000 mon.a (mon.0) 154 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0],b=[v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0],c=[v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0]} removed_ranks: {} 2024-02-23T00:23:43.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:42.771972+0000 mon.a (mon.0) 155 : cluster [DBG] fsmap 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:42.772021+0000 mon.a (mon.0) 156 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:42.772233+0000 mon.a (mon.0) 157 : cluster [DBG] mgrmap e13: y(active, since 80s) 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:42.772373+0000 mon.a (mon.0) 158 : cluster [INF] overall HEALTH_OK 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:42.775978+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:42.777400+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:42.772021+0000 mon.a (mon.0) 156 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:42.772233+0000 mon.a (mon.0) 157 : cluster [DBG] mgrmap e13: y(active, since 80s) 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:42.772373+0000 mon.a (mon.0) 158 : cluster [INF] overall HEALTH_OK 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:42.775978+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:42.777400+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:37.726659+0000 mon.a (mon.0) 144 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:23:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:37.726814+0000 mon.a (mon.0) 145 : cluster [INF] mon.a calling monitor election 2024-02-23T00:23:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:37.730492+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-23T00:23:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:37.740179+0000 mon.a (mon.0) 146 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:23:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:37.740288+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:38.062783+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:38.721687+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:39.721607+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:39.723503+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:40.063090+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:40.722008+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:41.722033+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:42.063382+0000 mgr.y (mgr.14146) 47 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:42.722104+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:42.754186+0000 mon.a (mon.0) 153 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:42.759763+0000 mon.a (mon.0) 154 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0],b=[v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0],c=[v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0]} removed_ranks: {} 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:42.771972+0000 mon.a (mon.0) 155 : cluster [DBG] fsmap 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:42.772021+0000 mon.a (mon.0) 156 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-23T00:23:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:42.772233+0000 mon.a (mon.0) 157 : cluster [DBG] mgrmap e13: y(active, since 80s) 2024-02-23T00:23:43.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:42.772373+0000 mon.a (mon.0) 158 : cluster [INF] overall HEALTH_OK 2024-02-23T00:23:43.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:42.775978+0000 mon.a (mon.0) 159 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:43.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:42.777400+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:44.110 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:43.722310+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:44.110 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:43.828598+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:44.325 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.322+0000 7f1c77efc700 1 -- 172.21.15.171:0/1145385873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 msgr2=0x7f1c700fc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:44.325 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.323+0000 7f1c77efc700 1 --2- 172.21.15.171:0/1145385873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 0x7f1c700fc910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f1c64004770 tx=0x7f1c64009cb0).stop 2024-02-23T00:23:44.325 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.323+0000 7f1c77efc700 1 -- 172.21.15.171:0/1145385873 shutdown_connections 2024-02-23T00:23:44.325 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.323+0000 7f1c77efc700 1 --2- 172.21.15.171:0/1145385873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 0x7f1c700fc910 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:44.325 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.323+0000 7f1c77efc700 1 -- 172.21.15.171:0/1145385873 >> 172.21.15.171:0/1145385873 conn(0x7f1c700f2100 msgr2=0x7f1c700f4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:44.325 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.323+0000 7f1c77efc700 1 -- 172.21.15.171:0/1145385873 shutdown_connections 2024-02-23T00:23:44.325 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.323+0000 7f1c77efc700 1 -- 172.21.15.171:0/1145385873 wait complete. 2024-02-23T00:23:44.328 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.323+0000 7f1c77efc700 1 librados: starting msgr at 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.323+0000 7f1c77efc700 1 librados: starting objecter 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 Processor -- start 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 -- start start 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 librados: setting wanted keys 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 librados: calling monclient init 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 0x7f1c701083c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c701088b0 0x7f1c70111270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c70108d30 0x7f1c70113770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c70113e30 con 0x7f1c700fc530 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c70113fa0 con 0x7f1c70108d30 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c77efc700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c70114160 con 0x7f1c701088b0 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c75c98700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 0x7f1c701083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c76499700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c70108d30 0x7f1c70113770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:44.329 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c75c98700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 0x7f1c701083c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.171:56726/0 (socket says 172.21.15.171:56726) 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c75c98700 1 -- 172.21.15.171:0/203138577 learned_addr learned my addr 172.21.15.171:0/203138577 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c75497700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c701088b0 0x7f1c70111270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c76499700 1 -- 172.21.15.171:0/203138577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c70108d30 msgr2=0x7f1c70113770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c76499700 1 -- 172.21.15.171:0/203138577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c70108d30 msgr2=0x7f1c70113770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c76499700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c70108d30 0x7f1c70113770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c76499700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c70108d30 0x7f1c70113770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c75497700 1 -- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c701088b0 msgr2=0x7f1c70111270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c75497700 1 -- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c701088b0 msgr2=0x7f1c70111270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c75497700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c701088b0 0x7f1c70111270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.324+0000 7f1c75497700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c701088b0 0x7f1c70111270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c75c98700 1 -- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c701088b0 msgr2=0x7f1c70111270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c75c98700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c701088b0 0x7f1c70111270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c75c98700 1 -- 172.21.15.171:0/203138577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c70108d30 msgr2=0x7f1c70113770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:23:44.330 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c75c98700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c70108d30 0x7f1c70113770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c75c98700 1 -- 172.21.15.171:0/203138577 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c64005040 con 0x7f1c700fc530 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c75c98700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 0x7f1c701083c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f1c64009b10 tx=0x7f1c64009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c62ffd700 10 client.?.objecter ms_handle_connect 0x7f1c700fc530 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c62ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c62ffd700 1 -- 172.21.15.171:0/203138577 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6400e030 con 0x7f1c700fc530 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c62ffd700 1 -- 172.21.15.171:0/203138577 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c640093b0 con 0x7f1c700fc530 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c701143e0 con 0x7f1c700fc530 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c62ffd700 1 -- 172.21.15.171:0/203138577 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64018070 con 0x7f1c700fc530 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c77efc700 10 client.14184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c701148f0 con 0x7f1c700fc530 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.325+0000 7f1c77efc700 1 librados: init done 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c62ffd700 1 -- 172.21.15.171:0/203138577 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7f1c6401d470 con 0x7f1c700fc530 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c62ffd700 1 --2- 172.21.15.171:0/203138577 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1c5c0344a0 0x7f1c5c036960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:44.331 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c62ffd700 1 -- 172.21.15.171:0/203138577 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f1c64044200 con 0x7f1c700fc530 2024-02-23T00:23:44.332 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c62ffd700 10 client.14184.objecter ms_dispatch 0x7f1c70001550 osd_map(4..4 src has 1..4) v4 2024-02-23T00:23:44.332 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c62ffd700 3 client.14184.objecter handle_osd_map got epochs [4,4] > 0 2024-02-23T00:23:44.332 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c62ffd700 3 client.14184.objecter handle_osd_map decoding full epoch 4 2024-02-23T00:23:44.332 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c62ffd700 20 client.14184.objecter dump_active .. 0 homeless 2024-02-23T00:23:44.332 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c7005fe40 con 0x7f1c700fc530 2024-02-23T00:23:44.332 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c75497700 1 --2- 172.21.15.171:0/203138577 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1c5c0344a0 0x7f1c5c036960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:44.332 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c62ffd700 10 client.14184.objecter ms_handle_connect 0x7f1c5c0344a0 2024-02-23T00:23:44.332 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.326+0000 7f1c75497700 1 --2- 172.21.15.171:0/203138577 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1c5c0344a0 0x7f1c5c036960 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1c6c006fd0 tx=0x7f1c6c008040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:44.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:43.722310+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:43.828598+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:43.722310+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:23:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:43.828598+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:44.333 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.331+0000 7f1c62ffd700 1 -- 172.21.15.171:0/203138577 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1c6401bcc0 con 0x7f1c700fc530 2024-02-23T00:23:44.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.570+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c700ffda0 con 0x7f1c700fc530 2024-02-23T00:23:44.575 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.571+0000 7f1c62ffd700 1 -- 172.21.15.171:0/203138577 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1259 (secure 0 0 0) 0x7f1c64025340 con 0x7f1c700fc530 2024-02-23T00:23:44.576 INFO:teuthology.orchestra.run.smithi171.stdout: 2024-02-23T00:23:44.576 INFO:teuthology.orchestra.run.smithi171.stdout:{"epoch":3,"fsid":"66db207c-d1e1-11ee-95c0-87774f69a715","modified":"2024-02-23T00:23:37.722458Z","created":"2024-02-23T00:21:26.056087Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3300","nonce":0},{"type":"v1","addr":"172.21.15.87:6789","nonce":0}]},"addr":"172.21.15.87:6789/0","public_addr":"172.21.15.87:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:3300","nonce":0},{"type":"v1","addr":"172.21.15.171:6789","nonce":0}]},"addr":"172.21.15.171:6789/0","public_addr":"172.21.15.171:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:3301","nonce":0},{"type":"v1","addr":"172.21.15.87:6790","nonce":0}]},"addr":"172.21.15.87:6790/0","public_addr":"172.21.15.87:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-23T00:23:44.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 10 librados: watch_flush enter 2024-02-23T00:23:44.581 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 10 librados: watch_flush exit 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 20 client.14184.objecter shutdown clearing up homeless session... 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 10 client.14184.objecter successfully canceled tick 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1c5c0344a0 msgr2=0x7f1c5c036960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 --2- 172.21.15.171:0/203138577 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1c5c0344a0 0x7f1c5c036960 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f1c6c006fd0 tx=0x7f1c6c008040).stop 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 msgr2=0x7f1c701083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 0x7f1c701083c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f1c64009b10 tx=0x7f1c64009cb0).stop 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 shutdown_connections 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c70108d30 0x7f1c70113770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 --2- 172.21.15.171:0/203138577 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1c5c0344a0 0x7f1c5c036960 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c700fc530 0x7f1c701083c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 --2- 172.21.15.171:0/203138577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c701088b0 0x7f1c70111270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.572+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 >> 172.21.15.171:0/203138577 conn(0x7f1c700f2100 msgr2=0x7f1c700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.573+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 shutdown_connections 2024-02-23T00:23:44.582 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.573+0000 7f1c77efc700 1 -- 172.21.15.171:0/203138577 wait complete. 2024-02-23T00:23:44.583 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:44.573+0000 7f1c77efc700 1 librados: shutdown 2024-02-23T00:23:44.583 INFO:teuthology.orchestra.run.smithi171.stderr:dumped monmap epoch 3 2024-02-23T00:23:45.217 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-23T00:23:45.217 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph config generate-minimal-conf 2024-02-23T00:23:45.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:44.063643+0000 mgr.y (mgr.14146) 48 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:45.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:44.572351+0000 mon.a (mon.0) 163 : audit [DBG] from='client.? 172.21.15.171:0/203138577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:23:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:44.063643+0000 mgr.y (mgr.14146) 48 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:44.572351+0000 mon.a (mon.0) 163 : audit [DBG] from='client.? 172.21.15.171:0/203138577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:23:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:44.063643+0000 mgr.y (mgr.14146) 48 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:44.572351+0000 mon.a (mon.0) 163 : audit [DBG] from='client.? 172.21.15.171:0/203138577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:45.528859+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:45.529403+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:45.530088+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:45.530988+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.conf 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:45.531193+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:45.556873+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:45.614762+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:45.620080+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:45.621044+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:45.621255+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:46.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:45.621812+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:45.622332+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:45.622863+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring daemon mon.a on smithi087 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:45.528859+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:45.529403+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:45.530088+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:45.530988+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.conf 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:45.531193+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:45.556873+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:45.614762+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:45.620080+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:45.621044+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-23T00:23:46.730 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:45.621255+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:46.730 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:45.621812+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-23T00:23:46.730 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:45.622332+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:46.730 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:45.622863+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring daemon mon.a on smithi087 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:45.528859+0000 mon.a (mon.0) 164 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:45.529403+0000 mon.a (mon.0) 165 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:45.530088+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:45.530988+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.conf 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:45.531193+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:45.556873+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:45.614762+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:45.620080+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:45.621044+0000 mgr.y (mgr.14146) 51 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-23T00:23:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:45.621255+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:46.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:45.621812+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-23T00:23:46.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:45.622332+0000 mon.a (mon.0) 172 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:46.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:45.622863+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring daemon mon.a on smithi087 2024-02-23T00:23:47.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 -- 172.21.15.87:0/4203303029 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 msgr2=0x7f24e80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:47.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 --2- 172.21.15.87:0/4203303029 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 0x7f24e80fce10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f24d8004870 tx=0x7f24d8009cb0).stop 2024-02-23T00:23:47.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 -- 172.21.15.87:0/4203303029 shutdown_connections 2024-02-23T00:23:47.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 --2- 172.21.15.87:0/4203303029 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24e80f72a0 0x7f24e81038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:47.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 --2- 172.21.15.87:0/4203303029 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 0x7f24e80fce10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:47.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 --2- 172.21.15.87:0/4203303029 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24e80fd390 0x7f24e80f6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:47.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 -- 172.21.15.87:0/4203303029 >> 172.21.15.87:0/4203303029 conn(0x7f24e80f2640 msgr2=0x7f24e80f4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 -- 172.21.15.87:0/4203303029 shutdown_connections 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 -- 172.21.15.87:0/4203303029 wait complete. 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 librados: starting msgr at 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.390+0000 7f24ec951700 1 librados: starting objecter 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 Processor -- start 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 -- start start 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 librados: setting wanted keys 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 librados: calling monclient init 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24e80f72a0 0x7f24e80f8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 0x7f24e80f8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24e80fd390 0x7f24e80f9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e8116540 con 0x7f24e80fd390 2024-02-23T00:23:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e81166b0 con 0x7f24e80f72a0 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24ec951700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e8116870 con 0x7f24e80fca30 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e5d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 0x7f24e80f8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e6d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24e80fd390 0x7f24e80f9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e5d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 0x7f24e80f8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:49908/0 (socket says 172.21.15.87:49908) 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e5d9b700 1 -- 172.21.15.87:0/476025914 learned_addr learned my addr 172.21.15.87:0/476025914 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e659c700 1 --2- 172.21.15.87:0/476025914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24e80f72a0 0x7f24e80f8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e5d9b700 1 -- 172.21.15.87:0/476025914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24e80f72a0 msgr2=0x7f24e80f8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e5d9b700 1 --2- 172.21.15.87:0/476025914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24e80f72a0 0x7f24e80f8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e5d9b700 1 -- 172.21.15.87:0/476025914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24e80fd390 msgr2=0x7f24e80f9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e5d9b700 1 --2- 172.21.15.87:0/476025914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24e80fd390 0x7f24e80f9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e5d9b700 1 -- 172.21.15.87:0/476025914 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc005040 con 0x7f24e80fca30 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e6d9d700 1 --2- 172.21.15.87:0/476025914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24e80fd390 0x7f24e80f9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24e5d9b700 1 --2- 172.21.15.87:0/476025914 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 0x7f24e80f8d20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f24dc0048f0 tx=0x7f24dc009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24d77fe700 10 client.?.objecter ms_handle_connect 0x7f24e80fca30 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.391+0000 7f24d77fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.392+0000 7f24d77fe700 1 -- 172.21.15.87:0/476025914 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc007b00 con 0x7f24e80fca30 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.392+0000 7f24d77fe700 1 -- 172.21.15.87:0/476025914 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24dc00f040 con 0x7f24e80fca30 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.392+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d8005040 con 0x7f24e80fca30 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.392+0000 7f24d77fe700 1 -- 172.21.15.87:0/476025914 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc014990 con 0x7f24e80fca30 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.392+0000 7f24ec951700 10 client.24104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.392+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e8116f30 con 0x7f24e80fca30 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.392+0000 7f24ec951700 1 librados: init done 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.392+0000 7f24d77fe700 1 -- 172.21.15.87:0/476025914 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7f24dc0133c0 con 0x7f24e80fca30 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.392+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e805f860 con 0x7f24e80fca30 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.393+0000 7f24d77fe700 1 --2- 172.21.15.87:0/476025914 >> v2:172.21.15.87:6800/1121018082 conn(0x7f24d0034230 0x7f24d00366f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.393+0000 7f24d77fe700 1 -- 172.21.15.87:0/476025914 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7f24dc042920 con 0x7f24e80fca30 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.393+0000 7f24d77fe700 10 client.24104.objecter ms_dispatch 0x7f24e8001550 osd_map(4..4 src has 1..4) v4 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.393+0000 7f24d77fe700 3 client.24104.objecter handle_osd_map got epochs [4,4] > 0 2024-02-23T00:23:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.393+0000 7f24d77fe700 3 client.24104.objecter handle_osd_map decoding full epoch 4 2024-02-23T00:23:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.393+0000 7f24d77fe700 20 client.24104.objecter dump_active .. 0 homeless 2024-02-23T00:23:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.393+0000 7f24e659c700 1 --2- 172.21.15.87:0/476025914 >> v2:172.21.15.87:6800/1121018082 conn(0x7f24d0034230 0x7f24d00366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.393+0000 7f24e659c700 1 --2- 172.21.15.87:0/476025914 >> v2:172.21.15.87:6800/1121018082 conn(0x7f24d0034230 0x7f24d00366f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f24d8009a00 tx=0x7f24d8009cb0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.393+0000 7f24d77fe700 10 client.24104.objecter ms_handle_connect 0x7f24d0034230 2024-02-23T00:23:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.397+0000 7f24d77fe700 1 -- 172.21.15.87:0/476025914 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f24dc01a270 con 0x7f24e80fca30 2024-02-23T00:23:47.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.566+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f24e805f360 con 0x7f24e80fca30 2024-02-23T00:23:47.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.566+0000 7f24d77fe700 1 -- 172.21.15.87:0/476025914 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7f24dc021170 con 0x7f24e80fca30 2024-02-23T00:23:47.568 INFO:teuthology.orchestra.run.smithi087.stdout:# minimal ceph.conf for 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:23:47.568 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2024-02-23T00:23:47.568 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:23:47.569 INFO:teuthology.orchestra.run.smithi087.stdout: mon_host = [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] 2024-02-23T00:23:47.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 10 librados: watch_flush enter 2024-02-23T00:23:47.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 10 librados: watch_flush exit 2024-02-23T00:23:47.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 20 client.24104.objecter shutdown clearing up homeless session... 2024-02-23T00:23:47.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 10 client.24104.objecter successfully canceled tick 2024-02-23T00:23:47.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 >> v2:172.21.15.87:6800/1121018082 conn(0x7f24d0034230 msgr2=0x7f24d00366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 --2- 172.21.15.87:0/476025914 >> v2:172.21.15.87:6800/1121018082 conn(0x7f24d0034230 0x7f24d00366f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f24d8009a00 tx=0x7f24d8009cb0).stop 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 msgr2=0x7f24e80f8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 --2- 172.21.15.87:0/476025914 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 0x7f24e80f8d20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f24dc0048f0 tx=0x7f24dc009cb0).stop 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 shutdown_connections 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 --2- 172.21.15.87:0/476025914 >> v2:172.21.15.87:6800/1121018082 conn(0x7f24d0034230 0x7f24d00366f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 --2- 172.21.15.87:0/476025914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24e80fd390 0x7f24e80f9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 --2- 172.21.15.87:0/476025914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24e80f72a0 0x7f24e80f8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 --2- 172.21.15.87:0/476025914 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24e80fca30 0x7f24e80f8d20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 >> 172.21.15.87:0/476025914 conn(0x7f24e80f2640 msgr2=0x7f24e80609e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 shutdown_connections 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 -- 172.21.15.87:0/476025914 wait complete. 2024-02-23T00:23:47.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:47.568+0000 7f24ec951700 1 librados: shutdown 2024-02-23T00:23:47.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:46.064011+0000 mgr.y (mgr.14146) 53 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:47.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:47.180846+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:47.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:47.567786+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? 172.21.15.87:0/476025914' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:46.064011+0000 mgr.y (mgr.14146) 53 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:47.180846+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:47.567786+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? 172.21.15.87:0/476025914' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:46.064011+0000 mgr.y (mgr.14146) 53 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:47.180846+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:47.567786+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? 172.21.15.87:0/476025914' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:48.468 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-23T00:23:48.469 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:23:48.469 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.conf 2024-02-23T00:23:48.502 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:23:48.502 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:23:48.570 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:23:48.570 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.conf 2024-02-23T00:23:48.608 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:23:48.608 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:23:48.679 INFO:tasks.cephadm:Adding mgr.y on smithi087 2024-02-23T00:23:48.679 INFO:tasks.cephadm:Adding mgr.x on smithi171 2024-02-23T00:23:48.680 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch apply mgr '2;smithi087=y;smithi171=x' 2024-02-23T00:23:49.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:48.064420+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:48.969311+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:48.970032+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:48.970619+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-23T00:23:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:48.971124+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:48.064420+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:48.969311+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:48.970032+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:48.970619+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-23T00:23:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:48.971124+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:48.064420+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:48.969311+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:48.970032+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:48.970619+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-23T00:23:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:48.971124+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:50.539 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 -- 172.21.15.171:0/1886279968 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fd680 msgr2=0x7fcf340ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:50.539 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 --2- 172.21.15.171:0/1886279968 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fd680 0x7fcf340ffb10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fcf2c009be0 tx=0x7fcf2c004d10).stop 2024-02-23T00:23:50.539 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 -- 172.21.15.171:0/1886279968 shutdown_connections 2024-02-23T00:23:50.539 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 --2- 172.21.15.171:0/1886279968 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fd680 0x7fcf340ffb10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:50.540 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 --2- 172.21.15.171:0/1886279968 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf34061670 0x7fcf340fa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:50.540 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 --2- 172.21.15.171:0/1886279968 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf340fac50 0x7fcf340fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:50.540 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 -- 172.21.15.171:0/1886279968 >> 172.21.15.171:0/1886279968 conn(0x7fcf340f2190 msgr2=0x7fcf340f45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:50.540 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 -- 172.21.15.171:0/1886279968 shutdown_connections 2024-02-23T00:23:50.540 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 -- 172.21.15.171:0/1886279968 wait complete. 2024-02-23T00:23:50.541 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 librados: starting msgr at 2024-02-23T00:23:50.541 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 librados: starting objecter 2024-02-23T00:23:50.541 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.531+0000 7fcf3b549700 1 Processor -- start 2024-02-23T00:23:50.541 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf3b549700 1 -- start start 2024-02-23T00:23:50.541 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf3b549700 1 librados: setting wanted keys 2024-02-23T00:23:50.541 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf3b549700 1 librados: calling monclient init 2024-02-23T00:23:50.542 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf3b549700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf34061670 0x7fcf34107f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:50.542 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf3b549700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fac50 0x7fcf34108440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:50.542 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf3b549700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf340fd680 0x7fcf3410f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:50.542 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf3b549700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410fa10 con 0x7fcf340fd680 2024-02-23T00:23:50.542 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf3b549700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410fb50 con 0x7fcf340fac50 2024-02-23T00:23:50.542 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf3b549700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410fc90 con 0x7fcf34061670 2024-02-23T00:23:50.543 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf38ae4700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fac50 0x7fcf34108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:50.543 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf392e5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf34061670 0x7fcf34107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:50.543 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf38ae4700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fac50 0x7fcf34108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:44300/0 (socket says 172.21.15.171:44300) 2024-02-23T00:23:50.543 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf38ae4700 1 -- 172.21.15.171:0/2879070202 learned_addr learned my addr 172.21.15.171:0/2879070202 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:23:50.543 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf39ae6700 1 --2- 172.21.15.171:0/2879070202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf340fd680 0x7fcf3410f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:50.543 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf38ae4700 1 -- 172.21.15.171:0/2879070202 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf34061670 msgr2=0x7fcf34107f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:50.544 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf38ae4700 1 --2- 172.21.15.171:0/2879070202 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf34061670 0x7fcf34107f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:50.544 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf38ae4700 1 -- 172.21.15.171:0/2879070202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf340fd680 msgr2=0x7fcf3410f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:50.544 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf38ae4700 1 --2- 172.21.15.171:0/2879070202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf340fd680 0x7fcf3410f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:50.544 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf38ae4700 1 -- 172.21.15.171:0/2879070202 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c0097c0 con 0x7fcf340fac50 2024-02-23T00:23:50.544 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf392e5700 1 --2- 172.21.15.171:0/2879070202 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf34061670 0x7fcf34107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:23:50.544 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf38ae4700 1 --2- 172.21.15.171:0/2879070202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fac50 0x7fcf34108440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fcf2400e940 tx=0x7fcf2400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:50.545 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf2a7fc700 10 client.?.objecter ms_handle_connect 0x7fcf340fac50 2024-02-23T00:23:50.545 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf2a7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:50.545 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf2a7fc700 1 -- 172.21.15.171:0/2879070202 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24004660 con 0x7fcf340fac50 2024-02-23T00:23:50.545 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.532+0000 7fcf2a7fc700 1 -- 172.21.15.171:0/2879070202 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf2400f040 con 0x7fcf340fac50 2024-02-23T00:23:50.545 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.533+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3410ff10 con 0x7fcf340fac50 2024-02-23T00:23:50.545 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.533+0000 7fcf2a7fc700 1 -- 172.21.15.171:0/2879070202 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf240147d0 con 0x7fcf340fac50 2024-02-23T00:23:50.545 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.533+0000 7fcf3b549700 10 client.24109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:50.546 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.533+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf34110440 con 0x7fcf340fac50 2024-02-23T00:23:50.546 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.533+0000 7fcf3b549700 1 librados: init done 2024-02-23T00:23:50.546 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.533+0000 7fcf2a7fc700 1 -- 172.21.15.171:0/2879070202 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7fcf24014c00 con 0x7fcf340fac50 2024-02-23T00:23:50.546 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf2a7fc700 1 --2- 172.21.15.171:0/2879070202 >> v2:172.21.15.87:6800/1121018082 conn(0x7fcf20034230 0x7fcf200366f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:50.546 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3405f360 con 0x7fcf340fac50 2024-02-23T00:23:50.546 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf2a7fc700 1 -- 172.21.15.171:0/2879070202 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fcf24042950 con 0x7fcf340fac50 2024-02-23T00:23:50.546 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf2a7fc700 10 client.24109.objecter ms_dispatch 0x7fcf34001550 osd_map(4..4 src has 1..4) v4 2024-02-23T00:23:50.546 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf2a7fc700 3 client.24109.objecter handle_osd_map got epochs [4,4] > 0 2024-02-23T00:23:50.547 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf2a7fc700 3 client.24109.objecter handle_osd_map decoding full epoch 4 2024-02-23T00:23:50.547 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf2a7fc700 20 client.24109.objecter dump_active .. 0 homeless 2024-02-23T00:23:50.547 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf392e5700 1 --2- 172.21.15.171:0/2879070202 >> v2:172.21.15.87:6800/1121018082 conn(0x7fcf20034230 0x7fcf200366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:50.547 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf2a7fc700 10 client.24109.objecter ms_handle_connect 0x7fcf20034230 2024-02-23T00:23:50.547 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.534+0000 7fcf392e5700 1 --2- 172.21.15.171:0/2879070202 >> v2:172.21.15.87:6800/1121018082 conn(0x7fcf20034230 0x7fcf200366f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fcf30011510 tx=0x7fcf3000b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:50.548 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.539+0000 7fcf2a7fc700 1 -- 172.21.15.171:0/2879070202 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fcf240112c0 con 0x7fcf340fac50 2024-02-23T00:23:50.724 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.722+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi087=y;smithi171=x", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf340f3cf0 con 0x7fcf20034230 2024-02-23T00:23:50.730 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.728+0000 7fcf2a7fc700 1 -- 172.21.15.171:0/2879070202 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fcf340f3cf0 con 0x7fcf20034230 2024-02-23T00:23:50.731 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled mgr update... 2024-02-23T00:23:50.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 10 librados: watch_flush enter 2024-02-23T00:23:50.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 10 librados: watch_flush exit 2024-02-23T00:23:50.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 20 client.24109.objecter shutdown clearing up homeless session... 2024-02-23T00:23:50.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 10 client.24109.objecter successfully canceled tick 2024-02-23T00:23:50.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 >> v2:172.21.15.87:6800/1121018082 conn(0x7fcf20034230 msgr2=0x7fcf200366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:50.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 --2- 172.21.15.171:0/2879070202 >> v2:172.21.15.87:6800/1121018082 conn(0x7fcf20034230 0x7fcf200366f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fcf30011510 tx=0x7fcf3000b040).stop 2024-02-23T00:23:50.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fac50 msgr2=0x7fcf34108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:50.733 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 --2- 172.21.15.171:0/2879070202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fac50 0x7fcf34108440 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fcf2400e940 tx=0x7fcf2400b040).stop 2024-02-23T00:23:50.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 shutdown_connections 2024-02-23T00:23:50.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 --2- 172.21.15.171:0/2879070202 >> v2:172.21.15.87:6800/1121018082 conn(0x7fcf20034230 0x7fcf200366f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:50.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 --2- 172.21.15.171:0/2879070202 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf340fd680 0x7fcf3410f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:50.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 --2- 172.21.15.171:0/2879070202 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf34061670 0x7fcf34107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:50.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 --2- 172.21.15.171:0/2879070202 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf340fac50 0x7fcf34108440 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:50.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 >> 172.21.15.171:0/2879070202 conn(0x7fcf340f2190 msgr2=0x7fcf340f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:50.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.730+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 shutdown_connections 2024-02-23T00:23:50.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.731+0000 7fcf3b549700 1 -- 172.21.15.171:0/2879070202 wait complete. 2024-02-23T00:23:50.734 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:23:50.731+0000 7fcf3b549700 1 librados: shutdown 2024-02-23T00:23:50.879 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:48.969815+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-23T00:23:50.879 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:48.971592+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.c on smithi087 2024-02-23T00:23:50.879 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:50.380990+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:50.879 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:50.381644+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:50.879 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:50.382151+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-23T00:23:50.880 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:50.382631+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:51.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:48.969815+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-23T00:23:51.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:48.971592+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.c on smithi087 2024-02-23T00:23:51.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:50.380990+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:51.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:50.381644+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:51.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:50.382151+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-23T00:23:51.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:50.382631+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:51.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:48.969815+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-23T00:23:51.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:48.971592+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.c on smithi087 2024-02-23T00:23:51.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:50.380990+0000 mon.a (mon.0) 178 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:51.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:50.381644+0000 mon.a (mon.0) 179 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-23T00:23:51.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:50.382151+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-23T00:23:51.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:50.382631+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:51.557 DEBUG:teuthology.orchestra.run.smithi171:mgr.x> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mgr.x.service 2024-02-23T00:23:51.560 INFO:tasks.cephadm:Deploying OSDs... 2024-02-23T00:23:51.560 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:23:51.560 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2024-02-23T00:23:51.583 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-23T00:23:51.583 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2024-02-23T00:23:51.589 INFO:journalctl@ceph.mgr.x.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:23:51.645 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-23T00:23:51.646 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-23T00:23:51.646 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 164767 Links: 1 2024-02-23T00:23:51.646 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-23T00:23:51.646 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-02-23T00:23:51.646 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-02-23 00:23:37.142906024 +0000 2024-02-23T00:23:51.646 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-02-23 00:12:32.092997153 +0000 2024-02-23T00:23:51.646 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-02-23 00:12:32.092997153 +0000 2024-02-23T00:23:51.646 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2024-02-23T00:23:51.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-23T00:23:51.719 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-02-23T00:23:51.719 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-02-23T00:23:51.719 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000277248 s, 1.8 MB/s 2024-02-23T00:23:51.720 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-23T00:23:51.782 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2024-02-23T00:23:51.842 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-23T00:23:51.843 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-23T00:23:51.843 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 165219 Links: 1 2024-02-23T00:23:51.843 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-23T00:23:51.843 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-02-23T00:23:51.843 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-02-23 00:23:37.142906024 +0000 2024-02-23T00:23:51.843 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-02-23 00:12:32.450989111 +0000 2024-02-23T00:23:51.843 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-02-23 00:12:32.450989111 +0000 2024-02-23T00:23:51.844 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2024-02-23T00:23:51.844 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-23T00:23:51.896 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:50.064754+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:51.896 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:50.381438+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-23T00:23:51.896 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:50.383056+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Reconfiguring daemon mon.b on smithi171 2024-02-23T00:23:51.896 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:50.725022+0000 mgr.y (mgr.14146) 60 : audit [DBG] from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi087=y;smithi171=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:51.896 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:50.726302+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Saving service mgr spec with placement smithi087=y;smithi171=x;count:2 2024-02-23T00:23:51.896 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:50.730057+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:51.910 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-02-23T00:23:51.910 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-02-23T00:23:51.910 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000225122 s, 2.3 MB/s 2024-02-23T00:23:51.912 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-23T00:23:51.971 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2024-02-23T00:23:52.030 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-23T00:23:52.030 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-23T00:23:52.030 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 166218 Links: 1 2024-02-23T00:23:52.031 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-23T00:23:52.031 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-02-23T00:23:52.031 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-02-23 00:23:37.142906024 +0000 2024-02-23T00:23:52.031 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-02-23 00:12:32.811981002 +0000 2024-02-23T00:23:52.031 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-02-23 00:12:32.811981002 +0000 2024-02-23T00:23:52.031 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2024-02-23T00:23:52.032 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-23T00:23:52.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:50.064754+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:52.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:50.381438+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-23T00:23:52.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:50.383056+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Reconfiguring daemon mon.b on smithi171 2024-02-23T00:23:52.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:50.725022+0000 mgr.y (mgr.14146) 60 : audit [DBG] from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi087=y;smithi171=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:52.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:50.726302+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Saving service mgr spec with placement smithi087=y;smithi171=x;count:2 2024-02-23T00:23:52.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:50.730057+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:50.064754+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:50.381438+0000 mgr.y (mgr.14146) 58 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-23T00:23:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:50.383056+0000 mgr.y (mgr.14146) 59 : cephadm [INF] Reconfiguring daemon mon.b on smithi171 2024-02-23T00:23:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:50.725022+0000 mgr.y (mgr.14146) 60 : audit [DBG] from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi087=y;smithi171=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:50.726302+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Saving service mgr spec with placement smithi087=y;smithi171=x;count:2 2024-02-23T00:23:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:50.730057+0000 mon.a (mon.0) 182 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:52.097 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-02-23T00:23:52.097 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-02-23T00:23:52.097 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000187838 s, 2.7 MB/s 2024-02-23T00:23:52.099 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-23T00:23:52.157 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2024-02-23T00:23:52.216 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-23T00:23:52.216 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-23T00:23:52.216 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 171103 Links: 1 2024-02-23T00:23:52.216 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-23T00:23:52.216 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2024-02-23T00:23:52.216 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2024-02-23 00:23:37.142906024 +0000 2024-02-23T00:23:52.217 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2024-02-23 00:12:33.190972488 +0000 2024-02-23T00:23:52.217 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2024-02-23 00:12:33.190972488 +0000 2024-02-23T00:23:52.217 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2024-02-23T00:23:52.217 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-23T00:23:52.282 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2024-02-23T00:23:52.282 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2024-02-23T00:23:52.282 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000209587 s, 2.4 MB/s 2024-02-23T00:23:52.283 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-23T00:23:52.343 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:23:52.343 DEBUG:teuthology.orchestra.run.smithi171:> dd if=/scratch_devs of=/dev/stdout 2024-02-23T00:23:52.362 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-23T00:23:52.362 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_1 2024-02-23T00:23:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-23T00:23:52.421 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-23T00:23:52.422 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 169096 Links: 1 2024-02-23T00:23:52.422 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-23T00:23:52.422 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-02-23T00:23:52.422 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-02-23 00:23:26.138222244 +0000 2024-02-23T00:23:52.422 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-02-23 00:12:32.100343646 +0000 2024-02-23T00:23:52.422 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-02-23 00:12:32.100343646 +0000 2024-02-23T00:23:52.423 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-02-23T00:23:52.423 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-23T00:23:52.491 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-02-23T00:23:52.491 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-02-23T00:23:52.492 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000234119 s, 2.2 MB/s 2024-02-23T00:23:52.493 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-23T00:23:52.552 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_2 2024-02-23T00:23:52.609 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-23T00:23:52.609 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-23T00:23:52.609 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 167510 Links: 1 2024-02-23T00:23:52.610 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-23T00:23:52.610 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-02-23T00:23:52.610 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-02-23 00:23:26.138222244 +0000 2024-02-23T00:23:52.610 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-02-23 00:12:32.446337037 +0000 2024-02-23T00:23:52.610 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-02-23 00:12:32.446337037 +0000 2024-02-23T00:23:52.610 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-02-23T00:23:52.610 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-23T00:23:52.675 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-02-23T00:23:52.676 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-02-23T00:23:52.676 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000210382 s, 2.4 MB/s 2024-02-23T00:23:52.677 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-23T00:23:52.735 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_3 2024-02-23T00:23:52.794 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-23T00:23:52.794 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-23T00:23:52.794 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 167613 Links: 1 2024-02-23T00:23:52.794 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-23T00:23:52.794 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-02-23T00:23:52.794 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-02-23 00:23:26.138222244 +0000 2024-02-23T00:23:52.794 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-02-23 00:12:32.798330314 +0000 2024-02-23T00:23:52.794 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-02-23 00:12:32.798330314 +0000 2024-02-23T00:23:52.794 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-02-23T00:23:52.795 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-23T00:23:52.860 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-02-23T00:23:52.860 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-02-23T00:23:52.860 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000184341 s, 2.8 MB/s 2024-02-23T00:23:52.861 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-23T00:23:52.922 DEBUG:teuthology.orchestra.run.smithi171:> stat /dev/vg_nvme/lv_4 2024-02-23T00:23:52.980 INFO:teuthology.orchestra.run.smithi171.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-23T00:23:52.980 INFO:teuthology.orchestra.run.smithi171.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-23T00:23:52.980 INFO:teuthology.orchestra.run.smithi171.stdout:Device: 6h/6d Inode: 170000 Links: 1 2024-02-23T00:23:52.980 INFO:teuthology.orchestra.run.smithi171.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-23T00:23:52.980 INFO:teuthology.orchestra.run.smithi171.stdout:Context: system_u:object_r:device_t:s0 2024-02-23T00:23:52.980 INFO:teuthology.orchestra.run.smithi171.stdout:Access: 2024-02-23 00:23:26.138222244 +0000 2024-02-23T00:23:52.981 INFO:teuthology.orchestra.run.smithi171.stdout:Modify: 2024-02-23 00:12:33.148323628 +0000 2024-02-23T00:23:52.981 INFO:teuthology.orchestra.run.smithi171.stdout:Change: 2024-02-23 00:12:33.148323628 +0000 2024-02-23T00:23:52.981 INFO:teuthology.orchestra.run.smithi171.stdout: Birth: - 2024-02-23T00:23:52.981 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-23T00:23:53.046 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records in 2024-02-23T00:23:53.047 INFO:teuthology.orchestra.run.smithi171.stderr:1+0 records out 2024-02-23T00:23:53.047 INFO:teuthology.orchestra.run.smithi171.stderr:512 bytes copied, 0.000223962 s, 2.3 MB/s 2024-02-23T00:23:53.048 DEBUG:teuthology.orchestra.run.smithi171:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-23T00:23:53.108 INFO:tasks.cephadm:Deploying osd.0 on smithi087 with /dev/vg_nvme/lv_4... 2024-02-23T00:23:53.108 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-23T00:23:53.576 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:52.065126+0000 mgr.y (mgr.14146) 62 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:53.576 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:52.202168+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:53.576 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:52.202996+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:53.576 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:52.204344+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:52.204960+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:52.208713+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:52.209917+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:52.212307+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:52.213054+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:52.213782+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:52.214318+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Deploying daemon mgr.x on smithi171 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:52.065126+0000 mgr.y (mgr.14146) 62 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:52.202168+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:52.202996+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:53.577 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:52.204344+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:53.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:52.204960+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:53.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:52.208713+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:53.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:52.209917+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-23T00:23:53.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:52.212307+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-23T00:23:53.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:52.213054+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-23T00:23:53.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:52.213782+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:53.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:52.214318+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Deploying daemon mgr.x on smithi171 2024-02-23T00:23:53.633 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:52.065126+0000 mgr.y (mgr.14146) 62 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:52.202168+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:52.202996+0000 mon.a (mon.0) 184 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:52.204344+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:52.204960+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:52.208713+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:52.209917+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:52.212307+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:52.213054+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:52.213782+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:53.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:52.214318+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Deploying daemon mgr.x on smithi171 2024-02-23T00:23:53.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:53 smithi171 systemd[1]: Starting Ceph mgr.x for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:23:54.692 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:54.407+0000 7f0b5c278700 1 -- 172.21.15.171:0/1372563415 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cab409c4e0 con 0x55cab4071400 2024-02-23T00:23:54.692 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:54 smithi171 bash[40898]: bd0c9c71f98ba1ee5e51e0fab4c5674455f114d60e7ab42e50edc5876cab411c 2024-02-23T00:23:54.692 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:54 smithi171 systemd[1]: Started Ceph mgr.x for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:23:54.692 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:54.532+0000 7f0b6c873680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-23T00:23:54.693 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:54.618+0000 7f0b6c873680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-23T00:23:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-23T00:23:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-23T00:23:55.338 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10+0 records in 2024-02-23T00:23:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2024-02-23T00:23:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112887 s, 929 MB/s 2024-02-23T00:23:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping successful for: 2024-02-23T00:23:55.442 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:55.017+0000 7f0b6c873680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-23T00:23:55.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:54.065436+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:55.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:54.552074+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:55.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:54.554383+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:55.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:55.538220+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:55.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:55.538773+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:55.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:55.539325+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:55.544947+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:55.546500+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:55.547132+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:55.547607+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:54.065436+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:54.552074+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:54.554383+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:55.538220+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:55.538773+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:55.539325+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:55.544947+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:55.691 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:55.546500+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-23T00:23:55.692 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:55.547132+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-23T00:23:55.692 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:55.547607+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:55.883 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:55.798+0000 7f0b6c873680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-23T00:23:55.883 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:54.065436+0000 mgr.y (mgr.14146) 64 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:55.884 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:54.552074+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:55.884 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:54.554383+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:55.884 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:55.538220+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:55.884 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:55.538773+0000 mon.a (mon.0) 195 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:55.884 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:55.539325+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:55.884 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:55.544947+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:55.884 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:55.546500+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-23T00:23:55.884 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:55.547132+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-23T00:23:55.884 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:55.547607+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:55.988 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch daemon add osd smithi087:vg_nvme/lv_4 2024-02-23T00:23:56.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:55.881+0000 7f0b6c873680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-23T00:23:56.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:56.093+0000 7f0b6c873680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-23T00:23:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:55.546252+0000 mgr.y (mgr.14146) 65 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-23T00:23:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:23:55.548045+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Reconfiguring daemon mgr.y on smithi087 2024-02-23T00:23:56.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:55.546252+0000 mgr.y (mgr.14146) 65 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-23T00:23:56.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:23:55.548045+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Reconfiguring daemon mgr.y on smithi087 2024-02-23T00:23:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:55.546252+0000 mgr.y (mgr.14146) 65 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-23T00:23:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:23:55.548045+0000 mgr.y (mgr.14146) 66 : cephadm [INF] Reconfiguring daemon mgr.y on smithi087 2024-02-23T00:23:57.326 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:57.031+0000 7f0b6c873680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-23T00:23:57.326 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:57.157+0000 7f0b6c873680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:23:57.326 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:57.236+0000 7f0b6c873680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-23T00:23:57.326 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:57.325+0000 7f0b6c873680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-23T00:23:57.682 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:56.065882+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:57.682 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:57.185116+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:57.682 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:56.065882+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:57.682 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:57.185116+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:57.692 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:57.520+0000 7f0b6c873680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-23T00:23:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:56.065882+0000 mgr.y (mgr.14146) 67 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:57.185116+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:58.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:57.799+0000 7f0b6c873680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-23T00:23:58.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:57.897+0000 7f0b6c873680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-23T00:23:58.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.275+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/3940037525 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd9d8061b80 msgr2=0x7fd9d80fcb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.275+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/3940037525 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd9d8061b80 0x7fd9d80fcb60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd9cc004870 tx=0x7fd9cc009cb0).stop 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.275+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/3940037525 shutdown_connections 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.275+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/3940037525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9d80ff930 0x7fd9d8101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.275+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/3940037525 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd9d8061b80 0x7fd9d80fcb60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.275+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/3940037525 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80fd050 0x7fd9d80ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.275+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/3940037525 >> 172.21.15.87:0/3940037525 conn(0x7fd9d80f2180 msgr2=0x7fd9d80f45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.275+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/3940037525 shutdown_connections 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.275+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/3940037525 wait complete. 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 librados: starting msgr at 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 librados: starting objecter 2024-02-23T00:23:58.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 Processor -- start 2024-02-23T00:23:58.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 -- start start 2024-02-23T00:23:58.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 librados: setting wanted keys 2024-02-23T00:23:58.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 librados: calling monclient init 2024-02-23T00:23:58.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd9d8061b80 0x7fd9d810a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:58.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9d80fd050 0x7fd9d810a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:58.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80ff930 0x7fd9d8111860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:58.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d8111d50 con 0x7fd9d80ff930 2024-02-23T00:23:58.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d8111e90 con 0x7fd9d80fd050 2024-02-23T00:23:58.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9dc8b2700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d8111fd0 con 0x7fd9d8061b80 2024-02-23T00:23:58.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9d659c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd9d8061b80 0x7fd9d810a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:58.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9d659c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd9d8061b80 0x7fd9d810a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:48252/0 (socket says 172.21.15.87:48252) 2024-02-23T00:23:58.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9d6d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80ff930 0x7fd9d8111860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9d659c700 1 -- 172.21.15.87:0/2365074797 learned_addr learned my addr 172.21.15.87:0/2365074797 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.276+0000 7fd9d5d9b700 1 --2- 172.21.15.87:0/2365074797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9d80fd050 0x7fd9d810a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9d6d9d700 1 -- 172.21.15.87:0/2365074797 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd9d8061b80 msgr2=0x7fd9d810a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9d6d9d700 1 --2- 172.21.15.87:0/2365074797 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd9d8061b80 0x7fd9d810a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9d6d9d700 1 -- 172.21.15.87:0/2365074797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9d80fd050 msgr2=0x7fd9d810a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9d6d9d700 1 --2- 172.21.15.87:0/2365074797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9d80fd050 0x7fd9d810a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9d6d9d700 1 -- 172.21.15.87:0/2365074797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c8005040 con 0x7fd9d80ff930 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9d6d9d700 1 --2- 172.21.15.87:0/2365074797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80ff930 0x7fd9d8111860 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd9d000eb40 tx=0x7fd9d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9c77fe700 10 client.?.objecter ms_handle_connect 0x7fd9d80ff930 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9c77fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:23:58.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9c77fe700 1 -- 172.21.15.87:0/2365074797 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0009870 con 0x7fd9d80ff930 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9cc005040 con 0x7fd9d80ff930 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9c77fe700 1 -- 172.21.15.87:0/2365074797 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9d000f040 con 0x7fd9d80ff930 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9c77fe700 1 -- 172.21.15.87:0/2365074797 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0007d60 con 0x7fd9d80ff930 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9dc8b2700 10 client.14208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d8112680 con 0x7fd9d80ff930 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.277+0000 7fd9dc8b2700 1 librados: init done 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9c77fe700 1 -- 172.21.15.87:0/2365074797 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 13) v1 ==== 38638+0+0 (secure 0 0 0) 0x7fd9d000c7c0 con 0x7fd9d80ff930 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d8104b00 con 0x7fd9d80ff930 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9c77fe700 1 --2- 172.21.15.87:0/2365074797 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd9c0034230 0x7fd9c00366f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9c77fe700 1 -- 172.21.15.87:0/2365074797 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (secure 0 0 0) 0x7fd9d0042b00 con 0x7fd9d80ff930 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9c77fe700 10 client.14208.objecter ms_dispatch 0x7fd9d8001550 osd_map(4..4 src has 1..4) v4 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9c77fe700 3 client.14208.objecter handle_osd_map got epochs [4,4] > 0 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9c77fe700 3 client.14208.objecter handle_osd_map decoding full epoch 4 2024-02-23T00:23:58.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9c77fe700 20 client.14208.objecter dump_active .. 0 homeless 2024-02-23T00:23:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9d659c700 1 --2- 172.21.15.87:0/2365074797 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd9c0034230 0x7fd9c00366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:23:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9c77fe700 10 client.14208.objecter ms_handle_connect 0x7fd9c0034230 2024-02-23T00:23:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.278+0000 7fd9d659c700 1 --2- 172.21.15.87:0/2365074797 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd9c0034230 0x7fd9c00366f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd9cc009f20 tx=0x7fd9cc00b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:23:58.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.283+0000 7fd9c77fe700 1 -- 172.21.15.87:0/2365074797 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd9d0044e60 con 0x7fd9d80ff930 2024-02-23T00:23:58.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:23:58.457+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9d80f3c10 con 0x7fd9c0034230 2024-02-23T00:23:58.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:58.462020+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:23:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:58.463861+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:23:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:58.464336+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:58.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:58.462020+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:23:58.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:58.463861+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:23:58.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:58.464336+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:58.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:58.621+0000 7f0b6c873680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-23T00:23:58.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:58.702+0000 7f0b6c873680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-23T00:23:58.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:58.794+0000 7f0b6c873680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-23T00:23:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:58.462020+0000 mon.a (mon.0) 202 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:23:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:58.463861+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:23:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:58.464336+0000 mon.a (mon.0) 204 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:59.226 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:58.965+0000 7f0b6c873680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-23T00:23:59.488 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:59.224+0000 7f0b6c873680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-23T00:23:59.488 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:59.354+0000 7f0b6c873680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-23T00:23:59.743 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:59.486+0000 7f0b6c873680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:23:59.743 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:59.658+0000 7f0b6c873680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-23T00:23:59.743 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:23:59.742+0000 7f0b6c873680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-23T00:23:59.743 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:58.066469+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:59.744 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:58.459849+0000 mgr.y (mgr.14146) 69 : audit [DBG] from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:59.744 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:58.598728+0000 mon.a (mon.0) 205 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:59.744 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:58.599647+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:59.744 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:58.600859+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:59.744 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:58.601442+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:59.744 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:23:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:58.608238+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:59.886 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:58.066469+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:59.886 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:58.459849+0000 mgr.y (mgr.14146) 69 : audit [DBG] from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:59.886 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:58.598728+0000 mon.a (mon.0) 205 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:59.886 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:58.599647+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:59.886 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:58.600859+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:59.886 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:58.601442+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:59.886 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:58.608238+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:59.887 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:58.066469+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:23:59.887 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:58.459849+0000 mgr.y (mgr.14146) 69 : audit [DBG] from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:23:59.887 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:58.598728+0000 mon.a (mon.0) 205 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:23:59.887 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:58.599647+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:23:59.887 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:58.600859+0000 mon.a (mon.0) 207 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:23:59.887 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:58.601442+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:23:59.887 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:23:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:58.608238+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:00.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:00.606+0000 7fd9c77fe700 1 -- 172.21.15.87:0/2365074797 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7fd9d00147e0 con 0x7fd9d80ff930 2024-02-23T00:24:00.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:23:59.756405+0000 mon.a (mon.0) 210 : cluster [DBG] Standby manager daemon x started 2024-02-23T00:24:00.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:59.757387+0000 mon.b (mon.1) 3 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-23T00:24:00.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:59.757719+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-23T00:24:00.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:59.758642+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-23T00:24:00.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:23:59.758915+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:23:59.756405+0000 mon.a (mon.0) 210 : cluster [DBG] Standby manager daemon x started 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:59.757387+0000 mon.b (mon.1) 3 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:59.757719+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:59.758642+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:23:59.758915+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:23:59.756405+0000 mon.a (mon.0) 210 : cluster [DBG] Standby manager daemon x started 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:59.757387+0000 mon.b (mon.1) 3 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:59.757719+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:59.758642+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-23T00:24:01.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:23:59.758915+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.171:0/4059942763' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-23T00:24:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:00.066739+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:00.609420+0000 mon.a (mon.0) 211 : cluster [DBG] mgrmap e14: y(active, since 98s), standbys: x 2024-02-23T00:24:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:00.609540+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-23T00:24:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:00.945344+0000 mon.c (mon.2) 3 : audit [INF] from='client.? 172.21.15.87:0/3205588355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7"}]: dispatch 2024-02-23T00:24:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:00.945660+0000 mon.a (mon.0) 213 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7"}]: dispatch 2024-02-23T00:24:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:00.954748+0000 mon.a (mon.0) 214 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7"}]': finished 2024-02-23T00:24:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:00.954804+0000 mon.a (mon.0) 215 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-23T00:24:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:00.954925+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:01.556911+0000 mon.c (mon.2) 4 : audit [DBG] from='client.? 172.21.15.87:0/3016833125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:24:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:00.066739+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:00.609420+0000 mon.a (mon.0) 211 : cluster [DBG] mgrmap e14: y(active, since 98s), standbys: x 2024-02-23T00:24:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:00.609540+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-23T00:24:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:00.945344+0000 mon.c (mon.2) 3 : audit [INF] from='client.? 172.21.15.87:0/3205588355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7"}]: dispatch 2024-02-23T00:24:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:00.945660+0000 mon.a (mon.0) 213 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7"}]: dispatch 2024-02-23T00:24:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:00.954748+0000 mon.a (mon.0) 214 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7"}]': finished 2024-02-23T00:24:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:00.954804+0000 mon.a (mon.0) 215 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-23T00:24:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:00.954925+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:01.556911+0000 mon.c (mon.2) 4 : audit [DBG] from='client.? 172.21.15.87:0/3016833125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:24:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:00.066739+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:00.609420+0000 mon.a (mon.0) 211 : cluster [DBG] mgrmap e14: y(active, since 98s), standbys: x 2024-02-23T00:24:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:00.609540+0000 mon.a (mon.0) 212 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-23T00:24:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:00.945344+0000 mon.c (mon.2) 3 : audit [INF] from='client.? 172.21.15.87:0/3205588355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7"}]: dispatch 2024-02-23T00:24:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:00.945660+0000 mon.a (mon.0) 213 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7"}]: dispatch 2024-02-23T00:24:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:00.954748+0000 mon.a (mon.0) 214 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7"}]': finished 2024-02-23T00:24:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:00.954804+0000 mon.a (mon.0) 215 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-23T00:24:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:00.954925+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:01.556911+0000 mon.c (mon.2) 4 : audit [DBG] from='client.? 172.21.15.87:0/3016833125' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:24:03.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:03.278+0000 7fd9d7fff700 10 client.14208.objecter tick 2024-02-23T00:24:03.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:02.067236+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:04.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:02.067236+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:04.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:02.067236+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:05.769 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:04.067501+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:05.769 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:04.067501+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:04.067501+0000 mgr.y (mgr.14146) 72 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:07.916 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:06.067876+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:07.916 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:07.198336+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-23T00:24:07.916 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:07.199018+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:07.916 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:06.067876+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:07.917 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:07.198336+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-23T00:24:07.917 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:07.199018+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:07.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:06.067876+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:07.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:07.198336+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-23T00:24:07.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:07.199018+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:08.279+0000 7fd9d7fff700 10 client.14208.objecter tick 2024-02-23T00:24:08.882 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:24:07.199580+0000 mgr.y (mgr.14146) 74 : cephadm [INF] Deploying daemon osd.0 on smithi087 2024-02-23T00:24:08.882 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:24:07.199580+0000 mgr.y (mgr.14146) 74 : cephadm [INF] Deploying daemon osd.0 on smithi087 2024-02-23T00:24:08.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:24:07.199580+0000 mgr.y (mgr.14146) 74 : cephadm [INF] Deploying daemon osd.0 on smithi087 2024-02-23T00:24:09.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:08.068234+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:09.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:08.068234+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:09.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:08.068234+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:11.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:10.071520+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:11.983 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:10.071520+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:11.983 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:10.071520+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:12.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:12.251248+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:12.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:12.256796+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:13.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:12.251248+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:13.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:12.256796+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:12.251248+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:12.256796+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:13.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:13.280+0000 7fd9d7fff700 10 client.14208.objecter tick 2024-02-23T00:24:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:12.072085+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:14.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:12.072085+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:14.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:12.072085+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:14.072496+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:15.314547+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:15.315197+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:15.315795+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:15.319681+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:15.349765+0000 mon.a (mon.0) 225 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-23T00:24:16.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:14.072496+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:16.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:15.314547+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:16.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:15.315197+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:16.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:15.315795+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:16.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:15.319681+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:16.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:15.349765+0000 mon.a (mon.0) 225 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-23T00:24:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:14.072496+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:15.314547+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:15.315197+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:15.315795+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:15.319681+0000 mon.a (mon.0) 224 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:15.349765+0000 mon.a (mon.0) 225 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-23T00:24:16.153 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 0 on host 'smithi087' 2024-02-23T00:24:16.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.152+0000 7fd9c77fe700 1 -- 172.21.15.87:0/2365074797 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd9d80f3c10 con 0x7fd9c0034230 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.153+0000 7fd9dc8b2700 10 librados: watch_flush enter 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 10 librados: watch_flush exit 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 20 client.14208.objecter shutdown clearing up homeless session... 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 10 client.14208.objecter successfully canceled tick 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd9c0034230 msgr2=0x7fd9c00366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/2365074797 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd9c0034230 0x7fd9c00366f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd9cc009f20 tx=0x7fd9cc00b040).stop 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80ff930 msgr2=0x7fd9d8111860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/2365074797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80ff930 0x7fd9d8111860 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd9d000eb40 tx=0x7fd9d000b040).stop 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 shutdown_connections 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/2365074797 >> v2:172.21.15.87:6800/1121018082 conn(0x7fd9c0034230 0x7fd9c00366f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/2365074797 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd9d80ff930 0x7fd9d8111860 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:16.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/2365074797 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd9d8061b80 0x7fd9d810a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:16.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 --2- 172.21.15.87:0/2365074797 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd9d80fd050 0x7fd9d810a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:16.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 >> 172.21.15.87:0/2365074797 conn(0x7fd9d80f2180 msgr2=0x7fd9d80fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:16.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 shutdown_connections 2024-02-23T00:24:16.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 -- 172.21.15.87:0/2365074797 wait complete. 2024-02-23T00:24:16.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:16.154+0000 7fd9dc8b2700 1 librados: shutdown 2024-02-23T00:24:16.814 DEBUG:teuthology.orchestra.run.smithi087:osd.0> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.0.service 2024-02-23T00:24:16.817 INFO:tasks.cephadm:Deploying osd.1 on smithi087 with /dev/vg_nvme/lv_3... 2024-02-23T00:24:16.817 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-23T00:24:16.847 INFO:journalctl@ceph.osd.0.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:24:17.411 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[48310]: debug 2024-02-23T00:24:17.325+0000 7fc9c95eb700 -1 osd.0 0 waiting for initial osdmap 2024-02-23T00:24:17.411 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[48310]: debug 2024-02-23T00:24:17.327+0000 7fc9c3d63700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-23T00:24:17.411 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:16.072833+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:16.151760+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:16.186444+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:16.322336+0000 mon.a (mon.0) 228 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:16.322419+0000 mon.a (mon.0) 229 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:16.322544+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:16.322910+0000 mon.a (mon.0) 231 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:16.072833+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:16.151760+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:16.186444+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:17.412 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:16.322336+0000 mon.a (mon.0) 228 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-23T00:24:17.413 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:16.322419+0000 mon.a (mon.0) 229 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-23T00:24:17.413 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:16.322544+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:17.413 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:16.322910+0000 mon.a (mon.0) 231 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:24:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:16.072833+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:16.151760+0000 mon.a (mon.0) 226 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:16.186444+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:16.322336+0000 mon.a (mon.0) 228 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-23T00:24:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:16.322419+0000 mon.a (mon.0) 229 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-23T00:24:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:16.322544+0000 mon.a (mon.0) 230 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:16.322910+0000 mon.a (mon.0) 231 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:24:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:17.323760+0000 mon.a (mon.0) 232 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:24:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:17.323820+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-23T00:24:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:17.323944+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:17.326719+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:17.323760+0000 mon.a (mon.0) 232 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:24:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:17.323820+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-23T00:24:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:17.323944+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:18.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:17.326719+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:17.323760+0000 mon.a (mon.0) 232 : audit [INF] from='osd.0 v2:172.21.15.87:6801/2153191863' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:24:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:17.323820+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-23T00:24:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:17.323944+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:17.326719+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:19.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:16.311527+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:24:19.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:16.311574+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:24:19.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:18.073124+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:19.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:18.328853+0000 mon.a (mon.0) 236 : cluster [INF] osd.0 v2:172.21.15.87:6801/2153191863 boot 2024-02-23T00:24:19.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:18.328889+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-23T00:24:19.338 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:18.329336+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:19.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:16.311527+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:24:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:16.311574+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:24:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:18.073124+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:18.328853+0000 mon.a (mon.0) 236 : cluster [INF] osd.0 v2:172.21.15.87:6801/2153191863 boot 2024-02-23T00:24:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:18.328889+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-23T00:24:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:18.329336+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:16.311527+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:24:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:16.311574+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:24:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:18.073124+0000 mgr.y (mgr.14146) 80 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-23T00:24:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:18.328853+0000 mon.a (mon.0) 236 : cluster [INF] osd.0 v2:172.21.15.87:6801/2153191863 boot 2024-02-23T00:24:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:18.328889+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-23T00:24:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:18.329336+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:24:20.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:19.329496+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-23T00:24:20.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:19.329496+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-23T00:24:20.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:19.329496+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-23T00:24:21.019 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-23T00:24:21.019 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-23T00:24:21.020 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10+0 records in 2024-02-23T00:24:21.020 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2024-02-23T00:24:21.020 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011353 s, 924 MB/s 2024-02-23T00:24:21.020 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping successful for: 2024-02-23T00:24:21.733 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:20.073583+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:21.733 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:24:20.614481+0000 mgr.y (mgr.14146) 82 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:24:21.733 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:20.621684+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:21.734 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:20.622503+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:24:21.734 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:24:20.622939+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 16098M 2024-02-23T00:24:21.734 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:20.630189+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:21.734 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:20.631058+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:21.734 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:20.631622+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:21.734 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:20.636732+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:21.735 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:20.073583+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:21.735 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:24:20.614481+0000 mgr.y (mgr.14146) 82 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:24:21.735 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:20.621684+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:21.735 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:20.622503+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:24:21.735 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:24:20.622939+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 16098M 2024-02-23T00:24:21.735 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:20.630189+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:21.736 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:20.631058+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:21.736 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:20.631622+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:21.736 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:20.636732+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:21.753 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch daemon add osd smithi087:vg_nvme/lv_3 2024-02-23T00:24:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:20.073583+0000 mgr.y (mgr.14146) 81 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:24:20.614481+0000 mgr.y (mgr.14146) 82 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:24:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:20.621684+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:20.622503+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:24:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:24:20.622939+0000 mgr.y (mgr.14146) 83 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 16098M 2024-02-23T00:24:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:20.630189+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:20.631058+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:20.631622+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:20.636732+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:23.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 -- 172.21.15.87:0/1341941443 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 msgr2=0x7f95d40ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:23.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 --2- 172.21.15.87:0/1341941443 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 0x7f95d40ffb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f95cc009be0 tx=0x7f95cc004d10).stop 2024-02-23T00:24:23.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 -- 172.21.15.87:0/1341941443 shutdown_connections 2024-02-23T00:24:23.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 --2- 172.21.15.87:0/1341941443 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 0x7f95d40ffb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:23.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 --2- 172.21.15.87:0/1341941443 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95d40616d0 0x7f95d40fa670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:23.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 --2- 172.21.15.87:0/1341941443 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95d40fabf0 0x7f95d40fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:23.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 -- 172.21.15.87:0/1341941443 >> 172.21.15.87:0/1341941443 conn(0x7f95d40f2190 msgr2=0x7f95d40f45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:23.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 -- 172.21.15.87:0/1341941443 shutdown_connections 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 -- 172.21.15.87:0/1341941443 wait complete. 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 librados: starting msgr at 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 librados: starting objecter 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 Processor -- start 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.658+0000 7f95da684700 1 -- start start 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95da684700 1 librados: setting wanted keys 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95da684700 1 librados: calling monclient init 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95da684700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95d40616d0 0x7f95d410a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95da684700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95d40fabf0 0x7f95d410a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95da684700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 0x7f95d4111730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95da684700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4111c20 con 0x7f95d40fabf0 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95da684700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4111d60 con 0x7f95d40616d0 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95da684700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4111ea0 con 0x7f95d40fd680 2024-02-23T00:24:23.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d8c21700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 0x7f95d4111730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95d40fabf0 0x7f95d410a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d8c21700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 0x7f95d4111730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:57518/0 (socket says 172.21.15.87:57518) 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d8c21700 1 -- 172.21.15.87:0/2347681253 learned_addr learned my addr 172.21.15.87:0/2347681253 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d3fff700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95d40616d0 0x7f95d410a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d8c21700 1 -- 172.21.15.87:0/2347681253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95d40616d0 msgr2=0x7f95d410a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d8c21700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95d40616d0 0x7f95d410a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d8c21700 1 -- 172.21.15.87:0/2347681253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95d40fabf0 msgr2=0x7f95d410a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d8c21700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95d40fabf0 0x7f95d410a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d8c21700 1 -- 172.21.15.87:0/2347681253 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c4005040 con 0x7f95d40fd680 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d3fff700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95d40616d0 0x7f95d410a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d37fe700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95d40fabf0 0x7f95d410a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d8c21700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 0x7f95d4111730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f95cc004a30 tx=0x7f95cc004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:23.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d17fa700 10 client.?.objecter ms_handle_connect 0x7f95d40fd680 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d17fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d17fa700 1 -- 172.21.15.87:0/2347681253 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc00e030 con 0x7f95d40fd680 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.659+0000 7f95d17fa700 1 -- 172.21.15.87:0/2347681253 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95cc007bf0 con 0x7f95d40fd680 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.660+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95cc0097c0 con 0x7f95d40fd680 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.660+0000 7f95d17fa700 1 -- 172.21.15.87:0/2347681253 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc018070 con 0x7f95d40fd680 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.660+0000 7f95da684700 10 client.24149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.660+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d41123c0 con 0x7f95d40fd680 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.660+0000 7f95da684700 1 librados: init done 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.661+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d41029f0 con 0x7f95d40fd680 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.661+0000 7f95d17fa700 1 -- 172.21.15.87:0/2347681253 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f95cc01d420 con 0x7f95d40fd680 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.661+0000 7f95d17fa700 1 --2- 172.21.15.87:0/2347681253 >> v2:172.21.15.87:6800/1121018082 conn(0x7f95c0062ba0 0x7f95c0065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.661+0000 7f95d3fff700 1 --2- 172.21.15.87:0/2347681253 >> v2:172.21.15.87:6800/1121018082 conn(0x7f95c0062ba0 0x7f95c0065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.661+0000 7f95d17fa700 1 -- 172.21.15.87:0/2347681253 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1484+0+0 (secure 0 0 0) 0x7f95cc07cfc0 con 0x7f95d40fd680 2024-02-23T00:24:23.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.661+0000 7f95d17fa700 10 client.24149.objecter ms_dispatch 0x7f95d4001550 osd_map(9..9 src has 1..9) v4 2024-02-23T00:24:23.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.661+0000 7f95d17fa700 3 client.24149.objecter handle_osd_map got epochs [9,9] > 0 2024-02-23T00:24:23.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.661+0000 7f95d17fa700 3 client.24149.objecter handle_osd_map decoding full epoch 9 2024-02-23T00:24:23.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.662+0000 7f95d17fa700 20 client.24149.objecter dump_active .. 0 homeless 2024-02-23T00:24:23.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.662+0000 7f95d17fa700 10 client.24149.objecter ms_handle_connect 0x7f95c0062ba0 2024-02-23T00:24:23.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.662+0000 7f95d3fff700 1 --2- 172.21.15.87:0/2347681253 >> v2:172.21.15.87:6800/1121018082 conn(0x7f95c0062ba0 0x7f95c0065060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f95bc004ca0 tx=0x7f95bc00b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:23.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.666+0000 7f95d17fa700 1 -- 172.21.15.87:0/2347681253 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f95cc0abf60 con 0x7f95d40fd680 2024-02-23T00:24:23.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:22.074037+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:23.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:22.074037+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:23.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:23.839+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f95d40fe280 con 0x7f95c0062ba0 2024-02-23T00:24:23.941 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:22.074037+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:23.843199+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:24:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:23.845009+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:24:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:23.845462+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:24.943 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:23.843199+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:24:24.943 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:23.845009+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:24:24.943 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:23.845462+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:24.943 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:23.843199+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:24:24.943 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:23.845009+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:24:24.943 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:23.845462+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:23.841179+0000 mgr.y (mgr.14146) 85 : audit [DBG] from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:24:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:24.074393+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:25.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:23.841179+0000 mgr.y (mgr.14146) 85 : audit [DBG] from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:24:25.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:24.074393+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:25.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:23.841179+0000 mgr.y (mgr.14146) 85 : audit [DBG] from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:24:25.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:24.074393+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:26.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:26.207449+0000 mon.a (mon.0) 249 : audit [INF] from='client.? 172.21.15.87:0/1480216905' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85525ce8-38d5-4840-8057-547cb176c859"}]: dispatch 2024-02-23T00:24:26.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:26.211114+0000 mon.a (mon.0) 250 : audit [INF] from='client.? 172.21.15.87:0/1480216905' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85525ce8-38d5-4840-8057-547cb176c859"}]': finished 2024-02-23T00:24:26.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:26.211149+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-23T00:24:26.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:26.211235+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:27.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:26.207449+0000 mon.a (mon.0) 249 : audit [INF] from='client.? 172.21.15.87:0/1480216905' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85525ce8-38d5-4840-8057-547cb176c859"}]: dispatch 2024-02-23T00:24:27.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:26.211114+0000 mon.a (mon.0) 250 : audit [INF] from='client.? 172.21.15.87:0/1480216905' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85525ce8-38d5-4840-8057-547cb176c859"}]': finished 2024-02-23T00:24:27.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:26.211149+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-23T00:24:27.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:26.211235+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:27.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:26.207449+0000 mon.a (mon.0) 249 : audit [INF] from='client.? 172.21.15.87:0/1480216905' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "85525ce8-38d5-4840-8057-547cb176c859"}]: dispatch 2024-02-23T00:24:27.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:26.211114+0000 mon.a (mon.0) 250 : audit [INF] from='client.? 172.21.15.87:0/1480216905' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85525ce8-38d5-4840-8057-547cb176c859"}]': finished 2024-02-23T00:24:27.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:26.211149+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-23T00:24:27.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:26.211235+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:27.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:26.074675+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:27.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:26.817418+0000 mon.c (mon.2) 5 : audit [DBG] from='client.? 172.21.15.87:0/3836874930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:24:28.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:26.074675+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:28.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:26.817418+0000 mon.c (mon.2) 5 : audit [DBG] from='client.? 172.21.15.87:0/3836874930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:24:28.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:26.074675+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:28.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:26.817418+0000 mon.c (mon.2) 5 : audit [DBG] from='client.? 172.21.15.87:0/3836874930' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:24:28.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:28.661+0000 7f95d9e83700 10 client.24149.objecter tick 2024-02-23T00:24:29.749 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:28.075033+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:29.750 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:28.075033+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:28.075033+0000 mgr.y (mgr.14146) 88 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:31.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:30.075446+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:31.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:30.075446+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:31.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:30.075446+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:32.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:32.606322+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-23T00:24:32.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:32.606984+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:33.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:32.606322+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-23T00:24:33.065 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:32.606984+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:33.066 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:32.606322+0000 mon.a (mon.0) 253 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-23T00:24:33.066 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:32.606984+0000 mon.a (mon.0) 254 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:33.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:33.662+0000 7f95d9e83700 10 client.24149.objecter tick 2024-02-23T00:24:33.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:32.075647+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:33.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:24:32.607528+0000 mgr.y (mgr.14146) 91 : cephadm [INF] Deploying daemon osd.1 on smithi087 2024-02-23T00:24:33.968 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:32.075647+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:33.968 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:24:32.607528+0000 mgr.y (mgr.14146) 91 : cephadm [INF] Deploying daemon osd.1 on smithi087 2024-02-23T00:24:33.968 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:32.075647+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:33.968 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:24:32.607528+0000 mgr.y (mgr.14146) 91 : cephadm [INF] Deploying daemon osd.1 on smithi087 2024-02-23T00:24:35.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:34.076039+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:35.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:34.076039+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:35.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:34.076039+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:37.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:36.076365+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:37.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:37.622329+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:37.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:37.635514+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:37.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:36.076365+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:37.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:37.622329+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:37.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:37.635514+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:36.076365+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:37.622329+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:37.635514+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:38.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:38.663+0000 7f95d9e83700 10 client.24149.objecter tick 2024-02-23T00:24:39.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:38.076599+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:40.032 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:38.076599+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:40.032 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:38.076599+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:41.007 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 1 on host 'smithi087' 2024-02-23T00:24:41.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.005+0000 7f95d17fa700 1 -- 172.21.15.87:0/2347681253 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f95d40fe280 con 0x7f95c0062ba0 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.007+0000 7f95da684700 10 librados: watch_flush enter 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.007+0000 7f95da684700 10 librados: watch_flush exit 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.007+0000 7f95da684700 20 client.24149.objecter shutdown clearing up homeless session... 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.007+0000 7f95da684700 10 client.24149.objecter successfully canceled tick 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.007+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 >> v2:172.21.15.87:6800/1121018082 conn(0x7f95c0062ba0 msgr2=0x7f95c0065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.007+0000 7f95da684700 1 --2- 172.21.15.87:0/2347681253 >> v2:172.21.15.87:6800/1121018082 conn(0x7f95c0062ba0 0x7f95c0065060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f95bc004ca0 tx=0x7f95bc00b040).stop 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 msgr2=0x7f95d4111730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 0x7f95d4111730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f95cc004a30 tx=0x7f95cc004d10).stop 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 shutdown_connections 2024-02-23T00:24:41.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95d40fd680 0x7f95d4111730 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:41.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95d40616d0 0x7f95d410a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:41.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 --2- 172.21.15.87:0/2347681253 >> v2:172.21.15.87:6800/1121018082 conn(0x7f95c0062ba0 0x7f95c0065060 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:41.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 --2- 172.21.15.87:0/2347681253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95d40fabf0 0x7f95d410a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:41.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 >> 172.21.15.87:0/2347681253 conn(0x7f95d40f2190 msgr2=0x7f95d40f4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:41.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 shutdown_connections 2024-02-23T00:24:41.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 -- 172.21.15.87:0/2347681253 wait complete. 2024-02-23T00:24:41.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:41.008+0000 7f95da684700 1 librados: shutdown 2024-02-23T00:24:41.649 DEBUG:teuthology.orchestra.run.smithi087:osd.1> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.1.service 2024-02-23T00:24:41.651 INFO:tasks.cephadm:Deploying osd.2 on smithi087 with /dev/vg_nvme/lv_2... 2024-02-23T00:24:41.651 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-23T00:24:41.682 INFO:journalctl@ceph.osd.1.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:24:41.870 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:40.076931+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:41.871 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:40.733460+0000 mon.a (mon.0) 257 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-23T00:24:41.871 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:41.005893+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:41.871 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:41.042236+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:41.871 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:40.076931+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:41.871 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:40.733460+0000 mon.a (mon.0) 257 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-23T00:24:41.871 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:41.005893+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:41.871 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:41.042236+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:40.076931+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:40.733460+0000 mon.a (mon.0) 257 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-23T00:24:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:41.005893+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:41.042236+0000 mon.a (mon.0) 259 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:42.788 INFO:journalctl@ceph.osd.1.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-1[52702]: debug 2024-02-23T00:24:42.682+0000 7f108e77d700 -1 osd.1 0 waiting for initial osdmap 2024-02-23T00:24:42.789 INFO:journalctl@ceph.osd.1.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-1[52702]: debug 2024-02-23T00:24:42.685+0000 7f1087ef3700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-23T00:24:42.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:41.677783+0000 mon.a (mon.0) 260 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-23T00:24:42.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:41.677851+0000 mon.a (mon.0) 261 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-23T00:24:42.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:41.677977+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:42.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:41.678343+0000 mon.a (mon.0) 263 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:24:42.789 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:41.677783+0000 mon.a (mon.0) 260 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-23T00:24:42.790 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:41.677851+0000 mon.a (mon.0) 261 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-23T00:24:42.790 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:41.677977+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:42.790 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:41.678343+0000 mon.a (mon.0) 263 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:24:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:41.677783+0000 mon.a (mon.0) 260 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-23T00:24:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:41.677851+0000 mon.a (mon.0) 261 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-23T00:24:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:41.677977+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:41.678343+0000 mon.a (mon.0) 263 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:24:44.060 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:42.077248+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:44.060 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:42.681053+0000 mon.a (mon.0) 264 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:24:44.060 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:42.681099+0000 mon.a (mon.0) 265 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-23T00:24:44.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:42.681459+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:42.687517+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:43.684111+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.061 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:42.077248+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:44.061 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:42.681053+0000 mon.a (mon.0) 264 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:24:44.061 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:42.681099+0000 mon.a (mon.0) 265 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-23T00:24:44.061 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:42.681459+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.061 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:42.687517+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.061 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:43.684111+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:42.077248+0000 mgr.y (mgr.14146) 96 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:42.681053+0000 mon.a (mon.0) 264 : audit [INF] from='osd.1 v2:172.21.15.87:6805/685354171' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:24:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:42.681099+0000 mon.a (mon.0) 265 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-23T00:24:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:42.681459+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:42.687517+0000 mon.a (mon.0) 267 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:43.684111+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:41.738065+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:24:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:41.738119+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:24:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:43.687720+0000 mon.a (mon.0) 269 : cluster [INF] osd.1 v2:172.21.15.87:6805/685354171 boot 2024-02-23T00:24:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:43.687745+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-23T00:24:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:43.688605+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:44.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:41.738065+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:24:44.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:41.738119+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:24:44.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:43.687720+0000 mon.a (mon.0) 269 : cluster [INF] osd.1 v2:172.21.15.87:6805/685354171 boot 2024-02-23T00:24:44.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:43.687745+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-23T00:24:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:43.688605+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:41.738065+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:24:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:41.738119+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:24:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:43.687720+0000 mon.a (mon.0) 269 : cluster [INF] osd.1 v2:172.21.15.87:6805/685354171 boot 2024-02-23T00:24:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:43.687745+0000 mon.a (mon.0) 270 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-23T00:24:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:43.688605+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:24:45.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:44.077668+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:45.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:44.698583+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-23T00:24:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.529724+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:45.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:44.077668+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:45.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:44.698583+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-23T00:24:45.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.529724+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:46.085 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-23T00:24:46.085 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-23T00:24:46.085 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10+0 records in 2024-02-23T00:24:46.085 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2024-02-23T00:24:46.086 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114693 s, 914 MB/s 2024-02-23T00:24:46.086 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping successful for: 2024-02-23T00:24:46.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:44.077668+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-23T00:24:46.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:44.698583+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-23T00:24:46.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.529724+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:46.723 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch daemon add osd smithi087:vg_nvme/lv_2 2024-02-23T00:24:47.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:24:45.525255+0000 mgr.y (mgr.14146) 98 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:24:47.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.825197+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:24:47.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.825878+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:24:47.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.831614+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:47.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.832894+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:47.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.833508+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:47.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.838654+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.840436+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.841561+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.842061+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:45.846357+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:47.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:24:45.525255+0000 mgr.y (mgr.14146) 98 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:24:47.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.825197+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:24:47.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.825878+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:24:47.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.831614+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:47.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.832894+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:47.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.833508+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:47.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.838654+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:47.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.840436+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:47.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.841561+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:47.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.842061+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:47.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:45.846357+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:24:45.525255+0000 mgr.y (mgr.14146) 98 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:24:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.825197+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:24:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.825878+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:24:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.831614+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.832894+0000 mon.a (mon.0) 277 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.833508+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.838654+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.840436+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:24:47.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.841561+0000 mon.a (mon.0) 281 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:47.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.842061+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:24:47.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:45.846357+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:24:48.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:24:45.826308+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 8049M 2024-02-23T00:24:48.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:46.077987+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:48.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:24:45.826308+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 8049M 2024-02-23T00:24:48.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:46.077987+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:48.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:24:45.826308+0000 mgr.y (mgr.14146) 99 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 8049M 2024-02-23T00:24:48.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:46.077987+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:48.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.806+0000 7f953a9be700 1 -- 172.21.15.87:0/1531050887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f7270 msgr2=0x7f95340f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:48.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.806+0000 7f953a9be700 1 --2- 172.21.15.87:0/1531050887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f7270 0x7f95340f7ca0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9528004870 tx=0x7f9528009cb0).stop 2024-02-23T00:24:48.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.806+0000 7f953a9be700 1 -- 172.21.15.87:0/1531050887 shutdown_connections 2024-02-23T00:24:48.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.806+0000 7f953a9be700 1 --2- 172.21.15.87:0/1531050887 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95340f8380 0x7f95340feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.806+0000 7f953a9be700 1 --2- 172.21.15.87:0/1531050887 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95340f6960 0x7f95340f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.806+0000 7f953a9be700 1 --2- 172.21.15.87:0/1531050887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f7270 0x7f95340f7ca0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.806+0000 7f953a9be700 1 -- 172.21.15.87:0/1531050887 >> 172.21.15.87:0/1531050887 conn(0x7f95340f2180 msgr2=0x7f95340f45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.806+0000 7f953a9be700 1 -- 172.21.15.87:0/1531050887 shutdown_connections 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.806+0000 7f953a9be700 1 -- 172.21.15.87:0/1531050887 wait complete. 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 librados: starting msgr at 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 librados: starting objecter 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 Processor -- start 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 -- start start 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 librados: setting wanted keys 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 librados: calling monclient init 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f6960 0x7f9534108460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95340f7270 0x7f9534108950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:48.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95340f8380 0x7f953410fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953410ff20 con 0x7f95340f6960 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9534110060 con 0x7f95340f8380 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.807+0000 7f953a9be700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95341101a0 con 0x7f95340f7270 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f9533fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f6960 0x7f9534108460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95337fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95340f7270 0x7f9534108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95337fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95340f7270 0x7f9534108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:47016/0 (socket says 172.21.15.87:47016) 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f9533fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f6960 0x7f9534108460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58910/0 (socket says 172.21.15.87:58910) 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95337fe700 1 -- 172.21.15.87:0/2603981098 learned_addr learned my addr 172.21.15.87:0/2603981098 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f9538f5b700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95340f8380 0x7f953410fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95337fe700 1 -- 172.21.15.87:0/2603981098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95340f8380 msgr2=0x7f953410fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95337fe700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95340f8380 0x7f953410fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95337fe700 1 -- 172.21.15.87:0/2603981098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f6960 msgr2=0x7f9534108460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:24:48.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95337fe700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f6960 0x7f9534108460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:24:48.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95337fe700 1 -- 172.21.15.87:0/2603981098 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9524005040 con 0x7f95340f7270 2024-02-23T00:24:48.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f9533fff700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f6960 0x7f9534108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:24:48.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f9538f5b700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95340f8380 0x7f953410fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:24:48.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95337fe700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95340f7270 0x7f9534108950 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9528004710 tx=0x7f952800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:48.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95317fa700 10 client.?.objecter ms_handle_connect 0x7f95340f7270 2024-02-23T00:24:48.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95317fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:24:48.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95317fa700 1 -- 172.21.15.87:0/2603981098 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952800e030 con 0x7f95340f7270 2024-02-23T00:24:48.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9528005040 con 0x7f95340f7270 2024-02-23T00:24:48.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95317fa700 1 -- 172.21.15.87:0/2603981098 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9528007d50 con 0x7f95340f7270 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f95317fa700 1 -- 172.21.15.87:0/2603981098 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9528019070 con 0x7f95340f7270 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f953a9be700 10 client.24176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95340f9130 con 0x7f95340f7270 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.808+0000 7f953a9be700 1 librados: init done 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.809+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f953405ffd0 con 0x7f95340f7270 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.810+0000 7f95317fa700 1 -- 172.21.15.87:0/2603981098 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f952801e420 con 0x7f95340f7270 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.810+0000 7f95317fa700 1 --2- 172.21.15.87:0/2603981098 >> v2:172.21.15.87:6800/1121018082 conn(0x7f951c062ba0 0x7f951c065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.810+0000 7f9533fff700 1 --2- 172.21.15.87:0/2603981098 >> v2:172.21.15.87:6800/1121018082 conn(0x7f951c062ba0 0x7f951c065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.811+0000 7f95317fa700 1 -- 172.21.15.87:0/2603981098 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 1776+0+0 (secure 0 0 0) 0x7f952807d2a0 con 0x7f95340f7270 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.811+0000 7f95317fa700 10 client.24176.objecter ms_dispatch 0x7f9534001550 osd_map(14..14 src has 1..14) v4 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.811+0000 7f95317fa700 3 client.24176.objecter handle_osd_map got epochs [14,14] > 0 2024-02-23T00:24:48.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.811+0000 7f95317fa700 3 client.24176.objecter handle_osd_map decoding full epoch 14 2024-02-23T00:24:48.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.811+0000 7f95317fa700 20 client.24176.objecter dump_active .. 0 homeless 2024-02-23T00:24:48.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.811+0000 7f95317fa700 10 client.24176.objecter ms_handle_connect 0x7f951c062ba0 2024-02-23T00:24:48.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.811+0000 7f9533fff700 1 --2- 172.21.15.87:0/2603981098 >> v2:172.21.15.87:6800/1121018082 conn(0x7f951c062ba0 0x7f951c065060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f9524009760 tx=0x7f952400b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:24:48.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.814+0000 7f95317fa700 1 -- 172.21.15.87:0/2603981098 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f952804e570 con 0x7f95340f7270 2024-02-23T00:24:48.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:48.987+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f95341019b0 con 0x7f951c062ba0 2024-02-23T00:24:49.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:48.078416+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:48.990825+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:24:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:48.992644+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:24:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:48.993113+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:48.078416+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:48.990825+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:24:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:48.992644+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:24:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:48.993113+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:50.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:48.078416+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:50.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:48.990825+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:24:50.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:48.992644+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:24:50.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:48.993113+0000 mon.a (mon.0) 286 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:51.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:48.988700+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:24:51.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:48.988700+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:24:51.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:48.988700+0000 mgr.y (mgr.14146) 102 : audit [DBG] from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:24:52.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:50.078699+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:51.384087+0000 mon.c (mon.2) 6 : audit [INF] from='client.? 172.21.15.87:0/3977974355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa"}]: dispatch 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:51.384418+0000 mon.a (mon.0) 287 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa"}]: dispatch 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:51.389992+0000 mon.a (mon.0) 288 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa"}]': finished 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:51.390037+0000 mon.a (mon.0) 289 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:51.390151+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:50.078699+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:51.384087+0000 mon.c (mon.2) 6 : audit [INF] from='client.? 172.21.15.87:0/3977974355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa"}]: dispatch 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:51.384418+0000 mon.a (mon.0) 287 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa"}]: dispatch 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:51.389992+0000 mon.a (mon.0) 288 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa"}]': finished 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:51.390037+0000 mon.a (mon.0) 289 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-23T00:24:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:51.390151+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:24:52.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:50.078699+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:52.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:51.384087+0000 mon.c (mon.2) 6 : audit [INF] from='client.? 172.21.15.87:0/3977974355' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa"}]: dispatch 2024-02-23T00:24:52.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:51.384418+0000 mon.a (mon.0) 287 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa"}]: dispatch 2024-02-23T00:24:52.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:51.389992+0000 mon.a (mon.0) 288 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa"}]': finished 2024-02-23T00:24:52.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:51.390037+0000 mon.a (mon.0) 289 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-23T00:24:52.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:51.390151+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:24:53.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:51.984629+0000 mon.c (mon.2) 7 : audit [DBG] from='client.? 172.21.15.87:0/1856955309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:24:53.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:51.984629+0000 mon.c (mon.2) 7 : audit [DBG] from='client.? 172.21.15.87:0/1856955309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:24:53.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:51.984629+0000 mon.c (mon.2) 7 : audit [DBG] from='client.? 172.21.15.87:0/1856955309' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:24:53.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:53.809+0000 7f953a1bd700 10 client.24176.objecter tick 2024-02-23T00:24:54.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:52.079101+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:54.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:52.079101+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:54.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:52.079101+0000 mgr.y (mgr.14146) 104 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:55.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:54.079587+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:55.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:54.079587+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:56.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:54.079587+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:57.831 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:56.080016+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:57.831 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:56.080016+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:58.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:56.080016+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:58.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:24:58.810+0000 7f953a1bd700 10 client.24176.objecter tick 2024-02-23T00:24:59.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:57.849537+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-23T00:24:59.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:24:57.850185+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:59.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:57.849537+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-23T00:24:59.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:24:57.850185+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:59.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:57.849537+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-23T00:24:59.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:24:57.850185+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:24:59.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:24:57.850760+0000 mgr.y (mgr.14146) 107 : cephadm [INF] Deploying daemon osd.2 on smithi087 2024-02-23T00:24:59.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:24:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:24:58.080306+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:24:59.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:24:57.850760+0000 mgr.y (mgr.14146) 107 : cephadm [INF] Deploying daemon osd.2 on smithi087 2024-02-23T00:24:59.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:24:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:24:58.080306+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:00.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:24:57.850760+0000 mgr.y (mgr.14146) 107 : cephadm [INF] Deploying daemon osd.2 on smithi087 2024-02-23T00:25:00.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:24:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:24:58.080306+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:00.080587+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:00.080587+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:00.080587+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:03.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:03.810+0000 7f953a1bd700 10 client.24176.objecter tick 2024-02-23T00:25:03.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:02.080866+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:03.114420+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:03.147732+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:03.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:02.080866+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:03.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:03.114420+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:03.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:03.147732+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:04.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:02.080866+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:04.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:03.114420+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:04.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:03.147732+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:06.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:04.081135+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:06.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:04.081135+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:06.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:04.081135+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:06.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.637+0000 7f95317fa700 1 -- 172.21.15.87:0/2603981098 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f95341019b0 con 0x7f951c062ba0 2024-02-23T00:25:06.640 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 2 on host 'smithi087' 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.639+0000 7f953a9be700 10 librados: watch_flush enter 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.639+0000 7f953a9be700 10 librados: watch_flush exit 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.639+0000 7f953a9be700 20 client.24176.objecter shutdown clearing up homeless session... 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.639+0000 7f953a9be700 10 client.24176.objecter successfully canceled tick 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.639+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 >> v2:172.21.15.87:6800/1121018082 conn(0x7f951c062ba0 msgr2=0x7f951c065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 --2- 172.21.15.87:0/2603981098 >> v2:172.21.15.87:6800/1121018082 conn(0x7f951c062ba0 0x7f951c065060 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f9524009760 tx=0x7f952400b040).stop 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95340f7270 msgr2=0x7f9534108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95340f7270 0x7f9534108950 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9528004710 tx=0x7f952800b040).stop 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 shutdown_connections 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95340f8380 0x7f953410fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:06.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 --2- 172.21.15.87:0/2603981098 >> v2:172.21.15.87:6800/1121018082 conn(0x7f951c062ba0 0x7f951c065060 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:06.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95340f6960 0x7f9534108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:06.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 --2- 172.21.15.87:0/2603981098 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95340f7270 0x7f9534108950 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:06.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 >> 172.21.15.87:0/2603981098 conn(0x7f95340f2180 msgr2=0x7f9534061070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:06.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 shutdown_connections 2024-02-23T00:25:06.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 -- 172.21.15.87:0/2603981098 wait complete. 2024-02-23T00:25:06.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:06.640+0000 7f953a9be700 1 librados: shutdown 2024-02-23T00:25:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:05.933178+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:05.933901+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:05.934554+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:05.942001+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:06.081676+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:06.232014+0000 mon.c (mon.2) 8 : audit [INF] from='osd.2 v2:172.21.15.87:6809/3605394228' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-23T00:25:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:06.232290+0000 mon.a (mon.0) 299 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-23T00:25:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:06.638498+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:06.659193+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:07.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:05.933178+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:05.933901+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:05.934554+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:05.942001+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:06.081676+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:06.232014+0000 mon.c (mon.2) 8 : audit [INF] from='osd.2 v2:172.21.15.87:6809/3605394228' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-23T00:25:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:06.232290+0000 mon.a (mon.0) 299 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-23T00:25:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:06.638498+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:06.659193+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:07.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:05.933178+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:05.933901+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:05.934554+0000 mon.a (mon.0) 297 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:05.942001+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:06.081676+0000 mgr.y (mgr.14146) 112 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:06.232014+0000 mon.c (mon.2) 8 : audit [INF] from='osd.2 v2:172.21.15.87:6809/3605394228' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-23T00:25:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:06.232290+0000 mon.a (mon.0) 299 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-23T00:25:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:06.638498+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:06.659193+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:07.400 DEBUG:teuthology.orchestra.run.smithi087:osd.2> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.2.service 2024-02-23T00:25:07.402 INFO:tasks.cephadm:Deploying osd.3 on smithi087 with /dev/vg_nvme/lv_1... 2024-02-23T00:25:07.403 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-23T00:25:07.434 INFO:journalctl@ceph.osd.2.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:25:08.333 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-2[57400]: debug 2024-02-23T00:25:07.957+0000 7f684c9d9700 -1 osd.2 0 waiting for initial osdmap 2024-02-23T00:25:08.333 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-2[57400]: debug 2024-02-23T00:25:07.960+0000 7f6849155700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-23T00:25:08.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:06.955906+0000 mon.a (mon.0) 302 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-23T00:25:08.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:06.955955+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-23T00:25:08.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:06.956051+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:08.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:06.956479+0000 mon.c (mon.2) 9 : audit [INF] from='osd.2 v2:172.21.15.87:6809/3605394228' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:25:08.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:06.956769+0000 mon.a (mon.0) 305 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:25:08.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:06.955906+0000 mon.a (mon.0) 302 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-23T00:25:08.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:06.955955+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-23T00:25:08.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:06.956051+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:08.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:06.956479+0000 mon.c (mon.2) 9 : audit [INF] from='osd.2 v2:172.21.15.87:6809/3605394228' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:25:08.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:06.956769+0000 mon.a (mon.0) 305 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:25:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:06.955906+0000 mon.a (mon.0) 302 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-23T00:25:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:06.955955+0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-23T00:25:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:06.956051+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:06.956479+0000 mon.c (mon.2) 9 : audit [INF] from='osd.2 v2:172.21.15.87:6809/3605394228' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:25:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:06.956769+0000 mon.a (mon.0) 305 : audit [INF] from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:25:09.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:07.951898+0000 mon.a (mon.0) 306 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:25:09.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:07.951974+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-23T00:25:09.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:07.952330+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:09.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:07.955503+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:09.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:08.081940+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:09.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:08.955258+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:09.567 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:07.951898+0000 mon.a (mon.0) 306 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:25:09.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:07.951974+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-23T00:25:09.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:07.952330+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:09.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:07.955503+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:09.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:08.081940+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:09.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:08.955258+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:07.951898+0000 mon.a (mon.0) 306 : audit [INF] from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:25:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:07.951974+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-23T00:25:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:07.952330+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:07.955503+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:08.081940+0000 mgr.y (mgr.14146) 113 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-23T00:25:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:08.955258+0000 mon.a (mon.0) 310 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:07.253499+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:07.253536+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:08.961034+0000 mon.a (mon.0) 311 : cluster [INF] osd.2 v2:172.21.15.87:6809/3605394228 boot 2024-02-23T00:25:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:08.961067+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-23T00:25:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:08.961430+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:08.961746+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-23T00:25:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:09.961157+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-23T00:25:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:09.961226+0000 mon.a (mon.0) 316 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-23T00:25:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:09.961885+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-23T00:25:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:07.253499+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:07.253536+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:08.961034+0000 mon.a (mon.0) 311 : cluster [INF] osd.2 v2:172.21.15.87:6809/3605394228 boot 2024-02-23T00:25:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:08.961067+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-23T00:25:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:08.961430+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:08.961746+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-23T00:25:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:09.961157+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-23T00:25:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:09.961226+0000 mon.a (mon.0) 316 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-23T00:25:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:09.961885+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-23T00:25:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:07.253499+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:07.253536+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:08.961034+0000 mon.a (mon.0) 311 : cluster [INF] osd.2 v2:172.21.15.87:6809/3605394228 boot 2024-02-23T00:25:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:08.961067+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-23T00:25:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:08.961430+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:25:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:08.961746+0000 mon.a (mon.0) 314 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-23T00:25:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:09.961157+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-23T00:25:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:09.961226+0000 mon.a (mon.0) 316 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-23T00:25:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:09.961885+0000 mon.a (mon.0) 317 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-23T00:25:10.739 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-23T00:25:10.739 INFO:teuthology.orchestra.run.smithi087.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-23T00:25:10.739 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10+0 records in 2024-02-23T00:25:10.739 INFO:teuthology.orchestra.run.smithi087.stderr:10+0 records out 2024-02-23T00:25:10.739 INFO:teuthology.orchestra.run.smithi087.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112265 s, 934 MB/s 2024-02-23T00:25:10.739 INFO:teuthology.orchestra.run.smithi087.stderr:--> Zapping successful for: 2024-02-23T00:25:11.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:10.082385+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:11.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:10.967694+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-23T00:25:11.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:10.967772+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-23T00:25:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:10.082385+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:10.967694+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-23T00:25:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:10.967772+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-23T00:25:11.691 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:10.082385+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:10.967694+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-23T00:25:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:10.967772+0000 mon.a (mon.0) 319 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-23T00:25:12.109 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch daemon add osd smithi087:vg_nvme/lv_1 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:11.969517+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:12.082648+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v96: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:25:12.189876+0000 mgr.y (mgr.14146) 116 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:12.194632+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:12.195520+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:12.196117+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:12.196656+0000 mon.a (mon.0) 324 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:25:12.197144+0000 mgr.y (mgr.14146) 117 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 5366M 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:12.201592+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:12.202418+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:12.203040+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:12.207756+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:11.969517+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:12.082648+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v96: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:25:12.189876+0000 mgr.y (mgr.14146) 116 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:12.194632+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:12.195520+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:12.196117+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:12.196656+0000 mon.a (mon.0) 324 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:13.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:25:12.197144+0000 mgr.y (mgr.14146) 117 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 5366M 2024-02-23T00:25:13.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:12.201592+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:13.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:12.202418+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:13.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:12.203040+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:13.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:12.207756+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:11.969517+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:12.082648+0000 mgr.y (mgr.14146) 115 : cluster [DBG] pgmap v96: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:25:12.189876+0000 mgr.y (mgr.14146) 116 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:12.194632+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:12.195520+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:12.196117+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:12.196656+0000 mon.a (mon.0) 324 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:25:12.197144+0000 mgr.y (mgr.14146) 117 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 5366M 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:12.201592+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:12.202418+0000 mon.a (mon.0) 326 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:12.203040+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:12.207756+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 -- 172.21.15.87:0/729614070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f7a80 msgr2=0x7f01740f7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 --2- 172.21.15.87:0/729614070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f7a80 0x7f01740f7ea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0170004870 tx=0x7f0170009cb0).stop 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 -- 172.21.15.87:0/729614070 shutdown_connections 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 --2- 172.21.15.87:0/729614070 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01740f9670 0x7f01740fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 --2- 172.21.15.87:0/729614070 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f7a80 0x7f01740f7ea0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 --2- 172.21.15.87:0/729614070 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01740f8d20 0x7f01740f9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 -- 172.21.15.87:0/729614070 >> 172.21.15.87:0/729614070 conn(0x7f01740f32a0 msgr2=0x7f01740f56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 -- 172.21.15.87:0/729614070 shutdown_connections 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 -- 172.21.15.87:0/729614070 wait complete. 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 librados: starting msgr at 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 librados: starting objecter 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 Processor -- start 2024-02-23T00:25:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 -- start start 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 librados: setting wanted keys 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 librados: calling monclient init 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01740f7a80 0x7f0174107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f8d20 0x7f01741084b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01740f9670 0x7f017410f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017410fa80 con 0x7f01740f8d20 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017410fbc0 con 0x7f01740f7a80 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.110+0000 7f017c48d700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017410fd00 con 0x7f01740f9670 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f0179a28700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f8d20 0x7f01741084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017aa2a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01740f9670 0x7f017410f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f0179a28700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f8d20 0x7f01741084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48186/0 (socket says 172.21.15.87:48186) 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017aa2a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01740f9670 0x7f017410f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50358/0 (socket says 172.21.15.87:50358) 2024-02-23T00:25:14.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f0179a28700 1 -- 172.21.15.87:0/1850926941 learned_addr learned my addr 172.21.15.87:0/1850926941 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017a229700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01740f7a80 0x7f0174107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017aa2a700 1 -- 172.21.15.87:0/1850926941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01740f7a80 msgr2=0x7f0174107fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017aa2a700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01740f7a80 0x7f0174107fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017aa2a700 1 -- 172.21.15.87:0/1850926941 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f8d20 msgr2=0x7f01741084b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017aa2a700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f8d20 0x7f01741084b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017aa2a700 1 -- 172.21.15.87:0/1850926941 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c0097c0 con 0x7f01740f9670 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017a229700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01740f7a80 0x7f0174107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f0179a28700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f8d20 0x7f01741084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017aa2a700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01740f9670 0x7f017410f590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f016c001cb0 tx=0x7f016c004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f016b7fe700 10 client.?.objecter ms_handle_connect 0x7f01740f9670 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f016b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f016b7fe700 1 -- 172.21.15.87:0/1850926941 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c00fa30 con 0x7f01740f9670 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f016b7fe700 1 -- 172.21.15.87:0/1850926941 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f016c0095a0 con 0x7f01740f9670 2024-02-23T00:25:14.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0170005040 con 0x7f01740f9670 2024-02-23T00:25:14.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f016b7fe700 1 -- 172.21.15.87:0/1850926941 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c007d50 con 0x7f01740f9670 2024-02-23T00:25:14.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017c48d700 10 client.24203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:25:14.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0174110330 con 0x7f01740f9670 2024-02-23T00:25:14.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.111+0000 7f017c48d700 1 librados: init done 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.112+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017405f360 con 0x7f01740f9670 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f016b7fe700 1 -- 172.21.15.87:0/1850926941 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f016c0090d0 con 0x7f01740f9670 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f016b7fe700 1 --2- 172.21.15.87:0/1850926941 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0160062ba0 0x7f0160065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f016b7fe700 1 -- 172.21.15.87:0/1850926941 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 2484+0+0 (secure 0 0 0) 0x7f016c07c2d0 con 0x7f01740f9670 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f016b7fe700 10 client.24203.objecter ms_dispatch 0x7f0174001550 osd_map(21..21 src has 1..21) v4 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f016b7fe700 3 client.24203.objecter handle_osd_map got epochs [21,21] > 0 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f016b7fe700 3 client.24203.objecter handle_osd_map decoding full epoch 21 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f017a229700 1 --2- 172.21.15.87:0/1850926941 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0160062ba0 0x7f0160065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f016b7fe700 20 client.24203.objecter dump_active .. 0 homeless 2024-02-23T00:25:14.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f016b7fe700 10 client.24203.objecter ms_handle_connect 0x7f0160062ba0 2024-02-23T00:25:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.113+0000 7f017a229700 1 --2- 172.21.15.87:0/1850926941 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0160062ba0 0x7f0160065060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0170009da0 tx=0x7f017000b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:14.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.117+0000 7f016b7fe700 1 -- 172.21.15.87:0/1850926941 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f016c04d370 con 0x7f01740f9670 2024-02-23T00:25:14.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:14.290+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f01740fee70 con 0x7f0160062ba0 2024-02-23T00:25:15.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:14.082926+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:15.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:14.292286+0000 mgr.y (mgr.14146) 119 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:25:15.418 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:14.294387+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:25:15.418 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:14.296190+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:25:15.418 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:14.296718+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:15.418 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:14.082926+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:15.418 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:14.292286+0000 mgr.y (mgr.14146) 119 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:25:15.418 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:14.294387+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:25:15.418 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:14.296190+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:25:15.418 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:14.296718+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:14.082926+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:14.292286+0000 mgr.y (mgr.14146) 119 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi087:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:25:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:14.294387+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:25:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:14.296190+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:25:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:14.296718+0000 mon.a (mon.0) 331 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:16.083418+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:17.002638+0000 mon.a (mon.0) 332 : audit [INF] from='client.? 172.21.15.87:0/557244181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a"}]: dispatch 2024-02-23T00:25:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:17.007042+0000 mon.a (mon.0) 333 : audit [INF] from='client.? 172.21.15.87:0/557244181' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a"}]': finished 2024-02-23T00:25:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:17.007095+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-23T00:25:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:17.007232+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:16.083418+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:17.002638+0000 mon.a (mon.0) 332 : audit [INF] from='client.? 172.21.15.87:0/557244181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a"}]: dispatch 2024-02-23T00:25:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:17.007042+0000 mon.a (mon.0) 333 : audit [INF] from='client.? 172.21.15.87:0/557244181' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a"}]': finished 2024-02-23T00:25:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:17.007095+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-23T00:25:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:17.007232+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:17.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:16.083418+0000 mgr.y (mgr.14146) 120 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:17.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:17.002638+0000 mon.a (mon.0) 332 : audit [INF] from='client.? 172.21.15.87:0/557244181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a"}]: dispatch 2024-02-23T00:25:17.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:17.007042+0000 mon.a (mon.0) 333 : audit [INF] from='client.? 172.21.15.87:0/557244181' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a"}]': finished 2024-02-23T00:25:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:17.007095+0000 mon.a (mon.0) 334 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-23T00:25:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:17.007232+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:18.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:17.583744+0000 mon.a (mon.0) 336 : audit [DBG] from='client.? 172.21.15.87:0/3602547250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:25:18.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:17.583744+0000 mon.a (mon.0) 336 : audit [DBG] from='client.? 172.21.15.87:0/3602547250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:25:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:17.583744+0000 mon.a (mon.0) 336 : audit [DBG] from='client.? 172.21.15.87:0/3602547250' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:25:19.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:19.112+0000 7f017bc8c700 10 client.24203.objecter tick 2024-02-23T00:25:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:18.083687+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:19.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:18.083687+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:18.083687+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:20.084141+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:21.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:20.084141+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:21.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:20.084141+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:22.084491+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:22.084491+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:23.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:22.084491+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:24.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:24.113+0000 7f017bc8c700 10 client.24203.objecter tick 2024-02-23T00:25:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:23.208162+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-23T00:25:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:23.208763+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:25:23.209258+0000 mgr.y (mgr.14146) 124 : cephadm [INF] Deploying daemon osd.3 on smithi087 2024-02-23T00:25:24.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:23.208162+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-23T00:25:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:23.208763+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:25:23.209258+0000 mgr.y (mgr.14146) 124 : cephadm [INF] Deploying daemon osd.3 on smithi087 2024-02-23T00:25:24.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:23.208162+0000 mon.a (mon.0) 337 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-23T00:25:24.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:23.208763+0000 mon.a (mon.0) 338 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:24.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:25:23.209258+0000 mgr.y (mgr.14146) 124 : cephadm [INF] Deploying daemon osd.3 on smithi087 2024-02-23T00:25:25.165 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:24.084859+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:25.165 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:24.290628+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:25.165 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:24.290709+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:24.084859+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:24.290628+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:24.290709+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:25.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:24.084859+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:25.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:24.290628+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:25.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:24.290709+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:26.298 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:25.161669+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-23T00:25:26.298 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:25.162052+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:26.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:25.161669+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-23T00:25:26.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:25.162052+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:25.161669+0000 mon.a (mon.0) 339 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-23T00:25:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:25.162052+0000 mon.a (mon.0) 340 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:26.085433+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:27.452 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:26.085433+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:27.452 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:26.085433+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:29.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:29.114+0000 7f017bc8c700 10 client.24203.objecter tick 2024-02-23T00:25:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:28.085707+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:28.394420+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:28.409986+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:29.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:28.085707+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:29.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:28.394420+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:29.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:28.409986+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:28.085707+0000 mgr.y (mgr.14146) 127 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:28.394420+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:28.409986+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:30.086089+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:31.192126+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:31.192947+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:31.193665+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:31.198787+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:30.086089+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:31.192126+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:31.192947+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:31.193665+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:31.198787+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:30.086089+0000 mgr.y (mgr.14146) 128 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:31.192126+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:31.192947+0000 mon.a (mon.0) 344 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:31.193665+0000 mon.a (mon.0) 345 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:31.198787+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:32.013 INFO:teuthology.orchestra.run.smithi087.stdout:Created osd(s) 3 on host 'smithi087' 2024-02-23T00:25:32.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.011+0000 7f016b7fe700 1 -- 172.21.15.87:0/1850926941 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f01740fee70 con 0x7f0160062ba0 2024-02-23T00:25:32.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 10 librados: watch_flush enter 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 10 librados: watch_flush exit 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 20 client.24203.objecter shutdown clearing up homeless session... 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 10 client.24203.objecter successfully canceled tick 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0160062ba0 msgr2=0x7f0160065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 --2- 172.21.15.87:0/1850926941 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0160062ba0 0x7f0160065060 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0170009da0 tx=0x7f017000b040).stop 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01740f9670 msgr2=0x7f017410f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01740f9670 0x7f017410f590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f016c001cb0 tx=0x7f016c004c50).stop 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 shutdown_connections 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01740f9670 0x7f017410f590 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01740f7a80 0x7f0174107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 --2- 172.21.15.87:0/1850926941 >> v2:172.21.15.87:6800/1121018082 conn(0x7f0160062ba0 0x7f0160065060 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:32.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 --2- 172.21.15.87:0/1850926941 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01740f8d20 0x7f01741084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:32.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.013+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 >> 172.21.15.87:0/1850926941 conn(0x7f01740f32a0 msgr2=0x7f01740f6660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:32.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.014+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 shutdown_connections 2024-02-23T00:25:32.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.014+0000 7f017c48d700 1 -- 172.21.15.87:0/1850926941 wait complete. 2024-02-23T00:25:32.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:25:32.014+0000 7f017c48d700 1 librados: shutdown 2024-02-23T00:25:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:31.507851+0000 mon.a (mon.0) 347 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-23T00:25:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:32.011101+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:32.016200+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:32.755 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:31.507851+0000 mon.a (mon.0) 347 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-23T00:25:32.755 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:32.011101+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:32.756 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:32.016200+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:32.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:31.507851+0000 mon.a (mon.0) 347 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-23T00:25:32.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:32.011101+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:32.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:32.016200+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:32.782 DEBUG:teuthology.orchestra.run.smithi087:osd.3> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.3.service 2024-02-23T00:25:32.784 INFO:tasks.cephadm:Deploying osd.4 on smithi171 with /dev/vg_nvme/lv_4... 2024-02-23T00:25:32.784 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-23T00:25:32.815 INFO:journalctl@ceph.osd.3.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:25:33.514 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:32.086400+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:33.514 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:32.407504+0000 mon.a (mon.0) 350 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-23T00:25:33.514 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:32.407559+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-23T00:25:33.514 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:32.407685+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:33.514 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:32.408060+0000 mon.a (mon.0) 353 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:25:33.751 INFO:journalctl@ceph.osd.3.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-3[61865]: debug 2024-02-23T00:25:33.412+0000 7fda8148b700 -1 osd.3 0 waiting for initial osdmap 2024-02-23T00:25:33.751 INFO:journalctl@ceph.osd.3.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-3[61865]: debug 2024-02-23T00:25:33.417+0000 7fda7a400700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-23T00:25:33.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:32.086400+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:33.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:32.407504+0000 mon.a (mon.0) 350 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-23T00:25:33.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:32.407559+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-23T00:25:33.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:32.407685+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:33.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:32.408060+0000 mon.a (mon.0) 353 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:25:33.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:32.086400+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:33.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:32.407504+0000 mon.a (mon.0) 350 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-23T00:25:33.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:32.407559+0000 mon.a (mon.0) 351 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-23T00:25:33.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:32.407685+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:33.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:32.408060+0000 mon.a (mon.0) 353 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:25:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:33.409453+0000 mon.a (mon.0) 354 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:25:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:33.409514+0000 mon.a (mon.0) 355 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-02-23T00:25:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:33.409887+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:33.414770+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:34.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:33.409453+0000 mon.a (mon.0) 354 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:25:34.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:33.409514+0000 mon.a (mon.0) 355 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-02-23T00:25:34.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:33.409887+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:34.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:33.414770+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:33.409453+0000 mon.a (mon.0) 354 : audit [INF] from='osd.3 v2:172.21.15.87:6813/128059790' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]': finished 2024-02-23T00:25:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:33.409514+0000 mon.a (mon.0) 355 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-02-23T00:25:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:33.409887+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:33.414770+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:34.977 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-23T00:25:34.977 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-23T00:25:34.977 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-02-23T00:25:34.977 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-02-23T00:25:34.977 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111394 s, 941 MB/s 2024-02-23T00:25:34.977 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-02-23T00:25:35.627 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:32.517835+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:35.627 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:32.517874+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:35.627 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:34.086699+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:35.627 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:34.412619+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:35.627 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:34.415439+0000 mon.a (mon.0) 359 : cluster [INF] osd.3 v2:172.21.15.87:6813/128059790 boot 2024-02-23T00:25:35.627 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:34.415471+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-23T00:25:35.627 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:34.416703+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:35.646 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch daemon add osd smithi171:vg_nvme/lv_4 2024-02-23T00:25:35.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:32.517835+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:32.517874+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:34.086699+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:34.412619+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:34.415439+0000 mon.a (mon.0) 359 : cluster [INF] osd.3 v2:172.21.15.87:6813/128059790 boot 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:34.415471+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:34.416703+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:32.517835+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:32.517874+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:34.086699+0000 mgr.y (mgr.14146) 130 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:34.412619+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:34.415439+0000 mon.a (mon.0) 359 : cluster [INF] osd.3 v2:172.21.15.87:6813/128059790 boot 2024-02-23T00:25:35.731 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:34.415471+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-23T00:25:35.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:34.416703+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:25:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:35.425994+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-23T00:25:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:35.966830+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:35.967660+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:35.968252+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:35.968751+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:35.969218+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:35.972531+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:35.973910+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:35.974427+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:35.978177+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:35.425994+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:35.966830+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:35.967660+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:35.968252+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:35.968751+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:35.969218+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:35.972531+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:35.973910+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:35.974427+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:35.978177+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:35.425994+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:35.966830+0000 mon.a (mon.0) 363 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:35.967660+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:35.968252+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:35.968751+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:35.969218+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:35.972531+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:35.973910+0000 mon.a (mon.0) 369 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:35.974427+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:35.978177+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:37.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.459+0000 7fb94f1c4700 1 -- 172.21.15.171:0/2036728061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 msgr2=0x7fb9480f9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:37.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.459+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/2036728061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 0x7fb9480f9180 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fb9340048a0 tx=0x7fb934009cb0).stop 2024-02-23T00:25:37.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.459+0000 7fb94f1c4700 1 -- 172.21.15.171:0/2036728061 shutdown_connections 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.459+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/2036728061 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9480f9670 0x7fb9480fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.459+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/2036728061 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9480f7a80 0x7fb9480f7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.459+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/2036728061 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 0x7fb9480f9180 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.459+0000 7fb94f1c4700 1 -- 172.21.15.171:0/2036728061 >> 172.21.15.171:0/2036728061 conn(0x7fb9480f32a0 msgr2=0x7fb9480f56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.459+0000 7fb94f1c4700 1 -- 172.21.15.171:0/2036728061 shutdown_connections 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.459+0000 7fb94f1c4700 1 -- 172.21.15.171:0/2036728061 wait complete. 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 librados: starting msgr at 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 librados: starting objecter 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 Processor -- start 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 -- start start 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 librados: setting wanted keys 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 librados: calling monclient init 2024-02-23T00:25:37.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9480f7a80 0x7fb94806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 0x7fb94806b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9480f9670 0x7fb94806b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94806bb40 con 0x7fb9480f7a80 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94806bcb0 con 0x7fb9480f8d20 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94f1c4700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94806be70 con 0x7fb9480f9670 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb947fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 0x7fb94806b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb947fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 0x7fb94806b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:41390/0 (socket says 172.21.15.171:41390) 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb947fff700 1 -- 172.21.15.171:0/1866013527 learned_addr learned my addr 172.21.15.171:0/1866013527 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94cf60700 1 --2- 172.21.15.171:0/1866013527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9480f7a80 0x7fb94806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb94d761700 1 --2- 172.21.15.171:0/1866013527 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9480f9670 0x7fb94806b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb947fff700 1 -- 172.21.15.171:0/1866013527 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9480f9670 msgr2=0x7fb94806b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:37.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb947fff700 1 --2- 172.21.15.171:0/1866013527 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9480f9670 0x7fb94806b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb947fff700 1 -- 172.21.15.171:0/1866013527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9480f7a80 msgr2=0x7fb94806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb947fff700 1 --2- 172.21.15.171:0/1866013527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9480f7a80 0x7fb94806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb947fff700 1 -- 172.21.15.171:0/1866013527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb934005040 con 0x7fb9480f8d20 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb947fff700 1 --2- 172.21.15.171:0/1866013527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 0x7fb94806b030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb934005010 tx=0x7fb934009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.460+0000 7fb945ffb700 10 client.?.objecter ms_handle_connect 0x7fb9480f8d20 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.461+0000 7fb945ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.461+0000 7fb945ffb700 1 -- 172.21.15.171:0/1866013527 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93400e030 con 0x7fb9480f8d20 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.461+0000 7fb945ffb700 1 -- 172.21.15.171:0/1866013527 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb934004100 con 0x7fb9480f8d20 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.461+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb948112110 con 0x7fb9480f8d20 2024-02-23T00:25:37.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.461+0000 7fb945ffb700 1 -- 172.21.15.171:0/1866013527 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934018070 con 0x7fb9480f8d20 2024-02-23T00:25:37.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.461+0000 7fb94f1c4700 10 client.24193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:25:37.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.461+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9481125a0 con 0x7fb9480f8d20 2024-02-23T00:25:37.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.461+0000 7fb94f1c4700 1 librados: init done 2024-02-23T00:25:37.468 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.462+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94805f360 con 0x7fb9480f8d20 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.462+0000 7fb945ffb700 1 -- 172.21.15.171:0/1866013527 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7fb93401d420 con 0x7fb9480f8d20 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.463+0000 7fb945ffb700 1 --2- 172.21.15.171:0/1866013527 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb938062ba0 0x7fb938065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.463+0000 7fb945ffb700 1 -- 172.21.15.171:0/1866013527 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 2776+0+0 (secure 0 0 0) 0x7fb93407d480 con 0x7fb9480f8d20 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.463+0000 7fb945ffb700 10 client.24193.objecter ms_dispatch 0x7fb948001550 osd_map(27..27 src has 1..27) v4 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.463+0000 7fb945ffb700 3 client.24193.objecter handle_osd_map got epochs [27,27] > 0 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.463+0000 7fb945ffb700 3 client.24193.objecter handle_osd_map decoding full epoch 27 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.463+0000 7fb945ffb700 20 client.24193.objecter dump_active .. 0 homeless 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.463+0000 7fb94cf60700 1 --2- 172.21.15.171:0/1866013527 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb938062ba0 0x7fb938065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.463+0000 7fb945ffb700 10 client.24193.objecter ms_handle_connect 0x7fb938062ba0 2024-02-23T00:25:37.469 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.463+0000 7fb94cf60700 1 --2- 172.21.15.171:0/1866013527 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb938062ba0 0x7fb938065060 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb9480f8b80 tx=0x7fb93c00b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:25:37.470 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.467+0000 7fb945ffb700 1 -- 172.21.15.171:0/1866013527 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb93404e400 con 0x7fb9480f8d20 2024-02-23T00:25:37.639 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:37.637+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9480fee70 con 0x7fb938062ba0 2024-02-23T00:25:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:25:35.962379+0000 mgr.y (mgr.14146) 131 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:25:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:25:35.969609+0000 mgr.y (mgr.14146) 132 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 4024M 2024-02-23T00:25:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:36.086998+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:37.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:25:35.962379+0000 mgr.y (mgr.14146) 131 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:25:37.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:25:35.969609+0000 mgr.y (mgr.14146) 132 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 4024M 2024-02-23T00:25:37.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:36.086998+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:37.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:25:35.962379+0000 mgr.y (mgr.14146) 131 : cephadm [INF] Detected new or changed devices on smithi087 2024-02-23T00:25:37.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:25:35.969609+0000 mgr.y (mgr.14146) 132 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 4024M 2024-02-23T00:25:37.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:36.086998+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:38.720 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:37.640263+0000 mgr.y (mgr.14146) 134 : audit [DBG] from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:25:38.720 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:37.642675+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:25:38.720 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:37.645002+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:25:38.720 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:37.645713+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:38.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:37.640263+0000 mgr.y (mgr.14146) 134 : audit [DBG] from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:25:38.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:37.642675+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:25:38.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:37.645002+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:25:38.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:37.645713+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:37.640263+0000 mgr.y (mgr.14146) 134 : audit [DBG] from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:25:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:37.642675+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:25:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:37.645002+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:25:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:37.645713+0000 mon.a (mon.0) 374 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:39.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:38.087470+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:39.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:38.087470+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:39.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:38.087470+0000 mgr.y (mgr.14146) 135 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:39.977728+0000 mon.b (mon.1) 7 : audit [INF] from='client.? 172.21.15.171:0/2080868555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c"}]: dispatch 2024-02-23T00:25:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:39.978319+0000 mon.a (mon.0) 375 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c"}]: dispatch 2024-02-23T00:25:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:39.983638+0000 mon.a (mon.0) 376 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c"}]': finished 2024-02-23T00:25:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:39.983716+0000 mon.a (mon.0) 377 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-23T00:25:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:39.983866+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:39.977728+0000 mon.b (mon.1) 7 : audit [INF] from='client.? 172.21.15.171:0/2080868555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c"}]: dispatch 2024-02-23T00:25:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:39.978319+0000 mon.a (mon.0) 375 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c"}]: dispatch 2024-02-23T00:25:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:39.983638+0000 mon.a (mon.0) 376 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c"}]': finished 2024-02-23T00:25:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:39.983716+0000 mon.a (mon.0) 377 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-23T00:25:40.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:39.983866+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:39.977728+0000 mon.b (mon.1) 7 : audit [INF] from='client.? 172.21.15.171:0/2080868555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c"}]: dispatch 2024-02-23T00:25:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:39.978319+0000 mon.a (mon.0) 375 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c"}]: dispatch 2024-02-23T00:25:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:39.983638+0000 mon.a (mon.0) 376 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c"}]': finished 2024-02-23T00:25:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:39.983716+0000 mon.a (mon.0) 377 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-23T00:25:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:39.983866+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:41.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:40.087833+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:41.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:40.562287+0000 mon.b (mon.1) 8 : audit [DBG] from='client.? 172.21.15.171:0/2997978960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:25:41.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:40.087833+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:41.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:40.562287+0000 mon.b (mon.1) 8 : audit [DBG] from='client.? 172.21.15.171:0/2997978960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:25:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:40.087833+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:40.562287+0000 mon.b (mon.1) 8 : audit [DBG] from='client.? 172.21.15.171:0/2997978960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:25:42.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:42.462+0000 7fb94e9c3700 10 client.24193.objecter tick 2024-02-23T00:25:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:42.088200+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:43.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:42.088200+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:42.088200+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:45.793 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:44.088660+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:45.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:44.088660+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:45.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:44.088660+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:46.777 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:46.077094+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-23T00:25:46.777 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:46.077863+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:46.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:46.077094+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-23T00:25:46.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:46.077863+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:46.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:46.077094+0000 mon.a (mon.0) 379 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-23T00:25:46.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:46.077863+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:47.464 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:47.463+0000 7fb94e9c3700 10 client.24193.objecter tick 2024-02-23T00:25:47.704 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:25:46.078484+0000 mgr.y (mgr.14146) 139 : cephadm [INF] Deploying daemon osd.4 on smithi171 2024-02-23T00:25:47.705 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:46.089159+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:47.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:25:46.078484+0000 mgr.y (mgr.14146) 139 : cephadm [INF] Deploying daemon osd.4 on smithi171 2024-02-23T00:25:47.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:46.089159+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:25:46.078484+0000 mgr.y (mgr.14146) 139 : cephadm [INF] Deploying daemon osd.4 on smithi171 2024-02-23T00:25:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:46.089159+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:48.089561+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:48.089561+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:48.089561+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:51.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:50.089911+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:51.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:50.979453+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:51.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:50.980177+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:51.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:50.089911+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:50.979453+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:50.980177+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:51.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:50.089911+0000 mgr.y (mgr.14146) 142 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:51.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:50.979453+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:51.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:50.980177+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:52.465 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:52.464+0000 7fb94e9c3700 10 client.24193.objecter tick 2024-02-23T00:25:53.725 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:52.090391+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:53.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:52.090391+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:53.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:52.090391+0000 mgr.y (mgr.14146) 143 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:54.287 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 4 on host 'smithi171' 2024-02-23T00:25:54.287 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.285+0000 7fb945ffb700 1 -- 172.21.15.171:0/1866013527 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb9480fee70 con 0x7fb938062ba0 2024-02-23T00:25:54.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.287+0000 7fb94f1c4700 10 librados: watch_flush enter 2024-02-23T00:25:54.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.287+0000 7fb94f1c4700 10 librados: watch_flush exit 2024-02-23T00:25:54.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.287+0000 7fb94f1c4700 20 client.24193.objecter shutdown clearing up homeless session... 2024-02-23T00:25:54.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.287+0000 7fb94f1c4700 10 client.24193.objecter successfully canceled tick 2024-02-23T00:25:54.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.287+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb938062ba0 msgr2=0x7fb938065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:54.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.287+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/1866013527 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb938062ba0 0x7fb938065060 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fb9480f8b80 tx=0x7fb93c00b040).stop 2024-02-23T00:25:54.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 msgr2=0x7fb94806b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:25:54.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/1866013527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 0x7fb94806b030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb934005010 tx=0x7fb934009cb0).stop 2024-02-23T00:25:54.289 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 shutdown_connections 2024-02-23T00:25:54.290 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/1866013527 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9480f9670 0x7fb94806b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:54.290 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/1866013527 >> v2:172.21.15.87:6800/1121018082 conn(0x7fb938062ba0 0x7fb938065060 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:54.290 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/1866013527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9480f7a80 0x7fb94806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:54.290 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 --2- 172.21.15.171:0/1866013527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9480f8d20 0x7fb94806b030 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:25:54.290 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 >> 172.21.15.171:0/1866013527 conn(0x7fb9480f32a0 msgr2=0x7fb9480f66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:25:54.290 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 shutdown_connections 2024-02-23T00:25:54.290 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 -- 172.21.15.171:0/1866013527 wait complete. 2024-02-23T00:25:54.290 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:25:54.288+0000 7fb94f1c4700 1 librados: shutdown 2024-02-23T00:25:54.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:53.575432+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:54.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:53.971631+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:54.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:53.972525+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:54.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:53.977421+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:54.070679+0000 mon.b (mon.1) 9 : audit [INF] from='osd.4 v2:172.21.15.171:6800/4175844757' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:54.071058+0000 mon.a (mon.0) 387 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:54.285115+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:54.294685+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:53.575432+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:53.971631+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:53.972525+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:53.977421+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:54.070679+0000 mon.b (mon.1) 9 : audit [INF] from='osd.4 v2:172.21.15.171:6800/4175844757' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:54.071058+0000 mon.a (mon.0) 387 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:54.285115+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:54.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:54.294685+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:54.899 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:53.575432+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:54.899 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:53.971631+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:54.899 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:53.972525+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:54.899 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:53.977421+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:54.899 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:54.070679+0000 mon.b (mon.1) 9 : audit [INF] from='osd.4 v2:172.21.15.171:6800/4175844757' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-23T00:25:54.899 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:54.071058+0000 mon.a (mon.0) 387 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-23T00:25:54.899 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:54.285115+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:54.899 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:54.294685+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:25:54.921 DEBUG:teuthology.orchestra.run.smithi171:osd.4> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.4.service 2024-02-23T00:25:54.923 INFO:tasks.cephadm:Deploying osd.5 on smithi171 with /dev/vg_nvme/lv_3... 2024-02-23T00:25:54.923 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-23T00:25:54.953 INFO:journalctl@ceph.osd.4.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:25:55.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:54.090722+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:54.987821+0000 mon.a (mon.0) 390 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:54.987881+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:54.988042+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:54.988810+0000 mon.b (mon.1) 10 : audit [INF] from='osd.4 v2:172.21.15.171:6800/4175844757' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:54.989157+0000 mon.a (mon.0) 393 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:54.090722+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:54.987821+0000 mon.a (mon.0) 390 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:54.987881+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:54.988042+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:54.988810+0000 mon.b (mon.1) 10 : audit [INF] from='osd.4 v2:172.21.15.171:6800/4175844757' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:25:55.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:54.989157+0000 mon.a (mon.0) 393 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:25:55.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:54.090722+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:55.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:54.987821+0000 mon.a (mon.0) 390 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-23T00:25:55.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:54.987881+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-23T00:25:55.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:54.988042+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:55.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:54.988810+0000 mon.b (mon.1) 10 : audit [INF] from='osd.4 v2:172.21.15.171:6800/4175844757' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:25:55.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:54.989157+0000 mon.a (mon.0) 393 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:25:56.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:25:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:25:55.995+0000 7f0bcefa8700 -1 osd.4 0 waiting for initial osdmap 2024-02-23T00:25:56.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:25:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:25:55.999+0000 7f0bc9f21700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-23T00:25:57.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:55.993929+0000 mon.a (mon.0) 394 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:25:57.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:55.994038+0000 mon.a (mon.0) 395 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-23T00:25:57.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:55.994242+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:57.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:55.996867+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:57.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:56.091137+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:57.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:55.993929+0000 mon.a (mon.0) 394 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:25:57.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:55.994038+0000 mon.a (mon.0) 395 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-23T00:25:57.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:55.994242+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:57.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:55.996867+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:57.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:56.091137+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:57.426 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:55.993929+0000 mon.a (mon.0) 394 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:25:57.426 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:55.994038+0000 mon.a (mon.0) 395 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-23T00:25:57.427 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:55.994242+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:57.427 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:55.996867+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:57.427 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:56.091137+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v127: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-23T00:25:58.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:55.027522+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:58.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:55.027593+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:58.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:56.997625+0000 mon.a (mon.0) 398 : cluster [INF] osd.4 v2:172.21.15.171:6800/4175844757 boot 2024-02-23T00:25:58.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:56.997709+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-23T00:25:58.279 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:56.998000+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:58.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:55.027522+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:58.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:55.027593+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:56.997625+0000 mon.a (mon.0) 398 : cluster [INF] osd.4 v2:172.21.15.171:6800/4175844757 boot 2024-02-23T00:25:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:56.997709+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-23T00:25:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:56.998000+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:58.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:55.027522+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:25:58.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:55.027593+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:25:58.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:56.997625+0000 mon.a (mon.0) 398 : cluster [INF] osd.4 v2:172.21.15.171:6800/4175844757 boot 2024-02-23T00:25:58.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:56.997709+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-23T00:25:58.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:56.998000+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:25:58.854 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-23T00:25:58.855 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-23T00:25:58.855 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-02-23T00:25:58.855 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-02-23T00:25:58.855 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127229 s, 824 MB/s 2024-02-23T00:25:58.855 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-02-23T00:25:59.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:57.999307+0000 mon.a (mon.0) 401 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-23T00:25:59.277 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:58.091516+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:25:59.277 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:25:58.321289+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:25:59.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:58.326330+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:59.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:58.327198+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:59.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:25:58.327799+0000 mgr.y (mgr.14146) 148 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 17122M 2024-02-23T00:25:59.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:58.331398+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:59.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:58.332331+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:59.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:58.333049+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:59.278 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:25:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:25:58.336983+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:57.999307+0000 mon.a (mon.0) 401 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:58.091516+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:25:58.321289+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:58.326330+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:58.327198+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:25:58.327799+0000 mgr.y (mgr.14146) 148 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 17122M 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:58.331398+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:58.332331+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:58.333049+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:59.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:25:58.336983+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:57.999307+0000 mon.a (mon.0) 401 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:58.091516+0000 mgr.y (mgr.14146) 146 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:25:58.321289+0000 mgr.y (mgr.14146) 147 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:58.326330+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:58.327198+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:25:58.327799+0000 mgr.y (mgr.14146) 148 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 17122M 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:58.331398+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:58.332331+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:58.333049+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:25:59.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:25:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:25:58.336983+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:25:59.514 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch daemon add osd smithi171:vg_nvme/lv_3 2024-02-23T00:26:00.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:25:59.001254+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-23T00:26:00.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:25:59.001254+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-23T00:26:00.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:25:59.001254+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-23T00:26:01.083 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:00.091908+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:01.301 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 -- 172.21.15.171:0/552746962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f6990 msgr2=0x7f11800f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 --2- 172.21.15.171:0/552746962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f6990 0x7f11800f6db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f11740048a0 tx=0x7f1174009cb0).stop 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 -- 172.21.15.171:0/552746962 shutdown_connections 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 --2- 172.21.15.171:0/552746962 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11800f8580 0x7f11800fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 --2- 172.21.15.171:0/552746962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f6990 0x7f11800f6db0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 --2- 172.21.15.171:0/552746962 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11800f7c30 0x7f11800f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 -- 172.21.15.171:0/552746962 >> 172.21.15.171:0/552746962 conn(0x7f11800f2190 msgr2=0x7f11800f45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 -- 172.21.15.171:0/552746962 shutdown_connections 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 -- 172.21.15.171:0/552746962 wait complete. 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 librados: starting msgr at 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.300+0000 7f1186220700 1 librados: starting objecter 2024-02-23T00:26:01.302 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 Processor -- start 2024-02-23T00:26:01.305 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 -- start start 2024-02-23T00:26:01.305 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 librados: setting wanted keys 2024-02-23T00:26:01.305 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 librados: calling monclient init 2024-02-23T00:26:01.305 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11800f6990 0x7f1180061490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:01.305 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11800f7c30 0x7f118010cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:01.305 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f8580 0x7f118010f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:01.305 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118010fbb0 con 0x7f11800f7c30 2024-02-23T00:26:01.305 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118010fd20 con 0x7f11800f8580 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f1186220700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118010fee0 con 0x7f11800f6990 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117ffff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f8580 0x7f118010f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117ffff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f8580 0x7f118010f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:43718/0 (socket says 172.21.15.171:43718) 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117ffff700 1 -- 172.21.15.171:0/295932142 learned_addr learned my addr 172.21.15.171:0/295932142 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117f7fe700 1 --2- 172.21.15.171:0/295932142 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11800f6990 0x7f1180061490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117effd700 1 --2- 172.21.15.171:0/295932142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11800f7c30 0x7f118010cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117ffff700 1 -- 172.21.15.171:0/295932142 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11800f6990 msgr2=0x7f1180061490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117ffff700 1 --2- 172.21.15.171:0/295932142 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11800f6990 0x7f1180061490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117ffff700 1 -- 172.21.15.171:0/295932142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11800f7c30 msgr2=0x7f118010cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117ffff700 1 --2- 172.21.15.171:0/295932142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11800f7c30 0x7f118010cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.301+0000 7f117ffff700 1 -- 172.21.15.171:0/295932142 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1174005040 con 0x7f11800f8580 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f117ffff700 1 --2- 172.21.15.171:0/295932142 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f8580 0x7f118010f460 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f117800bc30 tx=0x7f1178009840).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:01.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f117cff9700 10 client.?.objecter ms_handle_connect 0x7f11800f8580 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f117cff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f117cff9700 1 -- 172.21.15.171:0/295932142 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117800c630 con 0x7f11800f8580 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f117cff9700 1 -- 172.21.15.171:0/295932142 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1178010070 con 0x7f11800f8580 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11801101c0 con 0x7f11800f8580 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f117cff9700 1 -- 172.21.15.171:0/295932142 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1178007810 con 0x7f11800f8580 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f1186220700 10 client.24220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1180110700 con 0x7f11800f8580 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.302+0000 7f1186220700 1 librados: init done 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.303+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1180101870 con 0x7f11800f8580 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.303+0000 7f117cff9700 1 -- 172.21.15.171:0/295932142 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f1178007bc0 con 0x7f11800f8580 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.303+0000 7f117cff9700 1 --2- 172.21.15.171:0/295932142 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1168062ba0 0x7f1168065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.303+0000 7f117cff9700 1 -- 172.21.15.171:0/295932142 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 3178+0+0 (secure 0 0 0) 0x7f117807b5d0 con 0x7f11800f8580 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.303+0000 7f117cff9700 10 client.24220.objecter ms_dispatch 0x7f1180001550 osd_map(33..33 src has 1..33) v4 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.303+0000 7f117cff9700 3 client.24220.objecter handle_osd_map got epochs [33,33] > 0 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.303+0000 7f117cff9700 3 client.24220.objecter handle_osd_map decoding full epoch 33 2024-02-23T00:26:01.307 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.304+0000 7f117cff9700 20 client.24220.objecter dump_active .. 0 homeless 2024-02-23T00:26:01.308 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.304+0000 7f117f7fe700 1 --2- 172.21.15.171:0/295932142 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1168062ba0 0x7f1168065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:01.308 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.304+0000 7f117f7fe700 1 --2- 172.21.15.171:0/295932142 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1168062ba0 0x7f1168065060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1174009da0 tx=0x7f1174009cb0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:01.308 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.304+0000 7f117cff9700 10 client.24220.objecter ms_handle_connect 0x7f1168062ba0 2024-02-23T00:26:01.309 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.308+0000 7f117cff9700 1 -- 172.21.15.171:0/295932142 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f117804c460 con 0x7f11800f8580 2024-02-23T00:26:01.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:00.091908+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:00.091908+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:01.483 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:01.482+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f11800fdd70 con 0x7f1168062ba0 2024-02-23T00:26:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:01.484077+0000 mgr.y (mgr.14146) 150 : audit [DBG] from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:26:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:01.486467+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:26:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:01.488946+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:26:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:01.489635+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:02.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:01.484077+0000 mgr.y (mgr.14146) 150 : audit [DBG] from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:26:02.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:01.486467+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:26:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:01.488946+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:26:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:01.489635+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:02.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:01.484077+0000 mgr.y (mgr.14146) 150 : audit [DBG] from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:26:02.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:01.486467+0000 mon.a (mon.0) 409 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:26:02.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:01.488946+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:26:02.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:01.489635+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:03.100 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:02.092268+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:03.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:02.092268+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:03.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:02.092268+0000 mgr.y (mgr.14146) 151 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:04.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:03.874819+0000 mon.b (mon.1) 11 : audit [INF] from='client.? 172.21.15.171:0/2507797760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d"}]: dispatch 2024-02-23T00:26:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:03.875423+0000 mon.a (mon.0) 412 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d"}]: dispatch 2024-02-23T00:26:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:03.879872+0000 mon.a (mon.0) 413 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d"}]': finished 2024-02-23T00:26:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:03.879928+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-23T00:26:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:03.880070+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:03.874819+0000 mon.b (mon.1) 11 : audit [INF] from='client.? 172.21.15.171:0/2507797760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d"}]: dispatch 2024-02-23T00:26:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:03.875423+0000 mon.a (mon.0) 412 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d"}]: dispatch 2024-02-23T00:26:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:03.879872+0000 mon.a (mon.0) 413 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d"}]': finished 2024-02-23T00:26:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:03.879928+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-23T00:26:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:03.880070+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:03.874819+0000 mon.b (mon.1) 11 : audit [INF] from='client.? 172.21.15.171:0/2507797760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d"}]: dispatch 2024-02-23T00:26:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:03.875423+0000 mon.a (mon.0) 412 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d"}]: dispatch 2024-02-23T00:26:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:03.879872+0000 mon.a (mon.0) 413 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d"}]': finished 2024-02-23T00:26:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:03.879928+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-23T00:26:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:03.880070+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:05.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:04.092720+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:05.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:04.462748+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? 172.21.15.171:0/1380235358' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:26:05.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:04.092720+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:05.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:04.462748+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? 172.21.15.171:0/1380235358' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:26:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:04.092720+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:04.462748+0000 mon.b (mon.1) 12 : audit [DBG] from='client.? 172.21.15.171:0/1380235358' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:26:06.304 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:06.303+0000 7f1185a1f700 10 client.24220.objecter tick 2024-02-23T00:26:07.335 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:06.093204+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:06.093204+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:07.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:06.093204+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:07.333984+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:07.334063+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:08.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:07.333984+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:08.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:07.334063+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:07.333984+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:07.334063+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:09.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:08.093608+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:09.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:08.154335+0000 mon.a (mon.0) 416 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-23T00:26:09.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:08.154512+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:09.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:08.093608+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:09.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:08.154335+0000 mon.a (mon.0) 416 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-23T00:26:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:08.154512+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:08.093608+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:08.154335+0000 mon.a (mon.0) 416 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-23T00:26:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:08.154512+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:11.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:10.091070+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-23T00:26:11.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:10.091886+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:11.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:26:10.092544+0000 mgr.y (mgr.14146) 155 : cephadm [INF] Deploying daemon osd.5 on smithi171 2024-02-23T00:26:11.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:10.093939+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:11.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:10.091070+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-23T00:26:11.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:10.091886+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:11.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:26:10.092544+0000 mgr.y (mgr.14146) 155 : cephadm [INF] Deploying daemon osd.5 on smithi171 2024-02-23T00:26:11.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:10.093939+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:11.189 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:10.091070+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-23T00:26:11.189 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:10.091886+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:11.189 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:26:10.092544+0000 mgr.y (mgr.14146) 155 : cephadm [INF] Deploying daemon osd.5 on smithi171 2024-02-23T00:26:11.189 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:10.093939+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:11.305 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:11.304+0000 7f1185a1f700 10 client.24220.objecter tick 2024-02-23T00:26:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:12.094292+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:13.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:12.094292+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:13.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:12.094292+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:15.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:14.094634+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:15.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:14.952650+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:15.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:14.992890+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:14.094634+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:14.952650+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:14.992890+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:14.094634+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:14.952650+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:14.992890+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:16.306 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:16.304+0000 7f1185a1f700 10 client.24220.objecter tick 2024-02-23T00:26:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:16.094954+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:16.094954+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:16.094954+0000 mgr.y (mgr.14146) 159 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:18.066 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.063+0000 7f117cff9700 1 -- 172.21.15.171:0/295932142 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f11800fdd70 con 0x7f1168062ba0 2024-02-23T00:26:18.066 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 5 on host 'smithi171' 2024-02-23T00:26:18.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 10 librados: watch_flush enter 2024-02-23T00:26:18.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 10 librados: watch_flush exit 2024-02-23T00:26:18.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 20 client.24220.objecter shutdown clearing up homeless session... 2024-02-23T00:26:18.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 10 client.24220.objecter successfully canceled tick 2024-02-23T00:26:18.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1168062ba0 msgr2=0x7f1168065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:18.068 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 --2- 172.21.15.171:0/295932142 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1168062ba0 0x7f1168065060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1174009da0 tx=0x7f1174009cb0).stop 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f8580 msgr2=0x7f118010f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 --2- 172.21.15.171:0/295932142 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f8580 0x7f118010f460 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f117800bc30 tx=0x7f1178009840).stop 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 shutdown_connections 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 --2- 172.21.15.171:0/295932142 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11800f8580 0x7f118010f460 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 --2- 172.21.15.171:0/295932142 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11800f6990 0x7f1180061490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 --2- 172.21.15.171:0/295932142 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1168062ba0 0x7f1168065060 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 --2- 172.21.15.171:0/295932142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11800f7c30 0x7f118010cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 >> 172.21.15.171:0/295932142 conn(0x7f11800f2190 msgr2=0x7f11800f4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 shutdown_connections 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 -- 172.21.15.171:0/295932142 wait complete. 2024-02-23T00:26:18.069 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:18.066+0000 7f1186220700 1 librados: shutdown 2024-02-23T00:26:18.434 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:18.038041+0000 mon.b (mon.1) 13 : audit [INF] from='osd.5 v2:172.21.15.171:6804/2550013795' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-23T00:26:18.434 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:18.038450+0000 mon.a (mon.0) 422 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-23T00:26:18.434 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:18.064636+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:18.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:18.038041+0000 mon.b (mon.1) 13 : audit [INF] from='osd.5 v2:172.21.15.171:6804/2550013795' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-23T00:26:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:18.038450+0000 mon.a (mon.0) 422 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-23T00:26:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:18.064636+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:18.038041+0000 mon.b (mon.1) 13 : audit [INF] from='osd.5 v2:172.21.15.171:6804/2550013795' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-23T00:26:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:18.038450+0000 mon.a (mon.0) 422 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-23T00:26:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:18.064636+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:18.768 DEBUG:teuthology.orchestra.run.smithi171:osd.5> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.5.service 2024-02-23T00:26:18.770 INFO:tasks.cephadm:Deploying osd.6 on smithi171 with /dev/vg_nvme/lv_2... 2024-02-23T00:26:18.770 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-23T00:26:18.801 INFO:journalctl@ceph.osd.5.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:26:19.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:26:19.170+0000 7f5eba45f700 -1 osd.5 0 waiting for initial osdmap 2024-02-23T00:26:19.287 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:26:19.179+0000 7f5eb6bdb700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-23T00:26:19.287 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:17.632603+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:19.287 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:17.632702+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:19.287 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:18.095307+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:19.288 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:18.165935+0000 mon.a (mon.0) 424 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-23T00:26:19.288 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:18.166038+0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-23T00:26:19.288 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:18.166185+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:19.288 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:18.166561+0000 mon.b (mon.1) 14 : audit [INF] from='osd.5 v2:172.21.15.171:6804/2550013795' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:19.288 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:18.166817+0000 mon.a (mon.0) 427 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:19.288 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:18.340588+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:19.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:17.632603+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:17.632702+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:18.095307+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:18.165935+0000 mon.a (mon.0) 424 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:18.166038+0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:18.166185+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:18.166561+0000 mon.b (mon.1) 14 : audit [INF] from='osd.5 v2:172.21.15.171:6804/2550013795' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:18.166817+0000 mon.a (mon.0) 427 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:18.340588+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:17.632603+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:17.632702+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:18.095307+0000 mgr.y (mgr.14146) 160 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:18.165935+0000 mon.a (mon.0) 424 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-23T00:26:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:18.166038+0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-23T00:26:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:18.166185+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:18.166561+0000 mon.b (mon.1) 14 : audit [INF] from='osd.5 v2:172.21.15.171:6804/2550013795' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:18.166817+0000 mon.a (mon.0) 427 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:18.340588+0000 mon.a (mon.0) 428 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:19.168850+0000 mon.a (mon.0) 429 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:26:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:19.168948+0000 mon.a (mon.0) 430 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-23T00:26:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:19.169450+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:19.173876+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:19.830436+0000 mon.a (mon.0) 433 : cluster [INF] osd.5 v2:172.21.15.171:6804/2550013795 boot 2024-02-23T00:26:20.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:19.830495+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-23T00:26:20.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:19.830707+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:19.168850+0000 mon.a (mon.0) 429 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:26:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:19.168948+0000 mon.a (mon.0) 430 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-23T00:26:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:19.169450+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:19.173876+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:19.830436+0000 mon.a (mon.0) 433 : cluster [INF] osd.5 v2:172.21.15.171:6804/2550013795 boot 2024-02-23T00:26:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:19.830495+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-23T00:26:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:19.830707+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:20.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:19.168850+0000 mon.a (mon.0) 429 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:26:20.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:19.168948+0000 mon.a (mon.0) 430 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2024-02-23T00:26:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:19.169450+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:19.173876+0000 mon.a (mon.0) 432 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:19.830436+0000 mon.a (mon.0) 433 : cluster [INF] osd.5 v2:172.21.15.171:6804/2550013795 boot 2024-02-23T00:26:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:19.830495+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-23T00:26:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:19.830707+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:26:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:19.048169+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:19.048235+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:20.095762+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v147: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:20.839357+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-23T00:26:21.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:19.048169+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:21.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:19.048235+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:21.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:20.095762+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v147: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:21.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:20.839357+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-23T00:26:21.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:19.048169+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:21.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:19.048235+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:21.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:20.095762+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v147: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-23T00:26:21.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:20.839357+0000 mon.a (mon.0) 436 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-23T00:26:22.154 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-23T00:26:22.154 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-23T00:26:22.154 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-02-23T00:26:22.154 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-02-23T00:26:22.154 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144665 s, 725 MB/s 2024-02-23T00:26:22.154 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-02-23T00:26:23.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:21.835355+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-23T00:26:23.791 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:22.096224+0000 mgr.y (mgr.14146) 162 : cluster [DBG] pgmap v150: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:23.791 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:21.835355+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-23T00:26:23.792 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:22.096224+0000 mgr.y (mgr.14146) 162 : cluster [DBG] pgmap v150: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:23.792 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:21.835355+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2024-02-23T00:26:23.792 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:22.096224+0000 mgr.y (mgr.14146) 162 : cluster [DBG] pgmap v150: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:23.793 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch daemon add osd smithi171:vg_nvme/lv_2 2024-02-23T00:26:24.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:26:22.836166+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:26:24.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.843267+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.844252+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:24.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.844975+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:24.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:26:22.845462+0000 mgr.y (mgr.14146) 164 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 8561M 2024-02-23T00:26:24.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.849652+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.850537+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:24.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.851215+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:24.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.856006+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.858185+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:24.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.859587+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:24.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.860234+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:24.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:22.864828+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:26:22.836166+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.843267+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.844252+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.844975+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:26:22.845462+0000 mgr.y (mgr.14146) 164 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 8561M 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.849652+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.850537+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.851215+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.856006+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.858185+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.859587+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.860234+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:22.864828+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:26:22.836166+0000 mgr.y (mgr.14146) 163 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.843267+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.844252+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.844975+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:26:22.845462+0000 mgr.y (mgr.14146) 164 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 8561M 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.849652+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.850537+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:24.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.851215+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:24.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.856006+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:24.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.858185+0000 mon.a (mon.0) 445 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:24.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.859587+0000 mon.a (mon.0) 446 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:24.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.860234+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:24.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:22.864828+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:25.116 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:24.096663+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:25.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:24.096663+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:24.096663+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 peering; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:25.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.621+0000 7f2625a1a700 1 -- 172.21.15.171:0/3709310525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f7270 msgr2=0x7f26200f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.621+0000 7f2625a1a700 1 --2- 172.21.15.171:0/3709310525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f7270 0x7f26200f7ca0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2610004870 tx=0x7f2610009cb0).stop 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.621+0000 7f2625a1a700 1 -- 172.21.15.171:0/3709310525 shutdown_connections 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.621+0000 7f2625a1a700 1 --2- 172.21.15.171:0/3709310525 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26200f8380 0x7f26200feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.621+0000 7f2625a1a700 1 --2- 172.21.15.171:0/3709310525 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26200f6960 0x7f26200f6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.621+0000 7f2625a1a700 1 --2- 172.21.15.171:0/3709310525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f7270 0x7f26200f7ca0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.621+0000 7f2625a1a700 1 -- 172.21.15.171:0/3709310525 >> 172.21.15.171:0/3709310525 conn(0x7f26200f2160 msgr2=0x7f26200f4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.621+0000 7f2625a1a700 1 -- 172.21.15.171:0/3709310525 shutdown_connections 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.621+0000 7f2625a1a700 1 -- 172.21.15.171:0/3709310525 wait complete. 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 librados: starting msgr at 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 librados: starting objecter 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 Processor -- start 2024-02-23T00:26:25.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 -- start start 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 librados: setting wanted keys 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 librados: calling monclient init 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26200f6960 0x7f2620108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26200f7270 0x7f2620108640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f8380 0x7f262010f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262010fc10 con 0x7f26200f7270 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262010fd50 con 0x7f26200f8380 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f2625a1a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262010fe90 con 0x7f26200f6960 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f261f7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f8380 0x7f262010f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f261f7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f8380 0x7f262010f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:56214/0 (socket says 172.21.15.171:56214) 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f261f7fe700 1 -- 172.21.15.171:0/2881131858 learned_addr learned my addr 172.21.15.171:0/2881131858 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f261e7fc700 1 --2- 172.21.15.171:0/2881131858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26200f7270 0x7f2620108640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.622+0000 7f261effd700 1 --2- 172.21.15.171:0/2881131858 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26200f6960 0x7f2620108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:25.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f261f7fe700 1 -- 172.21.15.171:0/2881131858 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26200f6960 msgr2=0x7f2620108150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f261f7fe700 1 --2- 172.21.15.171:0/2881131858 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26200f6960 0x7f2620108150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f261f7fe700 1 -- 172.21.15.171:0/2881131858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26200f7270 msgr2=0x7f2620108640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f261f7fe700 1 --2- 172.21.15.171:0/2881131858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26200f7270 0x7f2620108640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f261f7fe700 1 -- 172.21.15.171:0/2881131858 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2610005040 con 0x7f26200f8380 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f261f7fe700 1 --2- 172.21.15.171:0/2881131858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f8380 0x7f262010f720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f261800c870 tx=0x7f2618007dd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f2607fff700 10 client.?.objecter ms_handle_connect 0x7f26200f8380 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f2607fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f2607fff700 1 -- 172.21.15.171:0/2881131858 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2618010510 con 0x7f26200f8380 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f2607fff700 1 -- 172.21.15.171:0/2881131858 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2618010b40 con 0x7f26200f8380 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2620110170 con 0x7f26200f8380 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f2607fff700 1 -- 172.21.15.171:0/2881131858 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2618012ae0 con 0x7f26200f8380 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f2625a1a700 10 client.24247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:26:25.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2620110660 con 0x7f26200f8380 2024-02-23T00:26:25.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.623+0000 7f2625a1a700 1 librados: init done 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.624+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2620101810 con 0x7f26200f8380 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.624+0000 7f2607fff700 1 -- 172.21.15.171:0/2881131858 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f2618010670 con 0x7f26200f8380 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.625+0000 7f2607fff700 1 --2- 172.21.15.171:0/2881131858 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2608062ba0 0x7f2608065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.625+0000 7f2607fff700 1 -- 172.21.15.171:0/2881131858 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 3470+0+0 (secure 0 0 0) 0x7f261807bf40 con 0x7f26200f8380 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.625+0000 7f2607fff700 10 client.24247.objecter ms_dispatch 0x7f2620001550 osd_map(40..40 src has 1..40) v4 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.625+0000 7f2607fff700 3 client.24247.objecter handle_osd_map got epochs [40,40] > 0 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.625+0000 7f2607fff700 3 client.24247.objecter handle_osd_map decoding full epoch 40 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.625+0000 7f2607fff700 20 client.24247.objecter dump_active .. 0 homeless 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.625+0000 7f261effd700 1 --2- 172.21.15.171:0/2881131858 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2608062ba0 0x7f2608065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.625+0000 7f2607fff700 10 client.24247.objecter ms_handle_connect 0x7f2608062ba0 2024-02-23T00:26:25.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.625+0000 7f261effd700 1 --2- 172.21.15.171:0/2881131858 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2608062ba0 0x7f2608065060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f26140041d0 tx=0x7f2614009040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:25.631 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.629+0000 7f2607fff700 1 -- 172.21.15.171:0/2881131858 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f261804cc80 con 0x7f26200f8380 2024-02-23T00:26:25.802 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:25.800+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f26200fe0f0 con 0x7f2608062ba0 2024-02-23T00:26:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:25.805240+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:26:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:25.807558+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:26:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:25.808243+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:26.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:25.805240+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:26:26.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:25.807558+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:26:26.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:25.808243+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:25.805240+0000 mon.a (mon.0) 449 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:26:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:25.807558+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:26:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:25.808243+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:25.802830+0000 mgr.y (mgr.14146) 166 : audit [DBG] from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:26:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:26.097019+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:27.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:25.802830+0000 mgr.y (mgr.14146) 166 : audit [DBG] from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:26:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:26.097019+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:25.802830+0000 mgr.y (mgr.14146) 166 : audit [DBG] from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:26:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:26.097019+0000 mgr.y (mgr.14146) 167 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:28.097480+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:28.214966+0000 mon.b (mon.1) 15 : audit [INF] from='client.? 172.21.15.171:0/3388284139' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01"}]: dispatch 2024-02-23T00:26:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:28.215682+0000 mon.a (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01"}]: dispatch 2024-02-23T00:26:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:28.220458+0000 mon.a (mon.0) 453 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01"}]': finished 2024-02-23T00:26:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:28.220545+0000 mon.a (mon.0) 454 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-23T00:26:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:28.220702+0000 mon.a (mon.0) 455 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:28.811309+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? 172.21.15.171:0/393774225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:26:29.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:28.097480+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:28.214966+0000 mon.b (mon.1) 15 : audit [INF] from='client.? 172.21.15.171:0/3388284139' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01"}]: dispatch 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:28.215682+0000 mon.a (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01"}]: dispatch 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:28.220458+0000 mon.a (mon.0) 453 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01"}]': finished 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:28.220545+0000 mon.a (mon.0) 454 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:28.220702+0000 mon.a (mon.0) 455 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:28.811309+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? 172.21.15.171:0/393774225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:28.097480+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:28.214966+0000 mon.b (mon.1) 15 : audit [INF] from='client.? 172.21.15.171:0/3388284139' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01"}]: dispatch 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:28.215682+0000 mon.a (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01"}]: dispatch 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:28.220458+0000 mon.a (mon.0) 453 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01"}]': finished 2024-02-23T00:26:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:28.220545+0000 mon.a (mon.0) 454 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-23T00:26:29.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:28.220702+0000 mon.a (mon.0) 455 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:29.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:28.811309+0000 mon.b (mon.1) 16 : audit [DBG] from='client.? 172.21.15.171:0/393774225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:26:30.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:30.624+0000 7f2625219700 10 client.24247.objecter tick 2024-02-23T00:26:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:30.097859+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:30.097859+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:30.097859+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:33.329 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:32.098328+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:32.098328+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:33.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:32.098328+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:33.053294+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:26:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:33.054252+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:26:34.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:33.053294+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:26:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:33.054252+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:26:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:33.053294+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:26:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:33.054252+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:26:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:34.098751+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:34.333944+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-23T00:26:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:34.334755+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:26:34.335397+0000 mgr.y (mgr.14146) 172 : cephadm [INF] Deploying daemon osd.6 on smithi171 2024-02-23T00:26:35.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:34.098751+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:34.333944+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-23T00:26:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:34.334755+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:26:34.335397+0000 mgr.y (mgr.14146) 172 : cephadm [INF] Deploying daemon osd.6 on smithi171 2024-02-23T00:26:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:34.098751+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:34.333944+0000 mon.a (mon.0) 456 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-23T00:26:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:34.334755+0000 mon.a (mon.0) 457 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:26:34.335397+0000 mgr.y (mgr.14146) 172 : cephadm [INF] Deploying daemon osd.6 on smithi171 2024-02-23T00:26:35.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:35.625+0000 7f2625219700 10 client.24247.objecter tick 2024-02-23T00:26:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:36.099142+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:37.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:36.099142+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:36.099142+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:38.099600+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:39.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:38.099600+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:38.099600+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:40.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:40.626+0000 7f2625219700 10 client.24247.objecter tick 2024-02-23T00:26:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:39.453809+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:39.470166+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:40.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:39.453809+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:39.470166+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:39.453809+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:39.470166+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:41.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:40.099997+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:41.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:40.099997+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:40.099997+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:42.812 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.811+0000 7f2607fff700 1 -- 172.21.15.171:0/2881131858 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f26200fe0f0 con 0x7f2608062ba0 2024-02-23T00:26:42.812 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 6 on host 'smithi171' 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 10 librados: watch_flush enter 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 10 librados: watch_flush exit 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 20 client.24247.objecter shutdown clearing up homeless session... 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 10 client.24247.objecter successfully canceled tick 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2608062ba0 msgr2=0x7f2608065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 --2- 172.21.15.171:0/2881131858 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2608062ba0 0x7f2608065060 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f26140041d0 tx=0x7f2614009040).stop 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f8380 msgr2=0x7f262010f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 --2- 172.21.15.171:0/2881131858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f8380 0x7f262010f720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f261800c870 tx=0x7f2618007dd0).stop 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 shutdown_connections 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 --2- 172.21.15.171:0/2881131858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26200f8380 0x7f262010f720 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:42.815 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 --2- 172.21.15.171:0/2881131858 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26200f6960 0x7f2620108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:42.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 --2- 172.21.15.171:0/2881131858 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2608062ba0 0x7f2608065060 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:42.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 --2- 172.21.15.171:0/2881131858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26200f7270 0x7f2620108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:42.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 >> 172.21.15.171:0/2881131858 conn(0x7f26200f2160 msgr2=0x7f26200617b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:42.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 shutdown_connections 2024-02-23T00:26:42.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 -- 172.21.15.171:0/2881131858 wait complete. 2024-02-23T00:26:42.816 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:42.813+0000 7f2625a1a700 1 librados: shutdown 2024-02-23T00:26:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:42.100457+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:42.107522+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:42.108374+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:42.109151+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:42.113575+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:42.589471+0000 mon.b (mon.1) 17 : audit [INF] from='osd.6 v2:172.21.15.171:6808/2509408498' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-23T00:26:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:42.589835+0000 mon.a (mon.0) 464 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-23T00:26:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:42.811022+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:42.831216+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:43.511 DEBUG:teuthology.orchestra.run.smithi171:osd.6> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.6.service 2024-02-23T00:26:43.514 INFO:tasks.cephadm:Deploying osd.7 on smithi171 with /dev/vg_nvme/lv_1... 2024-02-23T00:26:43.514 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-23T00:26:43.544 INFO:journalctl@ceph.osd.6.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:26:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:42.100457+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:42.107522+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:42.108374+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:42.109151+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:42.113575+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:42.589471+0000 mon.b (mon.1) 17 : audit [INF] from='osd.6 v2:172.21.15.171:6808/2509408498' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-23T00:26:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:42.589835+0000 mon.a (mon.0) 464 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-23T00:26:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:42.811022+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:42.831216+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:42.100457+0000 mgr.y (mgr.14146) 176 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:42.107522+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:42.108374+0000 mon.a (mon.0) 461 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:42.109151+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:42.113575+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:42.589471+0000 mon.b (mon.1) 17 : audit [INF] from='osd.6 v2:172.21.15.171:6808/2509408498' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-23T00:26:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:42.589835+0000 mon.a (mon.0) 464 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-23T00:26:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:42.811022+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:42.831216+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:26:44.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:43.464897+0000 mon.a (mon.0) 467 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-23T00:26:44.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:43.464988+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-23T00:26:44.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:43.465607+0000 mon.b (mon.1) 18 : audit [INF] from='osd.6 v2:172.21.15.171:6808/2509408498' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:44.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:43.465962+0000 mon.a (mon.0) 469 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:44.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:43.466475+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:44.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:43.464897+0000 mon.a (mon.0) 467 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-23T00:26:44.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:43.464988+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-23T00:26:44.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:43.465607+0000 mon.b (mon.1) 18 : audit [INF] from='osd.6 v2:172.21.15.171:6808/2509408498' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:44.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:43.465962+0000 mon.a (mon.0) 469 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:44.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:43.466475+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:44.911 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:43.464897+0000 mon.a (mon.0) 467 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-23T00:26:44.911 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:43.464988+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-23T00:26:44.911 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:43.465607+0000 mon.b (mon.1) 18 : audit [INF] from='osd.6 v2:172.21.15.171:6808/2509408498' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:44.911 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:43.465962+0000 mon.a (mon.0) 469 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:26:44.912 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:43.466475+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:44.912 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:26:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:26:44.475+0000 7fef560e0700 -1 osd.6 0 waiting for initial osdmap 2024-02-23T00:26:44.912 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:26:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:26:44.481+0000 7fef50057700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-23T00:26:45.689 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:44.100797+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:44.473041+0000 mon.a (mon.0) 471 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:26:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:44.473132+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-23T00:26:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:44.473453+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:44.476225+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:45.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:44.100797+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:44.473041+0000 mon.a (mon.0) 471 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:26:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:44.473132+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-23T00:26:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:44.473453+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:44.476225+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:45.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:44.100797+0000 mgr.y (mgr.14146) 177 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-23T00:26:45.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:44.473041+0000 mon.a (mon.0) 471 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:26:45.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:44.473132+0000 mon.a (mon.0) 472 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-23T00:26:45.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:44.473453+0000 mon.a (mon.0) 473 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:45.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:44.476225+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:46.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:43.583363+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:46.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:43.583408+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:46.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:44.529202+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:46.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:44.529283+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:46.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:45.472290+0000 mon.a (mon.0) 475 : cluster [INF] osd.6 v2:172.21.15.171:6808/2509408498 boot 2024-02-23T00:26:46.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:45.472415+0000 mon.a (mon.0) 476 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-02-23T00:26:46.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:45.480704+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:46.840 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:46.475035+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-23T00:26:46.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:43.583363+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:46.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:43.583408+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:46.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:44.529202+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:46.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:44.529283+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:46.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:45.472290+0000 mon.a (mon.0) 475 : cluster [INF] osd.6 v2:172.21.15.171:6808/2509408498 boot 2024-02-23T00:26:46.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:45.472415+0000 mon.a (mon.0) 476 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-02-23T00:26:46.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:45.480704+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:46.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:46.475035+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-23T00:26:46.868 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-23T00:26:46.868 INFO:teuthology.orchestra.run.smithi171.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-23T00:26:46.868 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10+0 records in 2024-02-23T00:26:46.868 INFO:teuthology.orchestra.run.smithi171.stderr:10+0 records out 2024-02-23T00:26:46.868 INFO:teuthology.orchestra.run.smithi171.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126187 s, 831 MB/s 2024-02-23T00:26:46.868 INFO:teuthology.orchestra.run.smithi171.stderr:--> Zapping successful for: 2024-02-23T00:26:46.880 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:43.583363+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:46.880 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:43.583408+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:46.880 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:44.529202+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:46.881 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:44.529283+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:46.881 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:45.472290+0000 mon.a (mon.0) 475 : cluster [INF] osd.6 v2:172.21.15.171:6808/2509408498 boot 2024-02-23T00:26:46.881 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:45.472415+0000 mon.a (mon.0) 476 : cluster [DBG] osdmap e44: 7 total, 7 up, 7 in 2024-02-23T00:26:46.881 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:45.480704+0000 mon.a (mon.0) 477 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:26:46.881 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:46.475035+0000 mon.a (mon.0) 478 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-23T00:26:47.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:46.101152+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:47.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:47.473984+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-23T00:26:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:46.101152+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:47.473984+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-23T00:26:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:46.101152+0000 mgr.y (mgr.14146) 178 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:47.473984+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-23T00:26:48.298 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch daemon add osd smithi171:vg_nvme/lv_1 2024-02-23T00:26:49.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:26:47.915232+0000 mgr.y (mgr.14146) 179 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:26:49.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:47.924291+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:49.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:47.925481+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:49.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:47.926140+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:49.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:47.926745+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:49.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:26:47.927183+0000 mgr.y (mgr.14146) 180 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 5707M 2024-02-23T00:26:49.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:47.933277+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:49.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:47.934239+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:49.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:47.934900+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:49.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:47.941750+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:49.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:48.101736+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:26:47.915232+0000 mgr.y (mgr.14146) 179 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:47.924291+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:47.925481+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:47.926140+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:47.926745+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:26:47.927183+0000 mgr.y (mgr.14146) 180 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 5707M 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:47.933277+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:47.934239+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:47.934900+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:47.941750+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:48.101736+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:26:47.915232+0000 mgr.y (mgr.14146) 179 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:47.924291+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:47.925481+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:47.926140+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:47.926745+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:26:47.927183+0000 mgr.y (mgr.14146) 180 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 5707M 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:47.933277+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:47.934239+0000 mon.a (mon.0) 485 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:47.934900+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:47.941750+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:26:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:48.101736+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:50.141 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 -- 172.21.15.171:0/3179144907 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f8d20 msgr2=0x7f877c0f9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:50.141 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 --2- 172.21.15.171:0/3179144907 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f8d20 0x7f877c0f9180 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f876c0048a0 tx=0x7f876c009cb0).stop 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 -- 172.21.15.171:0/3179144907 shutdown_connections 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 --2- 172.21.15.171:0/3179144907 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f877c0f9670 0x7f877c0fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 --2- 172.21.15.171:0/3179144907 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f877c0f7a80 0x7f877c0f7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 --2- 172.21.15.171:0/3179144907 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f8d20 0x7f877c0f9180 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 -- 172.21.15.171:0/3179144907 >> 172.21.15.171:0/3179144907 conn(0x7f877c0f32a0 msgr2=0x7f877c0f56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 -- 172.21.15.171:0/3179144907 shutdown_connections 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 -- 172.21.15.171:0/3179144907 wait complete. 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 librados: starting msgr at 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 librados: starting objecter 2024-02-23T00:26:50.142 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.136+0000 7f87812a9700 1 Processor -- start 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 -- start start 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 librados: setting wanted keys 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 librados: calling monclient init 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f877c0f7a80 0x7f877c107f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f877c0f8d20 0x7f877c1083f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f9670 0x7f877c10f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c10f9c0 con 0x7f877c0f7a80 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c10fb00 con 0x7f877c0f9670 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877c10fc40 con 0x7f877c0f8d20 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877b7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f9670 0x7f877c10f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877b7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f9670 0x7f877c10f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:50938/0 (socket says 172.21.15.171:50938) 2024-02-23T00:26:50.143 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877b7fe700 1 -- 172.21.15.171:0/231252344 learned_addr learned my addr 172.21.15.171:0/231252344 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877affd700 1 --2- 172.21.15.171:0/231252344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f877c0f7a80 0x7f877c107f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877a7fc700 1 --2- 172.21.15.171:0/231252344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f877c0f8d20 0x7f877c1083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877b7fe700 1 -- 172.21.15.171:0/231252344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f877c0f8d20 msgr2=0x7f877c1083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877b7fe700 1 --2- 172.21.15.171:0/231252344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f877c0f8d20 0x7f877c1083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877b7fe700 1 -- 172.21.15.171:0/231252344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f877c0f7a80 msgr2=0x7f877c107f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877b7fe700 1 --2- 172.21.15.171:0/231252344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f877c0f7a80 0x7f877c107f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877b7fe700 1 -- 172.21.15.171:0/231252344 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c005040 con 0x7f877c0f9670 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f877b7fe700 1 --2- 172.21.15.171:0/231252344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f9670 0x7f877c10f4d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f877400c960 tx=0x7f8774004d60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f8763fff700 10 client.?.objecter ms_handle_connect 0x7f877c0f9670 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f8763fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f8763fff700 1 -- 172.21.15.171:0/231252344 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877400a520 con 0x7f877c0f9670 2024-02-23T00:26:50.144 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c10ff20 con 0x7f877c0f9670 2024-02-23T00:26:50.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f8763fff700 1 -- 172.21.15.171:0/231252344 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f877400ab50 con 0x7f877c0f9670 2024-02-23T00:26:50.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f8763fff700 1 -- 172.21.15.171:0/231252344 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8774012ba0 con 0x7f877c0f9670 2024-02-23T00:26:50.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 10 client.24274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:26:50.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c110400 con 0x7f877c0f9670 2024-02-23T00:26:50.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.137+0000 7f87812a9700 1 librados: init done 2024-02-23T00:26:50.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.138+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c102860 con 0x7f877c0f9670 2024-02-23T00:26:50.145 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.139+0000 7f8763fff700 1 -- 172.21.15.171:0/231252344 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f877400a680 con 0x7f877c0f9670 2024-02-23T00:26:50.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.139+0000 7f8763fff700 1 --2- 172.21.15.171:0/231252344 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8764062ba0 0x7f8764065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:26:50.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.139+0000 7f8763fff700 1 -- 172.21.15.171:0/231252344 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 3762+0+0 (secure 0 0 0) 0x7f877407bbb0 con 0x7f877c0f9670 2024-02-23T00:26:50.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.139+0000 7f8763fff700 10 client.24274.objecter ms_dispatch 0x7f877c001550 osd_map(47..47 src has 1..47) v4 2024-02-23T00:26:50.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.139+0000 7f8763fff700 3 client.24274.objecter handle_osd_map got epochs [47,47] > 0 2024-02-23T00:26:50.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.139+0000 7f8763fff700 3 client.24274.objecter handle_osd_map decoding full epoch 47 2024-02-23T00:26:50.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.139+0000 7f8763fff700 20 client.24274.objecter dump_active .. 0 homeless 2024-02-23T00:26:50.146 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.140+0000 7f877affd700 1 --2- 172.21.15.171:0/231252344 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8764062ba0 0x7f8764065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:26:50.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.140+0000 7f8763fff700 10 client.24274.objecter ms_handle_connect 0x7f8764062ba0 2024-02-23T00:26:50.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.140+0000 7f877affd700 1 --2- 172.21.15.171:0/231252344 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8764062ba0 0x7f8764065060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f877c0f8b80 tx=0x7f8770009040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:26:50.147 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.144+0000 7f8763fff700 1 -- 172.21.15.171:0/231252344 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8774048f60 con 0x7f877c0f9670 2024-02-23T00:26:50.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:49.127292+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:50.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:49.127381+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:50.315 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:50.314+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f877c0feea0 con 0x7f8764062ba0 2024-02-23T00:26:50.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:49.127292+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:50.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:49.127381+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:50.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:49.127292+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:26:50.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:49.127381+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:26:51.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:49.950320+0000 mon.a (mon.0) 488 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-23T00:26:51.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:50.102026+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:50.315478+0000 mgr.y (mgr.14146) 183 : audit [DBG] from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:26:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:50.317607+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:26:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:50.319656+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:26:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:50.320248+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:51.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:49.950320+0000 mon.a (mon.0) 488 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-23T00:26:51.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:50.102026+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:51.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:50.315478+0000 mgr.y (mgr.14146) 183 : audit [DBG] from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:26:51.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:50.317607+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:26:51.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:50.319656+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:26:51.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:50.320248+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:51.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:49.950320+0000 mon.a (mon.0) 488 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-23T00:26:51.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:50.102026+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:51.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:50.315478+0000 mgr.y (mgr.14146) 183 : audit [DBG] from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi171:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:26:51.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:50.317607+0000 mon.a (mon.0) 489 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-23T00:26:51.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:50.319656+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-23T00:26:51.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:50.320248+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:52.102518+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:52.602225+0000 mon.b (mon.1) 19 : audit [INF] from='client.? 172.21.15.171:0/2803426468' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939"}]: dispatch 2024-02-23T00:26:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:52.602835+0000 mon.a (mon.0) 492 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939"}]: dispatch 2024-02-23T00:26:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:52.607497+0000 mon.a (mon.0) 493 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939"}]': finished 2024-02-23T00:26:53.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:52.607595+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2024-02-23T00:26:53.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:52.607763+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:26:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:52.102518+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:52.602225+0000 mon.b (mon.1) 19 : audit [INF] from='client.? 172.21.15.171:0/2803426468' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939"}]: dispatch 2024-02-23T00:26:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:52.602835+0000 mon.a (mon.0) 492 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939"}]: dispatch 2024-02-23T00:26:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:52.607497+0000 mon.a (mon.0) 493 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939"}]': finished 2024-02-23T00:26:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:52.607595+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2024-02-23T00:26:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:52.607763+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:26:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:52.102518+0000 mgr.y (mgr.14146) 184 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:52.602225+0000 mon.b (mon.1) 19 : audit [INF] from='client.? 172.21.15.171:0/2803426468' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939"}]: dispatch 2024-02-23T00:26:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:52.602835+0000 mon.a (mon.0) 492 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939"}]: dispatch 2024-02-23T00:26:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:52.607497+0000 mon.a (mon.0) 493 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939"}]': finished 2024-02-23T00:26:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:52.607595+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2024-02-23T00:26:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:52.607763+0000 mon.a (mon.0) 495 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:26:54.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:53.365100+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? 172.21.15.171:0/3428279112' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:26:54.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:53.365100+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? 172.21.15.171:0/3428279112' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:26:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:53.365100+0000 mon.b (mon.1) 20 : audit [DBG] from='client.? 172.21.15.171:0/3428279112' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-23T00:26:55.139 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:26:55.138+0000 7f8780aa8700 10 client.24274.objecter tick 2024-02-23T00:26:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:54.102941+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:55.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:54.102941+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:55.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:54.102941+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v174: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:56.103419+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:57.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:56.103419+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:56.103419+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:26:58.103833+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:58.926833+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-23T00:26:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:26:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:26:58.927604+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:59.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:26:58.103833+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:59.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:58.926833+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-23T00:26:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:26:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:26:58.927604+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:26:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:26:58.103833+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:26:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:58.926833+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-23T00:26:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:26:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:26:58.927604+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:00.140 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:00.139+0000 7f8780aa8700 10 client.24274.objecter tick 2024-02-23T00:27:00.338 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:26:58.928220+0000 mgr.y (mgr.14146) 188 : cephadm [INF] Deploying daemon osd.7 on smithi171 2024-02-23T00:27:00.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:26:58.928220+0000 mgr.y (mgr.14146) 188 : cephadm [INF] Deploying daemon osd.7 on smithi171 2024-02-23T00:27:00.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:26:58.928220+0000 mgr.y (mgr.14146) 188 : cephadm [INF] Deploying daemon osd.7 on smithi171 2024-02-23T00:27:01.331 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:00.104237+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:01.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:00.104237+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:01.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:00.104237+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:03.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:02.104594+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:03.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:02.104594+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:03.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:02.104594+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:05.141 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:05.139+0000 7f8780aa8700 10 client.24274.objecter tick 2024-02-23T00:27:05.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:03.963689+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:05.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:03.997040+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:05.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:04.104915+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:05.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:03.963689+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:05.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:03.997040+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:05.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:04.104915+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:05.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:03.963689+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:05.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:03.997040+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:05.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:04.104915+0000 mgr.y (mgr.14146) 191 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:07.371 INFO:teuthology.orchestra.run.smithi171.stdout:Created osd(s) 7 on host 'smithi171' 2024-02-23T00:27:07.371 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.369+0000 7f8763fff700 1 -- 172.21.15.171:0/231252344 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f877c0feea0 con 0x7f8764062ba0 2024-02-23T00:27:07.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 10 librados: watch_flush enter 2024-02-23T00:27:07.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 10 librados: watch_flush exit 2024-02-23T00:27:07.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 20 client.24274.objecter shutdown clearing up homeless session... 2024-02-23T00:27:07.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 10 client.24274.objecter successfully canceled tick 2024-02-23T00:27:07.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8764062ba0 msgr2=0x7f8764065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:07.373 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 --2- 172.21.15.171:0/231252344 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8764062ba0 0x7f8764065060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f877c0f8b80 tx=0x7f8770009040).stop 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f9670 msgr2=0x7f877c10f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 --2- 172.21.15.171:0/231252344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f9670 0x7f877c10f4d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f877400c960 tx=0x7f8774004d60).stop 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 shutdown_connections 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 --2- 172.21.15.171:0/231252344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f877c0f9670 0x7f877c10f4d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 --2- 172.21.15.171:0/231252344 >> v2:172.21.15.87:6800/1121018082 conn(0x7f8764062ba0 0x7f8764065060 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 --2- 172.21.15.171:0/231252344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f877c0f7a80 0x7f877c107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 --2- 172.21.15.171:0/231252344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f877c0f8d20 0x7f877c1083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 >> 172.21.15.171:0/231252344 conn(0x7f877c0f32a0 msgr2=0x7f877c0f6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 shutdown_connections 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 -- 172.21.15.171:0/231252344 wait complete. 2024-02-23T00:27:07.374 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:07.371+0000 7f87812a9700 1 librados: shutdown 2024-02-23T00:27:07.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:06.105335+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:06.624568+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.085419+0000 mon.b (mon.1) 21 : audit [INF] from='osd.7 v2:172.21.15.171:6812/373787000' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.085703+0000 mon.a (mon.0) 501 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.369674+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.481872+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.482667+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.483398+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.484153+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.490565+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.491579+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.492270+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:07.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.497429+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:07.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.499638+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:08.035 DEBUG:teuthology.orchestra.run.smithi171:osd.7> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.7.service 2024-02-23T00:27:08.038 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-23T00:27:08.038 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd stat -f json 2024-02-23T00:27:08.067 INFO:journalctl@ceph.osd.7.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:27:08.071 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:06.105335+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:08.071 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:06.624568+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:08.071 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.085419+0000 mon.b (mon.1) 21 : audit [INF] from='osd.7 v2:172.21.15.171:6812/373787000' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-23T00:27:08.071 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.085703+0000 mon.a (mon.0) 501 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-23T00:27:08.071 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.369674+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:08.071 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.481872+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.482667+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.483398+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.484153+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.490565+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.491579+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.492270+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.497429+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.499638+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:06.105335+0000 mgr.y (mgr.14146) 192 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:08.072 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:06.624568+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.085419+0000 mon.b (mon.1) 21 : audit [INF] from='osd.7 v2:172.21.15.171:6812/373787000' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.085703+0000 mon.a (mon.0) 501 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.369674+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.481872+0000 mon.a (mon.0) 503 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.482667+0000 mon.a (mon.0) 504 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.483398+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.484153+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.490565+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.491579+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.492270+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.497429+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:08.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.499638+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:08.718 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:07.484822+0000 mgr.y (mgr.14146) 193 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 4280M 2024-02-23T00:27:08.718 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.625356+0000 mon.a (mon.0) 512 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-23T00:27:08.718 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:07.625424+0000 mon.a (mon.0) 513 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-23T00:27:08.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.625784+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:08.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.628287+0000 mon.b (mon.1) 22 : audit [INF] from='osd.7 v2:172.21.15.171:6812/373787000' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:27:08.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:07.633376+0000 mon.a (mon.0) 515 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:27:08.719 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:07.484822+0000 mgr.y (mgr.14146) 193 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 4280M 2024-02-23T00:27:08.719 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.625356+0000 mon.a (mon.0) 512 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-23T00:27:08.719 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:07.625424+0000 mon.a (mon.0) 513 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-23T00:27:08.719 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.625784+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:08.719 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.628287+0000 mon.b (mon.1) 22 : audit [INF] from='osd.7 v2:172.21.15.171:6812/373787000' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:27:08.719 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:07.633376+0000 mon.a (mon.0) 515 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:27:08.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:27:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:27:08.637+0000 7fbed8fad700 -1 osd.7 0 waiting for initial osdmap 2024-02-23T00:27:08.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:27:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:27:08.646+0000 7fbed3725700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-23T00:27:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:07.484822+0000 mgr.y (mgr.14146) 193 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 4280M 2024-02-23T00:27:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.625356+0000 mon.a (mon.0) 512 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-23T00:27:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:07.625424+0000 mon.a (mon.0) 513 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-23T00:27:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.625784+0000 mon.a (mon.0) 514 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.628287+0000 mon.b (mon.1) 22 : audit [INF] from='osd.7 v2:172.21.15.171:6812/373787000' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:27:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:07.633376+0000 mon.a (mon.0) 515 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]: dispatch 2024-02-23T00:27:09.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:08.105663+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:09.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:08.636447+0000 mon.a (mon.0) 516 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:27:09.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:08.636512+0000 mon.a (mon.0) 517 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-23T00:27:09.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:08.636909+0000 mon.a (mon.0) 518 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:09.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:08.639762+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:08.105663+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:08.636447+0000 mon.a (mon.0) 516 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:27:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:08.636512+0000 mon.a (mon.0) 517 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-23T00:27:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:08.636909+0000 mon.a (mon.0) 518 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:08.639762+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:09.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:08.105663+0000 mgr.y (mgr.14146) 194 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:27:09.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:08.636447+0000 mon.a (mon.0) 516 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi171", "root=default"]}]': finished 2024-02-23T00:27:09.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:08.636512+0000 mon.a (mon.0) 517 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-23T00:27:09.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:08.636909+0000 mon.a (mon.0) 518 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:09.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:08.639762+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:10.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.164+0000 7f3489737700 1 -- 172.21.15.87:0/85645736 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f99d0 msgr2=0x7f34840f5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:10.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.164+0000 7f3489737700 1 --2- 172.21.15.87:0/85645736 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f99d0 0x7f34840f5be0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f34740047b0 tx=0x7f3474009cb0).stop 2024-02-23T00:27:10.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.164+0000 7f3489737700 1 -- 172.21.15.87:0/85645736 shutdown_connections 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.164+0000 7f3489737700 1 --2- 172.21.15.87:0/85645736 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34840f61b0 0x7f34840f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.164+0000 7f3489737700 1 --2- 172.21.15.87:0/85645736 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34840f90c0 0x7f34840f94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.164+0000 7f3489737700 1 --2- 172.21.15.87:0/85645736 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f99d0 0x7f34840f5be0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.164+0000 7f3489737700 1 -- 172.21.15.87:0/85645736 >> 172.21.15.87:0/85645736 conn(0x7f34840f1900 msgr2=0x7f34840f3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.164+0000 7f3489737700 1 -- 172.21.15.87:0/85645736 shutdown_connections 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.164+0000 7f3489737700 1 -- 172.21.15.87:0/85645736 wait complete. 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 librados: starting msgr at 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 librados: starting objecter 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 Processor -- start 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 -- start start 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 librados: setting wanted keys 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 librados: calling monclient init 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34840f61b0 0x7f34840f87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:10.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f90c0 0x7f34840f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34840f99d0 0x7f34840f7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840f7920 con 0x7f34840f61b0 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34840f7a90 con 0x7f34840f99d0 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3489737700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34840f7c50 con 0x7f34840f90c0 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f34827fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f90c0 0x7f34840f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f3482ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34840f61b0 0x7f34840f87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f34827fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f90c0 0x7f34840f6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:45312/0 (socket says 172.21.15.87:45312) 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.165+0000 7f34827fc700 1 -- 172.21.15.87:0/262082277 learned_addr learned my addr 172.21.15.87:0/262082277 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f34837fe700 1 --2- 172.21.15.87:0/262082277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34840f99d0 0x7f34840f7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f34827fc700 1 -- 172.21.15.87:0/262082277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34840f99d0 msgr2=0x7f34840f7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f34827fc700 1 --2- 172.21.15.87:0/262082277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34840f99d0 0x7f34840f7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f34827fc700 1 -- 172.21.15.87:0/262082277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34840f61b0 msgr2=0x7f34840f87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f34827fc700 1 --2- 172.21.15.87:0/262082277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34840f61b0 0x7f34840f87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:10.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f34827fc700 1 -- 172.21.15.87:0/262082277 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c005040 con 0x7f34840f90c0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f3482ffd700 1 --2- 172.21.15.87:0/262082277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34840f61b0 0x7f34840f87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f34827fc700 1 --2- 172.21.15.87:0/262082277 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f90c0 0x7f34840f6e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f3474009b10 tx=0x7f3474009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f347bfff700 10 client.?.objecter ms_handle_connect 0x7f34840f90c0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f347bfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f347bfff700 1 -- 172.21.15.87:0/262082277 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474018070 con 0x7f34840f90c0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f347bfff700 1 -- 172.21.15.87:0/262082277 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3474007d50 con 0x7f34840f90c0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3474005040 con 0x7f34840f90c0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f347bfff700 1 -- 172.21.15.87:0/262082277 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347401dce0 con 0x7f34840f90c0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f3489737700 10 client.24305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3484112540 con 0x7f34840f90c0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.166+0000 7f3489737700 1 librados: init done 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.167+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348405ffd0 con 0x7f34840f90c0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.167+0000 7f347bfff700 1 -- 172.21.15.87:0/262082277 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f347401d420 con 0x7f34840f90c0 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.168+0000 7f347bfff700 1 --2- 172.21.15.87:0/262082277 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3470062ba0 0x7f3470065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:10.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.168+0000 7f347bfff700 1 -- 172.21.15.87:0/262082277 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4054+0+0 (secure 0 0 0) 0x7f347407d520 con 0x7f34840f90c0 2024-02-23T00:27:10.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.168+0000 7f347bfff700 10 client.24305.objecter ms_dispatch 0x7f3484001550 osd_map(52..52 src has 1..52) v4 2024-02-23T00:27:10.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.168+0000 7f347bfff700 3 client.24305.objecter handle_osd_map got epochs [52,52] > 0 2024-02-23T00:27:10.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.168+0000 7f347bfff700 3 client.24305.objecter handle_osd_map decoding full epoch 52 2024-02-23T00:27:10.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.168+0000 7f3482ffd700 1 --2- 172.21.15.87:0/262082277 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3470062ba0 0x7f3470065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:10.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.168+0000 7f347bfff700 20 client.24305.objecter dump_active .. 0 homeless 2024-02-23T00:27:10.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.168+0000 7f347bfff700 10 client.24305.objecter ms_handle_connect 0x7f3470062ba0 2024-02-23T00:27:10.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.168+0000 7f3482ffd700 1 --2- 172.21.15.87:0/262082277 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3470062ba0 0x7f3470065060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f346c004c80 tx=0x7f346c00b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:10.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.172+0000 7f347bfff700 1 -- 172.21.15.87:0/262082277 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f347404e040 con 0x7f34840f90c0 2024-02-23T00:27:10.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.333+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f34840008d0 con 0x7f34840f90c0 2024-02-23T00:27:10.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.333+0000 7f347bfff700 1 -- 172.21.15.87:0/262082277 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v52) v1 ==== 74+0+130 (secure 0 0 0) 0x7f34740043d0 con 0x7f34840f90c0 2024-02-23T00:27:10.334 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:27:10.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.334+0000 7f3489737700 10 librados: watch_flush enter 2024-02-23T00:27:10.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 10 librados: watch_flush exit 2024-02-23T00:27:10.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 20 client.24305.objecter shutdown clearing up homeless session... 2024-02-23T00:27:10.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 10 client.24305.objecter successfully canceled tick 2024-02-23T00:27:10.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3470062ba0 msgr2=0x7f3470065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:10.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 --2- 172.21.15.87:0/262082277 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3470062ba0 0x7f3470065060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f346c004c80 tx=0x7f346c00b040).stop 2024-02-23T00:27:10.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f90c0 msgr2=0x7f34840f6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:10.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 --2- 172.21.15.87:0/262082277 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f90c0 0x7f34840f6e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f3474009b10 tx=0x7f3474009cb0).stop 2024-02-23T00:27:10.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 shutdown_connections 2024-02-23T00:27:10.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 --2- 172.21.15.87:0/262082277 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f34840f99d0 0x7f34840f7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:10.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 --2- 172.21.15.87:0/262082277 >> v2:172.21.15.87:6800/1121018082 conn(0x7f3470062ba0 0x7f3470065060 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:10.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 --2- 172.21.15.87:0/262082277 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f34840f61b0 0x7f34840f87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:10.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 --2- 172.21.15.87:0/262082277 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f34840f90c0 0x7f34840f6e10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:10.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 >> 172.21.15.87:0/262082277 conn(0x7f34840f1900 msgr2=0x7f34840f3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:10.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 shutdown_connections 2024-02-23T00:27:10.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 -- 172.21.15.87:0/262082277 wait complete. 2024-02-23T00:27:10.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:10.335+0000 7f3489737700 1 librados: shutdown 2024-02-23T00:27:10.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:08.063247+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:08.063301+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:09.639132+0000 mon.a (mon.0) 520 : cluster [INF] osd.7 v2:172.21.15.171:6812/373787000 boot 2024-02-23T00:27:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:09.639176+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2024-02-23T00:27:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:09.639582+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:09.836642+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-02-23T00:27:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:10.333583+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.87:0/262082277' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-23T00:27:10.949 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:08.063247+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:08.063301+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:09.639132+0000 mon.a (mon.0) 520 : cluster [INF] osd.7 v2:172.21.15.171:6812/373787000 boot 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:09.639176+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:09.639582+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:09.836642+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:10.333583+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.87:0/262082277' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:08.063247+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:08.063301+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:09.639132+0000 mon.a (mon.0) 520 : cluster [INF] osd.7 v2:172.21.15.171:6812/373787000 boot 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:09.639176+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e51: 8 total, 8 up, 8 in 2024-02-23T00:27:10.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:09.639582+0000 mon.a (mon.0) 522 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:10.951 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:09.836642+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2024-02-23T00:27:10.951 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:10.333583+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.87:0/262082277' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-23T00:27:10.971 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":52,"num_osds":8,"num_up_osds":8,"osd_up_since":1708648029,"num_in_osds":8,"osd_in_since":1708648012,"num_remapped_pgs":0} 2024-02-23T00:27:10.971 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi087 2024-02-23T00:27:10.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi087=foo.a' 2024-02-23T00:27:11.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:10.106023+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v186: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:11.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:10.838464+0000 mon.a (mon.0) 524 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:27:11.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:10.845874+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-23T00:27:11.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:11.202838+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:11.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:11.203606+0000 mon.a (mon.0) 527 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:11.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:11.204292+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:11.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:11.209806+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:12.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:10.106023+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v186: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:10.838464+0000 mon.a (mon.0) 524 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:10.845874+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:11.202838+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:11.203606+0000 mon.a (mon.0) 527 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:11.204292+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:11.209806+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:10.106023+0000 mgr.y (mgr.14146) 195 : cluster [DBG] pgmap v186: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:10.838464+0000 mon.a (mon.0) 524 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:10.845874+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:11.202838+0000 mon.a (mon.0) 526 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:11.203606+0000 mon.a (mon.0) 527 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:12.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:11.204292+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:12.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:11.209806+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:12.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.823+0000 7f506cd30700 1 -- 172.21.15.171:0/2797632548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f7c30 msgr2=0x7f50680f8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:12.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.823+0000 7f506cd30700 1 --2- 172.21.15.171:0/2797632548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f7c30 0x7f50680f8090 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f50580048a0 tx=0x7f5058009cb0).stop 2024-02-23T00:27:12.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.823+0000 7f506cd30700 1 -- 172.21.15.171:0/2797632548 shutdown_connections 2024-02-23T00:27:12.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.823+0000 7f506cd30700 1 --2- 172.21.15.171:0/2797632548 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50680f8580 0x7f50680fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:27:12.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.823+0000 7f506cd30700 1 --2- 172.21.15.171:0/2797632548 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50680f6990 0x7f50680f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:12.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.823+0000 7f506cd30700 1 --2- 172.21.15.171:0/2797632548 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f7c30 0x7f50680f8090 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:12.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.823+0000 7f506cd30700 1 -- 172.21.15.171:0/2797632548 >> 172.21.15.171:0/2797632548 conn(0x7f50680f2190 msgr2=0x7f50680f45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:12.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.823+0000 7f506cd30700 1 -- 172.21.15.171:0/2797632548 shutdown_connections 2024-02-23T00:27:12.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.823+0000 7f506cd30700 1 -- 172.21.15.171:0/2797632548 wait complete. 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 librados: starting msgr at 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 librados: starting objecter 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 Processor -- start 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 -- start start 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 librados: setting wanted keys 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 librados: calling monclient init 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f6990 0x7f506810a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50680f7c30 0x7f506810a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50680f8580 0x7f50681119b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068111ea0 con 0x7f50680f7c30 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068111fe0 con 0x7f50680f6990 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506cd30700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5068112120 con 0x7f50680f8580 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506659c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f6990 0x7f506810a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:12.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506659c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f6990 0x7f506810a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:42942/0 (socket says 172.21.15.171:42942) 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.824+0000 7f506659c700 1 -- 172.21.15.171:0/652064345 learned_addr learned my addr 172.21.15.171:0/652064345 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506659c700 1 -- 172.21.15.171:0/652064345 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50680f8580 msgr2=0x7f50681119b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f5065d9b700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50680f7c30 0x7f506810a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f5066d9d700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50680f8580 0x7f50681119b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506659c700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50680f8580 0x7f50681119b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506659c700 1 -- 172.21.15.171:0/652064345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50680f7c30 msgr2=0x7f506810a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506659c700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50680f7c30 0x7f506810a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506659c700 1 -- 172.21.15.171:0/652064345 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5058005040 con 0x7f50680f6990 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f5065d9b700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50680f7c30 0x7f506810a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506659c700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f6990 0x7f506810a3e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f505c00c810 tx=0x7f505c007f90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f50577fe700 10 client.?.objecter ms_handle_connect 0x7f50680f6990 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f50577fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f50577fe700 1 -- 172.21.15.171:0/652064345 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c010510 con 0x7f50680f6990 2024-02-23T00:27:12.830 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f50577fe700 1 -- 172.21.15.171:0/652064345 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f505c010b40 con 0x7f50680f6990 2024-02-23T00:27:12.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5068112400 con 0x7f50680f6990 2024-02-23T00:27:12.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f50577fe700 1 -- 172.21.15.171:0/652064345 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c012ba0 con 0x7f50680f6990 2024-02-23T00:27:12.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506cd30700 10 client.24307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:12.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50681128f0 con 0x7f50680f6990 2024-02-23T00:27:12.831 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.825+0000 7f506cd30700 1 librados: init done 2024-02-23T00:27:12.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.826+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506805f360 con 0x7f50680f6990 2024-02-23T00:27:12.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.826+0000 7f50577fe700 1 -- 172.21.15.171:0/652064345 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f505c010670 con 0x7f50680f6990 2024-02-23T00:27:12.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.827+0000 7f50577fe700 1 --2- 172.21.15.171:0/652064345 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5050062ba0 0x7f5050065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:12.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.827+0000 7f50577fe700 1 -- 172.21.15.171:0/652064345 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4054+0+0 (secure 0 0 0) 0x7f505c07c070 con 0x7f50680f6990 2024-02-23T00:27:12.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.827+0000 7f50577fe700 10 client.24307.objecter ms_dispatch 0x7f5068001550 osd_map(53..53 src has 1..53) v4 2024-02-23T00:27:12.832 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.827+0000 7f50577fe700 3 client.24307.objecter handle_osd_map got epochs [53,53] > 0 2024-02-23T00:27:12.833 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.827+0000 7f50577fe700 3 client.24307.objecter handle_osd_map decoding full epoch 53 2024-02-23T00:27:12.833 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.827+0000 7f50577fe700 20 client.24307.objecter dump_active .. 0 homeless 2024-02-23T00:27:12.833 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.827+0000 7f5065d9b700 1 --2- 172.21.15.171:0/652064345 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5050062ba0 0x7f5050065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:12.833 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.827+0000 7f50577fe700 10 client.24307.objecter ms_handle_connect 0x7f5050062ba0 2024-02-23T00:27:12.833 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.827+0000 7f5065d9b700 1 --2- 172.21.15.171:0/652064345 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5050062ba0 0x7f5050065060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f5058005010 tx=0x7f5058009cb0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:12.833 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:12.831+0000 7f50577fe700 1 -- 172.21.15.171:0/652064345 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f505c04cb60 con 0x7f50680f6990 2024-02-23T00:27:12.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:11.198277+0000 mgr.y (mgr.14146) 196 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:27:13.006 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.004+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi087=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f50680fd4f0 con 0x7f5050062ba0 2024-02-23T00:27:13.013 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.011+0000 7f50577fe700 1 -- 172.21.15.171:0/652064345 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f50680fd4f0 con 0x7f5050062ba0 2024-02-23T00:27:13.013 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled rgw.foo.a update... 2024-02-23T00:27:13.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 10 librados: watch_flush enter 2024-02-23T00:27:13.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 10 librados: watch_flush exit 2024-02-23T00:27:13.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 20 client.24307.objecter shutdown clearing up homeless session... 2024-02-23T00:27:13.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 10 client.24307.objecter successfully canceled tick 2024-02-23T00:27:13.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5050062ba0 msgr2=0x7f5050065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:13.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 1 --2- 172.21.15.171:0/652064345 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5050062ba0 0x7f5050065060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f5058005010 tx=0x7f5058009cb0).stop 2024-02-23T00:27:13.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f6990 msgr2=0x7f506810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:13.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f6990 0x7f506810a3e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f505c00c810 tx=0x7f505c007f90).stop 2024-02-23T00:27:13.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 shutdown_connections 2024-02-23T00:27:13.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50680f8580 0x7f50681119b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:13.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50680f6990 0x7f506810a3e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:13.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 1 --2- 172.21.15.171:0/652064345 >> v2:172.21.15.87:6800/1121018082 conn(0x7f5050062ba0 0x7f5050065060 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:13.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.012+0000 7f506cd30700 1 --2- 172.21.15.171:0/652064345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50680f7c30 0x7f506810a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:13.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.013+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 >> 172.21.15.171:0/652064345 conn(0x7f50680f2190 msgr2=0x7f5068061200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:13.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.013+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 shutdown_connections 2024-02-23T00:27:13.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.013+0000 7f506cd30700 1 -- 172.21.15.171:0/652064345 wait complete. 2024-02-23T00:27:13.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:13.013+0000 7f506cd30700 1 librados: shutdown 2024-02-23T00:27:13.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:11.198277+0000 mgr.y (mgr.14146) 196 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:27:13.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:11.198277+0000 mgr.y (mgr.14146) 196 : cephadm [INF] Detected new or changed devices on smithi171 2024-02-23T00:27:13.698 DEBUG:teuthology.orchestra.run.smithi087:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@rgw.foo.a.service 2024-02-23T00:27:13.701 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi171 2024-02-23T00:27:13.701 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool create iscsi 3 3 replicated 2024-02-23T00:27:13.730 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:27:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:12.106356+0000 mgr.y (mgr.14146) 197 : cluster [DBG] pgmap v188: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.012029+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.029953+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.031572+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.032274+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.036885+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.042205+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.042773+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-23T00:27:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.047513+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-23T00:27:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.052752+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.053451+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:12.106356+0000 mgr.y (mgr.14146) 197 : cluster [DBG] pgmap v188: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.012029+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.029953+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.031572+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.032274+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:13.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.036885+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.042205+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.042773+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-23T00:27:13.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.047513+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-23T00:27:13.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.052752+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.053451+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:12.106356+0000 mgr.y (mgr.14146) 197 : cluster [DBG] pgmap v188: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.012029+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.029953+0000 mon.a (mon.0) 531 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.031572+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.032274+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.036885+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.042205+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.042773+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-23T00:27:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.047513+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-23T00:27:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.052752+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.053451+0000 mon.a (mon.0) 539 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:14.767 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:13.006626+0000 mgr.y (mgr.14146) 198 : audit [DBG] from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi087=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:14.767 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:13.008184+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi087=foo.a;count:1 2024-02-23T00:27:14.767 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:13.038989+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi087=foo.a;count:1 2024-02-23T00:27:14.767 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:13.054045+0000 mgr.y (mgr.14146) 201 : cephadm [INF] Deploying daemon rgw.foo.a on smithi087 2024-02-23T00:27:14.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:13.006626+0000 mgr.y (mgr.14146) 198 : audit [DBG] from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi087=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:14.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:13.008184+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi087=foo.a;count:1 2024-02-23T00:27:14.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:13.038989+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi087=foo.a;count:1 2024-02-23T00:27:14.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:13.054045+0000 mgr.y (mgr.14146) 201 : cephadm [INF] Deploying daemon rgw.foo.a on smithi087 2024-02-23T00:27:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:13.006626+0000 mgr.y (mgr.14146) 198 : audit [DBG] from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi087=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:13.008184+0000 mgr.y (mgr.14146) 199 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi087=foo.a;count:1 2024-02-23T00:27:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:13.038989+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi087=foo.a;count:1 2024-02-23T00:27:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:13.054045+0000 mgr.y (mgr.14146) 201 : cephadm [INF] Deploying daemon rgw.foo.a on smithi087 2024-02-23T00:27:15.082 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:27:14 smithi087 systemd[1]: Starting Ceph rgw.foo.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:27:15.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.570+0000 7f288fc73700 1 -- 172.21.15.171:0/1976810198 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28880f8370 msgr2=0x7f28880f8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:15.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.570+0000 7f288fc73700 1 --2- 172.21.15.171:0/1976810198 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28880f8370 0x7f28880f8da0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f2884004870 tx=0x7f2884009cb0).stop 2024-02-23T00:27:15.598 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.570+0000 7f288fc73700 1 -- 172.21.15.171:0/1976810198 shutdown_connections 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.570+0000 7f288fc73700 1 --2- 172.21.15.171:0/1976810198 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28880f9480 0x7f28880ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.570+0000 7f288fc73700 1 --2- 172.21.15.171:0/1976810198 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28880f7a60 0x7f28880f7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.570+0000 7f288fc73700 1 --2- 172.21.15.171:0/1976810198 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28880f8370 0x7f28880f8da0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.570+0000 7f288fc73700 1 -- 172.21.15.171:0/1976810198 >> 172.21.15.171:0/1976810198 conn(0x7f28880f32a0 msgr2=0x7f28880f56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 -- 172.21.15.171:0/1976810198 shutdown_connections 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 -- 172.21.15.171:0/1976810198 wait complete. 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 librados: starting msgr at 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 librados: starting objecter 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 Processor -- start 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 -- start start 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 librados: setting wanted keys 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 librados: calling monclient init 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28880f7a60 0x7f2888108040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:15.599 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28880f8370 0x7f2888108530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28880f9480 0x7f288810f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288810fb00 con 0x7f28880f8370 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288810fc40 con 0x7f28880f7a60 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288fc73700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288810fd80 con 0x7f28880f9480 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288da0f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28880f7a60 0x7f2888108040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288da0f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28880f7a60 0x7f2888108040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:55500/0 (socket says 172.21.15.171:55500) 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288da0f700 1 -- 172.21.15.171:0/1742735883 learned_addr learned my addr 172.21.15.171:0/1742735883 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.571+0000 7f288d20e700 1 --2- 172.21.15.171:0/1742735883 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28880f8370 0x7f2888108530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288e210700 1 --2- 172.21.15.171:0/1742735883 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28880f9480 0x7f288810f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288da0f700 1 -- 172.21.15.171:0/1742735883 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28880f9480 msgr2=0x7f288810f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288da0f700 1 --2- 172.21.15.171:0/1742735883 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28880f9480 0x7f288810f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288da0f700 1 -- 172.21.15.171:0/1742735883 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28880f8370 msgr2=0x7f2888108530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288da0f700 1 --2- 172.21.15.171:0/1742735883 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28880f8370 0x7f2888108530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:15.600 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288da0f700 1 -- 172.21.15.171:0/1742735883 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c005040 con 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288da0f700 1 --2- 172.21.15.171:0/1742735883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28880f7a60 0x7f2888108040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f287c0048f0 tx=0x7f287c009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f287affd700 10 client.?.objecter ms_handle_connect 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f287affd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f287affd700 1 -- 172.21.15.171:0/1742735883 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c007b00 con 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f287affd700 1 -- 172.21.15.171:0/1742735883 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f287c00f040 con 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28800097c0 con 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f287affd700 1 -- 172.21.15.171:0/1742735883 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c014990 con 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288fc73700 10 client.24319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2884005040 con 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.572+0000 7f288fc73700 1 librados: init done 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.573+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288805ffd0 con 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.573+0000 7f287affd700 1 -- 172.21.15.171:0/1742735883 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f287c010500 con 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.574+0000 7f287affd700 1 --2- 172.21.15.171:0/1742735883 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2874062ba0 0x7f2874065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.574+0000 7f287affd700 1 -- 172.21.15.171:0/1742735883 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4054+0+0 (secure 0 0 0) 0x7f287c07bc90 con 0x7f28880f7a60 2024-02-23T00:27:15.601 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.574+0000 7f287affd700 10 client.24319.objecter ms_dispatch 0x7f2888001550 osd_map(53..53 src has 1..53) v4 2024-02-23T00:27:15.602 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.574+0000 7f287affd700 3 client.24319.objecter handle_osd_map got epochs [53,53] > 0 2024-02-23T00:27:15.602 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.574+0000 7f287affd700 3 client.24319.objecter handle_osd_map decoding full epoch 53 2024-02-23T00:27:15.602 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.574+0000 7f288d20e700 1 --2- 172.21.15.171:0/1742735883 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2874062ba0 0x7f2874065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:15.602 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.574+0000 7f287affd700 20 client.24319.objecter dump_active .. 0 homeless 2024-02-23T00:27:15.602 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.574+0000 7f287affd700 10 client.24319.objecter ms_handle_connect 0x7f2874062ba0 2024-02-23T00:27:15.602 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.574+0000 7f288d20e700 1 --2- 172.21.15.171:0/1742735883 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2874062ba0 0x7f2874065060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f2884009da0 tx=0x7f288400b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:15.602 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.578+0000 7f287affd700 1 -- 172.21.15.171:0/1742735883 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f287c049f60 con 0x7f28880f7a60 2024-02-23T00:27:15.660 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:27:15 smithi087 bash[64454]: 425454b0d35f42bddbd1dc0a24c556c04f8d07a2c69fa5017aa422f00bf6de2d 2024-02-23T00:27:15.660 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:27:15 smithi087 systemd[1]: Started Ceph rgw.foo.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:27:15.746 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.743+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f28880feee0 con 0x7f28880f7a60 2024-02-23T00:27:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:14.106688+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v189: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:15.567555+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:15.568642+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:15.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.994+0000 7f287affd700 1 -- 172.21.15.171:0/1742735883 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'iscsi' created v54) v1 ==== 154+0+0 (secure 0 0 0) 0x7f287c04c710 con 0x7f28880f7a60 2024-02-23T00:27:15.999 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 10 librados: watch_flush enter 2024-02-23T00:27:16.000 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 10 librados: watch_flush exit 2024-02-23T00:27:16.000 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 20 client.24319.objecter shutdown clearing up homeless session... 2024-02-23T00:27:16.000 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 10 client.24319.objecter successfully canceled tick 2024-02-23T00:27:16.000 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2874062ba0 msgr2=0x7f2874065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:16.000 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 --2- 172.21.15.171:0/1742735883 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2874062ba0 0x7f2874065060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f2884009da0 tx=0x7f288400b040).stop 2024-02-23T00:27:16.000 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28880f7a60 msgr2=0x7f2888108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:16.000 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 --2- 172.21.15.171:0/1742735883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28880f7a60 0x7f2888108040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f287c0048f0 tx=0x7f287c009cb0).stop 2024-02-23T00:27:16.000 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 shutdown_connections 2024-02-23T00:27:16.001 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 --2- 172.21.15.171:0/1742735883 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28880f9480 0x7f288810f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:16.001 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 --2- 172.21.15.171:0/1742735883 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28880f7a60 0x7f2888108040 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:16.001 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 --2- 172.21.15.171:0/1742735883 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2874062ba0 0x7f2874065060 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:16.001 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 --2- 172.21.15.171:0/1742735883 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28880f8370 0x7f2888108530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:16.001 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 >> 172.21.15.171:0/1742735883 conn(0x7f28880f32a0 msgr2=0x7f28880f55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:16.001 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 shutdown_connections 2024-02-23T00:27:16.001 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 -- 172.21.15.171:0/1742735883 wait complete. 2024-02-23T00:27:16.001 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:15.997+0000 7f288fc73700 1 librados: shutdown 2024-02-23T00:27:16.002 INFO:teuthology.orchestra.run.smithi171.stderr:pool 'iscsi' created 2024-02-23T00:27:16.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:14.106688+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v189: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:16.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:15.567555+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:16.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:15.568642+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:16.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:14.106688+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v189: 1 pgs: 1 peering; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:16.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:15.567555+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:16.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:15.568642+0000 mon.a (mon.0) 541 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:16.618 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool application enable iscsi rbd 2024-02-23T00:27:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:15.745870+0000 mon.b (mon.1) 23 : audit [INF] from='client.? 172.21.15.171:0/1742735883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-23T00:27:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:15.746223+0000 mon.a (mon.0) 542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-23T00:27:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:15.995751+0000 mon.a (mon.0) 543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-23T00:27:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:15.995841+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-23T00:27:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:16.000275+0000 mon.a (mon.0) 545 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-23T00:27:17.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:15.745870+0000 mon.b (mon.1) 23 : audit [INF] from='client.? 172.21.15.171:0/1742735883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-23T00:27:17.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:15.746223+0000 mon.a (mon.0) 542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-23T00:27:17.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:15.995751+0000 mon.a (mon.0) 543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-23T00:27:17.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:15.995841+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-23T00:27:17.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:16.000275+0000 mon.a (mon.0) 545 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-23T00:27:17.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:15.745870+0000 mon.b (mon.1) 23 : audit [INF] from='client.? 172.21.15.171:0/1742735883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-23T00:27:17.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:15.746223+0000 mon.a (mon.0) 542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-23T00:27:17.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:15.995751+0000 mon.a (mon.0) 543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-23T00:27:17.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:15.995841+0000 mon.a (mon.0) 544 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-23T00:27:17.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:16.000275+0000 mon.a (mon.0) 545 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-23T00:27:17.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:16.107170+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v191: 36 pgs: 35 unknown, 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:17.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:16.887319+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:17.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:16.888007+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:17.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:16.888612+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:17.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:16.990699+0000 mon.a (mon.0) 549 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-23T00:27:17.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:16.990723+0000 mon.a (mon.0) 550 : cluster [INF] Cluster is now healthy 2024-02-23T00:27:17.951 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:17.002077+0000 mon.a (mon.0) 551 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-23T00:27:17.951 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:17.002222+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-23T00:27:17.951 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:17.515089+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:18.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:16.107170+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v191: 36 pgs: 35 unknown, 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:18.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:16.887319+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:18.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:16.888007+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:18.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:16.888612+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:16.990699+0000 mon.a (mon.0) 549 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:16.990723+0000 mon.a (mon.0) 550 : cluster [INF] Cluster is now healthy 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:17.002077+0000 mon.a (mon.0) 551 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:17.002222+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:17.515089+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:16.107170+0000 mgr.y (mgr.14146) 203 : cluster [DBG] pgmap v191: 36 pgs: 35 unknown, 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:16.887319+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:16.888007+0000 mon.a (mon.0) 547 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:16.888612+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:16.990699+0000 mon.a (mon.0) 549 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-23T00:27:18.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:16.990723+0000 mon.a (mon.0) 550 : cluster [INF] Cluster is now healthy 2024-02-23T00:27:18.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:17.002077+0000 mon.a (mon.0) 551 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-23T00:27:18.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:17.002222+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-23T00:27:18.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:17.515089+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:18.606 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 -- 172.21.15.171:0/4150876669 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f8d20 msgr2=0x7fa0700f9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 --2- 172.21.15.171:0/4150876669 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f8d20 0x7fa0700f9180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa0640048a0 tx=0x7fa064009cb0).stop 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 -- 172.21.15.171:0/4150876669 shutdown_connections 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 --2- 172.21.15.171:0/4150876669 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa0700f9670 0x7fa0700fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 --2- 172.21.15.171:0/4150876669 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0700f7a80 0x7fa0700f7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 --2- 172.21.15.171:0/4150876669 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f8d20 0x7fa0700f9180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 -- 172.21.15.171:0/4150876669 >> 172.21.15.171:0/4150876669 conn(0x7fa0700f32a0 msgr2=0x7fa0700f56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 -- 172.21.15.171:0/4150876669 shutdown_connections 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 -- 172.21.15.171:0/4150876669 wait complete. 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 librados: starting msgr at 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 librados: starting objecter 2024-02-23T00:27:18.607 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 Processor -- start 2024-02-23T00:27:18.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 -- start start 2024-02-23T00:27:18.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 librados: setting wanted keys 2024-02-23T00:27:18.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.602+0000 7fa07644e700 1 librados: calling monclient init 2024-02-23T00:27:18.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0700f7a80 0x7fa070107f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:18.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa0700f8d20 0x7fa070108420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:18.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f9670 0x7fa07010f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:18.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07010f9f0 con 0x7fa0700f7a80 2024-02-23T00:27:18.608 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa07010fb30 con 0x7fa0700f9670 2024-02-23T00:27:18.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07010fc70 con 0x7fa0700f8d20 2024-02-23T00:27:18.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa0749eb700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f9670 0x7fa07010f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:18.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa0749eb700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f9670 0x7fa07010f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:55512/0 (socket says 172.21.15.171:55512) 2024-02-23T00:27:18.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa0749eb700 1 -- 172.21.15.171:0/3446228781 learned_addr learned my addr 172.21.15.171:0/3446228781 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:27:18.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa06f7fe700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa0700f8d20 0x7fa070108420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:18.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa06ffff700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0700f7a80 0x7fa070107f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:18.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa0749eb700 1 -- 172.21.15.171:0/3446228781 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa0700f8d20 msgr2=0x7fa070108420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:18.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa0749eb700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa0700f8d20 0x7fa070108420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:18.609 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa0749eb700 1 -- 172.21.15.171:0/3446228781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0700f7a80 msgr2=0x7fa070107f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:18.610 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa0749eb700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0700f7a80 0x7fa070107f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:18.610 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa0749eb700 1 -- 172.21.15.171:0/3446228781 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064005040 con 0x7fa0700f9670 2024-02-23T00:27:18.610 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa06ffff700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0700f7a80 0x7fa070107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:27:18.610 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa0749eb700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f9670 0x7fa07010f500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fa06800d970 tx=0x7fa068004e60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:18.610 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa06d7fa700 10 client.?.objecter ms_handle_connect 0x7fa0700f9670 2024-02-23T00:27:18.610 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa06d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:18.614 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa06d7fa700 1 -- 172.21.15.171:0/3446228781 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06800de60 con 0x7fa0700f9670 2024-02-23T00:27:18.614 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa06d7fa700 1 -- 172.21.15.171:0/3446228781 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa06800f040 con 0x7fa0700f9670 2024-02-23T00:27:18.614 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa06d7fa700 1 -- 172.21.15.171:0/3446228781 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0680036c0 con 0x7fa0700f9670 2024-02-23T00:27:18.614 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07010ff50 con 0x7fa0700f9670 2024-02-23T00:27:18.614 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 10 client.24346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:18.615 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa070110380 con 0x7fa0700f9670 2024-02-23T00:27:18.615 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.603+0000 7fa07644e700 1 librados: init done 2024-02-23T00:27:18.615 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.604+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa070102860 con 0x7fa0700f9670 2024-02-23T00:27:18.615 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.605+0000 7fa06d7fa700 1 -- 172.21.15.171:0/3446228781 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7fa068003ab0 con 0x7fa0700f9670 2024-02-23T00:27:18.615 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.605+0000 7fa06d7fa700 1 --2- 172.21.15.171:0/3446228781 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa058062ba0 0x7fa058065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:18.615 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.605+0000 7fa06d7fa700 1 -- 172.21.15.171:0/3446228781 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5111+0+0 (secure 0 0 0) 0x7fa06807c860 con 0x7fa0700f9670 2024-02-23T00:27:18.615 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.605+0000 7fa06d7fa700 10 client.24346.objecter ms_dispatch 0x7fa070001550 osd_map(56..56 src has 1..56) v4 2024-02-23T00:27:18.615 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.606+0000 7fa06d7fa700 3 client.24346.objecter handle_osd_map got epochs [56,56] > 0 2024-02-23T00:27:18.615 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.606+0000 7fa06d7fa700 3 client.24346.objecter handle_osd_map decoding full epoch 56 2024-02-23T00:27:18.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.606+0000 7fa06d7fa700 20 client.24346.objecter dump_active .. 0 homeless 2024-02-23T00:27:18.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.606+0000 7fa06ffff700 1 --2- 172.21.15.171:0/3446228781 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa058062ba0 0x7fa058065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:18.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.606+0000 7fa06d7fa700 10 client.24346.objecter ms_handle_connect 0x7fa058062ba0 2024-02-23T00:27:18.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.606+0000 7fa06ffff700 1 --2- 172.21.15.171:0/3446228781 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa058062ba0 0x7fa058065060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa0700f8b80 tx=0x7fa060009d80).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:18.616 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.609+0000 7fa06d7fa700 1 -- 172.21.15.171:0/3446228781 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa06804cf60 con 0x7fa0700f9670 2024-02-23T00:27:18.780 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:18.773+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"} v 0) v1 -- 0x7fa070028540 con 0x7fa0700f9670 2024-02-23T00:27:18.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:16.892561+0000 mgr.y (mgr.14146) 204 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-23T00:27:18.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:18.004763+0000 mon.a (mon.0) 554 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-23T00:27:18.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:18.005807+0000 mon.a (mon.0) 555 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-23T00:27:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:18.010706+0000 mon.c (mon.2) 11 : audit [INF] from='client.? 172.21.15.87:0/2178799724' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-23T00:27:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:18.011125+0000 mon.a (mon.0) 556 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-23T00:27:19.009 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.006+0000 7fa06d7fa700 1 -- 172.21.15.171:0/3446228781 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]=0 enabled application 'rbd' on pool 'iscsi' v57) v1 ==== 147+0+0 (secure 0 0 0) 0x7fa068021020 con 0x7fa0700f9670 2024-02-23T00:27:19.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.012+0000 7fa07644e700 10 librados: watch_flush enter 2024-02-23T00:27:19.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.012+0000 7fa07644e700 10 librados: watch_flush exit 2024-02-23T00:27:19.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.012+0000 7fa07644e700 20 client.24346.objecter shutdown clearing up homeless session... 2024-02-23T00:27:19.015 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.012+0000 7fa07644e700 10 client.24346.objecter successfully canceled tick 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.012+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa058062ba0 msgr2=0x7fa058065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.012+0000 7fa07644e700 1 --2- 172.21.15.171:0/3446228781 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa058062ba0 0x7fa058065060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa0700f8b80 tx=0x7fa060009d80).stop 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.012+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f9670 msgr2=0x7fa07010f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.012+0000 7fa07644e700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f9670 0x7fa07010f500 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fa06800d970 tx=0x7fa068004e60).stop 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.013+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 shutdown_connections 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.013+0000 7fa07644e700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa0700f9670 0x7fa07010f500 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.013+0000 7fa07644e700 1 --2- 172.21.15.171:0/3446228781 >> v2:172.21.15.87:6800/1121018082 conn(0x7fa058062ba0 0x7fa058065060 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.013+0000 7fa07644e700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa0700f7a80 0x7fa070107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.013+0000 7fa07644e700 1 --2- 172.21.15.171:0/3446228781 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa0700f8d20 0x7fa070108420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.013+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 >> 172.21.15.171:0/3446228781 conn(0x7fa0700f32a0 msgr2=0x7fa0700f66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.013+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 shutdown_connections 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.013+0000 7fa07644e700 1 -- 172.21.15.171:0/3446228781 wait complete. 2024-02-23T00:27:19.016 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:19.013+0000 7fa07644e700 1 librados: shutdown 2024-02-23T00:27:19.019 INFO:teuthology.orchestra.run.smithi171.stderr:enabled application 'rbd' on pool 'iscsi' 2024-02-23T00:27:19.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:16.892561+0000 mgr.y (mgr.14146) 204 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-23T00:27:19.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:18.004763+0000 mon.a (mon.0) 554 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-23T00:27:19.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:18.005807+0000 mon.a (mon.0) 555 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-23T00:27:19.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:18.010706+0000 mon.c (mon.2) 11 : audit [INF] from='client.? 172.21.15.87:0/2178799724' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-23T00:27:19.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:18.011125+0000 mon.a (mon.0) 556 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-23T00:27:19.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:16.892561+0000 mgr.y (mgr.14146) 204 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-23T00:27:19.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:18.004763+0000 mon.a (mon.0) 554 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-23T00:27:19.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:18.005807+0000 mon.a (mon.0) 555 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-23T00:27:19.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:18.010706+0000 mon.c (mon.2) 11 : audit [INF] from='client.? 172.21.15.87:0/2178799724' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-23T00:27:19.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:18.011125+0000 mon.a (mon.0) 556 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-23T00:27:19.675 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch apply iscsi iscsi user password --placement '1;smithi171=iscsi.a' 2024-02-23T00:27:19.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:18.107618+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v194: 68 pgs: 66 unknown, 2 active+clean; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:19.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:18.775215+0000 mon.b (mon.1) 24 : audit [INF] from='client.? 172.21.15.171:0/3446228781' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-23T00:27:19.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:18.775831+0000 mon.a (mon.0) 557 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-23T00:27:19.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:19.007821+0000 mon.a (mon.0) 558 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-23T00:27:19.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:19.008062+0000 mon.a (mon.0) 559 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-23T00:27:19.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:19.008169+0000 mon.a (mon.0) 560 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-23T00:27:19.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:19.008259+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-23T00:27:20.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:18.107618+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v194: 68 pgs: 66 unknown, 2 active+clean; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:20.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:18.775215+0000 mon.b (mon.1) 24 : audit [INF] from='client.? 172.21.15.171:0/3446228781' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-23T00:27:20.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:18.775831+0000 mon.a (mon.0) 557 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-23T00:27:20.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:19.007821+0000 mon.a (mon.0) 558 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-23T00:27:20.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:19.008062+0000 mon.a (mon.0) 559 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-23T00:27:20.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:19.008169+0000 mon.a (mon.0) 560 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-23T00:27:20.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:19.008259+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-23T00:27:20.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:18.107618+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v194: 68 pgs: 66 unknown, 2 active+clean; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:20.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:18.775215+0000 mon.b (mon.1) 24 : audit [INF] from='client.? 172.21.15.171:0/3446228781' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-23T00:27:20.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:18.775831+0000 mon.a (mon.0) 557 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-23T00:27:20.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:19.007821+0000 mon.a (mon.0) 558 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-23T00:27:20.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:19.008062+0000 mon.a (mon.0) 559 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-23T00:27:20.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:19.008169+0000 mon.a (mon.0) 560 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-23T00:27:20.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:19.008259+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-23T00:27:21.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:20.007379+0000 mon.a (mon.0) 562 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-23T00:27:21.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:20.009353+0000 mon.a (mon.0) 563 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-23T00:27:21.187 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:20.013717+0000 mon.c (mon.2) 12 : audit [INF] from='client.? 172.21.15.87:0/2178799724' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-23T00:27:21.187 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:20.014076+0000 mon.a (mon.0) 564 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-23T00:27:21.187 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:20.108017+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v197: 100 pgs: 67 unknown, 33 active+clean; 965 B data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-23T00:27:21.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:20.007379+0000 mon.a (mon.0) 562 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-23T00:27:21.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:20.009353+0000 mon.a (mon.0) 563 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-23T00:27:21.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:20.013717+0000 mon.c (mon.2) 12 : audit [INF] from='client.? 172.21.15.87:0/2178799724' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-23T00:27:21.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:20.014076+0000 mon.a (mon.0) 564 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-23T00:27:21.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:20.108017+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v197: 100 pgs: 67 unknown, 33 active+clean; 965 B data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-23T00:27:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:20.007379+0000 mon.a (mon.0) 562 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-23T00:27:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:20.009353+0000 mon.a (mon.0) 563 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-23T00:27:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:20.013717+0000 mon.c (mon.2) 12 : audit [INF] from='client.? 172.21.15.87:0/2178799724' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-23T00:27:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:20.014076+0000 mon.a (mon.0) 564 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-23T00:27:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:20.108017+0000 mgr.y (mgr.14146) 206 : cluster [DBG] pgmap v197: 100 pgs: 67 unknown, 33 active+clean; 965 B data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-23T00:27:21.684 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 -- 172.21.15.171:0/261101113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f8370 msgr2=0x7fe6400f8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:21.684 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 --2- 172.21.15.171:0/261101113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f8370 0x7fe6400f8da0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fe630004870 tx=0x7fe630009cb0).stop 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 -- 172.21.15.171:0/261101113 shutdown_connections 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 --2- 172.21.15.171:0/261101113 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6400f9480 0x7fe6400ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 --2- 172.21.15.171:0/261101113 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6400f7a60 0x7fe6400f7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 --2- 172.21.15.171:0/261101113 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f8370 0x7fe6400f8da0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 -- 172.21.15.171:0/261101113 >> 172.21.15.171:0/261101113 conn(0x7fe6400f32a0 msgr2=0x7fe6400f56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 -- 172.21.15.171:0/261101113 shutdown_connections 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 -- 172.21.15.171:0/261101113 wait complete. 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 librados: starting msgr at 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 librados: starting objecter 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 Processor -- start 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 -- start start 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 librados: setting wanted keys 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.677+0000 7fe64585e700 1 librados: calling monclient init 2024-02-23T00:27:21.685 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6400f7a60 0x7fe640107ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6400f8370 0x7fe6401084e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f9480 0x7fe64010f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64010fab0 con 0x7fe6400f8370 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64010fbf0 con 0x7fe6400f9480 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64010fd30 con 0x7fe6400f7a60 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63f7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f9480 0x7fe64010f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63f7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f9480 0x7fe64010f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:55522/0 (socket says 172.21.15.171:55522) 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63f7fe700 1 -- 172.21.15.171:0/124110720 learned_addr learned my addr 172.21.15.171:0/124110720 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63e7fc700 1 --2- 172.21.15.171:0/124110720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6400f8370 0x7fe6401084e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63effd700 1 --2- 172.21.15.171:0/124110720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6400f7a60 0x7fe640107ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63f7fe700 1 -- 172.21.15.171:0/124110720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6400f7a60 msgr2=0x7fe640107ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63f7fe700 1 --2- 172.21.15.171:0/124110720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6400f7a60 0x7fe640107ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63f7fe700 1 -- 172.21.15.171:0/124110720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6400f8370 msgr2=0x7fe6401084e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:21.686 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63f7fe700 1 --2- 172.21.15.171:0/124110720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6400f8370 0x7fe6401084e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63f7fe700 1 -- 172.21.15.171:0/124110720 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe630005040 con 0x7fe6400f9480 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe63f7fe700 1 --2- 172.21.15.171:0/124110720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f9480 0x7fe64010f5c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fe63800ec20 tx=0x7fe63800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe637fff700 10 client.?.objecter ms_handle_connect 0x7fe6400f9480 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe637fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe637fff700 1 -- 172.21.15.171:0/124110720 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638009950 con 0x7fe6400f9480 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe637fff700 1 -- 172.21.15.171:0/124110720 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe63800f040 con 0x7fe6400f9480 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64010ffb0 con 0x7fe6400f9480 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe637fff700 1 -- 172.21.15.171:0/124110720 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe638007df0 con 0x7fe6400f9480 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 10 client.24364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe640110460 con 0x7fe6400f9480 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.678+0000 7fe64585e700 1 librados: init done 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.679+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe640102970 con 0x7fe6400f9480 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe637fff700 1 -- 172.21.15.171:0/124110720 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7fe6380092e0 con 0x7fe6400f9480 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe637fff700 1 --2- 172.21.15.171:0/124110720 >> v2:172.21.15.87:6800/1121018082 conn(0x7fe62c062ba0 0x7fe62c065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:21.687 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe637fff700 1 -- 172.21.15.171:0/124110720 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fe63807c2c0 con 0x7fe6400f9480 2024-02-23T00:27:21.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe637fff700 10 client.24364.objecter ms_dispatch 0x7fe640001550 osd_map(59..59 src has 1..59) v4 2024-02-23T00:27:21.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe637fff700 3 client.24364.objecter handle_osd_map got epochs [59,59] > 0 2024-02-23T00:27:21.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe637fff700 3 client.24364.objecter handle_osd_map decoding full epoch 59 2024-02-23T00:27:21.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe637fff700 20 client.24364.objecter dump_active .. 0 homeless 2024-02-23T00:27:21.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe63effd700 1 --2- 172.21.15.171:0/124110720 >> v2:172.21.15.87:6800/1121018082 conn(0x7fe62c062ba0 0x7fe62c065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:21.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe637fff700 10 client.24364.objecter ms_handle_connect 0x7fe62c062ba0 2024-02-23T00:27:21.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.680+0000 7fe63effd700 1 --2- 172.21.15.171:0/124110720 >> v2:172.21.15.87:6800/1121018082 conn(0x7fe62c062ba0 0x7fe62c065060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe628004e60 tx=0x7fe62800b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:21.688 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.684+0000 7fe637fff700 1 -- 172.21.15.171:0/124110720 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe638049f60 con 0x7fe6400f9480 2024-02-23T00:27:21.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.855+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi171=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fe640059120 con 0x7fe62c062ba0 2024-02-23T00:27:21.867 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.865+0000 7fe637fff700 1 -- 172.21.15.171:0/124110720 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fe640059120 con 0x7fe62c062ba0 2024-02-23T00:27:21.867 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled iscsi.iscsi update... 2024-02-23T00:27:21.869 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.866+0000 7fe64585e700 10 librados: watch_flush enter 2024-02-23T00:27:21.869 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 10 librados: watch_flush exit 2024-02-23T00:27:21.869 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 20 client.24364.objecter shutdown clearing up homeless session... 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 10 client.24364.objecter successfully canceled tick 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 >> v2:172.21.15.87:6800/1121018082 conn(0x7fe62c062ba0 msgr2=0x7fe62c065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 --2- 172.21.15.171:0/124110720 >> v2:172.21.15.87:6800/1121018082 conn(0x7fe62c062ba0 0x7fe62c065060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe628004e60 tx=0x7fe62800b040).stop 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f9480 msgr2=0x7fe64010f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 --2- 172.21.15.171:0/124110720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f9480 0x7fe64010f5c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fe63800ec20 tx=0x7fe63800b040).stop 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 shutdown_connections 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 --2- 172.21.15.171:0/124110720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6400f9480 0x7fe64010f5c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 --2- 172.21.15.171:0/124110720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6400f7a60 0x7fe640107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 --2- 172.21.15.171:0/124110720 >> v2:172.21.15.87:6800/1121018082 conn(0x7fe62c062ba0 0x7fe62c065060 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 --2- 172.21.15.171:0/124110720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6400f8370 0x7fe6401084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 >> 172.21.15.171:0/124110720 conn(0x7fe6400f32a0 msgr2=0x7fe6400f55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 shutdown_connections 2024-02-23T00:27:21.871 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 -- 172.21.15.171:0/124110720 wait complete. 2024-02-23T00:27:21.871 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:21.867+0000 7fe64585e700 1 librados: shutdown 2024-02-23T00:27:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:21.013247+0000 mon.a (mon.0) 565 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-23T00:27:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:21.013483+0000 mon.a (mon.0) 566 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-23T00:27:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:21.013579+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-23T00:27:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:21.866142+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:22.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:21.013247+0000 mon.a (mon.0) 565 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-23T00:27:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:21.013483+0000 mon.a (mon.0) 566 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-23T00:27:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:21.013579+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-23T00:27:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:21.866142+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:22.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:21.013247+0000 mon.a (mon.0) 565 : audit [INF] from='client.? 172.21.15.87:0/3823246443' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-23T00:27:22.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:21.013483+0000 mon.a (mon.0) 566 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-23T00:27:22.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:21.013579+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-23T00:27:22.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:21.866142+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:22.496 DEBUG:teuthology.orchestra.run.smithi171:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@iscsi.iscsi.a.service 2024-02-23T00:27:22.499 INFO:tasks.cephadm:Adding prometheus.a on smithi171 2024-02-23T00:27:22.499 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch apply prometheus '1;smithi171=a' 2024-02-23T00:27:22.528 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:21.857762+0000 mgr.y (mgr.14146) 207 : audit [DBG] from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi171=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:21.859685+0000 mgr.y (mgr.14146) 208 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi171=iscsi.a;count:1 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:22.015175+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:22.017146+0000 mon.b (mon.1) 25 : audit [INF] from='client.? 172.21.15.87:0/4203450800' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:22.024485+0000 mon.c (mon.2) 13 : audit [INF] from='client.? 172.21.15.87:0/707689632' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:22.025127+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:22.025913+0000 mon.a (mon.0) 571 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:22.108402+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v200: 132 pgs: 12 creating+peering, 59 active+clean, 61 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:23.017927+0000 mon.a (mon.0) 572 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:23.018093+0000 mon.a (mon.0) 573 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-23T00:27:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:23.018156+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:21.857762+0000 mgr.y (mgr.14146) 207 : audit [DBG] from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi171=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:21.859685+0000 mgr.y (mgr.14146) 208 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi171=iscsi.a;count:1 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:22.015175+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:22.017146+0000 mon.b (mon.1) 25 : audit [INF] from='client.? 172.21.15.87:0/4203450800' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:22.024485+0000 mon.c (mon.2) 13 : audit [INF] from='client.? 172.21.15.87:0/707689632' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:22.025127+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:22.025913+0000 mon.a (mon.0) 571 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:22.108402+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v200: 132 pgs: 12 creating+peering, 59 active+clean, 61 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:23.017927+0000 mon.a (mon.0) 572 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:23.018093+0000 mon.a (mon.0) 573 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-23T00:27:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:23.018156+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-23T00:27:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:21.857762+0000 mgr.y (mgr.14146) 207 : audit [DBG] from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi171=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:21.859685+0000 mgr.y (mgr.14146) 208 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi171=iscsi.a;count:1 2024-02-23T00:27:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:22.015175+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-23T00:27:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:22.017146+0000 mon.b (mon.1) 25 : audit [INF] from='client.? 172.21.15.87:0/4203450800' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:22.024485+0000 mon.c (mon.2) 13 : audit [INF] from='client.? 172.21.15.87:0/707689632' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:22.025127+0000 mon.a (mon.0) 570 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:22.025913+0000 mon.a (mon.0) 571 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-23T00:27:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:22.108402+0000 mgr.y (mgr.14146) 209 : cluster [DBG] pgmap v200: 132 pgs: 12 creating+peering, 59 active+clean, 61 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-23T00:27:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:23.017927+0000 mon.a (mon.0) 572 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-23T00:27:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:23.018093+0000 mon.a (mon.0) 573 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-23T00:27:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:23.018156+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-23T00:27:24.086 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:23.027296+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.87:0/4203450800' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.086 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:23.029889+0000 mon.c (mon.2) 14 : audit [INF] from='client.? 172.21.15.87:0/707689632' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.086 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:23.035958+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.087 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:23.038721+0000 mon.a (mon.0) 576 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.087 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.019276+0000 mon.a (mon.0) 577 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-23T00:27:24.087 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.019402+0000 mon.a (mon.0) 578 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-23T00:27:24.087 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:24.019470+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-23T00:27:24.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:23.027296+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.87:0/4203450800' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:23.029889+0000 mon.c (mon.2) 14 : audit [INF] from='client.? 172.21.15.87:0/707689632' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:23.035958+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:23.038721+0000 mon.a (mon.0) 576 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.019276+0000 mon.a (mon.0) 577 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.019402+0000 mon.a (mon.0) 578 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:24.019470+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:23.027296+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.87:0/4203450800' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:23.029889+0000 mon.c (mon.2) 14 : audit [INF] from='client.? 172.21.15.87:0/707689632' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:23.035958+0000 mon.a (mon.0) 575 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:23.038721+0000 mon.a (mon.0) 576 : audit [INF] from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.019276+0000 mon.a (mon.0) 577 : audit [INF] from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.019402+0000 mon.a (mon.0) 578 : audit [INF] from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-23T00:27:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:24.019470+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-23T00:27:24.623 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.620+0000 7f28846a0700 1 -- 172.21.15.171:0/3259788057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f6990 msgr2=0x7f287c0f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.620+0000 7f28846a0700 1 --2- 172.21.15.171:0/3259788057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f6990 0x7f287c0f6db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f28700048a0 tx=0x7f2870009cb0).stop 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.620+0000 7f28846a0700 1 -- 172.21.15.171:0/3259788057 shutdown_connections 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.620+0000 7f28846a0700 1 --2- 172.21.15.171:0/3259788057 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f287c0f8580 0x7f287c0fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.620+0000 7f28846a0700 1 --2- 172.21.15.171:0/3259788057 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f6990 0x7f287c0f6db0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.620+0000 7f28846a0700 1 --2- 172.21.15.171:0/3259788057 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f287c0f7c30 0x7f287c0f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.620+0000 7f28846a0700 1 -- 172.21.15.171:0/3259788057 >> 172.21.15.171:0/3259788057 conn(0x7f287c0f2190 msgr2=0x7f287c0f45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 -- 172.21.15.171:0/3259788057 shutdown_connections 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 -- 172.21.15.171:0/3259788057 wait complete. 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 librados: starting msgr at 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 librados: starting objecter 2024-02-23T00:27:24.624 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 Processor -- start 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 -- start start 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 librados: setting wanted keys 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 librados: calling monclient init 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f287c0f6990 0x7f287c108200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f7c30 0x7f287c1086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f287c0f8580 0x7f287c10f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c10fcc0 con 0x7f287c0f8580 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c10fe00 con 0x7f287c0f7c30 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f28846a0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c10ff40 con 0x7f287c0f6990 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f2881c3b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f7c30 0x7f287c1086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f2881c3b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f7c30 0x7f287c1086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:47750/0 (socket says 172.21.15.171:47750) 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f288243c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f287c0f6990 0x7f287c108200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:24.625 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.621+0000 7f2881c3b700 1 -- 172.21.15.171:0/2163205775 learned_addr learned my addr 172.21.15.171:0/2163205775 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:27:24.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f2882c3d700 1 --2- 172.21.15.171:0/2163205775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f287c0f8580 0x7f287c10f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:24.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f2881c3b700 1 -- 172.21.15.171:0/2163205775 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f287c0f6990 msgr2=0x7f287c108200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:24.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f2881c3b700 1 --2- 172.21.15.171:0/2163205775 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f287c0f6990 0x7f287c108200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:24.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f2881c3b700 1 -- 172.21.15.171:0/2163205775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f287c0f8580 msgr2=0x7f287c10f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:24.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f2881c3b700 1 --2- 172.21.15.171:0/2163205775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f287c0f8580 0x7f287c10f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:24.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f2881c3b700 1 -- 172.21.15.171:0/2163205775 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2870005040 con 0x7f287c0f7c30 2024-02-23T00:27:24.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f2881c3b700 1 --2- 172.21.15.171:0/2163205775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f7c30 0x7f287c1086f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f287800c810 tx=0x7f2878004330).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:24.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f286f7fe700 10 client.?.objecter ms_handle_connect 0x7f287c0f7c30 2024-02-23T00:27:24.626 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f286f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:24.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f286f7fe700 1 -- 172.21.15.171:0/2163205775 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28780043a0 con 0x7f287c0f7c30 2024-02-23T00:27:24.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f286f7fe700 1 -- 172.21.15.171:0/2163205775 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f287800e970 con 0x7f287c0f7c30 2024-02-23T00:27:24.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f286f7fe700 1 -- 172.21.15.171:0/2163205775 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878012990 con 0x7f287c0f7c30 2024-02-23T00:27:24.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287c0f8eb0 con 0x7f287c0f7c30 2024-02-23T00:27:24.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f28846a0700 10 client.24388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:24.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287c0f93f0 con 0x7f287c0f7c30 2024-02-23T00:27:24.627 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.622+0000 7f28846a0700 1 librados: init done 2024-02-23T00:27:24.628 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.623+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287c05f360 con 0x7f287c0f7c30 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.623+0000 7f286f7fe700 1 -- 172.21.15.171:0/2163205775 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 14) v1 ==== 77029+0+0 (secure 0 0 0) 0x7f287800eae0 con 0x7f287c0f7c30 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.624+0000 7f286f7fe700 1 --2- 172.21.15.171:0/2163205775 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2868062ba0 0x7f2868065060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.624+0000 7f286f7fe700 1 -- 172.21.15.171:0/2163205775 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5884+0+0 (secure 0 0 0) 0x7f287807bf40 con 0x7f287c0f7c30 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.624+0000 7f286f7fe700 10 client.24388.objecter ms_dispatch 0x7f287c001550 osd_map(62..62 src has 1..62) v4 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.624+0000 7f286f7fe700 3 client.24388.objecter handle_osd_map got epochs [62,62] > 0 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.624+0000 7f286f7fe700 3 client.24388.objecter handle_osd_map decoding full epoch 62 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.624+0000 7f286f7fe700 20 client.24388.objecter dump_active .. 0 homeless 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.624+0000 7f288243c700 1 --2- 172.21.15.171:0/2163205775 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2868062ba0 0x7f2868065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.624+0000 7f286f7fe700 10 client.24388.objecter ms_handle_connect 0x7f2868062ba0 2024-02-23T00:27:24.629 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.624+0000 7f288243c700 1 --2- 172.21.15.171:0/2163205775 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2868062ba0 0x7f2868065060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2870009da0 tx=0x7f2870009cb0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:24.630 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.628+0000 7f286f7fe700 1 -- 172.21.15.171:0/2163205775 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2878048f60 con 0x7f287c0f7c30 2024-02-23T00:27:24.820 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.818+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi171=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f287c0fe080 con 0x7f2868062ba0 2024-02-23T00:27:24.826 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.824+0000 7f286f7fe700 1 -- 172.21.15.171:0/2163205775 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f287c0fe080 con 0x7f2868062ba0 2024-02-23T00:27:24.826 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled prometheus update... 2024-02-23T00:27:24.827 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 10 librados: watch_flush enter 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 10 librados: watch_flush exit 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 20 client.24388.objecter shutdown clearing up homeless session... 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 10 client.24388.objecter successfully canceled tick 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2868062ba0 msgr2=0x7f2868065060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 --2- 172.21.15.171:0/2163205775 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2868062ba0 0x7f2868065060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2870009da0 tx=0x7f2870009cb0).stop 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f7c30 msgr2=0x7f287c1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 --2- 172.21.15.171:0/2163205775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f7c30 0x7f287c1086f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f287800c810 tx=0x7f2878004330).stop 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 shutdown_connections 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 --2- 172.21.15.171:0/2163205775 >> v2:172.21.15.87:6800/1121018082 conn(0x7f2868062ba0 0x7f2868065060 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 --2- 172.21.15.171:0/2163205775 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f287c0f8580 0x7f287c10f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 --2- 172.21.15.171:0/2163205775 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f287c0f6990 0x7f287c108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 --2- 172.21.15.171:0/2163205775 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f287c0f7c30 0x7f287c1086f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:24.828 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 >> 172.21.15.171:0/2163205775 conn(0x7f287c0f2190 msgr2=0x7f287c061200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:24.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 shutdown_connections 2024-02-23T00:27:24.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 -- 172.21.15.171:0/2163205775 wait complete. 2024-02-23T00:27:24.829 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:24.826+0000 7f28846a0700 1 librados: shutdown 2024-02-23T00:27:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:23.618837+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:23.618923+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:24.108890+0000 mgr.y (mgr.14146) 210 : cluster [DBG] pgmap v203: 132 pgs: 17 creating+peering, 72 active+clean, 43 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s 2024-02-23T00:27:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.197930+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.205027+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.211189+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:24.218125+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-23T00:27:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.230247+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-23T00:27:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.234219+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-23T00:27:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.243600+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:24.244560+0000 mgr.y (mgr.14146) 212 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi171 2024-02-23T00:27:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:24.499441+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:24.499491+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.824964+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:23.618837+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:23.618923+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:24.108890+0000 mgr.y (mgr.14146) 210 : cluster [DBG] pgmap v203: 132 pgs: 17 creating+peering, 72 active+clean, 43 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.197930+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.205027+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.211189+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:24.218125+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.230247+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.234219+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.243600+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:24.244560+0000 mgr.y (mgr.14146) 212 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi171 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:24.499441+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:24.499491+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.824964+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:23.618837+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:23.618923+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:24.108890+0000 mgr.y (mgr.14146) 210 : cluster [DBG] pgmap v203: 132 pgs: 17 creating+peering, 72 active+clean, 43 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.197930+0000 mon.a (mon.0) 580 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.205027+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.211189+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:24.218125+0000 mgr.y (mgr.14146) 211 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.230247+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-23T00:27:25.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.234219+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-23T00:27:25.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.243600+0000 mon.a (mon.0) 585 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:25.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:24.244560+0000 mgr.y (mgr.14146) 212 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi171 2024-02-23T00:27:25.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:24.499441+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:25.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:24.499491+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:25.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.824964+0000 mon.a (mon.0) 586 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:26.033 DEBUG:teuthology.orchestra.run.smithi171:prometheus.a> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@prometheus.a.service 2024-02-23T00:27:26.036 INFO:tasks.cephadm:Adding node-exporter.a on smithi087 2024-02-23T00:27:26.036 INFO:tasks.cephadm:Adding node-exporter.b on smithi171 2024-02-23T00:27:26.036 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch apply node-exporter '2;smithi087=a;smithi171=b' 2024-02-23T00:27:26.067 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:27:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:24.820270+0000 mgr.y (mgr.14146) 213 : audit [DBG] from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi171=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:24.821724+0000 mgr.y (mgr.14146) 214 : cephadm [INF] Saving service prometheus spec with placement smithi171=a;count:1 2024-02-23T00:27:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:25.239698+0000 mon.a (mon.0) 587 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-23T00:27:26.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:24.820270+0000 mgr.y (mgr.14146) 213 : audit [DBG] from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi171=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:24.821724+0000 mgr.y (mgr.14146) 214 : cephadm [INF] Saving service prometheus spec with placement smithi171=a;count:1 2024-02-23T00:27:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:25.239698+0000 mon.a (mon.0) 587 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-23T00:27:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:24.820270+0000 mgr.y (mgr.14146) 213 : audit [DBG] from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi171=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:24.821724+0000 mgr.y (mgr.14146) 214 : cephadm [INF] Saving service prometheus spec with placement smithi171=a;count:1 2024-02-23T00:27:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:25.239698+0000 mon.a (mon.0) 587 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-23T00:27:26.717 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:26 smithi171 systemd[1]: Starting Ceph iscsi.iscsi.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:27:27.375 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:26.109444+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v205: 132 pgs: 5 creating+peering, 127 active+clean; 4.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 115 KiB/s rd, 6.8 KiB/s wr, 256 op/s 2024-02-23T00:27:27.375 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:26.248407+0000 mon.a (mon.0) 588 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-02-23T00:27:27.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:26.109444+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v205: 132 pgs: 5 creating+peering, 127 active+clean; 4.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 115 KiB/s rd, 6.8 KiB/s wr, 256 op/s 2024-02-23T00:27:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:26.248407+0000 mon.a (mon.0) 588 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-02-23T00:27:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:26.109444+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v205: 132 pgs: 5 creating+peering, 127 active+clean; 4.4 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 115 KiB/s rd, 6.8 KiB/s wr, 256 op/s 2024-02-23T00:27:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:26.248407+0000 mon.a (mon.0) 588 : cluster [DBG] mgrmap e15: y(active, since 5m), standbys: x 2024-02-23T00:27:27.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:27 smithi171 bash[60572]: log file path now is '/var/log/tcmu-runner.log' 2024-02-23T00:27:28.851 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 -- 172.21.15.171:0/2680935477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 msgr2=0x7f48840fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:28.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 --2- 172.21.15.171:0/2680935477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 0x7f48840fff70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f487c009c10 tx=0x7f487c004d10).stop 2024-02-23T00:27:28.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 -- 172.21.15.171:0/2680935477 shutdown_connections 2024-02-23T00:27:28.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 --2- 172.21.15.171:0/2680935477 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 0x7f48840fff70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:28.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 --2- 172.21.15.171:0/2680935477 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f48840f7a80 0x7f48840f7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:28.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 --2- 172.21.15.171:0/2680935477 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48840f8d20 0x7f48840f9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:28.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 -- 172.21.15.171:0/2680935477 >> 172.21.15.171:0/2680935477 conn(0x7f48840f32a0 msgr2=0x7f48840f56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:28.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 -- 172.21.15.171:0/2680935477 shutdown_connections 2024-02-23T00:27:28.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 -- 172.21.15.171:0/2680935477 wait complete. 2024-02-23T00:27:28.852 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 librados: starting msgr at 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 librados: starting objecter 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.789+0000 7f4889f56700 1 Processor -- start 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 -- start start 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 librados: setting wanted keys 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 librados: calling monclient init 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48840f7a80 0x7f488406c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f48840f8d20 0x7f488406b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 0x7f488406b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488406bb40 con 0x7f48840f7a80 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488406bcb0 con 0x7f48840f9670 2024-02-23T00:27:28.853 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488406be70 con 0x7f48840f8d20 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4883fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 0x7f488406b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4883fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 0x7f488406b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:47764/0 (socket says 172.21.15.171:47764) 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4883fff700 1 -- 172.21.15.171:0/931779759 learned_addr learned my addr 172.21.15.171:0/931779759 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f48837fe700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48840f7a80 0x7f488406c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4882ffd700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f48840f8d20 0x7f488406b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4883fff700 1 -- 172.21.15.171:0/931779759 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f48840f8d20 msgr2=0x7f488406b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4883fff700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f48840f8d20 0x7f488406b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4883fff700 1 -- 172.21.15.171:0/931779759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48840f7a80 msgr2=0x7f488406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4883fff700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48840f7a80 0x7f488406c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4883fff700 1 -- 172.21.15.171:0/931779759 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487c0097c0 con 0x7f48840f9670 2024-02-23T00:27:28.854 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f48837fe700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48840f7a80 0x7f488406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4883fff700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 0x7f488406b520 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f487c004750 tx=0x7f487c004d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4880ff9700 10 client.?.objecter ms_handle_connect 0x7f48840f9670 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4880ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4880ff9700 1 -- 172.21.15.171:0/931779759 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c00e040 con 0x7f48840f9670 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4884111ff0 con 0x7f48840f9670 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4880ff9700 1 -- 172.21.15.171:0/931779759 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f487c018040 con 0x7f48840f9670 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.790+0000 7f4880ff9700 1 -- 172.21.15.171:0/931779759 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c01d690 con 0x7f48840f9670 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.791+0000 7f4889f56700 10 client.24394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.791+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4884112420 con 0x7f48840f9670 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.791+0000 7f4889f56700 1 librados: init done 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.791+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488405f360 con 0x7f48840f9670 2024-02-23T00:27:28.855 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.792+0000 7f4880ff9700 1 -- 172.21.15.171:0/931779759 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 15) v1 ==== 77080+0+0 (secure 0 0 0) 0x7f487c013420 con 0x7f48840f9670 2024-02-23T00:27:28.856 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.792+0000 7f4880ff9700 1 --2- 172.21.15.171:0/931779759 >> v2:172.21.15.87:6800/1121018082 conn(0x7f486c062c20 0x7f486c0650e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:28.856 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.792+0000 7f4880ff9700 1 -- 172.21.15.171:0/931779759 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5884+0+0 (secure 0 0 0) 0x7f487c07e5b0 con 0x7f48840f9670 2024-02-23T00:27:28.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.792+0000 7f4880ff9700 10 client.24394.objecter ms_dispatch 0x7f4884001550 osd_map(63..63 src has 1..63) v4 2024-02-23T00:27:28.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.792+0000 7f4880ff9700 3 client.24394.objecter handle_osd_map got epochs [63,63] > 0 2024-02-23T00:27:28.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.792+0000 7f4880ff9700 3 client.24394.objecter handle_osd_map decoding full epoch 63 2024-02-23T00:27:28.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.793+0000 7f48837fe700 1 --2- 172.21.15.171:0/931779759 >> v2:172.21.15.87:6800/1121018082 conn(0x7f486c062c20 0x7f486c0650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:28.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.793+0000 7f4880ff9700 20 client.24394.objecter dump_active .. 0 homeless 2024-02-23T00:27:28.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.793+0000 7f4880ff9700 10 client.24394.objecter ms_handle_connect 0x7f486c062c20 2024-02-23T00:27:28.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.793+0000 7f48837fe700 1 --2- 172.21.15.171:0/931779759 >> v2:172.21.15.87:6800/1121018082 conn(0x7f486c062c20 0x7f486c0650e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f4874007fd0 tx=0x7f487400b040).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:28.857 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.797+0000 7f4880ff9700 1 -- 172.21.15.171:0/931779759 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f487c04e800 con 0x7f48840f9670 2024-02-23T00:27:28.982 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.981+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi087=a;smithi171=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f4884059120 con 0x7f486c062c20 2024-02-23T00:27:28.989 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.988+0000 7f4880ff9700 1 -- 172.21.15.171:0/931779759 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f4884059120 con 0x7f486c062c20 2024-02-23T00:27:28.989 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled node-exporter update... 2024-02-23T00:27:28.991 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.990+0000 7f4889f56700 10 librados: watch_flush enter 2024-02-23T00:27:28.991 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.990+0000 7f4889f56700 10 librados: watch_flush exit 2024-02-23T00:27:28.991 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.990+0000 7f4889f56700 20 client.24394.objecter shutdown clearing up homeless session... 2024-02-23T00:27:28.991 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.990+0000 7f4889f56700 10 client.24394.objecter successfully canceled tick 2024-02-23T00:27:28.991 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.990+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 >> v2:172.21.15.87:6800/1121018082 conn(0x7f486c062c20 msgr2=0x7f486c0650e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:28.992 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.990+0000 7f4889f56700 1 --2- 172.21.15.171:0/931779759 >> v2:172.21.15.87:6800/1121018082 conn(0x7f486c062c20 0x7f486c0650e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f4874007fd0 tx=0x7f487400b040).stop 2024-02-23T00:27:28.992 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.990+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 msgr2=0x7f488406b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:28.992 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.990+0000 7f4889f56700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 0x7f488406b520 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f487c004750 tx=0x7f487c004d10).stop 2024-02-23T00:27:28.993 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.992+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 shutdown_connections 2024-02-23T00:27:28.993 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.992+0000 7f4889f56700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f48840f9670 0x7f488406b520 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:28.993 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.992+0000 7f4889f56700 1 --2- 172.21.15.171:0/931779759 >> v2:172.21.15.87:6800/1121018082 conn(0x7f486c062c20 0x7f486c0650e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:28.993 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.992+0000 7f4889f56700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f48840f7a80 0x7f488406c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:28.993 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.992+0000 7f4889f56700 1 --2- 172.21.15.171:0/931779759 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f48840f8d20 0x7f488406b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:28.993 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.992+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 >> 172.21.15.171:0/931779759 conn(0x7f48840f32a0 msgr2=0x7f48840fbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:28.993 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.992+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 shutdown_connections 2024-02-23T00:27:28.993 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.992+0000 7f4889f56700 1 -- 172.21.15.171:0/931779759 wait complete. 2024-02-23T00:27:28.993 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:28.992+0000 7f4889f56700 1 librados: shutdown 2024-02-23T00:27:29.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:28 smithi171 bash[60699]: 1ae55962627b58601b43a6ed20982216890b4b038f500848eb177ad5973ac958 2024-02-23T00:27:29.251 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:28 smithi171 systemd[1]: Started Ceph iscsi.iscsi.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:27:29.251 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug Started the configuration object watcher 2024-02-23T00:27:29.251 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug Checking for config object changes every 1s 2024-02-23T00:27:29.251 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug Processing osd blocklist entries for this node 2024-02-23T00:27:29.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:28.110052+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v206: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 105 KiB/s rd, 6.0 KiB/s wr, 235 op/s 2024-02-23T00:27:29.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:28.988323+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:29.018099+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:29.019456+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:28.110052+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v206: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 105 KiB/s rd, 6.0 KiB/s wr, 235 op/s 2024-02-23T00:27:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:28.988323+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:29.018099+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:29.019456+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:29.628 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:28.110052+0000 mgr.y (mgr.14146) 216 : cluster [DBG] pgmap v206: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 105 KiB/s rd, 6.0 KiB/s wr, 235 op/s 2024-02-23T00:27:29.628 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:28.988323+0000 mon.a (mon.0) 589 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:29.628 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:29.018099+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:29.628 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:29.019456+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:29.628 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug Reading the configuration object to update local LIO configuration 2024-02-23T00:27:29.628 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug Configuration does not have an entry for this host(smithi171.front.sepia.ceph.com) - nothing to define to LIO 2024-02-23T00:27:29.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-23T00:27:29.652 DEBUG:teuthology.orchestra.run.smithi087:node-exporter.a> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@node-exporter.a.service 2024-02-23T00:27:29.656 DEBUG:teuthology.orchestra.run.smithi171:node-exporter.b> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@node-exporter.b.service 2024-02-23T00:27:29.658 INFO:tasks.cephadm:Adding alertmanager.a on smithi087 2024-02-23T00:27:29.658 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch apply alertmanager '1;smithi087=a' 2024-02-23T00:27:29.687 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:27:29.688 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:27:30.364 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:28.982815+0000 mgr.y (mgr.14146) 217 : audit [DBG] from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi087=a;smithi171=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:30.365 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:28.984305+0000 mgr.y (mgr.14146) 218 : cephadm [INF] Saving service node-exporter spec with placement smithi087=a;smithi171=b;count:2 2024-02-23T00:27:30.365 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:29.357928+0000 mon.b (mon.1) 27 : audit [DBG] from='client.? 172.21.15.171:0/3328903024' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-23T00:27:30.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:28.982815+0000 mgr.y (mgr.14146) 217 : audit [DBG] from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi087=a;smithi171=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:28.984305+0000 mgr.y (mgr.14146) 218 : cephadm [INF] Saving service node-exporter spec with placement smithi087=a;smithi171=b;count:2 2024-02-23T00:27:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:29.357928+0000 mon.b (mon.1) 27 : audit [DBG] from='client.? 172.21.15.171:0/3328903024' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-23T00:27:30.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:28.982815+0000 mgr.y (mgr.14146) 217 : audit [DBG] from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi087=a;smithi171=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:30.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:28.984305+0000 mgr.y (mgr.14146) 218 : cephadm [INF] Saving service node-exporter spec with placement smithi087=a;smithi171=b;count:2 2024-02-23T00:27:30.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:29.357928+0000 mon.b (mon.1) 27 : audit [DBG] from='client.? 172.21.15.171:0/3328903024' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-23T00:27:31.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:30.110659+0000 mgr.y (mgr.14146) 219 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 5.2 KiB/s wr, 214 op/s 2024-02-23T00:27:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:30.110659+0000 mgr.y (mgr.14146) 219 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 5.2 KiB/s wr, 214 op/s 2024-02-23T00:27:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:30.110659+0000 mgr.y (mgr.14146) 219 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 5.2 KiB/s wr, 214 op/s 2024-02-23T00:27:32.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.021+0000 7f1166516700 1 -- 172.21.15.171:0/3584125280 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f7c30 msgr2=0x7f11600f8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:32.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.021+0000 7f1166516700 1 --2- 172.21.15.171:0/3584125280 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f7c30 0x7f11600f8090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f11540048a0 tx=0x7f1154009cb0).stop 2024-02-23T00:27:32.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.021+0000 7f1166516700 1 -- 172.21.15.171:0/3584125280 shutdown_connections 2024-02-23T00:27:32.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.021+0000 7f1166516700 1 --2- 172.21.15.171:0/3584125280 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11600f8580 0x7f11600fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:32.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.021+0000 7f1166516700 1 --2- 172.21.15.171:0/3584125280 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11600f6990 0x7f11600f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:32.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.021+0000 7f1166516700 1 --2- 172.21.15.171:0/3584125280 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f7c30 0x7f11600f8090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:32.023 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.021+0000 7f1166516700 1 -- 172.21.15.171:0/3584125280 >> 172.21.15.171:0/3584125280 conn(0x7f11600f2190 msgr2=0x7f11600f45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:32.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.021+0000 7f1166516700 1 -- 172.21.15.171:0/3584125280 shutdown_connections 2024-02-23T00:27:32.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.021+0000 7f1166516700 1 -- 172.21.15.171:0/3584125280 wait complete. 2024-02-23T00:27:32.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 librados: starting msgr at 2024-02-23T00:27:32.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 librados: starting objecter 2024-02-23T00:27:32.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 Processor -- start 2024-02-23T00:27:32.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 -- start start 2024-02-23T00:27:32.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 librados: setting wanted keys 2024-02-23T00:27:32.024 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 librados: calling monclient init 2024-02-23T00:27:32.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f6990 0x7f1160106e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:32.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11600f7c30 0x7f1160107360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:32.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11600f8580 0x7f116010e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:32.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116010e930 con 0x7f11600f7c30 2024-02-23T00:27:32.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116010ea70 con 0x7f11600f6990 2024-02-23T00:27:32.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f1166516700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116010ebb0 con 0x7f11600f8580 2024-02-23T00:27:32.027 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f115ffff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f6990 0x7f1160106e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f115ffff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f6990 0x7f1160106e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:47838/0 (socket says 172.21.15.171:47838) 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.022+0000 7f115ffff700 1 -- 172.21.15.171:0/3634672831 learned_addr learned my addr 172.21.15.171:0/3634672831 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115f7fe700 1 --2- 172.21.15.171:0/3634672831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11600f7c30 0x7f1160107360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f1164ab3700 1 --2- 172.21.15.171:0/3634672831 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11600f8580 0x7f116010e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115ffff700 1 -- 172.21.15.171:0/3634672831 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11600f8580 msgr2=0x7f116010e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115ffff700 1 --2- 172.21.15.171:0/3634672831 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11600f8580 0x7f116010e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115ffff700 1 -- 172.21.15.171:0/3634672831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11600f7c30 msgr2=0x7f1160107360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115ffff700 1 --2- 172.21.15.171:0/3634672831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11600f7c30 0x7f1160107360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115ffff700 1 -- 172.21.15.171:0/3634672831 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1154005040 con 0x7f11600f6990 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115ffff700 1 --2- 172.21.15.171:0/3634672831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f6990 0x7f1160106e70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f11500028a0 tx=0x7f1150009f00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115d7fa700 10 client.?.objecter ms_handle_connect 0x7f11600f6990 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115d7fa700 1 -- 172.21.15.171:0/3634672831 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150007b40 con 0x7f11600f6990 2024-02-23T00:27:32.028 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115d7fa700 1 -- 172.21.15.171:0/3634672831 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f115000f040 con 0x7f11600f6990 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116010ee90 con 0x7f11600f6990 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f115d7fa700 1 -- 172.21.15.171:0/3634672831 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150014710 con 0x7f11600f6990 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f1166516700 10 client.24424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116010f310 con 0x7f11600f6990 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.023+0000 7f1166516700 1 librados: init done 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.024+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116005f360 con 0x7f11600f6990 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.024+0000 7f115d7fa700 1 -- 172.21.15.171:0/3634672831 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 15) v1 ==== 77080+0+0 (secure 0 0 0) 0x7f1150009110 con 0x7f11600f6990 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.025+0000 7f115d7fa700 1 --2- 172.21.15.171:0/3634672831 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1148062c20 0x7f11480650e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.025+0000 7f115d7fa700 1 -- 172.21.15.171:0/3634672831 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5884+0+0 (secure 0 0 0) 0x7f115007c620 con 0x7f11600f6990 2024-02-23T00:27:32.029 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.025+0000 7f115d7fa700 10 client.24424.objecter ms_dispatch 0x7f1160001550 osd_map(63..63 src has 1..63) v4 2024-02-23T00:27:32.030 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.025+0000 7f115d7fa700 3 client.24424.objecter handle_osd_map got epochs [63,63] > 0 2024-02-23T00:27:32.030 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.025+0000 7f115d7fa700 3 client.24424.objecter handle_osd_map decoding full epoch 63 2024-02-23T00:27:32.030 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.025+0000 7f115d7fa700 20 client.24424.objecter dump_active .. 0 homeless 2024-02-23T00:27:32.030 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.025+0000 7f115f7fe700 1 --2- 172.21.15.171:0/3634672831 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1148062c20 0x7f11480650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:32.030 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.026+0000 7f115d7fa700 10 client.24424.objecter ms_handle_connect 0x7f1148062c20 2024-02-23T00:27:32.030 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.026+0000 7f115f7fe700 1 --2- 172.21.15.171:0/3634672831 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1148062c20 0x7f11480650e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1154009da0 tx=0x7f1154009cb0).ready entity=mgr.14146 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:32.030 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.029+0000 7f115d7fa700 1 -- 172.21.15.171:0/3634672831 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f115004c810 con 0x7f11600f6990 2024-02-23T00:27:32.220 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.219+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi087=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f1160059120 con 0x7f1148062c20 2024-02-23T00:27:32.226 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.225+0000 7f115d7fa700 1 -- 172.21.15.171:0/3634672831 <== mgr.14146 v2:172.21.15.87:6800/1121018082 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f1160059120 con 0x7f1148062c20 2024-02-23T00:27:32.226 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled alertmanager update... 2024-02-23T00:27:32.228 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 10 librados: watch_flush enter 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 10 librados: watch_flush exit 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 20 client.24424.objecter shutdown clearing up homeless session... 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 10 client.24424.objecter successfully canceled tick 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1148062c20 msgr2=0x7f11480650e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 --2- 172.21.15.171:0/3634672831 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1148062c20 0x7f11480650e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f1154009da0 tx=0x7f1154009cb0).stop 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f6990 msgr2=0x7f1160106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 --2- 172.21.15.171:0/3634672831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f6990 0x7f1160106e70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f11500028a0 tx=0x7f1150009f00).stop 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 shutdown_connections 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 --2- 172.21.15.171:0/3634672831 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11600f8580 0x7f116010e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 --2- 172.21.15.171:0/3634672831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11600f6990 0x7f1160106e70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 --2- 172.21.15.171:0/3634672831 >> v2:172.21.15.87:6800/1121018082 conn(0x7f1148062c20 0x7f11480650e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 --2- 172.21.15.171:0/3634672831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11600f7c30 0x7f1160107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:32.229 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 >> 172.21.15.171:0/3634672831 conn(0x7f11600f2190 msgr2=0x7f11600f4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:32.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 shutdown_connections 2024-02-23T00:27:32.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 -- 172.21.15.171:0/3634672831 wait complete. 2024-02-23T00:27:32.230 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:32.227+0000 7f1166516700 1 librados: shutdown 2024-02-23T00:27:33.451 DEBUG:teuthology.orchestra.run.smithi087:alertmanager.a> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@alertmanager.a.service 2024-02-23T00:27:33.454 INFO:tasks.cephadm:Adding grafana.a on smithi171 2024-02-23T00:27:33.455 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph orch apply grafana '1;smithi171=a' 2024-02-23T00:27:33.482 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:32.111403+0000 mgr.y (mgr.14146) 220 : cluster [DBG] pgmap v208: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 KiB/s rd, 4.6 KiB/s wr, 203 op/s 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:32.220485+0000 mgr.y (mgr.14146) 221 : audit [DBG] from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi087=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:32.222055+0000 mgr.y (mgr.14146) 222 : cephadm [INF] Saving service alertmanager spec with placement smithi087=a;count:1 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:32.225765+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:32.526349+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:33.136387+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:33.137267+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:33.138117+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:33.143508+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:33.148550+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:32.111403+0000 mgr.y (mgr.14146) 220 : cluster [DBG] pgmap v208: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 KiB/s rd, 4.6 KiB/s wr, 203 op/s 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:32.220485+0000 mgr.y (mgr.14146) 221 : audit [DBG] from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi087=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:33.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:32.222055+0000 mgr.y (mgr.14146) 222 : cephadm [INF] Saving service alertmanager spec with placement smithi087=a;count:1 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:32.225765+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:32.526349+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:33.136387+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:33.137267+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:33.138117+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:33.143508+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:33.148550+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:32.111403+0000 mgr.y (mgr.14146) 220 : cluster [DBG] pgmap v208: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 95 KiB/s rd, 4.6 KiB/s wr, 203 op/s 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:32.220485+0000 mgr.y (mgr.14146) 221 : audit [DBG] from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi087=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:27:33.484 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:32.222055+0000 mgr.y (mgr.14146) 222 : cephadm [INF] Saving service alertmanager spec with placement smithi087=a;count:1 2024-02-23T00:27:33.485 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:32.225765+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.485 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:32.526349+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.485 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:33.136387+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.485 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:33.137267+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:33.485 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:33.138117+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:33.485 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:33.143508+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' 2024-02-23T00:27:33.485 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:33.148550+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-23T00:27:33.487 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:27:34.442 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ignoring --setuser ceph since I am not root 2024-02-23T00:27:34.442 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ignoring --setgroup ceph since I am not root 2024-02-23T00:27:34.442 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:34.183+0000 7f1ff6a51700 1 -- 172.21.15.171:0/451375272 <== mon.1 v2:172.21.15.171:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d4128324e0 con 0x55d412808c00 2024-02-23T00:27:34.442 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:34.305+0000 7f2006c2e680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-23T00:27:34.442 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:34.385+0000 7f2006c2e680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-23T00:27:34.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ignoring --setuser ceph since I am not root 2024-02-23T00:27:34.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ignoring --setgroup ceph since I am not root 2024-02-23T00:27:34.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:34.187+0000 7f6efd0c6700 1 -- 172.21.15.87:0/3796776378 <== mon.2 v2:172.21.15.87:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5607401404e0 con 0x560740116c00 2024-02-23T00:27:34.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:34.309+0000 7f6f0d2a3680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-23T00:27:34.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:34.390+0000 7f6f0d2a3680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-23T00:27:34.970 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:34.756+0000 7f2006c2e680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-23T00:27:35.082 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:34.748+0000 7f6f0d2a3680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-23T00:27:35.236 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:34.150145+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-23T00:27:35.236 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:34.150192+0000 mon.a (mon.0) 600 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-23T00:27:35.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.471+0000 7ff46653b700 1 -- 172.21.15.171:0/1796548831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4600fcfe0 msgr2=0x7ff4600ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:35.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.471+0000 7ff46653b700 1 --2- 172.21.15.171:0/1796548831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4600fcfe0 0x7ff4600ff3d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff454004870 tx=0x7ff454009cb0).stop 2024-02-23T00:27:35.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 -- 172.21.15.171:0/1796548831 shutdown_connections 2024-02-23T00:27:35.477 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 --2- 172.21.15.171:0/1796548831 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4600ff8c0 0x7ff460101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 --2- 172.21.15.171:0/1796548831 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff460061650 0x7ff4600fcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 --2- 172.21.15.171:0/1796548831 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4600fcfe0 0x7ff4600ff3d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 -- 172.21.15.171:0/1796548831 >> 172.21.15.171:0/1796548831 conn(0x7ff4600f2190 msgr2=0x7ff4600f45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 -- 172.21.15.171:0/1796548831 shutdown_connections 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 -- 172.21.15.171:0/1796548831 wait complete. 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 librados: starting msgr at 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 librados: starting objecter 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 Processor -- start 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 -- start start 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 librados: setting wanted keys 2024-02-23T00:27:35.478 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.472+0000 7ff46653b700 1 librados: calling monclient init 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff460061650 0x7ff46010a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4600fcfe0 0x7ff46010a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4600ff8c0 0x7ff4601117d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff460111cc0 con 0x7ff4600fcfe0 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff460111e00 con 0x7ff460061650 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff460111f40 con 0x7ff4600ff8c0 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45ffff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff460061650 0x7ff46010a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45ffff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff460061650 0x7ff46010a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:42906/0 (socket says 172.21.15.171:42906) 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45ffff700 1 -- 172.21.15.171:0/3143907707 learned_addr learned my addr 172.21.15.171:0/3143907707 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff464ad8700 1 --2- 172.21.15.171:0/3143907707 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4600ff8c0 0x7ff4601117d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45f7fe700 1 --2- 172.21.15.171:0/3143907707 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4600fcfe0 0x7ff46010a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45ffff700 1 -- 172.21.15.171:0/3143907707 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4600ff8c0 msgr2=0x7ff4601117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45ffff700 1 --2- 172.21.15.171:0/3143907707 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4600ff8c0 0x7ff4601117d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:35.479 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45ffff700 1 -- 172.21.15.171:0/3143907707 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4600fcfe0 msgr2=0x7ff46010a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45ffff700 1 --2- 172.21.15.171:0/3143907707 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4600fcfe0 0x7ff46010a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45ffff700 1 -- 172.21.15.171:0/3143907707 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff454005040 con 0x7ff460061650 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45ffff700 1 --2- 172.21.15.171:0/3143907707 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff460061650 0x7ff46010a200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff4500027f0 tx=0x7ff450002b00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45d7fa700 10 client.?.objecter ms_handle_connect 0x7ff460061650 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff450007b40 con 0x7ff460061650 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff460112220 con 0x7ff460061650 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff45000f040 con 0x7ff460061650 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4500146f0 con 0x7ff460061650 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 10 client.24442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff460112710 con 0x7ff460061650 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.473+0000 7ff46653b700 1 librados: init done 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.474+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46005f360 con 0x7ff460061650 2024-02-23T00:27:35.480 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.475+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 16) v1 ==== 77094+0+0 (secure 0 0 0) 0x7ff450009110 con 0x7ff460061650 2024-02-23T00:27:35.481 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.476+0000 7ff45d7fa700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 0x7ff448065130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:35.481 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.476+0000 7ff45f7fe700 1 -- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 msgr2=0x7ff448065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1121018082 2024-02-23T00:27:35.481 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.476+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5884+0+0 (secure 0 0 0) 0x7ff45007c830 con 0x7ff460061650 2024-02-23T00:27:35.481 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.476+0000 7ff45d7fa700 10 client.24442.objecter ms_dispatch 0x7ff460001550 osd_map(63..63 src has 1..63) v4 2024-02-23T00:27:35.481 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.476+0000 7ff45d7fa700 3 client.24442.objecter handle_osd_map got epochs [63,63] > 0 2024-02-23T00:27:35.481 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.476+0000 7ff45d7fa700 3 client.24442.objecter handle_osd_map decoding full epoch 63 2024-02-23T00:27:35.481 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.476+0000 7ff45f7fe700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 0x7ff448065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-23T00:27:35.481 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.476+0000 7ff45d7fa700 20 client.24442.objecter dump_active .. 0 homeless 2024-02-23T00:27:35.483 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.482+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff45004c960 con 0x7ff460061650 2024-02-23T00:27:35.493 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:34.150145+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-23T00:27:35.493 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:34.150192+0000 mon.a (mon.0) 600 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-23T00:27:35.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:34.150145+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14146 172.21.15.87:0/1426589195' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-23T00:27:35.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:34.150192+0000 mon.a (mon.0) 600 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-23T00:27:35.678 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.677+0000 7ff45f7fe700 1 -- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 msgr2=0x7ff448065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1121018082 2024-02-23T00:27:35.678 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.677+0000 7ff45f7fe700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 0x7ff448065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-23T00:27:35.705 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:35.703+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 --> v2:172.21.15.87:6800/1121018082 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi171=a", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4600f3c50 con 0x7ff448062c70 2024-02-23T00:27:35.788 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:35.491+0000 7f6f0d2a3680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-23T00:27:35.789 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:35.576+0000 7f6f0d2a3680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-23T00:27:35.838 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:35.537+0000 7f2006c2e680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-23T00:27:35.838 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:35.624+0000 7f2006c2e680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-23T00:27:36.079 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:36.078+0000 7ff45f7fe700 1 -- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 msgr2=0x7ff448065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1121018082 2024-02-23T00:27:36.079 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:36.078+0000 7ff45f7fe700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 0x7ff448065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-23T00:27:36.082 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:35.786+0000 7f6f0d2a3680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-23T00:27:36.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:35.836+0000 7f2006c2e680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-23T00:27:36.880 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:36.879+0000 7ff45f7fe700 1 -- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 msgr2=0x7ff448065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1121018082 2024-02-23T00:27:36.880 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:36.879+0000 7ff45f7fe700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 0x7ff448065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-23T00:27:37.012 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:36.717+0000 7f6f0d2a3680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-23T00:27:37.012 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:36.843+0000 7f6f0d2a3680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:27:37.012 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:36.921+0000 7f6f0d2a3680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-23T00:27:37.066 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:36.768+0000 7f2006c2e680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-23T00:27:37.066 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:36.895+0000 7f2006c2e680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:27:37.066 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:36.974+0000 7f2006c2e680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-23T00:27:37.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:37.009+0000 7f6f0d2a3680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-23T00:27:37.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:37.207+0000 7f6f0d2a3680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-23T00:27:37.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:35.753387+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:37.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:35.753434+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:37.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:36.148610+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-23T00:27:37.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:35.753387+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:37.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:35.753434+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:37.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:36.148610+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-23T00:27:37.442 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:37.064+0000 7f2006c2e680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-23T00:27:37.442 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:37.262+0000 7f2006c2e680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-23T00:27:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:35.753387+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:35.753434+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:36.148610+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-23T00:27:37.832 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:37.485+0000 7f6f0d2a3680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-23T00:27:37.832 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:37.583+0000 7f6f0d2a3680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-23T00:27:37.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:37.539+0000 7f2006c2e680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-23T00:27:37.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:37.637+0000 7f2006c2e680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-23T00:27:38.482 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:38.480+0000 7ff45f7fe700 1 -- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 msgr2=0x7ff448065130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.87:6800/1121018082 2024-02-23T00:27:38.482 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:38.481+0000 7ff45f7fe700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 0x7ff448065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-23T00:27:38.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:38.298+0000 7f6f0d2a3680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-23T00:27:38.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:38.380+0000 7f6f0d2a3680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-23T00:27:38.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:38.471+0000 7f6f0d2a3680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-23T00:27:38.692 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:38.355+0000 7f2006c2e680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-23T00:27:38.692 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:38.437+0000 7f2006c2e680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-23T00:27:38.692 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:38.528+0000 7f2006c2e680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-23T00:27:38.904 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:38.648+0000 7f6f0d2a3680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-23T00:27:38.963 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:38.702+0000 7f2006c2e680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-23T00:27:39.173 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:38.902+0000 7f6f0d2a3680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-23T00:27:39.173 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:39.034+0000 7f6f0d2a3680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-23T00:27:39.233 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:38.962+0000 7f2006c2e680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-23T00:27:39.233 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:39.098+0000 7f2006c2e680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-23T00:27:39.430 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:39.171+0000 7f6f0d2a3680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:27:39.430 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:39.344+0000 7f6f0d2a3680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-23T00:27:39.451 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:39.450+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mgrmap(e 17) v1 ==== 76859+0+0 (secure 0 0 0) 0x7ff45007a520 con 0x7ff460061650 2024-02-23T00:27:39.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:39.450+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 msgr2=0x7ff448065130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:27:39.452 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:39.450+0000 7ff45d7fa700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1121018082 conn(0x7ff448062c70 0x7ff448065130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:27:39.490 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:39.232+0000 7f2006c2e680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-23T00:27:39.490 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:39.402+0000 7f2006c2e680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-23T00:27:39.684 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:38.540998+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:39.684 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:38.541060+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:38.596184+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:39.442977+0000 mon.a (mon.0) 603 : cluster [INF] Active manager daemon y restarted 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:39.443285+0000 mon.a (mon.0) 604 : cluster [INF] Activating manager daemon y 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:39.450217+0000 mon.a (mon.0) 605 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:39.450685+0000 mon.a (mon.0) 606 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00748258s), standbys: x 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.453766+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.453954+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.454123+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.454788+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.454993+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.455221+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.455468+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:27:39.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.455696+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:27:39.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.456015+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:27:39.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.456313+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:27:39.686 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:27:39.428+0000 7f6f0d2a3680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-23T00:27:39.686 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [79B blob data] 2024-02-23T00:27:39.686 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [80B blob data] 2024-02-23T00:27:39.686 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:27:39] ENGINE Bus STARTING 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:38.540998+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:38.541060+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:38.596184+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:39.442977+0000 mon.a (mon.0) 603 : cluster [INF] Active manager daemon y restarted 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:39.443285+0000 mon.a (mon.0) 604 : cluster [INF] Activating manager daemon y 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:39.450217+0000 mon.a (mon.0) 605 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:39.450685+0000 mon.a (mon.0) 606 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00748258s), standbys: x 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.453766+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.453954+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.454123+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.454788+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.454993+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-23T00:27:39.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.455221+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.455468+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.455696+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.456015+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.456313+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.456508+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.456671+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.456834+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.456999+0000 mon.a (mon.0) 620 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.457112+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.457492+0000 mon.a (mon.0) 622 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:39.471833+0000 mon.a (mon.0) 623 : cluster [INF] Manager daemon y is now available 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:39.502863+0000 mon.a (mon.0) 624 : cluster [DBG] Standby manager daemon x restarted 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:39.502949+0000 mon.a (mon.0) 625 : cluster [DBG] Standby manager daemon x started 2024-02-23T00:27:39.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.504543+0000 mon.b (mon.1) 28 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.505062+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.506945+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.507951+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.568746+0000 mon.a (mon.0) 626 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.570000+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.456508+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.456671+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.456834+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.456999+0000 mon.a (mon.0) 620 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.457112+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.457492+0000 mon.a (mon.0) 622 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:27:39.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:39.471833+0000 mon.a (mon.0) 623 : cluster [INF] Manager daemon y is now available 2024-02-23T00:27:39.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:39.502863+0000 mon.a (mon.0) 624 : cluster [DBG] Standby manager daemon x restarted 2024-02-23T00:27:39.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:39.502949+0000 mon.a (mon.0) 625 : cluster [DBG] Standby manager daemon x started 2024-02-23T00:27:39.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.504543+0000 mon.b (mon.1) 28 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-23T00:27:39.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.505062+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-23T00:27:39.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.506945+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-23T00:27:39.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.507951+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-23T00:27:39.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.568746+0000 mon.a (mon.0) 626 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:39.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.570000+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:39.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: debug 2024-02-23T00:27:39.488+0000 7f2006c2e680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-23T00:27:39.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: [23/Feb/2024:00:27:39] ENGINE Bus STARTING 2024-02-23T00:27:39.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: CherryPy Checker: 2024-02-23T00:27:39.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: The Application mounted at '' has an empty config. 2024-02-23T00:27:39.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: 2024-02-23T00:27:39.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: [23/Feb/2024:00:27:39] ENGINE Serving on http://:::9283 2024-02-23T00:27:39.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: [23/Feb/2024:00:27:39] ENGINE Bus STARTED 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:38.540998+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:38.541060+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:38.596184+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:39.442977+0000 mon.a (mon.0) 603 : cluster [INF] Active manager daemon y restarted 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:39.443285+0000 mon.a (mon.0) 604 : cluster [INF] Activating manager daemon y 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:39.450217+0000 mon.a (mon.0) 605 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:39.450685+0000 mon.a (mon.0) 606 : cluster [DBG] mgrmap e17: y(active, starting, since 0.00748258s), standbys: x 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.453766+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.453954+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.454123+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-23T00:27:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.454788+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.454993+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.455221+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.455468+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.455696+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.456015+0000 mon.a (mon.0) 615 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.456313+0000 mon.a (mon.0) 616 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.456508+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.456671+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.456834+0000 mon.a (mon.0) 619 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.456999+0000 mon.a (mon.0) 620 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.457112+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.457492+0000 mon.a (mon.0) 622 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:39.471833+0000 mon.a (mon.0) 623 : cluster [INF] Manager daemon y is now available 2024-02-23T00:27:39.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:39.502863+0000 mon.a (mon.0) 624 : cluster [DBG] Standby manager daemon x restarted 2024-02-23T00:27:39.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:39.502949+0000 mon.a (mon.0) 625 : cluster [DBG] Standby manager daemon x started 2024-02-23T00:27:39.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.504543+0000 mon.b (mon.1) 28 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-23T00:27:39.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.505062+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-23T00:27:39.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.506945+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-23T00:27:39.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.507951+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.171:0/2833003639' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-23T00:27:39.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.568746+0000 mon.a (mon.0) 626 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:39.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.570000+0000 mon.a (mon.0) 627 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:27:40.082 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: CherryPy Checker: 2024-02-23T00:27:40.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: The Application mounted at '' has an empty config. 2024-02-23T00:27:40.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: 2024-02-23T00:27:40.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:27:39] ENGINE Serving on http://:::9283 2024-02-23T00:27:40.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:27:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:27:39] ENGINE Bus STARTED 2024-02-23T00:27:40.455 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.454+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 <== mon.1 v2:172.21.15.171:3300/0 8 ==== mgrmap(e 18) v1 ==== 76894+0+0 (secure 0 0 0) 0x7ff45007a8a0 con 0x7ff460061650 2024-02-23T00:27:40.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.454+0000 7ff45d7fa700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff448032740 0x7ff44806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:40.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.454+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi171=a", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4600f3c50 con 0x7ff448032740 2024-02-23T00:27:40.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.454+0000 7ff45f7fe700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff448032740 0x7ff44806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:40.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.455+0000 7ff45d7fa700 10 client.24442.objecter ms_handle_connect 0x7ff448032740 2024-02-23T00:27:40.456 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.455+0000 7ff45f7fe700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff448032740 0x7ff44806e160 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff454004ee0 tx=0x7ff45400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:40.463 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.462+0000 7ff45d7fa700 1 -- 172.21.15.171:0/3143907707 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff4600f3c50 con 0x7ff448032740 2024-02-23T00:27:40.464 INFO:teuthology.orchestra.run.smithi171.stdout:Scheduled grafana update... 2024-02-23T00:27:40.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.464+0000 7ff46653b700 10 librados: watch_flush enter 2024-02-23T00:27:40.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.464+0000 7ff46653b700 10 librados: watch_flush exit 2024-02-23T00:27:40.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.464+0000 7ff46653b700 20 client.24442.objecter shutdown clearing up homeless session... 2024-02-23T00:27:40.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.464+0000 7ff46653b700 10 client.24442.objecter successfully canceled tick 2024-02-23T00:27:40.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.464+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff448032740 msgr2=0x7ff44806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:40.466 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.464+0000 7ff46653b700 1 --2- 172.21.15.171:0/3143907707 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff448032740 0x7ff44806e160 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff454004ee0 tx=0x7ff45400b040).stop 2024-02-23T00:27:40.467 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.464+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff460061650 msgr2=0x7ff46010a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:40.467 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.464+0000 7ff46653b700 1 --2- 172.21.15.171:0/3143907707 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff460061650 0x7ff46010a200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ff4500027f0 tx=0x7ff450002b00).stop 2024-02-23T00:27:40.467 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.464+0000 7ff464ad8700 1 -- 172.21.15.171:0/3143907707 reap_dead start 2024-02-23T00:27:40.467 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.465+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 shutdown_connections 2024-02-23T00:27:40.467 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.465+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 >> 172.21.15.171:0/3143907707 conn(0x7ff4600f2190 msgr2=0x7ff4600fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:40.467 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.465+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 shutdown_connections 2024-02-23T00:27:40.467 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.465+0000 7ff46653b700 1 -- 172.21.15.171:0/3143907707 wait complete. 2024-02-23T00:27:40.467 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:40.465+0000 7ff46653b700 1 librados: shutdown 2024-02-23T00:27:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.611925+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.626489+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:27:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.648120+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-23T00:27:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:39.850645+0000 mon.a (mon.0) 631 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-23T00:27:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.851082+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-23T00:27:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:39.851808+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-23T00:27:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:40.455336+0000 mon.a (mon.0) 634 : cluster [DBG] mgrmap e18: y(active, since 1.01212s), standbys: x 2024-02-23T00:27:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:40.462981+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.611925+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.626489+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:27:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.648120+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-23T00:27:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:39.850645+0000 mon.a (mon.0) 631 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-23T00:27:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.851082+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-23T00:27:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:39.851808+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-23T00:27:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:40.455336+0000 mon.a (mon.0) 634 : cluster [DBG] mgrmap e18: y(active, since 1.01212s), standbys: x 2024-02-23T00:27:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:40.462981+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:40.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:27:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.611925+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.626489+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-23T00:27:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.648120+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-23T00:27:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:39.850645+0000 mon.a (mon.0) 631 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-23T00:27:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.851082+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-23T00:27:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:39.851808+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-23T00:27:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:40.455336+0000 mon.a (mon.0) 634 : cluster [DBG] mgrmap e18: y(active, since 1.01212s), standbys: x 2024-02-23T00:27:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:40.462981+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:41.509 DEBUG:teuthology.orchestra.run.smithi171:grafana.a> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@grafana.a.service 2024-02-23T00:27:41.551 INFO:tasks.cephadm:Setting up client nodes... 2024-02-23T00:27:41.552 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-23T00:27:41.583 INFO:journalctl@ceph.grafana.a.smithi171.stdout:-- Logs begin at Fri 2024-02-23 00:01:54 UTC. -- 2024-02-23T00:27:42.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:40.459749+0000 mgr.y (mgr.24437) 3 : cephadm [INF] Saving service grafana spec with placement smithi171=a;count:1 2024-02-23T00:27:42.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:40.459956+0000 mgr.y (mgr.24437) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:42.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:40.848276+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-23T00:27:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:40.848326+0000 mon.a (mon.0) 637 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-23T00:27:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:41.095951+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:41.374875+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:41.375459+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:41.375917+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:41.376452+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:41.380875+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:42.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:40.459749+0000 mgr.y (mgr.24437) 3 : cephadm [INF] Saving service grafana spec with placement smithi171=a;count:1 2024-02-23T00:27:42.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:40.459956+0000 mgr.y (mgr.24437) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:42.287 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:40.848276+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-23T00:27:42.287 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:40.848326+0000 mon.a (mon.0) 637 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-23T00:27:42.287 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:41.095951+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:42.287 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:41.374875+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.287 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:41.375459+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.287 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:41.375917+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.287 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:41.376452+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.287 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:41.380875+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:42.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:40.459749+0000 mgr.y (mgr.24437) 3 : cephadm [INF] Saving service grafana spec with placement smithi171=a;count:1 2024-02-23T00:27:42.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:40.459956+0000 mgr.y (mgr.24437) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:42.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:40.848276+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-23T00:27:42.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:40.848326+0000 mon.a (mon.0) 637 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-23T00:27:42.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:41.095951+0000 mon.a (mon.0) 638 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:42.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:41.374875+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:41.375459+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:41.375917+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:41.376452+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:42.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:41.380875+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:41.376837+0000 mgr.y (mgr.24437) 5 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 3768M 2024-02-23T00:27:43.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:41.455904+0000 mgr.y (mgr.24437) 6 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:42.384851+0000 mon.a (mon.0) 644 : cluster [DBG] mgrmap e19: y(active, since 2s), standbys: x 2024-02-23T00:27:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.440717+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.712575+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.713181+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.713719+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.714227+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.718680+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.719399+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:43.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.719945+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:43.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.775197+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.805174+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:42.808670+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:41.376837+0000 mgr.y (mgr.24437) 5 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 3768M 2024-02-23T00:27:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:41.455904+0000 mgr.y (mgr.24437) 6 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:42.384851+0000 mon.a (mon.0) 644 : cluster [DBG] mgrmap e19: y(active, since 2s), standbys: x 2024-02-23T00:27:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.440717+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.086 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.712575+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.713181+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.087 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.713719+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.714227+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.718680+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.719399+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:43.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.719945+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:43.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.775197+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.805174+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:42.808670+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:41.376837+0000 mgr.y (mgr.24437) 5 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 3768M 2024-02-23T00:27:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:41.455904+0000 mgr.y (mgr.24437) 6 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:42.384851+0000 mon.a (mon.0) 644 : cluster [DBG] mgrmap e19: y(active, since 2s), standbys: x 2024-02-23T00:27:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.440717+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.712575+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.713181+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.713719+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.714227+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:27:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.718680+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.719399+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:27:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.719945+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:27:43.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.775197+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.805174+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:42.808670+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:43.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.694+0000 7fb35b9f3700 1 -- 172.21.15.87:0/1445845889 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3540fdb60 msgr2=0x7fb3540fdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:43.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.694+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/1445845889 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3540fdb60 0x7fb3540fdf40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb3440047b0 tx=0x7fb344009cb0).stop 2024-02-23T00:27:43.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.694+0000 7fb35b9f3700 1 -- 172.21.15.87:0/1445845889 shutdown_connections 2024-02-23T00:27:43.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.694+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/1445845889 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3540f6a20 0x7fb3541049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.694+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/1445845889 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3540fdb60 0x7fb3540fdf40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.694+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/1445845889 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3540fe4c0 0x7fb3540f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.694+0000 7fb35b9f3700 1 -- 172.21.15.87:0/1445845889 >> 172.21.15.87:0/1445845889 conn(0x7fb35406c300 msgr2=0x7fb35406c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.694+0000 7fb35b9f3700 1 -- 172.21.15.87:0/1445845889 shutdown_connections 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.694+0000 7fb35b9f3700 1 -- 172.21.15.87:0/1445845889 wait complete. 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 librados: starting msgr at 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 librados: starting objecter 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 Processor -- start 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 -- start start 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 librados: setting wanted keys 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 librados: calling monclient init 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3540f6a20 0x7fb3540f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3540fdb60 0x7fb3540f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3540fe4c0 0x7fb3540f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:43.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb354116560 con 0x7fb3540fe4c0 2024-02-23T00:27:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3541166d0 con 0x7fb3540f6a20 2024-02-23T00:27:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35b9f3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb354116890 con 0x7fb3540fdb60 2024-02-23T00:27:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb359f90700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3540fe4c0 0x7fb3540f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb359f90700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3540fe4c0 0x7fb3540f7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48318/0 (socket says 172.21.15.87:48318) 2024-02-23T00:27:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb359f90700 1 -- 172.21.15.87:0/3297231975 learned_addr learned my addr 172.21.15.87:0/3297231975 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:27:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb358f8e700 1 --2- 172.21.15.87:0/3297231975 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3540fdb60 0x7fb3540f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:43.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb35978f700 1 --2- 172.21.15.87:0/3297231975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3540f6a20 0x7fb3540f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:43.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb359f90700 1 -- 172.21.15.87:0/3297231975 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3540fdb60 msgr2=0x7fb3540f7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb359f90700 1 --2- 172.21.15.87:0/3297231975 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3540fdb60 0x7fb3540f7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb359f90700 1 -- 172.21.15.87:0/3297231975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3540f6a20 msgr2=0x7fb3540f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb359f90700 1 --2- 172.21.15.87:0/3297231975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3540f6a20 0x7fb3540f7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb359f90700 1 -- 172.21.15.87:0/3297231975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c0097c0 con 0x7fb3540fe4c0 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb359f90700 1 --2- 172.21.15.87:0/3297231975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3540fe4c0 0x7fb3540f7ff0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb34c001df0 tx=0x7fb34c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb34a7fc700 10 client.?.objecter ms_handle_connect 0x7fb3540fe4c0 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.695+0000 7fb34a7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.696+0000 7fb34a7fc700 1 -- 172.21.15.87:0/3297231975 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c009a10 con 0x7fb3540fe4c0 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.696+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb344005040 con 0x7fb3540fe4c0 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.696+0000 7fb34a7fc700 1 -- 172.21.15.87:0/3297231975 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb34c0095a0 con 0x7fb3540fe4c0 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.696+0000 7fb34a7fc700 1 -- 172.21.15.87:0/3297231975 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c007e30 con 0x7fb3540fe4c0 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.696+0000 7fb35b9f3700 10 client.14535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:43.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.696+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb354116ed0 con 0x7fb3540fe4c0 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.696+0000 7fb35b9f3700 1 librados: init done 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.697+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35405ffd0 con 0x7fb3540fe4c0 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.697+0000 7fb34a7fc700 1 -- 172.21.15.87:0/3297231975 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb34c0090d0 con 0x7fb3540fe4c0 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.697+0000 7fb34a7fc700 1 --2- 172.21.15.87:0/3297231975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb340062c90 0x7fb340065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.697+0000 7fb34a7fc700 1 -- 172.21.15.87:0/3297231975 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6056+0+0 (secure 0 0 0) 0x7fb34c07d2f0 con 0x7fb3540fe4c0 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.697+0000 7fb34a7fc700 10 client.14535.objecter ms_dispatch 0x7fb354001550 osd_map(68..68 src has 1..68) v4 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.697+0000 7fb34a7fc700 3 client.14535.objecter handle_osd_map got epochs [68,68] > 0 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.697+0000 7fb34a7fc700 3 client.14535.objecter handle_osd_map decoding full epoch 68 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.697+0000 7fb35978f700 1 --2- 172.21.15.87:0/3297231975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb340062c90 0x7fb340065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.697+0000 7fb34a7fc700 20 client.14535.objecter dump_active .. 0 homeless 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.698+0000 7fb34a7fc700 10 client.14535.objecter ms_handle_connect 0x7fb340062c90 2024-02-23T00:27:43.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.698+0000 7fb35978f700 1 --2- 172.21.15.87:0/3297231975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb340062c90 0x7fb340065150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fb344009f20 tx=0x7fb344009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:43.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.702+0000 7fb34a7fc700 1 -- 172.21.15.87:0/3297231975 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb34c050fd0 con 0x7fb3540fe4c0 2024-02-23T00:27:43.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.948+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fb3540f8af0 con 0x7fb3540fe4c0 2024-02-23T00:27:43.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.952+0000 7fb34a7fc700 1 -- 172.21.15.87:0/3297231975 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7fb34c050bc0 con 0x7fb3540fe4c0 2024-02-23T00:27:43.953 INFO:teuthology.orchestra.run.smithi087.stdout:[client.0] 2024-02-23T00:27:43.953 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQB/5tdlrPmXOBAAfrzxFys2X8eCitmcg9n2WA== 2024-02-23T00:27:43.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 10 librados: watch_flush enter 2024-02-23T00:27:43.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 10 librados: watch_flush exit 2024-02-23T00:27:43.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 20 client.14535.objecter shutdown clearing up homeless session... 2024-02-23T00:27:43.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 10 client.14535.objecter successfully canceled tick 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb340062c90 msgr2=0x7fb340065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/3297231975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb340062c90 0x7fb340065150 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fb344009f20 tx=0x7fb344009cb0).stop 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3540fe4c0 msgr2=0x7fb3540f7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/3297231975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3540fe4c0 0x7fb3540f7ff0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb34c001df0 tx=0x7fb34c004d10).stop 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 shutdown_connections 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/3297231975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb340062c90 0x7fb340065150 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/3297231975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3540fe4c0 0x7fb3540f7ff0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/3297231975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3540f6a20 0x7fb3540f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 --2- 172.21.15.87:0/3297231975 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3540fdb60 0x7fb3540f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 >> 172.21.15.87:0/3297231975 conn(0x7fb35406c300 msgr2=0x7fb3540fb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 shutdown_connections 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 -- 172.21.15.87:0/3297231975 wait complete. 2024-02-23T00:27:43.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:43.954+0000 7fb35b9f3700 1 librados: shutdown 2024-02-23T00:27:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:42.714636+0000 mgr.y (mgr.24437) 7 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 4024M 2024-02-23T00:27:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:42.720923+0000 mgr.y (mgr.24437) 8 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.conf 2024-02-23T00:27:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:42.721206+0000 mgr.y (mgr.24437) 9 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2024-02-23T00:27:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:42.746048+0000 mgr.y (mgr.24437) 10 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:27:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:42.758170+0000 mgr.y (mgr.24437) 11 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:27:44.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:42.818169+0000 mgr.y (mgr.24437) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi087 2024-02-23T00:27:44.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:42.714636+0000 mgr.y (mgr.24437) 7 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 4024M 2024-02-23T00:27:44.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:42.720923+0000 mgr.y (mgr.24437) 8 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.conf 2024-02-23T00:27:44.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:42.721206+0000 mgr.y (mgr.24437) 9 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2024-02-23T00:27:44.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:42.746048+0000 mgr.y (mgr.24437) 10 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:27:44.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:42.758170+0000 mgr.y (mgr.24437) 11 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:27:44.310 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:42.818169+0000 mgr.y (mgr.24437) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi087 2024-02-23T00:27:44.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:42.714636+0000 mgr.y (mgr.24437) 7 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 4024M 2024-02-23T00:27:44.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:42.720923+0000 mgr.y (mgr.24437) 8 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.conf 2024-02-23T00:27:44.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:42.721206+0000 mgr.y (mgr.24437) 9 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.conf 2024-02-23T00:27:44.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:42.746048+0000 mgr.y (mgr.24437) 10 : cephadm [INF] Updating smithi171:/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:27:44.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:42.758170+0000 mgr.y (mgr.24437) 11 : cephadm [INF] Updating smithi087:/etc/ceph/ceph.client.admin.keyring 2024-02-23T00:27:44.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:42.818169+0000 mgr.y (mgr.24437) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi087 2024-02-23T00:27:44.772 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:27:44.772 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-23T00:27:44.772 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-23T00:27:44.813 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-23T00:27:45.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:43.456290+0000 mgr.y (mgr.24437) 13 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:45.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:43.949328+0000 mon.a (mon.0) 656 : audit [INF] from='client.? 172.21.15.87:0/3297231975' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-23T00:27:45.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:43.952928+0000 mon.a (mon.0) 657 : audit [INF] from='client.? 172.21.15.87:0/3297231975' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-23T00:27:45.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:44.843926+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-23T00:27:45.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:43.456290+0000 mgr.y (mgr.24437) 13 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:45.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:43.949328+0000 mon.a (mon.0) 656 : audit [INF] from='client.? 172.21.15.87:0/3297231975' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-23T00:27:45.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:43.952928+0000 mon.a (mon.0) 657 : audit [INF] from='client.? 172.21.15.87:0/3297231975' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-23T00:27:45.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:44.843926+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-23T00:27:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:43.456290+0000 mgr.y (mgr.24437) 13 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:27:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:43.949328+0000 mon.a (mon.0) 656 : audit [INF] from='client.? 172.21.15.87:0/3297231975' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-23T00:27:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:43.952928+0000 mon.a (mon.0) 657 : audit [INF] from='client.? 172.21.15.87:0/3297231975' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-23T00:27:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:44.843926+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 -- 172.21.15.171:0/2572760311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 msgr2=0x7f62c40f8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/2572760311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 0x7f62c40f8f10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f62ac004870 tx=0x7f62ac009cb0).stop 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 -- 172.21.15.171:0/2572760311 shutdown_connections 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/2572760311 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f62c4102090 0x7f62c4104480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/2572760311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 0x7f62c40f8f10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/2572760311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62c40f9490 0x7f62c4101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 -- 172.21.15.171:0/2572760311 >> 172.21.15.171:0/2572760311 conn(0x7f62c40f2640 msgr2=0x7f62c40f4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 -- 172.21.15.171:0/2572760311 shutdown_connections 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 -- 172.21.15.171:0/2572760311 wait complete. 2024-02-23T00:27:46.761 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 librados: starting msgr at 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.757+0000 7f62ca8f0700 1 librados: starting objecter 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 Processor -- start 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 -- start start 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 librados: setting wanted keys 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 librados: calling monclient init 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 0x7f62c40f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62c40f9490 0x7f62c40f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f62c4102090 0x7f62c40f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4116550 con 0x7f62c40f9490 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c41166c0 con 0x7f62c40f8b30 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62ca8f0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4116880 con 0x7f62c4102090 2024-02-23T00:27:46.762 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62c3fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 0x7f62c40f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62c3fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 0x7f62c40f6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.171:48568/0 (socket says 172.21.15.171:48568) 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62c3fff700 1 -- 172.21.15.171:0/3032440929 learned_addr learned my addr 172.21.15.171:0/3032440929 (peer_addr_for_me v2:172.21.15.171:0/0) 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62c37fe700 1 --2- 172.21.15.171:0/3032440929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62c40f9490 0x7f62c40f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62c8e8d700 1 --2- 172.21.15.171:0/3032440929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f62c4102090 0x7f62c40f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62c3fff700 1 -- 172.21.15.171:0/3032440929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f62c4102090 msgr2=0x7f62c40f74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.758+0000 7f62c3fff700 1 --2- 172.21.15.171:0/3032440929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f62c4102090 0x7f62c40f74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62c3fff700 1 -- 172.21.15.171:0/3032440929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62c40f9490 msgr2=0x7f62c40f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62c3fff700 1 --2- 172.21.15.171:0/3032440929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62c40f9490 0x7f62c40f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62c3fff700 1 -- 172.21.15.171:0/3032440929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac005040 con 0x7f62c40f8b30 2024-02-23T00:27:46.763 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62c3fff700 1 --2- 172.21.15.171:0/3032440929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 0x7f62c40f6ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f62ac009b10 tx=0x7f62ac007f90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:46.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62c17fa700 10 client.?.objecter ms_handle_connect 0x7f62c40f8b30 2024-02-23T00:27:46.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62c17fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:46.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62c17fa700 1 -- 172.21.15.171:0/3032440929 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac00e030 con 0x7f62c40f8b30 2024-02-23T00:27:46.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62c17fa700 1 -- 172.21.15.171:0/3032440929 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62ac0092e0 con 0x7f62c40f8b30 2024-02-23T00:27:46.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c4116b00 con 0x7f62c40f8b30 2024-02-23T00:27:46.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62c17fa700 1 -- 172.21.15.171:0/3032440929 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac018070 con 0x7f62c40f8b30 2024-02-23T00:27:46.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62ca8f0700 10 client.24463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:46.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c4116f00 con 0x7f62c40f8b30 2024-02-23T00:27:46.764 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.759+0000 7f62ca8f0700 1 librados: init done 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.760+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c405f860 con 0x7f62c40f8b30 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.760+0000 7f62c17fa700 1 -- 172.21.15.171:0/3032440929 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f62ac027440 con 0x7f62c40f8b30 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.760+0000 7f62c17fa700 1 --2- 172.21.15.171:0/3032440929 >> v2:172.21.15.87:6800/1231597463 conn(0x7f62b0062c90 0x7f62b0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.760+0000 7f62c17fa700 1 -- 172.21.15.171:0/3032440929 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f62ac07dc10 con 0x7f62c40f8b30 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.760+0000 7f62c17fa700 10 client.24463.objecter ms_dispatch 0x7f62c4001550 osd_map(69..69 src has 1..69) v4 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.760+0000 7f62c17fa700 3 client.24463.objecter handle_osd_map got epochs [69,69] > 0 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.760+0000 7f62c17fa700 3 client.24463.objecter handle_osd_map decoding full epoch 69 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.761+0000 7f62c37fe700 1 --2- 172.21.15.171:0/3032440929 >> v2:172.21.15.87:6800/1231597463 conn(0x7f62b0062c90 0x7f62b0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.761+0000 7f62c17fa700 20 client.24463.objecter dump_active .. 0 homeless 2024-02-23T00:27:46.766 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.761+0000 7f62c17fa700 10 client.24463.objecter ms_handle_connect 0x7f62b0062c90 2024-02-23T00:27:46.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.761+0000 7f62c37fe700 1 --2- 172.21.15.171:0/3032440929 >> v2:172.21.15.87:6800/1231597463 conn(0x7f62b0062c90 0x7f62b0065150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f62c40f8280 tx=0x7f62b400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:46.767 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.765+0000 7f62c17fa700 1 -- 172.21.15.171:0/3032440929 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f62ac051930 con 0x7f62c40f8b30 2024-02-23T00:27:46.988 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.987+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f62c405f360 con 0x7f62c40f8b30 2024-02-23T00:27:46.994 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.993+0000 7f62c17fa700 1 -- 172.21.15.171:0/3032440929 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f62ac004490 con 0x7f62c40f8b30 2024-02-23T00:27:46.994 INFO:teuthology.orchestra.run.smithi171.stdout:[client.1] 2024-02-23T00:27:46.994 INFO:teuthology.orchestra.run.smithi171.stdout: key = AQCC5tdlWdPqOhAAjofVzDof84uZsuBZR+hm2g== 2024-02-23T00:27:46.996 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 10 librados: watch_flush enter 2024-02-23T00:27:46.996 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 10 librados: watch_flush exit 2024-02-23T00:27:46.996 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 20 client.24463.objecter shutdown clearing up homeless session... 2024-02-23T00:27:46.996 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 10 client.24463.objecter successfully canceled tick 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 >> v2:172.21.15.87:6800/1231597463 conn(0x7f62b0062c90 msgr2=0x7f62b0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/3032440929 >> v2:172.21.15.87:6800/1231597463 conn(0x7f62b0062c90 0x7f62b0065150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f62c40f8280 tx=0x7f62b400b040).stop 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 msgr2=0x7f62c40f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/3032440929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 0x7f62c40f6ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f62ac009b10 tx=0x7f62ac007f90).stop 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 shutdown_connections 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/3032440929 >> v2:172.21.15.87:6800/1231597463 conn(0x7f62b0062c90 0x7f62b0065150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/3032440929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f62c4102090 0x7f62c40f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/3032440929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f62c40f8b30 0x7f62c40f6ae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 --2- 172.21.15.171:0/3032440929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f62c40f9490 0x7f62c40f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 >> 172.21.15.171:0/3032440929 conn(0x7f62c40f2640 msgr2=0x7f62c40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 shutdown_connections 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 -- 172.21.15.171:0/3032440929 wait complete. 2024-02-23T00:27:46.997 INFO:teuthology.orchestra.run.smithi171.stderr:2024-02-23T00:27:46.995+0000 7f62ca8f0700 1 librados: shutdown 2024-02-23T00:27:47.191 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:45.456745+0000 mgr.y (mgr.24437) 14 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-23T00:27:47.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:45.456745+0000 mgr.y (mgr.24437) 14 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-23T00:27:47.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:45.456745+0000 mgr.y (mgr.24437) 14 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-23T00:27:47.650 DEBUG:teuthology.orchestra.run.smithi171:> set -ex 2024-02-23T00:27:47.650 DEBUG:teuthology.orchestra.run.smithi171:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-23T00:27:47.650 DEBUG:teuthology.orchestra.run.smithi171:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-23T00:27:47.691 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump --format=json 2024-02-23T00:27:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:46.987985+0000 mon.b (mon.1) 32 : audit [INF] from='client.? 172.21.15.171:0/3032440929' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-23T00:27:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:46.988301+0000 mon.a (mon.0) 659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-23T00:27:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:46.991509+0000 mon.a (mon.0) 660 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-23T00:27:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:47.010709+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:47.010766+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:48.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:46.987985+0000 mon.b (mon.1) 32 : audit [INF] from='client.? 172.21.15.171:0/3032440929' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-23T00:27:48.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:46.988301+0000 mon.a (mon.0) 659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-23T00:27:48.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:46.991509+0000 mon.a (mon.0) 660 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-23T00:27:48.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:47.010709+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:48.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:47.010766+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:48.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:46.987985+0000 mon.b (mon.1) 32 : audit [INF] from='client.? 172.21.15.171:0/3032440929' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-23T00:27:48.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:46.988301+0000 mon.a (mon.0) 659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-23T00:27:48.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:46.991509+0000 mon.a (mon.0) 660 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-23T00:27:48.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:47.010709+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:48.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:47.010766+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:49.022 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:27:49.268 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:47.457244+0000 mgr.y (mgr.24437) 15 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-23T00:27:49.268 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:48.004759+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-23T00:27:49.269 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:47.457244+0000 mgr.y (mgr.24437) 15 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-23T00:27:49.269 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:48.004759+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-23T00:27:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:47.457244+0000 mgr.y (mgr.24437) 15 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-23T00:27:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:48.004759+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-23T00:27:50.845 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:27:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:27:51.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:49.457727+0000 mgr.y (mgr.24437) 16 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:27:51.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:49.847540+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-23T00:27:51.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:49.457727+0000 mgr.y (mgr.24437) 16 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:27:51.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:49.847540+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-23T00:27:51.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:49.457727+0000 mgr.y (mgr.24437) 16 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:27:51.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:49.847540+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-23T00:27:52.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:50.469223+0000 mgr.y (mgr.24437) 17 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:27:52.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:50.469223+0000 mgr.y (mgr.24437) 17 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:27:52.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:50.469223+0000 mgr.y (mgr.24437) 17 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:27:52.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.810+0000 7f1f8b189700 1 -- 172.21.15.87:0/1803352817 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840f7400 msgr2=0x7f1f84103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:52.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.810+0000 7f1f8b189700 1 --2- 172.21.15.87:0/1803352817 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840f7400 0x7f1f84103a20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f1f7c009c30 tx=0x7f1f7c004c50).stop 2024-02-23T00:27:52.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.810+0000 7f1f8b189700 1 -- 172.21.15.87:0/1803352817 shutdown_connections 2024-02-23T00:27:52.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.810+0000 7f1f8b189700 1 --2- 172.21.15.87:0/1803352817 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840f7400 0x7f1f84103a20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.810+0000 7f1f8b189700 1 --2- 172.21.15.87:0/1803352817 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f840fcbb0 0x7f1f840fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.810+0000 7f1f8b189700 1 --2- 172.21.15.87:0/1803352817 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f840fd510 0x7f1f840f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.810+0000 7f1f8b189700 1 -- 172.21.15.87:0/1803352817 >> 172.21.15.87:0/1803352817 conn(0x7f1f840f2820 msgr2=0x7f1f840f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.810+0000 7f1f8b189700 1 -- 172.21.15.87:0/1803352817 shutdown_connections 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.810+0000 7f1f8b189700 1 -- 172.21.15.87:0/1803352817 wait complete. 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 librados: starting msgr at 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 librados: starting objecter 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 Processor -- start 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 -- start start 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 librados: setting wanted keys 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 librados: calling monclient init 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f840f7400 0x7f1f8410e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:52.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840fcbb0 0x7f1f8410e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f840fd510 0x7f1f841081e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f841086d0 con 0x7f1f840f7400 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f84108840 con 0x7f1f840fd510 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f8b189700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f84108a00 con 0x7f1f840fcbb0 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f88f25700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f840f7400 0x7f1f8410e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f83fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840fcbb0 0x7f1f8410e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f83fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840fcbb0 0x7f1f8410e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37858/0 (socket says 172.21.15.87:37858) 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f88f25700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f840f7400 0x7f1f8410e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53058/0 (socket says 172.21.15.87:53058) 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f83fff700 1 -- 172.21.15.87:0/420777370 learned_addr learned my addr 172.21.15.87:0/420777370 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f89726700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f840fd510 0x7f1f841081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f83fff700 1 -- 172.21.15.87:0/420777370 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f840fd510 msgr2=0x7f1f841081e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f83fff700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f840fd510 0x7f1f841081e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f83fff700 1 -- 172.21.15.87:0/420777370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f840f7400 msgr2=0x7f1f8410e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:52.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f83fff700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f840f7400 0x7f1f8410e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f83fff700 1 -- 172.21.15.87:0/420777370 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f70005040 con 0x7f1f840fcbb0 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f88f25700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f840f7400 0x7f1f8410e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.811+0000 7f1f89726700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f840fd510 0x7f1f841081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f83fff700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840fcbb0 0x7f1f8410e530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f1f70009400 tx=0x7f1f70009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f81ffb700 10 client.?.objecter ms_handle_connect 0x7f1f840fcbb0 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f81ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f81ffb700 1 -- 172.21.15.87:0/420777370 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70007a40 con 0x7f1f840fcbb0 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f81ffb700 1 -- 172.21.15.87:0/420777370 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f7000f040 con 0x7f1f840fcbb0 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c0097c0 con 0x7f1f840fcbb0 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f81ffb700 1 -- 172.21.15.87:0/420777370 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f70014aa0 con 0x7f1f840fcbb0 2024-02-23T00:27:52.650 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:27:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f8b189700 10 client.24473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f84109040 con 0x7f1f840fcbb0 2024-02-23T00:27:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.812+0000 7f1f8b189700 1 librados: init done 2024-02-23T00:27:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.813+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8405fa60 con 0x7f1f840fcbb0 2024-02-23T00:27:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.813+0000 7f1f81ffb700 1 -- 172.21.15.87:0/420777370 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1f7000e630 con 0x7f1f840fcbb0 2024-02-23T00:27:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.813+0000 7f1f81ffb700 1 --2- 172.21.15.87:0/420777370 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f74062c90 0x7f1f74065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.813+0000 7f1f81ffb700 1 -- 172.21.15.87:0/420777370 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f1f7007c720 con 0x7f1f840fcbb0 2024-02-23T00:27:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.813+0000 7f1f81ffb700 10 client.24473.objecter ms_dispatch 0x7f1f84001550 osd_map(71..71 src has 1..71) v4 2024-02-23T00:27:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.813+0000 7f1f81ffb700 3 client.24473.objecter handle_osd_map got epochs [71,71] > 0 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.813+0000 7f1f81ffb700 3 client.24473.objecter handle_osd_map decoding full epoch 71 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.813+0000 7f1f88f25700 1 --2- 172.21.15.87:0/420777370 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f74062c90 0x7f1f74065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.814+0000 7f1f81ffb700 20 client.24473.objecter dump_active .. 0 homeless 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.814+0000 7f1f81ffb700 10 client.24473.objecter ms_handle_connect 0x7f1f74062c90 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.814+0000 7f1f88f25700 1 --2- 172.21.15.87:0/420777370 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f74062c90 0x7f1f74065150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f1f780103c0 tx=0x7f1f7800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.818+0000 7f1f81ffb700 1 -- 172.21.15.87:0/420777370 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1f70049f60 con 0x7f1f840fcbb0 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.986+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f84106120 con 0x7f1f840fcbb0 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.987+0000 7f1f81ffb700 1 -- 172.21.15.87:0/420777370 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+19010 (secure 0 0 0) 0x7f1f70050110 con 0x7f1f840fcbb0 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 10 librados: watch_flush enter 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 10 librados: watch_flush exit 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 20 client.24473.objecter shutdown clearing up homeless session... 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 10 client.24473.objecter successfully canceled tick 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f74062c90 msgr2=0x7f1f74065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 --2- 172.21.15.87:0/420777370 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f74062c90 0x7f1f74065150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f1f780103c0 tx=0x7f1f7800b040).stop 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840fcbb0 msgr2=0x7f1f8410e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840fcbb0 0x7f1f8410e530 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f1f70009400 tx=0x7f1f70009cb0).stop 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 shutdown_connections 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 --2- 172.21.15.87:0/420777370 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f74062c90 0x7f1f74065150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1f840fd510 0x7f1f841081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1f840f7400 0x7f1f8410e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 --2- 172.21.15.87:0/420777370 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1f840fcbb0 0x7f1f8410e530 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 >> 172.21.15.87:0/420777370 conn(0x7f1f840f2820 msgr2=0x7f1f840f4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 shutdown_connections 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 -- 172.21.15.87:0/420777370 wait complete. 2024-02-23T00:27:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:51.989+0000 7f1f8b189700 1 librados: shutdown 2024-02-23T00:27:52.654 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":71,"fsid":"66db207c-d1e1-11ee-95c0-87774f69a715","created":"2024-02-23T00:21:28.861318+0000","modified":"2024-02-23T00:27:49.840929+0000","last_up_change":"2024-02-23T00:27:09.633391+0000","last_in_change":"2024-02-23T00:26:52.603467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-23T00:25:09.208309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-23T00:27:15.562186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-23T00:27:15.989206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-23T00:27:17.282440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-23T00:27:19.262164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-23T00:27:21.319951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"e3549bad-2891-47b9-bf77-132cadc20bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":2153191863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":2153191863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":2153191863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":2153191863}]},"public_addr":"172.21.15.87:6801/2153191863","cluster_addr":"172.21.15.87:6802/2153191863","heartbeat_back_addr":"172.21.15.87:6804/2153191863","heartbeat_front_addr":"172.21.15.87:6803/2153191863","state":["exists","up"]},{"osd":1,"uuid":"85525ce8-38d5-4840-8057-547cb176c859","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":685354171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":685354171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":685354171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":685354171}]},"public_addr":"172.21.15.87:6805/685354171","cluster_addr":"172.21.15.87:6806/685354171","heartbeat_back_addr":"172.21.15.87:6808/685354171","heartbeat_front_addr":"172.21.15.87:6807/685354171","state":["exists","up"]},{"osd":2,"uuid":"48ea64b6-ed15-48d2-b0ed-aa4053938aaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3605394228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3605394228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3605394228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3605394228}]},"public_addr":"172.21.15.87:6809/3605394228","cluster_addr":"172.21.15.87:6810/3605394228","heartbeat_back_addr":"172.21.15.87:6812/3605394228","heartbeat_front_addr":"172.21.15.87:6811/3605394228","state":["exists","up"]},{"osd":3,"uuid":"cb8705aa-e114-4dfd-ac9d-ec12f85f877a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":128059790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":128059790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":128059790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":128059790}]},"public_addr":"172.21.15.87:6813/128059790","cluster_addr":"172.21.15.87:6814/128059790","heartbeat_back_addr":"172.21.15.87:6816/128059790","heartbeat_front_addr":"172.21.15.87:6815/128059790","state":["exists","up"]},{"osd":4,"uuid":"c3734a04-7343-4251-9b5b-bedb9e8b882c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4175844757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6801","nonce":4175844757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6803","nonce":4175844757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4175844757}]},"public_addr":"172.21.15.171:6800/4175844757","cluster_addr":"172.21.15.171:6801/4175844757","heartbeat_back_addr":"172.21.15.171:6803/4175844757","heartbeat_front_addr":"172.21.15.171:6802/4175844757","state":["exists","up"]},{"osd":5,"uuid":"fe86232d-96b9-4cbf-9cac-373286fa103d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2550013795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6805","nonce":2550013795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6807","nonce":2550013795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2550013795}]},"public_addr":"172.21.15.171:6804/2550013795","cluster_addr":"172.21.15.171:6805/2550013795","heartbeat_back_addr":"172.21.15.171:6807/2550013795","heartbeat_front_addr":"172.21.15.171:6806/2550013795","state":["exists","up"]},{"osd":6,"uuid":"bfc5dd69-5cfb-409f-985f-83ee0dd4dc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2509408498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2509408498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6811","nonce":2509408498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":2509408498}]},"public_addr":"172.21.15.171:6808/2509408498","cluster_addr":"172.21.15.171:6809/2509408498","heartbeat_back_addr":"172.21.15.171:6811/2509408498","heartbeat_front_addr":"172.21.15.171:6810/2509408498","state":["exists","up"]},{"osd":7,"uuid":"29ddcf97-369a-4a2c-b3e2-142e84ccf939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":373787000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":373787000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6815","nonce":373787000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":373787000}]},"public_addr":"172.21.15.171:6812/373787000","cluster_addr":"172.21.15.171:6813/373787000","heartbeat_back_addr":"172.21.15.171:6815/373787000","heartbeat_front_addr":"172.21.15.171:6814/373787000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:26:49.127385+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:24.499494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:35.753447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:26:44.529285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:23.618927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:38.541075+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:47.010768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:08.063305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:6800/1121018082":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/2138166229":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/27794119":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/1675126043":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/3767016525":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/969918521":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/795095363":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2273151327":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/1685406414":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2093733106":"2024-02-24T00:21:58.144574+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-23T00:27:53.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:51.458406+0000 mgr.y (mgr.24437) 18 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:27:53.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:51.987077+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? 172.21.15.87:0/420777370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:27:53.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:52.302263+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:53.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:52.302312+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:53.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:51.458406+0000 mgr.y (mgr.24437) 18 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:27:53.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:51.987077+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? 172.21.15.87:0/420777370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:27:53.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:52.302263+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:53.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:52.302312+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:53.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:51.458406+0000 mgr.y (mgr.24437) 18 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:27:53.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:51.987077+0000 mon.c (mon.2) 15 : audit [DBG] from='client.? 172.21.15.87:0/420777370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:27:53.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:52.302263+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:27:53.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:52.302312+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:27:53.268 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}] 2024-02-23T00:27:53.268 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2024-02-23T00:27:54.147 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:52.858472+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-23T00:27:54.147 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:52.858472+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-23T00:27:54.147 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:27:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[67003]: level=info ts=2024-02-23T00:27:54.102Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-23T00:27:54.188 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:27:54.191 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:52.858472+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-23T00:27:54.408 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:27:54 smithi087 bash[66975]: 9d3ea2bdbd02e251a75bf985c7ee0610ed681c3525376d8491f236078f51efbf 2024-02-23T00:27:54.409 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:27:54 smithi087 systemd[1]: Started Ceph alertmanager.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:27:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:53.458740+0000 mgr.y (mgr.24437) 19 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:27:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:54.216323+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:54.614191+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:55.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:53.458740+0000 mgr.y (mgr.24437) 19 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:27:55.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:54.216323+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:55.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:54.614191+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:55.084 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:54 smithi087 systemd[1]: Starting Ceph node-exporter.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:27:55.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:53.458740+0000 mgr.y (mgr.24437) 19 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:27:55.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:54.216323+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:55.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:54.614191+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/354109645 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3f40fcbb0 msgr2=0x7fc3f40fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/354109645 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3f40fcbb0 0x7fc3f40fcf90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fc3e8004870 tx=0x7fc3e8009cb0).stop 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/354109645 shutdown_connections 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/354109645 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3f40f7400 0x7fc3f4103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/354109645 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3f40fcbb0 0x7fc3f40fcf90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/354109645 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3f40fd510 0x7fc3f40f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/354109645 >> 172.21.15.87:0/354109645 conn(0x7fc3f40f2820 msgr2=0x7fc3f40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/354109645 shutdown_connections 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/354109645 wait complete. 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 librados: starting msgr at 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.987+0000 7fc3f8c9b700 1 librados: starting objecter 2024-02-23T00:27:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 Processor -- start 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 -- start start 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 librados: setting wanted keys 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 librados: calling monclient init 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3f40f7400 0x7fc3f410d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3f40fcbb0 0x7fc3f410d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3f40fd510 0x7fc3f41072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f41077b0 con 0x7fc3f40f7400 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4107920 con 0x7fc3f40fd510 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f8c9b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4107ae0 con 0x7fc3f40fcbb0 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3f40f7400 0x7fc3f410d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f1d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3f40fcbb0 0x7fc3f410d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3f40f7400 0x7fc3f410d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53074/0 (socket says 172.21.15.87:53074) 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f259c700 1 -- 172.21.15.87:0/3965415777 learned_addr learned my addr 172.21.15.87:0/3965415777 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:27:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f2d9d700 1 --2- 172.21.15.87:0/3965415777 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3f40fd510 0x7fc3f41072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f259c700 1 -- 172.21.15.87:0/3965415777 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3f40fcbb0 msgr2=0x7fc3f410d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f259c700 1 --2- 172.21.15.87:0/3965415777 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3f40fcbb0 0x7fc3f410d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f259c700 1 -- 172.21.15.87:0/3965415777 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3f40fd510 msgr2=0x7fc3f41072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f259c700 1 --2- 172.21.15.87:0/3965415777 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3f40fd510 0x7fc3f41072c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f259c700 1 -- 172.21.15.87:0/3965415777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e4005040 con 0x7fc3f40f7400 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.988+0000 7fc3f1d9b700 1 --2- 172.21.15.87:0/3965415777 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3f40fcbb0 0x7fc3f410d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3f259c700 1 --2- 172.21.15.87:0/3965415777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3f40f7400 0x7fc3f410d120 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc3e8009eb0 tx=0x7fc3e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3e37fe700 10 client.?.objecter ms_handle_connect 0x7fc3f40f7400 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3e37fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3e37fe700 1 -- 172.21.15.87:0/3965415777 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e800e070 con 0x7fc3f40f7400 2024-02-23T00:27:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3e37fe700 1 -- 172.21.15.87:0/3965415777 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3e8013da0 con 0x7fc3f40f7400 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3e37fe700 1 -- 172.21.15.87:0/3965415777 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8019070 con 0x7fc3f40f7400 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e8005040 con 0x7fc3f40f7400 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3f8c9b700 10 client.14553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f4107f80 con 0x7fc3f40f7400 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.989+0000 7fc3f8c9b700 1 librados: init done 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.990+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f4108170 con 0x7fc3f40f7400 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.990+0000 7fc3e37fe700 1 -- 172.21.15.87:0/3965415777 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc3e801e420 con 0x7fc3f40f7400 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.991+0000 7fc3e37fe700 1 --2- 172.21.15.87:0/3965415777 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc3dc062c90 0x7fc3dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.991+0000 7fc3e37fe700 1 -- 172.21.15.87:0/3965415777 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6056+0+0 (secure 0 0 0) 0x7fc3e807d530 con 0x7fc3f40f7400 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.991+0000 7fc3e37fe700 10 client.14553.objecter ms_dispatch 0x7fc3f4001550 osd_map(72..72 src has 1..72) v4 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.991+0000 7fc3e37fe700 3 client.14553.objecter handle_osd_map got epochs [72,72] > 0 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.991+0000 7fc3e37fe700 3 client.14553.objecter handle_osd_map decoding full epoch 72 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.991+0000 7fc3f1d9b700 1 --2- 172.21.15.87:0/3965415777 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc3dc062c90 0x7fc3dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.991+0000 7fc3e37fe700 20 client.14553.objecter dump_active .. 0 homeless 2024-02-23T00:27:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.991+0000 7fc3e37fe700 10 client.14553.objecter ms_handle_connect 0x7fc3dc062c90 2024-02-23T00:27:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.991+0000 7fc3f1d9b700 1 --2- 172.21.15.87:0/3965415777 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc3dc062c90 0x7fc3dc065150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fc3f4108a60 tx=0x7fc3e400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:27:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:55.995+0000 7fc3e37fe700 1 -- 172.21.15.87:0/3965415777 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc3e8051280 con 0x7fc3f40f7400 2024-02-23T00:27:56.076 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:54.218916+0000 mgr.y (mgr.24437) 20 : cephadm [INF] Deploying daemon node-exporter.a on smithi087 2024-02-23T00:27:56.077 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:54.218916+0000 mgr.y (mgr.24437) 20 : cephadm [INF] Deploying daemon node-exporter.a on smithi087 2024-02-23T00:27:56.077 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:55 smithi087 bash[67322]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-23T00:27:56.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.162+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc3f405fa60 con 0x7fc3f40f7400 2024-02-23T00:27:56.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.163+0000 7fc3e37fe700 1 -- 172.21.15.87:0/3965415777 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v72) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc3e8023020 con 0x7fc3f40f7400 2024-02-23T00:27:56.164 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 1 2024-02-23T00:27:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.164+0000 7fc3f8c9b700 10 librados: watch_flush enter 2024-02-23T00:27:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.164+0000 7fc3f8c9b700 10 librados: watch_flush exit 2024-02-23T00:27:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.164+0000 7fc3f8c9b700 20 client.14553.objecter shutdown clearing up homeless session... 2024-02-23T00:27:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.164+0000 7fc3f8c9b700 10 client.14553.objecter successfully canceled tick 2024-02-23T00:27:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc3dc062c90 msgr2=0x7fc3dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/3965415777 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc3dc062c90 0x7fc3dc065150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fc3f4108a60 tx=0x7fc3e400b040).stop 2024-02-23T00:27:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3f40f7400 msgr2=0x7fc3f410d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:27:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/3965415777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3f40f7400 0x7fc3f410d120 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc3e8009eb0 tx=0x7fc3e800b040).stop 2024-02-23T00:27:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 shutdown_connections 2024-02-23T00:27:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/3965415777 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc3dc062c90 0x7fc3dc065150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/3965415777 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3f40fd510 0x7fc3f41072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/3965415777 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3f40f7400 0x7fc3f410d120 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 --2- 172.21.15.87:0/3965415777 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3f40fcbb0 0x7fc3f410d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:27:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 >> 172.21.15.87:0/3965415777 conn(0x7fc3f40f2820 msgr2=0x7fc3f40fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:27:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 shutdown_connections 2024-02-23T00:27:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 -- 172.21.15.87:0/3965415777 wait complete. 2024-02-23T00:27:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:27:56.165+0000 7fc3f8c9b700 1 librados: shutdown 2024-02-23T00:27:56.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:54.218916+0000 mgr.y (mgr.24437) 20 : cephadm [INF] Deploying daemon node-exporter.a on smithi087 2024-02-23T00:27:56.333 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:56 smithi087 bash[67322]: Getting image source signatures 2024-02-23T00:27:56.333 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:56 smithi087 bash[67322]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-23T00:27:56.333 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:56 smithi087 bash[67322]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-23T00:27:56.333 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:56 smithi087 bash[67322]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-23T00:27:56.333 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:27:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[67003]: level=info ts=2024-02-23T00:27:56.075Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000177308s 2024-02-23T00:27:57.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:55.459211+0000 mgr.y (mgr.24437) 21 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-23T00:27:57.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:56.163499+0000 mon.a (mon.0) 666 : audit [DBG] from='client.? 172.21.15.87:0/3965415777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-23T00:27:57.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:55.459211+0000 mgr.y (mgr.24437) 21 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-23T00:27:57.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:56.163499+0000 mon.a (mon.0) 666 : audit [DBG] from='client.? 172.21.15.87:0/3965415777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-23T00:27:57.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:55.459211+0000 mgr.y (mgr.24437) 21 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-23T00:27:57.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:56.163499+0000 mon.a (mon.0) 666 : audit [DBG] from='client.? 172.21.15.87:0/3965415777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-23T00:27:57.391 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool get .rgw.root pg_num 2024-02-23T00:27:57.582 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:57 smithi087 bash[67322]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-23T00:27:57.582 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:57 smithi087 bash[67322]: Writing manifest to image destination 2024-02-23T00:27:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:27:58.549 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.229Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-23T00:27:58.549 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.229Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-23T00:27:58.549 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.230Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-23T00:27:58.549 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.230Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-23T00:27:58.549 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-23T00:27:58.549 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=arp 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=edac 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-23T00:27:58.550 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=os 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-23T00:27:58.551 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=stat 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=time 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=uname 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-23T00:27:58.552 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-23T00:27:58.553 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-23T00:27:58.553 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-23T00:27:58.553 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a[67472]: ts=2024-02-23T00:27:58.231Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-23T00:27:58.553 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 bash[67322]: aed9b37913ca3c81d085c6fdbb0b0826256d0a13567ec0447b0e11ba9fbe2782 2024-02-23T00:27:58.553 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:27:58 smithi087 systemd[1]: Started Ceph node-exporter.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:27:59.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:57.459662+0000 mgr.y (mgr.24437) 22 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:27:59.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:57.665253+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:27:59.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:57.668199+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:27:59.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:27:58.375284+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:59.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:27:58 smithi171 systemd[1]: Starting Ceph node-exporter.b for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:27:59.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:27:59 smithi171 bash[62419]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-23T00:27:59.277 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:57.459662+0000 mgr.y (mgr.24437) 22 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:27:59.277 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:57.665253+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:27:59.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:57.668199+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:27:59.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:27:58.375284+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:59.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:57.459662+0000 mgr.y (mgr.24437) 22 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:27:59.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:57.665253+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:27:59.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:57.668199+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:27:59.278 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:27:58.375284+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:27:59.692 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:27:59 smithi171 bash[62419]: Getting image source signatures 2024-02-23T00:27:59.692 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:27:59 smithi171 bash[62419]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-23T00:27:59.692 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:27:59 smithi171 bash[62419]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-23T00:27:59.692 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:27:59 smithi171 bash[62419]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-23T00:28:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 -- 172.21.15.87:0/3170261898 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 msgr2=0x7f91140f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 --2- 172.21.15.87:0/3170261898 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 0x7f91140f9090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f90fc004870 tx=0x7f90fc009cb0).stop 2024-02-23T00:28:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 -- 172.21.15.87:0/3170261898 shutdown_connections 2024-02-23T00:28:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 --2- 172.21.15.87:0/3170261898 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9114102210 0x7f9114104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 --2- 172.21.15.87:0/3170261898 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 0x7f91140f9090 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 --2- 172.21.15.87:0/3170261898 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91140f9610 0x7f9114101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 -- 172.21.15.87:0/3170261898 >> 172.21.15.87:0/3170261898 conn(0x7f91140f2820 msgr2=0x7f91140f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 -- 172.21.15.87:0/3170261898 shutdown_connections 2024-02-23T00:28:00.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 -- 172.21.15.87:0/3170261898 wait complete. 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 librados: starting msgr at 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 librados: starting objecter 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 Processor -- start 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 -- start start 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 librados: setting wanted keys 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.138+0000 7f911a8d6700 1 librados: calling monclient init 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f911a8d6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 0x7f911410e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f911a8d6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91140f9610 0x7f911410e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f911a8d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9114102210 0x7f9114108340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f911a8d6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9114108830 con 0x7f9114102210 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f911a8d6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91141089a0 con 0x7f91140f9610 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f911a8d6700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9114108b60 con 0x7f91140f8cb0 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9113fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 0x7f911410e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9113fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 0x7f911410e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37890/0 (socket says 172.21.15.87:37890) 2024-02-23T00:28:00.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9113fff700 1 -- 172.21.15.87:0/745811701 learned_addr learned my addr 172.21.15.87:0/745811701 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9118e73700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9114102210 0x7f9114108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f91137fe700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91140f9610 0x7f911410e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9113fff700 1 -- 172.21.15.87:0/745811701 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91140f9610 msgr2=0x7f911410e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9113fff700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91140f9610 0x7f911410e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9113fff700 1 -- 172.21.15.87:0/745811701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9114102210 msgr2=0x7f9114108340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9113fff700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9114102210 0x7f9114108340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9113fff700 1 -- 172.21.15.87:0/745811701 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c0097c0 con 0x7f91140f8cb0 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9118e73700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9114102210 0x7f9114108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f9113fff700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 0x7f911410e1a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f90fc009eb0 tx=0x7f90fc00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f91117fa700 10 client.?.objecter ms_handle_connect 0x7f91140f8cb0 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f91117fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f91117fa700 1 -- 172.21.15.87:0/745811701 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc00e070 con 0x7f91140f8cb0 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f91117fa700 1 -- 172.21.15.87:0/745811701 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90fc013da0 con 0x7f91140f8cb0 2024-02-23T00:28:00.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.139+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc005040 con 0x7f91140f8cb0 2024-02-23T00:28:00.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.140+0000 7f91117fa700 1 -- 172.21.15.87:0/745811701 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc019070 con 0x7f91140f8cb0 2024-02-23T00:28:00.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.140+0000 7f911a8d6700 10 client.24485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:00.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.140+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9114109000 con 0x7f91140f8cb0 2024-02-23T00:28:00.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.140+0000 7f911a8d6700 1 librados: init done 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.140+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91141091f0 con 0x7f91140f8cb0 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.141+0000 7f91117fa700 1 -- 172.21.15.87:0/745811701 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f90fc01e420 con 0x7f91140f8cb0 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.141+0000 7f91117fa700 1 --2- 172.21.15.87:0/745811701 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9100062c90 0x7f9100065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.141+0000 7f91137fe700 1 --2- 172.21.15.87:0/745811701 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9100062c90 0x7f9100065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.141+0000 7f91117fa700 1 -- 172.21.15.87:0/745811701 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f90fc07d4c0 con 0x7f91140f8cb0 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.141+0000 7f91117fa700 10 client.24485.objecter ms_dispatch 0x7f9114001550 osd_map(72..72 src has 1..72) v4 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.141+0000 7f91117fa700 3 client.24485.objecter handle_osd_map got epochs [72,72] > 0 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.141+0000 7f91117fa700 3 client.24485.objecter handle_osd_map decoding full epoch 72 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.142+0000 7f91117fa700 20 client.24485.objecter dump_active .. 0 homeless 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.142+0000 7f91117fa700 10 client.24485.objecter ms_handle_connect 0x7f9100062c90 2024-02-23T00:28:00.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.142+0000 7f91137fe700 1 --2- 172.21.15.87:0/745811701 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9100062c90 0x7f9100065150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9114109ae0 tx=0x7f910400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:00.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.146+0000 7f91117fa700 1 -- 172.21.15.87:0/745811701 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f90fc051280 con 0x7f91140f8cb0 2024-02-23T00:28:00.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:27:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:27:58.376109+0000 mgr.y (mgr.24437) 23 : cephadm [INF] Deploying daemon node-exporter.b on smithi171 2024-02-23T00:28:00.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.308+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"} v 0) v1 -- 0x7f911405fa60 con 0x7f91140f8cb0 2024-02-23T00:28:00.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.308+0000 7f91117fa700 1 -- 172.21.15.87:0/745811701 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]=0 v72) v1 ==== 98+0+11 (secure 0 0 0) 0x7f911405fa60 con 0x7f91140f8cb0 2024-02-23T00:28:00.313 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-02-23T00:28:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 10 librados: watch_flush enter 2024-02-23T00:28:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 10 librados: watch_flush exit 2024-02-23T00:28:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 20 client.24485.objecter shutdown clearing up homeless session... 2024-02-23T00:28:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 10 client.24485.objecter successfully canceled tick 2024-02-23T00:28:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9100062c90 msgr2=0x7f9100065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 --2- 172.21.15.87:0/745811701 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9100062c90 0x7f9100065150 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9114109ae0 tx=0x7f910400b040).stop 2024-02-23T00:28:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 msgr2=0x7f911410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 0x7f911410e1a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f90fc009eb0 tx=0x7f90fc00b040).stop 2024-02-23T00:28:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 shutdown_connections 2024-02-23T00:28:00.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 --2- 172.21.15.87:0/745811701 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9100062c90 0x7f9100065150 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:00.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9114102210 0x7f9114108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:00.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91140f8cb0 0x7f911410e1a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:00.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 --2- 172.21.15.87:0/745811701 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91140f9610 0x7f911410e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:00.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 >> 172.21.15.87:0/745811701 conn(0x7f91140f2820 msgr2=0x7f91140f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:00.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 shutdown_connections 2024-02-23T00:28:00.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 -- 172.21.15.87:0/745811701 wait complete. 2024-02-23T00:28:00.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:00.310+0000 7f911a8d6700 1 librados: shutdown 2024-02-23T00:28:00.331 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:27:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:27:58.376109+0000 mgr.y (mgr.24437) 23 : cephadm [INF] Deploying daemon node-exporter.b on smithi171 2024-02-23T00:28:00.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:27:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:27:58.376109+0000 mgr.y (mgr.24437) 23 : cephadm [INF] Deploying daemon node-exporter.b on smithi171 2024-02-23T00:28:00.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:28:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:28:01.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:27:59.460129+0000 mgr.y (mgr.24437) 24 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:28:01.073 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:00.309894+0000 mon.c (mon.2) 16 : audit [DBG] from='client.? 172.21.15.87:0/745811701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-23T00:28:01.073 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:27:59.460129+0000 mgr.y (mgr.24437) 24 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:28:01.073 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:00.309894+0000 mon.c (mon.2) 16 : audit [DBG] from='client.? 172.21.15.87:0/745811701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-23T00:28:01.093 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool get iscsi pg_num 2024-02-23T00:28:01.189 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:00 smithi171 bash[62419]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-23T00:28:01.189 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:27:59.460129+0000 mgr.y (mgr.24437) 24 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:28:01.189 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:00.309894+0000 mon.c (mon.2) 16 : audit [DBG] from='client.? 172.21.15.87:0/745811701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-23T00:28:01.360 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:01.442 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 bash[62419]: Writing manifest to image destination 2024-02-23T00:28:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:00.480156+0000 mgr.y (mgr.24437) 25 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:02.192 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.908Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.908Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=arp 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-23T00:28:02.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=edac 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-23T00:28:02.194 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=os 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=stat 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-23T00:28:02.195 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=time 2024-02-23T00:28:02.196 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-23T00:28:02.196 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=uname 2024-02-23T00:28:02.196 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-23T00:28:02.196 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-23T00:28:02.196 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.909Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-23T00:28:02.196 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.910Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-23T00:28:02.196 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b[62477]: ts=2024-02-23T00:28:01.910Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-23T00:28:02.196 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:02 smithi171 bash[62419]: 5084d4ab5b9aa2968b5ede48c6b812e6ddf7cbfa71158224c3472dc08ac3b6ee 2024-02-23T00:28:02.196 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:28:02 smithi171 systemd[1]: Started Ceph node-exporter.b for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:28:02.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:00.480156+0000 mgr.y (mgr.24437) 25 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:02.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:00.480156+0000 mgr.y (mgr.24437) 25 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1704044046 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f8cb0 msgr2=0x7f4e180f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1704044046 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f8cb0 0x7f4e180f9090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f4e080047b0 tx=0x7f4e08009cb0).stop 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1704044046 shutdown_connections 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1704044046 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e18102210 0x7f4e18104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1704044046 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f8cb0 0x7f4e180f9090 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1704044046 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e180f9610 0x7f4e18101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1704044046 >> 172.21.15.87:0/1704044046 conn(0x7f4e180f2820 msgr2=0x7f4e180f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1704044046 shutdown_connections 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1704044046 wait complete. 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 librados: starting msgr at 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 librados: starting objecter 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 Processor -- start 2024-02-23T00:28:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 -- start start 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 librados: setting wanted keys 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.288+0000 7f4e1e8f5700 1 librados: calling monclient init 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e180f8cb0 0x7f4e1810e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f9610 0x7f4e1810e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e18102210 0x7f4e181083e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e181088d0 con 0x7f4e180f9610 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e18108a40 con 0x7f4e180f8cb0 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e18108c00 con 0x7f4e18102210 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e0ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f9610 0x7f4e1810e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1ce92700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e18102210 0x7f4e181083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e0ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f9610 0x7f4e1810e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40250/0 (socket says 172.21.15.87:40250) 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e0ffff700 1 -- 172.21.15.87:0/1790751214 learned_addr learned my addr 172.21.15.87:0/1790751214 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e17fff700 1 --2- 172.21.15.87:0/1790751214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e180f8cb0 0x7f4e1810e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e0ffff700 1 -- 172.21.15.87:0/1790751214 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e18102210 msgr2=0x7f4e181083e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e0ffff700 1 --2- 172.21.15.87:0/1790751214 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e18102210 0x7f4e181083e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e0ffff700 1 -- 172.21.15.87:0/1790751214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e180f8cb0 msgr2=0x7f4e1810e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e0ffff700 1 --2- 172.21.15.87:0/1790751214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e180f8cb0 0x7f4e1810e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e0ffff700 1 -- 172.21.15.87:0/1790751214 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100097c0 con 0x7f4e180f9610 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1ce92700 1 --2- 172.21.15.87:0/1790751214 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e18102210 0x7f4e181083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e0ffff700 1 --2- 172.21.15.87:0/1790751214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f9610 0x7f4e1810e730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4e0000e970 tx=0x7f4e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e15ffb700 10 client.?.objecter ms_handle_connect 0x7f4e180f9610 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e15ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e15ffb700 1 -- 172.21.15.87:0/1790751214 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00004660 con 0x7f4e180f9610 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e15ffb700 1 -- 172.21.15.87:0/1790751214 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e0000f040 con 0x7f4e180f9610 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e08005040 con 0x7f4e180f9610 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e15ffb700 1 -- 172.21.15.87:0/1790751214 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e00014790 con 0x7f4e180f9610 2024-02-23T00:28:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 10 client.14565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:03.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e18115070 con 0x7f4e180f9610 2024-02-23T00:28:03.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.289+0000 7f4e1e8f5700 1 librados: init done 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.290+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1805fa60 con 0x7f4e180f9610 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e15ffb700 1 -- 172.21.15.87:0/1790751214 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4e000090d0 con 0x7f4e180f9610 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e15ffb700 1 --2- 172.21.15.87:0/1790751214 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e04062c90 0x7f4e04065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e15ffb700 1 -- 172.21.15.87:0/1790751214 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f4e0007c560 con 0x7f4e180f9610 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e15ffb700 10 client.14565.objecter ms_dispatch 0x7f4e18001550 osd_map(72..72 src has 1..72) v4 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e15ffb700 3 client.14565.objecter handle_osd_map got epochs [72,72] > 0 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e15ffb700 3 client.14565.objecter handle_osd_map decoding full epoch 72 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e17fff700 1 --2- 172.21.15.87:0/1790751214 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e04062c90 0x7f4e04065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e15ffb700 20 client.14565.objecter dump_active .. 0 homeless 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e15ffb700 10 client.14565.objecter ms_handle_connect 0x7f4e04062c90 2024-02-23T00:28:03.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.291+0000 7f4e17fff700 1 --2- 172.21.15.87:0/1790751214 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e04062c90 0x7f4e04065150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f4e08005010 tx=0x7f4e08009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:03.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.295+0000 7f4e15ffb700 1 -- 172.21.15.87:0/1790751214 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4e0004af60 con 0x7f4e180f9610 2024-02-23T00:28:03.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:01.460821+0000 mgr.y (mgr.24437) 26 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:03.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:02.064180+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:03.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:28:02.074645+0000 mgr.y (mgr.24437) 27 : cephadm [INF] Deploying daemon prometheus.a on smithi171 2024-02-23T00:28:03.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:01.460821+0000 mgr.y (mgr.24437) 26 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:03.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:02.064180+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:03.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:28:02.074645+0000 mgr.y (mgr.24437) 27 : cephadm [INF] Deploying daemon prometheus.a on smithi171 2024-02-23T00:28:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:01.460821+0000 mgr.y (mgr.24437) 26 : cluster [DBG] pgmap v20: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:02.064180+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:28:02.074645+0000 mgr.y (mgr.24437) 27 : cephadm [INF] Deploying daemon prometheus.a on smithi171 2024-02-23T00:28:03.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.458+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"} v 0) v1 -- 0x7f4e18107290 con 0x7f4e180f9610 2024-02-23T00:28:03.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.458+0000 7f4e15ffb700 1 -- 172.21.15.87:0/1790751214 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]=0 v72) v1 ==== 94+0+10 (secure 0 0 0) 0x7f4e000190a0 con 0x7f4e180f9610 2024-02-23T00:28:03.460 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 3 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.459+0000 7f4e1e8f5700 10 librados: watch_flush enter 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 10 librados: watch_flush exit 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 20 client.14565.objecter shutdown clearing up homeless session... 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 10 client.14565.objecter successfully canceled tick 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e04062c90 msgr2=0x7f4e04065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1790751214 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e04062c90 0x7f4e04065150 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f4e08005010 tx=0x7f4e08009cb0).stop 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f9610 msgr2=0x7f4e1810e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1790751214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f9610 0x7f4e1810e730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4e0000e970 tx=0x7f4e0000b040).stop 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 shutdown_connections 2024-02-23T00:28:03.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1790751214 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e04062c90 0x7f4e04065150 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:03.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1790751214 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e18102210 0x7f4e181083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:03.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1790751214 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e180f8cb0 0x7f4e1810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:03.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 --2- 172.21.15.87:0/1790751214 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e180f9610 0x7f4e1810e730 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:03.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 >> 172.21.15.87:0/1790751214 conn(0x7f4e180f2820 msgr2=0x7f4e180f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:03.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 shutdown_connections 2024-02-23T00:28:03.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 -- 172.21.15.87:0/1790751214 wait complete. 2024-02-23T00:28:03.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:03.460+0000 7f4e1e8f5700 1 librados: shutdown 2024-02-23T00:28:04.098 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool get default.rgw.log pg_num 2024-02-23T00:28:04.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:03.459669+0000 mon.a (mon.0) 669 : audit [DBG] from='client.? 172.21.15.87:0/1790751214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-23T00:28:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:03.459669+0000 mon.a (mon.0) 669 : audit [DBG] from='client.? 172.21.15.87:0/1790751214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-23T00:28:04.333 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[67003]: level=info ts=2024-02-23T00:28:04.077Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001720901s 2024-02-23T00:28:04.367 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:03.459669+0000 mon.a (mon.0) 669 : audit [DBG] from='client.? 172.21.15.87:0/1790751214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-23T00:28:05.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:03.461159+0000 mgr.y (mgr.24437) 28 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 965 B/s rd, 0 op/s 2024-02-23T00:28:05.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:04.620510+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:05.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:03.461159+0000 mgr.y (mgr.24437) 28 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 965 B/s rd, 0 op/s 2024-02-23T00:28:05.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:04.620510+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:05.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:03.461159+0000 mgr.y (mgr.24437) 28 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 965 B/s rd, 0 op/s 2024-02-23T00:28:05.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:04.620510+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:06.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.204+0000 7f9758001700 1 -- 172.21.15.87:0/3277902060 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97500fd420 msgr2=0x7f97500f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:06.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.204+0000 7f9758001700 1 --2- 172.21.15.87:0/3277902060 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97500fd420 0x7f97500f6c50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f974c004870 tx=0x7f974c009cb0).stop 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.205+0000 7f9758001700 1 -- 172.21.15.87:0/3277902060 shutdown_connections 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.205+0000 7f9758001700 1 --2- 172.21.15.87:0/3277902060 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97500f7330 0x7f9750103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.205+0000 7f9758001700 1 --2- 172.21.15.87:0/3277902060 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97500fcac0 0x7f97500fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.205+0000 7f9758001700 1 --2- 172.21.15.87:0/3277902060 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97500fd420 0x7f97500f6c50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.205+0000 7f9758001700 1 -- 172.21.15.87:0/3277902060 >> 172.21.15.87:0/3277902060 conn(0x7f97500f26f0 msgr2=0x7f97500f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.205+0000 7f9758001700 1 -- 172.21.15.87:0/3277902060 shutdown_connections 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.205+0000 7f9758001700 1 -- 172.21.15.87:0/3277902060 wait complete. 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.205+0000 7f9758001700 1 librados: starting msgr at 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.205+0000 7f9758001700 1 librados: starting objecter 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 Processor -- start 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 -- start start 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 librados: setting wanted keys 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 librados: calling monclient init 2024-02-23T00:28:06.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97500f7330 0x7f975010cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97500fcac0 0x7f975010d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97500fd420 0x7f9750107120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750107610 con 0x7f97500fcac0 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750107780 con 0x7f97500f7330 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9758001700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9750107940 con 0x7f97500fd420 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975559c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97500fcac0 0x7f975010d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975559c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97500fcac0 0x7f975010d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40274/0 (socket says 172.21.15.87:40274) 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975559c700 1 -- 172.21.15.87:0/3562358615 learned_addr learned my addr 172.21.15.87:0/3562358615 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975659e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97500fd420 0x7f9750107120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9755d9d700 1 --2- 172.21.15.87:0/3562358615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97500f7330 0x7f975010cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975559c700 1 -- 172.21.15.87:0/3562358615 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97500fd420 msgr2=0x7f9750107120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975559c700 1 --2- 172.21.15.87:0/3562358615 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97500fd420 0x7f9750107120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975559c700 1 -- 172.21.15.87:0/3562358615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97500f7330 msgr2=0x7f975010cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975559c700 1 --2- 172.21.15.87:0/3562358615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97500f7330 0x7f975010cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975559c700 1 -- 172.21.15.87:0/3562358615 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9740005040 con 0x7f97500fcac0 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975659e700 1 --2- 172.21.15.87:0/3562358615 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97500fd420 0x7f9750107120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f975559c700 1 --2- 172.21.15.87:0/3562358615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97500fcac0 0x7f975010d470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f974c009eb0 tx=0x7f974c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9746ffd700 10 client.?.objecter ms_handle_connect 0x7f97500fcac0 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.206+0000 7f9746ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.207+0000 7f9746ffd700 1 -- 172.21.15.87:0/3562358615 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c00e070 con 0x7f97500fcac0 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.207+0000 7f9746ffd700 1 -- 172.21.15.87:0/3562358615 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f974c013da0 con 0x7f97500fcac0 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.207+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c005040 con 0x7f97500fcac0 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.207+0000 7f9746ffd700 1 -- 172.21.15.87:0/3562358615 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974c019070 con 0x7f97500fcac0 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.207+0000 7f9758001700 10 client.14571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.207+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9750107eb0 con 0x7f97500fcac0 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.207+0000 7f9758001700 1 librados: init done 2024-02-23T00:28:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.207+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97500606d0 con 0x7f97500fcac0 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.208+0000 7f9746ffd700 1 -- 172.21.15.87:0/3562358615 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f974c01e420 con 0x7f97500fcac0 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.208+0000 7f9746ffd700 1 --2- 172.21.15.87:0/3562358615 >> v2:172.21.15.87:6800/1231597463 conn(0x7f973c062c90 0x7f973c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.208+0000 7f9755d9d700 1 --2- 172.21.15.87:0/3562358615 >> v2:172.21.15.87:6800/1231597463 conn(0x7f973c062c90 0x7f973c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.208+0000 7f9746ffd700 1 -- 172.21.15.87:0/3562358615 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f974c07d470 con 0x7f97500fcac0 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.208+0000 7f9746ffd700 10 client.14571.objecter ms_dispatch 0x7f9750001550 osd_map(72..72 src has 1..72) v4 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.208+0000 7f9746ffd700 3 client.14571.objecter handle_osd_map got epochs [72,72] > 0 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.208+0000 7f9746ffd700 3 client.14571.objecter handle_osd_map decoding full epoch 72 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.209+0000 7f9746ffd700 20 client.14571.objecter dump_active .. 0 homeless 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.209+0000 7f9746ffd700 10 client.14571.objecter ms_handle_connect 0x7f973c062c90 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.209+0000 7f9755d9d700 1 --2- 172.21.15.87:0/3562358615 >> v2:172.21.15.87:6800/1231597463 conn(0x7f973c062c90 0x7f973c065150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9740009350 tx=0x7f974000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.213+0000 7f9746ffd700 1 -- 172.21.15.87:0/3562358615 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f974c051150 con 0x7f97500fcac0 2024-02-23T00:28:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.375+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"} v 0) v1 -- 0x7f9750105fb0 con 0x7f97500fcac0 2024-02-23T00:28:06.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.375+0000 7f9746ffd700 1 -- 172.21.15.87:0/3562358615 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7f974c0230a0 con 0x7f97500fcac0 2024-02-23T00:28:06.378 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-02-23T00:28:06.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 10 librados: watch_flush enter 2024-02-23T00:28:06.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 10 librados: watch_flush exit 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 20 client.14571.objecter shutdown clearing up homeless session... 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 10 client.14571.objecter successfully canceled tick 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 >> v2:172.21.15.87:6800/1231597463 conn(0x7f973c062c90 msgr2=0x7f973c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 --2- 172.21.15.87:0/3562358615 >> v2:172.21.15.87:6800/1231597463 conn(0x7f973c062c90 0x7f973c065150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9740009350 tx=0x7f974000b040).stop 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97500fcac0 msgr2=0x7f975010d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 --2- 172.21.15.87:0/3562358615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97500fcac0 0x7f975010d470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f974c009eb0 tx=0x7f974c00b040).stop 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 shutdown_connections 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 --2- 172.21.15.87:0/3562358615 >> v2:172.21.15.87:6800/1231597463 conn(0x7f973c062c90 0x7f973c065150 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 --2- 172.21.15.87:0/3562358615 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97500fd420 0x7f9750107120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 --2- 172.21.15.87:0/3562358615 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97500f7330 0x7f975010cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 --2- 172.21.15.87:0/3562358615 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97500fcac0 0x7f975010d470 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 >> 172.21.15.87:0/3562358615 conn(0x7f97500f26f0 msgr2=0x7f97500f41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 shutdown_connections 2024-02-23T00:28:06.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 -- 172.21.15.87:0/3562358615 wait complete. 2024-02-23T00:28:06.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:06.377+0000 7f9758001700 1 librados: shutdown 2024-02-23T00:28:07.232 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:05.461658+0000 mgr.y (mgr.24437) 29 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:07.232 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:06.377126+0000 mon.a (mon.0) 671 : audit [DBG] from='client.? 172.21.15.87:0/3562358615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-23T00:28:07.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:05.461658+0000 mgr.y (mgr.24437) 29 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:07.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:06.377126+0000 mon.a (mon.0) 671 : audit [DBG] from='client.? 172.21.15.87:0/3562358615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-23T00:28:07.253 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool get default.rgw.control pg_num 2024-02-23T00:28:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:05.461658+0000 mgr.y (mgr.24437) 29 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:06.377126+0000 mon.a (mon.0) 671 : audit [DBG] from='client.? 172.21.15.87:0/3562358615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-23T00:28:07.522 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:09.236 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:07.461982+0000 mgr.y (mgr.24437) 30 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:28:09.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:07.461982+0000 mgr.y (mgr.24437) 30 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:28:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:07.461982+0000 mgr.y (mgr.24437) 30 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:28:09.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.498+0000 7f2945d60700 1 -- 172.21.15.87:0/2426138751 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fdbb0 msgr2=0x7f29400fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.498+0000 7f2945d60700 1 --2- 172.21.15.87:0/2426138751 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fdbb0 0x7f29400fdf90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f2930004870 tx=0x7f2930009cb0).stop 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.498+0000 7f2945d60700 1 -- 172.21.15.87:0/2426138751 shutdown_connections 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.498+0000 7f2945d60700 1 --2- 172.21.15.87:0/2426138751 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29400f8420 0x7f2940104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.498+0000 7f2945d60700 1 --2- 172.21.15.87:0/2426138751 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fdbb0 0x7f29400fdf90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.498+0000 7f2945d60700 1 --2- 172.21.15.87:0/2426138751 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29400fe510 0x7f29400f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.498+0000 7f2945d60700 1 -- 172.21.15.87:0/2426138751 >> 172.21.15.87:0/2426138751 conn(0x7f29400f3800 msgr2=0x7f29400f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.498+0000 7f2945d60700 1 -- 172.21.15.87:0/2426138751 shutdown_connections 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.498+0000 7f2945d60700 1 -- 172.21.15.87:0/2426138751 wait complete. 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.499+0000 7f2945d60700 1 librados: starting msgr at 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.499+0000 7f2945d60700 1 librados: starting objecter 2024-02-23T00:28:09.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.499+0000 7f2945d60700 1 Processor -- start 2024-02-23T00:28:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.499+0000 7f2945d60700 1 -- start start 2024-02-23T00:28:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.499+0000 7f2945d60700 1 librados: setting wanted keys 2024-02-23T00:28:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.499+0000 7f2945d60700 1 librados: calling monclient init 2024-02-23T00:28:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29400f8420 0x7f294010e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29400fdbb0 0x7f294010e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fe510 0x7f2940108280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2940108770 con 0x7f29400f8420 2024-02-23T00:28:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29401088e0 con 0x7f29400fdbb0 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2940108aa0 con 0x7f29400fe510 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fe510 0x7f2940108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fe510 0x7f2940108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50958/0 (socket says 172.21.15.87:50958) 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293ffff700 1 -- 172.21.15.87:0/2947194805 learned_addr learned my addr 172.21.15.87:0/2947194805 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293f7fe700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29400f8420 0x7f294010e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293effd700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29400fdbb0 0x7f294010e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293ffff700 1 -- 172.21.15.87:0/2947194805 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29400fdbb0 msgr2=0x7f294010e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293ffff700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29400fdbb0 0x7f294010e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293ffff700 1 -- 172.21.15.87:0/2947194805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29400f8420 msgr2=0x7f294010e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293ffff700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29400f8420 0x7f294010e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293ffff700 1 -- 172.21.15.87:0/2947194805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29380097c0 con 0x7f29400fe510 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293effd700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29400fdbb0 0x7f294010e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293ffff700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fe510 0x7f2940108280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2938001df0 tx=0x7f2938004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293cff9700 10 client.?.objecter ms_handle_connect 0x7f29400fe510 2024-02-23T00:28:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293cff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293cff9700 1 -- 172.21.15.87:0/2947194805 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938009a10 con 0x7f29400fe510 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293cff9700 1 -- 172.21.15.87:0/2947194805 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29380095a0 con 0x7f29400fe510 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2930005040 con 0x7f29400fe510 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f293cff9700 1 -- 172.21.15.87:0/2947194805 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2938007da0 con 0x7f29400fe510 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 10 client.24503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2940114f10 con 0x7f29400fe510 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.500+0000 7f2945d60700 1 librados: init done 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.501+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294005ee80 con 0x7f29400fe510 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293cff9700 1 -- 172.21.15.87:0/2947194805 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f29380090d0 con 0x7f29400fe510 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293cff9700 1 --2- 172.21.15.87:0/2947194805 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2928062c90 0x7f2928065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293cff9700 1 -- 172.21.15.87:0/2947194805 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f293807d2d0 con 0x7f29400fe510 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293cff9700 10 client.24503.objecter ms_dispatch 0x7f2940001550 osd_map(72..72 src has 1..72) v4 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293cff9700 3 client.24503.objecter handle_osd_map got epochs [72,72] > 0 2024-02-23T00:28:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293cff9700 3 client.24503.objecter handle_osd_map decoding full epoch 72 2024-02-23T00:28:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293f7fe700 1 --2- 172.21.15.87:0/2947194805 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2928062c90 0x7f2928065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293cff9700 20 client.24503.objecter dump_active .. 0 homeless 2024-02-23T00:28:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293cff9700 10 client.24503.objecter ms_handle_connect 0x7f2928062c90 2024-02-23T00:28:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.502+0000 7f293f7fe700 1 --2- 172.21.15.87:0/2947194805 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2928062c90 0x7f2928065150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f29300043d0 tx=0x7f293000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:09.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.506+0000 7f293cff9700 1 -- 172.21.15.87:0/2947194805 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2938051050 con 0x7f29400fe510 2024-02-23T00:28:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.674+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"} v 0) v1 -- 0x7f2940108d80 con 0x7f29400fe510 2024-02-23T00:28:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.674+0000 7f293cff9700 1 -- 172.21.15.87:0/2947194805 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]=0 v72) v1 ==== 108+0+11 (secure 0 0 0) 0x7f2940108d80 con 0x7f29400fe510 2024-02-23T00:28:09.676 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-02-23T00:28:09.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 10 librados: watch_flush enter 2024-02-23T00:28:09.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 10 librados: watch_flush exit 2024-02-23T00:28:09.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 20 client.24503.objecter shutdown clearing up homeless session... 2024-02-23T00:28:09.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 10 client.24503.objecter successfully canceled tick 2024-02-23T00:28:09.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2928062c90 msgr2=0x7f2928065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 --2- 172.21.15.87:0/2947194805 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2928062c90 0x7f2928065150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f29300043d0 tx=0x7f293000b040).stop 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fe510 msgr2=0x7f2940108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fe510 0x7f2940108280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2938001df0 tx=0x7f2938004c50).stop 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 shutdown_connections 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 --2- 172.21.15.87:0/2947194805 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2928062c90 0x7f2928065150 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29400fe510 0x7f2940108280 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29400f8420 0x7f294010e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 --2- 172.21.15.87:0/2947194805 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29400fdbb0 0x7f294010e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 >> 172.21.15.87:0/2947194805 conn(0x7f29400f3800 msgr2=0x7f29400f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 shutdown_connections 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 -- 172.21.15.87:0/2947194805 wait complete. 2024-02-23T00:28:09.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:09.676+0000 7f2945d60700 1 librados: shutdown 2024-02-23T00:28:10.294 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:09.675745+0000 mon.c (mon.2) 17 : audit [DBG] from='client.? 172.21.15.87:0/2947194805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-23T00:28:10.294 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:09.675745+0000 mon.c (mon.2) 17 : audit [DBG] from='client.? 172.21.15.87:0/2947194805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-23T00:28:10.313 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool get default.rgw.meta pg_num 2024-02-23T00:28:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:09.675745+0000 mon.c (mon.2) 17 : audit [DBG] from='client.? 172.21.15.87:0/2947194805' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-23T00:28:10.580 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:10.909 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:28:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:28:11.130 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:09.462315+0000 mgr.y (mgr.24437) 31 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:28:11.131 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:10.269466+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:11.131 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:10.269511+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:11.131 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:09.462315+0000 mgr.y (mgr.24437) 31 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:28:11.131 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:10.269466+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:11.131 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:10.269511+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:11.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:09.462315+0000 mgr.y (mgr.24437) 31 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:28:11.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:10.269466+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:11.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:10.269511+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:12.193 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:10.490980+0000 mgr.y (mgr.24437) 32 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:12.193 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:11.091596+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-23T00:28:12.193 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:10.490980+0000 mgr.y (mgr.24437) 32 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:12.193 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:11.091596+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-23T00:28:12.428 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:10.490980+0000 mgr.y (mgr.24437) 32 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:12.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:11.091596+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3825627021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc4102210 msgr2=0x7f1fc4104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3825627021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc4102210 0x7f1fc4104600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1fbc009be0 tx=0x7f1fbc004d10).stop 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3825627021 shutdown_connections 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3825627021 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc4102210 0x7f1fc4104600 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3825627021 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc40f8cb0 0x7f1fc40f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3825627021 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fc40f9610 0x7f1fc4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3825627021 >> 172.21.15.87:0/3825627021 conn(0x7f1fc40f2820 msgr2=0x7f1fc40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3825627021 shutdown_connections 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3825627021 wait complete. 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 librados: starting msgr at 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 librados: starting objecter 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.431+0000 7f1fcafd0700 1 Processor -- start 2024-02-23T00:28:12.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fcafd0700 1 -- start start 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fcafd0700 1 librados: setting wanted keys 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fcafd0700 1 librados: calling monclient init 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fcafd0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc40f8cb0 0x7f1fc410dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fcafd0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc40f9610 0x7f1fc410e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fcafd0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fc4102210 0x7f1fc4108170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fcafd0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4108660 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fcafd0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc41087d0 con 0x7f1fc40f9610 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fcafd0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc4108990 con 0x7f1fc4102210 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc8d6c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc40f8cb0 0x7f1fc410dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc8d6c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc40f8cb0 0x7f1fc410dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52486/0 (socket says 172.21.15.87:52486) 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc8d6c700 1 -- 172.21.15.87:0/3674441140 learned_addr learned my addr 172.21.15.87:0/3674441140 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc956d700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fc4102210 0x7f1fc4108170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc3fff700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc40f9610 0x7f1fc410e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:12.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc8d6c700 1 -- 172.21.15.87:0/3674441140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fc4102210 msgr2=0x7f1fc4108170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc8d6c700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fc4102210 0x7f1fc4108170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc8d6c700 1 -- 172.21.15.87:0/3674441140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc40f9610 msgr2=0x7f1fc410e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc8d6c700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc40f9610 0x7f1fc410e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc8d6c700 1 -- 172.21.15.87:0/3674441140 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb0005040 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc3fff700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc40f9610 0x7f1fc410e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc956d700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fc4102210 0x7f1fc4108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc8d6c700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc40f8cb0 0x7f1fc410dfd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f1fb80041d0 tx=0x7f1fb8009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc1ffb700 10 client.?.objecter ms_handle_connect 0x7f1fc40f8cb0 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc1ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.432+0000 7f1fc1ffb700 1 -- 172.21.15.87:0/3674441140 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8007b40 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.433+0000 7f1fc1ffb700 1 -- 172.21.15.87:0/3674441140 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fb800f040 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.433+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fbc0097c0 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.433+0000 7f1fc1ffb700 1 -- 172.21.15.87:0/3674441140 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8014810 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.433+0000 7f1fcafd0700 10 client.14583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:12.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.433+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc4114e00 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.433+0000 7f1fcafd0700 1 librados: init done 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.434+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc405fa60 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.434+0000 7f1fc1ffb700 1 -- 172.21.15.87:0/3674441140 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1fb8009110 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.434+0000 7f1fc1ffb700 1 --2- 172.21.15.87:0/3674441140 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1fb4062c90 0x7f1fb4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.434+0000 7f1fc1ffb700 1 -- 172.21.15.87:0/3674441140 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f1fb807caf0 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.434+0000 7f1fc1ffb700 10 client.14583.objecter ms_dispatch 0x7f1fc4001550 osd_map(73..73 src has 1..73) v4 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.434+0000 7f1fc1ffb700 3 client.14583.objecter handle_osd_map got epochs [73,73] > 0 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.434+0000 7f1fc1ffb700 3 client.14583.objecter handle_osd_map decoding full epoch 73 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.434+0000 7f1fc3fff700 1 --2- 172.21.15.87:0/3674441140 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1fb4062c90 0x7f1fb4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.435+0000 7f1fc1ffb700 20 client.14583.objecter dump_active .. 0 homeless 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.435+0000 7f1fc1ffb700 10 client.14583.objecter ms_handle_connect 0x7f1fb4062c90 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.435+0000 7f1fc3fff700 1 --2- 172.21.15.87:0/3674441140 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1fb4062c90 0x7f1fb4065150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f1fb0007fd0 tx=0x7f1fb0009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:12.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.439+0000 7f1fc1ffb700 1 -- 172.21.15.87:0/3674441140 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1fb8050790 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.603+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"} v 0) v1 -- 0x7f1fc4109650 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.603+0000 7f1fc1ffb700 1 -- 172.21.15.87:0/3674441140 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]=0 v73) v1 ==== 105+0+11 (secure 0 0 0) 0x7f1fb8050380 con 0x7f1fc40f8cb0 2024-02-23T00:28:12.606 INFO:teuthology.orchestra.run.smithi087.stdout:pg_num: 32 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 10 librados: watch_flush enter 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 10 librados: watch_flush exit 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 20 client.14583.objecter shutdown clearing up homeless session... 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 10 client.14583.objecter successfully canceled tick 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1fb4062c90 msgr2=0x7f1fb4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3674441140 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1fb4062c90 0x7f1fb4065150 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f1fb0007fd0 tx=0x7f1fb0009cb0).stop 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc40f8cb0 msgr2=0x7f1fc410dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc40f8cb0 0x7f1fc410dfd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f1fb80041d0 tx=0x7f1fb8009f60).stop 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 shutdown_connections 2024-02-23T00:28:12.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3674441140 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1fb4062c90 0x7f1fb4065150 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:12.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fc4102210 0x7f1fc4108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:12.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fc40f8cb0 0x7f1fc410dfd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:12.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 --2- 172.21.15.87:0/3674441140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fc40f9610 0x7f1fc410e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:12.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 >> 172.21.15.87:0/3674441140 conn(0x7f1fc40f2820 msgr2=0x7f1fc40f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:12.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 shutdown_connections 2024-02-23T00:28:12.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 -- 172.21.15.87:0/3674441140 wait complete. 2024-02-23T00:28:12.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:12.605+0000 7f1fcafd0700 1 librados: shutdown 2024-02-23T00:28:13.103 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:12 smithi171 systemd[1]: Starting Ceph prometheus.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:28:14.479 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:11.462733+0000 mgr.y (mgr.24437) 33 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:14.479 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:12.605066+0000 mon.a (mon.0) 673 : audit [DBG] from='client.? 172.21.15.87:0/3674441140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-23T00:28:14.480 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:11.462733+0000 mgr.y (mgr.24437) 33 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:14.480 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:12.605066+0000 mon.a (mon.0) 673 : audit [DBG] from='client.? 172.21.15.87:0/3674441140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-23T00:28:14.480 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:11.462733+0000 mgr.y (mgr.24437) 33 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:14.480 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:12.605066+0000 mon.a (mon.0) 673 : audit [DBG] from='client.? 172.21.15.87:0/3674441140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.338Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.338Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.338Z caller=main.go:518 level=info host_details="(Linux 4.18.0-540.el8.x86_64 #1 SMP Fri Feb 9 14:58:00 UTC 2024 x86_64 smithi171 (none))" 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.338Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.338Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.339Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.340Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.340Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.343Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.343Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.594µs 2024-02-23T00:28:14.481 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.343Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-23T00:28:14.482 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.343Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-23T00:28:14.482 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.343Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=22.489µs wal_replay_duration=130.297µs total_replay_duration=166.696µs 2024-02-23T00:28:14.482 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.343Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-23T00:28:14.482 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.343Z caller=main.go:947 level=info msg="TSDB started" 2024-02-23T00:28:14.482 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.343Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-23T00:28:14.482 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.359Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.645086ms db_storage=581ns remote_storage=1.108µs web_handler=258ns query_engine=591ns scrape=449.126µs scrape_sd=21.717µs notify=17.649µs notify_sd=8.319µs rules=14.941316ms 2024-02-23T00:28:14.482 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:13.359Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-23T00:28:14.482 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 bash[62932]: e5439be0ffe6321950fedbff3b17d2019c275ffecd8d011add57fad59e5a73da 2024-02-23T00:28:14.482 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:13 smithi171 systemd[1]: Started Ceph prometheus.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:28:14.483 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-23T00:28:14.483 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-23T00:28:14.483 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph mgr dump --format=json 2024-02-23T00:28:14.484 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:13] ENGINE Bus STOPPING 2024-02-23T00:28:14.484 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-23T00:28:14.484 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:13] ENGINE Bus STOPPED 2024-02-23T00:28:14.484 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:13] ENGINE Bus STARTING 2024-02-23T00:28:14.484 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:13] ENGINE Serving on http://:::9283 2024-02-23T00:28:14.484 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:13] ENGINE Bus STARTED 2024-02-23T00:28:14.768 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:13.459164+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:13.463070+0000 mgr.y (mgr.24437) 34 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:13.579172+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:13.584289+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:13.586580+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-23T00:28:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:13.586943+0000 mgr.y (mgr.24437) 35 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-23T00:28:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:13.590734+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:28:13.599825+0000 mgr.y (mgr.24437) 36 : cephadm [INF] Deploying daemon grafana.a on smithi171 2024-02-23T00:28:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:13.459164+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:13.463070+0000 mgr.y (mgr.24437) 34 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:13.579172+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:13.584289+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:13.586580+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-23T00:28:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:13.586943+0000 mgr.y (mgr.24437) 35 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-23T00:28:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:13.590734+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:28:13.599825+0000 mgr.y (mgr.24437) 36 : cephadm [INF] Deploying daemon grafana.a on smithi171 2024-02-23T00:28:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:13.459164+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:13.463070+0000 mgr.y (mgr.24437) 34 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:13.579172+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:13.584289+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:13.586580+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-23T00:28:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:13.586943+0000 mgr.y (mgr.24437) 35 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-23T00:28:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:13.590734+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:28:13.599825+0000 mgr.y (mgr.24437) 36 : cephadm [INF] Deploying daemon grafana.a on smithi171 2024-02-23T00:28:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:14.631576+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:14.850407+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-23T00:28:16.029 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:14.631576+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:16.029 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:14.850407+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-23T00:28:16.029 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:14.631576+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:16.030 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:14.850407+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-23T00:28:16.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 -- 172.21.15.87:0/3441412215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f9670 msgr2=0x7fc940101d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:16.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3441412215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f9670 0x7fc940101d80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc93c004870 tx=0x7fc93c009cb0).stop 2024-02-23T00:28:16.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 -- 172.21.15.87:0/3441412215 shutdown_connections 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3441412215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc940102270 0x7fc940104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3441412215 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9400f8d10 0x7fc9400f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3441412215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f9670 0x7fc940101d80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 -- 172.21.15.87:0/3441412215 >> 172.21.15.87:0/3441412215 conn(0x7fc9400f2820 msgr2=0x7fc9400f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 -- 172.21.15.87:0/3441412215 shutdown_connections 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 -- 172.21.15.87:0/3441412215 wait complete. 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 librados: starting msgr at 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.581+0000 7fc947cfa700 1 librados: starting objecter 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 Processor -- start 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 -- start start 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 librados: setting wanted keys 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 librados: calling monclient init 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f8d10 0x7fc9400f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:16.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9400f9670 0x7fc9400f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc940102270 0x7fc9400f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9401166b0 con 0x7fc9400f8d10 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc940116820 con 0x7fc940102270 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc947cfa700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9401169e0 con 0x7fc9400f9670 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945a96700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f8d10 0x7fc9400f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945a96700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f8d10 0x7fc9400f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52504/0 (socket says 172.21.15.87:52504) 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945295700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9400f9670 0x7fc9400f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945a96700 1 -- 172.21.15.87:0/3073720858 learned_addr learned my addr 172.21.15.87:0/3073720858 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc946297700 1 --2- 172.21.15.87:0/3073720858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc940102270 0x7fc9400f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945a96700 1 -- 172.21.15.87:0/3073720858 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9400f9670 msgr2=0x7fc9400f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945a96700 1 --2- 172.21.15.87:0/3073720858 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9400f9670 0x7fc9400f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945a96700 1 -- 172.21.15.87:0/3073720858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc940102270 msgr2=0x7fc9400f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945a96700 1 --2- 172.21.15.87:0/3073720858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc940102270 0x7fc9400f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:16.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945a96700 1 -- 172.21.15.87:0/3073720858 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc930005040 con 0x7fc9400f8d10 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945295700 1 --2- 172.21.15.87:0/3073720858 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9400f9670 0x7fc9400f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc945a96700 1 --2- 172.21.15.87:0/3073720858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f8d10 0x7fc9400f6cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc9300048f0 tx=0x7fc930009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc936ffd700 10 client.?.objecter ms_handle_connect 0x7fc9400f8d10 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc936ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.582+0000 7fc936ffd700 1 -- 172.21.15.87:0/3073720858 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930007b00 con 0x7fc9400f8d10 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.583+0000 7fc936ffd700 1 -- 172.21.15.87:0/3073720858 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc93000f040 con 0x7fc9400f8d10 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.583+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c005040 con 0x7fc9400f8d10 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.583+0000 7fc936ffd700 1 -- 172.21.15.87:0/3073720858 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930014990 con 0x7fc9400f8d10 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.583+0000 7fc947cfa700 10 client.14589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.583+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc940117020 con 0x7fc9400f8d10 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.583+0000 7fc947cfa700 1 librados: init done 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.584+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94005fa60 con 0x7fc9400f8d10 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.584+0000 7fc936ffd700 1 -- 172.21.15.87:0/3073720858 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc930010500 con 0x7fc9400f8d10 2024-02-23T00:28:16.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.584+0000 7fc936ffd700 1 --2- 172.21.15.87:0/3073720858 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc92c062c90 0x7fc92c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:16.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.584+0000 7fc936ffd700 1 -- 172.21.15.87:0/3073720858 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7fc93007c530 con 0x7fc9400f8d10 2024-02-23T00:28:16.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.584+0000 7fc945295700 1 --2- 172.21.15.87:0/3073720858 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc92c062c90 0x7fc92c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:16.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.584+0000 7fc936ffd700 10 client.14589.objecter ms_dispatch 0x7fc940001550 osd_map(74..74 src has 1..74) v4 2024-02-23T00:28:16.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.584+0000 7fc936ffd700 3 client.14589.objecter handle_osd_map got epochs [74,74] > 0 2024-02-23T00:28:16.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.584+0000 7fc936ffd700 3 client.14589.objecter handle_osd_map decoding full epoch 74 2024-02-23T00:28:16.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.584+0000 7fc936ffd700 20 client.14589.objecter dump_active .. 0 homeless 2024-02-23T00:28:16.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.585+0000 7fc936ffd700 10 client.14589.objecter ms_handle_connect 0x7fc92c062c90 2024-02-23T00:28:16.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.585+0000 7fc945295700 1 --2- 172.21.15.87:0/3073720858 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc92c062c90 0x7fc92c065150 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc93c004ee0 tx=0x7fc93c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:16.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.589+0000 7fc936ffd700 1 -- 172.21.15.87:0/3073720858 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc930049f60 con 0x7fc9400f8d10 2024-02-23T00:28:16.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.795+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc940107300 con 0x7fc9400f8d10 2024-02-23T00:28:16.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.798+0000 7fc936ffd700 1 -- 172.21.15.87:0/3073720858 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+151935 (secure 0 0 0) 0x7fc930047190 con 0x7fc9400f8d10 2024-02-23T00:28:16.802 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 10 librados: watch_flush enter 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 10 librados: watch_flush exit 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 20 client.14589.objecter shutdown clearing up homeless session... 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 10 client.14589.objecter successfully canceled tick 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc92c062c90 msgr2=0x7fc92c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3073720858 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc92c062c90 0x7fc92c065150 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc93c004ee0 tx=0x7fc93c00b040).stop 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f8d10 msgr2=0x7fc9400f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3073720858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f8d10 0x7fc9400f6cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc9300048f0 tx=0x7fc930009cb0).stop 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 shutdown_connections 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3073720858 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc92c062c90 0x7fc92c065150 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:16.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3073720858 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc940102270 0x7fc9400f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:16.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3073720858 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9400f8d10 0x7fc9400f6cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:16.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 --2- 172.21.15.87:0/3073720858 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9400f9670 0x7fc9400f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:16.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 >> 172.21.15.87:0/3073720858 conn(0x7fc9400f2820 msgr2=0x7fc9400f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:16.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 shutdown_connections 2024-02-23T00:28:16.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 -- 172.21.15.87:0/3073720858 wait complete. 2024-02-23T00:28:16.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:16.800+0000 7fc947cfa700 1 librados: shutdown 2024-02-23T00:28:17.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:15.463631+0000 mgr.y (mgr.24437) 37 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:17.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:16.797052+0000 mon.a (mon.0) 681 : audit [DBG] from='client.? 172.21.15.87:0/3073720858' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-23T00:28:17.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:15.463631+0000 mgr.y (mgr.24437) 37 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:17.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:16.797052+0000 mon.a (mon.0) 681 : audit [DBG] from='client.? 172.21.15.87:0/3073720858' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-23T00:28:17.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:15.463631+0000 mgr.y (mgr.24437) 37 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:17.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:16.797052+0000 mon.a (mon.0) 681 : audit [DBG] from='client.? 172.21.15.87:0/3073720858' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-23T00:28:17.627 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":19,"active_gid":24437,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":1231597463}]},"active_addr":"172.21.15.87:6800/1231597463","active_change":"2024-02-23T00:27:39.443191+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":24430,"name":"x","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.87:8443/","prometheus":"http://172.21.15.87:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":66,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":2027873087}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.87:0","nonce":87454471}]}]} 2024-02-23T00:28:17.630 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-23T00:28:17.631 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-23T00:28:17.631 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump --format=json 2024-02-23T00:28:17.901 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:18.144 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:16.557200+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.e scrub starts 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:16.558020+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.e scrub ok 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:17.027154+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:17.030638+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:17.044719+0000 osd.3 (osd.3) 7 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:17.047611+0000 osd.3 (osd.3) 8 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:16.557200+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.e scrub starts 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:16.558020+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.e scrub ok 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:17.027154+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:17.030638+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:17.044719+0000 osd.3 (osd.3) 7 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:28:18.145 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:17.047611+0000 osd.3 (osd.3) 8 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:28:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:16.557200+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.e scrub starts 2024-02-23T00:28:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:16.558020+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.e scrub ok 2024-02-23T00:28:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:17.027154+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:28:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:17.030638+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:28:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:17.044719+0000 osd.3 (osd.3) 7 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:28:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:17.047611+0000 osd.3 (osd.3) 8 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:28:19.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:17.463996+0000 mgr.y (mgr.24437) 38 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:19.304 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:17.463996+0000 mgr.y (mgr.24437) 38 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:19.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:17.463996+0000 mgr.y (mgr.24437) 38 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:19.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 -- 172.21.15.87:0/4126777962 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26500fdbb0 msgr2=0x7f26500fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:19.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 --2- 172.21.15.87:0/4126777962 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26500fdbb0 0x7f26500fdf90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2640004870 tx=0x7f2640009cb0).stop 2024-02-23T00:28:19.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 -- 172.21.15.87:0/4126777962 shutdown_connections 2024-02-23T00:28:19.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 --2- 172.21.15.87:0/4126777962 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26500f8420 0x7f2650104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 --2- 172.21.15.87:0/4126777962 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26500fdbb0 0x7f26500fdf90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 --2- 172.21.15.87:0/4126777962 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26500fe510 0x7f26500f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 -- 172.21.15.87:0/4126777962 >> 172.21.15.87:0/4126777962 conn(0x7f26500f3800 msgr2=0x7f26500f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 -- 172.21.15.87:0/4126777962 shutdown_connections 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 -- 172.21.15.87:0/4126777962 wait complete. 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 librados: starting msgr at 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.833+0000 7f2657fc9700 1 librados: starting objecter 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 Processor -- start 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 -- start start 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 librados: setting wanted keys 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 librados: calling monclient init 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26500f8420 0x7f265006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:19.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26500fdbb0 0x7f265006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26500fe510 0x7f265006bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2650116580 con 0x7f26500f8420 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26501166f0 con 0x7f26500fdbb0 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2657fc9700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26501168b0 con 0x7f26500fe510 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2655d65700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26500f8420 0x7f265006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2656566700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26500fe510 0x7f265006bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2655d65700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26500f8420 0x7f265006b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52530/0 (socket says 172.21.15.87:52530) 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2655d65700 1 -- 172.21.15.87:0/1105282859 learned_addr learned my addr 172.21.15.87:0/1105282859 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2655564700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26500fdbb0 0x7f265006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2656566700 1 -- 172.21.15.87:0/1105282859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26500fdbb0 msgr2=0x7f265006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2656566700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26500fdbb0 0x7f265006b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2656566700 1 -- 172.21.15.87:0/1105282859 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26500f8420 msgr2=0x7f265006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2656566700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26500f8420 0x7f265006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:19.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2656566700 1 -- 172.21.15.87:0/1105282859 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26480097c0 con 0x7f26500fe510 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2655564700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26500fdbb0 0x7f265006b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2655d65700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26500f8420 0x7f265006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2656566700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26500fe510 0x7f265006bab0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2648001cb0 tx=0x7f2648004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2646ffd700 10 client.?.objecter ms_handle_connect 0x7f26500fe510 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2646ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.834+0000 7f2646ffd700 1 -- 172.21.15.87:0/1105282859 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264800fa30 con 0x7f26500fe510 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.835+0000 7f2646ffd700 1 -- 172.21.15.87:0/1105282859 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26480095a0 con 0x7f26500fe510 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.835+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2640005040 con 0x7f26500fe510 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.835+0000 7f2646ffd700 1 -- 172.21.15.87:0/1105282859 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2648007d50 con 0x7f26500fe510 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.835+0000 7f2657fc9700 10 client.24521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.835+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2650116ea0 con 0x7f26500fe510 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.835+0000 7f2657fc9700 1 librados: init done 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.836+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26501070a0 con 0x7f26500fe510 2024-02-23T00:28:19.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.836+0000 7f2646ffd700 1 -- 172.21.15.87:0/1105282859 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f26480090d0 con 0x7f26500fe510 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.836+0000 7f2646ffd700 1 --2- 172.21.15.87:0/1105282859 >> v2:172.21.15.87:6800/1231597463 conn(0x7f263c062c90 0x7f263c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.836+0000 7f2646ffd700 1 -- 172.21.15.87:0/1105282859 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f264807d290 con 0x7f26500fe510 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.836+0000 7f2646ffd700 10 client.24521.objecter ms_dispatch 0x7f2650001550 osd_map(74..74 src has 1..74) v4 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.836+0000 7f2646ffd700 3 client.24521.objecter handle_osd_map got epochs [74,74] > 0 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.836+0000 7f2646ffd700 3 client.24521.objecter handle_osd_map decoding full epoch 74 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.836+0000 7f2655d65700 1 --2- 172.21.15.87:0/1105282859 >> v2:172.21.15.87:6800/1231597463 conn(0x7f263c062c90 0x7f263c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.837+0000 7f2646ffd700 20 client.24521.objecter dump_active .. 0 homeless 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.837+0000 7f2646ffd700 10 client.24521.objecter ms_handle_connect 0x7f263c062c90 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.837+0000 7f2655d65700 1 --2- 172.21.15.87:0/1105282859 >> v2:172.21.15.87:6800/1231597463 conn(0x7f263c062c90 0x7f263c065150 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f26400043d0 tx=0x7f264000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:19.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:19.841+0000 7f2646ffd700 1 -- 172.21.15.87:0/1105282859 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2648050f70 con 0x7f26500fe510 2024-02-23T00:28:20.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.004+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f265005fa60 con 0x7f26500fe510 2024-02-23T00:28:20.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.005+0000 7f2646ffd700 1 -- 172.21.15.87:0/1105282859 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+19010 (secure 0 0 0) 0x7f2648050b60 con 0x7f26500fe510 2024-02-23T00:28:20.008 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:28:20.008 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":74,"fsid":"66db207c-d1e1-11ee-95c0-87774f69a715","created":"2024-02-23T00:21:28.861318+0000","modified":"2024-02-23T00:28:14.846722+0000","last_up_change":"2024-02-23T00:27:09.633391+0000","last_in_change":"2024-02-23T00:26:52.603467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-23T00:25:09.208309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-23T00:27:15.562186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-23T00:27:15.989206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-23T00:27:17.282440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-23T00:27:19.262164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-23T00:27:21.319951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"e3549bad-2891-47b9-bf77-132cadc20bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":2153191863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":2153191863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":2153191863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":2153191863}]},"public_addr":"172.21.15.87:6801/2153191863","cluster_addr":"172.21.15.87:6802/2153191863","heartbeat_back_addr":"172.21.15.87:6804/2153191863","heartbeat_front_addr":"172.21.15.87:6803/2153191863","state":["exists","up"]},{"osd":1,"uuid":"85525ce8-38d5-4840-8057-547cb176c859","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":685354171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":685354171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":685354171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":685354171}]},"public_addr":"172.21.15.87:6805/685354171","cluster_addr":"172.21.15.87:6806/685354171","heartbeat_back_addr":"172.21.15.87:6808/685354171","heartbeat_front_addr":"172.21.15.87:6807/685354171","state":["exists","up"]},{"osd":2,"uuid":"48ea64b6-ed15-48d2-b0ed-aa4053938aaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3605394228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3605394228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3605394228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3605394228}]},"public_addr":"172.21.15.87:6809/3605394228","cluster_addr":"172.21.15.87:6810/3605394228","heartbeat_back_addr":"172.21.15.87:6812/3605394228","heartbeat_front_addr":"172.21.15.87:6811/3605394228","state":["exists","up"]},{"osd":3,"uuid":"cb8705aa-e114-4dfd-ac9d-ec12f85f877a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":128059790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":128059790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":128059790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":128059790}]},"public_addr":"172.21.15.87:6813/128059790","cluster_addr":"172.21.15.87:6814/128059790","heartbeat_back_addr":"172.21.15.87:6816/128059790","heartbeat_front_addr":"172.21.15.87:6815/128059790","state":["exists","up"]},{"osd":4,"uuid":"c3734a04-7343-4251-9b5b-bedb9e8b882c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4175844757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6801","nonce":4175844757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6803","nonce":4175844757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4175844757}]},"public_addr":"172.21.15.171:6800/4175844757","cluster_addr":"172.21.15.171:6801/4175844757","heartbeat_back_addr":"172.21.15.171:6803/4175844757","heartbeat_front_addr":"172.21.15.171:6802/4175844757","state":["exists","up"]},{"osd":5,"uuid":"fe86232d-96b9-4cbf-9cac-373286fa103d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2550013795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6805","nonce":2550013795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6807","nonce":2550013795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2550013795}]},"public_addr":"172.21.15.171:6804/2550013795","cluster_addr":"172.21.15.171:6805/2550013795","heartbeat_back_addr":"172.21.15.171:6807/2550013795","heartbeat_front_addr":"172.21.15.171:6806/2550013795","state":["exists","up"]},{"osd":6,"uuid":"bfc5dd69-5cfb-409f-985f-83ee0dd4dc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2509408498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2509408498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6811","nonce":2509408498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":2509408498}]},"public_addr":"172.21.15.171:6808/2509408498","cluster_addr":"172.21.15.171:6809/2509408498","heartbeat_back_addr":"172.21.15.171:6811/2509408498","heartbeat_front_addr":"172.21.15.171:6810/2509408498","state":["exists","up"]},{"osd":7,"uuid":"29ddcf97-369a-4a2c-b3e2-142e84ccf939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":373787000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":373787000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6815","nonce":373787000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":373787000}]},"public_addr":"172.21.15.171:6812/373787000","cluster_addr":"172.21.15.171:6813/373787000","heartbeat_back_addr":"172.21.15.171:6815/373787000","heartbeat_front_addr":"172.21.15.171:6814/373787000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:52.302314+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:24.499494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:35.753447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:28:10.269514+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:23.618927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:38.541075+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:47.010768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:08.063305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:6800/1121018082":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/2138166229":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/27794119":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/1675126043":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/3767016525":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/969918521":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/795095363":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2273151327":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/1685406414":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2093733106":"2024-02-24T00:21:58.144574+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-23T00:28:20.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 10 librados: watch_flush enter 2024-02-23T00:28:20.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 10 librados: watch_flush exit 2024-02-23T00:28:20.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 20 client.24521.objecter shutdown clearing up homeless session... 2024-02-23T00:28:20.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 10 client.24521.objecter successfully canceled tick 2024-02-23T00:28:20.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 >> v2:172.21.15.87:6800/1231597463 conn(0x7f263c062c90 msgr2=0x7f263c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 --2- 172.21.15.87:0/1105282859 >> v2:172.21.15.87:6800/1231597463 conn(0x7f263c062c90 0x7f263c065150 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f26400043d0 tx=0x7f264000b040).stop 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26500fe510 msgr2=0x7f265006bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26500fe510 0x7f265006bab0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2648001cb0 tx=0x7f2648004c50).stop 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 shutdown_connections 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 --2- 172.21.15.87:0/1105282859 >> v2:172.21.15.87:6800/1231597463 conn(0x7f263c062c90 0x7f263c065150 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f26500fe510 0x7f265006bab0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f26500f8420 0x7f265006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 --2- 172.21.15.87:0/1105282859 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f26500fdbb0 0x7f265006b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 >> 172.21.15.87:0/1105282859 conn(0x7f26500f3800 msgr2=0x7f26500f53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:20.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 shutdown_connections 2024-02-23T00:28:20.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 -- 172.21.15.87:0/1105282859 wait complete. 2024-02-23T00:28:20.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:20.007+0000 7f2657fc9700 1 librados: shutdown 2024-02-23T00:28:20.765 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-23T00:28:20.765 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump --format=json 2024-02-23T00:28:20.866 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:28:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:28:21.035 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:21.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:19.464392+0000 mgr.y (mgr.24437) 39 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-02-23T00:28:21.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:20.006406+0000 mon.c (mon.2) 18 : audit [DBG] from='client.? 172.21.15.87:0/1105282859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:28:21.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:20.070897+0000 osd.3 (osd.3) 9 : cluster [DBG] 4.10 deep-scrub starts 2024-02-23T00:28:21.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:20.075480+0000 osd.3 (osd.3) 10 : cluster [DBG] 4.10 deep-scrub ok 2024-02-23T00:28:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:19.464392+0000 mgr.y (mgr.24437) 39 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-02-23T00:28:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:20.006406+0000 mon.c (mon.2) 18 : audit [DBG] from='client.? 172.21.15.87:0/1105282859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:28:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:20.070897+0000 osd.3 (osd.3) 9 : cluster [DBG] 4.10 deep-scrub starts 2024-02-23T00:28:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:20.075480+0000 osd.3 (osd.3) 10 : cluster [DBG] 4.10 deep-scrub ok 2024-02-23T00:28:21.191 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:19.464392+0000 mgr.y (mgr.24437) 39 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-02-23T00:28:21.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:20.006406+0000 mon.c (mon.2) 18 : audit [DBG] from='client.? 172.21.15.87:0/1105282859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:28:21.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:20.070897+0000 osd.3 (osd.3) 9 : cluster [DBG] 4.10 deep-scrub starts 2024-02-23T00:28:21.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:20.075480+0000 osd.3 (osd.3) 10 : cluster [DBG] 4.10 deep-scrub ok 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:20.500993+0000 mgr.y (mgr.24437) 40 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:20.503719+0000 osd.2 (osd.2) 9 : cluster [DBG] 2.c scrub starts 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:20.504616+0000 osd.2 (osd.2) 10 : cluster [DBG] 2.c scrub ok 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:20.769559+0000 osd.7 (osd.7) 5 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:20.772471+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:21.020593+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:21.021612+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:21.120009+0000 osd.3 (osd.3) 11 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:21.123491+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:28:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:21.154263+0000 osd.0 (osd.0) 9 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:28:22.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:21.157732+0000 osd.0 (osd.0) 10 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:28:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:20.500993+0000 mgr.y (mgr.24437) 40 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:20.503719+0000 osd.2 (osd.2) 9 : cluster [DBG] 2.c scrub starts 2024-02-23T00:28:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:20.504616+0000 osd.2 (osd.2) 10 : cluster [DBG] 2.c scrub ok 2024-02-23T00:28:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:20.769559+0000 osd.7 (osd.7) 5 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:28:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:20.772471+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:28:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:21.020593+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:28:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:21.021612+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:21.120009+0000 osd.3 (osd.3) 11 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:21.123491+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:21.154263+0000 osd.0 (osd.0) 9 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:21.157732+0000 osd.0 (osd.0) 10 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:20.500993+0000 mgr.y (mgr.24437) 40 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:20.503719+0000 osd.2 (osd.2) 9 : cluster [DBG] 2.c scrub starts 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:20.504616+0000 osd.2 (osd.2) 10 : cluster [DBG] 2.c scrub ok 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:20.769559+0000 osd.7 (osd.7) 5 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:20.772471+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:21.020593+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:28:22.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:21.021612+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:28:22.229 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:21.120009+0000 osd.3 (osd.3) 11 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:28:22.229 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:21.123491+0000 osd.3 (osd.3) 12 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:28:22.229 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:21.154263+0000 osd.0 (osd.0) 9 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:28:22.229 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:21.157732+0000 osd.0 (osd.0) 10 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:28:22.504 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:28:22] "GET /metrics HTTP/1.1" 200 172154 "" "Prometheus/2.33.4" 2024-02-23T00:28:23.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:21.465063+0000 mgr.y (mgr.24437) 41 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:23.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:22.157748+0000 osd.3 (osd.3) 13 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:28:23.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:22.160637+0000 osd.3 (osd.3) 14 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:28:23.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:21.465063+0000 mgr.y (mgr.24437) 41 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:23.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:22.157748+0000 osd.3 (osd.3) 13 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:28:23.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:22.160637+0000 osd.3 (osd.3) 14 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:28:23.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:21.465063+0000 mgr.y (mgr.24437) 41 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:23.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:22.157748+0000 osd.3 (osd.3) 13 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:28:23.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:22.160637+0000 osd.3 (osd.3) 14 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:28:23.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 -- 172.21.15.87:0/1153649310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 msgr2=0x7fe17c101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 --2- 172.21.15.87:0/1153649310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 0x7fe17c101cd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe1680047b0 tx=0x7fe168009cb0).stop 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 -- 172.21.15.87:0/1153649310 shutdown_connections 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 --2- 172.21.15.87:0/1153649310 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe17c1021c0 0x7fe17c1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 --2- 172.21.15.87:0/1153649310 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe17c0f8e20 0x7fe17c0f9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 --2- 172.21.15.87:0/1153649310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 0x7fe17c101cd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 -- 172.21.15.87:0/1153649310 >> 172.21.15.87:0/1153649310 conn(0x7fe17c0f26f0 msgr2=0x7fe17c0f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 -- 172.21.15.87:0/1153649310 shutdown_connections 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 -- 172.21.15.87:0/1153649310 wait complete. 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 librados: starting msgr at 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.398+0000 7fe183045700 1 librados: starting objecter 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 Processor -- start 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 -- start start 2024-02-23T00:28:23.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 librados: setting wanted keys 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 librados: calling monclient init 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe17c0f8e20 0x7fe17c10e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 0x7fe17c10e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe17c1021c0 0x7fe17c1084a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c108990 con 0x7fe17c0f96f0 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c108b00 con 0x7fe17c0f8e20 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe183045700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c108cc0 con 0x7fe17c1021c0 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 0x7fe17c10e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe1815e2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe17c1021c0 0x7fe17c1084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe17bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 0x7fe17c10e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42334/0 (socket says 172.21.15.87:42334) 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe17bfff700 1 -- 172.21.15.87:0/8919053 learned_addr learned my addr 172.21.15.87:0/8919053 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe1815e2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe17c1021c0 0x7fe17c1084a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:42082/0 (socket says 172.21.15.87:42082) 2024-02-23T00:28:23.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe180de1700 1 --2- 172.21.15.87:0/8919053 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe17c0f8e20 0x7fe17c10e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe17bfff700 1 -- 172.21.15.87:0/8919053 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe17c1021c0 msgr2=0x7fe17c1084a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe17bfff700 1 --2- 172.21.15.87:0/8919053 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe17c1021c0 0x7fe17c1084a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe17bfff700 1 -- 172.21.15.87:0/8919053 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe17c0f8e20 msgr2=0x7fe17c10e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe17bfff700 1 --2- 172.21.15.87:0/8919053 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe17c0f8e20 0x7fe17c10e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe17bfff700 1 -- 172.21.15.87:0/8919053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1740097c0 con 0x7fe17c0f96f0 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe17bfff700 1 --2- 172.21.15.87:0/8919053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 0x7fe17c10e7f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe168009b10 tx=0x7fe168009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe179ffb700 10 client.?.objecter ms_handle_connect 0x7fe17c0f96f0 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe179ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.399+0000 7fe179ffb700 1 -- 172.21.15.87:0/8919053 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe168018070 con 0x7fe17c0f96f0 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.400+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe168005040 con 0x7fe17c0f96f0 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.400+0000 7fe179ffb700 1 -- 172.21.15.87:0/8919053 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe168004180 con 0x7fe17c0f96f0 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.400+0000 7fe179ffb700 1 -- 172.21.15.87:0/8919053 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16801ddb0 con 0x7fe17c0f96f0 2024-02-23T00:28:23.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.400+0000 7fe183045700 10 client.14601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:23.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.400+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c109240 con 0x7fe17c0f96f0 2024-02-23T00:28:23.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.400+0000 7fe183045700 1 librados: init done 2024-02-23T00:28:23.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.401+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17c05ee80 con 0x7fe17c0f96f0 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.401+0000 7fe179ffb700 1 -- 172.21.15.87:0/8919053 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fe16801d470 con 0x7fe17c0f96f0 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.402+0000 7fe179ffb700 1 --2- 172.21.15.87:0/8919053 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe16c062c90 0x7fe16c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.402+0000 7fe179ffb700 1 -- 172.21.15.87:0/8919053 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7fe16807d990 con 0x7fe17c0f96f0 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.402+0000 7fe179ffb700 10 client.14601.objecter ms_dispatch 0x7fe17c001550 osd_map(74..74 src has 1..74) v4 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.402+0000 7fe179ffb700 3 client.14601.objecter handle_osd_map got epochs [74,74] > 0 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.402+0000 7fe179ffb700 3 client.14601.objecter handle_osd_map decoding full epoch 74 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.402+0000 7fe180de1700 1 --2- 172.21.15.87:0/8919053 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe16c062c90 0x7fe16c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.402+0000 7fe179ffb700 20 client.14601.objecter dump_active .. 0 homeless 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.402+0000 7fe179ffb700 10 client.14601.objecter ms_handle_connect 0x7fe16c062c90 2024-02-23T00:28:23.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.402+0000 7fe180de1700 1 --2- 172.21.15.87:0/8919053 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe16c062c90 0x7fe16c065150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe170006fd0 tx=0x7fe170009060).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:23.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.405+0000 7fe179ffb700 1 -- 172.21.15.87:0/8919053 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe1680516b0 con 0x7fe17c0f96f0 2024-02-23T00:28:23.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.568+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe17c1071f0 con 0x7fe17c0f96f0 2024-02-23T00:28:23.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.569+0000 7fe179ffb700 1 -- 172.21.15.87:0/8919053 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+19010 (secure 0 0 0) 0x7fe1680512a0 con 0x7fe17c0f96f0 2024-02-23T00:28:23.571 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:28:23.571 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":74,"fsid":"66db207c-d1e1-11ee-95c0-87774f69a715","created":"2024-02-23T00:21:28.861318+0000","modified":"2024-02-23T00:28:14.846722+0000","last_up_change":"2024-02-23T00:27:09.633391+0000","last_in_change":"2024-02-23T00:26:52.603467+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-23T00:25:09.208309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-23T00:27:15.562186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-23T00:27:15.989206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-23T00:27:17.282440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-23T00:27:19.262164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-23T00:27:21.319951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"e3549bad-2891-47b9-bf77-132cadc20bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":2153191863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":2153191863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":2153191863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":2153191863}]},"public_addr":"172.21.15.87:6801/2153191863","cluster_addr":"172.21.15.87:6802/2153191863","heartbeat_back_addr":"172.21.15.87:6804/2153191863","heartbeat_front_addr":"172.21.15.87:6803/2153191863","state":["exists","up"]},{"osd":1,"uuid":"85525ce8-38d5-4840-8057-547cb176c859","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":685354171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":685354171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":685354171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":685354171}]},"public_addr":"172.21.15.87:6805/685354171","cluster_addr":"172.21.15.87:6806/685354171","heartbeat_back_addr":"172.21.15.87:6808/685354171","heartbeat_front_addr":"172.21.15.87:6807/685354171","state":["exists","up"]},{"osd":2,"uuid":"48ea64b6-ed15-48d2-b0ed-aa4053938aaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3605394228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3605394228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3605394228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3605394228}]},"public_addr":"172.21.15.87:6809/3605394228","cluster_addr":"172.21.15.87:6810/3605394228","heartbeat_back_addr":"172.21.15.87:6812/3605394228","heartbeat_front_addr":"172.21.15.87:6811/3605394228","state":["exists","up"]},{"osd":3,"uuid":"cb8705aa-e114-4dfd-ac9d-ec12f85f877a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":128059790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":128059790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":128059790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":128059790}]},"public_addr":"172.21.15.87:6813/128059790","cluster_addr":"172.21.15.87:6814/128059790","heartbeat_back_addr":"172.21.15.87:6816/128059790","heartbeat_front_addr":"172.21.15.87:6815/128059790","state":["exists","up"]},{"osd":4,"uuid":"c3734a04-7343-4251-9b5b-bedb9e8b882c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4175844757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6801","nonce":4175844757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6803","nonce":4175844757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4175844757}]},"public_addr":"172.21.15.171:6800/4175844757","cluster_addr":"172.21.15.171:6801/4175844757","heartbeat_back_addr":"172.21.15.171:6803/4175844757","heartbeat_front_addr":"172.21.15.171:6802/4175844757","state":["exists","up"]},{"osd":5,"uuid":"fe86232d-96b9-4cbf-9cac-373286fa103d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2550013795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6805","nonce":2550013795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6807","nonce":2550013795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2550013795}]},"public_addr":"172.21.15.171:6804/2550013795","cluster_addr":"172.21.15.171:6805/2550013795","heartbeat_back_addr":"172.21.15.171:6807/2550013795","heartbeat_front_addr":"172.21.15.171:6806/2550013795","state":["exists","up"]},{"osd":6,"uuid":"bfc5dd69-5cfb-409f-985f-83ee0dd4dc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2509408498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2509408498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6811","nonce":2509408498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":2509408498}]},"public_addr":"172.21.15.171:6808/2509408498","cluster_addr":"172.21.15.171:6809/2509408498","heartbeat_back_addr":"172.21.15.171:6811/2509408498","heartbeat_front_addr":"172.21.15.171:6810/2509408498","state":["exists","up"]},{"osd":7,"uuid":"29ddcf97-369a-4a2c-b3e2-142e84ccf939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":373787000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":373787000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6815","nonce":373787000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":373787000}]},"public_addr":"172.21.15.171:6812/373787000","cluster_addr":"172.21.15.171:6813/373787000","heartbeat_back_addr":"172.21.15.171:6815/373787000","heartbeat_front_addr":"172.21.15.171:6814/373787000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:52.302314+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:24.499494+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:35.753447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:28:10.269514+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:23.618927+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:38.541075+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:47.010768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:27:08.063305+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:6800/1121018082":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/2138166229":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/27794119":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/1675126043":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/3767016525":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/969918521":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/795095363":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2273151327":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/1685406414":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2093733106":"2024-02-24T00:21:58.144574+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-23T00:28:23.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 10 librados: watch_flush enter 2024-02-23T00:28:23.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 10 librados: watch_flush exit 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 20 client.14601.objecter shutdown clearing up homeless session... 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 10 client.14601.objecter successfully canceled tick 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe16c062c90 msgr2=0x7fe16c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 --2- 172.21.15.87:0/8919053 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe16c062c90 0x7fe16c065150 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe170006fd0 tx=0x7fe170009060).stop 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 msgr2=0x7fe17c10e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 --2- 172.21.15.87:0/8919053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 0x7fe17c10e7f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fe168009b10 tx=0x7fe168009cb0).stop 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 shutdown_connections 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 --2- 172.21.15.87:0/8919053 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe16c062c90 0x7fe16c065150 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 --2- 172.21.15.87:0/8919053 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe17c1021c0 0x7fe17c1084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 --2- 172.21.15.87:0/8919053 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe17c0f8e20 0x7fe17c10e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 --2- 172.21.15.87:0/8919053 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe17c0f96f0 0x7fe17c10e7f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.570+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 >> 172.21.15.87:0/8919053 conn(0x7fe17c0f26f0 msgr2=0x7fe17c0f3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.571+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 shutdown_connections 2024-02-23T00:28:23.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.571+0000 7fe183045700 1 -- 172.21.15.87:0/8919053 wait complete. 2024-02-23T00:28:23.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:23.571+0000 7fe183045700 1 librados: shutdown 2024-02-23T00:28:23.878 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:28:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:28:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:28:24.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:23.569675+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.87:0/8919053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:28:24.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:23.569675+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.87:0/8919053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:28:24.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:23.569675+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.87:0/8919053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:28:24.319 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-23T00:28:24.586 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:23.465412+0000 mgr.y (mgr.24437) 42 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:24.001903+0000 osd.5 (osd.5) 7 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:28:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:24.002873+0000 osd.5 (osd.5) 8 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:28:25.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:23.465412+0000 mgr.y (mgr.24437) 42 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:25.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:24.001903+0000 osd.5 (osd.5) 7 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:28:25.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:24.002873+0000 osd.5 (osd.5) 8 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:23.465412+0000 mgr.y (mgr.24437) 42 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:24.001903+0000 osd.5 (osd.5) 7 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:28:25.088 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:24.002873+0000 osd.5 (osd.5) 8 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:28:26.019 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:24.952445+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.7 deep-scrub starts 2024-02-23T00:28:26.019 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:24.953344+0000 osd.5 (osd.5) 10 : cluster [DBG] 5.7 deep-scrub ok 2024-02-23T00:28:26.019 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 systemd[1]: Starting Ceph grafana.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:28:26.221 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:24.952445+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.7 deep-scrub starts 2024-02-23T00:28:26.221 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:24.953344+0000 osd.5 (osd.5) 10 : cluster [DBG] 5.7 deep-scrub ok 2024-02-23T00:28:26.221 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:24.952445+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.7 deep-scrub starts 2024-02-23T00:28:26.221 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:24.953344+0000 osd.5 (osd.5) 10 : cluster [DBG] 5.7 deep-scrub ok 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.460+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/641169655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4b40fe510 msgr2=0x7fc4b40f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.460+0000 7fc4ba2a2700 1 --2- 172.21.15.87:0/641169655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4b40fe510 0x7fc4b40f7d40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc4a40047b0 tx=0x7fc4a4009cb0).stop 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.460+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/641169655 shutdown_connections 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.460+0000 7fc4ba2a2700 1 --2- 172.21.15.87:0/641169655 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc4b40f8420 0x7fc4b4104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.460+0000 7fc4ba2a2700 1 --2- 172.21.15.87:0/641169655 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc4b40fdbb0 0x7fc4b40fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.460+0000 7fc4ba2a2700 1 --2- 172.21.15.87:0/641169655 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4b40fe510 0x7fc4b40f7d40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.460+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/641169655 >> 172.21.15.87:0/641169655 conn(0x7fc4b40f3800 msgr2=0x7fc4b40f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.460+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/641169655 shutdown_connections 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.460+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/641169655 wait complete. 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 librados: starting msgr at 2024-02-23T00:28:26.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 librados: starting objecter 2024-02-23T00:28:26.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 Processor -- start 2024-02-23T00:28:26.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 -- start start 2024-02-23T00:28:26.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 librados: setting wanted keys 2024-02-23T00:28:26.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 librados: calling monclient init 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4b40f8420 0x7fc4b410e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc4b40fdbb0 0x7fc4b410e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc4b40fe510 0x7fc4b4108230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4108720 con 0x7fc4b40f8420 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4108890 con 0x7fc4b40fe510 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4ba2a2700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4108a50 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4b37fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc4b40fdbb0 0x7fc4b410e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4b37fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc4b40fdbb0 0x7fc4b410e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:42092/0 (socket says 172.21.15.87:42092) 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4b37fe700 1 -- 172.21.15.87:0/508400986 learned_addr learned my addr 172.21.15.87:0/508400986 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4b3fff700 1 --2- 172.21.15.87:0/508400986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4b40f8420 0x7fc4b410e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.461+0000 7fc4b883f700 1 --2- 172.21.15.87:0/508400986 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc4b40fe510 0x7fc4b4108230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:26.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b37fe700 1 -- 172.21.15.87:0/508400986 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc4b40fe510 msgr2=0x7fc4b4108230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b37fe700 1 --2- 172.21.15.87:0/508400986 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc4b40fe510 0x7fc4b4108230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b37fe700 1 -- 172.21.15.87:0/508400986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4b40f8420 msgr2=0x7fc4b410e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b37fe700 1 --2- 172.21.15.87:0/508400986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4b40f8420 0x7fc4b410e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b37fe700 1 -- 172.21.15.87:0/508400986 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0097c0 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b3fff700 1 --2- 172.21.15.87:0/508400986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc4b40f8420 0x7fc4b410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b37fe700 1 --2- 172.21.15.87:0/508400986 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc4b40fdbb0 0x7fc4b410e580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fc4a40044e0 tx=0x7fc4a4007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b1775700 10 client.?.objecter ms_handle_connect 0x7fc4b40fdbb0 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b1775700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b1775700 1 -- 172.21.15.87:0/508400986 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4018070 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b1775700 1 -- 172.21.15.87:0/508400986 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4a40092e0 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a4005040 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4b1775700 1 -- 172.21.15.87:0/508400986 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a401dc50 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4ba2a2700 10 client.24533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b4108f70 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.462+0000 7fc4ba2a2700 1 librados: init done 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.463+0000 7fc4ba2a2700 10 client.24533.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.463+0000 7fc4ba2a2700 15 client.24533.objecter _session_command_op_assign -1 1 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.463+0000 7fc4ba2a2700 20 client.24533.objecter _get_session osd=-1 returning homeless 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.463+0000 7fc4ba2a2700 10 client.24533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.463+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc4b40606d0 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.463+0000 7fc4b1775700 1 -- 172.21.15.87:0/508400986 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc4a401c5d0 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.463+0000 7fc4b1775700 1 --2- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc4a0062c90 0x7fc4a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 1 -- 172.21.15.87:0/508400986 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6056+0+0 (secure 0 0 0) 0x7fc4a407dda0 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 10 client.24533.objecter ms_dispatch 0x7fc4b4001550 osd_map(74..74 src has 1..74) v4 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 3 client.24533.objecter handle_osd_map got epochs [74,74] > 0 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 3 client.24533.objecter handle_osd_map decoding full epoch 74 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b3fff700 1 --2- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc4a0062c90 0x7fc4a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:26.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 10 client.24533.objecter checking command 1 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 1 --2- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc4a006c4b0 0x7fc4a006e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 20 client.24533.objecter _get_session s=0x7fc4a006be00 osd=0 3 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 20 client.24533.objecter put_session s=0x7fc4a006be00 osd=0 3 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 15 client.24533.objecter _session_command_op_remove -1 1 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 20 client.24533.objecter _get_session s=0x7fc4a006be00 osd=0 3 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 20 client.24533.objecter get_session s=0x7fc4a006be00 osd=0 3 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 15 client.24533.objecter _session_command_op_assign 0 1 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 20 client.24533.objecter put_session s=0x7fc4a006be00 osd=0 4 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 10 client.24533.objecter _send_command 1 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 1 -- 172.21.15.87:0/508400986 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4a006ef90 con 0x7fc4a006c4b0 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 20 client.24533.objecter dump_active .. 0 homeless 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 1 -- 172.21.15.87:0/508400986 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4a40803c0 con 0x7fc4b40fdbb0 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b883f700 1 --2- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc4a006c4b0 0x7fc4a006e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:26.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 10 client.24533.objecter ms_handle_connect 0x7fc4a0062c90 2024-02-23T00:28:26.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b3fff700 1 --2- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc4a0062c90 0x7fc4a0065150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fc49c007bf0 tx=0x7fc49c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:26.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b1775700 10 client.24533.objecter ms_handle_connect 0x7fc4a006c4b0 2024-02-23T00:28:26.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.464+0000 7fc4b883f700 1 --2- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc4a006c4b0 0x7fc4a006e950 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:26.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.465+0000 7fc4b1775700 1 -- 172.21.15.87:0/508400986 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc4a006ef90 con 0x7fc4a006c4b0 2024-02-23T00:28:26.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.465+0000 7fc4b1775700 10 client.24533.objecter ms_dispatch 0x7fc4b4001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:28:26.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.465+0000 7fc4b1775700 10 client.24533.objecter _finish_command 1 = system:0 2024-02-23T00:28:26.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.465+0000 7fc4b1775700 20 client.24533.objecter put_session s=0x7fc4a006be00 osd=0 4 2024-02-23T00:28:26.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.465+0000 7fc4b1775700 15 client.24533.objecter _session_command_op_remove 0 1 2024-02-23T00:28:26.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 10 client.24533.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:28:26.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 15 client.24533.objecter _session_command_op_assign -1 2 2024-02-23T00:28:26.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 20 client.24533.objecter _get_session s=0x7fc4a006be00 osd=0 3 2024-02-23T00:28:26.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 20 client.24533.objecter put_session s=0x7fc4a006be00 osd=0 3 2024-02-23T00:28:26.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 20 client.24533.objecter _get_session s=0x7fc4a006be00 osd=0 3 2024-02-23T00:28:26.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 15 client.24533.objecter _session_command_op_remove -1 2 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 20 client.24533.objecter get_session s=0x7fc4a006be00 osd=0 3 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 15 client.24533.objecter _session_command_op_assign 0 2 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 20 client.24533.objecter put_session s=0x7fc4a006be00 osd=0 4 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 10 client.24533.objecter _send_command 2 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc4b405ee80 con 0x7fc4a006c4b0 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4b1775700 1 -- 172.21.15.87:0/508400986 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc4b405ee80 con 0x7fc4a006c4b0 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4b1775700 10 client.24533.objecter ms_dispatch 0x7fc4b4001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4b1775700 10 client.24533.objecter _finish_command 2 = system:0 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4b1775700 20 client.24533.objecter put_session s=0x7fc4a006be00 osd=0 4 2024-02-23T00:28:26.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4b1775700 15 client.24533.objecter _session_command_op_remove 0 2 2024-02-23T00:28:26.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 10 librados: watch_flush enter 2024-02-23T00:28:26.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 10 librados: watch_flush exit 2024-02-23T00:28:26.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 10 client.24533.objecter close_session for osd.0 2024-02-23T00:28:26.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc4a006c4b0 msgr2=0x7fc4a006e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:26.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 --2- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc4a006c4b0 0x7fc4a006e950 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:26.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 20 client.24533.objecter put_session s=0x7fc4a006be00 osd=0 1 2024-02-23T00:28:26.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 20 client.24533.objecter shutdown clearing up homeless session... 2024-02-23T00:28:26.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 10 client.24533.objecter successfully canceled tick 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc4a0062c90 msgr2=0x7fc4a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 --2- 172.21.15.87:0/508400986 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc4a0062c90 0x7fc4a0065150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fc49c007bf0 tx=0x7fc49c00b040).stop 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc4b40fdbb0 msgr2=0x7fc4b410e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 --2- 172.21.15.87:0/508400986 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc4b40fdbb0 0x7fc4b410e580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fc4a40044e0 tx=0x7fc4a4007f90).stop 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4b883f700 1 -- 172.21.15.87:0/508400986 reap_dead start 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 shutdown_connections 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 >> 172.21.15.87:0/508400986 conn(0x7fc4b40f3800 msgr2=0x7fc4b40f52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 shutdown_connections 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.479+0000 7fc4ba2a2700 1 -- 172.21.15.87:0/508400986 wait complete. 2024-02-23T00:28:26.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:26.480+0000 7fc4ba2a2700 1 librados: shutdown 2024-02-23T00:28:26.619 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-23T00:28:26.619 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-23T00:28:26.619 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-23T00:28:26.619 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-23T00:28:26.619 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="App mode production" logger=settings 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-23T00:28:26.620 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-23T00:28:26.621 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-02-23T00:28:26.622 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-02-23T00:28:26.623 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-23T00:28:26.623 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-02-23T00:28:26.623 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-02-23T00:28:26.623 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-02-23T00:28:26.623 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-02-23T00:28:26.623 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-02-23T00:28:26.623 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-02-23T00:28:26.871 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-02-23T00:28:26.871 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-02-23T00:28:26.871 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-23T00:28:26.871 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-23T00:28:26.871 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-23T00:28:26.871 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-23T00:28:26.871 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-23T00:28:26.871 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-23T00:28:26.871 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-23T00:28:26.872 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 bash[63299]: ba47117050a0bd21275a3f66ad60ca0321ad2ed163ad4c39e5896aabd3fea030 2024-02-23T00:28:26.873 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 systemd[1]: Started Ceph grafana.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-23T00:28:26.874 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-23T00:28:26.875 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-23T00:28:26.876 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-23T00:28:26.880 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-23T00:28:26.881 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-23T00:28:26.882 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-23T00:28:26.883 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-23T00:28:26.884 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-23T00:28:26.885 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-23T00:28:26.886 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-23T00:28:26.887 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-23T00:28:26.888 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-23T00:28:26.889 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-23T00:28:27.119 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:25.455271+0000 osd.4 (osd.4) 5 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:28:27.119 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:25.457322+0000 osd.4 (osd.4) 6 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:28:27.120 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:25.465905+0000 mgr.y (mgr.24437) 43 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:28:27.120 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:26.538465+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.d scrub starts 2024-02-23T00:28:27.120 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:26.541629+0000 osd.1 (osd.1) 8 : cluster [DBG] 2.d scrub ok 2024-02-23T00:28:27.120 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:26.759120+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:27.120 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:26.760242+0000 mon.a (mon.0) 684 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:28:27.121 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-23T00:28:27.121 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-23T00:28:27.121 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-23T00:28:27.121 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-23T00:28:27.121 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-23T00:28:27.121 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-23T00:28:27.121 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-23T00:28:27.121 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-23T00:28:27.122 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-23T00:28:27.123 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-23T00:28:27.124 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-23T00:28:27.124 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-23T00:28:27.124 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-23T00:28:27.124 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-23T00:28:27.124 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-23T00:28:27.124 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-23T00:28:27.124 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-23T00:28:27.124 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-23T00:28:27.124 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-23T00:28:27.125 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-23T00:28:27.126 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:26+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-23T00:28:27.127 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-23T00:28:27.128 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-23T00:28:27.128 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-23T00:28:27.128 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-23T00:28:27.128 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-23T00:28:27.128 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-23T00:28:27.128 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-23T00:28:27.128 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-23T00:28:27.128 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-23T00:28:27.128 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-23T00:28:27.129 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-23T00:28:27.130 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-23T00:28:27.131 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-23T00:28:27.131 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-23T00:28:27.131 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-23T00:28:27.131 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-23T00:28:27.131 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-23T00:28:27.212 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:25.455271+0000 osd.4 (osd.4) 5 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:28:27.212 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:25.457322+0000 osd.4 (osd.4) 6 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:28:27.212 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:25.465905+0000 mgr.y (mgr.24437) 43 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:28:27.212 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:26.538465+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.d scrub starts 2024-02-23T00:28:27.212 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:26.541629+0000 osd.1 (osd.1) 8 : cluster [DBG] 2.d scrub ok 2024-02-23T00:28:27.212 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:26.759120+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:27.212 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:26.760242+0000 mon.a (mon.0) 684 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:28:27.213 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:25.455271+0000 osd.4 (osd.4) 5 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:28:27.213 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:25.457322+0000 osd.4 (osd.4) 6 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:28:27.213 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:25.465905+0000 mgr.y (mgr.24437) 43 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:28:27.213 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:26.538465+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.d scrub starts 2024-02-23T00:28:27.213 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:26.541629+0000 osd.1 (osd.1) 8 : cluster [DBG] 2.d scrub ok 2024-02-23T00:28:27.213 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:26.759120+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:27.213 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:26.760242+0000 mon.a (mon.0) 684 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:28:27.230 INFO:teuthology.orchestra.run.smithi087.stdout:34359738419 2024-02-23T00:28:27.231 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-23T00:28:27.442 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-23T00:28:27.442 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=532.363961ms 2024-02-23T00:28:27.442 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-23T00:28:27.442 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-23T00:28:27.442 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-23T00:28:27.442 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-23T00:28:27.443 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-23T00:28:27.443 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-23T00:28:27.443 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-23T00:28:27.443 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-23T00:28:27.443 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-23T00:28:27.443 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-23T00:28:27.443 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:28:27+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-23T00:28:28.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:27.521037+0000 osd.1 (osd.1) 9 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:28:28.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:27.523950+0000 osd.1 (osd.1) 10 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:28:28.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:27.521037+0000 osd.1 (osd.1) 9 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:28:28.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:27.523950+0000 osd.1 (osd.1) 10 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:28:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:27.521037+0000 osd.1 (osd.1) 9 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:28:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:27.523950+0000 osd.1 (osd.1) 10 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:28:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:29.185 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:27.295696+0000 osd.7 (osd.7) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:29.185 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:27.295753+0000 osd.7 (osd.7) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:29.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:27.466458+0000 mgr.y (mgr.24437) 44 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:28:29.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:27.555394+0000 osd.2 (osd.2) 11 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:28:29.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:27.557135+0000 osd.2 (osd.2) 12 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:28:29.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:27.905170+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-23T00:28:29.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:28.090875+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.a scrub starts 2024-02-23T00:28:29.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:28.095006+0000 osd.6 (osd.6) 10 : cluster [DBG] 4.a scrub ok 2024-02-23T00:28:29.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:28.219066+0000 osd.0 (osd.0) 11 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:28:29.186 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:28.220072+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:28:29.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:27.295696+0000 osd.7 (osd.7) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:29.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:27.295753+0000 osd.7 (osd.7) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:29.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:27.466458+0000 mgr.y (mgr.24437) 44 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:28:29.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:27.555394+0000 osd.2 (osd.2) 11 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:28:29.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:27.557135+0000 osd.2 (osd.2) 12 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:28:29.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:27.905170+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-23T00:28:29.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:28.090875+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.a scrub starts 2024-02-23T00:28:29.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:28.095006+0000 osd.6 (osd.6) 10 : cluster [DBG] 4.a scrub ok 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:28.219066+0000 osd.0 (osd.0) 11 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:28.220072+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:27.295696+0000 osd.7 (osd.7) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:27.295753+0000 osd.7 (osd.7) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:27.466458+0000 mgr.y (mgr.24437) 44 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:27.555394+0000 osd.2 (osd.2) 11 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:27.557135+0000 osd.2 (osd.2) 12 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:27.905170+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:28.090875+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.a scrub starts 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:28.095006+0000 osd.6 (osd.6) 10 : cluster [DBG] 4.a scrub ok 2024-02-23T00:28:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:28.219066+0000 osd.0 (osd.0) 11 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:28:29.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:28.220072+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:28:30.038 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:28.675377+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:28:30.038 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:28.676832+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:28:30.038 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:29.085575+0000 osd.3 (osd.3) 15 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:28:30.038 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:29.086434+0000 osd.3 (osd.3) 16 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:28:30.039 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:29.645747+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:30.039 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:29.858673+0000 mon.a (mon.0) 687 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-23T00:28:30.039 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:28.675377+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:28:30.039 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:28.676832+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:28:30.039 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:29.085575+0000 osd.3 (osd.3) 15 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:28:30.039 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:29.086434+0000 osd.3 (osd.3) 16 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:28:30.039 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:29.645747+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:30.039 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:29.858673+0000 mon.a (mon.0) 687 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-23T00:28:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:28.675377+0000 osd.7 (osd.7) 9 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:28:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:28.676832+0000 osd.7 (osd.7) 10 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:28:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:29.085575+0000 osd.3 (osd.3) 15 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:28:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:29.086434+0000 osd.3 (osd.3) 16 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:28:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:29.645747+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:30.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:29.858673+0000 mon.a (mon.0) 687 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-23T00:28:30.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.292+0000 7f4f5793a700 1 -- 172.21.15.87:0/1765464308 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f500f8bb0 msgr2=0x7f4f500f8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:30.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.292+0000 7f4f5793a700 1 --2- 172.21.15.87:0/1765464308 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f500f8bb0 0x7f4f500f8f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f4f4c004870 tx=0x7f4f4c009cb0).stop 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.292+0000 7f4f5793a700 1 -- 172.21.15.87:0/1765464308 shutdown_connections 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 --2- 172.21.15.87:0/1765464308 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f50102110 0x7f4f50104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 --2- 172.21.15.87:0/1765464308 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f500f8bb0 0x7f4f500f8f90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 --2- 172.21.15.87:0/1765464308 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f500f9510 0x7f4f50101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 -- 172.21.15.87:0/1765464308 >> 172.21.15.87:0/1765464308 conn(0x7f4f500f26e0 msgr2=0x7f4f500f4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 -- 172.21.15.87:0/1765464308 shutdown_connections 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 -- 172.21.15.87:0/1765464308 wait complete. 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 librados: starting msgr at 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 librados: starting objecter 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 Processor -- start 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 -- start start 2024-02-23T00:28:30.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 librados: setting wanted keys 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 librados: calling monclient init 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f500f8bb0 0x7f4f5010e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f500f9510 0x7f4f5010e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f50102110 0x7f4f501083e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f501088d0 con 0x7f4f500f9510 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50108a40 con 0x7f4f50102110 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.293+0000 7f4f5793a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50108c00 con 0x7f4f500f8bb0 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f556d6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f500f8bb0 0x7f4f5010e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f556d6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f500f8bb0 0x7f4f5010e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:42112/0 (socket says 172.21.15.87:42112) 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f556d6700 1 -- 172.21.15.87:0/3681040715 learned_addr learned my addr 172.21.15.87:0/3681040715 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f54ed5700 1 --2- 172.21.15.87:0/3681040715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f500f9510 0x7f4f5010e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:30.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f55ed7700 1 --2- 172.21.15.87:0/3681040715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f50102110 0x7f4f501083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f556d6700 1 -- 172.21.15.87:0/3681040715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f50102110 msgr2=0x7f4f501083e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f556d6700 1 --2- 172.21.15.87:0/3681040715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f50102110 0x7f4f501083e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f556d6700 1 -- 172.21.15.87:0/3681040715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f500f9510 msgr2=0x7f4f5010e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f556d6700 1 --2- 172.21.15.87:0/3681040715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f500f9510 0x7f4f5010e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f556d6700 1 -- 172.21.15.87:0/3681040715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f40005040 con 0x7f4f500f8bb0 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f556d6700 1 --2- 172.21.15.87:0/3681040715 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f500f8bb0 0x7f4f5010e240 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f4f4c009eb0 tx=0x7f4f4c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f467fc700 10 client.?.objecter ms_handle_connect 0x7f4f500f8bb0 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f467fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f467fc700 1 -- 172.21.15.87:0/3681040715 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c00e070 con 0x7f4f500f8bb0 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f467fc700 1 -- 172.21.15.87:0/3681040715 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f4c013da0 con 0x7f4f500f8bb0 2024-02-23T00:28:30.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c005040 con 0x7f4f500f8bb0 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f467fc700 1 -- 172.21.15.87:0/3681040715 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c019070 con 0x7f4f500f8bb0 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f5793a700 10 client.24539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f501090a0 con 0x7f4f500f8bb0 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.294+0000 7f4f5793a700 1 librados: init done 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.295+0000 7f4f5793a700 10 client.24539.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.295+0000 7f4f5793a700 15 client.24539.objecter _session_command_op_assign -1 1 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.295+0000 7f4f5793a700 20 client.24539.objecter _get_session osd=-1 returning homeless 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.295+0000 7f4f5793a700 10 client.24539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.295+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f5005f970 con 0x7f4f500f8bb0 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.295+0000 7f4f467fc700 1 -- 172.21.15.87:0/3681040715 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4f4c01e420 con 0x7f4f500f8bb0 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 1 --2- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f3c062c90 0x7f4f3c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:30.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 1 -- 172.21.15.87:0/3681040715 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f4f4c07d420 con 0x7f4f500f8bb0 2024-02-23T00:28:30.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 10 client.24539.objecter ms_dispatch 0x7f4f50001550 osd_map(76..76 src has 1..76) v4 2024-02-23T00:28:30.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 3 client.24539.objecter handle_osd_map got epochs [76,76] > 0 2024-02-23T00:28:30.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 3 client.24539.objecter handle_osd_map decoding full epoch 76 2024-02-23T00:28:30.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f54ed5700 1 --2- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f3c062c90 0x7f4f3c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:30.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 10 client.24539.objecter checking command 1 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 1 --2- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6805/685354171 conn(0x7f4f3c06c5b0 0x7f4f3c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 20 client.24539.objecter _get_session s=0x7f4f3c06bf00 osd=1 3 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 20 client.24539.objecter put_session s=0x7f4f3c06bf00 osd=1 3 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 15 client.24539.objecter _session_command_op_remove -1 1 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 20 client.24539.objecter _get_session s=0x7f4f3c06bf00 osd=1 3 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 20 client.24539.objecter get_session s=0x7f4f3c06bf00 osd=1 3 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 15 client.24539.objecter _session_command_op_assign 1 1 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 20 client.24539.objecter put_session s=0x7f4f3c06bf00 osd=1 4 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 10 client.24539.objecter _send_command 1 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 1 -- 172.21.15.87:0/3681040715 --> v2:172.21.15.87:6805/685354171 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f3c06f090 con 0x7f4f3c06c5b0 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 20 client.24539.objecter dump_active .. 0 homeless 2024-02-23T00:28:30.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 1 -- 172.21.15.87:0/3681040715 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f4c07f900 con 0x7f4f500f8bb0 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f55ed7700 1 --2- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6805/685354171 conn(0x7f4f3c06c5b0 0x7f4f3c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 10 client.24539.objecter ms_handle_connect 0x7f4f3c062c90 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f54ed5700 1 --2- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f3c062c90 0x7f4f3c065150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4f50109b80 tx=0x7f4f4000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f467fc700 10 client.24539.objecter ms_handle_connect 0x7f4f3c06c5b0 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.296+0000 7f4f55ed7700 1 --2- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6805/685354171 conn(0x7f4f3c06c5b0 0x7f4f3c06ea50 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.297+0000 7f4f467fc700 1 -- 172.21.15.87:0/3681040715 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4f3c06f090 con 0x7f4f3c06c5b0 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.297+0000 7f4f467fc700 10 client.24539.objecter ms_dispatch 0x7f4f50001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.297+0000 7f4f467fc700 10 client.24539.objecter _finish_command 1 = system:0 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.297+0000 7f4f467fc700 20 client.24539.objecter put_session s=0x7f4f3c06bf00 osd=1 4 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.297+0000 7f4f467fc700 15 client.24539.objecter _session_command_op_remove 1 1 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 10 client.24539.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:28:30.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 15 client.24539.objecter _session_command_op_assign -1 2 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 20 client.24539.objecter _get_session s=0x7f4f3c06bf00 osd=1 3 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 20 client.24539.objecter put_session s=0x7f4f3c06bf00 osd=1 3 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 20 client.24539.objecter _get_session s=0x7f4f3c06bf00 osd=1 3 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 15 client.24539.objecter _session_command_op_remove -1 2 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 20 client.24539.objecter get_session s=0x7f4f3c06bf00 osd=1 3 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 15 client.24539.objecter _session_command_op_assign 1 2 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 20 client.24539.objecter put_session s=0x7f4f3c06bf00 osd=1 4 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 10 client.24539.objecter _send_command 2 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4f5005ee80 con 0x7f4f3c06c5b0 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f467fc700 1 -- 172.21.15.87:0/3681040715 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4f5005ee80 con 0x7f4f3c06c5b0 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f467fc700 10 client.24539.objecter ms_dispatch 0x7f4f50001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:28:30.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f467fc700 10 client.24539.objecter _finish_command 2 = system:0 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f467fc700 20 client.24539.objecter put_session s=0x7f4f3c06bf00 osd=1 4 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.311+0000 7f4f467fc700 15 client.24539.objecter _session_command_op_remove 1 2 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 10 librados: watch_flush enter 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 10 librados: watch_flush exit 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 10 client.24539.objecter close_session for osd.1 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6805/685354171 conn(0x7f4f3c06c5b0 msgr2=0x7f4f3c06ea50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 --2- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6805/685354171 conn(0x7f4f3c06c5b0 0x7f4f3c06ea50 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 20 client.24539.objecter put_session s=0x7f4f3c06bf00 osd=1 1 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 20 client.24539.objecter shutdown clearing up homeless session... 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 10 client.24539.objecter successfully canceled tick 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f3c062c90 msgr2=0x7f4f3c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 --2- 172.21.15.87:0/3681040715 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f3c062c90 0x7f4f3c065150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4f50109b80 tx=0x7f4f4000b040).stop 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f500f8bb0 msgr2=0x7f4f5010e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:30.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 --2- 172.21.15.87:0/3681040715 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f500f8bb0 0x7f4f5010e240 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f4f4c009eb0 tx=0x7f4f4c00b040).stop 2024-02-23T00:28:30.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f55ed7700 1 -- 172.21.15.87:0/3681040715 reap_dead start 2024-02-23T00:28:30.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 shutdown_connections 2024-02-23T00:28:30.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 >> 172.21.15.87:0/3681040715 conn(0x7f4f500f26e0 msgr2=0x7f4f500f3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:30.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 shutdown_connections 2024-02-23T00:28:30.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 -- 172.21.15.87:0/3681040715 wait complete. 2024-02-23T00:28:30.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:30.312+0000 7f4f5793a700 1 librados: shutdown 2024-02-23T00:28:30.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:28:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:28:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:29.466833+0000 mgr.y (mgr.24437) 45 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:29.522941+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.d scrub starts 2024-02-23T00:28:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:29.526500+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.d scrub ok 2024-02-23T00:28:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:30.083117+0000 osd.3 (osd.3) 17 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:28:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:30.086005+0000 osd.3 (osd.3) 18 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:28:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:30.233158+0000 osd.0 (osd.0) 13 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:28:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:30.236185+0000 osd.0 (osd.0) 14 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:29.466833+0000 mgr.y (mgr.24437) 45 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:29.522941+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.d scrub starts 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:29.526500+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.d scrub ok 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:30.083117+0000 osd.3 (osd.3) 17 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:30.086005+0000 osd.3 (osd.3) 18 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:30.233158+0000 osd.0 (osd.0) 13 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:30.236185+0000 osd.0 (osd.0) 14 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:29.466833+0000 mgr.y (mgr.24437) 45 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:29.522941+0000 osd.4 (osd.4) 7 : cluster [DBG] 4.d scrub starts 2024-02-23T00:28:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:29.526500+0000 osd.4 (osd.4) 8 : cluster [DBG] 4.d scrub ok 2024-02-23T00:28:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:30.083117+0000 osd.3 (osd.3) 17 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:28:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:30.086005+0000 osd.3 (osd.3) 18 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:28:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:30.233158+0000 osd.0 (osd.0) 13 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:28:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:30.236185+0000 osd.0 (osd.0) 14 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:28:31.368 INFO:teuthology.orchestra.run.smithi087.stdout:55834574893 2024-02-23T00:28:31.368 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-23T00:28:31.784 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:32.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:30.511737+0000 mgr.y (mgr.24437) 46 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:32.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:31.577296+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:32.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:30.511737+0000 mgr.y (mgr.24437) 46 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:32.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:31.577296+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:32.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:30.511737+0000 mgr.y (mgr.24437) 46 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:32.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:31.577296+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:32.522 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:28:32] "GET /metrics HTTP/1.1" 200 172163 "" "Prometheus/2.33.4" 2024-02-23T00:28:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:31.467442+0000 mgr.y (mgr.24437) 47 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:31.554252+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e scrub starts 2024-02-23T00:28:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:31.557715+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e scrub ok 2024-02-23T00:28:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:31.644856+0000 osd.7 (osd.7) 11 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:28:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:31.646089+0000 osd.7 (osd.7) 12 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:28:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:32.087974+0000 osd.3 (osd.3) 19 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:28:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:32.090865+0000 osd.3 (osd.3) 20 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:28:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:32.599579+0000 osd.1 (osd.1) 11 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:28:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:32.603084+0000 osd.1 (osd.1) 12 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:28:33.279 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:31.467442+0000 mgr.y (mgr.24437) 47 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:33.279 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:31.554252+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e scrub starts 2024-02-23T00:28:33.279 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:31.557715+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e scrub ok 2024-02-23T00:28:33.279 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:31.644856+0000 osd.7 (osd.7) 11 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:28:33.279 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:31.646089+0000 osd.7 (osd.7) 12 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:28:33.279 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:32.087974+0000 osd.3 (osd.3) 19 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:28:33.279 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:32.090865+0000 osd.3 (osd.3) 20 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:28:33.279 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:32.599579+0000 osd.1 (osd.1) 11 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:32.603084+0000 osd.1 (osd.1) 12 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:31.467442+0000 mgr.y (mgr.24437) 47 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:31.554252+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.e scrub starts 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:31.557715+0000 osd.4 (osd.4) 10 : cluster [DBG] 5.e scrub ok 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:31.644856+0000 osd.7 (osd.7) 11 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:31.646089+0000 osd.7 (osd.7) 12 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:32.087974+0000 osd.3 (osd.3) 19 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:32.090865+0000 osd.3 (osd.3) 20 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:32.599579+0000 osd.1 (osd.1) 11 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:28:33.280 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:32.603084+0000 osd.1 (osd.1) 12 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:28:34.168 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:28:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:28:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.071059+0000 osd.3 (osd.3) 21 : cluster [DBG] 4.1d deep-scrub starts 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.077838+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.1d deep-scrub ok 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.088435+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.090449+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:33.167708+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:33.168530+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:33.169218+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:33.173445+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:28:33.177085+0000 mgr.y (mgr.24437) 48 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:28:33.180156+0000 mgr.y (mgr.24437) 49 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi087 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.613899+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:28:34.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.614958+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:28:34.171 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.932517+0000 osd.5 (osd.5) 11 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:28:34.171 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.933520+0000 osd.5 (osd.5) 12 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.234+0000 7fe33eb93700 1 -- 172.21.15.87:0/3151741851 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3380fcac0 msgr2=0x7fe3380fcea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.234+0000 7fe33eb93700 1 --2- 172.21.15.87:0/3151741851 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3380fcac0 0x7fe3380fcea0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe32c004870 tx=0x7fe32c009cb0).stop 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.234+0000 7fe33eb93700 1 -- 172.21.15.87:0/3151741851 shutdown_connections 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.234+0000 7fe33eb93700 1 --2- 172.21.15.87:0/3151741851 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3380f7330 0x7fe338103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.234+0000 7fe33eb93700 1 --2- 172.21.15.87:0/3151741851 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3380fcac0 0x7fe3380fcea0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.234+0000 7fe33eb93700 1 --2- 172.21.15.87:0/3151741851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3380fd420 0x7fe3380f6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.234+0000 7fe33eb93700 1 -- 172.21.15.87:0/3151741851 >> 172.21.15.87:0/3151741851 conn(0x7fe3380f26f0 msgr2=0x7fe3380f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.235+0000 7fe33eb93700 1 -- 172.21.15.87:0/3151741851 shutdown_connections 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.235+0000 7fe33eb93700 1 -- 172.21.15.87:0/3151741851 wait complete. 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.235+0000 7fe33eb93700 1 librados: starting msgr at 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.235+0000 7fe33eb93700 1 librados: starting objecter 2024-02-23T00:28:34.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.235+0000 7fe33eb93700 1 Processor -- start 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.235+0000 7fe33eb93700 1 -- start start 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.235+0000 7fe33eb93700 1 librados: setting wanted keys 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.235+0000 7fe33eb93700 1 librados: calling monclient init 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33eb93700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3380f7330 0x7fe33810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33eb93700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3380fcac0 0x7fe33810d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33eb93700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3380fd420 0x7fe338107190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33eb93700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe338107680 con 0x7fe3380fd420 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33eb93700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3381077f0 con 0x7fe3380f7330 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33eb93700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3381079b0 con 0x7fe3380fcac0 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33d130700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3380fd420 0x7fe338107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33d130700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3380fd420 0x7fe338107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56260/0 (socket says 172.21.15.87:56260) 2024-02-23T00:28:34.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe337fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3380fcac0 0x7fe33810d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33d130700 1 -- 172.21.15.87:0/476635920 learned_addr learned my addr 172.21.15.87:0/476635920 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33c92f700 1 --2- 172.21.15.87:0/476635920 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3380f7330 0x7fe33810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33d130700 1 -- 172.21.15.87:0/476635920 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3380fcac0 msgr2=0x7fe33810d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33d130700 1 --2- 172.21.15.87:0/476635920 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3380fcac0 0x7fe33810d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33d130700 1 -- 172.21.15.87:0/476635920 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3380f7330 msgr2=0x7fe33810cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33d130700 1 --2- 172.21.15.87:0/476635920 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3380f7330 0x7fe33810cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33d130700 1 -- 172.21.15.87:0/476635920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3300097c0 con 0x7fe3380fd420 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe337fff700 1 --2- 172.21.15.87:0/476635920 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3380fcac0 0x7fe33810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33d130700 1 --2- 172.21.15.87:0/476635920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3380fd420 0x7fe338107190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe330001df0 tx=0x7fe330004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe335ffb700 10 client.?.objecter ms_handle_connect 0x7fe3380fd420 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe335ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe335ffb700 1 -- 172.21.15.87:0/476635920 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330009a10 con 0x7fe3380fd420 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe335ffb700 1 -- 172.21.15.87:0/476635920 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3300090d0 con 0x7fe3380fd420 2024-02-23T00:28:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c005040 con 0x7fe3380fd420 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.236+0000 7fe335ffb700 1 -- 172.21.15.87:0/476635920 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330007d30 con 0x7fe3380fd420 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.237+0000 7fe33eb93700 10 client.14619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.237+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe338107fe0 con 0x7fe3380fd420 2024-02-23T00:28:34.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.237+0000 7fe33eb93700 1 librados: init done 2024-02-23T00:28:34.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.237+0000 7fe33eb93700 10 client.14619.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:28:34.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.237+0000 7fe33eb93700 15 client.14619.objecter _session_command_op_assign -1 1 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.237+0000 7fe33eb93700 20 client.14619.objecter _get_session osd=-1 returning homeless 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.237+0000 7fe33eb93700 10 client.14619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.237+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe3380606d0 con 0x7fe3380fd420 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.238+0000 7fe335ffb700 1 -- 172.21.15.87:0/476635920 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fe330009240 con 0x7fe3380fd420 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.238+0000 7fe335ffb700 1 --2- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe328062c90 0x7fe328065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 1 -- 172.21.15.87:0/476635920 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6056+0+0 (secure 0 0 0) 0x7fe33007d750 con 0x7fe3380fd420 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 10 client.14619.objecter ms_dispatch 0x7fe338001550 osd_map(76..76 src has 1..76) v4 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 3 client.14619.objecter handle_osd_map got epochs [76,76] > 0 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 3 client.14619.objecter handle_osd_map decoding full epoch 76 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe33c92f700 1 --2- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe328062c90 0x7fe328065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 10 client.14619.objecter checking command 1 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 1 --2- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe32806c530 0x7fe32806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 20 client.14619.objecter _get_session s=0x7fe32806be80 osd=2 3 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 20 client.14619.objecter put_session s=0x7fe32806be80 osd=2 3 2024-02-23T00:28:34.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 15 client.14619.objecter _session_command_op_remove -1 1 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 20 client.14619.objecter _get_session s=0x7fe32806be80 osd=2 3 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 20 client.14619.objecter get_session s=0x7fe32806be80 osd=2 3 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 15 client.14619.objecter _session_command_op_assign 2 1 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 20 client.14619.objecter put_session s=0x7fe32806be80 osd=2 4 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 10 client.14619.objecter _send_command 1 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 1 -- 172.21.15.87:0/476635920 --> v2:172.21.15.87:6809/3605394228 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe32806f010 con 0x7fe32806c530 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 20 client.14619.objecter dump_active .. 0 homeless 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 1 -- 172.21.15.87:0/476635920 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe33007da40 con 0x7fe3380fd420 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe337fff700 1 --2- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe32806c530 0x7fe32806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 10 client.14619.objecter ms_handle_connect 0x7fe328062c90 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe33c92f700 1 --2- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe328062c90 0x7fe328065150 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe32c009a00 tx=0x7fe32c018040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe335ffb700 10 client.14619.objecter ms_handle_connect 0x7fe32806c530 2024-02-23T00:28:34.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.239+0000 7fe337fff700 1 --2- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe32806c530 0x7fe32806e9d0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:34.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.240+0000 7fe335ffb700 1 -- 172.21.15.87:0/476635920 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe32806f010 con 0x7fe32806c530 2024-02-23T00:28:34.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.240+0000 7fe335ffb700 10 client.14619.objecter ms_dispatch 0x7fe338001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:28:34.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.240+0000 7fe335ffb700 10 client.14619.objecter _finish_command 1 = system:0 2024-02-23T00:28:34.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.240+0000 7fe335ffb700 20 client.14619.objecter put_session s=0x7fe32806be80 osd=2 4 2024-02-23T00:28:34.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.240+0000 7fe335ffb700 15 client.14619.objecter _session_command_op_remove 2 1 2024-02-23T00:28:34.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 10 client.14619.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:28:34.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 15 client.14619.objecter _session_command_op_assign -1 2 2024-02-23T00:28:34.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 20 client.14619.objecter _get_session s=0x7fe32806be80 osd=2 3 2024-02-23T00:28:34.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 20 client.14619.objecter put_session s=0x7fe32806be80 osd=2 3 2024-02-23T00:28:34.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 20 client.14619.objecter _get_session s=0x7fe32806be80 osd=2 3 2024-02-23T00:28:34.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 15 client.14619.objecter _session_command_op_remove -1 2 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 20 client.14619.objecter get_session s=0x7fe32806be80 osd=2 3 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 15 client.14619.objecter _session_command_op_assign 2 2 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 20 client.14619.objecter put_session s=0x7fe32806be80 osd=2 4 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 10 client.14619.objecter _send_command 2 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 --> v2:172.21.15.87:6809/3605394228 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe33805ee80 con 0x7fe32806c530 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe335ffb700 1 -- 172.21.15.87:0/476635920 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe33805ee80 con 0x7fe32806c530 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe335ffb700 10 client.14619.objecter ms_dispatch 0x7fe338001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe335ffb700 10 client.14619.objecter _finish_command 2 = system:0 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe335ffb700 20 client.14619.objecter put_session s=0x7fe32806be80 osd=2 4 2024-02-23T00:28:34.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe335ffb700 15 client.14619.objecter _session_command_op_remove 2 2 2024-02-23T00:28:34.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 10 librados: watch_flush enter 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 10 librados: watch_flush exit 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 10 client.14619.objecter close_session for osd.2 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe32806c530 msgr2=0x7fe32806e9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 1 --2- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe32806c530 0x7fe32806e9d0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 20 client.14619.objecter put_session s=0x7fe32806be80 osd=2 1 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 20 client.14619.objecter shutdown clearing up homeless session... 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 10 client.14619.objecter successfully canceled tick 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe328062c90 msgr2=0x7fe328065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 1 --2- 172.21.15.87:0/476635920 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe328062c90 0x7fe328065150 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fe32c009a00 tx=0x7fe32c018040).stop 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3380fd420 msgr2=0x7fe338107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33eb93700 1 --2- 172.21.15.87:0/476635920 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3380fd420 0x7fe338107190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe330001df0 tx=0x7fe330004c50).stop 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.254+0000 7fe33d130700 1 -- 172.21.15.87:0/476635920 reap_dead start 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.255+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 shutdown_connections 2024-02-23T00:28:34.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.255+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 >> 172.21.15.87:0/476635920 conn(0x7fe3380f26f0 msgr2=0x7fe3380fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:34.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.255+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 shutdown_connections 2024-02-23T00:28:34.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.255+0000 7fe33eb93700 1 -- 172.21.15.87:0/476635920 wait complete. 2024-02-23T00:28:34.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:34.255+0000 7fe33eb93700 1 librados: shutdown 2024-02-23T00:28:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.071059+0000 osd.3 (osd.3) 21 : cluster [DBG] 4.1d deep-scrub starts 2024-02-23T00:28:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.077838+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.1d deep-scrub ok 2024-02-23T00:28:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.088435+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:28:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.090449+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:28:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:33.167708+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:33.168530+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:28:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:33.169218+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:28:34.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:33.173445+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:34.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:28:33.177085+0000 mgr.y (mgr.24437) 48 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-23T00:28:34.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:28:33.180156+0000 mgr.y (mgr.24437) 49 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi087 2024-02-23T00:28:34.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.613899+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:28:34.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.614958+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:28:34.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.932517+0000 osd.5 (osd.5) 11 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:28:34.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.933520+0000 osd.5 (osd.5) 12 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:28:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.071059+0000 osd.3 (osd.3) 21 : cluster [DBG] 4.1d deep-scrub starts 2024-02-23T00:28:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.077838+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.1d deep-scrub ok 2024-02-23T00:28:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.088435+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:28:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.090449+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:28:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:33.167708+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:33.168530+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:28:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:33.169218+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:28:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:33.173445+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:28:33.177085+0000 mgr.y (mgr.24437) 48 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-23T00:28:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:28:33.180156+0000 mgr.y (mgr.24437) 49 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi087 2024-02-23T00:28:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.613899+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:28:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.614958+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:28:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.932517+0000 osd.5 (osd.5) 11 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:28:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.933520+0000 osd.5 (osd.5) 12 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:28:35.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.467928+0000 mgr.y (mgr.24437) 50 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.502505+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:33.503235+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:34.063555+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:34.066437+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:34.606871+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:34.610329+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.467928+0000 mgr.y (mgr.24437) 50 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.502505+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:33.503235+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:28:35.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:34.063555+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:28:35.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:34.066437+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:28:35.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:34.606871+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:28:35.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:34.610329+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:28:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.467928+0000 mgr.y (mgr.24437) 50 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:28:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.502505+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:28:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:33.503235+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:28:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:34.063555+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:28:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:34.066437+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:28:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:34.606871+0000 osd.7 (osd.7) 13 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:28:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:34.610329+0000 osd.7 (osd.7) 14 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:28:35.924 INFO:teuthology.orchestra.run.smithi087.stdout:77309411370 2024-02-23T00:28:35.925 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-23T00:28:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:35.055752+0000 osd.3 (osd.3) 23 : cluster [DBG] 6.b scrub starts 2024-02-23T00:28:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:35.058769+0000 osd.3 (osd.3) 24 : cluster [DBG] 6.b scrub ok 2024-02-23T00:28:36.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:35.055752+0000 osd.3 (osd.3) 23 : cluster [DBG] 6.b scrub starts 2024-02-23T00:28:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:35.058769+0000 osd.3 (osd.3) 24 : cluster [DBG] 6.b scrub ok 2024-02-23T00:28:36.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:35.055752+0000 osd.3 (osd.3) 23 : cluster [DBG] 6.b scrub starts 2024-02-23T00:28:36.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:35.058769+0000 osd.3 (osd.3) 24 : cluster [DBG] 6.b scrub ok 2024-02-23T00:28:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:35.468503+0000 mgr.y (mgr.24437) 51 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:36.067449+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:28:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:36.070953+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:28:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:36.207446+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:28:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:36.209106+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:28:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:36.535098+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:28:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:36.536955+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:28:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:36.673006+0000 osd.1 (osd.1) 13 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:28:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:36.676575+0000 osd.1 (osd.1) 14 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:28:37.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:35.468503+0000 mgr.y (mgr.24437) 51 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:36.067449+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:28:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:36.070953+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:28:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:36.207446+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:28:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:36.209106+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:28:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:36.535098+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:28:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:36.536955+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:28:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:36.673006+0000 osd.1 (osd.1) 13 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:28:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:36.676575+0000 osd.1 (osd.1) 14 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:28:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:35.468503+0000 mgr.y (mgr.24437) 51 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:36.067449+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:28:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:36.070953+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:28:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:36.207446+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:28:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:36.209106+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:28:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:36.535098+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:28:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:36.536955+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:28:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:36.673006+0000 osd.1 (osd.1) 13 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:28:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:36.676575+0000 osd.1 (osd.1) 14 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:28:37.682 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:38.264 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:37.036744+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:28:38.264 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:37.040210+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:28:38.264 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:37.103361+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:28:38.264 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:37.106304+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:28:38.264 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:37.036744+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:28:38.265 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:37.040210+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:28:38.265 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:37.103361+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:28:38.265 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:37.106304+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:28:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:37.036744+0000 osd.3 (osd.3) 25 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:28:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:37.040210+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:28:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:37.103361+0000 osd.6 (osd.6) 17 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:28:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:37.106304+0000 osd.6 (osd.6) 18 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:28:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:37.468883+0000 mgr.y (mgr.24437) 52 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:28:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:38.953875+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.15 deep-scrub starts 2024-02-23T00:28:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:38.956426+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.15 deep-scrub ok 2024-02-23T00:28:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:37.468883+0000 mgr.y (mgr.24437) 52 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:28:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:38.953875+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.15 deep-scrub starts 2024-02-23T00:28:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:38.956426+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.15 deep-scrub ok 2024-02-23T00:28:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:37.468883+0000 mgr.y (mgr.24437) 52 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:28:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:38.953875+0000 osd.5 (osd.5) 13 : cluster [DBG] 5.15 deep-scrub starts 2024-02-23T00:28:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:38.956426+0000 osd.5 (osd.5) 14 : cluster [DBG] 5.15 deep-scrub ok 2024-02-23T00:28:39.584 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:39 smithi087 systemd[1]: Stopping Ceph alertmanager.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:28:39.584 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:39 smithi087 bash[70599]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager.a" found: no such container 2024-02-23T00:28:40.153 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[67003]: level=info ts=2024-02-23T00:28:40.151Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-23T00:28:40.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.132+0000 7f8284a2f700 1 -- 172.21.15.87:0/528411397 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f82800fd510 msgr2=0x7f82800f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:40.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.132+0000 7f8284a2f700 1 --2- 172.21.15.87:0/528411397 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f82800fd510 0x7f82800f6d40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f82740047b0 tx=0x7f8274009cb0).stop 2024-02-23T00:28:40.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 -- 172.21.15.87:0/528411397 shutdown_connections 2024-02-23T00:28:40.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 --2- 172.21.15.87:0/528411397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f82800f7400 0x7f8280103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:40.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 --2- 172.21.15.87:0/528411397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82800fcbb0 0x7f82800fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:28:40.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 --2- 172.21.15.87:0/528411397 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f82800fd510 0x7f82800f6d40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:40.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 -- 172.21.15.87:0/528411397 >> 172.21.15.87:0/528411397 conn(0x7f82800f2820 msgr2=0x7f82800f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:40.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 -- 172.21.15.87:0/528411397 shutdown_connections 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 -- 172.21.15.87:0/528411397 wait complete. 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 librados: starting msgr at 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 librados: starting objecter 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 Processor -- start 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 -- start start 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 librados: setting wanted keys 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.133+0000 7f8284a2f700 1 librados: calling monclient init 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f82800f7400 0x7f8280109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f82800fcbb0 0x7f8280109820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82800fd510 0x7f8280109de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8280114520 con 0x7f82800fcbb0 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8280114690 con 0x7f82800fd510 2024-02-23T00:28:40.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8280114850 con 0x7f82800f7400 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827e59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f82800f7400 0x7f8280109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827e59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f82800f7400 0x7f8280109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40616/0 (socket says 172.21.15.87:40616) 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827e59c700 1 -- 172.21.15.87:0/2846831881 learned_addr learned my addr 172.21.15.87:0/2846831881 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827dd9b700 1 --2- 172.21.15.87:0/2846831881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f82800fcbb0 0x7f8280109820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827ed9d700 1 --2- 172.21.15.87:0/2846831881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82800fd510 0x7f8280109de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827e59c700 1 -- 172.21.15.87:0/2846831881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82800fd510 msgr2=0x7f8280109de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827e59c700 1 --2- 172.21.15.87:0/2846831881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82800fd510 0x7f8280109de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827e59c700 1 -- 172.21.15.87:0/2846831881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f82800fcbb0 msgr2=0x7f8280109820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827e59c700 1 --2- 172.21.15.87:0/2846831881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f82800fcbb0 0x7f8280109820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827e59c700 1 -- 172.21.15.87:0/2846831881 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82780097c0 con 0x7f82800f7400 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827ed9d700 1 --2- 172.21.15.87:0/2846831881 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f82800fd510 0x7f8280109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:28:40.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827dd9b700 1 --2- 172.21.15.87:0/2846831881 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f82800fcbb0 0x7f8280109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f827e59c700 1 --2- 172.21.15.87:0/2846831881 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f82800f7400 0x7f8280109330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f827000b530 tx=0x7f82700090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f826f7fe700 10 client.?.objecter ms_handle_connect 0x7f82800f7400 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f826f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f826f7fe700 1 -- 172.21.15.87:0/2846831881 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827000d4a0 con 0x7f82800f7400 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f826f7fe700 1 -- 172.21.15.87:0/2846831881 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f827000dad0 con 0x7f82800f7400 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8274005040 con 0x7f82800f7400 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f826f7fe700 1 -- 172.21.15.87:0/2846831881 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270012a50 con 0x7f82800f7400 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 10 client.24551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8280114e90 con 0x7f82800f7400 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.134+0000 7f8284a2f700 1 librados: init done 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.135+0000 7f8284a2f700 10 client.24551.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.135+0000 7f8284a2f700 15 client.24551.objecter _session_command_op_assign -1 1 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.135+0000 7f8284a2f700 20 client.24551.objecter _get_session osd=-1 returning homeless 2024-02-23T00:28:40.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.135+0000 7f8284a2f700 10 client.24551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.135+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f828005fa60 con 0x7f82800f7400 2024-02-23T00:28:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 1 -- 172.21.15.87:0/2846831881 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f827000d600 con 0x7f82800f7400 2024-02-23T00:28:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 1 --2- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8268062c90 0x7f8268065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f827dd9b700 1 --2- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8268062c90 0x7f8268065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 1 -- 172.21.15.87:0/2846831881 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6056+0+0 (secure 0 0 0) 0x7f827007cb30 con 0x7f82800f7400 2024-02-23T00:28:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 10 client.24551.objecter ms_dispatch 0x7f8280001550 osd_map(76..76 src has 1..76) v4 2024-02-23T00:28:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 3 client.24551.objecter handle_osd_map got epochs [76,76] > 0 2024-02-23T00:28:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 3 client.24551.objecter handle_osd_map decoding full epoch 76 2024-02-23T00:28:40.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 10 client.24551.objecter checking command 1 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 1 --2- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6813/128059790 conn(0x7f826806c4b0 0x7f826806e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 20 client.24551.objecter _get_session s=0x7f826806be00 osd=3 3 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 20 client.24551.objecter put_session s=0x7f826806be00 osd=3 3 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 15 client.24551.objecter _session_command_op_remove -1 1 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 20 client.24551.objecter _get_session s=0x7f826806be00 osd=3 3 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 20 client.24551.objecter get_session s=0x7f826806be00 osd=3 3 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 15 client.24551.objecter _session_command_op_assign 3 1 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 20 client.24551.objecter put_session s=0x7f826806be00 osd=3 4 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 10 client.24551.objecter _send_command 1 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 1 -- 172.21.15.87:0/2846831881 --> v2:172.21.15.87:6813/128059790 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f826806ef90 con 0x7f826806c4b0 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 20 client.24551.objecter dump_active .. 0 homeless 2024-02-23T00:28:40.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 1 -- 172.21.15.87:0/2846831881 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8270081220 con 0x7f82800f7400 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f827ed9d700 1 --2- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6813/128059790 conn(0x7f826806c4b0 0x7f826806e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f826f7fe700 10 client.24551.objecter ms_handle_connect 0x7f8268062c90 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.136+0000 7f827dd9b700 1 --2- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8268062c90 0x7f8268065150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8274004e20 tx=0x7f8274009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.137+0000 7f826f7fe700 10 client.24551.objecter ms_handle_connect 0x7f826806c4b0 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.137+0000 7f827ed9d700 1 --2- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6813/128059790 conn(0x7f826806c4b0 0x7f826806e950 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.137+0000 7f826f7fe700 1 -- 172.21.15.87:0/2846831881 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f826806ef90 con 0x7f826806c4b0 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.137+0000 7f826f7fe700 10 client.24551.objecter ms_dispatch 0x7f8280001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.137+0000 7f826f7fe700 10 client.24551.objecter _finish_command 1 = system:0 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.137+0000 7f826f7fe700 20 client.24551.objecter put_session s=0x7f826806be00 osd=3 4 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.137+0000 7f826f7fe700 15 client.24551.objecter _session_command_op_remove 3 1 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 10 client.24551.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 15 client.24551.objecter _session_command_op_assign -1 2 2024-02-23T00:28:40.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 20 client.24551.objecter _get_session s=0x7f826806be00 osd=3 3 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 20 client.24551.objecter put_session s=0x7f826806be00 osd=3 3 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 20 client.24551.objecter _get_session s=0x7f826806be00 osd=3 3 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 15 client.24551.objecter _session_command_op_remove -1 2 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 20 client.24551.objecter get_session s=0x7f826806be00 osd=3 3 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 15 client.24551.objecter _session_command_op_assign 3 2 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 20 client.24551.objecter put_session s=0x7f826806be00 osd=3 4 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 10 client.24551.objecter _send_command 2 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.151+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 --> v2:172.21.15.87:6813/128059790 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f828005ee80 con 0x7f826806c4b0 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f826f7fe700 1 -- 172.21.15.87:0/2846831881 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f828005ee80 con 0x7f826806c4b0 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f826f7fe700 10 client.24551.objecter ms_dispatch 0x7f8280001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f826f7fe700 10 client.24551.objecter _finish_command 2 = system:0 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f826f7fe700 20 client.24551.objecter put_session s=0x7f826806be00 osd=3 4 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f826f7fe700 15 client.24551.objecter _session_command_op_remove 3 2 2024-02-23T00:28:40.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 10 librados: watch_flush enter 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 10 librados: watch_flush exit 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 10 client.24551.objecter close_session for osd.3 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6813/128059790 conn(0x7f826806c4b0 msgr2=0x7f826806e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 --2- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6813/128059790 conn(0x7f826806c4b0 0x7f826806e950 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 20 client.24551.objecter put_session s=0x7f826806be00 osd=3 1 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 20 client.24551.objecter shutdown clearing up homeless session... 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 10 client.24551.objecter successfully canceled tick 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8268062c90 msgr2=0x7f8268065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 --2- 172.21.15.87:0/2846831881 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8268062c90 0x7f8268065150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8274004e20 tx=0x7f8274009cb0).stop 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f82800f7400 msgr2=0x7f8280109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 --2- 172.21.15.87:0/2846831881 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f82800f7400 0x7f8280109330 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f827000b530 tx=0x7f82700090f0).stop 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f827ed9d700 1 -- 172.21.15.87:0/2846831881 reap_dead start 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 shutdown_connections 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 >> 172.21.15.87:0/2846831881 conn(0x7f82800f2820 msgr2=0x7f82800fb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 shutdown_connections 2024-02-23T00:28:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 -- 172.21.15.87:0/2846831881 wait complete. 2024-02-23T00:28:40.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:40.152+0000 7f8284a2f700 1 librados: shutdown 2024-02-23T00:28:40.523 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:39.073617+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.11 scrub starts 2024-02-23T00:28:40.523 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:39.076627+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.11 scrub ok 2024-02-23T00:28:40.523 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:39.475876+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-23T00:28:40.523 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:39.476099+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-23T00:28:40.523 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:39.476257+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-23T00:28:40.523 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:39.476426+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-02-23T00:28:40.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:39.493230+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:28:40.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:39.496155+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:28:40.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:39.613730+0000 osd.4 (osd.4) 15 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:28:40.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:39.614997+0000 osd.4 (osd.4) 16 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:28:40.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:39.625458+0000 osd.1 (osd.1) 15 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:28:40.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:39.629070+0000 osd.1 (osd.1) 16 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:28:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:39.073617+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.11 scrub starts 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:39.076627+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.11 scrub ok 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:39.475876+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:39.476099+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:39.476257+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:39.476426+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:39.493230+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:39.496155+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:39.613730+0000 osd.4 (osd.4) 15 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:39.614997+0000 osd.4 (osd.4) 16 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:39.625458+0000 osd.1 (osd.1) 15 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:28:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:39.629070+0000 osd.1 (osd.1) 16 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:39.073617+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.11 scrub starts 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:39.076627+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.11 scrub ok 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:39.475876+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:39.476099+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:39.476257+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:39.476426+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:39.493230+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:39.496155+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:39.613730+0000 osd.4 (osd.4) 15 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:39.614997+0000 osd.4 (osd.4) 16 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:39.625458+0000 osd.1 (osd.1) 15 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:28:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:39.629070+0000 osd.1 (osd.1) 16 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:28:40.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:28:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:28:40.954 INFO:teuthology.orchestra.run.smithi087.stdout:111669149735 2024-02-23T00:28:40.955 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-23T00:28:41.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:39.469522+0000 mgr.y (mgr.24437) 53 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:41.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:40.196766+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-02-23T00:28:41.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:40.196837+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-23T00:28:41.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:40.196891+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-23T00:28:41.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:40.196934+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-02-23T00:28:41.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:40.196981+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-23T00:28:41.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:40.317378+0000 osd.1 (osd.1) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:41.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:40.317450+0000 osd.1 (osd.1) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:41.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:40.417840+0000 osd.4 (osd.4) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:41.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:40.417932+0000 osd.4 (osd.4) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:41.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:40.626941+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.e scrub starts 2024-02-23T00:28:41.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:40.630133+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.e scrub ok 2024-02-23T00:28:41.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:40.998582+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:28:41.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:40.999509+0000 osd.5 (osd.5) 16 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:39.469522+0000 mgr.y (mgr.24437) 53 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:40.196766+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:40.196837+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:40.196891+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:40.196934+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:40.196981+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:40.317378+0000 osd.1 (osd.1) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:40.317450+0000 osd.1 (osd.1) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:40.417840+0000 osd.4 (osd.4) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:40.417932+0000 osd.4 (osd.4) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:40.626941+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.e scrub starts 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:40.630133+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.e scrub ok 2024-02-23T00:28:41.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:40.998582+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:28:41.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:40.999509+0000 osd.5 (osd.5) 16 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:28:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:39.469522+0000 mgr.y (mgr.24437) 53 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:28:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:40.196766+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2024-02-23T00:28:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:40.196837+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-23T00:28:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:40.196891+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-23T00:28:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:40.196934+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-02-23T00:28:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:40.196981+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-23T00:28:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:40.317378+0000 osd.1 (osd.1) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:40.317450+0000 osd.1 (osd.1) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:40.417840+0000 osd.4 (osd.4) 17 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:40.417932+0000 osd.4 (osd.4) 18 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:40.626941+0000 osd.4 (osd.4) 19 : cluster [DBG] 6.e scrub starts 2024-02-23T00:28:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:40.630133+0000 osd.4 (osd.4) 20 : cluster [DBG] 6.e scrub ok 2024-02-23T00:28:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:40.998582+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:28:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:40.999509+0000 osd.5 (osd.5) 16 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:28:41.832 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:41 smithi087 bash[70677]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a 2024-02-23T00:28:41.840 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:42.309 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:28:42] "GET /metrics HTTP/1.1" 200 172168 "" "Prometheus/2.33.4" 2024-02-23T00:28:42.309 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:42 smithi087 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-23T00:28:42.309 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:42 smithi087 systemd[1]: Stopped Ceph alertmanager.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:28:42.309 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:42 smithi087 systemd[1]: Starting Ceph alertmanager.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:40.522131+0000 mgr.y (mgr.24437) 54 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.110545+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.113675+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.196749+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.469956+0000 mgr.y (mgr.24437) 55 : cluster [DBG] pgmap v46: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.517522+0000 osd.7 (osd.7) 17 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.520408+0000 osd.7 (osd.7) 18 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.614965+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.617867+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:40.522131+0000 mgr.y (mgr.24437) 54 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:42.310 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.110545+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:28:42.311 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.113675+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:28:42.311 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.196749+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-23T00:28:42.311 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.469956+0000 mgr.y (mgr.24437) 55 : cluster [DBG] pgmap v46: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:42.311 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.517522+0000 osd.7 (osd.7) 17 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:28:42.311 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.520408+0000 osd.7 (osd.7) 18 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:28:42.311 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.614965+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:28:42.311 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.617867+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:28:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:40.522131+0000 mgr.y (mgr.24437) 54 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.110545+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:28:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.113675+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:28:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.196749+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-23T00:28:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.469956+0000 mgr.y (mgr.24437) 55 : cluster [DBG] pgmap v46: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.517522+0000 osd.7 (osd.7) 17 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:28:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.520408+0000 osd.7 (osd.7) 18 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:28:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.614965+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:28:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.617867+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:28:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.610690+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:28:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:41.611445+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:28:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:42.142913+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:28:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:42.144825+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:28:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:42.246058+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:28:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:42.249504+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:28:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.610690+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:28:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:41.611445+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:28:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:42.142913+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:28:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:42.144825+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:28:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:42.246058+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:28:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:42.249504+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:28:43.507 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.610690+0000 osd.2 (osd.2) 15 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:28:43.551 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:41.611445+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:28:43.551 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:42.142913+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:28:43.552 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:42.144825+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:28:43.552 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:42.246058+0000 osd.0 (osd.0) 17 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:28:43.552 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:42.249504+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:28:43.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 -- 172.21.15.87:0/4208433925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c900fdbb0 msgr2=0x7f8c900fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:43.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 --2- 172.21.15.87:0/4208433925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c900fdbb0 0x7f8c900fdf90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8c84004870 tx=0x7f8c84009cb0).stop 2024-02-23T00:28:43.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 -- 172.21.15.87:0/4208433925 shutdown_connections 2024-02-23T00:28:43.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 --2- 172.21.15.87:0/4208433925 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c900f8420 0x7f8c90104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:43.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 --2- 172.21.15.87:0/4208433925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c900fdbb0 0x7f8c900fdf90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:43.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 --2- 172.21.15.87:0/4208433925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c900fe510 0x7f8c900f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:43.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 -- 172.21.15.87:0/4208433925 >> 172.21.15.87:0/4208433925 conn(0x7f8c900f3800 msgr2=0x7f8c900f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 -- 172.21.15.87:0/4208433925 shutdown_connections 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 -- 172.21.15.87:0/4208433925 wait complete. 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 librados: starting msgr at 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.584+0000 7f8c96f68700 1 librados: starting objecter 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 Processor -- start 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 -- start start 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 librados: setting wanted keys 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 librados: calling monclient init 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c900f8420 0x7f8c9010e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c900fdbb0 0x7f8c9010e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c900fe510 0x7f8c90108220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c90108710 con 0x7f8c900fdbb0 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c90108880 con 0x7f8c900fe510 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c96f68700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c901089f0 con 0x7f8c900f8420 2024-02-23T00:28:43.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c94d04700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c900f8420 0x7f8c9010e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c94d04700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c900f8420 0x7f8c9010e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:52674/0 (socket says 172.21.15.87:52674) 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c94d04700 1 -- 172.21.15.87:0/3273784612 learned_addr learned my addr 172.21.15.87:0/3273784612 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c8ffff700 1 --2- 172.21.15.87:0/3273784612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c900fdbb0 0x7f8c9010e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c95505700 1 --2- 172.21.15.87:0/3273784612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c900fe510 0x7f8c90108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c94d04700 1 -- 172.21.15.87:0/3273784612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c900fe510 msgr2=0x7f8c90108220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c94d04700 1 --2- 172.21.15.87:0/3273784612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c900fe510 0x7f8c90108220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c94d04700 1 -- 172.21.15.87:0/3273784612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c900fdbb0 msgr2=0x7f8c9010e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c94d04700 1 --2- 172.21.15.87:0/3273784612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c900fdbb0 0x7f8c9010e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c94d04700 1 -- 172.21.15.87:0/3273784612 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c880097c0 con 0x7f8c900f8420 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c8ffff700 1 --2- 172.21.15.87:0/3273784612 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c900fdbb0 0x7f8c9010e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c95505700 1 --2- 172.21.15.87:0/3273784612 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c900fe510 0x7f8c90108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c94d04700 1 --2- 172.21.15.87:0/3273784612 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c900f8420 0x7f8c9010e040 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8c84009eb0 tx=0x7f8c8400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c8dffb700 10 client.?.objecter ms_handle_connect 0x7f8c900f8420 2024-02-23T00:28:43.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c8dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:43.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.585+0000 7f8c8dffb700 1 -- 172.21.15.87:0/3273784612 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8400e070 con 0x7f8c900f8420 2024-02-23T00:28:43.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.586+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c84005040 con 0x7f8c900f8420 2024-02-23T00:28:43.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.586+0000 7f8c8dffb700 1 -- 172.21.15.87:0/3273784612 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c84013da0 con 0x7f8c900f8420 2024-02-23T00:28:43.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.586+0000 7f8c8dffb700 1 -- 172.21.15.87:0/3273784612 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c84019070 con 0x7f8c900f8420 2024-02-23T00:28:43.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.586+0000 7f8c96f68700 10 client.24557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:43.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.586+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c90108f40 con 0x7f8c900f8420 2024-02-23T00:28:43.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.586+0000 7f8c96f68700 1 librados: init done 2024-02-23T00:28:43.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c96f68700 10 client.24557.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:28:43.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c96f68700 15 client.24557.objecter _session_command_op_assign -1 1 2024-02-23T00:28:43.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c96f68700 20 client.24557.objecter _get_session osd=-1 returning homeless 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c96f68700 10 client.24557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c9005ee80 con 0x7f8c900f8420 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c8dffb700 1 -- 172.21.15.87:0/3273784612 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f8c8401e420 con 0x7f8c900f8420 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c8dffb700 1 --2- 172.21.15.87:0/3273784612 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c80062c90 0x7f8c80065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c8ffff700 1 --2- 172.21.15.87:0/3273784612 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c80062c90 0x7f8c80065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c8dffb700 1 -- 172.21.15.87:0/3273784612 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f8c8407d680 con 0x7f8c900f8420 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.587+0000 7f8c8dffb700 10 client.24557.objecter ms_dispatch 0x7f8c90001550 osd_map(78..78 src has 1..78) v4 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 3 client.24557.objecter handle_osd_map got epochs [78,78] > 0 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 3 client.24557.objecter handle_osd_map decoding full epoch 78 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 10 client.24557.objecter checking command 1 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 1 --2- 172.21.15.87:0/3273784612 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c8006c650 0x7f8c8006eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 20 client.24557.objecter _get_session s=0x7f8c8006bfa0 osd=4 3 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 20 client.24557.objecter put_session s=0x7f8c8006bfa0 osd=4 3 2024-02-23T00:28:43.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 15 client.24557.objecter _session_command_op_remove -1 1 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 20 client.24557.objecter _get_session s=0x7f8c8006bfa0 osd=4 3 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 20 client.24557.objecter get_session s=0x7f8c8006bfa0 osd=4 3 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 15 client.24557.objecter _session_command_op_assign 4 1 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 20 client.24557.objecter put_session s=0x7f8c8006bfa0 osd=4 4 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 10 client.24557.objecter _send_command 1 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 1 -- 172.21.15.87:0/3273784612 --> v2:172.21.15.171:6800/4175844757 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c8006f130 con 0x7f8c8006c650 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 20 client.24557.objecter dump_active .. 0 homeless 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 1 -- 172.21.15.87:0/3273784612 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c8407fca0 con 0x7f8c900f8420 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 10 client.24557.objecter ms_handle_connect 0x7f8c80062c90 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8ffff700 1 --2- 172.21.15.87:0/3273784612 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c80062c90 0x7f8c80065150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8c901099c0 tx=0x7f8c7c009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c95505700 1 --2- 172.21.15.87:0/3273784612 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c8006c650 0x7f8c8006eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c8dffb700 10 client.24557.objecter ms_handle_connect 0x7f8c8006c650 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.588+0000 7f8c95505700 1 --2- 172.21.15.87:0/3273784612 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c8006c650 0x7f8c8006eaf0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:43.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.589+0000 7f8c8dffb700 1 -- 172.21.15.87:0/3273784612 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8c8006f130 con 0x7f8c8006c650 2024-02-23T00:28:43.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.589+0000 7f8c8dffb700 10 client.24557.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:28:43.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.589+0000 7f8c8dffb700 10 client.24557.objecter _finish_command 1 = system:0 2024-02-23T00:28:43.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.589+0000 7f8c8dffb700 20 client.24557.objecter put_session s=0x7f8c8006bfa0 osd=4 4 2024-02-23T00:28:43.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.589+0000 7f8c8dffb700 15 client.24557.objecter _session_command_op_remove 4 1 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 10 client.24557.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 15 client.24557.objecter _session_command_op_assign -1 2 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 20 client.24557.objecter _get_session s=0x7f8c8006bfa0 osd=4 3 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 20 client.24557.objecter put_session s=0x7f8c8006bfa0 osd=4 3 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 20 client.24557.objecter _get_session s=0x7f8c8006bfa0 osd=4 3 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 15 client.24557.objecter _session_command_op_remove -1 2 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 20 client.24557.objecter get_session s=0x7f8c8006bfa0 osd=4 3 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 15 client.24557.objecter _session_command_op_assign 4 2 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 20 client.24557.objecter put_session s=0x7f8c8006bfa0 osd=4 4 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 10 client.24557.objecter _send_command 2 2024-02-23T00:28:43.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.603+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 --> v2:172.21.15.171:6800/4175844757 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c901070a0 con 0x7f8c8006c650 2024-02-23T00:28:43.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c8dffb700 1 -- 172.21.15.87:0/3273784612 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8c901070a0 con 0x7f8c8006c650 2024-02-23T00:28:43.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c8dffb700 10 client.24557.objecter ms_dispatch 0x7f8c90001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:28:43.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c8dffb700 10 client.24557.objecter _finish_command 2 = system:0 2024-02-23T00:28:43.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c8dffb700 20 client.24557.objecter put_session s=0x7f8c8006bfa0 osd=4 4 2024-02-23T00:28:43.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c8dffb700 15 client.24557.objecter _session_command_op_remove 4 2 2024-02-23T00:28:43.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 10 librados: watch_flush enter 2024-02-23T00:28:43.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 10 librados: watch_flush exit 2024-02-23T00:28:43.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 10 client.24557.objecter close_session for osd.4 2024-02-23T00:28:43.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c8006c650 msgr2=0x7f8c8006eaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 --2- 172.21.15.87:0/3273784612 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c8006c650 0x7f8c8006eaf0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 20 client.24557.objecter put_session s=0x7f8c8006bfa0 osd=4 1 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 20 client.24557.objecter shutdown clearing up homeless session... 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 10 client.24557.objecter successfully canceled tick 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c80062c90 msgr2=0x7f8c80065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 --2- 172.21.15.87:0/3273784612 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c80062c90 0x7f8c80065150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8c901099c0 tx=0x7f8c7c009040).stop 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c900f8420 msgr2=0x7f8c9010e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 --2- 172.21.15.87:0/3273784612 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c900f8420 0x7f8c9010e040 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8c84009eb0 tx=0x7f8c8400b040).stop 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c95505700 1 -- 172.21.15.87:0/3273784612 reap_dead start 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 shutdown_connections 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 >> 172.21.15.87:0/3273784612 conn(0x7f8c900f3800 msgr2=0x7f8c900f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 shutdown_connections 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 -- 172.21.15.87:0/3273784612 wait complete. 2024-02-23T00:28:43.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:43.604+0000 7f8c96f68700 1 librados: shutdown 2024-02-23T00:28:43.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:28:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:28:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:28:43.942 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:28:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:28:43.506Z caller=notifier.go:526 level=error component=notifier alertmanager=http://smithi087.front.sepia.ceph.com:9093/api/v2/alerts count=1 msg="Error sending alert" err="Post \"http://smithi087.front.sepia.ceph.com:9093/api/v2/alerts\": dial tcp 172.21.15.87:9093: connect: connection refused" 2024-02-23T00:28:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:43.439722+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:28:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:43.440782+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:28:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:43.470466+0000 mgr.y (mgr.24437) 56 : cluster [DBG] pgmap v47: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:28:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:43.557705+0000 osd.4 (osd.4) 21 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:28:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:43.560601+0000 osd.4 (osd.4) 22 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:28:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:43.439722+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:28:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:43.440782+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:28:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:43.470466+0000 mgr.y (mgr.24437) 56 : cluster [DBG] pgmap v47: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:28:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:43.557705+0000 osd.4 (osd.4) 21 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:28:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:43.560601+0000 osd.4 (osd.4) 22 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:28:44.382 INFO:teuthology.orchestra.run.smithi087.stdout:133143986211 2024-02-23T00:28:44.383 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-23T00:28:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:43.439722+0000 osd.7 (osd.7) 19 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:28:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:43.440782+0000 osd.7 (osd.7) 20 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:28:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:43.470466+0000 mgr.y (mgr.24437) 56 : cluster [DBG] pgmap v47: 132 pgs: 4 peering, 128 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:28:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:43.557705+0000 osd.4 (osd.4) 21 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:28:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:43.560601+0000 osd.4 (osd.4) 22 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:28:45.165 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:45.298 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STOPPING 2024-02-23T00:28:45.298 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-23T00:28:45.299 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STOPPED 2024-02-23T00:28:45.299 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STARTING 2024-02-23T00:28:45.299 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:45.008Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-23T00:28:45.299 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:45.008Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-23T00:28:45.299 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:45.009Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.87 port=9094 2024-02-23T00:28:45.299 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:45.010Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-23T00:28:45.299 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:45.037Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-23T00:28:45.299 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:45.038Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-23T00:28:45.299 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:45.040Z caller=main.go:518 msg=Listening address=:9093 2024-02-23T00:28:45.299 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:45.040Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-23T00:28:45.300 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 bash[71060]: d9d250691d53cef27333aae02b7fa3c5c67f91859914d49f716dc91364ab4d5a 2024-02-23T00:28:45.300 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:45 smithi087 systemd[1]: Started Ceph alertmanager.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:28:45.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:43.989102+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:28:45.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:43.992066+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:28:45.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:44.582995+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.e scrub starts 2024-02-23T00:28:45.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:44.586722+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.e scrub ok 2024-02-23T00:28:45.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.157797+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.161040+0000 mon.a (mon.0) 704 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-23T00:28:45.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.161381+0000 mgr.y (mgr.24437) 57 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.162160+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi087.front.sepia.ceph.com:9093"}]: dispatch 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.162475+0000 mgr.y (mgr.24437) 58 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi087.front.sepia.ceph.com:9093"}]: dispatch 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.169677+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.174231+0000 mon.a (mon.0) 707 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.174752+0000 mgr.y (mgr.24437) 59 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.178024+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi171.front.sepia.ceph.com:9095"}]: dispatch 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.178538+0000 mgr.y (mgr.24437) 60 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi171.front.sepia.ceph.com:9095"}]: dispatch 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.185419+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.190697+0000 mon.a (mon.0) 710 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-23T00:28:45.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:43.989102+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:43.992066+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:44.582995+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.e scrub starts 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:44.586722+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.e scrub ok 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.157797+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.161040+0000 mon.a (mon.0) 704 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.161381+0000 mgr.y (mgr.24437) 57 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.162160+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi087.front.sepia.ceph.com:9093"}]: dispatch 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.162475+0000 mgr.y (mgr.24437) 58 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi087.front.sepia.ceph.com:9093"}]: dispatch 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.169677+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.174231+0000 mon.a (mon.0) 707 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.174752+0000 mgr.y (mgr.24437) 59 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.178024+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi171.front.sepia.ceph.com:9095"}]: dispatch 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.178538+0000 mgr.y (mgr.24437) 60 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi171.front.sepia.ceph.com:9095"}]: dispatch 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.185419+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.190697+0000 mon.a (mon.0) 710 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-23T00:28:45.303 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.191096+0000 mgr.y (mgr.24437) 61 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-23T00:28:45.303 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.193322+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi171.front.sepia.ceph.com:3000"}]: dispatch 2024-02-23T00:28:45.303 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.193690+0000 mgr.y (mgr.24437) 62 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi171.front.sepia.ceph.com:3000"}]: dispatch 2024-02-23T00:28:45.303 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.201688+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.303 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:45.203583+0000 mon.a (mon.0) 713 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:28:45.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.191096+0000 mgr.y (mgr.24437) 61 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-23T00:28:45.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.193322+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi171.front.sepia.ceph.com:3000"}]: dispatch 2024-02-23T00:28:45.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.193690+0000 mgr.y (mgr.24437) 62 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi171.front.sepia.ceph.com:3000"}]: dispatch 2024-02-23T00:28:45.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.201688+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:45.203583+0000 mon.a (mon.0) 713 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Serving on http://:::9283 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STARTED 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STOPPING 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STOPPED 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STARTING 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Serving on http://:::9283 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STARTED 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STOPPING 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STOPPED 2024-02-23T00:28:45.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STARTING 2024-02-23T00:28:45.584 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Serving on http://:::9283 2024-02-23T00:28:45.584 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:28:45] ENGINE Bus STARTED 2024-02-23T00:28:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:43.989102+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:28:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:43.992066+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:28:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:44.582995+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.e scrub starts 2024-02-23T00:28:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:44.586722+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.e scrub ok 2024-02-23T00:28:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.157797+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.161040+0000 mon.a (mon.0) 704 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.161381+0000 mgr.y (mgr.24437) 57 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.162160+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi087.front.sepia.ceph.com:9093"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.162475+0000 mgr.y (mgr.24437) 58 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi087.front.sepia.ceph.com:9093"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.169677+0000 mon.a (mon.0) 706 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.174231+0000 mon.a (mon.0) 707 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.174752+0000 mgr.y (mgr.24437) 59 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.178024+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi171.front.sepia.ceph.com:9095"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.178538+0000 mgr.y (mgr.24437) 60 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi171.front.sepia.ceph.com:9095"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.185419+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.190697+0000 mon.a (mon.0) 710 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.191096+0000 mgr.y (mgr.24437) 61 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.193322+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi171.front.sepia.ceph.com:3000"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.193690+0000 mgr.y (mgr.24437) 62 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi171.front.sepia.ceph.com:3000"}]: dispatch 2024-02-23T00:28:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.201688+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:45.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:45.203583+0000 mon.a (mon.0) 713 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:28:46.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:44.615372+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.d scrub starts 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:44.617032+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.d scrub ok 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:44.983919+0000 osd.6 (osd.6) 21 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:44.987415+0000 osd.6 (osd.6) 22 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:45.131717+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:45.134831+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:45.471124+0000 mgr.y (mgr.24437) 63 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:44.615372+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.d scrub starts 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:44.617032+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.d scrub ok 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:44.983919+0000 osd.6 (osd.6) 21 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:28:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:44.987415+0000 osd.6 (osd.6) 22 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:28:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:45.131717+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:28:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:45.134831+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:28:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:45.471124+0000 mgr.y (mgr.24437) 63 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-02-23T00:28:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:44.615372+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.d scrub starts 2024-02-23T00:28:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:44.617032+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.d scrub ok 2024-02-23T00:28:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:44.983919+0000 osd.6 (osd.6) 21 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:28:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:44.987415+0000 osd.6 (osd.6) 22 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:28:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:45.131717+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:28:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:45.134831+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:28:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:45.471124+0000 mgr.y (mgr.24437) 63 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-02-23T00:28:47.180 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:47.011Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000180206s 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.133+0000 7fddd3270700 1 -- 172.21.15.87:0/1957885611 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fddcc0f8420 msgr2=0x7fddcc104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.133+0000 7fddd3270700 1 --2- 172.21.15.87:0/1957885611 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fddcc0f8420 0x7fddcc104a20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fddc4009c30 tx=0x7fddc4004c50).stop 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.133+0000 7fddd3270700 1 -- 172.21.15.87:0/1957885611 shutdown_connections 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.133+0000 7fddd3270700 1 --2- 172.21.15.87:0/1957885611 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fddcc0f8420 0x7fddcc104a20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.133+0000 7fddd3270700 1 --2- 172.21.15.87:0/1957885611 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fddcc0fdbb0 0x7fddcc0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.133+0000 7fddd3270700 1 --2- 172.21.15.87:0/1957885611 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fddcc0fe510 0x7fddcc0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.133+0000 7fddd3270700 1 -- 172.21.15.87:0/1957885611 >> 172.21.15.87:0/1957885611 conn(0x7fddcc0f3800 msgr2=0x7fddcc0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 -- 172.21.15.87:0/1957885611 shutdown_connections 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 -- 172.21.15.87:0/1957885611 wait complete. 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 librados: starting msgr at 2024-02-23T00:28:47.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 librados: starting objecter 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 Processor -- start 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 -- start start 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 librados: setting wanted keys 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 librados: calling monclient init 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fddcc0f8420 0x7fddcc06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fddcc0fdbb0 0x7fddcc06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fddcc0fe510 0x7fddcc06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc1163f0 con 0x7fddcc0f8420 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc116560 con 0x7fddcc0fe510 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd3270700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc116720 con 0x7fddcc0fdbb0 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd080b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fddcc0fdbb0 0x7fddcc06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd100c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fddcc0f8420 0x7fddcc06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.134+0000 7fddd080b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fddcc0fdbb0 0x7fddcc06b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:52692/0 (socket says 172.21.15.87:52692) 2024-02-23T00:28:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd080b700 1 -- 172.21.15.87:0/831167578 learned_addr learned my addr 172.21.15.87:0/831167578 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd180d700 1 --2- 172.21.15.87:0/831167578 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fddcc0fe510 0x7fddcc06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd100c700 1 -- 172.21.15.87:0/831167578 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fddcc0fdbb0 msgr2=0x7fddcc06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd100c700 1 --2- 172.21.15.87:0/831167578 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fddcc0fdbb0 0x7fddcc06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd100c700 1 -- 172.21.15.87:0/831167578 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fddcc0fe510 msgr2=0x7fddcc06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd100c700 1 --2- 172.21.15.87:0/831167578 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fddcc0fe510 0x7fddcc06bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd100c700 1 -- 172.21.15.87:0/831167578 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc0005040 con 0x7fddcc0f8420 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd080b700 1 --2- 172.21.15.87:0/831167578 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fddcc0fdbb0 0x7fddcc06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd100c700 1 --2- 172.21.15.87:0/831167578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fddcc0f8420 0x7fddcc06b0d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fddbc00e8f0 tx=0x7fddbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddca777700 10 client.?.objecter ms_handle_connect 0x7fddcc0f8420 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddca777700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddca777700 1 -- 172.21.15.87:0/831167578 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc004620 con 0x7fddcc0f8420 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddca777700 1 -- 172.21.15.87:0/831167578 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddbc00f040 con 0x7fddcc0f8420 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc40097c0 con 0x7fddcc0f8420 2024-02-23T00:28:47.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddca777700 1 -- 172.21.15.87:0/831167578 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddbc014830 con 0x7fddcc0f8420 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd3270700 10 client.14637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddcc116e30 con 0x7fddcc0f8420 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.135+0000 7fddd3270700 1 librados: init done 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.136+0000 7fddd3270700 10 client.14637.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.136+0000 7fddd3270700 15 client.14637.objecter _session_command_op_assign -1 1 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.136+0000 7fddd3270700 20 client.14637.objecter _get_session osd=-1 returning homeless 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.136+0000 7fddd3270700 10 client.14637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.136+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fddcc05ee80 con 0x7fddcc0f8420 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.136+0000 7fddca777700 1 -- 172.21.15.87:0/831167578 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fddbc0090d0 con 0x7fddcc0f8420 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 1 --2- 172.21.15.87:0/831167578 >> v2:172.21.15.87:6800/1231597463 conn(0x7fddb4062c90 0x7fddb4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddd080b700 1 --2- 172.21.15.87:0/831167578 >> v2:172.21.15.87:6800/1231597463 conn(0x7fddb4062c90 0x7fddb4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 1 -- 172.21.15.87:0/831167578 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(78..78 src has 28..78) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fddbc07c3e0 con 0x7fddcc0f8420 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 10 client.14637.objecter ms_dispatch 0x7fddcc001550 osd_map(78..78 src has 28..78) v4 2024-02-23T00:28:47.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 3 client.14637.objecter handle_osd_map got epochs [78,78] > 0 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 3 client.14637.objecter handle_osd_map decoding full epoch 78 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 10 client.14637.objecter checking command 1 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 1 --2- 172.21.15.87:0/831167578 >> v2:172.21.15.171:6804/2550013795 conn(0x7fddb406c4e0 0x7fddb406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 20 client.14637.objecter _get_session s=0x7fddbc004620 osd=5 3 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 20 client.14637.objecter put_session s=0x7fddbc004620 osd=5 3 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 15 client.14637.objecter _session_command_op_remove -1 1 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 20 client.14637.objecter _get_session s=0x7fddbc004620 osd=5 3 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 20 client.14637.objecter get_session s=0x7fddbc004620 osd=5 3 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 15 client.14637.objecter _session_command_op_assign 5 1 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 20 client.14637.objecter put_session s=0x7fddbc004620 osd=5 4 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 10 client.14637.objecter _send_command 1 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 1 -- 172.21.15.87:0/831167578 --> v2:172.21.15.171:6804/2550013795 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddb406efc0 con 0x7fddb406c4e0 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 20 client.14637.objecter dump_active .. 0 homeless 2024-02-23T00:28:47.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 1 -- 172.21.15.87:0/831167578 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddbc07ea00 con 0x7fddcc0f8420 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 10 client.14637.objecter ms_handle_connect 0x7fddb4062c90 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddd080b700 1 --2- 172.21.15.87:0/831167578 >> v2:172.21.15.87:6800/1231597463 conn(0x7fddb4062c90 0x7fddb4065150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fddc0009fd0 tx=0x7fddc000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddd180d700 1 --2- 172.21.15.87:0/831167578 >> v2:172.21.15.171:6804/2550013795 conn(0x7fddb406c4e0 0x7fddb406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddca777700 10 client.14637.objecter ms_handle_connect 0x7fddb406c4e0 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.137+0000 7fddd180d700 1 --2- 172.21.15.87:0/831167578 >> v2:172.21.15.171:6804/2550013795 conn(0x7fddb406c4e0 0x7fddb406e980 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.138+0000 7fddca777700 1 -- 172.21.15.87:0/831167578 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fddb406efc0 con 0x7fddb406c4e0 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.138+0000 7fddca777700 10 client.14637.objecter ms_dispatch 0x7fddcc001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.138+0000 7fddca777700 10 client.14637.objecter _finish_command 1 = system:0 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.138+0000 7fddca777700 20 client.14637.objecter put_session s=0x7fddbc004620 osd=5 4 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.138+0000 7fddca777700 15 client.14637.objecter _session_command_op_remove 5 1 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 10 client.14637.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 15 client.14637.objecter _session_command_op_assign -1 2 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 20 client.14637.objecter _get_session s=0x7fddbc004620 osd=5 3 2024-02-23T00:28:47.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 20 client.14637.objecter put_session s=0x7fddbc004620 osd=5 3 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 20 client.14637.objecter _get_session s=0x7fddbc004620 osd=5 3 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 15 client.14637.objecter _session_command_op_remove -1 2 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 20 client.14637.objecter get_session s=0x7fddbc004620 osd=5 3 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 15 client.14637.objecter _session_command_op_assign 5 2 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 20 client.14637.objecter put_session s=0x7fddbc004620 osd=5 4 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 10 client.14637.objecter _send_command 2 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 --> v2:172.21.15.171:6804/2550013795 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fddcc107110 con 0x7fddb406c4e0 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddca777700 1 -- 172.21.15.87:0/831167578 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fddcc107110 con 0x7fddb406c4e0 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddca777700 10 client.14637.objecter ms_dispatch 0x7fddcc001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddca777700 10 client.14637.objecter _finish_command 2 = system:0 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddca777700 20 client.14637.objecter put_session s=0x7fddbc004620 osd=5 4 2024-02-23T00:28:47.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.152+0000 7fddca777700 15 client.14637.objecter _session_command_op_remove 5 2 2024-02-23T00:28:47.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 10 librados: watch_flush enter 2024-02-23T00:28:47.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 10 librados: watch_flush exit 2024-02-23T00:28:47.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 10 client.14637.objecter close_session for osd.5 2024-02-23T00:28:47.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 >> v2:172.21.15.171:6804/2550013795 conn(0x7fddb406c4e0 msgr2=0x7fddb406e980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:47.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 --2- 172.21.15.87:0/831167578 >> v2:172.21.15.171:6804/2550013795 conn(0x7fddb406c4e0 0x7fddb406e980 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:47.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 20 client.14637.objecter put_session s=0x7fddbc004620 osd=5 1 2024-02-23T00:28:47.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 20 client.14637.objecter shutdown clearing up homeless session... 2024-02-23T00:28:47.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 10 client.14637.objecter successfully canceled tick 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 >> v2:172.21.15.87:6800/1231597463 conn(0x7fddb4062c90 msgr2=0x7fddb4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 --2- 172.21.15.87:0/831167578 >> v2:172.21.15.87:6800/1231597463 conn(0x7fddb4062c90 0x7fddb4065150 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fddc0009fd0 tx=0x7fddc000b040).stop 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fddcc0f8420 msgr2=0x7fddcc06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 --2- 172.21.15.87:0/831167578 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fddcc0f8420 0x7fddcc06b0d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fddbc00e8f0 tx=0x7fddbc00b040).stop 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd180d700 1 -- 172.21.15.87:0/831167578 reap_dead start 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 shutdown_connections 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 >> 172.21.15.87:0/831167578 conn(0x7fddcc0f3800 msgr2=0x7fddcc0f5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 shutdown_connections 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 -- 172.21.15.87:0/831167578 wait complete. 2024-02-23T00:28:47.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:47.153+0000 7fddd3270700 1 librados: shutdown 2024-02-23T00:28:47.516 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:46.112788+0000 osd.5 (osd.5) 17 : cluster [DBG] 4.8 deep-scrub starts 2024-02-23T00:28:47.516 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:46.118912+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.8 deep-scrub ok 2024-02-23T00:28:47.516 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:46.216685+0000 osd.0 (osd.0) 19 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:28:47.516 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:46.217503+0000 osd.0 (osd.0) 20 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:28:47.517 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:46.393442+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.1d deep-scrub starts 2024-02-23T00:28:47.517 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:46.396466+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.1d deep-scrub ok 2024-02-23T00:28:47.517 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:46.618414+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.f scrub starts 2024-02-23T00:28:47.517 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:46.621971+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2024-02-23T00:28:47.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:46.112788+0000 osd.5 (osd.5) 17 : cluster [DBG] 4.8 deep-scrub starts 2024-02-23T00:28:47.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:46.118912+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.8 deep-scrub ok 2024-02-23T00:28:47.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:46.216685+0000 osd.0 (osd.0) 19 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:28:47.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:46.217503+0000 osd.0 (osd.0) 20 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:28:47.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:46.393442+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.1d deep-scrub starts 2024-02-23T00:28:47.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:46.396466+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.1d deep-scrub ok 2024-02-23T00:28:47.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:46.618414+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.f scrub starts 2024-02-23T00:28:47.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:46.621971+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2024-02-23T00:28:47.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:46.112788+0000 osd.5 (osd.5) 17 : cluster [DBG] 4.8 deep-scrub starts 2024-02-23T00:28:47.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:46.118912+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.8 deep-scrub ok 2024-02-23T00:28:47.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:46.216685+0000 osd.0 (osd.0) 19 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:28:47.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:46.217503+0000 osd.0 (osd.0) 20 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:28:47.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:46.393442+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.1d deep-scrub starts 2024-02-23T00:28:47.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:46.396466+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.1d deep-scrub ok 2024-02-23T00:28:47.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:46.618414+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.f scrub starts 2024-02-23T00:28:47.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:46.621971+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.f scrub ok 2024-02-23T00:28:48.001 INFO:teuthology.orchestra.run.smithi087.stdout:163208757279 2024-02-23T00:28:48.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-23T00:28:48.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:47.070675+0000 osd.3 (osd.3) 35 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:28:48.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:47.073548+0000 osd.3 (osd.3) 36 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:28:48.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:47.089491+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.d scrub starts 2024-02-23T00:28:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:47.090879+0000 osd.5 (osd.5) 20 : cluster [DBG] 6.d scrub ok 2024-02-23T00:28:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:47.471720+0000 mgr.y (mgr.24437) 64 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-02-23T00:28:48.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:47.070675+0000 osd.3 (osd.3) 35 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:28:48.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:47.073548+0000 osd.3 (osd.3) 36 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:28:48.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:47.089491+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.d scrub starts 2024-02-23T00:28:48.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:47.090879+0000 osd.5 (osd.5) 20 : cluster [DBG] 6.d scrub ok 2024-02-23T00:28:48.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:47.471720+0000 mgr.y (mgr.24437) 64 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-02-23T00:28:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:47.070675+0000 osd.3 (osd.3) 35 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:28:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:47.073548+0000 osd.3 (osd.3) 36 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:28:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:47.089491+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.d scrub starts 2024-02-23T00:28:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:47.090879+0000 osd.5 (osd.5) 20 : cluster [DBG] 6.d scrub ok 2024-02-23T00:28:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:47.471720+0000 mgr.y (mgr.24437) 64 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-02-23T00:28:48.785 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:49.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:47.676857+0000 osd.2 (osd.2) 19 : cluster [DBG] 6.f scrub starts 2024-02-23T00:28:49.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:47.677726+0000 osd.2 (osd.2) 20 : cluster [DBG] 6.f scrub ok 2024-02-23T00:28:49.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:48.331724+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.1c deep-scrub starts 2024-02-23T00:28:49.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:48.333632+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.1c deep-scrub ok 2024-02-23T00:28:49.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.011800+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.12 deep-scrub starts 2024-02-23T00:28:49.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.012689+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.12 deep-scrub ok 2024-02-23T00:28:49.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:47.676857+0000 osd.2 (osd.2) 19 : cluster [DBG] 6.f scrub starts 2024-02-23T00:28:49.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:47.677726+0000 osd.2 (osd.2) 20 : cluster [DBG] 6.f scrub ok 2024-02-23T00:28:49.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:48.331724+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.1c deep-scrub starts 2024-02-23T00:28:49.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:48.333632+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.1c deep-scrub ok 2024-02-23T00:28:49.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.011800+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.12 deep-scrub starts 2024-02-23T00:28:49.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.012689+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.12 deep-scrub ok 2024-02-23T00:28:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:47.676857+0000 osd.2 (osd.2) 19 : cluster [DBG] 6.f scrub starts 2024-02-23T00:28:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:47.677726+0000 osd.2 (osd.2) 20 : cluster [DBG] 6.f scrub ok 2024-02-23T00:28:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:48.331724+0000 osd.7 (osd.7) 23 : cluster [DBG] 6.1c deep-scrub starts 2024-02-23T00:28:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:48.333632+0000 osd.7 (osd.7) 24 : cluster [DBG] 6.1c deep-scrub ok 2024-02-23T00:28:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.011800+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.12 deep-scrub starts 2024-02-23T00:28:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.012689+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.12 deep-scrub ok 2024-02-23T00:28:50.449 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.184858+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:28:50.449 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.187782+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:28:50.449 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.328227+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:28:50.449 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.331133+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.472186+0000 mgr.y (mgr.24437) 65 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35 B/s, 1 objects/s recovering 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:49.525777+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.662937+0000 osd.1 (osd.1) 23 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.663805+0000 osd.1 (osd.1) 24 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:49.999025+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:50.000611+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.184858+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.187782+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.328227+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.331133+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:28:50.450 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.472186+0000 mgr.y (mgr.24437) 65 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35 B/s, 1 objects/s recovering 2024-02-23T00:28:50.451 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:49.525777+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:50.451 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.662937+0000 osd.1 (osd.1) 23 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:28:50.451 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.663805+0000 osd.1 (osd.1) 24 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:28:50.451 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:49.999025+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:28:50.451 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:50.000611+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:28:50.534 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.184858+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:28:50.535 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.187782+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:28:50.535 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.328227+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:28:50.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.331133+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:28:50.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.472186+0000 mgr.y (mgr.24437) 65 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35 B/s, 1 objects/s recovering 2024-02-23T00:28:50.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:49.525777+0000 mon.a (mon.0) 714 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:50.537 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.662937+0000 osd.1 (osd.1) 23 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:28:50.537 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.663805+0000 osd.1 (osd.1) 24 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:28:50.537 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:49.999025+0000 osd.5 (osd.5) 23 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:28:50.537 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:50.000611+0000 osd.5 (osd.5) 24 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:28:50.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:28:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:28:51.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.982+0000 7f11c965a700 1 -- 172.21.15.87:0/1621131512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c40fdc60 msgr2=0x7f11c40fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:51.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.982+0000 7f11c965a700 1 --2- 172.21.15.87:0/1621131512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c40fdc60 0x7f11c40fe040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f11ac004870 tx=0x7f11ac009cb0).stop 2024-02-23T00:28:51.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.982+0000 7f11c965a700 1 -- 172.21.15.87:0/1621131512 shutdown_connections 2024-02-23T00:28:51.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.982+0000 7f11c965a700 1 --2- 172.21.15.87:0/1621131512 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c40f6400 0x7f11c4104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:51.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.982+0000 7f11c965a700 1 --2- 172.21.15.87:0/1621131512 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c40fdc60 0x7f11c40fe040 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:51.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.982+0000 7f11c965a700 1 --2- 172.21.15.87:0/1621131512 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c40fe5c0 0x7f11c40f5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:51.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.982+0000 7f11c965a700 1 -- 172.21.15.87:0/1621131512 >> 172.21.15.87:0/1621131512 conn(0x7f11c406ca90 msgr2=0x7f11c40f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.982+0000 7f11c965a700 1 -- 172.21.15.87:0/1621131512 shutdown_connections 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.982+0000 7f11c965a700 1 -- 172.21.15.87:0/1621131512 wait complete. 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 librados: starting msgr at 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 librados: starting objecter 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 Processor -- start 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 -- start start 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 librados: setting wanted keys 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 librados: calling monclient init 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c40f6400 0x7f11c410e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c40fdc60 0x7f11c410e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c40fe5c0 0x7f11c41081f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c41086e0 con 0x7f11c40fe5c0 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4108850 con 0x7f11c40f6400 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c965a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4108a10 con 0x7f11c40fdc60 2024-02-23T00:28:51.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c40fe5c0 0x7f11c41081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c27fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c40fdc60 0x7f11c410e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c40fe5c0 0x7f11c41081f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46516/0 (socket says 172.21.15.87:46516) 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c37fe700 1 -- 172.21.15.87:0/2004567403 learned_addr learned my addr 172.21.15.87:0/2004567403 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.983+0000 7f11c2ffd700 1 --2- 172.21.15.87:0/2004567403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c40f6400 0x7f11c410e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c37fe700 1 -- 172.21.15.87:0/2004567403 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c40fdc60 msgr2=0x7f11c410e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c37fe700 1 --2- 172.21.15.87:0/2004567403 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c40fdc60 0x7f11c410e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c37fe700 1 -- 172.21.15.87:0/2004567403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c40f6400 msgr2=0x7f11c410e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c37fe700 1 --2- 172.21.15.87:0/2004567403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c40f6400 0x7f11c410e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c37fe700 1 -- 172.21.15.87:0/2004567403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc0097c0 con 0x7f11c40fe5c0 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c2ffd700 1 --2- 172.21.15.87:0/2004567403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c40f6400 0x7f11c410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c37fe700 1 --2- 172.21.15.87:0/2004567403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c40fe5c0 0x7f11c41081f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f11bc001df0 tx=0x7f11bc004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11bbfff700 10 client.?.objecter ms_handle_connect 0x7f11c40fe5c0 2024-02-23T00:28:51.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11bbfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11bbfff700 1 -- 172.21.15.87:0/2004567403 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc009a10 con 0x7f11c40fe5c0 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac005040 con 0x7f11c40fe5c0 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11bbfff700 1 -- 172.21.15.87:0/2004567403 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11bc0095a0 con 0x7f11c40fe5c0 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11bbfff700 1 -- 172.21.15.87:0/2004567403 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc007da0 con 0x7f11c40fe5c0 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c965a700 10 client.14643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c4109000 con 0x7f11c40fe5c0 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.984+0000 7f11c965a700 1 librados: init done 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.985+0000 7f11c965a700 10 client.14643.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.985+0000 7f11c965a700 15 client.14643.objecter _session_command_op_assign -1 1 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.985+0000 7f11c965a700 20 client.14643.objecter _get_session osd=-1 returning homeless 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.985+0000 7f11c965a700 10 client.14643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.985+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f11c40606d0 con 0x7f11c40fe5c0 2024-02-23T00:28:51.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.985+0000 7f11bbfff700 1 -- 172.21.15.87:0/2004567403 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f11bc0090d0 con 0x7f11c40fe5c0 2024-02-23T00:28:51.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 1 --2- 172.21.15.87:0/2004567403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f11b0062c90 0x7f11b0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:51.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 1 -- 172.21.15.87:0/2004567403 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(78..78 src has 28..78) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f11bc07d950 con 0x7f11c40fe5c0 2024-02-23T00:28:51.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 10 client.14643.objecter ms_dispatch 0x7f11c4001550 osd_map(78..78 src has 28..78) v4 2024-02-23T00:28:51.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 3 client.14643.objecter handle_osd_map got epochs [78,78] > 0 2024-02-23T00:28:51.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 3 client.14643.objecter handle_osd_map decoding full epoch 78 2024-02-23T00:28:51.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11c2ffd700 1 --2- 172.21.15.87:0/2004567403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f11b0062c90 0x7f11b0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:51.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 10 client.14643.objecter checking command 1 2024-02-23T00:28:51.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 1 --2- 172.21.15.87:0/2004567403 >> v2:172.21.15.171:6808/2509408498 conn(0x7f11b006c650 0x7f11b006eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:51.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 20 client.14643.objecter _get_session s=0x7f11b006bfa0 osd=6 3 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 20 client.14643.objecter put_session s=0x7f11b006bfa0 osd=6 3 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 15 client.14643.objecter _session_command_op_remove -1 1 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 20 client.14643.objecter _get_session s=0x7f11b006bfa0 osd=6 3 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 20 client.14643.objecter get_session s=0x7f11b006bfa0 osd=6 3 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 15 client.14643.objecter _session_command_op_assign 6 1 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 20 client.14643.objecter put_session s=0x7f11b006bfa0 osd=6 4 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 10 client.14643.objecter _send_command 1 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 1 -- 172.21.15.87:0/2004567403 --> v2:172.21.15.171:6808/2509408498 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11b006f130 con 0x7f11b006c650 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 20 client.14643.objecter dump_active .. 0 homeless 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 1 -- 172.21.15.87:0/2004567403 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11bc07dc40 con 0x7f11c40fe5c0 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 10 client.14643.objecter ms_handle_connect 0x7f11b0062c90 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11c2ffd700 1 --2- 172.21.15.87:0/2004567403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f11b0062c90 0x7f11b0065150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f11ac0043d0 tx=0x7f11ac00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11c27fc700 1 --2- 172.21.15.87:0/2004567403 >> v2:172.21.15.171:6808/2509408498 conn(0x7f11b006c650 0x7f11b006eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11bbfff700 10 client.14643.objecter ms_handle_connect 0x7f11b006c650 2024-02-23T00:28:51.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.986+0000 7f11c27fc700 1 --2- 172.21.15.87:0/2004567403 >> v2:172.21.15.171:6808/2509408498 conn(0x7f11b006c650 0x7f11b006eaf0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:51.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.987+0000 7f11bbfff700 1 -- 172.21.15.87:0/2004567403 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f11b006f130 con 0x7f11b006c650 2024-02-23T00:28:51.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.987+0000 7f11bbfff700 10 client.14643.objecter ms_dispatch 0x7f11c4001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.987+0000 7f11bbfff700 10 client.14643.objecter _finish_command 1 = system:0 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.987+0000 7f11bbfff700 20 client.14643.objecter put_session s=0x7f11b006bfa0 osd=6 4 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:50.987+0000 7f11bbfff700 15 client.14643.objecter _session_command_op_remove 6 1 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 10 client.14643.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 15 client.14643.objecter _session_command_op_assign -1 2 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 20 client.14643.objecter _get_session s=0x7f11b006bfa0 osd=6 3 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 20 client.14643.objecter put_session s=0x7f11b006bfa0 osd=6 3 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 20 client.14643.objecter _get_session s=0x7f11b006bfa0 osd=6 3 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 15 client.14643.objecter _session_command_op_remove -1 2 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 20 client.14643.objecter get_session s=0x7f11b006bfa0 osd=6 3 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 15 client.14643.objecter _session_command_op_assign 6 2 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 20 client.14643.objecter put_session s=0x7f11b006bfa0 osd=6 4 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 10 client.14643.objecter _send_command 2 2024-02-23T00:28:51.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.001+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 --> v2:172.21.15.171:6808/2509408498 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f11c405ee80 con 0x7f11b006c650 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11bbfff700 1 -- 172.21.15.87:0/2004567403 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f11c405ee80 con 0x7f11b006c650 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11bbfff700 10 client.14643.objecter ms_dispatch 0x7f11c4001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11bbfff700 10 client.14643.objecter _finish_command 2 = system:0 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11bbfff700 20 client.14643.objecter put_session s=0x7f11b006bfa0 osd=6 4 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11bbfff700 15 client.14643.objecter _session_command_op_remove 6 2 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 10 librados: watch_flush enter 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 10 librados: watch_flush exit 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 10 client.14643.objecter close_session for osd.6 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 >> v2:172.21.15.171:6808/2509408498 conn(0x7f11b006c650 msgr2=0x7f11b006eaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 1 --2- 172.21.15.87:0/2004567403 >> v2:172.21.15.171:6808/2509408498 conn(0x7f11b006c650 0x7f11b006eaf0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 20 client.14643.objecter put_session s=0x7f11b006bfa0 osd=6 1 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 20 client.14643.objecter shutdown clearing up homeless session... 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 10 client.14643.objecter successfully canceled tick 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f11b0062c90 msgr2=0x7f11b0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:51.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 1 --2- 172.21.15.87:0/2004567403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f11b0062c90 0x7f11b0065150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f11ac0043d0 tx=0x7f11ac00b040).stop 2024-02-23T00:28:51.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c40fe5c0 msgr2=0x7f11c41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:51.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.002+0000 7f11c965a700 1 --2- 172.21.15.87:0/2004567403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c40fe5c0 0x7f11c41081f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f11bc001df0 tx=0x7f11bc004c50).stop 2024-02-23T00:28:51.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.003+0000 7f11c37fe700 1 -- 172.21.15.87:0/2004567403 reap_dead start 2024-02-23T00:28:51.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.003+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 shutdown_connections 2024-02-23T00:28:51.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.003+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 >> 172.21.15.87:0/2004567403 conn(0x7f11c406ca90 msgr2=0x7f11c406dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:51.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.003+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 shutdown_connections 2024-02-23T00:28:51.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.003+0000 7f11c965a700 1 -- 172.21.15.87:0/2004567403 wait complete. 2024-02-23T00:28:51.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:51.003+0000 7f11c965a700 1 librados: shutdown 2024-02-23T00:28:51.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:50.173314+0000 osd.0 (osd.0) 23 : cluster [DBG] 6.18 deep-scrub starts 2024-02-23T00:28:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:50.176204+0000 osd.0 (osd.0) 24 : cluster [DBG] 6.18 deep-scrub ok 2024-02-23T00:28:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:50.343202+0000 osd.7 (osd.7) 27 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:28:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:50.344315+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:28:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:50.533172+0000 mgr.y (mgr.24437) 66 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:50.173314+0000 osd.0 (osd.0) 23 : cluster [DBG] 6.18 deep-scrub starts 2024-02-23T00:28:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:50.176204+0000 osd.0 (osd.0) 24 : cluster [DBG] 6.18 deep-scrub ok 2024-02-23T00:28:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:50.343202+0000 osd.7 (osd.7) 27 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:28:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:50.344315+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:28:51.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:50.533172+0000 mgr.y (mgr.24437) 66 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:50.173314+0000 osd.0 (osd.0) 23 : cluster [DBG] 6.18 deep-scrub starts 2024-02-23T00:28:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:50.176204+0000 osd.0 (osd.0) 24 : cluster [DBG] 6.18 deep-scrub ok 2024-02-23T00:28:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:50.343202+0000 osd.7 (osd.7) 27 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:28:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:50.344315+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:28:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:50.533172+0000 mgr.y (mgr.24437) 66 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:28:52.203 INFO:teuthology.orchestra.run.smithi087.stdout:188978561051 2024-02-23T00:28:52.204 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-23T00:28:52.515 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:28:52] "GET /metrics HTTP/1.1" 200 172168 "" "Prometheus/2.33.4" 2024-02-23T00:28:52.515 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:50.670907+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:28:52.515 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:50.671984+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:28:52.515 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:51.473003+0000 mgr.y (mgr.24437) 67 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 996 B/s rd, 0 op/s; 32 B/s, 1 objects/s recovering 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:51.661353+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.4 deep-scrub starts 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:51.664363+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.4 deep-scrub ok 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:51.681308+0000 osd.4 (osd.4) 25 : cluster [DBG] 2.f scrub starts 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:51.684784+0000 osd.4 (osd.4) 26 : cluster [DBG] 2.f scrub ok 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:50.670907+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:50.671984+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:51.473003+0000 mgr.y (mgr.24437) 67 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 996 B/s rd, 0 op/s; 32 B/s, 1 objects/s recovering 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:51.661353+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.4 deep-scrub starts 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:51.664363+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.4 deep-scrub ok 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:51.681308+0000 osd.4 (osd.4) 25 : cluster [DBG] 2.f scrub starts 2024-02-23T00:28:52.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:51.684784+0000 osd.4 (osd.4) 26 : cluster [DBG] 2.f scrub ok 2024-02-23T00:28:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:50.670907+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:28:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:50.671984+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:28:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:51.473003+0000 mgr.y (mgr.24437) 67 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 996 B/s rd, 0 op/s; 32 B/s, 1 objects/s recovering 2024-02-23T00:28:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:51.661353+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.4 deep-scrub starts 2024-02-23T00:28:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:51.664363+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.4 deep-scrub ok 2024-02-23T00:28:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:51.681308+0000 osd.4 (osd.4) 25 : cluster [DBG] 2.f scrub starts 2024-02-23T00:28:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:51.684784+0000 osd.4 (osd.4) 26 : cluster [DBG] 2.f scrub ok 2024-02-23T00:28:52.879 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:51.634082+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:51.634895+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:52.089781+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:52.093174+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:52.651137+0000 osd.1 (osd.1) 27 : cluster [DBG] 3.0 deep-scrub starts 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:52.653818+0000 osd.1 (osd.1) 28 : cluster [DBG] 3.0 deep-scrub ok 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:52.861088+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:52.947688+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:52.948597+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:28:53.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:51.634082+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:28:53.471 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:51.634895+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:28:53.471 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:52.089781+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:28:53.471 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:52.093174+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:28:53.471 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:52.651137+0000 osd.1 (osd.1) 27 : cluster [DBG] 3.0 deep-scrub starts 2024-02-23T00:28:53.471 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:52.653818+0000 osd.1 (osd.1) 28 : cluster [DBG] 3.0 deep-scrub ok 2024-02-23T00:28:53.471 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:52.861088+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:53.471 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:52.947688+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:28:53.471 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:52.948597+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:28:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:51.634082+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:28:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:51.634895+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:28:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:52.089781+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:28:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:52.093174+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:28:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:52.651137+0000 osd.1 (osd.1) 27 : cluster [DBG] 3.0 deep-scrub starts 2024-02-23T00:28:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:52.653818+0000 osd.1 (osd.1) 28 : cluster [DBG] 3.0 deep-scrub ok 2024-02-23T00:28:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:52.861088+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:52.947688+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:28:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:52.948597+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:28:54.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:28:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:28:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:28:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:53.182549+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-02-23T00:28:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:53.186012+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-02-23T00:28:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:53.283288+0000 mon.a (mon.0) 716 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:28:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:53.283988+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:28:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:53.288500+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:53.292472+0000 mon.a (mon.0) 719 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:28:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:53.473389+0000 mgr.y (mgr.24437) 68 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-02-23T00:28:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:53.662199+0000 osd.1 (osd.1) 29 : cluster [DBG] 5.12 deep-scrub starts 2024-02-23T00:28:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:53.665089+0000 osd.1 (osd.1) 30 : cluster [DBG] 5.12 deep-scrub ok 2024-02-23T00:28:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:53.182549+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-02-23T00:28:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:53.186012+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-02-23T00:28:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:53.283288+0000 mon.a (mon.0) 716 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:28:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:53.283988+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:28:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:53.288500+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:53.292472+0000 mon.a (mon.0) 719 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:28:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:53.473389+0000 mgr.y (mgr.24437) 68 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-02-23T00:28:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:53.662199+0000 osd.1 (osd.1) 29 : cluster [DBG] 5.12 deep-scrub starts 2024-02-23T00:28:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:53.665089+0000 osd.1 (osd.1) 30 : cluster [DBG] 5.12 deep-scrub ok 2024-02-23T00:28:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:53.182549+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-02-23T00:28:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:53.186012+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-02-23T00:28:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:53.283288+0000 mon.a (mon.0) 716 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:28:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:53.283988+0000 mon.a (mon.0) 717 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:28:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:53.288500+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:28:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:53.292472+0000 mon.a (mon.0) 719 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:28:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:53.473389+0000 mgr.y (mgr.24437) 68 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-02-23T00:28:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:53.662199+0000 osd.1 (osd.1) 29 : cluster [DBG] 5.12 deep-scrub starts 2024-02-23T00:28:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:53.665089+0000 osd.1 (osd.1) 30 : cluster [DBG] 5.12 deep-scrub ok 2024-02-23T00:28:55.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.185+0000 7f50c7bab700 1 -- 172.21.15.87:0/1658921795 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50c00fe510 msgr2=0x7f50c00f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:55.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.185+0000 7f50c7bab700 1 --2- 172.21.15.87:0/1658921795 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50c00fe510 0x7f50c00f7d40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f50bc004870 tx=0x7f50bc009cb0).stop 2024-02-23T00:28:55.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.185+0000 7f50c7bab700 1 -- 172.21.15.87:0/1658921795 shutdown_connections 2024-02-23T00:28:55.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.185+0000 7f50c7bab700 1 --2- 172.21.15.87:0/1658921795 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50c00f8420 0x7f50c0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:55.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.185+0000 7f50c7bab700 1 --2- 172.21.15.87:0/1658921795 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50c00fdbb0 0x7f50c00fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:55.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.185+0000 7f50c7bab700 1 --2- 172.21.15.87:0/1658921795 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50c00fe510 0x7f50c00f7d40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.185+0000 7f50c7bab700 1 -- 172.21.15.87:0/1658921795 >> 172.21.15.87:0/1658921795 conn(0x7f50c00f3800 msgr2=0x7f50c00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.185+0000 7f50c7bab700 1 -- 172.21.15.87:0/1658921795 shutdown_connections 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.185+0000 7f50c7bab700 1 -- 172.21.15.87:0/1658921795 wait complete. 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 librados: starting msgr at 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 librados: starting objecter 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 Processor -- start 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 -- start start 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 librados: setting wanted keys 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 librados: calling monclient init 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50c00f8420 0x7f50c006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50c00fdbb0 0x7f50c006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50c00fe510 0x7f50c006bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c0116430 con 0x7f50c00fe510 2024-02-23T00:28:55.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c01165a0 con 0x7f50c00fdbb0 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.186+0000 7f50c7bab700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c0116760 con 0x7f50c00f8420 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c6148700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50c00fe510 0x7f50c006bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c5947700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50c00f8420 0x7f50c006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c6148700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50c00fe510 0x7f50c006bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36268/0 (socket says 172.21.15.87:36268) 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c6148700 1 -- 172.21.15.87:0/3975603505 learned_addr learned my addr 172.21.15.87:0/3975603505 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c5146700 1 --2- 172.21.15.87:0/3975603505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50c00fdbb0 0x7f50c006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c6148700 1 -- 172.21.15.87:0/3975603505 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50c00f8420 msgr2=0x7f50c006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c6148700 1 --2- 172.21.15.87:0/3975603505 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50c00f8420 0x7f50c006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c6148700 1 -- 172.21.15.87:0/3975603505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50c00fdbb0 msgr2=0x7f50c006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c6148700 1 --2- 172.21.15.87:0/3975603505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50c00fdbb0 0x7f50c006b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c6148700 1 -- 172.21.15.87:0/3975603505 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b80097c0 con 0x7f50c00fe510 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c5146700 1 --2- 172.21.15.87:0/3975603505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50c00fdbb0 0x7f50c006b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c5947700 1 --2- 172.21.15.87:0/3975603505 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50c00f8420 0x7f50c006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c6148700 1 --2- 172.21.15.87:0/3975603505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50c00fe510 0x7f50c006bab0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f50b8001df0 tx=0x7f50b8004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:55.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50b6ffd700 10 client.?.objecter ms_handle_connect 0x7f50c00fe510 2024-02-23T00:28:55.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50b6ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:28:55.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50b6ffd700 1 -- 172.21.15.87:0/3975603505 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8009a10 con 0x7f50c00fe510 2024-02-23T00:28:55.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50b6ffd700 1 -- 172.21.15.87:0/3975603505 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50b80095a0 con 0x7f50c00fe510 2024-02-23T00:28:55.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50b6ffd700 1 -- 172.21.15.87:0/3975603505 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b8007da0 con 0x7f50c00fe510 2024-02-23T00:28:55.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc005040 con 0x7f50c00fe510 2024-02-23T00:28:55.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c7bab700 10 client.14649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c0116dd0 con 0x7f50c00fe510 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.187+0000 7f50c7bab700 1 librados: init done 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.188+0000 7f50c7bab700 10 client.14649.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.188+0000 7f50c7bab700 15 client.14649.objecter _session_command_op_assign -1 1 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.188+0000 7f50c7bab700 20 client.14649.objecter _get_session osd=-1 returning homeless 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.188+0000 7f50c7bab700 10 client.14649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.188+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50c005ee80 con 0x7f50c00fe510 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 1 -- 172.21.15.87:0/3975603505 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f50b80090d0 con 0x7f50c00fe510 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 1 --2- 172.21.15.87:0/3975603505 >> v2:172.21.15.87:6800/1231597463 conn(0x7f50ac062c90 0x7f50ac065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 1 -- 172.21.15.87:0/3975603505 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(78..78 src has 28..78) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f50b807d420 con 0x7f50c00fe510 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 10 client.14649.objecter ms_dispatch 0x7f50c0001550 osd_map(78..78 src has 28..78) v4 2024-02-23T00:28:55.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 3 client.14649.objecter handle_osd_map got epochs [78,78] > 0 2024-02-23T00:28:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 3 client.14649.objecter handle_osd_map decoding full epoch 78 2024-02-23T00:28:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50c5947700 1 --2- 172.21.15.87:0/3975603505 >> v2:172.21.15.87:6800/1231597463 conn(0x7f50ac062c90 0x7f50ac065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 10 client.14649.objecter checking command 1 2024-02-23T00:28:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 1 --2- 172.21.15.87:0/3975603505 >> v2:172.21.15.171:6812/373787000 conn(0x7f50ac06c650 0x7f50ac06eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:28:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 20 client.14649.objecter _get_session s=0x7f50ac06bfa0 osd=7 3 2024-02-23T00:28:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 20 client.14649.objecter put_session s=0x7f50ac06bfa0 osd=7 3 2024-02-23T00:28:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 15 client.14649.objecter _session_command_op_remove -1 1 2024-02-23T00:28:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 20 client.14649.objecter _get_session s=0x7f50ac06bfa0 osd=7 3 2024-02-23T00:28:55.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 20 client.14649.objecter get_session s=0x7f50ac06bfa0 osd=7 3 2024-02-23T00:28:55.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 15 client.14649.objecter _session_command_op_assign 7 1 2024-02-23T00:28:55.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 20 client.14649.objecter put_session s=0x7f50ac06bfa0 osd=7 4 2024-02-23T00:28:55.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 10 client.14649.objecter _send_command 1 2024-02-23T00:28:55.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 1 -- 172.21.15.87:0/3975603505 --> v2:172.21.15.171:6812/373787000 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50ac06f130 con 0x7f50ac06c650 2024-02-23T00:28:55.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 20 client.14649.objecter dump_active .. 0 homeless 2024-02-23T00:28:55.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 1 -- 172.21.15.87:0/3975603505 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50b807fa40 con 0x7f50c00fe510 2024-02-23T00:28:55.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50b6ffd700 10 client.14649.objecter ms_handle_connect 0x7f50ac062c90 2024-02-23T00:28:55.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.189+0000 7f50c5947700 1 --2- 172.21.15.87:0/3975603505 >> v2:172.21.15.87:6800/1231597463 conn(0x7f50ac062c90 0x7f50ac065150 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f50b000f2f0 tx=0x7f50b0009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:55.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.190+0000 7f50c5146700 1 --2- 172.21.15.87:0/3975603505 >> v2:172.21.15.171:6812/373787000 conn(0x7f50ac06c650 0x7f50ac06eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:28:55.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.190+0000 7f50b6ffd700 10 client.14649.objecter ms_handle_connect 0x7f50ac06c650 2024-02-23T00:28:55.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.190+0000 7f50c5146700 1 --2- 172.21.15.87:0/3975603505 >> v2:172.21.15.171:6812/373787000 conn(0x7f50ac06c650 0x7f50ac06eaf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:28:55.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.191+0000 7f50b6ffd700 1 -- 172.21.15.87:0/3975603505 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f50ac06f130 con 0x7f50ac06c650 2024-02-23T00:28:55.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.191+0000 7f50b6ffd700 10 client.14649.objecter ms_dispatch 0x7f50c0001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:28:55.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.191+0000 7f50b6ffd700 10 client.14649.objecter _finish_command 1 = system:0 2024-02-23T00:28:55.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.191+0000 7f50b6ffd700 20 client.14649.objecter put_session s=0x7f50ac06bfa0 osd=7 4 2024-02-23T00:28:55.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.191+0000 7f50b6ffd700 15 client.14649.objecter _session_command_op_remove 7 1 2024-02-23T00:28:55.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 10 client.14649.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:28:55.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 15 client.14649.objecter _session_command_op_assign -1 2 2024-02-23T00:28:55.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 20 client.14649.objecter _get_session s=0x7f50ac06bfa0 osd=7 3 2024-02-23T00:28:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 20 client.14649.objecter put_session s=0x7f50ac06bfa0 osd=7 3 2024-02-23T00:28:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 20 client.14649.objecter _get_session s=0x7f50ac06bfa0 osd=7 3 2024-02-23T00:28:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 15 client.14649.objecter _session_command_op_remove -1 2 2024-02-23T00:28:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 20 client.14649.objecter get_session s=0x7f50ac06bfa0 osd=7 3 2024-02-23T00:28:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 15 client.14649.objecter _session_command_op_assign 7 2 2024-02-23T00:28:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 20 client.14649.objecter put_session s=0x7f50ac06bfa0 osd=7 4 2024-02-23T00:28:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 10 client.14649.objecter _send_command 2 2024-02-23T00:28:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.205+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 --> v2:172.21.15.171:6812/373787000 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50c01070a0 con 0x7f50ac06c650 2024-02-23T00:28:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50b6ffd700 1 -- 172.21.15.87:0/3975603505 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f50c01070a0 con 0x7f50ac06c650 2024-02-23T00:28:55.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50b6ffd700 10 client.14649.objecter ms_dispatch 0x7f50c0001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:28:55.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50b6ffd700 10 client.14649.objecter _finish_command 2 = system:0 2024-02-23T00:28:55.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50b6ffd700 20 client.14649.objecter put_session s=0x7f50ac06bfa0 osd=7 4 2024-02-23T00:28:55.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50b6ffd700 15 client.14649.objecter _session_command_op_remove 7 2 2024-02-23T00:28:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50c7bab700 10 librados: watch_flush enter 2024-02-23T00:28:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50c7bab700 10 librados: watch_flush exit 2024-02-23T00:28:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50c7bab700 10 client.14649.objecter close_session for osd.7 2024-02-23T00:28:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 >> v2:172.21.15.171:6812/373787000 conn(0x7f50ac06c650 msgr2=0x7f50ac06eaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50c7bab700 1 --2- 172.21.15.87:0/3975603505 >> v2:172.21.15.171:6812/373787000 conn(0x7f50ac06c650 0x7f50ac06eaf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:28:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50c7bab700 20 client.14649.objecter put_session s=0x7f50ac06bfa0 osd=7 1 2024-02-23T00:28:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50c7bab700 20 client.14649.objecter shutdown clearing up homeless session... 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.206+0000 7f50c7bab700 10 client.14649.objecter successfully canceled tick 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 >> v2:172.21.15.87:6800/1231597463 conn(0x7f50ac062c90 msgr2=0x7f50ac065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c7bab700 1 --2- 172.21.15.87:0/3975603505 >> v2:172.21.15.87:6800/1231597463 conn(0x7f50ac062c90 0x7f50ac065150 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f50b000f2f0 tx=0x7f50b0009040).stop 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50c00fe510 msgr2=0x7f50c006bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c7bab700 1 --2- 172.21.15.87:0/3975603505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50c00fe510 0x7f50c006bab0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f50b8001df0 tx=0x7f50b8004c50).stop 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c6148700 1 -- 172.21.15.87:0/3975603505 reap_dead start 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 shutdown_connections 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 >> 172.21.15.87:0/3975603505 conn(0x7f50c00f3800 msgr2=0x7f50c00f5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 shutdown_connections 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c7bab700 1 -- 172.21.15.87:0/3975603505 wait complete. 2024-02-23T00:28:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:55.207+0000 7f50c7bab700 1 librados: shutdown 2024-02-23T00:28:55.332 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:28:55.013Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002163871s 2024-02-23T00:28:55.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:53.628651+0000 osd.2 (osd.2) 25 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:28:55.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:53.629565+0000 osd.2 (osd.2) 26 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:28:55.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:54.653805+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.1b deep-scrub starts 2024-02-23T00:28:55.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:54.657169+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.1b deep-scrub ok 2024-02-23T00:28:55.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:54.904311+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub starts 2024-02-23T00:28:55.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:54.905253+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.f scrub ok 2024-02-23T00:28:55.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:53.628651+0000 osd.2 (osd.2) 25 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:28:55.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:53.629565+0000 osd.2 (osd.2) 26 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:28:55.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:54.653805+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.1b deep-scrub starts 2024-02-23T00:28:55.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:54.657169+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.1b deep-scrub ok 2024-02-23T00:28:55.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:54.904311+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub starts 2024-02-23T00:28:55.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:54.905253+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.f scrub ok 2024-02-23T00:28:55.669 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:53.628651+0000 osd.2 (osd.2) 25 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:28:55.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:53.629565+0000 osd.2 (osd.2) 26 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:28:55.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:54.653805+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.1b deep-scrub starts 2024-02-23T00:28:55.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:54.657169+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.1b deep-scrub ok 2024-02-23T00:28:55.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:54.904311+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.f scrub starts 2024-02-23T00:28:55.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:54.905253+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.f scrub ok 2024-02-23T00:28:56.394 INFO:teuthology.orchestra.run.smithi087.stdout:219043332118 2024-02-23T00:28:56.395 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-23T00:28:56.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:54.991350+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:54.994271+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:55.228158+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:55.230076+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:55.473916+0000 mgr.y (mgr.24437) 69 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:55.676631+0000 osd.1 (osd.1) 31 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:55.679518+0000 osd.1 (osd.1) 32 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:55.905521+0000 osd.5 (osd.5) 29 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:55.907522+0000 osd.5 (osd.5) 30 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:54.991350+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:54.994271+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:55.228158+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:28:56.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:55.230076+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:28:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:55.473916+0000 mgr.y (mgr.24437) 69 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-23T00:28:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:55.676631+0000 osd.1 (osd.1) 31 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:28:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:55.679518+0000 osd.1 (osd.1) 32 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:28:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:55.905521+0000 osd.5 (osd.5) 29 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:28:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:55.907522+0000 osd.5 (osd.5) 30 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:28:56.682 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:28:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:54.991350+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:28:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:54.994271+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:28:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:55.228158+0000 osd.0 (osd.0) 27 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:28:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:55.230076+0000 osd.0 (osd.0) 28 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:28:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:55.473916+0000 mgr.y (mgr.24437) 69 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-23T00:28:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:55.676631+0000 osd.1 (osd.1) 31 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:28:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:55.679518+0000 osd.1 (osd.1) 32 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:28:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:55.905521+0000 osd.5 (osd.5) 29 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:28:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:55.907522+0000 osd.5 (osd.5) 30 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:28:57.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:55.812162+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:55.812205+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:56.019846+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:56.022830+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:56.266333+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:56.952893+0000 osd.5 (osd.5) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:56.952986+0000 osd.5 (osd.5) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:55.812162+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:55.812205+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:56.019846+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:28:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:56.022830+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:28:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:56.266333+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-23T00:28:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:56.952893+0000 osd.5 (osd.5) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:56.952986+0000 osd.5 (osd.5) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:55.812162+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:55.812205+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:56.019846+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:28:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:56.022830+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:28:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:56.266333+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-23T00:28:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:56.952893+0000 osd.5 (osd.5) 31 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:28:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:56.952986+0000 osd.5 (osd.5) 32 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:28:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:57.034239+0000 osd.3 (osd.3) 39 : cluster [DBG] 4.11 deep-scrub starts 2024-02-23T00:28:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:57.039655+0000 osd.3 (osd.3) 40 : cluster [DBG] 4.11 deep-scrub ok 2024-02-23T00:28:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:57.299547+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-23T00:28:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:57.474294+0000 mgr.y (mgr.24437) 70 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:57.677546+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:28:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:57.678516+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:28:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:57.034239+0000 osd.3 (osd.3) 39 : cluster [DBG] 4.11 deep-scrub starts 2024-02-23T00:28:58.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:57.039655+0000 osd.3 (osd.3) 40 : cluster [DBG] 4.11 deep-scrub ok 2024-02-23T00:28:58.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:57.299547+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-23T00:28:58.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:57.474294+0000 mgr.y (mgr.24437) 70 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:58.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:57.677546+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:28:58.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:28:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:57.678516+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:28:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:57.034239+0000 osd.3 (osd.3) 39 : cluster [DBG] 4.11 deep-scrub starts 2024-02-23T00:28:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:57.039655+0000 osd.3 (osd.3) 40 : cluster [DBG] 4.11 deep-scrub ok 2024-02-23T00:28:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:57.299547+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-23T00:28:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:57.474294+0000 mgr.y (mgr.24437) 70 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:28:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:57.677546+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:28:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:28:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:57.678516+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 -- 172.21.15.87:0/2708588299 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 msgr2=0x7f23e80fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2708588299 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 0x7f23e80fcf90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f23e4004870 tx=0x7f23e4009cb0).stop 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 -- 172.21.15.87:0/2708588299 shutdown_connections 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2708588299 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23e80f7400 0x7f23e8103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2708588299 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 0x7f23e80fcf90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2708588299 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23e80fd510 0x7f23e80f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 -- 172.21.15.87:0/2708588299 >> 172.21.15.87:0/2708588299 conn(0x7f23e80f2820 msgr2=0x7f23e80f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 -- 172.21.15.87:0/2708588299 shutdown_connections 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 -- 172.21.15.87:0/2708588299 wait complete. 2024-02-23T00:29:00.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 librados: starting msgr at 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 librados: starting objecter 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 Processor -- start 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 -- start start 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 librados: setting wanted keys 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.996+0000 7f23f04d6700 1 librados: calling monclient init 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23e80f7400 0x7f23e810e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 0x7f23e810e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23e80fd510 0x7f23e8108440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e8108930 con 0x7f23e80fd510 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e8108aa0 con 0x7f23e80f7400 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e8108c60 con 0x7f23e80fcbb0 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eda71700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 0x7f23e810e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eea73700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23e80fd510 0x7f23e8108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:00.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eda71700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 0x7f23e810e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:54948/0 (socket says 172.21.15.87:54948) 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eda71700 1 -- 172.21.15.87:0/2298569310 learned_addr learned my addr 172.21.15.87:0/2298569310 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23ee272700 1 --2- 172.21.15.87:0/2298569310 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23e80f7400 0x7f23e810e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eda71700 1 -- 172.21.15.87:0/2298569310 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23e80f7400 msgr2=0x7f23e810e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eda71700 1 --2- 172.21.15.87:0/2298569310 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23e80f7400 0x7f23e810e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eda71700 1 -- 172.21.15.87:0/2298569310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23e80fd510 msgr2=0x7f23e8108440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eda71700 1 --2- 172.21.15.87:0/2298569310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23e80fd510 0x7f23e8108440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eda71700 1 -- 172.21.15.87:0/2298569310 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d8005040 con 0x7f23e80fcbb0 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eea73700 1 --2- 172.21.15.87:0/2298569310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23e80fd510 0x7f23e8108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23eda71700 1 --2- 172.21.15.87:0/2298569310 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 0x7f23e810e790 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f23d80048f0 tx=0x7f23d8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23df7fe700 10 client.?.objecter ms_handle_connect 0x7f23e80fcbb0 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23df7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23df7fe700 1 -- 172.21.15.87:0/2298569310 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8007b00 con 0x7f23e80fcbb0 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23df7fe700 1 -- 172.21.15.87:0/2298569310 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23d800f040 con 0x7f23e80fcbb0 2024-02-23T00:29:00.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23df7fe700 1 -- 172.21.15.87:0/2298569310 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8014990 con 0x7f23e80fcbb0 2024-02-23T00:29:00.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e4005040 con 0x7f23e80fcbb0 2024-02-23T00:29:00.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 10 client.24581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:00.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e8109250 con 0x7f23e80fcbb0 2024-02-23T00:29:00.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.997+0000 7f23f04d6700 1 librados: init done 2024-02-23T00:29:00.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.998+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e805fa60 con 0x7f23e80fcbb0 2024-02-23T00:29:00.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23df7fe700 1 -- 172.21.15.87:0/2298569310 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f23d8010500 con 0x7f23e80fcbb0 2024-02-23T00:29:00.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23df7fe700 1 --2- 172.21.15.87:0/2298569310 >> v2:172.21.15.87:6800/1231597463 conn(0x7f23d4062c90 0x7f23d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:00.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23df7fe700 1 -- 172.21.15.87:0/2298569310 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(80..80 src has 28..80) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f23d807c5f0 con 0x7f23e80fcbb0 2024-02-23T00:29:00.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23df7fe700 10 client.24581.objecter ms_dispatch 0x7f23e8001550 osd_map(80..80 src has 28..80) v4 2024-02-23T00:29:00.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23df7fe700 3 client.24581.objecter handle_osd_map got epochs [80,80] > 0 2024-02-23T00:29:00.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23df7fe700 3 client.24581.objecter handle_osd_map decoding full epoch 80 2024-02-23T00:29:00.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23ee272700 1 --2- 172.21.15.87:0/2298569310 >> v2:172.21.15.87:6800/1231597463 conn(0x7f23d4062c90 0x7f23d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:00.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23df7fe700 20 client.24581.objecter dump_active .. 0 homeless 2024-02-23T00:29:00.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23df7fe700 10 client.24581.objecter ms_handle_connect 0x7f23d4062c90 2024-02-23T00:29:00.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:28:59.999+0000 7f23ee272700 1 --2- 172.21.15.87:0/2298569310 >> v2:172.21.15.87:6800/1231597463 conn(0x7f23d4062c90 0x7f23d4065150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f23e4009a00 tx=0x7f23e4009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:00.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.003+0000 7f23df7fe700 1 -- 172.21.15.87:0/2298569310 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f23d8049f60 con 0x7f23e80fcbb0 2024-02-23T00:29:00.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.171+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f23e8106130 con 0x7f23e80fcbb0 2024-02-23T00:29:00.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.171+0000 7f23df7fe700 1 -- 172.21.15.87:0/2298569310 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f23d8050260 con 0x7f23e80fcbb0 2024-02-23T00:29:00.174 INFO:teuthology.orchestra.run.smithi087.stdout:34359738425 2024-02-23T00:29:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 10 librados: watch_flush enter 2024-02-23T00:29:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 10 librados: watch_flush exit 2024-02-23T00:29:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 20 client.24581.objecter shutdown clearing up homeless session... 2024-02-23T00:29:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 10 client.24581.objecter successfully canceled tick 2024-02-23T00:29:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 >> v2:172.21.15.87:6800/1231597463 conn(0x7f23d4062c90 msgr2=0x7f23d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2298569310 >> v2:172.21.15.87:6800/1231597463 conn(0x7f23d4062c90 0x7f23d4065150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f23e4009a00 tx=0x7f23e4009cb0).stop 2024-02-23T00:29:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 msgr2=0x7f23e810e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:00.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2298569310 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 0x7f23e810e790 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f23d80048f0 tx=0x7f23d8009cb0).stop 2024-02-23T00:29:00.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 shutdown_connections 2024-02-23T00:29:00.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2298569310 >> v2:172.21.15.87:6800/1231597463 conn(0x7f23d4062c90 0x7f23d4065150 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:00.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2298569310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23e80fd510 0x7f23e8108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:00.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2298569310 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23e80f7400 0x7f23e810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:00.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 --2- 172.21.15.87:0/2298569310 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23e80fcbb0 0x7f23e810e790 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:00.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 >> 172.21.15.87:0/2298569310 conn(0x7f23e80f2820 msgr2=0x7f23e80f32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:00.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 shutdown_connections 2024-02-23T00:29:00.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 -- 172.21.15.87:0/2298569310 wait complete. 2024-02-23T00:29:00.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:00.173+0000 7f23f04d6700 1 librados: shutdown 2024-02-23T00:29:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:58.380622+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.15 deep-scrub starts 2024-02-23T00:29:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:58.383637+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 deep-scrub ok 2024-02-23T00:29:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:28:59.087247+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:59.223992+0000 osd.0 (osd.0) 29 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:29:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:59.226886+0000 osd.0 (osd.0) 30 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:29:00.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:58.380622+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.15 deep-scrub starts 2024-02-23T00:29:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:58.383637+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 deep-scrub ok 2024-02-23T00:29:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:28:59.087247+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:59.223992+0000 osd.0 (osd.0) 29 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:29:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:59.226886+0000 osd.0 (osd.0) 30 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:29:00.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:58.380622+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.15 deep-scrub starts 2024-02-23T00:29:00.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:58.383637+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 deep-scrub ok 2024-02-23T00:29:00.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:28:59.087247+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:00.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:59.223992+0000 osd.0 (osd.0) 29 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:29:00.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:59.226886+0000 osd.0 (osd.0) 30 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:29:00.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:29:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:29:00.965 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738419 got 34359738425 for osd.0 2024-02-23T00:29:00.965 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-23T00:29:01.238 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:01.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:59.474665+0000 mgr.y (mgr.24437) 71 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:59.684073+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:28:59.685037+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:00.172884+0000 mon.c (mon.2) 19 : audit [DBG] from='client.? 172.21.15.87:0/2298569310' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:00.184715+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:00.187780+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:00.700869+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.a deep-scrub starts 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:00.703732+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.a deep-scrub ok 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:59.474665+0000 mgr.y (mgr.24437) 71 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:59.684073+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:29:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:28:59.685037+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:29:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:00.172884+0000 mon.c (mon.2) 19 : audit [DBG] from='client.? 172.21.15.87:0/2298569310' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-23T00:29:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:00.184715+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:29:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:00.187780+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:29:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:00.700869+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.a deep-scrub starts 2024-02-23T00:29:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:00.703732+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.a deep-scrub ok 2024-02-23T00:29:01.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:59.474665+0000 mgr.y (mgr.24437) 71 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:29:01.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:59.684073+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:29:01.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:28:59.685037+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:29:01.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:00.172884+0000 mon.c (mon.2) 19 : audit [DBG] from='client.? 172.21.15.87:0/2298569310' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-23T00:29:01.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:00.184715+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:29:01.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:00.187780+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:29:01.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:00.700869+0000 osd.1 (osd.1) 33 : cluster [DBG] 2.a deep-scrub starts 2024-02-23T00:29:01.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:00.703732+0000 osd.1 (osd.1) 34 : cluster [DBG] 2.a deep-scrub ok 2024-02-23T00:29:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:00.318870+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:29:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:00.319977+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:29:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:00.543083+0000 mgr.y (mgr.24437) 72 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.023063+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:29:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.026571+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:29:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.232586+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:29:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.234352+0000 osd.0 (osd.0) 34 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:29:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.731028+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:29:02.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.733283+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:29:02.443 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:02] "GET /metrics HTTP/1.1" 200 172159 "" "Prometheus/2.33.4" 2024-02-23T00:29:02.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:00.318870+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:29:02.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:00.319977+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:29:02.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:00.543083+0000 mgr.y (mgr.24437) 72 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:02.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.023063+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:29:02.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.026571+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:29:02.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.232586+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:29:02.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.234352+0000 osd.0 (osd.0) 34 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:29:02.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.731028+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:29:02.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.733283+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:29:02.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:00.318870+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:29:02.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:00.319977+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:29:02.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:00.543083+0000 mgr.y (mgr.24437) 72 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:02.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.023063+0000 osd.3 (osd.3) 41 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:29:02.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.026571+0000 osd.3 (osd.3) 42 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:29:02.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.232586+0000 osd.0 (osd.0) 33 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:29:02.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.234352+0000 osd.0 (osd.0) 34 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:29:02.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.731028+0000 osd.4 (osd.4) 33 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:29:02.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.733283+0000 osd.4 (osd.4) 34 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:29:03.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.022+0000 7fefcb3ad700 1 -- 172.21.15.87:0/3690055500 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fefc40f7400 msgr2=0x7fefc4103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:03.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.022+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/3690055500 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fefc40f7400 0x7fefc4103a20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fefbc009be0 tx=0x7fefbc004d10).stop 2024-02-23T00:29:03.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.022+0000 7fefcb3ad700 1 -- 172.21.15.87:0/3690055500 shutdown_connections 2024-02-23T00:29:03.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.022+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/3690055500 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fefc40f7400 0x7fefc4103a20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.022+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/3690055500 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefc40fcbb0 0x7fefc40fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.022+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/3690055500 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc40fd510 0x7fefc40f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.022+0000 7fefcb3ad700 1 -- 172.21.15.87:0/3690055500 >> 172.21.15.87:0/3690055500 conn(0x7fefc40f2820 msgr2=0x7fefc40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.023+0000 7fefcb3ad700 1 -- 172.21.15.87:0/3690055500 shutdown_connections 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.023+0000 7fefcb3ad700 1 -- 172.21.15.87:0/3690055500 wait complete. 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.023+0000 7fefcb3ad700 1 librados: starting msgr at 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.023+0000 7fefcb3ad700 1 librados: starting objecter 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.023+0000 7fefcb3ad700 1 Processor -- start 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.023+0000 7fefcb3ad700 1 -- start start 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.023+0000 7fefcb3ad700 1 librados: setting wanted keys 2024-02-23T00:29:03.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.023+0000 7fefcb3ad700 1 librados: calling monclient init 2024-02-23T00:29:03.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefcb3ad700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fefc40f7400 0x7fefc41101e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefcb3ad700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefc40fcbb0 0x7fefc41106d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefcb3ad700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc40fd510 0x7fefc410a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefcb3ad700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc410a8b0 con 0x7fefc40fcbb0 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefcb3ad700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc410aa20 con 0x7fefc40fd510 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefcb3ad700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc410ab90 con 0x7fefc40f7400 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc8948700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefc40fcbb0 0x7fefc41106d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc8948700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefc40fcbb0 0x7fefc41106d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50374/0 (socket says 172.21.15.87:50374) 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc8948700 1 -- 172.21.15.87:0/2221097441 learned_addr learned my addr 172.21.15.87:0/2221097441 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc9149700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fefc40f7400 0x7fefc41101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc994a700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc40fd510 0x7fefc410a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc8948700 1 -- 172.21.15.87:0/2221097441 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fefc40f7400 msgr2=0x7fefc41101e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:03.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc8948700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fefc40f7400 0x7fefc41101e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc8948700 1 -- 172.21.15.87:0/2221097441 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc40fd510 msgr2=0x7fefc410a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc8948700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc40fd510 0x7fefc410a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc8948700 1 -- 172.21.15.87:0/2221097441 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc0005040 con 0x7fefc40fcbb0 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc994a700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc40fd510 0x7fefc410a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefc8948700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefc40fcbb0 0x7fefc41106d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fefb400e990 tx=0x7fefb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefba7fc700 10 client.?.objecter ms_handle_connect 0x7fefc40fcbb0 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.024+0000 7fefba7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.025+0000 7fefba7fc700 1 -- 172.21.15.87:0/2221097441 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4004680 con 0x7fefc40fcbb0 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.025+0000 7fefba7fc700 1 -- 172.21.15.87:0/2221097441 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefb400f040 con 0x7fefc40fcbb0 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.025+0000 7fefba7fc700 1 -- 172.21.15.87:0/2221097441 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb40146b0 con 0x7fefc40fcbb0 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.025+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefbc0097c0 con 0x7fefc40fcbb0 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.025+0000 7fefcb3ad700 10 client.14661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:03.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.025+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc410b120 con 0x7fefc40fcbb0 2024-02-23T00:29:03.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.025+0000 7fefcb3ad700 1 librados: init done 2024-02-23T00:29:03.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.026+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc405fa60 con 0x7fefc40fcbb0 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefba7fc700 1 -- 172.21.15.87:0/2221097441 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fefb4014ae0 con 0x7fefc40fcbb0 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefba7fc700 1 --2- 172.21.15.87:0/2221097441 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefb0062c90 0x7fefb0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefba7fc700 1 -- 172.21.15.87:0/2221097441 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(80..80 src has 28..80) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fefb407c4e0 con 0x7fefc40fcbb0 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefc9149700 1 --2- 172.21.15.87:0/2221097441 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefb0062c90 0x7fefb0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefba7fc700 10 client.14661.objecter ms_dispatch 0x7fefc4001550 osd_map(80..80 src has 28..80) v4 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefba7fc700 3 client.14661.objecter handle_osd_map got epochs [80,80] > 0 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefba7fc700 3 client.14661.objecter handle_osd_map decoding full epoch 80 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefba7fc700 20 client.14661.objecter dump_active .. 0 homeless 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefba7fc700 10 client.14661.objecter ms_handle_connect 0x7fefb0062c90 2024-02-23T00:29:03.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.027+0000 7fefc9149700 1 --2- 172.21.15.87:0/2221097441 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefb0062c90 0x7fefb0065150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fefc0009350 tx=0x7fefc000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:03.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.031+0000 7fefba7fc700 1 -- 172.21.15.87:0/2221097441 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fefb4049f60 con 0x7fefc40fcbb0 2024-02-23T00:29:03.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.196+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fefc41061b0 con 0x7fefc40fcbb0 2024-02-23T00:29:03.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.197+0000 7fefba7fc700 1 -- 172.21.15.87:0/2221097441 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fefb404ff90 con 0x7fefc40fcbb0 2024-02-23T00:29:03.199 INFO:teuthology.orchestra.run.smithi087.stdout:55834574900 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 10 librados: watch_flush enter 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 10 librados: watch_flush exit 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 20 client.14661.objecter shutdown clearing up homeless session... 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 10 client.14661.objecter successfully canceled tick 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefb0062c90 msgr2=0x7fefb0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/2221097441 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefb0062c90 0x7fefb0065150 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fefc0009350 tx=0x7fefc000b040).stop 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefc40fcbb0 msgr2=0x7fefc41106d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefc40fcbb0 0x7fefc41106d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fefb400e990 tx=0x7fefb400b040).stop 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 shutdown_connections 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/2221097441 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefb0062c90 0x7fefb0065150 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fefc40fd510 0x7fefc410a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:03.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fefc40f7400 0x7fefc41101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:03.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.198+0000 7fefcb3ad700 1 --2- 172.21.15.87:0/2221097441 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fefc40fcbb0 0x7fefc41106d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:03.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.199+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 >> 172.21.15.87:0/2221097441 conn(0x7fefc40f2820 msgr2=0x7fefc4060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:03.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.199+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 shutdown_connections 2024-02-23T00:29:03.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.199+0000 7fefcb3ad700 1 -- 172.21.15.87:0/2221097441 wait complete. 2024-02-23T00:29:03.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:03.199+0000 7fefcb3ad700 1 librados: shutdown 2024-02-23T00:29:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.475314+0000 mgr.y (mgr.24437) 73 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.775005+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.b scrub starts 2024-02-23T00:29:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.775937+0000 osd.2 (osd.2) 30 : cluster [DBG] 5.b scrub ok 2024-02-23T00:29:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.985504+0000 osd.3 (osd.3) 43 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:29:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:01.986458+0000 osd.3 (osd.3) 44 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:29:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:02.337551+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:29:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:02.340446+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:29:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:02.386527+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:03.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:02.387321+0000 mon.a (mon.0) 724 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:29:03.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:02.388047+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:29:03.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:02.391895+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:03.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:02.708269+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.15 deep-scrub starts 2024-02-23T00:29:03.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:02.711277+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.15 deep-scrub ok 2024-02-23T00:29:03.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:03.016118+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:29:03.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:03.017045+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:29:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.475314+0000 mgr.y (mgr.24437) 73 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.775005+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.b scrub starts 2024-02-23T00:29:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.775937+0000 osd.2 (osd.2) 30 : cluster [DBG] 5.b scrub ok 2024-02-23T00:29:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.985504+0000 osd.3 (osd.3) 43 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:29:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:01.986458+0000 osd.3 (osd.3) 44 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:29:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:02.337551+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:29:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:02.340446+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:02.386527+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:02.387321+0000 mon.a (mon.0) 724 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:02.388047+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:02.391895+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:02.708269+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.15 deep-scrub starts 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:02.711277+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.15 deep-scrub ok 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:03.016118+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:03.017045+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.475314+0000 mgr.y (mgr.24437) 73 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.775005+0000 osd.2 (osd.2) 29 : cluster [DBG] 5.b scrub starts 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.775937+0000 osd.2 (osd.2) 30 : cluster [DBG] 5.b scrub ok 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.985504+0000 osd.3 (osd.3) 43 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:29:03.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:01.986458+0000 osd.3 (osd.3) 44 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:02.337551+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:02.340446+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:02.386527+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:02.387321+0000 mon.a (mon.0) 724 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:02.388047+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:02.391895+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:02.708269+0000 osd.1 (osd.1) 35 : cluster [DBG] 2.15 deep-scrub starts 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:02.711277+0000 osd.1 (osd.1) 36 : cluster [DBG] 2.15 deep-scrub ok 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:03.016118+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:29:03.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:03.017045+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:29:04.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574900 for osd.1 2024-02-23T00:29:04.080 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-23T00:29:04.109 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:29:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:29:04.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:02.767831+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.a scrub starts 2024-02-23T00:29:04.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:02.768982+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.a scrub ok 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:02.966186+0000 osd.3 (osd.3) 45 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:02.969654+0000 osd.3 (osd.3) 46 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:03.198431+0000 mon.a (mon.0) 727 : audit [DBG] from='client.? 172.21.15.87:0/2221097441' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:03.687540+0000 osd.1 (osd.1) 37 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:03.691013+0000 osd.1 (osd.1) 38 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:03.789628+0000 osd.4 (osd.4) 35 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:03.790523+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:04.048455+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.19 deep-scrub starts 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:04.049408+0000 osd.5 (osd.5) 36 : cluster [DBG] 6.19 deep-scrub ok 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:02.767831+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.a scrub starts 2024-02-23T00:29:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:02.768982+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.a scrub ok 2024-02-23T00:29:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:02.966186+0000 osd.3 (osd.3) 45 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:29:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:02.969654+0000 osd.3 (osd.3) 46 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:29:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:03.198431+0000 mon.a (mon.0) 727 : audit [DBG] from='client.? 172.21.15.87:0/2221097441' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-23T00:29:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:03.687540+0000 osd.1 (osd.1) 37 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:29:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:03.691013+0000 osd.1 (osd.1) 38 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:29:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:03.789628+0000 osd.4 (osd.4) 35 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:29:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:03.790523+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:29:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:04.048455+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.19 deep-scrub starts 2024-02-23T00:29:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:04.049408+0000 osd.5 (osd.5) 36 : cluster [DBG] 6.19 deep-scrub ok 2024-02-23T00:29:04.352 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:02.767831+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.a scrub starts 2024-02-23T00:29:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:02.768982+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.a scrub ok 2024-02-23T00:29:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:02.966186+0000 osd.3 (osd.3) 45 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:29:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:02.969654+0000 osd.3 (osd.3) 46 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:29:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:03.198431+0000 mon.a (mon.0) 727 : audit [DBG] from='client.? 172.21.15.87:0/2221097441' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-23T00:29:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:03.687540+0000 osd.1 (osd.1) 37 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:29:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:03.691013+0000 osd.1 (osd.1) 38 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:29:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:03.789628+0000 osd.4 (osd.4) 35 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:29:04.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:03.790523+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:29:04.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:04.048455+0000 osd.5 (osd.5) 35 : cluster [DBG] 6.19 deep-scrub starts 2024-02-23T00:29:04.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:04.049408+0000 osd.5 (osd.5) 36 : cluster [DBG] 6.19 deep-scrub ok 2024-02-23T00:29:05.386 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:03.475697+0000 mgr.y (mgr.24437) 74 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:29:05.386 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:03.713757+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:05.386 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:03.713831+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:05.386 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:04.006580+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:29:05.386 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:04.010336+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:29:05.386 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:04.109925+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-23T00:29:05.386 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:04.224964+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:04.227853+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:04.266468+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:04.267421+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:03.475697+0000 mgr.y (mgr.24437) 74 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:03.713757+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:03.713831+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:04.006580+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:04.010336+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:04.109925+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:04.224964+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:04.227853+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:29:05.387 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:04.266468+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:29:05.388 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:04.267421+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:29:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:03.475697+0000 mgr.y (mgr.24437) 74 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:29:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:03.713757+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:03.713831+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:04.006580+0000 osd.3 (osd.3) 47 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:29:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:04.010336+0000 osd.3 (osd.3) 48 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:29:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:04.109925+0000 mon.a (mon.0) 728 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-23T00:29:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:04.224964+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:29:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:04.227853+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:29:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:04.266468+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:29:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:04.267421+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.378+0000 7ffa7ed48700 1 -- 172.21.15.87:0/1145128892 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa780fe510 msgr2=0x7ffa780f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.378+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/1145128892 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa780fe510 0x7ffa780f7d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ffa640047b0 tx=0x7ffa64009cb0).stop 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.378+0000 7ffa7ed48700 1 -- 172.21.15.87:0/1145128892 shutdown_connections 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.378+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/1145128892 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa780f8420 0x7ffa78104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.378+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/1145128892 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffa780fdbb0 0x7ffa780fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.378+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/1145128892 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa780fe510 0x7ffa780f7d40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.378+0000 7ffa7ed48700 1 -- 172.21.15.87:0/1145128892 >> 172.21.15.87:0/1145128892 conn(0x7ffa780f3800 msgr2=0x7ffa780f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.378+0000 7ffa7ed48700 1 -- 172.21.15.87:0/1145128892 shutdown_connections 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.378+0000 7ffa7ed48700 1 -- 172.21.15.87:0/1145128892 wait complete. 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 librados: starting msgr at 2024-02-23T00:29:06.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 librados: starting objecter 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 Processor -- start 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 -- start start 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 librados: setting wanted keys 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 librados: calling monclient init 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffa780f8420 0x7ffa7810e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa780fdbb0 0x7ffa7810e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa780fe510 0x7ffa78108210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa78108700 con 0x7ffa780fdbb0 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa78108870 con 0x7ffa780fe510 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7ed48700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa78108a30 con 0x7ffa780f8420 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7cae4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffa780f8420 0x7ffa7810e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7cae4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffa780f8420 0x7ffa7810e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35318/0 (socket says 172.21.15.87:35318) 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7cae4700 1 -- 172.21.15.87:0/129754348 learned_addr learned my addr 172.21.15.87:0/129754348 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:06.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7d2e5700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa780fe510 0x7ffa78108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa77fff700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa780fdbb0 0x7ffa7810e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7cae4700 1 -- 172.21.15.87:0/129754348 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa780fe510 msgr2=0x7ffa78108210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7cae4700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa780fe510 0x7ffa78108210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7cae4700 1 -- 172.21.15.87:0/129754348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa780fdbb0 msgr2=0x7ffa7810e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7cae4700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa780fdbb0 0x7ffa7810e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7cae4700 1 -- 172.21.15.87:0/129754348 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c005040 con 0x7ffa780f8420 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa77fff700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa780fdbb0 0x7ffa7810e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa7cae4700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffa780f8420 0x7ffa7810e070 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7ffa6c009400 tx=0x7ffa6c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa75ffb700 10 client.?.objecter ms_handle_connect 0x7ffa780f8420 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.379+0000 7ffa75ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.380+0000 7ffa75ffb700 1 -- 172.21.15.87:0/129754348 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c009870 con 0x7ffa780f8420 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.380+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa64005040 con 0x7ffa780f8420 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.380+0000 7ffa75ffb700 1 -- 172.21.15.87:0/129754348 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa6c00f040 con 0x7ffa780f8420 2024-02-23T00:29:06.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.380+0000 7ffa75ffb700 1 -- 172.21.15.87:0/129754348 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c014b50 con 0x7ffa780f8420 2024-02-23T00:29:06.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.380+0000 7ffa7ed48700 10 client.24593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:06.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.380+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa78114ea0 con 0x7ffa780f8420 2024-02-23T00:29:06.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.380+0000 7ffa7ed48700 1 librados: init done 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.381+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7805ee80 con 0x7ffa780f8420 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.381+0000 7ffa75ffb700 1 -- 172.21.15.87:0/129754348 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ffa6c00e840 con 0x7ffa780f8420 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.381+0000 7ffa75ffb700 1 --2- 172.21.15.87:0/129754348 >> v2:172.21.15.87:6800/1231597463 conn(0x7ffa68062c90 0x7ffa68065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.381+0000 7ffa75ffb700 1 -- 172.21.15.87:0/129754348 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(81..81 src has 28..81) v4 ==== 6172+0+0 (secure 0 0 0) 0x7ffa6c07c720 con 0x7ffa780f8420 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.381+0000 7ffa75ffb700 10 client.24593.objecter ms_dispatch 0x7ffa78001550 osd_map(81..81 src has 28..81) v4 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.381+0000 7ffa75ffb700 3 client.24593.objecter handle_osd_map got epochs [81,81] > 0 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.381+0000 7ffa75ffb700 3 client.24593.objecter handle_osd_map decoding full epoch 81 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.381+0000 7ffa77fff700 1 --2- 172.21.15.87:0/129754348 >> v2:172.21.15.87:6800/1231597463 conn(0x7ffa68062c90 0x7ffa68065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.382+0000 7ffa75ffb700 20 client.24593.objecter dump_active .. 0 homeless 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.382+0000 7ffa75ffb700 10 client.24593.objecter ms_handle_connect 0x7ffa68062c90 2024-02-23T00:29:06.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.382+0000 7ffa77fff700 1 --2- 172.21.15.87:0/129754348 >> v2:172.21.15.87:6800/1231597463 conn(0x7ffa68062c90 0x7ffa68065150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ffa64009bc0 tx=0x7ffa64007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:06.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.386+0000 7ffa75ffb700 1 -- 172.21.15.87:0/129754348 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ffa6c049f60 con 0x7ffa780f8420 2024-02-23T00:29:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:05.278807+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.b scrub starts 2024-02-23T00:29:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:05.279876+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.b scrub ok 2024-02-23T00:29:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:05.638203+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.1d deep-scrub starts 2024-02-23T00:29:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:05.641450+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.1d deep-scrub ok 2024-02-23T00:29:06.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.548+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ffa78107110 con 0x7ffa780f8420 2024-02-23T00:29:06.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.549+0000 7ffa75ffb700 1 -- 172.21.15.87:0/129754348 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffa6c050390 con 0x7ffa780f8420 2024-02-23T00:29:06.551 INFO:teuthology.orchestra.run.smithi087.stdout:77309411377 2024-02-23T00:29:06.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 10 librados: watch_flush enter 2024-02-23T00:29:06.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 10 librados: watch_flush exit 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 20 client.24593.objecter shutdown clearing up homeless session... 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 10 client.24593.objecter successfully canceled tick 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 >> v2:172.21.15.87:6800/1231597463 conn(0x7ffa68062c90 msgr2=0x7ffa68065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/129754348 >> v2:172.21.15.87:6800/1231597463 conn(0x7ffa68062c90 0x7ffa68065150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ffa64009bc0 tx=0x7ffa64007f90).stop 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffa780f8420 msgr2=0x7ffa7810e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffa780f8420 0x7ffa7810e070 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7ffa6c009400 tx=0x7ffa6c009cb0).stop 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 shutdown_connections 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/129754348 >> v2:172.21.15.87:6800/1231597463 conn(0x7ffa68062c90 0x7ffa68065150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffa780fe510 0x7ffa78108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffa780f8420 0x7ffa7810e070 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 --2- 172.21.15.87:0/129754348 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffa780fdbb0 0x7ffa7810e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 >> 172.21.15.87:0/129754348 conn(0x7ffa780f3800 msgr2=0x7ffa780f5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 shutdown_connections 2024-02-23T00:29:06.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.550+0000 7ffa7ed48700 1 -- 172.21.15.87:0/129754348 wait complete. 2024-02-23T00:29:06.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:06.551+0000 7ffa7ed48700 1 librados: shutdown 2024-02-23T00:29:06.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:05.278807+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.b scrub starts 2024-02-23T00:29:06.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:05.279876+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.b scrub ok 2024-02-23T00:29:06.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:05.638203+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.1d deep-scrub starts 2024-02-23T00:29:06.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:05.641450+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.1d deep-scrub ok 2024-02-23T00:29:06.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:05.278807+0000 osd.7 (osd.7) 37 : cluster [DBG] 2.b scrub starts 2024-02-23T00:29:06.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:05.279876+0000 osd.7 (osd.7) 38 : cluster [DBG] 2.b scrub ok 2024-02-23T00:29:06.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:05.638203+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.1d deep-scrub starts 2024-02-23T00:29:06.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:05.641450+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.1d deep-scrub ok 2024-02-23T00:29:07.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:05.476541+0000 mgr.y (mgr.24437) 75 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:07.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:06.550252+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? 172.21.15.87:0/129754348' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-23T00:29:07.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:06.818559+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.18 deep-scrub starts 2024-02-23T00:29:07.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:06.822812+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.18 deep-scrub ok 2024-02-23T00:29:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:05.476541+0000 mgr.y (mgr.24437) 75 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:06.550252+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? 172.21.15.87:0/129754348' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-23T00:29:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:06.818559+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.18 deep-scrub starts 2024-02-23T00:29:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:06.822812+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.18 deep-scrub ok 2024-02-23T00:29:07.364 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411370 got 77309411377 for osd.2 2024-02-23T00:29:07.364 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-23T00:29:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:05.476541+0000 mgr.y (mgr.24437) 75 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:06.550252+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? 172.21.15.87:0/129754348' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-23T00:29:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:06.818559+0000 osd.4 (osd.4) 37 : cluster [DBG] 4.18 deep-scrub starts 2024-02-23T00:29:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:06.822812+0000 osd.4 (osd.4) 38 : cluster [DBG] 4.18 deep-scrub ok 2024-02-23T00:29:07.634 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:09.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:07.476879+0000 mgr.y (mgr.24437) 76 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:09.232 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:07.957042+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-02-23T00:29:09.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:07.960137+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-02-23T00:29:09.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:08.062470+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.a scrub starts 2024-02-23T00:29:09.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:08.063394+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.a scrub ok 2024-02-23T00:29:09.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:08.576996+0000 osd.1 (osd.1) 41 : cluster [DBG] 5.c scrub starts 2024-02-23T00:29:09.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:08.580097+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.c scrub ok 2024-02-23T00:29:09.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:08.816390+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.c scrub starts 2024-02-23T00:29:09.234 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:08.818229+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.c scrub ok 2024-02-23T00:29:09.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:07.476879+0000 mgr.y (mgr.24437) 76 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:09.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:07.957042+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-02-23T00:29:09.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:07.960137+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-02-23T00:29:09.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:08.062470+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.a scrub starts 2024-02-23T00:29:09.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:08.063394+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.a scrub ok 2024-02-23T00:29:09.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:08.576996+0000 osd.1 (osd.1) 41 : cluster [DBG] 5.c scrub starts 2024-02-23T00:29:09.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:08.580097+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.c scrub ok 2024-02-23T00:29:09.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:08.816390+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.c scrub starts 2024-02-23T00:29:09.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:08.818229+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.c scrub ok 2024-02-23T00:29:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:07.476879+0000 mgr.y (mgr.24437) 76 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:07.957042+0000 osd.3 (osd.3) 49 : cluster [DBG] 6.c scrub starts 2024-02-23T00:29:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:07.960137+0000 osd.3 (osd.3) 50 : cluster [DBG] 6.c scrub ok 2024-02-23T00:29:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:08.062470+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.a scrub starts 2024-02-23T00:29:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:08.063394+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.a scrub ok 2024-02-23T00:29:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:08.576996+0000 osd.1 (osd.1) 41 : cluster [DBG] 5.c scrub starts 2024-02-23T00:29:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:08.580097+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.c scrub ok 2024-02-23T00:29:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:08.816390+0000 osd.4 (osd.4) 39 : cluster [DBG] 4.c scrub starts 2024-02-23T00:29:09.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:08.818229+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.c scrub ok 2024-02-23T00:29:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 -- 172.21.15.87:0/2471555937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9800fe510 msgr2=0x7fc9800f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 --2- 172.21.15.87:0/2471555937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9800fe510 0x7fc9800f7d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fc974004870 tx=0x7fc974009cb0).stop 2024-02-23T00:29:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 -- 172.21.15.87:0/2471555937 shutdown_connections 2024-02-23T00:29:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 --2- 172.21.15.87:0/2471555937 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9800f8420 0x7fc980104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:29:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 --2- 172.21.15.87:0/2471555937 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9800fdbb0 0x7fc9800fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 --2- 172.21.15.87:0/2471555937 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9800fe510 0x7fc9800f7d40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 -- 172.21.15.87:0/2471555937 >> 172.21.15.87:0/2471555937 conn(0x7fc9800f3800 msgr2=0x7fc9800f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 -- 172.21.15.87:0/2471555937 shutdown_connections 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 -- 172.21.15.87:0/2471555937 wait complete. 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 librados: starting msgr at 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.664+0000 7fc9869ff700 1 librados: starting objecter 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 Processor -- start 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 -- start start 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 librados: setting wanted keys 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 librados: calling monclient init 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9800f8420 0x7fc98010e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9800fdbb0 0x7fc98010e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9800fe510 0x7fc980108300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9801087f0 con 0x7fc9800fe510 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc980108960 con 0x7fc9800f8420 2024-02-23T00:29:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc9869ff700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc980108b20 con 0x7fc9800fdbb0 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97f7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9800fdbb0 0x7fc98010e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97f7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9800fdbb0 0x7fc98010e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35350/0 (socket says 172.21.15.87:35350) 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97f7fe700 1 -- 172.21.15.87:0/724696235 learned_addr learned my addr 172.21.15.87:0/724696235 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc984f9c700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9800fe510 0x7fc980108300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97ffff700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9800f8420 0x7fc98010e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97f7fe700 1 -- 172.21.15.87:0/724696235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9800f8420 msgr2=0x7fc98010e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97f7fe700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9800f8420 0x7fc98010e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97f7fe700 1 -- 172.21.15.87:0/724696235 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9800fe510 msgr2=0x7fc980108300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97f7fe700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9800fe510 0x7fc980108300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97f7fe700 1 -- 172.21.15.87:0/724696235 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc974005040 con 0x7fc9800fdbb0 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc984f9c700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9800fe510 0x7fc980108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97ffff700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9800f8420 0x7fc98010e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:29:09.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97f7fe700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9800fdbb0 0x7fc98010e650 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc974009eb0 tx=0x7fc97400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97d7fa700 10 client.?.objecter ms_handle_connect 0x7fc9800fdbb0 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.665+0000 7fc97d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.666+0000 7fc97d7fa700 1 -- 172.21.15.87:0/724696235 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97400e070 con 0x7fc9800fdbb0 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.666+0000 7fc97d7fa700 1 -- 172.21.15.87:0/724696235 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc974013da0 con 0x7fc9800fdbb0 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.666+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc980108da0 con 0x7fc9800fdbb0 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.666+0000 7fc97d7fa700 1 -- 172.21.15.87:0/724696235 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974019070 con 0x7fc9800fdbb0 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.666+0000 7fc9869ff700 10 client.24599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.666+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc980114f90 con 0x7fc9800fdbb0 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.666+0000 7fc9869ff700 1 librados: init done 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.667+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98005ee80 con 0x7fc9800fdbb0 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.667+0000 7fc97d7fa700 1 -- 172.21.15.87:0/724696235 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc97401e420 con 0x7fc9800fdbb0 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.667+0000 7fc97d7fa700 1 --2- 172.21.15.87:0/724696235 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc968062c90 0x7fc968065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:09.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.667+0000 7fc97d7fa700 1 -- 172.21.15.87:0/724696235 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(81..81 src has 28..81) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fc97407d5a0 con 0x7fc9800fdbb0 2024-02-23T00:29:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.667+0000 7fc97d7fa700 10 client.24599.objecter ms_dispatch 0x7fc980001550 osd_map(81..81 src has 28..81) v4 2024-02-23T00:29:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.667+0000 7fc97d7fa700 3 client.24599.objecter handle_osd_map got epochs [81,81] > 0 2024-02-23T00:29:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.667+0000 7fc97d7fa700 3 client.24599.objecter handle_osd_map decoding full epoch 81 2024-02-23T00:29:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.667+0000 7fc97ffff700 1 --2- 172.21.15.87:0/724696235 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc968062c90 0x7fc968065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.668+0000 7fc97d7fa700 20 client.24599.objecter dump_active .. 0 homeless 2024-02-23T00:29:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.668+0000 7fc97d7fa700 10 client.24599.objecter ms_handle_connect 0x7fc968062c90 2024-02-23T00:29:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.668+0000 7fc97ffff700 1 --2- 172.21.15.87:0/724696235 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc968062c90 0x7fc968065150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc970009f20 tx=0x7fc97000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:09.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.672+0000 7fc97d7fa700 1 -- 172.21.15.87:0/724696235 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc974051280 con 0x7fc9800fdbb0 2024-02-23T00:29:09.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.835+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc9801097e0 con 0x7fc9800fdbb0 2024-02-23T00:29:09.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.835+0000 7fc97d7fa700 1 -- 172.21.15.87:0/724696235 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc974023020 con 0x7fc9800fdbb0 2024-02-23T00:29:09.838 INFO:teuthology.orchestra.run.smithi087.stdout:111669149741 2024-02-23T00:29:09.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 10 librados: watch_flush enter 2024-02-23T00:29:09.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 10 librados: watch_flush exit 2024-02-23T00:29:09.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 20 client.24599.objecter shutdown clearing up homeless session... 2024-02-23T00:29:09.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 10 client.24599.objecter successfully canceled tick 2024-02-23T00:29:09.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc968062c90 msgr2=0x7fc968065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 --2- 172.21.15.87:0/724696235 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc968062c90 0x7fc968065150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc970009f20 tx=0x7fc97000b040).stop 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9800fdbb0 msgr2=0x7fc98010e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9800fdbb0 0x7fc98010e650 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc974009eb0 tx=0x7fc97400b040).stop 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 shutdown_connections 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 --2- 172.21.15.87:0/724696235 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc968062c90 0x7fc968065150 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc9800fe510 0x7fc980108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc9800f8420 0x7fc98010e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 --2- 172.21.15.87:0/724696235 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc9800fdbb0 0x7fc98010e650 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 >> 172.21.15.87:0/724696235 conn(0x7fc9800f3800 msgr2=0x7fc9800fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 shutdown_connections 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 -- 172.21.15.87:0/724696235 wait complete. 2024-02-23T00:29:09.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:09.837+0000 7fc9869ff700 1 librados: shutdown 2024-02-23T00:29:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:09.832611+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:29:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:09.834736+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:29:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:09.836971+0000 mon.c (mon.2) 21 : audit [DBG] from='client.? 172.21.15.87:0/724696235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-23T00:29:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:10.042996+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:29:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:10.043962+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:29:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:09.832611+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:29:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:09.834736+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:29:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:09.836971+0000 mon.c (mon.2) 21 : audit [DBG] from='client.? 172.21.15.87:0/724696235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-23T00:29:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:10.042996+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:29:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:10.043962+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:29:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:09.832611+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:29:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:09.834736+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:29:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:09.836971+0000 mon.c (mon.2) 21 : audit [DBG] from='client.? 172.21.15.87:0/724696235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-23T00:29:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:10.042996+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:29:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:10.043962+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:29:10.643 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149735 got 111669149741 for osd.3 2024-02-23T00:29:10.643 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-23T00:29:10.909 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:10.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:29:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:29:11.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:09.477306+0000 mgr.y (mgr.24437) 77 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:11.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:10.129868+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:29:11.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:10.130724+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:29:11.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:10.573076+0000 osd.1 (osd.1) 43 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:29:11.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:10.573978+0000 osd.1 (osd.1) 44 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:29:11.377 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:10.827446+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:29:11.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:10.830384+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:29:11.378 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:09.477306+0000 mgr.y (mgr.24437) 77 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:11.378 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:10.129868+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:29:11.378 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:10.130724+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:29:11.378 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:10.573076+0000 osd.1 (osd.1) 43 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:29:11.378 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:10.573978+0000 osd.1 (osd.1) 44 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:29:11.378 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:10.827446+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:29:11.378 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:10.830384+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:29:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:09.477306+0000 mgr.y (mgr.24437) 77 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:10.129868+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:29:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:10.130724+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:29:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:10.573076+0000 osd.1 (osd.1) 43 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:29:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:10.573978+0000 osd.1 (osd.1) 44 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:29:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:10.827446+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:29:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:10.830384+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:29:12.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:10.546671+0000 mgr.y (mgr.24437) 78 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:12.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:11.579853+0000 osd.1 (osd.1) 45 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:29:12.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:11.580691+0000 osd.1 (osd.1) 46 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:29:12.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:10.546671+0000 mgr.y (mgr.24437) 78 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:12.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:11.579853+0000 osd.1 (osd.1) 45 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:29:12.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:11.580691+0000 osd.1 (osd.1) 46 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:29:12.546 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:12] "GET /metrics HTTP/1.1" 200 172179 "" "Prometheus/2.33.4" 2024-02-23T00:29:12.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:10.546671+0000 mgr.y (mgr.24437) 78 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:12.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:11.579853+0000 osd.1 (osd.1) 45 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:29:12.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:11.580691+0000 osd.1 (osd.1) 46 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:29:12.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 -- 172.21.15.87:0/2015993993 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f9610 msgr2=0x7ff224101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:12.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 --2- 172.21.15.87:0/2015993993 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f9610 0x7ff224101d20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff218004870 tx=0x7ff218009cb0).stop 2024-02-23T00:29:12.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 -- 172.21.15.87:0/2015993993 shutdown_connections 2024-02-23T00:29:12.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 --2- 172.21.15.87:0/2015993993 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff224102210 0x7ff224104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:12.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 --2- 172.21.15.87:0/2015993993 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2240f8cb0 0x7ff2240f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:12.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 --2- 172.21.15.87:0/2015993993 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f9610 0x7ff224101d20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:12.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 -- 172.21.15.87:0/2015993993 >> 172.21.15.87:0/2015993993 conn(0x7ff2240f2820 msgr2=0x7ff2240f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 -- 172.21.15.87:0/2015993993 shutdown_connections 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 -- 172.21.15.87:0/2015993993 wait complete. 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 librados: starting msgr at 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.819+0000 7ff228ad1700 1 librados: starting objecter 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 Processor -- start 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 -- start start 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 librados: setting wanted keys 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 librados: calling monclient init 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f8cb0 0x7ff2240f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2240f9610 0x7ff2240f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff224102210 0x7ff2240f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff224116670 con 0x7ff224102210 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2241167e0 con 0x7ff2240f9610 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff228ad1700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2241169a0 con 0x7ff2240f8cb0 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff22259c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f8cb0 0x7ff2240f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff22259c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f8cb0 0x7ff2240f6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:46602/0 (socket says 172.21.15.87:46602) 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff22259c700 1 -- 172.21.15.87:0/4214362880 learned_addr learned my addr 172.21.15.87:0/4214362880 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff222d9d700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff224102210 0x7ff2240f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff221d9b700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2240f9610 0x7ff2240f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff22259c700 1 -- 172.21.15.87:0/4214362880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2240f9610 msgr2=0x7ff2240f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:12.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff22259c700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2240f9610 0x7ff2240f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff22259c700 1 -- 172.21.15.87:0/4214362880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff224102210 msgr2=0x7ff2240f7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff22259c700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff224102210 0x7ff2240f7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff22259c700 1 -- 172.21.15.87:0/4214362880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c0097c0 con 0x7ff2240f8cb0 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.820+0000 7ff222d9d700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff224102210 0x7ff2240f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff22259c700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f8cb0 0x7ff2240f6c60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff2140041d0 tx=0x7ff214009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff2137fe700 10 client.?.objecter ms_handle_connect 0x7ff2240f8cb0 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff2137fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff2137fe700 1 -- 172.21.15.87:0/4214362880 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214007b40 con 0x7ff2240f8cb0 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff2137fe700 1 -- 172.21.15.87:0/4214362880 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff21400f040 con 0x7ff2240f8cb0 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff218005040 con 0x7ff2240f8cb0 2024-02-23T00:29:12.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff2137fe700 1 -- 172.21.15.87:0/4214362880 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff214014860 con 0x7ff2240f8cb0 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff228ad1700 10 client.24605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff224116fe0 con 0x7ff2240f8cb0 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.821+0000 7ff228ad1700 1 librados: init done 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.822+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22405fa60 con 0x7ff2240f8cb0 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.822+0000 7ff2137fe700 1 -- 172.21.15.87:0/4214362880 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff214014c50 con 0x7ff2240f8cb0 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.822+0000 7ff2137fe700 1 --2- 172.21.15.87:0/4214362880 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff20c062c90 0x7ff20c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.823+0000 7ff2137fe700 1 -- 172.21.15.87:0/4214362880 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(81..81 src has 28..81) v4 ==== 6172+0+0 (secure 0 0 0) 0x7ff21407c580 con 0x7ff2240f8cb0 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.823+0000 7ff221d9b700 1 --2- 172.21.15.87:0/4214362880 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff20c062c90 0x7ff20c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.823+0000 7ff2137fe700 10 client.24605.objecter ms_dispatch 0x7ff224001550 osd_map(81..81 src has 28..81) v4 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.823+0000 7ff2137fe700 3 client.24605.objecter handle_osd_map got epochs [81,81] > 0 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.823+0000 7ff2137fe700 3 client.24605.objecter handle_osd_map decoding full epoch 81 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.823+0000 7ff2137fe700 20 client.24605.objecter dump_active .. 0 homeless 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.823+0000 7ff2137fe700 10 client.24605.objecter ms_handle_connect 0x7ff20c062c90 2024-02-23T00:29:12.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.823+0000 7ff221d9b700 1 --2- 172.21.15.87:0/4214362880 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff20c062c90 0x7ff20c065150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff218009a00 tx=0x7ff21800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:12.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.827+0000 7ff2137fe700 1 -- 172.21.15.87:0/4214362880 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff2140501f0 con 0x7ff2240f8cb0 2024-02-23T00:29:12.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.993+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff2240f80f0 con 0x7ff2240f8cb0 2024-02-23T00:29:12.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.994+0000 7ff2137fe700 1 -- 172.21.15.87:0/4214362880 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff21404fde0 con 0x7ff2240f8cb0 2024-02-23T00:29:12.995 INFO:teuthology.orchestra.run.smithi087.stdout:133143986217 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.995+0000 7ff228ad1700 10 librados: watch_flush enter 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.995+0000 7ff228ad1700 10 librados: watch_flush exit 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.995+0000 7ff228ad1700 20 client.24605.objecter shutdown clearing up homeless session... 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.995+0000 7ff228ad1700 10 client.24605.objecter successfully canceled tick 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.995+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff20c062c90 msgr2=0x7ff20c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.995+0000 7ff228ad1700 1 --2- 172.21.15.87:0/4214362880 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff20c062c90 0x7ff20c065150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff218009a00 tx=0x7ff21800b040).stop 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.995+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f8cb0 msgr2=0x7ff2240f6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.995+0000 7ff228ad1700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f8cb0 0x7ff2240f6c60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff2140041d0 tx=0x7ff214009f60).stop 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.996+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 shutdown_connections 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.996+0000 7ff228ad1700 1 --2- 172.21.15.87:0/4214362880 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff20c062c90 0x7ff20c065150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.996+0000 7ff228ad1700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff224102210 0x7ff2240f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.996+0000 7ff228ad1700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2240f8cb0 0x7ff2240f6c60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:12.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.996+0000 7ff228ad1700 1 --2- 172.21.15.87:0/4214362880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2240f9610 0x7ff2240f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:12.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.996+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 >> 172.21.15.87:0/4214362880 conn(0x7ff2240f2820 msgr2=0x7ff2240f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:12.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.996+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 shutdown_connections 2024-02-23T00:29:12.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.996+0000 7ff228ad1700 1 -- 172.21.15.87:0/4214362880 wait complete. 2024-02-23T00:29:12.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:12.996+0000 7ff228ad1700 1 librados: shutdown 2024-02-23T00:29:13.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:11.477891+0000 mgr.y (mgr.24437) 79 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:13.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:12.994277+0000 mon.c (mon.2) 22 : audit [DBG] from='client.? 172.21.15.87:0/4214362880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-23T00:29:13.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:11.477891+0000 mgr.y (mgr.24437) 79 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:13.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:12.994277+0000 mon.c (mon.2) 22 : audit [DBG] from='client.? 172.21.15.87:0/4214362880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-23T00:29:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:11.477891+0000 mgr.y (mgr.24437) 79 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:12.994277+0000 mon.c (mon.2) 22 : audit [DBG] from='client.? 172.21.15.87:0/4214362880' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-23T00:29:13.841 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986211 got 133143986217 for osd.4 2024-02-23T00:29:13.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-23T00:29:14.110 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:14.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:29:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:29:15.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:13.478231+0000 mgr.y (mgr.24437) 80 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:15.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:14.085136+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.16 scrub starts 2024-02-23T00:29:15.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:14.086289+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.16 scrub ok 2024-02-23T00:29:15.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:13.478231+0000 mgr.y (mgr.24437) 80 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:15.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:14.085136+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.16 scrub starts 2024-02-23T00:29:15.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:14.086289+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.16 scrub ok 2024-02-23T00:29:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:13.478231+0000 mgr.y (mgr.24437) 80 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:14.085136+0000 osd.0 (osd.0) 39 : cluster [DBG] 6.16 scrub starts 2024-02-23T00:29:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:14.086289+0000 osd.0 (osd.0) 40 : cluster [DBG] 6.16 scrub ok 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.893+0000 7f9ea3890700 1 -- 172.21.15.87:0/1712780043 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e9c0fe510 msgr2=0x7f9e9c0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.893+0000 7f9ea3890700 1 --2- 172.21.15.87:0/1712780043 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e9c0fe510 0x7f9e9c0f7d40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f9e98004870 tx=0x7f9e98009cb0).stop 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.893+0000 7f9ea3890700 1 -- 172.21.15.87:0/1712780043 shutdown_connections 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.893+0000 7f9ea3890700 1 --2- 172.21.15.87:0/1712780043 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9c0f8420 0x7f9e9c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.893+0000 7f9ea3890700 1 --2- 172.21.15.87:0/1712780043 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e9c0fdbb0 0x7f9e9c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.893+0000 7f9ea3890700 1 --2- 172.21.15.87:0/1712780043 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e9c0fe510 0x7f9e9c0f7d40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.893+0000 7f9ea3890700 1 -- 172.21.15.87:0/1712780043 >> 172.21.15.87:0/1712780043 conn(0x7f9e9c0f3800 msgr2=0x7f9e9c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.893+0000 7f9ea3890700 1 -- 172.21.15.87:0/1712780043 shutdown_connections 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.893+0000 7f9ea3890700 1 -- 172.21.15.87:0/1712780043 wait complete. 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 librados: starting msgr at 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 librados: starting objecter 2024-02-23T00:29:15.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 Processor -- start 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 -- start start 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 librados: setting wanted keys 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 librados: calling monclient init 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9c0f8420 0x7f9e9c10e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e9c0fdbb0 0x7f9e9c10e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e9c0fe510 0x7f9e9c1082f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c1087e0 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c108950 con 0x7f9e9c0f8420 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea3890700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9c108b10 con 0x7f9e9c0fe510 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea0e2b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e9c0fdbb0 0x7f9e9c10e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea1e2d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e9c0fe510 0x7f9e9c1082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea0e2b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e9c0fdbb0 0x7f9e9c10e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53552/0 (socket says 172.21.15.87:53552) 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.894+0000 7f9ea0e2b700 1 -- 172.21.15.87:0/3616758982 learned_addr learned my addr 172.21.15.87:0/3616758982 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:15.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea162c700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9c0f8420 0x7f9e9c10e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea0e2b700 1 -- 172.21.15.87:0/3616758982 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e9c0fe510 msgr2=0x7f9e9c1082f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea0e2b700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e9c0fe510 0x7f9e9c1082f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea0e2b700 1 -- 172.21.15.87:0/3616758982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9c0f8420 msgr2=0x7f9e9c10e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea0e2b700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9c0f8420 0x7f9e9c10e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea0e2b700 1 -- 172.21.15.87:0/3616758982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e90005040 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea1e2d700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e9c0fe510 0x7f9e9c1082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea162c700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9c0f8420 0x7f9e9c10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea0e2b700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e9c0fdbb0 0x7f9e9c10e640 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f9e98009eb0 tx=0x7f9e9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9e8e7fc700 10 client.?.objecter ms_handle_connect 0x7f9e9c0fdbb0 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9e8e7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9e8e7fc700 1 -- 172.21.15.87:0/3616758982 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9800e070 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e98005040 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9e8e7fc700 1 -- 172.21.15.87:0/3616758982 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e98013da0 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9e8e7fc700 1 -- 172.21.15.87:0/3616758982 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98019070 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea3890700 10 client.14685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c109000 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.895+0000 7f9ea3890700 1 librados: init done 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.896+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9c05ee80 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.896+0000 7f9e8e7fc700 1 -- 172.21.15.87:0/3616758982 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f9e9801e420 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.897+0000 7f9e8e7fc700 1 --2- 172.21.15.87:0/3616758982 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e88062c90 0x7f9e88065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.897+0000 7f9e8e7fc700 1 -- 172.21.15.87:0/3616758982 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(82..82 src has 28..82) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f9e9807d5a0 con 0x7f9e9c0fdbb0 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.897+0000 7f9e8e7fc700 10 client.14685.objecter ms_dispatch 0x7f9e9c001550 osd_map(82..82 src has 28..82) v4 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.897+0000 7f9e8e7fc700 3 client.14685.objecter handle_osd_map got epochs [82,82] > 0 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.897+0000 7f9e8e7fc700 3 client.14685.objecter handle_osd_map decoding full epoch 82 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.897+0000 7f9ea162c700 1 --2- 172.21.15.87:0/3616758982 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e88062c90 0x7f9e88065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.897+0000 7f9e8e7fc700 20 client.14685.objecter dump_active .. 0 homeless 2024-02-23T00:29:15.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.897+0000 7f9e8e7fc700 10 client.14685.objecter ms_handle_connect 0x7f9e88062c90 2024-02-23T00:29:15.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.897+0000 7f9ea162c700 1 --2- 172.21.15.87:0/3616758982 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e88062c90 0x7f9e88065150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f9e90007fd0 tx=0x7f9e90009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:15.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:15.901+0000 7f9e8e7fc700 1 -- 172.21.15.87:0/3616758982 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9e98051280 con 0x7f9e9c0fdbb0 2024-02-23T00:29:16.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.063+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9e9c1098a0 con 0x7f9e9c0fdbb0 2024-02-23T00:29:16.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.064+0000 7f9e8e7fc700 1 -- 172.21.15.87:0/3616758982 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9e98023020 con 0x7f9e9c0fdbb0 2024-02-23T00:29:16.065 INFO:teuthology.orchestra.run.smithi087.stdout:163208757284 2024-02-23T00:29:16.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 10 librados: watch_flush enter 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 10 librados: watch_flush exit 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 20 client.14685.objecter shutdown clearing up homeless session... 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 10 client.14685.objecter successfully canceled tick 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e88062c90 msgr2=0x7f9e88065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 --2- 172.21.15.87:0/3616758982 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e88062c90 0x7f9e88065150 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f9e90007fd0 tx=0x7f9e90009cb0).stop 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e9c0fdbb0 msgr2=0x7f9e9c10e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e9c0fdbb0 0x7f9e9c10e640 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f9e98009eb0 tx=0x7f9e9800b040).stop 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 shutdown_connections 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 --2- 172.21.15.87:0/3616758982 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e88062c90 0x7f9e88065150 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e9c0fe510 0x7f9e9c1082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e9c0f8420 0x7f9e9c10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 --2- 172.21.15.87:0/3616758982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e9c0fdbb0 0x7f9e9c10e640 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:16.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.065+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 >> 172.21.15.87:0/3616758982 conn(0x7f9e9c0f3800 msgr2=0x7f9e9c0fb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:16.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.066+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 shutdown_connections 2024-02-23T00:29:16.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.066+0000 7f9ea3890700 1 -- 172.21.15.87:0/3616758982 wait complete. 2024-02-23T00:29:16.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:16.066+0000 7f9ea3890700 1 librados: shutdown 2024-02-23T00:29:16.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:14.462556+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:16.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:14.462617+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:16.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:15.137236+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-23T00:29:16.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:16.064436+0000 mon.a (mon.0) 730 : audit [DBG] from='client.? 172.21.15.87:0/3616758982' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-23T00:29:16.405 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:14.462556+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:16.405 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:14.462617+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:16.405 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:15.137236+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-23T00:29:16.405 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:16.064436+0000 mon.a (mon.0) 730 : audit [DBG] from='client.? 172.21.15.87:0/3616758982' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-23T00:29:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:14.462556+0000 osd.0 (osd.0) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:14.462617+0000 osd.0 (osd.0) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:15.137236+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-23T00:29:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:16.064436+0000 mon.a (mon.0) 730 : audit [DBG] from='client.? 172.21.15.87:0/3616758982' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-23T00:29:16.831 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757279 got 163208757284 for osd.5 2024-02-23T00:29:16.832 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-23T00:29:17.100 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:17.401 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:15.478780+0000 mgr.y (mgr.24437) 81 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:17.401 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:15.478780+0000 mgr.y (mgr.24437) 81 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:15.478780+0000 mgr.y (mgr.24437) 81 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:19.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 -- 172.21.15.87:0/205206602 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2d80f8d10 msgr2=0x7fd2d80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:19.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 --2- 172.21.15.87:0/205206602 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2d80f8d10 0x7fd2d80f90f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fd2c80047b0 tx=0x7fd2c8009cb0).stop 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 -- 172.21.15.87:0/205206602 shutdown_connections 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 --2- 172.21.15.87:0/205206602 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2d8102270 0x7fd2d8104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 --2- 172.21.15.87:0/205206602 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2d80f8d10 0x7fd2d80f90f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 --2- 172.21.15.87:0/205206602 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2d80f9670 0x7fd2d8101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 -- 172.21.15.87:0/205206602 >> 172.21.15.87:0/205206602 conn(0x7fd2d80f2820 msgr2=0x7fd2d80f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 -- 172.21.15.87:0/205206602 shutdown_connections 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 -- 172.21.15.87:0/205206602 wait complete. 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 librados: starting msgr at 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 librados: starting objecter 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 Processor -- start 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 -- start start 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 librados: setting wanted keys 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.006+0000 7fd2e0023700 1 librados: calling monclient init 2024-02-23T00:29:19.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2d80f8d10 0x7fd2d80f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2d80f9670 0x7fd2d80f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2d8102270 0x7fd2d80f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d81167c0 con 0x7fd2d80f9670 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d8116930 con 0x7fd2d8102270 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d8116af0 con 0x7fd2d80f8d10 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dddbf700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2d80f8d10 0x7fd2d80f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dd5be700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2d80f9670 0x7fd2d80f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dddbf700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2d80f8d10 0x7fd2d80f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:46648/0 (socket says 172.21.15.87:46648) 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dddbf700 1 -- 172.21.15.87:0/3957235419 learned_addr learned my addr 172.21.15.87:0/3957235419 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2de5c0700 1 --2- 172.21.15.87:0/3957235419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2d8102270 0x7fd2d80f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dddbf700 1 -- 172.21.15.87:0/3957235419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2d8102270 msgr2=0x7fd2d80f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dddbf700 1 --2- 172.21.15.87:0/3957235419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2d8102270 0x7fd2d80f76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dddbf700 1 -- 172.21.15.87:0/3957235419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2d80f9670 msgr2=0x7fd2d80f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dddbf700 1 --2- 172.21.15.87:0/3957235419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2d80f9670 0x7fd2d80f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dddbf700 1 -- 172.21.15.87:0/3957235419 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d4005040 con 0x7fd2d80f8d10 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dd5be700 1 --2- 172.21.15.87:0/3957235419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2d80f9670 0x7fd2d80f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2dddbf700 1 --2- 172.21.15.87:0/3957235419 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2d80f8d10 0x7fd2d80f6cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd2c8009b10 tx=0x7fd2c8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2ceffd700 10 client.?.objecter ms_handle_connect 0x7fd2d80f8d10 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2ceffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2ceffd700 1 -- 172.21.15.87:0/3957235419 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8018070 con 0x7fd2d80f8d10 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2ceffd700 1 -- 172.21.15.87:0/3957235419 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2c8013e70 con 0x7fd2d80f8d10 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c8005040 con 0x7fd2d80f8d10 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2ceffd700 1 -- 172.21.15.87:0/3957235419 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c801ddb0 con 0x7fd2d80f8d10 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 10 client.24617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d8116f90 con 0x7fd2d80f8d10 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.007+0000 7fd2e0023700 1 librados: init done 2024-02-23T00:29:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.008+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d805fa60 con 0x7fd2d80f8d10 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2ceffd700 1 -- 172.21.15.87:0/3957235419 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd2c801d420 con 0x7fd2d80f8d10 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2ceffd700 1 --2- 172.21.15.87:0/3957235419 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2c4062c90 0x7fd2c4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2ceffd700 1 -- 172.21.15.87:0/3957235419 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(82..82 src has 28..82) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fd2c807ded0 con 0x7fd2d80f8d10 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2ceffd700 10 client.24617.objecter ms_dispatch 0x7fd2d8001550 osd_map(82..82 src has 28..82) v4 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2ceffd700 3 client.24617.objecter handle_osd_map got epochs [82,82] > 0 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2ceffd700 3 client.24617.objecter handle_osd_map decoding full epoch 82 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2dd5be700 1 --2- 172.21.15.87:0/3957235419 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2c4062c90 0x7fd2c4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2ceffd700 20 client.24617.objecter dump_active .. 0 homeless 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2ceffd700 10 client.24617.objecter ms_handle_connect 0x7fd2c4062c90 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.009+0000 7fd2dd5be700 1 --2- 172.21.15.87:0/3957235419 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2c4062c90 0x7fd2c4065150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fd2d80f8460 tx=0x7fd2d400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:19.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.013+0000 7fd2ceffd700 1 -- 172.21.15.87:0/3957235419 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd2c8051b40 con 0x7fd2d80f8d10 2024-02-23T00:29:19.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.181+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd2d80f81a0 con 0x7fd2d80f8d10 2024-02-23T00:29:19.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.181+0000 7fd2ceffd700 1 -- 172.21.15.87:0/3957235419 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd2c8051730 con 0x7fd2d80f8d10 2024-02-23T00:29:19.183 INFO:teuthology.orchestra.run.smithi087.stdout:188978561056 2024-02-23T00:29:19.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 10 librados: watch_flush enter 2024-02-23T00:29:19.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 10 librados: watch_flush exit 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 20 client.24617.objecter shutdown clearing up homeless session... 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 10 client.24617.objecter successfully canceled tick 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2c4062c90 msgr2=0x7fd2c4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 --2- 172.21.15.87:0/3957235419 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2c4062c90 0x7fd2c4065150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fd2d80f8460 tx=0x7fd2d400b040).stop 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2d80f8d10 msgr2=0x7fd2d80f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 --2- 172.21.15.87:0/3957235419 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2d80f8d10 0x7fd2d80f6cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd2c8009b10 tx=0x7fd2c8009cb0).stop 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 shutdown_connections 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 --2- 172.21.15.87:0/3957235419 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2c4062c90 0x7fd2c4065150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 --2- 172.21.15.87:0/3957235419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2d8102270 0x7fd2d80f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:19.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 --2- 172.21.15.87:0/3957235419 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2d80f8d10 0x7fd2d80f6cc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:19.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 --2- 172.21.15.87:0/3957235419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2d80f9670 0x7fd2d80f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:19.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 >> 172.21.15.87:0/3957235419 conn(0x7fd2d80f2820 msgr2=0x7fd2d80f3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:19.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 shutdown_connections 2024-02-23T00:29:19.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 -- 172.21.15.87:0/3957235419 wait complete. 2024-02-23T00:29:19.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:19.183+0000 7fd2e0023700 1 librados: shutdown 2024-02-23T00:29:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:17.479086+0000 mgr.y (mgr.24437) 82 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:19.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:17.479086+0000 mgr.y (mgr.24437) 82 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:17.479086+0000 mgr.y (mgr.24437) 82 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:20.004 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561051 got 188978561056 for osd.6 2024-02-23T00:29:20.004 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-23T00:29:20.271 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:20.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:19.181675+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? 172.21.15.87:0/3957235419' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-23T00:29:20.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:19.181675+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? 172.21.15.87:0/3957235419' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-23T00:29:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:19.181675+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? 172.21.15.87:0/3957235419' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-23T00:29:20.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:29:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:29:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:19.479421+0000 mgr.y (mgr.24437) 83 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:21.558 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:19.479421+0000 mgr.y (mgr.24437) 83 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:21.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:19.479421+0000 mgr.y (mgr.24437) 83 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 -- 172.21.15.87:0/3377375634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00fd510 msgr2=0x7f60f00f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/3377375634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00fd510 0x7f60f00f6d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f60dc004870 tx=0x7f60dc009cb0).stop 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 -- 172.21.15.87:0/3377375634 shutdown_connections 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/3377375634 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f00f7400 0x7f60f0103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/3377375634 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60f00fcbb0 0x7f60f00fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/3377375634 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00fd510 0x7f60f00f6d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 -- 172.21.15.87:0/3377375634 >> 172.21.15.87:0/3377375634 conn(0x7f60f00f2820 msgr2=0x7f60f00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 -- 172.21.15.87:0/3377375634 shutdown_connections 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 -- 172.21.15.87:0/3377375634 wait complete. 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 librados: starting msgr at 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.063+0000 7f60f6fa7700 1 librados: starting objecter 2024-02-23T00:29:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 Processor -- start 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 -- start start 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 librados: setting wanted keys 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 librados: calling monclient init 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00f7400 0x7f60f0109320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f00fcbb0 0x7f60f0109810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60f00fd510 0x7f60f0109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f0114510 con 0x7f60f00f7400 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f0114680 con 0x7f60f00fcbb0 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f6fa7700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f0114840 con 0x7f60f00fd510 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f4d43700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00f7400 0x7f60f0109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f5544700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60f00fd510 0x7f60f0109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f4d43700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00f7400 0x7f60f0109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35432/0 (socket says 172.21.15.87:35432) 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f5544700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60f00fd510 0x7f60f0109dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34366/0 (socket says 172.21.15.87:34366) 2024-02-23T00:29:22.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f4d43700 1 -- 172.21.15.87:0/1794026426 learned_addr learned my addr 172.21.15.87:0/1794026426 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60effff700 1 --2- 172.21.15.87:0/1794026426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f00fcbb0 0x7f60f0109810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f4d43700 1 -- 172.21.15.87:0/1794026426 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60f00fd510 msgr2=0x7f60f0109dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f4d43700 1 --2- 172.21.15.87:0/1794026426 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60f00fd510 0x7f60f0109dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f4d43700 1 -- 172.21.15.87:0/1794026426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f00fcbb0 msgr2=0x7f60f0109810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f4d43700 1 --2- 172.21.15.87:0/1794026426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f00fcbb0 0x7f60f0109810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f4d43700 1 -- 172.21.15.87:0/1794026426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc005040 con 0x7f60f00f7400 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60f4d43700 1 --2- 172.21.15.87:0/1794026426 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00f7400 0x7f60f0109320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f60e400e970 tx=0x7f60e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60edffb700 10 client.?.objecter ms_handle_connect 0x7f60f00f7400 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60edffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.064+0000 7f60edffb700 1 -- 172.21.15.87:0/1794026426 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e4004660 con 0x7f60f00f7400 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.065+0000 7f60edffb700 1 -- 172.21.15.87:0/1794026426 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60e400f040 con 0x7f60f00f7400 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.065+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f0114b20 con 0x7f60f00f7400 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.065+0000 7f60edffb700 1 -- 172.21.15.87:0/1794026426 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e4014790 con 0x7f60f00f7400 2024-02-23T00:29:22.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.065+0000 7f60f6fa7700 10 client.14697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:22.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.065+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f0115060 con 0x7f60f00f7400 2024-02-23T00:29:22.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.065+0000 7f60f6fa7700 1 librados: init done 2024-02-23T00:29:22.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.066+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f005fa60 con 0x7f60f00f7400 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.066+0000 7f60edffb700 1 -- 172.21.15.87:0/1794026426 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f60e40090d0 con 0x7f60f00f7400 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.066+0000 7f60edffb700 1 --2- 172.21.15.87:0/1794026426 >> v2:172.21.15.87:6800/1231597463 conn(0x7f60e0062c90 0x7f60e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.066+0000 7f60edffb700 1 -- 172.21.15.87:0/1794026426 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(82..82 src has 28..82) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f60e407c570 con 0x7f60f00f7400 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.066+0000 7f60edffb700 10 client.14697.objecter ms_dispatch 0x7f60f0001550 osd_map(82..82 src has 28..82) v4 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.066+0000 7f60edffb700 3 client.14697.objecter handle_osd_map got epochs [82,82] > 0 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.066+0000 7f60edffb700 3 client.14697.objecter handle_osd_map decoding full epoch 82 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.066+0000 7f60effff700 1 --2- 172.21.15.87:0/1794026426 >> v2:172.21.15.87:6800/1231597463 conn(0x7f60e0062c90 0x7f60e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.066+0000 7f60edffb700 20 client.14697.objecter dump_active .. 0 homeless 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.067+0000 7f60edffb700 10 client.14697.objecter ms_handle_connect 0x7f60e0062c90 2024-02-23T00:29:22.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.067+0000 7f60effff700 1 --2- 172.21.15.87:0/1794026426 >> v2:172.21.15.87:6800/1231597463 conn(0x7f60e0062c90 0x7f60e0065150 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f60dc004ee0 tx=0x7f60dc009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:22.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.070+0000 7f60edffb700 1 -- 172.21.15.87:0/1794026426 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f60e404af60 con 0x7f60f00f7400 2024-02-23T00:29:22.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.241+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f60f0061460 con 0x7f60f00f7400 2024-02-23T00:29:22.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.241+0000 7f60edffb700 1 -- 172.21.15.87:0/1794026426 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f60e4019090 con 0x7f60f00f7400 2024-02-23T00:29:22.242 INFO:teuthology.orchestra.run.smithi087.stdout:219043332124 2024-02-23T00:29:22.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 10 librados: watch_flush enter 2024-02-23T00:29:22.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 10 librados: watch_flush exit 2024-02-23T00:29:22.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 20 client.14697.objecter shutdown clearing up homeless session... 2024-02-23T00:29:22.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 10 client.14697.objecter successfully canceled tick 2024-02-23T00:29:22.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 >> v2:172.21.15.87:6800/1231597463 conn(0x7f60e0062c90 msgr2=0x7f60e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:22.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/1794026426 >> v2:172.21.15.87:6800/1231597463 conn(0x7f60e0062c90 0x7f60e0065150 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f60dc004ee0 tx=0x7f60dc009cb0).stop 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00f7400 msgr2=0x7f60f0109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/1794026426 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00f7400 0x7f60f0109320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f60e400e970 tx=0x7f60e400b040).stop 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 shutdown_connections 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/1794026426 >> v2:172.21.15.87:6800/1231597463 conn(0x7f60e0062c90 0x7f60e0065150 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/1794026426 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60f00fd510 0x7f60f0109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/1794026426 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60f00f7400 0x7f60f0109320 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 --2- 172.21.15.87:0/1794026426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60f00fcbb0 0x7f60f0109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 >> 172.21.15.87:0/1794026426 conn(0x7f60f00f2820 msgr2=0x7f60f00f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 shutdown_connections 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 -- 172.21.15.87:0/1794026426 wait complete. 2024-02-23T00:29:22.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:22.243+0000 7f60f6fa7700 1 librados: shutdown 2024-02-23T00:29:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:20.550598+0000 mgr.y (mgr.24437) 84 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:22.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:20.550598+0000 mgr.y (mgr.24437) 84 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:20.550598+0000 mgr.y (mgr.24437) 84 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:22.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:22] "GET /metrics HTTP/1.1" 200 172179 "" "Prometheus/2.33.4" 2024-02-23T00:29:23.029 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332118 got 219043332124 for osd.7 2024-02-23T00:29:23.029 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-23T00:29:23.029 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:29:23.300 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:23.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:21.480003+0000 mgr.y (mgr.24437) 85 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:23.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:22.241769+0000 mon.a (mon.0) 731 : audit [DBG] from='client.? 172.21.15.87:0/1794026426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-23T00:29:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:21.480003+0000 mgr.y (mgr.24437) 85 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:22.241769+0000 mon.a (mon.0) 731 : audit [DBG] from='client.? 172.21.15.87:0/1794026426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-23T00:29:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:21.480003+0000 mgr.y (mgr.24437) 85 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:22.241769+0000 mon.a (mon.0) 731 : audit [DBG] from='client.? 172.21.15.87:0/1794026426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-23T00:29:24.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:29:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 -- 172.21.15.87:0/1783984316 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40fd4a0 msgr2=0x7f23a40f74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 --2- 172.21.15.87:0/1783984316 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40fd4a0 0x7f23a40f74c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f23940047b0 tx=0x7f2394009cb0).stop 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 -- 172.21.15.87:0/1783984316 shutdown_connections 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 --2- 172.21.15.87:0/1783984316 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23a40f7ba0 0x7f23a4103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 --2- 172.21.15.87:0/1783984316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23a40fcb40 0x7f23a40fcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 --2- 172.21.15.87:0/1783984316 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40fd4a0 0x7f23a40f74c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 -- 172.21.15.87:0/1783984316 >> 172.21.15.87:0/1783984316 conn(0x7f23a406c370 msgr2=0x7f23a406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 -- 172.21.15.87:0/1783984316 shutdown_connections 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 -- 172.21.15.87:0/1783984316 wait complete. 2024-02-23T00:29:25.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 librados: starting msgr at 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 librados: starting objecter 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 Processor -- start 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 -- start start 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 librados: setting wanted keys 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.353+0000 7f23aa504700 1 librados: calling monclient init 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23aa504700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40f7ba0 0x7f23a410e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23aa504700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23a40fcb40 0x7f23a410e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23aa504700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23a40fd4a0 0x7f23a4108320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23aa504700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a4108810 con 0x7f23a40fd4a0 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23aa504700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a4108980 con 0x7f23a40fcb40 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23aa504700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a4108b40 con 0x7f23a40f7ba0 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a3fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40f7ba0 0x7f23a410e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:25.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a8aa1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23a40fd4a0 0x7f23a4108320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a3fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40f7ba0 0x7f23a410e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34386/0 (socket says 172.21.15.87:34386) 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a8aa1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23a40fd4a0 0x7f23a4108320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35440/0 (socket says 172.21.15.87:35440) 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a3fff700 1 -- 172.21.15.87:0/1916817404 learned_addr learned my addr 172.21.15.87:0/1916817404 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a37fe700 1 --2- 172.21.15.87:0/1916817404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23a40fcb40 0x7f23a410e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a3fff700 1 -- 172.21.15.87:0/1916817404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23a40fcb40 msgr2=0x7f23a410e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a3fff700 1 --2- 172.21.15.87:0/1916817404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23a40fcb40 0x7f23a410e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a3fff700 1 -- 172.21.15.87:0/1916817404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23a40fd4a0 msgr2=0x7f23a4108320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a3fff700 1 --2- 172.21.15.87:0/1916817404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23a40fd4a0 0x7f23a4108320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a3fff700 1 -- 172.21.15.87:0/1916817404 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c0097c0 con 0x7f23a40f7ba0 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a8aa1700 1 --2- 172.21.15.87:0/1916817404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23a40fd4a0 0x7f23a4108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a3fff700 1 --2- 172.21.15.87:0/1916817404 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40f7ba0 0x7f23a410e180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f239800c810 tx=0x7f239800cb20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a17fa700 10 client.?.objecter ms_handle_connect 0x7f23a40f7ba0 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.354+0000 7f23a17fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:25.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.355+0000 7f23a17fa700 1 -- 172.21.15.87:0/1916817404 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239800e4a0 con 0x7f23a40f7ba0 2024-02-23T00:29:25.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.355+0000 7f23a17fa700 1 -- 172.21.15.87:0/1916817404 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f239800ead0 con 0x7f23a40f7ba0 2024-02-23T00:29:25.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.355+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2394005040 con 0x7f23a40f7ba0 2024-02-23T00:29:25.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.355+0000 7f23a17fa700 1 -- 172.21.15.87:0/1916817404 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398012a70 con 0x7f23a40f7ba0 2024-02-23T00:29:25.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.355+0000 7f23aa504700 10 client.24629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:25.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.355+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a4109180 con 0x7f23a40f7ba0 2024-02-23T00:29:25.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.355+0000 7f23aa504700 1 librados: init done 2024-02-23T00:29:25.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.356+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a405f950 con 0x7f23a40f7ba0 2024-02-23T00:29:25.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.356+0000 7f23a17fa700 1 -- 172.21.15.87:0/1916817404 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f239801b3b0 con 0x7f23a40f7ba0 2024-02-23T00:29:25.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.356+0000 7f23a17fa700 1 --2- 172.21.15.87:0/1916817404 >> v2:172.21.15.87:6800/1231597463 conn(0x7f238c062c90 0x7f238c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:25.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.356+0000 7f23a17fa700 1 -- 172.21.15.87:0/1916817404 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(82..82 src has 28..82) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f239807c1a0 con 0x7f23a40f7ba0 2024-02-23T00:29:25.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.356+0000 7f23a17fa700 10 client.24629.objecter ms_dispatch 0x7f23a4001550 osd_map(82..82 src has 28..82) v4 2024-02-23T00:29:25.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.356+0000 7f23a17fa700 3 client.24629.objecter handle_osd_map got epochs [82,82] > 0 2024-02-23T00:29:25.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.356+0000 7f23a17fa700 3 client.24629.objecter handle_osd_map decoding full epoch 82 2024-02-23T00:29:25.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.356+0000 7f23a37fe700 1 --2- 172.21.15.87:0/1916817404 >> v2:172.21.15.87:6800/1231597463 conn(0x7f238c062c90 0x7f238c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:25.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.357+0000 7f23a17fa700 20 client.24629.objecter dump_active .. 0 homeless 2024-02-23T00:29:25.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.357+0000 7f23a17fa700 10 client.24629.objecter ms_handle_connect 0x7f238c062c90 2024-02-23T00:29:25.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.357+0000 7f23a37fe700 1 --2- 172.21.15.87:0/1916817404 >> v2:172.21.15.87:6800/1231597463 conn(0x7f238c062c90 0x7f238c065150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f23a4109ac0 tx=0x7f2394009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:25.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.361+0000 7f23a17fa700 1 -- 172.21.15.87:0/1916817404 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2398048f60 con 0x7f23a40f7ba0 2024-02-23T00:29:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:23.480365+0000 mgr.y (mgr.24437) 86 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:25.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.520+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23a4109800 con 0x7f238c062c90 2024-02-23T00:29:25.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.530+0000 7f23a17fa700 1 -- 172.21.15.87:0/1916817404 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357299 (secure 0 0 0) 0x7f23a4109800 con 0x7f238c062c90 2024-02-23T00:29:25.536 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:29:25.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 10 librados: watch_flush enter 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 10 librados: watch_flush exit 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 20 client.24629.objecter shutdown clearing up homeless session... 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 10 client.24629.objecter successfully canceled tick 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 >> v2:172.21.15.87:6800/1231597463 conn(0x7f238c062c90 msgr2=0x7f238c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 --2- 172.21.15.87:0/1916817404 >> v2:172.21.15.87:6800/1231597463 conn(0x7f238c062c90 0x7f238c065150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f23a4109ac0 tx=0x7f2394009cb0).stop 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40f7ba0 msgr2=0x7f23a410e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 --2- 172.21.15.87:0/1916817404 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40f7ba0 0x7f23a410e180 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f239800c810 tx=0x7f239800cb20).stop 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 shutdown_connections 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 --2- 172.21.15.87:0/1916817404 >> v2:172.21.15.87:6800/1231597463 conn(0x7f238c062c90 0x7f238c065150 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 --2- 172.21.15.87:0/1916817404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f23a40fd4a0 0x7f23a4108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 --2- 172.21.15.87:0/1916817404 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f23a40f7ba0 0x7f23a410e180 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 --2- 172.21.15.87:0/1916817404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f23a40fcb40 0x7f23a410e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:25.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 >> 172.21.15.87:0/1916817404 conn(0x7f23a406c370 msgr2=0x7f23a40fb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:25.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 shutdown_connections 2024-02-23T00:29:25.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 -- 172.21.15.87:0/1916817404 wait complete. 2024-02-23T00:29:25.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:25.533+0000 7f23aa504700 1 librados: shutdown 2024-02-23T00:29:25.541 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:29:25.554 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:23.480365+0000 mgr.y (mgr.24437) 86 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:25.554 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:23.480365+0000 mgr.y (mgr.24437) 86 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:26.234 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-02-23T00:29:25.480531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1226,"num_read_kb":946,"num_write":620,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":694,"ondisk_log_size":694,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381744,"kb_used_data":4976,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579472,"statfs":{"total":767960285184,"available":765521379328,"internally_reserved":0,"allocated":5095424,"data_stored":2321961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.345804"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165705+0000","last_change":"2024-02-23T00:28:52.093263+0000","last_active":"2024-02-23T00:29:04.165705+0000","last_peered":"2024-02-23T00:29:04.165705+0000","last_clean":"2024-02-23T00:29:04.165705+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:29:04.165705+0000","last_undegraded":"2024-02-23T00:29:04.165705+0000","last_fullsized":"2024-02-23T00:29:04.165705+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:52.093177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.093177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.109974+0000","last_change":"2024-02-23T00:28:57.678604+0000","last_active":"2024-02-23T00:29:04.109974+0000","last_peered":"2024-02-23T00:29:04.109974+0000","last_clean":"2024-02-23T00:29:04.109974+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:29:04.109974+0000","last_undegraded":"2024-02-23T00:29:04.109974+0000","last_fullsized":"2024-02-23T00:29:04.109974+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:57.678522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.678522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166389+0000","last_change":"2024-02-23T00:28:46.217573+0000","last_active":"2024-02-23T00:29:04.166389+0000","last_peered":"2024-02-23T00:29:04.166389+0000","last_clean":"2024-02-23T00:29:04.166389+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:29:04.166389+0000","last_undegraded":"2024-02-23T00:29:04.166389+0000","last_fullsized":"2024-02-23T00:29:04.166389+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:46.217507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.217507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'39","reported_seq":124,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165323+0000","last_change":"2024-02-23T00:28:37.040298+0000","last_active":"2024-02-23T00:29:04.165323+0000","last_peered":"2024-02-23T00:29:04.165323+0000","last_clean":"2024-02-23T00:29:04.165323+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:29:04.165323+0000","last_undegraded":"2024-02-23T00:29:04.165323+0000","last_fullsized":"2024-02-23T00:29:04.165323+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'39","last_scrub_stamp":"2024-02-23T00:28:37.040215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:37.040215+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110191+0000","last_change":"2024-02-23T00:28:59.685150+0000","last_active":"2024-02-23T00:29:04.110191+0000","last_peered":"2024-02-23T00:29:04.110191+0000","last_clean":"2024-02-23T00:29:04.110191+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:29:04.110191+0000","last_undegraded":"2024-02-23T00:29:04.110191+0000","last_fullsized":"2024-02-23T00:29:04.110191+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:59.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:59.685045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112331+0000","last_change":"2024-02-23T00:28:49.663875+0000","last_active":"2024-02-23T00:29:04.112331+0000","last_peered":"2024-02-23T00:29:04.112331+0000","last_clean":"2024-02-23T00:29:04.112331+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:29:04.112331+0000","last_undegraded":"2024-02-23T00:29:04.112331+0000","last_fullsized":"2024-02-23T00:29:04.112331+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.663808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.663808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166335+0000","last_change":"2024-02-23T00:28:47.073631+0000","last_active":"2024-02-23T00:29:04.166335+0000","last_peered":"2024-02-23T00:29:04.166335+0000","last_clean":"2024-02-23T00:29:04.166335+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:29:04.166335+0000","last_undegraded":"2024-02-23T00:29:04.166335+0000","last_fullsized":"2024-02-23T00:29:04.166335+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:47.073551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.073551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":75,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:06.822930+0000","last_change":"2024-02-23T00:29:06.822899+0000","last_active":"2024-02-23T00:29:06.822930+0000","last_peered":"2024-02-23T00:29:06.822930+0000","last_clean":"2024-02-23T00:29:06.822930+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:29:06.822930+0000","last_undegraded":"2024-02-23T00:29:06.822930+0000","last_fullsized":"2024-02-23T00:29:06.822930+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:29:06.822818+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167297+0000","last_change":"2024-02-23T00:29:04.049502+0000","last_active":"2024-02-23T00:29:04.167297+0000","last_peered":"2024-02-23T00:29:04.167297+0000","last_clean":"2024-02-23T00:29:04.167297+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:29:04.167297+0000","last_undegraded":"2024-02-23T00:29:04.167297+0000","last_fullsized":"2024-02-23T00:29:04.167297+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.049414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166268+0000","last_change":"2024-02-23T00:29:00.320083+0000","last_active":"2024-02-23T00:29:04.166268+0000","last_peered":"2024-02-23T00:29:04.166268+0000","last_clean":"2024-02-23T00:29:04.166268+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:29:04.166268+0000","last_undegraded":"2024-02-23T00:29:04.166268+0000","last_fullsized":"2024-02-23T00:29:04.166268+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.319985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.319985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167512+0000","last_change":"2024-02-23T00:28:46.396553+0000","last_active":"2024-02-23T00:29:04.167512+0000","last_peered":"2024-02-23T00:29:04.167512+0000","last_clean":"2024-02-23T00:29:04.167512+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:29:04.167512+0000","last_undegraded":"2024-02-23T00:29:04.167512+0000","last_fullsized":"2024-02-23T00:29:04.167512+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.396471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110342+0000","last_change":"2024-02-23T00:28:54.657266+0000","last_active":"2024-02-23T00:29:04.110342+0000","last_peered":"2024-02-23T00:29:04.110342+0000","last_clean":"2024-02-23T00:29:04.110342+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:29:04.110342+0000","last_undegraded":"2024-02-23T00:29:04.110342+0000","last_fullsized":"2024-02-23T00:29:04.110342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.657177+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165496+0000","last_change":"2024-02-23T00:28:50.176282+0000","last_active":"2024-02-23T00:29:04.165496+0000","last_peered":"2024-02-23T00:29:04.165496+0000","last_clean":"2024-02-23T00:29:04.165496+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:29:04.165496+0000","last_undegraded":"2024-02-23T00:29:04.165496+0000","last_fullsized":"2024-02-23T00:29:04.165496+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.176208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167187+0000","last_change":"2024-02-23T00:28:33.933658+0000","last_active":"2024-02-23T00:29:04.167187+0000","last_peered":"2024-02-23T00:29:04.167187+0000","last_clean":"2024-02-23T00:29:04.167187+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:29:04.167187+0000","last_undegraded":"2024-02-23T00:29:04.167187+0000","last_fullsized":"2024-02-23T00:29:04.167187+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.933529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.933529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.267535+0000","last_change":"2024-02-23T00:29:04.267507+0000","last_active":"2024-02-23T00:29:04.267535+0000","last_peered":"2024-02-23T00:29:04.267535+0000","last_clean":"2024-02-23T00:29:04.267535+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:29:04.267535+0000","last_undegraded":"2024-02-23T00:29:04.267535+0000","last_fullsized":"2024-02-23T00:29:04.267535+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.267427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.267427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110473+0000","last_change":"2024-02-23T00:28:25.457421+0000","last_active":"2024-02-23T00:29:04.110473+0000","last_peered":"2024-02-23T00:29:04.110473+0000","last_clean":"2024-02-23T00:29:04.110473+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:29:04.110473+0000","last_undegraded":"2024-02-23T00:29:04.110473+0000","last_fullsized":"2024-02-23T00:29:04.110473+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:28:25.457329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:25.457329+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166564+0000","last_change":"2024-02-23T00:28:29.086512+0000","last_active":"2024-02-23T00:29:04.166564+0000","last_peered":"2024-02-23T00:29:04.166564+0000","last_clean":"2024-02-23T00:29:04.166564+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:29:04.166564+0000","last_undegraded":"2024-02-23T00:29:04.166564+0000","last_fullsized":"2024-02-23T00:29:04.166564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:29.086437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:29.086437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110635+0000","last_change":"2024-02-23T00:29:03.790608+0000","last_active":"2024-02-23T00:29:04.110635+0000","last_peered":"2024-02-23T00:29:04.110635+0000","last_clean":"2024-02-23T00:29:04.110635+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:29:04.110635+0000","last_undegraded":"2024-02-23T00:29:04.110635+0000","last_fullsized":"2024-02-23T00:29:04.110635+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.790530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.790530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165837+0000","last_change":"2024-02-23T00:28:30.086083+0000","last_active":"2024-02-23T00:29:04.165837+0000","last_peered":"2024-02-23T00:29:04.165837+0000","last_clean":"2024-02-23T00:29:04.165837+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:29:04.165837+0000","last_undegraded":"2024-02-23T00:29:04.165837+0000","last_fullsized":"2024-02-23T00:29:04.165837+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:30.086009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:30.086009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":83,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166086+0000","last_change":"2024-02-23T00:28:33.077947+0000","last_active":"2024-02-23T00:29:04.166086+0000","last_peered":"2024-02-23T00:29:04.166086+0000","last_clean":"2024-02-23T00:29:04.166086+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:29:04.166086+0000","last_undegraded":"2024-02-23T00:29:04.166086+0000","last_fullsized":"2024-02-23T00:29:04.166086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.077848+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.830507+0000","last_change":"2024-02-23T00:29:10.830479+0000","last_active":"2024-02-23T00:29:10.830507+0000","last_peered":"2024-02-23T00:29:10.830507+0000","last_clean":"2024-02-23T00:29:10.830507+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:29:10.830507+0000","last_undegraded":"2024-02-23T00:29:10.830507+0000","last_fullsized":"2024-02-23T00:29:10.830507+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.830389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.830389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:05.641550+0000","last_change":"2024-02-23T00:29:05.641525+0000","last_active":"2024-02-23T00:29:05.641550+0000","last_peered":"2024-02-23T00:29:05.641550+0000","last_clean":"2024-02-23T00:29:05.641550+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:29:05.641550+0000","last_undegraded":"2024-02-23T00:29:05.641550+0000","last_fullsized":"2024-02-23T00:29:05.641550+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_clean_scrub_stamp":"2024-02-23T00:29:05.641454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165791+0000","last_change":"2024-02-23T00:28:17.030758+0000","last_active":"2024-02-23T00:29:04.165791+0000","last_peered":"2024-02-23T00:29:04.165791+0000","last_clean":"2024-02-23T00:29:04.165791+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:29:04.165791+0000","last_undegraded":"2024-02-23T00:29:04.165791+0000","last_fullsized":"2024-02-23T00:29:04.165791+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:17.030655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:17.030655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.113086+0000","last_change":"2024-02-23T00:28:27.557204+0000","last_active":"2024-02-23T00:29:04.113086+0000","last_peered":"2024-02-23T00:29:04.113086+0000","last_clean":"2024-02-23T00:29:04.113086+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:29:04.113086+0000","last_undegraded":"2024-02-23T00:29:04.113086+0000","last_fullsized":"2024-02-23T00:29:04.113086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:28:27.557140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:27.557140+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111713+0000","last_change":"2024-02-23T00:28:27.524062+0000","last_active":"2024-02-23T00:29:04.111713+0000","last_peered":"2024-02-23T00:29:04.111713+0000","last_clean":"2024-02-23T00:29:04.111713+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:29:04.111713+0000","last_undegraded":"2024-02-23T00:29:04.111713+0000","last_fullsized":"2024-02-23T00:29:04.111713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:27.523959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:27.523959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.130821+0000","last_change":"2024-02-23T00:29:10.130797+0000","last_active":"2024-02-23T00:29:10.130821+0000","last_peered":"2024-02-23T00:29:10.130821+0000","last_clean":"2024-02-23T00:29:10.130821+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:29:10.130821+0000","last_undegraded":"2024-02-23T00:29:10.130821+0000","last_fullsized":"2024-02-23T00:29:10.130821+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.130727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.130727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165445+0000","last_change":"2024-02-23T00:28:32.090929+0000","last_active":"2024-02-23T00:29:04.165445+0000","last_peered":"2024-02-23T00:29:04.165445+0000","last_clean":"2024-02-23T00:29:04.165445+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:29:04.165445+0000","last_undegraded":"2024-02-23T00:29:04.165445+0000","last_fullsized":"2024-02-23T00:29:04.165445+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:32.090869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:32.090869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167021+0000","last_change":"2024-02-23T00:28:44.987515+0000","last_active":"2024-02-23T00:29:04.167021+0000","last_peered":"2024-02-23T00:29:04.167021+0000","last_clean":"2024-02-23T00:29:04.167021+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:29:04.167021+0000","last_undegraded":"2024-02-23T00:29:04.167021+0000","last_fullsized":"2024-02-23T00:29:04.167021+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:28:44.987419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.987419+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167041+0000","last_change":"2024-02-23T00:28:48.333707+0000","last_active":"2024-02-23T00:29:04.167041+0000","last_peered":"2024-02-23T00:29:04.167041+0000","last_clean":"2024-02-23T00:29:04.167041+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:29:04.167041+0000","last_undegraded":"2024-02-23T00:29:04.167041+0000","last_fullsized":"2024-02-23T00:29:04.167041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:28:48.333638+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"62'8","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.349098+0000","last_change":"2024-02-23T00:28:36.071048+0000","last_active":"2024-02-23T00:29:20.349098+0000","last_peered":"2024-02-23T00:29:20.349098+0000","last_clean":"2024-02-23T00:29:20.349098+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:29:20.349098+0000","last_undegraded":"2024-02-23T00:29:20.349098+0000","last_fullsized":"2024-02-23T00:29:20.349098+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:36.070959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.070959+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165452+0000","last_change":"2024-02-23T00:29:03.017128+0000","last_active":"2024-02-23T00:29:04.165452+0000","last_peered":"2024-02-23T00:29:04.165452+0000","last_clean":"2024-02-23T00:29:04.165452+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:29:04.165452+0000","last_undegraded":"2024-02-23T00:29:04.165452+0000","last_fullsized":"2024-02-23T00:29:04.165452+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.017051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.017051+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'16","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166313+0000","last_change":"2024-02-23T00:28:55.230148+0000","last_active":"2024-02-23T00:29:04.166313+0000","last_peered":"2024-02-23T00:29:04.166313+0000","last_clean":"2024-02-23T00:29:04.166313+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:29:04.166313+0000","last_undegraded":"2024-02-23T00:29:04.166313+0000","last_fullsized":"2024-02-23T00:29:04.166313+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:28:55.230081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.230081+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165952+0000","last_change":"2024-02-23T00:28:41.113778+0000","last_active":"2024-02-23T00:29:04.165952+0000","last_peered":"2024-02-23T00:29:04.165952+0000","last_clean":"2024-02-23T00:29:04.165952+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:29:04.165952+0000","last_undegraded":"2024-02-23T00:29:04.165952+0000","last_fullsized":"2024-02-23T00:29:04.165952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.113684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.113684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167420+0000","last_change":"2024-02-23T00:28:41.520505+0000","last_active":"2024-02-23T00:29:04.167420+0000","last_peered":"2024-02-23T00:29:04.167420+0000","last_clean":"2024-02-23T00:29:04.167420+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:29:04.167420+0000","last_undegraded":"2024-02-23T00:29:04.167420+0000","last_fullsized":"2024-02-23T00:29:04.167420+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.520412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.520412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167154+0000","last_change":"2024-02-23T00:28:37.106379+0000","last_active":"2024-02-23T00:29:04.167154+0000","last_peered":"2024-02-23T00:29:04.167154+0000","last_clean":"2024-02-23T00:29:04.167154+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:29:04.167154+0000","last_undegraded":"2024-02-23T00:29:04.167154+0000","last_fullsized":"2024-02-23T00:29:04.167154+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:37.106309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:37.106309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":80,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165684+0000","last_change":"2024-02-23T00:28:57.039721+0000","last_active":"2024-02-23T00:29:04.165684+0000","last_peered":"2024-02-23T00:29:04.165684+0000","last_clean":"2024-02-23T00:29:04.165684+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:29:04.165684+0000","last_undegraded":"2024-02-23T00:29:04.165684+0000","last_fullsized":"2024-02-23T00:29:04.165684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.039660+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166057+0000","last_change":"2024-02-23T00:28:43.440882+0000","last_active":"2024-02-23T00:29:04.166057+0000","last_peered":"2024-02-23T00:29:04.166057+0000","last_clean":"2024-02-23T00:29:04.166057+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:29:04.166057+0000","last_undegraded":"2024-02-23T00:29:04.166057+0000","last_fullsized":"2024-02-23T00:29:04.166057+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.440788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.440788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166062+0000","last_change":"2024-02-23T00:28:56.022911+0000","last_active":"2024-02-23T00:29:04.166062+0000","last_peered":"2024-02-23T00:29:04.166062+0000","last_clean":"2024-02-23T00:29:04.166062+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:29:04.166062+0000","last_undegraded":"2024-02-23T00:29:04.166062+0000","last_fullsized":"2024-02-23T00:29:04.166062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:56.022835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:56.022835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167959+0000","last_change":"2024-02-23T00:28:24.002963+0000","last_active":"2024-02-23T00:29:04.167959+0000","last_peered":"2024-02-23T00:29:04.167959+0000","last_clean":"2024-02-23T00:29:04.167959+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:29:04.167959+0000","last_undegraded":"2024-02-23T00:29:04.167959+0000","last_fullsized":"2024-02-23T00:29:04.167959+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:24.002878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:24.002878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165126+0000","last_change":"2024-02-23T00:28:20.075559+0000","last_active":"2024-02-23T00:29:04.165126+0000","last_peered":"2024-02-23T00:29:04.165126+0000","last_clean":"2024-02-23T00:29:04.165126+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:29:04.165126+0000","last_undegraded":"2024-02-23T00:29:04.165126+0000","last_fullsized":"2024-02-23T00:29:04.165126+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:28:20.075487+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165247+0000","last_change":"2024-02-23T00:28:39.076715+0000","last_active":"2024-02-23T00:29:04.165247+0000","last_peered":"2024-02-23T00:29:04.165247+0000","last_clean":"2024-02-23T00:29:04.165247+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:29:04.165247+0000","last_undegraded":"2024-02-23T00:29:04.165247+0000","last_fullsized":"2024-02-23T00:29:04.165247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.076631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.076631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111882+0000","last_change":"2024-02-23T00:28:53.665149+0000","last_active":"2024-02-23T00:29:04.111882+0000","last_peered":"2024-02-23T00:29:04.111882+0000","last_clean":"2024-02-23T00:29:04.111882+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:29:04.111882+0000","last_undegraded":"2024-02-23T00:29:04.111882+0000","last_fullsized":"2024-02-23T00:29:04.111882+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.665092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111817+0000","last_change":"2024-02-23T00:29:02.711400+0000","last_active":"2024-02-23T00:29:04.111817+0000","last_peered":"2024-02-23T00:29:04.111817+0000","last_clean":"2024-02-23T00:29:04.111817+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:29:04.111817+0000","last_undegraded":"2024-02-23T00:29:04.111817+0000","last_fullsized":"2024-02-23T00:29:04.111817+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.711284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":79,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110548+0000","last_change":"2024-02-23T00:28:41.214195+0000","last_active":"2024-02-23T00:29:04.110548+0000","last_peered":"2024-02-23T00:29:04.110548+0000","last_clean":"2024-02-23T00:29:04.110548+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:29:04.110548+0000","last_undegraded":"2024-02-23T00:29:04.110548+0000","last_fullsized":"2024-02-23T00:29:04.110548+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:28:36.536962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.536962+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165027+0000","last_change":"2024-02-23T00:28:49.187859+0000","last_active":"2024-02-23T00:29:04.165027+0000","last_peered":"2024-02-23T00:29:04.165027+0000","last_clean":"2024-02-23T00:29:04.165027+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:29:04.165027+0000","last_undegraded":"2024-02-23T00:29:04.165027+0000","last_fullsized":"2024-02-23T00:29:04.165027+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.187786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.187786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165874+0000","last_change":"2024-02-23T00:28:22.160736+0000","last_active":"2024-02-23T00:29:04.165874+0000","last_peered":"2024-02-23T00:29:04.165874+0000","last_clean":"2024-02-23T00:29:04.165874+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:29:04.165874+0000","last_undegraded":"2024-02-23T00:29:04.165874+0000","last_fullsized":"2024-02-23T00:29:04.165874+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:22.160642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:22.160642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166211+0000","last_change":"2024-02-23T00:28:21.021722+0000","last_active":"2024-02-23T00:29:04.166211+0000","last_peered":"2024-02-23T00:29:04.166211+0000","last_clean":"2024-02-23T00:29:04.166211+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:29:04.166211+0000","last_undegraded":"2024-02-23T00:29:04.166211+0000","last_fullsized":"2024-02-23T00:29:04.166211+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:21.021619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.021619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111782+0000","last_change":"2024-02-23T00:28:36.676648+0000","last_active":"2024-02-23T00:29:04.111782+0000","last_peered":"2024-02-23T00:29:04.111782+0000","last_clean":"2024-02-23T00:29:04.111782+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:29:04.111782+0000","last_undegraded":"2024-02-23T00:29:04.111782+0000","last_fullsized":"2024-02-23T00:29:04.111782+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:28:36.676582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.676582+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111356+0000","last_change":"2024-02-23T00:28:43.560702+0000","last_active":"2024-02-23T00:29:04.111356+0000","last_peered":"2024-02-23T00:29:04.111356+0000","last_clean":"2024-02-23T00:29:04.111356+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:29:04.111356+0000","last_undegraded":"2024-02-23T00:29:04.111356+0000","last_fullsized":"2024-02-23T00:29:04.111356+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.560606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.560606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"62'8","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348794+0000","last_change":"2024-02-23T00:28:21.123596+0000","last_active":"2024-02-23T00:29:20.348794+0000","last_peered":"2024-02-23T00:29:20.348794+0000","last_clean":"2024-02-23T00:29:20.348794+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:29:20.348794+0000","last_undegraded":"2024-02-23T00:29:20.348794+0000","last_fullsized":"2024-02-23T00:29:20.348794+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:21.123498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.123498+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165830+0000","last_change":"2024-02-23T00:28:30.236265+0000","last_active":"2024-02-23T00:29:04.165830+0000","last_peered":"2024-02-23T00:29:04.165830+0000","last_clean":"2024-02-23T00:29:04.165830+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:29:04.165830+0000","last_undegraded":"2024-02-23T00:29:04.165830+0000","last_fullsized":"2024-02-23T00:29:04.165830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:30.236188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:30.236188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'15","reported_seq":68,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166655+0000","last_change":"2024-02-23T00:28:55.907616+0000","last_active":"2024-02-23T00:29:04.166655+0000","last_peered":"2024-02-23T00:29:04.166655+0000","last_clean":"2024-02-23T00:29:04.166655+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:29:04.166655+0000","last_undegraded":"2024-02-23T00:29:04.166655+0000","last_fullsized":"2024-02-23T00:29:04.166655+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:28:55.907528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.907528+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:14.086398+0000","last_change":"2024-02-23T00:29:14.086374+0000","last_active":"2024-02-23T00:29:14.086398+0000","last_peered":"2024-02-23T00:29:14.086398+0000","last_clean":"2024-02-23T00:29:14.086398+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:29:14.086398+0000","last_undegraded":"2024-02-23T00:29:14.086398+0000","last_fullsized":"2024-02-23T00:29:14.086398+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:14.086293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:14.086293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.349069+0000","last_change":"2024-02-23T00:28:38.956507+0000","last_active":"2024-02-23T00:29:20.349069+0000","last_peered":"2024-02-23T00:29:20.349069+0000","last_clean":"2024-02-23T00:29:20.349069+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:29:20.349069+0000","last_undegraded":"2024-02-23T00:29:20.349069+0000","last_fullsized":"2024-02-23T00:29:20.349069+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_clean_scrub_stamp":"2024-02-23T00:28:38.956431+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165830+0000","last_change":"2024-02-23T00:28:49.012817+0000","last_active":"2024-02-23T00:29:04.165830+0000","last_peered":"2024-02-23T00:29:04.165830+0000","last_clean":"2024-02-23T00:29:04.165830+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:29:04.165830+0000","last_undegraded":"2024-02-23T00:29:04.165830+0000","last_fullsized":"2024-02-23T00:29:04.165830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.012698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":84,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166786+0000","last_change":"2024-02-23T00:29:01.026660+0000","last_active":"2024-02-23T00:29:04.166786+0000","last_peered":"2024-02-23T00:29:04.166786+0000","last_clean":"2024-02-23T00:29:04.166786+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:29:04.166786+0000","last_undegraded":"2024-02-23T00:29:04.166786+0000","last_fullsized":"2024-02-23T00:29:04.166786+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:29:01.026575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.026575+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167407+0000","last_change":"2024-02-23T00:28:58.383733+0000","last_active":"2024-02-23T00:29:04.167407+0000","last_peered":"2024-02-23T00:29:04.167407+0000","last_clean":"2024-02-23T00:29:04.167407+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:29:04.167407+0000","last_undegraded":"2024-02-23T00:29:04.167407+0000","last_fullsized":"2024-02-23T00:29:04.167407+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:28:58.383642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.044071+0000","last_change":"2024-02-23T00:29:10.044046+0000","last_active":"2024-02-23T00:29:10.044071+0000","last_peered":"2024-02-23T00:29:10.044071+0000","last_clean":"2024-02-23T00:29:10.044071+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:29:10.044071+0000","last_undegraded":"2024-02-23T00:29:10.044071+0000","last_fullsized":"2024-02-23T00:29:10.044071+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.043966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.043966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166141+0000","last_change":"2024-02-23T00:28:34.066522+0000","last_active":"2024-02-23T00:29:04.166141+0000","last_peered":"2024-02-23T00:29:04.166141+0000","last_clean":"2024-02-23T00:29:04.166141+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:29:04.166141+0000","last_undegraded":"2024-02-23T00:29:04.166141+0000","last_fullsized":"2024-02-23T00:29:04.166141+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:34.066441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:34.066441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":82,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166525+0000","last_change":"2024-02-23T00:28:42.144931+0000","last_active":"2024-02-23T00:29:04.166525+0000","last_peered":"2024-02-23T00:29:04.166525+0000","last_clean":"2024-02-23T00:29:04.166525+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:29:04.166525+0000","last_undegraded":"2024-02-23T00:29:04.166525+0000","last_fullsized":"2024-02-23T00:29:04.166525+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:28:42.144833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:42.144833+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112842+0000","last_change":"2024-02-23T00:28:53.629641+0000","last_active":"2024-02-23T00:29:04.112842+0000","last_peered":"2024-02-23T00:29:04.112842+0000","last_clean":"2024-02-23T00:29:04.112842+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:29:04.112842+0000","last_undegraded":"2024-02-23T00:29:04.112842+0000","last_fullsized":"2024-02-23T00:29:04.112842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:53.629568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.629568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166601+0000","last_change":"2024-02-23T00:29:01.986532+0000","last_active":"2024-02-23T00:29:04.166601+0000","last_peered":"2024-02-23T00:29:04.166601+0000","last_clean":"2024-02-23T00:29:04.166601+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:29:04.166601+0000","last_undegraded":"2024-02-23T00:29:04.166601+0000","last_fullsized":"2024-02-23T00:29:04.166601+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:01.986461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.986461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112289+0000","last_change":"2024-02-23T00:28:41.611517+0000","last_active":"2024-02-23T00:29:04.112289+0000","last_peered":"2024-02-23T00:29:04.112289+0000","last_clean":"2024-02-23T00:29:04.112289+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:29:04.112289+0000","last_undegraded":"2024-02-23T00:29:04.112289+0000","last_fullsized":"2024-02-23T00:29:04.112289+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.611448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.611448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166225+0000","last_change":"2024-02-23T00:29:01.234433+0000","last_active":"2024-02-23T00:29:04.166225+0000","last_peered":"2024-02-23T00:29:04.166225+0000","last_clean":"2024-02-23T00:29:04.166225+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:29:04.166225+0000","last_undegraded":"2024-02-23T00:29:04.166225+0000","last_fullsized":"2024-02-23T00:29:04.166225+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:29:01.234356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.234356+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165913+0000","last_change":"2024-02-23T00:28:35.058858+0000","last_active":"2024-02-23T00:29:04.165913+0000","last_peered":"2024-02-23T00:29:04.165913+0000","last_clean":"2024-02-23T00:29:04.165913+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:29:04.165913+0000","last_undegraded":"2024-02-23T00:29:04.165913+0000","last_fullsized":"2024-02-23T00:29:04.165913+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:35.058773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:35.058773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111158+0000","last_change":"2024-02-23T00:28:51.684865+0000","last_active":"2024-02-23T00:29:04.111158+0000","last_peered":"2024-02-23T00:29:04.111158+0000","last_clean":"2024-02-23T00:29:04.111158+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:29:04.111158+0000","last_undegraded":"2024-02-23T00:29:04.111158+0000","last_fullsized":"2024-02-23T00:29:04.111158+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:28:51.684789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.684789+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":118,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110345+0000","last_change":"2024-02-23T00:29:01.733374+0000","last_active":"2024-02-23T00:29:04.110345+0000","last_peered":"2024-02-23T00:29:04.110345+0000","last_clean":"2024-02-23T00:29:04.110345+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:29:04.110345+0000","last_undegraded":"2024-02-23T00:29:04.110345+0000","last_fullsized":"2024-02-23T00:29:04.110345+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:29:01.733287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.733287+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112261+0000","last_change":"2024-02-23T00:28:51.634965+0000","last_active":"2024-02-23T00:29:04.112261+0000","last_peered":"2024-02-23T00:29:04.112261+0000","last_clean":"2024-02-23T00:29:04.112261+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:29:04.112261+0000","last_undegraded":"2024-02-23T00:29:04.112261+0000","last_fullsized":"2024-02-23T00:29:04.112261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:51.634901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.634901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.227949+0000","last_change":"2024-02-23T00:29:04.227928+0000","last_active":"2024-02-23T00:29:04.227949+0000","last_peered":"2024-02-23T00:29:04.227949+0000","last_clean":"2024-02-23T00:29:04.227949+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:29:04.227949+0000","last_undegraded":"2024-02-23T00:29:04.227949+0000","last_fullsized":"2024-02-23T00:29:04.227949+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.227864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.227864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111585+0000","last_change":"2024-02-23T00:28:51.664433+0000","last_active":"2024-02-23T00:29:04.111585+0000","last_peered":"2024-02-23T00:29:04.111585+0000","last_clean":"2024-02-23T00:29:04.111585+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:29:04.111585+0000","last_undegraded":"2024-02-23T00:29:04.111585+0000","last_fullsized":"2024-02-23T00:29:04.111585+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.664367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111833+0000","last_change":"2024-02-23T00:29:03.691092+0000","last_active":"2024-02-23T00:29:04.111833+0000","last_peered":"2024-02-23T00:29:04.111833+0000","last_clean":"2024-02-23T00:29:04.111833+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:29:04.111833+0000","last_undegraded":"2024-02-23T00:29:04.111833+0000","last_fullsized":"2024-02-23T00:29:04.111833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:29:03.691018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.691018+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":74,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348722+0000","last_change":"2024-02-23T00:28:42.249575+0000","last_active":"2024-02-23T00:29:20.348722+0000","last_peered":"2024-02-23T00:29:20.348722+0000","last_clean":"2024-02-23T00:29:20.348722+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:29:20.348722+0000","last_undegraded":"2024-02-23T00:29:20.348722+0000","last_fullsized":"2024-02-23T00:29:20.348722+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:42.249510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:42.249510+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166389+0000","last_change":"2024-02-23T00:28:45.134966+0000","last_active":"2024-02-23T00:29:04.166389+0000","last_peered":"2024-02-23T00:29:04.166389+0000","last_clean":"2024-02-23T00:29:04.166389+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:29:04.166389+0000","last_undegraded":"2024-02-23T00:29:04.166389+0000","last_fullsized":"2024-02-23T00:29:04.166389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:45.134840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:45.134840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165915+0000","last_change":"2024-02-23T00:28:50.000698+0000","last_active":"2024-02-23T00:29:04.165915+0000","last_peered":"2024-02-23T00:29:04.165915+0000","last_clean":"2024-02-23T00:29:04.165915+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:29:04.165915+0000","last_undegraded":"2024-02-23T00:29:04.165915+0000","last_fullsized":"2024-02-23T00:29:04.165915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:28:50.000624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.000624+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":128,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348789+0000","last_change":"2024-02-23T00:28:39.629165+0000","last_active":"2024-02-23T00:29:20.348789+0000","last_peered":"2024-02-23T00:29:20.348789+0000","last_clean":"2024-02-23T00:29:20.348789+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:29:20.348789+0000","last_undegraded":"2024-02-23T00:29:20.348789+0000","last_fullsized":"2024-02-23T00:29:20.348789+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:28:39.629075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.629075+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165991+0000","last_change":"2024-02-23T00:28:28.220174+0000","last_active":"2024-02-23T00:29:04.165991+0000","last_peered":"2024-02-23T00:29:04.165991+0000","last_clean":"2024-02-23T00:29:04.165991+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:29:04.165991+0000","last_undegraded":"2024-02-23T00:29:04.165991+0000","last_fullsized":"2024-02-23T00:29:04.165991+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:28.220080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.220080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167568+0000","last_change":"2024-02-23T00:28:39.496236+0000","last_active":"2024-02-23T00:29:04.167568+0000","last_peered":"2024-02-23T00:29:04.167568+0000","last_clean":"2024-02-23T00:29:04.167568+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:29:04.167568+0000","last_undegraded":"2024-02-23T00:29:04.167568+0000","last_fullsized":"2024-02-23T00:29:04.167568+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.496160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.496160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":167,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:24.259201+0000","last_change":"2024-02-23T00:28:52.653896+0000","last_active":"2024-02-23T00:29:24.259201+0000","last_peered":"2024-02-23T00:29:24.259201+0000","last_clean":"2024-02-23T00:29:24.259201+0000","last_became_active":"2024-02-23T00:27:17.009670+0000","last_became_peered":"2024-02-23T00:27:17.009670+0000","last_unstale":"2024-02-23T00:29:24.259201+0000","last_undegraded":"2024-02-23T00:29:24.259201+0000","last_fullsized":"2024-02-23T00:29:24.259201+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.653823+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.113286+0000","last_change":"2024-02-23T00:28:33.503311+0000","last_active":"2024-02-23T00:29:04.113286+0000","last_peered":"2024-02-23T00:29:04.113286+0000","last_clean":"2024-02-23T00:29:04.113286+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:29:04.113286+0000","last_undegraded":"2024-02-23T00:29:04.113286+0000","last_fullsized":"2024-02-23T00:29:04.113286+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.503241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.503241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":81,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112582+0000","last_change":"2024-02-23T00:28:32.603154+0000","last_active":"2024-02-23T00:29:04.112582+0000","last_peered":"2024-02-23T00:29:04.112582+0000","last_clean":"2024-02-23T00:29:04.112582+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:29:04.112582+0000","last_undegraded":"2024-02-23T00:29:04.112582+0000","last_fullsized":"2024-02-23T00:29:04.112582+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:28:32.603088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:32.603088+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112491+0000","last_change":"2024-02-23T00:28:50.672048+0000","last_active":"2024-02-23T00:29:04.112491+0000","last_peered":"2024-02-23T00:29:04.112491+0000","last_clean":"2024-02-23T00:29:04.112491+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:29:04.112491+0000","last_undegraded":"2024-02-23T00:29:04.112491+0000","last_fullsized":"2024-02-23T00:29:04.112491+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.671988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.671988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.168280+0000","last_change":"2024-02-23T00:28:40.999596+0000","last_active":"2024-02-23T00:29:04.168280+0000","last_peered":"2024-02-23T00:29:04.168280+0000","last_clean":"2024-02-23T00:29:04.168280+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:29:04.168280+0000","last_undegraded":"2024-02-23T00:29:04.168280+0000","last_fullsized":"2024-02-23T00:29:04.168280+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:40.999514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:40.999514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165328+0000","last_change":"2024-02-23T00:28:17.047708+0000","last_active":"2024-02-23T00:29:04.165328+0000","last_peered":"2024-02-23T00:29:04.165328+0000","last_clean":"2024-02-23T00:29:04.165328+0000","last_became_active":"2024-02-23T00:27:17.012683+0000","last_became_peered":"2024-02-23T00:27:17.012683+0000","last_unstale":"2024-02-23T00:29:04.165328+0000","last_undegraded":"2024-02-23T00:29:04.165328+0000","last_fullsized":"2024-02-23T00:29:04.165328+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:17.047614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:17.047614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167144+0000","last_change":"2024-02-23T00:28:52.948687+0000","last_active":"2024-02-23T00:29:04.167144+0000","last_peered":"2024-02-23T00:29:04.167144+0000","last_clean":"2024-02-23T00:29:04.167144+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:29:04.167144+0000","last_undegraded":"2024-02-23T00:29:04.167144+0000","last_fullsized":"2024-02-23T00:29:04.167144+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:52.948601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.948601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:25.267851+0000","last_change":"2024-02-23T00:29:25.267851+0000","last_active":"2024-02-23T00:29:25.267851+0000","last_peered":"2024-02-23T00:29:25.267851+0000","last_clean":"2024-02-23T00:29:25.267851+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:29:25.267851+0000","last_undegraded":"2024-02-23T00:29:25.267851+0000","last_fullsized":"2024-02-23T00:29:25.267851+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:29:25.267773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166538+0000","last_change":"2024-02-23T00:28:33.090534+0000","last_active":"2024-02-23T00:29:04.166538+0000","last_peered":"2024-02-23T00:29:04.166538+0000","last_clean":"2024-02-23T00:29:04.166538+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:29:04.166538+0000","last_undegraded":"2024-02-23T00:29:04.166538+0000","last_fullsized":"2024-02-23T00:29:04.166538+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:28:33.090454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.090454+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166849+0000","last_change":"2024-02-23T00:28:50.344419+0000","last_active":"2024-02-23T00:29:04.166849+0000","last_peered":"2024-02-23T00:29:04.166849+0000","last_clean":"2024-02-23T00:29:04.166849+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:29:04.166849+0000","last_undegraded":"2024-02-23T00:29:04.166849+0000","last_fullsized":"2024-02-23T00:29:04.166849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.344324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.344324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111422+0000","last_change":"2024-02-23T00:28:55.679584+0000","last_active":"2024-02-23T00:29:04.111422+0000","last_peered":"2024-02-23T00:29:04.111422+0000","last_clean":"2024-02-23T00:29:04.111422+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:29:04.111422+0000","last_undegraded":"2024-02-23T00:29:04.111422+0000","last_fullsized":"2024-02-23T00:29:04.111422+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:55.679522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.679522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165482+0000","last_change":"2024-02-23T00:28:59.226967+0000","last_active":"2024-02-23T00:29:04.165482+0000","last_peered":"2024-02-23T00:29:04.165482+0000","last_clean":"2024-02-23T00:29:04.165482+0000","last_became_active":"2024-02-23T00:27:17.009826+0000","last_became_peered":"2024-02-23T00:27:17.009826+0000","last_unstale":"2024-02-23T00:29:04.165482+0000","last_undegraded":"2024-02-23T00:29:04.165482+0000","last_fullsized":"2024-02-23T00:29:04.165482+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:59.226890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:59.226890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166978+0000","last_change":"2024-02-23T00:29:02.340533+0000","last_active":"2024-02-23T00:29:04.166978+0000","last_peered":"2024-02-23T00:29:04.166978+0000","last_clean":"2024-02-23T00:29:04.166978+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:29:04.166978+0000","last_undegraded":"2024-02-23T00:29:04.166978+0000","last_fullsized":"2024-02-23T00:29:04.166978+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.340450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.340450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165741+0000","last_change":"2024-02-23T00:28:36.209181+0000","last_active":"2024-02-23T00:29:04.165741+0000","last_peered":"2024-02-23T00:29:04.165741+0000","last_clean":"2024-02-23T00:29:04.165741+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:29:04.165741+0000","last_undegraded":"2024-02-23T00:29:04.165741+0000","last_fullsized":"2024-02-23T00:29:04.165741+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:28:36.209111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.209111+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.168250+0000","last_change":"2024-02-23T00:28:24.953423+0000","last_active":"2024-02-23T00:29:04.168250+0000","last_peered":"2024-02-23T00:29:04.168250+0000","last_clean":"2024-02-23T00:29:04.168250+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:29:04.168250+0000","last_undegraded":"2024-02-23T00:29:04.168250+0000","last_fullsized":"2024-02-23T00:29:04.168250+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:28:24.953349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:11.580783+0000","last_change":"2024-02-23T00:29:11.580760+0000","last_active":"2024-02-23T00:29:11.580783+0000","last_peered":"2024-02-23T00:29:11.580783+0000","last_clean":"2024-02-23T00:29:11.580783+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:29:11.580783+0000","last_undegraded":"2024-02-23T00:29:11.580783+0000","last_fullsized":"2024-02-23T00:29:11.580783+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:11.580697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:11.580697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167628+0000","last_change":"2024-02-23T00:28:20.772584+0000","last_active":"2024-02-23T00:29:04.167628+0000","last_peered":"2024-02-23T00:29:04.167628+0000","last_clean":"2024-02-23T00:29:04.167628+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:29:04.167628+0000","last_undegraded":"2024-02-23T00:29:04.167628+0000","last_fullsized":"2024-02-23T00:29:04.167628+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:20.772479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:20.772479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":90,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165786+0000","last_change":"2024-02-23T00:28:21.157813+0000","last_active":"2024-02-23T00:29:04.165786+0000","last_peered":"2024-02-23T00:29:04.165786+0000","last_clean":"2024-02-23T00:29:04.165786+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:29:04.165786+0000","last_undegraded":"2024-02-23T00:29:04.165786+0000","last_fullsized":"2024-02-23T00:29:04.165786+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:28:21.157735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.157735+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165865+0000","last_change":"2024-02-23T00:28:43.992150+0000","last_active":"2024-02-23T00:29:04.165865+0000","last_peered":"2024-02-23T00:29:04.165865+0000","last_clean":"2024-02-23T00:29:04.165865+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:29:04.165865+0000","last_undegraded":"2024-02-23T00:29:04.165865+0000","last_fullsized":"2024-02-23T00:29:04.165865+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.992070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.992070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110748+0000","last_change":"2024-02-23T00:28:33.615049+0000","last_active":"2024-02-23T00:29:04.110748+0000","last_peered":"2024-02-23T00:29:04.110748+0000","last_clean":"2024-02-23T00:29:04.110748+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:29:04.110748+0000","last_undegraded":"2024-02-23T00:29:04.110748+0000","last_fullsized":"2024-02-23T00:29:04.110748+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.614962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.614962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112075+0000","last_change":"2024-02-23T00:28:41.617945+0000","last_active":"2024-02-23T00:29:04.112075+0000","last_peered":"2024-02-23T00:29:04.112075+0000","last_clean":"2024-02-23T00:29:04.112075+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:29:04.112075+0000","last_undegraded":"2024-02-23T00:29:04.112075+0000","last_fullsized":"2024-02-23T00:29:04.112075+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.617870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.617870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":101,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165052+0000","last_change":"2024-02-23T00:29:04.010419+0000","last_active":"2024-02-23T00:29:04.165052+0000","last_peered":"2024-02-23T00:29:04.165052+0000","last_clean":"2024-02-23T00:29:04.165052+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:29:04.165052+0000","last_undegraded":"2024-02-23T00:29:04.165052+0000","last_fullsized":"2024-02-23T00:29:04.165052+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:29:04.010348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.010348+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110724+0000","last_change":"2024-02-23T00:28:39.615097+0000","last_active":"2024-02-23T00:29:04.110724+0000","last_peered":"2024-02-23T00:29:04.110724+0000","last_clean":"2024-02-23T00:29:04.110724+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:29:04.110724+0000","last_undegraded":"2024-02-23T00:29:04.110724+0000","last_fullsized":"2024-02-23T00:29:04.110724+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.615002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.615002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166706+0000","last_change":"2024-02-23T00:28:31.646180+0000","last_active":"2024-02-23T00:29:04.166706+0000","last_peered":"2024-02-23T00:29:04.166706+0000","last_clean":"2024-02-23T00:29:04.166706+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:29:04.166706+0000","last_undegraded":"2024-02-23T00:29:04.166706+0000","last_fullsized":"2024-02-23T00:29:04.166706+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:31.646096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:31.646096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167113+0000","last_change":"2024-02-23T00:28:54.994367+0000","last_active":"2024-02-23T00:29:04.167113+0000","last_peered":"2024-02-23T00:29:04.167113+0000","last_clean":"2024-02-23T00:29:04.167113+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:29:04.167113+0000","last_undegraded":"2024-02-23T00:29:04.167113+0000","last_fullsized":"2024-02-23T00:29:04.167113+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:54.994276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.994276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":75,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:09.834901+0000","last_change":"2024-02-23T00:29:09.834851+0000","last_active":"2024-02-23T00:29:09.834901+0000","last_peered":"2024-02-23T00:29:09.834901+0000","last_clean":"2024-02-23T00:29:09.834901+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:29:09.834901+0000","last_undegraded":"2024-02-23T00:29:09.834901+0000","last_fullsized":"2024-02-23T00:29:09.834901+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:29:09.834741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:09.834741+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348815+0000","last_change":"2024-02-23T00:29:02.969742+0000","last_active":"2024-02-23T00:29:20.348815+0000","last_peered":"2024-02-23T00:29:20.348815+0000","last_clean":"2024-02-23T00:29:20.348815+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:29:20.348815+0000","last_undegraded":"2024-02-23T00:29:20.348815+0000","last_fullsized":"2024-02-23T00:29:20.348815+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:29:02.969659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.969659+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:07.960264+0000","last_change":"2024-02-23T00:29:07.960232+0000","last_active":"2024-02-23T00:29:07.960264+0000","last_peered":"2024-02-23T00:29:07.960264+0000","last_clean":"2024-02-23T00:29:07.960264+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:29:07.960264+0000","last_undegraded":"2024-02-23T00:29:07.960264+0000","last_fullsized":"2024-02-23T00:29:07.960264+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:07.960141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:07.960141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.168156+0000","last_change":"2024-02-23T00:28:54.905338+0000","last_active":"2024-02-23T00:29:04.168156+0000","last_peered":"2024-02-23T00:29:04.168156+0000","last_clean":"2024-02-23T00:29:04.168156+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:29:04.168156+0000","last_undegraded":"2024-02-23T00:29:04.168156+0000","last_fullsized":"2024-02-23T00:29:04.168156+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:54.905258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.905258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167187+0000","last_change":"2024-02-23T00:28:34.610448+0000","last_active":"2024-02-23T00:29:04.167187+0000","last_peered":"2024-02-23T00:29:04.167187+0000","last_clean":"2024-02-23T00:29:04.167187+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:29:04.167187+0000","last_undegraded":"2024-02-23T00:29:04.167187+0000","last_fullsized":"2024-02-23T00:29:04.167187+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:28:34.610334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:34.610334+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'17","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110910+0000","last_change":"2024-02-23T00:28:44.586817+0000","last_active":"2024-02-23T00:29:04.110910+0000","last_peered":"2024-02-23T00:29:04.110910+0000","last_clean":"2024-02-23T00:29:04.110910+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:29:04.110910+0000","last_undegraded":"2024-02-23T00:29:04.110910+0000","last_fullsized":"2024-02-23T00:29:04.110910+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:28:44.586731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.586731+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166329+0000","last_change":"2024-02-23T00:28:47.090951+0000","last_active":"2024-02-23T00:29:04.166329+0000","last_peered":"2024-02-23T00:29:04.166329+0000","last_clean":"2024-02-23T00:29:04.166329+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:29:04.166329+0000","last_undegraded":"2024-02-23T00:29:04.166329+0000","last_fullsized":"2024-02-23T00:29:04.166329+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:28:47.090884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.090884+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.349062+0000","last_change":"2024-02-23T00:28:31.557831+0000","last_active":"2024-02-23T00:29:20.349062+0000","last_peered":"2024-02-23T00:29:20.349062+0000","last_clean":"2024-02-23T00:29:20.349062+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:29:20.349062+0000","last_undegraded":"2024-02-23T00:29:20.349062+0000","last_fullsized":"2024-02-23T00:29:20.349062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:31.557737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:31.557737+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.574074+0000","last_change":"2024-02-23T00:29:10.574049+0000","last_active":"2024-02-23T00:29:10.574074+0000","last_peered":"2024-02-23T00:29:10.574074+0000","last_clean":"2024-02-23T00:29:10.574074+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:29:10.574074+0000","last_undegraded":"2024-02-23T00:29:10.574074+0000","last_fullsized":"2024-02-23T00:29:10.574074+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.573983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.573983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'15","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112648+0000","last_change":"2024-02-23T00:28:46.622106+0000","last_active":"2024-02-23T00:29:04.112648+0000","last_peered":"2024-02-23T00:29:04.112648+0000","last_clean":"2024-02-23T00:29:04.112648+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:29:04.112648+0000","last_undegraded":"2024-02-23T00:29:04.112648+0000","last_fullsized":"2024-02-23T00:29:04.112648+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:28:46.621982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.621982+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110668+0000","last_change":"2024-02-23T00:28:40.630272+0000","last_active":"2024-02-23T00:29:04.110668+0000","last_peered":"2024-02-23T00:29:04.110668+0000","last_clean":"2024-02-23T00:29:04.110668+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:29:04.110668+0000","last_undegraded":"2024-02-23T00:29:04.110668+0000","last_fullsized":"2024-02-23T00:29:04.110668+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:40.630153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:40.630153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348734+0000","last_change":"2024-02-23T00:28:44.617114+0000","last_active":"2024-02-23T00:29:20.348734+0000","last_peered":"2024-02-23T00:29:20.348734+0000","last_clean":"2024-02-23T00:29:20.348734+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:29:20.348734+0000","last_undegraded":"2024-02-23T00:29:20.348734+0000","last_fullsized":"2024-02-23T00:29:20.348734+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:44.617038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.617038+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":50,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111700+0000","last_change":"2024-02-23T00:29:00.703800+0000","last_active":"2024-02-23T00:29:04.111700+0000","last_peered":"2024-02-23T00:29:04.111700+0000","last_clean":"2024-02-23T00:29:04.111700+0000","last_became_active":"2024-02-23T00:28:41.203031+0000","last_became_peered":"2024-02-23T00:28:41.203031+0000","last_unstale":"2024-02-23T00:29:04.111700+0000","last_undegraded":"2024-02-23T00:29:04.111700+0000","last_fullsized":"2024-02-23T00:29:04.111700+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.703737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:08.818370+0000","last_change":"2024-02-23T00:29:08.818324+0000","last_active":"2024-02-23T00:29:08.818370+0000","last_peered":"2024-02-23T00:29:08.818370+0000","last_clean":"2024-02-23T00:29:08.818370+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:29:08.818370+0000","last_undegraded":"2024-02-23T00:29:08.818370+0000","last_fullsized":"2024-02-23T00:29:08.818370+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:29:08.818235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.818235+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112456+0000","last_change":"2024-02-23T00:28:47.677799+0000","last_active":"2024-02-23T00:29:04.112456+0000","last_peered":"2024-02-23T00:29:04.112456+0000","last_clean":"2024-02-23T00:29:04.112456+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:29:04.112456+0000","last_undegraded":"2024-02-23T00:29:04.112456+0000","last_fullsized":"2024-02-23T00:29:04.112456+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:47.677731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.677731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:08.580191+0000","last_change":"2024-02-23T00:29:08.580168+0000","last_active":"2024-02-23T00:29:08.580191+0000","last_peered":"2024-02-23T00:29:08.580191+0000","last_clean":"2024-02-23T00:29:08.580191+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:29:08.580191+0000","last_undegraded":"2024-02-23T00:29:08.580191+0000","last_fullsized":"2024-02-23T00:29:08.580191+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.580102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.580102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:05.279997+0000","last_change":"2024-02-23T00:29:05.279969+0000","last_active":"2024-02-23T00:29:05.279997+0000","last_peered":"2024-02-23T00:29:05.279997+0000","last_clean":"2024-02-23T00:29:05.279997+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:29:05.279997+0000","last_undegraded":"2024-02-23T00:29:05.279997+0000","last_fullsized":"2024-02-23T00:29:05.279997+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:05.279882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:05.279882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110952+0000","last_change":"2024-02-23T00:28:29.526599+0000","last_active":"2024-02-23T00:29:04.110952+0000","last_peered":"2024-02-23T00:29:04.110952+0000","last_clean":"2024-02-23T00:29:04.110952+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:29:04.110952+0000","last_undegraded":"2024-02-23T00:29:04.110952+0000","last_fullsized":"2024-02-23T00:29:04.110952+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:28:29.526505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:29.526505+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166861+0000","last_change":"2024-02-23T00:28:49.331226+0000","last_active":"2024-02-23T00:29:04.166861+0000","last_peered":"2024-02-23T00:29:04.166861+0000","last_clean":"2024-02-23T00:29:04.166861+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:29:04.166861+0000","last_undegraded":"2024-02-23T00:29:04.166861+0000","last_fullsized":"2024-02-23T00:29:04.166861+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.331139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.331139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.113319+0000","last_change":"2024-02-23T00:29:01.776038+0000","last_active":"2024-02-23T00:29:04.113319+0000","last_peered":"2024-02-23T00:29:04.113319+0000","last_clean":"2024-02-23T00:29:04.113319+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:29:04.113319+0000","last_undegraded":"2024-02-23T00:29:04.113319+0000","last_fullsized":"2024-02-23T00:29:04.113319+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:01.775943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.775943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:24.809313+0000","last_change":"2024-02-23T00:29:24.809313+0000","last_active":"2024-02-23T00:29:24.809313+0000","last_peered":"2024-02-23T00:29:24.809313+0000","last_clean":"2024-02-23T00:29:24.809313+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:29:24.809313+0000","last_undegraded":"2024-02-23T00:29:24.809313+0000","last_fullsized":"2024-02-23T00:29:24.809313+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:24.809192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:24.809192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166687+0000","last_change":"2024-02-23T00:28:28.095098+0000","last_active":"2024-02-23T00:29:04.166687+0000","last_peered":"2024-02-23T00:29:04.166687+0000","last_clean":"2024-02-23T00:29:04.166687+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:29:04.166687+0000","last_undegraded":"2024-02-23T00:29:04.166687+0000","last_fullsized":"2024-02-23T00:29:04.166687+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:28:28.095011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.095011+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165173+0000","last_change":"2024-02-23T00:29:00.187868+0000","last_active":"2024-02-23T00:29:04.165173+0000","last_peered":"2024-02-23T00:29:04.165173+0000","last_clean":"2024-02-23T00:29:04.165173+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:29:04.165173+0000","last_undegraded":"2024-02-23T00:29:04.165173+0000","last_fullsized":"2024-02-23T00:29:04.165173+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.187784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.187784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112896+0000","last_change":"2024-02-23T00:29:02.769052+0000","last_active":"2024-02-23T00:29:04.112896+0000","last_peered":"2024-02-23T00:29:04.112896+0000","last_clean":"2024-02-23T00:29:04.112896+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:29:04.112896+0000","last_undegraded":"2024-02-23T00:29:04.112896+0000","last_fullsized":"2024-02-23T00:29:04.112896+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.768986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.768986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111919+0000","last_change":"2024-02-23T00:28:26.541700+0000","last_active":"2024-02-23T00:29:04.111919+0000","last_peered":"2024-02-23T00:29:04.111919+0000","last_clean":"2024-02-23T00:29:04.111919+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:29:04.111919+0000","last_undegraded":"2024-02-23T00:29:04.111919+0000","last_fullsized":"2024-02-23T00:29:04.111919+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:26.541633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:26.541633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":70,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165871+0000","last_change":"2024-02-23T00:28:53.186087+0000","last_active":"2024-02-23T00:29:04.165871+0000","last_peered":"2024-02-23T00:29:04.165871+0000","last_clean":"2024-02-23T00:29:04.165871+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:29:04.165871+0000","last_undegraded":"2024-02-23T00:29:04.165871+0000","last_fullsized":"2024-02-23T00:29:04.165871+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:28:53.186017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.186017+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:08.063504+0000","last_change":"2024-02-23T00:29:08.063475+0000","last_active":"2024-02-23T00:29:08.063504+0000","last_peered":"2024-02-23T00:29:08.063504+0000","last_clean":"2024-02-23T00:29:08.063504+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:29:08.063504+0000","last_undegraded":"2024-02-23T00:29:08.063504+0000","last_fullsized":"2024-02-23T00:29:08.063504+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.063398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.063398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112616+0000","last_change":"2024-02-23T00:28:16.558091+0000","last_active":"2024-02-23T00:29:04.112616+0000","last_peered":"2024-02-23T00:29:04.112616+0000","last_clean":"2024-02-23T00:29:04.112616+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:29:04.112616+0000","last_undegraded":"2024-02-23T00:29:04.112616+0000","last_fullsized":"2024-02-23T00:29:04.112616+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:16.558024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:16.558024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":101,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165621+0000","last_change":"2024-02-23T00:28:46.118981+0000","last_active":"2024-02-23T00:29:04.165621+0000","last_peered":"2024-02-23T00:29:04.165621+0000","last_clean":"2024-02-23T00:29:04.165621+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:29:04.165621+0000","last_undegraded":"2024-02-23T00:29:04.165621+0000","last_fullsized":"2024-02-23T00:29:04.165621+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.118918+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'8","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.349079+0000","last_change":"2024-02-23T00:28:28.676907+0000","last_active":"2024-02-23T00:29:20.349079+0000","last_peered":"2024-02-23T00:29:20.349079+0000","last_clean":"2024-02-23T00:29:20.349079+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:29:20.349079+0000","last_undegraded":"2024-02-23T00:29:20.349079+0000","last_fullsized":"2024-02-23T00:29:20.349079+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:28.676837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.676837+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":616,"ondisk_log_size":616,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332125,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":290664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.502,"15min":0.501},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68300000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.441}]},{"osd":1,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.61799999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.49399999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.41599999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.28599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.46999999999999997}]},{"osd":5,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42799999999999999,"15min":0.42699999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.436,"15min":0.42799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.63400000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40500000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.47699999999999998,"15min":0.47099999999999997},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.46000000000000002,"15min":0.45400000000000001},"min":{"1min":0.33900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.52600000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561058,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":289807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.28599999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.51000000000000001,"15min":0.51700000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69399999999999995,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.60899999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.57799999999999996,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.31,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70299999999999996,"5min":0.754,"15min":0.754},"last":0.53900000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.55100000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.504}]},{"osd":3,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48499999999999999,"15min":0.48699999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66800000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.69199999999999995}]},{"osd":4,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.378},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40899999999999997,"15min":0.40600000000000003},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61499999999999999,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.435}]},{"osd":5,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42399999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61699999999999999,"5min":0.996,"15min":0.996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.42199999999999999,"15min":0.41999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.48299999999999998,"15min":0.47599999999999998},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.46700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297748,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447404,"statfs":{"total":95995035648,"available":95690141696,"internally_reserved":0,"allocated":667648,"data_stored":291667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.32800000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.44},{"interface":"front","average":{"1min":0.371,"5min":0.372,"15min":0.35099999999999998},"min":{"1min":0.215,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54600000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.37,"15min":0.371},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33400000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.41399999999999998,"15min":0.42599999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.49299999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.438,"15min":0.435},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.38400000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72399999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Feb 23 00:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79100000000000004,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.59399999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.71599999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.52300000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:29:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78500000000000003,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.505},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.60899999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.752,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":290324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34399999999999997,"15min":0.32800000000000001},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.501,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.371},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35599999999999998,"15min":0.33300000000000002},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.49099999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.42099999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.38500000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.46899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.34499999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.43099999999999999,"15min":0.44700000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56200000000000006,"5min":0.629,"15min":0.629},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.313,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53100000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.41199999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.39100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.48999999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.58199999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.59199999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41299999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.623},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":630784,"data_stored":290227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.33700000000000002,"15min":0.32900000000000001},"min":{"1min":0.23799999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.36099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.34300000000000003,"15min":0.32900000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.42699999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.35599999999999998,"15min":0.34000000000000002},"min":{"1min":0.245,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47599999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35099999999999998,"15min":0.33900000000000002},"min":{"1min":0.25900000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.44500000000000001,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:28:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40100000000000002,"15min":0.41199999999999998},"min":{"1min":0.26900000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.49299999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.42999999999999999,"15min":0.438},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.54400000000000004,"5min":0.63,"15min":0.63},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Feb 23 00:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57699999999999996}]},{"osd":5,"last update":"Fri Feb 23 00:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.622},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58799999999999997,"15min":0.59299999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.56799999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77600000000000002,"5min":1.117,"15min":1.117},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80600000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.66500000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":289924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.36699999999999999,"15min":0.371},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.49199999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.249},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.38400000000000001,"15min":0.39700000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.50800000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.26500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.40699999999999997,"15min":0.42699999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.437,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.52600000000000002,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.28799999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.43099999999999999,"15min":0.439},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54900000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.47799999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.30299999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.499},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.30299999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68799999999999994,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65500000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56499999999999995,"15min":0.56899999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72899999999999998,"5min":0.752,"15min":0.752},"last":0.71599999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76600000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.59299999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.61199999999999999},"min":{"1min":0.498,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72499999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":289907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.35699999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66600000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.33600000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.5,"15min":0.52000000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51000000000000001,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.65200000000000002,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.309}]},{"osd":2,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67200000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.379,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.70299999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.433}]},{"osd":3,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69099999999999995,"5min":0.755,"15min":0.755},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.50800000000000001,"15min":0.53000000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.40600000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.36299999999999999,"15min":0.36099999999999999},"min":{"1min":0.308,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.39400000000000002},"min":{"1min":0.28000000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.55500000000000005,"5min":0.62,"15min":0.62},"last":0.46899999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.34599999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.55500000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.43099999999999999,"15min":0.42899999999999999},"min":{"1min":0.29399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.56399999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.45600000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.46999999999999997,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54800000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757287,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":289441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.505},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70599999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.498},{"interface":"front","average":{"1min":0.497,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.34599999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.623}]},{"osd":1,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47999999999999998,"15min":0.47799999999999998},"min":{"1min":0.34999999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67100000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.49399999999999999,"15min":0.495},"min":{"1min":0.39900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.70399999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58299999999999996}]},{"osd":2,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66100000000000003,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36699999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73299999999999998,"5min":0.753,"15min":0.753},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.38600000000000001,"15min":0.371},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39400000000000002,"15min":0.372},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.39500000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.434,"15min":0.42399999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.442,"15min":0.434},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69599999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:29:26.242 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:29:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:24.807896+0000 osd.2 (osd.2) 33 : cluster [DBG] 2.c scrub starts 2024-02-23T00:29:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:24.809184+0000 osd.2 (osd.2) 34 : cluster [DBG] 2.c scrub ok 2024-02-23T00:29:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:25.264792+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 deep-scrub starts 2024-02-23T00:29:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:25.267755+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 deep-scrub ok 2024-02-23T00:29:26.543 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:24.807896+0000 osd.2 (osd.2) 33 : cluster [DBG] 2.c scrub starts 2024-02-23T00:29:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:24.809184+0000 osd.2 (osd.2) 34 : cluster [DBG] 2.c scrub ok 2024-02-23T00:29:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:25.264792+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 deep-scrub starts 2024-02-23T00:29:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:25.267755+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 deep-scrub ok 2024-02-23T00:29:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:24.807896+0000 osd.2 (osd.2) 33 : cluster [DBG] 2.c scrub starts 2024-02-23T00:29:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:24.809184+0000 osd.2 (osd.2) 34 : cluster [DBG] 2.c scrub ok 2024-02-23T00:29:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:25.264792+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 deep-scrub starts 2024-02-23T00:29:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:25.267755+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 deep-scrub ok 2024-02-23T00:29:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:25.480875+0000 mgr.y (mgr.24437) 87 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:29:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:25.520875+0000 mgr.y (mgr.24437) 88 : audit [DBG] from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:25.480875+0000 mgr.y (mgr.24437) 87 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:29:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:25.520875+0000 mgr.y (mgr.24437) 88 : audit [DBG] from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:25.480875+0000 mgr.y (mgr.24437) 87 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:29:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:25.520875+0000 mgr.y (mgr.24437) 88 : audit [DBG] from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:28.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.351+0000 7f4c34210700 1 -- 172.21.15.87:0/2332118172 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0f7400 msgr2=0x7f4c2c103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:28.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.351+0000 7f4c34210700 1 --2- 172.21.15.87:0/2332118172 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0f7400 0x7f4c2c103a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f4c24009c30 tx=0x7f4c24004c50).stop 2024-02-23T00:29:28.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.351+0000 7f4c34210700 1 -- 172.21.15.87:0/2332118172 shutdown_connections 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.351+0000 7f4c34210700 1 --2- 172.21.15.87:0/2332118172 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0f7400 0x7f4c2c103a20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.351+0000 7f4c34210700 1 --2- 172.21.15.87:0/2332118172 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c2c0fcbb0 0x7f4c2c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.351+0000 7f4c34210700 1 --2- 172.21.15.87:0/2332118172 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c2c0fd510 0x7f4c2c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.351+0000 7f4c34210700 1 -- 172.21.15.87:0/2332118172 >> 172.21.15.87:0/2332118172 conn(0x7f4c2c0f2820 msgr2=0x7f4c2c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.351+0000 7f4c34210700 1 -- 172.21.15.87:0/2332118172 shutdown_connections 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.351+0000 7f4c34210700 1 -- 172.21.15.87:0/2332118172 wait complete. 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 librados: starting msgr at 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 librados: starting objecter 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 Processor -- start 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 -- start start 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 librados: setting wanted keys 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 librados: calling monclient init 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c2c0f7400 0x7f4c2c109090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:28.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0fcbb0 0x7f4c2c109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c2c0fd510 0x7f4c2c109b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c114270 con 0x7f4c2c0fd510 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c1143e0 con 0x7f4c2c0f7400 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c34210700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c1145a0 con 0x7f4c2c0fcbb0 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c317ab700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0fcbb0 0x7f4c2c109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c327ad700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c2c0fd510 0x7f4c2c109b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c327ad700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c2c0fd510 0x7f4c2c109b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35452/0 (socket says 172.21.15.87:35452) 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c317ab700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0fcbb0 0x7f4c2c109580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34408/0 (socket says 172.21.15.87:34408) 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c327ad700 1 -- 172.21.15.87:0/3139008341 learned_addr learned my addr 172.21.15.87:0/3139008341 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c31fac700 1 --2- 172.21.15.87:0/3139008341 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c2c0f7400 0x7f4c2c109090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c327ad700 1 -- 172.21.15.87:0/3139008341 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0fcbb0 msgr2=0x7f4c2c109580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c327ad700 1 --2- 172.21.15.87:0/3139008341 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0fcbb0 0x7f4c2c109580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:28.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c327ad700 1 -- 172.21.15.87:0/3139008341 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c2c0f7400 msgr2=0x7f4c2c109090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c327ad700 1 --2- 172.21.15.87:0/3139008341 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c2c0f7400 0x7f4c2c109090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c327ad700 1 -- 172.21.15.87:0/3139008341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c005040 con 0x7f4c2c0fd510 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c317ab700 1 --2- 172.21.15.87:0/3139008341 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0fcbb0 0x7f4c2c109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c327ad700 1 --2- 172.21.15.87:0/3139008341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c2c0fd510 0x7f4c2c109b60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f4c24004e10 tx=0x7f4c24004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c22ffd700 10 client.?.objecter ms_handle_connect 0x7f4c2c0fd510 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.352+0000 7f4c22ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.353+0000 7f4c22ffd700 1 -- 172.21.15.87:0/3139008341 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2400e040 con 0x7f4c2c0fd510 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.353+0000 7f4c22ffd700 1 -- 172.21.15.87:0/3139008341 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c24018040 con 0x7f4c2c0fd510 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.353+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c240097c0 con 0x7f4c2c0fd510 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.353+0000 7f4c22ffd700 1 -- 172.21.15.87:0/3139008341 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24003ae0 con 0x7f4c2c0fd510 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.353+0000 7f4c34210700 10 client.14709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.353+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c1149e0 con 0x7f4c2c0fd510 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.353+0000 7f4c34210700 1 librados: init done 2024-02-23T00:29:28.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.354+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c05fa60 con 0x7f4c2c0fd510 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c22ffd700 1 -- 172.21.15.87:0/3139008341 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4c24004020 con 0x7f4c2c0fd510 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c22ffd700 1 --2- 172.21.15.87:0/3139008341 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4c18062c90 0x7f4c18065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c22ffd700 1 -- 172.21.15.87:0/3139008341 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(82..82 src has 28..82) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f4c2407e140 con 0x7f4c2c0fd510 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c22ffd700 10 client.14709.objecter ms_dispatch 0x7f4c2c001550 osd_map(82..82 src has 28..82) v4 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c22ffd700 3 client.14709.objecter handle_osd_map got epochs [82,82] > 0 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c22ffd700 3 client.14709.objecter handle_osd_map decoding full epoch 82 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c31fac700 1 --2- 172.21.15.87:0/3139008341 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4c18062c90 0x7f4c18065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c22ffd700 20 client.14709.objecter dump_active .. 0 homeless 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c22ffd700 10 client.14709.objecter ms_handle_connect 0x7f4c18062c90 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.355+0000 7f4c31fac700 1 --2- 172.21.15.87:0/3139008341 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4c18062c90 0x7f4c18065150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f4c1c004870 tx=0x7f4c1c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:28.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.359+0000 7f4c22ffd700 1 -- 172.21.15.87:0/3139008341 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4c24051e50 con 0x7f4c2c0fd510 2024-02-23T00:29:28.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.517+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c2c061440 con 0x7f4c18062c90 2024-02-23T00:29:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.527+0000 7f4c22ffd700 1 -- 172.21.15.87:0/3139008341 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357299 (secure 0 0 0) 0x7f4c2c061440 con 0x7f4c18062c90 2024-02-23T00:29:28.533 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:29:28.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 10 librados: watch_flush enter 2024-02-23T00:29:28.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 10 librados: watch_flush exit 2024-02-23T00:29:28.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 20 client.14709.objecter shutdown clearing up homeless session... 2024-02-23T00:29:28.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 10 client.14709.objecter successfully canceled tick 2024-02-23T00:29:28.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4c18062c90 msgr2=0x7f4c18065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:28.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 --2- 172.21.15.87:0/3139008341 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4c18062c90 0x7f4c18065150 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f4c1c004870 tx=0x7f4c1c00b040).stop 2024-02-23T00:29:28.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c2c0fd510 msgr2=0x7f4c2c109b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:28.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 --2- 172.21.15.87:0/3139008341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c2c0fd510 0x7f4c2c109b60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f4c24004e10 tx=0x7f4c24004c50).stop 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 shutdown_connections 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 --2- 172.21.15.87:0/3139008341 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4c18062c90 0x7f4c18065150 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 --2- 172.21.15.87:0/3139008341 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4c2c0fd510 0x7f4c2c109b60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 --2- 172.21.15.87:0/3139008341 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4c2c0f7400 0x7f4c2c109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 --2- 172.21.15.87:0/3139008341 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4c2c0fcbb0 0x7f4c2c109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 >> 172.21.15.87:0/3139008341 conn(0x7f4c2c0f2820 msgr2=0x7f4c2c0fab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 shutdown_connections 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 -- 172.21.15.87:0/3139008341 wait complete. 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:28.530+0000 7f4c34210700 1 librados: shutdown 2024-02-23T00:29:28.537 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:29:29.323 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:27.481277+0000 mgr.y (mgr.24437) 89 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:29.323 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:27.873849+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.e scrub starts 2024-02-23T00:29:29.323 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:27.875102+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.e scrub ok 2024-02-23T00:29:29.323 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:27.481277+0000 mgr.y (mgr.24437) 89 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:29.324 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:27.873849+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.e scrub starts 2024-02-23T00:29:29.324 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:27.875102+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.e scrub ok 2024-02-23T00:29:29.345 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-02-23T00:29:27.481118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1226,"num_read_kb":946,"num_write":620,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":694,"ondisk_log_size":694,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381744,"kb_used_data":4976,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579472,"statfs":{"total":767960285184,"available":765521379328,"internally_reserved":0,"allocated":5095424,"data_stored":2321961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002683"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165705+0000","last_change":"2024-02-23T00:28:52.093263+0000","last_active":"2024-02-23T00:29:04.165705+0000","last_peered":"2024-02-23T00:29:04.165705+0000","last_clean":"2024-02-23T00:29:04.165705+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:29:04.165705+0000","last_undegraded":"2024-02-23T00:29:04.165705+0000","last_fullsized":"2024-02-23T00:29:04.165705+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:52.093177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.093177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.109974+0000","last_change":"2024-02-23T00:28:57.678604+0000","last_active":"2024-02-23T00:29:04.109974+0000","last_peered":"2024-02-23T00:29:04.109974+0000","last_clean":"2024-02-23T00:29:04.109974+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:29:04.109974+0000","last_undegraded":"2024-02-23T00:29:04.109974+0000","last_fullsized":"2024-02-23T00:29:04.109974+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:57.678522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.678522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166389+0000","last_change":"2024-02-23T00:28:46.217573+0000","last_active":"2024-02-23T00:29:04.166389+0000","last_peered":"2024-02-23T00:29:04.166389+0000","last_clean":"2024-02-23T00:29:04.166389+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:29:04.166389+0000","last_undegraded":"2024-02-23T00:29:04.166389+0000","last_fullsized":"2024-02-23T00:29:04.166389+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:46.217507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.217507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'39","reported_seq":124,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165323+0000","last_change":"2024-02-23T00:28:37.040298+0000","last_active":"2024-02-23T00:29:04.165323+0000","last_peered":"2024-02-23T00:29:04.165323+0000","last_clean":"2024-02-23T00:29:04.165323+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:29:04.165323+0000","last_undegraded":"2024-02-23T00:29:04.165323+0000","last_fullsized":"2024-02-23T00:29:04.165323+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'39","last_scrub_stamp":"2024-02-23T00:28:37.040215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:37.040215+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110191+0000","last_change":"2024-02-23T00:28:59.685150+0000","last_active":"2024-02-23T00:29:04.110191+0000","last_peered":"2024-02-23T00:29:04.110191+0000","last_clean":"2024-02-23T00:29:04.110191+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:29:04.110191+0000","last_undegraded":"2024-02-23T00:29:04.110191+0000","last_fullsized":"2024-02-23T00:29:04.110191+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:59.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:59.685045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112331+0000","last_change":"2024-02-23T00:28:49.663875+0000","last_active":"2024-02-23T00:29:04.112331+0000","last_peered":"2024-02-23T00:29:04.112331+0000","last_clean":"2024-02-23T00:29:04.112331+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:29:04.112331+0000","last_undegraded":"2024-02-23T00:29:04.112331+0000","last_fullsized":"2024-02-23T00:29:04.112331+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.663808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.663808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166335+0000","last_change":"2024-02-23T00:28:47.073631+0000","last_active":"2024-02-23T00:29:04.166335+0000","last_peered":"2024-02-23T00:29:04.166335+0000","last_clean":"2024-02-23T00:29:04.166335+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:29:04.166335+0000","last_undegraded":"2024-02-23T00:29:04.166335+0000","last_fullsized":"2024-02-23T00:29:04.166335+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:47.073551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.073551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":75,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:06.822930+0000","last_change":"2024-02-23T00:29:06.822899+0000","last_active":"2024-02-23T00:29:06.822930+0000","last_peered":"2024-02-23T00:29:06.822930+0000","last_clean":"2024-02-23T00:29:06.822930+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:29:06.822930+0000","last_undegraded":"2024-02-23T00:29:06.822930+0000","last_fullsized":"2024-02-23T00:29:06.822930+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:29:06.822818+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167297+0000","last_change":"2024-02-23T00:29:04.049502+0000","last_active":"2024-02-23T00:29:04.167297+0000","last_peered":"2024-02-23T00:29:04.167297+0000","last_clean":"2024-02-23T00:29:04.167297+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:29:04.167297+0000","last_undegraded":"2024-02-23T00:29:04.167297+0000","last_fullsized":"2024-02-23T00:29:04.167297+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.049414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166268+0000","last_change":"2024-02-23T00:29:00.320083+0000","last_active":"2024-02-23T00:29:04.166268+0000","last_peered":"2024-02-23T00:29:04.166268+0000","last_clean":"2024-02-23T00:29:04.166268+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:29:04.166268+0000","last_undegraded":"2024-02-23T00:29:04.166268+0000","last_fullsized":"2024-02-23T00:29:04.166268+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.319985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.319985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167512+0000","last_change":"2024-02-23T00:28:46.396553+0000","last_active":"2024-02-23T00:29:04.167512+0000","last_peered":"2024-02-23T00:29:04.167512+0000","last_clean":"2024-02-23T00:29:04.167512+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:29:04.167512+0000","last_undegraded":"2024-02-23T00:29:04.167512+0000","last_fullsized":"2024-02-23T00:29:04.167512+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.396471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110342+0000","last_change":"2024-02-23T00:28:54.657266+0000","last_active":"2024-02-23T00:29:04.110342+0000","last_peered":"2024-02-23T00:29:04.110342+0000","last_clean":"2024-02-23T00:29:04.110342+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:29:04.110342+0000","last_undegraded":"2024-02-23T00:29:04.110342+0000","last_fullsized":"2024-02-23T00:29:04.110342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.657177+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165496+0000","last_change":"2024-02-23T00:28:50.176282+0000","last_active":"2024-02-23T00:29:04.165496+0000","last_peered":"2024-02-23T00:29:04.165496+0000","last_clean":"2024-02-23T00:29:04.165496+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:29:04.165496+0000","last_undegraded":"2024-02-23T00:29:04.165496+0000","last_fullsized":"2024-02-23T00:29:04.165496+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.176208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167187+0000","last_change":"2024-02-23T00:28:33.933658+0000","last_active":"2024-02-23T00:29:04.167187+0000","last_peered":"2024-02-23T00:29:04.167187+0000","last_clean":"2024-02-23T00:29:04.167187+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:29:04.167187+0000","last_undegraded":"2024-02-23T00:29:04.167187+0000","last_fullsized":"2024-02-23T00:29:04.167187+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.933529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.933529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.267535+0000","last_change":"2024-02-23T00:29:04.267507+0000","last_active":"2024-02-23T00:29:04.267535+0000","last_peered":"2024-02-23T00:29:04.267535+0000","last_clean":"2024-02-23T00:29:04.267535+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:29:04.267535+0000","last_undegraded":"2024-02-23T00:29:04.267535+0000","last_fullsized":"2024-02-23T00:29:04.267535+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.267427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.267427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110473+0000","last_change":"2024-02-23T00:28:25.457421+0000","last_active":"2024-02-23T00:29:04.110473+0000","last_peered":"2024-02-23T00:29:04.110473+0000","last_clean":"2024-02-23T00:29:04.110473+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:29:04.110473+0000","last_undegraded":"2024-02-23T00:29:04.110473+0000","last_fullsized":"2024-02-23T00:29:04.110473+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:28:25.457329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:25.457329+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166564+0000","last_change":"2024-02-23T00:28:29.086512+0000","last_active":"2024-02-23T00:29:04.166564+0000","last_peered":"2024-02-23T00:29:04.166564+0000","last_clean":"2024-02-23T00:29:04.166564+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:29:04.166564+0000","last_undegraded":"2024-02-23T00:29:04.166564+0000","last_fullsized":"2024-02-23T00:29:04.166564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:29.086437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:29.086437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110635+0000","last_change":"2024-02-23T00:29:03.790608+0000","last_active":"2024-02-23T00:29:04.110635+0000","last_peered":"2024-02-23T00:29:04.110635+0000","last_clean":"2024-02-23T00:29:04.110635+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:29:04.110635+0000","last_undegraded":"2024-02-23T00:29:04.110635+0000","last_fullsized":"2024-02-23T00:29:04.110635+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.790530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.790530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165837+0000","last_change":"2024-02-23T00:28:30.086083+0000","last_active":"2024-02-23T00:29:04.165837+0000","last_peered":"2024-02-23T00:29:04.165837+0000","last_clean":"2024-02-23T00:29:04.165837+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:29:04.165837+0000","last_undegraded":"2024-02-23T00:29:04.165837+0000","last_fullsized":"2024-02-23T00:29:04.165837+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:30.086009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:30.086009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":83,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166086+0000","last_change":"2024-02-23T00:28:33.077947+0000","last_active":"2024-02-23T00:29:04.166086+0000","last_peered":"2024-02-23T00:29:04.166086+0000","last_clean":"2024-02-23T00:29:04.166086+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:29:04.166086+0000","last_undegraded":"2024-02-23T00:29:04.166086+0000","last_fullsized":"2024-02-23T00:29:04.166086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.077848+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.830507+0000","last_change":"2024-02-23T00:29:10.830479+0000","last_active":"2024-02-23T00:29:10.830507+0000","last_peered":"2024-02-23T00:29:10.830507+0000","last_clean":"2024-02-23T00:29:10.830507+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:29:10.830507+0000","last_undegraded":"2024-02-23T00:29:10.830507+0000","last_fullsized":"2024-02-23T00:29:10.830507+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.830389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.830389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:05.641550+0000","last_change":"2024-02-23T00:29:05.641525+0000","last_active":"2024-02-23T00:29:05.641550+0000","last_peered":"2024-02-23T00:29:05.641550+0000","last_clean":"2024-02-23T00:29:05.641550+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:29:05.641550+0000","last_undegraded":"2024-02-23T00:29:05.641550+0000","last_fullsized":"2024-02-23T00:29:05.641550+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_clean_scrub_stamp":"2024-02-23T00:29:05.641454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165791+0000","last_change":"2024-02-23T00:28:17.030758+0000","last_active":"2024-02-23T00:29:04.165791+0000","last_peered":"2024-02-23T00:29:04.165791+0000","last_clean":"2024-02-23T00:29:04.165791+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:29:04.165791+0000","last_undegraded":"2024-02-23T00:29:04.165791+0000","last_fullsized":"2024-02-23T00:29:04.165791+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:17.030655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:17.030655+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.113086+0000","last_change":"2024-02-23T00:28:27.557204+0000","last_active":"2024-02-23T00:29:04.113086+0000","last_peered":"2024-02-23T00:29:04.113086+0000","last_clean":"2024-02-23T00:29:04.113086+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:29:04.113086+0000","last_undegraded":"2024-02-23T00:29:04.113086+0000","last_fullsized":"2024-02-23T00:29:04.113086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:28:27.557140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:27.557140+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111713+0000","last_change":"2024-02-23T00:28:27.524062+0000","last_active":"2024-02-23T00:29:04.111713+0000","last_peered":"2024-02-23T00:29:04.111713+0000","last_clean":"2024-02-23T00:29:04.111713+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:29:04.111713+0000","last_undegraded":"2024-02-23T00:29:04.111713+0000","last_fullsized":"2024-02-23T00:29:04.111713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:27.523959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:27.523959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.130821+0000","last_change":"2024-02-23T00:29:10.130797+0000","last_active":"2024-02-23T00:29:10.130821+0000","last_peered":"2024-02-23T00:29:10.130821+0000","last_clean":"2024-02-23T00:29:10.130821+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:29:10.130821+0000","last_undegraded":"2024-02-23T00:29:10.130821+0000","last_fullsized":"2024-02-23T00:29:10.130821+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.130727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.130727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165445+0000","last_change":"2024-02-23T00:28:32.090929+0000","last_active":"2024-02-23T00:29:04.165445+0000","last_peered":"2024-02-23T00:29:04.165445+0000","last_clean":"2024-02-23T00:29:04.165445+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:29:04.165445+0000","last_undegraded":"2024-02-23T00:29:04.165445+0000","last_fullsized":"2024-02-23T00:29:04.165445+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:32.090869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:32.090869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167021+0000","last_change":"2024-02-23T00:28:44.987515+0000","last_active":"2024-02-23T00:29:04.167021+0000","last_peered":"2024-02-23T00:29:04.167021+0000","last_clean":"2024-02-23T00:29:04.167021+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:29:04.167021+0000","last_undegraded":"2024-02-23T00:29:04.167021+0000","last_fullsized":"2024-02-23T00:29:04.167021+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:28:44.987419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.987419+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167041+0000","last_change":"2024-02-23T00:28:48.333707+0000","last_active":"2024-02-23T00:29:04.167041+0000","last_peered":"2024-02-23T00:29:04.167041+0000","last_clean":"2024-02-23T00:29:04.167041+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:29:04.167041+0000","last_undegraded":"2024-02-23T00:29:04.167041+0000","last_fullsized":"2024-02-23T00:29:04.167041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:28:48.333638+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"62'8","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.349098+0000","last_change":"2024-02-23T00:28:36.071048+0000","last_active":"2024-02-23T00:29:20.349098+0000","last_peered":"2024-02-23T00:29:20.349098+0000","last_clean":"2024-02-23T00:29:20.349098+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:29:20.349098+0000","last_undegraded":"2024-02-23T00:29:20.349098+0000","last_fullsized":"2024-02-23T00:29:20.349098+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:36.070959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.070959+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165452+0000","last_change":"2024-02-23T00:29:03.017128+0000","last_active":"2024-02-23T00:29:04.165452+0000","last_peered":"2024-02-23T00:29:04.165452+0000","last_clean":"2024-02-23T00:29:04.165452+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:29:04.165452+0000","last_undegraded":"2024-02-23T00:29:04.165452+0000","last_fullsized":"2024-02-23T00:29:04.165452+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.017051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.017051+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'16","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166313+0000","last_change":"2024-02-23T00:28:55.230148+0000","last_active":"2024-02-23T00:29:04.166313+0000","last_peered":"2024-02-23T00:29:04.166313+0000","last_clean":"2024-02-23T00:29:04.166313+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:29:04.166313+0000","last_undegraded":"2024-02-23T00:29:04.166313+0000","last_fullsized":"2024-02-23T00:29:04.166313+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:28:55.230081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.230081+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165952+0000","last_change":"2024-02-23T00:28:41.113778+0000","last_active":"2024-02-23T00:29:04.165952+0000","last_peered":"2024-02-23T00:29:04.165952+0000","last_clean":"2024-02-23T00:29:04.165952+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:29:04.165952+0000","last_undegraded":"2024-02-23T00:29:04.165952+0000","last_fullsized":"2024-02-23T00:29:04.165952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.113684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.113684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167420+0000","last_change":"2024-02-23T00:28:41.520505+0000","last_active":"2024-02-23T00:29:04.167420+0000","last_peered":"2024-02-23T00:29:04.167420+0000","last_clean":"2024-02-23T00:29:04.167420+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:29:04.167420+0000","last_undegraded":"2024-02-23T00:29:04.167420+0000","last_fullsized":"2024-02-23T00:29:04.167420+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.520412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.520412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167154+0000","last_change":"2024-02-23T00:28:37.106379+0000","last_active":"2024-02-23T00:29:04.167154+0000","last_peered":"2024-02-23T00:29:04.167154+0000","last_clean":"2024-02-23T00:29:04.167154+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:29:04.167154+0000","last_undegraded":"2024-02-23T00:29:04.167154+0000","last_fullsized":"2024-02-23T00:29:04.167154+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:37.106309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:37.106309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":80,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165684+0000","last_change":"2024-02-23T00:28:57.039721+0000","last_active":"2024-02-23T00:29:04.165684+0000","last_peered":"2024-02-23T00:29:04.165684+0000","last_clean":"2024-02-23T00:29:04.165684+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:29:04.165684+0000","last_undegraded":"2024-02-23T00:29:04.165684+0000","last_fullsized":"2024-02-23T00:29:04.165684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.039660+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166057+0000","last_change":"2024-02-23T00:28:43.440882+0000","last_active":"2024-02-23T00:29:04.166057+0000","last_peered":"2024-02-23T00:29:04.166057+0000","last_clean":"2024-02-23T00:29:04.166057+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:29:04.166057+0000","last_undegraded":"2024-02-23T00:29:04.166057+0000","last_fullsized":"2024-02-23T00:29:04.166057+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.440788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.440788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166062+0000","last_change":"2024-02-23T00:28:56.022911+0000","last_active":"2024-02-23T00:29:04.166062+0000","last_peered":"2024-02-23T00:29:04.166062+0000","last_clean":"2024-02-23T00:29:04.166062+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:29:04.166062+0000","last_undegraded":"2024-02-23T00:29:04.166062+0000","last_fullsized":"2024-02-23T00:29:04.166062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:56.022835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:56.022835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167959+0000","last_change":"2024-02-23T00:28:24.002963+0000","last_active":"2024-02-23T00:29:04.167959+0000","last_peered":"2024-02-23T00:29:04.167959+0000","last_clean":"2024-02-23T00:29:04.167959+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:29:04.167959+0000","last_undegraded":"2024-02-23T00:29:04.167959+0000","last_fullsized":"2024-02-23T00:29:04.167959+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:24.002878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:24.002878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165126+0000","last_change":"2024-02-23T00:28:20.075559+0000","last_active":"2024-02-23T00:29:04.165126+0000","last_peered":"2024-02-23T00:29:04.165126+0000","last_clean":"2024-02-23T00:29:04.165126+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:29:04.165126+0000","last_undegraded":"2024-02-23T00:29:04.165126+0000","last_fullsized":"2024-02-23T00:29:04.165126+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:28:20.075487+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165247+0000","last_change":"2024-02-23T00:28:39.076715+0000","last_active":"2024-02-23T00:29:04.165247+0000","last_peered":"2024-02-23T00:29:04.165247+0000","last_clean":"2024-02-23T00:29:04.165247+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:29:04.165247+0000","last_undegraded":"2024-02-23T00:29:04.165247+0000","last_fullsized":"2024-02-23T00:29:04.165247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.076631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.076631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111882+0000","last_change":"2024-02-23T00:28:53.665149+0000","last_active":"2024-02-23T00:29:04.111882+0000","last_peered":"2024-02-23T00:29:04.111882+0000","last_clean":"2024-02-23T00:29:04.111882+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:29:04.111882+0000","last_undegraded":"2024-02-23T00:29:04.111882+0000","last_fullsized":"2024-02-23T00:29:04.111882+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.665092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111817+0000","last_change":"2024-02-23T00:29:02.711400+0000","last_active":"2024-02-23T00:29:04.111817+0000","last_peered":"2024-02-23T00:29:04.111817+0000","last_clean":"2024-02-23T00:29:04.111817+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:29:04.111817+0000","last_undegraded":"2024-02-23T00:29:04.111817+0000","last_fullsized":"2024-02-23T00:29:04.111817+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.711284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":79,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110548+0000","last_change":"2024-02-23T00:28:41.214195+0000","last_active":"2024-02-23T00:29:04.110548+0000","last_peered":"2024-02-23T00:29:04.110548+0000","last_clean":"2024-02-23T00:29:04.110548+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:29:04.110548+0000","last_undegraded":"2024-02-23T00:29:04.110548+0000","last_fullsized":"2024-02-23T00:29:04.110548+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:28:36.536962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.536962+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165027+0000","last_change":"2024-02-23T00:28:49.187859+0000","last_active":"2024-02-23T00:29:04.165027+0000","last_peered":"2024-02-23T00:29:04.165027+0000","last_clean":"2024-02-23T00:29:04.165027+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:29:04.165027+0000","last_undegraded":"2024-02-23T00:29:04.165027+0000","last_fullsized":"2024-02-23T00:29:04.165027+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.187786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.187786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165874+0000","last_change":"2024-02-23T00:28:22.160736+0000","last_active":"2024-02-23T00:29:04.165874+0000","last_peered":"2024-02-23T00:29:04.165874+0000","last_clean":"2024-02-23T00:29:04.165874+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:29:04.165874+0000","last_undegraded":"2024-02-23T00:29:04.165874+0000","last_fullsized":"2024-02-23T00:29:04.165874+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:22.160642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:22.160642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166211+0000","last_change":"2024-02-23T00:28:21.021722+0000","last_active":"2024-02-23T00:29:04.166211+0000","last_peered":"2024-02-23T00:29:04.166211+0000","last_clean":"2024-02-23T00:29:04.166211+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:29:04.166211+0000","last_undegraded":"2024-02-23T00:29:04.166211+0000","last_fullsized":"2024-02-23T00:29:04.166211+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:21.021619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.021619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111782+0000","last_change":"2024-02-23T00:28:36.676648+0000","last_active":"2024-02-23T00:29:04.111782+0000","last_peered":"2024-02-23T00:29:04.111782+0000","last_clean":"2024-02-23T00:29:04.111782+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:29:04.111782+0000","last_undegraded":"2024-02-23T00:29:04.111782+0000","last_fullsized":"2024-02-23T00:29:04.111782+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:28:36.676582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.676582+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111356+0000","last_change":"2024-02-23T00:28:43.560702+0000","last_active":"2024-02-23T00:29:04.111356+0000","last_peered":"2024-02-23T00:29:04.111356+0000","last_clean":"2024-02-23T00:29:04.111356+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:29:04.111356+0000","last_undegraded":"2024-02-23T00:29:04.111356+0000","last_fullsized":"2024-02-23T00:29:04.111356+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.560606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.560606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"62'8","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348794+0000","last_change":"2024-02-23T00:28:21.123596+0000","last_active":"2024-02-23T00:29:20.348794+0000","last_peered":"2024-02-23T00:29:20.348794+0000","last_clean":"2024-02-23T00:29:20.348794+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:29:20.348794+0000","last_undegraded":"2024-02-23T00:29:20.348794+0000","last_fullsized":"2024-02-23T00:29:20.348794+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:21.123498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.123498+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165830+0000","last_change":"2024-02-23T00:28:30.236265+0000","last_active":"2024-02-23T00:29:04.165830+0000","last_peered":"2024-02-23T00:29:04.165830+0000","last_clean":"2024-02-23T00:29:04.165830+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:29:04.165830+0000","last_undegraded":"2024-02-23T00:29:04.165830+0000","last_fullsized":"2024-02-23T00:29:04.165830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:30.236188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:30.236188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'15","reported_seq":68,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166655+0000","last_change":"2024-02-23T00:28:55.907616+0000","last_active":"2024-02-23T00:29:04.166655+0000","last_peered":"2024-02-23T00:29:04.166655+0000","last_clean":"2024-02-23T00:29:04.166655+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:29:04.166655+0000","last_undegraded":"2024-02-23T00:29:04.166655+0000","last_fullsized":"2024-02-23T00:29:04.166655+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:28:55.907528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.907528+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:14.086398+0000","last_change":"2024-02-23T00:29:14.086374+0000","last_active":"2024-02-23T00:29:14.086398+0000","last_peered":"2024-02-23T00:29:14.086398+0000","last_clean":"2024-02-23T00:29:14.086398+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:29:14.086398+0000","last_undegraded":"2024-02-23T00:29:14.086398+0000","last_fullsized":"2024-02-23T00:29:14.086398+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:14.086293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:14.086293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.349069+0000","last_change":"2024-02-23T00:28:38.956507+0000","last_active":"2024-02-23T00:29:20.349069+0000","last_peered":"2024-02-23T00:29:20.349069+0000","last_clean":"2024-02-23T00:29:20.349069+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:29:20.349069+0000","last_undegraded":"2024-02-23T00:29:20.349069+0000","last_fullsized":"2024-02-23T00:29:20.349069+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_clean_scrub_stamp":"2024-02-23T00:28:38.956431+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165830+0000","last_change":"2024-02-23T00:28:49.012817+0000","last_active":"2024-02-23T00:29:04.165830+0000","last_peered":"2024-02-23T00:29:04.165830+0000","last_clean":"2024-02-23T00:29:04.165830+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:29:04.165830+0000","last_undegraded":"2024-02-23T00:29:04.165830+0000","last_fullsized":"2024-02-23T00:29:04.165830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.012698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":84,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166786+0000","last_change":"2024-02-23T00:29:01.026660+0000","last_active":"2024-02-23T00:29:04.166786+0000","last_peered":"2024-02-23T00:29:04.166786+0000","last_clean":"2024-02-23T00:29:04.166786+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:29:04.166786+0000","last_undegraded":"2024-02-23T00:29:04.166786+0000","last_fullsized":"2024-02-23T00:29:04.166786+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:29:01.026575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.026575+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167407+0000","last_change":"2024-02-23T00:28:58.383733+0000","last_active":"2024-02-23T00:29:04.167407+0000","last_peered":"2024-02-23T00:29:04.167407+0000","last_clean":"2024-02-23T00:29:04.167407+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:29:04.167407+0000","last_undegraded":"2024-02-23T00:29:04.167407+0000","last_fullsized":"2024-02-23T00:29:04.167407+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:28:58.383642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.044071+0000","last_change":"2024-02-23T00:29:10.044046+0000","last_active":"2024-02-23T00:29:10.044071+0000","last_peered":"2024-02-23T00:29:10.044071+0000","last_clean":"2024-02-23T00:29:10.044071+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:29:10.044071+0000","last_undegraded":"2024-02-23T00:29:10.044071+0000","last_fullsized":"2024-02-23T00:29:10.044071+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.043966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.043966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166141+0000","last_change":"2024-02-23T00:28:34.066522+0000","last_active":"2024-02-23T00:29:04.166141+0000","last_peered":"2024-02-23T00:29:04.166141+0000","last_clean":"2024-02-23T00:29:04.166141+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:29:04.166141+0000","last_undegraded":"2024-02-23T00:29:04.166141+0000","last_fullsized":"2024-02-23T00:29:04.166141+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:34.066441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:34.066441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":82,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166525+0000","last_change":"2024-02-23T00:28:42.144931+0000","last_active":"2024-02-23T00:29:04.166525+0000","last_peered":"2024-02-23T00:29:04.166525+0000","last_clean":"2024-02-23T00:29:04.166525+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:29:04.166525+0000","last_undegraded":"2024-02-23T00:29:04.166525+0000","last_fullsized":"2024-02-23T00:29:04.166525+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:28:42.144833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:42.144833+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112842+0000","last_change":"2024-02-23T00:28:53.629641+0000","last_active":"2024-02-23T00:29:04.112842+0000","last_peered":"2024-02-23T00:29:04.112842+0000","last_clean":"2024-02-23T00:29:04.112842+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:29:04.112842+0000","last_undegraded":"2024-02-23T00:29:04.112842+0000","last_fullsized":"2024-02-23T00:29:04.112842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:53.629568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.629568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166601+0000","last_change":"2024-02-23T00:29:01.986532+0000","last_active":"2024-02-23T00:29:04.166601+0000","last_peered":"2024-02-23T00:29:04.166601+0000","last_clean":"2024-02-23T00:29:04.166601+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:29:04.166601+0000","last_undegraded":"2024-02-23T00:29:04.166601+0000","last_fullsized":"2024-02-23T00:29:04.166601+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:01.986461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.986461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112289+0000","last_change":"2024-02-23T00:28:41.611517+0000","last_active":"2024-02-23T00:29:04.112289+0000","last_peered":"2024-02-23T00:29:04.112289+0000","last_clean":"2024-02-23T00:29:04.112289+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:29:04.112289+0000","last_undegraded":"2024-02-23T00:29:04.112289+0000","last_fullsized":"2024-02-23T00:29:04.112289+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.611448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.611448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166225+0000","last_change":"2024-02-23T00:29:01.234433+0000","last_active":"2024-02-23T00:29:04.166225+0000","last_peered":"2024-02-23T00:29:04.166225+0000","last_clean":"2024-02-23T00:29:04.166225+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:29:04.166225+0000","last_undegraded":"2024-02-23T00:29:04.166225+0000","last_fullsized":"2024-02-23T00:29:04.166225+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:29:01.234356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.234356+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165913+0000","last_change":"2024-02-23T00:28:35.058858+0000","last_active":"2024-02-23T00:29:04.165913+0000","last_peered":"2024-02-23T00:29:04.165913+0000","last_clean":"2024-02-23T00:29:04.165913+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:29:04.165913+0000","last_undegraded":"2024-02-23T00:29:04.165913+0000","last_fullsized":"2024-02-23T00:29:04.165913+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:35.058773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:35.058773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111158+0000","last_change":"2024-02-23T00:28:51.684865+0000","last_active":"2024-02-23T00:29:04.111158+0000","last_peered":"2024-02-23T00:29:04.111158+0000","last_clean":"2024-02-23T00:29:04.111158+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:29:04.111158+0000","last_undegraded":"2024-02-23T00:29:04.111158+0000","last_fullsized":"2024-02-23T00:29:04.111158+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:28:51.684789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.684789+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":118,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110345+0000","last_change":"2024-02-23T00:29:01.733374+0000","last_active":"2024-02-23T00:29:04.110345+0000","last_peered":"2024-02-23T00:29:04.110345+0000","last_clean":"2024-02-23T00:29:04.110345+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:29:04.110345+0000","last_undegraded":"2024-02-23T00:29:04.110345+0000","last_fullsized":"2024-02-23T00:29:04.110345+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:29:01.733287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.733287+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112261+0000","last_change":"2024-02-23T00:28:51.634965+0000","last_active":"2024-02-23T00:29:04.112261+0000","last_peered":"2024-02-23T00:29:04.112261+0000","last_clean":"2024-02-23T00:29:04.112261+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:29:04.112261+0000","last_undegraded":"2024-02-23T00:29:04.112261+0000","last_fullsized":"2024-02-23T00:29:04.112261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:51.634901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.634901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.227949+0000","last_change":"2024-02-23T00:29:04.227928+0000","last_active":"2024-02-23T00:29:04.227949+0000","last_peered":"2024-02-23T00:29:04.227949+0000","last_clean":"2024-02-23T00:29:04.227949+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:29:04.227949+0000","last_undegraded":"2024-02-23T00:29:04.227949+0000","last_fullsized":"2024-02-23T00:29:04.227949+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.227864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.227864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111585+0000","last_change":"2024-02-23T00:28:51.664433+0000","last_active":"2024-02-23T00:29:04.111585+0000","last_peered":"2024-02-23T00:29:04.111585+0000","last_clean":"2024-02-23T00:29:04.111585+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:29:04.111585+0000","last_undegraded":"2024-02-23T00:29:04.111585+0000","last_fullsized":"2024-02-23T00:29:04.111585+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.664367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111833+0000","last_change":"2024-02-23T00:29:03.691092+0000","last_active":"2024-02-23T00:29:04.111833+0000","last_peered":"2024-02-23T00:29:04.111833+0000","last_clean":"2024-02-23T00:29:04.111833+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:29:04.111833+0000","last_undegraded":"2024-02-23T00:29:04.111833+0000","last_fullsized":"2024-02-23T00:29:04.111833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:29:03.691018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.691018+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":74,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348722+0000","last_change":"2024-02-23T00:28:42.249575+0000","last_active":"2024-02-23T00:29:20.348722+0000","last_peered":"2024-02-23T00:29:20.348722+0000","last_clean":"2024-02-23T00:29:20.348722+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:29:20.348722+0000","last_undegraded":"2024-02-23T00:29:20.348722+0000","last_fullsized":"2024-02-23T00:29:20.348722+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:42.249510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:42.249510+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166389+0000","last_change":"2024-02-23T00:28:45.134966+0000","last_active":"2024-02-23T00:29:04.166389+0000","last_peered":"2024-02-23T00:29:04.166389+0000","last_clean":"2024-02-23T00:29:04.166389+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:29:04.166389+0000","last_undegraded":"2024-02-23T00:29:04.166389+0000","last_fullsized":"2024-02-23T00:29:04.166389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:45.134840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:45.134840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165915+0000","last_change":"2024-02-23T00:28:50.000698+0000","last_active":"2024-02-23T00:29:04.165915+0000","last_peered":"2024-02-23T00:29:04.165915+0000","last_clean":"2024-02-23T00:29:04.165915+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:29:04.165915+0000","last_undegraded":"2024-02-23T00:29:04.165915+0000","last_fullsized":"2024-02-23T00:29:04.165915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:28:50.000624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.000624+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":128,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348789+0000","last_change":"2024-02-23T00:28:39.629165+0000","last_active":"2024-02-23T00:29:20.348789+0000","last_peered":"2024-02-23T00:29:20.348789+0000","last_clean":"2024-02-23T00:29:20.348789+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:29:20.348789+0000","last_undegraded":"2024-02-23T00:29:20.348789+0000","last_fullsized":"2024-02-23T00:29:20.348789+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:28:39.629075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.629075+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165991+0000","last_change":"2024-02-23T00:28:28.220174+0000","last_active":"2024-02-23T00:29:04.165991+0000","last_peered":"2024-02-23T00:29:04.165991+0000","last_clean":"2024-02-23T00:29:04.165991+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:29:04.165991+0000","last_undegraded":"2024-02-23T00:29:04.165991+0000","last_fullsized":"2024-02-23T00:29:04.165991+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:28.220080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.220080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167568+0000","last_change":"2024-02-23T00:28:39.496236+0000","last_active":"2024-02-23T00:29:04.167568+0000","last_peered":"2024-02-23T00:29:04.167568+0000","last_clean":"2024-02-23T00:29:04.167568+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:29:04.167568+0000","last_undegraded":"2024-02-23T00:29:04.167568+0000","last_fullsized":"2024-02-23T00:29:04.167568+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.496160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.496160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":167,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:24.259201+0000","last_change":"2024-02-23T00:28:52.653896+0000","last_active":"2024-02-23T00:29:24.259201+0000","last_peered":"2024-02-23T00:29:24.259201+0000","last_clean":"2024-02-23T00:29:24.259201+0000","last_became_active":"2024-02-23T00:27:17.009670+0000","last_became_peered":"2024-02-23T00:27:17.009670+0000","last_unstale":"2024-02-23T00:29:24.259201+0000","last_undegraded":"2024-02-23T00:29:24.259201+0000","last_fullsized":"2024-02-23T00:29:24.259201+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.653823+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.113286+0000","last_change":"2024-02-23T00:28:33.503311+0000","last_active":"2024-02-23T00:29:04.113286+0000","last_peered":"2024-02-23T00:29:04.113286+0000","last_clean":"2024-02-23T00:29:04.113286+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:29:04.113286+0000","last_undegraded":"2024-02-23T00:29:04.113286+0000","last_fullsized":"2024-02-23T00:29:04.113286+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.503241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.503241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":81,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112582+0000","last_change":"2024-02-23T00:28:32.603154+0000","last_active":"2024-02-23T00:29:04.112582+0000","last_peered":"2024-02-23T00:29:04.112582+0000","last_clean":"2024-02-23T00:29:04.112582+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:29:04.112582+0000","last_undegraded":"2024-02-23T00:29:04.112582+0000","last_fullsized":"2024-02-23T00:29:04.112582+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:28:32.603088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:32.603088+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112491+0000","last_change":"2024-02-23T00:28:50.672048+0000","last_active":"2024-02-23T00:29:04.112491+0000","last_peered":"2024-02-23T00:29:04.112491+0000","last_clean":"2024-02-23T00:29:04.112491+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:29:04.112491+0000","last_undegraded":"2024-02-23T00:29:04.112491+0000","last_fullsized":"2024-02-23T00:29:04.112491+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.671988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.671988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.168280+0000","last_change":"2024-02-23T00:28:40.999596+0000","last_active":"2024-02-23T00:29:04.168280+0000","last_peered":"2024-02-23T00:29:04.168280+0000","last_clean":"2024-02-23T00:29:04.168280+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:29:04.168280+0000","last_undegraded":"2024-02-23T00:29:04.168280+0000","last_fullsized":"2024-02-23T00:29:04.168280+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:40.999514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:40.999514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165328+0000","last_change":"2024-02-23T00:28:17.047708+0000","last_active":"2024-02-23T00:29:04.165328+0000","last_peered":"2024-02-23T00:29:04.165328+0000","last_clean":"2024-02-23T00:29:04.165328+0000","last_became_active":"2024-02-23T00:27:17.012683+0000","last_became_peered":"2024-02-23T00:27:17.012683+0000","last_unstale":"2024-02-23T00:29:04.165328+0000","last_undegraded":"2024-02-23T00:29:04.165328+0000","last_fullsized":"2024-02-23T00:29:04.165328+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:17.047614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:17.047614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167144+0000","last_change":"2024-02-23T00:28:52.948687+0000","last_active":"2024-02-23T00:29:04.167144+0000","last_peered":"2024-02-23T00:29:04.167144+0000","last_clean":"2024-02-23T00:29:04.167144+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:29:04.167144+0000","last_undegraded":"2024-02-23T00:29:04.167144+0000","last_fullsized":"2024-02-23T00:29:04.167144+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:52.948601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.948601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:25.267851+0000","last_change":"2024-02-23T00:29:25.267851+0000","last_active":"2024-02-23T00:29:25.267851+0000","last_peered":"2024-02-23T00:29:25.267851+0000","last_clean":"2024-02-23T00:29:25.267851+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:29:25.267851+0000","last_undegraded":"2024-02-23T00:29:25.267851+0000","last_fullsized":"2024-02-23T00:29:25.267851+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:29:25.267773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166538+0000","last_change":"2024-02-23T00:28:33.090534+0000","last_active":"2024-02-23T00:29:04.166538+0000","last_peered":"2024-02-23T00:29:04.166538+0000","last_clean":"2024-02-23T00:29:04.166538+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:29:04.166538+0000","last_undegraded":"2024-02-23T00:29:04.166538+0000","last_fullsized":"2024-02-23T00:29:04.166538+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:28:33.090454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.090454+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166849+0000","last_change":"2024-02-23T00:28:50.344419+0000","last_active":"2024-02-23T00:29:04.166849+0000","last_peered":"2024-02-23T00:29:04.166849+0000","last_clean":"2024-02-23T00:29:04.166849+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:29:04.166849+0000","last_undegraded":"2024-02-23T00:29:04.166849+0000","last_fullsized":"2024-02-23T00:29:04.166849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.344324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.344324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111422+0000","last_change":"2024-02-23T00:28:55.679584+0000","last_active":"2024-02-23T00:29:04.111422+0000","last_peered":"2024-02-23T00:29:04.111422+0000","last_clean":"2024-02-23T00:29:04.111422+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:29:04.111422+0000","last_undegraded":"2024-02-23T00:29:04.111422+0000","last_fullsized":"2024-02-23T00:29:04.111422+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:55.679522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.679522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165482+0000","last_change":"2024-02-23T00:28:59.226967+0000","last_active":"2024-02-23T00:29:04.165482+0000","last_peered":"2024-02-23T00:29:04.165482+0000","last_clean":"2024-02-23T00:29:04.165482+0000","last_became_active":"2024-02-23T00:27:17.009826+0000","last_became_peered":"2024-02-23T00:27:17.009826+0000","last_unstale":"2024-02-23T00:29:04.165482+0000","last_undegraded":"2024-02-23T00:29:04.165482+0000","last_fullsized":"2024-02-23T00:29:04.165482+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:59.226890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:59.226890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166978+0000","last_change":"2024-02-23T00:29:02.340533+0000","last_active":"2024-02-23T00:29:04.166978+0000","last_peered":"2024-02-23T00:29:04.166978+0000","last_clean":"2024-02-23T00:29:04.166978+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:29:04.166978+0000","last_undegraded":"2024-02-23T00:29:04.166978+0000","last_fullsized":"2024-02-23T00:29:04.166978+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.340450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.340450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165741+0000","last_change":"2024-02-23T00:28:36.209181+0000","last_active":"2024-02-23T00:29:04.165741+0000","last_peered":"2024-02-23T00:29:04.165741+0000","last_clean":"2024-02-23T00:29:04.165741+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:29:04.165741+0000","last_undegraded":"2024-02-23T00:29:04.165741+0000","last_fullsized":"2024-02-23T00:29:04.165741+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:28:36.209111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.209111+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.168250+0000","last_change":"2024-02-23T00:28:24.953423+0000","last_active":"2024-02-23T00:29:04.168250+0000","last_peered":"2024-02-23T00:29:04.168250+0000","last_clean":"2024-02-23T00:29:04.168250+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:29:04.168250+0000","last_undegraded":"2024-02-23T00:29:04.168250+0000","last_fullsized":"2024-02-23T00:29:04.168250+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:28:24.953349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:11.580783+0000","last_change":"2024-02-23T00:29:11.580760+0000","last_active":"2024-02-23T00:29:11.580783+0000","last_peered":"2024-02-23T00:29:11.580783+0000","last_clean":"2024-02-23T00:29:11.580783+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:29:11.580783+0000","last_undegraded":"2024-02-23T00:29:11.580783+0000","last_fullsized":"2024-02-23T00:29:11.580783+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:11.580697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:11.580697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167628+0000","last_change":"2024-02-23T00:28:20.772584+0000","last_active":"2024-02-23T00:29:04.167628+0000","last_peered":"2024-02-23T00:29:04.167628+0000","last_clean":"2024-02-23T00:29:04.167628+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:29:04.167628+0000","last_undegraded":"2024-02-23T00:29:04.167628+0000","last_fullsized":"2024-02-23T00:29:04.167628+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:20.772479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:20.772479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":90,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165786+0000","last_change":"2024-02-23T00:28:21.157813+0000","last_active":"2024-02-23T00:29:04.165786+0000","last_peered":"2024-02-23T00:29:04.165786+0000","last_clean":"2024-02-23T00:29:04.165786+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:29:04.165786+0000","last_undegraded":"2024-02-23T00:29:04.165786+0000","last_fullsized":"2024-02-23T00:29:04.165786+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:28:21.157735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.157735+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165865+0000","last_change":"2024-02-23T00:28:43.992150+0000","last_active":"2024-02-23T00:29:04.165865+0000","last_peered":"2024-02-23T00:29:04.165865+0000","last_clean":"2024-02-23T00:29:04.165865+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:29:04.165865+0000","last_undegraded":"2024-02-23T00:29:04.165865+0000","last_fullsized":"2024-02-23T00:29:04.165865+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.992070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.992070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110748+0000","last_change":"2024-02-23T00:28:33.615049+0000","last_active":"2024-02-23T00:29:04.110748+0000","last_peered":"2024-02-23T00:29:04.110748+0000","last_clean":"2024-02-23T00:29:04.110748+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:29:04.110748+0000","last_undegraded":"2024-02-23T00:29:04.110748+0000","last_fullsized":"2024-02-23T00:29:04.110748+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.614962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.614962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112075+0000","last_change":"2024-02-23T00:28:41.617945+0000","last_active":"2024-02-23T00:29:04.112075+0000","last_peered":"2024-02-23T00:29:04.112075+0000","last_clean":"2024-02-23T00:29:04.112075+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:29:04.112075+0000","last_undegraded":"2024-02-23T00:29:04.112075+0000","last_fullsized":"2024-02-23T00:29:04.112075+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.617870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.617870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":101,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165052+0000","last_change":"2024-02-23T00:29:04.010419+0000","last_active":"2024-02-23T00:29:04.165052+0000","last_peered":"2024-02-23T00:29:04.165052+0000","last_clean":"2024-02-23T00:29:04.165052+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:29:04.165052+0000","last_undegraded":"2024-02-23T00:29:04.165052+0000","last_fullsized":"2024-02-23T00:29:04.165052+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:29:04.010348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.010348+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110724+0000","last_change":"2024-02-23T00:28:39.615097+0000","last_active":"2024-02-23T00:29:04.110724+0000","last_peered":"2024-02-23T00:29:04.110724+0000","last_clean":"2024-02-23T00:29:04.110724+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:29:04.110724+0000","last_undegraded":"2024-02-23T00:29:04.110724+0000","last_fullsized":"2024-02-23T00:29:04.110724+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.615002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.615002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166706+0000","last_change":"2024-02-23T00:28:31.646180+0000","last_active":"2024-02-23T00:29:04.166706+0000","last_peered":"2024-02-23T00:29:04.166706+0000","last_clean":"2024-02-23T00:29:04.166706+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:29:04.166706+0000","last_undegraded":"2024-02-23T00:29:04.166706+0000","last_fullsized":"2024-02-23T00:29:04.166706+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:31.646096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:31.646096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167113+0000","last_change":"2024-02-23T00:28:54.994367+0000","last_active":"2024-02-23T00:29:04.167113+0000","last_peered":"2024-02-23T00:29:04.167113+0000","last_clean":"2024-02-23T00:29:04.167113+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:29:04.167113+0000","last_undegraded":"2024-02-23T00:29:04.167113+0000","last_fullsized":"2024-02-23T00:29:04.167113+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:54.994276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.994276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":75,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:09.834901+0000","last_change":"2024-02-23T00:29:09.834851+0000","last_active":"2024-02-23T00:29:09.834901+0000","last_peered":"2024-02-23T00:29:09.834901+0000","last_clean":"2024-02-23T00:29:09.834901+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:29:09.834901+0000","last_undegraded":"2024-02-23T00:29:09.834901+0000","last_fullsized":"2024-02-23T00:29:09.834901+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:29:09.834741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:09.834741+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348815+0000","last_change":"2024-02-23T00:29:02.969742+0000","last_active":"2024-02-23T00:29:20.348815+0000","last_peered":"2024-02-23T00:29:20.348815+0000","last_clean":"2024-02-23T00:29:20.348815+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:29:20.348815+0000","last_undegraded":"2024-02-23T00:29:20.348815+0000","last_fullsized":"2024-02-23T00:29:20.348815+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:29:02.969659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.969659+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:07.960264+0000","last_change":"2024-02-23T00:29:07.960232+0000","last_active":"2024-02-23T00:29:07.960264+0000","last_peered":"2024-02-23T00:29:07.960264+0000","last_clean":"2024-02-23T00:29:07.960264+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:29:07.960264+0000","last_undegraded":"2024-02-23T00:29:07.960264+0000","last_fullsized":"2024-02-23T00:29:07.960264+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:07.960141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:07.960141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.168156+0000","last_change":"2024-02-23T00:28:54.905338+0000","last_active":"2024-02-23T00:29:04.168156+0000","last_peered":"2024-02-23T00:29:04.168156+0000","last_clean":"2024-02-23T00:29:04.168156+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:29:04.168156+0000","last_undegraded":"2024-02-23T00:29:04.168156+0000","last_fullsized":"2024-02-23T00:29:04.168156+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:54.905258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.905258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167187+0000","last_change":"2024-02-23T00:28:34.610448+0000","last_active":"2024-02-23T00:29:04.167187+0000","last_peered":"2024-02-23T00:29:04.167187+0000","last_clean":"2024-02-23T00:29:04.167187+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:29:04.167187+0000","last_undegraded":"2024-02-23T00:29:04.167187+0000","last_fullsized":"2024-02-23T00:29:04.167187+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:28:34.610334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:34.610334+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'17","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110910+0000","last_change":"2024-02-23T00:28:44.586817+0000","last_active":"2024-02-23T00:29:04.110910+0000","last_peered":"2024-02-23T00:29:04.110910+0000","last_clean":"2024-02-23T00:29:04.110910+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:29:04.110910+0000","last_undegraded":"2024-02-23T00:29:04.110910+0000","last_fullsized":"2024-02-23T00:29:04.110910+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:28:44.586731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.586731+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166329+0000","last_change":"2024-02-23T00:28:47.090951+0000","last_active":"2024-02-23T00:29:04.166329+0000","last_peered":"2024-02-23T00:29:04.166329+0000","last_clean":"2024-02-23T00:29:04.166329+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:29:04.166329+0000","last_undegraded":"2024-02-23T00:29:04.166329+0000","last_fullsized":"2024-02-23T00:29:04.166329+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:28:47.090884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.090884+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.349062+0000","last_change":"2024-02-23T00:28:31.557831+0000","last_active":"2024-02-23T00:29:20.349062+0000","last_peered":"2024-02-23T00:29:20.349062+0000","last_clean":"2024-02-23T00:29:20.349062+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:29:20.349062+0000","last_undegraded":"2024-02-23T00:29:20.349062+0000","last_fullsized":"2024-02-23T00:29:20.349062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:31.557737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:31.557737+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.574074+0000","last_change":"2024-02-23T00:29:10.574049+0000","last_active":"2024-02-23T00:29:10.574074+0000","last_peered":"2024-02-23T00:29:10.574074+0000","last_clean":"2024-02-23T00:29:10.574074+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:29:10.574074+0000","last_undegraded":"2024-02-23T00:29:10.574074+0000","last_fullsized":"2024-02-23T00:29:10.574074+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.573983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.573983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'15","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112648+0000","last_change":"2024-02-23T00:28:46.622106+0000","last_active":"2024-02-23T00:29:04.112648+0000","last_peered":"2024-02-23T00:29:04.112648+0000","last_clean":"2024-02-23T00:29:04.112648+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:29:04.112648+0000","last_undegraded":"2024-02-23T00:29:04.112648+0000","last_fullsized":"2024-02-23T00:29:04.112648+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:28:46.621982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.621982+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110668+0000","last_change":"2024-02-23T00:28:40.630272+0000","last_active":"2024-02-23T00:29:04.110668+0000","last_peered":"2024-02-23T00:29:04.110668+0000","last_clean":"2024-02-23T00:29:04.110668+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:29:04.110668+0000","last_undegraded":"2024-02-23T00:29:04.110668+0000","last_fullsized":"2024-02-23T00:29:04.110668+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:40.630153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:40.630153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.348734+0000","last_change":"2024-02-23T00:28:44.617114+0000","last_active":"2024-02-23T00:29:20.348734+0000","last_peered":"2024-02-23T00:29:20.348734+0000","last_clean":"2024-02-23T00:29:20.348734+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:29:20.348734+0000","last_undegraded":"2024-02-23T00:29:20.348734+0000","last_fullsized":"2024-02-23T00:29:20.348734+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:44.617038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.617038+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":50,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111700+0000","last_change":"2024-02-23T00:29:00.703800+0000","last_active":"2024-02-23T00:29:04.111700+0000","last_peered":"2024-02-23T00:29:04.111700+0000","last_clean":"2024-02-23T00:29:04.111700+0000","last_became_active":"2024-02-23T00:28:41.203031+0000","last_became_peered":"2024-02-23T00:28:41.203031+0000","last_unstale":"2024-02-23T00:29:04.111700+0000","last_undegraded":"2024-02-23T00:29:04.111700+0000","last_fullsized":"2024-02-23T00:29:04.111700+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.703737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:08.818370+0000","last_change":"2024-02-23T00:29:08.818324+0000","last_active":"2024-02-23T00:29:08.818370+0000","last_peered":"2024-02-23T00:29:08.818370+0000","last_clean":"2024-02-23T00:29:08.818370+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:29:08.818370+0000","last_undegraded":"2024-02-23T00:29:08.818370+0000","last_fullsized":"2024-02-23T00:29:08.818370+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:29:08.818235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.818235+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112456+0000","last_change":"2024-02-23T00:28:47.677799+0000","last_active":"2024-02-23T00:29:04.112456+0000","last_peered":"2024-02-23T00:29:04.112456+0000","last_clean":"2024-02-23T00:29:04.112456+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:29:04.112456+0000","last_undegraded":"2024-02-23T00:29:04.112456+0000","last_fullsized":"2024-02-23T00:29:04.112456+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:47.677731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.677731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:08.580191+0000","last_change":"2024-02-23T00:29:08.580168+0000","last_active":"2024-02-23T00:29:08.580191+0000","last_peered":"2024-02-23T00:29:08.580191+0000","last_clean":"2024-02-23T00:29:08.580191+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:29:08.580191+0000","last_undegraded":"2024-02-23T00:29:08.580191+0000","last_fullsized":"2024-02-23T00:29:08.580191+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.580102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.580102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:05.279997+0000","last_change":"2024-02-23T00:29:05.279969+0000","last_active":"2024-02-23T00:29:05.279997+0000","last_peered":"2024-02-23T00:29:05.279997+0000","last_clean":"2024-02-23T00:29:05.279997+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:29:05.279997+0000","last_undegraded":"2024-02-23T00:29:05.279997+0000","last_fullsized":"2024-02-23T00:29:05.279997+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:05.279882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:05.279882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110952+0000","last_change":"2024-02-23T00:28:29.526599+0000","last_active":"2024-02-23T00:29:04.110952+0000","last_peered":"2024-02-23T00:29:04.110952+0000","last_clean":"2024-02-23T00:29:04.110952+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:29:04.110952+0000","last_undegraded":"2024-02-23T00:29:04.110952+0000","last_fullsized":"2024-02-23T00:29:04.110952+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:28:29.526505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:29.526505+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166861+0000","last_change":"2024-02-23T00:28:49.331226+0000","last_active":"2024-02-23T00:29:04.166861+0000","last_peered":"2024-02-23T00:29:04.166861+0000","last_clean":"2024-02-23T00:29:04.166861+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:29:04.166861+0000","last_undegraded":"2024-02-23T00:29:04.166861+0000","last_fullsized":"2024-02-23T00:29:04.166861+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.331139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.331139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.113319+0000","last_change":"2024-02-23T00:29:01.776038+0000","last_active":"2024-02-23T00:29:04.113319+0000","last_peered":"2024-02-23T00:29:04.113319+0000","last_clean":"2024-02-23T00:29:04.113319+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:29:04.113319+0000","last_undegraded":"2024-02-23T00:29:04.113319+0000","last_fullsized":"2024-02-23T00:29:04.113319+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:01.775943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.775943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:24.809313+0000","last_change":"2024-02-23T00:29:24.809313+0000","last_active":"2024-02-23T00:29:24.809313+0000","last_peered":"2024-02-23T00:29:24.809313+0000","last_clean":"2024-02-23T00:29:24.809313+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:29:24.809313+0000","last_undegraded":"2024-02-23T00:29:24.809313+0000","last_fullsized":"2024-02-23T00:29:24.809313+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:24.809192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:24.809192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166687+0000","last_change":"2024-02-23T00:28:28.095098+0000","last_active":"2024-02-23T00:29:04.166687+0000","last_peered":"2024-02-23T00:29:04.166687+0000","last_clean":"2024-02-23T00:29:04.166687+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:29:04.166687+0000","last_undegraded":"2024-02-23T00:29:04.166687+0000","last_fullsized":"2024-02-23T00:29:04.166687+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:28:28.095011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.095011+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165173+0000","last_change":"2024-02-23T00:29:00.187868+0000","last_active":"2024-02-23T00:29:04.165173+0000","last_peered":"2024-02-23T00:29:04.165173+0000","last_clean":"2024-02-23T00:29:04.165173+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:29:04.165173+0000","last_undegraded":"2024-02-23T00:29:04.165173+0000","last_fullsized":"2024-02-23T00:29:04.165173+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.187784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.187784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112896+0000","last_change":"2024-02-23T00:29:02.769052+0000","last_active":"2024-02-23T00:29:04.112896+0000","last_peered":"2024-02-23T00:29:04.112896+0000","last_clean":"2024-02-23T00:29:04.112896+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:29:04.112896+0000","last_undegraded":"2024-02-23T00:29:04.112896+0000","last_fullsized":"2024-02-23T00:29:04.112896+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.768986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.768986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111919+0000","last_change":"2024-02-23T00:28:26.541700+0000","last_active":"2024-02-23T00:29:04.111919+0000","last_peered":"2024-02-23T00:29:04.111919+0000","last_clean":"2024-02-23T00:29:04.111919+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:29:04.111919+0000","last_undegraded":"2024-02-23T00:29:04.111919+0000","last_fullsized":"2024-02-23T00:29:04.111919+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:26.541633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:26.541633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":70,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165871+0000","last_change":"2024-02-23T00:28:53.186087+0000","last_active":"2024-02-23T00:29:04.165871+0000","last_peered":"2024-02-23T00:29:04.165871+0000","last_clean":"2024-02-23T00:29:04.165871+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:29:04.165871+0000","last_undegraded":"2024-02-23T00:29:04.165871+0000","last_fullsized":"2024-02-23T00:29:04.165871+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:28:53.186017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.186017+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:08.063504+0000","last_change":"2024-02-23T00:29:08.063475+0000","last_active":"2024-02-23T00:29:08.063504+0000","last_peered":"2024-02-23T00:29:08.063504+0000","last_clean":"2024-02-23T00:29:08.063504+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:29:08.063504+0000","last_undegraded":"2024-02-23T00:29:08.063504+0000","last_fullsized":"2024-02-23T00:29:08.063504+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.063398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.063398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112616+0000","last_change":"2024-02-23T00:28:16.558091+0000","last_active":"2024-02-23T00:29:04.112616+0000","last_peered":"2024-02-23T00:29:04.112616+0000","last_clean":"2024-02-23T00:29:04.112616+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:29:04.112616+0000","last_undegraded":"2024-02-23T00:29:04.112616+0000","last_fullsized":"2024-02-23T00:29:04.112616+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:16.558024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:16.558024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":101,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165621+0000","last_change":"2024-02-23T00:28:46.118981+0000","last_active":"2024-02-23T00:29:04.165621+0000","last_peered":"2024-02-23T00:29:04.165621+0000","last_clean":"2024-02-23T00:29:04.165621+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:29:04.165621+0000","last_undegraded":"2024-02-23T00:29:04.165621+0000","last_fullsized":"2024-02-23T00:29:04.165621+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.118918+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'8","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:20.349079+0000","last_change":"2024-02-23T00:28:28.676907+0000","last_active":"2024-02-23T00:29:20.349079+0000","last_peered":"2024-02-23T00:29:20.349079+0000","last_clean":"2024-02-23T00:29:20.349079+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:29:20.349079+0000","last_undegraded":"2024-02-23T00:29:20.349079+0000","last_fullsized":"2024-02-23T00:29:20.349079+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:28.676837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.676837+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":616,"ondisk_log_size":616,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332125,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":290664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.502,"15min":0.501},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68300000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.441}]},{"osd":1,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.61799999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.49399999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.41599999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.28599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.46999999999999997}]},{"osd":5,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42799999999999999,"15min":0.42699999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.436,"15min":0.42799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.63400000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40500000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.47699999999999998,"15min":0.47099999999999997},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.46000000000000002,"15min":0.45400000000000001},"min":{"1min":0.33900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.52600000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561058,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":289807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.28599999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.51000000000000001,"15min":0.51700000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69399999999999995,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.60899999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.57799999999999996,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.31,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70299999999999996,"5min":0.754,"15min":0.754},"last":0.53900000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.55100000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.504}]},{"osd":3,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48499999999999999,"15min":0.48699999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66800000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.69199999999999995}]},{"osd":4,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.378},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40899999999999997,"15min":0.40600000000000003},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61499999999999999,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.435}]},{"osd":5,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42399999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61699999999999999,"5min":0.996,"15min":0.996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.42199999999999999,"15min":0.41999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.49099999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.48299999999999998,"15min":0.47599999999999998},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.46700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297748,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447404,"statfs":{"total":95995035648,"available":95690141696,"internally_reserved":0,"allocated":667648,"data_stored":291667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.32800000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.44},{"interface":"front","average":{"1min":0.371,"5min":0.372,"15min":0.35099999999999998},"min":{"1min":0.215,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54600000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.37,"15min":0.371},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33400000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.41399999999999998,"15min":0.42599999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.49299999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.438,"15min":0.435},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.38400000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72399999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Feb 23 00:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79100000000000004,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.59399999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.71599999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.52300000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:29:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78500000000000003,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.505},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.60899999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.752,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":290324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34399999999999997,"15min":0.32800000000000001},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.501,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.371},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35599999999999998,"15min":0.33300000000000002},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.49099999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.42099999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.38500000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.46899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.34499999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.43099999999999999,"15min":0.44700000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56200000000000006,"5min":0.629,"15min":0.629},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.313,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53100000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.41199999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.39100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.48999999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.58199999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.59199999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41299999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.623},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":630784,"data_stored":290227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.33700000000000002,"15min":0.32900000000000001},"min":{"1min":0.23799999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.36099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.34300000000000003,"15min":0.32900000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.42699999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.50700000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.35599999999999998,"15min":0.34000000000000002},"min":{"1min":0.245,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47599999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35099999999999998,"15min":0.33900000000000002},"min":{"1min":0.25900000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.44500000000000001,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:28:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40100000000000002,"15min":0.41199999999999998},"min":{"1min":0.26900000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.49299999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.42999999999999999,"15min":0.438},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.54400000000000004,"5min":0.63,"15min":0.63},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Feb 23 00:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57699999999999996}]},{"osd":5,"last update":"Fri Feb 23 00:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.622},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58799999999999997,"15min":0.59299999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.56799999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77600000000000002,"5min":1.117,"15min":1.117},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80600000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.66500000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":289924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.36699999999999999,"15min":0.371},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.49199999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.249},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.38400000000000001,"15min":0.39700000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.50800000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.26500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.40699999999999997,"15min":0.42699999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.437,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.52600000000000002,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.28799999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.43099999999999999,"15min":0.439},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54900000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.47799999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.30299999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.499},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.30299999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68799999999999994,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65500000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:28:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56499999999999995,"15min":0.56899999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72899999999999998,"5min":0.752,"15min":0.752},"last":0.71599999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76600000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.59299999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.61199999999999999},"min":{"1min":0.498,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.72499999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":289907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.35699999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66600000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.33600000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.5,"15min":0.52000000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51000000000000001,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.65200000000000002,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.309}]},{"osd":2,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67200000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.379,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.70299999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.433}]},{"osd":3,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69099999999999995,"5min":0.755,"15min":0.755},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.50800000000000001,"15min":0.53000000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.40600000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.36299999999999999,"15min":0.36099999999999999},"min":{"1min":0.308,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.48799999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.39800000000000002,"15min":0.39400000000000002},"min":{"1min":0.28000000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.55500000000000005,"5min":0.62,"15min":0.62},"last":0.46899999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.34599999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.55500000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.43099999999999999,"15min":0.42899999999999999},"min":{"1min":0.29399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.56399999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.45600000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.46999999999999997,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54800000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757287,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":289441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.505},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70599999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.498},{"interface":"front","average":{"1min":0.497,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.34599999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.623}]},{"osd":1,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47999999999999998,"15min":0.47799999999999998},"min":{"1min":0.34999999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67100000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.49399999999999999,"15min":0.495},"min":{"1min":0.39900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.70399999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58299999999999996}]},{"osd":2,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66100000000000003,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36699999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73299999999999998,"5min":0.753,"15min":0.753},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.38600000000000001,"15min":0.371},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39400000000000002,"15min":0.372},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.39500000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.434,"15min":0.42399999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.442,"15min":0.434},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69599999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:29:29.351 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-23T00:29:29.351 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-23T00:29:29.352 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-23T00:29:29.352 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph health --format=json 2024-02-23T00:29:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:27.481277+0000 mgr.y (mgr.24437) 89 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:27.873849+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.e scrub starts 2024-02-23T00:29:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:27.875102+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.e scrub ok 2024-02-23T00:29:29.624 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:30.292 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:28.519157+0000 mgr.y (mgr.24437) 90 : audit [DBG] from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:30.293 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:28.875600+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:29:30.293 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:28.878485+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:29:30.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:28.519157+0000 mgr.y (mgr.24437) 90 : audit [DBG] from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:30.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:28.875600+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:29:30.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:28.878485+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:29:30.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:28.519157+0000 mgr.y (mgr.24437) 90 : audit [DBG] from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:30.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:28.875600+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:29:30.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:28.878485+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:29:30.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:29:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:29:31.423 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:29.481609+0000 mgr.y (mgr.24437) 91 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:31.423 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:29.481609+0000 mgr.y (mgr.24437) 91 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:29.481609+0000 mgr.y (mgr.24437) 91 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:31.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.678+0000 7fac7f154700 1 -- 172.21.15.87:0/1925138661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 msgr2=0x7fac780f6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:31.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.678+0000 7fac7f154700 1 --2- 172.21.15.87:0/1925138661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 0x7fac780f6450 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fac60004870 tx=0x7fac60009cb0).stop 2024-02-23T00:29:31.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.678+0000 7fac7f154700 1 -- 172.21.15.87:0/1925138661 shutdown_connections 2024-02-23T00:29:31.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.678+0000 7fac7f154700 1 --2- 172.21.15.87:0/1925138661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fac780f6b10 0x7fac781030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:31.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.678+0000 7fac7f154700 1 --2- 172.21.15.87:0/1925138661 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac780fc240 0x7fac780fc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:31.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.678+0000 7fac7f154700 1 --2- 172.21.15.87:0/1925138661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 0x7fac780f6450 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.678+0000 7fac7f154700 1 -- 172.21.15.87:0/1925138661 >> 172.21.15.87:0/1925138661 conn(0x7fac7806c370 msgr2=0x7fac7806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.678+0000 7fac7f154700 1 -- 172.21.15.87:0/1925138661 shutdown_connections 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.678+0000 7fac7f154700 1 -- 172.21.15.87:0/1925138661 wait complete. 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 librados: starting msgr at 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 librados: starting objecter 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 Processor -- start 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 -- start start 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 librados: setting wanted keys 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 librados: calling monclient init 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fac780f6b10 0x7fac781091a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac780fc240 0x7fac78109690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 0x7fac78109c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac78114390 con 0x7fac780fcba0 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac78114500 con 0x7fac780fc240 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7f154700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac781146c0 con 0x7fac780f6b10 2024-02-23T00:29:31.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7cef0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fac780f6b10 0x7fac781091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7d6f1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 0x7fac78109c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7d6f1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 0x7fac78109c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50678/0 (socket says 172.21.15.87:50678) 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7d6f1700 1 -- 172.21.15.87:0/3658902410 learned_addr learned my addr 172.21.15.87:0/3658902410 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac7cef0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fac780f6b10 0x7fac781091a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:43732/0 (socket says 172.21.15.87:43732) 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.679+0000 7fac77fff700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac780fc240 0x7fac78109690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7d6f1700 1 -- 172.21.15.87:0/3658902410 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fac780f6b10 msgr2=0x7fac781091a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7d6f1700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fac780f6b10 0x7fac781091a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7d6f1700 1 -- 172.21.15.87:0/3658902410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac780fc240 msgr2=0x7fac78109690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7d6f1700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac780fc240 0x7fac78109690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7d6f1700 1 -- 172.21.15.87:0/3658902410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac700097c0 con 0x7fac780fcba0 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac77fff700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac780fc240 0x7fac78109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7cef0700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fac780f6b10 0x7fac781091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7d6f1700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 0x7fac78109c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fac70001df0 tx=0x7fac70004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:31.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac75ffb700 10 client.?.objecter ms_handle_connect 0x7fac780fcba0 2024-02-23T00:29:31.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac75ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:31.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac75ffb700 1 -- 172.21.15.87:0/3658902410 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac70009a10 con 0x7fac780fcba0 2024-02-23T00:29:31.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac60005040 con 0x7fac780fcba0 2024-02-23T00:29:31.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac75ffb700 1 -- 172.21.15.87:0/3658902410 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac700095a0 con 0x7fac780fcba0 2024-02-23T00:29:31.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac75ffb700 1 -- 172.21.15.87:0/3658902410 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac70007da0 con 0x7fac780fcba0 2024-02-23T00:29:31.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7f154700 10 client.14715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:31.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac78114cb0 con 0x7fac780fcba0 2024-02-23T00:29:31.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.680+0000 7fac7f154700 1 librados: init done 2024-02-23T00:29:31.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.681+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7805fa40 con 0x7fac780fcba0 2024-02-23T00:29:31.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.681+0000 7fac75ffb700 1 -- 172.21.15.87:0/3658902410 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fac700090d0 con 0x7fac780fcba0 2024-02-23T00:29:31.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.682+0000 7fac75ffb700 1 --2- 172.21.15.87:0/3658902410 >> v2:172.21.15.87:6800/1231597463 conn(0x7fac64062c90 0x7fac64065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:31.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.682+0000 7fac75ffb700 1 -- 172.21.15.87:0/3658902410 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(82..82 src has 28..82) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fac7007d420 con 0x7fac780fcba0 2024-02-23T00:29:31.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.682+0000 7fac75ffb700 10 client.14715.objecter ms_dispatch 0x7fac78001550 osd_map(82..82 src has 28..82) v4 2024-02-23T00:29:31.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.682+0000 7fac75ffb700 3 client.14715.objecter handle_osd_map got epochs [82,82] > 0 2024-02-23T00:29:31.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.682+0000 7fac75ffb700 3 client.14715.objecter handle_osd_map decoding full epoch 82 2024-02-23T00:29:31.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.682+0000 7fac7cef0700 1 --2- 172.21.15.87:0/3658902410 >> v2:172.21.15.87:6800/1231597463 conn(0x7fac64062c90 0x7fac64065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:31.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.682+0000 7fac75ffb700 20 client.14715.objecter dump_active .. 0 homeless 2024-02-23T00:29:31.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.682+0000 7fac75ffb700 10 client.14715.objecter ms_handle_connect 0x7fac64062c90 2024-02-23T00:29:31.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.682+0000 7fac7cef0700 1 --2- 172.21.15.87:0/3658902410 >> v2:172.21.15.87:6800/1231597463 conn(0x7fac64062c90 0x7fac64065150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fac68010420 tx=0x7fac6800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:31.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.686+0000 7fac75ffb700 1 -- 172.21.15.87:0/3658902410 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fac70051090 con 0x7fac780fcba0 2024-02-23T00:29:31.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.892+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fac7805ee80 con 0x7fac780fcba0 2024-02-23T00:29:31.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.893+0000 7fac75ffb700 1 -- 172.21.15.87:0/3658902410 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7fac70050c80 con 0x7fac780fcba0 2024-02-23T00:29:31.895 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:29:31.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 10 librados: watch_flush enter 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 10 librados: watch_flush exit 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 20 client.14715.objecter shutdown clearing up homeless session... 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 10 client.14715.objecter successfully canceled tick 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 >> v2:172.21.15.87:6800/1231597463 conn(0x7fac64062c90 msgr2=0x7fac64065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 --2- 172.21.15.87:0/3658902410 >> v2:172.21.15.87:6800/1231597463 conn(0x7fac64062c90 0x7fac64065150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fac68010420 tx=0x7fac6800b040).stop 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 msgr2=0x7fac78109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 0x7fac78109c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fac70001df0 tx=0x7fac70004c50).stop 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 shutdown_connections 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 --2- 172.21.15.87:0/3658902410 >> v2:172.21.15.87:6800/1231597463 conn(0x7fac64062c90 0x7fac64065150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fac780fcba0 0x7fac78109c50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:31.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fac780f6b10 0x7fac781091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:31.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 --2- 172.21.15.87:0/3658902410 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fac780fc240 0x7fac78109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:31.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 >> 172.21.15.87:0/3658902410 conn(0x7fac7806c370 msgr2=0x7fac780fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:31.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 shutdown_connections 2024-02-23T00:29:31.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.894+0000 7fac7f154700 1 -- 172.21.15.87:0/3658902410 wait complete. 2024-02-23T00:29:31.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:31.895+0000 7fac7f154700 1 librados: shutdown 2024-02-23T00:29:32.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:32] "GET /metrics HTTP/1.1" 200 172176 "" "Prometheus/2.33.4" 2024-02-23T00:29:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:30.561492+0000 mgr.y (mgr.24437) 92 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:31.894263+0000 mon.a (mon.0) 732 : audit [DBG] from='client.? 172.21.15.87:0/3658902410' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-23T00:29:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:30.561492+0000 mgr.y (mgr.24437) 92 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:31.894263+0000 mon.a (mon.0) 732 : audit [DBG] from='client.? 172.21.15.87:0/3658902410' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-23T00:29:32.678 INFO:teuthology.orchestra.run.smithi087.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-23T00:29:32.678 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-23T00:29:32.679 INFO:tasks.cephadm:Setup complete, yielding 2024-02-23T00:29:32.679 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-23T00:29:32.715 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-23T00:29:32.716 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'pacific-release'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', 'flag\\(s\\) set', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(PG_', '\\(OSD_', '\\(POOL_', '\\(CEPHADM_STRAY_DAEMON\\)', 'PG_', 'degraded', 'backfill', 'mons down', 'OSD_', 'is down', 'acting', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'eb66ed921e744301e6863be6353618d63967ea59'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-ksirivad-pacific-release-whitelist', 'sha1': '4f758cd8e8854d1f126da86e254076852bf059a9'}} 2024-02-23T00:29:32.716 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-23T00:29:32.716 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-23T00:29:32.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:30.561492+0000 mgr.y (mgr.24437) 92 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:32.717 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:31.894263+0000 mon.a (mon.0) 732 : audit [DBG] from='client.? 172.21.15.87:0/3658902410' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-23T00:29:32.987 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:33.504 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:31.482183+0000 mgr.y (mgr.24437) 93 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:33.504 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:31.821524+0000 osd.2 (osd.2) 37 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:29:33.504 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:31.823400+0000 osd.2 (osd.2) 38 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:29:33.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:32.119509+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:29:33.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:32.122423+0000 osd.3 (osd.3) 52 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:29:33.505 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:31.482183+0000 mgr.y (mgr.24437) 93 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:33.505 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:31.821524+0000 osd.2 (osd.2) 37 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:29:33.505 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:31.823400+0000 osd.2 (osd.2) 38 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:29:33.505 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:32.119509+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:29:33.505 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:32.122423+0000 osd.3 (osd.3) 52 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:29:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:31.482183+0000 mgr.y (mgr.24437) 93 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:31.821524+0000 osd.2 (osd.2) 37 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:29:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:31.823400+0000 osd.2 (osd.2) 38 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:29:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:32.119509+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:29:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:32.122423+0000 osd.3 (osd.3) 52 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:29:34.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:29:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:29:35.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 -- 172.21.15.87:0/926126927 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240f7400 msgr2=0x7f9e24103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:35.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/926126927 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240f7400 0x7f9e24103a20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f9e1c009c30 tx=0x7f9e1c004c50).stop 2024-02-23T00:29:35.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 -- 172.21.15.87:0/926126927 shutdown_connections 2024-02-23T00:29:35.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/926126927 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240f7400 0x7f9e24103a20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:35.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/926126927 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e240fcbb0 0x7f9e240fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:35.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/926126927 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e240fd510 0x7f9e240f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 -- 172.21.15.87:0/926126927 >> 172.21.15.87:0/926126927 conn(0x7f9e240f2820 msgr2=0x7f9e240f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 -- 172.21.15.87:0/926126927 shutdown_connections 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 -- 172.21.15.87:0/926126927 wait complete. 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 librados: starting msgr at 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.015+0000 7f9e28ba8700 1 librados: starting objecter 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 Processor -- start 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 -- start start 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 librados: setting wanted keys 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 librados: calling monclient init 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e240f7400 0x7f9e24106de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e240fcbb0 0x7f9e241072d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240fd510 0x7f9e241077c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:35.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24116460 con 0x7f9e240f7400 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e241165a0 con 0x7f9e240fcbb0 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e28ba8700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24116710 con 0x7f9e240fd510 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e22d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240fd510 0x7f9e241077c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e22d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240fd510 0x7f9e241077c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:43738/0 (socket says 172.21.15.87:43738) 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e22d9d700 1 -- 172.21.15.87:0/529272757 learned_addr learned my addr 172.21.15.87:0/529272757 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e2259c700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e240f7400 0x7f9e24106de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e21d9b700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e240fcbb0 0x7f9e241072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e22d9d700 1 -- 172.21.15.87:0/529272757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e240fcbb0 msgr2=0x7f9e241072d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e22d9d700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e240fcbb0 0x7f9e241072d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e22d9d700 1 -- 172.21.15.87:0/529272757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e240f7400 msgr2=0x7f9e24106de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e22d9d700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e240f7400 0x7f9e24106de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e22d9d700 1 -- 172.21.15.87:0/529272757 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c005040 con 0x7f9e240fd510 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e2259c700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e240f7400 0x7f9e24106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:35.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e22d9d700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240fd510 0x7f9e241077c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f9e1c004e10 tx=0x7f9e1c004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e1b7fe700 10 client.?.objecter ms_handle_connect 0x7f9e240fd510 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.016+0000 7f9e1b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.017+0000 7f9e1b7fe700 1 -- 172.21.15.87:0/529272757 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c00e040 con 0x7f9e240fd510 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.017+0000 7f9e1b7fe700 1 -- 172.21.15.87:0/529272757 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e1c018040 con 0x7f9e240fd510 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.017+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c0097c0 con 0x7f9e240fd510 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.017+0000 7f9e1b7fe700 1 -- 172.21.15.87:0/529272757 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c003ae0 con 0x7f9e240fd510 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.017+0000 7f9e28ba8700 10 client.24647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.017+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e24116b50 con 0x7f9e240fd510 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.017+0000 7f9e28ba8700 1 librados: init done 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.018+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2405fa60 con 0x7f9e240fd510 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.018+0000 7f9e1b7fe700 1 -- 172.21.15.87:0/529272757 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f9e1c004020 con 0x7f9e240fd510 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.018+0000 7f9e1b7fe700 1 --2- 172.21.15.87:0/529272757 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e10062c90 0x7f9e10065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.018+0000 7f9e1b7fe700 1 -- 172.21.15.87:0/529272757 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(82..82 src has 28..82) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f9e1c07db50 con 0x7f9e240fd510 2024-02-23T00:29:35.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.018+0000 7f9e1b7fe700 10 client.24647.objecter ms_dispatch 0x7f9e24001550 osd_map(82..82 src has 28..82) v4 2024-02-23T00:29:35.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.018+0000 7f9e1b7fe700 3 client.24647.objecter handle_osd_map got epochs [82,82] > 0 2024-02-23T00:29:35.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.018+0000 7f9e1b7fe700 3 client.24647.objecter handle_osd_map decoding full epoch 82 2024-02-23T00:29:35.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.018+0000 7f9e2259c700 1 --2- 172.21.15.87:0/529272757 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e10062c90 0x7f9e10065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:35.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.018+0000 7f9e1b7fe700 20 client.24647.objecter dump_active .. 0 homeless 2024-02-23T00:29:35.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.019+0000 7f9e1b7fe700 10 client.24647.objecter ms_handle_connect 0x7f9e10062c90 2024-02-23T00:29:35.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.019+0000 7f9e2259c700 1 --2- 172.21.15.87:0/529272757 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e10062c90 0x7f9e10065150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9e0c004c80 tx=0x7f9e0c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:35.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.023+0000 7f9e1b7fe700 1 -- 172.21.15.87:0/529272757 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9e1c0517c0 con 0x7f9e240fd510 2024-02-23T00:29:35.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.188+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) v1 -- 0x7f9e24061440 con 0x7f9e240fd510 2024-02-23T00:29:35.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.193+0000 7f9e1b7fe700 1 -- 172.21.15.87:0/529272757 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v27)=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7f9e1c0513b0 con 0x7f9e240fd510 2024-02-23T00:29:35.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.195+0000 7f9e28ba8700 10 librados: watch_flush enter 2024-02-23T00:29:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.195+0000 7f9e28ba8700 10 librados: watch_flush exit 2024-02-23T00:29:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.195+0000 7f9e28ba8700 20 client.24647.objecter shutdown clearing up homeless session... 2024-02-23T00:29:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.195+0000 7f9e28ba8700 10 client.24647.objecter successfully canceled tick 2024-02-23T00:29:35.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.197+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e10062c90 msgr2=0x7f9e10065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:35.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.197+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/529272757 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e10062c90 0x7f9e10065150 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f9e0c004c80 tx=0x7f9e0c00b040).stop 2024-02-23T00:29:35.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.197+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240fd510 msgr2=0x7f9e241077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:35.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.197+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240fd510 0x7f9e241077c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f9e1c004e10 tx=0x7f9e1c004c50).stop 2024-02-23T00:29:35.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.198+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 shutdown_connections 2024-02-23T00:29:35.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.198+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/529272757 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9e10062c90 0x7f9e10065150 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0x7f9e0c004c80 tx=0x7f9e0c00b040).stop 2024-02-23T00:29:35.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.198+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9e240fd510 0x7f9e241077c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:35.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.198+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9e240f7400 0x7f9e24106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:35.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.198+0000 7f9e28ba8700 1 --2- 172.21.15.87:0/529272757 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9e240fcbb0 0x7f9e241072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:35.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.198+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 >> 172.21.15.87:0/529272757 conn(0x7f9e240f2820 msgr2=0x7f9e240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:35.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.199+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 shutdown_connections 2024-02-23T00:29:35.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.199+0000 7f9e28ba8700 1 -- 172.21.15.87:0/529272757 wait complete. 2024-02-23T00:29:35.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:35.199+0000 7f9e28ba8700 1 librados: shutdown 2024-02-23T00:29:35.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:33.482600+0000 mgr.y (mgr.24437) 94 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:35.194484+0000 mon.a (mon.0) 733 : audit [INF] from='client.? ' entity='client.admin' 2024-02-23T00:29:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:33.482600+0000 mgr.y (mgr.24437) 94 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:35.194484+0000 mon.a (mon.0) 733 : audit [INF] from='client.? ' entity='client.admin' 2024-02-23T00:29:35.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:29:35] ENGINE Bus STOPPING 2024-02-23T00:29:35.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:29:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-23T00:29:35.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:29:35] ENGINE Bus STOPPED 2024-02-23T00:29:35.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:29:35] ENGINE Bus STARTING 2024-02-23T00:29:35.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:29:35] ENGINE Serving on http://:::9283 2024-02-23T00:29:35.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:29:35] ENGINE Bus STARTED 2024-02-23T00:29:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:33.482600+0000 mgr.y (mgr.24437) 94 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:35.194484+0000 mon.a (mon.0) 733 : audit [INF] from='client.? ' entity='client.admin' 2024-02-23T00:29:35.942 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-23T00:29:35.942 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-23T00:29:35.942 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:29:36.210 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:36.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:35.224955+0000 mon.a (mon.0) 734 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:29:36.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:35.226319+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:29:36.479 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:35.226842+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:29:36.479 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:35.231480+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:36.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:35.224955+0000 mon.a (mon.0) 734 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:29:36.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:35.226319+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:29:36.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:35.226842+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:29:36.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:35.231480+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:35.224955+0000 mon.a (mon.0) 734 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:29:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:35.226319+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:29:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:35.226842+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:29:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:35.231480+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:29:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:35.483110+0000 mgr.y (mgr.24437) 95 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:35.483110+0000 mgr.y (mgr.24437) 95 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:35.483110+0000 mgr.y (mgr.24437) 95 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:38.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.124+0000 7f73f06ab700 1 -- 172.21.15.87:0/3694186258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80f8420 msgr2=0x7f73e8104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:38.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.124+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3694186258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80f8420 0x7f73e8104a20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f73e0009be0 tx=0x7f73e0004d10).stop 2024-02-23T00:29:38.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.124+0000 7f73f06ab700 1 -- 172.21.15.87:0/3694186258 shutdown_connections 2024-02-23T00:29:38.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.124+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3694186258 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80f8420 0x7f73e8104a20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.124+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3694186258 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73e80fdbb0 0x7f73e80fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:29:38.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.124+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3694186258 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73e80fe510 0x7f73e80f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.124+0000 7f73f06ab700 1 -- 172.21.15.87:0/3694186258 >> 172.21.15.87:0/3694186258 conn(0x7f73e80f3800 msgr2=0x7f73e80f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:38.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.124+0000 7f73f06ab700 1 -- 172.21.15.87:0/3694186258 shutdown_connections 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.124+0000 7f73f06ab700 1 -- 172.21.15.87:0/3694186258 wait complete. 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 librados: starting msgr at 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 librados: starting objecter 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 Processor -- start 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 -- start start 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 librados: setting wanted keys 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 librados: calling monclient init 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73e80f8420 0x7f73e810dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80fdbb0 0x7f73e810e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73e80fe510 0x7f73e8108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e8108640 con 0x7f73e80fdbb0 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e81087b0 con 0x7f73e80fe510 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.125+0000 7f73f06ab700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e8108970 con 0x7f73e80f8420 2024-02-23T00:29:38.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73edc46700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80fdbb0 0x7f73e810e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73edc46700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80fdbb0 0x7f73e810e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:50716/0 (socket says 172.21.15.87:50716) 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73edc46700 1 -- 172.21.15.87:0/3510854421 learned_addr learned my addr 172.21.15.87:0/3510854421 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73ee447700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73e80f8420 0x7f73e810dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73eec48700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73e80fe510 0x7f73e8108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73edc46700 1 -- 172.21.15.87:0/3510854421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73e80f8420 msgr2=0x7f73e810dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73edc46700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73e80f8420 0x7f73e810dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73edc46700 1 -- 172.21.15.87:0/3510854421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73e80fe510 msgr2=0x7f73e8108150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73edc46700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73e80fe510 0x7f73e8108150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73edc46700 1 -- 172.21.15.87:0/3510854421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e4005040 con 0x7f73e80fdbb0 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73ee447700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73e80f8420 0x7f73e810dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73eec48700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73e80fe510 0x7f73e8108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73edc46700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80fdbb0 0x7f73e810e4a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f73e40048f0 tx=0x7f73e4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73df7fe700 10 client.?.objecter ms_handle_connect 0x7f73e80fdbb0 2024-02-23T00:29:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73df7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73df7fe700 1 -- 172.21.15.87:0/3510854421 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4007b00 con 0x7f73e80fdbb0 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73df7fe700 1 -- 172.21.15.87:0/3510854421 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73e400f040 con 0x7f73e80fdbb0 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e00097c0 con 0x7f73e80fdbb0 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73df7fe700 1 -- 172.21.15.87:0/3510854421 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4014a00 con 0x7f73e80fdbb0 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73f06ab700 10 client.14727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e8108fb0 con 0x7f73e80fdbb0 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.126+0000 7f73f06ab700 1 librados: init done 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.127+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e805ee80 con 0x7f73e80fdbb0 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.127+0000 7f73df7fe700 1 -- 172.21.15.87:0/3510854421 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f73e400e580 con 0x7f73e80fdbb0 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.128+0000 7f73df7fe700 1 --2- 172.21.15.87:0/3510854421 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73d4062c90 0x7f73d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.128+0000 7f73df7fe700 1 -- 172.21.15.87:0/3510854421 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(82..82 src has 28..82) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f73e407c650 con 0x7f73e80fdbb0 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.128+0000 7f73ee447700 1 --2- 172.21.15.87:0/3510854421 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73d4062c90 0x7f73d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.128+0000 7f73df7fe700 10 client.14727.objecter ms_dispatch 0x7f73e8001550 osd_map(82..82 src has 28..82) v4 2024-02-23T00:29:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.128+0000 7f73df7fe700 3 client.14727.objecter handle_osd_map got epochs [82,82] > 0 2024-02-23T00:29:38.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.128+0000 7f73df7fe700 3 client.14727.objecter handle_osd_map decoding full epoch 82 2024-02-23T00:29:38.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.128+0000 7f73df7fe700 20 client.14727.objecter dump_active .. 0 homeless 2024-02-23T00:29:38.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.128+0000 7f73df7fe700 10 client.14727.objecter ms_handle_connect 0x7f73d4062c90 2024-02-23T00:29:38.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.128+0000 7f73ee447700 1 --2- 172.21.15.87:0/3510854421 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73d4062c90 0x7f73d4065150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f73d8006fd0 tx=0x7f73d8009060).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:38.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.132+0000 7f73df7fe700 1 -- 172.21.15.87:0/3510854421 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f73e4049f60 con 0x7f73e80fdbb0 2024-02-23T00:29:38.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.294+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73e8109700 con 0x7f73d4062c90 2024-02-23T00:29:38.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.303+0000 7f73df7fe700 1 -- 172.21.15.87:0/3510854421 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357379 (secure 0 0 0) 0x7f73e8109700 con 0x7f73d4062c90 2024-02-23T00:29:38.311 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:29:38.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 10 librados: watch_flush enter 2024-02-23T00:29:38.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 10 librados: watch_flush exit 2024-02-23T00:29:38.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 20 client.14727.objecter shutdown clearing up homeless session... 2024-02-23T00:29:38.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 10 client.14727.objecter successfully canceled tick 2024-02-23T00:29:38.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73d4062c90 msgr2=0x7f73d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:38.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3510854421 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73d4062c90 0x7f73d4065150 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f73d8006fd0 tx=0x7f73d8009060).stop 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80fdbb0 msgr2=0x7f73e810e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80fdbb0 0x7f73e810e4a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f73e40048f0 tx=0x7f73e4009cb0).stop 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 shutdown_connections 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3510854421 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73d4062c90 0x7f73d4065150 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73e80fe510 0x7f73e8108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73e80f8420 0x7f73e810dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 --2- 172.21.15.87:0/3510854421 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73e80fdbb0 0x7f73e810e4a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 >> 172.21.15.87:0/3510854421 conn(0x7f73e80f3800 msgr2=0x7f73e80f5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 shutdown_connections 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.306+0000 7f73f06ab700 1 -- 172.21.15.87:0/3510854421 wait complete. 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:38.307+0000 7f73f06ab700 1 librados: shutdown 2024-02-23T00:29:38.315 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:29:38.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:37.517438+0000 osd.1 (osd.1) 47 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:29:38.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:37.520437+0000 osd.1 (osd.1) 48 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:29:38.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:37.517438+0000 osd.1 (osd.1) 47 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:29:38.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:37.520437+0000 osd.1 (osd.1) 48 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:29:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:37.517438+0000 osd.1 (osd.1) 47 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:29:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:37.520437+0000 osd.1 (osd.1) 48 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:29:39.084 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-02-23T00:29:37.483393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1236,"num_read_kb":956,"num_write":620,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":694,"ondisk_log_size":694,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381744,"kb_used_data":4976,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579472,"statfs":{"total":767960285184,"available":765521379328,"internally_reserved":0,"allocated":5095424,"data_stored":2321961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002862"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165705+0000","last_change":"2024-02-23T00:28:52.093263+0000","last_active":"2024-02-23T00:29:04.165705+0000","last_peered":"2024-02-23T00:29:04.165705+0000","last_clean":"2024-02-23T00:29:04.165705+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:29:04.165705+0000","last_undegraded":"2024-02-23T00:29:04.165705+0000","last_fullsized":"2024-02-23T00:29:04.165705+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:52.093177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.093177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.109974+0000","last_change":"2024-02-23T00:28:57.678604+0000","last_active":"2024-02-23T00:29:04.109974+0000","last_peered":"2024-02-23T00:29:04.109974+0000","last_clean":"2024-02-23T00:29:04.109974+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:29:04.109974+0000","last_undegraded":"2024-02-23T00:29:04.109974+0000","last_fullsized":"2024-02-23T00:29:04.109974+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:57.678522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.678522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166389+0000","last_change":"2024-02-23T00:28:46.217573+0000","last_active":"2024-02-23T00:29:04.166389+0000","last_peered":"2024-02-23T00:29:04.166389+0000","last_clean":"2024-02-23T00:29:04.166389+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:29:04.166389+0000","last_undegraded":"2024-02-23T00:29:04.166389+0000","last_fullsized":"2024-02-23T00:29:04.166389+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:46.217507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.217507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'39","reported_seq":124,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165323+0000","last_change":"2024-02-23T00:28:37.040298+0000","last_active":"2024-02-23T00:29:04.165323+0000","last_peered":"2024-02-23T00:29:04.165323+0000","last_clean":"2024-02-23T00:29:04.165323+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:29:04.165323+0000","last_undegraded":"2024-02-23T00:29:04.165323+0000","last_fullsized":"2024-02-23T00:29:04.165323+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'39","last_scrub_stamp":"2024-02-23T00:28:37.040215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:37.040215+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110191+0000","last_change":"2024-02-23T00:28:59.685150+0000","last_active":"2024-02-23T00:29:04.110191+0000","last_peered":"2024-02-23T00:29:04.110191+0000","last_clean":"2024-02-23T00:29:04.110191+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:29:04.110191+0000","last_undegraded":"2024-02-23T00:29:04.110191+0000","last_fullsized":"2024-02-23T00:29:04.110191+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:59.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:59.685045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112331+0000","last_change":"2024-02-23T00:28:49.663875+0000","last_active":"2024-02-23T00:29:04.112331+0000","last_peered":"2024-02-23T00:29:04.112331+0000","last_clean":"2024-02-23T00:29:04.112331+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:29:04.112331+0000","last_undegraded":"2024-02-23T00:29:04.112331+0000","last_fullsized":"2024-02-23T00:29:04.112331+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.663808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.663808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166335+0000","last_change":"2024-02-23T00:28:47.073631+0000","last_active":"2024-02-23T00:29:04.166335+0000","last_peered":"2024-02-23T00:29:04.166335+0000","last_clean":"2024-02-23T00:29:04.166335+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:29:04.166335+0000","last_undegraded":"2024-02-23T00:29:04.166335+0000","last_fullsized":"2024-02-23T00:29:04.166335+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:47.073551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.073551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":75,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:06.822930+0000","last_change":"2024-02-23T00:29:06.822899+0000","last_active":"2024-02-23T00:29:06.822930+0000","last_peered":"2024-02-23T00:29:06.822930+0000","last_clean":"2024-02-23T00:29:06.822930+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:29:06.822930+0000","last_undegraded":"2024-02-23T00:29:06.822930+0000","last_fullsized":"2024-02-23T00:29:06.822930+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:29:06.822818+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167297+0000","last_change":"2024-02-23T00:29:04.049502+0000","last_active":"2024-02-23T00:29:04.167297+0000","last_peered":"2024-02-23T00:29:04.167297+0000","last_clean":"2024-02-23T00:29:04.167297+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:29:04.167297+0000","last_undegraded":"2024-02-23T00:29:04.167297+0000","last_fullsized":"2024-02-23T00:29:04.167297+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.049414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166268+0000","last_change":"2024-02-23T00:29:00.320083+0000","last_active":"2024-02-23T00:29:04.166268+0000","last_peered":"2024-02-23T00:29:04.166268+0000","last_clean":"2024-02-23T00:29:04.166268+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:29:04.166268+0000","last_undegraded":"2024-02-23T00:29:04.166268+0000","last_fullsized":"2024-02-23T00:29:04.166268+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.319985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.319985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167512+0000","last_change":"2024-02-23T00:28:46.396553+0000","last_active":"2024-02-23T00:29:04.167512+0000","last_peered":"2024-02-23T00:29:04.167512+0000","last_clean":"2024-02-23T00:29:04.167512+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:29:04.167512+0000","last_undegraded":"2024-02-23T00:29:04.167512+0000","last_fullsized":"2024-02-23T00:29:04.167512+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.396471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110342+0000","last_change":"2024-02-23T00:28:54.657266+0000","last_active":"2024-02-23T00:29:04.110342+0000","last_peered":"2024-02-23T00:29:04.110342+0000","last_clean":"2024-02-23T00:29:04.110342+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:29:04.110342+0000","last_undegraded":"2024-02-23T00:29:04.110342+0000","last_fullsized":"2024-02-23T00:29:04.110342+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.657177+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165496+0000","last_change":"2024-02-23T00:28:50.176282+0000","last_active":"2024-02-23T00:29:04.165496+0000","last_peered":"2024-02-23T00:29:04.165496+0000","last_clean":"2024-02-23T00:29:04.165496+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:29:04.165496+0000","last_undegraded":"2024-02-23T00:29:04.165496+0000","last_fullsized":"2024-02-23T00:29:04.165496+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.176208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167187+0000","last_change":"2024-02-23T00:28:33.933658+0000","last_active":"2024-02-23T00:29:04.167187+0000","last_peered":"2024-02-23T00:29:04.167187+0000","last_clean":"2024-02-23T00:29:04.167187+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:29:04.167187+0000","last_undegraded":"2024-02-23T00:29:04.167187+0000","last_fullsized":"2024-02-23T00:29:04.167187+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.933529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.933529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.267535+0000","last_change":"2024-02-23T00:29:04.267507+0000","last_active":"2024-02-23T00:29:04.267535+0000","last_peered":"2024-02-23T00:29:04.267535+0000","last_clean":"2024-02-23T00:29:04.267535+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:29:04.267535+0000","last_undegraded":"2024-02-23T00:29:04.267535+0000","last_fullsized":"2024-02-23T00:29:04.267535+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.267427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.267427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110473+0000","last_change":"2024-02-23T00:28:25.457421+0000","last_active":"2024-02-23T00:29:04.110473+0000","last_peered":"2024-02-23T00:29:04.110473+0000","last_clean":"2024-02-23T00:29:04.110473+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:29:04.110473+0000","last_undegraded":"2024-02-23T00:29:04.110473+0000","last_fullsized":"2024-02-23T00:29:04.110473+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:28:25.457329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:25.457329+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166564+0000","last_change":"2024-02-23T00:28:29.086512+0000","last_active":"2024-02-23T00:29:04.166564+0000","last_peered":"2024-02-23T00:29:04.166564+0000","last_clean":"2024-02-23T00:29:04.166564+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:29:04.166564+0000","last_undegraded":"2024-02-23T00:29:04.166564+0000","last_fullsized":"2024-02-23T00:29:04.166564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:29.086437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:29.086437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110635+0000","last_change":"2024-02-23T00:29:03.790608+0000","last_active":"2024-02-23T00:29:04.110635+0000","last_peered":"2024-02-23T00:29:04.110635+0000","last_clean":"2024-02-23T00:29:04.110635+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:29:04.110635+0000","last_undegraded":"2024-02-23T00:29:04.110635+0000","last_fullsized":"2024-02-23T00:29:04.110635+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.790530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.790530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165837+0000","last_change":"2024-02-23T00:28:30.086083+0000","last_active":"2024-02-23T00:29:04.165837+0000","last_peered":"2024-02-23T00:29:04.165837+0000","last_clean":"2024-02-23T00:29:04.165837+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:29:04.165837+0000","last_undegraded":"2024-02-23T00:29:04.165837+0000","last_fullsized":"2024-02-23T00:29:04.165837+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:30.086009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:30.086009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":83,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166086+0000","last_change":"2024-02-23T00:28:33.077947+0000","last_active":"2024-02-23T00:29:04.166086+0000","last_peered":"2024-02-23T00:29:04.166086+0000","last_clean":"2024-02-23T00:29:04.166086+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:29:04.166086+0000","last_undegraded":"2024-02-23T00:29:04.166086+0000","last_fullsized":"2024-02-23T00:29:04.166086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.077848+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.830507+0000","last_change":"2024-02-23T00:29:10.830479+0000","last_active":"2024-02-23T00:29:10.830507+0000","last_peered":"2024-02-23T00:29:10.830507+0000","last_clean":"2024-02-23T00:29:10.830507+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:29:10.830507+0000","last_undegraded":"2024-02-23T00:29:10.830507+0000","last_fullsized":"2024-02-23T00:29:10.830507+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.830389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.830389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:05.641550+0000","last_change":"2024-02-23T00:29:05.641525+0000","last_active":"2024-02-23T00:29:05.641550+0000","last_peered":"2024-02-23T00:29:05.641550+0000","last_clean":"2024-02-23T00:29:05.641550+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:29:05.641550+0000","last_undegraded":"2024-02-23T00:29:05.641550+0000","last_fullsized":"2024-02-23T00:29:05.641550+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_clean_scrub_stamp":"2024-02-23T00:29:05.641454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:28.878569+0000","last_change":"2024-02-23T00:29:28.878569+0000","last_active":"2024-02-23T00:29:28.878569+0000","last_peered":"2024-02-23T00:29:28.878569+0000","last_clean":"2024-02-23T00:29:28.878569+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:29:28.878569+0000","last_undegraded":"2024-02-23T00:29:28.878569+0000","last_fullsized":"2024-02-23T00:29:28.878569+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:28.878491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:28.878491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":69,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:31.823473+0000","last_change":"2024-02-23T00:29:31.823473+0000","last_active":"2024-02-23T00:29:31.823473+0000","last_peered":"2024-02-23T00:29:31.823473+0000","last_clean":"2024-02-23T00:29:31.823473+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:29:31.823473+0000","last_undegraded":"2024-02-23T00:29:31.823473+0000","last_fullsized":"2024-02-23T00:29:31.823473+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:29:31.823405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:31.823405+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111713+0000","last_change":"2024-02-23T00:28:27.524062+0000","last_active":"2024-02-23T00:29:04.111713+0000","last_peered":"2024-02-23T00:29:04.111713+0000","last_clean":"2024-02-23T00:29:04.111713+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:29:04.111713+0000","last_undegraded":"2024-02-23T00:29:04.111713+0000","last_fullsized":"2024-02-23T00:29:04.111713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:27.523959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:27.523959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.130821+0000","last_change":"2024-02-23T00:29:10.130797+0000","last_active":"2024-02-23T00:29:10.130821+0000","last_peered":"2024-02-23T00:29:10.130821+0000","last_clean":"2024-02-23T00:29:10.130821+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:29:10.130821+0000","last_undegraded":"2024-02-23T00:29:10.130821+0000","last_fullsized":"2024-02-23T00:29:10.130821+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.130727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.130727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165445+0000","last_change":"2024-02-23T00:28:32.090929+0000","last_active":"2024-02-23T00:29:04.165445+0000","last_peered":"2024-02-23T00:29:04.165445+0000","last_clean":"2024-02-23T00:29:04.165445+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:29:04.165445+0000","last_undegraded":"2024-02-23T00:29:04.165445+0000","last_fullsized":"2024-02-23T00:29:04.165445+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:32.090869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:32.090869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167021+0000","last_change":"2024-02-23T00:28:44.987515+0000","last_active":"2024-02-23T00:29:04.167021+0000","last_peered":"2024-02-23T00:29:04.167021+0000","last_clean":"2024-02-23T00:29:04.167021+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:29:04.167021+0000","last_undegraded":"2024-02-23T00:29:04.167021+0000","last_fullsized":"2024-02-23T00:29:04.167021+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:28:44.987419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.987419+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167041+0000","last_change":"2024-02-23T00:28:48.333707+0000","last_active":"2024-02-23T00:29:04.167041+0000","last_peered":"2024-02-23T00:29:04.167041+0000","last_clean":"2024-02-23T00:29:04.167041+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:29:04.167041+0000","last_undegraded":"2024-02-23T00:29:04.167041+0000","last_fullsized":"2024-02-23T00:29:04.167041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:28:48.333638+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"62'8","reported_seq":75,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:30.351316+0000","last_change":"2024-02-23T00:28:36.071048+0000","last_active":"2024-02-23T00:29:30.351316+0000","last_peered":"2024-02-23T00:29:30.351316+0000","last_clean":"2024-02-23T00:29:30.351316+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:29:30.351316+0000","last_undegraded":"2024-02-23T00:29:30.351316+0000","last_fullsized":"2024-02-23T00:29:30.351316+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:36.070959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.070959+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165452+0000","last_change":"2024-02-23T00:29:03.017128+0000","last_active":"2024-02-23T00:29:04.165452+0000","last_peered":"2024-02-23T00:29:04.165452+0000","last_clean":"2024-02-23T00:29:04.165452+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:29:04.165452+0000","last_undegraded":"2024-02-23T00:29:04.165452+0000","last_fullsized":"2024-02-23T00:29:04.165452+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.017051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.017051+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'16","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166313+0000","last_change":"2024-02-23T00:28:55.230148+0000","last_active":"2024-02-23T00:29:04.166313+0000","last_peered":"2024-02-23T00:29:04.166313+0000","last_clean":"2024-02-23T00:29:04.166313+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:29:04.166313+0000","last_undegraded":"2024-02-23T00:29:04.166313+0000","last_fullsized":"2024-02-23T00:29:04.166313+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:28:55.230081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.230081+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165952+0000","last_change":"2024-02-23T00:28:41.113778+0000","last_active":"2024-02-23T00:29:04.165952+0000","last_peered":"2024-02-23T00:29:04.165952+0000","last_clean":"2024-02-23T00:29:04.165952+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:29:04.165952+0000","last_undegraded":"2024-02-23T00:29:04.165952+0000","last_fullsized":"2024-02-23T00:29:04.165952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.113684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.113684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167420+0000","last_change":"2024-02-23T00:28:41.520505+0000","last_active":"2024-02-23T00:29:04.167420+0000","last_peered":"2024-02-23T00:29:04.167420+0000","last_clean":"2024-02-23T00:29:04.167420+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:29:04.167420+0000","last_undegraded":"2024-02-23T00:29:04.167420+0000","last_fullsized":"2024-02-23T00:29:04.167420+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.520412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.520412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167154+0000","last_change":"2024-02-23T00:28:37.106379+0000","last_active":"2024-02-23T00:29:04.167154+0000","last_peered":"2024-02-23T00:29:04.167154+0000","last_clean":"2024-02-23T00:29:04.167154+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:29:04.167154+0000","last_undegraded":"2024-02-23T00:29:04.167154+0000","last_fullsized":"2024-02-23T00:29:04.167154+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:37.106309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:37.106309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":80,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165684+0000","last_change":"2024-02-23T00:28:57.039721+0000","last_active":"2024-02-23T00:29:04.165684+0000","last_peered":"2024-02-23T00:29:04.165684+0000","last_clean":"2024-02-23T00:29:04.165684+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:29:04.165684+0000","last_undegraded":"2024-02-23T00:29:04.165684+0000","last_fullsized":"2024-02-23T00:29:04.165684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.039660+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166057+0000","last_change":"2024-02-23T00:28:43.440882+0000","last_active":"2024-02-23T00:29:04.166057+0000","last_peered":"2024-02-23T00:29:04.166057+0000","last_clean":"2024-02-23T00:29:04.166057+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:29:04.166057+0000","last_undegraded":"2024-02-23T00:29:04.166057+0000","last_fullsized":"2024-02-23T00:29:04.166057+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.440788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.440788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166062+0000","last_change":"2024-02-23T00:28:56.022911+0000","last_active":"2024-02-23T00:29:04.166062+0000","last_peered":"2024-02-23T00:29:04.166062+0000","last_clean":"2024-02-23T00:29:04.166062+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:29:04.166062+0000","last_undegraded":"2024-02-23T00:29:04.166062+0000","last_fullsized":"2024-02-23T00:29:04.166062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:56.022835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:56.022835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167959+0000","last_change":"2024-02-23T00:28:24.002963+0000","last_active":"2024-02-23T00:29:04.167959+0000","last_peered":"2024-02-23T00:29:04.167959+0000","last_clean":"2024-02-23T00:29:04.167959+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:29:04.167959+0000","last_undegraded":"2024-02-23T00:29:04.167959+0000","last_fullsized":"2024-02-23T00:29:04.167959+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:24.002878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:24.002878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165126+0000","last_change":"2024-02-23T00:28:20.075559+0000","last_active":"2024-02-23T00:29:04.165126+0000","last_peered":"2024-02-23T00:29:04.165126+0000","last_clean":"2024-02-23T00:29:04.165126+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:29:04.165126+0000","last_undegraded":"2024-02-23T00:29:04.165126+0000","last_fullsized":"2024-02-23T00:29:04.165126+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:28:20.075487+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165247+0000","last_change":"2024-02-23T00:28:39.076715+0000","last_active":"2024-02-23T00:29:04.165247+0000","last_peered":"2024-02-23T00:29:04.165247+0000","last_clean":"2024-02-23T00:29:04.165247+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:29:04.165247+0000","last_undegraded":"2024-02-23T00:29:04.165247+0000","last_fullsized":"2024-02-23T00:29:04.165247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.076631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.076631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111882+0000","last_change":"2024-02-23T00:28:53.665149+0000","last_active":"2024-02-23T00:29:04.111882+0000","last_peered":"2024-02-23T00:29:04.111882+0000","last_clean":"2024-02-23T00:29:04.111882+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:29:04.111882+0000","last_undegraded":"2024-02-23T00:29:04.111882+0000","last_fullsized":"2024-02-23T00:29:04.111882+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.665092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111817+0000","last_change":"2024-02-23T00:29:02.711400+0000","last_active":"2024-02-23T00:29:04.111817+0000","last_peered":"2024-02-23T00:29:04.111817+0000","last_clean":"2024-02-23T00:29:04.111817+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:29:04.111817+0000","last_undegraded":"2024-02-23T00:29:04.111817+0000","last_fullsized":"2024-02-23T00:29:04.111817+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.711284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":79,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110548+0000","last_change":"2024-02-23T00:28:41.214195+0000","last_active":"2024-02-23T00:29:04.110548+0000","last_peered":"2024-02-23T00:29:04.110548+0000","last_clean":"2024-02-23T00:29:04.110548+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:29:04.110548+0000","last_undegraded":"2024-02-23T00:29:04.110548+0000","last_fullsized":"2024-02-23T00:29:04.110548+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:28:36.536962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.536962+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165027+0000","last_change":"2024-02-23T00:28:49.187859+0000","last_active":"2024-02-23T00:29:04.165027+0000","last_peered":"2024-02-23T00:29:04.165027+0000","last_clean":"2024-02-23T00:29:04.165027+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:29:04.165027+0000","last_undegraded":"2024-02-23T00:29:04.165027+0000","last_fullsized":"2024-02-23T00:29:04.165027+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.187786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.187786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:32.122532+0000","last_change":"2024-02-23T00:29:32.122532+0000","last_active":"2024-02-23T00:29:32.122532+0000","last_peered":"2024-02-23T00:29:32.122532+0000","last_clean":"2024-02-23T00:29:32.122532+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:29:32.122532+0000","last_undegraded":"2024-02-23T00:29:32.122532+0000","last_fullsized":"2024-02-23T00:29:32.122532+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:32.122430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:32.122430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166211+0000","last_change":"2024-02-23T00:28:21.021722+0000","last_active":"2024-02-23T00:29:04.166211+0000","last_peered":"2024-02-23T00:29:04.166211+0000","last_clean":"2024-02-23T00:29:04.166211+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:29:04.166211+0000","last_undegraded":"2024-02-23T00:29:04.166211+0000","last_fullsized":"2024-02-23T00:29:04.166211+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:21.021619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.021619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111782+0000","last_change":"2024-02-23T00:28:36.676648+0000","last_active":"2024-02-23T00:29:04.111782+0000","last_peered":"2024-02-23T00:29:04.111782+0000","last_clean":"2024-02-23T00:29:04.111782+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:29:04.111782+0000","last_undegraded":"2024-02-23T00:29:04.111782+0000","last_fullsized":"2024-02-23T00:29:04.111782+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:28:36.676582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.676582+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111356+0000","last_change":"2024-02-23T00:28:43.560702+0000","last_active":"2024-02-23T00:29:04.111356+0000","last_peered":"2024-02-23T00:29:04.111356+0000","last_clean":"2024-02-23T00:29:04.111356+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:29:04.111356+0000","last_undegraded":"2024-02-23T00:29:04.111356+0000","last_fullsized":"2024-02-23T00:29:04.111356+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.560606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.560606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"62'8","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:30.350846+0000","last_change":"2024-02-23T00:28:21.123596+0000","last_active":"2024-02-23T00:29:30.350846+0000","last_peered":"2024-02-23T00:29:30.350846+0000","last_clean":"2024-02-23T00:29:30.350846+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:29:30.350846+0000","last_undegraded":"2024-02-23T00:29:30.350846+0000","last_fullsized":"2024-02-23T00:29:30.350846+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:21.123498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.123498+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165830+0000","last_change":"2024-02-23T00:28:30.236265+0000","last_active":"2024-02-23T00:29:04.165830+0000","last_peered":"2024-02-23T00:29:04.165830+0000","last_clean":"2024-02-23T00:29:04.165830+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:29:04.165830+0000","last_undegraded":"2024-02-23T00:29:04.165830+0000","last_fullsized":"2024-02-23T00:29:04.165830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:30.236188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:30.236188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'15","reported_seq":68,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166655+0000","last_change":"2024-02-23T00:28:55.907616+0000","last_active":"2024-02-23T00:29:04.166655+0000","last_peered":"2024-02-23T00:29:04.166655+0000","last_clean":"2024-02-23T00:29:04.166655+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:29:04.166655+0000","last_undegraded":"2024-02-23T00:29:04.166655+0000","last_fullsized":"2024-02-23T00:29:04.166655+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:28:55.907528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.907528+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:14.086398+0000","last_change":"2024-02-23T00:29:14.086374+0000","last_active":"2024-02-23T00:29:14.086398+0000","last_peered":"2024-02-23T00:29:14.086398+0000","last_clean":"2024-02-23T00:29:14.086398+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:29:14.086398+0000","last_undegraded":"2024-02-23T00:29:14.086398+0000","last_fullsized":"2024-02-23T00:29:14.086398+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:14.086293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:14.086293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":72,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:30.351315+0000","last_change":"2024-02-23T00:28:38.956507+0000","last_active":"2024-02-23T00:29:30.351315+0000","last_peered":"2024-02-23T00:29:30.351315+0000","last_clean":"2024-02-23T00:29:30.351315+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:29:30.351315+0000","last_undegraded":"2024-02-23T00:29:30.351315+0000","last_fullsized":"2024-02-23T00:29:30.351315+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_clean_scrub_stamp":"2024-02-23T00:28:38.956431+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165830+0000","last_change":"2024-02-23T00:28:49.012817+0000","last_active":"2024-02-23T00:29:04.165830+0000","last_peered":"2024-02-23T00:29:04.165830+0000","last_clean":"2024-02-23T00:29:04.165830+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:29:04.165830+0000","last_undegraded":"2024-02-23T00:29:04.165830+0000","last_fullsized":"2024-02-23T00:29:04.165830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.012698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":84,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166786+0000","last_change":"2024-02-23T00:29:01.026660+0000","last_active":"2024-02-23T00:29:04.166786+0000","last_peered":"2024-02-23T00:29:04.166786+0000","last_clean":"2024-02-23T00:29:04.166786+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:29:04.166786+0000","last_undegraded":"2024-02-23T00:29:04.166786+0000","last_fullsized":"2024-02-23T00:29:04.166786+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:29:01.026575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.026575+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167407+0000","last_change":"2024-02-23T00:28:58.383733+0000","last_active":"2024-02-23T00:29:04.167407+0000","last_peered":"2024-02-23T00:29:04.167407+0000","last_clean":"2024-02-23T00:29:04.167407+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:29:04.167407+0000","last_undegraded":"2024-02-23T00:29:04.167407+0000","last_fullsized":"2024-02-23T00:29:04.167407+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:28:58.383642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.044071+0000","last_change":"2024-02-23T00:29:10.044046+0000","last_active":"2024-02-23T00:29:10.044071+0000","last_peered":"2024-02-23T00:29:10.044071+0000","last_clean":"2024-02-23T00:29:10.044071+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:29:10.044071+0000","last_undegraded":"2024-02-23T00:29:10.044071+0000","last_fullsized":"2024-02-23T00:29:10.044071+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.043966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.043966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166141+0000","last_change":"2024-02-23T00:28:34.066522+0000","last_active":"2024-02-23T00:29:04.166141+0000","last_peered":"2024-02-23T00:29:04.166141+0000","last_clean":"2024-02-23T00:29:04.166141+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:29:04.166141+0000","last_undegraded":"2024-02-23T00:29:04.166141+0000","last_fullsized":"2024-02-23T00:29:04.166141+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:34.066441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:34.066441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":82,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166525+0000","last_change":"2024-02-23T00:28:42.144931+0000","last_active":"2024-02-23T00:29:04.166525+0000","last_peered":"2024-02-23T00:29:04.166525+0000","last_clean":"2024-02-23T00:29:04.166525+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:29:04.166525+0000","last_undegraded":"2024-02-23T00:29:04.166525+0000","last_fullsized":"2024-02-23T00:29:04.166525+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:28:42.144833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:42.144833+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112842+0000","last_change":"2024-02-23T00:28:53.629641+0000","last_active":"2024-02-23T00:29:04.112842+0000","last_peered":"2024-02-23T00:29:04.112842+0000","last_clean":"2024-02-23T00:29:04.112842+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:29:04.112842+0000","last_undegraded":"2024-02-23T00:29:04.112842+0000","last_fullsized":"2024-02-23T00:29:04.112842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:53.629568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.629568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166601+0000","last_change":"2024-02-23T00:29:01.986532+0000","last_active":"2024-02-23T00:29:04.166601+0000","last_peered":"2024-02-23T00:29:04.166601+0000","last_clean":"2024-02-23T00:29:04.166601+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:29:04.166601+0000","last_undegraded":"2024-02-23T00:29:04.166601+0000","last_fullsized":"2024-02-23T00:29:04.166601+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:01.986461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.986461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112289+0000","last_change":"2024-02-23T00:28:41.611517+0000","last_active":"2024-02-23T00:29:04.112289+0000","last_peered":"2024-02-23T00:29:04.112289+0000","last_clean":"2024-02-23T00:29:04.112289+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:29:04.112289+0000","last_undegraded":"2024-02-23T00:29:04.112289+0000","last_fullsized":"2024-02-23T00:29:04.112289+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.611448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.611448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166225+0000","last_change":"2024-02-23T00:29:01.234433+0000","last_active":"2024-02-23T00:29:04.166225+0000","last_peered":"2024-02-23T00:29:04.166225+0000","last_clean":"2024-02-23T00:29:04.166225+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:29:04.166225+0000","last_undegraded":"2024-02-23T00:29:04.166225+0000","last_fullsized":"2024-02-23T00:29:04.166225+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:29:01.234356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.234356+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165913+0000","last_change":"2024-02-23T00:28:35.058858+0000","last_active":"2024-02-23T00:29:04.165913+0000","last_peered":"2024-02-23T00:29:04.165913+0000","last_clean":"2024-02-23T00:29:04.165913+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:29:04.165913+0000","last_undegraded":"2024-02-23T00:29:04.165913+0000","last_fullsized":"2024-02-23T00:29:04.165913+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:35.058773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:35.058773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111158+0000","last_change":"2024-02-23T00:28:51.684865+0000","last_active":"2024-02-23T00:29:04.111158+0000","last_peered":"2024-02-23T00:29:04.111158+0000","last_clean":"2024-02-23T00:29:04.111158+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:29:04.111158+0000","last_undegraded":"2024-02-23T00:29:04.111158+0000","last_fullsized":"2024-02-23T00:29:04.111158+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:28:51.684789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.684789+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":118,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110345+0000","last_change":"2024-02-23T00:29:01.733374+0000","last_active":"2024-02-23T00:29:04.110345+0000","last_peered":"2024-02-23T00:29:04.110345+0000","last_clean":"2024-02-23T00:29:04.110345+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:29:04.110345+0000","last_undegraded":"2024-02-23T00:29:04.110345+0000","last_fullsized":"2024-02-23T00:29:04.110345+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:29:01.733287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.733287+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112261+0000","last_change":"2024-02-23T00:28:51.634965+0000","last_active":"2024-02-23T00:29:04.112261+0000","last_peered":"2024-02-23T00:29:04.112261+0000","last_clean":"2024-02-23T00:29:04.112261+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:29:04.112261+0000","last_undegraded":"2024-02-23T00:29:04.112261+0000","last_fullsized":"2024-02-23T00:29:04.112261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:51.634901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.634901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.227949+0000","last_change":"2024-02-23T00:29:04.227928+0000","last_active":"2024-02-23T00:29:04.227949+0000","last_peered":"2024-02-23T00:29:04.227949+0000","last_clean":"2024-02-23T00:29:04.227949+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:29:04.227949+0000","last_undegraded":"2024-02-23T00:29:04.227949+0000","last_fullsized":"2024-02-23T00:29:04.227949+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.227864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.227864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111585+0000","last_change":"2024-02-23T00:28:51.664433+0000","last_active":"2024-02-23T00:29:04.111585+0000","last_peered":"2024-02-23T00:29:04.111585+0000","last_clean":"2024-02-23T00:29:04.111585+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:29:04.111585+0000","last_undegraded":"2024-02-23T00:29:04.111585+0000","last_fullsized":"2024-02-23T00:29:04.111585+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.664367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111833+0000","last_change":"2024-02-23T00:29:03.691092+0000","last_active":"2024-02-23T00:29:04.111833+0000","last_peered":"2024-02-23T00:29:04.111833+0000","last_clean":"2024-02-23T00:29:04.111833+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:29:04.111833+0000","last_undegraded":"2024-02-23T00:29:04.111833+0000","last_fullsized":"2024-02-23T00:29:04.111833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:29:03.691018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.691018+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":76,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:30.350841+0000","last_change":"2024-02-23T00:28:42.249575+0000","last_active":"2024-02-23T00:29:30.350841+0000","last_peered":"2024-02-23T00:29:30.350841+0000","last_clean":"2024-02-23T00:29:30.350841+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:29:30.350841+0000","last_undegraded":"2024-02-23T00:29:30.350841+0000","last_fullsized":"2024-02-23T00:29:30.350841+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:42.249510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:42.249510+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166389+0000","last_change":"2024-02-23T00:28:45.134966+0000","last_active":"2024-02-23T00:29:04.166389+0000","last_peered":"2024-02-23T00:29:04.166389+0000","last_clean":"2024-02-23T00:29:04.166389+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:29:04.166389+0000","last_undegraded":"2024-02-23T00:29:04.166389+0000","last_fullsized":"2024-02-23T00:29:04.166389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:45.134840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:45.134840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":45,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165915+0000","last_change":"2024-02-23T00:28:50.000698+0000","last_active":"2024-02-23T00:29:04.165915+0000","last_peered":"2024-02-23T00:29:04.165915+0000","last_clean":"2024-02-23T00:29:04.165915+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:29:04.165915+0000","last_undegraded":"2024-02-23T00:29:04.165915+0000","last_fullsized":"2024-02-23T00:29:04.165915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:28:50.000624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.000624+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":130,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:30.350847+0000","last_change":"2024-02-23T00:28:39.629165+0000","last_active":"2024-02-23T00:29:30.350847+0000","last_peered":"2024-02-23T00:29:30.350847+0000","last_clean":"2024-02-23T00:29:30.350847+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:29:30.350847+0000","last_undegraded":"2024-02-23T00:29:30.350847+0000","last_fullsized":"2024-02-23T00:29:30.350847+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:28:39.629075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.629075+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165991+0000","last_change":"2024-02-23T00:28:28.220174+0000","last_active":"2024-02-23T00:29:04.165991+0000","last_peered":"2024-02-23T00:29:04.165991+0000","last_clean":"2024-02-23T00:29:04.165991+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:29:04.165991+0000","last_undegraded":"2024-02-23T00:29:04.165991+0000","last_fullsized":"2024-02-23T00:29:04.165991+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:28.220080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.220080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167568+0000","last_change":"2024-02-23T00:28:39.496236+0000","last_active":"2024-02-23T00:29:04.167568+0000","last_peered":"2024-02-23T00:29:04.167568+0000","last_clean":"2024-02-23T00:29:04.167568+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:29:04.167568+0000","last_undegraded":"2024-02-23T00:29:04.167568+0000","last_fullsized":"2024-02-23T00:29:04.167568+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.496160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.496160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":177,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:34.277591+0000","last_change":"2024-02-23T00:28:52.653896+0000","last_active":"2024-02-23T00:29:34.277591+0000","last_peered":"2024-02-23T00:29:34.277591+0000","last_clean":"2024-02-23T00:29:34.277591+0000","last_became_active":"2024-02-23T00:27:17.009670+0000","last_became_peered":"2024-02-23T00:27:17.009670+0000","last_unstale":"2024-02-23T00:29:34.277591+0000","last_undegraded":"2024-02-23T00:29:34.277591+0000","last_fullsized":"2024-02-23T00:29:34.277591+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.653823+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.113286+0000","last_change":"2024-02-23T00:28:33.503311+0000","last_active":"2024-02-23T00:29:04.113286+0000","last_peered":"2024-02-23T00:29:04.113286+0000","last_clean":"2024-02-23T00:29:04.113286+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:29:04.113286+0000","last_undegraded":"2024-02-23T00:29:04.113286+0000","last_fullsized":"2024-02-23T00:29:04.113286+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.503241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.503241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":81,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112582+0000","last_change":"2024-02-23T00:28:32.603154+0000","last_active":"2024-02-23T00:29:04.112582+0000","last_peered":"2024-02-23T00:29:04.112582+0000","last_clean":"2024-02-23T00:29:04.112582+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:29:04.112582+0000","last_undegraded":"2024-02-23T00:29:04.112582+0000","last_fullsized":"2024-02-23T00:29:04.112582+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:28:32.603088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:32.603088+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112491+0000","last_change":"2024-02-23T00:28:50.672048+0000","last_active":"2024-02-23T00:29:04.112491+0000","last_peered":"2024-02-23T00:29:04.112491+0000","last_clean":"2024-02-23T00:29:04.112491+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:29:04.112491+0000","last_undegraded":"2024-02-23T00:29:04.112491+0000","last_fullsized":"2024-02-23T00:29:04.112491+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.671988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.671988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.168280+0000","last_change":"2024-02-23T00:28:40.999596+0000","last_active":"2024-02-23T00:29:04.168280+0000","last_peered":"2024-02-23T00:29:04.168280+0000","last_clean":"2024-02-23T00:29:04.168280+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:29:04.168280+0000","last_undegraded":"2024-02-23T00:29:04.168280+0000","last_fullsized":"2024-02-23T00:29:04.168280+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:40.999514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:40.999514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165328+0000","last_change":"2024-02-23T00:28:17.047708+0000","last_active":"2024-02-23T00:29:04.165328+0000","last_peered":"2024-02-23T00:29:04.165328+0000","last_clean":"2024-02-23T00:29:04.165328+0000","last_became_active":"2024-02-23T00:27:17.012683+0000","last_became_peered":"2024-02-23T00:27:17.012683+0000","last_unstale":"2024-02-23T00:29:04.165328+0000","last_undegraded":"2024-02-23T00:29:04.165328+0000","last_fullsized":"2024-02-23T00:29:04.165328+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:17.047614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:17.047614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167144+0000","last_change":"2024-02-23T00:28:52.948687+0000","last_active":"2024-02-23T00:29:04.167144+0000","last_peered":"2024-02-23T00:29:04.167144+0000","last_clean":"2024-02-23T00:29:04.167144+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:29:04.167144+0000","last_undegraded":"2024-02-23T00:29:04.167144+0000","last_fullsized":"2024-02-23T00:29:04.167144+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:52.948601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.948601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:25.267851+0000","last_change":"2024-02-23T00:29:25.267851+0000","last_active":"2024-02-23T00:29:25.267851+0000","last_peered":"2024-02-23T00:29:25.267851+0000","last_clean":"2024-02-23T00:29:25.267851+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:29:25.267851+0000","last_undegraded":"2024-02-23T00:29:25.267851+0000","last_fullsized":"2024-02-23T00:29:25.267851+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:29:25.267773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166538+0000","last_change":"2024-02-23T00:28:33.090534+0000","last_active":"2024-02-23T00:29:04.166538+0000","last_peered":"2024-02-23T00:29:04.166538+0000","last_clean":"2024-02-23T00:29:04.166538+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:29:04.166538+0000","last_undegraded":"2024-02-23T00:29:04.166538+0000","last_fullsized":"2024-02-23T00:29:04.166538+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:28:33.090454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.090454+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166849+0000","last_change":"2024-02-23T00:28:50.344419+0000","last_active":"2024-02-23T00:29:04.166849+0000","last_peered":"2024-02-23T00:29:04.166849+0000","last_clean":"2024-02-23T00:29:04.166849+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:29:04.166849+0000","last_undegraded":"2024-02-23T00:29:04.166849+0000","last_fullsized":"2024-02-23T00:29:04.166849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.344324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.344324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111422+0000","last_change":"2024-02-23T00:28:55.679584+0000","last_active":"2024-02-23T00:29:04.111422+0000","last_peered":"2024-02-23T00:29:04.111422+0000","last_clean":"2024-02-23T00:29:04.111422+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:29:04.111422+0000","last_undegraded":"2024-02-23T00:29:04.111422+0000","last_fullsized":"2024-02-23T00:29:04.111422+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:55.679522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.679522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165482+0000","last_change":"2024-02-23T00:28:59.226967+0000","last_active":"2024-02-23T00:29:04.165482+0000","last_peered":"2024-02-23T00:29:04.165482+0000","last_clean":"2024-02-23T00:29:04.165482+0000","last_became_active":"2024-02-23T00:27:17.009826+0000","last_became_peered":"2024-02-23T00:27:17.009826+0000","last_unstale":"2024-02-23T00:29:04.165482+0000","last_undegraded":"2024-02-23T00:29:04.165482+0000","last_fullsized":"2024-02-23T00:29:04.165482+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:59.226890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:59.226890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166978+0000","last_change":"2024-02-23T00:29:02.340533+0000","last_active":"2024-02-23T00:29:04.166978+0000","last_peered":"2024-02-23T00:29:04.166978+0000","last_clean":"2024-02-23T00:29:04.166978+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:29:04.166978+0000","last_undegraded":"2024-02-23T00:29:04.166978+0000","last_fullsized":"2024-02-23T00:29:04.166978+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.340450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.340450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165741+0000","last_change":"2024-02-23T00:28:36.209181+0000","last_active":"2024-02-23T00:29:04.165741+0000","last_peered":"2024-02-23T00:29:04.165741+0000","last_clean":"2024-02-23T00:29:04.165741+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:29:04.165741+0000","last_undegraded":"2024-02-23T00:29:04.165741+0000","last_fullsized":"2024-02-23T00:29:04.165741+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:28:36.209111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.209111+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.168250+0000","last_change":"2024-02-23T00:28:24.953423+0000","last_active":"2024-02-23T00:29:04.168250+0000","last_peered":"2024-02-23T00:29:04.168250+0000","last_clean":"2024-02-23T00:29:04.168250+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:29:04.168250+0000","last_undegraded":"2024-02-23T00:29:04.168250+0000","last_fullsized":"2024-02-23T00:29:04.168250+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:28:24.953349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:11.580783+0000","last_change":"2024-02-23T00:29:11.580760+0000","last_active":"2024-02-23T00:29:11.580783+0000","last_peered":"2024-02-23T00:29:11.580783+0000","last_clean":"2024-02-23T00:29:11.580783+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:29:11.580783+0000","last_undegraded":"2024-02-23T00:29:11.580783+0000","last_fullsized":"2024-02-23T00:29:11.580783+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:11.580697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:11.580697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167628+0000","last_change":"2024-02-23T00:28:20.772584+0000","last_active":"2024-02-23T00:29:04.167628+0000","last_peered":"2024-02-23T00:29:04.167628+0000","last_clean":"2024-02-23T00:29:04.167628+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:29:04.167628+0000","last_undegraded":"2024-02-23T00:29:04.167628+0000","last_fullsized":"2024-02-23T00:29:04.167628+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:20.772479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:20.772479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":90,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165786+0000","last_change":"2024-02-23T00:28:21.157813+0000","last_active":"2024-02-23T00:29:04.165786+0000","last_peered":"2024-02-23T00:29:04.165786+0000","last_clean":"2024-02-23T00:29:04.165786+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:29:04.165786+0000","last_undegraded":"2024-02-23T00:29:04.165786+0000","last_fullsized":"2024-02-23T00:29:04.165786+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:28:21.157735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.157735+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165865+0000","last_change":"2024-02-23T00:28:43.992150+0000","last_active":"2024-02-23T00:29:04.165865+0000","last_peered":"2024-02-23T00:29:04.165865+0000","last_clean":"2024-02-23T00:29:04.165865+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:29:04.165865+0000","last_undegraded":"2024-02-23T00:29:04.165865+0000","last_fullsized":"2024-02-23T00:29:04.165865+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.992070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.992070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110748+0000","last_change":"2024-02-23T00:28:33.615049+0000","last_active":"2024-02-23T00:29:04.110748+0000","last_peered":"2024-02-23T00:29:04.110748+0000","last_clean":"2024-02-23T00:29:04.110748+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:29:04.110748+0000","last_undegraded":"2024-02-23T00:29:04.110748+0000","last_fullsized":"2024-02-23T00:29:04.110748+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.614962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.614962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112075+0000","last_change":"2024-02-23T00:28:41.617945+0000","last_active":"2024-02-23T00:29:04.112075+0000","last_peered":"2024-02-23T00:29:04.112075+0000","last_clean":"2024-02-23T00:29:04.112075+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:29:04.112075+0000","last_undegraded":"2024-02-23T00:29:04.112075+0000","last_fullsized":"2024-02-23T00:29:04.112075+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.617870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.617870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":101,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165052+0000","last_change":"2024-02-23T00:29:04.010419+0000","last_active":"2024-02-23T00:29:04.165052+0000","last_peered":"2024-02-23T00:29:04.165052+0000","last_clean":"2024-02-23T00:29:04.165052+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:29:04.165052+0000","last_undegraded":"2024-02-23T00:29:04.165052+0000","last_fullsized":"2024-02-23T00:29:04.165052+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:29:04.010348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.010348+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110724+0000","last_change":"2024-02-23T00:28:39.615097+0000","last_active":"2024-02-23T00:29:04.110724+0000","last_peered":"2024-02-23T00:29:04.110724+0000","last_clean":"2024-02-23T00:29:04.110724+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:29:04.110724+0000","last_undegraded":"2024-02-23T00:29:04.110724+0000","last_fullsized":"2024-02-23T00:29:04.110724+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.615002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.615002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166706+0000","last_change":"2024-02-23T00:28:31.646180+0000","last_active":"2024-02-23T00:29:04.166706+0000","last_peered":"2024-02-23T00:29:04.166706+0000","last_clean":"2024-02-23T00:29:04.166706+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:29:04.166706+0000","last_undegraded":"2024-02-23T00:29:04.166706+0000","last_fullsized":"2024-02-23T00:29:04.166706+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:31.646096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:31.646096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167113+0000","last_change":"2024-02-23T00:28:54.994367+0000","last_active":"2024-02-23T00:29:04.167113+0000","last_peered":"2024-02-23T00:29:04.167113+0000","last_clean":"2024-02-23T00:29:04.167113+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:29:04.167113+0000","last_undegraded":"2024-02-23T00:29:04.167113+0000","last_fullsized":"2024-02-23T00:29:04.167113+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:54.994276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.994276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":75,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:09.834901+0000","last_change":"2024-02-23T00:29:09.834851+0000","last_active":"2024-02-23T00:29:09.834901+0000","last_peered":"2024-02-23T00:29:09.834901+0000","last_clean":"2024-02-23T00:29:09.834901+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:29:09.834901+0000","last_undegraded":"2024-02-23T00:29:09.834901+0000","last_fullsized":"2024-02-23T00:29:09.834901+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:29:09.834741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:09.834741+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:30.350879+0000","last_change":"2024-02-23T00:29:02.969742+0000","last_active":"2024-02-23T00:29:30.350879+0000","last_peered":"2024-02-23T00:29:30.350879+0000","last_clean":"2024-02-23T00:29:30.350879+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:29:30.350879+0000","last_undegraded":"2024-02-23T00:29:30.350879+0000","last_fullsized":"2024-02-23T00:29:30.350879+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:29:02.969659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.969659+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:07.960264+0000","last_change":"2024-02-23T00:29:07.960232+0000","last_active":"2024-02-23T00:29:07.960264+0000","last_peered":"2024-02-23T00:29:07.960264+0000","last_clean":"2024-02-23T00:29:07.960264+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:29:07.960264+0000","last_undegraded":"2024-02-23T00:29:07.960264+0000","last_fullsized":"2024-02-23T00:29:07.960264+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:07.960141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:07.960141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.168156+0000","last_change":"2024-02-23T00:28:54.905338+0000","last_active":"2024-02-23T00:29:04.168156+0000","last_peered":"2024-02-23T00:29:04.168156+0000","last_clean":"2024-02-23T00:29:04.168156+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:29:04.168156+0000","last_undegraded":"2024-02-23T00:29:04.168156+0000","last_fullsized":"2024-02-23T00:29:04.168156+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:54.905258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.905258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.167187+0000","last_change":"2024-02-23T00:28:34.610448+0000","last_active":"2024-02-23T00:29:04.167187+0000","last_peered":"2024-02-23T00:29:04.167187+0000","last_clean":"2024-02-23T00:29:04.167187+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:29:04.167187+0000","last_undegraded":"2024-02-23T00:29:04.167187+0000","last_fullsized":"2024-02-23T00:29:04.167187+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:28:34.610334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:34.610334+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'17","reported_seq":73,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110910+0000","last_change":"2024-02-23T00:28:44.586817+0000","last_active":"2024-02-23T00:29:04.110910+0000","last_peered":"2024-02-23T00:29:04.110910+0000","last_clean":"2024-02-23T00:29:04.110910+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:29:04.110910+0000","last_undegraded":"2024-02-23T00:29:04.110910+0000","last_fullsized":"2024-02-23T00:29:04.110910+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:28:44.586731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.586731+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166329+0000","last_change":"2024-02-23T00:28:47.090951+0000","last_active":"2024-02-23T00:29:04.166329+0000","last_peered":"2024-02-23T00:29:04.166329+0000","last_clean":"2024-02-23T00:29:04.166329+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:29:04.166329+0000","last_undegraded":"2024-02-23T00:29:04.166329+0000","last_fullsized":"2024-02-23T00:29:04.166329+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:28:47.090884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.090884+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:30.351320+0000","last_change":"2024-02-23T00:28:31.557831+0000","last_active":"2024-02-23T00:29:30.351320+0000","last_peered":"2024-02-23T00:29:30.351320+0000","last_clean":"2024-02-23T00:29:30.351320+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:29:30.351320+0000","last_undegraded":"2024-02-23T00:29:30.351320+0000","last_fullsized":"2024-02-23T00:29:30.351320+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:31.557737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:31.557737+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:10.574074+0000","last_change":"2024-02-23T00:29:10.574049+0000","last_active":"2024-02-23T00:29:10.574074+0000","last_peered":"2024-02-23T00:29:10.574074+0000","last_clean":"2024-02-23T00:29:10.574074+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:29:10.574074+0000","last_undegraded":"2024-02-23T00:29:10.574074+0000","last_fullsized":"2024-02-23T00:29:10.574074+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.573983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.573983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'15","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112648+0000","last_change":"2024-02-23T00:28:46.622106+0000","last_active":"2024-02-23T00:29:04.112648+0000","last_peered":"2024-02-23T00:29:04.112648+0000","last_clean":"2024-02-23T00:29:04.112648+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:29:04.112648+0000","last_undegraded":"2024-02-23T00:29:04.112648+0000","last_fullsized":"2024-02-23T00:29:04.112648+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:28:46.621982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.621982+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110668+0000","last_change":"2024-02-23T00:28:40.630272+0000","last_active":"2024-02-23T00:29:04.110668+0000","last_peered":"2024-02-23T00:29:04.110668+0000","last_clean":"2024-02-23T00:29:04.110668+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:29:04.110668+0000","last_undegraded":"2024-02-23T00:29:04.110668+0000","last_fullsized":"2024-02-23T00:29:04.110668+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:40.630153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:40.630153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:30.350841+0000","last_change":"2024-02-23T00:28:44.617114+0000","last_active":"2024-02-23T00:29:30.350841+0000","last_peered":"2024-02-23T00:29:30.350841+0000","last_clean":"2024-02-23T00:29:30.350841+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:29:30.350841+0000","last_undegraded":"2024-02-23T00:29:30.350841+0000","last_fullsized":"2024-02-23T00:29:30.350841+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:44.617038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.617038+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":50,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111700+0000","last_change":"2024-02-23T00:29:00.703800+0000","last_active":"2024-02-23T00:29:04.111700+0000","last_peered":"2024-02-23T00:29:04.111700+0000","last_clean":"2024-02-23T00:29:04.111700+0000","last_became_active":"2024-02-23T00:28:41.203031+0000","last_became_peered":"2024-02-23T00:28:41.203031+0000","last_unstale":"2024-02-23T00:29:04.111700+0000","last_undegraded":"2024-02-23T00:29:04.111700+0000","last_fullsized":"2024-02-23T00:29:04.111700+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.703737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:08.818370+0000","last_change":"2024-02-23T00:29:08.818324+0000","last_active":"2024-02-23T00:29:08.818370+0000","last_peered":"2024-02-23T00:29:08.818370+0000","last_clean":"2024-02-23T00:29:08.818370+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:29:08.818370+0000","last_undegraded":"2024-02-23T00:29:08.818370+0000","last_fullsized":"2024-02-23T00:29:08.818370+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:29:08.818235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.818235+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112456+0000","last_change":"2024-02-23T00:28:47.677799+0000","last_active":"2024-02-23T00:29:04.112456+0000","last_peered":"2024-02-23T00:29:04.112456+0000","last_clean":"2024-02-23T00:29:04.112456+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:29:04.112456+0000","last_undegraded":"2024-02-23T00:29:04.112456+0000","last_fullsized":"2024-02-23T00:29:04.112456+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:47.677731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.677731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:08.580191+0000","last_change":"2024-02-23T00:29:08.580168+0000","last_active":"2024-02-23T00:29:08.580191+0000","last_peered":"2024-02-23T00:29:08.580191+0000","last_clean":"2024-02-23T00:29:08.580191+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:29:08.580191+0000","last_undegraded":"2024-02-23T00:29:08.580191+0000","last_fullsized":"2024-02-23T00:29:08.580191+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.580102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.580102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:05.279997+0000","last_change":"2024-02-23T00:29:05.279969+0000","last_active":"2024-02-23T00:29:05.279997+0000","last_peered":"2024-02-23T00:29:05.279997+0000","last_clean":"2024-02-23T00:29:05.279997+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:29:05.279997+0000","last_undegraded":"2024-02-23T00:29:05.279997+0000","last_fullsized":"2024-02-23T00:29:05.279997+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:05.279882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:05.279882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.110952+0000","last_change":"2024-02-23T00:28:29.526599+0000","last_active":"2024-02-23T00:29:04.110952+0000","last_peered":"2024-02-23T00:29:04.110952+0000","last_clean":"2024-02-23T00:29:04.110952+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:29:04.110952+0000","last_undegraded":"2024-02-23T00:29:04.110952+0000","last_fullsized":"2024-02-23T00:29:04.110952+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:28:29.526505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:29.526505+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166861+0000","last_change":"2024-02-23T00:28:49.331226+0000","last_active":"2024-02-23T00:29:04.166861+0000","last_peered":"2024-02-23T00:29:04.166861+0000","last_clean":"2024-02-23T00:29:04.166861+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:29:04.166861+0000","last_undegraded":"2024-02-23T00:29:04.166861+0000","last_fullsized":"2024-02-23T00:29:04.166861+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.331139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.331139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.113319+0000","last_change":"2024-02-23T00:29:01.776038+0000","last_active":"2024-02-23T00:29:04.113319+0000","last_peered":"2024-02-23T00:29:04.113319+0000","last_clean":"2024-02-23T00:29:04.113319+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:29:04.113319+0000","last_undegraded":"2024-02-23T00:29:04.113319+0000","last_fullsized":"2024-02-23T00:29:04.113319+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:01.775943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.775943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:24.809313+0000","last_change":"2024-02-23T00:29:24.809313+0000","last_active":"2024-02-23T00:29:24.809313+0000","last_peered":"2024-02-23T00:29:24.809313+0000","last_clean":"2024-02-23T00:29:24.809313+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:29:24.809313+0000","last_undegraded":"2024-02-23T00:29:24.809313+0000","last_fullsized":"2024-02-23T00:29:24.809313+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:24.809192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:24.809192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":85,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.166687+0000","last_change":"2024-02-23T00:28:28.095098+0000","last_active":"2024-02-23T00:29:04.166687+0000","last_peered":"2024-02-23T00:29:04.166687+0000","last_clean":"2024-02-23T00:29:04.166687+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:29:04.166687+0000","last_undegraded":"2024-02-23T00:29:04.166687+0000","last_fullsized":"2024-02-23T00:29:04.166687+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:28:28.095011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.095011+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165173+0000","last_change":"2024-02-23T00:29:00.187868+0000","last_active":"2024-02-23T00:29:04.165173+0000","last_peered":"2024-02-23T00:29:04.165173+0000","last_clean":"2024-02-23T00:29:04.165173+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:29:04.165173+0000","last_undegraded":"2024-02-23T00:29:04.165173+0000","last_fullsized":"2024-02-23T00:29:04.165173+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.187784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.187784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.112896+0000","last_change":"2024-02-23T00:29:02.769052+0000","last_active":"2024-02-23T00:29:04.112896+0000","last_peered":"2024-02-23T00:29:04.112896+0000","last_clean":"2024-02-23T00:29:04.112896+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:29:04.112896+0000","last_undegraded":"2024-02-23T00:29:04.112896+0000","last_fullsized":"2024-02-23T00:29:04.112896+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.768986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.768986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.111919+0000","last_change":"2024-02-23T00:28:26.541700+0000","last_active":"2024-02-23T00:29:04.111919+0000","last_peered":"2024-02-23T00:29:04.111919+0000","last_clean":"2024-02-23T00:29:04.111919+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:29:04.111919+0000","last_undegraded":"2024-02-23T00:29:04.111919+0000","last_fullsized":"2024-02-23T00:29:04.111919+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:26.541633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:26.541633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":70,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165871+0000","last_change":"2024-02-23T00:28:53.186087+0000","last_active":"2024-02-23T00:29:04.165871+0000","last_peered":"2024-02-23T00:29:04.165871+0000","last_clean":"2024-02-23T00:29:04.165871+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:29:04.165871+0000","last_undegraded":"2024-02-23T00:29:04.165871+0000","last_fullsized":"2024-02-23T00:29:04.165871+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:28:53.186017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.186017+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:08.063504+0000","last_change":"2024-02-23T00:29:08.063475+0000","last_active":"2024-02-23T00:29:08.063504+0000","last_peered":"2024-02-23T00:29:08.063504+0000","last_clean":"2024-02-23T00:29:08.063504+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:29:08.063504+0000","last_undegraded":"2024-02-23T00:29:08.063504+0000","last_fullsized":"2024-02-23T00:29:08.063504+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.063398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.063398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:27.875203+0000","last_change":"2024-02-23T00:29:27.875203+0000","last_active":"2024-02-23T00:29:27.875203+0000","last_peered":"2024-02-23T00:29:27.875203+0000","last_clean":"2024-02-23T00:29:27.875203+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:29:27.875203+0000","last_undegraded":"2024-02-23T00:29:27.875203+0000","last_fullsized":"2024-02-23T00:29:27.875203+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:27.875106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:27.875106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":101,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-23T00:29:04.165621+0000","last_change":"2024-02-23T00:28:46.118981+0000","last_active":"2024-02-23T00:29:04.165621+0000","last_peered":"2024-02-23T00:29:04.165621+0000","last_clean":"2024-02-23T00:29:04.165621+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:29:04.165621+0000","last_undegraded":"2024-02-23T00:29:04.165621+0000","last_fullsized":"2024-02-23T00:29:04.165621+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.118918+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'8","reported_seq":72,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:30.351335+0000","last_change":"2024-02-23T00:28:28.676907+0000","last_active":"2024-02-23T00:29:30.351335+0000","last_peered":"2024-02-23T00:29:30.351335+0000","last_clean":"2024-02-23T00:29:30.351335+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:29:30.351335+0000","last_undegraded":"2024-02-23T00:29:30.351335+0000","last_fullsized":"2024-02-23T00:29:30.351335+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:28.676837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.676837+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":616,"ondisk_log_size":616,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":127,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332127,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":290664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.502,"15min":0.501},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.437},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68300000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51600000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.61799999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.505}]},{"osd":4,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.28599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.35199999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42799999999999999,"15min":0.42699999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.436,"15min":0.42799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.63400000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.379}]},{"osd":6,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.47699999999999998,"15min":0.47099999999999997},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.46000000000000002,"15min":0.45400000000000001},"min":{"1min":0.33900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.39200000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561060,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":289807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.28599999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.51000000000000001,"15min":0.51700000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69399999999999995,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.46700000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.57799999999999996,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.31,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70299999999999996,"5min":0.754,"15min":0.754},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.55100000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.49399999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48499999999999999,"15min":0.48699999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66800000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.44500000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40899999999999997,"15min":0.40600000000000003},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61499999999999999,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.36299999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42399999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61699999999999999,"5min":0.996,"15min":0.996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.42199999999999999,"15min":0.41999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.45700000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.48299999999999998,"15min":0.47599999999999998},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.38800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297748,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447404,"statfs":{"total":95995035648,"available":95690141696,"internally_reserved":0,"allocated":667648,"data_stored":291667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.32800000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.372,"15min":0.35099999999999998},"min":{"1min":0.215,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54600000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.38}]},{"osd":2,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.37,"15min":0.371},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.433}]},{"osd":3,"last update":"Fri Feb 23 00:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.41399999999999998,"15min":0.42599999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.49299999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.438,"15min":0.435},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.39200000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72399999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.64800000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79100000000000004,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.623},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.68600000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.71599999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.85999999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:29:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78500000000000003,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.60899999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.752,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.71399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738433,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":290324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34399999999999997,"15min":0.32800000000000001},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.501,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35599999999999998,"15min":0.33300000000000002},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.49099999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.46000000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.38500000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.46899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.45000000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.43099999999999999,"15min":0.44700000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56200000000000006,"5min":0.629,"15min":0.629},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.313,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53100000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.497}]},{"osd":4,"last update":"Fri Feb 23 00:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.39100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.745}]},{"osd":5,"last update":"Fri Feb 23 00:29:26 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.432,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.72199999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41299999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.79000000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411383,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":630784,"data_stored":290227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.33700000000000002,"15min":0.32900000000000001},"min":{"1min":0.23799999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.36099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.34300000000000003,"15min":0.32900000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.42699999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.45800000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.35599999999999998,"15min":0.34000000000000002},"min":{"1min":0.245,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47599999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35099999999999998,"15min":0.33900000000000002},"min":{"1min":0.25900000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.44500000000000001,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.35799999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:28:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40100000000000002,"15min":0.41199999999999998},"min":{"1min":0.26900000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.49299999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.433},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.42999999999999999,"15min":0.438},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.54400000000000004,"5min":0.63,"15min":0.63},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.505},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58699999999999997}]},{"osd":5,"last update":"Fri Feb 23 00:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55000000000000004,"15min":0.54500000000000004},"min":{"1min":0.35599999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.76500000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73899999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.59899999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58799999999999997,"15min":0.59299999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.60799999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77600000000000002,"5min":1.117,"15min":1.117},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80600000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.52700000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":289924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.36699999999999999,"15min":0.371},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.49199999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.38400000000000001,"15min":0.39700000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.50800000000000001,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.378}]},{"osd":1,"last update":"Fri Feb 23 00:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.40699999999999997,"15min":0.42699999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.437,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.52600000000000002,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.316}]},{"osd":2,"last update":"Fri Feb 23 00:28:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.43099999999999999,"15min":0.439},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54900000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.47799999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.29899999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.30299999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68799999999999994,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.48299999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.38100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.52500000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76600000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51100000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.61199999999999999},"min":{"1min":0.498,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.47599999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986222,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":289907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.35699999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66600000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.55900000000000005}]},{"osd":1,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.5,"15min":0.52000000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51000000000000001,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.65200000000000002,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67200000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.504},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.379,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.70299999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69099999999999995,"5min":0.755,"15min":0.755},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.50800000000000001,"15min":0.53000000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.49199999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.38300000000000001,"15min":0.36799999999999999},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.42099999999999999,"15min":0.40200000000000002},"min":{"1min":0.34200000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.34599999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.55500000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.43099999999999999,"15min":0.42899999999999999},"min":{"1min":0.29399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.56399999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.53800000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.46999999999999997,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.622}]}]},{"osd":5,"up_from":38,"seq":163208757289,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":289441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.505},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70599999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.34599999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47999999999999998,"15min":0.47799999999999998},"min":{"1min":0.34999999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67100000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.49399999999999999,"15min":0.495},"min":{"1min":0.39900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.70399999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66100000000000003,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36699999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.56399999999999995}]},{"osd":3,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73299999999999998,"5min":0.753,"15min":0.753},"last":0.68600000000000005}]},{"osd":4,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.38600000000000001,"15min":0.371},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39400000000000002,"15min":0.372},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.53900000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.434,"15min":0.42399999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.442,"15min":0.434},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.69499999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69599999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.71899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:29:39.090 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:29:39.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:37.483566+0000 mgr.y (mgr.24437) 96 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:39.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:38.094428+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.14 deep-scrub starts 2024-02-23T00:29:39.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:38.098438+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.14 deep-scrub ok 2024-02-23T00:29:39.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:38.854888+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:29:39.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:38.857025+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:29:39.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:37.483566+0000 mgr.y (mgr.24437) 96 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:39.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:38.094428+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.14 deep-scrub starts 2024-02-23T00:29:39.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:38.098438+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.14 deep-scrub ok 2024-02-23T00:29:39.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:38.854888+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:29:39.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:38.857025+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:29:39.362 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:37.483566+0000 mgr.y (mgr.24437) 96 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:38.094428+0000 osd.3 (osd.3) 53 : cluster [DBG] 5.14 deep-scrub starts 2024-02-23T00:29:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:38.098438+0000 osd.3 (osd.3) 54 : cluster [DBG] 5.14 deep-scrub ok 2024-02-23T00:29:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:38.854888+0000 osd.4 (osd.4) 45 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:29:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:38.857025+0000 osd.4 (osd.4) 46 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:29:40.566 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:38.295569+0000 mgr.y (mgr.24437) 97 : audit [DBG] from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:40.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:38.880059+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.1f deep-scrub starts 2024-02-23T00:29:40.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:38.884126+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.1f deep-scrub ok 2024-02-23T00:29:40.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:39.107490+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:29:40.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:39.108377+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:29:40.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:39.219087+0000 osd.3 (osd.3) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:40.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:39.219133+0000 osd.3 (osd.3) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:40.640 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:38.295569+0000 mgr.y (mgr.24437) 97 : audit [DBG] from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:40.640 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:38.880059+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.1f deep-scrub starts 2024-02-23T00:29:40.640 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:38.884126+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.1f deep-scrub ok 2024-02-23T00:29:40.640 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:39.107490+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:29:40.640 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:39.108377+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:29:40.640 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:39.219087+0000 osd.3 (osd.3) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:40.641 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:39.219133+0000 osd.3 (osd.3) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:40.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:38.295569+0000 mgr.y (mgr.24437) 97 : audit [DBG] from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:40.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:38.880059+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.1f deep-scrub starts 2024-02-23T00:29:40.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:38.884126+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.1f deep-scrub ok 2024-02-23T00:29:40.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:39.107490+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:29:40.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:39.108377+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:29:40.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:39.219087+0000 osd.3 (osd.3) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:40.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:39.219133+0000 osd.3 (osd.3) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:40.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:29:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:29:41.254 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:39.483948+0000 mgr.y (mgr.24437) 98 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:41.254 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:39.778234+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:29:41.254 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:39.778980+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:40.213646+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:40.565040+0000 mgr.y (mgr.24437) 99 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:40.939411+0000 osd.4 (osd.4) 47 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:40.940441+0000 osd.4 (osd.4) 48 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:40.945846+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:40.945907+0000 osd.7 (osd.7) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:39.483948+0000 mgr.y (mgr.24437) 98 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:39.778234+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:39.778980+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:40.213646+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-23T00:29:41.255 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:40.565040+0000 mgr.y (mgr.24437) 99 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:41.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:40.939411+0000 osd.4 (osd.4) 47 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:29:41.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:40.940441+0000 osd.4 (osd.4) 48 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:29:41.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:40.945846+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:41.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:40.945907+0000 osd.7 (osd.7) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:41.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.512+0000 7f497bd98700 1 -- 172.21.15.87:0/3970433282 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740f8420 msgr2=0x7f4974104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:41.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.512+0000 7f497bd98700 1 --2- 172.21.15.87:0/3970433282 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740f8420 0x7f4974104a20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f496c009c30 tx=0x7f496c004c50).stop 2024-02-23T00:29:41.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.512+0000 7f497bd98700 1 -- 172.21.15.87:0/3970433282 shutdown_connections 2024-02-23T00:29:41.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.512+0000 7f497bd98700 1 --2- 172.21.15.87:0/3970433282 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740f8420 0x7f4974104a20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.512+0000 7f497bd98700 1 --2- 172.21.15.87:0/3970433282 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49740fdbb0 0x7f49740fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.512+0000 7f497bd98700 1 --2- 172.21.15.87:0/3970433282 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49740fe510 0x7f49740f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.512+0000 7f497bd98700 1 -- 172.21.15.87:0/3970433282 >> 172.21.15.87:0/3970433282 conn(0x7f49740f3800 msgr2=0x7f49740f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.512+0000 7f497bd98700 1 -- 172.21.15.87:0/3970433282 shutdown_connections 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.512+0000 7f497bd98700 1 -- 172.21.15.87:0/3970433282 wait complete. 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 librados: starting msgr at 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 librados: starting objecter 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 Processor -- start 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 -- start start 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 librados: setting wanted keys 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 librados: calling monclient init 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49740f8420 0x7f497410de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740fdbb0 0x7f497410e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:41.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49740fe510 0x7f4974108030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4974108520 con 0x7f49740fe510 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4974108690 con 0x7f49740f8420 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497bd98700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4974108850 con 0x7f49740fdbb0 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f4979333700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740fdbb0 0x7f497410e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497a335700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49740fe510 0x7f4974108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f4979333700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740fdbb0 0x7f497410e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:59732/0 (socket says 172.21.15.87:59732) 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f497a335700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49740fe510 0x7f4974108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43778/0 (socket says 172.21.15.87:43778) 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f4979333700 1 -- 172.21.15.87:0/2217657526 learned_addr learned my addr 172.21.15.87:0/2217657526 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.513+0000 7f4979b34700 1 --2- 172.21.15.87:0/2217657526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49740f8420 0x7f497410de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f4979333700 1 -- 172.21.15.87:0/2217657526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49740f8420 msgr2=0x7f497410de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f4979333700 1 --2- 172.21.15.87:0/2217657526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49740f8420 0x7f497410de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f4979333700 1 -- 172.21.15.87:0/2217657526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49740fe510 msgr2=0x7f4974108030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f4979333700 1 --2- 172.21.15.87:0/2217657526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49740fe510 0x7f4974108030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f4979333700 1 -- 172.21.15.87:0/2217657526 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4964005040 con 0x7f49740fdbb0 2024-02-23T00:29:41.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f497a335700 1 --2- 172.21.15.87:0/2217657526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49740fe510 0x7f4974108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f4979333700 1 --2- 172.21.15.87:0/2217657526 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740fdbb0 0x7f497410e380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f4964009400 tx=0x7f4964009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f496affd700 10 client.?.objecter ms_handle_connect 0x7f49740fdbb0 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f496affd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f496affd700 1 -- 172.21.15.87:0/2217657526 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4964007a40 con 0x7f49740fdbb0 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f496affd700 1 -- 172.21.15.87:0/2217657526 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f496400f040 con 0x7f49740fdbb0 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496c0097c0 con 0x7f49740fdbb0 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f496affd700 1 -- 172.21.15.87:0/2217657526 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4964014a70 con 0x7f49740fdbb0 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f497bd98700 10 client.24659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4974108e90 con 0x7f49740fdbb0 2024-02-23T00:29:41.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.514+0000 7f497bd98700 1 librados: init done 2024-02-23T00:29:41.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.515+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497405ee80 con 0x7f49740fdbb0 2024-02-23T00:29:41.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.515+0000 7f496affd700 1 -- 172.21.15.87:0/2217657526 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4964010530 con 0x7f49740fdbb0 2024-02-23T00:29:41.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.516+0000 7f496affd700 1 --2- 172.21.15.87:0/2217657526 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4960062c90 0x7f4960065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:41.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.516+0000 7f496affd700 1 -- 172.21.15.87:0/2217657526 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(84..84 src has 28..84) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f496407c650 con 0x7f49740fdbb0 2024-02-23T00:29:41.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.516+0000 7f496affd700 10 client.24659.objecter ms_dispatch 0x7f4974001550 osd_map(84..84 src has 28..84) v4 2024-02-23T00:29:41.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.516+0000 7f496affd700 3 client.24659.objecter handle_osd_map got epochs [84,84] > 0 2024-02-23T00:29:41.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.516+0000 7f496affd700 3 client.24659.objecter handle_osd_map decoding full epoch 84 2024-02-23T00:29:41.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.516+0000 7f4979b34700 1 --2- 172.21.15.87:0/2217657526 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4960062c90 0x7f4960065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:41.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.516+0000 7f496affd700 20 client.24659.objecter dump_active .. 0 homeless 2024-02-23T00:29:41.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.516+0000 7f496affd700 10 client.24659.objecter ms_handle_connect 0x7f4960062c90 2024-02-23T00:29:41.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.516+0000 7f4979b34700 1 --2- 172.21.15.87:0/2217657526 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4960062c90 0x7f4960065150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f4970004de0 tx=0x7f497000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:41.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.520+0000 7f496affd700 1 -- 172.21.15.87:0/2217657526 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4964049f60 con 0x7f49740fdbb0 2024-02-23T00:29:41.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.685+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f497405fa60 con 0x7f4960062c90 2024-02-23T00:29:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:39.483948+0000 mgr.y (mgr.24437) 98 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:39.778234+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:29:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:39.778980+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:29:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:40.213646+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-23T00:29:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:40.565040+0000 mgr.y (mgr.24437) 99 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:40.939411+0000 osd.4 (osd.4) 47 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:29:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:40.940441+0000 osd.4 (osd.4) 48 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:29:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:40.945846+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:40.945907+0000 osd.7 (osd.7) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:41.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.695+0000 7f496affd700 1 -- 172.21.15.87:0/2217657526 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357477 (secure 0 0 0) 0x7f497405fa60 con 0x7f4960062c90 2024-02-23T00:29:41.702 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:29:41.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.697+0000 7f497bd98700 10 librados: watch_flush enter 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.697+0000 7f497bd98700 10 librados: watch_flush exit 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.697+0000 7f497bd98700 20 client.24659.objecter shutdown clearing up homeless session... 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.697+0000 7f497bd98700 10 client.24659.objecter successfully canceled tick 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4960062c90 msgr2=0x7f4960065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 --2- 172.21.15.87:0/2217657526 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4960062c90 0x7f4960065150 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f4970004de0 tx=0x7f497000b040).stop 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740fdbb0 msgr2=0x7f497410e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 --2- 172.21.15.87:0/2217657526 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740fdbb0 0x7f497410e380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f4964009400 tx=0x7f4964009cb0).stop 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 shutdown_connections 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 --2- 172.21.15.87:0/2217657526 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4960062c90 0x7f4960065150 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 --2- 172.21.15.87:0/2217657526 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49740fe510 0x7f4974108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 --2- 172.21.15.87:0/2217657526 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49740f8420 0x7f497410de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 --2- 172.21.15.87:0/2217657526 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49740fdbb0 0x7f497410e380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:41.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 >> 172.21.15.87:0/2217657526 conn(0x7f49740f3800 msgr2=0x7f49740f50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:41.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 shutdown_connections 2024-02-23T00:29:41.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 -- 172.21.15.87:0/2217657526 wait complete. 2024-02-23T00:29:41.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:41.698+0000 7f497bd98700 1 librados: shutdown 2024-02-23T00:29:41.707 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:29:42.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:41.219501+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-23T00:29:42.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:41.484309+0000 mgr.y (mgr.24437) 100 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:42.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:41.686829+0000 mgr.y (mgr.24437) 101 : audit [DBG] from='client.24659 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:42.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:41.752540+0000 osd.1 (osd.1) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:42.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:41.752606+0000 osd.1 (osd.1) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:42.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:41.219501+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-23T00:29:42.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:41.484309+0000 mgr.y (mgr.24437) 100 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:42.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:41.686829+0000 mgr.y (mgr.24437) 101 : audit [DBG] from='client.24659 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:42.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:41.752540+0000 osd.1 (osd.1) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:42.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:41.752606+0000 osd.1 (osd.1) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:42.522 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:42] "GET /metrics HTTP/1.1" 200 172167 "" "Prometheus/2.33.4" 2024-02-23T00:29:42.540 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-02-23T00:29:41.484088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":224,"num_object_clones":0,"num_object_copies":672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":1241,"num_read_kb":961,"num_write":620,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":696,"ondisk_log_size":696,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381756,"kb_used_data":4988,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579460,"statfs":{"total":767960285184,"available":765521367040,"internally_reserved":0,"allocated":5107712,"data_stored":2328375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001669"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.135917+0000","last_change":"2024-02-23T00:28:52.093263+0000","last_active":"2024-02-23T00:29:15.135917+0000","last_peered":"2024-02-23T00:29:15.135917+0000","last_clean":"2024-02-23T00:29:15.135917+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:29:15.135917+0000","last_undegraded":"2024-02-23T00:29:15.135917+0000","last_fullsized":"2024-02-23T00:29:15.135917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:52.093177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.093177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.135680+0000","last_change":"2024-02-23T00:28:57.678604+0000","last_active":"2024-02-23T00:29:15.135680+0000","last_peered":"2024-02-23T00:29:15.135680+0000","last_clean":"2024-02-23T00:29:15.135680+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:29:15.135680+0000","last_undegraded":"2024-02-23T00:29:15.135680+0000","last_fullsized":"2024-02-23T00:29:15.135680+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:57.678522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.678522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350151+0000","last_change":"2024-02-23T00:28:46.217573+0000","last_active":"2024-02-23T00:29:15.350151+0000","last_peered":"2024-02-23T00:29:15.350151+0000","last_clean":"2024-02-23T00:29:15.350151+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:29:15.350151+0000","last_undegraded":"2024-02-23T00:29:15.350151+0000","last_fullsized":"2024-02-23T00:29:15.350151+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:46.217507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.217507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"63'39","reported_seq":125,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136217+0000","last_change":"2024-02-23T00:28:37.040298+0000","last_active":"2024-02-23T00:29:15.136217+0000","last_peered":"2024-02-23T00:29:15.136217+0000","last_clean":"2024-02-23T00:29:15.136217+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:29:15.136217+0000","last_undegraded":"2024-02-23T00:29:15.136217+0000","last_fullsized":"2024-02-23T00:29:15.136217+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'39","last_scrub_stamp":"2024-02-23T00:28:37.040215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:37.040215+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.135929+0000","last_change":"2024-02-23T00:28:59.685150+0000","last_active":"2024-02-23T00:29:15.135929+0000","last_peered":"2024-02-23T00:29:15.135929+0000","last_clean":"2024-02-23T00:29:15.135929+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:29:15.135929+0000","last_undegraded":"2024-02-23T00:29:15.135929+0000","last_fullsized":"2024-02-23T00:29:15.135929+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:59.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:59.685045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.166826+0000","last_change":"2024-02-23T00:28:49.663875+0000","last_active":"2024-02-23T00:29:15.166826+0000","last_peered":"2024-02-23T00:29:15.166826+0000","last_clean":"2024-02-23T00:29:15.166826+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:29:15.166826+0000","last_undegraded":"2024-02-23T00:29:15.166826+0000","last_fullsized":"2024-02-23T00:29:15.166826+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.663808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.663808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136688+0000","last_change":"2024-02-23T00:28:47.073631+0000","last_active":"2024-02-23T00:29:15.136688+0000","last_peered":"2024-02-23T00:29:15.136688+0000","last_clean":"2024-02-23T00:29:15.136688+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:29:15.136688+0000","last_undegraded":"2024-02-23T00:29:15.136688+0000","last_fullsized":"2024-02-23T00:29:15.136688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:47.073551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.073551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":76,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.135850+0000","last_change":"2024-02-23T00:29:06.822899+0000","last_active":"2024-02-23T00:29:15.135850+0000","last_peered":"2024-02-23T00:29:15.135850+0000","last_clean":"2024-02-23T00:29:15.135850+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:29:15.135850+0000","last_undegraded":"2024-02-23T00:29:15.135850+0000","last_fullsized":"2024-02-23T00:29:15.135850+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:29:06.822818+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.260844+0000","last_change":"2024-02-23T00:29:04.049502+0000","last_active":"2024-02-23T00:29:15.260844+0000","last_peered":"2024-02-23T00:29:15.260844+0000","last_clean":"2024-02-23T00:29:15.260844+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:29:15.260844+0000","last_undegraded":"2024-02-23T00:29:15.260844+0000","last_fullsized":"2024-02-23T00:29:15.260844+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.049414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277492+0000","last_change":"2024-02-23T00:29:00.320083+0000","last_active":"2024-02-23T00:29:15.277492+0000","last_peered":"2024-02-23T00:29:15.277492+0000","last_clean":"2024-02-23T00:29:15.277492+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:29:15.277492+0000","last_undegraded":"2024-02-23T00:29:15.277492+0000","last_fullsized":"2024-02-23T00:29:15.277492+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.319985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.319985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277723+0000","last_change":"2024-02-23T00:28:46.396553+0000","last_active":"2024-02-23T00:29:15.277723+0000","last_peered":"2024-02-23T00:29:15.277723+0000","last_clean":"2024-02-23T00:29:15.277723+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:29:15.277723+0000","last_undegraded":"2024-02-23T00:29:15.277723+0000","last_fullsized":"2024-02-23T00:29:15.277723+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.396471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":68,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136041+0000","last_change":"2024-02-23T00:28:54.657266+0000","last_active":"2024-02-23T00:29:15.136041+0000","last_peered":"2024-02-23T00:29:15.136041+0000","last_clean":"2024-02-23T00:29:15.136041+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:29:15.136041+0000","last_undegraded":"2024-02-23T00:29:15.136041+0000","last_fullsized":"2024-02-23T00:29:15.136041+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.657177+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.349763+0000","last_change":"2024-02-23T00:28:50.176282+0000","last_active":"2024-02-23T00:29:15.349763+0000","last_peered":"2024-02-23T00:29:15.349763+0000","last_clean":"2024-02-23T00:29:15.349763+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:29:15.349763+0000","last_undegraded":"2024-02-23T00:29:15.349763+0000","last_fullsized":"2024-02-23T00:29:15.349763+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.176208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261120+0000","last_change":"2024-02-23T00:28:33.933658+0000","last_active":"2024-02-23T00:29:15.261120+0000","last_peered":"2024-02-23T00:29:15.261120+0000","last_clean":"2024-02-23T00:29:15.261120+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:29:15.261120+0000","last_undegraded":"2024-02-23T00:29:15.261120+0000","last_fullsized":"2024-02-23T00:29:15.261120+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.933529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.933529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277453+0000","last_change":"2024-02-23T00:29:04.267507+0000","last_active":"2024-02-23T00:29:15.277453+0000","last_peered":"2024-02-23T00:29:15.277453+0000","last_clean":"2024-02-23T00:29:15.277453+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:29:15.277453+0000","last_undegraded":"2024-02-23T00:29:15.277453+0000","last_fullsized":"2024-02-23T00:29:15.277453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.267427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.267427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":91,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:38.857163+0000","last_change":"2024-02-23T00:29:38.857115+0000","last_active":"2024-02-23T00:29:38.857163+0000","last_peered":"2024-02-23T00:29:38.857163+0000","last_clean":"2024-02-23T00:29:38.857163+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:29:38.857163+0000","last_undegraded":"2024-02-23T00:29:38.857163+0000","last_fullsized":"2024-02-23T00:29:38.857163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:29:38.857029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.857029+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136644+0000","last_change":"2024-02-23T00:28:29.086512+0000","last_active":"2024-02-23T00:29:15.136644+0000","last_peered":"2024-02-23T00:29:15.136644+0000","last_clean":"2024-02-23T00:29:15.136644+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:29:15.136644+0000","last_undegraded":"2024-02-23T00:29:15.136644+0000","last_fullsized":"2024-02-23T00:29:15.136644+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:29.086437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:29.086437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136300+0000","last_change":"2024-02-23T00:29:03.790608+0000","last_active":"2024-02-23T00:29:15.136300+0000","last_peered":"2024-02-23T00:29:15.136300+0000","last_clean":"2024-02-23T00:29:15.136300+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:29:15.136300+0000","last_undegraded":"2024-02-23T00:29:15.136300+0000","last_fullsized":"2024-02-23T00:29:15.136300+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.790530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.790530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136091+0000","last_change":"2024-02-23T00:28:30.086083+0000","last_active":"2024-02-23T00:29:15.136091+0000","last_peered":"2024-02-23T00:29:15.136091+0000","last_clean":"2024-02-23T00:29:15.136091+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:29:15.136091+0000","last_undegraded":"2024-02-23T00:29:15.136091+0000","last_fullsized":"2024-02-23T00:29:15.136091+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:30.086009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:30.086009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":84,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136295+0000","last_change":"2024-02-23T00:28:33.077947+0000","last_active":"2024-02-23T00:29:15.136295+0000","last_peered":"2024-02-23T00:29:15.136295+0000","last_clean":"2024-02-23T00:29:15.136295+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:29:15.136295+0000","last_undegraded":"2024-02-23T00:29:15.136295+0000","last_fullsized":"2024-02-23T00:29:15.136295+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.077848+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136621+0000","last_change":"2024-02-23T00:29:10.830479+0000","last_active":"2024-02-23T00:29:15.136621+0000","last_peered":"2024-02-23T00:29:15.136621+0000","last_clean":"2024-02-23T00:29:15.136621+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:29:15.136621+0000","last_undegraded":"2024-02-23T00:29:15.136621+0000","last_fullsized":"2024-02-23T00:29:15.136621+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.830389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.830389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167447+0000","last_change":"2024-02-23T00:29:05.641525+0000","last_active":"2024-02-23T00:29:15.167447+0000","last_peered":"2024-02-23T00:29:15.167447+0000","last_clean":"2024-02-23T00:29:15.167447+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:29:15.167447+0000","last_undegraded":"2024-02-23T00:29:15.167447+0000","last_fullsized":"2024-02-23T00:29:15.167447+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_clean_scrub_stamp":"2024-02-23T00:29:05.641454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:28.878602+0000","last_change":"2024-02-23T00:29:28.878569+0000","last_active":"2024-02-23T00:29:28.878602+0000","last_peered":"2024-02-23T00:29:28.878602+0000","last_clean":"2024-02-23T00:29:28.878602+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:29:28.878602+0000","last_undegraded":"2024-02-23T00:29:28.878602+0000","last_fullsized":"2024-02-23T00:29:28.878602+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:28.878491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:28.878491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:31.823495+0000","last_change":"2024-02-23T00:29:31.823473+0000","last_active":"2024-02-23T00:29:31.823495+0000","last_peered":"2024-02-23T00:29:31.823495+0000","last_clean":"2024-02-23T00:29:31.823495+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:29:31.823495+0000","last_undegraded":"2024-02-23T00:29:31.823495+0000","last_fullsized":"2024-02-23T00:29:31.823495+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:29:31.823405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:31.823405+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:37.520530+0000","last_change":"2024-02-23T00:29:37.520506+0000","last_active":"2024-02-23T00:29:37.520530+0000","last_peered":"2024-02-23T00:29:37.520530+0000","last_clean":"2024-02-23T00:29:37.520530+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:29:37.520530+0000","last_undegraded":"2024-02-23T00:29:37.520530+0000","last_fullsized":"2024-02-23T00:29:37.520530+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:37.520440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:37.520440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350304+0000","last_change":"2024-02-23T00:29:10.130797+0000","last_active":"2024-02-23T00:29:15.350304+0000","last_peered":"2024-02-23T00:29:15.350304+0000","last_clean":"2024-02-23T00:29:15.350304+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:29:15.350304+0000","last_undegraded":"2024-02-23T00:29:15.350304+0000","last_fullsized":"2024-02-23T00:29:15.350304+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.130727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.130727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136294+0000","last_change":"2024-02-23T00:28:32.090929+0000","last_active":"2024-02-23T00:29:15.136294+0000","last_peered":"2024-02-23T00:29:15.136294+0000","last_clean":"2024-02-23T00:29:15.136294+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:29:15.136294+0000","last_undegraded":"2024-02-23T00:29:15.136294+0000","last_fullsized":"2024-02-23T00:29:15.136294+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:32.090869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:32.090869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":74,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350928+0000","last_change":"2024-02-23T00:28:44.987515+0000","last_active":"2024-02-23T00:29:15.350928+0000","last_peered":"2024-02-23T00:29:15.350928+0000","last_clean":"2024-02-23T00:29:15.350928+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:29:15.350928+0000","last_undegraded":"2024-02-23T00:29:15.350928+0000","last_fullsized":"2024-02-23T00:29:15.350928+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:28:44.987419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.987419+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277657+0000","last_change":"2024-02-23T00:28:48.333707+0000","last_active":"2024-02-23T00:29:15.277657+0000","last_peered":"2024-02-23T00:29:15.277657+0000","last_clean":"2024-02-23T00:29:15.277657+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:29:15.277657+0000","last_undegraded":"2024-02-23T00:29:15.277657+0000","last_fullsized":"2024-02-23T00:29:15.277657+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:28:48.333638+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":83,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.355578+0000","last_change":"2024-02-23T00:29:38.884205+0000","last_active":"2024-02-23T00:29:40.355578+0000","last_peered":"2024-02-23T00:29:40.355578+0000","last_clean":"2024-02-23T00:29:40.355578+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:29:40.355578+0000","last_undegraded":"2024-02-23T00:29:40.355578+0000","last_fullsized":"2024-02-23T00:29:40.355578+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.884131+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.260789+0000","last_change":"2024-02-23T00:29:03.017128+0000","last_active":"2024-02-23T00:29:15.260789+0000","last_peered":"2024-02-23T00:29:15.260789+0000","last_clean":"2024-02-23T00:29:15.260789+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:29:15.260789+0000","last_undegraded":"2024-02-23T00:29:15.260789+0000","last_fullsized":"2024-02-23T00:29:15.260789+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.017051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.017051+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"63'16","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350263+0000","last_change":"2024-02-23T00:28:55.230148+0000","last_active":"2024-02-23T00:29:15.350263+0000","last_peered":"2024-02-23T00:29:15.350263+0000","last_clean":"2024-02-23T00:29:15.350263+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:29:15.350263+0000","last_undegraded":"2024-02-23T00:29:15.350263+0000","last_fullsized":"2024-02-23T00:29:15.350263+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:28:55.230081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.230081+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136281+0000","last_change":"2024-02-23T00:28:41.113778+0000","last_active":"2024-02-23T00:29:15.136281+0000","last_peered":"2024-02-23T00:29:15.136281+0000","last_clean":"2024-02-23T00:29:15.136281+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:29:15.136281+0000","last_undegraded":"2024-02-23T00:29:15.136281+0000","last_fullsized":"2024-02-23T00:29:15.136281+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.113684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.113684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277372+0000","last_change":"2024-02-23T00:28:41.520505+0000","last_active":"2024-02-23T00:29:15.277372+0000","last_peered":"2024-02-23T00:29:15.277372+0000","last_clean":"2024-02-23T00:29:15.277372+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:29:15.277372+0000","last_undegraded":"2024-02-23T00:29:15.277372+0000","last_fullsized":"2024-02-23T00:29:15.277372+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.520412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.520412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.351045+0000","last_change":"2024-02-23T00:28:37.106379+0000","last_active":"2024-02-23T00:29:15.351045+0000","last_peered":"2024-02-23T00:29:15.351045+0000","last_clean":"2024-02-23T00:29:15.351045+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:29:15.351045+0000","last_undegraded":"2024-02-23T00:29:15.351045+0000","last_fullsized":"2024-02-23T00:29:15.351045+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:37.106309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:37.106309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":81,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136535+0000","last_change":"2024-02-23T00:28:57.039721+0000","last_active":"2024-02-23T00:29:15.136535+0000","last_peered":"2024-02-23T00:29:15.136535+0000","last_clean":"2024-02-23T00:29:15.136535+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:29:15.136535+0000","last_undegraded":"2024-02-23T00:29:15.136535+0000","last_fullsized":"2024-02-23T00:29:15.136535+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.039660+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277191+0000","last_change":"2024-02-23T00:28:43.440882+0000","last_active":"2024-02-23T00:29:15.277191+0000","last_peered":"2024-02-23T00:29:15.277191+0000","last_clean":"2024-02-23T00:29:15.277191+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:29:15.277191+0000","last_undegraded":"2024-02-23T00:29:15.277191+0000","last_fullsized":"2024-02-23T00:29:15.277191+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.440788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.440788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350476+0000","last_change":"2024-02-23T00:28:56.022911+0000","last_active":"2024-02-23T00:29:15.350476+0000","last_peered":"2024-02-23T00:29:15.350476+0000","last_clean":"2024-02-23T00:29:15.350476+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:29:15.350476+0000","last_undegraded":"2024-02-23T00:29:15.350476+0000","last_fullsized":"2024-02-23T00:29:15.350476+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:56.022835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:56.022835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261549+0000","last_change":"2024-02-23T00:28:24.002963+0000","last_active":"2024-02-23T00:29:15.261549+0000","last_peered":"2024-02-23T00:29:15.261549+0000","last_clean":"2024-02-23T00:29:15.261549+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:29:15.261549+0000","last_undegraded":"2024-02-23T00:29:15.261549+0000","last_fullsized":"2024-02-23T00:29:15.261549+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:24.002878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:24.002878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.135861+0000","last_change":"2024-02-23T00:28:20.075559+0000","last_active":"2024-02-23T00:29:15.135861+0000","last_peered":"2024-02-23T00:29:15.135861+0000","last_clean":"2024-02-23T00:29:15.135861+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:29:15.135861+0000","last_undegraded":"2024-02-23T00:29:15.135861+0000","last_fullsized":"2024-02-23T00:29:15.135861+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:28:20.075487+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136125+0000","last_change":"2024-02-23T00:28:39.076715+0000","last_active":"2024-02-23T00:29:15.136125+0000","last_peered":"2024-02-23T00:29:15.136125+0000","last_clean":"2024-02-23T00:29:15.136125+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:29:15.136125+0000","last_undegraded":"2024-02-23T00:29:15.136125+0000","last_fullsized":"2024-02-23T00:29:15.136125+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.076631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.076631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167113+0000","last_change":"2024-02-23T00:28:53.665149+0000","last_active":"2024-02-23T00:29:15.167113+0000","last_peered":"2024-02-23T00:29:15.167113+0000","last_clean":"2024-02-23T00:29:15.167113+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:29:15.167113+0000","last_undegraded":"2024-02-23T00:29:15.167113+0000","last_fullsized":"2024-02-23T00:29:15.167113+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.665092+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167283+0000","last_change":"2024-02-23T00:29:02.711400+0000","last_active":"2024-02-23T00:29:15.167283+0000","last_peered":"2024-02-23T00:29:15.167283+0000","last_clean":"2024-02-23T00:29:15.167283+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:29:15.167283+0000","last_undegraded":"2024-02-23T00:29:15.167283+0000","last_fullsized":"2024-02-23T00:29:15.167283+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.711284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":80,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136158+0000","last_change":"2024-02-23T00:28:41.214195+0000","last_active":"2024-02-23T00:29:15.136158+0000","last_peered":"2024-02-23T00:29:15.136158+0000","last_clean":"2024-02-23T00:29:15.136158+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:29:15.136158+0000","last_undegraded":"2024-02-23T00:29:15.136158+0000","last_fullsized":"2024-02-23T00:29:15.136158+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:28:36.536962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.536962+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.349470+0000","last_change":"2024-02-23T00:28:49.187859+0000","last_active":"2024-02-23T00:29:15.349470+0000","last_peered":"2024-02-23T00:29:15.349470+0000","last_clean":"2024-02-23T00:29:15.349470+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:29:15.349470+0000","last_undegraded":"2024-02-23T00:29:15.349470+0000","last_fullsized":"2024-02-23T00:29:15.349470+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.187786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.187786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:32.122572+0000","last_change":"2024-02-23T00:29:32.122532+0000","last_active":"2024-02-23T00:29:32.122572+0000","last_peered":"2024-02-23T00:29:32.122572+0000","last_clean":"2024-02-23T00:29:32.122572+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:29:32.122572+0000","last_undegraded":"2024-02-23T00:29:32.122572+0000","last_fullsized":"2024-02-23T00:29:32.122572+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:32.122430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:32.122430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350714+0000","last_change":"2024-02-23T00:28:21.021722+0000","last_active":"2024-02-23T00:29:15.350714+0000","last_peered":"2024-02-23T00:29:15.350714+0000","last_clean":"2024-02-23T00:29:15.350714+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:29:15.350714+0000","last_undegraded":"2024-02-23T00:29:15.350714+0000","last_fullsized":"2024-02-23T00:29:15.350714+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:21.021619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.021619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167013+0000","last_change":"2024-02-23T00:28:36.676648+0000","last_active":"2024-02-23T00:29:15.167013+0000","last_peered":"2024-02-23T00:29:15.167013+0000","last_clean":"2024-02-23T00:29:15.167013+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:29:15.167013+0000","last_undegraded":"2024-02-23T00:29:15.167013+0000","last_fullsized":"2024-02-23T00:29:15.167013+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:28:36.676582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.676582+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136826+0000","last_change":"2024-02-23T00:28:43.560702+0000","last_active":"2024-02-23T00:29:15.136826+0000","last_peered":"2024-02-23T00:29:15.136826+0000","last_clean":"2024-02-23T00:29:15.136826+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:29:15.136826+0000","last_undegraded":"2024-02-23T00:29:15.136826+0000","last_fullsized":"2024-02-23T00:29:15.136826+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.560606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.560606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"82'9","reported_seq":81,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.212828+0000","last_change":"2024-02-23T00:29:38.098504+0000","last_active":"2024-02-23T00:29:40.212828+0000","last_peered":"2024-02-23T00:29:40.212828+0000","last_clean":"2024-02-23T00:29:40.212828+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:29:40.212828+0000","last_undegraded":"2024-02-23T00:29:40.212828+0000","last_fullsized":"2024-02-23T00:29:40.212828+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.098442+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.349884+0000","last_change":"2024-02-23T00:28:30.236265+0000","last_active":"2024-02-23T00:29:15.349884+0000","last_peered":"2024-02-23T00:29:15.349884+0000","last_clean":"2024-02-23T00:29:15.349884+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:29:15.349884+0000","last_undegraded":"2024-02-23T00:29:15.349884+0000","last_fullsized":"2024-02-23T00:29:15.349884+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:30.236188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:30.236188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"63'15","reported_seq":69,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261535+0000","last_change":"2024-02-23T00:28:55.907616+0000","last_active":"2024-02-23T00:29:15.261535+0000","last_peered":"2024-02-23T00:29:15.261535+0000","last_clean":"2024-02-23T00:29:15.261535+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:29:15.261535+0000","last_undegraded":"2024-02-23T00:29:15.261535+0000","last_fullsized":"2024-02-23T00:29:15.261535+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:28:55.907528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.907528+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350039+0000","last_change":"2024-02-23T00:29:14.086374+0000","last_active":"2024-02-23T00:29:15.350039+0000","last_peered":"2024-02-23T00:29:15.350039+0000","last_clean":"2024-02-23T00:29:15.350039+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:29:15.350039+0000","last_undegraded":"2024-02-23T00:29:15.350039+0000","last_fullsized":"2024-02-23T00:29:15.350039+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:14.086293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:14.086293+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":74,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.355898+0000","last_change":"2024-02-23T00:28:38.956507+0000","last_active":"2024-02-23T00:29:40.355898+0000","last_peered":"2024-02-23T00:29:40.355898+0000","last_clean":"2024-02-23T00:29:40.355898+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:29:40.355898+0000","last_undegraded":"2024-02-23T00:29:40.355898+0000","last_fullsized":"2024-02-23T00:29:40.355898+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_clean_scrub_stamp":"2024-02-23T00:28:38.956431+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261001+0000","last_change":"2024-02-23T00:28:49.012817+0000","last_active":"2024-02-23T00:29:15.261001+0000","last_peered":"2024-02-23T00:29:15.261001+0000","last_clean":"2024-02-23T00:29:15.261001+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:29:15.261001+0000","last_undegraded":"2024-02-23T00:29:15.261001+0000","last_fullsized":"2024-02-23T00:29:15.261001+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.012698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":85,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136169+0000","last_change":"2024-02-23T00:29:01.026660+0000","last_active":"2024-02-23T00:29:15.136169+0000","last_peered":"2024-02-23T00:29:15.136169+0000","last_clean":"2024-02-23T00:29:15.136169+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:29:15.136169+0000","last_undegraded":"2024-02-23T00:29:15.136169+0000","last_fullsized":"2024-02-23T00:29:15.136169+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:29:01.026575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.026575+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.276921+0000","last_change":"2024-02-23T00:28:58.383733+0000","last_active":"2024-02-23T00:29:15.276921+0000","last_peered":"2024-02-23T00:29:15.276921+0000","last_clean":"2024-02-23T00:29:15.276921+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:29:15.276921+0000","last_undegraded":"2024-02-23T00:29:15.276921+0000","last_fullsized":"2024-02-23T00:29:15.276921+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:28:58.383642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261529+0000","last_change":"2024-02-23T00:29:10.044046+0000","last_active":"2024-02-23T00:29:15.261529+0000","last_peered":"2024-02-23T00:29:15.261529+0000","last_clean":"2024-02-23T00:29:15.261529+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:29:15.261529+0000","last_undegraded":"2024-02-23T00:29:15.261529+0000","last_fullsized":"2024-02-23T00:29:15.261529+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.043966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.043966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350377+0000","last_change":"2024-02-23T00:28:34.066522+0000","last_active":"2024-02-23T00:29:15.350377+0000","last_peered":"2024-02-23T00:29:15.350377+0000","last_clean":"2024-02-23T00:29:15.350377+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:29:15.350377+0000","last_undegraded":"2024-02-23T00:29:15.350377+0000","last_fullsized":"2024-02-23T00:29:15.350377+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:34.066441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:34.066441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":83,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136580+0000","last_change":"2024-02-23T00:28:42.144931+0000","last_active":"2024-02-23T00:29:15.136580+0000","last_peered":"2024-02-23T00:29:15.136580+0000","last_clean":"2024-02-23T00:29:15.136580+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:29:15.136580+0000","last_undegraded":"2024-02-23T00:29:15.136580+0000","last_fullsized":"2024-02-23T00:29:15.136580+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:28:42.144833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:42.144833+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.137076+0000","last_change":"2024-02-23T00:28:53.629641+0000","last_active":"2024-02-23T00:29:15.137076+0000","last_peered":"2024-02-23T00:29:15.137076+0000","last_clean":"2024-02-23T00:29:15.137076+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:29:15.137076+0000","last_undegraded":"2024-02-23T00:29:15.137076+0000","last_fullsized":"2024-02-23T00:29:15.137076+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:53.629568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.629568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136648+0000","last_change":"2024-02-23T00:29:01.986532+0000","last_active":"2024-02-23T00:29:15.136648+0000","last_peered":"2024-02-23T00:29:15.136648+0000","last_clean":"2024-02-23T00:29:15.136648+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:29:15.136648+0000","last_undegraded":"2024-02-23T00:29:15.136648+0000","last_fullsized":"2024-02-23T00:29:15.136648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:01.986461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.986461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136833+0000","last_change":"2024-02-23T00:28:41.611517+0000","last_active":"2024-02-23T00:29:15.136833+0000","last_peered":"2024-02-23T00:29:15.136833+0000","last_clean":"2024-02-23T00:29:15.136833+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:29:15.136833+0000","last_undegraded":"2024-02-23T00:29:15.136833+0000","last_fullsized":"2024-02-23T00:29:15.136833+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.611448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.611448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":86,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350158+0000","last_change":"2024-02-23T00:29:01.234433+0000","last_active":"2024-02-23T00:29:15.350158+0000","last_peered":"2024-02-23T00:29:15.350158+0000","last_clean":"2024-02-23T00:29:15.350158+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:29:15.350158+0000","last_undegraded":"2024-02-23T00:29:15.350158+0000","last_fullsized":"2024-02-23T00:29:15.350158+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:29:01.234356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.234356+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.138282+0000","last_change":"2024-02-23T00:28:35.058858+0000","last_active":"2024-02-23T00:29:15.138282+0000","last_peered":"2024-02-23T00:29:15.138282+0000","last_clean":"2024-02-23T00:29:15.138282+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:29:15.138282+0000","last_undegraded":"2024-02-23T00:29:15.138282+0000","last_fullsized":"2024-02-23T00:29:15.138282+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:35.058773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:35.058773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136708+0000","last_change":"2024-02-23T00:28:51.684865+0000","last_active":"2024-02-23T00:29:15.136708+0000","last_peered":"2024-02-23T00:29:15.136708+0000","last_clean":"2024-02-23T00:29:15.136708+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:29:15.136708+0000","last_undegraded":"2024-02-23T00:29:15.136708+0000","last_fullsized":"2024-02-23T00:29:15.136708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:28:51.684789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.684789+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":119,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136126+0000","last_change":"2024-02-23T00:29:01.733374+0000","last_active":"2024-02-23T00:29:15.136126+0000","last_peered":"2024-02-23T00:29:15.136126+0000","last_clean":"2024-02-23T00:29:15.136126+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:29:15.136126+0000","last_undegraded":"2024-02-23T00:29:15.136126+0000","last_fullsized":"2024-02-23T00:29:15.136126+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:29:01.733287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.733287+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136794+0000","last_change":"2024-02-23T00:28:51.634965+0000","last_active":"2024-02-23T00:29:15.136794+0000","last_peered":"2024-02-23T00:29:15.136794+0000","last_clean":"2024-02-23T00:29:15.136794+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:29:15.136794+0000","last_undegraded":"2024-02-23T00:29:15.136794+0000","last_fullsized":"2024-02-23T00:29:15.136794+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:51.634901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.634901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.349495+0000","last_change":"2024-02-23T00:29:04.227928+0000","last_active":"2024-02-23T00:29:15.349495+0000","last_peered":"2024-02-23T00:29:15.349495+0000","last_clean":"2024-02-23T00:29:15.349495+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:29:15.349495+0000","last_undegraded":"2024-02-23T00:29:15.349495+0000","last_fullsized":"2024-02-23T00:29:15.349495+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.227864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.227864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167139+0000","last_change":"2024-02-23T00:28:51.664433+0000","last_active":"2024-02-23T00:29:15.167139+0000","last_peered":"2024-02-23T00:29:15.167139+0000","last_clean":"2024-02-23T00:29:15.167139+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:29:15.167139+0000","last_undegraded":"2024-02-23T00:29:15.167139+0000","last_fullsized":"2024-02-23T00:29:15.167139+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:28:51.664367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167086+0000","last_change":"2024-02-23T00:29:03.691092+0000","last_active":"2024-02-23T00:29:15.167086+0000","last_peered":"2024-02-23T00:29:15.167086+0000","last_clean":"2024-02-23T00:29:15.167086+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:29:15.167086+0000","last_undegraded":"2024-02-23T00:29:15.167086+0000","last_fullsized":"2024-02-23T00:29:15.167086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:29:03.691018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.691018+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":78,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.354721+0000","last_change":"2024-02-23T00:28:42.249575+0000","last_active":"2024-02-23T00:29:40.354721+0000","last_peered":"2024-02-23T00:29:40.354721+0000","last_clean":"2024-02-23T00:29:40.354721+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:29:40.354721+0000","last_undegraded":"2024-02-23T00:29:40.354721+0000","last_fullsized":"2024-02-23T00:29:40.354721+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:42.249510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:42.249510+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136734+0000","last_change":"2024-02-23T00:28:45.134966+0000","last_active":"2024-02-23T00:29:15.136734+0000","last_peered":"2024-02-23T00:29:15.136734+0000","last_clean":"2024-02-23T00:29:15.136734+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:29:15.136734+0000","last_undegraded":"2024-02-23T00:29:15.136734+0000","last_fullsized":"2024-02-23T00:29:15.136734+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:45.134840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:45.134840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":46,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261025+0000","last_change":"2024-02-23T00:28:50.000698+0000","last_active":"2024-02-23T00:29:15.261025+0000","last_peered":"2024-02-23T00:29:15.261025+0000","last_clean":"2024-02-23T00:29:15.261025+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:29:15.261025+0000","last_undegraded":"2024-02-23T00:29:15.261025+0000","last_fullsized":"2024-02-23T00:29:15.261025+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:28:50.000624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.000624+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":132,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.353911+0000","last_change":"2024-02-23T00:28:39.629165+0000","last_active":"2024-02-23T00:29:40.353911+0000","last_peered":"2024-02-23T00:29:40.353911+0000","last_clean":"2024-02-23T00:29:40.353911+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:29:40.353911+0000","last_undegraded":"2024-02-23T00:29:40.353911+0000","last_fullsized":"2024-02-23T00:29:40.353911+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:28:39.629075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.629075+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350193+0000","last_change":"2024-02-23T00:28:28.220174+0000","last_active":"2024-02-23T00:29:15.350193+0000","last_peered":"2024-02-23T00:29:15.350193+0000","last_clean":"2024-02-23T00:29:15.350193+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:29:15.350193+0000","last_undegraded":"2024-02-23T00:29:15.350193+0000","last_fullsized":"2024-02-23T00:29:15.350193+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:28.220080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.220080+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.276914+0000","last_change":"2024-02-23T00:28:39.496236+0000","last_active":"2024-02-23T00:29:15.276914+0000","last_peered":"2024-02-23T00:29:15.276914+0000","last_clean":"2024-02-23T00:29:15.276914+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:29:15.276914+0000","last_undegraded":"2024-02-23T00:29:15.276914+0000","last_fullsized":"2024-02-23T00:29:15.276914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.496160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.496160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":183,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.288885+0000","last_change":"2024-02-23T00:28:52.653896+0000","last_active":"2024-02-23T00:29:40.288885+0000","last_peered":"2024-02-23T00:29:40.288885+0000","last_clean":"2024-02-23T00:29:40.288885+0000","last_became_active":"2024-02-23T00:27:17.009670+0000","last_became_peered":"2024-02-23T00:27:17.009670+0000","last_unstale":"2024-02-23T00:29:40.288885+0000","last_undegraded":"2024-02-23T00:29:40.288885+0000","last_fullsized":"2024-02-23T00:29:40.288885+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.653823+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":137,"num_read_kb":132,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:39.779070+0000","last_change":"2024-02-23T00:29:39.779046+0000","last_active":"2024-02-23T00:29:39.779070+0000","last_peered":"2024-02-23T00:29:39.779070+0000","last_clean":"2024-02-23T00:29:39.779070+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:29:39.779070+0000","last_undegraded":"2024-02-23T00:29:39.779070+0000","last_fullsized":"2024-02-23T00:29:39.779070+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:39.778983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:39.778983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":82,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167398+0000","last_change":"2024-02-23T00:28:32.603154+0000","last_active":"2024-02-23T00:29:15.167398+0000","last_peered":"2024-02-23T00:29:15.167398+0000","last_clean":"2024-02-23T00:29:15.167398+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:29:15.167398+0000","last_undegraded":"2024-02-23T00:29:15.167398+0000","last_fullsized":"2024-02-23T00:29:15.167398+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:28:32.603088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:32.603088+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.137137+0000","last_change":"2024-02-23T00:28:50.672048+0000","last_active":"2024-02-23T00:29:15.137137+0000","last_peered":"2024-02-23T00:29:15.137137+0000","last_clean":"2024-02-23T00:29:15.137137+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:29:15.137137+0000","last_undegraded":"2024-02-23T00:29:15.137137+0000","last_fullsized":"2024-02-23T00:29:15.137137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.671988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.671988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261718+0000","last_change":"2024-02-23T00:28:40.999596+0000","last_active":"2024-02-23T00:29:15.261718+0000","last_peered":"2024-02-23T00:29:15.261718+0000","last_clean":"2024-02-23T00:29:15.261718+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:29:15.261718+0000","last_undegraded":"2024-02-23T00:29:15.261718+0000","last_fullsized":"2024-02-23T00:29:15.261718+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:40.999514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:40.999514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.135886+0000","last_change":"2024-02-23T00:28:17.047708+0000","last_active":"2024-02-23T00:29:15.135886+0000","last_peered":"2024-02-23T00:29:15.135886+0000","last_clean":"2024-02-23T00:29:15.135886+0000","last_became_active":"2024-02-23T00:27:17.012683+0000","last_became_peered":"2024-02-23T00:27:17.012683+0000","last_unstale":"2024-02-23T00:29:15.135886+0000","last_undegraded":"2024-02-23T00:29:15.135886+0000","last_fullsized":"2024-02-23T00:29:15.135886+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:17.047614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:17.047614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261103+0000","last_change":"2024-02-23T00:28:52.948687+0000","last_active":"2024-02-23T00:29:15.261103+0000","last_peered":"2024-02-23T00:29:15.261103+0000","last_clean":"2024-02-23T00:29:15.261103+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:29:15.261103+0000","last_undegraded":"2024-02-23T00:29:15.261103+0000","last_fullsized":"2024-02-23T00:29:15.261103+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:52.948601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:52.948601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:25.267882+0000","last_change":"2024-02-23T00:29:25.267851+0000","last_active":"2024-02-23T00:29:25.267882+0000","last_peered":"2024-02-23T00:29:25.267882+0000","last_clean":"2024-02-23T00:29:25.267882+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:29:25.267882+0000","last_undegraded":"2024-02-23T00:29:25.267882+0000","last_fullsized":"2024-02-23T00:29:25.267882+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:29:25.267773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":62,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350844+0000","last_change":"2024-02-23T00:28:33.090534+0000","last_active":"2024-02-23T00:29:15.350844+0000","last_peered":"2024-02-23T00:29:15.350844+0000","last_clean":"2024-02-23T00:29:15.350844+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:29:15.350844+0000","last_undegraded":"2024-02-23T00:29:15.350844+0000","last_fullsized":"2024-02-23T00:29:15.350844+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:28:33.090454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.090454+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277569+0000","last_change":"2024-02-23T00:28:50.344419+0000","last_active":"2024-02-23T00:29:15.277569+0000","last_peered":"2024-02-23T00:29:15.277569+0000","last_clean":"2024-02-23T00:29:15.277569+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:29:15.277569+0000","last_undegraded":"2024-02-23T00:29:15.277569+0000","last_fullsized":"2024-02-23T00:29:15.277569+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:50.344324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:50.344324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167005+0000","last_change":"2024-02-23T00:28:55.679584+0000","last_active":"2024-02-23T00:29:15.167005+0000","last_peered":"2024-02-23T00:29:15.167005+0000","last_clean":"2024-02-23T00:29:15.167005+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:29:15.167005+0000","last_undegraded":"2024-02-23T00:29:15.167005+0000","last_fullsized":"2024-02-23T00:29:15.167005+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:55.679522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.679522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.349795+0000","last_change":"2024-02-23T00:28:59.226967+0000","last_active":"2024-02-23T00:29:15.349795+0000","last_peered":"2024-02-23T00:29:15.349795+0000","last_clean":"2024-02-23T00:29:15.349795+0000","last_became_active":"2024-02-23T00:27:17.009826+0000","last_became_peered":"2024-02-23T00:27:17.009826+0000","last_unstale":"2024-02-23T00:29:15.349795+0000","last_undegraded":"2024-02-23T00:29:15.349795+0000","last_fullsized":"2024-02-23T00:29:15.349795+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:59.226890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:59.226890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277121+0000","last_change":"2024-02-23T00:29:02.340533+0000","last_active":"2024-02-23T00:29:15.277121+0000","last_peered":"2024-02-23T00:29:15.277121+0000","last_clean":"2024-02-23T00:29:15.277121+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:29:15.277121+0000","last_undegraded":"2024-02-23T00:29:15.277121+0000","last_fullsized":"2024-02-23T00:29:15.277121+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.340450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.340450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":58,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350103+0000","last_change":"2024-02-23T00:28:36.209181+0000","last_active":"2024-02-23T00:29:15.350103+0000","last_peered":"2024-02-23T00:29:15.350103+0000","last_clean":"2024-02-23T00:29:15.350103+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:29:15.350103+0000","last_undegraded":"2024-02-23T00:29:15.350103+0000","last_fullsized":"2024-02-23T00:29:15.350103+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:28:36.209111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:36.209111+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:39.108490+0000","last_change":"2024-02-23T00:29:39.108458+0000","last_active":"2024-02-23T00:29:39.108490+0000","last_peered":"2024-02-23T00:29:39.108490+0000","last_clean":"2024-02-23T00:29:39.108490+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:29:39.108490+0000","last_undegraded":"2024-02-23T00:29:39.108490+0000","last_fullsized":"2024-02-23T00:29:39.108490+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:39.108381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:29:39.108381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.166793+0000","last_change":"2024-02-23T00:29:11.580760+0000","last_active":"2024-02-23T00:29:15.166793+0000","last_peered":"2024-02-23T00:29:15.166793+0000","last_clean":"2024-02-23T00:29:15.166793+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:29:15.166793+0000","last_undegraded":"2024-02-23T00:29:15.166793+0000","last_fullsized":"2024-02-23T00:29:15.166793+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:11.580697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:11.580697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277846+0000","last_change":"2024-02-23T00:28:20.772584+0000","last_active":"2024-02-23T00:29:15.277846+0000","last_peered":"2024-02-23T00:29:15.277846+0000","last_clean":"2024-02-23T00:29:15.277846+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:29:15.277846+0000","last_undegraded":"2024-02-23T00:29:15.277846+0000","last_fullsized":"2024-02-23T00:29:15.277846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:20.772479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:20.772479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":91,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.349882+0000","last_change":"2024-02-23T00:28:21.157813+0000","last_active":"2024-02-23T00:29:15.349882+0000","last_peered":"2024-02-23T00:29:15.349882+0000","last_clean":"2024-02-23T00:29:15.349882+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:29:15.349882+0000","last_undegraded":"2024-02-23T00:29:15.349882+0000","last_fullsized":"2024-02-23T00:29:15.349882+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:28:21.157735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:21.157735+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350728+0000","last_change":"2024-02-23T00:28:43.992150+0000","last_active":"2024-02-23T00:29:15.350728+0000","last_peered":"2024-02-23T00:29:15.350728+0000","last_clean":"2024-02-23T00:29:15.350728+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:29:15.350728+0000","last_undegraded":"2024-02-23T00:29:15.350728+0000","last_fullsized":"2024-02-23T00:29:15.350728+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:43.992070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:43.992070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136242+0000","last_change":"2024-02-23T00:28:33.615049+0000","last_active":"2024-02-23T00:29:15.136242+0000","last_peered":"2024-02-23T00:29:15.136242+0000","last_clean":"2024-02-23T00:29:15.136242+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:29:15.136242+0000","last_undegraded":"2024-02-23T00:29:15.136242+0000","last_fullsized":"2024-02-23T00:29:15.136242+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:33.614962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:33.614962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167326+0000","last_change":"2024-02-23T00:28:41.617945+0000","last_active":"2024-02-23T00:29:15.167326+0000","last_peered":"2024-02-23T00:29:15.167326+0000","last_clean":"2024-02-23T00:29:15.167326+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:29:15.167326+0000","last_undegraded":"2024-02-23T00:29:15.167326+0000","last_fullsized":"2024-02-23T00:29:15.167326+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:41.617870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:41.617870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":102,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.135791+0000","last_change":"2024-02-23T00:29:04.010419+0000","last_active":"2024-02-23T00:29:15.135791+0000","last_peered":"2024-02-23T00:29:15.135791+0000","last_clean":"2024-02-23T00:29:15.135791+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:29:15.135791+0000","last_undegraded":"2024-02-23T00:29:15.135791+0000","last_fullsized":"2024-02-23T00:29:15.135791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:29:04.010348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.010348+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136145+0000","last_change":"2024-02-23T00:28:39.615097+0000","last_active":"2024-02-23T00:29:15.136145+0000","last_peered":"2024-02-23T00:29:15.136145+0000","last_clean":"2024-02-23T00:29:15.136145+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:29:15.136145+0000","last_undegraded":"2024-02-23T00:29:15.136145+0000","last_fullsized":"2024-02-23T00:29:15.136145+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:39.615002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:39.615002+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.276895+0000","last_change":"2024-02-23T00:28:31.646180+0000","last_active":"2024-02-23T00:29:15.276895+0000","last_peered":"2024-02-23T00:29:15.276895+0000","last_clean":"2024-02-23T00:29:15.276895+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:29:15.276895+0000","last_undegraded":"2024-02-23T00:29:15.276895+0000","last_fullsized":"2024-02-23T00:29:15.276895+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:31.646096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:31.646096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350994+0000","last_change":"2024-02-23T00:28:54.994367+0000","last_active":"2024-02-23T00:29:15.350994+0000","last_peered":"2024-02-23T00:29:15.350994+0000","last_clean":"2024-02-23T00:29:15.350994+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:29:15.350994+0000","last_undegraded":"2024-02-23T00:29:15.350994+0000","last_fullsized":"2024-02-23T00:29:15.350994+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:54.994276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.994276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":76,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.135853+0000","last_change":"2024-02-23T00:29:09.834851+0000","last_active":"2024-02-23T00:29:15.135853+0000","last_peered":"2024-02-23T00:29:15.135853+0000","last_clean":"2024-02-23T00:29:15.135853+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:29:15.135853+0000","last_undegraded":"2024-02-23T00:29:15.135853+0000","last_fullsized":"2024-02-23T00:29:15.135853+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:29:09.834741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:09.834741+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":75,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.211824+0000","last_change":"2024-02-23T00:29:02.969742+0000","last_active":"2024-02-23T00:29:40.211824+0000","last_peered":"2024-02-23T00:29:40.211824+0000","last_clean":"2024-02-23T00:29:40.211824+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:29:40.211824+0000","last_undegraded":"2024-02-23T00:29:40.211824+0000","last_fullsized":"2024-02-23T00:29:40.211824+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:29:02.969659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.969659+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136033+0000","last_change":"2024-02-23T00:29:07.960232+0000","last_active":"2024-02-23T00:29:15.136033+0000","last_peered":"2024-02-23T00:29:15.136033+0000","last_clean":"2024-02-23T00:29:15.136033+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:29:15.136033+0000","last_undegraded":"2024-02-23T00:29:15.136033+0000","last_fullsized":"2024-02-23T00:29:15.136033+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:07.960141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:07.960141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261641+0000","last_change":"2024-02-23T00:28:54.905338+0000","last_active":"2024-02-23T00:29:15.261641+0000","last_peered":"2024-02-23T00:29:15.261641+0000","last_clean":"2024-02-23T00:29:15.261641+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:29:15.261641+0000","last_undegraded":"2024-02-23T00:29:15.261641+0000","last_fullsized":"2024-02-23T00:29:15.261641+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:54.905258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.905258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277254+0000","last_change":"2024-02-23T00:28:34.610448+0000","last_active":"2024-02-23T00:29:15.277254+0000","last_peered":"2024-02-23T00:29:15.277254+0000","last_clean":"2024-02-23T00:29:15.277254+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:29:15.277254+0000","last_undegraded":"2024-02-23T00:29:15.277254+0000","last_fullsized":"2024-02-23T00:29:15.277254+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:28:34.610334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:34.610334+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"63'17","reported_seq":74,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136610+0000","last_change":"2024-02-23T00:28:44.586817+0000","last_active":"2024-02-23T00:29:15.136610+0000","last_peered":"2024-02-23T00:29:15.136610+0000","last_clean":"2024-02-23T00:29:15.136610+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:29:15.136610+0000","last_undegraded":"2024-02-23T00:29:15.136610+0000","last_fullsized":"2024-02-23T00:29:15.136610+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:28:44.586731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.586731+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.261342+0000","last_change":"2024-02-23T00:28:47.090951+0000","last_active":"2024-02-23T00:29:15.261342+0000","last_peered":"2024-02-23T00:29:15.261342+0000","last_clean":"2024-02-23T00:29:15.261342+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:29:15.261342+0000","last_undegraded":"2024-02-23T00:29:15.261342+0000","last_fullsized":"2024-02-23T00:29:15.261342+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:28:47.090884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.090884+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":75,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.215697+0000","last_change":"2024-02-23T00:28:31.557831+0000","last_active":"2024-02-23T00:29:40.215697+0000","last_peered":"2024-02-23T00:29:40.215697+0000","last_clean":"2024-02-23T00:29:40.215697+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:29:40.215697+0000","last_undegraded":"2024-02-23T00:29:40.215697+0000","last_fullsized":"2024-02-23T00:29:40.215697+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:31.557737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:31.557737+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.166889+0000","last_change":"2024-02-23T00:29:10.574049+0000","last_active":"2024-02-23T00:29:15.166889+0000","last_peered":"2024-02-23T00:29:15.166889+0000","last_clean":"2024-02-23T00:29:15.166889+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:29:15.166889+0000","last_undegraded":"2024-02-23T00:29:15.166889+0000","last_fullsized":"2024-02-23T00:29:15.166889+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.573983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.573983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"62'15","reported_seq":65,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167440+0000","last_change":"2024-02-23T00:28:46.622106+0000","last_active":"2024-02-23T00:29:15.167440+0000","last_peered":"2024-02-23T00:29:15.167440+0000","last_clean":"2024-02-23T00:29:15.167440+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:29:15.167440+0000","last_undegraded":"2024-02-23T00:29:15.167440+0000","last_fullsized":"2024-02-23T00:29:15.167440+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:28:46.621982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.621982+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136406+0000","last_change":"2024-02-23T00:28:40.630272+0000","last_active":"2024-02-23T00:29:15.136406+0000","last_peered":"2024-02-23T00:29:15.136406+0000","last_clean":"2024-02-23T00:29:15.136406+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:29:15.136406+0000","last_undegraded":"2024-02-23T00:29:15.136406+0000","last_fullsized":"2024-02-23T00:29:15.136406+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:40.630153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:40.630153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":75,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.214277+0000","last_change":"2024-02-23T00:28:44.617114+0000","last_active":"2024-02-23T00:29:40.214277+0000","last_peered":"2024-02-23T00:29:40.214277+0000","last_clean":"2024-02-23T00:29:40.214277+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:29:40.214277+0000","last_undegraded":"2024-02-23T00:29:40.214277+0000","last_fullsized":"2024-02-23T00:29:40.214277+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:44.617038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:44.617038+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167060+0000","last_change":"2024-02-23T00:29:00.703800+0000","last_active":"2024-02-23T00:29:15.167060+0000","last_peered":"2024-02-23T00:29:15.167060+0000","last_clean":"2024-02-23T00:29:15.167060+0000","last_became_active":"2024-02-23T00:28:41.203031+0000","last_became_peered":"2024-02-23T00:28:41.203031+0000","last_unstale":"2024-02-23T00:29:15.167060+0000","last_undegraded":"2024-02-23T00:29:15.167060+0000","last_fullsized":"2024-02-23T00:29:15.167060+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.703737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136062+0000","last_change":"2024-02-23T00:29:08.818324+0000","last_active":"2024-02-23T00:29:15.136062+0000","last_peered":"2024-02-23T00:29:15.136062+0000","last_clean":"2024-02-23T00:29:15.136062+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:29:15.136062+0000","last_undegraded":"2024-02-23T00:29:15.136062+0000","last_fullsized":"2024-02-23T00:29:15.136062+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:29:08.818235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.818235+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.137217+0000","last_change":"2024-02-23T00:28:47.677799+0000","last_active":"2024-02-23T00:29:15.137217+0000","last_peered":"2024-02-23T00:29:15.137217+0000","last_clean":"2024-02-23T00:29:15.137217+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:29:15.137217+0000","last_undegraded":"2024-02-23T00:29:15.137217+0000","last_fullsized":"2024-02-23T00:29:15.137217+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:47.677731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.677731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167028+0000","last_change":"2024-02-23T00:29:08.580168+0000","last_active":"2024-02-23T00:29:15.167028+0000","last_peered":"2024-02-23T00:29:15.167028+0000","last_clean":"2024-02-23T00:29:15.167028+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:29:15.167028+0000","last_undegraded":"2024-02-23T00:29:15.167028+0000","last_fullsized":"2024-02-23T00:29:15.167028+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.580102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.580102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":41,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277470+0000","last_change":"2024-02-23T00:29:05.279969+0000","last_active":"2024-02-23T00:29:15.277470+0000","last_peered":"2024-02-23T00:29:15.277470+0000","last_clean":"2024-02-23T00:29:15.277470+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:29:15.277470+0000","last_undegraded":"2024-02-23T00:29:15.277470+0000","last_fullsized":"2024-02-23T00:29:15.277470+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:05.279882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:05.279882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":68,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136630+0000","last_change":"2024-02-23T00:28:29.526599+0000","last_active":"2024-02-23T00:29:15.136630+0000","last_peered":"2024-02-23T00:29:15.136630+0000","last_clean":"2024-02-23T00:29:15.136630+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:29:15.136630+0000","last_undegraded":"2024-02-23T00:29:15.136630+0000","last_fullsized":"2024-02-23T00:29:15.136630+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:28:29.526505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:29.526505+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.277028+0000","last_change":"2024-02-23T00:28:49.331226+0000","last_active":"2024-02-23T00:29:15.277028+0000","last_peered":"2024-02-23T00:29:15.277028+0000","last_clean":"2024-02-23T00:29:15.277028+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:29:15.277028+0000","last_undegraded":"2024-02-23T00:29:15.277028+0000","last_fullsized":"2024-02-23T00:29:15.277028+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.331139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.331139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136983+0000","last_change":"2024-02-23T00:29:01.776038+0000","last_active":"2024-02-23T00:29:15.136983+0000","last_peered":"2024-02-23T00:29:15.136983+0000","last_clean":"2024-02-23T00:29:15.136983+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:29:15.136983+0000","last_undegraded":"2024-02-23T00:29:15.136983+0000","last_fullsized":"2024-02-23T00:29:15.136983+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:01.775943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.775943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:24.809360+0000","last_change":"2024-02-23T00:29:24.809313+0000","last_active":"2024-02-23T00:29:24.809360+0000","last_peered":"2024-02-23T00:29:24.809360+0000","last_clean":"2024-02-23T00:29:24.809360+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:29:24.809360+0000","last_undegraded":"2024-02-23T00:29:24.809360+0000","last_fullsized":"2024-02-23T00:29:24.809360+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:24.809192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:24.809192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":86,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350578+0000","last_change":"2024-02-23T00:28:28.095098+0000","last_active":"2024-02-23T00:29:15.350578+0000","last_peered":"2024-02-23T00:29:15.350578+0000","last_clean":"2024-02-23T00:29:15.350578+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:29:15.350578+0000","last_undegraded":"2024-02-23T00:29:15.350578+0000","last_fullsized":"2024-02-23T00:29:15.350578+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:28:28.095011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.095011+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.349564+0000","last_change":"2024-02-23T00:29:00.187868+0000","last_active":"2024-02-23T00:29:15.349564+0000","last_peered":"2024-02-23T00:29:15.349564+0000","last_clean":"2024-02-23T00:29:15.349564+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:29:15.349564+0000","last_undegraded":"2024-02-23T00:29:15.349564+0000","last_fullsized":"2024-02-23T00:29:15.349564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.187784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.187784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.136560+0000","last_change":"2024-02-23T00:29:02.769052+0000","last_active":"2024-02-23T00:29:15.136560+0000","last_peered":"2024-02-23T00:29:15.136560+0000","last_clean":"2024-02-23T00:29:15.136560+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:29:15.136560+0000","last_undegraded":"2024-02-23T00:29:15.136560+0000","last_fullsized":"2024-02-23T00:29:15.136560+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.768986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.768986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.167391+0000","last_change":"2024-02-23T00:28:26.541700+0000","last_active":"2024-02-23T00:29:15.167391+0000","last_peered":"2024-02-23T00:29:15.167391+0000","last_clean":"2024-02-23T00:29:15.167391+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:29:15.167391+0000","last_undegraded":"2024-02-23T00:29:15.167391+0000","last_fullsized":"2024-02-23T00:29:15.167391+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:26.541633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:26.541633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":71,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.350001+0000","last_change":"2024-02-23T00:28:53.186087+0000","last_active":"2024-02-23T00:29:15.350001+0000","last_peered":"2024-02-23T00:29:15.350001+0000","last_clean":"2024-02-23T00:29:15.350001+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:29:15.350001+0000","last_undegraded":"2024-02-23T00:29:15.350001+0000","last_fullsized":"2024-02-23T00:29:15.350001+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:28:53.186017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:53.186017+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.260915+0000","last_change":"2024-02-23T00:29:08.063475+0000","last_active":"2024-02-23T00:29:15.260915+0000","last_peered":"2024-02-23T00:29:15.260915+0000","last_clean":"2024-02-23T00:29:15.260915+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:29:15.260915+0000","last_undegraded":"2024-02-23T00:29:15.260915+0000","last_fullsized":"2024-02-23T00:29:15.260915+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.063398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.063398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:27.875242+0000","last_change":"2024-02-23T00:29:27.875203+0000","last_active":"2024-02-23T00:29:27.875242+0000","last_peered":"2024-02-23T00:29:27.875242+0000","last_clean":"2024-02-23T00:29:27.875242+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:29:27.875242+0000","last_undegraded":"2024-02-23T00:29:27.875242+0000","last_fullsized":"2024-02-23T00:29:27.875242+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:27.875106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:27.875106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":102,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-23T00:29:15.260902+0000","last_change":"2024-02-23T00:28:46.118981+0000","last_active":"2024-02-23T00:29:15.260902+0000","last_peered":"2024-02-23T00:29:15.260902+0000","last_clean":"2024-02-23T00:29:15.260902+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:29:15.260902+0000","last_undegraded":"2024-02-23T00:29:15.260902+0000","last_fullsized":"2024-02-23T00:29:15.260902+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:28:46.118918+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'8","reported_seq":74,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-23T00:29:40.356447+0000","last_change":"2024-02-23T00:28:28.676907+0000","last_active":"2024-02-23T00:29:40.356447+0000","last_peered":"2024-02-23T00:29:40.356447+0000","last_clean":"2024-02-23T00:29:40.356447+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:29:40.356447+0000","last_undegraded":"2024-02-23T00:29:40.356447+0000","last_fullsized":"2024-02-23T00:29:40.356447+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:28:28.676837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:28.676837+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":616,"ondisk_log_size":616,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":137,"num_read_kb":132,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332128,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":290664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.502,"15min":0.501},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68300000000000005,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.57599999999999996}]},{"osd":1,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64300000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52300000000000002,"15min":0.52500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.61799999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.69999999999999996}]},{"osd":4,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.41699999999999998,"15min":0.41099999999999998},"min":{"1min":0.28599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.626}]},{"osd":5,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42799999999999999,"15min":0.42699999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64100000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.436,"15min":0.42799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.63400000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.73599999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.47699999999999998,"15min":0.47099999999999997},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.46000000000000002,"15min":0.45400000000000001},"min":{"1min":0.33900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.69299999999999995,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.74299999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561061,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":289807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48799999999999999,"15min":0.49399999999999999},"min":{"1min":0.28599999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.51000000000000001,"15min":0.51700000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69399999999999995,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.42399999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.57799999999999996,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49099999999999999,"15min":0.495},"min":{"1min":0.31,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70299999999999996,"5min":0.754,"15min":0.754},"last":0.44500000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.49299999999999999,"15min":0.495},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.52900000000000003,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.55100000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.49399999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48499999999999999,"15min":0.48699999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51600000000000001,"15min":0.52100000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66800000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40899999999999997,"15min":0.40600000000000003},"min":{"1min":0.29099999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61499999999999999,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.42399999999999999,"15min":0.42599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61699999999999999,"5min":0.996,"15min":0.996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.436,"5min":0.42199999999999999,"15min":0.41999999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63400000000000001,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.57399999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.48299999999999998,"15min":0.47599999999999998},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.65500000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297760,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447392,"statfs":{"total":95995035648,"available":95690129408,"internally_reserved":0,"allocated":679936,"data_stored":298081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.34999999999999998,"15min":0.32800000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.372,"15min":0.35099999999999998},"min":{"1min":0.215,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.54600000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.37,"15min":0.371},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.23999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.39600000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:28:42 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.41399999999999998,"15min":0.42599999999999999},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.49299999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.438,"15min":0.435},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.38}]},{"osd":4,"last update":"Fri Feb 23 00:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.72399999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.625},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.83299999999999996}]},{"osd":5,"last update":"Fri Feb 23 00:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.57499999999999996},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79100000000000004,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.78600000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.71599999999999997,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Feb 23 00:29:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.78500000000000003,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60499999999999998,"15min":0.60899999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.752,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.82499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":290324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34399999999999997,"15min":0.32800000000000001},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.501,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35599999999999998,"15min":0.33300000000000002},"min":{"1min":0.23699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.49099999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.45700000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.38500000000000001,"15min":0.39100000000000001},"min":{"1min":0.27100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.46899999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36599999999999999,"15min":0.36099999999999999},"min":{"1min":0.253,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.47299999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:28:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.43099999999999999,"15min":0.44700000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56200000000000006,"5min":0.629,"15min":0.629},"last":0.505},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.313,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.53100000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.59999999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54400000000000004,"15min":0.55400000000000005},"min":{"1min":0.39100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55100000000000005,"15min":0.55900000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:29:26 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76300000000000001,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.432,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.56899999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41299999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.40500000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.54700000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411384,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":630784,"data_stored":290227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.33700000000000002,"15min":0.32900000000000001},"min":{"1min":0.23799999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.36099999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.34300000000000003,"15min":0.32900000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.42699999999999999,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.35599999999999998,"15min":0.34000000000000002},"min":{"1min":0.245,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.47599999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35099999999999998,"15min":0.33900000000000002},"min":{"1min":0.25900000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.44500000000000001,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.33200000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:28:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40100000000000002,"15min":0.41199999999999998},"min":{"1min":0.26900000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.49299999999999999,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.42999999999999999,"15min":0.438},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.54400000000000004,"5min":0.63,"15min":0.63},"last":0.40500000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76000000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:29:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.55000000000000004,"15min":0.54500000000000004},"min":{"1min":0.35599999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.76500000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73899999999999999,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.51300000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:28:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58799999999999997,"15min":0.59299999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.77600000000000002,"5min":1.117,"15min":1.117},"last":0.627},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80600000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.58599999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":289924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.37},"min":{"1min":0.23400000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.374,"15min":0.39400000000000002},"min":{"1min":0.223,"5min":0.191,"15min":0.191},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.38600000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39200000000000002,"15min":0.42199999999999999},"min":{"1min":0.27900000000000003,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.49399999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40600000000000003,"15min":0.41199999999999998},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.56899999999999995,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.44700000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:29:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.42299999999999999,"15min":0.437},"min":{"1min":0.29699999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53900000000000003,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.40600000000000003,"15min":0.42399999999999999},"min":{"1min":0.27900000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56299999999999994,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.43099999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56799999999999995,"15min":0.57599999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77000000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56899999999999995,"15min":0.58299999999999996},"min":{"1min":0.30299999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.68799999999999994,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.622}]},{"osd":5,"last update":"Fri Feb 23 00:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.38100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.63100000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76600000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.69399999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:29:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.61199999999999999},"min":{"1min":0.498,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.76900000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986223,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":289907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.50700000000000001,"15min":0.51500000000000001},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.35699999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66600000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.621}]},{"osd":1,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.5,"15min":0.52000000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51000000000000001,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.65200000000000002,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67200000000000004,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.379,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.70299999999999996,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.65900000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:28:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51800000000000002,"15min":0.52300000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69099999999999995,"5min":0.755,"15min":0.755},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.50800000000000001,"15min":0.53000000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.53100000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:29:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.38300000000000001,"15min":0.36799999999999999},"min":{"1min":0.31900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.42099999999999999,"15min":0.40200000000000002},"min":{"1min":0.34200000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.47799999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.34599999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.55500000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.43099999999999999,"15min":0.42899999999999999},"min":{"1min":0.29399999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.56399999999999995,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.504}]},{"osd":7,"last update":"Fri Feb 23 00:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.46999999999999997,"15min":0.46500000000000002},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.39400000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.53600000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757290,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":289441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.505},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70599999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.34599999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47999999999999998,"15min":0.47799999999999998},"min":{"1min":0.34999999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67100000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.49399999999999999,"15min":0.495},"min":{"1min":0.39900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.70399999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58199999999999996}]},{"osd":2,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.503,"15min":0.501},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66100000000000003,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.495},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.36699999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.63400000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52100000000000002,"15min":0.52300000000000002},"min":{"1min":0.36799999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73299999999999998,"5min":0.753,"15min":0.753},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:29:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.38600000000000001,"15min":0.371},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39400000000000002,"15min":0.372},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.66400000000000003,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.44600000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:28:50 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.434,"15min":0.42399999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.442,"15min":0.434},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.51300000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:29:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48999999999999999,"15min":0.48599999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.69599999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.50600000000000001,"15min":0.505},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.66200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:29:42.546 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-23T00:29:42.547 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump 2024-02-23T00:29:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:41.219501+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-23T00:29:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:41.484309+0000 mgr.y (mgr.24437) 100 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:41.686829+0000 mgr.y (mgr.24437) 101 : audit [DBG] from='client.24659 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:29:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:41.752540+0000 osd.1 (osd.1) 49 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:41.752606+0000 osd.1 (osd.1) 50 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:42.814 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:43.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:42.224039+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-23T00:29:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:42.503702+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:29:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:42.507302+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:29:43.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:42.224039+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-23T00:29:43.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:42.503702+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:29:43.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:42.507302+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:29:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:42.224039+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-23T00:29:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:42.503702+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:29:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:42.507302+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:29:44.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:29:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:29:44.493 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:42.926683+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:29:44.494 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:42.929088+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:29:44.494 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:43.104597+0000 osd.3 (osd.3) 57 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:29:44.494 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:43.108152+0000 osd.3 (osd.3) 58 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:29:44.494 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:43.484674+0000 mgr.y (mgr.24437) 102 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:44.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:42.926683+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:29:44.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:42.929088+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:29:44.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:43.104597+0000 osd.3 (osd.3) 57 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:29:44.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:43.108152+0000 osd.3 (osd.3) 58 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:29:44.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:43.484674+0000 mgr.y (mgr.24437) 102 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:42.926683+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:29:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:42.929088+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:29:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:43.104597+0000 osd.3 (osd.3) 57 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:29:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:43.108152+0000 osd.3 (osd.3) 58 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:29:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:43.484674+0000 mgr.y (mgr.24437) 102 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:29:44.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.770+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2279684812 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0f7400 msgr2=0x7f99dc103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:44.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.770+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2279684812 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0f7400 0x7f99dc103a20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f99d4009be0 tx=0x7f99d4004d10).stop 2024-02-23T00:29:44.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2279684812 shutdown_connections 2024-02-23T00:29:44.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2279684812 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0f7400 0x7f99dc103a20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2279684812 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99dc0fcbb0 0x7f99dc0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2279684812 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99dc0fd510 0x7f99dc0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2279684812 >> 172.21.15.87:0/2279684812 conn(0x7f99dc0f2820 msgr2=0x7f99dc0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2279684812 shutdown_connections 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2279684812 wait complete. 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 librados: starting msgr at 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 librados: starting objecter 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 Processor -- start 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 -- start start 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 librados: setting wanted keys 2024-02-23T00:29:44.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.771+0000 7f99e0f7c700 1 librados: calling monclient init 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99dc0f7400 0x7f99dc10ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0fcbb0 0x7f99dc10d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99dc0fd510 0x7f99dc1070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc107590 con 0x7f99dc0fd510 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc107700 con 0x7f99dc0f7400 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc1078c0 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99d9d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0fcbb0 0x7f99dc10d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99d9d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0fcbb0 0x7f99dc10d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:59748/0 (socket says 172.21.15.87:59748) 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99d9d9b700 1 -- 172.21.15.87:0/2782943901 learned_addr learned my addr 172.21.15.87:0/2782943901 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99dad9d700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99dc0fd510 0x7f99dc1070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99da59c700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99dc0f7400 0x7f99dc10ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99d9d9b700 1 -- 172.21.15.87:0/2782943901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99dc0f7400 msgr2=0x7f99dc10ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99d9d9b700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99dc0f7400 0x7f99dc10ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99d9d9b700 1 -- 172.21.15.87:0/2782943901 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99dc0fd510 msgr2=0x7f99dc1070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99d9d9b700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99dc0fd510 0x7f99dc1070a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99d9d9b700 1 -- 172.21.15.87:0/2782943901 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc005040 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99da59c700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99dc0f7400 0x7f99dc10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99dad9d700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99dc0fd510 0x7f99dc1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99d9d9b700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0fcbb0 0x7f99dc10d3a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f99d00041d0 tx=0x7f99d0009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99cb7fe700 10 client.?.objecter ms_handle_connect 0x7f99dc0fcbb0 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99cb7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99cb7fe700 1 -- 172.21.15.87:0/2782943901 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000ee90 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99cb7fe700 1 -- 172.21.15.87:0/2782943901 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99d00095a0 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d40097c0 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99cb7fe700 1 -- 172.21.15.87:0/2782943901 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d000dbb0 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 10 client.24665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99dc107ea0 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.772+0000 7f99e0f7c700 1 librados: init done 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.773+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc05fa60 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99cb7fe700 1 -- 172.21.15.87:0/2782943901 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f99d00090d0 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99cb7fe700 1 --2- 172.21.15.87:0/2782943901 >> v2:172.21.15.87:6800/1231597463 conn(0x7f99c4062c90 0x7f99c4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99cb7fe700 1 -- 172.21.15.87:0/2782943901 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(85..85 src has 28..85) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f99d007c620 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99cb7fe700 10 client.24665.objecter ms_dispatch 0x7f99dc001550 osd_map(85..85 src has 28..85) v4 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99cb7fe700 3 client.24665.objecter handle_osd_map got epochs [85,85] > 0 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99cb7fe700 3 client.24665.objecter handle_osd_map decoding full epoch 85 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99da59c700 1 --2- 172.21.15.87:0/2782943901 >> v2:172.21.15.87:6800/1231597463 conn(0x7f99c4062c90 0x7f99c4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99cb7fe700 20 client.24665.objecter dump_active .. 0 homeless 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99cb7fe700 10 client.24665.objecter ms_handle_connect 0x7f99c4062c90 2024-02-23T00:29:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.774+0000 7f99da59c700 1 --2- 172.21.15.87:0/2782943901 >> v2:172.21.15.87:6800/1231597463 conn(0x7f99c4062c90 0x7f99c4065150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f99cc007fd0 tx=0x7f99cc009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:44.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.778+0000 7f99cb7fe700 1 -- 172.21.15.87:0/2782943901 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f99d0047f60 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.945+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f99dc106120 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.945+0000 7f99cb7fe700 1 -- 172.21.15.87:0/2782943901 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v85) v1 ==== 56+0+4011 (secure 0 0 0) 0x7f99d0016070 con 0x7f99dc0fcbb0 2024-02-23T00:29:44.947 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 85 2024-02-23T00:29:44.947 INFO:teuthology.orchestra.run.smithi087.stdout:fsid 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:29:44.947 INFO:teuthology.orchestra.run.smithi087.stdout:created 2024-02-23T00:21:28.861318+0000 2024-02-23T00:29:44.947 INFO:teuthology.orchestra.run.smithi087.stdout:modified 2024-02-23T00:29:42.216298+0000 2024-02-23T00:29:44.947 INFO:teuthology.orchestra.run.smithi087.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:crush_version 18 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:full_ratio 0.95 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:backfillfull_ratio 0.9 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:nearfull_ratio 0.85 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:require_min_compat_client luminous 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:min_compat_client luminous 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:require_osd_release pacific 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:stretch_mode_enabled false 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 56 flags hashpspool stripe_width 0 application rgw 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:pool 3 'iscsi' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 57 flags hashpspool stripe_width 0 application rbd 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 58 flags hashpspool stripe_width 0 application rgw 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 60 flags hashpspool stripe_width 0 application rgw 2024-02-23T00:29:44.948 INFO:teuthology.orchestra.run.smithi087.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:max_osd 8 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0 up in weight 1 up_from 8 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6801/2153191863 v2:172.21.15.87:6802/2153191863 exists,up e3549bad-2891-47b9-bf77-132cadc20bf7 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1 up in weight 1 up_from 13 up_thru 77 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6805/685354171 v2:172.21.15.87:6806/685354171 exists,up 85525ce8-38d5-4840-8057-547cb176c859 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2 up in weight 1 up_from 18 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6809/3605394228 v2:172.21.15.87:6810/3605394228 exists,up 48ea64b6-ed15-48d2-b0ed-aa4053938aaa 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3 up in weight 1 up_from 26 up_thru 77 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6813/128059790 v2:172.21.15.87:6814/128059790 exists,up cb8705aa-e114-4dfd-ac9d-ec12f85f877a 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4 up in weight 1 up_from 31 up_thru 77 down_at 0 last_clean_interval [0,0) v2:172.21.15.171:6800/4175844757 v2:172.21.15.171:6801/4175844757 exists,up c3734a04-7343-4251-9b5b-bedb9e8b882c 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5 up in weight 1 up_from 38 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.171:6804/2550013795 v2:172.21.15.171:6805/2550013795 exists,up fe86232d-96b9-4cbf-9cac-373286fa103d 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6 up in weight 1 up_from 44 up_thru 58 down_at 0 last_clean_interval [0,0) v2:172.21.15.171:6808/2509408498 v2:172.21.15.171:6809/2509408498 exists,up bfc5dd69-5cfb-409f-985f-83ee0dd4dc01 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7 up in weight 1 up_from 51 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.171:6812/373787000 v2:172.21.15.171:6813/373787000 exists,up 29ddcf97-369a-4a2c-b3e2-142e84ccf939 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 2.a [7,2] 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 4.9 [1,2] 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 4.13 [1,2] 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:pg_upmap_items 4.17 [1,5] 2024-02-23T00:29:44.949 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:6800/1121018082 expires 2024-02-24T00:27:39.443000+0000 2024-02-23T00:29:44.950 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:6800/2138166229 expires 2024-02-24T00:22:22.047081+0000 2024-02-23T00:29:44.950 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/27794119 expires 2024-02-24T00:27:39.443000+0000 2024-02-23T00:29:44.950 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/1675126043 expires 2024-02-24T00:27:39.443000+0000 2024-02-23T00:29:44.950 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/3767016525 expires 2024-02-24T00:22:22.047081+0000 2024-02-23T00:29:44.950 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/969918521 expires 2024-02-24T00:22:22.047081+0000 2024-02-23T00:29:44.950 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/795095363 expires 2024-02-24T00:21:58.144574+0000 2024-02-23T00:29:44.950 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/2273151327 expires 2024-02-24T00:27:39.443000+0000 2024-02-23T00:29:44.950 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:6800/1685406414 expires 2024-02-24T00:21:58.144574+0000 2024-02-23T00:29:44.950 INFO:teuthology.orchestra.run.smithi087.stdout:blocklist 172.21.15.87:0/2093733106 expires 2024-02-24T00:21:58.144574+0000 2024-02-23T00:29:44.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 10 librados: watch_flush enter 2024-02-23T00:29:44.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 10 librados: watch_flush exit 2024-02-23T00:29:44.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 20 client.24665.objecter shutdown clearing up homeless session... 2024-02-23T00:29:44.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 10 client.24665.objecter successfully canceled tick 2024-02-23T00:29:44.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 >> v2:172.21.15.87:6800/1231597463 conn(0x7f99c4062c90 msgr2=0x7f99c4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2782943901 >> v2:172.21.15.87:6800/1231597463 conn(0x7f99c4062c90 0x7f99c4065150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f99cc007fd0 tx=0x7f99cc009cb0).stop 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0fcbb0 msgr2=0x7f99dc10d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0fcbb0 0x7f99dc10d3a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f99d00041d0 tx=0x7f99d0009f60).stop 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 shutdown_connections 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2782943901 >> v2:172.21.15.87:6800/1231597463 conn(0x7f99c4062c90 0x7f99c4065150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f99dc0fd510 0x7f99dc1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f99dc0f7400 0x7f99dc10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 --2- 172.21.15.87:0/2782943901 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f99dc0fcbb0 0x7f99dc10d3a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 >> 172.21.15.87:0/2782943901 conn(0x7f99dc0f2820 msgr2=0x7f99dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 shutdown_connections 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 -- 172.21.15.87:0/2782943901 wait complete. 2024-02-23T00:29:44.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:44.947+0000 7f99e0f7c700 1 librados: shutdown 2024-02-23T00:29:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:44.526696+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:29:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:44.530160+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:29:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:44.946652+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? 172.21.15.87:0/2782943901' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-23T00:29:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:44.526696+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:29:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:44.530160+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:29:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:44.946652+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? 172.21.15.87:0/2782943901' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-23T00:29:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:44.526696+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:29:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:44.530160+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:29:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:44.946652+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? 172.21.15.87:0/2782943901' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-23T00:29:45.832 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6801/2153191863 v2:172.21.15.87:6802/2153191863 exists,up e3549bad-2891-47b9-bf77-132cadc20bf7', 'osd.1 up in weight 1 up_from 13 up_thru 77 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6805/685354171 v2:172.21.15.87:6806/685354171 exists,up 85525ce8-38d5-4840-8057-547cb176c859', 'osd.2 up in weight 1 up_from 18 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6809/3605394228 v2:172.21.15.87:6810/3605394228 exists,up 48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'osd.3 up in weight 1 up_from 26 up_thru 77 down_at 0 last_clean_interval [0,0) v2:172.21.15.87:6813/128059790 v2:172.21.15.87:6814/128059790 exists,up cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'osd.4 up in weight 1 up_from 31 up_thru 77 down_at 0 last_clean_interval [0,0) v2:172.21.15.171:6800/4175844757 v2:172.21.15.171:6801/4175844757 exists,up c3734a04-7343-4251-9b5b-bedb9e8b882c', 'osd.5 up in weight 1 up_from 38 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.171:6804/2550013795 v2:172.21.15.171:6805/2550013795 exists,up fe86232d-96b9-4cbf-9cac-373286fa103d', 'osd.6 up in weight 1 up_from 44 up_thru 58 down_at 0 last_clean_interval [0,0) v2:172.21.15.171:6808/2509408498 v2:172.21.15.171:6809/2509408498 exists,up bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'osd.7 up in weight 1 up_from 51 up_thru 60 down_at 0 last_clean_interval [0,0) v2:172.21.15.171:6812/373787000 v2:172.21.15.171:6813/373787000 exists,up 29ddcf97-369a-4a2c-b3e2-142e84ccf939'] 2024-02-23T00:29:45.833 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon mon.a config show 2024-02-23T00:29:46.099 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:45.485620+0000 mgr.y (mgr.24437) 103 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-23T00:29:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:46.044755+0000 osd.4 (osd.4) 49 : cluster [DBG] 5.e scrub starts 2024-02-23T00:29:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:46.048258+0000 osd.4 (osd.4) 50 : cluster [DBG] 5.e scrub ok 2024-02-23T00:29:46.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:45.485620+0000 mgr.y (mgr.24437) 103 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-23T00:29:46.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:46.044755+0000 osd.4 (osd.4) 49 : cluster [DBG] 5.e scrub starts 2024-02-23T00:29:46.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:46.048258+0000 osd.4 (osd.4) 50 : cluster [DBG] 5.e scrub ok 2024-02-23T00:29:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:45.485620+0000 mgr.y (mgr.24437) 103 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-23T00:29:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:46.044755+0000 osd.4 (osd.4) 49 : cluster [DBG] 5.e scrub starts 2024-02-23T00:29:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:46.048258+0000 osd.4 (osd.4) 50 : cluster [DBG] 5.e scrub ok 2024-02-23T00:29:47.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:46.504087+0000 osd.7 (osd.7) 45 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:29:47.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:46.507008+0000 osd.7 (osd.7) 46 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:29:47.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:47.032358+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:29:47.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:47.033295+0000 osd.4 (osd.4) 52 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:29:47.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:46.504087+0000 osd.7 (osd.7) 45 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:29:47.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:46.507008+0000 osd.7 (osd.7) 46 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:29:47.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:47.032358+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:29:47.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:47.033295+0000 osd.4 (osd.4) 52 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:29:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:46.504087+0000 osd.7 (osd.7) 45 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:29:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:46.507008+0000 osd.7 (osd.7) 46 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:29:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:47.032358+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:29:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:47.033295+0000 osd.4 (osd.4) 52 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:29:48.073 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:29:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "name": "mon.a", 2024-02-23T00:29:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster": "ceph", 2024-02-23T00:29:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-23T00:29:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "admin_socket_mode": "", 2024-02-23T00:29:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-02-23T00:29:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "auth_client_required": "cephx, none", 2024-02-23T00:29:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "auth_cluster_required": "cephx", 2024-02-23T00:29:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "auth_debug": "true", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "auth_service_required": "cephx", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "auth_supported": "", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_aio": "true", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_aio_poll_ms": "250", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_aio_reap_max": "16", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_async_discard": "false", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_block_size": "4096", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_debug_aio": "false", 2024-02-23T00:29:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_debug_inflight_ios": "false", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_enable_discard": "false", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_flock_retry": "3", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_inject_crash": "0", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_ioring": "false", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_ioring_hipri": "false", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bdev_type": "", 2024-02-23T00:29:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_alloc_size": "1048576", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_allocator": "hybrid", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_buffered_io": "true", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_check_for_zeros": "false", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_compact_log_sync": "false", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_max_log_runway": "4194304", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_max_prefetch": "1048576", 2024-02-23T00:29:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_min_flush_size": "524288", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_min_log_runway": "1048576", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_replay_recovery": "false", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluefs_sync_write": "false", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_allocator": "hybrid", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-23T00:29:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_create": "true", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_db_create": "false", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_db_path": "", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_db_size": "0", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_path": "", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_preallocate_file": "false", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_size": "107374182400", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_wal_create": "false", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_wal_path": "", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_block_wal_size": "100663296", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_bluefs": "true", 2024-02-23T00:29:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_autotune": "true", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_size": "0", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_cache_type": "2q", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_clone_cow": "true", 2024-02-23T00:29:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_compression_mode": "none", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_csum_type": "crc32c", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_freelist": "false", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_inject_bug21040": "false", 2024-02-23T00:29:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_small_allocations": "0", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-23T00:29:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_default_buffered_read": "true", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_default_buffered_write": "false", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-23T00:29:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_on_mount": "false", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_on_umount": "false", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_ignore_data_csum": "false", 2024-02-23T00:29:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_log_op_age": "5.000000", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_max_alloc_size": "0", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_max_blob_size": "0", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_max_deferred_txc": "32", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_min_alloc_size": "0", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-23T00:29:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_nid_prealloc": "1024", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_retry_disk_reads": "3", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_rocksdb_cf": "true", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-23T00:29:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_spdk_mem": "512", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_tracing": "false", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-23T00:29:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-23T00:29:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "cephx_cluster_require_signatures": "false", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "cephx_cluster_require_version": "2", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "cephx_require_signatures": "false", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "cephx_require_version": "2", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "cephx_service_require_signatures": "false", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "cephx_service_require_version": "2", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "cephx_sign_messages": "true", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "chdir": "", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "client_acl_type": "", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "client_asio_thread_count": "2", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "client_cache_mid": "0.750000", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "client_cache_size": "16384", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "client_caps_release_delay": "5", 2024-02-23T00:29:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "client_check_pool_perm": "true", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_debug_force_sync_read": "false", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_debug_getattr_caps": "false", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_debug_inject_tick_delay": "0", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_die_on_failed_remount": "false", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_dirsize_rbytes": "true", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_force_lazyio": "false", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_fs": "", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_inject_release_failure": "false", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_max_inline_size": "4096", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-23T00:29:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "client_mds_namespace": "", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_metadata": "", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_mount_gid": "-1", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_mount_timeout": "300", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_mount_uid": "-1", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_mountpoint": "/", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_notify_timeout": "10", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_oc": "true", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_oc_max_dirty": "104857600", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_oc_max_objects": "1000", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_oc_size": "209715200", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_oc_target_dirty": "8388608", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_permissions": "true", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_quota": "true", 2024-02-23T00:29:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "client_quota_df": "true", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_readahead_max_bytes": "0", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_readahead_max_periods": "4", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_readahead_min": "131072", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_reconnect_stale": "false", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_shutdown_timeout": "30", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_snapdir": ".snap", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_tick_interval": "1", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_trace": "", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_try_dentry_invalidate": "false", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_use_faked_inos": "false", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "client_use_random_mds": "false", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "clog_to_graylog": "false", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-23T00:29:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "clog_to_graylog_port": "12201", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "clog_to_monitors": "default=true", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "clog_to_syslog": "false", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "clog_to_syslog_level": "info", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "-", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_network": "", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_network_interface": "", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "compressor_zlib_isal": "false", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "compressor_zlib_level": "5", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "compressor_zlib_winsize": "-15", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "compressor_zstd_level": "1", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:44277403522cf37edeee63b04643fd6c6e9416f13c3731a771e3cc8e53b7d010", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-23T00:29:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "crimson_osd_obc_lru_size": "10", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_location": "", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_location_hook": "", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_location_hook_timeout": "10", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "daemonize": "false", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_allow_any_pool_priority": "false", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_asok": "1/5", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_asok_assert_abort": "false", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_asserts_on_shutdown": "true", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_auth": "1/5", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_bdev": "1/3", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_bluefs": "1/5", 2024-02-23T00:29:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_bluestore": "1/5", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_buffer": "0/1", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_cephfs_mirror": "0/5", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_cephsqlite": "0/5", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_civetweb": "1/10", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_client": "0/5", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_compressor": "1/5", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_context": "0/1", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_crush": "1/1", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_crypto": "1/5", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_deliberately_leak_memory": "false", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_disable_randomized_ping": "false", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_dpdk": "1/5", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_eventtrace": "1/5", 2024-02-23T00:29:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_filer": "0/1", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_filestore": "1/3", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_finisher": "1/1", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_fuse": "1/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_heartbeat_testing_span": "0", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_heartbeatmap": "1/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_javaclient": "1/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_journal": "1/3", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_journaler": "0/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_kstore": "1/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_leveldb": "4/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_lockdep": "0/1", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_mds": "1/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_mds_balancer": "1/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_mds_locker": "1/5", 2024-02-23T00:29:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_mds_log": "1/5", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_mds_log_expire": "1/5", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_mds_migrator": "1/5", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_memdb": "4/5", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_mgr": "2/5", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_mgrc": "20/20", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_mon": "20/20", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_monc": "0/10", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_ms": "1/1", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_none": "0/5", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_objclass": "0/5", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_objectcacher": "0/5", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_objecter": "0/1", 2024-02-23T00:29:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_optracker": "0/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_osd": "1/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_paxos": "20/20", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_perfcounter": "1/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_prioritycache": "1/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_rados": "0/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_rbd": "0/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_rbd_mirror": "0/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_rbd_pwl": "0/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_rbd_replay": "0/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_refs": "0/0", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_reserver": "1/1", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_rgw": "1/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_rgw_sync": "1/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_rocksdb": "4/5", 2024-02-23T00:29:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_striper": "0/1", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_test": "0/5", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_throttle": "1/1", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_timer": "0/1", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "debug_tp": "0/5", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "device_failure_prediction_mode": "none", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "err_to_graylog": "false", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "err_to_stderr": "true", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "err_to_syslog": "false", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "event_tracing": "false", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "fake_statfs_for_testing": "0", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "fatal_signal_handlers": "true", 2024-02-23T00:29:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "filer_max_purge_ops": "10", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filer_max_truncate_ops": "128", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_apply_finisher_threads": "1", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_blackhole": "false", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_btrfs_clone_range": "true", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_btrfs_snap": "true", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_caller_concurrency": "10", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_collect_device_partition_information": "true", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_commit_timeout": "600.000000", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_debug_inject_read_err": "false", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_debug_omap_check": "false", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_debug_verify_split": "false", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_dump_file": "", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-23T00:29:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_fadvise": "true", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_fail_eio": "true", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_fd_cache_shards": "16", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_fd_cache_size": "128", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_fiemap": "false", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_fiemap_threshold": "4096", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_inject_stall": "0", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_journal_parallel": "false", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_journal_trailing": "false", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_journal_writeahead": "false", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_kill_at": "0", 2024-02-23T00:29:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_inline_xattrs": "0", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_xattr_value_size": "0", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-23T00:29:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_merge_threshold": "-10", 2024-02-23T00:29:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-23T00:29:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_odsync_write": "false", 2024-02-23T00:29:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_omap_backend": "rocksdb", 2024-02-23T00:29:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_omap_backend_path": "", 2024-02-23T00:29:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-23T00:29:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-23T00:29:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_op_thread_timeout": "60", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_op_threads": "2", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_punch_hole": "false", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-23T00:29:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_queue_max_ops": "50", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_seek_data_hole": "false", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_sloppy_crc": "false", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_splice": "false", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_split_multiple": "2", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_split_rand_factor": "20", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_update_to": "1000", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-23T00:29:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_enable": "true", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_xfs_extsize": "false", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "filestore_zfs_snap": "false", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "fio_dir": "/tmp/fio", 2024-02-23T00:29:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_allow_other": "true", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_atomic_o_trunc": "true", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_big_writes": "true", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_debug": "false", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_default_permissions": "false", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_disable_pagecache": "false", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_max_write": "0", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_multithreaded": "true", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_require_active_mds": "true", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_set_user_groups": "true", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_splice_move": "true", 2024-02-23T00:29:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_splice_read": "true", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_splice_write": "true", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "fuse_use_invalidate_cb": "true", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "gss_target_name": "ceph", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_file": "", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_inject_failure": "0", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_interval": "5", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "host": "", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-23T00:29:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "inject_early_sigterm": "false", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_aio": "true", 2024-02-23T00:29:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_align_min_size": "65536", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_block_align": "true", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_block_size": "4096", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_dio": "true", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_discard": "false", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_force_aio": "false", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_ignore_corruption": "false", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_max_write_bytes": "10485760", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_max_write_entries": "100", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_replay_from": "0", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-23T00:29:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_write_header_frequency": "0", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "journal_zero_on_create": "false", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "journaler_prefetch_periods": "10", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "journaler_prezero_periods": "5", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "journaler_write_head_interval": "15", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "key": "", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "keyfile": "", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_backend": "rocksdb", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_default_stripe_size": "65536", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_fsck_on_mount": "false", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_max_bytes": "67108864", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_max_ops": "512", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_nid_prealloc": "1024", 2024-02-23T00:29:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_onode_map_size": "1024", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_sync_submit_transaction": "false", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "kstore_sync_transaction": "false", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_block_size": "65536", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_bloom_size": "0", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_cache_size": "536870912", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_compact_on_mount": "false", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_compression": "false", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_log": "", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_log_to_ceph_log": "true", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_max_open_files": "0", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_paranoid": "false", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "leveldb_write_buffer_size": "33554432", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "librados_thread_count": "2", 2024-02-23T00:29:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "lockdep": "false", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "lockdep_force_backtrace": "false", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_coarse_timestamps": "true", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_flush_on_exit": "false", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_graylog_host": "127.0.0.1", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_graylog_port": "12201", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_max_new": "1000", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_max_recent": "10000", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_stderr_prefix": "debug ", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_stop_at_utilization": "0.970000", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_file": "true", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_graylog": "false", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_stderr": "false", 2024-02-23T00:29:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "log_to_syslog": "false", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "max_rotating_auth_attempts": "10", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_action_on_write_error": "1", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_alternate_name_max": "8192", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_asio_thread_count": "2", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_export_pin": "true", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_fragment_dirs": "true", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_fragment_interval": "5", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_interval": "10", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_max": "-1", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_max_until": "-1", 2024-02-23T00:29:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_merge_size": "50", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_midchunk": "0.300000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_min_start": "0.200000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_minchunk": "0.001000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_mode": "0", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_need_max": "1.200000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_need_min": "0.800000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_split_bits": "3", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_split_size": "10000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_target_decay": "10.000000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-23T00:29:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_beacon_grace": "15.000000", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_beacon_interval": "4.000000", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_cache_mid": "0.700000", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_cache_release_free_interval": "10", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_cache_reservation": "0.050000", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_cache_trim_interval": "1", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_cache_trim_threshold": "262144", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_client_prealloc_inos": "1000", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-23T00:29:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_connect_bootstrapping": "false", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_damage_table_max_entries": "10000", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_debug_auth_pins": "false", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_debug_frag": "false", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_debug_scatterstat": "false", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_debug_subtrees": "false", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_decay_halflife": "5.000000", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_default_dir_hash": "2", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_defer_session_stale": "true", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_deny_all_reconnect": "false", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_dir_keys_per_op": "16384", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_dir_max_commit_size": "10", 2024-02-23T00:29:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_dir_max_entries": "0", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_dump_cache_on_map": "false", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_early_reply": "true", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_enable_op_tracker": "true", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_enforce_unique_name": "true", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_export_ephemeral_random": "true", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-23T00:29:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_health_summarize_threshold": "10", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_inject_health_dummy": "false", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_inject_migrator_session_race": "false", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_join_fs": "", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_journal_format": "1", 2024-02-23T00:29:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_create_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_export_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_import_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_journal_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_journal_expire_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_journal_replay_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_link_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_mdstable_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_openc_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_rename_at": "0", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_log_events_per_segment": "1024", 2024-02-23T00:29:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_log_max_events": "-1", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_log_max_segments": "128", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_log_pause": "false", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_log_segment_size": "0", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_log_warn_factor": "2.000000", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_caps_per_client": "1048576", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_completed_flushes": "100000", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_completed_requests": "100000", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_export_size": "20971520", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_file_recover": "32", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_purge_files": "64", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_purge_ops": "8192", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-23T00:29:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_snaps_per_dir": "100", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_max_xattr_pairs_size": "65536", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_metrics_update_interval": "2", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_min_caps_per_client": "100", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_min_caps_working_set": "10000", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_numa_node": "-1", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_op_complaint_time": "30.000000", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_op_history_duration": "600", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_op_history_size": "20", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_op_log_threshold": "5", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_ping_grace": "15", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_ping_interval": "5", 2024-02-23T00:29:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_recall_max_caps": "30000", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_recall_warning_threshold": "262144", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_replay_interval": "1.000000", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_root_ino_gid": "0", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_root_ino_uid": "0", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-23T00:29:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_shutdown_check": "0", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_skip_ino": "0", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_snap_max_uid": "4294967294", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_snap_min_uid": "0", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_snap_rstat": "false", 2024-02-23T00:29:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_standby_replay_damaged": "false", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_thrash_exports": "0", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_thrash_fragments": "0", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_tick_interval": "5.000000", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_valgrind_exit": "false", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_verify_backtrace": "1", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_verify_scatter": "false", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mds_wipe_sessions": "false", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "mempool_debug": "false", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "memstore_device_bytes": "1073741824", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "memstore_page_set": "false", 2024-02-23T00:29:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "memstore_page_size": "65536", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_client_bytes": "134217728", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_client_messages": "512", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_disabled_modules": "", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_max_pg_num_change": "128", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_mds_bytes": "134217728", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_mds_messages": "128", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_mon_bytes": "134217728", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_mon_messages": "128", 2024-02-23T00:29:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_osd_bytes": "536870912", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_osd_messages": "8192", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_standby_modules": "true", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_stats_period": "5", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_stats_threshold": "5", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_tick_period": "2", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_allow_pool_delete": "true", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_allow_pool_size_one": "false", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_auth_validate_all_caps": "true", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_bytes": "104857600", 2024-02-23T00:29:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_directed_command_retry": "2", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_hunt_parallel": "3", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_log_interval": "1.000000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_ping_interval": "10.000000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_file_level": "debug", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_to_file": "true", 2024-02-23T00:29:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_to_stderr": "true", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_compact_on_bootstrap": "false", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_compact_on_start": "false", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_compact_on_trim": "true", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_cpu_threads": "4", 2024-02-23T00:29:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_crush_min_required_version": "hammer", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_daemon_bytes": "419430400", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_data_avail_crit": "5", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_data_avail_warn": "5", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_data_size_warn": "16106127360", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_dump_json": "false", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_dump_transactions": "false", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_extra_checks": "false", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_no_require_octopus": "false", 2024-02-23T00:29:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_no_require_pacific": "false", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_down_mkfs_grace": "60", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_election_default_strategy": "1", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_election_timeout": "5.000000", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_elector_ping_divisor": "2", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_enable_op_tracker": "true", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_fake_pool_delete": "false", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_force_quorum_join": "false", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_globalid_prealloc": "10000", 2024-02-23T00:29:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_health_detail_to_clog": "true", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_health_log_update_period": "5", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_health_max_detail": "50", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_health_to_clog": "true", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_health_to_clog_interval": "600", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_host": "[v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0]", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_host_override": "", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_initial_members": "", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-23T00:29:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_lease": "5.000000", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_log_max_summary": "50", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_max_log_epochs": "500", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_max_osd": "10000", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_max_pg_per_osd": "10000", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_max_pool_pg_num": "65536", 2024-02-23T00:29:48.132 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_mds_force_trim_to": "0", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_mds_skip_sanity": "false", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_memory_autotune": "true", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_memory_target": "2147483648", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_mgr_beacon_grace": "30", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_mgr_digest_period": "5", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_mgr_inactive_grace": "60", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_min_osdmap_epochs": "50", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_op_complaint_time": "30", 2024-02-23T00:29:48.133 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_op_history_duration": "600", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_op_history_size": "20", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_op_history_slow_op_size": "20", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_op_log_threshold": "5", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_auto_mark_in": "false", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_cache_size": "500", 2024-02-23T00:29:48.134 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_down_out_interval": "600", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_force_trim_to": "0", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-23T00:29:48.135 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_min_down_reporters": "2", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_report_timeout": "900", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-23T00:29:48.136 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_pg_stuck_threshold": "60", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_probe_timeout": "2.000000", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_reweight_max_change": "0.050000", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_reweight_max_osds": "4", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-23T00:29:48.137 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_scrub_interval": "86400", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_scrub_max_keys": "100", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_scrub_timeout": "300", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_session_timeout": "300", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_smart_report_timeout": "5", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_stat_smooth_intervals": "6", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_stretch_pool_min_size": "2", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_stretch_pool_size": "4", 2024-02-23T00:29:48.138 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_sync_debug": "false", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_sync_provider_kill_at": "0", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_sync_requester_kill_at": "0", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_sync_timeout": "60.000000", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_target_pg_per_osd": "100", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_tick_interval": "5", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_timecheck_interval": "300.000000", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_older_version_delay": "604800", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-23T00:29:48.139 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_misplaced": "false", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_older_version": "true", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_pool_no_app": "false", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-23T00:29:48.140 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "monmap": "", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_op_threads": "3", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_cm": "false", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_device_name": "", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_dscp": "96", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_local_gid": "", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_port_num": "1", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-23T00:29:48.141 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_sl": "3", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_support_srq": "true", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_rdma_type": "ib", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_async_reap_threshold": "5", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_before_connect": "false", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_ipv4": "true", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_ipv6": "false", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_msgr1": "false", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_msgr2": "true", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_port_max": "7300", 2024-02-23T00:29:48.142 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_port_min": "6800", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_retry_count": "3", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_bind_retry_delay": "5", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_blackhole_client": "false", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_blackhole_mds": "false", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_blackhole_mgr": "false", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_blackhole_mon": "false", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_blackhole_osd": "false", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_client_mode": "crc secure", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_cluster_mode": "crc secure", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_cluster_type": "", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_connection_idle_timeout": "900", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_connection_ready_timeout": "10", 2024-02-23T00:29:48.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_crc_data": "true", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_crc_header": "true", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_die_on_bad_msg": "false", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_die_on_bug": "true", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_die_on_old_message": "true", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_die_on_skipped_message": "false", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_coremask": "0xF", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_hugepages": "", 2024-02-23T00:29:48.144 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_lro": "true", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_memory_channel": "4", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_pmd": "", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_port_id": "0", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_dump_on_send": "false", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_initial_backoff": "0.200000", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_inject_delay_max": "1.000000", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_inject_delay_type": "", 2024-02-23T00:29:48.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_inject_socket_failures": "0", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_learn_addr_from_peer": "true", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_max_accept_failures": "4", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_max_backoff": "15.000000", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_mon_client_mode": "secure crc", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_mon_service_mode": "secure crc", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_pq_min_cost": "65536", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_public_type": "", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_service_mode": "crc secure", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_tcp_listen_backlog": "512", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_tcp_nodelay": "true", 2024-02-23T00:29:48.146 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_tcp_rcvbuf": "0", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "ms_type": "async+posix", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "no_config_file": "false", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "objecter_completion_locks_per_session": "32", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "objecter_inflight_ops": "1024", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "objecter_tick_interval": "5.000000", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "objecter_timeout": "10.000000", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "objectstore_blackhole": "false", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "openssl_engine_opts": "", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_agent_delay_time": "5.000000", 2024-02-23T00:29:48.147 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_agent_hist_halflife": "1000", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_agent_max_low_ops": "2", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_agent_max_ops": "4", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_agent_slop": "0.020000", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_asio_thread_count": "2", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_async_recovery_min_cost": "100", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_backfill_scan_max": "512", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_backfill_scan_min": "64", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_backoff_on_degraded": "false", 2024-02-23T00:29:48.148 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_backoff_on_peering": "false", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_backoff_on_unfound": "true", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_beacon_report_interval": "300", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_bench_duration": "30", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_bench_max_block_size": "67108864", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_blkin_trace_all": "false", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_check_for_log_corruption": "false", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-23T00:29:48.149 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_class_update_on_start": "true", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_client_message_cap": "256", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_client_message_size_cap": "524288000", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_client_op_priority": "63", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_client_watch_timeout": "30", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_command_max_records": "256", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_command_thread_suicide_timeout": "900", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_command_thread_timeout": "600", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_compact_on_start": "false", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_crush_update_on_start": "true", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_crush_update_weight_set": "true", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-23T00:29:48.150 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_feed_pullee": "-1", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_misdirected_ops": "false", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_no_acting_change": "false", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_no_purge_strays": "false", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_op_order": "false", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-23T00:29:48.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_shutdown": "false", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_verify_snaps": "false", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_deep_scrub_keys": "1024", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_deep_scrub_stride": "524288", 2024-02-23T00:29:48.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_default_notify_timeout": "30", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_delete_sleep": "0.000000", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_discard_disconnected_ops": "true", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_enable_op_tracker": "true", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_fast_info": "true", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_fast_shutdown": "true", 2024-02-23T00:29:48.153 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_function_tracing": "false", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_heartbeat_grace": "20", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_heartbeat_interval": "6", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_heartbeat_min_peers": "10", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_heartbeat_min_size": "2000", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_heartbeat_stale": "600", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_hit_set_max_size": "100000", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_hit_set_min_size": "1000", 2024-02-23T00:29:48.154 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_journal_size": "5120", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_kill_backfill_at": "0", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_map_cache_size": "50", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_map_dedup": "true", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_map_message_max": "40", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_map_share_max_epochs": "40", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_attr_name_len": "100", 2024-02-23T00:29:48.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_attr_size": "0", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_backfills": "1", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_markdown_count": "5", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_markdown_period": "600", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_object_name_len": "2048", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_object_namespace_len": "256", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_object_size": "134217728", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_pg_blocked_by": "16", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_pg_log_entries": "10000", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_pgls": "1024", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_push_cost": "8388608", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_push_objects": "10", 2024-02-23T00:29:48.156 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_scrubs": "1", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_trimming_pgs": "2", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_write_op_reply_len": "64", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_max_write_size": "90", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_cost_per_byte_usec_hdd": "5.200000", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_cost_per_io_usec_hdd": "25000.000000", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-23T00:29:48.157 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_profile": "high_client_ops", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_client_lim": "999999", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_client_res": "1", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_memory_base": "805306368", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_memory_cache_min": "134217728", 2024-02-23T00:29:48.158 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_memory_target": "4294967296", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_memory_target_autotune": "false", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_min_pg_log_entries": "250", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_min_recovery_priority": "0", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mon_report_interval": "5", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_num_cache_shards": "32", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_num_op_tracker_shard": "32", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_numa_auto_affinity": "true", 2024-02-23T00:29:48.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_numa_node": "-1", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_numa_prefer_iface": "true", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_objecter_finishers": "1", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_objectstore": "bluestore", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_objectstore_fuse": "false", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_objectstore_tracing": "false", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_complaint_time": "30.000000", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_history_duration": "600", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_history_size": "20", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_history_slow_op_size": "20", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_log_threshold": "5", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_num_shards": "0", 2024-02-23T00:29:48.160 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_num_shards_hdd": "5", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_num_shards_ssd": "8", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_pq_min_cost": "65536", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_queue": "wpq", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_queue_cut_off": "high", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_op_thread_timeout": "15", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_open_classes_on_start": "true", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_os_flags": "0", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_peering_op_priority": "255", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_delete_cost": "1048576", 2024-02-23T00:29:48.161 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_delete_priority": "5", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_log_trim_max": "10000", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_log_trim_min": "100", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pg_stat_report_interval_max": "500", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-23T00:29:48.162 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_flags": "0", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_min_size": "0", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_pg_num": "32", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_pgp_num": "0", 2024-02-23T00:29:48.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_size": "3", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_default_type": "replicated", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_push_per_object_cost": "1000", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recover_clone_overlap": "true", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_cost": "20971520", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_max_active": "0", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-23T00:29:48.164 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_max_single_start": "1", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_op_priority": "3", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_priority": "5", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_sleep": "0.000000", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_repair_during_recovery": "false", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_requested_scrub_priority": "120", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-23T00:29:48.165 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_auto_repair": "false", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_begin_hour": "0", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_begin_week_day": "0", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_chunk_max": "25", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_chunk_min": "5", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_cost": "52428800", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_during_recovery": "false", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_end_hour": "0", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_end_week_day": "0", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_invalid_stats": "true", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-23T00:29:48.166 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_max_preemptions": "5", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_priority": "5", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_scrub_sleep": "0.000000", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_skip_data_digest": "false", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_smart_report_timeout": "5", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_snap_trim_cost": "1048576", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_snap_trim_priority": "5", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-23T00:29:48.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_target_transaction_size": "30", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_tracing": "false", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_use_stale_snap": "false", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "osdc_blkin_trace_all": "false", 2024-02-23T00:29:48.168 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_kill_at": "0", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_max_join_drift": "10", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_min": "500", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_min_wait": "0.050000", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_propose_interval": "1.000000", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_service_trim_max": "500", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_service_trim_min": "10", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_stash_full_interval": "25", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_trim_max": "500", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "paxos_trim_min": "250", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "perf": "true", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "pid_file": "", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-23T00:29:48.169 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "-", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrv": "", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "public_bind_addr": "-", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "public_network": "", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "public_network_interface": "", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "qat_compressor_enabled": "false", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "rados_mon_op_timeout": "0", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "rados_osd_op_timeout": "0", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "rados_tracing": "false", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_atime_update_interval": "60", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_balance_parent_reads": "false", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_balance_snap_reads": "false", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_blkin_trace_all": "false", 2024-02-23T00:29:48.170 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_cache": "true", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_cache_policy": "writearound", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_cache_size": "33554432", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_clone_copy_on_read": "false", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_compression_hint": "none", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_concurrent_management_ops": "10", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_clone_format": "auto", 2024-02-23T00:29:48.171 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_data_pool": "", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_features": "61", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_format": "2", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_map_options": "", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_order": "22", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_pool": "rbd", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_stripe_count": "0", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_default_stripe_unit": "0", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_enable_alloc_hint": "true", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-23T00:29:48.172 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_io_scheduler": "simple", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_order": "24", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_pool": "", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_journal_splay_width": "4", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_localize_parent_reads": "false", 2024-02-23T00:29:48.173 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_localize_snap_reads": "false", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-23T00:29:48.174 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-23T00:29:48.175 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_mtime_update_interval": "60", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_non_blocking_aio": "true", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_op_thread_timeout": "60", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_op_threads": "1", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_parent_cache_enabled": "false", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_plugins": "", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_bps_burst": "0", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-23T00:29:48.176 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_bps_limit": "0", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_iops_burst": "0", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_iops_limit": "0", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-23T00:29:48.177 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_read_from_replica_policy": "default", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_skip_partial_discard": "true", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_tracing": "false", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_validate_names": "true", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd_validate_pool": "true", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "restapi_base_url": "", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "restapi_log_level": "", 2024-02-23T00:29:48.178 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_acl_grants_max_num": "100", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_admin_entry": "admin", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_barbican_url": "", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_cache_enabled": "true", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_cache_expiry_interval": "900", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_cache_lru_size": "10000", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_content_length_compat": "false", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_copy_obj_progress": "true", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-23T00:29:48.179 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_cors_rules_max_num": "100", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_cross_domain_policy": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_kmip_addr": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_kmip_password": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_kmip_username": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_require_ssl": "true", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-23T00:29:48.180 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_addr": "", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_auth": "token", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_namespace": "", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_prefix": "", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_token_file": "", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_curl_buffersize": "524288", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_curl_low_speed_time": "300", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-23T00:29:48.181 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_data_log_changes_size": "1000", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_data_log_num_shards": "128", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_data_log_window": "30", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_data_notify_interval_msec": "200", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-23T00:29:48.182 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dns_name": "", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dns_s3website_name": "", 2024-02-23T00:29:48.183 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_dynamic_resharding": "true", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_enable_gc_threads": "true", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_enable_lc_threads": "true", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_enable_ops_log": "false", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_enable_quota_threads": "true", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_enable_static_website": "false", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_enable_usage_log": "false", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_enforce_swift_acls": "true", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_exit_timeout_secs": "120", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_expose_bucket": "false", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_extended_http_attrs": "", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_fcgi_socket_backlog": "1024", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-23T00:29:48.184 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_frontends": "beast port=7480", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_gc_max_deferred": "50", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_gc_max_objs": "32", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_gc_processor_period": "3600", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_host": "", 2024-02-23T00:29:48.185 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_init_timeout": "300", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_admin_domain": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_admin_password": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_admin_password_path": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_admin_project": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_admin_tenant": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_admin_token": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_admin_token_path": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_admin_user": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_api_version": "2", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_barbican_domain": "", 2024-02-23T00:29:48.186 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_barbican_password": "", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_barbican_project": "", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_barbican_user": "", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_url": "", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_lc_debug_interval": "-1", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_lc_lock_max_time": "90", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_lc_max_objs": "32", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_lc_max_rules": "1000", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_lc_max_worker": "3", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_lc_thread_delay": "0", 2024-02-23T00:29:48.187 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ldap_dnattr": "uid", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ldap_searchfilter": "", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_log_http_headers": "", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_log_object_name_utc": "false", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_attr_name_len": "0", 2024-02-23T00:29:48.188 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_attr_size": "0", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_chunk_size": "4194304", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_listing_results": "1000", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_notify_retries": "10", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_put_param_size": "1048576", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_put_size": "5368709120", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_max_slo_entries": "1000", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_md_log_max_shards": "64", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-23T00:29:48.189 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_mp_lock_max_time": "600", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_max_gc": "300", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-23T00:29:48.190 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_num_async_rados_threads": "32", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_num_control_oids": "8", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_num_rados_handles": "1", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_numa_node": "-1", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_objexp_chunk_size": "100", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_objexp_gc_interval": "600", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_op_thread_timeout": "600", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_op_tracing": "false", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_opa_token": "", 2024-02-23T00:29:48.191 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_opa_url": "", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_opa_verify_ssl": "true", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ops_log_file_path": "", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ops_log_rados": "true", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_ops_log_socket_path": "", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_period_push_interval": "2.000000", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_port": "", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_print_continue": "true", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-23T00:29:48.192 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_rados_tracing": "false", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_realm": "", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_region": "", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_request_uri": "", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_reshard_batch_size": "64", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-23T00:29:48.193 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_reshard_max_aio": "128", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_reshard_num_logs": "16", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_reshard_thread_interval": "600", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_resolve_cname": "false", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_run_sync_thread": "true", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_scheduler_type": "throttler", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_script_uri": "", 2024-02-23T00:29:48.194 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_service_provider_name": "", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_socket_path": "", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sts_client_id": "", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sts_client_secret": "", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sts_entry": "sts", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sts_key": "sts", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sts_min_session_duration": "900", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sts_token_introspection_url": "", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_account_in_url": "false", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_auth_entry": "auth", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_auth_url": "", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_custom_header": "", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-23T00:29:48.195 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_need_stats": "true", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_tenant_name": "", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_token_expiration": "86400", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_url": "", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_url_prefix": "swift", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_lease_period": "120", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-23T00:29:48.196 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_thread_pool_size": "512", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_torrent_comment": "", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_torrent_createby": "", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_torrent_encoding": "", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_torrent_flag": "false", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_torrent_origin": "", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_torrent_tracker": "", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_trust_forwarded_https": "false", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_usage_max_shards": "32", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_usage_max_user_shards": "1", 2024-02-23T00:29:48.197 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_use_opa_authz": "false", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_user_max_buckets": "1000", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_user_unique_email": "true", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_verify_ssl": "true", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_zone": "", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_zonegroup": "", 2024-02-23T00:29:48.198 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_block_size": "4096", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_cache_size": "536870912", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_cf_compact_on_deletion": "false", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-23T00:29:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_index_type": "binary_search", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_partition_filters": "false", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_perf": "false", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "run_dir": "/var/run/ceph", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "setgroup": "ceph", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "setuser": "ceph", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "setuser_match_path": "", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "thp": "false", 2024-02-23T00:29:48.200 INFO:teuthology.orchestra.run.smithi087.stdout: "threadpool_default_timeout": "60", 2024-02-23T00:29:48.201 INFO:teuthology.orchestra.run.smithi087.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-23T00:29:48.201 INFO:teuthology.orchestra.run.smithi087.stdout: "throttler_perf_counter": "true" 2024-02-23T00:29:48.201 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:29:48.472 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:47.043947+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:29:48.472 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:47.044794+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:29:48.472 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:47.486038+0000 mgr.y (mgr.24437) 104 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:29:48.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:47.043947+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:29:48.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:47.044794+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:29:48.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:47.486038+0000 mgr.y (mgr.24437) 104 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:29:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:47.043947+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:29:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:47.044794+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:29:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:47.486038+0000 mgr.y (mgr.24437) 104 : cluster [DBG] pgmap v86: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:29:48.816 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-23T00:29:49.084 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:47.779566+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:29:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:47.780456+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:29:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:48.455707+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.d scrub starts 2024-02-23T00:29:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:48.458710+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.d scrub ok 2024-02-23T00:29:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:48.483993+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:29:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:48.485579+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:29:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:48.659801+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:48.659849+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:47.779566+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:29:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:47.780456+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:29:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:48.455707+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.d scrub starts 2024-02-23T00:29:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:48.458710+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.d scrub ok 2024-02-23T00:29:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:48.483993+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:29:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:48.485579+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:29:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:48.659801+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:48.659849+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:47.779566+0000 osd.2 (osd.2) 41 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:29:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:47.780456+0000 osd.2 (osd.2) 42 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:29:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:48.455707+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.d scrub starts 2024-02-23T00:29:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:48.458710+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.d scrub ok 2024-02-23T00:29:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:48.483993+0000 osd.7 (osd.7) 47 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:29:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:48.485579+0000 osd.7 (osd.7) 48 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:29:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:48.659801+0000 osd.4 (osd.4) 53 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:29:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:48.659849+0000 osd.4 (osd.4) 54 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:29:50.569 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:49.053223+0000 osd.3 (osd.3) 61 : cluster [DBG] 2.19 deep-scrub starts 2024-02-23T00:29:50.569 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:49.056180+0000 osd.3 (osd.3) 62 : cluster [DBG] 2.19 deep-scrub ok 2024-02-23T00:29:50.569 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:49.259367+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-23T00:29:50.569 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:49.486431+0000 mgr.y (mgr.24437) 105 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:29:50.569 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:49.502194+0000 osd.7 (osd.7) 49 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:29:50.570 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:49.503087+0000 osd.7 (osd.7) 50 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:49.053223+0000 osd.3 (osd.3) 61 : cluster [DBG] 2.19 deep-scrub starts 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:49.056180+0000 osd.3 (osd.3) 62 : cluster [DBG] 2.19 deep-scrub ok 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:49.259367+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:49.486431+0000 mgr.y (mgr.24437) 105 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:49.502194+0000 osd.7 (osd.7) 49 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:49.503087+0000 osd.7 (osd.7) 50 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:49.053223+0000 osd.3 (osd.3) 61 : cluster [DBG] 2.19 deep-scrub starts 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:49.056180+0000 osd.3 (osd.3) 62 : cluster [DBG] 2.19 deep-scrub ok 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:49.259367+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:49.486431+0000 mgr.y (mgr.24437) 105 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:29:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:49.502194+0000 osd.7 (osd.7) 49 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:29:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:49.503087+0000 osd.7 (osd.7) 50 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:29:50.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:29:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:29:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.180+0000 7f28c52fd700 1 -- 172.21.15.87:0/562256967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c00f9270 msgr2=0x7f28c00f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.180+0000 7f28c52fd700 1 --2- 172.21.15.87:0/562256967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c00f9270 0x7f28c00f6c50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f28b00047b0 tx=0x7f28b0009cb0).stop 2024-02-23T00:29:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.180+0000 7f28c52fd700 1 -- 172.21.15.87:0/562256967 shutdown_connections 2024-02-23T00:29:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.180+0000 7f28c52fd700 1 --2- 172.21.15.87:0/562256967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00f7330 0x7f28c0103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.180+0000 7f28c52fd700 1 --2- 172.21.15.87:0/562256967 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00f89a0 0x7f28c00f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.180+0000 7f28c52fd700 1 --2- 172.21.15.87:0/562256967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c00f9270 0x7f28c00f6c50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.180+0000 7f28c52fd700 1 -- 172.21.15.87:0/562256967 >> 172.21.15.87:0/562256967 conn(0x7f28c00f26f0 msgr2=0x7f28c00f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.180+0000 7f28c52fd700 1 -- 172.21.15.87:0/562256967 shutdown_connections 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.180+0000 7f28c52fd700 1 -- 172.21.15.87:0/562256967 wait complete. 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 librados: starting msgr at 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 librados: starting objecter 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 Processor -- start 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 -- start start 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 librados: setting wanted keys 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 librados: calling monclient init 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c00f7330 0x7f28c01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00f89a0 0x7f28c01096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00f9270 0x7f28c0109ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c01143b0 con 0x7f28c00f89a0 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0114520 con 0x7f28c00f9270 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28c52fd700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c01146e0 con 0x7f28c00f7330 2024-02-23T00:29:51.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28beffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c00f7330 0x7f28c01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28be7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00f89a0 0x7f28c01096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28beffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c00f7330 0x7f28c01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:59768/0 (socket says 172.21.15.87:59768) 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28beffd700 1 -- 172.21.15.87:0/1897447782 learned_addr learned my addr 172.21.15.87:0/1897447782 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.181+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00f9270 0x7f28c0109ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28beffd700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00f9270 msgr2=0x7f28c0109ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28beffd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00f9270 0x7f28c0109ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28beffd700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00f89a0 msgr2=0x7f28c01096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28beffd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00f89a0 0x7f28c01096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28beffd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b80097c0 con 0x7f28c00f7330 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28be7fc700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00f89a0 0x7f28c01096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28beffd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c00f7330 0x7f28c01091d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f28a800c7c0 tx=0x7f28a8007ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28b7fff700 10 client.?.objecter ms_handle_connect 0x7f28c00f7330 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28b7fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8004430 con 0x7f28c00f7330 2024-02-23T00:29:51.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b0005040 con 0x7f28c00f7330 2024-02-23T00:29:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28a800e970 con 0x7f28c00f7330 2024-02-23T00:29:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a8013a70 con 0x7f28c00f7330 2024-02-23T00:29:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28c52fd700 10 client.24671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c0114d70 con 0x7f28c00f7330 2024-02-23T00:29:51.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.182+0000 7f28c52fd700 1 librados: init done 2024-02-23T00:29:51.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.183+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28c005ee80 con 0x7f28c00f7330 2024-02-23T00:29:51.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.183+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f28a800eae0 con 0x7f28c00f7330 2024-02-23T00:29:51.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28ac062c90 0x7f28ac065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28be7fc700 1 --2- 172.21.15.87:0/1897447782 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28ac062c90 0x7f28ac065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(86..86 src has 28..86) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f28a807c6b0 con 0x7f28c00f7330 2024-02-23T00:29:51.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 10 client.24671.objecter ms_dispatch 0x7f28c0001550 osd_map(86..86 src has 28..86) v4 2024-02-23T00:29:51.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 3 client.24671.objecter handle_osd_map got epochs [86,86] > 0 2024-02-23T00:29:51.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 3 client.24671.objecter handle_osd_map decoding full epoch 86 2024-02-23T00:29:51.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 20 client.24671.objecter dump_active .. 0 homeless 2024-02-23T00:29:51.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1259 (secure 0 0 0) 0x7f28a8083090 con 0x7f28c00f7330 2024-02-23T00:29:51.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28be7fc700 1 --2- 172.21.15.87:0/1897447782 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28ac062c90 0x7f28ac065150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f28c00617c0 tx=0x7f28b0007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:51.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 10 client.24671.objecter ms_handle_connect 0x7f28ac062c90 2024-02-23T00:29:51.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28c52fd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fd990 0x7f28c00fdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0114a20 con 0x7f28c00fd990 2024-02-23T00:29:51.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fd990 0x7f28c00fdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28bf7fe700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28c01060f0 con 0x7f28c00fd990 2024-02-23T00:29:51.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 10 client.24671.objecter ms_handle_connect 0x7f28c00fd990 2024-02-23T00:29:51.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28b7fff700 10 client.24671.objecter resend_mon_ops 2024-02-23T00:29:51.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.184+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fd990 0x7f28c00fdd70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f28b8001fb0 tx=0x7f28b8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:51.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.185+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8011da0 con 0x7f28c00fd990 2024-02-23T00:29:51.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.185+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.0 v2:172.21.15.87:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f28b800f030 con 0x7f28c00fd990 2024-02-23T00:29:51.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.185+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fd990 msgr2=0x7f28c00fdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.185+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fd990 0x7f28c00fdd70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f28b8001fb0 tx=0x7f28b8004d10).stop 2024-02-23T00:29:51.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.185+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fd990 msgr2=0x7f28c00fdd70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:29:51.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.185+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fd990 0x7f28c00fdd70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.192+0000 7f28c52fd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fe260 0x7f28c0118fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.192+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c00fe660 con 0x7f28c00fe260 2024-02-23T00:29:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.192+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fe260 0x7f28c0118fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.192+0000 7f28bf7fe700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f28c00fe7d0 con 0x7f28c00fe260 2024-02-23T00:29:51.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.192+0000 7f28b7fff700 10 client.24671.objecter ms_handle_connect 0x7f28c00fe260 2024-02-23T00:29:51.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.192+0000 7f28b7fff700 10 client.24671.objecter resend_mon_ops 2024-02-23T00:29:51.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.192+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fe260 0x7f28c0118fb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f28b8001fb0 tx=0x7f28b801a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:51.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.192+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8016b20 con 0x7f28c00fe260 2024-02-23T00:29:51.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.193+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.0 v2:172.21.15.87:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (secure 0 0 0) 0x7f28b8011b40 con 0x7f28c00fe260 2024-02-23T00:29:51.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.193+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fe260 msgr2=0x7f28c0118fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.193+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fe260 0x7f28c0118fb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f28b8001fb0 tx=0x7f28b801a000).stop 2024-02-23T00:29:51.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.193+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fe260 msgr2=0x7f28c0118fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:29:51.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.193+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28c00fe260 0x7f28c0118fb0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0x7f28b8001fb0 tx=0x7f28b801a000).stop 2024-02-23T00:29:51.197 INFO:teuthology.orchestra.run.smithi087.stdout:mon.a: {} 2024-02-23T00:29:51.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.194+0000 7f28c52fd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00273c0 0x7f28c00277d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.194+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c00fe970 con 0x7f28c00273c0 2024-02-23T00:29:51.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.194+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00273c0 0x7f28c00277d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.194+0000 7f28bf7fe700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28c0027cc0 con 0x7f28c00273c0 2024-02-23T00:29:51.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.194+0000 7f28b7fff700 10 client.24671.objecter ms_handle_connect 0x7f28c00273c0 2024-02-23T00:29:51.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.194+0000 7f28b7fff700 10 client.24671.objecter resend_mon_ops 2024-02-23T00:29:51.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.194+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00273c0 0x7f28c00277d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f28b8004d50 tx=0x7f28b8022000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:51.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.195+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8031030 con 0x7f28c00273c0 2024-02-23T00:29:51.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.195+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.1 v2:172.21.15.171:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f28b8016850 con 0x7f28c00273c0 2024-02-23T00:29:51.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.195+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00273c0 msgr2=0x7f28c00277d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.195+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00273c0 0x7f28c00277d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f28b8004d50 tx=0x7f28b8022000).stop 2024-02-23T00:29:51.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.195+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00273c0 msgr2=0x7f28c00277d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:29:51.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.195+0000 7f28bf7fe700 1 -- 172.21.15.87:0/1897447782 reap_dead start 2024-02-23T00:29:51.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.195+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c00273c0 0x7f28c00277d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.201+0000 7f28c52fd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c01096c0 0x7f28c0027e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.201+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0109aa0 con 0x7f28c01096c0 2024-02-23T00:29:51.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.202+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c01096c0 0x7f28c0027e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.202+0000 7f28bf7fe700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f28c0109df0 con 0x7f28c01096c0 2024-02-23T00:29:51.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.202+0000 7f28b7fff700 10 client.24671.objecter ms_handle_connect 0x7f28c01096c0 2024-02-23T00:29:51.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.202+0000 7f28b7fff700 10 client.24671.objecter resend_mon_ops 2024-02-23T00:29:51.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.202+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c01096c0 0x7f28c0027e90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f28c00f50d0 tx=0x7f28b802f070).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:51.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.202+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8022ea0 con 0x7f28c01096c0 2024-02-23T00:29:51.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.1 v2:172.21.15.171:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (secure 0 0 0) 0x7f28b8022570 con 0x7f28c01096c0 2024-02-23T00:29:51.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c01096c0 msgr2=0x7f28c0027e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c01096c0 0x7f28c0027e90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f28c00f50d0 tx=0x7f28b802f070).stop 2024-02-23T00:29:51.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c01096c0 msgr2=0x7f28c0027e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:29:51.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28c01096c0 0x7f28c0027e90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.206 INFO:teuthology.orchestra.run.smithi087.stdout:mon.b: {} 2024-02-23T00:29:51.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28c52fd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0028380 0x7f28c0112240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0028760 con 0x7f28c0028380 2024-02-23T00:29:51.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0028380 0x7f28c0112240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28bf7fe700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28c0113d10 con 0x7f28c0028380 2024-02-23T00:29:51.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28b7fff700 10 client.24671.objecter ms_handle_connect 0x7f28c0028380 2024-02-23T00:29:51.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0028380 0x7f28c0112240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f28b8004d50 tx=0x7f28b803a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:51.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28b7fff700 10 client.24671.objecter resend_mon_ops 2024-02-23T00:29:51.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.204+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8039b30 con 0x7f28c0028380 2024-02-23T00:29:51.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.205+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f28b8022c90 con 0x7f28c0028380 2024-02-23T00:29:51.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.205+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0028380 msgr2=0x7f28c0112240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.205+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0028380 0x7f28c0112240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f28b8004d50 tx=0x7f28b803a000).stop 2024-02-23T00:29:51.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.205+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0028380 msgr2=0x7f28c0112240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:29:51.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.205+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0028380 0x7f28c0112240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.211+0000 7f28c52fd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0112730 0x7f28c011f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.211+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0114060 con 0x7f28c0112730 2024-02-23T00:29:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.211+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0112730 0x7f28c011f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.211+0000 7f28bf7fe700 1 -- 172.21.15.87:0/1897447782 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7f28c011f9c0 con 0x7f28c0112730 2024-02-23T00:29:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.211+0000 7f28b7fff700 10 client.24671.objecter ms_handle_connect 0x7f28c0112730 2024-02-23T00:29:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.211+0000 7f28b7fff700 10 client.24671.objecter resend_mon_ops 2024-02-23T00:29:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.211+0000 7f28bf7fe700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0112730 0x7f28c011f4d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f28b8002010 tx=0x7f28b8043070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.212+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b8039ea0 con 0x7f28c0112730 2024-02-23T00:29:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 <== mon.2 v2:172.21.15.87:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (secure 0 0 0) 0x7f28b8039920 con 0x7f28c0112730 2024-02-23T00:29:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0112730 msgr2=0x7f28c011f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0112730 0x7f28c011f4d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f28b8002010 tx=0x7f28b8043070).stop 2024-02-23T00:29:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28b7fff700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0112730 msgr2=0x7f28c011f4d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:29:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28b7fff700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c0112730 0x7f28c011f4d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:51.215 INFO:teuthology.orchestra.run.smithi087.stdout:mon.c: {} 2024-02-23T00:29:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 10 librados: watch_flush enter 2024-02-23T00:29:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 10 librados: watch_flush exit 2024-02-23T00:29:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 20 client.24671.objecter shutdown clearing up homeless session... 2024-02-23T00:29:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 10 client.24671.objecter successfully canceled tick 2024-02-23T00:29:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28ac062c90 msgr2=0x7f28ac065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 1 --2- 172.21.15.87:0/1897447782 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28ac062c90 0x7f28ac065150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f28c00617c0 tx=0x7f28b0007f90).stop 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c00f7330 msgr2=0x7f28c01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 1 --2- 172.21.15.87:0/1897447782 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28c00f7330 0x7f28c01091d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f28a800c7c0 tx=0x7f28a8007ef0).stop 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28bf7fe700 1 -- 172.21.15.87:0/1897447782 reap_dead start 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 shutdown_connections 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 >> 172.21.15.87:0/1897447782 conn(0x7f28c00f26f0 msgr2=0x7f28c00f3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 shutdown_connections 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 1 -- 172.21.15.87:0/1897447782 wait complete. 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:51.213+0000 7f28c52fd700 1 librados: shutdown 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-23T00:29:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-23T00:29:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:50.049305+0000 osd.3 (osd.3) 63 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:29:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:50.052465+0000 osd.3 (osd.3) 64 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:29:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:50.163381+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:29:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:50.166822+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:29:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:50.567733+0000 mgr.y (mgr.24437) 106 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:51.185163+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? 172.21.15.87:0/1897447782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:50.049305+0000 osd.3 (osd.3) 63 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:29:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:50.052465+0000 osd.3 (osd.3) 64 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:29:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:50.163381+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:29:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:50.166822+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:29:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:50.567733+0000 mgr.y (mgr.24437) 106 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:51.185163+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? 172.21.15.87:0/1897447782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:50.049305+0000 osd.3 (osd.3) 63 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:29:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:50.052465+0000 osd.3 (osd.3) 64 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:29:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:50.163381+0000 osd.0 (osd.0) 43 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:29:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:50.166822+0000 osd.0 (osd.0) 44 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:29:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:50.567733+0000 mgr.y (mgr.24437) 106 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:29:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:51.185163+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? 172.21.15.87:0/1897447782' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:29:51.940 INFO:teuthology.run_tasks:Running task workunit... 2024-02-23T00:29:51.954 INFO:tasks.workunit:Pulling workunits from ref 4f758cd8e8854d1f126da86e254076852bf059a9 2024-02-23T00:29:51.954 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-23T00:29:51.955 DEBUG:teuthology.orchestra.run.smithi087:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-23T00:29:51.955 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-23T00:29:51.955 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:29:51.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:29:51.956 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-23T00:29:51.956 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-23T00:29:51.956 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-23T00:29:51.956 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-23T00:29:51.957 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-23T00:29:51.957 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:29:51.957 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-23T00:29:51.957 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:29:51.958 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-23T00:29:51.958 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-23T00:29:51.959 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-23T00:29:51.959 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:29:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:29:51.979 INFO:teuthology.orchestra.run.smithi087.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-23T00:29:51.979 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-23T00:29:51.998 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:52.028 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:29:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:52.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:29:52.038 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-23T00:29:52.039 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-23T00:29:52.099 INFO:tasks.workunit:timeout=3h 2024-02-23T00:29:52.099 INFO:tasks.workunit:cleanup=True 2024-02-23T00:29:52.099 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/kamoltat/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4f758cd8e8854d1f126da86e254076852bf059a9 2024-02-23T00:29:52.132 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:29:52.157 INFO:tasks.workunit.client.0.smithi087.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-23T00:29:52.159 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:52.178 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:29:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:52.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:29:52.226 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:29:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:52] "GET /metrics HTTP/1.1" 200 172167 "" "Prometheus/2.33.4" 2024-02-23T00:29:52.251 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:29:52.258 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:52.279 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:52.283 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:29:52.287 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:29:52.317 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:52.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:51.019862+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:29:52.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:51.021052+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:29:52.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:51.487025+0000 mgr.y (mgr.24437) 107 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:29:52.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:51.019862+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:29:52.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:51.021052+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:29:52.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:51.487025+0000 mgr.y (mgr.24437) 107 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:29:52.531 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:51.019862+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:29:52.531 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:51.021052+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:29:52.531 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:51.487025+0000 mgr.y (mgr.24437) 107 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:29:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:53.009959+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:29:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:53.011081+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:29:53.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:53.009959+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:29:53.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:53.011081+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:29:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:53.009959+0000 osd.5 (osd.5) 43 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:29:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:53.011081+0000 osd.5 (osd.5) 44 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:29:53.768 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:29:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:29:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:29:54.049 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:29:54.049 INFO:teuthology.orchestra.run.smithi171.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-23T00:29:54.049 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:29:54.404 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:53.080703+0000 osd.6 (osd.6) 37 : cluster [DBG] 4.a scrub starts 2024-02-23T00:29:54.404 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:53.084371+0000 osd.6 (osd.6) 38 : cluster [DBG] 4.a scrub ok 2024-02-23T00:29:54.404 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:53.272146+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:29:54.404 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:53.274150+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:29:54.404 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:53.487390+0000 mgr.y (mgr.24437) 108 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:54.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:53.080703+0000 osd.6 (osd.6) 37 : cluster [DBG] 4.a scrub starts 2024-02-23T00:29:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:53.084371+0000 osd.6 (osd.6) 38 : cluster [DBG] 4.a scrub ok 2024-02-23T00:29:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:53.272146+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:29:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:53.274150+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:29:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:53.487390+0000 mgr.y (mgr.24437) 108 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:53.080703+0000 osd.6 (osd.6) 37 : cluster [DBG] 4.a scrub starts 2024-02-23T00:29:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:53.084371+0000 osd.6 (osd.6) 38 : cluster [DBG] 4.a scrub ok 2024-02-23T00:29:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:53.272146+0000 osd.0 (osd.0) 45 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:29:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:53.274150+0000 osd.0 (osd.0) 46 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:29:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:53.487390+0000 mgr.y (mgr.24437) 108 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:54.594 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:29:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:54.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:29:54.699 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:29:54.727 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:55.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.292+0000 7f6c979cd700 1 -- 172.21.15.87:0/2841301882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c900fe530 msgr2=0x7f6c900f63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.292+0000 7f6c979cd700 1 --2- 172.21.15.87:0/2841301882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c900fe530 0x7f6c900f63b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f6c80004870 tx=0x7f6c80009cb0).stop 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.292+0000 7f6c979cd700 1 -- 172.21.15.87:0/2841301882 shutdown_connections 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.292+0000 7f6c979cd700 1 --2- 172.21.15.87:0/2841301882 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c900f6a90 0x7f6c90104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.292+0000 7f6c979cd700 1 --2- 172.21.15.87:0/2841301882 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c900fdbd0 0x7f6c900fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.292+0000 7f6c979cd700 1 --2- 172.21.15.87:0/2841301882 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c900fe530 0x7f6c900f63b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.292+0000 7f6c979cd700 1 -- 172.21.15.87:0/2841301882 >> 172.21.15.87:0/2841301882 conn(0x7f6c9006c370 msgr2=0x7f6c9006c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.292+0000 7f6c979cd700 1 -- 172.21.15.87:0/2841301882 shutdown_connections 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.292+0000 7f6c979cd700 1 -- 172.21.15.87:0/2841301882 wait complete. 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 librados: starting msgr at 2024-02-23T00:29:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 librados: starting objecter 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 Processor -- start 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 -- start start 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 librados: setting wanted keys 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 librados: calling monclient init 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c900f6a90 0x7f6c9010e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c900fdbd0 0x7f6c9010e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c900fe530 0x7f6c90108260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90108750 con 0x7f6c900fe530 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901088c0 con 0x7f6c900f6a90 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c979cd700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90108a80 con 0x7f6c900fdbd0 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c94f68700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c900fdbd0 0x7f6c9010e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:55.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c94f68700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c900fdbd0 0x7f6c9010e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:59984/0 (socket says 172.21.15.87:59984) 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c94f68700 1 -- 172.21.15.87:0/3678053283 learned_addr learned my addr 172.21.15.87:0/3678053283 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c95f6a700 1 --2- 172.21.15.87:0/3678053283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c900fe530 0x7f6c90108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.293+0000 7f6c95769700 1 --2- 172.21.15.87:0/3678053283 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c900f6a90 0x7f6c9010e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c95f6a700 1 -- 172.21.15.87:0/3678053283 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c900fdbd0 msgr2=0x7f6c9010e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c95f6a700 1 --2- 172.21.15.87:0/3678053283 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c900fdbd0 0x7f6c9010e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c95f6a700 1 -- 172.21.15.87:0/3678053283 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c900f6a90 msgr2=0x7f6c9010e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c95f6a700 1 --2- 172.21.15.87:0/3678053283 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c900f6a90 0x7f6c9010e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c95f6a700 1 -- 172.21.15.87:0/3678053283 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c880097c0 con 0x7f6c900fe530 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c95f6a700 1 --2- 172.21.15.87:0/3678053283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c900fe530 0x7f6c90108260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f6c88001cb0 tx=0x7f6c88004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c867fc700 10 client.?.objecter ms_handle_connect 0x7f6c900fe530 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c867fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c867fc700 1 -- 172.21.15.87:0/3678053283 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880079b0 con 0x7f6c900fe530 2024-02-23T00:29:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c867fc700 1 -- 172.21.15.87:0/3678053283 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c880090d0 con 0x7f6c900fe530 2024-02-23T00:29:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c867fc700 1 -- 172.21.15.87:0/3678053283 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800fc20 con 0x7f6c900fe530 2024-02-23T00:29:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c80005040 con 0x7f6c900fe530 2024-02-23T00:29:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c979cd700 10 client.14757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c90109070 con 0x7f6c900fe530 2024-02-23T00:29:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.294+0000 7f6c979cd700 1 librados: init done 2024-02-23T00:29:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.295+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9005fa60 con 0x7f6c900fe530 2024-02-23T00:29:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.295+0000 7f6c867fc700 1 -- 172.21.15.87:0/3678053283 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f6c8800fd80 con 0x7f6c900fe530 2024-02-23T00:29:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.296+0000 7f6c867fc700 1 --2- 172.21.15.87:0/3678053283 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6c7c062c90 0x7f6c7c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.296+0000 7f6c867fc700 1 -- 172.21.15.87:0/3678053283 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(86..86 src has 28..86) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f6c8807ccb0 con 0x7f6c900fe530 2024-02-23T00:29:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.296+0000 7f6c867fc700 10 client.14757.objecter ms_dispatch 0x7f6c90001550 osd_map(86..86 src has 28..86) v4 2024-02-23T00:29:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.296+0000 7f6c867fc700 3 client.14757.objecter handle_osd_map got epochs [86,86] > 0 2024-02-23T00:29:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.296+0000 7f6c867fc700 3 client.14757.objecter handle_osd_map decoding full epoch 86 2024-02-23T00:29:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.296+0000 7f6c95769700 1 --2- 172.21.15.87:0/3678053283 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6c7c062c90 0x7f6c7c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.296+0000 7f6c867fc700 20 client.14757.objecter dump_active .. 0 homeless 2024-02-23T00:29:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.296+0000 7f6c867fc700 10 client.14757.objecter ms_handle_connect 0x7f6c7c062c90 2024-02-23T00:29:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.296+0000 7f6c95769700 1 --2- 172.21.15.87:0/3678053283 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6c7c062c90 0x7f6c7c065150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f6c8c004d20 tx=0x7f6c8c009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.300+0000 7f6c867fc700 1 -- 172.21.15.87:0/3678053283 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6c880509c0 con 0x7f6c900fe530 2024-02-23T00:29:55.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:55.484+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f6c9005ee80 con 0x7f6c900fe530 2024-02-23T00:29:55.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:54.068456+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:29:55.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:54.071367+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:29:55.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:54.236871+0000 osd.0 (osd.0) 47 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:29:55.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:54.237943+0000 osd.0 (osd.0) 48 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:29:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:54.068456+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:29:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:54.071367+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:29:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:54.236871+0000 osd.0 (osd.0) 47 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:29:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:54.237943+0000 osd.0 (osd.0) 48 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:29:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:54.068456+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:29:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:54.071367+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:29:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:54.236871+0000 osd.0 (osd.0) 47 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:29:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:54.237943+0000 osd.0 (osd.0) 48 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:29:55.877 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:29:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:55.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:29:55.980 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:29:56.016 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:56.038 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:29:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:56.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:29:56.142 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:29:56.176 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:56.195 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:29:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:56.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:29:56.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.295+0000 7f6c867fc700 1 -- 172.21.15.87:0/3678053283 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v87) v1 ==== 87+0+0 (secure 0 0 0) 0x7f6c880505b0 con 0x7f6c900fe530 2024-02-23T00:29:56.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 10 librados: watch_flush enter 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 10 librados: watch_flush exit 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 20 client.14757.objecter shutdown clearing up homeless session... 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 10 client.14757.objecter successfully canceled tick 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6c7c062c90 msgr2=0x7f6c7c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 1 --2- 172.21.15.87:0/3678053283 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6c7c062c90 0x7f6c7c065150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f6c8c004d20 tx=0x7f6c8c009040).stop 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c900fe530 msgr2=0x7f6c90108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 1 --2- 172.21.15.87:0/3678053283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c900fe530 0x7f6c90108260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f6c88001cb0 tx=0x7f6c88004c50).stop 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 shutdown_connections 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 1 --2- 172.21.15.87:0/3678053283 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6c7c062c90 0x7f6c7c065150 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 1 --2- 172.21.15.87:0/3678053283 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c900fe530 0x7f6c90108260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 1 --2- 172.21.15.87:0/3678053283 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c900f6a90 0x7f6c9010e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.297+0000 7f6c979cd700 1 --2- 172.21.15.87:0/3678053283 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c900fdbd0 0x7f6c9010e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.298+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 >> 172.21.15.87:0/3678053283 conn(0x7f6c9006c370 msgr2=0x7f6c900f58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:56.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.298+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 shutdown_connections 2024-02-23T00:29:56.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.298+0000 7f6c979cd700 1 -- 172.21.15.87:0/3678053283 wait complete. 2024-02-23T00:29:56.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.298+0000 7f6c979cd700 1 librados: shutdown 2024-02-23T00:29:56.300 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:29:56.301 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:29:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:56.354 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:29:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:56.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:29:56.454 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:55.234030+0000 osd.0 (osd.0) 49 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:29:56.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:55.237036+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:29:56.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:55.485260+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.87:0/3678053283' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:29:56.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:55.487897+0000 mgr.y (mgr.24437) 109 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:56.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:56.038292+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.e deep-scrub starts 2024-02-23T00:29:56.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:56.044325+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.e deep-scrub ok 2024-02-23T00:29:56.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:55.234030+0000 osd.0 (osd.0) 49 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:29:56.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:55.237036+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:29:56.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:55.485260+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.87:0/3678053283' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:29:56.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:55.487897+0000 mgr.y (mgr.24437) 109 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:56.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:56.038292+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.e deep-scrub starts 2024-02-23T00:29:56.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:56.044325+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.e deep-scrub ok 2024-02-23T00:29:56.461 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:29:56.494 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:56.515 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:29:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:56.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:29:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 -- 172.21.15.87:0/878215644 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 msgr2=0x7f7bdc103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 --2- 172.21.15.87:0/878215644 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 0x7f7bdc103f60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7bcc0047b0 tx=0x7f7bcc009cb0).stop 2024-02-23T00:29:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 -- 172.21.15.87:0/878215644 shutdown_connections 2024-02-23T00:29:56.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 --2- 172.21.15.87:0/878215644 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bdc104450 0x7f7bdc106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 --2- 172.21.15.87:0/878215644 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bdc06b110 0x7f7bdc06b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 --2- 172.21.15.87:0/878215644 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 0x7f7bdc103f60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 -- 172.21.15.87:0/878215644 >> 172.21.15.87:0/878215644 conn(0x7f7bdc0f37f0 msgr2=0x7f7bdc0f5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 -- 172.21.15.87:0/878215644 shutdown_connections 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 -- 172.21.15.87:0/878215644 wait complete. 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 librados: starting msgr at 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.444+0000 7f7be3578700 1 librados: starting objecter 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 Processor -- start 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 -- start start 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 librados: setting wanted keys 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 librados: calling monclient init 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bdc06b110 0x7f7bdc1103a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 0x7f7bdc110890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bdc104450 0x7f7bdc10a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc10aa30 con 0x7f7bdc06b110 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc10aba0 con 0x7f7bdc104450 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be3578700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc10ad60 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be0b13700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 0x7f7bdc110890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be1314700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bdc06b110 0x7f7bdc1103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be0b13700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 0x7f7bdc110890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:59994/0 (socket says 172.21.15.87:59994) 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be0b13700 1 -- 172.21.15.87:0/2426931312 learned_addr learned my addr 172.21.15.87:0/2426931312 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be1b15700 1 --2- 172.21.15.87:0/2426931312 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bdc104450 0x7f7bdc10a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be0b13700 1 -- 172.21.15.87:0/2426931312 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bdc104450 msgr2=0x7f7bdc10a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be0b13700 1 --2- 172.21.15.87:0/2426931312 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7bdc104450 0x7f7bdc10a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be0b13700 1 -- 172.21.15.87:0/2426931312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bdc06b110 msgr2=0x7f7bdc1103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be0b13700 1 --2- 172.21.15.87:0/2426931312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bdc06b110 0x7f7bdc1103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be0b13700 1 -- 172.21.15.87:0/2426931312 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd8005040 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be1314700 1 --2- 172.21.15.87:0/2426931312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7bdc06b110 0x7f7bdc1103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7be0b13700 1 --2- 172.21.15.87:0/2426931312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 0x7f7bdc110890 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7bcc009b10 tx=0x7f7bcc009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7bd27fc700 10 client.?.objecter ms_handle_connect 0x7f7bdc06b9e0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7bd27fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.445+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc018070 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.446+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bcc005040 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.446+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bcc013d60 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.446+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc01dd00 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.446+0000 7f7be3578700 10 client.24689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.446+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc10b200 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.446+0000 7f7be3578700 1 librados: init done 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.446+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7bdc10b3f0 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.447+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7bcc009070 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7bd27fc700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7bc8062c90 0x7f7bc8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(87..87 src has 28..87) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f7bcc07e450 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 osd_map(87..87 src has 28..87) v4 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be1314700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7bc8062c90 0x7f7bc8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7bd27fc700 3 client.24689.objecter handle_osd_map got epochs [87,87] > 0 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7bd27fc700 3 client.24689.objecter handle_osd_map decoding full epoch 87 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7bd27fc700 20 client.24689.objecter dump_active .. 0 homeless 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v87) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7bcc07e930 con 0x7f7bdc06b9e0 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7bd27fc700 10 client.24689.objecter ms_handle_connect 0x7f7bc8062c90 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be1314700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7bc8062c90 0x7f7bc8065150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7bd8004c80 tx=0x7f7bd800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 10 client.24689.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 1 2024-02-23T00:29:56.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7bdc028540 0x7f7bdc0fdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.567 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:29:56.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc047540 osd=0 3 2024-02-23T00:29:56.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc047540 osd=0 3 2024-02-23T00:29:56.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc047540 osd=0 3 2024-02-23T00:29:56.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 1 2024-02-23T00:29:56.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc047540 osd=0 3 2024-02-23T00:29:56.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 0 1 2024-02-23T00:29:56.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc047540 osd=0 4 2024-02-23T00:29:56.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 10 client.24689.objecter _send_command 1 2024-02-23T00:29:56.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bdc1092d0 con 0x7f7bdc028540 2024-02-23T00:29:56.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be1b15700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7bdc028540 0x7f7bdc0fdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7bd27fc700 10 client.24689.objecter ms_handle_connect 0x7f7bdc028540 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.448+0000 7f7be1b15700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7bdc028540 0x7f7bdc0fdf10 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.449+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7bdc1092d0 con 0x7f7bdc028540 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.449+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.449+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 1 = system:0 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.449+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc047540 osd=0 4 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.449+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 0 1 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 10 client.24689.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 2 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc047540 osd=0 3 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc047540 osd=0 3 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc047540 osd=0 3 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 2 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc047540 osd=0 3 2024-02-23T00:29:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 0 2 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc047540 osd=0 4 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 10 client.24689.objecter _send_command 2 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.465+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7bdc059e00 con 0x7f7bdc028540 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.467+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7bdc059e00 con 0x7f7bdc028540 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.467+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.467+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.467+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc047540 osd=0 4 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.467+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 0 2 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 10 client.24689.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 3 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6805/685354171 conn(0x7f7bdc0fea80 0x7f7bdc0fee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc0fe400 osd=1 3 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc0fe400 osd=1 3 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc0fe400 osd=1 3 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 3 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc0fe400 osd=1 3 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 1 3 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc0fe400 osd=1 4 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 10 client.24689.objecter _send_command 3 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bdc0ff350 con 0x7f7bdc0fea80 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be1314700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6805/685354171 conn(0x7f7bdc0fea80 0x7f7bdc0fee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7bd27fc700 10 client.24689.objecter ms_handle_connect 0x7f7bdc0fea80 2024-02-23T00:29:56.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.468+0000 7f7be1314700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6805/685354171 conn(0x7f7bdc0fea80 0x7f7bdc0fee60 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.469+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7bdc0ff350 con 0x7f7bdc0fea80 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.469+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.469+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 3 = system:0 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.469+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc0fe400 osd=1 4 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.469+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 1 3 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 10 client.24689.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 4 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc0fe400 osd=1 3 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc0fe400 osd=1 3 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc0fe400 osd=1 3 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 4 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc0fe400 osd=1 3 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 1 4 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc0fe400 osd=1 4 2024-02-23T00:29:56.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 10 client.24689.objecter _send_command 4 2024-02-23T00:29:56.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.483+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7bdc11b1f0 con 0x7f7bdc0fea80 2024-02-23T00:29:56.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7bdc11b1f0 con 0x7f7bdc0fea80 2024-02-23T00:29:56.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:29:56.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc0fe400 osd=1 4 2024-02-23T00:29:56.573 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:29:56.573 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:29:56.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 1 4 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 10 client.24689.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 5 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7bdc0594d0 0x7f7bdc11e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc059360 osd=2 3 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc059360 osd=2 3 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc059360 osd=2 3 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 5 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc059360 osd=2 3 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 2 5 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc059360 osd=2 4 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 10 client.24689.objecter _send_command 5 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bdc0281d0 con 0x7f7bdc0594d0 2024-02-23T00:29:56.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.485+0000 7f7be0b13700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7bdc0594d0 0x7f7bdc11e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.486+0000 7f7bd27fc700 10 client.24689.objecter ms_handle_connect 0x7f7bdc0594d0 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.486+0000 7f7be0b13700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7bdc0594d0 0x7f7bdc11e580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.486+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7bdc0281d0 con 0x7f7bdc0594d0 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.486+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.486+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 5 = system:0 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.486+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc059360 osd=2 4 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.486+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 2 5 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 10 client.24689.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 6 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc059360 osd=2 3 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc059360 osd=2 3 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc059360 osd=2 3 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 6 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc059360 osd=2 3 2024-02-23T00:29:56.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 2 6 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc059360 osd=2 4 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 10 client.24689.objecter _send_command 6 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.501+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7bdc027600 con 0x7f7bdc0594d0 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7bdc027600 con 0x7f7bdc0594d0 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc059360 osd=2 4 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 2 6 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 10 client.24689.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 7 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6813/128059790 conn(0x7f7bdc11c0d0 0x7f7bdc121c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc027a10 osd=3 3 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc027a10 osd=3 3 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc027a10 osd=3 3 2024-02-23T00:29:56.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 7 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc027a10 osd=3 3 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 3 7 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc027a10 osd=3 4 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 10 client.24689.objecter _send_command 7 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bdc124920 con 0x7f7bdc11c0d0 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.502+0000 7f7be1b15700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6813/128059790 conn(0x7f7bdc11c0d0 0x7f7bdc121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.503+0000 7f7bd27fc700 10 client.24689.objecter ms_handle_connect 0x7f7bdc11c0d0 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.503+0000 7f7be1b15700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6813/128059790 conn(0x7f7bdc11c0d0 0x7f7bdc121c00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.503+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7bdc124920 con 0x7f7bdc11c0d0 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.503+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.503+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 7 = system:0 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.503+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc027a10 osd=3 4 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.503+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 3 7 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 10 client.24689.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:29:56.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 8 2024-02-23T00:29:56.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc027a10 osd=3 3 2024-02-23T00:29:56.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc027a10 osd=3 3 2024-02-23T00:29:56.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc027a10 osd=3 3 2024-02-23T00:29:56.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 8 2024-02-23T00:29:56.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc027a10 osd=3 3 2024-02-23T00:29:56.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 3 8 2024-02-23T00:29:56.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc027a10 osd=3 4 2024-02-23T00:29:56.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 10 client.24689.objecter _send_command 8 2024-02-23T00:29:56.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.518+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7bdc11b630 con 0x7f7bdc11c0d0 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.519+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7bdc11b630 con 0x7f7bdc11c0d0 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.519+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.519+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.519+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc027a10 osd=3 4 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.519+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 3 8 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.519+0000 7f7be3578700 10 client.24689.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.519+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 9 2024-02-23T00:29:56.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.519+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7bdc11b7d0 0x7f7bdc123310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc11b630 osd=4 3 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11b630 osd=4 3 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc11b630 osd=4 3 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 9 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc11b630 osd=4 3 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 4 9 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11b630 osd=4 4 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be3578700 10 client.24689.objecter _send_command 9 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bdc11f990 con 0x7f7bdc11b7d0 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be1314700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7bdc11b7d0 0x7f7bdc123310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7be1314700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7bdc11b7d0 0x7f7bdc123310 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.520+0000 7f7bd27fc700 10 client.24689.objecter ms_handle_connect 0x7f7bdc11b7d0 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.521+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7bdc11f990 con 0x7f7bdc11b7d0 2024-02-23T00:29:56.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.521+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.521+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 9 = system:0 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.521+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc11b630 osd=4 4 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.521+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 4 9 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 10 client.24689.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 10 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc11b630 osd=4 3 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11b630 osd=4 3 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc11b630 osd=4 3 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 10 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc11b630 osd=4 3 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 4 10 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11b630 osd=4 4 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 10 client.24689.objecter _send_command 10 2024-02-23T00:29:56.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.537+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7bdc11f990 con 0x7f7bdc11b7d0 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7bdc11f990 con 0x7f7bdc11b7d0 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc11b630 osd=4 4 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 4 10 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 10 client.24689.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 11 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7bdc123d40 0x7f7bdc1241a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc11f990 osd=5 3 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11f990 osd=5 3 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc11f990 osd=5 3 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 11 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc11f990 osd=5 3 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 5 11 2024-02-23T00:29:56.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11f990 osd=5 4 2024-02-23T00:29:56.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 10 client.24689.objecter _send_command 11 2024-02-23T00:29:56.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.539+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bdc124780 con 0x7f7bdc123d40 2024-02-23T00:29:56.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.540+0000 7f7be0b13700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7bdc123d40 0x7f7bdc1241a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.540+0000 7f7bd27fc700 10 client.24689.objecter ms_handle_connect 0x7f7bdc123d40 2024-02-23T00:29:56.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.540+0000 7f7be0b13700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7bdc123d40 0x7f7bdc1241a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.583 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.541+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7bdc124780 con 0x7f7bdc123d40 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.541+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.541+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 11 = system:0 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.541+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc11f990 osd=5 4 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.541+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 5 11 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.555+0000 7f7be3578700 10 client.24689.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 12 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc11f990 osd=5 3 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11f990 osd=5 3 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc11f990 osd=5 3 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 12 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc11f990 osd=5 3 2024-02-23T00:29:56.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 5 12 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11f990 osd=5 4 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 10 client.24689.objecter _send_command 12 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.556+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7bdc124780 con 0x7f7bdc123d40 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.557+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7bdc124780 con 0x7f7bdc123d40 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.557+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.557+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.557+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc11f990 osd=5 4 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.557+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 5 12 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 10 client.24689.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 13 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7bdc12a6e0 0x7f7bdc12ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc124780 osd=6 3 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc124780 osd=6 3 2024-02-23T00:29:56.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc124780 osd=6 3 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 13 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc124780 osd=6 3 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 6 13 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc124780 osd=6 4 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 10 client.24689.objecter _send_command 13 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bdc12e500 con 0x7f7bdc12a6e0 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be1b15700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7bdc12a6e0 0x7f7bdc12ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7bd27fc700 10 client.24689.objecter ms_handle_connect 0x7f7bdc12a6e0 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.558+0000 7f7be1b15700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7bdc12a6e0 0x7f7bdc12ab40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.559+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7bdc12e500 con 0x7f7bdc12a6e0 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.559+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.559+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 13 = system:0 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.559+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc124780 osd=6 4 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.559+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 6 13 2024-02-23T00:29:56.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 10 client.24689.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 14 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc124780 osd=6 3 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc124780 osd=6 3 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc124780 osd=6 3 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 14 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc124780 osd=6 3 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 6 14 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc124780 osd=6 4 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 10 client.24689.objecter _send_command 14 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.574+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7bdc128b60 con 0x7f7bdc12a6e0 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7bdc128b60 con 0x7f7bdc12a6e0 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc124780 osd=6 4 2024-02-23T00:29:56.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 6 14 2024-02-23T00:29:56.588 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 10 client.24689.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 15 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6812/373787000 conn(0x7f7bdc129210 0x7f7bdc129670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc128b60 osd=7 3 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc128b60 osd=7 3 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc128b60 osd=7 3 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 15 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc128b60 osd=7 3 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 7 15 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc128b60 osd=7 4 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 10 client.24689.objecter _send_command 15 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.576+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bdc12a030 con 0x7f7bdc129210 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.577+0000 7f7be1314700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6812/373787000 conn(0x7f7bdc129210 0x7f7bdc129670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:29:56.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.577+0000 7f7bd27fc700 10 client.24689.objecter ms_handle_connect 0x7f7bdc129210 2024-02-23T00:29:56.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.577+0000 7f7be1314700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6812/373787000 conn(0x7f7bdc129210 0x7f7bdc129670 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:29:56.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.578+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7bdc12a030 con 0x7f7bdc129210 2024-02-23T00:29:56.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.578+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:29:56.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.578+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 15 = system:0 2024-02-23T00:29:56.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.578+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc128b60 osd=7 4 2024-02-23T00:29:56.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.578+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 7 15 2024-02-23T00:29:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 10 client.24689.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:29:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign -1 16 2024-02-23T00:29:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc128b60 osd=7 3 2024-02-23T00:29:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc128b60 osd=7 3 2024-02-23T00:29:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 20 client.24689.objecter _get_session s=0x7f7bdc128b60 osd=7 3 2024-02-23T00:29:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 15 client.24689.objecter _session_command_op_remove -1 16 2024-02-23T00:29:56.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 20 client.24689.objecter get_session s=0x7f7bdc128b60 osd=7 3 2024-02-23T00:29:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 15 client.24689.objecter _session_command_op_assign 7 16 2024-02-23T00:29:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc128b60 osd=7 4 2024-02-23T00:29:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 10 client.24689.objecter _send_command 16 2024-02-23T00:29:56.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.592+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7bdc12a030 con 0x7f7bdc129210 2024-02-23T00:29:56.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7bd27fc700 1 -- 172.21.15.87:0/2426931312 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7bdc12a030 con 0x7f7bdc129210 2024-02-23T00:29:56.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7bd27fc700 10 client.24689.objecter ms_dispatch 0x7f7bdc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:29:56.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7bd27fc700 10 client.24689.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7bd27fc700 20 client.24689.objecter put_session s=0x7f7bdc128b60 osd=7 4 2024-02-23T00:29:56.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7bd27fc700 15 client.24689.objecter _session_command_op_remove 7 16 2024-02-23T00:29:56.595 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 10 librados: watch_flush enter 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 10 librados: watch_flush exit 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 10 client.24689.objecter close_session for osd.0 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7bdc028540 msgr2=0x7f7bdc0fdf10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7bdc028540 0x7f7bdc0fdf10 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc047540 osd=0 1 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 10 client.24689.objecter close_session for osd.1 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6805/685354171 conn(0x7f7bdc0fea80 msgr2=0x7f7bdc0fee60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6805/685354171 conn(0x7f7bdc0fea80 0x7f7bdc0fee60 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc0fe400 osd=1 1 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 10 client.24689.objecter close_session for osd.2 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7bdc0594d0 msgr2=0x7f7bdc11e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7bdc0594d0 0x7f7bdc11e580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc059360 osd=2 1 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 10 client.24689.objecter close_session for osd.3 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6813/128059790 conn(0x7f7bdc11c0d0 msgr2=0x7f7bdc121c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be1b15700 1 -- 172.21.15.87:0/2426931312 reap_dead start 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.594+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6813/128059790 conn(0x7f7bdc11c0d0 0x7f7bdc121c00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc027a10 osd=3 1 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 10 client.24689.objecter close_session for osd.4 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7bdc11b7d0 msgr2=0x7f7bdc123310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7bdc11b7d0 0x7f7bdc123310 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11b630 osd=4 1 2024-02-23T00:29:56.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 10 client.24689.objecter close_session for osd.5 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7bdc123d40 msgr2=0x7f7bdc1241a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7bdc123d40 0x7f7bdc1241a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc11f990 osd=5 1 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 10 client.24689.objecter close_session for osd.6 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7bdc12a6e0 msgr2=0x7f7bdc12ab40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7bdc12a6e0 0x7f7bdc12ab40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc124780 osd=6 1 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 10 client.24689.objecter close_session for osd.7 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6812/373787000 conn(0x7f7bdc129210 msgr2=0x7f7bdc129670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.171:6812/373787000 conn(0x7f7bdc129210 0x7f7bdc129670 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 20 client.24689.objecter put_session s=0x7f7bdc128b60 osd=7 1 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 20 client.24689.objecter shutdown clearing up homeless session... 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 10 client.24689.objecter successfully canceled tick 2024-02-23T00:29:56.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be1b15700 1 -- 172.21.15.87:0/2426931312 reap_dead start 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7bc8062c90 msgr2=0x7f7bc8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7bc8062c90 0x7f7bc8065150 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7bd8004c80 tx=0x7f7bd800b040).stop 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 msgr2=0x7f7bdc110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 0x7f7bdc110890 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7bcc009b10 tx=0x7f7bcc009cb0).stop 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 shutdown_connections 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7bc8062c90 0x7f7bc8065150 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 --2- 172.21.15.87:0/2426931312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7bdc06b9e0 0x7f7bdc110890 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 >> 172.21.15.87:0/2426931312 conn(0x7f7bdc0f37f0 msgr2=0x7f7bdc0f4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 shutdown_connections 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 -- 172.21.15.87:0/2426931312 wait complete. 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:29:56.595+0000 7f7be3578700 1 librados: shutdown 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.601 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:29:56.619 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:29:56.648 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:55.234030+0000 osd.0 (osd.0) 49 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:29:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:55.237036+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:29:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:55.485260+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.87:0/3678053283' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:29:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:55.487897+0000 mgr.y (mgr.24437) 109 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:29:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:56.038292+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.e deep-scrub starts 2024-02-23T00:29:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:56.044325+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.e deep-scrub ok 2024-02-23T00:29:57.398 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:29:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:57.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:29:57.502 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:29:57.534 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:57.555 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:29:57.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:56.281000+0000 mon.a (mon.0) 743 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:29:57.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:56.295692+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.87:0/3678053283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:29:57.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:56.295743+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-23T00:29:57.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:29:56.447632+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? 172.21.15.87:0/2426931312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:29:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:57.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:29:57.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:56.281000+0000 mon.a (mon.0) 743 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:29:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:56.295692+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.87:0/3678053283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:29:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:56.295743+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-23T00:29:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:29:56.447632+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? 172.21.15.87:0/2426931312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:29:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:56.281000+0000 mon.a (mon.0) 743 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:29:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:56.295692+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.87:0/3678053283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:29:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:56.295743+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-23T00:29:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:29:56.447632+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? 172.21.15.87:0/2426931312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:29:57.659 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:29:57.689 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:57.710 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:29:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:57.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:29:57.814 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:29:57.843 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:58.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:57.488178+0000 mgr.y (mgr.24437) 110 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:58.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:57.560891+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.f deep-scrub starts 2024-02-23T00:29:58.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:29:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:57.567416+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.f deep-scrub ok 2024-02-23T00:29:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:57.488178+0000 mgr.y (mgr.24437) 110 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:57.560891+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.f deep-scrub starts 2024-02-23T00:29:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:29:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:57.567416+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.f deep-scrub ok 2024-02-23T00:29:58.392 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:29:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:58.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:29:58.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:29:58.529 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:58.542 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:29:58.542 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:29:58.550 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:29:58.552 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:57.488178+0000 mgr.y (mgr.24437) 110 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:29:58.552 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:57.560891+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.f deep-scrub starts 2024-02-23T00:29:58.552 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:29:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:57.567416+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.f deep-scrub ok 2024-02-23T00:29:58.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:58.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:29:58.654 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:29:58.686 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:58.708 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:29:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:58.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:29:58.812 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:29:58.840 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:58.863 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:29:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:58.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:29:58.967 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:29:58.996 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:59.444 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:29:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:29:59.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:29:59.548 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:29:59.574 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:29:59.771 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:29:59.771 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:29:59.771 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-23T00:29:59.771 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight-by-utilization 110 0.05 2024-02-23T00:30:00.024 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:00.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:00.040 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-23T00:30:00.099 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:00.128 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:00.161 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:00.183 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:00.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:00.287 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:00.314 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:00.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:00.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:00.400 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:30:00.470 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:00.503 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:00.524 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:00.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:00.628 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:00.657 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:29:59.488613+0000 mgr.y (mgr.24437) 111 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:00.657 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:00.000093+0000 mon.a (mon.0) 746 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set 2024-02-23T00:30:00.657 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:00.000116+0000 mon.a (mon.0) 747 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-23T00:30:00.657 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:29:59.488613+0000 mgr.y (mgr.24437) 111 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:00.657 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:00.000093+0000 mon.a (mon.0) 746 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set 2024-02-23T00:30:00.657 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:00.000116+0000 mon.a (mon.0) 747 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-23T00:30:00.659 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:00.731 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:00.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:00.836 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:00.866 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:29:59.488613+0000 mgr.y (mgr.24437) 111 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:00.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:00.000093+0000 mon.a (mon.0) 746 : cluster [WRN] Health detail: HEALTH_WARN noscrub flag(s) set 2024-02-23T00:30:00.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:00.000116+0000 mon.a (mon.0) 747 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-23T00:30:00.867 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:30:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:30:00.869 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:00.889 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:00.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:00.994 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:01.022 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:01.042 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:01.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:01.080 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:01.146 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:01.173 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:01.192 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:01.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:01.296 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:01.323 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:01.341 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:01.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:01.445 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:01.447 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:30:01.472 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:01.491 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:01.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:01.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:01.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:01.621 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:01.683 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:00.578657+0000 mgr.y (mgr.24437) 112 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:01.683 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:00.578657+0000 mgr.y (mgr.24437) 112 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:00.578657+0000 mgr.y (mgr.24437) 112 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:02.073 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:02.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:02.178 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:02.205 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:02.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:30:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:02] "GET /metrics HTTP/1.1" 200 172196 "" "Prometheus/2.33.4" 2024-02-23T00:30:02.454 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:02.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:02.476 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:02.558 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:02.586 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:02.729 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:01.489172+0000 mgr.y (mgr.24437) 113 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:02.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:01.489172+0000 mgr.y (mgr.24437) 113 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:02.735 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:02.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:02.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:02.743 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:02.843 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:02.874 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:01.489172+0000 mgr.y (mgr.24437) 113 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:02.876 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:02.898 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:02.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:03.002 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:03.033 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:03.053 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:03.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:03.157 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:03.190 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.219+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/4202852092 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 msgr2=0x7f5d580f8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.219+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/4202852092 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 0x7f5d580f8f90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f5d48004870 tx=0x7f5d48009cb0).stop 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.219+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/4202852092 shutdown_connections 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.219+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/4202852092 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d58102110 0x7f5d58104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.219+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/4202852092 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 0x7f5d580f8f90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.219+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/4202852092 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5d580f9510 0x7f5d58101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.219+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/4202852092 >> 172.21.15.87:0/4202852092 conn(0x7f5d580f26e0 msgr2=0x7f5d580f4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.219+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/4202852092 shutdown_connections 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.219+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/4202852092 wait complete. 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 librados: starting msgr at 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 librados: starting objecter 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 Processor -- start 2024-02-23T00:30:03.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 -- start start 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 librados: setting wanted keys 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 librados: calling monclient init 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 0x7f5d580f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5d580f9510 0x7f5d580f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d58102110 0x7f5d580f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d58116650 con 0x7f5d580f9510 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d581167c0 con 0x7f5d58102110 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5e4e4700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d58116980 con 0x7f5d580f8bb0 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d57fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 0x7f5d580f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d4ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5d580f9510 0x7f5d580f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d57fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 0x7f5d580f6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40718/0 (socket says 172.21.15.87:40718) 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d4ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5d580f9510 0x7f5d580f7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57394/0 (socket says 172.21.15.87:57394) 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d57fff700 1 -- 172.21.15.87:0/3821768412 learned_addr learned my addr 172.21.15.87:0/3821768412 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:03.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5ca81700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d58102110 0x7f5d580f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d57fff700 1 -- 172.21.15.87:0/3821768412 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d58102110 msgr2=0x7f5d580f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d57fff700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d58102110 0x7f5d580f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d57fff700 1 -- 172.21.15.87:0/3821768412 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5d580f9510 msgr2=0x7f5d580f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d57fff700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5d580f9510 0x7f5d580f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d57fff700 1 -- 172.21.15.87:0/3821768412 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d40005040 con 0x7f5d580f8bb0 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d5ca81700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d58102110 0x7f5d580f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d4ffff700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5d580f9510 0x7f5d580f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d57fff700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 0x7f5d580f6b60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5d48009eb0 tx=0x7f5d4800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d55ffb700 10 client.?.objecter ms_handle_connect 0x7f5d580f8bb0 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.220+0000 7f5d55ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.221+0000 7f5d55ffb700 1 -- 172.21.15.87:0/3821768412 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4800e070 con 0x7f5d580f8bb0 2024-02-23T00:30:03.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.221+0000 7f5d55ffb700 1 -- 172.21.15.87:0/3821768412 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d48013da0 con 0x7f5d580f8bb0 2024-02-23T00:30:03.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.221+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d48005040 con 0x7f5d580f8bb0 2024-02-23T00:30:03.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.221+0000 7f5d55ffb700 1 -- 172.21.15.87:0/3821768412 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d48019070 con 0x7f5d580f8bb0 2024-02-23T00:30:03.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.221+0000 7f5d5e4e4700 10 client.24695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:03.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.221+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d58116e70 con 0x7f5d580f8bb0 2024-02-23T00:30:03.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.221+0000 7f5d5e4e4700 1 librados: init done 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.222+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5805f970 con 0x7f5d580f8bb0 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.222+0000 7f5d55ffb700 1 -- 172.21.15.87:0/3821768412 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f5d4801e420 con 0x7f5d580f8bb0 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.222+0000 7f5d55ffb700 1 --2- 172.21.15.87:0/3821768412 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5d44062c90 0x7f5d44065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.222+0000 7f5d55ffb700 1 -- 172.21.15.87:0/3821768412 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(87..87 src has 28..87) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f5d4807d4d0 con 0x7f5d580f8bb0 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.222+0000 7f5d55ffb700 10 client.24695.objecter ms_dispatch 0x7f5d58001550 osd_map(87..87 src has 28..87) v4 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.222+0000 7f5d55ffb700 3 client.24695.objecter handle_osd_map got epochs [87,87] > 0 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.222+0000 7f5d55ffb700 3 client.24695.objecter handle_osd_map decoding full epoch 87 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.222+0000 7f5d4ffff700 1 --2- 172.21.15.87:0/3821768412 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5d44062c90 0x7f5d44065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.222+0000 7f5d55ffb700 20 client.24695.objecter dump_active .. 0 homeless 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.223+0000 7f5d55ffb700 10 client.24695.objecter ms_handle_connect 0x7f5d44062c90 2024-02-23T00:30:03.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.223+0000 7f5d4ffff700 1 --2- 172.21.15.87:0/3821768412 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5d44062c90 0x7f5d44065150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5d580f8300 tx=0x7f5d4000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:03.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.227+0000 7f5d55ffb700 1 -- 172.21.15.87:0/3821768412 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5d480511e0 con 0x7f5d580f8bb0 2024-02-23T00:30:03.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.402+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}) v1 -- 0x7f5d580f80b0 con 0x7f5d44062c90 2024-02-23T00:30:03.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.403+0000 7f5d55ffb700 1 -- 172.21.15.87:0/3821768412 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+296 (secure 0 0 0) 0x7f5d580f80b0 con 0x7f5d44062c90 2024-02-23T00:30:03.404 INFO:teuthology.orchestra.run.smithi087.stdout:moved 0 / 396 (0%) 2024-02-23T00:30:03.404 INFO:teuthology.orchestra.run.smithi087.stdout:avg 49.5 2024-02-23T00:30:03.404 INFO:teuthology.orchestra.run.smithi087.stdout:stddev 5.22015 -> 5.22015 (expected baseline 6.58122) 2024-02-23T00:30:03.404 INFO:teuthology.orchestra.run.smithi087.stdout:min osd.2 with 40 -> 40 pgs (0.808081 -> 0.808081 * mean) 2024-02-23T00:30:03.404 INFO:teuthology.orchestra.run.smithi087.stdout:max osd.7 with 56 -> 56 pgs (1.13131 -> 1.13131 * mean) 2024-02-23T00:30:03.404 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:30:03.405 INFO:teuthology.orchestra.run.smithi087.stdout:oload 110 2024-02-23T00:30:03.405 INFO:teuthology.orchestra.run.smithi087.stdout:max_change 0.05 2024-02-23T00:30:03.405 INFO:teuthology.orchestra.run.smithi087.stdout:max_change_osds 4 2024-02-23T00:30:03.405 INFO:teuthology.orchestra.run.smithi087.stdout:average_utilization 0.0032 2024-02-23T00:30:03.405 INFO:teuthology.orchestra.run.smithi087.stdout:overload_utilization 0.0035 2024-02-23T00:30:03.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 10 librados: watch_flush enter 2024-02-23T00:30:03.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 10 librados: watch_flush exit 2024-02-23T00:30:03.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 20 client.24695.objecter shutdown clearing up homeless session... 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 10 client.24695.objecter successfully canceled tick 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5d44062c90 msgr2=0x7f5d44065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/3821768412 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5d44062c90 0x7f5d44065150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5d580f8300 tx=0x7f5d4000b040).stop 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 msgr2=0x7f5d580f6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 0x7f5d580f6b60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5d48009eb0 tx=0x7f5d4800b040).stop 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 shutdown_connections 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/3821768412 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5d44062c90 0x7f5d44065150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5d58102110 0x7f5d580f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5d580f8bb0 0x7f5d580f6b60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 --2- 172.21.15.87:0/3821768412 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5d580f9510 0x7f5d580f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 >> 172.21.15.87:0/3821768412 conn(0x7f5d580f26e0 msgr2=0x7f5d580f3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 shutdown_connections 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 -- 172.21.15.87:0/3821768412 wait complete. 2024-02-23T00:30:03.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:03.405+0000 7f5d5e4e4700 1 librados: shutdown 2024-02-23T00:30:03.408 INFO:teuthology.orchestra.run.smithi087.stderr:no change 2024-02-23T00:30:04.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:30:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:30:04.545 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:04.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:04.651 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:04.685 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:04.706 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:04.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:04.810 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:04.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:03.403017+0000 mgr.y (mgr.24437) 114 : audit [DBG] from='client.24695 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:04.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:03.489512+0000 mgr.y (mgr.24437) 115 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:04.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:04.216558+0000 osd.2 (osd.2) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:04.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:04.216608+0000 osd.2 (osd.2) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:03.403017+0000 mgr.y (mgr.24437) 114 : audit [DBG] from='client.24695 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:03.489512+0000 mgr.y (mgr.24437) 115 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:04.216558+0000 osd.2 (osd.2) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:04.216608+0000 osd.2 (osd.2) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:04.836 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:03.403017+0000 mgr.y (mgr.24437) 114 : audit [DBG] from='client.24695 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:03.489512+0000 mgr.y (mgr.24437) 115 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:04.216558+0000 osd.2 (osd.2) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:04.216608+0000 osd.2 (osd.2) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:05.034 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:05.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:05.039 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-23T00:30:05.039 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd test-reweight-by-utilization 110 1.0 2024-02-23T00:30:05.139 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:05.166 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:05.344 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:05.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:05.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:05.449 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:05.482 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:05.503 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:05.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:05.607 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:05.634 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:03.936143+0000 osd.6 (osd.6) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:05.634 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:03.936206+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:05.634 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:04.552753+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-23T00:30:05.634 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:03.936143+0000 osd.6 (osd.6) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:05.634 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:03.936206+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:05.635 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:04.552753+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-23T00:30:05.636 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:03.936143+0000 osd.6 (osd.6) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:03.936206+0000 osd.6 (osd.6) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:04.552753+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-23T00:30:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 -- 172.21.15.87:0/2449262158 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b2c0f8d10 msgr2=0x7f2b2c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/2449262158 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b2c0f8d10 0x7f2b2c0f90f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f2b1c004870 tx=0x7f2b1c009cb0).stop 2024-02-23T00:30:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 -- 172.21.15.87:0/2449262158 shutdown_connections 2024-02-23T00:30:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/2449262158 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b2c102270 0x7f2b2c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/2449262158 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b2c0f8d10 0x7f2b2c0f90f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/2449262158 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b2c0f9670 0x7f2b2c101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 -- 172.21.15.87:0/2449262158 >> 172.21.15.87:0/2449262158 conn(0x7f2b2c0f2820 msgr2=0x7f2b2c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 -- 172.21.15.87:0/2449262158 shutdown_connections 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 -- 172.21.15.87:0/2449262158 wait complete. 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 librados: starting msgr at 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 librados: starting objecter 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.146+0000 7f2b31bf5700 1 Processor -- start 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b31bf5700 1 -- start start 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b31bf5700 1 librados: setting wanted keys 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b31bf5700 1 librados: calling monclient init 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b31bf5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b2c0f8d10 0x7f2b2c0f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b31bf5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b2c0f9670 0x7f2b2c0f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b31bf5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b2c102270 0x7f2b2c0f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:06.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b31bf5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c116660 con 0x7f2b2c0f9670 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b31bf5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c1167d0 con 0x7f2b2c0f8d10 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b31bf5700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c116990 con 0x7f2b2c102270 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b2c0f9670 0x7f2b2c0f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b2c0f9670 0x7f2b2c0f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57404/0 (socket says 172.21.15.87:57404) 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2affd700 1 -- 172.21.15.87:0/1506701478 learned_addr learned my addr 172.21.15.87:0/1506701478 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2bfff700 1 --2- 172.21.15.87:0/1506701478 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b2c102270 0x7f2b2c0f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2b7fe700 1 --2- 172.21.15.87:0/1506701478 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b2c0f8d10 0x7f2b2c0f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2affd700 1 -- 172.21.15.87:0/1506701478 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b2c102270 msgr2=0x7f2b2c0f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2affd700 1 --2- 172.21.15.87:0/1506701478 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b2c102270 0x7f2b2c0f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2affd700 1 -- 172.21.15.87:0/1506701478 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b2c0f8d10 msgr2=0x7f2b2c0f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2affd700 1 --2- 172.21.15.87:0/1506701478 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b2c0f8d10 0x7f2b2c0f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.147+0000 7f2b2affd700 1 -- 172.21.15.87:0/1506701478 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b240097c0 con 0x7f2b2c0f9670 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b2affd700 1 --2- 172.21.15.87:0/1506701478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b2c0f9670 0x7f2b2c0f71b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f2b2000c810 tx=0x7f2b2000cb20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b28ff9700 10 client.?.objecter ms_handle_connect 0x7f2b2c0f9670 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b28ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b28ff9700 1 -- 172.21.15.87:0/1506701478 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20010510 con 0x7f2b2c0f9670 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b28ff9700 1 -- 172.21.15.87:0/1506701478 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b20010b40 con 0x7f2b2c0f9670 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b28ff9700 1 -- 172.21.15.87:0/1506701478 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b20012a90 con 0x7f2b2c0f9670 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c005040 con 0x7f2b2c0f9670 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b31bf5700 10 client.14775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c116f80 con 0x7f2b2c0f9670 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.148+0000 7f2b31bf5700 1 librados: init done 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.149+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c05fa60 con 0x7f2b2c0f9670 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.149+0000 7f2b28ff9700 1 -- 172.21.15.87:0/1506701478 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f2b20010cb0 con 0x7f2b2c0f9670 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.149+0000 7f2b28ff9700 1 --2- 172.21.15.87:0/1506701478 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2b14062c90 0x7f2b14065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:06.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.150+0000 7f2b28ff9700 1 -- 172.21.15.87:0/1506701478 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(88..88 src has 38..88) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f2b2007c9c0 con 0x7f2b2c0f9670 2024-02-23T00:30:06.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.150+0000 7f2b28ff9700 10 client.14775.objecter ms_dispatch 0x7f2b2c001550 osd_map(88..88 src has 38..88) v4 2024-02-23T00:30:06.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.150+0000 7f2b28ff9700 3 client.14775.objecter handle_osd_map got epochs [88,88] > 0 2024-02-23T00:30:06.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.150+0000 7f2b28ff9700 3 client.14775.objecter handle_osd_map decoding full epoch 88 2024-02-23T00:30:06.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.150+0000 7f2b2b7fe700 1 --2- 172.21.15.87:0/1506701478 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2b14062c90 0x7f2b14065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:06.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.150+0000 7f2b28ff9700 20 client.14775.objecter dump_active .. 0 homeless 2024-02-23T00:30:06.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.150+0000 7f2b28ff9700 10 client.14775.objecter ms_handle_connect 0x7f2b14062c90 2024-02-23T00:30:06.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.150+0000 7f2b2b7fe700 1 --2- 172.21.15.87:0/1506701478 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2b14062c90 0x7f2b14065150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f2b1c0043d0 tx=0x7f2b1c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:06.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.154+0000 7f2b28ff9700 1 -- 172.21.15.87:0/1506701478 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2b20050630 con 0x7f2b2c0f9670 2024-02-23T00:30:06.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.344+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f2b2c107330 con 0x7f2b2c0f9670 2024-02-23T00:30:06.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.879+0000 7f2b28ff9700 1 -- 172.21.15.87:0/1506701478 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v89) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2b20050220 con 0x7f2b2c0f9670 2024-02-23T00:30:06.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 10 librados: watch_flush enter 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 10 librados: watch_flush exit 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 20 client.14775.objecter shutdown clearing up homeless session... 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 10 client.14775.objecter successfully canceled tick 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2b14062c90 msgr2=0x7f2b14065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/1506701478 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2b14062c90 0x7f2b14065150 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f2b1c0043d0 tx=0x7f2b1c00b040).stop 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b2c0f9670 msgr2=0x7f2b2c0f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/1506701478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b2c0f9670 0x7f2b2c0f71b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f2b2000c810 tx=0x7f2b2000cb20).stop 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 shutdown_connections 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/1506701478 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2b14062c90 0x7f2b14065150 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/1506701478 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b2c102270 0x7f2b2c0f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/1506701478 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b2c0f8d10 0x7f2b2c0f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 --2- 172.21.15.87:0/1506701478 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b2c0f9670 0x7f2b2c0f71b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:06.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.881+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 >> 172.21.15.87:0/1506701478 conn(0x7f2b2c0f2820 msgr2=0x7f2b2c0f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:06.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.882+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 shutdown_connections 2024-02-23T00:30:06.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.882+0000 7f2b31bf5700 1 -- 172.21.15.87:0/1506701478 wait complete. 2024-02-23T00:30:06.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:06.882+0000 7f2b31bf5700 1 librados: shutdown 2024-02-23T00:30:06.884 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:30:07.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:05.490092+0000 mgr.y (mgr.24437) 116 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:30:07.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:06.345436+0000 mon.a (mon.0) 749 : audit [INF] from='client.? 172.21.15.87:0/1506701478' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:30:07.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:05.490092+0000 mgr.y (mgr.24437) 116 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:30:07.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:06.345436+0000 mon.a (mon.0) 749 : audit [INF] from='client.? 172.21.15.87:0/1506701478' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:30:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:05.490092+0000 mgr.y (mgr.24437) 116 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:30:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:06.345436+0000 mon.a (mon.0) 749 : audit [INF] from='client.? 172.21.15.87:0/1506701478' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:30:07.290 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:07.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:07.394 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:07.422 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:07.494 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:07.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:07.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:07.600 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:07.632 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:07.653 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:07.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:07.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:07.757 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:07.788 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:07.808 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:07.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:07.912 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:07.940 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:07.959 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:07.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:08.062 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:08.090 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:08.116 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:08.118 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:06.874991+0000 mon.a (mon.0) 750 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:08.118 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:06.879271+0000 mon.a (mon.0) 751 : audit [INF] from='client.? 172.21.15.87:0/1506701478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:30:08.118 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:06.879329+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-23T00:30:08.118 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:06.874991+0000 mon.a (mon.0) 750 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:08.118 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:06.879271+0000 mon.a (mon.0) 751 : audit [INF] from='client.? 172.21.15.87:0/1506701478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:30:08.118 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:06.879329+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-23T00:30:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:08.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:08.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:06.874991+0000 mon.a (mon.0) 750 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:08.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:06.879271+0000 mon.a (mon.0) 751 : audit [INF] from='client.? 172.21.15.87:0/1506701478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:30:08.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:06.879329+0000 mon.a (mon.0) 752 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-23T00:30:08.220 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:08.248 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:08.681 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:08.945 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:07.490440+0000 mgr.y (mgr.24437) 117 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:08.945 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:07.490440+0000 mgr.y (mgr.24437) 117 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:08.996 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:08.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:09.099 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:09.129 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:09.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:07.490440+0000 mgr.y (mgr.24437) 117 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:09.253 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:09.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:09.356 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:09.382 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:09.406 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:09.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:09.510 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:09.535 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:30:09.535 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:30:09.542 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:09.563 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:09.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:09.667 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:09.698 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.788+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1550782146 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1bb40fdbb0 msgr2=0x7f1bb40fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.788+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1550782146 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1bb40fdbb0 0x7f1bb40fdf90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1ba4004870 tx=0x7f1ba4009cb0).stop 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.788+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1550782146 shutdown_connections 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.788+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1550782146 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bb40f8420 0x7f1bb4104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.788+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1550782146 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1bb40fdbb0 0x7f1bb40fdf90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.788+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1550782146 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1bb40fe510 0x7f1bb40f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.788+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1550782146 >> 172.21.15.87:0/1550782146 conn(0x7f1bb40f3800 msgr2=0x7f1bb40f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.788+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1550782146 shutdown_connections 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.788+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1550782146 wait complete. 2024-02-23T00:30:09.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.789+0000 7f1bb92ab700 1 librados: starting msgr at 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.789+0000 7f1bb92ab700 1 librados: starting objecter 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.789+0000 7f1bb92ab700 1 Processor -- start 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.789+0000 7f1bb92ab700 1 -- start start 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.789+0000 7f1bb92ab700 1 librados: setting wanted keys 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.789+0000 7f1bb92ab700 1 librados: calling monclient init 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb92ab700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1bb40f8420 0x7f1bb410e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb92ab700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bb40fdbb0 0x7f1bb410e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb92ab700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1bb40fe510 0x7f1bb4108210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb92ab700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4108700 con 0x7f1bb40fe510 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb92ab700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4108870 con 0x7f1bb40fdbb0 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb92ab700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4108a30 con 0x7f1bb40f8420 2024-02-23T00:30:09.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1bb40fe510 0x7f1bb4108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1bb40fe510 0x7f1bb4108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:57408/0 (socket says 172.21.15.87:57408) 2024-02-23T00:30:09.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb37fe700 1 -- 172.21.15.87:0/1178975245 learned_addr learned my addr 172.21.15.87:0/1178975245 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:09.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb2ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1bb40f8420 0x7f1bb410e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb27fc700 1 --2- 172.21.15.87:0/1178975245 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bb40fdbb0 0x7f1bb410e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb2ffd700 1 -- 172.21.15.87:0/1178975245 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bb40fdbb0 msgr2=0x7f1bb410e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb2ffd700 1 --2- 172.21.15.87:0/1178975245 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1bb40fdbb0 0x7f1bb410e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb2ffd700 1 -- 172.21.15.87:0/1178975245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1bb40fe510 msgr2=0x7f1bb4108210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb2ffd700 1 --2- 172.21.15.87:0/1178975245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1bb40fe510 0x7f1bb4108210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb2ffd700 1 -- 172.21.15.87:0/1178975245 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba8005040 con 0x7f1bb40f8420 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb37fe700 1 --2- 172.21.15.87:0/1178975245 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1bb40fe510 0x7f1bb4108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb2ffd700 1 --2- 172.21.15.87:0/1178975245 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1bb40f8420 0x7f1bb410e070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1ba4004870 tx=0x7f1ba400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1b9bfff700 10 client.?.objecter ms_handle_connect 0x7f1bb40f8420 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1b9bfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba400e070 con 0x7f1bb40f8420 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba4005040 con 0x7f1bb40f8420 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.790+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ba4013da0 con 0x7f1bb40f8420 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.791+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4019070 con 0x7f1bb40f8420 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.791+0000 7f1bb92ab700 10 client.24707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.791+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb4108fa0 con 0x7f1bb40f8420 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.791+0000 7f1bb92ab700 1 librados: init done 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1bb4107060 con 0x7f1bb40f8420 2024-02-23T00:30:09.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1ba401e420 con 0x7f1bb40f8420 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1b9bfff700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b9c062c90 0x7f1b9c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(89..89 src has 38..89) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f1ba407d2e0 con 0x7f1bb40f8420 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 osd_map(89..89 src has 38..89) v4 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1b9bfff700 3 client.24707.objecter handle_osd_map got epochs [89,89] > 0 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1b9bfff700 3 client.24707.objecter handle_osd_map decoding full epoch 89 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1bb27fc700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b9c062c90 0x7f1b9c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1b9bfff700 20 client.24707.objecter dump_active .. 0 homeless 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.792+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v89) v1 ==== 54+0+16 (secure 0 0 0) 0x7f1ba4084070 con 0x7f1bb40f8420 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1b9bfff700 10 client.24707.objecter ms_handle_connect 0x7f1b9c062c90 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb27fc700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b9c062c90 0x7f1b9c065150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1bb41099b0 tx=0x7f1ba800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 1 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1bb40fee60 0x7f1bb40ff240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb41096f0 osd=0 3 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41096f0 osd=0 3 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb41096f0 osd=0 3 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 1 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb41096f0 osd=0 3 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 0 1 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41096f0 osd=0 4 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 10 client.24707.objecter _send_command 1 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb40ff730 con 0x7f1bb40fee60 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb37fe700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1bb40fee60 0x7f1bb40ff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1b9bfff700 10 client.24707.objecter ms_handle_connect 0x7f1bb40fee60 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.793+0000 7f1bb37fe700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1bb40fee60 0x7f1bb40ff240 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.794+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1bb40ff730 con 0x7f1bb40fee60 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.794+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.794+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 1 = system:0 2024-02-23T00:30:09.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.794+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb41096f0 osd=0 4 2024-02-23T00:30:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.794+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 0 1 2024-02-23T00:30:09.804 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:09.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 2 2024-02-23T00:30:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb41096f0 osd=0 3 2024-02-23T00:30:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41096f0 osd=0 3 2024-02-23T00:30:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb41096f0 osd=0 3 2024-02-23T00:30:09.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 2 2024-02-23T00:30:09.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb41096f0 osd=0 3 2024-02-23T00:30:09.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 0 2 2024-02-23T00:30:09.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41096f0 osd=0 4 2024-02-23T00:30:09.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 10 client.24707.objecter _send_command 2 2024-02-23T00:30:09.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.810+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1bb40ff8f0 con 0x7f1bb40fee60 2024-02-23T00:30:09.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1bb40ff8f0 con 0x7f1bb40fee60 2024-02-23T00:30:09.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:09.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb41096f0 osd=0 4 2024-02-23T00:30:09.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 0 2 2024-02-23T00:30:09.813 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 3 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6805/685354171 conn(0x7f1bb4028540 0x7f1bb4118ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb40f5ef0 osd=1 3 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40f5ef0 osd=1 3 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb40f5ef0 osd=1 3 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 3 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb40f5ef0 osd=1 3 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 1 3 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40f5ef0 osd=1 4 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 10 client.24707.objecter _send_command 3 2024-02-23T00:30:09.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb40f6060 con 0x7f1bb4028540 2024-02-23T00:30:09.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.812+0000 7f1bb2ffd700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6805/685354171 conn(0x7f1bb4028540 0x7f1bb4118ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.813+0000 7f1b9bfff700 10 client.24707.objecter ms_handle_connect 0x7f1bb4028540 2024-02-23T00:30:09.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.813+0000 7f1bb2ffd700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6805/685354171 conn(0x7f1bb4028540 0x7f1bb4118ec0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.813+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1bb40f6060 con 0x7f1bb4028540 2024-02-23T00:30:09.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.813+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:30:09.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.813+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 3 = system:0 2024-02-23T00:30:09.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.813+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb40f5ef0 osd=1 4 2024-02-23T00:30:09.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.813+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 1 3 2024-02-23T00:30:09.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:09.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 4 2024-02-23T00:30:09.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb40f5ef0 osd=1 3 2024-02-23T00:30:09.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40f5ef0 osd=1 3 2024-02-23T00:30:09.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb40f5ef0 osd=1 3 2024-02-23T00:30:09.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 4 2024-02-23T00:30:09.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb40f5ef0 osd=1 3 2024-02-23T00:30:09.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 1 4 2024-02-23T00:30:09.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40f5ef0 osd=1 4 2024-02-23T00:30:09.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 10 client.24707.objecter _send_command 4 2024-02-23T00:30:09.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.828+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1bb405a040 con 0x7f1bb4028540 2024-02-23T00:30:09.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.829+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1bb405a040 con 0x7f1bb4028540 2024-02-23T00:30:09.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.829+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:09.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.829+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.829+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb40f5ef0 osd=1 4 2024-02-23T00:30:09.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.829+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 1 4 2024-02-23T00:30:09.831 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:30:09.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:09.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 5 2024-02-23T00:30:09.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1bb40594d0 0x7f1bb411c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4059360 osd=2 3 2024-02-23T00:30:09.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4059360 osd=2 3 2024-02-23T00:30:09.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4059360 osd=2 3 2024-02-23T00:30:09.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 5 2024-02-23T00:30:09.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb4059360 osd=2 3 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 2 5 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4059360 osd=2 4 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 10 client.24707.objecter _send_command 5 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb40280a0 con 0x7f1bb40594d0 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb27fc700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1bb40594d0 0x7f1bb411c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1b9bfff700 10 client.24707.objecter ms_handle_connect 0x7f1bb40594d0 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.830+0000 7f1bb27fc700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1bb40594d0 0x7f1bb411c540 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.831+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1bb40280a0 con 0x7f1bb40594d0 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.831+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.831+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 5 = system:0 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.831+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb4059360 osd=2 4 2024-02-23T00:30:09.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.831+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 2 5 2024-02-23T00:30:09.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:09.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 6 2024-02-23T00:30:09.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4059360 osd=2 3 2024-02-23T00:30:09.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4059360 osd=2 3 2024-02-23T00:30:09.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4059360 osd=2 3 2024-02-23T00:30:09.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 6 2024-02-23T00:30:09.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb4059360 osd=2 3 2024-02-23T00:30:09.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 2 6 2024-02-23T00:30:09.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4059360 osd=2 4 2024-02-23T00:30:09.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 10 client.24707.objecter _send_command 6 2024-02-23T00:30:09.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.846+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1bb4028350 con 0x7f1bb40594d0 2024-02-23T00:30:09.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1bb4028350 con 0x7f1bb40594d0 2024-02-23T00:30:09.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:09.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb4059360 osd=2 4 2024-02-23T00:30:09.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 2 6 2024-02-23T00:30:09.849 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:30:09.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:09.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 7 2024-02-23T00:30:09.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6813/128059790 conn(0x7f1bb411a090 0x7f1bb411fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb40278f0 osd=3 3 2024-02-23T00:30:09.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40278f0 osd=3 3 2024-02-23T00:30:09.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb40278f0 osd=3 3 2024-02-23T00:30:09.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 7 2024-02-23T00:30:09.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb40278f0 osd=3 3 2024-02-23T00:30:09.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 3 7 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40278f0 osd=3 4 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 10 client.24707.objecter _send_command 7 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb4122840 con 0x7f1bb411a090 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb37fe700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6813/128059790 conn(0x7f1bb411a090 0x7f1bb411fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1b9bfff700 10 client.24707.objecter ms_handle_connect 0x7f1bb411a090 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.848+0000 7f1bb37fe700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6813/128059790 conn(0x7f1bb411a090 0x7f1bb411fbc0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.849+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1bb4122840 con 0x7f1bb411a090 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.849+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.849+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 7 = system:0 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.849+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb40278f0 osd=3 4 2024-02-23T00:30:09.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.849+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 3 7 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 8 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb40278f0 osd=3 3 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40278f0 osd=3 3 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb40278f0 osd=3 3 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 8 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb40278f0 osd=3 3 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 3 8 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40278f0 osd=3 4 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 10 client.24707.objecter _send_command 8 2024-02-23T00:30:09.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.864+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1bb41195f0 con 0x7f1bb411a090 2024-02-23T00:30:09.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1bb41195f0 con 0x7f1bb411a090 2024-02-23T00:30:09.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:09.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb40278f0 osd=3 4 2024-02-23T00:30:09.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 3 8 2024-02-23T00:30:09.867 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 9 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1bb41197a0 0x7f1bb4121230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb41195f0 osd=4 3 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41195f0 osd=4 3 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb41195f0 osd=4 3 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 9 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb41195f0 osd=4 3 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 4 9 2024-02-23T00:30:09.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41195f0 osd=4 4 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 10 client.24707.objecter _send_command 9 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.865+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb411d8d0 con 0x7f1bb41197a0 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.866+0000 7f1bb2ffd700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1bb41197a0 0x7f1bb4121230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.866+0000 7f1b9bfff700 10 client.24707.objecter ms_handle_connect 0x7f1bb41197a0 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.866+0000 7f1bb2ffd700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1bb41197a0 0x7f1bb4121230 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.867+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1bb411d8d0 con 0x7f1bb41197a0 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.867+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.867+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 9 = system:0 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.867+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb41195f0 osd=4 4 2024-02-23T00:30:09.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.867+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 4 9 2024-02-23T00:30:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 10 2024-02-23T00:30:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb41195f0 osd=4 3 2024-02-23T00:30:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41195f0 osd=4 3 2024-02-23T00:30:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb41195f0 osd=4 3 2024-02-23T00:30:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 10 2024-02-23T00:30:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb41195f0 osd=4 3 2024-02-23T00:30:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 4 10 2024-02-23T00:30:09.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41195f0 osd=4 4 2024-02-23T00:30:09.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 10 client.24707.objecter _send_command 10 2024-02-23T00:30:09.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.882+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1bb411d8d0 con 0x7f1bb41197a0 2024-02-23T00:30:09.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.883+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1bb411d8d0 con 0x7f1bb41197a0 2024-02-23T00:30:09.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.883+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:09.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.883+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.883+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb41195f0 osd=4 4 2024-02-23T00:30:09.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.883+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 4 10 2024-02-23T00:30:09.885 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:30:09.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:09.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 11 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1bb4121c30 0x7f1bb4122090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb411d8d0 osd=5 3 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb411d8d0 osd=5 3 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb411d8d0 osd=5 3 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 11 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb411d8d0 osd=5 3 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 5 11 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb411d8d0 osd=5 4 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 10 client.24707.objecter _send_command 11 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb4122580 con 0x7f1bb4121c30 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb27fc700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1bb4121c30 0x7f1bb4122090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1b9bfff700 10 client.24707.objecter ms_handle_connect 0x7f1bb4121c30 2024-02-23T00:30:09.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.884+0000 7f1bb27fc700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1bb4121c30 0x7f1bb4122090 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.885+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1bb4122580 con 0x7f1bb4121c30 2024-02-23T00:30:09.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.885+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:30:09.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.885+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 11 = system:0 2024-02-23T00:30:09.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.885+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb411d8d0 osd=5 4 2024-02-23T00:30:09.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.885+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 5 11 2024-02-23T00:30:09.907 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.910+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.910+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 12 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.911+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb411d8d0 osd=5 3 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.911+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb411d8d0 osd=5 3 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.911+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb411d8d0 osd=5 3 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.911+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 12 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.911+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb411d8d0 osd=5 3 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.911+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 5 12 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.911+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb411d8d0 osd=5 4 2024-02-23T00:30:09.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.911+0000 7f1bb92ab700 10 client.24707.objecter _send_command 12 2024-02-23T00:30:09.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.911+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1bb4122580 con 0x7f1bb4121c30 2024-02-23T00:30:09.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.912+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1bb4122580 con 0x7f1bb4121c30 2024-02-23T00:30:09.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.912+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:09.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.912+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.912+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb411d8d0 osd=5 4 2024-02-23T00:30:09.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.912+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 5 12 2024-02-23T00:30:09.915 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:30:09.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:09.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 13 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1bb4124f80 0x7f1bb41253e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4122580 osd=6 3 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4122580 osd=6 3 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4122580 osd=6 3 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 13 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb4122580 osd=6 3 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 6 13 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4122580 osd=6 4 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 10 client.24707.objecter _send_command 13 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb4126f90 con 0x7f1bb4124f80 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.913+0000 7f1bb37fe700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1bb4124f80 0x7f1bb41253e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.914+0000 7f1b9bfff700 10 client.24707.objecter ms_handle_connect 0x7f1bb4124f80 2024-02-23T00:30:09.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.914+0000 7f1bb37fe700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1bb4124f80 0x7f1bb41253e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.914+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1bb4126f90 con 0x7f1bb4124f80 2024-02-23T00:30:09.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.914+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:30:09.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.914+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 13 = system:0 2024-02-23T00:30:09.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.914+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb4122580 osd=6 4 2024-02-23T00:30:09.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.914+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 6 13 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 14 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4122580 osd=6 3 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4122580 osd=6 3 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4122580 osd=6 3 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 14 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb4122580 osd=6 3 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 6 14 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4122580 osd=6 4 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 10 client.24707.objecter _send_command 14 2024-02-23T00:30:09.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.931+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1bb4126f90 con 0x7f1bb4124f80 2024-02-23T00:30:09.934 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:09.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1bb4126f90 con 0x7f1bb4124f80 2024-02-23T00:30:09.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:09.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb4122580 osd=6 4 2024-02-23T00:30:09.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 6 14 2024-02-23T00:30:09.935 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:30:09.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:09.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 15 2024-02-23T00:30:09.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6812/373787000 conn(0x7f1bb412db00 0x7f1bb412df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:09.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4126f90 osd=7 3 2024-02-23T00:30:09.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4126f90 osd=7 3 2024-02-23T00:30:09.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4126f90 osd=7 3 2024-02-23T00:30:09.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 15 2024-02-23T00:30:09.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb4126f90 osd=7 3 2024-02-23T00:30:09.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 7 15 2024-02-23T00:30:09.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4126f90 osd=7 4 2024-02-23T00:30:09.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 10 client.24707.objecter _send_command 15 2024-02-23T00:30:09.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb412e560 con 0x7f1bb412db00 2024-02-23T00:30:09.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.933+0000 7f1bb2ffd700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6812/373787000 conn(0x7f1bb412db00 0x7f1bb412df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:09.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.934+0000 7f1b9bfff700 10 client.24707.objecter ms_handle_connect 0x7f1bb412db00 2024-02-23T00:30:09.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.934+0000 7f1bb2ffd700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6812/373787000 conn(0x7f1bb412db00 0x7f1bb412df40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:09.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.935+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1bb412e560 con 0x7f1bb412db00 2024-02-23T00:30:09.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.935+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:30:09.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.935+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 15 = system:0 2024-02-23T00:30:09.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.935+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb4126f90 osd=7 4 2024-02-23T00:30:09.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.935+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 7 15 2024-02-23T00:30:09.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 10 client.24707.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:09.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign -1 16 2024-02-23T00:30:09.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4126f90 osd=7 3 2024-02-23T00:30:09.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4126f90 osd=7 3 2024-02-23T00:30:09.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 20 client.24707.objecter _get_session s=0x7f1bb4126f90 osd=7 3 2024-02-23T00:30:09.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_remove -1 16 2024-02-23T00:30:09.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 20 client.24707.objecter get_session s=0x7f1bb4126f90 osd=7 3 2024-02-23T00:30:09.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 15 client.24707.objecter _session_command_op_assign 7 16 2024-02-23T00:30:09.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4126f90 osd=7 4 2024-02-23T00:30:09.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 10 client.24707.objecter _send_command 16 2024-02-23T00:30:09.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.951+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1bb412e560 con 0x7f1bb412db00 2024-02-23T00:30:09.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1b9bfff700 1 -- 172.21.15.87:0/1178975245 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1bb412e560 con 0x7f1bb412db00 2024-02-23T00:30:09.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1b9bfff700 10 client.24707.objecter ms_dispatch 0x7f1bb4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:09.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1b9bfff700 10 client.24707.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1b9bfff700 20 client.24707.objecter put_session s=0x7f1bb4126f90 osd=7 4 2024-02-23T00:30:09.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1b9bfff700 15 client.24707.objecter _session_command_op_remove 7 16 2024-02-23T00:30:09.955 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:30:09.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 librados: watch_flush enter 2024-02-23T00:30:09.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 librados: watch_flush exit 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 client.24707.objecter close_session for osd.0 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1bb40fee60 msgr2=0x7f1bb40ff240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1bb40fee60 0x7f1bb40ff240 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41096f0 osd=0 1 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 client.24707.objecter close_session for osd.1 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6805/685354171 conn(0x7f1bb4028540 msgr2=0x7f1bb4118ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6805/685354171 conn(0x7f1bb4028540 0x7f1bb4118ec0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40f5ef0 osd=1 1 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 client.24707.objecter close_session for osd.2 2024-02-23T00:30:09.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1bb40594d0 msgr2=0x7f1bb411c540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1bb40594d0 0x7f1bb411c540 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4059360 osd=2 1 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 client.24707.objecter close_session for osd.3 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6813/128059790 conn(0x7f1bb411a090 msgr2=0x7f1bb411fbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb37fe700 1 -- 172.21.15.87:0/1178975245 reap_dead start 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6813/128059790 conn(0x7f1bb411a090 0x7f1bb411fbc0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb40278f0 osd=3 1 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 client.24707.objecter close_session for osd.4 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1bb41197a0 msgr2=0x7f1bb4121230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1bb41197a0 0x7f1bb4121230 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb41195f0 osd=4 1 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 client.24707.objecter close_session for osd.5 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1bb4121c30 msgr2=0x7f1bb4122090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1bb4121c30 0x7f1bb4122090 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb411d8d0 osd=5 1 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 client.24707.objecter close_session for osd.6 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1bb4124f80 msgr2=0x7f1bb41253e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1bb4124f80 0x7f1bb41253e0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4122580 osd=6 1 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 client.24707.objecter close_session for osd.7 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6812/373787000 conn(0x7f1bb412db00 msgr2=0x7f1bb412df40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.171:6812/373787000 conn(0x7f1bb412db00 0x7f1bb412df40 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 20 client.24707.objecter put_session s=0x7f1bb4126f90 osd=7 1 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 20 client.24707.objecter shutdown clearing up homeless session... 2024-02-23T00:30:09.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 10 client.24707.objecter successfully canceled tick 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb37fe700 1 -- 172.21.15.87:0/1178975245 reap_dead start 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b9c062c90 msgr2=0x7f1b9c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b9c062c90 0x7f1b9c065150 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1bb41099b0 tx=0x7f1ba800b040).stop 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1bb40f8420 msgr2=0x7f1bb410e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.953+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1bb40f8420 0x7f1bb410e070 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1ba4004870 tx=0x7f1ba400b040).stop 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.954+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 shutdown_connections 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.954+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b9c062c90 0x7f1b9c065150 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.954+0000 7f1bb92ab700 1 --2- 172.21.15.87:0/1178975245 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1bb40f8420 0x7f1bb410e070 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.954+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 >> 172.21.15.87:0/1178975245 conn(0x7f1bb40f3800 msgr2=0x7f1bb40f5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.954+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 shutdown_connections 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.954+0000 7f1bb92ab700 1 -- 172.21.15.87:0/1178975245 wait complete. 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:09.954+0000 7f1bb92ab700 1 librados: shutdown 2024-02-23T00:30:09.960 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.961 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.961 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.961 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.961 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.961 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.961 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:30:09.961 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:30:10.086 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:10.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:10.190 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:10.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:08.915924+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:10.196 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:08.915984+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:10.196 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:09.623901+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:10.196 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:09.792843+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? 172.21.15.87:0/1178975245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:10.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:08.915924+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:10.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:08.915984+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:10.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:09.623901+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:10.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:09.792843+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? 172.21.15.87:0/1178975245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:10.215 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:08.915924+0000 osd.5 (osd.5) 45 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:10.215 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:08.915984+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:10.215 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:09.623901+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:10.215 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:09.792843+0000 mon.c (mon.2) 27 : audit [DBG] from='client.? 172.21.15.87:0/1178975245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:10.216 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:10.413 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:10.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:10.519 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:10.545 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:10.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:30:10.895 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:30:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:30:11.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:09.490829+0000 mgr.y (mgr.24437) 118 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:30:11.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:09.895243+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-23T00:30:11.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:09.490829+0000 mgr.y (mgr.24437) 118 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:30:11.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:09.895243+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-23T00:30:11.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:09.490829+0000 mgr.y (mgr.24437) 118 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:30:11.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:09.895243+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-23T00:30:12.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:10.588937+0000 mgr.y (mgr.24437) 119 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:12.223 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:10.588937+0000 mgr.y (mgr.24437) 119 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:12.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:10.588937+0000 mgr.y (mgr.24437) 119 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:12.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:30:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:12] "GET /metrics HTTP/1.1" 200 172257 "" "Prometheus/2.33.4" 2024-02-23T00:30:13.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:11.491435+0000 mgr.y (mgr.24437) 120 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:30:13.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:11.491435+0000 mgr.y (mgr.24437) 120 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:30:13.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:11.491435+0000 mgr.y (mgr.24437) 120 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:30:14.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:30:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:30:15.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:13.491809+0000 mgr.y (mgr.24437) 121 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:30:15.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:13.491809+0000 mgr.y (mgr.24437) 121 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:30:15.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:13.491809+0000 mgr.y (mgr.24437) 121 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:30:16.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-23T00:30:16.498 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:16.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:16.549 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:16.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:16.634 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:16.936 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:16.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:16.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:17.043 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:17.076 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:17.097 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:17.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:17.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:15.492535+0000 mgr.y (mgr.24437) 122 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:17.201 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:17.231 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:17.251 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:17.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:17.330 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:15.492535+0000 mgr.y (mgr.24437) 122 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:17.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:15.492535+0000 mgr.y (mgr.24437) 122 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:17.355 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:17.383 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:17.402 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:17.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:17.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:17.505 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:17.532 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:17.551 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:17.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:17.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:17.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:30:17.686 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:17.948 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:17.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:17.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:17.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.982+0000 7f0330914700 1 -- 172.21.15.87:0/1815755261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f032c0fe530 msgr2=0x7f032c0f63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:17.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.982+0000 7f0330914700 1 --2- 172.21.15.87:0/1815755261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f032c0fe530 0x7f032c0f63b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0320004870 tx=0x7f0320009cb0).stop 2024-02-23T00:30:17.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.982+0000 7f0330914700 1 -- 172.21.15.87:0/1815755261 shutdown_connections 2024-02-23T00:30:17.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.982+0000 7f0330914700 1 --2- 172.21.15.87:0/1815755261 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f032c0f6a90 0x7f032c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:30:17.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.982+0000 7f0330914700 1 --2- 172.21.15.87:0/1815755261 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f032c0fdbd0 0x7f032c0fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:30:17.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.982+0000 7f0330914700 1 --2- 172.21.15.87:0/1815755261 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f032c0fe530 0x7f032c0f63b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:17.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.982+0000 7f0330914700 1 -- 172.21.15.87:0/1815755261 >> 172.21.15.87:0/1815755261 conn(0x7f032c06c370 msgr2=0x7f032c06c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:17.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.983+0000 7f0330914700 1 -- 172.21.15.87:0/1815755261 shutdown_connections 2024-02-23T00:30:17.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.983+0000 7f0330914700 1 -- 172.21.15.87:0/1815755261 wait complete. 2024-02-23T00:30:17.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.983+0000 7f0330914700 1 librados: starting msgr at 2024-02-23T00:30:17.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.983+0000 7f0330914700 1 librados: starting objecter 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.983+0000 7f0330914700 1 Processor -- start 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.983+0000 7f0330914700 1 -- start start 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.983+0000 7f0330914700 1 librados: setting wanted keys 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.983+0000 7f0330914700 1 librados: calling monclient init 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f0330914700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f032c0f6a90 0x7f032c103a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f0330914700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f032c0fdbd0 0x7f032c103f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f0330914700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f032c0fe530 0x7f032c1230c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f0330914700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c104730 con 0x7f032c0f6a90 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f0330914700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c1235b0 con 0x7f032c0fdbd0 2024-02-23T00:30:17.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f0330914700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032c123770 con 0x7f032c0fe530 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f032a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f032c0f6a90 0x7f032c103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f032ad9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f032c0fe530 0x7f032c1230c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f032a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f032c0f6a90 0x7f032c103a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36680/0 (socket says 172.21.15.87:36680) 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f032a59c700 1 -- 172.21.15.87:0/1462762390 learned_addr learned my addr 172.21.15.87:0/1462762390 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f0329d9b700 1 --2- 172.21.15.87:0/1462762390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f032c0fdbd0 0x7f032c103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f032a59c700 1 -- 172.21.15.87:0/1462762390 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f032c0fe530 msgr2=0x7f032c1230c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f032a59c700 1 --2- 172.21.15.87:0/1462762390 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f032c0fe530 0x7f032c1230c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f032a59c700 1 -- 172.21.15.87:0/1462762390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f032c0fdbd0 msgr2=0x7f032c103f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f032a59c700 1 --2- 172.21.15.87:0/1462762390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f032c0fdbd0 0x7f032c103f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:17.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.984+0000 7f032a59c700 1 -- 172.21.15.87:0/1462762390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0320005040 con 0x7f032c0f6a90 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f032a59c700 1 --2- 172.21.15.87:0/1462762390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f032c0f6a90 0x7f032c103a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f031c00b530 tx=0x7f031c0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f031b7fe700 10 client.?.objecter ms_handle_connect 0x7f032c0f6a90 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f031b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f031b7fe700 1 -- 172.21.15.87:0/1462762390 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031c002320 con 0x7f032c0f6a90 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c1239f0 con 0x7f032c0f6a90 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f0330914700 10 client.14784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c123f30 con 0x7f032c0f6a90 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f0330914700 1 librados: init done 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f031b7fe700 1 -- 172.21.15.87:0/1462762390 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f031c002950 con 0x7f032c0f6a90 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.985+0000 7f031b7fe700 1 -- 172.21.15.87:0/1462762390 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031c012950 con 0x7f032c0f6a90 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.986+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c004f50 con 0x7f032c0f6a90 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.986+0000 7f031b7fe700 1 -- 172.21.15.87:0/1462762390 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f031c002ac0 con 0x7f032c0f6a90 2024-02-23T00:30:17.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.987+0000 7f031b7fe700 1 --2- 172.21.15.87:0/1462762390 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0314062c90 0x7f0314065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:17.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.989+0000 7f031b7fe700 1 -- 172.21.15.87:0/1462762390 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(90..90 src has 38..90) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f031c009160 con 0x7f032c0f6a90 2024-02-23T00:30:17.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.989+0000 7f031b7fe700 10 client.14784.objecter ms_dispatch 0x7f032c001550 osd_map(90..90 src has 38..90) v4 2024-02-23T00:30:17.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.989+0000 7f031b7fe700 3 client.14784.objecter handle_osd_map got epochs [90,90] > 0 2024-02-23T00:30:17.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.989+0000 7f031b7fe700 3 client.14784.objecter handle_osd_map decoding full epoch 90 2024-02-23T00:30:17.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.989+0000 7f031b7fe700 20 client.14784.objecter dump_active .. 0 homeless 2024-02-23T00:30:17.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.991+0000 7f031b7fe700 1 -- 172.21.15.87:0/1462762390 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f031c081060 con 0x7f032c0f6a90 2024-02-23T00:30:17.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.996+0000 7f0329d9b700 1 --2- 172.21.15.87:0/1462762390 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0314062c90 0x7f0314065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:17.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.996+0000 7f0329d9b700 1 --2- 172.21.15.87:0/1462762390 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0314062c90 0x7f0314065150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f032c107b50 tx=0x7f0320007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:17.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:17.996+0000 7f031b7fe700 10 client.14784.objecter ms_handle_connect 0x7f0314062c90 2024-02-23T00:30:18.060 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:18.089 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:18.109 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:18.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:18.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.162+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}) v1 -- 0x7f030c000bf0 con 0x7f0314062c90 2024-02-23T00:30:18.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.163+0000 7f031b7fe700 1 -- 172.21.15.87:0/1462762390 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+293 (secure 0 0 0) 0x7f030c000bf0 con 0x7f0314062c90 2024-02-23T00:30:18.165 INFO:teuthology.orchestra.run.smithi087.stdout:moved 0 / 396 (0%) 2024-02-23T00:30:18.165 INFO:teuthology.orchestra.run.smithi087.stdout:avg 49.5 2024-02-23T00:30:18.165 INFO:teuthology.orchestra.run.smithi087.stdout:stddev 5.22015 -> 5.22015 (expected baseline 6.58122) 2024-02-23T00:30:18.165 INFO:teuthology.orchestra.run.smithi087.stdout:min osd.2 with 40 -> 40 pgs (0.808081 -> 0.808081 * mean) 2024-02-23T00:30:18.165 INFO:teuthology.orchestra.run.smithi087.stdout:max osd.7 with 56 -> 56 pgs (1.13131 -> 1.13131 * mean) 2024-02-23T00:30:18.165 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:30:18.165 INFO:teuthology.orchestra.run.smithi087.stdout:oload 110 2024-02-23T00:30:18.165 INFO:teuthology.orchestra.run.smithi087.stdout:max_change 1 2024-02-23T00:30:18.166 INFO:teuthology.orchestra.run.smithi087.stdout:max_change_osds 4 2024-02-23T00:30:18.166 INFO:teuthology.orchestra.run.smithi087.stdout:average_utilization 0.0032 2024-02-23T00:30:18.166 INFO:teuthology.orchestra.run.smithi087.stdout:overload_utilization 0.0035 2024-02-23T00:30:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 10 librados: watch_flush enter 2024-02-23T00:30:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 10 librados: watch_flush exit 2024-02-23T00:30:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 20 client.14784.objecter shutdown clearing up homeless session... 2024-02-23T00:30:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 10 client.14784.objecter successfully canceled tick 2024-02-23T00:30:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0314062c90 msgr2=0x7f0314065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 --2- 172.21.15.87:0/1462762390 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0314062c90 0x7f0314065150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f032c107b50 tx=0x7f0320007f90).stop 2024-02-23T00:30:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f032c0f6a90 msgr2=0x7f032c103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 --2- 172.21.15.87:0/1462762390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f032c0f6a90 0x7f032c103a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f031c00b530 tx=0x7f031c0090f0).stop 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 shutdown_connections 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 --2- 172.21.15.87:0/1462762390 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0314062c90 0x7f0314065150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 --2- 172.21.15.87:0/1462762390 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f032c0fe530 0x7f032c1230c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 --2- 172.21.15.87:0/1462762390 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f032c0f6a90 0x7f032c103a70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 --2- 172.21.15.87:0/1462762390 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f032c0fdbd0 0x7f032c103f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 >> 172.21.15.87:0/1462762390 conn(0x7f032c06c370 msgr2=0x7f032c0f9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 shutdown_connections 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 -- 172.21.15.87:0/1462762390 wait complete. 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:18.165+0000 7f0330914700 1 librados: shutdown 2024-02-23T00:30:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:no change 2024-02-23T00:30:18.212 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:18.239 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:18.259 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:18.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:18.290 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:18.362 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:18.391 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:18.408 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:18.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:18.512 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:18.547 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:18.747 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:18.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:18.862 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:18.891 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:18.910 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:18.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:19.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:19.047 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:17.492914+0000 mgr.y (mgr.24437) 123 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:19.047 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:18.164318+0000 mgr.y (mgr.24437) 124 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:19.047 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:17.492914+0000 mgr.y (mgr.24437) 123 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:19.047 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:18.164318+0000 mgr.y (mgr.24437) 124 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:19.050 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:19.086 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:19.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:19.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:17.492914+0000 mgr.y (mgr.24437) 123 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:19.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:18.164318+0000 mgr.y (mgr.24437) 124 : audit [DBG] from='client.14784 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:19.193 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:19.222 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:19.242 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:19.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:19.346 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:19.383 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:19.443 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:19.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:19.550 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:19.580 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:19.601 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:19.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:19.704 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:19.733 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:19.753 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:19.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:19.857 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:19.884 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:19.903 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:19.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:20.007 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:20.034 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:20.053 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:20.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:20.157 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:20.184 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:20.203 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:20.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:20.306 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:20.333 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:20.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.347+0000 7f74e0bf2700 1 -- 172.21.15.87:0/494435505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74dc0fe510 msgr2=0x7f74dc0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:20.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.347+0000 7f74e0bf2700 1 --2- 172.21.15.87:0/494435505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74dc0fe510 0x7f74dc0f7d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f74d00047b0 tx=0x7f74d0009cb0).stop 2024-02-23T00:30:20.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.348+0000 7f74e0bf2700 1 -- 172.21.15.87:0/494435505 shutdown_connections 2024-02-23T00:30:20.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.348+0000 7f74e0bf2700 1 --2- 172.21.15.87:0/494435505 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74dc0f8420 0x7f74dc104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:20.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.348+0000 7f74e0bf2700 1 --2- 172.21.15.87:0/494435505 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74dc0fdbb0 0x7f74dc0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:20.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.348+0000 7f74e0bf2700 1 --2- 172.21.15.87:0/494435505 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74dc0fe510 0x7f74dc0f7d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:20.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.348+0000 7f74e0bf2700 1 -- 172.21.15.87:0/494435505 >> 172.21.15.87:0/494435505 conn(0x7f74dc0f3800 msgr2=0x7f74dc0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:20.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.348+0000 7f74e0bf2700 1 -- 172.21.15.87:0/494435505 shutdown_connections 2024-02-23T00:30:20.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.348+0000 7f74e0bf2700 1 -- 172.21.15.87:0/494435505 wait complete. 2024-02-23T00:30:20.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 librados: starting msgr at 2024-02-23T00:30:20.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 librados: starting objecter 2024-02-23T00:30:20.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 Processor -- start 2024-02-23T00:30:20.352 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:20.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 -- start start 2024-02-23T00:30:20.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 librados: setting wanted keys 2024-02-23T00:30:20.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 librados: calling monclient init 2024-02-23T00:30:20.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74dc0f8420 0x7f74dc067fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74dc0fdbb0 0x7f74dc0684d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74dc0fe510 0x7f74dc0631e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc063760 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc0638d0 con 0x7f74dc0fe510 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.349+0000 7f74e0bf2700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc063a90 con 0x7f74dc0f8420 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74dad9d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74dc0fe510 0x7f74dc0631e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74dad9d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74dc0fe510 0x7f74dc0631e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.87:52382/0 (socket says 172.21.15.87:52382) 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74dad9d700 1 -- 172.21.15.87:0/1130772623 learned_addr learned my addr 172.21.15.87:0/1130772623 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74d9d9b700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74dc0fdbb0 0x7f74dc0684d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74da59c700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74dc0f8420 0x7f74dc067fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74d9d9b700 1 -- 172.21.15.87:0/1130772623 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74dc0f8420 msgr2=0x7f74dc067fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74d9d9b700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74dc0f8420 0x7f74dc067fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:20.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74d9d9b700 1 -- 172.21.15.87:0/1130772623 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74dc0fe510 msgr2=0x7f74dc0631e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:20.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74d9d9b700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74dc0fe510 0x7f74dc0631e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:20.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74d9d9b700 1 -- 172.21.15.87:0/1130772623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d0005040 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74d9d9b700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74dc0fdbb0 0x7f74dc0684d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f74d0004650 tx=0x7f74d0007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:20.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74cb7fe700 10 client.?.objecter ms_handle_connect 0x7f74dc0fdbb0 2024-02-23T00:30:20.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74cb7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:20.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74cb7fe700 1 -- 172.21.15.87:0/1130772623 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d0018070 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74e0bf2700 1 -- 172.21.15.87:0/1130772623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc063d10 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:20.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74cb7fe700 1 -- 172.21.15.87:0/1130772623 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74d00093b0 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74e0bf2700 10 client.14790.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:20.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74cb7fe700 1 -- 172.21.15.87:0/1130772623 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d001ddb0 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74e0bf2700 1 -- 172.21.15.87:0/1130772623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc06bdc0 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74e0bf2700 1 librados: init done 2024-02-23T00:30:20.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.350+0000 7f74dad9d700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74dc0fe510 0x7f74dc0631e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:30:20.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.352+0000 7f74c97fa700 1 -- 172.21.15.87:0/1130772623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c0004f20 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:20.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.354+0000 7f74cb7fe700 1 -- 172.21.15.87:0/1130772623 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f74d001d420 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.354+0000 7f74cb7fe700 1 --2- 172.21.15.87:0/1130772623 >> v2:172.21.15.87:6800/1231597463 conn(0x7f74c4062c90 0x7f74c4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:20.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.354+0000 7f74da59c700 1 --2- 172.21.15.87:0/1130772623 >> v2:172.21.15.87:6800/1231597463 conn(0x7f74c4062c90 0x7f74c4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:20.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.355+0000 7f74cb7fe700 10 client.14790.objecter ms_handle_connect 0x7f74c4062c90 2024-02-23T00:30:20.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.355+0000 7f74da59c700 1 --2- 172.21.15.87:0/1130772623 >> v2:172.21.15.87:6800/1231597463 conn(0x7f74c4062c90 0x7f74c4065150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f74cc009a00 tx=0x7f74cc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:20.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.360+0000 7f74cb7fe700 1 -- 172.21.15.87:0/1130772623 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(90..90 src has 38..90) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f74d0052230 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.360+0000 7f74cb7fe700 10 client.14790.objecter ms_dispatch 0x7f74dc001550 osd_map(90..90 src has 38..90) v4 2024-02-23T00:30:20.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.360+0000 7f74cb7fe700 3 client.14790.objecter handle_osd_map got epochs [90,90] > 0 2024-02-23T00:30:20.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.360+0000 7f74cb7fe700 3 client.14790.objecter handle_osd_map decoding full epoch 90 2024-02-23T00:30:20.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.360+0000 7f74cb7fe700 20 client.14790.objecter dump_active .. 0 homeless 2024-02-23T00:30:20.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.361+0000 7f74cb7fe700 1 -- 172.21.15.87:0/1130772623 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f74d0052000 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:20.491 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:20.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.565+0000 7f74c97fa700 1 -- 172.21.15.87:0/1130772623 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f74c00054a0 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.737 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-23T00:30:20.737 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-02-23T00:30:20.757 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:20.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:20.862 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:20.897 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:20.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:30:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:30:20.949 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.947+0000 7f74cb7fe700 1 -- 172.21.15.87:0/1130772623 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v91) v1 ==== 91+0+0 (secure 0 0 0) 0x7f74d0051bf0 con 0x7f74dc0fdbb0 2024-02-23T00:30:20.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 10 librados: watch_flush enter 2024-02-23T00:30:20.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 10 librados: watch_flush exit 2024-02-23T00:30:20.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 20 client.14790.objecter shutdown clearing up homeless session... 2024-02-23T00:30:20.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 10 client.14790.objecter successfully canceled tick 2024-02-23T00:30:20.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 -- 172.21.15.87:0/1130772623 >> v2:172.21.15.87:6800/1231597463 conn(0x7f74c4062c90 msgr2=0x7f74c4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:20.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 --2- 172.21.15.87:0/1130772623 >> v2:172.21.15.87:6800/1231597463 conn(0x7f74c4062c90 0x7f74c4065150 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f74cc009a00 tx=0x7f74cc00b040).stop 2024-02-23T00:30:20.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 -- 172.21.15.87:0/1130772623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74dc0fdbb0 msgr2=0x7f74dc0684d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:20.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74dc0fdbb0 0x7f74dc0684d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f74d0004650 tx=0x7f74d0007f90).stop 2024-02-23T00:30:20.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 -- 172.21.15.87:0/1130772623 shutdown_connections 2024-02-23T00:30:20.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 --2- 172.21.15.87:0/1130772623 >> v2:172.21.15.87:6800/1231597463 conn(0x7f74c4062c90 0x7f74c4065150 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:20.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f74dc0fe510 0x7f74dc0631e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:20.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f74dc0f8420 0x7f74dc067fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:20.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 --2- 172.21.15.87:0/1130772623 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f74dc0fdbb0 0x7f74dc0684d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:20.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 -- 172.21.15.87:0/1130772623 >> 172.21.15.87:0/1130772623 conn(0x7f74dc0f3800 msgr2=0x7f74dc0fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:20.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 -- 172.21.15.87:0/1130772623 shutdown_connections 2024-02-23T00:30:20.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.956+0000 7f74c97fa700 1 -- 172.21.15.87:0/1130772623 wait complete. 2024-02-23T00:30:20.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:20.957+0000 7f74c97fa700 1 librados: shutdown 2024-02-23T00:30:20.962 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:30:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:19.493269+0000 mgr.y (mgr.24437) 125 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:20.566838+0000 mon.a (mon.0) 755 : audit [INF] from='client.? 172.21.15.87:0/1130772623' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:30:21.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:19.493269+0000 mgr.y (mgr.24437) 125 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:21.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:20.566838+0000 mon.a (mon.0) 755 : audit [INF] from='client.? 172.21.15.87:0/1130772623' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:30:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:19.493269+0000 mgr.y (mgr.24437) 125 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:20.566838+0000 mon.a (mon.0) 755 : audit [INF] from='client.? 172.21.15.87:0/1130772623' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:30:21.658 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:21.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:21.763 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:21.796 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:21.818 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:21.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:21.881 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:21.922 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:21.952 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:20.575179+0000 osd.0 (osd.0) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:20.575253+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:20.599221+0000 mgr.y (mgr.24437) 126 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:20.942730+0000 mon.a (mon.0) 756 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:20.947819+0000 mon.a (mon.0) 757 : audit [INF] from='client.? 172.21.15.87:0/1130772623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:30:22.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:20.947871+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-23T00:30:22.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:20.575179+0000 osd.0 (osd.0) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:20.575253+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:20.599221+0000 mgr.y (mgr.24437) 126 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:20.942730+0000 mon.a (mon.0) 756 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:20.947819+0000 mon.a (mon.0) 757 : audit [INF] from='client.? 172.21.15.87:0/1130772623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:30:22.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:20.947871+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-23T00:30:22.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:20.575179+0000 osd.0 (osd.0) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:22.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:20.575253+0000 osd.0 (osd.0) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:22.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:20.599221+0000 mgr.y (mgr.24437) 126 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:22.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:20.942730+0000 mon.a (mon.0) 756 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:22.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:20.947819+0000 mon.a (mon.0) 757 : audit [INF] from='client.? 172.21.15.87:0/1130772623' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:30:22.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:20.947871+0000 mon.a (mon.0) 758 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-23T00:30:22.412 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:22.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:22.517 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:22.549 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:22.570 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:22.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:22.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:30:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:22] "GET /metrics HTTP/1.1" 200 172257 "" "Prometheus/2.33.4" 2024-02-23T00:30:22.663 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:30:22.663 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:30:22.663 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:30:22.663 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:30:22.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.663 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24344.0:64 5.3 5:c609908c:::notify.5:head [watch watch cookie 94727417080272] snapc 0=[] ondisk+write+known_if_redirected e59)", 2024-02-23T00:30:22.663 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:21.038127+0000", 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 181.62014123399999, 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00183734, 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24344", 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2178799724", 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 64 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.038127+0000", 2024-02-23T00:30:22.664 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.038127+0000", 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.038126+0000", 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8599999999999998e-06 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.038132+0000", 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0832000000000001e-05 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.038143+0000", 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2799999999999998e-06 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.038150+0000", 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1501999999999997e-05 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.666 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.038191+0000", 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5480000000000002e-05 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for rw locks", 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.038267+0000", 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00070825100000000002 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.038975+0000", 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1179999999999994e-05 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.667 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039036+0000", 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016228600000000001 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 5,6", 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039199+0000", 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00034765399999999998 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039546+0000", 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025281300000000001 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039799+0000", 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0222000000000002e-05 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039839+0000", 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0602000000000001e-05 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039890+0000", 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4379000000000001e-05 2024-02-23T00:30:22.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039964+0000", 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00183734 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:102 5.3 5:c609908c:::notify.5:head [watch watch cookie 94532396652160] snapc 0=[] ondisk+write+known_if_redirected e59)", 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:21.039073+0000", 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 181.61919499199999, 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001901916, 2024-02-23T00:30:22.670 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 102 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039073+0000", 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039073+0000", 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039072+0000", 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0600000000000002e-06 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039079+0000", 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1953000000000001e-05 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.672 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039091+0000", 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8109999999999997e-06 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039098+0000", 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000346977 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039445+0000", 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3910000000000001e-05 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for rw locks", 2024-02-23T00:30:22.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039529+0000", 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046140999999999999 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.039990+0000", 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1814000000000002e-05 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.040042+0000", 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00010019800000000001 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 5,6", 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.040142+0000", 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037985500000000001 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.040522+0000", 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000223167 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.040745+0000", 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000144229 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.040889+0000", 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.9626999999999999e-05 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.040919+0000", 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6196999999999997e-05 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:21.040975+0000", 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001901916 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.676 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:4 4.b 4:d242335b:gc::gc.2:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e61)", 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.027823+0000", 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.63044561699999, 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020889189999999998, 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.677 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 4 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.027823+0000", 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.027823+0000", 2024-02-23T00:30:22.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.027820+0000", 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1661000000000001e-05 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.027832+0000", 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5965999999999999e-05 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.027848+0000", 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7639999999999994e-06 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.027856+0000", 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8090999999999994e-05 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.027944+0000", 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013339600000000001 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.680 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.028078+0000", 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00032967100000000002 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.028408+0000", 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00079672499999999997 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.029204+0000", 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00014231500000000001 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.029347+0000", 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046086899999999998 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.029807+0000", 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1316999999999996e-05 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.029869+0000", 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2753000000000002e-05 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.029912+0000", 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020889189999999998 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:96 4.16 4:6b0d4d95:lc::lc.20:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.091315+0000", 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.56695294400001, 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020150620000000002, 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.683 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 96 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.091315+0000", 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.091315+0000", 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:30:22.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.091313+0000", 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0468999999999999e-05 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.091324+0000", 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2565000000000001e-05 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.091336+0000", 2024-02-23T00:30:22.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.225e-05 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.091349+0000", 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6675000000000003e-05 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.091395+0000", 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000123934 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.091519+0000", 2024-02-23T00:30:22.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00020446799999999999 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.091724+0000", 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00075583099999999999 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.092480+0000", 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6869000000000001e-05 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.092497+0000", 2024-02-23T00:30:22.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00076390699999999998 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.093260+0000", 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4552000000000003e-05 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.093305+0000", 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5361999999999999e-05 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.688 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.093330+0000", 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020150620000000002 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:118 4.3 4:c31f41d7:gc::gc.10:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.103306+0000", 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.554961827, 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0017157310000000001, 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.689 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 118 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.103306+0000", 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.103306+0000", 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.103305+0000", 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0627e-05 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.103315+0000", 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8542999999999998e-05 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.103334+0000", 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9890000000000001e-05 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.103354+0000", 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.282e-05 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.103396+0000", 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1118999999999995e-05 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.103458+0000", 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000128577 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 5,7", 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.103586+0000", 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0012451090000000001 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.104831+0000", 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6934000000000001e-05 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.104848+0000", 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8722999999999997e-05 2024-02-23T00:30:22.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.104937+0000", 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8547999999999997e-05 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.104985+0000", 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.6690000000000003e-05 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.105022+0000", 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0017157310000000001 2024-02-23T00:30:22.694 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:170 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.116262+0000", 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.542006499, 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027659260000000001, 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.695 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 170 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116262+0000", 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116262+0000", 2024-02-23T00:30:22.696 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116261+0000", 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.451e-06 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116266+0000", 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0998999999999999e-05 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116277+0000", 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8269999999999997e-06 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116282+0000", 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.2673999999999999e-05 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116315+0000", 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000103157 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.698 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116418+0000", 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000116199 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116534+0000", 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000375437 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116910+0000", 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00070015599999999996 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.117610+0000", 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001332305 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.118942+0000", 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6996000000000002e-05 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.118989+0000", 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8662999999999999e-05 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.119028+0000", 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027659260000000001 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:172 4.b 4:d5e0c857:lc::lc.5:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.116291+0000", 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.54197763799999, 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027858240000000001, 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.701 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 172 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116291+0000", 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116291+0000", 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.702 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116290+0000", 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4539999999999999e-06 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116294+0000", 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3480000000000003e-06 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116299+0000", 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0960000000000003e-06 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116303+0000", 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1882000000000002e-05 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116355+0000", 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00010502599999999999 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116460+0000", 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00019528899999999999 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.116655+0000", 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0003634 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.117019+0000", 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000336862 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.117355+0000", 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016745550000000001 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.119030+0000", 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6324000000000001e-05 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.119056+0000", 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0078999999999999e-05 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.119076+0000", 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027858240000000001 2024-02-23T00:30:22.706 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:190 4.16 4:6a988a9b:reshard::reshard.0000000007:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.121635+0000", 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.53663361400001, 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001692217, 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 190 2024-02-23T00:30:22.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.121635+0000", 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.121635+0000", 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.121634+0000", 2024-02-23T00:30:22.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.311e-05 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.121647+0000", 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4620000000000001e-05 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.121661+0000", 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9910000000000002e-06 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.121666+0000", 2024-02-23T00:30:22.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5493000000000001e-05 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.121702+0000", 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5223999999999995e-05 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.121797+0000", 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000175336 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.121972+0000", 2024-02-23T00:30:22.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00031809699999999999 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.122290+0000", 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00057861200000000005 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.122869+0000", 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000404991 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123274+0000", 2024-02-23T00:30:22.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1371000000000002e-05 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123305+0000", 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1393e-05 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123327+0000", 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001692217 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:199 4.6 4:64760185:::obj_delete_at_hint.0000000010:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.123896+0000", 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.53437213300001, 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0018890440000000001, 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 199 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.713 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123896+0000", 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123896+0000", 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123895+0000", 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8309999999999997e-06 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123901+0000", 2024-02-23T00:30:22.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0584999999999999e-05 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123911+0000", 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2650000000000002e-06 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123918+0000", 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9255999999999997e-05 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.123967+0000", 2024-02-23T00:30:22.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7255000000000006e-05 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.124034+0000", 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00014708000000000001 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.124181+0000", 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00078197200000000003 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.716 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.124963+0000", 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00072889500000000002 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.125692+0000", 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7602000000000003e-05 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.125740+0000", 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5746999999999999e-05 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.125765+0000", 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9683000000000001e-05 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.125785+0000", 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0018890440000000001 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:269 4.3 4:c57bc89b:::obj_delete_at_hint.0000000013:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.138863+0000", 2024-02-23T00:30:22.718 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.51940551199999, 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0030020910000000001, 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 269 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.138863+0000", 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.138863+0000", 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.138862+0000", 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3449999999999996e-06 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.138867+0000", 2024-02-23T00:30:22.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1924e-05 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.138879+0000", 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0039999999999996e-06 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.138885+0000", 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0219000000000002e-05 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.138925+0000", 2024-02-23T00:30:22.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3203999999999999e-05 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.138999+0000", 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000181416 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 5,7", 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.139180+0000", 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00039142700000000002 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.139571+0000", 2024-02-23T00:30:22.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00052836599999999995 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140100+0000", 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001674771 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.141775+0000", 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0800000000000002e-05 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.141825+0000", 2024-02-23T00:30:22.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9449999999999997e-05 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.141865+0000", 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0030020910000000001 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:273 4.3 4:c134263f:lc::lc.25:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.140210+0000", 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.51805777499999, 2024-02-23T00:30:22.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001778733, 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 273 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140210+0000", 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.725 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140210+0000", 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140209+0000", 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7510000000000002e-06 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140216+0000", 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3067e-05 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.726 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140229+0000", 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0129999999999996e-06 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140236+0000", 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0478000000000001e-05 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140276+0000", 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011613899999999999 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.727 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140393+0000", 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021926199999999999 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 5,7", 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140612+0000", 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00032341200000000003 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.140935+0000", 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00048473199999999998 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.141420+0000", 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00048161299999999998 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.141902+0000", 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2122000000000001e-05 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.141944+0000", 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5392999999999997e-05 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.141989+0000", 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001778733 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:297 4.3 4:c623b173:gc::gc.16:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.730 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.145870+0000", 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.51239775499999, 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0023326150000000001, 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 297 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.731 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.145870+0000", 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.145870+0000", 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.732 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.145870+0000", 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1099999999999999e-06 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.145876+0000", 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8040000000000007e-06 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.145885+0000", 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7790000000000003e-06 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.733 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.145889+0000", 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5698000000000003e-05 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.145925+0000", 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5026999999999998e-05 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146000+0000", 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000151309 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 5,7", 2024-02-23T00:30:22.734 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146151+0000", 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00038681000000000002 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146538+0000", 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00023965600000000001 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146778+0000", 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0013324929999999999 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148110+0000", 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1411e-05 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148162+0000", 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1362999999999997e-05 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148203+0000", 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0023326150000000001 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24373.0:61 5.3 5:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.146849+0000", 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.511419617, 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0023258580000000001, 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24373", 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1092262198", 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 61 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.737 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146849+0000", 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146849+0000", 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146847+0000", 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3890000000000002e-06 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.738 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146854+0000", 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3743e-05 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146867+0000", 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0199999999999997e-06 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146875+0000", 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5250999999999998e-05 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.739 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146910+0000", 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7014000000000006e-05 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.146977+0000", 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012751599999999999 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 5,6", 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.147104+0000", 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00048858099999999998 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.740 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.741 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.147593+0000", 2024-02-23T00:30:22.741 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001289739 2024-02-23T00:30:22.741 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.741 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.741 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.741 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148883+0000", 2024-02-23T00:30:22.741 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000193866 2024-02-23T00:30:22.741 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.741 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.149076+0000", 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2502999999999994e-05 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.149139+0000", 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5466999999999998e-05 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.149174+0000", 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0023258580000000001 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.742 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:305 4.16 4:6849fd06:gc::gc.17:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.148399+0000", 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.50986941799999, 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027694080000000001, 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 305 2024-02-23T00:30:22.743 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148399+0000", 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148399+0000", 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.744 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148398+0000", 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0662000000000001e-05 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148408+0000", 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0421e-05 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148429+0000", 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8859999999999998e-06 2024-02-23T00:30:22.745 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148436+0000", 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4683e-05 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148470+0000", 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2588000000000004e-05 2024-02-23T00:30:22.746 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148543+0000", 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000179006 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.148722+0000", 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00058446300000000002 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.747 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.149306+0000", 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011071899999999999 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.149417+0000", 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0017034369999999999 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.151121+0000", 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7988e-05 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.748 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.151149+0000", 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.967e-05 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.151168+0000", 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027694080000000001 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.749 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:421 4.16 4:6d4431ed:lc::lc.7:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.175752+0000", 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.48251666499999, 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020380989999999998, 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.750 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 421 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.175752+0000", 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.175752+0000", 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.751 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.175751+0000", 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6019999999999999e-06 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.175756+0000", 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9210000000000004e-06 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.752 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.175766+0000", 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3080000000000004e-06 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.175773+0000", 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.8311999999999999e-05 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.175801+0000", 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2534999999999996e-05 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.175873+0000", 2024-02-23T00:30:22.753 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000176756 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.176050+0000", 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028012799999999999 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.176330+0000", 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00036071999999999998 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.754 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.176691+0000", 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001052603 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.177744+0000", 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6210000000000001e-05 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.177770+0000", 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9853000000000001e-05 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.755 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.177790+0000", 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020380989999999998 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:518 4.b 4:d5e0c857:lc::lc.5:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.200993+0000", 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.45727563400001, 2024-02-23T00:30:22.756 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027332599999999999, 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 518 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.200993+0000", 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.757 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.200993+0000", 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.200991+0000", 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6409999999999996e-06 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.200998+0000", 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2952999999999999e-05 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.758 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.201011+0000", 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4799999999999998e-06 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.201018+0000", 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1802e-05 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.201109+0000", 2024-02-23T00:30:22.759 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6514000000000006e-05 2024-02-23T00:30:22.760 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.760 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.760 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.760 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.201186+0000", 2024-02-23T00:30:22.760 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016334699999999999 2024-02-23T00:30:22.760 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.760 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.760 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.201349+0000", 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037272199999999997 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.201722+0000", 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000220903 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.201943+0000", 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001696052 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.761 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.203639+0000", 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9143e-05 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.203688+0000", 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7789e-05 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.762 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.203726+0000", 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027332599999999999 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:610 4.b 4:d242335b:gc::gc.2:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.231259+0000", 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.427008766, 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0033861780000000001, 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.763 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 610 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231259+0000", 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.764 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231259+0000", 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231258+0000", 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.409e-06 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231265+0000", 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3913e-05 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.765 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231279+0000", 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4230000000000002e-06 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231285+0000", 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8664000000000001e-05 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231324+0000", 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7911000000000002e-05 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.766 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.767 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231382+0000", 2024-02-23T00:30:22.767 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000144948 2024-02-23T00:30:22.767 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.767 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.767 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:30:22.767 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231527+0000", 2024-02-23T00:30:22.767 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00039107200000000001 2024-02-23T00:30:22.767 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.768 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.768 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.768 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.231918+0000", 2024-02-23T00:30:22.768 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00032120199999999998 2024-02-23T00:30:22.768 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.768 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.768 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.768 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.232239+0000", 2024-02-23T00:30:22.768 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0023267909999999999 2024-02-23T00:30:22.769 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.769 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.769 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.769 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234566+0000", 2024-02-23T00:30:22.769 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9496999999999997e-05 2024-02-23T00:30:22.769 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.769 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.769 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.769 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234615+0000", 2024-02-23T00:30:22.770 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0537000000000003e-05 2024-02-23T00:30:22.770 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.770 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.770 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.770 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234646+0000", 2024-02-23T00:30:22.770 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0033861780000000001 2024-02-23T00:30:22.770 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.770 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.771 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.771 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.771 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.771 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24362.0:617 4.16 4:6d4431ed:lc::lc.7:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-23T00:30:22.771 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:27:24.234785+0000", 2024-02-23T00:30:22.771 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 178.42348338799999, 2024-02-23T00:30:22.772 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027337389999999998, 2024-02-23T00:30:22.772 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.772 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:30:22.772 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.772 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24362", 2024-02-23T00:30:22.772 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/707689632", 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 617 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234785+0000", 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.773 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234785+0000", 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234784+0000", 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0440000000000001e-06 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234790+0000", 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0723999999999999e-05 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.774 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234800+0000", 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5220000000000007e-06 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234809+0000", 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8166e-05 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234847+0000", 2024-02-23T00:30:22.775 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9545e-05 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.234907+0000", 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000112402 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.235019+0000", 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00034010900000000001 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.776 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.235359+0000", 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021387000000000001 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.235573+0000", 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0018699459999999999 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.237443+0000", 2024-02-23T00:30:22.777 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1452000000000002e-05 2024-02-23T00:30:22.778 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.778 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.778 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:30:22.778 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.237484+0000", 2024-02-23T00:30:22.778 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4255999999999999e-05 2024-02-23T00:30:22.778 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.778 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.778 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.778 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:27:24.237519+0000", 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027337389999999998 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:600 5.3 5:c609908c:::notify.5:head [watch ping cookie 94532396652160] snapc 0=[] ondisk+write+known_if_redirected e82)", 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:29:15.348096+0000", 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 67.310172234000007, 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0021250650000000002, 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.779 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 600 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.348096+0000", 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.780 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.348096+0000", 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.348094+0000", 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1099999999999997e-06 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.781 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.348101+0000", 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8065999999999999e-05 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.348119+0000", 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3699999999999997e-06 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.348127+0000", 2024-02-23T00:30:22.782 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4156000000000001e-05 2024-02-23T00:30:22.783 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.783 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.783 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.783 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.348181+0000", 2024-02-23T00:30:22.783 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3549999999999999e-06 2024-02-23T00:30:22.783 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.783 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.783 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:30:22.783 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.348188+0000", 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0018576599999999999 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.350046+0000", 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4357000000000006e-05 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.350130+0000", 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0673000000000004e-05 2024-02-23T00:30:22.784 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:15.350221+0000", 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0021250650000000002 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:645 5.3 5:c609908c:::notify.5:head [watch ping cookie 94532396652160] snapc 0=[] ondisk+write+known_if_redirected e83)", 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:29:40.352111+0000", 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 42.306157310000003, 2024-02-23T00:30:22.785 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002817782, 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 645 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:30:22.786 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.352111+0000", 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.352111+0000", 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.352109+0000", 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0445000000000001e-05 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.787 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.352119+0000", 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1787e-05 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.352141+0000", 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5219999999999996e-06 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:30:22.788 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.352150+0000", 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3937999999999997e-05 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.352214+0000", 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7959999999999998e-06 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:30:22.789 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.352221+0000", 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002548505 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.354770+0000", 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8054999999999998e-05 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.354838+0000", 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0910999999999997e-05 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:30:22.790 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:30:22.791 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:29:40.354929+0000", 2024-02-23T00:30:22.791 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002817782 2024-02-23T00:30:22.791 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.791 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.791 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.791 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:30:22.791 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:30:22.791 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:30:22.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:22.820 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:22.987 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:22.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:23.093 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:23.122 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:23.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:21.493567+0000 mgr.y (mgr.24437) 127 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:22.242438+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:30:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:22.245353+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:30:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:21.493567+0000 mgr.y (mgr.24437) 127 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:22.242438+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:30:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:22.245353+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:30:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:21.493567+0000 mgr.y (mgr.24437) 127 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:22.242438+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:30:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:22.245353+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:30:23.509 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-23T00:30:23.672 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:23.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:23.779 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:23.811 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:23.833 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:23.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:23.936 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:23.969 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:23.972 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:30:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:30:24.248 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:22.779705+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.d scrub starts 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:22.781307+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.d scrub ok 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:22.857577+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:22.860558+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.077849+0000 osd.0 (osd.0) 53 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.080866+0000 osd.0 (osd.0) 54 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.595910+0000 osd.1 (osd.1) 57 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.600299+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:22.779705+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.d scrub starts 2024-02-23T00:30:24.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:22.781307+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.d scrub ok 2024-02-23T00:30:24.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:22.857577+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:30:24.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:22.860558+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:30:24.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.077849+0000 osd.0 (osd.0) 53 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:30:24.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.080866+0000 osd.0 (osd.0) 54 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:30:24.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.595910+0000 osd.1 (osd.1) 57 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:30:24.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.600299+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:30:24.410 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:30:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:22.779705+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.d scrub starts 2024-02-23T00:30:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:22.781307+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.d scrub ok 2024-02-23T00:30:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:22.857577+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:30:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:22.860558+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:30:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.077849+0000 osd.0 (osd.0) 53 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:30:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.080866+0000 osd.0 (osd.0) 54 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:30:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.595910+0000 osd.1 (osd.1) 57 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:30:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.600299+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:30:24.626 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:24.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:24.656 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:24.740 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:24.771 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:24.801 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:24.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:24.909 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:24.936 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:25.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.493993+0000 mgr.y (mgr.24437) 128 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:25.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.736454+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:30:25.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.737864+0000 osd.2 (osd.2) 48 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:30:25.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.834202+0000 osd.3 (osd.3) 67 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:30:25.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.839560+0000 osd.3 (osd.3) 68 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:30:25.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.951414+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:30:25.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:23.953324+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:30:25.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:24.047408+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:30:25.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:24.050879+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:30:25.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.493993+0000 mgr.y (mgr.24437) 128 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:25.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.736454+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:30:25.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.737864+0000 osd.2 (osd.2) 48 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:30:25.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.834202+0000 osd.3 (osd.3) 67 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:30:25.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.839560+0000 osd.3 (osd.3) 68 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:30:25.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.951414+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:30:25.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:23.953324+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:30:25.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:24.047408+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:30:25.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:24.050879+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:30:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.493993+0000 mgr.y (mgr.24437) 128 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.736454+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:30:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.737864+0000 osd.2 (osd.2) 48 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:30:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.834202+0000 osd.3 (osd.3) 67 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:30:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.839560+0000 osd.3 (osd.3) 68 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:30:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.951414+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:30:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:23.953324+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:30:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:24.047408+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:30:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:24.050879+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:30:25.991 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:25.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:26.097 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:26.130 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:26.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:24.706746+0000 osd.2 (osd.2) 49 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:30:26.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:24.708039+0000 osd.2 (osd.2) 50 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:30:26.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:24.792656+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:30:26.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:24.802124+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:30:26.237 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:25.051957+0000 osd.5 (osd.5) 47 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:30:26.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:25.054134+0000 osd.5 (osd.5) 48 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:30:26.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:25.061097+0000 osd.0 (osd.0) 57 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:30:26.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:25.064096+0000 osd.0 (osd.0) 58 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:30:26.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:25.160325+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:30:26.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:25.163294+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:30:26.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:25.590838+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:30:26.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:25.593829+0000 osd.1 (osd.1) 60 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:30:26.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:24.706746+0000 osd.2 (osd.2) 49 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:30:26.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:24.708039+0000 osd.2 (osd.2) 50 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:24.792656+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:24.802124+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:25.051957+0000 osd.5 (osd.5) 47 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:25.054134+0000 osd.5 (osd.5) 48 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:25.061097+0000 osd.0 (osd.0) 57 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:25.064096+0000 osd.0 (osd.0) 58 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:25.160325+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:25.163294+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:25.590838+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:30:26.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:25.593829+0000 osd.1 (osd.1) 60 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:30:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:24.706746+0000 osd.2 (osd.2) 49 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:30:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:24.708039+0000 osd.2 (osd.2) 50 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:30:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:24.792656+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:30:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:24.802124+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:30:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:25.051957+0000 osd.5 (osd.5) 47 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:30:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:25.054134+0000 osd.5 (osd.5) 48 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:30:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:25.061097+0000 osd.0 (osd.0) 57 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:30:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:25.064096+0000 osd.0 (osd.0) 58 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:30:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:25.160325+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:30:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:25.163294+0000 osd.6 (osd.6) 46 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:30:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:25.590838+0000 osd.1 (osd.1) 59 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:30:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:25.593829+0000 osd.1 (osd.1) 60 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:30:27.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.604+0000 7fc5ed10f700 1 -- 172.21.15.87:0/730410206 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5e80fdbb0 msgr2=0x7fc5e80fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.604+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/730410206 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5e80fdbb0 0x7fc5e80fdf90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc5d8004870 tx=0x7fc5d8009cb0).stop 2024-02-23T00:30:27.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 -- 172.21.15.87:0/730410206 shutdown_connections 2024-02-23T00:30:27.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/730410206 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5e80f8420 0x7fc5e8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/730410206 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5e80fdbb0 0x7fc5e80fdf90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/730410206 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5e80fe510 0x7fc5e80f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.026 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 -- 172.21.15.87:0/730410206 >> 172.21.15.87:0/730410206 conn(0x7fc5e80f3800 msgr2=0x7fc5e80f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 -- 172.21.15.87:0/730410206 shutdown_connections 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 -- 172.21.15.87:0/730410206 wait complete. 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 librados: starting msgr at 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 librados: starting objecter 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.605+0000 7fc5ed10f700 1 Processor -- start 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5ed10f700 1 -- start start 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5ed10f700 1 librados: setting wanted keys 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5ed10f700 1 librados: calling monclient init 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5ed10f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5e80f8420 0x7fc5e810e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5ed10f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5e80fdbb0 0x7fc5e810e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5ed10f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5e80fe510 0x7fc5e81081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5ed10f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81086b0 con 0x7fc5e80f8420 2024-02-23T00:30:27.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5ed10f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e8108820 con 0x7fc5e80fe510 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5ed10f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e81089e0 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5e6d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5e80f8420 0x7fc5e810e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5e659c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5e80fdbb0 0x7fc5e810e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5e659c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5e80fdbb0 0x7fc5e810e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34836/0 (socket says 172.21.15.87:34836) 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5e659c700 1 -- 172.21.15.87:0/3233119657 learned_addr learned my addr 172.21.15.87:0/3233119657 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.606+0000 7fc5e759e700 1 --2- 172.21.15.87:0/3233119657 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5e80fe510 0x7fc5e81081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5e659c700 1 -- 172.21.15.87:0/3233119657 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5e80fe510 msgr2=0x7fc5e81081c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5e659c700 1 --2- 172.21.15.87:0/3233119657 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5e80fe510 0x7fc5e81081c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5e659c700 1 -- 172.21.15.87:0/3233119657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5e80f8420 msgr2=0x7fc5e810e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5e659c700 1 --2- 172.21.15.87:0/3233119657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5e80f8420 0x7fc5e810e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5e659c700 1 -- 172.21.15.87:0/3233119657 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8005040 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5e6d9d700 1 --2- 172.21.15.87:0/3233119657 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5e80f8420 0x7fc5e810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5e659c700 1 --2- 172.21.15.87:0/3233119657 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5e80fdbb0 0x7fc5e810e510 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc5dc0027f0 tx=0x7fc5dc002b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5cffff700 10 client.?.objecter ms_handle_connect 0x7fc5e80fdbb0 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5cffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc007b40 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5dc00f040 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.607+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc0146f0 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.608+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e8108c60 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.608+0000 7fc5ed10f700 10 client.24722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.608+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e806af10 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.608+0000 7fc5ed10f700 1 librados: init done 2024-02-23T00:30:27.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.608+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fc5e805ee80 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.610+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc5dc009110 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5cffff700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc5d0062ce0 0x7fc5d00651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(91..91 src has 38..91) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fc5dc07cb90 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 osd_map(91..91 src has 38..91) v4 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5cffff700 3 client.24722.objecter handle_osd_map got epochs [91,91] > 0 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5cffff700 3 client.24722.objecter handle_osd_map decoding full epoch 91 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5e6d9d700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc5d0062ce0 0x7fc5d00651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5cffff700 20 client.24722.objecter dump_active .. 0 homeless 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v91) v1 ==== 54+0+16 (secure 0 0 0) 0x7fc5dc07d0a0 con 0x7fc5e80fdbb0 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5cffff700 10 client.24722.objecter ms_handle_connect 0x7fc5d0062ce0 2024-02-23T00:30:27.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5e6d9d700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc5d0062ce0 0x7fc5d00651a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fc5d8009a00 tx=0x7fc5d8009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 1 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc5e80fee60 0x7fc5e80ff260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8107110 osd=0 3 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8107110 osd=0 3 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8107110 osd=0 3 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 1 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8107110 osd=0 3 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 0 1 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8107110 osd=0 4 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 10 client.24722.objecter _send_command 1 2024-02-23T00:30:27.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.611+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5e81096a0 con 0x7fc5e80fee60 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.612+0000 7fc5e759e700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc5e80fee60 0x7fc5e80ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.612+0000 7fc5e759e700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc5e80fee60 0x7fc5e80ff260 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.612+0000 7fc5cffff700 10 client.24722.objecter ms_handle_connect 0x7fc5e80fee60 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.613+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc5e81096a0 con 0x7fc5e80fee60 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.613+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.613+0000 7fc5cffff700 10 client.24722.objecter _finish_command 1 = system:0 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.613+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8107110 osd=0 4 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.613+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 0 1 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 2 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8107110 osd=0 3 2024-02-23T00:30:27.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8107110 osd=0 3 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8107110 osd=0 3 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 2 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8107110 osd=0 3 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 0 2 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8107110 osd=0 4 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 10 client.24722.objecter _send_command 2 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.642+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc5e80273c0 con 0x7fc5e80fee60 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc5e80273c0 con 0x7fc5e80fee60 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5cffff700 10 client.24722.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8107110 osd=0 4 2024-02-23T00:30:27.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 0 2 2024-02-23T00:30:27.034 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:30:27.034 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:30:27.034 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:30:27.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:27.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 3 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6805/685354171 conn(0x7fc5e80598f0 0x7fc5e8059d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8028710 osd=1 3 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8028710 osd=1 3 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8028710 osd=1 3 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 3 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8028710 osd=1 3 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 1 3 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8028710 osd=1 4 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 10 client.24722.objecter _send_command 3 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.644+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5e806c7d0 con 0x7fc5e80598f0 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.645+0000 7fc5e6d9d700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6805/685354171 conn(0x7fc5e80598f0 0x7fc5e8059d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.645+0000 7fc5cffff700 10 client.24722.objecter ms_handle_connect 0x7fc5e80598f0 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.645+0000 7fc5e6d9d700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6805/685354171 conn(0x7fc5e80598f0 0x7fc5e8059d10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.645+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc5e806c7d0 con 0x7fc5e80598f0 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.645+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.645+0000 7fc5cffff700 10 client.24722.objecter _finish_command 3 = system:0 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.645+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8028710 osd=1 4 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.645+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 1 3 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 4 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8028710 osd=1 3 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8028710 osd=1 3 2024-02-23T00:30:27.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8028710 osd=1 3 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 4 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8028710 osd=1 3 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 1 4 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8028710 osd=1 4 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 10 client.24722.objecter _send_command 4 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.660+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc5e806cc00 con 0x7fc5e80598f0 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc5e806cc00 con 0x7fc5e80598f0 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5cffff700 10 client.24722.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8028710 osd=1 4 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 1 4 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 5 2024-02-23T00:30:27.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6809/3605394228 conn(0x7fc5e806b870 0x7fc5e806bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e806b1f0 osd=2 3 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e806b1f0 osd=2 3 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e806b1f0 osd=2 3 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 5 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e806b1f0 osd=2 3 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 2 5 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e806b1f0 osd=2 4 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 10 client.24722.objecter _send_command 5 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5e806c1a0 con 0x7fc5e806b870 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5e659c700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6809/3605394228 conn(0x7fc5e806b870 0x7fc5e806bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5cffff700 10 client.24722.objecter ms_handle_connect 0x7fc5e806b870 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.663+0000 7fc5e659c700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6809/3605394228 conn(0x7fc5e806b870 0x7fc5e806bcb0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.664+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc5e806c1a0 con 0x7fc5e806b870 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.664+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:30:27.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.664+0000 7fc5cffff700 10 client.24722.objecter _finish_command 5 = system:0 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.664+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e806b1f0 osd=2 4 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.664+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 2 5 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 6 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e806b1f0 osd=2 3 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e806b1f0 osd=2 3 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e806b1f0 osd=2 3 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 6 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e806b1f0 osd=2 3 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 2 6 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e806b1f0 osd=2 4 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 10 client.24722.objecter _send_command 6 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.688+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc5e806c600 con 0x7fc5e806b870 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc5e806c600 con 0x7fc5e806b870 2024-02-23T00:30:27.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:27.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5cffff700 10 client.24722.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e806b1f0 osd=2 4 2024-02-23T00:30:27.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 2 6 2024-02-23T00:30:27.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:27.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 7 2024-02-23T00:30:27.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6813/128059790 conn(0x7fc5e80594d0 0x7fc5e811e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.040 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:30:27.040 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8059360 osd=3 3 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8059360 osd=3 3 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8059360 osd=3 3 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 7 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8059360 osd=3 3 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 3 7 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8059360 osd=3 4 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 10 client.24722.objecter _send_command 7 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.690+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5e8028270 con 0x7fc5e80594d0 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.691+0000 7fc5e759e700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6813/128059790 conn(0x7fc5e80594d0 0x7fc5e811e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.691+0000 7fc5cffff700 10 client.24722.objecter ms_handle_connect 0x7fc5e80594d0 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.691+0000 7fc5e759e700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6813/128059790 conn(0x7fc5e80594d0 0x7fc5e811e010 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.692+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc5e8028270 con 0x7fc5e80594d0 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.692+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.692+0000 7fc5cffff700 10 client.24722.objecter _finish_command 7 = system:0 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.692+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8059360 osd=3 4 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.692+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 3 7 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 8 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8059360 osd=3 3 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8059360 osd=3 3 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8059360 osd=3 3 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 8 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8059360 osd=3 3 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 3 8 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8059360 osd=3 4 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 10 client.24722.objecter _send_command 8 2024-02-23T00:30:27.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.719+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc5e811e500 con 0x7fc5e80594d0 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.721+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc5e811e500 con 0x7fc5e80594d0 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.721+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.721+0000 7fc5cffff700 10 client.24722.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.721+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8059360 osd=3 4 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.721+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 3 8 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 9 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6800/4175844757 conn(0x7fc5e811ec10 0x7fc5e8121000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e811e500 osd=4 3 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e811e500 osd=4 3 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e811e500 osd=4 3 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 9 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e811e500 osd=4 3 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 4 9 2024-02-23T00:30:27.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e811e500 osd=4 4 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 10 client.24722.objecter _send_command 9 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5e8123c50 con 0x7fc5e811ec10 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5e6d9d700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6800/4175844757 conn(0x7fc5e811ec10 0x7fc5e8121000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5cffff700 10 client.24722.objecter ms_handle_connect 0x7fc5e811ec10 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.722+0000 7fc5e6d9d700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6800/4175844757 conn(0x7fc5e811ec10 0x7fc5e8121000 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.723+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc5e8123c50 con 0x7fc5e811ec10 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.723+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.723+0000 7fc5cffff700 10 client.24722.objecter _finish_command 9 = system:0 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.723+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e811e500 osd=4 4 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.723+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 4 9 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 10 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e811e500 osd=4 3 2024-02-23T00:30:27.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e811e500 osd=4 3 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e811e500 osd=4 3 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 10 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e811e500 osd=4 3 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 4 10 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e811e500 osd=4 4 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 10 client.24722.objecter _send_command 10 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.752+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc5e81214f0 con 0x7fc5e811ec10 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc5e81214f0 con 0x7fc5e811ec10 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5cffff700 10 client.24722.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e811e500 osd=4 4 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 4 10 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:27.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 11 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6804/2550013795 conn(0x7fc5e8121bd0 0x7fc5e8123fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e81214f0 osd=5 3 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e81214f0 osd=5 3 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e81214f0 osd=5 3 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 11 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e81214f0 osd=5 3 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 5 11 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e81214f0 osd=5 4 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 10 client.24722.objecter _send_command 11 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5e8126d90 con 0x7fc5e8121bd0 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.754+0000 7fc5e659c700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6804/2550013795 conn(0x7fc5e8121bd0 0x7fc5e8123fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.755+0000 7fc5cffff700 10 client.24722.objecter ms_handle_connect 0x7fc5e8121bd0 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.755+0000 7fc5e659c700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6804/2550013795 conn(0x7fc5e8121bd0 0x7fc5e8123fc0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.756+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc5e8126d90 con 0x7fc5e8121bd0 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.756+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.756+0000 7fc5cffff700 10 client.24722.objecter _finish_command 11 = system:0 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.756+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e81214f0 osd=5 4 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.756+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 5 11 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 12 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e81214f0 osd=5 3 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e81214f0 osd=5 3 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e81214f0 osd=5 3 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 12 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e81214f0 osd=5 3 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 5 12 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e81214f0 osd=5 4 2024-02-23T00:30:27.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 10 client.24722.objecter _send_command 12 2024-02-23T00:30:27.048 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:30:27.048 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:30:27.048 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:30:27.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.784+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc5e8126d90 con 0x7fc5e8121bd0 2024-02-23T00:30:27.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.785+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc5e8126d90 con 0x7fc5e8121bd0 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.785+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.785+0000 7fc5cffff700 10 client.24722.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.785+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e81214f0 osd=5 4 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.785+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 5 12 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 13 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6808/2509408498 conn(0x7fc5e8126090 0x7fc5e8126510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8126d90 osd=6 3 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126d90 osd=6 3 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8126d90 osd=6 3 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 13 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8126d90 osd=6 3 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 6 13 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126d90 osd=6 4 2024-02-23T00:30:27.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 10 client.24722.objecter _send_command 13 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5e8126b80 con 0x7fc5e8126090 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5e759e700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6808/2509408498 conn(0x7fc5e8126090 0x7fc5e8126510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5cffff700 10 client.24722.objecter ms_handle_connect 0x7fc5e8126090 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.786+0000 7fc5e759e700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6808/2509408498 conn(0x7fc5e8126090 0x7fc5e8126510 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.787+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc5e8126b80 con 0x7fc5e8126090 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.787+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.787+0000 7fc5cffff700 10 client.24722.objecter _finish_command 13 = system:0 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.787+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8126d90 osd=6 4 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.787+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 6 13 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 14 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8126d90 osd=6 3 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126d90 osd=6 3 2024-02-23T00:30:27.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8126d90 osd=6 3 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 14 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8126d90 osd=6 3 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 6 14 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126d90 osd=6 4 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 10 client.24722.objecter _send_command 14 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.815+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc5e8126b80 con 0x7fc5e8126090 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc5e8126b80 con 0x7fc5e8126090 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5cffff700 10 client.24722.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8126d90 osd=6 4 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 6 14 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 15 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6812/373787000 conn(0x7fc5e8129540 0x7fc5e8129980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8126b80 osd=7 3 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126b80 osd=7 3 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8126b80 osd=7 3 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 15 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8126b80 osd=7 3 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 7 15 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126b80 osd=7 4 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 10 client.24722.objecter _send_command 15 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.817+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5e8129ff0 con 0x7fc5e8129540 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.818+0000 7fc5e6d9d700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6812/373787000 conn(0x7fc5e8129540 0x7fc5e8129980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.818+0000 7fc5cffff700 10 client.24722.objecter ms_handle_connect 0x7fc5e8129540 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.818+0000 7fc5e6d9d700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6812/373787000 conn(0x7fc5e8129540 0x7fc5e8129980 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.819+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc5e8129ff0 con 0x7fc5e8129540 2024-02-23T00:30:27.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.819+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.819+0000 7fc5cffff700 10 client.24722.objecter _finish_command 15 = system:0 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.819+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8126b80 osd=7 4 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.819+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 7 15 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 10 client.24722.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign -1 16 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8126b80 osd=7 3 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126b80 osd=7 3 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 20 client.24722.objecter _get_session s=0x7fc5e8126b80 osd=7 3 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_remove -1 16 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 20 client.24722.objecter get_session s=0x7fc5e8126b80 osd=7 3 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 15 client.24722.objecter _session_command_op_assign 7 16 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126b80 osd=7 4 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 10 client.24722.objecter _send_command 16 2024-02-23T00:30:27.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.847+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fc5e8129ff0 con 0x7fc5e8129540 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5cffff700 1 -- 172.21.15.87:0/3233119657 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fc5e8129ff0 con 0x7fc5e8129540 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5cffff700 10 client.24722.objecter ms_dispatch 0x7fc5e8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5cffff700 10 client.24722.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5cffff700 20 client.24722.objecter put_session s=0x7fc5e8126b80 osd=7 4 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5cffff700 15 client.24722.objecter _session_command_op_remove 7 16 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 librados: watch_flush enter 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 librados: watch_flush exit 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 client.24722.objecter close_session for osd.0 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc5e80fee60 msgr2=0x7fc5e80ff260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6801/2153191863 conn(0x7fc5e80fee60 0x7fc5e80ff260 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8107110 osd=0 1 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 client.24722.objecter close_session for osd.1 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6805/685354171 conn(0x7fc5e80598f0 msgr2=0x7fc5e8059d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6805/685354171 conn(0x7fc5e80598f0 0x7fc5e8059d10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8028710 osd=1 1 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 client.24722.objecter close_session for osd.2 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6809/3605394228 conn(0x7fc5e806b870 msgr2=0x7fc5e806bcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6809/3605394228 conn(0x7fc5e806b870 0x7fc5e806bcb0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e806b1f0 osd=2 1 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 client.24722.objecter close_session for osd.3 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6813/128059790 conn(0x7fc5e80594d0 msgr2=0x7fc5e811e010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6813/128059790 conn(0x7fc5e80594d0 0x7fc5e811e010 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8059360 osd=3 1 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5e759e700 1 -- 172.21.15.87:0/3233119657 reap_dead start 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 client.24722.objecter close_session for osd.4 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6800/4175844757 conn(0x7fc5e811ec10 msgr2=0x7fc5e8121000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6800/4175844757 conn(0x7fc5e811ec10 0x7fc5e8121000 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5e759e700 1 -- 172.21.15.87:0/3233119657 reap_dead start 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e811e500 osd=4 1 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 client.24722.objecter close_session for osd.5 2024-02-23T00:30:27.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6804/2550013795 conn(0x7fc5e8121bd0 msgr2=0x7fc5e8123fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6804/2550013795 conn(0x7fc5e8121bd0 0x7fc5e8123fc0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e81214f0 osd=5 1 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 client.24722.objecter close_session for osd.6 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6808/2509408498 conn(0x7fc5e8126090 msgr2=0x7fc5e8126510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6808/2509408498 conn(0x7fc5e8126090 0x7fc5e8126510 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126d90 osd=6 1 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 client.24722.objecter close_session for osd.7 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6812/373787000 conn(0x7fc5e8129540 msgr2=0x7fc5e8129980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.171:6812/373787000 conn(0x7fc5e8129540 0x7fc5e8129980 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 20 client.24722.objecter put_session s=0x7fc5e8126b80 osd=7 1 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 20 client.24722.objecter shutdown clearing up homeless session... 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 10 client.24722.objecter successfully canceled tick 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc5d0062ce0 msgr2=0x7fc5d00651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc5d0062ce0 0x7fc5d00651a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fc5d8009a00 tx=0x7fc5d8009cb0).stop 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5e80fdbb0 msgr2=0x7fc5e810e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 --2- 172.21.15.87:0/3233119657 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5e80fdbb0 0x7fc5e810e510 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc5dc0027f0 tx=0x7fc5dc002b00).stop 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5e759e700 1 -- 172.21.15.87:0/3233119657 reap_dead start 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 shutdown_connections 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.849+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 >> 172.21.15.87:0/3233119657 conn(0x7fc5e80f3800 msgr2=0x7fc5e80f5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.850+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 shutdown_connections 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.850+0000 7fc5ed10f700 1 -- 172.21.15.87:0/3233119657 wait complete. 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:26.850+0000 7fc5ed10f700 1 librados: shutdown 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.057 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.058 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:30:27.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:25.494720+0000 mgr.y (mgr.24437) 129 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:26.058306+0000 osd.5 (osd.5) 49 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:26.059240+0000 osd.5 (osd.5) 50 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:26.073128+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:26.074256+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:26.167593+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:26.170477+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:26.611054+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? 172.21.15.87:0/3233119657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:25.494720+0000 mgr.y (mgr.24437) 129 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:26.058306+0000 osd.5 (osd.5) 49 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:30:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:26.059240+0000 osd.5 (osd.5) 50 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:30:27.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:26.073128+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:30:27.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:26.074256+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:30:27.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:26.167593+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:30:27.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:26.170477+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:30:27.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:26.611054+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? 172.21.15.87:0/3233119657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:25.494720+0000 mgr.y (mgr.24437) 129 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:26.058306+0000 osd.5 (osd.5) 49 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:30:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:26.059240+0000 osd.5 (osd.5) 50 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:30:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:26.073128+0000 osd.0 (osd.0) 59 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:30:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:26.074256+0000 osd.0 (osd.0) 60 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:30:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:26.167593+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:30:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:26.170477+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:30:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:26.611054+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? 172.21.15.87:0/3233119657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:28.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:26.670846+0000 osd.2 (osd.2) 51 : cluster [DBG] 6.f scrub starts 2024-02-23T00:30:28.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:26.671881+0000 osd.2 (osd.2) 52 : cluster [DBG] 6.f scrub ok 2024-02-23T00:30:28.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:27.148438+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:30:28.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:27.151338+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:30:28.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:26.670846+0000 osd.2 (osd.2) 51 : cluster [DBG] 6.f scrub starts 2024-02-23T00:30:28.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:26.671881+0000 osd.2 (osd.2) 52 : cluster [DBG] 6.f scrub ok 2024-02-23T00:30:28.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:27.148438+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:30:28.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:27.151338+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:30:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:26.670846+0000 osd.2 (osd.2) 51 : cluster [DBG] 6.f scrub starts 2024-02-23T00:30:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:26.671881+0000 osd.2 (osd.2) 52 : cluster [DBG] 6.f scrub ok 2024-02-23T00:30:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:27.148438+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:30:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:27.151338+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:30:28.985 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:28.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:29.090 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:29.123 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:29.144 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:29.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:29.248 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:29.259 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:27.495090+0000 mgr.y (mgr.24437) 130 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:29.259 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:27.495090+0000 mgr.y (mgr.24437) 130 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:29.276 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:27.495090+0000 mgr.y (mgr.24437) 130 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:29.278 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:29.299 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:29.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:29.402 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:29.442 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.458+0000 7f012ee3c700 1 -- 172.21.15.87:0/684659713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01280fcb40 msgr2=0x7f01280fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.458+0000 7f012ee3c700 1 --2- 172.21.15.87:0/684659713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01280fcb40 0x7f01280fcf20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0124004830 tx=0x7f0124009cb0).stop 2024-02-23T00:30:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.458+0000 7f012ee3c700 1 -- 172.21.15.87:0/684659713 shutdown_connections 2024-02-23T00:30:29.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.458+0000 7f012ee3c700 1 --2- 172.21.15.87:0/684659713 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01280f7ba0 0x7f0128103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:29.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.458+0000 7f012ee3c700 1 --2- 172.21.15.87:0/684659713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01280fcb40 0x7f01280fcf20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:29.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.458+0000 7f012ee3c700 1 --2- 172.21.15.87:0/684659713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01280fd4a0 0x7f01280f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:29.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.458+0000 7f012ee3c700 1 -- 172.21.15.87:0/684659713 >> 172.21.15.87:0/684659713 conn(0x7f012806c370 msgr2=0x7f012806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:29.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.460+0000 7f012ee3c700 1 -- 172.21.15.87:0/684659713 shutdown_connections 2024-02-23T00:30:29.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.465+0000 7f012ee3c700 1 -- 172.21.15.87:0/684659713 wait complete. 2024-02-23T00:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.466+0000 7f012ee3c700 1 librados: starting msgr at 2024-02-23T00:30:29.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.466+0000 7f012ee3c700 1 librados: starting objecter 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.466+0000 7f012ee3c700 1 Processor -- start 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012ee3c700 1 -- start start 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012ee3c700 1 librados: setting wanted keys 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012ee3c700 1 librados: calling monclient init 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012ee3c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01280f7ba0 0x7f01281061d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012ee3c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01280fcb40 0x7f01281066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012ee3c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01280fd4a0 0x7f0128106bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012ee3c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012810a2a0 con 0x7f01280fcb40 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012ee3c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012810a420 con 0x7f01280f7ba0 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012ee3c700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012810a5e0 con 0x7f01280fd4a0 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f011ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01280fcb40 0x7f01281066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:29.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f011ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01280fcb40 0x7f01281066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39514/0 (socket says 172.21.15.87:39514) 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f011ffff700 1 -- 172.21.15.87:0/1029226120 learned_addr learned my addr 172.21.15.87:0/1029226120 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012d3d9700 1 --2- 172.21.15.87:0/1029226120 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01280fd4a0 0x7f0128106bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f012cbd8700 1 --2- 172.21.15.87:0/1029226120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01280f7ba0 0x7f01281061d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f011ffff700 1 -- 172.21.15.87:0/1029226120 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01280fd4a0 msgr2=0x7f0128106bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f011ffff700 1 --2- 172.21.15.87:0/1029226120 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01280fd4a0 0x7f0128106bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f011ffff700 1 -- 172.21.15.87:0/1029226120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01280f7ba0 msgr2=0x7f01281061d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f011ffff700 1 --2- 172.21.15.87:0/1029226120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01280f7ba0 0x7f01281061d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.467+0000 7f011ffff700 1 -- 172.21.15.87:0/1029226120 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0118005040 con 0x7f01280fcb40 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f011ffff700 1 --2- 172.21.15.87:0/1029226120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01280fcb40 0x7f01281066c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f01180048f0 tx=0x7f0118009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f011dffb700 10 client.?.objecter ms_handle_connect 0x7f01280fcb40 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f011dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f011dffb700 1 -- 172.21.15.87:0/1029226120 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118007b00 con 0x7f01280fcb40 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f011dffb700 1 -- 172.21.15.87:0/1029226120 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f011800f040 con 0x7f01280fcb40 2024-02-23T00:30:29.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0124005040 con 0x7f01280fcb40 2024-02-23T00:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f012ee3c700 10 client.14802.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012810ac70 con 0x7f01280fcb40 2024-02-23T00:30:29.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f012ee3c700 1 librados: init done 2024-02-23T00:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.468+0000 7f011dffb700 1 -- 172.21.15.87:0/1029226120 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118014aa0 con 0x7f01280fcb40 2024-02-23T00:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.470+0000 7f011dffb700 1 -- 172.21.15.87:0/1029226120 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f01180133c0 con 0x7f01280fcb40 2024-02-23T00:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.470+0000 7f011dffb700 1 --2- 172.21.15.87:0/1029226120 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0110062ce0 0x7f01100651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.471+0000 7f011dffb700 1 -- 172.21.15.87:0/1029226120 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(91..91 src has 38..91) v4 ==== 6172+0+0 (secure 0 0 0) 0x7f011807c6c0 con 0x7f01280fcb40 2024-02-23T00:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.471+0000 7f011dffb700 10 client.14802.objecter ms_dispatch 0x7f0128001550 osd_map(91..91 src has 38..91) v4 2024-02-23T00:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.471+0000 7f011dffb700 3 client.14802.objecter handle_osd_map got epochs [91,91] > 0 2024-02-23T00:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.471+0000 7f011dffb700 3 client.14802.objecter handle_osd_map decoding full epoch 91 2024-02-23T00:30:29.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.471+0000 7f011dffb700 20 client.14802.objecter dump_active .. 0 homeless 2024-02-23T00:30:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.473+0000 7f012cbd8700 1 --2- 172.21.15.87:0/1029226120 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0110062ce0 0x7f01100651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:29.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.473+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0114004f50 con 0x7f01280fcb40 2024-02-23T00:30:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.473+0000 7f012cbd8700 1 --2- 172.21.15.87:0/1029226120 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0110062ce0 0x7f01100651a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f01240115a0 tx=0x7f012400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:29.477 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.473+0000 7f011dffb700 10 client.14802.objecter ms_handle_connect 0x7f0110062ce0 2024-02-23T00:30:29.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.477+0000 7f011dffb700 1 -- 172.21.15.87:0/1029226120 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0118050290 con 0x7f01280fcb40 2024-02-23T00:30:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:29.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:29.691 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:29.723 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:29.744 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:29.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:29.754 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:30:29.755 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:30:29.772 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:29.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.782+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}) v1 -- 0x7f0114002840 con 0x7f0110062ce0 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:moved 0 / 396 (0%) 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:avg 49.5 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:stddev 5.22015 -> 5.22015 (expected baseline 6.58122) 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:min osd.2 with 40 -> 40 pgs (0.808081 -> 0.808081 * mean) 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:max osd.7 with 56 -> 56 pgs (1.13131 -> 1.13131 * mean) 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:oload 110 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:max_change 3 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:max_change_osds 4 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:average_utilization 0.0032 2024-02-23T00:30:29.787 INFO:teuthology.orchestra.run.smithi087.stdout:overload_utilization 0.0035 2024-02-23T00:30:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.783+0000 7f011dffb700 1 -- 172.21.15.87:0/1029226120 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+293 (secure 0 0 0) 0x7f0114002840 con 0x7f0110062ce0 2024-02-23T00:30:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 10 librados: watch_flush enter 2024-02-23T00:30:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 10 librados: watch_flush exit 2024-02-23T00:30:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 20 client.14802.objecter shutdown clearing up homeless session... 2024-02-23T00:30:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 10 client.14802.objecter successfully canceled tick 2024-02-23T00:30:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0110062ce0 msgr2=0x7f01100651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:29.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 --2- 172.21.15.87:0/1029226120 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0110062ce0 0x7f01100651a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f01240115a0 tx=0x7f012400b040).stop 2024-02-23T00:30:29.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01280fcb40 msgr2=0x7f01281066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:29.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 --2- 172.21.15.87:0/1029226120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01280fcb40 0x7f01281066c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f01180048f0 tx=0x7f0118009cb0).stop 2024-02-23T00:30:29.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 shutdown_connections 2024-02-23T00:30:29.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 --2- 172.21.15.87:0/1029226120 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0110062ce0 0x7f01100651a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:29.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 --2- 172.21.15.87:0/1029226120 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f01280fd4a0 0x7f0128106bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:29.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 --2- 172.21.15.87:0/1029226120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f01280f7ba0 0x7f01281061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:29.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 --2- 172.21.15.87:0/1029226120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f01280fcb40 0x7f01281066c0 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0x7f01180048f0 tx=0x7f0118009cb0).stop 2024-02-23T00:30:29.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.785+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 >> 172.21.15.87:0/1029226120 conn(0x7f012806c370 msgr2=0x7f01280f5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:29.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.787+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 shutdown_connections 2024-02-23T00:30:29.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.787+0000 7f012ee3c700 1 -- 172.21.15.87:0/1029226120 wait complete. 2024-02-23T00:30:29.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:29.789+0000 7f012ee3c700 1 librados: shutdown 2024-02-23T00:30:29.802 INFO:teuthology.orchestra.run.smithi087.stderr:no change 2024-02-23T00:30:29.827 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:30:29.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:29.911 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:30.072 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:30.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:30.178 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:30.212 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:30.232 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:30.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:30.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:28.675185+0000 osd.2 (osd.2) 53 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:30:30.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:28.676320+0000 osd.2 (osd.2) 54 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:30:30.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:29.048535+0000 osd.4 (osd.4) 59 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:30:30.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:29.051433+0000 osd.4 (osd.4) 60 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:30:30.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:29.522654+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:30:30.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:29.527456+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:30:30.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:28.675185+0000 osd.2 (osd.2) 53 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:30:30.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:28.676320+0000 osd.2 (osd.2) 54 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:30:30.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:29.048535+0000 osd.4 (osd.4) 59 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:30:30.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:29.051433+0000 osd.4 (osd.4) 60 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:30:30.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:29.522654+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:30:30.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:29.527456+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:30:30.337 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:30.367 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:30.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:28.675185+0000 osd.2 (osd.2) 53 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:30:30.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:28.676320+0000 osd.2 (osd.2) 54 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:30:30.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:29.048535+0000 osd.4 (osd.4) 59 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:30:30.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:29.051433+0000 osd.4 (osd.4) 60 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:30:30.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:29.522654+0000 osd.1 (osd.1) 61 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:30:30.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:29.527456+0000 osd.1 (osd.1) 62 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:30:30.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:30:30.961 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:30.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:30.987 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-23T00:30:31.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:31.108 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:31.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:29.495476+0000 mgr.y (mgr.24437) 131 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:29.784515+0000 mgr.y (mgr.24437) 132 : audit [DBG] from='client.14802 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:29.840190+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:30:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:29.843946+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:30:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:30.204285+0000 osd.6 (osd.6) 51 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:30:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:30.207842+0000 osd.6 (osd.6) 52 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:30:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:30.482923+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:30:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:30.485894+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:30:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:29.495476+0000 mgr.y (mgr.24437) 131 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:29.784515+0000 mgr.y (mgr.24437) 132 : audit [DBG] from='client.14802 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:29.840190+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:30:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:29.843946+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:30:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:30.204285+0000 osd.6 (osd.6) 51 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:30:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:30.207842+0000 osd.6 (osd.6) 52 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:30:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:30.482923+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:30:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:30.485894+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:30:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:29.495476+0000 mgr.y (mgr.24437) 131 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:29.784515+0000 mgr.y (mgr.24437) 132 : audit [DBG] from='client.14802 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:29.840190+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:30:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:29.843946+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:30:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:30.204285+0000 osd.6 (osd.6) 51 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:30:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:30.207842+0000 osd.6 (osd.6) 52 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:30:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:30.482923+0000 osd.1 (osd.1) 63 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:30:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:30.485894+0000 osd.1 (osd.1) 64 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:30:32.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:30.129261+0000 osd.5 (osd.5) 51 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:30:32.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:30.130945+0000 osd.5 (osd.5) 52 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:30:32.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:30.496546+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:30:32.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:30.499427+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:30:32.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:30.607224+0000 mgr.y (mgr.24437) 133 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:32.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:31.062359+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.e scrub starts 2024-02-23T00:30:32.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:31.065380+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.e scrub ok 2024-02-23T00:30:32.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:31.437924+0000 osd.1 (osd.1) 65 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:31.441410+0000 osd.1 (osd.1) 66 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:30.129261+0000 osd.5 (osd.5) 51 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:30.130945+0000 osd.5 (osd.5) 52 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:30.496546+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:30.499427+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:30.607224+0000 mgr.y (mgr.24437) 133 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:31.062359+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.e scrub starts 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:31.065380+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.e scrub ok 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:31.437924+0000 osd.1 (osd.1) 65 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:30:32.006 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:31.441410+0000 osd.1 (osd.1) 66 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:30:32.148 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-23T00:30:32.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd test-reweight-by-utilization 110 1.0 2024-02-23T00:30:32.166 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:32.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:32.221 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:32.270 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:32.301 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:30:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:32] "GET /metrics HTTP/1.1" 200 172255 "" "Prometheus/2.33.4" 2024-02-23T00:30:32.304 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:32.329 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:32.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:32.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:32.441 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:30.129261+0000 osd.5 (osd.5) 51 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:30:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:30.130945+0000 osd.5 (osd.5) 52 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:30:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:30.496546+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:30:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:30.499427+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:30:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:30.607224+0000 mgr.y (mgr.24437) 133 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:31.062359+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.e scrub starts 2024-02-23T00:30:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:31.065380+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.e scrub ok 2024-02-23T00:30:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:31.437924+0000 osd.1 (osd.1) 65 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:30:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:31.441410+0000 osd.1 (osd.1) 66 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:30:32.474 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:32.495 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:32.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:32.600 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:32.631 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:32.652 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:32.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:32.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:32.784 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:32.786 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:32.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:32.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:32.910 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:32.938 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:32.958 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:32.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:33.058 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:31.496067+0000 mgr.y (mgr.24437) 134 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-02-23T00:30:33.058 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:31.826632+0000 osd.3 (osd.3) 73 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:30:33.058 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:31.829690+0000 osd.3 (osd.3) 74 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:30:33.058 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:31.983573+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:30:33.058 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:31.986609+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:30:33.058 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:31.496067+0000 mgr.y (mgr.24437) 134 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-02-23T00:30:33.058 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:31.826632+0000 osd.3 (osd.3) 73 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:30:33.058 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:31.829690+0000 osd.3 (osd.3) 74 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:30:33.059 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:31.983573+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:30:33.059 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:31.986609+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:30:33.062 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:33.087 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:31.496067+0000 mgr.y (mgr.24437) 134 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-02-23T00:30:33.087 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:31.826632+0000 osd.3 (osd.3) 73 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:30:33.087 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:31.829690+0000 osd.3 (osd.3) 74 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:30:33.088 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:31.983573+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:30:33.088 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:31.986609+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:30:33.089 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:33.107 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:33.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:33.211 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:33.241 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:33.491 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:33.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:33.598 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:33.626 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:33.645 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:33.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:33.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:33.749 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:33.777 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:33.795 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:33.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:33.899 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:33.923 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:30:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:30:33.925 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:33.944 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:33.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:34.047 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:34.074 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:34.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:32.476903+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:30:34.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:32.479806+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:32.875594+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:32.878563+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:32.950230+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:32.953135+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:32.476903+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:32.479806+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:32.875594+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:32.878563+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:32.950230+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:30:34.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:32.953135+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:30:34.092 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:34.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:32.476903+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:30:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:32.479806+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:30:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:32.875594+0000 osd.3 (osd.3) 75 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:30:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:32.878563+0000 osd.3 (osd.3) 76 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:30:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:32.950230+0000 osd.0 (osd.0) 63 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:30:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:32.953135+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:30:34.196 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:34.222 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:34.240 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:34.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:34.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:34.344 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:34.370 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:34.389 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:34.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:34.493 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:34.519 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:34.538 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:34.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:34.642 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:34.668 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:34.687 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:34.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:34.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:34.822 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:35.015 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:33.496472+0000 mgr.y (mgr.24437) 135 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:30:35.015 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:33.525788+0000 osd.7 (osd.7) 55 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:30:35.015 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:33.528698+0000 osd.7 (osd.7) 56 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:30:35.015 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:33.902518+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-02-23T00:30:35.015 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:33.906029+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-02-23T00:30:35.015 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:33.496472+0000 mgr.y (mgr.24437) 135 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:30:35.015 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:33.525788+0000 osd.7 (osd.7) 55 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:30:35.015 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:33.528698+0000 osd.7 (osd.7) 56 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:30:35.015 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:33.902518+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-02-23T00:30:35.016 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:33.906029+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-02-23T00:30:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:33.496472+0000 mgr.y (mgr.24437) 135 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:30:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:33.525788+0000 osd.7 (osd.7) 55 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:30:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:33.528698+0000 osd.7 (osd.7) 56 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:30:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:33.902518+0000 osd.0 (osd.0) 65 : cluster [DBG] 4.b scrub starts 2024-02-23T00:30:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:33.906029+0000 osd.0 (osd.0) 66 : cluster [DBG] 4.b scrub ok 2024-02-23T00:30:35.845 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:30:35.845 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:30:35.845 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.764+0000 7fb279a7b700 1 -- 172.21.15.87:0/213596973 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2740f6a90 msgr2=0x7fb274104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.764+0000 7fb279a7b700 1 --2- 172.21.15.87:0/213596973 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2740f6a90 0x7fb274104a20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb26c009be0 tx=0x7fb26c004d10).stop 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.764+0000 7fb279a7b700 1 -- 172.21.15.87:0/213596973 shutdown_connections 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.764+0000 7fb279a7b700 1 --2- 172.21.15.87:0/213596973 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2740f6a90 0x7fb274104a20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.764+0000 7fb279a7b700 1 --2- 172.21.15.87:0/213596973 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2740fdbd0 0x7fb2740fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.764+0000 7fb279a7b700 1 --2- 172.21.15.87:0/213596973 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740fe530 0x7fb2740f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.764+0000 7fb279a7b700 1 -- 172.21.15.87:0/213596973 >> 172.21.15.87:0/213596973 conn(0x7fb27406c370 msgr2=0x7fb27406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.764+0000 7fb279a7b700 1 -- 172.21.15.87:0/213596973 shutdown_connections 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.764+0000 7fb279a7b700 1 -- 172.21.15.87:0/213596973 wait complete. 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.765+0000 7fb279a7b700 1 librados: starting msgr at 2024-02-23T00:30:35.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.765+0000 7fb279a7b700 1 librados: starting objecter 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.765+0000 7fb279a7b700 1 Processor -- start 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.765+0000 7fb279a7b700 1 -- start start 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.765+0000 7fb279a7b700 1 librados: setting wanted keys 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.765+0000 7fb279a7b700 1 librados: calling monclient init 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb279a7b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2740f6a90 0x7fb2740f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb279a7b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2740fdbd0 0x7fb2740f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb279a7b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740fe530 0x7fb2740f8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb279a7b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2741163d0 con 0x7fb2740f6a90 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb279a7b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb274116540 con 0x7fb2740fe530 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb279a7b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb274116700 con 0x7fb2740fdbd0 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb273fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740fe530 0x7fb2740f8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb273fff700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740fe530 0x7fb2740f8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.87:37668/0 (socket says 172.21.15.87:37668) 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb273fff700 1 -- 172.21.15.87:0/1724646080 learned_addr learned my addr 172.21.15.87:0/1724646080 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb2737fe700 1 --2- 172.21.15.87:0/1724646080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2740f6a90 0x7fb2740f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb272ffd700 1 --2- 172.21.15.87:0/1724646080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2740fdbd0 0x7fb2740f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb273fff700 1 -- 172.21.15.87:0/1724646080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2740fdbd0 msgr2=0x7fb2740f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb273fff700 1 --2- 172.21.15.87:0/1724646080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2740fdbd0 0x7fb2740f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb273fff700 1 -- 172.21.15.87:0/1724646080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2740f6a90 msgr2=0x7fb2740f7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb273fff700 1 --2- 172.21.15.87:0/1724646080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2740f6a90 0x7fb2740f7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb273fff700 1 -- 172.21.15.87:0/1724646080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0097c0 con 0x7fb2740fe530 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb273fff700 1 --2- 172.21.15.87:0/1724646080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740fe530 0x7fb2740f8060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fb26c004ed0 tx=0x7fb26c004d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.766+0000 7fb270f74700 10 client.?.objecter ms_handle_connect 0x7fb2740fe530 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.767+0000 7fb270f74700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.767+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c00e040 con 0x7fb2740fe530 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.767+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb26c018040 con 0x7fb2740fe530 2024-02-23T00:30:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.767+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c01c3f0 con 0x7fb2740fe530 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.767+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb274116980 con 0x7fb2740fe530 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.767+0000 7fb279a7b700 10 client.24661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.767+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb274116d30 con 0x7fb2740fe530 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.767+0000 7fb279a7b700 1 librados: init done 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.768+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb27405fa60 con 0x7fb2740fe530 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.768+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb26c01c820 con 0x7fb2740fe530 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.768+0000 7fb270f74700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb25c062c90 0x7fb25c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(91..91 src has 38..91) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fb26c07d970 con 0x7fb2740fe530 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 osd_map(91..91 src has 38..91) v4 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb270f74700 3 client.24661.objecter handle_osd_map got epochs [91,91] > 0 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb270f74700 3 client.24661.objecter handle_osd_map decoding full epoch 91 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb270f74700 20 client.24661.objecter dump_active .. 0 homeless 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v91) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb26c082220 con 0x7fb2740fe530 2024-02-23T00:30:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb2737fe700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb25c062c90 0x7fb25c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 10 client.24661.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 1 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb2740feee0 0x7fb2740ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27405ee80 osd=0 3 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27405ee80 osd=0 3 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27405ee80 osd=0 3 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 1 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27405ee80 osd=0 3 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 0 1 2024-02-23T00:30:35.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27405ee80 osd=0 4 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 10 client.24661.objecter _send_command 1 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2741070d0 con 0x7fb2740feee0 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb272ffd700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb2740feee0 0x7fb2740ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb2737fe700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb25c062c90 0x7fb25c065150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fb2640048b0 tx=0x7fb26400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.769+0000 7fb270f74700 10 client.24661.objecter ms_handle_connect 0x7fb25c062c90 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.770+0000 7fb270f74700 10 client.24661.objecter ms_handle_connect 0x7fb2740feee0 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.770+0000 7fb272ffd700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb2740feee0 0x7fb2740ff2c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.770+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb2741070d0 con 0x7fb2740feee0 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.770+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.770+0000 7fb270f74700 10 client.24661.objecter _finish_command 1 = system:0 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.770+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27405ee80 osd=0 4 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.770+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 0 1 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 10 client.24661.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 2 2024-02-23T00:30:35.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27405ee80 osd=0 3 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27405ee80 osd=0 3 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27405ee80 osd=0 3 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 2 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27405ee80 osd=0 3 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 0 2 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27405ee80 osd=0 4 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 10 client.24661.objecter _send_command 2 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.785+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb2740273c0 con 0x7fb2740feee0 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb2740273c0 con 0x7fb2740feee0 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb270f74700 10 client.24661.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27405ee80 osd=0 4 2024-02-23T00:30:35.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 0 2 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 10 client.24661.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 3 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6805/685354171 conn(0x7fb2740598f0 0x7fb274059d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb274028710 osd=1 3 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274028710 osd=1 3 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb274028710 osd=1 3 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 3 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb274028710 osd=1 3 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 1 3 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274028710 osd=1 4 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 10 client.24661.objecter _send_command 3 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb27411d6d0 con 0x7fb2740598f0 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.788+0000 7fb2737fe700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6805/685354171 conn(0x7fb2740598f0 0x7fb274059d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.789+0000 7fb270f74700 10 client.24661.objecter ms_handle_connect 0x7fb2740598f0 2024-02-23T00:30:35.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.789+0000 7fb2737fe700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6805/685354171 conn(0x7fb2740598f0 0x7fb274059d10 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.789+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb27411d6d0 con 0x7fb2740598f0 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.790+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.790+0000 7fb270f74700 10 client.24661.objecter _finish_command 3 = system:0 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.790+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb274028710 osd=1 4 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.790+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 1 3 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 10 client.24661.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 4 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb274028710 osd=1 3 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274028710 osd=1 3 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb274028710 osd=1 3 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 4 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb274028710 osd=1 3 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 1 4 2024-02-23T00:30:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274028710 osd=1 4 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 10 client.24661.objecter _send_command 4 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.804+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb274059360 con 0x7fb2740598f0 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb274059360 con 0x7fb2740598f0 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb270f74700 10 client.24661.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb274028710 osd=1 4 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 1 4 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 10 client.24661.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 5 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb27411c0f0 0x7fb27411c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb274027d60 osd=2 3 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274027d60 osd=2 3 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb274027d60 osd=2 3 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 5 2024-02-23T00:30:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb274027d60 osd=2 3 2024-02-23T00:30:35.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 2 5 2024-02-23T00:30:35.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274027d60 osd=2 4 2024-02-23T00:30:35.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 10 client.24661.objecter _send_command 5 2024-02-23T00:30:35.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2740283e0 con 0x7fb27411c0f0 2024-02-23T00:30:35.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb272ffd700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb27411c0f0 0x7fb27411c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb270f74700 10 client.24661.objecter ms_handle_connect 0x7fb27411c0f0 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.807+0000 7fb272ffd700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb27411c0f0 0x7fb27411c530 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.808+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb2740283e0 con 0x7fb27411c0f0 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.808+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.808+0000 7fb270f74700 10 client.24661.objecter _finish_command 5 = system:0 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.808+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb274027d60 osd=2 4 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.808+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 2 5 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 10 client.24661.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 6 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb274027d60 osd=2 3 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274027d60 osd=2 3 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb274027d60 osd=2 3 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 6 2024-02-23T00:30:35.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb274027d60 osd=2 3 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 2 6 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274027d60 osd=2 4 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 10 client.24661.objecter _send_command 6 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.823+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb27411d1f0 con 0x7fb27411c0f0 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.825+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb27411d1f0 con 0x7fb27411c0f0 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.825+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.825+0000 7fb270f74700 10 client.24661.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.825+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb274027d60 osd=2 4 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.825+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 2 6 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 10 client.24661.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 7 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6813/128059790 conn(0x7fb274027940 0x7fb274121d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb2740277d0 osd=3 3 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb2740277d0 osd=3 3 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb2740277d0 osd=3 3 2024-02-23T00:30:35.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 7 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb2740277d0 osd=3 3 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 3 7 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb2740277d0 osd=3 4 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 10 client.24661.objecter _send_command 7 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2741249c0 con 0x7fb274027940 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb273fff700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6813/128059790 conn(0x7fb274027940 0x7fb274121d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb270f74700 10 client.24661.objecter ms_handle_connect 0x7fb274027940 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.826+0000 7fb273fff700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6813/128059790 conn(0x7fb274027940 0x7fb274121d70 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.827+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb2741249c0 con 0x7fb274027940 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.827+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.827+0000 7fb270f74700 10 client.24661.objecter _finish_command 7 = system:0 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.827+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb2740277d0 osd=3 4 2024-02-23T00:30:35.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.827+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 3 7 2024-02-23T00:30:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 10 client.24661.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 8 2024-02-23T00:30:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb2740277d0 osd=3 3 2024-02-23T00:30:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb2740277d0 osd=3 3 2024-02-23T00:30:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb2740277d0 osd=3 3 2024-02-23T00:30:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 8 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb2740277d0 osd=3 3 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 3 8 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb2740277d0 osd=3 4 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 10 client.24661.objecter _send_command 8 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.855+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb27411cc60 con 0x7fb274027940 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb27411cc60 con 0x7fb274027940 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb270f74700 10 client.24661.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb2740277d0 osd=3 4 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 3 8 2024-02-23T00:30:35.861 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:30:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 10 client.24661.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 9 2024-02-23T00:30:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb27411bc50 0x7fb2741233e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27411cc60 osd=4 3 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cc60 osd=4 3 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27411cc60 osd=4 3 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 9 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27411cc60 osd=4 3 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 4 9 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cc60 osd=4 4 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 10 client.24661.objecter _send_command 9 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.858+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb27411cea0 con 0x7fb27411bc50 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.859+0000 7fb2737fe700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb27411bc50 0x7fb2741233e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.859+0000 7fb270f74700 10 client.24661.objecter ms_handle_connect 0x7fb27411bc50 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.859+0000 7fb2737fe700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb27411bc50 0x7fb2741233e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.860+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb27411cea0 con 0x7fb27411bc50 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.860+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:30:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.860+0000 7fb270f74700 10 client.24661.objecter _finish_command 9 = system:0 2024-02-23T00:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.860+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27411cc60 osd=4 4 2024-02-23T00:30:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.860+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 4 9 2024-02-23T00:30:35.877 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:35.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:35.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 10 client.24661.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:35.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 10 2024-02-23T00:30:35.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27411cc60 osd=4 3 2024-02-23T00:30:35.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cc60 osd=4 3 2024-02-23T00:30:35.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27411cc60 osd=4 3 2024-02-23T00:30:35.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 10 2024-02-23T00:30:35.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27411cc60 osd=4 3 2024-02-23T00:30:35.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 4 10 2024-02-23T00:30:35.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cc60 osd=4 4 2024-02-23T00:30:35.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 10 client.24661.objecter _send_command 10 2024-02-23T00:30:35.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.887+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb27411cea0 con 0x7fb27411bc50 2024-02-23T00:30:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb27411cea0 con 0x7fb27411bc50 2024-02-23T00:30:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb270f74700 10 client.24661.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27411cc60 osd=4 4 2024-02-23T00:30:35.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 4 10 2024-02-23T00:30:35.891 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:30:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 10 client.24661.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 11 2024-02-23T00:30:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb274123de0 0x7fb274124220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27411cea0 osd=5 3 2024-02-23T00:30:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cea0 osd=5 3 2024-02-23T00:30:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27411cea0 osd=5 3 2024-02-23T00:30:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 11 2024-02-23T00:30:35.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27411cea0 osd=5 3 2024-02-23T00:30:35.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 5 11 2024-02-23T00:30:35.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cea0 osd=5 4 2024-02-23T00:30:35.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 10 client.24661.objecter _send_command 11 2024-02-23T00:30:35.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.889+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb274124840 con 0x7fb274123de0 2024-02-23T00:30:35.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.890+0000 7fb272ffd700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb274123de0 0x7fb274124220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.890+0000 7fb270f74700 10 client.24661.objecter ms_handle_connect 0x7fb274123de0 2024-02-23T00:30:35.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.890+0000 7fb272ffd700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb274123de0 0x7fb274124220 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.890+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb26800f040 con 0x7fb274123de0 2024-02-23T00:30:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.890+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:30:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.890+0000 7fb270f74700 10 client.24661.objecter _finish_command 11 = system:0 2024-02-23T00:30:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.890+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27411cea0 osd=5 4 2024-02-23T00:30:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.890+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 5 11 2024-02-23T00:30:35.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 10 client.24661.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:35.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 12 2024-02-23T00:30:35.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27411cea0 osd=5 3 2024-02-23T00:30:35.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cea0 osd=5 3 2024-02-23T00:30:35.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27411cea0 osd=5 3 2024-02-23T00:30:35.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 12 2024-02-23T00:30:35.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27411cea0 osd=5 3 2024-02-23T00:30:35.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 5 12 2024-02-23T00:30:35.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cea0 osd=5 4 2024-02-23T00:30:35.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 10 client.24661.objecter _send_command 12 2024-02-23T00:30:35.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.905+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb274128b50 con 0x7fb274123de0 2024-02-23T00:30:35.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb274128b50 con 0x7fb274123de0 2024-02-23T00:30:35.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:35.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb270f74700 10 client.24661.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27411cea0 osd=5 4 2024-02-23T00:30:35.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 5 12 2024-02-23T00:30:35.909 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 10 client.24661.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 13 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb27412a950 0x7fb274128b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27412a2d0 osd=6 3 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412a2d0 osd=6 3 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27412a2d0 osd=6 3 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 13 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27412a2d0 osd=6 3 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 6 13 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412a2d0 osd=6 4 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 10 client.24661.objecter _send_command 13 2024-02-23T00:30:35.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb27412ae90 con 0x7fb27412a950 2024-02-23T00:30:35.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.907+0000 7fb273fff700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb27412a950 0x7fb274128b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.908+0000 7fb270f74700 10 client.24661.objecter ms_handle_connect 0x7fb27412a950 2024-02-23T00:30:35.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.908+0000 7fb273fff700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb27412a950 0x7fb274128b50 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.908+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb27412ae90 con 0x7fb27412a950 2024-02-23T00:30:35.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.908+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:30:35.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.908+0000 7fb270f74700 10 client.24661.objecter _finish_command 13 = system:0 2024-02-23T00:30:35.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.908+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27412a2d0 osd=6 4 2024-02-23T00:30:35.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.908+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 6 13 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 10 client.24661.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 14 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27412a2d0 osd=6 3 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412a2d0 osd=6 3 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27412a2d0 osd=6 3 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 14 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27412a2d0 osd=6 3 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 6 14 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412a2d0 osd=6 4 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 10 client.24661.objecter _send_command 14 2024-02-23T00:30:35.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.923+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb27412ae90 con 0x7fb27412a950 2024-02-23T00:30:35.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb27412ae90 con 0x7fb27412a950 2024-02-23T00:30:35.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:35.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb270f74700 10 client.24661.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27412a2d0 osd=6 4 2024-02-23T00:30:35.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 6 14 2024-02-23T00:30:35.927 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:30:35.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 10 client.24661.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:35.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 15 2024-02-23T00:30:35.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6812/373787000 conn(0x7fb274129550 0x7fb2741299b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:35.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27412ae90 osd=7 3 2024-02-23T00:30:35.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412ae90 osd=7 3 2024-02-23T00:30:35.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27412ae90 osd=7 3 2024-02-23T00:30:35.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 15 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27412ae90 osd=7 3 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 7 15 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412ae90 osd=7 4 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 10 client.24661.objecter _send_command 15 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb274129ea0 con 0x7fb274129550 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb2737fe700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6812/373787000 conn(0x7fb274129550 0x7fb2741299b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb270f74700 10 client.24661.objecter ms_handle_connect 0x7fb274129550 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.925+0000 7fb2737fe700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6812/373787000 conn(0x7fb274129550 0x7fb2741299b0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.926+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb274129ea0 con 0x7fb274129550 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.926+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.926+0000 7fb270f74700 10 client.24661.objecter _finish_command 15 = system:0 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.926+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27412ae90 osd=7 4 2024-02-23T00:30:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.926+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 7 15 2024-02-23T00:30:35.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 10 client.24661.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:30:35.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign -1 16 2024-02-23T00:30:35.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27412ae90 osd=7 3 2024-02-23T00:30:35.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412ae90 osd=7 3 2024-02-23T00:30:35.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 20 client.24661.objecter _get_session s=0x7fb27412ae90 osd=7 3 2024-02-23T00:30:35.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_remove -1 16 2024-02-23T00:30:35.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 20 client.24661.objecter get_session s=0x7fb27412ae90 osd=7 3 2024-02-23T00:30:35.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 15 client.24661.objecter _session_command_op_assign 7 16 2024-02-23T00:30:35.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412ae90 osd=7 4 2024-02-23T00:30:35.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 10 client.24661.objecter _send_command 16 2024-02-23T00:30:35.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.954+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb274129ea0 con 0x7fb274129550 2024-02-23T00:30:35.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb270f74700 1 -- 172.21.15.87:0/1724646080 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb274129ea0 con 0x7fb274129550 2024-02-23T00:30:35.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb270f74700 10 client.24661.objecter ms_dispatch 0x7fb274001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:30:35.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb270f74700 10 client.24661.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb270f74700 20 client.24661.objecter put_session s=0x7fb27412ae90 osd=7 4 2024-02-23T00:30:35.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb270f74700 15 client.24661.objecter _session_command_op_remove 7 16 2024-02-23T00:30:35.958 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:30:35.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 librados: watch_flush enter 2024-02-23T00:30:35.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 librados: watch_flush exit 2024-02-23T00:30:35.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 client.24661.objecter close_session for osd.0 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb2740feee0 msgr2=0x7fb2740ff2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb2740feee0 0x7fb2740ff2c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27405ee80 osd=0 1 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 client.24661.objecter close_session for osd.1 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6805/685354171 conn(0x7fb2740598f0 msgr2=0x7fb274059d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6805/685354171 conn(0x7fb2740598f0 0x7fb274059d10 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274028710 osd=1 1 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 client.24661.objecter close_session for osd.2 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb27411c0f0 msgr2=0x7fb27411c530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb27411c0f0 0x7fb27411c530 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb274027d60 osd=2 1 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 client.24661.objecter close_session for osd.3 2024-02-23T00:30:35.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6813/128059790 conn(0x7fb274027940 msgr2=0x7fb274121d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6813/128059790 conn(0x7fb274027940 0x7fb274121d70 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb2740277d0 osd=3 1 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 client.24661.objecter close_session for osd.4 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb27411bc50 msgr2=0x7fb2741233e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb27411bc50 0x7fb2741233e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb273fff700 1 -- 172.21.15.87:0/1724646080 reap_dead start 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cc60 osd=4 1 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 client.24661.objecter close_session for osd.5 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb274123de0 msgr2=0x7fb274124220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb274123de0 0x7fb274124220 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb273fff700 1 -- 172.21.15.87:0/1724646080 reap_dead start 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb273fff700 1 -- 172.21.15.87:0/1724646080 reap_dead start 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27411cea0 osd=5 1 2024-02-23T00:30:35.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 client.24661.objecter close_session for osd.6 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb27412a950 msgr2=0x7fb274128b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb27412a950 0x7fb274128b50 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412a2d0 osd=6 1 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 10 client.24661.objecter close_session for osd.7 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6812/373787000 conn(0x7fb274129550 msgr2=0x7fb2741299b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.956+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.171:6812/373787000 conn(0x7fb274129550 0x7fb2741299b0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 20 client.24661.objecter put_session s=0x7fb27412ae90 osd=7 1 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 20 client.24661.objecter shutdown clearing up homeless session... 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 10 client.24661.objecter successfully canceled tick 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb25c062c90 msgr2=0x7fb25c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb25c062c90 0x7fb25c065150 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fb2640048b0 tx=0x7fb26400b040).stop 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740fe530 msgr2=0x7fb2740f8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 1 --2- 172.21.15.87:0/1724646080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2740fe530 0x7fb2740f8060 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fb26c004ed0 tx=0x7fb26c004d10).stop 2024-02-23T00:30:35.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb273fff700 1 -- 172.21.15.87:0/1724646080 reap_dead start 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 shutdown_connections 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 >> 172.21.15.87:0/1724646080 conn(0x7fb27406c370 msgr2=0x7fb2740fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 shutdown_connections 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 1 -- 172.21.15.87:0/1724646080 wait complete. 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:35.957+0000 7fb279a7b700 1 librados: shutdown 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:30:35.983 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:36.025 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:34.560956+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:30:36.025 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:34.561762+0000 osd.7 (osd.7) 58 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:30:36.025 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:34.858719+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.b scrub starts 2024-02-23T00:30:36.025 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:34.560956+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:34.561762+0000 osd.7 (osd.7) 58 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:34.858719+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.b scrub starts 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:34.861735+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.b scrub ok 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:34.868550+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:34.869434+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:35.235842+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:35.628500+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:35.629274+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:35.634553+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:36.026 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:35.770153+0000 mon.b (mon.1) 33 : audit [DBG] from='client.? 172.21.15.87:0/1724646080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:36.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:34.861735+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.b scrub ok 2024-02-23T00:30:36.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:34.868550+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:30:36.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:34.869434+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:30:36.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:35.235842+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:30:36.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:35.628500+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:30:36.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:35.629274+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:30:36.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:35.634553+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:36.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:35.770153+0000 mon.b (mon.1) 33 : audit [DBG] from='client.? 172.21.15.87:0/1724646080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:36.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 -- 172.21.15.87:0/119780992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb0c0f6b10 msgr2=0x7fbb0c1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:36.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 --2- 172.21.15.87:0/119780992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb0c0f6b10 0x7fbb0c1030b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fbb04004d30 tx=0x7fbb0400bd30).stop 2024-02-23T00:30:36.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 -- 172.21.15.87:0/119780992 shutdown_connections 2024-02-23T00:30:36.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 --2- 172.21.15.87:0/119780992 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb0c0f6b10 0x7fbb0c1030b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:36.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 --2- 172.21.15.87:0/119780992 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb0c0fc240 0x7fbb0c0fc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:36.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 --2- 172.21.15.87:0/119780992 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb0c0fcba0 0x7fbb0c0f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:36.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 -- 172.21.15.87:0/119780992 >> 172.21.15.87:0/119780992 conn(0x7fbb0c06c370 msgr2=0x7fbb0c06c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:36.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 -- 172.21.15.87:0/119780992 shutdown_connections 2024-02-23T00:30:36.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 -- 172.21.15.87:0/119780992 wait complete. 2024-02-23T00:30:36.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 librados: starting msgr at 2024-02-23T00:30:36.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.426+0000 7fbb141de700 1 librados: starting objecter 2024-02-23T00:30:36.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 Processor -- start 2024-02-23T00:30:36.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 -- start start 2024-02-23T00:30:36.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 librados: setting wanted keys 2024-02-23T00:30:36.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 librados: calling monclient init 2024-02-23T00:30:36.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb0c0f6b10 0x7fbb0c10bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:36.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb0c0fc240 0x7fbb0c10c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:36.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb0c0fcba0 0x7fbb0c105e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:36.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c106370 con 0x7fbb0c0fcba0 2024-02-23T00:30:36.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c1064e0 con 0x7fbb0c0fc240 2024-02-23T00:30:36.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb141de700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0c1066a0 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb11f7a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb0c0f6b10 0x7fbb0c10bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb11f7a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb0c0f6b10 0x7fbb0c10bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:48646/0 (socket says 172.21.15.87:48646) 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb11f7a700 1 -- 172.21.15.87:0/3262819577 learned_addr learned my addr 172.21.15.87:0/3262819577 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb11779700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb0c0fc240 0x7fbb0c10c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.427+0000 7fbb1277b700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb0c0fcba0 0x7fbb0c105e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb11f7a700 1 -- 172.21.15.87:0/3262819577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb0c0fc240 msgr2=0x7fbb0c10c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb11f7a700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb0c0fc240 0x7fbb0c10c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb11f7a700 1 -- 172.21.15.87:0/3262819577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb0c0fcba0 msgr2=0x7fbb0c105e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb11f7a700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb0c0fcba0 0x7fbb0c105e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb11f7a700 1 -- 172.21.15.87:0/3262819577 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb040048f0 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb1277b700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb0c0fcba0 0x7fbb0c105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb11f7a700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb0c0f6b10 0x7fbb0c10bce0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fbb0800a520 tx=0x7fbb0800c040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb02ffd700 10 client.?.objecter ms_handle_connect 0x7fbb0c0f6b10 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb02ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:36.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb02ffd700 1 -- 172.21.15.87:0/3262819577 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0800a9b0 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb02ffd700 1 -- 172.21.15.87:0/3262819577 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb08008bd0 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb02ffd700 1 -- 172.21.15.87:0/3262819577 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08013b60 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0c106980 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb141de700 10 client.24740.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:36.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0c062700 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.428+0000 7fbb141de700 1 librados: init done 2024-02-23T00:30:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.429+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0c05fa60 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.430+0000 7fbb02ffd700 1 -- 172.21.15.87:0/3262819577 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fbb0800c7b0 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.431+0000 7fbb02ffd700 1 --2- 172.21.15.87:0/3262819577 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaf8062ce0 0x7fbaf80651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:36.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.431+0000 7fbb02ffd700 1 -- 172.21.15.87:0/3262819577 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(91..91 src has 38..91) v4 ==== 6172+0+0 (secure 0 0 0) 0x7fbb0800f070 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.431+0000 7fbb02ffd700 10 client.24740.objecter ms_dispatch 0x7fbb0c001550 osd_map(91..91 src has 38..91) v4 2024-02-23T00:30:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.431+0000 7fbb02ffd700 3 client.24740.objecter handle_osd_map got epochs [91,91] > 0 2024-02-23T00:30:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.431+0000 7fbb02ffd700 3 client.24740.objecter handle_osd_map decoding full epoch 91 2024-02-23T00:30:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.431+0000 7fbb11779700 1 --2- 172.21.15.87:0/3262819577 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaf8062ce0 0x7fbaf80651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.431+0000 7fbb02ffd700 20 client.24740.objecter dump_active .. 0 homeless 2024-02-23T00:30:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.431+0000 7fbb02ffd700 10 client.24740.objecter ms_handle_connect 0x7fbaf8062ce0 2024-02-23T00:30:36.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.431+0000 7fbb11779700 1 --2- 172.21.15.87:0/3262819577 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaf8062ce0 0x7fbaf80651a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fbb0c107620 tx=0x7fbafc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:36.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.435+0000 7fbb02ffd700 1 -- 172.21.15.87:0/3262819577 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbb08050a80 con 0x7fbb0c0f6b10 2024-02-23T00:30:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:34.560956+0000 osd.7 (osd.7) 57 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:30:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:34.561762+0000 osd.7 (osd.7) 58 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:30:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:34.858719+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.b scrub starts 2024-02-23T00:30:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:34.861735+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.b scrub ok 2024-02-23T00:30:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:34.868550+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:30:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:34.869434+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:30:36.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:35.235842+0000 mon.a (mon.0) 759 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:30:36.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:35.628500+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:30:36.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:35.629274+0000 mon.a (mon.0) 761 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:30:36.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:35.634553+0000 mon.a (mon.0) 762 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:36.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:35.770153+0000 mon.b (mon.1) 33 : audit [DBG] from='client.? 172.21.15.87:0/1724646080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:36.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:36.666+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fbb0c05ee80 con 0x7fbb0c0f6b10 2024-02-23T00:30:37.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.028+0000 7fbb02ffd700 1 -- 172.21.15.87:0/3262819577 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v92) v1 ==== 101+0+0 (secure 0 0 0) 0x7fbb08050670 con 0x7fbb0c0f6b10 2024-02-23T00:30:37.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.030+0000 7fbb141de700 10 librados: watch_flush enter 2024-02-23T00:30:37.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 10 librados: watch_flush exit 2024-02-23T00:30:37.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 20 client.24740.objecter shutdown clearing up homeless session... 2024-02-23T00:30:37.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 10 client.24740.objecter successfully canceled tick 2024-02-23T00:30:37.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaf8062ce0 msgr2=0x7fbaf80651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:37.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 --2- 172.21.15.87:0/3262819577 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaf8062ce0 0x7fbaf80651a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fbb0c107620 tx=0x7fbafc00b040).stop 2024-02-23T00:30:37.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb0c0f6b10 msgr2=0x7fbb0c10bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:37.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb0c0f6b10 0x7fbb0c10bce0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fbb0800a520 tx=0x7fbb0800c040).stop 2024-02-23T00:30:37.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 shutdown_connections 2024-02-23T00:30:37.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 --2- 172.21.15.87:0/3262819577 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaf8062ce0 0x7fbaf80651a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:37.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb0c0fcba0 0x7fbb0c105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:37.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb0c0f6b10 0x7fbb0c10bce0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:37.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 --2- 172.21.15.87:0/3262819577 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb0c0fc240 0x7fbb0c10c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:37.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.031+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 >> 172.21.15.87:0/3262819577 conn(0x7fbb0c06c370 msgr2=0x7fbb0c0f50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:37.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.032+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 shutdown_connections 2024-02-23T00:30:37.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.032+0000 7fbb141de700 1 -- 172.21.15.87:0/3262819577 wait complete. 2024-02-23T00:30:37.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:37.032+0000 7fbb141de700 1 librados: shutdown 2024-02-23T00:30:37.035 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:30:37.180 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:37.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:35.117318+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.d scrub starts 2024-02-23T00:30:37.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:35.120904+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.d scrub ok 2024-02-23T00:30:37.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:35.497015+0000 mgr.y (mgr.24437) 136 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:37.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:35.522015+0000 osd.7 (osd.7) 59 : cluster [DBG] 6.1c scrub starts 2024-02-23T00:30:37.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:35.523153+0000 osd.7 (osd.7) 60 : cluster [DBG] 6.1c scrub ok 2024-02-23T00:30:37.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:35.842361+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:30:37.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:35.845379+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:30:37.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:36.668537+0000 mon.c (mon.2) 29 : audit [INF] from='client.? 172.21.15.87:0/3262819577' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:30:37.187 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:36.668910+0000 mon.a (mon.0) 763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:30:37.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:35.117318+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.d scrub starts 2024-02-23T00:30:37.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:35.120904+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.d scrub ok 2024-02-23T00:30:37.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:35.497015+0000 mgr.y (mgr.24437) 136 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:37.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:35.522015+0000 osd.7 (osd.7) 59 : cluster [DBG] 6.1c scrub starts 2024-02-23T00:30:37.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:35.523153+0000 osd.7 (osd.7) 60 : cluster [DBG] 6.1c scrub ok 2024-02-23T00:30:37.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:35.842361+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:30:37.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:35.845379+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:30:37.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:36.668537+0000 mon.c (mon.2) 29 : audit [INF] from='client.? 172.21.15.87:0/3262819577' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:30:37.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:36.668910+0000 mon.a (mon.0) 763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:30:37.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:37.290 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:37.320 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:35.117318+0000 osd.4 (osd.4) 63 : cluster [DBG] 4.d scrub starts 2024-02-23T00:30:37.320 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:35.120904+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.d scrub ok 2024-02-23T00:30:37.320 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:35.497015+0000 mgr.y (mgr.24437) 136 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:37.320 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:35.522015+0000 osd.7 (osd.7) 59 : cluster [DBG] 6.1c scrub starts 2024-02-23T00:30:37.320 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:35.523153+0000 osd.7 (osd.7) 60 : cluster [DBG] 6.1c scrub ok 2024-02-23T00:30:37.320 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:35.842361+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:30:37.321 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:35.845379+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:30:37.321 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:36.668537+0000 mon.c (mon.2) 29 : audit [INF] from='client.? 172.21.15.87:0/3262819577' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:30:37.321 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:36.668910+0000 mon.a (mon.0) 763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:30:37.322 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:37.344 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:37.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:37.448 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:37.477 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:37.497 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:37.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:37.601 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:37.630 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:37.874 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:37.878 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:37.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:37.986 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:38.011 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:38.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:36.893289+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:30:38.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:36.894214+0000 osd.3 (osd.3) 80 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:30:38.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.018268+0000 mon.a (mon.0) 764 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:30:38.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.018295+0000 mon.a (mon.0) 765 : cluster [INF] Cluster is now healthy 2024-02-23T00:30:38.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:37.028959+0000 mon.a (mon.0) 766 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:30:38.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.029025+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-23T00:30:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:36.893289+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:30:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:36.894214+0000 osd.3 (osd.3) 80 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:30:38.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.018268+0000 mon.a (mon.0) 764 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:30:38.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.018295+0000 mon.a (mon.0) 765 : cluster [INF] Cluster is now healthy 2024-02-23T00:30:38.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:37.028959+0000 mon.a (mon.0) 766 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:30:38.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.029025+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-23T00:30:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:36.893289+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:30:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:36.894214+0000 osd.3 (osd.3) 80 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:30:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.018268+0000 mon.a (mon.0) 764 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:30:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.018295+0000 mon.a (mon.0) 765 : cluster [INF] Cluster is now healthy 2024-02-23T00:30:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:37.028959+0000 mon.a (mon.0) 766 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:30:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.029025+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-23T00:30:38.663 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:38.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:38.769 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:38.801 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:38.823 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:38.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:38.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:38.927 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:38.956 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:38.975 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:38.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:39.079 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:39.104 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.497406+0000 mgr.y (mgr.24437) 137 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:39.104 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.529310+0000 osd.7 (osd.7) 61 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:30:39.104 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.532434+0000 osd.7 (osd.7) 62 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:30:39.105 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.888127+0000 osd.0 (osd.0) 71 : cluster [DBG] 6.16 deep-scrub starts 2024-02-23T00:30:39.105 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.889332+0000 osd.0 (osd.0) 72 : cluster [DBG] 6.16 deep-scrub ok 2024-02-23T00:30:39.105 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.896389+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:30:39.105 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:37.899839+0000 osd.3 (osd.3) 82 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:30:39.105 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:38.430997+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:30:39.105 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:38.432672+0000 osd.1 (osd.1) 68 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:30:39.106 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:39.125 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:39.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:39.229 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:39.255 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:39.274 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:39.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:39.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.497406+0000 mgr.y (mgr.24437) 137 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:39.321 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.529310+0000 osd.7 (osd.7) 61 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:30:39.321 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.532434+0000 osd.7 (osd.7) 62 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:30:39.321 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.888127+0000 osd.0 (osd.0) 71 : cluster [DBG] 6.16 deep-scrub starts 2024-02-23T00:30:39.321 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.889332+0000 osd.0 (osd.0) 72 : cluster [DBG] 6.16 deep-scrub ok 2024-02-23T00:30:39.321 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.896389+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:30:39.321 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:37.899839+0000 osd.3 (osd.3) 82 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:30:39.321 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:38.430997+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:30:39.321 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:38.432672+0000 osd.1 (osd.1) 68 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:30:39.321 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.497406+0000 mgr.y (mgr.24437) 137 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:39.322 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.529310+0000 osd.7 (osd.7) 61 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:30:39.322 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.532434+0000 osd.7 (osd.7) 62 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:30:39.322 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.888127+0000 osd.0 (osd.0) 71 : cluster [DBG] 6.16 deep-scrub starts 2024-02-23T00:30:39.322 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.889332+0000 osd.0 (osd.0) 72 : cluster [DBG] 6.16 deep-scrub ok 2024-02-23T00:30:39.322 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.896389+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:30:39.322 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:37.899839+0000 osd.3 (osd.3) 82 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:30:39.322 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:38.430997+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:30:39.322 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:38.432672+0000 osd.1 (osd.1) 68 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:30:39.378 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:39.404 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:39.423 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:39.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:39.526 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:39.556 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:39.577 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:39.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:39.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:39.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:30:39.682 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:39.708 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:39.727 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:39.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr:Note: switching to '4f758cd8e8854d1f126da86e254076852bf059a9'. 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr:state without impacting any branches by switching back to a branch. 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr: git switch -c 2024-02-23T00:30:39.809 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-02-23T00:30:39.810 INFO:tasks.workunit.client.0.smithi087.stderr:Or undo this operation with: 2024-02-23T00:30:39.810 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-02-23T00:30:39.810 INFO:tasks.workunit.client.0.smithi087.stderr: git switch - 2024-02-23T00:30:39.810 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-02-23T00:30:39.810 INFO:tasks.workunit.client.0.smithi087.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-23T00:30:39.810 INFO:tasks.workunit.client.0.smithi087.stderr: 2024-02-23T00:30:39.810 INFO:tasks.workunit.client.0.smithi087.stderr:HEAD is now at 4f758cd8e88 [DNM] qa/suites: added more whitelisting 2024-02-23T00:30:39.814 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-23T00:30:39.831 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:39.884 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:39.909 INFO:tasks.workunit.client.0.smithi087.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-23T00:30:39.911 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-23T00:30:39.911 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-23T00:30:39.948 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:30:39.949 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:30:39.949 INFO:teuthology.orchestra.run.smithi087.stdout: "complaint_time": 30, 2024-02-23T00:30:39.949 INFO:teuthology.orchestra.run.smithi087.stdout: "num_blocked_ops": 0 2024-02-23T00:30:39.949 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:30:40.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:38.548301+0000 osd.7 (osd.7) 63 : cluster [DBG] 5.1a deep-scrub starts 2024-02-23T00:30:40.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:38.549221+0000 osd.7 (osd.7) 64 : cluster [DBG] 5.1a deep-scrub ok 2024-02-23T00:30:40.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:38.735623+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.a scrub starts 2024-02-23T00:30:40.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:38.736485+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.a scrub ok 2024-02-23T00:30:40.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:38.896279+0000 osd.0 (osd.0) 73 : cluster [DBG] 4.1e deep-scrub starts 2024-02-23T00:30:40.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:38.900753+0000 osd.0 (osd.0) 74 : cluster [DBG] 4.1e deep-scrub ok 2024-02-23T00:30:40.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:39.624799+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:40.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:38.548301+0000 osd.7 (osd.7) 63 : cluster [DBG] 5.1a deep-scrub starts 2024-02-23T00:30:40.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:38.549221+0000 osd.7 (osd.7) 64 : cluster [DBG] 5.1a deep-scrub ok 2024-02-23T00:30:40.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:38.735623+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.a scrub starts 2024-02-23T00:30:40.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:38.736485+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.a scrub ok 2024-02-23T00:30:40.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:38.896279+0000 osd.0 (osd.0) 73 : cluster [DBG] 4.1e deep-scrub starts 2024-02-23T00:30:40.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:38.900753+0000 osd.0 (osd.0) 74 : cluster [DBG] 4.1e deep-scrub ok 2024-02-23T00:30:40.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:39.624799+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:38.548301+0000 osd.7 (osd.7) 63 : cluster [DBG] 5.1a deep-scrub starts 2024-02-23T00:30:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:38.549221+0000 osd.7 (osd.7) 64 : cluster [DBG] 5.1a deep-scrub ok 2024-02-23T00:30:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:38.735623+0000 osd.2 (osd.2) 55 : cluster [DBG] 5.a scrub starts 2024-02-23T00:30:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:38.736485+0000 osd.2 (osd.2) 56 : cluster [DBG] 5.a scrub ok 2024-02-23T00:30:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:38.896279+0000 osd.0 (osd.0) 73 : cluster [DBG] 4.1e deep-scrub starts 2024-02-23T00:30:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:38.900753+0000 osd.0 (osd.0) 74 : cluster [DBG] 4.1e deep-scrub ok 2024-02-23T00:30:40.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:39.624799+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:40.661 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:30:40.935 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:40.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:40.941 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:40.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:30:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:30:41.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:41.044 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:39.497806+0000 mgr.y (mgr.24437) 138 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:41.044 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:39.567755+0000 osd.7 (osd.7) 65 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:30:41.045 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:39.570663+0000 osd.7 (osd.7) 66 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:30:41.045 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:40.180564+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.1a deep-scrub starts 2024-02-23T00:30:41.045 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:40.183449+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.1a deep-scrub ok 2024-02-23T00:30:41.045 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:40.497427+0000 osd.3 (osd.3) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:41.045 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:40.497478+0000 osd.3 (osd.3) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:41.071 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:41.092 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:41.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:41.196 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:41.225 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:41.244 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:41.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:41.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:39.497806+0000 mgr.y (mgr.24437) 138 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:41.332 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:39.567755+0000 osd.7 (osd.7) 65 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:30:41.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:39.570663+0000 osd.7 (osd.7) 66 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:30:41.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:40.180564+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.1a deep-scrub starts 2024-02-23T00:30:41.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:40.183449+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.1a deep-scrub ok 2024-02-23T00:30:41.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:40.497427+0000 osd.3 (osd.3) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:41.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:40.497478+0000 osd.3 (osd.3) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:41.348 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:41.380 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:41.387 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-23T00:30:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:39.497806+0000 mgr.y (mgr.24437) 138 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:30:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:39.567755+0000 osd.7 (osd.7) 65 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:30:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:39.570663+0000 osd.7 (osd.7) 66 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:30:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:40.180564+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.1a deep-scrub starts 2024-02-23T00:30:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:40.183449+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.1a deep-scrub ok 2024-02-23T00:30:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:40.497427+0000 osd.3 (osd.3) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:40.497478+0000 osd.3 (osd.3) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:41.522 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-23T00:30:41.563 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-23T00:30:41.627 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-23T00:30:41.629 INFO:tasks.workunit.client.0.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-23T00:30:41.629 INFO:tasks.workunit.client.0.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-23T00:30:41.630 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2024-02-23T00:30:41.631 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-23T00:30:41.659 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:41.679 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-02-23T00:30:41.680 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-02-23T00:30:41.680 DEBUG:teuthology.orchestra.run.smithi087:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f758cd8e8854d1f126da86e254076852bf059a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-02-23T00:30:41.774 INFO:tasks.workunit.client.0.smithi087.stderr:+ parallel=1 2024-02-23T00:30:41.774 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' '' = --serial ']' 2024-02-23T00:30:41.774 INFO:tasks.workunit.client.0.smithi087.stderr:+ color= 2024-02-23T00:30:41.774 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' -t 1 ']' 2024-02-23T00:30:41.774 INFO:tasks.workunit.client.0.smithi087.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-02-23T00:30:41.775 INFO:tasks.workunit.client.0.smithi087.stderr:+ declare -A pids 2024-02-23T00:30:41.775 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.775 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.775 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_aio 2024-02-23T00:30:41.775 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_aio' 2024-02-23T00:30:41.775 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_aio 2024-02-23T00:30:41.775 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.776 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_aio 2024-02-23T00:30:41.776 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82006 2024-02-23T00:30:41.777 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_aio on pid 82006' 2024-02-23T00:30:41.777 INFO:tasks.workunit.client.0.smithi087.stdout:test api_aio on pid 82006 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82006 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_aio_pp 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_aio_pp' 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_aio_pp 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_aio 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_aio.log 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_aio: /' 2024-02-23T00:30:41.779 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_aio_pp 2024-02-23T00:30:41.780 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82014 2024-02-23T00:30:41.780 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_aio_pp on pid 82014' 2024-02-23T00:30:41.780 INFO:tasks.workunit.client.0.smithi087.stdout:test api_aio_pp on pid 82014 2024-02-23T00:30:41.781 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82014 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_io 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_io' 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_io 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_aio_pp 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_aio_pp: /' 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_io 2024-02-23T00:30:41.782 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82022 2024-02-23T00:30:41.783 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_io on pid 82022' 2024-02-23T00:30:41.783 INFO:tasks.workunit.client.0.smithi087.stdout:test api_io on pid 82022 2024-02-23T00:30:41.784 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82022 2024-02-23T00:30:41.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-02-23T00:30:41.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.785 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_io_pp 2024-02-23T00:30:41.785 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_io_pp' 2024-02-23T00:30:41.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_io_pp 2024-02-23T00:30:41.786 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.786 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_io 2024-02-23T00:30:41.786 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_io.log 2024-02-23T00:30:41.786 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_io: /' 2024-02-23T00:30:41.786 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_io_pp 2024-02-23T00:30:41.786 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82030 2024-02-23T00:30:41.786 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_io_pp on pid 82030' 2024-02-23T00:30:41.787 INFO:tasks.workunit.client.0.smithi087.stdout:test api_io_pp on pid 82030 2024-02-23T00:30:41.788 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82030 2024-02-23T00:30:41.788 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.788 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_asio 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_asio' 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_asio 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_io_pp 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_io_pp: /' 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_asio 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82038 2024-02-23T00:30:41.789 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_asio on pid 82038' 2024-02-23T00:30:41.790 INFO:tasks.workunit.client.0.smithi087.stdout:test api_asio on pid 82038 2024-02-23T00:30:41.791 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82038 2024-02-23T00:30:41.791 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_list 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_list' 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_list 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_asio 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_asio.log 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_asio: /' 2024-02-23T00:30:41.792 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_list 2024-02-23T00:30:41.793 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82046 2024-02-23T00:30:41.793 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_list on pid 82046' 2024-02-23T00:30:41.793 INFO:tasks.workunit.client.0.smithi087.stdout:test api_list on pid 82046 2024-02-23T00:30:41.794 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82046 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_lock 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_lock' 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_lock 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_list 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_list.log 2024-02-23T00:30:41.795 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_list: /' 2024-02-23T00:30:41.796 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_lock 2024-02-23T00:30:41.796 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82054 2024-02-23T00:30:41.796 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_lock on pid 82054' 2024-02-23T00:30:41.796 INFO:tasks.workunit.client.0.smithi087.stdout:test api_lock on pid 82054 2024-02-23T00:30:41.797 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82054 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_lock_pp 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_lock_pp' 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_lock_pp 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_lock 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_lock.log 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_lock: /' 2024-02-23T00:30:41.798 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_lock_pp 2024-02-23T00:30:41.799 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82062 2024-02-23T00:30:41.799 INFO:tasks.workunit.client.0.smithi087.stdout:test api_lock_pp on pid 82062 2024-02-23T00:30:41.800 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_lock_pp on pid 82062' 2024-02-23T00:30:41.800 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82062 2024-02-23T00:30:41.800 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_misc 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_misc' 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_misc 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_lock_pp 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_lock_pp: /' 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_misc 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82070 2024-02-23T00:30:41.801 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_misc on pid 82070' 2024-02-23T00:30:41.802 INFO:tasks.workunit.client.0.smithi087.stdout:test api_misc on pid 82070 2024-02-23T00:30:41.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82070 2024-02-23T00:30:41.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-02-23T00:30:41.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_misc_pp 2024-02-23T00:30:41.803 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_misc_pp' 2024-02-23T00:30:41.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_misc_pp 2024-02-23T00:30:41.803 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.804 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_misc 2024-02-23T00:30:41.804 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_misc.log 2024-02-23T00:30:41.804 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_misc: /' 2024-02-23T00:30:41.804 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_misc_pp 2024-02-23T00:30:41.804 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82078 2024-02-23T00:30:41.804 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_misc_pp on pid 82078' 2024-02-23T00:30:41.805 INFO:tasks.workunit.client.0.smithi087.stdout:test api_misc_pp on pid 82078 2024-02-23T00:30:41.806 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82078 2024-02-23T00:30:41.806 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.806 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_tier_pp 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_tier_pp' 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_tier_pp 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_misc_pp 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_misc_pp: /' 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_tier_pp 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82086 2024-02-23T00:30:41.807 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_tier_pp on pid 82086' 2024-02-23T00:30:41.808 INFO:tasks.workunit.client.0.smithi087.stdout:test api_tier_pp on pid 82086 2024-02-23T00:30:41.810 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82086 2024-02-23T00:30:41.810 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.810 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-02-23T00:30:41.810 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.810 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_pool 2024-02-23T00:30:41.810 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_pool' 2024-02-23T00:30:41.811 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_pool 2024-02-23T00:30:41.811 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.811 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_tier_pp 2024-02-23T00:30:41.811 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-02-23T00:30:41.811 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_tier_pp: /' 2024-02-23T00:30:41.811 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_pool 2024-02-23T00:30:41.811 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82094 2024-02-23T00:30:41.811 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_pool on pid 82094' 2024-02-23T00:30:41.811 INFO:tasks.workunit.client.0.smithi087.stdout:test api_pool on pid 82094 2024-02-23T00:30:41.813 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82094 2024-02-23T00:30:41.813 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.813 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-02-23T00:30:41.813 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_snapshots 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_snapshots' 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_snapshots 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_pool 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_pool.log 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_pool: /' 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_snapshots 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82102 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_snapshots on pid 82102' 2024-02-23T00:30:41.814 INFO:tasks.workunit.client.0.smithi087.stdout:test api_snapshots on pid 82102 2024-02-23T00:30:41.816 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82102 2024-02-23T00:30:41.816 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.816 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.816 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-02-23T00:30:41.816 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_snapshots_pp 2024-02-23T00:30:41.816 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_snapshots_pp' 2024-02-23T00:30:41.816 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_snapshots_pp 2024-02-23T00:30:41.816 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.816 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_snapshots 2024-02-23T00:30:41.817 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-02-23T00:30:41.817 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_snapshots: /' 2024-02-23T00:30:41.817 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_snapshots_pp 2024-02-23T00:30:41.817 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82110 2024-02-23T00:30:41.817 INFO:tasks.workunit.client.0.smithi087.stdout:test api_snapshots_pp on pid 82110 2024-02-23T00:30:41.818 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_snapshots_pp on pid 82110' 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82110 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_stat 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_stat' 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_stat 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_snapshots_pp 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_stat 2024-02-23T00:30:41.819 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82118 2024-02-23T00:30:41.820 INFO:tasks.workunit.client.0.smithi087.stdout:test api_stat on pid 82118 2024-02-23T00:30:41.822 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_stat on pid 82118' 2024-02-23T00:30:41.823 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82118 2024-02-23T00:30:41.823 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.823 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.823 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-02-23T00:30:41.823 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_stat_pp 2024-02-23T00:30:41.823 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_stat_pp' 2024-02-23T00:30:41.823 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_stat_pp 2024-02-23T00:30:41.823 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.823 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_stat 2024-02-23T00:30:41.824 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_stat.log 2024-02-23T00:30:41.824 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_stat: /' 2024-02-23T00:30:41.824 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_stat_pp 2024-02-23T00:30:41.824 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82126 2024-02-23T00:30:41.824 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_stat_pp on pid 82126' 2024-02-23T00:30:41.824 INFO:tasks.workunit.client.0.smithi087.stdout:test api_stat_pp on pid 82126 2024-02-23T00:30:41.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82126 2024-02-23T00:30:41.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-02-23T00:30:41.826 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_watch_notify 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_watch_notify' 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_watch_notify 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_stat_pp 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_stat_pp: /' 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_watch_notify 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82134 2024-02-23T00:30:41.827 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_watch_notify on pid 82134' 2024-02-23T00:30:41.829 INFO:tasks.workunit.client.0.smithi087.stdout:test api_watch_notify on pid 82134 2024-02-23T00:30:41.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82134 2024-02-23T00:30:41.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-02-23T00:30:41.830 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.830 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_watch_notify_pp 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_watch_notify_pp' 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_watch_notify_pp 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_watch_notify 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_watch_notify: /' 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_watch_notify_pp 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82142 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_watch_notify_pp on pid 82142' 2024-02-23T00:30:41.831 INFO:tasks.workunit.client.0.smithi087.stdout:test api_watch_notify_pp on pid 82142 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82142 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_cmd 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_cmd' 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_cmd 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_cmd 2024-02-23T00:30:41.833 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82150 2024-02-23T00:30:41.834 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_cmd on pid 82150' 2024-02-23T00:30:41.834 INFO:tasks.workunit.client.0.smithi087.stdout:test api_cmd on pid 82150 2024-02-23T00:30:41.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82150 2024-02-23T00:30:41.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.835 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-02-23T00:30:41.835 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_cmd_pp 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_cmd_pp' 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_cmd_pp 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_cmd 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_cmd.log 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_cmd: /' 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_cmd_pp 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82158 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_cmd_pp on pid 82158' 2024-02-23T00:30:41.836 INFO:tasks.workunit.client.0.smithi087.stdout:test api_cmd_pp on pid 82158 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82158 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_service 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_service' 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_service 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_cmd_pp 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-02-23T00:30:41.838 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-02-23T00:30:41.839 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_service 2024-02-23T00:30:41.839 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82166 2024-02-23T00:30:41.839 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_service on pid 82166' 2024-02-23T00:30:41.839 INFO:tasks.workunit.client.0.smithi087.stdout:test api_service on pid 82166 2024-02-23T00:30:41.840 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82166 2024-02-23T00:30:41.840 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.840 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.840 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-02-23T00:30:41.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_service_pp 2024-02-23T00:30:41.840 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_service_pp' 2024-02-23T00:30:41.840 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.840 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_service 2024-02-23T00:30:41.841 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_service.log 2024-02-23T00:30:41.841 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_service_pp 2024-02-23T00:30:41.841 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_service_pp 2024-02-23T00:30:41.841 INFO:tasks.workunit.client.0.smithi087.stdout:test api_service_pp on pid 82174 2024-02-23T00:30:41.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82174 2024-02-23T00:30:41.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_service_pp on pid 82174' 2024-02-23T00:30:41.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82174 2024-02-23T00:30:41.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_service: /' 2024-02-23T00:30:41.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.842 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-02-23T00:30:41.842 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_c_write_operations 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_c_write_operations' 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_c_write_operations 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_service_pp 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_service_pp: /' 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_c_write_operations 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82182 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_c_write_operations on pid 82182' 2024-02-23T00:30:41.843 INFO:tasks.workunit.client.0.smithi087.stdout:test api_c_write_operations on pid 82182 2024-02-23T00:30:41.844 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82182 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s api_c_read_operations 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' api_c_read_operations' 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo api_c_read_operations 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_c_write_operations 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-02-23T00:30:41.845 INFO:tasks.workunit.client.0.smithi087.stdout:test api_c_read_operations on pid 82190 2024-02-23T00:30:41.846 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=api_c_read_operations 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82190 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test api_c_read_operations on pid 82190' 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82190 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s list_parallel 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' list_parallel' 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo list_parallel 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_api_c_read_operations 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-02-23T00:30:41.847 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=list_parallel 2024-02-23T00:30:41.848 INFO:tasks.workunit.client.0.smithi087.stdout:test list_parallel on pid 82198 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82198 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test list_parallel on pid 82198' 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82198 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s open_pools_parallel 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' open_pools_parallel' 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo open_pools_parallel 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_list_parallel 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_list_parallel.log 2024-02-23T00:30:41.849 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ list_parallel: /' 2024-02-23T00:30:41.850 INFO:tasks.workunit.client.0.smithi087.stdout:test open_pools_parallel on pid 82206 2024-02-23T00:30:41.851 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=open_pools_parallel 2024-02-23T00:30:41.851 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82206 2024-02-23T00:30:41.851 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test open_pools_parallel on pid 82206' 2024-02-23T00:30:41.851 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82206 2024-02-23T00:30:41.851 INFO:tasks.workunit.client.0.smithi087.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-23T00:30:41.851 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.851 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-02-23T00:30:41.851 INFO:tasks.workunit.client.0.smithi087.stderr:++ printf %25s delete_pools_parallel 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ r=' delete_pools_parallel' 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stderr:++ echo delete_pools_parallel 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stderr:++ awk '{print $1}' 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_open_pools_parallel 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ ff=delete_pools_parallel 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82214 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stderr:+ echo 'test delete_pools_parallel on pid 82214' 2024-02-23T00:30:41.852 INFO:tasks.workunit.client.0.smithi087.stdout:test delete_pools_parallel on pid 82214 2024-02-23T00:30:41.853 INFO:tasks.workunit.client.0.smithi087.stderr:+ pids[$f]=82214 2024-02-23T00:30:41.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ ret=0 2024-02-23T00:30:41.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-02-23T00:30:41.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ '[' 1 -eq 1 ']' 2024-02-23T00:30:41.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:30:41.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82102 2024-02-23T00:30:41.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82102 2024-02-23T00:30:41.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ ceph_test_rados_delete_pools_parallel 2024-02-23T00:30:41.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-02-23T00:30:41.854 INFO:tasks.workunit.client.0.smithi087.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-02-23T00:30:42.032 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:42.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:42.183 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:42.212 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:42.233 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:42.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:42.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:42.300 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:42] "GET /metrics HTTP/1.1" 200 172248 "" "Prometheus/2.33.4" 2024-02-23T00:30:42.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:40.143431+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:30:42.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:40.144217+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:30:42.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:40.618182+0000 mgr.y (mgr.24437) 139 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:42.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:40.786265+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.b deep-scrub starts 2024-02-23T00:30:42.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:40.787187+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.b deep-scrub ok 2024-02-23T00:30:42.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:41.040094+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-23T00:30:42.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:41.076028+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:30:42.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:41.077672+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:30:42.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:40.143431+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:30:42.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:40.144217+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:30:42.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:40.618182+0000 mgr.y (mgr.24437) 139 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:42.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:40.786265+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.b deep-scrub starts 2024-02-23T00:30:42.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:40.787187+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.b deep-scrub ok 2024-02-23T00:30:42.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:41.040094+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-23T00:30:42.303 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:41.076028+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:30:42.303 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:41.077672+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:30:42.337 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:42.365 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:42.383 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:42.386 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:40.143431+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:30:42.386 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:40.144217+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:30:42.386 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:40.618182+0000 mgr.y (mgr.24437) 139 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:42.386 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:40.786265+0000 osd.2 (osd.2) 57 : cluster [DBG] 5.b deep-scrub starts 2024-02-23T00:30:42.386 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:40.787187+0000 osd.2 (osd.2) 58 : cluster [DBG] 5.b deep-scrub ok 2024-02-23T00:30:42.386 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:41.040094+0000 mon.a (mon.0) 769 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-23T00:30:42.386 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:41.076028+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:30:42.386 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:41.077672+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:30:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:42.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:42.488 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:42.515 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:42.534 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:42.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:42.638 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:42.664 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:42.683 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:42.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:42.787 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:42.817 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:43.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.203+0000 7f4f88c08700 1 -- 172.21.15.87:0/1945589514 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 msgr2=0x7f4f84104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:43.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.203+0000 7f4f88c08700 1 --2- 172.21.15.87:0/1945589514 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 0x7f4f84104a20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f4f7c009be0 tx=0x7f4f7c004d10).stop 2024-02-23T00:30:43.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.203+0000 7f4f88c08700 1 -- 172.21.15.87:0/1945589514 shutdown_connections 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.203+0000 7f4f88c08700 1 --2- 172.21.15.87:0/1945589514 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 0x7f4f84104a20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.203+0000 7f4f88c08700 1 --2- 172.21.15.87:0/1945589514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f840fdbb0 0x7f4f840fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.203+0000 7f4f88c08700 1 --2- 172.21.15.87:0/1945589514 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f840fe510 0x7f4f840f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.203+0000 7f4f88c08700 1 -- 172.21.15.87:0/1945589514 >> 172.21.15.87:0/1945589514 conn(0x7f4f840f3800 msgr2=0x7f4f840f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.203+0000 7f4f88c08700 1 -- 172.21.15.87:0/1945589514 shutdown_connections 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.203+0000 7f4f88c08700 1 -- 172.21.15.87:0/1945589514 wait complete. 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.204+0000 7f4f88c08700 1 librados: starting msgr at 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.204+0000 7f4f88c08700 1 librados: starting objecter 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.204+0000 7f4f88c08700 1 Processor -- start 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.204+0000 7f4f88c08700 1 -- start start 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.204+0000 7f4f88c08700 1 librados: setting wanted keys 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.204+0000 7f4f88c08700 1 librados: calling monclient init 2024-02-23T00:30:43.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 0x7f4f8410de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f840fdbb0 0x7f4f8410e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f840fe510 0x7f4f84108050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84108540 con 0x7f4f840fe510 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f841086b0 con 0x7f4f840fdbb0 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84108870 con 0x7f4f840f8420 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f8259c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 0x7f4f8410de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f8259c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 0x7f4f8410de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40682/0 (socket says 172.21.15.87:40682) 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f8259c700 1 -- 172.21.15.87:0/62695553 learned_addr learned my addr 172.21.15.87:0/62695553 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f82d9d700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f840fe510 0x7f4f84108050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f81d9b700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f840fdbb0 0x7f4f8410e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f8259c700 1 -- 172.21.15.87:0/62695553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f840fdbb0 msgr2=0x7f4f8410e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:43.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f8259c700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f840fdbb0 0x7f4f8410e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f8259c700 1 -- 172.21.15.87:0/62695553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f840fe510 msgr2=0x7f4f84108050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f8259c700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f840fe510 0x7f4f84108050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f8259c700 1 -- 172.21.15.87:0/62695553 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c0097c0 con 0x7f4f840f8420 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f81d9b700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f840fdbb0 0x7f4f8410e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f82d9d700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f840fe510 0x7f4f84108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f8259c700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 0x7f4f8410de60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f4f740041d0 tx=0x7f4f74009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f737fe700 10 client.?.objecter ms_handle_connect 0x7f4f840f8420 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f737fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f737fe700 1 -- 172.21.15.87:0/62695553 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74007b40 con 0x7f4f840f8420 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f84108b50 con 0x7f4f840f8420 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f737fe700 1 -- 172.21.15.87:0/62695553 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f7400f040 con 0x7f4f840f8420 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f737fe700 1 -- 172.21.15.87:0/62695553 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74014770 con 0x7f4f840f8420 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 10 client.24833.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:43.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f84108ed0 con 0x7f4f840f8420 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.205+0000 7f4f88c08700 1 librados: init done 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.206+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f84107110 con 0x7f4f840f8420 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f737fe700 1 -- 172.21.15.87:0/62695553 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4f74009110 con 0x7f4f840f8420 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f737fe700 1 --2- 172.21.15.87:0/62695553 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f6c062c90 0x7f4f6c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f737fe700 1 -- 172.21.15.87:0/62695553 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(94..94 src has 38..94) v4 ==== 7657+0+0 (secure 0 0 0) 0x7f4f7407d100 con 0x7f4f840f8420 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f737fe700 10 client.24833.objecter ms_dispatch 0x7f4f84001550 osd_map(94..94 src has 38..94) v4 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f737fe700 3 client.24833.objecter handle_osd_map got epochs [94,94] > 0 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f737fe700 3 client.24833.objecter handle_osd_map decoding full epoch 94 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f81d9b700 1 --2- 172.21.15.87:0/62695553 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f6c062c90 0x7f4f6c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f737fe700 20 client.24833.objecter dump_active .. 0 homeless 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f737fe700 10 client.24833.objecter ms_handle_connect 0x7f4f6c062c90 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.207+0000 7f4f81d9b700 1 --2- 172.21.15.87:0/62695553 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f6c062c90 0x7f4f6c065150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4f78009f20 tx=0x7f4f7800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:43.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.211+0000 7f4f737fe700 1 -- 172.21.15.87:0/62695553 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4f740507a0 con 0x7f4f840f8420 2024-02-23T00:30:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:43.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:43.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:43.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.385+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}) v1 -- 0x7f4f8405fbb0 con 0x7f4f6c062c90 2024-02-23T00:30:43.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.386+0000 7f4f737fe700 1 -- 172.21.15.87:0/62695553 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+295 (secure 0 0 0) 0x7f4f8405fbb0 con 0x7f4f6c062c90 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:moved 0 / 780 (0%) 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:avg 97.5 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:stddev 8.70345 -> 8.70345 (expected baseline 9.23648) 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:min osd.2 with 82 -> 82 pgs (0.841026 -> 0.841026 * mean) 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:max osd.7 with 110 -> 110 pgs (1.12821 -> 1.12821 * mean) 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:oload 110 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:max_change 1 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:max_change_osds 4 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:average_utilization 0.0032 2024-02-23T00:30:43.387 INFO:teuthology.orchestra.run.smithi087.stdout:overload_utilization 0.0035 2024-02-23T00:30:43.388 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:41.120918+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:30:43.388 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:41.121913+0000 osd.5 (osd.5) 56 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:30:43.388 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:41.498202+0000 mgr.y (mgr.24437) 140 : cluster [DBG] pgmap v121: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:43.388 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:42.155207+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:43.388 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:42.177851+0000 mon.b (mon.1) 35 : audit [DBG] from='client.? 172.21.15.87:0/2585156511' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-23T00:30:43.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:41.120918+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:30:43.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:41.121913+0000 osd.5 (osd.5) 56 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:30:43.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:41.498202+0000 mgr.y (mgr.24437) 140 : cluster [DBG] pgmap v121: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:43.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:42.155207+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:43.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:42.177851+0000 mon.b (mon.1) 35 : audit [DBG] from='client.? 172.21.15.87:0/2585156511' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-23T00:30:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 10 librados: watch_flush enter 2024-02-23T00:30:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 10 librados: watch_flush exit 2024-02-23T00:30:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 20 client.24833.objecter shutdown clearing up homeless session... 2024-02-23T00:30:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 10 client.24833.objecter successfully canceled tick 2024-02-23T00:30:43.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f6c062c90 msgr2=0x7f4f6c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 --2- 172.21.15.87:0/62695553 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f6c062c90 0x7f4f6c065150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f4f78009f20 tx=0x7f4f7800b040).stop 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 msgr2=0x7f4f8410de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 0x7f4f8410de60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f4f740041d0 tx=0x7f4f74009f60).stop 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 shutdown_connections 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 --2- 172.21.15.87:0/62695553 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f6c062c90 0x7f4f6c065150 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f840fe510 0x7f4f84108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f840f8420 0x7f4f8410de60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 --2- 172.21.15.87:0/62695553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f840fdbb0 0x7f4f8410e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 >> 172.21.15.87:0/62695553 conn(0x7f4f840f3800 msgr2=0x7f4f840fbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 shutdown_connections 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 -- 172.21.15.87:0/62695553 wait complete. 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:43.388+0000 7f4f88c08700 1 librados: shutdown 2024-02-23T00:30:43.391 INFO:teuthology.orchestra.run.smithi087.stderr:no change 2024-02-23T00:30:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:41.120918+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:30:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:41.121913+0000 osd.5 (osd.5) 56 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:30:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:41.498202+0000 mgr.y (mgr.24437) 140 : cluster [DBG] pgmap v121: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:30:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:42.155207+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:42.177851+0000 mon.b (mon.1) 35 : audit [DBG] from='client.? 172.21.15.87:0/2585156511' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-23T00:30:43.469 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:43.495 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:43.972 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:43.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:43.981 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-23T00:30:44.077 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:44.106 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:44.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:30:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:30:44.229 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:44.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:44.334 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:44.379 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:44.402 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:44.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:44.505 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:44.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:43.142061+0000 mon.c (mon.2) 30 : audit [INF] from='client.? 172.21.15.87:0/2540195451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:44.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:43.146248+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.87:0/2395726451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:44.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:43.148941+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.87:0/2285581802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:44.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:43.620219+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-23T00:30:44.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:43.620478+0000 mon.a (mon.0) 771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:44.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:43.621311+0000 mon.b (mon.1) 38 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:44.530 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:43.142061+0000 mon.c (mon.2) 30 : audit [INF] from='client.? 172.21.15.87:0/2540195451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:44.531 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:43.146248+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.87:0/2395726451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:44.531 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:43.148941+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.87:0/2285581802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:44.531 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:43.620219+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-23T00:30:44.531 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:43.620478+0000 mon.a (mon.0) 771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:44.531 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:43.621311+0000 mon.b (mon.1) 38 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:44.531 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:43.142061+0000 mon.c (mon.2) 30 : audit [INF] from='client.? 172.21.15.87:0/2540195451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:43.146248+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.87:0/2395726451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:44.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:43.148941+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.87:0/2285581802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:44.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:43.620219+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-23T00:30:44.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:43.620478+0000 mon.a (mon.0) 771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:44.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:43.621311+0000 mon.b (mon.1) 38 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:44.731 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:44.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:44.834 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:44.860 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:45.008 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:45.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:45.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:45.113 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:45.141 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:45.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 -- 172.21.15.87:0/432788286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4550102210 msgr2=0x7f4550104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 --2- 172.21.15.87:0/432788286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4550102210 0x7f4550104600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f4548009be0 tx=0x7f4548004d10).stop 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 -- 172.21.15.87:0/432788286 shutdown_connections 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 --2- 172.21.15.87:0/432788286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4550102210 0x7f4550104600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 --2- 172.21.15.87:0/432788286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45500f8cb0 0x7f45500f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 --2- 172.21.15.87:0/432788286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f45500f9610 0x7f4550101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 -- 172.21.15.87:0/432788286 >> 172.21.15.87:0/432788286 conn(0x7f45500f2820 msgr2=0x7f45500f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 -- 172.21.15.87:0/432788286 shutdown_connections 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 -- 172.21.15.87:0/432788286 wait complete. 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 librados: starting msgr at 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.248+0000 7f4554c3c700 1 librados: starting objecter 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 Processor -- start 2024-02-23T00:30:45.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 -- start start 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 librados: setting wanted keys 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 librados: calling monclient init 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f45500f8cb0 0x7f455010df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45500f9610 0x7f455010e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102210 0x7f4550108170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550108660 con 0x7f4550102210 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501087d0 con 0x7f45500f9610 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f4554c3c700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4550108940 con 0x7f45500f8cb0 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454e59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f45500f8cb0 0x7f455010df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454ed9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102210 0x7f4550108170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454e59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f45500f8cb0 0x7f455010df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40736/0 (socket says 172.21.15.87:40736) 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454e59c700 1 -- 172.21.15.87:0/1662271628 learned_addr learned my addr 172.21.15.87:0/1662271628 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454dd9b700 1 --2- 172.21.15.87:0/1662271628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45500f9610 0x7f455010e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454e59c700 1 -- 172.21.15.87:0/1662271628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45500f9610 msgr2=0x7f455010e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454e59c700 1 --2- 172.21.15.87:0/1662271628 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f45500f9610 0x7f455010e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454e59c700 1 -- 172.21.15.87:0/1662271628 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102210 msgr2=0x7f4550108170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454e59c700 1 --2- 172.21.15.87:0/1662271628 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4550102210 0x7f4550108170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.249+0000 7f454e59c700 1 -- 172.21.15.87:0/1662271628 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480097c0 con 0x7f45500f8cb0 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f454e59c700 1 --2- 172.21.15.87:0/1662271628 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f45500f8cb0 0x7f455010df90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f454000c810 tx=0x7f4540004330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f453f7fe700 10 client.?.objecter ms_handle_connect 0x7f45500f8cb0 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f453f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45400043a0 con 0x7f45500f8cb0 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45400109e0 con 0x7f45500f8cb0 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4550108c20 con 0x7f45500f8cb0 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540012ab0 con 0x7f45500f8cb0 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f4554c3c700 10 client.24848.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:45.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4550114e00 con 0x7f45500f8cb0 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f4554c3c700 1 librados: init done 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.250+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f455005fa60 con 0x7f45500f8cb0 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.251+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4540010b50 con 0x7f45500f8cb0 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f453f7fe700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4538062c90 0x7f4538065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(95..95 src has 38..95) v4 ==== 9508+0+0 (secure 0 0 0) 0x7f454007e1f0 con 0x7f45500f8cb0 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 osd_map(95..95 src has 38..95) v4 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f453f7fe700 3 client.24848.objecter handle_osd_map got epochs [95,95] > 0 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f453f7fe700 3 client.24848.objecter handle_osd_map decoding full epoch 95 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f454dd9b700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4538062c90 0x7f4538065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f453f7fe700 20 client.24848.objecter dump_active .. 0 homeless 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v95) v1 ==== 54+0+16 (secure 0 0 0) 0x7f454007e6d0 con 0x7f45500f8cb0 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f453f7fe700 10 client.24848.objecter ms_handle_connect 0x7f4538062c90 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f454dd9b700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4538062c90 0x7f4538065150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f4550109910 tx=0x7f4544009de0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 10 client.24848.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 1 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6801/2153191863 conn(0x7f45500fc1a0 0x7f45500fc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455005ee80 osd=0 3 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455005ee80 osd=0 3 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455005ee80 osd=0 3 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 1 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f455005ee80 osd=0 3 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 0 1 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455005ee80 osd=0 4 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 10 client.24848.objecter _send_command 1 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4550107290 con 0x7f45500fc1a0 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.252+0000 7f454ed9d700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6801/2153191863 conn(0x7f45500fc1a0 0x7f45500fc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.253+0000 7f453f7fe700 10 client.24848.objecter ms_handle_connect 0x7f45500fc1a0 2024-02-23T00:30:45.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.253+0000 7f454ed9d700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6801/2153191863 conn(0x7f45500fc1a0 0x7f45500fc5a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.253+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4550107290 con 0x7f45500fc1a0 2024-02-23T00:30:45.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.253+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:30:45.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.253+0000 7f453f7fe700 10 client.24848.objecter _finish_command 1 = system:0 2024-02-23T00:30:45.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.253+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f455005ee80 osd=0 4 2024-02-23T00:30:45.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.253+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 0 1 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 10 client.24848.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 2 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455005ee80 osd=0 3 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455005ee80 osd=0 3 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455005ee80 osd=0 3 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 2 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f455005ee80 osd=0 3 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 0 2 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455005ee80 osd=0 4 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 10 client.24848.objecter _send_command 2 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.268+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f45500273c0 con 0x7f45500fc1a0 2024-02-23T00:30:45.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f45500273c0 con 0x7f45500fc1a0 2024-02-23T00:30:45.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:45.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f453f7fe700 10 client.24848.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f455005ee80 osd=0 4 2024-02-23T00:30:45.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 0 2 2024-02-23T00:30:45.275 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:30:45.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 10 client.24848.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:45.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 3 2024-02-23T00:30:45.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6805/685354171 conn(0x7f45500598f0 0x7f4550059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550028710 osd=1 3 2024-02-23T00:30:45.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550028710 osd=1 3 2024-02-23T00:30:45.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550028710 osd=1 3 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 3 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f4550028710 osd=1 3 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 1 3 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550028710 osd=1 4 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 10 client.24848.objecter _send_command 3 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.269+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f455011b7a0 con 0x7f45500598f0 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.270+0000 7f454e59c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6805/685354171 conn(0x7f45500598f0 0x7f4550059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.270+0000 7f453f7fe700 10 client.24848.objecter ms_handle_connect 0x7f45500598f0 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.270+0000 7f454e59c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6805/685354171 conn(0x7f45500598f0 0x7f4550059cf0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.270+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f455011b7a0 con 0x7f45500598f0 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.270+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.270+0000 7f453f7fe700 10 client.24848.objecter _finish_command 3 = system:0 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.270+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f4550028710 osd=1 4 2024-02-23T00:30:45.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.270+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 1 3 2024-02-23T00:30:45.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 10 client.24848.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:45.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 4 2024-02-23T00:30:45.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550028710 osd=1 3 2024-02-23T00:30:45.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550028710 osd=1 3 2024-02-23T00:30:45.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550028710 osd=1 3 2024-02-23T00:30:45.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 4 2024-02-23T00:30:45.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f4550028710 osd=1 3 2024-02-23T00:30:45.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 1 4 2024-02-23T00:30:45.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550028710 osd=1 4 2024-02-23T00:30:45.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 10 client.24848.objecter _send_command 4 2024-02-23T00:30:45.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.285+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4550059360 con 0x7f45500598f0 2024-02-23T00:30:45.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4550059360 con 0x7f45500598f0 2024-02-23T00:30:45.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:45.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f453f7fe700 10 client.24848.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f4550028710 osd=1 4 2024-02-23T00:30:45.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 1 4 2024-02-23T00:30:45.288 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:30:45.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 10 client.24848.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:45.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 5 2024-02-23T00:30:45.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4550028270 0x7f455011a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550059710 osd=2 3 2024-02-23T00:30:45.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550059710 osd=2 3 2024-02-23T00:30:45.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550059710 osd=2 3 2024-02-23T00:30:45.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 5 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f4550059710 osd=2 3 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 2 5 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550059710 osd=2 4 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 10 client.24848.objecter _send_command 5 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f455011a6b0 con 0x7f4550028270 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.287+0000 7f454dd9b700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4550028270 0x7f455011a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.288+0000 7f453f7fe700 10 client.24848.objecter ms_handle_connect 0x7f4550028270 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.288+0000 7f454dd9b700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4550028270 0x7f455011a1c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.288+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f455011a6b0 con 0x7f4550028270 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.288+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.288+0000 7f453f7fe700 10 client.24848.objecter _finish_command 5 = system:0 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.288+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f4550059710 osd=2 4 2024-02-23T00:30:45.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.288+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 2 5 2024-02-23T00:30:45.291 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:45.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:45.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 10 client.24848.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:45.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 6 2024-02-23T00:30:45.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550059710 osd=2 3 2024-02-23T00:30:45.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550059710 osd=2 3 2024-02-23T00:30:45.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550059710 osd=2 3 2024-02-23T00:30:45.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 6 2024-02-23T00:30:45.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f4550059710 osd=2 3 2024-02-23T00:30:45.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 2 6 2024-02-23T00:30:45.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550059710 osd=2 4 2024-02-23T00:30:45.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 10 client.24848.objecter _send_command 6 2024-02-23T00:30:45.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.307+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f45500277d0 con 0x7f4550028270 2024-02-23T00:30:45.310 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f45500277d0 con 0x7f4550028270 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f453f7fe700 10 client.24848.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f4550059710 osd=2 4 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 2 6 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 10 client.24848.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 7 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6813/128059790 conn(0x7f455011d920 0x7f455011fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550027bb0 osd=3 3 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550027bb0 osd=3 3 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550027bb0 osd=3 3 2024-02-23T00:30:45.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 7 2024-02-23T00:30:45.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f4550027bb0 osd=3 3 2024-02-23T00:30:45.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 3 7 2024-02-23T00:30:45.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550027bb0 osd=3 4 2024-02-23T00:30:45.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 10 client.24848.objecter _send_command 7 2024-02-23T00:30:45.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.309+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f455011b560 con 0x7f455011d920 2024-02-23T00:30:45.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.310+0000 7f454ed9d700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6813/128059790 conn(0x7f455011d920 0x7f455011fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.310+0000 7f453f7fe700 10 client.24848.objecter ms_handle_connect 0x7f455011d920 2024-02-23T00:30:45.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.310+0000 7f454ed9d700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6813/128059790 conn(0x7f455011d920 0x7f455011fd10 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.311+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f455011b560 con 0x7f455011d920 2024-02-23T00:30:45.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.311+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:30:45.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.311+0000 7f453f7fe700 10 client.24848.objecter _finish_command 7 = system:0 2024-02-23T00:30:45.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.311+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f4550027bb0 osd=3 4 2024-02-23T00:30:45.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.311+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 3 7 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 10 client.24848.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 8 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550027bb0 osd=3 3 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550027bb0 osd=3 3 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550027bb0 osd=3 3 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 8 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f4550027bb0 osd=3 3 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 3 8 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550027bb0 osd=3 4 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 10 client.24848.objecter _send_command 8 2024-02-23T00:30:45.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.328+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f455011aac0 con 0x7f455011d920 2024-02-23T00:30:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f455011aac0 con 0x7f455011d920 2024-02-23T00:30:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f453f7fe700 10 client.24848.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f4550027bb0 osd=3 4 2024-02-23T00:30:45.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 3 8 2024-02-23T00:30:45.331 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:30:45.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 10 client.24848.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 9 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4550119d20 0x7f45500f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455011ad00 osd=4 3 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ad00 osd=4 3 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455011ad00 osd=4 3 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 9 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f455011ad00 osd=4 3 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 4 9 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ad00 osd=4 4 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 10 client.24848.objecter _send_command 9 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.329+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f455011ae70 con 0x7f4550119d20 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.330+0000 7f454e59c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4550119d20 0x7f45500f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.330+0000 7f453f7fe700 10 client.24848.objecter ms_handle_connect 0x7f4550119d20 2024-02-23T00:30:45.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.330+0000 7f454e59c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4550119d20 0x7f45500f7c20 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.331+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f455011ae70 con 0x7f4550119d20 2024-02-23T00:30:45.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.331+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:30:45.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.331+0000 7f453f7fe700 10 client.24848.objecter _finish_command 9 = system:0 2024-02-23T00:30:45.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.331+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f455011ad00 osd=4 4 2024-02-23T00:30:45.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.331+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 4 9 2024-02-23T00:30:45.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 10 client.24848.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 10 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455011ad00 osd=4 3 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ad00 osd=4 3 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455011ad00 osd=4 3 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 10 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f455011ad00 osd=4 3 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 4 10 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ad00 osd=4 4 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 10 client.24848.objecter _send_command 10 2024-02-23T00:30:45.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.346+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f455011ae70 con 0x7f4550119d20 2024-02-23T00:30:45.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.347+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f455011ae70 con 0x7f4550119d20 2024-02-23T00:30:45.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.347+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:45.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f453f7fe700 10 client.24848.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f455011ad00 osd=4 4 2024-02-23T00:30:45.349 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 4 10 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 10 client.24848.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 11 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6804/2550013795 conn(0x7f45500f86c0 0x7f4550124220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455011ae70 osd=5 3 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ae70 osd=5 3 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455011ae70 osd=5 3 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 11 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f455011ae70 osd=5 3 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 5 11 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ae70 osd=5 4 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 10 client.24848.objecter _send_command 11 2024-02-23T00:30:45.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4550125cf0 con 0x7f45500f86c0 2024-02-23T00:30:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f454dd9b700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6804/2550013795 conn(0x7f45500f86c0 0x7f4550124220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f454dd9b700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6804/2550013795 conn(0x7f45500f86c0 0x7f4550124220 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.348+0000 7f453f7fe700 10 client.24848.objecter ms_handle_connect 0x7f45500f86c0 2024-02-23T00:30:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.349+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4550125cf0 con 0x7f45500f86c0 2024-02-23T00:30:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.349+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:30:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.349+0000 7f453f7fe700 10 client.24848.objecter _finish_command 11 = system:0 2024-02-23T00:30:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.349+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f455011ae70 osd=5 4 2024-02-23T00:30:45.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.349+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 5 11 2024-02-23T00:30:45.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 10 client.24848.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:45.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 12 2024-02-23T00:30:45.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455011ae70 osd=5 3 2024-02-23T00:30:45.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ae70 osd=5 3 2024-02-23T00:30:45.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f455011ae70 osd=5 3 2024-02-23T00:30:45.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 12 2024-02-23T00:30:45.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f455011ae70 osd=5 3 2024-02-23T00:30:45.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 5 12 2024-02-23T00:30:45.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ae70 osd=5 4 2024-02-23T00:30:45.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 10 client.24848.objecter _send_command 12 2024-02-23T00:30:45.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.364+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4550125ee0 con 0x7f45500f86c0 2024-02-23T00:30:45.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4550125ee0 con 0x7f45500f86c0 2024-02-23T00:30:45.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:45.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f453f7fe700 10 client.24848.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f455011ae70 osd=5 4 2024-02-23T00:30:45.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 5 12 2024-02-23T00:30:45.367 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:30:45.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 10 client.24848.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:45.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 13 2024-02-23T00:30:45.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4550126560 0x7f4550124710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550125ee0 osd=6 3 2024-02-23T00:30:45.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550125ee0 osd=6 3 2024-02-23T00:30:45.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550125ee0 osd=6 3 2024-02-23T00:30:45.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 13 2024-02-23T00:30:45.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f4550125ee0 osd=6 3 2024-02-23T00:30:45.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 6 13 2024-02-23T00:30:45.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550125ee0 osd=6 4 2024-02-23T00:30:45.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 10 client.24848.objecter _send_command 13 2024-02-23T00:30:45.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.366+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45501269c0 con 0x7f4550126560 2024-02-23T00:30:45.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.367+0000 7f454ed9d700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4550126560 0x7f4550124710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.367+0000 7f454ed9d700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4550126560 0x7f4550124710 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.367+0000 7f453f7fe700 10 client.24848.objecter ms_handle_connect 0x7f4550126560 2024-02-23T00:30:45.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.368+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f45501269c0 con 0x7f4550126560 2024-02-23T00:30:45.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.368+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:30:45.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.368+0000 7f453f7fe700 10 client.24848.objecter _finish_command 13 = system:0 2024-02-23T00:30:45.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.368+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f4550125ee0 osd=6 4 2024-02-23T00:30:45.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.368+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 6 13 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 10 client.24848.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 14 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550125ee0 osd=6 3 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550125ee0 osd=6 3 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f4550125ee0 osd=6 3 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 14 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f4550125ee0 osd=6 3 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 6 14 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550125ee0 osd=6 4 2024-02-23T00:30:45.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 10 client.24848.objecter _send_command 14 2024-02-23T00:30:45.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.382+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f45501269c0 con 0x7f4550126560 2024-02-23T00:30:45.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f45501269c0 con 0x7f4550126560 2024-02-23T00:30:45.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:45.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f453f7fe700 10 client.24848.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f4550125ee0 osd=6 4 2024-02-23T00:30:45.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 6 14 2024-02-23T00:30:45.386 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:30:45.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 10 client.24848.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:30:45.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 15 2024-02-23T00:30:45.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6812/373787000 conn(0x7f4550125110 0x7f4550125570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:45.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f45501269c0 osd=7 3 2024-02-23T00:30:45.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f45501269c0 osd=7 3 2024-02-23T00:30:45.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f45501269c0 osd=7 3 2024-02-23T00:30:45.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 15 2024-02-23T00:30:45.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f45501269c0 osd=7 3 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 7 15 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f45501269c0 osd=7 4 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 10 client.24848.objecter _send_command 15 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4550125aa0 con 0x7f4550125110 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f454e59c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6812/373787000 conn(0x7f4550125110 0x7f4550125570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f454e59c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6812/373787000 conn(0x7f4550125110 0x7f4550125570 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.384+0000 7f453f7fe700 10 client.24848.objecter ms_handle_connect 0x7f4550125110 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.385+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4550125aa0 con 0x7f4550125110 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.385+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.385+0000 7f453f7fe700 10 client.24848.objecter _finish_command 15 = system:0 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.385+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f45501269c0 osd=7 4 2024-02-23T00:30:45.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.385+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 7 15 2024-02-23T00:30:45.394 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:45.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 10 client.24848.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:30:45.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign -1 16 2024-02-23T00:30:45.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f45501269c0 osd=7 3 2024-02-23T00:30:45.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f45501269c0 osd=7 3 2024-02-23T00:30:45.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 20 client.24848.objecter _get_session s=0x7f45501269c0 osd=7 3 2024-02-23T00:30:45.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_remove -1 16 2024-02-23T00:30:45.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 20 client.24848.objecter get_session s=0x7f45501269c0 osd=7 3 2024-02-23T00:30:45.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 15 client.24848.objecter _session_command_op_assign 7 16 2024-02-23T00:30:45.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f45501269c0 osd=7 4 2024-02-23T00:30:45.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 10 client.24848.objecter _send_command 16 2024-02-23T00:30:45.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.400+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4550125aa0 con 0x7f4550125110 2024-02-23T00:30:45.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.401+0000 7f453f7fe700 1 -- 172.21.15.87:0/1662271628 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4550125aa0 con 0x7f4550125110 2024-02-23T00:30:45.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.401+0000 7f453f7fe700 10 client.24848.objecter ms_dispatch 0x7f4550001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:30:45.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.401+0000 7f453f7fe700 10 client.24848.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.401+0000 7f453f7fe700 20 client.24848.objecter put_session s=0x7f45501269c0 osd=7 4 2024-02-23T00:30:45.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.401+0000 7f453f7fe700 15 client.24848.objecter _session_command_op_remove 7 16 2024-02-23T00:30:45.403 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 librados: watch_flush enter 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 librados: watch_flush exit 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 client.24848.objecter close_session for osd.0 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6801/2153191863 conn(0x7f45500fc1a0 msgr2=0x7f45500fc5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6801/2153191863 conn(0x7f45500fc1a0 0x7f45500fc5a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455005ee80 osd=0 1 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 client.24848.objecter close_session for osd.1 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6805/685354171 conn(0x7f45500598f0 msgr2=0x7f4550059cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6805/685354171 conn(0x7f45500598f0 0x7f4550059cf0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550028710 osd=1 1 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 client.24848.objecter close_session for osd.2 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4550028270 msgr2=0x7f455011a1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4550028270 0x7f455011a1c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550059710 osd=2 1 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 client.24848.objecter close_session for osd.3 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6813/128059790 conn(0x7f455011d920 msgr2=0x7f455011fd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f454ed9d700 1 -- 172.21.15.87:0/1662271628 reap_dead start 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6813/128059790 conn(0x7f455011d920 0x7f455011fd10 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550027bb0 osd=3 1 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 client.24848.objecter close_session for osd.4 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4550119d20 msgr2=0x7f45500f7c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4550119d20 0x7f45500f7c20 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ad00 osd=4 1 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 client.24848.objecter close_session for osd.5 2024-02-23T00:30:45.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6804/2550013795 conn(0x7f45500f86c0 msgr2=0x7f4550124220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6804/2550013795 conn(0x7f45500f86c0 0x7f4550124220 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f455011ae70 osd=5 1 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 client.24848.objecter close_session for osd.6 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4550126560 msgr2=0x7f4550124710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4550126560 0x7f4550124710 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f4550125ee0 osd=6 1 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 client.24848.objecter close_session for osd.7 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6812/373787000 conn(0x7f4550125110 msgr2=0x7f4550125570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.171:6812/373787000 conn(0x7f4550125110 0x7f4550125570 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 20 client.24848.objecter put_session s=0x7f45501269c0 osd=7 1 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 20 client.24848.objecter shutdown clearing up homeless session... 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 10 client.24848.objecter successfully canceled tick 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f454ed9d700 1 -- 172.21.15.87:0/1662271628 reap_dead start 2024-02-23T00:30:45.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4538062c90 msgr2=0x7f4538065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4538062c90 0x7f4538065150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f4550109910 tx=0x7f4544009de0).stop 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f45500f8cb0 msgr2=0x7f455010df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f45500f8cb0 0x7f455010df90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f454000c810 tx=0x7f4540004330).stop 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 shutdown_connections 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4538062c90 0x7f4538065150 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 --2- 172.21.15.87:0/1662271628 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f45500f8cb0 0x7f455010df90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 >> 172.21.15.87:0/1662271628 conn(0x7f45500f2820 msgr2=0x7f45500f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 shutdown_connections 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 -- 172.21.15.87:0/1662271628 wait complete. 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:45.402+0000 7f4554c3c700 1 librados: shutdown 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.408 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.409 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.409 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.409 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.409 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:30:45.426 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:45.447 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:45.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:45.552 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:45.587 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:45.820 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:42.979485+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:45.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:42.979528+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:45.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:43.385639+0000 mgr.y (mgr.24437) 141 : audit [DBG] from='client.24833 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:45.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:43.498685+0000 mgr.y (mgr.24437) 142 : cluster [DBG] pgmap v123: 260 pgs: 128 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-02-23T00:30:45.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:44.351483+0000 mon.a (mon.0) 772 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:46.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:42.979485+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:46.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:42.979528+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:46.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:43.385639+0000 mgr.y (mgr.24437) 141 : audit [DBG] from='client.24833 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:46.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:43.498685+0000 mgr.y (mgr.24437) 142 : cluster [DBG] pgmap v123: 260 pgs: 128 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-02-23T00:30:46.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:44.351483+0000 mon.a (mon.0) 772 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:46.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:42.979485+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:30:46.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:42.979528+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:30:46.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:43.385639+0000 mgr.y (mgr.24437) 141 : audit [DBG] from='client.24833 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:46.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:43.498685+0000 mgr.y (mgr.24437) 142 : cluster [DBG] pgmap v123: 260 pgs: 128 unknown, 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2024-02-23T00:30:46.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:44.351483+0000 mon.a (mon.0) 772 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:46.738 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:46.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:46.841 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:30:46.873 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:46.894 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:30:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:46.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:30:46.998 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:47.030 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:47.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:47.053 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:47.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:47.154 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:47.185 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:47.206 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:47.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:47.283 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:47.288 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-23T00:30:47.288 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd test-reweight-by-utilization 110 0.05 2024-02-23T00:30:47.310 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:47.341 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:47.360 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:47.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:47.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.330+0000 7f308ffff700 1 --2- 172.21.15.87:0/3806809140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30900fdbb0 0x7f30900fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.330+0000 7f3096330700 1 -- 172.21.15.87:0/3806809140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 msgr2=0x7f30900f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.330+0000 7f3096330700 1 --2- 172.21.15.87:0/3806809140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 0x7f30900f7d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3084004870 tx=0x7f3084009cb0).stop 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.330+0000 7f3096330700 1 -- 172.21.15.87:0/3806809140 shutdown_connections 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.330+0000 7f3096330700 1 --2- 172.21.15.87:0/3806809140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30900f8420 0x7f3090104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.330+0000 7f3096330700 1 --2- 172.21.15.87:0/3806809140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30900fdbb0 0x7f30900fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 --2- 172.21.15.87:0/3806809140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 0x7f30900f7d40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 -- 172.21.15.87:0/3806809140 >> 172.21.15.87:0/3806809140 conn(0x7f30900f3800 msgr2=0x7f30900f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 -- 172.21.15.87:0/3806809140 shutdown_connections 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 -- 172.21.15.87:0/3806809140 wait complete. 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 librados: starting msgr at 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 librados: starting objecter 2024-02-23T00:30:47.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 Processor -- start 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 -- start start 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 librados: setting wanted keys 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.331+0000 7f3096330700 1 librados: calling monclient init 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f3096330700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30900f8420 0x7f309010df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f3096330700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30900fdbb0 0x7f309010e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f3096330700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 0x7f3090108130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f3096330700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3090108620 con 0x7f30900f8420 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f3096330700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3090108790 con 0x7f30900fe510 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f3096330700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3090108950 con 0x7f30900fdbb0 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f30948cd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 0x7f3090108130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f30948cd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 0x7f3090108130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.87:49798/0 (socket says 172.21.15.87:49798) 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f30948cd700 1 -- 172.21.15.87:0/3597061316 learned_addr learned my addr 172.21.15.87:0/3597061316 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:47.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f308f7fe700 1 --2- 172.21.15.87:0/3597061316 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30900fdbb0 0x7f309010e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f30948cd700 1 -- 172.21.15.87:0/3597061316 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30900fdbb0 msgr2=0x7f309010e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f30948cd700 1 --2- 172.21.15.87:0/3597061316 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30900fdbb0 0x7f309010e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f30948cd700 1 -- 172.21.15.87:0/3597061316 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30900f8420 msgr2=0x7f309010df90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f30948cd700 1 --2- 172.21.15.87:0/3597061316 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30900f8420 0x7f309010df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.332+0000 7f30948cd700 1 -- 172.21.15.87:0/3597061316 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3084005040 con 0x7f30900fe510 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.333+0000 7f30948cd700 1 --2- 172.21.15.87:0/3597061316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 0x7f3090108130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f308800d970 tx=0x7f3088004e60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.334+0000 7f308d7fa700 10 client.?.objecter ms_handle_connect 0x7f30900fe510 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.334+0000 7f308d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.334+0000 7f308d7fa700 1 -- 172.21.15.87:0/3597061316 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308800de60 con 0x7f30900fe510 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.334+0000 7f308d7fa700 1 -- 172.21.15.87:0/3597061316 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f308800f040 con 0x7f30900fe510 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.334+0000 7f308d7fa700 1 -- 172.21.15.87:0/3597061316 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30880036c0 con 0x7f30900fe510 2024-02-23T00:30:47.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.334+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3090108c30 con 0x7f30900fe510 2024-02-23T00:30:47.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.334+0000 7f3096330700 10 client.24868.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:47.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.334+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309006af10 con 0x7f30900fe510 2024-02-23T00:30:47.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.334+0000 7f3096330700 1 librados: init done 2024-02-23T00:30:47.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.335+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309005ee80 con 0x7f30900fe510 2024-02-23T00:30:47.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.336+0000 7f308d7fa700 1 -- 172.21.15.87:0/3597061316 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3088003ab0 con 0x7f30900fe510 2024-02-23T00:30:47.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.336+0000 7f308d7fa700 1 --2- 172.21.15.87:0/3597061316 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3078062ce0 0x7f30780651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:47.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.336+0000 7f308d7fa700 1 -- 172.21.15.87:0/3597061316 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(96..96 src has 38..96) v4 ==== 11779+0+0 (secure 0 0 0) 0x7f308807dc40 con 0x7f30900fe510 2024-02-23T00:30:47.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.336+0000 7f308d7fa700 10 client.24868.objecter ms_dispatch 0x7f3090001550 osd_map(96..96 src has 38..96) v4 2024-02-23T00:30:47.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.336+0000 7f308d7fa700 3 client.24868.objecter handle_osd_map got epochs [96,96] > 0 2024-02-23T00:30:47.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.336+0000 7f308d7fa700 3 client.24868.objecter handle_osd_map decoding full epoch 96 2024-02-23T00:30:47.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.336+0000 7f308d7fa700 20 client.24868.objecter dump_active .. 0 homeless 2024-02-23T00:30:47.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.336+0000 7f308ffff700 1 --2- 172.21.15.87:0/3597061316 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3078062ce0 0x7f30780651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:47.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.337+0000 7f308d7fa700 10 client.24868.objecter ms_handle_connect 0x7f3078062ce0 2024-02-23T00:30:47.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.337+0000 7f308ffff700 1 --2- 172.21.15.87:0/3597061316 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3078062ce0 0x7f30780651a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3080004a90 tx=0x7f3080007b40).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:47.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.341+0000 7f308d7fa700 1 -- 172.21.15.87:0/3597061316 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3088050280 con 0x7f30900fe510 2024-02-23T00:30:47.464 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:47.490 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:47.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:47.667+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f3090107110 con 0x7f30900fe510 2024-02-23T00:30:47.696 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:47.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:47.800 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:44.362084+0000 mon.c (mon.2) 31 : audit [INF] from='client.? 172.21.15.87:0/1747884445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:44.362431+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.87:0/499549712' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:44.363226+0000 mon.c (mon.2) 32 : audit [INF] from='client.? 172.21.15.87:0/2173949184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:44.363668+0000 mon.c (mon.2) 33 : audit [INF] from='client.? 172.21.15.87:0/2024050357' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:44.367673+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.87:0/166096809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:44.540108+0000 osd.1 (osd.1) 71 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:44.543155+0000 osd.1 (osd.1) 72 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:44.883501+0000 osd.0 (osd.0) 75 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:44.885328+0000 osd.0 (osd.0) 76 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:30:47.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:45.251529+0000 mon.c (mon.2) 34 : audit [DBG] from='client.? 172.21.15.87:0/1662271628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:45.566854+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:45.809836+0000 mon.a (mon.0) 774 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:45.814477+0000 mon.a (mon.0) 775 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:45.814650+0000 mon.a (mon.0) 776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:44.362084+0000 mon.c (mon.2) 31 : audit [INF] from='client.? 172.21.15.87:0/1747884445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:44.362431+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.87:0/499549712' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:44.363226+0000 mon.c (mon.2) 32 : audit [INF] from='client.? 172.21.15.87:0/2173949184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:44.363668+0000 mon.c (mon.2) 33 : audit [INF] from='client.? 172.21.15.87:0/2024050357' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:44.367673+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.87:0/166096809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:44.540108+0000 osd.1 (osd.1) 71 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:44.543155+0000 osd.1 (osd.1) 72 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:30:47.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:44.883501+0000 osd.0 (osd.0) 75 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:30:47.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:44.885328+0000 osd.0 (osd.0) 76 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:30:47.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:45.251529+0000 mon.c (mon.2) 34 : audit [DBG] from='client.? 172.21.15.87:0/1662271628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:47.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:45.566854+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-23T00:30:47.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:45.809836+0000 mon.a (mon.0) 774 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:45.814477+0000 mon.a (mon.0) 775 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:45.814650+0000 mon.a (mon.0) 776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.828 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:44.362084+0000 mon.c (mon.2) 31 : audit [INF] from='client.? 172.21.15.87:0/1747884445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:44.362431+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.87:0/499549712' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:44.363226+0000 mon.c (mon.2) 32 : audit [INF] from='client.? 172.21.15.87:0/2173949184' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:44.363668+0000 mon.c (mon.2) 33 : audit [INF] from='client.? 172.21.15.87:0/2024050357' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:44.367673+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.87:0/166096809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:44.540108+0000 osd.1 (osd.1) 71 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:30:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:44.543155+0000 osd.1 (osd.1) 72 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:30:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:44.883501+0000 osd.0 (osd.0) 75 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:30:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:44.885328+0000 osd.0 (osd.0) 76 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:30:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:45.251529+0000 mon.c (mon.2) 34 : audit [DBG] from='client.? 172.21.15.87:0/1662271628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:45.566854+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-23T00:30:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:45.809836+0000 mon.a (mon.0) 774 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:45.814477+0000 mon.a (mon.0) 775 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:45.814650+0000 mon.a (mon.0) 776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.332 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:48.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.582+0000 7f308d7fa700 1 -- 172.21.15.87:0/3597061316 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v98) v1 ==== 87+0+0 (secure 0 0 0) 0x7f308804fe70 con 0x7f30900fe510 2024-02-23T00:30:48.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.586+0000 7f3096330700 10 librados: watch_flush enter 2024-02-23T00:30:48.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.588+0000 7f3096330700 10 librados: watch_flush exit 2024-02-23T00:30:48.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.588+0000 7f3096330700 20 client.24868.objecter shutdown clearing up homeless session... 2024-02-23T00:30:48.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.588+0000 7f3096330700 10 client.24868.objecter successfully canceled tick 2024-02-23T00:30:48.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.588+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3078062ce0 msgr2=0x7f30780651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:48.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.588+0000 7f3096330700 1 --2- 172.21.15.87:0/3597061316 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3078062ce0 0x7f30780651a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3080004a90 tx=0x7f3080007b40).stop 2024-02-23T00:30:48.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.588+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 msgr2=0x7f3090108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:48.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.588+0000 7f3096330700 1 --2- 172.21.15.87:0/3597061316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 0x7f3090108130 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f308800d970 tx=0x7f3088004e60).stop 2024-02-23T00:30:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.590+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 shutdown_connections 2024-02-23T00:30:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.590+0000 7f3096330700 1 --2- 172.21.15.87:0/3597061316 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3078062ce0 0x7f30780651a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.590+0000 7f3096330700 1 --2- 172.21.15.87:0/3597061316 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30900fe510 0x7f3090108130 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.590+0000 7f3096330700 1 --2- 172.21.15.87:0/3597061316 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30900f8420 0x7f309010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:30:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.590+0000 7f3096330700 1 --2- 172.21.15.87:0/3597061316 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30900fdbb0 0x7f309010e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.590+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 >> 172.21.15.87:0/3597061316 conn(0x7f30900f3800 msgr2=0x7f30900f5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:48.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.595+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 shutdown_connections 2024-02-23T00:30:48.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.598+0000 7f3096330700 1 -- 172.21.15.87:0/3597061316 wait complete. 2024-02-23T00:30:48.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:48.599+0000 7f3096330700 1 librados: shutdown 2024-02-23T00:30:48.618 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:30:48.725 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:30:48.811 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:42.633957+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:42.634748+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:43.620019+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.b scrub starts 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:43.620845+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.b scrub ok 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:45.023155+0000 osd.5 (osd.5) 57 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:45.024053+0000 osd.5 (osd.5) 58 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:45.499760+0000 mgr.y (mgr.24437) 143 : cluster [DBG] pgmap v125: 420 pgs: 231 creating+peering, 132 active+clean, 57 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:45.819276+0000 mon.a (mon.0) 777 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:45.825553+0000 mon.c (mon.2) 35 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:45.840685+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.87:0/3080165175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:46.578516+0000 osd.1 (osd.1) 73 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:46.579413+0000 osd.1 (osd.1) 74 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:30:48.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:46.845122+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:46.848060+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.089256+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.090889+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.139044+0000 osd.4 (osd.4) 67 : cluster [DBG] 2.f scrub starts 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.142503+0000 osd.4 (osd.4) 68 : cluster [DBG] 2.f scrub ok 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.500323+0000 mgr.y (mgr.24437) 144 : cluster [DBG] pgmap v127: 612 pgs: 231 creating+peering, 132 active+clean, 249 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.536348+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.548642+0000 mon.a (mon.0) 779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.549772+0000 mon.a (mon.0) 780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.552718+0000 mon.a (mon.0) 781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.553050+0000 mon.a (mon.0) 782 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.553346+0000 mon.a (mon.0) 783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:48.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.554218+0000 mon.a (mon.0) 784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.554739+0000 mon.a (mon.0) 785 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.556737+0000 mon.a (mon.0) 786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.568682+0000 mon.a (mon.0) 787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.568930+0000 mon.a (mon.0) 788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.569025+0000 mon.a (mon.0) 789 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.571901+0000 mon.b (mon.1) 42 : audit [INF] from='client.? 172.21.15.87:0/2261177883' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.573995+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.87:0/2672054735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.581712+0000 mon.c (mon.2) 36 : audit [INF] from='client.? 172.21.15.87:0/2449886674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.586622+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.87:0/1721605290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.587981+0000 mon.c (mon.2) 37 : audit [INF] from='client.? 172.21.15.87:0/261903153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.610264+0000 mon.b (mon.1) 45 : audit [DBG] from='client.? 172.21.15.87:0/3061443299' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.622685+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:30:48.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.629139+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:47.668751+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.87:0/3597061316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.839859+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.11 deep-scrub starts 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:47.842921+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.11 deep-scrub ok 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.088203+0000 mon.b (mon.1) 47 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:48.088409+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.090897+0000 mon.a (mon.0) 791 : audit [INF] from='client.? 172.21.15.87:0/1104851555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi087-82139-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.091125+0000 mon.a (mon.0) 792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.091295+0000 mon.a (mon.0) 793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.092488+0000 mon.a (mon.0) 794 : audit [INF] from='client.? 172.21.15.87:0/5693960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi087-82019-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.092749+0000 mon.a (mon.0) 795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.092966+0000 mon.a (mon.0) 796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.093621+0000 mon.a (mon.0) 797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.094619+0000 mon.a (mon.0) 798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:30:48.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.095222+0000 mon.a (mon.0) 799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:48.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:42.633957+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:30:48.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:42.634748+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:30:48.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:43.620019+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.b scrub starts 2024-02-23T00:30:48.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:43.620845+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.b scrub ok 2024-02-23T00:30:48.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:45.023155+0000 osd.5 (osd.5) 57 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:30:48.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:45.024053+0000 osd.5 (osd.5) 58 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:30:48.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:45.499760+0000 mgr.y (mgr.24437) 143 : cluster [DBG] pgmap v125: 420 pgs: 231 creating+peering, 132 active+clean, 57 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-23T00:30:48.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:45.819276+0000 mon.a (mon.0) 777 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:45.825553+0000 mon.c (mon.2) 35 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:45.840685+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.87:0/3080165175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:46.578516+0000 osd.1 (osd.1) 73 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:46.579413+0000 osd.1 (osd.1) 74 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:46.845122+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:46.848060+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.089256+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.090889+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.139044+0000 osd.4 (osd.4) 67 : cluster [DBG] 2.f scrub starts 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.142503+0000 osd.4 (osd.4) 68 : cluster [DBG] 2.f scrub ok 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.500323+0000 mgr.y (mgr.24437) 144 : cluster [DBG] pgmap v127: 612 pgs: 231 creating+peering, 132 active+clean, 249 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.536348+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.548642+0000 mon.a (mon.0) 779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.549772+0000 mon.a (mon.0) 780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.552718+0000 mon.a (mon.0) 781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.553050+0000 mon.a (mon.0) 782 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.553346+0000 mon.a (mon.0) 783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.554218+0000 mon.a (mon.0) 784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.554739+0000 mon.a (mon.0) 785 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.556737+0000 mon.a (mon.0) 786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.568682+0000 mon.a (mon.0) 787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.568930+0000 mon.a (mon.0) 788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.569025+0000 mon.a (mon.0) 789 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.571901+0000 mon.b (mon.1) 42 : audit [INF] from='client.? 172.21.15.87:0/2261177883' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.573995+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.87:0/2672054735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.581712+0000 mon.c (mon.2) 36 : audit [INF] from='client.? 172.21.15.87:0/2449886674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.818 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.586622+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.87:0/1721605290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.587981+0000 mon.c (mon.2) 37 : audit [INF] from='client.? 172.21.15.87:0/261903153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.610264+0000 mon.b (mon.1) 45 : audit [DBG] from='client.? 172.21.15.87:0/3061443299' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.622685+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.629139+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:47.668751+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.87:0/3597061316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.839859+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.11 deep-scrub starts 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:47.842921+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.11 deep-scrub ok 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.088203+0000 mon.b (mon.1) 47 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:48.088409+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.090897+0000 mon.a (mon.0) 791 : audit [INF] from='client.? 172.21.15.87:0/1104851555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi087-82139-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.091125+0000 mon.a (mon.0) 792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.091295+0000 mon.a (mon.0) 793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.092488+0000 mon.a (mon.0) 794 : audit [INF] from='client.? 172.21.15.87:0/5693960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi087-82019-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.092749+0000 mon.a (mon.0) 795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.092966+0000 mon.a (mon.0) 796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.093621+0000 mon.a (mon.0) 797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.094619+0000 mon.a (mon.0) 798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:30:48.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.095222+0000 mon.a (mon.0) 799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:48.820 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-02-23T00:30:48.820 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [----------] Global test environment set-up. 2024-02-23T00:30:48.820 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [----------] 12 tests from AsioRados 2024-02-23T00:30:48.820 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-02-23T00:30:48.820 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-02-23T00:30:48.820 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (27 ms) 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (5 ms) 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (2 ms) 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-02-23T00:30:48.821 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1 ms) 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2 ms) 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [----------] 12 tests from AsioRados (45 ms total) 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [----------] Global test environment tear-down 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (6700 ms total) 2024-02-23T00:30:48.822 INFO:tasks.workunit.client.0.smithi087.stdout: api_asio: [ PASSED ] 12 tests. 2024-02-23T00:30:48.883 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:48.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:42.633957+0000 osd.7 (osd.7) 67 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:30:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:42.634748+0000 osd.7 (osd.7) 68 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:30:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:43.620019+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.b scrub starts 2024-02-23T00:30:48.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:43.620845+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.b scrub ok 2024-02-23T00:30:48.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:45.023155+0000 osd.5 (osd.5) 57 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:30:48.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:45.024053+0000 osd.5 (osd.5) 58 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:30:48.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:45.499760+0000 mgr.y (mgr.24437) 143 : cluster [DBG] pgmap v125: 420 pgs: 231 creating+peering, 132 active+clean, 57 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-23T00:30:48.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:45.819276+0000 mon.a (mon.0) 777 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:45.825553+0000 mon.c (mon.2) 35 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:45.840685+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.87:0/3080165175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:46.578516+0000 osd.1 (osd.1) 73 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:30:48.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:46.579413+0000 osd.1 (osd.1) 74 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:30:48.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:46.845122+0000 osd.3 (osd.3) 85 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:30:48.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:46.848060+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:30:48.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.089256+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:30:48.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.090889+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:30:48.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.139044+0000 osd.4 (osd.4) 67 : cluster [DBG] 2.f scrub starts 2024-02-23T00:30:48.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.142503+0000 osd.4 (osd.4) 68 : cluster [DBG] 2.f scrub ok 2024-02-23T00:30:48.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.500323+0000 mgr.y (mgr.24437) 144 : cluster [DBG] pgmap v127: 612 pgs: 231 creating+peering, 132 active+clean, 249 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:30:48.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.536348+0000 mon.a (mon.0) 778 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-23T00:30:48.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.548642+0000 mon.a (mon.0) 779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.549772+0000 mon.a (mon.0) 780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.552718+0000 mon.a (mon.0) 781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.553050+0000 mon.a (mon.0) 782 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.553346+0000 mon.a (mon.0) 783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:48.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.554218+0000 mon.a (mon.0) 784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.554739+0000 mon.a (mon.0) 785 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.556737+0000 mon.a (mon.0) 786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.568682+0000 mon.a (mon.0) 787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.568930+0000 mon.a (mon.0) 788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.569025+0000 mon.a (mon.0) 789 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:48.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.571901+0000 mon.b (mon.1) 42 : audit [INF] from='client.? 172.21.15.87:0/2261177883' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.573995+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.87:0/2672054735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.581712+0000 mon.c (mon.2) 36 : audit [INF] from='client.? 172.21.15.87:0/2449886674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.586622+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.87:0/1721605290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.587981+0000 mon.c (mon.2) 37 : audit [INF] from='client.? 172.21.15.87:0/261903153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.610264+0000 mon.b (mon.1) 45 : audit [DBG] from='client.? 172.21.15.87:0/3061443299' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-23T00:30:48.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.622685+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:30:48.949 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.629139+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:30:48.949 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:47.668751+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.87:0/3597061316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:30:48.949 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.839859+0000 osd.3 (osd.3) 87 : cluster [DBG] 6.11 deep-scrub starts 2024-02-23T00:30:48.949 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:47.842921+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.11 deep-scrub ok 2024-02-23T00:30:48.949 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.088203+0000 mon.b (mon.1) 47 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:48.949 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:48.088409+0000 mon.a (mon.0) 790 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-23T00:30:48.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.090897+0000 mon.a (mon.0) 791 : audit [INF] from='client.? 172.21.15.87:0/1104851555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi087-82139-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.091125+0000 mon.a (mon.0) 792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.091295+0000 mon.a (mon.0) 793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.092488+0000 mon.a (mon.0) 794 : audit [INF] from='client.? 172.21.15.87:0/5693960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi087-82019-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.092749+0000 mon.a (mon.0) 795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.092966+0000 mon.a (mon.0) 796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.950 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.093621+0000 mon.a (mon.0) 797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:48.951 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.094619+0000 mon.a (mon.0) 798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:30:48.951 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.095222+0000 mon.a (mon.0) 799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:48.994 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:49.023 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:49.046 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:49.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:49.150 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:49.182 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:49.204 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:49.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:49.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:49.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-02-23T00:30:49.214 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] Global test environment set-up. 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 7 tests from LibRadosList 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjects (298 ms) 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (75 ms) 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo1,foo2,foo3 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo1 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo2 2024-02-23T00:30:49.215 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo3 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo1,foo4,foo5 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo4 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo5 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo1 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo6,foo7 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo7 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo6 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo4 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo5 2024-02-23T00:30:49.216 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns2:foo7 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns2:foo6 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo1 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo1 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo2 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo3 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (96 ms) 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 1 0 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 10 0 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 13 0 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 7 0 2024-02-23T00:30:49.217 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 14 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 0 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 15 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 11 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 5 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 8 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 6 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 3 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 4 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 12 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 9 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2 0 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-23T00:30:49.218 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (86 ms) 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: x cursor=MIN 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=1 cursor=19:02547ec2:::1:head 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=10 cursor=19:52ea6a34:::10:head 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=13 cursor=19:566253c9:::13:head 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=7 cursor=19:5c6b0b28:::7:head 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=14 cursor=19:62a1935d:::14:head 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=0 cursor=19:6cac518f:::0:head 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=15 cursor=19:863748b0:::15:head 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=11 cursor=19:89d3ae78:::11:head 2024-02-23T00:30:49.219 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=5 cursor=19:b29083e3:::5:head 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=8 cursor=19:bd63b0f1:::8:head 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=6 cursor=19:c4fdafeb:::6:head 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=3 cursor=19:cfc208b3:::3:head 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=4 cursor=19:d83876eb:::4:head 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=12 cursor=19:de5d7c5f:::12:head 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=9 cursor=19:e960b815:::9:head 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > oid=2 cursor=19:f905c69b:::2:head 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: FIRST> seek to MIN oid=1 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=1 cursor=19:02547ec2:::1:head 2024-02-23T00:30:49.220 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:02547ec2:::1:head 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:02547ec2:::1:head -> 1 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=10 cursor=19:52ea6a34:::10:head 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:52ea6a34:::10:head 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:52ea6a34:::10:head -> 10 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=13 cursor=19:566253c9:::13:head 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:566253c9:::13:head 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:566253c9:::13:head -> 13 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=7 cursor=19:5c6b0b28:::7:head 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:5c6b0b28:::7:head 2024-02-23T00:30:49.221 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:5c6b0b28:::7:head -> 7 2024-02-23T00:30:49.276 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=14 cursor=19:62a1935d:::14:head 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:62a1935d:::14:head 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:62a1935d:::14:head -> 14 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=0 cursor=19:6cac518f:::0:head 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:6cac518f:::0:head 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:6cac518f:::0:head -> 0 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=15 cursor=19:863748b0:::15:head 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:863748b0:::15:head 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:863748b0:::15:head -> 15 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=11 cursor=19:89d3ae78:::11:head 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:89d3ae78:::11:head 2024-02-23T00:30:49.277 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:89d3ae78:::11:head -> 11 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=5 cursor=19:b29083e3:::5:head 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:b29083e3:::5:head 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:b29083e3:::5:head -> 5 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=8 cursor=19:bd63b0f1:::8:head 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:bd63b0f1:::8:head 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:bd63b0f1:::8:head -> 8 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=6 cursor=19:c4fdafeb:::6:head 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:c4fdafeb:::6:head 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:c4fdafeb:::6:head -> 6 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=3 cursor=19:cfc208b3:::3:head 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:cfc208b3:::3:head 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:cfc208b3:::3:head -> 3 2024-02-23T00:30:49.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=4 cursor=19:d83876eb:::4:head 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:d83876eb:::4:head 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:d83876eb:::4:head -> 4 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=12 cursor=19:de5d7c5f:::12:head 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:de5d7c5f:::12:head 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:de5d7c5f:::12:head -> 12 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=9 cursor=19:e960b815:::9:head 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:e960b815:::9:head 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:e960b815:::9:head -> 9 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : oid=2 cursor=19:f905c69b:::2:head 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:f905c69b:::2:head 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:f905c69b:::2:head -> 2 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:566253c9:::13:head 2024-02-23T00:30:49.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:566253c9:::13:head expected=19:566253c9:::13:head 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:566253c9:::13:head -> 13 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=13 expected=13 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:5c6b0b28:::7:head 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:5c6b0b28:::7:head expected=19:5c6b0b28:::7:head 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:5c6b0b28:::7:head -> 7 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=7 expected=7 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:52ea6a34:::10:head 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:52ea6a34:::10:head expected=19:52ea6a34:::10:head 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:52ea6a34:::10:head -> 10 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=10 expected=10 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:f905c69b:::2:head 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:f905c69b:::2:head expected=19:f905c69b:::2:head 2024-02-23T00:30:49.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:f905c69b:::2:head -> 2 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=2 expected=2 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:e960b815:::9:head 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:e960b815:::9:head expected=19:e960b815:::9:head 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:e960b815:::9:head -> 9 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=9 expected=9 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:de5d7c5f:::12:head 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:de5d7c5f:::12:head expected=19:de5d7c5f:::12:head 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:de5d7c5f:::12:head -> 12 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=12 expected=12 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:d83876eb:::4:head 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:d83876eb:::4:head expected=19:d83876eb:::4:head 2024-02-23T00:30:49.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:d83876eb:::4:head -> 4 2024-02-23T00:30:49.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=4 expected=4 2024-02-23T00:30:49.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:cfc208b3:::3:head 2024-02-23T00:30:49.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:cfc208b3:::3:head expected=19:cfc208b3:::3:head 2024-02-23T00:30:49.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:cfc208b3:::3:head -> 3 2024-02-23T00:30:49.307 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:49.340 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:49.365 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:49.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:49.469 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:49.500 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:49.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : e list_parallel: process_1_[82240]: starting. 2024-02-23T00:30:49.510 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[82240]: creating pool ceph_test_rados_list_parallel.smithi087-82203 2024-02-23T00:30:49.510 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[82240]: created object 0... 2024-02-23T00:30:49.510 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[82240]: created object 25... 2024-02-23T00:30:49.510 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[82240]: created object 49... 2024-02-23T00:30:49.510 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[82240]: finishing. 2024-02-23T00:30:49.510 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_1_[82240]: shutting down. 2024-02-23T00:30:49.510 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[82241]: starting. 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[82241]: listing objects. 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[82241]: listed object 0... 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[82241]: listed object 25... 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[82241]: saw 50 objects 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_2_[82241]: shutting down. 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[84112]: starting. 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[84112]: creating pool ceph_test_rados_list_parallel.smithi087-82203 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[84112]: created object 0... 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[84112]: created object 25... 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[84112]: created object 49... 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[84112]: finishing. 2024-02-23T00:30:49.511 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_3_[84112]: shutting down. 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[84114]: starting. 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[84114]: removed 25 objects... 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[84114]: removed half of the objects 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[84114]: removed 50 objects... 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[84114]: removed 50 objects 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_5_[84114]: shutting down. 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[84113]: starting. 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[84113]: listing objects. 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[84113]: saw 0 objects 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_4_[84113]: shutting down. 2024-02-23T00:30:49.512 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[84180]: starting. 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[84180]: creating pool ceph_test_rados_list_parallel.smithi087-82203 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[84180]: created object 0... 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[84180]: created object 25... 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[84180]: created object 49... 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[84180]: finishing. 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_6_[84180]: shutting down. 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[84181]: starting. 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[84181]: listing objects. 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[84181]: listed object 0... 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[84181]: listed object 25... 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[84181]: listed object 50... 2024-02-23T00:30:49.513 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[84181]: saw 56 objects 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_7_[84181]: shutting down. 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[84182]: starting. 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[84182]: added 25 objects... 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[84182]: added half of the objects 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[84182]: added 50 objects... 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[84182]: added 50 objects 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_8_[84182]: shutting down. 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:49.514 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[84238]: starting. 2024-02-23T00:30:49.523 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:49.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:49.629 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:49.725 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:49.869 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[84238]: creating pool ceph_t api_c_read_operations: Running main() from gmock_main.cc 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (188 ms) 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (8 ms) 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (26 ms) 2024-02-23T00:30:49.870 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (35 ms) 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (16 ms) 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (12 ms) 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (8 ms) 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (6 ms) 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (7 ms) 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-02-23T00:30:49.871 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (13 ms) 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (25 ms) 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (13 ms) 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (18 ms) 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (384 ms total) 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (7828 ms total) 2024-02-23T00:30:49.872 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (154 ms) 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (953 ms) 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (6711 ms) 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (7819 ms total) 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-02-23T00:30:49.873 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (7820 ms total) 2024-02-23T00:30:49.874 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-02-23T00:30:49.924 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: Running main() from gmock_main.cc 2024-02-23T00:30:49.924 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-02-23T00:30:49.924 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [----------] Global test environment set-up. 2024-02-23T00:30:49.924 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-02-23T00:30:49.924 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-02-23T00:30:49.924 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (143 ms) 2024-02-23T00:30:49.924 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (20 ms) 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (7685 ms) 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:45.819276+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:45.825553+0000 mon.c [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:45.840685+0000 mon.b [INF] from='client.? 172.21.15.87:0/3080165175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.548642+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.549772+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.552718+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.553050+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.553346+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:49.925 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.554218+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.554739+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.556737+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.568682+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi087-82123-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.568930+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi087-82059-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.569025+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.571901+0000 mon.b [INF] from='client.? 172.21.15.87:0/2261177883' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.573995+0000 mon.b [INF] from='client.? 172.21.15.87:0/2672054735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.581712+0000 mon.c [INF] from='client.? 172.21.15.87:0/2449886674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.586622+0000 mon.b [INF] from='client.? 172.21.15.87:0/1721605290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.587981+0000 mon.c [INF] from='client.? 172.21.15.87:0/261903153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:47.668751+0000 mon.b [INF] from='client.? 172.21.15.87:0/3597061316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.088203+0000 mon.b [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.090897+0000 mon.a [INF] from='client.? 172.21.15.87:0/1104851555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi087-82139-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.926 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.091125+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.927 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.091295+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.927 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.092488+0000 mon.a [INF] from='client.? 172.21.15.87:0/5693960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi087-82019-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.927 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.092749+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.927 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.092966+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:49.927 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.093621+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:50.272 INFO:tasks.workunit.client.0.smithi087.stdout: est_rados_list_parallel.smithi087-82203 2024-02-23T00:30:50.272 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[84238]: created object 0... 2024-02-23T00:30:50.272 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[84238]: created object 25... 2024-02-23T00:30:50.272 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[84238]: created object 49... 2024-02-23T00:30:50.273 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[84238]: finishing. 2024-02-23T00:30:50.273 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_9_[84238]: shutting down. 2024-02-23T00:30:50.273 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.273 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.273 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.273 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[84239]: starting. 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[84239]: listing objects. 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[84239]: listed object 0... 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[84239]: listed object 25... 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[84239]: listed object 50... 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[84239]: listed object 75... 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[84239]: saw 97 objects 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_10_[84239]: shutting down. 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[84242]: starting. 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[84242]: removed 25 objects... 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[84242]: removed half of the objects 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[84242]: removed 50 objects... 2024-02-23T00:30:50.274 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[84242]: removed 50 objects 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_13_[84242]: shutting down. 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[84240]: starting. 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[84240]: added 25 objects... 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[84240]: added half of the objects 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[84240]: added 50 objects... 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[84240]: added 50 objects 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_11_[84240]: shutting down. 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.275 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[84241]: starting. 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[84241]: added 25 objects... 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[84241]: added half of the objects 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[84241]: added 50 objects... 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[84241]: added 50 objects 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_12_[84241]: shutting down. 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[84388]: starting. 2024-02-23T00:30:50.276 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[84388]: creating pool ceph_test_rados_list_parallel.smithi087-82203 2024-02-23T00:30:50.277 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[84388]: created object 0... 2024-02-23T00:30:50.277 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[84388]: created object 25... 2024-02-23T00:30:50.277 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[84388]: created object 49... 2024-02-23T00:30:50.277 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[84388]: finishing. 2024-02-23T00:30:50.277 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_14_[84388]: shutting down. 2024-02-23T00:30:50.277 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.277 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.277 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.278 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:50.279 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: starting. 2024-02-23T00:30:50.279 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: listing objects. 2024-02-23T00:30:50.279 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: listed object 0... 2024-02-23T00:30:50.279 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: listed object 25... 2024-02-23T00:30:50.279 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: listed object 50... 2024-02-23T00:30:50.413 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: listed object api_cmd: got: 2024-02-23T00:30:48.094619+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:30:50.413 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.095222+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:50.413 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.562751+0000 mon.a [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:50.413 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.569512+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.413 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.569624+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.569743+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.569808+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.569888+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.569967+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570114+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570211+0000 mon.a [INF] from='client.? 172.21.15.87:0/1104851555' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi087-82139-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570273+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570342+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570427+0000 mon.a [INF] from='client.? 172.21.15.87:0/5693960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi087-82019-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570481+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.414 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570541+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:48.562751+0000 mon.a (mon.0) 800 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.569512+0000 mon.a (mon.0) 801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.569624+0000 mon.a (mon.0) 802 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.569743+0000 mon.a (mon.0) 803 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.569808+0000 mon.a (mon.0) 804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.569888+0000 mon.a (mon.0) 805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.569967+0000 mon.a (mon.0) 806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570114+0000 mon.a (mon.0) 807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570211+0000 mon.a (mon.0) 808 : audit [INF] from='client.? 172.21.15.87:0/1104851555' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi087-82139-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570273+0000 mon.a (mon.0) 809 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570342+0000 mon.a (mon.0) 810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570427+0000 mon.a (mon.0) 811 : audit [INF] from='client.? 172.21.15.87:0/5693960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi087-82019-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:48.562751+0000 mon.a (mon.0) 800 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.569512+0000 mon.a (mon.0) 801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.569624+0000 mon.a (mon.0) 802 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.569743+0000 mon.a (mon.0) 803 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.569808+0000 mon.a (mon.0) 804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.569888+0000 mon.a (mon.0) 805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.569967+0000 mon.a (mon.0) 806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570114+0000 mon.a (mon.0) 807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570211+0000 mon.a (mon.0) 808 : audit [INF] from='client.? 172.21.15.87:0/1104851555' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi087-82139-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570273+0000 mon.a (mon.0) 809 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570342+0000 mon.a (mon.0) 810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570427+0000 mon.a (mon.0) 811 : audit [INF] from='client.? 172.21.15.87:0/5693960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi087-82019-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570481+0000 mon.a (mon.0) 812 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570541+0000 mon.a (mon.0) 813 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570627+0000 mon.a (mon.0) 814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570730+0000 mon.a (mon.0) 815 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.570789+0000 mon.a (mon.0) 816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:48.570872+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.580544+0000 mon.b (mon.1) 48 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.588788+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.87:0/3465680593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.931085+0000 mon.a (mon.0) 818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:48.932160+0000 mon.a (mon.0) 819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:49.371828+0000 mon.c (mon.2) 38 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570481+0000 mon.a (mon.0) 812 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570541+0000 mon.a (mon.0) 813 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570627+0000 mon.a (mon.0) 814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570730+0000 mon.a (mon.0) 815 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:30:50.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.570789+0000 mon.a (mon.0) 816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:50.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:48.570872+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-23T00:30:50.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.580544+0000 mon.b (mon.1) 48 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:50.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.588788+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.87:0/3465680593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:50.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.931085+0000 mon.a (mon.0) 818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:50.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:48.932160+0000 mon.a (mon.0) 819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:50.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:49.371828+0000 mon.c (mon.2) 38 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:50.693 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:30:50.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:48.562751+0000 mon.a (mon.0) 800 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:30:50.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.569512+0000 mon.a (mon.0) 801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.569624+0000 mon.a (mon.0) 802 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi087-82051-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.569743+0000 mon.a (mon.0) 803 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi087-82035-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.569808+0000 mon.a (mon.0) 804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi087-82115-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.569888+0000 mon.a (mon.0) 805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi087-82131-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.569967+0000 mon.a (mon.0) 806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi087-82195-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570114+0000 mon.a (mon.0) 807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi087-82107-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570211+0000 mon.a (mon.0) 808 : audit [INF] from='client.? 172.21.15.87:0/1104851555' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi087-82139-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570273+0000 mon.a (mon.0) 809 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi087-82011-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570342+0000 mon.a (mon.0) 810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82163-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570427+0000 mon.a (mon.0) 811 : audit [INF] from='client.? 172.21.15.87:0/5693960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi087-82019-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570481+0000 mon.a (mon.0) 812 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi087-82067-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570541+0000 mon.a (mon.0) 813 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82155-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570627+0000 mon.a (mon.0) 814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570730+0000 mon.a (mon.0) 815 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.570789+0000 mon.a (mon.0) 816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:48.570872+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.580544+0000 mon.b (mon.1) 48 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.588788+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.87:0/3465680593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:50.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.931085+0000 mon.a (mon.0) 818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:50.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:48.932160+0000 mon.a (mon.0) 819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:50.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:49.371828+0000 mon.c (mon.2) 38 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:51.090 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:51.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:51.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:51.195 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:51.232 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:51.460 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570627+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi087-82091-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:51.460 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570730+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:30:51.460 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.570789+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:51.460 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.580544+0000 mon.b [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.588788+0000 mon.b [INF] from='client.? 172.21.15.87:0/3465680593' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.931085+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:48.932160+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:49.849578+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:49.852165+0000 mon.b [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:49.883523+0000 mon.b [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:49.916050+0000 mon.b [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:49.918500+0000 mon.c [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:49.947159+0000 mon.b [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:50.151881+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.461 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:50.153138+0000 mon.c [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.650 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:30:51.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:50.683678+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "poo 75... 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: listed object 100... 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: listed object 125... 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: saw 150 objects 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_15_[84389]: shutting down. 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[84390]: starting. 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[84390]: added 25 objects... 2024-02-23T00:30:51.694 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[84390]: added half of the objects 2024-02-23T00:30:51.695 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[84390]: added 50 objects... 2024-02-23T00:30:51.695 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[84390]: added 50 objects 2024-02-23T00:30:51.695 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: process_16_[84390]: shutting down. 2024-02-23T00:30:51.695 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:51.695 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:51.695 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:51.695 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:51.695 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******************************* 2024-02-23T00:30:51.695 INFO:tasks.workunit.client.0.smithi087.stdout: list_parallel: ******* SUCCESS ********** 2024-02-23T00:30:51.771 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:49.500970+0000 mgr.y (mgr.24437) 145 : cluster [DBG] pgmap v130: 900 pgs: 231 creating+peering, 132 active+clean, 537 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-02-23T00:30:51.771 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:49.686470+0000 mon.b (mon.1) 50 : audit [DBG] from='client.? 172.21.15.87:0/175550475' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-23T00:30:51.772 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:49.849578+0000 mon.a (mon.0) 820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:51.772 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:49.850033+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-23T00:30:51.772 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:49.852165+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.772 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:49.883523+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:51.772 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:49.916050+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-23T00:30:51.772 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:49.918500+0000 mon.c (mon.2) 39 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.773 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:49.947159+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:51.773 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.151881+0000 mon.a (mon.0) 822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.773 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.153138+0000 mon.c (mon.2) 40 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.773 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.372985+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:51.773 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.683678+0000 mon.a (mon.0) 823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:51.774 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:49.500970+0000 mgr.y (mgr.24437) 145 : cluster [DBG] pgmap v130: 900 pgs: 231 creating+peering, 132 active+clean, 537 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-02-23T00:30:51.774 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:49.686470+0000 mon.b (mon.1) 50 : audit [DBG] from='client.? 172.21.15.87:0/175550475' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-23T00:30:51.774 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:49.849578+0000 mon.a (mon.0) 820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:51.774 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:49.850033+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-23T00:30:51.774 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:49.852165+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:49.883523+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:51.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:49.916050+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-23T00:30:51.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:49.918500+0000 mon.c (mon.2) 39 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:49.947159+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:51.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.151881+0000 mon.a (mon.0) 822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.776 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.153138+0000 mon.c (mon.2) 40 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.776 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.372985+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:51.776 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.683678+0000 mon.a (mon.0) 823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:51.776 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.702331+0000 mon.c (mon.2) 42 : audit [INF] from='client.? 172.21.15.87:0/633734636' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.777 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.711131+0000 mon.c (mon.2) 43 : audit [INF] from='client.? 172.21.15.87:0/789129464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.777 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.714810+0000 mon.c (mon.2) 44 : audit [INF] from='client.? 172.21.15.87:0/4111446920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.777 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:50.947266+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-23T00:30:51.777 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.950380+0000 mon.a (mon.0) 825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.777 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.953296+0000 mon.c (mon.2) 45 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:51.777 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.200889+0000 mon.a (mon.0) 826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.777 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.201597+0000 mon.a (mon.0) 827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.202051+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.203915+0000 mon.a (mon.0) 829 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:51.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.205182+0000 mon.a (mon.0) 830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.205422+0000 mon.a (mon.0) 831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:51.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.702331+0000 mon.c (mon.2) 42 : audit [INF] from='client.? 172.21.15.87:0/633734636' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.778 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.711131+0000 mon.c (mon.2) 43 : audit [INF] from='client.? 172.21.15.87:0/789129464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.714810+0000 mon.c (mon.2) 44 : audit [INF] from='client.? 172.21.15.87:0/4111446920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:50.947266+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-23T00:30:51.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.950380+0000 mon.a (mon.0) 825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.953296+0000 mon.c (mon.2) 45 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:51.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.200889+0000 mon.a (mon.0) 826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.201597+0000 mon.a (mon.0) 827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.202051+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.203915+0000 mon.a (mon.0) 829 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:51.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.205182+0000 mon.a (mon.0) 830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.205422+0000 mon.a (mon.0) 831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:49.500970+0000 mgr.y (mgr.24437) 145 : cluster [DBG] pgmap v130: 900 pgs: 231 creating+peering, 132 active+clean, 537 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-02-23T00:30:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:49.686470+0000 mon.b (mon.1) 50 : audit [DBG] from='client.? 172.21.15.87:0/175550475' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-23T00:30:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:49.849578+0000 mon.a (mon.0) 820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi087-82027-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:49.850033+0000 mon.a (mon.0) 821 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-23T00:30:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:49.852165+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:49.883523+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:49.916050+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-23T00:30:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:49.918500+0000 mon.c (mon.2) 39 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:49.947159+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.151881+0000 mon.a (mon.0) 822 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.153138+0000 mon.c (mon.2) 40 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.372985+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.683678+0000 mon.a (mon.0) 823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi087-82147-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:51.954 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.702331+0000 mon.c (mon.2) 42 : audit [INF] from='client.? 172.21.15.87:0/633734636' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.954 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.711131+0000 mon.c (mon.2) 43 : audit [INF] from='client.? 172.21.15.87:0/789129464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.954 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.714810+0000 mon.c (mon.2) 44 : audit [INF] from='client.? 172.21.15.87:0/4111446920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.954 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:50.947266+0000 mon.a (mon.0) 824 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-23T00:30:51.954 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.950380+0000 mon.a (mon.0) 825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:51.954 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.953296+0000 mon.c (mon.2) 45 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:51.955 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.200889+0000 mon.a (mon.0) 826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.955 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.201597+0000 mon.a (mon.0) 827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.955 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.202051+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.955 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.203915+0000 mon.a (mon.0) 829 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:51.955 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.205182+0000 mon.a (mon.0) 830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:51.955 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.205422+0000 mon.a (mon.0) 831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [----------] Global test environment set-up. 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [----------] 4 tests from LibRadosService 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5111 ms) 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ OK ] LibRadosService.RegisterLate (28 ms) 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: cluster: 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: id: 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: health: HEALTH_OK 2024-02-23T00:30:52.003 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: services: 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mgr: y(active, since 3m), standbys: x 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: data: 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pools: 15 pools, 420 pgs 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: objects: 224 objects, 5.7 KiB 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pgs: 13.571% pgs unknown 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 55.000% pgs not active 2024-02-23T00:30:52.004 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 231 creating+peering 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 132 active+clean 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 57 unknown 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: io: 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: client: 1.7 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: cluster: 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: id: 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: health: HEALTH_WARN 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: noscrub flag(s) set 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: services: 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-02-23T00:30:52.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mgr: y(active, since 3m), standbys: x 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: flags noscrub 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: foo: 11 portals active (1 hosts, 3 zones) 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: data: 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pools: 21 pools, 612 pgs 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: objects: 224 objects, 5.7 KiB 2024-02-23T00:30:52.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pgs: 40.686% pgs unknown 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 37.745% pgs not active 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 249 unknown 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 231 creating+peering 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 132 active+clean 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: io: 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: cluster: 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: id: 66db207c-d1e1-11ee-95c0-87774f69a715 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: health: HEALTH_WARN 2024-02-23T00:30:52.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: noscrub flag(s) set 2024-02-23T00:30:52.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:30:52.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: services: 2024-02-23T00:30:52.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 7m) 2024-02-23T00:30:52.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: mgr: y(active, since 3m), standbys: x 2024-02-23T00:30:52.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-23T00:30:52.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: flags noscrub 2024-02-23T00:30:52.012 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:52.018 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:30:52.018 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:30:52.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:52] "GET /metrics HTTP/1.1" 200 172248 "" "Prometheus/2.33.4" 2024-02-23T00:30:52.472 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: foo: 16 portals actil_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:52.472 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:50.702331+0000 mon.c [INF] from='client.? 172.21.15.87:0/633734636' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:50.711131+0000 mon.c [INF] from='client.? 172.21.15.87:0/789129464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:50.714810+0000 mon.c [INF] from='client.? 172.21.15.87:0/4111446920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:50.950380+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:50.953296+0000 mon.c [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.200889+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.201597+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.202051+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.203915+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.205182+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.205422+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:50.151852+0000 client.admin [INF] onexx 2024-02-23T00:30:52.473 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.206141+0000 mon.b [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.456958+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.457170+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.463426+0000 mon.b [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.465552+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.465736+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.466368+0000 mon.b [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.466613+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.466688+0000 mon.b [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.466918+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.539401+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.539947+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.540347+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:30:52.474 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.540755+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.541132+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.541521+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.541901+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.542246+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.542600+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.542945+0000 mon.a [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.685044+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.685139+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.685213+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.685362+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.685480+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.685676+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.475 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.685750+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.476 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.688351+0000 mon.b [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.476 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.688627+0000 mon.b [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.476 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.688757+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.476 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.689617+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.476 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.730153+0000 mon.c [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.476 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.743620+0000 mon.c [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:52.476 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.749265+0000 mon.b [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.476 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.763683+0000 mon.b [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.476 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.995774+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/1137656896 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 msgr2=0x7f1c480fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/1137656896 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 0x7f1c480fcf90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f1c38004870 tx=0x7f1c38009cb0).stop 2024-02-23T00:30:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/1137656896 shutdown_connections 2024-02-23T00:30:52.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/1137656896 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c480f7400 0x7f1c48103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/1137656896 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 0x7f1c480fcf90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/1137656896 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c480fd510 0x7f1c480f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/1137656896 >> 172.21.15.87:0/1137656896 conn(0x7f1c480f2820 msgr2=0x7f1c480f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/1137656896 shutdown_connections 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/1137656896 wait complete. 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 librados: starting msgr at 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.225+0000 7f1c4d5c0700 1 librados: starting objecter 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 Processor -- start 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 -- start start 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 librados: setting wanted keys 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 librados: calling monclient init 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c480f7400 0x7f1c48109320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 0x7f1c48109810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:52.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c480fd510 0x7f1c48109d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c481144e0 con 0x7f1c480fd510 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48114650 con 0x7f1c480f7400 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c4d5c0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c48114810 con 0x7f1c480fcbb0 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c467fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 0x7f1c48109810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c467fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 0x7f1c48109810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39808/0 (socket says 172.21.15.87:39808) 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c467fc700 1 -- 172.21.15.87:0/2539754710 learned_addr learned my addr 172.21.15.87:0/2539754710 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c477fe700 1 --2- 172.21.15.87:0/2539754710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c480fd510 0x7f1c48109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c46ffd700 1 --2- 172.21.15.87:0/2539754710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c480f7400 0x7f1c48109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c467fc700 1 -- 172.21.15.87:0/2539754710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c480f7400 msgr2=0x7f1c48109320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c467fc700 1 --2- 172.21.15.87:0/2539754710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c480f7400 0x7f1c48109320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c467fc700 1 -- 172.21.15.87:0/2539754710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c480fd510 msgr2=0x7f1c48109d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c467fc700 1 --2- 172.21.15.87:0/2539754710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c480fd510 0x7f1c48109d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c467fc700 1 -- 172.21.15.87:0/2539754710 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c400097c0 con 0x7f1c480fcbb0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.226+0000 7f1c467fc700 1 --2- 172.21.15.87:0/2539754710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 0x7f1c48109810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1c3c00e990 tx=0x7f1c3c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.227+0000 7f1c2ffff700 10 client.?.objecter ms_handle_connect 0x7f1c480fcbb0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.227+0000 7f1c2ffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.227+0000 7f1c2ffff700 1 -- 172.21.15.87:0/2539754710 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c004680 con 0x7f1c480fcbb0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.227+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c38005040 con 0x7f1c480fcbb0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.227+0000 7f1c4d5c0700 10 client.25103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.227+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c48107530 con 0x7f1c480fcbb0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.227+0000 7f1c2ffff700 1 -- 172.21.15.87:0/2539754710 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c3c00f040 con 0x7f1c480fcbb0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.227+0000 7f1c4d5c0700 1 librados: init done 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.227+0000 7f1c2ffff700 1 -- 172.21.15.87:0/2539754710 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c0146f0 con 0x7f1c480fcbb0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.228+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c34004f50 con 0x7f1c480fcbb0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.228+0000 7f1c2ffff700 1 -- 172.21.15.87:0/2539754710 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1c3c014b20 con 0x7f1c480fcbb0 2024-02-23T00:30:52.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.228+0000 7f1c2ffff700 1 --2- 172.21.15.87:0/2539754710 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1c30062c90 0x7f1c30065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:30:52.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.228+0000 7f1c2ffff700 1 -- 172.21.15.87:0/2539754710 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(101..101 src has 38..101) v4 ==== 13725+0+0 (secure 0 0 0) 0x7f1c3c07e430 con 0x7f1c480fcbb0 2024-02-23T00:30:52.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.228+0000 7f1c2ffff700 10 client.25103.objecter ms_dispatch 0x7f1c48001550 osd_map(101..101 src has 38..101) v4 2024-02-23T00:30:52.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.228+0000 7f1c2ffff700 3 client.25103.objecter handle_osd_map got epochs [101,101] > 0 2024-02-23T00:30:52.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.228+0000 7f1c2ffff700 3 client.25103.objecter handle_osd_map decoding full epoch 101 2024-02-23T00:30:52.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.228+0000 7f1c46ffd700 1 --2- 172.21.15.87:0/2539754710 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1c30062c90 0x7f1c30065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:30:52.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.229+0000 7f1c2ffff700 20 client.25103.objecter dump_active .. 0 homeless 2024-02-23T00:30:52.495 INFO:teuthology.orchestra.run.smithi087.stdout:moved 0 / 1860 (0%) 2024-02-23T00:30:52.495 INFO:teuthology.orchestra.run.smithi087.stdout:avg 232.5 2024-02-23T00:30:52.495 INFO:teuthology.orchestra.run.smithi087.stdout:stddev 13.1244 -> 13.1244 (expected baseline 14.2632) 2024-02-23T00:30:52.496 INFO:teuthology.orchestra.run.smithi087.stdout:min osd.1 with 203 -> 203 pgs (0.873118 -> 0.873118 * mean) 2024-02-23T00:30:52.496 INFO:teuthology.orchestra.run.smithi087.stdout:max osd.4 with 247 -> 247 pgs (1.06237 -> 1.06237 * mean) 2024-02-23T00:30:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:30:52.496 INFO:teuthology.orchestra.run.smithi087.stdout:oload 110 2024-02-23T00:30:52.496 INFO:teuthology.orchestra.run.smithi087.stdout:max_change 0.05 2024-02-23T00:30:52.496 INFO:teuthology.orchestra.run.smithi087.stdout:max_change_osds 4 2024-02-23T00:30:52.496 INFO:teuthology.orchestra.run.smithi087.stdout:average_utilization 0.0032 2024-02-23T00:30:52.496 INFO:teuthology.orchestra.run.smithi087.stdout:overload_utilization 0.0036 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.229+0000 7f1c2ffff700 10 client.25103.objecter ms_handle_connect 0x7f1c30062c90 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.229+0000 7f1c46ffd700 1 --2- 172.21.15.87:0/2539754710 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1c30062c90 0x7f1c30065150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f1c38009a00 tx=0x7f1c38009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.233+0000 7f1c2ffff700 1 -- 172.21.15.87:0/2539754710 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1c3c050280 con 0x7f1c480fcbb0 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.396+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}) v1 -- 0x7f1c34000bf0 con 0x7f1c30062c90 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.398+0000 7f1c2ffff700 1 -- 172.21.15.87:0/2539754710 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 no change) v1 ==== 17+0+302 (secure 0 0 0) 0x7f1c34000bf0 con 0x7f1c30062c90 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 10 librados: watch_flush enter 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 10 librados: watch_flush exit 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 20 client.25103.objecter shutdown clearing up homeless session... 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 10 client.25103.objecter successfully canceled tick 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1c30062c90 msgr2=0x7f1c30065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/2539754710 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1c30062c90 0x7f1c30065150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f1c38009a00 tx=0x7f1c38009cb0).stop 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 msgr2=0x7f1c48109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:30:52.497 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/2539754710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 0x7f1c48109810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f1c3c00e990 tx=0x7f1c3c00b040).stop 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 shutdown_connections 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/2539754710 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1c30062c90 0x7f1c30065150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/2539754710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1c480fd510 0x7f1c48109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/2539754710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1c480f7400 0x7f1c48109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 --2- 172.21.15.87:0/2539754710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1c480fcbb0 0x7f1c48109810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 >> 172.21.15.87:0/2539754710 conn(0x7f1c480f2820 msgr2=0x7f1c480fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 shutdown_connections 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 -- 172.21.15.87:0/2539754710 wait complete. 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:52.400+0000 7f1c4d5c0700 1 librados: shutdown 2024-02-23T00:30:52.498 INFO:teuthology.orchestra.run.smithi087.stderr:no change 2024-02-23T00:30:52.682 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:30:52.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:50.151852+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-02-23T00:30:52.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:50.628864+0000 mgr.y (mgr.24437) 146 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:52.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.206141+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.374318+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:52.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.456958+0000 mon.a (mon.0) 832 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-23T00:30:52.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.457170+0000 mon.a (mon.0) 833 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.685 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.463426+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.465552+0000 mon.a (mon.0) 834 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.465736+0000 mon.a (mon.0) 835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.466368+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.466613+0000 mon.a (mon.0) 836 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.466688+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.466918+0000 mon.a (mon.0) 837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.686 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:50.151852+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-02-23T00:30:52.686 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:50.628864+0000 mgr.y (mgr.24437) 146 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:52.686 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.206141+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.374318+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.456958+0000 mon.a (mon.0) 832 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.457170+0000 mon.a (mon.0) 833 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.463426+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.465552+0000 mon.a (mon.0) 834 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.465736+0000 mon.a (mon.0) 835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.466368+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.466613+0000 mon.a (mon.0) 836 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.466688+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.466918+0000 mon.a (mon.0) 837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:51.501496+0000 mgr.y (mgr.24437) 147 : cluster [DBG] pgmap v133: 684 pgs: 104 unknown, 64 creating+peering, 516 active+clean; 34 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 139 KiB/s wr, 624 op/s 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.539401+0000 mon.a (mon.0) 838 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.539586+0000 mgr.y (mgr.24437) 148 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.539947+0000 mon.a (mon.0) 839 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.540046+0000 mgr.y (mgr.24437) 149 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.540347+0000 mon.a (mon.0) 840 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.540444+0000 mgr.y (mgr.24437) 150 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.540755+0000 mon.a (mon.0) 841 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.540847+0000 mgr.y (mgr.24437) 151 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.541132+0000 mon.a (mon.0) 842 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.541223+0000 mgr.y (mgr.24437) 152 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.541521+0000 mon.a (mon.0) 843 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:30:52.688 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.541615+0000 mgr.y (mgr.24437) 153 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.541901+0000 mon.a (mon.0) 844 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.541993+0000 mgr.y (mgr.24437) 154 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.542246+0000 mon.a (mon.0) 845 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:51.501496+0000 mgr.y (mgr.24437) 147 : cluster [DBG] pgmap v133: 684 pgs: 104 unknown, 64 creating+peering, 516 active+clean; 34 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 139 KiB/s wr, 624 op/s 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.539401+0000 mon.a (mon.0) 838 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.539586+0000 mgr.y (mgr.24437) 148 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.539947+0000 mon.a (mon.0) 839 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.540046+0000 mgr.y (mgr.24437) 149 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.540347+0000 mon.a (mon.0) 840 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.540444+0000 mgr.y (mgr.24437) 150 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:30:52.689 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.540755+0000 mon.a (mon.0) 841 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.540847+0000 mgr.y (mgr.24437) 151 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.541132+0000 mon.a (mon.0) 842 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.541223+0000 mgr.y (mgr.24437) 152 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.541521+0000 mon.a (mon.0) 843 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.541615+0000 mgr.y (mgr.24437) 153 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.541901+0000 mon.a (mon.0) 844 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.541993+0000 mgr.y (mgr.24437) 154 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.542246+0000 mon.a (mon.0) 845 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.542333+0000 mgr.y (mgr.24437) 155 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:30:52.690 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.542600+0000 mon.a (mon.0) 846 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:30:52.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.542686+0000 mgr.y (mgr.24437) 156 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:30:52.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.542945+0000 mon.a (mon.0) 847 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:30:52.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.543032+0000 mgr.y (mgr.24437) 157 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:30:52.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.685044+0000 mon.a (mon.0) 848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.685139+0000 mon.a (mon.0) 849 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.685213+0000 mon.a (mon.0) 850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.685362+0000 mon.a (mon.0) 851 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.685480+0000 mon.a (mon.0) 852 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.691 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.685676+0000 mon.a (mon.0) 853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.692 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.685750+0000 mon.a (mon.0) 854 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.692 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:51.685832+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-23T00:30:52.692 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.542333+0000 mgr.y (mgr.24437) 155 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:30:52.692 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.542600+0000 mon.a (mon.0) 846 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:30:52.692 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.542686+0000 mgr.y (mgr.24437) 156 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:30:52.692 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.542945+0000 mon.a (mon.0) 847 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:30:52.692 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.543032+0000 mgr.y (mgr.24437) 157 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.685044+0000 mon.a (mon.0) 848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.685139+0000 mon.a (mon.0) 849 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.685213+0000 mon.a (mon.0) 850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.685362+0000 mon.a (mon.0) 851 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.685480+0000 mon.a (mon.0) 852 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.685676+0000 mon.a (mon.0) 853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.685750+0000 mon.a (mon.0) 854 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:51.685832+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.688351+0000 mon.b (mon.1) 59 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.688627+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.688757+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.689617+0000 mon.a (mon.0) 857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.730153+0000 mon.c (mon.2) 47 : audit [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.693 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.743620+0000 mon.c (mon.2) 48 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:52.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.749265+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.763683+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.995774+0000 mon.a (mon.0) 858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.997161+0000 mon.a (mon.0) 859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:52.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.997872+0000 mon.a (mon.0) 860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:52.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.688351+0000 mon.b (mon.1) 59 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.688627+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.688757+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.689617+0000 mon.a (mon.0) 857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.730153+0000 mon.c (mon.2) 47 : audit [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.743620+0000 mon.c (mon.2) 48 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.749265+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.763683+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.995774+0000 mon.a (mon.0) 858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.997161+0000 mon.a (mon.0) 859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:52.696 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.997872+0000 mon.a (mon.0) 860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.697 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:51.998872+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.697 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.000605+0000 mon.b (mon.1) 64 : audit [DBG] from='client.? 172.21.15.87:0/214922659' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-23T00:30:52.697 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.001876+0000 mon.a (mon.0) 861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.697 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.002386+0000 mon.a (mon.0) 862 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:52.697 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.003018+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.697 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.004613+0000 mon.a (mon.0) 863 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:52.697 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.007060+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.007356+0000 mon.a (mon.0) 865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.008159+0000 mon.a (mon.0) 866 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi087-82027-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.008720+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.008897+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.008945+0000 mon.a (mon.0) 867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.009411+0000 mon.a (mon.0) 868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.375535+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:51.998872+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.000605+0000 mon.b (mon.1) 64 : audit [DBG] from='client.? 172.21.15.87:0/214922659' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.001876+0000 mon.a (mon.0) 861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.002386+0000 mon.a (mon.0) 862 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.003018+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.004613+0000 mon.a (mon.0) 863 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.007060+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.007356+0000 mon.a (mon.0) 865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.008159+0000 mon.a (mon.0) 866 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi087-82027-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.008720+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.008897+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.008945+0000 mon.a (mon.0) 867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.009411+0000 mon.a (mon.0) 868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.375535+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:52.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:30:52.833 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:50.151852+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-02-23T00:30:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:50.628864+0000 mgr.y (mgr.24437) 146 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:30:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.206141+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.374318+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.456958+0000 mon.a (mon.0) 832 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-23T00:30:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.457170+0000 mon.a (mon.0) 833 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.463426+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.465552+0000 mon.a (mon.0) 834 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.465736+0000 mon.a (mon.0) 835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.466368+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.466613+0000 mon.a (mon.0) 836 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.466688+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.466918+0000 mon.a (mon.0) 837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:51.501496+0000 mgr.y (mgr.24437) 147 : cluster [DBG] pgmap v133: 684 pgs: 104 unknown, 64 creating+peering, 516 active+clean; 34 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 139 KiB/s wr, 624 op/s 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.539401+0000 mon.a (mon.0) 838 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.539586+0000 mgr.y (mgr.24437) 148 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.539947+0000 mon.a (mon.0) 839 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.540046+0000 mgr.y (mgr.24437) 149 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.540347+0000 mon.a (mon.0) 840 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.540444+0000 mgr.y (mgr.24437) 150 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:30:52.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.540755+0000 mon.a (mon.0) 841 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.540847+0000 mgr.y (mgr.24437) 151 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.541132+0000 mon.a (mon.0) 842 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.541223+0000 mgr.y (mgr.24437) 152 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.541521+0000 mon.a (mon.0) 843 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.541615+0000 mgr.y (mgr.24437) 153 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.541901+0000 mon.a (mon.0) 844 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.541993+0000 mgr.y (mgr.24437) 154 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.542246+0000 mon.a (mon.0) 845 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.542333+0000 mgr.y (mgr.24437) 155 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.542600+0000 mon.a (mon.0) 846 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.542686+0000 mgr.y (mgr.24437) 156 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.542945+0000 mon.a (mon.0) 847 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:30:52.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.543032+0000 mgr.y (mgr.24437) 157 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.685044+0000 mon.a (mon.0) 848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi087-82011-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.685139+0000 mon.a (mon.0) 849 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.685213+0000 mon.a (mon.0) 850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.685362+0000 mon.a (mon.0) 851 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.685480+0000 mon.a (mon.0) 852 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.685676+0000 mon.a (mon.0) 853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi087-82123-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.685750+0000 mon.a (mon.0) 854 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:51.685832+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.688351+0000 mon.b (mon.1) 59 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.688627+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.688757+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.689617+0000 mon.a (mon.0) 857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.730153+0000 mon.c (mon.2) 47 : audit [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.743620+0000 mon.c (mon.2) 48 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:52.946 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.749265+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.763683+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.995774+0000 mon.a (mon.0) 858 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.997161+0000 mon.a (mon.0) 859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.997872+0000 mon.a (mon.0) 860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:51.998872+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.000605+0000 mon.b (mon.1) 64 : audit [DBG] from='client.? 172.21.15.87:0/214922659' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.001876+0000 mon.a (mon.0) 861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.002386+0000 mon.a (mon.0) 862 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.003018+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.004613+0000 mon.a (mon.0) 863 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.007060+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.007356+0000 mon.a (mon.0) 865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.008159+0000 mon.a (mon.0) 866 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi087-82027-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.008720+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.008897+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.008945+0000 mon.a (mon.0) 867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.009411+0000 mon.a (mon.0) 868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:52.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.375535+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:53.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-23T00:30:53.797 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:30:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:53.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:30:53.900 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:53.929 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:53.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:30:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:30:53.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.397366+0000 mgr.y (mgr.24437) 158 : audit [DBG] from='client.25103 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:53.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:52.457873+0000 mon.a (mon.0) 869 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:30:53.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.470179+0000 mon.a (mon.0) 870 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-23T00:30:53.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.471708+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-23T00:30:53.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:52.471752+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:52.471869+0000 mon.a (mon.0) 871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.193298+0000 mon.a (mon.0) 872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.194436+0000 mon.b (mon.1) 69 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.376775+0000 mon.c (mon.2) 50 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.433308+0000 mon.a (mon.0) 873 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi087-82027-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.433414+0000 mon.a (mon.0) 874 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.433507+0000 mon.a (mon.0) 875 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:53.433590+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.434310+0000 mon.b (mon.1) 70 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.434585+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.434861+0000 mon.b (mon.1) 72 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.435844+0000 mon.a (mon.0) 877 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi087-82027-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:53.949 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[82232]: starting. 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[82232]: creating pool ceph_test_rados_open_pools_parallel.smithi087-82211 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[82232]: created object 0... 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[82232]: created object 25... 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[82232]: created object 49... 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[82232]: finishing. 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_1_[82232]: shutting down. 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_2_[82233]: starting. 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_2_[82233]: rados_pool_create. 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_2_[82233]: rados_ioctx_create. 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_2_[82233]: shutting down. 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******************************* 2024-02-23T00:30:53.950 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[84437]: starting. 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[84437]: creating pool ceph_test_rados_open_pools_parallel.smithi087-82211 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[84437]: created object 0... 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[84437]: created object 25... 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[84437]: created object 49... 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[84437]: finishing. 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_3_[84437]: shutting down. 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******************************* 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_4_[84438]: starting. 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_4_[84438]: rados_pool_create. 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_4_[84438]: rados_ioctx_create. 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: process_4_[84438]: shutting down. 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******************************* 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******************************* 2024-02-23T00:30:53.951 INFO:tasks.workunit.client.0.smithi087.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-02-23T00:30:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:53.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:54.036 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.397366+0000 mgr.y (mgr.24437) 158 : audit [DBG] from='client.25103 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:54.036 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:52.457873+0000 mon.a (mon.0) 869 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.470179+0000 mon.a (mon.0) 870 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.471708+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:52.471752+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:52.471869+0000 mon.a (mon.0) 871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.193298+0000 mon.a (mon.0) 872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.194436+0000 mon.b (mon.1) 69 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.376775+0000 mon.c (mon.2) 50 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.433308+0000 mon.a (mon.0) 873 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi087-82027-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.433414+0000 mon.a (mon.0) 874 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.433507+0000 mon.a (mon.0) 875 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:54.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:53.433590+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.397366+0000 mgr.y (mgr.24437) 158 : audit [DBG] from='client.25103 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:52.457873+0000 mon.a (mon.0) 869 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.470179+0000 mon.a (mon.0) 870 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.471708+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:52.471752+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:52.471869+0000 mon.a (mon.0) 871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.193298+0000 mon.a (mon.0) 872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.194436+0000 mon.b (mon.1) 69 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.376775+0000 mon.c (mon.2) 50 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.433308+0000 mon.a (mon.0) 873 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi087-82027-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.433414+0000 mon.a (mon.0) 874 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:54.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.433507+0000 mon.a (mon.0) 875 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:30:54.039 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:53.433590+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-23T00:30:54.039 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.434310+0000 mon.b (mon.1) 70 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:54.039 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.434585+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:54.039 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.434861+0000 mon.b (mon.1) 72 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:54.039 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.435844+0000 mon.a (mon.0) 877 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi087-82027-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:54.039 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.434310+0000 mon.b (mon.1) 70 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:54.040 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.434585+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:54.040 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.434861+0000 mon.b (mon.1) 72 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:54.040 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.435844+0000 mon.a (mon.0) 877 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi087-82027-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:54.040 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:54.053 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:30:54.086 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:54.284 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:30:54.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:54.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:30:54.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:54.421 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:54.768 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:54.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:54.872 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:54.904 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:54.926 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:54.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:54.941 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.683859+0000 mon.a (mon.0) 878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-23T00:30:54.963 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.683859+0000 mon.a (mon.0) 878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-23T00:30:54.964 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.683859+0000 mon.a (mon.0) 878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-23T00:30:54.964 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[82247]: starting. 2024-02-23T00:30:54.964 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[82247]: creating pool ceph_test_rados_delete_pools_parallel.smithi087-82215 2024-02-23T00:30:54.964 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[82247]: created object 0... 2024-02-23T00:30:54.964 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[82247]: created object 25... 2024-02-23T00:30:54.964 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[82247]: created object 49... 2024-02-23T00:30:54.964 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[82247]: finishing. 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_1_[82247]: shutting down. 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_2_[82248]: starting. 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_2_[82248]: deleting pool ceph_test_rados_delete_pools_parallel.smithi087-82215 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_2_[82248]: shutting down. 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[84430]: starting. 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[84430]: creating pool ceph_test_rados_delete_pools_parallel.smithi087-82215 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[84430]: created object 0... 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[84430]: created object 25... 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[84430]: created object 49... 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[84430]: finishing. 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_3_[84430]: shutting down. 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[84432]: starting. 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[84432]: listing objects. 2024-02-23T00:30:54.965 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[84432]: listed object 0... 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[84432]: listed object 25... 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[84432]: saw 50 objects 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_5_[84432]: shutting down. 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_4_[84431]: starting. 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_4_[84431]: deleting pool ceph_test_rados_delete_pools_parallel.smithi087-82215 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: process_4_[84431]: shutting down. 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******************************* 2024-02-23T00:30:54.966 INFO:tasks.workunit.client.0.smithi087.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-02-23T00:30:55.030 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:30:55.056 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:55.074 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:30:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:55.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:30:55.178 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:55.205 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:55.233 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:55.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:55.337 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:55.364 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:55.383 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:55.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:55.486 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:55.513 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:55.689 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.997161+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:55.689 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.997872+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:55.689 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:51.998872+0000 mon.b [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:55.689 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.001876+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:55.689 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.002386+0000 mon.a [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:55.689 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.003018+0000 mon.b [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:55.689 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.004613+0000 mon.a [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:55.689 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.007060+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.007356+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.008159+0000 mon.a [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi087-82027-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.008720+0000 mon.b [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.008897+0000 mon.b [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.008945+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi087-82059-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:52.009411+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:53.683859+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:53.695200+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:53.695288+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:53.695358+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:53.947149+0000 mon.c [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.684464+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.690 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.691171+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.691 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.932305+0000 mon.a [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:55.691 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.933001+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:55.691 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.933230+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:55.691 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.933600+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:55.691 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.937154+0000 mon.a [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi087-82027-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:30:55.691 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.937397+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:55.691 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.937514+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:55.691 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.956487+0000 mon.a [INF] from='client.? 172.21.15.87:0/270589134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.691 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.956719+0000 mon.a [INF] from='client.? 172.21.15.87:0/3387560150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.819 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:52.225633+0000 osd.5 (osd.5) 61 : cluster [DBG] 10.6 deep-scrub starts 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:52.226563+0000 osd.5 (osd.5) 62 : cluster [DBG] 10.6 deep-scrub ok 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:52.965497+0000 osd.6 (osd.6) 55 : cluster [DBG] 10.0 deep-scrub starts 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:52.970962+0000 osd.6 (osd.6) 56 : cluster [DBG] 10.0 deep-scrub ok 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:53.078692+0000 osd.4 (osd.4) 69 : cluster [DBG] 10.1 deep-scrub starts 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:53.080392+0000 osd.4 (osd.4) 70 : cluster [DBG] 10.1 deep-scrub ok 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:53.501981+0000 mgr.y (mgr.24437) 159 : cluster [DBG] pgmap v136: 516 pgs: 128 unknown, 32 creating+peering, 356 active+clean; 9.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 22 KiB/s wr, 472 op/s 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.695200+0000 mon.a (mon.0) 879 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.695288+0000 mon.a (mon.0) 880 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.695358+0000 mon.a (mon.0) 881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:53.947149+0000 mon.c (mon.2) 51 : audit [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.378063+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:54.681669+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.684464+0000 mon.a (mon.0) 883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.691171+0000 mon.a (mon.0) 884 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:52.225633+0000 osd.5 (osd.5) 61 : cluster [DBG] 10.6 deep-scrub starts 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:52.226563+0000 osd.5 (osd.5) 62 : cluster [DBG] 10.6 deep-scrub ok 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:52.965497+0000 osd.6 (osd.6) 55 : cluster [DBG] 10.0 deep-scrub starts 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:52.970962+0000 osd.6 (osd.6) 56 : cluster [DBG] 10.0 deep-scrub ok 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:53.078692+0000 osd.4 (osd.4) 69 : cluster [DBG] 10.1 deep-scrub starts 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:53.080392+0000 osd.4 (osd.4) 70 : cluster [DBG] 10.1 deep-scrub ok 2024-02-23T00:30:55.821 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:53.501981+0000 mgr.y (mgr.24437) 159 : cluster [DBG] pgmap v136: 516 pgs: 128 unknown, 32 creating+peering, 356 active+clean; 9.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 22 KiB/s wr, 472 op/s 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.695200+0000 mon.a (mon.0) 879 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.695288+0000 mon.a (mon.0) 880 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.695358+0000 mon.a (mon.0) 881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:53.947149+0000 mon.c (mon.2) 51 : audit [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.378063+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:54.681669+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.684464+0000 mon.a (mon.0) 883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.691171+0000 mon.a (mon.0) 884 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.932305+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.933001+0000 mon.a (mon.0) 886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:55.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.933230+0000 mon.a (mon.0) 887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.933600+0000 mon.a (mon.0) 888 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.937154+0000 mon.a (mon.0) 889 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi087-82027-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.937397+0000 mon.a (mon.0) 890 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.937514+0000 mon.a (mon.0) 891 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:54.938317+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.956487+0000 mon.a (mon.0) 893 : audit [INF] from='client.? 172.21.15.87:0/270589134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.956719+0000 mon.a (mon.0) 894 : audit [INF] from='client.? 172.21.15.87:0/3387560150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.964970+0000 mon.c (mon.2) 53 : audit [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.965185+0000 mon.c (mon.2) 54 : audit [INF] from='client.? 172.21.15.87:0/1266179889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.932305+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:55.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.933001+0000 mon.a (mon.0) 886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.933230+0000 mon.a (mon.0) 887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.933600+0000 mon.a (mon.0) 888 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.937154+0000 mon.a (mon.0) 889 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi087-82027-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.937397+0000 mon.a (mon.0) 890 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.937514+0000 mon.a (mon.0) 891 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:54.938317+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.956487+0000 mon.a (mon.0) 893 : audit [INF] from='client.? 172.21.15.87:0/270589134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.956719+0000 mon.a (mon.0) 894 : audit [INF] from='client.? 172.21.15.87:0/3387560150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.964970+0000 mon.c (mon.2) 53 : audit [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-23T00:30:55.824 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.965185+0000 mon.c (mon.2) 54 : audit [INF] from='client.? 172.21.15.87:0/1266179889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.965812+0000 mon.a (mon.0) 895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.966198+0000 mon.a (mon.0) 896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.968878+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.969105+0000 mon.b (mon.1) 74 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.971035+0000 mon.a (mon.0) 897 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:54.971280+0000 mon.a (mon.0) 898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:55.378978+0000 mon.c (mon.2) 55 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.965812+0000 mon.a (mon.0) 895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.966198+0000 mon.a (mon.0) 896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.968878+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.969105+0000 mon.b (mon.1) 74 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.971035+0000 mon.a (mon.0) 897 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:55.826 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:54.971280+0000 mon.a (mon.0) 898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.826 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:55.378978+0000 mon.c (mon.2) 55 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:52.225633+0000 osd.5 (osd.5) 61 : cluster [DBG] 10.6 deep-scrub starts 2024-02-23T00:30:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:52.226563+0000 osd.5 (osd.5) 62 : cluster [DBG] 10.6 deep-scrub ok 2024-02-23T00:30:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:52.965497+0000 osd.6 (osd.6) 55 : cluster [DBG] 10.0 deep-scrub starts 2024-02-23T00:30:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:52.970962+0000 osd.6 (osd.6) 56 : cluster [DBG] 10.0 deep-scrub ok 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:53.078692+0000 osd.4 (osd.4) 69 : cluster [DBG] 10.1 deep-scrub starts 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:53.080392+0000 osd.4 (osd.4) 70 : cluster [DBG] 10.1 deep-scrub ok 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:53.501981+0000 mgr.y (mgr.24437) 159 : cluster [DBG] pgmap v136: 516 pgs: 128 unknown, 32 creating+peering, 356 active+clean; 9.8 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 22 KiB/s wr, 472 op/s 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.695200+0000 mon.a (mon.0) 879 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi087-82123-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.695288+0000 mon.a (mon.0) 880 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi087-82035-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.695358+0000 mon.a (mon.0) 881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi087-82131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:53.947149+0000 mon.c (mon.2) 51 : audit [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.378063+0000 mon.c (mon.2) 52 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:54.681669+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.684464+0000 mon.a (mon.0) 883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.691171+0000 mon.a (mon.0) 884 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.932305+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:30:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.933001+0000 mon.a (mon.0) 886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.933230+0000 mon.a (mon.0) 887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.933600+0000 mon.a (mon.0) 888 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.937154+0000 mon.a (mon.0) 889 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi087-82027-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.937397+0000 mon.a (mon.0) 890 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi087-82019-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.937514+0000 mon.a (mon.0) 891 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:54.938317+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.956487+0000 mon.a (mon.0) 893 : audit [INF] from='client.? 172.21.15.87:0/270589134' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.956719+0000 mon.a (mon.0) 894 : audit [INF] from='client.? 172.21.15.87:0/3387560150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.964970+0000 mon.c (mon.2) 53 : audit [INF] from='client.? 172.21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.965185+0000 mon.c (mon.2) 54 : audit [INF] from='client.? 172.21.15.87:0/1266179889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.965812+0000 mon.a (mon.0) 895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.966198+0000 mon.a (mon.0) 896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.968878+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:55.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.969105+0000 mon.b (mon.1) 74 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.971035+0000 mon.a (mon.0) 897 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:55.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:54.971280+0000 mon.a (mon.0) 898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:55.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:55.378978+0000 mon.c (mon.2) 55 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:56.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.964970+0000 mon.c [INF] from='client.? 172. api_io_pp: Running main() from gmock_main.cc 2024-02-23T00:30:56.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-02-23T00:30:56.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] Global test environment set-up. 2024-02-23T00:30:56.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-02-23T00:30:56.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: seed 82035 2024-02-23T00:30:56.090 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (168 ms) 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (20 ms) 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (6 ms) 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (6 ms) 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (13 ms) 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (5 ms) 2024-02-23T00:30:56.091 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (9 ms) 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (11 ms) 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5 ms) 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (14 ms) 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4 ms) 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (15 ms) 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (11 ms) 2024-02-23T00:30:56.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (10 ms) 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (23 ms) 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (5 ms) 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (9 ms) 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (3 ms) 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (6 ms) 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (344 ms total) 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: 2024-02-23T00:30:56.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (2357 ms) 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (13 ms) 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3 ms) 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (3 ms) 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-02-23T00:30:56.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (4 ms) 2024-02-23T00:30:56.162 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:56.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:30:56.266 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:56.292 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:56.311 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:56.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:56.415 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:30:56.445 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:56.653 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:30:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:56.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN 21.15.87:0/3829298547' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.965185+0000 mon.c [INF] from='client.? 172.21.15.87:0/1266179889' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.965812+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.966198+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.968878+0000 mon.b [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.969105+0000 mon.b [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.971035+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:54.971280+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:55.684866+0000 mon.b [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:55.684970+0000 client.admin [INF] threexx 2024-02-23T00:30:56.693 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:55.685416+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-23T00:30:56.694 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.208569+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:56.694 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.209562+0000 mon.b [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:56.694 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.444415+0000 mon.a [INF] from='client.? 172.21.15.87:0/270589134' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.694 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.444477+0000 mon.a [INF] from='client.? 172.21.15.87:0/3387560150' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.694 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.444576+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-23T00:30:56.694 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.444656+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:55.277219+0000 osd.5 (osd.5) 63 : cluster [DBG] 10.9 deep-scrub starts 2024-02-23T00:30:56.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:55.312741+0000 osd.5 (osd.5) 64 : cluster [DBG] 10.9 deep-scrub ok 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:55.503636+0000 mgr.y (mgr.24437) 160 : cluster [DBG] pgmap v139: 644 pgs: 166 creating+peering, 478 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 31 op/s 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:55.684866+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:55.684970+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:55.685416+0000 mon.a (mon.0) 899 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:55.719193+0000 osd.1 (osd.1) 75 : cluster [DBG] 10.2 deep-scrub starts 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:55.722661+0000 osd.1 (osd.1) 76 : cluster [DBG] 10.2 deep-scrub ok 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.208569+0000 mon.a (mon.0) 900 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.209562+0000 mon.b (mon.1) 76 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.380296+0000 mon.c (mon.2) 56 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.444415+0000 mon.a (mon.0) 901 : audit [INF] from='client.? 172.21.15.87:0/270589134' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.444477+0000 mon.a (mon.0) 902 : audit [INF] from='client.? 172.21.15.87:0/3387560150' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.444576+0000 mon.a (mon.0) 903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-23T00:30:56.695 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.444656+0000 mon.a (mon.0) 904 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.444761+0000 mon.a (mon.0) 905 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.444874+0000 mon.a (mon.0) 906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:56.445033+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.445580+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.455725+0000 mon.a (mon.0) 908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.455811+0000 mon.a (mon.0) 909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:56.455958+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.481319+0000 mon.a (mon.0) 911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.481826+0000 mon.a (mon.0) 912 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.494007+0000 mon.b (mon.1) 78 : audit [DBG] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.494642+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.696 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.494877+0000 mon.a (mon.0) 913 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:55.277219+0000 osd.5 (osd.5) 63 : cluster [DBG] 10.9 deep-scrub starts 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:55.312741+0000 osd.5 (osd.5) 64 : cluster [DBG] 10.9 deep-scrub ok 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:55.503636+0000 mgr.y (mgr.24437) 160 : cluster [DBG] pgmap v139: 644 pgs: 166 creating+peering, 478 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 31 op/s 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:55.684866+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:55.684970+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:55.685416+0000 mon.a (mon.0) 899 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:55.719193+0000 osd.1 (osd.1) 75 : cluster [DBG] 10.2 deep-scrub starts 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:55.722661+0000 osd.1 (osd.1) 76 : cluster [DBG] 10.2 deep-scrub ok 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.208569+0000 mon.a (mon.0) 900 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:56.697 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.209562+0000 mon.b (mon.1) 76 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.380296+0000 mon.c (mon.2) 56 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.444415+0000 mon.a (mon.0) 901 : audit [INF] from='client.? 172.21.15.87:0/270589134' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.444477+0000 mon.a (mon.0) 902 : audit [INF] from='client.? 172.21.15.87:0/3387560150' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.444576+0000 mon.a (mon.0) 903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.444656+0000 mon.a (mon.0) 904 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.444761+0000 mon.a (mon.0) 905 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.444874+0000 mon.a (mon.0) 906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:56.445033+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.445580+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.455725+0000 mon.a (mon.0) 908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.455811+0000 mon.a (mon.0) 909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:56.455958+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.481319+0000 mon.a (mon.0) 911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:56.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.481826+0000 mon.a (mon.0) 912 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:56.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.494007+0000 mon.b (mon.1) 78 : audit [DBG] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-23T00:30:56.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.494642+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.494877+0000 mon.a (mon.0) 913 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.757 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:30:56.784 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:56.802 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:30:56.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:56.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:30:56.906 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:30:56.932 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:55.277219+0000 osd.5 (osd.5) 63 : cluster [DBG] 10.9 deep-scrub starts 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:55.312741+0000 osd.5 (osd.5) 64 : cluster [DBG] 10.9 deep-scrub ok 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:55.503636+0000 mgr.y (mgr.24437) 160 : cluster [DBG] pgmap v139: 644 pgs: 166 creating+peering, 478 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 31 op/s 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:55.684866+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:55.684970+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:55.685416+0000 mon.a (mon.0) 899 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:55.719193+0000 osd.1 (osd.1) 75 : cluster [DBG] 10.2 deep-scrub starts 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:55.722661+0000 osd.1 (osd.1) 76 : cluster [DBG] 10.2 deep-scrub ok 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.208569+0000 mon.a (mon.0) 900 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.209562+0000 mon.b (mon.1) 76 : audit [INF] from='client.? 172.21.15.87:0/4186855760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.380296+0000 mon.c (mon.2) 56 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.444415+0000 mon.a (mon.0) 901 : audit [INF] from='client.? 172.21.15.87:0/270589134' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.444477+0000 mon.a (mon.0) 902 : audit [INF] from='client.? 172.21.15.87:0/3387560150' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.444576+0000 mon.a (mon.0) 903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi087-82019-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.444656+0000 mon.a (mon.0) 904 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi087-82011-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.444761+0000 mon.a (mon.0) 905 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.444874+0000 mon.a (mon.0) 906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:56.445033+0000 mon.a (mon.0) 907 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.445580+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.455725+0000 mon.a (mon.0) 908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.455811+0000 mon.a (mon.0) 909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:56.455958+0000 mon.a (mon.0) 910 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.481319+0000 mon.a (mon.0) 911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.481826+0000 mon.a (mon.0) 912 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:30:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.494007+0000 mon.b (mon.1) 78 : audit [DBG] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-23T00:30:56.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.494642+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.494877+0000 mon.a (mon.0) 913 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:56.951 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:30:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:30:56.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:30:57.055 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:30:57.083 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:30:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.691326+0000 mon.a (mon.0) 914 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-23T00:30:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.692833+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-23T00:30:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:56.692881+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-02-23T00:30:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.693017+0000 mon.a (mon.0) 915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-23T00:30:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:56.874275+0000 mon.a (mon.0) 916 : audit [DBG] from='client.? 172.21.15.87:0/1683571677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.381193+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.457040+0000 mon.a (mon.0) 917 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.457144+0000 mon.a (mon.0) 918 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.457284+0000 mon.a (mon.0) 919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:57.457361+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.467688+0000 mon.c (mon.2) 58 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.469095+0000 mon.a (mon.0) 921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.470530+0000 mon.b (mon.1) 81 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.471008+0000 mon.a (mon.0) 922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.471106+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-4-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:30:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.471592+0000 mon.a (mon.0) 923 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-4-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:30:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.475636+0000 mon.a (mon.0) 924 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.478813+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2"}]: dispatch 2024-02-23T00:30:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.483638+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.489347+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.500165+0000 mon.a (mon.0) 925 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:57.500414+0000 mon.a (mon.0) 926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.691326+0000 mon.a (mon.0) 914 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-23T00:30:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.692833+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-23T00:30:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:56.692881+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.693017+0000 mon.a (mon.0) 915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:56.874275+0000 mon.a (mon.0) 916 : audit [DBG] from='client.? 172.21.15.87:0/1683571677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.381193+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.457040+0000 mon.a (mon.0) 917 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.457144+0000 mon.a (mon.0) 918 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.457284+0000 mon.a (mon.0) 919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:57.457361+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.467688+0000 mon.c (mon.2) 58 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.469095+0000 mon.a (mon.0) 921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.470530+0000 mon.b (mon.1) 81 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.471008+0000 mon.a (mon.0) 922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.471106+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-4-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.471592+0000 mon.a (mon.0) 923 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-4-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.475636+0000 mon.a (mon.0) 924 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.478813+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2"}]: dispatch 2024-02-23T00:30:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.483638+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.489347+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.500165+0000 mon.a (mon.0) 925 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:57.500414+0000 mon.a (mon.0) 926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.691326+0000 mon.a (mon.0) 914 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.692833+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.87:0/888932476' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:56.692881+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.693017+0000 mon.a (mon.0) 915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:56.874275+0000 mon.a (mon.0) 916 : audit [DBG] from='client.? 172.21.15.87:0/1683571677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.381193+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.457040+0000 mon.a (mon.0) 917 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]': finished 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.457144+0000 mon.a (mon.0) 918 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.457284+0000 mon.a (mon.0) 919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:57.457361+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.467688+0000 mon.c (mon.2) 58 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.469095+0000 mon.a (mon.0) 921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.470530+0000 mon.b (mon.1) 81 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.471008+0000 mon.a (mon.0) 922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.471106+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-4-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.471592+0000 mon.a (mon.0) 923 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-4-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.475636+0000 mon.a (mon.0) 924 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.478813+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2"}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.483638+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.489347+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.500165+0000 mon.a (mon.0) 925 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:58.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:57.500414+0000 mon.a (mon.0) 926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:58.750 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:30:58.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:30:58.750 INFO:tasks.thrashosds.thrasher:Growing pool iscsi 2024-02-23T00:30:58.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.4 api_stat_pp: Running main() from gmock_main.cc 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: seed 82131 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (194 ms) 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (8 ms) 2024-02-23T00:30:59.508 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (2 ms) 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (4 ms) 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (23 ms) 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (231 ms total) 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (2348 ms) 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (10 ms) 2024-02-23T00:30:59.509 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (7 ms) 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (2367 ms total) 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (17443 ms total) 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: Running main() from gmock_main.cc 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [==========] Running 23 tests from 2 test suites. 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] Global test environment set-up. 2024-02-23T00:30:59.510 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] 13 tests from LibRadosIo 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (58 ms) 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: no timeout :/ 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: no timeout :/ 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: no timeout :/ 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: no timeout :/ 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: no timeout :/ 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (46 ms) 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (10 ms) 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-02-23T00:30:59.511 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.Checksum (2 ms) 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (6 ms) 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (3 ms) 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5 ms) 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.TruncTest (5 ms) 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3 ms) 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (3 ms) 2024-02-23T00:30:59.512 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.RmXattr (7 ms) 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIo.XattrIter (3 ms) 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] 13 tests from LibRadosIo (151 ms total) 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1106 ms) 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (6 ms) 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (4 ms) 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-02-23T00:30:59.513 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (9 ms) 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (3 ms) 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (4 ms) 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1154 ms total) 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [----------] Global test environment tear-down 2024-02-23T00:30:59.514 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [==========] 23 tests from 2 test suites ran. (17390 ms total) 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: api_io: [ PASSED ] 23 tests. 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (4 ms) 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (7 ms) 2024-02-23T00:30:59.515 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-02-23T00:30:59.516 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2024-02-23T00:30:59.516 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-02-23T00:30:59.516 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-02-23T00:30:59.516 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (7 ms) 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (4 ms) 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (7 ms) 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (2438 ms total) 2024-02-23T00:30:59.517 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: 2024-02-23T00:30:59.518 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [----------] Global test environment tear-down 2024-02-23T00:30:59.518 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (17433 ms total) 2024-02-23T00:30:59.518 INFO:tasks.workunit.client.0.smithi087.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-02-23T00:30:59.520 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: Running main() from gmock_main.cc 2024-02-23T00:30:59.520 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2024-02-23T00:30:59.520 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] Global test environment set-up. 2024-02-23T00:30:59.520 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] 4 tests from LibRadosStat 2024-02-23T00:30:59.520 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-02-23T00:30:59.520 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStat.Stat (66 ms) 2024-02-23T00:30:59.520 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStat.StatNS (1132 ms) 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (6 ms) 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (6 ms) 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] 4 tests from LibRadosStat (1210 ms total) 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (2336 ms) 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (26 ms) 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-02-23T00:30:59.521 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-02-23T00:30:59.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-02-23T00:30:59.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (2365 ms total) 2024-02-23T00:30:59.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: 2024-02-23T00:30:59.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [----------] Global test environment tear-down 2024-02-23T00:30:59.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (17424 ms total) 2024-02-23T00:30:59.522 INFO:tasks.workunit.client.0.smithi087.stdout: api_stat: [ PASSED ] 8 tests. 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:57.504264+0000 mgr.y (mgr.24437) 161 : cluster [DBG] pgmap v143: 532 pgs: 16 unknown, 48 creating+peering, 468 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 39 MiB/s wr, 80 op/s 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:57.828463+0000 osd.2 (osd.2) 59 : cluster [DBG] 10.3 deep-scrub starts 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:57.830063+0000 osd.2 (osd.2) 60 : cluster [DBG] 10.3 deep-scrub ok 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:57.970559+0000 osd.6 (osd.6) 57 : cluster [DBG] 10.7 deep-scrub starts 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:57.973443+0000 osd.6 (osd.6) 58 : cluster [DBG] 10.7 deep-scrub ok 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.244313+0000 mon.a (mon.0) 927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.382044+0000 mon.c (mon.2) 59 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:58.478566+0000 mon.a (mon.0) 928 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:58.478973+0000 mon.a (mon.0) 929 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.485439+0000 mon.a (mon.0) 930 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:30:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.485514+0000 mon.a (mon.0) 931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.485577+0000 mon.a (mon.0) 932 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-4-cache", "mode": "writeback"}]': finished 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.485649+0000 mon.a (mon.0) 933 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.485702+0000 mon.a (mon.0) 934 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.485768+0000 mon.a (mon.0) 935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:57.504264+0000 mgr.y (mgr.24437) 161 : cluster [DBG] pgmap v143: 532 pgs: 16 unknown, 48 creating+peering, 468 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 39 MiB/s wr, 80 op/s 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:57.828463+0000 osd.2 (osd.2) 59 : cluster [DBG] 10.3 deep-scrub starts 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:57.830063+0000 osd.2 (osd.2) 60 : cluster [DBG] 10.3 deep-scrub ok 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:57.970559+0000 osd.6 (osd.6) 57 : cluster [DBG] 10.7 deep-scrub starts 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:57.973443+0000 osd.6 (osd.6) 58 : cluster [DBG] 10.7 deep-scrub ok 2024-02-23T00:30:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.244313+0000 mon.a (mon.0) 927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-23T00:30:59.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.382044+0000 mon.c (mon.2) 59 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:59.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:58.478566+0000 mon.a (mon.0) 928 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:30:59.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:58.478973+0000 mon.a (mon.0) 929 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:30:59.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.485439+0000 mon.a (mon.0) 930 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:30:59.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.485514+0000 mon.a (mon.0) 931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:30:59.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.485577+0000 mon.a (mon.0) 932 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-4-cache", "mode": "writeback"}]': finished 2024-02-23T00:30:59.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.485649+0000 mon.a (mon.0) 933 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:30:59.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.485702+0000 mon.a (mon.0) 934 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:30:59.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.485768+0000 mon.a (mon.0) 935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:59.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:58.485992+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-23T00:30:59.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.492098+0000 mon.a (mon.0) 937 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:59.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.493732+0000 mon.a (mon.0) 938 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.496862+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:59.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.498420+0000 mon.c (mon.2) 60 : audit [INF] from='client.? 172.21.15.87:0/1241496180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.501491+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:59.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.501632+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.507848+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.87:0/4137506582' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi087-82011-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.508048+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.87:0/635270104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.508743+0000 mon.a (mon.0) 939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.509179+0000 mon.a (mon.0) 940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.509429+0000 mon.a (mon.0) 941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi087-82011-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.509646+0000 mon.a (mon.0) 942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.511927+0000 mon.b (mon.1) 91 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.513467+0000 mon.a (mon.0) 943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.521886+0000 mon.c (mon.2) 61 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:59.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.522208+0000 mon.a (mon.0) 944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.588 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.522523+0000 mon.a (mon.0) 945 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:59.588 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.535193+0000 mon.a (mon.0) 946 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:30:59.588 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.536500+0000 mon.a (mon.0) 947 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:30:59.588 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.536812+0000 mon.a (mon.0) 948 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:59.588 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.537113+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-4"}]: dispatch 2024-02-23T00:30:59.588 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:58.537294+0000 mon.a (mon.0) 949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-4"}]: dispatch 2024-02-23T00:30:59.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:58.485992+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-23T00:30:59.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.492098+0000 mon.a (mon.0) 937 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.493732+0000 mon.a (mon.0) 938 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.496862+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.498420+0000 mon.c (mon.2) 60 : audit [INF] from='client.? 172.21.15.87:0/1241496180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.501491+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.501632+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.507848+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.87:0/4137506582' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi087-82011-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.508048+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.87:0/635270104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.508743+0000 mon.a (mon.0) 939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.509179+0000 mon.a (mon.0) 940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.509429+0000 mon.a (mon.0) 941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi087-82011-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.509646+0000 mon.a (mon.0) 942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.511927+0000 mon.b (mon.1) 91 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.513467+0000 mon.a (mon.0) 943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.521886+0000 mon.c (mon.2) 61 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:59.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.522208+0000 mon.a (mon.0) 944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.522523+0000 mon.a (mon.0) 945 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:59.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.535193+0000 mon.a (mon.0) 946 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:30:59.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.536500+0000 mon.a (mon.0) 947 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:30:59.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.536812+0000 mon.a (mon.0) 948 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:59.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.537113+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-4"}]: dispatch 2024-02-23T00:30:59.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:30:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:58.537294+0000 mon.a (mon.0) 949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-4"}]: dispatch 2024-02-23T00:30:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:57.504264+0000 mgr.y (mgr.24437) 161 : cluster [DBG] pgmap v143: 532 pgs: 16 unknown, 48 creating+peering, 468 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 13 MiB/s rd, 39 MiB/s wr, 80 op/s 2024-02-23T00:30:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:57.828463+0000 osd.2 (osd.2) 59 : cluster [DBG] 10.3 deep-scrub starts 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:57.830063+0000 osd.2 (osd.2) 60 : cluster [DBG] 10.3 deep-scrub ok 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:57.970559+0000 osd.6 (osd.6) 57 : cluster [DBG] 10.7 deep-scrub starts 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:57.973443+0000 osd.6 (osd.6) 58 : cluster [DBG] 10.7 deep-scrub ok 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.244313+0000 mon.a (mon.0) 927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.382044+0000 mon.c (mon.2) 59 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:58.478566+0000 mon.a (mon.0) 928 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:58.478973+0000 mon.a (mon.0) 929 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.485439+0000 mon.a (mon.0) 930 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.485514+0000 mon.a (mon.0) 931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.485577+0000 mon.a (mon.0) 932 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-4-cache", "mode": "writeback"}]': finished 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.485649+0000 mon.a (mon.0) 933 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.485702+0000 mon.a (mon.0) 934 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:30:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.485768+0000 mon.a (mon.0) 935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:58.485992+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.492098+0000 mon.a (mon.0) 937 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.493732+0000 mon.a (mon.0) 938 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.496862+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.87:0/1206734460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.498420+0000 mon.c (mon.2) 60 : audit [INF] from='client.? 172.21.15.87:0/1241496180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.501491+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.87:0/2547026741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.501632+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.507848+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.87:0/4137506582' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi087-82011-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.508048+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.87:0/635270104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.508743+0000 mon.a (mon.0) 939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.509179+0000 mon.a (mon.0) 940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.509429+0000 mon.a (mon.0) 941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi087-82011-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.509646+0000 mon.a (mon.0) 942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.511927+0000 mon.b (mon.1) 91 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:30:59.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.513467+0000 mon.a (mon.0) 943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:30:59.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.521886+0000 mon.c (mon.2) 61 : audit [INF] from='client.? 172.21.15.87:0/1350246346' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:59.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.522208+0000 mon.a (mon.0) 944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:30:59.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.522523+0000 mon.a (mon.0) 945 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]: dispatch 2024-02-23T00:30:59.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.535193+0000 mon.a (mon.0) 946 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:30:59.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.536500+0000 mon.a (mon.0) 947 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:30:59.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.536812+0000 mon.a (mon.0) 948 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:30:59.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.537113+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-4"}]: dispatch 2024-02-23T00:30:59.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:30:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:58.537294+0000 mon.a (mon.0) 949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-4"}]: dispatch 2024-02-23T00:31:00.248 INFO:tasks.workunit.client.0.smithi087.stdout:44761+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:31:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.444874+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.445580+0000 mon.b [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:31:00.248 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.455725+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi087-82067-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.455811+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi087-82059-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.481319+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi087-82147-1"}]: dispatch 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.481826+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-4", "overlaypool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.494642+0000 mon.b [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: got: 2024-02-23T00:30:56.494877+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi087-82075-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (10354 ms) 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (18203 ms total) 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [----------] Global test environment tear-down 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (18203 ms total) 2024-02-23T00:31:00.249 INFO:tasks.workunit.client.0.smithi087.stdout: api_cmd: [ PASSED ] 4 tests. 2024-02-23T00:31:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:58.686964+0000 osd.0 (osd.0) 77 : cluster [DBG] 10.4 deep-scrub starts 2024-02-23T00:31:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:58.688673+0000 osd.0 (osd.0) 78 : cluster [DBG] 10.4 deep-scrub ok 2024-02-23T00:31:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.382899+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485170+0000 mon.a (mon.0) 950 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:31:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485225+0000 mon.a (mon.0) 951 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485274+0000 mon.a (mon.0) 952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:31:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485360+0000 mon.a (mon.0) 953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:31:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485419+0000 mon.a (mon.0) 954 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi087-82011-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485501+0000 mon.a (mon.0) 955 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485557+0000 mon.a (mon.0) 956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-23T00:31:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485680+0000 mon.a (mon.0) 957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485775+0000 mon.a (mon.0) 958 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:31:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485896+0000 mon.a (mon.0) 959 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:00.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:58.686964+0000 osd.0 (osd.0) 77 : cluster [DBG] 10.4 deep-scrub starts 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:58.688673+0000 osd.0 (osd.0) 78 : cluster [DBG] 10.4 deep-scrub ok 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.382899+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485170+0000 mon.a (mon.0) 950 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485225+0000 mon.a (mon.0) 951 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485274+0000 mon.a (mon.0) 952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485360+0000 mon.a (mon.0) 953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485419+0000 mon.a (mon.0) 954 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi087-82011-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485501+0000 mon.a (mon.0) 955 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485557+0000 mon.a (mon.0) 956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485680+0000 mon.a (mon.0) 957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485775+0000 mon.a (mon.0) 958 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:31:00.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485896+0000 mon.a (mon.0) 959 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:00.589 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.485958+0000 mon.a (mon.0) 960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-4"}]': finished 2024-02-23T00:31:00.589 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:59.486008+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-23T00:31:00.589 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.493430+0000 mon.a (mon.0) 962 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi087-82139-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:31:00.589 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.493580+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:31:00.589 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.493860+0000 mon.b (mon.1) 94 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-23T00:31:00.589 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.497871+0000 mon.a (mon.0) 963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:31:00.589 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.498207+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:59.504887+0000 mgr.y (mgr.24437) 162 : cluster [DBG] pgmap v146: 628 pgs: 144 unknown, 48 creating+peering, 436 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:30:59.553213+0000 mon.a (mon.0) 965 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pg_num": 4}]: dispatch 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:59.983154+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:30:59.983247+0000 osd.7 (osd.7) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.485958+0000 mon.a (mon.0) 960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-4"}]': finished 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:59.486008+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.493430+0000 mon.a (mon.0) 962 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi087-82139-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.493580+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.493860+0000 mon.b (mon.1) 94 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-23T00:31:00.590 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.497871+0000 mon.a (mon.0) 963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:31:00.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.498207+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-23T00:31:00.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:59.504887+0000 mgr.y (mgr.24437) 162 : cluster [DBG] pgmap v146: 628 pgs: 144 unknown, 48 creating+peering, 436 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:00.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:30:59.553213+0000 mon.a (mon.0) 965 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pg_num": 4}]: dispatch 2024-02-23T00:31:00.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:59.983154+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:00.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:30:59.983247+0000 osd.7 (osd.7) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:00.641 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:58.686964+0000 osd.0 (osd.0) 77 : cluster [DBG] 10.4 deep-scrub starts 2024-02-23T00:31:00.641 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:58.688673+0000 osd.0 (osd.0) 78 : cluster [DBG] 10.4 deep-scrub ok 2024-02-23T00:31:00.641 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.382899+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:00.641 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485170+0000 mon.a (mon.0) 950 : audit [INF] from='client.? 172.21.15.87:0/1226496269' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi087-82027-15"}]': finished 2024-02-23T00:31:00.641 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485225+0000 mon.a (mon.0) 951 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.641 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485274+0000 mon.a (mon.0) 952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi087-82123-6"}]': finished 2024-02-23T00:31:00.641 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485360+0000 mon.a (mon.0) 953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi087-82035-22"}]': finished 2024-02-23T00:31:00.641 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485419+0000 mon.a (mon.0) 954 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi087-82011-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.641 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485501+0000 mon.a (mon.0) 955 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485557+0000 mon.a (mon.0) 956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485680+0000 mon.a (mon.0) 957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485775+0000 mon.a (mon.0) 958 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi087-82131-7"}]': finished 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485896+0000 mon.a (mon.0) 959 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.485958+0000 mon.a (mon.0) 960 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-4"}]': finished 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:59.486008+0000 mon.a (mon.0) 961 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.493430+0000 mon.a (mon.0) 962 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi087-82139-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.493580+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.87:0/668592513' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.493860+0000 mon.b (mon.1) 94 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.497871+0000 mon.a (mon.0) 963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]: dispatch 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.498207+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:59.504887+0000 mgr.y (mgr.24437) 162 : cluster [DBG] pgmap v146: 628 pgs: 144 unknown, 48 creating+peering, 436 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:30:59.553213+0000 mon.a (mon.0) 965 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pg_num": 4}]: dispatch 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:59.983154+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:00.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:30:59.983247+0000 osd.7 (osd.7) 74 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:00.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:31:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:31:01.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.383941+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:01.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:00.483777+0000 mon.a (mon.0) 966 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:01.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.639854+0000 mgr.y (mgr.24437) 163 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:01.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.733586+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:31:01.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.733664+0000 mon.a (mon.0) 968 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-23T00:31:01.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.733776+0000 mon.a (mon.0) 969 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pg_num": 4}]': finished 2024-02-23T00:31:01.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:00.733885+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.735057+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.735274+0000 mon.a (mon.0) 971 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.735623+0000 mon.b (mon.1) 96 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.739420+0000 mon.a (mon.0) 972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.739859+0000 mon.a (mon.0) 973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.740490+0000 mon.b (mon.1) 97 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1"}]: dispatch 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:00.740711+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1"}]: dispatch 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.383941+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:00.483777+0000 mon.a (mon.0) 966 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.639854+0000 mgr.y (mgr.24437) 163 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.733586+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.733664+0000 mon.a (mon.0) 968 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.733776+0000 mon.a (mon.0) 969 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pg_num": 4}]': finished 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:00.733885+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.735057+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.735274+0000 mon.a (mon.0) 971 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.735623+0000 mon.b (mon.1) 96 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.739420+0000 mon.a (mon.0) 972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.739859+0000 mon.a (mon.0) 973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.740490+0000 mon.b (mon.1) 97 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1"}]: dispatch 2024-02-23T00:31:01.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:00.740711+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1"}]: dispatch 2024-02-23T00:31:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.383941+0000 mon.c (mon.2) 63 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:00.483777+0000 mon.a (mon.0) 966 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.639854+0000 mgr.y (mgr.24437) 163 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.733586+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-4", "tierpool": "test-rados-api-smithi087-82091-4-cache"}]': finished 2024-02-23T00:31:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.733664+0000 mon.a (mon.0) 968 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-23T00:31:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.733776+0000 mon.a (mon.0) 969 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pg_num": 4}]': finished 2024-02-23T00:31:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:00.733885+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-23T00:31:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.735057+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.735274+0000 mon.a (mon.0) 971 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.735623+0000 mon.b (mon.1) 96 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.739420+0000 mon.a (mon.0) 972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.739859+0000 mon.a (mon.0) 973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.740490+0000 mon.b (mon.1) 97 : audit [INF] from='client.? 172.21.15.87:0/1648023973' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1"}]: dispatch 2024-02-23T00:31:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:00.740711+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1"}]: dispatch 2024-02-23T00:31:02.483 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:02] "GET /metrics HTTP/1.1" 200 216269 "" "Prometheus/2.33.4" 2024-02-23T00:31:02.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:00.617516+0000 osd.0 (osd.0) 79 : cluster [DBG] 10.8 deep-scrub starts 2024-02-23T00:31:02.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:00.671196+0000 osd.0 (osd.0) 80 : cluster [DBG] 10.8 deep-scrub ok 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.385343+0000 mon.c (mon.2) 64 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.476685+0000 mon.a (mon.0) 975 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi087-82139-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]': finished 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.476761+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.476839+0000 mon.a (mon.0) 977 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.476947+0000 mon.a (mon.0) 978 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.477023+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1"}]': finished 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:01.477165+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.482289+0000 mon.c (mon.2) 65 : audit [INF] from='client.? 172.21.15.87:0/4218332242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.482502+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.484241+0000 mon.c (mon.2) 66 : audit [INF] from='client.? 172.21.15.87:0/458030738' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi087-82011-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.484908+0000 mon.a (mon.0) 981 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]: dispatch 2024-02-23T00:31:02.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.485222+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:02.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.485652+0000 mon.a (mon.0) 983 : audit [INF] from='client.? 172.21.15.87:0/92013801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.493122+0000 mon.a (mon.0) 984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.493250+0000 mon.b (mon.1) 99 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:02.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.493450+0000 mon.a (mon.0) 985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi087-82011-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:01.494276+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:02.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:01.505522+0000 mgr.y (mgr.24437) 164 : cluster [DBG] pgmap v149: 560 pgs: 1 active+clean+snaptrim, 108 unknown, 451 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.7 KiB/s wr, 2 op/s 2024-02-23T00:31:02.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:01.693306+0000 osd.2 (osd.2) 61 : cluster [DBG] 10.5 deep-scrub starts 2024-02-23T00:31:02.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:01.694254+0000 osd.2 (osd.2) 62 : cluster [DBG] 10.5 deep-scrub ok 2024-02-23T00:31:02.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:02.393918+0000 mon.c (mon.2) 67 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:02.553 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-02-23T00:31:02.553 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-02-23T00:31:02.553 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-02-23T00:31:02.553 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-02-23T00:31:02.553 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: seed 82067 2024-02-23T00:31:02.553 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-02-23T00:31:02.553 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (174 ms) 2024-02-23T00:31:02.553 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-02-23T00:31:02.553 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (17 ms) 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1083 ms) 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1011 ms) 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (3 ms) 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (3 ms) 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2298 ms total) 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: 2024-02-23T00:31:02.554 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1230 ms) 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (3 ms) 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1004 ms) 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1005 ms) 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (3 ms) 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (4 ms) 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-02-23T00:31:02.555 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-02-23T00:31:02.556 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-02-23T00:31:02.556 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3256 ms total) 2024-02-23T00:31:02.556 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: 2024-02-23T00:31:02.556 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-02-23T00:31:02.556 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (20510 ms total) 2024-02-23T00:31:02.556 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-02-23T00:31:02.556 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: Running main() from gmock_main.cc 2024-02-23T00:31:02.556 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-02-23T00:31:02.556 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] Global test environment set-up. 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (69 ms) 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockShared (4 ms) 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (2228 ms) 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1042 ms) 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (4 ms) 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-02-23T00:31:02.557 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5 ms) 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] 8 tests from LibRadosLock (3359 ms total) 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1066 ms) 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (7 ms) 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1004 ms) 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1011 ms) 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-02-23T00:31:02.558 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3 ms) 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3103 ms total) 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [----------] Global test environment tear-down 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (20511 ms total) 2024-02-23T00:31:02.559 INFO:tasks.workunit.client.0.smithi087.stdout: api_lock: [ PASSED ] 16 tests. 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:00.617516+0000 osd.0 (osd.0) 79 : cluster [DBG] 10.8 deep-scrub starts 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:00.671196+0000 osd.0 (osd.0) 80 : cluster [DBG] 10.8 deep-scrub ok 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.385343+0000 mon.c (mon.2) 64 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.476685+0000 mon.a (mon.0) 975 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi087-82139-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]': finished 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.476761+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.476839+0000 mon.a (mon.0) 977 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.476947+0000 mon.a (mon.0) 978 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.477023+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1"}]': finished 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:01.477165+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.482289+0000 mon.c (mon.2) 65 : audit [INF] from='client.? 172.21.15.87:0/4218332242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.482502+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:02.840 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.484241+0000 mon.c (mon.2) 66 : audit [INF] from='client.? 172.21.15.87:0/458030738' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi087-82011-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.484908+0000 mon.a (mon.0) 981 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]: dispatch 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.485222+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.485652+0000 mon.a (mon.0) 983 : audit [INF] from='client.? 172.21.15.87:0/92013801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.493122+0000 mon.a (mon.0) 984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.493250+0000 mon.b (mon.1) 99 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.493450+0000 mon.a (mon.0) 985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi087-82011-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:01.494276+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:01.505522+0000 mgr.y (mgr.24437) 164 : cluster [DBG] pgmap v149: 560 pgs: 1 active+clean+snaptrim, 108 unknown, 451 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.7 KiB/s wr, 2 op/s 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:01.693306+0000 osd.2 (osd.2) 61 : cluster [DBG] 10.5 deep-scrub starts 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:01.694254+0000 osd.2 (osd.2) 62 : cluster [DBG] 10.5 deep-scrub ok 2024-02-23T00:31:02.841 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:02.393918+0000 mon.c (mon.2) 67 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:02.876 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:31:02.877 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:31:02.877 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:31:02.877 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:31:02.877 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:31:02.877 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:31:02.877 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:31:02.877 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:31:02.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.869+0000 7f1b272e5700 1 -- 172.21.15.87:0/3335663935 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b200fcbb0 msgr2=0x7f1b200fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.869+0000 7f1b272e5700 1 --2- 172.21.15.87:0/3335663935 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b200fcbb0 0x7f1b200fcf90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1b1c004870 tx=0x7f1b1c009cb0).stop 2024-02-23T00:31:02.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.869+0000 7f1b272e5700 1 -- 172.21.15.87:0/3335663935 shutdown_connections 2024-02-23T00:31:02.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 --2- 172.21.15.87:0/3335663935 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b200f7400 0x7f1b20103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 --2- 172.21.15.87:0/3335663935 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b200fcbb0 0x7f1b200fcf90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 --2- 172.21.15.87:0/3335663935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b200fd510 0x7f1b200f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 -- 172.21.15.87:0/3335663935 >> 172.21.15.87:0/3335663935 conn(0x7f1b200f2820 msgr2=0x7f1b200f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 -- 172.21.15.87:0/3335663935 shutdown_connections 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 -- 172.21.15.87:0/3335663935 wait complete. 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 librados: starting msgr at 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 librados: starting objecter 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 Processor -- start 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 -- start start 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 librados: setting wanted keys 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 librados: calling monclient init 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b200f7400 0x7f1b2010d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b200fcbb0 0x7f1b2010d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b200fd510 0x7f1b20107340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b20107830 con 0x7f1b200f7400 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b201079a0 con 0x7f1b200fcbb0 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b272e5700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b20107b60 con 0x7f1b200fd510 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b25081700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b200f7400 0x7f1b2010d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b25882700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b200fd510 0x7f1b20107340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b25081700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b200f7400 0x7f1b2010d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58304/0 (socket says 172.21.15.87:58304) 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.870+0000 7f1b25081700 1 -- 172.21.15.87:0/1683571677 learned_addr learned my addr 172.21.15.87:0/1683571677 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b25081700 1 -- 172.21.15.87:0/1683571677 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b200fd510 msgr2=0x7f1b20107340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b25081700 1 --2- 172.21.15.87:0/1683571677 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b200fd510 0x7f1b20107340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b25081700 1 -- 172.21.15.87:0/1683571677 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b200fcbb0 msgr2=0x7f1b2010d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b24880700 1 --2- 172.21.15.87:0/1683571677 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b200fcbb0 0x7f1b2010d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b25081700 1 --2- 172.21.15.87:0/1683571677 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b200fcbb0 0x7f1b2010d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b25081700 1 -- 172.21.15.87:0/1683571677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b10005040 con 0x7f1b200f7400 2024-02-23T00:31:02.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b25882700 1 --2- 172.21.15.87:0/1683571677 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b200fd510 0x7f1b20107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b25081700 1 --2- 172.21.15.87:0/1683571677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b200f7400 0x7f1b2010d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1b1c009eb0 tx=0x7f1b1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b16777700 10 client.?.objecter ms_handle_connect 0x7f1b200f7400 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b16777700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c00e070 con 0x7f1b200f7400 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b1c013da0 con 0x7f1b200f7400 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b1c005040 con 0x7f1b200f7400 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c019070 con 0x7f1b200f7400 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b272e5700 10 client.15141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b20108000 con 0x7f1b200f7400 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.871+0000 7f1b272e5700 1 librados: init done 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.872+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1b201081f0 con 0x7f1b200f7400 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.872+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1b1c01e420 con 0x7f1b200f7400 2024-02-23T00:31:02.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b16777700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b0c062c90 0x7f1b0c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b24880700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b0c062c90 0x7f1b0c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(106..106 src has 38..106) v4 ==== 17078+0+0 (secure 0 0 0) 0x7f1b1c07fff0 con 0x7f1b200f7400 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 osd_map(106..106 src has 38..106) v4 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b16777700 3 client.15141.objecter handle_osd_map got epochs [106,106] > 0 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b16777700 3 client.15141.objecter handle_osd_map decoding full epoch 106 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b16777700 20 client.15141.objecter dump_active .. 0 homeless 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v106) v1 ==== 54+0+16 (secure 0 0 0) 0x7f1b1c087070 con 0x7f1b200f7400 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b16777700 10 client.15141.objecter ms_handle_connect 0x7f1b0c062c90 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b24880700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b0c062c90 0x7f1b0c065150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1b10009f20 tx=0x7f1b10009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 10 client.15141.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 1 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1b20061a90 0x7f1b200273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b2005fa60 osd=0 3 2024-02-23T00:31:02.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b2005fa60 osd=0 3 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b2005fa60 osd=0 3 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 1 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b2005fa60 osd=0 3 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 0 1 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b2005fa60 osd=0 4 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 10 client.15141.objecter _send_command 1 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b2005ee80 con 0x7f1b20061a90 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.873+0000 7f1b25882700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1b20061a90 0x7f1b200273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.874+0000 7f1b16777700 10 client.15141.objecter ms_handle_connect 0x7f1b20061a90 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.874+0000 7f1b25882700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1b20061a90 0x7f1b200273c0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.874+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1b2005ee80 con 0x7f1b20061a90 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.874+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.874+0000 7f1b16777700 10 client.15141.objecter _finish_command 1 = system:0 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.874+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b2005fa60 osd=0 4 2024-02-23T00:31:02.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.874+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 0 1 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 10 client.15141.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 2 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b2005fa60 osd=0 3 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b2005fa60 osd=0 3 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b2005fa60 osd=0 3 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 2 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b2005fa60 osd=0 3 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 0 2 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b2005fa60 osd=0 4 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 10 client.15141.objecter _send_command 2 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.889+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b201060e0 con 0x7f1b20061a90 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.890+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1b201060e0 con 0x7f1b20061a90 2024-02-23T00:31:02.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.890+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.890+0000 7f1b16777700 10 client.15141.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.890+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b2005fa60 osd=0 4 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.890+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 0 2 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.890+0000 7f1b272e5700 10 client.15141.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 3 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6805/685354171 conn(0x7f1b20028590 0x7f1b200fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20061230 osd=1 3 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20061230 osd=1 3 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20061230 osd=1 3 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 3 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20061230 osd=1 3 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 1 3 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20061230 osd=1 4 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 10 client.15141.objecter _send_command 3 2024-02-23T00:31:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b200613a0 con 0x7f1b20028590 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b25081700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6805/685354171 conn(0x7f1b20028590 0x7f1b200fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b16777700 10 client.15141.objecter ms_handle_connect 0x7f1b20028590 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b25081700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6805/685354171 conn(0x7f1b20028590 0x7f1b200fd9b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1b200613a0 con 0x7f1b20028590 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b16777700 10 client.15141.objecter _finish_command 3 = system:0 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20061230 osd=1 4 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.891+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 1 3 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 10 client.15141.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 4 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20061230 osd=1 3 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20061230 osd=1 3 2024-02-23T00:31:02.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20061230 osd=1 3 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 4 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20061230 osd=1 3 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 1 4 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20061230 osd=1 4 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 10 client.15141.objecter _send_command 4 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.906+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b200ff020 con 0x7f1b20028590 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.907+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1b200ff020 con 0x7f1b20028590 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.907+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.907+0000 7f1b16777700 10 client.15141.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20061230 osd=1 4 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 1 4 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 10 client.15141.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:02.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 5 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1b20059e00 0x7f1b2011e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20027df0 osd=2 3 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20027df0 osd=2 3 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20027df0 osd=2 3 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 5 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20027df0 osd=2 3 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 2 5 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20027df0 osd=2 4 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 10 client.15141.objecter _send_command 5 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b20121010 con 0x7f1b20059e00 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b24880700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1b20059e00 0x7f1b2011e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b24880700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1b20059e00 0x7f1b2011e3c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.908+0000 7f1b16777700 10 client.15141.objecter ms_handle_connect 0x7f1b20059e00 2024-02-23T00:31:02.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.909+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1b20121010 con 0x7f1b20059e00 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.909+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.909+0000 7f1b16777700 10 client.15141.objecter _finish_command 5 = system:0 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.909+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20027df0 osd=2 4 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.909+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 2 5 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 10 client.15141.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 6 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20027df0 osd=2 3 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20027df0 osd=2 3 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20027df0 osd=2 3 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 6 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20027df0 osd=2 3 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 2 6 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20027df0 osd=2 4 2024-02-23T00:31:02.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 10 client.15141.objecter _send_command 6 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.924+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b20059360 con 0x7f1b20059e00 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1b20059360 con 0x7f1b20059e00 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b16777700 10 client.15141.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20027df0 osd=2 4 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 2 6 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 10 client.15141.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 7 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6813/128059790 conn(0x7f1b2011fa30 0x7f1b2011fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b200fe670 osd=3 3 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b200fe670 osd=3 3 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b200fe670 osd=3 3 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 7 2024-02-23T00:31:02.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b200fe670 osd=3 3 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 3 7 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b200fe670 osd=3 4 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 10 client.15141.objecter _send_command 7 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b200fee10 con 0x7f1b2011fa30 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b25882700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6813/128059790 conn(0x7f1b2011fa30 0x7f1b2011fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b16777700 10 client.15141.objecter ms_handle_connect 0x7f1b2011fa30 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.926+0000 7f1b25882700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6813/128059790 conn(0x7f1b2011fa30 0x7f1b2011fe70 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.927+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1b200fee10 con 0x7f1b2011fa30 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.927+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.927+0000 7f1b16777700 10 client.15141.objecter _finish_command 7 = system:0 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.927+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b200fe670 osd=3 4 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.927+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 3 7 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 10 client.15141.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:02.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 8 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b200fe670 osd=3 3 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b200fe670 osd=3 3 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b200fe670 osd=3 3 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 8 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b200fe670 osd=3 3 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 3 8 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b200fe670 osd=3 4 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 10 client.15141.objecter _send_command 8 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.941+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b20120b60 con 0x7f1b2011fa30 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1b20120b60 con 0x7f1b2011fa30 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b16777700 10 client.15141.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b200fe670 osd=3 4 2024-02-23T00:31:02.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 3 8 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 10 client.15141.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 9 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1b201231e0 0x7f1b201255d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20120390 osd=4 3 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120390 osd=4 3 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20120390 osd=4 3 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 9 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20120390 osd=4 3 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 4 9 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120390 osd=4 4 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 10 client.15141.objecter _send_command 9 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b20120b40 con 0x7f1b201231e0 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.943+0000 7f1b25081700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1b201231e0 0x7f1b201255d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.944+0000 7f1b16777700 10 client.15141.objecter ms_handle_connect 0x7f1b201231e0 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.944+0000 7f1b25081700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1b201231e0 0x7f1b201255d0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.945+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1b20120b40 con 0x7f1b201231e0 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.945+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.945+0000 7f1b16777700 10 client.15141.objecter _finish_command 9 = system:0 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.945+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20120390 osd=4 4 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.945+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 4 9 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 10 client.15141.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 10 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20120390 osd=4 3 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120390 osd=4 3 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20120390 osd=4 3 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 10 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20120390 osd=4 3 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 4 10 2024-02-23T00:31:02.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120390 osd=4 4 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 10 client.15141.objecter _send_command 10 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.959+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b20120b40 con 0x7f1b201231e0 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1b20120b40 con 0x7f1b201231e0 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b16777700 10 client.15141.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20120390 osd=4 4 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 4 10 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 10 client.15141.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 11 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1b20127640 0x7f1b20127aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20120b40 osd=5 3 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120b40 osd=5 3 2024-02-23T00:31:02.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20120b40 osd=5 3 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 11 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20120b40 osd=5 3 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 5 11 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120b40 osd=5 4 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 10 client.15141.objecter _send_command 11 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b2012b460 con 0x7f1b20127640 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.961+0000 7f1b24880700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1b20127640 0x7f1b20127aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.962+0000 7f1b16777700 10 client.15141.objecter ms_handle_connect 0x7f1b20127640 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.962+0000 7f1b24880700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1b20127640 0x7f1b20127aa0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.962+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1b2012b460 con 0x7f1b20127640 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.962+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.962+0000 7f1b16777700 10 client.15141.objecter _finish_command 11 = system:0 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.962+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20120b40 osd=5 4 2024-02-23T00:31:02.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.962+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 5 11 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 10 client.15141.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 12 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20120b40 osd=5 3 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120b40 osd=5 3 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20120b40 osd=5 3 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 12 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20120b40 osd=5 3 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 5 12 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120b40 osd=5 4 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 10 client.15141.objecter _send_command 12 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.976+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b20125ac0 con 0x7f1b20127640 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1b20125ac0 con 0x7f1b20127640 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b16777700 10 client.15141.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20120b40 osd=5 4 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 5 12 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 10 client.15141.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 13 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1b201261a0 0x7f1b20126600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20125ac0 osd=6 3 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20125ac0 osd=6 3 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20125ac0 osd=6 3 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 13 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20125ac0 osd=6 3 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 6 13 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20125ac0 osd=6 4 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 10 client.15141.objecter _send_command 13 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b20126bb0 con 0x7f1b201261a0 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.978+0000 7f1b25882700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1b201261a0 0x7f1b20126600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.979+0000 7f1b16777700 10 client.15141.objecter ms_handle_connect 0x7f1b201261a0 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.979+0000 7f1b25882700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1b201261a0 0x7f1b20126600 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.979+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1b20126bb0 con 0x7f1b201261a0 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.979+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.979+0000 7f1b16777700 10 client.15141.objecter _finish_command 13 = system:0 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.979+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20125ac0 osd=6 4 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.979+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 6 13 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 10 client.15141.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 14 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20125ac0 osd=6 3 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20125ac0 osd=6 3 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20125ac0 osd=6 3 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 14 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20125ac0 osd=6 3 2024-02-23T00:31:02.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 6 14 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20125ac0 osd=6 4 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 10 client.15141.objecter _send_command 14 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.993+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b20126bb0 con 0x7f1b201261a0 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1b20126bb0 con 0x7f1b201261a0 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b16777700 10 client.15141.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20125ac0 osd=6 4 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 6 14 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 10 client.15141.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 15 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6812/373787000 conn(0x7f1b2012c800 0x7f1b2012ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20126bb0 osd=7 3 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20126bb0 osd=7 3 2024-02-23T00:31:02.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20126bb0 osd=7 3 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 15 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20126bb0 osd=7 3 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 7 15 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20126bb0 osd=7 4 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 10 client.15141.objecter _send_command 15 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.995+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b20131840 con 0x7f1b2012c800 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.996+0000 7f1b25081700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6812/373787000 conn(0x7f1b2012c800 0x7f1b2012ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.996+0000 7f1b25081700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6812/373787000 conn(0x7f1b2012c800 0x7f1b2012ebf0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.996+0000 7f1b16777700 10 client.15141.objecter ms_handle_connect 0x7f1b2012c800 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.997+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1b20131840 con 0x7f1b2012c800 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.997+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.997+0000 7f1b16777700 10 client.15141.objecter _finish_command 15 = system:0 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.997+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20126bb0 osd=7 4 2024-02-23T00:31:02.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:56.997+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 7 15 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 10 client.15141.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign -1 16 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20126bb0 osd=7 3 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20126bb0 osd=7 3 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 20 client.15141.objecter _get_session s=0x7f1b20126bb0 osd=7 3 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_remove -1 16 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 20 client.15141.objecter get_session s=0x7f1b20126bb0 osd=7 3 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 15 client.15141.objecter _session_command_op_assign 7 16 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20126bb0 osd=7 4 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 10 client.15141.objecter _send_command 16 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.011+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b2012f0e0 con 0x7f1b2012c800 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.012+0000 7f1b16777700 1 -- 172.21.15.87:0/1683571677 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1b2012f0e0 con 0x7f1b2012c800 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.012+0000 7f1b16777700 10 client.15141.objecter ms_dispatch 0x7f1b20001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.012+0000 7f1b16777700 10 client.15141.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.012+0000 7f1b16777700 20 client.15141.objecter put_session s=0x7f1b20126bb0 osd=7 4 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.012+0000 7f1b16777700 15 client.15141.objecter _session_command_op_remove 7 16 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 librados: watch_flush enter 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 librados: watch_flush exit 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 client.15141.objecter close_session for osd.0 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1b20061a90 msgr2=0x7f1b200273c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6801/2153191863 conn(0x7f1b20061a90 0x7f1b200273c0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b2005fa60 osd=0 1 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 client.15141.objecter close_session for osd.1 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6805/685354171 conn(0x7f1b20028590 msgr2=0x7f1b200fd9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6805/685354171 conn(0x7f1b20028590 0x7f1b200fd9b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20061230 osd=1 1 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 client.15141.objecter close_session for osd.2 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1b20059e00 msgr2=0x7f1b2011e3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6809/3605394228 conn(0x7f1b20059e00 0x7f1b2011e3c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20027df0 osd=2 1 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 client.15141.objecter close_session for osd.3 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b25882700 1 -- 172.21.15.87:0/1683571677 reap_dead start 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6813/128059790 conn(0x7f1b2011fa30 msgr2=0x7f1b2011fe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6813/128059790 conn(0x7f1b2011fa30 0x7f1b2011fe70 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b200fe670 osd=3 1 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 client.15141.objecter close_session for osd.4 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1b201231e0 msgr2=0x7f1b201255d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1b201231e0 0x7f1b201255d0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120390 osd=4 1 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 client.15141.objecter close_session for osd.5 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1b20127640 msgr2=0x7f1b20127aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1b20127640 0x7f1b20127aa0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20120b40 osd=5 1 2024-02-23T00:31:02.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 client.15141.objecter close_session for osd.6 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1b201261a0 msgr2=0x7f1b20126600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1b201261a0 0x7f1b20126600 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20125ac0 osd=6 1 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 client.15141.objecter close_session for osd.7 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6812/373787000 conn(0x7f1b2012c800 msgr2=0x7f1b2012ebf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.171:6812/373787000 conn(0x7f1b2012c800 0x7f1b2012ebf0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 20 client.15141.objecter put_session s=0x7f1b20126bb0 osd=7 1 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 20 client.15141.objecter shutdown clearing up homeless session... 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 10 client.15141.objecter successfully canceled tick 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b0c062c90 msgr2=0x7f1b0c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b0c062c90 0x7f1b0c065150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f1b10009f20 tx=0x7f1b10009cb0).stop 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b200f7400 msgr2=0x7f1b2010d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b25882700 1 -- 172.21.15.87:0/1683571677 reap_dead start 2024-02-23T00:31:02.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 --2- 172.21.15.87:0/1683571677 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b200f7400 0x7f1b2010d150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1b1c009eb0 tx=0x7f1b1c00b040).stop 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b25882700 1 -- 172.21.15.87:0/1683571677 reap_dead start 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 shutdown_connections 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 >> 172.21.15.87:0/1683571677 conn(0x7f1b200f2820 msgr2=0x7f1b200fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 shutdown_connections 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 -- 172.21.15.87:0/1683571677 wait complete. 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:30:57.013+0000 7f1b272e5700 1 librados: shutdown 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:31:02.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:00.617516+0000 osd.0 (osd.0) 79 : cluster [DBG] 10.8 deep-scrub starts 2024-02-23T00:31:02.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:00.671196+0000 osd.0 (osd.0) 80 : cluster [DBG] 10.8 deep-scrub ok 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.385343+0000 mon.c (mon.2) 64 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.476685+0000 mon.a (mon.0) 975 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi087-82139-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]': finished 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.476761+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.476839+0000 mon.a (mon.0) 977 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.476947+0000 mon.a (mon.0) 978 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.477023+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi087-82075-1","app":"app1","key":"key1"}]': finished 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:01.477165+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.482289+0000 mon.c (mon.2) 65 : audit [INF] from='client.? 172.21.15.87:0/4218332242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.482502+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.87:0/205797397' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.484241+0000 mon.c (mon.2) 66 : audit [INF] from='client.? 172.21.15.87:0/458030738' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi087-82011-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.484908+0000 mon.a (mon.0) 981 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]: dispatch 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.485222+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]: dispatch 2024-02-23T00:31:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.485652+0000 mon.a (mon.0) 983 : audit [INF] from='client.? 172.21.15.87:0/92013801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.493122+0000 mon.a (mon.0) 984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.493250+0000 mon.b (mon.1) 99 : audit [INF] from='client.? 172.21.15.87:0/1200380376' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.493450+0000 mon.a (mon.0) 985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi087-82011-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:01.494276+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]: dispatch 2024-02-23T00:31:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:01.505522+0000 mgr.y (mgr.24437) 164 : cluster [DBG] pgmap v149: 560 pgs: 1 active+clean+snaptrim, 108 unknown, 451 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.7 KiB/s wr, 2 op/s 2024-02-23T00:31:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:01.693306+0000 osd.2 (osd.2) 61 : cluster [DBG] 10.5 deep-scrub starts 2024-02-23T00:31:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:01.694254+0000 osd.2 (osd.2) 62 : cluster [DBG] 10.5 deep-scrub ok 2024-02-23T00:31:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:02.393918+0000 mon.c (mon.2) 67 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:03.558 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:03.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:03.671 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:03.699 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:03.719 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:03.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:03.823 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:02.486938+0000 mon.a (mon.0) 987 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]': finished 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:02.486978+0000 mon.a (mon.0) 988 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:02.487020+0000 mon.a (mon.0) 989 : audit [INF] from='client.? 172.21.15.87:0/92013801' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:02.487091+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:02.487151+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi087-82011-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:02.487186+0000 mon.a (mon.0) 992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:02.487236+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:02.494328+0000 mon.a (mon.0) 994 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "mode": "writeback"}]: dispatch 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:02.498424+0000 mon.a (mon.0) 995 : audit [INF] from='client.? 172.21.15.87:0/3965564315' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi087-82107-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:03.401683+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:03.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:02.486938+0000 mon.a (mon.0) 987 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]': finished 2024-02-23T00:31:03.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:02.486978+0000 mon.a (mon.0) 988 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:31:03.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:02.487020+0000 mon.a (mon.0) 989 : audit [INF] from='client.? 172.21.15.87:0/92013801' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:03.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:02.487091+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:03.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:02.487151+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi087-82011-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:03.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:02.487186+0000 mon.a (mon.0) 992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:31:03.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:02.487236+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-23T00:31:03.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:02.494328+0000 mon.a (mon.0) 994 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "mode": "writeback"}]: dispatch 2024-02-23T00:31:03.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:02.498424+0000 mon.a (mon.0) 995 : audit [INF] from='client.? 172.21.15.87:0/3965564315' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi087-82107-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:03.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:03.401683+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:03.847 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:31:03.848 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:02.486938+0000 mon.a (mon.0) 987 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]': finished 2024-02-23T00:31:03.852 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:02.486978+0000 mon.a (mon.0) 988 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi087-82067-10"}]': finished 2024-02-23T00:31:03.852 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:02.487020+0000 mon.a (mon.0) 989 : audit [INF] from='client.? 172.21.15.87:0/92013801' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:03.852 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:02.487091+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:03.852 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:02.487151+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi087-82011-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:03.852 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:02.487186+0000 mon.a (mon.0) 992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi087-82059-10"}]': finished 2024-02-23T00:31:03.852 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:02.487236+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-23T00:31:03.852 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:02.494328+0000 mon.a (mon.0) 994 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "mode": "writeback"}]: dispatch 2024-02-23T00:31:03.853 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:02.498424+0000 mon.a (mon.0) 995 : audit [INF] from='client.? 172.21.15.87:0/3965564315' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi087-82107-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:03.853 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:03.401683+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:03.853 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:03.870 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:03.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:03.974 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:04.001 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:04.020 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:04.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:04.124 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:04.211 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:04.297 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:04.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:04.365 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:31:04.365 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:31:04.365 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:31:04.365 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:31:04.411 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:04.437 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:04.456 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:04.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:04.560 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:04.586 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:04.605 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:04.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:04.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:31:04.736 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:04.756 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:31:04.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:04.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:31:04.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:04.944 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:05.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:03.484370+0000 mon.a (mon.0) 996 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:05.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:03.506297+0000 mgr.y (mgr.24437) 165 : cluster [DBG] pgmap v151: 560 pgs: 1 active+clean+snaptrim, 140 unknown, 419 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.7 KiB/s wr, 5 op/s 2024-02-23T00:31:05.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:03.525461+0000 mon.a (mon.0) 997 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "mode": "writeback"}]': finished 2024-02-23T00:31:05.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:03.525528+0000 mon.a (mon.0) 998 : audit [INF] from='client.? 172.21.15.87:0/3965564315' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi087-82107-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:05.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:03.525735+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-23T00:31:05.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:03.548965+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:31:05.204 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:04.409104+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.241+0000 7fe32efee700 1 -- 172.21.15.87:0/1871348009 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280f7400 msgr2=0x7fe328103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.241+0000 7fe32efee700 1 --2- 172.21.15.87:0/1871348009 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280f7400 0x7fe328103a20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fe320009be0 tx=0x7fe320004d10).stop 2024-02-23T00:31:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.243+0000 7fe32efee700 1 -- 172.21.15.87:0/1871348009 shutdown_connections 2024-02-23T00:31:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.243+0000 7fe32efee700 1 --2- 172.21.15.87:0/1871348009 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280f7400 0x7fe328103a20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.243+0000 7fe32efee700 1 --2- 172.21.15.87:0/1871348009 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3280fcbb0 0x7fe3280fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:05.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.243+0000 7fe32efee700 1 --2- 172.21.15.87:0/1871348009 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3280fd510 0x7fe3280f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.243+0000 7fe32efee700 1 -- 172.21.15.87:0/1871348009 >> 172.21.15.87:0/1871348009 conn(0x7fe3280f2820 msgr2=0x7fe3280f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.243+0000 7fe32efee700 1 -- 172.21.15.87:0/1871348009 shutdown_connections 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.243+0000 7fe32efee700 1 -- 172.21.15.87:0/1871348009 wait complete. 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.244+0000 7fe32efee700 1 librados: starting msgr at 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.244+0000 7fe32efee700 1 librados: starting objecter 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.244+0000 7fe32efee700 1 Processor -- start 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.246+0000 7fe32efee700 1 -- start start 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.246+0000 7fe32efee700 1 librados: setting wanted keys 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.246+0000 7fe32efee700 1 librados: calling monclient init 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.246+0000 7fe32efee700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3280f7400 0x7fe32810ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.246+0000 7fe32efee700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280fcbb0 0x7fe32810b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.246+0000 7fe32efee700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3280fd510 0x7fe328106020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:05.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.246+0000 7fe32efee700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe328112ce0 con 0x7fe3280fd510 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.246+0000 7fe32efee700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe328106510 con 0x7fe3280f7400 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.246+0000 7fe32efee700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3281066a0 con 0x7fe3280fcbb0 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.248+0000 7fe327fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280fcbb0 0x7fe32810b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.248+0000 7fe327fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280fcbb0 0x7fe32810b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:36570/0 (socket says 172.21.15.87:36570) 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.248+0000 7fe327fff700 1 -- 172.21.15.87:0/1505395554 learned_addr learned my addr 172.21.15.87:0/1505395554 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.249+0000 7fe32cd8a700 1 --2- 172.21.15.87:0/1505395554 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3280f7400 0x7fe32810ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.250+0000 7fe327fff700 1 -- 172.21.15.87:0/1505395554 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3280f7400 msgr2=0x7fe32810ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.250+0000 7fe327fff700 1 --2- 172.21.15.87:0/1505395554 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3280f7400 0x7fe32810ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.250+0000 7fe327fff700 1 -- 172.21.15.87:0/1505395554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3280fd510 msgr2=0x7fe328106020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.250+0000 7fe327fff700 1 --2- 172.21.15.87:0/1505395554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3280fd510 0x7fe328106020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:05.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.250+0000 7fe327fff700 1 -- 172.21.15.87:0/1505395554 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3200097c0 con 0x7fe3280fcbb0 2024-02-23T00:31:05.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.254+0000 7fe327fff700 1 --2- 172.21.15.87:0/1505395554 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280fcbb0 0x7fe32810b320 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fe31400b530 tx=0x7fe3140090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:05.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.254+0000 7fe325ffb700 10 client.?.objecter ms_handle_connect 0x7fe3280fcbb0 2024-02-23T00:31:05.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.254+0000 7fe325ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:05.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.260+0000 7fe325ffb700 1 -- 172.21.15.87:0/1505395554 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31400d4a0 con 0x7fe3280fcbb0 2024-02-23T00:31:05.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.260+0000 7fe325ffb700 1 -- 172.21.15.87:0/1505395554 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe31400dad0 con 0x7fe3280fcbb0 2024-02-23T00:31:05.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.260+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe328106980 con 0x7fe3280fcbb0 2024-02-23T00:31:05.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.260+0000 7fe325ffb700 1 -- 172.21.15.87:0/1505395554 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314012a50 con 0x7fe3280fcbb0 2024-02-23T00:31:05.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.260+0000 7fe32efee700 10 client.26702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:05.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.260+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe328123e90 con 0x7fe3280fcbb0 2024-02-23T00:31:05.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.260+0000 7fe32efee700 1 librados: init done 2024-02-23T00:31:05.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.262+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32805fa60 con 0x7fe3280fcbb0 2024-02-23T00:31:05.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.264+0000 7fe325ffb700 1 -- 172.21.15.87:0/1505395554 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fe31400d600 con 0x7fe3280fcbb0 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.264+0000 7fe325ffb700 1 --2- 172.21.15.87:0/1505395554 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe318062ce0 0x7fe3180651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.264+0000 7fe325ffb700 1 -- 172.21.15.87:0/1505395554 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(114..114 src has 38..114) v4 ==== 11906+0+0 (secure 0 0 0) 0x7fe31407eb60 con 0x7fe3280fcbb0 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.264+0000 7fe325ffb700 10 client.26702.objecter ms_dispatch 0x7fe328001550 osd_map(114..114 src has 38..114) v4 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.264+0000 7fe325ffb700 3 client.26702.objecter handle_osd_map got epochs [114,114] > 0 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.264+0000 7fe325ffb700 3 client.26702.objecter handle_osd_map decoding full epoch 114 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.265+0000 7fe325ffb700 20 client.26702.objecter dump_active .. 0 homeless 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.265+0000 7fe32cd8a700 1 --2- 172.21.15.87:0/1505395554 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe318062ce0 0x7fe3180651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.265+0000 7fe32cd8a700 1 --2- 172.21.15.87:0/1505395554 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe318062ce0 0x7fe3180651a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fe31c004be0 tx=0x7fe31c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.265+0000 7fe325ffb700 10 client.26702.objecter ms_handle_connect 0x7fe318062ce0 2024-02-23T00:31:05.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.268+0000 7fe325ffb700 1 -- 172.21.15.87:0/1505395554 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe314051120 con 0x7fe3280fcbb0 2024-02-23T00:31:05.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:05.567+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe32805ee80 con 0x7fe3280fcbb0 2024-02-23T00:31:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:03.484370+0000 mon.a (mon.0) 996 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:03.506297+0000 mgr.y (mgr.24437) 165 : cluster [DBG] pgmap v151: 560 pgs: 1 active+clean+snaptrim, 140 unknown, 419 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.7 KiB/s wr, 5 op/s 2024-02-23T00:31:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:03.525461+0000 mon.a (mon.0) 997 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "mode": "writeback"}]': finished 2024-02-23T00:31:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:03.525528+0000 mon.a (mon.0) 998 : audit [INF] from='client.? 172.21.15.87:0/3965564315' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi087-82107-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:03.525735+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-23T00:31:05.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:03.548965+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:31:05.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:04.409104+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:05.592 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:31:05.627 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:05.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:05.650 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-23T00:31:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:03.484370+0000 mon.a (mon.0) 996 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:03.506297+0000 mgr.y (mgr.24437) 165 : cluster [DBG] pgmap v151: 560 pgs: 1 active+clean+snaptrim, 140 unknown, 419 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 6.7 KiB/s wr, 5 op/s 2024-02-23T00:31:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:03.525461+0000 mon.a (mon.0) 997 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "mode": "writeback"}]': finished 2024-02-23T00:31:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:03.525528+0000 mon.a (mon.0) 998 : audit [INF] from='client.? 172.21.15.87:0/3965564315' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi087-82107-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:03.525735+0000 mon.a (mon.0) 999 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-23T00:31:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:03.548965+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:31:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:04.409104+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:05.719 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:05.736 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:05.762 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:05.781 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:05.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:05.885 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:05.911 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:05.930 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:05.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:05.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:06.034 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:06.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.200+0000 7fe325ffb700 1 -- 172.21.15.87:0/1505395554 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v115) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe314050d10 con 0x7fe3280fcbb0 2024-02-23T00:31:06.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.216+0000 7fe32efee700 10 librados: watch_flush enter 2024-02-23T00:31:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.221+0000 7fe32efee700 10 librados: watch_flush exit 2024-02-23T00:31:06.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.221+0000 7fe32efee700 20 client.26702.objecter shutdown clearing up homeless session... 2024-02-23T00:31:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.221+0000 7fe32efee700 10 client.26702.objecter successfully canceled tick 2024-02-23T00:31:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.221+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe318062ce0 msgr2=0x7fe3180651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.221+0000 7fe32efee700 1 --2- 172.21.15.87:0/1505395554 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe318062ce0 0x7fe3180651a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fe31c004be0 tx=0x7fe31c00b040).stop 2024-02-23T00:31:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.221+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280fcbb0 msgr2=0x7fe32810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.221+0000 7fe32efee700 1 --2- 172.21.15.87:0/1505395554 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280fcbb0 0x7fe32810b320 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fe31400b530 tx=0x7fe3140090f0).stop 2024-02-23T00:31:06.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.221+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 shutdown_connections 2024-02-23T00:31:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.221+0000 7fe32efee700 1 --2- 172.21.15.87:0/1505395554 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe318062ce0 0x7fe3180651a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.222+0000 7fe32efee700 1 --2- 172.21.15.87:0/1505395554 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe3280fd510 0x7fe328106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.222+0000 7fe32efee700 1 --2- 172.21.15.87:0/1505395554 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe3280f7400 0x7fe32810ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.222+0000 7fe32efee700 1 --2- 172.21.15.87:0/1505395554 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe3280fcbb0 0x7fe32810b320 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.222+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 >> 172.21.15.87:0/1505395554 conn(0x7fe3280f2820 msgr2=0x7fe3280fab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.222+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 shutdown_connections 2024-02-23T00:31:06.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.222+0000 7fe32efee700 1 -- 172.21.15.87:0/1505395554 wait complete. 2024-02-23T00:31:06.227 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:06.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:06.222+0000 7fe32efee700 1 librados: shutdown 2024-02-23T00:31:06.236 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:31:06.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:06.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:06.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:06.458 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:06.823 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:06.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:06.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:06.848 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-02-23T00:31:06.848 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify_test_cb 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (214 ms) 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94856416788560 err -107 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (7 ms) 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94856416788560 err -107 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (29 ms) 2024-02-23T00:31:06.849 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 14856 notify_id 420906795008 cookie 94856416809088 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (10 ms) 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 14856 notify_id 420906795008 cookie 94856416809088 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (13 ms) 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 14856 notify_id 420906795009 cookie 94856416834384 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (20 ms) 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 14856 notify_id 420906795010 cookie 94856416828064 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 14856 notify_id 420906795010 cookie 94856416834384 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (22 ms) 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 14856 notify_id 420906795009 cookie 94856416830464 2024-02-23T00:31:06.850 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 14856 notify_id 425201762306 cookie 94856416830464 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3008 ms) 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94856416830464 err -107 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_cb from 14856 notify_id 455266533378 cookie 94856416830464 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94856416830464 err -107 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9335 ms total) 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-02-23T00:31:06.851 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: watch_notify_test_cb 2024-02-23T00:31:06.852 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1123 ms) 2024-02-23T00:31:06.852 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1123 ms total) 2024-02-23T00:31:06.852 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: 2024-02-23T00:31:06.852 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-02-23T00:31:06.852 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (24788 ms total) 2024-02-23T00:31:06.852 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1185 ms) 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (5 ms) 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1190 ms total) 2024-02-23T00:31:06.897 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (54 ms) 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4107 ms) 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (76 ms) 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (11 ms) 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402138000 notify_id 485331304451 notifier_gid 15147 2024-02-23T00:31:06.898 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (7 ms) 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402367200 notify_id 485331304450 notifier_gid 15147 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (37 ms) 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402138000 notify_id 485331304451 notifier_gid 15147 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (42 ms) 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402138000 notify_id 485331304452 notifier_gid 15147 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (32 ms) 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402138000 notify_id 485331304453 notifier_gid 15147 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (6 ms) 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402138000 notify_id 485331304452 notifier_gid 15147 2024-02-23T00:31:06.899 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (7 ms) 2024-02-23T00:31:06.900 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-02-23T00:31:06.900 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: trying... 2024-02-23T00:31:06.900 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402367200 notify_id 485331304454 notifier_gid 15147 2024-02-23T00:31:06.900 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: timed out 2024-02-23T00:31:06.900 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushing 2024-02-23T00:31:06.927 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:07.007 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:07.120 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.167632+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]': finished 2024-02-23T00:31:07.120 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:05.167781+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-23T00:31:07.120 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.171032+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? 172.21.15.87:0/611985872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.120 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.176913+0000 mon.c (mon.2) 70 : audit [INF] from='client.? 172.21.15.87:0/3255892024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi087-82011-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.120 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.184223+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.193404+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.196791+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.87:0/2330528609' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.206986+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.207220+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.237452+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi087-82011-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.410167+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.568890+0000 mon.c (mon.2) 72 : audit [INF] from='client.? 172.21.15.87:0/1505395554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:05.569284+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:06.164978+0000 mon.a (mon.0) 1009 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.167632+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]': finished 2024-02-23T00:31:07.121 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:05.167781+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.171032+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? 172.21.15.87:0/611985872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.176913+0000 mon.c (mon.2) 70 : audit [INF] from='client.? 172.21.15.87:0/3255892024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi087-82011-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.184223+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.193404+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.196791+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.87:0/2330528609' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.206986+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.207220+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.237452+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi087-82011-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.410167+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.568890+0000 mon.c (mon.2) 72 : audit [INF] from='client.? 172.21.15.87:0/1505395554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:05.569284+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:07.122 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:06.164978+0000 mon.a (mon.0) 1009 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.167632+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi087-82139-12"}]': finished 2024-02-23T00:31:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:05.167781+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-23T00:31:07.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.171032+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? 172.21.15.87:0/611985872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.176913+0000 mon.c (mon.2) 70 : audit [INF] from='client.? 172.21.15.87:0/3255892024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi087-82011-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.184223+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.193404+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi087-82139-12"}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.196791+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.87:0/2330528609' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.206986+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.207220+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.237452+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi087-82011-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.410167+0000 mon.c (mon.2) 71 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.568890+0000 mon.c (mon.2) 72 : audit [INF] from='client.? 172.21.15.87:0/1505395554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:05.569284+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:07.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:06.164978+0000 mon.a (mon.0) 1009 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:07.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:07.751 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:07.759 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:07.785 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:07.804 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:07.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:07.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:07.908 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:07.935 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:05.508512+0000 mgr.y (mgr.24437) 166 : cluster [DBG] pgmap v154: 552 pgs: 128 creating+peering, 1 active+clean+snaptrim, 423 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-02-23T00:31:07.935 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:06.177725+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? 172.21.15.87:0/611985872' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:07.935 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:06.177841+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi087-82139-12"}]': finished 2024-02-23T00:31:07.935 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:06.177944+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:07.935 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:06.178003+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:07.935 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:06.178078+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi087-82011-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:07.935 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:06.178181+0000 mon.a (mon.0) 1015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:31:07.935 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:06.178231+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-23T00:31:07.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:06.316790+0000 osd.2 (osd.2) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:07.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:06.316850+0000 osd.2 (osd.2) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:07.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:06.427382+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:07.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:06.825294+0000 mon.a (mon.0) 1017 : cluster [WRN] pool 'PoolQuotaPP_smithi087-82019-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-23T00:31:07.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:06.826179+0000 mon.a (mon.0) 1018 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-23T00:31:07.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:06.889273+0000 mon.b (mon.1) 102 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:07.425760+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-23T00:31:07.936 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:07.433937+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.937 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:07.456994+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:07.937 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:05.508512+0000 mgr.y (mgr.24437) 166 : cluster [DBG] pgmap v154: 552 pgs: 128 creating+peering, 1 active+clean+snaptrim, 423 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-02-23T00:31:07.937 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:06.177725+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? 172.21.15.87:0/611985872' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:07.937 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:06.177841+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi087-82139-12"}]': finished 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:06.177944+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:06.178003+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:06.178078+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi087-82011-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:06.178181+0000 mon.a (mon.0) 1015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:06.178231+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:06.316790+0000 osd.2 (osd.2) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:06.316850+0000 osd.2 (osd.2) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:06.427382+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:06.825294+0000 mon.a (mon.0) 1017 : cluster [WRN] pool 'PoolQuotaPP_smithi087-82019-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:06.826179+0000 mon.a (mon.0) 1018 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-23T00:31:07.938 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:06.889273+0000 mon.b (mon.1) 102 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.939 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:07.425760+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-23T00:31:07.939 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:07.433937+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:07.939 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:07.456994+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:07.971 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:08.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:05.508512+0000 mgr.y (mgr.24437) 166 : cluster [DBG] pgmap v154: 552 pgs: 128 creating+peering, 1 active+clean+snaptrim, 423 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-02-23T00:31:08.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:06.177725+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? 172.21.15.87:0/611985872' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:08.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:06.177841+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? 172.21.15.87:0/3542609165' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi087-82139-12"}]': finished 2024-02-23T00:31:08.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:06.177944+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:06.178003+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:06.178078+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi087-82011-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:06.178181+0000 mon.a (mon.0) 1015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:06.178231+0000 mon.a (mon.0) 1016 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:06.316790+0000 osd.2 (osd.2) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:06.316850+0000 osd.2 (osd.2) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:06.427382+0000 mon.c (mon.2) 73 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:06.825294+0000 mon.a (mon.0) 1017 : cluster [WRN] pool 'PoolQuotaPP_smithi087-82019-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:06.826179+0000 mon.a (mon.0) 1018 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:06.889273+0000 mon.b (mon.1) 102 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:07.425760+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:07.433937+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:08.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:07.456994+0000 mon.c (mon.2) 74 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:08.298 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:08.572 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:08.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:08.691 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:08.718 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:08.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:08.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:08.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:08.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:08.925 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:09.195 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:07.886192+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:09.196 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:07.886364+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-23T00:31:09.196 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:07.928317+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.87:0/2607672493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:09.196 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:07.928797+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:09.196 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:08.071917+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:09.196 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:08.461139+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:09.196 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:07.886192+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:09.197 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:07.886364+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-23T00:31:09.197 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:07.928317+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.87:0/2607672493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:09.197 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:07.928797+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:09.197 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:08.071917+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:09.197 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:08.461139+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:09.313 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:31:09.392 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:09.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:07.886192+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:07.886364+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-23T00:31:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:07.928317+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.87:0/2607672493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:07.928797+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:08.071917+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:08.461139+0000 mon.c (mon.2) 75 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:09.503 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:09.731 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:10.133 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:10.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:10.237 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:10.282 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:10.397 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:10.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:10.413 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:07.511750+0000 mgr.y (mgr.24437) 167 : cluster [DBG] pgmap v157: 552 pgs: 32 unknown, 128 creating+peering, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1022 B/s rd, 5.5 KiB/s wr, 4 op/s 2024-02-23T00:31:10.413 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:09.127346+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:10.413 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:09.128225+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-23T00:31:10.413 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:09.142234+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:09.148070+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.87:0/71734753' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi087-82011-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:09.148419+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.87:0/3024219439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:09.178327+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi087-82011-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:09.181180+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:09.474507+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:09.964764+0000 mon.a (mon.0) 1029 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:10.139348+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:10.139481+0000 mon.a (mon.0) 1031 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi087-82011-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:10.139561+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:10.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:10.139698+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-23T00:31:10.415 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:07.511750+0000 mgr.y (mgr.24437) 167 : cluster [DBG] pgmap v157: 552 pgs: 32 unknown, 128 creating+peering, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1022 B/s rd, 5.5 KiB/s wr, 4 op/s 2024-02-23T00:31:10.415 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:09.127346+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:10.415 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:09.128225+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-23T00:31:10.415 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:09.142234+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:10.415 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:09.148070+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.87:0/71734753' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi087-82011-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.415 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:09.148419+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.87:0/3024219439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.415 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:09.178327+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi087-82011-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.415 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:09.181180+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.415 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:09.474507+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:10.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:09.964764+0000 mon.a (mon.0) 1029 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:10.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:10.139348+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:10.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:10.139481+0000 mon.a (mon.0) 1031 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi087-82011-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:10.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:10.139561+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:10.419 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:10.139698+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-23T00:31:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:07.511750+0000 mgr.y (mgr.24437) 167 : cluster [DBG] pgmap v157: 552 pgs: 32 unknown, 128 creating+peering, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1022 B/s rd, 5.5 KiB/s wr, 4 op/s 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:09.127346+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:09.128225+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:09.142234+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:09.148070+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.87:0/71734753' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi087-82011-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:09.148419+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.87:0/3024219439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:09.178327+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi087-82011-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:09.181180+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:09.474507+0000 mon.c (mon.2) 76 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:09.964764+0000 mon.a (mon.0) 1029 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:10.139348+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:10.139481+0000 mon.a (mon.0) 1031 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi087-82011-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:10.139561+0000 mon.a (mon.0) 1032 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82187-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:10.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:10.139698+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-23T00:31:10.502 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:10.539 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:10.775 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:10.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:10.879 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:10.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:31:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:31:10.963 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:11.173 INFO:tasks.workunit.client.0.smithi087.stdout: api_ api_c_write_operations: Running main() from gmock_main.cc 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-02-23T00:31:11.174 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (7848 ms) 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3559 ms) 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (4063 ms) 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3236 ms) 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2829 ms) 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (4350 ms) 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (3255 ms) 2024-02-23T00:31:11.175 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (29140 ms total) 2024-02-23T00:31:11.176 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: 2024-02-23T00:31:11.176 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-02-23T00:31:11.176 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (29140 ms total) 2024-02-23T00:31:11.176 INFO:tasks.workunit.client.0.smithi087.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-02-23T00:31:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:09.522652+0000 mgr.y (mgr.24437) 168 : cluster [DBG] pgmap v160: 584 pgs: 160 unknown, 32 creating+peering, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:10.159195+0000 mon.b (mon.1) 107 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:10.162867+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:10.478523+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:11.153982+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-7"}]': finished 2024-02-23T00:31:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:11.154133+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-23T00:31:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:11.169518+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? 172.21.15.87:0/786827881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:11.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:09.522652+0000 mgr.y (mgr.24437) 168 : cluster [DBG] pgmap v160: 584 pgs: 160 unknown, 32 creating+peering, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:11.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:10.159195+0000 mon.b (mon.1) 107 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:11.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:10.162867+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:11.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:10.478523+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:11.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:11.153982+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-7"}]': finished 2024-02-23T00:31:11.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:11.154133+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-23T00:31:11.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:11.169518+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? 172.21.15.87:0/786827881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:11.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:09.522652+0000 mgr.y (mgr.24437) 168 : cluster [DBG] pgmap v160: 584 pgs: 160 unknown, 32 creating+peering, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:11.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:10.159195+0000 mon.b (mon.1) 107 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:11.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:10.162867+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:11.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:10.478523+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:11.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:11.153982+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-7"}]': finished 2024-02-23T00:31:11.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:11.154133+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-23T00:31:11.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:11.169518+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? 172.21.15.87:0/786827881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:11.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.842+0000 7efc39f83700 1 -- 172.21.15.87:0/203224935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 msgr2=0x7efc34103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:11.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.842+0000 7efc39f83700 1 --2- 172.21.15.87:0/203224935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 0x7efc34103920 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7efc28004870 tx=0x7efc28009cb0).stop 2024-02-23T00:31:11.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.844+0000 7efc39f83700 1 -- 172.21.15.87:0/203224935 shutdown_connections 2024-02-23T00:31:11.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.844+0000 7efc39f83700 1 --2- 172.21.15.87:0/203224935 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 0x7efc34103920 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:11.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.844+0000 7efc39f83700 1 --2- 172.21.15.87:0/203224935 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc340fcab0 0x7efc340fce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:11.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.844+0000 7efc39f83700 1 --2- 172.21.15.87:0/203224935 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc340fd410 0x7efc340f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:11.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.844+0000 7efc39f83700 1 -- 172.21.15.87:0/203224935 >> 172.21.15.87:0/203224935 conn(0x7efc340708b0 msgr2=0x7efc340f4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:11.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.845+0000 7efc39f83700 1 -- 172.21.15.87:0/203224935 shutdown_connections 2024-02-23T00:31:11.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.845+0000 7efc39f83700 1 -- 172.21.15.87:0/203224935 wait complete. 2024-02-23T00:31:11.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.846+0000 7efc39f83700 1 librados: starting msgr at 2024-02-23T00:31:11.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.846+0000 7efc39f83700 1 librados: starting objecter 2024-02-23T00:31:11.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.846+0000 7efc39f83700 1 Processor -- start 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.846+0000 7efc39f83700 1 -- start start 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.846+0000 7efc39f83700 1 librados: setting wanted keys 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.846+0000 7efc39f83700 1 librados: calling monclient init 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc39f83700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 0x7efc3410acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc39f83700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc340fcab0 0x7efc3410b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc39f83700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc340fd410 0x7efc34105f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc39f83700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc34106510 con 0x7efc340f7320 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc39f83700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc34106680 con 0x7efc340fcab0 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc39f83700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc34106840 con 0x7efc340fd410 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc337fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 0x7efc3410acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc337fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 0x7efc3410acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51742/0 (socket says 172.21.15.87:51742) 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc337fe700 1 -- 172.21.15.87:0/1526244710 learned_addr learned my addr 172.21.15.87:0/1526244710 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:11.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.847+0000 7efc33fff700 1 --2- 172.21.15.87:0/1526244710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc340fd410 0x7efc34105f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:11.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.851+0000 7efc337fe700 1 -- 172.21.15.87:0/1526244710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc340fd410 msgr2=0x7efc34105f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:11.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.851+0000 7efc337fe700 1 --2- 172.21.15.87:0/1526244710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc340fd410 0x7efc34105f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:11.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.851+0000 7efc337fe700 1 -- 172.21.15.87:0/1526244710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc340fcab0 msgr2=0x7efc3410b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:11.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.851+0000 7efc337fe700 1 --2- 172.21.15.87:0/1526244710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc340fcab0 0x7efc3410b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:11.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.851+0000 7efc337fe700 1 -- 172.21.15.87:0/1526244710 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc28005040 con 0x7efc340f7320 2024-02-23T00:31:11.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.852+0000 7efc337fe700 1 --2- 172.21.15.87:0/1526244710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 0x7efc3410acc0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7efc2400b530 tx=0x7efc240090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:11.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.852+0000 7efc30ff9700 10 client.?.objecter ms_handle_connect 0x7efc340f7320 2024-02-23T00:31:11.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.852+0000 7efc30ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:11.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.858+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc24002320 con 0x7efc340f7320 2024-02-23T00:31:11.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.858+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc24002950 con 0x7efc340f7320 2024-02-23T00:31:11.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.858+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc24012930 con 0x7efc340f7320 2024-02-23T00:31:11.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.860+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc34106b20 con 0x7efc340f7320 2024-02-23T00:31:11.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.860+0000 7efc39f83700 10 client.19308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:11.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.860+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc34112cc0 con 0x7efc340f7320 2024-02-23T00:31:11.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.860+0000 7efc39f83700 1 librados: init done 2024-02-23T00:31:11.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:11.861+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc34107880 con 0x7efc340f7320 2024-02-23T00:31:12.147 INFO:tasks.workunit.client.0.smithi087.stdout:ve (1 hosts, 3 zones) 2024-02-23T00:31:12.147 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-02-23T00:31:12.147 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-23T00:31:12.147 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: data: 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pools: 30 pools, 900 pgs 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: objects: 224 objects, 5.7 KiB 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: pgs: 59.667% pgs unknown 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 25.667% pgs not active 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 537 unknown 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 231 creating+peering 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 132 active+clean 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: io: 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: client: 994 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:31:12.148 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:31:12.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4837 ms) 2024-02-23T00:31:12.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ RUN ] LibRadosService.Status 2024-02-23T00:31:12.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ OK ] LibRadosService.Status (20099 ms) 2024-02-23T00:31:12.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [----------] 4 tests from LibRadosService (30075 ms total) 2024-02-23T00:31:12.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: 2024-02-23T00:31:12.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [----------] Global test environment tear-down 2024-02-23T00:31:12.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [==========] 4 tests from 1 test suite ran. (30075 ms total) 2024-02-23T00:31:12.149 INFO:tasks.workunit.client.0.smithi087.stdout: api_service: [ PASSED ] 4 tests. 2024-02-23T00:31:12.181 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:12.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:12.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:12.293 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:31:12.320 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:12.339 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:31:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:12.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:31:12.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.425+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7efc24002480 con 0x7efc340f7320 2024-02-23T00:31:12.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.425+0000 7efc30ff9700 1 --2- 172.21.15.87:0/1526244710 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc1c062ce0 0x7efc1c0651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:12.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.425+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(121..121 src has 38..121) v4 ==== 11848+0+0 (secure 0 0 0) 0x7efc2407e490 con 0x7efc340f7320 2024-02-23T00:31:12.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.425+0000 7efc30ff9700 10 client.19308.objecter ms_dispatch 0x7efc34001550 osd_map(121..121 src has 38..121) v4 2024-02-23T00:31:12.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.425+0000 7efc30ff9700 3 client.19308.objecter handle_osd_map got epochs [121,121] > 0 2024-02-23T00:31:12.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.425+0000 7efc30ff9700 3 client.19308.objecter handle_osd_map decoding full epoch 121 2024-02-23T00:31:12.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.426+0000 7efc30ff9700 10 client.19308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:12.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.426+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=122}) v3 -- 0x7efc1c077b50 con 0x7efc340f7320 2024-02-23T00:31:12.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.426+0000 7efc30ff9700 20 client.19308.objecter dump_active .. 0 homeless 2024-02-23T00:31:12.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.426+0000 7efc32ffd700 1 --2- 172.21.15.87:0/1526244710 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc1c062ce0 0x7efc1c0651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:12.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.428+0000 7efc32ffd700 1 --2- 172.21.15.87:0/1526244710 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc1c062ce0 0x7efc1c0651a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7efc341076a0 tx=0x7efc2c009370).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:12.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.428+0000 7efc30ff9700 10 client.19308.objecter ms_handle_connect 0x7efc1c062ce0 2024-02-23T00:31:12.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.431+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efc24050af0 con 0x7efc340f7320 2024-02-23T00:31:12.442 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:12.471 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:12.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:10.651441+0000 mgr.y (mgr.24437) 169 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:12.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:11.463787+0000 mon.b (mon.1) 108 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:12.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:11.482246+0000 mon.c (mon.2) 78 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:12.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:11.529836+0000 mgr.y (mgr.24437) 170 : cluster [DBG] pgmap v163: 520 pgs: 32 unknown, 488 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 4 op/s 2024-02-23T00:31:12.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:11.842982+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:12.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:11.848672+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? 172.21.15.87:0/786827881' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:12.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:11.849110+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-23T00:31:12.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:11.859682+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.87:0/3939712127' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi087-82011-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:12.522 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:12] "GET /metrics HTTP/1.1" 200 211966 "" "Prometheus/2.33.4" 2024-02-23T00:31:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:12.523 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:12.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:12.588 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [----------] Global test environment set-up. 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5290 ms) 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (38 ms) 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20051 ms) 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: attempt 0 of 20 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5082 ms) 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30461 ms total) 2024-02-23T00:31:12.589 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: 2024-02-23T00:31:12.590 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [----------] Global test environment tear-down 2024-02-23T00:31:12.590 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30461 ms total) 2024-02-23T00:31:12.590 INFO:tasks.workunit.client.0.smithi087.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-02-23T00:31:12.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:10.651441+0000 mgr.y (mgr.24437) 169 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:11.463787+0000 mon.b (mon.1) 108 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:11.482246+0000 mon.c (mon.2) 78 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:11.529836+0000 mgr.y (mgr.24437) 170 : cluster [DBG] pgmap v163: 520 pgs: 32 unknown, 488 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 4 op/s 2024-02-23T00:31:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:11.842982+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:11.848672+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? 172.21.15.87:0/786827881' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:11.849110+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-23T00:31:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:11.859682+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.87:0/3939712127' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi087-82011-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:12.713 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:12.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.774+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc341073e0 con 0x7efc1c062ce0 2024-02-23T00:31:12.785 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:10.651441+0000 mgr.y (mgr.24437) 169 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:12.786 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:11.463787+0000 mon.b (mon.1) 108 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:12.786 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:11.482246+0000 mon.c (mon.2) 78 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:12.786 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:11.529836+0000 mgr.y (mgr.24437) 170 : cluster [DBG] pgmap v163: 520 pgs: 32 unknown, 488 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.7 KiB/s wr, 4 op/s 2024-02-23T00:31:12.786 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:11.842982+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:12.786 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:11.848672+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? 172.21.15.87:0/786827881' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:12.786 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:11.849110+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-23T00:31:12.786 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:11.859682+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.87:0/3939712127' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi087-82011-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:12.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.792+0000 7f4ed9590700 1 -- 172.21.15.87:0/81410914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed406cc20 msgr2=0x7f4ed406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:12.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.792+0000 7f4ed9590700 1 --2- 172.21.15.87:0/81410914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed406cc20 0x7f4ed406b170 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 rx=0x7f4ebc004870 tx=0x7f4ebc009cb0).stop 2024-02-23T00:31:12.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.792+0000 7f4ed9590700 1 -- 172.21.15.87:0/81410914 shutdown_connections 2024-02-23T00:31:12.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.792+0000 7f4ed9590700 1 --2- 172.21.15.87:0/81410914 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed4108720 0x7f4ed410ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:12.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.792+0000 7f4ed9590700 1 --2- 172.21.15.87:0/81410914 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed406cc20 0x7f4ed406b170 secure :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 rx=0x7f4ebc004870 tx=0x7f4ebc009cb0).stop 2024-02-23T00:31:12.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.792+0000 7f4ed9590700 1 --2- 172.21.15.87:0/81410914 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4ed406b660 0x7f4ed406bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:12.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.792+0000 7f4ed9590700 1 -- 172.21.15.87:0/81410914 >> 172.21.15.87:0/81410914 conn(0x7f4ed40f3800 msgr2=0x7f4ed40f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:12.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.792+0000 7f4ed9590700 1 -- 172.21.15.87:0/81410914 shutdown_connections 2024-02-23T00:31:12.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.792+0000 7f4ed9590700 1 -- 172.21.15.87:0/81410914 wait complete. 2024-02-23T00:31:12.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 librados: starting msgr at 2024-02-23T00:31:12.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 librados: starting objecter 2024-02-23T00:31:12.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 Processor -- start 2024-02-23T00:31:12.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 -- start start 2024-02-23T00:31:12.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 librados: setting wanted keys 2024-02-23T00:31:12.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 librados: calling monclient init 2024-02-23T00:31:12.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed406b660 0x7f4ed406a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:12.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4ed4108720 0x7f4ed406a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:12.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed40633f0 0x7f4ed4063870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:12.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed4063e90 con 0x7f4ed406b660 2024-02-23T00:31:12.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed4064000 con 0x7f4ed40633f0 2024-02-23T00:31:12.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.793+0000 7f4ed9590700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40641c0 con 0x7f4ed4108720 2024-02-23T00:31:12.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.795+0000 7f4ed37fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed40633f0 0x7f4ed4063870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:12.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.795+0000 7f4ed37fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed40633f0 0x7f4ed4063870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.87:41880/0 (socket says 172.21.15.87:41880) 2024-02-23T00:31:12.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.795+0000 7f4ed37fe700 1 -- 172.21.15.87:0/2989193080 learned_addr learned my addr 172.21.15.87:0/2989193080 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:12.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.797+0000 7f4ed37fe700 1 -- 172.21.15.87:0/2989193080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4ed4108720 msgr2=0x7f4ed406a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:12.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.797+0000 7f4ed37fe700 1 --2- 172.21.15.87:0/2989193080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4ed4108720 0x7f4ed406a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:12.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.797+0000 7f4ed37fe700 1 -- 172.21.15.87:0/2989193080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed406b660 msgr2=0x7f4ed406a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:12.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.797+0000 7f4ed37fe700 1 --2- 172.21.15.87:0/2989193080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4ed406b660 0x7f4ed406a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:12.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.797+0000 7f4ed37fe700 1 -- 172.21.15.87:0/2989193080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc005040 con 0x7f4ed40633f0 2024-02-23T00:31:12.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.798+0000 7f4ed37fe700 1 --2- 172.21.15.87:0/2989193080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed40633f0 0x7f4ed4063870 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f4ecc00bb30 tx=0x7f4ecc0097e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:12.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.798+0000 7f4ecbfff700 10 client.?.objecter ms_handle_connect 0x7f4ed40633f0 2024-02-23T00:31:12.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.798+0000 7f4ecbfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:12.806 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.814+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc00c940 con 0x7f4ed40633f0 2024-02-23T00:31:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.814+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ecc015420 con 0x7f4ed40633f0 2024-02-23T00:31:12.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.814+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc0143d0 con 0x7f4ed40633f0 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.818+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed4067f40 con 0x7f4ed40633f0 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.818+0000 7f4ed9590700 10 client.32281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.818+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed40683b0 con 0x7f4ed40633f0 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.818+0000 7f4ed9590700 1 librados: init done 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.821+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4ed405ee80 con 0x7f4ed40633f0 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4ecc012020 con 0x7f4ed40633f0 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ec0062ce0 0x7f4ec00651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(121..121 src has 38..121) v4 ==== 11848+0+0 (secure 0 0 0) 0x7f4ecc07e110 con 0x7f4ed40633f0 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 osd_map(121..121 src has 38..121) v4 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 3 client.32281.objecter handle_osd_map got epochs [121,121] > 0 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 3 client.32281.objecter handle_osd_map decoding full epoch 121 2024-02-23T00:31:12.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 10 client.32281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:12.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=122}) v3 -- 0x7f4ec0077b50 con 0x7f4ed40633f0 2024-02-23T00:31:12.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 20 client.32281.objecter dump_active .. 0 homeless 2024-02-23T00:31:12.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.826+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v121) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4ecc085070 con 0x7f4ed40633f0 2024-02-23T00:31:12.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.827+0000 7f4ed2ffd700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ec0062ce0 0x7f4ec00651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:12.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.827+0000 7f4ecbfff700 10 client.32281.objecter ms_handle_connect 0x7f4ec0062ce0 2024-02-23T00:31:12.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.827+0000 7f4ed2ffd700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ec0062ce0 0x7f4ec00651a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f4ebc004710 tx=0x7f4ebc009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:12.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 10 client.32281.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:12.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 1 2024-02-23T00:31:12.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4ed4106710 0x7f4ed4106af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:12.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed410d780 osd=0 3 2024-02-23T00:31:12.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed410d780 osd=0 3 2024-02-23T00:31:12.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed410d780 osd=0 3 2024-02-23T00:31:12.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 1 2024-02-23T00:31:12.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed410d780 osd=0 3 2024-02-23T00:31:12.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 0 1 2024-02-23T00:31:12.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed410d780 osd=0 4 2024-02-23T00:31:12.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 10 client.32281.objecter _send_command 1 2024-02-23T00:31:12.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.829+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ed40648d0 con 0x7f4ed4106710 2024-02-23T00:31:12.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.839+0000 7f4ed27fc700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4ed4106710 0x7f4ed4106af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:12.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.839+0000 7f4ed27fc700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4ed4106710 0x7f4ed4106af0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:12.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.840+0000 7f4ecbfff700 10 client.32281.objecter ms_handle_connect 0x7f4ed4106710 2024-02-23T00:31:12.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.841+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4ed40648d0 con 0x7f4ed4106710 2024-02-23T00:31:12.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.841+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:31:12.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.841+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 1 = system:0 2024-02-23T00:31:12.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.841+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed410d780 osd=0 4 2024-02-23T00:31:12.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.841+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 0 1 2024-02-23T00:31:12.896 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:12.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:12.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.901+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 <== mon.0 v2:172.21.15.87:3300/0 7 ==== osd_map(122..122 src has 38..122) v4 ==== 2543+0+0 (secure 0 0 0) 0x7efc24050d20 con 0x7efc340f7320 2024-02-23T00:31:12.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.901+0000 7efc30ff9700 10 client.19308.objecter ms_dispatch 0x7efc34001550 osd_map(122..122 src has 38..122) v4 2024-02-23T00:31:12.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.901+0000 7efc30ff9700 3 client.19308.objecter handle_osd_map got epochs [122,122] > 121 2024-02-23T00:31:12.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.901+0000 7efc30ff9700 3 client.19308.objecter handle_osd_map decoding incremental epoch 122 2024-02-23T00:31:12.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.901+0000 7efc30ff9700 10 client.19308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:12.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.901+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=123}) v3 -- 0x7efc1c0725a0 con 0x7efc340f7320 2024-02-23T00:31:12.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.901+0000 7efc30ff9700 20 client.19308.objecter dump_active .. 0 homeless 2024-02-23T00:31:12.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.903+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== mon.1 v2:172.21.15.171:3300/0 7 ==== osd_map(122..122 src has 38..122) v4 ==== 2543+0+0 (secure 0 0 0) 0x7f4ecc015b20 con 0x7f4ed40633f0 2024-02-23T00:31:12.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.903+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 osd_map(122..122 src has 38..122) v4 2024-02-23T00:31:12.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.903+0000 7f4ecbfff700 3 client.32281.objecter handle_osd_map got epochs [122,122] > 121 2024-02-23T00:31:12.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.903+0000 7f4ecbfff700 3 client.32281.objecter handle_osd_map decoding incremental epoch 122 2024-02-23T00:31:12.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.903+0000 7f4ecbfff700 10 client.32281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:12.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.903+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=123}) v3 -- 0x7f4ec00725a0 con 0x7f4ed40633f0 2024-02-23T00:31:12.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.903+0000 7f4ecbfff700 20 client.32281.objecter dump_active .. 0 homeless 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 10 client.32281.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 2 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed410d780 osd=0 3 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed410d780 osd=0 3 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed410d780 osd=0 3 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 2 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed410d780 osd=0 3 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 0 2 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed410d780 osd=0 4 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 10 client.32281.objecter _send_command 2 2024-02-23T00:31:12.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:12.911+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4ed4106fe0 con 0x7f4ed4106710 2024-02-23T00:31:13.003 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:13.063 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:31:13.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.008+0000 7efc30ff9700 1 -- 172.21.15.87:0/1526244710 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+1327580 (secure 0 0 0) 0x7efc341073e0 con 0x7efc1c062ce0 2024-02-23T00:31:13.106 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:31:13.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.100+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4ed4106fe0 con 0x7f4ed4106710 2024-02-23T00:31:13.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.100+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.100+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed410d780 osd=0 4 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 0 2 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 10 client.32281.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 3 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6805/685354171 conn(0x7f4ed40278d0 0x7f4ed4027cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed41073f0 osd=1 3 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed41073f0 osd=1 3 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed41073f0 osd=1 3 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 3 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed41073f0 osd=1 3 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 1 3 2024-02-23T00:31:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed41073f0 osd=1 4 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 10 client.32281.objecter _send_command 3 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ed4107560 con 0x7f4ed40278d0 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.101+0000 7f4ed2ffd700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6805/685354171 conn(0x7f4ed40278d0 0x7f4ed4027cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.102+0000 7f4ed2ffd700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6805/685354171 conn(0x7f4ed40278d0 0x7f4ed4027cb0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.102+0000 7f4ecbfff700 10 client.32281.objecter ms_handle_connect 0x7f4ed40278d0 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.103+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4ed4107560 con 0x7f4ed40278d0 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.103+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.103+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 3 = system:0 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.103+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed41073f0 osd=1 4 2024-02-23T00:31:13.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.103+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 1 3 2024-02-23T00:31:13.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.112+0000 7efc39f83700 10 librados: watch_flush enter 2024-02-23T00:31:13.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.112+0000 7efc39f83700 10 librados: watch_flush exit 2024-02-23T00:31:13.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.112+0000 7efc39f83700 20 client.19308.objecter shutdown clearing up homeless session... 2024-02-23T00:31:13.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.112+0000 7efc39f83700 10 client.19308.objecter successfully canceled tick 2024-02-23T00:31:13.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.113+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc1c062ce0 msgr2=0x7efc1c0651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.113+0000 7efc39f83700 1 --2- 172.21.15.87:0/1526244710 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc1c062ce0 0x7efc1c0651a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7efc341076a0 tx=0x7efc2c009370).stop 2024-02-23T00:31:13.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.113+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 msgr2=0x7efc3410acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.113+0000 7efc39f83700 1 --2- 172.21.15.87:0/1526244710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 0x7efc3410acc0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7efc2400b530 tx=0x7efc240090f0).stop 2024-02-23T00:31:13.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.114+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 shutdown_connections 2024-02-23T00:31:13.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.114+0000 7efc39f83700 1 --2- 172.21.15.87:0/1526244710 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc1c062ce0 0x7efc1c0651a0 secure :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0x7efc341076a0 tx=0x7efc2c009370).stop 2024-02-23T00:31:13.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.114+0000 7efc39f83700 1 --2- 172.21.15.87:0/1526244710 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc340fd410 0x7efc34105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.114+0000 7efc39f83700 1 --2- 172.21.15.87:0/1526244710 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc340f7320 0x7efc3410acc0 secure :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0x7efc2400b530 tx=0x7efc240090f0).stop 2024-02-23T00:31:13.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.114+0000 7efc39f83700 1 --2- 172.21.15.87:0/1526244710 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc340fcab0 0x7efc3410b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:13.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.114+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 >> 172.21.15.87:0/1526244710 conn(0x7efc340708b0 msgr2=0x7efc3406d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:13.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 10 client.32281.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 4 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed41073f0 osd=1 3 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed41073f0 osd=1 3 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed41073f0 osd=1 3 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 4 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed41073f0 osd=1 3 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 1 4 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed41073f0 osd=1 4 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 10 client.32281.objecter _send_command 4 2024-02-23T00:31:13.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.127+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4ed40f5d60 con 0x7f4ed40278d0 2024-02-23T00:31:13.148 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:31:13.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4ed40f5d60 con 0x7f4ed40278d0 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed41073f0 osd=1 4 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 1 4 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 10 client.32281.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 5 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4ed4100420 0x7f4ed4102850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed40f6170 osd=2 3 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed40f6170 osd=2 3 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed40f6170 osd=2 3 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 5 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed40f6170 osd=2 3 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 2 5 2024-02-23T00:31:13.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed40f6170 osd=2 4 2024-02-23T00:31:13.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 10 client.32281.objecter _send_command 5 2024-02-23T00:31:13.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.146+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ed4104320 con 0x7f4ed4100420 2024-02-23T00:31:13.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.167+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 shutdown_connections 2024-02-23T00:31:13.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.167+0000 7efc39f83700 1 -- 172.21.15.87:0/1526244710 wait complete. 2024-02-23T00:31:13.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.169+0000 7efc39f83700 1 librados: shutdown 2024-02-23T00:31:13.197 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:31:13.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.197+0000 7f4ed27fc700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4ed4100420 0x7f4ed4102850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:13.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.197+0000 7f4ed27fc700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4ed4100420 0x7f4ed4102850 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:13.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.197+0000 7f4ecbfff700 10 client.32281.objecter ms_handle_connect 0x7f4ed4100420 2024-02-23T00:31:13.205 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:13.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.198+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4ed4104320 con 0x7f4ed4100420 2024-02-23T00:31:13.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.198+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:31:13.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.198+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 5 = system:0 2024-02-23T00:31:13.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.198+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed40f6170 osd=2 4 2024-02-23T00:31:13.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.198+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 2 5 2024-02-23T00:31:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 10 client.32281.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 6 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed40f6170 osd=2 3 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed40f6170 osd=2 3 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed40f6170 osd=2 3 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 6 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed40f6170 osd=2 3 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 2 6 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed40f6170 osd=2 4 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 10 client.32281.objecter _send_command 6 2024-02-23T00:31:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.224+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4ed4104540 con 0x7f4ed4100420 2024-02-23T00:31:13.273 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.263+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4ed4104540 con 0x7f4ed4100420 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.263+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.263+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.263+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed40f6170 osd=2 4 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.263+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 2 6 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 10 client.32281.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 7 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6813/128059790 conn(0x7f4ed4103ec0 0x7f4ed4137f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4028620 osd=3 3 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4028620 osd=3 3 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4028620 osd=3 3 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 7 2024-02-23T00:31:13.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed4028620 osd=3 3 2024-02-23T00:31:13.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 3 7 2024-02-23T00:31:13.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4028620 osd=3 4 2024-02-23T00:31:13.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 10 client.32281.objecter _send_command 7 2024-02-23T00:31:13.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ed4059e00 con 0x7f4ed4103ec0 2024-02-23T00:31:13.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.270+0000 7f4ed37fe700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6813/128059790 conn(0x7f4ed4103ec0 0x7f4ed4137f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:13.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.271+0000 7f4ed37fe700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6813/128059790 conn(0x7f4ed4103ec0 0x7f4ed4137f20 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:13.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.271+0000 7f4ecbfff700 10 client.32281.objecter ms_handle_connect 0x7f4ed4103ec0 2024-02-23T00:31:13.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.275+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4ed4059e00 con 0x7f4ed4103ec0 2024-02-23T00:31:13.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.275+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:31:13.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.275+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 7 = system:0 2024-02-23T00:31:13.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.275+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed4028620 osd=3 4 2024-02-23T00:31:13.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.275+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 3 7 2024-02-23T00:31:13.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 10 client.32281.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:13.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 8 2024-02-23T00:31:13.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4028620 osd=3 3 2024-02-23T00:31:13.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4028620 osd=3 3 2024-02-23T00:31:13.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4028620 osd=3 3 2024-02-23T00:31:13.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 8 2024-02-23T00:31:13.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed4028620 osd=3 3 2024-02-23T00:31:13.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 3 8 2024-02-23T00:31:13.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4028620 osd=3 4 2024-02-23T00:31:13.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 10 client.32281.objecter _send_command 8 2024-02-23T00:31:13.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.306+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4ed4138410 con 0x7f4ed4103ec0 2024-02-23T00:31:13.312 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:31:13.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4ed4138410 con 0x7f4ed4103ec0 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed4028620 osd=3 4 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 3 8 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 10 client.32281.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 9 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4ed4138af0 0x7f4ed413aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4138410 osd=4 3 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4138410 osd=4 3 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4138410 osd=4 3 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 9 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed4138410 osd=4 3 2024-02-23T00:31:13.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 4 9 2024-02-23T00:31:13.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4138410 osd=4 4 2024-02-23T00:31:13.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 10 client.32281.objecter _send_command 9 2024-02-23T00:31:13.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.309+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ed413dbc0 con 0x7f4ed4138af0 2024-02-23T00:31:13.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.313+0000 7f4ed2ffd700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4ed4138af0 0x7f4ed413aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:13.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.313+0000 7f4ed2ffd700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4ed4138af0 0x7f4ed413aee0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:13.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.318+0000 7f4ecbfff700 10 client.32281.objecter ms_handle_connect 0x7f4ed4138af0 2024-02-23T00:31:13.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.318+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4ed413dbc0 con 0x7f4ed4138af0 2024-02-23T00:31:13.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.318+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:31:13.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.318+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 9 = system:0 2024-02-23T00:31:13.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.318+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed4138410 osd=4 4 2024-02-23T00:31:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.318+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 4 9 2024-02-23T00:31:13.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 10 client.32281.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:13.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 10 2024-02-23T00:31:13.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4138410 osd=4 3 2024-02-23T00:31:13.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4138410 osd=4 3 2024-02-23T00:31:13.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4138410 osd=4 3 2024-02-23T00:31:13.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 10 2024-02-23T00:31:13.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed4138410 osd=4 3 2024-02-23T00:31:13.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 4 10 2024-02-23T00:31:13.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4138410 osd=4 4 2024-02-23T00:31:13.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 10 client.32281.objecter _send_command 10 2024-02-23T00:31:13.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.348+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4ed413dbc0 con 0x7f4ed4138af0 2024-02-23T00:31:13.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.372+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4ed413dbc0 con 0x7f4ed4138af0 2024-02-23T00:31:13.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.372+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:13.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.372+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.372+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed4138410 osd=4 4 2024-02-23T00:31:13.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.372+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 4 10 2024-02-23T00:31:13.389 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 10 client.32281.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 11 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4ed413cf20 0x7f4ed413d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed413dbc0 osd=5 3 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413dbc0 osd=5 3 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed413dbc0 osd=5 3 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 11 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed413dbc0 osd=5 3 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 5 11 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413dbc0 osd=5 4 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 10 client.32281.objecter _send_command 11 2024-02-23T00:31:13.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.387+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ed413da20 con 0x7f4ed413cf20 2024-02-23T00:31:13.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.388+0000 7f4ed27fc700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4ed413cf20 0x7f4ed413d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:13.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.398+0000 7f4ed27fc700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4ed413cf20 0x7f4ed413d380 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:13.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.398+0000 7f4ecbfff700 10 client.32281.objecter ms_handle_connect 0x7f4ed413cf20 2024-02-23T00:31:13.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.399+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4ed413da20 con 0x7f4ed413cf20 2024-02-23T00:31:13.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.399+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:31:13.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.399+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 11 = system:0 2024-02-23T00:31:13.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.399+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed413dbc0 osd=5 4 2024-02-23T00:31:13.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.399+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 5 11 2024-02-23T00:31:13.406 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:31:13.406 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:31:13.406 INFO:teuthology.orchestra.run.smithi087.stdout: "complaint_time": 30, 2024-02-23T00:31:13.406 INFO:teuthology.orchestra.run.smithi087.stdout: "num_blocked_ops": 0 2024-02-23T00:31:13.406 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:31:13.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 10 client.32281.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:13.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 12 2024-02-23T00:31:13.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed413dbc0 osd=5 3 2024-02-23T00:31:13.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413dbc0 osd=5 3 2024-02-23T00:31:13.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed413dbc0 osd=5 3 2024-02-23T00:31:13.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 12 2024-02-23T00:31:13.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed413dbc0 osd=5 3 2024-02-23T00:31:13.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 5 12 2024-02-23T00:31:13.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413dbc0 osd=5 4 2024-02-23T00:31:13.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 10 client.32281.objecter _send_command 12 2024-02-23T00:31:13.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.423+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4ed413da20 con 0x7f4ed413cf20 2024-02-23T00:31:13.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.427+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4ed413da20 con 0x7f4ed413cf20 2024-02-23T00:31:13.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.427+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:13.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.427+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.427+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed413dbc0 osd=5 4 2024-02-23T00:31:13.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.427+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 5 12 2024-02-23T00:31:13.446 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:31:13.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 10 client.32281.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:13.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 13 2024-02-23T00:31:13.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4ed4140300 0x7f4ed4140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:13.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed413da20 osd=6 3 2024-02-23T00:31:13.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413da20 osd=6 3 2024-02-23T00:31:13.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed413da20 osd=6 3 2024-02-23T00:31:13.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 13 2024-02-23T00:31:13.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed413da20 osd=6 3 2024-02-23T00:31:13.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 6 13 2024-02-23T00:31:13.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413da20 osd=6 4 2024-02-23T00:31:13.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 10 client.32281.objecter _send_command 13 2024-02-23T00:31:13.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ed4140e00 con 0x7f4ed4140300 2024-02-23T00:31:13.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.440+0000 7f4ed37fe700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4ed4140300 0x7f4ed4140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:13.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.453+0000 7f4ed37fe700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4ed4140300 0x7f4ed4140760 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:13.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.453+0000 7f4ecbfff700 10 client.32281.objecter ms_handle_connect 0x7f4ed4140300 2024-02-23T00:31:13.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.454+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4ed4140e00 con 0x7f4ed4140300 2024-02-23T00:31:13.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.454+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:31:13.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.454+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 13 = system:0 2024-02-23T00:31:13.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.454+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed413da20 osd=6 4 2024-02-23T00:31:13.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.454+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 6 13 2024-02-23T00:31:13.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 10 client.32281.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:13.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 14 2024-02-23T00:31:13.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed413da20 osd=6 3 2024-02-23T00:31:13.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413da20 osd=6 3 2024-02-23T00:31:13.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed413da20 osd=6 3 2024-02-23T00:31:13.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 14 2024-02-23T00:31:13.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed413da20 osd=6 3 2024-02-23T00:31:13.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 6 14 2024-02-23T00:31:13.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413da20 osd=6 4 2024-02-23T00:31:13.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 10 client.32281.objecter _send_command 14 2024-02-23T00:31:13.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.485+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4ed4140e00 con 0x7f4ed4140300 2024-02-23T00:31:13.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.492+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4ed4140e00 con 0x7f4ed4140300 2024-02-23T00:31:13.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.492+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:13.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.492+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.492+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed413da20 osd=6 4 2024-02-23T00:31:13.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.492+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 6 14 2024-02-23T00:31:13.507 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 10 client.32281.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 15 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6812/373787000 conn(0x7f4ed4143540 0x7f4ed41439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4140e00 osd=7 3 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4140e00 osd=7 3 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4140e00 osd=7 3 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 15 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed4140e00 osd=7 3 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 7 15 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4140e00 osd=7 4 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 10 client.32281.objecter _send_command 15 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.505+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ed4143fc0 con 0x7f4ed4143540 2024-02-23T00:31:13.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.506+0000 7f4ed2ffd700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6812/373787000 conn(0x7f4ed4143540 0x7f4ed41439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:13.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.521+0000 7f4ed2ffd700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6812/373787000 conn(0x7f4ed4143540 0x7f4ed41439a0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:13.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.521+0000 7f4ecbfff700 10 client.32281.objecter ms_handle_connect 0x7f4ed4143540 2024-02-23T00:31:13.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.526+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4ed4143fc0 con 0x7f4ed4143540 2024-02-23T00:31:13.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.526+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:31:13.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.527+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 15 = system:0 2024-02-23T00:31:13.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.527+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed4140e00 osd=7 4 2024-02-23T00:31:13.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.527+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 7 15 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 10 client.32281.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign -1 16 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4140e00 osd=7 3 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4140e00 osd=7 3 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 20 client.32281.objecter _get_session s=0x7f4ed4140e00 osd=7 3 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_remove -1 16 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 20 client.32281.objecter get_session s=0x7f4ed4140e00 osd=7 3 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 15 client.32281.objecter _session_command_op_assign 7 16 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4140e00 osd=7 4 2024-02-23T00:31:13.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 10 client.32281.objecter _send_command 16 2024-02-23T00:31:13.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.586+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4ed4143fc0 con 0x7f4ed4143540 2024-02-23T00:31:13.598 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:31:13.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.595+0000 7f4ecbfff700 1 -- 172.21.15.87:0/2989193080 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4ed4143fc0 con 0x7f4ed4143540 2024-02-23T00:31:13.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.595+0000 7f4ecbfff700 10 client.32281.objecter ms_dispatch 0x7f4ed4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:13.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.595+0000 7f4ecbfff700 10 client.32281.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.595+0000 7f4ecbfff700 20 client.32281.objecter put_session s=0x7f4ed4140e00 osd=7 4 2024-02-23T00:31:13.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.595+0000 7f4ecbfff700 15 client.32281.objecter _session_command_op_remove 7 16 2024-02-23T00:31:13.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.595+0000 7f4ed9590700 10 librados: watch_flush enter 2024-02-23T00:31:13.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 librados: watch_flush exit 2024-02-23T00:31:13.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 client.32281.objecter close_session for osd.0 2024-02-23T00:31:13.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4ed4106710 msgr2=0x7f4ed4106af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4ed4106710 0x7f4ed4106af0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed410d780 osd=0 1 2024-02-23T00:31:13.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 client.32281.objecter close_session for osd.1 2024-02-23T00:31:13.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6805/685354171 conn(0x7f4ed40278d0 msgr2=0x7f4ed4027cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6805/685354171 conn(0x7f4ed40278d0 0x7f4ed4027cb0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed41073f0 osd=1 1 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 client.32281.objecter close_session for osd.2 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4ed4100420 msgr2=0x7f4ed4102850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4ed4100420 0x7f4ed4102850 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed40f6170 osd=2 1 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 client.32281.objecter close_session for osd.3 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6813/128059790 conn(0x7f4ed4103ec0 msgr2=0x7f4ed4137f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6813/128059790 conn(0x7f4ed4103ec0 0x7f4ed4137f20 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4028620 osd=3 1 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 client.32281.objecter close_session for osd.4 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4ed4138af0 msgr2=0x7f4ed413aee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4ed4138af0 0x7f4ed413aee0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4138410 osd=4 1 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 client.32281.objecter close_session for osd.5 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4ed413cf20 msgr2=0x7f4ed413d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4ed413cf20 0x7f4ed413d380 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413dbc0 osd=5 1 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 client.32281.objecter close_session for osd.6 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4ed4140300 msgr2=0x7f4ed4140760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4ed4140300 0x7f4ed4140760 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed413da20 osd=6 1 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 client.32281.objecter close_session for osd.7 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6812/373787000 conn(0x7f4ed4143540 msgr2=0x7f4ed41439a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.171:6812/373787000 conn(0x7f4ed4143540 0x7f4ed41439a0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 20 client.32281.objecter put_session s=0x7f4ed4140e00 osd=7 1 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 20 client.32281.objecter shutdown clearing up homeless session... 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed9590700 10 client.32281.objecter successfully canceled tick 2024-02-23T00:31:13.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed37fe700 1 -- 172.21.15.87:0/2989193080 reap_dead start 2024-02-23T00:31:13.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed37fe700 1 -- 172.21.15.87:0/2989193080 reap_dead start 2024-02-23T00:31:13.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed37fe700 1 -- 172.21.15.87:0/2989193080 reap_dead start 2024-02-23T00:31:13.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed37fe700 1 -- 172.21.15.87:0/2989193080 reap_dead start 2024-02-23T00:31:13.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.597+0000 7f4ed37fe700 1 -- 172.21.15.87:0/2989193080 reap_dead start 2024-02-23T00:31:13.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.599+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ec0062ce0 msgr2=0x7f4ec00651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.599+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ec0062ce0 0x7f4ec00651a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f4ebc004710 tx=0x7f4ebc009cb0).stop 2024-02-23T00:31:13.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.599+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed40633f0 msgr2=0x7f4ed4063870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:13.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.599+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed40633f0 0x7f4ed4063870 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f4ecc00bb30 tx=0x7f4ecc0097e0).stop 2024-02-23T00:31:13.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.601+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 shutdown_connections 2024-02-23T00:31:13.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.601+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4ed40633f0 0x7f4ed4063870 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.601+0000 7f4ed9590700 1 --2- 172.21.15.87:0/2989193080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ec0062ce0 0x7f4ec00651a0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:13.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.601+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 >> 172.21.15.87:0/2989193080 conn(0x7f4ed40f3800 msgr2=0x7f4ed40f4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:13.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.604+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 shutdown_connections 2024-02-23T00:31:13.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.605+0000 7f4ed9590700 1 -- 172.21.15.87:0/2989193080 wait complete. 2024-02-23T00:31:13.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:13.606+0000 7f4ed9590700 1 librados: shutdown 2024-02-23T00:31:13.617 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.617 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.617 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.618 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.618 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.618 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.618 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.618 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:11.261804+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:11.261858+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:11.892785+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:11.892841+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.389805+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi087-82011-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.487820+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.573035+0000 mon.a (mon.0) 1042 : audit [DBG] from='client.? 172.21.15.87:0/1379110377' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.577639+0000 mgr.y (mgr.24437) 171 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.797573+0000 mgr.y (mgr.24437) 172 : audit [DBG] from='client.19308 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.827908+0000 mon.b (mon.1) 110 : audit [DBG] from='client.? 172.21.15.87:0/2989193080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:12.860483+0000 mon.a (mon.0) 1043 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-23T00:31:13.852 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.905765+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.905912+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi087-82011-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:12.906022+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.927482+0000 mon.b (mon.1) 111 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:12.938407+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:11.261804+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:11.261858+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:11.892785+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:11.892841+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.389805+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi087-82011-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.487820+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.573035+0000 mon.a (mon.0) 1042 : audit [DBG] from='client.? 172.21.15.87:0/1379110377' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-23T00:31:13.853 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.577639+0000 mgr.y (mgr.24437) 171 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-23T00:31:13.854 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.797573+0000 mgr.y (mgr.24437) 172 : audit [DBG] from='client.19308 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:31:13.854 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.827908+0000 mon.b (mon.1) 110 : audit [DBG] from='client.? 172.21.15.87:0/2989193080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:13.854 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:12.860483+0000 mon.a (mon.0) 1043 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-23T00:31:13.854 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.905765+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:31:13.854 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.905912+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi087-82011-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:13.854 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:12.906022+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-23T00:31:13.854 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.927482+0000 mon.b (mon.1) 111 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:13.854 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:12.938407+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:13.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:31:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:11.261804+0000 osd.5 (osd.5) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:11.261858+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:11.892785+0000 osd.4 (osd.4) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:11.892841+0000 osd.4 (osd.4) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.389805+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi087-82011-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.487820+0000 mon.c (mon.2) 79 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.573035+0000 mon.a (mon.0) 1042 : audit [DBG] from='client.? 172.21.15.87:0/1379110377' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.577639+0000 mgr.y (mgr.24437) 171 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.797573+0000 mgr.y (mgr.24437) 172 : audit [DBG] from='client.19308 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.827908+0000 mon.b (mon.1) 110 : audit [DBG] from='client.? 172.21.15.87:0/2989193080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:12.860483+0000 mon.a (mon.0) 1043 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.905765+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.905912+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi087-82011-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:12.906022+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.927482+0000 mon.b (mon.1) 111 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:12.938407+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7"}]: dispatch 2024-02-23T00:31:14.553 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:14.556 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-23T00:31:14.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:14.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:14.664 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:14.765 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:14.949 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:14.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:15.049 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:15.130 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:13.490618+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:15.130 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:13.530452+0000 mgr.y (mgr.24437) 173 : cluster [DBG] pgmap v166: 552 pgs: 96 unknown, 456 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:31:15.130 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:13.968830+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7"}]': finished 2024-02-23T00:31:15.130 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:13.968962+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-23T00:31:15.130 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:14.512251+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:15.130 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:13.490618+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:15.131 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:13.530452+0000 mgr.y (mgr.24437) 173 : cluster [DBG] pgmap v166: 552 pgs: 96 unknown, 456 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:31:15.131 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:13.968830+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7"}]': finished 2024-02-23T00:31:15.131 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:13.968962+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-23T00:31:15.131 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:14.512251+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:15.131 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:15.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:13.490618+0000 mon.c (mon.2) 80 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:15.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:13.530452+0000 mgr.y (mgr.24437) 173 : cluster [DBG] pgmap v166: 552 pgs: 96 unknown, 456 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:31:15.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:13.968830+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-7"}]': finished 2024-02-23T00:31:15.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:13.968962+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-23T00:31:15.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:14.512251+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:15.229 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:15.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:15.352 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:15.436 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:15.803 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:15.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:15.909 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:15.935 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:15.954 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:15.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:16.058 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:16.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:14.953936+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-23T00:31:16.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:14.993500+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.87:0/3809788133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:14.999058+0000 mon.b (mon.1) 113 : audit [INF] from='client.? 172.21.15.87:0/81502563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi087-82011-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.065587+0000 mon.b (mon.1) 114 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.480172+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.480731+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi087-82011-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.487327+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.494774+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.515706+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.856820+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.857725+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:16.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.865607+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:16.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:14.953936+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-23T00:31:16.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:14.993500+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.87:0/3809788133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:14.999058+0000 mon.b (mon.1) 113 : audit [INF] from='client.? 172.21.15.87:0/81502563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi087-82011-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.065587+0000 mon.b (mon.1) 114 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.480172+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.480731+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi087-82011-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.487327+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.494774+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.515706+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:16.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.856820+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.857725+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:16.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.865607+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:16.088 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:16.109 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:16.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:16.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:14.953936+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-23T00:31:16.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:14.993500+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.87:0/3809788133' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:14.999058+0000 mon.b (mon.1) 113 : audit [INF] from='client.? 172.21.15.87:0/81502563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi087-82011-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.065587+0000 mon.b (mon.1) 114 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.480172+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.480731+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi087-82011-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:16.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.487327+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.494774+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.515706+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:16.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.856820+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:16.195 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.857725+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:16.195 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.865607+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:16.213 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:16.240 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:16.259 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:16.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:16.363 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:16.541 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:16.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:16.760 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:16.786 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:16.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:16.810 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:16.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:16.812 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-02-23T00:31:11.848126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151054463,"num_objects":301,"num_object_clones":0,"num_object_copies":903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":301,"num_whiteouts":0,"num_read":2362,"num_read_kb":50221,"num_write":1806,"num_write_kb":153152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1952,"ondisk_log_size":1952,"up":1464,"acting":1464,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1494,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2908736,"kb_used_data":454144,"kb_used_omap":0,"kb_used_meta":2454528,"kb_avail":747052480,"statfs":{"total":767960285184,"available":764981739520,"internally_reserved":0,"allocated":465043456,"data_stored":459184871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2513436672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":54,"apply_latency_ms":54,"commit_latency_ns":54000000,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8585,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":10,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.939681"},"pg_stats":[{"pgid":"72.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.1c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.221028+0000","last_change":"2024-02-23T00:31:09.220627+0000","last_active":"2024-02-23T00:31:09.221028+0000","last_peered":"2024-02-23T00:31:09.221028+0000","last_clean":"2024-02-23T00:31:09.221028+0000","last_became_active":"2024-02-23T00:31:09.220372+0000","last_became_peered":"2024-02-23T00:31:09.220372+0000","last_unstale":"2024-02-23T00:31:09.221028+0000","last_undegraded":"2024-02-23T00:31:09.221028+0000","last_fullsized":"2024-02-23T00:31:09.221028+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.1d","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.155839+0000","last_change":"2024-02-23T00:31:08.014165+0000","last_active":"2024-02-23T00:31:09.155839+0000","last_peered":"2024-02-23T00:31:09.155839+0000","last_clean":"2024-02-23T00:31:09.155839+0000","last_became_active":"2024-02-23T00:31:08.013887+0000","last_became_peered":"2024-02-23T00:31:08.013887+0000","last_unstale":"2024-02-23T00:31:09.155839+0000","last_undegraded":"2024-02-23T00:31:09.155839+0000","last_fullsized":"2024-02-23T00:31:09.155839+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.1d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.186731+0000","last_change":"2024-02-23T00:31:09.186252+0000","last_active":"2024-02-23T00:31:09.186731+0000","last_peered":"2024-02-23T00:31:09.186731+0000","last_clean":"2024-02-23T00:31:09.186731+0000","last_became_active":"2024-02-23T00:31:09.185962+0000","last_became_peered":"2024-02-23T00:31:09.185962+0000","last_unstale":"2024-02-23T00:31:09.186731+0000","last_undegraded":"2024-02-23T00:31:09.186731+0000","last_fullsized":"2024-02-23T00:31:09.186731+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1c","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174570+0000","last_change":"2024-02-23T00:31:08.068372+0000","last_active":"2024-02-23T00:31:09.174570+0000","last_peered":"2024-02-23T00:31:09.174570+0000","last_clean":"2024-02-23T00:31:09.174570+0000","last_became_active":"2024-02-23T00:31:08.068073+0000","last_became_peered":"2024-02-23T00:31:08.068073+0000","last_unstale":"2024-02-23T00:31:09.174570+0000","last_undegraded":"2024-02-23T00:31:09.174570+0000","last_fullsized":"2024-02-23T00:31:09.174570+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.1e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.243787+0000","last_change":"2024-02-23T00:31:09.241751+0000","last_active":"2024-02-23T00:31:09.243787+0000","last_peered":"2024-02-23T00:31:09.243787+0000","last_clean":"2024-02-23T00:31:09.243787+0000","last_became_active":"2024-02-23T00:31:09.241154+0000","last_became_peered":"2024-02-23T00:31:09.241154+0000","last_unstale":"2024-02-23T00:31:09.243787+0000","last_undegraded":"2024-02-23T00:31:09.243787+0000","last_fullsized":"2024-02-23T00:31:09.243787+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1f","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.119559+0000","last_change":"2024-02-23T00:31:08.039893+0000","last_active":"2024-02-23T00:31:09.119559+0000","last_peered":"2024-02-23T00:31:09.119559+0000","last_clean":"2024-02-23T00:31:09.119559+0000","last_became_active":"2024-02-23T00:31:08.039581+0000","last_became_peered":"2024-02-23T00:31:08.039581+0000","last_unstale":"2024-02-23T00:31:09.119559+0000","last_undegraded":"2024-02-23T00:31:09.119559+0000","last_fullsized":"2024-02-23T00:31:09.119559+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"72.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.1f","version":"119'1","reported_seq":11,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.217866+0000","last_change":"2024-02-23T00:31:09.237865+0000","last_active":"2024-02-23T00:31:10.217866+0000","last_peered":"2024-02-23T00:31:10.217866+0000","last_clean":"2024-02-23T00:31:10.217866+0000","last_became_active":"2024-02-23T00:31:09.237584+0000","last_became_peered":"2024-02-23T00:31:09.237584+0000","last_unstale":"2024-02-23T00:31:10.217866+0000","last_undegraded":"2024-02-23T00:31:10.217866+0000","last_fullsized":"2024-02-23T00:31:10.217866+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.1e","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128428+0000","last_change":"2024-02-23T00:31:08.044606+0000","last_active":"2024-02-23T00:31:09.128428+0000","last_peered":"2024-02-23T00:31:09.128428+0000","last_clean":"2024-02-23T00:31:09.128428+0000","last_became_active":"2024-02-23T00:31:08.044292+0000","last_became_peered":"2024-02-23T00:31:08.044292+0000","last_unstale":"2024-02-23T00:31:09.128428+0000","last_undegraded":"2024-02-23T00:31:09.128428+0000","last_fullsized":"2024-02-23T00:31:09.128428+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.18","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.139011+0000","last_change":"2024-02-23T00:31:09.138421+0000","last_active":"2024-02-23T00:31:09.139011+0000","last_peered":"2024-02-23T00:31:09.139011+0000","last_clean":"2024-02-23T00:31:09.139011+0000","last_became_active":"2024-02-23T00:31:09.138161+0000","last_became_peered":"2024-02-23T00:31:09.138161+0000","last_unstale":"2024-02-23T00:31:09.139011+0000","last_undegraded":"2024-02-23T00:31:09.139011+0000","last_fullsized":"2024-02-23T00:31:09.139011+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.19","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122100+0000","last_change":"2024-02-23T00:31:07.966898+0000","last_active":"2024-02-23T00:31:09.122100+0000","last_peered":"2024-02-23T00:31:09.122100+0000","last_clean":"2024-02-23T00:31:09.122100+0000","last_became_active":"2024-02-23T00:31:07.966439+0000","last_became_peered":"2024-02-23T00:31:07.966439+0000","last_unstale":"2024-02-23T00:31:09.122100+0000","last_undegraded":"2024-02-23T00:31:09.122100+0000","last_fullsized":"2024-02-23T00:31:09.122100+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"72.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.19","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.186753+0000","last_change":"2024-02-23T00:31:09.186383+0000","last_active":"2024-02-23T00:31:09.186753+0000","last_peered":"2024-02-23T00:31:09.186753+0000","last_clean":"2024-02-23T00:31:09.186753+0000","last_became_active":"2024-02-23T00:31:09.186101+0000","last_became_peered":"2024-02-23T00:31:09.186101+0000","last_unstale":"2024-02-23T00:31:09.186753+0000","last_undegraded":"2024-02-23T00:31:09.186753+0000","last_fullsized":"2024-02-23T00:31:09.186753+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.18","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.148262+0000","last_change":"2024-02-23T00:31:07.968832+0000","last_active":"2024-02-23T00:31:09.148262+0000","last_peered":"2024-02-23T00:31:09.148262+0000","last_clean":"2024-02-23T00:31:09.148262+0000","last_became_active":"2024-02-23T00:31:07.968561+0000","last_became_peered":"2024-02-23T00:31:07.968561+0000","last_unstale":"2024-02-23T00:31:09.148262+0000","last_undegraded":"2024-02-23T00:31:09.148262+0000","last_fullsized":"2024-02-23T00:31:09.148262+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.1a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.243688+0000","last_change":"2024-02-23T00:31:09.240372+0000","last_active":"2024-02-23T00:31:09.243688+0000","last_peered":"2024-02-23T00:31:09.243688+0000","last_clean":"2024-02-23T00:31:09.243688+0000","last_became_active":"2024-02-23T00:31:09.240090+0000","last_became_peered":"2024-02-23T00:31:09.240090+0000","last_unstale":"2024-02-23T00:31:09.243688+0000","last_undegraded":"2024-02-23T00:31:09.243688+0000","last_fullsized":"2024-02-23T00:31:09.243688+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.1b","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.150333+0000","last_change":"2024-02-23T00:31:08.026257+0000","last_active":"2024-02-23T00:31:09.150333+0000","last_peered":"2024-02-23T00:31:09.150333+0000","last_clean":"2024-02-23T00:31:09.150333+0000","last_became_active":"2024-02-23T00:31:08.025981+0000","last_became_peered":"2024-02-23T00:31:08.025981+0000","last_unstale":"2024-02-23T00:31:09.150333+0000","last_undegraded":"2024-02-23T00:31:09.150333+0000","last_fullsized":"2024-02-23T00:31:09.150333+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.1b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.177996+0000","last_change":"2024-02-23T00:31:09.177794+0000","last_active":"2024-02-23T00:31:09.177996+0000","last_peered":"2024-02-23T00:31:09.177996+0000","last_clean":"2024-02-23T00:31:09.177996+0000","last_became_active":"2024-02-23T00:31:09.177521+0000","last_became_peered":"2024-02-23T00:31:09.177521+0000","last_unstale":"2024-02-23T00:31:09.177996+0000","last_undegraded":"2024-02-23T00:31:09.177996+0000","last_fullsized":"2024-02-23T00:31:09.177996+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.1a","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.124742+0000","last_change":"2024-02-23T00:31:08.034014+0000","last_active":"2024-02-23T00:31:09.124742+0000","last_peered":"2024-02-23T00:31:09.124742+0000","last_clean":"2024-02-23T00:31:09.124742+0000","last_became_active":"2024-02-23T00:31:08.033702+0000","last_became_peered":"2024-02-23T00:31:08.033702+0000","last_unstale":"2024-02-23T00:31:09.124742+0000","last_undegraded":"2024-02-23T00:31:09.124742+0000","last_fullsized":"2024-02-23T00:31:09.124742+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"72.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.14","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.225697+0000","last_change":"2024-02-23T00:31:09.225179+0000","last_active":"2024-02-23T00:31:09.225697+0000","last_peered":"2024-02-23T00:31:09.225697+0000","last_clean":"2024-02-23T00:31:09.225697+0000","last_became_active":"2024-02-23T00:31:09.225015+0000","last_became_peered":"2024-02-23T00:31:09.225015+0000","last_unstale":"2024-02-23T00:31:09.225697+0000","last_undegraded":"2024-02-23T00:31:09.225697+0000","last_fullsized":"2024-02-23T00:31:09.225697+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.15","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138583+0000","last_change":"2024-02-23T00:31:08.034672+0000","last_active":"2024-02-23T00:31:09.138583+0000","last_peered":"2024-02-23T00:31:09.138583+0000","last_clean":"2024-02-23T00:31:09.138583+0000","last_became_active":"2024-02-23T00:31:08.034340+0000","last_became_peered":"2024-02-23T00:31:08.034340+0000","last_unstale":"2024-02-23T00:31:09.138583+0000","last_undegraded":"2024-02-23T00:31:09.138583+0000","last_fullsized":"2024-02-23T00:31:09.138583+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"72.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.15","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.224161+0000","last_change":"2024-02-23T00:31:09.221593+0000","last_active":"2024-02-23T00:31:09.224161+0000","last_peered":"2024-02-23T00:31:09.224161+0000","last_clean":"2024-02-23T00:31:09.224161+0000","last_became_active":"2024-02-23T00:31:09.221263+0000","last_became_peered":"2024-02-23T00:31:09.221263+0000","last_unstale":"2024-02-23T00:31:09.224161+0000","last_undegraded":"2024-02-23T00:31:09.224161+0000","last_fullsized":"2024-02-23T00:31:09.224161+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.14","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121879+0000","last_change":"2024-02-23T00:31:07.989330+0000","last_active":"2024-02-23T00:31:09.121879+0000","last_peered":"2024-02-23T00:31:09.121879+0000","last_clean":"2024-02-23T00:31:09.121879+0000","last_became_active":"2024-02-23T00:31:07.989048+0000","last_became_peered":"2024-02-23T00:31:07.989048+0000","last_unstale":"2024-02-23T00:31:09.121879+0000","last_undegraded":"2024-02-23T00:31:09.121879+0000","last_fullsized":"2024-02-23T00:31:09.121879+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"72.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.16","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.140081+0000","last_change":"2024-02-23T00:31:09.139701+0000","last_active":"2024-02-23T00:31:09.140081+0000","last_peered":"2024-02-23T00:31:09.140081+0000","last_clean":"2024-02-23T00:31:09.140081+0000","last_became_active":"2024-02-23T00:31:09.139421+0000","last_became_peered":"2024-02-23T00:31:09.139421+0000","last_unstale":"2024-02-23T00:31:09.140081+0000","last_undegraded":"2024-02-23T00:31:09.140081+0000","last_fullsized":"2024-02-23T00:31:09.140081+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.17","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.227805+0000","last_change":"2024-02-23T00:31:07.965578+0000","last_active":"2024-02-23T00:31:09.227805+0000","last_peered":"2024-02-23T00:31:09.227805+0000","last_clean":"2024-02-23T00:31:09.227805+0000","last_became_active":"2024-02-23T00:31:07.965306+0000","last_became_peered":"2024-02-23T00:31:07.965306+0000","last_unstale":"2024-02-23T00:31:09.227805+0000","last_undegraded":"2024-02-23T00:31:09.227805+0000","last_fullsized":"2024-02-23T00:31:09.227805+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.17","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.243580+0000","last_change":"2024-02-23T00:31:09.234637+0000","last_active":"2024-02-23T00:31:09.243580+0000","last_peered":"2024-02-23T00:31:09.243580+0000","last_clean":"2024-02-23T00:31:09.243580+0000","last_became_active":"2024-02-23T00:31:09.234324+0000","last_became_peered":"2024-02-23T00:31:09.234324+0000","last_unstale":"2024-02-23T00:31:09.243580+0000","last_undegraded":"2024-02-23T00:31:09.243580+0000","last_fullsized":"2024-02-23T00:31:09.243580+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.16","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128989+0000","last_change":"2024-02-23T00:31:08.057505+0000","last_active":"2024-02-23T00:31:09.128989+0000","last_peered":"2024-02-23T00:31:09.128989+0000","last_clean":"2024-02-23T00:31:09.128989+0000","last_became_active":"2024-02-23T00:31:08.057285+0000","last_became_peered":"2024-02-23T00:31:08.057285+0000","last_unstale":"2024-02-23T00:31:09.128989+0000","last_undegraded":"2024-02-23T00:31:09.128989+0000","last_fullsized":"2024-02-23T00:31:09.128989+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.10","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.240414+0000","last_change":"2024-02-23T00:31:09.238999+0000","last_active":"2024-02-23T00:31:09.240414+0000","last_peered":"2024-02-23T00:31:09.240414+0000","last_clean":"2024-02-23T00:31:09.240414+0000","last_became_active":"2024-02-23T00:31:09.238717+0000","last_became_peered":"2024-02-23T00:31:09.238717+0000","last_unstale":"2024-02-23T00:31:09.240414+0000","last_undegraded":"2024-02-23T00:31:09.240414+0000","last_fullsized":"2024-02-23T00:31:09.240414+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.11","version":"117'1","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.162058+0000","last_change":"2024-02-23T00:31:08.070174+0000","last_active":"2024-02-23T00:31:09.162058+0000","last_peered":"2024-02-23T00:31:09.162058+0000","last_clean":"2024-02-23T00:31:09.162058+0000","last_became_active":"2024-02-23T00:31:08.069324+0000","last_became_peered":"2024-02-23T00:31:08.069324+0000","last_unstale":"2024-02-23T00:31:09.162058+0000","last_undegraded":"2024-02-23T00:31:09.162058+0000","last_fullsized":"2024-02-23T00:31:09.162058+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"72.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.11","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.226484+0000","last_change":"2024-02-23T00:31:09.207760+0000","last_active":"2024-02-23T00:31:09.226484+0000","last_peered":"2024-02-23T00:31:09.226484+0000","last_clean":"2024-02-23T00:31:09.226484+0000","last_became_active":"2024-02-23T00:31:09.207089+0000","last_became_peered":"2024-02-23T00:31:09.207089+0000","last_unstale":"2024-02-23T00:31:09.226484+0000","last_undegraded":"2024-02-23T00:31:09.226484+0000","last_fullsized":"2024-02-23T00:31:09.226484+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.10","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.119410+0000","last_change":"2024-02-23T00:31:07.985975+0000","last_active":"2024-02-23T00:31:09.119410+0000","last_peered":"2024-02-23T00:31:09.119410+0000","last_clean":"2024-02-23T00:31:09.119410+0000","last_became_active":"2024-02-23T00:31:07.985694+0000","last_became_peered":"2024-02-23T00:31:07.985694+0000","last_unstale":"2024-02-23T00:31:09.119410+0000","last_undegraded":"2024-02-23T00:31:09.119410+0000","last_fullsized":"2024-02-23T00:31:09.119410+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"72.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.12","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.175687+0000","last_change":"2024-02-23T00:31:09.175411+0000","last_active":"2024-02-23T00:31:09.175687+0000","last_peered":"2024-02-23T00:31:09.175687+0000","last_clean":"2024-02-23T00:31:09.175687+0000","last_became_active":"2024-02-23T00:31:09.175111+0000","last_became_peered":"2024-02-23T00:31:09.175111+0000","last_unstale":"2024-02-23T00:31:09.175687+0000","last_undegraded":"2024-02-23T00:31:09.175687+0000","last_fullsized":"2024-02-23T00:31:09.175687+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.13","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120901+0000","last_change":"2024-02-23T00:31:08.036739+0000","last_active":"2024-02-23T00:31:09.120901+0000","last_peered":"2024-02-23T00:31:09.120901+0000","last_clean":"2024-02-23T00:31:09.120901+0000","last_became_active":"2024-02-23T00:31:08.036425+0000","last_became_peered":"2024-02-23T00:31:08.036425+0000","last_unstale":"2024-02-23T00:31:09.120901+0000","last_undegraded":"2024-02-23T00:31:09.120901+0000","last_fullsized":"2024-02-23T00:31:09.120901+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"72.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.13","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.213529+0000","last_change":"2024-02-23T00:31:09.211608+0000","last_active":"2024-02-23T00:31:09.213529+0000","last_peered":"2024-02-23T00:31:09.213529+0000","last_clean":"2024-02-23T00:31:09.213529+0000","last_became_active":"2024-02-23T00:31:09.211304+0000","last_became_peered":"2024-02-23T00:31:09.211304+0000","last_unstale":"2024-02-23T00:31:09.213529+0000","last_undegraded":"2024-02-23T00:31:09.213529+0000","last_fullsized":"2024-02-23T00:31:09.213529+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.12","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120075+0000","last_change":"2024-02-23T00:31:08.037384+0000","last_active":"2024-02-23T00:31:09.120075+0000","last_peered":"2024-02-23T00:31:09.120075+0000","last_clean":"2024-02-23T00:31:09.120075+0000","last_became_active":"2024-02-23T00:31:08.037179+0000","last_became_peered":"2024-02-23T00:31:08.037179+0000","last_unstale":"2024-02-23T00:31:09.120075+0000","last_undegraded":"2024-02-23T00:31:09.120075+0000","last_fullsized":"2024-02-23T00:31:09.120075+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"72.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.c","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.243744+0000","last_change":"2024-02-23T00:31:09.242134+0000","last_active":"2024-02-23T00:31:09.243744+0000","last_peered":"2024-02-23T00:31:09.243744+0000","last_clean":"2024-02-23T00:31:09.243744+0000","last_became_active":"2024-02-23T00:31:09.241661+0000","last_became_peered":"2024-02-23T00:31:09.241661+0000","last_unstale":"2024-02-23T00:31:09.243744+0000","last_undegraded":"2024-02-23T00:31:09.243744+0000","last_fullsized":"2024-02-23T00:31:09.243744+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.d","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.127645+0000","last_change":"2024-02-23T00:31:08.043913+0000","last_active":"2024-02-23T00:31:09.127645+0000","last_peered":"2024-02-23T00:31:09.127645+0000","last_clean":"2024-02-23T00:31:09.127645+0000","last_became_active":"2024-02-23T00:31:08.039502+0000","last_became_peered":"2024-02-23T00:31:08.039502+0000","last_unstale":"2024-02-23T00:31:09.127645+0000","last_undegraded":"2024-02-23T00:31:09.127645+0000","last_fullsized":"2024-02-23T00:31:09.127645+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.d","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.213696+0000","last_change":"2024-02-23T00:31:09.213246+0000","last_active":"2024-02-23T00:31:09.213696+0000","last_peered":"2024-02-23T00:31:09.213696+0000","last_clean":"2024-02-23T00:31:09.213696+0000","last_became_active":"2024-02-23T00:31:09.213015+0000","last_became_peered":"2024-02-23T00:31:09.213015+0000","last_unstale":"2024-02-23T00:31:09.213696+0000","last_undegraded":"2024-02-23T00:31:09.213696+0000","last_fullsized":"2024-02-23T00:31:09.213696+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.c","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132321+0000","last_change":"2024-02-23T00:31:07.989228+0000","last_active":"2024-02-23T00:31:09.132321+0000","last_peered":"2024-02-23T00:31:09.132321+0000","last_clean":"2024-02-23T00:31:09.132321+0000","last_became_active":"2024-02-23T00:31:07.988917+0000","last_became_peered":"2024-02-23T00:31:07.988917+0000","last_unstale":"2024-02-23T00:31:09.132321+0000","last_undegraded":"2024-02-23T00:31:09.132321+0000","last_fullsized":"2024-02-23T00:31:09.132321+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"72.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.e","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.243769+0000","last_change":"2024-02-23T00:31:09.241654+0000","last_active":"2024-02-23T00:31:09.243769+0000","last_peered":"2024-02-23T00:31:09.243769+0000","last_clean":"2024-02-23T00:31:09.243769+0000","last_became_active":"2024-02-23T00:31:09.240836+0000","last_became_peered":"2024-02-23T00:31:09.240836+0000","last_unstale":"2024-02-23T00:31:09.243769+0000","last_undegraded":"2024-02-23T00:31:09.243769+0000","last_fullsized":"2024-02-23T00:31:09.243769+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"66.f","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.155645+0000","last_change":"2024-02-23T00:31:08.016254+0000","last_active":"2024-02-23T00:31:09.155645+0000","last_peered":"2024-02-23T00:31:09.155645+0000","last_clean":"2024-02-23T00:31:09.155645+0000","last_became_active":"2024-02-23T00:31:08.015973+0000","last_became_peered":"2024-02-23T00:31:08.015973+0000","last_unstale":"2024-02-23T00:31:09.155645+0000","last_undegraded":"2024-02-23T00:31:09.155645+0000","last_fullsized":"2024-02-23T00:31:09.155645+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.f","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.226357+0000","last_change":"2024-02-23T00:31:09.215621+0000","last_active":"2024-02-23T00:31:09.226357+0000","last_peered":"2024-02-23T00:31:09.226357+0000","last_clean":"2024-02-23T00:31:09.226357+0000","last_became_active":"2024-02-23T00:31:09.215304+0000","last_became_peered":"2024-02-23T00:31:09.215304+0000","last_unstale":"2024-02-23T00:31:09.226357+0000","last_undegraded":"2024-02-23T00:31:09.226357+0000","last_fullsized":"2024-02-23T00:31:09.226357+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.e","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.149829+0000","last_change":"2024-02-23T00:31:08.052289+0000","last_active":"2024-02-23T00:31:09.149829+0000","last_peered":"2024-02-23T00:31:09.149829+0000","last_clean":"2024-02-23T00:31:09.149829+0000","last_became_active":"2024-02-23T00:31:08.051925+0000","last_became_peered":"2024-02-23T00:31:08.051925+0000","last_unstale":"2024-02-23T00:31:09.149829+0000","last_undegraded":"2024-02-23T00:31:09.149829+0000","last_fullsized":"2024-02-23T00:31:09.149829+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.8","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.211809+0000","last_change":"2024-02-23T00:31:09.211604+0000","last_active":"2024-02-23T00:31:09.211809+0000","last_peered":"2024-02-23T00:31:09.211809+0000","last_clean":"2024-02-23T00:31:09.211809+0000","last_became_active":"2024-02-23T00:31:09.211345+0000","last_became_peered":"2024-02-23T00:31:09.211345+0000","last_unstale":"2024-02-23T00:31:09.211809+0000","last_undegraded":"2024-02-23T00:31:09.211809+0000","last_fullsized":"2024-02-23T00:31:09.211809+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.9","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.171204+0000","last_change":"2024-02-23T00:31:08.017593+0000","last_active":"2024-02-23T00:31:09.171204+0000","last_peered":"2024-02-23T00:31:09.171204+0000","last_clean":"2024-02-23T00:31:09.171204+0000","last_became_active":"2024-02-23T00:31:08.017304+0000","last_became_peered":"2024-02-23T00:31:08.017304+0000","last_unstale":"2024-02-23T00:31:09.171204+0000","last_undegraded":"2024-02-23T00:31:09.171204+0000","last_fullsized":"2024-02-23T00:31:09.171204+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"72.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.9","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.186970+0000","last_change":"2024-02-23T00:31:09.186329+0000","last_active":"2024-02-23T00:31:09.186970+0000","last_peered":"2024-02-23T00:31:09.186970+0000","last_clean":"2024-02-23T00:31:09.186970+0000","last_became_active":"2024-02-23T00:31:09.186057+0000","last_became_peered":"2024-02-23T00:31:09.186057+0000","last_unstale":"2024-02-23T00:31:09.186970+0000","last_undegraded":"2024-02-23T00:31:09.186970+0000","last_fullsized":"2024-02-23T00:31:09.186970+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.8","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.148175+0000","last_change":"2024-02-23T00:31:07.968724+0000","last_active":"2024-02-23T00:31:09.148175+0000","last_peered":"2024-02-23T00:31:09.148175+0000","last_clean":"2024-02-23T00:31:09.148175+0000","last_became_active":"2024-02-23T00:31:07.968322+0000","last_became_peered":"2024-02-23T00:31:07.968322+0000","last_unstale":"2024-02-23T00:31:09.148175+0000","last_undegraded":"2024-02-23T00:31:09.148175+0000","last_fullsized":"2024-02-23T00:31:09.148175+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.a","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.181854+0000","last_change":"2024-02-23T00:31:09.181633+0000","last_active":"2024-02-23T00:31:09.181854+0000","last_peered":"2024-02-23T00:31:09.181854+0000","last_clean":"2024-02-23T00:31:09.181854+0000","last_became_active":"2024-02-23T00:31:09.181378+0000","last_became_peered":"2024-02-23T00:31:09.181378+0000","last_unstale":"2024-02-23T00:31:09.181854+0000","last_undegraded":"2024-02-23T00:31:09.181854+0000","last_fullsized":"2024-02-23T00:31:09.181854+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.b","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.119613+0000","last_change":"2024-02-23T00:31:08.036709+0000","last_active":"2024-02-23T00:31:09.119613+0000","last_peered":"2024-02-23T00:31:09.119613+0000","last_clean":"2024-02-23T00:31:09.119613+0000","last_became_active":"2024-02-23T00:31:08.036387+0000","last_became_peered":"2024-02-23T00:31:08.036387+0000","last_unstale":"2024-02-23T00:31:09.119613+0000","last_undegraded":"2024-02-23T00:31:09.119613+0000","last_fullsized":"2024-02-23T00:31:09.119613+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"72.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.b","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.207193+0000","last_change":"2024-02-23T00:31:09.193344+0000","last_active":"2024-02-23T00:31:09.207193+0000","last_peered":"2024-02-23T00:31:09.207193+0000","last_clean":"2024-02-23T00:31:09.207193+0000","last_became_active":"2024-02-23T00:31:09.193062+0000","last_became_peered":"2024-02-23T00:31:09.193062+0000","last_unstale":"2024-02-23T00:31:09.207193+0000","last_undegraded":"2024-02-23T00:31:09.207193+0000","last_fullsized":"2024-02-23T00:31:09.207193+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.a","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.148198+0000","last_change":"2024-02-23T00:31:08.052706+0000","last_active":"2024-02-23T00:31:09.148198+0000","last_peered":"2024-02-23T00:31:09.148198+0000","last_clean":"2024-02-23T00:31:09.148198+0000","last_became_active":"2024-02-23T00:31:08.052541+0000","last_became_peered":"2024-02-23T00:31:08.052541+0000","last_unstale":"2024-02-23T00:31:09.148198+0000","last_undegraded":"2024-02-23T00:31:09.148198+0000","last_fullsized":"2024-02-23T00:31:09.148198+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"72.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.4","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.238314+0000","last_change":"2024-02-23T00:31:09.237679+0000","last_active":"2024-02-23T00:31:09.238314+0000","last_peered":"2024-02-23T00:31:09.238314+0000","last_clean":"2024-02-23T00:31:09.238314+0000","last_became_active":"2024-02-23T00:31:09.237324+0000","last_became_peered":"2024-02-23T00:31:09.237324+0000","last_unstale":"2024-02-23T00:31:09.238314+0000","last_undegraded":"2024-02-23T00:31:09.238314+0000","last_fullsized":"2024-02-23T00:31:09.238314+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.5","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.127719+0000","last_change":"2024-02-23T00:31:08.058111+0000","last_active":"2024-02-23T00:31:09.127719+0000","last_peered":"2024-02-23T00:31:09.127719+0000","last_clean":"2024-02-23T00:31:09.127719+0000","last_became_active":"2024-02-23T00:31:08.057695+0000","last_became_peered":"2024-02-23T00:31:08.057695+0000","last_unstale":"2024-02-23T00:31:09.127719+0000","last_undegraded":"2024-02-23T00:31:09.127719+0000","last_fullsized":"2024-02-23T00:31:09.127719+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"72.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.5","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.242814+0000","last_change":"2024-02-23T00:31:09.232175+0000","last_active":"2024-02-23T00:31:09.242814+0000","last_peered":"2024-02-23T00:31:09.242814+0000","last_clean":"2024-02-23T00:31:09.242814+0000","last_became_active":"2024-02-23T00:31:09.230101+0000","last_became_peered":"2024-02-23T00:31:09.230101+0000","last_unstale":"2024-02-23T00:31:09.242814+0000","last_undegraded":"2024-02-23T00:31:09.242814+0000","last_fullsized":"2024-02-23T00:31:09.242814+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"66.4","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132395+0000","last_change":"2024-02-23T00:31:08.039860+0000","last_active":"2024-02-23T00:31:09.132395+0000","last_peered":"2024-02-23T00:31:09.132395+0000","last_clean":"2024-02-23T00:31:09.132395+0000","last_became_active":"2024-02-23T00:31:08.039515+0000","last_became_peered":"2024-02-23T00:31:08.039515+0000","last_unstale":"2024-02-23T00:31:09.132395+0000","last_undegraded":"2024-02-23T00:31:09.132395+0000","last_fullsized":"2024-02-23T00:31:09.132395+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"72.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.6","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.222117+0000","last_change":"2024-02-23T00:31:09.221910+0000","last_active":"2024-02-23T00:31:09.222117+0000","last_peered":"2024-02-23T00:31:09.222117+0000","last_clean":"2024-02-23T00:31:09.222117+0000","last_became_active":"2024-02-23T00:31:09.221624+0000","last_became_peered":"2024-02-23T00:31:09.221624+0000","last_unstale":"2024-02-23T00:31:09.222117+0000","last_undegraded":"2024-02-23T00:31:09.222117+0000","last_fullsized":"2024-02-23T00:31:09.222117+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.7","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135008+0000","last_change":"2024-02-23T00:31:07.989078+0000","last_active":"2024-02-23T00:31:09.135008+0000","last_peered":"2024-02-23T00:31:09.135008+0000","last_clean":"2024-02-23T00:31:09.135008+0000","last_became_active":"2024-02-23T00:31:07.988827+0000","last_became_peered":"2024-02-23T00:31:07.988827+0000","last_unstale":"2024-02-23T00:31:09.135008+0000","last_undegraded":"2024-02-23T00:31:09.135008+0000","last_fullsized":"2024-02-23T00:31:09.135008+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"72.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.7","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.235044+0000","last_change":"2024-02-23T00:31:09.234823+0000","last_active":"2024-02-23T00:31:09.235044+0000","last_peered":"2024-02-23T00:31:09.235044+0000","last_clean":"2024-02-23T00:31:09.235044+0000","last_became_active":"2024-02-23T00:31:09.234564+0000","last_became_peered":"2024-02-23T00:31:09.234564+0000","last_unstale":"2024-02-23T00:31:09.235044+0000","last_undegraded":"2024-02-23T00:31:09.235044+0000","last_fullsized":"2024-02-23T00:31:09.235044+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"66.6","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126914+0000","last_change":"2024-02-23T00:31:08.039566+0000","last_active":"2024-02-23T00:31:09.126914+0000","last_peered":"2024-02-23T00:31:09.126914+0000","last_clean":"2024-02-23T00:31:09.126914+0000","last_became_active":"2024-02-23T00:31:08.038904+0000","last_became_peered":"2024-02-23T00:31:08.038904+0000","last_unstale":"2024-02-23T00:31:09.126914+0000","last_undegraded":"2024-02-23T00:31:09.126914+0000","last_fullsized":"2024-02-23T00:31:09.126914+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"72.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.0","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.221791+0000","last_change":"2024-02-23T00:31:09.221500+0000","last_active":"2024-02-23T00:31:09.221791+0000","last_peered":"2024-02-23T00:31:09.221791+0000","last_clean":"2024-02-23T00:31:09.221791+0000","last_became_active":"2024-02-23T00:31:09.221229+0000","last_became_peered":"2024-02-23T00:31:09.221229+0000","last_unstale":"2024-02-23T00:31:09.221791+0000","last_undegraded":"2024-02-23T00:31:09.221791+0000","last_fullsized":"2024-02-23T00:31:09.221791+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"66.1","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130216+0000","last_change":"2024-02-23T00:31:08.040716+0000","last_active":"2024-02-23T00:31:09.130216+0000","last_peered":"2024-02-23T00:31:09.130216+0000","last_clean":"2024-02-23T00:31:09.130216+0000","last_became_active":"2024-02-23T00:31:08.040500+0000","last_became_peered":"2024-02-23T00:31:08.040500+0000","last_unstale":"2024-02-23T00:31:09.130216+0000","last_undegraded":"2024-02-23T00:31:09.130216+0000","last_fullsized":"2024-02-23T00:31:09.130216+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"72.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.1","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.236167+0000","last_change":"2024-02-23T00:31:09.235868+0000","last_active":"2024-02-23T00:31:09.236167+0000","last_peered":"2024-02-23T00:31:09.236167+0000","last_clean":"2024-02-23T00:31:09.236167+0000","last_became_active":"2024-02-23T00:31:09.235578+0000","last_became_peered":"2024-02-23T00:31:09.235578+0000","last_unstale":"2024-02-23T00:31:09.236167+0000","last_undegraded":"2024-02-23T00:31:09.236167+0000","last_fullsized":"2024-02-23T00:31:09.236167+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"66.0","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130795+0000","last_change":"2024-02-23T00:31:07.972534+0000","last_active":"2024-02-23T00:31:09.130795+0000","last_peered":"2024-02-23T00:31:09.130795+0000","last_clean":"2024-02-23T00:31:09.130795+0000","last_became_active":"2024-02-23T00:31:07.972132+0000","last_became_peered":"2024-02-23T00:31:07.972132+0000","last_unstale":"2024-02-23T00:31:09.130795+0000","last_undegraded":"2024-02-23T00:31:09.130795+0000","last_fullsized":"2024-02-23T00:31:09.130795+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"72.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.2","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.221009+0000","last_change":"2024-02-23T00:31:09.220670+0000","last_active":"2024-02-23T00:31:09.221009+0000","last_peered":"2024-02-23T00:31:09.221009+0000","last_clean":"2024-02-23T00:31:09.221009+0000","last_became_active":"2024-02-23T00:31:09.220336+0000","last_became_peered":"2024-02-23T00:31:09.220336+0000","last_unstale":"2024-02-23T00:31:09.221009+0000","last_undegraded":"2024-02-23T00:31:09.221009+0000","last_fullsized":"2024-02-23T00:31:09.221009+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"66.3","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.119296+0000","last_change":"2024-02-23T00:31:07.972120+0000","last_active":"2024-02-23T00:31:09.119296+0000","last_peered":"2024-02-23T00:31:09.119296+0000","last_clean":"2024-02-23T00:31:09.119296+0000","last_became_active":"2024-02-23T00:31:07.971885+0000","last_became_peered":"2024-02-23T00:31:07.971885+0000","last_unstale":"2024-02-23T00:31:09.119296+0000","last_undegraded":"2024-02-23T00:31:09.119296+0000","last_fullsized":"2024-02-23T00:31:09.119296+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"72.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.834361+0000","last_change":"2024-02-23T00:31:11.834361+0000","last_active":"2024-02-23T00:31:11.834361+0000","last_peered":"2024-02-23T00:31:11.834361+0000","last_clean":"2024-02-23T00:31:11.834361+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.834361+0000","last_undegraded":"2024-02-23T00:31:11.834361+0000","last_fullsized":"2024-02-23T00:31:11.834361+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.834361+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.834361+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:11.137211+0000","last_change":"2024-02-23T00:31:11.137211+0000","last_active":"2024-02-23T00:31:11.137211+0000","last_peered":"2024-02-23T00:31:11.137211+0000","last_clean":"2024-02-23T00:31:11.137211+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:11.137211+0000","last_undegraded":"2024-02-23T00:31:11.137211+0000","last_fullsized":"2024-02-23T00:31:11.137211+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:11.137211+0000","last_clean_scrub_stamp":"2024-02-23T00:31:11.137211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"67.3","version":"0'0","reported_seq":10,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.233315+0000","last_change":"2024-02-23T00:31:09.233117+0000","last_active":"2024-02-23T00:31:09.233315+0000","last_peered":"2024-02-23T00:31:09.233315+0000","last_clean":"2024-02-23T00:31:09.233315+0000","last_became_active":"2024-02-23T00:31:09.232780+0000","last_became_peered":"2024-02-23T00:31:09.232780+0000","last_unstale":"2024-02-23T00:31:09.233315+0000","last_undegraded":"2024-02-23T00:31:09.233315+0000","last_fullsized":"2024-02-23T00:31:09.233315+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:07.864982+0000","last_clean_scrub_stamp":"2024-02-23T00:31:07.864982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"66.2","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.148527+0000","last_change":"2024-02-23T00:31:08.029567+0000","last_active":"2024-02-23T00:31:09.148527+0000","last_peered":"2024-02-23T00:31:09.148527+0000","last_clean":"2024-02-23T00:31:09.148527+0000","last_became_active":"2024-02-23T00:31:08.029274+0000","last_became_peered":"2024-02-23T00:31:08.029274+0000","last_unstale":"2024-02-23T00:31:09.148527+0000","last_undegraded":"2024-02-23T00:31:09.148527+0000","last_fullsized":"2024-02-23T00:31:09.148527+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:06.825331+0000","last_clean_scrub_stamp":"2024-02-23T00:31:06.825331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.0","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.157117+0000","last_change":"2024-02-23T00:31:06.334878+0000","last_active":"2024-02-23T00:31:09.157117+0000","last_peered":"2024-02-23T00:31:09.157117+0000","last_clean":"2024-02-23T00:31:09.157117+0000","last_became_active":"2024-02-23T00:31:06.330477+0000","last_became_peered":"2024-02-23T00:31:06.330477+0000","last_unstale":"2024-02-23T00:31:09.157117+0000","last_undegraded":"2024-02-23T00:31:09.157117+0000","last_fullsized":"2024-02-23T00:31:09.157117+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.2","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.211857+0000","last_change":"2024-02-23T00:31:09.211857+0000","last_active":"2024-02-23T00:31:09.211857+0000","last_peered":"2024-02-23T00:31:09.211857+0000","last_clean":"2024-02-23T00:31:09.211857+0000","last_became_active":"2024-02-23T00:31:03.660156+0000","last_became_peered":"2024-02-23T00:31:03.660156+0000","last_unstale":"2024-02-23T00:31:09.211857+0000","last_undegraded":"2024-02-23T00:31:09.211857+0000","last_fullsized":"2024-02-23T00:31:09.211857+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.b","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120251+0000","last_change":"2024-02-23T00:30:59.528472+0000","last_active":"2024-02-23T00:31:09.120251+0000","last_peered":"2024-02-23T00:31:09.120251+0000","last_clean":"2024-02-23T00:31:09.120251+0000","last_became_active":"2024-02-23T00:30:59.528139+0000","last_became_peered":"2024-02-23T00:30:59.528139+0000","last_unstale":"2024-02-23T00:31:09.120251+0000","last_undegraded":"2024-02-23T00:31:09.120251+0000","last_fullsized":"2024-02-23T00:31:09.120251+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"49.e","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.183050+0000","last_change":"2024-02-23T00:30:56.063253+0000","last_active":"2024-02-23T00:31:09.183050+0000","last_peered":"2024-02-23T00:31:09.183050+0000","last_clean":"2024-02-23T00:31:09.183050+0000","last_became_active":"2024-02-23T00:30:56.063001+0000","last_became_peered":"2024-02-23T00:30:56.063001+0000","last_unstale":"2024-02-23T00:31:09.183050+0000","last_undegraded":"2024-02-23T00:31:09.183050+0000","last_fullsized":"2024-02-23T00:31:09.183050+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"44.13","version":"107'2","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.195389+0000","last_change":"2024-02-23T00:30:55.001409+0000","last_active":"2024-02-23T00:31:09.195389+0000","last_peered":"2024-02-23T00:31:09.195389+0000","last_clean":"2024-02-23T00:31:09.195389+0000","last_became_active":"2024-02-23T00:30:55.001137+0000","last_became_peered":"2024-02-23T00:30:55.001137+0000","last_unstale":"2024-02-23T00:31:09.195389+0000","last_undegraded":"2024-02-23T00:31:09.195389+0000","last_fullsized":"2024-02-23T00:31:09.195389+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8192,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.162560+0000","last_change":"2024-02-23T00:30:52.778011+0000","last_active":"2024-02-23T00:31:09.162560+0000","last_peered":"2024-02-23T00:31:09.162560+0000","last_clean":"2024-02-23T00:31:09.162560+0000","last_became_active":"2024-02-23T00:30:52.777680+0000","last_became_peered":"2024-02-23T00:30:52.777680+0000","last_unstale":"2024-02-23T00:31:09.162560+0000","last_undegraded":"2024-02-23T00:31:09.162560+0000","last_fullsized":"2024-02-23T00:31:09.162560+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.1","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122051+0000","last_change":"2024-02-23T00:31:06.258401+0000","last_active":"2024-02-23T00:31:09.122051+0000","last_peered":"2024-02-23T00:31:09.122051+0000","last_clean":"2024-02-23T00:31:09.122051+0000","last_became_active":"2024-02-23T00:31:06.258140+0000","last_became_peered":"2024-02-23T00:31:06.258140+0000","last_unstale":"2024-02-23T00:31:09.122051+0000","last_undegraded":"2024-02-23T00:31:09.122051+0000","last_fullsized":"2024-02-23T00:31:09.122051+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.3","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.158576+0000","last_change":"2024-02-23T00:31:09.158576+0000","last_active":"2024-02-23T00:31:09.158576+0000","last_peered":"2024-02-23T00:31:09.158576+0000","last_clean":"2024-02-23T00:31:09.158576+0000","last_became_active":"2024-02-23T00:31:03.611979+0000","last_became_peered":"2024-02-23T00:31:03.611979+0000","last_unstale":"2024-02-23T00:31:09.158576+0000","last_undegraded":"2024-02-23T00:31:09.158576+0000","last_fullsized":"2024-02-23T00:31:09.158576+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.a","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122943+0000","last_change":"2024-02-23T00:30:59.529301+0000","last_active":"2024-02-23T00:31:09.122943+0000","last_peered":"2024-02-23T00:31:09.122943+0000","last_clean":"2024-02-23T00:31:09.122943+0000","last_became_active":"2024-02-23T00:30:59.528978+0000","last_became_peered":"2024-02-23T00:30:59.528978+0000","last_unstale":"2024-02-23T00:31:09.122943+0000","last_undegraded":"2024-02-23T00:31:09.122943+0000","last_fullsized":"2024-02-23T00:31:09.122943+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"49.f","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.192872+0000","last_change":"2024-02-23T00:30:56.062184+0000","last_active":"2024-02-23T00:31:09.192872+0000","last_peered":"2024-02-23T00:31:09.192872+0000","last_clean":"2024-02-23T00:31:09.192872+0000","last_became_active":"2024-02-23T00:30:56.061736+0000","last_became_peered":"2024-02-23T00:30:56.061736+0000","last_unstale":"2024-02-23T00:31:09.192872+0000","last_undegraded":"2024-02-23T00:31:09.192872+0000","last_fullsized":"2024-02-23T00:31:09.192872+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"44.12","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129267+0000","last_change":"2024-02-23T00:30:54.996108+0000","last_active":"2024-02-23T00:31:09.129267+0000","last_peered":"2024-02-23T00:31:09.129267+0000","last_clean":"2024-02-23T00:31:09.129267+0000","last_became_active":"2024-02-23T00:30:54.995715+0000","last_became_peered":"2024-02-23T00:30:54.995715+0000","last_unstale":"2024-02-23T00:31:09.129267+0000","last_undegraded":"2024-02-23T00:31:09.129267+0000","last_fullsized":"2024-02-23T00:31:09.129267+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125837+0000","last_change":"2024-02-23T00:30:52.749161+0000","last_active":"2024-02-23T00:31:09.125837+0000","last_peered":"2024-02-23T00:31:09.125837+0000","last_clean":"2024-02-23T00:31:09.125837+0000","last_became_active":"2024-02-23T00:30:52.748972+0000","last_became_peered":"2024-02-23T00:30:52.748972+0000","last_unstale":"2024-02-23T00:31:09.125837+0000","last_undegraded":"2024-02-23T00:31:09.125837+0000","last_fullsized":"2024-02-23T00:31:09.125837+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.2","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.148655+0000","last_change":"2024-02-23T00:31:06.329994+0000","last_active":"2024-02-23T00:31:09.148655+0000","last_peered":"2024-02-23T00:31:09.148655+0000","last_clean":"2024-02-23T00:31:09.148655+0000","last_became_active":"2024-02-23T00:31:06.329726+0000","last_became_peered":"2024-02-23T00:31:06.329726+0000","last_unstale":"2024-02-23T00:31:09.148655+0000","last_undegraded":"2024-02-23T00:31:09.148655+0000","last_fullsized":"2024-02-23T00:31:09.148655+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.169432+0000","last_change":"2024-02-23T00:31:09.169432+0000","last_active":"2024-02-23T00:31:09.169432+0000","last_peered":"2024-02-23T00:31:09.169432+0000","last_clean":"2024-02-23T00:31:09.169432+0000","last_became_active":"2024-02-23T00:31:03.665678+0000","last_became_peered":"2024-02-23T00:31:03.665678+0000","last_unstale":"2024-02-23T00:31:09.169432+0000","last_undegraded":"2024-02-23T00:31:09.169432+0000","last_fullsized":"2024-02-23T00:31:09.169432+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.9","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130655+0000","last_change":"2024-02-23T00:30:59.537277+0000","last_active":"2024-02-23T00:31:09.130655+0000","last_peered":"2024-02-23T00:31:09.130655+0000","last_clean":"2024-02-23T00:31:09.130655+0000","last_became_active":"2024-02-23T00:30:59.536810+0000","last_became_peered":"2024-02-23T00:30:59.536810+0000","last_unstale":"2024-02-23T00:31:09.130655+0000","last_undegraded":"2024-02-23T00:31:09.130655+0000","last_fullsized":"2024-02-23T00:31:09.130655+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"49.c","version":"106'1","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174356+0000","last_change":"2024-02-23T00:30:56.025967+0000","last_active":"2024-02-23T00:31:09.174356+0000","last_peered":"2024-02-23T00:31:09.174356+0000","last_clean":"2024-02-23T00:31:09.174356+0000","last_became_active":"2024-02-23T00:30:56.024820+0000","last_became_peered":"2024-02-23T00:30:56.024820+0000","last_unstale":"2024-02-23T00:31:09.174356+0000","last_undegraded":"2024-02-23T00:31:09.174356+0000","last_fullsized":"2024-02-23T00:31:09.174356+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"44.11","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131587+0000","last_change":"2024-02-23T00:30:54.982617+0000","last_active":"2024-02-23T00:31:09.131587+0000","last_peered":"2024-02-23T00:31:09.131587+0000","last_clean":"2024-02-23T00:31:09.131587+0000","last_became_active":"2024-02-23T00:30:54.981877+0000","last_became_peered":"2024-02-23T00:30:54.981877+0000","last_unstale":"2024-02-23T00:31:09.131587+0000","last_undegraded":"2024-02-23T00:31:09.131587+0000","last_fullsized":"2024-02-23T00:31:09.131587+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.1b","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.195531+0000","last_change":"2024-02-23T00:30:52.761108+0000","last_active":"2024-02-23T00:31:09.195531+0000","last_peered":"2024-02-23T00:31:09.195531+0000","last_clean":"2024-02-23T00:31:09.195531+0000","last_became_active":"2024-02-23T00:30:52.760868+0000","last_became_peered":"2024-02-23T00:30:52.760868+0000","last_unstale":"2024-02-23T00:31:09.195531+0000","last_undegraded":"2024-02-23T00:31:09.195531+0000","last_fullsized":"2024-02-23T00:31:09.195531+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"63.3","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120106+0000","last_change":"2024-02-23T00:31:06.259266+0000","last_active":"2024-02-23T00:31:09.120106+0000","last_peered":"2024-02-23T00:31:09.120106+0000","last_clean":"2024-02-23T00:31:09.120106+0000","last_became_active":"2024-02-23T00:31:06.258997+0000","last_became_peered":"2024-02-23T00:31:06.258997+0000","last_unstale":"2024-02-23T00:31:09.120106+0000","last_undegraded":"2024-02-23T00:31:09.120106+0000","last_fullsized":"2024-02-23T00:31:09.120106+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.180867+0000","last_change":"2024-02-23T00:31:09.180867+0000","last_active":"2024-02-23T00:31:09.180867+0000","last_peered":"2024-02-23T00:31:09.180867+0000","last_clean":"2024-02-23T00:31:09.180867+0000","last_became_active":"2024-02-23T00:31:03.610801+0000","last_became_peered":"2024-02-23T00:31:03.610801+0000","last_unstale":"2024-02-23T00:31:09.180867+0000","last_undegraded":"2024-02-23T00:31:09.180867+0000","last_fullsized":"2024-02-23T00:31:09.180867+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.8","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.160680+0000","last_change":"2024-02-23T00:30:59.524372+0000","last_active":"2024-02-23T00:31:09.160680+0000","last_peered":"2024-02-23T00:31:09.160680+0000","last_clean":"2024-02-23T00:31:09.160680+0000","last_became_active":"2024-02-23T00:30:59.521701+0000","last_became_peered":"2024-02-23T00:30:59.521701+0000","last_unstale":"2024-02-23T00:31:09.160680+0000","last_undegraded":"2024-02-23T00:31:09.160680+0000","last_fullsized":"2024-02-23T00:31:09.160680+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"49.d","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.124551+0000","last_change":"2024-02-23T00:30:56.033953+0000","last_active":"2024-02-23T00:31:09.124551+0000","last_peered":"2024-02-23T00:31:09.124551+0000","last_clean":"2024-02-23T00:31:09.124551+0000","last_became_active":"2024-02-23T00:30:56.033514+0000","last_became_peered":"2024-02-23T00:30:56.033514+0000","last_unstale":"2024-02-23T00:31:09.124551+0000","last_undegraded":"2024-02-23T00:31:09.124551+0000","last_fullsized":"2024-02-23T00:31:09.124551+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"44.10","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.160753+0000","last_change":"2024-02-23T00:30:54.994086+0000","last_active":"2024-02-23T00:31:09.160753+0000","last_peered":"2024-02-23T00:31:09.160753+0000","last_clean":"2024-02-23T00:31:09.160753+0000","last_became_active":"2024-02-23T00:30:54.993786+0000","last_became_peered":"2024-02-23T00:30:54.993786+0000","last_unstale":"2024-02-23T00:31:09.160753+0000","last_undegraded":"2024-02-23T00:31:09.160753+0000","last_fullsized":"2024-02-23T00:31:09.160753+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.1a","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138501+0000","last_change":"2024-02-23T00:30:52.746690+0000","last_active":"2024-02-23T00:31:09.138501+0000","last_peered":"2024-02-23T00:31:09.138501+0000","last_clean":"2024-02-23T00:31:09.138501+0000","last_became_active":"2024-02-23T00:30:52.746395+0000","last_became_peered":"2024-02-23T00:30:52.746395+0000","last_unstale":"2024-02-23T00:31:09.138501+0000","last_undegraded":"2024-02-23T00:31:09.138501+0000","last_fullsized":"2024-02-23T00:31:09.138501+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.4","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125150+0000","last_change":"2024-02-23T00:31:06.215224+0000","last_active":"2024-02-23T00:31:09.125150+0000","last_peered":"2024-02-23T00:31:09.125150+0000","last_clean":"2024-02-23T00:31:09.125150+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:31:09.125150+0000","last_undegraded":"2024-02-23T00:31:09.125150+0000","last_fullsized":"2024-02-23T00:31:09.125150+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.6","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131819+0000","last_change":"2024-02-23T00:31:09.131819+0000","last_active":"2024-02-23T00:31:09.131819+0000","last_peered":"2024-02-23T00:31:09.131819+0000","last_clean":"2024-02-23T00:31:09.131819+0000","last_became_active":"2024-02-23T00:31:03.622928+0000","last_became_peered":"2024-02-23T00:31:03.622928+0000","last_unstale":"2024-02-23T00:31:09.131819+0000","last_undegraded":"2024-02-23T00:31:09.131819+0000","last_fullsized":"2024-02-23T00:31:09.131819+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"56.3","version":"113'8","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132893+0000","last_change":"2024-02-23T00:31:01.529570+0000","last_active":"2024-02-23T00:31:09.132893+0000","last_peered":"2024-02-23T00:31:09.132893+0000","last_clean":"2024-02-23T00:31:09.132893+0000","last_became_active":"2024-02-23T00:31:01.529279+0000","last_became_peered":"2024-02-23T00:31:01.529279+0000","last_unstale":"2024-02-23T00:31:09.132893+0000","last_undegraded":"2024-02-23T00:31:09.132893+0000","last_fullsized":"2024-02-23T00:31:09.132893+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:00.482621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:00.482621+0000","last_clean_scrub_stamp":"2024-02-23T00:31:00.482621+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"52.f","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129284+0000","last_change":"2024-02-23T00:30:59.542614+0000","last_active":"2024-02-23T00:31:09.129284+0000","last_peered":"2024-02-23T00:31:09.129284+0000","last_clean":"2024-02-23T00:31:09.129284+0000","last_became_active":"2024-02-23T00:30:59.542332+0000","last_became_peered":"2024-02-23T00:30:59.542332+0000","last_unstale":"2024-02-23T00:31:09.129284+0000","last_undegraded":"2024-02-23T00:31:09.129284+0000","last_fullsized":"2024-02-23T00:31:09.129284+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"49.a","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138636+0000","last_change":"2024-02-23T00:30:56.040802+0000","last_active":"2024-02-23T00:31:09.138636+0000","last_peered":"2024-02-23T00:31:09.138636+0000","last_clean":"2024-02-23T00:31:09.138636+0000","last_became_active":"2024-02-23T00:30:56.039854+0000","last_became_peered":"2024-02-23T00:30:56.039854+0000","last_unstale":"2024-02-23T00:31:09.138636+0000","last_undegraded":"2024-02-23T00:31:09.138636+0000","last_fullsized":"2024-02-23T00:31:09.138636+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"44.17","version":"112'2","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132409+0000","last_change":"2024-02-23T00:30:56.020403+0000","last_active":"2024-02-23T00:31:09.132409+0000","last_peered":"2024-02-23T00:31:09.132409+0000","last_clean":"2024-02-23T00:31:09.132409+0000","last_became_active":"2024-02-23T00:30:56.020125+0000","last_became_peered":"2024-02-23T00:30:56.020125+0000","last_unstale":"2024-02-23T00:31:09.132409+0000","last_undegraded":"2024-02-23T00:31:09.132409+0000","last_fullsized":"2024-02-23T00:31:09.132409+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8192,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.1d","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.184274+0000","last_change":"2024-02-23T00:30:52.766060+0000","last_active":"2024-02-23T00:31:09.184274+0000","last_peered":"2024-02-23T00:31:09.184274+0000","last_clean":"2024-02-23T00:31:09.184274+0000","last_became_active":"2024-02-23T00:30:52.765863+0000","last_became_peered":"2024-02-23T00:30:52.765863+0000","last_unstale":"2024-02-23T00:31:09.184274+0000","last_undegraded":"2024-02-23T00:31:09.184274+0000","last_fullsized":"2024-02-23T00:31:09.184274+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.5","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128592+0000","last_change":"2024-02-23T00:31:06.258729+0000","last_active":"2024-02-23T00:31:09.128592+0000","last_peered":"2024-02-23T00:31:09.128592+0000","last_clean":"2024-02-23T00:31:09.128592+0000","last_became_active":"2024-02-23T00:31:06.258422+0000","last_became_peered":"2024-02-23T00:31:06.258422+0000","last_unstale":"2024-02-23T00:31:09.128592+0000","last_undegraded":"2024-02-23T00:31:09.128592+0000","last_fullsized":"2024-02-23T00:31:09.128592+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.7","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126560+0000","last_change":"2024-02-23T00:31:09.126560+0000","last_active":"2024-02-23T00:31:09.126560+0000","last_peered":"2024-02-23T00:31:09.126560+0000","last_clean":"2024-02-23T00:31:09.126560+0000","last_became_active":"2024-02-23T00:31:03.659710+0000","last_became_peered":"2024-02-23T00:31:03.659710+0000","last_unstale":"2024-02-23T00:31:09.126560+0000","last_undegraded":"2024-02-23T00:31:09.126560+0000","last_fullsized":"2024-02-23T00:31:09.126560+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"56.2","version":"113'20","reported_seq":52,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123217+0000","last_change":"2024-02-23T00:31:01.521764+0000","last_active":"2024-02-23T00:31:09.123217+0000","last_peered":"2024-02-23T00:31:09.123217+0000","last_clean":"2024-02-23T00:31:09.123217+0000","last_became_active":"2024-02-23T00:31:01.521457+0000","last_became_peered":"2024-02-23T00:31:01.521457+0000","last_unstale":"2024-02-23T00:31:09.123217+0000","last_undegraded":"2024-02-23T00:31:09.123217+0000","last_fullsized":"2024-02-23T00:31:09.123217+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:00.482621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:00.482621+0000","last_clean_scrub_stamp":"2024-02-23T00:31:00.482621+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":640,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"52.e","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.127275+0000","last_change":"2024-02-23T00:30:59.523704+0000","last_active":"2024-02-23T00:31:09.127275+0000","last_peered":"2024-02-23T00:31:09.127275+0000","last_clean":"2024-02-23T00:31:09.127275+0000","last_became_active":"2024-02-23T00:30:59.523283+0000","last_became_peered":"2024-02-23T00:30:59.523283+0000","last_unstale":"2024-02-23T00:31:09.127275+0000","last_undegraded":"2024-02-23T00:31:09.127275+0000","last_fullsized":"2024-02-23T00:31:09.127275+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"49.b","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120517+0000","last_change":"2024-02-23T00:30:56.032200+0000","last_active":"2024-02-23T00:31:09.120517+0000","last_peered":"2024-02-23T00:31:09.120517+0000","last_clean":"2024-02-23T00:31:09.120517+0000","last_became_active":"2024-02-23T00:30:56.031819+0000","last_became_peered":"2024-02-23T00:30:56.031819+0000","last_unstale":"2024-02-23T00:31:09.120517+0000","last_undegraded":"2024-02-23T00:31:09.120517+0000","last_fullsized":"2024-02-23T00:31:09.120517+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"44.16","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135071+0000","last_change":"2024-02-23T00:30:56.053247+0000","last_active":"2024-02-23T00:31:09.135071+0000","last_peered":"2024-02-23T00:31:09.135071+0000","last_clean":"2024-02-23T00:31:09.135071+0000","last_became_active":"2024-02-23T00:30:56.052968+0000","last_became_peered":"2024-02-23T00:30:56.052968+0000","last_unstale":"2024-02-23T00:31:09.135071+0000","last_undegraded":"2024-02-23T00:31:09.135071+0000","last_fullsized":"2024-02-23T00:31:09.135071+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.1c","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132775+0000","last_change":"2024-02-23T00:30:52.758720+0000","last_active":"2024-02-23T00:31:09.132775+0000","last_peered":"2024-02-23T00:31:09.132775+0000","last_clean":"2024-02-23T00:31:09.132775+0000","last_became_active":"2024-02-23T00:30:52.757850+0000","last_became_peered":"2024-02-23T00:30:52.757850+0000","last_unstale":"2024-02-23T00:31:09.132775+0000","last_undegraded":"2024-02-23T00:31:09.132775+0000","last_fullsized":"2024-02-23T00:31:09.132775+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.6","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128613+0000","last_change":"2024-02-23T00:31:06.314136+0000","last_active":"2024-02-23T00:31:09.128613+0000","last_peered":"2024-02-23T00:31:09.128613+0000","last_clean":"2024-02-23T00:31:09.128613+0000","last_became_active":"2024-02-23T00:31:06.313441+0000","last_became_peered":"2024-02-23T00:31:06.313441+0000","last_unstale":"2024-02-23T00:31:09.128613+0000","last_undegraded":"2024-02-23T00:31:09.128613+0000","last_fullsized":"2024-02-23T00:31:09.128613+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.4","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.164526+0000","last_change":"2024-02-23T00:31:09.164526+0000","last_active":"2024-02-23T00:31:09.164526+0000","last_peered":"2024-02-23T00:31:09.164526+0000","last_clean":"2024-02-23T00:31:09.164526+0000","last_became_active":"2024-02-23T00:31:03.619226+0000","last_became_peered":"2024-02-23T00:31:03.619226+0000","last_unstale":"2024-02-23T00:31:09.164526+0000","last_undegraded":"2024-02-23T00:31:09.164526+0000","last_fullsized":"2024-02-23T00:31:09.164526+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"56.1","version":"118'15","reported_seq":37,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.989855+0000","last_change":"2024-02-23T00:31:01.531346+0000","last_active":"2024-02-23T00:31:09.989855+0000","last_peered":"2024-02-23T00:31:09.989855+0000","last_clean":"2024-02-23T00:31:09.989855+0000","last_became_active":"2024-02-23T00:31:01.531134+0000","last_became_peered":"2024-02-23T00:31:01.531134+0000","last_unstale":"2024-02-23T00:31:09.989855+0000","last_undegraded":"2024-02-23T00:31:09.989855+0000","last_fullsized":"2024-02-23T00:31:09.989855+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:00.482621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:00.482621+0000","last_clean_scrub_stamp":"2024-02-23T00:31:00.482621+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"52.d","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128028+0000","last_change":"2024-02-23T00:30:59.521719+0000","last_active":"2024-02-23T00:31:09.128028+0000","last_peered":"2024-02-23T00:31:09.128028+0000","last_clean":"2024-02-23T00:31:09.128028+0000","last_became_active":"2024-02-23T00:30:59.521397+0000","last_became_peered":"2024-02-23T00:30:59.521397+0000","last_unstale":"2024-02-23T00:31:09.128028+0000","last_undegraded":"2024-02-23T00:31:09.128028+0000","last_fullsized":"2024-02-23T00:31:09.128028+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"49.8","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125717+0000","last_change":"2024-02-23T00:30:56.049140+0000","last_active":"2024-02-23T00:31:09.125717+0000","last_peered":"2024-02-23T00:31:09.125717+0000","last_clean":"2024-02-23T00:31:09.125717+0000","last_became_active":"2024-02-23T00:30:56.048468+0000","last_became_peered":"2024-02-23T00:30:56.048468+0000","last_unstale":"2024-02-23T00:31:09.125717+0000","last_undegraded":"2024-02-23T00:31:09.125717+0000","last_fullsized":"2024-02-23T00:31:09.125717+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"44.15","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128067+0000","last_change":"2024-02-23T00:30:54.992979+0000","last_active":"2024-02-23T00:31:09.128067+0000","last_peered":"2024-02-23T00:31:09.128067+0000","last_clean":"2024-02-23T00:31:09.128067+0000","last_became_active":"2024-02-23T00:30:54.992685+0000","last_became_peered":"2024-02-23T00:30:54.992685+0000","last_unstale":"2024-02-23T00:31:09.128067+0000","last_undegraded":"2024-02-23T00:31:09.128067+0000","last_fullsized":"2024-02-23T00:31:09.128067+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1f","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.228213+0000","last_change":"2024-02-23T00:30:52.780134+0000","last_active":"2024-02-23T00:31:09.228213+0000","last_peered":"2024-02-23T00:31:09.228213+0000","last_clean":"2024-02-23T00:31:09.228213+0000","last_became_active":"2024-02-23T00:30:52.779908+0000","last_became_peered":"2024-02-23T00:30:52.779908+0000","last_unstale":"2024-02-23T00:31:09.228213+0000","last_undegraded":"2024-02-23T00:31:09.228213+0000","last_fullsized":"2024-02-23T00:31:09.228213+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"63.7","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.182594+0000","last_change":"2024-02-23T00:31:06.339686+0000","last_active":"2024-02-23T00:31:09.182594+0000","last_peered":"2024-02-23T00:31:09.182594+0000","last_clean":"2024-02-23T00:31:09.182594+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:31:09.182594+0000","last_undegraded":"2024-02-23T00:31:09.182594+0000","last_fullsized":"2024-02-23T00:31:09.182594+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.5","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.182064+0000","last_change":"2024-02-23T00:31:09.182064+0000","last_active":"2024-02-23T00:31:09.182064+0000","last_peered":"2024-02-23T00:31:09.182064+0000","last_clean":"2024-02-23T00:31:09.182064+0000","last_became_active":"2024-02-23T00:31:03.605994+0000","last_became_peered":"2024-02-23T00:31:03.605994+0000","last_unstale":"2024-02-23T00:31:09.182064+0000","last_undegraded":"2024-02-23T00:31:09.182064+0000","last_fullsized":"2024-02-23T00:31:09.182064+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"56.0","version":"117'4","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.916676+0000","last_change":"2024-02-23T00:31:01.520966+0000","last_active":"2024-02-23T00:31:09.916676+0000","last_peered":"2024-02-23T00:31:09.916676+0000","last_clean":"2024-02-23T00:31:09.916676+0000","last_became_active":"2024-02-23T00:31:01.520595+0000","last_became_peered":"2024-02-23T00:31:01.520595+0000","last_unstale":"2024-02-23T00:31:09.916676+0000","last_undegraded":"2024-02-23T00:31:09.916676+0000","last_fullsized":"2024-02-23T00:31:09.916676+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:00.482621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:00.482621+0000","last_clean_scrub_stamp":"2024-02-23T00:31:00.482621+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"52.c","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.159593+0000","last_change":"2024-02-23T00:30:59.543518+0000","last_active":"2024-02-23T00:31:09.159593+0000","last_peered":"2024-02-23T00:31:09.159593+0000","last_clean":"2024-02-23T00:31:09.159593+0000","last_became_active":"2024-02-23T00:30:59.543257+0000","last_became_peered":"2024-02-23T00:30:59.543257+0000","last_unstale":"2024-02-23T00:31:09.159593+0000","last_undegraded":"2024-02-23T00:31:09.159593+0000","last_fullsized":"2024-02-23T00:31:09.159593+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"49.9","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130574+0000","last_change":"2024-02-23T00:30:56.050073+0000","last_active":"2024-02-23T00:31:09.130574+0000","last_peered":"2024-02-23T00:31:09.130574+0000","last_clean":"2024-02-23T00:31:09.130574+0000","last_became_active":"2024-02-23T00:30:56.049416+0000","last_became_peered":"2024-02-23T00:30:56.049416+0000","last_unstale":"2024-02-23T00:31:09.130574+0000","last_undegraded":"2024-02-23T00:31:09.130574+0000","last_fullsized":"2024-02-23T00:31:09.130574+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"44.14","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.159235+0000","last_change":"2024-02-23T00:30:54.981891+0000","last_active":"2024-02-23T00:31:09.159235+0000","last_peered":"2024-02-23T00:31:09.159235+0000","last_clean":"2024-02-23T00:31:09.159235+0000","last_became_active":"2024-02-23T00:30:54.981621+0000","last_became_peered":"2024-02-23T00:30:54.981621+0000","last_unstale":"2024-02-23T00:31:09.159235+0000","last_undegraded":"2024-02-23T00:31:09.159235+0000","last_fullsized":"2024-02-23T00:31:09.159235+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1e","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.217207+0000","last_change":"2024-02-23T00:30:52.780792+0000","last_active":"2024-02-23T00:31:09.217207+0000","last_peered":"2024-02-23T00:31:09.217207+0000","last_clean":"2024-02-23T00:31:09.217207+0000","last_became_active":"2024-02-23T00:30:52.779789+0000","last_became_peered":"2024-02-23T00:30:52.779789+0000","last_unstale":"2024-02-23T00:31:09.217207+0000","last_undegraded":"2024-02-23T00:31:09.217207+0000","last_fullsized":"2024-02-23T00:31:09.217207+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"63.8","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126766+0000","last_change":"2024-02-23T00:31:06.364034+0000","last_active":"2024-02-23T00:31:09.126766+0000","last_peered":"2024-02-23T00:31:09.126766+0000","last_clean":"2024-02-23T00:31:09.126766+0000","last_became_active":"2024-02-23T00:31:06.363442+0000","last_became_peered":"2024-02-23T00:31:06.363442+0000","last_unstale":"2024-02-23T00:31:09.126766+0000","last_undegraded":"2024-02-23T00:31:09.126766+0000","last_fullsized":"2024-02-23T00:31:09.126766+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.a","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.211820+0000","last_change":"2024-02-23T00:31:09.211820+0000","last_active":"2024-02-23T00:31:09.211820+0000","last_peered":"2024-02-23T00:31:09.211820+0000","last_clean":"2024-02-23T00:31:09.211820+0000","last_became_active":"2024-02-23T00:31:03.575402+0000","last_became_peered":"2024-02-23T00:31:03.575402+0000","last_unstale":"2024-02-23T00:31:09.211820+0000","last_undegraded":"2024-02-23T00:31:09.211820+0000","last_fullsized":"2024-02-23T00:31:09.211820+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.3","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120176+0000","last_change":"2024-02-23T00:30:59.529295+0000","last_active":"2024-02-23T00:31:09.120176+0000","last_peered":"2024-02-23T00:31:09.120176+0000","last_clean":"2024-02-23T00:31:09.120176+0000","last_became_active":"2024-02-23T00:30:59.528812+0000","last_became_peered":"2024-02-23T00:30:59.528812+0000","last_unstale":"2024-02-23T00:31:09.120176+0000","last_undegraded":"2024-02-23T00:31:09.120176+0000","last_fullsized":"2024-02-23T00:31:09.120176+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"49.6","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.188027+0000","last_change":"2024-02-23T00:30:56.034974+0000","last_active":"2024-02-23T00:31:09.188027+0000","last_peered":"2024-02-23T00:31:09.188027+0000","last_clean":"2024-02-23T00:31:09.188027+0000","last_became_active":"2024-02-23T00:30:56.034359+0000","last_became_peered":"2024-02-23T00:30:56.034359+0000","last_unstale":"2024-02-23T00:31:09.188027+0000","last_undegraded":"2024-02-23T00:31:09.188027+0000","last_fullsized":"2024-02-23T00:31:09.188027+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"44.1b","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.168935+0000","last_change":"2024-02-23T00:30:54.996818+0000","last_active":"2024-02-23T00:31:09.168935+0000","last_peered":"2024-02-23T00:31:09.168935+0000","last_clean":"2024-02-23T00:31:09.168935+0000","last_became_active":"2024-02-23T00:30:54.996510+0000","last_became_peered":"2024-02-23T00:30:54.996510+0000","last_unstale":"2024-02-23T00:31:09.168935+0000","last_undegraded":"2024-02-23T00:31:09.168935+0000","last_fullsized":"2024-02-23T00:31:09.168935+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122608+0000","last_change":"2024-02-23T00:30:52.766385+0000","last_active":"2024-02-23T00:31:09.122608+0000","last_peered":"2024-02-23T00:31:09.122608+0000","last_clean":"2024-02-23T00:31:09.122608+0000","last_became_active":"2024-02-23T00:30:52.766094+0000","last_became_peered":"2024-02-23T00:30:52.766094+0000","last_unstale":"2024-02-23T00:31:09.122608+0000","last_undegraded":"2024-02-23T00:31:09.122608+0000","last_fullsized":"2024-02-23T00:31:09.122608+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.9","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.171365+0000","last_change":"2024-02-23T00:31:06.356529+0000","last_active":"2024-02-23T00:31:09.171365+0000","last_peered":"2024-02-23T00:31:09.171365+0000","last_clean":"2024-02-23T00:31:09.171365+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:31:09.171365+0000","last_undegraded":"2024-02-23T00:31:09.171365+0000","last_fullsized":"2024-02-23T00:31:09.171365+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.b","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.158447+0000","last_change":"2024-02-23T00:31:09.158447+0000","last_active":"2024-02-23T00:31:09.158447+0000","last_peered":"2024-02-23T00:31:09.158447+0000","last_clean":"2024-02-23T00:31:09.158447+0000","last_became_active":"2024-02-23T00:31:03.611625+0000","last_became_peered":"2024-02-23T00:31:03.611625+0000","last_unstale":"2024-02-23T00:31:09.158447+0000","last_undegraded":"2024-02-23T00:31:09.158447+0000","last_fullsized":"2024-02-23T00:31:09.158447+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.2","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.148809+0000","last_change":"2024-02-23T00:30:59.531955+0000","last_active":"2024-02-23T00:31:09.148809+0000","last_peered":"2024-02-23T00:31:09.148809+0000","last_clean":"2024-02-23T00:31:09.148809+0000","last_became_active":"2024-02-23T00:30:59.531314+0000","last_became_peered":"2024-02-23T00:30:59.531314+0000","last_unstale":"2024-02-23T00:31:09.148809+0000","last_undegraded":"2024-02-23T00:31:09.148809+0000","last_fullsized":"2024-02-23T00:31:09.148809+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"49.7","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.192951+0000","last_change":"2024-02-23T00:30:56.060667+0000","last_active":"2024-02-23T00:31:09.192951+0000","last_peered":"2024-02-23T00:31:09.192951+0000","last_clean":"2024-02-23T00:31:09.192951+0000","last_became_active":"2024-02-23T00:30:56.060087+0000","last_became_peered":"2024-02-23T00:30:56.060087+0000","last_unstale":"2024-02-23T00:31:09.192951+0000","last_undegraded":"2024-02-23T00:31:09.192951+0000","last_fullsized":"2024-02-23T00:31:09.192951+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"44.1a","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125512+0000","last_change":"2024-02-23T00:30:56.049985+0000","last_active":"2024-02-23T00:31:09.125512+0000","last_peered":"2024-02-23T00:31:09.125512+0000","last_clean":"2024-02-23T00:31:09.125512+0000","last_became_active":"2024-02-23T00:30:56.049367+0000","last_became_peered":"2024-02-23T00:30:56.049367+0000","last_unstale":"2024-02-23T00:31:09.125512+0000","last_undegraded":"2024-02-23T00:31:09.125512+0000","last_fullsized":"2024-02-23T00:31:09.125512+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.10","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130707+0000","last_change":"2024-02-23T00:30:52.772757+0000","last_active":"2024-02-23T00:31:09.130707+0000","last_peered":"2024-02-23T00:31:09.130707+0000","last_clean":"2024-02-23T00:31:09.130707+0000","last_became_active":"2024-02-23T00:30:52.770301+0000","last_became_peered":"2024-02-23T00:30:52.770301+0000","last_unstale":"2024-02-23T00:31:09.130707+0000","last_undegraded":"2024-02-23T00:31:09.130707+0000","last_fullsized":"2024-02-23T00:31:09.130707+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.a","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122716+0000","last_change":"2024-02-23T00:31:06.313835+0000","last_active":"2024-02-23T00:31:09.122716+0000","last_peered":"2024-02-23T00:31:09.122716+0000","last_clean":"2024-02-23T00:31:09.122716+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:31:09.122716+0000","last_undegraded":"2024-02-23T00:31:09.122716+0000","last_fullsized":"2024-02-23T00:31:09.122716+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.8","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122512+0000","last_change":"2024-02-23T00:31:09.122512+0000","last_active":"2024-02-23T00:31:09.122512+0000","last_peered":"2024-02-23T00:31:09.122512+0000","last_clean":"2024-02-23T00:31:09.122512+0000","last_became_active":"2024-02-23T00:31:03.669219+0000","last_became_peered":"2024-02-23T00:31:03.669219+0000","last_unstale":"2024-02-23T00:31:09.122512+0000","last_undegraded":"2024-02-23T00:31:09.122512+0000","last_fullsized":"2024-02-23T00:31:09.122512+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.1","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131316+0000","last_change":"2024-02-23T00:30:59.530496+0000","last_active":"2024-02-23T00:31:09.131316+0000","last_peered":"2024-02-23T00:31:09.131316+0000","last_clean":"2024-02-23T00:31:09.131316+0000","last_became_active":"2024-02-23T00:30:59.528662+0000","last_became_peered":"2024-02-23T00:30:59.528662+0000","last_unstale":"2024-02-23T00:31:09.131316+0000","last_undegraded":"2024-02-23T00:31:09.131316+0000","last_fullsized":"2024-02-23T00:31:09.131316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"49.4","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.175873+0000","last_change":"2024-02-23T00:30:56.063766+0000","last_active":"2024-02-23T00:31:09.175873+0000","last_peered":"2024-02-23T00:31:09.175873+0000","last_clean":"2024-02-23T00:31:09.175873+0000","last_became_active":"2024-02-23T00:30:56.063037+0000","last_became_peered":"2024-02-23T00:30:56.063037+0000","last_unstale":"2024-02-23T00:31:09.175873+0000","last_undegraded":"2024-02-23T00:31:09.175873+0000","last_fullsized":"2024-02-23T00:31:09.175873+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"44.19","version":"114'1","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130794+0000","last_change":"2024-02-23T00:30:56.045492+0000","last_active":"2024-02-23T00:31:09.130794+0000","last_peered":"2024-02-23T00:31:09.130794+0000","last_clean":"2024-02-23T00:31:09.130794+0000","last_became_active":"2024-02-23T00:30:56.045158+0000","last_became_peered":"2024-02-23T00:30:56.045158+0000","last_unstale":"2024-02-23T00:31:09.130794+0000","last_undegraded":"2024-02-23T00:31:09.130794+0000","last_fullsized":"2024-02-23T00:31:09.130794+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4096,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.13","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121349+0000","last_change":"2024-02-23T00:30:52.722934+0000","last_active":"2024-02-23T00:31:09.121349+0000","last_peered":"2024-02-23T00:31:09.121349+0000","last_clean":"2024-02-23T00:31:09.121349+0000","last_became_active":"2024-02-23T00:30:52.722405+0000","last_became_peered":"2024-02-23T00:30:52.722405+0000","last_unstale":"2024-02-23T00:31:09.121349+0000","last_undegraded":"2024-02-23T00:31:09.121349+0000","last_fullsized":"2024-02-23T00:31:09.121349+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.b","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132586+0000","last_change":"2024-02-23T00:31:06.294911+0000","last_active":"2024-02-23T00:31:09.132586+0000","last_peered":"2024-02-23T00:31:09.132586+0000","last_clean":"2024-02-23T00:31:09.132586+0000","last_became_active":"2024-02-23T00:31:06.294667+0000","last_became_peered":"2024-02-23T00:31:06.294667+0000","last_unstale":"2024-02-23T00:31:09.132586+0000","last_undegraded":"2024-02-23T00:31:09.132586+0000","last_fullsized":"2024-02-23T00:31:09.132586+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.9","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.180748+0000","last_change":"2024-02-23T00:31:09.180748+0000","last_active":"2024-02-23T00:31:09.180748+0000","last_peered":"2024-02-23T00:31:09.180748+0000","last_clean":"2024-02-23T00:31:09.180748+0000","last_became_active":"2024-02-23T00:31:03.667544+0000","last_became_peered":"2024-02-23T00:31:03.667544+0000","last_unstale":"2024-02-23T00:31:09.180748+0000","last_undegraded":"2024-02-23T00:31:09.180748+0000","last_fullsized":"2024-02-23T00:31:09.180748+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.0","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.119629+0000","last_change":"2024-02-23T00:30:59.510751+0000","last_active":"2024-02-23T00:31:09.119629+0000","last_peered":"2024-02-23T00:31:09.119629+0000","last_clean":"2024-02-23T00:31:09.119629+0000","last_became_active":"2024-02-23T00:30:59.510436+0000","last_became_peered":"2024-02-23T00:30:59.510436+0000","last_unstale":"2024-02-23T00:31:09.119629+0000","last_undegraded":"2024-02-23T00:31:09.119629+0000","last_fullsized":"2024-02-23T00:31:09.119629+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"49.5","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.136092+0000","last_change":"2024-02-23T00:30:56.052536+0000","last_active":"2024-02-23T00:31:09.136092+0000","last_peered":"2024-02-23T00:31:09.136092+0000","last_clean":"2024-02-23T00:31:09.136092+0000","last_became_active":"2024-02-23T00:30:56.052106+0000","last_became_peered":"2024-02-23T00:30:56.052106+0000","last_unstale":"2024-02-23T00:31:09.136092+0000","last_undegraded":"2024-02-23T00:31:09.136092+0000","last_fullsized":"2024-02-23T00:31:09.136092+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"44.18","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121709+0000","last_change":"2024-02-23T00:30:55.984338+0000","last_active":"2024-02-23T00:31:09.121709+0000","last_peered":"2024-02-23T00:31:09.121709+0000","last_clean":"2024-02-23T00:31:09.121709+0000","last_became_active":"2024-02-23T00:30:55.983690+0000","last_became_peered":"2024-02-23T00:30:55.983690+0000","last_unstale":"2024-02-23T00:31:09.121709+0000","last_undegraded":"2024-02-23T00:31:09.121709+0000","last_fullsized":"2024-02-23T00:31:09.121709+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138423+0000","last_change":"2024-02-23T00:30:52.764195+0000","last_active":"2024-02-23T00:31:09.138423+0000","last_peered":"2024-02-23T00:31:09.138423+0000","last_clean":"2024-02-23T00:31:09.138423+0000","last_became_active":"2024-02-23T00:30:52.763903+0000","last_became_peered":"2024-02-23T00:30:52.763903+0000","last_unstale":"2024-02-23T00:31:09.138423+0000","last_undegraded":"2024-02-23T00:31:09.138423+0000","last_fullsized":"2024-02-23T00:31:09.138423+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.c","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129525+0000","last_change":"2024-02-23T00:31:06.234496+0000","last_active":"2024-02-23T00:31:09.129525+0000","last_peered":"2024-02-23T00:31:09.129525+0000","last_clean":"2024-02-23T00:31:09.129525+0000","last_became_active":"2024-02-23T00:31:06.233844+0000","last_became_peered":"2024-02-23T00:31:06.233844+0000","last_unstale":"2024-02-23T00:31:09.129525+0000","last_undegraded":"2024-02-23T00:31:09.129525+0000","last_fullsized":"2024-02-23T00:31:09.129525+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.e","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131726+0000","last_change":"2024-02-23T00:31:09.131726+0000","last_active":"2024-02-23T00:31:09.131726+0000","last_peered":"2024-02-23T00:31:09.131726+0000","last_clean":"2024-02-23T00:31:09.131726+0000","last_became_active":"2024-02-23T00:31:03.570897+0000","last_became_peered":"2024-02-23T00:31:03.570897+0000","last_unstale":"2024-02-23T00:31:09.131726+0000","last_undegraded":"2024-02-23T00:31:09.131726+0000","last_fullsized":"2024-02-23T00:31:09.131726+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.7","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135364+0000","last_change":"2024-02-23T00:30:59.538606+0000","last_active":"2024-02-23T00:31:09.135364+0000","last_peered":"2024-02-23T00:31:09.135364+0000","last_clean":"2024-02-23T00:31:09.135364+0000","last_became_active":"2024-02-23T00:30:59.537639+0000","last_became_peered":"2024-02-23T00:30:59.537639+0000","last_unstale":"2024-02-23T00:31:09.135364+0000","last_undegraded":"2024-02-23T00:31:09.135364+0000","last_fullsized":"2024-02-23T00:31:09.135364+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"49.2","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129065+0000","last_change":"2024-02-23T00:30:56.047979+0000","last_active":"2024-02-23T00:31:09.129065+0000","last_peered":"2024-02-23T00:31:09.129065+0000","last_clean":"2024-02-23T00:31:09.129065+0000","last_became_active":"2024-02-23T00:30:56.047594+0000","last_became_peered":"2024-02-23T00:30:56.047594+0000","last_unstale":"2024-02-23T00:31:09.129065+0000","last_undegraded":"2024-02-23T00:31:09.129065+0000","last_fullsized":"2024-02-23T00:31:09.129065+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"44.1f","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.193027+0000","last_change":"2024-02-23T00:30:54.989923+0000","last_active":"2024-02-23T00:31:09.193027+0000","last_peered":"2024-02-23T00:31:09.193027+0000","last_clean":"2024-02-23T00:31:09.193027+0000","last_became_active":"2024-02-23T00:30:54.989656+0000","last_became_peered":"2024-02-23T00:30:54.989656+0000","last_unstale":"2024-02-23T00:31:09.193027+0000","last_undegraded":"2024-02-23T00:31:09.193027+0000","last_fullsized":"2024-02-23T00:31:09.193027+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.15","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125180+0000","last_change":"2024-02-23T00:30:52.729544+0000","last_active":"2024-02-23T00:31:09.125180+0000","last_peered":"2024-02-23T00:31:09.125180+0000","last_clean":"2024-02-23T00:31:09.125180+0000","last_became_active":"2024-02-23T00:30:52.729222+0000","last_became_peered":"2024-02-23T00:30:52.729222+0000","last_unstale":"2024-02-23T00:31:09.125180+0000","last_undegraded":"2024-02-23T00:31:09.125180+0000","last_fullsized":"2024-02-23T00:31:09.125180+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.d","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.124194+0000","last_change":"2024-02-23T00:31:06.262505+0000","last_active":"2024-02-23T00:31:09.124194+0000","last_peered":"2024-02-23T00:31:09.124194+0000","last_clean":"2024-02-23T00:31:09.124194+0000","last_became_active":"2024-02-23T00:31:06.262251+0000","last_became_peered":"2024-02-23T00:31:06.262251+0000","last_unstale":"2024-02-23T00:31:09.124194+0000","last_undegraded":"2024-02-23T00:31:09.124194+0000","last_fullsized":"2024-02-23T00:31:09.124194+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.f","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.140493+0000","last_change":"2024-02-23T00:31:09.140493+0000","last_active":"2024-02-23T00:31:09.140493+0000","last_peered":"2024-02-23T00:31:09.140493+0000","last_clean":"2024-02-23T00:31:09.140493+0000","last_became_active":"2024-02-23T00:31:03.560364+0000","last_became_peered":"2024-02-23T00:31:03.560364+0000","last_unstale":"2024-02-23T00:31:09.140493+0000","last_undegraded":"2024-02-23T00:31:09.140493+0000","last_fullsized":"2024-02-23T00:31:09.140493+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.6","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.134950+0000","last_change":"2024-02-23T00:30:59.534155+0000","last_active":"2024-02-23T00:31:09.134950+0000","last_peered":"2024-02-23T00:31:09.134950+0000","last_clean":"2024-02-23T00:31:09.134950+0000","last_became_active":"2024-02-23T00:30:59.533442+0000","last_became_peered":"2024-02-23T00:30:59.533442+0000","last_unstale":"2024-02-23T00:31:09.134950+0000","last_undegraded":"2024-02-23T00:31:09.134950+0000","last_fullsized":"2024-02-23T00:31:09.134950+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"49.3","version":"106'3","reported_seq":27,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.195224+0000","last_change":"2024-02-23T00:30:56.037642+0000","last_active":"2024-02-23T00:31:09.195224+0000","last_peered":"2024-02-23T00:31:09.195224+0000","last_clean":"2024-02-23T00:31:09.195224+0000","last_became_active":"2024-02-23T00:30:56.036780+0000","last_became_peered":"2024-02-23T00:30:56.036780+0000","last_unstale":"2024-02-23T00:31:09.195224+0000","last_undegraded":"2024-02-23T00:31:09.195224+0000","last_fullsized":"2024-02-23T00:31:09.195224+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"44.1e","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.127566+0000","last_change":"2024-02-23T00:30:56.056667+0000","last_active":"2024-02-23T00:31:09.127566+0000","last_peered":"2024-02-23T00:31:09.127566+0000","last_clean":"2024-02-23T00:31:09.127566+0000","last_became_active":"2024-02-23T00:30:56.056209+0000","last_became_peered":"2024-02-23T00:30:56.056209+0000","last_unstale":"2024-02-23T00:31:09.127566+0000","last_undegraded":"2024-02-23T00:31:09.127566+0000","last_fullsized":"2024-02-23T00:31:09.127566+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.14","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132848+0000","last_change":"2024-02-23T00:30:52.757916+0000","last_active":"2024-02-23T00:31:09.132848+0000","last_peered":"2024-02-23T00:31:09.132848+0000","last_clean":"2024-02-23T00:31:09.132848+0000","last_became_active":"2024-02-23T00:30:52.757444+0000","last_became_peered":"2024-02-23T00:30:52.757444+0000","last_unstale":"2024-02-23T00:31:09.132848+0000","last_undegraded":"2024-02-23T00:31:09.132848+0000","last_fullsized":"2024-02-23T00:31:09.132848+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.e","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.182695+0000","last_change":"2024-02-23T00:31:06.369628+0000","last_active":"2024-02-23T00:31:09.182695+0000","last_peered":"2024-02-23T00:31:09.182695+0000","last_clean":"2024-02-23T00:31:09.182695+0000","last_became_active":"2024-02-23T00:31:06.356847+0000","last_became_peered":"2024-02-23T00:31:06.356847+0000","last_unstale":"2024-02-23T00:31:09.182695+0000","last_undegraded":"2024-02-23T00:31:09.182695+0000","last_fullsized":"2024-02-23T00:31:09.182695+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.c","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125867+0000","last_change":"2024-02-23T00:31:09.125867+0000","last_active":"2024-02-23T00:31:09.125867+0000","last_peered":"2024-02-23T00:31:09.125867+0000","last_clean":"2024-02-23T00:31:09.125867+0000","last_became_active":"2024-02-23T00:31:03.630978+0000","last_became_peered":"2024-02-23T00:31:03.630978+0000","last_unstale":"2024-02-23T00:31:09.125867+0000","last_undegraded":"2024-02-23T00:31:09.125867+0000","last_fullsized":"2024-02-23T00:31:09.125867+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.5","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.183836+0000","last_change":"2024-02-23T00:30:59.542559+0000","last_active":"2024-02-23T00:31:09.183836+0000","last_peered":"2024-02-23T00:31:09.183836+0000","last_clean":"2024-02-23T00:31:09.183836+0000","last_became_active":"2024-02-23T00:30:59.542265+0000","last_became_peered":"2024-02-23T00:30:59.542265+0000","last_unstale":"2024-02-23T00:31:09.183836+0000","last_undegraded":"2024-02-23T00:31:09.183836+0000","last_fullsized":"2024-02-23T00:31:09.183836+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"49.0","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121497+0000","last_change":"2024-02-23T00:30:56.033794+0000","last_active":"2024-02-23T00:31:09.121497+0000","last_peered":"2024-02-23T00:31:09.121497+0000","last_clean":"2024-02-23T00:31:09.121497+0000","last_became_active":"2024-02-23T00:30:56.033189+0000","last_became_peered":"2024-02-23T00:30:56.033189+0000","last_unstale":"2024-02-23T00:31:09.121497+0000","last_undegraded":"2024-02-23T00:31:09.121497+0000","last_fullsized":"2024-02-23T00:31:09.121497+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"44.1d","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.137938+0000","last_change":"2024-02-23T00:30:56.048744+0000","last_active":"2024-02-23T00:31:09.137938+0000","last_peered":"2024-02-23T00:31:09.137938+0000","last_clean":"2024-02-23T00:31:09.137938+0000","last_became_active":"2024-02-23T00:30:56.048468+0000","last_became_peered":"2024-02-23T00:30:56.048468+0000","last_unstale":"2024-02-23T00:31:09.137938+0000","last_undegraded":"2024-02-23T00:31:09.137938+0000","last_fullsized":"2024-02-23T00:31:09.137938+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135598+0000","last_change":"2024-02-23T00:30:52.751376+0000","last_active":"2024-02-23T00:31:09.135598+0000","last_peered":"2024-02-23T00:31:09.135598+0000","last_clean":"2024-02-23T00:31:09.135598+0000","last_became_active":"2024-02-23T00:30:52.750976+0000","last_became_peered":"2024-02-23T00:30:52.750976+0000","last_unstale":"2024-02-23T00:31:09.135598+0000","last_undegraded":"2024-02-23T00:31:09.135598+0000","last_fullsized":"2024-02-23T00:31:09.135598+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.f","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.184758+0000","last_change":"2024-02-23T00:31:06.364880+0000","last_active":"2024-02-23T00:31:09.184758+0000","last_peered":"2024-02-23T00:31:09.184758+0000","last_clean":"2024-02-23T00:31:09.184758+0000","last_became_active":"2024-02-23T00:31:06.364609+0000","last_became_peered":"2024-02-23T00:31:06.364609+0000","last_unstale":"2024-02-23T00:31:09.184758+0000","last_undegraded":"2024-02-23T00:31:09.184758+0000","last_fullsized":"2024-02-23T00:31:09.184758+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.d","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128007+0000","last_change":"2024-02-23T00:31:09.128007+0000","last_active":"2024-02-23T00:31:09.128007+0000","last_peered":"2024-02-23T00:31:09.128007+0000","last_clean":"2024-02-23T00:31:09.128007+0000","last_became_active":"2024-02-23T00:31:03.583989+0000","last_became_peered":"2024-02-23T00:31:03.583989+0000","last_unstale":"2024-02-23T00:31:09.128007+0000","last_undegraded":"2024-02-23T00:31:09.128007+0000","last_fullsized":"2024-02-23T00:31:09.128007+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.4","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129849+0000","last_change":"2024-02-23T00:30:59.531311+0000","last_active":"2024-02-23T00:31:09.129849+0000","last_peered":"2024-02-23T00:31:09.129849+0000","last_clean":"2024-02-23T00:31:09.129849+0000","last_became_active":"2024-02-23T00:30:59.531010+0000","last_became_peered":"2024-02-23T00:30:59.531010+0000","last_unstale":"2024-02-23T00:31:09.129849+0000","last_undegraded":"2024-02-23T00:31:09.129849+0000","last_fullsized":"2024-02-23T00:31:09.129849+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"49.1","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122274+0000","last_change":"2024-02-23T00:30:56.043975+0000","last_active":"2024-02-23T00:31:09.122274+0000","last_peered":"2024-02-23T00:31:09.122274+0000","last_clean":"2024-02-23T00:31:09.122274+0000","last_became_active":"2024-02-23T00:30:56.043709+0000","last_became_peered":"2024-02-23T00:30:56.043709+0000","last_unstale":"2024-02-23T00:31:09.122274+0000","last_undegraded":"2024-02-23T00:31:09.122274+0000","last_fullsized":"2024-02-23T00:31:09.122274+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"44.1c","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174386+0000","last_change":"2024-02-23T00:30:54.996098+0000","last_active":"2024-02-23T00:31:09.174386+0000","last_peered":"2024-02-23T00:31:09.174386+0000","last_clean":"2024-02-23T00:31:09.174386+0000","last_became_active":"2024-02-23T00:30:54.995752+0000","last_became_peered":"2024-02-23T00:30:54.995752+0000","last_unstale":"2024-02-23T00:31:09.174386+0000","last_undegraded":"2024-02-23T00:31:09.174386+0000","last_fullsized":"2024-02-23T00:31:09.174386+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.217246+0000","last_change":"2024-02-23T00:30:52.780962+0000","last_active":"2024-02-23T00:31:09.217246+0000","last_peered":"2024-02-23T00:31:09.217246+0000","last_clean":"2024-02-23T00:31:09.217246+0000","last_became_active":"2024-02-23T00:30:52.780128+0000","last_became_peered":"2024-02-23T00:30:52.780128+0000","last_unstale":"2024-02-23T00:31:09.217246+0000","last_undegraded":"2024-02-23T00:31:09.217246+0000","last_fullsized":"2024-02-23T00:31:09.217246+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"63.10","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.146759+0000","last_change":"2024-02-23T00:31:06.354463+0000","last_active":"2024-02-23T00:31:09.146759+0000","last_peered":"2024-02-23T00:31:09.146759+0000","last_clean":"2024-02-23T00:31:09.146759+0000","last_became_active":"2024-02-23T00:31:06.342164+0000","last_became_peered":"2024-02-23T00:31:06.342164+0000","last_unstale":"2024-02-23T00:31:09.146759+0000","last_undegraded":"2024-02-23T00:31:09.146759+0000","last_fullsized":"2024-02-23T00:31:09.146759+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.12","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132005+0000","last_change":"2024-02-23T00:31:09.132005+0000","last_active":"2024-02-23T00:31:09.132005+0000","last_peered":"2024-02-23T00:31:09.132005+0000","last_clean":"2024-02-23T00:31:09.132005+0000","last_became_active":"2024-02-23T00:31:03.660680+0000","last_became_peered":"2024-02-23T00:31:03.660680+0000","last_unstale":"2024-02-23T00:31:09.132005+0000","last_undegraded":"2024-02-23T00:31:09.132005+0000","last_fullsized":"2024-02-23T00:31:09.132005+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.1b","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.156511+0000","last_change":"2024-02-23T00:30:59.531214+0000","last_active":"2024-02-23T00:31:09.156511+0000","last_peered":"2024-02-23T00:31:09.156511+0000","last_clean":"2024-02-23T00:31:09.156511+0000","last_became_active":"2024-02-23T00:30:59.530915+0000","last_became_peered":"2024-02-23T00:30:59.530915+0000","last_unstale":"2024-02-23T00:31:09.156511+0000","last_undegraded":"2024-02-23T00:31:09.156511+0000","last_fullsized":"2024-02-23T00:31:09.156511+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"49.1e","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128922+0000","last_change":"2024-02-23T00:30:56.022856+0000","last_active":"2024-02-23T00:31:09.128922+0000","last_peered":"2024-02-23T00:31:09.128922+0000","last_clean":"2024-02-23T00:31:09.128922+0000","last_became_active":"2024-02-23T00:30:56.022579+0000","last_became_peered":"2024-02-23T00:30:56.022579+0000","last_unstale":"2024-02-23T00:31:09.128922+0000","last_undegraded":"2024-02-23T00:31:09.128922+0000","last_fullsized":"2024-02-23T00:31:09.128922+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"44.3","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131508+0000","last_change":"2024-02-23T00:30:54.998724+0000","last_active":"2024-02-23T00:31:09.131508+0000","last_peered":"2024-02-23T00:31:09.131508+0000","last_clean":"2024-02-23T00:31:09.131508+0000","last_became_active":"2024-02-23T00:30:54.998446+0000","last_became_peered":"2024-02-23T00:30:54.998446+0000","last_unstale":"2024-02-23T00:31:09.131508+0000","last_undegraded":"2024-02-23T00:31:09.131508+0000","last_fullsized":"2024-02-23T00:31:09.131508+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.162637+0000","last_change":"2024-02-23T00:30:52.778331+0000","last_active":"2024-02-23T00:31:09.162637+0000","last_peered":"2024-02-23T00:31:09.162637+0000","last_clean":"2024-02-23T00:31:09.162637+0000","last_became_active":"2024-02-23T00:30:52.778082+0000","last_became_peered":"2024-02-23T00:30:52.778082+0000","last_unstale":"2024-02-23T00:31:09.162637+0000","last_undegraded":"2024-02-23T00:31:09.162637+0000","last_fullsized":"2024-02-23T00:31:09.162637+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.11","version":"117'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122024+0000","last_change":"2024-02-23T00:31:06.330557+0000","last_active":"2024-02-23T00:31:09.122024+0000","last_peered":"2024-02-23T00:31:09.122024+0000","last_clean":"2024-02-23T00:31:09.122024+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:31:09.122024+0000","last_undegraded":"2024-02-23T00:31:09.122024+0000","last_fullsized":"2024-02-23T00:31:09.122024+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.13","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135971+0000","last_change":"2024-02-23T00:31:09.135971+0000","last_active":"2024-02-23T00:31:09.135971+0000","last_peered":"2024-02-23T00:31:09.135971+0000","last_clean":"2024-02-23T00:31:09.135971+0000","last_became_active":"2024-02-23T00:31:03.572018+0000","last_became_peered":"2024-02-23T00:31:03.572018+0000","last_unstale":"2024-02-23T00:31:09.135971+0000","last_undegraded":"2024-02-23T00:31:09.135971+0000","last_fullsized":"2024-02-23T00:31:09.135971+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.1a","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.172301+0000","last_change":"2024-02-23T00:30:59.539212+0000","last_active":"2024-02-23T00:31:09.172301+0000","last_peered":"2024-02-23T00:31:09.172301+0000","last_clean":"2024-02-23T00:31:09.172301+0000","last_became_active":"2024-02-23T00:30:59.536523+0000","last_became_peered":"2024-02-23T00:30:59.536523+0000","last_unstale":"2024-02-23T00:31:09.172301+0000","last_undegraded":"2024-02-23T00:31:09.172301+0000","last_fullsized":"2024-02-23T00:31:09.172301+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"49.1f","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135478+0000","last_change":"2024-02-23T00:30:56.054187+0000","last_active":"2024-02-23T00:31:09.135478+0000","last_peered":"2024-02-23T00:31:09.135478+0000","last_clean":"2024-02-23T00:31:09.135478+0000","last_became_active":"2024-02-23T00:30:56.053854+0000","last_became_peered":"2024-02-23T00:30:56.053854+0000","last_unstale":"2024-02-23T00:31:09.135478+0000","last_undegraded":"2024-02-23T00:31:09.135478+0000","last_fullsized":"2024-02-23T00:31:09.135478+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"44.2","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.149034+0000","last_change":"2024-02-23T00:30:54.989904+0000","last_active":"2024-02-23T00:31:09.149034+0000","last_peered":"2024-02-23T00:31:09.149034+0000","last_clean":"2024-02-23T00:31:09.149034+0000","last_became_active":"2024-02-23T00:30:54.989398+0000","last_became_peered":"2024-02-23T00:30:54.989398+0000","last_unstale":"2024-02-23T00:31:09.149034+0000","last_undegraded":"2024-02-23T00:31:09.149034+0000","last_fullsized":"2024-02-23T00:31:09.149034+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131292+0000","last_change":"2024-02-23T00:30:52.771046+0000","last_active":"2024-02-23T00:31:09.131292+0000","last_peered":"2024-02-23T00:31:09.131292+0000","last_clean":"2024-02-23T00:31:09.131292+0000","last_became_active":"2024-02-23T00:30:52.770745+0000","last_became_peered":"2024-02-23T00:30:52.770745+0000","last_unstale":"2024-02-23T00:31:09.131292+0000","last_undegraded":"2024-02-23T00:31:09.131292+0000","last_fullsized":"2024-02-23T00:31:09.131292+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.12","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.134802+0000","last_change":"2024-02-23T00:31:06.281866+0000","last_active":"2024-02-23T00:31:09.134802+0000","last_peered":"2024-02-23T00:31:09.134802+0000","last_clean":"2024-02-23T00:31:09.134802+0000","last_became_active":"2024-02-23T00:31:06.281619+0000","last_became_peered":"2024-02-23T00:31:06.281619+0000","last_unstale":"2024-02-23T00:31:09.134802+0000","last_undegraded":"2024-02-23T00:31:09.134802+0000","last_fullsized":"2024-02-23T00:31:09.134802+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.10","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132149+0000","last_change":"2024-02-23T00:31:09.132149+0000","last_active":"2024-02-23T00:31:09.132149+0000","last_peered":"2024-02-23T00:31:09.132149+0000","last_clean":"2024-02-23T00:31:09.132149+0000","last_became_active":"2024-02-23T00:31:03.627906+0000","last_became_peered":"2024-02-23T00:31:03.627906+0000","last_unstale":"2024-02-23T00:31:09.132149+0000","last_undegraded":"2024-02-23T00:31:09.132149+0000","last_fullsized":"2024-02-23T00:31:09.132149+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.19","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121268+0000","last_change":"2024-02-23T00:30:59.533403+0000","last_active":"2024-02-23T00:31:09.121268+0000","last_peered":"2024-02-23T00:31:09.121268+0000","last_clean":"2024-02-23T00:31:09.121268+0000","last_became_active":"2024-02-23T00:30:59.533051+0000","last_became_peered":"2024-02-23T00:30:59.533051+0000","last_unstale":"2024-02-23T00:31:09.121268+0000","last_undegraded":"2024-02-23T00:31:09.121268+0000","last_fullsized":"2024-02-23T00:31:09.121268+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"49.1c","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.175799+0000","last_change":"2024-02-23T00:30:56.056330+0000","last_active":"2024-02-23T00:31:09.175799+0000","last_peered":"2024-02-23T00:31:09.175799+0000","last_clean":"2024-02-23T00:31:09.175799+0000","last_became_active":"2024-02-23T00:30:56.056136+0000","last_became_peered":"2024-02-23T00:30:56.056136+0000","last_unstale":"2024-02-23T00:31:09.175799+0000","last_undegraded":"2024-02-23T00:31:09.175799+0000","last_fullsized":"2024-02-23T00:31:09.175799+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"44.1","version":"115'2","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122413+0000","last_change":"2024-02-23T00:30:56.054512+0000","last_active":"2024-02-23T00:31:09.122413+0000","last_peered":"2024-02-23T00:31:09.122413+0000","last_clean":"2024-02-23T00:31:09.122413+0000","last_became_active":"2024-02-23T00:30:56.054208+0000","last_became_peered":"2024-02-23T00:30:56.054208+0000","last_unstale":"2024-02-23T00:31:09.122413+0000","last_undegraded":"2024-02-23T00:31:09.122413+0000","last_fullsized":"2024-02-23T00:31:09.122413+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8192,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.b","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120639+0000","last_change":"2024-02-23T00:30:52.751155+0000","last_active":"2024-02-23T00:31:09.120639+0000","last_peered":"2024-02-23T00:31:09.120639+0000","last_clean":"2024-02-23T00:31:09.120639+0000","last_became_active":"2024-02-23T00:30:52.750861+0000","last_became_peered":"2024-02-23T00:30:52.750861+0000","last_unstale":"2024-02-23T00:31:09.120639+0000","last_undegraded":"2024-02-23T00:31:09.120639+0000","last_fullsized":"2024-02-23T00:31:09.120639+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.13","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132518+0000","last_change":"2024-02-23T00:31:06.231713+0000","last_active":"2024-02-23T00:31:09.132518+0000","last_peered":"2024-02-23T00:31:09.132518+0000","last_clean":"2024-02-23T00:31:09.132518+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:31:09.132518+0000","last_undegraded":"2024-02-23T00:31:09.132518+0000","last_fullsized":"2024-02-23T00:31:09.132518+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.11","version":"119'1","reported_seq":24,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.166966+0000","last_change":"2024-02-23T00:31:09.131585+0000","last_active":"2024-02-23T00:31:10.166966+0000","last_peered":"2024-02-23T00:31:10.166966+0000","last_clean":"2024-02-23T00:31:10.166966+0000","last_became_active":"2024-02-23T00:31:03.622659+0000","last_became_peered":"2024-02-23T00:31:03.622659+0000","last_unstale":"2024-02-23T00:31:10.166966+0000","last_undegraded":"2024-02-23T00:31:10.166966+0000","last_fullsized":"2024-02-23T00:31:10.166966+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.18","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.158388+0000","last_change":"2024-02-23T00:30:59.545172+0000","last_active":"2024-02-23T00:31:09.158388+0000","last_peered":"2024-02-23T00:31:09.158388+0000","last_clean":"2024-02-23T00:31:09.158388+0000","last_became_active":"2024-02-23T00:30:59.544890+0000","last_became_peered":"2024-02-23T00:30:59.544890+0000","last_unstale":"2024-02-23T00:31:09.158388+0000","last_undegraded":"2024-02-23T00:31:09.158388+0000","last_fullsized":"2024-02-23T00:31:09.158388+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"49.1d","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129245+0000","last_change":"2024-02-23T00:30:56.031398+0000","last_active":"2024-02-23T00:31:09.129245+0000","last_peered":"2024-02-23T00:31:09.129245+0000","last_clean":"2024-02-23T00:31:09.129245+0000","last_became_active":"2024-02-23T00:30:56.031101+0000","last_became_peered":"2024-02-23T00:30:56.031101+0000","last_unstale":"2024-02-23T00:31:09.129245+0000","last_undegraded":"2024-02-23T00:31:09.129245+0000","last_fullsized":"2024-02-23T00:31:09.129245+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"44.0","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.119932+0000","last_change":"2024-02-23T00:30:56.028941+0000","last_active":"2024-02-23T00:31:09.119932+0000","last_peered":"2024-02-23T00:31:09.119932+0000","last_clean":"2024-02-23T00:31:09.119932+0000","last_became_active":"2024-02-23T00:30:56.028635+0000","last_became_peered":"2024-02-23T00:30:56.028635+0000","last_unstale":"2024-02-23T00:31:09.119932+0000","last_undegraded":"2024-02-23T00:31:09.119932+0000","last_fullsized":"2024-02-23T00:31:09.119932+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125637+0000","last_change":"2024-02-23T00:30:52.774746+0000","last_active":"2024-02-23T00:31:09.125637+0000","last_peered":"2024-02-23T00:31:09.125637+0000","last_clean":"2024-02-23T00:31:09.125637+0000","last_became_active":"2024-02-23T00:30:52.769195+0000","last_became_peered":"2024-02-23T00:30:52.769195+0000","last_unstale":"2024-02-23T00:31:09.125637+0000","last_undegraded":"2024-02-23T00:31:09.125637+0000","last_fullsized":"2024-02-23T00:31:09.125637+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.14","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129525+0000","last_change":"2024-02-23T00:31:06.234864+0000","last_active":"2024-02-23T00:31:09.129525+0000","last_peered":"2024-02-23T00:31:09.129525+0000","last_clean":"2024-02-23T00:31:09.129525+0000","last_became_active":"2024-02-23T00:31:06.234521+0000","last_became_peered":"2024-02-23T00:31:06.234521+0000","last_unstale":"2024-02-23T00:31:09.129525+0000","last_undegraded":"2024-02-23T00:31:09.129525+0000","last_fullsized":"2024-02-23T00:31:09.129525+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.16","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.134703+0000","last_change":"2024-02-23T00:31:09.134703+0000","last_active":"2024-02-23T00:31:09.134703+0000","last_peered":"2024-02-23T00:31:09.134703+0000","last_clean":"2024-02-23T00:31:09.134703+0000","last_became_active":"2024-02-23T00:31:03.569920+0000","last_became_peered":"2024-02-23T00:31:03.569920+0000","last_unstale":"2024-02-23T00:31:09.134703+0000","last_undegraded":"2024-02-23T00:31:09.134703+0000","last_fullsized":"2024-02-23T00:31:09.134703+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.1f","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.160721+0000","last_change":"2024-02-23T00:30:59.527593+0000","last_active":"2024-02-23T00:31:09.160721+0000","last_peered":"2024-02-23T00:31:09.160721+0000","last_clean":"2024-02-23T00:31:09.160721+0000","last_became_active":"2024-02-23T00:30:59.526959+0000","last_became_peered":"2024-02-23T00:30:59.526959+0000","last_unstale":"2024-02-23T00:31:09.160721+0000","last_undegraded":"2024-02-23T00:31:09.160721+0000","last_fullsized":"2024-02-23T00:31:09.160721+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"49.1a","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125371+0000","last_change":"2024-02-23T00:30:56.038449+0000","last_active":"2024-02-23T00:31:09.125371+0000","last_peered":"2024-02-23T00:31:09.125371+0000","last_clean":"2024-02-23T00:31:09.125371+0000","last_became_active":"2024-02-23T00:30:56.038151+0000","last_became_peered":"2024-02-23T00:30:56.038151+0000","last_unstale":"2024-02-23T00:31:09.125371+0000","last_undegraded":"2024-02-23T00:31:09.125371+0000","last_fullsized":"2024-02-23T00:31:09.125371+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"44.7","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.185144+0000","last_change":"2024-02-23T00:30:54.973574+0000","last_active":"2024-02-23T00:31:09.185144+0000","last_peered":"2024-02-23T00:31:09.185144+0000","last_clean":"2024-02-23T00:31:09.185144+0000","last_became_active":"2024-02-23T00:30:54.973081+0000","last_became_peered":"2024-02-23T00:30:54.973081+0000","last_unstale":"2024-02-23T00:31:09.185144+0000","last_undegraded":"2024-02-23T00:31:09.185144+0000","last_fullsized":"2024-02-23T00:31:09.185144+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.d","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129007+0000","last_change":"2024-02-23T00:30:52.774871+0000","last_active":"2024-02-23T00:31:09.129007+0000","last_peered":"2024-02-23T00:31:09.129007+0000","last_clean":"2024-02-23T00:31:09.129007+0000","last_became_active":"2024-02-23T00:30:52.773418+0000","last_became_peered":"2024-02-23T00:30:52.773418+0000","last_unstale":"2024-02-23T00:31:09.129007+0000","last_undegraded":"2024-02-23T00:31:09.129007+0000","last_fullsized":"2024-02-23T00:31:09.129007+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.15","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128322+0000","last_change":"2024-02-23T00:31:06.229865+0000","last_active":"2024-02-23T00:31:09.128322+0000","last_peered":"2024-02-23T00:31:09.128322+0000","last_clean":"2024-02-23T00:31:09.128322+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:31:09.128322+0000","last_undegraded":"2024-02-23T00:31:09.128322+0000","last_fullsized":"2024-02-23T00:31:09.128322+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.17","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.230139+0000","last_change":"2024-02-23T00:31:09.230139+0000","last_active":"2024-02-23T00:31:09.230139+0000","last_peered":"2024-02-23T00:31:09.230139+0000","last_clean":"2024-02-23T00:31:09.230139+0000","last_became_active":"2024-02-23T00:31:03.662107+0000","last_became_peered":"2024-02-23T00:31:03.662107+0000","last_unstale":"2024-02-23T00:31:09.230139+0000","last_undegraded":"2024-02-23T00:31:09.230139+0000","last_fullsized":"2024-02-23T00:31:09.230139+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.1e","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130952+0000","last_change":"2024-02-23T00:30:59.527074+0000","last_active":"2024-02-23T00:31:09.130952+0000","last_peered":"2024-02-23T00:31:09.130952+0000","last_clean":"2024-02-23T00:31:09.130952+0000","last_became_active":"2024-02-23T00:30:59.526816+0000","last_became_peered":"2024-02-23T00:30:59.526816+0000","last_unstale":"2024-02-23T00:31:09.130952+0000","last_undegraded":"2024-02-23T00:31:09.130952+0000","last_fullsized":"2024-02-23T00:31:09.130952+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"49.1b","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129865+0000","last_change":"2024-02-23T00:30:56.036897+0000","last_active":"2024-02-23T00:31:09.129865+0000","last_peered":"2024-02-23T00:31:09.129865+0000","last_clean":"2024-02-23T00:31:09.129865+0000","last_became_active":"2024-02-23T00:30:56.036587+0000","last_became_peered":"2024-02-23T00:30:56.036587+0000","last_unstale":"2024-02-23T00:31:09.129865+0000","last_undegraded":"2024-02-23T00:31:09.129865+0000","last_fullsized":"2024-02-23T00:31:09.129865+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"44.6","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.127828+0000","last_change":"2024-02-23T00:30:56.048819+0000","last_active":"2024-02-23T00:31:09.127828+0000","last_peered":"2024-02-23T00:31:09.127828+0000","last_clean":"2024-02-23T00:31:09.127828+0000","last_became_active":"2024-02-23T00:30:56.048364+0000","last_became_peered":"2024-02-23T00:30:56.048364+0000","last_unstale":"2024-02-23T00:31:09.127828+0000","last_undegraded":"2024-02-23T00:31:09.127828+0000","last_fullsized":"2024-02-23T00:31:09.127828+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.159165+0000","last_change":"2024-02-23T00:30:52.740833+0000","last_active":"2024-02-23T00:31:09.159165+0000","last_peered":"2024-02-23T00:31:09.159165+0000","last_clean":"2024-02-23T00:31:09.159165+0000","last_became_active":"2024-02-23T00:30:52.740595+0000","last_became_peered":"2024-02-23T00:30:52.740595+0000","last_unstale":"2024-02-23T00:31:09.159165+0000","last_undegraded":"2024-02-23T00:31:09.159165+0000","last_fullsized":"2024-02-23T00:31:09.159165+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.16","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128547+0000","last_change":"2024-02-23T00:31:06.367333+0000","last_active":"2024-02-23T00:31:09.128547+0000","last_peered":"2024-02-23T00:31:09.128547+0000","last_clean":"2024-02-23T00:31:09.128547+0000","last_became_active":"2024-02-23T00:31:06.367069+0000","last_became_peered":"2024-02-23T00:31:06.367069+0000","last_unstale":"2024-02-23T00:31:09.128547+0000","last_undegraded":"2024-02-23T00:31:09.128547+0000","last_fullsized":"2024-02-23T00:31:09.128547+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.14","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.200486+0000","last_change":"2024-02-23T00:31:09.200486+0000","last_active":"2024-02-23T00:31:09.200486+0000","last_peered":"2024-02-23T00:31:09.200486+0000","last_clean":"2024-02-23T00:31:09.200486+0000","last_became_active":"2024-02-23T00:31:03.661950+0000","last_became_peered":"2024-02-23T00:31:03.661950+0000","last_unstale":"2024-02-23T00:31:09.200486+0000","last_undegraded":"2024-02-23T00:31:09.200486+0000","last_fullsized":"2024-02-23T00:31:09.200486+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.1d","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.210741+0000","last_change":"2024-02-23T00:30:59.539648+0000","last_active":"2024-02-23T00:31:09.210741+0000","last_peered":"2024-02-23T00:31:09.210741+0000","last_clean":"2024-02-23T00:31:09.210741+0000","last_became_active":"2024-02-23T00:30:59.539333+0000","last_became_peered":"2024-02-23T00:30:59.539333+0000","last_unstale":"2024-02-23T00:31:09.210741+0000","last_undegraded":"2024-02-23T00:31:09.210741+0000","last_fullsized":"2024-02-23T00:31:09.210741+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"49.18","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131191+0000","last_change":"2024-02-23T00:30:56.045553+0000","last_active":"2024-02-23T00:31:09.131191+0000","last_peered":"2024-02-23T00:31:09.131191+0000","last_clean":"2024-02-23T00:31:09.131191+0000","last_became_active":"2024-02-23T00:30:56.041362+0000","last_became_peered":"2024-02-23T00:30:56.041362+0000","last_unstale":"2024-02-23T00:31:09.131191+0000","last_undegraded":"2024-02-23T00:31:09.131191+0000","last_fullsized":"2024-02-23T00:31:09.131191+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"44.5","version":"110'2","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126409+0000","last_change":"2024-02-23T00:30:56.047816+0000","last_active":"2024-02-23T00:31:09.126409+0000","last_peered":"2024-02-23T00:31:09.126409+0000","last_clean":"2024-02-23T00:31:09.126409+0000","last_became_active":"2024-02-23T00:30:56.046934+0000","last_became_peered":"2024-02-23T00:30:56.046934+0000","last_unstale":"2024-02-23T00:31:09.126409+0000","last_undegraded":"2024-02-23T00:31:09.126409+0000","last_fullsized":"2024-02-23T00:31:09.126409+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8192,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.f","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.193102+0000","last_change":"2024-02-23T00:30:52.735680+0000","last_active":"2024-02-23T00:31:09.193102+0000","last_peered":"2024-02-23T00:31:09.193102+0000","last_clean":"2024-02-23T00:31:09.193102+0000","last_became_active":"2024-02-23T00:30:52.735388+0000","last_became_peered":"2024-02-23T00:30:52.735388+0000","last_unstale":"2024-02-23T00:31:09.193102+0000","last_undegraded":"2024-02-23T00:31:09.193102+0000","last_fullsized":"2024-02-23T00:31:09.193102+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.17","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122326+0000","last_change":"2024-02-23T00:31:06.320348+0000","last_active":"2024-02-23T00:31:09.122326+0000","last_peered":"2024-02-23T00:31:09.122326+0000","last_clean":"2024-02-23T00:31:09.122326+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:31:09.122326+0000","last_undegraded":"2024-02-23T00:31:09.122326+0000","last_fullsized":"2024-02-23T00:31:09.122326+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.15","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.139060+0000","last_change":"2024-02-23T00:31:09.139060+0000","last_active":"2024-02-23T00:31:09.139060+0000","last_peered":"2024-02-23T00:31:09.139060+0000","last_clean":"2024-02-23T00:31:09.139060+0000","last_became_active":"2024-02-23T00:31:03.612941+0000","last_became_peered":"2024-02-23T00:31:03.612941+0000","last_unstale":"2024-02-23T00:31:09.139060+0000","last_undegraded":"2024-02-23T00:31:09.139060+0000","last_fullsized":"2024-02-23T00:31:09.139060+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.1c","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.175663+0000","last_change":"2024-02-23T00:30:59.528227+0000","last_active":"2024-02-23T00:31:09.175663+0000","last_peered":"2024-02-23T00:31:09.175663+0000","last_clean":"2024-02-23T00:31:09.175663+0000","last_became_active":"2024-02-23T00:30:59.527923+0000","last_became_peered":"2024-02-23T00:30:59.527923+0000","last_unstale":"2024-02-23T00:31:09.175663+0000","last_undegraded":"2024-02-23T00:31:09.175663+0000","last_fullsized":"2024-02-23T00:31:09.175663+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"49.19","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.127051+0000","last_change":"2024-02-23T00:30:56.046567+0000","last_active":"2024-02-23T00:31:09.127051+0000","last_peered":"2024-02-23T00:31:09.127051+0000","last_clean":"2024-02-23T00:31:09.127051+0000","last_became_active":"2024-02-23T00:30:56.046267+0000","last_became_peered":"2024-02-23T00:30:56.046267+0000","last_unstale":"2024-02-23T00:31:09.127051+0000","last_undegraded":"2024-02-23T00:31:09.127051+0000","last_fullsized":"2024-02-23T00:31:09.127051+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"44.4","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.199680+0000","last_change":"2024-02-23T00:30:54.998722+0000","last_active":"2024-02-23T00:31:09.199680+0000","last_peered":"2024-02-23T00:31:09.199680+0000","last_clean":"2024-02-23T00:31:09.199680+0000","last_became_active":"2024-02-23T00:30:54.998433+0000","last_became_peered":"2024-02-23T00:30:54.998433+0000","last_unstale":"2024-02-23T00:31:09.199680+0000","last_undegraded":"2024-02-23T00:31:09.199680+0000","last_fullsized":"2024-02-23T00:31:09.199680+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.e","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.127710+0000","last_change":"2024-02-23T00:30:52.724963+0000","last_active":"2024-02-23T00:31:09.127710+0000","last_peered":"2024-02-23T00:31:09.127710+0000","last_clean":"2024-02-23T00:31:09.127710+0000","last_became_active":"2024-02-23T00:30:52.722641+0000","last_became_peered":"2024-02-23T00:30:52.722641+0000","last_unstale":"2024-02-23T00:31:09.127710+0000","last_undegraded":"2024-02-23T00:31:09.127710+0000","last_fullsized":"2024-02-23T00:31:09.127710+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.18","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130902+0000","last_change":"2024-02-23T00:31:06.332340+0000","last_active":"2024-02-23T00:31:09.130902+0000","last_peered":"2024-02-23T00:31:09.130902+0000","last_clean":"2024-02-23T00:31:09.130902+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:31:09.130902+0000","last_undegraded":"2024-02-23T00:31:09.130902+0000","last_fullsized":"2024-02-23T00:31:09.130902+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1a","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.140526+0000","last_change":"2024-02-23T00:31:09.140526+0000","last_active":"2024-02-23T00:31:09.140526+0000","last_peered":"2024-02-23T00:31:09.140526+0000","last_clean":"2024-02-23T00:31:09.140526+0000","last_became_active":"2024-02-23T00:31:03.566930+0000","last_became_peered":"2024-02-23T00:31:03.566930+0000","last_unstale":"2024-02-23T00:31:09.140526+0000","last_undegraded":"2024-02-23T00:31:09.140526+0000","last_fullsized":"2024-02-23T00:31:09.140526+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.13","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129943+0000","last_change":"2024-02-23T00:30:59.533980+0000","last_active":"2024-02-23T00:31:09.129943+0000","last_peered":"2024-02-23T00:31:09.129943+0000","last_clean":"2024-02-23T00:31:09.129943+0000","last_became_active":"2024-02-23T00:30:59.531620+0000","last_became_peered":"2024-02-23T00:30:59.531620+0000","last_unstale":"2024-02-23T00:31:09.129943+0000","last_undegraded":"2024-02-23T00:31:09.129943+0000","last_fullsized":"2024-02-23T00:31:09.129943+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"49.16","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130774+0000","last_change":"2024-02-23T00:30:56.026326+0000","last_active":"2024-02-23T00:31:09.130774+0000","last_peered":"2024-02-23T00:31:09.130774+0000","last_clean":"2024-02-23T00:31:09.130774+0000","last_became_active":"2024-02-23T00:30:56.026030+0000","last_became_peered":"2024-02-23T00:30:56.026030+0000","last_unstale":"2024-02-23T00:31:09.130774+0000","last_undegraded":"2024-02-23T00:31:09.130774+0000","last_fullsized":"2024-02-23T00:31:09.130774+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"44.b","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132424+0000","last_change":"2024-02-23T00:30:56.024352+0000","last_active":"2024-02-23T00:31:09.132424+0000","last_peered":"2024-02-23T00:31:09.132424+0000","last_clean":"2024-02-23T00:31:09.132424+0000","last_became_active":"2024-02-23T00:30:56.024071+0000","last_became_peered":"2024-02-23T00:30:56.024071+0000","last_unstale":"2024-02-23T00:31:09.132424+0000","last_undegraded":"2024-02-23T00:31:09.132424+0000","last_fullsized":"2024-02-23T00:31:09.132424+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.150965+0000","last_change":"2024-02-23T00:30:52.766561+0000","last_active":"2024-02-23T00:31:09.150965+0000","last_peered":"2024-02-23T00:31:09.150965+0000","last_clean":"2024-02-23T00:31:09.150965+0000","last_became_active":"2024-02-23T00:30:52.762168+0000","last_became_peered":"2024-02-23T00:30:52.762168+0000","last_unstale":"2024-02-23T00:31:09.150965+0000","last_undegraded":"2024-02-23T00:31:09.150965+0000","last_fullsized":"2024-02-23T00:31:09.150965+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"63.19","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121975+0000","last_change":"2024-02-23T00:31:06.254019+0000","last_active":"2024-02-23T00:31:09.121975+0000","last_peered":"2024-02-23T00:31:09.121975+0000","last_clean":"2024-02-23T00:31:09.121975+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:31:09.121975+0000","last_undegraded":"2024-02-23T00:31:09.121975+0000","last_fullsized":"2024-02-23T00:31:09.121975+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.1b","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.223975+0000","last_change":"2024-02-23T00:31:09.223975+0000","last_active":"2024-02-23T00:31:09.223975+0000","last_peered":"2024-02-23T00:31:09.223975+0000","last_clean":"2024-02-23T00:31:09.223975+0000","last_became_active":"2024-02-23T00:31:03.632642+0000","last_became_peered":"2024-02-23T00:31:03.632642+0000","last_unstale":"2024-02-23T00:31:09.223975+0000","last_undegraded":"2024-02-23T00:31:09.223975+0000","last_fullsized":"2024-02-23T00:31:09.223975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.12","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.148731+0000","last_change":"2024-02-23T00:30:59.528876+0000","last_active":"2024-02-23T00:31:09.148731+0000","last_peered":"2024-02-23T00:31:09.148731+0000","last_clean":"2024-02-23T00:31:09.148731+0000","last_became_active":"2024-02-23T00:30:59.528605+0000","last_became_peered":"2024-02-23T00:30:59.528605+0000","last_unstale":"2024-02-23T00:31:09.148731+0000","last_undegraded":"2024-02-23T00:31:09.148731+0000","last_fullsized":"2024-02-23T00:31:09.148731+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"49.17","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.185078+0000","last_change":"2024-02-23T00:30:56.024331+0000","last_active":"2024-02-23T00:31:09.185078+0000","last_peered":"2024-02-23T00:31:09.185078+0000","last_clean":"2024-02-23T00:31:09.185078+0000","last_became_active":"2024-02-23T00:30:56.023736+0000","last_became_peered":"2024-02-23T00:30:56.023736+0000","last_unstale":"2024-02-23T00:31:09.185078+0000","last_undegraded":"2024-02-23T00:31:09.185078+0000","last_fullsized":"2024-02-23T00:31:09.185078+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"44.a","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135868+0000","last_change":"2024-02-23T00:30:56.059187+0000","last_active":"2024-02-23T00:31:09.135868+0000","last_peered":"2024-02-23T00:31:09.135868+0000","last_clean":"2024-02-23T00:31:09.135868+0000","last_became_active":"2024-02-23T00:30:56.058905+0000","last_became_peered":"2024-02-23T00:30:56.058905+0000","last_unstale":"2024-02-23T00:31:09.135868+0000","last_undegraded":"2024-02-23T00:31:09.135868+0000","last_fullsized":"2024-02-23T00:31:09.135868+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120010+0000","last_change":"2024-02-23T00:30:52.767139+0000","last_active":"2024-02-23T00:31:09.120010+0000","last_peered":"2024-02-23T00:31:09.120010+0000","last_clean":"2024-02-23T00:31:09.120010+0000","last_became_active":"2024-02-23T00:30:52.766835+0000","last_became_peered":"2024-02-23T00:30:52.766835+0000","last_unstale":"2024-02-23T00:31:09.120010+0000","last_undegraded":"2024-02-23T00:31:09.120010+0000","last_fullsized":"2024-02-23T00:31:09.120010+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.1a","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128715+0000","last_change":"2024-02-23T00:31:06.367656+0000","last_active":"2024-02-23T00:31:09.128715+0000","last_peered":"2024-02-23T00:31:09.128715+0000","last_clean":"2024-02-23T00:31:09.128715+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:31:09.128715+0000","last_undegraded":"2024-02-23T00:31:09.128715+0000","last_fullsized":"2024-02-23T00:31:09.128715+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.18","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131423+0000","last_change":"2024-02-23T00:31:09.131423+0000","last_active":"2024-02-23T00:31:09.131423+0000","last_peered":"2024-02-23T00:31:09.131423+0000","last_clean":"2024-02-23T00:31:09.131423+0000","last_became_active":"2024-02-23T00:31:03.574159+0000","last_became_peered":"2024-02-23T00:31:03.574159+0000","last_unstale":"2024-02-23T00:31:09.131423+0000","last_undegraded":"2024-02-23T00:31:09.131423+0000","last_fullsized":"2024-02-23T00:31:09.131423+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.11","version":"109'3","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131674+0000","last_change":"2024-02-23T00:30:59.530655+0000","last_active":"2024-02-23T00:31:09.131674+0000","last_peered":"2024-02-23T00:31:09.131674+0000","last_clean":"2024-02-23T00:31:09.131674+0000","last_became_active":"2024-02-23T00:30:59.529045+0000","last_became_peered":"2024-02-23T00:30:59.529045+0000","last_unstale":"2024-02-23T00:31:09.131674+0000","last_undegraded":"2024-02-23T00:31:09.131674+0000","last_fullsized":"2024-02-23T00:31:09.131674+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"49.14","version":"105'1","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.158987+0000","last_change":"2024-02-23T00:30:55.958237+0000","last_active":"2024-02-23T00:31:09.158987+0000","last_peered":"2024-02-23T00:31:09.158987+0000","last_clean":"2024-02-23T00:31:09.158987+0000","last_became_active":"2024-02-23T00:30:55.957886+0000","last_became_peered":"2024-02-23T00:30:55.957886+0000","last_unstale":"2024-02-23T00:31:09.158987+0000","last_undegraded":"2024-02-23T00:31:09.158987+0000","last_fullsized":"2024-02-23T00:31:09.158987+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"44.9","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.139135+0000","last_change":"2024-02-23T00:30:54.971035+0000","last_active":"2024-02-23T00:31:09.139135+0000","last_peered":"2024-02-23T00:31:09.139135+0000","last_clean":"2024-02-23T00:31:09.139135+0000","last_became_active":"2024-02-23T00:30:54.970425+0000","last_became_peered":"2024-02-23T00:30:54.970425+0000","last_unstale":"2024-02-23T00:31:09.139135+0000","last_undegraded":"2024-02-23T00:31:09.139135+0000","last_fullsized":"2024-02-23T00:31:09.139135+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133942+0000","last_change":"2024-02-23T00:30:52.721508+0000","last_active":"2024-02-23T00:31:09.133942+0000","last_peered":"2024-02-23T00:31:09.133942+0000","last_clean":"2024-02-23T00:31:09.133942+0000","last_became_active":"2024-02-23T00:30:52.720801+0000","last_became_peered":"2024-02-23T00:30:52.720801+0000","last_unstale":"2024-02-23T00:31:09.133942+0000","last_undegraded":"2024-02-23T00:31:09.133942+0000","last_fullsized":"2024-02-23T00:31:09.133942+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.1b","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120028+0000","last_change":"2024-02-23T00:31:06.280687+0000","last_active":"2024-02-23T00:31:09.120028+0000","last_peered":"2024-02-23T00:31:09.120028+0000","last_clean":"2024-02-23T00:31:09.120028+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:31:09.120028+0000","last_undegraded":"2024-02-23T00:31:09.120028+0000","last_fullsized":"2024-02-23T00:31:09.120028+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.19","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.127981+0000","last_change":"2024-02-23T00:31:09.127981+0000","last_active":"2024-02-23T00:31:09.127981+0000","last_peered":"2024-02-23T00:31:09.127981+0000","last_clean":"2024-02-23T00:31:09.127981+0000","last_became_active":"2024-02-23T00:31:03.659737+0000","last_became_peered":"2024-02-23T00:31:03.659737+0000","last_unstale":"2024-02-23T00:31:09.127981+0000","last_undegraded":"2024-02-23T00:31:09.127981+0000","last_fullsized":"2024-02-23T00:31:09.127981+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.10","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121793+0000","last_change":"2024-02-23T00:30:59.524889+0000","last_active":"2024-02-23T00:31:09.121793+0000","last_peered":"2024-02-23T00:31:09.121793+0000","last_clean":"2024-02-23T00:31:09.121793+0000","last_became_active":"2024-02-23T00:30:59.524481+0000","last_became_peered":"2024-02-23T00:30:59.524481+0000","last_unstale":"2024-02-23T00:31:09.121793+0000","last_undegraded":"2024-02-23T00:31:09.121793+0000","last_fullsized":"2024-02-23T00:31:09.121793+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"49.15","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128925+0000","last_change":"2024-02-23T00:30:56.022820+0000","last_active":"2024-02-23T00:31:09.128925+0000","last_peered":"2024-02-23T00:31:09.128925+0000","last_clean":"2024-02-23T00:31:09.128925+0000","last_became_active":"2024-02-23T00:30:56.022522+0000","last_became_peered":"2024-02-23T00:30:56.022522+0000","last_unstale":"2024-02-23T00:31:09.128925+0000","last_undegraded":"2024-02-23T00:31:09.128925+0000","last_fullsized":"2024-02-23T00:31:09.128925+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"44.8","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121772+0000","last_change":"2024-02-23T00:30:56.033953+0000","last_active":"2024-02-23T00:31:09.121772+0000","last_peered":"2024-02-23T00:31:09.121772+0000","last_clean":"2024-02-23T00:31:09.121772+0000","last_became_active":"2024-02-23T00:30:56.033517+0000","last_became_peered":"2024-02-23T00:30:56.033517+0000","last_unstale":"2024-02-23T00:31:09.121772+0000","last_undegraded":"2024-02-23T00:31:09.121772+0000","last_fullsized":"2024-02-23T00:31:09.121772+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.172401+0000","last_change":"2024-02-23T00:30:52.765449+0000","last_active":"2024-02-23T00:31:09.172401+0000","last_peered":"2024-02-23T00:31:09.172401+0000","last_clean":"2024-02-23T00:31:09.172401+0000","last_became_active":"2024-02-23T00:30:52.764271+0000","last_became_peered":"2024-02-23T00:30:52.764271+0000","last_unstale":"2024-02-23T00:31:09.172401+0000","last_undegraded":"2024-02-23T00:31:09.172401+0000","last_fullsized":"2024-02-23T00:31:09.172401+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.1c","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.175587+0000","last_change":"2024-02-23T00:31:06.338954+0000","last_active":"2024-02-23T00:31:09.175587+0000","last_peered":"2024-02-23T00:31:09.175587+0000","last_clean":"2024-02-23T00:31:09.175587+0000","last_became_active":"2024-02-23T00:31:06.338540+0000","last_became_peered":"2024-02-23T00:31:06.338540+0000","last_unstale":"2024-02-23T00:31:09.175587+0000","last_undegraded":"2024-02-23T00:31:09.175587+0000","last_fullsized":"2024-02-23T00:31:09.175587+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1e","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138724+0000","last_change":"2024-02-23T00:31:09.138724+0000","last_active":"2024-02-23T00:31:09.138724+0000","last_peered":"2024-02-23T00:31:09.138724+0000","last_clean":"2024-02-23T00:31:09.138724+0000","last_became_active":"2024-02-23T00:31:03.627871+0000","last_became_peered":"2024-02-23T00:31:03.627871+0000","last_unstale":"2024-02-23T00:31:09.138724+0000","last_undegraded":"2024-02-23T00:31:09.138724+0000","last_fullsized":"2024-02-23T00:31:09.138724+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.17","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122633+0000","last_change":"2024-02-23T00:30:59.530587+0000","last_active":"2024-02-23T00:31:09.122633+0000","last_peered":"2024-02-23T00:31:09.122633+0000","last_clean":"2024-02-23T00:31:09.122633+0000","last_became_active":"2024-02-23T00:30:59.530278+0000","last_became_peered":"2024-02-23T00:30:59.530278+0000","last_unstale":"2024-02-23T00:31:09.122633+0000","last_undegraded":"2024-02-23T00:31:09.122633+0000","last_fullsized":"2024-02-23T00:31:09.122633+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"49.12","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125444+0000","last_change":"2024-02-23T00:30:56.035717+0000","last_active":"2024-02-23T00:31:09.125444+0000","last_peered":"2024-02-23T00:31:09.125444+0000","last_clean":"2024-02-23T00:31:09.125444+0000","last_became_active":"2024-02-23T00:30:56.035321+0000","last_became_peered":"2024-02-23T00:30:56.035321+0000","last_unstale":"2024-02-23T00:31:09.125444+0000","last_undegraded":"2024-02-23T00:31:09.125444+0000","last_fullsized":"2024-02-23T00:31:09.125444+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"44.f","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.228121+0000","last_change":"2024-02-23T00:30:54.991802+0000","last_active":"2024-02-23T00:31:09.228121+0000","last_peered":"2024-02-23T00:31:09.228121+0000","last_clean":"2024-02-23T00:31:09.228121+0000","last_became_active":"2024-02-23T00:30:54.991450+0000","last_became_peered":"2024-02-23T00:30:54.991450+0000","last_unstale":"2024-02-23T00:31:09.228121+0000","last_undegraded":"2024-02-23T00:31:09.228121+0000","last_fullsized":"2024-02-23T00:31:09.228121+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132190+0000","last_change":"2024-02-23T00:30:52.740341+0000","last_active":"2024-02-23T00:31:09.132190+0000","last_peered":"2024-02-23T00:31:09.132190+0000","last_clean":"2024-02-23T00:31:09.132190+0000","last_became_active":"2024-02-23T00:30:52.739788+0000","last_became_peered":"2024-02-23T00:30:52.739788+0000","last_unstale":"2024-02-23T00:31:09.132190+0000","last_undegraded":"2024-02-23T00:31:09.132190+0000","last_fullsized":"2024-02-23T00:31:09.132190+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.1d","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174553+0000","last_change":"2024-02-23T00:31:06.353639+0000","last_active":"2024-02-23T00:31:09.174553+0000","last_peered":"2024-02-23T00:31:09.174553+0000","last_clean":"2024-02-23T00:31:09.174553+0000","last_became_active":"2024-02-23T00:31:06.337635+0000","last_became_peered":"2024-02-23T00:31:06.337635+0000","last_unstale":"2024-02-23T00:31:09.174553+0000","last_undegraded":"2024-02-23T00:31:09.174553+0000","last_fullsized":"2024-02-23T00:31:09.174553+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1f","version":"118'6","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.089398+0000","last_change":"2024-02-23T00:31:10.089398+0000","last_active":"2024-02-23T00:31:10.089398+0000","last_peered":"2024-02-23T00:31:10.089398+0000","last_clean":"2024-02-23T00:31:10.089398+0000","last_became_active":"2024-02-23T00:31:03.667554+0000","last_became_peered":"2024-02-23T00:31:03.667554+0000","last_unstale":"2024-02-23T00:31:10.089398+0000","last_undegraded":"2024-02-23T00:31:10.089398+0000","last_fullsized":"2024-02-23T00:31:10.089398+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.16","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.187955+0000","last_change":"2024-02-23T00:30:59.545438+0000","last_active":"2024-02-23T00:31:09.187955+0000","last_peered":"2024-02-23T00:31:09.187955+0000","last_clean":"2024-02-23T00:31:09.187955+0000","last_became_active":"2024-02-23T00:30:59.544896+0000","last_became_peered":"2024-02-23T00:30:59.544896+0000","last_unstale":"2024-02-23T00:31:09.187955+0000","last_undegraded":"2024-02-23T00:31:09.187955+0000","last_fullsized":"2024-02-23T00:31:09.187955+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"49.13","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120446+0000","last_change":"2024-02-23T00:30:56.048194+0000","last_active":"2024-02-23T00:31:09.120446+0000","last_peered":"2024-02-23T00:31:09.120446+0000","last_clean":"2024-02-23T00:31:09.120446+0000","last_became_active":"2024-02-23T00:30:56.047010+0000","last_became_peered":"2024-02-23T00:30:56.047010+0000","last_unstale":"2024-02-23T00:31:09.120446+0000","last_undegraded":"2024-02-23T00:31:09.120446+0000","last_fullsized":"2024-02-23T00:31:09.120446+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"44.e","version":"111'1","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.183181+0000","last_change":"2024-02-23T00:30:54.991316+0000","last_active":"2024-02-23T00:31:09.183181+0000","last_peered":"2024-02-23T00:31:09.183181+0000","last_clean":"2024-02-23T00:31:09.183181+0000","last_became_active":"2024-02-23T00:30:54.991124+0000","last_became_peered":"2024-02-23T00:30:54.991124+0000","last_unstale":"2024-02-23T00:31:09.183181+0000","last_undegraded":"2024-02-23T00:31:09.183181+0000","last_fullsized":"2024-02-23T00:31:09.183181+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4096,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133117+0000","last_change":"2024-02-23T00:30:52.763424+0000","last_active":"2024-02-23T00:31:09.133117+0000","last_peered":"2024-02-23T00:31:09.133117+0000","last_clean":"2024-02-23T00:31:09.133117+0000","last_became_active":"2024-02-23T00:30:52.763137+0000","last_became_peered":"2024-02-23T00:30:52.763137+0000","last_unstale":"2024-02-23T00:31:09.133117+0000","last_undegraded":"2024-02-23T00:31:09.133117+0000","last_fullsized":"2024-02-23T00:31:09.133117+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1e","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126991+0000","last_change":"2024-02-23T00:31:06.209150+0000","last_active":"2024-02-23T00:31:09.126991+0000","last_peered":"2024-02-23T00:31:09.126991+0000","last_clean":"2024-02-23T00:31:09.126991+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:31:09.126991+0000","last_undegraded":"2024-02-23T00:31:09.126991+0000","last_fullsized":"2024-02-23T00:31:09.126991+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1c","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125724+0000","last_change":"2024-02-23T00:31:09.125724+0000","last_active":"2024-02-23T00:31:09.125724+0000","last_peered":"2024-02-23T00:31:09.125724+0000","last_clean":"2024-02-23T00:31:09.125724+0000","last_became_active":"2024-02-23T00:31:03.569076+0000","last_became_peered":"2024-02-23T00:31:03.569076+0000","last_unstale":"2024-02-23T00:31:09.125724+0000","last_undegraded":"2024-02-23T00:31:09.125724+0000","last_fullsized":"2024-02-23T00:31:09.125724+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.15","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.134193+0000","last_change":"2024-02-23T00:30:59.528289+0000","last_active":"2024-02-23T00:31:09.134193+0000","last_peered":"2024-02-23T00:31:09.134193+0000","last_clean":"2024-02-23T00:31:09.134193+0000","last_became_active":"2024-02-23T00:30:59.528001+0000","last_became_peered":"2024-02-23T00:30:59.528001+0000","last_unstale":"2024-02-23T00:31:09.134193+0000","last_undegraded":"2024-02-23T00:31:09.134193+0000","last_fullsized":"2024-02-23T00:31:09.134193+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"49.10","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.119797+0000","last_change":"2024-02-23T00:30:56.051768+0000","last_active":"2024-02-23T00:31:09.119797+0000","last_peered":"2024-02-23T00:31:09.119797+0000","last_clean":"2024-02-23T00:31:09.119797+0000","last_became_active":"2024-02-23T00:30:56.051205+0000","last_became_peered":"2024-02-23T00:30:56.051205+0000","last_unstale":"2024-02-23T00:31:09.119797+0000","last_undegraded":"2024-02-23T00:31:09.119797+0000","last_fullsized":"2024-02-23T00:31:09.119797+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"44.d","version":"0'0","reported_seq":24,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129326+0000","last_change":"2024-02-23T00:30:56.045015+0000","last_active":"2024-02-23T00:31:09.129326+0000","last_peered":"2024-02-23T00:31:09.129326+0000","last_clean":"2024-02-23T00:31:09.129326+0000","last_became_active":"2024-02-23T00:30:56.044784+0000","last_became_peered":"2024-02-23T00:30:56.044784+0000","last_unstale":"2024-02-23T00:31:09.129326+0000","last_undegraded":"2024-02-23T00:31:09.129326+0000","last_fullsized":"2024-02-23T00:31:09.129326+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.7","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.185212+0000","last_change":"2024-02-23T00:30:52.773433+0000","last_active":"2024-02-23T00:31:09.185212+0000","last_peered":"2024-02-23T00:31:09.185212+0000","last_clean":"2024-02-23T00:31:09.185212+0000","last_became_active":"2024-02-23T00:30:52.771195+0000","last_became_peered":"2024-02-23T00:30:52.771195+0000","last_unstale":"2024-02-23T00:31:09.185212+0000","last_undegraded":"2024-02-23T00:31:09.185212+0000","last_fullsized":"2024-02-23T00:31:09.185212+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.1f","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122259+0000","last_change":"2024-02-23T00:31:06.355927+0000","last_active":"2024-02-23T00:31:09.122259+0000","last_peered":"2024-02-23T00:31:09.122259+0000","last_clean":"2024-02-23T00:31:09.122259+0000","last_became_active":"2024-02-23T00:31:06.355623+0000","last_became_peered":"2024-02-23T00:31:06.355623+0000","last_unstale":"2024-02-23T00:31:09.122259+0000","last_undegraded":"2024-02-23T00:31:09.122259+0000","last_fullsized":"2024-02-23T00:31:09.122259+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1d","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.182203+0000","last_change":"2024-02-23T00:31:09.182203+0000","last_active":"2024-02-23T00:31:09.182203+0000","last_peered":"2024-02-23T00:31:09.182203+0000","last_clean":"2024-02-23T00:31:09.182203+0000","last_became_active":"2024-02-23T00:31:03.624465+0000","last_became_peered":"2024-02-23T00:31:03.624465+0000","last_unstale":"2024-02-23T00:31:09.182203+0000","last_undegraded":"2024-02-23T00:31:09.182203+0000","last_fullsized":"2024-02-23T00:31:09.182203+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:02.477975+0000","last_clean_scrub_stamp":"2024-02-23T00:31:02.477975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"2"}]},{"pgid":"52.14","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.159516+0000","last_change":"2024-02-23T00:30:59.544745+0000","last_active":"2024-02-23T00:31:09.159516+0000","last_peered":"2024-02-23T00:31:09.159516+0000","last_clean":"2024-02-23T00:31:09.159516+0000","last_became_active":"2024-02-23T00:30:59.544473+0000","last_became_peered":"2024-02-23T00:30:59.544473+0000","last_unstale":"2024-02-23T00:31:09.159516+0000","last_undegraded":"2024-02-23T00:31:09.159516+0000","last_fullsized":"2024-02-23T00:31:09.159516+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:58.477626+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.477626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"49.11","version":"0'0","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130493+0000","last_change":"2024-02-23T00:30:56.056405+0000","last_active":"2024-02-23T00:31:09.130493+0000","last_peered":"2024-02-23T00:31:09.130493+0000","last_clean":"2024-02-23T00:31:09.130493+0000","last_became_active":"2024-02-23T00:30:56.056150+0000","last_became_peered":"2024-02-23T00:30:56.056150+0000","last_unstale":"2024-02-23T00:31:09.130493+0000","last_undegraded":"2024-02-23T00:31:09.130493+0000","last_fullsized":"2024-02-23T00:31:09.130493+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:54.928599+0000","last_clean_scrub_stamp":"2024-02-23T00:30:54.928599+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"44.c","version":"113'1","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128177+0000","last_change":"2024-02-23T00:30:56.017685+0000","last_active":"2024-02-23T00:31:09.128177+0000","last_peered":"2024-02-23T00:31:09.128177+0000","last_clean":"2024-02-23T00:31:09.128177+0000","last_became_active":"2024-02-23T00:30:56.017366+0000","last_became_peered":"2024-02-23T00:30:56.017366+0000","last_unstale":"2024-02-23T00:31:09.128177+0000","last_undegraded":"2024-02-23T00:31:09.128177+0000","last_fullsized":"2024-02-23T00:31:09.128177+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:53.684564+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.684564+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4096,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":26,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.217286+0000","last_change":"2024-02-23T00:30:52.781125+0000","last_active":"2024-02-23T00:31:09.217286+0000","last_peered":"2024-02-23T00:31:09.217286+0000","last_clean":"2024-02-23T00:31:09.217286+0000","last_became_active":"2024-02-23T00:30:52.780465+0000","last_became_peered":"2024-02-23T00:30:52.780465+0000","last_unstale":"2024-02-23T00:31:09.217286+0000","last_undegraded":"2024-02-23T00:31:09.217286+0000","last_fullsized":"2024-02-23T00:31:09.217286+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:51.678455+0000","last_clean_scrub_stamp":"2024-02-23T00:30:51.678455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128009+0000","last_change":"2024-02-23T00:30:52.226674+0000","last_active":"2024-02-23T00:31:09.128009+0000","last_peered":"2024-02-23T00:31:09.128009+0000","last_clean":"2024-02-23T00:31:09.128009+0000","last_became_active":"2024-02-23T00:30:48.745022+0000","last_became_peered":"2024-02-23T00:30:48.745022+0000","last_unstale":"2024-02-23T00:31:09.128009+0000","last_undegraded":"2024-02-23T00:31:09.128009+0000","last_fullsized":"2024-02-23T00:31:09.128009+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:52.226569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:52.226569+0000","last_clean_scrub_stamp":"2024-02-23T00:30:52.226569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'8","reported_seq":133,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.143425+0000","last_change":"2024-02-23T00:29:48.485677+0000","last_active":"2024-02-23T00:31:10.143425+0000","last_peered":"2024-02-23T00:31:10.143425+0000","last_clean":"2024-02-23T00:31:10.143425+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:31:10.143425+0000","last_undegraded":"2024-02-23T00:31:10.143425+0000","last_fullsized":"2024-02-23T00:31:10.143425+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:29:48.485586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:48.485586+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":143,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122134+0000","last_change":"2024-02-23T00:30:25.054238+0000","last_active":"2024-02-23T00:31:09.122134+0000","last_peered":"2024-02-23T00:31:09.122134+0000","last_clean":"2024-02-23T00:31:09.122134+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:31:09.122134+0000","last_undegraded":"2024-02-23T00:31:09.122134+0000","last_fullsized":"2024-02-23T00:31:09.122134+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:30:25.054140+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.054140+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130788+0000","last_change":"2024-02-23T00:29:27.875203+0000","last_active":"2024-02-23T00:31:09.130788+0000","last_peered":"2024-02-23T00:31:09.130788+0000","last_clean":"2024-02-23T00:31:09.130788+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:31:09.130788+0000","last_undegraded":"2024-02-23T00:31:09.130788+0000","last_fullsized":"2024-02-23T00:31:09.130788+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:27.875106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:27.875106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":70,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123392+0000","last_change":"2024-02-23T00:29:08.063475+0000","last_active":"2024-02-23T00:31:09.123392+0000","last_peered":"2024-02-23T00:31:09.123392+0000","last_clean":"2024-02-23T00:31:09.123392+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:31:09.123392+0000","last_undegraded":"2024-02-23T00:31:09.123392+0000","last_fullsized":"2024-02-23T00:31:09.123392+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.063398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.063398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1f","version":"98'5","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133242+0000","last_change":"2024-02-23T00:30:48.756420+0000","last_active":"2024-02-23T00:31:09.133242+0000","last_peered":"2024-02-23T00:31:09.133242+0000","last_clean":"2024-02-23T00:31:09.133242+0000","last_became_active":"2024-02-23T00:30:48.755826+0000","last_became_peered":"2024-02-23T00:30:48.755826+0000","last_unstale":"2024-02-23T00:31:09.133242+0000","last_undegraded":"2024-02-23T00:31:09.133242+0000","last_fullsized":"2024-02-23T00:31:09.133242+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.149980+0000","last_change":"2024-02-23T00:30:48.782537+0000","last_active":"2024-02-23T00:31:09.149980+0000","last_peered":"2024-02-23T00:31:09.149980+0000","last_clean":"2024-02-23T00:31:09.149980+0000","last_became_active":"2024-02-23T00:30:48.777752+0000","last_became_peered":"2024-02-23T00:30:48.777752+0000","last_unstale":"2024-02-23T00:31:09.149980+0000","last_undegraded":"2024-02-23T00:31:09.149980+0000","last_fullsized":"2024-02-23T00:31:09.149980+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128188+0000","last_change":"2024-02-23T00:31:01.694323+0000","last_active":"2024-02-23T00:31:09.128188+0000","last_peered":"2024-02-23T00:31:09.128188+0000","last_clean":"2024-02-23T00:31:09.128188+0000","last_became_active":"2024-02-23T00:30:45.845153+0000","last_became_peered":"2024-02-23T00:30:45.845153+0000","last_unstale":"2024-02-23T00:31:09.128188+0000","last_undegraded":"2024-02-23T00:31:09.128188+0000","last_fullsized":"2024-02-23T00:31:09.128188+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:01.694259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:01.694259+0000","last_clean_scrub_stamp":"2024-02-23T00:31:01.694259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":112,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.196008+0000","last_change":"2024-02-23T00:30:33.906112+0000","last_active":"2024-02-23T00:31:09.196008+0000","last_peered":"2024-02-23T00:31:09.196008+0000","last_clean":"2024-02-23T00:31:09.196008+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:31:09.196008+0000","last_undegraded":"2024-02-23T00:31:09.196008+0000","last_fullsized":"2024-02-23T00:31:09.196008+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:30:33.906033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:33.906033+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.184103+0000","last_change":"2024-02-23T00:29:48.458818+0000","last_active":"2024-02-23T00:31:09.184103+0000","last_peered":"2024-02-23T00:31:09.184103+0000","last_clean":"2024-02-23T00:31:09.184103+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:31:09.184103+0000","last_undegraded":"2024-02-23T00:31:09.184103+0000","last_fullsized":"2024-02-23T00:31:09.184103+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:48.458726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:48.458726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129427+0000","last_change":"2024-02-23T00:30:38.736599+0000","last_active":"2024-02-23T00:31:09.129427+0000","last_peered":"2024-02-23T00:31:09.129427+0000","last_clean":"2024-02-23T00:31:09.129427+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:31:09.129427+0000","last_undegraded":"2024-02-23T00:31:09.129427+0000","last_fullsized":"2024-02-23T00:31:09.129427+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:38.736492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.736492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.151626+0000","last_change":"2024-02-23T00:30:32.953255+0000","last_active":"2024-02-23T00:31:09.151626+0000","last_peered":"2024-02-23T00:31:09.151626+0000","last_clean":"2024-02-23T00:31:09.151626+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:31:09.151626+0000","last_undegraded":"2024-02-23T00:31:09.151626+0000","last_fullsized":"2024-02-23T00:31:09.151626+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:32.953143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:32.953143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133311+0000","last_change":"2024-02-23T00:30:48.734879+0000","last_active":"2024-02-23T00:31:09.133311+0000","last_peered":"2024-02-23T00:31:09.133311+0000","last_clean":"2024-02-23T00:31:09.133311+0000","last_became_active":"2024-02-23T00:30:48.734140+0000","last_became_peered":"2024-02-23T00:30:48.734140+0000","last_unstale":"2024-02-23T00:31:09.133311+0000","last_undegraded":"2024-02-23T00:31:09.133311+0000","last_fullsized":"2024-02-23T00:31:09.133311+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133757+0000","last_change":"2024-02-23T00:30:48.713856+0000","last_active":"2024-02-23T00:31:09.133757+0000","last_peered":"2024-02-23T00:31:09.133757+0000","last_clean":"2024-02-23T00:31:09.133757+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:31:09.133757+0000","last_undegraded":"2024-02-23T00:31:09.133757+0000","last_fullsized":"2024-02-23T00:31:09.133757+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.4","version":"97'1","reported_seq":39,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.199919+0000","last_change":"2024-02-23T00:30:58.688751+0000","last_active":"2024-02-23T00:31:09.199919+0000","last_peered":"2024-02-23T00:31:09.199919+0000","last_clean":"2024-02-23T00:31:09.199919+0000","last_became_active":"2024-02-23T00:30:45.851941+0000","last_became_peered":"2024-02-23T00:30:45.851941+0000","last_unstale":"2024-02-23T00:31:09.199919+0000","last_undegraded":"2024-02-23T00:31:09.199919+0000","last_fullsized":"2024-02-23T00:31:09.199919+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'1","last_scrub_stamp":"2024-02-23T00:30:58.688677+0000","last_deep_scrub":"97'1","last_deep_scrub_stamp":"2024-02-23T00:30:58.688677+0000","last_clean_scrub_stamp":"2024-02-23T00:30:58.688677+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":127,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135692+0000","last_change":"2024-02-23T00:29:53.084484+0000","last_active":"2024-02-23T00:31:09.135692+0000","last_peered":"2024-02-23T00:31:09.135692+0000","last_clean":"2024-02-23T00:31:09.135692+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:31:09.135692+0000","last_undegraded":"2024-02-23T00:31:09.135692+0000","last_fullsized":"2024-02-23T00:31:09.135692+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:29:53.084379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:53.084379+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.159708+0000","last_change":"2024-02-23T00:29:24.809313+0000","last_active":"2024-02-23T00:31:09.159708+0000","last_peered":"2024-02-23T00:31:09.159708+0000","last_clean":"2024-02-23T00:31:09.159708+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:31:09.159708+0000","last_undegraded":"2024-02-23T00:31:09.159708+0000","last_fullsized":"2024-02-23T00:31:09.159708+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:24.809192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:24.809192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131683+0000","last_change":"2024-02-23T00:30:40.787256+0000","last_active":"2024-02-23T00:31:09.131683+0000","last_peered":"2024-02-23T00:31:09.131683+0000","last_clean":"2024-02-23T00:31:09.131683+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:31:09.131683+0000","last_undegraded":"2024-02-23T00:31:09.131683+0000","last_fullsized":"2024-02-23T00:31:09.131683+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.787191+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131416+0000","last_change":"2024-02-23T00:30:30.499519+0000","last_active":"2024-02-23T00:31:09.131416+0000","last_peered":"2024-02-23T00:31:09.131416+0000","last_clean":"2024-02-23T00:31:09.131416+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:31:09.131416+0000","last_undegraded":"2024-02-23T00:31:09.131416+0000","last_fullsized":"2024-02-23T00:31:09.131416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:30.499432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.499432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1d","version":"98'7","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138733+0000","last_change":"2024-02-23T00:30:48.742500+0000","last_active":"2024-02-23T00:31:09.138733+0000","last_peered":"2024-02-23T00:31:09.138733+0000","last_clean":"2024-02-23T00:31:09.138733+0000","last_became_active":"2024-02-23T00:30:48.741929+0000","last_became_peered":"2024-02-23T00:30:48.741929+0000","last_unstale":"2024-02-23T00:31:09.138733+0000","last_undegraded":"2024-02-23T00:31:09.138733+0000","last_fullsized":"2024-02-23T00:31:09.138733+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.162295+0000","last_change":"2024-02-23T00:30:48.673703+0000","last_active":"2024-02-23T00:31:09.162295+0000","last_peered":"2024-02-23T00:31:09.162295+0000","last_clean":"2024-02-23T00:31:09.162295+0000","last_became_active":"2024-02-23T00:30:48.673452+0000","last_became_peered":"2024-02-23T00:30:48.673452+0000","last_unstale":"2024-02-23T00:31:09.162295+0000","last_undegraded":"2024-02-23T00:31:09.162295+0000","last_fullsized":"2024-02-23T00:31:09.162295+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"98'1","reported_seq":39,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.142699+0000","last_change":"2024-02-23T00:30:57.830134+0000","last_active":"2024-02-23T00:31:09.142699+0000","last_peered":"2024-02-23T00:31:09.142699+0000","last_clean":"2024-02-23T00:31:09.142699+0000","last_became_active":"2024-02-23T00:30:45.855798+0000","last_became_peered":"2024-02-23T00:30:45.855798+0000","last_unstale":"2024-02-23T00:31:09.142699+0000","last_undegraded":"2024-02-23T00:31:09.142699+0000","last_fullsized":"2024-02-23T00:31:09.142699+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'1","last_scrub_stamp":"2024-02-23T00:30:57.830068+0000","last_deep_scrub":"98'1","last_deep_scrub_stamp":"2024-02-23T00:30:57.830068+0000","last_clean_scrub_stamp":"2024-02-23T00:30:57.830068+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132401+0000","last_change":"2024-02-23T00:30:35.121011+0000","last_active":"2024-02-23T00:31:09.132401+0000","last_peered":"2024-02-23T00:31:09.132401+0000","last_clean":"2024-02-23T00:31:09.132401+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:31:09.132401+0000","last_undegraded":"2024-02-23T00:31:09.132401+0000","last_fullsized":"2024-02-23T00:31:09.132401+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:30:35.120909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.120909+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":82,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133274+0000","last_change":"2024-02-23T00:30:43.620922+0000","last_active":"2024-02-23T00:31:09.133274+0000","last_peered":"2024-02-23T00:31:09.133274+0000","last_clean":"2024-02-23T00:31:09.133274+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:31:09.133274+0000","last_undegraded":"2024-02-23T00:31:09.133274+0000","last_fullsized":"2024-02-23T00:31:09.133274+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.620849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.620849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.176500+0000","last_change":"2024-02-23T00:29:08.580168+0000","last_active":"2024-02-23T00:31:09.176500+0000","last_peered":"2024-02-23T00:31:09.176500+0000","last_clean":"2024-02-23T00:31:09.176500+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:31:09.176500+0000","last_undegraded":"2024-02-23T00:31:09.176500+0000","last_fullsized":"2024-02-23T00:31:09.176500+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.580102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.580102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.161173+0000","last_change":"2024-02-23T00:30:26.671989+0000","last_active":"2024-02-23T00:31:09.161173+0000","last_peered":"2024-02-23T00:31:09.161173+0000","last_clean":"2024-02-23T00:31:09.161173+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:31:09.161173+0000","last_undegraded":"2024-02-23T00:31:09.161173+0000","last_fullsized":"2024-02-23T00:31:09.161173+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:26.671888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:26.671888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1a","version":"98'5","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.173074+0000","last_change":"2024-02-23T00:30:48.726547+0000","last_active":"2024-02-23T00:31:09.173074+0000","last_peered":"2024-02-23T00:31:09.173074+0000","last_clean":"2024-02-23T00:31:09.173074+0000","last_became_active":"2024-02-23T00:30:48.725016+0000","last_became_peered":"2024-02-23T00:30:48.725016+0000","last_unstale":"2024-02-23T00:31:09.173074+0000","last_undegraded":"2024-02-23T00:31:09.173074+0000","last_fullsized":"2024-02-23T00:31:09.173074+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135981+0000","last_change":"2024-02-23T00:30:48.716832+0000","last_active":"2024-02-23T00:31:09.135981+0000","last_peered":"2024-02-23T00:31:09.135981+0000","last_clean":"2024-02-23T00:31:09.135981+0000","last_became_active":"2024-02-23T00:30:48.716552+0000","last_became_peered":"2024-02-23T00:30:48.716552+0000","last_unstale":"2024-02-23T00:31:09.135981+0000","last_undegraded":"2024-02-23T00:31:09.135981+0000","last_fullsized":"2024-02-23T00:31:09.135981+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2","version":"98'4","reported_seq":43,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.149525+0000","last_change":"2024-02-23T00:30:55.722743+0000","last_active":"2024-02-23T00:31:09.149525+0000","last_peered":"2024-02-23T00:31:09.149525+0000","last_clean":"2024-02-23T00:31:09.149525+0000","last_became_active":"2024-02-23T00:30:45.849309+0000","last_became_peered":"2024-02-23T00:30:45.849309+0000","last_unstale":"2024-02-23T00:31:09.149525+0000","last_undegraded":"2024-02-23T00:31:09.149525+0000","last_fullsized":"2024-02-23T00:31:09.149525+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'4","last_scrub_stamp":"2024-02-23T00:30:55.722665+0000","last_deep_scrub":"98'4","last_deep_scrub_stamp":"2024-02-23T00:30:55.722665+0000","last_clean_scrub_stamp":"2024-02-23T00:30:55.722665+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":91,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128589+0000","last_change":"2024-02-23T00:29:08.818324+0000","last_active":"2024-02-23T00:31:09.128589+0000","last_peered":"2024-02-23T00:31:09.128589+0000","last_clean":"2024-02-23T00:31:09.128589+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:31:09.128589+0000","last_undegraded":"2024-02-23T00:31:09.128589+0000","last_fullsized":"2024-02-23T00:31:09.128589+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:29:08.818235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.818235+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.149603+0000","last_change":"2024-02-23T00:29:00.703800+0000","last_active":"2024-02-23T00:31:09.149603+0000","last_peered":"2024-02-23T00:31:09.149603+0000","last_clean":"2024-02-23T00:31:09.149603+0000","last_became_active":"2024-02-23T00:28:41.203031+0000","last_became_peered":"2024-02-23T00:28:41.203031+0000","last_unstale":"2024-02-23T00:31:09.149603+0000","last_undegraded":"2024-02-23T00:31:09.149603+0000","last_fullsized":"2024-02-23T00:31:09.149603+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:00.703737+0000","last_clean_scrub_stamp":"2024-02-23T00:29:00.703737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":134,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.203653+0000","last_change":"2024-02-23T00:30:22.781434+0000","last_active":"2024-02-23T00:31:10.203653+0000","last_peered":"2024-02-23T00:31:10.203653+0000","last_clean":"2024-02-23T00:31:10.203653+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:31:10.203653+0000","last_undegraded":"2024-02-23T00:31:10.203653+0000","last_fullsized":"2024-02-23T00:31:10.203653+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:30:22.781318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.781318+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131746+0000","last_change":"2024-02-23T00:30:31.065478+0000","last_active":"2024-02-23T00:31:09.131746+0000","last_peered":"2024-02-23T00:31:09.131746+0000","last_clean":"2024-02-23T00:31:09.131746+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:31:09.131746+0000","last_undegraded":"2024-02-23T00:31:09.131746+0000","last_fullsized":"2024-02-23T00:31:09.131746+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.065384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.065384+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"98'10","reported_seq":41,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.124571+0000","last_change":"2024-02-23T00:30:48.671089+0000","last_active":"2024-02-23T00:31:09.124571+0000","last_peered":"2024-02-23T00:31:09.124571+0000","last_clean":"2024-02-23T00:31:09.124571+0000","last_became_active":"2024-02-23T00:30:48.669268+0000","last_became_peered":"2024-02-23T00:30:48.669268+0000","last_unstale":"2024-02-23T00:31:09.124571+0000","last_undegraded":"2024-02-23T00:31:09.124571+0000","last_fullsized":"2024-02-23T00:31:09.124571+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131826+0000","last_change":"2024-02-23T00:30:48.656349+0000","last_active":"2024-02-23T00:31:09.131826+0000","last_peered":"2024-02-23T00:31:09.131826+0000","last_clean":"2024-02-23T00:31:09.131826+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:31:09.131826+0000","last_undegraded":"2024-02-23T00:31:09.131826+0000","last_fullsized":"2024-02-23T00:31:09.131826+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"98'1","reported_seq":39,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122670+0000","last_change":"2024-02-23T00:30:53.080463+0000","last_active":"2024-02-23T00:31:09.122670+0000","last_peered":"2024-02-23T00:31:09.122670+0000","last_clean":"2024-02-23T00:31:09.122670+0000","last_became_active":"2024-02-23T00:30:48.696494+0000","last_became_peered":"2024-02-23T00:30:48.696494+0000","last_unstale":"2024-02-23T00:31:09.122670+0000","last_undegraded":"2024-02-23T00:31:09.122670+0000","last_fullsized":"2024-02-23T00:31:09.122670+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'1","last_scrub_stamp":"2024-02-23T00:30:53.080396+0000","last_deep_scrub":"98'1","last_deep_scrub_stamp":"2024-02-23T00:30:53.080396+0000","last_clean_scrub_stamp":"2024-02-23T00:30:53.080396+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"87'22","reported_seq":113,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.193672+0000","last_change":"2024-02-23T00:29:57.567523+0000","last_active":"2024-02-23T00:31:09.193672+0000","last_peered":"2024-02-23T00:31:09.193672+0000","last_clean":"2024-02-23T00:31:09.193672+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:31:09.193672+0000","last_undegraded":"2024-02-23T00:31:09.193672+0000","last_fullsized":"2024-02-23T00:31:09.193672+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:29:57.567427+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174501+0000","last_change":"2024-02-23T00:29:10.574049+0000","last_active":"2024-02-23T00:31:09.174501+0000","last_peered":"2024-02-23T00:31:09.174501+0000","last_clean":"2024-02-23T00:31:09.174501+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:31:09.174501+0000","last_undegraded":"2024-02-23T00:31:09.174501+0000","last_fullsized":"2024-02-23T00:31:09.174501+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.573983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.573983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":134,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.145863+0000","last_change":"2024-02-23T00:29:46.048380+0000","last_active":"2024-02-23T00:31:10.145863+0000","last_peered":"2024-02-23T00:31:10.145863+0000","last_clean":"2024-02-23T00:31:10.145863+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:31:10.145863+0000","last_undegraded":"2024-02-23T00:31:10.145863+0000","last_fullsized":"2024-02-23T00:31:10.145863+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:29:46.048267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:46.048267+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":71,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125654+0000","last_change":"2024-02-23T00:28:47.090951+0000","last_active":"2024-02-23T00:31:09.125654+0000","last_peered":"2024-02-23T00:31:09.125654+0000","last_clean":"2024-02-23T00:31:09.125654+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:31:09.125654+0000","last_undegraded":"2024-02-23T00:31:09.125654+0000","last_fullsized":"2024-02-23T00:31:09.125654+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:28:47.090884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.090884+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.18","version":"98'2","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122447+0000","last_change":"2024-02-23T00:30:48.696478+0000","last_active":"2024-02-23T00:31:09.122447+0000","last_peered":"2024-02-23T00:31:09.122447+0000","last_clean":"2024-02-23T00:31:09.122447+0000","last_became_active":"2024-02-23T00:30:48.693791+0000","last_became_peered":"2024-02-23T00:30:48.693791+0000","last_unstale":"2024-02-23T00:31:09.122447+0000","last_undegraded":"2024-02-23T00:31:09.122447+0000","last_fullsized":"2024-02-23T00:31:09.122447+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.183657+0000","last_change":"2024-02-23T00:30:48.713004+0000","last_active":"2024-02-23T00:31:09.183657+0000","last_peered":"2024-02-23T00:31:09.183657+0000","last_clean":"2024-02-23T00:31:09.183657+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:31:09.183657+0000","last_undegraded":"2024-02-23T00:31:09.183657+0000","last_fullsized":"2024-02-23T00:31:09.183657+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"99'1006","reported_seq":2044,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121952+0000","last_change":"2024-02-23T00:30:52.971058+0000","last_active":"2024-02-23T00:31:09.121952+0000","last_peered":"2024-02-23T00:31:09.121952+0000","last_clean":"2024-02-23T00:31:09.121952+0000","last_became_active":"2024-02-23T00:30:48.675769+0000","last_became_peered":"2024-02-23T00:30:48.675769+0000","last_unstale":"2024-02-23T00:31:09.121952+0000","last_undegraded":"2024-02-23T00:31:09.121952+0000","last_fullsized":"2024-02-23T00:31:09.121952+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'1006","last_scrub_stamp":"2024-02-23T00:30:52.970968+0000","last_deep_scrub":"99'1006","last_deep_scrub_stamp":"2024-02-23T00:30:52.970968+0000","last_clean_scrub_stamp":"2024-02-23T00:30:52.970968+0000","log_size":1006,"ondisk_log_size":1006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1000,"num_read_kb":0,"num_write":1002,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"86'23","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131355+0000","last_change":"2024-02-23T00:29:56.044401+0000","last_active":"2024-02-23T00:31:09.131355+0000","last_peered":"2024-02-23T00:31:09.131355+0000","last_clean":"2024-02-23T00:31:09.131355+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:31:09.131355+0000","last_undegraded":"2024-02-23T00:31:09.131355+0000","last_fullsized":"2024-02-23T00:31:09.131355+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'23","last_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:29:56.044331+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131662+0000","last_change":"2024-02-23T00:29:42.507393+0000","last_active":"2024-02-23T00:31:09.131662+0000","last_peered":"2024-02-23T00:31:09.131662+0000","last_clean":"2024-02-23T00:31:09.131662+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:31:09.131662+0000","last_undegraded":"2024-02-23T00:31:09.131662+0000","last_fullsized":"2024-02-23T00:31:09.131662+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:29:42.507310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:42.507310+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":72,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123273+0000","last_change":"2024-02-23T00:28:54.905338+0000","last_active":"2024-02-23T00:31:09.123273+0000","last_peered":"2024-02-23T00:31:09.123273+0000","last_clean":"2024-02-23T00:31:09.123273+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:31:09.123273+0000","last_undegraded":"2024-02-23T00:31:09.123273+0000","last_fullsized":"2024-02-23T00:31:09.123273+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:54.905258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:54.905258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":72,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174659+0000","last_change":"2024-02-23T00:29:07.960232+0000","last_active":"2024-02-23T00:31:09.174659+0000","last_peered":"2024-02-23T00:31:09.174659+0000","last_clean":"2024-02-23T00:31:09.174659+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:31:09.174659+0000","last_undegraded":"2024-02-23T00:31:09.174659+0000","last_fullsized":"2024-02-23T00:31:09.174659+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:07.960141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:07.960141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174942+0000","last_change":"2024-02-23T00:30:48.772504+0000","last_active":"2024-02-23T00:31:09.174942+0000","last_peered":"2024-02-23T00:31:09.174942+0000","last_clean":"2024-02-23T00:31:09.174942+0000","last_became_active":"2024-02-23T00:30:48.771852+0000","last_became_peered":"2024-02-23T00:30:48.771852+0000","last_unstale":"2024-02-23T00:31:09.174942+0000","last_undegraded":"2024-02-23T00:31:09.174942+0000","last_fullsized":"2024-02-23T00:31:09.174942+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.176794+0000","last_change":"2024-02-23T00:30:48.742570+0000","last_active":"2024-02-23T00:31:09.176794+0000","last_peered":"2024-02-23T00:31:09.176794+0000","last_clean":"2024-02-23T00:31:09.176794+0000","last_became_active":"2024-02-23T00:30:48.740636+0000","last_became_peered":"2024-02-23T00:30:48.740636+0000","last_unstale":"2024-02-23T00:31:09.176794+0000","last_undegraded":"2024-02-23T00:31:09.176794+0000","last_fullsized":"2024-02-23T00:31:09.176794+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132808+0000","last_change":"2024-02-23T00:30:48.727072+0000","last_active":"2024-02-23T00:31:09.132808+0000","last_peered":"2024-02-23T00:31:09.132808+0000","last_clean":"2024-02-23T00:31:09.132808+0000","last_became_active":"2024-02-23T00:30:48.719268+0000","last_became_peered":"2024-02-23T00:30:48.719268+0000","last_unstale":"2024-02-23T00:31:09.132808+0000","last_undegraded":"2024-02-23T00:31:09.132808+0000","last_fullsized":"2024-02-23T00:31:09.132808+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":129,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.154056+0000","last_change":"2024-02-23T00:29:02.969742+0000","last_active":"2024-02-23T00:31:10.154056+0000","last_peered":"2024-02-23T00:31:10.154056+0000","last_clean":"2024-02-23T00:31:10.154056+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:31:10.154056+0000","last_undegraded":"2024-02-23T00:31:10.154056+0000","last_fullsized":"2024-02-23T00:31:10.154056+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:29:02.969659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.969659+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":112,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122785+0000","last_change":"2024-02-23T00:29:09.834851+0000","last_active":"2024-02-23T00:31:09.122785+0000","last_peered":"2024-02-23T00:31:09.122785+0000","last_clean":"2024-02-23T00:31:09.122785+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:31:09.122785+0000","last_undegraded":"2024-02-23T00:31:09.122785+0000","last_fullsized":"2024-02-23T00:31:09.122785+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:29:09.834741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:09.834741+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":82,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132875+0000","last_change":"2024-02-23T00:30:26.170570+0000","last_active":"2024-02-23T00:31:09.132875+0000","last_peered":"2024-02-23T00:31:09.132875+0000","last_clean":"2024-02-23T00:31:09.132875+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:31:09.132875+0000","last_undegraded":"2024-02-23T00:31:09.132875+0000","last_fullsized":"2024-02-23T00:31:09.132875+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:26.170482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:26.170482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133661+0000","last_change":"2024-02-23T00:29:49.503208+0000","last_active":"2024-02-23T00:31:09.133661+0000","last_peered":"2024-02-23T00:31:09.133661+0000","last_clean":"2024-02-23T00:31:09.133661+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:31:09.133661+0000","last_undegraded":"2024-02-23T00:31:09.133661+0000","last_fullsized":"2024-02-23T00:31:09.133661+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:49.503096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:49.503096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.16","version":"98'5","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129066+0000","last_change":"2024-02-23T00:30:48.695224+0000","last_active":"2024-02-23T00:31:09.129066+0000","last_peered":"2024-02-23T00:31:09.129066+0000","last_clean":"2024-02-23T00:31:09.129066+0000","last_became_active":"2024-02-23T00:30:48.688249+0000","last_became_peered":"2024-02-23T00:30:48.688249+0000","last_unstale":"2024-02-23T00:31:09.129066+0000","last_undegraded":"2024-02-23T00:31:09.129066+0000","last_fullsized":"2024-02-23T00:31:09.129066+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.181587+0000","last_change":"2024-02-23T00:30:48.782732+0000","last_active":"2024-02-23T00:31:09.181587+0000","last_peered":"2024-02-23T00:31:09.181587+0000","last_clean":"2024-02-23T00:31:09.181587+0000","last_became_active":"2024-02-23T00:30:48.778293+0000","last_became_peered":"2024-02-23T00:30:48.778293+0000","last_unstale":"2024-02-23T00:31:09.181587+0000","last_undegraded":"2024-02-23T00:31:09.181587+0000","last_fullsized":"2024-02-23T00:31:09.181587+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"100'5","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131522+0000","last_change":"2024-02-23T00:30:48.657775+0000","last_active":"2024-02-23T00:31:09.131522+0000","last_peered":"2024-02-23T00:31:09.131522+0000","last_clean":"2024-02-23T00:31:09.131522+0000","last_became_active":"2024-02-23T00:30:48.657502+0000","last_became_peered":"2024-02-23T00:30:48.657502+0000","last_unstale":"2024-02-23T00:31:09.131522+0000","last_undegraded":"2024-02-23T00:31:09.131522+0000","last_fullsized":"2024-02-23T00:31:09.131522+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165835,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123098+0000","last_change":"2024-02-23T00:29:47.033380+0000","last_active":"2024-02-23T00:31:09.123098+0000","last_peered":"2024-02-23T00:31:09.123098+0000","last_clean":"2024-02-23T00:31:09.123098+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:31:09.123098+0000","last_undegraded":"2024-02-23T00:31:09.123098+0000","last_fullsized":"2024-02-23T00:31:09.123098+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:47.033301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:47.033301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131076+0000","last_change":"2024-02-23T00:29:04.010419+0000","last_active":"2024-02-23T00:31:09.131076+0000","last_peered":"2024-02-23T00:31:09.131076+0000","last_clean":"2024-02-23T00:31:09.131076+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:31:09.131076+0000","last_undegraded":"2024-02-23T00:31:09.131076+0000","last_fullsized":"2024-02-23T00:31:09.131076+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:29:04.010348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.010348+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.183320+0000","last_change":"2024-02-23T00:30:25.593913+0000","last_active":"2024-02-23T00:31:09.183320+0000","last_peered":"2024-02-23T00:31:09.183320+0000","last_clean":"2024-02-23T00:31:09.183320+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:31:09.183320+0000","last_undegraded":"2024-02-23T00:31:09.183320+0000","last_fullsized":"2024-02-23T00:31:09.183320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.593833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.593833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126143+0000","last_change":"2024-02-23T00:29:40.940545+0000","last_active":"2024-02-23T00:31:09.126143+0000","last_peered":"2024-02-23T00:31:09.126143+0000","last_clean":"2024-02-23T00:31:09.126143+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:31:09.126143+0000","last_undegraded":"2024-02-23T00:31:09.126143+0000","last_fullsized":"2024-02-23T00:31:09.126143+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:40.940460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:40.940460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.17","version":"98'5","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123665+0000","last_change":"2024-02-23T00:30:48.716942+0000","last_active":"2024-02-23T00:31:09.123665+0000","last_peered":"2024-02-23T00:31:09.123665+0000","last_clean":"2024-02-23T00:31:09.123665+0000","last_became_active":"2024-02-23T00:30:48.715717+0000","last_became_peered":"2024-02-23T00:30:48.715717+0000","last_unstale":"2024-02-23T00:31:09.123665+0000","last_undegraded":"2024-02-23T00:31:09.123665+0000","last_fullsized":"2024-02-23T00:31:09.123665+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.172671+0000","last_change":"2024-02-23T00:30:48.776316+0000","last_active":"2024-02-23T00:31:09.172671+0000","last_peered":"2024-02-23T00:31:09.172671+0000","last_clean":"2024-02-23T00:31:09.172671+0000","last_became_active":"2024-02-23T00:30:48.769602+0000","last_became_peered":"2024-02-23T00:30:48.769602+0000","last_unstale":"2024-02-23T00:31:09.172671+0000","last_undegraded":"2024-02-23T00:31:09.172671+0000","last_fullsized":"2024-02-23T00:31:09.172671+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132277+0000","last_change":"2024-02-23T00:30:48.759250+0000","last_active":"2024-02-23T00:31:09.132277+0000","last_peered":"2024-02-23T00:31:09.132277+0000","last_clean":"2024-02-23T00:31:09.132277+0000","last_became_active":"2024-02-23T00:30:48.759107+0000","last_became_peered":"2024-02-23T00:30:48.759107+0000","last_unstale":"2024-02-23T00:31:09.132277+0000","last_undegraded":"2024-02-23T00:31:09.132277+0000","last_fullsized":"2024-02-23T00:31:09.132277+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135767+0000","last_change":"2024-02-23T00:30:27.151423+0000","last_active":"2024-02-23T00:31:09.135767+0000","last_peered":"2024-02-23T00:31:09.135767+0000","last_clean":"2024-02-23T00:31:09.135767+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:31:09.135767+0000","last_undegraded":"2024-02-23T00:31:09.135767+0000","last_fullsized":"2024-02-23T00:31:09.135767+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:27.151342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:27.151342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":132,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.195860+0000","last_change":"2024-02-23T00:29:50.166913+0000","last_active":"2024-02-23T00:31:09.195860+0000","last_peered":"2024-02-23T00:31:09.195860+0000","last_clean":"2024-02-23T00:31:09.195860+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:31:09.195860+0000","last_undegraded":"2024-02-23T00:31:09.195860+0000","last_fullsized":"2024-02-23T00:31:09.195860+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:29:50.166826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:50.166826+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":82,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138447+0000","last_change":"2024-02-23T00:29:46.507110+0000","last_active":"2024-02-23T00:31:09.138447+0000","last_peered":"2024-02-23T00:31:09.138447+0000","last_clean":"2024-02-23T00:31:09.138447+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:31:09.138447+0000","last_undegraded":"2024-02-23T00:31:09.138447+0000","last_fullsized":"2024-02-23T00:31:09.138447+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:46.507012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:46.507012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174581+0000","last_change":"2024-02-23T00:30:46.579484+0000","last_active":"2024-02-23T00:31:09.174581+0000","last_peered":"2024-02-23T00:31:09.174581+0000","last_clean":"2024-02-23T00:31:09.174581+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:31:09.174581+0000","last_undegraded":"2024-02-23T00:31:09.174581+0000","last_fullsized":"2024-02-23T00:31:09.174581+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.579417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.579417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"98'2","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133241+0000","last_change":"2024-02-23T00:30:48.715663+0000","last_active":"2024-02-23T00:31:09.133241+0000","last_peered":"2024-02-23T00:31:09.133241+0000","last_clean":"2024-02-23T00:31:09.133241+0000","last_became_active":"2024-02-23T00:30:48.715382+0000","last_became_peered":"2024-02-23T00:30:48.715382+0000","last_unstale":"2024-02-23T00:31:09.133241+0000","last_undegraded":"2024-02-23T00:31:09.133241+0000","last_fullsized":"2024-02-23T00:31:09.133241+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174654+0000","last_change":"2024-02-23T00:30:48.707450+0000","last_active":"2024-02-23T00:31:09.174654+0000","last_peered":"2024-02-23T00:31:09.174654+0000","last_clean":"2024-02-23T00:31:09.174654+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:31:09.174654+0000","last_undegraded":"2024-02-23T00:31:09.174654+0000","last_fullsized":"2024-02-23T00:31:09.174654+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.162367+0000","last_change":"2024-02-23T00:31:00.671290+0000","last_active":"2024-02-23T00:31:09.162367+0000","last_peered":"2024-02-23T00:31:09.162367+0000","last_clean":"2024-02-23T00:31:09.162367+0000","last_became_active":"2024-02-23T00:30:45.855458+0000","last_became_peered":"2024-02-23T00:30:45.855458+0000","last_unstale":"2024-02-23T00:31:09.162367+0000","last_undegraded":"2024-02-23T00:31:09.162367+0000","last_fullsized":"2024-02-23T00:31:09.162367+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:00.671201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:00.671201+0000","last_clean_scrub_stamp":"2024-02-23T00:31:00.671201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123393+0000","last_change":"2024-02-23T00:29:39.108458+0000","last_active":"2024-02-23T00:31:09.123393+0000","last_peered":"2024-02-23T00:31:09.123393+0000","last_clean":"2024-02-23T00:31:09.123393+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:31:09.123393+0000","last_undegraded":"2024-02-23T00:31:09.123393+0000","last_fullsized":"2024-02-23T00:31:09.123393+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:39.108381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:29:39.108381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.217458+0000","last_change":"2024-02-23T00:29:53.274270+0000","last_active":"2024-02-23T00:31:09.217458+0000","last_peered":"2024-02-23T00:31:09.217458+0000","last_clean":"2024-02-23T00:31:09.217458+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:31:09.217458+0000","last_undegraded":"2024-02-23T00:31:09.217458+0000","last_fullsized":"2024-02-23T00:31:09.217458+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:29:53.274159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:53.274159+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131489+0000","last_change":"2024-02-23T00:29:02.340533+0000","last_active":"2024-02-23T00:31:09.131489+0000","last_peered":"2024-02-23T00:31:09.131489+0000","last_clean":"2024-02-23T00:31:09.131489+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:31:09.131489+0000","last_undegraded":"2024-02-23T00:31:09.131489+0000","last_fullsized":"2024-02-23T00:31:09.131489+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.340450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.340450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.151267+0000","last_change":"2024-02-23T00:30:35.845452+0000","last_active":"2024-02-23T00:31:09.151267+0000","last_peered":"2024-02-23T00:31:09.151267+0000","last_clean":"2024-02-23T00:31:09.151267+0000","last_became_active":"2024-02-23T00:27:17.009826+0000","last_became_peered":"2024-02-23T00:27:17.009826+0000","last_unstale":"2024-02-23T00:31:09.151267+0000","last_undegraded":"2024-02-23T00:31:09.151267+0000","last_fullsized":"2024-02-23T00:31:09.151267+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":72,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.176585+0000","last_change":"2024-02-23T00:28:55.679584+0000","last_active":"2024-02-23T00:31:09.176585+0000","last_peered":"2024-02-23T00:31:09.176585+0000","last_clean":"2024-02-23T00:31:09.176585+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:31:09.176585+0000","last_undegraded":"2024-02-23T00:31:09.176585+0000","last_fullsized":"2024-02-23T00:31:09.176585+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:55.679522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.679522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"98'7","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133686+0000","last_change":"2024-02-23T00:30:48.707448+0000","last_active":"2024-02-23T00:31:09.133686+0000","last_peered":"2024-02-23T00:31:09.133686+0000","last_clean":"2024-02-23T00:31:09.133686+0000","last_became_active":"2024-02-23T00:30:48.706820+0000","last_became_peered":"2024-02-23T00:30:48.706820+0000","last_unstale":"2024-02-23T00:31:09.133686+0000","last_undegraded":"2024-02-23T00:31:09.133686+0000","last_fullsized":"2024-02-23T00:31:09.133686+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128740+0000","last_change":"2024-02-23T00:30:48.711294+0000","last_active":"2024-02-23T00:31:09.128740+0000","last_peered":"2024-02-23T00:31:09.128740+0000","last_clean":"2024-02-23T00:31:09.128740+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:31:09.128740+0000","last_undegraded":"2024-02-23T00:31:09.128740+0000","last_fullsized":"2024-02-23T00:31:09.128740+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.b","version":"98'1","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.134201+0000","last_change":"2024-02-23T00:30:48.743397+0000","last_active":"2024-02-23T00:31:09.134201+0000","last_peered":"2024-02-23T00:31:09.134201+0000","last_clean":"2024-02-23T00:31:09.134201+0000","last_became_active":"2024-02-23T00:30:48.743034+0000","last_became_peered":"2024-02-23T00:30:48.743034+0000","last_unstale":"2024-02-23T00:31:09.134201+0000","last_undegraded":"2024-02-23T00:31:09.134201+0000","last_fullsized":"2024-02-23T00:31:09.134201+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133686+0000","last_change":"2024-02-23T00:30:42.634821+0000","last_active":"2024-02-23T00:31:09.133686+0000","last_peered":"2024-02-23T00:31:09.133686+0000","last_clean":"2024-02-23T00:31:09.133686+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:31:09.133686+0000","last_undegraded":"2024-02-23T00:31:09.133686+0000","last_fullsized":"2024-02-23T00:31:09.133686+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:42.634753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:42.634753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":103,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132360+0000","last_change":"2024-02-23T00:29:42.929178+0000","last_active":"2024-02-23T00:31:09.132360+0000","last_peered":"2024-02-23T00:31:09.132360+0000","last_clean":"2024-02-23T00:31:09.132360+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:31:09.132360+0000","last_undegraded":"2024-02-23T00:31:09.132360+0000","last_fullsized":"2024-02-23T00:31:09.132360+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:29:42.929096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:42.929096+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":85,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131579+0000","last_change":"2024-02-23T00:29:25.267851+0000","last_active":"2024-02-23T00:31:09.131579+0000","last_peered":"2024-02-23T00:31:09.131579+0000","last_clean":"2024-02-23T00:31:09.131579+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:31:09.131579+0000","last_undegraded":"2024-02-23T00:31:09.131579+0000","last_fullsized":"2024-02-23T00:31:09.131579+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:29:25.267773+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":81,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121086+0000","last_change":"2024-02-23T00:30:40.144284+0000","last_active":"2024-02-23T00:31:09.121086+0000","last_peered":"2024-02-23T00:31:09.121086+0000","last_clean":"2024-02-23T00:31:09.121086+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:31:09.121086+0000","last_undegraded":"2024-02-23T00:31:09.121086+0000","last_fullsized":"2024-02-23T00:31:09.121086+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.144220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.144220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.172744+0000","last_change":"2024-02-23T00:29:50.052543+0000","last_active":"2024-02-23T00:31:09.172744+0000","last_peered":"2024-02-23T00:31:09.172744+0000","last_clean":"2024-02-23T00:31:09.172744+0000","last_became_active":"2024-02-23T00:27:17.012683+0000","last_became_peered":"2024-02-23T00:27:17.012683+0000","last_unstale":"2024-02-23T00:31:09.172744+0000","last_undegraded":"2024-02-23T00:31:09.172744+0000","last_fullsized":"2024-02-23T00:31:09.172744+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:50.052469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:50.052469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123467+0000","last_change":"2024-02-23T00:30:45.024137+0000","last_active":"2024-02-23T00:31:09.123467+0000","last_peered":"2024-02-23T00:31:09.123467+0000","last_clean":"2024-02-23T00:31:09.123467+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:31:09.123467+0000","last_undegraded":"2024-02-23T00:31:09.123467+0000","last_fullsized":"2024-02-23T00:31:09.123467+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.024058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.024058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138758+0000","last_change":"2024-02-23T00:30:48.734267+0000","last_active":"2024-02-23T00:31:09.138758+0000","last_peered":"2024-02-23T00:31:09.138758+0000","last_clean":"2024-02-23T00:31:09.138758+0000","last_became_active":"2024-02-23T00:30:48.726403+0000","last_became_peered":"2024-02-23T00:30:48.726403+0000","last_unstale":"2024-02-23T00:31:09.138758+0000","last_undegraded":"2024-02-23T00:31:09.138758+0000","last_fullsized":"2024-02-23T00:31:09.138758+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132529+0000","last_change":"2024-02-23T00:30:48.664253+0000","last_active":"2024-02-23T00:31:09.132529+0000","last_peered":"2024-02-23T00:31:09.132529+0000","last_clean":"2024-02-23T00:31:09.132529+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:31:09.132529+0000","last_undegraded":"2024-02-23T00:31:09.132529+0000","last_fullsized":"2024-02-23T00:31:09.132529+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.9","version":"104'8","reported_seq":50,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126762+0000","last_change":"2024-02-23T00:30:55.312818+0000","last_active":"2024-02-23T00:31:09.126762+0000","last_peered":"2024-02-23T00:31:09.126762+0000","last_clean":"2024-02-23T00:31:09.126762+0000","last_became_active":"2024-02-23T00:30:48.664389+0000","last_became_peered":"2024-02-23T00:30:48.664389+0000","last_unstale":"2024-02-23T00:31:09.126762+0000","last_undegraded":"2024-02-23T00:31:09.126762+0000","last_fullsized":"2024-02-23T00:31:09.126762+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'8","last_scrub_stamp":"2024-02-23T00:30:55.312747+0000","last_deep_scrub":"104'8","last_deep_scrub_stamp":"2024-02-23T00:30:55.312747+0000","last_clean_scrub_stamp":"2024-02-23T00:30:55.312747+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165843,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":2,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130930+0000","last_change":"2024-02-23T00:30:23.737978+0000","last_active":"2024-02-23T00:31:09.130930+0000","last_peered":"2024-02-23T00:31:09.130930+0000","last_clean":"2024-02-23T00:31:09.130930+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:31:09.130930+0000","last_undegraded":"2024-02-23T00:31:09.130930+0000","last_fullsized":"2024-02-23T00:31:09.130930+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:23.737872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.737872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":123,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.193488+0000","last_change":"2024-02-23T00:29:44.530235+0000","last_active":"2024-02-23T00:31:09.193488+0000","last_peered":"2024-02-23T00:31:09.193488+0000","last_clean":"2024-02-23T00:31:09.193488+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:31:09.193488+0000","last_undegraded":"2024-02-23T00:31:09.193488+0000","last_fullsized":"2024-02-23T00:31:09.193488+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:29:44.530164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:44.530164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.127392+0000","last_change":"2024-02-23T00:29:39.779046+0000","last_active":"2024-02-23T00:31:09.127392+0000","last_peered":"2024-02-23T00:31:09.127392+0000","last_clean":"2024-02-23T00:31:09.127392+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:31:09.127392+0000","last_undegraded":"2024-02-23T00:31:09.127392+0000","last_fullsized":"2024-02-23T00:31:09.127392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:39.778983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:39.778983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":314,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.161826+0000","last_change":"2024-02-23T00:30:38.432744+0000","last_active":"2024-02-23T00:31:10.161826+0000","last_peered":"2024-02-23T00:31:10.161826+0000","last_clean":"2024-02-23T00:31:10.161826+0000","last_became_active":"2024-02-23T00:27:17.009670+0000","last_became_peered":"2024-02-23T00:27:17.009670+0000","last_unstale":"2024-02-23T00:31:10.161826+0000","last_undegraded":"2024-02-23T00:31:10.161826+0000","last_fullsized":"2024-02-23T00:31:10.161826+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:30:38.432677+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.432677+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":227,"num_read_kb":222,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138383+0000","last_change":"2024-02-23T00:30:32.479956+0000","last_active":"2024-02-23T00:31:09.138383+0000","last_peered":"2024-02-23T00:31:09.138383+0000","last_clean":"2024-02-23T00:31:09.138383+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:31:09.138383+0000","last_undegraded":"2024-02-23T00:31:09.138383+0000","last_fullsized":"2024-02-23T00:31:09.138383+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:32.479814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:32.479814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.158030+0000","last_change":"2024-02-23T00:30:48.712509+0000","last_active":"2024-02-23T00:31:09.158030+0000","last_peered":"2024-02-23T00:31:09.158030+0000","last_clean":"2024-02-23T00:31:09.158030+0000","last_became_active":"2024-02-23T00:30:48.712247+0000","last_became_peered":"2024-02-23T00:30:48.712247+0000","last_unstale":"2024-02-23T00:31:09.158030+0000","last_undegraded":"2024-02-23T00:31:09.158030+0000","last_fullsized":"2024-02-23T00:31:09.158030+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128106+0000","last_change":"2024-02-23T00:30:48.692888+0000","last_active":"2024-02-23T00:31:09.128106+0000","last_peered":"2024-02-23T00:31:09.128106+0000","last_clean":"2024-02-23T00:31:09.128106+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:31:09.128106+0000","last_undegraded":"2024-02-23T00:31:09.128106+0000","last_fullsized":"2024-02-23T00:31:09.128106+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"100'4","reported_seq":37,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135545+0000","last_change":"2024-02-23T00:30:48.756342+0000","last_active":"2024-02-23T00:31:09.135545+0000","last_peered":"2024-02-23T00:31:09.135545+0000","last_clean":"2024-02-23T00:31:09.135545+0000","last_became_active":"2024-02-23T00:30:48.755689+0000","last_became_peered":"2024-02-23T00:30:48.755689+0000","last_unstale":"2024-02-23T00:31:09.135545+0000","last_undegraded":"2024-02-23T00:31:09.135545+0000","last_fullsized":"2024-02-23T00:31:09.135545+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165835,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.216648+0000","last_change":"2024-02-23T00:29:54.238061+0000","last_active":"2024-02-23T00:31:09.216648+0000","last_peered":"2024-02-23T00:31:09.216648+0000","last_clean":"2024-02-23T00:31:09.216648+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:31:09.216648+0000","last_undegraded":"2024-02-23T00:31:09.216648+0000","last_fullsized":"2024-02-23T00:31:09.216648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:54.237950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:54.237950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":191,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.192027+0000","last_change":"2024-02-23T00:30:29.527577+0000","last_active":"2024-02-23T00:31:10.192027+0000","last_peered":"2024-02-23T00:31:10.192027+0000","last_clean":"2024-02-23T00:31:10.192027+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:31:10.192027+0000","last_undegraded":"2024-02-23T00:31:10.192027+0000","last_fullsized":"2024-02-23T00:31:10.192027+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:30:29.527466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.527466+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":87,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123656+0000","last_change":"2024-02-23T00:30:47.090981+0000","last_active":"2024-02-23T00:31:09.123656+0000","last_peered":"2024-02-23T00:31:09.123656+0000","last_clean":"2024-02-23T00:31:09.123656+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:31:09.123656+0000","last_undegraded":"2024-02-23T00:31:09.123656+0000","last_fullsized":"2024-02-23T00:31:09.123656+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:30:47.090896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.090896+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.188558+0000","last_change":"2024-02-23T00:30:22.860665+0000","last_active":"2024-02-23T00:31:09.188558+0000","last_peered":"2024-02-23T00:31:09.188558+0000","last_clean":"2024-02-23T00:31:09.188558+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:31:09.188558+0000","last_undegraded":"2024-02-23T00:31:09.188558+0000","last_fullsized":"2024-02-23T00:31:09.188558+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:22.860565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.860565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.13","version":"98'9","reported_seq":40,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131617+0000","last_change":"2024-02-23T00:30:48.766029+0000","last_active":"2024-02-23T00:31:09.131617+0000","last_peered":"2024-02-23T00:31:09.131617+0000","last_clean":"2024-02-23T00:31:09.131617+0000","last_became_active":"2024-02-23T00:30:48.760177+0000","last_became_peered":"2024-02-23T00:30:48.760177+0000","last_unstale":"2024-02-23T00:31:09.131617+0000","last_undegraded":"2024-02-23T00:31:09.131617+0000","last_fullsized":"2024-02-23T00:31:09.131617+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131224+0000","last_change":"2024-02-23T00:30:48.701388+0000","last_active":"2024-02-23T00:31:09.131224+0000","last_peered":"2024-02-23T00:31:09.131224+0000","last_clean":"2024-02-23T00:31:09.131224+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:31:09.131224+0000","last_undegraded":"2024-02-23T00:31:09.131224+0000","last_fullsized":"2024-02-23T00:31:09.131224+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"99'2","reported_seq":37,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133386+0000","last_change":"2024-02-23T00:30:48.747631+0000","last_active":"2024-02-23T00:31:09.133386+0000","last_peered":"2024-02-23T00:31:09.133386+0000","last_clean":"2024-02-23T00:31:09.133386+0000","last_became_active":"2024-02-23T00:30:48.746677+0000","last_became_peered":"2024-02-23T00:30:48.746677+0000","last_unstale":"2024-02-23T00:31:09.133386+0000","last_undegraded":"2024-02-23T00:31:09.133386+0000","last_fullsized":"2024-02-23T00:31:09.133386+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.201425+0000","last_change":"2024-02-23T00:30:24.050993+0000","last_active":"2024-02-23T00:31:10.201425+0000","last_peered":"2024-02-23T00:31:10.201425+0000","last_clean":"2024-02-23T00:31:10.201425+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:31:10.201425+0000","last_undegraded":"2024-02-23T00:31:10.201425+0000","last_fullsized":"2024-02-23T00:31:10.201425+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:30:24.050887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:24.050887+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":111,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.149836+0000","last_change":"2024-02-23T00:30:31.441530+0000","last_active":"2024-02-23T00:31:09.149836+0000","last_peered":"2024-02-23T00:31:09.149836+0000","last_clean":"2024-02-23T00:31:09.149836+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:31:09.149836+0000","last_undegraded":"2024-02-23T00:31:09.149836+0000","last_fullsized":"2024-02-23T00:31:09.149836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:30:31.441418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.441418+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.176262+0000","last_change":"2024-02-23T00:30:44.543241+0000","last_active":"2024-02-23T00:31:09.176262+0000","last_peered":"2024-02-23T00:31:09.176262+0000","last_clean":"2024-02-23T00:31:09.176262+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:31:09.176262+0000","last_undegraded":"2024-02-23T00:31:09.176262+0000","last_fullsized":"2024-02-23T00:31:09.176262+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:44.543161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:30:44.543161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.161006+0000","last_change":"2024-02-23T00:30:25.064207+0000","last_active":"2024-02-23T00:31:09.161006+0000","last_peered":"2024-02-23T00:31:09.161006+0000","last_clean":"2024-02-23T00:31:09.161006+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:31:09.161006+0000","last_undegraded":"2024-02-23T00:31:09.161006+0000","last_fullsized":"2024-02-23T00:31:09.161006+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.064103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.064103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138894+0000","last_change":"2024-02-23T00:30:48.708244+0000","last_active":"2024-02-23T00:31:09.138894+0000","last_peered":"2024-02-23T00:31:09.138894+0000","last_clean":"2024-02-23T00:31:09.138894+0000","last_became_active":"2024-02-23T00:30:48.706292+0000","last_became_peered":"2024-02-23T00:30:48.706292+0000","last_unstale":"2024-02-23T00:31:09.138894+0000","last_undegraded":"2024-02-23T00:31:09.138894+0000","last_fullsized":"2024-02-23T00:31:09.138894+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120579+0000","last_change":"2024-02-23T00:30:48.616685+0000","last_active":"2024-02-23T00:31:09.120579+0000","last_peered":"2024-02-23T00:31:09.120579+0000","last_clean":"2024-02-23T00:31:09.120579+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:31:09.120579+0000","last_undegraded":"2024-02-23T00:31:09.120579+0000","last_fullsized":"2024-02-23T00:31:09.120579+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132671+0000","last_change":"2024-02-23T00:30:57.973513+0000","last_active":"2024-02-23T00:31:09.132671+0000","last_peered":"2024-02-23T00:31:09.132671+0000","last_clean":"2024-02-23T00:31:09.132671+0000","last_became_active":"2024-02-23T00:30:48.716293+0000","last_became_peered":"2024-02-23T00:30:48.716293+0000","last_unstale":"2024-02-23T00:31:09.132671+0000","last_undegraded":"2024-02-23T00:31:09.132671+0000","last_fullsized":"2024-02-23T00:31:09.132671+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:57.973446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:57.973446+0000","last_clean_scrub_stamp":"2024-02-23T00:30:57.973446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126115+0000","last_change":"2024-02-23T00:30:28.676431+0000","last_active":"2024-02-23T00:31:09.126115+0000","last_peered":"2024-02-23T00:31:09.126115+0000","last_clean":"2024-02-23T00:31:09.126115+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:31:09.126115+0000","last_undegraded":"2024-02-23T00:31:09.126115+0000","last_fullsized":"2024-02-23T00:31:09.126115+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:28.676326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:28.676326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":155,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122962+0000","last_change":"2024-02-23T00:29:01.733374+0000","last_active":"2024-02-23T00:31:09.122962+0000","last_peered":"2024-02-23T00:31:09.122962+0000","last_clean":"2024-02-23T00:31:09.122962+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:31:09.122962+0000","last_undegraded":"2024-02-23T00:31:09.122962+0000","last_fullsized":"2024-02-23T00:31:09.122962+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:29:01.733287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:01.733287+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132742+0000","last_change":"2024-02-23T00:30:47.142583+0000","last_active":"2024-02-23T00:31:09.132742+0000","last_peered":"2024-02-23T00:31:09.132742+0000","last_clean":"2024-02-23T00:31:09.132742+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:31:09.132742+0000","last_undegraded":"2024-02-23T00:31:09.132742+0000","last_fullsized":"2024-02-23T00:31:09.132742+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:30:47.142508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.142508+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.169274+0000","last_change":"2024-02-23T00:30:34.861845+0000","last_active":"2024-02-23T00:31:09.169274+0000","last_peered":"2024-02-23T00:31:09.169274+0000","last_clean":"2024-02-23T00:31:09.169274+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:31:09.169274+0000","last_undegraded":"2024-02-23T00:31:09.169274+0000","last_fullsized":"2024-02-23T00:31:09.169274+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:34.861741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:34.861741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.188688+0000","last_change":"2024-02-23T00:30:48.706243+0000","last_active":"2024-02-23T00:31:09.188688+0000","last_peered":"2024-02-23T00:31:09.188688+0000","last_clean":"2024-02-23T00:31:09.188688+0000","last_became_active":"2024-02-23T00:30:48.705705+0000","last_became_peered":"2024-02-23T00:30:48.705705+0000","last_unstale":"2024-02-23T00:31:09.188688+0000","last_undegraded":"2024-02-23T00:31:09.188688+0000","last_fullsized":"2024-02-23T00:31:09.188688+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120718+0000","last_change":"2024-02-23T00:30:48.674743+0000","last_active":"2024-02-23T00:31:09.120718+0000","last_peered":"2024-02-23T00:31:09.120718+0000","last_clean":"2024-02-23T00:31:09.120718+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:31:09.120718+0000","last_undegraded":"2024-02-23T00:31:09.120718+0000","last_fullsized":"2024-02-23T00:31:09.120718+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.18","version":"99'1","reported_seq":37,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125973+0000","last_change":"2024-02-23T00:30:45.852203+0000","last_active":"2024-02-23T00:31:09.125973+0000","last_peered":"2024-02-23T00:31:09.125973+0000","last_clean":"2024-02-23T00:31:09.125973+0000","last_became_active":"2024-02-23T00:30:45.851851+0000","last_became_peered":"2024-02-23T00:30:45.851851+0000","last_unstale":"2024-02-23T00:31:09.125973+0000","last_undegraded":"2024-02-23T00:31:09.125973+0000","last_fullsized":"2024-02-23T00:31:09.125973+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":127,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.217536+0000","last_change":"2024-02-23T00:30:44.885410+0000","last_active":"2024-02-23T00:31:09.217536+0000","last_peered":"2024-02-23T00:31:09.217536+0000","last_clean":"2024-02-23T00:31:09.217536+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:31:09.217536+0000","last_undegraded":"2024-02-23T00:31:09.217536+0000","last_fullsized":"2024-02-23T00:31:09.217536+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:30:44.885334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:44.885334+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126048+0000","last_change":"2024-02-23T00:29:47.780564+0000","last_active":"2024-02-23T00:31:09.126048+0000","last_peered":"2024-02-23T00:31:09.126048+0000","last_clean":"2024-02-23T00:31:09.126048+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:31:09.126048+0000","last_undegraded":"2024-02-23T00:31:09.126048+0000","last_fullsized":"2024-02-23T00:31:09.126048+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:47.780463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:47.780463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.185557+0000","last_change":"2024-02-23T00:30:36.894317+0000","last_active":"2024-02-23T00:31:09.185557+0000","last_peered":"2024-02-23T00:31:09.185557+0000","last_clean":"2024-02-23T00:31:09.185557+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:31:09.185557+0000","last_undegraded":"2024-02-23T00:31:09.185557+0000","last_fullsized":"2024-02-23T00:31:09.185557+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:36.894220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:36.894220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.159480+0000","last_change":"2024-02-23T00:30:24.708147+0000","last_active":"2024-02-23T00:31:09.159480+0000","last_peered":"2024-02-23T00:31:09.159480+0000","last_clean":"2024-02-23T00:31:09.159480+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:31:09.159480+0000","last_undegraded":"2024-02-23T00:31:09.159480+0000","last_fullsized":"2024-02-23T00:31:09.159480+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:24.708045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:24.708045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"98'5","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174868+0000","last_change":"2024-02-23T00:30:48.725952+0000","last_active":"2024-02-23T00:31:09.174868+0000","last_peered":"2024-02-23T00:31:09.174868+0000","last_clean":"2024-02-23T00:31:09.174868+0000","last_became_active":"2024-02-23T00:30:48.725566+0000","last_became_peered":"2024-02-23T00:30:48.725566+0000","last_unstale":"2024-02-23T00:31:09.174868+0000","last_undegraded":"2024-02-23T00:31:09.174868+0000","last_fullsized":"2024-02-23T00:31:09.174868+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.183037+0000","last_change":"2024-02-23T00:30:48.691960+0000","last_active":"2024-02-23T00:31:09.183037+0000","last_peered":"2024-02-23T00:31:09.183037+0000","last_clean":"2024-02-23T00:31:09.183037+0000","last_became_active":"2024-02-23T00:30:48.691713+0000","last_became_peered":"2024-02-23T00:30:48.691713+0000","last_unstale":"2024-02-23T00:31:09.183037+0000","last_undegraded":"2024-02-23T00:31:09.183037+0000","last_fullsized":"2024-02-23T00:31:09.183037+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174269+0000","last_change":"2024-02-23T00:30:45.840678+0000","last_active":"2024-02-23T00:31:09.174269+0000","last_peered":"2024-02-23T00:31:09.174269+0000","last_clean":"2024-02-23T00:31:09.174269+0000","last_became_active":"2024-02-23T00:30:45.840423+0000","last_became_peered":"2024-02-23T00:30:45.840423+0000","last_unstale":"2024-02-23T00:31:09.174269+0000","last_undegraded":"2024-02-23T00:31:09.174269+0000","last_fullsized":"2024-02-23T00:31:09.174269+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":124,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.185424+0000","last_change":"2024-02-23T00:30:24.802266+0000","last_active":"2024-02-23T00:31:09.185424+0000","last_peered":"2024-02-23T00:31:09.185424+0000","last_clean":"2024-02-23T00:31:09.185424+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:31:09.185424+0000","last_undegraded":"2024-02-23T00:31:09.185424+0000","last_fullsized":"2024-02-23T00:31:09.185424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:30:24.802134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:24.802134+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133302+0000","last_change":"2024-02-23T00:29:54.071478+0000","last_active":"2024-02-23T00:31:09.133302+0000","last_peered":"2024-02-23T00:31:09.133302+0000","last_clean":"2024-02-23T00:31:09.133302+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:31:09.133302+0000","last_undegraded":"2024-02-23T00:31:09.133302+0000","last_fullsized":"2024-02-23T00:31:09.133302+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:54.071374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:54.071374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":72,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128078+0000","last_change":"2024-02-23T00:29:10.044046+0000","last_active":"2024-02-23T00:31:09.128078+0000","last_peered":"2024-02-23T00:31:09.128078+0000","last_clean":"2024-02-23T00:31:09.128078+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:31:09.128078+0000","last_undegraded":"2024-02-23T00:31:09.128078+0000","last_fullsized":"2024-02-23T00:31:09.128078+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.043966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.043966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138155+0000","last_change":"2024-02-23T00:30:37.532548+0000","last_active":"2024-02-23T00:31:09.138155+0000","last_peered":"2024-02-23T00:31:09.138155+0000","last_clean":"2024-02-23T00:31:09.138155+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:31:09.138155+0000","last_undegraded":"2024-02-23T00:31:09.138155+0000","last_fullsized":"2024-02-23T00:31:09.138155+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:37.532441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.532441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"98'5","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131189+0000","last_change":"2024-02-23T00:30:48.712709+0000","last_active":"2024-02-23T00:31:09.131189+0000","last_peered":"2024-02-23T00:31:09.131189+0000","last_clean":"2024-02-23T00:31:09.131189+0000","last_became_active":"2024-02-23T00:30:48.711845+0000","last_became_peered":"2024-02-23T00:30:48.711845+0000","last_unstale":"2024-02-23T00:31:09.131189+0000","last_undegraded":"2024-02-23T00:31:09.131189+0000","last_fullsized":"2024-02-23T00:31:09.131189+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132598+0000","last_change":"2024-02-23T00:30:48.703519+0000","last_active":"2024-02-23T00:31:09.132598+0000","last_peered":"2024-02-23T00:31:09.132598+0000","last_clean":"2024-02-23T00:31:09.132598+0000","last_became_active":"2024-02-23T00:30:48.703232+0000","last_became_peered":"2024-02-23T00:30:48.703232+0000","last_unstale":"2024-02-23T00:31:09.132598+0000","last_undegraded":"2024-02-23T00:31:09.132598+0000","last_fullsized":"2024-02-23T00:31:09.132598+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135290+0000","last_change":"2024-02-23T00:30:48.756105+0000","last_active":"2024-02-23T00:31:09.135290+0000","last_peered":"2024-02-23T00:31:09.135290+0000","last_clean":"2024-02-23T00:31:09.135290+0000","last_became_active":"2024-02-23T00:30:48.754003+0000","last_became_peered":"2024-02-23T00:30:48.754003+0000","last_unstale":"2024-02-23T00:31:09.135290+0000","last_undegraded":"2024-02-23T00:31:09.135290+0000","last_fullsized":"2024-02-23T00:31:09.135290+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174715+0000","last_change":"2024-02-23T00:30:37.899931+0000","last_active":"2024-02-23T00:31:09.174715+0000","last_peered":"2024-02-23T00:31:09.174715+0000","last_clean":"2024-02-23T00:31:09.174715+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:31:09.174715+0000","last_undegraded":"2024-02-23T00:31:09.174715+0000","last_fullsized":"2024-02-23T00:31:09.174715+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:30:37.899845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.899845+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":81,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123516+0000","last_change":"2024-02-23T00:29:53.011199+0000","last_active":"2024-02-23T00:31:09.123516+0000","last_peered":"2024-02-23T00:31:09.123516+0000","last_clean":"2024-02-23T00:31:09.123516+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:31:09.123516+0000","last_undegraded":"2024-02-23T00:31:09.123516+0000","last_fullsized":"2024-02-23T00:31:09.123516+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:53.011089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:29:53.011089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":133,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.146483+0000","last_change":"2024-02-23T00:30:30.131030+0000","last_active":"2024-02-23T00:31:10.146483+0000","last_peered":"2024-02-23T00:31:10.146483+0000","last_clean":"2024-02-23T00:31:10.146483+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:31:10.146483+0000","last_undegraded":"2024-02-23T00:31:10.146483+0000","last_fullsized":"2024-02-23T00:31:10.146483+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:30:30.130950+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.130950+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.217385+0000","last_change":"2024-02-23T00:30:37.889427+0000","last_active":"2024-02-23T00:31:09.217385+0000","last_peered":"2024-02-23T00:31:09.217385+0000","last_clean":"2024-02-23T00:31:09.217385+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:31:09.217385+0000","last_undegraded":"2024-02-23T00:31:09.217385+0000","last_fullsized":"2024-02-23T00:31:09.217385+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:37.889336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:37.889336+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.889336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.3","version":"98'5","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121429+0000","last_change":"2024-02-23T00:30:48.683688+0000","last_active":"2024-02-23T00:31:09.121429+0000","last_peered":"2024-02-23T00:31:09.121429+0000","last_clean":"2024-02-23T00:31:09.121429+0000","last_became_active":"2024-02-23T00:30:48.682705+0000","last_became_peered":"2024-02-23T00:30:48.682705+0000","last_unstale":"2024-02-23T00:31:09.121429+0000","last_undegraded":"2024-02-23T00:31:09.121429+0000","last_fullsized":"2024-02-23T00:31:09.121429+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128530+0000","last_change":"2024-02-23T00:30:48.688217+0000","last_active":"2024-02-23T00:31:09.128530+0000","last_peered":"2024-02-23T00:31:09.128530+0000","last_clean":"2024-02-23T00:31:09.128530+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:31:09.128530+0000","last_undegraded":"2024-02-23T00:31:09.128530+0000","last_fullsized":"2024-02-23T00:31:09.128530+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"99'2","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.169131+0000","last_change":"2024-02-23T00:30:45.849077+0000","last_active":"2024-02-23T00:31:09.169131+0000","last_peered":"2024-02-23T00:31:09.169131+0000","last_clean":"2024-02-23T00:31:09.169131+0000","last_became_active":"2024-02-23T00:30:45.848757+0000","last_became_peered":"2024-02-23T00:30:45.848757+0000","last_unstale":"2024-02-23T00:31:09.169131+0000","last_undegraded":"2024-02-23T00:31:09.169131+0000","last_fullsized":"2024-02-23T00:31:09.169131+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"63'15","reported_seq":105,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125310+0000","last_change":"2024-02-23T00:28:55.907616+0000","last_active":"2024-02-23T00:31:09.125310+0000","last_peered":"2024-02-23T00:31:09.125310+0000","last_clean":"2024-02-23T00:31:09.125310+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:31:09.125310+0000","last_undegraded":"2024-02-23T00:31:09.125310+0000","last_fullsized":"2024-02-23T00:31:09.125310+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:28:55.907528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:28:55.907528+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.195934+0000","last_change":"2024-02-23T00:29:55.237158+0000","last_active":"2024-02-23T00:31:09.195934+0000","last_peered":"2024-02-23T00:31:09.195934+0000","last_clean":"2024-02-23T00:31:09.195934+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:31:09.195934+0000","last_undegraded":"2024-02-23T00:31:09.195934+0000","last_fullsized":"2024-02-23T00:31:09.195934+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:55.237043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:55.237043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'9","reported_seq":135,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.155661+0000","last_change":"2024-02-23T00:29:38.098504+0000","last_active":"2024-02-23T00:31:10.155661+0000","last_peered":"2024-02-23T00:31:10.155661+0000","last_clean":"2024-02-23T00:31:10.155661+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:31:10.155661+0000","last_undegraded":"2024-02-23T00:31:10.155661+0000","last_fullsized":"2024-02-23T00:31:10.155661+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.098442+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132945+0000","last_change":"2024-02-23T00:30:29.051519+0000","last_active":"2024-02-23T00:31:09.132945+0000","last_peered":"2024-02-23T00:31:09.132945+0000","last_clean":"2024-02-23T00:31:09.132945+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:31:09.132945+0000","last_undegraded":"2024-02-23T00:31:09.132945+0000","last_fullsized":"2024-02-23T00:31:09.132945+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:29.051438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.051438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135947+0000","last_change":"2024-02-23T00:30:48.734850+0000","last_active":"2024-02-23T00:31:09.135947+0000","last_peered":"2024-02-23T00:31:09.135947+0000","last_clean":"2024-02-23T00:31:09.135947+0000","last_became_active":"2024-02-23T00:30:48.732126+0000","last_became_peered":"2024-02-23T00:30:48.732126+0000","last_unstale":"2024-02-23T00:31:09.135947+0000","last_undegraded":"2024-02-23T00:31:09.135947+0000","last_fullsized":"2024-02-23T00:31:09.135947+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.185623+0000","last_change":"2024-02-23T00:30:48.717524+0000","last_active":"2024-02-23T00:31:09.185623+0000","last_peered":"2024-02-23T00:31:09.185623+0000","last_clean":"2024-02-23T00:31:09.185623+0000","last_became_active":"2024-02-23T00:30:48.716104+0000","last_became_peered":"2024-02-23T00:30:48.716104+0000","last_unstale":"2024-02-23T00:31:09.185623+0000","last_undegraded":"2024-02-23T00:31:09.185623+0000","last_fullsized":"2024-02-23T00:31:09.185623+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1c","version":"98'1","reported_seq":35,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130226+0000","last_change":"2024-02-23T00:30:48.684299+0000","last_active":"2024-02-23T00:31:09.130226+0000","last_peered":"2024-02-23T00:31:09.130226+0000","last_clean":"2024-02-23T00:31:09.130226+0000","last_became_active":"2024-02-23T00:30:48.683921+0000","last_became_peered":"2024-02-23T00:30:48.683921+0000","last_unstale":"2024-02-23T00:31:09.130226+0000","last_undegraded":"2024-02-23T00:31:09.130226+0000","last_fullsized":"2024-02-23T00:31:09.130226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1024,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.149684+0000","last_change":"2024-02-23T00:30:23.600452+0000","last_active":"2024-02-23T00:31:09.149684+0000","last_peered":"2024-02-23T00:31:09.149684+0000","last_clean":"2024-02-23T00:31:09.149684+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:31:09.149684+0000","last_undegraded":"2024-02-23T00:31:09.149684+0000","last_fullsized":"2024-02-23T00:31:09.149684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:30:23.600311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.600311+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130209+0000","last_change":"2024-02-23T00:29:51.021137+0000","last_active":"2024-02-23T00:31:09.130209+0000","last_peered":"2024-02-23T00:31:09.130209+0000","last_clean":"2024-02-23T00:31:09.130209+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:31:09.130209+0000","last_undegraded":"2024-02-23T00:31:09.130209+0000","last_fullsized":"2024-02-23T00:31:09.130209+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:51.021056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:51.021056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.169236+0000","last_change":"2024-02-23T00:29:32.122532+0000","last_active":"2024-02-23T00:31:09.169236+0000","last_peered":"2024-02-23T00:31:09.169236+0000","last_clean":"2024-02-23T00:31:09.169236+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:31:09.169236+0000","last_undegraded":"2024-02-23T00:31:09.169236+0000","last_fullsized":"2024-02-23T00:31:09.169236+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:32.122430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:32.122430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.160884+0000","last_change":"2024-02-23T00:30:23.080974+0000","last_active":"2024-02-23T00:31:09.160884+0000","last_peered":"2024-02-23T00:31:09.160884+0000","last_clean":"2024-02-23T00:31:09.160884+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:31:09.160884+0000","last_undegraded":"2024-02-23T00:31:09.160884+0000","last_fullsized":"2024-02-23T00:31:09.160884+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:23.080873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.080873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132529+0000","last_change":"2024-02-23T00:30:48.672730+0000","last_active":"2024-02-23T00:31:09.132529+0000","last_peered":"2024-02-23T00:31:09.132529+0000","last_clean":"2024-02-23T00:31:09.132529+0000","last_became_active":"2024-02-23T00:30:48.671603+0000","last_became_peered":"2024-02-23T00:30:48.671603+0000","last_unstale":"2024-02-23T00:31:09.132529+0000","last_undegraded":"2024-02-23T00:31:09.132529+0000","last_fullsized":"2024-02-23T00:31:09.132529+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120502+0000","last_change":"2024-02-23T00:30:48.668969+0000","last_active":"2024-02-23T00:31:09.120502+0000","last_peered":"2024-02-23T00:31:09.120502+0000","last_clean":"2024-02-23T00:31:09.120502+0000","last_became_active":"2024-02-23T00:30:48.666975+0000","last_became_peered":"2024-02-23T00:30:48.666975+0000","last_unstale":"2024-02-23T00:31:09.120502+0000","last_undegraded":"2024-02-23T00:31:09.120502+0000","last_fullsized":"2024-02-23T00:31:09.120502+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129080+0000","last_change":"2024-02-23T00:30:48.672408+0000","last_active":"2024-02-23T00:31:09.129080+0000","last_peered":"2024-02-23T00:31:09.129080+0000","last_clean":"2024-02-23T00:31:09.129080+0000","last_became_active":"2024-02-23T00:30:48.665189+0000","last_became_peered":"2024-02-23T00:30:48.665189+0000","last_unstale":"2024-02-23T00:31:09.129080+0000","last_undegraded":"2024-02-23T00:31:09.129080+0000","last_fullsized":"2024-02-23T00:31:09.129080+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":121,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.124793+0000","last_change":"2024-02-23T00:30:41.077799+0000","last_active":"2024-02-23T00:31:09.124793+0000","last_peered":"2024-02-23T00:31:09.124793+0000","last_clean":"2024-02-23T00:31:09.124793+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:31:09.124793+0000","last_undegraded":"2024-02-23T00:31:09.124793+0000","last_fullsized":"2024-02-23T00:31:09.124793+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:30:41.077681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.077681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.183929+0000","last_change":"2024-02-23T00:29:02.711400+0000","last_active":"2024-02-23T00:31:09.183929+0000","last_peered":"2024-02-23T00:31:09.183929+0000","last_clean":"2024-02-23T00:31:09.183929+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:31:09.183929+0000","last_undegraded":"2024-02-23T00:31:09.183929+0000","last_fullsized":"2024-02-23T00:31:09.183929+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:29:02.711284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.149758+0000","last_change":"2024-02-23T00:30:30.486006+0000","last_active":"2024-02-23T00:31:09.149758+0000","last_peered":"2024-02-23T00:31:09.149758+0000","last_clean":"2024-02-23T00:31:09.149758+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:31:09.149758+0000","last_undegraded":"2024-02-23T00:31:09.149758+0000","last_fullsized":"2024-02-23T00:31:09.149758+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:30.485900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.485900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.162886+0000","last_change":"2024-02-23T00:30:47.843046+0000","last_active":"2024-02-23T00:31:09.162886+0000","last_peered":"2024-02-23T00:31:09.162886+0000","last_clean":"2024-02-23T00:31:09.162886+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:31:09.162886+0000","last_undegraded":"2024-02-23T00:31:09.162886+0000","last_fullsized":"2024-02-23T00:31:09.162886+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.842931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.176664+0000","last_change":"2024-02-23T00:30:48.721760+0000","last_active":"2024-02-23T00:31:09.176664+0000","last_peered":"2024-02-23T00:31:09.176664+0000","last_clean":"2024-02-23T00:31:09.176664+0000","last_became_active":"2024-02-23T00:30:48.721541+0000","last_became_peered":"2024-02-23T00:30:48.721541+0000","last_unstale":"2024-02-23T00:31:09.176664+0000","last_undegraded":"2024-02-23T00:31:09.176664+0000","last_fullsized":"2024-02-23T00:31:09.176664+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":43,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133597+0000","last_change":"2024-02-23T00:30:48.705018+0000","last_active":"2024-02-23T00:31:09.133597+0000","last_peered":"2024-02-23T00:31:09.133597+0000","last_clean":"2024-02-23T00:31:09.133597+0000","last_became_active":"2024-02-23T00:30:48.703281+0000","last_became_peered":"2024-02-23T00:30:48.703281+0000","last_unstale":"2024-02-23T00:31:09.133597+0000","last_undegraded":"2024-02-23T00:31:09.133597+0000","last_fullsized":"2024-02-23T00:31:09.133597+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1e","version":"100'2","reported_seq":43,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.217331+0000","last_change":"2024-02-23T00:30:45.864076+0000","last_active":"2024-02-23T00:31:09.217331+0000","last_peered":"2024-02-23T00:31:09.217331+0000","last_clean":"2024-02-23T00:31:09.217331+0000","last_became_active":"2024-02-23T00:30:45.863776+0000","last_became_peered":"2024-02-23T00:30:45.863776+0000","last_unstale":"2024-02-23T00:31:09.217331+0000","last_undegraded":"2024-02-23T00:31:09.217331+0000","last_fullsized":"2024-02-23T00:31:09.217331+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.130995+0000","last_change":"2024-02-23T00:29:43.108230+0000","last_active":"2024-02-23T00:31:09.130995+0000","last_peered":"2024-02-23T00:31:09.130995+0000","last_clean":"2024-02-23T00:31:09.130995+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:31:09.130995+0000","last_undegraded":"2024-02-23T00:31:09.130995+0000","last_fullsized":"2024-02-23T00:31:09.130995+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:29:43.108156+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:29:43.108156+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":81,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129560+0000","last_change":"2024-02-23T00:30:26.059320+0000","last_active":"2024-02-23T00:31:09.129560+0000","last_peered":"2024-02-23T00:31:09.129560+0000","last_clean":"2024-02-23T00:31:09.129560+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:31:09.129560+0000","last_undegraded":"2024-02-23T00:31:09.129560+0000","last_fullsized":"2024-02-23T00:31:09.129560+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:26.059244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:26.059244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133233+0000","last_change":"2024-02-23T00:30:25.163377+0000","last_active":"2024-02-23T00:31:09.133233+0000","last_peered":"2024-02-23T00:31:09.133233+0000","last_clean":"2024-02-23T00:31:09.133233+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:31:09.133233+0000","last_undegraded":"2024-02-23T00:31:09.133233+0000","last_fullsized":"2024-02-23T00:31:09.133233+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.163297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.163297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138108+0000","last_change":"2024-02-23T00:30:34.561832+0000","last_active":"2024-02-23T00:31:09.138108+0000","last_peered":"2024-02-23T00:31:09.138108+0000","last_clean":"2024-02-23T00:31:09.138108+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:31:09.138108+0000","last_undegraded":"2024-02-23T00:31:09.138108+0000","last_fullsized":"2024-02-23T00:31:09.138108+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:34.561766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:34.561766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.160919+0000","last_change":"2024-02-23T00:30:48.709585+0000","last_active":"2024-02-23T00:31:09.160919+0000","last_peered":"2024-02-23T00:31:09.160919+0000","last_clean":"2024-02-23T00:31:09.160919+0000","last_became_active":"2024-02-23T00:30:48.709325+0000","last_became_peered":"2024-02-23T00:30:48.709325+0000","last_unstale":"2024-02-23T00:31:09.160919+0000","last_undegraded":"2024-02-23T00:31:09.160919+0000","last_fullsized":"2024-02-23T00:31:09.160919+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.149908+0000","last_change":"2024-02-23T00:30:48.696251+0000","last_active":"2024-02-23T00:31:09.149908+0000","last_peered":"2024-02-23T00:31:09.149908+0000","last_clean":"2024-02-23T00:31:09.149908+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:31:09.149908+0000","last_undegraded":"2024-02-23T00:31:09.149908+0000","last_fullsized":"2024-02-23T00:31:09.149908+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1f","version":"100'1","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135701+0000","last_change":"2024-02-23T00:30:48.729600+0000","last_active":"2024-02-23T00:31:09.135701+0000","last_peered":"2024-02-23T00:31:09.135701+0000","last_clean":"2024-02-23T00:31:09.135701+0000","last_became_active":"2024-02-23T00:30:48.728700+0000","last_became_peered":"2024-02-23T00:30:48.728700+0000","last_unstale":"2024-02-23T00:31:09.135701+0000","last_undegraded":"2024-02-23T00:31:09.135701+0000","last_fullsized":"2024-02-23T00:31:09.135701+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":117,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.163483+0000","last_change":"2024-02-23T00:28:57.039721+0000","last_active":"2024-02-23T00:31:09.163483+0000","last_peered":"2024-02-23T00:31:09.163483+0000","last_clean":"2024-02-23T00:31:09.163483+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:31:09.163483+0000","last_undegraded":"2024-02-23T00:31:09.163483+0000","last_fullsized":"2024-02-23T00:31:09.163483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.039660+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":82,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132741+0000","last_change":"2024-02-23T00:30:22.245444+0000","last_active":"2024-02-23T00:31:09.132741+0000","last_peered":"2024-02-23T00:31:09.132741+0000","last_clean":"2024-02-23T00:31:09.132741+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:31:09.132741+0000","last_undegraded":"2024-02-23T00:31:09.132741+0000","last_fullsized":"2024-02-23T00:31:09.132741+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:22.245359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.245359+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131910+0000","last_change":"2024-02-23T00:30:33.528773+0000","last_active":"2024-02-23T00:31:09.131910+0000","last_peered":"2024-02-23T00:31:09.131910+0000","last_clean":"2024-02-23T00:31:09.131910+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:31:09.131910+0000","last_undegraded":"2024-02-23T00:31:09.131910+0000","last_fullsized":"2024-02-23T00:31:09.131910+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:33.528703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:33.528703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.169354+0000","last_change":"2024-02-23T00:30:31.829814+0000","last_active":"2024-02-23T00:31:09.169354+0000","last_peered":"2024-02-23T00:31:09.169354+0000","last_clean":"2024-02-23T00:31:09.169354+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:31:09.169354+0000","last_undegraded":"2024-02-23T00:31:09.169354+0000","last_fullsized":"2024-02-23T00:31:09.169354+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.829697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.829697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.6","version":"98'9","reported_seq":40,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131966+0000","last_change":"2024-02-23T00:30:48.706598+0000","last_active":"2024-02-23T00:31:09.131966+0000","last_peered":"2024-02-23T00:31:09.131966+0000","last_clean":"2024-02-23T00:31:09.131966+0000","last_became_active":"2024-02-23T00:30:48.699445+0000","last_became_peered":"2024-02-23T00:30:48.699445+0000","last_unstale":"2024-02-23T00:31:09.131966+0000","last_undegraded":"2024-02-23T00:31:09.131966+0000","last_fullsized":"2024-02-23T00:31:09.131966+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125141+0000","last_change":"2024-02-23T00:30:48.723611+0000","last_active":"2024-02-23T00:31:09.125141+0000","last_peered":"2024-02-23T00:31:09.125141+0000","last_clean":"2024-02-23T00:31:09.125141+0000","last_became_active":"2024-02-23T00:30:48.722602+0000","last_became_peered":"2024-02-23T00:30:48.722602+0000","last_unstale":"2024-02-23T00:31:09.125141+0000","last_undegraded":"2024-02-23T00:31:09.125141+0000","last_fullsized":"2024-02-23T00:31:09.125141+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.10","version":"98'2","reported_seq":35,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120234+0000","last_change":"2024-02-23T00:30:48.709763+0000","last_active":"2024-02-23T00:31:09.120234+0000","last_peered":"2024-02-23T00:31:09.120234+0000","last_clean":"2024-02-23T00:31:09.120234+0000","last_became_active":"2024-02-23T00:30:48.709247+0000","last_became_peered":"2024-02-23T00:30:48.709247+0000","last_unstale":"2024-02-23T00:31:09.120234+0000","last_undegraded":"2024-02-23T00:31:09.120234+0000","last_fullsized":"2024-02-23T00:31:09.120234+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"92'22","reported_seq":117,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.217713+0000","last_change":"2024-02-23T00:30:38.900864+0000","last_active":"2024-02-23T00:31:09.217713+0000","last_peered":"2024-02-23T00:31:09.217713+0000","last_clean":"2024-02-23T00:31:09.217713+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:31:09.217713+0000","last_undegraded":"2024-02-23T00:31:09.217713+0000","last_fullsized":"2024-02-23T00:31:09.217713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.900761+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122208+0000","last_change":"2024-02-23T00:29:03.017128+0000","last_active":"2024-02-23T00:31:09.122208+0000","last_peered":"2024-02-23T00:31:09.122208+0000","last_clean":"2024-02-23T00:31:09.122208+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:31:09.122208+0000","last_undegraded":"2024-02-23T00:31:09.122208+0000","last_fullsized":"2024-02-23T00:31:09.122208+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.017051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.017051+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-23T00:31:10.153586+0000","last_change":"2024-02-23T00:29:38.884205+0000","last_active":"2024-02-23T00:31:10.153586+0000","last_peered":"2024-02-23T00:31:10.153586+0000","last_clean":"2024-02-23T00:31:10.153586+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:31:10.153586+0000","last_undegraded":"2024-02-23T00:31:10.153586+0000","last_fullsized":"2024-02-23T00:31:10.153586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.884131+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133824+0000","last_change":"2024-02-23T00:30:35.523228+0000","last_active":"2024-02-23T00:31:09.133824+0000","last_peered":"2024-02-23T00:31:09.133824+0000","last_clean":"2024-02-23T00:31:09.133824+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:31:09.133824+0000","last_undegraded":"2024-02-23T00:31:09.133824+0000","last_fullsized":"2024-02-23T00:31:09.133824+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:30:35.523157+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.523157+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131508+0000","last_change":"2024-02-23T00:30:48.695030+0000","last_active":"2024-02-23T00:31:09.131508+0000","last_peered":"2024-02-23T00:31:09.131508+0000","last_clean":"2024-02-23T00:31:09.131508+0000","last_became_active":"2024-02-23T00:30:48.694604+0000","last_became_peered":"2024-02-23T00:30:48.694604+0000","last_unstale":"2024-02-23T00:31:09.131508+0000","last_undegraded":"2024-02-23T00:31:09.131508+0000","last_fullsized":"2024-02-23T00:31:09.131508+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133898+0000","last_change":"2024-02-23T00:30:48.748188+0000","last_active":"2024-02-23T00:31:09.133898+0000","last_peered":"2024-02-23T00:31:09.133898+0000","last_clean":"2024-02-23T00:31:09.133898+0000","last_became_active":"2024-02-23T00:30:48.745982+0000","last_became_peered":"2024-02-23T00:30:48.745982+0000","last_unstale":"2024-02-23T00:31:09.133898+0000","last_undegraded":"2024-02-23T00:31:09.133898+0000","last_fullsized":"2024-02-23T00:31:09.133898+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"100'2","reported_seq":35,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.124660+0000","last_change":"2024-02-23T00:30:48.739601+0000","last_active":"2024-02-23T00:31:09.124660+0000","last_peered":"2024-02-23T00:31:09.124660+0000","last_clean":"2024-02-23T00:31:09.124660+0000","last_became_active":"2024-02-23T00:30:48.736094+0000","last_became_peered":"2024-02-23T00:30:48.736094+0000","last_unstale":"2024-02-23T00:31:09.124660+0000","last_undegraded":"2024-02-23T00:31:09.124660+0000","last_fullsized":"2024-02-23T00:31:09.124660+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2800,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":115,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133029+0000","last_change":"2024-02-23T00:30:30.207937+0000","last_active":"2024-02-23T00:31:09.133029+0000","last_peered":"2024-02-23T00:31:09.133029+0000","last_clean":"2024-02-23T00:31:09.133029+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:31:09.133029+0000","last_undegraded":"2024-02-23T00:31:09.133029+0000","last_fullsized":"2024-02-23T00:31:09.133029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:30:30.207850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.207850+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.163223+0000","last_change":"2024-02-23T00:29:49.056255+0000","last_active":"2024-02-23T00:31:09.163223+0000","last_peered":"2024-02-23T00:31:09.163223+0000","last_clean":"2024-02-23T00:31:09.163223+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:31:09.163223+0000","last_undegraded":"2024-02-23T00:31:09.163223+0000","last_fullsized":"2024-02-23T00:31:09.163223+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:49.056184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:49.056184+0000","last_clean_scrub_stamp":"2024-02-23T00:29:49.056184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.217758+0000","last_change":"2024-02-23T00:30:34.869530+0000","last_active":"2024-02-23T00:31:09.217758+0000","last_peered":"2024-02-23T00:31:09.217758+0000","last_clean":"2024-02-23T00:31:09.217758+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:31:09.217758+0000","last_undegraded":"2024-02-23T00:31:09.217758+0000","last_fullsized":"2024-02-23T00:31:09.217758+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:34.869440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:34.869440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.183736+0000","last_change":"2024-02-23T00:29:37.520506+0000","last_active":"2024-02-23T00:31:09.183736+0000","last_peered":"2024-02-23T00:31:09.183736+0000","last_clean":"2024-02-23T00:31:09.183736+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:31:09.183736+0000","last_undegraded":"2024-02-23T00:31:09.183736+0000","last_fullsized":"2024-02-23T00:31:09.183736+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:37.520440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:37.520440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.8","version":"98'2","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.158165+0000","last_change":"2024-02-23T00:30:48.782205+0000","last_active":"2024-02-23T00:31:09.158165+0000","last_peered":"2024-02-23T00:31:09.158165+0000","last_clean":"2024-02-23T00:31:09.158165+0000","last_became_active":"2024-02-23T00:30:48.781887+0000","last_became_peered":"2024-02-23T00:30:48.781887+0000","last_unstale":"2024-02-23T00:31:09.158165+0000","last_undegraded":"2024-02-23T00:31:09.158165+0000","last_fullsized":"2024-02-23T00:31:09.158165+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138819+0000","last_change":"2024-02-23T00:30:48.737859+0000","last_active":"2024-02-23T00:31:09.138819+0000","last_peered":"2024-02-23T00:31:09.138819+0000","last_clean":"2024-02-23T00:31:09.138819+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:31:09.138819+0000","last_undegraded":"2024-02-23T00:31:09.138819+0000","last_fullsized":"2024-02-23T00:31:09.138819+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135838+0000","last_change":"2024-02-23T00:30:48.699036+0000","last_active":"2024-02-23T00:31:09.135838+0000","last_peered":"2024-02-23T00:31:09.135838+0000","last_clean":"2024-02-23T00:31:09.135838+0000","last_became_active":"2024-02-23T00:30:48.698750+0000","last_became_peered":"2024-02-23T00:30:48.698750+0000","last_unstale":"2024-02-23T00:31:09.135838+0000","last_undegraded":"2024-02-23T00:31:09.135838+0000","last_fullsized":"2024-02-23T00:31:09.135838+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.159929+0000","last_change":"2024-02-23T00:29:31.823473+0000","last_active":"2024-02-23T00:31:09.159929+0000","last_peered":"2024-02-23T00:31:09.159929+0000","last_clean":"2024-02-23T00:31:09.159929+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:31:09.159929+0000","last_undegraded":"2024-02-23T00:31:09.159929+0000","last_fullsized":"2024-02-23T00:31:09.159929+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:29:31.823405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:31.823405+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":88,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.135914+0000","last_change":"2024-02-23T00:30:40.183533+0000","last_active":"2024-02-23T00:31:09.135914+0000","last_peered":"2024-02-23T00:31:09.135914+0000","last_clean":"2024-02-23T00:31:09.135914+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:31:09.135914+0000","last_undegraded":"2024-02-23T00:31:09.135914+0000","last_fullsized":"2024-02-23T00:31:09.135914+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.183453+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.184238+0000","last_change":"2024-02-23T00:29:05.641525+0000","last_active":"2024-02-23T00:31:09.184238+0000","last_peered":"2024-02-23T00:31:09.184238+0000","last_clean":"2024-02-23T00:31:09.184238+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:31:09.184238+0000","last_undegraded":"2024-02-23T00:31:09.184238+0000","last_fullsized":"2024-02-23T00:31:09.184238+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:05.641454+0000","last_clean_scrub_stamp":"2024-02-23T00:29:05.641454+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":72,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131439+0000","last_change":"2024-02-23T00:29:10.830479+0000","last_active":"2024-02-23T00:31:09.131439+0000","last_peered":"2024-02-23T00:31:09.131439+0000","last_clean":"2024-02-23T00:31:09.131439+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:31:09.131439+0000","last_undegraded":"2024-02-23T00:31:09.131439+0000","last_fullsized":"2024-02-23T00:31:09.131439+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.830389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.830389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.b","version":"98'2","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.124659+0000","last_change":"2024-02-23T00:30:48.695895+0000","last_active":"2024-02-23T00:31:09.124659+0000","last_peered":"2024-02-23T00:31:09.124659+0000","last_clean":"2024-02-23T00:31:09.124659+0000","last_became_active":"2024-02-23T00:30:48.695060+0000","last_became_peered":"2024-02-23T00:30:48.695060+0000","last_unstale":"2024-02-23T00:31:09.124659+0000","last_undegraded":"2024-02-23T00:31:09.124659+0000","last_fullsized":"2024-02-23T00:31:09.124659+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128458+0000","last_change":"2024-02-23T00:30:48.711374+0000","last_active":"2024-02-23T00:31:09.128458+0000","last_peered":"2024-02-23T00:31:09.128458+0000","last_clean":"2024-02-23T00:31:09.128458+0000","last_became_active":"2024-02-23T00:30:48.710049+0000","last_became_peered":"2024-02-23T00:30:48.710049+0000","last_unstale":"2024-02-23T00:31:09.128458+0000","last_undegraded":"2024-02-23T00:31:09.128458+0000","last_fullsized":"2024-02-23T00:31:09.128458+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.156765+0000","last_change":"2024-02-23T00:30:45.857144+0000","last_active":"2024-02-23T00:31:09.156765+0000","last_peered":"2024-02-23T00:31:09.156765+0000","last_clean":"2024-02-23T00:31:09.156765+0000","last_became_active":"2024-02-23T00:30:45.856020+0000","last_became_peered":"2024-02-23T00:30:45.856020+0000","last_unstale":"2024-02-23T00:31:09.156765+0000","last_undegraded":"2024-02-23T00:31:09.156765+0000","last_fullsized":"2024-02-23T00:31:09.156765+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":125,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.184494+0000","last_change":"2024-02-23T00:30:29.844070+0000","last_active":"2024-02-23T00:31:09.184494+0000","last_peered":"2024-02-23T00:31:09.184494+0000","last_clean":"2024-02-23T00:31:09.184494+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:31:09.184494+0000","last_undegraded":"2024-02-23T00:31:09.184494+0000","last_fullsized":"2024-02-23T00:31:09.184494+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:30:29.843954+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.843954+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.169203+0000","last_change":"2024-02-23T00:30:32.878666+0000","last_active":"2024-02-23T00:31:09.169203+0000","last_peered":"2024-02-23T00:31:09.169203+0000","last_clean":"2024-02-23T00:31:09.169203+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:31:09.169203+0000","last_undegraded":"2024-02-23T00:31:09.169203+0000","last_fullsized":"2024-02-23T00:31:09.169203+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:32.878569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:32.878569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.128498+0000","last_change":"2024-02-23T00:29:03.790608+0000","last_active":"2024-02-23T00:31:09.128498+0000","last_peered":"2024-02-23T00:31:09.128498+0000","last_clean":"2024-02-23T00:31:09.128498+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:31:09.128498+0000","last_undegraded":"2024-02-23T00:31:09.128498+0000","last_fullsized":"2024-02-23T00:31:09.128498+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:03.790530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:03.790530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.185348+0000","last_change":"2024-02-23T00:29:47.044859+0000","last_active":"2024-02-23T00:31:09.185348+0000","last_peered":"2024-02-23T00:31:09.185348+0000","last_clean":"2024-02-23T00:31:09.185348+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:31:09.185348+0000","last_undegraded":"2024-02-23T00:31:09.185348+0000","last_fullsized":"2024-02-23T00:31:09.185348+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:47.044799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:47.044799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.a","version":"98'10","reported_seq":41,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.129511+0000","last_change":"2024-02-23T00:30:48.751955+0000","last_active":"2024-02-23T00:31:09.129511+0000","last_peered":"2024-02-23T00:31:09.129511+0000","last_clean":"2024-02-23T00:31:09.129511+0000","last_became_active":"2024-02-23T00:30:48.726997+0000","last_became_peered":"2024-02-23T00:30:48.726997+0000","last_unstale":"2024-02-23T00:31:09.129511+0000","last_undegraded":"2024-02-23T00:31:09.129511+0000","last_fullsized":"2024-02-23T00:31:09.129511+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.122950+0000","last_change":"2024-02-23T00:30:48.679036+0000","last_active":"2024-02-23T00:31:09.122950+0000","last_peered":"2024-02-23T00:31:09.122950+0000","last_clean":"2024-02-23T00:31:09.122950+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:31:09.122950+0000","last_undegraded":"2024-02-23T00:31:09.122950+0000","last_fullsized":"2024-02-23T00:31:09.122950+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.14","version":"98'3","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.176116+0000","last_change":"2024-02-23T00:30:45.853178+0000","last_active":"2024-02-23T00:31:09.176116+0000","last_peered":"2024-02-23T00:31:09.176116+0000","last_clean":"2024-02-23T00:31:09.176116+0000","last_became_active":"2024-02-23T00:30:45.852871+0000","last_became_peered":"2024-02-23T00:30:45.852871+0000","last_unstale":"2024-02-23T00:31:09.176116+0000","last_undegraded":"2024-02-23T00:31:09.176116+0000","last_fullsized":"2024-02-23T00:31:09.176116+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":127,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125950+0000","last_change":"2024-02-23T00:29:38.857115+0000","last_active":"2024-02-23T00:31:09.125950+0000","last_peered":"2024-02-23T00:31:09.125950+0000","last_clean":"2024-02-23T00:31:09.125950+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:31:09.125950+0000","last_undegraded":"2024-02-23T00:31:09.125950+0000","last_fullsized":"2024-02-23T00:31:09.125950+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:29:38.857029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.857029+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":82,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.133502+0000","last_change":"2024-02-23T00:30:47.629232+0000","last_active":"2024-02-23T00:31:09.133502+0000","last_peered":"2024-02-23T00:31:09.133502+0000","last_clean":"2024-02-23T00:31:09.133502+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:31:09.133502+0000","last_undegraded":"2024-02-23T00:31:09.133502+0000","last_fullsized":"2024-02-23T00:31:09.133502+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.629146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.629146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.121158+0000","last_change":"2024-02-23T00:30:41.121986+0000","last_active":"2024-02-23T00:31:09.121158+0000","last_peered":"2024-02-23T00:31:09.121158+0000","last_clean":"2024-02-23T00:31:09.121158+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:31:09.121158+0000","last_undegraded":"2024-02-23T00:31:09.121158+0000","last_fullsized":"2024-02-23T00:31:09.121158+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:41.121916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.121916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.162161+0000","last_change":"2024-02-23T00:30:31.986713+0000","last_active":"2024-02-23T00:31:09.162161+0000","last_peered":"2024-02-23T00:31:09.162161+0000","last_clean":"2024-02-23T00:31:09.162161+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:31:09.162161+0000","last_undegraded":"2024-02-23T00:31:09.162161+0000","last_fullsized":"2024-02-23T00:31:09.162161+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.986615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.986615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.d","version":"98'5","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.184386+0000","last_change":"2024-02-23T00:30:48.749817+0000","last_active":"2024-02-23T00:31:09.184386+0000","last_peered":"2024-02-23T00:31:09.184386+0000","last_clean":"2024-02-23T00:31:09.184386+0000","last_became_active":"2024-02-23T00:30:48.740849+0000","last_became_peered":"2024-02-23T00:30:48.740849+0000","last_unstale":"2024-02-23T00:31:09.184386+0000","last_undegraded":"2024-02-23T00:31:09.184386+0000","last_fullsized":"2024-02-23T00:31:09.184386+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131374+0000","last_change":"2024-02-23T00:30:48.735843+0000","last_active":"2024-02-23T00:31:09.131374+0000","last_peered":"2024-02-23T00:31:09.131374+0000","last_clean":"2024-02-23T00:31:09.131374+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:31:09.131374+0000","last_undegraded":"2024-02-23T00:31:09.131374+0000","last_fullsized":"2024-02-23T00:31:09.131374+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.184408+0000","last_change":"2024-02-23T00:30:45.845873+0000","last_active":"2024-02-23T00:31:09.184408+0000","last_peered":"2024-02-23T00:31:09.184408+0000","last_clean":"2024-02-23T00:31:09.184408+0000","last_became_active":"2024-02-23T00:30:45.845563+0000","last_became_peered":"2024-02-23T00:30:45.845563+0000","last_unstale":"2024-02-23T00:31:09.184408+0000","last_undegraded":"2024-02-23T00:31:09.184408+0000","last_fullsized":"2024-02-23T00:31:09.184408+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.124991+0000","last_change":"2024-02-23T00:30:23.953417+0000","last_active":"2024-02-23T00:31:09.124991+0000","last_peered":"2024-02-23T00:31:09.124991+0000","last_clean":"2024-02-23T00:31:09.124991+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:31:09.124991+0000","last_undegraded":"2024-02-23T00:31:09.124991+0000","last_fullsized":"2024-02-23T00:31:09.124991+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:30:23.953330+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.953330+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":82,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138223+0000","last_change":"2024-02-23T00:30:39.570738+0000","last_active":"2024-02-23T00:31:09.138223+0000","last_peered":"2024-02-23T00:31:09.138223+0000","last_clean":"2024-02-23T00:31:09.138223+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:31:09.138223+0000","last_undegraded":"2024-02-23T00:31:09.138223+0000","last_fullsized":"2024-02-23T00:31:09.138223+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:39.570668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:30:39.570668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":78,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138012+0000","last_change":"2024-02-23T00:30:38.549305+0000","last_active":"2024-02-23T00:31:09.138012+0000","last_peered":"2024-02-23T00:31:09.138012+0000","last_clean":"2024-02-23T00:31:09.138012+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:31:09.138012+0000","last_undegraded":"2024-02-23T00:31:09.138012+0000","last_fullsized":"2024-02-23T00:31:09.138012+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.549228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":70,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.126987+0000","last_change":"2024-02-23T00:29:04.049502+0000","last_active":"2024-02-23T00:31:09.126987+0000","last_peered":"2024-02-23T00:31:09.126987+0000","last_clean":"2024-02-23T00:31:09.126987+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:31:09.126987+0000","last_undegraded":"2024-02-23T00:31:09.126987+0000","last_fullsized":"2024-02-23T00:31:09.126987+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:04.049414+0000","last_clean_scrub_stamp":"2024-02-23T00:29:04.049414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.182958+0000","last_change":"2024-02-23T00:30:48.766372+0000","last_active":"2024-02-23T00:31:09.182958+0000","last_peered":"2024-02-23T00:31:09.182958+0000","last_clean":"2024-02-23T00:31:09.182958+0000","last_became_active":"2024-02-23T00:30:48.762637+0000","last_became_peered":"2024-02-23T00:30:48.762637+0000","last_unstale":"2024-02-23T00:31:09.182958+0000","last_undegraded":"2024-02-23T00:31:09.182958+0000","last_fullsized":"2024-02-23T00:31:09.182958+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174785+0000","last_change":"2024-02-23T00:30:48.725798+0000","last_active":"2024-02-23T00:31:09.174785+0000","last_peered":"2024-02-23T00:31:09.174785+0000","last_clean":"2024-02-23T00:31:09.174785+0000","last_became_active":"2024-02-23T00:30:48.725162+0000","last_became_peered":"2024-02-23T00:30:48.725162+0000","last_unstale":"2024-02-23T00:31:09.174785+0000","last_undegraded":"2024-02-23T00:31:09.174785+0000","last_fullsized":"2024-02-23T00:31:09.174785+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.16","version":"100'7","reported_seq":44,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131144+0000","last_change":"2024-02-23T00:30:48.688397+0000","last_active":"2024-02-23T00:31:09.131144+0000","last_peered":"2024-02-23T00:31:09.131144+0000","last_clean":"2024-02-23T00:31:09.131144+0000","last_became_active":"2024-02-23T00:30:48.688113+0000","last_became_peered":"2024-02-23T00:30:48.688113+0000","last_unstale":"2024-02-23T00:31:09.131144+0000","last_undegraded":"2024-02-23T00:31:09.131144+0000","last_fullsized":"2024-02-23T00:31:09.131144+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165843,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":5,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":112,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120094+0000","last_change":"2024-02-23T00:29:06.822899+0000","last_active":"2024-02-23T00:31:09.120094+0000","last_peered":"2024-02-23T00:31:09.120094+0000","last_clean":"2024-02-23T00:31:09.120094+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:31:09.120094+0000","last_undegraded":"2024-02-23T00:31:09.120094+0000","last_fullsized":"2024-02-23T00:31:09.120094+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:29:06.822818+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.188483+0000","last_change":"2024-02-23T00:28:47.073631+0000","last_active":"2024-02-23T00:31:09.188483+0000","last_peered":"2024-02-23T00:31:09.188483+0000","last_clean":"2024-02-23T00:31:09.188483+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:31:09.188483+0000","last_undegraded":"2024-02-23T00:31:09.188483+0000","last_fullsized":"2024-02-23T00:31:09.188483+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:47.073551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:28:47.073551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.174428+0000","last_change":"2024-02-23T00:28:49.663875+0000","last_active":"2024-02-23T00:31:09.174428+0000","last_peered":"2024-02-23T00:31:09.174428+0000","last_clean":"2024-02-23T00:31:09.174428+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:31:09.174428+0000","last_undegraded":"2024-02-23T00:31:09.174428+0000","last_fullsized":"2024-02-23T00:31:09.174428+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:49.663808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:49.663808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":72,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.125899+0000","last_change":"2024-02-23T00:28:59.685150+0000","last_active":"2024-02-23T00:31:09.125899+0000","last_peered":"2024-02-23T00:31:09.125899+0000","last_clean":"2024-02-23T00:31:09.125899+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:31:09.125899+0000","last_undegraded":"2024-02-23T00:31:09.125899+0000","last_fullsized":"2024-02-23T00:31:09.125899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:59.685045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:28:59.685045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.123585+0000","last_change":"2024-02-23T00:30:48.742680+0000","last_active":"2024-02-23T00:31:09.123585+0000","last_peered":"2024-02-23T00:31:09.123585+0000","last_clean":"2024-02-23T00:31:09.123585+0000","last_became_active":"2024-02-23T00:30:48.741161+0000","last_became_peered":"2024-02-23T00:30:48.741161+0000","last_unstale":"2024-02-23T00:31:09.123585+0000","last_undegraded":"2024-02-23T00:31:09.123585+0000","last_fullsized":"2024-02-23T00:31:09.123585+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.138244+0000","last_change":"2024-02-23T00:30:48.698701+0000","last_active":"2024-02-23T00:31:09.138244+0000","last_peered":"2024-02-23T00:31:09.138244+0000","last_clean":"2024-02-23T00:31:09.138244+0000","last_became_active":"2024-02-23T00:30:48.697963+0000","last_became_peered":"2024-02-23T00:30:48.697963+0000","last_unstale":"2024-02-23T00:31:09.138244+0000","last_undegraded":"2024-02-23T00:31:09.138244+0000","last_fullsized":"2024-02-23T00:31:09.138244+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.132601+0000","last_change":"2024-02-23T00:30:48.726534+0000","last_active":"2024-02-23T00:31:09.132601+0000","last_peered":"2024-02-23T00:31:09.132601+0000","last_clean":"2024-02-23T00:31:09.132601+0000","last_became_active":"2024-02-23T00:30:48.724953+0000","last_became_peered":"2024-02-23T00:30:48.724953+0000","last_unstale":"2024-02-23T00:31:09.132601+0000","last_undegraded":"2024-02-23T00:31:09.132601+0000","last_fullsized":"2024-02-23T00:31:09.132601+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:43.122033+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.122033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"63'39","reported_seq":166,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.163093+0000","last_change":"2024-02-23T00:30:23.839691+0000","last_active":"2024-02-23T00:31:09.163093+0000","last_peered":"2024-02-23T00:31:09.163093+0000","last_clean":"2024-02-23T00:31:09.163093+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:31:09.163093+0000","last_undegraded":"2024-02-23T00:31:09.163093+0000","last_fullsized":"2024-02-23T00:31:09.163093+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'39","last_scrub_stamp":"2024-02-23T00:30:23.839568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.839568+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.228253+0000","last_change":"2024-02-23T00:30:26.074372+0000","last_active":"2024-02-23T00:31:09.228253+0000","last_peered":"2024-02-23T00:31:09.228253+0000","last_clean":"2024-02-23T00:31:09.228253+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:31:09.228253+0000","last_undegraded":"2024-02-23T00:31:09.228253+0000","last_fullsized":"2024-02-23T00:31:09.228253+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:26.074262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:26.074262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.120422+0000","last_change":"2024-02-23T00:28:57.678604+0000","last_active":"2024-02-23T00:31:09.120422+0000","last_peered":"2024-02-23T00:31:09.120422+0000","last_clean":"2024-02-23T00:31:09.120422+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:31:09.120422+0000","last_undegraded":"2024-02-23T00:31:09.120422+0000","last_fullsized":"2024-02-23T00:31:09.120422+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:28:57.678522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:28:57.678522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.169032+0000","last_change":"2024-02-23T00:30:46.848126+0000","last_active":"2024-02-23T00:31:09.169032+0000","last_peered":"2024-02-23T00:31:09.169032+0000","last_clean":"2024-02-23T00:31:09.169032+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:31:09.169032+0000","last_undegraded":"2024-02-23T00:31:09.169032+0000","last_fullsized":"2024-02-23T00:31:09.169032+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.848063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.848063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.131546+0000","last_change":"2024-02-23T00:30:48.696186+0000","last_active":"2024-02-23T00:31:09.131546+0000","last_peered":"2024-02-23T00:31:09.131546+0000","last_clean":"2024-02-23T00:31:09.131546+0000","last_became_active":"2024-02-23T00:30:48.695912+0000","last_became_peered":"2024-02-23T00:30:48.695912+0000","last_unstale":"2024-02-23T00:31:09.131546+0000","last_undegraded":"2024-02-23T00:31:09.131546+0000","last_fullsized":"2024-02-23T00:31:09.131546+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-23T00:31:09.181664+0000","last_change":"2024-02-23T00:30:48.722860+0000","last_active":"2024-02-23T00:31:09.181664+0000","last_peered":"2024-02-23T00:31:09.181664+0000","last_clean":"2024-02-23T00:31:09.181664+0000","last_became_active":"2024-02-23T00:30:48.722569+0000","last_became_peered":"2024-02-23T00:30:48.722569+0000","last_unstale":"2024-02-23T00:31:09.181664+0000","last_undegraded":"2024-02-23T00:31:09.181664+0000","last_fullsized":"2024-02-23T00:31:09.181664+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":72,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":71,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":67,"num_pg":32,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":96,"acting":96,"num_store_stats":8},{"poolid":66,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":96,"acting":96,"num_store_stats":8},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":52,"num_pg":32,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":49,"num_pg":32,"stat_sum":{"num_bytes":651,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":56,"num_pg":4,"stat_sum":{"num_bytes":1536,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":10,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":12,"acting":12,"num_store_stats":6},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":635,"ondisk_log_size":635,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":227,"num_read_kb":222,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":9,"acting":9,"num_store_stats":7},{"poolid":38,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":44,"num_pg":32,"stat_sum":{"num_bytes":45056,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":150998978,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1043,"num_write_kb":152978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":453267456,"data_stored":452996934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1054,"ondisk_log_size":1054,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332146,"num_pgs":199,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355912,"kb_used_data":50624,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93389240,"statfs":{"total":95995035648,"available":95630581760,"internally_reserved":0,"allocated":51838976,"data_stored":51109947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.315,"5min":0.311,"15min":0.311},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.40200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":5.7009999999999996}]},{"osd":1,"last update":"Fri Feb 23 00:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.36899999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.71799999999999997,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.313,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.911,"5min":1.911,"15min":1.911},"last":0.41899999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88900000000000001,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.52500000000000002,"15min":0.52400000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.39900000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.33100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.79800000000000004,"5min":1.121,"15min":1.121},"last":5.6749999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.38700000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.69999999999999996,"5min":1.208,"15min":1.208},"last":5.6520000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.434,"15min":0.41699999999999998},"min":{"1min":0.32500000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67900000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.35199999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.311}]},{"osd":5,"last update":"Fri Feb 23 00:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.44700000000000001,"15min":0.433},"min":{"1min":0.33900000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.46100000000000002,"15min":0.437},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.36699999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:30:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.49299999999999999,"15min":0.47699999999999998},"min":{"1min":0.33900000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.47999999999999998,"15min":0.46000000000000002},"min":{"1min":0.38900000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.38}]}]},{"osd":6,"up_from":44,"seq":188978561079,"num_pgs":172,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":384560,"kb_used_data":75176,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93360592,"statfs":{"total":95995035648,"available":95601246208,"internally_reserved":0,"allocated":76980224,"data_stored":76261176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.27400000000000002,"5min":0.22,"15min":0.22},"max":{"1min":3.3140000000000001,"5min":3.3140000000000001,"15min":3.3140000000000001},"last":94.906000000000006},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.57799999999999996,"15min":0.54000000000000004},"min":{"1min":0.35999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":8.0269999999999992,"5min":8.0269999999999992,"15min":8.0269999999999992},"last":3.6190000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":0.68100000000000005,"15min":0.56299999999999994},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":17.849,"5min":17.849,"15min":17.849},"last":9.8960000000000008},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.60399999999999998,"15min":0.53300000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.574,"5min":11.574,"15min":11.574},"last":74.513999999999996}]},{"osd":2,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":0.77600000000000002,"15min":0.58999999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":25.055,"5min":25.055,"15min":25.055},"last":25.550999999999998},{"interface":"front","average":{"1min":1.911,"5min":0.79800000000000004,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":26.622,"5min":26.622,"15min":26.622},"last":3.5169999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":0.72899999999999998,"15min":0.56899999999999995},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.307},"max":{"1min":17.908999999999999,"5min":17.908999999999999,"15min":17.908999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":2.9449999999999998,"5min":1.014,"15min":0.68700000000000006},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":30.834,"5min":30.834,"15min":30.834},"last":35.203000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.80300000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":3.7839999999999998},{"interface":"front","average":{"1min":1.415,"5min":0.63500000000000001,"15min":0.48099999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":17.718,"5min":17.718,"15min":17.718},"last":3.1259999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.53300000000000003,"15min":0.46300000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":8.0990000000000002,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":4.4210000000000003},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.54300000000000004,"15min":0.46000000000000002},"min":{"1min":0.32100000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.1370000000000005,"5min":8.1370000000000005,"15min":8.1370000000000005},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:30:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48899999999999999,"15min":0.47499999999999998},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":17.696000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.497,"15min":0.48099999999999998},"min":{"1min":0.314,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.70499999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":15.231999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":197,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360012,"kb_used_data":50628,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93385140,"statfs":{"total":95995035648,"available":95626383360,"internally_reserved":0,"allocated":51843072,"data_stored":51112664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.677,"5min":2.677,"15min":2.677},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.41899999999999998,"15min":0.36699999999999999},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.20799999999999999},"max":{"1min":4.7309999999999999,"5min":4.7309999999999999,"15min":4.7309999999999999},"last":0.372}]},{"osd":2,"last update":"Fri Feb 23 00:30:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.36099999999999999,"15min":0.36799999999999999},"min":{"1min":0.22900000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.495,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.378,"15min":0.372},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55800000000000005,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.87}]},{"osd":3,"last update":"Fri Feb 23 00:30:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38600000000000001,"15min":0.41599999999999998},"min":{"1min":0.26000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54400000000000004,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.61899999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:31:01 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57099999999999995,"15min":0.56100000000000005},"min":{"1min":0.44700000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55600000000000005,"15min":0.55100000000000005},"min":{"1min":0.40200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78900000000000003,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:30:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.443,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.75600000000000001,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.42199999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:30:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.56999999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82999999999999996,"5min":1.228,"15min":1.228},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.503,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.97299999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.49399999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.997},{"interface":"front","average":{"1min":0.624,"5min":0.60699999999999998,"15min":0.60999999999999999},"min":{"1min":0.436,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.57099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":163,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355880,"kb_used_data":50592,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93389272,"statfs":{"total":95995035648,"available":95630614528,"internally_reserved":0,"allocated":51806208,"data_stored":51097182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.35899999999999999,"15min":0.33300000000000002},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":4.0410000000000004},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.371,"15min":0.33800000000000002},"min":{"1min":0.23400000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.64900000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.378,"15min":0.38800000000000001},"min":{"1min":0.28399999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.50800000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.36299999999999999,"15min":0.35999999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.48099999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":4.0750000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:30:41 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.41299999999999998,"15min":0.441},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":4.1760000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42199999999999999,"15min":0.442},"min":{"1min":0.28599999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":8.4199999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:31:08 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.622,"15min":0.57999999999999996},"min":{"1min":0.39000000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":8.3599999999999994,"5min":8.3599999999999994,"15min":8.3599999999999994},"last":8.3599999999999994},{"interface":"front","average":{"1min":1.232,"5min":0.68400000000000005,"15min":0.60399999999999998},"min":{"1min":0.45000000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":10.641,"5min":10.641,"15min":10.641},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56200000000000006,"15min":0.55500000000000005},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.39900000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.62}]},{"osd":6,"last update":"Fri Feb 23 00:30:57 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":0.82599999999999996,"15min":0.62},"min":{"1min":0.36599999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":33.082999999999998,"5min":33.082999999999998,"15min":33.082999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.63100000000000001,"15min":0.58699999999999997},"min":{"1min":0.45400000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":7.1600000000000001,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":8.3930000000000007}]},{"osd":7,"last update":"Fri Feb 23 00:30:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56499999999999995,"15min":0.56100000000000005},"min":{"1min":0.432,"5min":0.373,"15min":0.373},"max":{"1min":0.75600000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":8.3219999999999992},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57099999999999995,"15min":0.56299999999999994},"min":{"1min":0.39900000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.79000000000000004,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.58999999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":191,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405060,"kb_used_data":99772,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93340092,"statfs":{"total":95995035648,"available":95580254208,"internally_reserved":0,"allocated":102166528,"data_stored":101431053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34000000000000002,"15min":0.33000000000000002},"min":{"1min":0.22,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52200000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":2.8900000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.34999999999999998,"15min":0.33200000000000002},"min":{"1min":0.22700000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.50700000000000001,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":2.8039999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:30:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36299999999999999,"15min":0.34300000000000003},"min":{"1min":0.26700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":2.5350000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.35199999999999998,"15min":0.33900000000000002},"min":{"1min":0.23799999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.499,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":1.8680000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:30:49 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.47399999999999998,"15min":0.436},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":7.8319999999999999,"5min":7.8319999999999999,"15min":7.8319999999999999},"last":7.5270000000000001},{"interface":"front","average":{"1min":1.0569999999999999,"5min":0.54400000000000004,"15min":0.47599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":7.6669999999999998,"5min":7.6669999999999998,"15min":7.6669999999999998},"last":2.3599999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.60199999999999998,"15min":0.56200000000000006},"min":{"1min":0.40000000000000002,"5min":0.308,"15min":0.308},"max":{"1min":3.4359999999999999,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.60699999999999998,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.319,"5min":3.319,"15min":3.319},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.38300000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.88400000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":1.9410000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.59399999999999997,"15min":0.56599999999999995},"min":{"1min":0.375,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3940000000000001,"5min":3.3940000000000001,"15min":3.3940000000000001},"last":0.59599999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:30:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.59999999999999998,"15min":0.56599999999999995},"min":{"1min":0.40699999999999997,"5min":0.317,"15min":0.317},"max":{"1min":3.3399999999999999,"5min":3.3399999999999999,"15min":3.3399999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57299999999999995,"15min":0.58799999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81000000000000005,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":1.9039999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:30:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.39600000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80600000000000005,"5min":1.117,"15min":1.117},"last":1.974},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.93799999999999994,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.62}]}]},{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":175,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306784,"kb_used_data":1496,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93438368,"statfs":{"total":95995035648,"available":95680888832,"internally_reserved":0,"allocated":1531904,"data_stored":763606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.35499999999999998,"15min":0.36699999999999999},"min":{"1min":0.19500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.498,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.36199999999999999,"15min":0.39000000000000001},"min":{"1min":0.216,"5min":0.191,"15min":0.191},"max":{"1min":0.52400000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.439}]},{"osd":1,"last update":"Fri Feb 23 00:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.41799999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40600000000000003,"15min":0.41199999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:30:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.40500000000000003,"15min":0.43099999999999999},"min":{"1min":0.29099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.504,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39600000000000002,"15min":0.42099999999999999},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.60599999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.34599999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:31:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.33500000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.61599999999999999,"15min":0.59799999999999998},"min":{"1min":0.32900000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":8.0359999999999996,"5min":8.0359999999999996,"15min":8.0359999999999996},"last":0.40100000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79100000000000004,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71499999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.72099999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:30:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.64400000000000002,"15min":0.59999999999999998},"min":{"1min":0.36299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.093,"5min":8.093,"15min":8.093},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.7170000000000001,"5min":1.0229999999999999,"15min":0.747},"min":{"1min":0.39800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":45.232999999999997,"5min":45.232999999999997,"15min":45.232999999999997},"last":0.42299999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.42399999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69399999999999995,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.76900000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.73399999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986241,"num_pgs":202,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335444,"kb_used_data":26060,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93409708,"statfs":{"total":95995035648,"available":95651540992,"internally_reserved":0,"allocated":26685440,"data_stored":25953557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:59 2024","interfaces":[{"interface":"back","average":{"1min":2.02,"5min":0.80200000000000005,"15min":0.61299999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":17.231999999999999,"5min":17.231999999999999,"15min":17.231999999999999},"last":0.20000000000000001},{"interface":"front","average":{"1min":3.548,"5min":1.1279999999999999,"15min":0.72899999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":36.171999999999997,"5min":36.171999999999997,"15min":36.171999999999997},"last":4}]},{"osd":1,"last update":"Fri Feb 23 00:30:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8029999999999999,"5min":1.1459999999999999,"15min":0.73499999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":41.137999999999998,"5min":41.137999999999998,"15min":41.137999999999998},"last":11.477},{"interface":"front","average":{"1min":2.835,"5min":0.97099999999999997,"15min":0.67100000000000004},"min":{"1min":0.34300000000000003,"5min":0.309,"15min":0.309},"max":{"1min":26.210000000000001,"5min":26.210000000000001,"15min":26.210000000000001},"last":2.1760000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:30:59 2024","interfaces":[{"interface":"back","average":{"1min":5.0389999999999997,"5min":1.4139999999999999,"15min":0.83099999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":67.253,"5min":67.253,"15min":67.253},"last":0.42599999999999999},{"interface":"front","average":{"1min":2.093,"5min":0.84499999999999997,"15min":0.63},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":25.864000000000001,"5min":25.864000000000001,"15min":25.864000000000001},"last":11.731999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:30:59 2024","interfaces":[{"interface":"back","average":{"1min":3.5350000000000001,"5min":1.129,"15min":0.72699999999999998},"min":{"1min":0.35999999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":27.550000000000001,"5min":27.550000000000001,"15min":27.550000000000001},"last":7.0819999999999999},{"interface":"front","average":{"1min":2.7360000000000002,"5min":0.94499999999999995,"15min":0.67500000000000004},"min":{"1min":0.35299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":30.613,"5min":30.613,"15min":30.613},"last":21.053000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:30:28 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.40999999999999998,"15min":0.377},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.221},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.45300000000000001,"15min":0.41299999999999998},"min":{"1min":0.30599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.29899999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.53100000000000003,"15min":0.47499999999999998},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.254},{"interface":"front","average":{"1min":2.6309999999999998,"5min":0.89000000000000001,"15min":0.58199999999999996},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":45.814999999999998,"5min":45.814999999999998,"15min":45.814999999999998},"last":0.23100000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:30:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.48899999999999999,"15min":0.47099999999999997},"min":{"1min":0.36799999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.308},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.53400000000000003,"15min":0.51400000000000001},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.28299999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757308,"num_pgs":195,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405084,"kb_used_data":99796,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93340068,"statfs":{"total":95995035648,"available":95580229632,"internally_reserved":0,"allocated":102191104,"data_stored":101455686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.49299999999999999,"15min":0.503},"min":{"1min":0.313,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65600000000000003,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":1.117},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.52200000000000002,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.653,"5min":3.653,"15min":3.653},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48199999999999998,"15min":0.47799999999999998},"min":{"1min":0.33400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":3.96},{"interface":"front","average":{"1min":0.501,"5min":0.495,"15min":0.495},"min":{"1min":0.36399999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51400000000000001,"15min":0.505},"min":{"1min":0.36099999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.63700000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.56000000000000005}]},{"osd":3,"last update":"Fri Feb 23 00:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.53200000000000003,"15min":0.498},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.762,"5min":3.762,"15min":3.762},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.54900000000000004,"15min":0.53200000000000003},"min":{"1min":0.38200000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"last":40.865000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:30:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40200000000000002,"15min":0.376},"min":{"1min":0.314,"5min":0.219,"15min":0.219},"max":{"1min":0.60999999999999999,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.41799999999999998,"15min":0.38},"min":{"1min":0.29999999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60799999999999998,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.48199999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:30:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.55500000000000005,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.1440000000000001,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":5.8879999999999999,"5min":1.5489999999999999,"15min":0.80300000000000005},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":103.61499999999999,"5min":103.61499999999999,"15min":103.61499999999999},"last":0.433}]},{"osd":7,"last update":"Fri Feb 23 00:30:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.496,"15min":0.48799999999999999},"min":{"1min":0.34200000000000003,"5min":0.31,"15min":0.31},"max":{"1min":0.68600000000000005,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.51800000000000002,"15min":0.50900000000000001},"min":{"1min":0.34200000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71899999999999997,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50364416,"data_stored":50334500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50360320,"data_stored":50332774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100696064,"data_stored":100663354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25210880,"data_stored":25166974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":100700160,"data_stored":100666161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":75542528,"data_stored":75498622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":50372608,"data_stored":50334515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":44,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":44,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":44,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":44,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":44,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":44,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":44,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":44,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":56,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":66,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":67,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:31:16.835 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-23T00:31:16.835 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set iscsi pg_num 13 2024-02-23T00:31:16.912 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:16.986 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:17.225 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:17.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:17.334 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:17.414 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:17.526 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:17.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:17.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:17.727 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:17.812 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:17.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:17.924 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:17.952 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:17.971 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:17.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:17.976 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.034+0000 7fab9691b700 1 -- 172.21.15.87:0/3869215982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab900692d0 msgr2=0x7fab900696b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.034+0000 7fab9691b700 1 --2- 172.21.15.87:0/3869215982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab900692d0 0x7fab900696b0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 rx=0x7fab80004870 tx=0x7fab80009cb0).stop 2024-02-23T00:31:18.075 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.077+0000 7fab9691b700 1 -- 172.21.15.87:0/3869215982 shutdown_connections 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.077+0000 7fab9691b700 1 --2- 172.21.15.87:0/3869215982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab901044a0 0x7fab90104920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.077+0000 7fab9691b700 1 --2- 172.21.15.87:0/3869215982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab900692d0 0x7fab900696b0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.077+0000 7fab9691b700 1 --2- 172.21.15.87:0/3869215982 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab90069ba0 0x7fab90103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.077+0000 7fab9691b700 1 -- 172.21.15.87:0/3869215982 >> 172.21.15.87:0/3869215982 conn(0x7fab90063db0 msgr2=0x7fab90067560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.077+0000 7fab9691b700 1 -- 172.21.15.87:0/3869215982 shutdown_connections 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.077+0000 7fab9691b700 1 -- 172.21.15.87:0/3869215982 wait complete. 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 librados: starting msgr at 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 librados: starting objecter 2024-02-23T00:31:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 Processor -- start 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 -- start start 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 librados: setting wanted keys 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 librados: calling monclient init 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab900692d0 0x7fab9010c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab90069ba0 0x7fab9010c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab901044a0 0x7fab9011a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9011ac90 con 0x7fab900692d0 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9011ae00 con 0x7fab90069ba0 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.078+0000 7fab9691b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9011afc0 con 0x7fab901044a0 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.080+0000 7fab94eb8700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab901044a0 0x7fab9011a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.080+0000 7fab94eb8700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab901044a0 0x7fab9011a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:43392/0 (socket says 172.21.15.87:43392) 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.080+0000 7fab94eb8700 1 -- 172.21.15.87:0/1762469599 learned_addr learned my addr 172.21.15.87:0/1762469599 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:18.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.080+0000 7fab8ffff700 1 --2- 172.21.15.87:0/1762469599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab900692d0 0x7fab9010c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:15.531935+0000 mgr.y (mgr.24437) 174 : cluster [DBG] pgmap v169: 424 pgs: 64 creating+peering, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.970522+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.970634+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi087-82011-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.970728+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:15.970780+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.980075+0000 mon.b (mon.1) 117 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi087-82107-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.980990+0000 mon.b (mon.1) 118 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.985823+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.87:0/1156606555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:15.999895+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi087-82107-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:18.094 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:16.000445+0000 mon.a (mon.0) 1061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:16.002668+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:16.141101+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4"}]: dispatch 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:16.525462+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:17.327810+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:17.328010+0000 mon.a (mon.0) 1065 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:17.328256+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4"}]': finished 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:15.531935+0000 mgr.y (mgr.24437) 174 : cluster [DBG] pgmap v169: 424 pgs: 64 creating+peering, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.970522+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.970634+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi087-82011-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.970728+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:18.095 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:15.970780+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.980075+0000 mon.b (mon.1) 117 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi087-82107-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.980990+0000 mon.b (mon.1) 118 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.985823+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.87:0/1156606555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:15.999895+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi087-82107-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:16.000445+0000 mon.a (mon.0) 1061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:16.002668+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:16.141101+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4"}]: dispatch 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:16.525462+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:17.327810+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:17.328010+0000 mon.a (mon.0) 1065 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.096 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:17.328256+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4"}]': finished 2024-02-23T00:31:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.086+0000 7fab8f7fe700 1 --2- 172.21.15.87:0/1762469599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab90069ba0 0x7fab9010c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab8f7fe700 1 -- 172.21.15.87:0/1762469599 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab901044a0 msgr2=0x7fab9011a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab8f7fe700 1 --2- 172.21.15.87:0/1762469599 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab901044a0 0x7fab9011a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab8f7fe700 1 -- 172.21.15.87:0/1762469599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab900692d0 msgr2=0x7fab9010c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab8f7fe700 1 --2- 172.21.15.87:0/1762469599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab900692d0 0x7fab9010c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab8f7fe700 1 -- 172.21.15.87:0/1762469599 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab80005040 con 0x7fab90069ba0 2024-02-23T00:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab8f7fe700 1 --2- 172.21.15.87:0/1762469599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab90069ba0 0x7fab9010c580 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7fab8400b4e0 tx=0x7fab84009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab8d7fa700 10 client.?.objecter ms_handle_connect 0x7fab90069ba0 2024-02-23T00:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab8d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab8d7fa700 1 -- 172.21.15.87:0/1762469599 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84002320 con 0x7fab90069ba0 2024-02-23T00:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9011b2a0 con 0x7fab90069ba0 2024-02-23T00:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab9691b700 10 client.35755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9011b7e0 con 0x7fab90069ba0 2024-02-23T00:31:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.092+0000 7fab9691b700 1 librados: init done 2024-02-23T00:31:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.094+0000 7fab8d7fa700 1 -- 172.21.15.87:0/1762469599 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab84002950 con 0x7fab90069ba0 2024-02-23T00:31:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.094+0000 7fab8d7fa700 1 -- 172.21.15.87:0/1762469599 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84012a20 con 0x7fab90069ba0 2024-02-23T00:31:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.094+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab90060280 con 0x7fab90069ba0 2024-02-23T00:31:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.095+0000 7fab8d7fa700 1 -- 172.21.15.87:0/1762469599 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fab8401c440 con 0x7fab90069ba0 2024-02-23T00:31:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.095+0000 7fab8d7fa700 1 --2- 172.21.15.87:0/1762469599 >> v2:172.21.15.87:6800/1231597463 conn(0x7fab78062ce0 0x7fab780651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.095+0000 7fab8d7fa700 1 -- 172.21.15.87:0/1762469599 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(126..126 src has 38..126) v4 ==== 10820+0+0 (secure 0 0 0) 0x7fab8407ec90 con 0x7fab90069ba0 2024-02-23T00:31:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.095+0000 7fab8d7fa700 10 client.35755.objecter ms_dispatch 0x7fab90001550 osd_map(126..126 src has 38..126) v4 2024-02-23T00:31:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.095+0000 7fab8d7fa700 3 client.35755.objecter handle_osd_map got epochs [126,126] > 0 2024-02-23T00:31:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.095+0000 7fab8d7fa700 3 client.35755.objecter handle_osd_map decoding full epoch 126 2024-02-23T00:31:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.096+0000 7fab8d7fa700 20 client.35755.objecter dump_active .. 0 homeless 2024-02-23T00:31:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.096+0000 7fab8ffff700 1 --2- 172.21.15.87:0/1762469599 >> v2:172.21.15.87:6800/1231597463 conn(0x7fab78062ce0 0x7fab780651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.096+0000 7fab8ffff700 1 --2- 172.21.15.87:0/1762469599 >> v2:172.21.15.87:6800/1231597463 conn(0x7fab78062ce0 0x7fab780651a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7fab80009da0 tx=0x7fab80007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:18.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.096+0000 7fab8d7fa700 10 client.35755.objecter ms_handle_connect 0x7fab78062ce0 2024-02-23T00:31:18.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.100+0000 7fab8d7fa700 1 -- 172.21.15.87:0/1762469599 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fab840516f0 con 0x7fab90069ba0 2024-02-23T00:31:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:15.531935+0000 mgr.y (mgr.24437) 174 : cluster [DBG] pgmap v169: 424 pgs: 64 creating+peering, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:31:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.970522+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.970634+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi087-82011-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.970728+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:15.970780+0000 mon.a (mon.0) 1059 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-23T00:31:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.980075+0000 mon.b (mon.1) 117 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi087-82107-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.980990+0000 mon.b (mon.1) 118 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.985823+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.87:0/1156606555' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:15.999895+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi087-82107-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:18.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:16.000445+0000 mon.a (mon.0) 1061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:16.002668+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:18.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:16.141101+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4"}]: dispatch 2024-02-23T00:31:18.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:16.525462+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:18.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:17.327810+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:17.328010+0000 mon.a (mon.0) 1065 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:18.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:17.328256+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4"}]': finished 2024-02-23T00:31:18.416 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:18.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:18.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:18.528 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:18.554 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:18.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:18.555+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fab90116040 con 0x7fab90069ba0 2024-02-23T00:31:18.573 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:18.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:18.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:18.677 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:31:18.703 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:18.722 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:31:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:18.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:31:18.826 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:18.852 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:18.872 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:18.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:18.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:19.062 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:19.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:17.328322+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-23T00:31:19.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:17.337693+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]: dispatch 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:17.351065+0000 mon.b (mon.1) 120 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:17.533722+0000 mgr.y (mgr.24437) 175 : cluster [DBG] pgmap v172: 424 pgs: 64 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:17.542346+0000 mon.c (mon.2) 84 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:17.825316+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:18.220432+0000 mon.a (mon.0) 1070 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.235237+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi087-82107-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]': finished 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.235327+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]': finished 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.235427+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:18.235495+0000 mon.a (mon.0) 1074 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.242563+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.87:0/945355696' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi087-82011-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.250508+0000 mon.b (mon.1) 122 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.264085+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-23T00:31:19.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.265985+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi087-82011-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:19.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.266370+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:19.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.556827+0000 mon.b (mon.1) 123 : audit [INF] from='client.? 172.21.15.87:0/1762469599' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:31:19.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.575931+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:19.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:18.746542+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:31:19.236 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:31:19.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.236+0000 7fab8d7fa700 1 -- 172.21.15.87:0/1762469599 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v128) v1 ==== 91+0+0 (secure 0 0 0) 0x7fab840512e0 con 0x7fab90069ba0 2024-02-23T00:31:19.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.242+0000 7fab9691b700 10 librados: watch_flush enter 2024-02-23T00:31:19.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.242+0000 7fab9691b700 10 librados: watch_flush exit 2024-02-23T00:31:19.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.242+0000 7fab9691b700 20 client.35755.objecter shutdown clearing up homeless session... 2024-02-23T00:31:19.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.242+0000 7fab9691b700 10 client.35755.objecter successfully canceled tick 2024-02-23T00:31:19.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.243+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 >> v2:172.21.15.87:6800/1231597463 conn(0x7fab78062ce0 msgr2=0x7fab780651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:19.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.243+0000 7fab9691b700 1 --2- 172.21.15.87:0/1762469599 >> v2:172.21.15.87:6800/1231597463 conn(0x7fab78062ce0 0x7fab780651a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7fab80009da0 tx=0x7fab80007f90).stop 2024-02-23T00:31:19.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.243+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab90069ba0 msgr2=0x7fab9010c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:19.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.243+0000 7fab9691b700 1 --2- 172.21.15.87:0/1762469599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab90069ba0 0x7fab9010c580 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 rx=0x7fab8400b4e0 tx=0x7fab84009060).stop 2024-02-23T00:31:19.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.269+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 shutdown_connections 2024-02-23T00:31:19.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.269+0000 7fab9691b700 1 --2- 172.21.15.87:0/1762469599 >> v2:172.21.15.87:6800/1231597463 conn(0x7fab78062ce0 0x7fab780651a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:19.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.269+0000 7fab9691b700 1 --2- 172.21.15.87:0/1762469599 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fab901044a0 0x7fab9011a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:19.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.269+0000 7fab9691b700 1 --2- 172.21.15.87:0/1762469599 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fab900692d0 0x7fab9010c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:19.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.269+0000 7fab9691b700 1 --2- 172.21.15.87:0/1762469599 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fab90069ba0 0x7fab9010c580 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:19.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.269+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 >> 172.21.15.87:0/1762469599 conn(0x7fab90063db0 msgr2=0x7fab900fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:19.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.290+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 shutdown_connections 2024-02-23T00:31:19.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.290+0000 7fab9691b700 1 -- 172.21.15.87:0/1762469599 wait complete. 2024-02-23T00:31:19.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:19.300+0000 7fab9691b700 1 librados: shutdown 2024-02-23T00:31:19.317 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:31:19.349 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:17.328322+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-23T00:31:19.349 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:17.337693+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]: dispatch 2024-02-23T00:31:19.349 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:17.351065+0000 mon.b (mon.1) 120 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:19.349 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:17.533722+0000 mgr.y (mgr.24437) 175 : cluster [DBG] pgmap v172: 424 pgs: 64 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:31:19.349 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:17.542346+0000 mon.c (mon.2) 84 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:17.825316+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:18.220432+0000 mon.a (mon.0) 1070 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.235237+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi087-82107-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]': finished 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.235327+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]': finished 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.235427+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:18.235495+0000 mon.a (mon.0) 1074 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.242563+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.87:0/945355696' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi087-82011-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.250508+0000 mon.b (mon.1) 122 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.264085+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.265985+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi087-82011-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.266370+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.556827+0000 mon.b (mon.1) 123 : audit [INF] from='client.? 172.21.15.87:0/1762469599' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:31:19.350 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.575931+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:18.746542+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:17.328322+0000 mon.a (mon.0) 1067 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:17.337693+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]: dispatch 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:17.351065+0000 mon.b (mon.1) 120 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:17.533722+0000 mgr.y (mgr.24437) 175 : cluster [DBG] pgmap v172: 424 pgs: 64 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:17.542346+0000 mon.c (mon.2) 84 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:17.825316+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:18.220432+0000 mon.a (mon.0) 1070 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.235237+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi087-82107-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]': finished 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.235327+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6"}]': finished 2024-02-23T00:31:19.351 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.235427+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:19.352 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:18.235495+0000 mon.a (mon.0) 1074 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-23T00:31:19.352 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.242563+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.87:0/945355696' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi087-82011-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:19.352 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.250508+0000 mon.b (mon.1) 122 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:19.352 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.264085+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-23T00:31:19.352 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.265985+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi087-82011-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:19.352 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.266370+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:19.352 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.556827+0000 mon.b (mon.1) 123 : audit [INF] from='client.? 172.21.15.87:0/1762469599' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:31:19.352 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.575931+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:19.352 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:18.746542+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:31:19.552 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:19.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:19.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:19.656 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:19.718 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:20.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.231081+0000 mon.a (mon.0) 1079 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:20.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:19.235990+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-23T00:31:20.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:19.236079+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi087-82011-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:20.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:19.236180+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-9"}]': finished 2024-02-23T00:31:20.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:19.237136+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:31:20.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.237188+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-23T00:31:20.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:19.247224+0000 mon.b (mon.1) 124 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-9", "mode": "writeback"}]: dispatch 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:19.249189+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-9", "mode": "writeback"}]: dispatch 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:19.249368+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? 172.21.15.87:0/2268997341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.375459+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.393713+0000 osd.0 (osd.0) 82 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:19.596044+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.231081+0000 mon.a (mon.0) 1079 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:19.235990+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:19.236079+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi087-82011-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:19.236180+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-9"}]': finished 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:19.237136+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:31:20.090 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.237188+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-23T00:31:20.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:19.247224+0000 mon.b (mon.1) 124 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-9", "mode": "writeback"}]: dispatch 2024-02-23T00:31:20.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:19.249189+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-9", "mode": "writeback"}]: dispatch 2024-02-23T00:31:20.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:19.249368+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? 172.21.15.87:0/2268997341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:20.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.375459+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:31:20.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.393713+0000 osd.0 (osd.0) 82 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:31:20.091 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:19.596044+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:20.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.231081+0000 mon.a (mon.0) 1079 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:20.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:19.235990+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? 172.21.15.87:0/2103098061' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi087-82147-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-23T00:31:20.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:19.236079+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi087-82011-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:20.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:19.236180+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-9"}]': finished 2024-02-23T00:31:20.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:19.237136+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:31:20.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.237188+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-23T00:31:20.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:19.247224+0000 mon.b (mon.1) 124 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-9", "mode": "writeback"}]: dispatch 2024-02-23T00:31:20.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:19.249189+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-9", "mode": "writeback"}]: dispatch 2024-02-23T00:31:20.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:19.249368+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? 172.21.15.87:0/2268997341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:20.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.375459+0000 osd.0 (osd.0) 81 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:31:20.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.393713+0000 osd.0 (osd.0) 82 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:31:20.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:19.596044+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:20.312 INFO:tasks.workunit.client.0.smithi087.stdout:watch_notify_pp: flushed 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3018 ms) 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: trying... 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402367200 notify_id 498216206343 notifier_gid 15147 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: timed out 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushing 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushed 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3021 ms) 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: List watches 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify2 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402367200 notify_id 506806140936 notifier_gid 15147 2024-02-23T00:31:20.313 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify2 done 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: watch_check 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: unwatch2 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushing 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: done 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3165 ms) 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: List watches 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify2 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: handle_notify cookie 93958402367200 notify_id 523986010121 notifier_gid 15147 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: notify2 done 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: watch_check 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: unwatch2 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: flushing 2024-02-23T00:31:20.314 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: done 2024-02-23T00:31:20.315 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3058 ms) 2024-02-23T00:31:20.315 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16641 ms total) 2024-02-23T00:31:20.315 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: 2024-02-23T00:31:20.315 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-02-23T00:31:20.315 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (38261 ms total) 2024-02-23T00:31:20.315 INFO:tasks.workunit.client.0.smithi087.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-02-23T00:31:20.781 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:20.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:20.904 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:20.940 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:20.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:31:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:31:21.309 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:21.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:21.412 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:21.481 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:21.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:21.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.540557+0000 mgr.y (mgr.24437) 176 : cluster [DBG] pgmap v175: 492 pgs: 136 unknown, 356 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:21.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.757319+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:31:21.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.760197+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:31:21.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.967937+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:31:21.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.971415+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:31:21.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.979955+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.c scrub starts 2024-02-23T00:31:21.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:19.981981+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.c scrub ok 2024-02-23T00:31:21.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:20.233382+0000 mon.a (mon.0) 1087 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:21.619 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:20.261077+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-9", "mode": "writeback"}]': finished 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:20.261159+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? 172.21.15.87:0/2268997341' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:20.261292+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:20.332760+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:20.333544+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:20.613932+0000 mon.c (mon.2) 87 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:20.662090+0000 mgr.y (mgr.24437) 177 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:20.709025+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.338403+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.540557+0000 mgr.y (mgr.24437) 176 : cluster [DBG] pgmap v175: 492 pgs: 136 unknown, 356 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.757319+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.760197+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:31:21.620 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.967937+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.971415+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.979955+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.c scrub starts 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:19.981981+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.c scrub ok 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:20.233382+0000 mon.a (mon.0) 1087 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:20.261077+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-9", "mode": "writeback"}]': finished 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:20.261159+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? 172.21.15.87:0/2268997341' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:20.261292+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:20.332760+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:20.333544+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:20.613932+0000 mon.c (mon.2) 87 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:20.662090+0000 mgr.y (mgr.24437) 177 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:20.709025+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:21.621 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.338403+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:21.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.540557+0000 mgr.y (mgr.24437) 176 : cluster [DBG] pgmap v175: 492 pgs: 136 unknown, 356 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.757319+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:31:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.760197+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:31:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.967937+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:31:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.971415+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:31:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.979955+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.c scrub starts 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:19.981981+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.c scrub ok 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:20.233382+0000 mon.a (mon.0) 1087 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:20.261077+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-9", "mode": "writeback"}]': finished 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:20.261159+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? 172.21.15.87:0/2268997341' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi087-82019-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:20.261292+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:20.332760+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:20.333544+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:20.613932+0000 mon.c (mon.2) 87 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:20.662090+0000 mgr.y (mgr.24437) 177 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:20.709025+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.338403+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:21.804 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:22.523 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:20.630213+0000 osd.2 (osd.2) 65 : cluster [DBG] 2.c scrub starts 2024-02-23T00:31:22.523 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:20.632726+0000 osd.2 (osd.2) 66 : cluster [DBG] 2.c scrub ok 2024-02-23T00:31:22.523 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.082065+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:31:22.523 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.086602+0000 osd.3 (osd.3) 90 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:31:22.524 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.363014+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.d scrub starts 2024-02-23T00:31:22.524 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.366530+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.d scrub ok 2024-02-23T00:31:22.524 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.395631+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:31:22.524 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.395694+0000 mon.a (mon.0) 1093 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-23T00:31:22.524 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.405576+0000 mon.b (mon.1) 126 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:22.524 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:22] "GET /metrics HTTP/1.1" 200 211966 "" "Prometheus/2.33.4" 2024-02-23T00:31:22.552 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:22.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:22.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.601+0000 7fa8a7004700 1 -- 172.21.15.87:0/3536607218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8a00fd4a0 msgr2=0x7fa8a0101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:22.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.601+0000 7fa8a7004700 1 --2- 172.21.15.87:0/3536607218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8a00fd4a0 0x7fa8a0101b40 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fa894004870 tx=0x7fa894009cb0).stop 2024-02-23T00:31:22.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.601+0000 7fa8a7004700 1 -- 172.21.15.87:0/3536607218 shutdown_connections 2024-02-23T00:31:22.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.601+0000 7fa8a7004700 1 --2- 172.21.15.87:0/3536607218 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa8a01020c0 0x7fa8a0104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:22.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.601+0000 7fa8a7004700 1 --2- 172.21.15.87:0/3536607218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8a00fcb40 0x7fa8a00fcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:22.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.601+0000 7fa8a7004700 1 --2- 172.21.15.87:0/3536607218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8a00fd4a0 0x7fa8a0101b40 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:22.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.601+0000 7fa8a7004700 1 -- 172.21.15.87:0/3536607218 >> 172.21.15.87:0/3536607218 conn(0x7fa8a006c370 msgr2=0x7fa8a006c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.602+0000 7fa8a7004700 1 -- 172.21.15.87:0/3536607218 shutdown_connections 2024-02-23T00:31:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.602+0000 7fa8a7004700 1 -- 172.21.15.87:0/3536607218 wait complete. 2024-02-23T00:31:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.602+0000 7fa8a7004700 1 librados: starting msgr at 2024-02-23T00:31:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.602+0000 7fa8a7004700 1 librados: starting objecter 2024-02-23T00:31:22.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 Processor -- start 2024-02-23T00:31:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 -- start start 2024-02-23T00:31:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 librados: setting wanted keys 2024-02-23T00:31:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 librados: calling monclient init 2024-02-23T00:31:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8a00fcb40 0x7fa8a0063340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8a00fd4a0 0x7fa8a0063830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa8a01020c0 0x7fa8a0109610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a0063fe0 con 0x7fa8a00fd4a0 2024-02-23T00:31:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a0109b00 con 0x7fa8a00fcb40 2024-02-23T00:31:22.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.603+0000 7fa8a7004700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a0109cc0 con 0x7fa8a01020c0 2024-02-23T00:31:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.605+0000 7fa8a4da0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8a00fcb40 0x7fa8a0063340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:22.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa8a4da0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8a00fcb40 0x7fa8a0063340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.87:52708/0 (socket says 172.21.15.87:52708) 2024-02-23T00:31:22.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa8a4da0700 1 -- 172.21.15.87:0/884223873 learned_addr learned my addr 172.21.15.87:0/884223873 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa8a55a1700 1 --2- 172.21.15.87:0/884223873 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa8a01020c0 0x7fa8a0109610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa8a4da0700 1 -- 172.21.15.87:0/884223873 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa8a01020c0 msgr2=0x7fa8a0109610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa8a4da0700 1 --2- 172.21.15.87:0/884223873 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa8a01020c0 0x7fa8a0109610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa8a4da0700 1 -- 172.21.15.87:0/884223873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8a00fd4a0 msgr2=0x7fa8a0063830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa8a4da0700 1 --2- 172.21.15.87:0/884223873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8a00fd4a0 0x7fa8a0063830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa8a4da0700 1 -- 172.21.15.87:0/884223873 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa894005040 con 0x7fa8a00fcb40 2024-02-23T00:31:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa8a4da0700 1 --2- 172.21.15.87:0/884223873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8a00fcb40 0x7fa8a0063340 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 rx=0x7fa88c0041d0 tx=0x7fa88c009f60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa89dffb700 10 client.?.objecter ms_handle_connect 0x7fa8a00fcb40 2024-02-23T00:31:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.606+0000 7fa89dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.607+0000 7fa89dffb700 1 -- 172.21.15.87:0/884223873 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c007b40 con 0x7fa8a00fcb40 2024-02-23T00:31:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.607+0000 7fa89dffb700 1 -- 172.21.15.87:0/884223873 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa88c00f040 con 0x7fa8a00fcb40 2024-02-23T00:31:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.607+0000 7fa89dffb700 1 -- 172.21.15.87:0/884223873 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00d8c0 con 0x7fa8a00fcb40 2024-02-23T00:31:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.607+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a0109fa0 con 0x7fa8a00fcb40 2024-02-23T00:31:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.607+0000 7fa8a7004700 10 client.38785.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:22.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.607+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a010a440 con 0x7fa8a00fcb40 2024-02-23T00:31:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.607+0000 7fa8a7004700 1 librados: init done 2024-02-23T00:31:22.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.608+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a005f9d0 con 0x7fa8a00fcb40 2024-02-23T00:31:22.663 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:22.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.629+0000 7fa89dffb700 1 -- 172.21.15.87:0/884223873 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fa88c009320 con 0x7fa8a00fcb40 2024-02-23T00:31:22.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.630+0000 7fa89dffb700 1 --2- 172.21.15.87:0/884223873 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa890062c90 0x7fa890065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:22.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.630+0000 7fa89dffb700 1 -- 172.21.15.87:0/884223873 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(131..131 src has 38..131) v4 ==== 11207+0+0 (secure 0 0 0) 0x7fa88c07e120 con 0x7fa8a00fcb40 2024-02-23T00:31:22.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.630+0000 7fa89dffb700 10 client.38785.objecter ms_dispatch 0x7fa8a0001550 osd_map(131..131 src has 38..131) v4 2024-02-23T00:31:22.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.630+0000 7fa89dffb700 3 client.38785.objecter handle_osd_map got epochs [131,131] > 0 2024-02-23T00:31:22.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.630+0000 7fa89dffb700 3 client.38785.objecter handle_osd_map decoding full epoch 131 2024-02-23T00:31:22.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.630+0000 7fa89dffb700 20 client.38785.objecter dump_active .. 0 homeless 2024-02-23T00:31:22.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.640+0000 7fa89dffb700 1 -- 172.21.15.87:0/884223873 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa88c0509a0 con 0x7fa8a00fcb40 2024-02-23T00:31:22.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.700+0000 7fa89ffff700 1 --2- 172.21.15.87:0/884223873 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa890062c90 0x7fa890065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:22.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.715+0000 7fa89ffff700 1 --2- 172.21.15.87:0/884223873 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa890062c90 0x7fa890065150 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7fa894009da0 tx=0x7fa894009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:22.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:22.715+0000 7fa89dffb700 10 client.38785.objecter ms_handle_connect 0x7fa890062c90 2024-02-23T00:31:22.844 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.405826+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.844 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.406283+0000 mon.b (mon.1) 128 : audit [INF] from='client.? 172.21.15.87:0/685070663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi087-82011-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.844 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.422086+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:22.844 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.422304+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.844 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.422528+0000 mon.a (mon.0) 1096 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi087-82011-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.844 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.550054+0000 mgr.y (mgr.24437) 178 : cluster [DBG] pgmap v178: 428 pgs: 64 unknown, 364 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.641023+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.821364+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.828709+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.954944+0000 mon.a (mon.0) 1097 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.970607+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9"}]': finished 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.970661+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.970705+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi087-82011-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.970807+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:21.985480+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.87:0/338034338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi087-82019-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.493328+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi087-82019-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.493808+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:31:22.845 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:20.630213+0000 osd.2 (osd.2) 65 : cluster [DBG] 2.c scrub starts 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:20.632726+0000 osd.2 (osd.2) 66 : cluster [DBG] 2.c scrub ok 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.082065+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.086602+0000 osd.3 (osd.3) 90 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.363014+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.d scrub starts 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.366530+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.d scrub ok 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.395631+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.395694+0000 mon.a (mon.0) 1093 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.405576+0000 mon.b (mon.1) 126 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.405826+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.406283+0000 mon.b (mon.1) 128 : audit [INF] from='client.? 172.21.15.87:0/685070663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi087-82011-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.422086+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.422304+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.846 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.422528+0000 mon.a (mon.0) 1096 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi087-82011-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.550054+0000 mgr.y (mgr.24437) 178 : cluster [DBG] pgmap v178: 428 pgs: 64 unknown, 364 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.641023+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.821364+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.828709+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.954944+0000 mon.a (mon.0) 1097 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.970607+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9"}]': finished 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.970661+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.970705+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi087-82011-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.970807+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:21.985480+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.87:0/338034338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi087-82019-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.493328+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi087-82019-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.847 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.493808+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:20.630213+0000 osd.2 (osd.2) 65 : cluster [DBG] 2.c scrub starts 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:20.632726+0000 osd.2 (osd.2) 66 : cluster [DBG] 2.c scrub ok 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.082065+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.086602+0000 osd.3 (osd.3) 90 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.363014+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.d scrub starts 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.366530+0000 osd.5 (osd.5) 70 : cluster [DBG] 6.d scrub ok 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.395631+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.395694+0000 mon.a (mon.0) 1093 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.405576+0000 mon.b (mon.1) 126 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.405826+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.406283+0000 mon.b (mon.1) 128 : audit [INF] from='client.? 172.21.15.87:0/685070663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi087-82011-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.422086+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9"}]: dispatch 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.422304+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.422528+0000 mon.a (mon.0) 1096 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi087-82011-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.550054+0000 mgr.y (mgr.24437) 178 : cluster [DBG] pgmap v178: 428 pgs: 64 unknown, 364 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.641023+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.821364+0000 osd.7 (osd.7) 77 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.828709+0000 osd.7 (osd.7) 78 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.954944+0000 mon.a (mon.0) 1097 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.970607+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-9"}]': finished 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.970661+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82099-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.970705+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi087-82011-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.970807+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-23T00:31:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:21.985480+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.87:0/338034338' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi087-82019-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.493328+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi087-82019-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:22.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.493808+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:31:22.980 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:23.024 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:23.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:23.132 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:23.159 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:23.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:23.165+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"} v 0) v1 -- 0x7fa8a005ee80 con 0x7fa8a00fcb40 2024-02-23T00:31:23.179 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:23.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:23.283 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:23.376 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:23.414 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:23.418 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:23.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:23.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:23.523 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:23.531 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:31:23.531 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:31:23.550 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:23.570 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:23.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:23.673 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:31:23.700 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:23.720 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:31:23.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:23.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:31:23.823 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:31:23.848 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:31:23.850 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:23.868 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:31:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:23.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:31:23.973 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:23.999 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:24.035 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:24.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:24.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.039+0000 7fa89dffb700 1 -- 172.21.15.87:0/884223873 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]=0 set pool 3 pg_num to 13 v133) v1 ==== 130+0+0 (secure 0 0 0) 0x7fa88c050e60 con 0x7fa8a00fcb40 2024-02-23T00:31:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.042+0000 7fa8a7004700 10 librados: watch_flush enter 2024-02-23T00:31:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.042+0000 7fa8a7004700 10 librados: watch_flush exit 2024-02-23T00:31:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.042+0000 7fa8a7004700 20 client.38785.objecter shutdown clearing up homeless session... 2024-02-23T00:31:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.042+0000 7fa8a7004700 10 client.38785.objecter successfully canceled tick 2024-02-23T00:31:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.043+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa890062c90 msgr2=0x7fa890065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.043+0000 7fa8a7004700 1 --2- 172.21.15.87:0/884223873 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa890062c90 0x7fa890065150 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 rx=0x7fa894009da0 tx=0x7fa894009cb0).stop 2024-02-23T00:31:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.043+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8a00fcb40 msgr2=0x7fa8a0063340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:24.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.043+0000 7fa8a7004700 1 --2- 172.21.15.87:0/884223873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8a00fcb40 0x7fa8a0063340 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 rx=0x7fa88c0041d0 tx=0x7fa88c009f60).stop 2024-02-23T00:31:24.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.044+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 shutdown_connections 2024-02-23T00:31:24.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.044+0000 7fa8a7004700 1 --2- 172.21.15.87:0/884223873 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa890062c90 0x7fa890065150 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:24.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.044+0000 7fa8a7004700 1 --2- 172.21.15.87:0/884223873 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa8a01020c0 0x7fa8a0109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:24.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.044+0000 7fa8a7004700 1 --2- 172.21.15.87:0/884223873 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa8a00fcb40 0x7fa8a0063340 unknown :-1 s=CLOSED pgs=3554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:24.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.044+0000 7fa8a7004700 1 --2- 172.21.15.87:0/884223873 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa8a00fd4a0 0x7fa8a0063830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:24.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.044+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 >> 172.21.15.87:0/884223873 conn(0x7fa8a006c370 msgr2=0x7fa8a00fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:24.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.044+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 shutdown_connections 2024-02-23T00:31:24.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.044+0000 7fa8a7004700 1 -- 172.21.15.87:0/884223873 wait complete. 2024-02-23T00:31:24.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:24.044+0000 7fa8a7004700 1 librados: shutdown 2024-02-23T00:31:24.069 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 3 pg_num to 13 2024-02-23T00:31:24.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.951637+0000 osd.1 (osd.1) 77 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:31:24.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:21.959588+0000 osd.1 (osd.1) 78 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:31:24.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:22.127826+0000 osd.3 (osd.3) 91 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:22.137158+0000 osd.3 (osd.3) 92 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:22.511505+0000 mon.a (mon.0) 1104 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.515798+0000 mgr.y (mgr.24437) 179 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.529693+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.560458+0000 mgr.y (mgr.24437) 180 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.581860+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.587353+0000 mgr.y (mgr.24437) 181 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.590475+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.591127+0000 mgr.y (mgr.24437) 182 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.593004+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.593353+0000 mgr.y (mgr.24437) 183 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.594394+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:31:24.087 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.594584+0000 mgr.y (mgr.24437) 184 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.601272+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.601543+0000 mgr.y (mgr.24437) 185 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.602188+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.602319+0000 mgr.y (mgr.24437) 186 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.602647+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.602754+0000 mgr.y (mgr.24437) 187 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.603111+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.603225+0000 mgr.y (mgr.24437) 188 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:22.653526+0000 mon.c (mon.2) 89 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:23.004488+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi087-82019-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:23.004696+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:23.030908+0000 mon.b (mon.1) 130 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache", "force_nonempty":""}]: dispatch 2024-02-23T00:31:24.088 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:23.036935+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache", "force_nonempty":""}]: dispatch 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:23.167182+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.87:0/884223873' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]: dispatch 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:23.531861+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]: dispatch 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.951637+0000 osd.1 (osd.1) 77 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:21.959588+0000 osd.1 (osd.1) 78 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:22.127826+0000 osd.3 (osd.3) 91 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:22.137158+0000 osd.3 (osd.3) 92 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:22.511505+0000 mon.a (mon.0) 1104 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.515798+0000 mgr.y (mgr.24437) 179 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.529693+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.560458+0000 mgr.y (mgr.24437) 180 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:31:24.089 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.581860+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.587353+0000 mgr.y (mgr.24437) 181 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.590475+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.591127+0000 mgr.y (mgr.24437) 182 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.593004+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.593353+0000 mgr.y (mgr.24437) 183 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.594394+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.594584+0000 mgr.y (mgr.24437) 184 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.601272+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.601543+0000 mgr.y (mgr.24437) 185 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.602188+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.602319+0000 mgr.y (mgr.24437) 186 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.602647+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.602754+0000 mgr.y (mgr.24437) 187 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:31:24.090 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.603111+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:31:24.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.603225+0000 mgr.y (mgr.24437) 188 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:31:24.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:22.653526+0000 mon.c (mon.2) 89 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:24.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:23.004488+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi087-82019-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:24.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:23.004696+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-23T00:31:24.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:23.030908+0000 mon.b (mon.1) 130 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache", "force_nonempty":""}]: dispatch 2024-02-23T00:31:24.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:23.036935+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache", "force_nonempty":""}]: dispatch 2024-02-23T00:31:24.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:23.167182+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.87:0/884223873' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]: dispatch 2024-02-23T00:31:24.091 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:23.531861+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]: dispatch 2024-02-23T00:31:24.139 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:24.164 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.951637+0000 osd.1 (osd.1) 77 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:21.959588+0000 osd.1 (osd.1) 78 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:22.127826+0000 osd.3 (osd.3) 91 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:22.137158+0000 osd.3 (osd.3) 92 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:22.511505+0000 mon.a (mon.0) 1104 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.515798+0000 mgr.y (mgr.24437) 179 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.529693+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.560458+0000 mgr.y (mgr.24437) 180 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.581860+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.587353+0000 mgr.y (mgr.24437) 181 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.590475+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:31:24.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.591127+0000 mgr.y (mgr.24437) 182 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.593004+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.593353+0000 mgr.y (mgr.24437) 183 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.594394+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.594584+0000 mgr.y (mgr.24437) 184 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.601272+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.601543+0000 mgr.y (mgr.24437) 185 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.602188+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.602319+0000 mgr.y (mgr.24437) 186 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.602647+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.602754+0000 mgr.y (mgr.24437) 187 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.603111+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.603225+0000 mgr.y (mgr.24437) 188 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:22.653526+0000 mon.c (mon.2) 89 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:24.166 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:23.004488+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi087-82019-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:24.167 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:23.004696+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-23T00:31:24.167 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:23.030908+0000 mon.b (mon.1) 130 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache", "force_nonempty":""}]: dispatch 2024-02-23T00:31:24.167 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:23.036935+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache", "force_nonempty":""}]: dispatch 2024-02-23T00:31:24.167 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:23.167182+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.87:0/884223873' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]: dispatch 2024-02-23T00:31:24.167 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:23.531861+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]: dispatch 2024-02-23T00:31:24.167 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:24.186 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:24.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:24.260 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:31:24.292 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:24.443 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:22.972755+0000 osd.1 (osd.1) 79 : cluster [DBG] 10.2 deep-scrub starts 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:23.002647+0000 osd.1 (osd.1) 80 : cluster [DBG] 10.2 deep-scrub ok 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:23.446454+0000 osd.0 (osd.0) 83 : cluster [DBG] 10.4 deep-scrub starts 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:23.456478+0000 osd.0 (osd.0) 84 : cluster [DBG] 10.4 deep-scrub ok 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:23.552988+0000 mgr.y (mgr.24437) 189 : cluster [DBG] pgmap v181: 428 pgs: 96 unknown, 332 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:23.608677+0000 osd.2 (osd.2) 67 : cluster [DBG] 10.3 deep-scrub starts 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:23.614685+0000 osd.2 (osd.2) 68 : cluster [DBG] 10.3 deep-scrub ok 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:23.662748+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:24.038265+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache", "force_nonempty":""}]': finished 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:24.038436+0000 mon.a (mon.0) 1119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]': finished 2024-02-23T00:31:25.344 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:24.038495+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:24.040936+0000 mon.b (mon.1) 132 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82099-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:24.043933+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:24.075683+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82099-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:24.075879+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:24.082196+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? 172.21.15.87:0/1894717922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi087-82011-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:24.653825+0000 mon.a (mon.0) 1124 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:24.665100+0000 mon.c (mon.2) 91 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:22.972755+0000 osd.1 (osd.1) 79 : cluster [DBG] 10.2 deep-scrub starts 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:23.002647+0000 osd.1 (osd.1) 80 : cluster [DBG] 10.2 deep-scrub ok 2024-02-23T00:31:25.345 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:23.446454+0000 osd.0 (osd.0) 83 : cluster [DBG] 10.4 deep-scrub starts 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:23.456478+0000 osd.0 (osd.0) 84 : cluster [DBG] 10.4 deep-scrub ok 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:23.552988+0000 mgr.y (mgr.24437) 189 : cluster [DBG] pgmap v181: 428 pgs: 96 unknown, 332 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:23.608677+0000 osd.2 (osd.2) 67 : cluster [DBG] 10.3 deep-scrub starts 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:23.614685+0000 osd.2 (osd.2) 68 : cluster [DBG] 10.3 deep-scrub ok 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:23.662748+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:24.038265+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache", "force_nonempty":""}]': finished 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:24.038436+0000 mon.a (mon.0) 1119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]': finished 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:24.038495+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:24.040936+0000 mon.b (mon.1) 132 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82099-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:24.043933+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:24.075683+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82099-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:24.075879+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:24.082196+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? 172.21.15.87:0/1894717922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi087-82011-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.346 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:24.653825+0000 mon.a (mon.0) 1124 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:25.347 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:24.665100+0000 mon.c (mon.2) 91 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:22.972755+0000 osd.1 (osd.1) 79 : cluster [DBG] 10.2 deep-scrub starts 2024-02-23T00:31:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:23.002647+0000 osd.1 (osd.1) 80 : cluster [DBG] 10.2 deep-scrub ok 2024-02-23T00:31:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:23.446454+0000 osd.0 (osd.0) 83 : cluster [DBG] 10.4 deep-scrub starts 2024-02-23T00:31:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:23.456478+0000 osd.0 (osd.0) 84 : cluster [DBG] 10.4 deep-scrub ok 2024-02-23T00:31:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:23.552988+0000 mgr.y (mgr.24437) 189 : cluster [DBG] pgmap v181: 428 pgs: 96 unknown, 332 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:23.608677+0000 osd.2 (osd.2) 67 : cluster [DBG] 10.3 deep-scrub starts 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:23.614685+0000 osd.2 (osd.2) 68 : cluster [DBG] 10.3 deep-scrub ok 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:23.662748+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:24.038265+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache", "force_nonempty":""}]': finished 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:24.038436+0000 mon.a (mon.0) 1119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "13"}]': finished 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:24.038495+0000 mon.a (mon.0) 1120 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:24.040936+0000 mon.b (mon.1) 132 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82099-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:24.043933+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:24.075683+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82099-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:24.075879+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:24.082196+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? 172.21.15.87:0/1894717922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi087-82011-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:24.653825+0000 mon.a (mon.0) 1124 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:24.665100+0000 mon.c (mon.2) 91 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:25.445 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:25.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:25.546 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:25.597 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:25.978 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:25.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:25.995 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-23T00:31:26.087 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:26.131 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:26.150 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:26.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:26.229 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:25.016982+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82099-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:26.229 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:25.017083+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:26.229 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:25.017187+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? 172.21.15.87:0/1894717922' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi087-82011-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:25.017303+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:25.062721+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.87:0/2964548532' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi087-82019-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:25.064103+0000 mon.a (mon.0) 1129 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi087-82019-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:25.069202+0000 mon.b (mon.1) 135 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache"}]: dispatch 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:25.093722+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache"}]: dispatch 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:25.511328+0000 osd.0 (osd.0) 85 : cluster [DBG] 10.8 deep-scrub starts 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:25.514390+0000 osd.0 (osd.0) 86 : cluster [DBG] 10.8 deep-scrub ok 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:25.556030+0000 mon.a (mon.0) 1131 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:25.670849+0000 mon.c (mon.2) 92 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:26.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:25.016982+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82099-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:25.017083+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:25.017187+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? 172.21.15.87:0/1894717922' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi087-82011-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:25.017303+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:25.062721+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.87:0/2964548532' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi087-82019-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:25.064103+0000 mon.a (mon.0) 1129 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi087-82019-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:25.069202+0000 mon.b (mon.1) 135 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache"}]: dispatch 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:25.093722+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache"}]: dispatch 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:25.511328+0000 osd.0 (osd.0) 85 : cluster [DBG] 10.8 deep-scrub starts 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:25.514390+0000 osd.0 (osd.0) 86 : cluster [DBG] 10.8 deep-scrub ok 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:25.556030+0000 mon.a (mon.0) 1131 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-23T00:31:26.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:25.670849+0000 mon.c (mon.2) 92 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:26.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:26.326 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:25.016982+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82099-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:25.017083+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:25.017187+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? 172.21.15.87:0/1894717922' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi087-82011-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:25.017303+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-23T00:31:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:25.062721+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.87:0/2964548532' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi087-82019-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:25.064103+0000 mon.a (mon.0) 1129 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi087-82019-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:25.069202+0000 mon.b (mon.1) 135 : audit [INF] from='client.? 172.21.15.87:0/255780493' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache"}]: dispatch 2024-02-23T00:31:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:25.093722+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache"}]: dispatch 2024-02-23T00:31:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:25.511328+0000 osd.0 (osd.0) 85 : cluster [DBG] 10.8 deep-scrub starts 2024-02-23T00:31:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:25.514390+0000 osd.0 (osd.0) 86 : cluster [DBG] 10.8 deep-scrub ok 2024-02-23T00:31:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:25.556030+0000 mon.a (mon.0) 1131 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-23T00:31:26.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:25.670849+0000 mon.c (mon.2) 92 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:26.709 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:26.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:26.813 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:26.839 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:26.857 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:26.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:26.961 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:26.988 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:27.015 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:27.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:27.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:27.149 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:27.168 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:27.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:27.272 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:27.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.293+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3183969299 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad41044a0 msgr2=0x7f0ad4104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.293+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3183969299 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad41044a0 0x7f0ad4104920 secure :-1 s=READY pgs=4405 cs=0 l=1 rev1=1 rx=0x7f0acc009be0 tx=0x7f0acc004d10).stop 2024-02-23T00:31:27.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.293+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3183969299 shutdown_connections 2024-02-23T00:31:27.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.293+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3183969299 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad41044a0 0x7f0ad4104920 unknown :-1 s=CLOSED pgs=4405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.293+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3183969299 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ad40692d0 0x7f0ad40696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.293+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3183969299 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0ad4069ba0 0x7f0ad4103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.293+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3183969299 >> 172.21.15.87:0/3183969299 conn(0x7f0ad4063db0 msgr2=0x7f0ad4067560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:27.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.293+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3183969299 shutdown_connections 2024-02-23T00:31:27.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.293+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3183969299 wait complete. 2024-02-23T00:31:27.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 librados: starting msgr at 2024-02-23T00:31:27.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 librados: starting objecter 2024-02-23T00:31:27.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 Processor -- start 2024-02-23T00:31:27.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 -- start start 2024-02-23T00:31:27.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 librados: setting wanted keys 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 librados: calling monclient init 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ad40692d0 0x7f0ad410ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0ad4069ba0 0x7f0ad410cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad41044a0 0x7f0ad410d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4111540 con 0x7f0ad40692d0 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.294+0000 7f0adbf7a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41116b0 con 0x7f0ad41044a0 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0adbf7a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4111870 con 0x7f0ad4069ba0 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ad9d16700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ad40692d0 0x7f0ad410ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ad9d16700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ad40692d0 0x7f0ad410ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:36332/0 (socket says 172.21.15.87:36332) 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ad9d16700 1 -- 172.21.15.87:0/3117103828 learned_addr learned my addr 172.21.15.87:0/3117103828 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:27.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ada517700 1 --2- 172.21.15.87:0/3117103828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad41044a0 0x7f0ad410d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ada517700 1 -- 172.21.15.87:0/3117103828 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0ad4069ba0 msgr2=0x7f0ad410cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ada517700 1 --2- 172.21.15.87:0/3117103828 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0ad4069ba0 0x7f0ad410cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:31:27.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ada517700 1 -- 172.21.15.87:0/3117103828 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ad40692d0 msgr2=0x7f0ad410ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ada517700 1 --2- 172.21.15.87:0/3117103828 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0ad40692d0 0x7f0ad410ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ada517700 1 -- 172.21.15.87:0/3117103828 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc0097c0 con 0x7f0ad41044a0 2024-02-23T00:31:27.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.295+0000 7f0ada517700 1 --2- 172.21.15.87:0/3117103828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad41044a0 0x7f0ad410d450 secure :-1 s=READY pgs=4407 cs=0 l=1 rev1=1 rx=0x7f0acc00b780 tx=0x7f0acc004d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.296+0000 7f0acaffd700 10 client.?.objecter ms_handle_connect 0x7f0ad41044a0 2024-02-23T00:31:27.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.296+0000 7f0acaffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:27.304 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:27.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.296+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc00e040 con 0x7f0ad41044a0 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.296+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4111af0 con 0x7f0ad41044a0 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.296+0000 7f0adbf7a700 10 client.41980.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.296+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad4111f30 con 0x7f0ad41044a0 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.296+0000 7f0adbf7a700 1 librados: init done 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.297+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0acc018040 con 0x7f0ad41044a0 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.297+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc013d50 con 0x7f0ad41044a0 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.299+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0acc007bf0 con 0x7f0ad41044a0 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.299+0000 7f0acaffd700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0ac0062ce0 0x7f0ac00651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.299+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(136..136 src has 38..136) v4 ==== 10939+0+0 (secure 0 0 0) 0x7f0acc07ed30 con 0x7f0ad41044a0 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.299+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 osd_map(136..136 src has 38..136) v4 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.299+0000 7f0acaffd700 3 client.41980.objecter handle_osd_map got epochs [136,136] > 0 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.299+0000 7f0acaffd700 3 client.41980.objecter handle_osd_map decoding full epoch 136 2024-02-23T00:31:27.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.300+0000 7f0acaffd700 20 client.41980.objecter dump_active .. 0 homeless 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.301+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0ab8004f30 con 0x7f0ad41044a0 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.301+0000 7f0ad9d16700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0ac0062ce0 0x7f0ac00651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v136) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0acc021070 con 0x7f0ad41044a0 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 1 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0ab8004560 0x7f0ab80049a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab80051d0 osd=0 3 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80051d0 osd=0 3 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab80051d0 osd=0 3 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 1 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab80051d0 osd=0 3 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 0 1 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80051d0 osd=0 4 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 10 client.41980.objecter _send_command 1 2024-02-23T00:31:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.302+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ab8008960 con 0x7f0ab8004560 2024-02-23T00:31:27.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.303+0000 7f0ad9d16700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0ac0062ce0 0x7f0ac00651a0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 rx=0x7f0ad0009fd0 tx=0x7f0ad000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.303+0000 7f0acaffd700 10 client.41980.objecter ms_handle_connect 0x7f0ac0062ce0 2024-02-23T00:31:27.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.307+0000 7f0ad9515700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0ab8004560 0x7f0ab80049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.307+0000 7f0ad9515700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0ab8004560 0x7f0ab80049a0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.307+0000 7f0acaffd700 10 client.41980.objecter ms_handle_connect 0x7f0ab8004560 2024-02-23T00:31:27.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.308+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ab8008960 con 0x7f0ab8004560 2024-02-23T00:31:27.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.308+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:31:27.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.308+0000 7f0acaffd700 10 client.41980.objecter _finish_command 1 = system:0 2024-02-23T00:31:27.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.308+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab80051d0 osd=0 4 2024-02-23T00:31:27.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.308+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 0 1 2024-02-23T00:31:27.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:27.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 2 2024-02-23T00:31:27.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab80051d0 osd=0 3 2024-02-23T00:31:27.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80051d0 osd=0 3 2024-02-23T00:31:27.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab80051d0 osd=0 3 2024-02-23T00:31:27.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 2 2024-02-23T00:31:27.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab80051d0 osd=0 3 2024-02-23T00:31:27.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 0 2 2024-02-23T00:31:27.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80051d0 osd=0 4 2024-02-23T00:31:27.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 10 client.41980.objecter _send_command 2 2024-02-23T00:31:27.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.335+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0ab80081f0 con 0x7f0ab8004560 2024-02-23T00:31:27.352 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:27.367 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:27.372 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:31:27.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.366+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0ab80081f0 con 0x7f0ab8004560 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.366+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.366+0000 7f0acaffd700 10 client.41980.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.366+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab80051d0 osd=0 4 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.366+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 0 2 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 3 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6805/685354171 conn(0x7f0ab8002c90 0x7f0ab8003120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8008650 osd=1 3 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8008650 osd=1 3 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8008650 osd=1 3 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 3 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab8008650 osd=1 3 2024-02-23T00:31:27.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 1 3 2024-02-23T00:31:27.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8008650 osd=1 4 2024-02-23T00:31:27.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 10 client.41980.objecter _send_command 3 2024-02-23T00:31:27.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.367+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ab80036a0 con 0x7f0ab8002c90 2024-02-23T00:31:27.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.371+0000 7f0ad9d16700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6805/685354171 conn(0x7f0ab8002c90 0x7f0ab8003120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.372+0000 7f0ad9d16700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6805/685354171 conn(0x7f0ab8002c90 0x7f0ab8003120 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.373+0000 7f0acaffd700 10 client.41980.objecter ms_handle_connect 0x7f0ab8002c90 2024-02-23T00:31:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.376+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ab80036a0 con 0x7f0ab8002c90 2024-02-23T00:31:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.376+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:31:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.376+0000 7f0acaffd700 10 client.41980.objecter _finish_command 3 = system:0 2024-02-23T00:31:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.376+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab8008650 osd=1 4 2024-02-23T00:31:27.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.376+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 1 3 2024-02-23T00:31:27.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:27.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 4 2024-02-23T00:31:27.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8008650 osd=1 3 2024-02-23T00:31:27.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8008650 osd=1 3 2024-02-23T00:31:27.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8008650 osd=1 3 2024-02-23T00:31:27.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 4 2024-02-23T00:31:27.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab8008650 osd=1 3 2024-02-23T00:31:27.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 1 4 2024-02-23T00:31:27.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8008650 osd=1 4 2024-02-23T00:31:27.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 10 client.41980.objecter _send_command 4 2024-02-23T00:31:27.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.409+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0ab8004090 con 0x7f0ab8002c90 2024-02-23T00:31:27.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0ab8004090 con 0x7f0ab8002c90 2024-02-23T00:31:27.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:27.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0acaffd700 10 client.41980.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab8008650 osd=1 4 2024-02-23T00:31:27.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 1 4 2024-02-23T00:31:27.418 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:31:27.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 5 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0ab800aff0 0x7f0ab800d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8007d20 osd=2 3 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8007d20 osd=2 3 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8007d20 osd=2 3 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 5 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab8007d20 osd=2 3 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 2 5 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8007d20 osd=2 4 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 10 client.41980.objecter _send_command 5 2024-02-23T00:31:27.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.416+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ab8010030 con 0x7f0ab800aff0 2024-02-23T00:31:27.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.426+0000 7f0ad9515700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0ab800aff0 0x7f0ab800d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.427+0000 7f0ad9515700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0ab800aff0 0x7f0ab800d3e0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.429+0000 7f0acaffd700 10 client.41980.objecter ms_handle_connect 0x7f0ab800aff0 2024-02-23T00:31:27.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.430+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ab8010030 con 0x7f0ab800aff0 2024-02-23T00:31:27.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.430+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:31:27.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.430+0000 7f0acaffd700 10 client.41980.objecter _finish_command 5 = system:0 2024-02-23T00:31:27.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.430+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab8007d20 osd=2 4 2024-02-23T00:31:27.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.430+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 2 5 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 6 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8007d20 osd=2 3 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8007d20 osd=2 3 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8007d20 osd=2 3 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 6 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab8007d20 osd=2 3 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 2 6 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8007d20 osd=2 4 2024-02-23T00:31:27.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 10 client.41980.objecter _send_command 6 2024-02-23T00:31:27.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.448+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0ab8003b00 con 0x7f0ab800aff0 2024-02-23T00:31:27.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0ab8003b00 con 0x7f0ab800aff0 2024-02-23T00:31:27.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:27.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0acaffd700 10 client.41980.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab8007d20 osd=2 4 2024-02-23T00:31:27.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 2 6 2024-02-23T00:31:27.453 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:31:27.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:27.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 7 2024-02-23T00:31:27.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6813/128059790 conn(0x7f0ab800ea50 0x7f0ab800ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8001bb0 osd=3 3 2024-02-23T00:31:27.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8001bb0 osd=3 3 2024-02-23T00:31:27.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8001bb0 osd=3 3 2024-02-23T00:31:27.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 7 2024-02-23T00:31:27.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab8001bb0 osd=3 3 2024-02-23T00:31:27.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 3 7 2024-02-23T00:31:27.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8001bb0 osd=3 4 2024-02-23T00:31:27.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 10 client.41980.objecter _send_command 7 2024-02-23T00:31:27.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.451+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ab8002230 con 0x7f0ab800ea50 2024-02-23T00:31:27.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.452+0000 7f0ada517700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6813/128059790 conn(0x7f0ab800ea50 0x7f0ab800ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.453+0000 7f0ada517700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6813/128059790 conn(0x7f0ab800ea50 0x7f0ab800ef00 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.453+0000 7f0acaffd700 10 client.41980.objecter ms_handle_connect 0x7f0ab800ea50 2024-02-23T00:31:27.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.453+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ab8002230 con 0x7f0ab800ea50 2024-02-23T00:31:27.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.453+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:31:27.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.453+0000 7f0acaffd700 10 client.41980.objecter _finish_command 7 = system:0 2024-02-23T00:31:27.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.453+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab8001bb0 osd=3 4 2024-02-23T00:31:27.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.453+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 3 7 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 8 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8001bb0 osd=3 3 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8001bb0 osd=3 3 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8001bb0 osd=3 3 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 8 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab8001bb0 osd=3 3 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 3 8 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8001bb0 osd=3 4 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 10 client.41980.objecter _send_command 8 2024-02-23T00:31:27.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.472+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0ab800fbc0 con 0x7f0ab800ea50 2024-02-23T00:31:27.487 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:27.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:27.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.498+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0ab800fbc0 con 0x7f0ab800ea50 2024-02-23T00:31:27.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.498+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:27.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.498+0000 7f0acaffd700 10 client.41980.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.498+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab8001bb0 osd=3 4 2024-02-23T00:31:27.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.498+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 3 8 2024-02-23T00:31:27.507 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:31:27.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:27.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 9 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0ab80121e0 0x7f0ab80145d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab800fbc0 osd=4 3 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab800fbc0 osd=4 3 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab800fbc0 osd=4 3 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 9 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab800fbc0 osd=4 3 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 4 9 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab800fbc0 osd=4 4 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 10 client.41980.objecter _send_command 9 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ab8017260 con 0x7f0ab80121e0 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.506+0000 7f0ad9d16700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0ab80121e0 0x7f0ab80145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.507+0000 7f0ad9d16700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0ab80121e0 0x7f0ab80145d0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.511+0000 7f0acaffd700 10 client.41980.objecter ms_handle_connect 0x7f0ab80121e0 2024-02-23T00:31:27.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.511+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ab8017260 con 0x7f0ab80121e0 2024-02-23T00:31:27.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.511+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:31:27.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.511+0000 7f0acaffd700 10 client.41980.objecter _finish_command 9 = system:0 2024-02-23T00:31:27.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.511+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab800fbc0 osd=4 4 2024-02-23T00:31:27.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.511+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 4 9 2024-02-23T00:31:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 10 2024-02-23T00:31:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab800fbc0 osd=4 3 2024-02-23T00:31:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab800fbc0 osd=4 3 2024-02-23T00:31:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab800fbc0 osd=4 3 2024-02-23T00:31:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 10 2024-02-23T00:31:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab800fbc0 osd=4 3 2024-02-23T00:31:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 4 10 2024-02-23T00:31:27.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab800fbc0 osd=4 4 2024-02-23T00:31:27.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 10 client.41980.objecter _send_command 10 2024-02-23T00:31:27.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.542+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0ab8017260 con 0x7f0ab80121e0 2024-02-23T00:31:27.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0ab8017260 con 0x7f0ab80121e0 2024-02-23T00:31:27.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:27.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0acaffd700 10 client.41980.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab800fbc0 osd=4 4 2024-02-23T00:31:27.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 4 10 2024-02-23T00:31:27.546 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:31:27.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 11 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0ab8015140 0x7f0ab8015540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8014ac0 osd=5 3 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8014ac0 osd=5 3 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8014ac0 osd=5 3 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 11 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab8014ac0 osd=5 3 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 5 11 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8014ac0 osd=5 4 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 10 client.41980.objecter _send_command 11 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ab80170a0 con 0x7f0ab8015140 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.544+0000 7f0ad9515700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0ab8015140 0x7f0ab8015540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.545+0000 7f0ad9515700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0ab8015140 0x7f0ab8015540 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.545+0000 7f0acaffd700 10 client.41980.objecter ms_handle_connect 0x7f0ab8015140 2024-02-23T00:31:27.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.545+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ac400f040 con 0x7f0ab8015140 2024-02-23T00:31:27.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.545+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:31:27.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.545+0000 7f0acaffd700 10 client.41980.objecter _finish_command 11 = system:0 2024-02-23T00:31:27.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.545+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab8014ac0 osd=5 4 2024-02-23T00:31:27.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.545+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 5 11 2024-02-23T00:31:27.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:27.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 12 2024-02-23T00:31:27.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8014ac0 osd=5 3 2024-02-23T00:31:27.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8014ac0 osd=5 3 2024-02-23T00:31:27.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab8014ac0 osd=5 3 2024-02-23T00:31:27.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 12 2024-02-23T00:31:27.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab8014ac0 osd=5 3 2024-02-23T00:31:27.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 5 12 2024-02-23T00:31:27.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8014ac0 osd=5 4 2024-02-23T00:31:27.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 10 client.41980.objecter _send_command 12 2024-02-23T00:31:27.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.573+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0ab80194c0 con 0x7f0ab8015140 2024-02-23T00:31:27.579 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.577+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0ab80194c0 con 0x7f0ab8015140 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.577+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.577+0000 7f0acaffd700 10 client.41980.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.577+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab8014ac0 osd=5 4 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.577+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 5 12 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.578+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.578+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 13 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.578+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0ab8019be0 0x7f0ab801a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.578+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab80194c0 osd=6 3 2024-02-23T00:31:27.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.578+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80194c0 osd=6 3 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.578+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab80194c0 osd=6 3 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.578+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 13 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.578+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab80194c0 osd=6 3 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.578+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 6 13 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.579+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80194c0 osd=6 4 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.579+0000 7f0adbf7a700 10 client.41980.objecter _send_command 13 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.579+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ab801d8f0 con 0x7f0ab8019be0 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.579+0000 7f0ada517700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0ab8019be0 0x7f0ab801a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.579+0000 7f0acaffd700 10 client.41980.objecter ms_handle_connect 0x7f0ab8019be0 2024-02-23T00:31:27.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.579+0000 7f0ada517700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0ab8019be0 0x7f0ab801a060 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.581+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ab801d8f0 con 0x7f0ab8019be0 2024-02-23T00:31:27.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.581+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:31:27.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.581+0000 7f0acaffd700 10 client.41980.objecter _finish_command 13 = system:0 2024-02-23T00:31:27.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.581+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab80194c0 osd=6 4 2024-02-23T00:31:27.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.581+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 6 13 2024-02-23T00:31:27.598 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:25.388706+0000 osd.5 (osd.5) 71 : cluster [DBG] 10.6 deep-scrub starts 2024-02-23T00:31:27.598 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:25.389503+0000 osd.5 (osd.5) 72 : cluster [DBG] 10.6 deep-scrub ok 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:25.554585+0000 mgr.y (mgr.24437) 190 : cluster [DBG] pgmap v184: 492 pgs: 32 creating+peering, 2 active+clean+snaptrim, 458 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:25.960182+0000 osd.6 (osd.6) 61 : cluster [DBG] 10.0 deep-scrub starts 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:25.970869+0000 osd.6 (osd.6) 62 : cluster [DBG] 10.0 deep-scrub ok 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:26.073885+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi087-82019-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:26.073982+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache"}]': finished 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:26.074141+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:26.074185+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:26.213287+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:26.693769+0000 mon.c (mon.2) 93 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:26.705388+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:27.599 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:26.919094+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:31:27.600 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:26.920130+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:31:27.600 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:26.994981+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:27.600 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:26.995069+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-23T00:31:27.600 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:27.001944+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:27.600 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:27.006922+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.87:0/244424669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi087-82011-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:27.600 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:27.009368+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:27.600 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:27.009764+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi087-82011-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:27.600 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:27.302123+0000 mon.b (mon.1) 139 : audit [DBG] from='client.? 172.21.15.87:0/3117103828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:25.388706+0000 osd.5 (osd.5) 71 : cluster [DBG] 10.6 deep-scrub starts 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:25.389503+0000 osd.5 (osd.5) 72 : cluster [DBG] 10.6 deep-scrub ok 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:25.554585+0000 mgr.y (mgr.24437) 190 : cluster [DBG] pgmap v184: 492 pgs: 32 creating+peering, 2 active+clean+snaptrim, 458 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:25.960182+0000 osd.6 (osd.6) 61 : cluster [DBG] 10.0 deep-scrub starts 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:25.970869+0000 osd.6 (osd.6) 62 : cluster [DBG] 10.0 deep-scrub ok 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:26.073885+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi087-82019-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:26.073982+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache"}]': finished 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:26.074141+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:26.074185+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-23T00:31:27.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:26.213287+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:26.693769+0000 mon.c (mon.2) 93 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:26.705388+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:26.919094+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:26.920130+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:26.994981+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:26.995069+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:27.001944+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:27.006922+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.87:0/244424669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi087-82011-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:27.009368+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:27.009764+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi087-82011-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:27.602 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:27.302123+0000 mon.b (mon.1) 139 : audit [DBG] from='client.? 172.21.15.87:0/3117103828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:27.603 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 14 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab80194c0 osd=6 3 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80194c0 osd=6 3 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab80194c0 osd=6 3 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 14 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab80194c0 osd=6 3 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 6 14 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80194c0 osd=6 4 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 10 client.41980.objecter _send_command 14 2024-02-23T00:31:27.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.610+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0ab801d8f0 con 0x7f0ab8019be0 2024-02-23T00:31:27.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.619+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0ab801d8f0 con 0x7f0ab8019be0 2024-02-23T00:31:27.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.619+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:27.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.619+0000 7f0acaffd700 10 client.41980.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.619+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab80194c0 osd=6 4 2024-02-23T00:31:27.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.619+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 6 14 2024-02-23T00:31:27.624 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:31:27.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:27.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 15 2024-02-23T00:31:27.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6812/373787000 conn(0x7f0ab801cc10 0x7f0ab801d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:27.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab801d8f0 osd=7 3 2024-02-23T00:31:27.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab801d8f0 osd=7 3 2024-02-23T00:31:27.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab801d8f0 osd=7 3 2024-02-23T00:31:27.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 15 2024-02-23T00:31:27.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab801d8f0 osd=7 3 2024-02-23T00:31:27.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 7 15 2024-02-23T00:31:27.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab801d8f0 osd=7 4 2024-02-23T00:31:27.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 10 client.41980.objecter _send_command 15 2024-02-23T00:31:27.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.624+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ab8020cd0 con 0x7f0ab801cc10 2024-02-23T00:31:27.629 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:27.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.629+0000 7f0ad9d16700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6812/373787000 conn(0x7f0ab801cc10 0x7f0ab801d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:27.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.638+0000 7f0acaffd700 10 client.41980.objecter ms_handle_connect 0x7f0ab801cc10 2024-02-23T00:31:27.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.638+0000 7f0ad9d16700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6812/373787000 conn(0x7f0ab801cc10 0x7f0ab801d070 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:27.649 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:27.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:27.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.654+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0ab8020cd0 con 0x7f0ab801cc10 2024-02-23T00:31:27.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.654+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:31:27.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.654+0000 7f0acaffd700 10 client.41980.objecter _finish_command 15 = system:0 2024-02-23T00:31:27.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.654+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab801d8f0 osd=7 4 2024-02-23T00:31:27.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.654+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 7 15 2024-02-23T00:31:27.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 10 client.41980.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:27.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign -1 16 2024-02-23T00:31:27.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab801d8f0 osd=7 3 2024-02-23T00:31:27.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab801d8f0 osd=7 3 2024-02-23T00:31:27.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 20 client.41980.objecter _get_session s=0x7f0ab801d8f0 osd=7 3 2024-02-23T00:31:27.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_remove -1 16 2024-02-23T00:31:27.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 20 client.41980.objecter get_session s=0x7f0ab801d8f0 osd=7 3 2024-02-23T00:31:27.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 15 client.41980.objecter _session_command_op_assign 7 16 2024-02-23T00:31:27.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab801d8f0 osd=7 4 2024-02-23T00:31:27.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 10 client.41980.objecter _send_command 16 2024-02-23T00:31:27.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.681+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0ab8015a30 con 0x7f0ab801cc10 2024-02-23T00:31:27.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.683+0000 7f0acaffd700 1 -- 172.21.15.87:0/3117103828 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0ab8015a30 con 0x7f0ab801cc10 2024-02-23T00:31:27.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.683+0000 7f0acaffd700 10 client.41980.objecter ms_dispatch 0x7f0ad4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:27.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.683+0000 7f0acaffd700 10 client.41980.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.683+0000 7f0acaffd700 20 client.41980.objecter put_session s=0x7f0ab801d8f0 osd=7 4 2024-02-23T00:31:27.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.683+0000 7f0acaffd700 15 client.41980.objecter _session_command_op_remove 7 16 2024-02-23T00:31:27.685 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:31:27.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.684+0000 7f0adbf7a700 10 librados: watch_flush enter 2024-02-23T00:31:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 librados: watch_flush exit 2024-02-23T00:31:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 client.41980.objecter close_session for osd.0 2024-02-23T00:31:27.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0ab8004560 msgr2=0x7f0ab80049a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0ab8004560 0x7f0ab80049a0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80051d0 osd=0 1 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 client.41980.objecter close_session for osd.1 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6805/685354171 conn(0x7f0ab8002c90 msgr2=0x7f0ab8003120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6805/685354171 conn(0x7f0ab8002c90 0x7f0ab8003120 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8008650 osd=1 1 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 client.41980.objecter close_session for osd.2 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0ab800aff0 msgr2=0x7f0ab800d3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0ab800aff0 0x7f0ab800d3e0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8007d20 osd=2 1 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 client.41980.objecter close_session for osd.3 2024-02-23T00:31:27.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6813/128059790 conn(0x7f0ab800ea50 msgr2=0x7f0ab800ef00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6813/128059790 conn(0x7f0ab800ea50 0x7f0ab800ef00 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8001bb0 osd=3 1 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 client.41980.objecter close_session for osd.4 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0ab80121e0 msgr2=0x7f0ab80145d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0ab80121e0 0x7f0ab80145d0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab800fbc0 osd=4 1 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 client.41980.objecter close_session for osd.5 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0ada517700 1 -- 172.21.15.87:0/3117103828 reap_dead start 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0ab8015140 msgr2=0x7f0ab8015540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0ab8015140 0x7f0ab8015540 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0ada517700 1 -- 172.21.15.87:0/3117103828 reap_dead start 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0ada517700 1 -- 172.21.15.87:0/3117103828 reap_dead start 2024-02-23T00:31:27.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab8014ac0 osd=5 1 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 client.41980.objecter close_session for osd.6 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0ab8019be0 msgr2=0x7f0ab801a060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0ab8019be0 0x7f0ab801a060 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab80194c0 osd=6 1 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 client.41980.objecter close_session for osd.7 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6812/373787000 conn(0x7f0ab801cc10 msgr2=0x7f0ab801d070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.171:6812/373787000 conn(0x7f0ab801cc10 0x7f0ab801d070 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 20 client.41980.objecter put_session s=0x7f0ab801d8f0 osd=7 1 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 20 client.41980.objecter shutdown clearing up homeless session... 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 10 client.41980.objecter successfully canceled tick 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0ac0062ce0 msgr2=0x7f0ac00651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0ac0062ce0 0x7f0ac00651a0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 rx=0x7f0ad0009fd0 tx=0x7f0ad000b040).stop 2024-02-23T00:31:27.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad41044a0 msgr2=0x7f0ad410d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:27.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 --2- 172.21.15.87:0/3117103828 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0ad41044a0 0x7f0ad410d450 secure :-1 s=READY pgs=4407 cs=0 l=1 rev1=1 rx=0x7f0acc00b780 tx=0x7f0acc004d10).stop 2024-02-23T00:31:27.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0ada517700 1 -- 172.21.15.87:0/3117103828 reap_dead start 2024-02-23T00:31:27.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 shutdown_connections 2024-02-23T00:31:27.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.687+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 >> 172.21.15.87:0/3117103828 conn(0x7f0ad4063db0 msgr2=0x7f0ad4064f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:27.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:25.388706+0000 osd.5 (osd.5) 71 : cluster [DBG] 10.6 deep-scrub starts 2024-02-23T00:31:27.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:25.389503+0000 osd.5 (osd.5) 72 : cluster [DBG] 10.6 deep-scrub ok 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:25.554585+0000 mgr.y (mgr.24437) 190 : cluster [DBG] pgmap v184: 492 pgs: 32 creating+peering, 2 active+clean+snaptrim, 458 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:25.960182+0000 osd.6 (osd.6) 61 : cluster [DBG] 10.0 deep-scrub starts 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:25.970869+0000 osd.6 (osd.6) 62 : cluster [DBG] 10.0 deep-scrub ok 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:26.073885+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi087-82019-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:26.073982+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82099-10", "tierpool":"test-rados-api-smithi087-82099-10-cache"}]': finished 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:26.074141+0000 mon.a (mon.0) 1134 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:26.074185+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:26.213287+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:26.693769+0000 mon.c (mon.2) 93 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:26.705388+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:26.919094+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:26.920130+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:26.994981+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:26.995069+0000 mon.a (mon.0) 1138 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-23T00:31:27.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:27.001944+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:27.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:27.006922+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.87:0/244424669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi087-82011-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:27.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:27.009368+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:27.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:27.009764+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi087-82011-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:27.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:27.302123+0000 mon.b (mon.1) 139 : audit [DBG] from='client.? 172.21.15.87:0/3117103828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:27.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.707+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 shutdown_connections 2024-02-23T00:31:27.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.726+0000 7f0adbf7a700 1 -- 172.21.15.87:0/3117103828 wait complete. 2024-02-23T00:31:27.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:27.726+0000 7f0adbf7a700 1 librados: shutdown 2024-02-23T00:31:27.751 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.751 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.751 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.751 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.751 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.751 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.751 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.752 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:31:27.752 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:31:27.779 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:27.798 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:31:27.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:27.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:31:27.902 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:27.977 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: Running main() from gmock_main.cc 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [----------] Global test environment set-up. 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolList (7496 ms) 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3618 ms) 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (4289 ms) 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-02-23T00:31:28.037 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3252 ms) 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2866 ms) 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6579 ms) 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4852 ms) 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5230 ms) 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7767 ms) 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [----------] 9 tests from LibRadosPools (45949 ms total) 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [----------] Global test environment tear-down 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (45949 ms total) 2024-02-23T00:31:28.038 INFO:tasks.workunit.client.0.smithi087.stdout: api_pool: [ PASSED ] 9 tests. 2024-02-23T00:31:28.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:26.689736+0000 osd.0 (osd.0) 87 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:31:28.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:26.704980+0000 osd.0 (osd.0) 88 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:31:28.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:27.171928+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:31:28.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:27.176191+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:31:28.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:27.414876+0000 osd.5 (osd.5) 73 : cluster [DBG] 10.9 deep-scrub starts 2024-02-23T00:31:28.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:27.449796+0000 osd.5 (osd.5) 74 : cluster [DBG] 10.9 deep-scrub ok 2024-02-23T00:31:28.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:27.556195+0000 mgr.y (mgr.24437) 191 : cluster [DBG] pgmap v187: 438 pgs: 42 unknown, 2 active+clean+snaptrim, 394 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-02-23T00:31:28.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:27.557831+0000 mon.a (mon.0) 1141 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]: dispatch 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:27.722973+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.007677+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-11"}]': finished 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.007769+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi087-82011-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.007972+0000 mon.a (mon.0) 1144 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]': finished 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:28.008021+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.013541+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.87:0/2059300124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi087-82019-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.013912+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi087-82019-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.015904+0000 mon.b (mon.1) 141 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-11", "mode": "writeback"}]: dispatch 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.017090+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-11", "mode": "writeback"}]: dispatch 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.024281+0000 mon.b (mon.1) 142 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:28.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.028518+0000 mon.a (mon.0) 1148 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:26.689736+0000 osd.0 (osd.0) 87 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:26.704980+0000 osd.0 (osd.0) 88 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:27.171928+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:27.176191+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:27.414876+0000 osd.5 (osd.5) 73 : cluster [DBG] 10.9 deep-scrub starts 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:27.449796+0000 osd.5 (osd.5) 74 : cluster [DBG] 10.9 deep-scrub ok 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:27.556195+0000 mgr.y (mgr.24437) 191 : cluster [DBG] pgmap v187: 438 pgs: 42 unknown, 2 active+clean+snaptrim, 394 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:27.557831+0000 mon.a (mon.0) 1141 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]: dispatch 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:27.722973+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.007677+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-11"}]': finished 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.007769+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi087-82011-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:28.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.007972+0000 mon.a (mon.0) 1144 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]': finished 2024-02-23T00:31:28.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:28.008021+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-02-23T00:31:28.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.013541+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.87:0/2059300124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi087-82019-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:28.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.013912+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi087-82019-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:28.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.015904+0000 mon.b (mon.1) 141 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-11", "mode": "writeback"}]: dispatch 2024-02-23T00:31:28.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.017090+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-11", "mode": "writeback"}]: dispatch 2024-02-23T00:31:28.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.024281+0000 mon.b (mon.1) 142 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:28.839 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.028518+0000 mon.a (mon.0) 1148 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:28.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:26.689736+0000 osd.0 (osd.0) 87 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:31:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:26.704980+0000 osd.0 (osd.0) 88 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:31:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:27.171928+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:31:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:27.176191+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:31:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:27.414876+0000 osd.5 (osd.5) 73 : cluster [DBG] 10.9 deep-scrub starts 2024-02-23T00:31:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:27.449796+0000 osd.5 (osd.5) 74 : cluster [DBG] 10.9 deep-scrub ok 2024-02-23T00:31:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:27.556195+0000 mgr.y (mgr.24437) 191 : cluster [DBG] pgmap v187: 438 pgs: 42 unknown, 2 active+clean+snaptrim, 394 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-02-23T00:31:28.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:27.557831+0000 mon.a (mon.0) 1141 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]: dispatch 2024-02-23T00:31:28.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:27.722973+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:28.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.007677+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-11"}]': finished 2024-02-23T00:31:28.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.007769+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi087-82011-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:28.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.007972+0000 mon.a (mon.0) 1144 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "13"}]': finished 2024-02-23T00:31:28.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:28.008021+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-02-23T00:31:28.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.013541+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.87:0/2059300124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi087-82019-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:28.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.013912+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi087-82019-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:28.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.015904+0000 mon.b (mon.1) 141 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-11", "mode": "writeback"}]: dispatch 2024-02-23T00:31:28.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.017090+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-11", "mode": "writeback"}]: dispatch 2024-02-23T00:31:28.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.024281+0000 mon.b (mon.1) 142 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:28.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.028518+0000 mon.a (mon.0) 1148 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:29.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:27.752865+0000 osd.2 (osd.2) 69 : cluster [DBG] 10.5 deep-scrub starts 2024-02-23T00:31:29.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:27.755194+0000 osd.2 (osd.2) 70 : cluster [DBG] 10.5 deep-scrub ok 2024-02-23T00:31:29.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:28.733719+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:29.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:28.775578+0000 osd.2 (osd.2) 71 : cluster [DBG] 2.e scrub starts 2024-02-23T00:31:29.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:28.776782+0000 osd.2 (osd.2) 72 : cluster [DBG] 2.e scrub ok 2024-02-23T00:31:29.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:29.006855+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:29.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:29.012665+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi087-82019-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:29.514 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:29.012785+0000 mon.a (mon.0) 1151 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-11", "mode": "writeback"}]': finished 2024-02-23T00:31:29.514 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:29.012903+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]': finished 2024-02-23T00:31:29.514 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:29.013405+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-02-23T00:31:29.514 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:29.018115+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:29.514 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:29.032515+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:29.599 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:29.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:29.722 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:29.803 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:27.752865+0000 osd.2 (osd.2) 69 : cluster [DBG] 10.5 deep-scrub starts 2024-02-23T00:31:29.803 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:27.755194+0000 osd.2 (osd.2) 70 : cluster [DBG] 10.5 deep-scrub ok 2024-02-23T00:31:29.803 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:28.733719+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:29.803 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:28.775578+0000 osd.2 (osd.2) 71 : cluster [DBG] 2.e scrub starts 2024-02-23T00:31:29.803 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:28.776782+0000 osd.2 (osd.2) 72 : cluster [DBG] 2.e scrub ok 2024-02-23T00:31:29.804 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:29.006855+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:29.804 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:29.012665+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi087-82019-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:29.804 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:29.012785+0000 mon.a (mon.0) 1151 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-11", "mode": "writeback"}]': finished 2024-02-23T00:31:29.804 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:29.012903+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]': finished 2024-02-23T00:31:29.804 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:29.013405+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-02-23T00:31:29.804 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:29.018115+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:29.804 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:29.032515+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:29.817 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:27.752865+0000 osd.2 (osd.2) 69 : cluster [DBG] 10.5 deep-scrub starts 2024-02-23T00:31:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:27.755194+0000 osd.2 (osd.2) 70 : cluster [DBG] 10.5 deep-scrub ok 2024-02-23T00:31:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:28.733719+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:28.775578+0000 osd.2 (osd.2) 71 : cluster [DBG] 2.e scrub starts 2024-02-23T00:31:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:28.776782+0000 osd.2 (osd.2) 72 : cluster [DBG] 2.e scrub ok 2024-02-23T00:31:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:29.006855+0000 mon.a (mon.0) 1149 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:29.012665+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi087-82019-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:29.012785+0000 mon.a (mon.0) 1151 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-11", "mode": "writeback"}]': finished 2024-02-23T00:31:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:29.012903+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi087-82107-9"}]': finished 2024-02-23T00:31:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:29.013405+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-02-23T00:31:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:29.018115+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.87:0/2868093087' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:29.032515+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]: dispatch 2024-02-23T00:31:30.070 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:30.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:30.181 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:30.207 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:30.225 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:30.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:30.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:30.329 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:30.355 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:30.388 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:30.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:30.492 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:30.520 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:30.573 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:30.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:30.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:30.680 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:30.708 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:29.565055+0000 mgr.y (mgr.24437) 192 : cluster [DBG] pgmap v190: 398 pgs: 42 unknown, 1 active+clean+snaptrim, 355 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:29.736875+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:29.916421+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:29.920539+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:30.024069+0000 osd.4 (osd.4) 75 : cluster [DBG] 10.1 deep-scrub starts 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.030435+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]': finished 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:30.030716+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:30.036800+0000 osd.4 (osd.4) 76 : cluster [DBG] 10.1 deep-scrub ok 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.061942+0000 mon.a (mon.0) 1157 : audit [INF] from='client.? 172.21.15.87:0/2784157291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi087-82011-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.095266+0000 mon.c (mon.2) 97 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.182176+0000 mon.b (mon.1) 144 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.359060+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.359819+0000 mon.a (mon.0) 1159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:30.709 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.360470+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.360858+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.361555+0000 mon.c (mon.2) 99 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.361935+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:29.565055+0000 mgr.y (mgr.24437) 192 : cluster [DBG] pgmap v190: 398 pgs: 42 unknown, 1 active+clean+snaptrim, 355 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:29.736875+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:29.916421+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:29.920539+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:30.024069+0000 osd.4 (osd.4) 75 : cluster [DBG] 10.1 deep-scrub starts 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.030435+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]': finished 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:30.030716+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-02-23T00:31:30.710 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:30.036800+0000 osd.4 (osd.4) 76 : cluster [DBG] 10.1 deep-scrub ok 2024-02-23T00:31:30.711 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.061942+0000 mon.a (mon.0) 1157 : audit [INF] from='client.? 172.21.15.87:0/2784157291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi087-82011-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:30.711 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.095266+0000 mon.c (mon.2) 97 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.711 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.182176+0000 mon.b (mon.1) 144 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:30.711 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.359060+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.711 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.359819+0000 mon.a (mon.0) 1159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:30.711 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.360470+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.711 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.360858+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.711 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.361555+0000 mon.c (mon.2) 99 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:30.711 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.361935+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:30.711 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:30.905 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:30.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:30.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:31:30.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:29.565055+0000 mgr.y (mgr.24437) 192 : cluster [DBG] pgmap v190: 398 pgs: 42 unknown, 1 active+clean+snaptrim, 355 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:30.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:29.736875+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:30.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:29.916421+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:31:30.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:29.920539+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:30.024069+0000 osd.4 (osd.4) 75 : cluster [DBG] 10.1 deep-scrub starts 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.030435+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi087-82107-9"}]': finished 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:30.030716+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:30.036800+0000 osd.4 (osd.4) 76 : cluster [DBG] 10.1 deep-scrub ok 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.061942+0000 mon.a (mon.0) 1157 : audit [INF] from='client.? 172.21.15.87:0/2784157291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi087-82011-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.095266+0000 mon.c (mon.2) 97 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.182176+0000 mon.b (mon.1) 144 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.359060+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.359819+0000 mon.a (mon.0) 1159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.360470+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.360858+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.361555+0000 mon.c (mon.2) 99 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.361935+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:31.010 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:31.045 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:31.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-02-23T00:31:31.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] Global test environment set-up. 2024-02-23T00:31:31.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-02-23T00:31:31.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-02-23T00:31:31.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-02-23T00:31:31.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-23T00:31:31.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-02-23T00:31:31.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-02-23T00:31:31.065 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-02-23T00:30:42.183+0000 7f7c2ca3b700 -1 failed for service _ceph-mon._tcp 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-02-23T00:30:42.183+0000 7f7c2ca3b700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (142 ms) 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2011 ms) 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2153 ms total) 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55c071453f70 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 0x55c07145d450 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: started 2 aios 2024-02-23T00:31:31.066 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55c071453f70 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: waiting 0x55c07145d450 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: done. 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (9002 ms) 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (9002 ms total) 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.Exec (2 ms) 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (38 ms) 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (6 ms) 2024-02-23T00:31:31.067 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4995 ms) 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (28905 ms) 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] 8 tests from LibRadosMisc (33946 ms total) 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [----------] Global test environment tear-down 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (49008 ms total) 2024-02-23T00:31:31.068 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc: [ PASSED ] 12 tests. 2024-02-23T00:31:31.136 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:31.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:31.240 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:31.267 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:31.285 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:31.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:31.389 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:31.415 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:31.434 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:31.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:31.538 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:31.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:29.642647+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:31:31.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:29.650595+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:31:31.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:30.371606+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:31.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:30.371679+0000 osd.0 (osd.0) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:31.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:30.499726+0000 mon.a (mon.0) 1162 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:31.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.663619+0000 mgr.y (mgr.24437) 193 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:30.738545+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.033841+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? 172.21.15.87:0/2784157291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi087-82011-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.034111+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.034178+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.034253+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.036804+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.037233+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.038259+0000 mon.a (mon.0) 1167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.040463+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.047093+0000 mon.b (mon.1) 146 : audit [INF] from='client.? 172.21.15.87:0/3099212657' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:31.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.048700+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:31.565 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:31.583 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:31.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:31.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:31.686 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:31.710 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:31:31.710 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:31:31.711 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-23T00:31:31.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-23T00:31:31.714 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:29.642647+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:29.650595+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:30.371606+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:30.371679+0000 osd.0 (osd.0) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:30.499726+0000 mon.a (mon.0) 1162 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.663619+0000 mgr.y (mgr.24437) 193 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:30.738545+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.033841+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? 172.21.15.87:0/2784157291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi087-82011-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.034111+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.034178+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.034253+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.036804+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.037233+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:31.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.038259+0000 mon.a (mon.0) 1167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:31.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.040463+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:31.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.047093+0000 mon.b (mon.1) 146 : audit [INF] from='client.? 172.21.15.87:0/3099212657' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:31.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.048700+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:31.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:29.642647+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:31:31.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:29.650595+0000 osd.0 (osd.0) 90 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:31:31.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:30.371606+0000 osd.0 (osd.0) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:31.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:30.371679+0000 osd.0 (osd.0) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:31.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:30.499726+0000 mon.a (mon.0) 1162 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:31.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.663619+0000 mgr.y (mgr.24437) 193 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:30.738545+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.033841+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? 172.21.15.87:0/2784157291' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi087-82011-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.034111+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.034178+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.034253+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.036804+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.037233+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.038259+0000 mon.a (mon.0) 1167 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.040463+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11"}]: dispatch 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.047093+0000 mon.b (mon.1) 146 : audit [INF] from='client.? 172.21.15.87:0/3099212657' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:31.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.048700+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:31.718 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:31.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 -- 172.21.15.87:0/3763518050 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 msgr2=0x7f6a600f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:31.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3763518050 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 0x7f6a600f6d40 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 rx=0x7f6a54004870 tx=0x7f6a54009cb0).stop 2024-02-23T00:31:31.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 -- 172.21.15.87:0/3763518050 shutdown_connections 2024-02-23T00:31:31.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3763518050 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a600f7400 0x7f6a60103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:31.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3763518050 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a600fcbb0 0x7f6a600fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:31.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3763518050 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 0x7f6a600f6d40 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 -- 172.21.15.87:0/3763518050 >> 172.21.15.87:0/3763518050 conn(0x7f6a600f2820 msgr2=0x7f6a600f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 -- 172.21.15.87:0/3763518050 shutdown_connections 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 -- 172.21.15.87:0/3763518050 wait complete. 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 librados: starting msgr at 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.703+0000 7f6a6529d700 1 librados: starting objecter 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 Processor -- start 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 -- start start 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 librados: setting wanted keys 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 librados: calling monclient init 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a600f7400 0x7f6a6010e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a600fcbb0 0x7f6a6010e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 0x7f6a60108310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:31.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a60108800 con 0x7f6a600f7400 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a60108970 con 0x7f6a600fcbb0 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a6529d700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a60108b30 con 0x7f6a600fd510 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a5f7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 0x7f6a60108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a5f7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 0x7f6a60108310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44970/0 (socket says 172.21.15.87:44970) 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a5f7fe700 1 -- 172.21.15.87:0/3200302980 learned_addr learned my addr 172.21.15.87:0/3200302980 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a5effd700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a600f7400 0x7f6a6010e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a5e7fc700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a600fcbb0 0x7f6a6010e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.704+0000 7f6a5f7fe700 1 -- 172.21.15.87:0/3200302980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a600fcbb0 msgr2=0x7f6a6010e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a5f7fe700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a600fcbb0 0x7f6a6010e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a5f7fe700 1 -- 172.21.15.87:0/3200302980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a600f7400 msgr2=0x7f6a6010e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a5f7fe700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a600f7400 0x7f6a6010e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a5f7fe700 1 -- 172.21.15.87:0/3200302980 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a580097c0 con 0x7f6a600fd510 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a5effd700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a600f7400 0x7f6a6010e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:31:31.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a5f7fe700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 0x7f6a60108310 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7f6a58001df0 tx=0x7f6a58004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a47fff700 10 client.?.objecter ms_handle_connect 0x7f6a600fd510 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a47fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a47fff700 1 -- 172.21.15.87:0/3200302980 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58009a10 con 0x7f6a600fd510 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a54005040 con 0x7f6a600fd510 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a47fff700 1 -- 172.21.15.87:0/3200302980 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a580090d0 con 0x7f6a600fd510 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a47fff700 1 -- 172.21.15.87:0/3200302980 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58007ca0 con 0x7f6a600fd510 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.705+0000 7f6a6529d700 10 client.36695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.706+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a60114fa0 con 0x7f6a600fd510 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.706+0000 7f6a6529d700 1 librados: init done 2024-02-23T00:31:31.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.706+0000 7f6a47fff700 1 -- 172.21.15.87:0/3200302980 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f6a58009240 con 0x7f6a600fd510 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.706+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a601091c0 con 0x7f6a600fd510 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.707+0000 7f6a47fff700 1 --2- 172.21.15.87:0/3200302980 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a48062c90 0x7f6a48065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.707+0000 7f6a47fff700 1 -- 172.21.15.87:0/3200302980 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(140..140 src has 38..140) v4 ==== 9887+0+0 (secure 0 0 0) 0x7f6a5807e250 con 0x7f6a600fd510 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.707+0000 7f6a47fff700 10 client.36695.objecter ms_dispatch 0x7f6a60001550 osd_map(140..140 src has 38..140) v4 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.707+0000 7f6a47fff700 3 client.36695.objecter handle_osd_map got epochs [140,140] > 0 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.707+0000 7f6a47fff700 3 client.36695.objecter handle_osd_map decoding full epoch 140 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.707+0000 7f6a5effd700 1 --2- 172.21.15.87:0/3200302980 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a48062c90 0x7f6a48065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.707+0000 7f6a47fff700 20 client.36695.objecter dump_active .. 0 homeless 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.707+0000 7f6a47fff700 10 client.36695.objecter ms_handle_connect 0x7f6a48062c90 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.707+0000 7f6a5effd700 1 --2- 172.21.15.87:0/3200302980 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a48062c90 0x7f6a48065150 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 rx=0x7f6a500103c0 tx=0x7f6a5000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:31.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.712+0000 7f6a47fff700 1 -- 172.21.15.87:0/3200302980 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6a58050fa0 con 0x7f6a600fd510 2024-02-23T00:31:31.743 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:31.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:31.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:31.874 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:31.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:31.911+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f6a60106120 con 0x7f6a600fd510 2024-02-23T00:31:32.077 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:31:32.077 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:31:32.077 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:31:32.077 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:31:32.078 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:32.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:32.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:32.094 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:32.182 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:32.209 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:32.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.225+0000 7f6a47fff700 1 -- 172.21.15.87:0/3200302980 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v141) v1 ==== 101+0+0 (secure 0 0 0) 0x7f6a58050b90 con 0x7f6a600fd510 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.232+0000 7f6a6529d700 10 librados: watch_flush enter 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 10 librados: watch_flush exit 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 20 client.36695.objecter shutdown clearing up homeless session... 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 10 client.36695.objecter successfully canceled tick 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a48062c90 msgr2=0x7f6a48065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3200302980 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a48062c90 0x7f6a48065150 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 rx=0x7f6a500103c0 tx=0x7f6a5000b040).stop 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 msgr2=0x7f6a60108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 0x7f6a60108310 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7f6a58001df0 tx=0x7f6a58004c50).stop 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 shutdown_connections 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3200302980 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a48062c90 0x7f6a48065150 secure :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 rx=0x7f6a500103c0 tx=0x7f6a5000b040).stop 2024-02-23T00:31:32.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a600fd510 0x7f6a60108310 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:32.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a600f7400 0x7f6a6010e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:32.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 --2- 172.21.15.87:0/3200302980 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a600fcbb0 0x7f6a6010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:32.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.233+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 >> 172.21.15.87:0/3200302980 conn(0x7f6a600f2820 msgr2=0x7f6a600fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:32.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.251+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 shutdown_connections 2024-02-23T00:31:32.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.251+0000 7f6a6529d700 1 -- 172.21.15.87:0/3200302980 wait complete. 2024-02-23T00:31:32.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:32.251+0000 7f6a6529d700 1 librados: shutdown 2024-02-23T00:31:32.252 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:31:32.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:32] "GET /metrics HTTP/1.1" 200 203036 "" "Prometheus/2.33.4" 2024-02-23T00:31:32.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:31:32.413 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:32.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:32.518 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:32.546 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:32.759 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.322123+0000 osd.5 (osd.5) 75 : cluster [DBG] 6.a scrub starts 2024-02-23T00:31:32.759 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.322936+0000 osd.5 (osd.5) 76 : cluster [DBG] 6.a scrub ok 2024-02-23T00:31:32.759 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.739277+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:32.759 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.912526+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.87:0/3200302980' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:31.912900+0000 mon.a (mon.0) 1170 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.945545+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.946362+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.966879+0000 mon.a (mon.0) 1171 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.966903+0000 mon.a (mon.0) 1172 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:32.223443+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11"}]': finished 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:32.223614+0000 mon.a (mon.0) 1174 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:32.223791+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:32.223849+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.322123+0000 osd.5 (osd.5) 75 : cluster [DBG] 6.a scrub starts 2024-02-23T00:31:32.760 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.322936+0000 osd.5 (osd.5) 76 : cluster [DBG] 6.a scrub ok 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.739277+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.912526+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.87:0/3200302980' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:31.912900+0000 mon.a (mon.0) 1170 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.945545+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.946362+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.966879+0000 mon.a (mon.0) 1171 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.966903+0000 mon.a (mon.0) 1172 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:32.223443+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11"}]': finished 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:32.223614+0000 mon.a (mon.0) 1174 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:32.223791+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:31:32.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:32.223849+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-02-23T00:31:32.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.322123+0000 osd.5 (osd.5) 75 : cluster [DBG] 6.a scrub starts 2024-02-23T00:31:32.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.322936+0000 osd.5 (osd.5) 76 : cluster [DBG] 6.a scrub ok 2024-02-23T00:31:32.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.739277+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:32.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.912526+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.87:0/3200302980' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:32.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:31.912900+0000 mon.a (mon.0) 1170 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:32.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.945545+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:31:32.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.946362+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:31:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.966879+0000 mon.a (mon.0) 1171 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:31:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.966903+0000 mon.a (mon.0) 1172 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:31:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:32.223443+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-11"}]': finished 2024-02-23T00:31:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:32.223614+0000 mon.a (mon.0) 1174 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:32.223791+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:31:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:32.223849+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-02-23T00:31:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.565567+0000 mgr.y (mgr.24437) 194 : cluster [DBG] pgmap v193: 398 pgs: 32 creating+peering, 32 unknown, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 319 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 97 B/s, 0 objects/s recovering 2024-02-23T00:31:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.804331+0000 osd.2 (osd.2) 73 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:31:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:31.806679+0000 osd.2 (osd.2) 74 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:31:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:32.030237+0000 osd.6 (osd.6) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:32.030300+0000 osd.6 (osd.6) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:32.740422+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:32.871723+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:31:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:32.874674+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:31:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:32.973736+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]': finished 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:32.973914+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:32.980470+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.87:0/1307520964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi087-82011-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:32.981923+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi087-82011-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.565567+0000 mgr.y (mgr.24437) 194 : cluster [DBG] pgmap v193: 398 pgs: 32 creating+peering, 32 unknown, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 319 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 97 B/s, 0 objects/s recovering 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.804331+0000 osd.2 (osd.2) 73 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:31.806679+0000 osd.2 (osd.2) 74 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:32.030237+0000 osd.6 (osd.6) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:32.030300+0000 osd.6 (osd.6) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:32.740422+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:32.871723+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:31:33.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:32.874674+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:31:33.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:32.973736+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]': finished 2024-02-23T00:31:33.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:32.973914+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-02-23T00:31:33.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:32.980470+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.87:0/1307520964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi087-82011-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:33.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:32.981923+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi087-82011-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:33.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:31:33.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.565567+0000 mgr.y (mgr.24437) 194 : cluster [DBG] pgmap v193: 398 pgs: 32 creating+peering, 32 unknown, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 319 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 97 B/s, 0 objects/s recovering 2024-02-23T00:31:33.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.804331+0000 osd.2 (osd.2) 73 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:31:33.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:31.806679+0000 osd.2 (osd.2) 74 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:31:33.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:32.030237+0000 osd.6 (osd.6) 63 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:31:33.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:32.030300+0000 osd.6 (osd.6) 64 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:31:33.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:32.740422+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:32.871723+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:31:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:32.874674+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:31:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:32.973736+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]': finished 2024-02-23T00:31:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:32.973914+0000 mon.a (mon.0) 1178 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-02-23T00:31:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:32.980470+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.87:0/1307520964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi087-82011-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:32.981923+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi087-82011-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:33.380398+0000 osd.5 (osd.5) 77 : cluster [DBG] 5.f scrub starts 2024-02-23T00:31:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:33.381224+0000 osd.5 (osd.5) 78 : cluster [DBG] 5.f scrub ok 2024-02-23T00:31:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:33.566090+0000 mgr.y (mgr.24437) 195 : cluster [DBG] pgmap v196: 342 pgs: 40 unknown, 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 293 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 2 op/s; 97 B/s, 0 objects/s recovering 2024-02-23T00:31:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:33.741226+0000 mon.c (mon.2) 106 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:33.978012+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi087-82011-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:33.978236+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-02-23T00:31:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:33.986550+0000 mon.b (mon.1) 147 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:33.987298+0000 mon.a (mon.0) 1182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:33.989618+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.87:0/1873510578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi087-82019-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:33.999674+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi087-82019-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:33.380398+0000 osd.5 (osd.5) 77 : cluster [DBG] 5.f scrub starts 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:33.381224+0000 osd.5 (osd.5) 78 : cluster [DBG] 5.f scrub ok 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:33.566090+0000 mgr.y (mgr.24437) 195 : cluster [DBG] pgmap v196: 342 pgs: 40 unknown, 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 293 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 2 op/s; 97 B/s, 0 objects/s recovering 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:33.741226+0000 mon.c (mon.2) 106 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:33.978012+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi087-82011-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:33.978236+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:33.986550+0000 mon.b (mon.1) 147 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:33.987298+0000 mon.a (mon.0) 1182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:33.989618+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.87:0/1873510578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi087-82019-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:33.999674+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi087-82019-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:33.380398+0000 osd.5 (osd.5) 77 : cluster [DBG] 5.f scrub starts 2024-02-23T00:31:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:33.381224+0000 osd.5 (osd.5) 78 : cluster [DBG] 5.f scrub ok 2024-02-23T00:31:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:33.566090+0000 mgr.y (mgr.24437) 195 : cluster [DBG] pgmap v196: 342 pgs: 40 unknown, 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 293 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 2 op/s; 97 B/s, 0 objects/s recovering 2024-02-23T00:31:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:33.741226+0000 mon.c (mon.2) 106 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:33.978012+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi087-82011-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:33.978236+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-02-23T00:31:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:33.986550+0000 mon.b (mon.1) 147 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:33.987298+0000 mon.a (mon.0) 1182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:33.989618+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.87:0/1873510578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi087-82019-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:33.999674+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi087-82019-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:35.269 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:31:35.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:34.015359+0000 osd.6 (osd.6) 65 : cluster [DBG] 10.7 deep-scrub starts 2024-02-23T00:31:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:34.018246+0000 osd.6 (osd.6) 66 : cluster [DBG] 10.7 deep-scrub ok 2024-02-23T00:31:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:34.741994+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:34.978619+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:34.978781+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi087-82019-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:34.978932+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-02-23T00:31:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:34.015359+0000 osd.6 (osd.6) 65 : cluster [DBG] 10.7 deep-scrub starts 2024-02-23T00:31:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:34.018246+0000 osd.6 (osd.6) 66 : cluster [DBG] 10.7 deep-scrub ok 2024-02-23T00:31:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:34.741994+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:34.978619+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:34.978781+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi087-82019-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:34.978932+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-02-23T00:31:35.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:34.015359+0000 osd.6 (osd.6) 65 : cluster [DBG] 10.7 deep-scrub starts 2024-02-23T00:31:35.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:34.018246+0000 osd.6 (osd.6) 66 : cluster [DBG] 10.7 deep-scrub ok 2024-02-23T00:31:35.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:34.741994+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:35.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:34.978619+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:35.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:34.978781+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi087-82019-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:35.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:34.978932+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-02-23T00:31:36.087 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-23T00:31:36.103 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:36.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:36.211 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:36.238 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:36.256 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:36.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:36.360 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:36.387 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:36.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:36.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:36.509 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:36.537 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:35.419527+0000 osd.5 (osd.5) 79 : cluster [DBG] 4.15 deep-scrub starts 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:35.423222+0000 osd.5 (osd.5) 80 : cluster [DBG] 4.15 deep-scrub ok 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:35.567265+0000 mgr.y (mgr.24437) 196 : cluster [DBG] pgmap v199: 374 pgs: 374 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s wr, 25 op/s 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:35.640659+0000 mon.a (mon.0) 1187 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:35.742838+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:35.935076+0000 osd.1 (osd.1) 87 : cluster [DBG] 2.a deep-scrub starts 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:35.937964+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.a deep-scrub ok 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:35.981117+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:35.997287+0000 mon.a (mon.0) 1189 : audit [INF] from='client.? 172.21.15.87:0/2703256888' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi087-82011-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.278797+0000 mon.a (mon.0) 1190 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:31:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.279675+0000 mon.a (mon.0) 1191 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:31:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.284713+0000 mon.a (mon.0) 1192 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:35.419527+0000 osd.5 (osd.5) 79 : cluster [DBG] 4.15 deep-scrub starts 2024-02-23T00:31:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:35.423222+0000 osd.5 (osd.5) 80 : cluster [DBG] 4.15 deep-scrub ok 2024-02-23T00:31:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:35.567265+0000 mgr.y (mgr.24437) 196 : cluster [DBG] pgmap v199: 374 pgs: 374 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s wr, 25 op/s 2024-02-23T00:31:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:35.640659+0000 mon.a (mon.0) 1187 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:31:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:35.742838+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:35.935076+0000 osd.1 (osd.1) 87 : cluster [DBG] 2.a deep-scrub starts 2024-02-23T00:31:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:35.937964+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.a deep-scrub ok 2024-02-23T00:31:36.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:35.981117+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-02-23T00:31:36.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:35.997287+0000 mon.a (mon.0) 1189 : audit [INF] from='client.? 172.21.15.87:0/2703256888' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi087-82011-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:36.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.278797+0000 mon.a (mon.0) 1190 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:31:36.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.279675+0000 mon.a (mon.0) 1191 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:31:36.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.284713+0000 mon.a (mon.0) 1192 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:36.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:35.419527+0000 osd.5 (osd.5) 79 : cluster [DBG] 4.15 deep-scrub starts 2024-02-23T00:31:36.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:35.423222+0000 osd.5 (osd.5) 80 : cluster [DBG] 4.15 deep-scrub ok 2024-02-23T00:31:36.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:35.567265+0000 mgr.y (mgr.24437) 196 : cluster [DBG] pgmap v199: 374 pgs: 374 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s wr, 25 op/s 2024-02-23T00:31:36.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:35.640659+0000 mon.a (mon.0) 1187 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:31:36.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:35.742838+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:36.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:35.935076+0000 osd.1 (osd.1) 87 : cluster [DBG] 2.a deep-scrub starts 2024-02-23T00:31:36.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:35.937964+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.a deep-scrub ok 2024-02-23T00:31:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:35.981117+0000 mon.a (mon.0) 1188 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-02-23T00:31:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:35.997287+0000 mon.a (mon.0) 1189 : audit [INF] from='client.? 172.21.15.87:0/2703256888' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi087-82011-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.278797+0000 mon.a (mon.0) 1190 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:31:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.279675+0000 mon.a (mon.0) 1191 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:31:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.284713+0000 mon.a (mon.0) 1192 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:37.312 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:35.702359+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:35.705236+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:36.523040+0000 mon.a (mon.0) 1193 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.743579+0000 mon.c (mon.2) 110 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.744186+0000 mon.c (mon.2) 111 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.744504+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.974294+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? 172.21.15.87:0/2703256888' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi087-82011-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.974413+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]': finished 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:36.974479+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.980961+0000 mon.c (mon.2) 112 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:37.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.982593+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:37.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.983416+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:31:37.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:36.984448+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? 172.21.15.87:0/2170208941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi087-82019-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:37.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:37.001758+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:31:37.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:37.064979+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a deep-scrub starts 2024-02-23T00:31:37.789 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:37.065858+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a deep-scrub ok 2024-02-23T00:31:37.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:35.702359+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:31:37.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:35.705236+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:31:37.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:36.523040+0000 mon.a (mon.0) 1193 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:37.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.743579+0000 mon.c (mon.2) 110 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:37.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.744186+0000 mon.c (mon.2) 111 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:31:37.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.744504+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:31:37.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.974294+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? 172.21.15.87:0/2703256888' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi087-82011-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:37.789 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.974413+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]': finished 2024-02-23T00:31:37.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:36.974479+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-02-23T00:31:37.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.980961+0000 mon.c (mon.2) 112 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:37.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.982593+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:37.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.983416+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:31:37.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:36.984448+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? 172.21.15.87:0/2170208941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi087-82019-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:37.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:37.001758+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:31:37.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:37.064979+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a deep-scrub starts 2024-02-23T00:31:37.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:37.065858+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a deep-scrub ok 2024-02-23T00:31:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:37.791 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:37.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:37.892 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:37.919 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:35.702359+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:35.705236+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:36.523040+0000 mon.a (mon.0) 1193 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.743579+0000 mon.c (mon.2) 110 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.744186+0000 mon.c (mon.2) 111 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.744504+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.974294+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? 172.21.15.87:0/2703256888' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi087-82011-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.974413+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]': finished 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:36.974479+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-02-23T00:31:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.980961+0000 mon.c (mon.2) 112 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.982593+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.983416+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:31:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:36.984448+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? 172.21.15.87:0/2170208941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi087-82019-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:37.001758+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:31:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:37.064979+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a deep-scrub starts 2024-02-23T00:31:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:37.065858+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a deep-scrub ok 2024-02-23T00:31:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:38.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:38.221 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:38.250 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:38.352 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:38.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 -- 172.21.15.87:0/778536051 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fe510 msgr2=0x7f314c0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:38.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 --2- 172.21.15.87:0/778536051 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fe510 0x7f314c0f7d40 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 rx=0x7f3140004870 tx=0x7f3140009cb0).stop 2024-02-23T00:31:38.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 -- 172.21.15.87:0/778536051 shutdown_connections 2024-02-23T00:31:38.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 --2- 172.21.15.87:0/778536051 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f314c0f8420 0x7f314c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:38.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 --2- 172.21.15.87:0/778536051 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f314c0fdbb0 0x7f314c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:38.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 --2- 172.21.15.87:0/778536051 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fe510 0x7f314c0f7d40 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:38.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 -- 172.21.15.87:0/778536051 >> 172.21.15.87:0/778536051 conn(0x7f314c0f3800 msgr2=0x7f314c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:38.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 -- 172.21.15.87:0/778536051 shutdown_connections 2024-02-23T00:31:38.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 -- 172.21.15.87:0/778536051 wait complete. 2024-02-23T00:31:38.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 librados: starting msgr at 2024-02-23T00:31:38.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.351+0000 7f315191c700 1 librados: starting objecter 2024-02-23T00:31:38.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 Processor -- start 2024-02-23T00:31:38.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 -- start start 2024-02-23T00:31:38.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 librados: setting wanted keys 2024-02-23T00:31:38.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 librados: calling monclient init 2024-02-23T00:31:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:38.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f314c0f8420 0x7f314c10e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fdbb0 0x7f314c10e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f314c0fe510 0x7f314c108260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c108750 con 0x7f314c0fe510 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c1088c0 con 0x7f314c0f8420 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f315191c700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c108a80 con 0x7f314c0fdbb0 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f314a7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fdbb0 0x7f314c10e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f314affd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f314c0f8420 0x7f314c10e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f314a7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fdbb0 0x7f314c10e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:45064/0 (socket says 172.21.15.87:45064) 2024-02-23T00:31:38.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f314a7fc700 1 -- 172.21.15.87:0/3842658524 learned_addr learned my addr 172.21.15.87:0/3842658524 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.352+0000 7f314b7fe700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f314c0fe510 0x7f314c108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f314affd700 1 -- 172.21.15.87:0/3842658524 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fdbb0 msgr2=0x7f314c10e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f314affd700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fdbb0 0x7f314c10e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f314affd700 1 -- 172.21.15.87:0/3842658524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f314c0fe510 msgr2=0x7f314c108260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f314affd700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f314c0fe510 0x7f314c108260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f314affd700 1 -- 172.21.15.87:0/3842658524 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c005040 con 0x7f314c0f8420 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f314a7fc700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fdbb0 0x7f314c10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f314b7fe700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f314c0fe510 0x7f314c108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f314affd700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f314c0f8420 0x7f314c10e0c0 secure :-1 s=READY pgs=4962 cs=0 l=1 rev1=1 rx=0x7f313c004920 tx=0x7f313c009cb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f3133fff700 10 client.?.objecter ms_handle_connect 0x7f314c0f8420 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f3133fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f3133fff700 1 -- 172.21.15.87:0/3842658524 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c00e030 con 0x7f314c0f8420 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3140005040 con 0x7f314c0f8420 2024-02-23T00:31:38.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f315191c700 10 client.44212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c114ef0 con 0x7f314c0f8420 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f315191c700 1 librados: init done 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f3133fff700 1 -- 172.21.15.87:0/3842658524 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f313c00f040 con 0x7f314c0f8420 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.353+0000 7f3133fff700 1 -- 172.21.15.87:0/3842658524 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c0147f0 con 0x7f314c0f8420 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.354+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3138004f50 con 0x7f314c0f8420 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f3133fff700 1 -- 172.21.15.87:0/3842658524 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f313c014c20 con 0x7f314c0f8420 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f3133fff700 1 --2- 172.21.15.87:0/3842658524 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3134062c90 0x7f3134065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f3133fff700 1 -- 172.21.15.87:0/3842658524 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(147..147 src has 38..147) v4 ==== 10041+0+0 (secure 0 0 0) 0x7f313c07da10 con 0x7f314c0f8420 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f3133fff700 10 client.44212.objecter ms_dispatch 0x7f314c001550 osd_map(147..147 src has 38..147) v4 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f3133fff700 3 client.44212.objecter handle_osd_map got epochs [147,147] > 0 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f3133fff700 3 client.44212.objecter handle_osd_map decoding full epoch 147 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f314a7fc700 1 --2- 172.21.15.87:0/3842658524 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3134062c90 0x7f3134065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f3133fff700 20 client.44212.objecter dump_active .. 0 homeless 2024-02-23T00:31:38.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f3133fff700 10 client.44212.objecter ms_handle_connect 0x7f3134062c90 2024-02-23T00:31:38.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.355+0000 7f314a7fc700 1 --2- 172.21.15.87:0/3842658524 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3134062c90 0x7f3134065150 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f3140009a00 tx=0x7f314000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:38.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.359+0000 7f3133fff700 1 -- 172.21.15.87:0/3842658524 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f313c050720 con 0x7f314c0f8420 2024-02-23T00:31:38.457 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:38.485 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:38.503 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:38.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:38.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.525+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f3138005200 con 0x7f314c0f8420 2024-02-23T00:31:38.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.527+0000 7f3133fff700 1 -- 172.21.15.87:0/3842658524 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v147) v1 ==== 81+0+59009 (secure 0 0 0) 0x7f313c050310 con 0x7f314c0f8420 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 147, 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "created": "2024-02-23T00:21:28.861318+0000", 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "modified": "2024-02-23T00:31:37.970422+0000", 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "last_up_change": "2024-02-23T00:27:09.633391+0000", 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "last_in_change": "2024-02-23T00:26:52.603467+0000", 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_num": 5799936, 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_set": [ 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "pglog_hardlimit", 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "purged_snapdirs", 2024-02-23T00:31:38.531 INFO:teuthology.orchestra.run.smithi087.stdout: "recovery_deletes", 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "sortbitwise" 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_version": 30, 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "full_ratio": 0.94999998807907104, 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_snapshot": "", 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_max": 98, 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd": 8, 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "require_min_compat_client": "luminous", 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "min_compat_client": "luminous", 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "require_osd_release": "pacific", 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: "pools": [ 2024-02-23T00:31:38.532 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "device_health_metrics", 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:25:09.208309+0000", 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.533 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 1, 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 1, 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 1, 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 1, 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 1, 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "68", 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.534 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.535 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.536 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_max": 32, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_min": 1 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_devicehealth": {} 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 2, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": ".rgw.root", 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.562186+0000", 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.538 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "56", 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.539 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.540 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.541 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 3, 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "iscsi", 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.989206+0000", 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.542 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 13, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 13, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 13, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 13, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 13, 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.543 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "137", 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "135", 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.544 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.545 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.546 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd": {} 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 4, 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.log", 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:17.282440+0000", 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.547 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.548 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "58", 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.549 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.550 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.551 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 5, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.control", 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:19.262164+0000", 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.552 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "60", 2024-02-23T00:31:38.553 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.554 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.555 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 6, 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.meta", 2024-02-23T00:31:38.556 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:21.319951+0000", 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.557 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "62", 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.558 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.559 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.560 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_bias": 4 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 10, 2024-02-23T00:31:38.561 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "LibRadosMiscPP_smithi087-82083-1", 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:43.121428+0000", 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.562 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.563 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.564 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.565 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 19, 2024-02-23T00:31:38.566 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "LibRadosList_smithi087-82051-1", 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:45.322617+0000", 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.567 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 11, 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 11, 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 11, 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "147", 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.568 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.569 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.570 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 22, 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-23T00:31:38.571 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:46.071806+0000", 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.572 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.573 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.574 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.575 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-6", 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:31:04.375891+0000", 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:31:38.576 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.577 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "147", 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "140", 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "140", 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "140", 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.578 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 6, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 147, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.579 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.580 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 67, 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "LibRadosSnapshotsSelfManagedPP_smithi087-82115-7", 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:31:07.424556+0000", 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:31:38.581 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.582 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "142", 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.583 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 24, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 142, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.584 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.585 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:31:32.223322+0000", 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 3, 2024-02-23T00:31:38.586 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 1, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 8, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 8, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 8, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 8, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 8, 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.587 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "147", 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 6, 2024-02-23T00:31:38.588 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 147, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.589 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14", 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.590 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 8192, 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 95, 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-13", 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:31:33.242792+0000", 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.591 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.592 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "145", 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.593 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.594 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.595 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 98, 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "IsCompletePP_smithi087-82019-11", 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:31:36.278481+0000", 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 32769, 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,creating", 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:31:38.596 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:31:38.597 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "147", 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:31:38.598 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:31:38.599 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:31:38.600 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7", 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:31:38.601 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 8, 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 146, 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:31:38.602 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6802", 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6804", 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.603 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6803", 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6801/2153191863", 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6802/2153191863", 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6804/2153191863", 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6803/2153191863", 2024-02-23T00:31:38.604 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "85525ce8-38d5-4840-8057-547cb176c859", 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 13, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 146, 2024-02-23T00:31:38.605 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6805", 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6806", 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:31:38.606 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6808", 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.607 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6807", 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6805/685354171", 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6806/685354171", 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6808/685354171", 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6807/685354171", 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.608 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa", 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 18, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 146, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.609 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6809", 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6810", 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:31:38.610 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6812", 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6811", 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.611 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6809/3605394228", 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6810/3605394228", 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6812/3605394228", 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6811/3605394228", 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a", 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:31:38.612 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 26, 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 146, 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6813", 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:31:38.613 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6814", 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.614 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6816", 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6815", 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.615 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6813/128059790", 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6814/128059790", 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6816/128059790", 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6815/128059790", 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c", 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:31:38.616 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 31, 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 146, 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6800", 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.617 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6801", 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6803", 2024-02-23T00:31:38.618 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6802", 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6800/4175844757", 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6801/4175844757", 2024-02-23T00:31:38.619 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4175844757", 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4175844757", 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d", 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:31:38.620 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 38, 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 146, 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6804", 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.621 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6805", 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6807", 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:31:38.622 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6806", 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6804/2550013795", 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6805/2550013795", 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2550013795", 2024-02-23T00:31:38.623 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2550013795", 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01", 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:31:38.624 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 44, 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 146, 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6808", 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:31:38.625 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6809", 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6811", 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:31:38.626 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6810", 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6808/2509408498", 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6809/2509408498", 2024-02-23T00:31:38.627 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6811/2509408498", 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6810/2509408498", 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939", 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:31:38.628 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 51, 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 146, 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6812", 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.629 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6813", 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6815", 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:31:38.630 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6814", 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6812/373787000", 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6813/373787000", 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6815/373787000", 2024-02-23T00:31:38.631 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6814/373787000", 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_xinfo": [ 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:31:38.632 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:31:30.371681+0000", 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:30:42.979539+0000", 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:31:38.633 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:31:06.316853+0000", 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:31:38.634 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:30:40.497481+0000", 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:31:38.635 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:31:11.892843+0000", 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:31:11.261861+0000", 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.636 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:31:32.030302+0000", 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:31:38.637 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:30:59.983249+0000", 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap": [], 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap_items": [ 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.a", 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.638 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.9", 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.13", 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:31:38.639 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.17", 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 5 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_temp": [], 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_temp": [], 2024-02-23T00:31:38.640 INFO:teuthology.orchestra.run.smithi087.stdout: "blocklist": { 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1121018082": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/2138166229": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/27794119": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/1675126043": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/3767016525": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/969918521": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/795095363": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2273151327": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1685406414": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2093733106": "2024-02-24T00:21:58.144574+0000" 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "range_blocklist": {}, 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profiles": { 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "default": { 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:31:38.641 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van" 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14": { 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-device-class": "", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-root": "default", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van", 2024-02-23T00:31:38.642 INFO:teuthology.orchestra.run.smithi087.stdout: "w": "8" 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps_queue": [ 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 7, 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.643 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 21, 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:31:38.644 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 61, 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:31:38.645 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 78, 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.646 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 86, 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:31:38.647 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: "new_removed_snaps": [], 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: "new_purged_snaps": [], 2024-02-23T00:31:38.648 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_node_flags": {}, 2024-02-23T00:31:38.649 INFO:teuthology.orchestra.run.smithi087.stdout: "device_class_flags": {}, 2024-02-23T00:31:38.649 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode": { 2024-02-23T00:31:38.649 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_enabled": false, 2024-02-23T00:31:38.649 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_bucket_count": 0, 2024-02-23T00:31:38.649 INFO:teuthology.orchestra.run.smithi087.stdout: "degraded_stretch_mode": 0, 2024-02-23T00:31:38.649 INFO:teuthology.orchestra.run.smithi087.stdout: "recovering_stretch_mode": 0, 2024-02-23T00:31:38.649 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_bucket": 0 2024-02-23T00:31:38.649 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:38.649 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:31:38.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 10 librados: watch_flush enter 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 10 librados: watch_flush exit 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 20 client.44212.objecter shutdown clearing up homeless session... 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 10 client.44212.objecter successfully canceled tick 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3134062c90 msgr2=0x7f3134065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 --2- 172.21.15.87:0/3842658524 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3134062c90 0x7f3134065150 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 rx=0x7f3140009a00 tx=0x7f314000b040).stop 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f314c0f8420 msgr2=0x7f314c10e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f314c0f8420 0x7f314c10e0c0 secure :-1 s=READY pgs=4962 cs=0 l=1 rev1=1 rx=0x7f313c004920 tx=0x7f313c009cb0).stop 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 shutdown_connections 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 --2- 172.21.15.87:0/3842658524 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3134062c90 0x7f3134065150 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:38.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f314c0fe510 0x7f314c108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:38.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f314c0f8420 0x7f314c10e0c0 unknown :-1 s=CLOSED pgs=4962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:38.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 --2- 172.21.15.87:0/3842658524 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f314c0fdbb0 0x7f314c10e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:38.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.529+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 >> 172.21.15.87:0/3842658524 conn(0x7f314c0f3800 msgr2=0x7f314c0f5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:38.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.530+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 shutdown_connections 2024-02-23T00:31:38.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.530+0000 7f315191c700 1 -- 172.21.15.87:0/3842658524 wait complete. 2024-02-23T00:31:38.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:38.530+0000 7f315191c700 1 librados: shutdown 2024-02-23T00:31:38.652 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:38.652 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:37.567903+0000 mgr.y (mgr.24437) 197 : cluster [DBG] pgmap v202: 406 pgs: 64 unknown, 342 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 29 KiB/s wr, 27 op/s 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:37.568957+0000 mon.a (mon.0) 1200 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:37.978053+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? 172.21.15.87:0/2170208941' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi087-82019-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:37.978176+0000 mon.a (mon.0) 1202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]': finished 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:37.978294+0000 mon.a (mon.0) 1203 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:37.978336+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:37.988155+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:38.036169+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:38.038403+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:38.527853+0000 mon.b (mon.1) 148 : audit [DBG] from='client.? 172.21.15.87:0/3842658524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:31:38.653 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:37.567903+0000 mgr.y (mgr.24437) 197 : cluster [DBG] pgmap v202: 406 pgs: 64 unknown, 342 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 29 KiB/s wr, 27 op/s 2024-02-23T00:31:38.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:37.568957+0000 mon.a (mon.0) 1200 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-23T00:31:38.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:37.978053+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? 172.21.15.87:0/2170208941' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi087-82019-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:38.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:37.978176+0000 mon.a (mon.0) 1202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]': finished 2024-02-23T00:31:38.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:37.978294+0000 mon.a (mon.0) 1203 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-23T00:31:38.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:37.978336+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-02-23T00:31:38.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:37.988155+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:38.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:38.036169+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:31:38.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:38.038403+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:31:38.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:38.527853+0000 mon.b (mon.1) 148 : audit [DBG] from='client.? 172.21.15.87:0/3842658524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:31:38.682 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:38.780 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:38.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:38.885 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:38.912 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:37.567903+0000 mgr.y (mgr.24437) 197 : cluster [DBG] pgmap v202: 406 pgs: 64 unknown, 342 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 29 KiB/s wr, 27 op/s 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:37.568957+0000 mon.a (mon.0) 1200 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:37.978053+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? 172.21.15.87:0/2170208941' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi087-82019-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:37.978176+0000 mon.a (mon.0) 1202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]': finished 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:37.978294+0000 mon.a (mon.0) 1203 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:37.978336+0000 mon.a (mon.0) 1204 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:37.988155+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:38.036169+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:38.038403+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:31:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:38.527853+0000 mon.b (mon.1) 148 : audit [DBG] from='client.? 172.21.15.87:0/3842658524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:31:40.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:38.650166+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:31:40.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:38.653074+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:31:40.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:38.834582+0000 osd.7 (osd.7) 81 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:31:40.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:38.838197+0000 osd.7 (osd.7) 82 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:31:40.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:38.884705+0000 osd.1 (osd.1) 89 : cluster [DBG] 5.1d deep-scrub starts 2024-02-23T00:31:40.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:38.887593+0000 osd.1 (osd.1) 90 : cluster [DBG] 5.1d deep-scrub ok 2024-02-23T00:31:40.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:38.976998+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-02-23T00:31:40.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:38.991509+0000 mon.c (mon.2) 116 : audit [INF] from='client.? 172.21.15.87:0/2157738800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi087-82011-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.007740+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi087-82011-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.145208+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.276478+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.383654+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.637701+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:38.650166+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:38.653074+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:38.834582+0000 osd.7 (osd.7) 81 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:38.838197+0000 osd.7 (osd.7) 82 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:38.884705+0000 osd.1 (osd.1) 89 : cluster [DBG] 5.1d deep-scrub starts 2024-02-23T00:31:40.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:38.887593+0000 osd.1 (osd.1) 90 : cluster [DBG] 5.1d deep-scrub ok 2024-02-23T00:31:40.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:38.976998+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-02-23T00:31:40.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:38.991509+0000 mon.c (mon.2) 116 : audit [INF] from='client.? 172.21.15.87:0/2157738800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi087-82011-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:40.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.007740+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi087-82011-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:40.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.145208+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:40.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.276478+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:40.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.383654+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:40.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.637701+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:38.650166+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:31:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:38.653074+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:31:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:38.834582+0000 osd.7 (osd.7) 81 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:31:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:38.838197+0000 osd.7 (osd.7) 82 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:31:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:38.884705+0000 osd.1 (osd.1) 89 : cluster [DBG] 5.1d deep-scrub starts 2024-02-23T00:31:40.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:38.887593+0000 osd.1 (osd.1) 90 : cluster [DBG] 5.1d deep-scrub ok 2024-02-23T00:31:40.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:38.976998+0000 mon.a (mon.0) 1205 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-02-23T00:31:40.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:38.991509+0000 mon.c (mon.2) 116 : audit [INF] from='client.? 172.21.15.87:0/2157738800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi087-82011-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:40.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.007740+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi087-82011-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:40.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.145208+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:40.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.276478+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:31:40.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.383654+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:40.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.637701+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:40.944 INFO:tasks.thrashosds.thrasher:j is {'epoch': 147, 'fsid': '66db207c-d1e1-11ee-95c0-87774f69a715', 'created': '2024-02-23T00:21:28.861318+0000', 'modified': '2024-02-23T00:31:37.970422+0000', 'last_up_change': '2024-02-23T00:27:09.633391+0000', 'last_in_change': '2024-02-23T00:26:52.603467+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 30, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 98, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '135', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 10, 'pool_name': 'LibRadosMiscPP_smithi087-82083-1', 'create_time': '2024-02-23T00:30:43.121428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 19, 'pool_name': 'LibRadosList_smithi087-82051-1', 'create_time': '2024-02-23T00:30:45.322617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 22, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-23T00:30:46.071806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 63, 'pool_name': 'test-rados-api-smithi087-82091-6', 'create_time': '2024-02-23T00:31:04.375891+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '140', 'last_force_op_resend_prenautilus': '140', 'last_force_op_resend_preluminous': '140', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 147, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 67, 'pool_name': 'LibRadosSnapshotsSelfManagedPP_smithi087-82115-7', 'create_time': '2024-02-23T00:31:07.424556+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 24, 'snap_epoch': 142, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 93, 'pool_name': 'LibRadosSnapshotsSelfManagedEC_smithi087-82107-14', 'create_time': '2024-02-23T00:31:32.223322+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 147, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 95, 'pool_name': 'test-rados-api-smithi087-82091-13', 'create_time': '2024-02-23T00:31:33.242792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '145', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 98, 'pool_name': 'IsCompletePP_smithi087-82019-11', 'create_time': '2024-02-23T00:31:36.278481+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e3549bad-2891-47b9-bf77-132cadc20bf7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6801', 'nonce': 2153191863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6802', 'nonce': 2153191863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6804', 'nonce': 2153191863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6803', 'nonce': 2153191863}]}, 'public_addr': '172.21.15.87:6801/2153191863', 'cluster_addr': '172.21.15.87:6802/2153191863', 'heartbeat_back_addr': '172.21.15.87:6804/2153191863', 'heartbeat_front_addr': '172.21.15.87:6803/2153191863', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '85525ce8-38d5-4840-8057-547cb176c859', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6805', 'nonce': 685354171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6806', 'nonce': 685354171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6808', 'nonce': 685354171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6807', 'nonce': 685354171}]}, 'public_addr': '172.21.15.87:6805/685354171', 'cluster_addr': '172.21.15.87:6806/685354171', 'heartbeat_back_addr': '172.21.15.87:6808/685354171', 'heartbeat_front_addr': '172.21.15.87:6807/685354171', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6809', 'nonce': 3605394228}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6810', 'nonce': 3605394228}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6812', 'nonce': 3605394228}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6811', 'nonce': 3605394228}]}, 'public_addr': '172.21.15.87:6809/3605394228', 'cluster_addr': '172.21.15.87:6810/3605394228', 'heartbeat_back_addr': '172.21.15.87:6812/3605394228', 'heartbeat_front_addr': '172.21.15.87:6811/3605394228', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6813', 'nonce': 128059790}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6814', 'nonce': 128059790}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6816', 'nonce': 128059790}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6815', 'nonce': 128059790}]}, 'public_addr': '172.21.15.87:6813/128059790', 'cluster_addr': '172.21.15.87:6814/128059790', 'heartbeat_back_addr': '172.21.15.87:6816/128059790', 'heartbeat_front_addr': '172.21.15.87:6815/128059790', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c3734a04-7343-4251-9b5b-bedb9e8b882c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6800', 'nonce': 4175844757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6801', 'nonce': 4175844757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6803', 'nonce': 4175844757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6802', 'nonce': 4175844757}]}, 'public_addr': '172.21.15.171:6800/4175844757', 'cluster_addr': '172.21.15.171:6801/4175844757', 'heartbeat_back_addr': '172.21.15.171:6803/4175844757', 'heartbeat_front_addr': '172.21.15.171:6802/4175844757', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fe86232d-96b9-4cbf-9cac-373286fa103d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6804', 'nonce': 2550013795}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6805', 'nonce': 2550013795}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6807', 'nonce': 2550013795}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6806', 'nonce': 2550013795}]}, 'public_addr': '172.21.15.171:6804/2550013795', 'cluster_addr': '172.21.15.171:6805/2550013795', 'heartbeat_back_addr': '172.21.15.171:6807/2550013795', 'heartbeat_front_addr': '172.21.15.171:6806/2550013795', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6808', 'nonce': 2509408498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6809', 'nonce': 2509408498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6811', 'nonce': 2509408498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6810', 'nonce': 2509408498}]}, 'public_addr': '172.21.15.171:6808/2509408498', 'cluster_addr': '172.21.15.171:6809/2509408498', 'heartbeat_back_addr': '172.21.15.171:6811/2509408498', 'heartbeat_front_addr': '172.21.15.171:6810/2509408498', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '29ddcf97-369a-4a2c-b3e2-142e84ccf939', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6812', 'nonce': 373787000}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6813', 'nonce': 373787000}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6815', 'nonce': 373787000}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6814', 'nonce': 373787000}]}, 'public_addr': '172.21.15.171:6812/373787000', 'cluster_addr': '172.21.15.171:6813/373787000', 'heartbeat_back_addr': '172.21.15.171:6815/373787000', 'heartbeat_front_addr': '172.21.15.171:6814/373787000', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:31:30.371681+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:30:42.979539+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:31:06.316853+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:30:40.497481+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:31:11.892843+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:31:11.261861+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:31:32.030302+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:30:59.983249+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.a', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.87:6800/1121018082': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/2138166229': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/27794119': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/1675126043': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/3767016525': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/969918521': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/795095363': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2273151327': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/1685406414': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2093733106': '2024-02-24T00:21:58.144574+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 21, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 61, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 86, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-23T00:31:40.944 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.a 2024-02-23T00:31:40.944 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd rm-pg-upmap-items 2.a 2024-02-23T00:31:40.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:31:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:31:41.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:39.493868+0000 osd.5 (osd.5) 81 : cluster [DBG] 6.19 deep-scrub starts 2024-02-23T00:31:41.302 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:39.494729+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.19 deep-scrub ok 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:39.568413+0000 mgr.y (mgr.24437) 198 : cluster [DBG] pgmap v205: 374 pgs: 32 unknown, 342 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:39.640378+0000 osd.0 (osd.0) 97 : cluster [DBG] 3.6 deep-scrub starts 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:39.641614+0000 osd.0 (osd.0) 98 : cluster [DBG] 3.6 deep-scrub ok 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.982015+0000 mon.a (mon.0) 1209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi087-82011-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.982189+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:39.982262+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.995459+0000 mon.b (mon.1) 150 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.995791+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.87:0/1223102580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi087-82019-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.996630+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi087-82019-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:41.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:39.997398+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:31:41.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:40.028952+0000 osd.6 (osd.6) 67 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:31:41.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:40.030771+0000 osd.6 (osd.6) 68 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:31:41.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:40.070477+0000 osd.4 (osd.4) 81 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:31:41.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:40.071290+0000 osd.4 (osd.4) 82 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:31:41.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:40.384416+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:41.304 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:39.493868+0000 osd.5 (osd.5) 81 : cluster [DBG] 6.19 deep-scrub starts 2024-02-23T00:31:41.304 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:39.494729+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.19 deep-scrub ok 2024-02-23T00:31:41.304 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:39.568413+0000 mgr.y (mgr.24437) 198 : cluster [DBG] pgmap v205: 374 pgs: 32 unknown, 342 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:39.640378+0000 osd.0 (osd.0) 97 : cluster [DBG] 3.6 deep-scrub starts 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:39.641614+0000 osd.0 (osd.0) 98 : cluster [DBG] 3.6 deep-scrub ok 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.982015+0000 mon.a (mon.0) 1209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi087-82011-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.982189+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:39.982262+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.995459+0000 mon.b (mon.1) 150 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.995791+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.87:0/1223102580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi087-82019-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.996630+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi087-82019-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:39.997398+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:40.028952+0000 osd.6 (osd.6) 67 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:40.030771+0000 osd.6 (osd.6) 68 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:40.070477+0000 osd.4 (osd.4) 81 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:40.071290+0000 osd.4 (osd.4) 82 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:31:41.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:40.384416+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:41.349 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:39.493868+0000 osd.5 (osd.5) 81 : cluster [DBG] 6.19 deep-scrub starts 2024-02-23T00:31:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:39.494729+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.19 deep-scrub ok 2024-02-23T00:31:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:39.568413+0000 mgr.y (mgr.24437) 198 : cluster [DBG] pgmap v205: 374 pgs: 32 unknown, 342 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:39.640378+0000 osd.0 (osd.0) 97 : cluster [DBG] 3.6 deep-scrub starts 2024-02-23T00:31:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:39.641614+0000 osd.0 (osd.0) 98 : cluster [DBG] 3.6 deep-scrub ok 2024-02-23T00:31:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.982015+0000 mon.a (mon.0) 1209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi087-82011-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.982189+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:39.982262+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.995459+0000 mon.b (mon.1) 150 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.995791+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.87:0/1223102580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi087-82019-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.996630+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi087-82019-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:39.997398+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:40.028952+0000 osd.6 (osd.6) 67 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:40.030771+0000 osd.6 (osd.6) 68 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:40.070477+0000 osd.4 (osd.4) 81 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:40.071290+0000 osd.4 (osd.4) 82 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:31:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:40.384416+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:41.592 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:41.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:41.696 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:41.725 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:42.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:40.674601+0000 mgr.y (mgr.24437) 199 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:42.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:40.844609+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:31:42.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:40.847779+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:31:42.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:40.985804+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi087-82019-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:40.985915+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-13"}]': finished 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:40.986040+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:40.991461+0000 osd.6 (osd.6) 69 : cluster [DBG] 4.a scrub starts 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:40.991669+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-13", "mode": "writeback"}]: dispatch 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:41.010793+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-13", "mode": "writeback"}]: dispatch 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.019433+0000 osd.6 (osd.6) 70 : cluster [DBG] 4.a scrub ok 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.035025+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.036831+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:41.385644+0000 mon.c (mon.2) 120 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:41.569985+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.970619+0000 mon.a (mon.0) 1219 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:42.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:41.975932+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-13", "mode": "writeback"}]': finished 2024-02-23T00:31:42.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:40.674601+0000 mgr.y (mgr.24437) 199 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:42.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:40.844609+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:31:42.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:40.847779+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:31:42.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:40.985804+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi087-82019-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:42.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:40.985915+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-13"}]': finished 2024-02-23T00:31:42.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:40.986040+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-02-23T00:31:42.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:40.991461+0000 osd.6 (osd.6) 69 : cluster [DBG] 4.a scrub starts 2024-02-23T00:31:42.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:40.991669+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-13", "mode": "writeback"}]: dispatch 2024-02-23T00:31:42.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:41.010793+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-13", "mode": "writeback"}]: dispatch 2024-02-23T00:31:42.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.019433+0000 osd.6 (osd.6) 70 : cluster [DBG] 4.a scrub ok 2024-02-23T00:31:42.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.035025+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:31:42.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.036831+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:31:42.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:41.385644+0000 mon.c (mon.2) 120 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:42.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:41.569985+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-23T00:31:42.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.970619+0000 mon.a (mon.0) 1219 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:42.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:41.975932+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-13", "mode": "writeback"}]': finished 2024-02-23T00:31:42.227 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:41.976076+0000 mon.a (mon.0) 1221 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-23T00:31:42.228 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.976136+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-02-23T00:31:42.228 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:41.985611+0000 mon.a (mon.0) 1223 : audit [INF] from='client.? 172.21.15.87:0/643541833' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi087-82011-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:42.228 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:41.986585+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:42.228 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:41.987248+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:42.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:41.976076+0000 mon.a (mon.0) 1221 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-23T00:31:42.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.976136+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-02-23T00:31:42.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:41.985611+0000 mon.a (mon.0) 1223 : audit [INF] from='client.? 172.21.15.87:0/643541833' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi087-82011-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:42.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:41.986585+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:42.228 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:41.987248+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:40.674601+0000 mgr.y (mgr.24437) 199 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:40.844609+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:31:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:40.847779+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:31:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:40.985804+0000 mon.a (mon.0) 1214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi087-82019-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:40.985915+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-13"}]': finished 2024-02-23T00:31:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:40.986040+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:40.991461+0000 osd.6 (osd.6) 69 : cluster [DBG] 4.a scrub starts 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:40.991669+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-13", "mode": "writeback"}]: dispatch 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:41.010793+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-13", "mode": "writeback"}]: dispatch 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.019433+0000 osd.6 (osd.6) 70 : cluster [DBG] 4.a scrub ok 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.035025+0000 osd.4 (osd.4) 83 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.036831+0000 osd.4 (osd.4) 84 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:41.385644+0000 mon.c (mon.2) 120 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:41.569985+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.970619+0000 mon.a (mon.0) 1219 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:41.975932+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-13", "mode": "writeback"}]': finished 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:41.976076+0000 mon.a (mon.0) 1221 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.976136+0000 mon.a (mon.0) 1222 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:41.985611+0000 mon.a (mon.0) 1223 : audit [INF] from='client.? 172.21.15.87:0/643541833' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi087-82011-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:41.986585+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:41.987248+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:42.476 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:42.477 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:31:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:42] "GET /metrics HTTP/1.1" 200 196372 "" "Prometheus/2.33.4" 2024-02-23T00:31:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:42.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:42.580 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:42.611 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:42.969 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:42.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:43.073 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:43.100 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:43.127 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:43.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:43.229 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.568892+0000 mgr.y (mgr.24437) 200 : cluster [DBG] pgmap v208: 374 pgs: 32 creating+peering, 1 active+clean+snaptrim, 341 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 108 KiB/s wr, 110 op/s; 319 B/s, 4 objects/s recovering 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.611078+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.614313+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.650931+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.651753+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.942456+0000 osd.3 (osd.3) 95 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:41.945901+0000 osd.3 (osd.3) 96 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.109251+0000 mon.c (mon.2) 122 : audit [DBG] from='client.? 172.21.15.87:0/1026488857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.386391+0000 mon.c (mon.2) 123 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.701359+0000 mon.b (mon.1) 152 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.702073+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.702533+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-23T00:31:43.230 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.702941+0000 mon.b (mon.1) 155 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-23T00:31:43.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.703340+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-23T00:31:43.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.703749+0000 mon.b (mon.1) 157 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-23T00:31:43.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.704153+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-23T00:31:43.231 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.704589+0000 mon.b (mon.1) 159 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-23T00:31:43.231 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.568892+0000 mgr.y (mgr.24437) 200 : cluster [DBG] pgmap v208: 374 pgs: 32 creating+peering, 1 active+clean+snaptrim, 341 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 108 KiB/s wr, 110 op/s; 319 B/s, 4 objects/s recovering 2024-02-23T00:31:43.231 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.611078+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:31:43.231 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.614313+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:31:43.231 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.650931+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:31:43.231 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.651753+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.942456+0000 osd.3 (osd.3) 95 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:41.945901+0000 osd.3 (osd.3) 96 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.109251+0000 mon.c (mon.2) 122 : audit [DBG] from='client.? 172.21.15.87:0/1026488857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.386391+0000 mon.c (mon.2) 123 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.701359+0000 mon.b (mon.1) 152 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.702073+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.702533+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.702941+0000 mon.b (mon.1) 155 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.703340+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.703749+0000 mon.b (mon.1) 157 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.704153+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.704589+0000 mon.b (mon.1) 159 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.705001+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-23T00:31:43.232 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.705423+0000 mon.b (mon.1) 161 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.982719+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? 172.21.15.87:0/643541833' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi087-82011-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.982789+0000 mon.a (mon.0) 1226 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]': finished 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:42.982935+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.994394+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:43.006351+0000 mon.c (mon.2) 125 : audit [INF] from='client.? 172.21.15.87:0/361860434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi087-82019-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.705001+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.705423+0000 mon.b (mon.1) 161 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.982719+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? 172.21.15.87:0/643541833' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi087-82011-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.982789+0000 mon.a (mon.0) 1226 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]': finished 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:42.982935+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.994394+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:43.233 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:43.006351+0000 mon.c (mon.2) 125 : audit [INF] from='client.? 172.21.15.87:0/361860434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi087-82019-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:43.234 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:43.262 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:43.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.103+0000 7fa341103700 1 -- 172.21.15.87:0/1683930987 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa33c0fcbb0 msgr2=0x7fa33c0fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.103+0000 7fa341103700 1 --2- 172.21.15.87:0/1683930987 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa33c0fcbb0 0x7fa33c0fcf90 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 rx=0x7fa32c004870 tx=0x7fa32c009cb0).stop 2024-02-23T00:31:43.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.103+0000 7fa341103700 1 -- 172.21.15.87:0/1683930987 shutdown_connections 2024-02-23T00:31:43.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.103+0000 7fa341103700 1 --2- 172.21.15.87:0/1683930987 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa33c0f7400 0x7fa33c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.103+0000 7fa341103700 1 --2- 172.21.15.87:0/1683930987 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa33c0fcbb0 0x7fa33c0fcf90 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.103+0000 7fa341103700 1 --2- 172.21.15.87:0/1683930987 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa33c0fd510 0x7fa33c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.103+0000 7fa341103700 1 -- 172.21.15.87:0/1683930987 >> 172.21.15.87:0/1683930987 conn(0x7fa33c0f2820 msgr2=0x7fa33c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.103+0000 7fa341103700 1 -- 172.21.15.87:0/1683930987 shutdown_connections 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.103+0000 7fa341103700 1 -- 172.21.15.87:0/1683930987 wait complete. 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.104+0000 7fa341103700 1 librados: starting msgr at 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.104+0000 7fa341103700 1 librados: starting objecter 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.104+0000 7fa341103700 1 Processor -- start 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa341103700 1 -- start start 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa341103700 1 librados: setting wanted keys 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa341103700 1 librados: calling monclient init 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa341103700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa33c0f7400 0x7fa33c109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa341103700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa33c0fcbb0 0x7fa33c1096f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa341103700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa33c0fd510 0x7fa33c109d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa341103700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c114430 con 0x7fa33c0fcbb0 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa341103700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c1145a0 con 0x7fa33c0f7400 2024-02-23T00:31:43.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa341103700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c114760 con 0x7fa33c0fd510 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa33a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa33c0fcbb0 0x7fa33c1096f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa33b59e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa33c0fd510 0x7fa33c109d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa33b59e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa33c0fd510 0x7fa33c109d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:57068/0 (socket says 172.21.15.87:57068) 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa33a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa33c0fcbb0 0x7fa33c1096f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47594/0 (socket says 172.21.15.87:47594) 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa33b59e700 1 -- 172.21.15.87:0/1026488857 learned_addr learned my addr 172.21.15.87:0/1026488857 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa33ad9d700 1 --2- 172.21.15.87:0/1026488857 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa33c0f7400 0x7fa33c109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.105+0000 7fa33b59e700 1 -- 172.21.15.87:0/1026488857 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa33c0f7400 msgr2=0x7fa33c109200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa33b59e700 1 --2- 172.21.15.87:0/1026488857 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa33c0f7400 0x7fa33c109200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa33b59e700 1 -- 172.21.15.87:0/1026488857 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa33c0fcbb0 msgr2=0x7fa33c1096f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa33b59e700 1 --2- 172.21.15.87:0/1026488857 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa33c0fcbb0 0x7fa33c1096f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa33b59e700 1 -- 172.21.15.87:0/1026488857 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c005040 con 0x7fa33c0fd510 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa33b59e700 1 --2- 172.21.15.87:0/1026488857 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa33c0fd510 0x7fa33c109d20 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 rx=0x7fa334009fc0 tx=0x7fa33400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa323fff700 10 client.?.objecter ms_handle_connect 0x7fa33c0fd510 2024-02-23T00:31:43.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa323fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334009950 con 0x7fa33c0fd510 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa33400f040 con 0x7fa33c0fd510 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334007ca0 con 0x7fa33c0fd510 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33c107050 con 0x7fa33c0fd510 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa341103700 10 client.36752.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33c1074f0 con 0x7fa33c0fd510 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.106+0000 7fa341103700 1 librados: init done 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.107+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa33c05fa60 con 0x7fa33c0fd510 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.107+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fa334019030 con 0x7fa33c0fd510 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa323fff700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa324062c90 0x7fa324065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(151..151 src has 38..151) v4 ==== 9609+0+0 (secure 0 0 0) 0x7fa334081600 con 0x7fa33c0fd510 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 osd_map(151..151 src has 38..151) v4 2024-02-23T00:31:43.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa323fff700 3 client.36752.objecter handle_osd_map got epochs [151,151] > 0 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa323fff700 3 client.36752.objecter handle_osd_map decoding full epoch 151 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa33ad9d700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa324062c90 0x7fa324065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa323fff700 20 client.36752.objecter dump_active .. 0 homeless 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v151) v1 ==== 54+0+16 (secure 0 0 0) 0x7fa334088070 con 0x7fa33c0fd510 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa323fff700 10 client.36752.objecter ms_handle_connect 0x7fa324062c90 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa33ad9d700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa324062c90 0x7fa324065150 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7fa32c0043d0 tx=0x7fa32c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 10 client.36752.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 1 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6801/2153191863 conn(0x7fa33c0fdea0 0x7fa33c0fe280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c1060e0 osd=0 3 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c1060e0 osd=0 3 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c1060e0 osd=0 3 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 1 2024-02-23T00:31:43.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c1060e0 osd=0 3 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 0 1 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c1060e0 osd=0 4 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 10 client.36752.objecter _send_command 1 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa33c0fe770 con 0x7fa33c0fdea0 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.108+0000 7fa33a59c700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6801/2153191863 conn(0x7fa33c0fdea0 0x7fa33c0fe280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.109+0000 7fa33a59c700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6801/2153191863 conn(0x7fa33c0fdea0 0x7fa33c0fe280 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.109+0000 7fa323fff700 10 client.36752.objecter ms_handle_connect 0x7fa33c0fdea0 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.109+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa33c0fe770 con 0x7fa33c0fdea0 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.109+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.109+0000 7fa323fff700 10 client.36752.objecter _finish_command 1 = system:0 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.109+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c1060e0 osd=0 4 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.109+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 0 1 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 10 client.36752.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 2 2024-02-23T00:31:43.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c1060e0 osd=0 3 2024-02-23T00:31:43.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c1060e0 osd=0 3 2024-02-23T00:31:43.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c1060e0 osd=0 3 2024-02-23T00:31:43.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 2 2024-02-23T00:31:43.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c1060e0 osd=0 3 2024-02-23T00:31:43.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 0 2 2024-02-23T00:31:43.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c1060e0 osd=0 4 2024-02-23T00:31:43.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 10 client.36752.objecter _send_command 2 2024-02-23T00:31:43.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.134+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa33c0fe960 con 0x7fa33c0fdea0 2024-02-23T00:31:43.342 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.135+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fa33c0fe960 con 0x7fa33c0fdea0 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.135+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.135+0000 7fa323fff700 10 client.36752.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c1060e0 osd=0 4 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 0 2 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 10 client.36752.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 3 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6805/685354171 conn(0x7fa33c028540 0x7fa33c107710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c0fc4e0 osd=1 3 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c0fc4e0 osd=1 3 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c0fc4e0 osd=1 3 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 3 2024-02-23T00:31:43.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c0fc4e0 osd=1 3 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 1 3 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c0fc4e0 osd=1 4 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 10 client.36752.objecter _send_command 3 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa33c0fc650 con 0x7fa33c028540 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.136+0000 7fa33ad9d700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6805/685354171 conn(0x7fa33c028540 0x7fa33c107710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.137+0000 7fa323fff700 10 client.36752.objecter ms_handle_connect 0x7fa33c028540 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.137+0000 7fa33ad9d700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6805/685354171 conn(0x7fa33c028540 0x7fa33c107710 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.137+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa33c0fc650 con 0x7fa33c028540 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.137+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.137+0000 7fa323fff700 10 client.36752.objecter _finish_command 3 = system:0 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.137+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c0fc4e0 osd=1 4 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.137+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 1 3 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 10 client.36752.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:43.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 4 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c0fc4e0 osd=1 3 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c0fc4e0 osd=1 3 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c0fc4e0 osd=1 3 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 4 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c0fc4e0 osd=1 3 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 1 4 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c0fc4e0 osd=1 4 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 10 client.36752.objecter _send_command 4 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.157+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa33c05a040 con 0x7fa33c028540 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fa33c05a040 con 0x7fa33c028540 2024-02-23T00:31:43.345 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa323fff700 10 client.36752.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c0fc4e0 osd=1 4 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 1 4 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 10 client.36752.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 5 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6809/3605394228 conn(0x7fa33c027b90 0x7fa33c027fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c108d80 osd=2 3 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108d80 osd=2 3 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c108d80 osd=2 3 2024-02-23T00:31:43.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 5 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c108d80 osd=2 3 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 2 5 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108d80 osd=2 4 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 10 client.36752.objecter _send_command 5 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa33c108ef0 con 0x7fa33c027b90 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa33a59c700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6809/3605394228 conn(0x7fa33c027b90 0x7fa33c027fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa323fff700 10 client.36752.objecter ms_handle_connect 0x7fa33c027b90 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.159+0000 7fa33a59c700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6809/3605394228 conn(0x7fa33c027b90 0x7fa33c027fb0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.160+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa33c108ef0 con 0x7fa33c027b90 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.160+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.160+0000 7fa323fff700 10 client.36752.objecter _finish_command 5 = system:0 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.160+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c108d80 osd=2 4 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.160+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 2 5 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 10 client.36752.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:43.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 6 2024-02-23T00:31:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c108d80 osd=2 3 2024-02-23T00:31:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108d80 osd=2 3 2024-02-23T00:31:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c108d80 osd=2 3 2024-02-23T00:31:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 6 2024-02-23T00:31:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c108d80 osd=2 3 2024-02-23T00:31:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 2 6 2024-02-23T00:31:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108d80 osd=2 4 2024-02-23T00:31:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 10 client.36752.objecter _send_command 6 2024-02-23T00:31:43.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.175+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa33c027600 con 0x7fa33c027b90 2024-02-23T00:31:43.349 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:31:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fa33c027600 con 0x7fa33c027b90 2024-02-23T00:31:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa323fff700 10 client.36752.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c108d80 osd=2 4 2024-02-23T00:31:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 2 6 2024-02-23T00:31:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 10 client.36752.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 7 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6813/128059790 conn(0x7fa33c1088e0 0x7fa33c11dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c027a10 osd=3 3 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c027a10 osd=3 3 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c027a10 osd=3 3 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 7 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c027a10 osd=3 3 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 3 7 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c027a10 osd=3 4 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 10 client.36752.objecter _send_command 7 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa33c1207f0 con 0x7fa33c1088e0 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa33b59e700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6813/128059790 conn(0x7fa33c1088e0 0x7fa33c11dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa323fff700 10 client.36752.objecter ms_handle_connect 0x7fa33c1088e0 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.177+0000 7fa33b59e700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6813/128059790 conn(0x7fa33c1088e0 0x7fa33c11dba0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.178+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa33c1207f0 con 0x7fa33c1088e0 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.178+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:31:43.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.178+0000 7fa323fff700 10 client.36752.objecter _finish_command 7 = system:0 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.178+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c027a10 osd=3 4 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.178+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 3 7 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 10 client.36752.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 8 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c027a10 osd=3 3 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c027a10 osd=3 3 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c027a10 osd=3 3 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 8 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c027a10 osd=3 3 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 3 8 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c027a10 osd=3 4 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 10 client.36752.objecter _send_command 8 2024-02-23T00:31:43.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.193+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa33c107e40 con 0x7fa33c1088e0 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fa33c107e40 con 0x7fa33c1088e0 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa323fff700 10 client.36752.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c027a10 osd=3 4 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 3 8 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 10 client.36752.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 9 2024-02-23T00:31:43.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6800/4175844757 conn(0x7fa33c11b6f0 0x7fa33c11f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c107e40 osd=4 3 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c107e40 osd=4 3 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c107e40 osd=4 3 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 9 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c107e40 osd=4 3 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 4 9 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c107e40 osd=4 4 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 10 client.36752.objecter _send_command 9 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa33c108130 con 0x7fa33c11b6f0 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.195+0000 7fa33ad9d700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6800/4175844757 conn(0x7fa33c11b6f0 0x7fa33c11f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.196+0000 7fa323fff700 10 client.36752.objecter ms_handle_connect 0x7fa33c11b6f0 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.196+0000 7fa33ad9d700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6800/4175844757 conn(0x7fa33c11b6f0 0x7fa33c11f210 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.196+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa33c108130 con 0x7fa33c11b6f0 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.196+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.196+0000 7fa323fff700 10 client.36752.objecter _finish_command 9 = system:0 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.196+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c107e40 osd=4 4 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.196+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 4 9 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 10 client.36752.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 10 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c107e40 osd=4 3 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c107e40 osd=4 3 2024-02-23T00:31:43.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c107e40 osd=4 3 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 10 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c107e40 osd=4 3 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 4 10 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c107e40 osd=4 4 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 10 client.36752.objecter _send_command 10 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.212+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa33c108130 con 0x7fa33c11b6f0 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.213+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fa33c108130 con 0x7fa33c11b6f0 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.213+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.213+0000 7fa323fff700 10 client.36752.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.213+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c107e40 osd=4 4 2024-02-23T00:31:43.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.213+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 4 10 2024-02-23T00:31:43.356 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:31:43.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 10 client.36752.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:43.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 11 2024-02-23T00:31:43.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6804/2550013795 conn(0x7fa33c11fc10 0x7fa33c120070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c108130 osd=5 3 2024-02-23T00:31:43.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108130 osd=5 3 2024-02-23T00:31:43.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c108130 osd=5 3 2024-02-23T00:31:43.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 11 2024-02-23T00:31:43.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c108130 osd=5 3 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 5 11 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108130 osd=5 4 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 10 client.36752.objecter _send_command 11 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa33c120650 con 0x7fa33c11fc10 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa33a59c700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6804/2550013795 conn(0x7fa33c11fc10 0x7fa33c120070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa323fff700 10 client.36752.objecter ms_handle_connect 0x7fa33c11fc10 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.214+0000 7fa33a59c700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6804/2550013795 conn(0x7fa33c11fc10 0x7fa33c120070 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.215+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa33c120650 con 0x7fa33c11fc10 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.215+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.215+0000 7fa323fff700 10 client.36752.objecter _finish_command 11 = system:0 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.215+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c108130 osd=5 4 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.215+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 5 11 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 10 client.36752.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 12 2024-02-23T00:31:43.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c108130 osd=5 3 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108130 osd=5 3 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c108130 osd=5 3 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 12 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c108130 osd=5 3 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 5 12 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108130 osd=5 4 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 10 client.36752.objecter _send_command 12 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.231+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa33c120650 con 0x7fa33c11fc10 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fa33c120650 con 0x7fa33c11fc10 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa323fff700 10 client.36752.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c108130 osd=5 4 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 5 12 2024-02-23T00:31:43.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 10 client.36752.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 13 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6808/2509408498 conn(0x7fa33c126550 0x7fa33c1269b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c120650 osd=6 3 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c120650 osd=6 3 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c120650 osd=6 3 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 13 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c120650 osd=6 3 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 6 13 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c120650 osd=6 4 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 10 client.36752.objecter _send_command 13 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa33c12a340 con 0x7fa33c126550 2024-02-23T00:31:43.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.233+0000 7fa33b59e700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6808/2509408498 conn(0x7fa33c126550 0x7fa33c1269b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.360 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:31:43.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.234+0000 7fa323fff700 10 client.36752.objecter ms_handle_connect 0x7fa33c126550 2024-02-23T00:31:43.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.234+0000 7fa33b59e700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6808/2509408498 conn(0x7fa33c126550 0x7fa33c1269b0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.234+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa33c12a340 con 0x7fa33c126550 2024-02-23T00:31:43.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.234+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:31:43.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.234+0000 7fa323fff700 10 client.36752.objecter _finish_command 13 = system:0 2024-02-23T00:31:43.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.234+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c120650 osd=6 4 2024-02-23T00:31:43.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.234+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 6 13 2024-02-23T00:31:43.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 10 client.36752.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 14 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c120650 osd=6 3 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c120650 osd=6 3 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c120650 osd=6 3 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 14 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c120650 osd=6 3 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 6 14 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c120650 osd=6 4 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 10 client.36752.objecter _send_command 14 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.250+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa33c1249d0 con 0x7fa33c126550 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fa33c1249d0 con 0x7fa33c126550 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa323fff700 10 client.36752.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c120650 osd=6 4 2024-02-23T00:31:43.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 6 14 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 10 client.36752.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 15 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6812/373787000 conn(0x7fa33c124ee0 0x7fa33c125340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c125cf0 osd=7 3 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c125cf0 osd=7 3 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c125cf0 osd=7 3 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 15 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c125cf0 osd=7 3 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 7 15 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c125cf0 osd=7 4 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 10 client.36752.objecter _send_command 15 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.252+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa33c125830 con 0x7fa33c124ee0 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.253+0000 7fa33ad9d700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6812/373787000 conn(0x7fa33c124ee0 0x7fa33c125340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.253+0000 7fa323fff700 10 client.36752.objecter ms_handle_connect 0x7fa33c124ee0 2024-02-23T00:31:43.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.253+0000 7fa33ad9d700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6812/373787000 conn(0x7fa33c124ee0 0x7fa33c125340 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.254+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fa33c125830 con 0x7fa33c124ee0 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.254+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.254+0000 7fa323fff700 10 client.36752.objecter _finish_command 15 = system:0 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.254+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c125cf0 osd=7 4 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.254+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 7 15 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 10 client.36752.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign -1 16 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c125cf0 osd=7 3 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c125cf0 osd=7 3 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 20 client.36752.objecter _get_session s=0x7fa33c125cf0 osd=7 3 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 15 client.36752.objecter _session_command_op_remove -1 16 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 20 client.36752.objecter get_session s=0x7fa33c125cf0 osd=7 3 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 15 client.36752.objecter _session_command_op_assign 7 16 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c125cf0 osd=7 4 2024-02-23T00:31:43.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 10 client.36752.objecter _send_command 16 2024-02-23T00:31:43.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.269+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa33c125830 con 0x7fa33c124ee0 2024-02-23T00:31:43.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.270+0000 7fa323fff700 1 -- 172.21.15.87:0/1026488857 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fa33c125830 con 0x7fa33c124ee0 2024-02-23T00:31:43.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.270+0000 7fa323fff700 10 client.36752.objecter ms_dispatch 0x7fa33c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:31:43.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.270+0000 7fa323fff700 10 client.36752.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.364 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:31:43.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.270+0000 7fa323fff700 20 client.36752.objecter put_session s=0x7fa33c125cf0 osd=7 4 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.270+0000 7fa323fff700 15 client.36752.objecter _session_command_op_remove 7 16 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 librados: watch_flush enter 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 librados: watch_flush exit 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 client.36752.objecter close_session for osd.0 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6801/2153191863 conn(0x7fa33c0fdea0 msgr2=0x7fa33c0fe280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6801/2153191863 conn(0x7fa33c0fdea0 0x7fa33c0fe280 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c1060e0 osd=0 1 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 client.36752.objecter close_session for osd.1 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6805/685354171 conn(0x7fa33c028540 msgr2=0x7fa33c107710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6805/685354171 conn(0x7fa33c028540 0x7fa33c107710 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c0fc4e0 osd=1 1 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 client.36752.objecter close_session for osd.2 2024-02-23T00:31:43.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6809/3605394228 conn(0x7fa33c027b90 msgr2=0x7fa33c027fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6809/3605394228 conn(0x7fa33c027b90 0x7fa33c027fb0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108d80 osd=2 1 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 client.36752.objecter close_session for osd.3 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6813/128059790 conn(0x7fa33c1088e0 msgr2=0x7fa33c11dba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6813/128059790 conn(0x7fa33c1088e0 0x7fa33c11dba0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa33b59e700 1 -- 172.21.15.87:0/1026488857 reap_dead start 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c027a10 osd=3 1 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 client.36752.objecter close_session for osd.4 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6800/4175844757 conn(0x7fa33c11b6f0 msgr2=0x7fa33c11f210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6800/4175844757 conn(0x7fa33c11b6f0 0x7fa33c11f210 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa33b59e700 1 -- 172.21.15.87:0/1026488857 reap_dead start 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c107e40 osd=4 1 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 client.36752.objecter close_session for osd.5 2024-02-23T00:31:43.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6804/2550013795 conn(0x7fa33c11fc10 msgr2=0x7fa33c120070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6804/2550013795 conn(0x7fa33c11fc10 0x7fa33c120070 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c108130 osd=5 1 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 client.36752.objecter close_session for osd.6 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6808/2509408498 conn(0x7fa33c126550 msgr2=0x7fa33c1269b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6808/2509408498 conn(0x7fa33c126550 0x7fa33c1269b0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c120650 osd=6 1 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 client.36752.objecter close_session for osd.7 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6812/373787000 conn(0x7fa33c124ee0 msgr2=0x7fa33c125340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.171:6812/373787000 conn(0x7fa33c124ee0 0x7fa33c125340 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 20 client.36752.objecter put_session s=0x7fa33c125cf0 osd=7 1 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 20 client.36752.objecter shutdown clearing up homeless session... 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 10 client.36752.objecter successfully canceled tick 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa324062c90 msgr2=0x7fa324065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa324062c90 0x7fa324065150 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 rx=0x7fa32c0043d0 tx=0x7fa32c00b040).stop 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa33c0fd510 msgr2=0x7fa33c109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 --2- 172.21.15.87:0/1026488857 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa33c0fd510 0x7fa33c109d20 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 rx=0x7fa334009fc0 tx=0x7fa33400b040).stop 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa33b59e700 1 -- 172.21.15.87:0/1026488857 reap_dead start 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 shutdown_connections 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.271+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 >> 172.21.15.87:0/1026488857 conn(0x7fa33c0f2820 msgr2=0x7fa33c0fa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.272+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 shutdown_connections 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.272+0000 7fa341103700 1 -- 172.21.15.87:0/1026488857 wait complete. 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:42.272+0000 7fa341103700 1 librados: shutdown 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.368 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.369 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.369 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:31:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.568892+0000 mgr.y (mgr.24437) 200 : cluster [DBG] pgmap v208: 374 pgs: 32 creating+peering, 1 active+clean+snaptrim, 341 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 108 KiB/s wr, 110 op/s; 319 B/s, 4 objects/s recovering 2024-02-23T00:31:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.611078+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:31:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.614313+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.650931+0000 osd.2 (osd.2) 75 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.651753+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.942456+0000 osd.3 (osd.3) 95 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:41.945901+0000 osd.3 (osd.3) 96 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.109251+0000 mon.c (mon.2) 122 : audit [DBG] from='client.? 172.21.15.87:0/1026488857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.386391+0000 mon.c (mon.2) 123 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.701359+0000 mon.b (mon.1) 152 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.702073+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.702533+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.702941+0000 mon.b (mon.1) 155 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.703340+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.703749+0000 mon.b (mon.1) 157 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.704153+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-23T00:31:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.704589+0000 mon.b (mon.1) 159 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-23T00:31:43.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.705001+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-23T00:31:43.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.705423+0000 mon.b (mon.1) 161 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-23T00:31:43.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.982719+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? 172.21.15.87:0/643541833' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi087-82011-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:43.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.982789+0000 mon.a (mon.0) 1226 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]': finished 2024-02-23T00:31:43.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:42.982935+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-02-23T00:31:43.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.994394+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.87:0/3246607396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:43.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:43.006351+0000 mon.c (mon.2) 125 : audit [INF] from='client.? 172.21.15.87:0/361860434' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi087-82019-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:43.535 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:44.307 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:31:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:31:44.307 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: Running main() from gmock_main.cc 2024-02-23T00:31:44.307 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2024-02-23T00:31:44.307 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] Global test environment set-up. 2024-02-23T00:31:44.307 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-02-23T00:31:44.307 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-02-23T00:31:44.307 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (3083 ms) 2024-02-23T00:31:44.307 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (4749 ms) 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2046 ms) 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2243 ms) 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (12121 ms total) 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (5694 ms) 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4787 ms) 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (10481 ms total) 2024-02-23T00:31:44.308 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3166 ms) 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1624 ms) 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2030 ms) 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1934 ms) 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8754 ms total) 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4028 ms) 2024-02-23T00:31:44.309 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-02-23T00:31:44.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4013 ms) 2024-02-23T00:31:44.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8041 ms total) 2024-02-23T00:31:44.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: 2024-02-23T00:31:44.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [----------] Global test environment tear-down 2024-02-23T00:31:44.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (61966 ms total) 2024-02-23T00:31:44.310 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots: [ PASSED ] 12 tests. 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82182 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82182 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82038 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82038 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82142 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82142 2024-02-23T00:31:44.311 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:31:44.312 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82110 2024-02-23T00:31:44.312 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82110 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.701539+0000 mgr.y (mgr.24437) 201 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.702089+0000 mgr.y (mgr.24437) 202 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.702534+0000 mgr.y (mgr.24437) 203 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.702941+0000 mgr.y (mgr.24437) 204 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.703346+0000 mgr.y (mgr.24437) 205 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.703754+0000 mgr.y (mgr.24437) 206 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.704161+0000 mgr.y (mgr.24437) 207 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.704587+0000 mgr.y (mgr.24437) 208 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.705000+0000 mgr.y (mgr.24437) 209 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:42.705430+0000 mgr.y (mgr.24437) 210 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:42.997413+0000 osd.6 (osd.6) 71 : cluster [DBG] 95.0 scrub starts 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.001717+0000 osd.6 (osd.6) 72 : cluster [DBG] 95.0 scrub ok 2024-02-23T00:31:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:43.003726+0000 mon.a (mon.0) 1228 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:43.007098+0000 mon.a (mon.0) 1229 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi087-82019-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.016405+0000 osd.4 (osd.4) 85 : cluster [DBG] 95.1 scrub starts 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.020215+0000 osd.4 (osd.4) 86 : cluster [DBG] 95.1 scrub ok 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:43.387554+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:43.981354+0000 mon.a (mon.0) 1230 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]': finished 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:43.981498+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi087-82019-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.981576+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.701539+0000 mgr.y (mgr.24437) 201 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.702089+0000 mgr.y (mgr.24437) 202 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.702534+0000 mgr.y (mgr.24437) 203 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-23T00:31:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.702941+0000 mgr.y (mgr.24437) 204 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.703346+0000 mgr.y (mgr.24437) 205 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.703754+0000 mgr.y (mgr.24437) 206 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.704161+0000 mgr.y (mgr.24437) 207 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.704587+0000 mgr.y (mgr.24437) 208 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.705000+0000 mgr.y (mgr.24437) 209 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:42.705430+0000 mgr.y (mgr.24437) 210 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:42.997413+0000 osd.6 (osd.6) 71 : cluster [DBG] 95.0 scrub starts 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.001717+0000 osd.6 (osd.6) 72 : cluster [DBG] 95.0 scrub ok 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:43.003726+0000 mon.a (mon.0) 1228 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:43.007098+0000 mon.a (mon.0) 1229 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi087-82019-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.016405+0000 osd.4 (osd.4) 85 : cluster [DBG] 95.1 scrub starts 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.020215+0000 osd.4 (osd.4) 86 : cluster [DBG] 95.1 scrub ok 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:43.387554+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:43.981354+0000 mon.a (mon.0) 1230 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]': finished 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:43.981498+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi087-82019-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.981576+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-02-23T00:31:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.701539+0000 mgr.y (mgr.24437) 201 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-02-23T00:31:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.702089+0000 mgr.y (mgr.24437) 202 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-02-23T00:31:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.702534+0000 mgr.y (mgr.24437) 203 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-02-23T00:31:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.702941+0000 mgr.y (mgr.24437) 204 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-02-23T00:31:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.703346+0000 mgr.y (mgr.24437) 205 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.703754+0000 mgr.y (mgr.24437) 206 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.704161+0000 mgr.y (mgr.24437) 207 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.704587+0000 mgr.y (mgr.24437) 208 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.705000+0000 mgr.y (mgr.24437) 209 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:42.705430+0000 mgr.y (mgr.24437) 210 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:42.997413+0000 osd.6 (osd.6) 71 : cluster [DBG] 95.0 scrub starts 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.001717+0000 osd.6 (osd.6) 72 : cluster [DBG] 95.0 scrub ok 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:43.003726+0000 mon.a (mon.0) 1228 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]: dispatch 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:43.007098+0000 mon.a (mon.0) 1229 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi087-82019-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.016405+0000 osd.4 (osd.4) 85 : cluster [DBG] 95.1 scrub starts 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.020215+0000 osd.4 (osd.4) 86 : cluster [DBG] 95.1 scrub ok 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:43.387554+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:43.981354+0000 mon.a (mon.0) 1230 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi087-82107-14"}]': finished 2024-02-23T00:31:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:43.981498+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi087-82019-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.981576+0000 mon.a (mon.0) 1232 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-02-23T00:31:44.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 -- 172.21.15.87:0/202356616 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33600f7400 msgr2=0x7f3360103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:44.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 --2- 172.21.15.87:0/202356616 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33600f7400 0x7f3360103a20 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 rx=0x7f3358009be0 tx=0x7f3358004d10).stop 2024-02-23T00:31:44.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 -- 172.21.15.87:0/202356616 shutdown_connections 2024-02-23T00:31:44.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 --2- 172.21.15.87:0/202356616 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33600f7400 0x7f3360103a20 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:44.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 --2- 172.21.15.87:0/202356616 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33600fcbb0 0x7f33600fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:44.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 --2- 172.21.15.87:0/202356616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33600fd510 0x7f33600f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:44.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 -- 172.21.15.87:0/202356616 >> 172.21.15.87:0/202356616 conn(0x7f33600f2820 msgr2=0x7f33600f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:44.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 -- 172.21.15.87:0/202356616 shutdown_connections 2024-02-23T00:31:44.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 -- 172.21.15.87:0/202356616 wait complete. 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 librados: starting msgr at 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.860+0000 7f3364e40700 1 librados: starting objecter 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 Processor -- start 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 -- start start 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 librados: setting wanted keys 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 librados: calling monclient init 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33600f7400 0x7f336010e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33600fcbb0 0x7f336010e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33600fd510 0x7f33601081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33601086b0 con 0x7f33600f7400 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3360108820 con 0x7f33600fd510 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f3364e40700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33601089e0 con 0x7f33600fcbb0 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335dd9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33600fcbb0 0x7f336010e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:44.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335dd9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33600fcbb0 0x7f336010e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:57112/0 (socket says 172.21.15.87:57112) 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335dd9b700 1 -- 172.21.15.87:0/3167373880 learned_addr learned my addr 172.21.15.87:0/3167373880 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335e59c700 1 --2- 172.21.15.87:0/3167373880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33600f7400 0x7f336010e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335ed9d700 1 --2- 172.21.15.87:0/3167373880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33600fd510 0x7f33601081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335dd9b700 1 -- 172.21.15.87:0/3167373880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33600fd510 msgr2=0x7f33601081c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335dd9b700 1 --2- 172.21.15.87:0/3167373880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33600fd510 0x7f33601081c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335dd9b700 1 -- 172.21.15.87:0/3167373880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33600f7400 msgr2=0x7f336010e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335dd9b700 1 --2- 172.21.15.87:0/3167373880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33600f7400 0x7f336010e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335dd9b700 1 -- 172.21.15.87:0/3167373880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3350005040 con 0x7f33600fcbb0 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f335dd9b700 1 --2- 172.21.15.87:0/3167373880 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33600fcbb0 0x7f336010e510 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7f335400c790 tx=0x7f3354004330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f334f7fe700 10 client.?.objecter ms_handle_connect 0x7f33600fcbb0 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.861+0000 7f334f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.862+0000 7f334f7fe700 1 -- 172.21.15.87:0/3167373880 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540043a0 con 0x7f33600fcbb0 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.862+0000 7f334f7fe700 1 -- 172.21.15.87:0/3167373880 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f335400e970 con 0x7f33600fcbb0 2024-02-23T00:31:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.862+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33580097c0 con 0x7f33600fcbb0 2024-02-23T00:31:44.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.862+0000 7f334f7fe700 1 -- 172.21.15.87:0/3167373880 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540129f0 con 0x7f33600fcbb0 2024-02-23T00:31:44.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.862+0000 7f3364e40700 10 client.36764.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:44.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.862+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3360114e50 con 0x7f33600fcbb0 2024-02-23T00:31:44.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.862+0000 7f3364e40700 1 librados: init done 2024-02-23T00:31:44.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.863+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336005fa60 con 0x7f33600fcbb0 2024-02-23T00:31:44.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.863+0000 7f334f7fe700 1 -- 172.21.15.87:0/3167373880 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f335400e4a0 con 0x7f33600fcbb0 2024-02-23T00:31:44.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.863+0000 7f334f7fe700 1 --2- 172.21.15.87:0/3167373880 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3348062c90 0x7f3348065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:44.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.863+0000 7f334f7fe700 1 -- 172.21.15.87:0/3167373880 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(153..153 src has 38..153) v4 ==== 9186+0+0 (secure 0 0 0) 0x7f335407cd80 con 0x7f33600fcbb0 2024-02-23T00:31:44.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.863+0000 7f334f7fe700 10 client.36764.objecter ms_dispatch 0x7f3360001550 osd_map(153..153 src has 38..153) v4 2024-02-23T00:31:44.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.863+0000 7f334f7fe700 3 client.36764.objecter handle_osd_map got epochs [153,153] > 0 2024-02-23T00:31:44.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.863+0000 7f334f7fe700 3 client.36764.objecter handle_osd_map decoding full epoch 153 2024-02-23T00:31:44.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.863+0000 7f335e59c700 1 --2- 172.21.15.87:0/3167373880 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3348062c90 0x7f3348065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:44.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.864+0000 7f334f7fe700 20 client.36764.objecter dump_active .. 0 homeless 2024-02-23T00:31:44.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.864+0000 7f334f7fe700 10 client.36764.objecter ms_handle_connect 0x7f3348062c90 2024-02-23T00:31:44.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.864+0000 7f335e59c700 1 --2- 172.21.15.87:0/3167373880 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3348062c90 0x7f3348065150 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 rx=0x7f3350009350 tx=0x7f335000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:44.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:44.868+0000 7f334f7fe700 1 -- 172.21.15.87:0/3167373880 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3354048f60 con 0x7f33600fcbb0 2024-02-23T00:31:45.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.070+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f3360106180 con 0x7f33600fcbb0 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.421467+0000 osd.5 (osd.5) 83 : cluster [DBG] 95.3 scrub starts 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.422704+0000 osd.5 (osd.5) 84 : cluster [DBG] 95.3 scrub ok 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.569538+0000 mgr.y (mgr.24437) 211 : cluster [DBG] pgmap v211: 398 pgs: 64 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 100 KiB/s wr, 108 op/s; 319 B/s, 4 objects/s recovering 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.689027+0000 osd.0 (osd.0) 101 : cluster [DBG] 95.4 scrub starts 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.690727+0000 osd.0 (osd.0) 102 : cluster [DBG] 95.4 scrub ok 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.870071+0000 osd.7 (osd.7) 85 : cluster [DBG] 95.2 scrub starts 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.871865+0000 osd.7 (osd.7) 86 : cluster [DBG] 95.2 scrub ok 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.938670+0000 osd.3 (osd.3) 97 : cluster [DBG] 95.5 scrub starts 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:43.942397+0000 osd.3 (osd.3) 98 : cluster [DBG] 95.5 scrub ok 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:44.010476+0000 mon.a (mon.0) 1233 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:44.388520+0000 mon.c (mon.2) 127 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:44.831442+0000 osd.1 (osd.1) 91 : cluster [DBG] 95.8 scrub starts 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:44.833688+0000 osd.1 (osd.1) 92 : cluster [DBG] 95.8 scrub ok 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:44.984220+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:44.990045+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.87:0/2013654819' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi087-82011-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:44.995247+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi087-82011-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.421467+0000 osd.5 (osd.5) 83 : cluster [DBG] 95.3 scrub starts 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.422704+0000 osd.5 (osd.5) 84 : cluster [DBG] 95.3 scrub ok 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.569538+0000 mgr.y (mgr.24437) 211 : cluster [DBG] pgmap v211: 398 pgs: 64 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 100 KiB/s wr, 108 op/s; 319 B/s, 4 objects/s recovering 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.689027+0000 osd.0 (osd.0) 101 : cluster [DBG] 95.4 scrub starts 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.690727+0000 osd.0 (osd.0) 102 : cluster [DBG] 95.4 scrub ok 2024-02-23T00:31:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.870071+0000 osd.7 (osd.7) 85 : cluster [DBG] 95.2 scrub starts 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.871865+0000 osd.7 (osd.7) 86 : cluster [DBG] 95.2 scrub ok 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.938670+0000 osd.3 (osd.3) 97 : cluster [DBG] 95.5 scrub starts 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:43.942397+0000 osd.3 (osd.3) 98 : cluster [DBG] 95.5 scrub ok 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:44.010476+0000 mon.a (mon.0) 1233 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:44.388520+0000 mon.c (mon.2) 127 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:44.831442+0000 osd.1 (osd.1) 91 : cluster [DBG] 95.8 scrub starts 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:44.833688+0000 osd.1 (osd.1) 92 : cluster [DBG] 95.8 scrub ok 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:44.984220+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:44.990045+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.87:0/2013654819' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi087-82011-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:44.995247+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi087-82011-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.421467+0000 osd.5 (osd.5) 83 : cluster [DBG] 95.3 scrub starts 2024-02-23T00:31:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.422704+0000 osd.5 (osd.5) 84 : cluster [DBG] 95.3 scrub ok 2024-02-23T00:31:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.569538+0000 mgr.y (mgr.24437) 211 : cluster [DBG] pgmap v211: 398 pgs: 64 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 100 KiB/s wr, 108 op/s; 319 B/s, 4 objects/s recovering 2024-02-23T00:31:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.689027+0000 osd.0 (osd.0) 101 : cluster [DBG] 95.4 scrub starts 2024-02-23T00:31:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.690727+0000 osd.0 (osd.0) 102 : cluster [DBG] 95.4 scrub ok 2024-02-23T00:31:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.870071+0000 osd.7 (osd.7) 85 : cluster [DBG] 95.2 scrub starts 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.871865+0000 osd.7 (osd.7) 86 : cluster [DBG] 95.2 scrub ok 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.938670+0000 osd.3 (osd.3) 97 : cluster [DBG] 95.5 scrub starts 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:43.942397+0000 osd.3 (osd.3) 98 : cluster [DBG] 95.5 scrub ok 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:44.010476+0000 mon.a (mon.0) 1233 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:44.388520+0000 mon.c (mon.2) 127 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:44.831442+0000 osd.1 (osd.1) 91 : cluster [DBG] 95.8 scrub starts 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:44.833688+0000 osd.1 (osd.1) 92 : cluster [DBG] 95.8 scrub ok 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:44.984220+0000 mon.a (mon.0) 1234 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:44.990045+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.87:0/2013654819' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi087-82011-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:44.995247+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi087-82011-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:45.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.987+0000 7f334f7fe700 1 -- 172.21.15.87:0/3167373880 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v155) v1 ==== 87+0+0 (secure 0 0 0) 0x7f3354017030 con 0x7f33600fcbb0 2024-02-23T00:31:45.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.992+0000 7f3364e40700 10 librados: watch_flush enter 2024-02-23T00:31:45.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.992+0000 7f3364e40700 10 librados: watch_flush exit 2024-02-23T00:31:45.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.992+0000 7f3364e40700 20 client.36764.objecter shutdown clearing up homeless session... 2024-02-23T00:31:45.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.992+0000 7f3364e40700 10 client.36764.objecter successfully canceled tick 2024-02-23T00:31:45.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.992+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3348062c90 msgr2=0x7f3348065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:45.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.992+0000 7f3364e40700 1 --2- 172.21.15.87:0/3167373880 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3348062c90 0x7f3348065150 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 rx=0x7f3350009350 tx=0x7f335000b040).stop 2024-02-23T00:31:45.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.992+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33600fcbb0 msgr2=0x7f336010e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:45.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.992+0000 7f3364e40700 1 --2- 172.21.15.87:0/3167373880 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33600fcbb0 0x7f336010e510 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7f335400c790 tx=0x7f3354004330).stop 2024-02-23T00:31:45.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.997+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 shutdown_connections 2024-02-23T00:31:45.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.997+0000 7f3364e40700 1 --2- 172.21.15.87:0/3167373880 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3348062c90 0x7f3348065150 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:45.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.997+0000 7f3364e40700 1 --2- 172.21.15.87:0/3167373880 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33600fd510 0x7f33601081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:45.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.997+0000 7f3364e40700 1 --2- 172.21.15.87:0/3167373880 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33600f7400 0x7f336010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:45.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.997+0000 7f3364e40700 1 --2- 172.21.15.87:0/3167373880 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33600fcbb0 0x7f336010e510 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:45.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.997+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 >> 172.21.15.87:0/3167373880 conn(0x7f33600f2820 msgr2=0x7f3360060dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:46.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:45.999+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 shutdown_connections 2024-02-23T00:31:46.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:46.000+0000 7f3364e40700 1 -- 172.21.15.87:0/3167373880 wait complete. 2024-02-23T00:31:46.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:46.000+0000 7f3364e40700 1 librados: shutdown 2024-02-23T00:31:46.012 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:31:46.187 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:45.021688+0000 osd.6 (osd.6) 73 : cluster [DBG] 95.6 scrub starts 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:45.025151+0000 osd.6 (osd.6) 74 : cluster [DBG] 95.6 scrub ok 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:45.071740+0000 mon.c (mon.2) 129 : audit [INF] from='client.? 172.21.15.87:0/3167373880' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:45.266525+0000 mon.a (mon.0) 1236 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:45.389434+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:45.571966+0000 mon.a (mon.0) 1237 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:45.981300+0000 mon.a (mon.0) 1238 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:45.986536+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi087-82011-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:45.986699+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:45.986765+0000 mon.a (mon.0) 1241 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-23T00:31:46.188 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:45.986825+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:45.996484+0000 mon.a (mon.0) 1243 : audit [INF] from='client.? 172.21.15.87:0/662284483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi087-82019-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:45.021688+0000 osd.6 (osd.6) 73 : cluster [DBG] 95.6 scrub starts 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:45.025151+0000 osd.6 (osd.6) 74 : cluster [DBG] 95.6 scrub ok 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:45.071740+0000 mon.c (mon.2) 129 : audit [INF] from='client.? 172.21.15.87:0/3167373880' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:45.266525+0000 mon.a (mon.0) 1236 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:45.389434+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:45.571966+0000 mon.a (mon.0) 1237 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:45.981300+0000 mon.a (mon.0) 1238 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:45.986536+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi087-82011-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:46.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:45.986699+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:31:46.190 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:45.986765+0000 mon.a (mon.0) 1241 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-23T00:31:46.190 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:45.986825+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-02-23T00:31:46.190 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:45.996484+0000 mon.a (mon.0) 1243 : audit [INF] from='client.? 172.21.15.87:0/662284483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi087-82019-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:46.313 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:31:46.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:45.021688+0000 osd.6 (osd.6) 73 : cluster [DBG] 95.6 scrub starts 2024-02-23T00:31:46.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:45.025151+0000 osd.6 (osd.6) 74 : cluster [DBG] 95.6 scrub ok 2024-02-23T00:31:46.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:45.071740+0000 mon.c (mon.2) 129 : audit [INF] from='client.? 172.21.15.87:0/3167373880' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:31:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:45.266525+0000 mon.a (mon.0) 1236 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:31:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:45.389434+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:45.571966+0000 mon.a (mon.0) 1237 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-23T00:31:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:45.981300+0000 mon.a (mon.0) 1238 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:45.986536+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi087-82011-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:45.986699+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:31:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:45.986765+0000 mon.a (mon.0) 1241 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-23T00:31:46.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:45.986825+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-02-23T00:31:46.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:45.996484+0000 mon.a (mon.0) 1243 : audit [INF] from='client.? 172.21.15.87:0/662284483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi087-82019-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:46.642 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:46.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:46.746 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:46.778 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:46.799 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:46.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:46.903 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:46.933 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:46.953 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:46.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:47.057 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:47.085 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:47.103 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:47.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:47.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:47.158 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:31:47.158 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:31:47.206 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:31:47.233 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:47.251 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:31:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:47.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:31:47.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:45.570815+0000 mgr.y (mgr.24437) 212 : cluster [DBG] pgmap v214: 365 pgs: 32 creating+peering, 333 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 53 KiB/s rd, 76 op/s 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:45.896690+0000 osd.7 (osd.7) 87 : cluster [DBG] 95.7 scrub starts 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:45.900146+0000 osd.7 (osd.7) 88 : cluster [DBG] 95.7 scrub ok 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:45.960491+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:45.961382+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:46.390590+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:46.988191+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? 172.21.15.87:0/662284483' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi087-82019-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:46.988368+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:45.570815+0000 mgr.y (mgr.24437) 212 : cluster [DBG] pgmap v214: 365 pgs: 32 creating+peering, 333 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 53 KiB/s rd, 76 op/s 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:45.896690+0000 osd.7 (osd.7) 87 : cluster [DBG] 95.7 scrub starts 2024-02-23T00:31:47.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:45.900146+0000 osd.7 (osd.7) 88 : cluster [DBG] 95.7 scrub ok 2024-02-23T00:31:47.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:45.960491+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:31:47.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:45.961382+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:31:47.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:46.390590+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:47.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:46.988191+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? 172.21.15.87:0/662284483' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi087-82019-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:47.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:46.988368+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-02-23T00:31:47.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:47.383 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:47.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:45.570815+0000 mgr.y (mgr.24437) 212 : cluster [DBG] pgmap v214: 365 pgs: 32 creating+peering, 333 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 53 KiB/s rd, 76 op/s 2024-02-23T00:31:47.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:45.896690+0000 osd.7 (osd.7) 87 : cluster [DBG] 95.7 scrub starts 2024-02-23T00:31:47.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:45.900146+0000 osd.7 (osd.7) 88 : cluster [DBG] 95.7 scrub ok 2024-02-23T00:31:47.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:45.960491+0000 osd.4 (osd.4) 87 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:31:47.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:45.961382+0000 osd.4 (osd.4) 88 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:31:47.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:46.390590+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:47.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:46.988191+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? 172.21.15.87:0/662284483' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi087-82019-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:47.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:46.988368+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.750+0000 7f28dbdd0700 1 -- 172.21.15.87:0/3509816476 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 msgr2=0x7f28d4103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.750+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/3509816476 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 0x7f28d4103a20 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 rx=0x7f28cc009be0 tx=0x7f28cc004d10).stop 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.750+0000 7f28dbdd0700 1 -- 172.21.15.87:0/3509816476 shutdown_connections 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.750+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/3509816476 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 0x7f28d4103a20 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.750+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/3509816476 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28d40fcbb0 0x7f28d40fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.750+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/3509816476 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28d40fd510 0x7f28d40f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.750+0000 7f28dbdd0700 1 -- 172.21.15.87:0/3509816476 >> 172.21.15.87:0/3509816476 conn(0x7f28d40f2820 msgr2=0x7f28d40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.750+0000 7f28dbdd0700 1 -- 172.21.15.87:0/3509816476 shutdown_connections 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.750+0000 7f28dbdd0700 1 -- 172.21.15.87:0/3509816476 wait complete. 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 librados: starting msgr at 2024-02-23T00:31:47.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 librados: starting objecter 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 Processor -- start 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 -- start start 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 librados: setting wanted keys 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 librados: calling monclient init 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 0x7f28d41090a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28d40fcbb0 0x7f28d4109590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28d40fd510 0x7f28d4109ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4114230 con 0x7f28d40fcbb0 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d41143a0 con 0x7f28d40fd510 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28dbdd0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d4114560 con 0x7f28d40f7400 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d9b6c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 0x7f28d41090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d936b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28d40fcbb0 0x7f28d4109590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d936b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28d40fcbb0 0x7f28d4109590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47710/0 (socket says 172.21.15.87:47710) 2024-02-23T00:31:47.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d9b6c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 0x7f28d41090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:57178/0 (socket says 172.21.15.87:57178) 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d936b700 1 -- 172.21.15.87:0/4041043604 learned_addr learned my addr 172.21.15.87:0/4041043604 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28da36d700 1 --2- 172.21.15.87:0/4041043604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28d40fd510 0x7f28d4109ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d936b700 1 -- 172.21.15.87:0/4041043604 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 msgr2=0x7f28d41090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d936b700 1 --2- 172.21.15.87:0/4041043604 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 0x7f28d41090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d936b700 1 -- 172.21.15.87:0/4041043604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28d40fd510 msgr2=0x7f28d4109ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d936b700 1 --2- 172.21.15.87:0/4041043604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28d40fd510 0x7f28d4109ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.751+0000 7f28d936b700 1 -- 172.21.15.87:0/4041043604 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d0005040 con 0x7f28d40fcbb0 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28d9b6c700 1 --2- 172.21.15.87:0/4041043604 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 0x7f28d41090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28d936b700 1 --2- 172.21.15.87:0/4041043604 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28d40fcbb0 0x7f28d4109590 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7f28d00048f0 tx=0x7f28d0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28caffd700 10 client.?.objecter ms_handle_connect 0x7f28d40fcbb0 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28caffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28caffd700 1 -- 172.21.15.87:0/4041043604 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0007b00 con 0x7f28d40fcbb0 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28cc0097c0 con 0x7f28d40fcbb0 2024-02-23T00:31:47.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28dbdd0700 10 client.26331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d4114bc0 con 0x7f28d40fcbb0 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28dbdd0700 1 librados: init done 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28caffd700 1 -- 172.21.15.87:0/4041043604 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28d000f040 con 0x7f28d40fcbb0 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.752+0000 7f28caffd700 1 -- 172.21.15.87:0/4041043604 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0014970 con 0x7f28d40fcbb0 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28caffd700 1 -- 172.21.15.87:0/4041043604 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f28d000e790 con 0x7f28d40fcbb0 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b8004f50 con 0x7f28d40fcbb0 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28caffd700 1 --2- 172.21.15.87:0/4041043604 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28c0062c90 0x7f28c0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28caffd700 1 -- 172.21.15.87:0/4041043604 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(156..156 src has 38..156) v4 ==== 9180+0+0 (secure 0 0 0) 0x7f28d007d1a0 con 0x7f28d40fcbb0 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28caffd700 10 client.26331.objecter ms_dispatch 0x7f28d4001550 osd_map(156..156 src has 38..156) v4 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28caffd700 3 client.26331.objecter handle_osd_map got epochs [156,156] > 0 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28caffd700 3 client.26331.objecter handle_osd_map decoding full epoch 156 2024-02-23T00:31:47.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28d9b6c700 1 --2- 172.21.15.87:0/4041043604 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28c0062c90 0x7f28c0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28caffd700 20 client.26331.objecter dump_active .. 0 homeless 2024-02-23T00:31:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28caffd700 10 client.26331.objecter ms_handle_connect 0x7f28c0062c90 2024-02-23T00:31:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.754+0000 7f28d9b6c700 1 --2- 172.21.15.87:0/4041043604 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28c0062c90 0x7f28c0065150 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7f28c40041d0 tx=0x7f28c4009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:47.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.759+0000 7f28caffd700 1 -- 172.21.15.87:0/4041043604 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f28d0049f60 con 0x7f28d40fcbb0 2024-02-23T00:31:47.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.936+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "2.a"} v 0) v1 -- 0x7f28b8005200 con 0x7f28d40fcbb0 2024-02-23T00:31:47.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:47.994+0000 7f28caffd700 1 -- 172.21.15.87:0/4041043604 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "2.a"}]=0 clear 2.a pg_upmap_items mapping v157) v1 ==== 116+0+0 (secure 0 0 0) 0x7f28d0050210 con 0x7f28d40fcbb0 2024-02-23T00:31:48.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.002+0000 7f28dbdd0700 10 librados: watch_flush enter 2024-02-23T00:31:48.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 10 librados: watch_flush exit 2024-02-23T00:31:48.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 20 client.26331.objecter shutdown clearing up homeless session... 2024-02-23T00:31:48.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 10 client.26331.objecter successfully canceled tick 2024-02-23T00:31:48.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28c0062c90 msgr2=0x7f28c0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:48.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/4041043604 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28c0062c90 0x7f28c0065150 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 rx=0x7f28c40041d0 tx=0x7f28c4009040).stop 2024-02-23T00:31:48.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28d40fcbb0 msgr2=0x7f28d4109590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:48.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/4041043604 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28d40fcbb0 0x7f28d4109590 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 rx=0x7f28d00048f0 tx=0x7f28d0009cb0).stop 2024-02-23T00:31:48.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 shutdown_connections 2024-02-23T00:31:48.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/4041043604 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28c0062c90 0x7f28c0065150 secure :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 rx=0x7f28c40041d0 tx=0x7f28c4009040).stop 2024-02-23T00:31:48.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/4041043604 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28d40fd510 0x7f28d4109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:48.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/4041043604 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28d40f7400 0x7f28d41090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:48.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 --2- 172.21.15.87:0/4041043604 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28d40fcbb0 0x7f28d4109590 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:48.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.003+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 >> 172.21.15.87:0/4041043604 conn(0x7f28d40f2820 msgr2=0x7f28d40f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:48.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.005+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 shutdown_connections 2024-02-23T00:31:48.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.005+0000 7f28dbdd0700 1 -- 172.21.15.87:0/4041043604 wait complete. 2024-02-23T00:31:48.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:48.007+0000 7f28dbdd0700 1 librados: shutdown 2024-02-23T00:31:48.015 INFO:teuthology.orchestra.run.smithi087.stderr:clear 2.a pg_upmap_items mapping 2024-02-23T00:31:48.208 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-23T00:31:48.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:47.391420+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:48.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:47.938123+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? 172.21.15.87:0/4041043604' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.a"}]: dispatch 2024-02-23T00:31:48.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:47.989844+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? 172.21.15.87:0/4041043604' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.a"}]': finished 2024-02-23T00:31:48.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:47.989903+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-02-23T00:31:48.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:48.000673+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.87:0/2536529825' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82011-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:48.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:48.001125+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82011-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:48.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:47.391420+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:48.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:47.938123+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? 172.21.15.87:0/4041043604' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.a"}]: dispatch 2024-02-23T00:31:48.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:47.989844+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? 172.21.15.87:0/4041043604' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.a"}]': finished 2024-02-23T00:31:48.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:47.989903+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-02-23T00:31:48.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:48.000673+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.87:0/2536529825' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82011-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:48.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:48.001125+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82011-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:47.391420+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:47.938123+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? 172.21.15.87:0/4041043604' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.a"}]: dispatch 2024-02-23T00:31:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:47.989844+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? 172.21.15.87:0/4041043604' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.a"}]': finished 2024-02-23T00:31:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:47.989903+0000 mon.a (mon.0) 1248 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-02-23T00:31:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:48.000673+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.87:0/2536529825' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82011-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:48.001125+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82011-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:48.933 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:48.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:48.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:48.939 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:49.038 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:49.047 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:31:49.066 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:49.140 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:49.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:49.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:49.245 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:49.275 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:49.296 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:49.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:49.400 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:47.571270+0000 mgr.y (mgr.24437) 213 : cluster [DBG] pgmap v217: 365 pgs: 32 unknown, 333 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:47.752952+0000 osd.0 (osd.0) 103 : cluster [DBG] 95.9 scrub starts 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:47.756443+0000 osd.0 (osd.0) 104 : cluster [DBG] 95.9 scrub ok 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:48.392198+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:48.994404+0000 mon.a (mon.0) 1250 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82011-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:48.994608+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:49.005120+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? 172.21.15.87:0/2580512708' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi087-82019-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:47.571270+0000 mgr.y (mgr.24437) 213 : cluster [DBG] pgmap v217: 365 pgs: 32 unknown, 333 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:47.752952+0000 osd.0 (osd.0) 103 : cluster [DBG] 95.9 scrub starts 2024-02-23T00:31:49.416 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:47.756443+0000 osd.0 (osd.0) 104 : cluster [DBG] 95.9 scrub ok 2024-02-23T00:31:49.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:48.392198+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:49.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:48.994404+0000 mon.a (mon.0) 1250 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82011-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:49.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:48.994608+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-02-23T00:31:49.417 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:49.005120+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? 172.21.15.87:0/2580512708' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi087-82019-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:49.428 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:49.447 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:49.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:49.551 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:49.579 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:47.571270+0000 mgr.y (mgr.24437) 213 : cluster [DBG] pgmap v217: 365 pgs: 32 unknown, 333 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-23T00:31:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:47.752952+0000 osd.0 (osd.0) 103 : cluster [DBG] 95.9 scrub starts 2024-02-23T00:31:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:47.756443+0000 osd.0 (osd.0) 104 : cluster [DBG] 95.9 scrub ok 2024-02-23T00:31:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:48.392198+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:48.994404+0000 mon.a (mon.0) 1250 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82011-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:48.994608+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-02-23T00:31:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:49.005120+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? 172.21.15.87:0/2580512708' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi087-82019-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:50.427 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:50.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:49.392959+0000 mon.c (mon.2) 135 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:50.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:49.572742+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-23T00:31:50.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:49.992505+0000 mon.a (mon.0) 1254 : audit [INF] from='client.? 172.21.15.87:0/2580512708' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi087-82019-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:50.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:49.992699+0000 mon.a (mon.0) 1255 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-23T00:31:50.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:49.992753+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-02-23T00:31:50.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:50.013746+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? 172.21.15.87:0/3480094576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82115-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:50.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:49.392959+0000 mon.c (mon.2) 135 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:50.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:49.572742+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-23T00:31:50.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:49.992505+0000 mon.a (mon.0) 1254 : audit [INF] from='client.? 172.21.15.87:0/2580512708' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi087-82019-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:50.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:49.992699+0000 mon.a (mon.0) 1255 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-23T00:31:50.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:49.992753+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-02-23T00:31:50.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:50.013746+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? 172.21.15.87:0/3480094576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82115-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:50.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:50.532 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:50.560 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:50.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:49.392959+0000 mon.c (mon.2) 135 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:50.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:49.572742+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-23T00:31:50.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:49.992505+0000 mon.a (mon.0) 1254 : audit [INF] from='client.? 172.21.15.87:0/2580512708' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi087-82019-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:50.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:49.992699+0000 mon.a (mon.0) 1255 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-23T00:31:50.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:49.992753+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-02-23T00:31:50.681 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:50.013746+0000 mon.a (mon.0) 1257 : audit [INF] from='client.? 172.21.15.87:0/3480094576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82115-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:50.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:31:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:31:51.311 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:51.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:51.315 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:51.416 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:51.445 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:51.535 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:49.571649+0000 mgr.y (mgr.24437) 214 : cluster [DBG] pgmap v220: 364 pgs: 64 unknown, 300 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:51.536 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:50.394106+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:51.536 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:50.996114+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? 172.21.15.87:0/3480094576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82115-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:51.536 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:50.996222+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-02-23T00:31:51.536 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:51.014085+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? 172.21.15.87:0/1556106598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi087-82011-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:51.536 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:49.571649+0000 mgr.y (mgr.24437) 214 : cluster [DBG] pgmap v220: 364 pgs: 64 unknown, 300 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:51.536 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:50.394106+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:51.536 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:50.996114+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? 172.21.15.87:0/3480094576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82115-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:51.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:50.996222+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-02-23T00:31:51.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:51.014085+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? 172.21.15.87:0/1556106598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi087-82011-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:51.543 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:51.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:51.648 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:51.677 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:49.571649+0000 mgr.y (mgr.24437) 214 : cluster [DBG] pgmap v220: 364 pgs: 64 unknown, 300 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:50.394106+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:50.996114+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? 172.21.15.87:0/3480094576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82115-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:50.996222+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-02-23T00:31:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:51.014085+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? 172.21.15.87:0/1556106598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi087-82011-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:51.825 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:51.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:51.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:51.929 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:51.961 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:51.987 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:51.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:52.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4061124722 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5580f6a90 msgr2=0x7fb558104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4061124722 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5580f6a90 0x7fb558104a20 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 rx=0x7fb550009be0 tx=0x7fb550004d10).stop 2024-02-23T00:31:52.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4061124722 shutdown_connections 2024-02-23T00:31:52.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4061124722 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5580f6a90 0x7fb558104a20 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4061124722 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5580fdbd0 0x7fb5580fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4061124722 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5580fe530 0x7fb5580f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4061124722 >> 172.21.15.87:0/4061124722 conn(0x7fb55806c370 msgr2=0x7fb55806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:52.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4061124722 shutdown_connections 2024-02-23T00:31:52.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4061124722 wait complete. 2024-02-23T00:31:52.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 librados: starting msgr at 2024-02-23T00:31:52.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.078+0000 7fb55ee4b700 1 librados: starting objecter 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 Processor -- start 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 -- start start 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 librados: setting wanted keys 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 librados: calling monclient init 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5580f6a90 0x7fb55810e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5580fdbd0 0x7fb55810e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5580fe530 0x7fb558108230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb558108720 con 0x7fb5580fe530 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb558108890 con 0x7fb5580fdbd0 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55ee4b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb558108a50 con 0x7fb5580f6a90 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55cbe7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5580f6a90 0x7fb55810e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55d3e8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5580fe530 0x7fb558108230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55cbe7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5580f6a90 0x7fb55810e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50730/0 (socket says 172.21.15.87:50730) 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb55cbe7700 1 -- 172.21.15.87:0/4238619221 learned_addr learned my addr 172.21.15.87:0/4238619221 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.079+0000 7fb557fff700 1 --2- 172.21.15.87:0/4238619221 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5580fdbd0 0x7fb55810e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55d3e8700 1 -- 172.21.15.87:0/4238619221 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5580f6a90 msgr2=0x7fb55810e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55d3e8700 1 --2- 172.21.15.87:0/4238619221 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5580f6a90 0x7fb55810e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55d3e8700 1 -- 172.21.15.87:0/4238619221 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5580fdbd0 msgr2=0x7fb55810e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55d3e8700 1 --2- 172.21.15.87:0/4238619221 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5580fdbd0 0x7fb55810e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55d3e8700 1 -- 172.21.15.87:0/4238619221 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb544005040 con 0x7fb5580fe530 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55d3e8700 1 --2- 172.21.15.87:0/4238619221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5580fe530 0x7fb558108230 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7fb550004e00 tx=0x7fb550004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb555ffb700 10 client.?.objecter ms_handle_connect 0x7fb5580fe530 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb555ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:52.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55000e040 con 0x7fb5580fe530 2024-02-23T00:31:52.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb550018040 con 0x7fb5580fe530 2024-02-23T00:31:52.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5500097c0 con 0x7fb5580fe530 2024-02-23T00:31:52.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550013cf0 con 0x7fb5580fe530 2024-02-23T00:31:52.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55ee4b700 10 client.26373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:52.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb558108ef0 con 0x7fb5580fe530 2024-02-23T00:31:52.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.080+0000 7fb55ee4b700 1 librados: init done 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.081+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb5581090e0 con 0x7fb5580fe530 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb550013420 con 0x7fb5580fe530 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb555ffb700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb548062c90 0x7fb548065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(161..161 src has 38..161) v4 ==== 9519+0+0 (secure 0 0 0) 0x7fb55007e3c0 con 0x7fb5580fe530 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 osd_map(161..161 src has 38..161) v4 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb555ffb700 3 client.26373.objecter handle_osd_map got epochs [161,161] > 0 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb555ffb700 3 client.26373.objecter handle_osd_map decoding full epoch 161 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55cbe7700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb548062c90 0x7fb548065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb555ffb700 20 client.26373.objecter dump_active .. 0 homeless 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v161) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb550085070 con 0x7fb5580fe530 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:52.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 1 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb5580feee0 0x7fb5580ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb55805f970 osd=0 3 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55805f970 osd=0 3 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb55805f970 osd=0 3 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 1 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb55805f970 osd=0 3 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 0 1 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55805f970 osd=0 4 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 10 client.26373.objecter _send_command 1 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55805ee80 con 0x7fb5580feee0 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb555ffb700 10 client.26373.objecter ms_handle_connect 0x7fb548062c90 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.082+0000 7fb55cbe7700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb548062c90 0x7fb548065150 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 rx=0x7fb54c004e80 tx=0x7fb54c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.083+0000 7fb557fff700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb5580feee0 0x7fb5580ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.083+0000 7fb555ffb700 10 client.26373.objecter ms_handle_connect 0x7fb5580feee0 2024-02-23T00:31:52.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.083+0000 7fb557fff700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb5580feee0 0x7fb5580ff2c0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.083+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb55805ee80 con 0x7fb5580feee0 2024-02-23T00:31:52.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.083+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:31:52.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.083+0000 7fb555ffb700 10 client.26373.objecter _finish_command 1 = system:0 2024-02-23T00:31:52.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.083+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb55805f970 osd=0 4 2024-02-23T00:31:52.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.083+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 0 1 2024-02-23T00:31:52.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:52.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:52.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 2 2024-02-23T00:31:52.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb55805f970 osd=0 3 2024-02-23T00:31:52.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55805f970 osd=0 3 2024-02-23T00:31:52.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb55805f970 osd=0 3 2024-02-23T00:31:52.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 2 2024-02-23T00:31:52.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb55805f970 osd=0 3 2024-02-23T00:31:52.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 0 2 2024-02-23T00:31:52.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55805f970 osd=0 4 2024-02-23T00:31:52.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 10 client.26373.objecter _send_command 2 2024-02-23T00:31:52.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.100+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb5581070d0 con 0x7fb5580feee0 2024-02-23T00:31:52.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb5581070d0 con 0x7fb5580feee0 2024-02-23T00:31:52.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:52.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb555ffb700 10 client.26373.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb55805f970 osd=0 4 2024-02-23T00:31:52.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 0 2 2024-02-23T00:31:52.104 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:31:52.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:52.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 3 2024-02-23T00:31:52.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6805/685354171 conn(0x7fb5580598f0 0x7fb558059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558028540 osd=1 3 2024-02-23T00:31:52.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558028540 osd=1 3 2024-02-23T00:31:52.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558028540 osd=1 3 2024-02-23T00:31:52.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 3 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb558028540 osd=1 3 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 1 3 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558028540 osd=1 4 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 10 client.26373.objecter _send_command 3 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5580286b0 con 0x7fb5580598f0 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.102+0000 7fb55cbe7700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6805/685354171 conn(0x7fb5580598f0 0x7fb558059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.103+0000 7fb555ffb700 10 client.26373.objecter ms_handle_connect 0x7fb5580598f0 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.103+0000 7fb55cbe7700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6805/685354171 conn(0x7fb5580598f0 0x7fb558059cf0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.103+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb5580286b0 con 0x7fb5580598f0 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.103+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.103+0000 7fb555ffb700 10 client.26373.objecter _finish_command 3 = system:0 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.103+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb558028540 osd=1 4 2024-02-23T00:31:52.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.103+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 1 3 2024-02-23T00:31:52.120 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:52.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 4 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558028540 osd=1 3 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558028540 osd=1 3 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558028540 osd=1 3 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 4 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb558028540 osd=1 3 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 1 4 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558028540 osd=1 4 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 10 client.26373.objecter _send_command 4 2024-02-23T00:31:52.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.122+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb558059360 con 0x7fb5580598f0 2024-02-23T00:31:52.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.123+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb558059360 con 0x7fb5580598f0 2024-02-23T00:31:52.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.123+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:52.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.123+0000 7fb555ffb700 10 client.26373.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.123+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb558028540 osd=1 4 2024-02-23T00:31:52.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.123+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 1 4 2024-02-23T00:31:52.126 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:31:52.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.123+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.123+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 5 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.123+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb55811a060 0x7fb55811c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558027b90 osd=2 3 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558027b90 osd=2 3 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558027b90 osd=2 3 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 5 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb558027b90 osd=2 3 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 2 5 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558027b90 osd=2 4 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb55ee4b700 10 client.26373.objecter _send_command 5 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb558028210 con 0x7fb55811a060 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb557fff700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb55811a060 0x7fb55811c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb555ffb700 10 client.26373.objecter ms_handle_connect 0x7fb55811a060 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.124+0000 7fb557fff700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb55811a060 0x7fb55811c450 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.125+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb558028210 con 0x7fb55811a060 2024-02-23T00:31:52.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.125+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:31:52.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.125+0000 7fb555ffb700 10 client.26373.objecter _finish_command 5 = system:0 2024-02-23T00:31:52.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.125+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb558027b90 osd=2 4 2024-02-23T00:31:52.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.125+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 2 5 2024-02-23T00:31:52.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:52.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 6 2024-02-23T00:31:52.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558027b90 osd=2 3 2024-02-23T00:31:52.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558027b90 osd=2 3 2024-02-23T00:31:52.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558027b90 osd=2 3 2024-02-23T00:31:52.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 6 2024-02-23T00:31:52.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb558027b90 osd=2 3 2024-02-23T00:31:52.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 2 6 2024-02-23T00:31:52.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558027b90 osd=2 4 2024-02-23T00:31:52.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 10 client.26373.objecter _send_command 6 2024-02-23T00:31:52.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.141+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb558027600 con 0x7fb55811a060 2024-02-23T00:31:52.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.142+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb558027600 con 0x7fb55811a060 2024-02-23T00:31:52.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.142+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:52.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.142+0000 7fb555ffb700 10 client.26373.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.142+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb558027b90 osd=2 4 2024-02-23T00:31:52.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.142+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 2 6 2024-02-23T00:31:52.145 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:31:52.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:52.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 7 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6813/128059790 conn(0x7fb558119bc0 0x7fb55811e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580279e0 osd=3 3 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580279e0 osd=3 3 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580279e0 osd=3 3 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 7 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb5580279e0 osd=3 3 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 3 7 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580279e0 osd=3 4 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 10 client.26373.objecter _send_command 7 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55811ee40 con 0x7fb558119bc0 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55d3e8700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6813/128059790 conn(0x7fb558119bc0 0x7fb55811e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb555ffb700 10 client.26373.objecter ms_handle_connect 0x7fb558119bc0 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.143+0000 7fb55d3e8700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6813/128059790 conn(0x7fb558119bc0 0x7fb55811e950 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.144+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb55811ee40 con 0x7fb558119bc0 2024-02-23T00:31:52.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.144+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:31:52.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.144+0000 7fb555ffb700 10 client.26373.objecter _finish_command 7 = system:0 2024-02-23T00:31:52.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.144+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb5580279e0 osd=3 4 2024-02-23T00:31:52.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.144+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 3 7 2024-02-23T00:31:52.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:52.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 8 2024-02-23T00:31:52.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580279e0 osd=3 3 2024-02-23T00:31:52.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580279e0 osd=3 3 2024-02-23T00:31:52.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580279e0 osd=3 3 2024-02-23T00:31:52.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 8 2024-02-23T00:31:52.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb5580279e0 osd=3 3 2024-02-23T00:31:52.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 3 8 2024-02-23T00:31:52.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580279e0 osd=3 4 2024-02-23T00:31:52.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 10 client.26373.objecter _send_command 8 2024-02-23T00:31:52.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.158+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb558119120 con 0x7fb558119bc0 2024-02-23T00:31:52.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb558119120 con 0x7fb558119bc0 2024-02-23T00:31:52.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:52.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb555ffb700 10 client.26373.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb5580279e0 osd=3 4 2024-02-23T00:31:52.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 3 8 2024-02-23T00:31:52.162 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 9 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb558119290 0x7fb5581254c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558119120 osd=4 3 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558119120 osd=4 3 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558119120 osd=4 3 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 9 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb558119120 osd=4 3 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 4 9 2024-02-23T00:31:52.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558119120 osd=4 4 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 10 client.26373.objecter _send_command 9 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5580f9370 con 0x7fb558119290 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55cbe7700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb558119290 0x7fb5581254c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb555ffb700 10 client.26373.objecter ms_handle_connect 0x7fb558119290 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.160+0000 7fb55cbe7700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb558119290 0x7fb5581254c0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.161+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb5580f9370 con 0x7fb558119290 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.161+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.161+0000 7fb555ffb700 10 client.26373.objecter _finish_command 9 = system:0 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.161+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb558119120 osd=4 4 2024-02-23T00:31:52.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.161+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 4 9 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 10 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558119120 osd=4 3 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558119120 osd=4 3 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb558119120 osd=4 3 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 10 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb558119120 osd=4 3 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 4 10 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558119120 osd=4 4 2024-02-23T00:31:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 10 client.26373.objecter _send_command 10 2024-02-23T00:31:52.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.176+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb5580f9370 con 0x7fb558119290 2024-02-23T00:31:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb5580f9370 con 0x7fb558119290 2024-02-23T00:31:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb555ffb700 10 client.26373.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb558119120 osd=4 4 2024-02-23T00:31:52.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 4 10 2024-02-23T00:31:52.181 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 11 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb558125ec0 0x7fb5581282b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580f9370 osd=5 3 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f9370 osd=5 3 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580f9370 osd=5 3 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 11 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb5580f9370 osd=5 3 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 5 11 2024-02-23T00:31:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f9370 osd=5 4 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 10 client.26373.objecter _send_command 11 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.178+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb55812afc0 con 0x7fb558125ec0 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.179+0000 7fb557fff700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb558125ec0 0x7fb5581282b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.179+0000 7fb555ffb700 10 client.26373.objecter ms_handle_connect 0x7fb558125ec0 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.179+0000 7fb557fff700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb558125ec0 0x7fb5581282b0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.180+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb55812afc0 con 0x7fb558125ec0 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.180+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.180+0000 7fb555ffb700 10 client.26373.objecter _finish_command 11 = system:0 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.180+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb5580f9370 osd=5 4 2024-02-23T00:31:52.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.180+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 5 11 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 12 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580f9370 osd=5 3 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f9370 osd=5 3 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580f9370 osd=5 3 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 12 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb5580f9370 osd=5 3 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 5 12 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f9370 osd=5 4 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 10 client.26373.objecter _send_command 12 2024-02-23T00:31:52.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.194+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb55812afc0 con 0x7fb558125ec0 2024-02-23T00:31:52.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb55812afc0 con 0x7fb558125ec0 2024-02-23T00:31:52.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:52.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb555ffb700 10 client.26373.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb5580f9370 osd=5 4 2024-02-23T00:31:52.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 5 12 2024-02-23T00:31:52.200 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:31:52.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:52.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 13 2024-02-23T00:31:52.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb558129980 0x7fb5580f74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb55812afc0 osd=6 3 2024-02-23T00:31:52.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55812afc0 osd=6 3 2024-02-23T00:31:52.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb55812afc0 osd=6 3 2024-02-23T00:31:52.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 13 2024-02-23T00:31:52.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb55812afc0 osd=6 3 2024-02-23T00:31:52.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 6 13 2024-02-23T00:31:52.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55812afc0 osd=6 4 2024-02-23T00:31:52.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 10 client.26373.objecter _send_command 13 2024-02-23T00:31:52.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5580f8c80 con 0x7fb558129980 2024-02-23T00:31:52.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55d3e8700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb558129980 0x7fb5580f74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb55d3e8700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb558129980 0x7fb5580f74c0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.196+0000 7fb555ffb700 10 client.26373.objecter ms_handle_connect 0x7fb558129980 2024-02-23T00:31:52.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.197+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb5580f8c80 con 0x7fb558129980 2024-02-23T00:31:52.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.197+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:31:52.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.197+0000 7fb555ffb700 10 client.26373.objecter _finish_command 13 = system:0 2024-02-23T00:31:52.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.197+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb55812afc0 osd=6 4 2024-02-23T00:31:52.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.197+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 6 13 2024-02-23T00:31:52.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:52.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 14 2024-02-23T00:31:52.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb55812afc0 osd=6 3 2024-02-23T00:31:52.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55812afc0 osd=6 3 2024-02-23T00:31:52.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb55812afc0 osd=6 3 2024-02-23T00:31:52.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 14 2024-02-23T00:31:52.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb55812afc0 osd=6 3 2024-02-23T00:31:52.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 6 14 2024-02-23T00:31:52.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55812afc0 osd=6 4 2024-02-23T00:31:52.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 10 client.26373.objecter _send_command 14 2024-02-23T00:31:52.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.212+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb5580f8c80 con 0x7fb558129980 2024-02-23T00:31:52.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb5580f8c80 con 0x7fb558129980 2024-02-23T00:31:52.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:52.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb555ffb700 10 client.26373.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb55812afc0 osd=6 4 2024-02-23T00:31:52.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 6 14 2024-02-23T00:31:52.216 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:31:52.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:31:52.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 15 2024-02-23T00:31:52.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6812/373787000 conn(0x7fb5580f7ec0 0x7fb5580f8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:52.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580f8c80 osd=7 3 2024-02-23T00:31:52.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f8c80 osd=7 3 2024-02-23T00:31:52.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580f8c80 osd=7 3 2024-02-23T00:31:52.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 15 2024-02-23T00:31:52.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb5580f8c80 osd=7 3 2024-02-23T00:31:52.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 7 15 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f8c80 osd=7 4 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 10 client.26373.objecter _send_command 15 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5581295c0 con 0x7fb5580f7ec0 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.214+0000 7fb55cbe7700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6812/373787000 conn(0x7fb5580f7ec0 0x7fb5580f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.215+0000 7fb555ffb700 10 client.26373.objecter ms_handle_connect 0x7fb5580f7ec0 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.215+0000 7fb55cbe7700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6812/373787000 conn(0x7fb5580f7ec0 0x7fb5580f8300 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.215+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb5581295c0 con 0x7fb5580f7ec0 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.215+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.215+0000 7fb555ffb700 10 client.26373.objecter _finish_command 15 = system:0 2024-02-23T00:31:52.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.215+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb5580f8c80 osd=7 4 2024-02-23T00:31:52.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.215+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 7 15 2024-02-23T00:31:52.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 10 client.26373.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:31:52.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign -1 16 2024-02-23T00:31:52.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580f8c80 osd=7 3 2024-02-23T00:31:52.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f8c80 osd=7 3 2024-02-23T00:31:52.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 20 client.26373.objecter _get_session s=0x7fb5580f8c80 osd=7 3 2024-02-23T00:31:52.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_remove -1 16 2024-02-23T00:31:52.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 20 client.26373.objecter get_session s=0x7fb5580f8c80 osd=7 3 2024-02-23T00:31:52.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 15 client.26373.objecter _session_command_op_assign 7 16 2024-02-23T00:31:52.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f8c80 osd=7 4 2024-02-23T00:31:52.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 10 client.26373.objecter _send_command 16 2024-02-23T00:31:52.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.230+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb5581295c0 con 0x7fb5580f7ec0 2024-02-23T00:31:52.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb555ffb700 1 -- 172.21.15.87:0/4238619221 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb5581295c0 con 0x7fb5580f7ec0 2024-02-23T00:31:52.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb555ffb700 10 client.26373.objecter ms_dispatch 0x7fb558001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:31:52.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb555ffb700 10 client.26373.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb555ffb700 20 client.26373.objecter put_session s=0x7fb5580f8c80 osd=7 4 2024-02-23T00:31:52.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb555ffb700 15 client.26373.objecter _session_command_op_remove 7 16 2024-02-23T00:31:52.235 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:31:52.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 10 librados: watch_flush enter 2024-02-23T00:31:52.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 10 librados: watch_flush exit 2024-02-23T00:31:52.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 10 client.26373.objecter close_session for osd.0 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb5580feee0 msgr2=0x7fb5580ff2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb5580feee0 0x7fb5580ff2c0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55805f970 osd=0 1 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 10 client.26373.objecter close_session for osd.1 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6805/685354171 conn(0x7fb5580598f0 msgr2=0x7fb558059cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6805/685354171 conn(0x7fb5580598f0 0x7fb558059cf0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558028540 osd=1 1 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 10 client.26373.objecter close_session for osd.2 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb55811a060 msgr2=0x7fb55811c450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb55811a060 0x7fb55811c450 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558027b90 osd=2 1 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 10 client.26373.objecter close_session for osd.3 2024-02-23T00:31:52.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55d3e8700 1 -- 172.21.15.87:0/4238619221 reap_dead start 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6813/128059790 conn(0x7fb558119bc0 msgr2=0x7fb55811e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6813/128059790 conn(0x7fb558119bc0 0x7fb55811e950 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580279e0 osd=3 1 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 10 client.26373.objecter close_session for osd.4 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.232+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb558119290 msgr2=0x7fb5581254c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb558119290 0x7fb5581254c0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb558119120 osd=4 1 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 10 client.26373.objecter close_session for osd.5 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb558125ec0 msgr2=0x7fb5581282b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb558125ec0 0x7fb5581282b0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f9370 osd=5 1 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 10 client.26373.objecter close_session for osd.6 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb558129980 msgr2=0x7fb5580f74c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb558129980 0x7fb5580f74c0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb55812afc0 osd=6 1 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 10 client.26373.objecter close_session for osd.7 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6812/373787000 conn(0x7fb5580f7ec0 msgr2=0x7fb5580f8300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.171:6812/373787000 conn(0x7fb5580f7ec0 0x7fb5580f8300 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 20 client.26373.objecter put_session s=0x7fb5580f8c80 osd=7 1 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 20 client.26373.objecter shutdown clearing up homeless session... 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 10 client.26373.objecter successfully canceled tick 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55d3e8700 1 -- 172.21.15.87:0/4238619221 reap_dead start 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb548062c90 msgr2=0x7fb548065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb548062c90 0x7fb548065150 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 rx=0x7fb54c004e80 tx=0x7fb54c00b040).stop 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5580fe530 msgr2=0x7fb558108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5580fe530 0x7fb558108230 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 rx=0x7fb550004e00 tx=0x7fb550004d10).stop 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 shutdown_connections 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5580fe530 0x7fb558108230 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 --2- 172.21.15.87:0/4238619221 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb548062c90 0x7fb548065150 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 >> 172.21.15.87:0/4238619221 conn(0x7fb55806c370 msgr2=0x7fb5580f5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 shutdown_connections 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 -- 172.21.15.87:0/4238619221 wait complete. 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:52.233+0000 7fb55ee4b700 1 librados: shutdown 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.240 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:31:52.269 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:52.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:52.373 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:52.390 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:31:52.391 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:31:52.391 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:31:52.391 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:31:52.391 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.391 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24712.0:10 7.a 7:54825363:test-rados-api-smithi087-82115-4::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-23T00:31:52.391 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:30:57.476130+0000", 2024-02-23T00:31:52.391 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 54.906490624, 2024-02-23T00:31:52.391 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030623026000000001, 2024-02-23T00:31:52.392 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.392 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.392 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.392 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24712", 2024-02-23T00:31:52.392 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2285581802", 2024-02-23T00:31:52.392 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 10 2024-02-23T00:31:52.392 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.392 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.393 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.393 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.393 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.476130+0000", 2024-02-23T00:31:52.393 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.393 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.393 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.393 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.393 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.476130+0000", 2024-02-23T00:31:52.394 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:31:52.394 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.394 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.394 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.394 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.476128+0000", 2024-02-23T00:31:52.394 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.047e-05 2024-02-23T00:31:52.394 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.394 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.394 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.395 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.476138+0000", 2024-02-23T00:31:52.395 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0051071739999999999 2024-02-23T00:31:52.395 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.395 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.395 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.395 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.481245+0000", 2024-02-23T00:31:52.395 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1459e-05 2024-02-23T00:31:52.395 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.395 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.481257+0000", 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0026739429999999998 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.483931+0000", 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1996999999999993e-05 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.484023+0000", 2024-02-23T00:31:52.396 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000192447 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 2,5", 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.484215+0000", 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012288584999999999 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.496504+0000", 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0049043990000000003 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:31:52.397 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.501408+0000", 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0052414000000000002 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.506650+0000", 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2675e-05 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.506702+0000", 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0937999999999999e-05 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.398 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.506753+0000", 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030623026000000001 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.25145.0:2 55.15 55:ad551702:::test:head [create,write 0~4,writefull 0~2] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:30:59.533100+0000", 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 52.849521154000001, 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017801339999999999, 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.399 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.25145", 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1241496180", 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.533100+0000", 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.533100+0000", 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:31:52.400 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.533098+0000", 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6370000000000002e-06 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.533105+0000", 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2090000000000005e-06 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.401 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.533113+0000", 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3209999999999999e-06 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.533120+0000", 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3622000000000003e-05 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.533174+0000", 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4879999999999998e-06 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.402 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.533179+0000", 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0133244 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.546504+0000", 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012688700000000001 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.546631+0000", 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00017504900000000001 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.403 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 2,6", 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.546806+0000", 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0015590199999999999 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.548365+0000", 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016248289999999999 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.549990+0000", 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00082454200000000005 2024-02-23T00:31:52.404 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.550814+0000", 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7460999999999998e-05 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.550862+0000", 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9509999999999999e-05 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:59.550901+0000", 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017801339999999999 2024-02-23T00:31:52.405 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24712.0:19 7.2 7:40054bab:test-rados-api-smithi087-82115-6::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e111)", 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:01.502859+0000", 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 50.879762263000003, 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040250123999999998, 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.406 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24712", 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2285581802", 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 19 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502859+0000", 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502859+0000", 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.407 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502857+0000", 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.7179999999999996e-06 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502865+0000", 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0523000000000001e-05 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.408 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502875+0000", 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8630000000000002e-06 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502882+0000", 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00075966 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.503642+0000", 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2692e-05 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.409 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.503734+0000", 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021787099999999999 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.503952+0000", 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0062092650000000003 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.510162+0000", 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.022996850999999999 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.410 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.533158+0000", 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0098446820000000004 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.543003+0000", 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5009000000000001e-05 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.543058+0000", 2024-02-23T00:31:52.411 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0720999999999998e-05 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.543109+0000", 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040250123999999998 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24712.0:21 7.2 7:40054bab:test-rados-api-smithi087-82115-6::foo:head [create,delete] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.518065+0000", 2024-02-23T00:31:52.412 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 49.864555977000002, 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.083348715000000004, 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24712", 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2285581802", 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 21 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518065+0000", 2024-02-23T00:31:52.413 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.414 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.414 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.414 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518065+0000", 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518063+0000", 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5610000000000004e-06 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.415 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518070+0000", 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0468e-05 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518080+0000", 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8329999999999996e-06 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518087+0000", 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0055818969999999997 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.416 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.523669+0000", 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1359999999999999e-06 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.523676+0000", 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0062083249999999998 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.529885+0000", 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3505999999999995e-05 2024-02-23T00:31:52.417 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.529978+0000", 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00020373200000000001 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.530182+0000", 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010326180000000001 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.540508+0000", 2024-02-23T00:31:52.418 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.052153245000000001 2024-02-23T00:31:52.419 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.419 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.419 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.419 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.592661+0000", 2024-02-23T00:31:52.419 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0086538880000000002 2024-02-23T00:31:52.419 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.419 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.420 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.420 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.601315+0000", 2024-02-23T00:31:52.420 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4023000000000001e-05 2024-02-23T00:31:52.420 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.420 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.601369+0000", 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4456000000000003e-05 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.601414+0000", 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.083348715000000004 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.421 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.25166.0:2 59.6 e112/111 59:602f83fe:::foo:head v 112'1, mlcod=0'0)", 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.593159+0000", 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 49.789461559000003, 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020005182999999999, 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593159+0000", 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593159+0000", 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593159+0000", 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8780000000000004e-06 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.423 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.424 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.424 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593168+0000", 2024-02-23T00:31:52.424 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7429999999999999e-06 2024-02-23T00:31:52.424 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.424 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.424 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.424 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593174+0000", 2024-02-23T00:31:52.424 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7609999999999998e-06 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593178+0000", 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7953000000000005e-05 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593246+0000", 2024-02-23T00:31:52.425 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4884e-05 2024-02-23T00:31:52.426 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.426 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.426 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.426 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593261+0000", 2024-02-23T00:31:52.426 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019871346000000002 2024-02-23T00:31:52.426 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.426 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.426 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.427 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.613133+0000", 2024-02-23T00:31:52.427 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1909000000000002e-05 2024-02-23T00:31:52.427 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.427 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.427 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.427 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.613165+0000", 2024-02-23T00:31:52.427 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020005182999999999 2024-02-23T00:31:52.428 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.428 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.428 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.428 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.428 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.429 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.5.0:320 56.2 e113/110 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head v 113'1, mlcod=0'0)", 2024-02-23T00:31:52.429 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.596901+0000", 2024-02-23T00:31:52.429 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 48.785720208000001, 2024-02-23T00:31:52.429 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028157662, 2024-02-23T00:31:52.429 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.429 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.429 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.430 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.430 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.430 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596901+0000", 2024-02-23T00:31:52.430 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.430 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.430 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.431 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.431 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596901+0000", 2024-02-23T00:31:52.431 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:31:52.431 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.431 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.431 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.431 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596900+0000", 2024-02-23T00:31:52.431 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3729999999999998e-06 2024-02-23T00:31:52.431 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.432 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.432 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.432 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596907+0000", 2024-02-23T00:31:52.432 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7179999999999999e-06 2024-02-23T00:31:52.432 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.432 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.433 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.433 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596913+0000", 2024-02-23T00:31:52.433 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0419999999999997e-06 2024-02-23T00:31:52.433 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.433 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.433 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.433 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596920+0000", 2024-02-23T00:31:52.434 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8403999999999999e-05 2024-02-23T00:31:52.434 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.434 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.434 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.434 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596969+0000", 2024-02-23T00:31:52.434 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1588000000000001e-05 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596990+0000", 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028040088000000001 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.625030+0000", 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7909999999999999e-05 2024-02-23T00:31:52.435 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.625058+0000", 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028157662 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15147.0:8 56.2 e113/110 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head v 113'2, mlcod=0'0)", 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.596928+0000", 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 48.785692740000002, 2024-02-23T00:31:52.436 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028148210999999999, 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596928+0000", 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596928+0000", 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.437 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596928+0000", 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4100000000000008e-06 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596936+0000", 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9109999999999998e-06 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596942+0000", 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2209999999999999e-06 2024-02-23T00:31:52.438 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596946+0000", 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00096380100000000002 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.597910+0000", 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9411000000000001e-05 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.597930+0000", 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027132234000000002 2024-02-23T00:31:52.439 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.625062+0000", 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4535999999999999e-05 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.625076+0000", 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028148210999999999 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.440 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26218.0:2 62.15 62:ad551702:::test:head [create,write-same] snapc 0=[] ondisk+write+known_if_redirected e115)", 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:06.197041+0000", 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 46.185580023999997, 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.135993164, 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26218", 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2330528609", 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.441 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197041+0000", 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197041+0000", 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197038+0000", 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3602e-05 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.442 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197052+0000", 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00051941599999999997 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197571+0000", 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1411e-05 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197583+0000", 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015026843 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.443 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.212610+0000", 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8179999999999998e-06 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.212616+0000", 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074955025999999994 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.287571+0000", 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9249000000000001e-05 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.444 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.287641+0000", 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001974618 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 2,7", 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.289615+0000", 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020824579999999999 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.445 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.446 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.446 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.310440+0000", 2024-02-23T00:31:52.446 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00060312300000000001 2024-02-23T00:31:52.446 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.446 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.446 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:31:52.446 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.311043+0000", 2024-02-23T00:31:52.446 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021897838999999999 2024-02-23T00:31:52.447 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.447 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.447 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.447 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.332941+0000", 2024-02-23T00:31:52.447 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2295e-05 2024-02-23T00:31:52.447 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.447 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.447 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.447 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.332993+0000", 2024-02-23T00:31:52.448 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0945999999999997e-05 2024-02-23T00:31:52.448 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.448 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.448 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.448 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.333034+0000", 2024-02-23T00:31:52.448 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.135993164 2024-02-23T00:31:52.448 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.448 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.448 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.449 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.449 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.449 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:4 66.11 66:8d290297:test-rados-api-smithi087-82091-8::foo:head [writefull 0~5] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-23T00:31:52.449 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:08.056347+0000", 2024-02-23T00:31:52.449 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 44.326273872000002, 2024-02-23T00:31:52.449 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015103465999999999, 2024-02-23T00:31:52.449 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.449 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.450 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.450 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:31:52.450 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:31:52.450 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 4 2024-02-23T00:31:52.450 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.450 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.450 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.450 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.450 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.056347+0000", 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.056347+0000", 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.056345+0000", 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8125e-05 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.451 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.056363+0000", 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0720000000000001e-05 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.056374+0000", 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3989999999999998e-06 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.056381+0000", 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0026733619999999999 2024-02-23T00:31:52.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.059055+0000", 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3909999999999998e-06 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.059063+0000", 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010443427 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.069507+0000", 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000130825 2024-02-23T00:31:52.453 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.069637+0000", 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000153117 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 2,4", 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.069790+0000", 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00048223999999999999 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.070273+0000", 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000199903 2024-02-23T00:31:52.454 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.070473+0000", 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00090470999999999998 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.071377+0000", 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1693000000000001e-05 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.071429+0000", 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1501999999999998e-05 2024-02-23T00:31:52.455 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.071451+0000", 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015103465999999999 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15147.0:69 56.1 e118/110 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head v 118'14, mlcod=113'11)", 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.954974+0000", 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 42.427647116000003, 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023167466000000001, 2024-02-23T00:31:52.456 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954974+0000", 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954974+0000", 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.457 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954973+0000", 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2839999999999997e-06 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954979+0000", 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.463e-06 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954984+0000", 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8449999999999996e-06 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.458 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954988+0000", 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0044694829999999998 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.959457+0000", 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7171000000000001e-05 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.959485+0000", 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018617076 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.459 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.978102+0000", 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9558000000000003e-05 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.978141+0000", 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023167466000000001 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:5 66.11 66:8d290297:test-rados-api-smithi087-82091-8::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e120)", 2024-02-23T00:31:52.460 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.184196+0000", 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 41.198425188000002, 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.078409890999999995, 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 5 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184196+0000", 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184196+0000", 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999971 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184193+0000", 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4197e-05 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184207+0000", 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1573000000000002e-05 2024-02-23T00:31:52.462 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184229+0000", 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1409999999999994e-06 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184238+0000", 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014631701 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.198870+0000", 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5130000000000002e-06 2024-02-23T00:31:52.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.198879+0000", 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019356801999999999 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.218236+0000", 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5295000000000002e-05 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.218321+0000", 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044284507000000001 2024-02-23T00:31:52.464 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.262606+0000", 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.078409890999999995 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15147.0:75 56.1 e122/110 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head v 122'16, mlcod=118'14)", 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:13.000201+0000", 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 39.382419521000003, 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017591982999999999, 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.465 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.000201+0000", 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.000201+0000", 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.000201+0000", 2024-02-23T00:31:52.466 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5080000000000005e-06 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.000210+0000", 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2910000000000002e-06 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.000217+0000", 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8080000000000005e-06 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.000225+0000", 2024-02-23T00:31:52.467 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0010643969999999999 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.001289+0000", 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2946e-05 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.001312+0000", 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016441311 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.017753+0000", 2024-02-23T00:31:52.468 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9986e-05 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.017793+0000", 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017591982999999999 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27712.0:57 67.3 e138/117 67:c06d7414:test-rados-api-smithi087-82115-12::foo:head v 138'1, mlcod=0'0)", 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:29.022463+0000", 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 23.360157971, 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020343938999999998, 2024-02-23T00:31:52.469 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022463+0000", 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022463+0000", 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.470 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022462+0000", 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4499e-05 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022477+0000", 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5664000000000001e-05 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022492+0000", 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3310000000000004e-06 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.471 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022501+0000", 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7853000000000002e-05 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022568+0000", 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1843000000000001e-05 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022590+0000", 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020078691999999999 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.472 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.042669+0000", 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000137921 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.042807+0000", 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020343938999999998 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:256 86.e 86:70f8162d:test-rados-api-smithi087-82091-12::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-23T00:31:52.473 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:29.038511+0000", 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 23.344109840000002, 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.055373620999999998, 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 256 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038511+0000", 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038511+0000", 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038509+0000", 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4069999999999993e-06 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038518+0000", 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1795e-05 2024-02-23T00:31:52.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038529+0000", 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9810000000000003e-06 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038537+0000", 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6368000000000001e-05 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038584+0000", 2024-02-23T00:31:52.476 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.959e-06 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038590+0000", 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012529669 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.051119+0000", 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021244600000000001 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.051332+0000", 2024-02-23T00:31:52.477 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.039488665999999999 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.090821+0000", 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5575000000000007e-05 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for rw locks", 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.090906+0000", 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3004000000000006e-05 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.090999+0000", 2024-02-23T00:31:52.478 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9804e-05 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.091039+0000", 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0028457880000000001 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.093885+0000", 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.055373620999999998 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.479 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.41845.0:2 90.d e138/137 90:b3610dda:::oid_RoundTripPP2_:head v 138'1, mlcod=0'0)", 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:29.056696+0000", 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 23.325925350999999, 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017210487999999999, 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.056696+0000", 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.480 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.056696+0000", 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.056695+0000", 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8240000000000002e-06 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.056702+0000", 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.495e-06 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.056706+0000", 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.775e-06 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.056710+0000", 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020481890000000002 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.058759+0000", 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0979999999999999e-05 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.482 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.058780+0000", 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015093702000000001 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.073873+0000", 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.2752000000000001e-05 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.073906+0000", 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017210487999999999 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27712.0:59 67.3 e139/117 67:c06d7414:test-rados-api-smithi087-82115-12::foo:head v 139'3, mlcod=0'0)", 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.041654+0000", 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 22.340966697999999, 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016501597, 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.041654+0000", 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.041654+0000", 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.041654+0000", 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2900000000000008e-06 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.041663+0000", 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6359999999999997e-06 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.485 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.041670+0000", 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.7260000000000002e-06 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.041678+0000", 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0013193460000000001 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.042997+0000", 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6027000000000001e-05 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.486 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.043013+0000", 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015100451000000001 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.058113+0000", 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2361999999999999e-05 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.058156+0000", 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016501597 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.487 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27712.0:63 67.d 67:b74fee6d:test-rados-api-smithi087-82115-13::foo:head [write 0~128] snapc 16=[16] ondisk+write+known_if_redirected e140)", 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:31.046764+0000", 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 21.335856855999999, 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021316562000000001, 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27712", 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2607672493", 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 63 2024-02-23T00:31:52.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046764+0000", 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046764+0000", 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046762+0000", 2024-02-23T00:31:52.489 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5170000000000005e-06 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046772+0000", 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1219e-05 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046783+0000", 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4849999999999992e-06 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046791+0000", 2024-02-23T00:31:52.490 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000425838 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.047217+0000", 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0990000000000001e-06 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.047224+0000", 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011776207 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.059000+0000", 2024-02-23T00:31:52.491 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012097800000000001 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.059121+0000", 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000135659 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 6,7", 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.059257+0000", 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.007406035 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.066663+0000", 2024-02-23T00:31:52.492 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1823999999999999e-05 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.066695+0000", 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00128031 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.067975+0000", 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8869000000000002e-05 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.068034+0000", 2024-02-23T00:31:52.493 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6480000000000002e-05 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.068081+0000", 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021316562000000001 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.36707.0:2 96.18 96:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:34.989654+0000", 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 17.392966424000001, 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.038403145, 2024-02-23T00:31:52.494 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.36707", 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1873510578", 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989654+0000", 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.495 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989654+0000", 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989652+0000", 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1190000000000003e-06 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989661+0000", 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5359999999999999e-05 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.496 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989677+0000", 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0920000000000004e-06 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989686+0000", 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8553e-05 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989744+0000", 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.653e-06 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.497 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989753+0000", 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025073980999999999 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.014827+0000", 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000121211 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.014948+0000", 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015277699999999999 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.498 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.015101+0000", 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011323990000000001 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.016233+0000", 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0086185819999999996 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024852+0000", 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0031188610000000001 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.027971+0000", 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7979000000000003e-05 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.028019+0000", 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8761999999999999e-05 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.028058+0000", 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.038403145 2024-02-23T00:31:52.500 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24815.0:1044 19.0 19:00000000::::head [pgnls start_epoch 98] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e98)", 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:37.994463+0000", 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 14.388157464000001, 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015834966999999998, 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24815", 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3778460189", 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1044 2024-02-23T00:31:52.501 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.994463+0000", 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.994463+0000", 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.502 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.994462+0000", 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2499999999999995e-06 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.994471+0000", 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.594e-06 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.994480+0000", 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5480000000000004e-06 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.503 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.994488+0000", 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015656298999999999 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.010144+0000", 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1100999999999998e-05 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.010185+0000", 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000113101 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.504 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.010298+0000", 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015834966999999998 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24815.0:1045 19.10 19:08000000::::head [pgnls start_epoch 98] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_overlay+known_if_redirected e147)", 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:38.022061+0000", 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 14.360559976999999, 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.984081753, 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:31:52.505 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24815", 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3778460189", 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1045 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022061+0000", 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022061+0000", 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.506 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022059+0000", 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6950000000000003e-06 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022066+0000", 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3033999999999999e-05 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022079+0000", 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8210000000000001e-06 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.507 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022084+0000", 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00033546699999999999 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022419+0000", 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.7959999999999996e-06 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.508 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:31:52.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022427+0000", 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.983542099 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.005969+0000", 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5502e-05 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.006005+0000", 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013795699999999999 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:31:52.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.006143+0000", 2024-02-23T00:31:52.514 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.984081753 2024-02-23T00:31:52.514 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.514 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.514 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.514 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:31:52.514 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:31:52.514 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:31:52.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:50.679223+0000 mgr.y (mgr.24437) 215 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:52.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:51.394813+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:52.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:51.981472+0000 mon.a (mon.0) 1261 : audit [INF] from='client.? 172.21.15.87:0/1556106598' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi087-82011-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:52.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:51.981651+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-02-23T00:31:52.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:51.993734+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.87:0/2835242559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi087-82019-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:52.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:51.995500+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi087-82019-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:52.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:52.083622+0000 mon.a (mon.0) 1264 : audit [DBG] from='client.? 172.21.15.87:0/4238619221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:52.516 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:52] "GET /metrics HTTP/1.1" 200 196372 "" "Prometheus/2.33.4" 2024-02-23T00:31:52.516 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:50.679223+0000 mgr.y (mgr.24437) 215 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:52.516 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:51.394813+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:52.516 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:51.981472+0000 mon.a (mon.0) 1261 : audit [INF] from='client.? 172.21.15.87:0/1556106598' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi087-82011-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:52.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:51.981651+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-02-23T00:31:52.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:51.993734+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.87:0/2835242559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi087-82019-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:52.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:51.995500+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi087-82019-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:52.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:52.083622+0000 mon.a (mon.0) 1264 : audit [DBG] from='client.? 172.21.15.87:0/4238619221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:52.517 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:52.517 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:52.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:52.618 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:52.646 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:52.666 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:52.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:50.679223+0000 mgr.y (mgr.24437) 215 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:31:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:51.394813+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:51.981472+0000 mon.a (mon.0) 1261 : audit [INF] from='client.? 172.21.15.87:0/1556106598' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi087-82011-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:51.981651+0000 mon.a (mon.0) 1262 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-02-23T00:31:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:51.993734+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.87:0/2835242559' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi087-82019-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:51.995500+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi087-82019-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:52.083622+0000 mon.a (mon.0) 1264 : audit [DBG] from='client.? 172.21.15.87:0/4238619221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:31:52.771 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:31:52.799 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:52.818 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:31:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:52.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:31:52.921 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:52.948 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:53.097 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:53.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:53.201 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:53.231 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:53.332 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:53.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:53.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:53.436 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:53.462 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:51.572072+0000 mgr.y (mgr.24437) 216 : cluster [DBG] pgmap v223: 364 pgs: 32 creating+peering, 2 peering, 32 unknown, 298 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:52.396046+0000 mon.c (mon.2) 139 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:52.621802+0000 mon.a (mon.0) 1265 : audit [DBG] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-23T00:31:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:52.622415+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:52.980172+0000 mon.a (mon.0) 1267 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi087-82019-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:52.980415+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:52.980472+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-02-23T00:31:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:53.000905+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app2"}]: dispatch 2024-02-23T00:31:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:53.003677+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:51.572072+0000 mgr.y (mgr.24437) 216 : cluster [DBG] pgmap v223: 364 pgs: 32 creating+peering, 2 peering, 32 unknown, 298 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:52.396046+0000 mon.c (mon.2) 139 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:52.621802+0000 mon.a (mon.0) 1265 : audit [DBG] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-23T00:31:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:52.622415+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:52.980172+0000 mon.a (mon.0) 1267 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi087-82019-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:52.980415+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:52.980472+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-02-23T00:31:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:53.000905+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app2"}]: dispatch 2024-02-23T00:31:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:53.003677+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:51.572072+0000 mgr.y (mgr.24437) 216 : cluster [DBG] pgmap v223: 364 pgs: 32 creating+peering, 2 peering, 32 unknown, 298 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:31:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:52.396046+0000 mon.c (mon.2) 139 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:52.621802+0000 mon.a (mon.0) 1265 : audit [DBG] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-23T00:31:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:52.622415+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:52.980172+0000 mon.a (mon.0) 1267 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi087-82019-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:52.980415+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:52.980472+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-02-23T00:31:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:53.000905+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app2"}]: dispatch 2024-02-23T00:31:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:53.003677+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:53.860 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:31:53.860 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:31:53.860 INFO:tasks.thrashosds.thrasher:fixing pg num pool .rgw.root 2024-02-23T00:31:53.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:31:54.103 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-23T00:31:54.110 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:54.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:54.130 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:31:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:31:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:31:54.214 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:54.240 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:53.397052+0000 mon.c (mon.2) 140 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:53.573615+0000 mon.a (mon.0) 1272 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:53.985006+0000 mon.a (mon.0) 1273 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:53.985154+0000 mon.a (mon.0) 1274 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:53.985213+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:53.998853+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.000634+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.011783+0000 mon.c (mon.2) 141 : audit [INF] from='client.? 172.21.15.87:0/2136756363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi087-82011-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.012907+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi087-82011-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.050770+0000 mon.c (mon.2) 142 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.285668+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.286246+0000 mon.c (mon.2) 143 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:53.397052+0000 mon.c (mon.2) 140 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:54.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:53.573615+0000 mon.a (mon.0) 1272 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-23T00:31:54.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:53.985006+0000 mon.a (mon.0) 1273 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:54.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:53.985154+0000 mon.a (mon.0) 1274 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-23T00:31:54.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:53.985213+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-02-23T00:31:54.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:53.998853+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:31:54.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.000634+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:31:54.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.011783+0000 mon.c (mon.2) 141 : audit [INF] from='client.? 172.21.15.87:0/2136756363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi087-82011-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.012907+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi087-82011-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.050770+0000 mon.c (mon.2) 142 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.285668+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.286246+0000 mon.c (mon.2) 143 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.286520+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.286909+0000 mon.c (mon.2) 144 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.287138+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.286520+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.286909+0000 mon.c (mon.2) 144 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:54.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.287138+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:54.375 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:31:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:53.397052+0000 mon.c (mon.2) 140 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:53.573615+0000 mon.a (mon.0) 1272 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-23T00:31:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:53.985006+0000 mon.a (mon.0) 1273 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi087-82083-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:53.985154+0000 mon.a (mon.0) 1274 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-23T00:31:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:53.985213+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-02-23T00:31:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:53.998853+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:31:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.000634+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-23T00:31:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.011783+0000 mon.c (mon.2) 141 : audit [INF] from='client.? 172.21.15.87:0/2136756363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi087-82011-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.012907+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi087-82011-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.050770+0000 mon.c (mon.2) 142 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.285668+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.286246+0000 mon.c (mon.2) 143 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.286520+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.286909+0000 mon.c (mon.2) 144 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.287138+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:54.590 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:54.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:54.622 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:54.694 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:54.724 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:54.742 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:54.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:54.846 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:31:54.873 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:55.118 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:31:55.223 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:31:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:55.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:31:55.327 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:55.356 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:53.572575+0000 mgr.y (mgr.24437) 217 : cluster [DBG] pgmap v226: 363 pgs: 32 creating+peering, 1 peering, 32 unknown, 298 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.397777+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.639573+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.985583+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.985754+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi087-82011-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.985899+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:54.985983+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.989996+0000 mon.a (mon.0) 1287 : audit [INF] from='client.? 172.21.15.87:0/2638609709' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.991720+0000 mon.c (mon.2) 146 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi087-82115-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.991939+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-23T00:31:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:54.992374+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi087-82115-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:53.572575+0000 mgr.y (mgr.24437) 217 : cluster [DBG] pgmap v226: 363 pgs: 32 creating+peering, 1 peering, 32 unknown, 298 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.397777+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.639573+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.985583+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.985754+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi087-82011-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.985899+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:54.985983+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.989996+0000 mon.a (mon.0) 1287 : audit [INF] from='client.? 172.21.15.87:0/2638609709' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.991720+0000 mon.c (mon.2) 146 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi087-82115-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.991939+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-23T00:31:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:54.992374+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi087-82115-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:53.572575+0000 mgr.y (mgr.24437) 217 : cluster [DBG] pgmap v226: 363 pgs: 32 creating+peering, 1 peering, 32 unknown, 298 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:31:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.397777+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.639573+0000 mon.a (mon.0) 1282 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:31:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.985583+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-23T00:31:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.985754+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi087-82011-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.985899+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:31:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:54.985983+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-02-23T00:31:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.989996+0000 mon.a (mon.0) 1287 : audit [INF] from='client.? 172.21.15.87:0/2638609709' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.991720+0000 mon.c (mon.2) 146 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi087-82115-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.991939+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-23T00:31:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:54.992374+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi087-82115-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:31:55.933 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:55.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:56.037 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:56.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.055+0000 7f6bb69ba700 1 -- 172.21.15.87:0/2860117493 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6bb00f6a90 msgr2=0x7f6bb0104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:56.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.055+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/2860117493 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6bb00f6a90 0x7f6bb0104a20 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7f6ba8009be0 tx=0x7f6ba8004d10).stop 2024-02-23T00:31:56.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.055+0000 7f6bb69ba700 1 -- 172.21.15.87:0/2860117493 shutdown_connections 2024-02-23T00:31:56.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.055+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/2860117493 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6bb00f6a90 0x7f6bb0104a20 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:56.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.055+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/2860117493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6bb00fdbd0 0x7f6bb00fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:56.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.055+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/2860117493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6bb00fe530 0x7f6bb00f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:56.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.055+0000 7f6bb69ba700 1 -- 172.21.15.87:0/2860117493 >> 172.21.15.87:0/2860117493 conn(0x7f6bb006c370 msgr2=0x7f6bb006c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:56.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.055+0000 7f6bb69ba700 1 -- 172.21.15.87:0/2860117493 shutdown_connections 2024-02-23T00:31:56.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.056+0000 7f6bb69ba700 1 -- 172.21.15.87:0/2860117493 wait complete. 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.056+0000 7f6bb69ba700 1 librados: starting msgr at 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.056+0000 7f6bb69ba700 1 librados: starting objecter 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.056+0000 7f6bb69ba700 1 Processor -- start 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.056+0000 7f6bb69ba700 1 -- start start 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.056+0000 7f6bb69ba700 1 librados: setting wanted keys 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.056+0000 7f6bb69ba700 1 librados: calling monclient init 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb69ba700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6bb00f6a90 0x7f6bb010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb69ba700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6bb00fdbd0 0x7f6bb010e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb69ba700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6bb00fe530 0x7f6bb01081a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb69ba700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb0108690 con 0x7f6bb00f6a90 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb69ba700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb0108800 con 0x7f6bb00fdbd0 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb69ba700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb01089c0 con 0x7f6bb00fe530 2024-02-23T00:31:56.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6baffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6bb00f6a90 0x7f6bb010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb4f57700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6bb00fe530 0x7f6bb01081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6baffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6bb00f6a90 0x7f6bb010e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45216/0 (socket says 172.21.15.87:45216) 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6baffff700 1 -- 172.21.15.87:0/505606090 learned_addr learned my addr 172.21.15.87:0/505606090 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6baf7fe700 1 --2- 172.21.15.87:0/505606090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6bb00fdbd0 0x7f6bb010e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb4f57700 1 -- 172.21.15.87:0/505606090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6bb00fdbd0 msgr2=0x7f6bb010e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb4f57700 1 --2- 172.21.15.87:0/505606090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6bb00fdbd0 0x7f6bb010e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb4f57700 1 -- 172.21.15.87:0/505606090 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6bb00f6a90 msgr2=0x7f6bb010e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb4f57700 1 --2- 172.21.15.87:0/505606090 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6bb00f6a90 0x7f6bb010e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb4f57700 1 -- 172.21.15.87:0/505606090 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80097c0 con 0x7f6bb00fe530 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bb4f57700 1 --2- 172.21.15.87:0/505606090 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6bb00fe530 0x7f6bb01081a0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7f6ba8004e00 tx=0x7f6ba8004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bad7fa700 10 client.?.objecter ms_handle_connect 0x7f6bb00fe530 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.057+0000 7f6bad7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.058+0000 7f6bad7fa700 1 -- 172.21.15.87:0/505606090 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba800e040 con 0x7f6bb00fe530 2024-02-23T00:31:56.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.058+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb0108c40 con 0x7f6bb00fe530 2024-02-23T00:31:56.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.058+0000 7f6bad7fa700 1 -- 172.21.15.87:0/505606090 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ba8018040 con 0x7f6bb00fe530 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.058+0000 7f6bad7fa700 1 -- 172.21.15.87:0/505606090 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8013cf0 con 0x7f6bb00fe530 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.058+0000 7f6bb69ba700 10 client.36848.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.058+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb00f74c0 con 0x7f6bb00fe530 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.058+0000 7f6bb69ba700 1 librados: init done 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.059+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb005fa60 con 0x7f6bb00fe530 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.059+0000 7f6bad7fa700 1 -- 172.21.15.87:0/505606090 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f6ba8013420 con 0x7f6bb00fe530 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.060+0000 7f6bad7fa700 1 --2- 172.21.15.87:0/505606090 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6b98062c90 0x7f6b98065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.060+0000 7f6bad7fa700 1 -- 172.21.15.87:0/505606090 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(165..165 src has 38..165) v4 ==== 9266+0+0 (secure 0 0 0) 0x7f6ba807e250 con 0x7f6bb00fe530 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.060+0000 7f6baffff700 1 --2- 172.21.15.87:0/505606090 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6b98062c90 0x7f6b98065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.060+0000 7f6bad7fa700 10 client.36848.objecter ms_dispatch 0x7f6bb0001550 osd_map(165..165 src has 38..165) v4 2024-02-23T00:31:56.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.060+0000 7f6bad7fa700 3 client.36848.objecter handle_osd_map got epochs [165,165] > 0 2024-02-23T00:31:56.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.060+0000 7f6bad7fa700 3 client.36848.objecter handle_osd_map decoding full epoch 165 2024-02-23T00:31:56.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.060+0000 7f6bad7fa700 20 client.36848.objecter dump_active .. 0 homeless 2024-02-23T00:31:56.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.060+0000 7f6bad7fa700 10 client.36848.objecter ms_handle_connect 0x7f6b98062c90 2024-02-23T00:31:56.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.060+0000 7f6baffff700 1 --2- 172.21.15.87:0/505606090 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6b98062c90 0x7f6b98065150 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 rx=0x7f6ba0004e80 tx=0x7f6ba000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:56.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.064+0000 7f6bad7fa700 1 -- 172.21.15.87:0/505606090 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6ba8051310 con 0x7f6bb00fe530 2024-02-23T00:31:56.070 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:56.092 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:56.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:56.196 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:31:56.223 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:56.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.265+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f6bb01070d0 con 0x7f6bb00fe530 2024-02-23T00:31:56.328 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:31:56.328 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:31:56.328 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:31:56.328 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:31:56.429 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:31:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:56.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:31:56.518 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:55.398679+0000 mon.c (mon.2) 147 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:56.518 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:55.988183+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? 172.21.15.87:0/2638609709' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:56.518 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:55.988309+0000 mon.a (mon.0) 1291 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-23T00:31:56.518 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:56.246153+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-02-23T00:31:56.518 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:56.249195+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1"}]: dispatch 2024-02-23T00:31:56.533 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:56.562 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:56.582 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:56.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:56.671 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:55.398679+0000 mon.c (mon.2) 147 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:56.671 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:55.988183+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? 172.21.15.87:0/2638609709' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:56.671 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:55.988309+0000 mon.a (mon.0) 1291 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-23T00:31:56.671 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:56.246153+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-02-23T00:31:56.672 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:56.249195+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1"}]: dispatch 2024-02-23T00:31:56.686 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:31:56.715 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:56.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:55.398679+0000 mon.c (mon.2) 147 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:56.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:55.988183+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? 172.21.15.87:0/2638609709' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:56.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:55.988309+0000 mon.a (mon.0) 1291 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-23T00:31:56.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:56.246153+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-02-23T00:31:56.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:56.249195+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1"}]: dispatch 2024-02-23T00:31:56.914 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:31:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:56.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:31:57.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:56.997+0000 7f6bad7fa700 1 -- 172.21.15.87:0/505606090 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v166) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6ba8021030 con 0x7f6bb00fe530 2024-02-23T00:31:57.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.002+0000 7f6bb69ba700 10 librados: watch_flush enter 2024-02-23T00:31:57.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.002+0000 7f6bb69ba700 10 librados: watch_flush exit 2024-02-23T00:31:57.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.002+0000 7f6bb69ba700 20 client.36848.objecter shutdown clearing up homeless session... 2024-02-23T00:31:57.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.002+0000 7f6bb69ba700 10 client.36848.objecter successfully canceled tick 2024-02-23T00:31:57.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.003+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6b98062c90 msgr2=0x7f6b98065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:57.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.003+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/505606090 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6b98062c90 0x7f6b98065150 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 rx=0x7f6ba0004e80 tx=0x7f6ba000b040).stop 2024-02-23T00:31:57.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.003+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6bb00fe530 msgr2=0x7f6bb01081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:57.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.003+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/505606090 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6bb00fe530 0x7f6bb01081a0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7f6ba8004e00 tx=0x7f6ba8004d10).stop 2024-02-23T00:31:57.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.005+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 shutdown_connections 2024-02-23T00:31:57.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.005+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/505606090 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6b98062c90 0x7f6b98065150 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:57.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.005+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/505606090 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6bb00fe530 0x7f6bb01081a0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:57.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.005+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/505606090 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6bb00f6a90 0x7f6bb010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:57.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.005+0000 7f6bb69ba700 1 --2- 172.21.15.87:0/505606090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6bb00fdbd0 0x7f6bb010e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:57.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.005+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 >> 172.21.15.87:0/505606090 conn(0x7f6bb006c370 msgr2=0x7f6bb00f5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:57.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.006+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 shutdown_connections 2024-02-23T00:31:57.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.008+0000 7f6bb69ba700 1 -- 172.21.15.87:0/505606090 wait complete. 2024-02-23T00:31:57.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:57.010+0000 7f6bb69ba700 1 librados: shutdown 2024-02-23T00:31:57.018 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:57.019 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:31:57.030 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-23T00:31:57.048 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:57.068 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:57.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:57.172 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:31:57.199 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:57.314 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:31:57.323 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:31:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:57.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:31:57.427 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:31:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:57.528 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:31:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:57.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:31:57.639 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:57.667 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:55.573835+0000 mgr.y (mgr.24437) 218 : cluster [DBG] pgmap v229: 363 pgs: 32 creating+peering, 2 clean+premerge+peered, 329 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:56.267368+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.87:0/505606090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:56.399408+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:56.505778+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:56.985967+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:56.993896+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi087-82115-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]': finished 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:56.994050+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1"}]': finished 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:56.994158+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:56.994200+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:57.006676+0000 mon.c (mon.2) 150 : audit [INF] from='client.? 172.21.15.87:0/916251352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi087-82011-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:57.829 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:57.010677+0000 mon.a (mon.0) 1300 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi087-82011-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:57.830 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:57.400384+0000 mon.c (mon.2) 151 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:57.832 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:55.573835+0000 mgr.y (mgr.24437) 218 : cluster [DBG] pgmap v229: 363 pgs: 32 creating+peering, 2 clean+premerge+peered, 329 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:56.267368+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.87:0/505606090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:56.399408+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:56.505778+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:56.985967+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:56.993896+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi087-82115-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]': finished 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:56.994050+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1"}]': finished 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:56.994158+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:56.994200+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:57.006676+0000 mon.c (mon.2) 150 : audit [INF] from='client.? 172.21.15.87:0/916251352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi087-82011-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:57.010677+0000 mon.a (mon.0) 1300 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi087-82011-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:57.400384+0000 mon.c (mon.2) 151 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:57.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:55.573835+0000 mgr.y (mgr.24437) 218 : cluster [DBG] pgmap v229: 363 pgs: 32 creating+peering, 2 clean+premerge+peered, 329 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:56.267368+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.87:0/505606090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:56.399408+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:56.505778+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:56.985967+0000 mon.a (mon.0) 1295 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:56.993896+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi087-82115-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]': finished 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:56.994050+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? 172.21.15.87:0/3193797020' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi087-82083-1","app":"app1","key":"key1"}]': finished 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:56.994158+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:56.994200+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:57.006676+0000 mon.c (mon.2) 150 : audit [INF] from='client.? 172.21.15.87:0/916251352' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi087-82011-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:57.010677+0000 mon.a (mon.0) 1300 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi087-82011-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:57.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:57.400384+0000 mon.c (mon.2) 151 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: seed 82083 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (9 ms) 2024-02-23T00:31:58.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (1247 ms) 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (20 ms) 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (9 ms) 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (33 ms) 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (5 ms) 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (18 ms) 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (11 ms) 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-02-23T00:31:58.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (9 ms) 2024-02-23T00:31:58.020 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-02-23T00:31:58.020 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (14 ms) 2024-02-23T00:31:58.020 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-02-23T00:31:58.020 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (7 ms) 2024-02-23T00:31:58.020 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-02-23T00:31:58.020 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (4 ms) 2024-02-23T00:31:58.020 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (8 ms) 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: osd_max_attr_size = 0 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1703 ms) 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (480 ms) 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: done waiting, doing copies 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: waiting for final deep scrubs... 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: done waiting 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (61485 ms) 2024-02-23T00:31:58.021 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (2 ms) 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4391 ms) 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-02-23T00:31:58.022 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-02-23T00:31:58.127 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:31:58.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.138+0000 7fb420b1b700 1 -- 172.21.15.87:0/3757758573 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 msgr2=0x7fb41c0fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:58.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.138+0000 7fb420b1b700 1 --2- 172.21.15.87:0/3757758573 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 0x7fb41c0fcf90 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fb40c004870 tx=0x7fb40c009cb0).stop 2024-02-23T00:31:58.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.138+0000 7fb420b1b700 1 -- 172.21.15.87:0/3757758573 shutdown_connections 2024-02-23T00:31:58.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.138+0000 7fb420b1b700 1 --2- 172.21.15.87:0/3757758573 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb41c0f7400 0x7fb41c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:58.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.138+0000 7fb420b1b700 1 --2- 172.21.15.87:0/3757758573 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 0x7fb41c0fcf90 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:58.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.138+0000 7fb420b1b700 1 --2- 172.21.15.87:0/3757758573 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb41c0fd510 0x7fb41c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:58.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.138+0000 7fb420b1b700 1 -- 172.21.15.87:0/3757758573 >> 172.21.15.87:0/3757758573 conn(0x7fb41c0f2820 msgr2=0x7fb41c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:58.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.138+0000 7fb420b1b700 1 -- 172.21.15.87:0/3757758573 shutdown_connections 2024-02-23T00:31:58.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.138+0000 7fb420b1b700 1 -- 172.21.15.87:0/3757758573 wait complete. 2024-02-23T00:31:58.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 librados: starting msgr at 2024-02-23T00:31:58.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 librados: starting objecter 2024-02-23T00:31:58.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 Processor -- start 2024-02-23T00:31:58.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 -- start start 2024-02-23T00:31:58.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 librados: setting wanted keys 2024-02-23T00:31:58.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 librados: calling monclient init 2024-02-23T00:31:58.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb41c0f7400 0x7fb41c10d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:58.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 0x7fb41c10d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:58.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb41c0fd510 0x7fb41c1072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:58.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c1077b0 con 0x7fb41c0f7400 2024-02-23T00:31:58.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c107920 con 0x7fb41c0fd510 2024-02-23T00:31:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.139+0000 7fb420b1b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c107ae0 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb419d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 0x7fb41c10d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb41a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb41c0f7400 0x7fb41c10d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb419d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 0x7fb41c10d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50910/0 (socket says 172.21.15.87:50910) 2024-02-23T00:31:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb419d9b700 1 -- 172.21.15.87:0/2085538933 learned_addr learned my addr 172.21.15.87:0/2085538933 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:31:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb41ad9d700 1 --2- 172.21.15.87:0/2085538933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb41c0fd510 0x7fb41c1072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb419d9b700 1 -- 172.21.15.87:0/2085538933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb41c0fd510 msgr2=0x7fb41c1072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb419d9b700 1 --2- 172.21.15.87:0/2085538933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb41c0fd510 0x7fb41c1072c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb419d9b700 1 -- 172.21.15.87:0/2085538933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb41c0f7400 msgr2=0x7fb41c10d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb419d9b700 1 --2- 172.21.15.87:0/2085538933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb41c0f7400 0x7fb41c10d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb419d9b700 1 -- 172.21.15.87:0/2085538933 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4140097c0 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb41a59c700 1 --2- 172.21.15.87:0/2085538933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb41c0f7400 0x7fb41c10d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:31:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb419d9b700 1 --2- 172.21.15.87:0/2085538933 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 0x7fb41c10d610 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7fb41000e990 tx=0x7fb41000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb40b7fe700 10 client.?.objecter ms_handle_connect 0x7fb41c0fcbb0 2024-02-23T00:31:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb40b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:31:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb40b7fe700 1 -- 172.21.15.87:0/2085538933 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41000ee80 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb40b7fe700 1 -- 172.21.15.87:0/2085538933 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb41000f040 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c005040 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb40b7fe700 1 -- 172.21.15.87:0/2085538933 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41000c840 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb420b1b700 10 client.36872.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:31:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41c108060 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.140+0000 7fb420b1b700 1 librados: init done 2024-02-23T00:31:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.141+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41c05fa60 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.142+0000 7fb40b7fe700 1 -- 172.21.15.87:0/2085538933 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb41000cc70 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.142+0000 7fb40b7fe700 1 --2- 172.21.15.87:0/2085538933 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb404064ea0 0x7fb404067360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:31:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.142+0000 7fb40b7fe700 1 -- 172.21.15.87:0/2085538933 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(167..167 src has 38..167) v4 ==== 9622+0+0 (secure 0 0 0) 0x7fb41007d3c0 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.142+0000 7fb40b7fe700 10 client.36872.objecter ms_dispatch 0x7fb41c001550 osd_map(167..167 src has 38..167) v4 2024-02-23T00:31:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.142+0000 7fb40b7fe700 3 client.36872.objecter handle_osd_map got epochs [167,167] > 0 2024-02-23T00:31:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.142+0000 7fb40b7fe700 3 client.36872.objecter handle_osd_map decoding full epoch 167 2024-02-23T00:31:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.143+0000 7fb40b7fe700 20 client.36872.objecter dump_active .. 0 homeless 2024-02-23T00:31:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.143+0000 7fb41a59c700 1 --2- 172.21.15.87:0/2085538933 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb404064ea0 0x7fb404067360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:31:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.143+0000 7fb40b7fe700 10 client.36872.objecter ms_handle_connect 0x7fb404064ea0 2024-02-23T00:31:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.143+0000 7fb41a59c700 1 --2- 172.21.15.87:0/2085538933 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb404064ea0 0x7fb404067360 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 rx=0x7fb40c004ee0 tx=0x7fb40c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:31:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.147+0000 7fb40b7fe700 1 -- 172.21.15.87:0/2085538933 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb410049f60 con 0x7fb41c0fcbb0 2024-02-23T00:31:58.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.313+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb41c106180 con 0x7fb404064ea0 2024-02-23T00:31:58.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.335+0000 7fb40b7fe700 1 -- 172.21.15.87:0/2085538933 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+827143 (secure 0 0 0) 0x7fb41c106180 con 0x7fb404064ea0 2024-02-23T00:31:58.348 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.340+0000 7fb420b1b700 10 librados: watch_flush enter 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.340+0000 7fb420b1b700 10 librados: watch_flush exit 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.340+0000 7fb420b1b700 20 client.36872.objecter shutdown clearing up homeless session... 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.340+0000 7fb420b1b700 10 client.36872.objecter successfully canceled tick 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.340+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb404064ea0 msgr2=0x7fb404067360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.340+0000 7fb420b1b700 1 --2- 172.21.15.87:0/2085538933 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb404064ea0 0x7fb404067360 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 rx=0x7fb40c004ee0 tx=0x7fb40c00b040).stop 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 msgr2=0x7fb41c10d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 --2- 172.21.15.87:0/2085538933 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 0x7fb41c10d610 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7fb41000e990 tx=0x7fb41000b040).stop 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 shutdown_connections 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 --2- 172.21.15.87:0/2085538933 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb404064ea0 0x7fb404067360 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 --2- 172.21.15.87:0/2085538933 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb41c0fd510 0x7fb41c1072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 --2- 172.21.15.87:0/2085538933 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb41c0f7400 0x7fb41c10d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:58.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 --2- 172.21.15.87:0/2085538933 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb41c0fcbb0 0x7fb41c10d610 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:31:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 >> 172.21.15.87:0/2085538933 conn(0x7fb41c0f2820 msgr2=0x7fb41c0fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:31:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 shutdown_connections 2024-02-23T00:31:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 -- 172.21.15.87:0/2085538933 wait complete. 2024-02-23T00:31:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:31:58.341+0000 7fb420b1b700 1 librados: shutdown 2024-02-23T00:31:58.359 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:31:58.719 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:57.574313+0000 mgr.y (mgr.24437) 219 : cluster [DBG] pgmap v232: 338 pgs: 40 unknown, 1 clean+premerge+peered, 297 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:57.575474+0000 mon.a (mon.0) 1301 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:57.992467+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi087-82011-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:57.992642+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:57.992695+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.000326+0000 mon.c (mon.2) 152 : audit [INF] from='client.? 172.21.15.87:0/4037067096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi087-82019-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.016175+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi087-82019-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.037873+0000 mon.c (mon.2) 153 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.040497+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.041630+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.041923+0000 mon.a (mon.0) 1307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.721 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.042312+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82083-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:58.721 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.042587+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82083-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:58.721 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.401143+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:58.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:57.574313+0000 mgr.y (mgr.24437) 219 : cluster [DBG] pgmap v232: 338 pgs: 40 unknown, 1 clean+premerge+peered, 297 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:31:58.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:57.575474+0000 mon.a (mon.0) 1301 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-23T00:31:58.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:57.992467+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi087-82011-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:58.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:57.992642+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-23T00:31:58.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:57.992695+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-02-23T00:31:58.722 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.000326+0000 mon.c (mon.2) 152 : audit [INF] from='client.? 172.21.15.87:0/4037067096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi087-82019-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:58.722 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.016175+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi087-82019-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:58.722 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.037873+0000 mon.c (mon.2) 153 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.722 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.040497+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.722 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.041630+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.722 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.041923+0000 mon.a (mon.0) 1307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.722 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.042312+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82083-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:58.722 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.042587+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82083-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:58.722 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:31:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.401143+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:57.574313+0000 mgr.y (mgr.24437) 219 : cluster [DBG] pgmap v232: 338 pgs: 40 unknown, 1 clean+premerge+peered, 297 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:31:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:57.575474+0000 mon.a (mon.0) 1301 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-23T00:31:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:57.992467+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi087-82011-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:31:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:57.992642+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-23T00:31:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:57.992695+0000 mon.a (mon.0) 1304 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-02-23T00:31:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.000326+0000 mon.c (mon.2) 152 : audit [INF] from='client.? 172.21.15.87:0/4037067096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi087-82019-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.016175+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi087-82019-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:31:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.037873+0000 mon.c (mon.2) 153 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.040497+0000 mon.a (mon.0) 1306 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.041630+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.041923+0000 mon.a (mon.0) 1307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:31:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.042312+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82083-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.042587+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82083-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:31:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:31:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.401143+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:31:59.420 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:31:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:31:59.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:31:59.524 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:31:59.557 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:31:59.903 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:32:00.002 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.315229+0000 mgr.y (mgr.24437) 220 : audit [DBG] from='client.36872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:32:00.002 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.996283+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi087-82019-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:00.002 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:58.996444+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82083-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:00.003 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:58.996521+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-02-23T00:32:00.003 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:59.019648+0000 mon.c (mon.2) 157 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82083-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:00.003 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:59.020079+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82083-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:00.003 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:31:59.402242+0000 mon.c (mon.2) 158 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:00.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.315229+0000 mgr.y (mgr.24437) 220 : audit [DBG] from='client.36872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:32:00.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.996283+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi087-82019-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:00.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:58.996444+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82083-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:00.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:58.996521+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-02-23T00:32:00.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:59.019648+0000 mon.c (mon.2) 157 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82083-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:00.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:59.020079+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82083-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:00.005 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:31:59.402242+0000 mon.c (mon.2) 158 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.036+0000 7f96ba432700 1 -- 172.21.15.87:0/262952312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96b40fd510 msgr2=0x7f96b40f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.036+0000 7f96ba432700 1 --2- 172.21.15.87:0/262952312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96b40fd510 0x7f96b40f6d40 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7f96a8004870 tx=0x7f96a8009cb0).stop 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.036+0000 7f96ba432700 1 -- 172.21.15.87:0/262952312 shutdown_connections 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.036+0000 7f96ba432700 1 --2- 172.21.15.87:0/262952312 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96b40f7400 0x7f96b4103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.036+0000 7f96ba432700 1 --2- 172.21.15.87:0/262952312 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96b40fcbb0 0x7f96b40fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.036+0000 7f96ba432700 1 --2- 172.21.15.87:0/262952312 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96b40fd510 0x7f96b40f6d40 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.036+0000 7f96ba432700 1 -- 172.21.15.87:0/262952312 >> 172.21.15.87:0/262952312 conn(0x7f96b40f2820 msgr2=0x7f96b40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.036+0000 7f96ba432700 1 -- 172.21.15.87:0/262952312 shutdown_connections 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.036+0000 7f96ba432700 1 -- 172.21.15.87:0/262952312 wait complete. 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.037+0000 7f96ba432700 1 librados: starting msgr at 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.037+0000 7f96ba432700 1 librados: starting objecter 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.037+0000 7f96ba432700 1 Processor -- start 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.037+0000 7f96ba432700 1 -- start start 2024-02-23T00:32:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.037+0000 7f96ba432700 1 librados: setting wanted keys 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.037+0000 7f96ba432700 1 librados: calling monclient init 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96ba432700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96b40f7400 0x7f96b41069b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96ba432700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96b40fcbb0 0x7f96b4106ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96ba432700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96b40fd510 0x7f96b410ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96ba432700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b410b290 con 0x7f96b40f7400 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96ba432700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b410b400 con 0x7f96b40fd510 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96ba432700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b410b5c0 con 0x7f96b40fcbb0 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b37fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96b40fcbb0 0x7f96b4106ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b37fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96b40fcbb0 0x7f96b4106ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50962/0 (socket says 172.21.15.87:50962) 2024-02-23T00:32:00.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b37fe700 1 -- 172.21.15.87:0/1305282909 learned_addr learned my addr 172.21.15.87:0/1305282909 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b3fff700 1 --2- 172.21.15.87:0/1305282909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96b40f7400 0x7f96b41069b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b89cf700 1 --2- 172.21.15.87:0/1305282909 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96b40fd510 0x7f96b410ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b37fe700 1 -- 172.21.15.87:0/1305282909 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96b40fd510 msgr2=0x7f96b410ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b37fe700 1 --2- 172.21.15.87:0/1305282909 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96b40fd510 0x7f96b410ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b37fe700 1 -- 172.21.15.87:0/1305282909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96b40f7400 msgr2=0x7f96b41069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b37fe700 1 --2- 172.21.15.87:0/1305282909 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96b40f7400 0x7f96b41069b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b37fe700 1 -- 172.21.15.87:0/1305282909 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a8005040 con 0x7f96b40fcbb0 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b37fe700 1 --2- 172.21.15.87:0/1305282909 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96b40fcbb0 0x7f96b4106ea0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7f96a80043a0 tx=0x7f96a8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b17fa700 10 client.?.objecter ms_handle_connect 0x7f96b40fcbb0 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b17fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a800e030 con 0x7f96b40fcbb0 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96a8013da0 con 0x7f96b40fcbb0 2024-02-23T00:32:00.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.038+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8018070 con 0x7f96b40fcbb0 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.039+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b410b840 con 0x7f96b40fcbb0 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.039+0000 7f96ba432700 10 client.36884.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.039+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b4124030 con 0x7f96b40fcbb0 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.039+0000 7f96ba432700 1 librados: init done 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.040+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f96a801d420 con 0x7f96b40fcbb0 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96b17fa700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6800/1231597463 conn(0x7f969c062c90 0x7f969c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96b3fff700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6800/1231597463 conn(0x7f969c062c90 0x7f969c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96b17fa700 10 client.36884.objecter ms_handle_connect 0x7f969c062c90 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96b3fff700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6800/1231597463 conn(0x7f969c062c90 0x7f969c065150 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7f96b4107b70 tx=0x7f96a4009060).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f96b405fa60 con 0x7f96b40fcbb0 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(169..169 src has 38..169) v4 ==== 9684+0+0 (secure 0 0 0) 0x7f96a8051200 con 0x7f96b40fcbb0 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 osd_map(169..169 src has 38..169) v4 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96b17fa700 3 client.36884.objecter handle_osd_map got epochs [169,169] > 0 2024-02-23T00:32:00.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96b17fa700 3 client.36884.objecter handle_osd_map decoding full epoch 169 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.041+0000 7f96b17fa700 20 client.36884.objecter dump_active .. 0 homeless 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v169) v1 ==== 54+0+16 (secure 0 0 0) 0x7f96a8082060 con 0x7f96b40fcbb0 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 10 client.36884.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 1 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6801/2153191863 conn(0x7f96b40fdec0 0x7f96b40fe2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b405ee80 osd=0 3 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b405ee80 osd=0 3 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b405ee80 osd=0 3 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 1 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b405ee80 osd=0 3 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 0 1 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b405ee80 osd=0 4 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 10 client.36884.objecter _send_command 1 2024-02-23T00:32:00.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96b4106170 con 0x7f96b40fdec0 2024-02-23T00:32:00.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96b89cf700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6801/2153191863 conn(0x7f96b40fdec0 0x7f96b40fe2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96b17fa700 10 client.36884.objecter ms_handle_connect 0x7f96b40fdec0 2024-02-23T00:32:00.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.042+0000 7f96b89cf700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6801/2153191863 conn(0x7f96b40fdec0 0x7f96b40fe2a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.043+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f96b4106170 con 0x7f96b40fdec0 2024-02-23T00:32:00.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.043+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:32:00.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.043+0000 7f96b17fa700 10 client.36884.objecter _finish_command 1 = system:0 2024-02-23T00:32:00.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.043+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b405ee80 osd=0 4 2024-02-23T00:32:00.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.043+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 0 1 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 10 client.36884.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 2 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b405ee80 osd=0 3 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b405ee80 osd=0 3 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b405ee80 osd=0 3 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 2 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b405ee80 osd=0 3 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 0 2 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b405ee80 osd=0 4 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 10 client.36884.objecter _send_command 2 2024-02-23T00:32:00.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.067+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f96b40273c0 con 0x7f96b40fdec0 2024-02-23T00:32:00.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.068+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f96b40273c0 con 0x7f96b40fdec0 2024-02-23T00:32:00.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.068+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:00.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.068+0000 7f96b17fa700 10 client.36884.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b405ee80 osd=0 4 2024-02-23T00:32:00.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 0 2 2024-02-23T00:32:00.071 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:32:00.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 10 client.36884.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:00.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 3 2024-02-23T00:32:00.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6805/685354171 conn(0x7f96b40598f0 0x7f96b4059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4028710 osd=1 3 2024-02-23T00:32:00.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4028710 osd=1 3 2024-02-23T00:32:00.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4028710 osd=1 3 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 3 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4028710 osd=1 3 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 1 3 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4028710 osd=1 4 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 10 client.36884.objecter _send_command 3 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96b412a940 con 0x7f96b40598f0 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96b3fff700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6805/685354171 conn(0x7f96b40598f0 0x7f96b4059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96b17fa700 10 client.36884.objecter ms_handle_connect 0x7f96b40598f0 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.069+0000 7f96b3fff700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6805/685354171 conn(0x7f96b40598f0 0x7f96b4059cf0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.070+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f96b412a940 con 0x7f96b40598f0 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.070+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.070+0000 7f96b17fa700 10 client.36884.objecter _finish_command 3 = system:0 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.070+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4028710 osd=1 4 2024-02-23T00:32:00.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.070+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 1 3 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 10 client.36884.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 4 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4028710 osd=1 3 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4028710 osd=1 3 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4028710 osd=1 3 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 4 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4028710 osd=1 3 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 1 4 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4028710 osd=1 4 2024-02-23T00:32:00.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 10 client.36884.objecter _send_command 4 2024-02-23T00:32:00.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.084+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f96b4059360 con 0x7f96b40598f0 2024-02-23T00:32:00.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f96b4059360 con 0x7f96b40598f0 2024-02-23T00:32:00.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:00.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96b17fa700 10 client.36884.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4028710 osd=1 4 2024-02-23T00:32:00.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 1 4 2024-02-23T00:32:00.089 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:32:00.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 10 client.36884.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:00.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 5 2024-02-23T00:32:00.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6809/3605394228 conn(0x7f96b4028270 0x7f96b4129360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4059770 osd=2 3 2024-02-23T00:32:00.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4059770 osd=2 3 2024-02-23T00:32:00.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4059770 osd=2 3 2024-02-23T00:32:00.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 5 2024-02-23T00:32:00.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4059770 osd=2 3 2024-02-23T00:32:00.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 2 5 2024-02-23T00:32:00.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4059770 osd=2 4 2024-02-23T00:32:00.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 10 client.36884.objecter _send_command 5 2024-02-23T00:32:00.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96b4129850 con 0x7f96b4028270 2024-02-23T00:32:00.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.086+0000 7f96b37fe700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6809/3605394228 conn(0x7f96b4028270 0x7f96b4129360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.087+0000 7f96b17fa700 10 client.36884.objecter ms_handle_connect 0x7f96b4028270 2024-02-23T00:32:00.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.087+0000 7f96b37fe700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6809/3605394228 conn(0x7f96b4028270 0x7f96b4129360 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.087+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f96b4129850 con 0x7f96b4028270 2024-02-23T00:32:00.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.087+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:32:00.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.087+0000 7f96b17fa700 10 client.36884.objecter _finish_command 5 = system:0 2024-02-23T00:32:00.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.087+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4059770 osd=2 4 2024-02-23T00:32:00.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.087+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 2 5 2024-02-23T00:32:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 10 client.36884.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 6 2024-02-23T00:32:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4059770 osd=2 3 2024-02-23T00:32:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4059770 osd=2 3 2024-02-23T00:32:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4059770 osd=2 3 2024-02-23T00:32:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 6 2024-02-23T00:32:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4059770 osd=2 3 2024-02-23T00:32:00.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 2 6 2024-02-23T00:32:00.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4059770 osd=2 4 2024-02-23T00:32:00.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 10 client.36884.objecter _send_command 6 2024-02-23T00:32:00.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.111+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f96b40277d0 con 0x7f96b4028270 2024-02-23T00:32:00.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.112+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f96b40277d0 con 0x7f96b4028270 2024-02-23T00:32:00.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.112+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:00.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.112+0000 7f96b17fa700 10 client.36884.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.112+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4059770 osd=2 4 2024-02-23T00:32:00.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.112+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 2 6 2024-02-23T00:32:00.115 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 10 client.36884.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 7 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6813/128059790 conn(0x7f96b412cac0 0x7f96b412eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4027be0 osd=3 3 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4027be0 osd=3 3 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4027be0 osd=3 3 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 7 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4027be0 osd=3 3 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 3 7 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4027be0 osd=3 4 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 10 client.36884.objecter _send_command 7 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96b412a750 con 0x7f96b412cac0 2024-02-23T00:32:00.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96b89cf700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6813/128059790 conn(0x7f96b412cac0 0x7f96b412eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96b17fa700 10 client.36884.objecter ms_handle_connect 0x7f96b412cac0 2024-02-23T00:32:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.113+0000 7f96b89cf700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6813/128059790 conn(0x7f96b412cac0 0x7f96b412eeb0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.114+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f96b412a750 con 0x7f96b412cac0 2024-02-23T00:32:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.114+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:32:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.114+0000 7f96b17fa700 10 client.36884.objecter _finish_command 7 = system:0 2024-02-23T00:32:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.114+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4027be0 osd=3 4 2024-02-23T00:32:00.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.114+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 3 7 2024-02-23T00:32:00.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 10 client.36884.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:00.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 8 2024-02-23T00:32:00.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4027be0 osd=3 3 2024-02-23T00:32:00.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4027be0 osd=3 3 2024-02-23T00:32:00.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4027be0 osd=3 3 2024-02-23T00:32:00.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 8 2024-02-23T00:32:00.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4027be0 osd=3 3 2024-02-23T00:32:00.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 3 8 2024-02-23T00:32:00.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4027be0 osd=3 4 2024-02-23T00:32:00.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 10 client.36884.objecter _send_command 8 2024-02-23T00:32:00.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.130+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f96b4129cb0 con 0x7f96b412cac0 2024-02-23T00:32:00.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.131+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f96b4129cb0 con 0x7f96b412cac0 2024-02-23T00:32:00.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.131+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:00.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.131+0000 7f96b17fa700 10 client.36884.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.131+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4027be0 osd=3 4 2024-02-23T00:32:00.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.131+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 3 8 2024-02-23T00:32:00.134 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:32:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 10 client.36884.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 9 2024-02-23T00:32:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6800/4175844757 conn(0x7f96b4128ec0 0x7f96b4132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4129a70 osd=4 3 2024-02-23T00:32:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129a70 osd=4 3 2024-02-23T00:32:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4129a70 osd=4 3 2024-02-23T00:32:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 9 2024-02-23T00:32:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4129a70 osd=4 3 2024-02-23T00:32:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 4 9 2024-02-23T00:32:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129a70 osd=4 4 2024-02-23T00:32:00.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 10 client.36884.objecter _send_command 9 2024-02-23T00:32:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96b4129be0 con 0x7f96b4128ec0 2024-02-23T00:32:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96b3fff700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6800/4175844757 conn(0x7f96b4128ec0 0x7f96b4132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96b17fa700 10 client.36884.objecter ms_handle_connect 0x7f96b4128ec0 2024-02-23T00:32:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.132+0000 7f96b3fff700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6800/4175844757 conn(0x7f96b4128ec0 0x7f96b4132530 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.133+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f96b4129be0 con 0x7f96b4128ec0 2024-02-23T00:32:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.133+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:32:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.133+0000 7f96b17fa700 10 client.36884.objecter _finish_command 9 = system:0 2024-02-23T00:32:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.133+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4129a70 osd=4 4 2024-02-23T00:32:00.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.133+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 4 9 2024-02-23T00:32:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 10 client.36884.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 10 2024-02-23T00:32:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4129a70 osd=4 3 2024-02-23T00:32:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129a70 osd=4 3 2024-02-23T00:32:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4129a70 osd=4 3 2024-02-23T00:32:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 10 2024-02-23T00:32:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4129a70 osd=4 3 2024-02-23T00:32:00.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 4 10 2024-02-23T00:32:00.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129a70 osd=4 4 2024-02-23T00:32:00.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 10 client.36884.objecter _send_command 10 2024-02-23T00:32:00.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.148+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f96b4129be0 con 0x7f96b4128ec0 2024-02-23T00:32:00.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.150+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f96b4129be0 con 0x7f96b4128ec0 2024-02-23T00:32:00.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.150+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:00.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.150+0000 7f96b17fa700 10 client.36884.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.150+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4129a70 osd=4 4 2024-02-23T00:32:00.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.150+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 4 10 2024-02-23T00:32:00.153 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.150+0000 7f96ba432700 10 client.36884.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.150+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 11 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.150+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6804/2550013795 conn(0x7f96b4132f30 0x7f96b4135320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4129be0 osd=5 3 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129be0 osd=5 3 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4129be0 osd=5 3 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 11 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4129be0 osd=5 3 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 5 11 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129be0 osd=5 4 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96ba432700 10 client.36884.objecter _send_command 11 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96b4137f70 con 0x7f96b4132f30 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96b37fe700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6804/2550013795 conn(0x7f96b4132f30 0x7f96b4135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96b17fa700 10 client.36884.objecter ms_handle_connect 0x7f96b4132f30 2024-02-23T00:32:00.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.151+0000 7f96b37fe700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6804/2550013795 conn(0x7f96b4132f30 0x7f96b4135320 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.152+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f96b4137f70 con 0x7f96b4132f30 2024-02-23T00:32:00.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.152+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:32:00.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.152+0000 7f96b17fa700 10 client.36884.objecter _finish_command 11 = system:0 2024-02-23T00:32:00.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.152+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4129be0 osd=5 4 2024-02-23T00:32:00.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.152+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 5 11 2024-02-23T00:32:00.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 10 client.36884.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 12 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4129be0 osd=5 3 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129be0 osd=5 3 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4129be0 osd=5 3 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 12 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4129be0 osd=5 3 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 5 12 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129be0 osd=5 4 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 10 client.36884.objecter _send_command 12 2024-02-23T00:32:00.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.166+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f96b4135810 con 0x7f96b4132f30 2024-02-23T00:32:00.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f96b4135810 con 0x7f96b4132f30 2024-02-23T00:32:00.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:00.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96b17fa700 10 client.36884.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4129be0 osd=5 4 2024-02-23T00:32:00.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 5 12 2024-02-23T00:32:00.170 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 10 client.36884.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 13 2024-02-23T00:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6808/2509408498 conn(0x7f96b4135ef0 0x7f96b4136330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4135810 osd=6 3 2024-02-23T00:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4135810 osd=6 3 2024-02-23T00:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4135810 osd=6 3 2024-02-23T00:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 13 2024-02-23T00:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4135810 osd=6 3 2024-02-23T00:32:00.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 6 13 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4135810 osd=6 4 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 10 client.36884.objecter _send_command 13 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.168+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96b4137f30 con 0x7f96b4135ef0 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.169+0000 7f96b89cf700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6808/2509408498 conn(0x7f96b4135ef0 0x7f96b4136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.169+0000 7f96b17fa700 10 client.36884.objecter ms_handle_connect 0x7f96b4135ef0 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.169+0000 7f96b89cf700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6808/2509408498 conn(0x7f96b4135ef0 0x7f96b4136330 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.170+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f96b4137f30 con 0x7f96b4135ef0 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.170+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.170+0000 7f96b17fa700 10 client.36884.objecter _finish_command 13 = system:0 2024-02-23T00:32:00.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.170+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4135810 osd=6 4 2024-02-23T00:32:00.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.170+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 6 13 2024-02-23T00:32:00.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 10 client.36884.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:00.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 14 2024-02-23T00:32:00.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4135810 osd=6 3 2024-02-23T00:32:00.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4135810 osd=6 3 2024-02-23T00:32:00.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4135810 osd=6 3 2024-02-23T00:32:00.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 14 2024-02-23T00:32:00.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4135810 osd=6 3 2024-02-23T00:32:00.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 6 14 2024-02-23T00:32:00.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4135810 osd=6 4 2024-02-23T00:32:00.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 10 client.36884.objecter _send_command 14 2024-02-23T00:32:00.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.184+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f96b4137f30 con 0x7f96b4135ef0 2024-02-23T00:32:00.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f96b4137f30 con 0x7f96b4135ef0 2024-02-23T00:32:00.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:00.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96b17fa700 10 client.36884.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4135810 osd=6 4 2024-02-23T00:32:00.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 6 14 2024-02-23T00:32:00.190 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:32:00.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 10 client.36884.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:00.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 15 2024-02-23T00:32:00.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6812/373787000 conn(0x7f96b413a710 0x7f96b413ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:00.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4137f30 osd=7 3 2024-02-23T00:32:00.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4137f30 osd=7 3 2024-02-23T00:32:00.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4137f30 osd=7 3 2024-02-23T00:32:00.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 15 2024-02-23T00:32:00.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4137f30 osd=7 3 2024-02-23T00:32:00.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 7 15 2024-02-23T00:32:00.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4137f30 osd=7 4 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 10 client.36884.objecter _send_command 15 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96b413e500 con 0x7f96b413a710 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.186+0000 7f96b3fff700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6812/373787000 conn(0x7f96b413a710 0x7f96b413ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.187+0000 7f96b17fa700 10 client.36884.objecter ms_handle_connect 0x7f96b413a710 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.187+0000 7f96b3fff700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6812/373787000 conn(0x7f96b413a710 0x7f96b413ab70 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.187+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f96b413e500 con 0x7f96b413a710 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.187+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.187+0000 7f96b17fa700 10 client.36884.objecter _finish_command 15 = system:0 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.187+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4137f30 osd=7 4 2024-02-23T00:32:00.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.187+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 7 15 2024-02-23T00:32:00.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.315229+0000 mgr.y (mgr.24437) 220 : audit [DBG] from='client.36872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:32:00.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.996283+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi087-82019-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:00.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:58.996444+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82083-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:00.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:58.996521+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-02-23T00:32:00.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:59.019648+0000 mon.c (mon.2) 157 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82083-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:00.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:59.020079+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82083-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:00.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:31:59.402242+0000 mon.c (mon.2) 158 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:00.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 10 client.36884.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:00.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign -1 16 2024-02-23T00:32:00.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4137f30 osd=7 3 2024-02-23T00:32:00.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4137f30 osd=7 3 2024-02-23T00:32:00.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 20 client.36884.objecter _get_session s=0x7f96b4137f30 osd=7 3 2024-02-23T00:32:00.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 15 client.36884.objecter _session_command_op_remove -1 16 2024-02-23T00:32:00.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 20 client.36884.objecter get_session s=0x7f96b4137f30 osd=7 3 2024-02-23T00:32:00.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 15 client.36884.objecter _session_command_op_assign 7 16 2024-02-23T00:32:00.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4137f30 osd=7 4 2024-02-23T00:32:00.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 10 client.36884.objecter _send_command 16 2024-02-23T00:32:00.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.202+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f96b41368e0 con 0x7f96b413a710 2024-02-23T00:32:00.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.203+0000 7f96b17fa700 1 -- 172.21.15.87:0/1305282909 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f96b41368e0 con 0x7f96b413a710 2024-02-23T00:32:00.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.203+0000 7f96b17fa700 10 client.36884.objecter ms_dispatch 0x7f96b4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:00.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.203+0000 7f96b17fa700 10 client.36884.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.203+0000 7f96b17fa700 20 client.36884.objecter put_session s=0x7f96b4137f30 osd=7 4 2024-02-23T00:32:00.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.203+0000 7f96b17fa700 15 client.36884.objecter _session_command_op_remove 7 16 2024-02-23T00:32:00.207 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:32:00.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 librados: watch_flush enter 2024-02-23T00:32:00.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 librados: watch_flush exit 2024-02-23T00:32:00.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 client.36884.objecter close_session for osd.0 2024-02-23T00:32:00.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6801/2153191863 conn(0x7f96b40fdec0 msgr2=0x7f96b40fe2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6801/2153191863 conn(0x7f96b40fdec0 0x7f96b40fe2a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b405ee80 osd=0 1 2024-02-23T00:32:00.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 client.36884.objecter close_session for osd.1 2024-02-23T00:32:00.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6805/685354171 conn(0x7f96b40598f0 msgr2=0x7f96b4059cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6805/685354171 conn(0x7f96b40598f0 0x7f96b4059cf0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4028710 osd=1 1 2024-02-23T00:32:00.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 client.36884.objecter close_session for osd.2 2024-02-23T00:32:00.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6809/3605394228 conn(0x7f96b4028270 msgr2=0x7f96b4129360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6809/3605394228 conn(0x7f96b4028270 0x7f96b4129360 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4059770 osd=2 1 2024-02-23T00:32:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 client.36884.objecter close_session for osd.3 2024-02-23T00:32:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6813/128059790 conn(0x7f96b412cac0 msgr2=0x7f96b412eeb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6813/128059790 conn(0x7f96b412cac0 0x7f96b412eeb0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96b89cf700 1 -- 172.21.15.87:0/1305282909 reap_dead start 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4027be0 osd=3 1 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 client.36884.objecter close_session for osd.4 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6800/4175844757 conn(0x7f96b4128ec0 msgr2=0x7f96b4132530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6800/4175844757 conn(0x7f96b4128ec0 0x7f96b4132530 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96b89cf700 1 -- 172.21.15.87:0/1305282909 reap_dead start 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129a70 osd=4 1 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 client.36884.objecter close_session for osd.5 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6804/2550013795 conn(0x7f96b4132f30 msgr2=0x7f96b4135320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6804/2550013795 conn(0x7f96b4132f30 0x7f96b4135320 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4129be0 osd=5 1 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 client.36884.objecter close_session for osd.6 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6808/2509408498 conn(0x7f96b4135ef0 msgr2=0x7f96b4136330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6808/2509408498 conn(0x7f96b4135ef0 0x7f96b4136330 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4135810 osd=6 1 2024-02-23T00:32:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 client.36884.objecter close_session for osd.7 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6812/373787000 conn(0x7f96b413a710 msgr2=0x7f96b413ab70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.171:6812/373787000 conn(0x7f96b413a710 0x7f96b413ab70 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 20 client.36884.objecter put_session s=0x7f96b4137f30 osd=7 1 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 20 client.36884.objecter shutdown clearing up homeless session... 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 10 client.36884.objecter successfully canceled tick 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6800/1231597463 conn(0x7f969c062c90 msgr2=0x7f969c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> v2:172.21.15.87:6800/1231597463 conn(0x7f969c062c90 0x7f969c065150 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 rx=0x7f96b4107b70 tx=0x7f96a4009060).stop 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96b40fcbb0 msgr2=0x7f96b4106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 --2- 172.21.15.87:0/1305282909 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96b40fcbb0 0x7f96b4106ea0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7f96a80043a0 tx=0x7f96a8009cb0).stop 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96b89cf700 1 -- 172.21.15.87:0/1305282909 reap_dead start 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 shutdown_connections 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 >> 172.21.15.87:0/1305282909 conn(0x7f96b40f2820 msgr2=0x7f96b4060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 shutdown_connections 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 -- 172.21.15.87:0/1305282909 wait complete. 2024-02-23T00:32:00.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:00.204+0000 7f96ba432700 1 librados: shutdown 2024-02-23T00:32:00.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:32:00.474 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:32:00.475 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:32:00.476 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-02-23T00:31:57.989823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":15357,"num_objects":967,"num_object_clones":541,"num_object_copies":2901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":0,"num_read":1687,"num_read_kb":1322,"num_write":1269,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":1999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1997,"ondisk_log_size":1997,"up":798,"acting":798,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1021,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3113712,"kb_used_data":691120,"kb_used_omap":0,"kb_used_meta":2422528,"kb_avail":746847504,"statfs":{"total":767960285184,"available":764771844096,"internally_reserved":0,"allocated":707706880,"data_stored":690120193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2480668672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-226499134,"num_objects":-32,"num_object_clones":0,"num_object_copies":-96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-32,"num_whiteouts":0,"num_read":-1006,"num_read_kb":-49156,"num_write":-1038,"num_write_kb":-226707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.007208"},"pg_stats":[{"pgid":"114.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"112.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"112.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"112.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"112.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"112.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"112.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"112.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"112.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"114.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:57.987481+0000","last_change":"2024-02-23T00:31:57.987481+0000","last_active":"2024-02-23T00:31:57.987481+0000","last_peered":"2024-02-23T00:31:57.987481+0000","last_clean":"2024-02-23T00:31:57.987481+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:57.987481+0000","last_undegraded":"2024-02-23T00:31:57.987481+0000","last_fullsized":"2024-02-23T00:31:57.987481+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:57.987481+0000","last_clean_scrub_stamp":"2024-02-23T00:31:57.987481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"113.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:31:56.985128+0000","last_change":"2024-02-23T00:31:56.985128+0000","last_active":"2024-02-23T00:31:56.985128+0000","last_peered":"2024-02-23T00:31:56.985128+0000","last_clean":"2024-02-23T00:31:56.985128+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:31:56.985128+0000","last_undegraded":"2024-02-23T00:31:56.985128+0000","last_fullsized":"2024-02-23T00:31:56.985128+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:56.985128+0000","last_clean_scrub_stamp":"2024-02-23T00:31:56.985128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"95.0","version":"151'12","reported_seq":55,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993210+0000","last_change":"2024-02-23T00:31:43.001814+0000","last_active":"2024-02-23T00:31:53.993210+0000","last_peered":"2024-02-23T00:31:53.993210+0000","last_clean":"2024-02-23T00:31:53.993210+0000","last_became_active":"2024-02-23T00:31:35.004764+0000","last_became_peered":"2024-02-23T00:31:35.004764+0000","last_unstale":"2024-02-23T00:31:53.993210+0000","last_undegraded":"2024-02-23T00:31:53.993210+0000","last_fullsized":"2024-02-23T00:31:53.993210+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'12","last_scrub_stamp":"2024-02-23T00:31:43.001724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:43.001724+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.1","version":"151'9","reported_seq":51,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996518+0000","last_change":"2024-02-23T00:31:43.020285+0000","last_active":"2024-02-23T00:31:53.996518+0000","last_peered":"2024-02-23T00:31:53.996518+0000","last_clean":"2024-02-23T00:31:53.996518+0000","last_became_active":"2024-02-23T00:31:35.010873+0000","last_became_peered":"2024-02-23T00:31:35.010873+0000","last_unstale":"2024-02-23T00:31:53.996518+0000","last_undegraded":"2024-02-23T00:31:53.996518+0000","last_fullsized":"2024-02-23T00:31:53.996518+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'9","last_scrub_stamp":"2024-02-23T00:31:43.020221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:43.020221+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":96,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"95.2","version":"151'2","reported_seq":37,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.999573+0000","last_change":"2024-02-23T00:31:43.871944+0000","last_active":"2024-02-23T00:31:53.999573+0000","last_peered":"2024-02-23T00:31:53.999573+0000","last_clean":"2024-02-23T00:31:53.999573+0000","last_became_active":"2024-02-23T00:31:35.020429+0000","last_became_peered":"2024-02-23T00:31:35.020429+0000","last_unstale":"2024-02-23T00:31:53.999573+0000","last_undegraded":"2024-02-23T00:31:53.999573+0000","last_fullsized":"2024-02-23T00:31:53.999573+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'2","last_scrub_stamp":"2024-02-23T00:31:43.871869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:43.871869+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.3","version":"151'3","reported_seq":39,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.984035+0000","last_change":"2024-02-23T00:31:43.422777+0000","last_active":"2024-02-23T00:31:53.984035+0000","last_peered":"2024-02-23T00:31:53.984035+0000","last_clean":"2024-02-23T00:31:53.984035+0000","last_became_active":"2024-02-23T00:31:35.015366+0000","last_became_peered":"2024-02-23T00:31:35.015366+0000","last_unstale":"2024-02-23T00:31:53.984035+0000","last_undegraded":"2024-02-23T00:31:53.984035+0000","last_fullsized":"2024-02-23T00:31:53.984035+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'3","last_scrub_stamp":"2024-02-23T00:31:43.422710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:43.422710+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.4","version":"151'4","reported_seq":40,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003236+0000","last_change":"2024-02-23T00:31:43.690800+0000","last_active":"2024-02-23T00:31:54.003236+0000","last_peered":"2024-02-23T00:31:54.003236+0000","last_clean":"2024-02-23T00:31:54.003236+0000","last_became_active":"2024-02-23T00:31:35.007804+0000","last_became_peered":"2024-02-23T00:31:35.007804+0000","last_unstale":"2024-02-23T00:31:54.003236+0000","last_undegraded":"2024-02-23T00:31:54.003236+0000","last_fullsized":"2024-02-23T00:31:54.003236+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'4","last_scrub_stamp":"2024-02-23T00:31:43.690731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:43.690731+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"95.5","version":"151'16","reported_seq":62,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.991689+0000","last_change":"2024-02-23T00:31:43.942468+0000","last_active":"2024-02-23T00:31:53.991689+0000","last_peered":"2024-02-23T00:31:53.991689+0000","last_clean":"2024-02-23T00:31:53.991689+0000","last_became_active":"2024-02-23T00:31:34.997786+0000","last_became_peered":"2024-02-23T00:31:34.997786+0000","last_unstale":"2024-02-23T00:31:53.991689+0000","last_undegraded":"2024-02-23T00:31:53.991689+0000","last_fullsized":"2024-02-23T00:31:53.991689+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'16","last_scrub_stamp":"2024-02-23T00:31:43.942401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:43.942401+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":170,"num_objects":16,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":16,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.6","version":"151'7","reported_seq":98,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986956+0000","last_change":"2024-02-23T00:31:45.025227+0000","last_active":"2024-02-23T00:31:53.986956+0000","last_peered":"2024-02-23T00:31:53.986956+0000","last_clean":"2024-02-23T00:31:53.986956+0000","last_became_active":"2024-02-23T00:31:35.005902+0000","last_became_peered":"2024-02-23T00:31:35.005902+0000","last_unstale":"2024-02-23T00:31:53.986956+0000","last_undegraded":"2024-02-23T00:31:53.986956+0000","last_fullsized":"2024-02-23T00:31:53.986956+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'7","last_scrub_stamp":"2024-02-23T00:31:45.025156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:45.025156+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":57,"num_read_kb":57,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.7","version":"151'9","reported_seq":50,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.005690+0000","last_change":"2024-02-23T00:31:45.900247+0000","last_active":"2024-02-23T00:31:54.005690+0000","last_peered":"2024-02-23T00:31:54.005690+0000","last_clean":"2024-02-23T00:31:54.005690+0000","last_became_active":"2024-02-23T00:31:35.021720+0000","last_became_peered":"2024-02-23T00:31:35.021720+0000","last_unstale":"2024-02-23T00:31:54.005690+0000","last_undegraded":"2024-02-23T00:31:54.005690+0000","last_fullsized":"2024-02-23T00:31:54.005690+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'9","last_scrub_stamp":"2024-02-23T00:31:45.900152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:45.900152+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.8","version":"151'11","reported_seq":52,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.985632+0000","last_change":"2024-02-23T00:31:44.833764+0000","last_active":"2024-02-23T00:31:53.985632+0000","last_peered":"2024-02-23T00:31:53.985632+0000","last_clean":"2024-02-23T00:31:53.985632+0000","last_became_active":"2024-02-23T00:31:35.024709+0000","last_became_peered":"2024-02-23T00:31:35.024709+0000","last_unstale":"2024-02-23T00:31:53.985632+0000","last_undegraded":"2024-02-23T00:31:53.985632+0000","last_fullsized":"2024-02-23T00:31:53.985632+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'11","last_scrub_stamp":"2024-02-23T00:31:44.833693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:44.833693+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":115,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.9","version":"151'3","reported_seq":40,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006646+0000","last_change":"2024-02-23T00:31:47.756564+0000","last_active":"2024-02-23T00:31:54.006646+0000","last_peered":"2024-02-23T00:31:54.006646+0000","last_clean":"2024-02-23T00:31:54.006646+0000","last_became_active":"2024-02-23T00:31:35.010190+0000","last_became_peered":"2024-02-23T00:31:35.010190+0000","last_unstale":"2024-02-23T00:31:54.006646+0000","last_undegraded":"2024-02-23T00:31:54.006646+0000","last_fullsized":"2024-02-23T00:31:54.006646+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'3","last_scrub_stamp":"2024-02-23T00:31:47.756452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:47.756452+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"95.a","version":"151'3","reported_seq":34,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.984596+0000","last_change":"2024-02-23T00:31:35.016212+0000","last_active":"2024-02-23T00:31:53.984596+0000","last_peered":"2024-02-23T00:31:53.984596+0000","last_clean":"2024-02-23T00:31:53.984596+0000","last_became_active":"2024-02-23T00:31:35.015881+0000","last_became_peered":"2024-02-23T00:31:35.015881+0000","last_unstale":"2024-02-23T00:31:53.984596+0000","last_undegraded":"2024-02-23T00:31:53.984596+0000","last_fullsized":"2024-02-23T00:31:53.984596+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.b","version":"151'5","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.983951+0000","last_change":"2024-02-23T00:31:35.015276+0000","last_active":"2024-02-23T00:31:53.983951+0000","last_peered":"2024-02-23T00:31:53.983951+0000","last_clean":"2024-02-23T00:31:53.983951+0000","last_became_active":"2024-02-23T00:31:35.014983+0000","last_became_peered":"2024-02-23T00:31:35.014983+0000","last_unstale":"2024-02-23T00:31:53.983951+0000","last_undegraded":"2024-02-23T00:31:53.983951+0000","last_fullsized":"2024-02-23T00:31:53.983951+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.c","version":"151'9","reported_seq":44,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.985810+0000","last_change":"2024-02-23T00:31:35.017130+0000","last_active":"2024-02-23T00:31:53.985810+0000","last_peered":"2024-02-23T00:31:53.985810+0000","last_clean":"2024-02-23T00:31:53.985810+0000","last_became_active":"2024-02-23T00:31:35.015146+0000","last_became_peered":"2024-02-23T00:31:35.015146+0000","last_unstale":"2024-02-23T00:31:53.985810+0000","last_undegraded":"2024-02-23T00:31:53.985810+0000","last_fullsized":"2024-02-23T00:31:53.985810+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.d","version":"151'4","reported_seq":36,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993013+0000","last_change":"2024-02-23T00:31:35.007988+0000","last_active":"2024-02-23T00:31:53.993013+0000","last_peered":"2024-02-23T00:31:53.993013+0000","last_clean":"2024-02-23T00:31:53.993013+0000","last_became_active":"2024-02-23T00:31:35.007665+0000","last_became_peered":"2024-02-23T00:31:35.007665+0000","last_unstale":"2024-02-23T00:31:53.993013+0000","last_undegraded":"2024-02-23T00:31:53.993013+0000","last_fullsized":"2024-02-23T00:31:53.993013+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.e","version":"151'12","reported_seq":49,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003112+0000","last_change":"2024-02-23T00:31:35.020943+0000","last_active":"2024-02-23T00:31:54.003112+0000","last_peered":"2024-02-23T00:31:54.003112+0000","last_clean":"2024-02-23T00:31:54.003112+0000","last_became_active":"2024-02-23T00:31:35.020571+0000","last_became_peered":"2024-02-23T00:31:35.020571+0000","last_unstale":"2024-02-23T00:31:54.003112+0000","last_undegraded":"2024-02-23T00:31:54.003112+0000","last_fullsized":"2024-02-23T00:31:54.003112+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"95.f","version":"151'9","reported_seq":46,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988375+0000","last_change":"2024-02-23T00:31:35.014640+0000","last_active":"2024-02-23T00:31:53.988375+0000","last_peered":"2024-02-23T00:31:53.988375+0000","last_clean":"2024-02-23T00:31:53.988375+0000","last_became_active":"2024-02-23T00:31:35.014323+0000","last_became_peered":"2024-02-23T00:31:35.014323+0000","last_unstale":"2024-02-23T00:31:53.988375+0000","last_undegraded":"2024-02-23T00:31:53.988375+0000","last_fullsized":"2024-02-23T00:31:53.988375+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"95.10","version":"151'6","reported_seq":39,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989430+0000","last_change":"2024-02-23T00:31:35.005193+0000","last_active":"2024-02-23T00:31:53.989430+0000","last_peered":"2024-02-23T00:31:53.989430+0000","last_clean":"2024-02-23T00:31:53.989430+0000","last_became_active":"2024-02-23T00:31:35.004846+0000","last_became_peered":"2024-02-23T00:31:35.004846+0000","last_unstale":"2024-02-23T00:31:53.989430+0000","last_undegraded":"2024-02-23T00:31:53.989430+0000","last_fullsized":"2024-02-23T00:31:53.989430+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.11","version":"0'0","reported_seq":29,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.015473+0000","last_change":"2024-02-23T00:31:35.027636+0000","last_active":"2024-02-23T00:31:54.015473+0000","last_peered":"2024-02-23T00:31:54.015473+0000","last_clean":"2024-02-23T00:31:54.015473+0000","last_became_active":"2024-02-23T00:31:35.025517+0000","last_became_peered":"2024-02-23T00:31:35.025517+0000","last_unstale":"2024-02-23T00:31:54.015473+0000","last_undegraded":"2024-02-23T00:31:54.015473+0000","last_fullsized":"2024-02-23T00:31:54.015473+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.12","version":"151'5","reported_seq":37,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.987006+0000","last_change":"2024-02-23T00:31:35.018447+0000","last_active":"2024-02-23T00:31:53.987006+0000","last_peered":"2024-02-23T00:31:53.987006+0000","last_clean":"2024-02-23T00:31:53.987006+0000","last_became_active":"2024-02-23T00:31:35.018129+0000","last_became_peered":"2024-02-23T00:31:35.018129+0000","last_unstale":"2024-02-23T00:31:53.987006+0000","last_undegraded":"2024-02-23T00:31:53.987006+0000","last_fullsized":"2024-02-23T00:31:53.987006+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.13","version":"151'3","reported_seq":35,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.002634+0000","last_change":"2024-02-23T00:31:35.015167+0000","last_active":"2024-02-23T00:31:54.002634+0000","last_peered":"2024-02-23T00:31:54.002634+0000","last_clean":"2024-02-23T00:31:54.002634+0000","last_became_active":"2024-02-23T00:31:35.014877+0000","last_became_peered":"2024-02-23T00:31:35.014877+0000","last_unstale":"2024-02-23T00:31:54.002634+0000","last_undegraded":"2024-02-23T00:31:54.002634+0000","last_fullsized":"2024-02-23T00:31:54.002634+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"95.14","version":"151'9","reported_seq":45,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986331+0000","last_change":"2024-02-23T00:31:35.017258+0000","last_active":"2024-02-23T00:31:53.986331+0000","last_peered":"2024-02-23T00:31:53.986331+0000","last_clean":"2024-02-23T00:31:53.986331+0000","last_became_active":"2024-02-23T00:31:35.016593+0000","last_became_peered":"2024-02-23T00:31:35.016593+0000","last_unstale":"2024-02-23T00:31:53.986331+0000","last_undegraded":"2024-02-23T00:31:53.986331+0000","last_fullsized":"2024-02-23T00:31:53.986331+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":9,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.15","version":"151'11","reported_seq":51,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.011008+0000","last_change":"2024-02-23T00:31:35.026913+0000","last_active":"2024-02-23T00:31:54.011008+0000","last_peered":"2024-02-23T00:31:54.011008+0000","last_clean":"2024-02-23T00:31:54.011008+0000","last_became_active":"2024-02-23T00:31:35.026646+0000","last_became_peered":"2024-02-23T00:31:35.026646+0000","last_unstale":"2024-02-23T00:31:54.011008+0000","last_undegraded":"2024-02-23T00:31:54.011008+0000","last_fullsized":"2024-02-23T00:31:54.011008+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.16","version":"151'8","reported_seq":44,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989911+0000","last_change":"2024-02-23T00:31:35.020953+0000","last_active":"2024-02-23T00:31:53.989911+0000","last_peered":"2024-02-23T00:31:53.989911+0000","last_clean":"2024-02-23T00:31:53.989911+0000","last_became_active":"2024-02-23T00:31:35.006830+0000","last_became_peered":"2024-02-23T00:31:35.006830+0000","last_unstale":"2024-02-23T00:31:53.989911+0000","last_undegraded":"2024-02-23T00:31:53.989911+0000","last_fullsized":"2024-02-23T00:31:53.989911+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"95.17","version":"151'12","reported_seq":50,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989809+0000","last_change":"2024-02-23T00:31:35.008738+0000","last_active":"2024-02-23T00:31:53.989809+0000","last_peered":"2024-02-23T00:31:53.989809+0000","last_clean":"2024-02-23T00:31:53.989809+0000","last_became_active":"2024-02-23T00:31:35.008314+0000","last_became_peered":"2024-02-23T00:31:35.008314+0000","last_unstale":"2024-02-23T00:31:53.989809+0000","last_undegraded":"2024-02-23T00:31:53.989809+0000","last_fullsized":"2024-02-23T00:31:53.989809+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":126,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"95.18","version":"151'5","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993365+0000","last_change":"2024-02-23T00:31:35.014386+0000","last_active":"2024-02-23T00:31:53.993365+0000","last_peered":"2024-02-23T00:31:53.993365+0000","last_clean":"2024-02-23T00:31:53.993365+0000","last_became_active":"2024-02-23T00:31:35.013308+0000","last_became_peered":"2024-02-23T00:31:35.013308+0000","last_unstale":"2024-02-23T00:31:53.993365+0000","last_undegraded":"2024-02-23T00:31:53.993365+0000","last_fullsized":"2024-02-23T00:31:53.993365+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.19","version":"151'10","reported_seq":47,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.015390+0000","last_change":"2024-02-23T00:31:35.025791+0000","last_active":"2024-02-23T00:31:54.015390+0000","last_peered":"2024-02-23T00:31:54.015390+0000","last_clean":"2024-02-23T00:31:54.015390+0000","last_became_active":"2024-02-23T00:31:35.021002+0000","last_became_peered":"2024-02-23T00:31:35.021002+0000","last_unstale":"2024-02-23T00:31:54.015390+0000","last_undegraded":"2024-02-23T00:31:54.015390+0000","last_fullsized":"2024-02-23T00:31:54.015390+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"95.1a","version":"151'8","reported_seq":43,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.009092+0000","last_change":"2024-02-23T00:31:35.007306+0000","last_active":"2024-02-23T00:31:54.009092+0000","last_peered":"2024-02-23T00:31:54.009092+0000","last_clean":"2024-02-23T00:31:54.009092+0000","last_became_active":"2024-02-23T00:31:35.006925+0000","last_became_peered":"2024-02-23T00:31:35.006925+0000","last_unstale":"2024-02-23T00:31:54.009092+0000","last_undegraded":"2024-02-23T00:31:54.009092+0000","last_fullsized":"2024-02-23T00:31:54.009092+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"95.1b","version":"151'6","reported_seq":39,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990590+0000","last_change":"2024-02-23T00:31:35.008855+0000","last_active":"2024-02-23T00:31:53.990590+0000","last_peered":"2024-02-23T00:31:53.990590+0000","last_clean":"2024-02-23T00:31:53.990590+0000","last_became_active":"2024-02-23T00:31:35.008318+0000","last_became_peered":"2024-02-23T00:31:35.008318+0000","last_unstale":"2024-02-23T00:31:53.990590+0000","last_undegraded":"2024-02-23T00:31:53.990590+0000","last_fullsized":"2024-02-23T00:31:53.990590+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.1c","version":"151'5","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988760+0000","last_change":"2024-02-23T00:31:35.023480+0000","last_active":"2024-02-23T00:31:53.988760+0000","last_peered":"2024-02-23T00:31:53.988760+0000","last_clean":"2024-02-23T00:31:53.988760+0000","last_became_active":"2024-02-23T00:31:35.023202+0000","last_became_peered":"2024-02-23T00:31:35.023202+0000","last_unstale":"2024-02-23T00:31:53.988760+0000","last_undegraded":"2024-02-23T00:31:53.988760+0000","last_fullsized":"2024-02-23T00:31:53.988760+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"95.1d","version":"0'0","reported_seq":29,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.991607+0000","last_change":"2024-02-23T00:31:35.000486+0000","last_active":"2024-02-23T00:31:53.991607+0000","last_peered":"2024-02-23T00:31:53.991607+0000","last_clean":"2024-02-23T00:31:53.991607+0000","last_became_active":"2024-02-23T00:31:35.000189+0000","last_became_peered":"2024-02-23T00:31:35.000189+0000","last_unstale":"2024-02-23T00:31:53.991607+0000","last_undegraded":"2024-02-23T00:31:53.991607+0000","last_fullsized":"2024-02-23T00:31:53.991607+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.1e","version":"151'4","reported_seq":37,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.987892+0000","last_change":"2024-02-23T00:31:35.004152+0000","last_active":"2024-02-23T00:31:53.987892+0000","last_peered":"2024-02-23T00:31:53.987892+0000","last_clean":"2024-02-23T00:31:53.987892+0000","last_became_active":"2024-02-23T00:31:35.003831+0000","last_became_peered":"2024-02-23T00:31:35.003831+0000","last_unstale":"2024-02-23T00:31:53.987892+0000","last_undegraded":"2024-02-23T00:31:53.987892+0000","last_fullsized":"2024-02-23T00:31:53.987892+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"95.1f","version":"151'11","reported_seq":49,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989694+0000","last_change":"2024-02-23T00:31:35.015569+0000","last_active":"2024-02-23T00:31:53.989694+0000","last_peered":"2024-02-23T00:31:53.989694+0000","last_clean":"2024-02-23T00:31:53.989694+0000","last_became_active":"2024-02-23T00:31:35.014977+0000","last_became_peered":"2024-02-23T00:31:35.014977+0000","last_unstale":"2024-02-23T00:31:53.989694+0000","last_undegraded":"2024-02-23T00:31:53.989694+0000","last_fullsized":"2024-02-23T00:31:53.989694+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:33.970418+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.970418+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.0","version":"148'36","reported_seq":99,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986087+0000","last_change":"2024-02-23T00:31:32.134755+0000","last_active":"2024-02-23T00:31:53.986087+0000","last_peered":"2024-02-23T00:31:53.986087+0000","last_clean":"2024-02-23T00:31:53.986087+0000","last_became_active":"2024-02-23T00:31:06.330477+0000","last_became_peered":"2024-02-23T00:31:06.330477+0000","last_unstale":"2024-02-23T00:31:53.986087+0000","last_undegraded":"2024-02-23T00:31:53.986087+0000","last_fullsized":"2024-02-23T00:31:53.986087+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1","version":"148'45","reported_seq":101,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996904+0000","last_change":"2024-02-23T00:31:30.039304+0000","last_active":"2024-02-23T00:31:53.996904+0000","last_peered":"2024-02-23T00:31:53.996904+0000","last_clean":"2024-02-23T00:31:53.996904+0000","last_became_active":"2024-02-23T00:31:06.258140+0000","last_became_peered":"2024-02-23T00:31:06.258140+0000","last_unstale":"2024-02-23T00:31:53.996904+0000","last_undegraded":"2024-02-23T00:31:53.996904+0000","last_fullsized":"2024-02-23T00:31:53.996904+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.2","version":"148'10","reported_seq":74,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989017+0000","last_change":"2024-02-23T00:31:30.103692+0000","last_active":"2024-02-23T00:31:53.989017+0000","last_peered":"2024-02-23T00:31:53.989017+0000","last_clean":"2024-02-23T00:31:53.989017+0000","last_became_active":"2024-02-23T00:31:06.329726+0000","last_became_peered":"2024-02-23T00:31:06.329726+0000","last_unstale":"2024-02-23T00:31:53.989017+0000","last_undegraded":"2024-02-23T00:31:53.989017+0000","last_fullsized":"2024-02-23T00:31:53.989017+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.3","version":"148'9","reported_seq":72,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.984318+0000","last_change":"2024-02-23T00:31:30.048671+0000","last_active":"2024-02-23T00:31:53.984318+0000","last_peered":"2024-02-23T00:31:53.984318+0000","last_clean":"2024-02-23T00:31:53.984318+0000","last_became_active":"2024-02-23T00:31:06.258997+0000","last_became_peered":"2024-02-23T00:31:06.258997+0000","last_unstale":"2024-02-23T00:31:53.984318+0000","last_undegraded":"2024-02-23T00:31:53.984318+0000","last_fullsized":"2024-02-23T00:31:53.984318+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.4","version":"148'18","reported_seq":79,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986448+0000","last_change":"2024-02-23T00:31:30.059398+0000","last_active":"2024-02-23T00:31:53.986448+0000","last_peered":"2024-02-23T00:31:53.986448+0000","last_clean":"2024-02-23T00:31:53.986448+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:31:53.986448+0000","last_undegraded":"2024-02-23T00:31:53.986448+0000","last_fullsized":"2024-02-23T00:31:53.986448+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.5","version":"148'63","reported_seq":120,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006844+0000","last_change":"2024-02-23T00:31:30.041685+0000","last_active":"2024-02-23T00:31:54.006844+0000","last_peered":"2024-02-23T00:31:54.006844+0000","last_clean":"2024-02-23T00:31:54.006844+0000","last_became_active":"2024-02-23T00:31:06.258422+0000","last_became_peered":"2024-02-23T00:31:06.258422+0000","last_unstale":"2024-02-23T00:31:54.006844+0000","last_undegraded":"2024-02-23T00:31:54.006844+0000","last_fullsized":"2024-02-23T00:31:54.006844+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":63,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":364,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.6","version":"148'27","reported_seq":87,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990156+0000","last_change":"2024-02-23T00:31:30.105256+0000","last_active":"2024-02-23T00:31:53.990156+0000","last_peered":"2024-02-23T00:31:53.990156+0000","last_clean":"2024-02-23T00:31:53.990156+0000","last_became_active":"2024-02-23T00:31:06.313441+0000","last_became_peered":"2024-02-23T00:31:06.313441+0000","last_unstale":"2024-02-23T00:31:53.990156+0000","last_undegraded":"2024-02-23T00:31:53.990156+0000","last_fullsized":"2024-02-23T00:31:53.990156+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.7","version":"148'45","reported_seq":100,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010467+0000","last_change":"2024-02-23T00:31:32.134516+0000","last_active":"2024-02-23T00:31:54.010467+0000","last_peered":"2024-02-23T00:31:54.010467+0000","last_clean":"2024-02-23T00:31:54.010467+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:31:54.010467+0000","last_undegraded":"2024-02-23T00:31:54.010467+0000","last_fullsized":"2024-02-23T00:31:54.010467+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.8","version":"148'36","reported_seq":98,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.985925+0000","last_change":"2024-02-23T00:31:30.063925+0000","last_active":"2024-02-23T00:31:53.985925+0000","last_peered":"2024-02-23T00:31:53.985925+0000","last_clean":"2024-02-23T00:31:53.985925+0000","last_became_active":"2024-02-23T00:31:06.363442+0000","last_became_peered":"2024-02-23T00:31:06.363442+0000","last_unstale":"2024-02-23T00:31:53.985925+0000","last_undegraded":"2024-02-23T00:31:53.985925+0000","last_fullsized":"2024-02-23T00:31:53.985925+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.9","version":"148'9","reported_seq":72,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.015675+0000","last_change":"2024-02-23T00:31:32.134074+0000","last_active":"2024-02-23T00:31:54.015675+0000","last_peered":"2024-02-23T00:31:54.015675+0000","last_clean":"2024-02-23T00:31:54.015675+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:31:54.015675+0000","last_undegraded":"2024-02-23T00:31:54.015675+0000","last_fullsized":"2024-02-23T00:31:54.015675+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.a","version":"148'9","reported_seq":72,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.983954+0000","last_change":"2024-02-23T00:31:30.047066+0000","last_active":"2024-02-23T00:31:53.983954+0000","last_peered":"2024-02-23T00:31:53.983954+0000","last_clean":"2024-02-23T00:31:53.983954+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:31:53.983954+0000","last_undegraded":"2024-02-23T00:31:53.983954+0000","last_fullsized":"2024-02-23T00:31:53.983954+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.b","version":"148'18","reported_seq":80,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.999982+0000","last_change":"2024-02-23T00:31:32.058828+0000","last_active":"2024-02-23T00:31:53.999982+0000","last_peered":"2024-02-23T00:31:53.999982+0000","last_clean":"2024-02-23T00:31:53.999982+0000","last_became_active":"2024-02-23T00:31:06.294667+0000","last_became_peered":"2024-02-23T00:31:06.294667+0000","last_unstale":"2024-02-23T00:31:53.999982+0000","last_undegraded":"2024-02-23T00:31:53.999982+0000","last_fullsized":"2024-02-23T00:31:53.999982+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.c","version":"148'36","reported_seq":95,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988986+0000","last_change":"2024-02-23T00:31:30.049255+0000","last_active":"2024-02-23T00:31:53.988986+0000","last_peered":"2024-02-23T00:31:53.988986+0000","last_clean":"2024-02-23T00:31:53.988986+0000","last_became_active":"2024-02-23T00:31:06.233844+0000","last_became_peered":"2024-02-23T00:31:06.233844+0000","last_unstale":"2024-02-23T00:31:53.988986+0000","last_undegraded":"2024-02-23T00:31:53.988986+0000","last_fullsized":"2024-02-23T00:31:53.988986+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.d","version":"148'18","reported_seq":78,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.985476+0000","last_change":"2024-02-23T00:31:30.053419+0000","last_active":"2024-02-23T00:31:53.985476+0000","last_peered":"2024-02-23T00:31:53.985476+0000","last_clean":"2024-02-23T00:31:53.985476+0000","last_became_active":"2024-02-23T00:31:06.262251+0000","last_became_peered":"2024-02-23T00:31:06.262251+0000","last_unstale":"2024-02-23T00:31:53.985476+0000","last_undegraded":"2024-02-23T00:31:53.985476+0000","last_fullsized":"2024-02-23T00:31:53.985476+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.e","version":"148'41","reported_seq":106,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.004834+0000","last_change":"2024-02-23T00:31:32.134067+0000","last_active":"2024-02-23T00:31:54.004834+0000","last_peered":"2024-02-23T00:31:54.004834+0000","last_clean":"2024-02-23T00:31:54.004834+0000","last_became_active":"2024-02-23T00:31:06.356847+0000","last_became_peered":"2024-02-23T00:31:06.356847+0000","last_unstale":"2024-02-23T00:31:54.004834+0000","last_undegraded":"2024-02-23T00:31:54.004834+0000","last_fullsized":"2024-02-23T00:31:54.004834+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":213,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.f","version":"148'36","reported_seq":95,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989099+0000","last_change":"2024-02-23T00:31:30.099890+0000","last_active":"2024-02-23T00:31:53.989099+0000","last_peered":"2024-02-23T00:31:53.989099+0000","last_clean":"2024-02-23T00:31:53.989099+0000","last_became_active":"2024-02-23T00:31:06.364609+0000","last_became_peered":"2024-02-23T00:31:06.364609+0000","last_unstale":"2024-02-23T00:31:53.989099+0000","last_undegraded":"2024-02-23T00:31:53.989099+0000","last_fullsized":"2024-02-23T00:31:53.989099+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.10","version":"148'18","reported_seq":81,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986003+0000","last_change":"2024-02-23T00:31:32.134713+0000","last_active":"2024-02-23T00:31:53.986003+0000","last_peered":"2024-02-23T00:31:53.986003+0000","last_clean":"2024-02-23T00:31:53.986003+0000","last_became_active":"2024-02-23T00:31:06.342164+0000","last_became_peered":"2024-02-23T00:31:06.342164+0000","last_unstale":"2024-02-23T00:31:53.986003+0000","last_undegraded":"2024-02-23T00:31:53.986003+0000","last_fullsized":"2024-02-23T00:31:53.986003+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.11","version":"148'24","reported_seq":81,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996811+0000","last_change":"2024-02-23T00:31:32.062975+0000","last_active":"2024-02-23T00:31:53.996811+0000","last_peered":"2024-02-23T00:31:53.996811+0000","last_clean":"2024-02-23T00:31:53.996811+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:31:53.996811+0000","last_undegraded":"2024-02-23T00:31:53.996811+0000","last_fullsized":"2024-02-23T00:31:53.996811+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":113,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.12","version":"148'36","reported_seq":90,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993228+0000","last_change":"2024-02-23T00:31:30.044832+0000","last_active":"2024-02-23T00:31:53.993228+0000","last_peered":"2024-02-23T00:31:53.993228+0000","last_clean":"2024-02-23T00:31:53.993228+0000","last_became_active":"2024-02-23T00:31:06.281619+0000","last_became_peered":"2024-02-23T00:31:06.281619+0000","last_unstale":"2024-02-23T00:31:53.993228+0000","last_undegraded":"2024-02-23T00:31:53.993228+0000","last_fullsized":"2024-02-23T00:31:53.993228+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.13","version":"148'32","reported_seq":90,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.999865+0000","last_change":"2024-02-23T00:31:32.085929+0000","last_active":"2024-02-23T00:31:53.999865+0000","last_peered":"2024-02-23T00:31:53.999865+0000","last_clean":"2024-02-23T00:31:53.999865+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:31:53.999865+0000","last_undegraded":"2024-02-23T00:31:53.999865+0000","last_fullsized":"2024-02-23T00:31:53.999865+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":161,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.14","version":"148'45","reported_seq":100,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988886+0000","last_change":"2024-02-23T00:31:30.048990+0000","last_active":"2024-02-23T00:31:53.988886+0000","last_peered":"2024-02-23T00:31:53.988886+0000","last_clean":"2024-02-23T00:31:53.988886+0000","last_became_active":"2024-02-23T00:31:06.234521+0000","last_became_peered":"2024-02-23T00:31:06.234521+0000","last_unstale":"2024-02-23T00:31:53.988886+0000","last_undegraded":"2024-02-23T00:31:53.988886+0000","last_fullsized":"2024-02-23T00:31:53.988886+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.15","version":"148'36","reported_seq":98,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006731+0000","last_change":"2024-02-23T00:31:30.041396+0000","last_active":"2024-02-23T00:31:54.006731+0000","last_peered":"2024-02-23T00:31:54.006731+0000","last_clean":"2024-02-23T00:31:54.006731+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:31:54.006731+0000","last_undegraded":"2024-02-23T00:31:54.006731+0000","last_fullsized":"2024-02-23T00:31:54.006731+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.16","version":"148'27","reported_seq":88,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990065+0000","last_change":"2024-02-23T00:31:30.095851+0000","last_active":"2024-02-23T00:31:53.990065+0000","last_peered":"2024-02-23T00:31:53.990065+0000","last_clean":"2024-02-23T00:31:53.990065+0000","last_became_active":"2024-02-23T00:31:06.367069+0000","last_became_peered":"2024-02-23T00:31:06.367069+0000","last_unstale":"2024-02-23T00:31:53.990065+0000","last_undegraded":"2024-02-23T00:31:53.990065+0000","last_fullsized":"2024-02-23T00:31:53.990065+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.17","version":"148'45","reported_seq":105,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990145+0000","last_change":"2024-02-23T00:31:30.059346+0000","last_active":"2024-02-23T00:31:53.990145+0000","last_peered":"2024-02-23T00:31:53.990145+0000","last_clean":"2024-02-23T00:31:53.990145+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:31:53.990145+0000","last_undegraded":"2024-02-23T00:31:53.990145+0000","last_fullsized":"2024-02-23T00:31:53.990145+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.18","version":"148'32","reported_seq":91,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006008+0000","last_change":"2024-02-23T00:31:32.058614+0000","last_active":"2024-02-23T00:31:54.006008+0000","last_peered":"2024-02-23T00:31:54.006008+0000","last_clean":"2024-02-23T00:31:54.006008+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:31:54.006008+0000","last_undegraded":"2024-02-23T00:31:54.006008+0000","last_fullsized":"2024-02-23T00:31:54.006008+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.19","version":"148'27","reported_seq":91,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996714+0000","last_change":"2024-02-23T00:31:30.037526+0000","last_active":"2024-02-23T00:31:53.996714+0000","last_peered":"2024-02-23T00:31:53.996714+0000","last_clean":"2024-02-23T00:31:53.996714+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:31:53.996714+0000","last_undegraded":"2024-02-23T00:31:53.996714+0000","last_fullsized":"2024-02-23T00:31:53.996714+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.1a","version":"148'36","reported_seq":94,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996272+0000","last_change":"2024-02-23T00:31:30.086779+0000","last_active":"2024-02-23T00:31:53.996272+0000","last_peered":"2024-02-23T00:31:53.996272+0000","last_clean":"2024-02-23T00:31:53.996272+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:31:53.996272+0000","last_undegraded":"2024-02-23T00:31:53.996272+0000","last_fullsized":"2024-02-23T00:31:53.996272+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.1b","version":"148'27","reported_seq":86,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.984200+0000","last_change":"2024-02-23T00:31:30.048450+0000","last_active":"2024-02-23T00:31:53.984200+0000","last_peered":"2024-02-23T00:31:53.984200+0000","last_clean":"2024-02-23T00:31:53.984200+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:31:53.984200+0000","last_undegraded":"2024-02-23T00:31:53.984200+0000","last_fullsized":"2024-02-23T00:31:53.984200+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1c","version":"148'18","reported_seq":80,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.019639+0000","last_change":"2024-02-23T00:31:30.104026+0000","last_active":"2024-02-23T00:31:54.019639+0000","last_peered":"2024-02-23T00:31:54.019639+0000","last_clean":"2024-02-23T00:31:54.019639+0000","last_became_active":"2024-02-23T00:31:06.338540+0000","last_became_peered":"2024-02-23T00:31:06.338540+0000","last_unstale":"2024-02-23T00:31:54.019639+0000","last_undegraded":"2024-02-23T00:31:54.019639+0000","last_fullsized":"2024-02-23T00:31:54.019639+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1d","version":"0'0","reported_seq":63,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.011209+0000","last_change":"2024-02-23T00:31:30.093144+0000","last_active":"2024-02-23T00:31:54.011209+0000","last_peered":"2024-02-23T00:31:54.011209+0000","last_clean":"2024-02-23T00:31:54.011209+0000","last_became_active":"2024-02-23T00:31:06.337635+0000","last_became_peered":"2024-02-23T00:31:06.337635+0000","last_unstale":"2024-02-23T00:31:54.011209+0000","last_undegraded":"2024-02-23T00:31:54.011209+0000","last_fullsized":"2024-02-23T00:31:54.011209+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1e","version":"148'27","reported_seq":84,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990340+0000","last_change":"2024-02-23T00:31:30.052047+0000","last_active":"2024-02-23T00:31:53.990340+0000","last_peered":"2024-02-23T00:31:53.990340+0000","last_clean":"2024-02-23T00:31:53.990340+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:31:53.990340+0000","last_undegraded":"2024-02-23T00:31:53.990340+0000","last_fullsized":"2024-02-23T00:31:53.990340+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1f","version":"148'36","reported_seq":98,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990020+0000","last_change":"2024-02-23T00:31:30.059171+0000","last_active":"2024-02-23T00:31:53.990020+0000","last_peered":"2024-02-23T00:31:53.990020+0000","last_clean":"2024-02-23T00:31:53.990020+0000","last_became_active":"2024-02-23T00:31:06.355623+0000","last_became_peered":"2024-02-23T00:31:06.355623+0000","last_unstale":"2024-02-23T00:31:53.990020+0000","last_undegraded":"2024-02-23T00:31:53.990020+0000","last_fullsized":"2024-02-23T00:31:53.990020+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'8","reported_seq":187,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.991014+0000","last_change":"2024-02-23T00:29:48.485677+0000","last_active":"2024-02-23T00:31:54.991014+0000","last_peered":"2024-02-23T00:31:54.991014+0000","last_clean":"2024-02-23T00:31:54.991014+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:31:54.991014+0000","last_undegraded":"2024-02-23T00:31:54.991014+0000","last_fullsized":"2024-02-23T00:31:54.991014+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:29:48.485586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:48.485586+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":188,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.983208+0000","last_change":"2024-02-23T00:30:25.054238+0000","last_active":"2024-02-23T00:31:53.983208+0000","last_peered":"2024-02-23T00:31:53.983208+0000","last_clean":"2024-02-23T00:31:53.983208+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:31:53.983208+0000","last_undegraded":"2024-02-23T00:31:53.983208+0000","last_fullsized":"2024-02-23T00:31:53.983208+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:30:25.054140+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.054140+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":133,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990375+0000","last_change":"2024-02-23T00:31:28.776880+0000","last_active":"2024-02-23T00:31:53.990375+0000","last_peered":"2024-02-23T00:31:53.990375+0000","last_clean":"2024-02-23T00:31:53.990375+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:31:53.990375+0000","last_undegraded":"2024-02-23T00:31:53.990375+0000","last_fullsized":"2024-02-23T00:31:53.990375+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:28.776788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:28.776788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":120,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.984137+0000","last_change":"2024-02-23T00:31:31.323024+0000","last_active":"2024-02-23T00:31:53.984137+0000","last_peered":"2024-02-23T00:31:53.984137+0000","last_clean":"2024-02-23T00:31:53.984137+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:31:53.984137+0000","last_undegraded":"2024-02-23T00:31:53.984137+0000","last_fullsized":"2024-02-23T00:31:53.984137+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.322940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.322940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003301+0000","last_change":"2024-02-23T00:30:48.782537+0000","last_active":"2024-02-23T00:31:54.003301+0000","last_peered":"2024-02-23T00:31:54.003301+0000","last_clean":"2024-02-23T00:31:54.003301+0000","last_became_active":"2024-02-23T00:30:48.777752+0000","last_became_peered":"2024-02-23T00:30:48.777752+0000","last_unstale":"2024-02-23T00:31:54.003301+0000","last_undegraded":"2024-02-23T00:31:54.003301+0000","last_fullsized":"2024-02-23T00:31:54.003301+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":68,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.997311+0000","last_change":"2024-02-23T00:31:29.092048+0000","last_active":"2024-02-23T00:31:54.997311+0000","last_peered":"2024-02-23T00:31:54.997311+0000","last_clean":"2024-02-23T00:31:54.997311+0000","last_became_active":"2024-02-23T00:31:29.077539+0000","last_became_peered":"2024-02-23T00:31:29.077539+0000","last_unstale":"2024-02-23T00:31:54.997311+0000","last_undegraded":"2024-02-23T00:31:54.997311+0000","last_fullsized":"2024-02-23T00:31:54.997311+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:30:38.432677+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.432677+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":87,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":157,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003208+0000","last_change":"2024-02-23T00:30:33.906112+0000","last_active":"2024-02-23T00:31:54.003208+0000","last_peered":"2024-02-23T00:31:54.003208+0000","last_clean":"2024-02-23T00:31:54.003208+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:31:54.003208+0000","last_undegraded":"2024-02-23T00:31:54.003208+0000","last_fullsized":"2024-02-23T00:31:54.003208+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:30:33.906033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:33.906033+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.023100+0000","last_change":"2024-02-23T00:29:48.458818+0000","last_active":"2024-02-23T00:31:54.023100+0000","last_peered":"2024-02-23T00:31:54.023100+0000","last_clean":"2024-02-23T00:31:54.023100+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:31:54.023100+0000","last_undegraded":"2024-02-23T00:31:54.023100+0000","last_fullsized":"2024-02-23T00:31:54.023100+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:48.458726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:48.458726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996622+0000","last_change":"2024-02-23T00:30:38.736599+0000","last_active":"2024-02-23T00:31:53.996622+0000","last_peered":"2024-02-23T00:31:53.996622+0000","last_clean":"2024-02-23T00:31:53.996622+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:31:53.996622+0000","last_undegraded":"2024-02-23T00:31:53.996622+0000","last_fullsized":"2024-02-23T00:31:53.996622+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:38.736492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.736492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.007634+0000","last_change":"2024-02-23T00:30:32.953255+0000","last_active":"2024-02-23T00:31:54.007634+0000","last_peered":"2024-02-23T00:31:54.007634+0000","last_clean":"2024-02-23T00:31:54.007634+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:31:54.007634+0000","last_undegraded":"2024-02-23T00:31:54.007634+0000","last_fullsized":"2024-02-23T00:31:54.007634+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:32.953143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:32.953143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993115+0000","last_change":"2024-02-23T00:30:48.713856+0000","last_active":"2024-02-23T00:31:53.993115+0000","last_peered":"2024-02-23T00:31:53.993115+0000","last_clean":"2024-02-23T00:31:53.993115+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:31:53.993115+0000","last_undegraded":"2024-02-23T00:31:53.993115+0000","last_fullsized":"2024-02-23T00:31:53.993115+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":177,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993873+0000","last_change":"2024-02-23T00:31:41.019548+0000","last_active":"2024-02-23T00:31:53.993873+0000","last_peered":"2024-02-23T00:31:53.993873+0000","last_clean":"2024-02-23T00:31:53.993873+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:31:53.993873+0000","last_undegraded":"2024-02-23T00:31:53.993873+0000","last_fullsized":"2024-02-23T00:31:53.993873+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.019441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.019441+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":133,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.000123+0000","last_change":"2024-02-23T00:31:20.632804+0000","last_active":"2024-02-23T00:31:54.000123+0000","last_peered":"2024-02-23T00:31:54.000123+0000","last_clean":"2024-02-23T00:31:54.000123+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:31:54.000123+0000","last_undegraded":"2024-02-23T00:31:54.000123+0000","last_fullsized":"2024-02-23T00:31:54.000123+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:20.632730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:20.632730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.992673+0000","last_change":"2024-02-23T00:30:40.787256+0000","last_active":"2024-02-23T00:31:53.992673+0000","last_peered":"2024-02-23T00:31:53.992673+0000","last_clean":"2024-02-23T00:31:53.992673+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:31:53.992673+0000","last_undegraded":"2024-02-23T00:31:53.992673+0000","last_fullsized":"2024-02-23T00:31:53.992673+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.787191+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":121,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006149+0000","last_change":"2024-02-23T00:30:30.499519+0000","last_active":"2024-02-23T00:31:54.006149+0000","last_peered":"2024-02-23T00:31:54.006149+0000","last_clean":"2024-02-23T00:31:54.006149+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:31:54.006149+0000","last_undegraded":"2024-02-23T00:31:54.006149+0000","last_fullsized":"2024-02-23T00:31:54.006149+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:30.499432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.499432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006058+0000","last_change":"2024-02-23T00:30:48.673703+0000","last_active":"2024-02-23T00:31:54.006058+0000","last_peered":"2024-02-23T00:31:54.006058+0000","last_clean":"2024-02-23T00:31:54.006058+0000","last_became_active":"2024-02-23T00:30:48.673452+0000","last_became_peered":"2024-02-23T00:30:48.673452+0000","last_unstale":"2024-02-23T00:31:54.006058+0000","last_undegraded":"2024-02-23T00:31:54.006058+0000","last_fullsized":"2024-02-23T00:31:54.006058+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":137,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.994285+0000","last_change":"2024-02-23T00:31:29.088731+0000","last_active":"2024-02-23T00:31:53.994285+0000","last_peered":"2024-02-23T00:31:53.994285+0000","last_clean":"2024-02-23T00:31:53.994285+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:31:53.994285+0000","last_undegraded":"2024-02-23T00:31:53.994285+0000","last_fullsized":"2024-02-23T00:31:53.994285+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:50.052469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:50.052469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":154,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.008111+0000","last_change":"2024-02-23T00:30:35.121011+0000","last_active":"2024-02-23T00:31:54.008111+0000","last_peered":"2024-02-23T00:31:54.008111+0000","last_clean":"2024-02-23T00:31:54.008111+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:31:54.008111+0000","last_undegraded":"2024-02-23T00:31:54.008111+0000","last_fullsized":"2024-02-23T00:31:54.008111+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:30:35.120909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.120909+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":127,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.000480+0000","last_change":"2024-02-23T00:30:43.620922+0000","last_active":"2024-02-23T00:31:54.000480+0000","last_peered":"2024-02-23T00:31:54.000480+0000","last_clean":"2024-02-23T00:31:54.000480+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:31:54.000480+0000","last_undegraded":"2024-02-23T00:31:54.000480+0000","last_fullsized":"2024-02-23T00:31:54.000480+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.620849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.620849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":119,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.020382+0000","last_change":"2024-02-23T00:29:08.580168+0000","last_active":"2024-02-23T00:31:54.020382+0000","last_peered":"2024-02-23T00:31:54.020382+0000","last_clean":"2024-02-23T00:31:54.020382+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:31:54.020382+0000","last_undegraded":"2024-02-23T00:31:54.020382+0000","last_fullsized":"2024-02-23T00:31:54.020382+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:08.580102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:08.580102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989033+0000","last_change":"2024-02-23T00:30:26.671989+0000","last_active":"2024-02-23T00:31:53.989033+0000","last_peered":"2024-02-23T00:31:53.989033+0000","last_clean":"2024-02-23T00:31:53.989033+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:31:53.989033+0000","last_undegraded":"2024-02-23T00:31:53.989033+0000","last_fullsized":"2024-02-23T00:31:53.989033+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:26.671888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:26.671888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1a","version":"148'7","reported_seq":32,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996012+0000","last_change":"2024-02-23T00:31:39.016947+0000","last_active":"2024-02-23T00:31:53.996012+0000","last_peered":"2024-02-23T00:31:53.996012+0000","last_clean":"2024-02-23T00:31:53.996012+0000","last_became_active":"2024-02-23T00:31:38.998465+0000","last_became_peered":"2024-02-23T00:31:38.998465+0000","last_unstale":"2024-02-23T00:31:53.996012+0000","last_undegraded":"2024-02-23T00:31:53.996012+0000","last_fullsized":"2024-02-23T00:31:53.996012+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.004707+0000","last_change":"2024-02-23T00:30:48.716832+0000","last_active":"2024-02-23T00:31:54.004707+0000","last_peered":"2024-02-23T00:31:54.004707+0000","last_clean":"2024-02-23T00:31:54.004707+0000","last_became_active":"2024-02-23T00:30:48.716552+0000","last_became_peered":"2024-02-23T00:30:48.716552+0000","last_unstale":"2024-02-23T00:31:54.004707+0000","last_undegraded":"2024-02-23T00:31:54.004707+0000","last_fullsized":"2024-02-23T00:31:54.004707+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":137,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993704+0000","last_change":"2024-02-23T00:31:29.064566+0000","last_active":"2024-02-23T00:31:53.993704+0000","last_peered":"2024-02-23T00:31:53.993704+0000","last_clean":"2024-02-23T00:31:53.993704+0000","last_became_active":"2024-02-23T00:31:29.064244+0000","last_became_peered":"2024-02-23T00:31:29.064244+0000","last_unstale":"2024-02-23T00:31:53.993704+0000","last_undegraded":"2024-02-23T00:31:53.993704+0000","last_fullsized":"2024-02-23T00:31:53.993704+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":141,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.002252+0000","last_change":"2024-02-23T00:31:19.982059+0000","last_active":"2024-02-23T00:31:54.002252+0000","last_peered":"2024-02-23T00:31:54.002252+0000","last_clean":"2024-02-23T00:31:54.002252+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:31:54.002252+0000","last_undegraded":"2024-02-23T00:31:54.002252+0000","last_fullsized":"2024-02-23T00:31:54.002252+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:31:19.981985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.981985+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":146,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.002876+0000","last_change":"2024-02-23T00:31:49.024710+0000","last_active":"2024-02-23T00:31:54.002876+0000","last_peered":"2024-02-23T00:31:54.002876+0000","last_clean":"2024-02-23T00:31:54.002876+0000","last_became_active":"2024-02-23T00:31:49.024302+0000","last_became_peered":"2024-02-23T00:31:49.024302+0000","last_unstale":"2024-02-23T00:31:54.002876+0000","last_undegraded":"2024-02-23T00:31:54.002876+0000","last_fullsized":"2024-02-23T00:31:54.002876+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.937969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":188,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.989440+0000","last_change":"2024-02-23T00:30:22.781434+0000","last_active":"2024-02-23T00:31:54.989440+0000","last_peered":"2024-02-23T00:31:54.989440+0000","last_clean":"2024-02-23T00:31:54.989440+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:31:54.989440+0000","last_undegraded":"2024-02-23T00:31:54.989440+0000","last_fullsized":"2024-02-23T00:31:54.989440+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:30:22.781318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.781318+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006723+0000","last_change":"2024-02-23T00:30:31.065478+0000","last_active":"2024-02-23T00:31:54.006723+0000","last_peered":"2024-02-23T00:31:54.006723+0000","last_clean":"2024-02-23T00:31:54.006723+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:31:54.006723+0000","last_undegraded":"2024-02-23T00:31:54.006723+0000","last_fullsized":"2024-02-23T00:31:54.006723+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.065384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.065384+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"148'14","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.983939+0000","last_change":"2024-02-23T00:31:39.029997+0000","last_active":"2024-02-23T00:31:53.983939+0000","last_peered":"2024-02-23T00:31:53.983939+0000","last_clean":"2024-02-23T00:31:53.983939+0000","last_became_active":"2024-02-23T00:31:39.000032+0000","last_became_peered":"2024-02-23T00:31:39.000032+0000","last_unstale":"2024-02-23T00:31:53.983939+0000","last_undegraded":"2024-02-23T00:31:53.983939+0000","last_fullsized":"2024-02-23T00:31:53.983939+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006814+0000","last_change":"2024-02-23T00:30:48.656349+0000","last_active":"2024-02-23T00:31:54.006814+0000","last_peered":"2024-02-23T00:31:54.006814+0000","last_clean":"2024-02-23T00:31:54.006814+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:31:54.006814+0000","last_undegraded":"2024-02-23T00:31:54.006814+0000","last_fullsized":"2024-02-23T00:31:54.006814+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":36,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986588+0000","last_change":"2024-02-23T00:31:29.089821+0000","last_active":"2024-02-23T00:31:53.986588+0000","last_peered":"2024-02-23T00:31:53.986588+0000","last_clean":"2024-02-23T00:31:53.986588+0000","last_became_active":"2024-02-23T00:31:29.089319+0000","last_became_peered":"2024-02-23T00:31:29.089319+0000","last_unstale":"2024-02-23T00:31:53.986588+0000","last_undegraded":"2024-02-23T00:31:53.986588+0000","last_fullsized":"2024-02-23T00:31:53.986588+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:30:38.432677+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.432677+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":59,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"87'22","reported_seq":158,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.011043+0000","last_change":"2024-02-23T00:29:57.567523+0000","last_active":"2024-02-23T00:31:54.011043+0000","last_peered":"2024-02-23T00:31:54.011043+0000","last_clean":"2024-02-23T00:31:54.011043+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:31:54.011043+0000","last_undegraded":"2024-02-23T00:31:54.011043+0000","last_fullsized":"2024-02-23T00:31:54.011043+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:29:57.567427+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.018903+0000","last_change":"2024-02-23T00:31:29.920657+0000","last_active":"2024-02-23T00:31:54.018903+0000","last_peered":"2024-02-23T00:31:54.018903+0000","last_clean":"2024-02-23T00:31:54.018903+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:31:54.018903+0000","last_undegraded":"2024-02-23T00:31:54.018903+0000","last_fullsized":"2024-02-23T00:31:54.018903+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:29.920546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:29.920546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":188,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.997993+0000","last_change":"2024-02-23T00:29:46.048380+0000","last_active":"2024-02-23T00:31:54.997993+0000","last_peered":"2024-02-23T00:31:54.997993+0000","last_clean":"2024-02-23T00:31:54.997993+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:31:54.997993+0000","last_undegraded":"2024-02-23T00:31:54.997993+0000","last_fullsized":"2024-02-23T00:31:54.997993+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:29:46.048267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:46.048267+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":121,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989031+0000","last_change":"2024-02-23T00:31:21.366604+0000","last_active":"2024-02-23T00:31:53.989031+0000","last_peered":"2024-02-23T00:31:53.989031+0000","last_clean":"2024-02-23T00:31:53.989031+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:31:53.989031+0000","last_undegraded":"2024-02-23T00:31:53.989031+0000","last_fullsized":"2024-02-23T00:31:53.989031+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:31:21.366535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.366535+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.18","version":"98'2","reported_seq":32,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.985736+0000","last_change":"2024-02-23T00:31:39.033460+0000","last_active":"2024-02-23T00:31:53.985736+0000","last_peered":"2024-02-23T00:31:53.985736+0000","last_clean":"2024-02-23T00:31:53.985736+0000","last_became_active":"2024-02-23T00:31:39.007405+0000","last_became_peered":"2024-02-23T00:31:39.007405+0000","last_unstale":"2024-02-23T00:31:53.985736+0000","last_undegraded":"2024-02-23T00:31:53.985736+0000","last_fullsized":"2024-02-23T00:31:53.985736+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.011416+0000","last_change":"2024-02-23T00:30:48.713004+0000","last_active":"2024-02-23T00:31:54.011416+0000","last_peered":"2024-02-23T00:31:54.011416+0000","last_clean":"2024-02-23T00:31:54.011416+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:31:54.011416+0000","last_undegraded":"2024-02-23T00:31:54.011416+0000","last_fullsized":"2024-02-23T00:31:54.011416+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":137,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996613+0000","last_change":"2024-02-23T00:31:29.074410+0000","last_active":"2024-02-23T00:31:53.996613+0000","last_peered":"2024-02-23T00:31:53.996613+0000","last_clean":"2024-02-23T00:31:53.996613+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:31:53.996613+0000","last_undegraded":"2024-02-23T00:31:53.996613+0000","last_fullsized":"2024-02-23T00:31:53.996613+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"86'23","reported_seq":166,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006501+0000","last_change":"2024-02-23T00:29:56.044401+0000","last_active":"2024-02-23T00:31:54.006501+0000","last_peered":"2024-02-23T00:31:54.006501+0000","last_clean":"2024-02-23T00:31:54.006501+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:31:54.006501+0000","last_undegraded":"2024-02-23T00:31:54.006501+0000","last_fullsized":"2024-02-23T00:31:54.006501+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'23","last_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:29:56.044331+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":139,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010231+0000","last_change":"2024-02-23T00:31:38.838289+0000","last_active":"2024-02-23T00:31:54.010231+0000","last_peered":"2024-02-23T00:31:54.010231+0000","last_clean":"2024-02-23T00:31:54.010231+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:31:54.010231+0000","last_undegraded":"2024-02-23T00:31:54.010231+0000","last_fullsized":"2024-02-23T00:31:54.010231+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:31:38.838202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.838202+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.991695+0000","last_change":"2024-02-23T00:31:33.381301+0000","last_active":"2024-02-23T00:31:53.991695+0000","last_peered":"2024-02-23T00:31:53.991695+0000","last_clean":"2024-02-23T00:31:53.991695+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:31:53.991695+0000","last_undegraded":"2024-02-23T00:31:53.991695+0000","last_fullsized":"2024-02-23T00:31:53.991695+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.381229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.381229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":117,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988585+0000","last_change":"2024-02-23T00:29:07.960232+0000","last_active":"2024-02-23T00:31:53.988585+0000","last_peered":"2024-02-23T00:31:53.988585+0000","last_clean":"2024-02-23T00:31:53.988585+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:31:53.988585+0000","last_undegraded":"2024-02-23T00:31:53.988585+0000","last_fullsized":"2024-02-23T00:31:53.988585+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:07.960141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:07.960141+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":26,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003641+0000","last_change":"2024-02-23T00:31:39.013965+0000","last_active":"2024-02-23T00:31:54.003641+0000","last_peered":"2024-02-23T00:31:54.003641+0000","last_clean":"2024-02-23T00:31:54.003641+0000","last_became_active":"2024-02-23T00:31:39.013662+0000","last_became_peered":"2024-02-23T00:31:39.013662+0000","last_unstale":"2024-02-23T00:31:54.003641+0000","last_undegraded":"2024-02-23T00:31:54.003641+0000","last_fullsized":"2024-02-23T00:31:54.003641+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.020705+0000","last_change":"2024-02-23T00:30:48.742570+0000","last_active":"2024-02-23T00:31:54.020705+0000","last_peered":"2024-02-23T00:31:54.020705+0000","last_clean":"2024-02-23T00:31:54.020705+0000","last_became_active":"2024-02-23T00:30:48.740636+0000","last_became_peered":"2024-02-23T00:30:48.740636+0000","last_unstale":"2024-02-23T00:31:54.020705+0000","last_undegraded":"2024-02-23T00:31:54.020705+0000","last_fullsized":"2024-02-23T00:31:54.020705+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":41,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.009619+0000","last_change":"2024-02-23T00:31:39.641680+0000","last_active":"2024-02-23T00:31:54.009619+0000","last_peered":"2024-02-23T00:31:54.009619+0000","last_clean":"2024-02-23T00:31:54.009619+0000","last_became_active":"2024-02-23T00:31:29.108747+0000","last_became_peered":"2024-02-23T00:31:29.108747+0000","last_unstale":"2024-02-23T00:31:54.009619+0000","last_undegraded":"2024-02-23T00:31:54.009619+0000","last_fullsized":"2024-02-23T00:31:54.009619+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:31:39.641617+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":192,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.996006+0000","last_change":"2024-02-23T00:31:41.945984+0000","last_active":"2024-02-23T00:31:54.996006+0000","last_peered":"2024-02-23T00:31:54.996006+0000","last_clean":"2024-02-23T00:31:54.996006+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:31:54.996006+0000","last_undegraded":"2024-02-23T00:31:54.996006+0000","last_fullsized":"2024-02-23T00:31:54.996006+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":162,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.998119+0000","last_change":"2024-02-23T00:31:38.038523+0000","last_active":"2024-02-23T00:31:53.998119+0000","last_peered":"2024-02-23T00:31:53.998119+0000","last_clean":"2024-02-23T00:31:53.998119+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:31:53.998119+0000","last_undegraded":"2024-02-23T00:31:53.998119+0000","last_fullsized":"2024-02-23T00:31:53.998119+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:31:38.038413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.038413+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":127,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996049+0000","last_change":"2024-02-23T00:30:26.170570+0000","last_active":"2024-02-23T00:31:53.996049+0000","last_peered":"2024-02-23T00:31:53.996049+0000","last_clean":"2024-02-23T00:31:53.996049+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:31:53.996049+0000","last_undegraded":"2024-02-23T00:31:53.996049+0000","last_fullsized":"2024-02-23T00:31:53.996049+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:26.170482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:26.170482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":126,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.000978+0000","last_change":"2024-02-23T00:31:31.946435+0000","last_active":"2024-02-23T00:31:54.000978+0000","last_peered":"2024-02-23T00:31:54.000978+0000","last_clean":"2024-02-23T00:31:54.000978+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:31:54.000978+0000","last_undegraded":"2024-02-23T00:31:54.000978+0000","last_fullsized":"2024-02-23T00:31:54.000978+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.946366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.946366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.16","version":"148'7","reported_seq":35,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.004198+0000","last_change":"2024-02-23T00:31:39.181782+0000","last_active":"2024-02-23T00:31:54.004198+0000","last_peered":"2024-02-23T00:31:54.004198+0000","last_clean":"2024-02-23T00:31:54.004198+0000","last_became_active":"2024-02-23T00:31:38.996361+0000","last_became_peered":"2024-02-23T00:31:38.996361+0000","last_unstale":"2024-02-23T00:31:54.004198+0000","last_undegraded":"2024-02-23T00:31:54.004198+0000","last_fullsized":"2024-02-23T00:31:54.004198+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.004799+0000","last_change":"2024-02-23T00:30:48.782732+0000","last_active":"2024-02-23T00:31:54.004799+0000","last_peered":"2024-02-23T00:31:54.004799+0000","last_clean":"2024-02-23T00:31:54.004799+0000","last_became_active":"2024-02-23T00:30:48.778293+0000","last_became_peered":"2024-02-23T00:30:48.778293+0000","last_unstale":"2024-02-23T00:31:54.004799+0000","last_undegraded":"2024-02-23T00:31:54.004799+0000","last_fullsized":"2024-02-23T00:31:54.004799+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":137,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993304+0000","last_change":"2024-02-23T00:31:29.070859+0000","last_active":"2024-02-23T00:31:53.993304+0000","last_peered":"2024-02-23T00:31:53.993304+0000","last_clean":"2024-02-23T00:31:53.993304+0000","last_became_active":"2024-02-23T00:31:29.070521+0000","last_became_peered":"2024-02-23T00:31:29.070521+0000","last_unstale":"2024-02-23T00:31:53.993304+0000","last_undegraded":"2024-02-23T00:31:53.993304+0000","last_fullsized":"2024-02-23T00:31:53.993304+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.998488+0000","last_change":"2024-02-23T00:29:47.033380+0000","last_active":"2024-02-23T00:31:53.998488+0000","last_peered":"2024-02-23T00:31:53.998488+0000","last_clean":"2024-02-23T00:31:53.998488+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:31:53.998488+0000","last_undegraded":"2024-02-23T00:31:53.998488+0000","last_fullsized":"2024-02-23T00:31:53.998488+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:47.033301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:47.033301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":188,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986233+0000","last_change":"2024-02-23T00:31:21.086750+0000","last_active":"2024-02-23T00:31:53.986233+0000","last_peered":"2024-02-23T00:31:53.986233+0000","last_clean":"2024-02-23T00:31:53.986233+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:31:53.986233+0000","last_undegraded":"2024-02-23T00:31:53.986233+0000","last_fullsized":"2024-02-23T00:31:53.986233+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:31:21.086610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.086610+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.004966+0000","last_change":"2024-02-23T00:30:25.593913+0000","last_active":"2024-02-23T00:31:54.004966+0000","last_peered":"2024-02-23T00:31:54.004966+0000","last_clean":"2024-02-23T00:31:54.004966+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:31:54.004966+0000","last_undegraded":"2024-02-23T00:31:54.004966+0000","last_fullsized":"2024-02-23T00:31:54.004966+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.593833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.593833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.011677+0000","last_change":"2024-02-23T00:29:40.940545+0000","last_active":"2024-02-23T00:31:54.011677+0000","last_peered":"2024-02-23T00:31:54.011677+0000","last_clean":"2024-02-23T00:31:54.011677+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:31:54.011677+0000","last_undegraded":"2024-02-23T00:31:54.011677+0000","last_fullsized":"2024-02-23T00:31:54.011677+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:40.940460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:40.940460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.17","version":"148'7","reported_seq":48,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989185+0000","last_change":"2024-02-23T00:31:39.017837+0000","last_active":"2024-02-23T00:31:53.989185+0000","last_peered":"2024-02-23T00:31:53.989185+0000","last_clean":"2024-02-23T00:31:53.989185+0000","last_became_active":"2024-02-23T00:31:39.012456+0000","last_became_peered":"2024-02-23T00:31:39.012456+0000","last_unstale":"2024-02-23T00:31:53.989185+0000","last_undegraded":"2024-02-23T00:31:53.989185+0000","last_fullsized":"2024-02-23T00:31:53.989185+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988791+0000","last_change":"2024-02-23T00:30:48.776316+0000","last_active":"2024-02-23T00:31:53.988791+0000","last_peered":"2024-02-23T00:31:53.988791+0000","last_clean":"2024-02-23T00:31:53.988791+0000","last_became_active":"2024-02-23T00:30:48.769602+0000","last_became_peered":"2024-02-23T00:30:48.769602+0000","last_unstale":"2024-02-23T00:31:53.988791+0000","last_undegraded":"2024-02-23T00:31:53.988791+0000","last_fullsized":"2024-02-23T00:31:53.988791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":391,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.015805+0000","last_change":"2024-02-23T00:31:29.095695+0000","last_active":"2024-02-23T00:31:54.015805+0000","last_peered":"2024-02-23T00:31:54.015805+0000","last_clean":"2024-02-23T00:31:54.015805+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:31:54.015805+0000","last_undegraded":"2024-02-23T00:31:54.015805+0000","last_fullsized":"2024-02-23T00:31:54.015805+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:30:38.432677+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.432677+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":61,"num_read_kb":59,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993959+0000","last_change":"2024-02-23T00:30:27.151423+0000","last_active":"2024-02-23T00:31:53.993959+0000","last_peered":"2024-02-23T00:31:53.993959+0000","last_clean":"2024-02-23T00:31:53.993959+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:31:53.993959+0000","last_undegraded":"2024-02-23T00:31:53.993959+0000","last_fullsized":"2024-02-23T00:31:53.993959+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:27.151342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:27.151342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":182,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003039+0000","last_change":"2024-02-23T00:31:29.650718+0000","last_active":"2024-02-23T00:31:54.003039+0000","last_peered":"2024-02-23T00:31:54.003039+0000","last_clean":"2024-02-23T00:31:54.003039+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:31:54.003039+0000","last_undegraded":"2024-02-23T00:31:54.003039+0000","last_fullsized":"2024-02-23T00:31:54.003039+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:29.650602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:29.650602+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":132,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010380+0000","last_change":"2024-02-23T00:31:40.847863+0000","last_active":"2024-02-23T00:31:54.010380+0000","last_peered":"2024-02-23T00:31:54.010380+0000","last_clean":"2024-02-23T00:31:54.010380+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:31:54.010380+0000","last_undegraded":"2024-02-23T00:31:54.010380+0000","last_fullsized":"2024-02-23T00:31:54.010380+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.847784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.847784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.019048+0000","last_change":"2024-02-23T00:30:46.579484+0000","last_active":"2024-02-23T00:31:54.019048+0000","last_peered":"2024-02-23T00:31:54.019048+0000","last_clean":"2024-02-23T00:31:54.019048+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:31:54.019048+0000","last_undegraded":"2024-02-23T00:31:54.019048+0000","last_fullsized":"2024-02-23T00:31:54.019048+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.579417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.579417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.14","version":"98'2","reported_seq":30,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.015792+0000","last_change":"2024-02-23T00:31:39.105675+0000","last_active":"2024-02-23T00:31:54.015792+0000","last_peered":"2024-02-23T00:31:54.015792+0000","last_clean":"2024-02-23T00:31:54.015792+0000","last_became_active":"2024-02-23T00:31:39.009555+0000","last_became_peered":"2024-02-23T00:31:39.009555+0000","last_unstale":"2024-02-23T00:31:54.015792+0000","last_undegraded":"2024-02-23T00:31:54.015792+0000","last_fullsized":"2024-02-23T00:31:54.015792+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.019121+0000","last_change":"2024-02-23T00:30:48.707450+0000","last_active":"2024-02-23T00:31:54.019121+0000","last_peered":"2024-02-23T00:31:54.019121+0000","last_clean":"2024-02-23T00:31:54.019121+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:31:54.019121+0000","last_undegraded":"2024-02-23T00:31:54.019121+0000","last_fullsized":"2024-02-23T00:31:54.019121+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.994172+0000","last_change":"2024-02-23T00:29:39.108458+0000","last_active":"2024-02-23T00:31:53.994172+0000","last_peered":"2024-02-23T00:31:53.994172+0000","last_clean":"2024-02-23T00:31:53.994172+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:31:53.994172+0000","last_undegraded":"2024-02-23T00:31:53.994172+0000","last_fullsized":"2024-02-23T00:31:53.994172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:39.108381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:29:39.108381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":149,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.007296+0000","last_change":"2024-02-23T00:31:19.393840+0000","last_active":"2024-02-23T00:31:54.007296+0000","last_peered":"2024-02-23T00:31:54.007296+0000","last_clean":"2024-02-23T00:31:54.007296+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:31:54.007296+0000","last_undegraded":"2024-02-23T00:31:54.007296+0000","last_fullsized":"2024-02-23T00:31:54.007296+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:31:19.393720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.393720+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":127,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006224+0000","last_change":"2024-02-23T00:31:19.760264+0000","last_active":"2024-02-23T00:31:54.006224+0000","last_peered":"2024-02-23T00:31:54.006224+0000","last_clean":"2024-02-23T00:31:54.006224+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:31:54.006224+0000","last_undegraded":"2024-02-23T00:31:54.006224+0000","last_fullsized":"2024-02-23T00:31:54.006224+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:19.760201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.760201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":142,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006800+0000","last_change":"2024-02-23T00:31:38.653143+0000","last_active":"2024-02-23T00:31:54.006800+0000","last_peered":"2024-02-23T00:31:54.006800+0000","last_clean":"2024-02-23T00:31:54.006800+0000","last_became_active":"2024-02-23T00:31:27.053776+0000","last_became_peered":"2024-02-23T00:31:27.053776+0000","last_unstale":"2024-02-23T00:31:54.006800+0000","last_undegraded":"2024-02-23T00:31:54.006800+0000","last_fullsized":"2024-02-23T00:31:54.006800+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.653078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.653078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.020480+0000","last_change":"2024-02-23T00:31:21.959719+0000","last_active":"2024-02-23T00:31:54.020480+0000","last_peered":"2024-02-23T00:31:54.020480+0000","last_clean":"2024-02-23T00:31:54.020480+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:31:54.020480+0000","last_undegraded":"2024-02-23T00:31:54.020480+0000","last_fullsized":"2024-02-23T00:31:54.020480+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.959597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.959597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"148'9","reported_seq":53,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.019606+0000","last_change":"2024-02-23T00:31:39.037939+0000","last_active":"2024-02-23T00:31:54.019606+0000","last_peered":"2024-02-23T00:31:54.019606+0000","last_clean":"2024-02-23T00:31:54.019606+0000","last_became_active":"2024-02-23T00:31:39.012770+0000","last_became_peered":"2024-02-23T00:31:39.012770+0000","last_unstale":"2024-02-23T00:31:54.019606+0000","last_undegraded":"2024-02-23T00:31:54.019606+0000","last_fullsized":"2024-02-23T00:31:54.019606+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.002376+0000","last_change":"2024-02-23T00:30:48.711294+0000","last_active":"2024-02-23T00:31:54.002376+0000","last_peered":"2024-02-23T00:31:54.002376+0000","last_clean":"2024-02-23T00:31:54.002376+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:31:54.002376+0000","last_undegraded":"2024-02-23T00:31:54.002376+0000","last_fullsized":"2024-02-23T00:31:54.002376+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":123,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006067+0000","last_change":"2024-02-23T00:30:42.634821+0000","last_active":"2024-02-23T00:31:54.006067+0000","last_peered":"2024-02-23T00:31:54.006067+0000","last_clean":"2024-02-23T00:31:54.006067+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:31:54.006067+0000","last_undegraded":"2024-02-23T00:31:54.006067+0000","last_fullsized":"2024-02-23T00:31:54.006067+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:42.634753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:42.634753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":153,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.987610+0000","last_change":"2024-02-23T00:31:40.030848+0000","last_active":"2024-02-23T00:31:53.987610+0000","last_peered":"2024-02-23T00:31:53.987610+0000","last_clean":"2024-02-23T00:31:53.987610+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:31:53.987610+0000","last_undegraded":"2024-02-23T00:31:53.987610+0000","last_fullsized":"2024-02-23T00:31:53.987610+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:31:40.030776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.030776+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":135,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.009891+0000","last_change":"2024-02-23T00:31:21.828852+0000","last_active":"2024-02-23T00:31:54.009891+0000","last_peered":"2024-02-23T00:31:54.009891+0000","last_clean":"2024-02-23T00:31:54.009891+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:31:54.009891+0000","last_undegraded":"2024-02-23T00:31:54.009891+0000","last_fullsized":"2024-02-23T00:31:54.009891+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.828719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.828719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":126,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.985688+0000","last_change":"2024-02-23T00:30:40.144284+0000","last_active":"2024-02-23T00:31:53.985688+0000","last_peered":"2024-02-23T00:31:53.985688+0000","last_clean":"2024-02-23T00:31:53.985688+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:31:53.985688+0000","last_undegraded":"2024-02-23T00:31:53.985688+0000","last_fullsized":"2024-02-23T00:31:53.985688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.144220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.144220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":137,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.987234+0000","last_change":"2024-02-23T00:31:27.145186+0000","last_active":"2024-02-23T00:31:53.987234+0000","last_peered":"2024-02-23T00:31:53.987234+0000","last_clean":"2024-02-23T00:31:53.987234+0000","last_became_active":"2024-02-23T00:31:27.144976+0000","last_became_peered":"2024-02-23T00:31:27.144976+0000","last_unstale":"2024-02-23T00:31:53.987234+0000","last_undegraded":"2024-02-23T00:31:53.987234+0000","last_fullsized":"2024-02-23T00:31:53.987234+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:50.052469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:50.052469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":120,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.994267+0000","last_change":"2024-02-23T00:30:45.024137+0000","last_active":"2024-02-23T00:31:53.994267+0000","last_peered":"2024-02-23T00:31:53.994267+0000","last_clean":"2024-02-23T00:31:53.994267+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:31:53.994267+0000","last_undegraded":"2024-02-23T00:31:53.994267+0000","last_fullsized":"2024-02-23T00:31:53.994267+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.024058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.024058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":85,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.004798+0000","last_change":"2024-02-23T00:31:39.006330+0000","last_active":"2024-02-23T00:31:54.004798+0000","last_peered":"2024-02-23T00:31:54.004798+0000","last_clean":"2024-02-23T00:31:54.004798+0000","last_became_active":"2024-02-23T00:31:39.005510+0000","last_became_peered":"2024-02-23T00:31:39.005510+0000","last_unstale":"2024-02-23T00:31:54.004798+0000","last_undegraded":"2024-02-23T00:31:54.004798+0000","last_fullsized":"2024-02-23T00:31:54.004798+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006744+0000","last_change":"2024-02-23T00:30:48.664253+0000","last_active":"2024-02-23T00:31:54.006744+0000","last_peered":"2024-02-23T00:31:54.006744+0000","last_clean":"2024-02-23T00:31:54.006744+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:31:54.006744+0000","last_undegraded":"2024-02-23T00:31:54.006744+0000","last_fullsized":"2024-02-23T00:31:54.006744+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990517+0000","last_change":"2024-02-23T00:30:23.737978+0000","last_active":"2024-02-23T00:31:53.990517+0000","last_peered":"2024-02-23T00:31:53.990517+0000","last_clean":"2024-02-23T00:31:53.990517+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:31:53.990517+0000","last_undegraded":"2024-02-23T00:31:53.990517+0000","last_fullsized":"2024-02-23T00:31:53.990517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:23.737872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.737872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":168,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010836+0000","last_change":"2024-02-23T00:29:44.530235+0000","last_active":"2024-02-23T00:31:54.010836+0000","last_peered":"2024-02-23T00:31:54.010836+0000","last_clean":"2024-02-23T00:31:54.010836+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:31:54.010836+0000","last_undegraded":"2024-02-23T00:31:54.010836+0000","last_fullsized":"2024-02-23T00:31:54.010836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:29:44.530164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:44.530164+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":133,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993478+0000","last_change":"2024-02-23T00:31:41.651826+0000","last_active":"2024-02-23T00:31:53.993478+0000","last_peered":"2024-02-23T00:31:53.993478+0000","last_clean":"2024-02-23T00:31:53.993478+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:31:53.993478+0000","last_undegraded":"2024-02-23T00:31:53.993478+0000","last_fullsized":"2024-02-23T00:31:53.993478+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:41.651760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.651760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":384,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.987274+0000","last_change":"2024-02-23T00:31:27.040910+0000","last_active":"2024-02-23T00:31:53.987274+0000","last_peered":"2024-02-23T00:31:53.987274+0000","last_clean":"2024-02-23T00:31:53.987274+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:31:53.987274+0000","last_undegraded":"2024-02-23T00:31:53.987274+0000","last_fullsized":"2024-02-23T00:31:53.987274+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:30:38.432677+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.432677+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":59,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":121,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010296+0000","last_change":"2024-02-23T00:30:32.479956+0000","last_active":"2024-02-23T00:31:54.010296+0000","last_peered":"2024-02-23T00:31:54.010296+0000","last_clean":"2024-02-23T00:31:54.010296+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:31:54.010296+0000","last_undegraded":"2024-02-23T00:31:54.010296+0000","last_fullsized":"2024-02-23T00:31:54.010296+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:32.479814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:32.479814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":26,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.985779+0000","last_change":"2024-02-23T00:31:39.006489+0000","last_active":"2024-02-23T00:31:53.985779+0000","last_peered":"2024-02-23T00:31:53.985779+0000","last_clean":"2024-02-23T00:31:53.985779+0000","last_became_active":"2024-02-23T00:31:39.005730+0000","last_became_peered":"2024-02-23T00:31:39.005730+0000","last_unstale":"2024-02-23T00:31:53.985779+0000","last_undegraded":"2024-02-23T00:31:53.985779+0000","last_fullsized":"2024-02-23T00:31:53.985779+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988468+0000","last_change":"2024-02-23T00:30:48.692888+0000","last_active":"2024-02-23T00:31:53.988468+0000","last_peered":"2024-02-23T00:31:53.988468+0000","last_clean":"2024-02-23T00:31:53.988468+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:31:53.988468+0000","last_undegraded":"2024-02-23T00:31:53.988468+0000","last_fullsized":"2024-02-23T00:31:53.988468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":137,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.997028+0000","last_change":"2024-02-23T00:31:29.065838+0000","last_active":"2024-02-23T00:31:53.997028+0000","last_peered":"2024-02-23T00:31:53.997028+0000","last_clean":"2024-02-23T00:31:53.997028+0000","last_became_active":"2024-02-23T00:31:29.065548+0000","last_became_peered":"2024-02-23T00:31:29.065548+0000","last_unstale":"2024-02-23T00:31:53.997028+0000","last_undegraded":"2024-02-23T00:31:53.997028+0000","last_fullsized":"2024-02-23T00:31:53.997028+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":131,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.008447+0000","last_change":"2024-02-23T00:31:26.705106+0000","last_active":"2024-02-23T00:31:54.008447+0000","last_peered":"2024-02-23T00:31:54.008447+0000","last_clean":"2024-02-23T00:31:54.008447+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:31:54.008447+0000","last_undegraded":"2024-02-23T00:31:54.008447+0000","last_fullsized":"2024-02-23T00:31:54.008447+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:26.704988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:26.704988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":245,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:55.006220+0000","last_change":"2024-02-23T00:30:29.527577+0000","last_active":"2024-02-23T00:31:55.006220+0000","last_peered":"2024-02-23T00:31:55.006220+0000","last_clean":"2024-02-23T00:31:55.006220+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:31:55.006220+0000","last_undegraded":"2024-02-23T00:31:55.006220+0000","last_fullsized":"2024-02-23T00:31:55.006220+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:30:29.527466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.527466+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":132,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.984393+0000","last_change":"2024-02-23T00:30:47.090981+0000","last_active":"2024-02-23T00:31:53.984393+0000","last_peered":"2024-02-23T00:31:53.984393+0000","last_clean":"2024-02-23T00:31:53.984393+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:31:53.984393+0000","last_undegraded":"2024-02-23T00:31:53.984393+0000","last_fullsized":"2024-02-23T00:31:53.984393+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:30:47.090896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.090896+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988981+0000","last_change":"2024-02-23T00:30:22.860665+0000","last_active":"2024-02-23T00:31:53.988981+0000","last_peered":"2024-02-23T00:31:53.988981+0000","last_clean":"2024-02-23T00:31:53.988981+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:31:53.988981+0000","last_undegraded":"2024-02-23T00:31:53.988981+0000","last_fullsized":"2024-02-23T00:31:53.988981+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:22.860565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.860565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.13","version":"148'11","reported_seq":39,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.983640+0000","last_change":"2024-02-23T00:31:39.019330+0000","last_active":"2024-02-23T00:31:53.983640+0000","last_peered":"2024-02-23T00:31:53.983640+0000","last_clean":"2024-02-23T00:31:53.983640+0000","last_became_active":"2024-02-23T00:31:39.000272+0000","last_became_peered":"2024-02-23T00:31:39.000272+0000","last_unstale":"2024-02-23T00:31:53.983640+0000","last_undegraded":"2024-02-23T00:31:53.983640+0000","last_fullsized":"2024-02-23T00:31:53.983640+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990830+0000","last_change":"2024-02-23T00:30:48.701388+0000","last_active":"2024-02-23T00:31:53.990830+0000","last_peered":"2024-02-23T00:31:53.990830+0000","last_clean":"2024-02-23T00:31:53.990830+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:31:53.990830+0000","last_undegraded":"2024-02-23T00:31:53.990830+0000","last_fullsized":"2024-02-23T00:31:53.990830+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.984644+0000","last_change":"2024-02-23T00:31:29.059862+0000","last_active":"2024-02-23T00:31:53.984644+0000","last_peered":"2024-02-23T00:31:53.984644+0000","last_clean":"2024-02-23T00:31:53.984644+0000","last_became_active":"2024-02-23T00:31:29.059516+0000","last_became_peered":"2024-02-23T00:31:29.059516+0000","last_unstale":"2024-02-23T00:31:53.984644+0000","last_undegraded":"2024-02-23T00:31:53.984644+0000","last_fullsized":"2024-02-23T00:31:53.984644+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":191,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.988845+0000","last_change":"2024-02-23T00:30:24.050993+0000","last_active":"2024-02-23T00:31:54.988845+0000","last_peered":"2024-02-23T00:31:54.988845+0000","last_clean":"2024-02-23T00:31:54.988845+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:31:54.988845+0000","last_undegraded":"2024-02-23T00:31:54.988845+0000","last_fullsized":"2024-02-23T00:31:54.988845+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:30:24.050887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:24.050887+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":156,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003144+0000","last_change":"2024-02-23T00:30:31.441530+0000","last_active":"2024-02-23T00:31:54.003144+0000","last_peered":"2024-02-23T00:31:54.003144+0000","last_clean":"2024-02-23T00:31:54.003144+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:31:54.003144+0000","last_undegraded":"2024-02-23T00:31:54.003144+0000","last_fullsized":"2024-02-23T00:31:54.003144+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:30:31.441418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.441418+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.020198+0000","last_change":"2024-02-23T00:30:44.543241+0000","last_active":"2024-02-23T00:31:54.020198+0000","last_peered":"2024-02-23T00:31:54.020198+0000","last_clean":"2024-02-23T00:31:54.020198+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:31:54.020198+0000","last_undegraded":"2024-02-23T00:31:54.020198+0000","last_fullsized":"2024-02-23T00:31:54.020198+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:44.543161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:30:44.543161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.004967+0000","last_change":"2024-02-23T00:30:25.064207+0000","last_active":"2024-02-23T00:31:54.004967+0000","last_peered":"2024-02-23T00:31:54.004967+0000","last_clean":"2024-02-23T00:31:54.004967+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:31:54.004967+0000","last_undegraded":"2024-02-23T00:31:54.004967+0000","last_fullsized":"2024-02-23T00:31:54.004967+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.064103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.064103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":26,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006348+0000","last_change":"2024-02-23T00:31:39.004264+0000","last_active":"2024-02-23T00:31:54.006348+0000","last_peered":"2024-02-23T00:31:54.006348+0000","last_clean":"2024-02-23T00:31:54.006348+0000","last_became_active":"2024-02-23T00:31:39.003710+0000","last_became_peered":"2024-02-23T00:31:39.003710+0000","last_unstale":"2024-02-23T00:31:54.006348+0000","last_undegraded":"2024-02-23T00:31:54.006348+0000","last_fullsized":"2024-02-23T00:31:54.006348+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996108+0000","last_change":"2024-02-23T00:30:48.616685+0000","last_active":"2024-02-23T00:31:53.996108+0000","last_peered":"2024-02-23T00:31:53.996108+0000","last_clean":"2024-02-23T00:31:53.996108+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:31:53.996108+0000","last_undegraded":"2024-02-23T00:31:53.996108+0000","last_fullsized":"2024-02-23T00:31:53.996108+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.000206+0000","last_change":"2024-02-23T00:30:28.676431+0000","last_active":"2024-02-23T00:31:54.000206+0000","last_peered":"2024-02-23T00:31:54.000206+0000","last_clean":"2024-02-23T00:31:54.000206+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:31:54.000206+0000","last_undegraded":"2024-02-23T00:31:54.000206+0000","last_fullsized":"2024-02-23T00:31:54.000206+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:28.676326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:28.676326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":205,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.998401+0000","last_change":"2024-02-23T00:31:41.036909+0000","last_active":"2024-02-23T00:31:53.998401+0000","last_peered":"2024-02-23T00:31:53.998401+0000","last_clean":"2024-02-23T00:31:53.998401+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:31:53.998401+0000","last_undegraded":"2024-02-23T00:31:53.998401+0000","last_fullsized":"2024-02-23T00:31:53.998401+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.036837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.036837+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":141,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.009008+0000","last_change":"2024-02-23T00:30:47.142583+0000","last_active":"2024-02-23T00:31:54.009008+0000","last_peered":"2024-02-23T00:31:54.009008+0000","last_clean":"2024-02-23T00:31:54.009008+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:31:54.009008+0000","last_undegraded":"2024-02-23T00:31:54.009008+0000","last_fullsized":"2024-02-23T00:31:54.009008+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:30:47.142508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.142508+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.994472+0000","last_change":"2024-02-23T00:30:34.861845+0000","last_active":"2024-02-23T00:31:53.994472+0000","last_peered":"2024-02-23T00:31:53.994472+0000","last_clean":"2024-02-23T00:31:53.994472+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:31:53.994472+0000","last_undegraded":"2024-02-23T00:31:53.994472+0000","last_fullsized":"2024-02-23T00:31:53.994472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:34.861741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:34.861741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.984415+0000","last_change":"2024-02-23T00:30:48.674743+0000","last_active":"2024-02-23T00:31:53.984415+0000","last_peered":"2024-02-23T00:31:53.984415+0000","last_clean":"2024-02-23T00:31:53.984415+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:31:53.984415+0000","last_undegraded":"2024-02-23T00:31:53.984415+0000","last_fullsized":"2024-02-23T00:31:53.984415+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":172,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.007344+0000","last_change":"2024-02-23T00:30:44.885410+0000","last_active":"2024-02-23T00:31:54.007344+0000","last_peered":"2024-02-23T00:31:54.007344+0000","last_clean":"2024-02-23T00:31:54.007344+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:31:54.007344+0000","last_undegraded":"2024-02-23T00:31:54.007344+0000","last_fullsized":"2024-02-23T00:31:54.007344+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:30:44.885334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:44.885334+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.000113+0000","last_change":"2024-02-23T00:29:47.780564+0000","last_active":"2024-02-23T00:31:54.000113+0000","last_peered":"2024-02-23T00:31:54.000113+0000","last_clean":"2024-02-23T00:31:54.000113+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:31:54.000113+0000","last_undegraded":"2024-02-23T00:31:54.000113+0000","last_fullsized":"2024-02-23T00:31:54.000113+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:47.780463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:47.780463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.992570+0000","last_change":"2024-02-23T00:30:36.894317+0000","last_active":"2024-02-23T00:31:53.992570+0000","last_peered":"2024-02-23T00:31:53.992570+0000","last_clean":"2024-02-23T00:31:53.992570+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:31:53.992570+0000","last_undegraded":"2024-02-23T00:31:53.992570+0000","last_fullsized":"2024-02-23T00:31:53.992570+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:36.894220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:36.894220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.999842+0000","last_change":"2024-02-23T00:30:24.708147+0000","last_active":"2024-02-23T00:31:53.999842+0000","last_peered":"2024-02-23T00:31:53.999842+0000","last_clean":"2024-02-23T00:31:53.999842+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:31:53.999842+0000","last_undegraded":"2024-02-23T00:31:53.999842+0000","last_fullsized":"2024-02-23T00:31:53.999842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:24.708045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:24.708045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"148'7","reported_seq":83,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.019323+0000","last_change":"2024-02-23T00:30:48.725952+0000","last_active":"2024-02-23T00:31:54.019323+0000","last_peered":"2024-02-23T00:31:54.019323+0000","last_clean":"2024-02-23T00:31:54.019323+0000","last_became_active":"2024-02-23T00:30:48.725566+0000","last_became_peered":"2024-02-23T00:30:48.725566+0000","last_unstale":"2024-02-23T00:31:54.019323+0000","last_undegraded":"2024-02-23T00:31:54.019323+0000","last_fullsized":"2024-02-23T00:31:54.019323+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.991769+0000","last_change":"2024-02-23T00:30:48.691960+0000","last_active":"2024-02-23T00:31:53.991769+0000","last_peered":"2024-02-23T00:31:53.991769+0000","last_clean":"2024-02-23T00:31:53.991769+0000","last_became_active":"2024-02-23T00:30:48.691713+0000","last_became_peered":"2024-02-23T00:30:48.691713+0000","last_unstale":"2024-02-23T00:31:53.991769+0000","last_undegraded":"2024-02-23T00:31:53.991769+0000","last_fullsized":"2024-02-23T00:31:53.991769+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":169,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993217+0000","last_change":"2024-02-23T00:30:24.802266+0000","last_active":"2024-02-23T00:31:53.993217+0000","last_peered":"2024-02-23T00:31:53.993217+0000","last_clean":"2024-02-23T00:31:53.993217+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:31:53.993217+0000","last_undegraded":"2024-02-23T00:31:53.993217+0000","last_fullsized":"2024-02-23T00:31:53.993217+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:30:24.802134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:24.802134+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.992161+0000","last_change":"2024-02-23T00:29:54.071478+0000","last_active":"2024-02-23T00:31:53.992161+0000","last_peered":"2024-02-23T00:31:53.992161+0000","last_clean":"2024-02-23T00:31:53.992161+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:31:53.992161+0000","last_undegraded":"2024-02-23T00:31:53.992161+0000","last_fullsized":"2024-02-23T00:31:53.992161+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:54.071374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:54.071374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":117,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990561+0000","last_change":"2024-02-23T00:29:10.044046+0000","last_active":"2024-02-23T00:31:53.990561+0000","last_peered":"2024-02-23T00:31:53.990561+0000","last_clean":"2024-02-23T00:31:53.990561+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:31:53.990561+0000","last_undegraded":"2024-02-23T00:31:53.990561+0000","last_fullsized":"2024-02-23T00:31:53.990561+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.043966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.043966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":121,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010045+0000","last_change":"2024-02-23T00:30:37.532548+0000","last_active":"2024-02-23T00:31:54.010045+0000","last_peered":"2024-02-23T00:31:54.010045+0000","last_clean":"2024-02-23T00:31:54.010045+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:31:54.010045+0000","last_undegraded":"2024-02-23T00:31:54.010045+0000","last_fullsized":"2024-02-23T00:31:54.010045+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:37.532441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.532441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"148'7","reported_seq":83,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986497+0000","last_change":"2024-02-23T00:30:48.712709+0000","last_active":"2024-02-23T00:31:53.986497+0000","last_peered":"2024-02-23T00:31:53.986497+0000","last_clean":"2024-02-23T00:31:53.986497+0000","last_became_active":"2024-02-23T00:30:48.711845+0000","last_became_peered":"2024-02-23T00:30:48.711845+0000","last_unstale":"2024-02-23T00:31:53.986497+0000","last_undegraded":"2024-02-23T00:31:53.986497+0000","last_fullsized":"2024-02-23T00:31:53.986497+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.987664+0000","last_change":"2024-02-23T00:30:48.703519+0000","last_active":"2024-02-23T00:31:53.987664+0000","last_peered":"2024-02-23T00:31:53.987664+0000","last_clean":"2024-02-23T00:31:53.987664+0000","last_became_active":"2024-02-23T00:30:48.703232+0000","last_became_peered":"2024-02-23T00:30:48.703232+0000","last_unstale":"2024-02-23T00:31:53.987664+0000","last_undegraded":"2024-02-23T00:31:53.987664+0000","last_fullsized":"2024-02-23T00:31:53.987664+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":171,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988774+0000","last_change":"2024-02-23T00:30:37.899931+0000","last_active":"2024-02-23T00:31:53.988774+0000","last_peered":"2024-02-23T00:31:53.988774+0000","last_clean":"2024-02-23T00:31:53.988774+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:31:53.988774+0000","last_undegraded":"2024-02-23T00:31:53.988774+0000","last_fullsized":"2024-02-23T00:31:53.988774+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:30:37.899845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.899845+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":126,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.984256+0000","last_change":"2024-02-23T00:29:53.011199+0000","last_active":"2024-02-23T00:31:53.984256+0000","last_peered":"2024-02-23T00:31:53.984256+0000","last_clean":"2024-02-23T00:31:53.984256+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:31:53.984256+0000","last_undegraded":"2024-02-23T00:31:53.984256+0000","last_fullsized":"2024-02-23T00:31:53.984256+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:53.011089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:29:53.011089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":187,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.995624+0000","last_change":"2024-02-23T00:30:30.131030+0000","last_active":"2024-02-23T00:31:54.995624+0000","last_peered":"2024-02-23T00:31:54.995624+0000","last_clean":"2024-02-23T00:31:54.995624+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:31:54.995624+0000","last_undegraded":"2024-02-23T00:31:54.995624+0000","last_fullsized":"2024-02-23T00:31:54.995624+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:30:30.130950+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.130950+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.007170+0000","last_change":"2024-02-23T00:30:37.889427+0000","last_active":"2024-02-23T00:31:54.007170+0000","last_peered":"2024-02-23T00:31:54.007170+0000","last_clean":"2024-02-23T00:31:54.007170+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:31:54.007170+0000","last_undegraded":"2024-02-23T00:31:54.007170+0000","last_fullsized":"2024-02-23T00:31:54.007170+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:37.889336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:37.889336+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.889336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.3","version":"148'7","reported_seq":83,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988855+0000","last_change":"2024-02-23T00:30:48.683688+0000","last_active":"2024-02-23T00:31:53.988855+0000","last_peered":"2024-02-23T00:31:53.988855+0000","last_clean":"2024-02-23T00:31:53.988855+0000","last_became_active":"2024-02-23T00:30:48.682705+0000","last_became_peered":"2024-02-23T00:30:48.682705+0000","last_unstale":"2024-02-23T00:31:53.988855+0000","last_undegraded":"2024-02-23T00:31:53.988855+0000","last_fullsized":"2024-02-23T00:31:53.988855+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.991852+0000","last_change":"2024-02-23T00:30:48.688217+0000","last_active":"2024-02-23T00:31:53.991852+0000","last_peered":"2024-02-23T00:31:53.991852+0000","last_clean":"2024-02-23T00:31:53.991852+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:31:53.991852+0000","last_undegraded":"2024-02-23T00:31:53.991852+0000","last_fullsized":"2024-02-23T00:31:53.991852+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"144'20","reported_seq":160,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.985687+0000","last_change":"2024-02-23T00:31:35.423292+0000","last_active":"2024-02-23T00:31:53.985687+0000","last_peered":"2024-02-23T00:31:53.985687+0000","last_clean":"2024-02-23T00:31:53.985687+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:31:53.985687+0000","last_undegraded":"2024-02-23T00:31:53.985687+0000","last_fullsized":"2024-02-23T00:31:53.985687+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.423227+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":133,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003120+0000","last_change":"2024-02-23T00:31:35.705318+0000","last_active":"2024-02-23T00:31:54.003120+0000","last_peered":"2024-02-23T00:31:54.003120+0000","last_clean":"2024-02-23T00:31:54.003120+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:31:54.003120+0000","last_undegraded":"2024-02-23T00:31:54.003120+0000","last_fullsized":"2024-02-23T00:31:54.003120+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:35.705240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.705240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'9","reported_seq":189,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.997298+0000","last_change":"2024-02-23T00:29:38.098504+0000","last_active":"2024-02-23T00:31:54.997298+0000","last_peered":"2024-02-23T00:31:54.997298+0000","last_clean":"2024-02-23T00:31:54.997298+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:31:54.997298+0000","last_undegraded":"2024-02-23T00:31:54.997298+0000","last_fullsized":"2024-02-23T00:31:54.997298+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.098442+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.008660+0000","last_change":"2024-02-23T00:30:29.051519+0000","last_active":"2024-02-23T00:31:54.008660+0000","last_peered":"2024-02-23T00:31:54.008660+0000","last_clean":"2024-02-23T00:31:54.008660+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:31:54.008660+0000","last_undegraded":"2024-02-23T00:31:54.008660+0000","last_fullsized":"2024-02-23T00:31:54.008660+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:29.051438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.051438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":76,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.998380+0000","last_change":"2024-02-23T00:30:48.734850+0000","last_active":"2024-02-23T00:31:53.998380+0000","last_peered":"2024-02-23T00:31:53.998380+0000","last_clean":"2024-02-23T00:31:53.998380+0000","last_became_active":"2024-02-23T00:30:48.732126+0000","last_became_peered":"2024-02-23T00:30:48.732126+0000","last_unstale":"2024-02-23T00:31:53.998380+0000","last_undegraded":"2024-02-23T00:31:53.998380+0000","last_fullsized":"2024-02-23T00:31:53.998380+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.992707+0000","last_change":"2024-02-23T00:30:48.717524+0000","last_active":"2024-02-23T00:31:53.992707+0000","last_peered":"2024-02-23T00:31:53.992707+0000","last_clean":"2024-02-23T00:31:53.992707+0000","last_became_active":"2024-02-23T00:30:48.716104+0000","last_became_peered":"2024-02-23T00:30:48.716104+0000","last_unstale":"2024-02-23T00:31:53.992707+0000","last_undegraded":"2024-02-23T00:31:53.992707+0000","last_fullsized":"2024-02-23T00:31:53.992707+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":142,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.002969+0000","last_change":"2024-02-23T00:30:23.600452+0000","last_active":"2024-02-23T00:31:54.002969+0000","last_peered":"2024-02-23T00:31:54.002969+0000","last_clean":"2024-02-23T00:31:54.002969+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:31:54.002969+0000","last_undegraded":"2024-02-23T00:31:54.002969+0000","last_fullsized":"2024-02-23T00:31:54.002969+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:30:23.600311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.600311+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989158+0000","last_change":"2024-02-23T00:29:51.021137+0000","last_active":"2024-02-23T00:31:53.989158+0000","last_peered":"2024-02-23T00:31:53.989158+0000","last_clean":"2024-02-23T00:31:53.989158+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:31:53.989158+0000","last_undegraded":"2024-02-23T00:31:53.989158+0000","last_fullsized":"2024-02-23T00:31:53.989158+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:51.021056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:51.021056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.994380+0000","last_change":"2024-02-23T00:29:32.122532+0000","last_active":"2024-02-23T00:31:53.994380+0000","last_peered":"2024-02-23T00:31:53.994380+0000","last_clean":"2024-02-23T00:31:53.994380+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:31:53.994380+0000","last_undegraded":"2024-02-23T00:31:53.994380+0000","last_fullsized":"2024-02-23T00:31:53.994380+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:32.122430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:32.122430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":127,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.004836+0000","last_change":"2024-02-23T00:31:41.614428+0000","last_active":"2024-02-23T00:31:54.004836+0000","last_peered":"2024-02-23T00:31:54.004836+0000","last_clean":"2024-02-23T00:31:54.004836+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:31:54.004836+0000","last_undegraded":"2024-02-23T00:31:54.004836+0000","last_fullsized":"2024-02-23T00:31:54.004836+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:41.614319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.614319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":76,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.008773+0000","last_change":"2024-02-23T00:30:48.672730+0000","last_active":"2024-02-23T00:31:54.008773+0000","last_peered":"2024-02-23T00:31:54.008773+0000","last_clean":"2024-02-23T00:31:54.008773+0000","last_became_active":"2024-02-23T00:30:48.671603+0000","last_became_peered":"2024-02-23T00:30:48.671603+0000","last_unstale":"2024-02-23T00:31:54.008773+0000","last_undegraded":"2024-02-23T00:31:54.008773+0000","last_fullsized":"2024-02-23T00:31:54.008773+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996024+0000","last_change":"2024-02-23T00:30:48.668969+0000","last_active":"2024-02-23T00:31:53.996024+0000","last_peered":"2024-02-23T00:31:53.996024+0000","last_clean":"2024-02-23T00:31:53.996024+0000","last_became_active":"2024-02-23T00:30:48.666975+0000","last_became_peered":"2024-02-23T00:30:48.666975+0000","last_unstale":"2024-02-23T00:31:53.996024+0000","last_undegraded":"2024-02-23T00:31:53.996024+0000","last_fullsized":"2024-02-23T00:31:53.996024+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":166,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.999561+0000","last_change":"2024-02-23T00:30:41.077799+0000","last_active":"2024-02-23T00:31:53.999561+0000","last_peered":"2024-02-23T00:31:53.999561+0000","last_clean":"2024-02-23T00:31:53.999561+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:31:53.999561+0000","last_undegraded":"2024-02-23T00:31:53.999561+0000","last_fullsized":"2024-02-23T00:31:53.999561+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:30:41.077681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.077681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.021002+0000","last_change":"2024-02-23T00:31:32.874749+0000","last_active":"2024-02-23T00:31:54.021002+0000","last_peered":"2024-02-23T00:31:54.021002+0000","last_clean":"2024-02-23T00:31:54.021002+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:31:54.021002+0000","last_undegraded":"2024-02-23T00:31:54.021002+0000","last_fullsized":"2024-02-23T00:31:54.021002+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:32.874678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:31:32.874678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003058+0000","last_change":"2024-02-23T00:30:30.486006+0000","last_active":"2024-02-23T00:31:54.003058+0000","last_peered":"2024-02-23T00:31:54.003058+0000","last_clean":"2024-02-23T00:31:54.003058+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:31:54.003058+0000","last_undegraded":"2024-02-23T00:31:54.003058+0000","last_fullsized":"2024-02-23T00:31:54.003058+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:30.485900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.485900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.987630+0000","last_change":"2024-02-23T00:30:47.843046+0000","last_active":"2024-02-23T00:31:53.987630+0000","last_peered":"2024-02-23T00:31:53.987630+0000","last_clean":"2024-02-23T00:31:53.987630+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:31:53.987630+0000","last_undegraded":"2024-02-23T00:31:53.987630+0000","last_fullsized":"2024-02-23T00:31:53.987630+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.842931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":76,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.020566+0000","last_change":"2024-02-23T00:30:48.721760+0000","last_active":"2024-02-23T00:31:54.020566+0000","last_peered":"2024-02-23T00:31:54.020566+0000","last_clean":"2024-02-23T00:31:54.020566+0000","last_became_active":"2024-02-23T00:30:48.721541+0000","last_became_peered":"2024-02-23T00:30:48.721541+0000","last_unstale":"2024-02-23T00:31:54.020566+0000","last_undegraded":"2024-02-23T00:31:54.020566+0000","last_fullsized":"2024-02-23T00:31:54.020566+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":88,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.992954+0000","last_change":"2024-02-23T00:30:48.705018+0000","last_active":"2024-02-23T00:31:53.992954+0000","last_peered":"2024-02-23T00:31:53.992954+0000","last_clean":"2024-02-23T00:31:53.992954+0000","last_became_active":"2024-02-23T00:30:48.703281+0000","last_became_peered":"2024-02-23T00:30:48.703281+0000","last_unstale":"2024-02-23T00:31:53.992954+0000","last_undegraded":"2024-02-23T00:31:53.992954+0000","last_fullsized":"2024-02-23T00:31:53.992954+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":134,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986087+0000","last_change":"2024-02-23T00:29:43.108230+0000","last_active":"2024-02-23T00:31:53.986087+0000","last_peered":"2024-02-23T00:31:53.986087+0000","last_clean":"2024-02-23T00:31:53.986087+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:31:53.986087+0000","last_undegraded":"2024-02-23T00:31:53.986087+0000","last_fullsized":"2024-02-23T00:31:53.986087+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:29:43.108156+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:29:43.108156+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":126,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.991397+0000","last_change":"2024-02-23T00:30:26.059320+0000","last_active":"2024-02-23T00:31:53.991397+0000","last_peered":"2024-02-23T00:31:53.991397+0000","last_clean":"2024-02-23T00:31:53.991397+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:31:53.991397+0000","last_undegraded":"2024-02-23T00:31:53.991397+0000","last_fullsized":"2024-02-23T00:31:53.991397+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:26.059244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:26.059244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.992073+0000","last_change":"2024-02-23T00:30:25.163377+0000","last_active":"2024-02-23T00:31:53.992073+0000","last_peered":"2024-02-23T00:31:53.992073+0000","last_clean":"2024-02-23T00:31:53.992073+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:31:53.992073+0000","last_undegraded":"2024-02-23T00:31:53.992073+0000","last_fullsized":"2024-02-23T00:31:53.992073+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.163297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.163297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":121,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.000016+0000","last_change":"2024-02-23T00:30:34.561832+0000","last_active":"2024-02-23T00:31:54.000016+0000","last_peered":"2024-02-23T00:31:54.000016+0000","last_clean":"2024-02-23T00:31:54.000016+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:31:54.000016+0000","last_undegraded":"2024-02-23T00:31:54.000016+0000","last_fullsized":"2024-02-23T00:31:54.000016+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:34.561766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:34.561766+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":76,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988537+0000","last_change":"2024-02-23T00:30:48.709585+0000","last_active":"2024-02-23T00:31:53.988537+0000","last_peered":"2024-02-23T00:31:53.988537+0000","last_clean":"2024-02-23T00:31:53.988537+0000","last_became_active":"2024-02-23T00:30:48.709325+0000","last_became_peered":"2024-02-23T00:30:48.709325+0000","last_unstale":"2024-02-23T00:31:53.988537+0000","last_undegraded":"2024-02-23T00:31:53.988537+0000","last_fullsized":"2024-02-23T00:31:53.988537+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003226+0000","last_change":"2024-02-23T00:30:48.696251+0000","last_active":"2024-02-23T00:31:54.003226+0000","last_peered":"2024-02-23T00:31:54.003226+0000","last_clean":"2024-02-23T00:31:54.003226+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:31:54.003226+0000","last_undegraded":"2024-02-23T00:31:54.003226+0000","last_fullsized":"2024-02-23T00:31:54.003226+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":167,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.993141+0000","last_change":"2024-02-23T00:31:27.176319+0000","last_active":"2024-02-23T00:31:53.993141+0000","last_peered":"2024-02-23T00:31:53.993141+0000","last_clean":"2024-02-23T00:31:53.993141+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:31:53.993141+0000","last_undegraded":"2024-02-23T00:31:53.993141+0000","last_fullsized":"2024-02-23T00:31:53.993141+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:31:27.176199+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:31:27.176199+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":127,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.995878+0000","last_change":"2024-02-23T00:30:22.245444+0000","last_active":"2024-02-23T00:31:53.995878+0000","last_peered":"2024-02-23T00:31:53.995878+0000","last_clean":"2024-02-23T00:31:53.995878+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:31:53.995878+0000","last_undegraded":"2024-02-23T00:31:53.995878+0000","last_fullsized":"2024-02-23T00:31:53.995878+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:22.245359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.245359+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":123,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.011176+0000","last_change":"2024-02-23T00:30:33.528773+0000","last_active":"2024-02-23T00:31:54.011176+0000","last_peered":"2024-02-23T00:31:54.011176+0000","last_clean":"2024-02-23T00:31:54.011176+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:31:54.011176+0000","last_undegraded":"2024-02-23T00:31:54.011176+0000","last_fullsized":"2024-02-23T00:31:54.011176+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:33.528703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:33.528703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.994585+0000","last_change":"2024-02-23T00:30:31.829814+0000","last_active":"2024-02-23T00:31:53.994585+0000","last_peered":"2024-02-23T00:31:53.994585+0000","last_clean":"2024-02-23T00:31:53.994585+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:31:53.994585+0000","last_undegraded":"2024-02-23T00:31:53.994585+0000","last_fullsized":"2024-02-23T00:31:53.994585+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.829697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.829697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.6","version":"148'11","reported_seq":87,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006918+0000","last_change":"2024-02-23T00:30:48.706598+0000","last_active":"2024-02-23T00:31:54.006918+0000","last_peered":"2024-02-23T00:31:54.006918+0000","last_clean":"2024-02-23T00:31:54.006918+0000","last_became_active":"2024-02-23T00:30:48.699445+0000","last_became_peered":"2024-02-23T00:30:48.699445+0000","last_unstale":"2024-02-23T00:31:54.006918+0000","last_undegraded":"2024-02-23T00:31:54.006918+0000","last_fullsized":"2024-02-23T00:31:54.006918+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.999928+0000","last_change":"2024-02-23T00:30:48.723611+0000","last_active":"2024-02-23T00:31:53.999928+0000","last_peered":"2024-02-23T00:31:53.999928+0000","last_clean":"2024-02-23T00:31:53.999928+0000","last_became_active":"2024-02-23T00:30:48.722602+0000","last_became_peered":"2024-02-23T00:30:48.722602+0000","last_unstale":"2024-02-23T00:31:53.999928+0000","last_undegraded":"2024-02-23T00:31:53.999928+0000","last_fullsized":"2024-02-23T00:31:53.999928+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"92'22","reported_seq":162,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.007497+0000","last_change":"2024-02-23T00:30:38.900864+0000","last_active":"2024-02-23T00:31:54.007497+0000","last_peered":"2024-02-23T00:31:54.007497+0000","last_clean":"2024-02-23T00:31:54.007497+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:31:54.007497+0000","last_undegraded":"2024-02-23T00:31:54.007497+0000","last_fullsized":"2024-02-23T00:31:54.007497+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.900761+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":126,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.983307+0000","last_change":"2024-02-23T00:31:20.333610+0000","last_active":"2024-02-23T00:31:53.983307+0000","last_peered":"2024-02-23T00:31:53.983307+0000","last_clean":"2024-02-23T00:31:53.983307+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:31:53.983307+0000","last_undegraded":"2024-02-23T00:31:53.983307+0000","last_fullsized":"2024-02-23T00:31:53.983307+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:20.333547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:20.333547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":196,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.989633+0000","last_change":"2024-02-23T00:31:19.971484+0000","last_active":"2024-02-23T00:31:54.989633+0000","last_peered":"2024-02-23T00:31:54.989633+0000","last_clean":"2024-02-23T00:31:54.989633+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:31:54.989633+0000","last_undegraded":"2024-02-23T00:31:54.989633+0000","last_fullsized":"2024-02-23T00:31:54.989633+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:31:19.971419+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.971419+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":123,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006212+0000","last_change":"2024-02-23T00:30:35.523228+0000","last_active":"2024-02-23T00:31:54.006212+0000","last_peered":"2024-02-23T00:31:54.006212+0000","last_clean":"2024-02-23T00:31:54.006212+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:31:54.006212+0000","last_undegraded":"2024-02-23T00:31:54.006212+0000","last_fullsized":"2024-02-23T00:31:54.006212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:30:35.523157+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.523157+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":76,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.005049+0000","last_change":"2024-02-23T00:30:48.695030+0000","last_active":"2024-02-23T00:31:54.005049+0000","last_peered":"2024-02-23T00:31:54.005049+0000","last_clean":"2024-02-23T00:31:54.005049+0000","last_became_active":"2024-02-23T00:30:48.694604+0000","last_became_peered":"2024-02-23T00:30:48.694604+0000","last_unstale":"2024-02-23T00:31:54.005049+0000","last_undegraded":"2024-02-23T00:31:54.005049+0000","last_fullsized":"2024-02-23T00:31:54.005049+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006263+0000","last_change":"2024-02-23T00:30:48.748188+0000","last_active":"2024-02-23T00:31:54.006263+0000","last_peered":"2024-02-23T00:31:54.006263+0000","last_clean":"2024-02-23T00:31:54.006263+0000","last_became_active":"2024-02-23T00:30:48.745982+0000","last_became_peered":"2024-02-23T00:30:48.745982+0000","last_unstale":"2024-02-23T00:31:54.006263+0000","last_undegraded":"2024-02-23T00:31:54.006263+0000","last_fullsized":"2024-02-23T00:31:54.006263+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":160,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.998667+0000","last_change":"2024-02-23T00:30:30.207937+0000","last_active":"2024-02-23T00:31:53.998667+0000","last_peered":"2024-02-23T00:31:53.998667+0000","last_clean":"2024-02-23T00:31:53.998667+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:31:53.998667+0000","last_undegraded":"2024-02-23T00:31:53.998667+0000","last_fullsized":"2024-02-23T00:31:53.998667+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:30:30.207850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.207850+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.992304+0000","last_change":"2024-02-23T00:29:49.056255+0000","last_active":"2024-02-23T00:31:53.992304+0000","last_peered":"2024-02-23T00:31:53.992304+0000","last_clean":"2024-02-23T00:31:53.992304+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:31:53.992304+0000","last_undegraded":"2024-02-23T00:31:53.992304+0000","last_fullsized":"2024-02-23T00:31:53.992304+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:49.056184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:49.056184+0000","last_clean_scrub_stamp":"2024-02-23T00:29:49.056184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.007497+0000","last_change":"2024-02-23T00:30:34.869530+0000","last_active":"2024-02-23T00:31:54.007497+0000","last_peered":"2024-02-23T00:31:54.007497+0000","last_clean":"2024-02-23T00:31:54.007497+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:31:54.007497+0000","last_undegraded":"2024-02-23T00:31:54.007497+0000","last_fullsized":"2024-02-23T00:31:54.007497+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:34.869440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:34.869440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.011494+0000","last_change":"2024-02-23T00:29:37.520506+0000","last_active":"2024-02-23T00:31:54.011494+0000","last_peered":"2024-02-23T00:31:54.011494+0000","last_clean":"2024-02-23T00:31:54.011494+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:31:54.011494+0000","last_undegraded":"2024-02-23T00:31:54.011494+0000","last_fullsized":"2024-02-23T00:31:54.011494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:37.520440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:37.520440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.8","version":"98'2","reported_seq":78,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.987440+0000","last_change":"2024-02-23T00:30:48.782205+0000","last_active":"2024-02-23T00:31:53.987440+0000","last_peered":"2024-02-23T00:31:53.987440+0000","last_clean":"2024-02-23T00:31:53.987440+0000","last_became_active":"2024-02-23T00:30:48.781887+0000","last_became_peered":"2024-02-23T00:30:48.781887+0000","last_unstale":"2024-02-23T00:31:53.987440+0000","last_undegraded":"2024-02-23T00:31:53.987440+0000","last_fullsized":"2024-02-23T00:31:53.987440+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010588+0000","last_change":"2024-02-23T00:30:48.737859+0000","last_active":"2024-02-23T00:31:54.010588+0000","last_peered":"2024-02-23T00:31:54.010588+0000","last_clean":"2024-02-23T00:31:54.010588+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:31:54.010588+0000","last_undegraded":"2024-02-23T00:31:54.010588+0000","last_fullsized":"2024-02-23T00:31:54.010588+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":156,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.000330+0000","last_change":"2024-02-23T00:31:31.806791+0000","last_active":"2024-02-23T00:31:54.000330+0000","last_peered":"2024-02-23T00:31:54.000330+0000","last_clean":"2024-02-23T00:31:54.000330+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:31:54.000330+0000","last_undegraded":"2024-02-23T00:31:54.000330+0000","last_fullsized":"2024-02-23T00:31:54.000330+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:31:31.806687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.806687+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":133,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.994146+0000","last_change":"2024-02-23T00:30:40.183533+0000","last_active":"2024-02-23T00:31:53.994146+0000","last_peered":"2024-02-23T00:31:53.994146+0000","last_clean":"2024-02-23T00:31:53.994146+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:31:53.994146+0000","last_undegraded":"2024-02-23T00:31:53.994146+0000","last_fullsized":"2024-02-23T00:31:53.994146+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.183453+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":126,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.023682+0000","last_change":"2024-02-23T00:31:38.887671+0000","last_active":"2024-02-23T00:31:54.023682+0000","last_peered":"2024-02-23T00:31:54.023682+0000","last_clean":"2024-02-23T00:31:54.023682+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:31:54.023682+0000","last_undegraded":"2024-02-23T00:31:54.023682+0000","last_fullsized":"2024-02-23T00:31:54.023682+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.887597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":117,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006503+0000","last_change":"2024-02-23T00:29:10.830479+0000","last_active":"2024-02-23T00:31:54.006503+0000","last_peered":"2024-02-23T00:31:54.006503+0000","last_clean":"2024-02-23T00:31:54.006503+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:31:54.006503+0000","last_undegraded":"2024-02-23T00:31:54.006503+0000","last_fullsized":"2024-02-23T00:31:54.006503+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:10.830389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:10.830389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.b","version":"98'2","reported_seq":31,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.983776+0000","last_change":"2024-02-23T00:31:39.015830+0000","last_active":"2024-02-23T00:31:53.983776+0000","last_peered":"2024-02-23T00:31:53.983776+0000","last_clean":"2024-02-23T00:31:53.983776+0000","last_became_active":"2024-02-23T00:31:38.999851+0000","last_became_peered":"2024-02-23T00:31:38.999851+0000","last_unstale":"2024-02-23T00:31:53.983776+0000","last_undegraded":"2024-02-23T00:31:53.983776+0000","last_fullsized":"2024-02-23T00:31:53.983776+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.991761+0000","last_change":"2024-02-23T00:30:48.711374+0000","last_active":"2024-02-23T00:31:53.991761+0000","last_peered":"2024-02-23T00:31:53.991761+0000","last_clean":"2024-02-23T00:31:53.991761+0000","last_became_active":"2024-02-23T00:30:48.710049+0000","last_became_peered":"2024-02-23T00:30:48.710049+0000","last_unstale":"2024-02-23T00:31:53.991761+0000","last_undegraded":"2024-02-23T00:31:53.991761+0000","last_fullsized":"2024-02-23T00:31:53.991761+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":170,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.992290+0000","last_change":"2024-02-23T00:30:29.844070+0000","last_active":"2024-02-23T00:31:53.992290+0000","last_peered":"2024-02-23T00:31:53.992290+0000","last_clean":"2024-02-23T00:31:53.992290+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:31:53.992290+0000","last_undegraded":"2024-02-23T00:31:53.992290+0000","last_fullsized":"2024-02-23T00:31:53.992290+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:30:29.843954+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.843954+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.994382+0000","last_change":"2024-02-23T00:30:32.878666+0000","last_active":"2024-02-23T00:31:53.994382+0000","last_peered":"2024-02-23T00:31:53.994382+0000","last_clean":"2024-02-23T00:31:53.994382+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:31:53.994382+0000","last_undegraded":"2024-02-23T00:31:53.994382+0000","last_fullsized":"2024-02-23T00:31:53.994382+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:32.878569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:32.878569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.002206+0000","last_change":"2024-02-23T00:31:45.961466+0000","last_active":"2024-02-23T00:31:54.002206+0000","last_peered":"2024-02-23T00:31:54.002206+0000","last_clean":"2024-02-23T00:31:54.002206+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:31:54.002206+0000","last_undegraded":"2024-02-23T00:31:54.002206+0000","last_fullsized":"2024-02-23T00:31:54.002206+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:45.961386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:45.961386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.989256+0000","last_change":"2024-02-23T00:29:47.044859+0000","last_active":"2024-02-23T00:31:53.989256+0000","last_peered":"2024-02-23T00:31:53.989256+0000","last_clean":"2024-02-23T00:31:53.989256+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:31:53.989256+0000","last_undegraded":"2024-02-23T00:31:53.989256+0000","last_fullsized":"2024-02-23T00:31:53.989256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:47.044799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:47.044799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.a","version":"148'14","reported_seq":90,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.996790+0000","last_change":"2024-02-23T00:30:48.751955+0000","last_active":"2024-02-23T00:31:53.996790+0000","last_peered":"2024-02-23T00:31:53.996790+0000","last_clean":"2024-02-23T00:31:53.996790+0000","last_became_active":"2024-02-23T00:30:48.726997+0000","last_became_peered":"2024-02-23T00:30:48.726997+0000","last_unstale":"2024-02-23T00:31:53.996790+0000","last_undegraded":"2024-02-23T00:31:53.996790+0000","last_fullsized":"2024-02-23T00:31:53.996790+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.990500+0000","last_change":"2024-02-23T00:30:48.679036+0000","last_active":"2024-02-23T00:31:53.990500+0000","last_peered":"2024-02-23T00:31:53.990500+0000","last_clean":"2024-02-23T00:31:53.990500+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:31:53.990500+0000","last_undegraded":"2024-02-23T00:31:53.990500+0000","last_fullsized":"2024-02-23T00:31:53.990500+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":172,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010923+0000","last_change":"2024-02-23T00:29:38.857115+0000","last_active":"2024-02-23T00:31:54.010923+0000","last_peered":"2024-02-23T00:31:54.010923+0000","last_clean":"2024-02-23T00:31:54.010923+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:31:54.010923+0000","last_undegraded":"2024-02-23T00:31:54.010923+0000","last_fullsized":"2024-02-23T00:31:54.010923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:29:38.857029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.857029+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":127,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.005919+0000","last_change":"2024-02-23T00:30:47.629232+0000","last_active":"2024-02-23T00:31:54.005919+0000","last_peered":"2024-02-23T00:31:54.005919+0000","last_clean":"2024-02-23T00:31:54.005919+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:31:54.005919+0000","last_undegraded":"2024-02-23T00:31:54.005919+0000","last_fullsized":"2024-02-23T00:31:54.005919+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.629146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.629146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.986533+0000","last_change":"2024-02-23T00:30:41.121986+0000","last_active":"2024-02-23T00:31:53.986533+0000","last_peered":"2024-02-23T00:31:53.986533+0000","last_clean":"2024-02-23T00:31:53.986533+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:31:53.986533+0000","last_undegraded":"2024-02-23T00:31:53.986533+0000","last_fullsized":"2024-02-23T00:31:53.986533+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:41.121916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.121916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.005928+0000","last_change":"2024-02-23T00:30:31.986713+0000","last_active":"2024-02-23T00:31:54.005928+0000","last_peered":"2024-02-23T00:31:54.005928+0000","last_clean":"2024-02-23T00:31:54.005928+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:31:54.005928+0000","last_undegraded":"2024-02-23T00:31:54.005928+0000","last_fullsized":"2024-02-23T00:31:54.005928+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.986615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.986615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.d","version":"148'9","reported_seq":50,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006511+0000","last_change":"2024-02-23T00:31:53.000538+0000","last_active":"2024-02-23T00:31:54.006511+0000","last_peered":"2024-02-23T00:31:54.006511+0000","last_clean":"2024-02-23T00:31:54.006511+0000","last_became_active":"2024-02-23T00:31:53.000164+0000","last_became_peered":"2024-02-23T00:31:53.000164+0000","last_unstale":"2024-02-23T00:31:54.006511+0000","last_undegraded":"2024-02-23T00:31:54.006511+0000","last_fullsized":"2024-02-23T00:31:54.006511+0000","mapping_epoch":147,"log_start":"148'9","ondisk_log_start":"148'9","created":96,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.006093+0000","last_change":"2024-02-23T00:30:48.735843+0000","last_active":"2024-02-23T00:31:54.006093+0000","last_peered":"2024-02-23T00:31:54.006093+0000","last_clean":"2024-02-23T00:31:54.006093+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:31:54.006093+0000","last_undegraded":"2024-02-23T00:31:54.006093+0000","last_fullsized":"2024-02-23T00:31:54.006093+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":154,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.999791+0000","last_change":"2024-02-23T00:30:23.953417+0000","last_active":"2024-02-23T00:31:53.999791+0000","last_peered":"2024-02-23T00:31:53.999791+0000","last_clean":"2024-02-23T00:31:53.999791+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:31:53.999791+0000","last_undegraded":"2024-02-23T00:31:53.999791+0000","last_fullsized":"2024-02-23T00:31:53.999791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:30:23.953330+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.953330+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":127,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010681+0000","last_change":"2024-02-23T00:30:39.570738+0000","last_active":"2024-02-23T00:31:54.010681+0000","last_peered":"2024-02-23T00:31:54.010681+0000","last_clean":"2024-02-23T00:31:54.010681+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:31:54.010681+0000","last_undegraded":"2024-02-23T00:31:54.010681+0000","last_fullsized":"2024-02-23T00:31:54.010681+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:39.570668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:30:39.570668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":123,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.999903+0000","last_change":"2024-02-23T00:30:38.549305+0000","last_active":"2024-02-23T00:31:53.999903+0000","last_peered":"2024-02-23T00:31:53.999903+0000","last_clean":"2024-02-23T00:31:53.999903+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:31:53.999903+0000","last_undegraded":"2024-02-23T00:31:53.999903+0000","last_fullsized":"2024-02-23T00:31:53.999903+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.549228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":120,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.983404+0000","last_change":"2024-02-23T00:31:39.494827+0000","last_active":"2024-02-23T00:31:53.983404+0000","last_peered":"2024-02-23T00:31:53.983404+0000","last_clean":"2024-02-23T00:31:53.983404+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:31:53.983404+0000","last_undegraded":"2024-02-23T00:31:53.983404+0000","last_fullsized":"2024-02-23T00:31:53.983404+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:31:39.494737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":35,"reported_epoch":164,"state":"clean+premerge+peered","last_fresh":"2024-02-23T00:31:55.010095+0000","last_change":"2024-02-23T00:31:55.010095+0000","last_active":"2024-02-23T00:31:54.015430+0000","last_peered":"2024-02-23T00:31:55.010095+0000","last_clean":"2024-02-23T00:31:55.010095+0000","last_became_active":"2024-02-23T00:31:39.006874+0000","last_became_peered":"2024-02-23T00:31:55.008700+0000","last_unstale":"2024-02-23T00:31:55.010095+0000","last_undegraded":"2024-02-23T00:31:55.010095+0000","last_fullsized":"2024-02-23T00:31:55.010095+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.019243+0000","last_change":"2024-02-23T00:30:48.725798+0000","last_active":"2024-02-23T00:31:54.019243+0000","last_peered":"2024-02-23T00:31:54.019243+0000","last_clean":"2024-02-23T00:31:54.019243+0000","last_became_active":"2024-02-23T00:30:48.725162+0000","last_became_peered":"2024-02-23T00:30:48.725162+0000","last_unstale":"2024-02-23T00:31:54.019243+0000","last_undegraded":"2024-02-23T00:31:54.019243+0000","last_fullsized":"2024-02-23T00:31:54.019243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":157,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.995503+0000","last_change":"2024-02-23T00:29:06.822899+0000","last_active":"2024-02-23T00:31:53.995503+0000","last_peered":"2024-02-23T00:31:53.995503+0000","last_clean":"2024-02-23T00:31:53.995503+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:31:53.995503+0000","last_undegraded":"2024-02-23T00:31:53.995503+0000","last_fullsized":"2024-02-23T00:31:53.995503+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:29:06.822818+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":130,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988874+0000","last_change":"2024-02-23T00:31:22.137283+0000","last_active":"2024-02-23T00:31:53.988874+0000","last_peered":"2024-02-23T00:31:53.988874+0000","last_clean":"2024-02-23T00:31:53.988874+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:31:53.988874+0000","last_undegraded":"2024-02-23T00:31:53.988874+0000","last_fullsized":"2024-02-23T00:31:53.988874+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:22.137166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:22.137166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.018828+0000","last_change":"2024-02-23T00:31:26.920222+0000","last_active":"2024-02-23T00:31:54.018828+0000","last_peered":"2024-02-23T00:31:54.018828+0000","last_clean":"2024-02-23T00:31:54.018828+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:31:54.018828+0000","last_undegraded":"2024-02-23T00:31:54.018828+0000","last_fullsized":"2024-02-23T00:31:54.018828+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:26.920133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:26.920133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.010835+0000","last_change":"2024-02-23T00:31:37.065941+0000","last_active":"2024-02-23T00:31:54.010835+0000","last_peered":"2024-02-23T00:31:54.010835+0000","last_clean":"2024-02-23T00:31:54.010835+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:31:54.010835+0000","last_undegraded":"2024-02-23T00:31:54.010835+0000","last_fullsized":"2024-02-23T00:31:54.010835+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:31:37.065865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.f","version":"148'7","reported_seq":44,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988279+0000","last_change":"2024-02-23T00:31:45.006360+0000","last_active":"2024-02-23T00:31:53.988279+0000","last_peered":"2024-02-23T00:31:53.988279+0000","last_clean":"2024-02-23T00:31:53.988279+0000","last_became_active":"2024-02-23T00:31:45.006018+0000","last_became_peered":"2024-02-23T00:31:45.006018+0000","last_unstale":"2024-02-23T00:31:53.988279+0000","last_undegraded":"2024-02-23T00:31:53.988279+0000","last_fullsized":"2024-02-23T00:31:53.988279+0000","mapping_epoch":147,"log_start":"148'7","ondisk_log_start":"148'7","created":96,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.000198+0000","last_change":"2024-02-23T00:30:48.698701+0000","last_active":"2024-02-23T00:31:54.000198+0000","last_peered":"2024-02-23T00:31:54.000198+0000","last_clean":"2024-02-23T00:31:54.000198+0000","last_became_active":"2024-02-23T00:30:48.697963+0000","last_became_peered":"2024-02-23T00:30:48.697963+0000","last_unstale":"2024-02-23T00:31:54.000198+0000","last_undegraded":"2024-02-23T00:31:54.000198+0000","last_fullsized":"2024-02-23T00:31:54.000198+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"63'39","reported_seq":211,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.988243+0000","last_change":"2024-02-23T00:30:23.839691+0000","last_active":"2024-02-23T00:31:53.988243+0000","last_peered":"2024-02-23T00:31:53.988243+0000","last_clean":"2024-02-23T00:31:53.988243+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:31:53.988243+0000","last_undegraded":"2024-02-23T00:31:53.988243+0000","last_fullsized":"2024-02-23T00:31:53.988243+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'39","last_scrub_stamp":"2024-02-23T00:30:23.839568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.839568+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":128,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.009152+0000","last_change":"2024-02-23T00:30:26.074372+0000","last_active":"2024-02-23T00:31:54.009152+0000","last_peered":"2024-02-23T00:31:54.009152+0000","last_clean":"2024-02-23T00:31:54.009152+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:31:54.009152+0000","last_undegraded":"2024-02-23T00:31:54.009152+0000","last_fullsized":"2024-02-23T00:31:54.009152+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:26.074262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:26.074262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.995936+0000","last_change":"2024-02-23T00:31:40.071364+0000","last_active":"2024-02-23T00:31:53.995936+0000","last_peered":"2024-02-23T00:31:53.995936+0000","last_clean":"2024-02-23T00:31:53.995936+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:31:53.995936+0000","last_undegraded":"2024-02-23T00:31:53.995936+0000","last_fullsized":"2024-02-23T00:31:53.995936+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.071294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.071294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:53.994029+0000","last_change":"2024-02-23T00:30:46.848126+0000","last_active":"2024-02-23T00:31:53.994029+0000","last_peered":"2024-02-23T00:31:53.994029+0000","last_clean":"2024-02-23T00:31:53.994029+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:31:53.994029+0000","last_undegraded":"2024-02-23T00:31:53.994029+0000","last_fullsized":"2024-02-23T00:31:53.994029+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.848063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.848063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":44,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.003991+0000","last_change":"2024-02-23T00:31:49.020151+0000","last_active":"2024-02-23T00:31:54.003991+0000","last_peered":"2024-02-23T00:31:54.003991+0000","last_clean":"2024-02-23T00:31:54.003991+0000","last_became_active":"2024-02-23T00:31:49.019835+0000","last_became_peered":"2024-02-23T00:31:49.019835+0000","last_unstale":"2024-02-23T00:31:54.003991+0000","last_undegraded":"2024-02-23T00:31:54.003991+0000","last_fullsized":"2024-02-23T00:31:54.003991+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":75,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-02-23T00:31:54.004873+0000","last_change":"2024-02-23T00:30:48.722860+0000","last_active":"2024-02-23T00:31:54.004873+0000","last_peered":"2024-02-23T00:31:54.004873+0000","last_clean":"2024-02-23T00:31:54.004873+0000","last_became_active":"2024-02-23T00:30:48.722569+0000","last_became_peered":"2024-02-23T00:30:48.722569+0000","last_unstale":"2024-02-23T00:31:54.004873+0000","last_undegraded":"2024-02-23T00:31:54.004873+0000","last_fullsized":"2024-02-23T00:31:54.004873+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":95,"num_pg":32,"stat_sum":{"num_bytes":2281,"num_objects":221,"num_object_clones":141,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":222,"num_read_kb":222,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2715648,"data_stored":6843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":8},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":5227,"num_objects":505,"num_object_clones":400,"num_object_copies":1515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":86,"num_read_kb":1,"num_write":510,"num_write_kb":509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6205440,"data_stored":15681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":922,"ondisk_log_size":922,"up":96,"acting":96,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":19,"num_pg":28,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":84,"acting":84,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":640,"ondisk_log_size":640,"up":96,"acting":96,"num_store_stats":8},{"poolid":114,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":269,"num_read_kb":264,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":112,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":113,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":142,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":398592,"kb_used_data":101368,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93346560,"statfs":{"total":95995035648,"available":95586877440,"internally_reserved":0,"allocated":103800832,"data_stored":101989247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:31:25 2024","interfaces":[{"interface":"back","average":{"1min":5.7119999999999997,"5min":1.5509999999999999,"15min":0.85099999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":61.048999999999999,"5min":61.048999999999999,"15min":61.048999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":8.0839999999999996,"5min":2.032,"15min":1.032},"min":{"1min":0.32400000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":73.870999999999995,"5min":73.870999999999995,"15min":73.870999999999995},"last":2.323}]},{"osd":1,"last update":"Fri Feb 23 00:31:25 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":0.72599999999999998,"15min":0.59499999999999997},"min":{"1min":0.318,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":8.0890000000000004,"5min":8.0890000000000004,"15min":8.0890000000000004},"last":2.4550000000000001},{"interface":"front","average":{"1min":1.8009999999999999,"5min":0.77100000000000002,"15min":0.59299999999999997},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.907999999999999,"5min":13.907999999999999,"15min":13.907999999999999},"last":1.544}]},{"osd":2,"last update":"Fri Feb 23 00:31:25 2024","interfaces":[{"interface":"back","average":{"1min":2.234,"5min":0.86399999999999999,"15min":0.63900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":16.603999999999999,"5min":16.603999999999999,"15min":16.603999999999999},"last":2.0920000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":0.68100000000000005,"15min":0.57599999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":4.9710000000000001,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":2.1880000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:31:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5609999999999999,"5min":1.3300000000000001,"15min":0.79100000000000004},"min":{"1min":0.316,"5min":0.311,"15min":0.311},"max":{"1min":33.473999999999997,"5min":33.473999999999997,"15min":33.473999999999997},"last":1.077},{"interface":"front","average":{"1min":4.6719999999999997,"5min":1.3600000000000001,"15min":0.81000000000000005},"min":{"1min":0.378,"5min":0.312,"15min":0.312},"max":{"1min":27.195,"5min":27.195,"15min":27.195},"last":0.82099999999999995}]},{"osd":4,"last update":"Fri Feb 23 00:31:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.42199999999999999,"15min":0.41299999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55500000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.754},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.433,"15min":0.42599999999999999},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.55200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.73299999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:31:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.42999999999999999,"15min":0.42799999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57899999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.45500000000000002,"15min":0.435},"min":{"1min":0.318,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.51500000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.72099999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:31:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.48199999999999998,"15min":0.47299999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56599999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.57199999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.79500000000000004}]}]},{"osd":6,"up_from":44,"seq":188978561088,"num_pgs":119,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":378508,"kb_used_data":77188,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93366644,"statfs":{"total":95995035648,"available":95607443456,"internally_reserved":0,"allocated":79040512,"data_stored":76835268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.54200000000000004,"15min":0.51200000000000001},"min":{"1min":0.27400000000000002,"5min":0.22,"15min":0.22},"max":{"1min":3.3140000000000001,"5min":3.3140000000000001,"15min":3.3140000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.57799999999999996,"15min":0.54000000000000004},"min":{"1min":0.35999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":8.0269999999999992,"5min":8.0269999999999992,"15min":8.0269999999999992},"last":0.376}]},{"osd":1,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":0.68100000000000005,"15min":0.56299999999999994},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":17.849,"5min":17.849,"15min":17.849},"last":0.438},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.60399999999999998,"15min":0.53300000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":11.574,"5min":11.574,"15min":11.574},"last":0.44600000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":0.77600000000000002,"15min":0.58999999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":25.055,"5min":25.055,"15min":25.055},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.911,"5min":0.79800000000000004,"15min":0.63200000000000001},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":26.622,"5min":26.622,"15min":26.622},"last":0.47999999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":0.72899999999999998,"15min":0.56899999999999995},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.307},"max":{"1min":17.908999999999999,"5min":17.908999999999999,"15min":17.908999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":2.9449999999999998,"5min":1.014,"15min":0.68700000000000006},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":30.834,"5min":30.834,"15min":30.834},"last":0.46100000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.41699999999999998},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.80300000000000005,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.415,"5min":0.63500000000000001,"15min":0.48099999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":17.718,"5min":17.718,"15min":17.718},"last":0.35999999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.53300000000000003,"15min":0.46300000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":8.0990000000000002,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.54300000000000004,"15min":0.46000000000000002},"min":{"1min":0.32100000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.1370000000000005,"5min":8.1370000000000005,"15min":8.1370000000000005},"last":0.31}]},{"osd":7,"last update":"Fri Feb 23 00:31:18 2024","interfaces":[{"interface":"back","average":{"1min":2.2389999999999999,"5min":0.84299999999999997,"15min":0.59299999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":17.881,"5min":17.881,"15min":17.881},"last":0.39500000000000002},{"interface":"front","average":{"1min":2.0219999999999998,"5min":0.80700000000000005,"15min":0.58399999999999996},"min":{"1min":0.28699999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":17.396999999999998,"5min":17.396999999999998,"15min":17.396999999999998},"last":0.42999999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":155,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":379060,"kb_used_data":77740,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93366092,"statfs":{"total":95995035648,"available":95606878208,"internally_reserved":0,"allocated":79605760,"data_stored":76824247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.677,"5min":2.677,"15min":2.677},"last":2.4540000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.41899999999999998,"15min":0.36699999999999999},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.20799999999999999},"max":{"1min":4.7309999999999999,"5min":4.7309999999999999,"15min":4.7309999999999999},"last":3.9700000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:31:18 2024","interfaces":[{"interface":"back","average":{"1min":2.9409999999999998,"5min":0.875,"15min":0.53900000000000003},"min":{"1min":0.26600000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":29.140999999999998,"5min":29.140999999999998,"15min":29.140999999999998},"last":2.7570000000000001},{"interface":"front","average":{"1min":1.327,"5min":0.56899999999999995,"15min":0.436},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":15.273,"5min":15.273,"15min":15.273},"last":2.5059999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:31:43 2024","interfaces":[{"interface":"back","average":{"1min":3.5790000000000002,"5min":1.016,"15min":0.626},"min":{"1min":0.29499999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":21.013999999999999,"5min":21.013999999999999,"15min":21.013999999999999},"last":3.9449999999999998},{"interface":"front","average":{"1min":3.4590000000000001,"5min":1.02,"15min":0.629},"min":{"1min":0.316,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":29.234999999999999,"5min":29.234999999999999,"15min":29.234999999999999},"last":2.79}]},{"osd":4,"last update":"Fri Feb 23 00:31:01 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57099999999999995,"15min":0.56100000000000005},"min":{"1min":0.44700000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.752},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55600000000000005,"15min":0.55100000000000005},"min":{"1min":0.40200000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78900000000000003,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:31:29 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":0.80700000000000005,"15min":0.65700000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":11.904,"5min":11.904,"15min":11.904},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.734,"5min":0.81499999999999995,"15min":0.66500000000000004},"min":{"1min":0.38100000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":11.105,"5min":11.105,"15min":11.105},"last":3.8889999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:30:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.56999999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82999999999999996,"5min":1.228,"15min":1.228},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.503,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:31:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":0.72599999999999998,"15min":0.624},"min":{"1min":0.437,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":11.971,"5min":11.971,"15min":11.971},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.72599999999999998,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":11.135999999999999,"5min":11.135999999999999,"15min":11.135999999999999},"last":3.9209999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415416,"kb_used_data":101936,"kb_used_omap":0,"kb_used_meta":313472,"kb_avail":93329736,"statfs":{"total":95995035648,"available":95569649664,"internally_reserved":0,"allocated":104382464,"data_stored":101989958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":320995328},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:30:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.35899999999999999,"15min":0.33300000000000002},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.21299999999999999},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.371,"15min":0.33800000000000002},"min":{"1min":0.23400000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.247}]},{"osd":2,"last update":"Fri Feb 23 00:31:25 2024","interfaces":[{"interface":"back","average":{"1min":3.2650000000000001,"5min":0.95199999999999996,"15min":0.57999999999999996},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.253},"max":{"1min":23.809999999999999,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.6259999999999999,"5min":0.61699999999999999,"15min":0.44500000000000001},"min":{"1min":0.30099999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.965,"5min":15.965,"15min":15.965},"last":0.35399999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:31:42 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":1.014,"15min":0.64100000000000001},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":24.164999999999999,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":3.1509999999999998,"5min":0.96199999999999997,"15min":0.622},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":16.006,"5min":16.006,"15min":16.006},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:31:08 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.622,"15min":0.57999999999999996},"min":{"1min":0.39000000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":8.3599999999999994,"5min":8.3599999999999994,"15min":8.3599999999999994},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.232,"5min":0.68400000000000005,"15min":0.60399999999999998},"min":{"1min":0.45000000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":10.641,"5min":10.641,"15min":10.641},"last":0.41199999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:31:29 2024","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":1.012,"15min":0.70499999999999996},"min":{"1min":0.45400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":13.311,"5min":13.311,"15min":13.311},"last":0.435},{"interface":"front","average":{"1min":3.0569999999999999,"5min":1.0600000000000001,"15min":0.72499999999999998},"min":{"1min":0.38200000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":23.744,"5min":23.744,"15min":23.744},"last":0.47599999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:30:57 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":0.82599999999999996,"15min":0.62},"min":{"1min":0.36599999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":33.082999999999998,"5min":33.082999999999998,"15min":33.082999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.63100000000000001,"15min":0.58699999999999997},"min":{"1min":0.45400000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":7.1600000000000001,"5min":7.1600000000000001,"15min":7.1600000000000001},"last":0.40200000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.508,"5min":0.95499999999999996,"15min":0.69099999999999995},"min":{"1min":0.40500000000000003,"5min":0.373,"15min":0.373},"max":{"1min":31.251999999999999,"5min":31.251999999999999,"15min":31.251999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.63200000000000001,"15min":0.58399999999999996},"min":{"1min":0.41799999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":3.8450000000000002,"5min":3.8450000000000002,"15min":3.8450000000000002},"last":0.39500000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464308,"kb_used_data":150828,"kb_used_omap":0,"kb_used_meta":313472,"kb_avail":93280844,"statfs":{"total":95995035648,"available":95519584256,"internally_reserved":0,"allocated":154447872,"data_stored":152321261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":320995328},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:31:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0009999999999999,"5min":0.875,"15min":0.50900000000000001},"min":{"1min":0.23400000000000001,"5min":0.22,"15min":0.20999999999999999},"max":{"1min":30.890000000000001,"5min":30.890000000000001,"15min":30.890000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":4.6559999999999997,"5min":1.2170000000000001,"15min":0.621},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":31.498999999999999,"5min":31.498999999999999,"15min":31.498999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:31:19 2024","interfaces":[{"interface":"back","average":{"1min":2.3199999999999998,"5min":0.76000000000000001,"15min":0.47499999999999998},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":17.745999999999999,"5min":17.745999999999999,"15min":17.745999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":3.8900000000000001,"5min":1.0629999999999999,"15min":0.57599999999999996},"min":{"1min":0.255,"5min":0.23799999999999999,"15min":0.216},"max":{"1min":35.052,"5min":35.052,"15min":35.052},"last":0.443}]},{"osd":3,"last update":"Fri Feb 23 00:31:50 2024","interfaces":[{"interface":"back","average":{"1min":4.3220000000000001,"5min":1.2549999999999999,"15min":0.69699999999999995},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":43.5,"5min":43.5,"15min":43.5},"last":0.498},{"interface":"front","average":{"1min":2.5939999999999999,"5min":0.97399999999999998,"15min":0.61899999999999999},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":27.414000000000001,"5min":27.414000000000001,"15min":27.414000000000001},"last":0.439}]},{"osd":4,"last update":"Fri Feb 23 00:31:05 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.60199999999999998,"15min":0.56200000000000006},"min":{"1min":0.40000000000000002,"5min":0.308,"15min":0.308},"max":{"1min":3.4359999999999999,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.60699999999999998,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.319,"5min":3.319,"15min":3.319},"last":0.50900000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:31:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":0.89300000000000002,"15min":0.65900000000000003},"min":{"1min":0.34899999999999998,"5min":0.316,"15min":0.316},"max":{"1min":17.765000000000001,"5min":17.765000000000001,"15min":17.765000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.5249999999999999,"5min":0.78900000000000003,"15min":0.63100000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":14.589,"5min":14.589,"15min":14.589},"last":0.51900000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.67200000000000004,"15min":0.58999999999999997},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.6120000000000001,"5min":4.6120000000000001,"15min":4.6120000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":3.0030000000000001,"5min":1.054,"15min":0.748},"min":{"1min":0.37,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":20.574999999999999,"5min":20.574999999999999,"15min":20.574999999999999},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:31:15 2024","interfaces":[{"interface":"back","average":{"1min":6.3929999999999998,"5min":1.7410000000000001,"15min":0.97599999999999998},"min":{"1min":0.39500000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":66.632000000000005,"5min":66.632000000000005,"15min":66.632000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":3.1030000000000002,"5min":1.087,"15min":0.75900000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":19.603999999999999,"5min":19.603999999999999,"15min":19.603999999999999},"last":0.61899999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":148,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300252,"kb_used_data":3028,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444900,"statfs":{"total":95995035648,"available":95687577600,"internally_reserved":0,"allocated":3100672,"data_stored":1322477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:31:42 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":1.0529999999999999,"15min":0.59999999999999998},"min":{"1min":0.218,"5min":0.185,"15min":0.185},"max":{"1min":45.390999999999998,"5min":45.390999999999998,"15min":45.390999999999998},"last":0.29599999999999999},{"interface":"front","average":{"1min":3.7050000000000001,"5min":1.022,"15min":0.60999999999999999},"min":{"1min":0.251,"5min":0.191,"15min":0.191},"max":{"1min":30.186,"5min":30.186,"15min":30.186},"last":0.33000000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:31:42 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":0.629,"15min":0.501},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":18.039999999999999,"5min":18.039999999999999,"15min":18.039999999999999},"last":0.315},{"interface":"front","average":{"1min":3.5710000000000002,"5min":1.0369999999999999,"15min":0.622},"min":{"1min":0.29799999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":45.441000000000003,"5min":45.441000000000003,"15min":45.441000000000003},"last":0.26000000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:31:42 2024","interfaces":[{"interface":"back","average":{"1min":3.3450000000000002,"5min":0.98599999999999999,"15min":0.624},"min":{"1min":0.28100000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.40600000000000003},{"interface":"front","average":{"1min":3.0640000000000001,"5min":0.92200000000000004,"15min":0.59599999999999997},"min":{"1min":0.28699999999999998,"5min":0.185,"15min":0.185},"max":{"1min":24.183,"5min":24.183,"15min":24.183},"last":0.435}]},{"osd":4,"last update":"Fri Feb 23 00:31:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.33500000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.88700000000000001,"5min":0.61599999999999999,"15min":0.59799999999999998},"min":{"1min":0.32900000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":8.0359999999999996,"5min":8.0359999999999996,"15min":8.0359999999999996},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:31:31 2024","interfaces":[{"interface":"back","average":{"1min":3.0760000000000001,"5min":1.0720000000000001,"15min":0.73799999999999999},"min":{"1min":0.374,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":45.265000000000001,"5min":45.265000000000001,"15min":45.265000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":2.3879999999999999,"5min":0.93000000000000005,"15min":0.69999999999999996},"min":{"1min":0.40899999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":24.102,"5min":24.102,"15min":24.102},"last":0.51100000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:30:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.64400000000000002,"15min":0.59999999999999998},"min":{"1min":0.36299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":8.093,"5min":8.093,"15min":8.093},"last":0.56799999999999995},{"interface":"front","average":{"1min":2.7170000000000001,"5min":1.0229999999999999,"15min":0.747},"min":{"1min":0.39800000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":45.232999999999997,"5min":45.232999999999997,"15min":45.232999999999997},"last":0.5}]},{"osd":7,"last update":"Fri Feb 23 00:31:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":0.70999999999999996,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":8.1280000000000001,"5min":8.1280000000000001,"15min":8.1280000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":3.121,"5min":1.1100000000000001,"15min":0.77700000000000002},"min":{"1min":0.375,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":45.287999999999997,"5min":45.287999999999997,"15min":45.287999999999997},"last":0.52700000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986250,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353856,"kb_used_data":52536,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93391296,"statfs":{"total":95995035648,"available":95632687104,"internally_reserved":0,"allocated":53796864,"data_stored":51667824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:30:59 2024","interfaces":[{"interface":"back","average":{"1min":2.02,"5min":0.80200000000000005,"15min":0.61299999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":17.231999999999999,"5min":17.231999999999999,"15min":17.231999999999999},"last":0.38},{"interface":"front","average":{"1min":3.548,"5min":1.1279999999999999,"15min":0.72899999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":36.171999999999997,"5min":36.171999999999997,"15min":36.171999999999997},"last":0.36899999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:30:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8029999999999999,"5min":1.1459999999999999,"15min":0.73499999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":41.137999999999998,"5min":41.137999999999998,"15min":41.137999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.835,"5min":0.97099999999999997,"15min":0.67100000000000004},"min":{"1min":0.34300000000000003,"5min":0.309,"15min":0.309},"max":{"1min":26.210000000000001,"5min":26.210000000000001,"15min":26.210000000000001},"last":0.34999999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:30:59 2024","interfaces":[{"interface":"back","average":{"1min":5.0389999999999997,"5min":1.4139999999999999,"15min":0.83099999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":67.253,"5min":67.253,"15min":67.253},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.093,"5min":0.84499999999999997,"15min":0.63},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":25.864000000000001,"5min":25.864000000000001,"15min":25.864000000000001},"last":0.501}]},{"osd":3,"last update":"Fri Feb 23 00:30:59 2024","interfaces":[{"interface":"back","average":{"1min":3.5350000000000001,"5min":1.129,"15min":0.72699999999999998},"min":{"1min":0.35999999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":27.550000000000001,"5min":27.550000000000001,"15min":27.550000000000001},"last":1.7909999999999999},{"interface":"front","average":{"1min":2.7360000000000002,"5min":0.94499999999999995,"15min":0.67500000000000004},"min":{"1min":0.35299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":30.613,"5min":30.613,"15min":30.613},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:31:29 2024","interfaces":[{"interface":"back","average":{"1min":1.7589999999999999,"5min":0.68899999999999995,"15min":0.46999999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":14.242000000000001,"5min":14.242000000000001,"15min":14.242000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.466,"5min":0.66800000000000004,"15min":0.48399999999999999},"min":{"1min":0.28000000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":14.124000000000001,"5min":14.124000000000001,"15min":14.124000000000001},"last":0.40899999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.48799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":4.5860000000000003,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.92700000000000005,"15min":0.59499999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.8650000000000002,"5min":45.814999999999998,"15min":45.814999999999998},"last":0.40400000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:31:20 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":0.66500000000000004,"15min":0.53000000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":17.178999999999998,"5min":17.178999999999998,"15min":17.178999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.2,"5min":0.67300000000000004,"15min":0.56000000000000005},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":14.212,"5min":14.212,"15min":14.212},"last":0.47499999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":423720,"kb_used_data":126496,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93321432,"statfs":{"total":95995035648,"available":95561146368,"internally_reserved":0,"allocated":129531904,"data_stored":127169911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:31:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6789999999999998,"5min":1.127,"15min":0.71399999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":27.411000000000001,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":8.4670000000000005,"5min":2.1179999999999999,"15min":1.0309999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":103.57599999999999,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:31:28 2024","interfaces":[{"interface":"back","average":{"1min":4.5289999999999999,"5min":1.2929999999999999,"15min":0.749},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":38.838000000000001,"5min":38.838000000000001,"15min":38.838000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":10.669,"5min":2.5299999999999998,"15min":1.1739999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":127.794,"5min":127.794,"15min":127.794},"last":0.51000000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:31:28 2024","interfaces":[{"interface":"back","average":{"1min":6.2140000000000004,"5min":1.6559999999999999,"15min":0.88600000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":37.442999999999998,"5min":37.442999999999998,"15min":37.442999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":3.3199999999999998,"5min":1.073,"15min":0.69499999999999995},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":23.297999999999998,"5min":23.297999999999998,"15min":23.297999999999998},"last":0.63900000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:31:28 2024","interfaces":[{"interface":"back","average":{"1min":3.9540000000000002,"5min":1.226,"15min":0.72899999999999998},"min":{"1min":0.34999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":24.414999999999999,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":5.9050000000000002,"5min":1.6259999999999999,"15min":0.89100000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":40.865000000000002,"5min":40.865000000000002,"15min":40.865000000000002},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Feb 23 00:31:28 2024","interfaces":[{"interface":"back","average":{"1min":6.0869999999999997,"5min":1.5469999999999999,"15min":0.75800000000000001},"min":{"1min":0.23499999999999999,"5min":0.219,"15min":0.219},"max":{"1min":103.13,"5min":103.13,"15min":103.13},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.107,"5min":0.56699999999999995,"15min":0.42999999999999999},"min":{"1min":0.25700000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":12.795999999999999,"5min":12.795999999999999,"15min":12.795999999999999},"last":0.48499999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:30:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.55500000000000005,"15min":0.46400000000000002},"min":{"1min":0.32500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.1440000000000001,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":5.8879999999999999,"5min":1.5489999999999999,"15min":0.80300000000000005},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":103.61499999999999,"5min":103.61499999999999,"15min":103.61499999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:31:18 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.57499999999999996,"15min":0.51500000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.1820000000000004,"5min":7.1820000000000004,"15min":7.1820000000000004},"last":0.622},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.51500000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65800000000000003,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":905216,"data_stored":2296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1245184,"data_stored":3142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":806912,"data_stored":2038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":827392,"data_stored":2089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":913408,"data_stored":2302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":1097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":1075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":1129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":1157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":95,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:32:00.491 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set .rgw.root pgp_num 32 2024-02-23T00:32:00.508 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:00.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:00.612 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:32:00.641 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:00.740 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:32:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:00.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:32:00.841 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:32:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:32:00.844 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:00.872 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:01.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:31:59.574909+0000 mgr.y (mgr.24437) 221 : cluster [DBG] pgmap v235: 306 pgs: 40 unknown, 1 clean+premerge+peered, 265 active+clean; 15 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:32:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:00.043261+0000 mon.c (mon.2) 159 : audit [DBG] from='client.? 172.21.15.87:0/1305282909' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:00.264311+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-02-23T00:32:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:00.265623+0000 mon.c (mon.2) 160 : audit [INF] from='client.? 172.21.15.87:0/1861969848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi087-82011-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:00.266390+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi087-82011-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:01.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:00.403115+0000 mon.c (mon.2) 161 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:01.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:31:59.574909+0000 mgr.y (mgr.24437) 221 : cluster [DBG] pgmap v235: 306 pgs: 40 unknown, 1 clean+premerge+peered, 265 active+clean; 15 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:32:01.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:00.043261+0000 mon.c (mon.2) 159 : audit [DBG] from='client.? 172.21.15.87:0/1305282909' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:01.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:00.264311+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-02-23T00:32:01.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:00.265623+0000 mon.c (mon.2) 160 : audit [INF] from='client.? 172.21.15.87:0/1861969848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi087-82011-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:01.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:00.266390+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi087-82011-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:01.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:00.403115+0000 mon.c (mon.2) 161 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:01.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:31:59.574909+0000 mgr.y (mgr.24437) 221 : cluster [DBG] pgmap v235: 306 pgs: 40 unknown, 1 clean+premerge+peered, 265 active+clean; 15 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-23T00:32:01.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:00.043261+0000 mon.c (mon.2) 159 : audit [DBG] from='client.? 172.21.15.87:0/1305282909' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:01.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:00.264311+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-02-23T00:32:01.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:00.265623+0000 mon.c (mon.2) 160 : audit [INF] from='client.? 172.21.15.87:0/1861969848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi087-82011-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:01.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:00.266390+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi087-82011-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:01.165 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:00.403115+0000 mon.c (mon.2) 161 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:01.182 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-23T00:32:01.221 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:01.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:01.277 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:01.313 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:01.325 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:01.351 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:01.370 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:01.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:01.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:01.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:01.491 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:01.503 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:01.530 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:01.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:01.635 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:01.664 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:01.812 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:01.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:01.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:01.916 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:01.945 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:00.689914+0000 mgr.y (mgr.24437) 222 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:01.009444+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82083-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82083-24"}]': finished 2024-02-23T00:32:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:01.009675+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi087-82011-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:01.009788+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-02-23T00:32:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:01.019444+0000 mon.a (mon.0) 1318 : audit [INF] from='client.? 172.21.15.87:0/179348301' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi087-82019-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:01.404386+0000 mon.c (mon.2) 162 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:01.576808+0000 mon.a (mon.0) 1319 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-23T00:32:02.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:01.983790+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? 172.21.15.87:0/179348301' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi087-82019-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:01.983951+0000 mon.a (mon.0) 1321 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:01.984003+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:01.991996+0000 mon.c (mon.2) 163 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:01.992785+0000 mon.a (mon.0) 1323 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:00.689914+0000 mgr.y (mgr.24437) 222 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:01.009444+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82083-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82083-24"}]': finished 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:01.009675+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi087-82011-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:01.009788+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:01.019444+0000 mon.a (mon.0) 1318 : audit [INF] from='client.? 172.21.15.87:0/179348301' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi087-82019-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:01.404386+0000 mon.c (mon.2) 162 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:01.576808+0000 mon.a (mon.0) 1319 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-23T00:32:02.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:01.983790+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? 172.21.15.87:0/179348301' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi087-82019-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:02.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:01.983951+0000 mon.a (mon.0) 1321 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-23T00:32:02.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:01.984003+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-02-23T00:32:02.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:01.991996+0000 mon.c (mon.2) 163 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:02.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:01.992785+0000 mon.a (mon.0) 1323 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:02.367 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:00.689914+0000 mgr.y (mgr.24437) 222 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:02.367 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:01.009444+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82083-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82083-24"}]': finished 2024-02-23T00:32:02.367 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:01.009675+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi087-82011-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:02.367 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:01.009788+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-02-23T00:32:02.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:01.019444+0000 mon.a (mon.0) 1318 : audit [INF] from='client.? 172.21.15.87:0/179348301' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi087-82019-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:02.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:01.404386+0000 mon.c (mon.2) 162 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:02.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:01.576808+0000 mon.a (mon.0) 1319 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-23T00:32:02.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:01.983790+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? 172.21.15.87:0/179348301' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi087-82019-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:02.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:01.983951+0000 mon.a (mon.0) 1321 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-23T00:32:02.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:01.984003+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-02-23T00:32:02.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:01.991996+0000 mon.c (mon.2) 163 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:02.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:01.992785+0000 mon.a (mon.0) 1323 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:02.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:32:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:02] "GET /metrics HTTP/1.1" 200 191821 "" "Prometheus/2.33.4" 2024-02-23T00:32:02.795 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:02.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:02.900 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:03.209 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:32:03.289 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:32:03.290 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:32:03.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:01.575595+0000 mgr.y (mgr.24437) 223 : cluster [DBG] pgmap v238: 345 pgs: 40 unknown, 32 creating+peering, 1 peering, 272 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:03.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:02.006669+0000 mon.a (mon.0) 1324 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:03.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:02.405098+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:03.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:02.984863+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:03.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:02.985080+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-02-23T00:32:03.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:02.992119+0000 mon.c (mon.2) 165 : audit [INF] from='client.? 172.21.15.87:0/1431424293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi087-82011-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:02.995164+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi087-82011-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:02.997380+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:02.997853+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.480 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:03.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:01.575595+0000 mgr.y (mgr.24437) 223 : cluster [DBG] pgmap v238: 345 pgs: 40 unknown, 32 creating+peering, 1 peering, 272 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:02.006669+0000 mon.a (mon.0) 1324 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:02.405098+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:02.984863+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:02.985080+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:02.992119+0000 mon.c (mon.2) 165 : audit [INF] from='client.? 172.21.15.87:0/1431424293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi087-82011-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:02.995164+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi087-82011-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:02.997380+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:02.997853+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:01.575595+0000 mgr.y (mgr.24437) 223 : cluster [DBG] pgmap v238: 345 pgs: 40 unknown, 32 creating+peering, 1 peering, 272 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:02.006669+0000 mon.a (mon.0) 1324 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:02.405098+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:02.984863+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:02.985080+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-02-23T00:32:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:02.992119+0000 mon.c (mon.2) 165 : audit [INF] from='client.? 172.21.15.87:0/1431424293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi087-82011-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:02.995164+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi087-82011-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:02.997380+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:02.997853+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:03.584 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:03.611 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:03.924 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:32:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:32:03.944 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-23T00:32:04.233 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:04.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:03.405907+0000 mon.c (mon.2) 167 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:04.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:03.988816+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi087-82011-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:04.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:03.988907+0000 mon.a (mon.0) 1330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:04.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:03.989048+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-02-23T00:32:04.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:04.001025+0000 mon.c (mon.2) 168 : audit [INF] from='client.? 172.21.15.87:0/183937905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi087-82019-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:04.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:04.001686+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi087-82019-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:04.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:03.405907+0000 mon.c (mon.2) 167 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:04.457 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:03.988816+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi087-82011-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:04.457 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:03.988907+0000 mon.a (mon.0) 1330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:04.457 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:03.989048+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-02-23T00:32:04.457 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:04.001025+0000 mon.c (mon.2) 168 : audit [INF] from='client.? 172.21.15.87:0/183937905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi087-82019-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:04.457 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:04.001686+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi087-82019-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:04.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:03.405907+0000 mon.c (mon.2) 167 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:04.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:03.988816+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi087-82011-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:04.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:03.988907+0000 mon.a (mon.0) 1330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82083-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:04.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:03.989048+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-02-23T00:32:04.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:04.001025+0000 mon.c (mon.2) 168 : audit [INF] from='client.? 172.21.15.87:0/183937905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi087-82019-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:04.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:04.001686+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi087-82019-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:04.789 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 -- 172.21.15.87:0/4055360080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 msgr2=0x7f1b800f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 --2- 172.21.15.87:0/4055360080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 0x7f1b800f6d40 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7f1b7c004870 tx=0x7f1b7c009cb0).stop 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 -- 172.21.15.87:0/4055360080 shutdown_connections 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 --2- 172.21.15.87:0/4055360080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b800f7400 0x7f1b80103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 --2- 172.21.15.87:0/4055360080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b800fcbb0 0x7f1b800fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 --2- 172.21.15.87:0/4055360080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 0x7f1b800f6d40 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 -- 172.21.15.87:0/4055360080 >> 172.21.15.87:0/4055360080 conn(0x7f1b800f2820 msgr2=0x7f1b800f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 -- 172.21.15.87:0/4055360080 shutdown_connections 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 -- 172.21.15.87:0/4055360080 wait complete. 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 librados: starting msgr at 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 librados: starting objecter 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 Processor -- start 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 -- start start 2024-02-23T00:32:04.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 librados: setting wanted keys 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.676+0000 7f1b88611700 1 librados: calling monclient init 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b800f7400 0x7f1b801092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b800fcbb0 0x7f1b801097e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 0x7f1b80109da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b801144e0 con 0x7f1b800fcbb0 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80114650 con 0x7f1b800f7400 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80114810 con 0x7f1b800fd510 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b86bae700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 0x7f1b80109da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b85bac700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b800fcbb0 0x7f1b801097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b86bae700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 0x7f1b80109da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35804/0 (socket says 172.21.15.87:35804) 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b86bae700 1 -- 172.21.15.87:0/3384516841 learned_addr learned my addr 172.21.15.87:0/3384516841 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b863ad700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b800f7400 0x7f1b801092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b85bac700 1 -- 172.21.15.87:0/3384516841 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 msgr2=0x7f1b80109da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:04.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b85bac700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 0x7f1b80109da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b85bac700 1 -- 172.21.15.87:0/3384516841 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b800f7400 msgr2=0x7f1b801092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b85bac700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b800f7400 0x7f1b801092f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b85bac700 1 -- 172.21.15.87:0/3384516841 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b70005040 con 0x7f1b800fcbb0 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b86bae700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 0x7f1b80109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b863ad700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b800f7400 0x7f1b801092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b85bac700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b800fcbb0 0x7f1b801097e0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f1b7c004870 tx=0x7f1b7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b777fe700 10 client.?.objecter ms_handle_connect 0x7f1b800fcbb0 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b777fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b777fe700 1 -- 172.21.15.87:0/3384516841 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c00e070 con 0x7f1b800fcbb0 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7c005040 con 0x7f1b800fcbb0 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b777fe700 1 -- 172.21.15.87:0/3384516841 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b7c013da0 con 0x7f1b800fcbb0 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b777fe700 1 -- 172.21.15.87:0/3384516841 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c019070 con 0x7f1b800fcbb0 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 10 client.26451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b80114d00 con 0x7f1b800fcbb0 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.677+0000 7f1b88611700 1 librados: init done 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.678+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8005fa60 con 0x7f1b800fcbb0 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.679+0000 7f1b777fe700 1 -- 172.21.15.87:0/3384516841 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1b7c01e420 con 0x7f1b800fcbb0 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.680+0000 7f1b777fe700 1 --2- 172.21.15.87:0/3384516841 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b6c062c90 0x7f1b6c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.680+0000 7f1b777fe700 1 -- 172.21.15.87:0/3384516841 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(173..173 src has 38..173) v4 ==== 10915+0+0 (secure 0 0 0) 0x7f1b7c07e990 con 0x7f1b800fcbb0 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.680+0000 7f1b777fe700 10 client.26451.objecter ms_dispatch 0x7f1b80001550 osd_map(173..173 src has 38..173) v4 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.680+0000 7f1b777fe700 3 client.26451.objecter handle_osd_map got epochs [173,173] > 0 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.680+0000 7f1b777fe700 3 client.26451.objecter handle_osd_map decoding full epoch 173 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.680+0000 7f1b863ad700 1 --2- 172.21.15.87:0/3384516841 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b6c062c90 0x7f1b6c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.680+0000 7f1b777fe700 20 client.26451.objecter dump_active .. 0 homeless 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.680+0000 7f1b777fe700 10 client.26451.objecter ms_handle_connect 0x7f1b6c062c90 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.680+0000 7f1b863ad700 1 --2- 172.21.15.87:0/3384516841 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b6c062c90 0x7f1b6c065150 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 rx=0x7f1b70007fd0 tx=0x7f1b70009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:04.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.683+0000 7f1b777fe700 1 -- 172.21.15.87:0/3384516841 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1b7c0513b0 con 0x7f1b800fcbb0 2024-02-23T00:32:04.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.892+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"} v 0) v1 -- 0x7f1b8005ee80 con 0x7f1b800fcbb0 2024-02-23T00:32:04.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.792+0000 7f8e2bf64700 1 -- 172.21.15.87:0/3830279543 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e24102210 msgr2=0x7f8e24104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:04.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.792+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/3830279543 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e24102210 0x7f8e24104600 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7f8e1c009c30 tx=0x7f8e1c004c50).stop 2024-02-23T00:32:04.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.792+0000 7f8e2bf64700 1 -- 172.21.15.87:0/3830279543 shutdown_connections 2024-02-23T00:32:04.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.792+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/3830279543 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e24102210 0x7f8e24104600 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.792+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/3830279543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e240f8cb0 0x7f8e240f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.792+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/3830279543 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e240f9610 0x7f8e24101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.792+0000 7f8e2bf64700 1 -- 172.21.15.87:0/3830279543 >> 172.21.15.87:0/3830279543 conn(0x7f8e240f2820 msgr2=0x7f8e240f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:04.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 -- 172.21.15.87:0/3830279543 shutdown_connections 2024-02-23T00:32:04.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 -- 172.21.15.87:0/3830279543 wait complete. 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 librados: starting msgr at 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 librados: starting objecter 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 Processor -- start 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 -- start start 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 librados: setting wanted keys 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 librados: calling monclient init 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e240f8cb0 0x7f8e240f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e240f9610 0x7f8e240f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e24102210 0x7f8e240f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e241165a0 con 0x7f8e24102210 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e24116710 con 0x7f8e240f9610 2024-02-23T00:32:04.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e2bf64700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e241168d0 con 0x7f8e240f8cb0 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.793+0000 7f8e29d00700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e240f8cb0 0x7f8e240f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e29d00700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e240f8cb0 0x7f8e240f6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35822/0 (socket says 172.21.15.87:35822) 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e29d00700 1 -- 172.21.15.87:0/350336925 learned_addr learned my addr 172.21.15.87:0/350336925 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2a501700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e24102210 0x7f8e240f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e294ff700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e240f9610 0x7f8e240f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2a501700 1 -- 172.21.15.87:0/350336925 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e240f8cb0 msgr2=0x7f8e240f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2a501700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e240f8cb0 0x7f8e240f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2a501700 1 -- 172.21.15.87:0/350336925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e240f9610 msgr2=0x7f8e240f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2a501700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e240f9610 0x7f8e240f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2a501700 1 -- 172.21.15.87:0/350336925 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e14005040 con 0x7f8e24102210 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e29d00700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e240f8cb0 0x7f8e240f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:04.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2a501700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e24102210 0x7f8e240f7660 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7f8e1c004e10 tx=0x7f8e1c004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e1affd700 10 client.?.objecter ms_handle_connect 0x7f8e24102210 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e1affd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e1affd700 1 -- 172.21.15.87:0/350336925 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c00e040 con 0x7f8e24102210 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e1affd700 1 -- 172.21.15.87:0/350336925 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e1c018040 con 0x7f8e24102210 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c0097c0 con 0x7f8e24102210 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e1affd700 1 -- 172.21.15.87:0/350336925 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c01c3f0 con 0x7f8e24102210 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2bf64700 10 client.26457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e24116dc0 con 0x7f8e24102210 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.794+0000 7f8e2bf64700 1 librados: init done 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.795+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2405fa60 con 0x7f8e24102210 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.795+0000 7f8e1affd700 1 -- 172.21.15.87:0/350336925 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f8e1c013600 con 0x7f8e24102210 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.796+0000 7f8e1affd700 1 --2- 172.21.15.87:0/350336925 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e10062c90 0x7f8e10065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.796+0000 7f8e1affd700 1 -- 172.21.15.87:0/350336925 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(173..173 src has 38..173) v4 ==== 10915+0+0 (secure 0 0 0) 0x7f8e1c07ecf0 con 0x7f8e24102210 2024-02-23T00:32:04.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.796+0000 7f8e1affd700 10 client.26457.objecter ms_dispatch 0x7f8e24001550 osd_map(173..173 src has 38..173) v4 2024-02-23T00:32:04.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.796+0000 7f8e29d00700 1 --2- 172.21.15.87:0/350336925 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e10062c90 0x7f8e10065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:04.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.796+0000 7f8e1affd700 3 client.26457.objecter handle_osd_map got epochs [173,173] > 0 2024-02-23T00:32:04.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.796+0000 7f8e1affd700 3 client.26457.objecter handle_osd_map decoding full epoch 173 2024-02-23T00:32:04.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.796+0000 7f8e1affd700 20 client.26457.objecter dump_active .. 0 homeless 2024-02-23T00:32:04.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.796+0000 7f8e1affd700 10 client.26457.objecter ms_handle_connect 0x7f8e10062c90 2024-02-23T00:32:04.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.796+0000 7f8e29d00700 1 --2- 172.21.15.87:0/350336925 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e10062c90 0x7f8e10065150 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 rx=0x7f8e20004fd0 tx=0x7f8e2000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:04.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.800+0000 7f8e1affd700 1 -- 172.21.15.87:0/350336925 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8e1c051690 con 0x7f8e24102210 2024-02-23T00:32:04.926 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:04.961 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:04.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:05.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:04.997+0000 7f1b777fe700 1 -- 172.21.15.87:0/3384516841 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]=0 set pool 2 pgp_num to 32 v174) v1 ==== 136+0+0 (secure 0 0 0) 0x7f1b7c023070 con 0x7f1b800fcbb0 2024-02-23T00:32:05.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.000+0000 7f1b88611700 10 librados: watch_flush enter 2024-02-23T00:32:05.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.000+0000 7f1b88611700 10 librados: watch_flush exit 2024-02-23T00:32:05.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.000+0000 7f1b88611700 20 client.26451.objecter shutdown clearing up homeless session... 2024-02-23T00:32:05.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.000+0000 7f1b88611700 10 client.26451.objecter successfully canceled tick 2024-02-23T00:32:05.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.003+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b6c062c90 msgr2=0x7f1b6c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:05.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.003+0000 7f1b88611700 1 --2- 172.21.15.87:0/3384516841 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b6c062c90 0x7f1b6c065150 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 rx=0x7f1b70007fd0 tx=0x7f1b70009cb0).stop 2024-02-23T00:32:05.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.003+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b800fcbb0 msgr2=0x7f1b801097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:05.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.003+0000 7f1b88611700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b800fcbb0 0x7f1b801097e0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 rx=0x7f1b7c004870 tx=0x7f1b7c00b040).stop 2024-02-23T00:32:05.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.012+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 shutdown_connections 2024-02-23T00:32:05.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.012+0000 7f1b88611700 1 --2- 172.21.15.87:0/3384516841 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b6c062c90 0x7f1b6c065150 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.012+0000 7f1b88611700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1b800fd510 0x7f1b80109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.012+0000 7f1b88611700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1b800f7400 0x7f1b801092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.012+0000 7f1b88611700 1 --2- 172.21.15.87:0/3384516841 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1b800fcbb0 0x7f1b801097e0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.012+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 >> 172.21.15.87:0/3384516841 conn(0x7f1b800f2820 msgr2=0x7f1b800f4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:05.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.028+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 shutdown_connections 2024-02-23T00:32:05.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.035+0000 7f1b88611700 1 -- 172.21.15.87:0/3384516841 wait complete. 2024-02-23T00:32:05.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.036+0000 7f1b88611700 1 librados: shutdown 2024-02-23T00:32:05.039 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 2 pgp_num to 32 2024-02-23T00:32:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.046+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f8e24107290 con 0x7f8e24102210 2024-02-23T00:32:05.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:05.095 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:03.576043+0000 mgr.y (mgr.24437) 224 : cluster [DBG] pgmap v241: 345 pgs: 72 unknown, 1 peering, 272 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-23T00:32:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:04.406660+0000 mon.c (mon.2) 169 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:04.894524+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? 172.21.15.87:0/3384516841' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:32:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:04.991128+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi087-82019-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:04.991312+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? 172.21.15.87:0/3384516841' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:32:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:04.991366+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-02-23T00:32:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.042538+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.042846+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.043922+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.044167+0000 mon.a (mon.0) 1338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.044767+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi087-82011-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.045056+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi087-82011-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.048548+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? 172.21.15.87:0/350336925' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:32:05.544 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:05.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:03.576043+0000 mgr.y (mgr.24437) 224 : cluster [DBG] pgmap v241: 345 pgs: 72 unknown, 1 peering, 272 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-23T00:32:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:04.406660+0000 mon.c (mon.2) 169 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:04.894524+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? 172.21.15.87:0/3384516841' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:32:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:04.991128+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi087-82019-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:04.991312+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? 172.21.15.87:0/3384516841' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:32:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:04.991366+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-02-23T00:32:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.042538+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.042846+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.043922+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.044167+0000 mon.a (mon.0) 1338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.044767+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi087-82011-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:05.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.045056+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi087-82011-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:05.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.048548+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? 172.21.15.87:0/350336925' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:32:05.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:03.576043+0000 mgr.y (mgr.24437) 224 : cluster [DBG] pgmap v241: 345 pgs: 72 unknown, 1 peering, 272 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-23T00:32:05.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:04.406660+0000 mon.c (mon.2) 169 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:05.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:04.894524+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? 172.21.15.87:0/3384516841' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:32:05.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:04.991128+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi087-82019-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:05.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:04.991312+0000 mon.a (mon.0) 1335 : audit [INF] from='client.? 172.21.15.87:0/3384516841' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:32:05.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:04.991366+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-02-23T00:32:05.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.042538+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.042846+0000 mon.a (mon.0) 1337 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.043922+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.044167+0000 mon.a (mon.0) 1338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:05.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.044767+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi087-82011-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:05.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.045056+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi087-82011-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:05.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.048548+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? 172.21.15.87:0/350336925' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:32:05.648 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:32:05.676 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:05.774 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:32:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:05.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:32:05.878 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:05.908 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:05.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:05.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.992+0000 7f8e1affd700 1 -- 172.21.15.87:0/350336925 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v175) v1 ==== 91+0+0 (secure 0 0 0) 0x7f8e1c001250 con 0x7f8e24102210 2024-02-23T00:32:05.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.994+0000 7f8e2bf64700 10 librados: watch_flush enter 2024-02-23T00:32:05.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.994+0000 7f8e2bf64700 10 librados: watch_flush exit 2024-02-23T00:32:05.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.994+0000 7f8e2bf64700 20 client.26457.objecter shutdown clearing up homeless session... 2024-02-23T00:32:05.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.994+0000 7f8e2bf64700 10 client.26457.objecter successfully canceled tick 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e10062c90 msgr2=0x7f8e10065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/350336925 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e10062c90 0x7f8e10065150 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 rx=0x7f8e20004fd0 tx=0x7f8e2000b040).stop 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e24102210 msgr2=0x7f8e240f7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e24102210 0x7f8e240f7660 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 rx=0x7f8e1c004e10 tx=0x7f8e1c004c50).stop 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 shutdown_connections 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/350336925 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e10062c90 0x7f8e10065150 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8e24102210 0x7f8e240f7660 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8e240f8cb0 0x7f8e240f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 --2- 172.21.15.87:0/350336925 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8e240f9610 0x7f8e240f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 >> 172.21.15.87:0/350336925 conn(0x7f8e240f2820 msgr2=0x7f8e240f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:05.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 shutdown_connections 2024-02-23T00:32:05.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 -- 172.21.15.87:0/350336925 wait complete. 2024-02-23T00:32:05.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:05.995+0000 7f8e2bf64700 1 librados: shutdown 2024-02-23T00:32:05.999 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:32:06.031 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:32:06.031 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:32:06.064 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:06.094 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:06.354 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.407473+0000 mon.c (mon.2) 173 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:06.354 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.578169+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-23T00:32:06.354 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:05.988001+0000 mon.a (mon.0) 1342 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:06.354 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.992655+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi087-82011-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:06.354 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.992739+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.87:0/350336925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:32:06.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.992796+0000 mon.a (mon.0) 1345 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-23T00:32:06.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:05.992842+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-02-23T00:32:06.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.998599+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:06.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:05.998940+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:06.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:06.000852+0000 mon.c (mon.2) 175 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi087-82011-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:06.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:06.001444+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi087-82011-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.407473+0000 mon.c (mon.2) 173 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.578169+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-23T00:32:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:05.988001+0000 mon.a (mon.0) 1342 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.992655+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi087-82011-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.992739+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.87:0/350336925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:32:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.992796+0000 mon.a (mon.0) 1345 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-23T00:32:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:05.992842+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-02-23T00:32:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.998599+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:05.998940+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:06.000852+0000 mon.c (mon.2) 175 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi087-82011-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:06.001444+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi087-82011-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.407473+0000 mon.c (mon.2) 173 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.578169+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:05.988001+0000 mon.a (mon.0) 1342 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.992655+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi087-82011-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.992739+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.87:0/350336925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.992796+0000 mon.a (mon.0) 1345 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:05.992842+0000 mon.a (mon.0) 1346 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-02-23T00:32:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.998599+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:05.998940+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:06.000852+0000 mon.c (mon.2) 175 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi087-82011-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:06.001444+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi087-82011-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:06.702 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-23T00:32:06.745 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:06.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:06.848 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:06.875 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:06.894 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:06.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:06.998 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:07.023 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:07.026 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:07.228 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:07.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:07.334 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:07.362 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:07.382 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:07.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:07.486 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:07.510 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:05.577144+0000 mgr.y (mgr.24437) 225 : cluster [DBG] pgmap v244: 312 pgs: 1 active+clean+snaptrim, 311 active+clean; 16 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:06.367032+0000 osd.3 (osd.3) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:06.367301+0000 osd.3 (osd.3) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:06.408385+0000 mon.c (mon.2) 176 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:06.558481+0000 osd.0 (osd.0) 105 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:06.561370+0000 osd.0 (osd.0) 106 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:06.985071+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]': finished 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:06.992021+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:06.992862+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:07.227537+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.228700+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.229272+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:07.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.230279+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.87:0/359111152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi087-82019-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.231072+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi087-82019-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:05.577144+0000 mgr.y (mgr.24437) 225 : cluster [DBG] pgmap v244: 312 pgs: 1 active+clean+snaptrim, 311 active+clean; 16 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:06.367032+0000 osd.3 (osd.3) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:06.367301+0000 osd.3 (osd.3) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:06.408385+0000 mon.c (mon.2) 176 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:06.558481+0000 osd.0 (osd.0) 105 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:06.561370+0000 osd.0 (osd.0) 106 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:06.985071+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]': finished 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:06.992021+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:06.992862+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:07.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:07.227537+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-02-23T00:32:07.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.228700+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:07.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.229272+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:07.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.230279+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.87:0/359111152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi087-82019-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:07.513 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.231072+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi087-82019-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:07.513 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:07.525 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:05.577144+0000 mgr.y (mgr.24437) 225 : cluster [DBG] pgmap v244: 312 pgs: 1 active+clean+snaptrim, 311 active+clean; 16 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-23T00:32:07.525 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:06.367032+0000 osd.3 (osd.3) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:07.525 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:06.367301+0000 osd.3 (osd.3) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:07.525 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:06.408385+0000 mon.c (mon.2) 176 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:07.525 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:06.558481+0000 osd.0 (osd.0) 105 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:32:07.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:06.561370+0000 osd.0 (osd.0) 106 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:32:07.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:06.985071+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82083-24"}]': finished 2024-02-23T00:32:07.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:06.992021+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.87:0/2980124580' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:07.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:06.992862+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:07.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:07.227537+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-02-23T00:32:07.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.228700+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]: dispatch 2024-02-23T00:32:07.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.229272+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:07.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.230279+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.87:0/359111152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi087-82019-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:07.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.231072+0000 mon.a (mon.0) 1353 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi087-82019-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:07.274455+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:32:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:07.275811+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:32:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.409203+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:07.572730+0000 osd.0 (osd.0) 107 : cluster [DBG] 6.16 scrub starts 2024-02-23T00:32:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:07.573551+0000 osd.0 (osd.0) 108 : cluster [DBG] 6.16 scrub ok 2024-02-23T00:32:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.989671+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi087-82011-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi087-82011-26"}]': finished 2024-02-23T00:32:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.989805+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]': finished 2024-02-23T00:32:08.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.989880+0000 mon.a (mon.0) 1356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]': finished 2024-02-23T00:32:08.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.990009+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi087-82019-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:08.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:07.990093+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-02-23T00:32:08.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:07.994978+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:08.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:08.012306+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:07.274455+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:32:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:07.275811+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:32:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.409203+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:07.572730+0000 osd.0 (osd.0) 107 : cluster [DBG] 6.16 scrub starts 2024-02-23T00:32:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:07.573551+0000 osd.0 (osd.0) 108 : cluster [DBG] 6.16 scrub ok 2024-02-23T00:32:08.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.989671+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi087-82011-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi087-82011-26"}]': finished 2024-02-23T00:32:08.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.989805+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]': finished 2024-02-23T00:32:08.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.989880+0000 mon.a (mon.0) 1356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]': finished 2024-02-23T00:32:08.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.990009+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi087-82019-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:08.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:07.990093+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-02-23T00:32:08.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:07.994978+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:08.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:08.012306+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:08.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:07.274455+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:07.275811+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.409203+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:07.572730+0000 osd.0 (osd.0) 107 : cluster [DBG] 6.16 scrub starts 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:07.573551+0000 osd.0 (osd.0) 108 : cluster [DBG] 6.16 scrub ok 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.989671+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi087-82011-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi087-82011-26"}]': finished 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.989805+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82083-24"}]': finished 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.989880+0000 mon.a (mon.0) 1356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi087-82115-15"}]': finished 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.990009+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi087-82019-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:07.990093+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:07.994978+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.87:0/3011193832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:08.621 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:08.012306+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]: dispatch 2024-02-23T00:32:08.899 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:32:08.899 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:32:09.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 22 tests from LibRadosMisc api_snapshots_pp: Running main() from gmock_main.cc 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [==========] Running 20 tests from 5 test suites. 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: seed 82115 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (3073 ms) 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (4519 ms) 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2292 ms) 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-02-23T00:32:09.012 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2234 ms) 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4444 ms) 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (16562 ms total) 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4872 ms) 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4196 ms) 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5797 ms) 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3974 ms) 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-02-23T00:32:09.013 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2086 ms) 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi087-82115-7 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: waiting for snaps to purge 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17884 ms) 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (38809 ms total) 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (7 ms) 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5018 ms) 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5025 ms total) 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-02-23T00:32:09.014 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3012 ms) 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (1965 ms) 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2003 ms) 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2007 ms) 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8987 ms total) 2024-02-23T00:32:09.015 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-02-23T00:32:09.238 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:32:09.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:07.577590+0000 mgr.y (mgr.24437) 226 : cluster [DBG] pgmap v247: 296 pgs: 32 unknown, 264 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:08.282891+0000 mon.a (mon.0) 1360 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:08.410173+0000 mon.c (mon.2) 182 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:08.593843+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:08.596730+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:08.990273+0000 mon.a (mon.0) 1361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]': finished 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:08.990398+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.000375+0000 mon.a (mon.0) 1363 : audit [INF] from='client.? 172.21.15.87:0/2571447995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi087-82083-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.027652+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.028433+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.028828+0000 mon.a (mon.0) 1366 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:09.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:07.577590+0000 mgr.y (mgr.24437) 226 : cluster [DBG] pgmap v247: 296 pgs: 32 unknown, 264 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:08.282891+0000 mon.a (mon.0) 1360 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:08.410173+0000 mon.c (mon.2) 182 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:08.593843+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:08.596730+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:08.990273+0000 mon.a (mon.0) 1361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]': finished 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:08.990398+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.000375+0000 mon.a (mon.0) 1363 : audit [INF] from='client.? 172.21.15.87:0/2571447995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi087-82083-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.027652+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.028433+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.028828+0000 mon.a (mon.0) 1366 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:09.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.439+0000 7f77a592d700 1 -- 172.21.15.87:0/306537379 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00f7400 msgr2=0x7f77a0103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.439+0000 7f77a592d700 1 --2- 172.21.15.87:0/306537379 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00f7400 0x7f77a0103a20 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7f7798009be0 tx=0x7f7798004d10).stop 2024-02-23T00:32:09.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.439+0000 7f77a592d700 1 -- 172.21.15.87:0/306537379 shutdown_connections 2024-02-23T00:32:09.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.439+0000 7f77a592d700 1 --2- 172.21.15.87:0/306537379 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00f7400 0x7f77a0103a20 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.439+0000 7f77a592d700 1 --2- 172.21.15.87:0/306537379 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77a00fcbb0 0x7f77a00fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.439+0000 7f77a592d700 1 --2- 172.21.15.87:0/306537379 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77a00fd510 0x7f77a00f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.439+0000 7f77a592d700 1 -- 172.21.15.87:0/306537379 >> 172.21.15.87:0/306537379 conn(0x7f77a00f2820 msgr2=0x7f77a00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:09.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.439+0000 7f77a592d700 1 -- 172.21.15.87:0/306537379 shutdown_connections 2024-02-23T00:32:09.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.439+0000 7f77a592d700 1 -- 172.21.15.87:0/306537379 wait complete. 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.440+0000 7f77a592d700 1 librados: starting msgr at 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.440+0000 7f77a592d700 1 librados: starting objecter 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.440+0000 7f77a592d700 1 Processor -- start 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.440+0000 7f77a592d700 1 -- start start 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.440+0000 7f77a592d700 1 librados: setting wanted keys 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.440+0000 7f77a592d700 1 librados: calling monclient init 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f77a592d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77a00f7400 0x7f77a0109210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f77a592d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00fcbb0 0x7f77a0109700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f77a592d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77a00fd510 0x7f77a0109ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f77a592d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a01143f0 con 0x7f77a00f7400 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f77a592d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a0114560 con 0x7f77a00fd510 2024-02-23T00:32:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f77a592d700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a0114720 con 0x7f77a00fcbb0 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779e7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00fcbb0 0x7f77a0109700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779e7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00fcbb0 0x7f77a0109700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35930/0 (socket says 172.21.15.87:35930) 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779e7fc700 1 -- 172.21.15.87:0/2675956627 learned_addr learned my addr 172.21.15.87:0/2675956627 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77a00f7400 0x7f77a0109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779f7fe700 1 --2- 172.21.15.87:0/2675956627 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77a00fd510 0x7f77a0109ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779e7fc700 1 -- 172.21.15.87:0/2675956627 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77a00fd510 msgr2=0x7f77a0109ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779e7fc700 1 --2- 172.21.15.87:0/2675956627 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77a00fd510 0x7f77a0109ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779e7fc700 1 -- 172.21.15.87:0/2675956627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77a00f7400 msgr2=0x7f77a0109210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779e7fc700 1 --2- 172.21.15.87:0/2675956627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77a00f7400 0x7f77a0109210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779e7fc700 1 -- 172.21.15.87:0/2675956627 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7794005040 con 0x7f77a00fcbb0 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77a00f7400 0x7f77a0109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f779e7fc700 1 --2- 172.21.15.87:0/2675956627 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00fcbb0 0x7f77a0109700 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 rx=0x7f77940048f0 tx=0x7f7794009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f7787fff700 10 client.?.objecter ms_handle_connect 0x7f77a00fcbb0 2024-02-23T00:32:09.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f7787fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:09.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794007b00 con 0x7f77a00fcbb0 2024-02-23T00:32:09.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.441+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77980097c0 con 0x7f77a00fcbb0 2024-02-23T00:32:09.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.442+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f779400f040 con 0x7f77a00fcbb0 2024-02-23T00:32:09.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.442+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794014a00 con 0x7f77a00fcbb0 2024-02-23T00:32:09.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.442+0000 7f77a592d700 10 client.36941.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:09.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.442+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a0114db0 con 0x7f77a00fcbb0 2024-02-23T00:32:09.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.442+0000 7f77a592d700 1 librados: init done 2024-02-23T00:32:09.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.442+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f77a005fa60 con 0x7f77a00fcbb0 2024-02-23T00:32:09.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.443+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f779400e580 con 0x7f77a00fcbb0 2024-02-23T00:32:09.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.443+0000 7f7787fff700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7788062c90 0x7f7788065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.443+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(178..178 src has 38..178) v4 ==== 9241+0+0 (secure 0 0 0) 0x7f779407d1c0 con 0x7f77a00fcbb0 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.443+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 osd_map(178..178 src has 38..178) v4 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.443+0000 7f7787fff700 3 client.36941.objecter handle_osd_map got epochs [178,178] > 0 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.443+0000 7f7787fff700 3 client.36941.objecter handle_osd_map decoding full epoch 178 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.443+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7788062c90 0x7f7788065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.443+0000 7f7787fff700 20 client.36941.objecter dump_active .. 0 homeless 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f7787fff700 10 client.36941.objecter ms_handle_connect 0x7f7788062c90 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7788062c90 0x7f7788065150 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7f7790011510 tx=0x7f779000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v178) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7794084070 con 0x7f77a00fcbb0 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 10 client.36941.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 1 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6801/2153191863 conn(0x7f77a00fdea0 0x7f77a00fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a005ee80 osd=0 3 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a005ee80 osd=0 3 2024-02-23T00:32:09.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a005ee80 osd=0 3 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 1 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a005ee80 osd=0 3 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 0 1 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a005ee80 osd=0 4 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 10 client.36941.objecter _send_command 1 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77a01061c0 con 0x7f77a00fdea0 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.444+0000 7f779f7fe700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6801/2153191863 conn(0x7f77a00fdea0 0x7f77a00fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.445+0000 7f779f7fe700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6801/2153191863 conn(0x7f77a00fdea0 0x7f77a00fe2c0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.445+0000 7f7787fff700 10 client.36941.objecter ms_handle_connect 0x7f77a00fdea0 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.445+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77a01061c0 con 0x7f77a00fdea0 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.445+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.445+0000 7f7787fff700 10 client.36941.objecter _finish_command 1 = system:0 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.445+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a005ee80 osd=0 4 2024-02-23T00:32:09.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.445+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 0 1 2024-02-23T00:32:09.463 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:09.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 10 client.36941.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:09.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 2 2024-02-23T00:32:09.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a005ee80 osd=0 3 2024-02-23T00:32:09.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a005ee80 osd=0 3 2024-02-23T00:32:09.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a005ee80 osd=0 3 2024-02-23T00:32:09.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 2 2024-02-23T00:32:09.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a005ee80 osd=0 3 2024-02-23T00:32:09.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 0 2 2024-02-23T00:32:09.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a005ee80 osd=0 4 2024-02-23T00:32:09.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 10 client.36941.objecter _send_command 2 2024-02-23T00:32:09.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.463+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77a0116fe0 con 0x7f77a00fdea0 2024-02-23T00:32:09.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f77a0116fe0 con 0x7f77a00fdea0 2024-02-23T00:32:09.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:09.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f7787fff700 10 client.36941.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a005ee80 osd=0 4 2024-02-23T00:32:09.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 0 2 2024-02-23T00:32:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:09.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:09.467 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:32:09.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 10 client.36941.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:09.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 3 2024-02-23T00:32:09.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6805/685354171 conn(0x7f77a00f4be0 0x7f77a00273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a00f4a70 osd=1 3 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00f4a70 osd=1 3 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a00f4a70 osd=1 3 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 3 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a00f4a70 osd=1 3 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 1 3 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00f4a70 osd=1 4 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 10 client.36941.objecter _send_command 3 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77a0109010 con 0x7f77a00f4be0 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.465+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6805/685354171 conn(0x7f77a00f4be0 0x7f77a00273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.466+0000 7f7787fff700 10 client.36941.objecter ms_handle_connect 0x7f77a00f4be0 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.466+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6805/685354171 conn(0x7f77a00f4be0 0x7f77a00273c0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.466+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77a0109010 con 0x7f77a00f4be0 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.466+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.466+0000 7f7787fff700 10 client.36941.objecter _finish_command 3 = system:0 2024-02-23T00:32:09.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.466+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a00f4a70 osd=1 4 2024-02-23T00:32:09.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.466+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 1 3 2024-02-23T00:32:09.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 10 client.36941.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:09.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 4 2024-02-23T00:32:09.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a00f4a70 osd=1 3 2024-02-23T00:32:09.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00f4a70 osd=1 3 2024-02-23T00:32:09.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a00f4a70 osd=1 3 2024-02-23T00:32:09.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 4 2024-02-23T00:32:09.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a00f4a70 osd=1 3 2024-02-23T00:32:09.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 1 4 2024-02-23T00:32:09.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00f4a70 osd=1 4 2024-02-23T00:32:09.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 10 client.36941.objecter _send_command 4 2024-02-23T00:32:09.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.483+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77a00278b0 con 0x7f77a00f4be0 2024-02-23T00:32:09.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f77a00278b0 con 0x7f77a00f4be0 2024-02-23T00:32:09.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:09.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f7787fff700 10 client.36941.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a00f4a70 osd=1 4 2024-02-23T00:32:09.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 1 4 2024-02-23T00:32:09.486 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 10 client.36941.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 5 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6809/3605394228 conn(0x7f77a0117500 0x7f77a011e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0117390 osd=2 3 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0117390 osd=2 3 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0117390 osd=2 3 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 5 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a0117390 osd=2 3 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 2 5 2024-02-23T00:32:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0117390 osd=2 4 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 10 client.36941.objecter _send_command 5 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77a0028760 con 0x7f77a0117500 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f779e7fc700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6809/3605394228 conn(0x7f77a0117500 0x7f77a011e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f7787fff700 10 client.36941.objecter ms_handle_connect 0x7f77a0117500 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.485+0000 7f779e7fc700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6809/3605394228 conn(0x7f77a0117500 0x7f77a011e580 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.486+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77a0028760 con 0x7f77a0117500 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.486+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.486+0000 7f7787fff700 10 client.36941.objecter _finish_command 5 = system:0 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.486+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a0117390 osd=2 4 2024-02-23T00:32:09.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.486+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 2 5 2024-02-23T00:32:09.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 10 client.36941.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:09.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 6 2024-02-23T00:32:09.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0117390 osd=2 3 2024-02-23T00:32:09.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0117390 osd=2 3 2024-02-23T00:32:09.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0117390 osd=2 3 2024-02-23T00:32:09.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 6 2024-02-23T00:32:09.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a0117390 osd=2 3 2024-02-23T00:32:09.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 2 6 2024-02-23T00:32:09.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0117390 osd=2 4 2024-02-23T00:32:09.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 10 client.36941.objecter _send_command 6 2024-02-23T00:32:09.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.501+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77a0027cc0 con 0x7f77a0117500 2024-02-23T00:32:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.503+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f77a0027cc0 con 0x7f77a0117500 2024-02-23T00:32:09.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.503+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.503+0000 7f7787fff700 10 client.36941.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.503+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a0117390 osd=2 4 2024-02-23T00:32:09.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.503+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 2 6 2024-02-23T00:32:09.505 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 10 client.36941.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 7 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6813/128059790 conn(0x7f77a0059e00 0x7f77a0121c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a00280d0 osd=3 3 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00280d0 osd=3 3 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a00280d0 osd=3 3 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 7 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a00280d0 osd=3 3 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 3 7 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00280d0 osd=3 4 2024-02-23T00:32:09.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 10 client.36941.objecter _send_command 7 2024-02-23T00:32:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77a0124920 con 0x7f77a0059e00 2024-02-23T00:32:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f779f7fe700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6813/128059790 conn(0x7f77a0059e00 0x7f77a0121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f7787fff700 10 client.36941.objecter ms_handle_connect 0x7f77a0059e00 2024-02-23T00:32:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.504+0000 7f779f7fe700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6813/128059790 conn(0x7f77a0059e00 0x7f77a0121c00 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.505+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77a0124920 con 0x7f77a0059e00 2024-02-23T00:32:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.505+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:32:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.505+0000 7f7787fff700 10 client.36941.objecter _finish_command 7 = system:0 2024-02-23T00:32:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.505+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a00280d0 osd=3 4 2024-02-23T00:32:09.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.505+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 3 7 2024-02-23T00:32:09.516 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 10 client.36941.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 8 2024-02-23T00:32:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a00280d0 osd=3 3 2024-02-23T00:32:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00280d0 osd=3 3 2024-02-23T00:32:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a00280d0 osd=3 3 2024-02-23T00:32:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 8 2024-02-23T00:32:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a00280d0 osd=3 3 2024-02-23T00:32:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 3 8 2024-02-23T00:32:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00280d0 osd=3 4 2024-02-23T00:32:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 10 client.36941.objecter _send_command 8 2024-02-23T00:32:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.521+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77a0059360 con 0x7f77a0059e00 2024-02-23T00:32:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.522+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f77a0059360 con 0x7f77a0059e00 2024-02-23T00:32:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.522+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.522+0000 7f7787fff700 10 client.36941.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.522+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a00280d0 osd=3 4 2024-02-23T00:32:09.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.522+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 3 8 2024-02-23T00:32:09.524 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 10 client.36941.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 9 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6800/4175844757 conn(0x7f77a011f750 0x7f77a0123310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0059530 osd=4 3 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0059530 osd=4 3 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0059530 osd=4 3 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 9 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a0059530 osd=4 3 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 4 9 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0059530 osd=4 4 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 10 client.36941.objecter _send_command 9 2024-02-23T00:32:09.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77a00596a0 con 0x7f77a011f750 2024-02-23T00:32:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6800/4175844757 conn(0x7f77a011f750 0x7f77a0123310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f7787fff700 10 client.36941.objecter ms_handle_connect 0x7f77a011f750 2024-02-23T00:32:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.523+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6800/4175844757 conn(0x7f77a011f750 0x7f77a0123310 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.524+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77a00596a0 con 0x7f77a011f750 2024-02-23T00:32:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.524+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:32:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.524+0000 7f7787fff700 10 client.36941.objecter _finish_command 9 = system:0 2024-02-23T00:32:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.524+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a0059530 osd=4 4 2024-02-23T00:32:09.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.524+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 4 9 2024-02-23T00:32:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 10 client.36941.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 10 2024-02-23T00:32:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0059530 osd=4 3 2024-02-23T00:32:09.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0059530 osd=4 3 2024-02-23T00:32:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0059530 osd=4 3 2024-02-23T00:32:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 10 2024-02-23T00:32:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a0059530 osd=4 3 2024-02-23T00:32:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 4 10 2024-02-23T00:32:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0059530 osd=4 4 2024-02-23T00:32:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 10 client.36941.objecter _send_command 10 2024-02-23T00:32:09.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.540+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77a00596a0 con 0x7f77a011f750 2024-02-23T00:32:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f77a00596a0 con 0x7f77a011f750 2024-02-23T00:32:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f7787fff700 10 client.36941.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a0059530 osd=4 4 2024-02-23T00:32:09.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 4 10 2024-02-23T00:32:09.543 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 10 client.36941.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 11 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6804/2550013795 conn(0x7f77a0123f00 0x7f77a0124360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0123880 osd=5 3 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0123880 osd=5 3 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0123880 osd=5 3 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 11 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a0123880 osd=5 3 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 5 11 2024-02-23T00:32:09.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0123880 osd=5 4 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 10 client.36941.objecter _send_command 11 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77a012b2c0 con 0x7f77a0123f00 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.542+0000 7f779e7fc700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6804/2550013795 conn(0x7f77a0123f00 0x7f77a0124360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.543+0000 7f7787fff700 10 client.36941.objecter ms_handle_connect 0x7f77a0123f00 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.543+0000 7f779e7fc700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6804/2550013795 conn(0x7f77a0123f00 0x7f77a0124360 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.543+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77a012b2c0 con 0x7f77a0123f00 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.543+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.543+0000 7f7787fff700 10 client.36941.objecter _finish_command 11 = system:0 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.543+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a0123880 osd=5 4 2024-02-23T00:32:09.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.543+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 5 11 2024-02-23T00:32:09.553 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:07.577590+0000 mgr.y (mgr.24437) 226 : cluster [DBG] pgmap v247: 296 pgs: 32 unknown, 264 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:08.282891+0000 mon.a (mon.0) 1360 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:08.410173+0000 mon.c (mon.2) 182 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:08.593843+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:08.596730+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:08.990273+0000 mon.a (mon.0) 1361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi087-82115-15"}]': finished 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:08.990398+0000 mon.a (mon.0) 1362 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.000375+0000 mon.a (mon.0) 1363 : audit [INF] from='client.? 172.21.15.87:0/2571447995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi087-82083-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.027652+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.028433+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:09.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.028828+0000 mon.a (mon.0) 1366 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:09.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 10 client.36941.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:09.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 12 2024-02-23T00:32:09.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0123880 osd=5 3 2024-02-23T00:32:09.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0123880 osd=5 3 2024-02-23T00:32:09.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0123880 osd=5 3 2024-02-23T00:32:09.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 12 2024-02-23T00:32:09.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a0123880 osd=5 3 2024-02-23T00:32:09.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 5 12 2024-02-23T00:32:09.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0123880 osd=5 4 2024-02-23T00:32:09.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 10 client.36941.objecter _send_command 12 2024-02-23T00:32:09.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.558+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77a0128b60 con 0x7f77a0123f00 2024-02-23T00:32:09.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f77a0128b60 con 0x7f77a0123f00 2024-02-23T00:32:09.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:09.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f7787fff700 10 client.36941.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a0123880 osd=5 4 2024-02-23T00:32:09.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 5 12 2024-02-23T00:32:09.562 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:32:09.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 10 client.36941.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:09.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 13 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6808/2509408498 conn(0x7f77a0129240 0x7f77a01296a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0128b60 osd=6 3 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0128b60 osd=6 3 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0128b60 osd=6 3 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 13 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a0128b60 osd=6 3 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 6 13 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0128b60 osd=6 4 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 10 client.36941.objecter _send_command 13 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.560+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77a012b2a0 con 0x7f77a0129240 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.561+0000 7f779f7fe700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6808/2509408498 conn(0x7f77a0129240 0x7f77a01296a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.561+0000 7f779f7fe700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6808/2509408498 conn(0x7f77a0129240 0x7f77a01296a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.561+0000 7f7787fff700 10 client.36941.objecter ms_handle_connect 0x7f77a0129240 2024-02-23T00:32:09.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.562+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77a012b2a0 con 0x7f77a0129240 2024-02-23T00:32:09.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.562+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:32:09.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.562+0000 7f7787fff700 10 client.36941.objecter _finish_command 13 = system:0 2024-02-23T00:32:09.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.562+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a0128b60 osd=6 4 2024-02-23T00:32:09.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.562+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 6 13 2024-02-23T00:32:09.568 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:09.572 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 10 client.36941.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 14 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0128b60 osd=6 3 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0128b60 osd=6 3 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a0128b60 osd=6 3 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 14 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a0128b60 osd=6 3 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 6 14 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0128b60 osd=6 4 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 10 client.36941.objecter _send_command 14 2024-02-23T00:32:09.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.576+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77a012b2a0 con 0x7f77a0129240 2024-02-23T00:32:09.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f77a012b2a0 con 0x7f77a0129240 2024-02-23T00:32:09.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:09.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f7787fff700 10 client.36941.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a0128b60 osd=6 4 2024-02-23T00:32:09.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 6 14 2024-02-23T00:32:09.579 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 10 client.36941.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 15 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6812/373787000 conn(0x7f77a012da10 0x7f77a012de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a012b2a0 osd=7 3 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a012b2a0 osd=7 3 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a012b2a0 osd=7 3 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 15 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a012b2a0 osd=7 3 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 7 15 2024-02-23T00:32:09.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a012b2a0 osd=7 4 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 10 client.36941.objecter _send_command 15 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77a012e4a0 con 0x7f77a012da10 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6812/373787000 conn(0x7f77a012da10 0x7f77a012de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f7787fff700 10 client.36941.objecter ms_handle_connect 0x7f77a012da10 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.578+0000 7f779effd700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6812/373787000 conn(0x7f77a012da10 0x7f77a012de30 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.579+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f77a012e4a0 con 0x7f77a012da10 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.579+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.579+0000 7f7787fff700 10 client.36941.objecter _finish_command 15 = system:0 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.579+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a012b2a0 osd=7 4 2024-02-23T00:32:09.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.579+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 7 15 2024-02-23T00:32:09.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 10 client.36941.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:09.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign -1 16 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a012b2a0 osd=7 3 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a012b2a0 osd=7 3 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 20 client.36941.objecter _get_session s=0x7f77a012b2a0 osd=7 3 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 15 client.36941.objecter _session_command_op_remove -1 16 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 20 client.36941.objecter get_session s=0x7f77a012b2a0 osd=7 3 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 15 client.36941.objecter _session_command_op_assign 7 16 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a012b2a0 osd=7 4 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 10 client.36941.objecter _send_command 16 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.594+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f77a012e4a0 con 0x7f77a012da10 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f7787fff700 1 -- 172.21.15.87:0/2675956627 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f77a012e4a0 con 0x7f77a012da10 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f7787fff700 10 client.36941.objecter ms_dispatch 0x7f77a0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f7787fff700 10 client.36941.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f7787fff700 20 client.36941.objecter put_session s=0x7f77a012b2a0 osd=7 4 2024-02-23T00:32:09.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f7787fff700 15 client.36941.objecter _session_command_op_remove 7 16 2024-02-23T00:32:09.599 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:32:09.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 librados: watch_flush enter 2024-02-23T00:32:09.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 librados: watch_flush exit 2024-02-23T00:32:09.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 client.36941.objecter close_session for osd.0 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6801/2153191863 conn(0x7f77a00fdea0 msgr2=0x7f77a00fe2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6801/2153191863 conn(0x7f77a00fdea0 0x7f77a00fe2c0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a005ee80 osd=0 1 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 client.36941.objecter close_session for osd.1 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6805/685354171 conn(0x7f77a00f4be0 msgr2=0x7f77a00273c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6805/685354171 conn(0x7f77a00f4be0 0x7f77a00273c0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00f4a70 osd=1 1 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 client.36941.objecter close_session for osd.2 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6809/3605394228 conn(0x7f77a0117500 msgr2=0x7f77a011e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6809/3605394228 conn(0x7f77a0117500 0x7f77a011e580 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0117390 osd=2 1 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 client.36941.objecter close_session for osd.3 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f779f7fe700 1 -- 172.21.15.87:0/2675956627 reap_dead start 2024-02-23T00:32:09.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6813/128059790 conn(0x7f77a0059e00 msgr2=0x7f77a0121c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6813/128059790 conn(0x7f77a0059e00 0x7f77a0121c00 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a00280d0 osd=3 1 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 client.36941.objecter close_session for osd.4 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6800/4175844757 conn(0x7f77a011f750 msgr2=0x7f77a0123310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6800/4175844757 conn(0x7f77a011f750 0x7f77a0123310 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0059530 osd=4 1 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 client.36941.objecter close_session for osd.5 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6804/2550013795 conn(0x7f77a0123f00 msgr2=0x7f77a0124360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6804/2550013795 conn(0x7f77a0123f00 0x7f77a0124360 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0123880 osd=5 1 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 client.36941.objecter close_session for osd.6 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6808/2509408498 conn(0x7f77a0129240 msgr2=0x7f77a01296a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6808/2509408498 conn(0x7f77a0129240 0x7f77a01296a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a0128b60 osd=6 1 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 client.36941.objecter close_session for osd.7 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6812/373787000 conn(0x7f77a012da10 msgr2=0x7f77a012de30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.171:6812/373787000 conn(0x7f77a012da10 0x7f77a012de30 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 20 client.36941.objecter put_session s=0x7f77a012b2a0 osd=7 1 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 20 client.36941.objecter shutdown clearing up homeless session... 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.596+0000 7f77a592d700 10 client.36941.objecter successfully canceled tick 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f779f7fe700 1 -- 172.21.15.87:0/2675956627 reap_dead start 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7788062c90 msgr2=0x7f7788065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7788062c90 0x7f7788065150 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 rx=0x7f7790011510 tx=0x7f779000b040).stop 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00fcbb0 msgr2=0x7f77a0109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00fcbb0 0x7f77a0109700 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 rx=0x7f77940048f0 tx=0x7f7794009cb0).stop 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 shutdown_connections 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7788062c90 0x7f7788065150 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 --2- 172.21.15.87:0/2675956627 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77a00fcbb0 0x7f77a0109700 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:09.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 >> 172.21.15.87:0/2675956627 conn(0x7f77a00f2820 msgr2=0x7f77a00f4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 shutdown_connections 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 -- 172.21.15.87:0/2675956627 wait complete. 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:09.597+0000 7f77a592d700 1 librados: shutdown 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.604 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:32:09.605 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:09.618 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:09.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:09.723 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:09.751 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:09.869 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:09.949 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:09.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:10.053 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:10.080 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:10.177 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:10.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:10.281 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:10.311 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.009220+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.c scrub starts 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.012155+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.c scrub ok 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.107137+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.c scrub starts 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.110133+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.c scrub ok 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.410956+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.444371+0000 mon.c (mon.2) 184 : audit [DBG] from='client.? 172.21.15.87:0/2675956627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.516322+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.518142+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.579058+0000 mon.a (mon.0) 1367 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-23T00:32:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.586509+0000 osd.0 (osd.0) 111 : cluster [DBG] 4.b scrub starts 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.589977+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.b scrub ok 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.991426+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? 172.21.15.87:0/2571447995' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi087-82083-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.991550+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.991639+0000 mon.a (mon.0) 1370 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.991696+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.997391+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:09.999593+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:10.000076+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.87:0/284353186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82019-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:10.000376+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:10.000991+0000 mon.a (mon.0) 1374 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82019-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.009220+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.c scrub starts 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.012155+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.c scrub ok 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.107137+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.c scrub starts 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.110133+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.c scrub ok 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.410956+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.444371+0000 mon.c (mon.2) 184 : audit [DBG] from='client.? 172.21.15.87:0/2675956627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.516322+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.518142+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.579058+0000 mon.a (mon.0) 1367 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.586509+0000 osd.0 (osd.0) 111 : cluster [DBG] 4.b scrub starts 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.589977+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.b scrub ok 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.991426+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? 172.21.15.87:0/2571447995' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi087-82083-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.991550+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.991639+0000 mon.a (mon.0) 1370 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-23T00:32:10.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.991696+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-02-23T00:32:10.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.997391+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:10.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:09.999593+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:10.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:10.000076+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.87:0/284353186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82019-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:10.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:10.000376+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:10.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:10.000991+0000 mon.a (mon.0) 1374 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82019-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:10.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.009220+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.c scrub starts 2024-02-23T00:32:10.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.012155+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.c scrub ok 2024-02-23T00:32:10.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.107137+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.c scrub starts 2024-02-23T00:32:10.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.110133+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.c scrub ok 2024-02-23T00:32:10.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.410956+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:10.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.444371+0000 mon.c (mon.2) 184 : audit [DBG] from='client.? 172.21.15.87:0/2675956627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.516322+0000 osd.1 (osd.1) 95 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.518142+0000 osd.1 (osd.1) 96 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.579058+0000 mon.a (mon.0) 1367 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.586509+0000 osd.0 (osd.0) 111 : cluster [DBG] 4.b scrub starts 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.589977+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.b scrub ok 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.991426+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? 172.21.15.87:0/2571447995' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi087-82083-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.991550+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.991639+0000 mon.a (mon.0) 1370 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.991696+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.997391+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:09.999593+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:10.000076+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.87:0/284353186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82019-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:10.000376+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:10.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:10.000991+0000 mon.a (mon.0) 1374 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82019-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:10.940 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:32:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:32:11.446 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:32:11.447 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:32:11.447 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-23T00:32:11.447 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-23T00:32:11.447 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-23T00:32:11.447 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.577945+0000 mgr.y (mgr.24437) 227 : cluster [DBG] pgmap v250: 303 pgs: 40 unknown, 263 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.975483+0000 osd.3 (osd.3) 103 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:09.978372+0000 osd.3 (osd.3) 104 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:10.142802+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:10.145681+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:10.318972+0000 osd.5 (osd.5) 87 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:10.319739+0000 osd.5 (osd.5) 88 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:10.411766+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:10.901410+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:10.903181+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:32:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:11.238092+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]': finished 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:11.238169+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82019-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:11.238308+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:11.238719+0000 mon.c (mon.2) 187 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:11.240976+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.577945+0000 mgr.y (mgr.24437) 227 : cluster [DBG] pgmap v250: 303 pgs: 40 unknown, 263 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.975483+0000 osd.3 (osd.3) 103 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:09.978372+0000 osd.3 (osd.3) 104 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:10.142802+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:10.145681+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:32:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:10.318972+0000 osd.5 (osd.5) 87 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:32:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:10.319739+0000 osd.5 (osd.5) 88 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:32:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:10.411766+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:10.901410+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:32:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:10.903181+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:32:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:11.238092+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]': finished 2024-02-23T00:32:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:11.238169+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82019-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:11.238308+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-02-23T00:32:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:11.238719+0000 mon.c (mon.2) 187 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:11.240976+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:11.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.577945+0000 mgr.y (mgr.24437) 227 : cluster [DBG] pgmap v250: 303 pgs: 40 unknown, 263 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:11.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.975483+0000 osd.3 (osd.3) 103 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:32:11.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:09.978372+0000 osd.3 (osd.3) 104 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:32:11.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:10.142802+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:32:11.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:10.145681+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:32:11.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:10.318972+0000 osd.5 (osd.5) 87 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:32:11.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:10.319739+0000 osd.5 (osd.5) 88 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:32:11.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:10.411766+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:11.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:10.901410+0000 osd.4 (osd.4) 89 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:32:11.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:10.903181+0000 osd.4 (osd.4) 90 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:32:11.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:11.238092+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi087-82011-26"}]': finished 2024-02-23T00:32:11.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:11.238169+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi087-82019-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:11.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:11.238308+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-02-23T00:32:11.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:11.238719+0000 mon.c (mon.2) 187 : audit [INF] from='client.? 172.21.15.87:0/3927026979' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:11.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:11.240976+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]: dispatch 2024-02-23T00:32:11.660 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:11.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:11.764 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:11.793 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:11.865 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:11.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:11.887 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:32:11.887 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:32:11.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:32:11.887 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:32:11.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.887 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24329.0:1 2.12 2:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e54)", 2024-02-23T00:32:11.887 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:27:16.942446+0000", 2024-02-23T00:32:11.887 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 294.94114827999999, 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070525851000000001, 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24329", 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2824009082", 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:16.942446+0000", 2024-02-23T00:32:11.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:16.942446+0000", 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:16.942444+0000", 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3989999999999998e-06 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:16.942452+0000", 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0680000000000003e-06 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:16.942461+0000", 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6720000000000004e-06 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:16.942468+0000", 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8365999999999997e-05 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:16.942517+0000", 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6860000000000006e-06 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:16.942524+0000", 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070382966000000005 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:17.012907+0000", 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4511999999999995e-05 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:17.012972+0000", 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070525851000000001 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.24712.0:10 7.a e107/94 7:54825363:test-rados-api-smithi087-82115-4::foo:head v 107'3, mlcod=0'0)", 2024-02-23T00:32:11.892 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:30:57.494185+0000", 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 74.389409389999997, 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010984529999999999, 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.494185+0000", 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.494185+0000", 2024-02-23T00:32:11.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.494184+0000", 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0744000000000001e-05 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.494195+0000", 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1260000000000001e-05 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.494206+0000", 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2579999999999998e-06 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.494214+0000", 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00085235400000000005 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.495066+0000", 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1722e-05 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.495088+0000", 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010047960999999999 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.505136+0000", 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3624000000000002e-05 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:57.505169+0000", 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010984529999999999 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25166.0:2 59.6 59:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:02.557319+0000", 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 69.326275264000003, 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056150042999999997, 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25166", 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/458030738", 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557319+0000", 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557319+0000", 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557318+0000", 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8050000000000003e-06 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557324+0000", 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8560000000000007e-06 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557331+0000", 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4490000000000002e-06 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557339+0000", 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7904000000000002e-05 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557377+0000", 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127668 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557504+0000", 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000131311 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,6", 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557636+0000", 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033497099999999999 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557971+0000", 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000210115 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.558181+0000", 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055228405000000001 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.613409+0000", 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4125000000000003e-05 2024-02-23T00:32:11.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.613443+0000", 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5678999999999999e-05 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.613469+0000", 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056150042999999997 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:8 56.2 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.569023+0000", 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.314570777, 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.061982072999999999, 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 8 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569023+0000", 2024-02-23T00:32:11.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569023+0000", 2024-02-23T00:32:11.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:32:11.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569022+0000", 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9890000000000003e-06 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569029+0000", 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5150000000000006e-06 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569037+0000", 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0450000000000003e-06 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569045+0000", 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033288000000000002 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569377+0000", 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000205007 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569582+0000", 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070529399999999997 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.576635+0000", 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8623999999999996e-05 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.576734+0000", 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000176076 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.576910+0000", 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00079704099999999996 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.577707+0000", 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9565e-05 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.577807+0000", 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053133055999999998 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.630940+0000", 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2215999999999998e-05 2024-02-23T00:32:11.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.630972+0000", 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3599000000000001e-05 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631006+0000", 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.061982072999999999 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.909 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:9 56.2 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head [watch watch cookie 93958402142112] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.631729+0000", 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.251864972999996, 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018840933000000001, 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 9 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631729+0000", 2024-02-23T00:32:11.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631729+0000", 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631728+0000", 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8719999999999999e-06 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631734+0000", 2024-02-23T00:32:11.911 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1409999999999994e-06 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631743+0000", 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6979999999999997e-06 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631748+0000", 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5034999999999999e-05 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631783+0000", 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5764000000000002e-05 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631858+0000", 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104113 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631962+0000", 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00027199 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.632234+0000", 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017308100000000001 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.632408+0000", 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018079319999999999 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.650487+0000", 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0031000000000001e-05 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.650517+0000", 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3294000000000002e-05 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.650570+0000", 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018840933000000001 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:14 56.1 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:32:11.915 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.661664+0000", 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.221930417999999, 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044383775, 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 14 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661664+0000", 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661664+0000", 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661663+0000", 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3850000000000001e-06 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661668+0000", 2024-02-23T00:32:11.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5259999999999999e-06 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661678+0000", 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7389999999999999e-06 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661683+0000", 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7559000000000002e-05 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661710+0000", 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000168534 2024-02-23T00:32:11.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661879+0000", 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014226162000000001 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676105+0000", 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4277000000000006e-05 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676189+0000", 2024-02-23T00:32:11.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100592 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676290+0000", 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00034506400000000002 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676635+0000", 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000797236 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.677432+0000", 2024-02-23T00:32:11.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028506070000000001 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.705938+0000", 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3522000000000006e-05 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.706002+0000", 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5997e-05 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.706048+0000", 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044383775 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:16 56.1 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head [watch unwatch cookie 93958402138000] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.714409+0000", 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.169185385999995, 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021332225999999999, 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.922 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 16 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714409+0000", 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714409+0000", 2024-02-23T00:32:11.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714407+0000", 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.015e-06 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714413+0000", 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7249999999999997e-06 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714421+0000", 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1869999999999999e-06 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714425+0000", 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2351000000000002e-05 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714468+0000", 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012891899999999999 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714597+0000", 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00019076400000000001 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714787+0000", 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041226000000000001 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.715200+0000", 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001342099 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.716542+0000", 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019100005 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.735642+0000", 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1798999999999997e-05 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.735683+0000", 2024-02-23T00:32:11.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7598e-05 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.735741+0000", 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021332225999999999 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:26 56.2 56:59de2489:test-rados-api-smithi087-82147-11::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.753579+0000", 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.130015270000001, 2024-02-23T00:32:11.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010859437, 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 26 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.753579+0000", 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.753579+0000", 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.753578+0000", 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2143e-05 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.753590+0000", 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0166999999999999e-05 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.753600+0000", 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8210000000000001e-06 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.753605+0000", 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8408000000000001e-05 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.753654+0000", 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014086900000000001 2024-02-23T00:32:11.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.753794+0000", 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022362900000000002 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.756031+0000", 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6948999999999998e-05 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.756108+0000", 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1061000000000001e-05 2024-02-23T00:32:11.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.756199+0000", 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00032382400000000001 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.756523+0000", 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000185501 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.756708+0000", 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0075836050000000002 2024-02-23T00:32:11.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.764292+0000", 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7838999999999998e-05 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.764370+0000", 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8864999999999999e-05 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.764438+0000", 2024-02-23T00:32:11.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010859437 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:27 56.2 56:59de2489:test-rados-api-smithi087-82147-11::foo:head [watch watch cookie 93958402367200] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.766674+0000", 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.116919749999994, 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011887193000000001, 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.935 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 27 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.766674+0000", 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.766674+0000", 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.766673+0000", 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5060000000000003e-06 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.766679+0000", 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.516e-06 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.766688+0000", 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2889999999999997e-06 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.766694+0000", 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0320000000000001e-05 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.766724+0000", 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1302000000000004e-05 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.766785+0000", 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3776999999999993e-05 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.766879+0000", 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00025957600000000002 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.767139+0000", 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00023821399999999999 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.767377+0000", 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011078496 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.778455+0000", 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4341000000000001e-05 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.778500+0000", 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1919000000000001e-05 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.778562+0000", 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011887193000000001 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.940 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:37 56.1 56:972518f1:test-rados-api-smithi087-82147-12::foo:head [watch unwatch cookie 93958402138000] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.080383814000001, 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019757686999999999, 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 37 2024-02-23T00:32:11.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:11.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2400000000000001e-06 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803214+0000", 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9640000000000003e-06 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803222+0000", 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7230000000000002e-06 2024-02-23T00:32:11.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803226+0000", 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0100999999999999e-05 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803256+0000", 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8650999999999998e-05 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803314+0000", 2024-02-23T00:32:11.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0869e-05 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803405+0000", 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00032610700000000001 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803731+0000", 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012946180999999999 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.816678+0000", 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.006224963 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.822902+0000", 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8585000000000002e-05 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.822931+0000", 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7018999999999999e-05 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.822968+0000", 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019757686999999999 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:56 56.2 56:6c9b5892:test-rados-api-smithi087-82147-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.877562+0000", 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 68.006032722, 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015386004999999999, 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.947 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 56 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877562+0000", 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877562+0000", 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:32:11.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877560+0000", 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0372e-05 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877571+0000", 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1782e-05 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877582+0000", 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0769999999999996e-06 2024-02-23T00:32:11.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877588+0000", 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4140999999999997e-05 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877623+0000", 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016143599999999999 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877784+0000", 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011506860000000001 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.889291+0000", 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2185000000000003e-05 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.889343+0000", 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2941000000000003e-05 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.889436+0000", 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036122900000000002 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.889797+0000", 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000231883 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.890029+0000", 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028250570000000002 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.892854+0000", 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6941000000000003e-05 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.892911+0000", 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6390999999999998e-05 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.892948+0000", 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015386004999999999 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:67 56.0 56:2858fd9f:test-rados-api-smithi087-82147-17::foo:head [watch unwatch cookie 93958402367200] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:07.920664+0000", 2024-02-23T00:32:11.953 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 63.962930016000001, 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073703837999999994, 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 67 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920664+0000", 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920664+0000", 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920663+0000", 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8689999999999998e-06 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920669+0000", 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1250000000000001e-05 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920681+0000", 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8099999999999998e-06 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920688+0000", 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9345000000000003e-05 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920738+0000", 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109922 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920848+0000", 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016524700000000001 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,6", 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.921013+0000", 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00089610999999999998 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.921909+0000", 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125143 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.922034+0000", 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072257924000000001 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.994292+0000", 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4425000000000003e-05 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.994327+0000", 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1483000000000002e-05 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.994368+0000", 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073703837999999994 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:69 56.1 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e118)", 2024-02-23T00:32:11.959 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:09.916487+0000", 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 61.967107339999998, 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062093785999999998, 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 69 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916487+0000", 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916487+0000", 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916486+0000", 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7636000000000001e-05 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916514+0000", 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0845999999999999e-05 2024-02-23T00:32:11.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916524+0000", 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1185000000000001e-05 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916546+0000", 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3442999999999998e-05 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916579+0000", 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000177116 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916756+0000", 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077845400000000004 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.917535+0000", 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8349999999999994e-05 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.917603+0000", 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108934 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.917712+0000", 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00030745799999999998 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.918019+0000", 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032894684 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.950914+0000", 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027609002000000001 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.978523+0000", 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2388999999999998e-05 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.978555+0000", 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5239e-05 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.978581+0000", 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062093785999999998 2024-02-23T00:32:11.965 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:75 56.1 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head [watch unwatch cookie 93958402367200] snapc 0=[] ondisk+write+known_if_redirected e122)", 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:12.996174+0000", 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 58.887420228000003, 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.084863057000000006, 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 75 2024-02-23T00:32:11.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996174+0000", 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996174+0000", 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996173+0000", 2024-02-23T00:32:11.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1340000000000003e-06 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996178+0000", 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0209e-05 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996188+0000", 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9609999999999996e-06 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996193+0000", 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0006999999999999e-05 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996223+0000", 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0087000000000007e-05 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996313+0000", 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000276903 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996590+0000", 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00053075400000000003 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.997121+0000", 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070072748000000004 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.067194+0000", 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01373247 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.080926+0000", 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.193e-05 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.080968+0000", 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8838999999999996e-05 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081037+0000", 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.084863057000000006 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.971 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:76 56.1 56:a5875266:test-rados-api-smithi087-82147-19::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e122)", 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:13.081838+0000", 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 58.801755925999998, 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028322508, 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 76 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.972 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081838+0000", 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081838+0000", 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081837+0000", 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2859999999999996e-06 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081843+0000", 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0053e-05 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081853+0000", 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1229999999999999e-06 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081858+0000", 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0298999999999999e-05 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081889+0000", 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015674600000000001 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.082046+0000", 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011441438 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093487+0000", 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6672000000000002e-05 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093544+0000", 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108148 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093652+0000", 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00028719800000000001 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093939+0000", 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051637150000000001 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.099103+0000", 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011005768000000001 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.110108+0000", 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0571999999999998e-05 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.110139+0000", 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1756999999999999e-05 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.110161+0000", 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028322508 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24413.0:238 3.c 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e137)", 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:28.139198+0000", 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 43.744395859999997, 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94992057799999996, 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.978 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24413", 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.171:0/2453883515", 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 238 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139198+0000", 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139198+0000", 2024-02-23T00:32:11.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139197+0000", 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0800000000000002e-06 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139203+0000", 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1200000000000008e-06 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139212+0000", 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.344e-06 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139217+0000", 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9641000000000002e-05 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139257+0000", 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1140000000000002e-06 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139263+0000", 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.93888930100000001 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.078152+0000", 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013934400000000001 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for missing object", 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.078292+0000", 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010582111999999999 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.088874+0000", 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7443999999999997e-05 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.088971+0000", 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000147868 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.089119+0000", 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94992057799999996 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.983 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:578 63.a 63:54d171db:test-rados-api-smithi087-82091-14::foo7:head [writefull 0~11] snapc 5=[5,4] ondisk+write+known_if_redirected e146)", 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 34.861017826000001, 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044787916999999997, 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 578 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.984 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3959999999999997e-06 2024-02-23T00:32:11.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022581+0000", 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8709999999999997e-06 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022589+0000", 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2200000000000003e-06 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022593+0000", 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4632000000000002e-05 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022638+0000", 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012398899999999999 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022762+0000", 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00023933200000000001 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,4", 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.023001+0000", 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00054081799999999998 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:11.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.023542+0000", 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011881500000000001 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.023661+0000", 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043632126 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.067293+0000", 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3884999999999997e-05 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.067337+0000", 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7705e-05 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.067364+0000", 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044787916999999997 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDECSubOpWrite(93.0s2 150/142 ECSubWrite(tid=263, reqid=client.36632.0:18, at_version=150'7, trim_to=0'0, roll_forward_to=149'6))", 2024-02-23T00:32:11.989 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:41.003835+0000", 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 30.879759334999999, 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010928096999999999, 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.003835+0000", 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.003835+0000", 2024-02-23T00:32:11.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.003835+0000", 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1410999999999999e-05 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.003856+0000", 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1001e-05 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.003867+0000", 2024-02-23T00:32:11.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5220000000000007e-06 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.003876+0000", 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041319629999999998 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.008008+0000", 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1868999999999999e-05 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:32:11.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.008019+0000", 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0067203419999999998 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.014740+0000", 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3278999999999999e-05 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.014763+0000", 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010928096999999999 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36842.0:1 112.3s0 112:d6e58105:test-rados-api-smithi087-82115-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:57.017766+0000", 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 14.865828075, 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.01719962, 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36842", 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3011193832", 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:11.994 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017766+0000", 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017766+0000", 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017764+0000", 2024-02-23T00:32:11.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5610000000000004e-06 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017774+0000", 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1437000000000001e-05 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017785+0000", 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6439999999999995e-06 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:11.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017795+0000", 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020082722000000001 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.037878+0000", 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3429999999999998e-06 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.037882+0000", 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.99038801300000001 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.028270+0000", 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113621 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.028384+0000", 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00034479999999999998 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.028728+0000", 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011224329999999999 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.998 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.029851+0000", 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050385860000000003 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.034889+0000", 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6290000000000003e-05 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.034966+0000", 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.01719962 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:11.999 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26469.0:2 123.1c 123:3df5ee6d:test-rados-api-smithi087-82083-28::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected e179)", 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:10.007615+0000", 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 1.875978943, 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010299986000000001, 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26469", 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2571447995", 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:32:12.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.007615+0000", 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.007615+0000", 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:12.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.007613+0000", 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2782999999999999e-05 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.007626+0000", 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5339999999999999e-05 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.007641+0000", 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6076e-05 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.007657+0000", 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010108200000000001 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.007759+0000", 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9899999999999997e-06 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.007767+0000", 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070415909999999998 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:12.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.014808+0000", 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013520900000000001 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.014943+0000", 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000151547 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0,4", 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.015095+0000", 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00042922300000000001 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:12.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.015524+0000", 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0022221070000000001 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.017746+0000", 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0007999999999998e-05 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.017816+0000", 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4070000000000002e-05 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.017870+0000", 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5003000000000003e-05 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:10.017915+0000", 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010299986000000001 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:12.006 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:32:12.007 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:12.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp api_aio: Running main() from gmock_main.cc 2024-02-23T00:32:12.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [==========] Running 41 tests from 2 test suites. 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] Global test environment set-up. 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] 25 tests from LibRadosAio 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.TooBig (7806 ms) 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3581 ms) 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (4034 ms) 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3268 ms) 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2850 ms) 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-02-23T00:32:12.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (4312 ms) 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3274 ms) 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2805 ms) 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3354 ms) 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2944 ms) 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2757 ms) 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3058 ms) 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2922 ms) 2024-02-23T00:32:12.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3223 ms) 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.Flush (2768 ms) 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2987 ms) 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3010 ms) 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3002 ms) 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3006 ms) 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2987 ms) 2024-02-23T00:32:12.010 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2997 ms) 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3253 ms) 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2752 ms) 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3009 ms) 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3002 ms) 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] 25 tests from LibRadosAio (82961 ms total) 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-02-23T00:32:12.011 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6978 ms) 2024-02-23T00:32:12.044 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:12.071 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:12.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:12.177 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:12.206 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:12.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.168+0000 7f390c246700 1 -- 172.21.15.87:0/1655771336 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f8d10 msgr2=0x7f39040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:12.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.168+0000 7f390c246700 1 --2- 172.21.15.87:0/1655771336 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f8d10 0x7f39040f90f0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7f3900004870 tx=0x7f3900009cb0).stop 2024-02-23T00:32:12.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.168+0000 7f390c246700 1 -- 172.21.15.87:0/1655771336 shutdown_connections 2024-02-23T00:32:12.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.168+0000 7f390c246700 1 --2- 172.21.15.87:0/1655771336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3904102270 0x7f3904104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:12.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.168+0000 7f390c246700 1 --2- 172.21.15.87:0/1655771336 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f8d10 0x7f39040f90f0 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:12.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.168+0000 7f390c246700 1 --2- 172.21.15.87:0/1655771336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f39040f9670 0x7f3904101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:12.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.168+0000 7f390c246700 1 -- 172.21.15.87:0/1655771336 >> 172.21.15.87:0/1655771336 conn(0x7f39040f2820 msgr2=0x7f39040f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.168+0000 7f390c246700 1 -- 172.21.15.87:0/1655771336 shutdown_connections 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.168+0000 7f390c246700 1 -- 172.21.15.87:0/1655771336 wait complete. 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 librados: starting msgr at 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 librados: starting objecter 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 Processor -- start 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 -- start start 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 librados: setting wanted keys 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 librados: calling monclient init 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39040f8d10 0x7f390410e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f9670 0x7f390410e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3904102270 0x7f3904108350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904108840 con 0x7f39040f8d10 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39041089b0 con 0x7f3904102270 2024-02-23T00:32:12.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f390c246700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3904108b70 con 0x7f39040f9670 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f39097e1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f9670 0x7f390410e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f39097e1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f9670 0x7f390410e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:59994/0 (socket says 172.21.15.87:59994) 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f3909fe2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39040f8d10 0x7f390410e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.169+0000 7f39097e1700 1 -- 172.21.15.87:0/2446637229 learned_addr learned my addr 172.21.15.87:0/2446637229 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f390a7e3700 1 --2- 172.21.15.87:0/2446637229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3904102270 0x7f3904108350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f39097e1700 1 -- 172.21.15.87:0/2446637229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3904102270 msgr2=0x7f3904108350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f39097e1700 1 --2- 172.21.15.87:0/2446637229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3904102270 0x7f3904108350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f39097e1700 1 -- 172.21.15.87:0/2446637229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39040f8d10 msgr2=0x7f390410e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f39097e1700 1 --2- 172.21.15.87:0/2446637229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39040f8d10 0x7f390410e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f39097e1700 1 -- 172.21.15.87:0/2446637229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3900005040 con 0x7f39040f9670 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f390a7e3700 1 --2- 172.21.15.87:0/2446637229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3904102270 0x7f3904108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f39097e1700 1 --2- 172.21.15.87:0/2446637229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f9670 0x7f390410e6a0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f38f40041d0 tx=0x7f38f4009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f38faffd700 10 client.?.objecter ms_handle_connect 0x7f39040f9670 2024-02-23T00:32:12.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f38faffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f38faffd700 1 -- 172.21.15.87:0/2446637229 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f4007b40 con 0x7f39040f9670 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f38faffd700 1 -- 172.21.15.87:0/2446637229 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38f400f040 con 0x7f39040f9670 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3904108e50 con 0x7f39040f9670 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f38faffd700 1 -- 172.21.15.87:0/2446637229 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f4014810 con 0x7f39040f9670 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f390c246700 10 client.36965.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3904114fe0 con 0x7f39040f9670 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.170+0000 7f390c246700 1 librados: init done 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.171+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390405fa60 con 0x7f39040f9670 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.171+0000 7f38faffd700 1 -- 172.21.15.87:0/2446637229 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f38f4009110 con 0x7f39040f9670 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.172+0000 7f38faffd700 1 --2- 172.21.15.87:0/2446637229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f38f0062c90 0x7f38f0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.172+0000 7f3909fe2700 1 --2- 172.21.15.87:0/2446637229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f38f0062c90 0x7f38f0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.172+0000 7f38faffd700 1 -- 172.21.15.87:0/2446637229 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(181..181 src has 38..181) v4 ==== 9325+0+0 (secure 0 0 0) 0x7f38f407d7b0 con 0x7f39040f9670 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.172+0000 7f38faffd700 10 client.36965.objecter ms_dispatch 0x7f3904001550 osd_map(181..181 src has 38..181) v4 2024-02-23T00:32:12.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.172+0000 7f38faffd700 3 client.36965.objecter handle_osd_map got epochs [181,181] > 0 2024-02-23T00:32:12.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.172+0000 7f38faffd700 3 client.36965.objecter handle_osd_map decoding full epoch 181 2024-02-23T00:32:12.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.172+0000 7f38faffd700 20 client.36965.objecter dump_active .. 0 homeless 2024-02-23T00:32:12.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.172+0000 7f38faffd700 10 client.36965.objecter ms_handle_connect 0x7f38f0062c90 2024-02-23T00:32:12.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.172+0000 7f3909fe2700 1 --2- 172.21.15.87:0/2446637229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f38f0062c90 0x7f38f0065150 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f3900004560 tx=0x7f3900007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:12.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.176+0000 7f38faffd700 1 -- 172.21.15.87:0/2446637229 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f38f4050790 con 0x7f39040f9670 2024-02-23T00:32:12.243 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:12.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:12.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.356+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f39041072a0 con 0x7f39040f9670 2024-02-23T00:32:12.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:12.384 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:12.403 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:12.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:12.507 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:12.512 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:12] "GET /metrics HTTP/1.1" 200 192112 "" "Prometheus/2.33.4" 2024-02-23T00:32:12.535 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:12.556 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:12.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:12.562 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-23T00:32:12.581 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:12.661 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:12.688 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:12.707 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:12.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:12.811 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:12.813 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:10.693295+0000 mgr.y (mgr.24437) 228 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:12.813 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:11.412914+0000 mon.c (mon.2) 188 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:12.813 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:11.578403+0000 mgr.y (mgr.24437) 229 : cluster [DBG] pgmap v253: 295 pgs: 32 creating+peering, 2 peering, 261 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:12.813 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:11.741847+0000 osd.7 (osd.7) 89 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:11.743272+0000 osd.7 (osd.7) 90 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:11.911732+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:11.912500+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:11.986685+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]': finished 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:11.986804+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]': finished 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:11.986904+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:11.992985+0000 mon.c (mon.2) 189 : audit [INF] from='client.? 172.21.15.87:0/3435478986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi087-82083-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:11.999073+0000 mon.a (mon.0) 1382 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi087-82083-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:12.022813+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:12.024733+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:12.025185+0000 mon.a (mon.0) 1385 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi087-82011-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:12.356776+0000 mon.c (mon.2) 190 : audit [INF] from='client.? 172.21.15.87:0/2446637229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:12.357060+0000 mon.a (mon.0) 1386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:12.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:12.413685+0000 mon.c (mon.2) 191 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:10.693295+0000 mgr.y (mgr.24437) 228 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:11.412914+0000 mon.c (mon.2) 188 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:11.578403+0000 mgr.y (mgr.24437) 229 : cluster [DBG] pgmap v253: 295 pgs: 32 creating+peering, 2 peering, 261 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:11.741847+0000 osd.7 (osd.7) 89 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:11.743272+0000 osd.7 (osd.7) 90 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:11.911732+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:11.912500+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:11.986685+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]': finished 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:11.986804+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]': finished 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:11.986904+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-02-23T00:32:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:11.992985+0000 mon.c (mon.2) 189 : audit [INF] from='client.? 172.21.15.87:0/3435478986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi087-82083-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:11.999073+0000 mon.a (mon.0) 1382 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi087-82083-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:12.022813+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:12.024733+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:10.693295+0000 mgr.y (mgr.24437) 228 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:11.412914+0000 mon.c (mon.2) 188 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:11.578403+0000 mgr.y (mgr.24437) 229 : cluster [DBG] pgmap v253: 295 pgs: 32 creating+peering, 2 peering, 261 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:11.741847+0000 osd.7 (osd.7) 89 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:11.743272+0000 osd.7 (osd.7) 90 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:11.911732+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:11.912500+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:32:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:11.986685+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]': finished 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:11.986804+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi087-82011-26"}]': finished 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:11.986904+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:11.992985+0000 mon.c (mon.2) 189 : audit [INF] from='client.? 172.21.15.87:0/3435478986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi087-82083-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:11.999073+0000 mon.a (mon.0) 1382 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi087-82083-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:12.022813+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:12.024733+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:12.025185+0000 mon.a (mon.0) 1385 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi087-82011-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:12.356776+0000 mon.c (mon.2) 190 : audit [INF] from='client.? 172.21.15.87:0/2446637229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:12.357060+0000 mon.a (mon.0) 1386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:12.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:12.413685+0000 mon.c (mon.2) 191 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:12.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:12.025185+0000 mon.a (mon.0) 1385 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi087-82011-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:12.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:12.356776+0000 mon.c (mon.2) 190 : audit [INF] from='client.? 172.21.15.87:0/2446637229' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:12.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:12.357060+0000 mon.a (mon.0) 1386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:12.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:12.413685+0000 mon.c (mon.2) 191 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:12.837 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:12.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:32:12.855 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:12.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:12.959 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:12.988 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:13.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:12.998+0000 7f38faffd700 1 -- 172.21.15.87:0/2446637229 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v182) v1 ==== 101+0+0 (secure 0 0 0) 0x7f38f4050380 con 0x7f39040f9670 2024-02-23T00:32:13.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.002+0000 7f390c246700 10 librados: watch_flush enter 2024-02-23T00:32:13.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.004+0000 7f390c246700 10 librados: watch_flush exit 2024-02-23T00:32:13.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.004+0000 7f390c246700 20 client.36965.objecter shutdown clearing up homeless session... 2024-02-23T00:32:13.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.004+0000 7f390c246700 10 client.36965.objecter successfully canceled tick 2024-02-23T00:32:13.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.004+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f38f0062c90 msgr2=0x7f38f0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:13.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.004+0000 7f390c246700 1 --2- 172.21.15.87:0/2446637229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f38f0062c90 0x7f38f0065150 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 rx=0x7f3900004560 tx=0x7f3900007f90).stop 2024-02-23T00:32:13.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.004+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f9670 msgr2=0x7f390410e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:13.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.004+0000 7f390c246700 1 --2- 172.21.15.87:0/2446637229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f9670 0x7f390410e6a0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f38f40041d0 tx=0x7f38f4009f60).stop 2024-02-23T00:32:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.009+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 shutdown_connections 2024-02-23T00:32:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.009+0000 7f390c246700 1 --2- 172.21.15.87:0/2446637229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f38f0062c90 0x7f38f0065150 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.009+0000 7f390c246700 1 --2- 172.21.15.87:0/2446637229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3904102270 0x7f3904108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.009+0000 7f390c246700 1 --2- 172.21.15.87:0/2446637229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f39040f8d10 0x7f390410e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.009+0000 7f390c246700 1 --2- 172.21.15.87:0/2446637229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f39040f9670 0x7f390410e6a0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:13.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.009+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 >> 172.21.15.87:0/2446637229 conn(0x7f39040f2820 msgr2=0x7f39040f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:13.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.021+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 shutdown_connections 2024-02-23T00:32:13.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.021+0000 7f390c246700 1 -- 172.21.15.87:0/2446637229 wait complete. 2024-02-23T00:32:13.025 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:13.024+0000 7f390c246700 1 librados: shutdown 2024-02-23T00:32:13.029 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:32:13.052 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:13.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:13.156 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:13.182 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:13.333 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:11.983974+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:11.989324+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.233067+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.233953+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.385461+0000 osd.5 (osd.5) 89 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.386284+0000 osd.5 (osd.5) 90 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.720910+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.721714+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.980124+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:32:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.983017+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.984396+0000 mon.a (mon.0) 1387 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:12.993699+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi087-82083-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:12.993861+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi087-82011-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:12.993937+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:12.994005+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:13.003935+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.87:0/570645369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi087-82019-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:13.006417+0000 mon.a (mon.0) 1392 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi087-82011-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:13.007195+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi087-82019-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:13.414537+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:11.983974+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:32:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:11.989324+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.233067+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.233953+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.385461+0000 osd.5 (osd.5) 89 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.386284+0000 osd.5 (osd.5) 90 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.720910+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.721714+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.980124+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.983017+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.984396+0000 mon.a (mon.0) 1387 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:12.993699+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi087-82083-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:12.993861+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi087-82011-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:12.993937+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:12.994005+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:13.003935+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.87:0/570645369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi087-82019-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:13.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:13.006417+0000 mon.a (mon.0) 1392 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi087-82011-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:13.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:13.007195+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi087-82019-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:13.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:13.414537+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:13.836 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:32:13.836 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:11.983974+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:11.989324+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.233067+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.233953+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.385461+0000 osd.5 (osd.5) 89 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.386284+0000 osd.5 (osd.5) 90 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.720910+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.721714+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.980124+0000 osd.4 (osd.4) 91 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.983017+0000 osd.4 (osd.4) 92 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.984396+0000 mon.a (mon.0) 1387 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:12.993699+0000 mon.a (mon.0) 1388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi087-82083-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:12.993861+0000 mon.a (mon.0) 1389 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi087-82011-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:13.837 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:12.993937+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:32:13.838 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:12.994005+0000 mon.a (mon.0) 1391 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-02-23T00:32:13.838 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:13.003935+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.87:0/570645369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi087-82019-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:13.838 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:13.006417+0000 mon.a (mon.0) 1392 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi087-82011-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:13.838 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:13.007195+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi087-82019-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:13.838 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:13.414537+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:14.114 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:14.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:14.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:14.218 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:14.250 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:14.369 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:32:14.369 INFO:teuthology.orchestra.run.smithi171.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-23T00:32:14.369 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:32:14.374 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:14.376 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:14.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:14.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:14.481 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:14.509 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:14.652 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:13.100695+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:32:14.652 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:13.104165+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:32:14.652 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:13.578950+0000 mgr.y (mgr.24437) 230 : cluster [DBG] pgmap v256: 334 pgs: 72 unknown, 1 peering, 261 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:14.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:13.579941+0000 mon.a (mon.0) 1394 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:32:14.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:13.601616+0000 osd.0 (osd.0) 113 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:32:14.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:13.602446+0000 osd.0 (osd.0) 114 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:32:14.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:13.876968+0000 osd.2 (osd.2) 79 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:32:14.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:13.877932+0000 osd.2 (osd.2) 80 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:32:14.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:14.239798+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi087-82019-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:14.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:14.240168+0000 mon.a (mon.0) 1396 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:32:14.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:14.240364+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-02-23T00:32:14.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:14.415344+0000 mon.c (mon.2) 194 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:13.100695+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:13.104165+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:13.578950+0000 mgr.y (mgr.24437) 230 : cluster [DBG] pgmap v256: 334 pgs: 72 unknown, 1 peering, 261 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:13.579941+0000 mon.a (mon.0) 1394 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:13.601616+0000 osd.0 (osd.0) 113 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:13.602446+0000 osd.0 (osd.0) 114 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:13.876968+0000 osd.2 (osd.2) 79 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:13.877932+0000 osd.2 (osd.2) 80 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:14.239798+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi087-82019-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:14.240168+0000 mon.a (mon.0) 1396 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:32:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:14.240364+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:14.415344+0000 mon.c (mon.2) 194 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:13.100695+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:13.104165+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:13.578950+0000 mgr.y (mgr.24437) 230 : cluster [DBG] pgmap v256: 334 pgs: 72 unknown, 1 peering, 261 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:13.579941+0000 mon.a (mon.0) 1394 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:13.601616+0000 osd.0 (osd.0) 113 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:13.602446+0000 osd.0 (osd.0) 114 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:13.876968+0000 osd.2 (osd.2) 79 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:13.877932+0000 osd.2 (osd.2) 80 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:14.239798+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi087-82019-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:14.240168+0000 mon.a (mon.0) 1396 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:32:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:14.240364+0000 mon.a (mon.0) 1397 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-02-23T00:32:14.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:14.415344+0000 mon.c (mon.2) 194 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:15.260 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:15.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:15.363 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:32:15.392 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:15.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.769+0000 7f434d074700 1 -- 172.21.15.87:0/2775362036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43480fe510 msgr2=0x7f43480f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.769+0000 7f434d074700 1 --2- 172.21.15.87:0/2775362036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43480fe510 0x7f43480f7d40 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 rx=0x7f4338004870 tx=0x7f4338009cb0).stop 2024-02-23T00:32:15.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.769+0000 7f434d074700 1 -- 172.21.15.87:0/2775362036 shutdown_connections 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.769+0000 7f434d074700 1 --2- 172.21.15.87:0/2775362036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43480f8420 0x7f4348104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.769+0000 7f434d074700 1 --2- 172.21.15.87:0/2775362036 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43480fdbb0 0x7f43480fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.769+0000 7f434d074700 1 --2- 172.21.15.87:0/2775362036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43480fe510 0x7f43480f7d40 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.769+0000 7f434d074700 1 -- 172.21.15.87:0/2775362036 >> 172.21.15.87:0/2775362036 conn(0x7f43480f3800 msgr2=0x7f43480f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.769+0000 7f434d074700 1 -- 172.21.15.87:0/2775362036 shutdown_connections 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.769+0000 7f434d074700 1 -- 172.21.15.87:0/2775362036 wait complete. 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.770+0000 7f434d074700 1 librados: starting msgr at 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.770+0000 7f434d074700 1 librados: starting objecter 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.770+0000 7f434d074700 1 Processor -- start 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.770+0000 7f434d074700 1 -- start start 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.770+0000 7f434d074700 1 librados: setting wanted keys 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.770+0000 7f434d074700 1 librados: calling monclient init 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434d074700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43480f8420 0x7f434810e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434d074700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43480fdbb0 0x7f434810e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434d074700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43480fe510 0x7f4348108220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434d074700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4348108710 con 0x7f43480fe510 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434d074700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4348108880 con 0x7f43480fdbb0 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434d074700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4348108a40 con 0x7f43480f8420 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f4346d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43480f8420 0x7f434810e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434759e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43480fe510 0x7f4348108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f4346d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43480f8420 0x7f434810e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60062/0 (socket says 172.21.15.87:60062) 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f4346d9d700 1 -- 172.21.15.87:0/4002029457 learned_addr learned my addr 172.21.15.87:0/4002029457 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434659c700 1 --2- 172.21.15.87:0/4002029457 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43480fdbb0 0x7f434810e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434759e700 1 -- 172.21.15.87:0/4002029457 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43480f8420 msgr2=0x7f434810e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434759e700 1 --2- 172.21.15.87:0/4002029457 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43480f8420 0x7f434810e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434759e700 1 -- 172.21.15.87:0/4002029457 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43480fdbb0 msgr2=0x7f434810e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434759e700 1 --2- 172.21.15.87:0/4002029457 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43480fdbb0 0x7f434810e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434759e700 1 -- 172.21.15.87:0/4002029457 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c005040 con 0x7f43480fe510 2024-02-23T00:32:15.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f4346d9d700 1 --2- 172.21.15.87:0/4002029457 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43480f8420 0x7f434810e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f434759e700 1 --2- 172.21.15.87:0/4002029457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43480fe510 0x7f4348108220 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 rx=0x7f434000d8d0 tx=0x7f4340004db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f432ffff700 10 client.?.objecter ms_handle_connect 0x7f43480fe510 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f432ffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.771+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4340004020 con 0x7f43480fe510 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.772+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4338005040 con 0x7f43480fe510 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.772+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f434000f040 con 0x7f43480fe510 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.772+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4340003430 con 0x7f43480fe510 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.772+0000 7f434d074700 10 client.26523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.772+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4348109030 con 0x7f43480fe510 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.772+0000 7f434d074700 1 librados: init done 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.772+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4348107060 con 0x7f43480fe510 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.773+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4340004180 con 0x7f43480fe510 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.773+0000 7f432ffff700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4330062c90 0x7f4330065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(184..184 src has 38..184) v4 ==== 10313+0+0 (secure 0 0 0) 0x7f434007e050 con 0x7f43480fe510 2024-02-23T00:32:15.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 osd_map(184..184 src has 38..184) v4 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f432ffff700 3 client.26523.objecter handle_osd_map got epochs [184,184] > 0 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f4346d9d700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4330062c90 0x7f4330065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f432ffff700 3 client.26523.objecter handle_osd_map decoding full epoch 184 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f432ffff700 20 client.26523.objecter dump_active .. 0 homeless 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v184) v1 ==== 54+0+16 (secure 0 0 0) 0x7f434007e4f0 con 0x7f43480fe510 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f432ffff700 10 client.26523.objecter ms_handle_connect 0x7f4330062c90 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f4346d9d700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4330062c90 0x7f4330065150 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7f433c009810 tx=0x7f433c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 10 client.26523.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 1 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6801/2153191863 conn(0x7f43480fee20 0x7f43480ff200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348109700 osd=0 3 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348109700 osd=0 3 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348109700 osd=0 3 2024-02-23T00:32:15.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 1 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348109700 osd=0 3 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 0 1 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348109700 osd=0 4 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 10 client.26523.objecter _send_command 1 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43480ff6f0 con 0x7f43480fee20 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434659c700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6801/2153191863 conn(0x7f43480fee20 0x7f43480ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f432ffff700 10 client.26523.objecter ms_handle_connect 0x7f43480fee20 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.774+0000 7f434659c700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6801/2153191863 conn(0x7f43480fee20 0x7f43480ff200 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.775+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f43480ff6f0 con 0x7f43480fee20 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.775+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.775+0000 7f432ffff700 10 client.26523.objecter _finish_command 1 = system:0 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.775+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348109700 osd=0 4 2024-02-23T00:32:15.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.775+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 0 1 2024-02-23T00:32:15.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 10 client.26523.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 2 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348109700 osd=0 3 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348109700 osd=0 3 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348109700 osd=0 3 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 2 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348109700 osd=0 3 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 0 2 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348109700 osd=0 4 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 10 client.26523.objecter _send_command 2 2024-02-23T00:32:15.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.790+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f43480ff8b0 con 0x7f43480fee20 2024-02-23T00:32:15.792 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:14.240403+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:32:15.792 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:14.243779+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:32:15.792 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:14.647833+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:32:15.792 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:14.650822+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:32:15.792 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:14.995179+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi087-82011-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi087-82011-27"}]': finished 2024-02-23T00:32:15.792 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:14.995369+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-02-23T00:32:15.792 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:15.003743+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? 172.21.15.87:0/592950340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi087-82083-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:15.792 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:15.416436+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:15.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f43480ff8b0 con 0x7f43480fee20 2024-02-23T00:32:15.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:15.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f432ffff700 10 client.26523.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348109700 osd=0 4 2024-02-23T00:32:15.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 0 2 2024-02-23T00:32:15.793 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:32:15.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 10 client.26523.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:15.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 3 2024-02-23T00:32:15.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6805/685354171 conn(0x7f4348028540 0x7f4348118f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f43480fd460 osd=1 3 2024-02-23T00:32:15.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f43480fd460 osd=1 3 2024-02-23T00:32:15.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f43480fd460 osd=1 3 2024-02-23T00:32:15.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 3 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f43480fd460 osd=1 3 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 1 3 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f43480fd460 osd=1 4 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 10 client.26523.objecter _send_command 3 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43480fd5d0 con 0x7f4348028540 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.791+0000 7f4346d9d700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6805/685354171 conn(0x7f4348028540 0x7f4348118f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.792+0000 7f432ffff700 10 client.26523.objecter ms_handle_connect 0x7f4348028540 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.792+0000 7f4346d9d700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6805/685354171 conn(0x7f4348028540 0x7f4348118f30 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.792+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f43480fd5d0 con 0x7f4348028540 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.792+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.792+0000 7f432ffff700 10 client.26523.objecter _finish_command 3 = system:0 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.792+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f43480fd460 osd=1 4 2024-02-23T00:32:15.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.792+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 1 3 2024-02-23T00:32:15.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 10 client.26523.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:15.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 4 2024-02-23T00:32:15.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f43480fd460 osd=1 3 2024-02-23T00:32:15.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f43480fd460 osd=1 3 2024-02-23T00:32:15.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f43480fd460 osd=1 3 2024-02-23T00:32:15.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 4 2024-02-23T00:32:15.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f43480fd460 osd=1 3 2024-02-23T00:32:15.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 1 4 2024-02-23T00:32:15.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f43480fd460 osd=1 4 2024-02-23T00:32:15.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 10 client.26523.objecter _send_command 4 2024-02-23T00:32:15.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.807+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f434805a040 con 0x7f4348028540 2024-02-23T00:32:15.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f434805a040 con 0x7f4348028540 2024-02-23T00:32:15.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:15.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f432ffff700 10 client.26523.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f43480fd460 osd=1 4 2024-02-23T00:32:15.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 1 4 2024-02-23T00:32:15.810 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:32:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f434d074700 10 client.26523.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 5 2024-02-23T00:32:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43480594d0 0x7f434806c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348059360 osd=2 3 2024-02-23T00:32:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.808+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348059360 osd=2 3 2024-02-23T00:32:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348059360 osd=2 3 2024-02-23T00:32:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 5 2024-02-23T00:32:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348059360 osd=2 3 2024-02-23T00:32:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 2 5 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348059360 osd=2 4 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f434d074700 10 client.26523.objecter _send_command 5 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43480280a0 con 0x7f43480594d0 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f434659c700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43480594d0 0x7f434806c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f432ffff700 10 client.26523.objecter ms_handle_connect 0x7f43480594d0 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f434659c700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43480594d0 0x7f434806c090 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f43480280a0 con 0x7f43480594d0 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f432ffff700 10 client.26523.objecter _finish_command 5 = system:0 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348059360 osd=2 4 2024-02-23T00:32:15.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.809+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 2 5 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 10 client.26523.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 6 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348059360 osd=2 3 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348059360 osd=2 3 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348059360 osd=2 3 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 6 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348059360 osd=2 3 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 2 6 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348059360 osd=2 4 2024-02-23T00:32:15.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 10 client.26523.objecter _send_command 6 2024-02-23T00:32:15.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.824+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4348028320 con 0x7f43480594d0 2024-02-23T00:32:15.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4348028320 con 0x7f43480594d0 2024-02-23T00:32:15.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:15.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f432ffff700 10 client.26523.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348059360 osd=2 4 2024-02-23T00:32:15.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 2 6 2024-02-23T00:32:15.827 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 10 client.26523.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 7 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6813/128059790 conn(0x7f434811fd50 0x7f4348122140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348027840 osd=3 3 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348027840 osd=3 3 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348027840 osd=3 3 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 7 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348027840 osd=3 3 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 3 7 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348027840 osd=3 4 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 10 client.26523.objecter _send_command 7 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.825+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f434806cd50 con 0x7f434811fd50 2024-02-23T00:32:15.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.826+0000 7f434759e700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6813/128059790 conn(0x7f434811fd50 0x7f4348122140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.826+0000 7f432ffff700 10 client.26523.objecter ms_handle_connect 0x7f434811fd50 2024-02-23T00:32:15.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.826+0000 7f434759e700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6813/128059790 conn(0x7f434811fd50 0x7f4348122140 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.826+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f434806cd50 con 0x7f434811fd50 2024-02-23T00:32:15.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.826+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:32:15.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.826+0000 7f432ffff700 10 client.26523.objecter _finish_command 7 = system:0 2024-02-23T00:32:15.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.826+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348027840 osd=3 4 2024-02-23T00:32:15.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.826+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 3 7 2024-02-23T00:32:15.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:14.240403+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:32:15.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:14.243779+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:14.647833+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:14.650822+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:14.995179+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi087-82011-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi087-82011-27"}]': finished 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:14.995369+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:15.003743+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? 172.21.15.87:0/592950340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi087-82083-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:15.416436+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:14.240403+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:14.243779+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:14.647833+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:14.650822+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:14.995179+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi087-82011-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi087-82011-27"}]': finished 2024-02-23T00:32:15.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:14.995369+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-02-23T00:32:15.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:15.003743+0000 mon.a (mon.0) 1400 : audit [INF] from='client.? 172.21.15.87:0/592950340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi087-82083-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:15.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:15.416436+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:15.841 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:32:15.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 10 client.26523.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:15.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 8 2024-02-23T00:32:15.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348027840 osd=3 3 2024-02-23T00:32:15.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348027840 osd=3 3 2024-02-23T00:32:15.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348027840 osd=3 3 2024-02-23T00:32:15.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 8 2024-02-23T00:32:15.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348027840 osd=3 3 2024-02-23T00:32:15.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 3 8 2024-02-23T00:32:15.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348027840 osd=3 4 2024-02-23T00:32:15.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 10 client.26523.objecter _send_command 8 2024-02-23T00:32:15.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.841+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f434806cf70 con 0x7f434811fd50 2024-02-23T00:32:15.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.842+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f434806cf70 con 0x7f434811fd50 2024-02-23T00:32:15.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.842+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:15.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.842+0000 7f432ffff700 10 client.26523.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.842+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348027840 osd=3 4 2024-02-23T00:32:15.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.842+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 3 8 2024-02-23T00:32:15.844 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:32:15.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 10 client.26523.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:15.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 9 2024-02-23T00:32:15.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6800/4175844757 conn(0x7f434806b6e0 0x7f434806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f434806cf70 osd=4 3 2024-02-23T00:32:15.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f434806cf70 osd=4 3 2024-02-23T00:32:15.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f434806cf70 osd=4 3 2024-02-23T00:32:15.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 9 2024-02-23T00:32:15.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f434806cf70 osd=4 3 2024-02-23T00:32:15.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 4 9 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f434806cf70 osd=4 4 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 10 client.26523.objecter _send_command 9 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4348127fc0 con 0x7f434806b6e0 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f4346d9d700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6800/4175844757 conn(0x7f434806b6e0 0x7f434806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f4346d9d700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6800/4175844757 conn(0x7f434806b6e0 0x7f434806bb40 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.843+0000 7f432ffff700 10 client.26523.objecter ms_handle_connect 0x7f434806b6e0 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.844+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4348127fc0 con 0x7f434806b6e0 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.844+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.844+0000 7f432ffff700 10 client.26523.objecter _finish_command 9 = system:0 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.844+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f434806cf70 osd=4 4 2024-02-23T00:32:15.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.844+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 4 9 2024-02-23T00:32:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:15.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:32:15.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 10 client.26523.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 10 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f434806cf70 osd=4 3 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f434806cf70 osd=4 3 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f434806cf70 osd=4 3 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 10 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f434806cf70 osd=4 3 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 4 10 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f434806cf70 osd=4 4 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 10 client.26523.objecter _send_command 10 2024-02-23T00:32:15.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.859+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4348125860 con 0x7f434806b6e0 2024-02-23T00:32:15.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4348125860 con 0x7f434806b6e0 2024-02-23T00:32:15.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:15.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f432ffff700 10 client.26523.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f434806cf70 osd=4 4 2024-02-23T00:32:15.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 4 10 2024-02-23T00:32:15.862 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 10 client.26523.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 11 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4348127580 0x7f43481279c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348125860 osd=5 3 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125860 osd=5 3 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348125860 osd=5 3 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 11 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348125860 osd=5 3 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 5 11 2024-02-23T00:32:15.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125860 osd=5 4 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 10 client.26523.objecter _send_command 11 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.860+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f434812b2f0 con 0x7f4348127580 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.861+0000 7f434659c700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4348127580 0x7f43481279c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.861+0000 7f434659c700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4348127580 0x7f43481279c0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.861+0000 7f432ffff700 10 client.26523.objecter ms_handle_connect 0x7f4348127580 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.862+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f434812b2f0 con 0x7f4348127580 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.862+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.862+0000 7f432ffff700 10 client.26523.objecter _finish_command 11 = system:0 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.862+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348125860 osd=5 4 2024-02-23T00:32:15.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.862+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 5 11 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 10 client.26523.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 12 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348125860 osd=5 3 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125860 osd=5 3 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348125860 osd=5 3 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 12 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348125860 osd=5 3 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 5 12 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125860 osd=5 4 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 10 client.26523.objecter _send_command 12 2024-02-23T00:32:15.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.876+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4348125a60 con 0x7f4348127580 2024-02-23T00:32:15.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4348125a60 con 0x7f4348127580 2024-02-23T00:32:15.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:15.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f432ffff700 10 client.26523.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348125860 osd=5 4 2024-02-23T00:32:15.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 5 12 2024-02-23T00:32:15.879 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:32:15.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 10 client.26523.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:15.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 13 2024-02-23T00:32:15.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6808/2509408498 conn(0x7f43481260e0 0x7f4348126520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348125a60 osd=6 3 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125a60 osd=6 3 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348125a60 osd=6 3 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 13 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348125a60 osd=6 3 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 6 13 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125a60 osd=6 4 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 10 client.26523.objecter _send_command 13 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4348126b40 con 0x7f43481260e0 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.878+0000 7f434759e700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6808/2509408498 conn(0x7f43481260e0 0x7f4348126520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.879+0000 7f432ffff700 10 client.26523.objecter ms_handle_connect 0x7f43481260e0 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.879+0000 7f434759e700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6808/2509408498 conn(0x7f43481260e0 0x7f4348126520 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.879+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4348126b40 con 0x7f43481260e0 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.879+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.879+0000 7f432ffff700 10 client.26523.objecter _finish_command 13 = system:0 2024-02-23T00:32:15.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.879+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348125a60 osd=6 4 2024-02-23T00:32:15.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.879+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 6 13 2024-02-23T00:32:15.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 10 client.26523.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:15.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 14 2024-02-23T00:32:15.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348125a60 osd=6 3 2024-02-23T00:32:15.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125a60 osd=6 3 2024-02-23T00:32:15.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348125a60 osd=6 3 2024-02-23T00:32:15.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 14 2024-02-23T00:32:15.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348125a60 osd=6 3 2024-02-23T00:32:15.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 6 14 2024-02-23T00:32:15.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125a60 osd=6 4 2024-02-23T00:32:15.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 10 client.26523.objecter _send_command 14 2024-02-23T00:32:15.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.893+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4348126b40 con 0x7f43481260e0 2024-02-23T00:32:15.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.895+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4348126b40 con 0x7f43481260e0 2024-02-23T00:32:15.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.895+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:15.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.895+0000 7f432ffff700 10 client.26523.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.895+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348125a60 osd=6 4 2024-02-23T00:32:15.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.895+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 6 14 2024-02-23T00:32:15.897 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:32:15.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 10 client.26523.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:15.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 15 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6812/373787000 conn(0x7f434812c690 0x7f434812ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348126b40 osd=7 3 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348126b40 osd=7 3 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348126b40 osd=7 3 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 15 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348126b40 osd=7 3 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 7 15 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348126b40 osd=7 4 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 10 client.26523.objecter _send_command 15 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43481316d0 con 0x7f434812c690 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f4346d9d700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6812/373787000 conn(0x7f434812c690 0x7f434812ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f432ffff700 10 client.26523.objecter ms_handle_connect 0x7f434812c690 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.896+0000 7f4346d9d700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6812/373787000 conn(0x7f434812c690 0x7f434812ea80 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.897+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f43481316d0 con 0x7f434812c690 2024-02-23T00:32:15.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.897+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:32:15.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.897+0000 7f432ffff700 10 client.26523.objecter _finish_command 15 = system:0 2024-02-23T00:32:15.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.897+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348126b40 osd=7 4 2024-02-23T00:32:15.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.897+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 7 15 2024-02-23T00:32:15.901 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 10 client.26523.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign -1 16 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348126b40 osd=7 3 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348126b40 osd=7 3 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 20 client.26523.objecter _get_session s=0x7f4348126b40 osd=7 3 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 15 client.26523.objecter _session_command_op_remove -1 16 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 20 client.26523.objecter get_session s=0x7f4348126b40 osd=7 3 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 15 client.26523.objecter _session_command_op_assign 7 16 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348126b40 osd=7 4 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 10 client.26523.objecter _send_command 16 2024-02-23T00:32:15.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.913+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f434812ef70 con 0x7f434812c690 2024-02-23T00:32:15.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f432ffff700 1 -- 172.21.15.87:0/4002029457 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f434812ef70 con 0x7f434812c690 2024-02-23T00:32:15.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f432ffff700 10 client.26523.objecter ms_dispatch 0x7f4348001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:15.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f432ffff700 10 client.26523.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f432ffff700 20 client.26523.objecter put_session s=0x7f4348126b40 osd=7 4 2024-02-23T00:32:15.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f432ffff700 15 client.26523.objecter _session_command_op_remove 7 16 2024-02-23T00:32:15.916 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:32:15.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 librados: watch_flush enter 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 librados: watch_flush exit 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 client.26523.objecter close_session for osd.0 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6801/2153191863 conn(0x7f43480fee20 msgr2=0x7f43480ff200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6801/2153191863 conn(0x7f43480fee20 0x7f43480ff200 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348109700 osd=0 1 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 client.26523.objecter close_session for osd.1 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6805/685354171 conn(0x7f4348028540 msgr2=0x7f4348118f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6805/685354171 conn(0x7f4348028540 0x7f4348118f30 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f43480fd460 osd=1 1 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 client.26523.objecter close_session for osd.2 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43480594d0 msgr2=0x7f434806c090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43480594d0 0x7f434806c090 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348059360 osd=2 1 2024-02-23T00:32:15.918 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 client.26523.objecter close_session for osd.3 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6813/128059790 conn(0x7f434811fd50 msgr2=0x7f4348122140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6813/128059790 conn(0x7f434811fd50 0x7f4348122140 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434759e700 1 -- 172.21.15.87:0/4002029457 reap_dead start 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348027840 osd=3 1 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 client.26523.objecter close_session for osd.4 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6800/4175844757 conn(0x7f434806b6e0 msgr2=0x7f434806bb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6800/4175844757 conn(0x7f434806b6e0 0x7f434806bb40 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434759e700 1 -- 172.21.15.87:0/4002029457 reap_dead start 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f434806cf70 osd=4 1 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 client.26523.objecter close_session for osd.5 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4348127580 msgr2=0x7f43481279c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4348127580 0x7f43481279c0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125860 osd=5 1 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 client.26523.objecter close_session for osd.6 2024-02-23T00:32:15.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6808/2509408498 conn(0x7f43481260e0 msgr2=0x7f4348126520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6808/2509408498 conn(0x7f43481260e0 0x7f4348126520 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348125a60 osd=6 1 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 client.26523.objecter close_session for osd.7 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6812/373787000 conn(0x7f434812c690 msgr2=0x7f434812ea80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.171:6812/373787000 conn(0x7f434812c690 0x7f434812ea80 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 20 client.26523.objecter put_session s=0x7f4348126b40 osd=7 1 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 20 client.26523.objecter shutdown clearing up homeless session... 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 10 client.26523.objecter successfully canceled tick 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4330062c90 msgr2=0x7f4330065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4330062c90 0x7f4330065150 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7f433c009810 tx=0x7f433c00b040).stop 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43480fe510 msgr2=0x7f4348108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 --2- 172.21.15.87:0/4002029457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43480fe510 0x7f4348108220 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 rx=0x7f434000d8d0 tx=0x7f4340004db0).stop 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434759e700 1 -- 172.21.15.87:0/4002029457 reap_dead start 2024-02-23T00:32:15.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 shutdown_connections 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.915+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 >> 172.21.15.87:0/4002029457 conn(0x7f43480f3800 msgr2=0x7f43480fb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.916+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 shutdown_connections 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.916+0000 7f434d074700 1 -- 172.21.15.87:0/4002029457 wait complete. 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:15.916+0000 7f434d074700 1 librados: shutdown 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.921 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:32:15.948 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:15.976 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:16.175 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:16.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:16.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:16.280 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:16.308 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:16.344 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:16.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:16.452 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:32:16.479 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:15.088005+0000 osd.5 (osd.5) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:15.088107+0000 osd.5 (osd.5) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:15.580310+0000 mgr.y (mgr.24437) 231 : cluster [DBG] pgmap v259: 310 pgs: 40 creating+peering, 16 active+clean+snaptrim, 56 active+clean+snaptrim_wait, 2 clean+premerge+peered, 196 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-23T00:32:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:15.648756+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:32:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:15.651642+0000 osd.7 (osd.7) 96 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:15.773560+0000 mon.a (mon.0) 1401 : audit [DBG] from='client.? 172.21.15.87:0/4002029457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:15.854487+0000 osd.2 (osd.2) 81 : cluster [DBG] 6.f deep-scrub starts 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:15.855576+0000 osd.2 (osd.2) 82 : cluster [DBG] 6.f deep-scrub ok 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:15.997647+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? 172.21.15.87:0/592950340' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi087-82083-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:15.997864+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:16.005624+0000 mon.c (mon.2) 196 : audit [INF] from='client.? 172.21.15.87:0/3144085313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi087-82019-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:16.008077+0000 mon.a (mon.0) 1404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi087-82019-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:16.096832+0000 mon.b (mon.1) 163 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:16.097092+0000 mon.a (mon.0) 1405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:16.417332+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:15.088005+0000 osd.5 (osd.5) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:15.088107+0000 osd.5 (osd.5) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:15.580310+0000 mgr.y (mgr.24437) 231 : cluster [DBG] pgmap v259: 310 pgs: 40 creating+peering, 16 active+clean+snaptrim, 56 active+clean+snaptrim_wait, 2 clean+premerge+peered, 196 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:15.648756+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:15.651642+0000 osd.7 (osd.7) 96 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:15.773560+0000 mon.a (mon.0) 1401 : audit [DBG] from='client.? 172.21.15.87:0/4002029457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:15.854487+0000 osd.2 (osd.2) 81 : cluster [DBG] 6.f deep-scrub starts 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:15.855576+0000 osd.2 (osd.2) 82 : cluster [DBG] 6.f deep-scrub ok 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:15.997647+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? 172.21.15.87:0/592950340' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi087-82083-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:15.997864+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:16.005624+0000 mon.c (mon.2) 196 : audit [INF] from='client.? 172.21.15.87:0/3144085313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi087-82019-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:16.008077+0000 mon.a (mon.0) 1404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi087-82019-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:16.096832+0000 mon.b (mon.1) 163 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:16.097092+0000 mon.a (mon.0) 1405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:16.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:16.417332+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:16.628 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:15.088005+0000 osd.5 (osd.5) 91 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:15.088107+0000 osd.5 (osd.5) 92 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:15.580310+0000 mgr.y (mgr.24437) 231 : cluster [DBG] pgmap v259: 310 pgs: 40 creating+peering, 16 active+clean+snaptrim, 56 active+clean+snaptrim_wait, 2 clean+premerge+peered, 196 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:15.648756+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:15.651642+0000 osd.7 (osd.7) 96 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:15.773560+0000 mon.a (mon.0) 1401 : audit [DBG] from='client.? 172.21.15.87:0/4002029457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:15.854487+0000 osd.2 (osd.2) 81 : cluster [DBG] 6.f deep-scrub starts 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:15.855576+0000 osd.2 (osd.2) 82 : cluster [DBG] 6.f deep-scrub ok 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:15.997647+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? 172.21.15.87:0/592950340' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi087-82083-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:16.630 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:15.997864+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-02-23T00:32:16.631 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:16.005624+0000 mon.c (mon.2) 196 : audit [INF] from='client.? 172.21.15.87:0/3144085313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi087-82019-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:16.631 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:16.008077+0000 mon.a (mon.0) 1404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi087-82019-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:16.631 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:16.096832+0000 mon.b (mon.1) 163 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:16.631 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:16.097092+0000 mon.a (mon.0) 1405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:16.631 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:16.417332+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:16.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:16.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:32:16.732 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:16.760 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:16.769 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:16.797 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:16.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:16.901 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:16.922 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:32:16.922 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:32:16.927 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:17.050 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:17.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:17.154 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:17.180 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:17.629 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:16.520475+0000 mon.a (mon.0) 1406 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:16.827626+0000 osd.2 (osd.2) 83 : cluster [DBG] 6.14 deep-scrub starts 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:16.828486+0000 osd.2 (osd.2) 84 : cluster [DBG] 6.14 deep-scrub ok 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.000471+0000 mon.a (mon.0) 1407 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi087-82019-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.000649+0000 mon.a (mon.0) 1408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:17.000715+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.004445+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.005903+0000 mon.b (mon.1) 164 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.014139+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.045235+0000 mon.c (mon.2) 198 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.045533+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.632 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.046579+0000 mon.c (mon.2) 199 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.632 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.046909+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.632 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.047331+0000 mon.c (mon.2) 200 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:17.632 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.047602+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:17.632 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.418446+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:17.632 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:16.520475+0000 mon.a (mon.0) 1406 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:17.632 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:16.827626+0000 osd.2 (osd.2) 83 : cluster [DBG] 6.14 deep-scrub starts 2024-02-23T00:32:17.632 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:16.828486+0000 osd.2 (osd.2) 84 : cluster [DBG] 6.14 deep-scrub ok 2024-02-23T00:32:17.632 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.000471+0000 mon.a (mon.0) 1407 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi087-82019-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.000649+0000 mon.a (mon.0) 1408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:17.000715+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.004445+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.005903+0000 mon.b (mon.1) 164 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.014139+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.045235+0000 mon.c (mon.2) 198 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.045533+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.046579+0000 mon.c (mon.2) 199 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.046909+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.047331+0000 mon.c (mon.2) 200 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.047602+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:17.633 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.418446+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:17.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:16.520475+0000 mon.a (mon.0) 1406 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:16.827626+0000 osd.2 (osd.2) 83 : cluster [DBG] 6.14 deep-scrub starts 2024-02-23T00:32:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:16.828486+0000 osd.2 (osd.2) 84 : cluster [DBG] 6.14 deep-scrub ok 2024-02-23T00:32:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.000471+0000 mon.a (mon.0) 1407 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi087-82019-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.000649+0000 mon.a (mon.0) 1408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:17.000715+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-02-23T00:32:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.004445+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.005903+0000 mon.b (mon.1) 164 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:32:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.014139+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13"}]: dispatch 2024-02-23T00:32:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.045235+0000 mon.c (mon.2) 198 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.045533+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.046579+0000 mon.c (mon.2) 199 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.046909+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.047331+0000 mon.c (mon.2) 200 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.047602+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.418446+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:17.715 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-23T00:32:17.736 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:17.982 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:18.111 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:18.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:18.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:18.243 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:18.442 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:18.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:18.546 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:18.577 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.660+0000 7ff41537e700 1 -- 172.21.15.87:0/2049951056 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4100f6a10 msgr2=0x7ff4100f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:18.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.660+0000 7ff41537e700 1 --2- 172.21.15.87:0/2049951056 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4100f6a10 0x7ff4100f6df0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7ff4000048a0 tx=0x7ff400009cb0).stop 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.660+0000 7ff41537e700 1 -- 172.21.15.87:0/2049951056 shutdown_connections 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.660+0000 7ff41537e700 1 --2- 172.21.15.87:0/2049951056 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff410104570 0x7ff410106960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.660+0000 7ff41537e700 1 --2- 172.21.15.87:0/2049951056 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4100f6a10 0x7ff4100f6df0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.660+0000 7ff41537e700 1 --2- 172.21.15.87:0/2049951056 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4100f72e0 0x7ff410104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.661+0000 7ff41537e700 1 -- 172.21.15.87:0/2049951056 >> 172.21.15.87:0/2049951056 conn(0x7ff4100f26f0 msgr2=0x7ff4100f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.661+0000 7ff41537e700 1 -- 172.21.15.87:0/2049951056 shutdown_connections 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.661+0000 7ff41537e700 1 -- 172.21.15.87:0/2049951056 wait complete. 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.661+0000 7ff41537e700 1 librados: starting msgr at 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.661+0000 7ff41537e700 1 librados: starting objecter 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.661+0000 7ff41537e700 1 Processor -- start 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.661+0000 7ff41537e700 1 -- start start 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.661+0000 7ff41537e700 1 librados: setting wanted keys 2024-02-23T00:32:18.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.661+0000 7ff41537e700 1 librados: calling monclient init 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4100f6a10 0x7ff4100f8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4100f72e0 0x7ff4100f92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff410104570 0x7ff4100f97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100fceb0 con 0x7ff4100f72e0 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41011ab50 con 0x7ff4100f6a10 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41011ad10 con 0x7ff410104570 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40e7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4100f72e0 0x7ff4100f92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40f7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff410104570 0x7ff4100f97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40f7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff410104570 0x7ff4100f97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60108/0 (socket says 172.21.15.87:60108) 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40f7fe700 1 -- 172.21.15.87:0/1281780120 learned_addr learned my addr 172.21.15.87:0/1281780120 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40effd700 1 --2- 172.21.15.87:0/1281780120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4100f6a10 0x7ff4100f8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40f7fe700 1 -- 172.21.15.87:0/1281780120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4100f6a10 msgr2=0x7ff4100f8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40f7fe700 1 --2- 172.21.15.87:0/1281780120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4100f6a10 0x7ff4100f8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:18.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40f7fe700 1 -- 172.21.15.87:0/1281780120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4100f72e0 msgr2=0x7ff4100f92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40f7fe700 1 --2- 172.21.15.87:0/1281780120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4100f72e0 0x7ff4100f92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40f7fe700 1 -- 172.21.15.87:0/1281780120 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff404005040 con 0x7ff410104570 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff40f7fe700 1 --2- 172.21.15.87:0/1281780120 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff410104570 0x7ff4100f97c0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7ff40800d970 tx=0x7ff408004e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff3f7fff700 10 client.?.objecter ms_handle_connect 0x7ff410104570 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff3f7fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff3f7fff700 1 -- 172.21.15.87:0/1281780120 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff408004020 con 0x7ff410104570 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff400005040 con 0x7ff410104570 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff3f7fff700 1 -- 172.21.15.87:0/1281780120 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff40800f040 con 0x7ff410104570 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff3f7fff700 1 -- 172.21.15.87:0/1281780120 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4080036c0 con 0x7ff410104570 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 10 client.36998.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41011b3a0 con 0x7ff410104570 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.662+0000 7ff41537e700 1 librados: init done 2024-02-23T00:32:18.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.663+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41005f990 con 0x7ff410104570 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.664+0000 7ff3f7fff700 1 -- 172.21.15.87:0/1281780120 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff40800b510 con 0x7ff410104570 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.664+0000 7ff3f7fff700 1 --2- 172.21.15.87:0/1281780120 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3f8062c90 0x7ff3f8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.665+0000 7ff3f7fff700 1 -- 172.21.15.87:0/1281780120 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(187..187 src has 38..187) v4 ==== 9427+0+0 (secure 0 0 0) 0x7ff40807d880 con 0x7ff410104570 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.665+0000 7ff3f7fff700 10 client.36998.objecter ms_dispatch 0x7ff410001550 osd_map(187..187 src has 38..187) v4 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.665+0000 7ff3f7fff700 3 client.36998.objecter handle_osd_map got epochs [187,187] > 0 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.665+0000 7ff3f7fff700 3 client.36998.objecter handle_osd_map decoding full epoch 187 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.665+0000 7ff40effd700 1 --2- 172.21.15.87:0/1281780120 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3f8062c90 0x7ff3f8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.665+0000 7ff3f7fff700 20 client.36998.objecter dump_active .. 0 homeless 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.665+0000 7ff3f7fff700 10 client.36998.objecter ms_handle_connect 0x7ff3f8062c90 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.665+0000 7ff40effd700 1 --2- 172.21.15.87:0/1281780120 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3f8062c90 0x7ff3f8065150 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 rx=0x7ff400004710 tx=0x7ff400007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:18.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.668+0000 7ff3f7fff700 1 -- 172.21.15.87:0/1281780120 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff408050870 con 0x7ff410104570 2024-02-23T00:32:18.701 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:32:18.701 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:18.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:17.580802+0000 mgr.y (mgr.24437) 232 : cluster [DBG] pgmap v262: 301 pgs: 16 active+clean+snaptrim, 32 unknown, 56 active+clean+snaptrim_wait, 1 clean+premerge+peered, 196 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1 op/s 2024-02-23T00:32:18.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:17.581838+0000 mon.a (mon.0) 1415 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-23T00:32:18.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:17.589170+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:32:18.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:17.589946+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:32:18.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:17.980140+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:32:18.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:17.981258+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:32:18.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:17.996051+0000 mon.a (mon.0) 1416 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:18.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.000270+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi087-82011-27"}]': finished 2024-02-23T00:32:18.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.000350+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13"}]': finished 2024-02-23T00:32:18.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.000422+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:18.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.000527+0000 mon.a (mon.0) 1420 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-23T00:32:18.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:18.000574+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-02-23T00:32:18.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.003606+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:18.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.007135+0000 mon.c (mon.2) 202 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi087-82083-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:18.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.007913+0000 mon.a (mon.0) 1423 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi087-82083-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:18.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.419479+0000 mon.c (mon.2) 203 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:17.580802+0000 mgr.y (mgr.24437) 232 : cluster [DBG] pgmap v262: 301 pgs: 16 active+clean+snaptrim, 32 unknown, 56 active+clean+snaptrim_wait, 1 clean+premerge+peered, 196 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1 op/s 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:17.581838+0000 mon.a (mon.0) 1415 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:17.589170+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:17.589946+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:17.980140+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:17.981258+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:17.996051+0000 mon.a (mon.0) 1416 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.000270+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi087-82011-27"}]': finished 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.000350+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13"}]': finished 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.000422+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.000527+0000 mon.a (mon.0) 1420 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:18.000574+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-02-23T00:32:18.704 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.003606+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:18.705 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.007135+0000 mon.c (mon.2) 202 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi087-82083-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:18.705 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.007913+0000 mon.a (mon.0) 1423 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi087-82083-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:18.705 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.419479+0000 mon.c (mon.2) 203 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:18.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:18.807 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:18.834 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:18.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:18.852+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff41005ee80 con 0x7ff410104570 2024-02-23T00:32:18.858 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:18.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:18.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:17.580802+0000 mgr.y (mgr.24437) 232 : cluster [DBG] pgmap v262: 301 pgs: 16 active+clean+snaptrim, 32 unknown, 56 active+clean+snaptrim_wait, 1 clean+premerge+peered, 196 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 1 op/s 2024-02-23T00:32:18.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:17.581838+0000 mon.a (mon.0) 1415 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-23T00:32:18.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:17.589170+0000 osd.7 (osd.7) 97 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:32:18.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:17.589946+0000 osd.7 (osd.7) 98 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:17.980140+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:17.981258+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:17.996051+0000 mon.a (mon.0) 1416 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.000270+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi087-82011-27"}]': finished 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.000350+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-13"}]': finished 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.000422+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.000527+0000 mon.a (mon.0) 1420 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:18.000574+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.003606+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi087-82011-27"}]: dispatch 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.007135+0000 mon.c (mon.2) 202 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi087-82083-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.007913+0000 mon.a (mon.0) 1423 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi087-82083-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.419479+0000 mon.c (mon.2) 203 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:18.962 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:18.979 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:18.990 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:19.027 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:19.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:19.131 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:19.160 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:19.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.252+0000 7ff3f7fff700 1 -- 172.21.15.87:0/1281780120 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v188) v1 ==== 87+0+0 (secure 0 0 0) 0x7ff408021070 con 0x7ff410104570 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.254+0000 7ff41537e700 10 librados: watch_flush enter 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 10 librados: watch_flush exit 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 20 client.36998.objecter shutdown clearing up homeless session... 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 10 client.36998.objecter successfully canceled tick 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3f8062c90 msgr2=0x7ff3f8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 --2- 172.21.15.87:0/1281780120 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3f8062c90 0x7ff3f8065150 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 rx=0x7ff400004710 tx=0x7ff400007f90).stop 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff410104570 msgr2=0x7ff4100f97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 --2- 172.21.15.87:0/1281780120 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff410104570 0x7ff4100f97c0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7ff40800d970 tx=0x7ff408004e40).stop 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 shutdown_connections 2024-02-23T00:32:19.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 --2- 172.21.15.87:0/1281780120 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3f8062c90 0x7ff3f8065150 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:19.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 --2- 172.21.15.87:0/1281780120 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff410104570 0x7ff4100f97c0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:19.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 --2- 172.21.15.87:0/1281780120 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4100f6a10 0x7ff4100f8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:19.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 --2- 172.21.15.87:0/1281780120 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4100f72e0 0x7ff4100f92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:19.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 >> 172.21.15.87:0/1281780120 conn(0x7ff4100f26f0 msgr2=0x7ff4100f4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:19.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 shutdown_connections 2024-02-23T00:32:19.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 -- 172.21.15.87:0/1281780120 wait complete. 2024-02-23T00:32:19.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:19.255+0000 7ff41537e700 1 librados: shutdown 2024-02-23T00:32:19.257 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:32:19.508 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:19.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:19.612 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:19.639 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:19.731 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:18.603434+0000 osd.0 (osd.0) 115 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:32:19.731 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:18.606907+0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:32:19.731 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.852701+0000 mon.c (mon.2) 204 : audit [INF] from='client.? 172.21.15.87:0/1281780120' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:18.853132+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:18.998072+0000 mon.a (mon.0) 1425 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.002665+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi087-82011-27"}]': finished 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.248451+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:19.248541+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.249364+0000 mon.c (mon.2) 205 : audit [INF] from='client.? 172.21.15.87:0/3427795088' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi087-82019-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.272227+0000 mon.c (mon.2) 206 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.420331+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.494655+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi087-82019-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.494895+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.495450+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.495681+0000 mon.a (mon.0) 1431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.732 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.496053+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi087-82011-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:19.733 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:19.496281+0000 mon.a (mon.0) 1432 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi087-82011-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:19.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:19.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:18.603434+0000 osd.0 (osd.0) 115 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:32:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:18.606907+0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:32:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.852701+0000 mon.c (mon.2) 204 : audit [INF] from='client.? 172.21.15.87:0/1281780120' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:32:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:18.853132+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:32:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:18.998072+0000 mon.a (mon.0) 1425 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.002665+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi087-82011-27"}]': finished 2024-02-23T00:32:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.248451+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:32:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:19.248541+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-02-23T00:32:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.249364+0000 mon.c (mon.2) 205 : audit [INF] from='client.? 172.21.15.87:0/3427795088' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi087-82019-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.272227+0000 mon.c (mon.2) 206 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.420331+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.494655+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi087-82019-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.494895+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.495450+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.495681+0000 mon.a (mon.0) 1431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.496053+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi087-82011-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:19.496281+0000 mon.a (mon.0) 1432 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi087-82011-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:18.603434+0000 osd.0 (osd.0) 115 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:18.606907+0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.852701+0000 mon.c (mon.2) 204 : audit [INF] from='client.? 172.21.15.87:0/1281780120' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:18.853132+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:18.998072+0000 mon.a (mon.0) 1425 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.002665+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? 172.21.15.87:0/3439726987' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi087-82011-27"}]': finished 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.248451+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:19.248541+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.249364+0000 mon.c (mon.2) 205 : audit [INF] from='client.? 172.21.15.87:0/3427795088' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi087-82019-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.272227+0000 mon.c (mon.2) 206 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.420331+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.494655+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi087-82019-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:19.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.494895+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.495450+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.495681+0000 mon.a (mon.0) 1431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:19.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.496053+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi087-82011-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:19.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:19.496281+0000 mon.a (mon.0) 1432 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi087-82011-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:19.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:19.870 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:19.993 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:19.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:20.031 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:32:20.032 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:32:20.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:20.124 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:20.273 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:20.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:20.378 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:20.408 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:20.808 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:32:20.809 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:32:20.809 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-02-23T00:32:20.809 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-23T00:32:20.809 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-23T00:32:20.809 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-02-23T00:32:21.007 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:32:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:19.581358+0000 mgr.y (mgr.24437) 233 : cluster [DBG] pgmap v265: 269 pgs: 12 active+clean+snaptrim, 32 unknown, 28 active+clean+snaptrim_wait, 1 clean+premerge+peered, 196 active+clean; 28 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:20.009222+0000 mon.a (mon.0) 1433 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi087-82083-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi087-82083-36"}]': finished 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:20.009488+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi087-82019-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:20.009610+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi087-82011-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:20.009681+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:20.015912+0000 mon.c (mon.2) 210 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi087-82011-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:20.019320+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi087-82011-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:20.025568+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:20.027784+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:21.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:20.421557+0000 mon.c (mon.2) 211 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:19.581358+0000 mgr.y (mgr.24437) 233 : cluster [DBG] pgmap v265: 269 pgs: 12 active+clean+snaptrim, 32 unknown, 28 active+clean+snaptrim_wait, 1 clean+premerge+peered, 196 active+clean; 28 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:20.009222+0000 mon.a (mon.0) 1433 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi087-82083-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi087-82083-36"}]': finished 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:20.009488+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi087-82019-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:20.009610+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi087-82011-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:20.009681+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:20.015912+0000 mon.c (mon.2) 210 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi087-82011-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:20.019320+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi087-82011-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:20.025568+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:20.027784+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:21.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:20.421557+0000 mon.c (mon.2) 211 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:21.127 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-23T00:32:21.157 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:21.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:21.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:21.262 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:21.292 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:21.424 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:19.581358+0000 mgr.y (mgr.24437) 233 : cluster [DBG] pgmap v265: 269 pgs: 12 active+clean+snaptrim, 32 unknown, 28 active+clean+snaptrim_wait, 1 clean+premerge+peered, 196 active+clean; 28 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:20.009222+0000 mon.a (mon.0) 1433 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi087-82083-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi087-82083-36"}]': finished 2024-02-23T00:32:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:20.009488+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi087-82019-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:20.009610+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi087-82011-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:20.009681+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-02-23T00:32:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:20.015912+0000 mon.c (mon.2) 210 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi087-82011-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:20.019320+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi087-82011-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:20.025568+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:20.027784+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:20.421557+0000 mon.c (mon.2) 211 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:21.444 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:21.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 -- 172.21.15.87:0/523074929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4a0c0fc240 msgr2=0x7f4a0c0fc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 --2- 172.21.15.87:0/523074929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4a0c0fc240 0x7f4a0c0fc620 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 rx=0x7f4a08004870 tx=0x7f4a08009cb0).stop 2024-02-23T00:32:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 -- 172.21.15.87:0/523074929 shutdown_connections 2024-02-23T00:32:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 --2- 172.21.15.87:0/523074929 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4a0c0f6b10 0x7f4a0c1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 --2- 172.21.15.87:0/523074929 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4a0c0fc240 0x7f4a0c0fc620 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 --2- 172.21.15.87:0/523074929 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c0fcba0 0x7f4a0c0f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 -- 172.21.15.87:0/523074929 >> 172.21.15.87:0/523074929 conn(0x7f4a0c06c370 msgr2=0x7f4a0c06c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 -- 172.21.15.87:0/523074929 shutdown_connections 2024-02-23T00:32:21.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 -- 172.21.15.87:0/523074929 wait complete. 2024-02-23T00:32:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 librados: starting msgr at 2024-02-23T00:32:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.458+0000 7f4a13e93700 1 librados: starting objecter 2024-02-23T00:32:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 Processor -- start 2024-02-23T00:32:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 -- start start 2024-02-23T00:32:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 librados: setting wanted keys 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 librados: calling monclient init 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4a0c0f6b10 0x7f4a0c114790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c0fc240 0x7f4a0c114c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4a0c0fcba0 0x7f4a0c10ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c10f410 con 0x7f4a0c0f6b10 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c10f580 con 0x7f4a0c0fc240 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a13e93700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c10f740 con 0x7f4a0c0fcba0 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a11c2f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4a0c0f6b10 0x7f4a0c114790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a11c2f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4a0c0f6b10 0x7f4a0c114790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59814/0 (socket says 172.21.15.87:59814) 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.459+0000 7f4a11c2f700 1 -- 172.21.15.87:0/1081438002 learned_addr learned my addr 172.21.15.87:0/1081438002 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a1142e700 1 --2- 172.21.15.87:0/1081438002 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c0fc240 0x7f4a0c114c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a12430700 1 --2- 172.21.15.87:0/1081438002 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4a0c0fcba0 0x7f4a0c10ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a1142e700 1 -- 172.21.15.87:0/1081438002 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4a0c0fcba0 msgr2=0x7f4a0c10ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a1142e700 1 --2- 172.21.15.87:0/1081438002 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4a0c0fcba0 0x7f4a0c10ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a1142e700 1 -- 172.21.15.87:0/1081438002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4a0c0f6b10 msgr2=0x7f4a0c114790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a1142e700 1 --2- 172.21.15.87:0/1081438002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4a0c0f6b10 0x7f4a0c114790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a1142e700 1 -- 172.21.15.87:0/1081438002 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a08005040 con 0x7f4a0c0fc240 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a12430700 1 --2- 172.21.15.87:0/1081438002 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4a0c0fcba0 0x7f4a0c10ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a11c2f700 1 --2- 172.21.15.87:0/1081438002 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4a0c0f6b10 0x7f4a0c114790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a1142e700 1 --2- 172.21.15.87:0/1081438002 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c0fc240 0x7f4a0c114c80 secure :-1 s=READY pgs=4970 cs=0 l=1 rev1=1 rx=0x7f49fc002760 tx=0x7f49fc009e80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a02ffd700 10 client.?.objecter ms_handle_connect 0x7f4a0c0fc240 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a02ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc007b40 con 0x7f4a0c0fc240 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49fc00f040 con 0x7f4a0c0fc240 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc014790 con 0x7f4a0c0fc240 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a0c10fa20 con 0x7f4a0c0fc240 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a13e93700 10 client.44488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0c11b600 con 0x7f4a0c0fc240 2024-02-23T00:32:21.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.460+0000 7f4a13e93700 1 librados: init done 2024-02-23T00:32:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.461+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4a0c05fa40 con 0x7f4a0c0fc240 2024-02-23T00:32:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.462+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f49fc007ca0 con 0x7f4a0c0fc240 2024-02-23T00:32:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.462+0000 7f4a02ffd700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6800/1231597463 conn(0x7f49f8062c90 0x7f49f8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.462+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(190..190 src has 38..190) v4 ==== 10254+0+0 (secure 0 0 0) 0x7f49fc07d830 con 0x7f4a0c0fc240 2024-02-23T00:32:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.462+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 osd_map(190..190 src has 38..190) v4 2024-02-23T00:32:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.462+0000 7f4a02ffd700 3 client.44488.objecter handle_osd_map got epochs [190,190] > 0 2024-02-23T00:32:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.462+0000 7f4a02ffd700 3 client.44488.objecter handle_osd_map decoding full epoch 190 2024-02-23T00:32:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.462+0000 7f4a11c2f700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6800/1231597463 conn(0x7f49f8062c90 0x7f49f8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a02ffd700 20 client.44488.objecter dump_active .. 0 homeless 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v190) v1 ==== 54+0+16 (secure 0 0 0) 0x7f49fc07e0a0 con 0x7f4a0c0fc240 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 10 client.44488.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 1 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4a0c0598f0 0x7f4a0c059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c05ee80 osd=0 3 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c05ee80 osd=0 3 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c05ee80 osd=0 3 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 1 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c05ee80 osd=0 3 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 0 1 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c05ee80 osd=0 4 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 10 client.44488.objecter _send_command 1 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a0c10ff30 con 0x7f4a0c0598f0 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a02ffd700 10 client.44488.objecter ms_handle_connect 0x7f49f8062c90 2024-02-23T00:32:21.466 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a11c2f700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6800/1231597463 conn(0x7f49f8062c90 0x7f49f8065150 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 rx=0x7f4a08004560 tx=0x7f4a08007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a12430700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4a0c0598f0 0x7f4a0c059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a02ffd700 10 client.44488.objecter ms_handle_connect 0x7f4a0c0598f0 2024-02-23T00:32:21.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.463+0000 7f4a12430700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4a0c0598f0 0x7f4a0c059cd0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.464+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4a0c10ff30 con 0x7f4a0c0598f0 2024-02-23T00:32:21.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.464+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:32:21.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.464+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 1 = system:0 2024-02-23T00:32:21.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.464+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c05ee80 osd=0 4 2024-02-23T00:32:21.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.464+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 0 1 2024-02-23T00:32:21.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 10 client.44488.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:21.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 2 2024-02-23T00:32:21.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c05ee80 osd=0 3 2024-02-23T00:32:21.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c05ee80 osd=0 3 2024-02-23T00:32:21.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c05ee80 osd=0 3 2024-02-23T00:32:21.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 2 2024-02-23T00:32:21.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c05ee80 osd=0 3 2024-02-23T00:32:21.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 0 2 2024-02-23T00:32:21.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c05ee80 osd=0 4 2024-02-23T00:32:21.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 10 client.44488.objecter _send_command 2 2024-02-23T00:32:21.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.479+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4a0c106030 con 0x7f4a0c0598f0 2024-02-23T00:32:21.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4a0c106030 con 0x7f4a0c0598f0 2024-02-23T00:32:21.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:21.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c05ee80 osd=0 4 2024-02-23T00:32:21.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 0 2 2024-02-23T00:32:21.482 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:32:21.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 10 client.44488.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:21.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 3 2024-02-23T00:32:21.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6805/685354171 conn(0x7f4a0c028540 0x7f4a0c11f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c0595a0 osd=1 3 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0595a0 osd=1 3 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c0595a0 osd=1 3 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 3 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c0595a0 osd=1 3 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 1 3 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0595a0 osd=1 4 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 10 client.44488.objecter _send_command 3 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.481+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a0c059710 con 0x7f4a0c028540 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.482+0000 7f4a11c2f700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6805/685354171 conn(0x7f4a0c028540 0x7f4a0c11f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.482+0000 7f4a02ffd700 10 client.44488.objecter ms_handle_connect 0x7f4a0c028540 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.482+0000 7f4a11c2f700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6805/685354171 conn(0x7f4a0c028540 0x7f4a0c11f870 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.482+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4a0c059710 con 0x7f4a0c028540 2024-02-23T00:32:21.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.482+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:32:21.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.482+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 3 = system:0 2024-02-23T00:32:21.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.482+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c0595a0 osd=1 4 2024-02-23T00:32:21.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.482+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 1 3 2024-02-23T00:32:21.497 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 10 client.44488.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 4 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c0595a0 osd=1 3 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0595a0 osd=1 3 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c0595a0 osd=1 3 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 4 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c0595a0 osd=1 3 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 1 4 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0595a0 osd=1 4 2024-02-23T00:32:21.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 10 client.44488.objecter _send_command 4 2024-02-23T00:32:21.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.498+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4a0c0fdf40 con 0x7f4a0c028540 2024-02-23T00:32:21.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4a0c0fdf40 con 0x7f4a0c028540 2024-02-23T00:32:21.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:21.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c0595a0 osd=1 4 2024-02-23T00:32:21.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 1 4 2024-02-23T00:32:21.501 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:32:21.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 10 client.44488.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:21.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 5 2024-02-23T00:32:21.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4a0c0fd3d0 0x7f4a0c122ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c0fd260 osd=2 3 2024-02-23T00:32:21.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0fd260 osd=2 3 2024-02-23T00:32:21.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c0fd260 osd=2 3 2024-02-23T00:32:21.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 5 2024-02-23T00:32:21.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c0fd260 osd=2 3 2024-02-23T00:32:21.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 2 5 2024-02-23T00:32:21.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0fd260 osd=2 4 2024-02-23T00:32:21.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 10 client.44488.objecter _send_command 5 2024-02-23T00:32:21.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a0c0280a0 con 0x7f4a0c0fd3d0 2024-02-23T00:32:21.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a1142e700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4a0c0fd3d0 0x7f4a0c122ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a02ffd700 10 client.44488.objecter ms_handle_connect 0x7f4a0c0fd3d0 2024-02-23T00:32:21.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.500+0000 7f4a1142e700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4a0c0fd3d0 0x7f4a0c122ef0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.501+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4a0c0280a0 con 0x7f4a0c0fd3d0 2024-02-23T00:32:21.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.501+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:32:21.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.501+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 5 = system:0 2024-02-23T00:32:21.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.501+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c0fd260 osd=2 4 2024-02-23T00:32:21.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.501+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 2 5 2024-02-23T00:32:21.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 10 client.44488.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 6 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c0fd260 osd=2 3 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0fd260 osd=2 3 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c0fd260 osd=2 3 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 6 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c0fd260 osd=2 3 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 2 6 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0fd260 osd=2 4 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 10 client.44488.objecter _send_command 6 2024-02-23T00:32:21.518 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.516+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4a0c1233e0 con 0x7f4a0c0fd3d0 2024-02-23T00:32:21.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.518+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4a0c1233e0 con 0x7f4a0c0fd3d0 2024-02-23T00:32:21.519 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:21.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c0fd260 osd=2 4 2024-02-23T00:32:21.520 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 2 6 2024-02-23T00:32:21.520 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 10 client.44488.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 7 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6813/128059790 conn(0x7f4a0c027770 0x7f4a0c1266e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c027600 osd=3 3 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c027600 osd=3 3 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c027600 osd=3 3 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 7 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c027600 osd=3 3 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 3 7 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c027600 osd=3 4 2024-02-23T00:32:21.521 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 10 client.44488.objecter _send_command 7 2024-02-23T00:32:21.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a0c120a40 con 0x7f4a0c027770 2024-02-23T00:32:21.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a12430700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6813/128059790 conn(0x7f4a0c027770 0x7f4a0c1266e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a02ffd700 10 client.44488.objecter ms_handle_connect 0x7f4a0c027770 2024-02-23T00:32:21.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.519+0000 7f4a12430700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6813/128059790 conn(0x7f4a0c027770 0x7f4a0c1266e0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.520+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4a0c120a40 con 0x7f4a0c027770 2024-02-23T00:32:21.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.520+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:32:21.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.520+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 7 = system:0 2024-02-23T00:32:21.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.520+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c027600 osd=3 4 2024-02-23T00:32:21.522 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.520+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 3 7 2024-02-23T00:32:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 10 client.44488.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 8 2024-02-23T00:32:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c027600 osd=3 3 2024-02-23T00:32:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c027600 osd=3 3 2024-02-23T00:32:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c027600 osd=3 3 2024-02-23T00:32:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 8 2024-02-23T00:32:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c027600 osd=3 3 2024-02-23T00:32:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 3 8 2024-02-23T00:32:21.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c027600 osd=3 4 2024-02-23T00:32:21.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 10 client.44488.objecter _send_command 8 2024-02-23T00:32:21.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.539+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4a0c126bd0 con 0x7f4a0c027770 2024-02-23T00:32:21.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4a0c126bd0 con 0x7f4a0c027770 2024-02-23T00:32:21.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:21.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c027600 osd=3 4 2024-02-23T00:32:21.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 3 8 2024-02-23T00:32:21.543 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 10 client.44488.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 9 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4a0c1272e0 0x7f4a0c127740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c126bd0 osd=4 3 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c126bd0 osd=4 3 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c126bd0 osd=4 3 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 9 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c126bd0 osd=4 3 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 4 9 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c126bd0 osd=4 4 2024-02-23T00:32:21.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 10 client.44488.objecter _send_command 9 2024-02-23T00:32:21.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.541+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a0c129300 con 0x7f4a0c1272e0 2024-02-23T00:32:21.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.542+0000 7f4a11c2f700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4a0c1272e0 0x7f4a0c127740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.542+0000 7f4a02ffd700 10 client.44488.objecter ms_handle_connect 0x7f4a0c1272e0 2024-02-23T00:32:21.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.542+0000 7f4a11c2f700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4a0c1272e0 0x7f4a0c127740 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.543+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4a0c129300 con 0x7f4a0c1272e0 2024-02-23T00:32:21.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.543+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:32:21.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.543+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 9 = system:0 2024-02-23T00:32:21.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.543+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c126bd0 osd=4 4 2024-02-23T00:32:21.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.543+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 4 9 2024-02-23T00:32:21.549 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 10 client.44488.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 10 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c126bd0 osd=4 3 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c126bd0 osd=4 3 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c126bd0 osd=4 3 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 10 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c126bd0 osd=4 3 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 4 10 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c126bd0 osd=4 4 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 10 client.44488.objecter _send_command 10 2024-02-23T00:32:21.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.559+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4a0c129300 con 0x7f4a0c1272e0 2024-02-23T00:32:21.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4a0c129300 con 0x7f4a0c1272e0 2024-02-23T00:32:21.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:21.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c126bd0 osd=4 4 2024-02-23T00:32:21.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 4 10 2024-02-23T00:32:21.562 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:32:21.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 10 client.44488.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:21.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 11 2024-02-23T00:32:21.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4a0c128290 0x7f4a0c1286f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c129300 osd=5 3 2024-02-23T00:32:21.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c129300 osd=5 3 2024-02-23T00:32:21.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c129300 osd=5 3 2024-02-23T00:32:21.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 11 2024-02-23T00:32:21.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c129300 osd=5 3 2024-02-23T00:32:21.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 5 11 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c129300 osd=5 4 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 10 client.44488.objecter _send_command 11 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a0c128d90 con 0x7f4a0c128290 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a1142e700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4a0c128290 0x7f4a0c1286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a02ffd700 10 client.44488.objecter ms_handle_connect 0x7f4a0c128290 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.561+0000 7f4a1142e700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4a0c128290 0x7f4a0c1286f0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.562+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4a0c128d90 con 0x7f4a0c128290 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.562+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.562+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 11 = system:0 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.562+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c129300 osd=5 4 2024-02-23T00:32:21.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.562+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 5 11 2024-02-23T00:32:21.577 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 10 client.44488.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 12 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c129300 osd=5 3 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c129300 osd=5 3 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c129300 osd=5 3 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 12 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c129300 osd=5 3 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 5 12 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c129300 osd=5 4 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 10 client.44488.objecter _send_command 12 2024-02-23T00:32:21.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.578+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4a0c128d90 con 0x7f4a0c128290 2024-02-23T00:32:21.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4a0c128d90 con 0x7f4a0c128290 2024-02-23T00:32:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c129300 osd=5 4 2024-02-23T00:32:21.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 5 12 2024-02-23T00:32:21.581 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:32:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 10 client.44488.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 13 2024-02-23T00:32:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4a0c12da90 0x7f4a0c12fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c128d90 osd=6 3 2024-02-23T00:32:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c128d90 osd=6 3 2024-02-23T00:32:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c128d90 osd=6 3 2024-02-23T00:32:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 13 2024-02-23T00:32:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c128d90 osd=6 3 2024-02-23T00:32:21.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 6 13 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c128d90 osd=6 4 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 10 client.44488.objecter _send_command 13 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a0c132ad0 con 0x7f4a0c12da90 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a12430700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4a0c12da90 0x7f4a0c12fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a02ffd700 10 client.44488.objecter ms_handle_connect 0x7f4a0c12da90 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.580+0000 7f4a12430700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4a0c12da90 0x7f4a0c12fe80 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.581+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4a0c132ad0 con 0x7f4a0c12da90 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.581+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.581+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 13 = system:0 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.581+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c128d90 osd=6 4 2024-02-23T00:32:21.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.581+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 6 13 2024-02-23T00:32:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 10 client.44488.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 14 2024-02-23T00:32:21.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c128d90 osd=6 3 2024-02-23T00:32:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c128d90 osd=6 3 2024-02-23T00:32:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c128d90 osd=6 3 2024-02-23T00:32:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 14 2024-02-23T00:32:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c128d90 osd=6 3 2024-02-23T00:32:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 6 14 2024-02-23T00:32:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c128d90 osd=6 4 2024-02-23T00:32:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 10 client.44488.objecter _send_command 14 2024-02-23T00:32:21.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.596+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4a0c130370 con 0x7f4a0c12da90 2024-02-23T00:32:21.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.598+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4a0c130370 con 0x7f4a0c12da90 2024-02-23T00:32:21.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.598+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:21.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.598+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.598+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c128d90 osd=6 4 2024-02-23T00:32:21.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.598+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 6 14 2024-02-23T00:32:21.600 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:32:21.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.598+0000 7f4a13e93700 10 client.44488.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:21.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.598+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 15 2024-02-23T00:32:21.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.598+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6812/373787000 conn(0x7f4a0c130a90 0x7f4a0c130ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c130370 osd=7 3 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c130370 osd=7 3 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c130370 osd=7 3 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 15 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c130370 osd=7 3 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 7 15 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c130370 osd=7 4 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a13e93700 10 client.44488.objecter _send_command 15 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a0c135e50 con 0x7f4a0c130a90 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a11c2f700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6812/373787000 conn(0x7f4a0c130a90 0x7f4a0c130ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a02ffd700 10 client.44488.objecter ms_handle_connect 0x7f4a0c130a90 2024-02-23T00:32:21.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.599+0000 7f4a11c2f700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6812/373787000 conn(0x7f4a0c130a90 0x7f4a0c130ef0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:21.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.600+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4a0c135e50 con 0x7f4a0c130a90 2024-02-23T00:32:21.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.600+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:32:21.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.600+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 15 = system:0 2024-02-23T00:32:21.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.600+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c130370 osd=7 4 2024-02-23T00:32:21.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.600+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 7 15 2024-02-23T00:32:21.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 10 client.44488.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:21.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign -1 16 2024-02-23T00:32:21.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c130370 osd=7 3 2024-02-23T00:32:21.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c130370 osd=7 3 2024-02-23T00:32:21.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 20 client.44488.objecter _get_session s=0x7f4a0c130370 osd=7 3 2024-02-23T00:32:21.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_remove -1 16 2024-02-23T00:32:21.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 20 client.44488.objecter get_session s=0x7f4a0c130370 osd=7 3 2024-02-23T00:32:21.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 15 client.44488.objecter _session_command_op_assign 7 16 2024-02-23T00:32:21.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c130370 osd=7 4 2024-02-23T00:32:21.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 10 client.44488.objecter _send_command 16 2024-02-23T00:32:21.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.615+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4a0c1314d0 con 0x7f4a0c130a90 2024-02-23T00:32:21.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.616+0000 7f4a02ffd700 1 -- 172.21.15.87:0/1081438002 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4a0c1314d0 con 0x7f4a0c130a90 2024-02-23T00:32:21.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.616+0000 7f4a02ffd700 10 client.44488.objecter ms_dispatch 0x7f4a0c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:21.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.616+0000 7f4a02ffd700 10 client.44488.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.616+0000 7f4a02ffd700 20 client.44488.objecter put_session s=0x7f4a0c130370 osd=7 4 2024-02-23T00:32:21.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.616+0000 7f4a02ffd700 15 client.44488.objecter _session_command_op_remove 7 16 2024-02-23T00:32:21.618 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 librados: watch_flush enter 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 librados: watch_flush exit 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 client.44488.objecter close_session for osd.0 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4a0c0598f0 msgr2=0x7f4a0c059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6801/2153191863 conn(0x7f4a0c0598f0 0x7f4a0c059cd0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c05ee80 osd=0 1 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 client.44488.objecter close_session for osd.1 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6805/685354171 conn(0x7f4a0c028540 msgr2=0x7f4a0c11f870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6805/685354171 conn(0x7f4a0c028540 0x7f4a0c11f870 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0595a0 osd=1 1 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 client.44488.objecter close_session for osd.2 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4a0c0fd3d0 msgr2=0x7f4a0c122ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4a0c0fd3d0 0x7f4a0c122ef0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c0fd260 osd=2 1 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 client.44488.objecter close_session for osd.3 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6813/128059790 conn(0x7f4a0c027770 msgr2=0x7f4a0c1266e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6813/128059790 conn(0x7f4a0c027770 0x7f4a0c1266e0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a12430700 1 -- 172.21.15.87:0/1081438002 reap_dead start 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c027600 osd=3 1 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 client.44488.objecter close_session for osd.4 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4a0c1272e0 msgr2=0x7f4a0c127740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4a0c1272e0 0x7f4a0c127740 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c126bd0 osd=4 1 2024-02-23T00:32:21.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 client.44488.objecter close_session for osd.5 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4a0c128290 msgr2=0x7f4a0c1286f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4a0c128290 0x7f4a0c1286f0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c129300 osd=5 1 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 client.44488.objecter close_session for osd.6 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4a0c12da90 msgr2=0x7f4a0c12fe80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4a0c12da90 0x7f4a0c12fe80 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c128d90 osd=6 1 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 client.44488.objecter close_session for osd.7 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6812/373787000 conn(0x7f4a0c130a90 msgr2=0x7f4a0c130ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.171:6812/373787000 conn(0x7f4a0c130a90 0x7f4a0c130ef0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 20 client.44488.objecter put_session s=0x7f4a0c130370 osd=7 1 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 20 client.44488.objecter shutdown clearing up homeless session... 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 10 client.44488.objecter successfully canceled tick 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a12430700 1 -- 172.21.15.87:0/1081438002 reap_dead start 2024-02-23T00:32:21.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a12430700 1 -- 172.21.15.87:0/1081438002 reap_dead start 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a12430700 1 -- 172.21.15.87:0/1081438002 reap_dead start 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a12430700 1 -- 172.21.15.87:0/1081438002 reap_dead start 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6800/1231597463 conn(0x7f49f8062c90 msgr2=0x7f49f8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6800/1231597463 conn(0x7f49f8062c90 0x7f49f8065150 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 rx=0x7f4a08004560 tx=0x7f4a08007f90).stop 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c0fc240 msgr2=0x7f4a0c114c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c0fc240 0x7f4a0c114c80 secure :-1 s=READY pgs=4970 cs=0 l=1 rev1=1 rx=0x7f49fc002760 tx=0x7f49fc009e80).stop 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 shutdown_connections 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4a0c0fc240 0x7f4a0c114c80 unknown :-1 s=CLOSED pgs=4970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 --2- 172.21.15.87:0/1081438002 >> v2:172.21.15.87:6800/1231597463 conn(0x7f49f8062c90 0x7f49f8065150 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 >> 172.21.15.87:0/1081438002 conn(0x7f4a0c06c370 msgr2=0x7f4a0c0f5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 shutdown_connections 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 -- 172.21.15.87:0/1081438002 wait complete. 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:21.617+0000 7f4a13e93700 1 librados: shutdown 2024-02-23T00:32:21.623 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.624 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.624 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.624 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.624 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.624 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.624 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:32:21.624 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:32:22.026 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:22.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:22.130 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:22.158 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:22.230 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:22.232 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:20.704003+0000 mgr.y (mgr.24437) 234 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:22.232 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:21.264126+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:22.232 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:21.264257+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-02-23T00:32:22.232 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:21.422356+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:22.232 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:21.462403+0000 mon.b (mon.1) 166 : audit [DBG] from='client.? 172.21.15.87:0/1081438002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:22.232 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:21.582863+0000 mon.a (mon.0) 1441 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-23T00:32:22.233 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:21.991709+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi087-82011-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi087-82011-28"}]': finished 2024-02-23T00:32:22.233 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:21.991896+0000 mon.a (mon.0) 1443 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-23T00:32:22.233 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:21.991953+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-02-23T00:32:22.233 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.004192+0000 mon.c (mon.2) 213 : audit [INF] from='client.? 172.21.15.87:0/923895917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi087-82019-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:22.233 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.004473+0000 mon.c (mon.2) 214 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:22.233 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.007899+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi087-82019-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:22.233 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.008358+0000 mon.a (mon.0) 1446 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:22.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:22.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:22] "GET /metrics HTTP/1.1" 200 192112 "" "Prometheus/2.33.4" 2024-02-23T00:32:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:20.704003+0000 mgr.y (mgr.24437) 234 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:21.264126+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:21.264257+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-02-23T00:32:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:21.422356+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:21.462403+0000 mon.b (mon.1) 166 : audit [DBG] from='client.? 172.21.15.87:0/1081438002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:21.582863+0000 mon.a (mon.0) 1441 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-23T00:32:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:21.991709+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi087-82011-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi087-82011-28"}]': finished 2024-02-23T00:32:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:21.991896+0000 mon.a (mon.0) 1443 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-23T00:32:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:21.991953+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-02-23T00:32:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.004192+0000 mon.c (mon.2) 213 : audit [INF] from='client.? 172.21.15.87:0/923895917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi087-82019-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.004473+0000 mon.c (mon.2) 214 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.007899+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi087-82019-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.008358+0000 mon.a (mon.0) 1446 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:22.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:20.704003+0000 mgr.y (mgr.24437) 234 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:21.264126+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:21.264257+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:21.422356+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:21.462403+0000 mon.b (mon.1) 166 : audit [DBG] from='client.? 172.21.15.87:0/1081438002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:21.582863+0000 mon.a (mon.0) 1441 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:21.991709+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi087-82011-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi087-82011-28"}]': finished 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:21.991896+0000 mon.a (mon.0) 1443 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:21.991953+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.004192+0000 mon.c (mon.2) 213 : audit [INF] from='client.? 172.21.15.87:0/923895917' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi087-82019-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.004473+0000 mon.c (mon.2) 214 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.007899+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi087-82019-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.008358+0000 mon.a (mon.0) 1446 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:22.336 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:22.362 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:23.013 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:23.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:23.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:23.124 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:23.152 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:23.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:23.263 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:32:23.275 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:23.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:21.581758+0000 mgr.y (mgr.24437) 235 : cluster [DBG] pgmap v268: 276 pgs: 40 creating+peering, 14 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 1 peering, 195 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:23.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.014596+0000 mon.b (mon.1) 167 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:23.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.019191+0000 mon.a (mon.0) 1447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:23.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.423154+0000 mon.c (mon.2) 215 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:23.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.993321+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi087-82019-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:23.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.993417+0000 mon.a (mon.0) 1449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]': finished 2024-02-23T00:32:23.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.993550+0000 mon.a (mon.0) 1450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:23.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:22.993613+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-02-23T00:32:23.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.995423+0000 mon.b (mon.1) 168 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.995540+0000 mon.c (mon.2) 216 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.995752+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:22.996149+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:21.581758+0000 mgr.y (mgr.24437) 235 : cluster [DBG] pgmap v268: 276 pgs: 40 creating+peering, 14 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 1 peering, 195 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.014596+0000 mon.b (mon.1) 167 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.019191+0000 mon.a (mon.0) 1447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.423154+0000 mon.c (mon.2) 215 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.993321+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi087-82019-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.993417+0000 mon.a (mon.0) 1449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]': finished 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.993550+0000 mon.a (mon.0) 1450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:23.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:22.993613+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-02-23T00:32:23.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.995423+0000 mon.b (mon.1) 168 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:23.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.995540+0000 mon.c (mon.2) 216 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:23.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.995752+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:23.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:22.996149+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:23.301 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:21.581758+0000 mgr.y (mgr.24437) 235 : cluster [DBG] pgmap v268: 276 pgs: 40 creating+peering, 14 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 1 peering, 195 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s wr, 2 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:23.301 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.014596+0000 mon.b (mon.1) 167 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:23.301 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.019191+0000 mon.a (mon.0) 1447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:23.301 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.423154+0000 mon.c (mon.2) 215 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:23.301 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.993321+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi087-82019-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:23.301 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.993417+0000 mon.a (mon.0) 1449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi087-82083-36"}]': finished 2024-02-23T00:32:23.301 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.993550+0000 mon.a (mon.0) 1450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:23.301 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:22.993613+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-02-23T00:32:23.302 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.995423+0000 mon.b (mon.1) 168 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:23.302 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.995540+0000 mon.c (mon.2) 216 : audit [INF] from='client.? 172.21.15.87:0/3576868964' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:23.302 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.995752+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:23.302 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:22.996149+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]: dispatch 2024-02-23T00:32:23.303 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:23.325 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:32:23.325 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24344.0:53 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e59)", 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:27:21.016389+0000", 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 302.29979661599998, 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01501192, 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24344", 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2178799724", 2024-02-23T00:32:23.326 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 53 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.016389+0000", 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.016389+0000", 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.327 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.016387+0000", 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0538e-05 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.016398+0000", 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2442e-05 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.016410+0000", 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9279999999999993e-06 2024-02-23T00:32:23.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.016418+0000", 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0030323799999999999 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.019451+0000", 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.043e-05 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.019461+0000", 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082467349999999998 2024-02-23T00:32:23.329 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.027708+0000", 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012509899999999999 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.027833+0000", 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000174434 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4,7", 2024-02-23T00:32:23.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.028007+0000", 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013841280000000001 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.029391+0000", 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018620799999999999 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.029578+0000", 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001762617 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.031340+0000", 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0621999999999999e-05 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.031371+0000", 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0537000000000003e-05 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.031401+0000", 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01501192 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.332 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.14397.0:91 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e59)", 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:27:21.022692+0000", 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 302.293493877, 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087896180000000008, 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.14397", 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 91 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.333 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.022692+0000", 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.022692+0000", 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.022690+0000", 2024-02-23T00:32:23.334 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2579999999999995e-06 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.022698+0000", 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5566999999999999e-05 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.022714+0000", 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7729999999999999e-06 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.022722+0000", 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7735999999999998e-05 2024-02-23T00:32:23.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.022769+0000", 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9360000000000002e-06 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.022776+0000", 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054669260000000004 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.336 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.028243+0000", 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2482000000000002e-05 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.028316+0000", 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000128324 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4,7", 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.028444+0000", 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00095335600000000004 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.337 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.029397+0000", 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00037951100000000002 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.029777+0000", 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001658762 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.031436+0000", 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5442000000000001e-05 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.031461+0000", 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0625e-05 2024-02-23T00:32:23.338 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:27:21.031482+0000", 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087896180000000008 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27712.0:57 67.3 67:c06d7414:test-rados-api-smithi087-82115-12::foo:head [write 0~128] snapc 14=[14] ondisk+write+ordersnap+known_if_redirected e138)", 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:29.022087+0000", 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 54.294099449000001, 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030285401, 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.339 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27712", 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2607672493", 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 57 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.022087+0000", 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.022087+0000", 2024-02-23T00:32:23.340 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.022085+0000", 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3600000000000002e-06 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.022094+0000", 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1858000000000001e-05 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.022106+0000", 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.515e-06 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.022113+0000", 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8475e-05 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.022152+0000", 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000130278 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.022282+0000", 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015852499999999999 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.342 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0,3", 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.022441+0000", 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0029271029999999999 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.025368+0000", 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017769138 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.043137+0000", 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0091563500000000006 2024-02-23T00:32:23.343 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.344 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.344 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.344 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.052293+0000", 2024-02-23T00:32:23.344 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8924000000000002e-05 2024-02-23T00:32:23.344 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.344 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.344 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.344 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.052332+0000", 2024-02-23T00:32:23.344 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9907999999999998e-05 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.052372+0000", 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030285401 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.41845.0:2 90.d 90:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e138)", 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:29.042336+0000", 2024-02-23T00:32:23.345 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 54.273850338000003, 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032221679000000003, 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.41845", 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2059300124", 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.346 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.042336+0000", 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.042336+0000", 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.042334+0000", 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9580000000000007e-06 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.042342+0000", 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4983e-05 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.042357+0000", 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1379999999999996e-06 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.042366+0000", 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5310999999999998e-05 2024-02-23T00:32:23.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.042441+0000", 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0399999999999993e-06 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.042449+0000", 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011702669000000001 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.054152+0000", 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010567400000000001 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.054258+0000", 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113681 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.054371+0000", 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036325000000000002 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.054734+0000", 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00055018600000000003 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.055285+0000", 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019178015999999999 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.074463+0000", 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3726e-05 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.074516+0000", 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0988999999999997e-05 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.074557+0000", 2024-02-23T00:32:23.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032221679000000003 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27712.0:59 67.3 67:c06d7414:test-rados-api-smithi087-82115-12::foo:head [write 0~128] snapc 15=[15,14] ondisk+write+ordersnap+known_if_redirected e139)", 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:30.037148+0000", 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 53.279037950999999, 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047807215, 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27712", 2024-02-23T00:32:23.352 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2607672493", 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 59 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037148+0000", 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037148+0000", 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037146+0000", 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5389999999999996e-06 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037156+0000", 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0390999999999999e-05 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037166+0000", 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6610000000000003e-06 2024-02-23T00:32:23.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037173+0000", 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00072323199999999998 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037896+0000", 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0890000000000002e-06 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037903+0000", 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0016093780000000001 2024-02-23T00:32:23.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.039512+0000", 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0044999999999996e-05 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.039602+0000", 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000210879 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0,3", 2024-02-23T00:32:23.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.039813+0000", 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011312310000000001 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.051126+0000", 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072859370000000001 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.058411+0000", 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026478003999999999 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.084889+0000", 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.54e-05 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.084925+0000", 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0409e-05 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.084955+0000", 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.047807215 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27712.0:61 67.3 67:c06d7414:test-rados-api-smithi087-82115-12::foo:head [write 0~128] snapc 14=[14] ondisk+write+known_if_redirected e139)", 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:30.085600+0000", 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 53.230586011, 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016621252, 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27712", 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2607672493", 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 61 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.085600+0000", 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.085600+0000", 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.085599+0000", 2024-02-23T00:32:23.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.789e-06 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.085605+0000", 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8769999999999999e-06 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.085614+0000", 2024-02-23T00:32:23.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.454e-06 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.085618+0000", 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5559999999999999e-05 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.085644+0000", 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6536000000000002e-05 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.085700+0000", 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00010493900000000001 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0,3", 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.085805+0000", 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00030998400000000002 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.086115+0000", 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0019838239999999999 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.088099+0000", 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014059014 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.102158+0000", 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6161999999999997e-05 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.102194+0000", 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6968999999999999e-05 2024-02-23T00:32:23.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.102221+0000", 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016621252 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.44089.0:2 92.13 92:cbb9903e:::test_obj:head [write 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected e141)", 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:32.234964+0000", 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 51.081222283000002, 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012448242999999999, 2024-02-23T00:32:23.365 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.44089", 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3099212657", 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.234964+0000", 2024-02-23T00:32:23.366 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.234964+0000", 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.234962+0000", 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5920000000000003e-06 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.234970+0000", 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1578e-05 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.234981+0000", 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.109e-06 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.234988+0000", 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000101988 2024-02-23T00:32:23.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.235090+0000", 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120029 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.235210+0000", 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000150631 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:32:23.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.235361+0000", 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070158759999999999 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.242377+0000", 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001867711 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.244244+0000", 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003063423 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.247308+0000", 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4950000000000001e-05 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.247363+0000", 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9154999999999999e-05 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:32.247412+0000", 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012448242999999999 2024-02-23T00:32:23.371 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:572 63.c 63:32d270ed:test-rados-api-smithi087-82091-14::foo1:head [writefull 0~11] snapc 5=[5,4] ondisk+write+known_if_redirected e146)", 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:37.009765+0000", 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 46.30642126, 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004075983, 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 572 2024-02-23T00:32:23.372 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.009765+0000", 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.009765+0000", 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.009763+0000", 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0399e-05 2024-02-23T00:32:23.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.009773+0000", 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7223000000000001e-05 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.009791+0000", 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2969999999999998e-06 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.009798+0000", 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1858999999999999e-05 2024-02-23T00:32:23.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.009850+0000", 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5523000000000002e-05 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.009925+0000", 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014890899999999999 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0,4", 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.010074+0000", 2024-02-23T00:32:23.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00056905300000000003 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.010643+0000", 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00024854500000000003 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.010892+0000", 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028628730000000002 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.013755+0000", 2024-02-23T00:32:23.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4941999999999999e-05 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.013790+0000", 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1227999999999998e-05 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.013841+0000", 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004075983 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(client.26224.0:672 63.6 e147/114 63:6271ce74:test-rados-api-smithi087-82091-14::foo0:head v 147'17, mlcod=146'13)", 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:37.996224+0000", 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 45.319961769999999, 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.009279799, 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.996224+0000", 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.996224+0000", 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.996224+0000", 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6579999999999997e-06 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.996234+0000", 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6409999999999996e-06 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.996240+0000", 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9619999999999996e-06 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.996247+0000", 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3814999999999996e-05 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.996331+0000", 2024-02-23T00:32:23.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0662999999999999e-05 2024-02-23T00:32:23.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.996352+0000", 2024-02-23T00:32:23.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0091139029999999996 2024-02-23T00:32:23.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.005466+0000", 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8381999999999999e-05 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.005504+0000", 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.009279799 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:774 63.c 63:32d270ed:test-rados-api-smithi087-82091-14::foo1:head [writefull 0~11] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e148)", 2024-02-23T00:32:23.382 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:39.011079+0000", 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 44.305107454999998, 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050312170000000002, 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 774 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.011079+0000", 2024-02-23T00:32:23.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.011079+0000", 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.011077+0000", 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2049999999999996e-06 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.011084+0000", 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7899999999999994e-06 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.011094+0000", 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8740000000000003e-06 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.011099+0000", 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9150000000000003e-05 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.011138+0000", 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5725999999999998e-05 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.011234+0000", 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000203031 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0,4", 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.011437+0000", 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00059008800000000001 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.012027+0000", 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00097493299999999998 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.013002+0000", 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0030046729999999998 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.016006+0000", 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3466999999999994e-05 2024-02-23T00:32:23.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.016070+0000", 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9898000000000003e-05 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.016110+0000", 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050312170000000002 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDECSubOpWrite(93.0s1 149/142 ECSubWrite(tid=261, reqid=osd.7.0:261, at_version=149'6, trim_to=0'0, roll_forward_to=148'4))", 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:39.994900+0000", 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 43.321286186000002, 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013646978000000001, 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.994900+0000", 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.994900+0000", 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.994899+0000", 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9213e-05 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.994928+0000", 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.482e-05 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.994943+0000", 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9070000000000002e-06 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.994953+0000", 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010271885 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.005224+0000", 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0151e-05 2024-02-23T00:32:23.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.005235+0000", 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0032907190000000001 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.008525+0000", 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1466000000000001e-05 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.008547+0000", 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013646978000000001 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDECSubOpWrite(93.0s1 149/142 ECSubWrite(tid=262, reqid=osd.7.0:261, at_version=0'0, trim_to=0'0, roll_forward_to=149'6))", 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:40.009301+0000", 2024-02-23T00:32:23.393 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 43.306884740000001, 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.005067368, 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.009301+0000", 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.009301+0000", 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.009300+0000", 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3351e-05 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.009314+0000", 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2087e-05 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.009326+0000", 2024-02-23T00:32:23.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4529999999999996e-06 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.009333+0000", 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00056125399999999996 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.009894+0000", 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7410000000000004e-06 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.009904+0000", 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044426049999999996 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.014347+0000", 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1936000000000001e-05 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.014369+0000", 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.005067368 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:873 95.6 95:6271ce74:test-rados-api-smithi087-82091-14::foo0:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:41.998042+0000", 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 41.318143874, 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0074213539999999998, 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 873 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.998042+0000", 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.998042+0000", 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.998040+0000", 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0154999999999999e-05 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.998060+0000", 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1629e-05 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.998071+0000", 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1679999999999993e-06 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.998081+0000", 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2816999999999999e-05 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.998133+0000", 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00047475600000000001 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.998608+0000", 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0058596969999999996 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.004468+0000", 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112221 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.004580+0000", 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00040138400000000002 2024-02-23T00:32:23.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.004982+0000", 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2485999999999998e-05 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.005044+0000", 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041950100000000002 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.005464+0000", 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0074213539999999998 2024-02-23T00:32:23.403 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:950 95.1c 95:3d075ef2:test-rados-api-smithi087-82091-14::foo45:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:42.149889+0000", 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 41.166296682999999, 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004811259, 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:32:23.404 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 950 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.149889+0000", 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.149889+0000", 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.149888+0000", 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9569999999999999e-06 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.149894+0000", 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0472000000000001e-05 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.149905+0000", 2024-02-23T00:32:23.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9479999999999999e-06 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.149910+0000", 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1591999999999998e-05 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.149951+0000", 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000136544 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.150088+0000", 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00088392799999999997 2024-02-23T00:32:23.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.150972+0000", 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9015000000000003e-05 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.151031+0000", 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00070565199999999997 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.151737+0000", 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0637999999999998e-05 2024-02-23T00:32:23.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.151797+0000", 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0029034629999999998 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.154701+0000", 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004811259 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:952 95.0 95:04438bb4:test-rados-api-smithi087-82091-14::foo79:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:42.159200+0000", 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 41.156985820999999, 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.005336024, 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 952 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.159200+0000", 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.159200+0000", 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.159199+0000", 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5600000000000001e-06 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.159205+0000", 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8630000000000003e-06 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.159215+0000", 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5959999999999997e-06 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.159219+0000", 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2985000000000002e-05 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.159272+0000", 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000120226 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.159392+0000", 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0010703500000000001 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.160463+0000", 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016684600000000001 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.160630+0000", 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00087853599999999997 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.161508+0000", 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9760999999999996e-05 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.161608+0000", 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0002744 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.161882+0000", 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0727000000000003e-05 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.161943+0000", 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025931389999999999 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.164536+0000", 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.005336024 2024-02-23T00:32:23.415 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:957 95.0 95:00c72a10:test-rados-api-smithi087-82091-14::foo44:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:42.186839+0000", 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 41.129347283999998, 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054077830000000002, 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:32:23.416 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 957 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.186839+0000", 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.186839+0000", 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.186838+0000", 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2610000000000001e-06 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.186843+0000", 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.251e-06 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.186852+0000", 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2899999999999996e-06 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.186857+0000", 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7376999999999998e-05 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.186894+0000", 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000117909 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:23.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.187012+0000", 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013159840000000001 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.188328+0000", 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00020021700000000001 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.188528+0000", 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077187800000000004 2024-02-23T00:32:23.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.189300+0000", 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3778999999999999e-05 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.189384+0000", 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000331377 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.189715+0000", 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9787999999999997e-05 2024-02-23T00:32:23.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.189775+0000", 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0024715840000000002 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.192247+0000", 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054077830000000002 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36806.0:2 109.1c 109:384b83b2:::oid_RoundTripWriteFullPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e162)", 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:53.002546+0000", 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 30.313639791, 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0059211430000000002, 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36806", 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2835242559", 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.002546+0000", 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.002546+0000", 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.002545+0000", 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6449999999999999e-06 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.002551+0000", 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2029999999999998e-06 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.002560+0000", 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9519999999999997e-06 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.002567+0000", 2024-02-23T00:32:23.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7129999999999997e-05 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.002624+0000", 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8099999999999998e-06 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.002632+0000", 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040626869999999997 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.006695+0000", 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013247899999999999 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.006827+0000", 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00019956299999999999 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.007027+0000", 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00067307599999999999 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.007700+0000", 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00030813999999999999 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.008008+0000", 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00040241400000000003 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:32:23.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.008411+0000", 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1488000000000001e-05 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.008442+0000", 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5253999999999999e-05 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:53.008467+0000", 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0059211430000000002 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.429 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.4.0:479 63.19 e182/114 63:9c127ee7:test-rados-api-smithi087-82091-14::foo66:7 v 182'29, mlcod=148'25)", 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:12.998777+0000", 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 10.317408997999999, 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01433426, 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:12.998777+0000", 2024-02-23T00:32:23.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:12.998777+0000", 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:12.998777+0000", 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7520000000000008e-06 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:12.998787+0000", 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7166000000000001e-05 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:12.998804+0000", 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3819999999999993e-06 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:12.998813+0000", 2024-02-23T00:32:23.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000112612 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:12.998926+0000", 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3427000000000001e-05 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:12.998949+0000", 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0077358770000000004 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006685+0000", 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0064263150000000002 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.013111+0000", 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01433426 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.4.0:482 63.19 e182/114 63:9ecd68ad:test-rados-api-smithi087-82091-14::foo38:7 v 182'31, mlcod=148'25)", 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:13.000467+0000", 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 10.315718892, 2024-02-23T00:32:23.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012735105, 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000467+0000", 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000467+0000", 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000466+0000", 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2210000000000003e-06 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000475+0000", 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.52e-05 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000491+0000", 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9190000000000002e-06 2024-02-23T00:32:23.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000497+0000", 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1661999999999993e-05 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000559+0000", 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0404999999999999e-05 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000580+0000", 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012602423 2024-02-23T00:32:23.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.013182+0000", 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0293999999999999e-05 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.013202+0000", 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012735105 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.438 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37004.0:1 134.6s0 134:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e191)", 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:22.005484+0000", 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 1.3107021750000001, 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0088914980000001, 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37004", 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2538319794", 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:22.005484+0000", 2024-02-23T00:32:23.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:22.005484+0000", 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:22.005482+0000", 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3719999999999999e-06 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:22.005489+0000", 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5030000000000003e-06 2024-02-23T00:32:23.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:22.005499+0000", 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8849999999999999e-06 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:22.005507+0000", 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3978999999999999e-05 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:22.005561+0000", 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8600000000000001e-06 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:22.005563+0000", 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.004290482 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:23.009854+0000", 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000128491 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:23.009982+0000", 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00042572900000000002 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:32:23.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:23.010408+0000", 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002332223 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:23.012740+0000", 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001569188 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:23.014310+0000", 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5798999999999998e-05 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:32:23.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:32:23.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:23.014375+0000", 2024-02-23T00:32:23.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0088914980000001 2024-02-23T00:32:23.444 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.444 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.444 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.444 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:32:23.444 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:32:23.444 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:32:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:23.445 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:23.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:23.546 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:23.574 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:23.646 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:23.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:23.752 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:23.779 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:23.806 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:23.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:23.911 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:23.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:32:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplePP (69463 ms total) 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (129 ms) 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (129 ms total) 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (22 ms) 2024-02-23T00:32:24.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (3 ms) 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (25 ms total) 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (20 ms) 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (7 ms) 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (27 ms total) 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (27 ms) 2024-02-23T00:32:24.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (2 ms) 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (29 ms total) 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1026 ms) 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1026 ms total) 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (101914 ms total) 2024-02-23T00:32:24.019 INFO:tasks.workunit.client.0.smithi087.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-02-23T00:32:24.072 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:32:24.072 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-23T00:32:24.072 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:32:24.074 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-23T00:32:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:24.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:23.423901+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:24.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:23.997460+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-15"}]': finished 2024-02-23T00:32:24.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:23.997538+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]': finished 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:23.997659+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:23.999431+0000 mon.b (mon.1) 169 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-15", "mode": "writeback"}]: dispatch 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:24.000598+0000 mon.a (mon.0) 1457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-15", "mode": "writeback"}]: dispatch 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:24.004580+0000 mon.c (mon.2) 218 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:24.004906+0000 mon.a (mon.0) 1458 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:23.423901+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:23.997460+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-15"}]': finished 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:23.997538+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]': finished 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:23.997659+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:23.999431+0000 mon.b (mon.1) 169 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-15", "mode": "writeback"}]: dispatch 2024-02-23T00:32:24.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:24.000598+0000 mon.a (mon.0) 1457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-15", "mode": "writeback"}]: dispatch 2024-02-23T00:32:24.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:24.004580+0000 mon.c (mon.2) 218 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:24.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:24.004906+0000 mon.a (mon.0) 1458 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:24.088 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:24.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:24.192 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:24.219 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:24.348 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:23.423901+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:23.997460+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-15"}]': finished 2024-02-23T00:32:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:23.997538+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi087-82083-36"}]': finished 2024-02-23T00:32:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:23.997659+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-02-23T00:32:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:23.999431+0000 mon.b (mon.1) 169 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-15", "mode": "writeback"}]: dispatch 2024-02-23T00:32:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:24.000598+0000 mon.a (mon.0) 1457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-15", "mode": "writeback"}]: dispatch 2024-02-23T00:32:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:24.004580+0000 mon.c (mon.2) 218 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:24.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:24.004906+0000 mon.a (mon.0) 1458 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:24.571 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:32:24.699 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:24.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:24.802 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:24.830 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:23.582310+0000 mgr.y (mgr.24437) 236 : cluster [DBG] pgmap v271: 308 pgs: 32 creating+peering, 14 active+clean+snaptrim, 40 unknown, 26 active+clean+snaptrim_wait, 1 peering, 195 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 9.0 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:24.424746+0000 mon.c (mon.2) 219 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:24.638285+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:24.991890+0000 mon.a (mon.0) 1460 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:24.998523+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-15", "mode": "writeback"}]': finished 2024-02-23T00:32:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:24.998627+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]': finished 2024-02-23T00:32:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:24.998732+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-02-23T00:32:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.003649+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.005431+0000 mon.a (mon.0) 1464 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.005622+0000 mon.a (mon.0) 1465 : audit [INF] from='client.? 172.21.15.87:0/2722004722' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi087-82019-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.005911+0000 mon.a (mon.0) 1466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:23.582310+0000 mgr.y (mgr.24437) 236 : cluster [DBG] pgmap v271: 308 pgs: 32 creating+peering, 14 active+clean+snaptrim, 40 unknown, 26 active+clean+snaptrim_wait, 1 peering, 195 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 9.0 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:24.424746+0000 mon.c (mon.2) 219 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:24.638285+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:24.991890+0000 mon.a (mon.0) 1460 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:24.998523+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-15", "mode": "writeback"}]': finished 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:24.998627+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]': finished 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:24.998732+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.003649+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.005431+0000 mon.a (mon.0) 1464 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.005622+0000 mon.a (mon.0) 1465 : audit [INF] from='client.? 172.21.15.87:0/2722004722' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi087-82019-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.005911+0000 mon.a (mon.0) 1466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:23.582310+0000 mgr.y (mgr.24437) 236 : cluster [DBG] pgmap v271: 308 pgs: 32 creating+peering, 14 active+clean+snaptrim, 40 unknown, 26 active+clean+snaptrim_wait, 1 peering, 195 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 9.0 KiB/s wr, 5 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:24.424746+0000 mon.c (mon.2) 219 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:24.638285+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:24.991890+0000 mon.a (mon.0) 1460 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:24.998523+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-15", "mode": "writeback"}]': finished 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:24.998627+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi087-82011-28"}]': finished 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:24.998732+0000 mon.a (mon.0) 1463 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.003649+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.87:0/2538319794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.005431+0000 mon.a (mon.0) 1464 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.005622+0000 mon.a (mon.0) 1465 : audit [INF] from='client.? 172.21.15.87:0/2722004722' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi087-82019-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.005911+0000 mon.a (mon.0) 1466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]: dispatch 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.425515+0000 mon.c (mon.2) 221 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.584358+0000 mon.a (mon.0) 1467 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.999591+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]': finished 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.999687+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? 172.21.15.87:0/2722004722' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi087-82019-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.999763+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]': finished 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:25.999995+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:26.000085+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:26.005509+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.425515+0000 mon.c (mon.2) 221 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:26.265 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.584358+0000 mon.a (mon.0) 1467 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-23T00:32:26.266 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.999591+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]': finished 2024-02-23T00:32:26.266 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.999687+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? 172.21.15.87:0/2722004722' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi087-82019-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:26.266 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.999763+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]': finished 2024-02-23T00:32:26.266 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:25.999995+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-23T00:32:26.266 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:26.000085+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-02-23T00:32:26.266 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:26.005509+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:26.278 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:26.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:26.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:26.306 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:32:26.307 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:32:26.307 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:32:26.307 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:32:26.324 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.425515+0000 mon.c (mon.2) 221 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:26.324 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.584358+0000 mon.a (mon.0) 1467 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-23T00:32:26.324 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.999591+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]': finished 2024-02-23T00:32:26.324 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.999687+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? 172.21.15.87:0/2722004722' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi087-82019-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:26.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.999763+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi087-82011-28"}]': finished 2024-02-23T00:32:26.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:25.999995+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-23T00:32:26.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:26.000085+0000 mon.a (mon.0) 1472 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-02-23T00:32:26.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:26.005509+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]: dispatch 2024-02-23T00:32:26.382 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:26.412 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:26.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 -- 172.21.15.87:0/1478563982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f040c0f7330 msgr2=0x7f040c103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:26.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 --2- 172.21.15.87:0/1478563982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f040c0f7330 0x7f040c103930 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 rx=0x7f0404009c30 tx=0x7f0404004c50).stop 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 -- 172.21.15.87:0/1478563982 shutdown_connections 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 --2- 172.21.15.87:0/1478563982 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f040c0f7330 0x7f040c103930 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 --2- 172.21.15.87:0/1478563982 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f040c0fcac0 0x7f040c0fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 --2- 172.21.15.87:0/1478563982 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f040c0fd420 0x7f040c0f6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 -- 172.21.15.87:0/1478563982 >> 172.21.15.87:0/1478563982 conn(0x7f040c0f2710 msgr2=0x7f040c0f4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 -- 172.21.15.87:0/1478563982 shutdown_connections 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 -- 172.21.15.87:0/1478563982 wait complete. 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 librados: starting msgr at 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.518+0000 7f04110c4700 1 librados: starting objecter 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 Processor -- start 2024-02-23T00:32:26.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 -- start start 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 librados: setting wanted keys 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 librados: calling monclient init 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f040c0f7330 0x7f040c0f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f040c0fcac0 0x7f040c0f8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f040c0fd420 0x7f040c0f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c116310 con 0x7f040c0fcac0 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c116480 con 0x7f040c0f7330 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f04110c4700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040c116640 con 0x7f040c0fd420 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040b59e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f040c0fd420 0x7f040c0f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f040c0fcac0 0x7f040c0f8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040b59e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f040c0fd420 0x7f040c0f92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:58320/0 (socket says 172.21.15.87:58320) 2024-02-23T00:32:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040b59e700 1 -- 172.21.15.87:0/487596257 learned_addr learned my addr 172.21.15.87:0/487596257 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040ad9d700 1 --2- 172.21.15.87:0/487596257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f040c0f7330 0x7f040c0f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040b59e700 1 -- 172.21.15.87:0/487596257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f040c0f7330 msgr2=0x7f040c0f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040b59e700 1 --2- 172.21.15.87:0/487596257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f040c0f7330 0x7f040c0f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040b59e700 1 -- 172.21.15.87:0/487596257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f040c0fcac0 msgr2=0x7f040c0f8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040b59e700 1 --2- 172.21.15.87:0/487596257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f040c0fcac0 0x7f040c0f8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.519+0000 7f040b59e700 1 -- 172.21.15.87:0/487596257 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0400005040 con 0x7f040c0fd420 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f040b59e700 1 --2- 172.21.15.87:0/487596257 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f040c0fd420 0x7f040c0f92a0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f0404004860 tx=0x7f0404004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f03f3fff700 10 client.?.objecter ms_handle_connect 0x7f040c0fd420 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f03f3fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f03f3fff700 1 -- 172.21.15.87:0/487596257 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040400e040 con 0x7f040c0fd420 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f03f3fff700 1 -- 172.21.15.87:0/487596257 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0404018040 con 0x7f040c0fd420 2024-02-23T00:32:26.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04040097c0 con 0x7f040c0fd420 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f03f3fff700 1 -- 172.21.15.87:0/487596257 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040401c3f0 con 0x7f040c0fd420 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f04110c4700 10 client.37028.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040c116bb0 con 0x7f040c0fd420 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.520+0000 7f04110c4700 1 librados: init done 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.521+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040c0606d0 con 0x7f040c0fd420 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.521+0000 7f03f3fff700 1 -- 172.21.15.87:0/487596257 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0404013420 con 0x7f040c0fd420 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.521+0000 7f03f3fff700 1 --2- 172.21.15.87:0/487596257 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03f4062c90 0x7f03f4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.522+0000 7f03f3fff700 1 -- 172.21.15.87:0/487596257 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(195..195 src has 38..195) v4 ==== 8787+0+0 (secure 0 0 0) 0x7f040407e540 con 0x7f040c0fd420 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.522+0000 7f03f3fff700 10 client.37028.objecter ms_dispatch 0x7f040c001550 osd_map(195..195 src has 38..195) v4 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.522+0000 7f03f3fff700 3 client.37028.objecter handle_osd_map got epochs [195,195] > 0 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.522+0000 7f03f3fff700 3 client.37028.objecter handle_osd_map decoding full epoch 195 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.522+0000 7f040ad9d700 1 --2- 172.21.15.87:0/487596257 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03f4062c90 0x7f03f4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:26.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.522+0000 7f03f3fff700 20 client.37028.objecter dump_active .. 0 homeless 2024-02-23T00:32:26.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.522+0000 7f03f3fff700 10 client.37028.objecter ms_handle_connect 0x7f03f4062c90 2024-02-23T00:32:26.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.522+0000 7f040ad9d700 1 --2- 172.21.15.87:0/487596257 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03f4062c90 0x7f03f4065150 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7f0400004870 tx=0x7f040000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:26.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.526+0000 7f03f3fff700 1 -- 172.21.15.87:0/487596257 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f04040517e0 con 0x7f040c0fd420 2024-02-23T00:32:26.578 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:26.610 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:26.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:26.713 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:26.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:26.711+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f040c0f9da0 con 0x7f040c0fd420 2024-02-23T00:32:26.742 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:26.838 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:26.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:26.943 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:26.962 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-23T00:32:26.970 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:27.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.003+0000 7f03f3fff700 1 -- 172.21.15.87:0/487596257 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v196) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0404021070 con 0x7f040c0fd420 2024-02-23T00:32:27.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.006+0000 7f04110c4700 10 librados: watch_flush enter 2024-02-23T00:32:27.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.006+0000 7f04110c4700 10 librados: watch_flush exit 2024-02-23T00:32:27.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.006+0000 7f04110c4700 20 client.37028.objecter shutdown clearing up homeless session... 2024-02-23T00:32:27.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.006+0000 7f04110c4700 10 client.37028.objecter successfully canceled tick 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.012+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03f4062c90 msgr2=0x7f03f4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.012+0000 7f04110c4700 1 --2- 172.21.15.87:0/487596257 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03f4062c90 0x7f03f4065150 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 rx=0x7f0400004870 tx=0x7f040000b040).stop 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.013+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f040c0fd420 msgr2=0x7f040c0f92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.013+0000 7f04110c4700 1 --2- 172.21.15.87:0/487596257 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f040c0fd420 0x7f040c0f92a0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f0404004860 tx=0x7f0404004c50).stop 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.013+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 shutdown_connections 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.013+0000 7f04110c4700 1 --2- 172.21.15.87:0/487596257 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03f4062c90 0x7f03f4065150 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.013+0000 7f04110c4700 1 --2- 172.21.15.87:0/487596257 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f040c0fd420 0x7f040c0f92a0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.013+0000 7f04110c4700 1 --2- 172.21.15.87:0/487596257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f040c0f7330 0x7f040c0f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.013+0000 7f04110c4700 1 --2- 172.21.15.87:0/487596257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f040c0fcac0 0x7f040c0f8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:27.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.013+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 >> 172.21.15.87:0/487596257 conn(0x7f040c0f2710 msgr2=0x7f040c0f43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:27.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.013+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 shutdown_connections 2024-02-23T00:32:27.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.014+0000 7f04110c4700 1 -- 172.21.15.87:0/487596257 wait complete. 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout:: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4028 ms) 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3993 ms) 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3979 ms) 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12000 ms total) 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-02-23T00:32:27.017 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [==========] 20 tests from 5 test suites ran. (104981 ms total) 2024-02-23T00:32:27.018 INFO:tasks.workunit.client.0.smithi087.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-02-23T00:32:27.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:27.014+0000 7f04110c4700 1 librados: shutdown 2024-02-23T00:32:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:32:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82070 2024-02-23T00:32:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82070 2024-02-23T00:32:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:32:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82062 2024-02-23T00:32:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82062 2024-02-23T00:32:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:32:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82046 2024-02-23T00:32:27.019 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82046 2024-02-23T00:32:27.021 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:32:27.246 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:27.543 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:25.583384+0000 mgr.y (mgr.24437) 237 : cluster [DBG] pgmap v274: 291 pgs: 32 creating+peering, 14 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 227 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:27.543 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:26.027977+0000 mon.a (mon.0) 1474 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:27.543 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:26.036691+0000 mon.a (mon.0) 1475 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:27.543 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:26.042228+0000 mon.a (mon.0) 1476 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:26.044049+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi087-82011-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:26.052897+0000 mon.b (mon.1) 170 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:26.053159+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:26.426416+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:26.713320+0000 mon.c (mon.2) 223 : audit [INF] from='client.? 172.21.15.87:0/487596257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:26.713597+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:26.995852+0000 mon.a (mon.0) 1480 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:27.002831+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]': finished 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:27.002915+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi087-82011-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:27.002972+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:27.003035+0000 mon.a (mon.0) 1484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:27.003083+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-02-23T00:32:27.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:27.006216+0000 mon.a (mon.0) 1486 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi087-82011-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:27.545 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:27.008261+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:27.545 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:27.008788+0000 mon.a (mon.0) 1487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:25.583384+0000 mgr.y (mgr.24437) 237 : cluster [DBG] pgmap v274: 291 pgs: 32 creating+peering, 14 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 227 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:26.027977+0000 mon.a (mon.0) 1474 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:26.036691+0000 mon.a (mon.0) 1475 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:26.042228+0000 mon.a (mon.0) 1476 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:26.044049+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi087-82011-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:26.052897+0000 mon.b (mon.1) 170 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:26.053159+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:26.426416+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:26.713320+0000 mon.c (mon.2) 223 : audit [INF] from='client.? 172.21.15.87:0/487596257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:26.713597+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:26.995852+0000 mon.a (mon.0) 1480 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:27.002831+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]': finished 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:27.002915+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi087-82011-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:27.002972+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:27.003035+0000 mon.a (mon.0) 1484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:27.003083+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:27.006216+0000 mon.a (mon.0) 1486 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi087-82011-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:27.008261+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:27.008788+0000 mon.a (mon.0) 1487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:27.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:25.583384+0000 mgr.y (mgr.24437) 237 : cluster [DBG] pgmap v274: 291 pgs: 32 creating+peering, 14 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 227 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:27.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:26.027977+0000 mon.a (mon.0) 1474 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:32:27.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:26.036691+0000 mon.a (mon.0) 1475 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:27.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:26.042228+0000 mon.a (mon.0) 1476 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:27.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:26.044049+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi087-82011-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:27.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:26.052897+0000 mon.b (mon.1) 170 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:27.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:26.053159+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:27.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:26.426416+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:27.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:26.713320+0000 mon.c (mon.2) 223 : audit [INF] from='client.? 172.21.15.87:0/487596257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:26.713597+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:26.995852+0000 mon.a (mon.0) 1480 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:27.002831+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? 172.21.15.87:0/1917629916' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi087-82115-20"}]': finished 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:27.002915+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi087-82011-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:27.002972+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:27.003035+0000 mon.a (mon.0) 1484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:27.003083+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:27.006216+0000 mon.a (mon.0) 1486 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi087-82011-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:27.008261+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:27.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:27.008788+0000 mon.a (mon.0) 1487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15"}]: dispatch 2024-02-23T00:32:28.020 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:28.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:28.124 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:28.151 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:27.427284+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:27.997980+0000 mon.a (mon.0) 1488 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:28.247661+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15"}]': finished 2024-02-23T00:32:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:28.247837+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-02-23T00:32:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:28.250986+0000 mon.a (mon.0) 1491 : audit [INF] from='client.? 172.21.15.87:0/1507079809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi087-82019-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:28.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:27.427284+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:28.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:27.997980+0000 mon.a (mon.0) 1488 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:28.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:28.247661+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15"}]': finished 2024-02-23T00:32:28.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:28.247837+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-02-23T00:32:28.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:28.250986+0000 mon.a (mon.0) 1491 : audit [INF] from='client.? 172.21.15.87:0/1507079809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi087-82019-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:28.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:27.427284+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:28.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:27.997980+0000 mon.a (mon.0) 1488 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:28.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:28.247661+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-15"}]': finished 2024-02-23T00:32:28.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:28.247837+0000 mon.a (mon.0) 1490 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-02-23T00:32:28.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:28.250986+0000 mon.a (mon.0) 1491 : audit [INF] from='client.? 172.21.15.87:0/1507079809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi087-82019-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:28.599 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:28.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:28.704 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:28.732 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:28.804 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:28.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:28.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:28.909 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:28.938 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:28.959 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:28.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:29.063 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:29.092 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:32:29.092 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:32:29.092 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:32:29.092 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:32:29.092 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:32:29.095 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.092+0000 7f8386acd700 1 -- 172.21.15.87:0/1481636311 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83800fd410 msgr2=0x7f83800f6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.092+0000 7f8386acd700 1 --2- 172.21.15.87:0/1481636311 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83800fd410 0x7f83800f6c40 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7f8374004830 tx=0x7f8374009cb0).stop 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 -- 172.21.15.87:0/1481636311 shutdown_connections 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 --2- 172.21.15.87:0/1481636311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83800f7320 0x7f8380103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 --2- 172.21.15.87:0/1481636311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83800fcab0 0x7f83800fce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 --2- 172.21.15.87:0/1481636311 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83800fd410 0x7f83800f6c40 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 -- 172.21.15.87:0/1481636311 >> 172.21.15.87:0/1481636311 conn(0x7f83800f26e0 msgr2=0x7f83800f4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 -- 172.21.15.87:0/1481636311 shutdown_connections 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 -- 172.21.15.87:0/1481636311 wait complete. 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 librados: starting msgr at 2024-02-23T00:32:29.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 librados: starting objecter 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 Processor -- start 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 -- start start 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 librados: setting wanted keys 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.093+0000 7f8386acd700 1 librados: calling monclient init 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8386acd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83800f7320 0x7f83801146f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8386acd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83800fcab0 0x7f8380114be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8386acd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83800fd410 0x7f838010e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8386acd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838010edd0 con 0x7f83800f7320 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8386acd700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838010ef10 con 0x7f83800fcab0 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8386acd700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838010f0d0 con 0x7f83800fd410 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8384869700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83800f7320 0x7f83801146f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8384869700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83800f7320 0x7f83801146f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59894/0 (socket says 172.21.15.87:59894) 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8384869700 1 -- 172.21.15.87:0/138349848 learned_addr learned my addr 172.21.15.87:0/138349848 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:29.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f838506a700 1 --2- 172.21.15.87:0/138349848 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83800fd410 0x7f838010e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f838506a700 1 -- 172.21.15.87:0/138349848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83800fcab0 msgr2=0x7f8380114be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f838506a700 1 --2- 172.21.15.87:0/138349848 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f83800fcab0 0x7f8380114be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f838506a700 1 -- 172.21.15.87:0/138349848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83800f7320 msgr2=0x7f83801146f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f838506a700 1 --2- 172.21.15.87:0/138349848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83800f7320 0x7f83801146f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f838506a700 1 -- 172.21.15.87:0/138349848 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8374005040 con 0x7f83800fd410 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.094+0000 7f8384869700 1 --2- 172.21.15.87:0/138349848 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f83800f7320 0x7f83801146f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f838506a700 1 --2- 172.21.15.87:0/138349848 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83800fd410 0x7f838010e8e0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f837800eb70 tx=0x7f837800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f837df76700 10 client.?.objecter ms_handle_connect 0x7f83800fd410 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f837df76700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378009870 con 0x7f83800fd410 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f837800f040 con 0x7f83800fd410 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838010f350 con 0x7f83800fd410 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378007e40 con 0x7f83800fd410 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f8386acd700 10 client.37040.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:29.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838011b4f0 con 0x7f83800fd410 2024-02-23T00:32:29.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.095+0000 7f8386acd700 1 librados: init done 2024-02-23T00:32:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.096+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f838005f9d0 con 0x7f83800fd410 2024-02-23T00:32:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.096+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f83780092e0 con 0x7f83800fd410 2024-02-23T00:32:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.097+0000 7f837df76700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8368062c90 0x7f8368065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.097+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(198..198 src has 38..198) v4 ==== 9010+0+0 (secure 0 0 0) 0x7f837807d340 con 0x7f83800fd410 2024-02-23T00:32:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.097+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 osd_map(198..198 src has 38..198) v4 2024-02-23T00:32:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.097+0000 7f837df76700 3 client.37040.objecter handle_osd_map got epochs [198,198] > 0 2024-02-23T00:32:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.097+0000 7f837df76700 3 client.37040.objecter handle_osd_map decoding full epoch 198 2024-02-23T00:32:29.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.097+0000 7f837df76700 20 client.37040.objecter dump_active .. 0 homeless 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.097+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v198) v1 ==== 54+0+16 (secure 0 0 0) 0x7f8378084070 con 0x7f83800fd410 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8384869700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8368062c90 0x7f8368065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8384869700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8368062c90 0x7f8368065150 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 rx=0x7f8370004d40 tx=0x7f837000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 10 client.37040.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 1 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8380028540 0x7f83800fd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83801143c0 osd=0 3 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83801143c0 osd=0 3 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83801143c0 osd=0 3 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 1 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f83801143c0 osd=0 3 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 0 1 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83801143c0 osd=0 4 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 10 client.37040.objecter _send_command 1 2024-02-23T00:32:29.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.098+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f838010d6c0 con 0x7f8380028540 2024-02-23T00:32:29.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.099+0000 7f837df76700 10 client.37040.objecter ms_handle_connect 0x7f8368062c90 2024-02-23T00:32:29.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.099+0000 7f837ffff700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8380028540 0x7f83800fd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.099+0000 7f837df76700 10 client.37040.objecter ms_handle_connect 0x7f8380028540 2024-02-23T00:32:29.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.099+0000 7f837ffff700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8380028540 0x7f83800fd7f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.100+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f838010d6c0 con 0x7f8380028540 2024-02-23T00:32:29.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.100+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:32:29.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.100+0000 7f837df76700 10 client.37040.objecter _finish_command 1 = system:0 2024-02-23T00:32:29.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.100+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f83801143c0 osd=0 4 2024-02-23T00:32:29.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.100+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 0 1 2024-02-23T00:32:29.118 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:29.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 10 client.37040.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:29.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 2 2024-02-23T00:32:29.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83801143c0 osd=0 3 2024-02-23T00:32:29.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83801143c0 osd=0 3 2024-02-23T00:32:29.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83801143c0 osd=0 3 2024-02-23T00:32:29.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 2 2024-02-23T00:32:29.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f83801143c0 osd=0 3 2024-02-23T00:32:29.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 0 2 2024-02-23T00:32:29.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83801143c0 osd=0 4 2024-02-23T00:32:29.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 10 client.37040.objecter _send_command 2 2024-02-23T00:32:29.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.118+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f838005a040 con 0x7f8380028540 2024-02-23T00:32:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:29.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:29.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f838005a040 con 0x7f8380028540 2024-02-23T00:32:29.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:29.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f837df76700 10 client.37040.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f83801143c0 osd=0 4 2024-02-23T00:32:29.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 0 2 2024-02-23T00:32:29.123 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 10 client.37040.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 3 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6805/685354171 conn(0x7f8380027b90 0x7f8380027f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800fee60 osd=1 3 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fee60 osd=1 3 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800fee60 osd=1 3 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 3 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f83800fee60 osd=1 3 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 1 3 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fee60 osd=1 4 2024-02-23T00:32:29.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 10 client.37040.objecter _send_command 3 2024-02-23T00:32:29.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83800fefd0 con 0x7f8380027b90 2024-02-23T00:32:29.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8384869700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6805/685354171 conn(0x7f8380027b90 0x7f8380027f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f837df76700 10 client.37040.objecter ms_handle_connect 0x7f8380027b90 2024-02-23T00:32:29.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.120+0000 7f8384869700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6805/685354171 conn(0x7f8380027b90 0x7f8380027f70 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.121+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f83800fefd0 con 0x7f8380027b90 2024-02-23T00:32:29.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.121+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:32:29.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.121+0000 7f837df76700 10 client.37040.objecter _finish_command 3 = system:0 2024-02-23T00:32:29.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.121+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f83800fee60 osd=1 4 2024-02-23T00:32:29.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.121+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 1 3 2024-02-23T00:32:29.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 10 client.37040.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:29.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 4 2024-02-23T00:32:29.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800fee60 osd=1 3 2024-02-23T00:32:29.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fee60 osd=1 3 2024-02-23T00:32:29.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800fee60 osd=1 3 2024-02-23T00:32:29.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 4 2024-02-23T00:32:29.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f83800fee60 osd=1 3 2024-02-23T00:32:29.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 1 4 2024-02-23T00:32:29.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fee60 osd=1 4 2024-02-23T00:32:29.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 10 client.37040.objecter _send_command 4 2024-02-23T00:32:29.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.136+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8380027600 con 0x7f8380027b90 2024-02-23T00:32:29.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8380027600 con 0x7f8380027b90 2024-02-23T00:32:29.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:29.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f837df76700 10 client.37040.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f83800fee60 osd=1 4 2024-02-23T00:32:29.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 1 4 2024-02-23T00:32:29.140 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:32:29.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 10 client.37040.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:29.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 5 2024-02-23T00:32:29.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6809/3605394228 conn(0x7f83800fe9c0 0x7f8380122890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800279b0 osd=2 3 2024-02-23T00:32:29.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800279b0 osd=2 3 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800279b0 osd=2 3 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 5 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f83800279b0 osd=2 3 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 2 5 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800279b0 osd=2 4 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 10 client.37040.objecter _send_command 5 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8380125650 con 0x7f83800fe9c0 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.138+0000 7f837ffff700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6809/3605394228 conn(0x7f83800fe9c0 0x7f8380122890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.139+0000 7f837df76700 10 client.37040.objecter ms_handle_connect 0x7f83800fe9c0 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.139+0000 7f837ffff700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6809/3605394228 conn(0x7f83800fe9c0 0x7f8380122890 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.139+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8380125650 con 0x7f83800fe9c0 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.139+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:32:29.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.139+0000 7f837df76700 10 client.37040.objecter _finish_command 5 = system:0 2024-02-23T00:32:29.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.139+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f83800279b0 osd=2 4 2024-02-23T00:32:29.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.139+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 2 5 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 10 client.37040.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 6 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800279b0 osd=2 3 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800279b0 osd=2 3 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800279b0 osd=2 3 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 6 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f83800279b0 osd=2 3 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 2 6 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800279b0 osd=2 4 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 10 client.37040.objecter _send_command 6 2024-02-23T00:32:29.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.154+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f83800fdf20 con 0x7f83800fe9c0 2024-02-23T00:32:29.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.155+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f83800fdf20 con 0x7f83800fe9c0 2024-02-23T00:32:29.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.155+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:29.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.155+0000 7f837df76700 10 client.37040.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.155+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f83800279b0 osd=2 4 2024-02-23T00:32:29.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.155+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 2 6 2024-02-23T00:32:29.158 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.155+0000 7f8386acd700 10 client.37040.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.155+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 7 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.155+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6813/128059790 conn(0x7f83801203e0 0x7f8380123fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800fe330 osd=3 3 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fe330 osd=3 3 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800fe330 osd=3 3 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 7 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f83800fe330 osd=3 3 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 3 7 2024-02-23T00:32:29.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fe330 osd=3 4 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f8386acd700 10 client.37040.objecter _send_command 7 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8380124580 con 0x7f83801203e0 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f838506a700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6813/128059790 conn(0x7f83801203e0 0x7f8380123fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f837df76700 10 client.37040.objecter ms_handle_connect 0x7f83801203e0 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f838506a700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6813/128059790 conn(0x7f83801203e0 0x7f8380123fa0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8380124580 con 0x7f83801203e0 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f837df76700 10 client.37040.objecter _finish_command 7 = system:0 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f83800fe330 osd=3 4 2024-02-23T00:32:29.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.156+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 3 7 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 10 client.37040.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 8 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800fe330 osd=3 3 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fe330 osd=3 3 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f83800fe330 osd=3 3 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 8 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f83800fe330 osd=3 3 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 3 8 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fe330 osd=3 4 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 10 client.37040.objecter _send_command 8 2024-02-23T00:32:29.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.180+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f838011f940 con 0x7f83801203e0 2024-02-23T00:32:29.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.181+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f838011f940 con 0x7f83801203e0 2024-02-23T00:32:29.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.181+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:29.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.181+0000 7f837df76700 10 client.37040.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.181+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f83800fe330 osd=3 4 2024-02-23T00:32:29.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.181+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 3 8 2024-02-23T00:32:29.184 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 10 client.37040.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 9 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8380127880 0x7f8380129c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838011f700 osd=4 3 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f700 osd=4 3 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838011f700 osd=4 3 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 9 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f838011f700 osd=4 3 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 4 9 2024-02-23T00:32:29.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f700 osd=4 4 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 10 client.37040.objecter _send_command 9 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f838011f870 con 0x7f8380127880 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8384869700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8380127880 0x7f8380129c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f8384869700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8380127880 0x7f8380129c70 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.182+0000 7f837df76700 10 client.37040.objecter ms_handle_connect 0x7f8380127880 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.183+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f838011f870 con 0x7f8380127880 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.183+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.183+0000 7f837df76700 10 client.37040.objecter _finish_command 9 = system:0 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.183+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f838011f700 osd=4 4 2024-02-23T00:32:29.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.183+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 4 9 2024-02-23T00:32:29.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 10 client.37040.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:29.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 10 2024-02-23T00:32:29.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838011f700 osd=4 3 2024-02-23T00:32:29.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f700 osd=4 3 2024-02-23T00:32:29.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838011f700 osd=4 3 2024-02-23T00:32:29.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 10 2024-02-23T00:32:29.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f838011f700 osd=4 3 2024-02-23T00:32:29.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 4 10 2024-02-23T00:32:29.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f700 osd=4 4 2024-02-23T00:32:29.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 10 client.37040.objecter _send_command 10 2024-02-23T00:32:29.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.198+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f838011f870 con 0x7f8380127880 2024-02-23T00:32:29.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f838011f870 con 0x7f8380127880 2024-02-23T00:32:29.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:29.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f837df76700 10 client.37040.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f838011f700 osd=4 4 2024-02-23T00:32:29.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 4 10 2024-02-23T00:32:29.202 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:32:29.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 10 client.37040.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:29.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 11 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6804/2550013795 conn(0x7f838012a6b0 0x7f838012aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838011f870 osd=5 3 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f870 osd=5 3 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838011f870 osd=5 3 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 11 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f838011f870 osd=5 3 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 5 11 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f870 osd=5 4 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 10 client.37040.objecter _send_command 11 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.199+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f838012c720 con 0x7f838012a6b0 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.200+0000 7f837ffff700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6804/2550013795 conn(0x7f838012a6b0 0x7f838012aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.200+0000 7f837df76700 10 client.37040.objecter ms_handle_connect 0x7f838012a6b0 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.200+0000 7f837ffff700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6804/2550013795 conn(0x7f838012a6b0 0x7f838012aad0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.201+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f838012c720 con 0x7f838012a6b0 2024-02-23T00:32:29.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.201+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:32:29.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.201+0000 7f837df76700 10 client.37040.objecter _finish_command 11 = system:0 2024-02-23T00:32:29.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.201+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f838011f870 osd=5 4 2024-02-23T00:32:29.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.201+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 5 11 2024-02-23T00:32:29.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 10 client.37040.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:29.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 12 2024-02-23T00:32:29.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838011f870 osd=5 3 2024-02-23T00:32:29.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f870 osd=5 3 2024-02-23T00:32:29.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838011f870 osd=5 3 2024-02-23T00:32:29.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 12 2024-02-23T00:32:29.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f838011f870 osd=5 3 2024-02-23T00:32:29.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 5 12 2024-02-23T00:32:29.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f870 osd=5 4 2024-02-23T00:32:29.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 10 client.37040.objecter _send_command 12 2024-02-23T00:32:29.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.215+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f838012c720 con 0x7f838012a6b0 2024-02-23T00:32:29.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f838012c720 con 0x7f838012a6b0 2024-02-23T00:32:29.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:29.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f837df76700 10 client.37040.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f838011f870 osd=5 4 2024-02-23T00:32:29.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 5 12 2024-02-23T00:32:29.219 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 10 client.37040.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 13 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6808/2509408498 conn(0x7f838012eea0 0x7f838012f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838012c720 osd=6 3 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012c720 osd=6 3 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838012c720 osd=6 3 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 13 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f838012c720 osd=6 3 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 6 13 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012c720 osd=6 4 2024-02-23T00:32:29.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 10 client.37040.objecter _send_command 13 2024-02-23T00:32:29.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8380132c90 con 0x7f838012eea0 2024-02-23T00:32:29.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f838506a700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6808/2509408498 conn(0x7f838012eea0 0x7f838012f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f837df76700 10 client.37040.objecter ms_handle_connect 0x7f838012eea0 2024-02-23T00:32:29.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.217+0000 7f838506a700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6808/2509408498 conn(0x7f838012eea0 0x7f838012f300 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.218+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8380132c90 con 0x7f838012eea0 2024-02-23T00:32:29.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.218+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:32:29.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.218+0000 7f837df76700 10 client.37040.objecter _finish_command 13 = system:0 2024-02-23T00:32:29.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.218+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f838012c720 osd=6 4 2024-02-23T00:32:29.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.218+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 6 13 2024-02-23T00:32:29.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:29.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 10 client.37040.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:29.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 14 2024-02-23T00:32:29.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838012c720 osd=6 3 2024-02-23T00:32:29.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012c720 osd=6 3 2024-02-23T00:32:29.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838012c720 osd=6 3 2024-02-23T00:32:29.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 14 2024-02-23T00:32:29.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f838012c720 osd=6 3 2024-02-23T00:32:29.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 6 14 2024-02-23T00:32:29.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012c720 osd=6 4 2024-02-23T00:32:29.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 10 client.37040.objecter _send_command 14 2024-02-23T00:32:29.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.232+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f838012b080 con 0x7f838012eea0 2024-02-23T00:32:29.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f838012b080 con 0x7f838012eea0 2024-02-23T00:32:29.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:29.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f837df76700 10 client.37040.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f838012c720 osd=6 4 2024-02-23T00:32:29.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 6 14 2024-02-23T00:32:29.236 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:32:29.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 10 client.37040.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:29.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 15 2024-02-23T00:32:29.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6812/373787000 conn(0x7f838012b730 0x7f838012bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:29.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838012b080 osd=7 3 2024-02-23T00:32:29.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012b080 osd=7 3 2024-02-23T00:32:29.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838012b080 osd=7 3 2024-02-23T00:32:29.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 15 2024-02-23T00:32:29.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f838012b080 osd=7 3 2024-02-23T00:32:29.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 7 15 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012b080 osd=7 4 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 10 client.37040.objecter _send_command 15 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f838012c170 con 0x7f838012b730 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.234+0000 7f8384869700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6812/373787000 conn(0x7f838012b730 0x7f838012bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.235+0000 7f837df76700 10 client.37040.objecter ms_handle_connect 0x7f838012b730 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.235+0000 7f8384869700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6812/373787000 conn(0x7f838012b730 0x7f838012bb90 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.235+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f838012c170 con 0x7f838012b730 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.235+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.235+0000 7f837df76700 10 client.37040.objecter _finish_command 15 = system:0 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.235+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f838012b080 osd=7 4 2024-02-23T00:32:29.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.235+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 7 15 2024-02-23T00:32:29.250 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:29.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 10 client.37040.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:29.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign -1 16 2024-02-23T00:32:29.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838012b080 osd=7 3 2024-02-23T00:32:29.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012b080 osd=7 3 2024-02-23T00:32:29.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 20 client.37040.objecter _get_session s=0x7f838012b080 osd=7 3 2024-02-23T00:32:29.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 15 client.37040.objecter _session_command_op_remove -1 16 2024-02-23T00:32:29.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 20 client.37040.objecter get_session s=0x7f838012b080 osd=7 3 2024-02-23T00:32:29.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 15 client.37040.objecter _session_command_op_assign 7 16 2024-02-23T00:32:29.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012b080 osd=7 4 2024-02-23T00:32:29.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 10 client.37040.objecter _send_command 16 2024-02-23T00:32:29.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.251+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f838012c170 con 0x7f838012b730 2024-02-23T00:32:29.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f837df76700 1 -- 172.21.15.87:0/138349848 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f838012c170 con 0x7f838012b730 2024-02-23T00:32:29.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f837df76700 10 client.37040.objecter ms_dispatch 0x7f8380001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:29.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f837df76700 10 client.37040.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f837df76700 20 client.37040.objecter put_session s=0x7f838012b080 osd=7 4 2024-02-23T00:32:29.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f837df76700 15 client.37040.objecter _session_command_op_remove 7 16 2024-02-23T00:32:29.255 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 10 librados: watch_flush enter 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 10 librados: watch_flush exit 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 10 client.37040.objecter close_session for osd.0 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8380028540 msgr2=0x7f83800fd7f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8380028540 0x7f83800fd7f0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83801143c0 osd=0 1 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 10 client.37040.objecter close_session for osd.1 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6805/685354171 conn(0x7f8380027b90 msgr2=0x7f8380027f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6805/685354171 conn(0x7f8380027b90 0x7f8380027f70 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fee60 osd=1 1 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 10 client.37040.objecter close_session for osd.2 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6809/3605394228 conn(0x7f83800fe9c0 msgr2=0x7f8380122890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6809/3605394228 conn(0x7f83800fe9c0 0x7f8380122890 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800279b0 osd=2 1 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 10 client.37040.objecter close_session for osd.3 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6813/128059790 conn(0x7f83801203e0 msgr2=0x7f8380123fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6813/128059790 conn(0x7f83801203e0 0x7f8380123fa0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f83800fe330 osd=3 1 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 10 client.37040.objecter close_session for osd.4 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8380127880 msgr2=0x7f8380129c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8380127880 0x7f8380129c70 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.253+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f700 osd=4 1 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 10 client.37040.objecter close_session for osd.5 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6804/2550013795 conn(0x7f838012a6b0 msgr2=0x7f838012aad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6804/2550013795 conn(0x7f838012a6b0 0x7f838012aad0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838011f870 osd=5 1 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 10 client.37040.objecter close_session for osd.6 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6808/2509408498 conn(0x7f838012eea0 msgr2=0x7f838012f300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6808/2509408498 conn(0x7f838012eea0 0x7f838012f300 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012c720 osd=6 1 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 10 client.37040.objecter close_session for osd.7 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6812/373787000 conn(0x7f838012b730 msgr2=0x7f838012bb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.171:6812/373787000 conn(0x7f838012b730 0x7f838012bb90 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 20 client.37040.objecter put_session s=0x7f838012b080 osd=7 1 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 20 client.37040.objecter shutdown clearing up homeless session... 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 10 client.37040.objecter successfully canceled tick 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f838506a700 1 -- 172.21.15.87:0/138349848 reap_dead start 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f838506a700 1 -- 172.21.15.87:0/138349848 reap_dead start 2024-02-23T00:32:29.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f838506a700 1 -- 172.21.15.87:0/138349848 reap_dead start 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f838506a700 1 -- 172.21.15.87:0/138349848 reap_dead start 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f838506a700 1 -- 172.21.15.87:0/138349848 reap_dead start 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8368062c90 msgr2=0x7f8368065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8368062c90 0x7f8368065150 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 rx=0x7f8370004d40 tx=0x7f837000b040).stop 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83800fd410 msgr2=0x7f838010e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83800fd410 0x7f838010e8e0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7f837800eb70 tx=0x7f837800b040).stop 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 shutdown_connections 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8368062c90 0x7f8368065150 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 --2- 172.21.15.87:0/138349848 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f83800fd410 0x7f838010e8e0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 >> 172.21.15.87:0/138349848 conn(0x7f83800f26e0 msgr2=0x7f83800fa8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:29.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 shutdown_connections 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 -- 172.21.15.87:0/138349848 wait complete. 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:29.254+0000 7f8386acd700 1 librados: shutdown 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.261 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:32:29.269 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:29.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:29.372 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:29.397 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:27.583805+0000 mgr.y (mgr.24437) 238 : cluster [DBG] pgmap v277: 259 pgs: 14 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 227 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:29.397 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:28.428045+0000 mon.c (mon.2) 225 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:29.397 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:29.006411+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi087-82011-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi087-82011-29"}]': finished 2024-02-23T00:32:29.397 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:29.006640+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? 172.21.15.87:0/1507079809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi087-82019-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:29.397 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:29.006717+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-02-23T00:32:29.397 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:29.098208+0000 mon.c (mon.2) 226 : audit [DBG] from='client.? 172.21.15.87:0/138349848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:29.399 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:27.583805+0000 mgr.y (mgr.24437) 238 : cluster [DBG] pgmap v277: 259 pgs: 14 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 227 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:28.428045+0000 mon.c (mon.2) 225 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:29.006411+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi087-82011-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi087-82011-29"}]': finished 2024-02-23T00:32:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:29.006640+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? 172.21.15.87:0/1507079809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi087-82019-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:29.006717+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-02-23T00:32:29.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:29.098208+0000 mon.c (mon.2) 226 : audit [DBG] from='client.? 172.21.15.87:0/138349848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:27.583805+0000 mgr.y (mgr.24437) 238 : cluster [DBG] pgmap v277: 259 pgs: 14 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 227 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:29.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:28.428045+0000 mon.c (mon.2) 225 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:29.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:29.006411+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi087-82011-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi087-82011-29"}]': finished 2024-02-23T00:32:29.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:29.006640+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? 172.21.15.87:0/1507079809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi087-82019-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:29.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:29.006717+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-02-23T00:32:29.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:29.098208+0000 mon.c (mon.2) 226 : audit [DBG] from='client.? 172.21.15.87:0/138349848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:29.878 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-23T00:32:29.927 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:29.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:29.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:30.030 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:30.059 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:30.079 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:30.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:30.183 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:30.197 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:32:30.208 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:30.221 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:32:30.221 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:32:30.221 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-23T00:32:30.221 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-23T00:32:30.227 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:30.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:30.332 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:30.359 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:30.457 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:30.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:30.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:30.488 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:30.550 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:32:30.564 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:30.574 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:29.428921+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:30.574 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:29.585375+0000 mon.a (mon.0) 1495 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:29.887127+0000 osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:29.887337+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:30.005816+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:30.005883+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:30.015508+0000 mon.b (mon.1) 172 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:30.016309+0000 mon.a (mon.0) 1498 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:29.428921+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:29.585375+0000 mon.a (mon.0) 1495 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:29.887127+0000 osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:29.887337+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:30.575 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:30.005816+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-23T00:32:30.576 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:30.005883+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-02-23T00:32:30.576 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:30.015508+0000 mon.b (mon.1) 172 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:30.576 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:30.016309+0000 mon.a (mon.0) 1498 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:30.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:29.428921+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:30.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:29.585375+0000 mon.a (mon.0) 1495 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-23T00:32:30.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:29.887127+0000 osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:30.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:29.887337+0000 osd.7 (osd.7) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:30.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:30.005816+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-23T00:32:30.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:30.005883+0000 mon.a (mon.0) 1497 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-02-23T00:32:30.591 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:30.015508+0000 mon.b (mon.1) 172 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:30.591 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:30.016309+0000 mon.a (mon.0) 1498 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:30.592 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:30.690 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:30.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:30.795 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:30.825 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:30.847 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:30.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:30.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:32:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:32:30.950 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:30.976 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:31.413 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:31.424 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:31.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:31.529 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:32:31.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:29.584403+0000 mgr.y (mgr.24437) 239 : cluster [DBG] pgmap v280: 266 pgs: 14 active+clean+snaptrim, 40 unknown, 18 active+clean+snaptrim_wait, 194 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:31.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:30.427693+0000 osd.2 (osd.2) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:31.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:30.427927+0000 osd.2 (osd.2) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:31.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:30.429666+0000 mon.c (mon.2) 228 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:31.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.008520+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:31.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:31.008742+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-02-23T00:32:31.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.017496+0000 mon.a (mon.0) 1501 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:31.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.018736+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.87:0/1556845654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi087-82019-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:31.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.034286+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:31.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.034477+0000 mon.a (mon.0) 1503 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:31.560 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:31.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:29.584403+0000 mgr.y (mgr.24437) 239 : cluster [DBG] pgmap v280: 266 pgs: 14 active+clean+snaptrim, 40 unknown, 18 active+clean+snaptrim_wait, 194 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:30.427693+0000 osd.2 (osd.2) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:30.427927+0000 osd.2 (osd.2) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:30.429666+0000 mon.c (mon.2) 228 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.008520+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:31.008742+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.017496+0000 mon.a (mon.0) 1501 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.018736+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.87:0/1556845654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi087-82019-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.034286+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.034477+0000 mon.a (mon.0) 1503 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:29.584403+0000 mgr.y (mgr.24437) 239 : cluster [DBG] pgmap v280: 266 pgs: 14 active+clean+snaptrim, 40 unknown, 18 active+clean+snaptrim_wait, 194 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:30.427693+0000 osd.2 (osd.2) 85 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:30.427927+0000 osd.2 (osd.2) 86 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:30.429666+0000 mon.c (mon.2) 228 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.008520+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:31.008742+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-02-23T00:32:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.017496+0000 mon.a (mon.0) 1501 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.018736+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.87:0/1556845654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi087-82019-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.034286+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.034477+0000 mon.a (mon.0) 1503 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:32.012 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:32:32.305 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:32] "GET /metrics HTTP/1.1" 200 190009 "" "Prometheus/2.33.4" 2024-02-23T00:32:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:30.713956+0000 mgr.y (mgr.24437) 240 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.430804+0000 mon.c (mon.2) 229 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.993257+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi087-82011-29"}]': finished 2024-02-23T00:32:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.993324+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? 172.21.15.87:0/1556845654' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi087-82019-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.993447+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:31.993502+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-02-23T00:32:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.993855+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.997869+0000 mon.b (mon.1) 174 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:31.999287+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:30.713956+0000 mgr.y (mgr.24437) 240 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.430804+0000 mon.c (mon.2) 229 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.993257+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi087-82011-29"}]': finished 2024-02-23T00:32:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.993324+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? 172.21.15.87:0/1556845654' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi087-82019-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.993447+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:32.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:31.993502+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-02-23T00:32:32.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.993855+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:32.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.997869+0000 mon.b (mon.1) 174 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:32.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:31.999287+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:32.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:30.713956+0000 mgr.y (mgr.24437) 240 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:32.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.430804+0000 mon.c (mon.2) 229 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:32.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.993257+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi087-82011-29"}]': finished 2024-02-23T00:32:32.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.993324+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? 172.21.15.87:0/1556845654' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi087-82019-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:32.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.993447+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:32.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:31.993502+0000 mon.a (mon.0) 1507 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-02-23T00:32:32.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.993855+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi087-82011-29"}]: dispatch 2024-02-23T00:32:32.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.997869+0000 mon.b (mon.1) 174 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:32.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:31.999287+0000 mon.a (mon.0) 1509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:32.610 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:32:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:32.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:32:32.715 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:32.743 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:32.795 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:32.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:32.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:32.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:32.932 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:33.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-02-23T00:32:33.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [==========] Running 56 tests from 4 test suites. 2024-02-23T00:32:33.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-02-23T00:32:33.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 31 tests from LibRadosAio 2024-02-23T00:32:33.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-02-23T00:32:33.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (7830 ms) 2024-02-23T00:32:33.005 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (24109 ms) 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (7433 ms) 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2636 ms) 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2951 ms) 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3060 ms) 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2932 ms) 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3004 ms) 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-02-23T00:32:33.006 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3008 ms) 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2991 ms) 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3004 ms) 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3005 ms) 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3028 ms) 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2982 ms) 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3001 ms) 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3255 ms) 2024-02-23T00:32:33.007 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2736 ms) 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2997 ms) 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3002 ms) 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3003 ms) 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3009 ms) 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2994 ms) 2024-02-23T00:32:33.008 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-02-23T00:32:33.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3260 ms) 2024-02-23T00:32:33.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-02-23T00:32:33.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2740 ms) 2024-02-23T00:32:33.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-02-23T00:32:33.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3009 ms) 2024-02-23T00:32:33.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-02-23T00:32:33.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3006 ms) 2024-02-23T00:32:33.009 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-02-23T00:32:33.032 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:33.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:33.136 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:33.159 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:32:33.160 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:32:33.164 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:33.241 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:33.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2703660542 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 msgr2=0x7fb3d8104930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2703660542 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 0x7fb3d8104930 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 rx=0x7fb3d0009be0 tx=0x7fb3d0004d10).stop 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2703660542 shutdown_connections 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2703660542 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 0x7fb3d8104930 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2703660542 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3d80fdb70 0x7fb3d80fdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2703660542 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3d80fe440 0x7fb3d80f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2703660542 >> 172.21.15.87:0/2703660542 conn(0x7fb3d80f3610 msgr2=0x7fb3d80f5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2703660542 shutdown_connections 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2703660542 wait complete. 2024-02-23T00:32:33.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 librados: starting msgr at 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 librados: starting objecter 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.247+0000 7fb3dd1a2700 1 Processor -- start 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dd1a2700 1 -- start start 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dd1a2700 1 librados: setting wanted keys 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dd1a2700 1 librados: calling monclient init 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dd1a2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 0x7fb3d806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dd1a2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3d80fdb70 0x7fb3d806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dd1a2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3d80fe440 0x7fb3d806bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dd1a2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d81163c0 con 0x7fb3d80fdb70 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dd1a2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d8116530 con 0x7fb3d80fe440 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dd1a2700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d81166f0 con 0x7fb3d80f8250 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3d80fdb70 0x7fb3d806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d7fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 0x7fb3d806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:33.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3d80fdb70 0x7fb3d806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33300/0 (socket says 172.21.15.87:33300) 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d7fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 0x7fb3d806b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33214/0 (socket says 172.21.15.87:33214) 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d77fe700 1 -- 172.21.15.87:0/2712972799 learned_addr learned my addr 172.21.15.87:0/2712972799 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dc9a1700 1 --2- 172.21.15.87:0/2712972799 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3d80fe440 0x7fb3d806bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d7fff700 1 -- 172.21.15.87:0/2712972799 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3d80fe440 msgr2=0x7fb3d806bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d7fff700 1 --2- 172.21.15.87:0/2712972799 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3d80fe440 0x7fb3d806bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d7fff700 1 -- 172.21.15.87:0/2712972799 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3d80fdb70 msgr2=0x7fb3d806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d7fff700 1 --2- 172.21.15.87:0/2712972799 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3d80fdb70 0x7fb3d806b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3d7fff700 1 -- 172.21.15.87:0/2712972799 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d00097c0 con 0x7fb3d80f8250 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.248+0000 7fb3dc9a1700 1 --2- 172.21.15.87:0/2712972799 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3d80fe440 0x7fb3d806bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3d7fff700 1 --2- 172.21.15.87:0/2712972799 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 0x7fb3d806b0d0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 rx=0x7fb3c800e8f0 tx=0x7fb3c800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3d57fa700 10 client.?.objecter ms_handle_connect 0x7fb3d80f8250 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3d57fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3d57fa700 1 -- 172.21.15.87:0/2712972799 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8007b40 con 0x7fb3d80f8250 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d81169d0 con 0x7fb3d80f8250 2024-02-23T00:32:33.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3d57fa700 1 -- 172.21.15.87:0/2712972799 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3c800f040 con 0x7fb3d80f8250 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3d57fa700 1 -- 172.21.15.87:0/2712972799 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8014710 con 0x7fb3d80f8250 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3dd1a2700 10 client.37064.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d8116f40 con 0x7fb3d80f8250 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.249+0000 7fb3dd1a2700 1 librados: init done 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.250+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d805f8b0 con 0x7fb3d80f8250 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.250+0000 7fb3d57fa700 1 -- 172.21.15.87:0/2712972799 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb3c8014b40 con 0x7fb3d80f8250 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.250+0000 7fb3d57fa700 1 --2- 172.21.15.87:0/2712972799 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3c0062c90 0x7fb3c0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.251+0000 7fb3d57fa700 1 -- 172.21.15.87:0/2712972799 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(202..202 src has 38..202) v4 ==== 8221+0+0 (secure 0 0 0) 0x7fb3c807cdf0 con 0x7fb3d80f8250 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.251+0000 7fb3d57fa700 10 client.37064.objecter ms_dispatch 0x7fb3d8001550 osd_map(202..202 src has 38..202) v4 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.251+0000 7fb3d57fa700 3 client.37064.objecter handle_osd_map got epochs [202,202] > 0 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.251+0000 7fb3d57fa700 3 client.37064.objecter handle_osd_map decoding full epoch 202 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.251+0000 7fb3d77fe700 1 --2- 172.21.15.87:0/2712972799 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3c0062c90 0x7fb3c0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.251+0000 7fb3d57fa700 20 client.37064.objecter dump_active .. 0 homeless 2024-02-23T00:32:33.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.251+0000 7fb3d57fa700 10 client.37064.objecter ms_handle_connect 0x7fb3c0062c90 2024-02-23T00:32:33.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.251+0000 7fb3d77fe700 1 --2- 172.21.15.87:0/2712972799 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3c0062c90 0x7fb3c0065150 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7fb3cc009fd0 tx=0x7fb3cc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:33.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.256+0000 7fb3d57fa700 1 -- 172.21.15.87:0/2712972799 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb3c8049f60 con 0x7fb3d80f8250 2024-02-23T00:32:33.359 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:33.388 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:33.425 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:33.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:33.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.428+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fb3d8047540 con 0x7fb3d80f8250 2024-02-23T00:32:33.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.430+0000 7fb3d57fa700 1 -- 172.21.15.87:0/2712972799 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v202) v1 ==== 81+0+49649 (secure 0 0 0) 0x7fb3c8050220 con 0x7fb3d80f8250 2024-02-23T00:32:33.433 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:32:33.433 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:32:33.433 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 202, 2024-02-23T00:32:33.433 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:32:33.433 INFO:teuthology.orchestra.run.smithi087.stdout: "created": "2024-02-23T00:21:28.861318+0000", 2024-02-23T00:32:33.433 INFO:teuthology.orchestra.run.smithi087.stdout: "modified": "2024-02-23T00:32:32.990993+0000", 2024-02-23T00:32:33.433 INFO:teuthology.orchestra.run.smithi087.stdout: "last_up_change": "2024-02-23T00:27:09.633391+0000", 2024-02-23T00:32:33.433 INFO:teuthology.orchestra.run.smithi087.stdout: "last_in_change": "2024-02-23T00:26:52.603467+0000", 2024-02-23T00:32:33.433 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_num": 5806080, 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_set": [ 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "nodeep-scrub", 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "noscrub", 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "pglog_hardlimit", 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "purged_snapdirs", 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "recovery_deletes", 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "sortbitwise" 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_version": 46, 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "full_ratio": 0.94999998807907104, 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-23T00:32:33.434 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_snapshot": "", 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_max": 140, 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd": 8, 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "require_min_compat_client": "luminous", 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "min_compat_client": "luminous", 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "require_osd_release": "pacific", 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "pools": [ 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "device_health_metrics", 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:25:09.208309+0000", 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.435 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 1, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 1, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 1, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 1, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 1, 2024-02-23T00:32:33.436 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "68", 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.437 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.438 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.439 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_max": 32, 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_min": 1 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_devicehealth": {} 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.440 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 2, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": ".rgw.root", 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.562186+0000", 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.441 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:33.442 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "174", 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:33.443 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.444 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.445 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 3, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "iscsi", 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.989206+0000", 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.446 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 13, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 13, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 13, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 13, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 13, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.447 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "137", 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "135", 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:33.448 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.449 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd": {} 2024-02-23T00:32:33.450 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 4, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.log", 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:17.282440+0000", 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.451 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:33.452 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "58", 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:33.453 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.454 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:33.455 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 5, 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.control", 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:19.262164+0000", 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.456 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:33.457 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "60", 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.458 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.459 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.460 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 6, 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.meta", 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:21.319951+0000", 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.461 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:33.462 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "62", 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.463 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.464 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.464 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:33.464 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:33.464 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:33.464 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:33.464 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.464 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.464 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.464 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.465 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_bias": 4 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.466 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 19, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "LibRadosList_smithi087-82051-1", 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:45.322617+0000", 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.467 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 19, 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 11, 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 11, 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 11, 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 19, 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "19.13", 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 201, 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 200, 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 200, 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "148'11", 2024-02-23T00:32:33.468 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "148'7" 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "201", 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "201", 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "199", 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.469 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.470 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.471 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 22, 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:46.071806+0000", 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.472 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:33.473 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.474 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:33.475 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:33.475 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.475 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.475 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.475 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.476 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.477 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-6", 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:31:04.375891+0000", 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.478 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:33.479 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "202", 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "202", 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "202", 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "202", 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 13, 2024-02-23T00:32:33.480 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 201, 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [ 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: 139 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": 139, 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": 139, 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.481 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.482 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 139, 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-17", 2024-02-23T00:32:33.483 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:32:29.278809+0000", 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:33.484 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:33.485 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "202", 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "202", 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "202", 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "202", 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 13, 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 201, 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:33.486 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": 63, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:33.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:33.488 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7", 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 8, 2024-02-23T00:32:33.489 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 200, 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6802", 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6804", 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6803", 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6801/2153191863", 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6802/2153191863", 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6804/2153191863", 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6803/2153191863", 2024-02-23T00:32:33.492 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "85525ce8-38d5-4840-8057-547cb176c859", 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:33.493 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 13, 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 200, 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6805", 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.494 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6806", 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6808", 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.495 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6807", 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6805/685354171", 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6806/685354171", 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6808/685354171", 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6807/685354171", 2024-02-23T00:32:33.496 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa", 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:33.497 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 18, 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 200, 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6809", 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:33.498 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6810", 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6812", 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:32:33.499 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6811", 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6809/3605394228", 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6810/3605394228", 2024-02-23T00:32:33.500 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6812/3605394228", 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6811/3605394228", 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a", 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:33.501 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 26, 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 200, 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6813", 2024-02-23T00:32:33.502 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6814", 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:33.503 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6816", 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6815", 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:32:33.504 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6813/128059790", 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6814/128059790", 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6816/128059790", 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6815/128059790", 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:32:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c", 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 31, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 200, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.506 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6800", 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6801", 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:33.507 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6803", 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6802", 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:32:33.508 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6800/4175844757", 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6801/4175844757", 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4175844757", 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4175844757", 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:32:33.509 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d", 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 38, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 201, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.510 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6804", 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6805", 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.511 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6807", 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6806", 2024-02-23T00:32:33.512 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6804/2550013795", 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6805/2550013795", 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2550013795", 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2550013795", 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.513 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01", 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 44, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 200, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.514 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6808", 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6809", 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.515 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6811", 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6810", 2024-02-23T00:32:33.516 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6808/2509408498", 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6809/2509408498", 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6811/2509408498", 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6810/2509408498", 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.517 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939", 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 51, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 200, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6812", 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6813", 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.519 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6815", 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.520 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6814", 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6812/373787000", 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6813/373787000", 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6815/373787000", 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6814/373787000", 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.521 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_xinfo": [ 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:31:30.371681+0000", 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.522 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:09.518145+0000", 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:33.523 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:30.427929+0000", 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:06.367304+0000", 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:33.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:31:11.892843+0000", 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:32:33.525 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:15.088109+0000", 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:32:33.527 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:31:32.030302+0000", 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:33.528 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:29.887338+0000", 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap": [], 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap_items": [ 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.9", 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.529 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.13", 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.17", 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:32:33.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 5 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_temp": [], 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_temp": [], 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "blocklist": { 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1121018082": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/2138166229": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/27794119": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/1675126043": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:32:33.531 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/3767016525": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/969918521": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/795095363": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2273151327": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1685406414": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2093733106": "2024-02-24T00:21:58.144574+0000" 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "range_blocklist": {}, 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profiles": { 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "default": { 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:32:33.532 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van" 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps_queue": [ 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 7, 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 21, 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.534 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 61, 2024-02-23T00:32:33.535 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.536 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 4, 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 4 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 78, 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.538 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.538 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:32:33.538 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.539 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.539 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.539 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 86, 2024-02-23T00:32:33.539 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.539 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:32:33.540 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 95, 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 4, 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 4 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 107, 2024-02-23T00:32:33.541 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 112, 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.542 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 125, 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:33.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 10, 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 133, 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:33.544 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "new_removed_snaps": [], 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "new_purged_snaps": [], 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_node_flags": {}, 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "device_class_flags": {}, 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode": { 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_enabled": false, 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_bucket_count": 0, 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "degraded_stretch_mode": 0, 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "recovering_stretch_mode": 0, 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_bucket": 0 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:33.545 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:32:33.546 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:31.584829+0000 mgr.y (mgr.24437) 241 : cluster [DBG] pgmap v283: 290 pgs: 32 creating+peering, 2 peering, 14 active+clean+snaptrim, 32 unknown, 18 active+clean+snaptrim_wait, 192 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:32.431683+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:32.997193+0000 mon.a (mon.0) 1510 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi087-82011-29"}]': finished 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:32.997245+0000 mon.a (mon.0) 1511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-17"}]': finished 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:32.997330+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.000362+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-17", "mode": "writeback"}]: dispatch 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.002102+0000 mon.a (mon.0) 1513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-17", "mode": "writeback"}]: dispatch 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.020843+0000 mon.b (mon.1) 176 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.268915+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.269839+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.547 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.270001+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.548 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.270514+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi087-82011-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:33.548 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.270630+0000 mon.a (mon.0) 1516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi087-82011-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 10 librados: watch_flush enter 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 10 librados: watch_flush exit 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 20 client.37064.objecter shutdown clearing up homeless session... 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 10 client.37064.objecter successfully canceled tick 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3c0062c90 msgr2=0x7fb3c0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2712972799 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3c0062c90 0x7fb3c0065150 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 rx=0x7fb3cc009fd0 tx=0x7fb3cc00b040).stop 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 msgr2=0x7fb3d806b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2712972799 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 0x7fb3d806b0d0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 rx=0x7fb3c800e8f0 tx=0x7fb3c800b040).stop 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 shutdown_connections 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2712972799 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3c0062c90 0x7fb3c0065150 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2712972799 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3d80fe440 0x7fb3d806bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2712972799 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3d80f8250 0x7fb3d806b0d0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:33.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 --2- 172.21.15.87:0/2712972799 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3d80fdb70 0x7fb3d806b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:33.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 >> 172.21.15.87:0/2712972799 conn(0x7fb3d80f3610 msgr2=0x7fb3d80f4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:33.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 shutdown_connections 2024-02-23T00:32:33.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 -- 172.21.15.87:0/2712972799 wait complete. 2024-02-23T00:32:33.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:33.432+0000 7fb3dd1a2700 1 librados: shutdown 2024-02-23T00:32:33.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:31.584829+0000 mgr.y (mgr.24437) 241 : cluster [DBG] pgmap v283: 290 pgs: 32 creating+peering, 2 peering, 14 active+clean+snaptrim, 32 unknown, 18 active+clean+snaptrim_wait, 192 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:33.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:32.431683+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:33.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:32.997193+0000 mon.a (mon.0) 1510 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi087-82011-29"}]': finished 2024-02-23T00:32:33.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:32.997245+0000 mon.a (mon.0) 1511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-17"}]': finished 2024-02-23T00:32:33.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:32.997330+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.000362+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-17", "mode": "writeback"}]: dispatch 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.002102+0000 mon.a (mon.0) 1513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-17", "mode": "writeback"}]: dispatch 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.020843+0000 mon.b (mon.1) 176 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.268915+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.269839+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.270001+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.270514+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi087-82011-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.270630+0000 mon.a (mon.0) 1516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi087-82011-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:31.584829+0000 mgr.y (mgr.24437) 241 : cluster [DBG] pgmap v283: 290 pgs: 32 creating+peering, 2 peering, 14 active+clean+snaptrim, 32 unknown, 18 active+clean+snaptrim_wait, 192 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:32.431683+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:32.997193+0000 mon.a (mon.0) 1510 : audit [INF] from='client.? 172.21.15.87:0/2847938836' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi087-82011-29"}]': finished 2024-02-23T00:32:33.551 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:32.997245+0000 mon.a (mon.0) 1511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-17"}]': finished 2024-02-23T00:32:33.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:32.997330+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-02-23T00:32:33.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.000362+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-17", "mode": "writeback"}]: dispatch 2024-02-23T00:32:33.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.002102+0000 mon.a (mon.0) 1513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-17", "mode": "writeback"}]: dispatch 2024-02-23T00:32:33.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.020843+0000 mon.b (mon.1) 176 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.268915+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.269839+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.270001+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:33.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.270514+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi087-82011-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:33.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.270630+0000 mon.a (mon.0) 1516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi087-82011-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:33.574 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:33.593 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:33.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:33.704 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:33.736 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:33.922 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:32:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:32:33.940 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-23T00:32:33.979 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:33.984 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:33.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:34.088 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:34.240 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:34.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:34.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:34.314 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.430253+0000 mon.c (mon.2) 231 : audit [DBG] from='client.? 172.21.15.87:0/2712972799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:32:34.314 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.432390+0000 mon.c (mon.2) 232 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:34.314 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:33.586379+0000 mon.a (mon.0) 1517 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-23T00:32:34.345 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:34.375 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:34.394 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:34.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:34.518 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:34.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.430253+0000 mon.c (mon.2) 231 : audit [DBG] from='client.? 172.21.15.87:0/2712972799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:32:34.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.432390+0000 mon.c (mon.2) 232 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:34.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:33.586379+0000 mon.a (mon.0) 1517 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-23T00:32:34.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:33.993584+0000 mon.a (mon.0) 1518 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:34.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.001025+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-17", "mode": "writeback"}]': finished 2024-02-23T00:32:34.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.001098+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi087-82011-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.001157+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:34.001206+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.012846+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.87:0/900449550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi087-82019-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.014458+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi087-82019-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.014468+0000 mon.b (mon.1) 179 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi087-82011-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.014928+0000 mon.a (mon.0) 1524 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi087-82011-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.057921+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.058215+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.430253+0000 mon.c (mon.2) 231 : audit [DBG] from='client.? 172.21.15.87:0/2712972799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.432390+0000 mon.c (mon.2) 232 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:34.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:33.586379+0000 mon.a (mon.0) 1517 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:33.993584+0000 mon.a (mon.0) 1518 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.001025+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-17", "mode": "writeback"}]': finished 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.001098+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi087-82011-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.001157+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:34.001206+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.012846+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.87:0/900449550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi087-82019-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.014458+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi087-82019-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.014468+0000 mon.b (mon.1) 179 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi087-82011-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.014928+0000 mon.a (mon.0) 1524 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi087-82011-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.057921+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:34.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.058215+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:34.548 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.610+0000 7f8a828af700 1 -- 172.21.15.87:0/2400653851 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fd4a0 msgr2=0x7f8a7c0f74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.610+0000 7f8a828af700 1 --2- 172.21.15.87:0/2400653851 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fd4a0 0x7f8a7c0f74c0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7f8a70004870 tx=0x7f8a70009cb0).stop 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.610+0000 7f8a828af700 1 -- 172.21.15.87:0/2400653851 shutdown_connections 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.610+0000 7f8a828af700 1 --2- 172.21.15.87:0/2400653851 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c0f7ba0 0x7f8a7c103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.610+0000 7f8a828af700 1 --2- 172.21.15.87:0/2400653851 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a7c0fcb40 0x7f8a7c0fcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.610+0000 7f8a828af700 1 --2- 172.21.15.87:0/2400653851 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fd4a0 0x7f8a7c0f74c0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.610+0000 7f8a828af700 1 -- 172.21.15.87:0/2400653851 >> 172.21.15.87:0/2400653851 conn(0x7f8a7c06c370 msgr2=0x7f8a7c06c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.610+0000 7f8a828af700 1 -- 172.21.15.87:0/2400653851 shutdown_connections 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.610+0000 7f8a828af700 1 -- 172.21.15.87:0/2400653851 wait complete. 2024-02-23T00:32:34.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 librados: starting msgr at 2024-02-23T00:32:34.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 librados: starting objecter 2024-02-23T00:32:34.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 Processor -- start 2024-02-23T00:32:34.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 -- start start 2024-02-23T00:32:34.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 librados: setting wanted keys 2024-02-23T00:32:34.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 librados: calling monclient init 2024-02-23T00:32:34.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c0f7ba0 0x7f8a7c107070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fcb40 0x7f8a7c107560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a7c0fd4a0 0x7f8a7c107a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c1166f0 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c116830 con 0x7f8a7c0f7ba0 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a828af700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7c1169a0 con 0x7f8a7c0fcb40 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a7b7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fcb40 0x7f8a7c107560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.611+0000 7f8a80e4c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a7c0fd4a0 0x7f8a7c107a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a80e4c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a7c0fd4a0 0x7f8a7c107a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33318/0 (socket says 172.21.15.87:33318) 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a7b7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fcb40 0x7f8a7c107560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33244/0 (socket says 172.21.15.87:33244) 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a80e4c700 1 -- 172.21.15.87:0/2163695154 learned_addr learned my addr 172.21.15.87:0/2163695154 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a7bfff700 1 --2- 172.21.15.87:0/2163695154 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c0f7ba0 0x7f8a7c107070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a80e4c700 1 -- 172.21.15.87:0/2163695154 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fcb40 msgr2=0x7f8a7c107560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:34.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a80e4c700 1 --2- 172.21.15.87:0/2163695154 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fcb40 0x7f8a7c107560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a80e4c700 1 -- 172.21.15.87:0/2163695154 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c0f7ba0 msgr2=0x7f8a7c107070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a80e4c700 1 --2- 172.21.15.87:0/2163695154 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c0f7ba0 0x7f8a7c107070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a80e4c700 1 -- 172.21.15.87:0/2163695154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c005040 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a7b7fe700 1 --2- 172.21.15.87:0/2163695154 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fcb40 0x7f8a7c107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a80e4c700 1 --2- 172.21.15.87:0/2163695154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a7c0fd4a0 0x7f8a7c107a50 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f8a7400d970 tx=0x7f8a74004e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a797fa700 10 client.?.objecter ms_handle_connect 0x7f8a7c0fd4a0 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a797fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a797fa700 1 -- 172.21.15.87:0/2163695154 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a74004020 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a70005040 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a797fa700 1 -- 172.21.15.87:0/2163695154 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a7400f040 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a797fa700 1 -- 172.21.15.87:0/2163695154 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a740036c0 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a828af700 10 client.26616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a7c117030 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.612+0000 7f8a828af700 1 librados: init done 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.613+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7c05fa40 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.613+0000 7f8a797fa700 1 -- 172.21.15.87:0/2163695154 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f8a74003af0 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.614+0000 7f8a797fa700 1 --2- 172.21.15.87:0/2163695154 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a64062c90 0x7f8a64065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.614+0000 7f8a797fa700 1 -- 172.21.15.87:0/2163695154 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(203..203 src has 38..203) v4 ==== 8884+0+0 (secure 0 0 0) 0x7f8a7407cf40 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.614+0000 7f8a797fa700 10 client.26616.objecter ms_dispatch 0x7f8a7c001550 osd_map(203..203 src has 38..203) v4 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.614+0000 7f8a797fa700 3 client.26616.objecter handle_osd_map got epochs [203,203] > 0 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.614+0000 7f8a797fa700 3 client.26616.objecter handle_osd_map decoding full epoch 203 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.614+0000 7f8a7bfff700 1 --2- 172.21.15.87:0/2163695154 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a64062c90 0x7f8a64065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.614+0000 7f8a797fa700 20 client.26616.objecter dump_active .. 0 homeless 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.614+0000 7f8a797fa700 10 client.26616.objecter ms_handle_connect 0x7f8a64062c90 2024-02-23T00:32:34.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.614+0000 7f8a7bfff700 1 --2- 172.21.15.87:0/2163695154 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a64062c90 0x7f8a64065150 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 rx=0x7f8a6c009350 tx=0x7f8a6c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:34.620 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:34.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.619+0000 7f8a797fa700 1 -- 172.21.15.87:0/2163695154 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8a74048f60 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:34.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:33.993584+0000 mon.a (mon.0) 1518 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.001025+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-17", "mode": "writeback"}]': finished 2024-02-23T00:32:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.001098+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi087-82011-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.001157+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-23T00:32:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:34.001206+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-02-23T00:32:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.012846+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.87:0/900449550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi087-82019-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.014458+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi087-82019-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.014468+0000 mon.b (mon.1) 179 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi087-82011-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.014928+0000 mon.a (mon.0) 1524 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi087-82011-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.057921+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.058215+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:34.723 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:34.751 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:34.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:34.799+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f8a7c106100 con 0x7f8a7c0fd4a0 2024-02-23T00:32:34.993 INFO:tasks.thrashosds.thrasher:j is {'epoch': 202, 'fsid': '66db207c-d1e1-11ee-95c0-87774f69a715', 'created': '2024-02-23T00:21:28.861318+0000', 'modified': '2024-02-23T00:32:32.990993+0000', 'last_up_change': '2024-02-23T00:27:09.633391+0000', 'last_in_change': '2024-02-23T00:26:52.603467+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 46, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 140, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '135', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 19, 'pool_name': 'LibRadosList_smithi087-82051-1', 'create_time': '2024-02-23T00:30:45.322617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 19, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 19, 'last_pg_merge_meta': {'source_pgid': '19.13', 'ready_epoch': 201, 'last_epoch_started': 200, 'last_epoch_clean': 200, 'source_version': "148'11", 'target_version': "148'7"}, 'last_change': '201', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '201', 'last_force_op_resend_preluminous': '199', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 22, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-23T00:30:46.071806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 63, 'pool_name': 'test-rados-api-smithi087-82091-6', 'create_time': '2024-02-23T00:31:04.375891+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '202', 'last_force_op_resend': '202', 'last_force_op_resend_prenautilus': '202', 'last_force_op_resend_preluminous': '202', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 201, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [139], 'tier_of': -1, 'read_tier': 139, 'write_tier': 139, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 139, 'pool_name': 'test-rados-api-smithi087-82091-17', 'create_time': '2024-02-23T00:32:29.278809+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '202', 'last_force_op_resend': '202', 'last_force_op_resend_prenautilus': '202', 'last_force_op_resend_preluminous': '202', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 201, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 63, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e3549bad-2891-47b9-bf77-132cadc20bf7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 200, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6801', 'nonce': 2153191863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6802', 'nonce': 2153191863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6804', 'nonce': 2153191863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6803', 'nonce': 2153191863}]}, 'public_addr': '172.21.15.87:6801/2153191863', 'cluster_addr': '172.21.15.87:6802/2153191863', 'heartbeat_back_addr': '172.21.15.87:6804/2153191863', 'heartbeat_front_addr': '172.21.15.87:6803/2153191863', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '85525ce8-38d5-4840-8057-547cb176c859', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 200, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6805', 'nonce': 685354171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6806', 'nonce': 685354171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6808', 'nonce': 685354171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6807', 'nonce': 685354171}]}, 'public_addr': '172.21.15.87:6805/685354171', 'cluster_addr': '172.21.15.87:6806/685354171', 'heartbeat_back_addr': '172.21.15.87:6808/685354171', 'heartbeat_front_addr': '172.21.15.87:6807/685354171', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 200, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6809', 'nonce': 3605394228}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6810', 'nonce': 3605394228}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6812', 'nonce': 3605394228}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6811', 'nonce': 3605394228}]}, 'public_addr': '172.21.15.87:6809/3605394228', 'cluster_addr': '172.21.15.87:6810/3605394228', 'heartbeat_back_addr': '172.21.15.87:6812/3605394228', 'heartbeat_front_addr': '172.21.15.87:6811/3605394228', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 200, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6813', 'nonce': 128059790}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6814', 'nonce': 128059790}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6816', 'nonce': 128059790}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6815', 'nonce': 128059790}]}, 'public_addr': '172.21.15.87:6813/128059790', 'cluster_addr': '172.21.15.87:6814/128059790', 'heartbeat_back_addr': '172.21.15.87:6816/128059790', 'heartbeat_front_addr': '172.21.15.87:6815/128059790', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c3734a04-7343-4251-9b5b-bedb9e8b882c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 200, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6800', 'nonce': 4175844757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6801', 'nonce': 4175844757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6803', 'nonce': 4175844757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6802', 'nonce': 4175844757}]}, 'public_addr': '172.21.15.171:6800/4175844757', 'cluster_addr': '172.21.15.171:6801/4175844757', 'heartbeat_back_addr': '172.21.15.171:6803/4175844757', 'heartbeat_front_addr': '172.21.15.171:6802/4175844757', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fe86232d-96b9-4cbf-9cac-373286fa103d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 201, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6804', 'nonce': 2550013795}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6805', 'nonce': 2550013795}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6807', 'nonce': 2550013795}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6806', 'nonce': 2550013795}]}, 'public_addr': '172.21.15.171:6804/2550013795', 'cluster_addr': '172.21.15.171:6805/2550013795', 'heartbeat_back_addr': '172.21.15.171:6807/2550013795', 'heartbeat_front_addr': '172.21.15.171:6806/2550013795', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 200, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6808', 'nonce': 2509408498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6809', 'nonce': 2509408498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6811', 'nonce': 2509408498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6810', 'nonce': 2509408498}]}, 'public_addr': '172.21.15.171:6808/2509408498', 'cluster_addr': '172.21.15.171:6809/2509408498', 'heartbeat_back_addr': '172.21.15.171:6811/2509408498', 'heartbeat_front_addr': '172.21.15.171:6810/2509408498', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '29ddcf97-369a-4a2c-b3e2-142e84ccf939', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 200, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6812', 'nonce': 373787000}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6813', 'nonce': 373787000}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6815', 'nonce': 373787000}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6814', 'nonce': 373787000}]}, 'public_addr': '172.21.15.171:6812/373787000', 'cluster_addr': '172.21.15.171:6813/373787000', 'heartbeat_back_addr': '172.21.15.171:6815/373787000', 'heartbeat_front_addr': '172.21.15.171:6814/373787000', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:31:30.371681+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:09.518145+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:30.427929+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:06.367304+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:31:11.892843+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:15.088109+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:31:32.030302+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:29.887338+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.87:6800/1121018082': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/2138166229': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/27794119': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/1675126043': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/3767016525': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/969918521': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/795095363': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2273151327': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/1685406414': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2093733106': '2024-02-24T00:21:58.144574+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 21, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 61, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 63, 'snaps': [{'begin': 4, 'length': 4}, {'begin': 12, 'length': 1}]}, {'pool': 78, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 86, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-23T00:32:34.993 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:32:35.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.250+0000 7f8a797fa700 1 -- 172.21.15.87:0/2163695154 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v204) v1 ==== 91+0+0 (secure 0 0 0) 0x7f8a740500d0 con 0x7f8a7c0fd4a0 2024-02-23T00:32:35.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 10 librados: watch_flush enter 2024-02-23T00:32:35.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 10 librados: watch_flush exit 2024-02-23T00:32:35.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 20 client.26616.objecter shutdown clearing up homeless session... 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 10 client.26616.objecter successfully canceled tick 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a64062c90 msgr2=0x7f8a64065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 --2- 172.21.15.87:0/2163695154 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a64062c90 0x7f8a64065150 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 rx=0x7f8a6c009350 tx=0x7f8a6c00b040).stop 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a7c0fd4a0 msgr2=0x7f8a7c107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 --2- 172.21.15.87:0/2163695154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a7c0fd4a0 0x7f8a7c107a50 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 rx=0x7f8a7400d970 tx=0x7f8a74004e60).stop 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 shutdown_connections 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 --2- 172.21.15.87:0/2163695154 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a64062c90 0x7f8a64065150 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 --2- 172.21.15.87:0/2163695154 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8a7c0fd4a0 0x7f8a7c107a50 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 --2- 172.21.15.87:0/2163695154 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8a7c0f7ba0 0x7f8a7c107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 --2- 172.21.15.87:0/2163695154 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8a7c0fcb40 0x7f8a7c107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 >> 172.21.15.87:0/2163695154 conn(0x7f8a7c06c370 msgr2=0x7f8a7c0fa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:35.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 shutdown_connections 2024-02-23T00:32:35.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 -- 172.21.15.87:0/2163695154 wait complete. 2024-02-23T00:32:35.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:35.253+0000 7f8a828af700 1 librados: shutdown 2024-02-23T00:32:35.257 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:32:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:35.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:35.405 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:35.433 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:33.585368+0000 mgr.y (mgr.24437) 242 : cluster [DBG] pgmap v286: 257 pgs: 32 creating+peering, 1 peering, 14 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 192 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.433332+0000 mon.c (mon.2) 234 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:34.801155+0000 mon.a (mon.0) 1526 : audit [INF] from='client.? 172.21.15.87:0/2163695154' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:32:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:34.997415+0000 mon.a (mon.0) 1527 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:35.005173+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi087-82019-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:35.251579+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:35.251641+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? 172.21.15.87:0/2163695154' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:32:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.251691+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-02-23T00:32:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:35.252926+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:35.254747+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:33.585368+0000 mgr.y (mgr.24437) 242 : cluster [DBG] pgmap v286: 257 pgs: 32 creating+peering, 1 peering, 14 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 192 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.433332+0000 mon.c (mon.2) 234 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:34.801155+0000 mon.a (mon.0) 1526 : audit [INF] from='client.? 172.21.15.87:0/2163695154' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:32:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:34.997415+0000 mon.a (mon.0) 1527 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:35.005173+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi087-82019-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:35.251579+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:35.251641+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? 172.21.15.87:0/2163695154' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:32:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.251691+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-02-23T00:32:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:35.252926+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:35.254747+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:33.585368+0000 mgr.y (mgr.24437) 242 : cluster [DBG] pgmap v286: 257 pgs: 32 creating+peering, 1 peering, 14 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 192 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.433332+0000 mon.c (mon.2) 234 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:34.801155+0000 mon.a (mon.0) 1526 : audit [INF] from='client.? 172.21.15.87:0/2163695154' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:32:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:34.997415+0000 mon.a (mon.0) 1527 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:32:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:35.005173+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi087-82019-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:35.251579+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:35.251641+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? 172.21.15.87:0/2163695154' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:32:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.251691+0000 mon.a (mon.0) 1531 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-02-23T00:32:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:35.252926+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:35.254747+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17"}]: dispatch 2024-02-23T00:32:35.981 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:35.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:36.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:36.118 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:36.139 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:36.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:36.243 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:36.272 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:36.291 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:36.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:36.396 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:36.422 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:35.434110+0000 mon.c (mon.2) 235 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.495990+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.496332+0000 osd.6 (osd.6) 80 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.722288+0000 osd.7 (osd.7) 101 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:32:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.725178+0000 osd.7 (osd.7) 102 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:32:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.972410+0000 osd.4 (osd.4) 95 : cluster [DBG] 5.e scrub starts 2024-02-23T00:32:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.975882+0000 osd.4 (osd.4) 96 : cluster [DBG] 5.e scrub ok 2024-02-23T00:32:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.998955+0000 mon.a (mon.0) 1533 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:36.003355+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi087-82011-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi087-82011-30"}]': finished 2024-02-23T00:32:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:36.003502+0000 mon.a (mon.0) 1535 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17"}]': finished 2024-02-23T00:32:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:36.003583+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-02-23T00:32:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:36.288871+0000 mon.a (mon.0) 1537 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:32:36.444 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:36.445 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:36.448 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:36.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:36.556 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:35.434110+0000 mon.c (mon.2) 235 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.495990+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.496332+0000 osd.6 (osd.6) 80 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.722288+0000 osd.7 (osd.7) 101 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.725178+0000 osd.7 (osd.7) 102 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.972410+0000 osd.4 (osd.4) 95 : cluster [DBG] 5.e scrub starts 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.975882+0000 osd.4 (osd.4) 96 : cluster [DBG] 5.e scrub ok 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.998955+0000 mon.a (mon.0) 1533 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:36.003355+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi087-82011-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi087-82011-30"}]': finished 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:36.003502+0000 mon.a (mon.0) 1535 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17"}]': finished 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:36.003583+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-02-23T00:32:36.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:36.288871+0000 mon.a (mon.0) 1537 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:35.434110+0000 mon.c (mon.2) 235 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.495990+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.496332+0000 osd.6 (osd.6) 80 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.722288+0000 osd.7 (osd.7) 101 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.725178+0000 osd.7 (osd.7) 102 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.972410+0000 osd.4 (osd.4) 95 : cluster [DBG] 5.e scrub starts 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.975882+0000 osd.4 (osd.4) 96 : cluster [DBG] 5.e scrub ok 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.998955+0000 mon.a (mon.0) 1533 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:36.003355+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi087-82011-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi087-82011-30"}]': finished 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:36.003502+0000 mon.a (mon.0) 1535 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-17"}]': finished 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:36.003583+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-02-23T00:32:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:36.288871+0000 mon.a (mon.0) 1537 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:32:36.584 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:36.782 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:36.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:36.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:36.885 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:36.914 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:36.933 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:36.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:37.037 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:37.064 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:37.083 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:37.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:37.187 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:37.216 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:37.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.586316+0000 mgr.y (mgr.24437) 243 : cluster [DBG] pgmap v289: 289 pgs: 2 clean+premerge+peered, 260 active+clean, 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-23T00:32:37.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.623627+0000 osd.0 (osd.0) 117 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:32:37.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.624456+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:32:37.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.927448+0000 osd.2 (osd.2) 87 : cluster [DBG] 5.d scrub starts 2024-02-23T00:32:37.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:35.928972+0000 osd.2 (osd.2) 88 : cluster [DBG] 5.d scrub ok 2024-02-23T00:32:37.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:36.434915+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:37.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:36.446373+0000 osd.5 (osd.5) 93 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:32:37.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:36.447435+0000 osd.5 (osd.5) 94 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:32:37.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:36.711182+0000 osd.7 (osd.7) 103 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:36.714136+0000 osd.7 (osd.7) 104 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:36.998676+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:37.018423+0000 mon.a (mon.0) 1539 : audit [INF] from='client.? 172.21.15.87:0/4189795763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi087-82019-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:37.287058+0000 mon.a (mon.0) 1540 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.586316+0000 mgr.y (mgr.24437) 243 : cluster [DBG] pgmap v289: 289 pgs: 2 clean+premerge+peered, 260 active+clean, 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.623627+0000 osd.0 (osd.0) 117 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.624456+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.927448+0000 osd.2 (osd.2) 87 : cluster [DBG] 5.d scrub starts 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:35.928972+0000 osd.2 (osd.2) 88 : cluster [DBG] 5.d scrub ok 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:36.434915+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:37.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:36.446373+0000 osd.5 (osd.5) 93 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:32:37.399 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:36.447435+0000 osd.5 (osd.5) 94 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:32:37.399 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:36.711182+0000 osd.7 (osd.7) 103 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:32:37.399 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:36.714136+0000 osd.7 (osd.7) 104 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:32:37.399 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:36.998676+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-02-23T00:32:37.399 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:37.018423+0000 mon.a (mon.0) 1539 : audit [INF] from='client.? 172.21.15.87:0/4189795763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi087-82019-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:37.399 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:37.287058+0000 mon.a (mon.0) 1540 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.586316+0000 mgr.y (mgr.24437) 243 : cluster [DBG] pgmap v289: 289 pgs: 2 clean+premerge+peered, 260 active+clean, 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-23T00:32:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.623627+0000 osd.0 (osd.0) 117 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:32:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.624456+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:32:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.927448+0000 osd.2 (osd.2) 87 : cluster [DBG] 5.d scrub starts 2024-02-23T00:32:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:35.928972+0000 osd.2 (osd.2) 88 : cluster [DBG] 5.d scrub ok 2024-02-23T00:32:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:36.434915+0000 mon.c (mon.2) 236 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:36.446373+0000 osd.5 (osd.5) 93 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:32:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:36.447435+0000 osd.5 (osd.5) 94 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:32:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:36.711182+0000 osd.7 (osd.7) 103 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:32:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:36.714136+0000 osd.7 (osd.7) 104 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:32:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:36.998676+0000 mon.a (mon.0) 1538 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-02-23T00:32:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:37.018423+0000 mon.a (mon.0) 1539 : audit [INF] from='client.? 172.21.15.87:0/4189795763' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi087-82019-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:37.287058+0000 mon.a (mon.0) 1540 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:37.965 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:37.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:38.070 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:38.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:37.437118+0000 mon.c (mon.2) 237 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:38.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:37.439133+0000 osd.5 (osd.5) 95 : cluster [DBG] 3.c scrub starts 2024-02-23T00:32:38.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:37.440610+0000 osd.5 (osd.5) 96 : cluster [DBG] 3.c scrub ok 2024-02-23T00:32:38.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:37.442027+0000 osd.6 (osd.6) 81 : cluster [DBG] 3.a scrub starts 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:37.444965+0000 osd.6 (osd.6) 82 : cluster [DBG] 3.a scrub ok 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:37.586751+0000 mgr.y (mgr.24437) 244 : cluster [DBG] pgmap v292: 264 pgs: 40 unknown, 1 clean+premerge+peered, 196 active+clean, 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:37.587784+0000 mon.a (mon.0) 1541 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:37.998188+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? 172.21.15.87:0/4189795763' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi087-82019-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:37.998311+0000 mon.a (mon.0) 1543 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:37.998379+0000 mon.a (mon.0) 1544 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:38.005347+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:38.009000+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:38.009907+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:38.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:38.011780+0000 mon.a (mon.0) 1546 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:38.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:37.437118+0000 mon.c (mon.2) 237 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:38.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:37.439133+0000 osd.5 (osd.5) 95 : cluster [DBG] 3.c scrub starts 2024-02-23T00:32:38.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:37.440610+0000 osd.5 (osd.5) 96 : cluster [DBG] 3.c scrub ok 2024-02-23T00:32:38.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:37.442027+0000 osd.6 (osd.6) 81 : cluster [DBG] 3.a scrub starts 2024-02-23T00:32:38.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:37.444965+0000 osd.6 (osd.6) 82 : cluster [DBG] 3.a scrub ok 2024-02-23T00:32:38.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:37.586751+0000 mgr.y (mgr.24437) 244 : cluster [DBG] pgmap v292: 264 pgs: 40 unknown, 1 clean+premerge+peered, 196 active+clean, 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:38.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:37.587784+0000 mon.a (mon.0) 1541 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-23T00:32:38.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:37.998188+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? 172.21.15.87:0/4189795763' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi087-82019-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:38.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:37.998311+0000 mon.a (mon.0) 1543 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-23T00:32:38.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:37.998379+0000 mon.a (mon.0) 1544 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-02-23T00:32:38.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:38.005347+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:38.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:38.009000+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:38.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:38.009907+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:38.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:38.011780+0000 mon.a (mon.0) 1546 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:37.437118+0000 mon.c (mon.2) 237 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:37.439133+0000 osd.5 (osd.5) 95 : cluster [DBG] 3.c scrub starts 2024-02-23T00:32:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:37.440610+0000 osd.5 (osd.5) 96 : cluster [DBG] 3.c scrub ok 2024-02-23T00:32:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:37.442027+0000 osd.6 (osd.6) 81 : cluster [DBG] 3.a scrub starts 2024-02-23T00:32:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:37.444965+0000 osd.6 (osd.6) 82 : cluster [DBG] 3.a scrub ok 2024-02-23T00:32:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:37.586751+0000 mgr.y (mgr.24437) 244 : cluster [DBG] pgmap v292: 264 pgs: 40 unknown, 1 clean+premerge+peered, 196 active+clean, 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:37.587784+0000 mon.a (mon.0) 1541 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-23T00:32:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:37.998188+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? 172.21.15.87:0/4189795763' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi087-82019-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:37.998311+0000 mon.a (mon.0) 1543 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-23T00:32:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:37.998379+0000 mon.a (mon.0) 1544 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-02-23T00:32:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:38.005347+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:38.009000+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:38.009907+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:38.011780+0000 mon.a (mon.0) 1546 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:38.747 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:38.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:38.851 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:38.884 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:38.905 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:38.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:39.009 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:39.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.008+0000 7f07344bf700 1 -- 172.21.15.87:0/3733143197 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c069c40 msgr2=0x7f072c10d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.008+0000 7f07344bf700 1 --2- 172.21.15.87:0/3733143197 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c069c40 0x7f072c10d070 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 rx=0x7f0724009c30 tx=0x7f0724004c50).stop 2024-02-23T00:32:39.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.010+0000 7f07344bf700 1 -- 172.21.15.87:0/3733143197 shutdown_connections 2024-02-23T00:32:39.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.010+0000 7f07344bf700 1 --2- 172.21.15.87:0/3733143197 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c069c40 0x7f072c10d070 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.010+0000 7f07344bf700 1 --2- 172.21.15.87:0/3733143197 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f072c1001a0 0x7f072c100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:32:39.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.010+0000 7f07344bf700 1 --2- 172.21.15.87:0/3733143197 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f072c0692d0 0x7f072c069750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.010+0000 7f07344bf700 1 -- 172.21.15.87:0/3733143197 >> 172.21.15.87:0/3733143197 conn(0x7f072c063db0 msgr2=0x7f072c067560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:39.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.015+0000 7f07344bf700 1 -- 172.21.15.87:0/3733143197 shutdown_connections 2024-02-23T00:32:39.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.019+0000 7f07344bf700 1 -- 172.21.15.87:0/3733143197 wait complete. 2024-02-23T00:32:39.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.019+0000 7f07344bf700 1 librados: starting msgr at 2024-02-23T00:32:39.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.019+0000 7f07344bf700 1 librados: starting objecter 2024-02-23T00:32:39.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.019+0000 7f07344bf700 1 Processor -- start 2024-02-23T00:32:39.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.019+0000 7f07344bf700 1 -- start start 2024-02-23T00:32:39.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.019+0000 7f07344bf700 1 librados: setting wanted keys 2024-02-23T00:32:39.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.019+0000 7f07344bf700 1 librados: calling monclient init 2024-02-23T00:32:39.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.025+0000 7f07344bf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c0692d0 0x7f072c118a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.025+0000 7f07344bf700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f072c069c40 0x7f072c118f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.025+0000 7f07344bf700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f072c1001a0 0x7f072c112c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.025+0000 7f07344bf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c1130f0 con 0x7f072c0692d0 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.025+0000 7f07344bf700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c113260 con 0x7f072c1001a0 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.025+0000 7f07344bf700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c113420 con 0x7f072c069c40 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.025+0000 7f073225b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c0692d0 0x7f072c118a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.025+0000 7f073225b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c0692d0 0x7f072c118a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33348/0 (socket says 172.21.15.87:33348) 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.025+0000 7f073225b700 1 -- 172.21.15.87:0/3416446542 learned_addr learned my addr 172.21.15.87:0/3416446542 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.026+0000 7f0732a5c700 1 --2- 172.21.15.87:0/3416446542 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f072c1001a0 0x7f072c112c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.026+0000 7f073225b700 1 -- 172.21.15.87:0/3416446542 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f072c069c40 msgr2=0x7f072c118f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.026+0000 7f073225b700 1 --2- 172.21.15.87:0/3416446542 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f072c069c40 0x7f072c118f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:32:39.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.026+0000 7f073225b700 1 -- 172.21.15.87:0/3416446542 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f072c1001a0 msgr2=0x7f072c112c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.026+0000 7f073225b700 1 --2- 172.21.15.87:0/3416446542 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f072c1001a0 0x7f072c112c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.027+0000 7f073225b700 1 -- 172.21.15.87:0/3416446542 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07240097c0 con 0x7f072c0692d0 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.027+0000 7f073225b700 1 --2- 172.21.15.87:0/3416446542 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c0692d0 0x7f072c118a60 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7f072800b530 tx=0x7f07280090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.027+0000 7f07237fe700 10 client.?.objecter ms_handle_connect 0x7f072c0692d0 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.027+0000 7f07237fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.027+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0728002320 con 0x7f072c0692d0 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.027+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0728002950 con 0x7f072c0692d0 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.027+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07280129e0 con 0x7f072c0692d0 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.028+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c113700 con 0x7f072c0692d0 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.028+0000 7f07344bf700 10 client.26628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.028+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c11f890 con 0x7f072c0692d0 2024-02-23T00:32:39.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.028+0000 7f07344bf700 1 librados: init done 2024-02-23T00:32:39.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.035+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f072c060280 con 0x7f072c0692d0 2024-02-23T00:32:39.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.035+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0728002480 con 0x7f072c0692d0 2024-02-23T00:32:39.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.035+0000 7f07237fe700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0718062d20 0x7f07180651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.035+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(208..208 src has 38..208) v4 ==== 8323+0+0 (secure 0 0 0) 0x7f072807d110 con 0x7f072c0692d0 2024-02-23T00:32:39.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.035+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 osd_map(208..208 src has 38..208) v4 2024-02-23T00:32:39.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.035+0000 7f07237fe700 3 client.26628.objecter handle_osd_map got epochs [208,208] > 0 2024-02-23T00:32:39.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.035+0000 7f07237fe700 3 client.26628.objecter handle_osd_map decoding full epoch 208 2024-02-23T00:32:39.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.035+0000 7f07237fe700 20 client.26628.objecter dump_active .. 0 homeless 2024-02-23T00:32:39.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.036+0000 7f0731a5a700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0718062d20 0x7f07180651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.038+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v208) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0728084070 con 0x7f072c0692d0 2024-02-23T00:32:39.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.038+0000 7f0731a5a700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0718062d20 0x7f07180651e0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 rx=0x7f072c1143a0 tx=0x7f071c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.038+0000 7f07237fe700 10 client.26628.objecter ms_handle_connect 0x7f0718062d20 2024-02-23T00:32:39.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 10 client.26628.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:39.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 1 2024-02-23T00:32:39.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6801/2153191863 conn(0x7f072c1079b0 0x7f072c107d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c05ee80 osd=0 3 2024-02-23T00:32:39.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c05ee80 osd=0 3 2024-02-23T00:32:39.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c05ee80 osd=0 3 2024-02-23T00:32:39.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 1 2024-02-23T00:32:39.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c05ee80 osd=0 3 2024-02-23T00:32:39.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 0 1 2024-02-23T00:32:39.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c05ee80 osd=0 4 2024-02-23T00:32:39.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 10 client.26628.objecter _send_command 1 2024-02-23T00:32:39.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.039+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f072c111c20 con 0x7f072c1079b0 2024-02-23T00:32:39.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.040+0000 7f0732a5c700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6801/2153191863 conn(0x7f072c1079b0 0x7f072c107d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.041+0000 7f07237fe700 10 client.26628.objecter ms_handle_connect 0x7f072c1079b0 2024-02-23T00:32:39.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.041+0000 7f0732a5c700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6801/2153191863 conn(0x7f072c1079b0 0x7f072c107d90 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.042+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f072c111c20 con 0x7f072c1079b0 2024-02-23T00:32:39.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.042+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:32:39.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.042+0000 7f07237fe700 10 client.26628.objecter _finish_command 1 = system:0 2024-02-23T00:32:39.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.042+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c05ee80 osd=0 4 2024-02-23T00:32:39.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.042+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 0 1 2024-02-23T00:32:39.060 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 10 client.26628.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 2 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c05ee80 osd=0 3 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c05ee80 osd=0 3 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c05ee80 osd=0 3 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 2 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c05ee80 osd=0 3 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 0 2 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c05ee80 osd=0 4 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 10 client.26628.objecter _send_command 2 2024-02-23T00:32:39.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.059+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f072c0273c0 con 0x7f072c1079b0 2024-02-23T00:32:39.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.060+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f072c0273c0 con 0x7f072c1079b0 2024-02-23T00:32:39.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:39.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07237fe700 10 client.26628.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c05ee80 osd=0 4 2024-02-23T00:32:39.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 0 2 2024-02-23T00:32:39.063 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 10 client.26628.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 3 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6805/685354171 conn(0x7f072c0598f0 0x7f072c059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c028710 osd=1 3 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c028710 osd=1 3 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c028710 osd=1 3 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 3 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c028710 osd=1 3 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 1 3 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c028710 osd=1 4 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 10 client.26628.objecter _send_command 3 2024-02-23T00:32:39.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f072c126200 con 0x7f072c0598f0 2024-02-23T00:32:39.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f073225b700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6805/685354171 conn(0x7f072c0598f0 0x7f072c059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f07237fe700 10 client.26628.objecter ms_handle_connect 0x7f072c0598f0 2024-02-23T00:32:39.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.061+0000 7f073225b700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6805/685354171 conn(0x7f072c0598f0 0x7f072c059cd0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.062+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f072c126200 con 0x7f072c0598f0 2024-02-23T00:32:39.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.062+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:32:39.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.062+0000 7f07237fe700 10 client.26628.objecter _finish_command 3 = system:0 2024-02-23T00:32:39.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.062+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c028710 osd=1 4 2024-02-23T00:32:39.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.062+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 1 3 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 10 client.26628.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 4 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c028710 osd=1 3 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c028710 osd=1 3 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c028710 osd=1 3 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 4 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c028710 osd=1 3 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 1 4 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c028710 osd=1 4 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 10 client.26628.objecter _send_command 4 2024-02-23T00:32:39.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.078+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f072c059360 con 0x7f072c0598f0 2024-02-23T00:32:39.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.079+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f072c059360 con 0x7f072c0598f0 2024-02-23T00:32:39.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.079+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:39.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.079+0000 7f07237fe700 10 client.26628.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.079+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c028710 osd=1 4 2024-02-23T00:32:39.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.079+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 1 4 2024-02-23T00:32:39.082 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:32:39.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 10 client.26628.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 5 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6809/3605394228 conn(0x7f072c028270 0x7f072c124c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c059770 osd=2 3 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c059770 osd=2 3 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c059770 osd=2 3 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 5 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c059770 osd=2 3 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 2 5 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c059770 osd=2 4 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 10 client.26628.objecter _send_command 5 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f072c125110 con 0x7f072c028270 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f0731a5a700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6809/3605394228 conn(0x7f072c028270 0x7f072c124c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f0731a5a700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6809/3605394228 conn(0x7f072c028270 0x7f072c124c20 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.080+0000 7f07237fe700 10 client.26628.objecter ms_handle_connect 0x7f072c028270 2024-02-23T00:32:39.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.081+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f072c125110 con 0x7f072c028270 2024-02-23T00:32:39.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.081+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:32:39.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.081+0000 7f07237fe700 10 client.26628.objecter _finish_command 5 = system:0 2024-02-23T00:32:39.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.081+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c059770 osd=2 4 2024-02-23T00:32:39.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.081+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 2 5 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 10 client.26628.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 6 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c059770 osd=2 3 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c059770 osd=2 3 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c059770 osd=2 3 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 6 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c059770 osd=2 3 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 2 6 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c059770 osd=2 4 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 10 client.26628.objecter _send_command 6 2024-02-23T00:32:39.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.096+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f072c0277d0 con 0x7f072c028270 2024-02-23T00:32:39.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.097+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f072c0277d0 con 0x7f072c028270 2024-02-23T00:32:39.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.097+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:39.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.097+0000 7f07237fe700 10 client.26628.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.097+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c059770 osd=2 4 2024-02-23T00:32:39.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.097+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 2 6 2024-02-23T00:32:39.100 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:32:39.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.097+0000 7f07344bf700 10 client.26628.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.097+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 7 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6813/128059790 conn(0x7f072c128380 0x7f072c12a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c027be0 osd=3 3 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c027be0 osd=3 3 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c027be0 osd=3 3 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 7 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c027be0 osd=3 3 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 3 7 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c027be0 osd=3 4 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 10 client.26628.objecter _send_command 7 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f072c126090 con 0x7f072c128380 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f0732a5c700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6813/128059790 conn(0x7f072c128380 0x7f072c12a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f07237fe700 10 client.26628.objecter ms_handle_connect 0x7f072c128380 2024-02-23T00:32:39.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.098+0000 7f0732a5c700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6813/128059790 conn(0x7f072c128380 0x7f072c12a770 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.099+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f072c126090 con 0x7f072c128380 2024-02-23T00:32:39.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.099+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:32:39.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.099+0000 7f07237fe700 10 client.26628.objecter _finish_command 7 = system:0 2024-02-23T00:32:39.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.099+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c027be0 osd=3 4 2024-02-23T00:32:39.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.099+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 3 7 2024-02-23T00:32:39.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 10 client.26628.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:39.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 8 2024-02-23T00:32:39.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c027be0 osd=3 3 2024-02-23T00:32:39.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c027be0 osd=3 3 2024-02-23T00:32:39.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c027be0 osd=3 3 2024-02-23T00:32:39.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 8 2024-02-23T00:32:39.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c027be0 osd=3 3 2024-02-23T00:32:39.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 3 8 2024-02-23T00:32:39.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c027be0 osd=3 4 2024-02-23T00:32:39.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 10 client.26628.objecter _send_command 8 2024-02-23T00:32:39.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.114+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f072c125570 con 0x7f072c128380 2024-02-23T00:32:39.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f072c125570 con 0x7f072c128380 2024-02-23T00:32:39.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:39.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07237fe700 10 client.26628.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c027be0 osd=3 4 2024-02-23T00:32:39.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 3 8 2024-02-23T00:32:39.118 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 10 client.26628.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 9 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6800/4175844757 conn(0x7f072c124780 0x7f072c12bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c125710 osd=4 3 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125710 osd=4 3 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c125710 osd=4 3 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 9 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c125710 osd=4 3 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 4 9 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125710 osd=4 4 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 10 client.26628.objecter _send_command 9 2024-02-23T00:32:39.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f072c125880 con 0x7f072c124780 2024-02-23T00:32:39.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.116+0000 7f073225b700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6800/4175844757 conn(0x7f072c124780 0x7f072c12bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.117+0000 7f073225b700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6800/4175844757 conn(0x7f072c124780 0x7f072c12bde0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.117+0000 7f07237fe700 10 client.26628.objecter ms_handle_connect 0x7f072c124780 2024-02-23T00:32:39.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.118+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f072c125880 con 0x7f072c124780 2024-02-23T00:32:39.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.118+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:32:39.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.118+0000 7f07237fe700 10 client.26628.objecter _finish_command 9 = system:0 2024-02-23T00:32:39.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.118+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c125710 osd=4 4 2024-02-23T00:32:39.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.118+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 4 9 2024-02-23T00:32:39.132 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:39.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:39.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 10 client.26628.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:39.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 10 2024-02-23T00:32:39.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c125710 osd=4 3 2024-02-23T00:32:39.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125710 osd=4 3 2024-02-23T00:32:39.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c125710 osd=4 3 2024-02-23T00:32:39.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 10 2024-02-23T00:32:39.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c125710 osd=4 3 2024-02-23T00:32:39.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 4 10 2024-02-23T00:32:39.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125710 osd=4 4 2024-02-23T00:32:39.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 10 client.26628.objecter _send_command 10 2024-02-23T00:32:39.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.141+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f072c125880 con 0x7f072c124780 2024-02-23T00:32:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f072c125880 con 0x7f072c124780 2024-02-23T00:32:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07237fe700 10 client.26628.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c125710 osd=4 4 2024-02-23T00:32:39.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 4 10 2024-02-23T00:32:39.146 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:32:39.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 10 client.26628.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 11 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6804/2550013795 conn(0x7f072c12c820 0x7f072c12cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c125880 osd=5 3 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125880 osd=5 3 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c125880 osd=5 3 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 11 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c125880 osd=5 3 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 5 11 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125880 osd=5 4 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 10 client.26628.objecter _send_command 11 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.143+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f072c133bf0 con 0x7f072c12c820 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.144+0000 7f0731a5a700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6804/2550013795 conn(0x7f072c12c820 0x7f072c12cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.144+0000 7f07237fe700 10 client.26628.objecter ms_handle_connect 0x7f072c12c820 2024-02-23T00:32:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.144+0000 7f0731a5a700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6804/2550013795 conn(0x7f072c12c820 0x7f072c12cc40 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.145+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f072c133bf0 con 0x7f072c12c820 2024-02-23T00:32:39.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.145+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:32:39.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.145+0000 7f07237fe700 10 client.26628.objecter _finish_command 11 = system:0 2024-02-23T00:32:39.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.145+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c125880 osd=5 4 2024-02-23T00:32:39.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.145+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 5 11 2024-02-23T00:32:39.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 10 client.26628.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:39.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 12 2024-02-23T00:32:39.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c125880 osd=5 3 2024-02-23T00:32:39.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125880 osd=5 3 2024-02-23T00:32:39.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c125880 osd=5 3 2024-02-23T00:32:39.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 12 2024-02-23T00:32:39.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c125880 osd=5 3 2024-02-23T00:32:39.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 5 12 2024-02-23T00:32:39.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125880 osd=5 4 2024-02-23T00:32:39.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 10 client.26628.objecter _send_command 12 2024-02-23T00:32:39.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.159+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f072c131490 con 0x7f072c12c820 2024-02-23T00:32:39.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f072c131490 con 0x7f072c12c820 2024-02-23T00:32:39.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:39.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07237fe700 10 client.26628.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c125880 osd=5 4 2024-02-23T00:32:39.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 5 12 2024-02-23T00:32:39.164 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:32:39.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 10 client.26628.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 13 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6808/2509408498 conn(0x7f072c131b40 0x7f072c133f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c131490 osd=6 3 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c131490 osd=6 3 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c131490 osd=6 3 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 13 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c131490 osd=6 3 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 6 13 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c131490 osd=6 4 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 10 client.26628.objecter _send_command 13 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f072c136cf0 con 0x7f072c131b40 2024-02-23T00:32:39.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.161+0000 7f0732a5c700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6808/2509408498 conn(0x7f072c131b40 0x7f072c133f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.162+0000 7f07237fe700 10 client.26628.objecter ms_handle_connect 0x7f072c131b40 2024-02-23T00:32:39.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.162+0000 7f0732a5c700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6808/2509408498 conn(0x7f072c131b40 0x7f072c133f50 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.162+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f072c136cf0 con 0x7f072c131b40 2024-02-23T00:32:39.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.162+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:32:39.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.162+0000 7f07237fe700 10 client.26628.objecter _finish_command 13 = system:0 2024-02-23T00:32:39.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.162+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c131490 osd=6 4 2024-02-23T00:32:39.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.162+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 6 13 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 10 client.26628.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 14 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c131490 osd=6 3 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c131490 osd=6 3 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c131490 osd=6 3 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 14 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c131490 osd=6 3 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 6 14 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c131490 osd=6 4 2024-02-23T00:32:39.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 10 client.26628.objecter _send_command 14 2024-02-23T00:32:39.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.185+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f072c136cf0 con 0x7f072c131b40 2024-02-23T00:32:39.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f072c136cf0 con 0x7f072c131b40 2024-02-23T00:32:39.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:39.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07237fe700 10 client.26628.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c131490 osd=6 4 2024-02-23T00:32:39.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 6 14 2024-02-23T00:32:39.190 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:32:39.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 10 client.26628.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:39.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 15 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6812/373787000 conn(0x7f072c136020 0x7f072c1364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c136cf0 osd=7 3 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c136cf0 osd=7 3 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c136cf0 osd=7 3 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 15 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c136cf0 osd=7 3 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 7 15 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c136cf0 osd=7 4 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 10 client.26628.objecter _send_command 15 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.187+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f072c136a80 con 0x7f072c136020 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.188+0000 7f073225b700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6812/373787000 conn(0x7f072c136020 0x7f072c1364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.188+0000 7f07237fe700 10 client.26628.objecter ms_handle_connect 0x7f072c136020 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.188+0000 7f073225b700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6812/373787000 conn(0x7f072c136020 0x7f072c1364a0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:39.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.188+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f072c136a80 con 0x7f072c136020 2024-02-23T00:32:39.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.188+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:32:39.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.188+0000 7f07237fe700 10 client.26628.objecter _finish_command 15 = system:0 2024-02-23T00:32:39.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.189+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c136cf0 osd=7 4 2024-02-23T00:32:39.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.189+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 7 15 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 10 client.26628.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign -1 16 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c136cf0 osd=7 3 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c136cf0 osd=7 3 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 20 client.26628.objecter _get_session s=0x7f072c136cf0 osd=7 3 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 15 client.26628.objecter _session_command_op_remove -1 16 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 20 client.26628.objecter get_session s=0x7f072c136cf0 osd=7 3 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 15 client.26628.objecter _session_command_op_assign 7 16 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c136cf0 osd=7 4 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 10 client.26628.objecter _send_command 16 2024-02-23T00:32:39.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.203+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f072c136a80 con 0x7f072c136020 2024-02-23T00:32:39.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.204+0000 7f07237fe700 1 -- 172.21.15.87:0/3416446542 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f072c136a80 con 0x7f072c136020 2024-02-23T00:32:39.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.204+0000 7f07237fe700 10 client.26628.objecter ms_dispatch 0x7f072c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:32:39.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.204+0000 7f07237fe700 10 client.26628.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.204+0000 7f07237fe700 20 client.26628.objecter put_session s=0x7f072c136cf0 osd=7 4 2024-02-23T00:32:39.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.204+0000 7f07237fe700 15 client.26628.objecter _session_command_op_remove 7 16 2024-02-23T00:32:39.207 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:32:39.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 librados: watch_flush enter 2024-02-23T00:32:39.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 librados: watch_flush exit 2024-02-23T00:32:39.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 client.26628.objecter close_session for osd.0 2024-02-23T00:32:39.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6801/2153191863 conn(0x7f072c1079b0 msgr2=0x7f072c107d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6801/2153191863 conn(0x7f072c1079b0 0x7f072c107d90 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c05ee80 osd=0 1 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 client.26628.objecter close_session for osd.1 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6805/685354171 conn(0x7f072c0598f0 msgr2=0x7f072c059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6805/685354171 conn(0x7f072c0598f0 0x7f072c059cd0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c028710 osd=1 1 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 client.26628.objecter close_session for osd.2 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6809/3605394228 conn(0x7f072c028270 msgr2=0x7f072c124c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6809/3605394228 conn(0x7f072c028270 0x7f072c124c20 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c059770 osd=2 1 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 client.26628.objecter close_session for osd.3 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6813/128059790 conn(0x7f072c128380 msgr2=0x7f072c12a770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6813/128059790 conn(0x7f072c128380 0x7f072c12a770 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f0732a5c700 1 -- 172.21.15.87:0/3416446542 reap_dead start 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c027be0 osd=3 1 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 client.26628.objecter close_session for osd.4 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6800/4175844757 conn(0x7f072c124780 msgr2=0x7f072c12bde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6800/4175844757 conn(0x7f072c124780 0x7f072c12bde0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125710 osd=4 1 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 client.26628.objecter close_session for osd.5 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6804/2550013795 conn(0x7f072c12c820 msgr2=0x7f072c12cc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6804/2550013795 conn(0x7f072c12c820 0x7f072c12cc40 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c125880 osd=5 1 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 client.26628.objecter close_session for osd.6 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6808/2509408498 conn(0x7f072c131b40 msgr2=0x7f072c133f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6808/2509408498 conn(0x7f072c131b40 0x7f072c133f50 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c131490 osd=6 1 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 client.26628.objecter close_session for osd.7 2024-02-23T00:32:39.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6812/373787000 conn(0x7f072c136020 msgr2=0x7f072c1364a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.171:6812/373787000 conn(0x7f072c136020 0x7f072c1364a0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 20 client.26628.objecter put_session s=0x7f072c136cf0 osd=7 1 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 20 client.26628.objecter shutdown clearing up homeless session... 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 10 client.26628.objecter successfully canceled tick 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f0732a5c700 1 -- 172.21.15.87:0/3416446542 reap_dead start 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0718062d20 msgr2=0x7f07180651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0718062d20 0x7f07180651e0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 rx=0x7f072c1143a0 tx=0x7f071c00b040).stop 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c0692d0 msgr2=0x7f072c118a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c0692d0 0x7f072c118a60 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 rx=0x7f072800b530 tx=0x7f07280090f0).stop 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 shutdown_connections 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f072c0692d0 0x7f072c118a60 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 --2- 172.21.15.87:0/3416446542 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0718062d20 0x7f07180651e0 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 >> 172.21.15.87:0/3416446542 conn(0x7f072c063db0 msgr2=0x7f072c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:39.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 shutdown_connections 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 -- 172.21.15.87:0/3416446542 wait complete. 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:39.205+0000 7f07344bf700 1 librados: shutdown 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.212 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:32:39.237 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:39.262 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:37.929094+0000 osd.3 (osd.3) 107 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:37.932051+0000 osd.3 (osd.3) 108 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:38.325171+0000 osd.1 (osd.1) 101 : cluster [DBG] 2.d scrub starts 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:38.328233+0000 osd.1 (osd.1) 102 : cluster [DBG] 2.d scrub ok 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:38.438206+0000 mon.c (mon.2) 238 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:38.680782+0000 osd.7 (osd.7) 105 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:38.683681+0000 osd.7 (osd.7) 106 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:38.808817+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:38.809505+0000 mon.a (mon.0) 1548 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:38.815914+0000 mon.a (mon.0) 1549 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:39.002316+0000 mon.a (mon.0) 1550 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:39.002409+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]': finished 2024-02-23T00:32:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:39.002491+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-02-23T00:32:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:39.011336+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:39.017005+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:39.023225+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:39.027444+0000 mon.a (mon.0) 1554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:39.037938+0000 mon.a (mon.0) 1555 : audit [DBG] from='client.? 172.21.15.87:0/3416446542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:37.929094+0000 osd.3 (osd.3) 107 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:32:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:37.932051+0000 osd.3 (osd.3) 108 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:32:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:38.325171+0000 osd.1 (osd.1) 101 : cluster [DBG] 2.d scrub starts 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:38.328233+0000 osd.1 (osd.1) 102 : cluster [DBG] 2.d scrub ok 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:38.438206+0000 mon.c (mon.2) 238 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:38.680782+0000 osd.7 (osd.7) 105 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:38.683681+0000 osd.7 (osd.7) 106 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:38.808817+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:38.809505+0000 mon.a (mon.0) 1548 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:38.815914+0000 mon.a (mon.0) 1549 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:39.002316+0000 mon.a (mon.0) 1550 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:39.002409+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]': finished 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:39.002491+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:39.011336+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:39.017005+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:39.023225+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:39.027444+0000 mon.a (mon.0) 1554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:39.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:39.037938+0000 mon.a (mon.0) 1555 : audit [DBG] from='client.? 172.21.15.87:0/3416446542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:39.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:37.929094+0000 osd.3 (osd.3) 107 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:32:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:37.932051+0000 osd.3 (osd.3) 108 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:32:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:38.325171+0000 osd.1 (osd.1) 101 : cluster [DBG] 2.d scrub starts 2024-02-23T00:32:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:38.328233+0000 osd.1 (osd.1) 102 : cluster [DBG] 2.d scrub ok 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:38.438206+0000 mon.c (mon.2) 238 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:38.680782+0000 osd.7 (osd.7) 105 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:38.683681+0000 osd.7 (osd.7) 106 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:38.808817+0000 mon.a (mon.0) 1547 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:38.809505+0000 mon.a (mon.0) 1548 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:38.815914+0000 mon.a (mon.0) 1549 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:39.002316+0000 mon.a (mon.0) 1550 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:39.002409+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi087-82011-30"}]': finished 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:39.002491+0000 mon.a (mon.0) 1552 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:39.011336+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.87:0/1791382966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:39.017005+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]: dispatch 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:39.023225+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:39.027444+0000 mon.a (mon.0) 1554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:39.037938+0000 mon.a (mon.0) 1555 : audit [DBG] from='client.? 172.21.15.87:0/3416446542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:39.716 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:39.744 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:39.892 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:39.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:39.995 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:40.033 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:40.054 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:40.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:40.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:40.158 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:40.188 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:40.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:39.439149+0000 mon.c (mon.2) 239 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:39.587191+0000 mgr.y (mgr.24437) 245 : cluster [DBG] pgmap v295: 256 pgs: 32 unknown, 1 clean+premerge+peered, 196 active+clean, 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.004991+0000 mon.a (mon.0) 1556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]': finished 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.005111+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:40.005253+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.006707+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.008201+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.87:0/3734064806' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi087-82019-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.013157+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi087-82019-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.013421+0000 mon.a (mon.0) 1560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.031923+0000 mon.c (mon.2) 241 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.472 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.032189+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.473 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.032791+0000 mon.c (mon.2) 242 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.473 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.033007+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.473 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.033388+0000 mon.c (mon.2) 243 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi087-82011-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:40.473 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.033610+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi087-82011-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:40.473 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:39.439149+0000 mon.c (mon.2) 239 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:40.473 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:39.587191+0000 mgr.y (mgr.24437) 245 : cluster [DBG] pgmap v295: 256 pgs: 32 unknown, 1 clean+premerge+peered, 196 active+clean, 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:40.473 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.004991+0000 mon.a (mon.0) 1556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]': finished 2024-02-23T00:32:40.473 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.005111+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:40.473 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:40.005253+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.006707+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.008201+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.87:0/3734064806' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi087-82019-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.013157+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi087-82019-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.013421+0000 mon.a (mon.0) 1560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.031923+0000 mon.c (mon.2) 241 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.032189+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.032791+0000 mon.c (mon.2) 242 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.033007+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.033388+0000 mon.c (mon.2) 243 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi087-82011-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:40.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.033610+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi087-82011-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:39.439149+0000 mon.c (mon.2) 239 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:39.587191+0000 mgr.y (mgr.24437) 245 : cluster [DBG] pgmap v295: 256 pgs: 32 unknown, 1 clean+premerge+peered, 196 active+clean, 17 active+clean+snaptrim_wait, 10 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.004991+0000 mon.a (mon.0) 1556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi087-82011-30"}]': finished 2024-02-23T00:32:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.005111+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:40.005253+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.006707+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.008201+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.87:0/3734064806' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi087-82019-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.013157+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi087-82019-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.013421+0000 mon.a (mon.0) 1560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.031923+0000 mon.c (mon.2) 241 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.032189+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.032791+0000 mon.c (mon.2) 242 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.033007+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.033388+0000 mon.c (mon.2) 243 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi087-82011-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.033610+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi087-82011-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:41.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:32:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:32:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:39.641545+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:32:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:39.644475+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:40.323973+0000 osd.1 (osd.1) 103 : cluster [DBG] 4.f scrub starts 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:40.327513+0000 osd.1 (osd.1) 104 : cluster [DBG] 4.f scrub ok 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:40.342986+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:40.343858+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.440025+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:40.724292+0000 mgr.y (mgr.24437) 246 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:41.008756+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi087-82019-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:41.008848+0000 mon.a (mon.0) 1565 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-19"}]': finished 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:41.009036+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi087-82011-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.009115+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-02-23T00:32:41.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:41.013724+0000 mon.c (mon.2) 245 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi087-82011-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:41.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:41.014563+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-19", "mode": "writeback"}]: dispatch 2024-02-23T00:32:41.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:41.015132+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi087-82011-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:41.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:41.017836+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-19", "mode": "writeback"}]: dispatch 2024-02-23T00:32:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:39.641545+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:32:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:39.644475+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:32:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:40.323973+0000 osd.1 (osd.1) 103 : cluster [DBG] 4.f scrub starts 2024-02-23T00:32:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:40.327513+0000 osd.1 (osd.1) 104 : cluster [DBG] 4.f scrub ok 2024-02-23T00:32:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:40.342986+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:32:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:40.343858+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:32:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.440025+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:40.724292+0000 mgr.y (mgr.24437) 246 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:41.008756+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi087-82019-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:41.008848+0000 mon.a (mon.0) 1565 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-19"}]': finished 2024-02-23T00:32:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:41.009036+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi087-82011-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.009115+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-02-23T00:32:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:41.013724+0000 mon.c (mon.2) 245 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi087-82011-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:41.014563+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-19", "mode": "writeback"}]: dispatch 2024-02-23T00:32:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:41.015132+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi087-82011-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:41.017836+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-19", "mode": "writeback"}]: dispatch 2024-02-23T00:32:41.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:39.641545+0000 osd.0 (osd.0) 119 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:39.644475+0000 osd.0 (osd.0) 120 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:40.323973+0000 osd.1 (osd.1) 103 : cluster [DBG] 4.f scrub starts 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:40.327513+0000 osd.1 (osd.1) 104 : cluster [DBG] 4.f scrub ok 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:40.342986+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:40.343858+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.440025+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:40.724292+0000 mgr.y (mgr.24437) 246 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:41.008756+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi087-82019-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:41.008848+0000 mon.a (mon.0) 1565 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-19"}]': finished 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:41.009036+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi087-82011-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.009115+0000 mon.a (mon.0) 1567 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:41.013724+0000 mon.c (mon.2) 245 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi087-82011-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:41.014563+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-19", "mode": "writeback"}]: dispatch 2024-02-23T00:32:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:41.015132+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi087-82011-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:41.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:41.017836+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-19", "mode": "writeback"}]: dispatch 2024-02-23T00:32:42.499 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:42] "GET /metrics HTTP/1.1" 200 187646 "" "Prometheus/2.33.4" 2024-02-23T00:32:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:40.854179+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.a scrub starts 2024-02-23T00:32:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:40.855189+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2024-02-23T00:32:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.359513+0000 osd.1 (osd.1) 105 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:32:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.360639+0000 osd.1 (osd.1) 106 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:32:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:41.440933+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:41.588577+0000 mon.a (mon.0) 1570 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-23T00:32:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.921905+0000 osd.4 (osd.4) 97 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:32:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.922844+0000 osd.4 (osd.4) 98 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.993852+0000 mon.a (mon.0) 1571 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:42.237802+0000 mon.a (mon.0) 1572 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-19", "mode": "writeback"}]': finished 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:42.237900+0000 mon.a (mon.0) 1573 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:42.237949+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:40.854179+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.a scrub starts 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:40.855189+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.359513+0000 osd.1 (osd.1) 105 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.360639+0000 osd.1 (osd.1) 106 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:41.440933+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:41.588577+0000 mon.a (mon.0) 1570 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.921905+0000 osd.4 (osd.4) 97 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:32:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.922844+0000 osd.4 (osd.4) 98 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:32:42.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.993852+0000 mon.a (mon.0) 1571 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:42.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:42.237802+0000 mon.a (mon.0) 1572 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-19", "mode": "writeback"}]': finished 2024-02-23T00:32:42.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:42.237900+0000 mon.a (mon.0) 1573 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-23T00:32:42.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:42.237949+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-02-23T00:32:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:40.854179+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.a scrub starts 2024-02-23T00:32:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:40.855189+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2024-02-23T00:32:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.359513+0000 osd.1 (osd.1) 105 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:32:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.360639+0000 osd.1 (osd.1) 106 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:32:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:41.440933+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:41.588577+0000 mon.a (mon.0) 1570 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-23T00:32:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.921905+0000 osd.4 (osd.4) 97 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:32:42.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.922844+0000 osd.4 (osd.4) 98 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:32:42.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.993852+0000 mon.a (mon.0) 1571 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:42.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:42.237802+0000 mon.a (mon.0) 1572 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-19", "mode": "writeback"}]': finished 2024-02-23T00:32:42.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:42.237900+0000 mon.a (mon.0) 1573 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-23T00:32:42.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:42.237949+0000 mon.a (mon.0) 1574 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-02-23T00:32:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.587601+0000 mgr.y (mgr.24437) 247 : cluster [DBG] pgmap v298: 287 pgs: 1 peering, 32 creating+peering, 230 active+clean, 16 active+clean+snaptrim_wait, 8 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.672316+0000 osd.0 (osd.0) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.672555+0000 osd.0 (osd.0) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.860459+0000 osd.2 (osd.2) 91 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:32:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:41.861868+0000 osd.2 (osd.2) 92 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:32:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:42.184798+0000 osd.4 (osd.4) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:42.185293+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:42.261611+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:42.407218+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:42.410109+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:42.441703+0000 mon.c (mon.2) 247 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:42.494549+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:42.690517+0000 osd.7 (osd.7) 107 : cluster [DBG] 19.2 scrub starts 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:42.691626+0000 osd.7 (osd.7) 108 : cluster [DBG] 19.2 scrub ok 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:42.814869+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:42.815718+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:43.002923+0000 mon.a (mon.0) 1576 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi087-82011-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi087-82011-31"}]': finished 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:43.003109+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.003182+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-02-23T00:32:43.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:43.008278+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:43.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:43.009092+0000 mon.a (mon.0) 1579 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:43.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:43.015142+0000 mon.c (mon.2) 248 : audit [INF] from='client.? 172.21.15.87:0/450042771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi087-82019-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:43.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:43.016069+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi087-82019-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:43.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:43.442594+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:43.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.587601+0000 mgr.y (mgr.24437) 247 : cluster [DBG] pgmap v298: 287 pgs: 1 peering, 32 creating+peering, 230 active+clean, 16 active+clean+snaptrim_wait, 8 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:43.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.672316+0000 osd.0 (osd.0) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:43.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.672555+0000 osd.0 (osd.0) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:43.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.860459+0000 osd.2 (osd.2) 91 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:32:43.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:41.861868+0000 osd.2 (osd.2) 92 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:42.184798+0000 osd.4 (osd.4) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:42.185293+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:42.261611+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:42.407218+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:42.410109+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:42.441703+0000 mon.c (mon.2) 247 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:42.494549+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:42.690517+0000 osd.7 (osd.7) 107 : cluster [DBG] 19.2 scrub starts 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:42.691626+0000 osd.7 (osd.7) 108 : cluster [DBG] 19.2 scrub ok 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:42.814869+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:42.815718+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:32:43.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:43.002923+0000 mon.a (mon.0) 1576 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi087-82011-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi087-82011-31"}]': finished 2024-02-23T00:32:43.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:43.003109+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:43.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.003182+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-02-23T00:32:43.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:43.008278+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:43.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:43.009092+0000 mon.a (mon.0) 1579 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:43.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:43.015142+0000 mon.c (mon.2) 248 : audit [INF] from='client.? 172.21.15.87:0/450042771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi087-82019-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:43.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:43.016069+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi087-82019-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:43.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:43.442594+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:43.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:32:43.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.587601+0000 mgr.y (mgr.24437) 247 : cluster [DBG] pgmap v298: 287 pgs: 1 peering, 32 creating+peering, 230 active+clean, 16 active+clean+snaptrim_wait, 8 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:43.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.672316+0000 osd.0 (osd.0) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.672555+0000 osd.0 (osd.0) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.860459+0000 osd.2 (osd.2) 91 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:41.861868+0000 osd.2 (osd.2) 92 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:42.184798+0000 osd.4 (osd.4) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:42.185293+0000 osd.4 (osd.4) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:42.261611+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:42.407218+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:42.410109+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:42.441703+0000 mon.c (mon.2) 247 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:42.494549+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:42.690517+0000 osd.7 (osd.7) 107 : cluster [DBG] 19.2 scrub starts 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:42.691626+0000 osd.7 (osd.7) 108 : cluster [DBG] 19.2 scrub ok 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:42.814869+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:42.815718+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:43.002923+0000 mon.a (mon.0) 1576 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi087-82011-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi087-82011-31"}]': finished 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:43.003109+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.003182+0000 mon.a (mon.0) 1578 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:43.008278+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:43.009092+0000 mon.a (mon.0) 1579 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19"}]: dispatch 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:43.015142+0000 mon.c (mon.2) 248 : audit [INF] from='client.? 172.21.15.87:0/450042771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi087-82019-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:43.016069+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi087-82019-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:43.442594+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:44.533 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:32:44.533 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:32:44.533 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:32:44.533 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:32:44.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.319878+0000 osd.1 (osd.1) 107 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.322958+0000 osd.1 (osd.1) 108 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.558564+0000 osd.0 (osd.0) 123 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.560580+0000 osd.0 (osd.0) 124 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.588105+0000 mgr.y (mgr.24437) 248 : cluster [DBG] pgmap v301: 295 pgs: 40 unknown, 1 peering, 230 active+clean, 16 active+clean+snaptrim_wait, 8 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.844031+0000 osd.2 (osd.2) 95 : cluster [DBG] 19.7 scrub starts 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.845544+0000 osd.2 (osd.2) 96 : cluster [DBG] 19.7 scrub ok 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.997887+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:44.002222+0000 mon.a (mon.0) 1582 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19"}]': finished 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:44.002387+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi087-82019-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.002463+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-02-23T00:32:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:44.443524+0000 mon.c (mon.2) 250 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.319878+0000 osd.1 (osd.1) 107 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.322958+0000 osd.1 (osd.1) 108 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.558564+0000 osd.0 (osd.0) 123 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.560580+0000 osd.0 (osd.0) 124 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.588105+0000 mgr.y (mgr.24437) 248 : cluster [DBG] pgmap v301: 295 pgs: 40 unknown, 1 peering, 230 active+clean, 16 active+clean+snaptrim_wait, 8 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.844031+0000 osd.2 (osd.2) 95 : cluster [DBG] 19.7 scrub starts 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.845544+0000 osd.2 (osd.2) 96 : cluster [DBG] 19.7 scrub ok 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.997887+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:44.002222+0000 mon.a (mon.0) 1582 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19"}]': finished 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:44.002387+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi087-82019-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:44.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.002463+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-02-23T00:32:44.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:44.443524+0000 mon.c (mon.2) 250 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 -- 172.21.15.87:0/1357912789 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18540f7400 msgr2=0x7f1854103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 --2- 172.21.15.87:0/1357912789 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18540f7400 0x7f1854103a20 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7f184c009be0 tx=0x7f184c004d10).stop 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 -- 172.21.15.87:0/1357912789 shutdown_connections 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 --2- 172.21.15.87:0/1357912789 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18540f7400 0x7f1854103a20 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 --2- 172.21.15.87:0/1357912789 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18540fcbb0 0x7f18540fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 --2- 172.21.15.87:0/1357912789 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18540fd510 0x7f18540f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 -- 172.21.15.87:0/1357912789 >> 172.21.15.87:0/1357912789 conn(0x7f18540f2820 msgr2=0x7f18540f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 -- 172.21.15.87:0/1357912789 shutdown_connections 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 -- 172.21.15.87:0/1357912789 wait complete. 2024-02-23T00:32:44.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 librados: starting msgr at 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 librados: starting objecter 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.565+0000 7f185a738700 1 Processor -- start 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f185a738700 1 -- start start 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f185a738700 1 librados: setting wanted keys 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f185a738700 1 librados: calling monclient init 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f185a738700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18540f7400 0x7f185410cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f185a738700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18540fcbb0 0x7f185410d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f185a738700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18540fd510 0x7f1854107150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f185a738700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854107640 con 0x7f18540fd510 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f185a738700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18541077b0 con 0x7f18540fcbb0 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f185a738700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1854107970 con 0x7f18540f7400 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f1858cd5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18540fd510 0x7f1854107150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f1858cd5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18540fd510 0x7f1854107150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34882/0 (socket says 172.21.15.87:34882) 2024-02-23T00:32:44.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f1858cd5700 1 -- 172.21.15.87:0/2385900170 learned_addr learned my addr 172.21.15.87:0/2385900170 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f1853fff700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18540f7400 0x7f185410cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f18537fe700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18540fcbb0 0x7f185410d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f1853fff700 1 -- 172.21.15.87:0/2385900170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18540fcbb0 msgr2=0x7f185410d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f1853fff700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18540fcbb0 0x7f185410d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f1853fff700 1 -- 172.21.15.87:0/2385900170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18540fd510 msgr2=0x7f1854107150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f1853fff700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18540fd510 0x7f1854107150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.566+0000 7f1853fff700 1 -- 172.21.15.87:0/2385900170 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c0097c0 con 0x7f18540f7400 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f1858cd5700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18540fd510 0x7f1854107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f1853fff700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18540f7400 0x7f185410cf60 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f183c00b4e0 tx=0x7f183c009060).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f18517fa700 10 client.?.objecter ms_handle_connect 0x7f18540f7400 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f18517fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f18517fa700 1 -- 172.21.15.87:0/2385900170 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c00be30 con 0x7f18540f7400 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f18517fa700 1 -- 172.21.15.87:0/2385900170 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f183c0027f0 con 0x7f18540f7400 2024-02-23T00:32:44.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1854107c50 con 0x7f18540f7400 2024-02-23T00:32:44.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f18517fa700 1 -- 172.21.15.87:0/2385900170 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c012950 con 0x7f18540f7400 2024-02-23T00:32:44.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f185a738700 10 client.37100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:44.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1854107fd0 con 0x7f18540f7400 2024-02-23T00:32:44.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.567+0000 7f185a738700 1 librados: init done 2024-02-23T00:32:44.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.568+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f185405fa60 con 0x7f18540f7400 2024-02-23T00:32:44.780 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:32:44.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.568+0000 7f18517fa700 1 -- 172.21.15.87:0/2385900170 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f183c002320 con 0x7f18540f7400 2024-02-23T00:32:44.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.568+0000 7f18517fa700 1 --2- 172.21.15.87:0/2385900170 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1840062c90 0x7f1840065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:44.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.569+0000 7f18517fa700 1 -- 172.21.15.87:0/2385900170 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(213..213 src has 38..213) v4 ==== 9394+0+0 (secure 0 0 0) 0x7f183c07d5d0 con 0x7f18540f7400 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.569+0000 7f18517fa700 10 client.37100.objecter ms_dispatch 0x7f1854001550 osd_map(213..213 src has 38..213) v4 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.569+0000 7f18517fa700 3 client.37100.objecter handle_osd_map got epochs [213,213] > 0 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.569+0000 7f18517fa700 3 client.37100.objecter handle_osd_map decoding full epoch 213 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.569+0000 7f18537fe700 1 --2- 172.21.15.87:0/2385900170 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1840062c90 0x7f1840065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.569+0000 7f18517fa700 20 client.37100.objecter dump_active .. 0 homeless 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.569+0000 7f18537fe700 1 --2- 172.21.15.87:0/2385900170 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1840062c90 0x7f1840065150 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7f18541088a0 tx=0x7f184400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.569+0000 7f18517fa700 10 client.37100.objecter ms_handle_connect 0x7f1840062c90 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.573+0000 7f18517fa700 1 -- 172.21.15.87:0/2385900170 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f183c0505c0 con 0x7f18540f7400 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.739+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1854108650 con 0x7f1840062c90 2024-02-23T00:32:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.758+0000 7f18517fa700 1 -- 172.21.15.87:0/2385900170 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+731102 (secure 0 0 0) 0x7f1854108650 con 0x7f1840062c90 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 10 librados: watch_flush enter 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 10 librados: watch_flush exit 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 20 client.37100.objecter shutdown clearing up homeless session... 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 10 client.37100.objecter successfully canceled tick 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1840062c90 msgr2=0x7f1840065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 --2- 172.21.15.87:0/2385900170 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1840062c90 0x7f1840065150 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 rx=0x7f18541088a0 tx=0x7f184400b040).stop 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18540f7400 msgr2=0x7f185410cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18540f7400 0x7f185410cf60 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 rx=0x7f183c00b4e0 tx=0x7f183c009060).stop 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 shutdown_connections 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 --2- 172.21.15.87:0/2385900170 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1840062c90 0x7f1840065150 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18540fd510 0x7f1854107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18540f7400 0x7f185410cf60 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 --2- 172.21.15.87:0/2385900170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18540fcbb0 0x7f185410d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:44.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 >> 172.21.15.87:0/2385900170 conn(0x7f18540f2820 msgr2=0x7f18540f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:44.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 shutdown_connections 2024-02-23T00:32:44.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 -- 172.21.15.87:0/2385900170 wait complete. 2024-02-23T00:32:44.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:44.778+0000 7f185a738700 1 librados: shutdown 2024-02-23T00:32:44.799 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:32:44.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.319878+0000 osd.1 (osd.1) 107 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:32:44.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.322958+0000 osd.1 (osd.1) 108 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:32:44.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.558564+0000 osd.0 (osd.0) 123 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:32:44.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.560580+0000 osd.0 (osd.0) 124 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:32:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.588105+0000 mgr.y (mgr.24437) 248 : cluster [DBG] pgmap v301: 295 pgs: 40 unknown, 1 peering, 230 active+clean, 16 active+clean+snaptrim_wait, 8 active+clean+snaptrim; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.844031+0000 osd.2 (osd.2) 95 : cluster [DBG] 19.7 scrub starts 2024-02-23T00:32:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.845544+0000 osd.2 (osd.2) 96 : cluster [DBG] 19.7 scrub ok 2024-02-23T00:32:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.997887+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:44.002222+0000 mon.a (mon.0) 1582 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-19"}]': finished 2024-02-23T00:32:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:44.002387+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi087-82019-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.002463+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-02-23T00:32:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:44.443524+0000 mon.c (mon.2) 250 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:45.069 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:45.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:45.174 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:45.205 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:45.228 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:45.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:45.333 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:45.362 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:45.383 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:45.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:45.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:45.486 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:45.517 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:45.544 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:45.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:45.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:45.563 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-23T00:32:45.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:32:45.676 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.987410+0000 osd.3 (osd.3) 109 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:43.990878+0000 osd.3 (osd.3) 110 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.491099+0000 osd.6 (osd.6) 85 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.493999+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.576180+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.578064+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.639452+0000 osd.7 (osd.7) 109 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.640327+0000 osd.7 (osd.7) 110 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:44.740570+0000 mgr.y (mgr.24437) 249 : audit [DBG] from='client.37100 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:32:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.868194+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.870330+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.884334+0000 osd.2 (osd.2) 97 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:44.885202+0000 osd.2 (osd.2) 98 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.003702+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:45.020469+0000 mon.c (mon.2) 251 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:45.261735+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:45.444279+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.987410+0000 osd.3 (osd.3) 109 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:43.990878+0000 osd.3 (osd.3) 110 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.491099+0000 osd.6 (osd.6) 85 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.493999+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:32:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.576180+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.578064+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.639452+0000 osd.7 (osd.7) 109 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.640327+0000 osd.7 (osd.7) 110 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:44.740570+0000 mgr.y (mgr.24437) 249 : audit [DBG] from='client.37100 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.868194+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.870330+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.884334+0000 osd.2 (osd.2) 97 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:44.885202+0000 osd.2 (osd.2) 98 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.003702+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:45.020469+0000 mon.c (mon.2) 251 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:45.261735+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:45.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:45.444279+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.987410+0000 osd.3 (osd.3) 109 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:32:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:43.990878+0000 osd.3 (osd.3) 110 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:32:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.491099+0000 osd.6 (osd.6) 85 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:32:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.493999+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:32:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.576180+0000 osd.0 (osd.0) 125 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:32:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.578064+0000 osd.0 (osd.0) 126 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:32:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.639452+0000 osd.7 (osd.7) 109 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:32:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.640327+0000 osd.7 (osd.7) 110 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:32:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:44.740570+0000 mgr.y (mgr.24437) 249 : audit [DBG] from='client.37100 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:32:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.868194+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:32:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.870330+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:32:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.884334+0000 osd.2 (osd.2) 97 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:32:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:44.885202+0000 osd.2 (osd.2) 98 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:32:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.003702+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-02-23T00:32:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:45.020469+0000 mon.c (mon.2) 251 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:45.261735+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:45.444279+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:46.114 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-02-23T00:32:43.999504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10477,"num_objects":458,"num_object_clones":109,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":1383,"num_read_kb":1017,"num_write":1272,"num_write_kb":635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2263,"ondisk_log_size":2263,"up":762,"acting":762,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":771,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2455004,"kb_used_data":28012,"kb_used_omap":0,"kb_used_meta":2426880,"kb_avail":747506212,"statfs":{"total":767960285184,"available":765446361088,"internally_reserved":0,"allocated":28684288,"data_stored":14844033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2485125120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":112,"num_objects":-26,"num_object_clones":-28,"num_object_copies":-78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-124,"num_read_kb":-123,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":-261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997115"},"pg_stats":[{"pgid":"147.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.027599+0000","last_change":"2024-02-23T00:32:39.019171+0000","last_active":"2024-02-23T00:32:39.027599+0000","last_peered":"2024-02-23T00:32:39.027599+0000","last_clean":"2024-02-23T00:32:39.027599+0000","last_became_active":"2024-02-23T00:32:39.018879+0000","last_became_peered":"2024-02-23T00:32:39.018879+0000","last_unstale":"2024-02-23T00:32:39.027599+0000","last_undegraded":"2024-02-23T00:32:39.027599+0000","last_fullsized":"2024-02-23T00:32:39.027599+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"147.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.031239+0000","last_change":"2024-02-23T00:32:39.029301+0000","last_active":"2024-02-23T00:32:39.031239+0000","last_peered":"2024-02-23T00:32:39.031239+0000","last_clean":"2024-02-23T00:32:39.031239+0000","last_became_active":"2024-02-23T00:32:39.029057+0000","last_became_peered":"2024-02-23T00:32:39.029057+0000","last_unstale":"2024-02-23T00:32:39.031239+0000","last_undegraded":"2024-02-23T00:32:39.031239+0000","last_fullsized":"2024-02-23T00:32:39.031239+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"147.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.025499+0000","last_change":"2024-02-23T00:32:39.021334+0000","last_active":"2024-02-23T00:32:39.025499+0000","last_peered":"2024-02-23T00:32:39.025499+0000","last_clean":"2024-02-23T00:32:39.025499+0000","last_became_active":"2024-02-23T00:32:39.021102+0000","last_became_peered":"2024-02-23T00:32:39.021102+0000","last_unstale":"2024-02-23T00:32:39.025499+0000","last_undegraded":"2024-02-23T00:32:39.025499+0000","last_fullsized":"2024-02-23T00:32:39.025499+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"147.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.029756+0000","last_change":"2024-02-23T00:32:39.027651+0000","last_active":"2024-02-23T00:32:39.029756+0000","last_peered":"2024-02-23T00:32:39.029756+0000","last_clean":"2024-02-23T00:32:39.029756+0000","last_became_active":"2024-02-23T00:32:39.027360+0000","last_became_peered":"2024-02-23T00:32:39.027360+0000","last_unstale":"2024-02-23T00:32:39.029756+0000","last_undegraded":"2024-02-23T00:32:39.029756+0000","last_fullsized":"2024-02-23T00:32:39.029756+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"147.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.8","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.041579+0000","last_change":"2024-02-23T00:32:39.040096+0000","last_active":"2024-02-23T00:32:39.041579+0000","last_peered":"2024-02-23T00:32:39.041579+0000","last_clean":"2024-02-23T00:32:39.041579+0000","last_became_active":"2024-02-23T00:32:39.039643+0000","last_became_peered":"2024-02-23T00:32:39.039643+0000","last_unstale":"2024-02-23T00:32:39.041579+0000","last_undegraded":"2024-02-23T00:32:39.041579+0000","last_fullsized":"2024-02-23T00:32:39.041579+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"147.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.9","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.034269+0000","last_change":"2024-02-23T00:32:39.029906+0000","last_active":"2024-02-23T00:32:39.034269+0000","last_peered":"2024-02-23T00:32:39.034269+0000","last_clean":"2024-02-23T00:32:39.034269+0000","last_became_active":"2024-02-23T00:32:39.029610+0000","last_became_peered":"2024-02-23T00:32:39.029610+0000","last_unstale":"2024-02-23T00:32:39.034269+0000","last_undegraded":"2024-02-23T00:32:39.034269+0000","last_fullsized":"2024-02-23T00:32:39.034269+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"147.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.036788+0000","last_change":"2024-02-23T00:32:39.028442+0000","last_active":"2024-02-23T00:32:39.036788+0000","last_peered":"2024-02-23T00:32:39.036788+0000","last_clean":"2024-02-23T00:32:39.036788+0000","last_became_active":"2024-02-23T00:32:39.028172+0000","last_became_peered":"2024-02-23T00:32:39.028172+0000","last_unstale":"2024-02-23T00:32:39.036788+0000","last_undegraded":"2024-02-23T00:32:39.036788+0000","last_fullsized":"2024-02-23T00:32:39.036788+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"147.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.031730+0000","last_change":"2024-02-23T00:32:39.030266+0000","last_active":"2024-02-23T00:32:39.031730+0000","last_peered":"2024-02-23T00:32:39.031730+0000","last_clean":"2024-02-23T00:32:39.031730+0000","last_became_active":"2024-02-23T00:32:39.030108+0000","last_became_peered":"2024-02-23T00:32:39.030108+0000","last_unstale":"2024-02-23T00:32:39.031730+0000","last_undegraded":"2024-02-23T00:32:39.031730+0000","last_fullsized":"2024-02-23T00:32:39.031730+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"147.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"146.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.4","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.024777+0000","last_change":"2024-02-23T00:32:39.018602+0000","last_active":"2024-02-23T00:32:39.024777+0000","last_peered":"2024-02-23T00:32:39.024777+0000","last_clean":"2024-02-23T00:32:39.024777+0000","last_became_active":"2024-02-23T00:32:39.018297+0000","last_became_peered":"2024-02-23T00:32:39.018297+0000","last_unstale":"2024-02-23T00:32:39.024777+0000","last_undegraded":"2024-02-23T00:32:39.024777+0000","last_fullsized":"2024-02-23T00:32:39.024777+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"147.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"146.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.5","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.030155+0000","last_change":"2024-02-23T00:32:39.025812+0000","last_active":"2024-02-23T00:32:39.030155+0000","last_peered":"2024-02-23T00:32:39.030155+0000","last_clean":"2024-02-23T00:32:39.030155+0000","last_became_active":"2024-02-23T00:32:39.025408+0000","last_became_peered":"2024-02-23T00:32:39.025408+0000","last_unstale":"2024-02-23T00:32:39.030155+0000","last_undegraded":"2024-02-23T00:32:39.030155+0000","last_fullsized":"2024-02-23T00:32:39.030155+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"147.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"146.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.6","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.021416+0000","last_change":"2024-02-23T00:32:39.017498+0000","last_active":"2024-02-23T00:32:39.021416+0000","last_peered":"2024-02-23T00:32:39.021416+0000","last_clean":"2024-02-23T00:32:39.021416+0000","last_became_active":"2024-02-23T00:32:39.017176+0000","last_became_peered":"2024-02-23T00:32:39.017176+0000","last_unstale":"2024-02-23T00:32:39.021416+0000","last_undegraded":"2024-02-23T00:32:39.021416+0000","last_fullsized":"2024-02-23T00:32:39.021416+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"147.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"146.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.7","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.035431+0000","last_change":"2024-02-23T00:32:39.034149+0000","last_active":"2024-02-23T00:32:39.035431+0000","last_peered":"2024-02-23T00:32:39.035431+0000","last_clean":"2024-02-23T00:32:39.035431+0000","last_became_active":"2024-02-23T00:32:39.033882+0000","last_became_peered":"2024-02-23T00:32:39.033882+0000","last_unstale":"2024-02-23T00:32:39.035431+0000","last_undegraded":"2024-02-23T00:32:39.035431+0000","last_fullsized":"2024-02-23T00:32:39.035431+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"147.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"146.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.0","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.038721+0000","last_change":"2024-02-23T00:32:39.038501+0000","last_active":"2024-02-23T00:32:39.038721+0000","last_peered":"2024-02-23T00:32:39.038721+0000","last_clean":"2024-02-23T00:32:39.038721+0000","last_became_active":"2024-02-23T00:32:39.038283+0000","last_became_peered":"2024-02-23T00:32:39.038283+0000","last_unstale":"2024-02-23T00:32:39.038721+0000","last_undegraded":"2024-02-23T00:32:39.038721+0000","last_fullsized":"2024-02-23T00:32:39.038721+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"147.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"146.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.1","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.029765+0000","last_change":"2024-02-23T00:32:39.029109+0000","last_active":"2024-02-23T00:32:39.029765+0000","last_peered":"2024-02-23T00:32:39.029765+0000","last_clean":"2024-02-23T00:32:39.029765+0000","last_became_active":"2024-02-23T00:32:39.028799+0000","last_became_peered":"2024-02-23T00:32:39.028799+0000","last_unstale":"2024-02-23T00:32:39.029765+0000","last_undegraded":"2024-02-23T00:32:39.029765+0000","last_fullsized":"2024-02-23T00:32:39.029765+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"147.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"146.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.2","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.038133+0000","last_change":"2024-02-23T00:32:39.037665+0000","last_active":"2024-02-23T00:32:39.038133+0000","last_peered":"2024-02-23T00:32:39.038133+0000","last_clean":"2024-02-23T00:32:39.038133+0000","last_became_active":"2024-02-23T00:32:39.037293+0000","last_became_peered":"2024-02-23T00:32:39.037293+0000","last_unstale":"2024-02-23T00:32:39.038133+0000","last_undegraded":"2024-02-23T00:32:39.038133+0000","last_fullsized":"2024-02-23T00:32:39.038133+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"147.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"146.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.3","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.039081+0000","last_change":"2024-02-23T00:32:39.034318+0000","last_active":"2024-02-23T00:32:39.039081+0000","last_peered":"2024-02-23T00:32:39.039081+0000","last_clean":"2024-02-23T00:32:39.039081+0000","last_became_active":"2024-02-23T00:32:39.034046+0000","last_became_peered":"2024-02-23T00:32:39.034046+0000","last_unstale":"2024-02-23T00:32:39.039081+0000","last_undegraded":"2024-02-23T00:32:39.039081+0000","last_fullsized":"2024-02-23T00:32:39.039081+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"147.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.1c","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.033080+0000","last_change":"2024-02-23T00:32:39.032748+0000","last_active":"2024-02-23T00:32:39.033080+0000","last_peered":"2024-02-23T00:32:39.033080+0000","last_clean":"2024-02-23T00:32:39.033080+0000","last_became_active":"2024-02-23T00:32:39.032440+0000","last_became_peered":"2024-02-23T00:32:39.032440+0000","last_unstale":"2024-02-23T00:32:39.033080+0000","last_undegraded":"2024-02-23T00:32:39.033080+0000","last_fullsized":"2024-02-23T00:32:39.033080+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"147.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.1d","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.033275+0000","last_change":"2024-02-23T00:32:39.026715+0000","last_active":"2024-02-23T00:32:39.033275+0000","last_peered":"2024-02-23T00:32:39.033275+0000","last_clean":"2024-02-23T00:32:39.033275+0000","last_became_active":"2024-02-23T00:32:39.026380+0000","last_became_peered":"2024-02-23T00:32:39.026380+0000","last_unstale":"2024-02-23T00:32:39.033275+0000","last_undegraded":"2024-02-23T00:32:39.033275+0000","last_fullsized":"2024-02-23T00:32:39.033275+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"147.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.1e","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.025994+0000","last_change":"2024-02-23T00:32:39.025414+0000","last_active":"2024-02-23T00:32:39.025994+0000","last_peered":"2024-02-23T00:32:39.025994+0000","last_clean":"2024-02-23T00:32:39.025994+0000","last_became_active":"2024-02-23T00:32:39.025148+0000","last_became_peered":"2024-02-23T00:32:39.025148+0000","last_unstale":"2024-02-23T00:32:39.025994+0000","last_undegraded":"2024-02-23T00:32:39.025994+0000","last_fullsized":"2024-02-23T00:32:39.025994+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"147.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.1f","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.032742+0000","last_change":"2024-02-23T00:32:39.032130+0000","last_active":"2024-02-23T00:32:39.032742+0000","last_peered":"2024-02-23T00:32:39.032742+0000","last_clean":"2024-02-23T00:32:39.032742+0000","last_became_active":"2024-02-23T00:32:39.031719+0000","last_became_peered":"2024-02-23T00:32:39.031719+0000","last_unstale":"2024-02-23T00:32:39.032742+0000","last_undegraded":"2024-02-23T00:32:39.032742+0000","last_fullsized":"2024-02-23T00:32:39.032742+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"147.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.18","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.020369+0000","last_change":"2024-02-23T00:32:39.015769+0000","last_active":"2024-02-23T00:32:39.020369+0000","last_peered":"2024-02-23T00:32:39.020369+0000","last_clean":"2024-02-23T00:32:39.020369+0000","last_became_active":"2024-02-23T00:32:39.015508+0000","last_became_peered":"2024-02-23T00:32:39.015508+0000","last_unstale":"2024-02-23T00:32:39.020369+0000","last_undegraded":"2024-02-23T00:32:39.020369+0000","last_fullsized":"2024-02-23T00:32:39.020369+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"147.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.19","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.035395+0000","last_change":"2024-02-23T00:32:39.033581+0000","last_active":"2024-02-23T00:32:39.035395+0000","last_peered":"2024-02-23T00:32:39.035395+0000","last_clean":"2024-02-23T00:32:39.035395+0000","last_became_active":"2024-02-23T00:32:39.033300+0000","last_became_peered":"2024-02-23T00:32:39.033300+0000","last_unstale":"2024-02-23T00:32:39.035395+0000","last_undegraded":"2024-02-23T00:32:39.035395+0000","last_fullsized":"2024-02-23T00:32:39.035395+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"147.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.1a","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.039589+0000","last_change":"2024-02-23T00:32:39.035066+0000","last_active":"2024-02-23T00:32:39.039589+0000","last_peered":"2024-02-23T00:32:39.039589+0000","last_clean":"2024-02-23T00:32:39.039589+0000","last_became_active":"2024-02-23T00:32:39.034784+0000","last_became_peered":"2024-02-23T00:32:39.034784+0000","last_unstale":"2024-02-23T00:32:39.039589+0000","last_undegraded":"2024-02-23T00:32:39.039589+0000","last_fullsized":"2024-02-23T00:32:39.039589+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"147.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.1b","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.037933+0000","last_change":"2024-02-23T00:32:39.037663+0000","last_active":"2024-02-23T00:32:39.037933+0000","last_peered":"2024-02-23T00:32:39.037933+0000","last_clean":"2024-02-23T00:32:39.037933+0000","last_became_active":"2024-02-23T00:32:39.037412+0000","last_became_peered":"2024-02-23T00:32:39.037412+0000","last_unstale":"2024-02-23T00:32:39.037933+0000","last_undegraded":"2024-02-23T00:32:39.037933+0000","last_fullsized":"2024-02-23T00:32:39.037933+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"147.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.14","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.034276+0000","last_change":"2024-02-23T00:32:39.031510+0000","last_active":"2024-02-23T00:32:39.034276+0000","last_peered":"2024-02-23T00:32:39.034276+0000","last_clean":"2024-02-23T00:32:39.034276+0000","last_became_active":"2024-02-23T00:32:39.031215+0000","last_became_peered":"2024-02-23T00:32:39.031215+0000","last_unstale":"2024-02-23T00:32:39.034276+0000","last_undegraded":"2024-02-23T00:32:39.034276+0000","last_fullsized":"2024-02-23T00:32:39.034276+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"147.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.15","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.039136+0000","last_change":"2024-02-23T00:32:39.031467+0000","last_active":"2024-02-23T00:32:39.039136+0000","last_peered":"2024-02-23T00:32:39.039136+0000","last_clean":"2024-02-23T00:32:39.039136+0000","last_became_active":"2024-02-23T00:32:39.031160+0000","last_became_peered":"2024-02-23T00:32:39.031160+0000","last_unstale":"2024-02-23T00:32:39.039136+0000","last_undegraded":"2024-02-23T00:32:39.039136+0000","last_fullsized":"2024-02-23T00:32:39.039136+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"147.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.16","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.031291+0000","last_change":"2024-02-23T00:32:39.030620+0000","last_active":"2024-02-23T00:32:39.031291+0000","last_peered":"2024-02-23T00:32:39.031291+0000","last_clean":"2024-02-23T00:32:39.031291+0000","last_became_active":"2024-02-23T00:32:39.030322+0000","last_became_peered":"2024-02-23T00:32:39.030322+0000","last_unstale":"2024-02-23T00:32:39.031291+0000","last_undegraded":"2024-02-23T00:32:39.031291+0000","last_fullsized":"2024-02-23T00:32:39.031291+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"147.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.17","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.034536+0000","last_change":"2024-02-23T00:32:39.029425+0000","last_active":"2024-02-23T00:32:39.034536+0000","last_peered":"2024-02-23T00:32:39.034536+0000","last_clean":"2024-02-23T00:32:39.034536+0000","last_became_active":"2024-02-23T00:32:39.029155+0000","last_became_peered":"2024-02-23T00:32:39.029155+0000","last_unstale":"2024-02-23T00:32:39.034536+0000","last_undegraded":"2024-02-23T00:32:39.034536+0000","last_fullsized":"2024-02-23T00:32:39.034536+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"147.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.10","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.034251+0000","last_change":"2024-02-23T00:32:39.032846+0000","last_active":"2024-02-23T00:32:39.034251+0000","last_peered":"2024-02-23T00:32:39.034251+0000","last_clean":"2024-02-23T00:32:39.034251+0000","last_became_active":"2024-02-23T00:32:39.032557+0000","last_became_peered":"2024-02-23T00:32:39.032557+0000","last_unstale":"2024-02-23T00:32:39.034251+0000","last_undegraded":"2024-02-23T00:32:39.034251+0000","last_fullsized":"2024-02-23T00:32:39.034251+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"147.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.11","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.024076+0000","last_change":"2024-02-23T00:32:39.023516+0000","last_active":"2024-02-23T00:32:39.024076+0000","last_peered":"2024-02-23T00:32:39.024076+0000","last_clean":"2024-02-23T00:32:39.024076+0000","last_became_active":"2024-02-23T00:32:39.023208+0000","last_became_peered":"2024-02-23T00:32:39.023208+0000","last_unstale":"2024-02-23T00:32:39.024076+0000","last_undegraded":"2024-02-23T00:32:39.024076+0000","last_fullsized":"2024-02-23T00:32:39.024076+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"147.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.12","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.036557+0000","last_change":"2024-02-23T00:32:39.031200+0000","last_active":"2024-02-23T00:32:39.036557+0000","last_peered":"2024-02-23T00:32:39.036557+0000","last_clean":"2024-02-23T00:32:39.036557+0000","last_became_active":"2024-02-23T00:32:39.030792+0000","last_became_peered":"2024-02-23T00:32:39.030792+0000","last_unstale":"2024-02-23T00:32:39.036557+0000","last_undegraded":"2024-02-23T00:32:39.036557+0000","last_fullsized":"2024-02-23T00:32:39.036557+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"147.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:32:42.995554+0000","last_change":"2024-02-23T00:32:42.995554+0000","last_active":"2024-02-23T00:32:42.995554+0000","last_peered":"2024-02-23T00:32:42.995554+0000","last_clean":"2024-02-23T00:32:42.995554+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:32:42.995554+0000","last_undegraded":"2024-02-23T00:32:42.995554+0000","last_fullsized":"2024-02-23T00:32:42.995554+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:42.995554+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.995554+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"144.13","version":"0'0","reported_seq":10,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.039060+0000","last_change":"2024-02-23T00:32:39.032824+0000","last_active":"2024-02-23T00:32:39.039060+0000","last_peered":"2024-02-23T00:32:39.039060+0000","last_clean":"2024-02-23T00:32:39.039060+0000","last_became_active":"2024-02-23T00:32:39.032552+0000","last_became_peered":"2024-02-23T00:32:39.032552+0000","last_unstale":"2024-02-23T00:32:39.039060+0000","last_undegraded":"2024-02-23T00:32:39.039060+0000","last_fullsized":"2024-02-23T00:32:39.039060+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:37.993231+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.993231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.0","version":"204'52","reported_seq":166,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.012442+0000","last_change":"2024-02-23T00:32:37.099559+0000","last_active":"2024-02-23T00:32:39.012442+0000","last_peered":"2024-02-23T00:32:39.012442+0000","last_clean":"2024-02-23T00:32:39.012442+0000","last_became_active":"2024-02-23T00:31:06.330477+0000","last_became_peered":"2024-02-23T00:31:06.330477+0000","last_unstale":"2024-02-23T00:32:39.012442+0000","last_undegraded":"2024-02-23T00:32:39.012442+0000","last_fullsized":"2024-02-23T00:32:39.012442+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"63.1","version":"202'65","reported_seq":172,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.006681+0000","last_change":"2024-02-23T00:32:35.055450+0000","last_active":"2024-02-23T00:32:39.006681+0000","last_peered":"2024-02-23T00:32:39.006681+0000","last_clean":"2024-02-23T00:32:39.006681+0000","last_became_active":"2024-02-23T00:31:06.258140+0000","last_became_peered":"2024-02-23T00:31:06.258140+0000","last_unstale":"2024-02-23T00:32:39.006681+0000","last_undegraded":"2024-02-23T00:32:39.006681+0000","last_fullsized":"2024-02-23T00:32:39.006681+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":165,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"63.2","version":"204'16","reported_seq":139,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.014659+0000","last_change":"2024-02-23T00:32:37.116785+0000","last_active":"2024-02-23T00:32:39.014659+0000","last_peered":"2024-02-23T00:32:39.014659+0000","last_clean":"2024-02-23T00:32:39.014659+0000","last_became_active":"2024-02-23T00:31:06.329726+0000","last_became_peered":"2024-02-23T00:31:06.329726+0000","last_unstale":"2024-02-23T00:32:39.014659+0000","last_undegraded":"2024-02-23T00:32:39.014659+0000","last_fullsized":"2024-02-23T00:32:39.014659+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"63.3","version":"206'13","reported_seq":131,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.117871+0000","last_change":"2024-02-23T00:32:39.117871+0000","last_active":"2024-02-23T00:32:39.117871+0000","last_peered":"2024-02-23T00:32:39.117871+0000","last_clean":"2024-02-23T00:32:39.117871+0000","last_became_active":"2024-02-23T00:31:06.258997+0000","last_became_peered":"2024-02-23T00:31:06.258997+0000","last_unstale":"2024-02-23T00:32:39.117871+0000","last_undegraded":"2024-02-23T00:32:39.117871+0000","last_fullsized":"2024-02-23T00:32:39.117871+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"63.4","version":"208'26","reported_seq":137,"reported_epoch":208,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:32:39.119414+0000","last_change":"2024-02-23T00:32:39.117889+0000","last_active":"2024-02-23T00:32:39.119414+0000","last_peered":"2024-02-23T00:32:39.119414+0000","last_clean":"2024-02-23T00:32:39.119414+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:32:39.119414+0000","last_undegraded":"2024-02-23T00:32:39.119414+0000","last_fullsized":"2024-02-23T00:32:39.119414+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.5","version":"208'89","reported_seq":192,"reported_epoch":208,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:32:39.125171+0000","last_change":"2024-02-23T00:32:35.105484+0000","last_active":"2024-02-23T00:32:39.125171+0000","last_peered":"2024-02-23T00:32:39.125171+0000","last_clean":"2024-02-23T00:32:39.125171+0000","last_became_active":"2024-02-23T00:31:06.258422+0000","last_became_peered":"2024-02-23T00:31:06.258422+0000","last_unstale":"2024-02-23T00:32:39.125171+0000","last_undegraded":"2024-02-23T00:32:39.125171+0000","last_fullsized":"2024-02-23T00:32:39.125171+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":242,"num_objects":22,"num_object_clones":15,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.6","version":"206'51","reported_seq":173,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.117374+0000","last_change":"2024-02-23T00:32:39.117374+0000","last_active":"2024-02-23T00:32:39.117374+0000","last_peered":"2024-02-23T00:32:39.117374+0000","last_clean":"2024-02-23T00:32:39.117374+0000","last_became_active":"2024-02-23T00:31:06.313441+0000","last_became_peered":"2024-02-23T00:31:06.313441+0000","last_unstale":"2024-02-23T00:32:39.117374+0000","last_undegraded":"2024-02-23T00:32:39.117374+0000","last_fullsized":"2024-02-23T00:32:39.117374+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"63.7","version":"203'67","reported_seq":175,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.013413+0000","last_change":"2024-02-23T00:32:35.113067+0000","last_active":"2024-02-23T00:32:39.013413+0000","last_peered":"2024-02-23T00:32:39.013413+0000","last_clean":"2024-02-23T00:32:39.013413+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:32:39.013413+0000","last_undegraded":"2024-02-23T00:32:39.013413+0000","last_fullsized":"2024-02-23T00:32:39.013413+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":173,"num_objects":16,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"63.8","version":"198'68","reported_seq":188,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.024955+0000","last_change":"2024-02-23T00:32:31.058164+0000","last_active":"2024-02-23T00:32:39.024955+0000","last_peered":"2024-02-23T00:32:39.024955+0000","last_clean":"2024-02-23T00:32:39.024955+0000","last_became_active":"2024-02-23T00:31:06.363442+0000","last_became_peered":"2024-02-23T00:31:06.363442+0000","last_unstale":"2024-02-23T00:32:39.024955+0000","last_undegraded":"2024-02-23T00:32:39.024955+0000","last_fullsized":"2024-02-23T00:32:39.024955+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"63.9","version":"208'17","reported_seq":140,"reported_epoch":208,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:32:39.125707+0000","last_change":"2024-02-23T00:32:39.124101+0000","last_active":"2024-02-23T00:32:39.125707+0000","last_peered":"2024-02-23T00:32:39.125707+0000","last_clean":"2024-02-23T00:32:39.125707+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:32:39.125707+0000","last_undegraded":"2024-02-23T00:32:39.125707+0000","last_fullsized":"2024-02-23T00:32:39.125707+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"63.a","version":"198'13","reported_seq":131,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.006537+0000","last_change":"2024-02-23T00:32:31.046887+0000","last_active":"2024-02-23T00:32:39.006537+0000","last_peered":"2024-02-23T00:32:39.006537+0000","last_clean":"2024-02-23T00:32:39.006537+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:32:39.006537+0000","last_undegraded":"2024-02-23T00:32:39.006537+0000","last_fullsized":"2024-02-23T00:32:39.006537+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"63.b","version":"202'34","reported_seq":158,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.026233+0000","last_change":"2024-02-23T00:32:35.106187+0000","last_active":"2024-02-23T00:32:39.026233+0000","last_peered":"2024-02-23T00:32:39.026233+0000","last_clean":"2024-02-23T00:32:39.026233+0000","last_became_active":"2024-02-23T00:31:06.294667+0000","last_became_peered":"2024-02-23T00:31:06.294667+0000","last_unstale":"2024-02-23T00:32:39.026233+0000","last_undegraded":"2024-02-23T00:32:39.026233+0000","last_fullsized":"2024-02-23T00:32:39.026233+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"63.c","version":"208'68","reported_seq":179,"reported_epoch":208,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:32:39.125590+0000","last_change":"2024-02-23T00:32:37.112887+0000","last_active":"2024-02-23T00:32:39.125590+0000","last_peered":"2024-02-23T00:32:39.125590+0000","last_clean":"2024-02-23T00:32:39.125590+0000","last_became_active":"2024-02-23T00:31:06.233844+0000","last_became_peered":"2024-02-23T00:31:06.233844+0000","last_unstale":"2024-02-23T00:32:39.125590+0000","last_undegraded":"2024-02-23T00:32:39.125590+0000","last_fullsized":"2024-02-23T00:32:39.125590+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"2"}]},{"pgid":"63.d","version":"198'26","reported_seq":139,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.008150+0000","last_change":"2024-02-23T00:32:31.110226+0000","last_active":"2024-02-23T00:32:39.008150+0000","last_peered":"2024-02-23T00:32:39.008150+0000","last_clean":"2024-02-23T00:32:39.008150+0000","last_became_active":"2024-02-23T00:31:06.262251+0000","last_became_peered":"2024-02-23T00:31:06.262251+0000","last_unstale":"2024-02-23T00:32:39.008150+0000","last_undegraded":"2024-02-23T00:32:39.008150+0000","last_fullsized":"2024-02-23T00:32:39.008150+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"63.e","version":"208'61","reported_seq":177,"reported_epoch":208,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:32:39.119501+0000","last_change":"2024-02-23T00:32:39.117931+0000","last_active":"2024-02-23T00:32:39.119501+0000","last_peered":"2024-02-23T00:32:39.119501+0000","last_clean":"2024-02-23T00:32:39.119501+0000","last_became_active":"2024-02-23T00:31:06.356847+0000","last_became_peered":"2024-02-23T00:31:06.356847+0000","last_unstale":"2024-02-23T00:32:39.119501+0000","last_undegraded":"2024-02-23T00:32:39.119501+0000","last_fullsized":"2024-02-23T00:32:39.119501+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":115,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"63.f","version":"198'68","reported_seq":185,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019116+0000","last_change":"2024-02-23T00:32:31.062227+0000","last_active":"2024-02-23T00:32:39.019116+0000","last_peered":"2024-02-23T00:32:39.019116+0000","last_clean":"2024-02-23T00:32:39.019116+0000","last_became_active":"2024-02-23T00:31:06.364609+0000","last_became_peered":"2024-02-23T00:31:06.364609+0000","last_unstale":"2024-02-23T00:32:39.019116+0000","last_undegraded":"2024-02-23T00:32:39.019116+0000","last_fullsized":"2024-02-23T00:32:39.019116+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"63.10","version":"206'30","reported_seq":149,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.123752+0000","last_change":"2024-02-23T00:32:39.123752+0000","last_active":"2024-02-23T00:32:39.123752+0000","last_peered":"2024-02-23T00:32:39.123752+0000","last_clean":"2024-02-23T00:32:39.123752+0000","last_became_active":"2024-02-23T00:31:06.342164+0000","last_became_peered":"2024-02-23T00:31:06.342164+0000","last_unstale":"2024-02-23T00:32:39.123752+0000","last_undegraded":"2024-02-23T00:32:39.123752+0000","last_fullsized":"2024-02-23T00:32:39.123752+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"2"}]},{"pgid":"63.11","version":"202'36","reported_seq":149,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.006633+0000","last_change":"2024-02-23T00:32:35.105461+0000","last_active":"2024-02-23T00:32:39.006633+0000","last_peered":"2024-02-23T00:32:39.006633+0000","last_clean":"2024-02-23T00:32:39.006633+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:32:39.006633+0000","last_undegraded":"2024-02-23T00:32:39.006633+0000","last_fullsized":"2024-02-23T00:32:39.006633+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":53,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"63.12","version":"204'68","reported_seq":177,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.016332+0000","last_change":"2024-02-23T00:32:37.112880+0000","last_active":"2024-02-23T00:32:39.016332+0000","last_peered":"2024-02-23T00:32:39.016332+0000","last_clean":"2024-02-23T00:32:39.016332+0000","last_became_active":"2024-02-23T00:31:06.281619+0000","last_became_peered":"2024-02-23T00:31:06.281619+0000","last_unstale":"2024-02-23T00:32:39.016332+0000","last_undegraded":"2024-02-23T00:32:39.016332+0000","last_fullsized":"2024-02-23T00:32:39.016332+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"3"}]},{"pgid":"63.13","version":"200'56","reported_seq":176,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.026128+0000","last_change":"2024-02-23T00:32:35.046686+0000","last_active":"2024-02-23T00:32:39.026128+0000","last_peered":"2024-02-23T00:32:39.026128+0000","last_clean":"2024-02-23T00:32:39.026128+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:32:39.026128+0000","last_undegraded":"2024-02-23T00:32:39.026128+0000","last_fullsized":"2024-02-23T00:32:39.026128+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"63.14","version":"208'85","reported_seq":192,"reported_epoch":208,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:32:39.125704+0000","last_change":"2024-02-23T00:32:35.109610+0000","last_active":"2024-02-23T00:32:39.125704+0000","last_peered":"2024-02-23T00:32:39.125704+0000","last_clean":"2024-02-23T00:32:39.125704+0000","last_became_active":"2024-02-23T00:31:06.234521+0000","last_became_peered":"2024-02-23T00:31:06.234521+0000","last_unstale":"2024-02-23T00:32:39.125704+0000","last_undegraded":"2024-02-23T00:32:39.125704+0000","last_fullsized":"2024-02-23T00:32:39.125704+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"63.15","version":"208'56","reported_seq":169,"reported_epoch":208,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:32:39.099829+0000","last_change":"2024-02-23T00:32:39.098523+0000","last_active":"2024-02-23T00:32:39.099829+0000","last_peered":"2024-02-23T00:32:39.099829+0000","last_clean":"2024-02-23T00:32:39.099829+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:32:39.099829+0000","last_undegraded":"2024-02-23T00:32:39.099829+0000","last_fullsized":"2024-02-23T00:32:39.099829+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"63.16","version":"204'51","reported_seq":174,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007042+0000","last_change":"2024-02-23T00:32:37.107613+0000","last_active":"2024-02-23T00:32:39.007042+0000","last_peered":"2024-02-23T00:32:39.007042+0000","last_clean":"2024-02-23T00:32:39.007042+0000","last_became_active":"2024-02-23T00:31:06.367069+0000","last_became_peered":"2024-02-23T00:31:06.367069+0000","last_unstale":"2024-02-23T00:32:39.007042+0000","last_undegraded":"2024-02-23T00:32:39.007042+0000","last_fullsized":"2024-02-23T00:32:39.007042+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"63.17","version":"204'65","reported_seq":176,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.008943+0000","last_change":"2024-02-23T00:32:37.114427+0000","last_active":"2024-02-23T00:32:39.008943+0000","last_peered":"2024-02-23T00:32:39.008943+0000","last_clean":"2024-02-23T00:32:39.008943+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:32:39.008943+0000","last_undegraded":"2024-02-23T00:32:39.008943+0000","last_fullsized":"2024-02-23T00:32:39.008943+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":165,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"63.18","version":"202'56","reported_seq":177,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.018663+0000","last_change":"2024-02-23T00:32:35.046988+0000","last_active":"2024-02-23T00:32:39.018663+0000","last_peered":"2024-02-23T00:32:39.018663+0000","last_clean":"2024-02-23T00:32:39.018663+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:32:39.018663+0000","last_undegraded":"2024-02-23T00:32:39.018663+0000","last_fullsized":"2024-02-23T00:32:39.018663+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"63.19","version":"206'45","reported_seq":165,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.098518+0000","last_change":"2024-02-23T00:32:39.098518+0000","last_active":"2024-02-23T00:32:39.098518+0000","last_peered":"2024-02-23T00:32:39.098518+0000","last_clean":"2024-02-23T00:32:39.098518+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:32:39.098518+0000","last_undegraded":"2024-02-23T00:32:39.098518+0000","last_fullsized":"2024-02-23T00:32:39.098518+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"7","length":"1"}]},{"pgid":"63.1a","version":"202'68","reported_seq":184,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007500+0000","last_change":"2024-02-23T00:32:37.107437+0000","last_active":"2024-02-23T00:32:39.007500+0000","last_peered":"2024-02-23T00:32:39.007500+0000","last_clean":"2024-02-23T00:32:39.007500+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:32:39.007500+0000","last_undegraded":"2024-02-23T00:32:39.007500+0000","last_fullsized":"2024-02-23T00:32:39.007500+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"3"},{"start":"c","length":"1"}]},{"pgid":"63.1b","version":"208'37","reported_seq":146,"reported_epoch":208,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:32:39.083680+0000","last_change":"2024-02-23T00:32:39.081731+0000","last_active":"2024-02-23T00:32:39.083680+0000","last_peered":"2024-02-23T00:32:39.083680+0000","last_clean":"2024-02-23T00:32:39.083680+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:32:39.083680+0000","last_undegraded":"2024-02-23T00:32:39.083680+0000","last_fullsized":"2024-02-23T00:32:39.083680+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1c","version":"206'30","reported_seq":148,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.117852+0000","last_change":"2024-02-23T00:32:39.117852+0000","last_active":"2024-02-23T00:32:39.117852+0000","last_peered":"2024-02-23T00:32:39.117852+0000","last_clean":"2024-02-23T00:32:39.117852+0000","last_became_active":"2024-02-23T00:31:06.338540+0000","last_became_peered":"2024-02-23T00:31:06.338540+0000","last_unstale":"2024-02-23T00:32:39.117852+0000","last_undegraded":"2024-02-23T00:32:39.117852+0000","last_fullsized":"2024-02-23T00:32:39.117852+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"2"}]},{"pgid":"63.1d","version":"191'3","reported_seq":127,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.124070+0000","last_change":"2024-02-23T00:32:39.124070+0000","last_active":"2024-02-23T00:32:39.124070+0000","last_peered":"2024-02-23T00:32:39.124070+0000","last_clean":"2024-02-23T00:32:39.124070+0000","last_became_active":"2024-02-23T00:31:06.337635+0000","last_became_peered":"2024-02-23T00:31:06.337635+0000","last_unstale":"2024-02-23T00:32:39.124070+0000","last_undegraded":"2024-02-23T00:32:39.124070+0000","last_fullsized":"2024-02-23T00:32:39.124070+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"2"}]},{"pgid":"63.1e","version":"206'39","reported_seq":149,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.081705+0000","last_change":"2024-02-23T00:32:39.081705+0000","last_active":"2024-02-23T00:32:39.081705+0000","last_peered":"2024-02-23T00:32:39.081705+0000","last_clean":"2024-02-23T00:32:39.081705+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:32:39.081705+0000","last_undegraded":"2024-02-23T00:32:39.081705+0000","last_fullsized":"2024-02-23T00:32:39.081705+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":99,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"63.1f","version":"202'52","reported_seq":165,"reported_epoch":208,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:32:39.008805+0000","last_change":"2024-02-23T00:32:35.052955+0000","last_active":"2024-02-23T00:32:39.008805+0000","last_peered":"2024-02-23T00:32:39.008805+0000","last_clean":"2024-02-23T00:32:39.008805+0000","last_became_active":"2024-02-23T00:31:06.355623+0000","last_became_peered":"2024-02-23T00:31:06.355623+0000","last_unstale":"2024-02-23T00:32:39.008805+0000","last_undegraded":"2024-02-23T00:32:39.008805+0000","last_fullsized":"2024-02-23T00:32:39.008805+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"}]},{"pgid":"5.9","version":"62'8","reported_seq":246,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.007386+0000","last_change":"2024-02-23T00:32:11.743347+0000","last_active":"2024-02-23T00:32:40.007386+0000","last_peered":"2024-02-23T00:32:40.007386+0000","last_clean":"2024-02-23T00:32:40.007386+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:32:40.007386+0000","last_undegraded":"2024-02-23T00:32:40.007386+0000","last_fullsized":"2024-02-23T00:32:40.007386+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:11.743278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.743278+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":233,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006689+0000","last_change":"2024-02-23T00:30:25.054238+0000","last_active":"2024-02-23T00:32:39.006689+0000","last_peered":"2024-02-23T00:32:39.006689+0000","last_clean":"2024-02-23T00:32:39.006689+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:32:39.006689+0000","last_undegraded":"2024-02-23T00:32:39.006689+0000","last_fullsized":"2024-02-23T00:32:39.006689+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:30:25.054140+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.054140+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007291+0000","last_change":"2024-02-23T00:31:28.776880+0000","last_active":"2024-02-23T00:32:39.007291+0000","last_peered":"2024-02-23T00:32:39.007291+0000","last_clean":"2024-02-23T00:32:39.007291+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:32:39.007291+0000","last_undegraded":"2024-02-23T00:32:39.007291+0000","last_fullsized":"2024-02-23T00:32:39.007291+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:28.776788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:28.776788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":165,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006725+0000","last_change":"2024-02-23T00:31:31.323024+0000","last_active":"2024-02-23T00:32:39.006725+0000","last_peered":"2024-02-23T00:32:39.006725+0000","last_clean":"2024-02-23T00:32:39.006725+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:32:39.006725+0000","last_undegraded":"2024-02-23T00:32:39.006725+0000","last_fullsized":"2024-02-23T00:32:39.006725+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.322940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.322940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.016225+0000","last_change":"2024-02-23T00:30:48.782537+0000","last_active":"2024-02-23T00:32:39.016225+0000","last_peered":"2024-02-23T00:32:39.016225+0000","last_clean":"2024-02-23T00:32:39.016225+0000","last_became_active":"2024-02-23T00:30:48.777752+0000","last_became_peered":"2024-02-23T00:30:48.777752+0000","last_unstale":"2024-02-23T00:32:39.016225+0000","last_undegraded":"2024-02-23T00:32:39.016225+0000","last_fullsized":"2024-02-23T00:32:39.016225+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":163,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.016053+0000","last_change":"2024-02-23T00:32:37.440685+0000","last_active":"2024-02-23T00:32:40.016053+0000","last_peered":"2024-02-23T00:32:40.016053+0000","last_clean":"2024-02-23T00:32:40.016053+0000","last_became_active":"2024-02-23T00:31:29.077539+0000","last_became_peered":"2024-02-23T00:31:29.077539+0000","last_unstale":"2024-02-23T00:32:40.016053+0000","last_undegraded":"2024-02-23T00:32:40.016053+0000","last_fullsized":"2024-02-23T00:32:40.016053+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:37.440614+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.440614+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":207,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.001940+0000","last_change":"2024-02-23T00:32:09.590050+0000","last_active":"2024-02-23T00:32:39.001940+0000","last_peered":"2024-02-23T00:32:39.001940+0000","last_clean":"2024-02-23T00:32:39.001940+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:32:39.001940+0000","last_undegraded":"2024-02-23T00:32:39.001940+0000","last_fullsized":"2024-02-23T00:32:39.001940+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:32:09.589983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.589983+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.012671+0000","last_change":"2024-02-23T00:32:38.328306+0000","last_active":"2024-02-23T00:32:39.012671+0000","last_peered":"2024-02-23T00:32:39.012671+0000","last_clean":"2024-02-23T00:32:39.012671+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:32:39.012671+0000","last_undegraded":"2024-02-23T00:32:39.012671+0000","last_fullsized":"2024-02-23T00:32:39.012671+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.328237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.328237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007648+0000","last_change":"2024-02-23T00:30:38.736599+0000","last_active":"2024-02-23T00:32:39.007648+0000","last_peered":"2024-02-23T00:32:39.007648+0000","last_clean":"2024-02-23T00:32:39.007648+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:32:39.007648+0000","last_undegraded":"2024-02-23T00:32:39.007648+0000","last_fullsized":"2024-02-23T00:32:39.007648+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:38.736492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.736492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003488+0000","last_change":"2024-02-23T00:32:06.561440+0000","last_active":"2024-02-23T00:32:39.003488+0000","last_peered":"2024-02-23T00:32:39.003488+0000","last_clean":"2024-02-23T00:32:39.003488+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:32:39.003488+0000","last_undegraded":"2024-02-23T00:32:39.003488+0000","last_fullsized":"2024-02-23T00:32:39.003488+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:06.561374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:06.561374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.014394+0000","last_change":"2024-02-23T00:30:48.713856+0000","last_active":"2024-02-23T00:32:39.014394+0000","last_peered":"2024-02-23T00:32:39.014394+0000","last_clean":"2024-02-23T00:32:39.014394+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:32:39.014394+0000","last_undegraded":"2024-02-23T00:32:39.014394+0000","last_fullsized":"2024-02-23T00:32:39.014394+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":222,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.016586+0000","last_change":"2024-02-23T00:31:41.019548+0000","last_active":"2024-02-23T00:32:39.016586+0000","last_peered":"2024-02-23T00:32:39.016586+0000","last_clean":"2024-02-23T00:32:39.016586+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:32:39.016586+0000","last_undegraded":"2024-02-23T00:32:39.016586+0000","last_fullsized":"2024-02-23T00:32:39.016586+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.019441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.019441+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009779+0000","last_change":"2024-02-23T00:31:20.632804+0000","last_active":"2024-02-23T00:32:39.009779+0000","last_peered":"2024-02-23T00:32:39.009779+0000","last_clean":"2024-02-23T00:32:39.009779+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:32:39.009779+0000","last_undegraded":"2024-02-23T00:32:39.009779+0000","last_fullsized":"2024-02-23T00:32:39.009779+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:20.632730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:20.632730+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004061+0000","last_change":"2024-02-23T00:30:40.787256+0000","last_active":"2024-02-23T00:32:39.004061+0000","last_peered":"2024-02-23T00:32:39.004061+0000","last_clean":"2024-02-23T00:32:39.004061+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:32:39.004061+0000","last_undegraded":"2024-02-23T00:32:39.004061+0000","last_fullsized":"2024-02-23T00:32:39.004061+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.787191+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":171,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.018899+0000","last_change":"2024-02-23T00:32:38.683758+0000","last_active":"2024-02-23T00:32:39.018899+0000","last_peered":"2024-02-23T00:32:39.018899+0000","last_clean":"2024-02-23T00:32:39.018899+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:32:39.018899+0000","last_undegraded":"2024-02-23T00:32:39.018899+0000","last_fullsized":"2024-02-23T00:32:39.018899+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.683686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.683686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003145+0000","last_change":"2024-02-23T00:30:48.673703+0000","last_active":"2024-02-23T00:32:39.003145+0000","last_peered":"2024-02-23T00:32:39.003145+0000","last_clean":"2024-02-23T00:32:39.003145+0000","last_became_active":"2024-02-23T00:30:48.673452+0000","last_became_peered":"2024-02-23T00:30:48.673452+0000","last_unstale":"2024-02-23T00:32:39.003145+0000","last_undegraded":"2024-02-23T00:32:39.003145+0000","last_fullsized":"2024-02-23T00:32:39.003145+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":187,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.016777+0000","last_change":"2024-02-23T00:32:37.445055+0000","last_active":"2024-02-23T00:32:39.016777+0000","last_peered":"2024-02-23T00:32:39.016777+0000","last_clean":"2024-02-23T00:32:39.016777+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:32:39.016777+0000","last_undegraded":"2024-02-23T00:32:39.016777+0000","last_fullsized":"2024-02-23T00:32:39.016777+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.444973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.444973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":199,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004504+0000","last_change":"2024-02-23T00:30:35.121011+0000","last_active":"2024-02-23T00:32:39.004504+0000","last_peered":"2024-02-23T00:32:39.004504+0000","last_clean":"2024-02-23T00:32:39.004504+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:32:39.004504+0000","last_undegraded":"2024-02-23T00:32:39.004504+0000","last_fullsized":"2024-02-23T00:32:39.004504+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:30:35.120909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.120909+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.026343+0000","last_change":"2024-02-23T00:30:43.620922+0000","last_active":"2024-02-23T00:32:39.026343+0000","last_peered":"2024-02-23T00:32:39.026343+0000","last_clean":"2024-02-23T00:32:39.026343+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:32:39.026343+0000","last_undegraded":"2024-02-23T00:32:39.026343+0000","last_fullsized":"2024-02-23T00:32:39.026343+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:43.620849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:43.620849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.020823+0000","last_change":"2024-02-23T00:32:09.110225+0000","last_active":"2024-02-23T00:32:39.020823+0000","last_peered":"2024-02-23T00:32:39.020823+0000","last_clean":"2024-02-23T00:32:39.020823+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:32:39.020823+0000","last_undegraded":"2024-02-23T00:32:39.020823+0000","last_fullsized":"2024-02-23T00:32:39.020823+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.110138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009327+0000","last_change":"2024-02-23T00:32:15.855653+0000","last_active":"2024-02-23T00:32:39.009327+0000","last_peered":"2024-02-23T00:32:39.009327+0000","last_clean":"2024-02-23T00:32:39.009327+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:32:39.009327+0000","last_undegraded":"2024-02-23T00:32:39.009327+0000","last_fullsized":"2024-02-23T00:32:39.009327+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:32:15.855580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.026735+0000","last_change":"2024-02-23T00:30:48.716832+0000","last_active":"2024-02-23T00:32:39.026735+0000","last_peered":"2024-02-23T00:32:39.026735+0000","last_clean":"2024-02-23T00:32:39.026735+0000","last_became_active":"2024-02-23T00:30:48.716552+0000","last_became_peered":"2024-02-23T00:30:48.716552+0000","last_unstale":"2024-02-23T00:32:39.026735+0000","last_undegraded":"2024-02-23T00:32:39.026735+0000","last_fullsized":"2024-02-23T00:32:39.026735+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":182,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.024414+0000","last_change":"2024-02-23T00:31:29.064566+0000","last_active":"2024-02-23T00:32:39.024414+0000","last_peered":"2024-02-23T00:32:39.024414+0000","last_clean":"2024-02-23T00:32:39.024414+0000","last_became_active":"2024-02-23T00:31:29.064244+0000","last_became_peered":"2024-02-23T00:31:29.064244+0000","last_unstale":"2024-02-23T00:32:39.024414+0000","last_undegraded":"2024-02-23T00:32:39.024414+0000","last_fullsized":"2024-02-23T00:32:39.024414+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":186,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004069+0000","last_change":"2024-02-23T00:31:19.982059+0000","last_active":"2024-02-23T00:32:39.004069+0000","last_peered":"2024-02-23T00:32:39.004069+0000","last_clean":"2024-02-23T00:32:39.004069+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:32:39.004069+0000","last_undegraded":"2024-02-23T00:32:39.004069+0000","last_fullsized":"2024-02-23T00:32:39.004069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:31:19.981985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.981985+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":191,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.014970+0000","last_change":"2024-02-23T00:31:49.024710+0000","last_active":"2024-02-23T00:32:39.014970+0000","last_peered":"2024-02-23T00:32:39.014970+0000","last_clean":"2024-02-23T00:32:39.014970+0000","last_became_active":"2024-02-23T00:31:49.024302+0000","last_became_peered":"2024-02-23T00:31:49.024302+0000","last_unstale":"2024-02-23T00:32:39.014970+0000","last_undegraded":"2024-02-23T00:32:39.014970+0000","last_fullsized":"2024-02-23T00:32:39.014970+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.937969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":247,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.007051+0000","last_change":"2024-02-23T00:32:35.929043+0000","last_active":"2024-02-23T00:32:40.007051+0000","last_peered":"2024-02-23T00:32:40.007051+0000","last_clean":"2024-02-23T00:32:40.007051+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:32:40.007051+0000","last_undegraded":"2024-02-23T00:32:40.007051+0000","last_fullsized":"2024-02-23T00:32:40.007051+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.928977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.928977+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.008603+0000","last_change":"2024-02-23T00:30:31.065478+0000","last_active":"2024-02-23T00:32:39.008603+0000","last_peered":"2024-02-23T00:32:39.008603+0000","last_clean":"2024-02-23T00:32:39.008603+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:32:39.008603+0000","last_undegraded":"2024-02-23T00:32:39.008603+0000","last_fullsized":"2024-02-23T00:32:39.008603+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.065384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.065384+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.008696+0000","last_change":"2024-02-23T00:30:48.656349+0000","last_active":"2024-02-23T00:32:39.008696+0000","last_peered":"2024-02-23T00:32:39.008696+0000","last_clean":"2024-02-23T00:32:39.008696+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:32:39.008696+0000","last_undegraded":"2024-02-23T00:32:39.008696+0000","last_fullsized":"2024-02-23T00:32:39.008696+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":81,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.027773+0000","last_change":"2024-02-23T00:31:29.089821+0000","last_active":"2024-02-23T00:32:39.027773+0000","last_peered":"2024-02-23T00:32:39.027773+0000","last_clean":"2024-02-23T00:32:39.027773+0000","last_became_active":"2024-02-23T00:31:29.089319+0000","last_became_peered":"2024-02-23T00:31:29.089319+0000","last_unstale":"2024-02-23T00:32:39.027773+0000","last_undegraded":"2024-02-23T00:32:39.027773+0000","last_fullsized":"2024-02-23T00:32:39.027773+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:30:38.432677+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.432677+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":59,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"87'22","reported_seq":208,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.327594+0000","last_change":"2024-02-23T00:32:40.327594+0000","last_active":"2024-02-23T00:32:40.327594+0000","last_peered":"2024-02-23T00:32:40.327594+0000","last_clean":"2024-02-23T00:32:40.327594+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:32:40.327594+0000","last_undegraded":"2024-02-23T00:32:40.327594+0000","last_fullsized":"2024-02-23T00:32:40.327594+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:32:40.327519+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.327519+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":173,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.016178+0000","last_change":"2024-02-23T00:31:29.920657+0000","last_active":"2024-02-23T00:32:39.016178+0000","last_peered":"2024-02-23T00:32:39.016178+0000","last_clean":"2024-02-23T00:32:39.016178+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:32:39.016178+0000","last_undegraded":"2024-02-23T00:32:39.016178+0000","last_fullsized":"2024-02-23T00:32:39.016178+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:29.920546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:29.920546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":247,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.009686+0000","last_change":"2024-02-23T00:32:35.975973+0000","last_active":"2024-02-23T00:32:40.009686+0000","last_peered":"2024-02-23T00:32:40.009686+0000","last_clean":"2024-02-23T00:32:40.009686+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:32:40.009686+0000","last_undegraded":"2024-02-23T00:32:40.009686+0000","last_fullsized":"2024-02-23T00:32:40.009686+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.975889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.975889+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":166,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.008215+0000","last_change":"2024-02-23T00:31:21.366604+0000","last_active":"2024-02-23T00:32:39.008215+0000","last_peered":"2024-02-23T00:32:39.008215+0000","last_clean":"2024-02-23T00:32:39.008215+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:32:39.008215+0000","last_undegraded":"2024-02-23T00:32:39.008215+0000","last_fullsized":"2024-02-23T00:32:39.008215+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:31:21.366535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.366535+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.012231+0000","last_change":"2024-02-23T00:30:48.713004+0000","last_active":"2024-02-23T00:32:39.012231+0000","last_peered":"2024-02-23T00:32:39.012231+0000","last_clean":"2024-02-23T00:32:39.012231+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:32:39.012231+0000","last_undegraded":"2024-02-23T00:32:39.012231+0000","last_fullsized":"2024-02-23T00:32:39.012231+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":182,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003603+0000","last_change":"2024-02-23T00:31:29.074410+0000","last_active":"2024-02-23T00:32:39.003603+0000","last_peered":"2024-02-23T00:32:39.003603+0000","last_clean":"2024-02-23T00:32:39.003603+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:32:39.003603+0000","last_undegraded":"2024-02-23T00:32:39.003603+0000","last_fullsized":"2024-02-23T00:32:39.003603+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"86'23","reported_seq":211,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.008384+0000","last_change":"2024-02-23T00:29:56.044401+0000","last_active":"2024-02-23T00:32:39.008384+0000","last_peered":"2024-02-23T00:32:39.008384+0000","last_clean":"2024-02-23T00:32:39.008384+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:32:39.008384+0000","last_undegraded":"2024-02-23T00:32:39.008384+0000","last_fullsized":"2024-02-23T00:32:39.008384+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'23","last_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:29:56.044331+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":184,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019618+0000","last_change":"2024-02-23T00:31:38.838289+0000","last_active":"2024-02-23T00:32:39.019618+0000","last_peered":"2024-02-23T00:32:39.019618+0000","last_clean":"2024-02-23T00:32:39.019618+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:32:39.019618+0000","last_undegraded":"2024-02-23T00:32:39.019618+0000","last_fullsized":"2024-02-23T00:32:39.019618+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:31:38.838202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.838202+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009635+0000","last_change":"2024-02-23T00:31:33.381301+0000","last_active":"2024-02-23T00:32:39.009635+0000","last_peered":"2024-02-23T00:32:39.009635+0000","last_clean":"2024-02-23T00:32:39.009635+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:32:39.009635+0000","last_undegraded":"2024-02-23T00:32:39.009635+0000","last_fullsized":"2024-02-23T00:32:39.009635+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.381229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.381229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022670+0000","last_change":"2024-02-23T00:32:09.012254+0000","last_active":"2024-02-23T00:32:39.022670+0000","last_peered":"2024-02-23T00:32:39.022670+0000","last_clean":"2024-02-23T00:32:39.022670+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:32:39.022670+0000","last_undegraded":"2024-02-23T00:32:39.022670+0000","last_fullsized":"2024-02-23T00:32:39.022670+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.012161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.012161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.020950+0000","last_change":"2024-02-23T00:30:48.742570+0000","last_active":"2024-02-23T00:32:39.020950+0000","last_peered":"2024-02-23T00:32:39.020950+0000","last_clean":"2024-02-23T00:32:39.020950+0000","last_became_active":"2024-02-23T00:30:48.740636+0000","last_became_peered":"2024-02-23T00:30:48.740636+0000","last_unstale":"2024-02-23T00:32:39.020950+0000","last_undegraded":"2024-02-23T00:32:39.020950+0000","last_fullsized":"2024-02-23T00:32:39.020950+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":86,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004164+0000","last_change":"2024-02-23T00:31:39.641680+0000","last_active":"2024-02-23T00:32:39.004164+0000","last_peered":"2024-02-23T00:32:39.004164+0000","last_clean":"2024-02-23T00:32:39.004164+0000","last_became_active":"2024-02-23T00:31:29.108747+0000","last_became_peered":"2024-02-23T00:31:29.108747+0000","last_unstale":"2024-02-23T00:32:39.004164+0000","last_undegraded":"2024-02-23T00:32:39.004164+0000","last_fullsized":"2024-02-23T00:32:39.004164+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:31:39.641617+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":246,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.018527+0000","last_change":"2024-02-23T00:31:41.945984+0000","last_active":"2024-02-23T00:32:40.018527+0000","last_peered":"2024-02-23T00:32:40.018527+0000","last_clean":"2024-02-23T00:32:40.018527+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:32:40.018527+0000","last_undegraded":"2024-02-23T00:32:40.018527+0000","last_fullsized":"2024-02-23T00:32:40.018527+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":207,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006793+0000","last_change":"2024-02-23T00:31:38.038523+0000","last_active":"2024-02-23T00:32:39.006793+0000","last_peered":"2024-02-23T00:32:39.006793+0000","last_clean":"2024-02-23T00:32:39.006793+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:32:39.006793+0000","last_undegraded":"2024-02-23T00:32:39.006793+0000","last_fullsized":"2024-02-23T00:32:39.006793+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:31:38.038413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.038413+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.015017+0000","last_change":"2024-02-23T00:30:26.170570+0000","last_active":"2024-02-23T00:32:39.015017+0000","last_peered":"2024-02-23T00:32:39.015017+0000","last_clean":"2024-02-23T00:32:39.015017+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:32:39.015017+0000","last_undegraded":"2024-02-23T00:32:39.015017+0000","last_fullsized":"2024-02-23T00:32:39.015017+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:26.170482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:26.170482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":171,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.026723+0000","last_change":"2024-02-23T00:31:31.946435+0000","last_active":"2024-02-23T00:32:39.026723+0000","last_peered":"2024-02-23T00:32:39.026723+0000","last_clean":"2024-02-23T00:32:39.026723+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:32:39.026723+0000","last_undegraded":"2024-02-23T00:32:39.026723+0000","last_fullsized":"2024-02-23T00:32:39.026723+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.946366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.946366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.011117+0000","last_change":"2024-02-23T00:30:48.782732+0000","last_active":"2024-02-23T00:32:39.011117+0000","last_peered":"2024-02-23T00:32:39.011117+0000","last_clean":"2024-02-23T00:32:39.011117+0000","last_became_active":"2024-02-23T00:30:48.778293+0000","last_became_peered":"2024-02-23T00:30:48.778293+0000","last_unstale":"2024-02-23T00:32:39.011117+0000","last_undegraded":"2024-02-23T00:32:39.011117+0000","last_fullsized":"2024-02-23T00:32:39.011117+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":182,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019760+0000","last_change":"2024-02-23T00:31:29.070859+0000","last_active":"2024-02-23T00:32:39.019760+0000","last_peered":"2024-02-23T00:32:39.019760+0000","last_clean":"2024-02-23T00:32:39.019760+0000","last_became_active":"2024-02-23T00:31:29.070521+0000","last_became_peered":"2024-02-23T00:31:29.070521+0000","last_unstale":"2024-02-23T00:32:39.019760+0000","last_undegraded":"2024-02-23T00:32:39.019760+0000","last_fullsized":"2024-02-23T00:32:39.019760+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006949+0000","last_change":"2024-02-23T00:29:47.033380+0000","last_active":"2024-02-23T00:32:39.006949+0000","last_peered":"2024-02-23T00:32:39.006949+0000","last_clean":"2024-02-23T00:32:39.006949+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:32:39.006949+0000","last_undegraded":"2024-02-23T00:32:39.006949+0000","last_fullsized":"2024-02-23T00:32:39.006949+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:47.033301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:29:47.033301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":233,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.025231+0000","last_change":"2024-02-23T00:31:21.086750+0000","last_active":"2024-02-23T00:32:39.025231+0000","last_peered":"2024-02-23T00:32:39.025231+0000","last_clean":"2024-02-23T00:32:39.025231+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:32:39.025231+0000","last_undegraded":"2024-02-23T00:32:39.025231+0000","last_fullsized":"2024-02-23T00:32:39.025231+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:31:21.086610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.086610+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":173,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.011457+0000","last_change":"2024-02-23T00:30:25.593913+0000","last_active":"2024-02-23T00:32:39.011457+0000","last_peered":"2024-02-23T00:32:39.011457+0000","last_clean":"2024-02-23T00:32:39.011457+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:32:39.011457+0000","last_undegraded":"2024-02-23T00:32:39.011457+0000","last_fullsized":"2024-02-23T00:32:39.011457+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.593833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.593833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004297+0000","last_change":"2024-02-23T00:32:17.981331+0000","last_active":"2024-02-23T00:32:39.004297+0000","last_peered":"2024-02-23T00:32:39.004297+0000","last_clean":"2024-02-23T00:32:39.004297+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:32:39.004297+0000","last_undegraded":"2024-02-23T00:32:39.004297+0000","last_fullsized":"2024-02-23T00:32:39.004297+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.981262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.981262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.011358+0000","last_change":"2024-02-23T00:30:48.776316+0000","last_active":"2024-02-23T00:32:39.011358+0000","last_peered":"2024-02-23T00:32:39.011358+0000","last_clean":"2024-02-23T00:32:39.011358+0000","last_became_active":"2024-02-23T00:30:48.769602+0000","last_became_peered":"2024-02-23T00:30:48.769602+0000","last_unstale":"2024-02-23T00:32:39.011358+0000","last_undegraded":"2024-02-23T00:32:39.011358+0000","last_fullsized":"2024-02-23T00:32:39.011358+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":436,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.020577+0000","last_change":"2024-02-23T00:31:29.095695+0000","last_active":"2024-02-23T00:32:39.020577+0000","last_peered":"2024-02-23T00:32:39.020577+0000","last_clean":"2024-02-23T00:32:39.020577+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:32:39.020577+0000","last_undegraded":"2024-02-23T00:32:39.020577+0000","last_fullsized":"2024-02-23T00:32:39.020577+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:30:38.432677+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.432677+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":61,"num_read_kb":59,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.016627+0000","last_change":"2024-02-23T00:30:27.151423+0000","last_active":"2024-02-23T00:32:39.016627+0000","last_peered":"2024-02-23T00:32:39.016627+0000","last_clean":"2024-02-23T00:32:39.016627+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:32:39.016627+0000","last_undegraded":"2024-02-23T00:32:39.016627+0000","last_fullsized":"2024-02-23T00:32:39.016627+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:27.151342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:27.151342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":227,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.001768+0000","last_change":"2024-02-23T00:31:29.650718+0000","last_active":"2024-02-23T00:32:39.001768+0000","last_peered":"2024-02-23T00:32:39.001768+0000","last_clean":"2024-02-23T00:32:39.001768+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:32:39.001768+0000","last_undegraded":"2024-02-23T00:32:39.001768+0000","last_fullsized":"2024-02-23T00:32:39.001768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:29.650602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:29.650602+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":177,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022384+0000","last_change":"2024-02-23T00:31:40.847863+0000","last_active":"2024-02-23T00:32:39.022384+0000","last_peered":"2024-02-23T00:32:39.022384+0000","last_clean":"2024-02-23T00:32:39.022384+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:32:39.022384+0000","last_undegraded":"2024-02-23T00:32:39.022384+0000","last_fullsized":"2024-02-23T00:32:39.022384+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.847784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.847784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013608+0000","last_change":"2024-02-23T00:30:46.579484+0000","last_active":"2024-02-23T00:32:39.013608+0000","last_peered":"2024-02-23T00:32:39.013608+0000","last_clean":"2024-02-23T00:32:39.013608+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:32:39.013608+0000","last_undegraded":"2024-02-23T00:32:39.013608+0000","last_fullsized":"2024-02-23T00:32:39.013608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.579417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.579417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013680+0000","last_change":"2024-02-23T00:30:48.707450+0000","last_active":"2024-02-23T00:32:39.013680+0000","last_peered":"2024-02-23T00:32:39.013680+0000","last_clean":"2024-02-23T00:32:39.013680+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:32:39.013680+0000","last_undegraded":"2024-02-23T00:32:39.013680+0000","last_fullsized":"2024-02-23T00:32:39.013680+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009798+0000","last_change":"2024-02-23T00:32:12.386351+0000","last_active":"2024-02-23T00:32:39.009798+0000","last_peered":"2024-02-23T00:32:39.009798+0000","last_clean":"2024-02-23T00:32:39.009798+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:32:39.009798+0000","last_undegraded":"2024-02-23T00:32:39.009798+0000","last_fullsized":"2024-02-23T00:32:39.009798+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":194,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003907+0000","last_change":"2024-02-23T00:31:19.393840+0000","last_active":"2024-02-23T00:32:39.003907+0000","last_peered":"2024-02-23T00:32:39.003907+0000","last_clean":"2024-02-23T00:32:39.003907+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:32:39.003907+0000","last_undegraded":"2024-02-23T00:32:39.003907+0000","last_fullsized":"2024-02-23T00:32:39.003907+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:31:19.393720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.393720+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019038+0000","last_change":"2024-02-23T00:31:19.760264+0000","last_active":"2024-02-23T00:32:39.019038+0000","last_peered":"2024-02-23T00:32:39.019038+0000","last_clean":"2024-02-23T00:32:39.019038+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:32:39.019038+0000","last_undegraded":"2024-02-23T00:32:39.019038+0000","last_fullsized":"2024-02-23T00:32:39.019038+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:19.760201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.760201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":187,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.001247+0000","last_change":"2024-02-23T00:31:38.653143+0000","last_active":"2024-02-23T00:32:39.001247+0000","last_peered":"2024-02-23T00:32:39.001247+0000","last_clean":"2024-02-23T00:32:39.001247+0000","last_became_active":"2024-02-23T00:31:27.053776+0000","last_became_peered":"2024-02-23T00:31:27.053776+0000","last_unstale":"2024-02-23T00:32:39.001247+0000","last_undegraded":"2024-02-23T00:32:39.001247+0000","last_fullsized":"2024-02-23T00:32:39.001247+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.653078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.653078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.020867+0000","last_change":"2024-02-23T00:31:21.959719+0000","last_active":"2024-02-23T00:32:39.020867+0000","last_peered":"2024-02-23T00:32:39.020867+0000","last_clean":"2024-02-23T00:32:39.020867+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:32:39.020867+0000","last_undegraded":"2024-02-23T00:32:39.020867+0000","last_fullsized":"2024-02-23T00:32:39.020867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.959597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.959597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004359+0000","last_change":"2024-02-23T00:30:48.711294+0000","last_active":"2024-02-23T00:32:39.004359+0000","last_peered":"2024-02-23T00:32:39.004359+0000","last_clean":"2024-02-23T00:32:39.004359+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:32:39.004359+0000","last_undegraded":"2024-02-23T00:32:39.004359+0000","last_fullsized":"2024-02-23T00:32:39.004359+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":173,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.020891+0000","last_change":"2024-02-23T00:32:17.590019+0000","last_active":"2024-02-23T00:32:39.020891+0000","last_peered":"2024-02-23T00:32:39.020891+0000","last_clean":"2024-02-23T00:32:39.020891+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:32:39.020891+0000","last_undegraded":"2024-02-23T00:32:39.020891+0000","last_fullsized":"2024-02-23T00:32:39.020891+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.589952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.589952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":198,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013578+0000","last_change":"2024-02-23T00:31:40.030848+0000","last_active":"2024-02-23T00:32:39.013578+0000","last_peered":"2024-02-23T00:32:39.013578+0000","last_clean":"2024-02-23T00:32:39.013578+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:32:39.013578+0000","last_undegraded":"2024-02-23T00:32:39.013578+0000","last_fullsized":"2024-02-23T00:32:39.013578+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:31:40.030776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.030776+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":180,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019530+0000","last_change":"2024-02-23T00:31:21.828852+0000","last_active":"2024-02-23T00:32:39.019530+0000","last_peered":"2024-02-23T00:32:39.019530+0000","last_clean":"2024-02-23T00:32:39.019530+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:32:39.019530+0000","last_undegraded":"2024-02-23T00:32:39.019530+0000","last_fullsized":"2024-02-23T00:32:39.019530+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.828719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.828719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":171,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007904+0000","last_change":"2024-02-23T00:30:40.144284+0000","last_active":"2024-02-23T00:32:39.007904+0000","last_peered":"2024-02-23T00:32:39.007904+0000","last_clean":"2024-02-23T00:32:39.007904+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:32:39.007904+0000","last_undegraded":"2024-02-23T00:32:39.007904+0000","last_fullsized":"2024-02-23T00:32:39.007904+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.144220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.144220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":182,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.011434+0000","last_change":"2024-02-23T00:31:27.145186+0000","last_active":"2024-02-23T00:32:39.011434+0000","last_peered":"2024-02-23T00:32:39.011434+0000","last_clean":"2024-02-23T00:32:39.011434+0000","last_became_active":"2024-02-23T00:31:27.144976+0000","last_became_peered":"2024-02-23T00:31:27.144976+0000","last_unstale":"2024-02-23T00:32:39.011434+0000","last_undegraded":"2024-02-23T00:32:39.011434+0000","last_fullsized":"2024-02-23T00:32:39.011434+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:50.052469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:29:50.052469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":165,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009909+0000","last_change":"2024-02-23T00:30:45.024137+0000","last_active":"2024-02-23T00:32:39.009909+0000","last_peered":"2024-02-23T00:32:39.009909+0000","last_clean":"2024-02-23T00:32:39.009909+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:32:39.009909+0000","last_undegraded":"2024-02-23T00:32:39.009909+0000","last_fullsized":"2024-02-23T00:32:39.009909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.024058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.024058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004783+0000","last_change":"2024-02-23T00:30:48.664253+0000","last_active":"2024-02-23T00:32:39.004783+0000","last_peered":"2024-02-23T00:32:39.004783+0000","last_clean":"2024-02-23T00:32:39.004783+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:32:39.004783+0000","last_undegraded":"2024-02-23T00:32:39.004783+0000","last_fullsized":"2024-02-23T00:32:39.004783+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":174,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007448+0000","last_change":"2024-02-23T00:32:13.878058+0000","last_active":"2024-02-23T00:32:39.007448+0000","last_peered":"2024-02-23T00:32:39.007448+0000","last_clean":"2024-02-23T00:32:39.007448+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:32:39.007448+0000","last_undegraded":"2024-02-23T00:32:39.007448+0000","last_fullsized":"2024-02-23T00:32:39.007448+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:13.877940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.877940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":218,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013711+0000","last_change":"2024-02-23T00:32:13.104236+0000","last_active":"2024-02-23T00:32:39.013711+0000","last_peered":"2024-02-23T00:32:39.013711+0000","last_clean":"2024-02-23T00:32:39.013711+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:32:39.013711+0000","last_undegraded":"2024-02-23T00:32:39.013711+0000","last_fullsized":"2024-02-23T00:32:39.013711+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:32:13.104170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.104170+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007573+0000","last_change":"2024-02-23T00:31:41.651826+0000","last_active":"2024-02-23T00:32:39.007573+0000","last_peered":"2024-02-23T00:32:39.007573+0000","last_clean":"2024-02-23T00:32:39.007573+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:32:39.007573+0000","last_undegraded":"2024-02-23T00:32:39.007573+0000","last_fullsized":"2024-02-23T00:32:39.007573+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:41.651760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.651760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":429,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013045+0000","last_change":"2024-02-23T00:31:27.040910+0000","last_active":"2024-02-23T00:32:39.013045+0000","last_peered":"2024-02-23T00:32:39.013045+0000","last_clean":"2024-02-23T00:32:39.013045+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:32:39.013045+0000","last_undegraded":"2024-02-23T00:32:39.013045+0000","last_fullsized":"2024-02-23T00:32:39.013045+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:30:38.432677+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.432677+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":59,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":171,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022338+0000","last_change":"2024-02-23T00:32:14.650905+0000","last_active":"2024-02-23T00:32:39.022338+0000","last_peered":"2024-02-23T00:32:39.022338+0000","last_clean":"2024-02-23T00:32:39.022338+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:32:39.022338+0000","last_undegraded":"2024-02-23T00:32:39.022338+0000","last_fullsized":"2024-02-23T00:32:39.022338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.650827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.650827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006158+0000","last_change":"2024-02-23T00:30:48.692888+0000","last_active":"2024-02-23T00:32:39.006158+0000","last_peered":"2024-02-23T00:32:39.006158+0000","last_clean":"2024-02-23T00:32:39.006158+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:32:39.006158+0000","last_undegraded":"2024-02-23T00:32:39.006158+0000","last_fullsized":"2024-02-23T00:32:39.006158+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":182,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003323+0000","last_change":"2024-02-23T00:31:29.065838+0000","last_active":"2024-02-23T00:32:39.003323+0000","last_peered":"2024-02-23T00:32:39.003323+0000","last_clean":"2024-02-23T00:32:39.003323+0000","last_became_active":"2024-02-23T00:31:29.065548+0000","last_became_peered":"2024-02-23T00:31:29.065548+0000","last_unstale":"2024-02-23T00:32:39.003323+0000","last_undegraded":"2024-02-23T00:32:39.003323+0000","last_fullsized":"2024-02-23T00:32:39.003323+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:35.845382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.845382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":176,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.005262+0000","last_change":"2024-02-23T00:31:26.705106+0000","last_active":"2024-02-23T00:32:39.005262+0000","last_peered":"2024-02-23T00:32:39.005262+0000","last_clean":"2024-02-23T00:32:39.005262+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:32:39.005262+0000","last_undegraded":"2024-02-23T00:32:39.005262+0000","last_fullsized":"2024-02-23T00:32:39.005262+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:26.704988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:26.704988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":299,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.011130+0000","last_change":"2024-02-23T00:30:29.527577+0000","last_active":"2024-02-23T00:32:40.011130+0000","last_peered":"2024-02-23T00:32:40.011130+0000","last_clean":"2024-02-23T00:32:40.011130+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:32:40.011130+0000","last_undegraded":"2024-02-23T00:32:40.011130+0000","last_fullsized":"2024-02-23T00:32:40.011130+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:30:29.527466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.527466+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":177,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006904+0000","last_change":"2024-02-23T00:30:47.090981+0000","last_active":"2024-02-23T00:32:39.006904+0000","last_peered":"2024-02-23T00:32:39.006904+0000","last_clean":"2024-02-23T00:32:39.006904+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:32:39.006904+0000","last_undegraded":"2024-02-23T00:32:39.006904+0000","last_fullsized":"2024-02-23T00:32:39.006904+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:30:47.090896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.090896+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.018063+0000","last_change":"2024-02-23T00:30:22.860665+0000","last_active":"2024-02-23T00:32:39.018063+0000","last_peered":"2024-02-23T00:32:39.018063+0000","last_clean":"2024-02-23T00:32:39.018063+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:32:39.018063+0000","last_undegraded":"2024-02-23T00:32:39.018063+0000","last_fullsized":"2024-02-23T00:32:39.018063+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:22.860565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.860565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.016221+0000","last_change":"2024-02-23T00:30:48.701388+0000","last_active":"2024-02-23T00:32:39.016221+0000","last_peered":"2024-02-23T00:32:39.016221+0000","last_clean":"2024-02-23T00:32:39.016221+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:32:39.016221+0000","last_undegraded":"2024-02-23T00:32:39.016221+0000","last_fullsized":"2024-02-23T00:32:39.016221+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":86,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007237+0000","last_change":"2024-02-23T00:32:36.447499+0000","last_active":"2024-02-23T00:32:39.007237+0000","last_peered":"2024-02-23T00:32:39.007237+0000","last_clean":"2024-02-23T00:32:39.007237+0000","last_became_active":"2024-02-23T00:31:29.059516+0000","last_became_peered":"2024-02-23T00:31:29.059516+0000","last_unstale":"2024-02-23T00:32:39.007237+0000","last_undegraded":"2024-02-23T00:32:39.007237+0000","last_fullsized":"2024-02-23T00:32:39.007237+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.447439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.447439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":250,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.005950+0000","last_change":"2024-02-23T00:32:18.606981+0000","last_active":"2024-02-23T00:32:40.005950+0000","last_peered":"2024-02-23T00:32:40.005950+0000","last_clean":"2024-02-23T00:32:40.005950+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:32:40.005950+0000","last_undegraded":"2024-02-23T00:32:40.005950+0000","last_fullsized":"2024-02-23T00:32:40.005950+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:18.606911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:18.606911+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":201,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.015205+0000","last_change":"2024-02-23T00:30:31.441530+0000","last_active":"2024-02-23T00:32:39.015205+0000","last_peered":"2024-02-23T00:32:39.015205+0000","last_clean":"2024-02-23T00:32:39.015205+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:32:39.015205+0000","last_undegraded":"2024-02-23T00:32:39.015205+0000","last_fullsized":"2024-02-23T00:32:39.015205+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:30:31.441418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.441418+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.021306+0000","last_change":"2024-02-23T00:30:44.543241+0000","last_active":"2024-02-23T00:32:39.021306+0000","last_peered":"2024-02-23T00:32:39.021306+0000","last_clean":"2024-02-23T00:32:39.021306+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:32:39.021306+0000","last_undegraded":"2024-02-23T00:32:39.021306+0000","last_fullsized":"2024-02-23T00:32:39.021306+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:44.543161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:30:44.543161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.002175+0000","last_change":"2024-02-23T00:32:08.596816+0000","last_active":"2024-02-23T00:32:39.002175+0000","last_peered":"2024-02-23T00:32:39.002175+0000","last_clean":"2024-02-23T00:32:39.002175+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:32:39.002175+0000","last_undegraded":"2024-02-23T00:32:39.002175+0000","last_fullsized":"2024-02-23T00:32:39.002175+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:08.596734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:08.596734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003568+0000","last_change":"2024-02-23T00:30:48.616685+0000","last_active":"2024-02-23T00:32:39.003568+0000","last_peered":"2024-02-23T00:32:39.003568+0000","last_clean":"2024-02-23T00:32:39.003568+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:32:39.003568+0000","last_undegraded":"2024-02-23T00:32:39.003568+0000","last_fullsized":"2024-02-23T00:32:39.003568+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006195+0000","last_change":"2024-02-23T00:30:28.676431+0000","last_active":"2024-02-23T00:32:39.006195+0000","last_peered":"2024-02-23T00:32:39.006195+0000","last_clean":"2024-02-23T00:32:39.006195+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:32:39.006195+0000","last_undegraded":"2024-02-23T00:32:39.006195+0000","last_fullsized":"2024-02-23T00:32:39.006195+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:28.676326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:28.676326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":250,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006884+0000","last_change":"2024-02-23T00:31:41.036909+0000","last_active":"2024-02-23T00:32:39.006884+0000","last_peered":"2024-02-23T00:32:39.006884+0000","last_clean":"2024-02-23T00:32:39.006884+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:32:39.006884+0000","last_undegraded":"2024-02-23T00:32:39.006884+0000","last_fullsized":"2024-02-23T00:32:39.006884+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.036837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.036837+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":186,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004962+0000","last_change":"2024-02-23T00:30:47.142583+0000","last_active":"2024-02-23T00:32:39.004962+0000","last_peered":"2024-02-23T00:32:39.004962+0000","last_clean":"2024-02-23T00:32:39.004962+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:32:39.004962+0000","last_undegraded":"2024-02-23T00:32:39.004962+0000","last_fullsized":"2024-02-23T00:32:39.004962+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:30:47.142508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.142508+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.026783+0000","last_change":"2024-02-23T00:30:34.861845+0000","last_active":"2024-02-23T00:32:39.026783+0000","last_peered":"2024-02-23T00:32:39.026783+0000","last_clean":"2024-02-23T00:32:39.026783+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:32:39.026783+0000","last_undegraded":"2024-02-23T00:32:39.026783+0000","last_fullsized":"2024-02-23T00:32:39.026783+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:34.861741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:34.861741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007555+0000","last_change":"2024-02-23T00:30:48.674743+0000","last_active":"2024-02-23T00:32:39.007555+0000","last_peered":"2024-02-23T00:32:39.007555+0000","last_clean":"2024-02-23T00:32:39.007555+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:32:39.007555+0000","last_undegraded":"2024-02-23T00:32:39.007555+0000","last_fullsized":"2024-02-23T00:32:39.007555+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":217,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003926+0000","last_change":"2024-02-23T00:30:44.885410+0000","last_active":"2024-02-23T00:32:39.003926+0000","last_peered":"2024-02-23T00:32:39.003926+0000","last_clean":"2024-02-23T00:32:39.003926+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:32:39.003926+0000","last_undegraded":"2024-02-23T00:32:39.003926+0000","last_fullsized":"2024-02-23T00:32:39.003926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:30:44.885334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:44.885334+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006105+0000","last_change":"2024-02-23T00:32:11.912580+0000","last_active":"2024-02-23T00:32:39.006105+0000","last_peered":"2024-02-23T00:32:39.006105+0000","last_clean":"2024-02-23T00:32:39.006105+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:32:39.006105+0000","last_undegraded":"2024-02-23T00:32:39.006105+0000","last_fullsized":"2024-02-23T00:32:39.006105+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:11.912504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.912504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019564+0000","last_change":"2024-02-23T00:30:36.894317+0000","last_active":"2024-02-23T00:32:39.019564+0000","last_peered":"2024-02-23T00:32:39.019564+0000","last_clean":"2024-02-23T00:32:39.019564+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:32:39.019564+0000","last_undegraded":"2024-02-23T00:32:39.019564+0000","last_fullsized":"2024-02-23T00:32:39.019564+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:36.894220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:36.894220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009600+0000","last_change":"2024-02-23T00:32:16.828560+0000","last_active":"2024-02-23T00:32:39.009600+0000","last_peered":"2024-02-23T00:32:39.009600+0000","last_clean":"2024-02-23T00:32:39.009600+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:32:39.009600+0000","last_undegraded":"2024-02-23T00:32:39.009600+0000","last_fullsized":"2024-02-23T00:32:39.009600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_clean_scrub_stamp":"2024-02-23T00:32:16.828490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"148'9","reported_seq":138,"reported_epoch":209,"state":"peering","last_fresh":"2024-02-23T00:32:40.008496+0000","last_change":"2024-02-23T00:32:40.008731+0000","last_active":"2024-02-23T00:32:38.000512+0000","last_peered":"2024-02-23T00:32:39.039197+0000","last_clean":"2024-02-23T00:32:39.039197+0000","last_became_active":"2024-02-23T00:30:48.725566+0000","last_became_peered":"2024-02-23T00:32:39.032886+0000","last_unstale":"2024-02-23T00:32:40.008496+0000","last_undegraded":"2024-02-23T00:32:40.008496+0000","last_fullsized":"2024-02-23T00:32:40.008496+0000","mapping_epoch":96,"log_start":"148'9","ondisk_log_start":"148'9","created":96,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.023250+0000","last_change":"2024-02-23T00:30:48.691960+0000","last_active":"2024-02-23T00:32:39.023250+0000","last_peered":"2024-02-23T00:32:39.023250+0000","last_clean":"2024-02-23T00:32:39.023250+0000","last_became_active":"2024-02-23T00:30:48.691713+0000","last_became_peered":"2024-02-23T00:30:48.691713+0000","last_unstale":"2024-02-23T00:32:39.023250+0000","last_undegraded":"2024-02-23T00:32:39.023250+0000","last_fullsized":"2024-02-23T00:32:39.023250+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":214,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019382+0000","last_change":"2024-02-23T00:30:24.802266+0000","last_active":"2024-02-23T00:32:39.019382+0000","last_peered":"2024-02-23T00:32:39.019382+0000","last_clean":"2024-02-23T00:32:39.019382+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:32:39.019382+0000","last_undegraded":"2024-02-23T00:32:39.019382+0000","last_fullsized":"2024-02-23T00:32:39.019382+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:30:24.802134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:24.802134+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013951+0000","last_change":"2024-02-23T00:32:14.243873+0000","last_active":"2024-02-23T00:32:39.013951+0000","last_peered":"2024-02-23T00:32:39.013951+0000","last_clean":"2024-02-23T00:32:39.013951+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:32:39.013951+0000","last_undegraded":"2024-02-23T00:32:39.013951+0000","last_fullsized":"2024-02-23T00:32:39.013951+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.243786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.243786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009260+0000","last_change":"2024-02-23T00:32:07.275896+0000","last_active":"2024-02-23T00:32:39.009260+0000","last_peered":"2024-02-23T00:32:39.009260+0000","last_clean":"2024-02-23T00:32:39.009260+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:32:39.009260+0000","last_undegraded":"2024-02-23T00:32:39.009260+0000","last_fullsized":"2024-02-23T00:32:39.009260+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:07.275816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:07.275816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":171,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022180+0000","last_change":"2024-02-23T00:32:15.651736+0000","last_active":"2024-02-23T00:32:39.022180+0000","last_peered":"2024-02-23T00:32:39.022180+0000","last_clean":"2024-02-23T00:32:39.022180+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:32:39.022180+0000","last_undegraded":"2024-02-23T00:32:39.022180+0000","last_fullsized":"2024-02-23T00:32:39.022180+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:15.651651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:32:15.651651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"148'7","reported_seq":128,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.027686+0000","last_change":"2024-02-23T00:30:48.712709+0000","last_active":"2024-02-23T00:32:39.027686+0000","last_peered":"2024-02-23T00:32:39.027686+0000","last_clean":"2024-02-23T00:32:39.027686+0000","last_became_active":"2024-02-23T00:30:48.711845+0000","last_became_peered":"2024-02-23T00:30:48.711845+0000","last_unstale":"2024-02-23T00:32:39.027686+0000","last_undegraded":"2024-02-23T00:32:39.027686+0000","last_fullsized":"2024-02-23T00:32:39.027686+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013707+0000","last_change":"2024-02-23T00:30:48.703519+0000","last_active":"2024-02-23T00:32:39.013707+0000","last_peered":"2024-02-23T00:32:39.013707+0000","last_clean":"2024-02-23T00:32:39.013707+0000","last_became_active":"2024-02-23T00:30:48.703232+0000","last_became_peered":"2024-02-23T00:30:48.703232+0000","last_unstale":"2024-02-23T00:32:39.013707+0000","last_undegraded":"2024-02-23T00:32:39.013707+0000","last_fullsized":"2024-02-23T00:32:39.013707+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":216,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022795+0000","last_change":"2024-02-23T00:30:37.899931+0000","last_active":"2024-02-23T00:32:39.022795+0000","last_peered":"2024-02-23T00:32:39.022795+0000","last_clean":"2024-02-23T00:32:39.022795+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:32:39.022795+0000","last_undegraded":"2024-02-23T00:32:39.022795+0000","last_fullsized":"2024-02-23T00:32:39.022795+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:30:37.899845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.899845+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":176,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006812+0000","last_change":"2024-02-23T00:32:10.319807+0000","last_active":"2024-02-23T00:32:39.006812+0000","last_peered":"2024-02-23T00:32:39.006812+0000","last_clean":"2024-02-23T00:32:39.006812+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:32:39.006812+0000","last_undegraded":"2024-02-23T00:32:39.006812+0000","last_fullsized":"2024-02-23T00:32:39.006812+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.319744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.319744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":241,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.015489+0000","last_change":"2024-02-23T00:30:30.131030+0000","last_active":"2024-02-23T00:32:40.015489+0000","last_peered":"2024-02-23T00:32:40.015489+0000","last_clean":"2024-02-23T00:32:40.015489+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:32:40.015489+0000","last_undegraded":"2024-02-23T00:32:40.015489+0000","last_fullsized":"2024-02-23T00:32:40.015489+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:30:30.130950+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:28:38.956431+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.130950+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003840+0000","last_change":"2024-02-23T00:32:07.573617+0000","last_active":"2024-02-23T00:32:39.003840+0000","last_peered":"2024-02-23T00:32:39.003840+0000","last_clean":"2024-02-23T00:32:39.003840+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:32:39.003840+0000","last_undegraded":"2024-02-23T00:32:39.003840+0000","last_fullsized":"2024-02-23T00:32:39.003840+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:07.573555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:37.889336+0000","last_clean_scrub_stamp":"2024-02-23T00:32:07.573555+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.3","version":"148'11","reported_seq":146,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.008011+0000","last_change":"2024-02-23T00:32:33.010845+0000","last_active":"2024-02-23T00:32:39.008011+0000","last_peered":"2024-02-23T00:32:39.008011+0000","last_clean":"2024-02-23T00:32:39.008011+0000","last_became_active":"2024-02-23T00:32:33.010479+0000","last_became_peered":"2024-02-23T00:32:33.010479+0000","last_unstale":"2024-02-23T00:32:39.008011+0000","last_undegraded":"2024-02-23T00:32:39.008011+0000","last_fullsized":"2024-02-23T00:32:39.008011+0000","mapping_epoch":96,"log_start":"148'11","ondisk_log_start":"148'11","created":96,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009747+0000","last_change":"2024-02-23T00:30:48.688217+0000","last_active":"2024-02-23T00:32:39.009747+0000","last_peered":"2024-02-23T00:32:39.009747+0000","last_clean":"2024-02-23T00:32:39.009747+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:32:39.009747+0000","last_undegraded":"2024-02-23T00:32:39.009747+0000","last_fullsized":"2024-02-23T00:32:39.009747+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"144'20","reported_seq":205,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007914+0000","last_change":"2024-02-23T00:31:35.423292+0000","last_active":"2024-02-23T00:32:39.007914+0000","last_peered":"2024-02-23T00:32:39.007914+0000","last_clean":"2024-02-23T00:32:39.007914+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:32:39.007914+0000","last_undegraded":"2024-02-23T00:32:39.007914+0000","last_fullsized":"2024-02-23T00:32:39.007914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.423227+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.001851+0000","last_change":"2024-02-23T00:31:35.705318+0000","last_active":"2024-02-23T00:32:39.001851+0000","last_peered":"2024-02-23T00:32:39.001851+0000","last_clean":"2024-02-23T00:32:39.001851+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:32:39.001851+0000","last_undegraded":"2024-02-23T00:32:39.001851+0000","last_fullsized":"2024-02-23T00:32:39.001851+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:35.705240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.705240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'9","reported_seq":243,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.016526+0000","last_change":"2024-02-23T00:29:38.098504+0000","last_active":"2024-02-23T00:32:40.016526+0000","last_peered":"2024-02-23T00:32:40.016526+0000","last_clean":"2024-02-23T00:32:40.016526+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:32:40.016526+0000","last_undegraded":"2024-02-23T00:32:40.016526+0000","last_fullsized":"2024-02-23T00:32:40.016526+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.098442+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006243+0000","last_change":"2024-02-23T00:30:29.051519+0000","last_active":"2024-02-23T00:32:39.006243+0000","last_peered":"2024-02-23T00:32:39.006243+0000","last_clean":"2024-02-23T00:32:39.006243+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:32:39.006243+0000","last_undegraded":"2024-02-23T00:32:39.006243+0000","last_fullsized":"2024-02-23T00:32:39.006243+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:29.051438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.051438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":139,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022288+0000","last_change":"2024-02-23T00:32:37.017394+0000","last_active":"2024-02-23T00:32:39.022288+0000","last_peered":"2024-02-23T00:32:39.022288+0000","last_clean":"2024-02-23T00:32:39.022288+0000","last_became_active":"2024-02-23T00:32:37.017015+0000","last_became_peered":"2024-02-23T00:32:37.017015+0000","last_unstale":"2024-02-23T00:32:39.022288+0000","last_undegraded":"2024-02-23T00:32:39.022288+0000","last_fullsized":"2024-02-23T00:32:39.022288+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019689+0000","last_change":"2024-02-23T00:30:48.717524+0000","last_active":"2024-02-23T00:32:39.019689+0000","last_peered":"2024-02-23T00:32:39.019689+0000","last_clean":"2024-02-23T00:32:39.019689+0000","last_became_active":"2024-02-23T00:30:48.716104+0000","last_became_peered":"2024-02-23T00:30:48.716104+0000","last_unstale":"2024-02-23T00:32:39.019689+0000","last_undegraded":"2024-02-23T00:32:39.019689+0000","last_fullsized":"2024-02-23T00:32:39.019689+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":187,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.015052+0000","last_change":"2024-02-23T00:30:23.600452+0000","last_active":"2024-02-23T00:32:39.015052+0000","last_peered":"2024-02-23T00:32:39.015052+0000","last_clean":"2024-02-23T00:32:39.015052+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:32:39.015052+0000","last_undegraded":"2024-02-23T00:32:39.015052+0000","last_fullsized":"2024-02-23T00:32:39.015052+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:30:23.600311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.600311+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.014942+0000","last_change":"2024-02-23T00:32:12.234058+0000","last_active":"2024-02-23T00:32:39.014942+0000","last_peered":"2024-02-23T00:32:39.014942+0000","last_clean":"2024-02-23T00:32:39.014942+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:32:39.014942+0000","last_undegraded":"2024-02-23T00:32:39.014942+0000","last_fullsized":"2024-02-23T00:32:39.014942+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.233960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.233960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":174,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.026636+0000","last_change":"2024-02-23T00:32:09.978443+0000","last_active":"2024-02-23T00:32:39.026636+0000","last_peered":"2024-02-23T00:32:39.026636+0000","last_clean":"2024-02-23T00:32:39.026636+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:32:39.026636+0000","last_undegraded":"2024-02-23T00:32:39.026636+0000","last_fullsized":"2024-02-23T00:32:39.026636+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.978377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.978377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.002032+0000","last_change":"2024-02-23T00:31:41.614428+0000","last_active":"2024-02-23T00:32:39.002032+0000","last_peered":"2024-02-23T00:32:39.002032+0000","last_clean":"2024-02-23T00:32:39.002032+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:32:39.002032+0000","last_undegraded":"2024-02-23T00:32:39.002032+0000","last_fullsized":"2024-02-23T00:32:39.002032+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:41.614319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.614319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":139,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004642+0000","last_change":"2024-02-23T00:32:25.011453+0000","last_active":"2024-02-23T00:32:39.004642+0000","last_peered":"2024-02-23T00:32:39.004642+0000","last_clean":"2024-02-23T00:32:39.004642+0000","last_became_active":"2024-02-23T00:32:25.011119+0000","last_became_peered":"2024-02-23T00:32:25.011119+0000","last_unstale":"2024-02-23T00:32:39.004642+0000","last_undegraded":"2024-02-23T00:32:39.004642+0000","last_fullsized":"2024-02-23T00:32:39.004642+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003445+0000","last_change":"2024-02-23T00:30:48.668969+0000","last_active":"2024-02-23T00:32:39.003445+0000","last_peered":"2024-02-23T00:32:39.003445+0000","last_clean":"2024-02-23T00:32:39.003445+0000","last_became_active":"2024-02-23T00:30:48.666975+0000","last_became_peered":"2024-02-23T00:30:48.666975+0000","last_unstale":"2024-02-23T00:32:39.003445+0000","last_undegraded":"2024-02-23T00:32:39.003445+0000","last_fullsized":"2024-02-23T00:32:39.003445+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":211,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004504+0000","last_change":"2024-02-23T00:30:41.077799+0000","last_active":"2024-02-23T00:32:39.004504+0000","last_peered":"2024-02-23T00:32:39.004504+0000","last_clean":"2024-02-23T00:32:39.004504+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:32:39.004504+0000","last_undegraded":"2024-02-23T00:32:39.004504+0000","last_fullsized":"2024-02-23T00:32:39.004504+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:30:41.077681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.077681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":173,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.012534+0000","last_change":"2024-02-23T00:31:32.874749+0000","last_active":"2024-02-23T00:32:39.012534+0000","last_peered":"2024-02-23T00:32:39.012534+0000","last_clean":"2024-02-23T00:32:39.012534+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:32:39.012534+0000","last_undegraded":"2024-02-23T00:32:39.012534+0000","last_fullsized":"2024-02-23T00:32:39.012534+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:32.874678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:31:32.874678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.015126+0000","last_change":"2024-02-23T00:30:30.486006+0000","last_active":"2024-02-23T00:32:39.015126+0000","last_peered":"2024-02-23T00:32:39.015126+0000","last_clean":"2024-02-23T00:32:39.015126+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:32:39.015126+0000","last_undegraded":"2024-02-23T00:32:39.015126+0000","last_fullsized":"2024-02-23T00:32:39.015126+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:30.485900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.485900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.024525+0000","last_change":"2024-02-23T00:30:47.843046+0000","last_active":"2024-02-23T00:32:39.024525+0000","last_peered":"2024-02-23T00:32:39.024525+0000","last_clean":"2024-02-23T00:32:39.024525+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:32:39.024525+0000","last_undegraded":"2024-02-23T00:32:39.024525+0000","last_fullsized":"2024-02-23T00:32:39.024525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.842931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.4","version":"98'2","reported_seq":139,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.020910+0000","last_change":"2024-02-23T00:32:29.034101+0000","last_active":"2024-02-23T00:32:39.020910+0000","last_peered":"2024-02-23T00:32:39.020910+0000","last_clean":"2024-02-23T00:32:39.020910+0000","last_became_active":"2024-02-23T00:32:29.033750+0000","last_became_peered":"2024-02-23T00:32:29.033750+0000","last_unstale":"2024-02-23T00:32:39.020910+0000","last_undegraded":"2024-02-23T00:32:39.020910+0000","last_fullsized":"2024-02-23T00:32:39.020910+0000","mapping_epoch":96,"log_start":"98'2","ondisk_log_start":"98'2","created":96,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":133,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.014298+0000","last_change":"2024-02-23T00:30:48.705018+0000","last_active":"2024-02-23T00:32:39.014298+0000","last_peered":"2024-02-23T00:32:39.014298+0000","last_clean":"2024-02-23T00:32:39.014298+0000","last_became_active":"2024-02-23T00:30:48.703281+0000","last_became_peered":"2024-02-23T00:30:48.703281+0000","last_unstale":"2024-02-23T00:32:39.014298+0000","last_undegraded":"2024-02-23T00:32:39.014298+0000","last_fullsized":"2024-02-23T00:32:39.014298+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":184,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.025060+0000","last_change":"2024-02-23T00:32:11.989425+0000","last_active":"2024-02-23T00:32:39.025060+0000","last_peered":"2024-02-23T00:32:39.025060+0000","last_clean":"2024-02-23T00:32:39.025060+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:32:39.025060+0000","last_undegraded":"2024-02-23T00:32:39.025060+0000","last_fullsized":"2024-02-23T00:32:39.025060+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:32:11.989333+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.989333+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":176,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.343941+0000","last_change":"2024-02-23T00:32:40.343941+0000","last_active":"2024-02-23T00:32:40.343941+0000","last_peered":"2024-02-23T00:32:40.343941+0000","last_clean":"2024-02-23T00:32:40.343941+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:32:40.343941+0000","last_undegraded":"2024-02-23T00:32:40.343941+0000","last_fullsized":"2024-02-23T00:32:40.343941+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.343863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.343863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013865+0000","last_change":"2024-02-23T00:30:25.163377+0000","last_active":"2024-02-23T00:32:39.013865+0000","last_peered":"2024-02-23T00:32:39.013865+0000","last_clean":"2024-02-23T00:32:39.013865+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:32:39.013865+0000","last_undegraded":"2024-02-23T00:32:39.013865+0000","last_fullsized":"2024-02-23T00:32:39.013865+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.163297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.163297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":171,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022629+0000","last_change":"2024-02-23T00:32:12.721784+0000","last_active":"2024-02-23T00:32:39.022629+0000","last_peered":"2024-02-23T00:32:39.022629+0000","last_clean":"2024-02-23T00:32:39.022629+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:32:39.022629+0000","last_undegraded":"2024-02-23T00:32:39.022629+0000","last_fullsized":"2024-02-23T00:32:39.022629+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.721719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.721719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"148'7","reported_seq":139,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009180+0000","last_change":"2024-02-23T00:32:17.080494+0000","last_active":"2024-02-23T00:32:39.009180+0000","last_peered":"2024-02-23T00:32:39.009180+0000","last_clean":"2024-02-23T00:32:39.009180+0000","last_became_active":"2024-02-23T00:32:17.079954+0000","last_became_peered":"2024-02-23T00:32:17.079954+0000","last_unstale":"2024-02-23T00:32:39.009180+0000","last_undegraded":"2024-02-23T00:32:39.009180+0000","last_fullsized":"2024-02-23T00:32:39.009180+0000","mapping_epoch":96,"log_start":"148'7","ondisk_log_start":"148'7","created":96,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.015286+0000","last_change":"2024-02-23T00:30:48.696251+0000","last_active":"2024-02-23T00:32:39.015286+0000","last_peered":"2024-02-23T00:32:39.015286+0000","last_clean":"2024-02-23T00:32:39.015286+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:32:39.015286+0000","last_undegraded":"2024-02-23T00:32:39.015286+0000","last_fullsized":"2024-02-23T00:32:39.015286+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":212,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.025146+0000","last_change":"2024-02-23T00:31:27.176319+0000","last_active":"2024-02-23T00:32:39.025146+0000","last_peered":"2024-02-23T00:32:39.025146+0000","last_clean":"2024-02-23T00:32:39.025146+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:32:39.025146+0000","last_undegraded":"2024-02-23T00:32:39.025146+0000","last_fullsized":"2024-02-23T00:32:39.025146+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:31:27.176199+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:31:27.176199+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.014920+0000","last_change":"2024-02-23T00:30:22.245444+0000","last_active":"2024-02-23T00:32:39.014920+0000","last_peered":"2024-02-23T00:32:39.014920+0000","last_clean":"2024-02-23T00:32:39.014920+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:32:39.014920+0000","last_undegraded":"2024-02-23T00:32:39.014920+0000","last_fullsized":"2024-02-23T00:32:39.014920+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:22.245359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.245359+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":168,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019837+0000","last_change":"2024-02-23T00:30:33.528773+0000","last_active":"2024-02-23T00:32:39.019837+0000","last_peered":"2024-02-23T00:32:39.019837+0000","last_clean":"2024-02-23T00:32:39.019837+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:32:39.019837+0000","last_undegraded":"2024-02-23T00:32:39.019837+0000","last_fullsized":"2024-02-23T00:32:39.019837+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:33.528703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:33.528703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.026924+0000","last_change":"2024-02-23T00:30:31.829814+0000","last_active":"2024-02-23T00:32:39.026924+0000","last_peered":"2024-02-23T00:32:39.026924+0000","last_clean":"2024-02-23T00:32:39.026924+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:32:39.026924+0000","last_undegraded":"2024-02-23T00:32:39.026924+0000","last_fullsized":"2024-02-23T00:32:39.026924+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.829697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.829697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.6","version":"148'11","reported_seq":150,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.008839+0000","last_change":"2024-02-23T00:32:21.029226+0000","last_active":"2024-02-23T00:32:39.008839+0000","last_peered":"2024-02-23T00:32:39.008839+0000","last_clean":"2024-02-23T00:32:39.008839+0000","last_became_active":"2024-02-23T00:32:21.028872+0000","last_became_peered":"2024-02-23T00:32:21.028872+0000","last_unstale":"2024-02-23T00:32:39.008839+0000","last_undegraded":"2024-02-23T00:32:39.008839+0000","last_fullsized":"2024-02-23T00:32:39.008839+0000","mapping_epoch":96,"log_start":"148'11","ondisk_log_start":"148'11","created":96,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004826+0000","last_change":"2024-02-23T00:30:48.723611+0000","last_active":"2024-02-23T00:32:39.004826+0000","last_peered":"2024-02-23T00:32:39.004826+0000","last_clean":"2024-02-23T00:32:39.004826+0000","last_became_active":"2024-02-23T00:30:48.722602+0000","last_became_peered":"2024-02-23T00:30:48.722602+0000","last_unstale":"2024-02-23T00:32:39.004826+0000","last_undegraded":"2024-02-23T00:32:39.004826+0000","last_fullsized":"2024-02-23T00:32:39.004826+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"92'22","reported_seq":207,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004075+0000","last_change":"2024-02-23T00:30:38.900864+0000","last_active":"2024-02-23T00:32:39.004075+0000","last_peered":"2024-02-23T00:32:39.004075+0000","last_clean":"2024-02-23T00:32:39.004075+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:32:39.004075+0000","last_undegraded":"2024-02-23T00:32:39.004075+0000","last_fullsized":"2024-02-23T00:32:39.004075+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.900761+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":171,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.006782+0000","last_change":"2024-02-23T00:31:20.333610+0000","last_active":"2024-02-23T00:32:39.006782+0000","last_peered":"2024-02-23T00:32:39.006782+0000","last_clean":"2024-02-23T00:32:39.006782+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:32:39.006782+0000","last_undegraded":"2024-02-23T00:32:39.006782+0000","last_fullsized":"2024-02-23T00:32:39.006782+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:20.333547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:20.333547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":250,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-02-23T00:32:40.017019+0000","last_change":"2024-02-23T00:31:19.971484+0000","last_active":"2024-02-23T00:32:40.017019+0000","last_peered":"2024-02-23T00:32:40.017019+0000","last_clean":"2024-02-23T00:32:40.017019+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:32:40.017019+0000","last_undegraded":"2024-02-23T00:32:40.017019+0000","last_fullsized":"2024-02-23T00:32:40.017019+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:31:19.971419+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.971419+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":168,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.021021+0000","last_change":"2024-02-23T00:30:35.523228+0000","last_active":"2024-02-23T00:32:39.021021+0000","last_peered":"2024-02-23T00:32:39.021021+0000","last_clean":"2024-02-23T00:32:39.021021+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:32:39.021021+0000","last_undegraded":"2024-02-23T00:32:39.021021+0000","last_fullsized":"2024-02-23T00:32:39.021021+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:30:35.523157+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.523157+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":139,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.027432+0000","last_change":"2024-02-23T00:32:09.024454+0000","last_active":"2024-02-23T00:32:39.027432+0000","last_peered":"2024-02-23T00:32:39.027432+0000","last_clean":"2024-02-23T00:32:39.027432+0000","last_became_active":"2024-02-23T00:32:09.024182+0000","last_became_peered":"2024-02-23T00:32:09.024182+0000","last_unstale":"2024-02-23T00:32:39.027432+0000","last_undegraded":"2024-02-23T00:32:39.027432+0000","last_fullsized":"2024-02-23T00:32:39.027432+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.021083+0000","last_change":"2024-02-23T00:30:48.748188+0000","last_active":"2024-02-23T00:32:39.021083+0000","last_peered":"2024-02-23T00:32:39.021083+0000","last_clean":"2024-02-23T00:32:39.021083+0000","last_became_active":"2024-02-23T00:30:48.745982+0000","last_became_peered":"2024-02-23T00:30:48.745982+0000","last_unstale":"2024-02-23T00:32:39.021083+0000","last_undegraded":"2024-02-23T00:32:39.021083+0000","last_fullsized":"2024-02-23T00:32:39.021083+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":205,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.015224+0000","last_change":"2024-02-23T00:30:30.207937+0000","last_active":"2024-02-23T00:32:39.015224+0000","last_peered":"2024-02-23T00:32:39.015224+0000","last_clean":"2024-02-23T00:32:39.015224+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:32:39.015224+0000","last_undegraded":"2024-02-23T00:32:39.015224+0000","last_fullsized":"2024-02-23T00:32:39.015224+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:30:30.207850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.207850+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.025009+0000","last_change":"2024-02-23T00:32:37.932120+0000","last_active":"2024-02-23T00:32:39.025009+0000","last_peered":"2024-02-23T00:32:39.025009+0000","last_clean":"2024-02-23T00:32:39.025009+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:32:39.025009+0000","last_undegraded":"2024-02-23T00:32:39.025009+0000","last_fullsized":"2024-02-23T00:32:39.025009+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.932055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:49.056184+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.932055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":174,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004082+0000","last_change":"2024-02-23T00:32:35.624529+0000","last_active":"2024-02-23T00:32:39.004082+0000","last_peered":"2024-02-23T00:32:39.004082+0000","last_clean":"2024-02-23T00:32:39.004082+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:32:39.004082+0000","last_undegraded":"2024-02-23T00:32:39.004082+0000","last_fullsized":"2024-02-23T00:32:39.004082+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.624461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.624461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.012321+0000","last_change":"2024-02-23T00:32:10.145767+0000","last_active":"2024-02-23T00:32:39.012321+0000","last_peered":"2024-02-23T00:32:39.012321+0000","last_clean":"2024-02-23T00:32:39.012321+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:32:39.012321+0000","last_undegraded":"2024-02-23T00:32:39.012321+0000","last_fullsized":"2024-02-23T00:32:39.012321+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.145685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.145685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.8","version":"98'2","reported_seq":141,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013189+0000","last_change":"2024-02-23T00:32:13.036407+0000","last_active":"2024-02-23T00:32:39.013189+0000","last_peered":"2024-02-23T00:32:39.013189+0000","last_clean":"2024-02-23T00:32:39.013189+0000","last_became_active":"2024-02-23T00:32:13.036071+0000","last_became_peered":"2024-02-23T00:32:13.036071+0000","last_unstale":"2024-02-23T00:32:39.013189+0000","last_undegraded":"2024-02-23T00:32:39.013189+0000","last_fullsized":"2024-02-23T00:32:39.013189+0000","mapping_epoch":96,"log_start":"98'2","ondisk_log_start":"98'2","created":96,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022494+0000","last_change":"2024-02-23T00:30:48.737859+0000","last_active":"2024-02-23T00:32:39.022494+0000","last_peered":"2024-02-23T00:32:39.022494+0000","last_clean":"2024-02-23T00:32:39.022494+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:32:39.022494+0000","last_undegraded":"2024-02-23T00:32:39.022494+0000","last_fullsized":"2024-02-23T00:32:39.022494+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":201,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009981+0000","last_change":"2024-02-23T00:31:31.806791+0000","last_active":"2024-02-23T00:32:39.009981+0000","last_peered":"2024-02-23T00:32:39.009981+0000","last_clean":"2024-02-23T00:32:39.009981+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:32:39.009981+0000","last_undegraded":"2024-02-23T00:32:39.009981+0000","last_fullsized":"2024-02-23T00:32:39.009981+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:31:31.806687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.806687+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.016690+0000","last_change":"2024-02-23T00:30:40.183533+0000","last_active":"2024-02-23T00:32:39.016690+0000","last_peered":"2024-02-23T00:32:39.016690+0000","last_clean":"2024-02-23T00:32:39.016690+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:32:39.016690+0000","last_undegraded":"2024-02-23T00:32:39.016690+0000","last_fullsized":"2024-02-23T00:32:39.016690+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.183453+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":171,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.012798+0000","last_change":"2024-02-23T00:31:38.887671+0000","last_active":"2024-02-23T00:32:39.012798+0000","last_peered":"2024-02-23T00:32:39.012798+0000","last_clean":"2024-02-23T00:32:39.012798+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:32:39.012798+0000","last_undegraded":"2024-02-23T00:32:39.012798+0000","last_fullsized":"2024-02-23T00:32:39.012798+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.887597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.008504+0000","last_change":"2024-02-23T00:32:12.983101+0000","last_active":"2024-02-23T00:32:39.008504+0000","last_peered":"2024-02-23T00:32:39.008504+0000","last_clean":"2024-02-23T00:32:39.008504+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:32:39.008504+0000","last_undegraded":"2024-02-23T00:32:39.008504+0000","last_fullsized":"2024-02-23T00:32:39.008504+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.983023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.983023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.b","version":"148'14","reported_seq":94,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007252+0000","last_change":"2024-02-23T00:32:01.030834+0000","last_active":"2024-02-23T00:32:39.007252+0000","last_peered":"2024-02-23T00:32:39.007252+0000","last_clean":"2024-02-23T00:32:39.007252+0000","last_became_active":"2024-02-23T00:32:01.030404+0000","last_became_peered":"2024-02-23T00:32:01.030404+0000","last_unstale":"2024-02-23T00:32:39.007252+0000","last_undegraded":"2024-02-23T00:32:39.007252+0000","last_fullsized":"2024-02-23T00:32:39.007252+0000","mapping_epoch":147,"log_start":"148'14","ondisk_log_start":"148'14","created":96,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009670+0000","last_change":"2024-02-23T00:30:48.711374+0000","last_active":"2024-02-23T00:32:39.009670+0000","last_peered":"2024-02-23T00:32:39.009670+0000","last_clean":"2024-02-23T00:32:39.009670+0000","last_became_active":"2024-02-23T00:30:48.710049+0000","last_became_peered":"2024-02-23T00:30:48.710049+0000","last_unstale":"2024-02-23T00:32:39.009670+0000","last_undegraded":"2024-02-23T00:32:39.009670+0000","last_fullsized":"2024-02-23T00:32:39.009670+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":215,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.012760+0000","last_change":"2024-02-23T00:30:29.844070+0000","last_active":"2024-02-23T00:32:39.012760+0000","last_peered":"2024-02-23T00:32:39.012760+0000","last_clean":"2024-02-23T00:32:39.012760+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:32:39.012760+0000","last_undegraded":"2024-02-23T00:32:39.012760+0000","last_fullsized":"2024-02-23T00:32:39.012760+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:30:29.843954+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.843954+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":173,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.025257+0000","last_change":"2024-02-23T00:30:32.878666+0000","last_active":"2024-02-23T00:32:39.025257+0000","last_peered":"2024-02-23T00:32:39.025257+0000","last_clean":"2024-02-23T00:32:39.025257+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:32:39.025257+0000","last_undegraded":"2024-02-23T00:32:39.025257+0000","last_fullsized":"2024-02-23T00:32:39.025257+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:32.878569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:32.878569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004432+0000","last_change":"2024-02-23T00:31:45.961466+0000","last_active":"2024-02-23T00:32:39.004432+0000","last_peered":"2024-02-23T00:32:39.004432+0000","last_clean":"2024-02-23T00:32:39.004432+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:32:39.004432+0000","last_undegraded":"2024-02-23T00:32:39.004432+0000","last_fullsized":"2024-02-23T00:32:39.004432+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:45.961386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:45.961386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.019265+0000","last_change":"2024-02-23T00:29:47.044859+0000","last_active":"2024-02-23T00:32:39.019265+0000","last_peered":"2024-02-23T00:32:39.019265+0000","last_clean":"2024-02-23T00:32:39.019265+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:32:39.019265+0000","last_undegraded":"2024-02-23T00:32:39.019265+0000","last_fullsized":"2024-02-23T00:32:39.019265+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:29:47.044799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:29:47.044799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.a","version":"148'14","reported_seq":153,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007790+0000","last_change":"2024-02-23T00:32:05.034741+0000","last_active":"2024-02-23T00:32:39.007790+0000","last_peered":"2024-02-23T00:32:39.007790+0000","last_clean":"2024-02-23T00:32:39.007790+0000","last_became_active":"2024-02-23T00:32:05.034399+0000","last_became_peered":"2024-02-23T00:32:05.034399+0000","last_unstale":"2024-02-23T00:32:39.007790+0000","last_undegraded":"2024-02-23T00:32:39.007790+0000","last_fullsized":"2024-02-23T00:32:39.007790+0000","mapping_epoch":96,"log_start":"148'14","ondisk_log_start":"148'14","created":96,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009189+0000","last_change":"2024-02-23T00:30:48.679036+0000","last_active":"2024-02-23T00:32:39.009189+0000","last_peered":"2024-02-23T00:32:39.009189+0000","last_clean":"2024-02-23T00:32:39.009189+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:32:39.009189+0000","last_undegraded":"2024-02-23T00:32:39.009189+0000","last_fullsized":"2024-02-23T00:32:39.009189+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":217,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004008+0000","last_change":"2024-02-23T00:29:38.857115+0000","last_active":"2024-02-23T00:32:39.004008+0000","last_peered":"2024-02-23T00:32:39.004008+0000","last_clean":"2024-02-23T00:32:39.004008+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:32:39.004008+0000","last_undegraded":"2024-02-23T00:32:39.004008+0000","last_fullsized":"2024-02-23T00:32:39.004008+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:29:38.857029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:29:38.857029+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.020803+0000","last_change":"2024-02-23T00:30:47.629232+0000","last_active":"2024-02-23T00:32:39.020803+0000","last_peered":"2024-02-23T00:32:39.020803+0000","last_clean":"2024-02-23T00:32:39.020803+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:32:39.020803+0000","last_undegraded":"2024-02-23T00:32:39.020803+0000","last_fullsized":"2024-02-23T00:32:39.020803+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.629146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.629146+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.007910+0000","last_change":"2024-02-23T00:30:41.121986+0000","last_active":"2024-02-23T00:32:39.007910+0000","last_peered":"2024-02-23T00:32:39.007910+0000","last_clean":"2024-02-23T00:32:39.007910+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:32:39.007910+0000","last_undegraded":"2024-02-23T00:32:39.007910+0000","last_fullsized":"2024-02-23T00:32:39.007910+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:41.121916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.121916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":172,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.644625+0000","last_change":"2024-02-23T00:32:39.644590+0000","last_active":"2024-02-23T00:32:39.644625+0000","last_peered":"2024-02-23T00:32:39.644625+0000","last_clean":"2024-02-23T00:32:39.644625+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:32:39.644625+0000","last_undegraded":"2024-02-23T00:32:39.644625+0000","last_fullsized":"2024-02-23T00:32:39.644625+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:39.644483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:32:39.644483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.d","version":"148'9","reported_seq":95,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.002567+0000","last_change":"2024-02-23T00:31:53.000538+0000","last_active":"2024-02-23T00:32:39.002567+0000","last_peered":"2024-02-23T00:32:39.002567+0000","last_clean":"2024-02-23T00:32:39.002567+0000","last_became_active":"2024-02-23T00:31:53.000164+0000","last_became_peered":"2024-02-23T00:31:53.000164+0000","last_unstale":"2024-02-23T00:32:39.002567+0000","last_undegraded":"2024-02-23T00:32:39.002567+0000","last_fullsized":"2024-02-23T00:32:39.002567+0000","mapping_epoch":147,"log_start":"148'9","ondisk_log_start":"148'9","created":96,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":125,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.018816+0000","last_change":"2024-02-23T00:32:35.725253+0000","last_active":"2024-02-23T00:32:39.018816+0000","last_peered":"2024-02-23T00:32:39.018816+0000","last_clean":"2024-02-23T00:32:39.018816+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:32:39.018816+0000","last_undegraded":"2024-02-23T00:32:39.018816+0000","last_fullsized":"2024-02-23T00:32:39.018816+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.725182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.725182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":199,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.004684+0000","last_change":"2024-02-23T00:30:23.953417+0000","last_active":"2024-02-23T00:32:39.004684+0000","last_peered":"2024-02-23T00:32:39.004684+0000","last_clean":"2024-02-23T00:32:39.004684+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:32:39.004684+0000","last_undegraded":"2024-02-23T00:32:39.004684+0000","last_fullsized":"2024-02-23T00:32:39.004684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:30:23.953330+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.953330+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":177,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022228+0000","last_change":"2024-02-23T00:32:36.714218+0000","last_active":"2024-02-23T00:32:39.022228+0000","last_peered":"2024-02-23T00:32:39.022228+0000","last_clean":"2024-02-23T00:32:39.022228+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:32:39.022228+0000","last_undegraded":"2024-02-23T00:32:39.022228+0000","last_fullsized":"2024-02-23T00:32:39.022228+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.714139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.714139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":168,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022539+0000","last_change":"2024-02-23T00:30:38.549305+0000","last_active":"2024-02-23T00:32:39.022539+0000","last_peered":"2024-02-23T00:32:39.022539+0000","last_clean":"2024-02-23T00:32:39.022539+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:32:39.022539+0000","last_undegraded":"2024-02-23T00:32:39.022539+0000","last_fullsized":"2024-02-23T00:32:39.022539+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:30:38.549228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":165,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.008772+0000","last_change":"2024-02-23T00:31:39.494827+0000","last_active":"2024-02-23T00:32:39.008772+0000","last_peered":"2024-02-23T00:32:39.008772+0000","last_clean":"2024-02-23T00:32:39.008772+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:32:39.008772+0000","last_undegraded":"2024-02-23T00:32:39.008772+0000","last_fullsized":"2024-02-23T00:32:39.008772+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:31:39.494737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":89,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.020517+0000","last_change":"2024-02-23T00:31:57.022067+0000","last_active":"2024-02-23T00:32:39.020517+0000","last_peered":"2024-02-23T00:32:39.020517+0000","last_clean":"2024-02-23T00:32:39.020517+0000","last_became_active":"2024-02-23T00:31:57.021714+0000","last_became_peered":"2024-02-23T00:31:57.021714+0000","last_unstale":"2024-02-23T00:32:39.020517+0000","last_undegraded":"2024-02-23T00:32:39.020517+0000","last_fullsized":"2024-02-23T00:32:39.020517+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013806+0000","last_change":"2024-02-23T00:30:48.725798+0000","last_active":"2024-02-23T00:32:39.013806+0000","last_peered":"2024-02-23T00:32:39.013806+0000","last_clean":"2024-02-23T00:32:39.013806+0000","last_became_active":"2024-02-23T00:30:48.725162+0000","last_became_peered":"2024-02-23T00:30:48.725162+0000","last_unstale":"2024-02-23T00:32:39.013806+0000","last_undegraded":"2024-02-23T00:32:39.013806+0000","last_fullsized":"2024-02-23T00:32:39.013806+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":207,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003173+0000","last_change":"2024-02-23T00:32:10.903259+0000","last_active":"2024-02-23T00:32:39.003173+0000","last_peered":"2024-02-23T00:32:39.003173+0000","last_clean":"2024-02-23T00:32:39.003173+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:32:39.003173+0000","last_undegraded":"2024-02-23T00:32:39.003173+0000","last_fullsized":"2024-02-23T00:32:39.003173+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:32:10.903186+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.903186+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":175,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.017977+0000","last_change":"2024-02-23T00:31:22.137283+0000","last_active":"2024-02-23T00:32:39.017977+0000","last_peered":"2024-02-23T00:32:39.017977+0000","last_clean":"2024-02-23T00:32:39.017977+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:32:39.017977+0000","last_undegraded":"2024-02-23T00:32:39.017977+0000","last_fullsized":"2024-02-23T00:32:39.017977+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:22.137166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:22.137166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.013477+0000","last_change":"2024-02-23T00:31:26.920222+0000","last_active":"2024-02-23T00:32:39.013477+0000","last_peered":"2024-02-23T00:32:39.013477+0000","last_clean":"2024-02-23T00:32:39.013477+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:32:39.013477+0000","last_undegraded":"2024-02-23T00:32:39.013477+0000","last_fullsized":"2024-02-23T00:32:39.013477+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:26.920133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:26.920133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003702+0000","last_change":"2024-02-23T00:31:37.065941+0000","last_active":"2024-02-23T00:32:39.003702+0000","last_peered":"2024-02-23T00:32:39.003702+0000","last_clean":"2024-02-23T00:32:39.003702+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:32:39.003702+0000","last_undegraded":"2024-02-23T00:32:39.003702+0000","last_fullsized":"2024-02-23T00:32:39.003702+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:31:37.065865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.f","version":"148'7","reported_seq":89,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.009025+0000","last_change":"2024-02-23T00:31:45.006360+0000","last_active":"2024-02-23T00:32:39.009025+0000","last_peered":"2024-02-23T00:32:39.009025+0000","last_clean":"2024-02-23T00:32:39.009025+0000","last_became_active":"2024-02-23T00:31:45.006018+0000","last_became_peered":"2024-02-23T00:31:45.006018+0000","last_unstale":"2024-02-23T00:32:39.009025+0000","last_undegraded":"2024-02-23T00:32:39.009025+0000","last_fullsized":"2024-02-23T00:32:39.009025+0000","mapping_epoch":147,"log_start":"148'7","ondisk_log_start":"148'7","created":96,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.022800+0000","last_change":"2024-02-23T00:30:48.698701+0000","last_active":"2024-02-23T00:32:39.022800+0000","last_peered":"2024-02-23T00:32:39.022800+0000","last_clean":"2024-02-23T00:32:39.022800+0000","last_became_active":"2024-02-23T00:30:48.697963+0000","last_became_peered":"2024-02-23T00:30:48.697963+0000","last_unstale":"2024-02-23T00:32:39.022800+0000","last_undegraded":"2024-02-23T00:32:39.022800+0000","last_fullsized":"2024-02-23T00:32:39.022800+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"63'39","reported_seq":256,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.024750+0000","last_change":"2024-02-23T00:30:23.839691+0000","last_active":"2024-02-23T00:32:39.024750+0000","last_peered":"2024-02-23T00:32:39.024750+0000","last_clean":"2024-02-23T00:32:39.024750+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:32:39.024750+0000","last_undegraded":"2024-02-23T00:32:39.024750+0000","last_fullsized":"2024-02-23T00:32:39.024750+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'39","last_scrub_stamp":"2024-02-23T00:30:23.839568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.839568+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":178,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.002298+0000","last_change":"2024-02-23T00:32:13.602518+0000","last_active":"2024-02-23T00:32:39.002298+0000","last_peered":"2024-02-23T00:32:39.002298+0000","last_clean":"2024-02-23T00:32:39.002298+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:32:39.002298+0000","last_undegraded":"2024-02-23T00:32:39.002298+0000","last_fullsized":"2024-02-23T00:32:39.002298+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:13.602450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.602450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":169,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.003394+0000","last_change":"2024-02-23T00:31:40.071364+0000","last_active":"2024-02-23T00:32:39.003394+0000","last_peered":"2024-02-23T00:32:39.003394+0000","last_clean":"2024-02-23T00:32:39.003394+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:32:39.003394+0000","last_undegraded":"2024-02-23T00:32:39.003394+0000","last_fullsized":"2024-02-23T00:32:39.003394+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.071294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.071294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.025062+0000","last_change":"2024-02-23T00:30:46.848126+0000","last_active":"2024-02-23T00:32:39.025062+0000","last_peered":"2024-02-23T00:32:39.025062+0000","last_clean":"2024-02-23T00:32:39.025062+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:32:39.025062+0000","last_undegraded":"2024-02-23T00:32:39.025062+0000","last_fullsized":"2024-02-23T00:32:39.025062+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.848063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.848063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":89,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.008162+0000","last_change":"2024-02-23T00:31:49.020151+0000","last_active":"2024-02-23T00:32:39.008162+0000","last_peered":"2024-02-23T00:32:39.008162+0000","last_clean":"2024-02-23T00:32:39.008162+0000","last_became_active":"2024-02-23T00:31:49.019835+0000","last_became_peered":"2024-02-23T00:31:49.019835+0000","last_unstale":"2024-02-23T00:32:39.008162+0000","last_undegraded":"2024-02-23T00:32:39.008162+0000","last_fullsized":"2024-02-23T00:32:39.008162+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":120,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-02-23T00:32:39.011186+0000","last_change":"2024-02-23T00:30:48.722860+0000","last_active":"2024-02-23T00:32:39.011186+0000","last_peered":"2024-02-23T00:32:39.011186+0000","last_clean":"2024-02-23T00:32:39.011186+0000","last_became_active":"2024-02-23T00:30:48.722569+0000","last_became_peered":"2024-02-23T00:30:48.722569+0000","last_unstale":"2024-02-23T00:32:39.011186+0000","last_undegraded":"2024-02-23T00:32:39.011186+0000","last_fullsized":"2024-02-23T00:32:39.011186+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":147,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":146,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":144,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":2336,"num_objects":216,"num_object_clones":109,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":88,"num_read_kb":1,"num_write":514,"num_write_kb":513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2654208,"data_stored":7008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1511,"ondisk_log_size":1511,"up":96,"acting":96,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":19,"num_pg":16,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":640,"ondisk_log_size":640,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":681,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":185,"num_read_kb":181,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332164,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304648,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440504,"statfs":{"total":95995035648,"available":95683076096,"internally_reserved":0,"allocated":3403776,"data_stored":1851492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.0830000000000002,"15min":1.028},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":57.401000000000003,"5min":61.048999999999999,"15min":61.048999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":3.6440000000000001,"5min":2.6549999999999998,"15min":1.2390000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":65.558999999999997,"5min":73.870999999999995,"15min":73.870999999999995},"last":0.36799999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":2.028,"5min":1.026,"15min":0.69499999999999995},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.869,"5min":1.044,"15min":0.68400000000000005},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":29.423999999999999,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.95499999999999996,"15min":0.66900000000000004},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":9.5190000000000001,"5min":16.603999999999999,"15min":16.603999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.753,"15min":0.59999999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":7.5339999999999998,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.622}]},{"osd":3,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":3.2280000000000002,"5min":1.871,"15min":0.97099999999999997},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":57.729999999999997,"5min":57.729999999999997,"15min":57.729999999999997},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.613,"15min":0.89400000000000002},"min":{"1min":0.31900000000000001,"5min":0.312,"15min":0.312},"max":{"1min":27.573,"5min":27.573,"15min":27.573},"last":0.70699999999999996}]},{"osd":4,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41899999999999998,"15min":0.41199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.43099999999999999,"15min":0.42499999999999999},"min":{"1min":0.25700000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.65200000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.43099999999999999,"15min":0.42799999999999999},"min":{"1min":0.27200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45600000000000002,"15min":0.435},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.67100000000000004}]},{"osd":6,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.48099999999999998,"15min":0.47299999999999998},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48099999999999998,"15min":0.46100000000000002},"min":{"1min":0.30399999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.72799999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561097,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308832,"kb_used_data":3412,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436320,"statfs":{"total":95995035648,"available":95678791680,"internally_reserved":0,"allocated":3493888,"data_stored":1861488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:31:59 2024","interfaces":[{"interface":"back","average":{"1min":6.3810000000000002,"5min":1.718,"15min":0.90400000000000003},"min":{"1min":0.27400000000000002,"5min":0.22,"15min":0.22},"max":{"1min":94.906000000000006,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.252,"5min":0.72499999999999998,"15min":0.58899999999999997},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":9.468,"5min":9.468,"15min":9.468},"last":0.502}]},{"osd":1,"last update":"Fri Feb 23 00:31:59 2024","interfaces":[{"interface":"back","average":{"1min":7.29,"5min":2.0379999999999998,"15min":1.016},"min":{"1min":0.32000000000000001,"5min":0.316,"15min":0.316},"max":{"1min":59.347999999999999,"5min":59.347999999999999,"15min":59.347999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":11.893000000000001,"5min":2.883,"15min":1.2929999999999999},"min":{"1min":0.35199999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":87.221000000000004,"5min":87.221000000000004,"15min":87.221000000000004},"last":0.60699999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:31:59 2024","interfaces":[{"interface":"back","average":{"1min":3.7679999999999998,"5min":1.431,"15min":0.80800000000000005},"min":{"1min":0.39600000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":25.550999999999998,"5min":25.550999999999998,"15min":25.550999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":4.2210000000000001,"5min":1.532,"15min":0.877},"min":{"1min":0.36499999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":40.914000000000001,"5min":40.914000000000001,"15min":40.914000000000001},"last":0.56999999999999995}]},{"osd":3,"last update":"Fri Feb 23 00:31:59 2024","interfaces":[{"interface":"back","average":{"1min":3.7930000000000001,"5min":1.3899999999999999,"15min":0.78900000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":31.309999999999999,"5min":31.309999999999999,"15min":31.309999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":10.048,"5min":2.919,"15min":1.3220000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":86.311999999999998,"5min":86.311999999999998,"15min":86.311999999999998},"last":0.65400000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.45300000000000001,"15min":0.42599999999999999},"min":{"1min":0.24399999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.7839999999999998,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.65500000000000003,"15min":0.48799999999999999},"min":{"1min":0.26100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.1259999999999999,"5min":17.718,"15min":17.718},"last":0.59499999999999997}]},{"osd":5,"last update":"Fri Feb 23 00:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56299999999999994,"15min":0.47299999999999998},"min":{"1min":0.28199999999999997,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.4210000000000003,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.54000000000000004,"15min":0.45900000000000002},"min":{"1min":0.309,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67900000000000005,"5min":8.1370000000000005,"15min":8.1370000000000005},"last":0.66500000000000004}]},{"osd":7,"last update":"Fri Feb 23 00:32:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.82899999999999996,"15min":0.58799999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.52100000000000002,"5min":17.881,"15min":17.881},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.79700000000000004,"15min":0.58099999999999996},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54400000000000004,"5min":17.396999999999998,"15min":17.396999999999998},"last":0.72499999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309180,"kb_used_data":3760,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435972,"statfs":{"total":95995035648,"available":95678435328,"internally_reserved":0,"allocated":3850240,"data_stored":1863890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:31:58 2024","interfaces":[{"interface":"back","average":{"1min":3.8159999999999998,"5min":1.0669999999999999,"15min":0.57199999999999995},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":37.133000000000003,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.714,"5min":0.68200000000000005,"15min":0.45800000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20799999999999999},"max":{"1min":11.157999999999999,"5min":11.157999999999999,"15min":11.157999999999999},"last":0.28499999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.89500000000000002,"15min":0.54700000000000004},"min":{"1min":0.249,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.7570000000000001,"5min":29.140999999999998,"15min":29.140999999999998},"last":0.5},{"interface":"front","average":{"1min":0.875,"5min":0.65600000000000003,"15min":0.46999999999999997},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":8.8300000000000001,"5min":15.273,"15min":15.273},"last":0.41499999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:31:43 2024","interfaces":[{"interface":"back","average":{"1min":3.5790000000000002,"5min":1.016,"15min":0.626},"min":{"1min":0.29499999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":21.013999999999999,"5min":21.013999999999999,"15min":21.013999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":3.4590000000000001,"5min":1.02,"15min":0.629},"min":{"1min":0.316,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":29.234999999999999,"5min":29.234999999999999,"15min":29.234999999999999},"last":0.63500000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:32:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":0.76500000000000001,"15min":0.625},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":12.103,"5min":12.103,"15min":12.103},"last":0.54200000000000004},{"interface":"front","average":{"1min":3.3199999999999998,"5min":1.111,"15min":0.73599999999999999},"min":{"1min":0.33800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":29.212,"5min":29.212,"15min":29.212},"last":0.56699999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.79100000000000004,"15min":0.65100000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71799999999999997,"5min":11.904,"15min":11.904},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.83099999999999996,"15min":0.67000000000000004},"min":{"1min":0.378,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.8889999999999998,"5min":11.105,"15min":11.105},"last":0.52400000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:31:58 2024","interfaces":[{"interface":"back","average":{"1min":3.1739999999999999,"5min":1.099,"15min":0.74399999999999999},"min":{"1min":0.39100000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":29.187000000000001,"5min":29.187000000000001,"15min":29.187000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.72,"5min":0.83399999999999996,"15min":0.69199999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":11.946999999999999,"5min":11.946999999999999,"15min":11.946999999999999},"last":0.46200000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.80600000000000005,"15min":0.65000000000000002},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":8.8040000000000003,"5min":11.971,"15min":11.971},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.115,"5min":0.82699999999999996,"15min":0.68300000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.4379999999999997,"5min":11.135999999999999,"15min":11.135999999999999},"last":0.55700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304836,"kb_used_data":3504,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440316,"statfs":{"total":95995035648,"available":95682883584,"internally_reserved":0,"allocated":3588096,"data_stored":1850718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:31:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4550000000000001,"5min":0.76900000000000002,"15min":0.47499999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":15.986000000000001,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.2210000000000001,"5min":1.1299999999999999,"15min":0.59799999999999998},"min":{"1min":0.22600000000000001,"5min":0.216,"15min":0.216},"max":{"1min":23.832999999999998,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.255}]},{"osd":2,"last update":"Fri Feb 23 00:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.95499999999999996,"15min":0.58299999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.0620000000000001,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.318},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66400000000000003,"15min":0.46400000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.9379999999999997,"5min":15.965,"15min":15.965},"last":0.27400000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:31:42 2024","interfaces":[{"interface":"back","average":{"1min":3.4609999999999999,"5min":1.014,"15min":0.64100000000000001},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":24.164999999999999,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":3.1509999999999998,"5min":0.96199999999999997,"15min":0.622},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":16.006,"5min":16.006,"15min":16.006},"last":0.308}]},{"osd":4,"last update":"Fri Feb 23 00:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":0.80700000000000005,"15min":0.64100000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":15.917999999999999,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":2.8170000000000002,"5min":1.135,"15min":0.754},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":13.287000000000001,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.0680000000000001,"15min":0.72399999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.4660000000000002,"5min":13.311,"15min":13.311},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.0780000000000001,"15min":0.73099999999999998},"min":{"1min":0.38600000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.5139999999999998,"5min":23.744,"15min":23.744},"last":0.52200000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:31:59 2024","interfaces":[{"interface":"back","average":{"1min":3.0550000000000002,"5min":1.333,"15min":0.78900000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":13.334,"5min":33.082999999999998,"15min":33.082999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.9179999999999999,"5min":0.90200000000000002,"15min":0.67700000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":15.874000000000001,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.44400000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:32:18 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.175,"15min":0.76400000000000001},"min":{"1min":0.434,"5min":0.373,"15min":0.373},"max":{"1min":15.943,"5min":31.251999999999999,"15min":31.251999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":3.1059999999999999,"5min":1.141,"15min":0.753},"min":{"1min":0.38400000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":23.785,"5min":23.785,"15min":23.785},"last":0.48399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304760,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440392,"statfs":{"total":95995035648,"available":95682961408,"internally_reserved":0,"allocated":3514368,"data_stored":1851345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.4430000000000001,"5min":1.4850000000000001,"15min":0.71599999999999997},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":56.509,"5min":56.509,"15min":56.509},"last":0.39000000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.4179999999999999,"15min":0.69399999999999995},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":19.571999999999999,"5min":31.498999999999999,"15min":31.498999999999999},"last":0.34499999999999997}]},{"osd":1,"last update":"Fri Feb 23 00:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":0.96899999999999997,"15min":0.54900000000000004},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":19.324999999999999,"5min":19.324999999999999,"15min":19.324999999999999},"last":0.313},{"interface":"front","average":{"1min":1.534,"5min":1.288,"15min":0.65600000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.216},"max":{"1min":20.655000000000001,"5min":35.052,"15min":35.052},"last":0.36499999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:31:50 2024","interfaces":[{"interface":"back","average":{"1min":4.3220000000000001,"5min":1.2549999999999999,"15min":0.69699999999999995},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":43.5,"5min":43.5,"15min":43.5},"last":0.40200000000000002},{"interface":"front","average":{"1min":2.5939999999999999,"5min":0.97399999999999998,"15min":0.61899999999999999},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":27.414000000000001,"5min":27.414000000000001,"15min":27.414000000000001},"last":0.41399999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:32:08 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.76700000000000002,"15min":0.61699999999999999},"min":{"1min":0.36499999999999999,"5min":0.308,"15min":0.308},"max":{"1min":14.614000000000001,"5min":14.614000000000001,"15min":14.614000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.65300000000000002,"15min":0.58899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":4.6550000000000002,"5min":4.6550000000000002,"15min":4.6550000000000002},"last":0.56399999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:31:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":0.89300000000000002,"15min":0.65900000000000003},"min":{"1min":0.34899999999999998,"5min":0.316,"15min":0.316},"max":{"1min":17.765000000000001,"5min":17.765000000000001,"15min":17.765000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.5249999999999999,"5min":0.78900000000000003,"15min":0.63100000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":14.589,"5min":14.589,"15min":14.589},"last":0.54500000000000004}]},{"osd":6,"last update":"Fri Feb 23 00:31:54 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.67200000000000004,"15min":0.58999999999999997},"min":{"1min":0.32700000000000001,"5min":0.317,"15min":0.317},"max":{"1min":4.6120000000000001,"5min":4.6120000000000001,"15min":4.6120000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":3.0030000000000001,"5min":1.054,"15min":0.748},"min":{"1min":0.37,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":20.574999999999999,"5min":20.574999999999999,"15min":20.574999999999999},"last":0.54900000000000004}]},{"osd":7,"last update":"Fri Feb 23 00:32:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.8839999999999999,"15min":1.024},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":11.811999999999999,"5min":66.632000000000005,"15min":66.632000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.222,"5min":1.212,"15min":0.80100000000000005},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":11.781000000000001,"5min":19.603999999999999,"15min":19.603999999999999},"last":0.67800000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304652,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440500,"statfs":{"total":95995035648,"available":95683072000,"internally_reserved":0,"allocated":3403776,"data_stored":1850872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:31:42 2024","interfaces":[{"interface":"back","average":{"1min":3.8620000000000001,"5min":1.0529999999999999,"15min":0.59999999999999998},"min":{"1min":0.218,"5min":0.185,"15min":0.185},"max":{"1min":45.390999999999998,"5min":45.390999999999998,"15min":45.390999999999998},"last":0.217},{"interface":"front","average":{"1min":3.7050000000000001,"5min":1.022,"15min":0.60999999999999999},"min":{"1min":0.251,"5min":0.191,"15min":0.191},"max":{"1min":30.186,"5min":30.186,"15min":30.186},"last":0.311}]},{"osd":1,"last update":"Fri Feb 23 00:31:42 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":0.629,"15min":0.501},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":18.039999999999999,"5min":18.039999999999999,"15min":18.039999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":3.5710000000000002,"5min":1.0369999999999999,"15min":0.622},"min":{"1min":0.29799999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":45.441000000000003,"5min":45.441000000000003,"15min":45.441000000000003},"last":0.32200000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:31:42 2024","interfaces":[{"interface":"back","average":{"1min":3.3450000000000002,"5min":0.98599999999999999,"15min":0.624},"min":{"1min":0.28100000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":45.462000000000003,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.29799999999999999},{"interface":"front","average":{"1min":3.0640000000000001,"5min":0.92200000000000004,"15min":0.59599999999999997},"min":{"1min":0.28699999999999998,"5min":0.185,"15min":0.185},"max":{"1min":24.183,"5min":24.183,"15min":24.183},"last":0.45400000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:32:09 2024","interfaces":[{"interface":"back","average":{"1min":3.1219999999999999,"5min":1.085,"15min":0.749},"min":{"1min":0.29999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":24.058,"5min":24.058,"15min":24.058},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.2110000000000001,"5min":0.74099999999999999,"15min":0.64000000000000001},"min":{"1min":0.35699999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.391,"5min":8.0359999999999996,"15min":8.0359999999999996},"last":0.38400000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":1.056,"15min":0.73299999999999998},"min":{"1min":0.33900000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77200000000000002,"5min":45.265000000000001,"15min":45.265000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.91400000000000003,"15min":0.69399999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.113,"5min":24.102,"15min":24.102},"last":0.498}]},{"osd":6,"last update":"Fri Feb 23 00:31:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":0.75600000000000001,"15min":0.63800000000000001},"min":{"1min":0.35899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.4199999999999999,"5min":8.093,"15min":8.093},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.472,"5min":1.196,"15min":0.80500000000000005},"min":{"1min":0.371,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":8.5229999999999997,"5min":45.232999999999997,"15min":45.232999999999997},"last":0.41999999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:32:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.71199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.1240000000000001,"5min":8.1280000000000001,"15min":8.1280000000000001},"last":0.436},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.089,"15min":0.77000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65100000000000002,"5min":45.287999999999997,"15min":45.287999999999997},"last":0.48799999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986259,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313108,"kb_used_data":3596,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432044,"statfs":{"total":95995035648,"available":95674413056,"internally_reserved":0,"allocated":3682304,"data_stored":1851714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:00 2024","interfaces":[{"interface":"back","average":{"1min":3.1539999999999999,"5min":1.329,"15min":0.78900000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":21.419,"5min":21.419,"15min":21.419},"last":0.53400000000000003},{"interface":"front","average":{"1min":7.6509999999999998,"5min":2.5529999999999999,"15min":1.204},"min":{"1min":0.28399999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":63.462000000000003,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.35699999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:32:00 2024","interfaces":[{"interface":"back","average":{"1min":3.6739999999999999,"5min":1.7749999999999999,"15min":0.94499999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":16.318000000000001,"5min":41.137999999999998,"15min":41.137999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":2.4500000000000002,"5min":1.357,"15min":0.79900000000000004},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":10.491,"5min":26.210000000000001,"15min":26.210000000000001},"last":0.45800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:32:00 2024","interfaces":[{"interface":"back","average":{"1min":3.9140000000000001,"5min":2.089,"15min":1.056},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":41.886000000000003,"5min":67.253,"15min":67.253},"last":0.503},{"interface":"front","average":{"1min":5.4050000000000002,"5min":1.821,"15min":0.95499999999999996},"min":{"1min":0.318,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":41.927999999999997,"5min":41.927999999999997,"15min":41.927999999999997},"last":0.437}]},{"osd":3,"last update":"Fri Feb 23 00:32:00 2024","interfaces":[{"interface":"back","average":{"1min":4.7969999999999997,"5min":1.9830000000000001,"15min":1.0109999999999999},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":36.301000000000002,"5min":36.301000000000002,"15min":36.301000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":6.8360000000000003,"5min":2.2040000000000002,"15min":1.095},"min":{"1min":0.33100000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":42.064,"5min":42.064,"15min":42.064},"last":0.42199999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.754,"15min":0.49199999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.1360000000000001,"5min":14.242000000000001,"15min":14.242000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.69999999999999996,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6299999999999999,"5min":14.124000000000001,"15min":14.124000000000001},"last":0.40400000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:31:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.56799999999999995,"15min":0.48799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":4.5860000000000003,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.92700000000000005,"15min":0.59499999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.8650000000000002,"5min":45.814999999999998,"15min":45.814999999999998},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:32:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.753,"15min":0.55900000000000005},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.6630000000000003,"5min":17.178999999999998,"15min":17.178999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.028,"5min":0.77800000000000002,"15min":0.59499999999999997},"min":{"1min":0.35599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.1959999999999997,"5min":14.212,"15min":14.212},"last":0.57299999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757326,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304988,"kb_used_data":3660,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440164,"statfs":{"total":95995035648,"available":95682727936,"internally_reserved":0,"allocated":3747840,"data_stored":1862514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":1.115,"15min":0.70999999999999996},"min":{"1min":0.23400000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.67600000000000005,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.435,"5min":2.1070000000000002,"15min":1.0269999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.66000000000000003,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.29,"15min":0.748},"min":{"1min":0.27100000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72299999999999998,"5min":38.838000000000001,"15min":38.838000000000001},"last":0.5},{"interface":"front","average":{"1min":0.439,"5min":2.5179999999999998,"15min":1.1699999999999999},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.69599999999999995,"5min":127.794,"15min":127.794},"last":0.54600000000000004}]},{"osd":2,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.651,"15min":0.88400000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63,"5min":37.442999999999998,"15min":37.442999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":1.0620000000000001,"15min":0.69099999999999995},"min":{"1min":0.30499999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67200000000000004,"5min":23.297999999999998,"15min":23.297999999999998},"last":0.53600000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":1.226,"15min":0.72899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71799999999999997,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":1.6160000000000001,"15min":0.88800000000000001},"min":{"1min":0.33300000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.72199999999999998,"5min":40.865000000000002,"15min":40.865000000000002},"last":0.57199999999999995}]},{"osd":4,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.5489999999999999,"15min":0.75900000000000001},"min":{"1min":0.27400000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.63700000000000001,"5min":103.13,"15min":103.13},"last":0.432},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.57399999999999995,"15min":0.432},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.628,"5min":12.795999999999999,"15min":12.795999999999999},"last":0.44700000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:31:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.55900000000000005,"15min":0.46500000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69499999999999995,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.442,"5min":1.5509999999999999,"15min":0.80300000000000005},"min":{"1min":0.27400000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.622,"5min":103.61499999999999,"15min":103.61499999999999},"last":0.51500000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.56499999999999995,"15min":0.51100000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66300000000000003,"5min":7.1820000000000004,"15min":7.1820000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.69699999999999995,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.59999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":1495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":1311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":144,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":144,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":144,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":144,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":144,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":144,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":144,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":144,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:32:46.129 INFO:tasks.thrashosds.thrasher:No pools; doing nothing 2024-02-23T00:32:46.129 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:46.130 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:32:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:46.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:32:46.234 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:46.267 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:46.288 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:46.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:46.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:32:46.392 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:32:46.422 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:46.442 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:32:46.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:46.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:32:46.547 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:46.574 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:46.672 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:46.673 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.362909+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.366412+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.408742+0000 osd.5 (osd.5) 99 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.409543+0000 osd.5 (osd.5) 100 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.533067+0000 osd.0 (osd.0) 127 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.535984+0000 osd.0 (osd.0) 128 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.589074+0000 mgr.y (mgr.24437) 250 : cluster [DBG] pgmap v304: 222 pgs: 201 active+clean, 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:45.590068+0000 mon.a (mon.0) 1587 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.663045+0000 osd.7 (osd.7) 111 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:45.663901+0000 osd.7 (osd.7) 112 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:46.005754+0000 mon.a (mon.0) 1588 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]': finished 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:46.005864+0000 mon.a (mon.0) 1589 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.005925+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:46.014295+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:46.674 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:46.016490+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:46.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:46.017900+0000 mon.c (mon.2) 254 : audit [INF] from='client.? 172.21.15.87:0/4049231794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi087-82019-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:46.019062+0000 mon.b (mon.1) 190 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:46.019520+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi087-82019-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:46.019839+0000 mon.a (mon.0) 1593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:46.445081+0000 mon.c (mon.2) 255 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:46.675 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.362909+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:32:46.675 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.366412+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:32:46.675 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.408742+0000 osd.5 (osd.5) 99 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:32:46.675 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.409543+0000 osd.5 (osd.5) 100 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.533067+0000 osd.0 (osd.0) 127 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.535984+0000 osd.0 (osd.0) 128 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.589074+0000 mgr.y (mgr.24437) 250 : cluster [DBG] pgmap v304: 222 pgs: 201 active+clean, 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:45.590068+0000 mon.a (mon.0) 1587 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.663045+0000 osd.7 (osd.7) 111 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:45.663901+0000 osd.7 (osd.7) 112 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:46.005754+0000 mon.a (mon.0) 1588 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]': finished 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:46.005864+0000 mon.a (mon.0) 1589 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.005925+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:46.014295+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:46.016490+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:46.017900+0000 mon.c (mon.2) 254 : audit [INF] from='client.? 172.21.15.87:0/4049231794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi087-82019-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:46.019062+0000 mon.b (mon.1) 190 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:46.019520+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi087-82019-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.676 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:46.019839+0000 mon.a (mon.0) 1593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.677 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:46.445081+0000 mon.c (mon.2) 255 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:46.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.362909+0000 osd.1 (osd.1) 109 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:32:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.366412+0000 osd.1 (osd.1) 110 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:32:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.408742+0000 osd.5 (osd.5) 99 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:32:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.409543+0000 osd.5 (osd.5) 100 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:32:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.533067+0000 osd.0 (osd.0) 127 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:32:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.535984+0000 osd.0 (osd.0) 128 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.589074+0000 mgr.y (mgr.24437) 250 : cluster [DBG] pgmap v304: 222 pgs: 201 active+clean, 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:45.590068+0000 mon.a (mon.0) 1587 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.663045+0000 osd.7 (osd.7) 111 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:45.663901+0000 osd.7 (osd.7) 112 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:46.005754+0000 mon.a (mon.0) 1588 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi087-82011-31"}]': finished 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:46.005864+0000 mon.a (mon.0) 1589 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.005925+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:46.014295+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.87:0/758004491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:46.016490+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]: dispatch 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:46.017900+0000 mon.c (mon.2) 254 : audit [INF] from='client.? 172.21.15.87:0/4049231794' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi087-82019-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:46.019062+0000 mon.b (mon.1) 190 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:46.019520+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi087-82019-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:46.019839+0000 mon.a (mon.0) 1593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:46.445081+0000 mon.c (mon.2) 255 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:46.778 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:32:46.807 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:46.827 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:32:46.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:46.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:32:46.931 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:46.959 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:46.978 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:46.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:47.074 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:47.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:47.108 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:47.128 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:47.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:47.231 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:47.257 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:47.276 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:47.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:47.379 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:47.407 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:47.541 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:47.555 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:47.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:47.658 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:47.686 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.020586+0000 osd.3 (osd.3) 111 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.021657+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.409638+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.410626+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.501089+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.504037+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.505607+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.507346+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.652509+0000 osd.7 (osd.7) 113 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:32:47.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.655373+0000 osd.7 (osd.7) 114 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.776418+0000 osd.4 (osd.4) 103 : cluster [DBG] 4.e scrub starts 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.779973+0000 osd.4 (osd.4) 104 : cluster [DBG] 4.e scrub ok 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.884604+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.b scrub starts 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.885676+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.b scrub ok 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.000435+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]': finished 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.000582+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi087-82019-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.000661+0000 mon.a (mon.0) 1596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.000763+0000 mon.a (mon.0) 1597 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.064733+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.020586+0000 osd.3 (osd.3) 111 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.021657+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.409638+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.410626+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.501089+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.504037+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.505607+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.507346+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.652509+0000 osd.7 (osd.7) 113 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.655373+0000 osd.7 (osd.7) 114 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.776418+0000 osd.4 (osd.4) 103 : cluster [DBG] 4.e scrub starts 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.779973+0000 osd.4 (osd.4) 104 : cluster [DBG] 4.e scrub ok 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.884604+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.b scrub starts 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.885676+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.b scrub ok 2024-02-23T00:32:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.000435+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]': finished 2024-02-23T00:32:47.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.000582+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi087-82019-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:47.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.000661+0000 mon.a (mon.0) 1596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:47.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.000763+0000 mon.a (mon.0) 1597 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-02-23T00:32:47.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.064733+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.065076+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.066101+0000 mon.b (mon.1) 192 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.066458+0000 mon.a (mon.0) 1599 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.066914+0000 mon.b (mon.1) 193 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:47.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.067104+0000 mon.a (mon.0) 1600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.067133+0000 mon.b (mon.1) 194 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi087-82011-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.067519+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi087-82011-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:47.445971+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.065076+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.066101+0000 mon.b (mon.1) 192 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.066458+0000 mon.a (mon.0) 1599 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.066914+0000 mon.b (mon.1) 193 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.067104+0000 mon.a (mon.0) 1600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.067133+0000 mon.b (mon.1) 194 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi087-82011-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.067519+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi087-82011-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:47.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:47.445971+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:47.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:47.838 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:32:47.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:32:47.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.020586+0000 osd.3 (osd.3) 111 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.021657+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.409638+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.410626+0000 osd.5 (osd.5) 102 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.501089+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.504037+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.505607+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.507346+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.652509+0000 osd.7 (osd.7) 113 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.655373+0000 osd.7 (osd.7) 114 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.776418+0000 osd.4 (osd.4) 103 : cluster [DBG] 4.e scrub starts 2024-02-23T00:32:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.779973+0000 osd.4 (osd.4) 104 : cluster [DBG] 4.e scrub ok 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.884604+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.b scrub starts 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.885676+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.b scrub ok 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.000435+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi087-82011-31"}]': finished 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.000582+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi087-82019-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.000661+0000 mon.a (mon.0) 1596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.000763+0000 mon.a (mon.0) 1597 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.064733+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.065076+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.066101+0000 mon.b (mon.1) 192 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.066458+0000 mon.a (mon.0) 1599 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.066914+0000 mon.b (mon.1) 193 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.067104+0000 mon.a (mon.0) 1600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.067133+0000 mon.b (mon.1) 194 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi087-82011-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:47.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.067519+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi087-82011-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:47.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:47.445971+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:47.966 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:48.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.975434+0000 osd.3 (osd.3) 113 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:32:48.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:46.978399+0000 osd.3 (osd.3) 114 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:32:48.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.975434+0000 osd.3 (osd.3) 113 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:32:48.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:46.978399+0000 osd.3 (osd.3) 114 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:32:48.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.374831+0000 osd.5 (osd.5) 103 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:32:48.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.376687+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:32:48.593 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:48.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:48.697 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:48.727 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.432768+0000 osd.1 (osd.1) 111 : cluster [DBG] 19.c scrub starts 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.435760+0000 osd.1 (osd.1) 112 : cluster [DBG] 19.c scrub ok 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.481405+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.484282+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.529441+0000 osd.0 (osd.0) 131 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.532912+0000 osd.0 (osd.0) 132 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.589786+0000 mgr.y (mgr.24437) 251 : cluster [DBG] pgmap v307: 286 pgs: 64 unknown, 201 active+clean, 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.737094+0000 osd.4 (osd.4) 105 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:47.737868+0000 osd.4 (osd.4) 106 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:32:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:48.015939+0000 mon.a (mon.0) 1602 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:48.016031+0000 mon.a (mon.0) 1603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi087-82011-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:48.016136+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:48.018537+0000 mon.b (mon.1) 195 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi087-82011-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:48.019406+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:48.020465+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi087-82011-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:48.020985+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:48.446776+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.374831+0000 osd.5 (osd.5) 103 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.376687+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:32:48.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.432768+0000 osd.1 (osd.1) 111 : cluster [DBG] 19.c scrub starts 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.435760+0000 osd.1 (osd.1) 112 : cluster [DBG] 19.c scrub ok 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.481405+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.484282+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.529441+0000 osd.0 (osd.0) 131 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.532912+0000 osd.0 (osd.0) 132 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.589786+0000 mgr.y (mgr.24437) 251 : cluster [DBG] pgmap v307: 286 pgs: 64 unknown, 201 active+clean, 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.737094+0000 osd.4 (osd.4) 105 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:47.737868+0000 osd.4 (osd.4) 106 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:48.015939+0000 mon.a (mon.0) 1602 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:48.016031+0000 mon.a (mon.0) 1603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi087-82011-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:48.016136+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:48.018537+0000 mon.b (mon.1) 195 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi087-82011-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:48.019406+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:48.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:48.020465+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi087-82011-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:48.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:48.020985+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:48.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:48.446776+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.975434+0000 osd.3 (osd.3) 113 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:32:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:46.978399+0000 osd.3 (osd.3) 114 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:32:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.374831+0000 osd.5 (osd.5) 103 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:32:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.376687+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:32:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.432768+0000 osd.1 (osd.1) 111 : cluster [DBG] 19.c scrub starts 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.435760+0000 osd.1 (osd.1) 112 : cluster [DBG] 19.c scrub ok 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.481405+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.484282+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.529441+0000 osd.0 (osd.0) 131 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.532912+0000 osd.0 (osd.0) 132 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.589786+0000 mgr.y (mgr.24437) 251 : cluster [DBG] pgmap v307: 286 pgs: 64 unknown, 201 active+clean, 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.737094+0000 osd.4 (osd.4) 105 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:47.737868+0000 osd.4 (osd.4) 106 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:48.015939+0000 mon.a (mon.0) 1602 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:48.016031+0000 mon.a (mon.0) 1603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi087-82011-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:48.016136+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:48.018537+0000 mon.b (mon.1) 195 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi087-82011-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:48.019406+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:48.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:48.020465+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi087-82011-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:48.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:48.020985+0000 mon.a (mon.0) 1606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:48.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:48.446776+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:49.678 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:49.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:49.781 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:49.811 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:48.465186+0000 osd.1 (osd.1) 113 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:32:49.811 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:48.468154+0000 osd.1 (osd.1) 114 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:32:49.811 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:48.529496+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:32:49.811 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:48.536278+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:32:49.811 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:48.840110+0000 osd.2 (osd.2) 101 : cluster [DBG] 63.6 scrub starts 2024-02-23T00:32:49.811 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:48.841841+0000 osd.2 (osd.2) 102 : cluster [DBG] 63.6 scrub ok 2024-02-23T00:32:49.812 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:49.262005+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-21"}]': finished 2024-02-23T00:32:49.812 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:49.262144+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-02-23T00:32:49.812 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:49.263015+0000 mon.b (mon.1) 197 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-21", "mode": "writeback"}]: dispatch 2024-02-23T00:32:49.812 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:49.264496+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-21", "mode": "writeback"}]: dispatch 2024-02-23T00:32:49.812 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:49.264994+0000 mon.a (mon.0) 1610 : audit [INF] from='client.? 172.21.15.87:0/249967235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:49.812 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:49.447547+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:49.813 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:49.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:48.465186+0000 osd.1 (osd.1) 113 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:48.468154+0000 osd.1 (osd.1) 114 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:48.529496+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:48.536278+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:48.840110+0000 osd.2 (osd.2) 101 : cluster [DBG] 63.6 scrub starts 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:48.841841+0000 osd.2 (osd.2) 102 : cluster [DBG] 63.6 scrub ok 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:49.262005+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-21"}]': finished 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:49.262144+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:49.263015+0000 mon.b (mon.1) 197 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-21", "mode": "writeback"}]: dispatch 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:49.264496+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-21", "mode": "writeback"}]: dispatch 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:49.264994+0000 mon.a (mon.0) 1610 : audit [INF] from='client.? 172.21.15.87:0/249967235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:49.447547+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:48.465186+0000 osd.1 (osd.1) 113 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:48.468154+0000 osd.1 (osd.1) 114 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:48.529496+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:48.536278+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:48.840110+0000 osd.2 (osd.2) 101 : cluster [DBG] 63.6 scrub starts 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:48.841841+0000 osd.2 (osd.2) 102 : cluster [DBG] 63.6 scrub ok 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:49.262005+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-21"}]': finished 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:49.262144+0000 mon.a (mon.0) 1608 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:49.263015+0000 mon.b (mon.1) 197 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-21", "mode": "writeback"}]: dispatch 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:49.264496+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-21", "mode": "writeback"}]: dispatch 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:49.264994+0000 mon.a (mon.0) 1610 : audit [INF] from='client.? 172.21.15.87:0/249967235' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:49.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:49.447547+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:49.835 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:49.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:49.938 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:49.966 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:50.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:49.590444+0000 mgr.y (mgr.24437) 252 : cluster [DBG] pgmap v310: 285 pgs: 64 unknown, 200 active+clean, 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:50.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:49.591394+0000 mon.a (mon.0) 1611 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-23T00:32:50.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.015699+0000 mon.a (mon.0) 1612 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:50.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:50.021750+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi087-82011-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi087-82011-32"}]': finished 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:50.021945+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-21", "mode": "writeback"}]': finished 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:50.022022+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? 172.21.15.87:0/249967235' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:50.022088+0000 mon.a (mon.0) 1616 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.022146+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:50.087174+0000 mon.b (mon.1) 198 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:50.311364+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:50.448290+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:49.590444+0000 mgr.y (mgr.24437) 252 : cluster [DBG] pgmap v310: 285 pgs: 64 unknown, 200 active+clean, 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:49.591394+0000 mon.a (mon.0) 1611 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-23T00:32:50.718 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.015699+0000 mon.a (mon.0) 1612 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:50.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:50.021750+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi087-82011-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi087-82011-32"}]': finished 2024-02-23T00:32:50.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:50.021945+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-21", "mode": "writeback"}]': finished 2024-02-23T00:32:50.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:50.022022+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? 172.21.15.87:0/249967235' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:50.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:50.022088+0000 mon.a (mon.0) 1616 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-23T00:32:50.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.022146+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-02-23T00:32:50.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:50.087174+0000 mon.b (mon.1) 198 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:50.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:50.311364+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:50.719 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:50.448290+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:50.719 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:50.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:50.823 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:50.852 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:50.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 -- 172.21.15.87:0/2768086449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fecd00f8420 msgr2=0x7fecd0104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/2768086449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fecd00f8420 0x7fecd0104a20 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 rx=0x7fecc8009c30 tx=0x7fecc8004c50).stop 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 -- 172.21.15.87:0/2768086449 shutdown_connections 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/2768086449 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fecd00f8420 0x7fecd0104a20 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/2768086449 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecd00fdbb0 0x7fecd00fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/2768086449 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fecd00fe510 0x7fecd00f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 -- 172.21.15.87:0/2768086449 >> 172.21.15.87:0/2768086449 conn(0x7fecd00f3800 msgr2=0x7fecd00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 -- 172.21.15.87:0/2768086449 shutdown_connections 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 -- 172.21.15.87:0/2768086449 wait complete. 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 librados: starting msgr at 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 librados: starting objecter 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.934+0000 7fecd5fdb700 1 Processor -- start 2024-02-23T00:32:50.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 -- start start 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 librados: setting wanted keys 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 librados: calling monclient init 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fecd00f8420 0x7fecd010de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fecd00fdbb0 0x7fecd010e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecd00fe510 0x7fecd0108020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd0108510 con 0x7fecd00fdbb0 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd0108680 con 0x7fecd00fe510 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd0108840 con 0x7fecd00f8420 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fecd00fdbb0 0x7fecd010e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7feccf7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fecd00f8420 0x7fecd010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecceffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fecd00fdbb0 0x7fecd010e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34962/0 (socket says 172.21.15.87:34962) 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7feccf7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fecd00f8420 0x7fecd010de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39890/0 (socket says 172.21.15.87:39890) 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecceffd700 1 -- 172.21.15.87:0/1336638464 learned_addr learned my addr 172.21.15.87:0/1336638464 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:50.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7feccffff700 1 --2- 172.21.15.87:0/1336638464 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecd00fe510 0x7fecd0108020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecceffd700 1 -- 172.21.15.87:0/1336638464 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fecd00f8420 msgr2=0x7fecd010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecceffd700 1 --2- 172.21.15.87:0/1336638464 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fecd00f8420 0x7fecd010de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecceffd700 1 -- 172.21.15.87:0/1336638464 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecd00fe510 msgr2=0x7fecd0108020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecceffd700 1 --2- 172.21.15.87:0/1336638464 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fecd00fe510 0x7fecd0108020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecceffd700 1 -- 172.21.15.87:0/1336638464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc4005040 con 0x7fecd00fdbb0 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7feccf7fe700 1 --2- 172.21.15.87:0/1336638464 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fecd00f8420 0x7fecd010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecceffd700 1 --2- 172.21.15.87:0/1336638464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fecd00fdbb0 0x7fecd010e370 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 rx=0x7fecc4009400 tx=0x7fecc4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecccff9700 10 client.?.objecter ms_handle_connect 0x7fecd00fdbb0 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecccff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4007a40 con 0x7fecd00fdbb0 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc80097c0 con 0x7fecd00fdbb0 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecc400f040 con 0x7fecd00fdbb0 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4014a70 con 0x7fecd00fdbb0 2024-02-23T00:32:50.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 10 client.26679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd0114cb0 con 0x7fecd00fdbb0 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 librados: init done 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.936+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fecd005ee80 con 0x7fecd00fdbb0 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.938+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fecc4010530 con 0x7fecd00fdbb0 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6800/1231597463 conn(0x7fecb8062c90 0x7fecb8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(219..219 src has 38..219) v4 ==== 9393+0+0 (secure 0 0 0) 0x7fecc407d310 con 0x7fecd00fdbb0 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 osd_map(219..219 src has 38..219) v4 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 3 client.26679.objecter handle_osd_map got epochs [219,219] > 0 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 3 client.26679.objecter handle_osd_map decoding full epoch 219 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7feccf7fe700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6800/1231597463 conn(0x7fecb8062c90 0x7fecb8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 20 client.26679.objecter dump_active .. 0 homeless 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v219) v1 ==== 54+0+16 (secure 0 0 0) 0x7fecc4084070 con 0x7fecd00fdbb0 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 10 client.26679.objecter ms_handle_connect 0x7fecb8062c90 2024-02-23T00:32:50.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7feccf7fe700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6800/1231597463 conn(0x7fecb8062c90 0x7fecb8065150 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7fecc00103c0 tx=0x7fecc000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 1 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6801/2153191863 conn(0x7fecd00fee60 0x7fecd00ff240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0107110 osd=0 3 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0107110 osd=0 3 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0107110 osd=0 3 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 1 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0107110 osd=0 3 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 0 1 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0107110 osd=0 4 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 10 client.26679.objecter _send_command 1 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecd0109500 con 0x7fecd00fee60 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7feccffff700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6801/2153191863 conn(0x7fecd00fee60 0x7fecd00ff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 10 client.26679.objecter ms_handle_connect 0x7fecd00fee60 2024-02-23T00:32:50.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7feccffff700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6801/2153191863 conn(0x7fecd00fee60 0x7fecd00ff240 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:50.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fecd0109500 con 0x7fecd00fee60 2024-02-23T00:32:50.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:32:50.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 10 client.26679.objecter _finish_command 1 = system:0 2024-02-23T00:32:50.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0107110 osd=0 4 2024-02-23T00:32:50.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.939+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 0 1 2024-02-23T00:32:50.947 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:32:50.947 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:49.590444+0000 mgr.y (mgr.24437) 252 : cluster [DBG] pgmap v310: 285 pgs: 64 unknown, 200 active+clean, 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:49.591394+0000 mon.a (mon.0) 1611 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.015699+0000 mon.a (mon.0) 1612 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:50.021750+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi087-82011-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi087-82011-32"}]': finished 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:50.021945+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-21", "mode": "writeback"}]': finished 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:50.022022+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? 172.21.15.87:0/249967235' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:50.022088+0000 mon.a (mon.0) 1616 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.022146+0000 mon.a (mon.0) 1617 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:50.087174+0000 mon.b (mon.1) 198 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:50.311364+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:32:50.948 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:50.448290+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:50.953 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:50.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 2 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0107110 osd=0 3 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0107110 osd=0 3 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0107110 osd=0 3 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 2 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0107110 osd=0 3 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 0 2 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0107110 osd=0 4 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 10 client.26679.objecter _send_command 2 2024-02-23T00:32:50.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.955+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecd00273c0 con 0x7fecd00fee60 2024-02-23T00:32:50.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fecd00273c0 con 0x7fecd00fee60 2024-02-23T00:32:50.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:50.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecccff9700 10 client.26679.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:50.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0107110 osd=0 4 2024-02-23T00:32:50.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 0 2 2024-02-23T00:32:50.958 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:32:50.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:50.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 3 2024-02-23T00:32:50.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6805/685354171 conn(0x7fecd00598f0 0x7fecd0059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:50.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0028710 osd=1 3 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0028710 osd=1 3 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0028710 osd=1 3 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 3 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0028710 osd=1 3 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 1 3 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0028710 osd=1 4 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 10 client.26679.objecter _send_command 3 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecd011b5c0 con 0x7fecd00598f0 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7feccf7fe700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6805/685354171 conn(0x7fecd00598f0 0x7fecd0059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7fecccff9700 10 client.26679.objecter ms_handle_connect 0x7fecd00598f0 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.956+0000 7feccf7fe700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6805/685354171 conn(0x7fecd00598f0 0x7fecd0059cf0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.958+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fecd011b5c0 con 0x7fecd00598f0 2024-02-23T00:32:50.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.958+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:32:50.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.958+0000 7fecccff9700 10 client.26679.objecter _finish_command 3 = system:0 2024-02-23T00:32:50.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.958+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0028710 osd=1 4 2024-02-23T00:32:50.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.958+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 1 3 2024-02-23T00:32:50.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:50.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 4 2024-02-23T00:32:50.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0028710 osd=1 3 2024-02-23T00:32:50.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0028710 osd=1 3 2024-02-23T00:32:50.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0028710 osd=1 3 2024-02-23T00:32:50.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 4 2024-02-23T00:32:50.974 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0028710 osd=1 3 2024-02-23T00:32:50.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 1 4 2024-02-23T00:32:50.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0028710 osd=1 4 2024-02-23T00:32:50.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 10 client.26679.objecter _send_command 4 2024-02-23T00:32:50.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.973+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecd0059360 con 0x7fecd00598f0 2024-02-23T00:32:50.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fecd0059360 con 0x7fecd00598f0 2024-02-23T00:32:50.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:50.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecccff9700 10 client.26679.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:50.975 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0028710 osd=1 4 2024-02-23T00:32:50.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 1 4 2024-02-23T00:32:50.976 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:32:50.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 5 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6809/3605394228 conn(0x7fecd0119fe0 0x7fecd011a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0027d60 osd=2 3 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0027d60 osd=2 3 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0027d60 osd=2 3 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 5 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0027d60 osd=2 3 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 2 5 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0027d60 osd=2 4 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 10 client.26679.objecter _send_command 5 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecd00283e0 con 0x7fecd0119fe0 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.974+0000 7fecceffd700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6809/3605394228 conn(0x7fecd0119fe0 0x7fecd011a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:50.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.975+0000 7fecccff9700 10 client.26679.objecter ms_handle_connect 0x7fecd0119fe0 2024-02-23T00:32:50.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.975+0000 7fecceffd700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6809/3605394228 conn(0x7fecd0119fe0 0x7fecd011a400 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:50.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.975+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fecd00283e0 con 0x7fecd0119fe0 2024-02-23T00:32:50.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.975+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:32:50.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.975+0000 7fecccff9700 10 client.26679.objecter _finish_command 5 = system:0 2024-02-23T00:32:50.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.975+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0027d60 osd=2 4 2024-02-23T00:32:50.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.975+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 2 5 2024-02-23T00:32:50.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 6 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0027d60 osd=2 3 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0027d60 osd=2 3 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0027d60 osd=2 3 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 6 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0027d60 osd=2 3 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 2 6 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0027d60 osd=2 4 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 10 client.26679.objecter _send_command 6 2024-02-23T00:32:50.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.989+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecd011b0c0 con 0x7fecd0119fe0 2024-02-23T00:32:50.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fecd011b0c0 con 0x7fecd0119fe0 2024-02-23T00:32:50.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:50.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecccff9700 10 client.26679.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:50.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0027d60 osd=2 4 2024-02-23T00:32:50.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 2 6 2024-02-23T00:32:50.992 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:32:50.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:50.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 7 2024-02-23T00:32:50.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6813/128059790 conn(0x7fecd0027940 0x7fecd011fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:50.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd00277d0 osd=3 3 2024-02-23T00:32:50.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd00277d0 osd=3 3 2024-02-23T00:32:50.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd00277d0 osd=3 3 2024-02-23T00:32:50.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 7 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd00277d0 osd=3 3 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 3 7 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd00277d0 osd=3 4 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 10 client.26679.objecter _send_command 7 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecd01228b0 con 0x7fecd0027940 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7feccffff700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6813/128059790 conn(0x7fecd0027940 0x7fecd011fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7fecccff9700 10 client.26679.objecter ms_handle_connect 0x7fecd0027940 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.991+0000 7feccffff700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6813/128059790 conn(0x7fecd0027940 0x7fecd011fc60 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.992+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fecd01228b0 con 0x7fecd0027940 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.992+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.992+0000 7fecccff9700 10 client.26679.objecter _finish_command 7 = system:0 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.992+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd00277d0 osd=3 4 2024-02-23T00:32:50.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:50.992+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 3 7 2024-02-23T00:32:51.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 8 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd00277d0 osd=3 3 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd00277d0 osd=3 3 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd00277d0 osd=3 3 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 8 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd00277d0 osd=3 3 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 3 8 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd00277d0 osd=3 4 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 10 client.26679.objecter _send_command 8 2024-02-23T00:32:51.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.007+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecd011ab30 con 0x7fecd0027940 2024-02-23T00:32:51.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fecd011ab30 con 0x7fecd0027940 2024-02-23T00:32:51.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:51.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecccff9700 10 client.26679.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd00277d0 osd=3 4 2024-02-23T00:32:51.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 3 8 2024-02-23T00:32:51.010 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:32:51.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:51.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 9 2024-02-23T00:32:51.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6800/4175844757 conn(0x7fecd011aca0 0x7fecd01212d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd011ab30 osd=4 3 2024-02-23T00:32:51.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd011ab30 osd=4 3 2024-02-23T00:32:51.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd011ab30 osd=4 3 2024-02-23T00:32:51.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 9 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd011ab30 osd=4 3 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 4 9 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd011ab30 osd=4 4 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 10 client.26679.objecter _send_command 9 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecd0119b40 con 0x7fecd011aca0 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.009+0000 7feccf7fe700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6800/4175844757 conn(0x7fecd011aca0 0x7fecd01212d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.010+0000 7fecccff9700 10 client.26679.objecter ms_handle_connect 0x7fecd011aca0 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.010+0000 7feccf7fe700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6800/4175844757 conn(0x7fecd011aca0 0x7fecd01212d0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.011+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fecd0119b40 con 0x7fecd011aca0 2024-02-23T00:32:51.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.011+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:32:51.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.011+0000 7fecccff9700 10 client.26679.objecter _finish_command 9 = system:0 2024-02-23T00:32:51.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.011+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd011ab30 osd=4 4 2024-02-23T00:32:51.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.011+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 4 9 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 10 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd011ab30 osd=4 3 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd011ab30 osd=4 3 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd011ab30 osd=4 3 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 10 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd011ab30 osd=4 3 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 4 10 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd011ab30 osd=4 4 2024-02-23T00:32:51.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 10 client.26679.objecter _send_command 10 2024-02-23T00:32:51.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.031+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecd0119b40 con 0x7fecd011aca0 2024-02-23T00:32:51.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.041+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fecd0119b40 con 0x7fecd011aca0 2024-02-23T00:32:51.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.041+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:51.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.041+0000 7fecccff9700 10 client.26679.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.041+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd011ab30 osd=4 4 2024-02-23T00:32:51.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.041+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 4 10 2024-02-23T00:32:51.046 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 11 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6804/2550013795 conn(0x7fecd0121cd0 0x7fecd0122110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0119b40 osd=5 3 2024-02-23T00:32:51.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0119b40 osd=5 3 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0119b40 osd=5 3 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 11 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0119b40 osd=5 3 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 5 11 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0119b40 osd=5 4 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 10 client.26679.objecter _send_command 11 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecd0122600 con 0x7fecd0121cd0 2024-02-23T00:32:51.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.046+0000 7fecceffd700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6804/2550013795 conn(0x7fecd0121cd0 0x7fecd0122110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.047+0000 7fecccff9700 10 client.26679.objecter ms_handle_connect 0x7fecd0121cd0 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.047+0000 7fecceffd700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6804/2550013795 conn(0x7fecd0121cd0 0x7fecd0122110 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.048+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fecd0122600 con 0x7fecd0121cd0 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.048+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.048+0000 7fecccff9700 10 client.26679.objecter _finish_command 11 = system:0 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.048+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0119b40 osd=5 4 2024-02-23T00:32:51.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.048+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 5 11 2024-02-23T00:32:51.057 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:51.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:51.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 12 2024-02-23T00:32:51.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0119b40 osd=5 3 2024-02-23T00:32:51.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0119b40 osd=5 3 2024-02-23T00:32:51.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0119b40 osd=5 3 2024-02-23T00:32:51.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 12 2024-02-23T00:32:51.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0119b40 osd=5 3 2024-02-23T00:32:51.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 5 12 2024-02-23T00:32:51.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0119b40 osd=5 4 2024-02-23T00:32:51.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 10 client.26679.objecter _send_command 12 2024-02-23T00:32:51.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.062+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecd0122600 con 0x7fecd0121cd0 2024-02-23T00:32:51.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fecd0122600 con 0x7fecd0121cd0 2024-02-23T00:32:51.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:51.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecccff9700 10 client.26679.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0119b40 osd=5 4 2024-02-23T00:32:51.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 5 12 2024-02-23T00:32:51.066 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:32:51.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 13 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6808/2509408498 conn(0x7fecd006b420 0x7fecd0128a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0122600 osd=6 3 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0122600 osd=6 3 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0122600 osd=6 3 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 13 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0122600 osd=6 3 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 6 13 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0122600 osd=6 4 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 10 client.26679.objecter _send_command 13 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.064+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecd0128fb0 con 0x7fecd006b420 2024-02-23T00:32:51.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.065+0000 7feccffff700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6808/2509408498 conn(0x7fecd006b420 0x7fecd0128a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.065+0000 7fecccff9700 10 client.26679.objecter ms_handle_connect 0x7fecd006b420 2024-02-23T00:32:51.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.065+0000 7feccffff700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6808/2509408498 conn(0x7fecd006b420 0x7fecd0128a50 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:51.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.066+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fecd0128fb0 con 0x7fecd006b420 2024-02-23T00:32:51.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.066+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:32:51.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.066+0000 7fecccff9700 10 client.26679.objecter _finish_command 13 = system:0 2024-02-23T00:32:51.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.066+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0122600 osd=6 4 2024-02-23T00:32:51.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.066+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 6 13 2024-02-23T00:32:51.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.081+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.081+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 14 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.082+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0122600 osd=6 3 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.082+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0122600 osd=6 3 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.082+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0122600 osd=6 3 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.082+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 14 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.082+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0122600 osd=6 3 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.082+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 6 14 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.082+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0122600 osd=6 4 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.082+0000 7fecd5fdb700 10 client.26679.objecter _send_command 14 2024-02-23T00:32:51.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.082+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecd0128fb0 con 0x7fecd006b420 2024-02-23T00:32:51.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.083+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fecd0128fb0 con 0x7fecd006b420 2024-02-23T00:32:51.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.083+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:51.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.083+0000 7fecccff9700 10 client.26679.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.083+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0122600 osd=6 4 2024-02-23T00:32:51.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.083+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 6 14 2024-02-23T00:32:51.085 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:32:51.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.083+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:32:51.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.083+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 15 2024-02-23T00:32:51.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6812/373787000 conn(0x7fecd006bd90 0x7fecd006c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0128fb0 osd=7 3 2024-02-23T00:32:51.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0128fb0 osd=7 3 2024-02-23T00:32:51.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0128fb0 osd=7 3 2024-02-23T00:32:51.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 15 2024-02-23T00:32:51.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0128fb0 osd=7 3 2024-02-23T00:32:51.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 7 15 2024-02-23T00:32:51.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0128fb0 osd=7 4 2024-02-23T00:32:51.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 10 client.26679.objecter _send_command 15 2024-02-23T00:32:51.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecd006c700 con 0x7fecd006bd90 2024-02-23T00:32:51.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7feccf7fe700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6812/373787000 conn(0x7fecd006bd90 0x7fecd006c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.086 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:51.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7fecccff9700 10 client.26679.objecter ms_handle_connect 0x7fecd006bd90 2024-02-23T00:32:51.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.084+0000 7feccf7fe700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6812/373787000 conn(0x7fecd006bd90 0x7fecd006c210 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:51.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.085+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fecd006c700 con 0x7fecd006bd90 2024-02-23T00:32:51.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.085+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:32:51.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.085+0000 7fecccff9700 10 client.26679.objecter _finish_command 15 = system:0 2024-02-23T00:32:51.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.085+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0128fb0 osd=7 4 2024-02-23T00:32:51.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.085+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 7 15 2024-02-23T00:32:51.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 10 client.26679.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:32:51.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign -1 16 2024-02-23T00:32:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0128fb0 osd=7 3 2024-02-23T00:32:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0128fb0 osd=7 3 2024-02-23T00:32:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 20 client.26679.objecter _get_session s=0x7fecd0128fb0 osd=7 3 2024-02-23T00:32:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_remove -1 16 2024-02-23T00:32:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 20 client.26679.objecter get_session s=0x7fecd0128fb0 osd=7 3 2024-02-23T00:32:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 15 client.26679.objecter _session_command_op_assign 7 16 2024-02-23T00:32:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0128fb0 osd=7 4 2024-02-23T00:32:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 10 client.26679.objecter _send_command 16 2024-02-23T00:32:51.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.100+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fecd006c700 con 0x7fecd006bd90 2024-02-23T00:32:51.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.102+0000 7fecccff9700 1 -- 172.21.15.87:0/1336638464 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fecd006c700 con 0x7fecd006bd90 2024-02-23T00:32:51.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.102+0000 7fecccff9700 10 client.26679.objecter ms_dispatch 0x7fecd0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:32:51.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.102+0000 7fecccff9700 10 client.26679.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.102+0000 7fecccff9700 20 client.26679.objecter put_session s=0x7fecd0128fb0 osd=7 4 2024-02-23T00:32:51.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.102+0000 7fecccff9700 15 client.26679.objecter _session_command_op_remove 7 16 2024-02-23T00:32:51.104 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 librados: watch_flush enter 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 librados: watch_flush exit 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 client.26679.objecter close_session for osd.0 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6801/2153191863 conn(0x7fecd00fee60 msgr2=0x7fecd00ff240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6801/2153191863 conn(0x7fecd00fee60 0x7fecd00ff240 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0107110 osd=0 1 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 client.26679.objecter close_session for osd.1 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6805/685354171 conn(0x7fecd00598f0 msgr2=0x7fecd0059cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6805/685354171 conn(0x7fecd00598f0 0x7fecd0059cf0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0028710 osd=1 1 2024-02-23T00:32:51.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 client.26679.objecter close_session for osd.2 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6809/3605394228 conn(0x7fecd0119fe0 msgr2=0x7fecd011a400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6809/3605394228 conn(0x7fecd0119fe0 0x7fecd011a400 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0027d60 osd=2 1 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 client.26679.objecter close_session for osd.3 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6813/128059790 conn(0x7fecd0027940 msgr2=0x7fecd011fc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6813/128059790 conn(0x7fecd0027940 0x7fecd011fc60 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd00277d0 osd=3 1 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7feccffff700 1 -- 172.21.15.87:0/1336638464 reap_dead start 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 client.26679.objecter close_session for osd.4 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6800/4175844757 conn(0x7fecd011aca0 msgr2=0x7fecd01212d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6800/4175844757 conn(0x7fecd011aca0 0x7fecd01212d0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7feccffff700 1 -- 172.21.15.87:0/1336638464 reap_dead start 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd011ab30 osd=4 1 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 client.26679.objecter close_session for osd.5 2024-02-23T00:32:51.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6804/2550013795 conn(0x7fecd0121cd0 msgr2=0x7fecd0122110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6804/2550013795 conn(0x7fecd0121cd0 0x7fecd0122110 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0119b40 osd=5 1 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 client.26679.objecter close_session for osd.6 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6808/2509408498 conn(0x7fecd006b420 msgr2=0x7fecd0128a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6808/2509408498 conn(0x7fecd006b420 0x7fecd0128a50 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0122600 osd=6 1 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 client.26679.objecter close_session for osd.7 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6812/373787000 conn(0x7fecd006bd90 msgr2=0x7fecd006c210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.171:6812/373787000 conn(0x7fecd006bd90 0x7fecd006c210 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 20 client.26679.objecter put_session s=0x7fecd0128fb0 osd=7 1 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 20 client.26679.objecter shutdown clearing up homeless session... 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 10 client.26679.objecter successfully canceled tick 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6800/1231597463 conn(0x7fecb8062c90 msgr2=0x7fecb8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> v2:172.21.15.87:6800/1231597463 conn(0x7fecb8062c90 0x7fecb8065150 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 rx=0x7fecc00103c0 tx=0x7fecc000b040).stop 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fecd00fdbb0 msgr2=0x7fecd010e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 --2- 172.21.15.87:0/1336638464 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fecd00fdbb0 0x7fecd010e370 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 rx=0x7fecc4009400 tx=0x7fecc4009cb0).stop 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7feccffff700 1 -- 172.21.15.87:0/1336638464 reap_dead start 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 shutdown_connections 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 >> 172.21.15.87:0/1336638464 conn(0x7fecd00f3800 msgr2=0x7fecd00f5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 shutdown_connections 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 -- 172.21.15.87:0/1336638464 wait complete. 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.103+0000 7fecd5fdb700 1 librados: shutdown 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.108 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:32:51.130 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:32:51.131 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:32:51.131 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-23T00:32:51.131 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-23T00:32:51.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 -- 172.21.15.87:0/240406587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90600f8420 msgr2=0x7f9060104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 --2- 172.21.15.87:0/240406587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90600f8420 0x7f9060104a20 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 rx=0x7f9058009be0 tx=0x7f9058004d10).stop 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 -- 172.21.15.87:0/240406587 shutdown_connections 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 --2- 172.21.15.87:0/240406587 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90600f8420 0x7f9060104a20 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 --2- 172.21.15.87:0/240406587 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f90600fdbb0 0x7f90600fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 --2- 172.21.15.87:0/240406587 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90600fe510 0x7f90600f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 -- 172.21.15.87:0/240406587 >> 172.21.15.87:0/240406587 conn(0x7f90600f3800 msgr2=0x7f90600f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 -- 172.21.15.87:0/240406587 shutdown_connections 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 -- 172.21.15.87:0/240406587 wait complete. 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 librados: starting msgr at 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 librados: starting objecter 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.369+0000 7f9065d41700 1 Processor -- start 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f9065d41700 1 -- start start 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f9065d41700 1 librados: setting wanted keys 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f9065d41700 1 librados: calling monclient init 2024-02-23T00:32:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f9065d41700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f90600f8420 0x7f906010df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f9065d41700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90600fdbb0 0x7f906010e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f9065d41700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90600fe510 0x7f90601080b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f9065d41700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90601085a0 con 0x7f90600fe510 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f9065d41700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9060108710 con 0x7f90600f8420 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f9065d41700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90601088d0 con 0x7f90600fdbb0 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905effd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90600fdbb0 0x7f906010e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905effd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90600fdbb0 0x7f906010e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39464/0 (socket says 172.21.15.87:39464) 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905effd700 1 -- 172.21.15.87:0/631066976 learned_addr learned my addr 172.21.15.87:0/631066976 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905ffff700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90600fe510 0x7f90601080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905f7fe700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f90600f8420 0x7f906010df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905effd700 1 -- 172.21.15.87:0/631066976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f90600f8420 msgr2=0x7f906010df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905effd700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f90600f8420 0x7f906010df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905effd700 1 -- 172.21.15.87:0/631066976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90600fe510 msgr2=0x7f90601080b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905effd700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90600fe510 0x7f90601080b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905effd700 1 -- 172.21.15.87:0/631066976 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580097c0 con 0x7f90600fdbb0 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.370+0000 7f905ffff700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90600fe510 0x7f90601080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f905effd700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90600fdbb0 0x7f906010e400 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7f905400e970 tx=0x7f905400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f905cff9700 10 client.?.objecter ms_handle_connect 0x7f90600fdbb0 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f905cff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f905cff9700 1 -- 172.21.15.87:0/631066976 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054004660 con 0x7f90600fdbb0 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9060108bb0 con 0x7f90600fdbb0 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f905cff9700 1 -- 172.21.15.87:0/631066976 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f905400f040 con 0x7f90600fdbb0 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f905cff9700 1 -- 172.21.15.87:0/631066976 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054014790 con 0x7f90600fdbb0 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f9065d41700 10 client.37133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9060108f30 con 0x7f90600fdbb0 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.371+0000 7f9065d41700 1 librados: init done 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.372+0000 7f905cff9700 1 -- 172.21.15.87:0/631066976 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f90540090d0 con 0x7f90600fdbb0 2024-02-23T00:32:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.372+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90600606d0 con 0x7f90600fdbb0 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.372+0000 7f905cff9700 1 --2- 172.21.15.87:0/631066976 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9048062c90 0x7f9048065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.372+0000 7f905cff9700 1 -- 172.21.15.87:0/631066976 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(220..220 src has 38..220) v4 ==== 9768+0+0 (secure 0 0 0) 0x7f905407d360 con 0x7f90600fdbb0 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.373+0000 7f905cff9700 10 client.37133.objecter ms_dispatch 0x7f9060001550 osd_map(220..220 src has 38..220) v4 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.373+0000 7f905cff9700 3 client.37133.objecter handle_osd_map got epochs [220,220] > 0 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.373+0000 7f905cff9700 3 client.37133.objecter handle_osd_map decoding full epoch 220 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.373+0000 7f905f7fe700 1 --2- 172.21.15.87:0/631066976 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9048062c90 0x7f9048065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.373+0000 7f905cff9700 20 client.37133.objecter dump_active .. 0 homeless 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.373+0000 7f905cff9700 10 client.37133.objecter ms_handle_connect 0x7f9048062c90 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.373+0000 7f905f7fe700 1 --2- 172.21.15.87:0/631066976 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9048062c90 0x7f9048065150 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 rx=0x7f9050004dc0 tx=0x7f905000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.377+0000 7f905cff9700 1 -- 172.21.15.87:0/631066976 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9054050230 con 0x7f90600fdbb0 2024-02-23T00:32:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:51.556+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f90601071b0 con 0x7f90600fdbb0 2024-02-23T00:32:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.444958+0000 osd.1 (osd.1) 115 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:32:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.448586+0000 osd.1 (osd.1) 116 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:32:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.595951+0000 osd.0 (osd.0) 135 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:32:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.596852+0000 osd.0 (osd.0) 136 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:32:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.704205+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:32:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.705309+0000 osd.7 (osd.7) 115 : cluster [DBG] 6.1c scrub starts 2024-02-23T00:32:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.705310+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:32:51.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:50.706688+0000 osd.7 (osd.7) 116 : cluster [DBG] 6.1c scrub ok 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:50.728795+0000 mgr.y (mgr.24437) 253 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:50.939298+0000 mon.a (mon.0) 1619 : audit [DBG] from='client.? 172.21.15.87:0/1336638464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:51.024289+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:51.024362+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:51.030388+0000 mon.c (mon.2) 260 : audit [INF] from='client.? 172.21.15.87:0/1840183281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:51.030845+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:51.033690+0000 mon.b (mon.1) 199 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:51.034490+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:51.449268+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.444958+0000 osd.1 (osd.1) 115 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:32:51.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.448586+0000 osd.1 (osd.1) 116 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.595951+0000 osd.0 (osd.0) 135 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.596852+0000 osd.0 (osd.0) 136 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.704205+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.705309+0000 osd.7 (osd.7) 115 : cluster [DBG] 6.1c scrub starts 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.705310+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:50.706688+0000 osd.7 (osd.7) 116 : cluster [DBG] 6.1c scrub ok 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:50.728795+0000 mgr.y (mgr.24437) 253 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:50.939298+0000 mon.a (mon.0) 1619 : audit [DBG] from='client.? 172.21.15.87:0/1336638464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:51.024289+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:51.024362+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:51.030388+0000 mon.c (mon.2) 260 : audit [INF] from='client.? 172.21.15.87:0/1840183281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:51.030845+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:51.033690+0000 mon.b (mon.1) 199 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:51.034490+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:51.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:51.449268+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.444958+0000 osd.1 (osd.1) 115 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:32:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.448586+0000 osd.1 (osd.1) 116 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:32:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.595951+0000 osd.0 (osd.0) 135 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:32:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.596852+0000 osd.0 (osd.0) 136 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:32:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.704205+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:32:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.705309+0000 osd.7 (osd.7) 115 : cluster [DBG] 6.1c scrub starts 2024-02-23T00:32:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.705310+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:50.706688+0000 osd.7 (osd.7) 116 : cluster [DBG] 6.1c scrub ok 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:50.728795+0000 mgr.y (mgr.24437) 253 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:50.939298+0000 mon.a (mon.0) 1619 : audit [DBG] from='client.? 172.21.15.87:0/1336638464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:51.024289+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:51.024362+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:51.030388+0000 mon.c (mon.2) 260 : audit [INF] from='client.? 172.21.15.87:0/1840183281' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:51.030845+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:51.033690+0000 mon.b (mon.1) 199 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:51.034490+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21"}]: dispatch 2024-02-23T00:32:51.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:51.449268+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:52.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.007+0000 7f905cff9700 1 -- 172.21.15.87:0/631066976 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v221) v1 ==== 101+0+0 (secure 0 0 0) 0x7f9054019030 con 0x7f90600fdbb0 2024-02-23T00:32:52.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.010+0000 7f9065d41700 10 librados: watch_flush enter 2024-02-23T00:32:52.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.010+0000 7f9065d41700 10 librados: watch_flush exit 2024-02-23T00:32:52.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.010+0000 7f9065d41700 20 client.37133.objecter shutdown clearing up homeless session... 2024-02-23T00:32:52.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.010+0000 7f9065d41700 10 client.37133.objecter successfully canceled tick 2024-02-23T00:32:52.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.010+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9048062c90 msgr2=0x7f9048065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:52.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.010+0000 7f9065d41700 1 --2- 172.21.15.87:0/631066976 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9048062c90 0x7f9048065150 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 rx=0x7f9050004dc0 tx=0x7f905000b040).stop 2024-02-23T00:32:52.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.010+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90600fdbb0 msgr2=0x7f906010e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:52.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.010+0000 7f9065d41700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90600fdbb0 0x7f906010e400 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 rx=0x7f905400e970 tx=0x7f905400b040).stop 2024-02-23T00:32:52.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.017+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 shutdown_connections 2024-02-23T00:32:52.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.017+0000 7f9065d41700 1 --2- 172.21.15.87:0/631066976 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9048062c90 0x7f9048065150 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.017+0000 7f9065d41700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f90600fe510 0x7f90601080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.017+0000 7f9065d41700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f90600f8420 0x7f906010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.017+0000 7f9065d41700 1 --2- 172.21.15.87:0/631066976 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f90600fdbb0 0x7f906010e400 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.017+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 >> 172.21.15.87:0/631066976 conn(0x7f90600f3800 msgr2=0x7f90600f5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:52.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.018+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 shutdown_connections 2024-02-23T00:32:52.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.021+0000 7f9065d41700 1 -- 172.21.15.87:0/631066976 wait complete. 2024-02-23T00:32:52.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:52.021+0000 7f9065d41700 1 librados: shutdown 2024-02-23T00:32:52.026 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:32:52.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:52] "GET /metrics HTTP/1.1" 200 187646 "" "Prometheus/2.33.4" 2024-02-23T00:32:52.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:51.557133+0000 mon.c (mon.2) 262 : audit [INF] from='client.? 172.21.15.87:0/631066976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:52.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:51.557403+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:52.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:51.591133+0000 mgr.y (mgr.24437) 254 : cluster [DBG] pgmap v313: 325 pgs: 2 peering, 32 unknown, 8 creating+peering, 263 active+clean, 14 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-23T00:32:52.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:51.817777+0000 osd.2 (osd.2) 103 : cluster [DBG] 19.a scrub starts 2024-02-23T00:32:52.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:51.819396+0000 osd.2 (osd.2) 104 : cluster [DBG] 19.a scrub ok 2024-02-23T00:32:52.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:51.998234+0000 mon.a (mon.0) 1625 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:52.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:51.998261+0000 mon.a (mon.0) 1626 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:32:52.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:52.003121+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:52.003284+0000 mon.a (mon.0) 1628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21"}]': finished 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:52.003432+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:52.003483+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:52.015838+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:52.018232+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:52.450068+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:51.557133+0000 mon.c (mon.2) 262 : audit [INF] from='client.? 172.21.15.87:0/631066976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:51.557403+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:51.591133+0000 mgr.y (mgr.24437) 254 : cluster [DBG] pgmap v313: 325 pgs: 2 peering, 32 unknown, 8 creating+peering, 263 active+clean, 14 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-23T00:32:52.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:51.817777+0000 osd.2 (osd.2) 103 : cluster [DBG] 19.a scrub starts 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:51.819396+0000 osd.2 (osd.2) 104 : cluster [DBG] 19.a scrub ok 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:51.998234+0000 mon.a (mon.0) 1625 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:51.998261+0000 mon.a (mon.0) 1626 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:52.003121+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:52.003284+0000 mon.a (mon.0) 1628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21"}]': finished 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:52.003432+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:52.003483+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:52.015838+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:52.018232+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:52.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:52.450068+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:51.557133+0000 mon.c (mon.2) 262 : audit [INF] from='client.? 172.21.15.87:0/631066976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:51.557403+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:32:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:51.591133+0000 mgr.y (mgr.24437) 254 : cluster [DBG] pgmap v313: 325 pgs: 2 peering, 32 unknown, 8 creating+peering, 263 active+clean, 14 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-02-23T00:32:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:51.817777+0000 osd.2 (osd.2) 103 : cluster [DBG] 19.a scrub starts 2024-02-23T00:32:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:51.819396+0000 osd.2 (osd.2) 104 : cluster [DBG] 19.a scrub ok 2024-02-23T00:32:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:51.998234+0000 mon.a (mon.0) 1625 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:32:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:51.998261+0000 mon.a (mon.0) 1626 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:32:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:52.003121+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:52.003284+0000 mon.a (mon.0) 1628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-21"}]': finished 2024-02-23T00:32:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:52.003432+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:32:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:52.003483+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-02-23T00:32:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:52.015838+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:52.018232+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:52.450068+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:53.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:53.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:53.143 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:53.172 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:53.227 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:53.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:53.330 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:32:53.334 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:53.348 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:32:53.348 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:32:53.361 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:53.382 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:32:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:53.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:32:53.486 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:53.517 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:53.537 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:53.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:53.641 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:53.670 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:53.690 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:53.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:53.794 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:53.823 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:53.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:32:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:32:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:32:53.946 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:53.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:54.050 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:54.078 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:54.098 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:54.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:54.137 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:32:54.202 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:54.229 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:54.249 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:54.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:53.005210+0000 mon.a (mon.0) 1632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]': finished 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:53.005347+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:53.012213+0000 mon.b (mon.1) 201 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:53.013233+0000 mon.c (mon.2) 264 : audit [INF] from='client.? 172.21.15.87:0/3454566960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:53.265758+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:53.265962+0000 mon.a (mon.0) 1635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:53.451299+0000 mon.c (mon.2) 265 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:53.592807+0000 mon.a (mon.0) 1636 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:53.599960+0000 osd.0 (osd.0) 137 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:32:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:53.602875+0000 osd.0 (osd.0) 138 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:53.005210+0000 mon.a (mon.0) 1632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]': finished 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:53.005347+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:53.012213+0000 mon.b (mon.1) 201 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:53.013233+0000 mon.c (mon.2) 264 : audit [INF] from='client.? 172.21.15.87:0/3454566960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:53.265758+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:53.265962+0000 mon.a (mon.0) 1635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:53.451299+0000 mon.c (mon.2) 265 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:53.592807+0000 mon.a (mon.0) 1636 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:53.599960+0000 osd.0 (osd.0) 137 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:32:54.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:53.602875+0000 osd.0 (osd.0) 138 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:32:54.353 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:53.005210+0000 mon.a (mon.0) 1632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi087-82011-32"}]': finished 2024-02-23T00:32:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:53.005347+0000 mon.a (mon.0) 1633 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-02-23T00:32:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:53.012213+0000 mon.b (mon.1) 201 : audit [INF] from='client.? 172.21.15.87:0/843545378' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:53.013233+0000 mon.c (mon.2) 264 : audit [INF] from='client.? 172.21.15.87:0/3454566960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:53.265758+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:53.265962+0000 mon.a (mon.0) 1635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]: dispatch 2024-02-23T00:32:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:53.451299+0000 mon.c (mon.2) 265 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:54.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:53.592807+0000 mon.a (mon.0) 1636 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-23T00:32:54.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:53.599960+0000 osd.0 (osd.0) 137 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:32:54.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:53.602875+0000 osd.0 (osd.0) 138 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:32:54.830 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:54.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:54.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:32:54.933 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:54.989 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:53.384453+0000 osd.1 (osd.1) 117 : cluster [DBG] 22.1c deep-scrub starts 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:53.387491+0000 osd.1 (osd.1) 118 : cluster [DBG] 22.1c deep-scrub ok 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:53.591721+0000 mgr.y (mgr.24437) 255 : cluster [DBG] pgmap v316: 316 pgs: 1 peering, 64 unknown, 231 active+clean, 14 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:53.982207+0000 osd.3 (osd.3) 115 : cluster [DBG] 3.b scrub starts 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:53.985146+0000 osd.3 (osd.3) 116 : cluster [DBG] 3.b scrub ok 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.011646+0000 mon.a (mon.0) 1637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.011756+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]': finished 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.011845+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:54.011901+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-02-23T00:32:55.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.023133+0000 mon.b (mon.1) 202 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:55.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.033933+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:55.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.055430+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:55.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.056175+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:55.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.056459+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi087-82011-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:55.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.452090+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:55.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:54.638052+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:55.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:53.384453+0000 osd.1 (osd.1) 117 : cluster [DBG] 22.1c deep-scrub starts 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:53.387491+0000 osd.1 (osd.1) 118 : cluster [DBG] 22.1c deep-scrub ok 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:53.591721+0000 mgr.y (mgr.24437) 255 : cluster [DBG] pgmap v316: 316 pgs: 1 peering, 64 unknown, 231 active+clean, 14 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:53.982207+0000 osd.3 (osd.3) 115 : cluster [DBG] 3.b scrub starts 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:53.985146+0000 osd.3 (osd.3) 116 : cluster [DBG] 3.b scrub ok 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.011646+0000 mon.a (mon.0) 1637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.011756+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]': finished 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.011845+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:54.011901+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.023133+0000 mon.b (mon.1) 202 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.033933+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.055430+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.056175+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:55.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.056459+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi087-82011-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:55.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.452090+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:55.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:54.638052+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:55.237 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:55.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:55.247 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-23T00:32:55.307 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:55.340 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:55.368 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:53.384453+0000 osd.1 (osd.1) 117 : cluster [DBG] 22.1c deep-scrub starts 2024-02-23T00:32:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:53.387491+0000 osd.1 (osd.1) 118 : cluster [DBG] 22.1c deep-scrub ok 2024-02-23T00:32:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:53.591721+0000 mgr.y (mgr.24437) 255 : cluster [DBG] pgmap v316: 316 pgs: 1 peering, 64 unknown, 231 active+clean, 14 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-23T00:32:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:53.982207+0000 osd.3 (osd.3) 115 : cluster [DBG] 3.b scrub starts 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:53.985146+0000 osd.3 (osd.3) 116 : cluster [DBG] 3.b scrub ok 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.011646+0000 mon.a (mon.0) 1637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi087-82019-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.011756+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi087-82011-32"}]': finished 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.011845+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:54.011901+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.023133+0000 mon.b (mon.1) 202 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.033933+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.055430+0000 mon.a (mon.0) 1642 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.056175+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.056459+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi087-82011-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:32:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.452090+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:55.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:54.638052+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:32:55.616 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:55.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:55.721 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:55.754 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:55.774 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:55.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:55.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:55.879 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:32:55.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.883+0000 7f2d70450700 1 -- 172.21.15.87:0/3733520871 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d680f7400 msgr2=0x7f2d68103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:55.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.883+0000 7f2d70450700 1 --2- 172.21.15.87:0/3733520871 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d680f7400 0x7f2d68103a20 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f2d60009be0 tx=0x7f2d60004d10).stop 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.883+0000 7f2d70450700 1 -- 172.21.15.87:0/3733520871 shutdown_connections 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.883+0000 7f2d70450700 1 --2- 172.21.15.87:0/3733520871 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d680f7400 0x7f2d68103a20 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.883+0000 7f2d70450700 1 --2- 172.21.15.87:0/3733520871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d680fcbb0 0x7f2d680fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.883+0000 7f2d70450700 1 --2- 172.21.15.87:0/3733520871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d680fd510 0x7f2d680f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.883+0000 7f2d70450700 1 -- 172.21.15.87:0/3733520871 >> 172.21.15.87:0/3733520871 conn(0x7f2d680f2820 msgr2=0x7f2d680f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.883+0000 7f2d70450700 1 -- 172.21.15.87:0/3733520871 shutdown_connections 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.883+0000 7f2d70450700 1 -- 172.21.15.87:0/3733520871 wait complete. 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 librados: starting msgr at 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 librados: starting objecter 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 Processor -- start 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 -- start start 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 librados: setting wanted keys 2024-02-23T00:32:55.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 librados: calling monclient init 2024-02-23T00:32:55.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d680f7400 0x7f2d6810e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d680fcbb0 0x7f2d6810e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d680fd510 0x7f2d68108250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68108740 con 0x7f2d680f7400 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d681088b0 con 0x7f2d680fd510 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d70450700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d68108a70 con 0x7f2d680fcbb0 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6e1ec700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d680f7400 0x7f2d6810e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6e1ec700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d680f7400 0x7f2d6810e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58520/0 (socket says 172.21.15.87:58520) 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6e1ec700 1 -- 172.21.15.87:0/734820686 learned_addr learned my addr 172.21.15.87:0/734820686 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6d9eb700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d680fcbb0 0x7f2d6810e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6e9ed700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d680fd510 0x7f2d68108250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6e1ec700 1 -- 172.21.15.87:0/734820686 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d680fcbb0 msgr2=0x7f2d6810e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6e1ec700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d680fcbb0 0x7f2d6810e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:55.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6e1ec700 1 -- 172.21.15.87:0/734820686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d680fd510 msgr2=0x7f2d68108250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:55.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6e1ec700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d680fd510 0x7f2d68108250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:55.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.884+0000 7f2d6e1ec700 1 -- 172.21.15.87:0/734820686 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d64005040 con 0x7f2d680f7400 2024-02-23T00:32:55.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d6e9ed700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d680fd510 0x7f2d68108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:32:55.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d6e1ec700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d680f7400 0x7f2d6810e060 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 rx=0x7f2d64009a00 tx=0x7f2d64009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:55.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d5f7fe700 10 client.?.objecter ms_handle_connect 0x7f2d680f7400 2024-02-23T00:32:55.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d5f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:32:55.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d5f7fe700 1 -- 172.21.15.87:0/734820686 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64007b00 con 0x7f2d680f7400 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d600097c0 con 0x7f2d680f7400 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d70450700 10 client.26703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d681090a0 con 0x7f2d680f7400 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d70450700 1 librados: init done 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d5f7fe700 1 -- 172.21.15.87:0/734820686 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d64017070 con 0x7f2d680f7400 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.885+0000 7f2d5f7fe700 1 -- 172.21.15.87:0/734820686 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6401aa60 con 0x7f2d680f7400 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.887+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6805fa60 con 0x7f2d680f7400 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.887+0000 7f2d5f7fe700 1 -- 172.21.15.87:0/734820686 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f2d640253a0 con 0x7f2d680f7400 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.887+0000 7f2d5f7fe700 1 --2- 172.21.15.87:0/734820686 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d54062c90 0x7f2d54065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.887+0000 7f2d5f7fe700 1 -- 172.21.15.87:0/734820686 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(224..224 src has 38..224) v4 ==== 9276+0+0 (secure 0 0 0) 0x7f2d64081430 con 0x7f2d680f7400 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.887+0000 7f2d5f7fe700 10 client.26703.objecter ms_dispatch 0x7f2d68001550 osd_map(224..224 src has 38..224) v4 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.887+0000 7f2d5f7fe700 3 client.26703.objecter handle_osd_map got epochs [224,224] > 0 2024-02-23T00:32:55.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.887+0000 7f2d5f7fe700 3 client.26703.objecter handle_osd_map decoding full epoch 224 2024-02-23T00:32:55.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.887+0000 7f2d6d9eb700 1 --2- 172.21.15.87:0/734820686 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d54062c90 0x7f2d54065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:32:55.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.888+0000 7f2d5f7fe700 20 client.26703.objecter dump_active .. 0 homeless 2024-02-23T00:32:55.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.888+0000 7f2d5f7fe700 10 client.26703.objecter ms_handle_connect 0x7f2d54062c90 2024-02-23T00:32:55.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.888+0000 7f2d6d9eb700 1 --2- 172.21.15.87:0/734820686 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d54062c90 0x7f2d54065150 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7f2d681099a0 tx=0x7f2d5800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:32:55.891 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:55.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:55.895+0000 7f2d5f7fe700 1 -- 172.21.15.87:0/734820686 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2d640544c0 con 0x7f2d680f7400 2024-02-23T00:32:55.909 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:55.930 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:32:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:55.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:32:55.933 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:32:56.034 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:32:56.063 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:56.082 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:32:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:56.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:32:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.088+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2d681096e0 con 0x7f2d680f7400 2024-02-23T00:32:56.185 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:32:56.213 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:56.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.261+0000 7f2d5f7fe700 1 -- 172.21.15.87:0/734820686 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v225) v1 ==== 81+0+52539 (secure 0 0 0) 0x7f2d64021090 con 0x7f2d680f7400 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 225, 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: "created": "2024-02-23T00:21:28.861318+0000", 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: "modified": "2024-02-23T00:32:56.009414+0000", 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: "last_up_change": "2024-02-23T00:27:09.633391+0000", 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: "last_in_change": "2024-02-23T00:26:52.603467+0000", 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_num": 5799936, 2024-02-23T00:32:56.266 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_set": [ 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "pglog_hardlimit", 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "purged_snapdirs", 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "recovery_deletes", 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "sortbitwise" 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_version": 53, 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "full_ratio": 0.94999998807907104, 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_snapshot": "", 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_max": 154, 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd": 8, 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "require_min_compat_client": "luminous", 2024-02-23T00:32:56.267 INFO:teuthology.orchestra.run.smithi087.stdout: "min_compat_client": "luminous", 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "require_osd_release": "pacific", 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "pools": [ 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "device_health_metrics", 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:25:09.208309+0000", 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.268 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 1, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 1, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 1, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 1, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 1, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.269 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "68", 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.270 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.271 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.272 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_max": 32, 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_min": 1 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_devicehealth": {} 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 2, 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": ".rgw.root", 2024-02-23T00:32:56.273 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.562186+0000", 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:56.274 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "174", 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:56.275 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.276 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.277 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 3, 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "iscsi", 2024-02-23T00:32:56.278 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.989206+0000", 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 13, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 13, 2024-02-23T00:32:56.279 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 13, 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 13, 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 13, 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "137", 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:56.280 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "135", 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.281 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.282 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd": {} 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 4, 2024-02-23T00:32:56.283 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.log", 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:17.282440+0000", 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.284 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.285 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "58", 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.286 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.287 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.288 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 5, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.control", 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:19.262164+0000", 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.289 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.290 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "60", 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.291 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.292 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:32:56.293 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 6, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.meta", 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:21.319951+0000", 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.294 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.295 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "62", 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.296 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.297 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_bias": 4 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.298 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 19, 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "LibRadosList_smithi087-82051-1", 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:45.322617+0000", 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.299 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 13, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 11, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 11, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 11, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 13, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "19.d", 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 225, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 224, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 224, 2024-02-23T00:32:56.300 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "148'9", 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "225", 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "225", 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "223", 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.301 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.302 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.303 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:32:56.304 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.304 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.304 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.304 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 22, 2024-02-23T00:32:56.304 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-23T00:32:56.307 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:46.071806+0000", 2024-02-23T00:32:56.307 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.307 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.307 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.307 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.307 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.307 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.309 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.310 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.311 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-6", 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:31:04.375891+0000", 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.312 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.313 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "225", 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "220", 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "220", 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "220", 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 14, 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 225, 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.314 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.315 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 150, 2024-02-23T00:32:56.316 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "XattrsRoundTripPP_smithi087-82019-35", 2024-02-23T00:32:56.317 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:32:48.284463+0000", 2024-02-23T00:32:56.317 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.317 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.317 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.317 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.317 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.317 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.318 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "220", 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.319 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.320 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.321 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 154, 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-23", 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:32:53.265374+0000", 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:32:56.322 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:32:56.323 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "225", 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:32:56.324 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:32:56.325 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:32:56.326 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7", 2024-02-23T00:32:56.327 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 8, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 223, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.328 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6802", 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:56.329 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6804", 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6803", 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:32:56.330 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6801/2153191863", 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6802/2153191863", 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6804/2153191863", 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6803/2153191863", 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:32:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "85525ce8-38d5-4840-8057-547cb176c859", 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 13, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 223, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6805", 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6806", 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6808", 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6807", 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.334 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6805/685354171", 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6806/685354171", 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6808/685354171", 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6807/685354171", 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:32:56.335 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa", 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 18, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 223, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.336 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6809", 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6810", 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.337 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6812", 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6811", 2024-02-23T00:32:56.338 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6809/3605394228", 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6810/3605394228", 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6812/3605394228", 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6811/3605394228", 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.339 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a", 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 26, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 223, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.340 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6813", 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6814", 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.341 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6816", 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6815", 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:32:56.342 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6813/128059790", 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6814/128059790", 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6816/128059790", 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6815/128059790", 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c", 2024-02-23T00:32:56.343 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 31, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 223, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6800", 2024-02-23T00:32:56.344 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6801", 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.345 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6803", 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6802", 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6800/4175844757", 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6801/4175844757", 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4175844757", 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4175844757", 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d", 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:56.347 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 38, 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 223, 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6804", 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.348 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6805", 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.349 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6807", 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6806", 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.350 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6804/2550013795", 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6805/2550013795", 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2550013795", 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2550013795", 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01", 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:56.351 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 44, 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 223, 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6808", 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:32:56.352 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6809", 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.353 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6811", 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6810", 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.354 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6808/2509408498", 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6809/2509408498", 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6811/2509408498", 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6810/2509408498", 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939", 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:32:56.355 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 51, 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 223, 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6812", 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.356 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6813", 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6815", 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:32:56.357 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6814", 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6812/373787000", 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6813/373787000", 2024-02-23T00:32:56.358 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6815/373787000", 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6814/373787000", 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_xinfo": [ 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:56.359 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:41.672557+0000", 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:09.518145+0000", 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.360 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:30.427929+0000", 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:56.361 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:06.367304+0000", 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:42.185296+0000", 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.362 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:15.088109+0000", 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:56.363 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:35.496335+0000", 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:29.887338+0000", 2024-02-23T00:32:56.364 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap": [], 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap_items": [ 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.9", 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.365 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.13", 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.17", 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 5 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.366 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_temp": [], 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_temp": [], 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "blocklist": { 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1121018082": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/2138166229": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/27794119": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/1675126043": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/3767016525": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/969918521": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/795095363": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2273151327": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1685406414": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:32:56.367 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2093733106": "2024-02-24T00:21:58.144574+0000" 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "range_blocklist": {}, 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profiles": { 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "default": { 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van" 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "testprofile-ReturnValue_smithi087-82011-33": { 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-device-class": "", 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:32:56.368 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-root": "default", 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van", 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "w": "8" 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps_queue": [ 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 7, 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.369 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:32:56.370 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 21, 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.371 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 61, 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.372 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:32:56.373 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 78, 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.374 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 86, 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.375 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 95, 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.376 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 4, 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 4 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 107, 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.377 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 112, 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 125, 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.378 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 10, 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.379 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 133, 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: "new_removed_snaps": [], 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: "new_purged_snaps": [], 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_node_flags": {}, 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: "device_class_flags": {}, 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode": { 2024-02-23T00:32:56.380 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_enabled": false, 2024-02-23T00:32:56.381 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_bucket_count": 0, 2024-02-23T00:32:56.381 INFO:teuthology.orchestra.run.smithi087.stdout: "degraded_stretch_mode": 0, 2024-02-23T00:32:56.381 INFO:teuthology.orchestra.run.smithi087.stdout: "recovering_stretch_mode": 0, 2024-02-23T00:32:56.381 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_bucket": 0 2024-02-23T00:32:56.381 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:32:56.381 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:32:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.266+0000 7f2d70450700 10 librados: watch_flush enter 2024-02-23T00:32:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.266+0000 7f2d70450700 10 librados: watch_flush exit 2024-02-23T00:32:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.266+0000 7f2d70450700 20 client.26703.objecter shutdown clearing up homeless session... 2024-02-23T00:32:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.266+0000 7f2d70450700 10 client.26703.objecter successfully canceled tick 2024-02-23T00:32:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d54062c90 msgr2=0x7f2d54065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 --2- 172.21.15.87:0/734820686 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d54062c90 0x7f2d54065150 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 rx=0x7f2d681099a0 tx=0x7f2d5800b040).stop 2024-02-23T00:32:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d680f7400 msgr2=0x7f2d6810e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:32:56.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d680f7400 0x7f2d6810e060 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 rx=0x7f2d64009a00 tx=0x7f2d64009cb0).stop 2024-02-23T00:32:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 shutdown_connections 2024-02-23T00:32:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 --2- 172.21.15.87:0/734820686 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d54062c90 0x7f2d54065150 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d680fd510 0x7f2d68108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d680f7400 0x7f2d6810e060 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 --2- 172.21.15.87:0/734820686 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d680fcbb0 0x7f2d6810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:32:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 >> 172.21.15.87:0/734820686 conn(0x7f2d680f2820 msgr2=0x7f2d68060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:32:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 shutdown_connections 2024-02-23T00:32:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 -- 172.21.15.87:0/734820686 wait complete. 2024-02-23T00:32:56.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:32:56.267+0000 7f2d70450700 1 librados: shutdown 2024-02-23T00:32:56.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:55.012547+0000 mon.a (mon.0) 1646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:56.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:55.012677+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi087-82011-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:56.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:55.012729+0000 mon.a (mon.0) 1648 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-02-23T00:32:56.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:55.022157+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi087-82011-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:56.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:55.432087+0000 osd.6 (osd.6) 91 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:32:56.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:55.433027+0000 osd.6 (osd.6) 92 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:32:56.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:55.452896+0000 mon.c (mon.2) 267 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:56.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:55.012547+0000 mon.a (mon.0) 1646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:56.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:55.012677+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi087-82011-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:56.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:55.012729+0000 mon.a (mon.0) 1648 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-02-23T00:32:56.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:55.022157+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi087-82011-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:56.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:55.432087+0000 osd.6 (osd.6) 91 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:32:56.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:55.433027+0000 osd.6 (osd.6) 92 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:32:56.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:55.452896+0000 mon.c (mon.2) 267 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:55.012547+0000 mon.a (mon.0) 1646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:55.012677+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi087-82011-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:32:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:55.012729+0000 mon.a (mon.0) 1648 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-02-23T00:32:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:55.022157+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi087-82011-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:55.432087+0000 osd.6 (osd.6) 91 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:32:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:55.433027+0000 osd.6 (osd.6) 92 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:32:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:55.452896+0000 mon.c (mon.2) 267 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:57.540 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:55.592805+0000 mgr.y (mgr.24437) 256 : cluster [DBG] pgmap v319: 316 pgs: 2 clean+premerge+peered, 297 active+clean, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:55.990281+0000 osd.3 (osd.3) 117 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:55.991511+0000 osd.3 (osd.3) 118 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.258538+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:56.259778+0000 mon.a (mon.0) 1651 : audit [DBG] from='client.? 172.21.15.87:0/734820686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:56.275877+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:56.276156+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.393265+0000 osd.1 (osd.1) 119 : cluster [DBG] 22.b scrub starts 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.396147+0000 osd.1 (osd.1) 120 : cluster [DBG] 22.b scrub ok 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:56.453882+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:57.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.609956+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.610881+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:57.015300+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi087-82011-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi087-82011-33"}]': finished 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:57.015434+0000 mon.a (mon.0) 1654 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:57.015514+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:57.019698+0000 mon.b (mon.1) 204 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:57.025561+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:55.592805+0000 mgr.y (mgr.24437) 256 : cluster [DBG] pgmap v319: 316 pgs: 2 clean+premerge+peered, 297 active+clean, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:55.990281+0000 osd.3 (osd.3) 117 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:55.991511+0000 osd.3 (osd.3) 118 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.258538+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-02-23T00:32:57.542 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:56.259778+0000 mon.a (mon.0) 1651 : audit [DBG] from='client.? 172.21.15.87:0/734820686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:56.275877+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:56.276156+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.393265+0000 osd.1 (osd.1) 119 : cluster [DBG] 22.b scrub starts 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.396147+0000 osd.1 (osd.1) 120 : cluster [DBG] 22.b scrub ok 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:56.453882+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.609956+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.610881+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:57.015300+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi087-82011-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi087-82011-33"}]': finished 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:57.015434+0000 mon.a (mon.0) 1654 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:57.015514+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:57.019698+0000 mon.b (mon.1) 204 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:32:57.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:57.025561+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:32:57.561 INFO:tasks.thrashosds.thrasher:j is {'epoch': 225, 'fsid': '66db207c-d1e1-11ee-95c0-87774f69a715', 'created': '2024-02-23T00:21:28.861318+0000', 'modified': '2024-02-23T00:32:56.009414+0000', 'last_up_change': '2024-02-23T00:27:09.633391+0000', 'last_in_change': '2024-02-23T00:26:52.603467+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 53, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 154, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '135', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 19, 'pool_name': 'LibRadosList_smithi087-82051-1', 'create_time': '2024-02-23T00:30:45.322617+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '19.d', 'ready_epoch': 225, 'last_epoch_started': 224, 'last_epoch_clean': 224, 'source_version': "148'9", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '225', 'last_force_op_resend_preluminous': '223', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 22, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-23T00:30:46.071806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 63, 'pool_name': 'test-rados-api-smithi087-82091-6', 'create_time': '2024-02-23T00:31:04.375891+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '220', 'last_force_op_resend_prenautilus': '220', 'last_force_op_resend_preluminous': '220', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 14, 'snap_epoch': 225, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 150, 'pool_name': 'XattrsRoundTripPP_smithi087-82019-35', 'create_time': '2024-02-23T00:32:48.284463+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '220', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 154, 'pool_name': 'test-rados-api-smithi087-82091-23', 'create_time': '2024-02-23T00:32:53.265374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '225', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e3549bad-2891-47b9-bf77-132cadc20bf7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6801', 'nonce': 2153191863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6802', 'nonce': 2153191863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6804', 'nonce': 2153191863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6803', 'nonce': 2153191863}]}, 'public_addr': '172.21.15.87:6801/2153191863', 'cluster_addr': '172.21.15.87:6802/2153191863', 'heartbeat_back_addr': '172.21.15.87:6804/2153191863', 'heartbeat_front_addr': '172.21.15.87:6803/2153191863', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '85525ce8-38d5-4840-8057-547cb176c859', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6805', 'nonce': 685354171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6806', 'nonce': 685354171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6808', 'nonce': 685354171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6807', 'nonce': 685354171}]}, 'public_addr': '172.21.15.87:6805/685354171', 'cluster_addr': '172.21.15.87:6806/685354171', 'heartbeat_back_addr': '172.21.15.87:6808/685354171', 'heartbeat_front_addr': '172.21.15.87:6807/685354171', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6809', 'nonce': 3605394228}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6810', 'nonce': 3605394228}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6812', 'nonce': 3605394228}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6811', 'nonce': 3605394228}]}, 'public_addr': '172.21.15.87:6809/3605394228', 'cluster_addr': '172.21.15.87:6810/3605394228', 'heartbeat_back_addr': '172.21.15.87:6812/3605394228', 'heartbeat_front_addr': '172.21.15.87:6811/3605394228', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6813', 'nonce': 128059790}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6814', 'nonce': 128059790}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6816', 'nonce': 128059790}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6815', 'nonce': 128059790}]}, 'public_addr': '172.21.15.87:6813/128059790', 'cluster_addr': '172.21.15.87:6814/128059790', 'heartbeat_back_addr': '172.21.15.87:6816/128059790', 'heartbeat_front_addr': '172.21.15.87:6815/128059790', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c3734a04-7343-4251-9b5b-bedb9e8b882c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6800', 'nonce': 4175844757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6801', 'nonce': 4175844757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6803', 'nonce': 4175844757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6802', 'nonce': 4175844757}]}, 'public_addr': '172.21.15.171:6800/4175844757', 'cluster_addr': '172.21.15.171:6801/4175844757', 'heartbeat_back_addr': '172.21.15.171:6803/4175844757', 'heartbeat_front_addr': '172.21.15.171:6802/4175844757', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fe86232d-96b9-4cbf-9cac-373286fa103d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6804', 'nonce': 2550013795}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6805', 'nonce': 2550013795}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6807', 'nonce': 2550013795}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6806', 'nonce': 2550013795}]}, 'public_addr': '172.21.15.171:6804/2550013795', 'cluster_addr': '172.21.15.171:6805/2550013795', 'heartbeat_back_addr': '172.21.15.171:6807/2550013795', 'heartbeat_front_addr': '172.21.15.171:6806/2550013795', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6808', 'nonce': 2509408498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6809', 'nonce': 2509408498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6811', 'nonce': 2509408498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6810', 'nonce': 2509408498}]}, 'public_addr': '172.21.15.171:6808/2509408498', 'cluster_addr': '172.21.15.171:6809/2509408498', 'heartbeat_back_addr': '172.21.15.171:6811/2509408498', 'heartbeat_front_addr': '172.21.15.171:6810/2509408498', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '29ddcf97-369a-4a2c-b3e2-142e84ccf939', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 223, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6812', 'nonce': 373787000}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6813', 'nonce': 373787000}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6815', 'nonce': 373787000}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6814', 'nonce': 373787000}]}, 'public_addr': '172.21.15.171:6812/373787000', 'cluster_addr': '172.21.15.171:6813/373787000', 'heartbeat_back_addr': '172.21.15.171:6815/373787000', 'heartbeat_front_addr': '172.21.15.171:6814/373787000', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:41.672557+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:09.518145+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:30.427929+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:06.367304+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:42.185296+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:15.088109+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:35.496335+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:29.887338+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.87:6800/1121018082': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/2138166229': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/27794119': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/1675126043': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/3767016525': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/969918521': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/795095363': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2273151327': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/1685406414': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2093733106': '2024-02-24T00:21:58.144574+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-ReturnValue_smithi087-82011-33': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 21, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 61, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 63, 'snaps': [{'begin': 6, 'length': 2}, {'begin': 12, 'length': 1}]}, {'pool': 78, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 86, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-23T00:32:57.561 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:32:57.562 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:32:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:57.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:32:57.666 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:55.592805+0000 mgr.y (mgr.24437) 256 : cluster [DBG] pgmap v319: 316 pgs: 2 clean+premerge+peered, 297 active+clean, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:55.990281+0000 osd.3 (osd.3) 117 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:55.991511+0000 osd.3 (osd.3) 118 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.258538+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:56.259778+0000 mon.a (mon.0) 1651 : audit [DBG] from='client.? 172.21.15.87:0/734820686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:56.275877+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:56.276156+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.393265+0000 osd.1 (osd.1) 119 : cluster [DBG] 22.b scrub starts 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.396147+0000 osd.1 (osd.1) 120 : cluster [DBG] 22.b scrub ok 2024-02-23T00:32:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:56.453882+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.609956+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:32:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.610881+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:32:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:57.015300+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi087-82011-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi087-82011-33"}]': finished 2024-02-23T00:32:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:57.015434+0000 mon.a (mon.0) 1654 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:32:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:57.015514+0000 mon.a (mon.0) 1655 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-02-23T00:32:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:57.019698+0000 mon.b (mon.1) 204 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:32:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:57.025561+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:32:57.696 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:57.717 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:57.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:57.821 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:32:57.850 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:57.870 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:32:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:57.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:32:57.974 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:32:58.003 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:58.029 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:32:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:58.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:32:58.133 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:32:58.161 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:58.233 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:32:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:32:58.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:32:58.338 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:32:58.365 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:32:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.374843+0000 osd.5 (osd.5) 105 : cluster [DBG] 5.15 deep-scrub starts 2024-02-23T00:32:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.376365+0000 osd.5 (osd.5) 106 : cluster [DBG] 5.15 deep-scrub ok 2024-02-23T00:32:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.675235+0000 osd.4 (osd.4) 109 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:32:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.678349+0000 osd.4 (osd.4) 110 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:32:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.691205+0000 osd.7 (osd.7) 117 : cluster [DBG] 2.b scrub starts 2024-02-23T00:32:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:56.692032+0000 osd.7 (osd.7) 118 : cluster [DBG] 2.b scrub ok 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:57.419798+0000 osd.5 (osd.5) 107 : cluster [DBG] 19.b scrub starts 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:57.420930+0000 osd.5 (osd.5) 108 : cluster [DBG] 19.b scrub ok 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:57.454826+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:57.594169+0000 mon.a (mon.0) 1657 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:58.018183+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-23"}]': finished 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:58.018329+0000 mon.a (mon.0) 1659 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:58.018405+0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:58.024647+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.87:0/1314423278' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:58.026032+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-23", "mode": "writeback"}]: dispatch 2024-02-23T00:32:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:58.026328+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-23", "mode": "writeback"}]: dispatch 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.374843+0000 osd.5 (osd.5) 105 : cluster [DBG] 5.15 deep-scrub starts 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.376365+0000 osd.5 (osd.5) 106 : cluster [DBG] 5.15 deep-scrub ok 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.675235+0000 osd.4 (osd.4) 109 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.678349+0000 osd.4 (osd.4) 110 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.691205+0000 osd.7 (osd.7) 117 : cluster [DBG] 2.b scrub starts 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:56.692032+0000 osd.7 (osd.7) 118 : cluster [DBG] 2.b scrub ok 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:57.419798+0000 osd.5 (osd.5) 107 : cluster [DBG] 19.b scrub starts 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:57.420930+0000 osd.5 (osd.5) 108 : cluster [DBG] 19.b scrub ok 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:57.454826+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:57.594169+0000 mon.a (mon.0) 1657 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:58.018183+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-23"}]': finished 2024-02-23T00:32:58.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:58.018329+0000 mon.a (mon.0) 1659 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-23T00:32:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:58.018405+0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-02-23T00:32:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:58.024647+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.87:0/1314423278' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:58.026032+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-23", "mode": "writeback"}]: dispatch 2024-02-23T00:32:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:58.026328+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-23", "mode": "writeback"}]: dispatch 2024-02-23T00:32:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.374843+0000 osd.5 (osd.5) 105 : cluster [DBG] 5.15 deep-scrub starts 2024-02-23T00:32:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.376365+0000 osd.5 (osd.5) 106 : cluster [DBG] 5.15 deep-scrub ok 2024-02-23T00:32:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.675235+0000 osd.4 (osd.4) 109 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:32:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.678349+0000 osd.4 (osd.4) 110 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:32:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.691205+0000 osd.7 (osd.7) 117 : cluster [DBG] 2.b scrub starts 2024-02-23T00:32:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:56.692032+0000 osd.7 (osd.7) 118 : cluster [DBG] 2.b scrub ok 2024-02-23T00:32:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:57.419798+0000 osd.5 (osd.5) 107 : cluster [DBG] 19.b scrub starts 2024-02-23T00:32:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:57.420930+0000 osd.5 (osd.5) 108 : cluster [DBG] 19.b scrub ok 2024-02-23T00:32:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:57.454826+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:57.594169+0000 mon.a (mon.0) 1657 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-23T00:32:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:58.018183+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-23"}]': finished 2024-02-23T00:32:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:58.018329+0000 mon.a (mon.0) 1659 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-23T00:32:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:58.018405+0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-02-23T00:32:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:58.024647+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.87:0/1314423278' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:32:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:58.026032+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-23", "mode": "writeback"}]: dispatch 2024-02-23T00:32:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:58.026328+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-23", "mode": "writeback"}]: dispatch 2024-02-23T00:32:59.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:57.593206+0000 mgr.y (mgr.24437) 257 : cluster [DBG] pgmap v322: 259 pgs: 8 unknown, 1 clean+premerge+peered, 233 active+clean, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:59.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:57.835464+0000 osd.2 (osd.2) 105 : cluster [DBG] 63.16 scrub starts 2024-02-23T00:32:59.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:57.837191+0000 osd.2 (osd.2) 106 : cluster [DBG] 63.16 scrub ok 2024-02-23T00:32:59.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:58.455527+0000 mon.c (mon.2) 270 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:59.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:58.671927+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:32:59.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:58.673623+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:32:59.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:58.678733+0000 osd.7 (osd.7) 119 : cluster [DBG] 22.d scrub starts 2024-02-23T00:32:59.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:58.681622+0000 osd.7 (osd.7) 120 : cluster [DBG] 22.d scrub ok 2024-02-23T00:32:59.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.014732+0000 mon.a (mon.0) 1663 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:59.019057+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? 172.21.15.87:0/1314423278' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:59.019153+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-23", "mode": "writeback"}]': finished 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.019250+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:59.021313+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:57.593206+0000 mgr.y (mgr.24437) 257 : cluster [DBG] pgmap v322: 259 pgs: 8 unknown, 1 clean+premerge+peered, 233 active+clean, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:57.835464+0000 osd.2 (osd.2) 105 : cluster [DBG] 63.16 scrub starts 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:57.837191+0000 osd.2 (osd.2) 106 : cluster [DBG] 63.16 scrub ok 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:58.455527+0000 mon.c (mon.2) 270 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:58.671927+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:58.673623+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:32:59.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:58.678733+0000 osd.7 (osd.7) 119 : cluster [DBG] 22.d scrub starts 2024-02-23T00:32:59.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:58.681622+0000 osd.7 (osd.7) 120 : cluster [DBG] 22.d scrub ok 2024-02-23T00:32:59.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.014732+0000 mon.a (mon.0) 1663 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:59.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:59.019057+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? 172.21.15.87:0/1314423278' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:59.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:59.019153+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-23", "mode": "writeback"}]': finished 2024-02-23T00:32:59.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.019250+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-02-23T00:32:59.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:32:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:59.021313+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:32:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:57.593206+0000 mgr.y (mgr.24437) 257 : cluster [DBG] pgmap v322: 259 pgs: 8 unknown, 1 clean+premerge+peered, 233 active+clean, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:32:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:57.835464+0000 osd.2 (osd.2) 105 : cluster [DBG] 63.16 scrub starts 2024-02-23T00:32:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:57.837191+0000 osd.2 (osd.2) 106 : cluster [DBG] 63.16 scrub ok 2024-02-23T00:32:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:58.455527+0000 mon.c (mon.2) 270 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:32:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:58.671927+0000 osd.4 (osd.4) 111 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:32:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:58.673623+0000 osd.4 (osd.4) 112 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:32:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:58.678733+0000 osd.7 (osd.7) 119 : cluster [DBG] 22.d scrub starts 2024-02-23T00:32:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:58.681622+0000 osd.7 (osd.7) 120 : cluster [DBG] 22.d scrub ok 2024-02-23T00:32:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.014732+0000 mon.a (mon.0) 1663 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:32:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:59.019057+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? 172.21.15.87:0/1314423278' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:32:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:59.019153+0000 mon.a (mon.0) 1665 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-23", "mode": "writeback"}]': finished 2024-02-23T00:32:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.019250+0000 mon.a (mon.0) 1666 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-02-23T00:32:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:32:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:59.021313+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:33:00.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 -- 172.21.15.87:0/229889401 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 msgr2=0x7fa910104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 --2- 172.21.15.87:0/229889401 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 0x7fa910104600 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7fa908004870 tx=0x7fa908009cb0).stop 2024-02-23T00:33:00.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 -- 172.21.15.87:0/229889401 shutdown_connections 2024-02-23T00:33:00.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 --2- 172.21.15.87:0/229889401 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 0x7fa910104600 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 --2- 172.21.15.87:0/229889401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9100f8cb0 0x7fa9100f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 --2- 172.21.15.87:0/229889401 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9100f9610 0x7fa910101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 -- 172.21.15.87:0/229889401 >> 172.21.15.87:0/229889401 conn(0x7fa9100f2820 msgr2=0x7fa9100f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 -- 172.21.15.87:0/229889401 shutdown_connections 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 -- 172.21.15.87:0/229889401 wait complete. 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 librados: starting msgr at 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.080+0000 7fa915e9b700 1 librados: starting objecter 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 Processor -- start 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 -- start start 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 librados: setting wanted keys 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 librados: calling monclient init 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9100f8cb0 0x7fa9100f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9100f9610 0x7fa9100f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 0x7fa9100f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101163a0 con 0x7fa9100f9610 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa910116510 con 0x7fa9100f8cb0 2024-02-23T00:33:00.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa915e9b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9101166d0 con 0x7fa910102210 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 0x7fa9100f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90effd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9100f9610 0x7fa9100f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 0x7fa9100f7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39562/0 (socket says 172.21.15.87:39562) 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90ffff700 1 -- 172.21.15.87:0/3532636864 learned_addr learned my addr 172.21.15.87:0/3532636864 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90f7fe700 1 --2- 172.21.15.87:0/3532636864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9100f8cb0 0x7fa9100f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90ffff700 1 -- 172.21.15.87:0/3532636864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9100f8cb0 msgr2=0x7fa9100f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90ffff700 1 --2- 172.21.15.87:0/3532636864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9100f8cb0 0x7fa9100f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90ffff700 1 -- 172.21.15.87:0/3532636864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9100f9610 msgr2=0x7fa9100f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90ffff700 1 --2- 172.21.15.87:0/3532636864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9100f9610 0x7fa9100f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90ffff700 1 -- 172.21.15.87:0/3532636864 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9000097c0 con 0x7fa910102210 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90effd700 1 --2- 172.21.15.87:0/3532636864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9100f9610 0x7fa9100f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90ffff700 1 --2- 172.21.15.87:0/3532636864 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 0x7fa9100f7640 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fa908014f00 tx=0x7fa90800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90cff9700 10 client.?.objecter ms_handle_connect 0x7fa910102210 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.081+0000 7fa90cff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.082+0000 7fa90cff9700 1 -- 172.21.15.87:0/3532636864 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90800e030 con 0x7fa910102210 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.082+0000 7fa90cff9700 1 -- 172.21.15.87:0/3532636864 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa908009710 con 0x7fa910102210 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.082+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa908005040 con 0x7fa910102210 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.082+0000 7fa90cff9700 1 -- 172.21.15.87:0/3532636864 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908019070 con 0x7fa910102210 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.082+0000 7fa915e9b700 10 client.37169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.082+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa910116b70 con 0x7fa910102210 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.082+0000 7fa915e9b700 1 librados: init done 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.082+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91005fa60 con 0x7fa910102210 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.083+0000 7fa90cff9700 1 -- 172.21.15.87:0/3532636864 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fa90801e420 con 0x7fa910102210 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.083+0000 7fa90cff9700 1 --2- 172.21.15.87:0/3532636864 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa8fc062c90 0x7fa8fc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.083+0000 7fa90cff9700 1 -- 172.21.15.87:0/3532636864 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(229..229 src has 38..229) v4 ==== 9174+0+0 (secure 0 0 0) 0x7fa90807ea50 con 0x7fa910102210 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.083+0000 7fa90cff9700 10 client.37169.objecter ms_dispatch 0x7fa910001550 osd_map(229..229 src has 38..229) v4 2024-02-23T00:33:00.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.083+0000 7fa90cff9700 3 client.37169.objecter handle_osd_map got epochs [229,229] > 0 2024-02-23T00:33:00.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.083+0000 7fa90cff9700 3 client.37169.objecter handle_osd_map decoding full epoch 229 2024-02-23T00:33:00.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.083+0000 7fa90f7fe700 1 --2- 172.21.15.87:0/3532636864 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa8fc062c90 0x7fa8fc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.084+0000 7fa90cff9700 20 client.37169.objecter dump_active .. 0 homeless 2024-02-23T00:33:00.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.084+0000 7fa90cff9700 10 client.37169.objecter ms_handle_connect 0x7fa8fc062c90 2024-02-23T00:33:00.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.084+0000 7fa90f7fe700 1 --2- 172.21.15.87:0/3532636864 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa8fc062c90 0x7fa8fc065150 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7fa900001530 tx=0x7fa900004c50).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.088+0000 7fa90cff9700 1 -- 172.21.15.87:0/3532636864 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa908051ac0 con 0x7fa910102210 2024-02-23T00:33:00.115 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:00.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:00.219 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:00.251 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:00.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.268+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fa9100f8140 con 0x7fa910102210 2024-02-23T00:33:00.272 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:00.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:00.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 -- 172.21.15.87:0/2657585454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8940fe510 msgr2=0x7fd8940f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 --2- 172.21.15.87:0/2657585454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8940fe510 0x7fd8940f7d40 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 rx=0x7fd8880047b0 tx=0x7fd888009cb0).stop 2024-02-23T00:33:00.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 -- 172.21.15.87:0/2657585454 shutdown_connections 2024-02-23T00:33:00.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 --2- 172.21.15.87:0/2657585454 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8940f8420 0x7fd894104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 --2- 172.21.15.87:0/2657585454 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8940fdbb0 0x7fd8940fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 --2- 172.21.15.87:0/2657585454 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8940fe510 0x7fd8940f7d40 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 -- 172.21.15.87:0/2657585454 >> 172.21.15.87:0/2657585454 conn(0x7fd8940f3800 msgr2=0x7fd8940f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:00.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 -- 172.21.15.87:0/2657585454 shutdown_connections 2024-02-23T00:33:00.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 -- 172.21.15.87:0/2657585454 wait complete. 2024-02-23T00:33:00.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 librados: starting msgr at 2024-02-23T00:33:00.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.326+0000 7fd89c083700 1 librados: starting objecter 2024-02-23T00:33:00.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 Processor -- start 2024-02-23T00:33:00.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 -- start start 2024-02-23T00:33:00.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 librados: setting wanted keys 2024-02-23T00:33:00.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 librados: calling monclient init 2024-02-23T00:33:00.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8940f8420 0x7fd89406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8940fdbb0 0x7fd89406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8940fe510 0x7fd89406bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd894116670 con 0x7fd8940fdbb0 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8941167e0 con 0x7fd8940fe510 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89c083700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8941169a0 con 0x7fd8940f8420 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd899e1f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8940f8420 0x7fd89406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd899e1f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8940f8420 0x7fd89406b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:39586/0 (socket says 172.21.15.87:39586) 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd899e1f700 1 -- 172.21.15.87:0/1544669981 learned_addr learned my addr 172.21.15.87:0/1544669981 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89961e700 1 --2- 172.21.15.87:0/1544669981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8940fdbb0 0x7fd89406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89a620700 1 --2- 172.21.15.87:0/1544669981 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8940fe510 0x7fd89406bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd899e1f700 1 -- 172.21.15.87:0/1544669981 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8940fe510 msgr2=0x7fd89406bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd899e1f700 1 --2- 172.21.15.87:0/1544669981 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8940fe510 0x7fd89406bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd899e1f700 1 -- 172.21.15.87:0/1544669981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8940fdbb0 msgr2=0x7fd89406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd899e1f700 1 --2- 172.21.15.87:0/1544669981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8940fdbb0 0x7fd89406b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd899e1f700 1 -- 172.21.15.87:0/1544669981 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd884005040 con 0x7fd8940f8420 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.327+0000 7fd89961e700 1 --2- 172.21.15.87:0/1544669981 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8940fdbb0 0x7fd89406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd899e1f700 1 --2- 172.21.15.87:0/1544669981 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8940f8420 0x7fd89406b0d0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fd88c00db50 tx=0x7fd88c004ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd892ffd700 10 client.?.objecter ms_handle_connect 0x7fd8940f8420 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd892ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c0077a0 con 0x7fd8940f8420 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd888005040 con 0x7fd8940f8420 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd88c00f040 con 0x7fd8940f8420 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c013b60 con 0x7fd8940f8420 2024-02-23T00:33:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd89c083700 10 client.37175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd894117030 con 0x7fd8940f8420 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd89c083700 1 librados: init done 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.328+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd894107110 con 0x7fd8940f8420 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.329+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd88c00b750 con 0x7fd8940f8420 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd892ffd700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd87c062c90 0x7fd87c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(229..229 src has 38..229) v4 ==== 9174+0+0 (secure 0 0 0) 0x7fd88c07da40 con 0x7fd8940f8420 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 osd_map(229..229 src has 38..229) v4 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd892ffd700 3 client.37175.objecter handle_osd_map got epochs [229,229] > 0 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd892ffd700 3 client.37175.objecter handle_osd_map decoding full epoch 229 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89961e700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd87c062c90 0x7fd87c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd892ffd700 20 client.37175.objecter dump_active .. 0 homeless 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v229) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd88c07de60 con 0x7fd8940f8420 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89961e700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd87c062c90 0x7fd87c065150 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7fd89406c870 tx=0x7fd888007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd892ffd700 10 client.37175.objecter ms_handle_connect 0x7fd87c062c90 2024-02-23T00:33:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 10 client.37175.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 1 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd8940598f0 0x7fd894059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd89406c5b0 osd=0 3 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89406c5b0 osd=0 3 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd89406c5b0 osd=0 3 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 1 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd89406c5b0 osd=0 3 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 0 1 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89406c5b0 osd=0 4 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 10 client.37175.objecter _send_command 1 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd89405fa60 con 0x7fd8940598f0 2024-02-23T00:33:00.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.330+0000 7fd89a620700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd8940598f0 0x7fd894059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.331+0000 7fd89a620700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd8940598f0 0x7fd894059cf0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.331+0000 7fd892ffd700 10 client.37175.objecter ms_handle_connect 0x7fd8940598f0 2024-02-23T00:33:00.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.331+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd89405fa60 con 0x7fd8940598f0 2024-02-23T00:33:00.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.331+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:33:00.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.331+0000 7fd892ffd700 10 client.37175.objecter _finish_command 1 = system:0 2024-02-23T00:33:00.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.331+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd89406c5b0 osd=0 4 2024-02-23T00:33:00.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.331+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 0 1 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 10 client.37175.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 2 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd89406c5b0 osd=0 3 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89406c5b0 osd=0 3 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd89406c5b0 osd=0 3 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 2 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd89406c5b0 osd=0 3 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 0 2 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89406c5b0 osd=0 4 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 10 client.37175.objecter _send_command 2 2024-02-23T00:33:00.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.346+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd894028540 con 0x7fd8940598f0 2024-02-23T00:33:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.347+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd894028540 con 0x7fd8940598f0 2024-02-23T00:33:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.347+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.347+0000 7fd892ffd700 10 client.37175.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.347+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd89406c5b0 osd=0 4 2024-02-23T00:33:00.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.347+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 0 2 2024-02-23T00:33:00.349 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 10 client.37175.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 3 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6805/685354171 conn(0x7fd8940594d0 0x7fd89411b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894059360 osd=1 3 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894059360 osd=1 3 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894059360 osd=1 3 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 3 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894059360 osd=1 3 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 1 3 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894059360 osd=1 4 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 10 client.37175.objecter _send_command 3 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8940ff5d0 con 0x7fd8940594d0 2024-02-23T00:33:00.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd899e1f700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6805/685354171 conn(0x7fd8940594d0 0x7fd89411b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd892ffd700 10 client.37175.objecter ms_handle_connect 0x7fd8940594d0 2024-02-23T00:33:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd899e1f700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6805/685354171 conn(0x7fd8940594d0 0x7fd89411b240 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd8940ff5d0 con 0x7fd8940594d0 2024-02-23T00:33:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:33:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd892ffd700 10 client.37175.objecter _finish_command 3 = system:0 2024-02-23T00:33:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894059360 osd=1 4 2024-02-23T00:33:00.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.348+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 1 3 2024-02-23T00:33:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 10 client.37175.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 4 2024-02-23T00:33:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894059360 osd=1 3 2024-02-23T00:33:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894059360 osd=1 3 2024-02-23T00:33:00.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894059360 osd=1 3 2024-02-23T00:33:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 4 2024-02-23T00:33:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894059360 osd=1 3 2024-02-23T00:33:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 1 4 2024-02-23T00:33:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894059360 osd=1 4 2024-02-23T00:33:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 10 client.37175.objecter _send_command 4 2024-02-23T00:33:00.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.363+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd89411b730 con 0x7fd8940594d0 2024-02-23T00:33:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.364+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd89411b730 con 0x7fd8940594d0 2024-02-23T00:33:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.364+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.364+0000 7fd892ffd700 10 client.37175.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.364+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894059360 osd=1 4 2024-02-23T00:33:00.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.364+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 1 4 2024-02-23T00:33:00.366 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:33:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 10 client.37175.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 5 2024-02-23T00:33:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd8940feca0 0x7fd89411ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd8940feb30 osd=2 3 2024-02-23T00:33:00.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd8940feb30 osd=2 3 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd8940feb30 osd=2 3 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 5 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd8940feb30 osd=2 3 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 2 5 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd8940feb30 osd=2 4 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 10 client.37175.objecter _send_command 5 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8940280a0 con 0x7fd8940feca0 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89961e700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd8940feca0 0x7fd89411ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd892ffd700 10 client.37175.objecter ms_handle_connect 0x7fd8940feca0 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd89961e700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd8940feca0 0x7fd89411ea90 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd8940280a0 con 0x7fd8940feca0 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd892ffd700 10 client.37175.objecter _finish_command 5 = system:0 2024-02-23T00:33:00.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd8940feb30 osd=2 4 2024-02-23T00:33:00.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.365+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 2 5 2024-02-23T00:33:00.376 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:00.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 10 client.37175.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:00.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 6 2024-02-23T00:33:00.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd8940feb30 osd=2 3 2024-02-23T00:33:00.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd8940feb30 osd=2 3 2024-02-23T00:33:00.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd8940feb30 osd=2 3 2024-02-23T00:33:00.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 6 2024-02-23T00:33:00.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd8940feb30 osd=2 3 2024-02-23T00:33:00.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 2 6 2024-02-23T00:33:00.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd8940feb30 osd=2 4 2024-02-23T00:33:00.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 10 client.37175.objecter _send_command 6 2024-02-23T00:33:00.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.380+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd89411ef80 con 0x7fd8940feca0 2024-02-23T00:33:00.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.381+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd89411ef80 con 0x7fd8940feca0 2024-02-23T00:33:00.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.381+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:00.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.381+0000 7fd892ffd700 10 client.37175.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.381+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd8940feb30 osd=2 4 2024-02-23T00:33:00.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.381+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 2 6 2024-02-23T00:33:00.383 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 10 client.37175.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 7 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6813/128059790 conn(0x7fd894027770 0x7fd894122280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894027600 osd=3 3 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894027600 osd=3 3 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894027600 osd=3 3 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 7 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894027600 osd=3 3 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 3 7 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894027600 osd=3 4 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 10 client.37175.objecter _send_command 7 2024-02-23T00:33:00.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd89411c580 con 0x7fd894027770 2024-02-23T00:33:00.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89a620700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6813/128059790 conn(0x7fd894027770 0x7fd894122280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd892ffd700 10 client.37175.objecter ms_handle_connect 0x7fd894027770 2024-02-23T00:33:00.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.382+0000 7fd89a620700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6813/128059790 conn(0x7fd894027770 0x7fd894122280 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.383+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd89411c580 con 0x7fd894027770 2024-02-23T00:33:00.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.383+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:33:00.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.383+0000 7fd892ffd700 10 client.37175.objecter _finish_command 7 = system:0 2024-02-23T00:33:00.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.383+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894027600 osd=3 4 2024-02-23T00:33:00.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.383+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 3 7 2024-02-23T00:33:00.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 10 client.37175.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:00.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 8 2024-02-23T00:33:00.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894027600 osd=3 3 2024-02-23T00:33:00.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894027600 osd=3 3 2024-02-23T00:33:00.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894027600 osd=3 3 2024-02-23T00:33:00.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 8 2024-02-23T00:33:00.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894027600 osd=3 3 2024-02-23T00:33:00.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 3 8 2024-02-23T00:33:00.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894027600 osd=3 4 2024-02-23T00:33:00.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 10 client.37175.objecter _send_command 8 2024-02-23T00:33:00.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.399+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd894122770 con 0x7fd894027770 2024-02-23T00:33:00.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd894122770 con 0x7fd894027770 2024-02-23T00:33:00.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:00.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd892ffd700 10 client.37175.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894027600 osd=3 4 2024-02-23T00:33:00.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 3 8 2024-02-23T00:33:00.403 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 10 client.37175.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 9 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd894122df0 0x7fd8941251e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894122770 osd=4 3 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894122770 osd=4 3 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894122770 osd=4 3 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 9 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894122770 osd=4 3 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 4 9 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894122770 osd=4 4 2024-02-23T00:33:00.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 10 client.37175.objecter _send_command 9 2024-02-23T00:33:00.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd894127f70 con 0x7fd894122df0 2024-02-23T00:33:00.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.401+0000 7fd899e1f700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd894122df0 0x7fd8941251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.402+0000 7fd892ffd700 10 client.37175.objecter ms_handle_connect 0x7fd894122df0 2024-02-23T00:33:00.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.402+0000 7fd899e1f700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd894122df0 0x7fd8941251e0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:58.806460+0000 osd.2 (osd.2) 107 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:33:00.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:58.808098+0000 osd.2 (osd.2) 108 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:33:00.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:58.972320+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:33:00.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:58.975330+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:33:00.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.432918+0000 osd.5 (osd.5) 109 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.434075+0000 osd.5 (osd.5) 110 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.455569+0000 osd.6 (osd.6) 93 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:32:59.456799+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.459049+0000 osd.6 (osd.6) 94 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.679851+0000 osd.7 (osd.7) 121 : cluster [DBG] 19.9 deep-scrub starts 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.681029+0000 osd.7 (osd.7) 122 : cluster [DBG] 19.9 deep-scrub ok 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.022859+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi087-82011-33"}]': finished 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:00.022993+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.035218+0000 mon.a (mon.0) 1670 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.039791+0000 mon.c (mon.2) 272 : audit [INF] from='client.? 172.21.15.87:0/2545760482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.040183+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:00.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.054396+0000 mon.b (mon.1) 206 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:58.806460+0000 osd.2 (osd.2) 107 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:58.808098+0000 osd.2 (osd.2) 108 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:58.972320+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:58.975330+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.432918+0000 osd.5 (osd.5) 109 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.434075+0000 osd.5 (osd.5) 110 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.455569+0000 osd.6 (osd.6) 93 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:32:59.456799+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.459049+0000 osd.6 (osd.6) 94 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:33:00.407 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.679851+0000 osd.7 (osd.7) 121 : cluster [DBG] 19.9 deep-scrub starts 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.681029+0000 osd.7 (osd.7) 122 : cluster [DBG] 19.9 deep-scrub ok 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.022859+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi087-82011-33"}]': finished 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:00.022993+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.035218+0000 mon.a (mon.0) 1670 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.039791+0000 mon.c (mon.2) 272 : audit [INF] from='client.? 172.21.15.87:0/2545760482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.040183+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.054396+0000 mon.b (mon.1) 206 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.054664+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.268789+0000 mon.c (mon.2) 273 : audit [INF] from='client.? 172.21.15.87:0/3532636864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:00.408 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.269109+0000 mon.a (mon.0) 1673 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:00.409 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.054664+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:00.409 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.268789+0000 mon.c (mon.2) 273 : audit [INF] from='client.? 172.21.15.87:0/3532636864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:00.409 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.269109+0000 mon.a (mon.0) 1673 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:00.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.403+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd894127f70 con 0x7fd894122df0 2024-02-23T00:33:00.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.403+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:33:00.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.403+0000 7fd892ffd700 10 client.37175.objecter _finish_command 9 = system:0 2024-02-23T00:33:00.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.403+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894122770 osd=4 4 2024-02-23T00:33:00.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.403+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 4 9 2024-02-23T00:33:00.410 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 10 client.37175.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 10 2024-02-23T00:33:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894122770 osd=4 3 2024-02-23T00:33:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894122770 osd=4 3 2024-02-23T00:33:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894122770 osd=4 3 2024-02-23T00:33:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 10 2024-02-23T00:33:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894122770 osd=4 3 2024-02-23T00:33:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 4 10 2024-02-23T00:33:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894122770 osd=4 4 2024-02-23T00:33:00.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 10 client.37175.objecter _send_command 10 2024-02-23T00:33:00.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.420+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd894127f70 con 0x7fd894122df0 2024-02-23T00:33:00.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd894127f70 con 0x7fd894122df0 2024-02-23T00:33:00.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:00.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd892ffd700 10 client.37175.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894122770 osd=4 4 2024-02-23T00:33:00.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 4 10 2024-02-23T00:33:00.424 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:33:00.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 10 client.37175.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:00.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 11 2024-02-23T00:33:00.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd8941272d0 0x7fd8941276f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894127f70 osd=5 3 2024-02-23T00:33:00.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127f70 osd=5 3 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894127f70 osd=5 3 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 11 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894127f70 osd=5 3 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 5 11 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127f70 osd=5 4 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 10 client.37175.objecter _send_command 11 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.422+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd894127d60 con 0x7fd8941272d0 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.423+0000 7fd89961e700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd8941272d0 0x7fd8941276f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.423+0000 7fd892ffd700 10 client.37175.objecter ms_handle_connect 0x7fd8941272d0 2024-02-23T00:33:00.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.423+0000 7fd89961e700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd8941272d0 0x7fd8941276f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.424+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd894127d60 con 0x7fd8941272d0 2024-02-23T00:33:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.424+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:33:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.424+0000 7fd892ffd700 10 client.37175.objecter _finish_command 11 = system:0 2024-02-23T00:33:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.424+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894127f70 osd=5 4 2024-02-23T00:33:00.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.424+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 5 11 2024-02-23T00:33:00.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 10 client.37175.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:00.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 12 2024-02-23T00:33:00.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894127f70 osd=5 3 2024-02-23T00:33:00.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127f70 osd=5 3 2024-02-23T00:33:00.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894127f70 osd=5 3 2024-02-23T00:33:00.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 12 2024-02-23T00:33:00.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894127f70 osd=5 3 2024-02-23T00:33:00.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 5 12 2024-02-23T00:33:00.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127f70 osd=5 4 2024-02-23T00:33:00.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 10 client.37175.objecter _send_command 12 2024-02-23T00:33:00.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.439+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd894127d60 con 0x7fd8941272d0 2024-02-23T00:33:00.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd894127d60 con 0x7fd8941272d0 2024-02-23T00:33:00.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:00.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd892ffd700 10 client.37175.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894127f70 osd=5 4 2024-02-23T00:33:00.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 5 12 2024-02-23T00:33:00.442 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:33:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 10 client.37175.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 13 2024-02-23T00:33:00.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd89412a730 0x7fd89412ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894127d60 osd=6 3 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127d60 osd=6 3 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894127d60 osd=6 3 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 13 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894127d60 osd=6 3 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 6 13 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127d60 osd=6 4 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 10 client.37175.objecter _send_command 13 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.440+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd89412b140 con 0x7fd89412a730 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.441+0000 7fd89a620700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd89412a730 0x7fd89412ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.441+0000 7fd892ffd700 10 client.37175.objecter ms_handle_connect 0x7fd89412a730 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.441+0000 7fd89a620700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd89412a730 0x7fd89412ab90 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.442+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd89412b140 con 0x7fd89412a730 2024-02-23T00:33:00.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.442+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:33:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.442+0000 7fd892ffd700 10 client.37175.objecter _finish_command 13 = system:0 2024-02-23T00:33:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.442+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894127d60 osd=6 4 2024-02-23T00:33:00.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.442+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 6 13 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 10 client.37175.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 14 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894127d60 osd=6 3 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127d60 osd=6 3 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd894127d60 osd=6 3 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 14 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd894127d60 osd=6 3 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 6 14 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127d60 osd=6 4 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 10 client.37175.objecter _send_command 14 2024-02-23T00:33:00.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.456+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd89412b140 con 0x7fd89412a730 2024-02-23T00:33:00.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd89412b140 con 0x7fd89412a730 2024-02-23T00:33:00.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:00.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd892ffd700 10 client.37175.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd894127d60 osd=6 4 2024-02-23T00:33:00.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 6 14 2024-02-23T00:33:00.459 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:33:00.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 10 client.37175.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:00.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 15 2024-02-23T00:33:00.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6812/373787000 conn(0x7fd894126980 0x7fd894125ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:00.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd89412b140 osd=7 3 2024-02-23T00:33:00.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89412b140 osd=7 3 2024-02-23T00:33:00.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd89412b140 osd=7 3 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 15 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd89412b140 osd=7 3 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 7 15 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89412b140 osd=7 4 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 10 client.37175.objecter _send_command 15 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd894126350 con 0x7fd894126980 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd899e1f700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6812/373787000 conn(0x7fd894126980 0x7fd894125ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd892ffd700 10 client.37175.objecter ms_handle_connect 0x7fd894126980 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.458+0000 7fd899e1f700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6812/373787000 conn(0x7fd894126980 0x7fd894125ce0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.459+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd894126350 con 0x7fd894126980 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.459+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.459+0000 7fd892ffd700 10 client.37175.objecter _finish_command 15 = system:0 2024-02-23T00:33:00.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.459+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd89412b140 osd=7 4 2024-02-23T00:33:00.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.459+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 7 15 2024-02-23T00:33:00.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 10 client.37175.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:00.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign -1 16 2024-02-23T00:33:00.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd89412b140 osd=7 3 2024-02-23T00:33:00.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89412b140 osd=7 3 2024-02-23T00:33:00.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 20 client.37175.objecter _get_session s=0x7fd89412b140 osd=7 3 2024-02-23T00:33:00.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 15 client.37175.objecter _session_command_op_remove -1 16 2024-02-23T00:33:00.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 20 client.37175.objecter get_session s=0x7fd89412b140 osd=7 3 2024-02-23T00:33:00.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 15 client.37175.objecter _session_command_op_assign 7 16 2024-02-23T00:33:00.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89412b140 osd=7 4 2024-02-23T00:33:00.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 10 client.37175.objecter _send_command 16 2024-02-23T00:33:00.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.473+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fd894126350 con 0x7fd894126980 2024-02-23T00:33:00.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.475+0000 7fd892ffd700 1 -- 172.21.15.87:0/1544669981 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fd894126350 con 0x7fd894126980 2024-02-23T00:33:00.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.475+0000 7fd892ffd700 10 client.37175.objecter ms_dispatch 0x7fd894001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:00.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.475+0000 7fd892ffd700 10 client.37175.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.475+0000 7fd892ffd700 20 client.37175.objecter put_session s=0x7fd89412b140 osd=7 4 2024-02-23T00:33:00.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.475+0000 7fd892ffd700 15 client.37175.objecter _session_command_op_remove 7 16 2024-02-23T00:33:00.477 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:33:00.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 librados: watch_flush enter 2024-02-23T00:33:00.478 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 librados: watch_flush exit 2024-02-23T00:33:00.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 client.37175.objecter close_session for osd.0 2024-02-23T00:33:00.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd8940598f0 msgr2=0x7fd894059cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd8940598f0 0x7fd894059cf0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89406c5b0 osd=0 1 2024-02-23T00:33:00.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 client.37175.objecter close_session for osd.1 2024-02-23T00:33:00.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6805/685354171 conn(0x7fd8940594d0 msgr2=0x7fd89411b240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6805/685354171 conn(0x7fd8940594d0 0x7fd89411b240 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894059360 osd=1 1 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 client.37175.objecter close_session for osd.2 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd8940feca0 msgr2=0x7fd89411ea90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd8940feca0 0x7fd89411ea90 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd8940feb30 osd=2 1 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 client.37175.objecter close_session for osd.3 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6813/128059790 conn(0x7fd894027770 msgr2=0x7fd894122280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6813/128059790 conn(0x7fd894027770 0x7fd894122280 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89a620700 1 -- 172.21.15.87:0/1544669981 reap_dead start 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894027600 osd=3 1 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 client.37175.objecter close_session for osd.4 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd894122df0 msgr2=0x7fd8941251e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd894122df0 0x7fd8941251e0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894122770 osd=4 1 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 client.37175.objecter close_session for osd.5 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd8941272d0 msgr2=0x7fd8941276f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd8941272d0 0x7fd8941276f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127f70 osd=5 1 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 client.37175.objecter close_session for osd.6 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd89412a730 msgr2=0x7fd89412ab90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd89412a730 0x7fd89412ab90 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd894127d60 osd=6 1 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 client.37175.objecter close_session for osd.7 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6812/373787000 conn(0x7fd894126980 msgr2=0x7fd894125ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.171:6812/373787000 conn(0x7fd894126980 0x7fd894125ce0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 20 client.37175.objecter put_session s=0x7fd89412b140 osd=7 1 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 20 client.37175.objecter shutdown clearing up homeless session... 2024-02-23T00:33:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 10 client.37175.objecter successfully canceled tick 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89a620700 1 -- 172.21.15.87:0/1544669981 reap_dead start 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd87c062c90 msgr2=0x7fd87c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd87c062c90 0x7fd87c065150 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 rx=0x7fd89406c870 tx=0x7fd888007f90).stop 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8940f8420 msgr2=0x7fd89406b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8940f8420 0x7fd89406b0d0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7fd88c00db50 tx=0x7fd88c004ee0).stop 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 shutdown_connections 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd87c062c90 0x7fd87c065150 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 --2- 172.21.15.87:0/1544669981 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8940f8420 0x7fd89406b0d0 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 >> 172.21.15.87:0/1544669981 conn(0x7fd8940f3800 msgr2=0x7fd8940f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 shutdown_connections 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 -- 172.21.15.87:0/1544669981 wait complete. 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:00.476+0000 7fd89c083700 1 librados: shutdown 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.482 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.483 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.483 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.483 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.483 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.483 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:33:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:58.806460+0000 osd.2 (osd.2) 107 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:33:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:58.808098+0000 osd.2 (osd.2) 108 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:33:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:58.972320+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:33:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:58.975330+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:33:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.432918+0000 osd.5 (osd.5) 109 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.434075+0000 osd.5 (osd.5) 110 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.455569+0000 osd.6 (osd.6) 93 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:32:59.456799+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.459049+0000 osd.6 (osd.6) 94 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.679851+0000 osd.7 (osd.7) 121 : cluster [DBG] 19.9 deep-scrub starts 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.681029+0000 osd.7 (osd.7) 122 : cluster [DBG] 19.9 deep-scrub ok 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.022859+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi087-82011-33"}]': finished 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:00.022993+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.035218+0000 mon.a (mon.0) 1670 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi087-82011-33"}]: dispatch 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.039791+0000 mon.c (mon.2) 272 : audit [INF] from='client.? 172.21.15.87:0/2545760482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.040183+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.054396+0000 mon.b (mon.1) 206 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.054664+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.268789+0000 mon.c (mon.2) 273 : audit [INF] from='client.? 172.21.15.87:0/3532636864' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:00.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.269109+0000 mon.a (mon.0) 1673 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:01.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.026+0000 7fa90cff9700 1 -- 172.21.15.87:0/3532636864 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v230) v1 ==== 87+0+0 (secure 0 0 0) 0x7fa9080516b0 con 0x7fa910102210 2024-02-23T00:33:01.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.029+0000 7fa915e9b700 10 librados: watch_flush enter 2024-02-23T00:33:01.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.029+0000 7fa915e9b700 10 librados: watch_flush exit 2024-02-23T00:33:01.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.029+0000 7fa915e9b700 20 client.37169.objecter shutdown clearing up homeless session... 2024-02-23T00:33:01.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.029+0000 7fa915e9b700 10 client.37169.objecter successfully canceled tick 2024-02-23T00:33:01.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.029+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa8fc062c90 msgr2=0x7fa8fc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:01.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.029+0000 7fa915e9b700 1 --2- 172.21.15.87:0/3532636864 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa8fc062c90 0x7fa8fc065150 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7fa900001530 tx=0x7fa900004c50).stop 2024-02-23T00:33:01.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.029+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 msgr2=0x7fa9100f7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:01.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.029+0000 7fa915e9b700 1 --2- 172.21.15.87:0/3532636864 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 0x7fa9100f7640 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fa908014f00 tx=0x7fa90800b040).stop 2024-02-23T00:33:01.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.030+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 shutdown_connections 2024-02-23T00:33:01.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.030+0000 7fa915e9b700 1 --2- 172.21.15.87:0/3532636864 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa8fc062c90 0x7fa8fc065150 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:01.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.030+0000 7fa915e9b700 1 --2- 172.21.15.87:0/3532636864 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa910102210 0x7fa9100f7640 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:01.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.030+0000 7fa915e9b700 1 --2- 172.21.15.87:0/3532636864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9100f8cb0 0x7fa9100f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:01.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.030+0000 7fa915e9b700 1 --2- 172.21.15.87:0/3532636864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9100f9610 0x7fa9100f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:01.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.030+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 >> 172.21.15.87:0/3532636864 conn(0x7fa9100f2820 msgr2=0x7fa9100f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:01.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.042+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 shutdown_connections 2024-02-23T00:33:01.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.045+0000 7fa915e9b700 1 -- 172.21.15.87:0/3532636864 wait complete. 2024-02-23T00:33:01.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:01.048+0000 7fa915e9b700 1 librados: shutdown 2024-02-23T00:33:01.052 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:33:01.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:33:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:33:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.593729+0000 mgr.y (mgr.24437) 258 : cluster [DBG] pgmap v325: 283 pgs: 32 unknown, 1 clean+premerge+peered, 233 active+clean, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.848488+0000 osd.2 (osd.2) 109 : cluster [DBG] 2.c scrub starts 2024-02-23T00:33:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:32:59.849497+0000 osd.2 (osd.2) 110 : cluster [DBG] 2.c scrub ok 2024-02-23T00:33:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.329544+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? 172.21.15.87:0/1544669981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:00.438847+0000 osd.5 (osd.5) 111 : cluster [DBG] 6.7 deep-scrub starts 2024-02-23T00:33:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:00.439867+0000 osd.5 (osd.5) 112 : cluster [DBG] 6.7 deep-scrub ok 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.457582+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:00.668236+0000 osd.7 (osd.7) 123 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:00.671215+0000 osd.7 (osd.7) 124 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:01.018296+0000 mon.a (mon.0) 1674 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.022613+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi087-82011-33"}]': finished 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.022730+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.022806+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.022879+0000 mon.a (mon.0) 1678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:01.022939+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.028956+0000 mon.b (mon.1) 207 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.031697+0000 mon.a (mon.0) 1680 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.056867+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.299916+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.300427+0000 mon.a (mon.0) 1683 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi087-82011-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.593729+0000 mgr.y (mgr.24437) 258 : cluster [DBG] pgmap v325: 283 pgs: 32 unknown, 1 clean+premerge+peered, 233 active+clean, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.848488+0000 osd.2 (osd.2) 109 : cluster [DBG] 2.c scrub starts 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:32:59.849497+0000 osd.2 (osd.2) 110 : cluster [DBG] 2.c scrub ok 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.329544+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? 172.21.15.87:0/1544669981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:00.438847+0000 osd.5 (osd.5) 111 : cluster [DBG] 6.7 deep-scrub starts 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:00.439867+0000 osd.5 (osd.5) 112 : cluster [DBG] 6.7 deep-scrub ok 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.457582+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:00.668236+0000 osd.7 (osd.7) 123 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:00.671215+0000 osd.7 (osd.7) 124 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:01.018296+0000 mon.a (mon.0) 1674 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.022613+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi087-82011-33"}]': finished 2024-02-23T00:33:01.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.022730+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:01.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.022806+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:01.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.022879+0000 mon.a (mon.0) 1678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:33:01.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:01.022939+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-02-23T00:33:01.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.028956+0000 mon.b (mon.1) 207 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:33:01.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.031697+0000 mon.a (mon.0) 1680 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:33:01.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.056867+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:01.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.299916+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:01.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.300427+0000 mon.a (mon.0) 1683 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi087-82011-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.593729+0000 mgr.y (mgr.24437) 258 : cluster [DBG] pgmap v325: 283 pgs: 32 unknown, 1 clean+premerge+peered, 233 active+clean, 11 active+clean+snaptrim_wait, 6 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.848488+0000 osd.2 (osd.2) 109 : cluster [DBG] 2.c scrub starts 2024-02-23T00:33:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:32:59.849497+0000 osd.2 (osd.2) 110 : cluster [DBG] 2.c scrub ok 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.329544+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? 172.21.15.87:0/1544669981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:00.438847+0000 osd.5 (osd.5) 111 : cluster [DBG] 6.7 deep-scrub starts 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:00.439867+0000 osd.5 (osd.5) 112 : cluster [DBG] 6.7 deep-scrub ok 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.457582+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:00.668236+0000 osd.7 (osd.7) 123 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:00.671215+0000 osd.7 (osd.7) 124 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:01.018296+0000 mon.a (mon.0) 1674 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.022613+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? 172.21.15.87:0/3645231206' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi087-82011-33"}]': finished 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.022730+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.022806+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.022879+0000 mon.a (mon.0) 1678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:01.022939+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.028956+0000 mon.b (mon.1) 207 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:33:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.031697+0000 mon.a (mon.0) 1680 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23"}]: dispatch 2024-02-23T00:33:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.056867+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.299916+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.300427+0000 mon.a (mon.0) 1683 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi087-82011-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:02.321 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:02] "GET /metrics HTTP/1.1" 200 195443 "" "Prometheus/2.33.4" 2024-02-23T00:33:02.360 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:02.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:02.465 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:02.494 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:00.739050+0000 mgr.y (mgr.24437) 259 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:00.877472+0000 osd.2 (osd.2) 111 : cluster [DBG] 2.e scrub starts 2024-02-23T00:33:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:00.878388+0000 osd.2 (osd.2) 112 : cluster [DBG] 2.e scrub ok 2024-02-23T00:33:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.458491+0000 mon.c (mon.2) 276 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:01.595763+0000 mon.a (mon.0) 1684 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-23T00:33:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:02.002146+0000 mon.a (mon.0) 1685 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:02.009950+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23"}]': finished 2024-02-23T00:33:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:02.010067+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi087-82011-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:02.010126+0000 mon.a (mon.0) 1688 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-23T00:33:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:02.010170+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-02-23T00:33:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:02.014856+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi087-82011-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:02.023770+0000 mon.b (mon.1) 208 : audit [INF] from='client.? 172.21.15.87:0/1117214694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:02.024646+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:00.739050+0000 mgr.y (mgr.24437) 259 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:00.877472+0000 osd.2 (osd.2) 111 : cluster [DBG] 2.e scrub starts 2024-02-23T00:33:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:00.878388+0000 osd.2 (osd.2) 112 : cluster [DBG] 2.e scrub ok 2024-02-23T00:33:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.458491+0000 mon.c (mon.2) 276 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:01.595763+0000 mon.a (mon.0) 1684 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-23T00:33:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:02.002146+0000 mon.a (mon.0) 1685 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:02.009950+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23"}]': finished 2024-02-23T00:33:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:02.010067+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi087-82011-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:02.010126+0000 mon.a (mon.0) 1688 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-23T00:33:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:02.010170+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-02-23T00:33:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:02.014856+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi087-82011-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:02.023770+0000 mon.b (mon.1) 208 : audit [INF] from='client.? 172.21.15.87:0/1117214694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:02.024646+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:02.643 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:02.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:02.651 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:00.739050+0000 mgr.y (mgr.24437) 259 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:00.877472+0000 osd.2 (osd.2) 111 : cluster [DBG] 2.e scrub starts 2024-02-23T00:33:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:00.878388+0000 osd.2 (osd.2) 112 : cluster [DBG] 2.e scrub ok 2024-02-23T00:33:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.458491+0000 mon.c (mon.2) 276 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:01.595763+0000 mon.a (mon.0) 1684 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-23T00:33:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:02.002146+0000 mon.a (mon.0) 1685 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:02.009950+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-23"}]': finished 2024-02-23T00:33:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:02.010067+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi087-82011-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:02.010126+0000 mon.a (mon.0) 1688 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi087-82051-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-23T00:33:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:02.010170+0000 mon.a (mon.0) 1689 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-02-23T00:33:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:02.014856+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi087-82011-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:02.023770+0000 mon.b (mon.1) 208 : audit [INF] from='client.? 172.21.15.87:0/1117214694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:02.024646+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:02.752 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:02.781 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:02.879 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:02.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:02.984 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:03.004 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:33:03.004 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:33:03.004 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:33:03.004 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:33:03.004 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.004 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24757.0:10 21.9 21:928daad9:test-rados-api-smithi087-82107-4::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:30:57.492284+0000", 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 125.504185877, 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019562380000000001, 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24757", 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3080165175", 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 10 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.005 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492284+0000", 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492284+0000", 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999971 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492281+0000", 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3389e-05 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.006 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492295+0000", 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3093999999999999e-05 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492318+0000", 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7960000000000007e-06 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492327+0000", 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000104923 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.007 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492432+0000", 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9130999999999997e-05 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492531+0000", 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00020787600000000001 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492739+0000", 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010449754 2024-02-23T00:33:03.008 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.503189+0000", 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00069305099999999998 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.503882+0000", 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0078638679999999996 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.511746+0000", 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2425e-05 2024-02-23T00:33:03.009 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.511798+0000", 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7735000000000003e-05 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.511846+0000", 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019562380000000001 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.010 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.24712.0:14 7.5 e108/94 7:a3910452:test-rados-api-smithi087-82115-5::foo:head v 108'1, mlcod=0'0)", 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:30:58.510883+0000", 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 124.48558638900001, 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013710257, 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510883+0000", 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.011 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510883+0000", 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510883+0000", 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0350000000000001e-06 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510888+0000", 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.759e-06 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.012 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510895+0000", 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6359999999999994e-06 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510902+0000", 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000288127 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.511190+0000", 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.7677e-05 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.013 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.511208+0000", 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013359852 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.524568+0000", 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5661999999999999e-05 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.524594+0000", 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013710257 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.014 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:3 61.1f e114/112 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head v 114'1, mlcod=0'0)", 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:05.241186+0000", 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 117.75528382900001, 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0078894910000000002, 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241186+0000", 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.015 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241186+0000", 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241184+0000", 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2142e-05 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241196+0000", 2024-02-23T00:33:03.016 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4297999999999999e-05 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241210+0000", 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3780000000000003e-06 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241218+0000", 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0946999999999998e-05 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.017 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241279+0000", 2024-02-23T00:33:03.018 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0699999999999998e-05 2024-02-23T00:33:03.018 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241299+0000", 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0077120080000000002 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.249011+0000", 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3965000000000002e-05 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.020 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.249075+0000", 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0078894910000000002 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:10 61.1f e118/112 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head v 118'6, mlcod=115'3)", 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.218637+0000", 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 113.777832832, 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016531127999999999, 2024-02-23T00:33:03.021 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218637+0000", 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218637+0000", 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.022 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218637+0000", 2024-02-23T00:33:03.023 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5999999999999997e-06 2024-02-23T00:33:03.023 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.033 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.033 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.033 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218642+0000", 2024-02-23T00:33:03.033 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6220000000000003e-06 2024-02-23T00:33:03.033 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.033 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218649+0000", 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.703e-06 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218655+0000", 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010198169 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.228854+0000", 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5311e-05 2024-02-23T00:33:03.034 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.228869+0000", 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0062654290000000003 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.235134+0000", 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3626e-05 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.235168+0000", 2024-02-23T00:33:03.035 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016531127999999999 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27712.0:3 67.1f 67:f9c13a70:test-rados-api-smithi087-82115-8::foo:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected e119)", 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.170310+0000", 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 112.826159413, 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062446785999999997, 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27712", 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2607672493", 2024-02-23T00:33:03.036 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 3 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170310+0000", 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170310+0000", 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.037 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170308+0000", 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.1270000000000003e-06 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170316+0000", 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3158e-05 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170329+0000", 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2034e-05 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.038 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170341+0000", 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0090999999999999e-05 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170391+0000", 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4679999999999997e-06 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170398+0000", 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.047525959 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.039 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.217924+0000", 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000136811 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.218061+0000", 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00018028200000000001 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.218241+0000", 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0017470389999999999 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.040 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.219988+0000", 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.01262332 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.232611+0000", 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6244e-05 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.232658+0000", 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6916999999999997e-05 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.041 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.232714+0000", 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2565999999999999e-05 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.232757+0000", 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062446785999999997 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:807 5.3 5:c609908c:::notify.5:head [watch ping cookie 94532396652160] snapc 0=[] ondisk+write+known_if_redirected e114)", 2024-02-23T00:33:03.042 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.382657+0000", 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 112.613812747, 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010145018, 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 807 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382657+0000", 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.043 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382657+0000", 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382656+0000", 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0860000000000001e-06 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382661+0000", 2024-02-23T00:33:03.044 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5639999999999999e-06 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382669+0000", 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5300000000000002e-06 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382675+0000", 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0099372450000000008 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.392612+0000", 2024-02-23T00:33:03.045 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5774999999999998e-05 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.392698+0000", 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000103871 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.392802+0000", 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010145018 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.046 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27712.0:5 67.1f 67:f9c13a70:test-rados-api-smithi087-82115-8::foo:head [write 0~128] snapc 3=[3,2] ondisk+write+known_if_redirected e120)", 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.171931+0000", 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 111.824538307, 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.22139420000000001, 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27712", 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2607672493", 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 5 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.047 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171931+0000", 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171931+0000", 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171930+0000", 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5570000000000001e-06 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.048 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171937+0000", 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0461e-05 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171947+0000", 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9319999999999996e-06 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171955+0000", 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0064871850000000003 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.049 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.178442+0000", 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1840000000000002e-06 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.178449+0000", 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.18818955100000001 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.366639+0000", 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2639000000000005e-05 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.050 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.366731+0000", 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00018480499999999999 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.366916+0000", 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0048910660000000003 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.371807+0000", 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021056603 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:03.051 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.392864+0000", 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00030953399999999998 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.393173+0000", 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6799000000000006e-05 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.393270+0000", 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5291999999999998e-05 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.052 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.393326+0000", 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.22139420000000001 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:37 66.17 66:e8000000:%01:::head [pgnls start_epoch 120] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e120)", 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.430859+0000", 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 111.56561103, 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0083297739999999999, 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:33:03.053 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 37 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430859+0000", 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430859+0000", 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.054 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430856+0000", 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0530000000000001e-05 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430867+0000", 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4768e-05 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430882+0000", 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4930000000000001e-06 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.055 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430889+0000", 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0081343149999999996 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439024+0000", 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5954000000000001e-05 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439069+0000", 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011896399999999999 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.056 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439188+0000", 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0083297739999999999 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27712.0:57 67.3 e138/117 67:c06d7414:test-rados-api-smithi087-82115-12::foo:head v 138'1, mlcod=0'0)", 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:29.024212+0000", 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 93.972257310000003, 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024530395999999999, 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.057 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.059 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.059 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024212+0000", 2024-02-23T00:33:03.059 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.059 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.059 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.059 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.059 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024212+0000", 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024212+0000", 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7690000000000002e-06 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024221+0000", 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4880000000000004e-06 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.060 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024227+0000", 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6460000000000004e-06 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024234+0000", 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0069832150000000001 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.031217+0000", 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1175e-05 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.061 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.031238+0000", 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017474302000000001 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.048713+0000", 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0083999999999999e-05 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.048743+0000", 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024530395999999999 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDScrubReserve(10.1 REQUEST e138)", 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.013579+0000", 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 92.982890482000002, 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010003164, 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013579+0000", 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.063 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013579+0000", 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013577+0000", 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4552000000000001e-05 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013592+0000", 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2685999999999999e-05 2024-02-23T00:33:03.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013614+0000", 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0902e-05 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013625+0000", 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0098954390000000007 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.023521+0000", 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2829000000000001e-05 2024-02-23T00:33:03.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.023543+0000", 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8949000000000003e-05 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.023582+0000", 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010003164 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.4.0:339 10.1 e138/94 10:81ba0521:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa::a:head v 138'2, mlcod=0'0)", 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.026047+0000", 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 92.970423031999999, 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0074907230000000003, 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026047+0000", 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.067 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026047+0000", 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026045+0000", 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6579999999999997e-06 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026055+0000", 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0818999999999999e-05 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026065+0000", 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7590000000000003e-06 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026071+0000", 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5367000000000002e-05 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026107+0000", 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6110999999999999e-05 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026123+0000", 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0073568699999999997 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.033480+0000", 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7766999999999998e-05 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.033537+0000", 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0074907230000000003 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:672 63.6 e147/114 63:6271ce74:test-rados-api-smithi087-82091-14::foo0:head v 147'17, mlcod=146'13)", 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:37.995603+0000", 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 85.000866708000004, 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012932641, 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995603+0000", 2024-02-23T00:33:03.071 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995603+0000", 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995603+0000", 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4840000000000003e-06 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.072 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995608+0000", 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9030000000000001e-06 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995614+0000", 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5999999999999997e-06 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995620+0000", 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1352000000000005e-05 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.073 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995681+0000", 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0752000000000001e-05 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995702+0000", 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012792253 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.008494+0000", 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1677000000000001e-05 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.074 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.008536+0000", 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012932641 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.36722.0:2 99.6 99:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e149)", 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:39.997356+0000", 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 82.999114113000005, 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0097361589999999994, 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.36722", 2024-02-23T00:33:03.075 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2157738800", 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997356+0000", 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997356+0000", 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.076 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997354+0000", 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3889999999999999e-06 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997361+0000", 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0131e-05 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997371+0000", 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9399999999999996e-06 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.077 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997378+0000", 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0971999999999999e-05 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997429+0000", 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8999999999999995e-06 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997438+0000", 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003947908 2024-02-23T00:33:03.078 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.001386+0000", 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012212799999999999 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.001508+0000", 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000146413 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,4", 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.001655+0000", 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00048154200000000001 2024-02-23T00:33:03.079 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.002136+0000", 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000286619 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.002423+0000", 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0045833820000000004 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.007006+0000", 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4014000000000002e-05 2024-02-23T00:33:03.080 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.007050+0000", 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1578000000000001e-05 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.007092+0000", 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0097361589999999994 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.081 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.2.0:279 63.16 e182/114 63:689df377:test-rados-api-smithi087-82091-14::foo54:7 v 182'31, mlcod=148'25)", 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.012421+0000", 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 49.984048201, 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014084329, 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012421+0000", 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012421+0000", 2024-02-23T00:33:03.082 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012421+0000", 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9580000000000001e-06 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012425+0000", 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9779999999999996e-06 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.083 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012430+0000", 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9790000000000004e-06 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012434+0000", 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00029118500000000001 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012725+0000", 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6079999999999999e-05 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.084 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012741+0000", 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013727184999999999 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.026468+0000", 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7320000000000002e-05 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.026506+0000", 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014084329 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.085 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1311 63.12 e182/114 63:4961eb90:test-rados-api-smithi087-82091-14::foo67:7 v 182'38, mlcod=148'34)", 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.013199+0000", 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 49.983270832999999, 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018593278000000001, 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013199+0000", 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.086 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013199+0000", 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013198+0000", 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7669999999999998e-06 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013205+0000", 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5710000000000004e-06 2024-02-23T00:33:03.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013215+0000", 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0329999999999997e-06 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013221+0000", 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001101557 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.014322+0000", 2024-02-23T00:33:03.088 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6117000000000001e-05 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.014349+0000", 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017406123999999999 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.031755+0000", 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7462000000000001e-05 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.031792+0000", 2024-02-23T00:33:03.089 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018593278000000001 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1312 63.12 e182/114 63:49181275:test-rados-api-smithi087-82091-14::foo3:7 v 182'40, mlcod=148'34)", 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.013227+0000", 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 49.983243149000003, 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019144881999999998, 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.090 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013227+0000", 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013227+0000", 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013226+0000", 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7740000000000002e-06 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.091 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013230+0000", 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6500000000000004e-06 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013235+0000", 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2967e-05 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013248+0000", 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0072151000000000003 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.092 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.020463+0000", 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9933e-05 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.020483+0000", 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011861216000000001 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.032344+0000", 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.76e-05 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.093 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.032371+0000", 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019144881999999998 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1313 63.14 e182/114 63:2ca15095:test-rados-api-smithi087-82091-14::foo41:7 v 182'47, mlcod=148'43)", 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.017065+0000", 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 49.979404185999996, 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015321398999999999, 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.094 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017065+0000", 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017065+0000", 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017065+0000", 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2177999999999999e-05 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017077+0000", 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8810000000000002e-06 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017086+0000", 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3049999999999998e-06 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017092+0000", 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0015861499999999999 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.096 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.018678+0000", 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0152e-05 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.018698+0000", 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013653176 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.032352+0000", 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5281999999999999e-05 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.097 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.032387+0000", 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015321398999999999 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1314 63.14 e182/114 63:2cd9217a:test-rados-api-smithi087-82091-14::foo89:7 v 182'49, mlcod=148'43)", 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.021891+0000", 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 49.974579124999998, 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011822889, 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.098 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021891+0000", 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021891+0000", 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021890+0000", 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0390000000000005e-06 2024-02-23T00:33:03.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021898+0000", 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5710000000000003e-06 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021905+0000", 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2120000000000002e-06 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021910+0000", 2024-02-23T00:33:03.100 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0060575969999999996 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027967+0000", 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1897999999999999e-05 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027989+0000", 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0056844479999999999 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.033674+0000", 2024-02-23T00:33:03.101 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9691999999999999e-05 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.033713+0000", 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011822889 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.37106.0:2 149.1a e216/215 149:5ec95afd:::oid_ReadIntoBufferlist_:head v 216'1, mlcod=0'0)", 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.056252+0000", 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 15.940217435999999, 2024-02-23T00:33:03.102 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.049676218000000001, 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056252+0000", 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056252+0000", 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.103 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056252+0000", 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0475e-05 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056272+0000", 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8540000000000002e-06 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056278+0000", 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9689999999999999e-06 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.104 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056284+0000", 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000186269 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056470+0000", 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3541e-05 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056484+0000", 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.049403124999999999 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.105 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.105887+0000", 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1409999999999998e-05 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.105928+0000", 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.049676218000000001 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:1811 148.8 148:15d27fd9:test-rados-api-smithi087-82091-22::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e219)", 2024-02-23T00:33:03.106 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:50.035563+0000", 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 12.96090637, 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010372207, 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1811 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035563+0000", 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:03.107 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035563+0000", 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035561+0000", 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3500000000000002e-06 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035568+0000", 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2699999999999993e-06 2024-02-23T00:33:03.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035577+0000", 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.238e-06 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035584+0000", 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002828269 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.038413+0000", 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021697099999999999 2024-02-23T00:33:03.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.038630+0000", 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0024583119999999998 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.041088+0000", 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6991999999999997e-05 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for rw locks", 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.041165+0000", 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0042903780000000001 2024-02-23T00:33:03.110 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.045455+0000", 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7844999999999997e-05 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.045543+0000", 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00039248100000000003 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.045936+0000", 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010372207 2024-02-23T00:33:03.111 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.112 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.112 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.112 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:03.112 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:03.112 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:33:03.112 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:03.468 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:03.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:03.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:03.571 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:03.605 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:03.627 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:03.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:03.731 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:03.759 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:03.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:33:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:33:03.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:01.594141+0000 mgr.y (mgr.24437) 260 : cluster [DBG] pgmap v328: 314 pgs: 1 peering, 32 creating+peering, 259 active+clean, 17 active+clean+snaptrim_wait, 5 active+clean+snaptrim; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s wr, 6 op/s; 63 B/s, 0 objects/s recovering 2024-02-23T00:33:03.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:02.459985+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:03.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:03.251663+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:03.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:03.251786+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-02-23T00:33:04.007 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:04.009 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:01.594141+0000 mgr.y (mgr.24437) 260 : cluster [DBG] pgmap v328: 314 pgs: 1 peering, 32 creating+peering, 259 active+clean, 17 active+clean+snaptrim_wait, 5 active+clean+snaptrim; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s wr, 6 op/s; 63 B/s, 0 objects/s recovering 2024-02-23T00:33:04.010 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:02.459985+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:04.010 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:03.251663+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:04.010 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:03.251786+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-02-23T00:33:04.010 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:01.594141+0000 mgr.y (mgr.24437) 260 : cluster [DBG] pgmap v328: 314 pgs: 1 peering, 32 creating+peering, 259 active+clean, 17 active+clean+snaptrim_wait, 5 active+clean+snaptrim; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s wr, 6 op/s; 63 B/s, 0 objects/s recovering 2024-02-23T00:33:04.011 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:02.459985+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:04.011 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:03.251663+0000 mon.a (mon.0) 1692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:04.011 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:03.251786+0000 mon.a (mon.0) 1693 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-02-23T00:33:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:04.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:04.066 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:33:04.113 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:04.146 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:04.167 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:04.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:04.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:04.271 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:04.300 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:04.321 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:04.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:04.426 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:04.452 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:04.505 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:04.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:04.529 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-23T00:33:04.611 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:04.642 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:04.663 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:04.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:04.767 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:04.794 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:03.460757+0000 mon.c (mon.2) 278 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:03.594605+0000 mgr.y (mgr.24437) 261 : cluster [DBG] pgmap v331: 314 pgs: 1 peering, 32 unknown, 32 creating+peering, 235 active+clean, 9 active+clean+snaptrim_wait, 5 active+clean+snaptrim; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-02-23T00:33:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:04.019498+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi087-82011-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi087-82011-34"}]': finished 2024-02-23T00:33:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:04.019648+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-02-23T00:33:04.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:04.028332+0000 mon.c (mon.2) 279 : audit [INF] from='client.? 172.21.15.87:0/289455948' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:04.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:04.029157+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:04.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:04.031371+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:04.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:04.031993+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:04.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:04.462016+0000 mon.c (mon.2) 280 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:05.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:03.460757+0000 mon.c (mon.2) 278 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:05.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:03.594605+0000 mgr.y (mgr.24437) 261 : cluster [DBG] pgmap v331: 314 pgs: 1 peering, 32 unknown, 32 creating+peering, 235 active+clean, 9 active+clean+snaptrim_wait, 5 active+clean+snaptrim; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-02-23T00:33:05.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:04.019498+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi087-82011-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi087-82011-34"}]': finished 2024-02-23T00:33:05.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:04.019648+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-02-23T00:33:05.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:04.028332+0000 mon.c (mon.2) 279 : audit [INF] from='client.? 172.21.15.87:0/289455948' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:05.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:04.029157+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:05.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:04.031371+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:05.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:04.031993+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:05.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:04.462016+0000 mon.c (mon.2) 280 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:05.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:03.460757+0000 mon.c (mon.2) 278 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:03.594605+0000 mgr.y (mgr.24437) 261 : cluster [DBG] pgmap v331: 314 pgs: 1 peering, 32 unknown, 32 creating+peering, 235 active+clean, 9 active+clean+snaptrim_wait, 5 active+clean+snaptrim; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-02-23T00:33:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:04.019498+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi087-82011-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi087-82011-34"}]': finished 2024-02-23T00:33:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:04.019648+0000 mon.a (mon.0) 1695 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-02-23T00:33:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:04.028332+0000 mon.c (mon.2) 279 : audit [INF] from='client.? 172.21.15.87:0/289455948' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:04.029157+0000 mon.a (mon.0) 1696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:04.031371+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:04.031993+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:04.462016+0000 mon.c (mon.2) 280 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:05.451 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:33:05.574 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:05.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:05.678 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:05.704 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:05.902 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:05.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:06.006 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:06.047 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:06.068 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:06.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:06.132 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.141+0000 7fd832975700 1 -- 172.21.15.87:0/2275366128 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd82c0fdbb0 msgr2=0x7fd82c0fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.141+0000 7fd832975700 1 --2- 172.21.15.87:0/2275366128 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd82c0fdbb0 0x7fd82c0fdf90 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7fd81c004870 tx=0x7fd81c009cb0).stop 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.141+0000 7fd832975700 1 -- 172.21.15.87:0/2275366128 shutdown_connections 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.141+0000 7fd832975700 1 --2- 172.21.15.87:0/2275366128 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd82c0f8420 0x7fd82c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.141+0000 7fd832975700 1 --2- 172.21.15.87:0/2275366128 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd82c0fdbb0 0x7fd82c0fdf90 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.141+0000 7fd832975700 1 --2- 172.21.15.87:0/2275366128 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd82c0fe510 0x7fd82c0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.141+0000 7fd832975700 1 -- 172.21.15.87:0/2275366128 >> 172.21.15.87:0/2275366128 conn(0x7fd82c0f3800 msgr2=0x7fd82c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.141+0000 7fd832975700 1 -- 172.21.15.87:0/2275366128 shutdown_connections 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.141+0000 7fd832975700 1 -- 172.21.15.87:0/2275366128 wait complete. 2024-02-23T00:33:06.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.142+0000 7fd832975700 1 librados: starting msgr at 2024-02-23T00:33:06.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.142+0000 7fd832975700 1 librados: starting objecter 2024-02-23T00:33:06.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.142+0000 7fd832975700 1 Processor -- start 2024-02-23T00:33:06.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.142+0000 7fd832975700 1 -- start start 2024-02-23T00:33:06.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.142+0000 7fd832975700 1 librados: setting wanted keys 2024-02-23T00:33:06.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.142+0000 7fd832975700 1 librados: calling monclient init 2024-02-23T00:33:06.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd832975700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd82c0f8420 0x7fd82c10e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:06.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd832975700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd82c0fdbb0 0x7fd82c10e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:06.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd832975700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd82c0fe510 0x7fd82c1081b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:06.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd832975700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c1086a0 con 0x7fd82c0f8420 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd832975700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c108810 con 0x7fd82c0fdbb0 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd832975700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c1089d0 con 0x7fd82c0fe510 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd82bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd82c0f8420 0x7fd82c10e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd82bfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd82c0f8420 0x7fd82c10e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56744/0 (socket says 172.21.15.87:56744) 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd82bfff700 1 -- 172.21.15.87:0/3657276313 learned_addr learned my addr 172.21.15.87:0/3657276313 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd830f12700 1 --2- 172.21.15.87:0/3657276313 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd82c0fe510 0x7fd82c1081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd82b7fe700 1 --2- 172.21.15.87:0/3657276313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd82c0fdbb0 0x7fd82c10e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd82bfff700 1 -- 172.21.15.87:0/3657276313 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd82c0fe510 msgr2=0x7fd82c1081b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd82bfff700 1 --2- 172.21.15.87:0/3657276313 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd82c0fe510 0x7fd82c1081b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:06.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd82bfff700 1 -- 172.21.15.87:0/3657276313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd82c0fdbb0 msgr2=0x7fd82c10e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd82bfff700 1 --2- 172.21.15.87:0/3657276313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd82c0fdbb0 0x7fd82c10e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.143+0000 7fd82bfff700 1 -- 172.21.15.87:0/3657276313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8240097c0 con 0x7fd82c0f8420 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd82bfff700 1 --2- 172.21.15.87:0/3657276313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd82c0f8420 0x7fd82c10e010 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 rx=0x7fd81c009eb0 tx=0x7fd81c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd8297fa700 10 client.?.objecter ms_handle_connect 0x7fd82c0f8420 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd8297fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd8297fa700 1 -- 172.21.15.87:0/3657276313 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00e070 con 0x7fd82c0f8420 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c005040 con 0x7fd82c0f8420 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd832975700 10 client.26757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd82c108f40 con 0x7fd82c0f8420 2024-02-23T00:33:06.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd832975700 1 librados: init done 2024-02-23T00:33:06.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd8297fa700 1 -- 172.21.15.87:0/3657276313 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd81c01f070 con 0x7fd82c0f8420 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.144+0000 7fd8297fa700 1 -- 172.21.15.87:0/3657276313 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c01b8c0 con 0x7fd82c0f8420 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.145+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd818004f50 con 0x7fd82c0f8420 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.146+0000 7fd8297fa700 1 -- 172.21.15.87:0/3657276313 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd81c0092e0 con 0x7fd82c0f8420 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.146+0000 7fd8297fa700 1 --2- 172.21.15.87:0/3657276313 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd814062c90 0x7fd814065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.147+0000 7fd8297fa700 1 -- 172.21.15.87:0/3657276313 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(235..235 src has 38..235) v4 ==== 10525+0+0 (secure 0 0 0) 0x7fd81c0827c0 con 0x7fd82c0f8420 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.147+0000 7fd8297fa700 10 client.26757.objecter ms_dispatch 0x7fd82c001550 osd_map(235..235 src has 38..235) v4 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.147+0000 7fd8297fa700 3 client.26757.objecter handle_osd_map got epochs [235,235] > 0 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.147+0000 7fd8297fa700 3 client.26757.objecter handle_osd_map decoding full epoch 235 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.147+0000 7fd8297fa700 20 client.26757.objecter dump_active .. 0 homeless 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.147+0000 7fd82b7fe700 1 --2- 172.21.15.87:0/3657276313 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd814062c90 0x7fd814065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:06.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.147+0000 7fd8297fa700 10 client.26757.objecter ms_handle_connect 0x7fd814062c90 2024-02-23T00:33:06.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.147+0000 7fd82b7fe700 1 --2- 172.21.15.87:0/3657276313 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd814062c90 0x7fd814065150 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 rx=0x7fd82c109950 tx=0x7fd82000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:06.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.153+0000 7fd8297fa700 1 -- 172.21.15.87:0/3657276313 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd81c055420 con 0x7fd82c0f8420 2024-02-23T00:33:06.172 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:06.181 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:06.202 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:05.021655+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:06.202 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:05.021798+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:06.202 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:05.021908+0000 mon.a (mon.0) 1700 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-02-23T00:33:06.202 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:05.462819+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:06.202 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:05.021655+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:06.203 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:05.021798+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:06.203 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:05.021908+0000 mon.a (mon.0) 1700 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-02-23T00:33:06.203 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:05.462819+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:06.204 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:06.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.341+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd818000bf0 con 0x7fd814062c90 2024-02-23T00:33:06.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.371+0000 7fd8297fa700 1 -- 172.21.15.87:0/3657276313 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+999783 (secure 0 0 0) 0x7fd818000bf0 con 0x7fd814062c90 2024-02-23T00:33:06.385 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:33:06.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 10 librados: watch_flush enter 2024-02-23T00:33:06.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 10 librados: watch_flush exit 2024-02-23T00:33:06.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 20 client.26757.objecter shutdown clearing up homeless session... 2024-02-23T00:33:06.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 10 client.26757.objecter successfully canceled tick 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd814062c90 msgr2=0x7fd814065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 --2- 172.21.15.87:0/3657276313 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd814062c90 0x7fd814065150 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 rx=0x7fd82c109950 tx=0x7fd82000b040).stop 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd82c0f8420 msgr2=0x7fd82c10e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 --2- 172.21.15.87:0/3657276313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd82c0f8420 0x7fd82c10e010 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 rx=0x7fd81c009eb0 tx=0x7fd81c00b040).stop 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 shutdown_connections 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 --2- 172.21.15.87:0/3657276313 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd814062c90 0x7fd814065150 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 --2- 172.21.15.87:0/3657276313 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd82c0fe510 0x7fd82c1081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 --2- 172.21.15.87:0/3657276313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd82c0f8420 0x7fd82c10e010 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 --2- 172.21.15.87:0/3657276313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd82c0fdbb0 0x7fd82c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 >> 172.21.15.87:0/3657276313 conn(0x7fd82c0f3800 msgr2=0x7fd82c0f53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 shutdown_connections 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 -- 172.21.15.87:0/3657276313 wait complete. 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:06.377+0000 7fd832975700 1 librados: shutdown 2024-02-23T00:33:06.399 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:33:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:05.021655+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:05.021798+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:05.021908+0000 mon.a (mon.0) 1700 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-02-23T00:33:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:05.462819+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:05.596239+0000 mgr.y (mgr.24437) 262 : cluster [DBG] pgmap v334: 385 pgs: 376 active+clean, 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-02-23T00:33:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:06.016723+0000 mon.a (mon.0) 1701 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:33:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:06.021462+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-02-23T00:33:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:06.031618+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:06.040237+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? 172.21.15.87:0/2387320107' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:06.044500+0000 mon.b (mon.1) 210 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:06.044791+0000 mon.a (mon.0) 1705 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:06.463664+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:07.009044+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi087-82011-34"}]': finished 2024-02-23T00:33:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:07.009206+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? 172.21.15.87:0/2387320107' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:07.009274+0000 mon.a (mon.0) 1708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:07.009358+0000 mon.a (mon.0) 1709 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-02-23T00:33:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:07.016567+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:07.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:07.018213+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:07.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:07.021746+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:07.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:05.596239+0000 mgr.y (mgr.24437) 262 : cluster [DBG] pgmap v334: 385 pgs: 376 active+clean, 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-02-23T00:33:07.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:06.016723+0000 mon.a (mon.0) 1701 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:33:07.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:06.021462+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-02-23T00:33:07.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:06.031618+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:06.040237+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? 172.21.15.87:0/2387320107' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:06.044500+0000 mon.b (mon.1) 210 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:06.044791+0000 mon.a (mon.0) 1705 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:06.463664+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:07.009044+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi087-82011-34"}]': finished 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:07.009206+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? 172.21.15.87:0/2387320107' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:07.009274+0000 mon.a (mon.0) 1708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:07.009358+0000 mon.a (mon.0) 1709 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:07.016567+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:07.018213+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:07.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:07.021746+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:05.596239+0000 mgr.y (mgr.24437) 262 : cluster [DBG] pgmap v334: 385 pgs: 376 active+clean, 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-02-23T00:33:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:06.016723+0000 mon.a (mon.0) 1701 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:06.021462+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:06.031618+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:06.040237+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? 172.21.15.87:0/2387320107' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:06.044500+0000 mon.b (mon.1) 210 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:06.044791+0000 mon.a (mon.0) 1705 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:06.463664+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:07.009044+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi087-82011-34"}]': finished 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:07.009206+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? 172.21.15.87:0/2387320107' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi087-82019-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:07.009274+0000 mon.a (mon.0) 1708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:07.009358+0000 mon.a (mon.0) 1709 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:07.016567+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi087-82011-34"}]: dispatch 2024-02-23T00:33:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:07.018213+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:07.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:07.021746+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:07.458 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:07.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:07.474 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-02-23T00:33:06.018533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9170,"num_objects":359,"num_object_clones":2,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":1,"num_read":1231,"num_read_kb":865,"num_write":1288,"num_write_kb":647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2501,"ondisk_log_size":2501,"up":1131,"acting":1131,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":988,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2473796,"kb_used_data":30404,"kb_used_omap":0,"kb_used_meta":2443264,"kb_avail":747487420,"statfs":{"total":767960285184,"available":765427118080,"internally_reserved":0,"allocated":31133696,"data_stored":17045635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2501902336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-132,"num_objects":-12,"num_object_clones":-13,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":0,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":-5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011409"},"pg_stats":[{"pgid":"162.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.1e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.047302+0000","last_change":"2024-02-23T00:33:05.047302+0000","last_active":"2024-02-23T00:33:05.047302+0000","last_peered":"2024-02-23T00:33:05.047302+0000","last_clean":"2024-02-23T00:33:05.047302+0000","last_became_active":"2024-02-23T00:33:05.047028+0000","last_became_peered":"2024-02-23T00:33:05.047028+0000","last_unstale":"2024-02-23T00:33:05.047302+0000","last_undegraded":"2024-02-23T00:33:05.047302+0000","last_fullsized":"2024-02-23T00:33:05.047302+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"160.1f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.045119+0000","last_change":"2024-02-23T00:33:05.045119+0000","last_active":"2024-02-23T00:33:05.045119+0000","last_peered":"2024-02-23T00:33:05.045119+0000","last_clean":"2024-02-23T00:33:05.045119+0000","last_became_active":"2024-02-23T00:33:05.044755+0000","last_became_peered":"2024-02-23T00:33:05.044755+0000","last_unstale":"2024-02-23T00:33:05.045119+0000","last_undegraded":"2024-02-23T00:33:05.045119+0000","last_fullsized":"2024-02-23T00:33:05.045119+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"162.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.1f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.044837+0000","last_change":"2024-02-23T00:33:05.044837+0000","last_active":"2024-02-23T00:33:05.044837+0000","last_peered":"2024-02-23T00:33:05.044837+0000","last_clean":"2024-02-23T00:33:05.044837+0000","last_became_active":"2024-02-23T00:33:05.044530+0000","last_became_peered":"2024-02-23T00:33:05.044530+0000","last_unstale":"2024-02-23T00:33:05.044837+0000","last_undegraded":"2024-02-23T00:33:05.044837+0000","last_fullsized":"2024-02-23T00:33:05.044837+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"160.1e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.051263+0000","last_change":"2024-02-23T00:33:05.051263+0000","last_active":"2024-02-23T00:33:05.051263+0000","last_peered":"2024-02-23T00:33:05.051263+0000","last_clean":"2024-02-23T00:33:05.051263+0000","last_became_active":"2024-02-23T00:33:05.050993+0000","last_became_peered":"2024-02-23T00:33:05.050993+0000","last_unstale":"2024-02-23T00:33:05.051263+0000","last_undegraded":"2024-02-23T00:33:05.051263+0000","last_fullsized":"2024-02-23T00:33:05.051263+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"162.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.1c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.041219+0000","last_change":"2024-02-23T00:33:05.041219+0000","last_active":"2024-02-23T00:33:05.041219+0000","last_peered":"2024-02-23T00:33:05.041219+0000","last_clean":"2024-02-23T00:33:05.041219+0000","last_became_active":"2024-02-23T00:33:05.040956+0000","last_became_peered":"2024-02-23T00:33:05.040956+0000","last_unstale":"2024-02-23T00:33:05.041219+0000","last_undegraded":"2024-02-23T00:33:05.041219+0000","last_fullsized":"2024-02-23T00:33:05.041219+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"160.1d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.058212+0000","last_change":"2024-02-23T00:33:05.058212+0000","last_active":"2024-02-23T00:33:05.058212+0000","last_peered":"2024-02-23T00:33:05.058212+0000","last_clean":"2024-02-23T00:33:05.058212+0000","last_became_active":"2024-02-23T00:33:05.057929+0000","last_became_peered":"2024-02-23T00:33:05.057929+0000","last_unstale":"2024-02-23T00:33:05.058212+0000","last_undegraded":"2024-02-23T00:33:05.058212+0000","last_fullsized":"2024-02-23T00:33:05.058212+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"162.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.1d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.056336+0000","last_change":"2024-02-23T00:33:05.056336+0000","last_active":"2024-02-23T00:33:05.056336+0000","last_peered":"2024-02-23T00:33:05.056336+0000","last_clean":"2024-02-23T00:33:05.056336+0000","last_became_active":"2024-02-23T00:33:05.055803+0000","last_became_peered":"2024-02-23T00:33:05.055803+0000","last_unstale":"2024-02-23T00:33:05.056336+0000","last_undegraded":"2024-02-23T00:33:05.056336+0000","last_fullsized":"2024-02-23T00:33:05.056336+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"160.1c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.043917+0000","last_change":"2024-02-23T00:33:05.043917+0000","last_active":"2024-02-23T00:33:05.043917+0000","last_peered":"2024-02-23T00:33:05.043917+0000","last_clean":"2024-02-23T00:33:05.043917+0000","last_became_active":"2024-02-23T00:33:05.043631+0000","last_became_peered":"2024-02-23T00:33:05.043631+0000","last_unstale":"2024-02-23T00:33:05.043917+0000","last_undegraded":"2024-02-23T00:33:05.043917+0000","last_fullsized":"2024-02-23T00:33:05.043917+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"162.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.1a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.046901+0000","last_change":"2024-02-23T00:33:05.046901+0000","last_active":"2024-02-23T00:33:05.046901+0000","last_peered":"2024-02-23T00:33:05.046901+0000","last_clean":"2024-02-23T00:33:05.046901+0000","last_became_active":"2024-02-23T00:33:05.046510+0000","last_became_peered":"2024-02-23T00:33:05.046510+0000","last_unstale":"2024-02-23T00:33:05.046901+0000","last_undegraded":"2024-02-23T00:33:05.046901+0000","last_fullsized":"2024-02-23T00:33:05.046901+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"160.1b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.044276+0000","last_change":"2024-02-23T00:33:05.044276+0000","last_active":"2024-02-23T00:33:05.044276+0000","last_peered":"2024-02-23T00:33:05.044276+0000","last_clean":"2024-02-23T00:33:05.044276+0000","last_became_active":"2024-02-23T00:33:05.043904+0000","last_became_peered":"2024-02-23T00:33:05.043904+0000","last_unstale":"2024-02-23T00:33:05.044276+0000","last_undegraded":"2024-02-23T00:33:05.044276+0000","last_fullsized":"2024-02-23T00:33:05.044276+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"162.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.1b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.046139+0000","last_change":"2024-02-23T00:33:05.046139+0000","last_active":"2024-02-23T00:33:05.046139+0000","last_peered":"2024-02-23T00:33:05.046139+0000","last_clean":"2024-02-23T00:33:05.046139+0000","last_became_active":"2024-02-23T00:33:05.045804+0000","last_became_peered":"2024-02-23T00:33:05.045804+0000","last_unstale":"2024-02-23T00:33:05.046139+0000","last_undegraded":"2024-02-23T00:33:05.046139+0000","last_fullsized":"2024-02-23T00:33:05.046139+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"160.1a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.054347+0000","last_change":"2024-02-23T00:33:05.054347+0000","last_active":"2024-02-23T00:33:05.054347+0000","last_peered":"2024-02-23T00:33:05.054347+0000","last_clean":"2024-02-23T00:33:05.054347+0000","last_became_active":"2024-02-23T00:33:05.053999+0000","last_became_peered":"2024-02-23T00:33:05.053999+0000","last_unstale":"2024-02-23T00:33:05.054347+0000","last_undegraded":"2024-02-23T00:33:05.054347+0000","last_fullsized":"2024-02-23T00:33:05.054347+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"162.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.18","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.042677+0000","last_change":"2024-02-23T00:33:05.042677+0000","last_active":"2024-02-23T00:33:05.042677+0000","last_peered":"2024-02-23T00:33:05.042677+0000","last_clean":"2024-02-23T00:33:05.042677+0000","last_became_active":"2024-02-23T00:33:05.042417+0000","last_became_peered":"2024-02-23T00:33:05.042417+0000","last_unstale":"2024-02-23T00:33:05.042677+0000","last_undegraded":"2024-02-23T00:33:05.042677+0000","last_fullsized":"2024-02-23T00:33:05.042677+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"160.19","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.041358+0000","last_change":"2024-02-23T00:33:05.041358+0000","last_active":"2024-02-23T00:33:05.041358+0000","last_peered":"2024-02-23T00:33:05.041358+0000","last_clean":"2024-02-23T00:33:05.041358+0000","last_became_active":"2024-02-23T00:33:05.041174+0000","last_became_peered":"2024-02-23T00:33:05.041174+0000","last_unstale":"2024-02-23T00:33:05.041358+0000","last_undegraded":"2024-02-23T00:33:05.041358+0000","last_fullsized":"2024-02-23T00:33:05.041358+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"162.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.19","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.041194+0000","last_change":"2024-02-23T00:33:05.041194+0000","last_active":"2024-02-23T00:33:05.041194+0000","last_peered":"2024-02-23T00:33:05.041194+0000","last_clean":"2024-02-23T00:33:05.041194+0000","last_became_active":"2024-02-23T00:33:05.039844+0000","last_became_peered":"2024-02-23T00:33:05.039844+0000","last_unstale":"2024-02-23T00:33:05.041194+0000","last_undegraded":"2024-02-23T00:33:05.041194+0000","last_fullsized":"2024-02-23T00:33:05.041194+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"160.18","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.053521+0000","last_change":"2024-02-23T00:33:05.053521+0000","last_active":"2024-02-23T00:33:05.053521+0000","last_peered":"2024-02-23T00:33:05.053521+0000","last_clean":"2024-02-23T00:33:05.053521+0000","last_became_active":"2024-02-23T00:33:05.053190+0000","last_became_peered":"2024-02-23T00:33:05.053190+0000","last_unstale":"2024-02-23T00:33:05.053521+0000","last_undegraded":"2024-02-23T00:33:05.053521+0000","last_fullsized":"2024-02-23T00:33:05.053521+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"162.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.16","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.044903+0000","last_change":"2024-02-23T00:33:05.044903+0000","last_active":"2024-02-23T00:33:05.044903+0000","last_peered":"2024-02-23T00:33:05.044903+0000","last_clean":"2024-02-23T00:33:05.044903+0000","last_became_active":"2024-02-23T00:33:05.044574+0000","last_became_peered":"2024-02-23T00:33:05.044574+0000","last_unstale":"2024-02-23T00:33:05.044903+0000","last_undegraded":"2024-02-23T00:33:05.044903+0000","last_fullsized":"2024-02-23T00:33:05.044903+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"160.17","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.053598+0000","last_change":"2024-02-23T00:33:05.053598+0000","last_active":"2024-02-23T00:33:05.053598+0000","last_peered":"2024-02-23T00:33:05.053598+0000","last_clean":"2024-02-23T00:33:05.053598+0000","last_became_active":"2024-02-23T00:33:05.053247+0000","last_became_peered":"2024-02-23T00:33:05.053247+0000","last_unstale":"2024-02-23T00:33:05.053598+0000","last_undegraded":"2024-02-23T00:33:05.053598+0000","last_fullsized":"2024-02-23T00:33:05.053598+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"162.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.17","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.047086+0000","last_change":"2024-02-23T00:33:05.047086+0000","last_active":"2024-02-23T00:33:05.047086+0000","last_peered":"2024-02-23T00:33:05.047086+0000","last_clean":"2024-02-23T00:33:05.047086+0000","last_became_active":"2024-02-23T00:33:05.046816+0000","last_became_peered":"2024-02-23T00:33:05.046816+0000","last_unstale":"2024-02-23T00:33:05.047086+0000","last_undegraded":"2024-02-23T00:33:05.047086+0000","last_fullsized":"2024-02-23T00:33:05.047086+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"160.16","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.045601+0000","last_change":"2024-02-23T00:33:05.045601+0000","last_active":"2024-02-23T00:33:05.045601+0000","last_peered":"2024-02-23T00:33:05.045601+0000","last_clean":"2024-02-23T00:33:05.045601+0000","last_became_active":"2024-02-23T00:33:05.045315+0000","last_became_peered":"2024-02-23T00:33:05.045315+0000","last_unstale":"2024-02-23T00:33:05.045601+0000","last_undegraded":"2024-02-23T00:33:05.045601+0000","last_fullsized":"2024-02-23T00:33:05.045601+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"162.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.14","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.039228+0000","last_change":"2024-02-23T00:33:05.039228+0000","last_active":"2024-02-23T00:33:05.039228+0000","last_peered":"2024-02-23T00:33:05.039228+0000","last_clean":"2024-02-23T00:33:05.039228+0000","last_became_active":"2024-02-23T00:33:05.038883+0000","last_became_peered":"2024-02-23T00:33:05.038883+0000","last_unstale":"2024-02-23T00:33:05.039228+0000","last_undegraded":"2024-02-23T00:33:05.039228+0000","last_fullsized":"2024-02-23T00:33:05.039228+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"160.15","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.040307+0000","last_change":"2024-02-23T00:33:05.040307+0000","last_active":"2024-02-23T00:33:05.040307+0000","last_peered":"2024-02-23T00:33:05.040307+0000","last_clean":"2024-02-23T00:33:05.040307+0000","last_became_active":"2024-02-23T00:33:05.039876+0000","last_became_peered":"2024-02-23T00:33:05.039876+0000","last_unstale":"2024-02-23T00:33:05.040307+0000","last_undegraded":"2024-02-23T00:33:05.040307+0000","last_fullsized":"2024-02-23T00:33:05.040307+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"162.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.15","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.053627+0000","last_change":"2024-02-23T00:33:05.053627+0000","last_active":"2024-02-23T00:33:05.053627+0000","last_peered":"2024-02-23T00:33:05.053627+0000","last_clean":"2024-02-23T00:33:05.053627+0000","last_became_active":"2024-02-23T00:33:05.053307+0000","last_became_peered":"2024-02-23T00:33:05.053307+0000","last_unstale":"2024-02-23T00:33:05.053627+0000","last_undegraded":"2024-02-23T00:33:05.053627+0000","last_fullsized":"2024-02-23T00:33:05.053627+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"160.14","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.044840+0000","last_change":"2024-02-23T00:33:05.044840+0000","last_active":"2024-02-23T00:33:05.044840+0000","last_peered":"2024-02-23T00:33:05.044840+0000","last_clean":"2024-02-23T00:33:05.044840+0000","last_became_active":"2024-02-23T00:33:05.044483+0000","last_became_peered":"2024-02-23T00:33:05.044483+0000","last_unstale":"2024-02-23T00:33:05.044840+0000","last_undegraded":"2024-02-23T00:33:05.044840+0000","last_fullsized":"2024-02-23T00:33:05.044840+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"162.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.12","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.040543+0000","last_change":"2024-02-23T00:33:05.040543+0000","last_active":"2024-02-23T00:33:05.040543+0000","last_peered":"2024-02-23T00:33:05.040543+0000","last_clean":"2024-02-23T00:33:05.040543+0000","last_became_active":"2024-02-23T00:33:05.040241+0000","last_became_peered":"2024-02-23T00:33:05.040241+0000","last_unstale":"2024-02-23T00:33:05.040543+0000","last_undegraded":"2024-02-23T00:33:05.040543+0000","last_fullsized":"2024-02-23T00:33:05.040543+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"160.13","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.057653+0000","last_change":"2024-02-23T00:33:05.057653+0000","last_active":"2024-02-23T00:33:05.057653+0000","last_peered":"2024-02-23T00:33:05.057653+0000","last_clean":"2024-02-23T00:33:05.057653+0000","last_became_active":"2024-02-23T00:33:05.057375+0000","last_became_peered":"2024-02-23T00:33:05.057375+0000","last_unstale":"2024-02-23T00:33:05.057653+0000","last_undegraded":"2024-02-23T00:33:05.057653+0000","last_fullsized":"2024-02-23T00:33:05.057653+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"162.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.13","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.047221+0000","last_change":"2024-02-23T00:33:05.047221+0000","last_active":"2024-02-23T00:33:05.047221+0000","last_peered":"2024-02-23T00:33:05.047221+0000","last_clean":"2024-02-23T00:33:05.047221+0000","last_became_active":"2024-02-23T00:33:05.046917+0000","last_became_peered":"2024-02-23T00:33:05.046917+0000","last_unstale":"2024-02-23T00:33:05.047221+0000","last_undegraded":"2024-02-23T00:33:05.047221+0000","last_fullsized":"2024-02-23T00:33:05.047221+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"160.12","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.052359+0000","last_change":"2024-02-23T00:33:05.052359+0000","last_active":"2024-02-23T00:33:05.052359+0000","last_peered":"2024-02-23T00:33:05.052359+0000","last_clean":"2024-02-23T00:33:05.052359+0000","last_became_active":"2024-02-23T00:33:05.052038+0000","last_became_peered":"2024-02-23T00:33:05.052038+0000","last_unstale":"2024-02-23T00:33:05.052359+0000","last_undegraded":"2024-02-23T00:33:05.052359+0000","last_fullsized":"2024-02-23T00:33:05.052359+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"162.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.10","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.053992+0000","last_change":"2024-02-23T00:33:05.053992+0000","last_active":"2024-02-23T00:33:05.053992+0000","last_peered":"2024-02-23T00:33:05.053992+0000","last_clean":"2024-02-23T00:33:05.053992+0000","last_became_active":"2024-02-23T00:33:05.053526+0000","last_became_peered":"2024-02-23T00:33:05.053526+0000","last_unstale":"2024-02-23T00:33:05.053992+0000","last_undegraded":"2024-02-23T00:33:05.053992+0000","last_fullsized":"2024-02-23T00:33:05.053992+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"160.11","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.042686+0000","last_change":"2024-02-23T00:33:05.042686+0000","last_active":"2024-02-23T00:33:05.042686+0000","last_peered":"2024-02-23T00:33:05.042686+0000","last_clean":"2024-02-23T00:33:05.042686+0000","last_became_active":"2024-02-23T00:33:05.040725+0000","last_became_peered":"2024-02-23T00:33:05.040725+0000","last_unstale":"2024-02-23T00:33:05.042686+0000","last_undegraded":"2024-02-23T00:33:05.042686+0000","last_fullsized":"2024-02-23T00:33:05.042686+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"162.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.11","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.054246+0000","last_change":"2024-02-23T00:33:05.054246+0000","last_active":"2024-02-23T00:33:05.054246+0000","last_peered":"2024-02-23T00:33:05.054246+0000","last_clean":"2024-02-23T00:33:05.054246+0000","last_became_active":"2024-02-23T00:33:05.053952+0000","last_became_peered":"2024-02-23T00:33:05.053952+0000","last_unstale":"2024-02-23T00:33:05.054246+0000","last_undegraded":"2024-02-23T00:33:05.054246+0000","last_fullsized":"2024-02-23T00:33:05.054246+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"160.10","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.049439+0000","last_change":"2024-02-23T00:33:05.049439+0000","last_active":"2024-02-23T00:33:05.049439+0000","last_peered":"2024-02-23T00:33:05.049439+0000","last_clean":"2024-02-23T00:33:05.049439+0000","last_became_active":"2024-02-23T00:33:05.048941+0000","last_became_peered":"2024-02-23T00:33:05.048941+0000","last_unstale":"2024-02-23T00:33:05.049439+0000","last_undegraded":"2024-02-23T00:33:05.049439+0000","last_fullsized":"2024-02-23T00:33:05.049439+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"162.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.041729+0000","last_change":"2024-02-23T00:33:05.041729+0000","last_active":"2024-02-23T00:33:05.041729+0000","last_peered":"2024-02-23T00:33:05.041729+0000","last_clean":"2024-02-23T00:33:05.041729+0000","last_became_active":"2024-02-23T00:33:05.041515+0000","last_became_peered":"2024-02-23T00:33:05.041515+0000","last_unstale":"2024-02-23T00:33:05.041729+0000","last_undegraded":"2024-02-23T00:33:05.041729+0000","last_fullsized":"2024-02-23T00:33:05.041729+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"160.f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.057094+0000","last_change":"2024-02-23T00:33:05.057094+0000","last_active":"2024-02-23T00:33:05.057094+0000","last_peered":"2024-02-23T00:33:05.057094+0000","last_clean":"2024-02-23T00:33:05.057094+0000","last_became_active":"2024-02-23T00:33:05.056798+0000","last_became_peered":"2024-02-23T00:33:05.056798+0000","last_unstale":"2024-02-23T00:33:05.057094+0000","last_undegraded":"2024-02-23T00:33:05.057094+0000","last_fullsized":"2024-02-23T00:33:05.057094+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"162.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.f","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.049885+0000","last_change":"2024-02-23T00:33:05.049885+0000","last_active":"2024-02-23T00:33:05.049885+0000","last_peered":"2024-02-23T00:33:05.049885+0000","last_clean":"2024-02-23T00:33:05.049885+0000","last_became_active":"2024-02-23T00:33:05.047990+0000","last_became_peered":"2024-02-23T00:33:05.047990+0000","last_unstale":"2024-02-23T00:33:05.049885+0000","last_undegraded":"2024-02-23T00:33:05.049885+0000","last_fullsized":"2024-02-23T00:33:05.049885+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"160.e","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.044847+0000","last_change":"2024-02-23T00:33:05.044847+0000","last_active":"2024-02-23T00:33:05.044847+0000","last_peered":"2024-02-23T00:33:05.044847+0000","last_clean":"2024-02-23T00:33:05.044847+0000","last_became_active":"2024-02-23T00:33:05.044252+0000","last_became_peered":"2024-02-23T00:33:05.044252+0000","last_unstale":"2024-02-23T00:33:05.044847+0000","last_undegraded":"2024-02-23T00:33:05.044847+0000","last_fullsized":"2024-02-23T00:33:05.044847+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"162.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.046426+0000","last_change":"2024-02-23T00:33:05.046426+0000","last_active":"2024-02-23T00:33:05.046426+0000","last_peered":"2024-02-23T00:33:05.046426+0000","last_clean":"2024-02-23T00:33:05.046426+0000","last_became_active":"2024-02-23T00:33:05.046139+0000","last_became_peered":"2024-02-23T00:33:05.046139+0000","last_unstale":"2024-02-23T00:33:05.046426+0000","last_undegraded":"2024-02-23T00:33:05.046426+0000","last_fullsized":"2024-02-23T00:33:05.046426+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"160.d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.057435+0000","last_change":"2024-02-23T00:33:05.057435+0000","last_active":"2024-02-23T00:33:05.057435+0000","last_peered":"2024-02-23T00:33:05.057435+0000","last_clean":"2024-02-23T00:33:05.057435+0000","last_became_active":"2024-02-23T00:33:05.057046+0000","last_became_peered":"2024-02-23T00:33:05.057046+0000","last_unstale":"2024-02-23T00:33:05.057435+0000","last_undegraded":"2024-02-23T00:33:05.057435+0000","last_fullsized":"2024-02-23T00:33:05.057435+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"162.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.d","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.052216+0000","last_change":"2024-02-23T00:33:05.052216+0000","last_active":"2024-02-23T00:33:05.052216+0000","last_peered":"2024-02-23T00:33:05.052216+0000","last_clean":"2024-02-23T00:33:05.052216+0000","last_became_active":"2024-02-23T00:33:05.051928+0000","last_became_peered":"2024-02-23T00:33:05.051928+0000","last_unstale":"2024-02-23T00:33:05.052216+0000","last_undegraded":"2024-02-23T00:33:05.052216+0000","last_fullsized":"2024-02-23T00:33:05.052216+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"160.c","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.040983+0000","last_change":"2024-02-23T00:33:05.040983+0000","last_active":"2024-02-23T00:33:05.040983+0000","last_peered":"2024-02-23T00:33:05.040983+0000","last_clean":"2024-02-23T00:33:05.040983+0000","last_became_active":"2024-02-23T00:33:05.040737+0000","last_became_peered":"2024-02-23T00:33:05.040737+0000","last_unstale":"2024-02-23T00:33:05.040983+0000","last_undegraded":"2024-02-23T00:33:05.040983+0000","last_fullsized":"2024-02-23T00:33:05.040983+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"162.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.039152+0000","last_change":"2024-02-23T00:33:05.039152+0000","last_active":"2024-02-23T00:33:05.039152+0000","last_peered":"2024-02-23T00:33:05.039152+0000","last_clean":"2024-02-23T00:33:05.039152+0000","last_became_active":"2024-02-23T00:33:05.038873+0000","last_became_peered":"2024-02-23T00:33:05.038873+0000","last_unstale":"2024-02-23T00:33:05.039152+0000","last_undegraded":"2024-02-23T00:33:05.039152+0000","last_fullsized":"2024-02-23T00:33:05.039152+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"160.b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.049272+0000","last_change":"2024-02-23T00:33:05.049272+0000","last_active":"2024-02-23T00:33:05.049272+0000","last_peered":"2024-02-23T00:33:05.049272+0000","last_clean":"2024-02-23T00:33:05.049272+0000","last_became_active":"2024-02-23T00:33:05.049023+0000","last_became_peered":"2024-02-23T00:33:05.049023+0000","last_unstale":"2024-02-23T00:33:05.049272+0000","last_undegraded":"2024-02-23T00:33:05.049272+0000","last_fullsized":"2024-02-23T00:33:05.049272+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"162.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.b","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.045666+0000","last_change":"2024-02-23T00:33:05.045666+0000","last_active":"2024-02-23T00:33:05.045666+0000","last_peered":"2024-02-23T00:33:05.045666+0000","last_clean":"2024-02-23T00:33:05.045666+0000","last_became_active":"2024-02-23T00:33:05.045370+0000","last_became_peered":"2024-02-23T00:33:05.045370+0000","last_unstale":"2024-02-23T00:33:05.045666+0000","last_undegraded":"2024-02-23T00:33:05.045666+0000","last_fullsized":"2024-02-23T00:33:05.045666+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"160.a","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.042329+0000","last_change":"2024-02-23T00:33:05.042329+0000","last_active":"2024-02-23T00:33:05.042329+0000","last_peered":"2024-02-23T00:33:05.042329+0000","last_clean":"2024-02-23T00:33:05.042329+0000","last_became_active":"2024-02-23T00:33:05.041983+0000","last_became_peered":"2024-02-23T00:33:05.041983+0000","last_unstale":"2024-02-23T00:33:05.042329+0000","last_undegraded":"2024-02-23T00:33:05.042329+0000","last_fullsized":"2024-02-23T00:33:05.042329+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"162.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.8","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.046401+0000","last_change":"2024-02-23T00:33:05.046401+0000","last_active":"2024-02-23T00:33:05.046401+0000","last_peered":"2024-02-23T00:33:05.046401+0000","last_clean":"2024-02-23T00:33:05.046401+0000","last_became_active":"2024-02-23T00:33:05.046113+0000","last_became_peered":"2024-02-23T00:33:05.046113+0000","last_unstale":"2024-02-23T00:33:05.046401+0000","last_undegraded":"2024-02-23T00:33:05.046401+0000","last_fullsized":"2024-02-23T00:33:05.046401+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"160.9","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.039623+0000","last_change":"2024-02-23T00:33:05.039623+0000","last_active":"2024-02-23T00:33:05.039623+0000","last_peered":"2024-02-23T00:33:05.039623+0000","last_clean":"2024-02-23T00:33:05.039623+0000","last_became_active":"2024-02-23T00:33:05.039439+0000","last_became_peered":"2024-02-23T00:33:05.039439+0000","last_unstale":"2024-02-23T00:33:05.039623+0000","last_undegraded":"2024-02-23T00:33:05.039623+0000","last_fullsized":"2024-02-23T00:33:05.039623+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"162.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.9","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.046368+0000","last_change":"2024-02-23T00:33:05.046368+0000","last_active":"2024-02-23T00:33:05.046368+0000","last_peered":"2024-02-23T00:33:05.046368+0000","last_clean":"2024-02-23T00:33:05.046368+0000","last_became_active":"2024-02-23T00:33:05.046056+0000","last_became_peered":"2024-02-23T00:33:05.046056+0000","last_unstale":"2024-02-23T00:33:05.046368+0000","last_undegraded":"2024-02-23T00:33:05.046368+0000","last_fullsized":"2024-02-23T00:33:05.046368+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"160.8","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.042840+0000","last_change":"2024-02-23T00:33:05.042840+0000","last_active":"2024-02-23T00:33:05.042840+0000","last_peered":"2024-02-23T00:33:05.042840+0000","last_clean":"2024-02-23T00:33:05.042840+0000","last_became_active":"2024-02-23T00:33:05.042556+0000","last_became_peered":"2024-02-23T00:33:05.042556+0000","last_unstale":"2024-02-23T00:33:05.042840+0000","last_undegraded":"2024-02-23T00:33:05.042840+0000","last_fullsized":"2024-02-23T00:33:05.042840+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"162.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.6","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.051360+0000","last_change":"2024-02-23T00:33:05.051360+0000","last_active":"2024-02-23T00:33:05.051360+0000","last_peered":"2024-02-23T00:33:05.051360+0000","last_clean":"2024-02-23T00:33:05.051360+0000","last_became_active":"2024-02-23T00:33:05.051034+0000","last_became_peered":"2024-02-23T00:33:05.051034+0000","last_unstale":"2024-02-23T00:33:05.051360+0000","last_undegraded":"2024-02-23T00:33:05.051360+0000","last_fullsized":"2024-02-23T00:33:05.051360+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"160.7","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.049753+0000","last_change":"2024-02-23T00:33:05.049753+0000","last_active":"2024-02-23T00:33:05.049753+0000","last_peered":"2024-02-23T00:33:05.049753+0000","last_clean":"2024-02-23T00:33:05.049753+0000","last_became_active":"2024-02-23T00:33:05.049477+0000","last_became_peered":"2024-02-23T00:33:05.049477+0000","last_unstale":"2024-02-23T00:33:05.049753+0000","last_undegraded":"2024-02-23T00:33:05.049753+0000","last_fullsized":"2024-02-23T00:33:05.049753+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"162.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.7","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.046537+0000","last_change":"2024-02-23T00:33:05.046537+0000","last_active":"2024-02-23T00:33:05.046537+0000","last_peered":"2024-02-23T00:33:05.046537+0000","last_clean":"2024-02-23T00:33:05.046537+0000","last_became_active":"2024-02-23T00:33:05.046248+0000","last_became_peered":"2024-02-23T00:33:05.046248+0000","last_unstale":"2024-02-23T00:33:05.046537+0000","last_undegraded":"2024-02-23T00:33:05.046537+0000","last_fullsized":"2024-02-23T00:33:05.046537+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"160.6","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.054011+0000","last_change":"2024-02-23T00:33:05.054011+0000","last_active":"2024-02-23T00:33:05.054011+0000","last_peered":"2024-02-23T00:33:05.054011+0000","last_clean":"2024-02-23T00:33:05.054011+0000","last_became_active":"2024-02-23T00:33:05.053508+0000","last_became_peered":"2024-02-23T00:33:05.053508+0000","last_unstale":"2024-02-23T00:33:05.054011+0000","last_undegraded":"2024-02-23T00:33:05.054011+0000","last_fullsized":"2024-02-23T00:33:05.054011+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"162.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.4","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.046711+0000","last_change":"2024-02-23T00:33:05.046711+0000","last_active":"2024-02-23T00:33:05.046711+0000","last_peered":"2024-02-23T00:33:05.046711+0000","last_clean":"2024-02-23T00:33:05.046711+0000","last_became_active":"2024-02-23T00:33:05.046439+0000","last_became_peered":"2024-02-23T00:33:05.046439+0000","last_unstale":"2024-02-23T00:33:05.046711+0000","last_undegraded":"2024-02-23T00:33:05.046711+0000","last_fullsized":"2024-02-23T00:33:05.046711+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"160.5","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.057472+0000","last_change":"2024-02-23T00:33:05.057472+0000","last_active":"2024-02-23T00:33:05.057472+0000","last_peered":"2024-02-23T00:33:05.057472+0000","last_clean":"2024-02-23T00:33:05.057472+0000","last_became_active":"2024-02-23T00:33:05.057138+0000","last_became_peered":"2024-02-23T00:33:05.057138+0000","last_unstale":"2024-02-23T00:33:05.057472+0000","last_undegraded":"2024-02-23T00:33:05.057472+0000","last_fullsized":"2024-02-23T00:33:05.057472+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"162.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.5","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.045653+0000","last_change":"2024-02-23T00:33:05.045653+0000","last_active":"2024-02-23T00:33:05.045653+0000","last_peered":"2024-02-23T00:33:05.045653+0000","last_clean":"2024-02-23T00:33:05.045653+0000","last_became_active":"2024-02-23T00:33:05.045289+0000","last_became_peered":"2024-02-23T00:33:05.045289+0000","last_unstale":"2024-02-23T00:33:05.045653+0000","last_undegraded":"2024-02-23T00:33:05.045653+0000","last_fullsized":"2024-02-23T00:33:05.045653+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"160.4","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.054593+0000","last_change":"2024-02-23T00:33:05.054593+0000","last_active":"2024-02-23T00:33:05.054593+0000","last_peered":"2024-02-23T00:33:05.054593+0000","last_clean":"2024-02-23T00:33:05.054593+0000","last_became_active":"2024-02-23T00:33:05.054318+0000","last_became_peered":"2024-02-23T00:33:05.054318+0000","last_unstale":"2024-02-23T00:33:05.054593+0000","last_undegraded":"2024-02-23T00:33:05.054593+0000","last_fullsized":"2024-02-23T00:33:05.054593+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"162.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.2","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.057194+0000","last_change":"2024-02-23T00:33:05.057194+0000","last_active":"2024-02-23T00:33:05.057194+0000","last_peered":"2024-02-23T00:33:05.057194+0000","last_clean":"2024-02-23T00:33:05.057194+0000","last_became_active":"2024-02-23T00:33:05.056939+0000","last_became_peered":"2024-02-23T00:33:05.056939+0000","last_unstale":"2024-02-23T00:33:05.057194+0000","last_undegraded":"2024-02-23T00:33:05.057194+0000","last_fullsized":"2024-02-23T00:33:05.057194+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"160.3","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.045804+0000","last_change":"2024-02-23T00:33:05.045804+0000","last_active":"2024-02-23T00:33:05.045804+0000","last_peered":"2024-02-23T00:33:05.045804+0000","last_clean":"2024-02-23T00:33:05.045804+0000","last_became_active":"2024-02-23T00:33:05.045517+0000","last_became_peered":"2024-02-23T00:33:05.045517+0000","last_unstale":"2024-02-23T00:33:05.045804+0000","last_undegraded":"2024-02-23T00:33:05.045804+0000","last_fullsized":"2024-02-23T00:33:05.045804+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"162.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.3","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.042857+0000","last_change":"2024-02-23T00:33:05.042857+0000","last_active":"2024-02-23T00:33:05.042857+0000","last_peered":"2024-02-23T00:33:05.042857+0000","last_clean":"2024-02-23T00:33:05.042857+0000","last_became_active":"2024-02-23T00:33:05.042552+0000","last_became_peered":"2024-02-23T00:33:05.042552+0000","last_unstale":"2024-02-23T00:33:05.042857+0000","last_undegraded":"2024-02-23T00:33:05.042857+0000","last_fullsized":"2024-02-23T00:33:05.042857+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"160.2","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.048372+0000","last_change":"2024-02-23T00:33:05.048372+0000","last_active":"2024-02-23T00:33:05.048372+0000","last_peered":"2024-02-23T00:33:05.048372+0000","last_clean":"2024-02-23T00:33:05.048372+0000","last_became_active":"2024-02-23T00:33:05.048058+0000","last_became_peered":"2024-02-23T00:33:05.048058+0000","last_unstale":"2024-02-23T00:33:05.048372+0000","last_undegraded":"2024-02-23T00:33:05.048372+0000","last_fullsized":"2024-02-23T00:33:05.048372+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"162.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.0","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.049805+0000","last_change":"2024-02-23T00:33:05.049805+0000","last_active":"2024-02-23T00:33:05.049805+0000","last_peered":"2024-02-23T00:33:05.049805+0000","last_clean":"2024-02-23T00:33:05.049805+0000","last_became_active":"2024-02-23T00:33:05.049377+0000","last_became_peered":"2024-02-23T00:33:05.049377+0000","last_unstale":"2024-02-23T00:33:05.049805+0000","last_undegraded":"2024-02-23T00:33:05.049805+0000","last_fullsized":"2024-02-23T00:33:05.049805+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"160.1","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.043602+0000","last_change":"2024-02-23T00:33:05.043602+0000","last_active":"2024-02-23T00:33:05.043602+0000","last_peered":"2024-02-23T00:33:05.043602+0000","last_clean":"2024-02-23T00:33:05.043602+0000","last_became_active":"2024-02-23T00:33:05.043023+0000","last_became_peered":"2024-02-23T00:33:05.043023+0000","last_unstale":"2024-02-23T00:33:05.043602+0000","last_undegraded":"2024-02-23T00:33:05.043602+0000","last_fullsized":"2024-02-23T00:33:05.043602+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"162.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:06.015828+0000","last_change":"2024-02-23T00:33:06.015828+0000","last_active":"2024-02-23T00:33:06.015828+0000","last_peered":"2024-02-23T00:33:06.015828+0000","last_clean":"2024-02-23T00:33:06.015828+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:06.015828+0000","last_undegraded":"2024-02-23T00:33:06.015828+0000","last_fullsized":"2024-02-23T00:33:06.015828+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:06.015828+0000","last_clean_scrub_stamp":"2024-02-23T00:33:06.015828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"161.1","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.053593+0000","last_change":"2024-02-23T00:33:05.053593+0000","last_active":"2024-02-23T00:33:05.053593+0000","last_peered":"2024-02-23T00:33:05.053593+0000","last_clean":"2024-02-23T00:33:05.053593+0000","last_became_active":"2024-02-23T00:33:05.053304+0000","last_became_peered":"2024-02-23T00:33:05.053304+0000","last_unstale":"2024-02-23T00:33:05.053593+0000","last_undegraded":"2024-02-23T00:33:05.053593+0000","last_fullsized":"2024-02-23T00:33:05.053593+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"160.0","version":"0'0","reported_seq":9,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.053357+0000","last_change":"2024-02-23T00:33:05.053357+0000","last_active":"2024-02-23T00:33:05.053357+0000","last_peered":"2024-02-23T00:33:05.053357+0000","last_clean":"2024-02-23T00:33:05.053357+0000","last_became_active":"2024-02-23T00:33:05.053000+0000","last_became_peered":"2024-02-23T00:33:05.053000+0000","last_unstale":"2024-02-23T00:33:05.053357+0000","last_undegraded":"2024-02-23T00:33:05.053357+0000","last_fullsized":"2024-02-23T00:33:05.053357+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":233,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:04.006556+0000","last_clean_scrub_stamp":"2024-02-23T00:33:04.006556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"158.15","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023086+0000","last_change":"2024-02-23T00:33:03.259436+0000","last_active":"2024-02-23T00:33:04.023086+0000","last_peered":"2024-02-23T00:33:04.023086+0000","last_clean":"2024-02-23T00:33:04.023086+0000","last_became_active":"2024-02-23T00:33:03.259097+0000","last_became_peered":"2024-02-23T00:33:03.259097+0000","last_unstale":"2024-02-23T00:33:04.023086+0000","last_undegraded":"2024-02-23T00:33:04.023086+0000","last_fullsized":"2024-02-23T00:33:04.023086+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.16","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028355+0000","last_change":"2024-02-23T00:33:01.037450+0000","last_active":"2024-02-23T00:33:04.028355+0000","last_peered":"2024-02-23T00:33:04.028355+0000","last_clean":"2024-02-23T00:33:04.028355+0000","last_became_active":"2024-02-23T00:33:01.037156+0000","last_became_peered":"2024-02-23T00:33:01.037156+0000","last_unstale":"2024-02-23T00:33:04.028355+0000","last_undegraded":"2024-02-23T00:33:04.028355+0000","last_fullsized":"2024-02-23T00:33:04.028355+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.17","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021530+0000","last_change":"2024-02-23T00:32:59.043147+0000","last_active":"2024-02-23T00:33:04.021530+0000","last_peered":"2024-02-23T00:33:04.021530+0000","last_clean":"2024-02-23T00:33:04.021530+0000","last_became_active":"2024-02-23T00:32:59.042769+0000","last_became_peered":"2024-02-23T00:32:59.042769+0000","last_unstale":"2024-02-23T00:33:04.021530+0000","last_undegraded":"2024-02-23T00:33:04.021530+0000","last_fullsized":"2024-02-23T00:33:04.021530+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"158.14","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.033033+0000","last_change":"2024-02-23T00:33:03.030750+0000","last_active":"2024-02-23T00:33:04.033033+0000","last_peered":"2024-02-23T00:33:04.033033+0000","last_clean":"2024-02-23T00:33:04.033033+0000","last_became_active":"2024-02-23T00:33:03.030477+0000","last_became_peered":"2024-02-23T00:33:03.030477+0000","last_unstale":"2024-02-23T00:33:04.033033+0000","last_undegraded":"2024-02-23T00:33:04.033033+0000","last_fullsized":"2024-02-23T00:33:04.033033+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.17","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027237+0000","last_change":"2024-02-23T00:33:01.043881+0000","last_active":"2024-02-23T00:33:04.027237+0000","last_peered":"2024-02-23T00:33:04.027237+0000","last_clean":"2024-02-23T00:33:04.027237+0000","last_became_active":"2024-02-23T00:33:01.043558+0000","last_became_peered":"2024-02-23T00:33:01.043558+0000","last_unstale":"2024-02-23T00:33:04.027237+0000","last_undegraded":"2024-02-23T00:33:04.027237+0000","last_fullsized":"2024-02-23T00:33:04.027237+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"156.16","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023228+0000","last_change":"2024-02-23T00:32:59.035188+0000","last_active":"2024-02-23T00:33:04.023228+0000","last_peered":"2024-02-23T00:33:04.023228+0000","last_clean":"2024-02-23T00:33:04.023228+0000","last_became_active":"2024-02-23T00:32:59.034877+0000","last_became_peered":"2024-02-23T00:32:59.034877+0000","last_unstale":"2024-02-23T00:33:04.023228+0000","last_undegraded":"2024-02-23T00:33:04.023228+0000","last_fullsized":"2024-02-23T00:33:04.023228+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"158.17","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.035692+0000","last_change":"2024-02-23T00:33:03.258255+0000","last_active":"2024-02-23T00:33:04.035692+0000","last_peered":"2024-02-23T00:33:04.035692+0000","last_clean":"2024-02-23T00:33:04.035692+0000","last_became_active":"2024-02-23T00:33:03.257240+0000","last_became_peered":"2024-02-23T00:33:03.257240+0000","last_unstale":"2024-02-23T00:33:04.035692+0000","last_undegraded":"2024-02-23T00:33:04.035692+0000","last_fullsized":"2024-02-23T00:33:04.035692+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.14","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031324+0000","last_change":"2024-02-23T00:33:01.048094+0000","last_active":"2024-02-23T00:33:04.031324+0000","last_peered":"2024-02-23T00:33:04.031324+0000","last_clean":"2024-02-23T00:33:04.031324+0000","last_became_active":"2024-02-23T00:33:01.045971+0000","last_became_peered":"2024-02-23T00:33:01.045971+0000","last_unstale":"2024-02-23T00:33:04.031324+0000","last_undegraded":"2024-02-23T00:33:04.031324+0000","last_fullsized":"2024-02-23T00:33:04.031324+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"156.15","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025577+0000","last_change":"2024-02-23T00:32:59.035664+0000","last_active":"2024-02-23T00:33:04.025577+0000","last_peered":"2024-02-23T00:33:04.025577+0000","last_clean":"2024-02-23T00:33:04.025577+0000","last_became_active":"2024-02-23T00:32:59.035325+0000","last_became_peered":"2024-02-23T00:32:59.035325+0000","last_unstale":"2024-02-23T00:33:04.025577+0000","last_undegraded":"2024-02-23T00:33:04.025577+0000","last_fullsized":"2024-02-23T00:33:04.025577+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"158.16","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036863+0000","last_change":"2024-02-23T00:33:03.263827+0000","last_active":"2024-02-23T00:33:04.036863+0000","last_peered":"2024-02-23T00:33:04.036863+0000","last_clean":"2024-02-23T00:33:04.036863+0000","last_became_active":"2024-02-23T00:33:03.263469+0000","last_became_peered":"2024-02-23T00:33:03.263469+0000","last_unstale":"2024-02-23T00:33:04.036863+0000","last_undegraded":"2024-02-23T00:33:04.036863+0000","last_fullsized":"2024-02-23T00:33:04.036863+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.15","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023162+0000","last_change":"2024-02-23T00:33:01.047911+0000","last_active":"2024-02-23T00:33:04.023162+0000","last_peered":"2024-02-23T00:33:04.023162+0000","last_clean":"2024-02-23T00:33:04.023162+0000","last_became_active":"2024-02-23T00:33:01.047584+0000","last_became_peered":"2024-02-23T00:33:01.047584+0000","last_unstale":"2024-02-23T00:33:04.023162+0000","last_undegraded":"2024-02-23T00:33:04.023162+0000","last_fullsized":"2024-02-23T00:33:04.023162+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.14","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031439+0000","last_change":"2024-02-23T00:32:59.031036+0000","last_active":"2024-02-23T00:33:04.031439+0000","last_peered":"2024-02-23T00:33:04.031439+0000","last_clean":"2024-02-23T00:33:04.031439+0000","last_became_active":"2024-02-23T00:32:59.030743+0000","last_became_peered":"2024-02-23T00:32:59.030743+0000","last_unstale":"2024-02-23T00:33:04.031439+0000","last_undegraded":"2024-02-23T00:33:04.031439+0000","last_fullsized":"2024-02-23T00:33:04.031439+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"158.11","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023316+0000","last_change":"2024-02-23T00:33:03.183601+0000","last_active":"2024-02-23T00:33:04.023316+0000","last_peered":"2024-02-23T00:33:04.023316+0000","last_clean":"2024-02-23T00:33:04.023316+0000","last_became_active":"2024-02-23T00:33:03.183229+0000","last_became_peered":"2024-02-23T00:33:03.183229+0000","last_unstale":"2024-02-23T00:33:04.023316+0000","last_undegraded":"2024-02-23T00:33:04.023316+0000","last_fullsized":"2024-02-23T00:33:04.023316+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"157.12","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024153+0000","last_change":"2024-02-23T00:33:01.049176+0000","last_active":"2024-02-23T00:33:04.024153+0000","last_peered":"2024-02-23T00:33:04.024153+0000","last_clean":"2024-02-23T00:33:04.024153+0000","last_became_active":"2024-02-23T00:33:01.048878+0000","last_became_peered":"2024-02-23T00:33:01.048878+0000","last_unstale":"2024-02-23T00:33:04.024153+0000","last_undegraded":"2024-02-23T00:33:04.024153+0000","last_fullsized":"2024-02-23T00:33:04.024153+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.13","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036805+0000","last_change":"2024-02-23T00:32:59.037298+0000","last_active":"2024-02-23T00:33:04.036805+0000","last_peered":"2024-02-23T00:33:04.036805+0000","last_clean":"2024-02-23T00:33:04.036805+0000","last_became_active":"2024-02-23T00:32:59.037000+0000","last_became_peered":"2024-02-23T00:32:59.037000+0000","last_unstale":"2024-02-23T00:33:04.036805+0000","last_undegraded":"2024-02-23T00:33:04.036805+0000","last_fullsized":"2024-02-23T00:33:04.036805+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"158.10","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025230+0000","last_change":"2024-02-23T00:33:03.139279+0000","last_active":"2024-02-23T00:33:04.025230+0000","last_peered":"2024-02-23T00:33:04.025230+0000","last_clean":"2024-02-23T00:33:04.025230+0000","last_became_active":"2024-02-23T00:33:03.137831+0000","last_became_peered":"2024-02-23T00:33:03.137831+0000","last_unstale":"2024-02-23T00:33:04.025230+0000","last_undegraded":"2024-02-23T00:33:04.025230+0000","last_fullsized":"2024-02-23T00:33:04.025230+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.13","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021642+0000","last_change":"2024-02-23T00:33:01.048724+0000","last_active":"2024-02-23T00:33:04.021642+0000","last_peered":"2024-02-23T00:33:04.021642+0000","last_clean":"2024-02-23T00:33:04.021642+0000","last_became_active":"2024-02-23T00:33:01.048377+0000","last_became_peered":"2024-02-23T00:33:01.048377+0000","last_unstale":"2024-02-23T00:33:04.021642+0000","last_undegraded":"2024-02-23T00:33:04.021642+0000","last_fullsized":"2024-02-23T00:33:04.021642+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.12","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023500+0000","last_change":"2024-02-23T00:32:59.030659+0000","last_active":"2024-02-23T00:33:04.023500+0000","last_peered":"2024-02-23T00:33:04.023500+0000","last_clean":"2024-02-23T00:33:04.023500+0000","last_became_active":"2024-02-23T00:32:59.030370+0000","last_became_peered":"2024-02-23T00:32:59.030370+0000","last_unstale":"2024-02-23T00:33:04.023500+0000","last_undegraded":"2024-02-23T00:33:04.023500+0000","last_fullsized":"2024-02-23T00:33:04.023500+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"158.13","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028750+0000","last_change":"2024-02-23T00:33:03.028262+0000","last_active":"2024-02-23T00:33:04.028750+0000","last_peered":"2024-02-23T00:33:04.028750+0000","last_clean":"2024-02-23T00:33:04.028750+0000","last_became_active":"2024-02-23T00:33:03.027799+0000","last_became_peered":"2024-02-23T00:33:03.027799+0000","last_unstale":"2024-02-23T00:33:04.028750+0000","last_undegraded":"2024-02-23T00:33:04.028750+0000","last_fullsized":"2024-02-23T00:33:04.028750+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"157.10","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022375+0000","last_change":"2024-02-23T00:33:01.037762+0000","last_active":"2024-02-23T00:33:04.022375+0000","last_peered":"2024-02-23T00:33:04.022375+0000","last_clean":"2024-02-23T00:33:04.022375+0000","last_became_active":"2024-02-23T00:33:01.037486+0000","last_became_peered":"2024-02-23T00:33:01.037486+0000","last_unstale":"2024-02-23T00:33:04.022375+0000","last_undegraded":"2024-02-23T00:33:04.022375+0000","last_fullsized":"2024-02-23T00:33:04.022375+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.11","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030215+0000","last_change":"2024-02-23T00:32:59.039142+0000","last_active":"2024-02-23T00:33:04.030215+0000","last_peered":"2024-02-23T00:33:04.030215+0000","last_clean":"2024-02-23T00:33:04.030215+0000","last_became_active":"2024-02-23T00:32:59.038703+0000","last_became_peered":"2024-02-23T00:32:59.038703+0000","last_unstale":"2024-02-23T00:33:04.030215+0000","last_undegraded":"2024-02-23T00:33:04.030215+0000","last_fullsized":"2024-02-23T00:33:04.030215+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"158.12","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024312+0000","last_change":"2024-02-23T00:33:03.141476+0000","last_active":"2024-02-23T00:33:04.024312+0000","last_peered":"2024-02-23T00:33:04.024312+0000","last_clean":"2024-02-23T00:33:04.024312+0000","last_became_active":"2024-02-23T00:33:03.141225+0000","last_became_peered":"2024-02-23T00:33:03.141225+0000","last_unstale":"2024-02-23T00:33:04.024312+0000","last_undegraded":"2024-02-23T00:33:04.024312+0000","last_fullsized":"2024-02-23T00:33:04.024312+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.11","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030016+0000","last_change":"2024-02-23T00:33:01.050427+0000","last_active":"2024-02-23T00:33:04.030016+0000","last_peered":"2024-02-23T00:33:04.030016+0000","last_clean":"2024-02-23T00:33:04.030016+0000","last_became_active":"2024-02-23T00:33:01.049737+0000","last_became_peered":"2024-02-23T00:33:01.049737+0000","last_unstale":"2024-02-23T00:33:04.030016+0000","last_undegraded":"2024-02-23T00:33:04.030016+0000","last_fullsized":"2024-02-23T00:33:04.030016+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"156.10","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020313+0000","last_change":"2024-02-23T00:32:59.037690+0000","last_active":"2024-02-23T00:33:04.020313+0000","last_peered":"2024-02-23T00:33:04.020313+0000","last_clean":"2024-02-23T00:33:04.020313+0000","last_became_active":"2024-02-23T00:32:59.037387+0000","last_became_peered":"2024-02-23T00:32:59.037387+0000","last_unstale":"2024-02-23T00:33:04.020313+0000","last_undegraded":"2024-02-23T00:33:04.020313+0000","last_fullsized":"2024-02-23T00:33:04.020313+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"158.1d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039215+0000","last_change":"2024-02-23T00:33:03.265112+0000","last_active":"2024-02-23T00:33:04.039215+0000","last_peered":"2024-02-23T00:33:04.039215+0000","last_clean":"2024-02-23T00:33:04.039215+0000","last_became_active":"2024-02-23T00:33:03.264829+0000","last_became_peered":"2024-02-23T00:33:03.264829+0000","last_unstale":"2024-02-23T00:33:04.039215+0000","last_undegraded":"2024-02-23T00:33:04.039215+0000","last_fullsized":"2024-02-23T00:33:04.039215+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.1e","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023813+0000","last_change":"2024-02-23T00:33:01.049237+0000","last_active":"2024-02-23T00:33:04.023813+0000","last_peered":"2024-02-23T00:33:04.023813+0000","last_clean":"2024-02-23T00:33:04.023813+0000","last_became_active":"2024-02-23T00:33:01.048938+0000","last_became_peered":"2024-02-23T00:33:01.048938+0000","last_unstale":"2024-02-23T00:33:04.023813+0000","last_undegraded":"2024-02-23T00:33:04.023813+0000","last_fullsized":"2024-02-23T00:33:04.023813+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.1f","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021450+0000","last_change":"2024-02-23T00:32:59.034456+0000","last_active":"2024-02-23T00:33:04.021450+0000","last_peered":"2024-02-23T00:33:04.021450+0000","last_clean":"2024-02-23T00:33:04.021450+0000","last_became_active":"2024-02-23T00:32:59.034153+0000","last_became_peered":"2024-02-23T00:32:59.034153+0000","last_unstale":"2024-02-23T00:33:04.021450+0000","last_undegraded":"2024-02-23T00:33:04.021450+0000","last_fullsized":"2024-02-23T00:33:04.021450+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"158.1c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024590+0000","last_change":"2024-02-23T00:33:03.140132+0000","last_active":"2024-02-23T00:33:04.024590+0000","last_peered":"2024-02-23T00:33:04.024590+0000","last_clean":"2024-02-23T00:33:04.024590+0000","last_became_active":"2024-02-23T00:33:03.139468+0000","last_became_peered":"2024-02-23T00:33:03.139468+0000","last_unstale":"2024-02-23T00:33:04.024590+0000","last_undegraded":"2024-02-23T00:33:04.024590+0000","last_fullsized":"2024-02-23T00:33:04.024590+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.1f","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030327+0000","last_change":"2024-02-23T00:33:01.048884+0000","last_active":"2024-02-23T00:33:04.030327+0000","last_peered":"2024-02-23T00:33:04.030327+0000","last_clean":"2024-02-23T00:33:04.030327+0000","last_became_active":"2024-02-23T00:33:01.048602+0000","last_became_peered":"2024-02-23T00:33:01.048602+0000","last_unstale":"2024-02-23T00:33:04.030327+0000","last_undegraded":"2024-02-23T00:33:04.030327+0000","last_fullsized":"2024-02-23T00:33:04.030327+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"156.1e","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036573+0000","last_change":"2024-02-23T00:32:59.042656+0000","last_active":"2024-02-23T00:33:04.036573+0000","last_peered":"2024-02-23T00:33:04.036573+0000","last_clean":"2024-02-23T00:33:04.036573+0000","last_became_active":"2024-02-23T00:32:59.042346+0000","last_became_peered":"2024-02-23T00:32:59.042346+0000","last_unstale":"2024-02-23T00:33:04.036573+0000","last_undegraded":"2024-02-23T00:33:04.036573+0000","last_fullsized":"2024-02-23T00:33:04.036573+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"158.1f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027126+0000","last_change":"2024-02-23T00:33:03.180130+0000","last_active":"2024-02-23T00:33:04.027126+0000","last_peered":"2024-02-23T00:33:04.027126+0000","last_clean":"2024-02-23T00:33:04.027126+0000","last_became_active":"2024-02-23T00:33:03.179898+0000","last_became_peered":"2024-02-23T00:33:03.179898+0000","last_unstale":"2024-02-23T00:33:04.027126+0000","last_undegraded":"2024-02-23T00:33:04.027126+0000","last_fullsized":"2024-02-23T00:33:04.027126+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"157.1c","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024280+0000","last_change":"2024-02-23T00:33:01.049487+0000","last_active":"2024-02-23T00:33:04.024280+0000","last_peered":"2024-02-23T00:33:04.024280+0000","last_clean":"2024-02-23T00:33:04.024280+0000","last_became_active":"2024-02-23T00:33:01.049031+0000","last_became_peered":"2024-02-23T00:33:01.049031+0000","last_unstale":"2024-02-23T00:33:04.024280+0000","last_undegraded":"2024-02-23T00:33:04.024280+0000","last_fullsized":"2024-02-23T00:33:04.024280+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.1d","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023236+0000","last_change":"2024-02-23T00:32:59.040014+0000","last_active":"2024-02-23T00:33:04.023236+0000","last_peered":"2024-02-23T00:33:04.023236+0000","last_clean":"2024-02-23T00:33:04.023236+0000","last_became_active":"2024-02-23T00:32:59.039712+0000","last_became_peered":"2024-02-23T00:32:59.039712+0000","last_unstale":"2024-02-23T00:33:04.023236+0000","last_undegraded":"2024-02-23T00:33:04.023236+0000","last_fullsized":"2024-02-23T00:33:04.023236+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"158.1e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037518+0000","last_change":"2024-02-23T00:33:03.265359+0000","last_active":"2024-02-23T00:33:04.037518+0000","last_peered":"2024-02-23T00:33:04.037518+0000","last_clean":"2024-02-23T00:33:04.037518+0000","last_became_active":"2024-02-23T00:33:03.265093+0000","last_became_peered":"2024-02-23T00:33:03.265093+0000","last_unstale":"2024-02-23T00:33:04.037518+0000","last_undegraded":"2024-02-23T00:33:04.037518+0000","last_fullsized":"2024-02-23T00:33:04.037518+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.1d","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024799+0000","last_change":"2024-02-23T00:33:01.043740+0000","last_active":"2024-02-23T00:33:04.024799+0000","last_peered":"2024-02-23T00:33:04.024799+0000","last_clean":"2024-02-23T00:33:04.024799+0000","last_became_active":"2024-02-23T00:33:01.043376+0000","last_became_peered":"2024-02-23T00:33:01.043376+0000","last_unstale":"2024-02-23T00:33:04.024799+0000","last_undegraded":"2024-02-23T00:33:04.024799+0000","last_fullsized":"2024-02-23T00:33:04.024799+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"156.1c","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027118+0000","last_change":"2024-02-23T00:32:59.030860+0000","last_active":"2024-02-23T00:33:04.027118+0000","last_peered":"2024-02-23T00:33:04.027118+0000","last_clean":"2024-02-23T00:33:04.027118+0000","last_became_active":"2024-02-23T00:32:59.030546+0000","last_became_peered":"2024-02-23T00:32:59.030546+0000","last_unstale":"2024-02-23T00:33:04.027118+0000","last_undegraded":"2024-02-23T00:33:04.027118+0000","last_fullsized":"2024-02-23T00:33:04.027118+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"158.19","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022199+0000","last_change":"2024-02-23T00:33:03.028125+0000","last_active":"2024-02-23T00:33:04.022199+0000","last_peered":"2024-02-23T00:33:04.022199+0000","last_clean":"2024-02-23T00:33:04.022199+0000","last_became_active":"2024-02-23T00:33:03.027812+0000","last_became_peered":"2024-02-23T00:33:03.027812+0000","last_unstale":"2024-02-23T00:33:04.022199+0000","last_undegraded":"2024-02-23T00:33:04.022199+0000","last_fullsized":"2024-02-23T00:33:04.022199+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"157.1a","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028386+0000","last_change":"2024-02-23T00:33:01.037445+0000","last_active":"2024-02-23T00:33:04.028386+0000","last_peered":"2024-02-23T00:33:04.028386+0000","last_clean":"2024-02-23T00:33:04.028386+0000","last_became_active":"2024-02-23T00:33:01.037154+0000","last_became_peered":"2024-02-23T00:33:01.037154+0000","last_unstale":"2024-02-23T00:33:04.028386+0000","last_undegraded":"2024-02-23T00:33:04.028386+0000","last_fullsized":"2024-02-23T00:33:04.028386+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"156.1b","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021796+0000","last_change":"2024-02-23T00:32:59.037288+0000","last_active":"2024-02-23T00:33:04.021796+0000","last_peered":"2024-02-23T00:33:04.021796+0000","last_clean":"2024-02-23T00:33:04.021796+0000","last_became_active":"2024-02-23T00:32:59.036982+0000","last_became_peered":"2024-02-23T00:32:59.036982+0000","last_unstale":"2024-02-23T00:33:04.021796+0000","last_undegraded":"2024-02-23T00:33:04.021796+0000","last_fullsized":"2024-02-23T00:33:04.021796+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"158.18","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.019487+0000","last_change":"2024-02-23T00:33:03.024159+0000","last_active":"2024-02-23T00:33:04.019487+0000","last_peered":"2024-02-23T00:33:04.019487+0000","last_clean":"2024-02-23T00:33:04.019487+0000","last_became_active":"2024-02-23T00:33:03.023855+0000","last_became_peered":"2024-02-23T00:33:03.023855+0000","last_unstale":"2024-02-23T00:33:04.019487+0000","last_undegraded":"2024-02-23T00:33:04.019487+0000","last_fullsized":"2024-02-23T00:33:04.019487+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"157.1b","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036772+0000","last_change":"2024-02-23T00:33:01.043162+0000","last_active":"2024-02-23T00:33:04.036772+0000","last_peered":"2024-02-23T00:33:04.036772+0000","last_clean":"2024-02-23T00:33:04.036772+0000","last_became_active":"2024-02-23T00:33:01.042870+0000","last_became_peered":"2024-02-23T00:33:01.042870+0000","last_unstale":"2024-02-23T00:33:04.036772+0000","last_undegraded":"2024-02-23T00:33:04.036772+0000","last_fullsized":"2024-02-23T00:33:04.036772+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"156.1a","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036933+0000","last_change":"2024-02-23T00:32:59.041373+0000","last_active":"2024-02-23T00:33:04.036933+0000","last_peered":"2024-02-23T00:33:04.036933+0000","last_clean":"2024-02-23T00:33:04.036933+0000","last_became_active":"2024-02-23T00:32:59.041052+0000","last_became_peered":"2024-02-23T00:32:59.041052+0000","last_unstale":"2024-02-23T00:33:04.036933+0000","last_undegraded":"2024-02-23T00:33:04.036933+0000","last_fullsized":"2024-02-23T00:33:04.036933+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"158.1b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020747+0000","last_change":"2024-02-23T00:33:03.259246+0000","last_active":"2024-02-23T00:33:04.020747+0000","last_peered":"2024-02-23T00:33:04.020747+0000","last_clean":"2024-02-23T00:33:04.020747+0000","last_became_active":"2024-02-23T00:33:03.258953+0000","last_became_peered":"2024-02-23T00:33:03.258953+0000","last_unstale":"2024-02-23T00:33:04.020747+0000","last_undegraded":"2024-02-23T00:33:04.020747+0000","last_fullsized":"2024-02-23T00:33:04.020747+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"157.18","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037460+0000","last_change":"2024-02-23T00:33:01.043004+0000","last_active":"2024-02-23T00:33:04.037460+0000","last_peered":"2024-02-23T00:33:04.037460+0000","last_clean":"2024-02-23T00:33:04.037460+0000","last_became_active":"2024-02-23T00:33:01.042723+0000","last_became_peered":"2024-02-23T00:33:01.042723+0000","last_unstale":"2024-02-23T00:33:04.037460+0000","last_undegraded":"2024-02-23T00:33:04.037460+0000","last_fullsized":"2024-02-23T00:33:04.037460+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"156.19","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030144+0000","last_change":"2024-02-23T00:32:59.041589+0000","last_active":"2024-02-23T00:33:04.030144+0000","last_peered":"2024-02-23T00:33:04.030144+0000","last_clean":"2024-02-23T00:33:04.030144+0000","last_became_active":"2024-02-23T00:32:59.041264+0000","last_became_peered":"2024-02-23T00:32:59.041264+0000","last_unstale":"2024-02-23T00:33:04.030144+0000","last_undegraded":"2024-02-23T00:33:04.030144+0000","last_fullsized":"2024-02-23T00:33:04.030144+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"158.1a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023011+0000","last_change":"2024-02-23T00:33:03.261118+0000","last_active":"2024-02-23T00:33:04.023011+0000","last_peered":"2024-02-23T00:33:04.023011+0000","last_clean":"2024-02-23T00:33:04.023011+0000","last_became_active":"2024-02-23T00:33:03.260771+0000","last_became_peered":"2024-02-23T00:33:03.260771+0000","last_unstale":"2024-02-23T00:33:04.023011+0000","last_undegraded":"2024-02-23T00:33:04.023011+0000","last_fullsized":"2024-02-23T00:33:04.023011+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.19","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.019995+0000","last_change":"2024-02-23T00:33:01.034983+0000","last_active":"2024-02-23T00:33:04.019995+0000","last_peered":"2024-02-23T00:33:04.019995+0000","last_clean":"2024-02-23T00:33:04.019995+0000","last_became_active":"2024-02-23T00:33:01.034710+0000","last_became_peered":"2024-02-23T00:33:01.034710+0000","last_unstale":"2024-02-23T00:33:04.019995+0000","last_undegraded":"2024-02-23T00:33:04.019995+0000","last_fullsized":"2024-02-23T00:33:04.019995+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"156.18","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.019782+0000","last_change":"2024-02-23T00:32:59.037856+0000","last_active":"2024-02-23T00:33:04.019782+0000","last_peered":"2024-02-23T00:33:04.019782+0000","last_clean":"2024-02-23T00:33:04.019782+0000","last_became_active":"2024-02-23T00:32:59.037513+0000","last_became_peered":"2024-02-23T00:32:59.037513+0000","last_unstale":"2024-02-23T00:33:04.019782+0000","last_undegraded":"2024-02-23T00:33:04.019782+0000","last_fullsized":"2024-02-23T00:33:04.019782+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"158.5","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021093+0000","last_change":"2024-02-23T00:33:03.258436+0000","last_active":"2024-02-23T00:33:04.021093+0000","last_peered":"2024-02-23T00:33:04.021093+0000","last_clean":"2024-02-23T00:33:04.021093+0000","last_became_active":"2024-02-23T00:33:03.258171+0000","last_became_peered":"2024-02-23T00:33:03.258171+0000","last_unstale":"2024-02-23T00:33:04.021093+0000","last_undegraded":"2024-02-23T00:33:04.021093+0000","last_fullsized":"2024-02-23T00:33:04.021093+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"157.6","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.035385+0000","last_change":"2024-02-23T00:33:01.047566+0000","last_active":"2024-02-23T00:33:04.035385+0000","last_peered":"2024-02-23T00:33:04.035385+0000","last_clean":"2024-02-23T00:33:04.035385+0000","last_became_active":"2024-02-23T00:33:01.047251+0000","last_became_peered":"2024-02-23T00:33:01.047251+0000","last_unstale":"2024-02-23T00:33:04.035385+0000","last_undegraded":"2024-02-23T00:33:04.035385+0000","last_fullsized":"2024-02-23T00:33:04.035385+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"156.7","version":"230'3","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028760+0000","last_change":"2024-02-23T00:32:59.040847+0000","last_active":"2024-02-23T00:33:04.028760+0000","last_peered":"2024-02-23T00:33:04.028760+0000","last_clean":"2024-02-23T00:33:04.028760+0000","last_became_active":"2024-02-23T00:32:59.040539+0000","last_became_peered":"2024-02-23T00:32:59.040539+0000","last_unstale":"2024-02-23T00:33:04.028760+0000","last_undegraded":"2024-02-23T00:33:04.028760+0000","last_fullsized":"2024-02-23T00:33:04.028760+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"158.4","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024098+0000","last_change":"2024-02-23T00:33:03.140940+0000","last_active":"2024-02-23T00:33:04.024098+0000","last_peered":"2024-02-23T00:33:04.024098+0000","last_clean":"2024-02-23T00:33:04.024098+0000","last_became_active":"2024-02-23T00:33:03.139732+0000","last_became_peered":"2024-02-23T00:33:03.139732+0000","last_unstale":"2024-02-23T00:33:04.024098+0000","last_undegraded":"2024-02-23T00:33:04.024098+0000","last_fullsized":"2024-02-23T00:33:04.024098+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.7","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028670+0000","last_change":"2024-02-23T00:33:01.038120+0000","last_active":"2024-02-23T00:33:04.028670+0000","last_peered":"2024-02-23T00:33:04.028670+0000","last_clean":"2024-02-23T00:33:04.028670+0000","last_became_active":"2024-02-23T00:33:01.037846+0000","last_became_peered":"2024-02-23T00:33:01.037846+0000","last_unstale":"2024-02-23T00:33:04.028670+0000","last_undegraded":"2024-02-23T00:33:04.028670+0000","last_fullsized":"2024-02-23T00:33:04.028670+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.6","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.029600+0000","last_change":"2024-02-23T00:32:59.034390+0000","last_active":"2024-02-23T00:33:04.029600+0000","last_peered":"2024-02-23T00:33:04.029600+0000","last_clean":"2024-02-23T00:33:04.029600+0000","last_became_active":"2024-02-23T00:32:59.034079+0000","last_became_peered":"2024-02-23T00:32:59.034079+0000","last_unstale":"2024-02-23T00:33:04.029600+0000","last_undegraded":"2024-02-23T00:33:04.029600+0000","last_fullsized":"2024-02-23T00:33:04.029600+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"158.7","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028529+0000","last_change":"2024-02-23T00:33:03.261079+0000","last_active":"2024-02-23T00:33:04.028529+0000","last_peered":"2024-02-23T00:33:04.028529+0000","last_clean":"2024-02-23T00:33:04.028529+0000","last_became_active":"2024-02-23T00:33:03.260759+0000","last_became_peered":"2024-02-23T00:33:03.260759+0000","last_unstale":"2024-02-23T00:33:04.028529+0000","last_undegraded":"2024-02-23T00:33:04.028529+0000","last_fullsized":"2024-02-23T00:33:04.028529+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.4","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024048+0000","last_change":"2024-02-23T00:33:01.034335+0000","last_active":"2024-02-23T00:33:04.024048+0000","last_peered":"2024-02-23T00:33:04.024048+0000","last_clean":"2024-02-23T00:33:04.024048+0000","last_became_active":"2024-02-23T00:33:01.034015+0000","last_became_peered":"2024-02-23T00:33:01.034015+0000","last_unstale":"2024-02-23T00:33:04.024048+0000","last_undegraded":"2024-02-23T00:33:04.024048+0000","last_fullsized":"2024-02-23T00:33:04.024048+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"156.5","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039319+0000","last_change":"2024-02-23T00:32:59.044851+0000","last_active":"2024-02-23T00:33:04.039319+0000","last_peered":"2024-02-23T00:33:04.039319+0000","last_clean":"2024-02-23T00:33:04.039319+0000","last_became_active":"2024-02-23T00:32:59.044607+0000","last_became_peered":"2024-02-23T00:32:59.044607+0000","last_unstale":"2024-02-23T00:33:04.039319+0000","last_undegraded":"2024-02-23T00:33:04.039319+0000","last_fullsized":"2024-02-23T00:33:04.039319+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"158.6","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023640+0000","last_change":"2024-02-23T00:33:03.257982+0000","last_active":"2024-02-23T00:33:04.023640+0000","last_peered":"2024-02-23T00:33:04.023640+0000","last_clean":"2024-02-23T00:33:04.023640+0000","last_became_active":"2024-02-23T00:33:03.257604+0000","last_became_peered":"2024-02-23T00:33:03.257604+0000","last_unstale":"2024-02-23T00:33:04.023640+0000","last_undegraded":"2024-02-23T00:33:04.023640+0000","last_fullsized":"2024-02-23T00:33:04.023640+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.5","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027092+0000","last_change":"2024-02-23T00:33:01.036449+0000","last_active":"2024-02-23T00:33:04.027092+0000","last_peered":"2024-02-23T00:33:04.027092+0000","last_clean":"2024-02-23T00:33:04.027092+0000","last_became_active":"2024-02-23T00:33:01.036160+0000","last_became_peered":"2024-02-23T00:33:01.036160+0000","last_unstale":"2024-02-23T00:33:04.027092+0000","last_undegraded":"2024-02-23T00:33:04.027092+0000","last_fullsized":"2024-02-23T00:33:04.027092+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.4","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024355+0000","last_change":"2024-02-23T00:32:59.044961+0000","last_active":"2024-02-23T00:33:04.024355+0000","last_peered":"2024-02-23T00:33:04.024355+0000","last_clean":"2024-02-23T00:33:04.024355+0000","last_became_active":"2024-02-23T00:32:59.044629+0000","last_became_peered":"2024-02-23T00:32:59.044629+0000","last_unstale":"2024-02-23T00:33:04.024355+0000","last_undegraded":"2024-02-23T00:33:04.024355+0000","last_fullsized":"2024-02-23T00:33:04.024355+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"158.1","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025896+0000","last_change":"2024-02-23T00:33:03.178657+0000","last_active":"2024-02-23T00:33:04.025896+0000","last_peered":"2024-02-23T00:33:04.025896+0000","last_clean":"2024-02-23T00:33:04.025896+0000","last_became_active":"2024-02-23T00:33:03.178355+0000","last_became_peered":"2024-02-23T00:33:03.178355+0000","last_unstale":"2024-02-23T00:33:04.025896+0000","last_undegraded":"2024-02-23T00:33:04.025896+0000","last_fullsized":"2024-02-23T00:33:04.025896+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.2","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036747+0000","last_change":"2024-02-23T00:33:01.043276+0000","last_active":"2024-02-23T00:33:04.036747+0000","last_peered":"2024-02-23T00:33:04.036747+0000","last_clean":"2024-02-23T00:33:04.036747+0000","last_became_active":"2024-02-23T00:33:01.042855+0000","last_became_peered":"2024-02-23T00:33:01.042855+0000","last_unstale":"2024-02-23T00:33:04.036747+0000","last_undegraded":"2024-02-23T00:33:04.036747+0000","last_fullsized":"2024-02-23T00:33:04.036747+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"156.3","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023643+0000","last_change":"2024-02-23T00:32:59.037077+0000","last_active":"2024-02-23T00:33:04.023643+0000","last_peered":"2024-02-23T00:33:04.023643+0000","last_clean":"2024-02-23T00:33:04.023643+0000","last_became_active":"2024-02-23T00:32:59.036783+0000","last_became_peered":"2024-02-23T00:32:59.036783+0000","last_unstale":"2024-02-23T00:33:04.023643+0000","last_undegraded":"2024-02-23T00:33:04.023643+0000","last_fullsized":"2024-02-23T00:33:04.023643+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"158.0","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.019609+0000","last_change":"2024-02-23T00:33:03.260302+0000","last_active":"2024-02-23T00:33:04.019609+0000","last_peered":"2024-02-23T00:33:04.019609+0000","last_clean":"2024-02-23T00:33:04.019609+0000","last_became_active":"2024-02-23T00:33:03.260070+0000","last_became_peered":"2024-02-23T00:33:03.260070+0000","last_unstale":"2024-02-23T00:33:04.019609+0000","last_undegraded":"2024-02-23T00:33:04.019609+0000","last_fullsized":"2024-02-23T00:33:04.019609+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"157.3","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021203+0000","last_change":"2024-02-23T00:33:01.042914+0000","last_active":"2024-02-23T00:33:04.021203+0000","last_peered":"2024-02-23T00:33:04.021203+0000","last_clean":"2024-02-23T00:33:04.021203+0000","last_became_active":"2024-02-23T00:33:01.042561+0000","last_became_peered":"2024-02-23T00:33:01.042561+0000","last_unstale":"2024-02-23T00:33:04.021203+0000","last_undegraded":"2024-02-23T00:33:04.021203+0000","last_fullsized":"2024-02-23T00:33:04.021203+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"156.2","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032520+0000","last_change":"2024-02-23T00:32:59.040152+0000","last_active":"2024-02-23T00:33:04.032520+0000","last_peered":"2024-02-23T00:33:04.032520+0000","last_clean":"2024-02-23T00:33:04.032520+0000","last_became_active":"2024-02-23T00:32:59.039794+0000","last_became_peered":"2024-02-23T00:32:59.039794+0000","last_unstale":"2024-02-23T00:33:04.032520+0000","last_undegraded":"2024-02-23T00:33:04.032520+0000","last_fullsized":"2024-02-23T00:33:04.032520+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"158.3","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028900+0000","last_change":"2024-02-23T00:33:03.023658+0000","last_active":"2024-02-23T00:33:04.028900+0000","last_peered":"2024-02-23T00:33:04.028900+0000","last_clean":"2024-02-23T00:33:04.028900+0000","last_became_active":"2024-02-23T00:33:03.023114+0000","last_became_peered":"2024-02-23T00:33:03.023114+0000","last_unstale":"2024-02-23T00:33:04.028900+0000","last_undegraded":"2024-02-23T00:33:04.028900+0000","last_fullsized":"2024-02-23T00:33:04.028900+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"157.0","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.019673+0000","last_change":"2024-02-23T00:33:01.036841+0000","last_active":"2024-02-23T00:33:04.019673+0000","last_peered":"2024-02-23T00:33:04.019673+0000","last_clean":"2024-02-23T00:33:04.019673+0000","last_became_active":"2024-02-23T00:33:01.036458+0000","last_became_peered":"2024-02-23T00:33:01.036458+0000","last_unstale":"2024-02-23T00:33:04.019673+0000","last_undegraded":"2024-02-23T00:33:04.019673+0000","last_fullsized":"2024-02-23T00:33:04.019673+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"156.1","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.026081+0000","last_change":"2024-02-23T00:32:59.038986+0000","last_active":"2024-02-23T00:33:04.026081+0000","last_peered":"2024-02-23T00:33:04.026081+0000","last_clean":"2024-02-23T00:33:04.026081+0000","last_became_active":"2024-02-23T00:32:59.038690+0000","last_became_peered":"2024-02-23T00:32:59.038690+0000","last_unstale":"2024-02-23T00:33:04.026081+0000","last_undegraded":"2024-02-23T00:33:04.026081+0000","last_fullsized":"2024-02-23T00:33:04.026081+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"158.2","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023352+0000","last_change":"2024-02-23T00:33:03.183882+0000","last_active":"2024-02-23T00:33:04.023352+0000","last_peered":"2024-02-23T00:33:04.023352+0000","last_clean":"2024-02-23T00:33:04.023352+0000","last_became_active":"2024-02-23T00:33:03.183567+0000","last_became_peered":"2024-02-23T00:33:03.183567+0000","last_unstale":"2024-02-23T00:33:04.023352+0000","last_undegraded":"2024-02-23T00:33:04.023352+0000","last_fullsized":"2024-02-23T00:33:04.023352+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"157.1","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030075+0000","last_change":"2024-02-23T00:33:01.050595+0000","last_active":"2024-02-23T00:33:04.030075+0000","last_peered":"2024-02-23T00:33:04.030075+0000","last_clean":"2024-02-23T00:33:04.030075+0000","last_became_active":"2024-02-23T00:33:01.050143+0000","last_became_peered":"2024-02-23T00:33:01.050143+0000","last_unstale":"2024-02-23T00:33:04.030075+0000","last_undegraded":"2024-02-23T00:33:04.030075+0000","last_fullsized":"2024-02-23T00:33:04.030075+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"156.0","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023076+0000","last_change":"2024-02-23T00:32:59.033301+0000","last_active":"2024-02-23T00:33:04.023076+0000","last_peered":"2024-02-23T00:33:04.023076+0000","last_clean":"2024-02-23T00:33:04.023076+0000","last_became_active":"2024-02-23T00:32:59.033002+0000","last_became_peered":"2024-02-23T00:32:59.033002+0000","last_unstale":"2024-02-23T00:33:04.023076+0000","last_undegraded":"2024-02-23T00:33:04.023076+0000","last_fullsized":"2024-02-23T00:33:04.023076+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"158.d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039028+0000","last_change":"2024-02-23T00:33:03.264958+0000","last_active":"2024-02-23T00:33:04.039028+0000","last_peered":"2024-02-23T00:33:04.039028+0000","last_clean":"2024-02-23T00:33:04.039028+0000","last_became_active":"2024-02-23T00:33:03.264724+0000","last_became_peered":"2024-02-23T00:33:03.264724+0000","last_unstale":"2024-02-23T00:33:04.039028+0000","last_undegraded":"2024-02-23T00:33:04.039028+0000","last_fullsized":"2024-02-23T00:33:04.039028+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.e","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023044+0000","last_change":"2024-02-23T00:33:01.046045+0000","last_active":"2024-02-23T00:33:04.023044+0000","last_peered":"2024-02-23T00:33:04.023044+0000","last_clean":"2024-02-23T00:33:04.023044+0000","last_became_active":"2024-02-23T00:33:01.045714+0000","last_became_peered":"2024-02-23T00:33:01.045714+0000","last_unstale":"2024-02-23T00:33:04.023044+0000","last_undegraded":"2024-02-23T00:33:04.023044+0000","last_fullsized":"2024-02-23T00:33:04.023044+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.f","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022686+0000","last_change":"2024-02-23T00:32:59.027196+0000","last_active":"2024-02-23T00:33:04.022686+0000","last_peered":"2024-02-23T00:33:04.022686+0000","last_clean":"2024-02-23T00:33:04.022686+0000","last_became_active":"2024-02-23T00:32:59.026890+0000","last_became_peered":"2024-02-23T00:32:59.026890+0000","last_unstale":"2024-02-23T00:33:04.022686+0000","last_undegraded":"2024-02-23T00:33:04.022686+0000","last_fullsized":"2024-02-23T00:33:04.022686+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"158.c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.026724+0000","last_change":"2024-02-23T00:33:03.261238+0000","last_active":"2024-02-23T00:33:04.026724+0000","last_peered":"2024-02-23T00:33:04.026724+0000","last_clean":"2024-02-23T00:33:04.026724+0000","last_became_active":"2024-02-23T00:33:03.260948+0000","last_became_peered":"2024-02-23T00:33:03.260948+0000","last_unstale":"2024-02-23T00:33:04.026724+0000","last_undegraded":"2024-02-23T00:33:04.026724+0000","last_fullsized":"2024-02-23T00:33:04.026724+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.f","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030407+0000","last_change":"2024-02-23T00:33:01.042280+0000","last_active":"2024-02-23T00:33:04.030407+0000","last_peered":"2024-02-23T00:33:04.030407+0000","last_clean":"2024-02-23T00:33:04.030407+0000","last_became_active":"2024-02-23T00:33:01.042008+0000","last_became_peered":"2024-02-23T00:33:01.042008+0000","last_unstale":"2024-02-23T00:33:04.030407+0000","last_undegraded":"2024-02-23T00:33:04.030407+0000","last_fullsized":"2024-02-23T00:33:04.030407+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"156.e","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028380+0000","last_change":"2024-02-23T00:32:59.043940+0000","last_active":"2024-02-23T00:33:04.028380+0000","last_peered":"2024-02-23T00:33:04.028380+0000","last_clean":"2024-02-23T00:33:04.028380+0000","last_became_active":"2024-02-23T00:32:59.043639+0000","last_became_peered":"2024-02-23T00:32:59.043639+0000","last_unstale":"2024-02-23T00:33:04.028380+0000","last_undegraded":"2024-02-23T00:33:04.028380+0000","last_fullsized":"2024-02-23T00:33:04.028380+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"158.f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.035448+0000","last_change":"2024-02-23T00:33:03.143950+0000","last_active":"2024-02-23T00:33:04.035448+0000","last_peered":"2024-02-23T00:33:04.035448+0000","last_clean":"2024-02-23T00:33:04.035448+0000","last_became_active":"2024-02-23T00:33:03.143602+0000","last_became_peered":"2024-02-23T00:33:03.143602+0000","last_unstale":"2024-02-23T00:33:04.035448+0000","last_undegraded":"2024-02-23T00:33:04.035448+0000","last_fullsized":"2024-02-23T00:33:04.035448+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.c","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023975+0000","last_change":"2024-02-23T00:33:01.038849+0000","last_active":"2024-02-23T00:33:04.023975+0000","last_peered":"2024-02-23T00:33:04.023975+0000","last_clean":"2024-02-23T00:33:04.023975+0000","last_became_active":"2024-02-23T00:33:01.038566+0000","last_became_peered":"2024-02-23T00:33:01.038566+0000","last_unstale":"2024-02-23T00:33:04.023975+0000","last_undegraded":"2024-02-23T00:33:04.023975+0000","last_fullsized":"2024-02-23T00:33:04.023975+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"156.d","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025364+0000","last_change":"2024-02-23T00:32:59.042042+0000","last_active":"2024-02-23T00:33:04.025364+0000","last_peered":"2024-02-23T00:33:04.025364+0000","last_clean":"2024-02-23T00:33:04.025364+0000","last_became_active":"2024-02-23T00:32:59.041620+0000","last_became_peered":"2024-02-23T00:32:59.041620+0000","last_unstale":"2024-02-23T00:33:04.025364+0000","last_undegraded":"2024-02-23T00:33:04.025364+0000","last_fullsized":"2024-02-23T00:33:04.025364+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"158.e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024056+0000","last_change":"2024-02-23T00:33:03.256492+0000","last_active":"2024-02-23T00:33:04.024056+0000","last_peered":"2024-02-23T00:33:04.024056+0000","last_clean":"2024-02-23T00:33:04.024056+0000","last_became_active":"2024-02-23T00:33:03.256278+0000","last_became_peered":"2024-02-23T00:33:03.256278+0000","last_unstale":"2024-02-23T00:33:04.024056+0000","last_undegraded":"2024-02-23T00:33:04.024056+0000","last_fullsized":"2024-02-23T00:33:04.024056+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"157.d","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025227+0000","last_change":"2024-02-23T00:33:01.046386+0000","last_active":"2024-02-23T00:33:04.025227+0000","last_peered":"2024-02-23T00:33:04.025227+0000","last_clean":"2024-02-23T00:33:04.025227+0000","last_became_active":"2024-02-23T00:33:01.046028+0000","last_became_peered":"2024-02-23T00:33:01.046028+0000","last_unstale":"2024-02-23T00:33:04.025227+0000","last_undegraded":"2024-02-23T00:33:04.025227+0000","last_fullsized":"2024-02-23T00:33:04.025227+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"156.c","version":"234'3","reported_seq":18,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.049055+0000","last_change":"2024-02-23T00:32:59.032011+0000","last_active":"2024-02-23T00:33:05.049055+0000","last_peered":"2024-02-23T00:33:05.049055+0000","last_clean":"2024-02-23T00:33:05.049055+0000","last_became_active":"2024-02-23T00:32:59.031678+0000","last_became_peered":"2024-02-23T00:32:59.031678+0000","last_unstale":"2024-02-23T00:33:05.049055+0000","last_undegraded":"2024-02-23T00:33:05.049055+0000","last_fullsized":"2024-02-23T00:33:05.049055+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"158.9","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.026892+0000","last_change":"2024-02-23T00:33:03.141545+0000","last_active":"2024-02-23T00:33:04.026892+0000","last_peered":"2024-02-23T00:33:04.026892+0000","last_clean":"2024-02-23T00:33:04.026892+0000","last_became_active":"2024-02-23T00:33:03.141259+0000","last_became_peered":"2024-02-23T00:33:03.141259+0000","last_unstale":"2024-02-23T00:33:04.026892+0000","last_undegraded":"2024-02-23T00:33:04.026892+0000","last_fullsized":"2024-02-23T00:33:04.026892+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.a","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022819+0000","last_change":"2024-02-23T00:33:01.034065+0000","last_active":"2024-02-23T00:33:04.022819+0000","last_peered":"2024-02-23T00:33:04.022819+0000","last_clean":"2024-02-23T00:33:04.022819+0000","last_became_active":"2024-02-23T00:33:01.033778+0000","last_became_peered":"2024-02-23T00:33:01.033778+0000","last_unstale":"2024-02-23T00:33:04.022819+0000","last_undegraded":"2024-02-23T00:33:04.022819+0000","last_fullsized":"2024-02-23T00:33:04.022819+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.b","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020931+0000","last_change":"2024-02-23T00:32:59.032110+0000","last_active":"2024-02-23T00:33:04.020931+0000","last_peered":"2024-02-23T00:33:04.020931+0000","last_clean":"2024-02-23T00:33:04.020931+0000","last_became_active":"2024-02-23T00:32:59.031802+0000","last_became_peered":"2024-02-23T00:32:59.031802+0000","last_unstale":"2024-02-23T00:33:04.020931+0000","last_undegraded":"2024-02-23T00:33:04.020931+0000","last_fullsized":"2024-02-23T00:33:04.020931+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"158.8","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022914+0000","last_change":"2024-02-23T00:33:03.184335+0000","last_active":"2024-02-23T00:33:04.022914+0000","last_peered":"2024-02-23T00:33:04.022914+0000","last_clean":"2024-02-23T00:33:04.022914+0000","last_became_active":"2024-02-23T00:33:03.184049+0000","last_became_peered":"2024-02-23T00:33:03.184049+0000","last_unstale":"2024-02-23T00:33:04.022914+0000","last_undegraded":"2024-02-23T00:33:04.022914+0000","last_fullsized":"2024-02-23T00:33:04.022914+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"157.b","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025053+0000","last_change":"2024-02-23T00:33:01.040519+0000","last_active":"2024-02-23T00:33:04.025053+0000","last_peered":"2024-02-23T00:33:04.025053+0000","last_clean":"2024-02-23T00:33:04.025053+0000","last_became_active":"2024-02-23T00:33:01.040143+0000","last_became_peered":"2024-02-23T00:33:01.040143+0000","last_unstale":"2024-02-23T00:33:04.025053+0000","last_undegraded":"2024-02-23T00:33:04.025053+0000","last_fullsized":"2024-02-23T00:33:04.025053+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"156.a","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024229+0000","last_change":"2024-02-23T00:32:59.045113+0000","last_active":"2024-02-23T00:33:04.024229+0000","last_peered":"2024-02-23T00:33:04.024229+0000","last_clean":"2024-02-23T00:33:04.024229+0000","last_became_active":"2024-02-23T00:32:59.044753+0000","last_became_peered":"2024-02-23T00:32:59.044753+0000","last_unstale":"2024-02-23T00:33:04.024229+0000","last_undegraded":"2024-02-23T00:33:04.024229+0000","last_fullsized":"2024-02-23T00:33:04.024229+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"158.b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032477+0000","last_change":"2024-02-23T00:33:03.138566+0000","last_active":"2024-02-23T00:33:04.032477+0000","last_peered":"2024-02-23T00:33:04.032477+0000","last_clean":"2024-02-23T00:33:04.032477+0000","last_became_active":"2024-02-23T00:33:03.138227+0000","last_became_peered":"2024-02-23T00:33:03.138227+0000","last_unstale":"2024-02-23T00:33:04.032477+0000","last_undegraded":"2024-02-23T00:33:04.032477+0000","last_fullsized":"2024-02-23T00:33:04.032477+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.8","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032409+0000","last_change":"2024-02-23T00:33:01.046943+0000","last_active":"2024-02-23T00:33:04.032409+0000","last_peered":"2024-02-23T00:33:04.032409+0000","last_clean":"2024-02-23T00:33:04.032409+0000","last_became_active":"2024-02-23T00:33:01.046554+0000","last_became_peered":"2024-02-23T00:33:01.046554+0000","last_unstale":"2024-02-23T00:33:04.032409+0000","last_undegraded":"2024-02-23T00:33:04.032409+0000","last_fullsized":"2024-02-23T00:33:04.032409+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"156.9","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023587+0000","last_change":"2024-02-23T00:32:59.029841+0000","last_active":"2024-02-23T00:33:04.023587+0000","last_peered":"2024-02-23T00:33:04.023587+0000","last_clean":"2024-02-23T00:33:04.023587+0000","last_became_active":"2024-02-23T00:32:59.029558+0000","last_became_peered":"2024-02-23T00:32:59.029558+0000","last_unstale":"2024-02-23T00:33:04.023587+0000","last_undegraded":"2024-02-23T00:33:04.023587+0000","last_fullsized":"2024-02-23T00:33:04.023587+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"158.a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024073+0000","last_change":"2024-02-23T00:33:03.257836+0000","last_active":"2024-02-23T00:33:04.024073+0000","last_peered":"2024-02-23T00:33:04.024073+0000","last_clean":"2024-02-23T00:33:04.024073+0000","last_became_active":"2024-02-23T00:33:03.257498+0000","last_became_peered":"2024-02-23T00:33:03.257498+0000","last_unstale":"2024-02-23T00:33:04.024073+0000","last_undegraded":"2024-02-23T00:33:04.024073+0000","last_fullsized":"2024-02-23T00:33:04.024073+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:02.001567+0000","last_clean_scrub_stamp":"2024-02-23T00:33:02.001567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.9","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027101+0000","last_change":"2024-02-23T00:33:01.049693+0000","last_active":"2024-02-23T00:33:04.027101+0000","last_peered":"2024-02-23T00:33:04.027101+0000","last_clean":"2024-02-23T00:33:04.027101+0000","last_became_active":"2024-02-23T00:33:01.048973+0000","last_became_peered":"2024-02-23T00:33:01.048973+0000","last_unstale":"2024-02-23T00:33:04.027101+0000","last_undegraded":"2024-02-23T00:33:04.027101+0000","last_fullsized":"2024-02-23T00:33:04.027101+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.015881+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.015881+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.8","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.019820+0000","last_change":"2024-02-23T00:32:59.038086+0000","last_active":"2024-02-23T00:33:04.019820+0000","last_peered":"2024-02-23T00:33:04.019820+0000","last_clean":"2024-02-23T00:33:04.019820+0000","last_became_active":"2024-02-23T00:32:59.037658+0000","last_became_peered":"2024-02-23T00:32:59.037658+0000","last_unstale":"2024-02-23T00:33:04.019820+0000","last_undegraded":"2024-02-23T00:33:04.019820+0000","last_fullsized":"2024-02-23T00:33:04.019820+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:58.012615+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.012615+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.0","version":"222'68","reported_seq":218,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.038946+0000","last_change":"2024-02-23T00:33:02.043252+0000","last_active":"2024-02-23T00:33:04.038946+0000","last_peered":"2024-02-23T00:33:04.038946+0000","last_clean":"2024-02-23T00:33:04.038946+0000","last_became_active":"2024-02-23T00:31:06.330477+0000","last_became_peered":"2024-02-23T00:31:06.330477+0000","last_unstale":"2024-02-23T00:33:04.038946+0000","last_undegraded":"2024-02-23T00:33:04.038946+0000","last_fullsized":"2024-02-23T00:33:04.038946+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1","version":"226'85","reported_seq":229,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020261+0000","last_change":"2024-02-23T00:33:00.024266+0000","last_active":"2024-02-23T00:33:04.020261+0000","last_peered":"2024-02-23T00:33:04.020261+0000","last_clean":"2024-02-23T00:33:04.020261+0000","last_became_active":"2024-02-23T00:31:06.258140+0000","last_became_peered":"2024-02-23T00:31:06.258140+0000","last_unstale":"2024-02-23T00:33:04.020261+0000","last_undegraded":"2024-02-23T00:33:04.020261+0000","last_fullsized":"2024-02-23T00:33:04.020261+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.2","version":"216'18","reported_seq":177,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037447+0000","last_change":"2024-02-23T00:33:02.043677+0000","last_active":"2024-02-23T00:33:04.037447+0000","last_peered":"2024-02-23T00:33:04.037447+0000","last_clean":"2024-02-23T00:33:04.037447+0000","last_became_active":"2024-02-23T00:31:06.329726+0000","last_became_peered":"2024-02-23T00:31:06.329726+0000","last_unstale":"2024-02-23T00:33:04.037447+0000","last_undegraded":"2024-02-23T00:33:04.037447+0000","last_fullsized":"2024-02-23T00:33:04.037447+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.3","version":"234'18","reported_seq":170,"reported_epoch":234,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:33:05.176955+0000","last_change":"2024-02-23T00:33:05.176978+0000","last_active":"2024-02-23T00:33:05.176955+0000","last_peered":"2024-02-23T00:33:05.176955+0000","last_clean":"2024-02-23T00:33:05.176955+0000","last_became_active":"2024-02-23T00:31:06.258997+0000","last_became_peered":"2024-02-23T00:31:06.258997+0000","last_unstale":"2024-02-23T00:33:05.176955+0000","last_undegraded":"2024-02-23T00:33:05.176955+0000","last_fullsized":"2024-02-23T00:33:05.176955+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"}]},{"pgid":"63.4","version":"234'37","reported_seq":179,"reported_epoch":234,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:33:05.134778+0000","last_change":"2024-02-23T00:33:05.133475+0000","last_active":"2024-02-23T00:33:05.134778+0000","last_peered":"2024-02-23T00:33:05.134778+0000","last_clean":"2024-02-23T00:33:05.134778+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:33:05.134778+0000","last_undegraded":"2024-02-23T00:33:05.134778+0000","last_fullsized":"2024-02-23T00:33:05.134778+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":30,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"}]},{"pgid":"63.5","version":"234'124","reported_seq":265,"reported_epoch":234,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:33:05.172245+0000","last_change":"2024-02-23T00:33:05.172245+0000","last_active":"2024-02-23T00:33:05.172245+0000","last_peered":"2024-02-23T00:33:05.172245+0000","last_clean":"2024-02-23T00:33:05.172245+0000","last_became_active":"2024-02-23T00:31:06.258422+0000","last_became_peered":"2024-02-23T00:31:06.258422+0000","last_unstale":"2024-02-23T00:33:05.172245+0000","last_undegraded":"2024-02-23T00:33:05.172245+0000","last_fullsized":"2024-02-23T00:33:05.172245+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":124,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"63.6","version":"206'51","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028542+0000","last_change":"2024-02-23T00:33:00.032676+0000","last_active":"2024-02-23T00:33:04.028542+0000","last_peered":"2024-02-23T00:33:04.028542+0000","last_clean":"2024-02-23T00:33:04.028542+0000","last_became_active":"2024-02-23T00:31:06.313441+0000","last_became_peered":"2024-02-23T00:31:06.313441+0000","last_unstale":"2024-02-23T00:33:04.028542+0000","last_undegraded":"2024-02-23T00:33:04.028542+0000","last_fullsized":"2024-02-23T00:33:04.028542+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:32:48.841846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.841846+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.7","version":"229'92","reported_seq":237,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037413+0000","last_change":"2024-02-23T00:33:02.043270+0000","last_active":"2024-02-23T00:33:04.037413+0000","last_peered":"2024-02-23T00:33:04.037413+0000","last_clean":"2024-02-23T00:33:04.037413+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:33:04.037413+0000","last_undegraded":"2024-02-23T00:33:04.037413+0000","last_fullsized":"2024-02-23T00:33:04.037413+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.8","version":"219'70","reported_seq":222,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023097+0000","last_change":"2024-02-23T00:33:00.035333+0000","last_active":"2024-02-23T00:33:04.023097+0000","last_peered":"2024-02-23T00:33:04.023097+0000","last_clean":"2024-02-23T00:33:04.023097+0000","last_became_active":"2024-02-23T00:31:06.363442+0000","last_became_peered":"2024-02-23T00:31:06.363442+0000","last_unstale":"2024-02-23T00:33:04.023097+0000","last_undegraded":"2024-02-23T00:33:04.023097+0000","last_fullsized":"2024-02-23T00:33:04.023097+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.9","version":"208'17","reported_seq":175,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036683+0000","last_change":"2024-02-23T00:33:00.033893+0000","last_active":"2024-02-23T00:33:04.036683+0000","last_peered":"2024-02-23T00:33:04.036683+0000","last_clean":"2024-02-23T00:33:04.036683+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:33:04.036683+0000","last_undegraded":"2024-02-23T00:33:04.036683+0000","last_fullsized":"2024-02-23T00:33:04.036683+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.a","version":"222'17","reported_seq":173,"reported_epoch":234,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:33:05.177321+0000","last_change":"2024-02-23T00:33:05.177346+0000","last_active":"2024-02-23T00:33:05.177321+0000","last_peered":"2024-02-23T00:33:05.177321+0000","last_clean":"2024-02-23T00:33:05.177321+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:33:05.177321+0000","last_undegraded":"2024-02-23T00:33:05.177321+0000","last_fullsized":"2024-02-23T00:33:05.177321+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"63.b","version":"202'34","reported_seq":188,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032894+0000","last_change":"2024-02-23T00:33:00.037998+0000","last_active":"2024-02-23T00:33:04.032894+0000","last_peered":"2024-02-23T00:33:04.032894+0000","last_clean":"2024-02-23T00:33:04.032894+0000","last_became_active":"2024-02-23T00:31:06.294667+0000","last_became_peered":"2024-02-23T00:31:06.294667+0000","last_unstale":"2024-02-23T00:33:04.032894+0000","last_undegraded":"2024-02-23T00:33:04.032894+0000","last_fullsized":"2024-02-23T00:33:04.032894+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.c","version":"208'68","reported_seq":215,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024683+0000","last_change":"2024-02-23T00:33:00.031268+0000","last_active":"2024-02-23T00:33:04.024683+0000","last_peered":"2024-02-23T00:33:04.024683+0000","last_clean":"2024-02-23T00:33:04.024683+0000","last_became_active":"2024-02-23T00:31:06.233844+0000","last_became_peered":"2024-02-23T00:31:06.233844+0000","last_unstale":"2024-02-23T00:33:04.024683+0000","last_undegraded":"2024-02-23T00:33:04.024683+0000","last_fullsized":"2024-02-23T00:33:04.024683+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.d","version":"224'34","reported_seq":179,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:33:04.025811+0000","last_change":"2024-02-23T00:32:57.135465+0000","last_active":"2024-02-23T00:33:04.025811+0000","last_peered":"2024-02-23T00:33:04.025811+0000","last_clean":"2024-02-23T00:33:04.025811+0000","last_became_active":"2024-02-23T00:31:06.262251+0000","last_became_peered":"2024-02-23T00:31:06.262251+0000","last_unstale":"2024-02-23T00:33:04.025811+0000","last_undegraded":"2024-02-23T00:33:04.025811+0000","last_fullsized":"2024-02-23T00:33:04.025811+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"}]},{"pgid":"63.e","version":"220'73","reported_seq":225,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037614+0000","last_change":"2024-02-23T00:33:00.040538+0000","last_active":"2024-02-23T00:33:04.037614+0000","last_peered":"2024-02-23T00:33:04.037614+0000","last_clean":"2024-02-23T00:33:04.037614+0000","last_became_active":"2024-02-23T00:31:06.356847+0000","last_became_peered":"2024-02-23T00:31:06.356847+0000","last_unstale":"2024-02-23T00:33:04.037614+0000","last_undegraded":"2024-02-23T00:33:04.037614+0000","last_fullsized":"2024-02-23T00:33:04.037614+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.f","version":"219'69","reported_seq":216,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030544+0000","last_change":"2024-02-23T00:33:00.038123+0000","last_active":"2024-02-23T00:33:04.030544+0000","last_peered":"2024-02-23T00:33:04.030544+0000","last_clean":"2024-02-23T00:33:04.030544+0000","last_became_active":"2024-02-23T00:31:06.364609+0000","last_became_peered":"2024-02-23T00:31:06.364609+0000","last_unstale":"2024-02-23T00:33:04.030544+0000","last_undegraded":"2024-02-23T00:33:04.030544+0000","last_fullsized":"2024-02-23T00:33:04.030544+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.10","version":"218'34","reported_seq":188,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037808+0000","last_change":"2024-02-23T00:33:00.041791+0000","last_active":"2024-02-23T00:33:04.037808+0000","last_peered":"2024-02-23T00:33:04.037808+0000","last_clean":"2024-02-23T00:33:04.037808+0000","last_became_active":"2024-02-23T00:31:06.342164+0000","last_became_peered":"2024-02-23T00:31:06.342164+0000","last_unstale":"2024-02-23T00:33:04.037808+0000","last_undegraded":"2024-02-23T00:33:04.037808+0000","last_fullsized":"2024-02-23T00:33:04.037808+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.11","version":"212'40","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020058+0000","last_change":"2024-02-23T00:33:00.023883+0000","last_active":"2024-02-23T00:33:04.020058+0000","last_peered":"2024-02-23T00:33:04.020058+0000","last_clean":"2024-02-23T00:33:04.020058+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:33:04.020058+0000","last_undegraded":"2024-02-23T00:33:04.020058+0000","last_fullsized":"2024-02-23T00:33:04.020058+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.12","version":"204'68","reported_seq":210,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024470+0000","last_change":"2024-02-23T00:33:00.030555+0000","last_active":"2024-02-23T00:33:04.024470+0000","last_peered":"2024-02-23T00:33:04.024470+0000","last_clean":"2024-02-23T00:33:04.024470+0000","last_became_active":"2024-02-23T00:31:06.281619+0000","last_became_peered":"2024-02-23T00:31:06.281619+0000","last_unstale":"2024-02-23T00:33:04.024470+0000","last_undegraded":"2024-02-23T00:33:04.024470+0000","last_fullsized":"2024-02-23T00:33:04.024470+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.13","version":"200'56","reported_seq":206,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032795+0000","last_change":"2024-02-23T00:33:00.038010+0000","last_active":"2024-02-23T00:33:04.032795+0000","last_peered":"2024-02-23T00:33:04.032795+0000","last_clean":"2024-02-23T00:33:04.032795+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:33:04.032795+0000","last_undegraded":"2024-02-23T00:33:04.032795+0000","last_fullsized":"2024-02-23T00:33:04.032795+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.14","version":"208'85","reported_seq":228,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024499+0000","last_change":"2024-02-23T00:33:00.031035+0000","last_active":"2024-02-23T00:33:04.024499+0000","last_peered":"2024-02-23T00:33:04.024499+0000","last_clean":"2024-02-23T00:33:04.024499+0000","last_became_active":"2024-02-23T00:31:06.234521+0000","last_became_peered":"2024-02-23T00:31:06.234521+0000","last_unstale":"2024-02-23T00:33:04.024499+0000","last_undegraded":"2024-02-23T00:33:04.024499+0000","last_fullsized":"2024-02-23T00:33:04.024499+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.15","version":"220'68","reported_seq":218,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024955+0000","last_change":"2024-02-23T00:33:00.031524+0000","last_active":"2024-02-23T00:33:04.024955+0000","last_peered":"2024-02-23T00:33:04.024955+0000","last_clean":"2024-02-23T00:33:04.024955+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:33:04.024955+0000","last_undegraded":"2024-02-23T00:33:04.024955+0000","last_fullsized":"2024-02-23T00:33:04.024955+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.16","version":"204'51","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028469+0000","last_change":"2024-02-23T00:33:00.032292+0000","last_active":"2024-02-23T00:33:04.028469+0000","last_peered":"2024-02-23T00:33:04.028469+0000","last_clean":"2024-02-23T00:33:04.028469+0000","last_became_active":"2024-02-23T00:31:06.367069+0000","last_became_peered":"2024-02-23T00:31:06.367069+0000","last_unstale":"2024-02-23T00:33:04.028469+0000","last_undegraded":"2024-02-23T00:33:04.028469+0000","last_fullsized":"2024-02-23T00:33:04.028469+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:32:57.837196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:57.837196+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.17","version":"230'85","reported_seq":226,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:33:04.027557+0000","last_change":"2024-02-23T00:33:03.175527+0000","last_active":"2024-02-23T00:33:04.027557+0000","last_peered":"2024-02-23T00:33:04.027557+0000","last_clean":"2024-02-23T00:33:04.027557+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:33:04.027557+0000","last_undegraded":"2024-02-23T00:33:04.027557+0000","last_fullsized":"2024-02-23T00:33:04.027557+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"}]},{"pgid":"63.18","version":"211'57","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031288+0000","last_change":"2024-02-23T00:33:00.035022+0000","last_active":"2024-02-23T00:33:04.031288+0000","last_peered":"2024-02-23T00:33:04.031288+0000","last_clean":"2024-02-23T00:33:04.031288+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:33:04.031288+0000","last_undegraded":"2024-02-23T00:33:04.031288+0000","last_fullsized":"2024-02-23T00:33:04.031288+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.19","version":"216'51","reported_seq":207,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.019955+0000","last_change":"2024-02-23T00:33:00.023463+0000","last_active":"2024-02-23T00:33:04.019955+0000","last_peered":"2024-02-23T00:33:04.019955+0000","last_clean":"2024-02-23T00:33:04.019955+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:33:04.019955+0000","last_undegraded":"2024-02-23T00:33:04.019955+0000","last_fullsized":"2024-02-23T00:33:04.019955+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1a","version":"202'68","reported_seq":219,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023246+0000","last_change":"2024-02-23T00:33:00.023838+0000","last_active":"2024-02-23T00:33:04.023246+0000","last_peered":"2024-02-23T00:33:04.023246+0000","last_clean":"2024-02-23T00:33:04.023246+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:33:04.023246+0000","last_undegraded":"2024-02-23T00:33:04.023246+0000","last_fullsized":"2024-02-23T00:33:04.023246+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'68","last_scrub_stamp":"2024-02-23T00:32:58.808102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.808102+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1b","version":"234'49","reported_seq":190,"reported_epoch":234,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:33:05.178576+0000","last_change":"2024-02-23T00:33:05.177358+0000","last_active":"2024-02-23T00:33:05.178576+0000","last_peered":"2024-02-23T00:33:05.178576+0000","last_clean":"2024-02-23T00:33:05.178576+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:33:05.178576+0000","last_undegraded":"2024-02-23T00:33:05.178576+0000","last_fullsized":"2024-02-23T00:33:05.178576+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"}]},{"pgid":"63.1c","version":"216'34","reported_seq":187,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036623+0000","last_change":"2024-02-23T00:33:02.043705+0000","last_active":"2024-02-23T00:33:04.036623+0000","last_peered":"2024-02-23T00:33:04.036623+0000","last_clean":"2024-02-23T00:33:04.036623+0000","last_became_active":"2024-02-23T00:31:06.338540+0000","last_became_peered":"2024-02-23T00:31:06.338540+0000","last_unstale":"2024-02-23T00:33:04.036623+0000","last_undegraded":"2024-02-23T00:33:04.036623+0000","last_fullsized":"2024-02-23T00:33:04.036623+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1d","version":"229'10","reported_seq":172,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039307+0000","last_change":"2024-02-23T00:33:02.042975+0000","last_active":"2024-02-23T00:33:04.039307+0000","last_peered":"2024-02-23T00:33:04.039307+0000","last_clean":"2024-02-23T00:33:04.039307+0000","last_became_active":"2024-02-23T00:31:06.337635+0000","last_became_peered":"2024-02-23T00:31:06.337635+0000","last_unstale":"2024-02-23T00:33:04.039307+0000","last_undegraded":"2024-02-23T00:33:04.039307+0000","last_fullsized":"2024-02-23T00:33:04.039307+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1e","version":"232'51","reported_seq":193,"reported_epoch":234,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:33:05.133374+0000","last_change":"2024-02-23T00:33:05.133416+0000","last_active":"2024-02-23T00:33:05.133374+0000","last_peered":"2024-02-23T00:33:05.133374+0000","last_clean":"2024-02-23T00:33:05.133374+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:33:05.133374+0000","last_undegraded":"2024-02-23T00:33:05.133374+0000","last_fullsized":"2024-02-23T00:33:05.133374+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"}]},{"pgid":"63.1f","version":"228'68","reported_seq":211,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-23T00:33:04.027440+0000","last_change":"2024-02-23T00:33:01.129637+0000","last_active":"2024-02-23T00:33:04.027440+0000","last_peered":"2024-02-23T00:33:04.027440+0000","last_clean":"2024-02-23T00:33:04.027440+0000","last_became_active":"2024-02-23T00:31:06.355623+0000","last_became_peered":"2024-02-23T00:31:06.355623+0000","last_unstale":"2024-02-23T00:33:04.027440+0000","last_undegraded":"2024-02-23T00:33:04.027440+0000","last_fullsized":"2024-02-23T00:33:04.027440+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"}]},{"pgid":"5.9","version":"62'8","reported_seq":276,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.034296+0000","last_change":"2024-02-23T00:32:11.743347+0000","last_active":"2024-02-23T00:33:05.034296+0000","last_peered":"2024-02-23T00:33:05.034296+0000","last_clean":"2024-02-23T00:33:05.034296+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:33:05.034296+0000","last_undegraded":"2024-02-23T00:33:05.034296+0000","last_fullsized":"2024-02-23T00:33:05.034296+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:11.743278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.743278+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":263,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021953+0000","last_change":"2024-02-23T00:32:47.376760+0000","last_active":"2024-02-23T00:33:04.021953+0000","last_peered":"2024-02-23T00:33:04.021953+0000","last_clean":"2024-02-23T00:33:04.021953+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:33:04.021953+0000","last_undegraded":"2024-02-23T00:33:04.021953+0000","last_fullsized":"2024-02-23T00:33:04.021953+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:32:47.376691+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.376691+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028601+0000","last_change":"2024-02-23T00:33:00.878453+0000","last_active":"2024-02-23T00:33:04.028601+0000","last_peered":"2024-02-23T00:33:04.028601+0000","last_clean":"2024-02-23T00:33:04.028601+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:33:04.028601+0000","last_undegraded":"2024-02-23T00:33:04.028601+0000","last_fullsized":"2024-02-23T00:33:04.028601+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.878391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.878391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":190,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022750+0000","last_change":"2024-02-23T00:31:31.323024+0000","last_active":"2024-02-23T00:33:04.022750+0000","last_peered":"2024-02-23T00:33:04.022750+0000","last_clean":"2024-02-23T00:33:04.022750+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:33:04.022750+0000","last_undegraded":"2024-02-23T00:33:04.022750+0000","last_fullsized":"2024-02-23T00:33:04.022750+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.322940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.322940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":149,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.038965+0000","last_change":"2024-02-23T00:32:48.468225+0000","last_active":"2024-02-23T00:33:04.038965+0000","last_peered":"2024-02-23T00:33:04.038965+0000","last_clean":"2024-02-23T00:33:04.038965+0000","last_became_active":"2024-02-23T00:30:48.777752+0000","last_became_peered":"2024-02-23T00:30:48.777752+0000","last_unstale":"2024-02-23T00:33:04.038965+0000","last_undegraded":"2024-02-23T00:33:04.038965+0000","last_fullsized":"2024-02-23T00:33:04.038965+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":213,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.029219+0000","last_change":"2024-02-23T00:32:37.440685+0000","last_active":"2024-02-23T00:33:05.029219+0000","last_peered":"2024-02-23T00:33:05.029219+0000","last_clean":"2024-02-23T00:33:05.029219+0000","last_became_active":"2024-02-23T00:31:29.077539+0000","last_became_peered":"2024-02-23T00:31:29.077539+0000","last_unstale":"2024-02-23T00:33:05.029219+0000","last_undegraded":"2024-02-23T00:33:05.029219+0000","last_fullsized":"2024-02-23T00:33:05.029219+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:37.440614+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.440614+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":232,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021428+0000","last_change":"2024-02-23T00:32:09.590050+0000","last_active":"2024-02-23T00:33:04.021428+0000","last_peered":"2024-02-23T00:33:04.021428+0000","last_clean":"2024-02-23T00:33:04.021428+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:33:04.021428+0000","last_undegraded":"2024-02-23T00:33:04.021428+0000","last_fullsized":"2024-02-23T00:33:04.021428+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:32:09.589983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.589983+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.040660+0000","last_change":"2024-02-23T00:32:38.328306+0000","last_active":"2024-02-23T00:33:04.040660+0000","last_peered":"2024-02-23T00:33:04.040660+0000","last_clean":"2024-02-23T00:33:04.040660+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:33:04.040660+0000","last_undegraded":"2024-02-23T00:33:04.040660+0000","last_fullsized":"2024-02-23T00:33:04.040660+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.328237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.328237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023390+0000","last_change":"2024-02-23T00:32:40.855264+0000","last_active":"2024-02-23T00:33:04.023390+0000","last_peered":"2024-02-23T00:33:04.023390+0000","last_clean":"2024-02-23T00:33:04.023390+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:33:04.023390+0000","last_undegraded":"2024-02-23T00:33:04.023390+0000","last_fullsized":"2024-02-23T00:33:04.023390+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.855194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.855194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020482+0000","last_change":"2024-02-23T00:32:06.561440+0000","last_active":"2024-02-23T00:33:04.020482+0000","last_peered":"2024-02-23T00:33:04.020482+0000","last_clean":"2024-02-23T00:33:04.020482+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:33:04.020482+0000","last_undegraded":"2024-02-23T00:33:04.020482+0000","last_fullsized":"2024-02-23T00:33:04.020482+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:06.561374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:06.561374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027923+0000","last_change":"2024-02-23T00:30:48.713856+0000","last_active":"2024-02-23T00:33:04.027923+0000","last_peered":"2024-02-23T00:33:04.027923+0000","last_clean":"2024-02-23T00:33:04.027923+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:33:04.027923+0000","last_undegraded":"2024-02-23T00:33:04.027923+0000","last_fullsized":"2024-02-23T00:33:04.027923+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":247,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024803+0000","last_change":"2024-02-23T00:31:41.019548+0000","last_active":"2024-02-23T00:33:04.024803+0000","last_peered":"2024-02-23T00:33:04.024803+0000","last_clean":"2024-02-23T00:33:04.024803+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:33:04.024803+0000","last_undegraded":"2024-02-23T00:33:04.024803+0000","last_fullsized":"2024-02-23T00:33:04.024803+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.019441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.019441+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027139+0000","last_change":"2024-02-23T00:32:59.849579+0000","last_active":"2024-02-23T00:33:04.027139+0000","last_peered":"2024-02-23T00:33:04.027139+0000","last_clean":"2024-02-23T00:33:04.027139+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:33:04.027139+0000","last_undegraded":"2024-02-23T00:33:04.027139+0000","last_fullsized":"2024-02-23T00:33:04.027139+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.849503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.849503+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023566+0000","last_change":"2024-02-23T00:32:46.885745+0000","last_active":"2024-02-23T00:33:04.023566+0000","last_peered":"2024-02-23T00:33:04.023566+0000","last_clean":"2024-02-23T00:33:04.023566+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:33:04.023566+0000","last_undegraded":"2024-02-23T00:33:04.023566+0000","last_fullsized":"2024-02-23T00:33:04.023566+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.885681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.885681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031528+0000","last_change":"2024-02-23T00:32:38.683758+0000","last_active":"2024-02-23T00:33:04.031528+0000","last_peered":"2024-02-23T00:33:04.031528+0000","last_clean":"2024-02-23T00:33:04.031528+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:33:04.031528+0000","last_undegraded":"2024-02-23T00:33:04.031528+0000","last_fullsized":"2024-02-23T00:33:04.031528+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.683686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.683686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":150,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021281+0000","last_change":"2024-02-23T00:32:45.536091+0000","last_active":"2024-02-23T00:33:04.021281+0000","last_peered":"2024-02-23T00:33:04.021281+0000","last_clean":"2024-02-23T00:33:04.021281+0000","last_became_active":"2024-02-23T00:30:48.673452+0000","last_became_peered":"2024-02-23T00:30:48.673452+0000","last_unstale":"2024-02-23T00:33:04.021281+0000","last_undegraded":"2024-02-23T00:33:04.021281+0000","last_fullsized":"2024-02-23T00:33:04.021281+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.535992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.535992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":212,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025090+0000","last_change":"2024-02-23T00:32:37.445055+0000","last_active":"2024-02-23T00:33:04.025090+0000","last_peered":"2024-02-23T00:33:04.025090+0000","last_clean":"2024-02-23T00:33:04.025090+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:33:04.025090+0000","last_undegraded":"2024-02-23T00:33:04.025090+0000","last_fullsized":"2024-02-23T00:33:04.025090+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.444973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.444973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":224,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025707+0000","last_change":"2024-02-23T00:30:35.121011+0000","last_active":"2024-02-23T00:33:04.025707+0000","last_peered":"2024-02-23T00:33:04.025707+0000","last_clean":"2024-02-23T00:33:04.025707+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:33:04.025707+0000","last_undegraded":"2024-02-23T00:33:04.025707+0000","last_fullsized":"2024-02-23T00:33:04.025707+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:30:35.120909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.120909+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032992+0000","last_change":"2024-02-23T00:32:56.692114+0000","last_active":"2024-02-23T00:33:04.032992+0000","last_peered":"2024-02-23T00:33:04.032992+0000","last_clean":"2024-02-23T00:33:04.032992+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:33:04.032992+0000","last_undegraded":"2024-02-23T00:33:04.032992+0000","last_fullsized":"2024-02-23T00:33:04.032992+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.692037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.692037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":193,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.038974+0000","last_change":"2024-02-23T00:32:09.110225+0000","last_active":"2024-02-23T00:33:04.038974+0000","last_peered":"2024-02-23T00:33:04.038974+0000","last_clean":"2024-02-23T00:33:04.038974+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:33:04.038974+0000","last_undegraded":"2024-02-23T00:33:04.038974+0000","last_fullsized":"2024-02-23T00:33:04.038974+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.110138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028932+0000","last_change":"2024-02-23T00:32:15.855653+0000","last_active":"2024-02-23T00:33:04.028932+0000","last_peered":"2024-02-23T00:33:04.028932+0000","last_clean":"2024-02-23T00:33:04.028932+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:33:04.028932+0000","last_undegraded":"2024-02-23T00:33:04.028932+0000","last_fullsized":"2024-02-23T00:33:04.028932+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:32:15.855580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":150,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036380+0000","last_change":"2024-02-23T00:33:00.671292+0000","last_active":"2024-02-23T00:33:04.036380+0000","last_peered":"2024-02-23T00:33:04.036380+0000","last_clean":"2024-02-23T00:33:04.036380+0000","last_became_active":"2024-02-23T00:30:48.716552+0000","last_became_peered":"2024-02-23T00:30:48.716552+0000","last_unstale":"2024-02-23T00:33:04.036380+0000","last_undegraded":"2024-02-23T00:33:04.036380+0000","last_fullsized":"2024-02-23T00:33:04.036380+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.671219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.671219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":212,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024034+0000","last_change":"2024-02-23T00:32:53.985225+0000","last_active":"2024-02-23T00:33:04.024034+0000","last_peered":"2024-02-23T00:33:04.024034+0000","last_clean":"2024-02-23T00:33:04.024034+0000","last_became_active":"2024-02-23T00:31:29.064244+0000","last_became_peered":"2024-02-23T00:31:29.064244+0000","last_unstale":"2024-02-23T00:33:04.024034+0000","last_undegraded":"2024-02-23T00:33:04.024034+0000","last_fullsized":"2024-02-23T00:33:04.024034+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.985150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.985150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":211,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027526+0000","last_change":"2024-02-23T00:31:19.982059+0000","last_active":"2024-02-23T00:33:04.027526+0000","last_peered":"2024-02-23T00:33:04.027526+0000","last_clean":"2024-02-23T00:33:04.027526+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:33:04.027526+0000","last_undegraded":"2024-02-23T00:33:04.027526+0000","last_fullsized":"2024-02-23T00:33:04.027526+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:31:19.981985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.981985+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":215,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037665+0000","last_change":"2024-02-23T00:31:49.024710+0000","last_active":"2024-02-23T00:33:04.037665+0000","last_peered":"2024-02-23T00:33:04.037665+0000","last_clean":"2024-02-23T00:33:04.037665+0000","last_became_active":"2024-02-23T00:31:49.024302+0000","last_became_peered":"2024-02-23T00:31:49.024302+0000","last_unstale":"2024-02-23T00:33:04.037665+0000","last_undegraded":"2024-02-23T00:33:04.037665+0000","last_fullsized":"2024-02-23T00:33:04.037665+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.937969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":277,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.022836+0000","last_change":"2024-02-23T00:32:35.929043+0000","last_active":"2024-02-23T00:33:05.022836+0000","last_peered":"2024-02-23T00:33:05.022836+0000","last_clean":"2024-02-23T00:33:05.022836+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:33:05.022836+0000","last_undegraded":"2024-02-23T00:33:05.022836+0000","last_fullsized":"2024-02-23T00:33:05.022836+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.928977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.928977+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024314+0000","last_change":"2024-02-23T00:30:31.065478+0000","last_active":"2024-02-23T00:33:04.024314+0000","last_peered":"2024-02-23T00:33:04.024314+0000","last_clean":"2024-02-23T00:33:04.024314+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:33:04.024314+0000","last_undegraded":"2024-02-23T00:33:04.024314+0000","last_fullsized":"2024-02-23T00:33:04.024314+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.065384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.065384+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024399+0000","last_change":"2024-02-23T00:30:48.656349+0000","last_active":"2024-02-23T00:33:04.024399+0000","last_peered":"2024-02-23T00:33:04.024399+0000","last_clean":"2024-02-23T00:33:04.024399+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:33:04.024399+0000","last_undegraded":"2024-02-23T00:33:04.024399+0000","last_fullsized":"2024-02-23T00:33:04.024399+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":111,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024616+0000","last_change":"2024-02-23T00:32:58.975408+0000","last_active":"2024-02-23T00:33:04.024616+0000","last_peered":"2024-02-23T00:33:04.024616+0000","last_clean":"2024-02-23T00:33:04.024616+0000","last_became_active":"2024-02-23T00:31:29.089319+0000","last_became_peered":"2024-02-23T00:31:29.089319+0000","last_unstale":"2024-02-23T00:33:04.024616+0000","last_undegraded":"2024-02-23T00:33:04.024616+0000","last_fullsized":"2024-02-23T00:33:04.024616+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:58.975334+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.975334+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"87'22","reported_seq":232,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039163+0000","last_change":"2024-02-23T00:32:40.327594+0000","last_active":"2024-02-23T00:33:04.039163+0000","last_peered":"2024-02-23T00:33:04.039163+0000","last_clean":"2024-02-23T00:33:04.039163+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:33:04.039163+0000","last_undegraded":"2024-02-23T00:33:04.039163+0000","last_fullsized":"2024-02-23T00:33:04.039163+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:32:40.327519+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.327519+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036852+0000","last_change":"2024-02-23T00:31:29.920657+0000","last_active":"2024-02-23T00:33:04.036852+0000","last_peered":"2024-02-23T00:33:04.036852+0000","last_clean":"2024-02-23T00:33:04.036852+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:33:04.036852+0000","last_undegraded":"2024-02-23T00:33:04.036852+0000","last_fullsized":"2024-02-23T00:33:04.036852+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:29.920546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:29.920546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":277,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.035540+0000","last_change":"2024-02-23T00:32:35.975973+0000","last_active":"2024-02-23T00:33:05.035540+0000","last_peered":"2024-02-23T00:33:05.035540+0000","last_clean":"2024-02-23T00:33:05.035540+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:33:05.035540+0000","last_undegraded":"2024-02-23T00:33:05.035540+0000","last_fullsized":"2024-02-23T00:33:05.035540+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.975889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.975889+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.026251+0000","last_change":"2024-02-23T00:31:21.366604+0000","last_active":"2024-02-23T00:33:04.026251+0000","last_peered":"2024-02-23T00:33:04.026251+0000","last_clean":"2024-02-23T00:33:04.026251+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:33:04.026251+0000","last_undegraded":"2024-02-23T00:33:04.026251+0000","last_fullsized":"2024-02-23T00:33:04.026251+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:31:21.366535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.366535+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":144,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039486+0000","last_change":"2024-02-23T00:30:48.713004+0000","last_active":"2024-02-23T00:33:04.039486+0000","last_peered":"2024-02-23T00:33:04.039486+0000","last_clean":"2024-02-23T00:33:04.039486+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:33:04.039486+0000","last_undegraded":"2024-02-23T00:33:04.039486+0000","last_fullsized":"2024-02-23T00:33:04.039486+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":212,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.019851+0000","last_change":"2024-02-23T00:32:47.737933+0000","last_active":"2024-02-23T00:33:04.019851+0000","last_peered":"2024-02-23T00:33:04.019851+0000","last_clean":"2024-02-23T00:33:04.019851+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:33:04.019851+0000","last_undegraded":"2024-02-23T00:33:04.019851+0000","last_fullsized":"2024-02-23T00:33:04.019851+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:47.737874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.737874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"86'23","reported_seq":241,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023944+0000","last_change":"2024-02-23T00:32:46.780054+0000","last_active":"2024-02-23T00:33:04.023944+0000","last_peered":"2024-02-23T00:33:04.023944+0000","last_clean":"2024-02-23T00:33:04.023944+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:33:04.023944+0000","last_undegraded":"2024-02-23T00:33:04.023944+0000","last_fullsized":"2024-02-23T00:33:04.023944+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'23","last_scrub_stamp":"2024-02-23T00:32:46.779979+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.779979+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031823+0000","last_change":"2024-02-23T00:31:38.838289+0000","last_active":"2024-02-23T00:33:04.031823+0000","last_peered":"2024-02-23T00:33:04.031823+0000","last_clean":"2024-02-23T00:33:04.031823+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:33:04.031823+0000","last_undegraded":"2024-02-23T00:33:04.031823+0000","last_fullsized":"2024-02-23T00:33:04.031823+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:31:38.838202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.838202+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028116+0000","last_change":"2024-02-23T00:31:33.381301+0000","last_active":"2024-02-23T00:33:04.028116+0000","last_peered":"2024-02-23T00:33:04.028116+0000","last_clean":"2024-02-23T00:33:04.028116+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:33:04.028116+0000","last_undegraded":"2024-02-23T00:33:04.028116+0000","last_fullsized":"2024-02-23T00:33:04.028116+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.381229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.381229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031629+0000","last_change":"2024-02-23T00:32:09.012254+0000","last_active":"2024-02-23T00:33:04.031629+0000","last_peered":"2024-02-23T00:33:04.031629+0000","last_clean":"2024-02-23T00:33:04.031629+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:33:04.031629+0000","last_undegraded":"2024-02-23T00:33:04.031629+0000","last_fullsized":"2024-02-23T00:33:04.031629+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.012161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.012161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":149,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039062+0000","last_change":"2024-02-23T00:32:53.387558+0000","last_active":"2024-02-23T00:33:04.039062+0000","last_peered":"2024-02-23T00:33:04.039062+0000","last_clean":"2024-02-23T00:33:04.039062+0000","last_became_active":"2024-02-23T00:30:48.740636+0000","last_became_peered":"2024-02-23T00:30:48.740636+0000","last_unstale":"2024-02-23T00:33:04.039062+0000","last_undegraded":"2024-02-23T00:33:04.039062+0000","last_fullsized":"2024-02-23T00:33:04.039062+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.387496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":116,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021129+0000","last_change":"2024-02-23T00:32:56.610957+0000","last_active":"2024-02-23T00:33:04.021129+0000","last_peered":"2024-02-23T00:33:04.021129+0000","last_clean":"2024-02-23T00:33:04.021129+0000","last_became_active":"2024-02-23T00:31:29.108747+0000","last_became_peered":"2024-02-23T00:31:29.108747+0000","last_unstale":"2024-02-23T00:33:04.021129+0000","last_undegraded":"2024-02-23T00:33:04.021129+0000","last_fullsized":"2024-02-23T00:33:04.021129+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.610884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.610884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":276,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.038182+0000","last_change":"2024-02-23T00:31:41.945984+0000","last_active":"2024-02-23T00:33:05.038182+0000","last_peered":"2024-02-23T00:33:05.038182+0000","last_clean":"2024-02-23T00:33:05.038182+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:33:05.038182+0000","last_undegraded":"2024-02-23T00:33:05.038182+0000","last_fullsized":"2024-02-23T00:33:05.038182+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":232,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020856+0000","last_change":"2024-02-23T00:31:38.038523+0000","last_active":"2024-02-23T00:33:04.020856+0000","last_peered":"2024-02-23T00:33:04.020856+0000","last_clean":"2024-02-23T00:33:04.020856+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:33:04.020856+0000","last_undegraded":"2024-02-23T00:33:04.020856+0000","last_fullsized":"2024-02-23T00:33:04.020856+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:31:38.038413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.038413+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.026985+0000","last_change":"2024-02-23T00:32:46.504136+0000","last_active":"2024-02-23T00:33:04.026985+0000","last_peered":"2024-02-23T00:33:04.026985+0000","last_clean":"2024-02-23T00:33:04.026985+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:33:04.026985+0000","last_undegraded":"2024-02-23T00:33:04.026985+0000","last_fullsized":"2024-02-23T00:33:04.026985+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.504045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.504045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.033608+0000","last_change":"2024-02-23T00:31:31.946435+0000","last_active":"2024-02-23T00:33:04.033608+0000","last_peered":"2024-02-23T00:33:04.033608+0000","last_clean":"2024-02-23T00:33:04.033608+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:33:04.033608+0000","last_undegraded":"2024-02-23T00:33:04.033608+0000","last_fullsized":"2024-02-23T00:33:04.033608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.946366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.946366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":144,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037407+0000","last_change":"2024-02-23T00:30:48.782732+0000","last_active":"2024-02-23T00:33:04.037407+0000","last_peered":"2024-02-23T00:33:04.037407+0000","last_clean":"2024-02-23T00:33:04.037407+0000","last_became_active":"2024-02-23T00:30:48.778293+0000","last_became_peered":"2024-02-23T00:30:48.778293+0000","last_unstale":"2024-02-23T00:33:04.037407+0000","last_undegraded":"2024-02-23T00:33:04.037407+0000","last_fullsized":"2024-02-23T00:33:04.037407+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":212,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031143+0000","last_change":"2024-02-23T00:32:55.991569+0000","last_active":"2024-02-23T00:33:04.031143+0000","last_peered":"2024-02-23T00:33:04.031143+0000","last_clean":"2024-02-23T00:33:04.031143+0000","last_became_active":"2024-02-23T00:31:29.070521+0000","last_became_peered":"2024-02-23T00:31:29.070521+0000","last_unstale":"2024-02-23T00:33:04.031143+0000","last_undegraded":"2024-02-23T00:33:04.031143+0000","last_fullsized":"2024-02-23T00:33:04.031143+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:55.991514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:55.991514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021203+0000","last_change":"2024-02-23T00:32:41.922931+0000","last_active":"2024-02-23T00:33:04.021203+0000","last_peered":"2024-02-23T00:33:04.021203+0000","last_clean":"2024-02-23T00:33:04.021203+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:33:04.021203+0000","last_undegraded":"2024-02-23T00:33:04.021203+0000","last_fullsized":"2024-02-23T00:33:04.021203+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:41.922849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.922849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":258,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024363+0000","last_change":"2024-02-23T00:31:21.086750+0000","last_active":"2024-02-23T00:33:04.024363+0000","last_peered":"2024-02-23T00:33:04.024363+0000","last_clean":"2024-02-23T00:33:04.024363+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:33:04.024363+0000","last_undegraded":"2024-02-23T00:33:04.024363+0000","last_fullsized":"2024-02-23T00:33:04.024363+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:31:21.086610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.086610+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037711+0000","last_change":"2024-02-23T00:30:25.593913+0000","last_active":"2024-02-23T00:33:04.037711+0000","last_peered":"2024-02-23T00:33:04.037711+0000","last_clean":"2024-02-23T00:33:04.037711+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:33:04.037711+0000","last_undegraded":"2024-02-23T00:33:04.037711+0000","last_fullsized":"2024-02-23T00:33:04.037711+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.593833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.593833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.029113+0000","last_change":"2024-02-23T00:32:17.981331+0000","last_active":"2024-02-23T00:33:04.029113+0000","last_peered":"2024-02-23T00:33:04.029113+0000","last_clean":"2024-02-23T00:33:04.029113+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:33:04.029113+0000","last_undegraded":"2024-02-23T00:33:04.029113+0000","last_fullsized":"2024-02-23T00:33:04.029113+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.981262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.981262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023633+0000","last_change":"2024-02-23T00:30:48.776316+0000","last_active":"2024-02-23T00:33:04.023633+0000","last_peered":"2024-02-23T00:33:04.023633+0000","last_clean":"2024-02-23T00:33:04.023633+0000","last_became_active":"2024-02-23T00:30:48.769602+0000","last_became_peered":"2024-02-23T00:30:48.769602+0000","last_unstale":"2024-02-23T00:33:04.023633+0000","last_undegraded":"2024-02-23T00:33:04.023633+0000","last_fullsized":"2024-02-23T00:33:04.023633+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":465,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036541+0000","last_change":"2024-02-23T00:32:43.323030+0000","last_active":"2024-02-23T00:33:04.036541+0000","last_peered":"2024-02-23T00:33:04.036541+0000","last_clean":"2024-02-23T00:33:04.036541+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:33:04.036541+0000","last_undegraded":"2024-02-23T00:33:04.036541+0000","last_fullsized":"2024-02-23T00:33:04.036541+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:43.322963+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.322963+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024886+0000","last_change":"2024-02-23T00:32:42.410196+0000","last_active":"2024-02-23T00:33:04.024886+0000","last_peered":"2024-02-23T00:33:04.024886+0000","last_clean":"2024-02-23T00:33:04.024886+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:33:04.024886+0000","last_undegraded":"2024-02-23T00:33:04.024886+0000","last_fullsized":"2024-02-23T00:33:04.024886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.410112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.410112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":257,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021276+0000","last_change":"2024-02-23T00:32:47.533001+0000","last_active":"2024-02-23T00:33:04.021276+0000","last_peered":"2024-02-23T00:33:04.021276+0000","last_clean":"2024-02-23T00:33:04.021276+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:33:04.021276+0000","last_undegraded":"2024-02-23T00:33:04.021276+0000","last_fullsized":"2024-02-23T00:33:04.021276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:32:47.532917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.532917+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.034417+0000","last_change":"2024-02-23T00:31:40.847863+0000","last_active":"2024-02-23T00:33:04.034417+0000","last_peered":"2024-02-23T00:33:04.034417+0000","last_clean":"2024-02-23T00:33:04.034417+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:33:04.034417+0000","last_undegraded":"2024-02-23T00:33:04.034417+0000","last_fullsized":"2024-02-23T00:33:04.034417+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.847784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.847784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036916+0000","last_change":"2024-02-23T00:30:46.579484+0000","last_active":"2024-02-23T00:33:04.036916+0000","last_peered":"2024-02-23T00:33:04.036916+0000","last_clean":"2024-02-23T00:33:04.036916+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:33:04.036916+0000","last_undegraded":"2024-02-23T00:33:04.036916+0000","last_fullsized":"2024-02-23T00:33:04.036916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.579417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.579417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":144,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036926+0000","last_change":"2024-02-23T00:30:48.707450+0000","last_active":"2024-02-23T00:33:04.036926+0000","last_peered":"2024-02-23T00:33:04.036926+0000","last_clean":"2024-02-23T00:33:04.036926+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:33:04.036926+0000","last_undegraded":"2024-02-23T00:33:04.036926+0000","last_fullsized":"2024-02-23T00:33:04.036926+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028252+0000","last_change":"2024-02-23T00:32:12.386351+0000","last_active":"2024-02-23T00:33:04.028252+0000","last_peered":"2024-02-23T00:33:04.028252+0000","last_clean":"2024-02-23T00:33:04.028252+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:33:04.028252+0000","last_undegraded":"2024-02-23T00:33:04.028252+0000","last_fullsized":"2024-02-23T00:33:04.028252+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":224,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020651+0000","last_change":"2024-02-23T00:32:46.507450+0000","last_active":"2024-02-23T00:33:04.020651+0000","last_peered":"2024-02-23T00:33:04.020651+0000","last_clean":"2024-02-23T00:33:04.020651+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:33:04.020651+0000","last_undegraded":"2024-02-23T00:33:04.020651+0000","last_fullsized":"2024-02-23T00:33:04.020651+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:32:46.507354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.507354+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031676+0000","last_change":"2024-02-23T00:31:19.760264+0000","last_active":"2024-02-23T00:33:04.031676+0000","last_peered":"2024-02-23T00:33:04.031676+0000","last_clean":"2024-02-23T00:33:04.031676+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:33:04.031676+0000","last_undegraded":"2024-02-23T00:33:04.031676+0000","last_fullsized":"2024-02-23T00:33:04.031676+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:19.760201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.760201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":212,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020309+0000","last_change":"2024-02-23T00:31:38.653143+0000","last_active":"2024-02-23T00:33:04.020309+0000","last_peered":"2024-02-23T00:33:04.020309+0000","last_clean":"2024-02-23T00:33:04.020309+0000","last_became_active":"2024-02-23T00:31:27.053776+0000","last_became_peered":"2024-02-23T00:31:27.053776+0000","last_unstale":"2024-02-23T00:33:04.020309+0000","last_undegraded":"2024-02-23T00:33:04.020309+0000","last_fullsized":"2024-02-23T00:33:04.020309+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.653078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.653078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039005+0000","last_change":"2024-02-23T00:31:21.959719+0000","last_active":"2024-02-23T00:33:04.039005+0000","last_peered":"2024-02-23T00:33:04.039005+0000","last_clean":"2024-02-23T00:33:04.039005+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:33:04.039005+0000","last_undegraded":"2024-02-23T00:33:04.039005+0000","last_fullsized":"2024-02-23T00:33:04.039005+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.959597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.959597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027675+0000","last_change":"2024-02-23T00:30:48.711294+0000","last_active":"2024-02-23T00:33:04.027675+0000","last_peered":"2024-02-23T00:33:04.027675+0000","last_clean":"2024-02-23T00:33:04.027675+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:33:04.027675+0000","last_undegraded":"2024-02-23T00:33:04.027675+0000","last_fullsized":"2024-02-23T00:33:04.027675+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":198,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.033523+0000","last_change":"2024-02-23T00:32:17.590019+0000","last_active":"2024-02-23T00:33:04.033523+0000","last_peered":"2024-02-23T00:33:04.033523+0000","last_clean":"2024-02-23T00:33:04.033523+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:33:04.033523+0000","last_undegraded":"2024-02-23T00:33:04.033523+0000","last_fullsized":"2024-02-23T00:33:04.033523+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.589952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.589952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":223,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023322+0000","last_change":"2024-02-23T00:31:40.030848+0000","last_active":"2024-02-23T00:33:04.023322+0000","last_peered":"2024-02-23T00:33:04.023322+0000","last_clean":"2024-02-23T00:33:04.023322+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:33:04.023322+0000","last_undegraded":"2024-02-23T00:33:04.023322+0000","last_fullsized":"2024-02-23T00:33:04.023322+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:31:40.030776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.030776+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":205,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031755+0000","last_change":"2024-02-23T00:31:21.828852+0000","last_active":"2024-02-23T00:33:04.031755+0000","last_peered":"2024-02-23T00:33:04.031755+0000","last_clean":"2024-02-23T00:33:04.031755+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:33:04.031755+0000","last_undegraded":"2024-02-23T00:33:04.031755+0000","last_fullsized":"2024-02-23T00:33:04.031755+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.828719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.828719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":201,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028654+0000","last_change":"2024-02-23T00:32:46.410759+0000","last_active":"2024-02-23T00:33:04.028654+0000","last_peered":"2024-02-23T00:33:04.028654+0000","last_clean":"2024-02-23T00:33:04.028654+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:33:04.028654+0000","last_undegraded":"2024-02-23T00:33:04.028654+0000","last_fullsized":"2024-02-23T00:33:04.028654+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.410635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.410635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":212,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023727+0000","last_change":"2024-02-23T00:32:46.978493+0000","last_active":"2024-02-23T00:33:04.023727+0000","last_peered":"2024-02-23T00:33:04.023727+0000","last_clean":"2024-02-23T00:33:04.023727+0000","last_became_active":"2024-02-23T00:31:27.144976+0000","last_became_peered":"2024-02-23T00:31:27.144976+0000","last_unstale":"2024-02-23T00:33:04.023727+0000","last_undegraded":"2024-02-23T00:33:04.023727+0000","last_fullsized":"2024-02-23T00:33:04.023727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.978406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.978406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028339+0000","last_change":"2024-02-23T00:33:00.439952+0000","last_active":"2024-02-23T00:33:04.028339+0000","last_peered":"2024-02-23T00:33:04.028339+0000","last_clean":"2024-02-23T00:33:04.028339+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:33:04.028339+0000","last_undegraded":"2024-02-23T00:33:04.028339+0000","last_fullsized":"2024-02-23T00:33:04.028339+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.439872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022924+0000","last_change":"2024-02-23T00:30:48.664253+0000","last_active":"2024-02-23T00:33:04.022924+0000","last_peered":"2024-02-23T00:33:04.022924+0000","last_clean":"2024-02-23T00:33:04.022924+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:33:04.022924+0000","last_undegraded":"2024-02-23T00:33:04.022924+0000","last_fullsized":"2024-02-23T00:33:04.022924+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028678+0000","last_change":"2024-02-23T00:32:13.878058+0000","last_active":"2024-02-23T00:33:04.028678+0000","last_peered":"2024-02-23T00:33:04.028678+0000","last_clean":"2024-02-23T00:33:04.028678+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:33:04.028678+0000","last_undegraded":"2024-02-23T00:33:04.028678+0000","last_fullsized":"2024-02-23T00:33:04.028678+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:13.877940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.877940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":242,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037861+0000","last_change":"2024-02-23T00:32:13.104236+0000","last_active":"2024-02-23T00:33:04.037861+0000","last_peered":"2024-02-23T00:33:04.037861+0000","last_clean":"2024-02-23T00:33:04.037861+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:33:04.037861+0000","last_undegraded":"2024-02-23T00:33:04.037861+0000","last_fullsized":"2024-02-23T00:33:04.037861+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:32:13.104170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.104170+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":203,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.026600+0000","last_change":"2024-02-23T00:31:41.651826+0000","last_active":"2024-02-23T00:33:04.026600+0000","last_peered":"2024-02-23T00:33:04.026600+0000","last_clean":"2024-02-23T00:33:04.026600+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:33:04.026600+0000","last_undegraded":"2024-02-23T00:33:04.026600+0000","last_fullsized":"2024-02-23T00:33:04.026600+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:41.651760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.651760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":458,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039371+0000","last_change":"2024-02-23T00:32:41.360750+0000","last_active":"2024-02-23T00:33:04.039371+0000","last_peered":"2024-02-23T00:33:04.039371+0000","last_clean":"2024-02-23T00:33:04.039371+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:33:04.039371+0000","last_undegraded":"2024-02-23T00:33:04.039371+0000","last_fullsized":"2024-02-23T00:33:04.039371+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:41.360646+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.360646+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.034330+0000","last_change":"2024-02-23T00:32:14.650905+0000","last_active":"2024-02-23T00:33:04.034330+0000","last_peered":"2024-02-23T00:33:04.034330+0000","last_clean":"2024-02-23T00:33:04.034330+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:33:04.034330+0000","last_undegraded":"2024-02-23T00:33:04.034330+0000","last_fullsized":"2024-02-23T00:33:04.034330+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.650827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.650827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":150,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027978+0000","last_change":"2024-02-23T00:32:41.861943+0000","last_active":"2024-02-23T00:33:04.027978+0000","last_peered":"2024-02-23T00:33:04.027978+0000","last_clean":"2024-02-23T00:33:04.027978+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:33:04.027978+0000","last_undegraded":"2024-02-23T00:33:04.027978+0000","last_fullsized":"2024-02-23T00:33:04.027978+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:41.861874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.861874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":212,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021293+0000","last_change":"2024-02-23T00:32:50.705430+0000","last_active":"2024-02-23T00:33:04.021293+0000","last_peered":"2024-02-23T00:33:04.021293+0000","last_clean":"2024-02-23T00:33:04.021293+0000","last_became_active":"2024-02-23T00:31:29.065548+0000","last_became_peered":"2024-02-23T00:31:29.065548+0000","last_unstale":"2024-02-23T00:33:04.021293+0000","last_undegraded":"2024-02-23T00:33:04.021293+0000","last_fullsized":"2024-02-23T00:33:04.021293+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.705318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.705318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":206,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023372+0000","last_change":"2024-02-23T00:32:50.596925+0000","last_active":"2024-02-23T00:33:04.023372+0000","last_peered":"2024-02-23T00:33:04.023372+0000","last_clean":"2024-02-23T00:33:04.023372+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:33:04.023372+0000","last_undegraded":"2024-02-23T00:33:04.023372+0000","last_fullsized":"2024-02-23T00:33:04.023372+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":333,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.030908+0000","last_change":"2024-02-23T00:32:45.366480+0000","last_active":"2024-02-23T00:33:05.030908+0000","last_peered":"2024-02-23T00:33:05.030908+0000","last_clean":"2024-02-23T00:33:05.030908+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:33:05.030908+0000","last_undegraded":"2024-02-23T00:33:05.030908+0000","last_fullsized":"2024-02-23T00:33:05.030908+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:32:45.366416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.366416+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022850+0000","last_change":"2024-02-23T00:30:47.090981+0000","last_active":"2024-02-23T00:33:04.022850+0000","last_peered":"2024-02-23T00:33:04.022850+0000","last_clean":"2024-02-23T00:33:04.022850+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:33:04.022850+0000","last_undegraded":"2024-02-23T00:33:04.022850+0000","last_fullsized":"2024-02-23T00:33:04.022850+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:30:47.090896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.090896+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.029884+0000","last_change":"2024-02-23T00:30:22.860665+0000","last_active":"2024-02-23T00:33:04.029884+0000","last_peered":"2024-02-23T00:33:04.029884+0000","last_clean":"2024-02-23T00:33:04.029884+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:33:04.029884+0000","last_undegraded":"2024-02-23T00:33:04.029884+0000","last_fullsized":"2024-02-23T00:33:04.029884+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:22.860565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.860565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":150,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.029330+0000","last_change":"2024-02-23T00:32:44.885268+0000","last_active":"2024-02-23T00:33:04.029330+0000","last_peered":"2024-02-23T00:33:04.029330+0000","last_clean":"2024-02-23T00:33:04.029330+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:33:04.029330+0000","last_undegraded":"2024-02-23T00:33:04.029330+0000","last_fullsized":"2024-02-23T00:33:04.029330+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.885205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.885205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":111,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025441+0000","last_change":"2024-02-23T00:32:36.447499+0000","last_active":"2024-02-23T00:33:04.025441+0000","last_peered":"2024-02-23T00:33:04.025441+0000","last_clean":"2024-02-23T00:33:04.025441+0000","last_became_active":"2024-02-23T00:31:29.059516+0000","last_became_peered":"2024-02-23T00:31:29.059516+0000","last_unstale":"2024-02-23T00:33:04.025441+0000","last_undegraded":"2024-02-23T00:33:04.025441+0000","last_fullsized":"2024-02-23T00:33:04.025441+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.447439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.447439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":280,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.025787+0000","last_change":"2024-02-23T00:32:18.606981+0000","last_active":"2024-02-23T00:33:05.025787+0000","last_peered":"2024-02-23T00:33:05.025787+0000","last_clean":"2024-02-23T00:33:05.025787+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:33:05.025787+0000","last_undegraded":"2024-02-23T00:33:05.025787+0000","last_fullsized":"2024-02-23T00:33:05.025787+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:18.606911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:18.606911+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":225,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037789+0000","last_change":"2024-02-23T00:30:31.441530+0000","last_active":"2024-02-23T00:33:04.037789+0000","last_peered":"2024-02-23T00:33:04.037789+0000","last_clean":"2024-02-23T00:33:04.037789+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:33:04.037789+0000","last_undegraded":"2024-02-23T00:33:04.037789+0000","last_fullsized":"2024-02-23T00:33:04.037789+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:30:31.441418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.441418+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036679+0000","last_change":"2024-02-23T00:30:44.543241+0000","last_active":"2024-02-23T00:33:04.036679+0000","last_peered":"2024-02-23T00:33:04.036679+0000","last_clean":"2024-02-23T00:33:04.036679+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:33:04.036679+0000","last_undegraded":"2024-02-23T00:33:04.036679+0000","last_fullsized":"2024-02-23T00:33:04.036679+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:44.543161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:30:44.543161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020758+0000","last_change":"2024-02-23T00:32:08.596816+0000","last_active":"2024-02-23T00:33:04.020758+0000","last_peered":"2024-02-23T00:33:04.020758+0000","last_clean":"2024-02-23T00:33:04.020758+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:33:04.020758+0000","last_undegraded":"2024-02-23T00:33:04.020758+0000","last_fullsized":"2024-02-23T00:33:04.020758+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:08.596734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:08.596734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020423+0000","last_change":"2024-02-23T00:30:48.616685+0000","last_active":"2024-02-23T00:33:04.020423+0000","last_peered":"2024-02-23T00:33:04.020423+0000","last_clean":"2024-02-23T00:33:04.020423+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:33:04.020423+0000","last_undegraded":"2024-02-23T00:33:04.020423+0000","last_fullsized":"2024-02-23T00:33:04.020423+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022584+0000","last_change":"2024-02-23T00:32:42.815796+0000","last_active":"2024-02-23T00:33:04.022584+0000","last_peered":"2024-02-23T00:33:04.022584+0000","last_clean":"2024-02-23T00:33:04.022584+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:33:04.022584+0000","last_undegraded":"2024-02-23T00:33:04.022584+0000","last_fullsized":"2024-02-23T00:33:04.022584+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.815722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.815722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":275,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021101+0000","last_change":"2024-02-23T00:31:41.036909+0000","last_active":"2024-02-23T00:33:04.021101+0000","last_peered":"2024-02-23T00:33:04.021101+0000","last_clean":"2024-02-23T00:33:04.021101+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:33:04.021101+0000","last_undegraded":"2024-02-23T00:33:04.021101+0000","last_fullsized":"2024-02-23T00:33:04.021101+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.036837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.036837+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":211,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023076+0000","last_change":"2024-02-23T00:30:47.142583+0000","last_active":"2024-02-23T00:33:04.023076+0000","last_peered":"2024-02-23T00:33:04.023076+0000","last_clean":"2024-02-23T00:33:04.023076+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:33:04.023076+0000","last_undegraded":"2024-02-23T00:33:04.023076+0000","last_fullsized":"2024-02-23T00:33:04.023076+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:30:47.142508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.142508+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024462+0000","last_change":"2024-02-23T00:30:34.861845+0000","last_active":"2024-02-23T00:33:04.024462+0000","last_peered":"2024-02-23T00:33:04.024462+0000","last_clean":"2024-02-23T00:33:04.024462+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:33:04.024462+0000","last_undegraded":"2024-02-23T00:33:04.024462+0000","last_fullsized":"2024-02-23T00:33:04.024462+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:34.861741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:34.861741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":150,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025577+0000","last_change":"2024-02-23T00:32:59.434164+0000","last_active":"2024-02-23T00:33:04.025577+0000","last_peered":"2024-02-23T00:33:04.025577+0000","last_clean":"2024-02-23T00:33:04.025577+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:33:04.025577+0000","last_undegraded":"2024-02-23T00:33:04.025577+0000","last_fullsized":"2024-02-23T00:33:04.025577+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.434079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.434079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":247,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020750+0000","last_change":"2024-02-23T00:32:44.578143+0000","last_active":"2024-02-23T00:33:04.020750+0000","last_peered":"2024-02-23T00:33:04.020750+0000","last_clean":"2024-02-23T00:33:04.020750+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:33:04.020750+0000","last_undegraded":"2024-02-23T00:33:04.020750+0000","last_fullsized":"2024-02-23T00:33:04.020750+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:32:44.578068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.578068+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":203,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022529+0000","last_change":"2024-02-23T00:32:11.912580+0000","last_active":"2024-02-23T00:33:04.022529+0000","last_peered":"2024-02-23T00:33:04.022529+0000","last_clean":"2024-02-23T00:33:04.022529+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:33:04.022529+0000","last_undegraded":"2024-02-23T00:33:04.022529+0000","last_fullsized":"2024-02-23T00:33:04.022529+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:11.912504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.912504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030942+0000","last_change":"2024-02-23T00:30:36.894317+0000","last_active":"2024-02-23T00:33:04.030942+0000","last_peered":"2024-02-23T00:33:04.030942+0000","last_clean":"2024-02-23T00:33:04.030942+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:33:04.030942+0000","last_undegraded":"2024-02-23T00:33:04.030942+0000","last_fullsized":"2024-02-23T00:33:04.030942+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:36.894220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:36.894220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027009+0000","last_change":"2024-02-23T00:32:16.828560+0000","last_active":"2024-02-23T00:33:04.027009+0000","last_peered":"2024-02-23T00:33:04.027009+0000","last_clean":"2024-02-23T00:33:04.027009+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:33:04.027009+0000","last_undegraded":"2024-02-23T00:33:04.027009+0000","last_fullsized":"2024-02-23T00:33:04.027009+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_clean_scrub_stamp":"2024-02-23T00:32:16.828490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"148'9","reported_seq":170,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037286+0000","last_change":"2024-02-23T00:32:41.020450+0000","last_active":"2024-02-23T00:33:04.037286+0000","last_peered":"2024-02-23T00:33:04.037286+0000","last_clean":"2024-02-23T00:33:04.037286+0000","last_became_active":"2024-02-23T00:32:41.020138+0000","last_became_peered":"2024-02-23T00:32:41.020138+0000","last_unstale":"2024-02-23T00:33:04.037286+0000","last_undegraded":"2024-02-23T00:33:04.037286+0000","last_fullsized":"2024-02-23T00:33:04.037286+0000","mapping_epoch":96,"log_start":"148'9","ondisk_log_start":"148'9","created":96,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032172+0000","last_change":"2024-02-23T00:30:48.691960+0000","last_active":"2024-02-23T00:33:04.032172+0000","last_peered":"2024-02-23T00:33:04.032172+0000","last_clean":"2024-02-23T00:33:04.032172+0000","last_became_active":"2024-02-23T00:30:48.691713+0000","last_became_peered":"2024-02-23T00:30:48.691713+0000","last_unstale":"2024-02-23T00:33:04.032172+0000","last_undegraded":"2024-02-23T00:33:04.032172+0000","last_fullsized":"2024-02-23T00:33:04.032172+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":239,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030766+0000","last_change":"2024-02-23T00:30:24.802266+0000","last_active":"2024-02-23T00:33:04.030766+0000","last_peered":"2024-02-23T00:33:04.030766+0000","last_clean":"2024-02-23T00:33:04.030766+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:33:04.030766+0000","last_undegraded":"2024-02-23T00:33:04.030766+0000","last_fullsized":"2024-02-23T00:33:04.030766+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:30:24.802134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:24.802134+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":203,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027594+0000","last_change":"2024-02-23T00:32:14.243873+0000","last_active":"2024-02-23T00:33:04.027594+0000","last_peered":"2024-02-23T00:33:04.027594+0000","last_clean":"2024-02-23T00:33:04.027594+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:33:04.027594+0000","last_undegraded":"2024-02-23T00:33:04.027594+0000","last_fullsized":"2024-02-23T00:33:04.027594+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.243786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.243786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024302+0000","last_change":"2024-02-23T00:32:07.275896+0000","last_active":"2024-02-23T00:33:04.024302+0000","last_peered":"2024-02-23T00:33:04.024302+0000","last_clean":"2024-02-23T00:33:04.024302+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:33:04.024302+0000","last_undegraded":"2024-02-23T00:33:04.024302+0000","last_fullsized":"2024-02-23T00:33:04.024302+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:07.275816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:07.275816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.033950+0000","last_change":"2024-02-23T00:32:15.651736+0000","last_active":"2024-02-23T00:33:04.033950+0000","last_peered":"2024-02-23T00:33:04.033950+0000","last_clean":"2024-02-23T00:33:04.033950+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:33:04.033950+0000","last_undegraded":"2024-02-23T00:33:04.033950+0000","last_fullsized":"2024-02-23T00:33:04.033950+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:15.651651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:32:15.651651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"148'7","reported_seq":171,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024571+0000","last_change":"2024-02-23T00:32:45.016687+0000","last_active":"2024-02-23T00:33:04.024571+0000","last_peered":"2024-02-23T00:33:04.024571+0000","last_clean":"2024-02-23T00:33:04.024571+0000","last_became_active":"2024-02-23T00:32:45.016438+0000","last_became_peered":"2024-02-23T00:32:45.016438+0000","last_unstale":"2024-02-23T00:33:04.024571+0000","last_undegraded":"2024-02-23T00:33:04.024571+0000","last_fullsized":"2024-02-23T00:33:04.024571+0000","mapping_epoch":96,"log_start":"148'7","ondisk_log_start":"148'7","created":96,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":150,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023583+0000","last_change":"2024-02-23T00:32:55.433102+0000","last_active":"2024-02-23T00:33:04.023583+0000","last_peered":"2024-02-23T00:33:04.023583+0000","last_clean":"2024-02-23T00:33:04.023583+0000","last_became_active":"2024-02-23T00:30:48.703232+0000","last_became_peered":"2024-02-23T00:30:48.703232+0000","last_unstale":"2024-02-23T00:33:04.023583+0000","last_undegraded":"2024-02-23T00:33:04.023583+0000","last_fullsized":"2024-02-23T00:33:04.023583+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:55.433031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:55.433031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":241,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031775+0000","last_change":"2024-02-23T00:30:37.899931+0000","last_active":"2024-02-23T00:33:04.031775+0000","last_peered":"2024-02-23T00:33:04.031775+0000","last_clean":"2024-02-23T00:33:04.031775+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:33:04.031775+0000","last_undegraded":"2024-02-23T00:33:04.031775+0000","last_fullsized":"2024-02-23T00:33:04.031775+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:30:37.899845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.899845+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":201,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024712+0000","last_change":"2024-02-23T00:32:10.319807+0000","last_active":"2024-02-23T00:33:04.024712+0000","last_peered":"2024-02-23T00:33:04.024712+0000","last_clean":"2024-02-23T00:33:04.024712+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:33:04.024712+0000","last_undegraded":"2024-02-23T00:33:04.024712+0000","last_fullsized":"2024-02-23T00:33:04.024712+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.319744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.319744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":276,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.029500+0000","last_change":"2024-02-23T00:32:56.376450+0000","last_active":"2024-02-23T00:33:05.029500+0000","last_peered":"2024-02-23T00:33:05.029500+0000","last_clean":"2024-02-23T00:33:05.029500+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:33:05.029500+0000","last_undegraded":"2024-02-23T00:33:05.029500+0000","last_fullsized":"2024-02-23T00:33:05.029500+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020566+0000","last_change":"2024-02-23T00:32:07.573617+0000","last_active":"2024-02-23T00:33:04.020566+0000","last_peered":"2024-02-23T00:33:04.020566+0000","last_clean":"2024-02-23T00:33:04.020566+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:33:04.020566+0000","last_undegraded":"2024-02-23T00:33:04.020566+0000","last_fullsized":"2024-02-23T00:33:04.020566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:07.573555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:37.889336+0000","last_clean_scrub_stamp":"2024-02-23T00:32:07.573555+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.3","version":"148'14","reported_seq":189,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.054445+0000","last_change":"2024-02-23T00:33:05.054445+0000","last_active":"2024-02-23T00:33:05.054445+0000","last_peered":"2024-02-23T00:33:05.054445+0000","last_clean":"2024-02-23T00:33:05.054445+0000","last_became_active":"2024-02-23T00:33:05.054103+0000","last_became_peered":"2024-02-23T00:33:05.054103+0000","last_unstale":"2024-02-23T00:33:05.054445+0000","last_undegraded":"2024-02-23T00:33:05.054445+0000","last_fullsized":"2024-02-23T00:33:05.054445+0000","mapping_epoch":96,"log_start":"148'14","ondisk_log_start":"148'14","created":96,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":150,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024887+0000","last_change":"2024-02-23T00:32:45.409616+0000","last_active":"2024-02-23T00:33:04.024887+0000","last_peered":"2024-02-23T00:33:04.024887+0000","last_clean":"2024-02-23T00:33:04.024887+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:33:04.024887+0000","last_undegraded":"2024-02-23T00:33:04.024887+0000","last_fullsized":"2024-02-23T00:33:04.024887+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.409548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.409548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"144'20","reported_seq":230,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025903+0000","last_change":"2024-02-23T00:31:35.423292+0000","last_active":"2024-02-23T00:33:04.025903+0000","last_peered":"2024-02-23T00:33:04.025903+0000","last_clean":"2024-02-23T00:33:04.025903+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:33:04.025903+0000","last_undegraded":"2024-02-23T00:33:04.025903+0000","last_fullsized":"2024-02-23T00:33:04.025903+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.423227+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021351+0000","last_change":"2024-02-23T00:32:48.536354+0000","last_active":"2024-02-23T00:33:04.021351+0000","last_peered":"2024-02-23T00:33:04.021351+0000","last_clean":"2024-02-23T00:33:04.021351+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:33:04.021351+0000","last_undegraded":"2024-02-23T00:33:04.021351+0000","last_fullsized":"2024-02-23T00:33:04.021351+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.536283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.536283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'9","reported_seq":278,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.039720+0000","last_change":"2024-02-23T00:32:43.990958+0000","last_active":"2024-02-23T00:33:05.039720+0000","last_peered":"2024-02-23T00:33:05.039720+0000","last_clean":"2024-02-23T00:33:05.039720+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:33:05.039720+0000","last_undegraded":"2024-02-23T00:33:05.039720+0000","last_fullsized":"2024-02-23T00:33:05.039720+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:32:43.990886+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.990886+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023311+0000","last_change":"2024-02-23T00:32:56.678465+0000","last_active":"2024-02-23T00:33:04.023311+0000","last_peered":"2024-02-23T00:33:04.023311+0000","last_clean":"2024-02-23T00:33:04.023311+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:33:04.023311+0000","last_undegraded":"2024-02-23T00:33:04.023311+0000","last_fullsized":"2024-02-23T00:33:04.023311+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.678359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.678359+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":169,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032621+0000","last_change":"2024-02-23T00:32:42.691730+0000","last_active":"2024-02-23T00:33:04.032621+0000","last_peered":"2024-02-23T00:33:04.032621+0000","last_clean":"2024-02-23T00:33:04.032621+0000","last_became_active":"2024-02-23T00:32:37.017015+0000","last_became_peered":"2024-02-23T00:32:37.017015+0000","last_unstale":"2024-02-23T00:33:04.032621+0000","last_undegraded":"2024-02-23T00:33:04.032621+0000","last_fullsized":"2024-02-23T00:33:04.032621+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.691630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.691630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031065+0000","last_change":"2024-02-23T00:30:48.717524+0000","last_active":"2024-02-23T00:33:04.031065+0000","last_peered":"2024-02-23T00:33:04.031065+0000","last_clean":"2024-02-23T00:33:04.031065+0000","last_became_active":"2024-02-23T00:30:48.716104+0000","last_became_peered":"2024-02-23T00:30:48.716104+0000","last_unstale":"2024-02-23T00:33:04.031065+0000","last_undegraded":"2024-02-23T00:33:04.031065+0000","last_fullsized":"2024-02-23T00:33:04.031065+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":216,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037682+0000","last_change":"2024-02-23T00:32:50.448701+0000","last_active":"2024-02-23T00:33:04.037682+0000","last_peered":"2024-02-23T00:33:04.037682+0000","last_clean":"2024-02-23T00:33:04.037682+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:33:04.037682+0000","last_undegraded":"2024-02-23T00:33:04.037682+0000","last_fullsized":"2024-02-23T00:33:04.037682+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:32:50.448595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.448595+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":203,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024820+0000","last_change":"2024-02-23T00:32:12.234058+0000","last_active":"2024-02-23T00:33:04.024820+0000","last_peered":"2024-02-23T00:33:04.024820+0000","last_clean":"2024-02-23T00:33:04.024820+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:33:04.024820+0000","last_undegraded":"2024-02-23T00:33:04.024820+0000","last_fullsized":"2024-02-23T00:33:04.024820+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.233960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.233960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024391+0000","last_change":"2024-02-23T00:32:09.978443+0000","last_active":"2024-02-23T00:33:04.024391+0000","last_peered":"2024-02-23T00:33:04.024391+0000","last_clean":"2024-02-23T00:33:04.024391+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:33:04.024391+0000","last_undegraded":"2024-02-23T00:33:04.024391+0000","last_fullsized":"2024-02-23T00:33:04.024391+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.978377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.978377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020632+0000","last_change":"2024-02-23T00:32:53.602946+0000","last_active":"2024-02-23T00:33:04.020632+0000","last_peered":"2024-02-23T00:33:04.020632+0000","last_clean":"2024-02-23T00:33:04.020632+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:33:04.020632+0000","last_undegraded":"2024-02-23T00:33:04.020632+0000","last_fullsized":"2024-02-23T00:33:04.020632+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.602880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.602880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.5","version":"148'9","reported_seq":182,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025893+0000","last_change":"2024-02-23T00:32:57.090735+0000","last_active":"2024-02-23T00:33:04.025893+0000","last_peered":"2024-02-23T00:33:04.025893+0000","last_clean":"2024-02-23T00:33:04.025893+0000","last_became_active":"2024-02-23T00:32:57.090327+0000","last_became_peered":"2024-02-23T00:32:57.090327+0000","last_unstale":"2024-02-23T00:33:04.025893+0000","last_undegraded":"2024-02-23T00:33:04.025893+0000","last_fullsized":"2024-02-23T00:33:04.025893+0000","mapping_epoch":96,"log_start":"148'9","ondisk_log_start":"148'9","created":96,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020136+0000","last_change":"2024-02-23T00:30:48.668969+0000","last_active":"2024-02-23T00:33:04.020136+0000","last_peered":"2024-02-23T00:33:04.020136+0000","last_clean":"2024-02-23T00:33:04.020136+0000","last_became_active":"2024-02-23T00:30:48.666975+0000","last_became_peered":"2024-02-23T00:30:48.666975+0000","last_unstale":"2024-02-23T00:33:04.020136+0000","last_undegraded":"2024-02-23T00:33:04.020136+0000","last_fullsized":"2024-02-23T00:33:04.020136+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":236,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021616+0000","last_change":"2024-02-23T00:30:41.077799+0000","last_active":"2024-02-23T00:33:04.021616+0000","last_peered":"2024-02-23T00:33:04.021616+0000","last_clean":"2024-02-23T00:33:04.021616+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:33:04.021616+0000","last_undegraded":"2024-02-23T00:33:04.021616+0000","last_fullsized":"2024-02-23T00:33:04.021616+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:30:41.077681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.077681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039636+0000","last_change":"2024-02-23T00:31:32.874749+0000","last_active":"2024-02-23T00:33:04.039636+0000","last_peered":"2024-02-23T00:33:04.039636+0000","last_clean":"2024-02-23T00:33:04.039636+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:33:04.039636+0000","last_undegraded":"2024-02-23T00:33:04.039636+0000","last_fullsized":"2024-02-23T00:33:04.039636+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:32.874678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:31:32.874678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":193,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037769+0000","last_change":"2024-02-23T00:30:30.486006+0000","last_active":"2024-02-23T00:33:04.037769+0000","last_peered":"2024-02-23T00:33:04.037769+0000","last_clean":"2024-02-23T00:33:04.037769+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:33:04.037769+0000","last_undegraded":"2024-02-23T00:33:04.037769+0000","last_fullsized":"2024-02-23T00:33:04.037769+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:30.485900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.485900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023893+0000","last_change":"2024-02-23T00:30:47.843046+0000","last_active":"2024-02-23T00:33:04.023893+0000","last_peered":"2024-02-23T00:33:04.023893+0000","last_clean":"2024-02-23T00:33:04.023893+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:33:04.023893+0000","last_undegraded":"2024-02-23T00:33:04.023893+0000","last_fullsized":"2024-02-23T00:33:04.023893+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.842931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.4","version":"98'2","reported_seq":181,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039428+0000","last_change":"2024-02-23T00:33:01.049678+0000","last_active":"2024-02-23T00:33:04.039428+0000","last_peered":"2024-02-23T00:33:04.039428+0000","last_clean":"2024-02-23T00:33:04.039428+0000","last_became_active":"2024-02-23T00:33:01.049360+0000","last_became_peered":"2024-02-23T00:33:01.049360+0000","last_unstale":"2024-02-23T00:33:04.039428+0000","last_undegraded":"2024-02-23T00:33:04.039428+0000","last_fullsized":"2024-02-23T00:33:04.039428+0000","mapping_epoch":96,"log_start":"98'2","ondisk_log_start":"98'2","created":96,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":158,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027841+0000","last_change":"2024-02-23T00:30:48.705018+0000","last_active":"2024-02-23T00:33:04.027841+0000","last_peered":"2024-02-23T00:33:04.027841+0000","last_clean":"2024-02-23T00:33:04.027841+0000","last_became_active":"2024-02-23T00:30:48.703281+0000","last_became_peered":"2024-02-23T00:30:48.703281+0000","last_unstale":"2024-02-23T00:33:04.027841+0000","last_undegraded":"2024-02-23T00:33:04.027841+0000","last_fullsized":"2024-02-23T00:33:04.027841+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024207+0000","last_change":"2024-02-23T00:32:11.989425+0000","last_active":"2024-02-23T00:33:04.024207+0000","last_peered":"2024-02-23T00:33:04.024207+0000","last_clean":"2024-02-23T00:33:04.024207+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:33:04.024207+0000","last_undegraded":"2024-02-23T00:33:04.024207+0000","last_fullsized":"2024-02-23T00:33:04.024207+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:32:11.989333+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.989333+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":201,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024452+0000","last_change":"2024-02-23T00:32:40.343941+0000","last_active":"2024-02-23T00:33:04.024452+0000","last_peered":"2024-02-23T00:33:04.024452+0000","last_clean":"2024-02-23T00:33:04.024452+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:33:04.024452+0000","last_undegraded":"2024-02-23T00:33:04.024452+0000","last_fullsized":"2024-02-23T00:33:04.024452+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.343863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.343863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027581+0000","last_change":"2024-02-23T00:32:47.484343+0000","last_active":"2024-02-23T00:33:04.027581+0000","last_peered":"2024-02-23T00:33:04.027581+0000","last_clean":"2024-02-23T00:33:04.027581+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:33:04.027581+0000","last_undegraded":"2024-02-23T00:33:04.027581+0000","last_fullsized":"2024-02-23T00:33:04.027581+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:47.484285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.484285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032950+0000","last_change":"2024-02-23T00:32:12.721784+0000","last_active":"2024-02-23T00:33:04.032950+0000","last_peered":"2024-02-23T00:33:04.032950+0000","last_clean":"2024-02-23T00:33:04.032950+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:33:04.032950+0000","last_undegraded":"2024-02-23T00:33:04.032950+0000","last_fullsized":"2024-02-23T00:33:04.032950+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.721719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.721719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"148'7","reported_seq":187,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028869+0000","last_change":"2024-02-23T00:32:49.030723+0000","last_active":"2024-02-23T00:33:04.028869+0000","last_peered":"2024-02-23T00:33:04.028869+0000","last_clean":"2024-02-23T00:33:04.028869+0000","last_became_active":"2024-02-23T00:32:49.030411+0000","last_became_peered":"2024-02-23T00:32:49.030411+0000","last_unstale":"2024-02-23T00:33:04.028869+0000","last_undegraded":"2024-02-23T00:33:04.028869+0000","last_fullsized":"2024-02-23T00:33:04.028869+0000","mapping_epoch":96,"log_start":"148'7","ondisk_log_start":"148'7","created":96,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7","last_scrub_stamp":"2024-02-23T00:32:43.845548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.845548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":144,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037843+0000","last_change":"2024-02-23T00:30:48.696251+0000","last_active":"2024-02-23T00:33:04.037843+0000","last_peered":"2024-02-23T00:33:04.037843+0000","last_clean":"2024-02-23T00:33:04.037843+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:33:04.037843+0000","last_undegraded":"2024-02-23T00:33:04.037843+0000","last_fullsized":"2024-02-23T00:33:04.037843+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":237,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025682+0000","last_change":"2024-02-23T00:31:27.176319+0000","last_active":"2024-02-23T00:33:04.025682+0000","last_peered":"2024-02-23T00:33:04.025682+0000","last_clean":"2024-02-23T00:33:04.025682+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:33:04.025682+0000","last_undegraded":"2024-02-23T00:33:04.025682+0000","last_fullsized":"2024-02-23T00:33:04.025682+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:31:27.176199+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:31:27.176199+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.026898+0000","last_change":"2024-02-23T00:32:44.494080+0000","last_active":"2024-02-23T00:33:04.026898+0000","last_peered":"2024-02-23T00:33:04.026898+0000","last_clean":"2024-02-23T00:33:04.026898+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:33:04.026898+0000","last_undegraded":"2024-02-23T00:33:04.026898+0000","last_fullsized":"2024-02-23T00:33:04.026898+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.494004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.494004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":198,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032014+0000","last_change":"2024-02-23T00:32:46.655443+0000","last_active":"2024-02-23T00:33:04.032014+0000","last_peered":"2024-02-23T00:33:04.032014+0000","last_clean":"2024-02-23T00:33:04.032014+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:33:04.032014+0000","last_undegraded":"2024-02-23T00:33:04.032014+0000","last_fullsized":"2024-02-23T00:33:04.032014+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.655378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.655378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025198+0000","last_change":"2024-02-23T00:30:31.829814+0000","last_active":"2024-02-23T00:33:04.025198+0000","last_peered":"2024-02-23T00:33:04.025198+0000","last_clean":"2024-02-23T00:33:04.025198+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:33:04.025198+0000","last_undegraded":"2024-02-23T00:33:04.025198+0000","last_fullsized":"2024-02-23T00:33:04.025198+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.829697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.829697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.6","version":"148'11","reported_seq":193,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024571+0000","last_change":"2024-02-23T00:32:53.032240+0000","last_active":"2024-02-23T00:33:04.024571+0000","last_peered":"2024-02-23T00:33:04.024571+0000","last_clean":"2024-02-23T00:33:04.024571+0000","last_became_active":"2024-02-23T00:32:53.031873+0000","last_became_peered":"2024-02-23T00:32:53.031873+0000","last_unstale":"2024-02-23T00:33:04.024571+0000","last_undegraded":"2024-02-23T00:33:04.024571+0000","last_fullsized":"2024-02-23T00:33:04.024571+0000","mapping_epoch":96,"log_start":"148'11","ondisk_log_start":"148'11","created":96,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021934+0000","last_change":"2024-02-23T00:30:48.723611+0000","last_active":"2024-02-23T00:33:04.021934+0000","last_peered":"2024-02-23T00:33:04.021934+0000","last_clean":"2024-02-23T00:33:04.021934+0000","last_became_active":"2024-02-23T00:30:48.722602+0000","last_became_peered":"2024-02-23T00:30:48.722602+0000","last_unstale":"2024-02-23T00:33:04.021934+0000","last_undegraded":"2024-02-23T00:33:04.021934+0000","last_fullsized":"2024-02-23T00:33:04.021934+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"92'22","reported_seq":237,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020953+0000","last_change":"2024-02-23T00:32:43.560670+0000","last_active":"2024-02-23T00:33:04.020953+0000","last_peered":"2024-02-23T00:33:04.020953+0000","last_clean":"2024-02-23T00:33:04.020953+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:33:04.020953+0000","last_undegraded":"2024-02-23T00:33:04.020953+0000","last_fullsized":"2024-02-23T00:33:04.020953+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:32:43.560587+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.560587+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021959+0000","last_change":"2024-02-23T00:31:20.333610+0000","last_active":"2024-02-23T00:33:04.021959+0000","last_peered":"2024-02-23T00:33:04.021959+0000","last_clean":"2024-02-23T00:33:04.021959+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:33:04.021959+0000","last_undegraded":"2024-02-23T00:33:04.021959+0000","last_fullsized":"2024-02-23T00:33:04.021959+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:20.333547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:20.333547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":280,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-23T00:33:05.020508+0000","last_change":"2024-02-23T00:31:19.971484+0000","last_active":"2024-02-23T00:33:05.020508+0000","last_peered":"2024-02-23T00:33:05.020508+0000","last_clean":"2024-02-23T00:33:05.020508+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:33:05.020508+0000","last_undegraded":"2024-02-23T00:33:05.020508+0000","last_fullsized":"2024-02-23T00:33:05.020508+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:31:19.971419+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.971419+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":198,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.033728+0000","last_change":"2024-02-23T00:32:50.706776+0000","last_active":"2024-02-23T00:33:04.033728+0000","last_peered":"2024-02-23T00:33:04.033728+0000","last_clean":"2024-02-23T00:33:04.033728+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:33:04.033728+0000","last_undegraded":"2024-02-23T00:33:04.033728+0000","last_fullsized":"2024-02-23T00:33:04.033728+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:32:50.706693+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.706693+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":169,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030846+0000","last_change":"2024-02-23T00:32:59.681117+0000","last_active":"2024-02-23T00:33:04.030846+0000","last_peered":"2024-02-23T00:33:04.030846+0000","last_clean":"2024-02-23T00:33:04.030846+0000","last_became_active":"2024-02-23T00:32:09.024182+0000","last_became_peered":"2024-02-23T00:32:09.024182+0000","last_unstale":"2024-02-23T00:33:04.030846+0000","last_undegraded":"2024-02-23T00:33:04.030846+0000","last_fullsized":"2024-02-23T00:33:04.030846+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.681034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:59.681034+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.681034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.035857+0000","last_change":"2024-02-23T00:30:48.748188+0000","last_active":"2024-02-23T00:33:04.035857+0000","last_peered":"2024-02-23T00:33:04.035857+0000","last_clean":"2024-02-23T00:33:04.035857+0000","last_became_active":"2024-02-23T00:30:48.745982+0000","last_became_peered":"2024-02-23T00:30:48.745982+0000","last_unstale":"2024-02-23T00:33:04.035857+0000","last_undegraded":"2024-02-23T00:33:04.035857+0000","last_fullsized":"2024-02-23T00:33:04.035857+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":235,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027166+0000","last_change":"2024-02-23T00:32:59.459127+0000","last_active":"2024-02-23T00:33:04.027166+0000","last_peered":"2024-02-23T00:33:04.027166+0000","last_clean":"2024-02-23T00:33:04.027166+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:33:04.027166+0000","last_undegraded":"2024-02-23T00:33:04.027166+0000","last_fullsized":"2024-02-23T00:33:04.027166+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:32:59.459053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.459053+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":203,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025465+0000","last_change":"2024-02-23T00:32:37.932120+0000","last_active":"2024-02-23T00:33:04.025465+0000","last_peered":"2024-02-23T00:33:04.025465+0000","last_clean":"2024-02-23T00:33:04.025465+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:33:04.025465+0000","last_undegraded":"2024-02-23T00:33:04.025465+0000","last_fullsized":"2024-02-23T00:33:04.025465+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.932055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:49.056184+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.932055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021034+0000","last_change":"2024-02-23T00:32:35.624529+0000","last_active":"2024-02-23T00:33:04.021034+0000","last_peered":"2024-02-23T00:33:04.021034+0000","last_clean":"2024-02-23T00:33:04.021034+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:33:04.021034+0000","last_undegraded":"2024-02-23T00:33:04.021034+0000","last_fullsized":"2024-02-23T00:33:04.021034+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.624461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.624461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039497+0000","last_change":"2024-02-23T00:32:10.145767+0000","last_active":"2024-02-23T00:33:04.039497+0000","last_peered":"2024-02-23T00:33:04.039497+0000","last_clean":"2024-02-23T00:33:04.039497+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:33:04.039497+0000","last_undegraded":"2024-02-23T00:33:04.039497+0000","last_fullsized":"2024-02-23T00:33:04.039497+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.145685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.145685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.8","version":"98'2","reported_seq":165,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.039454+0000","last_change":"2024-02-23T00:32:13.036407+0000","last_active":"2024-02-23T00:33:04.039454+0000","last_peered":"2024-02-23T00:33:04.039454+0000","last_clean":"2024-02-23T00:33:04.039454+0000","last_became_active":"2024-02-23T00:32:13.036071+0000","last_became_peered":"2024-02-23T00:32:13.036071+0000","last_unstale":"2024-02-23T00:33:04.039454+0000","last_undegraded":"2024-02-23T00:33:04.039454+0000","last_fullsized":"2024-02-23T00:33:04.039454+0000","mapping_epoch":96,"log_start":"98'2","ondisk_log_start":"98'2","created":96,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":150,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.034986+0000","last_change":"2024-02-23T00:32:58.681718+0000","last_active":"2024-02-23T00:33:04.034986+0000","last_peered":"2024-02-23T00:33:04.034986+0000","last_clean":"2024-02-23T00:33:04.034986+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:33:04.034986+0000","last_undegraded":"2024-02-23T00:33:04.034986+0000","last_fullsized":"2024-02-23T00:33:04.034986+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.681626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.681626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":226,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027207+0000","last_change":"2024-02-23T00:31:31.806791+0000","last_active":"2024-02-23T00:33:04.027207+0000","last_peered":"2024-02-23T00:33:04.027207+0000","last_clean":"2024-02-23T00:33:04.027207+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:33:04.027207+0000","last_undegraded":"2024-02-23T00:33:04.027207+0000","last_fullsized":"2024-02-23T00:33:04.027207+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:31:31.806687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.806687+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":203,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024967+0000","last_change":"2024-02-23T00:30:40.183533+0000","last_active":"2024-02-23T00:33:04.024967+0000","last_peered":"2024-02-23T00:33:04.024967+0000","last_clean":"2024-02-23T00:33:04.024967+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:33:04.024967+0000","last_undegraded":"2024-02-23T00:33:04.024967+0000","last_fullsized":"2024-02-23T00:33:04.024967+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.183453+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.040707+0000","last_change":"2024-02-23T00:31:38.887671+0000","last_active":"2024-02-23T00:33:04.040707+0000","last_peered":"2024-02-23T00:33:04.040707+0000","last_clean":"2024-02-23T00:33:04.040707+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:33:04.040707+0000","last_undegraded":"2024-02-23T00:33:04.040707+0000","last_fullsized":"2024-02-23T00:33:04.040707+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.887597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024060+0000","last_change":"2024-02-23T00:32:12.983101+0000","last_active":"2024-02-23T00:33:04.024060+0000","last_peered":"2024-02-23T00:33:04.024060+0000","last_clean":"2024-02-23T00:33:04.024060+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:33:04.024060+0000","last_undegraded":"2024-02-23T00:33:04.024060+0000","last_fullsized":"2024-02-23T00:33:04.024060+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.983023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.983023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024563+0000","last_change":"2024-02-23T00:30:48.711374+0000","last_active":"2024-02-23T00:33:04.024563+0000","last_peered":"2024-02-23T00:33:04.024563+0000","last_clean":"2024-02-23T00:33:04.024563+0000","last_became_active":"2024-02-23T00:30:48.710049+0000","last_became_peered":"2024-02-23T00:30:48.710049+0000","last_unstale":"2024-02-23T00:33:04.024563+0000","last_undegraded":"2024-02-23T00:33:04.024563+0000","last_fullsized":"2024-02-23T00:33:04.024563+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":240,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.029865+0000","last_change":"2024-02-23T00:30:29.844070+0000","last_active":"2024-02-23T00:33:04.029865+0000","last_peered":"2024-02-23T00:33:04.029865+0000","last_clean":"2024-02-23T00:33:04.029865+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:33:04.029865+0000","last_undegraded":"2024-02-23T00:33:04.029865+0000","last_fullsized":"2024-02-23T00:33:04.029865+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:30:29.843954+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.843954+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":198,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024361+0000","last_change":"2024-02-23T00:30:32.878666+0000","last_active":"2024-02-23T00:33:04.024361+0000","last_peered":"2024-02-23T00:33:04.024361+0000","last_clean":"2024-02-23T00:33:04.024361+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:33:04.024361+0000","last_undegraded":"2024-02-23T00:33:04.024361+0000","last_fullsized":"2024-02-23T00:33:04.024361+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:32.878569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:32.878569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027440+0000","last_change":"2024-02-23T00:31:45.961466+0000","last_active":"2024-02-23T00:33:04.027440+0000","last_peered":"2024-02-23T00:33:04.027440+0000","last_clean":"2024-02-23T00:33:04.027440+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:33:04.027440+0000","last_undegraded":"2024-02-23T00:33:04.027440+0000","last_fullsized":"2024-02-23T00:33:04.027440+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:45.961386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:45.961386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.030686+0000","last_change":"2024-02-23T00:32:46.021732+0000","last_active":"2024-02-23T00:33:04.030686+0000","last_peered":"2024-02-23T00:33:04.030686+0000","last_clean":"2024-02-23T00:33:04.030686+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:33:04.030686+0000","last_undegraded":"2024-02-23T00:33:04.030686+0000","last_fullsized":"2024-02-23T00:33:04.030686+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.021662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.a","version":"148'14","reported_seq":183,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.023550+0000","last_change":"2024-02-23T00:32:51.819469+0000","last_active":"2024-02-23T00:33:04.023550+0000","last_peered":"2024-02-23T00:33:04.023550+0000","last_clean":"2024-02-23T00:33:04.023550+0000","last_became_active":"2024-02-23T00:32:05.034399+0000","last_became_peered":"2024-02-23T00:32:05.034399+0000","last_unstale":"2024-02-23T00:33:04.023550+0000","last_undegraded":"2024-02-23T00:33:04.023550+0000","last_fullsized":"2024-02-23T00:33:04.023550+0000","mapping_epoch":96,"log_start":"148'14","ondisk_log_start":"148'14","created":96,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'14","last_scrub_stamp":"2024-02-23T00:32:51.819400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:32:51.819400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.027731+0000","last_change":"2024-02-23T00:30:48.679036+0000","last_active":"2024-02-23T00:33:04.027731+0000","last_peered":"2024-02-23T00:33:04.027731+0000","last_clean":"2024-02-23T00:33:04.027731+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:33:04.027731+0000","last_undegraded":"2024-02-23T00:33:04.027731+0000","last_fullsized":"2024-02-23T00:33:04.027731+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":247,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028867+0000","last_change":"2024-02-23T00:32:44.870419+0000","last_active":"2024-02-23T00:33:04.028867+0000","last_peered":"2024-02-23T00:33:04.028867+0000","last_clean":"2024-02-23T00:33:04.028867+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:33:04.028867+0000","last_undegraded":"2024-02-23T00:33:04.028867+0000","last_fullsized":"2024-02-23T00:33:04.028867+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:32:44.870335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.870335+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.033281+0000","last_change":"2024-02-23T00:32:45.663968+0000","last_active":"2024-02-23T00:33:04.033281+0000","last_peered":"2024-02-23T00:33:04.033281+0000","last_clean":"2024-02-23T00:33:04.033281+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:33:04.033281+0000","last_undegraded":"2024-02-23T00:33:04.033281+0000","last_fullsized":"2024-02-23T00:33:04.033281+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.663905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.663905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028820+0000","last_change":"2024-02-23T00:30:41.121986+0000","last_active":"2024-02-23T00:33:04.028820+0000","last_peered":"2024-02-23T00:33:04.028820+0000","last_clean":"2024-02-23T00:33:04.028820+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:33:04.028820+0000","last_undegraded":"2024-02-23T00:33:04.028820+0000","last_fullsized":"2024-02-23T00:33:04.028820+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:41.121916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.121916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021166+0000","last_change":"2024-02-23T00:32:39.644590+0000","last_active":"2024-02-23T00:33:04.021166+0000","last_peered":"2024-02-23T00:33:04.021166+0000","last_clean":"2024-02-23T00:33:04.021166+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:33:04.021166+0000","last_undegraded":"2024-02-23T00:33:04.021166+0000","last_fullsized":"2024-02-23T00:33:04.021166+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:39.644483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:32:39.644483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":150,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.031438+0000","last_change":"2024-02-23T00:32:35.725253+0000","last_active":"2024-02-23T00:33:04.031438+0000","last_peered":"2024-02-23T00:33:04.031438+0000","last_clean":"2024-02-23T00:33:04.031438+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:33:04.031438+0000","last_undegraded":"2024-02-23T00:33:04.031438+0000","last_fullsized":"2024-02-23T00:33:04.031438+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.725182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.725182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":229,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021789+0000","last_change":"2024-02-23T00:32:58.673712+0000","last_active":"2024-02-23T00:33:04.021789+0000","last_peered":"2024-02-23T00:33:04.021789+0000","last_clean":"2024-02-23T00:33:04.021789+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:33:04.021789+0000","last_undegraded":"2024-02-23T00:33:04.021789+0000","last_fullsized":"2024-02-23T00:33:04.021789+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:32:58.673628+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.673628+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.034085+0000","last_change":"2024-02-23T00:32:36.714218+0000","last_active":"2024-02-23T00:33:04.034085+0000","last_peered":"2024-02-23T00:33:04.034085+0000","last_clean":"2024-02-23T00:33:04.034085+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:33:04.034085+0000","last_undegraded":"2024-02-23T00:33:04.034085+0000","last_fullsized":"2024-02-23T00:33:04.034085+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.714139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.714139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":198,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.032817+0000","last_change":"2024-02-23T00:32:44.640411+0000","last_active":"2024-02-23T00:33:04.032817+0000","last_peered":"2024-02-23T00:33:04.032817+0000","last_clean":"2024-02-23T00:33:04.032817+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:33:04.032817+0000","last_undegraded":"2024-02-23T00:33:04.032817+0000","last_fullsized":"2024-02-23T00:33:04.032817+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.640331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.640331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":190,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.022377+0000","last_change":"2024-02-23T00:31:39.494827+0000","last_active":"2024-02-23T00:33:04.022377+0000","last_peered":"2024-02-23T00:33:04.022377+0000","last_clean":"2024-02-23T00:33:04.022377+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:33:04.022377+0000","last_undegraded":"2024-02-23T00:33:04.022377+0000","last_fullsized":"2024-02-23T00:33:04.022377+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:31:39.494737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":144,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037208+0000","last_change":"2024-02-23T00:30:48.725798+0000","last_active":"2024-02-23T00:33:04.037208+0000","last_peered":"2024-02-23T00:33:04.037208+0000","last_clean":"2024-02-23T00:33:04.037208+0000","last_became_active":"2024-02-23T00:30:48.725162+0000","last_became_peered":"2024-02-23T00:30:48.725162+0000","last_unstale":"2024-02-23T00:33:04.037208+0000","last_undegraded":"2024-02-23T00:33:04.037208+0000","last_fullsized":"2024-02-23T00:33:04.037208+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":232,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.019906+0000","last_change":"2024-02-23T00:32:10.903259+0000","last_active":"2024-02-23T00:33:04.019906+0000","last_peered":"2024-02-23T00:33:04.019906+0000","last_clean":"2024-02-23T00:33:04.019906+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:33:04.019906+0000","last_undegraded":"2024-02-23T00:33:04.019906+0000","last_fullsized":"2024-02-23T00:33:04.019906+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:32:10.903186+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.903186+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.029824+0000","last_change":"2024-02-23T00:31:22.137283+0000","last_active":"2024-02-23T00:33:04.029824+0000","last_peered":"2024-02-23T00:33:04.029824+0000","last_clean":"2024-02-23T00:33:04.029824+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:33:04.029824+0000","last_undegraded":"2024-02-23T00:33:04.029824+0000","last_fullsized":"2024-02-23T00:33:04.029824+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:22.137166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:22.137166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":193,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.036798+0000","last_change":"2024-02-23T00:31:26.920222+0000","last_active":"2024-02-23T00:33:04.036798+0000","last_peered":"2024-02-23T00:33:04.036798+0000","last_clean":"2024-02-23T00:33:04.036798+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:33:04.036798+0000","last_undegraded":"2024-02-23T00:33:04.036798+0000","last_fullsized":"2024-02-23T00:33:04.036798+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:26.920133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:26.920133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.028817+0000","last_change":"2024-02-23T00:31:37.065941+0000","last_active":"2024-02-23T00:33:04.028817+0000","last_peered":"2024-02-23T00:33:04.028817+0000","last_clean":"2024-02-23T00:33:04.028817+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:33:04.028817+0000","last_undegraded":"2024-02-23T00:33:04.028817+0000","last_fullsized":"2024-02-23T00:33:04.028817+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:31:37.065865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":145,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.033037+0000","last_change":"2024-02-23T00:30:48.698701+0000","last_active":"2024-02-23T00:33:04.033037+0000","last_peered":"2024-02-23T00:33:04.033037+0000","last_clean":"2024-02-23T00:33:04.033037+0000","last_became_active":"2024-02-23T00:30:48.697963+0000","last_became_peered":"2024-02-23T00:30:48.697963+0000","last_unstale":"2024-02-23T00:33:04.033037+0000","last_undegraded":"2024-02-23T00:33:04.033037+0000","last_fullsized":"2024-02-23T00:33:04.033037+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"63'39","reported_seq":281,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.025336+0000","last_change":"2024-02-23T00:30:23.839691+0000","last_active":"2024-02-23T00:33:04.025336+0000","last_peered":"2024-02-23T00:33:04.025336+0000","last_clean":"2024-02-23T00:33:04.025336+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:33:04.025336+0000","last_undegraded":"2024-02-23T00:33:04.025336+0000","last_fullsized":"2024-02-23T00:33:04.025336+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'39","last_scrub_stamp":"2024-02-23T00:30:23.839568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.839568+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":203,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.021780+0000","last_change":"2024-02-23T00:32:13.602518+0000","last_active":"2024-02-23T00:33:04.021780+0000","last_peered":"2024-02-23T00:33:04.021780+0000","last_clean":"2024-02-23T00:33:04.021780+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:33:04.021780+0000","last_undegraded":"2024-02-23T00:33:04.021780+0000","last_fullsized":"2024-02-23T00:33:04.021780+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:13.602450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.602450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.020068+0000","last_change":"2024-02-23T00:31:40.071364+0000","last_active":"2024-02-23T00:33:04.020068+0000","last_peered":"2024-02-23T00:33:04.020068+0000","last_clean":"2024-02-23T00:33:04.020068+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:33:04.020068+0000","last_undegraded":"2024-02-23T00:33:04.020068+0000","last_fullsized":"2024-02-23T00:33:04.020068+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.071294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.071294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":192,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.024267+0000","last_change":"2024-02-23T00:30:46.848126+0000","last_active":"2024-02-23T00:33:04.024267+0000","last_peered":"2024-02-23T00:33:04.024267+0000","last_clean":"2024-02-23T00:33:04.024267+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:33:04.024267+0000","last_undegraded":"2024-02-23T00:33:04.024267+0000","last_fullsized":"2024-02-23T00:33:04.024267+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.848063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.848063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":149,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-23T00:33:04.037457+0000","last_change":"2024-02-23T00:32:56.396227+0000","last_active":"2024-02-23T00:33:04.037457+0000","last_peered":"2024-02-23T00:33:04.037457+0000","last_clean":"2024-02-23T00:33:04.037457+0000","last_became_active":"2024-02-23T00:30:48.722569+0000","last_became_peered":"2024-02-23T00:30:48.722569+0000","last_unstale":"2024-02-23T00:33:04.037457+0000","last_undegraded":"2024-02-23T00:33:04.037457+0000","last_fullsized":"2024-02-23T00:33:04.037457+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.396150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.396150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":162,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":161,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":160,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":158,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":157,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":156,"num_pg":32,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":1193,"num_objects":117,"num_object_clones":2,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":1,"num_read":92,"num_read_kb":1,"num_write":527,"num_write_kb":524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":3579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1750,"ondisk_log_size":1750,"up":96,"acting":96,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":19,"num_pg":11,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":640,"ondisk_log_size":640,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332169,"num_pgs":117,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309164,"kb_used_data":3740,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435988,"statfs":{"total":95995035648,"available":95678451712,"internally_reserved":0,"allocated":3829760,"data_stored":2127847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.0830000000000002,"15min":1.028},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":57.401000000000003,"5min":61.048999999999999,"15min":61.048999999999999},"last":0.436},{"interface":"front","average":{"1min":3.6440000000000001,"5min":2.6549999999999998,"15min":1.2390000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":65.558999999999997,"5min":73.870999999999995,"15min":73.870999999999995},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":2.028,"5min":1.026,"15min":0.69499999999999995},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.869,"5min":1.044,"15min":0.68400000000000005},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":29.423999999999999,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.60799999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.95499999999999996,"15min":0.66900000000000004},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":9.5190000000000001,"5min":16.603999999999999,"15min":16.603999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.753,"15min":0.59999999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":7.5339999999999998,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":3.2280000000000002,"5min":1.871,"15min":0.97099999999999997},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":57.729999999999997,"5min":57.729999999999997,"15min":57.729999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.613,"15min":0.89400000000000002},"min":{"1min":0.31900000000000001,"5min":0.312,"15min":0.312},"max":{"1min":27.573,"5min":27.573,"15min":27.573},"last":0.57599999999999996}]},{"osd":4,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41899999999999998,"15min":0.41199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.43099999999999999,"15min":0.42499999999999999},"min":{"1min":0.25700000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.43099999999999999,"15min":0.42799999999999999},"min":{"1min":0.27200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45600000000000002,"15min":0.435},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48699999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.48099999999999998,"15min":0.47299999999999998},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48099999999999998,"15min":0.46100000000000002},"min":{"1min":0.30399999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.45600000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561102,"num_pgs":122,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313272,"kb_used_data":3752,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431880,"statfs":{"total":95995035648,"available":95674245120,"internally_reserved":0,"allocated":3842048,"data_stored":2137948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":1.7090000000000001,"15min":0.90100000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.63100000000000001,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.71199999999999997,"15min":0.58399999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.70999999999999996,"5min":9.468,"15min":9.468},"last":0.375}]},{"osd":1,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":2.0299999999999998,"15min":1.0129999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.70899999999999996,"5min":59.347999999999999,"15min":59.347999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":2.8730000000000002,"15min":1.2889999999999999},"min":{"1min":0.28299999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.63800000000000001,"5min":87.221000000000004,"15min":87.221000000000004},"last":0.38700000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":1.427,"15min":0.80700000000000005},"min":{"1min":0.32900000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78800000000000003,"5min":25.550999999999998,"15min":25.550999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":1.5129999999999999,"15min":0.871},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.60799999999999998,"5min":40.914000000000001,"15min":40.914000000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":1.387,"15min":0.78800000000000003},"min":{"1min":0.30199999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76000000000000001,"5min":31.309999999999999,"15min":31.309999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.495,"5min":2.9129999999999998,"15min":1.3200000000000001},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72199999999999998,"5min":86.311999999999998,"15min":86.311999999999998},"last":0.44600000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.44800000000000001,"15min":0.42399999999999999},"min":{"1min":0.28599999999999998,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":0.68500000000000005,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.65900000000000003,"15min":0.48899999999999999},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59499999999999997,"5min":17.718,"15min":17.718},"last":0.436}]},{"osd":5,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.56399999999999995,"15min":0.47299999999999998},"min":{"1min":0.33300000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.73799999999999999,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.54700000000000004,"15min":0.46100000000000002},"min":{"1min":0.316,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66500000000000004,"5min":8.1370000000000005,"15min":8.1370000000000005},"last":0.47199999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:32:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.82899999999999996,"15min":0.58799999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.52100000000000002,"5min":17.881,"15min":17.881},"last":0.504},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.79700000000000004,"15min":0.58099999999999996},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54400000000000004,"5min":17.396999999999998,"15min":17.396999999999998},"last":0.50900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":149,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313456,"kb_used_data":3936,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431696,"statfs":{"total":95995035648,"available":95674056704,"internally_reserved":0,"allocated":4030464,"data_stored":2143853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.3169999999999999,"15min":0.65700000000000003},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":25.129000000000001,"5min":37.133000000000003,"15min":37.133000000000003},"last":2.2349999999999999},{"interface":"front","average":{"1min":1.7,"5min":0.93999999999999995,"15min":0.54900000000000004},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":26.690000000000001,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.93200000000000005}]},{"osd":2,"last update":"Fri Feb 23 00:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.89500000000000002,"15min":0.54700000000000004},"min":{"1min":0.249,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.7570000000000001,"5min":29.140999999999998,"15min":29.140999999999998},"last":0.316},{"interface":"front","average":{"1min":0.875,"5min":0.65600000000000003,"15min":0.46999999999999997},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":8.8300000000000001,"5min":15.273,"15min":15.273},"last":2.1909999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:32:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.1319999999999999,"15min":0.66400000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.294,"5min":21.013999999999999,"15min":21.013999999999999},"last":2.3919999999999999},{"interface":"front","average":{"1min":0.496,"5min":1.0289999999999999,"15min":0.63300000000000001},"min":{"1min":0.253,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.79,"5min":29.234999999999999,"15min":29.234999999999999},"last":10.382999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":0.92100000000000004,"15min":0.67700000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":16.231000000000002,"5min":16.231000000000002,"15min":16.231000000000002},"last":8.5559999999999992},{"interface":"front","average":{"1min":0.48799999999999999,"5min":1.101,"15min":0.73199999999999998},"min":{"1min":0.34599999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72299999999999998,"5min":29.212,"15min":29.212},"last":1.135}]},{"osd":5,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.79100000000000004,"15min":0.65100000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71799999999999997,"5min":11.904,"15min":11.904},"last":8.2579999999999991},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.83099999999999996,"15min":0.67000000000000004},"min":{"1min":0.378,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.8889999999999998,"5min":11.105,"15min":11.105},"last":8.4030000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":1.0860000000000001,"15min":0.73999999999999999},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68100000000000005,"5min":29.187000000000001,"15min":29.187000000000001},"last":10.416},{"interface":"front","average":{"1min":1.367,"5min":0.98299999999999998,"15min":0.74099999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":16.369,"5min":16.369,"15min":16.369},"last":8.2929999999999993}]},{"osd":7,"last update":"Fri Feb 23 00:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.80600000000000005,"15min":0.65000000000000002},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":8.8040000000000003,"5min":11.971,"15min":11.971},"last":15.295999999999999},{"interface":"front","average":{"1min":1.115,"5min":0.82699999999999996,"15min":0.68300000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.4379999999999997,"5min":11.135999999999999,"15min":11.135999999999999},"last":0.65800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305108,"kb_used_data":3784,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440044,"statfs":{"total":95995035648,"available":95682605056,"internally_reserved":0,"allocated":3874816,"data_stored":2126704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.82499999999999996,"15min":0.49399999999999999},"min":{"1min":0.215,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.7770000000000001,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":1.1779999999999999,"15min":0.61599999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.5469999999999997,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.23200000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.95499999999999996,"15min":0.58299999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.0620000000000001,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66400000000000003,"15min":0.46400000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.9379999999999997,"5min":15.965,"15min":15.965},"last":0.27100000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.0189999999999999,"15min":0.64000000000000001},"min":{"1min":0.28199999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.095,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.312},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.95999999999999996,"15min":0.62},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.245},"max":{"1min":1.046,"5min":16.006,"15min":16.006},"last":0.32200000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:32:11 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":0.80700000000000005,"15min":0.64100000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":15.917999999999999,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":2.8170000000000002,"5min":1.135,"15min":0.754},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":13.287000000000001,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.501}]},{"osd":5,"last update":"Fri Feb 23 00:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.0680000000000001,"15min":0.72399999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.4660000000000002,"5min":13.311,"15min":13.311},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.0780000000000001,"15min":0.73099999999999998},"min":{"1min":0.38600000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.5139999999999998,"5min":23.744,"15min":23.744},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.3740000000000001,"15min":0.80300000000000005},"min":{"1min":0.40100000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.5510000000000002,"5min":33.082999999999998,"15min":33.082999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0069999999999999,"15min":0.71199999999999997},"min":{"1min":0.38800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":9.8040000000000003,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.48899999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:32:18 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.175,"15min":0.76400000000000001},"min":{"1min":0.434,"5min":0.373,"15min":0.373},"max":{"1min":15.943,"5min":31.251999999999999,"15min":31.251999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":3.1059999999999999,"5min":1.141,"15min":0.753},"min":{"1min":0.38400000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":23.785,"5min":23.785,"15min":23.785},"last":0.495}]}]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":121,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305144,"kb_used_data":3812,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440008,"statfs":{"total":95995035648,"available":95682568192,"internally_reserved":0,"allocated":3903488,"data_stored":2127729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.4430000000000001,"5min":1.4850000000000001,"15min":0.71599999999999997},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":56.509,"5min":56.509,"15min":56.509},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.4179999999999999,"15min":0.69399999999999995},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":19.571999999999999,"5min":31.498999999999999,"15min":31.498999999999999},"last":0.34000000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":0.96899999999999997,"15min":0.54900000000000004},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":19.324999999999999,"5min":19.324999999999999,"15min":19.324999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.534,"5min":1.288,"15min":0.65600000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.216},"max":{"1min":20.655000000000001,"5min":35.052,"15min":35.052},"last":0.36299999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:32:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":1.25,"15min":0.69299999999999995},"min":{"1min":0.20999999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54600000000000004,"5min":43.5,"15min":43.5},"last":0.379},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.96499999999999997,"15min":0.61599999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57599999999999996,"5min":27.414000000000001,"15min":27.414000000000001},"last":0.38700000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:32:08 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.76700000000000002,"15min":0.61699999999999999},"min":{"1min":0.36499999999999999,"5min":0.308,"15min":0.308},"max":{"1min":14.614000000000001,"5min":14.614000000000001,"15min":14.614000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.65300000000000002,"15min":0.58899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":4.6550000000000002,"5min":4.6550000000000002,"15min":4.6550000000000002},"last":0.44800000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.88400000000000001,"15min":0.65600000000000003},"min":{"1min":0.32700000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.66100000000000003,"5min":17.765000000000001,"15min":17.765000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.78500000000000003,"15min":0.629},"min":{"1min":0.34899999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.81799999999999995,"5min":14.589,"15min":14.589},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.66700000000000004,"15min":0.58799999999999997},"min":{"1min":0.34399999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":4.6120000000000001,"15min":4.6120000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.038,"15min":0.74299999999999999},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69399999999999995,"5min":20.574999999999999,"15min":20.574999999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:32:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.8839999999999999,"15min":1.024},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":11.811999999999999,"5min":66.632000000000005,"15min":66.632000000000005},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.212,"15min":0.80100000000000005},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":11.781000000000001,"5min":19.603999999999999,"15min":19.603999999999999},"last":0.625}]}]},{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":122,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305048,"kb_used_data":3716,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440104,"statfs":{"total":95995035648,"available":95682666496,"internally_reserved":0,"allocated":3805184,"data_stored":2127055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":1.032,"15min":0.59499999999999997},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.503,"5min":45.390999999999998,"15min":45.390999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":1.004,"15min":0.60399999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.46600000000000003,"5min":30.186,"15min":30.186},"last":0.378}]},{"osd":1,"last update":"Fri Feb 23 00:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.61899999999999999,"15min":0.497},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.018,"5min":18.039999999999999,"15min":18.039999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":1.014,"15min":0.61799999999999999},"min":{"1min":0.26000000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63100000000000001,"5min":45.441000000000003,"15min":45.441000000000003},"last":0.28399999999999997}]},{"osd":2,"last update":"Fri Feb 23 00:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.96399999999999997,"15min":0.61899999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.49299999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.31},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.90200000000000002,"15min":0.59099999999999997},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.48099999999999998,"5min":24.183,"15min":24.183},"last":0.33500000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:32:09 2024","interfaces":[{"interface":"back","average":{"1min":3.1219999999999999,"5min":1.085,"15min":0.749},"min":{"1min":0.29999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":24.058,"5min":24.058,"15min":24.058},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.2110000000000001,"5min":0.74099999999999999,"15min":0.64000000000000001},"min":{"1min":0.35699999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.391,"5min":8.0359999999999996,"15min":8.0359999999999996},"last":0.66900000000000004}]},{"osd":5,"last update":"Fri Feb 23 00:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":1.056,"15min":0.73299999999999998},"min":{"1min":0.33900000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77200000000000002,"5min":45.265000000000001,"15min":45.265000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.91400000000000003,"15min":0.69399999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.113,"5min":24.102,"15min":24.102},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.745,"15min":0.63400000000000001},"min":{"1min":0.35199999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.093,"5min":8.093,"15min":8.093},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":1.179,"15min":0.79900000000000004},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75,"5min":45.232999999999997,"15min":45.232999999999997},"last":0.55900000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:32:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.71199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.1240000000000001,"5min":8.1280000000000001,"15min":8.1280000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.089,"15min":0.77000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65100000000000002,"5min":45.287999999999997,"15min":45.287999999999997},"last":0.68300000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986264,"num_pgs":127,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313344,"kb_used_data":3828,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431808,"statfs":{"total":95995035648,"available":95674171392,"internally_reserved":0,"allocated":3919872,"data_stored":2127557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.4159999999999999,"15min":0.81599999999999995},"min":{"1min":0.27700000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.0009999999999994,"5min":21.419,"15min":21.419},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.85599999999999998,"5min":2.6200000000000001,"15min":1.226},"min":{"1min":0.33900000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.1710000000000003,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.65900000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.9359999999999999,"15min":0.99299999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":13.044,"5min":41.137999999999998,"15min":41.137999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.444,"15min":0.82699999999999996},"min":{"1min":0.312,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":8.0310000000000006,"5min":26.210000000000001,"15min":26.210000000000001},"last":0.46000000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":2.1120000000000001,"15min":1.0609999999999999},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.5249999999999999,"5min":67.253,"15min":67.253},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":1.8540000000000001,"15min":0.96499999999999997},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.7280000000000002,"5min":41.927999999999997,"15min":41.927999999999997},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":2.0219999999999998,"15min":1.0229999999999999},"min":{"1min":0.35699999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.6949999999999998,"5min":36.301000000000002,"15min":36.301000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.626,"5min":2.2389999999999999,"15min":1.101},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.899,"5min":42.064,"15min":42.064},"last":0.52200000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.754,"15min":0.49199999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.1360000000000001,"5min":14.242000000000001,"15min":14.242000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.69999999999999996,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6299999999999999,"5min":14.124000000000001,"15min":14.124000000000001},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60899999999999999,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":3.6669999999999998,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.95199999999999996,"15min":0.60299999999999998},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.5640000000000001,"5min":45.814999999999998,"15min":45.814999999999998},"last":0.59199999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:32:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.753,"15min":0.55900000000000005},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.6630000000000003,"5min":17.178999999999998,"15min":17.178999999999998},"last":0.627},{"interface":"front","average":{"1min":1.028,"5min":0.77800000000000002,"15min":0.59499999999999997},"min":{"1min":0.35599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.1959999999999997,"5min":14.212,"15min":14.212},"last":0.68100000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757331,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309260,"kb_used_data":3836,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435892,"statfs":{"total":95995035648,"available":95678353408,"internally_reserved":0,"allocated":3928064,"data_stored":2126942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":1.115,"15min":0.70999999999999996},"min":{"1min":0.23400000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.67600000000000005,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.432},{"interface":"front","average":{"1min":0.435,"5min":2.1070000000000002,"15min":1.0269999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.66000000000000003,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.45400000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.29,"15min":0.748},"min":{"1min":0.27100000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72299999999999998,"5min":38.838000000000001,"15min":38.838000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.439,"5min":2.5179999999999998,"15min":1.1699999999999999},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.69599999999999995,"5min":127.794,"15min":127.794},"last":0.372}]},{"osd":2,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.651,"15min":0.88400000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63,"5min":37.442999999999998,"15min":37.442999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":1.0620000000000001,"15min":0.69099999999999995},"min":{"1min":0.30499999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67200000000000004,"5min":23.297999999999998,"15min":23.297999999999998},"last":0.51000000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":1.226,"15min":0.72899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71799999999999997,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.441},{"interface":"front","average":{"1min":0.47799999999999998,"5min":1.6160000000000001,"15min":0.88800000000000001},"min":{"1min":0.33300000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.72199999999999998,"5min":40.865000000000002,"15min":40.865000000000002},"last":0.47399999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.5489999999999999,"15min":0.75900000000000001},"min":{"1min":0.27400000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.63700000000000001,"5min":103.13,"15min":103.13},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.57399999999999995,"15min":0.432},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.628,"5min":12.795999999999999,"15min":12.795999999999999},"last":0.35599999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.57799999999999996,"15min":0.47199999999999998},"min":{"1min":0.35599999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69099999999999995,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":1.5640000000000001,"15min":0.80800000000000005},"min":{"1min":0.28399999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73299999999999998,"5min":103.61499999999999,"15min":103.61499999999999},"last":0.41799999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.56499999999999995,"15min":0.51100000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66300000000000003,"5min":7.1820000000000004,"15min":7.1820000000000004},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.69699999999999995,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.33100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":158,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":158,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":158,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":158,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":158,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":158,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":158,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":158,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":160,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":160,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":160,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":160,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":160,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":160,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":160,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":160,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:33:07.490 INFO:tasks.thrashosds.thrasher:No pools; doing nothing 2024-02-23T00:33:07.562 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:07.594 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:07.615 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:07.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:07.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:07.748 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:08.312 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:06.343034+0000 mgr.y (mgr.24437) 263 : audit [DBG] from='client.26757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:07.464477+0000 mon.c (mon.2) 283 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:08.012838+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi087-82011-34"}]': finished 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:08.012902+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-25"}]': finished 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:08.012979+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:08.017039+0000 mon.b (mon.1) 212 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-25", "mode": "writeback"}]: dispatch 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:08.018455+0000 mon.a (mon.0) 1715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-25", "mode": "writeback"}]: dispatch 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:06.343034+0000 mgr.y (mgr.24437) 263 : audit [DBG] from='client.26757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:07.464477+0000 mon.c (mon.2) 283 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:08.012838+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi087-82011-34"}]': finished 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:08.012902+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-25"}]': finished 2024-02-23T00:33:08.313 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:08.012979+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-02-23T00:33:08.314 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:08.017039+0000 mon.b (mon.1) 212 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-25", "mode": "writeback"}]: dispatch 2024-02-23T00:33:08.314 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:08.018455+0000 mon.a (mon.0) 1715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-25", "mode": "writeback"}]: dispatch 2024-02-23T00:33:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:06.343034+0000 mgr.y (mgr.24437) 263 : audit [DBG] from='client.26757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:33:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:07.464477+0000 mon.c (mon.2) 283 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:08.012838+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? 172.21.15.87:0/1200734777' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi087-82011-34"}]': finished 2024-02-23T00:33:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:08.012902+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-25"}]': finished 2024-02-23T00:33:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:08.012979+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-02-23T00:33:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:08.017039+0000 mon.b (mon.1) 212 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-25", "mode": "writeback"}]: dispatch 2024-02-23T00:33:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:08.018455+0000 mon.a (mon.0) 1715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-25", "mode": "writeback"}]: dispatch 2024-02-23T00:33:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:07.596888+0000 mgr.y (mgr.24437) 264 : cluster [DBG] pgmap v337: 409 pgs: 32 unknown, 368 active+clean, 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-23T00:33:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:08.036348+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:08.037859+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:08.040857+0000 mon.a (mon.0) 1718 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi087-82011-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:08.465387+0000 mon.c (mon.2) 284 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:09.008654+0000 mon.a (mon.0) 1719 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:09.014170+0000 mon.a (mon.0) 1720 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-25", "mode": "writeback"}]': finished 2024-02-23T00:33:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:09.014290+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi087-82011-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:09.014360+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:09.018704+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi087-82011-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:07.596888+0000 mgr.y (mgr.24437) 264 : cluster [DBG] pgmap v337: 409 pgs: 32 unknown, 368 active+clean, 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:08.036348+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:08.037859+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:08.040857+0000 mon.a (mon.0) 1718 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi087-82011-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:08.465387+0000 mon.c (mon.2) 284 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:09.008654+0000 mon.a (mon.0) 1719 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:09.014170+0000 mon.a (mon.0) 1720 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-25", "mode": "writeback"}]': finished 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:09.014290+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi087-82011-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:09.014360+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-02-23T00:33:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:09.018704+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi087-82011-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:07.596888+0000 mgr.y (mgr.24437) 264 : cluster [DBG] pgmap v337: 409 pgs: 32 unknown, 368 active+clean, 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-23T00:33:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:08.036348+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:08.037859+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:08.040857+0000 mon.a (mon.0) 1718 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi087-82011-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:08.465387+0000 mon.c (mon.2) 284 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:09.008654+0000 mon.a (mon.0) 1719 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:09.014170+0000 mon.a (mon.0) 1720 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-25", "mode": "writeback"}]': finished 2024-02-23T00:33:09.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:09.014290+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi087-82011-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:09.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:09.014360+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-02-23T00:33:09.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:09.018704+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi087-82011-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:09.701 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:09.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:09.803 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:09.835 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:09.857 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:09.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:09.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:09.960 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:09.992 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:10.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 -- 172.21.15.87:0/2508359870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 msgr2=0x7f09040f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2508359870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 0x7f09040f7d40 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f08f4004870 tx=0x7f08f4009cb0).stop 2024-02-23T00:33:10.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 -- 172.21.15.87:0/2508359870 shutdown_connections 2024-02-23T00:33:10.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2508359870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09040f8420 0x7f0904104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:33:10.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2508359870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09040fdbb0 0x7f09040fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2508359870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 0x7f09040f7d40 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 -- 172.21.15.87:0/2508359870 >> 172.21.15.87:0/2508359870 conn(0x7f09040f3800 msgr2=0x7f09040f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:10.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 -- 172.21.15.87:0/2508359870 shutdown_connections 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 -- 172.21.15.87:0/2508359870 wait complete. 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 librados: starting msgr at 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 librados: starting objecter 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 Processor -- start 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 -- start start 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 librados: setting wanted keys 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.382+0000 7f090bb6e700 1 librados: calling monclient init 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09040f8420 0x7f090410e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09040fdbb0 0x7f090410e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 0x7f09041082f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09041087e0 con 0x7f09040fdbb0 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0904108950 con 0x7f09040f8420 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0904108b10 con 0x7f09040fe510 2024-02-23T00:33:10.415 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f0909109700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09040fdbb0 0x7f090410e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090a10b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 0x7f09041082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090a10b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 0x7f09041082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:52584/0 (socket says 172.21.15.87:52584) 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090a10b700 1 -- 172.21.15.87:0/2665523323 learned_addr learned my addr 172.21.15.87:0/2665523323 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f0909109700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09040fdbb0 0x7f090410e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56778/0 (socket says 172.21.15.87:56778) 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090990a700 1 --2- 172.21.15.87:0/2665523323 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09040f8420 0x7f090410e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090a10b700 1 -- 172.21.15.87:0/2665523323 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09040f8420 msgr2=0x7f090410e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090a10b700 1 --2- 172.21.15.87:0/2665523323 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09040f8420 0x7f090410e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090a10b700 1 -- 172.21.15.87:0/2665523323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09040fdbb0 msgr2=0x7f090410e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090a10b700 1 --2- 172.21.15.87:0/2665523323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09040fdbb0 0x7f090410e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090a10b700 1 -- 172.21.15.87:0/2665523323 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f4005040 con 0x7f09040fe510 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f0909109700 1 --2- 172.21.15.87:0/2665523323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09040fdbb0 0x7f090410e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090a10b700 1 --2- 172.21.15.87:0/2665523323 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 0x7f09041082f0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f08fc00bc30 tx=0x7f08fc009840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f08faffd700 10 client.?.objecter ms_handle_connect 0x7f09040fe510 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f08faffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f08faffd700 1 -- 172.21.15.87:0/2665523323 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc00c6f0 con 0x7f09040fe510 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f08faffd700 1 -- 172.21.15.87:0/2665523323 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08fc010070 con 0x7f09040fe510 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0904108df0 con 0x7f09040fe510 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f08faffd700 1 -- 172.21.15.87:0/2665523323 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08fc007880 con 0x7f09040fe510 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 10 client.37217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0904114f80 con 0x7f09040fe510 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.383+0000 7f090bb6e700 1 librados: init done 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.384+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090405ee80 con 0x7f09040fe510 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.384+0000 7f08faffd700 1 -- 172.21.15.87:0/2665523323 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f08fc007c30 con 0x7f09040fe510 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.385+0000 7f08faffd700 1 --2- 172.21.15.87:0/2665523323 >> v2:172.21.15.87:6800/1231597463 conn(0x7f08f0062c90 0x7f08f0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.385+0000 7f08faffd700 1 -- 172.21.15.87:0/2665523323 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(239..239 src has 38..239) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f08fc07d140 con 0x7f09040fe510 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.385+0000 7f090990a700 1 --2- 172.21.15.87:0/2665523323 >> v2:172.21.15.87:6800/1231597463 conn(0x7f08f0062c90 0x7f08f0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.385+0000 7f08faffd700 10 client.37217.objecter ms_dispatch 0x7f0904001550 osd_map(239..239 src has 38..239) v4 2024-02-23T00:33:10.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.385+0000 7f08faffd700 3 client.37217.objecter handle_osd_map got epochs [239,239] > 0 2024-02-23T00:33:10.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.385+0000 7f08faffd700 3 client.37217.objecter handle_osd_map decoding full epoch 239 2024-02-23T00:33:10.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.385+0000 7f08faffd700 20 client.37217.objecter dump_active .. 0 homeless 2024-02-23T00:33:10.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.385+0000 7f08faffd700 10 client.37217.objecter ms_handle_connect 0x7f08f0062c90 2024-02-23T00:33:10.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.385+0000 7f090990a700 1 --2- 172.21.15.87:0/2665523323 >> v2:172.21.15.87:6800/1231597463 conn(0x7f08f0062c90 0x7f08f0065150 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f0900009f20 tx=0x7f090000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.389+0000 7f08faffd700 1 -- 172.21.15.87:0/2665523323 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f08fc050060 con 0x7f09040fe510 2024-02-23T00:33:10.440 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:10.442 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:09.075198+0000 mon.b (mon.1) 213 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:10.442 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:09.075530+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:10.442 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:09.466217+0000 mon.c (mon.2) 285 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:10.442 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:09.639229+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:10.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:09.075198+0000 mon.b (mon.1) 213 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:10.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:09.075530+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:10.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:09.466217+0000 mon.c (mon.2) 285 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:10.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:09.639229+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:10.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:10.544 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:10.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.569+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f090405fa60 con 0x7f09040fe510 2024-02-23T00:33:10.573 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:09.075198+0000 mon.b (mon.1) 213 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:10.573 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:09.075530+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:10.573 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:09.466217+0000 mon.c (mon.2) 285 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:10.573 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:09.639229+0000 mon.a (mon.0) 1725 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:10.575 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:10.596 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:10.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:10.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 -- 172.21.15.87:0/143546328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0fe510 msgr2=0x7f378c0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 --2- 172.21.15.87:0/143546328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0fe510 0x7f378c0f7d40 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 rx=0x7f377c004870 tx=0x7f377c009cb0).stop 2024-02-23T00:33:10.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 -- 172.21.15.87:0/143546328 shutdown_connections 2024-02-23T00:33:10.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 --2- 172.21.15.87:0/143546328 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f378c0f8420 0x7f378c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 --2- 172.21.15.87:0/143546328 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f378c0fdbb0 0x7f378c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 --2- 172.21.15.87:0/143546328 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0fe510 0x7f378c0f7d40 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 -- 172.21.15.87:0/143546328 >> 172.21.15.87:0/143546328 conn(0x7f378c0f3800 msgr2=0x7f378c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 -- 172.21.15.87:0/143546328 shutdown_connections 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 -- 172.21.15.87:0/143546328 wait complete. 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 librados: starting msgr at 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.654+0000 7f379118a700 1 librados: starting objecter 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 Processor -- start 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 -- start start 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 librados: setting wanted keys 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 librados: calling monclient init 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f378c0f8420 0x7f378c10dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f378c0fdbb0 0x7f378c10e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0fe510 0x7f378c108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c108640 con 0x7f378c0fe510 2024-02-23T00:33:10.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c1087b0 con 0x7f378c0f8420 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f379118a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378c108970 con 0x7f378c0fdbb0 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378b59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0fe510 0x7f378c108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378a59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f378c0fdbb0 0x7f378c10e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378b59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0fe510 0x7f378c108150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56796/0 (socket says 172.21.15.87:56796) 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378b59e700 1 -- 172.21.15.87:0/4047409660 learned_addr learned my addr 172.21.15.87:0/4047409660 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378ad9d700 1 --2- 172.21.15.87:0/4047409660 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f378c0f8420 0x7f378c10dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378a59c700 1 -- 172.21.15.87:0/4047409660 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f378c0f8420 msgr2=0x7f378c10dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378a59c700 1 --2- 172.21.15.87:0/4047409660 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f378c0f8420 0x7f378c10dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378a59c700 1 -- 172.21.15.87:0/4047409660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0fe510 msgr2=0x7f378c108150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378a59c700 1 --2- 172.21.15.87:0/4047409660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0fe510 0x7f378c108150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378a59c700 1 -- 172.21.15.87:0/4047409660 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37840097c0 con 0x7f378c0fdbb0 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.655+0000 7f378b59e700 1 --2- 172.21.15.87:0/4047409660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f378c0fe510 0x7f378c108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:10.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f378a59c700 1 --2- 172.21.15.87:0/4047409660 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f378c0fdbb0 0x7f378c10e4a0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f377c004870 tx=0x7f377c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f3773fff700 10 client.?.objecter ms_handle_connect 0x7f378c0fdbb0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f3773fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c00e070 con 0x7f378c0fdbb0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f377c013da0 con 0x7f378c0fdbb0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c005040 con 0x7f378c0fdbb0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c019070 con 0x7f378c0fdbb0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f379118a700 10 client.37223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378c108ee0 con 0x7f378c0fdbb0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.656+0000 7f379118a700 1 librados: init done 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.657+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f378c05ee80 con 0x7f378c0fdbb0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.657+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f377c0090d0 con 0x7f378c0fdbb0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.657+0000 7f3773fff700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3774062c90 0x7f3774065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(239..239 src has 38..239) v4 ==== 9416+0+0 (secure 0 0 0) 0x7f377c07e190 con 0x7f378c0fdbb0 2024-02-23T00:33:10.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 osd_map(239..239 src has 38..239) v4 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f3773fff700 3 client.37223.objecter handle_osd_map got epochs [239,239] > 0 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f3773fff700 3 client.37223.objecter handle_osd_map decoding full epoch 239 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f378ad9d700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3774062c90 0x7f3774065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f3773fff700 20 client.37223.objecter dump_active .. 0 homeless 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v239) v1 ==== 54+0+16 (secure 0 0 0) 0x7f377c085070 con 0x7f378c0fdbb0 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f3773fff700 10 client.37223.objecter ms_handle_connect 0x7f3774062c90 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f378ad9d700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3774062c90 0x7f3774065150 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 rx=0x7f37800041d0 tx=0x7f3780009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 10 client.37223.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 1 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6801/2153191863 conn(0x7f378c0fee20 0x7f378c0ff200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c1070a0 osd=0 3 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c1070a0 osd=0 3 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c1070a0 osd=0 3 2024-02-23T00:33:10.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 1 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c1070a0 osd=0 3 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 0 1 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c1070a0 osd=0 4 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 10 client.37223.objecter _send_command 1 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378c109630 con 0x7f378c0fee20 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f378b59e700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6801/2153191863 conn(0x7f378c0fee20 0x7f378c0ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f3773fff700 10 client.37223.objecter ms_handle_connect 0x7f378c0fee20 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.658+0000 7f378b59e700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6801/2153191863 conn(0x7f378c0fee20 0x7f378c0ff200 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.659+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f378c109630 con 0x7f378c0fee20 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.659+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.659+0000 7f3773fff700 10 client.37223.objecter _finish_command 1 = system:0 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.659+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c1070a0 osd=0 4 2024-02-23T00:33:10.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.659+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 0 1 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 10 client.37223.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 2 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c1070a0 osd=0 3 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c1070a0 osd=0 3 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c1070a0 osd=0 3 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 2 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c1070a0 osd=0 3 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 0 2 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c1070a0 osd=0 4 2024-02-23T00:33:10.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 10 client.37223.objecter _send_command 2 2024-02-23T00:33:10.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.674+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f378c0273c0 con 0x7f378c0fee20 2024-02-23T00:33:10.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.675+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f378c0273c0 con 0x7f378c0fee20 2024-02-23T00:33:10.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:10.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f3773fff700 10 client.37223.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c1070a0 osd=0 4 2024-02-23T00:33:10.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 0 2 2024-02-23T00:33:10.678 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 10 client.37223.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 3 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6805/685354171 conn(0x7f378c0598f0 0x7f378c059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c028710 osd=1 3 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c028710 osd=1 3 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c028710 osd=1 3 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 3 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c028710 osd=1 3 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 1 3 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c028710 osd=1 4 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 10 client.37223.objecter _send_command 3 2024-02-23T00:33:10.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378c11b560 con 0x7f378c0598f0 2024-02-23T00:33:10.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f378ad9d700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6805/685354171 conn(0x7f378c0598f0 0x7f378c059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f3773fff700 10 client.37223.objecter ms_handle_connect 0x7f378c0598f0 2024-02-23T00:33:10.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.676+0000 7f378ad9d700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6805/685354171 conn(0x7f378c0598f0 0x7f378c059cd0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.677+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f378c11b560 con 0x7f378c0598f0 2024-02-23T00:33:10.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.677+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:33:10.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.677+0000 7f3773fff700 10 client.37223.objecter _finish_command 3 = system:0 2024-02-23T00:33:10.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.677+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c028710 osd=1 4 2024-02-23T00:33:10.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.677+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 1 3 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 10 client.37223.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 4 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c028710 osd=1 3 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c028710 osd=1 3 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c028710 osd=1 3 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 4 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c028710 osd=1 3 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 1 4 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c028710 osd=1 4 2024-02-23T00:33:10.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 10 client.37223.objecter _send_command 4 2024-02-23T00:33:10.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.691+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f378c059360 con 0x7f378c0598f0 2024-02-23T00:33:10.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.692+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f378c059360 con 0x7f378c0598f0 2024-02-23T00:33:10.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.692+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:10.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f3773fff700 10 client.37223.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c028710 osd=1 4 2024-02-23T00:33:10.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 1 4 2024-02-23T00:33:10.695 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:33:10.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 10 client.37223.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:10.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 5 2024-02-23T00:33:10.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6809/3605394228 conn(0x7f378c119f80 0x7f378c11a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c027d60 osd=2 3 2024-02-23T00:33:10.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c027d60 osd=2 3 2024-02-23T00:33:10.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c027d60 osd=2 3 2024-02-23T00:33:10.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 5 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c027d60 osd=2 3 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 2 5 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c027d60 osd=2 4 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 10 client.37223.objecter _send_command 5 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378c0283e0 con 0x7f378c119f80 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f378a59c700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6809/3605394228 conn(0x7f378c119f80 0x7f378c11a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f3773fff700 10 client.37223.objecter ms_handle_connect 0x7f378c119f80 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.693+0000 7f378a59c700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6809/3605394228 conn(0x7f378c119f80 0x7f378c11a380 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.694+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f378c0283e0 con 0x7f378c119f80 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.694+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:33:10.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.694+0000 7f3773fff700 10 client.37223.objecter _finish_command 5 = system:0 2024-02-23T00:33:10.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.694+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c027d60 osd=2 4 2024-02-23T00:33:10.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.694+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 2 5 2024-02-23T00:33:10.700 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:10.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 10 client.37223.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:10.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 6 2024-02-23T00:33:10.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c027d60 osd=2 3 2024-02-23T00:33:10.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c027d60 osd=2 3 2024-02-23T00:33:10.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c027d60 osd=2 3 2024-02-23T00:33:10.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 6 2024-02-23T00:33:10.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c027d60 osd=2 3 2024-02-23T00:33:10.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 2 6 2024-02-23T00:33:10.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c027d60 osd=2 4 2024-02-23T00:33:10.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 10 client.37223.objecter _send_command 6 2024-02-23T00:33:10.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.708+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f378c11b040 con 0x7f378c119f80 2024-02-23T00:33:10.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f378c11b040 con 0x7f378c119f80 2024-02-23T00:33:10.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:10.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f3773fff700 10 client.37223.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c027d60 osd=2 4 2024-02-23T00:33:10.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 2 6 2024-02-23T00:33:10.712 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:33:10.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 10 client.37223.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:10.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 7 2024-02-23T00:33:10.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6813/128059790 conn(0x7f378c027940 0x7f378c11fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c0277d0 osd=3 3 2024-02-23T00:33:10.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c0277d0 osd=3 3 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c0277d0 osd=3 3 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 7 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c0277d0 osd=3 3 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 3 7 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c0277d0 osd=3 4 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 10 client.37223.objecter _send_command 7 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378c122850 con 0x7f378c027940 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.709+0000 7f378b59e700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6813/128059790 conn(0x7f378c027940 0x7f378c11fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.710+0000 7f3773fff700 10 client.37223.objecter ms_handle_connect 0x7f378c027940 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.710+0000 7f378b59e700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6813/128059790 conn(0x7f378c027940 0x7f378c11fc00 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.710+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f378c122850 con 0x7f378c027940 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.710+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.710+0000 7f3773fff700 10 client.37223.objecter _finish_command 7 = system:0 2024-02-23T00:33:10.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.710+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c0277d0 osd=3 4 2024-02-23T00:33:10.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.710+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 3 7 2024-02-23T00:33:10.729 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 10 client.37223.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 8 2024-02-23T00:33:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c0277d0 osd=3 3 2024-02-23T00:33:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c0277d0 osd=3 3 2024-02-23T00:33:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c0277d0 osd=3 3 2024-02-23T00:33:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 8 2024-02-23T00:33:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c0277d0 osd=3 3 2024-02-23T00:33:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 3 8 2024-02-23T00:33:10.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c0277d0 osd=3 4 2024-02-23T00:33:10.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 10 client.37223.objecter _send_command 8 2024-02-23T00:33:10.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.726+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f378c11aab0 con 0x7f378c027940 2024-02-23T00:33:10.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.728+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f378c11aab0 con 0x7f378c027940 2024-02-23T00:33:10.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.728+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:10.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.728+0000 7f3773fff700 10 client.37223.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.728+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c0277d0 osd=3 4 2024-02-23T00:33:10.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.728+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 3 8 2024-02-23T00:33:10.734 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:33:10.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 10 client.37223.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:10.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 9 2024-02-23T00:33:10.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6800/4175844757 conn(0x7f378c11ac50 0x7f378c121270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c11aab0 osd=4 3 2024-02-23T00:33:10.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c11aab0 osd=4 3 2024-02-23T00:33:10.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c11aab0 osd=4 3 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 9 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c11aab0 osd=4 3 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 4 9 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c11aab0 osd=4 4 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 10 client.37223.objecter _send_command 9 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378c119c60 con 0x7f378c11ac50 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f378ad9d700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6800/4175844757 conn(0x7f378c11ac50 0x7f378c121270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f3773fff700 10 client.37223.objecter ms_handle_connect 0x7f378c11ac50 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.729+0000 7f378ad9d700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6800/4175844757 conn(0x7f378c11ac50 0x7f378c121270 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.730+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f378c119c60 con 0x7f378c11ac50 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.730+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.730+0000 7f3773fff700 10 client.37223.objecter _finish_command 9 = system:0 2024-02-23T00:33:10.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.730+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c11aab0 osd=4 4 2024-02-23T00:33:10.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.730+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 4 9 2024-02-23T00:33:10.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 10 client.37223.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:10.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 10 2024-02-23T00:33:10.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c11aab0 osd=4 3 2024-02-23T00:33:10.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c11aab0 osd=4 3 2024-02-23T00:33:10.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c11aab0 osd=4 3 2024-02-23T00:33:10.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 10 2024-02-23T00:33:10.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c11aab0 osd=4 3 2024-02-23T00:33:10.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 4 10 2024-02-23T00:33:10.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c11aab0 osd=4 4 2024-02-23T00:33:10.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 10 client.37223.objecter _send_command 10 2024-02-23T00:33:10.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.744+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f378c119c60 con 0x7f378c11ac50 2024-02-23T00:33:10.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f378001e020 con 0x7f378c11ac50 2024-02-23T00:33:10.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:10.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f3773fff700 10 client.37223.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c11aab0 osd=4 4 2024-02-23T00:33:10.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 4 10 2024-02-23T00:33:10.749 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:33:10.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 10 client.37223.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:10.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 11 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6804/2550013795 conn(0x7f378c121de0 0x7f378c122220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c121760 osd=5 3 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c121760 osd=5 3 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c121760 osd=5 3 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 11 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c121760 osd=5 3 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 5 11 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c121760 osd=5 4 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 10 client.37223.objecter _send_command 11 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.746+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378c129140 con 0x7f378c121de0 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.747+0000 7f378a59c700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6804/2550013795 conn(0x7f378c121de0 0x7f378c122220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.747+0000 7f3773fff700 10 client.37223.objecter ms_handle_connect 0x7f378c121de0 2024-02-23T00:33:10.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.747+0000 7f378a59c700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6804/2550013795 conn(0x7f378c121de0 0x7f378c122220 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:10.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.748+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f378c129140 con 0x7f378c121de0 2024-02-23T00:33:10.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.748+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:33:10.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.748+0000 7f3773fff700 10 client.37223.objecter _finish_command 11 = system:0 2024-02-23T00:33:10.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.748+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c121760 osd=5 4 2024-02-23T00:33:10.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.748+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 5 11 2024-02-23T00:33:10.752 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:10.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:10.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 10 client.37223.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:10.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 12 2024-02-23T00:33:10.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c121760 osd=5 3 2024-02-23T00:33:10.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c121760 osd=5 3 2024-02-23T00:33:10.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c121760 osd=5 3 2024-02-23T00:33:10.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 12 2024-02-23T00:33:10.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c121760 osd=5 3 2024-02-23T00:33:10.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 5 12 2024-02-23T00:33:10.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c121760 osd=5 4 2024-02-23T00:33:10.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 10 client.37223.objecter _send_command 12 2024-02-23T00:33:10.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.762+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f378c12d6f0 con 0x7f378c121de0 2024-02-23T00:33:10.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f378c12d6f0 con 0x7f378c121de0 2024-02-23T00:33:10.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:10.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f3773fff700 10 client.37223.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c121760 osd=5 4 2024-02-23T00:33:10.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 5 12 2024-02-23T00:33:10.767 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:33:10.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 10 client.37223.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:10.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 13 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6808/2509408498 conn(0x7f378c12ddd0 0x7f378c12e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c12d6f0 osd=6 3 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12d6f0 osd=6 3 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c12d6f0 osd=6 3 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 13 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c12d6f0 osd=6 3 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 6 13 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12d6f0 osd=6 4 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 10 client.37223.objecter _send_command 13 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.764+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378c129140 con 0x7f378c12ddd0 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.765+0000 7f378b59e700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6808/2509408498 conn(0x7f378c12ddd0 0x7f378c12e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.765+0000 7f3773fff700 10 client.37223.objecter ms_handle_connect 0x7f378c12ddd0 2024-02-23T00:33:10.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.765+0000 7f378b59e700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6808/2509408498 conn(0x7f378c12ddd0 0x7f378c12e230 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.766+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f378c129140 con 0x7f378c12ddd0 2024-02-23T00:33:10.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.766+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:33:10.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.766+0000 7f3773fff700 10 client.37223.objecter _finish_command 13 = system:0 2024-02-23T00:33:10.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.766+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c12d6f0 osd=6 4 2024-02-23T00:33:10.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.766+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 6 13 2024-02-23T00:33:10.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 10 client.37223.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:10.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 14 2024-02-23T00:33:10.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c12d6f0 osd=6 3 2024-02-23T00:33:10.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12d6f0 osd=6 3 2024-02-23T00:33:10.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c12d6f0 osd=6 3 2024-02-23T00:33:10.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 14 2024-02-23T00:33:10.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c12d6f0 osd=6 3 2024-02-23T00:33:10.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 6 14 2024-02-23T00:33:10.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12d6f0 osd=6 4 2024-02-23T00:33:10.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 10 client.37223.objecter _send_command 14 2024-02-23T00:33:10.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.780+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f378c12e720 con 0x7f378c12ddd0 2024-02-23T00:33:10.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f378c12e720 con 0x7f378c12ddd0 2024-02-23T00:33:10.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:10.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f3773fff700 10 client.37223.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c12d6f0 osd=6 4 2024-02-23T00:33:10.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 6 14 2024-02-23T00:33:10.784 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 10 client.37223.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 15 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6812/373787000 conn(0x7f378c126ef0 0x7f378c127330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c12e720 osd=7 3 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12e720 osd=7 3 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c12e720 osd=7 3 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 15 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c12e720 osd=7 3 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 7 15 2024-02-23T00:33:10.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12e720 osd=7 4 2024-02-23T00:33:10.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 10 client.37223.objecter _send_command 15 2024-02-23T00:33:10.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f378c128e40 con 0x7f378c126ef0 2024-02-23T00:33:10.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.782+0000 7f378ad9d700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6812/373787000 conn(0x7f378c126ef0 0x7f378c127330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:10.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.783+0000 7f3773fff700 10 client.37223.objecter ms_handle_connect 0x7f378c126ef0 2024-02-23T00:33:10.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.783+0000 7f378ad9d700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6812/373787000 conn(0x7f378c126ef0 0x7f378c127330 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:10.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.783+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f378c128e40 con 0x7f378c126ef0 2024-02-23T00:33:10.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.783+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:33:10.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.783+0000 7f3773fff700 10 client.37223.objecter _finish_command 15 = system:0 2024-02-23T00:33:10.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.783+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c12e720 osd=7 4 2024-02-23T00:33:10.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.783+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 7 15 2024-02-23T00:33:10.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 10 client.37223.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:10.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign -1 16 2024-02-23T00:33:10.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c12e720 osd=7 3 2024-02-23T00:33:10.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12e720 osd=7 3 2024-02-23T00:33:10.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 20 client.37223.objecter _get_session s=0x7f378c12e720 osd=7 3 2024-02-23T00:33:10.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 15 client.37223.objecter _session_command_op_remove -1 16 2024-02-23T00:33:10.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 20 client.37223.objecter get_session s=0x7f378c12e720 osd=7 3 2024-02-23T00:33:10.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 15 client.37223.objecter _session_command_op_assign 7 16 2024-02-23T00:33:10.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12e720 osd=7 4 2024-02-23T00:33:10.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 10 client.37223.objecter _send_command 16 2024-02-23T00:33:10.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.797+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f378c128e40 con 0x7f378c126ef0 2024-02-23T00:33:10.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.799+0000 7f3773fff700 1 -- 172.21.15.87:0/4047409660 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f378c128e40 con 0x7f378c126ef0 2024-02-23T00:33:10.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.799+0000 7f3773fff700 10 client.37223.objecter ms_dispatch 0x7f378c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:10.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.799+0000 7f3773fff700 10 client.37223.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.799+0000 7f3773fff700 20 client.37223.objecter put_session s=0x7f378c12e720 osd=7 4 2024-02-23T00:33:10.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.799+0000 7f3773fff700 15 client.37223.objecter _session_command_op_remove 7 16 2024-02-23T00:33:10.802 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:33:10.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.799+0000 7f379118a700 10 librados: watch_flush enter 2024-02-23T00:33:10.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 librados: watch_flush exit 2024-02-23T00:33:10.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 client.37223.objecter close_session for osd.0 2024-02-23T00:33:10.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6801/2153191863 conn(0x7f378c0fee20 msgr2=0x7f378c0ff200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6801/2153191863 conn(0x7f378c0fee20 0x7f378c0ff200 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c1070a0 osd=0 1 2024-02-23T00:33:10.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 client.37223.objecter close_session for osd.1 2024-02-23T00:33:10.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6805/685354171 conn(0x7f378c0598f0 msgr2=0x7f378c059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6805/685354171 conn(0x7f378c0598f0 0x7f378c059cd0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c028710 osd=1 1 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 client.37223.objecter close_session for osd.2 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6809/3605394228 conn(0x7f378c119f80 msgr2=0x7f378c11a380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6809/3605394228 conn(0x7f378c119f80 0x7f378c11a380 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c027d60 osd=2 1 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 client.37223.objecter close_session for osd.3 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6813/128059790 conn(0x7f378c027940 msgr2=0x7f378c11fc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6813/128059790 conn(0x7f378c027940 0x7f378c11fc00 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f378b59e700 1 -- 172.21.15.87:0/4047409660 reap_dead start 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c0277d0 osd=3 1 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 client.37223.objecter close_session for osd.4 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6800/4175844757 conn(0x7f378c11ac50 msgr2=0x7f378c121270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.804 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6800/4175844757 conn(0x7f378c11ac50 0x7f378c121270 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f378b59e700 1 -- 172.21.15.87:0/4047409660 reap_dead start 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c11aab0 osd=4 1 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 client.37223.objecter close_session for osd.5 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6804/2550013795 conn(0x7f378c121de0 msgr2=0x7f378c122220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6804/2550013795 conn(0x7f378c121de0 0x7f378c122220 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c121760 osd=5 1 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 client.37223.objecter close_session for osd.6 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6808/2509408498 conn(0x7f378c12ddd0 msgr2=0x7f378c12e230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6808/2509408498 conn(0x7f378c12ddd0 0x7f378c12e230 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12d6f0 osd=6 1 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 client.37223.objecter close_session for osd.7 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6812/373787000 conn(0x7f378c126ef0 msgr2=0x7f378c127330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.171:6812/373787000 conn(0x7f378c126ef0 0x7f378c127330 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:10.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 20 client.37223.objecter put_session s=0x7f378c12e720 osd=7 1 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 20 client.37223.objecter shutdown clearing up homeless session... 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 10 client.37223.objecter successfully canceled tick 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3774062c90 msgr2=0x7f3774065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3774062c90 0x7f3774065150 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 rx=0x7f37800041d0 tx=0x7f3780009040).stop 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f378c0fdbb0 msgr2=0x7f378c10e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 --2- 172.21.15.87:0/4047409660 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f378c0fdbb0 0x7f378c10e4a0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f377c004870 tx=0x7f377c00b040).stop 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f378b59e700 1 -- 172.21.15.87:0/4047409660 reap_dead start 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 shutdown_connections 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 >> 172.21.15.87:0/4047409660 conn(0x7f378c0f3800 msgr2=0x7f378c0fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 shutdown_connections 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 -- 172.21.15.87:0/4047409660 wait complete. 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:10.800+0000 7f379118a700 1 librados: shutdown 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.806 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.807 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.807 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.807 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.807 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.807 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:33:10.855 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:10.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:33:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:33:10.883 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:10.902 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:10.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:11.006 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:11.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.019+0000 7f08faffd700 1 -- 172.21.15.87:0/2665523323 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v240) v1 ==== 97+0+0 (secure 0 0 0) 0x7f08fc04fc50 con 0x7f09040fe510 2024-02-23T00:33:11.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.022+0000 7f090bb6e700 10 librados: watch_flush enter 2024-02-23T00:33:11.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.022+0000 7f090bb6e700 10 librados: watch_flush exit 2024-02-23T00:33:11.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.022+0000 7f090bb6e700 20 client.37217.objecter shutdown clearing up homeless session... 2024-02-23T00:33:11.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.022+0000 7f090bb6e700 10 client.37217.objecter successfully canceled tick 2024-02-23T00:33:11.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.024+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 >> v2:172.21.15.87:6800/1231597463 conn(0x7f08f0062c90 msgr2=0x7f08f0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:11.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.024+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2665523323 >> v2:172.21.15.87:6800/1231597463 conn(0x7f08f0062c90 0x7f08f0065150 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 rx=0x7f0900009f20 tx=0x7f090000b040).stop 2024-02-23T00:33:11.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.024+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 msgr2=0x7f09041082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:11.027 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.024+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2665523323 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 0x7f09041082f0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f08fc00bc30 tx=0x7f08fc009840).stop 2024-02-23T00:33:11.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.027+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 shutdown_connections 2024-02-23T00:33:11.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.027+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2665523323 >> v2:172.21.15.87:6800/1231597463 conn(0x7f08f0062c90 0x7f08f0065150 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:11.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.027+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2665523323 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09040fe510 0x7f09041082f0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:11.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.027+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2665523323 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09040f8420 0x7f090410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:11.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.027+0000 7f090bb6e700 1 --2- 172.21.15.87:0/2665523323 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09040fdbb0 0x7f090410e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:11.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.027+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 >> 172.21.15.87:0/2665523323 conn(0x7f09040f3800 msgr2=0x7f09040fb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:11.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.028+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 shutdown_connections 2024-02-23T00:33:11.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.028+0000 7f090bb6e700 1 -- 172.21.15.87:0/2665523323 wait complete. 2024-02-23T00:33:11.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:11.031+0000 7f090bb6e700 1 librados: shutdown 2024-02-23T00:33:11.035 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:33:11.042 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:11.068 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:11.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:11.172 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:11.201 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:11.323 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:11.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:11.427 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:11.454 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:11.472 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:11.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:11.575 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:11.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:09.597397+0000 mgr.y (mgr.24437) 265 : cluster [DBG] pgmap v340: 345 pgs: 336 active+clean, 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:11.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:10.252293+0000 mon.a (mon.0) 1726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:10.252402+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:10.253176+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:10.253392+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:10.467108+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:10.571509+0000 mon.c (mon.2) 287 : audit [INF] from='client.? 172.21.15.87:0/2665523323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:10.571802+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:10.659059+0000 mon.c (mon.2) 288 : audit [DBG] from='client.? 172.21.15.87:0/4047409660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:11.012557+0000 mon.a (mon.0) 1730 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:11.012585+0000 mon.a (mon.0) 1731 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:11.016885+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi087-82011-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi087-82011-35"}]': finished 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:11.016971+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25"}]': finished 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:11.017059+0000 mon.a (mon.0) 1734 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:33:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:11.017115+0000 mon.a (mon.0) 1735 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:09.597397+0000 mgr.y (mgr.24437) 265 : cluster [DBG] pgmap v340: 345 pgs: 336 active+clean, 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:10.252293+0000 mon.a (mon.0) 1726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:10.252402+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:10.253176+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:10.253392+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:10.467108+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:10.571509+0000 mon.c (mon.2) 287 : audit [INF] from='client.? 172.21.15.87:0/2665523323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:10.571802+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:10.659059+0000 mon.c (mon.2) 288 : audit [DBG] from='client.? 172.21.15.87:0/4047409660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:11.012557+0000 mon.a (mon.0) 1730 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:11.012585+0000 mon.a (mon.0) 1731 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:11.016885+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi087-82011-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi087-82011-35"}]': finished 2024-02-23T00:33:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:11.016971+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25"}]': finished 2024-02-23T00:33:11.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:11.017059+0000 mon.a (mon.0) 1734 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:33:11.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:11.017115+0000 mon.a (mon.0) 1735 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-02-23T00:33:11.603 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:09.597397+0000 mgr.y (mgr.24437) 265 : cluster [DBG] pgmap v340: 345 pgs: 336 active+clean, 6 active+clean+snaptrim_wait, 3 active+clean+snaptrim; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:10.252293+0000 mon.a (mon.0) 1726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:10.252402+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-02-23T00:33:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:10.253176+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:10.253392+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25"}]: dispatch 2024-02-23T00:33:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:10.467108+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:11.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:10.571509+0000 mon.c (mon.2) 287 : audit [INF] from='client.? 172.21.15.87:0/2665523323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:10.571802+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:10.659059+0000 mon.c (mon.2) 288 : audit [DBG] from='client.? 172.21.15.87:0/4047409660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:11.012557+0000 mon.a (mon.0) 1730 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:11.012585+0000 mon.a (mon.0) 1731 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:11.016885+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi087-82011-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi087-82011-35"}]': finished 2024-02-23T00:33:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:11.016971+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-25"}]': finished 2024-02-23T00:33:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:11.017059+0000 mon.a (mon.0) 1734 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:33:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:11.017115+0000 mon.a (mon.0) 1735 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-02-23T00:33:12.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:33:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:12] "GET /metrics HTTP/1.1" 200 195897 "" "Prometheus/2.33.4" 2024-02-23T00:33:12.492 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:33:12.495 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:33:12.495 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.log 2024-02-23T00:33:12.495 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:33:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:10.749362+0000 mgr.y (mgr.24437) 266 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:11.467888+0000 mon.c (mon.2) 289 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:12.006015+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:33:12.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:12.011085+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-02-23T00:33:12.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:10.749362+0000 mgr.y (mgr.24437) 266 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:12.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:11.467888+0000 mon.c (mon.2) 289 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:12.006015+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:33:12.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:12.011085+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-02-23T00:33:12.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:10.749362+0000 mgr.y (mgr.24437) 266 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:12.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:11.467888+0000 mon.c (mon.2) 289 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:12.006015+0000 mon.a (mon.0) 1736 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:33:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:12.011085+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-02-23T00:33:13.083 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:13.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:13.188 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:13.215 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:13.365 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:13.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:13.469 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:13.497 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:13.570 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:13.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:13.594 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:33:13.594 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:33:13.674 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:13.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:11.597803+0000 mgr.y (mgr.24437) 267 : cluster [DBG] pgmap v343: 289 pgs: 8 unknown, 281 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-23T00:33:13.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:12.468605+0000 mon.c (mon.2) 290 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:13.021089+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:13.022651+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:13.031610+0000 mon.b (mon.1) 215 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:13.032968+0000 mon.b (mon.1) 216 : audit [INF] from='client.? 172.21.15.87:0/3091112879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi087-82019-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:13.033465+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:13.033894+0000 mon.a (mon.0) 1741 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi087-82019-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:13.469417+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:11.597803+0000 mgr.y (mgr.24437) 267 : cluster [DBG] pgmap v343: 289 pgs: 8 unknown, 281 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:12.468605+0000 mon.c (mon.2) 290 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:13.021089+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-02-23T00:33:13.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:13.022651+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:13.701 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:13.031610+0000 mon.b (mon.1) 215 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.701 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:13.032968+0000 mon.b (mon.1) 216 : audit [INF] from='client.? 172.21.15.87:0/3091112879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi087-82019-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.701 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:13.033465+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.701 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:13.033894+0000 mon.a (mon.0) 1741 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi087-82019-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.701 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:13.469417+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:13.702 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:13.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:33:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:11.597803+0000 mgr.y (mgr.24437) 267 : cluster [DBG] pgmap v343: 289 pgs: 8 unknown, 281 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-23T00:33:13.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:12.468605+0000 mon.c (mon.2) 290 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:13.021089+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-02-23T00:33:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:13.022651+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:13.031610+0000 mon.b (mon.1) 215 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:13.032968+0000 mon.b (mon.1) 216 : audit [INF] from='client.? 172.21.15.87:0/3091112879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi087-82019-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:13.033465+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:13.033894+0000 mon.a (mon.0) 1741 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi087-82019-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:13.469417+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:14.097 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:33:15.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:13.598205+0000 mgr.y (mgr.24437) 268 : cluster [DBG] pgmap v346: 281 pgs: 64 unknown, 217 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:15.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:14.017660+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi087-82011-35"}]': finished 2024-02-23T00:33:15.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:14.017883+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:15.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:14.017962+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi087-82019-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:15.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:14.018033+0000 mon.a (mon.0) 1745 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-02-23T00:33:15.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:14.018674+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:15.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:14.470262+0000 mon.c (mon.2) 292 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:15.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:13.598205+0000 mgr.y (mgr.24437) 268 : cluster [DBG] pgmap v346: 281 pgs: 64 unknown, 217 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:15.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:14.017660+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi087-82011-35"}]': finished 2024-02-23T00:33:15.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:14.017883+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:15.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:14.017962+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi087-82019-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:15.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:14.018033+0000 mon.a (mon.0) 1745 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-02-23T00:33:15.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:14.018674+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:15.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:14.470262+0000 mon.c (mon.2) 292 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:13.598205+0000 mgr.y (mgr.24437) 268 : cluster [DBG] pgmap v346: 281 pgs: 64 unknown, 217 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:14.017660+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi087-82011-35"}]': finished 2024-02-23T00:33:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:14.017883+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:14.017962+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi087-82019-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:14.018033+0000 mon.a (mon.0) 1745 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-02-23T00:33:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:14.018674+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi087-82011-35"}]: dispatch 2024-02-23T00:33:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:14.470262+0000 mon.c (mon.2) 292 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:16.005 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:15.027785+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi087-82011-35"}]': finished 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:15.027904+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:15.064138+0000 mon.a (mon.0) 1749 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:15.065758+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:15.066151+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:15.072901+0000 mon.b (mon.1) 217 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:15.309019+0000 mon.a (mon.0) 1752 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:15.471091+0000 mon.c (mon.2) 293 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:15.027785+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi087-82011-35"}]': finished 2024-02-23T00:33:16.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:15.027904+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-02-23T00:33:16.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:15.064138+0000 mon.a (mon.0) 1749 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:16.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:15.065758+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:16.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:15.066151+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:16.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:15.072901+0000 mon.b (mon.1) 217 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:16.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:15.309019+0000 mon.a (mon.0) 1752 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:16.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:15.471091+0000 mon.c (mon.2) 293 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:16.378 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:16.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:15.027785+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? 172.21.15.87:0/1360220292' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi087-82011-35"}]': finished 2024-02-23T00:33:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:15.027904+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-02-23T00:33:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:15.064138+0000 mon.a (mon.0) 1749 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:15.065758+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:15.066151+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:15.072901+0000 mon.b (mon.1) 217 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:16.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:15.309019+0000 mon.a (mon.0) 1752 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:16.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:15.471091+0000 mon.c (mon.2) 293 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:16.454 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:16.462 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:16.469 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:16.482 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:16.516 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:15.599162+0000 mgr.y (mgr.24437) 269 : cluster [DBG] pgmap v349: 249 pgs: 249 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.046113+0000 mon.a (mon.0) 1753 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.046211+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:16.046272+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.052486+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.87:0/850838230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi087-82019-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.053309+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.053757+0000 mon.a (mon.0) 1756 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi087-82011-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.054803+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi087-82019-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.056135+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:17.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.471944+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.472705+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:16.473038+0000 mon.a (mon.0) 1759 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:15.599162+0000 mgr.y (mgr.24437) 269 : cluster [DBG] pgmap v349: 249 pgs: 249 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.046113+0000 mon.a (mon.0) 1753 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.046211+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:16.046272+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.052486+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.87:0/850838230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi087-82019-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.053309+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.053757+0000 mon.a (mon.0) 1756 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi087-82011-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.054803+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi087-82019-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:17.275 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.056135+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:17.276 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.471944+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:17.276 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.472705+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:33:17.276 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:16.473038+0000 mon.a (mon.0) 1759 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:33:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:15.599162+0000 mgr.y (mgr.24437) 269 : cluster [DBG] pgmap v349: 249 pgs: 249 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-23T00:33:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.046113+0000 mon.a (mon.0) 1753 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.046211+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:16.046272+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-02-23T00:33:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.052486+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.87:0/850838230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi087-82019-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.053309+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.053757+0000 mon.a (mon.0) 1756 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi087-82011-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.054803+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi087-82019-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.056135+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.471944+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.472705+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:33:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:16.473038+0000 mon.a (mon.0) 1759 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]: dispatch 2024-02-23T00:33:17.765 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:17.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:17.869 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:17.900 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:17.999 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:18.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:18.103 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:18.130 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:18.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:18.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:18.271 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:18.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:17.256558+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi087-82019-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:18.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:17.256672+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-27"}]': finished 2024-02-23T00:33:18.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:17.256778+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]': finished 2024-02-23T00:33:18.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:17.256852+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-02-23T00:33:18.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:17.257762+0000 mon.c (mon.2) 296 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:18.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:17.258450+0000 mon.c (mon.2) 297 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:17.259132+0000 mon.c (mon.2) 298 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:17.259161+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-27", "mode": "writeback"}]: dispatch 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:17.262130+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-27", "mode": "writeback"}]: dispatch 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:17.269629+0000 mon.a (mon.0) 1765 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:18.011127+0000 mon.a (mon.0) 1766 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.017924+0000 mon.a (mon.0) 1767 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi087-82011-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi087-82011-36"}]': finished 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.018015+0000 mon.a (mon.0) 1768 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-27", "mode": "writeback"}]': finished 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:17.256558+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi087-82019-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:17.256672+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-27"}]': finished 2024-02-23T00:33:18.286 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:17.256778+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]': finished 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:17.256852+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:17.257762+0000 mon.c (mon.2) 296 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:17.258450+0000 mon.c (mon.2) 297 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:17.259132+0000 mon.c (mon.2) 298 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:17.259161+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-27", "mode": "writeback"}]: dispatch 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:17.262130+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-27", "mode": "writeback"}]: dispatch 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:17.269629+0000 mon.a (mon.0) 1765 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:18.011127+0000 mon.a (mon.0) 1766 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.017924+0000 mon.a (mon.0) 1767 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi087-82011-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi087-82011-36"}]': finished 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.018015+0000 mon.a (mon.0) 1768 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-27", "mode": "writeback"}]': finished 2024-02-23T00:33:18.287 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.018081+0000 mon.a (mon.0) 1769 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]': finished 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:18.018131+0000 mon.a (mon.0) 1770 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.019397+0000 mon.c (mon.2) 299 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.046075+0000 mon.c (mon.2) 300 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.046573+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.046974+0000 mon.c (mon.2) 301 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.047193+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.047565+0000 mon.c (mon.2) 302 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:18.047803+0000 mon.a (mon.0) 1773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.018081+0000 mon.a (mon.0) 1769 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]': finished 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:18.018131+0000 mon.a (mon.0) 1770 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.019397+0000 mon.c (mon.2) 299 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:18.288 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.046075+0000 mon.c (mon.2) 300 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.046573+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.046974+0000 mon.c (mon.2) 301 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.047193+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.047565+0000 mon.c (mon.2) 302 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:18.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:18.047803+0000 mon.a (mon.0) 1773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:18.303 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:18.325 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:18.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:18.372 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:33:18.428 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:18.455 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:17.256558+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi087-82019-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:17.256672+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-27"}]': finished 2024-02-23T00:33:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:17.256778+0000 mon.a (mon.0) 1762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pg_num","val":"11"}]': finished 2024-02-23T00:33:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:17.256852+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-02-23T00:33:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:17.257762+0000 mon.c (mon.2) 296 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:17.258450+0000 mon.c (mon.2) 297 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:17.259132+0000 mon.c (mon.2) 298 : audit [INF] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:17.259161+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-27", "mode": "writeback"}]: dispatch 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:17.262130+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-27", "mode": "writeback"}]: dispatch 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:17.269629+0000 mon.a (mon.0) 1765 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:18.011127+0000 mon.a (mon.0) 1766 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.017924+0000 mon.a (mon.0) 1767 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi087-82011-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi087-82011-36"}]': finished 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.018015+0000 mon.a (mon.0) 1768 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-27", "mode": "writeback"}]': finished 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.018081+0000 mon.a (mon.0) 1769 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi087-82051-1","var":"pgp_num","val":"11"}]': finished 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:18.018131+0000 mon.a (mon.0) 1770 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.019397+0000 mon.c (mon.2) 299 : audit [DBG] from='client.? 172.21.15.87:0/3778460189' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.046075+0000 mon.c (mon.2) 300 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.046573+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.046974+0000 mon.c (mon.2) 301 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.047193+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.047565+0000 mon.c (mon.2) 302 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:18.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:18.047803+0000 mon.a (mon.0) 1773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:18.804 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:18.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:18.909 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:18.942 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:18.963 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:18.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:19.067 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:19.095 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:19.114 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:19.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:19.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:19.246 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:19.274 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:19.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:19.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:19.378 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:19.406 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:19.432 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:19.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:19.536 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.535+0000 7fa144a8d700 1 -- 172.21.15.87:0/79670700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400fd510 msgr2=0x7fa1400f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.535+0000 7fa144a8d700 1 --2- 172.21.15.87:0/79670700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400fd510 0x7fa1400f6d40 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 rx=0x7fa134004870 tx=0x7fa134009cb0).stop 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.535+0000 7fa144a8d700 1 -- 172.21.15.87:0/79670700 shutdown_connections 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.535+0000 7fa144a8d700 1 --2- 172.21.15.87:0/79670700 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1400f7400 0x7fa140103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.535+0000 7fa144a8d700 1 --2- 172.21.15.87:0/79670700 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1400fcbb0 0x7fa1400fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.535+0000 7fa144a8d700 1 --2- 172.21.15.87:0/79670700 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400fd510 0x7fa1400f6d40 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.535+0000 7fa144a8d700 1 -- 172.21.15.87:0/79670700 >> 172.21.15.87:0/79670700 conn(0x7fa1400f2820 msgr2=0x7fa1400f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.535+0000 7fa144a8d700 1 -- 172.21.15.87:0/79670700 shutdown_connections 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.535+0000 7fa144a8d700 1 -- 172.21.15.87:0/79670700 wait complete. 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.536+0000 7fa144a8d700 1 librados: starting msgr at 2024-02-23T00:33:19.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.536+0000 7fa144a8d700 1 librados: starting objecter 2024-02-23T00:33:19.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.536+0000 7fa144a8d700 1 Processor -- start 2024-02-23T00:33:19.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.536+0000 7fa144a8d700 1 -- start start 2024-02-23T00:33:19.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.536+0000 7fa144a8d700 1 librados: setting wanted keys 2024-02-23T00:33:19.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.536+0000 7fa144a8d700 1 librados: calling monclient init 2024-02-23T00:33:19.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa144a8d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400f7400 0x7fa14010e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:19.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa144a8d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1400fcbb0 0x7fa14010e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:19.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa144a8d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1400fd510 0x7fa140108400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:19.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa144a8d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1401088f0 con 0x7fa1400f7400 2024-02-23T00:33:19.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa144a8d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa140108a60 con 0x7fa1400fd510 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa144a8d700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa140108c20 con 0x7fa1400fcbb0 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13dd9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1400fcbb0 0x7fa14010e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400f7400 0x7fa14010e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13dd9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1400fcbb0 0x7fa14010e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:46728/0 (socket says 172.21.15.87:46728) 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13dd9b700 1 -- 172.21.15.87:0/2405470840 learned_addr learned my addr 172.21.15.87:0/2405470840 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13e59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400f7400 0x7fa14010e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55676/0 (socket says 172.21.15.87:55676) 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13ed9d700 1 --2- 172.21.15.87:0/2405470840 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1400fd510 0x7fa140108400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13dd9b700 1 -- 172.21.15.87:0/2405470840 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1400fd510 msgr2=0x7fa140108400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13dd9b700 1 --2- 172.21.15.87:0/2405470840 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1400fd510 0x7fa140108400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13dd9b700 1 -- 172.21.15.87:0/2405470840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400f7400 msgr2=0x7fa14010e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13dd9b700 1 --2- 172.21.15.87:0/2405470840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400f7400 0x7fa14010e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13dd9b700 1 -- 172.21.15.87:0/2405470840 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa130005040 con 0x7fa1400fcbb0 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13e59c700 1 --2- 172.21.15.87:0/2405470840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400f7400 0x7fa14010e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:19.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa13dd9b700 1 --2- 172.21.15.87:0/2405470840 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1400fcbb0 0x7fa14010e750 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7fa1340043a0 tx=0x7fa134009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa12f7fe700 10 client.?.objecter ms_handle_connect 0x7fa1400fcbb0 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.537+0000 7fa12f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.538+0000 7fa12f7fe700 1 -- 172.21.15.87:0/2405470840 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa13400e030 con 0x7fa1400fcbb0 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.538+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa134005040 con 0x7fa1400fcbb0 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.538+0000 7fa12f7fe700 1 -- 172.21.15.87:0/2405470840 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa134013da0 con 0x7fa1400fcbb0 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.538+0000 7fa12f7fe700 1 -- 172.21.15.87:0/2405470840 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa134018070 con 0x7fa1400fcbb0 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.538+0000 7fa144a8d700 10 client.37247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.538+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa140109060 con 0x7fa1400fcbb0 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.538+0000 7fa144a8d700 1 librados: init done 2024-02-23T00:33:19.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.539+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14005fa60 con 0x7fa1400fcbb0 2024-02-23T00:33:19.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.539+0000 7fa12f7fe700 1 -- 172.21.15.87:0/2405470840 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fa13401d420 con 0x7fa1400fcbb0 2024-02-23T00:33:19.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.540+0000 7fa12f7fe700 1 --2- 172.21.15.87:0/2405470840 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa128062c90 0x7fa128065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:19.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.540+0000 7fa12f7fe700 1 -- 172.21.15.87:0/2405470840 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(248..248 src has 38..248) v4 ==== 9438+0+0 (secure 0 0 0) 0x7fa13407e890 con 0x7fa1400fcbb0 2024-02-23T00:33:19.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.540+0000 7fa12f7fe700 10 client.37247.objecter ms_dispatch 0x7fa140001550 osd_map(248..248 src has 38..248) v4 2024-02-23T00:33:19.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.540+0000 7fa12f7fe700 3 client.37247.objecter handle_osd_map got epochs [248,248] > 0 2024-02-23T00:33:19.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.540+0000 7fa12f7fe700 3 client.37247.objecter handle_osd_map decoding full epoch 248 2024-02-23T00:33:19.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.540+0000 7fa13e59c700 1 --2- 172.21.15.87:0/2405470840 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa128062c90 0x7fa128065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:19.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.540+0000 7fa12f7fe700 20 client.37247.objecter dump_active .. 0 homeless 2024-02-23T00:33:19.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.540+0000 7fa12f7fe700 10 client.37247.objecter ms_handle_connect 0x7fa128062c90 2024-02-23T00:33:19.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.540+0000 7fa13e59c700 1 --2- 172.21.15.87:0/2405470840 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa128062c90 0x7fa128065150 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 rx=0x7fa130009760 tx=0x7fa13000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:19.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.548+0000 7fa12f7fe700 1 -- 172.21.15.87:0/2405470840 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa134051800 con 0x7fa1400fcbb0 2024-02-23T00:33:19.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:17.599672+0000 mgr.y (mgr.24437) 270 : cluster [DBG] pgmap v352: 281 pgs: 32 unknown, 249 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-23T00:33:19.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:19.016856+0000 mon.a (mon.0) 1774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:19.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:19.016922+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-02-23T00:33:19.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:19.026518+0000 mon.c (mon.2) 303 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:19.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:19.026971+0000 mon.a (mon.0) 1776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:19.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:19.053389+0000 mon.b (mon.1) 221 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:19.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:19.053556+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:19.565 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:17.599672+0000 mgr.y (mgr.24437) 270 : cluster [DBG] pgmap v352: 281 pgs: 32 unknown, 249 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-23T00:33:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:19.016856+0000 mon.a (mon.0) 1774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:19.016922+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-02-23T00:33:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:19.026518+0000 mon.c (mon.2) 303 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:19.026971+0000 mon.a (mon.0) 1776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:19.053389+0000 mon.b (mon.1) 221 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:19.053556+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:17.599672+0000 mgr.y (mgr.24437) 270 : cluster [DBG] pgmap v352: 281 pgs: 32 unknown, 249 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-23T00:33:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:19.016856+0000 mon.a (mon.0) 1774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:19.016922+0000 mon.a (mon.0) 1775 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-02-23T00:33:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:19.026518+0000 mon.c (mon.2) 303 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:19.026971+0000 mon.a (mon.0) 1776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:19.053389+0000 mon.b (mon.1) 221 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:19.053556+0000 mon.a (mon.0) 1777 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:19.585 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:19.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:19.689 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:19.715 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.717+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa14005ef20 con 0x7fa128062c90 2024-02-23T00:33:19.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.736+0000 7fa12f7fe700 1 -- 172.21.15.87:0/2405470840 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+650968 (secure 0 0 0) 0x7fa14005ef20 con 0x7fa128062c90 2024-02-23T00:33:19.750 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:33:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 10 librados: watch_flush enter 2024-02-23T00:33:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 10 librados: watch_flush exit 2024-02-23T00:33:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 20 client.37247.objecter shutdown clearing up homeless session... 2024-02-23T00:33:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 10 client.37247.objecter successfully canceled tick 2024-02-23T00:33:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa128062c90 msgr2=0x7fa128065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 --2- 172.21.15.87:0/2405470840 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa128062c90 0x7fa128065150 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 rx=0x7fa130009760 tx=0x7fa13000b040).stop 2024-02-23T00:33:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1400fcbb0 msgr2=0x7fa14010e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 --2- 172.21.15.87:0/2405470840 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1400fcbb0 0x7fa14010e750 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7fa1340043a0 tx=0x7fa134009cb0).stop 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 shutdown_connections 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 --2- 172.21.15.87:0/2405470840 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa128062c90 0x7fa128065150 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 --2- 172.21.15.87:0/2405470840 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1400fd510 0x7fa140108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 --2- 172.21.15.87:0/2405470840 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1400f7400 0x7fa14010e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 --2- 172.21.15.87:0/2405470840 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1400fcbb0 0x7fa14010e750 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 >> 172.21.15.87:0/2405470840 conn(0x7fa1400f2820 msgr2=0x7fa1400f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 shutdown_connections 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 -- 172.21.15.87:0/2405470840 wait complete. 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:19.743+0000 7fa144a8d700 1 librados: shutdown 2024-02-23T00:33:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:33:20.414 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:20.765 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:20.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:20.884 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:20.917 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:20.937 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:20.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:20.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:20.995 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:33:21.026 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:33:21.027 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:33:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:33:21.042 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:21.072 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:21.115 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:21.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:21.219 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:21.249 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:21.268 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:21.270 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:19.600036+0000 mgr.y (mgr.24437) 271 : cluster [DBG] pgmap v355: 257 pgs: 8 unknown, 249 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:21.270 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:19.719319+0000 mgr.y (mgr.24437) 272 : audit [DBG] from='client.37247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:33:21.270 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:20.256549+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:21.270 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:20.256641+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-02-23T00:33:21.270 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:20.257362+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:21.271 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:20.257559+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:21.271 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:20.257770+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:21.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:21.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:19.600036+0000 mgr.y (mgr.24437) 271 : cluster [DBG] pgmap v355: 257 pgs: 8 unknown, 249 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:19.719319+0000 mgr.y (mgr.24437) 272 : audit [DBG] from='client.37247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:33:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:20.256549+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:20.256641+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-02-23T00:33:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:20.257362+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:20.257559+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:21.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:20.257770+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:21.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:19.600036+0000 mgr.y (mgr.24437) 271 : cluster [DBG] pgmap v355: 257 pgs: 8 unknown, 249 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:21.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:19.719319+0000 mgr.y (mgr.24437) 272 : audit [DBG] from='client.37247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:33:21.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:20.256549+0000 mon.a (mon.0) 1778 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:21.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:20.256641+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-02-23T00:33:21.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:20.257362+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:21.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:20.257559+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:21.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:20.257770+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27"}]: dispatch 2024-02-23T00:33:21.372 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:21.400 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:21.419 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:21.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:21.523 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:21.551 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:21.568 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:21.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:21.672 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:21.702 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:22.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:22] "GET /metrics HTTP/1.1" 200 195897 "" "Prometheus/2.33.4" 2024-02-23T00:33:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:20.754776+0000 mgr.y (mgr.24437) 273 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:21.017402+0000 mon.a (mon.0) 1782 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:21.025535+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-45"}]': finished 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:21.025635+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36"}]': finished 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:21.025688+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27"}]': finished 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:21.025747+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:21.027217+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:22.015623+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi087-82011-36"}]': finished 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:22.015803+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:20.754776+0000 mgr.y (mgr.24437) 273 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:21.017402+0000 mon.a (mon.0) 1782 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:21.025535+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-45"}]': finished 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:21.025635+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36"}]': finished 2024-02-23T00:33:22.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:21.025688+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27"}]': finished 2024-02-23T00:33:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:21.025747+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-02-23T00:33:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:21.027217+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:22.015623+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi087-82011-36"}]': finished 2024-02-23T00:33:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:22.015803+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-02-23T00:33:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:20.754776+0000 mgr.y (mgr.24437) 273 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:21.017402+0000 mon.a (mon.0) 1782 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:21.025535+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-45"}]': finished 2024-02-23T00:33:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:21.025635+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi087-82011-36"}]': finished 2024-02-23T00:33:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:21.025688+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-27"}]': finished 2024-02-23T00:33:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:21.025747+0000 mon.a (mon.0) 1786 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-02-23T00:33:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:21.027217+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi087-82011-36"}]: dispatch 2024-02-23T00:33:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:22.015623+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? 172.21.15.87:0/2316897923' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi087-82011-36"}]': finished 2024-02-23T00:33:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:22.015803+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-02-23T00:33:23.412 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-02-23T00:33:19.599810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9075,"num_objects":364,"num_object_clones":5,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":364,"num_whiteouts":1,"num_read":1242,"num_read_kb":875,"num_write":1291,"num_write_kb":653,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2513,"ondisk_log_size":2513,"up":747,"acting":747,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":800,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2491700,"kb_used_data":31932,"kb_used_omap":0,"kb_used_meta":2459648,"kb_avail":747469516,"statfs":{"total":767960285184,"available":765408784384,"internally_reserved":0,"allocated":32698368,"data_stored":17932761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2518679552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001396"},"pg_stats":[{"pgid":"164.1f","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.106706+0000","last_change":"2024-02-23T00:33:14.105553+0000","last_active":"2024-02-23T00:33:14.106706+0000","last_peered":"2024-02-23T00:33:14.106706+0000","last_clean":"2024-02-23T00:33:14.106706+0000","last_became_active":"2024-02-23T00:33:14.105225+0000","last_became_peered":"2024-02-23T00:33:14.105225+0000","last_unstale":"2024-02-23T00:33:14.106706+0000","last_undegraded":"2024-02-23T00:33:14.106706+0000","last_fullsized":"2024-02-23T00:33:14.106706+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"164.1e","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.102596+0000","last_change":"2024-02-23T00:33:14.102392+0000","last_active":"2024-02-23T00:33:14.102596+0000","last_peered":"2024-02-23T00:33:14.102596+0000","last_clean":"2024-02-23T00:33:14.102596+0000","last_became_active":"2024-02-23T00:33:14.102098+0000","last_became_peered":"2024-02-23T00:33:14.102098+0000","last_unstale":"2024-02-23T00:33:14.102596+0000","last_undegraded":"2024-02-23T00:33:14.102596+0000","last_fullsized":"2024-02-23T00:33:14.102596+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"164.1d","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.092462+0000","last_change":"2024-02-23T00:33:14.091395+0000","last_active":"2024-02-23T00:33:14.092462+0000","last_peered":"2024-02-23T00:33:14.092462+0000","last_clean":"2024-02-23T00:33:14.092462+0000","last_became_active":"2024-02-23T00:33:14.091091+0000","last_became_peered":"2024-02-23T00:33:14.091091+0000","last_unstale":"2024-02-23T00:33:14.092462+0000","last_undegraded":"2024-02-23T00:33:14.092462+0000","last_fullsized":"2024-02-23T00:33:14.092462+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"164.1c","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.102743+0000","last_change":"2024-02-23T00:33:14.096960+0000","last_active":"2024-02-23T00:33:14.102743+0000","last_peered":"2024-02-23T00:33:14.102743+0000","last_clean":"2024-02-23T00:33:14.102743+0000","last_became_active":"2024-02-23T00:33:14.096671+0000","last_became_peered":"2024-02-23T00:33:14.096671+0000","last_unstale":"2024-02-23T00:33:14.102743+0000","last_undegraded":"2024-02-23T00:33:14.102743+0000","last_fullsized":"2024-02-23T00:33:14.102743+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.1b","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.105616+0000","last_change":"2024-02-23T00:33:14.103402+0000","last_active":"2024-02-23T00:33:14.105616+0000","last_peered":"2024-02-23T00:33:14.105616+0000","last_clean":"2024-02-23T00:33:14.105616+0000","last_became_active":"2024-02-23T00:33:14.103085+0000","last_became_peered":"2024-02-23T00:33:14.103085+0000","last_unstale":"2024-02-23T00:33:14.105616+0000","last_undegraded":"2024-02-23T00:33:14.105616+0000","last_fullsized":"2024-02-23T00:33:14.105616+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"164.1a","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.104873+0000","last_change":"2024-02-23T00:33:14.100757+0000","last_active":"2024-02-23T00:33:14.104873+0000","last_peered":"2024-02-23T00:33:14.104873+0000","last_clean":"2024-02-23T00:33:14.104873+0000","last_became_active":"2024-02-23T00:33:14.100083+0000","last_became_peered":"2024-02-23T00:33:14.100083+0000","last_unstale":"2024-02-23T00:33:14.104873+0000","last_undegraded":"2024-02-23T00:33:14.104873+0000","last_fullsized":"2024-02-23T00:33:14.104873+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"164.19","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.102591+0000","last_change":"2024-02-23T00:33:14.097638+0000","last_active":"2024-02-23T00:33:14.102591+0000","last_peered":"2024-02-23T00:33:14.102591+0000","last_clean":"2024-02-23T00:33:14.102591+0000","last_became_active":"2024-02-23T00:33:14.097245+0000","last_became_peered":"2024-02-23T00:33:14.097245+0000","last_unstale":"2024-02-23T00:33:14.102591+0000","last_undegraded":"2024-02-23T00:33:14.102591+0000","last_fullsized":"2024-02-23T00:33:14.102591+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.18","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.099356+0000","last_change":"2024-02-23T00:33:14.093709+0000","last_active":"2024-02-23T00:33:14.099356+0000","last_peered":"2024-02-23T00:33:14.099356+0000","last_clean":"2024-02-23T00:33:14.099356+0000","last_became_active":"2024-02-23T00:33:14.093308+0000","last_became_peered":"2024-02-23T00:33:14.093308+0000","last_unstale":"2024-02-23T00:33:14.099356+0000","last_undegraded":"2024-02-23T00:33:14.099356+0000","last_fullsized":"2024-02-23T00:33:14.099356+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.17","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.104924+0000","last_change":"2024-02-23T00:33:14.101346+0000","last_active":"2024-02-23T00:33:14.104924+0000","last_peered":"2024-02-23T00:33:14.104924+0000","last_clean":"2024-02-23T00:33:14.104924+0000","last_became_active":"2024-02-23T00:33:14.101034+0000","last_became_peered":"2024-02-23T00:33:14.101034+0000","last_unstale":"2024-02-23T00:33:14.104924+0000","last_undegraded":"2024-02-23T00:33:14.104924+0000","last_fullsized":"2024-02-23T00:33:14.104924+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"164.16","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.099554+0000","last_change":"2024-02-23T00:33:14.096941+0000","last_active":"2024-02-23T00:33:14.099554+0000","last_peered":"2024-02-23T00:33:14.099554+0000","last_clean":"2024-02-23T00:33:14.099554+0000","last_became_active":"2024-02-23T00:33:14.096641+0000","last_became_peered":"2024-02-23T00:33:14.096641+0000","last_unstale":"2024-02-23T00:33:14.099554+0000","last_undegraded":"2024-02-23T00:33:14.099554+0000","last_fullsized":"2024-02-23T00:33:14.099554+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"164.15","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.107231+0000","last_change":"2024-02-23T00:33:14.106255+0000","last_active":"2024-02-23T00:33:14.107231+0000","last_peered":"2024-02-23T00:33:14.107231+0000","last_clean":"2024-02-23T00:33:14.107231+0000","last_became_active":"2024-02-23T00:33:14.105968+0000","last_became_peered":"2024-02-23T00:33:14.105968+0000","last_unstale":"2024-02-23T00:33:14.107231+0000","last_undegraded":"2024-02-23T00:33:14.107231+0000","last_fullsized":"2024-02-23T00:33:14.107231+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.14","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.094675+0000","last_change":"2024-02-23T00:33:14.094399+0000","last_active":"2024-02-23T00:33:14.094675+0000","last_peered":"2024-02-23T00:33:14.094675+0000","last_clean":"2024-02-23T00:33:14.094675+0000","last_became_active":"2024-02-23T00:33:14.094103+0000","last_became_peered":"2024-02-23T00:33:14.094103+0000","last_unstale":"2024-02-23T00:33:14.094675+0000","last_undegraded":"2024-02-23T00:33:14.094675+0000","last_fullsized":"2024-02-23T00:33:14.094675+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"164.13","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.100457+0000","last_change":"2024-02-23T00:33:14.099875+0000","last_active":"2024-02-23T00:33:14.100457+0000","last_peered":"2024-02-23T00:33:14.100457+0000","last_clean":"2024-02-23T00:33:14.100457+0000","last_became_active":"2024-02-23T00:33:14.099261+0000","last_became_peered":"2024-02-23T00:33:14.099261+0000","last_unstale":"2024-02-23T00:33:14.100457+0000","last_undegraded":"2024-02-23T00:33:14.100457+0000","last_fullsized":"2024-02-23T00:33:14.100457+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"164.12","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091981+0000","last_change":"2024-02-23T00:33:14.090422+0000","last_active":"2024-02-23T00:33:14.091981+0000","last_peered":"2024-02-23T00:33:14.091981+0000","last_clean":"2024-02-23T00:33:14.091981+0000","last_became_active":"2024-02-23T00:33:14.088969+0000","last_became_peered":"2024-02-23T00:33:14.088969+0000","last_unstale":"2024-02-23T00:33:14.091981+0000","last_undegraded":"2024-02-23T00:33:14.091981+0000","last_fullsized":"2024-02-23T00:33:14.091981+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"164.11","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.102572+0000","last_change":"2024-02-23T00:33:14.096098+0000","last_active":"2024-02-23T00:33:14.102572+0000","last_peered":"2024-02-23T00:33:14.102572+0000","last_clean":"2024-02-23T00:33:14.102572+0000","last_became_active":"2024-02-23T00:33:14.095707+0000","last_became_peered":"2024-02-23T00:33:14.095707+0000","last_unstale":"2024-02-23T00:33:14.102572+0000","last_undegraded":"2024-02-23T00:33:14.102572+0000","last_fullsized":"2024-02-23T00:33:14.102572+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.10","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.095272+0000","last_change":"2024-02-23T00:33:14.093219+0000","last_active":"2024-02-23T00:33:14.095272+0000","last_peered":"2024-02-23T00:33:14.095272+0000","last_clean":"2024-02-23T00:33:14.095272+0000","last_became_active":"2024-02-23T00:33:14.092927+0000","last_became_peered":"2024-02-23T00:33:14.092927+0000","last_unstale":"2024-02-23T00:33:14.095272+0000","last_undegraded":"2024-02-23T00:33:14.095272+0000","last_fullsized":"2024-02-23T00:33:14.095272+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"164.f","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.107013+0000","last_change":"2024-02-23T00:33:14.105491+0000","last_active":"2024-02-23T00:33:14.107013+0000","last_peered":"2024-02-23T00:33:14.107013+0000","last_clean":"2024-02-23T00:33:14.107013+0000","last_became_active":"2024-02-23T00:33:14.105202+0000","last_became_peered":"2024-02-23T00:33:14.105202+0000","last_unstale":"2024-02-23T00:33:14.107013+0000","last_undegraded":"2024-02-23T00:33:14.107013+0000","last_fullsized":"2024-02-23T00:33:14.107013+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"164.e","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.104853+0000","last_change":"2024-02-23T00:33:14.103086+0000","last_active":"2024-02-23T00:33:14.104853+0000","last_peered":"2024-02-23T00:33:14.104853+0000","last_clean":"2024-02-23T00:33:14.104853+0000","last_became_active":"2024-02-23T00:33:14.102525+0000","last_became_peered":"2024-02-23T00:33:14.102525+0000","last_unstale":"2024-02-23T00:33:14.104853+0000","last_undegraded":"2024-02-23T00:33:14.104853+0000","last_fullsized":"2024-02-23T00:33:14.104853+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"164.d","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.092947+0000","last_change":"2024-02-23T00:33:14.092362+0000","last_active":"2024-02-23T00:33:14.092947+0000","last_peered":"2024-02-23T00:33:14.092947+0000","last_clean":"2024-02-23T00:33:14.092947+0000","last_became_active":"2024-02-23T00:33:14.092031+0000","last_became_peered":"2024-02-23T00:33:14.092031+0000","last_unstale":"2024-02-23T00:33:14.092947+0000","last_undegraded":"2024-02-23T00:33:14.092947+0000","last_fullsized":"2024-02-23T00:33:14.092947+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"164.c","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.101721+0000","last_change":"2024-02-23T00:33:14.097269+0000","last_active":"2024-02-23T00:33:14.101721+0000","last_peered":"2024-02-23T00:33:14.101721+0000","last_clean":"2024-02-23T00:33:14.101721+0000","last_became_active":"2024-02-23T00:33:14.096985+0000","last_became_peered":"2024-02-23T00:33:14.096985+0000","last_unstale":"2024-02-23T00:33:14.101721+0000","last_undegraded":"2024-02-23T00:33:14.101721+0000","last_fullsized":"2024-02-23T00:33:14.101721+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"164.b","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.101823+0000","last_change":"2024-02-23T00:33:14.100846+0000","last_active":"2024-02-23T00:33:14.101823+0000","last_peered":"2024-02-23T00:33:14.101823+0000","last_clean":"2024-02-23T00:33:14.101823+0000","last_became_active":"2024-02-23T00:33:14.099342+0000","last_became_peered":"2024-02-23T00:33:14.099342+0000","last_unstale":"2024-02-23T00:33:14.101823+0000","last_undegraded":"2024-02-23T00:33:14.101823+0000","last_fullsized":"2024-02-23T00:33:14.101823+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.a","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.095467+0000","last_change":"2024-02-23T00:33:14.093486+0000","last_active":"2024-02-23T00:33:14.095467+0000","last_peered":"2024-02-23T00:33:14.095467+0000","last_clean":"2024-02-23T00:33:14.095467+0000","last_became_active":"2024-02-23T00:33:14.092823+0000","last_became_peered":"2024-02-23T00:33:14.092823+0000","last_unstale":"2024-02-23T00:33:14.095467+0000","last_undegraded":"2024-02-23T00:33:14.095467+0000","last_fullsized":"2024-02-23T00:33:14.095467+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"164.9","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.107718+0000","last_change":"2024-02-23T00:33:14.105588+0000","last_active":"2024-02-23T00:33:14.107718+0000","last_peered":"2024-02-23T00:33:14.107718+0000","last_clean":"2024-02-23T00:33:14.107718+0000","last_became_active":"2024-02-23T00:33:14.105279+0000","last_became_peered":"2024-02-23T00:33:14.105279+0000","last_unstale":"2024-02-23T00:33:14.107718+0000","last_undegraded":"2024-02-23T00:33:14.107718+0000","last_fullsized":"2024-02-23T00:33:14.107718+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"164.8","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.100544+0000","last_change":"2024-02-23T00:33:14.098965+0000","last_active":"2024-02-23T00:33:14.100544+0000","last_peered":"2024-02-23T00:33:14.100544+0000","last_clean":"2024-02-23T00:33:14.100544+0000","last_became_active":"2024-02-23T00:33:14.098669+0000","last_became_peered":"2024-02-23T00:33:14.098669+0000","last_unstale":"2024-02-23T00:33:14.100544+0000","last_undegraded":"2024-02-23T00:33:14.100544+0000","last_fullsized":"2024-02-23T00:33:14.100544+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"167.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:18.010767+0000","last_change":"2024-02-23T00:33:18.010767+0000","last_active":"2024-02-23T00:33:18.010767+0000","last_peered":"2024-02-23T00:33:18.010767+0000","last_clean":"2024-02-23T00:33:18.010767+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:18.010767+0000","last_undegraded":"2024-02-23T00:33:18.010767+0000","last_fullsized":"2024-02-23T00:33:18.010767+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_clean_scrub_stamp":"2024-02-23T00:33:18.010767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.3","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.089892+0000","last_change":"2024-02-23T00:33:14.089659+0000","last_active":"2024-02-23T00:33:14.089892+0000","last_peered":"2024-02-23T00:33:14.089892+0000","last_clean":"2024-02-23T00:33:14.089892+0000","last_became_active":"2024-02-23T00:33:14.089317+0000","last_became_peered":"2024-02-23T00:33:14.089317+0000","last_unstale":"2024-02-23T00:33:14.089892+0000","last_undegraded":"2024-02-23T00:33:14.089892+0000","last_fullsized":"2024-02-23T00:33:14.089892+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"167.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:18.010767+0000","last_change":"2024-02-23T00:33:18.010767+0000","last_active":"2024-02-23T00:33:18.010767+0000","last_peered":"2024-02-23T00:33:18.010767+0000","last_clean":"2024-02-23T00:33:18.010767+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:18.010767+0000","last_undegraded":"2024-02-23T00:33:18.010767+0000","last_fullsized":"2024-02-23T00:33:18.010767+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_clean_scrub_stamp":"2024-02-23T00:33:18.010767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.2","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.104920+0000","last_change":"2024-02-23T00:33:14.100910+0000","last_active":"2024-02-23T00:33:14.104920+0000","last_peered":"2024-02-23T00:33:14.104920+0000","last_clean":"2024-02-23T00:33:14.104920+0000","last_became_active":"2024-02-23T00:33:14.100558+0000","last_became_peered":"2024-02-23T00:33:14.100558+0000","last_unstale":"2024-02-23T00:33:14.104920+0000","last_undegraded":"2024-02-23T00:33:14.104920+0000","last_fullsized":"2024-02-23T00:33:14.104920+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"167.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:18.010767+0000","last_change":"2024-02-23T00:33:18.010767+0000","last_active":"2024-02-23T00:33:18.010767+0000","last_peered":"2024-02-23T00:33:18.010767+0000","last_clean":"2024-02-23T00:33:18.010767+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:18.010767+0000","last_undegraded":"2024-02-23T00:33:18.010767+0000","last_fullsized":"2024-02-23T00:33:18.010767+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_clean_scrub_stamp":"2024-02-23T00:33:18.010767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.1","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.102533+0000","last_change":"2024-02-23T00:33:14.094928+0000","last_active":"2024-02-23T00:33:14.102533+0000","last_peered":"2024-02-23T00:33:14.102533+0000","last_clean":"2024-02-23T00:33:14.102533+0000","last_became_active":"2024-02-23T00:33:14.094269+0000","last_became_peered":"2024-02-23T00:33:14.094269+0000","last_unstale":"2024-02-23T00:33:14.102533+0000","last_undegraded":"2024-02-23T00:33:14.102533+0000","last_fullsized":"2024-02-23T00:33:14.102533+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"167.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:18.010767+0000","last_change":"2024-02-23T00:33:18.010767+0000","last_active":"2024-02-23T00:33:18.010767+0000","last_peered":"2024-02-23T00:33:18.010767+0000","last_clean":"2024-02-23T00:33:18.010767+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:18.010767+0000","last_undegraded":"2024-02-23T00:33:18.010767+0000","last_fullsized":"2024-02-23T00:33:18.010767+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_clean_scrub_stamp":"2024-02-23T00:33:18.010767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.0","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.105492+0000","last_change":"2024-02-23T00:33:14.100973+0000","last_active":"2024-02-23T00:33:14.105492+0000","last_peered":"2024-02-23T00:33:14.105492+0000","last_clean":"2024-02-23T00:33:14.105492+0000","last_became_active":"2024-02-23T00:33:14.100647+0000","last_became_peered":"2024-02-23T00:33:14.100647+0000","last_unstale":"2024-02-23T00:33:14.105492+0000","last_undegraded":"2024-02-23T00:33:14.105492+0000","last_fullsized":"2024-02-23T00:33:14.105492+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"167.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:18.010767+0000","last_change":"2024-02-23T00:33:18.010767+0000","last_active":"2024-02-23T00:33:18.010767+0000","last_peered":"2024-02-23T00:33:18.010767+0000","last_clean":"2024-02-23T00:33:18.010767+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:18.010767+0000","last_undegraded":"2024-02-23T00:33:18.010767+0000","last_fullsized":"2024-02-23T00:33:18.010767+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_clean_scrub_stamp":"2024-02-23T00:33:18.010767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.7","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.106994+0000","last_change":"2024-02-23T00:33:14.105486+0000","last_active":"2024-02-23T00:33:14.106994+0000","last_peered":"2024-02-23T00:33:14.106994+0000","last_clean":"2024-02-23T00:33:14.106994+0000","last_became_active":"2024-02-23T00:33:14.105184+0000","last_became_peered":"2024-02-23T00:33:14.105184+0000","last_unstale":"2024-02-23T00:33:14.106994+0000","last_undegraded":"2024-02-23T00:33:14.106994+0000","last_fullsized":"2024-02-23T00:33:14.106994+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"167.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:18.010767+0000","last_change":"2024-02-23T00:33:18.010767+0000","last_active":"2024-02-23T00:33:18.010767+0000","last_peered":"2024-02-23T00:33:18.010767+0000","last_clean":"2024-02-23T00:33:18.010767+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:18.010767+0000","last_undegraded":"2024-02-23T00:33:18.010767+0000","last_fullsized":"2024-02-23T00:33:18.010767+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_clean_scrub_stamp":"2024-02-23T00:33:18.010767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.6","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.105695+0000","last_change":"2024-02-23T00:33:14.104212+0000","last_active":"2024-02-23T00:33:14.105695+0000","last_peered":"2024-02-23T00:33:14.105695+0000","last_clean":"2024-02-23T00:33:14.105695+0000","last_became_active":"2024-02-23T00:33:14.103797+0000","last_became_peered":"2024-02-23T00:33:14.103797+0000","last_unstale":"2024-02-23T00:33:14.105695+0000","last_undegraded":"2024-02-23T00:33:14.105695+0000","last_fullsized":"2024-02-23T00:33:14.105695+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"167.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:18.010767+0000","last_change":"2024-02-23T00:33:18.010767+0000","last_active":"2024-02-23T00:33:18.010767+0000","last_peered":"2024-02-23T00:33:18.010767+0000","last_clean":"2024-02-23T00:33:18.010767+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:18.010767+0000","last_undegraded":"2024-02-23T00:33:18.010767+0000","last_fullsized":"2024-02-23T00:33:18.010767+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_clean_scrub_stamp":"2024-02-23T00:33:18.010767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.5","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.033005+0000","last_change":"2024-02-23T00:33:14.032010+0000","last_active":"2024-02-23T00:33:14.033005+0000","last_peered":"2024-02-23T00:33:14.033005+0000","last_clean":"2024-02-23T00:33:14.033005+0000","last_became_active":"2024-02-23T00:33:14.031701+0000","last_became_peered":"2024-02-23T00:33:14.031701+0000","last_unstale":"2024-02-23T00:33:14.033005+0000","last_undegraded":"2024-02-23T00:33:14.033005+0000","last_fullsized":"2024-02-23T00:33:14.033005+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"167.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:18.010767+0000","last_change":"2024-02-23T00:33:18.010767+0000","last_active":"2024-02-23T00:33:18.010767+0000","last_peered":"2024-02-23T00:33:18.010767+0000","last_clean":"2024-02-23T00:33:18.010767+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:18.010767+0000","last_undegraded":"2024-02-23T00:33:18.010767+0000","last_fullsized":"2024-02-23T00:33:18.010767+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:18.010767+0000","last_clean_scrub_stamp":"2024-02-23T00:33:18.010767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.4","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.107302+0000","last_change":"2024-02-23T00:33:14.106514+0000","last_active":"2024-02-23T00:33:14.107302+0000","last_peered":"2024-02-23T00:33:14.107302+0000","last_clean":"2024-02-23T00:33:14.107302+0000","last_became_active":"2024-02-23T00:33:14.105846+0000","last_became_peered":"2024-02-23T00:33:14.105846+0000","last_unstale":"2024-02-23T00:33:14.107302+0000","last_undegraded":"2024-02-23T00:33:14.107302+0000","last_fullsized":"2024-02-23T00:33:14.107302+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:13.008272+0000","last_clean_scrub_stamp":"2024-02-23T00:33:13.008272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.0","version":"222'68","reported_seq":227,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084247+0000","last_change":"2024-02-23T00:33:02.043252+0000","last_active":"2024-02-23T00:33:14.084247+0000","last_peered":"2024-02-23T00:33:14.084247+0000","last_clean":"2024-02-23T00:33:14.084247+0000","last_became_active":"2024-02-23T00:31:06.330477+0000","last_became_peered":"2024-02-23T00:31:06.330477+0000","last_unstale":"2024-02-23T00:33:14.084247+0000","last_undegraded":"2024-02-23T00:33:14.084247+0000","last_fullsized":"2024-02-23T00:33:14.084247+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1","version":"226'85","reported_seq":240,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.018025+0000","last_change":"2024-02-23T00:33:00.024266+0000","last_active":"2024-02-23T00:33:14.018025+0000","last_peered":"2024-02-23T00:33:14.018025+0000","last_clean":"2024-02-23T00:33:14.018025+0000","last_became_active":"2024-02-23T00:31:06.258140+0000","last_became_peered":"2024-02-23T00:31:06.258140+0000","last_unstale":"2024-02-23T00:33:14.018025+0000","last_undegraded":"2024-02-23T00:33:14.018025+0000","last_fullsized":"2024-02-23T00:33:14.018025+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.2","version":"216'18","reported_seq":186,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091323+0000","last_change":"2024-02-23T00:33:02.043677+0000","last_active":"2024-02-23T00:33:14.091323+0000","last_peered":"2024-02-23T00:33:14.091323+0000","last_clean":"2024-02-23T00:33:14.091323+0000","last_became_active":"2024-02-23T00:31:06.329726+0000","last_became_peered":"2024-02-23T00:31:06.329726+0000","last_unstale":"2024-02-23T00:33:14.091323+0000","last_undegraded":"2024-02-23T00:33:14.091323+0000","last_fullsized":"2024-02-23T00:33:14.091323+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.3","version":"244'23","reported_seq":191,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.036063+0000","last_change":"2024-02-23T00:33:07.138945+0000","last_active":"2024-02-23T00:33:15.036063+0000","last_peered":"2024-02-23T00:33:15.036063+0000","last_clean":"2024-02-23T00:33:15.036063+0000","last_became_active":"2024-02-23T00:31:06.258997+0000","last_became_peered":"2024-02-23T00:31:06.258997+0000","last_unstale":"2024-02-23T00:33:15.036063+0000","last_undegraded":"2024-02-23T00:33:15.036063+0000","last_fullsized":"2024-02-23T00:33:15.036063+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.4","version":"236'39","reported_seq":200,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.025012+0000","last_change":"2024-02-23T00:33:09.140491+0000","last_active":"2024-02-23T00:33:14.025012+0000","last_peered":"2024-02-23T00:33:14.025012+0000","last_clean":"2024-02-23T00:33:14.025012+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:33:14.025012+0000","last_undegraded":"2024-02-23T00:33:14.025012+0000","last_fullsized":"2024-02-23T00:33:14.025012+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.5","version":"234'124","reported_seq":278,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.020855+0000","last_change":"2024-02-23T00:33:07.173887+0000","last_active":"2024-02-23T00:33:14.020855+0000","last_peered":"2024-02-23T00:33:14.020855+0000","last_clean":"2024-02-23T00:33:14.020855+0000","last_became_active":"2024-02-23T00:31:06.258422+0000","last_became_peered":"2024-02-23T00:31:06.258422+0000","last_unstale":"2024-02-23T00:33:14.020855+0000","last_undegraded":"2024-02-23T00:33:14.020855+0000","last_fullsized":"2024-02-23T00:33:14.020855+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":124,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.6","version":"206'51","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084280+0000","last_change":"2024-02-23T00:33:00.032676+0000","last_active":"2024-02-23T00:33:14.084280+0000","last_peered":"2024-02-23T00:33:14.084280+0000","last_clean":"2024-02-23T00:33:14.084280+0000","last_became_active":"2024-02-23T00:31:06.313441+0000","last_became_peered":"2024-02-23T00:31:06.313441+0000","last_unstale":"2024-02-23T00:33:14.084280+0000","last_undegraded":"2024-02-23T00:33:14.084280+0000","last_fullsized":"2024-02-23T00:33:14.084280+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:32:48.841846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.841846+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.7","version":"229'92","reported_seq":246,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.090145+0000","last_change":"2024-02-23T00:33:02.043270+0000","last_active":"2024-02-23T00:33:14.090145+0000","last_peered":"2024-02-23T00:33:14.090145+0000","last_clean":"2024-02-23T00:33:14.090145+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:33:14.090145+0000","last_undegraded":"2024-02-23T00:33:14.090145+0000","last_fullsized":"2024-02-23T00:33:14.090145+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.8","version":"219'70","reported_seq":233,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.017045+0000","last_change":"2024-02-23T00:33:00.035333+0000","last_active":"2024-02-23T00:33:14.017045+0000","last_peered":"2024-02-23T00:33:14.017045+0000","last_clean":"2024-02-23T00:33:14.017045+0000","last_became_active":"2024-02-23T00:31:06.363442+0000","last_became_peered":"2024-02-23T00:31:06.363442+0000","last_unstale":"2024-02-23T00:33:14.017045+0000","last_undegraded":"2024-02-23T00:33:14.017045+0000","last_fullsized":"2024-02-23T00:33:14.017045+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.9","version":"208'17","reported_seq":184,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.089330+0000","last_change":"2024-02-23T00:33:00.033893+0000","last_active":"2024-02-23T00:33:14.089330+0000","last_peered":"2024-02-23T00:33:14.089330+0000","last_clean":"2024-02-23T00:33:14.089330+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:33:14.089330+0000","last_undegraded":"2024-02-23T00:33:14.089330+0000","last_fullsized":"2024-02-23T00:33:14.089330+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.a","version":"244'20","reported_seq":187,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.033620+0000","last_change":"2024-02-23T00:33:07.137265+0000","last_active":"2024-02-23T00:33:15.033620+0000","last_peered":"2024-02-23T00:33:15.033620+0000","last_clean":"2024-02-23T00:33:15.033620+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:33:15.033620+0000","last_undegraded":"2024-02-23T00:33:15.033620+0000","last_fullsized":"2024-02-23T00:33:15.033620+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.b","version":"202'34","reported_seq":198,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.033223+0000","last_change":"2024-02-23T00:33:00.037998+0000","last_active":"2024-02-23T00:33:14.033223+0000","last_peered":"2024-02-23T00:33:14.033223+0000","last_clean":"2024-02-23T00:33:14.033223+0000","last_became_active":"2024-02-23T00:31:06.294667+0000","last_became_peered":"2024-02-23T00:31:06.294667+0000","last_unstale":"2024-02-23T00:33:14.033223+0000","last_undegraded":"2024-02-23T00:33:14.033223+0000","last_fullsized":"2024-02-23T00:33:14.033223+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.c","version":"208'68","reported_seq":224,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027454+0000","last_change":"2024-02-23T00:33:00.031268+0000","last_active":"2024-02-23T00:33:14.027454+0000","last_peered":"2024-02-23T00:33:14.027454+0000","last_clean":"2024-02-23T00:33:14.027454+0000","last_became_active":"2024-02-23T00:31:06.233844+0000","last_became_peered":"2024-02-23T00:31:06.233844+0000","last_unstale":"2024-02-23T00:33:14.027454+0000","last_undegraded":"2024-02-23T00:33:14.027454+0000","last_fullsized":"2024-02-23T00:33:14.027454+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.d","version":"224'34","reported_seq":195,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.024644+0000","last_change":"2024-02-23T00:33:07.137804+0000","last_active":"2024-02-23T00:33:14.024644+0000","last_peered":"2024-02-23T00:33:14.024644+0000","last_clean":"2024-02-23T00:33:14.024644+0000","last_became_active":"2024-02-23T00:31:06.262251+0000","last_became_peered":"2024-02-23T00:31:06.262251+0000","last_unstale":"2024-02-23T00:33:14.024644+0000","last_undegraded":"2024-02-23T00:33:14.024644+0000","last_fullsized":"2024-02-23T00:33:14.024644+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.e","version":"220'73","reported_seq":234,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085707+0000","last_change":"2024-02-23T00:33:00.040538+0000","last_active":"2024-02-23T00:33:14.085707+0000","last_peered":"2024-02-23T00:33:14.085707+0000","last_clean":"2024-02-23T00:33:14.085707+0000","last_became_active":"2024-02-23T00:31:06.356847+0000","last_became_peered":"2024-02-23T00:31:06.356847+0000","last_unstale":"2024-02-23T00:33:14.085707+0000","last_undegraded":"2024-02-23T00:33:14.085707+0000","last_fullsized":"2024-02-23T00:33:14.085707+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.f","version":"219'69","reported_seq":227,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083774+0000","last_change":"2024-02-23T00:33:00.038123+0000","last_active":"2024-02-23T00:33:14.083774+0000","last_peered":"2024-02-23T00:33:14.083774+0000","last_clean":"2024-02-23T00:33:14.083774+0000","last_became_active":"2024-02-23T00:31:06.364609+0000","last_became_peered":"2024-02-23T00:31:06.364609+0000","last_unstale":"2024-02-23T00:33:14.083774+0000","last_undegraded":"2024-02-23T00:33:14.083774+0000","last_fullsized":"2024-02-23T00:33:14.083774+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.10","version":"218'34","reported_seq":197,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084154+0000","last_change":"2024-02-23T00:33:00.041791+0000","last_active":"2024-02-23T00:33:14.084154+0000","last_peered":"2024-02-23T00:33:14.084154+0000","last_clean":"2024-02-23T00:33:14.084154+0000","last_became_active":"2024-02-23T00:31:06.342164+0000","last_became_peered":"2024-02-23T00:31:06.342164+0000","last_unstale":"2024-02-23T00:33:14.084154+0000","last_undegraded":"2024-02-23T00:33:14.084154+0000","last_fullsized":"2024-02-23T00:33:14.084154+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.11","version":"244'43","reported_seq":202,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.025206+0000","last_change":"2024-02-23T00:33:00.023883+0000","last_active":"2024-02-23T00:33:15.025206+0000","last_peered":"2024-02-23T00:33:15.025206+0000","last_clean":"2024-02-23T00:33:15.025206+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:33:15.025206+0000","last_undegraded":"2024-02-23T00:33:15.025206+0000","last_fullsized":"2024-02-23T00:33:15.025206+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.12","version":"204'68","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.028128+0000","last_change":"2024-02-23T00:33:00.030555+0000","last_active":"2024-02-23T00:33:14.028128+0000","last_peered":"2024-02-23T00:33:14.028128+0000","last_clean":"2024-02-23T00:33:14.028128+0000","last_became_active":"2024-02-23T00:31:06.281619+0000","last_became_peered":"2024-02-23T00:31:06.281619+0000","last_unstale":"2024-02-23T00:33:14.028128+0000","last_undegraded":"2024-02-23T00:33:14.028128+0000","last_fullsized":"2024-02-23T00:33:14.028128+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.13","version":"244'59","reported_seq":218,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.065984+0000","last_change":"2024-02-23T00:33:00.038010+0000","last_active":"2024-02-23T00:33:15.065984+0000","last_peered":"2024-02-23T00:33:15.065984+0000","last_clean":"2024-02-23T00:33:15.065984+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:33:15.065984+0000","last_undegraded":"2024-02-23T00:33:15.065984+0000","last_fullsized":"2024-02-23T00:33:15.065984+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.14","version":"208'85","reported_seq":237,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027345+0000","last_change":"2024-02-23T00:33:00.031035+0000","last_active":"2024-02-23T00:33:14.027345+0000","last_peered":"2024-02-23T00:33:14.027345+0000","last_clean":"2024-02-23T00:33:14.027345+0000","last_became_active":"2024-02-23T00:31:06.234521+0000","last_became_peered":"2024-02-23T00:31:06.234521+0000","last_unstale":"2024-02-23T00:33:14.027345+0000","last_undegraded":"2024-02-23T00:33:14.027345+0000","last_fullsized":"2024-02-23T00:33:14.027345+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.15","version":"220'68","reported_seq":229,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.020823+0000","last_change":"2024-02-23T00:33:00.031524+0000","last_active":"2024-02-23T00:33:14.020823+0000","last_peered":"2024-02-23T00:33:14.020823+0000","last_clean":"2024-02-23T00:33:14.020823+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:33:14.020823+0000","last_undegraded":"2024-02-23T00:33:14.020823+0000","last_fullsized":"2024-02-23T00:33:14.020823+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.16","version":"204'51","reported_seq":220,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084233+0000","last_change":"2024-02-23T00:33:00.032292+0000","last_active":"2024-02-23T00:33:14.084233+0000","last_peered":"2024-02-23T00:33:14.084233+0000","last_clean":"2024-02-23T00:33:14.084233+0000","last_became_active":"2024-02-23T00:31:06.367069+0000","last_became_peered":"2024-02-23T00:31:06.367069+0000","last_unstale":"2024-02-23T00:33:14.084233+0000","last_undegraded":"2024-02-23T00:33:14.084233+0000","last_fullsized":"2024-02-23T00:33:14.084233+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:32:57.837196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:57.837196+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.17","version":"230'85","reported_seq":242,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.030192+0000","last_change":"2024-02-23T00:33:07.138337+0000","last_active":"2024-02-23T00:33:14.030192+0000","last_peered":"2024-02-23T00:33:14.030192+0000","last_clean":"2024-02-23T00:33:14.030192+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:33:14.030192+0000","last_undegraded":"2024-02-23T00:33:14.030192+0000","last_fullsized":"2024-02-23T00:33:14.030192+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.18","version":"211'57","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032717+0000","last_change":"2024-02-23T00:33:00.035022+0000","last_active":"2024-02-23T00:33:14.032717+0000","last_peered":"2024-02-23T00:33:14.032717+0000","last_clean":"2024-02-23T00:33:14.032717+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:33:14.032717+0000","last_undegraded":"2024-02-23T00:33:14.032717+0000","last_fullsized":"2024-02-23T00:33:14.032717+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.19","version":"216'51","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.017869+0000","last_change":"2024-02-23T00:33:00.023463+0000","last_active":"2024-02-23T00:33:14.017869+0000","last_peered":"2024-02-23T00:33:14.017869+0000","last_clean":"2024-02-23T00:33:14.017869+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:33:14.017869+0000","last_undegraded":"2024-02-23T00:33:14.017869+0000","last_fullsized":"2024-02-23T00:33:14.017869+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1a","version":"202'68","reported_seq":230,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085444+0000","last_change":"2024-02-23T00:33:00.023838+0000","last_active":"2024-02-23T00:33:14.085444+0000","last_peered":"2024-02-23T00:33:14.085444+0000","last_clean":"2024-02-23T00:33:14.085444+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:33:14.085444+0000","last_undegraded":"2024-02-23T00:33:14.085444+0000","last_fullsized":"2024-02-23T00:33:14.085444+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'68","last_scrub_stamp":"2024-02-23T00:32:58.808102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.808102+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1b","version":"236'51","reported_seq":211,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027701+0000","last_change":"2024-02-23T00:33:09.181472+0000","last_active":"2024-02-23T00:33:14.027701+0000","last_peered":"2024-02-23T00:33:14.027701+0000","last_clean":"2024-02-23T00:33:14.027701+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:33:14.027701+0000","last_undegraded":"2024-02-23T00:33:14.027701+0000","last_fullsized":"2024-02-23T00:33:14.027701+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1c","version":"216'34","reported_seq":196,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085602+0000","last_change":"2024-02-23T00:33:02.043705+0000","last_active":"2024-02-23T00:33:14.085602+0000","last_peered":"2024-02-23T00:33:14.085602+0000","last_clean":"2024-02-23T00:33:14.085602+0000","last_became_active":"2024-02-23T00:31:06.338540+0000","last_became_peered":"2024-02-23T00:31:06.338540+0000","last_unstale":"2024-02-23T00:33:14.085602+0000","last_undegraded":"2024-02-23T00:33:14.085602+0000","last_fullsized":"2024-02-23T00:33:14.085602+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1d","version":"229'10","reported_seq":181,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091513+0000","last_change":"2024-02-23T00:33:02.042975+0000","last_active":"2024-02-23T00:33:14.091513+0000","last_peered":"2024-02-23T00:33:14.091513+0000","last_clean":"2024-02-23T00:33:14.091513+0000","last_became_active":"2024-02-23T00:31:06.337635+0000","last_became_peered":"2024-02-23T00:31:06.337635+0000","last_unstale":"2024-02-23T00:33:14.091513+0000","last_undegraded":"2024-02-23T00:33:14.091513+0000","last_fullsized":"2024-02-23T00:33:14.091513+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1e","version":"232'51","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.024633+0000","last_change":"2024-02-23T00:33:07.138597+0000","last_active":"2024-02-23T00:33:14.024633+0000","last_peered":"2024-02-23T00:33:14.024633+0000","last_clean":"2024-02-23T00:33:14.024633+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:33:14.024633+0000","last_undegraded":"2024-02-23T00:33:14.024633+0000","last_fullsized":"2024-02-23T00:33:14.024633+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"63.1f","version":"228'68","reported_seq":227,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.030075+0000","last_change":"2024-02-23T00:33:07.138080+0000","last_active":"2024-02-23T00:33:14.030075+0000","last_peered":"2024-02-23T00:33:14.030075+0000","last_clean":"2024-02-23T00:33:14.030075+0000","last_became_active":"2024-02-23T00:31:06.355623+0000","last_became_peered":"2024-02-23T00:31:06.355623+0000","last_unstale":"2024-02-23T00:33:14.030075+0000","last_undegraded":"2024-02-23T00:33:14.030075+0000","last_fullsized":"2024-02-23T00:33:14.030075+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"5.9","version":"62'8","reported_seq":287,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.066068+0000","last_change":"2024-02-23T00:32:11.743347+0000","last_active":"2024-02-23T00:33:15.066068+0000","last_peered":"2024-02-23T00:33:15.066068+0000","last_clean":"2024-02-23T00:33:15.066068+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:33:15.066068+0000","last_undegraded":"2024-02-23T00:33:15.066068+0000","last_fullsized":"2024-02-23T00:33:15.066068+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:11.743278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.743278+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":271,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.025508+0000","last_change":"2024-02-23T00:32:47.376760+0000","last_active":"2024-02-23T00:33:14.025508+0000","last_peered":"2024-02-23T00:33:14.025508+0000","last_clean":"2024-02-23T00:33:14.025508+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:33:14.025508+0000","last_undegraded":"2024-02-23T00:33:14.025508+0000","last_fullsized":"2024-02-23T00:33:14.025508+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:32:47.376691+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.376691+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084377+0000","last_change":"2024-02-23T00:33:00.878453+0000","last_active":"2024-02-23T00:33:14.084377+0000","last_peered":"2024-02-23T00:33:14.084377+0000","last_clean":"2024-02-23T00:33:14.084377+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:33:14.084377+0000","last_undegraded":"2024-02-23T00:33:14.084377+0000","last_fullsized":"2024-02-23T00:33:14.084377+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.878391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.878391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":198,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.023164+0000","last_change":"2024-02-23T00:31:31.323024+0000","last_active":"2024-02-23T00:33:14.023164+0000","last_peered":"2024-02-23T00:33:14.023164+0000","last_clean":"2024-02-23T00:33:14.023164+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:33:14.023164+0000","last_undegraded":"2024-02-23T00:33:14.023164+0000","last_fullsized":"2024-02-23T00:33:14.023164+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.322940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.322940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":157,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091969+0000","last_change":"2024-02-23T00:32:48.468225+0000","last_active":"2024-02-23T00:33:14.091969+0000","last_peered":"2024-02-23T00:33:14.091969+0000","last_clean":"2024-02-23T00:33:14.091969+0000","last_became_active":"2024-02-23T00:30:48.777752+0000","last_became_peered":"2024-02-23T00:30:48.777752+0000","last_unstale":"2024-02-23T00:33:14.091969+0000","last_undegraded":"2024-02-23T00:33:14.091969+0000","last_fullsized":"2024-02-23T00:33:14.091969+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":231,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.033859+0000","last_change":"2024-02-23T00:32:37.440685+0000","last_active":"2024-02-23T00:33:15.033859+0000","last_peered":"2024-02-23T00:33:15.033859+0000","last_clean":"2024-02-23T00:33:15.033859+0000","last_became_active":"2024-02-23T00:31:29.077539+0000","last_became_peered":"2024-02-23T00:31:29.077539+0000","last_unstale":"2024-02-23T00:33:15.033859+0000","last_undegraded":"2024-02-23T00:33:15.033859+0000","last_fullsized":"2024-02-23T00:33:15.033859+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:37.440614+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.440614+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":242,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.086107+0000","last_change":"2024-02-23T00:32:09.590050+0000","last_active":"2024-02-23T00:33:14.086107+0000","last_peered":"2024-02-23T00:33:14.086107+0000","last_clean":"2024-02-23T00:33:14.086107+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:33:14.086107+0000","last_undegraded":"2024-02-23T00:33:14.086107+0000","last_fullsized":"2024-02-23T00:33:14.086107+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:32:09.589983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.589983+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.092101+0000","last_change":"2024-02-23T00:32:38.328306+0000","last_active":"2024-02-23T00:33:14.092101+0000","last_peered":"2024-02-23T00:33:14.092101+0000","last_clean":"2024-02-23T00:33:14.092101+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:33:14.092101+0000","last_undegraded":"2024-02-23T00:33:14.092101+0000","last_fullsized":"2024-02-23T00:33:14.092101+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.328237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.328237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085585+0000","last_change":"2024-02-23T00:32:40.855264+0000","last_active":"2024-02-23T00:33:14.085585+0000","last_peered":"2024-02-23T00:33:14.085585+0000","last_clean":"2024-02-23T00:33:14.085585+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:33:14.085585+0000","last_undegraded":"2024-02-23T00:33:14.085585+0000","last_fullsized":"2024-02-23T00:33:14.085585+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.855194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.855194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.086799+0000","last_change":"2024-02-23T00:32:06.561440+0000","last_active":"2024-02-23T00:33:14.086799+0000","last_peered":"2024-02-23T00:33:14.086799+0000","last_clean":"2024-02-23T00:33:14.086799+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:33:14.086799+0000","last_undegraded":"2024-02-23T00:33:14.086799+0000","last_fullsized":"2024-02-23T00:33:14.086799+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:06.561374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:06.561374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.031198+0000","last_change":"2024-02-23T00:30:48.713856+0000","last_active":"2024-02-23T00:33:14.031198+0000","last_peered":"2024-02-23T00:33:14.031198+0000","last_clean":"2024-02-23T00:33:14.031198+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:33:14.031198+0000","last_undegraded":"2024-02-23T00:33:14.031198+0000","last_fullsized":"2024-02-23T00:33:14.031198+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":255,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.028504+0000","last_change":"2024-02-23T00:31:41.019548+0000","last_active":"2024-02-23T00:33:14.028504+0000","last_peered":"2024-02-23T00:33:14.028504+0000","last_clean":"2024-02-23T00:33:14.028504+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:33:14.028504+0000","last_undegraded":"2024-02-23T00:33:14.028504+0000","last_fullsized":"2024-02-23T00:33:14.028504+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.019441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.019441+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083173+0000","last_change":"2024-02-23T00:32:59.849579+0000","last_active":"2024-02-23T00:33:14.083173+0000","last_peered":"2024-02-23T00:33:14.083173+0000","last_clean":"2024-02-23T00:33:14.083173+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:33:14.083173+0000","last_undegraded":"2024-02-23T00:33:14.083173+0000","last_fullsized":"2024-02-23T00:33:14.083173+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.849503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.849503+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.081834+0000","last_change":"2024-02-23T00:32:46.885745+0000","last_active":"2024-02-23T00:33:14.081834+0000","last_peered":"2024-02-23T00:33:14.081834+0000","last_clean":"2024-02-23T00:33:14.081834+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:33:14.081834+0000","last_undegraded":"2024-02-23T00:33:14.081834+0000","last_fullsized":"2024-02-23T00:33:14.081834+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.885681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.885681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032828+0000","last_change":"2024-02-23T00:32:38.683758+0000","last_active":"2024-02-23T00:33:14.032828+0000","last_peered":"2024-02-23T00:33:14.032828+0000","last_clean":"2024-02-23T00:33:14.032828+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:33:14.032828+0000","last_undegraded":"2024-02-23T00:33:14.032828+0000","last_fullsized":"2024-02-23T00:33:14.032828+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.683686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.683686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":160,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.088272+0000","last_change":"2024-02-23T00:32:45.536091+0000","last_active":"2024-02-23T00:33:14.088272+0000","last_peered":"2024-02-23T00:33:14.088272+0000","last_clean":"2024-02-23T00:33:14.088272+0000","last_became_active":"2024-02-23T00:30:48.673452+0000","last_became_peered":"2024-02-23T00:30:48.673452+0000","last_unstale":"2024-02-23T00:33:14.088272+0000","last_undegraded":"2024-02-23T00:33:14.088272+0000","last_fullsized":"2024-02-23T00:33:14.088272+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.535992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.535992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":220,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032473+0000","last_change":"2024-02-23T00:32:37.445055+0000","last_active":"2024-02-23T00:33:14.032473+0000","last_peered":"2024-02-23T00:33:14.032473+0000","last_clean":"2024-02-23T00:33:14.032473+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:33:14.032473+0000","last_undegraded":"2024-02-23T00:33:14.032473+0000","last_fullsized":"2024-02-23T00:33:14.032473+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.444973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.444973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":234,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.021522+0000","last_change":"2024-02-23T00:30:35.121011+0000","last_active":"2024-02-23T00:33:14.021522+0000","last_peered":"2024-02-23T00:33:14.021522+0000","last_clean":"2024-02-23T00:33:14.021522+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:33:14.021522+0000","last_undegraded":"2024-02-23T00:33:14.021522+0000","last_fullsized":"2024-02-23T00:33:14.021522+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:30:35.120909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:35.120909+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":211,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.033404+0000","last_change":"2024-02-23T00:32:56.692114+0000","last_active":"2024-02-23T00:33:14.033404+0000","last_peered":"2024-02-23T00:33:14.033404+0000","last_clean":"2024-02-23T00:33:14.033404+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:33:14.033404+0000","last_undegraded":"2024-02-23T00:33:14.033404+0000","last_fullsized":"2024-02-23T00:33:14.033404+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.692037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.692037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085968+0000","last_change":"2024-02-23T00:32:09.110225+0000","last_active":"2024-02-23T00:33:14.085968+0000","last_peered":"2024-02-23T00:33:14.085968+0000","last_clean":"2024-02-23T00:33:14.085968+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:33:14.085968+0000","last_undegraded":"2024-02-23T00:33:14.085968+0000","last_fullsized":"2024-02-23T00:33:14.085968+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.110138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083181+0000","last_change":"2024-02-23T00:32:15.855653+0000","last_active":"2024-02-23T00:33:14.083181+0000","last_peered":"2024-02-23T00:33:14.083181+0000","last_clean":"2024-02-23T00:33:14.083181+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:33:14.083181+0000","last_undegraded":"2024-02-23T00:33:14.083181+0000","last_fullsized":"2024-02-23T00:33:14.083181+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:32:15.855580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":159,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.034879+0000","last_change":"2024-02-23T00:33:00.671292+0000","last_active":"2024-02-23T00:33:14.034879+0000","last_peered":"2024-02-23T00:33:14.034879+0000","last_clean":"2024-02-23T00:33:14.034879+0000","last_became_active":"2024-02-23T00:30:48.716552+0000","last_became_peered":"2024-02-23T00:30:48.716552+0000","last_unstale":"2024-02-23T00:33:14.034879+0000","last_undegraded":"2024-02-23T00:33:14.034879+0000","last_fullsized":"2024-02-23T00:33:14.034879+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.671219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.671219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":222,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.016906+0000","last_change":"2024-02-23T00:32:53.985225+0000","last_active":"2024-02-23T00:33:14.016906+0000","last_peered":"2024-02-23T00:33:14.016906+0000","last_clean":"2024-02-23T00:33:14.016906+0000","last_became_active":"2024-02-23T00:31:29.064244+0000","last_became_peered":"2024-02-23T00:31:29.064244+0000","last_unstale":"2024-02-23T00:33:14.016906+0000","last_undegraded":"2024-02-23T00:33:14.016906+0000","last_fullsized":"2024-02-23T00:33:14.016906+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.985150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.985150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":221,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.021046+0000","last_change":"2024-02-23T00:31:19.982059+0000","last_active":"2024-02-23T00:33:14.021046+0000","last_peered":"2024-02-23T00:33:14.021046+0000","last_clean":"2024-02-23T00:33:14.021046+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:33:14.021046+0000","last_undegraded":"2024-02-23T00:33:14.021046+0000","last_fullsized":"2024-02-23T00:33:14.021046+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:31:19.981985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.981985+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":223,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091544+0000","last_change":"2024-02-23T00:31:49.024710+0000","last_active":"2024-02-23T00:33:14.091544+0000","last_peered":"2024-02-23T00:33:14.091544+0000","last_clean":"2024-02-23T00:33:14.091544+0000","last_became_active":"2024-02-23T00:31:49.024302+0000","last_became_peered":"2024-02-23T00:31:49.024302+0000","last_unstale":"2024-02-23T00:33:14.091544+0000","last_undegraded":"2024-02-23T00:33:14.091544+0000","last_fullsized":"2024-02-23T00:33:14.091544+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.937969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":289,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.059679+0000","last_change":"2024-02-23T00:32:35.929043+0000","last_active":"2024-02-23T00:33:15.059679+0000","last_peered":"2024-02-23T00:33:15.059679+0000","last_clean":"2024-02-23T00:33:15.059679+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:33:15.059679+0000","last_undegraded":"2024-02-23T00:33:15.059679+0000","last_fullsized":"2024-02-23T00:33:15.059679+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.928977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.928977+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":202,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.022137+0000","last_change":"2024-02-23T00:30:31.065478+0000","last_active":"2024-02-23T00:33:14.022137+0000","last_peered":"2024-02-23T00:33:14.022137+0000","last_clean":"2024-02-23T00:33:14.022137+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:33:14.022137+0000","last_undegraded":"2024-02-23T00:33:14.022137+0000","last_fullsized":"2024-02-23T00:33:14.022137+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.065384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.065384+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":155,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.022131+0000","last_change":"2024-02-23T00:30:48.656349+0000","last_active":"2024-02-23T00:33:14.022131+0000","last_peered":"2024-02-23T00:33:14.022131+0000","last_clean":"2024-02-23T00:33:14.022131+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:33:14.022131+0000","last_undegraded":"2024-02-23T00:33:14.022131+0000","last_fullsized":"2024-02-23T00:33:14.022131+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":121,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083401+0000","last_change":"2024-02-23T00:32:58.975408+0000","last_active":"2024-02-23T00:33:14.083401+0000","last_peered":"2024-02-23T00:33:14.083401+0000","last_clean":"2024-02-23T00:33:14.083401+0000","last_became_active":"2024-02-23T00:31:29.089319+0000","last_became_peered":"2024-02-23T00:31:29.089319+0000","last_unstale":"2024-02-23T00:33:14.083401+0000","last_undegraded":"2024-02-23T00:33:14.083401+0000","last_fullsized":"2024-02-23T00:33:14.083401+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:58.975334+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.975334+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"87'22","reported_seq":240,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.090682+0000","last_change":"2024-02-23T00:32:40.327594+0000","last_active":"2024-02-23T00:33:14.090682+0000","last_peered":"2024-02-23T00:33:14.090682+0000","last_clean":"2024-02-23T00:33:14.090682+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:33:14.090682+0000","last_undegraded":"2024-02-23T00:33:14.090682+0000","last_fullsized":"2024-02-23T00:33:14.090682+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:32:40.327519+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.327519+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.089563+0000","last_change":"2024-02-23T00:31:29.920657+0000","last_active":"2024-02-23T00:33:14.089563+0000","last_peered":"2024-02-23T00:33:14.089563+0000","last_clean":"2024-02-23T00:33:14.089563+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:33:14.089563+0000","last_undegraded":"2024-02-23T00:33:14.089563+0000","last_fullsized":"2024-02-23T00:33:14.089563+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:29.920546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:29.920546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":289,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.028150+0000","last_change":"2024-02-23T00:32:35.975973+0000","last_active":"2024-02-23T00:33:15.028150+0000","last_peered":"2024-02-23T00:33:15.028150+0000","last_clean":"2024-02-23T00:33:15.028150+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:33:15.028150+0000","last_undegraded":"2024-02-23T00:33:15.028150+0000","last_fullsized":"2024-02-23T00:33:15.028150+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.975889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.975889+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.025531+0000","last_change":"2024-02-23T00:31:21.366604+0000","last_active":"2024-02-23T00:33:14.025531+0000","last_peered":"2024-02-23T00:33:14.025531+0000","last_clean":"2024-02-23T00:33:14.025531+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:33:14.025531+0000","last_undegraded":"2024-02-23T00:33:14.025531+0000","last_fullsized":"2024-02-23T00:33:14.025531+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:31:21.366535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.366535+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":152,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091747+0000","last_change":"2024-02-23T00:30:48.713004+0000","last_active":"2024-02-23T00:33:14.091747+0000","last_peered":"2024-02-23T00:33:14.091747+0000","last_clean":"2024-02-23T00:33:14.091747+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:33:14.091747+0000","last_undegraded":"2024-02-23T00:33:14.091747+0000","last_fullsized":"2024-02-23T00:33:14.091747+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":222,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.017791+0000","last_change":"2024-02-23T00:32:47.737933+0000","last_active":"2024-02-23T00:33:14.017791+0000","last_peered":"2024-02-23T00:33:14.017791+0000","last_clean":"2024-02-23T00:33:14.017791+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:33:14.017791+0000","last_undegraded":"2024-02-23T00:33:14.017791+0000","last_fullsized":"2024-02-23T00:33:14.017791+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:47.737874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.737874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"86'23","reported_seq":251,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.021797+0000","last_change":"2024-02-23T00:32:46.780054+0000","last_active":"2024-02-23T00:33:14.021797+0000","last_peered":"2024-02-23T00:33:14.021797+0000","last_clean":"2024-02-23T00:33:14.021797+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:33:14.021797+0000","last_undegraded":"2024-02-23T00:33:14.021797+0000","last_fullsized":"2024-02-23T00:33:14.021797+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'23","last_scrub_stamp":"2024-02-23T00:32:46.779979+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.779979+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.033136+0000","last_change":"2024-02-23T00:31:38.838289+0000","last_active":"2024-02-23T00:33:14.033136+0000","last_peered":"2024-02-23T00:33:14.033136+0000","last_clean":"2024-02-23T00:33:14.033136+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:33:14.033136+0000","last_undegraded":"2024-02-23T00:33:14.033136+0000","last_fullsized":"2024-02-23T00:33:14.033136+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:31:38.838202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.838202+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":200,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032431+0000","last_change":"2024-02-23T00:31:33.381301+0000","last_active":"2024-02-23T00:33:14.032431+0000","last_peered":"2024-02-23T00:33:14.032431+0000","last_clean":"2024-02-23T00:33:14.032431+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:33:14.032431+0000","last_undegraded":"2024-02-23T00:33:14.032431+0000","last_fullsized":"2024-02-23T00:33:14.032431+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.381229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.381229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":202,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084141+0000","last_change":"2024-02-23T00:32:09.012254+0000","last_active":"2024-02-23T00:33:14.084141+0000","last_peered":"2024-02-23T00:33:14.084141+0000","last_clean":"2024-02-23T00:33:14.084141+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:33:14.084141+0000","last_undegraded":"2024-02-23T00:33:14.084141+0000","last_fullsized":"2024-02-23T00:33:14.084141+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.012161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.012161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":157,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.086228+0000","last_change":"2024-02-23T00:32:53.387558+0000","last_active":"2024-02-23T00:33:14.086228+0000","last_peered":"2024-02-23T00:33:14.086228+0000","last_clean":"2024-02-23T00:33:14.086228+0000","last_became_active":"2024-02-23T00:30:48.740636+0000","last_became_peered":"2024-02-23T00:30:48.740636+0000","last_unstale":"2024-02-23T00:33:14.086228+0000","last_undegraded":"2024-02-23T00:33:14.086228+0000","last_fullsized":"2024-02-23T00:33:14.086228+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.387496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":126,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.095922+0000","last_change":"2024-02-23T00:32:56.610957+0000","last_active":"2024-02-23T00:33:14.095922+0000","last_peered":"2024-02-23T00:33:14.095922+0000","last_clean":"2024-02-23T00:33:14.095922+0000","last_became_active":"2024-02-23T00:31:29.108747+0000","last_became_peered":"2024-02-23T00:31:29.108747+0000","last_unstale":"2024-02-23T00:33:14.095922+0000","last_undegraded":"2024-02-23T00:33:14.095922+0000","last_fullsized":"2024-02-23T00:33:14.095922+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.610884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.610884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":288,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.027228+0000","last_change":"2024-02-23T00:31:41.945984+0000","last_active":"2024-02-23T00:33:15.027228+0000","last_peered":"2024-02-23T00:33:15.027228+0000","last_clean":"2024-02-23T00:33:15.027228+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:33:15.027228+0000","last_undegraded":"2024-02-23T00:33:15.027228+0000","last_fullsized":"2024-02-23T00:33:15.027228+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":242,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.018437+0000","last_change":"2024-02-23T00:31:38.038523+0000","last_active":"2024-02-23T00:33:14.018437+0000","last_peered":"2024-02-23T00:33:14.018437+0000","last_clean":"2024-02-23T00:33:14.018437+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:33:14.018437+0000","last_undegraded":"2024-02-23T00:33:14.018437+0000","last_fullsized":"2024-02-23T00:33:14.018437+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:31:38.038413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.038413+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027859+0000","last_change":"2024-02-23T00:32:46.504136+0000","last_active":"2024-02-23T00:33:14.027859+0000","last_peered":"2024-02-23T00:33:14.027859+0000","last_clean":"2024-02-23T00:33:14.027859+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:33:14.027859+0000","last_undegraded":"2024-02-23T00:33:14.027859+0000","last_fullsized":"2024-02-23T00:33:14.027859+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.504045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.504045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.034288+0000","last_change":"2024-02-23T00:31:31.946435+0000","last_active":"2024-02-23T00:33:14.034288+0000","last_peered":"2024-02-23T00:33:14.034288+0000","last_clean":"2024-02-23T00:33:14.034288+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:33:14.034288+0000","last_undegraded":"2024-02-23T00:33:14.034288+0000","last_fullsized":"2024-02-23T00:33:14.034288+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.946366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.946366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":152,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091103+0000","last_change":"2024-02-23T00:30:48.782732+0000","last_active":"2024-02-23T00:33:14.091103+0000","last_peered":"2024-02-23T00:33:14.091103+0000","last_clean":"2024-02-23T00:33:14.091103+0000","last_became_active":"2024-02-23T00:30:48.778293+0000","last_became_peered":"2024-02-23T00:30:48.778293+0000","last_unstale":"2024-02-23T00:33:14.091103+0000","last_undegraded":"2024-02-23T00:33:14.091103+0000","last_fullsized":"2024-02-23T00:33:14.091103+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":222,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084444+0000","last_change":"2024-02-23T00:32:55.991569+0000","last_active":"2024-02-23T00:33:14.084444+0000","last_peered":"2024-02-23T00:33:14.084444+0000","last_clean":"2024-02-23T00:33:14.084444+0000","last_became_active":"2024-02-23T00:31:29.070521+0000","last_became_peered":"2024-02-23T00:31:29.070521+0000","last_unstale":"2024-02-23T00:33:14.084444+0000","last_undegraded":"2024-02-23T00:33:14.084444+0000","last_fullsized":"2024-02-23T00:33:14.084444+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:55.991514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:55.991514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.018607+0000","last_change":"2024-02-23T00:32:41.922931+0000","last_active":"2024-02-23T00:33:14.018607+0000","last_peered":"2024-02-23T00:33:14.018607+0000","last_clean":"2024-02-23T00:33:14.018607+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:33:14.018607+0000","last_undegraded":"2024-02-23T00:33:14.018607+0000","last_fullsized":"2024-02-23T00:33:14.018607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:41.922849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.922849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":268,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083038+0000","last_change":"2024-02-23T00:31:21.086750+0000","last_active":"2024-02-23T00:33:14.083038+0000","last_peered":"2024-02-23T00:33:14.083038+0000","last_clean":"2024-02-23T00:33:14.083038+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:33:14.083038+0000","last_undegraded":"2024-02-23T00:33:14.083038+0000","last_fullsized":"2024-02-23T00:33:14.083038+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:31:21.086610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.086610+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085867+0000","last_change":"2024-02-23T00:30:25.593913+0000","last_active":"2024-02-23T00:33:14.085867+0000","last_peered":"2024-02-23T00:33:14.085867+0000","last_clean":"2024-02-23T00:33:14.085867+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:33:14.085867+0000","last_undegraded":"2024-02-23T00:33:14.085867+0000","last_fullsized":"2024-02-23T00:33:14.085867+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:25.593833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:25.593833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.019702+0000","last_change":"2024-02-23T00:32:17.981331+0000","last_active":"2024-02-23T00:33:14.019702+0000","last_peered":"2024-02-23T00:33:14.019702+0000","last_clean":"2024-02-23T00:33:14.019702+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:33:14.019702+0000","last_undegraded":"2024-02-23T00:33:14.019702+0000","last_fullsized":"2024-02-23T00:33:14.019702+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.981262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.981262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":155,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083876+0000","last_change":"2024-02-23T00:30:48.776316+0000","last_active":"2024-02-23T00:33:14.083876+0000","last_peered":"2024-02-23T00:33:14.083876+0000","last_clean":"2024-02-23T00:33:14.083876+0000","last_became_active":"2024-02-23T00:30:48.769602+0000","last_became_peered":"2024-02-23T00:30:48.769602+0000","last_unstale":"2024-02-23T00:33:14.083876+0000","last_undegraded":"2024-02-23T00:33:14.083876+0000","last_fullsized":"2024-02-23T00:33:14.083876+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":473,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085507+0000","last_change":"2024-02-23T00:32:43.323030+0000","last_active":"2024-02-23T00:33:14.085507+0000","last_peered":"2024-02-23T00:33:14.085507+0000","last_clean":"2024-02-23T00:33:14.085507+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:33:14.085507+0000","last_undegraded":"2024-02-23T00:33:14.085507+0000","last_fullsized":"2024-02-23T00:33:14.085507+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:43.322963+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.322963+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.028594+0000","last_change":"2024-02-23T00:32:42.410196+0000","last_active":"2024-02-23T00:33:14.028594+0000","last_peered":"2024-02-23T00:33:14.028594+0000","last_clean":"2024-02-23T00:33:14.028594+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:33:14.028594+0000","last_undegraded":"2024-02-23T00:33:14.028594+0000","last_fullsized":"2024-02-23T00:33:14.028594+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.410112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.410112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":267,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085940+0000","last_change":"2024-02-23T00:32:47.533001+0000","last_active":"2024-02-23T00:33:14.085940+0000","last_peered":"2024-02-23T00:33:14.085940+0000","last_clean":"2024-02-23T00:33:14.085940+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:33:14.085940+0000","last_undegraded":"2024-02-23T00:33:14.085940+0000","last_fullsized":"2024-02-23T00:33:14.085940+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:32:47.532917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.532917+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":211,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035792+0000","last_change":"2024-02-23T00:31:40.847863+0000","last_active":"2024-02-23T00:33:14.035792+0000","last_peered":"2024-02-23T00:33:14.035792+0000","last_clean":"2024-02-23T00:33:14.035792+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:33:14.035792+0000","last_undegraded":"2024-02-23T00:33:14.035792+0000","last_fullsized":"2024-02-23T00:33:14.035792+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.847784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.847784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.089645+0000","last_change":"2024-02-23T00:30:46.579484+0000","last_active":"2024-02-23T00:33:14.089645+0000","last_peered":"2024-02-23T00:33:14.089645+0000","last_clean":"2024-02-23T00:33:14.089645+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:33:14.089645+0000","last_undegraded":"2024-02-23T00:33:14.089645+0000","last_fullsized":"2024-02-23T00:33:14.089645+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.579417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.579417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":152,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.089718+0000","last_change":"2024-02-23T00:30:48.707450+0000","last_active":"2024-02-23T00:33:14.089718+0000","last_peered":"2024-02-23T00:33:14.089718+0000","last_clean":"2024-02-23T00:33:14.089718+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:33:14.089718+0000","last_undegraded":"2024-02-23T00:33:14.089718+0000","last_fullsized":"2024-02-23T00:33:14.089718+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032563+0000","last_change":"2024-02-23T00:32:12.386351+0000","last_active":"2024-02-23T00:33:14.032563+0000","last_peered":"2024-02-23T00:33:14.032563+0000","last_clean":"2024-02-23T00:33:14.032563+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:33:14.032563+0000","last_undegraded":"2024-02-23T00:33:14.032563+0000","last_fullsized":"2024-02-23T00:33:14.032563+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":234,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.095420+0000","last_change":"2024-02-23T00:32:46.507450+0000","last_active":"2024-02-23T00:33:14.095420+0000","last_peered":"2024-02-23T00:33:14.095420+0000","last_clean":"2024-02-23T00:33:14.095420+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:33:14.095420+0000","last_undegraded":"2024-02-23T00:33:14.095420+0000","last_fullsized":"2024-02-23T00:33:14.095420+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:32:46.507354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.507354+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032955+0000","last_change":"2024-02-23T00:31:19.760264+0000","last_active":"2024-02-23T00:33:14.032955+0000","last_peered":"2024-02-23T00:33:14.032955+0000","last_clean":"2024-02-23T00:33:14.032955+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:33:14.032955+0000","last_undegraded":"2024-02-23T00:33:14.032955+0000","last_fullsized":"2024-02-23T00:33:14.032955+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:19.760201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.760201+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":222,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.086480+0000","last_change":"2024-02-23T00:31:38.653143+0000","last_active":"2024-02-23T00:33:14.086480+0000","last_peered":"2024-02-23T00:33:14.086480+0000","last_clean":"2024-02-23T00:33:14.086480+0000","last_became_active":"2024-02-23T00:31:27.053776+0000","last_became_peered":"2024-02-23T00:31:27.053776+0000","last_unstale":"2024-02-23T00:33:14.086480+0000","last_undegraded":"2024-02-23T00:33:14.086480+0000","last_fullsized":"2024-02-23T00:33:14.086480+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.653078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.653078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.086052+0000","last_change":"2024-02-23T00:31:21.959719+0000","last_active":"2024-02-23T00:33:14.086052+0000","last_peered":"2024-02-23T00:33:14.086052+0000","last_clean":"2024-02-23T00:33:14.086052+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:33:14.086052+0000","last_undegraded":"2024-02-23T00:33:14.086052+0000","last_fullsized":"2024-02-23T00:33:14.086052+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.959597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.959597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":155,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.021285+0000","last_change":"2024-02-23T00:30:48.711294+0000","last_active":"2024-02-23T00:33:14.021285+0000","last_peered":"2024-02-23T00:33:14.021285+0000","last_clean":"2024-02-23T00:33:14.021285+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:33:14.021285+0000","last_undegraded":"2024-02-23T00:33:14.021285+0000","last_fullsized":"2024-02-23T00:33:14.021285+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035031+0000","last_change":"2024-02-23T00:32:17.590019+0000","last_active":"2024-02-23T00:33:14.035031+0000","last_peered":"2024-02-23T00:33:14.035031+0000","last_clean":"2024-02-23T00:33:14.035031+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:33:14.035031+0000","last_undegraded":"2024-02-23T00:33:14.035031+0000","last_fullsized":"2024-02-23T00:33:14.035031+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.589952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.589952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":231,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027457+0000","last_change":"2024-02-23T00:31:40.030848+0000","last_active":"2024-02-23T00:33:14.027457+0000","last_peered":"2024-02-23T00:33:14.027457+0000","last_clean":"2024-02-23T00:33:14.027457+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:33:14.027457+0000","last_undegraded":"2024-02-23T00:33:14.027457+0000","last_fullsized":"2024-02-23T00:33:14.027457+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:31:40.030776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.030776+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":214,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.033048+0000","last_change":"2024-02-23T00:31:21.828852+0000","last_active":"2024-02-23T00:33:14.033048+0000","last_peered":"2024-02-23T00:33:14.033048+0000","last_clean":"2024-02-23T00:33:14.033048+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:33:14.033048+0000","last_undegraded":"2024-02-23T00:33:14.033048+0000","last_fullsized":"2024-02-23T00:33:14.033048+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.828719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.828719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.028303+0000","last_change":"2024-02-23T00:32:46.410759+0000","last_active":"2024-02-23T00:33:14.028303+0000","last_peered":"2024-02-23T00:33:14.028303+0000","last_clean":"2024-02-23T00:33:14.028303+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:33:14.028303+0000","last_undegraded":"2024-02-23T00:33:14.028303+0000","last_fullsized":"2024-02-23T00:33:14.028303+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.410635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.410635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":222,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083954+0000","last_change":"2024-02-23T00:32:46.978493+0000","last_active":"2024-02-23T00:33:14.083954+0000","last_peered":"2024-02-23T00:33:14.083954+0000","last_clean":"2024-02-23T00:33:14.083954+0000","last_became_active":"2024-02-23T00:31:27.144976+0000","last_became_peered":"2024-02-23T00:31:27.144976+0000","last_unstale":"2024-02-23T00:33:14.083954+0000","last_undegraded":"2024-02-23T00:33:14.083954+0000","last_fullsized":"2024-02-23T00:33:14.083954+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.978406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.978406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":203,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032655+0000","last_change":"2024-02-23T00:33:00.439952+0000","last_active":"2024-02-23T00:33:14.032655+0000","last_peered":"2024-02-23T00:33:14.032655+0000","last_clean":"2024-02-23T00:33:14.032655+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:33:14.032655+0000","last_undegraded":"2024-02-23T00:33:14.032655+0000","last_fullsized":"2024-02-23T00:33:14.032655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.439872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":155,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.021323+0000","last_change":"2024-02-23T00:30:48.664253+0000","last_active":"2024-02-23T00:33:14.021323+0000","last_peered":"2024-02-23T00:33:14.021323+0000","last_clean":"2024-02-23T00:33:14.021323+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:33:14.021323+0000","last_undegraded":"2024-02-23T00:33:14.021323+0000","last_fullsized":"2024-02-23T00:33:14.021323+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084453+0000","last_change":"2024-02-23T00:32:13.878058+0000","last_active":"2024-02-23T00:33:14.084453+0000","last_peered":"2024-02-23T00:33:14.084453+0000","last_clean":"2024-02-23T00:33:14.084453+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:33:14.084453+0000","last_undegraded":"2024-02-23T00:33:14.084453+0000","last_fullsized":"2024-02-23T00:33:14.084453+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:13.877940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.877940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":250,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.090482+0000","last_change":"2024-02-23T00:32:13.104236+0000","last_active":"2024-02-23T00:33:14.090482+0000","last_peered":"2024-02-23T00:33:14.090482+0000","last_clean":"2024-02-23T00:33:14.090482+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:33:14.090482+0000","last_undegraded":"2024-02-23T00:33:14.090482+0000","last_fullsized":"2024-02-23T00:33:14.090482+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:32:13.104170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.104170+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":213,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.082560+0000","last_change":"2024-02-23T00:31:41.651826+0000","last_active":"2024-02-23T00:33:14.082560+0000","last_peered":"2024-02-23T00:33:14.082560+0000","last_clean":"2024-02-23T00:33:14.082560+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:33:14.082560+0000","last_undegraded":"2024-02-23T00:33:14.082560+0000","last_fullsized":"2024-02-23T00:33:14.082560+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:41.651760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.651760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":466,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084846+0000","last_change":"2024-02-23T00:32:41.360750+0000","last_active":"2024-02-23T00:33:14.084846+0000","last_peered":"2024-02-23T00:33:14.084846+0000","last_clean":"2024-02-23T00:33:14.084846+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:33:14.084846+0000","last_undegraded":"2024-02-23T00:33:14.084846+0000","last_fullsized":"2024-02-23T00:33:14.084846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:41.360646+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.360646+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035702+0000","last_change":"2024-02-23T00:32:14.650905+0000","last_active":"2024-02-23T00:33:14.035702+0000","last_peered":"2024-02-23T00:33:14.035702+0000","last_clean":"2024-02-23T00:33:14.035702+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:33:14.035702+0000","last_undegraded":"2024-02-23T00:33:14.035702+0000","last_fullsized":"2024-02-23T00:33:14.035702+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.650827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.650827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":160,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085687+0000","last_change":"2024-02-23T00:32:41.861943+0000","last_active":"2024-02-23T00:33:14.085687+0000","last_peered":"2024-02-23T00:33:14.085687+0000","last_clean":"2024-02-23T00:33:14.085687+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:33:14.085687+0000","last_undegraded":"2024-02-23T00:33:14.085687+0000","last_fullsized":"2024-02-23T00:33:14.085687+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:41.861874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.861874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":222,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.019923+0000","last_change":"2024-02-23T00:32:50.705430+0000","last_active":"2024-02-23T00:33:14.019923+0000","last_peered":"2024-02-23T00:33:14.019923+0000","last_clean":"2024-02-23T00:33:14.019923+0000","last_became_active":"2024-02-23T00:31:29.065548+0000","last_became_peered":"2024-02-23T00:31:29.065548+0000","last_unstale":"2024-02-23T00:33:14.019923+0000","last_undegraded":"2024-02-23T00:33:14.019923+0000","last_fullsized":"2024-02-23T00:33:14.019923+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.705318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.705318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":216,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.086447+0000","last_change":"2024-02-23T00:32:50.596925+0000","last_active":"2024-02-23T00:33:14.086447+0000","last_peered":"2024-02-23T00:33:14.086447+0000","last_clean":"2024-02-23T00:33:14.086447+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:33:14.086447+0000","last_undegraded":"2024-02-23T00:33:14.086447+0000","last_fullsized":"2024-02-23T00:33:14.086447+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":343,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.075115+0000","last_change":"2024-02-23T00:32:45.366480+0000","last_active":"2024-02-23T00:33:15.075115+0000","last_peered":"2024-02-23T00:33:15.075115+0000","last_clean":"2024-02-23T00:33:15.075115+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:33:15.075115+0000","last_undegraded":"2024-02-23T00:33:15.075115+0000","last_fullsized":"2024-02-23T00:33:15.075115+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:32:45.366416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.366416+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.023350+0000","last_change":"2024-02-23T00:30:47.090981+0000","last_active":"2024-02-23T00:33:14.023350+0000","last_peered":"2024-02-23T00:33:14.023350+0000","last_clean":"2024-02-23T00:33:14.023350+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:33:14.023350+0000","last_undegraded":"2024-02-23T00:33:14.023350+0000","last_fullsized":"2024-02-23T00:33:14.023350+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:30:47.090896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.090896+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":202,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083768+0000","last_change":"2024-02-23T00:30:22.860665+0000","last_active":"2024-02-23T00:33:14.083768+0000","last_peered":"2024-02-23T00:33:14.083768+0000","last_clean":"2024-02-23T00:33:14.083768+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:33:14.083768+0000","last_undegraded":"2024-02-23T00:33:14.083768+0000","last_fullsized":"2024-02-23T00:33:14.083768+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:22.860565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:22.860565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":160,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084589+0000","last_change":"2024-02-23T00:32:44.885268+0000","last_active":"2024-02-23T00:33:14.084589+0000","last_peered":"2024-02-23T00:33:14.084589+0000","last_clean":"2024-02-23T00:33:14.084589+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:33:14.084589+0000","last_undegraded":"2024-02-23T00:33:14.084589+0000","last_fullsized":"2024-02-23T00:33:14.084589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.885205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.885205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":119,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.024358+0000","last_change":"2024-02-23T00:32:36.447499+0000","last_active":"2024-02-23T00:33:14.024358+0000","last_peered":"2024-02-23T00:33:14.024358+0000","last_clean":"2024-02-23T00:33:14.024358+0000","last_became_active":"2024-02-23T00:31:29.059516+0000","last_became_peered":"2024-02-23T00:31:29.059516+0000","last_unstale":"2024-02-23T00:33:14.024358+0000","last_undegraded":"2024-02-23T00:33:14.024358+0000","last_fullsized":"2024-02-23T00:33:14.024358+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.447439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.447439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":292,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.190152+0000","last_change":"2024-02-23T00:32:18.606981+0000","last_active":"2024-02-23T00:33:15.190152+0000","last_peered":"2024-02-23T00:33:15.190152+0000","last_clean":"2024-02-23T00:33:15.190152+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:33:15.190152+0000","last_undegraded":"2024-02-23T00:33:15.190152+0000","last_fullsized":"2024-02-23T00:33:15.190152+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:18.606911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:18.606911+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":233,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091793+0000","last_change":"2024-02-23T00:30:31.441530+0000","last_active":"2024-02-23T00:33:14.091793+0000","last_peered":"2024-02-23T00:33:14.091793+0000","last_clean":"2024-02-23T00:33:14.091793+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:33:14.091793+0000","last_undegraded":"2024-02-23T00:33:14.091793+0000","last_fullsized":"2024-02-23T00:33:14.091793+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:30:31.441418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.441418+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085745+0000","last_change":"2024-02-23T00:30:44.543241+0000","last_active":"2024-02-23T00:33:14.085745+0000","last_peered":"2024-02-23T00:33:14.085745+0000","last_clean":"2024-02-23T00:33:14.085745+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:33:14.085745+0000","last_undegraded":"2024-02-23T00:33:14.085745+0000","last_fullsized":"2024-02-23T00:33:14.085745+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:44.543161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:30:44.543161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.087693+0000","last_change":"2024-02-23T00:32:08.596816+0000","last_active":"2024-02-23T00:33:14.087693+0000","last_peered":"2024-02-23T00:33:14.087693+0000","last_clean":"2024-02-23T00:33:14.087693+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:33:14.087693+0000","last_undegraded":"2024-02-23T00:33:14.087693+0000","last_fullsized":"2024-02-23T00:33:14.087693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:08.596734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:08.596734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":155,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.018756+0000","last_change":"2024-02-23T00:30:48.616685+0000","last_active":"2024-02-23T00:33:14.018756+0000","last_peered":"2024-02-23T00:33:14.018756+0000","last_clean":"2024-02-23T00:33:14.018756+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:33:14.018756+0000","last_undegraded":"2024-02-23T00:33:14.018756+0000","last_fullsized":"2024-02-23T00:33:14.018756+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.017620+0000","last_change":"2024-02-23T00:32:42.815796+0000","last_active":"2024-02-23T00:33:14.017620+0000","last_peered":"2024-02-23T00:33:14.017620+0000","last_clean":"2024-02-23T00:33:14.017620+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:33:14.017620+0000","last_undegraded":"2024-02-23T00:33:14.017620+0000","last_fullsized":"2024-02-23T00:33:14.017620+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.815722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.815722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":285,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.018582+0000","last_change":"2024-02-23T00:31:41.036909+0000","last_active":"2024-02-23T00:33:14.018582+0000","last_peered":"2024-02-23T00:33:14.018582+0000","last_clean":"2024-02-23T00:33:14.018582+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:33:14.018582+0000","last_undegraded":"2024-02-23T00:33:14.018582+0000","last_fullsized":"2024-02-23T00:33:14.018582+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.036837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.036837+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":221,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.021436+0000","last_change":"2024-02-23T00:30:47.142583+0000","last_active":"2024-02-23T00:33:14.021436+0000","last_peered":"2024-02-23T00:33:14.021436+0000","last_clean":"2024-02-23T00:33:14.021436+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:33:14.021436+0000","last_undegraded":"2024-02-23T00:33:14.021436+0000","last_fullsized":"2024-02-23T00:33:14.021436+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:30:47.142508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.142508+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":202,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.082809+0000","last_change":"2024-02-23T00:30:34.861845+0000","last_active":"2024-02-23T00:33:14.082809+0000","last_peered":"2024-02-23T00:33:14.082809+0000","last_clean":"2024-02-23T00:33:14.082809+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:33:14.082809+0000","last_undegraded":"2024-02-23T00:33:14.082809+0000","last_fullsized":"2024-02-23T00:33:14.082809+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:34.861741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:34.861741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027907+0000","last_change":"2024-02-23T00:32:59.434164+0000","last_active":"2024-02-23T00:33:14.027907+0000","last_peered":"2024-02-23T00:33:14.027907+0000","last_clean":"2024-02-23T00:33:14.027907+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:33:14.027907+0000","last_undegraded":"2024-02-23T00:33:14.027907+0000","last_fullsized":"2024-02-23T00:33:14.027907+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.434079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.434079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":257,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.095508+0000","last_change":"2024-02-23T00:32:44.578143+0000","last_active":"2024-02-23T00:33:14.095508+0000","last_peered":"2024-02-23T00:33:14.095508+0000","last_clean":"2024-02-23T00:33:14.095508+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:33:14.095508+0000","last_undegraded":"2024-02-23T00:33:14.095508+0000","last_fullsized":"2024-02-23T00:33:14.095508+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:32:44.578068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.578068+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":213,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.017364+0000","last_change":"2024-02-23T00:32:11.912580+0000","last_active":"2024-02-23T00:33:14.017364+0000","last_peered":"2024-02-23T00:33:14.017364+0000","last_clean":"2024-02-23T00:33:14.017364+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:33:14.017364+0000","last_undegraded":"2024-02-23T00:33:14.017364+0000","last_fullsized":"2024-02-23T00:33:14.017364+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:11.912504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.912504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084225+0000","last_change":"2024-02-23T00:30:36.894317+0000","last_active":"2024-02-23T00:33:14.084225+0000","last_peered":"2024-02-23T00:33:14.084225+0000","last_clean":"2024-02-23T00:33:14.084225+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:33:14.084225+0000","last_undegraded":"2024-02-23T00:33:14.084225+0000","last_fullsized":"2024-02-23T00:33:14.084225+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:36.894220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:36.894220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.082979+0000","last_change":"2024-02-23T00:32:16.828560+0000","last_active":"2024-02-23T00:33:14.082979+0000","last_peered":"2024-02-23T00:33:14.082979+0000","last_clean":"2024-02-23T00:33:14.082979+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:33:14.082979+0000","last_undegraded":"2024-02-23T00:33:14.082979+0000","last_fullsized":"2024-02-23T00:33:14.082979+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_clean_scrub_stamp":"2024-02-23T00:32:16.828490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1","version":"148'9","reported_seq":178,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.089929+0000","last_change":"2024-02-23T00:32:41.020450+0000","last_active":"2024-02-23T00:33:14.089929+0000","last_peered":"2024-02-23T00:33:14.089929+0000","last_clean":"2024-02-23T00:33:14.089929+0000","last_became_active":"2024-02-23T00:32:41.020138+0000","last_became_peered":"2024-02-23T00:32:41.020138+0000","last_unstale":"2024-02-23T00:33:14.089929+0000","last_undegraded":"2024-02-23T00:33:14.089929+0000","last_fullsized":"2024-02-23T00:33:14.089929+0000","mapping_epoch":96,"log_start":"148'9","ondisk_log_start":"148'9","created":96,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":155,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084681+0000","last_change":"2024-02-23T00:30:48.691960+0000","last_active":"2024-02-23T00:33:14.084681+0000","last_peered":"2024-02-23T00:33:14.084681+0000","last_clean":"2024-02-23T00:33:14.084681+0000","last_became_active":"2024-02-23T00:30:48.691713+0000","last_became_peered":"2024-02-23T00:30:48.691713+0000","last_unstale":"2024-02-23T00:33:14.084681+0000","last_undegraded":"2024-02-23T00:33:14.084681+0000","last_fullsized":"2024-02-23T00:33:14.084681+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":249,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084029+0000","last_change":"2024-02-23T00:30:24.802266+0000","last_active":"2024-02-23T00:33:14.084029+0000","last_peered":"2024-02-23T00:33:14.084029+0000","last_clean":"2024-02-23T00:33:14.084029+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:33:14.084029+0000","last_undegraded":"2024-02-23T00:33:14.084029+0000","last_fullsized":"2024-02-23T00:33:14.084029+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:30:24.802134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:24.802134+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":211,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.028411+0000","last_change":"2024-02-23T00:32:14.243873+0000","last_active":"2024-02-23T00:33:14.028411+0000","last_peered":"2024-02-23T00:33:14.028411+0000","last_clean":"2024-02-23T00:33:14.028411+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:33:14.028411+0000","last_undegraded":"2024-02-23T00:33:14.028411+0000","last_fullsized":"2024-02-23T00:33:14.028411+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.243786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.243786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":200,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.025139+0000","last_change":"2024-02-23T00:32:07.275896+0000","last_active":"2024-02-23T00:33:14.025139+0000","last_peered":"2024-02-23T00:33:14.025139+0000","last_clean":"2024-02-23T00:33:14.025139+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:33:14.025139+0000","last_undegraded":"2024-02-23T00:33:14.025139+0000","last_fullsized":"2024-02-23T00:33:14.025139+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:07.275816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:07.275816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035402+0000","last_change":"2024-02-23T00:32:15.651736+0000","last_active":"2024-02-23T00:33:14.035402+0000","last_peered":"2024-02-23T00:33:14.035402+0000","last_clean":"2024-02-23T00:33:14.035402+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:33:14.035402+0000","last_undegraded":"2024-02-23T00:33:14.035402+0000","last_fullsized":"2024-02-23T00:33:14.035402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:15.651651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:32:15.651651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"148'7","reported_seq":181,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083296+0000","last_change":"2024-02-23T00:32:45.016687+0000","last_active":"2024-02-23T00:33:14.083296+0000","last_peered":"2024-02-23T00:33:14.083296+0000","last_clean":"2024-02-23T00:33:14.083296+0000","last_became_active":"2024-02-23T00:32:45.016438+0000","last_became_peered":"2024-02-23T00:32:45.016438+0000","last_unstale":"2024-02-23T00:33:14.083296+0000","last_undegraded":"2024-02-23T00:33:14.083296+0000","last_fullsized":"2024-02-23T00:33:14.083296+0000","mapping_epoch":96,"log_start":"148'7","ondisk_log_start":"148'7","created":96,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027798+0000","last_change":"2024-02-23T00:32:55.433102+0000","last_active":"2024-02-23T00:33:14.027798+0000","last_peered":"2024-02-23T00:33:14.027798+0000","last_clean":"2024-02-23T00:33:14.027798+0000","last_became_active":"2024-02-23T00:30:48.703232+0000","last_became_peered":"2024-02-23T00:30:48.703232+0000","last_unstale":"2024-02-23T00:33:14.027798+0000","last_undegraded":"2024-02-23T00:33:14.027798+0000","last_fullsized":"2024-02-23T00:33:14.027798+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:55.433031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:55.433031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":251,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084279+0000","last_change":"2024-02-23T00:30:37.899931+0000","last_active":"2024-02-23T00:33:14.084279+0000","last_peered":"2024-02-23T00:33:14.084279+0000","last_clean":"2024-02-23T00:33:14.084279+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:33:14.084279+0000","last_undegraded":"2024-02-23T00:33:14.084279+0000","last_fullsized":"2024-02-23T00:33:14.084279+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:30:37.899845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.899845+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.023255+0000","last_change":"2024-02-23T00:32:10.319807+0000","last_active":"2024-02-23T00:33:14.023255+0000","last_peered":"2024-02-23T00:33:14.023255+0000","last_clean":"2024-02-23T00:33:14.023255+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:33:14.023255+0000","last_undegraded":"2024-02-23T00:33:14.023255+0000","last_fullsized":"2024-02-23T00:33:14.023255+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.319744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.319744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":286,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.035133+0000","last_change":"2024-02-23T00:32:56.376450+0000","last_active":"2024-02-23T00:33:15.035133+0000","last_peered":"2024-02-23T00:33:15.035133+0000","last_clean":"2024-02-23T00:33:15.035133+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:33:15.035133+0000","last_undegraded":"2024-02-23T00:33:15.035133+0000","last_fullsized":"2024-02-23T00:33:15.035133+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.095319+0000","last_change":"2024-02-23T00:32:07.573617+0000","last_active":"2024-02-23T00:33:14.095319+0000","last_peered":"2024-02-23T00:33:14.095319+0000","last_clean":"2024-02-23T00:33:14.095319+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:33:14.095319+0000","last_undegraded":"2024-02-23T00:33:14.095319+0000","last_fullsized":"2024-02-23T00:33:14.095319+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:07.573555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:37.889336+0000","last_clean_scrub_stamp":"2024-02-23T00:32:07.573555+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.3","version":"148'14","reported_seq":197,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.028581+0000","last_change":"2024-02-23T00:33:05.054445+0000","last_active":"2024-02-23T00:33:14.028581+0000","last_peered":"2024-02-23T00:33:14.028581+0000","last_clean":"2024-02-23T00:33:14.028581+0000","last_became_active":"2024-02-23T00:33:05.054103+0000","last_became_peered":"2024-02-23T00:33:05.054103+0000","last_unstale":"2024-02-23T00:33:14.028581+0000","last_undegraded":"2024-02-23T00:33:14.028581+0000","last_fullsized":"2024-02-23T00:33:14.028581+0000","mapping_epoch":96,"log_start":"148'14","ondisk_log_start":"148'14","created":96,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.025829+0000","last_change":"2024-02-23T00:32:45.409616+0000","last_active":"2024-02-23T00:33:14.025829+0000","last_peered":"2024-02-23T00:33:14.025829+0000","last_clean":"2024-02-23T00:33:14.025829+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:33:14.025829+0000","last_undegraded":"2024-02-23T00:33:14.025829+0000","last_fullsized":"2024-02-23T00:33:14.025829+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.409548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.409548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"144'20","reported_seq":238,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.024765+0000","last_change":"2024-02-23T00:31:35.423292+0000","last_active":"2024-02-23T00:33:14.024765+0000","last_peered":"2024-02-23T00:33:14.024765+0000","last_clean":"2024-02-23T00:33:14.024765+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:33:14.024765+0000","last_undegraded":"2024-02-23T00:33:14.024765+0000","last_fullsized":"2024-02-23T00:33:14.024765+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.423227+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.086029+0000","last_change":"2024-02-23T00:32:48.536354+0000","last_active":"2024-02-23T00:33:14.086029+0000","last_peered":"2024-02-23T00:33:14.086029+0000","last_clean":"2024-02-23T00:33:14.086029+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:33:14.086029+0000","last_undegraded":"2024-02-23T00:33:14.086029+0000","last_fullsized":"2024-02-23T00:33:14.086029+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.536283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.536283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'9","reported_seq":290,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.053295+0000","last_change":"2024-02-23T00:32:43.990958+0000","last_active":"2024-02-23T00:33:15.053295+0000","last_peered":"2024-02-23T00:33:15.053295+0000","last_clean":"2024-02-23T00:33:15.053295+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:33:15.053295+0000","last_undegraded":"2024-02-23T00:33:15.053295+0000","last_fullsized":"2024-02-23T00:33:15.053295+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:32:43.990886+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.990886+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.021594+0000","last_change":"2024-02-23T00:32:56.678465+0000","last_active":"2024-02-23T00:33:14.021594+0000","last_peered":"2024-02-23T00:33:14.021594+0000","last_clean":"2024-02-23T00:33:14.021594+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:33:14.021594+0000","last_undegraded":"2024-02-23T00:33:14.021594+0000","last_fullsized":"2024-02-23T00:33:14.021594+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.678359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.678359+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":178,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035237+0000","last_change":"2024-02-23T00:32:42.691730+0000","last_active":"2024-02-23T00:33:14.035237+0000","last_peered":"2024-02-23T00:33:14.035237+0000","last_clean":"2024-02-23T00:33:14.035237+0000","last_became_active":"2024-02-23T00:32:37.017015+0000","last_became_peered":"2024-02-23T00:32:37.017015+0000","last_unstale":"2024-02-23T00:33:14.035237+0000","last_undegraded":"2024-02-23T00:33:14.035237+0000","last_fullsized":"2024-02-23T00:33:14.035237+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.691630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.691630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":155,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084365+0000","last_change":"2024-02-23T00:30:48.717524+0000","last_active":"2024-02-23T00:33:14.084365+0000","last_peered":"2024-02-23T00:33:14.084365+0000","last_clean":"2024-02-23T00:33:14.084365+0000","last_became_active":"2024-02-23T00:30:48.716104+0000","last_became_peered":"2024-02-23T00:30:48.716104+0000","last_unstale":"2024-02-23T00:33:14.084365+0000","last_undegraded":"2024-02-23T00:33:14.084365+0000","last_fullsized":"2024-02-23T00:33:14.084365+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":224,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091631+0000","last_change":"2024-02-23T00:32:50.448701+0000","last_active":"2024-02-23T00:33:14.091631+0000","last_peered":"2024-02-23T00:33:14.091631+0000","last_clean":"2024-02-23T00:33:14.091631+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:33:14.091631+0000","last_undegraded":"2024-02-23T00:33:14.091631+0000","last_fullsized":"2024-02-23T00:33:14.091631+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:32:50.448595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.448595+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":211,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027628+0000","last_change":"2024-02-23T00:32:12.234058+0000","last_active":"2024-02-23T00:33:14.027628+0000","last_peered":"2024-02-23T00:33:14.027628+0000","last_clean":"2024-02-23T00:33:14.027628+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:33:14.027628+0000","last_undegraded":"2024-02-23T00:33:14.027628+0000","last_fullsized":"2024-02-23T00:33:14.027628+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.233960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.233960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.082720+0000","last_change":"2024-02-23T00:32:09.978443+0000","last_active":"2024-02-23T00:33:14.082720+0000","last_peered":"2024-02-23T00:33:14.082720+0000","last_clean":"2024-02-23T00:33:14.082720+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:33:14.082720+0000","last_undegraded":"2024-02-23T00:33:14.082720+0000","last_fullsized":"2024-02-23T00:33:14.082720+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.978377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.978377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":212,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.087046+0000","last_change":"2024-02-23T00:32:53.602946+0000","last_active":"2024-02-23T00:33:14.087046+0000","last_peered":"2024-02-23T00:33:14.087046+0000","last_clean":"2024-02-23T00:33:14.087046+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:33:14.087046+0000","last_undegraded":"2024-02-23T00:33:14.087046+0000","last_fullsized":"2024-02-23T00:33:14.087046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.602880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.602880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.5","version":"148'9","reported_seq":192,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.021678+0000","last_change":"2024-02-23T00:32:57.090735+0000","last_active":"2024-02-23T00:33:14.021678+0000","last_peered":"2024-02-23T00:33:14.021678+0000","last_clean":"2024-02-23T00:33:14.021678+0000","last_became_active":"2024-02-23T00:32:57.090327+0000","last_became_peered":"2024-02-23T00:32:57.090327+0000","last_unstale":"2024-02-23T00:33:14.021678+0000","last_undegraded":"2024-02-23T00:33:14.021678+0000","last_fullsized":"2024-02-23T00:33:14.021678+0000","mapping_epoch":96,"log_start":"148'9","ondisk_log_start":"148'9","created":96,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":155,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.018668+0000","last_change":"2024-02-23T00:30:48.668969+0000","last_active":"2024-02-23T00:33:14.018668+0000","last_peered":"2024-02-23T00:33:14.018668+0000","last_clean":"2024-02-23T00:33:14.018668+0000","last_became_active":"2024-02-23T00:30:48.666975+0000","last_became_peered":"2024-02-23T00:30:48.666975+0000","last_unstale":"2024-02-23T00:33:14.018668+0000","last_undegraded":"2024-02-23T00:33:14.018668+0000","last_fullsized":"2024-02-23T00:33:14.018668+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":246,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.020137+0000","last_change":"2024-02-23T00:30:41.077799+0000","last_active":"2024-02-23T00:33:14.020137+0000","last_peered":"2024-02-23T00:33:14.020137+0000","last_clean":"2024-02-23T00:33:14.020137+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:33:14.020137+0000","last_undegraded":"2024-02-23T00:33:14.020137+0000","last_fullsized":"2024-02-23T00:33:14.020137+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:30:41.077681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.077681+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.092026+0000","last_change":"2024-02-23T00:31:32.874749+0000","last_active":"2024-02-23T00:33:14.092026+0000","last_peered":"2024-02-23T00:33:14.092026+0000","last_clean":"2024-02-23T00:33:14.092026+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:33:14.092026+0000","last_undegraded":"2024-02-23T00:33:14.092026+0000","last_fullsized":"2024-02-23T00:33:14.092026+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:32.874678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:31:32.874678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091714+0000","last_change":"2024-02-23T00:30:30.486006+0000","last_active":"2024-02-23T00:33:14.091714+0000","last_peered":"2024-02-23T00:33:14.091714+0000","last_clean":"2024-02-23T00:33:14.091714+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:33:14.091714+0000","last_undegraded":"2024-02-23T00:33:14.091714+0000","last_fullsized":"2024-02-23T00:33:14.091714+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:30.485900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:53.665092+0000","last_clean_scrub_stamp":"2024-02-23T00:30:30.485900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":202,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.082740+0000","last_change":"2024-02-23T00:30:47.843046+0000","last_active":"2024-02-23T00:33:14.082740+0000","last_peered":"2024-02-23T00:33:14.082740+0000","last_clean":"2024-02-23T00:33:14.082740+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:33:14.082740+0000","last_undegraded":"2024-02-23T00:33:14.082740+0000","last_fullsized":"2024-02-23T00:33:14.082740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.842931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.4","version":"98'2","reported_seq":189,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.086137+0000","last_change":"2024-02-23T00:33:01.049678+0000","last_active":"2024-02-23T00:33:14.086137+0000","last_peered":"2024-02-23T00:33:14.086137+0000","last_clean":"2024-02-23T00:33:14.086137+0000","last_became_active":"2024-02-23T00:33:01.049360+0000","last_became_peered":"2024-02-23T00:33:01.049360+0000","last_unstale":"2024-02-23T00:33:14.086137+0000","last_undegraded":"2024-02-23T00:33:14.086137+0000","last_fullsized":"2024-02-23T00:33:14.086137+0000","mapping_epoch":96,"log_start":"98'2","ondisk_log_start":"98'2","created":96,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":166,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032561+0000","last_change":"2024-02-23T00:30:48.705018+0000","last_active":"2024-02-23T00:33:14.032561+0000","last_peered":"2024-02-23T00:33:14.032561+0000","last_clean":"2024-02-23T00:33:14.032561+0000","last_became_active":"2024-02-23T00:30:48.703281+0000","last_became_peered":"2024-02-23T00:30:48.703281+0000","last_unstale":"2024-02-23T00:33:14.032561+0000","last_undegraded":"2024-02-23T00:33:14.032561+0000","last_fullsized":"2024-02-23T00:33:14.032561+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.017383+0000","last_change":"2024-02-23T00:32:11.989425+0000","last_active":"2024-02-23T00:33:14.017383+0000","last_peered":"2024-02-23T00:33:14.017383+0000","last_clean":"2024-02-23T00:33:14.017383+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:33:14.017383+0000","last_undegraded":"2024-02-23T00:33:14.017383+0000","last_fullsized":"2024-02-23T00:33:14.017383+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:32:11.989333+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.989333+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.025546+0000","last_change":"2024-02-23T00:32:40.343941+0000","last_active":"2024-02-23T00:33:14.025546+0000","last_peered":"2024-02-23T00:33:14.025546+0000","last_clean":"2024-02-23T00:33:14.025546+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:33:14.025546+0000","last_undegraded":"2024-02-23T00:33:14.025546+0000","last_fullsized":"2024-02-23T00:33:14.025546+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.343863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.343863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.028324+0000","last_change":"2024-02-23T00:32:47.484343+0000","last_active":"2024-02-23T00:33:14.028324+0000","last_peered":"2024-02-23T00:33:14.028324+0000","last_clean":"2024-02-23T00:33:14.028324+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:33:14.028324+0000","last_undegraded":"2024-02-23T00:33:14.028324+0000","last_fullsized":"2024-02-23T00:33:14.028324+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:47.484285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.484285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035796+0000","last_change":"2024-02-23T00:32:12.721784+0000","last_active":"2024-02-23T00:33:14.035796+0000","last_peered":"2024-02-23T00:33:14.035796+0000","last_clean":"2024-02-23T00:33:14.035796+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:33:14.035796+0000","last_undegraded":"2024-02-23T00:33:14.035796+0000","last_fullsized":"2024-02-23T00:33:14.035796+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.721719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.721719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.7","version":"148'7","reported_seq":197,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.082971+0000","last_change":"2024-02-23T00:32:49.030723+0000","last_active":"2024-02-23T00:33:14.082971+0000","last_peered":"2024-02-23T00:33:14.082971+0000","last_clean":"2024-02-23T00:33:14.082971+0000","last_became_active":"2024-02-23T00:32:49.030411+0000","last_became_peered":"2024-02-23T00:32:49.030411+0000","last_unstale":"2024-02-23T00:33:14.082971+0000","last_undegraded":"2024-02-23T00:33:14.082971+0000","last_fullsized":"2024-02-23T00:33:14.082971+0000","mapping_epoch":96,"log_start":"148'7","ondisk_log_start":"148'7","created":96,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7","last_scrub_stamp":"2024-02-23T00:32:43.845548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.845548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":152,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091894+0000","last_change":"2024-02-23T00:30:48.696251+0000","last_active":"2024-02-23T00:33:14.091894+0000","last_peered":"2024-02-23T00:33:14.091894+0000","last_clean":"2024-02-23T00:33:14.091894+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:33:14.091894+0000","last_undegraded":"2024-02-23T00:33:14.091894+0000","last_fullsized":"2024-02-23T00:33:14.091894+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":247,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083374+0000","last_change":"2024-02-23T00:31:27.176319+0000","last_active":"2024-02-23T00:33:14.083374+0000","last_peered":"2024-02-23T00:33:14.083374+0000","last_clean":"2024-02-23T00:33:14.083374+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:33:14.083374+0000","last_undegraded":"2024-02-23T00:33:14.083374+0000","last_fullsized":"2024-02-23T00:33:14.083374+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:31:27.176199+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:31:27.176199+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027844+0000","last_change":"2024-02-23T00:32:44.494080+0000","last_active":"2024-02-23T00:33:14.027844+0000","last_peered":"2024-02-23T00:33:14.027844+0000","last_clean":"2024-02-23T00:33:14.027844+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:33:14.027844+0000","last_undegraded":"2024-02-23T00:33:14.027844+0000","last_fullsized":"2024-02-23T00:33:14.027844+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.494004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.494004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.034012+0000","last_change":"2024-02-23T00:32:46.655443+0000","last_active":"2024-02-23T00:33:14.034012+0000","last_peered":"2024-02-23T00:33:14.034012+0000","last_clean":"2024-02-23T00:33:14.034012+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:33:14.034012+0000","last_undegraded":"2024-02-23T00:33:14.034012+0000","last_fullsized":"2024-02-23T00:33:14.034012+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.655378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.655378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":202,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.082964+0000","last_change":"2024-02-23T00:30:31.829814+0000","last_active":"2024-02-23T00:33:14.082964+0000","last_peered":"2024-02-23T00:33:14.082964+0000","last_clean":"2024-02-23T00:33:14.082964+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:33:14.082964+0000","last_undegraded":"2024-02-23T00:33:14.082964+0000","last_fullsized":"2024-02-23T00:33:14.082964+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:31.829697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:31.829697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.6","version":"148'11","reported_seq":203,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.022291+0000","last_change":"2024-02-23T00:32:53.032240+0000","last_active":"2024-02-23T00:33:14.022291+0000","last_peered":"2024-02-23T00:33:14.022291+0000","last_clean":"2024-02-23T00:33:14.022291+0000","last_became_active":"2024-02-23T00:32:53.031873+0000","last_became_peered":"2024-02-23T00:32:53.031873+0000","last_unstale":"2024-02-23T00:33:14.022291+0000","last_undegraded":"2024-02-23T00:33:14.022291+0000","last_fullsized":"2024-02-23T00:33:14.022291+0000","mapping_epoch":96,"log_start":"148'11","ondisk_log_start":"148'11","created":96,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":155,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.020282+0000","last_change":"2024-02-23T00:30:48.723611+0000","last_active":"2024-02-23T00:33:14.020282+0000","last_peered":"2024-02-23T00:33:14.020282+0000","last_clean":"2024-02-23T00:33:14.020282+0000","last_became_active":"2024-02-23T00:30:48.722602+0000","last_became_peered":"2024-02-23T00:30:48.722602+0000","last_unstale":"2024-02-23T00:33:14.020282+0000","last_undegraded":"2024-02-23T00:33:14.020282+0000","last_fullsized":"2024-02-23T00:33:14.020282+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"92'22","reported_seq":247,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.095743+0000","last_change":"2024-02-23T00:32:43.560670+0000","last_active":"2024-02-23T00:33:14.095743+0000","last_peered":"2024-02-23T00:33:14.095743+0000","last_clean":"2024-02-23T00:33:14.095743+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:33:14.095743+0000","last_undegraded":"2024-02-23T00:33:14.095743+0000","last_fullsized":"2024-02-23T00:33:14.095743+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:32:43.560587+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.560587+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.025963+0000","last_change":"2024-02-23T00:31:20.333610+0000","last_active":"2024-02-23T00:33:14.025963+0000","last_peered":"2024-02-23T00:33:14.025963+0000","last_clean":"2024-02-23T00:33:14.025963+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:33:14.025963+0000","last_undegraded":"2024-02-23T00:33:14.025963+0000","last_fullsized":"2024-02-23T00:33:14.025963+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:20.333547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:20.333547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":290,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-23T00:33:15.193605+0000","last_change":"2024-02-23T00:31:19.971484+0000","last_active":"2024-02-23T00:33:15.193605+0000","last_peered":"2024-02-23T00:33:15.193605+0000","last_clean":"2024-02-23T00:33:15.193605+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:33:15.193605+0000","last_undegraded":"2024-02-23T00:33:15.193605+0000","last_fullsized":"2024-02-23T00:33:15.193605+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:31:19.971419+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.971419+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035128+0000","last_change":"2024-02-23T00:32:50.706776+0000","last_active":"2024-02-23T00:33:14.035128+0000","last_peered":"2024-02-23T00:33:14.035128+0000","last_clean":"2024-02-23T00:33:14.035128+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:33:14.035128+0000","last_undegraded":"2024-02-23T00:33:14.035128+0000","last_fullsized":"2024-02-23T00:33:14.035128+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:32:50.706693+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.706693+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":178,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.033962+0000","last_change":"2024-02-23T00:32:59.681117+0000","last_active":"2024-02-23T00:33:14.033962+0000","last_peered":"2024-02-23T00:33:14.033962+0000","last_clean":"2024-02-23T00:33:14.033962+0000","last_became_active":"2024-02-23T00:32:09.024182+0000","last_became_peered":"2024-02-23T00:32:09.024182+0000","last_unstale":"2024-02-23T00:33:14.033962+0000","last_undegraded":"2024-02-23T00:33:14.033962+0000","last_fullsized":"2024-02-23T00:33:14.033962+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.681034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:59.681034+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.681034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":154,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035163+0000","last_change":"2024-02-23T00:30:48.748188+0000","last_active":"2024-02-23T00:33:14.035163+0000","last_peered":"2024-02-23T00:33:14.035163+0000","last_clean":"2024-02-23T00:33:14.035163+0000","last_became_active":"2024-02-23T00:30:48.745982+0000","last_became_peered":"2024-02-23T00:30:48.745982+0000","last_unstale":"2024-02-23T00:33:14.035163+0000","last_undegraded":"2024-02-23T00:33:14.035163+0000","last_fullsized":"2024-02-23T00:33:14.035163+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":243,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.027972+0000","last_change":"2024-02-23T00:32:59.459127+0000","last_active":"2024-02-23T00:33:14.027972+0000","last_peered":"2024-02-23T00:33:14.027972+0000","last_clean":"2024-02-23T00:33:14.027972+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:33:14.027972+0000","last_undegraded":"2024-02-23T00:33:14.027972+0000","last_fullsized":"2024-02-23T00:33:14.027972+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:32:59.459053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.459053+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":213,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083103+0000","last_change":"2024-02-23T00:32:37.932120+0000","last_active":"2024-02-23T00:33:14.083103+0000","last_peered":"2024-02-23T00:33:14.083103+0000","last_clean":"2024-02-23T00:33:14.083103+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:33:14.083103+0000","last_undegraded":"2024-02-23T00:33:14.083103+0000","last_fullsized":"2024-02-23T00:33:14.083103+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.932055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:49.056184+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.932055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.095824+0000","last_change":"2024-02-23T00:32:35.624529+0000","last_active":"2024-02-23T00:33:14.095824+0000","last_peered":"2024-02-23T00:33:14.095824+0000","last_clean":"2024-02-23T00:33:14.095824+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:33:14.095824+0000","last_undegraded":"2024-02-23T00:33:14.095824+0000","last_fullsized":"2024-02-23T00:33:14.095824+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.624461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.624461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091842+0000","last_change":"2024-02-23T00:32:10.145767+0000","last_active":"2024-02-23T00:33:14.091842+0000","last_peered":"2024-02-23T00:33:14.091842+0000","last_clean":"2024-02-23T00:33:14.091842+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:33:14.091842+0000","last_undegraded":"2024-02-23T00:33:14.091842+0000","last_fullsized":"2024-02-23T00:33:14.091842+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.145685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.145685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.8","version":"98'2","reported_seq":173,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084988+0000","last_change":"2024-02-23T00:32:13.036407+0000","last_active":"2024-02-23T00:33:14.084988+0000","last_peered":"2024-02-23T00:33:14.084988+0000","last_clean":"2024-02-23T00:33:14.084988+0000","last_became_active":"2024-02-23T00:32:13.036071+0000","last_became_peered":"2024-02-23T00:32:13.036071+0000","last_unstale":"2024-02-23T00:33:14.084988+0000","last_undegraded":"2024-02-23T00:33:14.084988+0000","last_fullsized":"2024-02-23T00:33:14.084988+0000","mapping_epoch":96,"log_start":"98'2","ondisk_log_start":"98'2","created":96,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:30:45.807294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":159,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035995+0000","last_change":"2024-02-23T00:32:58.681718+0000","last_active":"2024-02-23T00:33:14.035995+0000","last_peered":"2024-02-23T00:33:14.035995+0000","last_clean":"2024-02-23T00:33:14.035995+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:33:14.035995+0000","last_undegraded":"2024-02-23T00:33:14.035995+0000","last_fullsized":"2024-02-23T00:33:14.035995+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.681626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.681626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":236,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.084636+0000","last_change":"2024-02-23T00:31:31.806791+0000","last_active":"2024-02-23T00:33:14.084636+0000","last_peered":"2024-02-23T00:33:14.084636+0000","last_clean":"2024-02-23T00:33:14.084636+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:33:14.084636+0000","last_undegraded":"2024-02-23T00:33:14.084636+0000","last_fullsized":"2024-02-23T00:33:14.084636+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:31:31.806687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.806687+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":211,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032510+0000","last_change":"2024-02-23T00:30:40.183533+0000","last_active":"2024-02-23T00:33:14.032510+0000","last_peered":"2024-02-23T00:33:14.032510+0000","last_clean":"2024-02-23T00:33:14.032510+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:33:14.032510+0000","last_undegraded":"2024-02-23T00:33:14.032510+0000","last_fullsized":"2024-02-23T00:33:14.032510+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:30:40.183453+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":203,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.092192+0000","last_change":"2024-02-23T00:31:38.887671+0000","last_active":"2024-02-23T00:33:14.092192+0000","last_peered":"2024-02-23T00:33:14.092192+0000","last_clean":"2024-02-23T00:33:14.092192+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:33:14.092192+0000","last_undegraded":"2024-02-23T00:33:14.092192+0000","last_fullsized":"2024-02-23T00:33:14.092192+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.887597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":202,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.021901+0000","last_change":"2024-02-23T00:32:12.983101+0000","last_active":"2024-02-23T00:33:14.021901+0000","last_peered":"2024-02-23T00:33:14.021901+0000","last_clean":"2024-02-23T00:33:14.021901+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:33:14.021901+0000","last_undegraded":"2024-02-23T00:33:14.021901+0000","last_fullsized":"2024-02-23T00:33:14.021901+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.983023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.983023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.025747+0000","last_change":"2024-02-23T00:30:48.711374+0000","last_active":"2024-02-23T00:33:14.025747+0000","last_peered":"2024-02-23T00:33:14.025747+0000","last_clean":"2024-02-23T00:33:14.025747+0000","last_became_active":"2024-02-23T00:30:48.710049+0000","last_became_peered":"2024-02-23T00:30:48.710049+0000","last_unstale":"2024-02-23T00:33:14.025747+0000","last_undegraded":"2024-02-23T00:33:14.025747+0000","last_fullsized":"2024-02-23T00:33:14.025747+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":250,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.017216+0000","last_change":"2024-02-23T00:30:29.844070+0000","last_active":"2024-02-23T00:33:14.017216+0000","last_peered":"2024-02-23T00:33:14.017216+0000","last_clean":"2024-02-23T00:33:14.017216+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:33:14.017216+0000","last_undegraded":"2024-02-23T00:33:14.017216+0000","last_fullsized":"2024-02-23T00:33:14.017216+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:30:29.843954+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.843954+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":208,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.082638+0000","last_change":"2024-02-23T00:30:32.878666+0000","last_active":"2024-02-23T00:33:14.082638+0000","last_peered":"2024-02-23T00:33:14.082638+0000","last_clean":"2024-02-23T00:33:14.082638+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:33:14.082638+0000","last_undegraded":"2024-02-23T00:33:14.082638+0000","last_fullsized":"2024-02-23T00:33:14.082638+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:32.878569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:30:32.878569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.020939+0000","last_change":"2024-02-23T00:31:45.961466+0000","last_active":"2024-02-23T00:33:14.020939+0000","last_peered":"2024-02-23T00:33:14.020939+0000","last_clean":"2024-02-23T00:33:14.020939+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:33:14.020939+0000","last_undegraded":"2024-02-23T00:33:14.020939+0000","last_fullsized":"2024-02-23T00:33:14.020939+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:45.961386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:45.961386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083938+0000","last_change":"2024-02-23T00:32:46.021732+0000","last_active":"2024-02-23T00:33:14.083938+0000","last_peered":"2024-02-23T00:33:14.083938+0000","last_clean":"2024-02-23T00:33:14.083938+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:33:14.083938+0000","last_undegraded":"2024-02-23T00:33:14.083938+0000","last_fullsized":"2024-02-23T00:33:14.083938+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.021662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.a","version":"148'14","reported_seq":193,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.085730+0000","last_change":"2024-02-23T00:32:51.819469+0000","last_active":"2024-02-23T00:33:14.085730+0000","last_peered":"2024-02-23T00:33:14.085730+0000","last_clean":"2024-02-23T00:33:14.085730+0000","last_became_active":"2024-02-23T00:32:05.034399+0000","last_became_peered":"2024-02-23T00:32:05.034399+0000","last_unstale":"2024-02-23T00:33:14.085730+0000","last_undegraded":"2024-02-23T00:33:14.085730+0000","last_fullsized":"2024-02-23T00:33:14.085730+0000","mapping_epoch":96,"log_start":"148'14","ondisk_log_start":"148'14","created":96,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'14","last_scrub_stamp":"2024-02-23T00:32:51.819400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:45.807294+0000","last_clean_scrub_stamp":"2024-02-23T00:32:51.819400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.030525+0000","last_change":"2024-02-23T00:30:48.679036+0000","last_active":"2024-02-23T00:33:14.030525+0000","last_peered":"2024-02-23T00:33:14.030525+0000","last_clean":"2024-02-23T00:33:14.030525+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:33:14.030525+0000","last_undegraded":"2024-02-23T00:33:14.030525+0000","last_fullsized":"2024-02-23T00:33:14.030525+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":257,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.019354+0000","last_change":"2024-02-23T00:32:44.870419+0000","last_active":"2024-02-23T00:33:14.019354+0000","last_peered":"2024-02-23T00:33:14.019354+0000","last_clean":"2024-02-23T00:33:14.019354+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:33:14.019354+0000","last_undegraded":"2024-02-23T00:33:14.019354+0000","last_fullsized":"2024-02-23T00:33:14.019354+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:32:44.870335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.870335+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":211,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.034495+0000","last_change":"2024-02-23T00:32:45.663968+0000","last_active":"2024-02-23T00:33:14.034495+0000","last_peered":"2024-02-23T00:33:14.034495+0000","last_clean":"2024-02-23T00:33:14.034495+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:33:14.034495+0000","last_undegraded":"2024-02-23T00:33:14.034495+0000","last_fullsized":"2024-02-23T00:33:14.034495+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.663905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.663905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":200,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.028391+0000","last_change":"2024-02-23T00:30:41.121986+0000","last_active":"2024-02-23T00:33:14.028391+0000","last_peered":"2024-02-23T00:33:14.028391+0000","last_clean":"2024-02-23T00:33:14.028391+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:33:14.028391+0000","last_undegraded":"2024-02-23T00:33:14.028391+0000","last_fullsized":"2024-02-23T00:33:14.028391+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:41.121916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:41.121916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.088143+0000","last_change":"2024-02-23T00:32:39.644590+0000","last_active":"2024-02-23T00:33:14.088143+0000","last_peered":"2024-02-23T00:33:14.088143+0000","last_clean":"2024-02-23T00:33:14.088143+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:33:14.088143+0000","last_undegraded":"2024-02-23T00:33:14.088143+0000","last_fullsized":"2024-02-23T00:33:14.088143+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:39.644483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:32:39.644483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":159,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.032838+0000","last_change":"2024-02-23T00:32:35.725253+0000","last_active":"2024-02-23T00:33:14.032838+0000","last_peered":"2024-02-23T00:33:14.032838+0000","last_clean":"2024-02-23T00:33:14.032838+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:33:14.032838+0000","last_undegraded":"2024-02-23T00:33:14.032838+0000","last_fullsized":"2024-02-23T00:33:14.032838+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.725182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.725182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":239,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.020420+0000","last_change":"2024-02-23T00:32:58.673712+0000","last_active":"2024-02-23T00:33:14.020420+0000","last_peered":"2024-02-23T00:33:14.020420+0000","last_clean":"2024-02-23T00:33:14.020420+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:33:14.020420+0000","last_undegraded":"2024-02-23T00:33:14.020420+0000","last_fullsized":"2024-02-23T00:33:14.020420+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:32:58.673628+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.673628+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":211,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035485+0000","last_change":"2024-02-23T00:32:36.714218+0000","last_active":"2024-02-23T00:33:14.035485+0000","last_peered":"2024-02-23T00:33:14.035485+0000","last_clean":"2024-02-23T00:33:14.035485+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:33:14.035485+0000","last_undegraded":"2024-02-23T00:33:14.035485+0000","last_fullsized":"2024-02-23T00:33:14.035485+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.714139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.714139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035785+0000","last_change":"2024-02-23T00:32:44.640411+0000","last_active":"2024-02-23T00:33:14.035785+0000","last_peered":"2024-02-23T00:33:14.035785+0000","last_clean":"2024-02-23T00:33:14.035785+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:33:14.035785+0000","last_undegraded":"2024-02-23T00:33:14.035785+0000","last_fullsized":"2024-02-23T00:33:14.035785+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.640331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.640331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":198,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.024234+0000","last_change":"2024-02-23T00:31:39.494827+0000","last_active":"2024-02-23T00:33:14.024234+0000","last_peered":"2024-02-23T00:33:14.024234+0000","last_clean":"2024-02-23T00:33:14.024234+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:33:14.024234+0000","last_undegraded":"2024-02-23T00:33:14.024234+0000","last_fullsized":"2024-02-23T00:33:14.024234+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:31:39.494737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":152,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.089845+0000","last_change":"2024-02-23T00:30:48.725798+0000","last_active":"2024-02-23T00:33:14.089845+0000","last_peered":"2024-02-23T00:33:14.089845+0000","last_clean":"2024-02-23T00:33:14.089845+0000","last_became_active":"2024-02-23T00:30:48.725162+0000","last_became_peered":"2024-02-23T00:30:48.725162+0000","last_unstale":"2024-02-23T00:33:14.089845+0000","last_undegraded":"2024-02-23T00:33:14.089845+0000","last_fullsized":"2024-02-23T00:33:14.089845+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":242,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.018224+0000","last_change":"2024-02-23T00:32:10.903259+0000","last_active":"2024-02-23T00:33:14.018224+0000","last_peered":"2024-02-23T00:33:14.018224+0000","last_clean":"2024-02-23T00:33:14.018224+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:33:14.018224+0000","last_undegraded":"2024-02-23T00:33:14.018224+0000","last_fullsized":"2024-02-23T00:33:14.018224+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:32:10.903186+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.903186+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.083682+0000","last_change":"2024-02-23T00:31:22.137283+0000","last_active":"2024-02-23T00:33:14.083682+0000","last_peered":"2024-02-23T00:33:14.083682+0000","last_clean":"2024-02-23T00:33:14.083682+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:33:14.083682+0000","last_undegraded":"2024-02-23T00:33:14.083682+0000","last_fullsized":"2024-02-23T00:33:14.083682+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:22.137166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:22.137166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.089489+0000","last_change":"2024-02-23T00:31:26.920222+0000","last_active":"2024-02-23T00:33:14.089489+0000","last_peered":"2024-02-23T00:33:14.089489+0000","last_clean":"2024-02-23T00:33:14.089489+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:33:14.089489+0000","last_undegraded":"2024-02-23T00:33:14.089489+0000","last_fullsized":"2024-02-23T00:33:14.089489+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:26.920133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:26.920133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.019254+0000","last_change":"2024-02-23T00:31:37.065941+0000","last_active":"2024-02-23T00:33:14.019254+0000","last_peered":"2024-02-23T00:33:14.019254+0000","last_clean":"2024-02-23T00:33:14.019254+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:33:14.019254+0000","last_undegraded":"2024-02-23T00:33:14.019254+0000","last_fullsized":"2024-02-23T00:33:14.019254+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:31:37.065865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":154,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.035894+0000","last_change":"2024-02-23T00:30:48.698701+0000","last_active":"2024-02-23T00:33:14.035894+0000","last_peered":"2024-02-23T00:33:14.035894+0000","last_clean":"2024-02-23T00:33:14.035894+0000","last_became_active":"2024-02-23T00:30:48.697963+0000","last_became_peered":"2024-02-23T00:30:48.697963+0000","last_unstale":"2024-02-23T00:33:14.035894+0000","last_undegraded":"2024-02-23T00:33:14.035894+0000","last_fullsized":"2024-02-23T00:33:14.035894+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"63'39","reported_seq":291,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.082963+0000","last_change":"2024-02-23T00:30:23.839691+0000","last_active":"2024-02-23T00:33:14.082963+0000","last_peered":"2024-02-23T00:33:14.082963+0000","last_clean":"2024-02-23T00:33:14.082963+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:33:14.082963+0000","last_undegraded":"2024-02-23T00:33:14.082963+0000","last_fullsized":"2024-02-23T00:33:14.082963+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'39","last_scrub_stamp":"2024-02-23T00:30:23.839568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:23.839568+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":213,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.095481+0000","last_change":"2024-02-23T00:32:13.602518+0000","last_active":"2024-02-23T00:33:14.095481+0000","last_peered":"2024-02-23T00:33:14.095481+0000","last_clean":"2024-02-23T00:33:14.095481+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:33:14.095481+0000","last_undegraded":"2024-02-23T00:33:14.095481+0000","last_fullsized":"2024-02-23T00:33:14.095481+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:13.602450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.602450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.018566+0000","last_change":"2024-02-23T00:31:40.071364+0000","last_active":"2024-02-23T00:33:14.018566+0000","last_peered":"2024-02-23T00:33:14.018566+0000","last_clean":"2024-02-23T00:33:14.018566+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:33:14.018566+0000","last_undegraded":"2024-02-23T00:33:14.018566+0000","last_fullsized":"2024-02-23T00:33:14.018566+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.071294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.071294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":202,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.017228+0000","last_change":"2024-02-23T00:30:46.848126+0000","last_active":"2024-02-23T00:33:14.017228+0000","last_peered":"2024-02-23T00:33:14.017228+0000","last_clean":"2024-02-23T00:33:14.017228+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:33:14.017228+0000","last_undegraded":"2024-02-23T00:33:14.017228+0000","last_fullsized":"2024-02-23T00:33:14.017228+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.848063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.848063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":157,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-23T00:33:14.091181+0000","last_change":"2024-02-23T00:32:56.396227+0000","last_active":"2024-02-23T00:33:14.091181+0000","last_peered":"2024-02-23T00:33:14.091181+0000","last_clean":"2024-02-23T00:33:14.091181+0000","last_became_active":"2024-02-23T00:30:48.722569+0000","last_became_peered":"2024-02-23T00:30:48.722569+0000","last_unstale":"2024-02-23T00:33:14.091181+0000","last_undegraded":"2024-02-23T00:33:14.091181+0000","last_fullsized":"2024-02-23T00:33:14.091181+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.396150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.396150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":164,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":1226,"num_objects":123,"num_object_clones":5,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":1,"num_read":93,"num_read_kb":1,"num_write":536,"num_write_kb":532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1486848,"data_stored":3678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1768,"ondisk_log_size":1768,"up":96,"acting":96,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":167,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":19,"num_pg":11,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":640,"ondisk_log_size":640,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332171,"num_pgs":117,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309344,"kb_used_data":3920,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435808,"statfs":{"total":95995035648,"available":95678267392,"internally_reserved":0,"allocated":4014080,"data_stored":2239452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.0830000000000002,"15min":1.028},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":57.401000000000003,"5min":61.048999999999999,"15min":61.048999999999999},"last":1.371},{"interface":"front","average":{"1min":3.6440000000000001,"5min":2.6549999999999998,"15min":1.2390000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":65.558999999999997,"5min":73.870999999999995,"15min":73.870999999999995},"last":1.1639999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":2.028,"5min":1.026,"15min":0.69499999999999995},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.869,"5min":1.044,"15min":0.68400000000000005},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":29.423999999999999,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.51600000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":0.95499999999999996,"15min":0.66900000000000004},"min":{"1min":0.33100000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":9.5190000000000001,"5min":16.603999999999999,"15min":16.603999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.753,"15min":0.59999999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":7.5339999999999998,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.48099999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":3.2280000000000002,"5min":1.871,"15min":0.97099999999999997},"min":{"1min":0.35899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":57.729999999999997,"5min":57.729999999999997,"15min":57.729999999999997},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.613,"15min":0.89400000000000002},"min":{"1min":0.31900000000000001,"5min":0.312,"15min":0.312},"max":{"1min":27.573,"5min":27.573,"15min":27.573},"last":0.80300000000000005}]},{"osd":4,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.41899999999999998,"15min":0.41199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":1.706},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.43099999999999999,"15min":0.42499999999999999},"min":{"1min":0.25700000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":3.294}]},{"osd":5,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.43099999999999999,"15min":0.42799999999999999},"min":{"1min":0.27200000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.45600000000000002,"15min":0.435},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72099999999999997,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":1.1950000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.48099999999999998,"15min":0.47299999999999998},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.68899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":2.1339999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.48099999999999998,"15min":0.46100000000000002},"min":{"1min":0.30399999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.83299999999999996}]}]},{"osd":6,"up_from":44,"seq":188978561104,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313444,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431708,"statfs":{"total":95995035648,"available":95674068992,"internally_reserved":0,"allocated":4018176,"data_stored":2238082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":1.7090000000000001,"15min":0.90100000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.63100000000000001,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.71199999999999997,"15min":0.58399999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.70999999999999996,"5min":9.468,"15min":9.468},"last":0.38200000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":2.0299999999999998,"15min":1.0129999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.70899999999999996,"5min":59.347999999999999,"15min":59.347999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":2.8730000000000002,"15min":1.2889999999999999},"min":{"1min":0.28299999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.63800000000000001,"5min":87.221000000000004,"15min":87.221000000000004},"last":0.436}]},{"osd":2,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":1.427,"15min":0.80700000000000005},"min":{"1min":0.32900000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78800000000000003,"5min":25.550999999999998,"15min":25.550999999999998},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":1.5129999999999999,"15min":0.871},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.60799999999999998,"5min":40.914000000000001,"15min":40.914000000000001},"last":0.34200000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":1.387,"15min":0.78800000000000003},"min":{"1min":0.30199999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76000000000000001,"5min":31.309999999999999,"15min":31.309999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.495,"5min":2.9129999999999998,"15min":1.3200000000000001},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72199999999999998,"5min":86.311999999999998,"15min":86.311999999999998},"last":0.35799999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.44800000000000001,"15min":0.42399999999999999},"min":{"1min":0.28599999999999998,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":0.68500000000000005,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.65900000000000003,"15min":0.48899999999999999},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59499999999999997,"5min":17.718,"15min":17.718},"last":0.39300000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.56399999999999995,"15min":0.47299999999999998},"min":{"1min":0.33300000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.73799999999999999,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.54700000000000004,"15min":0.46100000000000002},"min":{"1min":0.316,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66500000000000004,"5min":8.1370000000000005,"15min":8.1370000000000005},"last":0.31900000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:32:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.82899999999999996,"15min":0.58799999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.52100000000000002,"5min":17.881,"15min":17.881},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.79700000000000004,"15min":0.58099999999999996},"min":{"1min":0.316,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.54400000000000004,"5min":17.396999999999998,"15min":17.396999999999998},"last":0.45400000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313656,"kb_used_data":4132,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431496,"statfs":{"total":95995035648,"available":95673851904,"internally_reserved":0,"allocated":4231168,"data_stored":2250507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.3169999999999999,"15min":0.65700000000000003},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":25.129000000000001,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.248},{"interface":"front","average":{"1min":1.7,"5min":0.93999999999999995,"15min":0.54900000000000004},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":26.690000000000001,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.89500000000000002,"15min":0.54700000000000004},"min":{"1min":0.249,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.7570000000000001,"5min":29.140999999999998,"15min":29.140999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.875,"5min":0.65600000000000003,"15min":0.46999999999999997},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":8.8300000000000001,"5min":15.273,"15min":15.273},"last":0.318}]},{"osd":3,"last update":"Fri Feb 23 00:32:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.1319999999999999,"15min":0.66400000000000003},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.294,"5min":21.013999999999999,"15min":21.013999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.496,"5min":1.0289999999999999,"15min":0.63300000000000001},"min":{"1min":0.253,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.79,"5min":29.234999999999999,"15min":29.234999999999999},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":0.92100000000000004,"15min":0.67700000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":16.231000000000002,"5min":16.231000000000002,"15min":16.231000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":1.101,"15min":0.73199999999999998},"min":{"1min":0.34599999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72299999999999998,"5min":29.212,"15min":29.212},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.79100000000000004,"15min":0.65100000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71799999999999997,"5min":11.904,"15min":11.904},"last":0.627},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.83099999999999996,"15min":0.67000000000000004},"min":{"1min":0.378,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":3.8889999999999998,"5min":11.105,"15min":11.105},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":1.0860000000000001,"15min":0.73999999999999999},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68100000000000005,"5min":29.187000000000001,"15min":29.187000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.367,"5min":0.98299999999999998,"15min":0.74099999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":16.369,"5min":16.369,"15min":16.369},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:32:14 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.80600000000000005,"15min":0.65000000000000002},"min":{"1min":0.41699999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":8.8040000000000003,"5min":11.971,"15min":11.971},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.115,"5min":0.82699999999999996,"15min":0.68300000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":4.4379999999999997,"5min":11.135999999999999,"15min":11.135999999999999},"last":0.54400000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309392,"kb_used_data":3968,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435760,"statfs":{"total":95995035648,"available":95678218240,"internally_reserved":0,"allocated":4063232,"data_stored":2238323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.82499999999999996,"15min":0.49399999999999999},"min":{"1min":0.215,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.7770000000000001,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":1.1779999999999999,"15min":0.61599999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.5469999999999997,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.35399999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.95499999999999996,"15min":0.58299999999999996},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.0620000000000001,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.316},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66400000000000003,"15min":0.46400000000000002},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.9379999999999997,"5min":15.965,"15min":15.965},"last":0.33500000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":1.0189999999999999,"15min":0.64000000000000001},"min":{"1min":0.28199999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.095,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.95999999999999996,"15min":0.62},"min":{"1min":0.28100000000000003,"5min":0.245,"15min":0.245},"max":{"1min":1.046,"5min":16.006,"15min":16.006},"last":0.376}]},{"osd":4,"last update":"Fri Feb 23 00:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.82999999999999996,"15min":0.64600000000000002},"min":{"1min":0.36899999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.4780000000000002,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.175,"15min":0.76500000000000001},"min":{"1min":0.41499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.399,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.54200000000000004}]},{"osd":5,"last update":"Fri Feb 23 00:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.0680000000000001,"15min":0.72399999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.4660000000000002,"5min":13.311,"15min":13.311},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.0780000000000001,"15min":0.73099999999999998},"min":{"1min":0.38600000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.5139999999999998,"5min":23.744,"15min":23.744},"last":0.67800000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.3740000000000001,"15min":0.80300000000000005},"min":{"1min":0.40100000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.5510000000000002,"5min":33.082999999999998,"15min":33.082999999999998},"last":0.626},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0069999999999999,"15min":0.71199999999999997},"min":{"1min":0.38800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":9.8040000000000003,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:32:18 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.175,"15min":0.76400000000000001},"min":{"1min":0.434,"5min":0.373,"15min":0.373},"max":{"1min":15.943,"5min":31.251999999999999,"15min":31.251999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":3.1059999999999999,"5min":1.141,"15min":0.753},"min":{"1min":0.38400000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":23.785,"5min":23.785,"15min":23.785},"last":0.64300000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411427,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309412,"kb_used_data":3992,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435740,"statfs":{"total":95995035648,"available":95678197760,"internally_reserved":0,"allocated":4087808,"data_stored":2239211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.4430000000000001,"5min":1.4850000000000001,"15min":0.71599999999999997},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":56.509,"5min":56.509,"15min":56.509},"last":0.314},{"interface":"front","average":{"1min":1.427,"5min":1.4179999999999999,"15min":0.69399999999999995},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":19.571999999999999,"5min":31.498999999999999,"15min":31.498999999999999},"last":0.245}]},{"osd":1,"last update":"Fri Feb 23 00:32:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":0.96899999999999997,"15min":0.54900000000000004},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":19.324999999999999,"5min":19.324999999999999,"15min":19.324999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.288,"15min":0.65600000000000003},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.216},"max":{"1min":20.655000000000001,"5min":35.052,"15min":35.052},"last":0.29599999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:32:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":1.25,"15min":0.69299999999999995},"min":{"1min":0.20999999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54600000000000004,"5min":43.5,"15min":43.5},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.96499999999999997,"15min":0.61599999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57599999999999996,"5min":27.414000000000001,"15min":27.414000000000001},"last":0.34599999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.76700000000000002,"15min":0.61499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71199999999999997,"5min":14.614000000000001,"15min":14.614000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.65200000000000002,"15min":0.58599999999999997},"min":{"1min":0.316,"5min":0.307,"15min":0.307},"max":{"1min":0.73599999999999999,"5min":4.6550000000000002,"15min":4.6550000000000002},"last":0.68400000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.88400000000000001,"15min":0.65600000000000003},"min":{"1min":0.32700000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.66100000000000003,"5min":17.765000000000001,"15min":17.765000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.78500000000000003,"15min":0.629},"min":{"1min":0.34899999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.81799999999999995,"5min":14.589,"15min":14.589},"last":0.59099999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.66700000000000004,"15min":0.58799999999999997},"min":{"1min":0.34399999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":4.6120000000000001,"15min":4.6120000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.038,"15min":0.74299999999999999},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69399999999999995,"5min":20.574999999999999,"15min":20.574999999999999},"last":0.78400000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:32:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.8839999999999999,"15min":1.024},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":11.811999999999999,"5min":66.632000000000005,"15min":66.632000000000005},"last":0.628},{"interface":"front","average":{"1min":1.222,"5min":1.212,"15min":0.80100000000000005},"min":{"1min":0.39000000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":11.781000000000001,"5min":19.603999999999999,"15min":19.603999999999999},"last":0.70299999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":113,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309340,"kb_used_data":3920,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435812,"statfs":{"total":95995035648,"available":95678271488,"internally_reserved":0,"allocated":4014080,"data_stored":2238831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":1.032,"15min":0.59499999999999997},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.503,"5min":45.390999999999998,"15min":45.390999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":1.004,"15min":0.60399999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.46600000000000003,"5min":30.186,"15min":30.186},"last":0.313}]},{"osd":1,"last update":"Fri Feb 23 00:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.61899999999999999,"15min":0.497},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":1.018,"5min":18.039999999999999,"15min":18.039999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":1.014,"15min":0.61799999999999999},"min":{"1min":0.26000000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63100000000000001,"5min":45.441000000000003,"15min":45.441000000000003},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.96399999999999997,"15min":0.61899999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.49299999999999999,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.90200000000000002,"15min":0.59099999999999997},"min":{"1min":0.217,"5min":0.185,"15min":0.185},"max":{"1min":0.48099999999999998,"5min":24.183,"15min":24.183},"last":0.36899999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":1.071,"15min":0.74199999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":24.058,"15min":24.058},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.72899999999999998,"15min":0.63400000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66900000000000004,"5min":8.0359999999999996,"15min":8.0359999999999996},"last":0.57899999999999996}]},{"osd":5,"last update":"Fri Feb 23 00:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":1.056,"15min":0.73299999999999998},"min":{"1min":0.33900000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.77200000000000002,"5min":45.265000000000001,"15min":45.265000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.91400000000000003,"15min":0.69399999999999995},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.113,"5min":24.102,"15min":24.102},"last":0.55500000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.745,"15min":0.63400000000000001},"min":{"1min":0.35199999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.093,"5min":8.093,"15min":8.093},"last":0.625},{"interface":"front","average":{"1min":0.52700000000000002,"5min":1.179,"15min":0.79900000000000004},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75,"5min":45.232999999999997,"15min":45.232999999999997},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:32:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.71199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38400000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.1240000000000001,"5min":8.1280000000000001,"15min":8.1280000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.089,"15min":0.77000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65100000000000002,"5min":45.287999999999997,"15min":45.287999999999997},"last":0.59599999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986266,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317636,"kb_used_data":4020,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93427516,"statfs":{"total":95995035648,"available":95669776384,"internally_reserved":0,"allocated":4116480,"data_stored":2239183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.4159999999999999,"15min":0.81599999999999995},"min":{"1min":0.27700000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.0009999999999994,"5min":21.419,"15min":21.419},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":2.6200000000000001,"15min":1.226},"min":{"1min":0.33900000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.1710000000000003,"5min":63.462000000000003,"15min":63.462000000000003},"last":1.113}]},{"osd":1,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.9359999999999999,"15min":0.99299999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":13.044,"5min":41.137999999999998,"15min":41.137999999999998},"last":1.22},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.444,"15min":0.82699999999999996},"min":{"1min":0.312,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":8.0310000000000006,"5min":26.210000000000001,"15min":26.210000000000001},"last":2.5600000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":2.1120000000000001,"15min":1.0609999999999999},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.5249999999999999,"5min":67.253,"15min":67.253},"last":2.9009999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":1.8540000000000001,"15min":0.96499999999999997},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.7280000000000002,"5min":41.927999999999997,"15min":41.927999999999997},"last":1.266}]},{"osd":3,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":2.0219999999999998,"15min":1.0229999999999999},"min":{"1min":0.35699999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.6949999999999998,"5min":36.301000000000002,"15min":36.301000000000002},"last":0.626},{"interface":"front","average":{"1min":0.626,"5min":2.2389999999999999,"15min":1.101},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.899,"5min":42.064,"15min":42.064},"last":2.8580000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.754,"15min":0.49199999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":6.1360000000000001,"5min":14.242000000000001,"15min":14.242000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.69999999999999996,"15min":0.495},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6299999999999999,"5min":14.124000000000001,"15min":14.124000000000001},"last":2.2160000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60899999999999999,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":3.6669999999999998,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.95199999999999996,"15min":0.60299999999999998},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.5640000000000001,"5min":45.814999999999998,"15min":45.814999999999998},"last":0.77300000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:32:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.753,"15min":0.55900000000000005},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.6630000000000003,"5min":17.178999999999998,"15min":17.178999999999998},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.028,"5min":0.77800000000000002,"15min":0.59499999999999997},"min":{"1min":0.35599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.1959999999999997,"5min":14.212,"15min":14.212},"last":0.69899999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757333,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309476,"kb_used_data":4056,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435676,"statfs":{"total":95995035648,"available":95678132224,"internally_reserved":0,"allocated":4153344,"data_stored":2249172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":1.115,"15min":0.70999999999999996},"min":{"1min":0.23400000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.67600000000000005,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.435,"5min":2.1070000000000002,"15min":1.0269999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.66000000000000003,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.56799999999999995}]},{"osd":1,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.29,"15min":0.748},"min":{"1min":0.27100000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72299999999999998,"5min":38.838000000000001,"15min":38.838000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.439,"5min":2.5179999999999998,"15min":1.1699999999999999},"min":{"1min":0.28399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.69599999999999995,"5min":127.794,"15min":127.794},"last":0.47299999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.651,"15min":0.88400000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.63,"5min":37.442999999999998,"15min":37.442999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":1.0620000000000001,"15min":0.69099999999999995},"min":{"1min":0.30499999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67200000000000004,"5min":23.297999999999998,"15min":23.297999999999998},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":1.226,"15min":0.72899999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71799999999999997,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":1.6160000000000001,"15min":0.88800000000000001},"min":{"1min":0.33300000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.72199999999999998,"5min":40.865000000000002,"15min":40.865000000000002},"last":0.621}]},{"osd":4,"last update":"Fri Feb 23 00:32:32 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":1.5489999999999999,"15min":0.75900000000000001},"min":{"1min":0.27400000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.63700000000000001,"5min":103.13,"15min":103.13},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.57399999999999995,"15min":0.432},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.628,"5min":12.795999999999999,"15min":12.795999999999999},"last":0.55500000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.57799999999999996,"15min":0.47199999999999998},"min":{"1min":0.35599999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69099999999999995,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":1.5640000000000001,"15min":0.80800000000000005},"min":{"1min":0.28399999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73299999999999998,"5min":103.61499999999999,"15min":103.61499999999999},"last":0.65100000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.56499999999999995,"15min":0.51100000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.66300000000000003,"5min":7.1820000000000004,"15min":7.1820000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.32600000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.69699999999999995,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.68000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:33:23.423 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.log pgp_num 32 2024-02-23T00:33:23.452 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:23.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:23.556 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:21.600506+0000 mgr.y (mgr.24437) 274 : cluster [DBG] pgmap v358: 257 pgs: 8 unknown, 4 active+clean+snaptrim, 245 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:22.044891+0000 mon.c (mon.2) 304 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:22.280430+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:22.280929+0000 mon.c (mon.2) 305 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:22.281115+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:22.281481+0000 mon.c (mon.2) 306 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi087-82011-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:22.281683+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi087-82011-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:22.445053+0000 mon.c (mon.2) 307 : audit [DBG] from='client.? 172.21.15.87:0/1065919388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:23.038190+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi087-82011-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:23.038277+0000 mon.a (mon.0) 1794 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-02-23T00:33:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:23.044423+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi087-82011-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:23.044633+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:21.600506+0000 mgr.y (mgr.24437) 274 : cluster [DBG] pgmap v358: 257 pgs: 8 unknown, 4 active+clean+snaptrim, 245 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:22.044891+0000 mon.c (mon.2) 304 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:22.280430+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:22.280929+0000 mon.c (mon.2) 305 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:22.281115+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:22.281481+0000 mon.c (mon.2) 306 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi087-82011-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:22.281683+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi087-82011-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:22.445053+0000 mon.c (mon.2) 307 : audit [DBG] from='client.? 172.21.15.87:0/1065919388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:23.038190+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi087-82011-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:23.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:23.038277+0000 mon.a (mon.0) 1794 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-02-23T00:33:23.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:23.044423+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi087-82011-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:23.044633+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:23.044856+0000 mon.b (mon.1) 223 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:23.045665+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:23.045924+0000 mon.a (mon.0) 1796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi087-82011-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:23.046500+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:23.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:23.044856+0000 mon.b (mon.1) 223 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:23.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:23.045665+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:23.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:23.045924+0000 mon.a (mon.0) 1796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi087-82011-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:23.046500+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:23.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:21.600506+0000 mgr.y (mgr.24437) 274 : cluster [DBG] pgmap v358: 257 pgs: 8 unknown, 4 active+clean+snaptrim, 245 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-23T00:33:23.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:22.044891+0000 mon.c (mon.2) 304 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:22.280430+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:22.280929+0000 mon.c (mon.2) 305 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:22.281115+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:22.281481+0000 mon.c (mon.2) 306 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi087-82011-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:22.281683+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi087-82011-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:22.445053+0000 mon.c (mon.2) 307 : audit [DBG] from='client.? 172.21.15.87:0/1065919388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:23.038190+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi087-82011-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:23.038277+0000 mon.a (mon.0) 1794 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:23.044423+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi087-82011-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:23.044633+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:23.044856+0000 mon.b (mon.1) 223 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:23.045665+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:23.045924+0000 mon.a (mon.0) 1796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi087-82011-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:23.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:23.046500+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:23.590 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:23.611 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:23.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:23.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:23.715 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:23.742 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:23.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:33:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:33:24.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 -- 172.21.15.87:0/621406616 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0fcbb0 msgr2=0x7f551c0fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 --2- 172.21.15.87:0/621406616 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0fcbb0 0x7f551c0fcf90 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 rx=0x7f550c004870 tx=0x7f550c009cb0).stop 2024-02-23T00:33:24.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 -- 172.21.15.87:0/621406616 shutdown_connections 2024-02-23T00:33:24.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 --2- 172.21.15.87:0/621406616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f551c0f7400 0x7f551c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 --2- 172.21.15.87:0/621406616 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0fcbb0 0x7f551c0fcf90 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 --2- 172.21.15.87:0/621406616 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f551c0fd510 0x7f551c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 -- 172.21.15.87:0/621406616 >> 172.21.15.87:0/621406616 conn(0x7f551c0f2820 msgr2=0x7f551c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 -- 172.21.15.87:0/621406616 shutdown_connections 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 -- 172.21.15.87:0/621406616 wait complete. 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 librados: starting msgr at 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.440+0000 7f55218ea700 1 librados: starting objecter 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 Processor -- start 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 -- start start 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 librados: setting wanted keys 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 librados: calling monclient init 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0f7400 0x7f551c109310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f551c0fcbb0 0x7f551c109800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f551c0fd510 0x7f551c109dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c114500 con 0x7f551c0fcbb0 2024-02-23T00:33:24.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c114670 con 0x7f551c0fd510 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f55218ea700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551c114830 con 0x7f551c0f7400 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551a7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f551c0fcbb0 0x7f551c109800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0f7400 0x7f551c109310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0f7400 0x7f551c109310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44616/0 (socket says 172.21.15.87:44616) 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551affd700 1 -- 172.21.15.87:0/1065919388 learned_addr learned my addr 172.21.15.87:0/1065919388 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551a7fc700 1 --2- 172.21.15.87:0/1065919388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f551c0fcbb0 0x7f551c109800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41788/0 (socket says 172.21.15.87:41788) 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551b7fe700 1 --2- 172.21.15.87:0/1065919388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f551c0fd510 0x7f551c109dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551affd700 1 -- 172.21.15.87:0/1065919388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f551c0fd510 msgr2=0x7f551c109dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551affd700 1 --2- 172.21.15.87:0/1065919388 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f551c0fd510 0x7f551c109dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551affd700 1 -- 172.21.15.87:0/1065919388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f551c0fcbb0 msgr2=0x7f551c109800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551affd700 1 --2- 172.21.15.87:0/1065919388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f551c0fcbb0 0x7f551c109800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551affd700 1 -- 172.21.15.87:0/1065919388 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c005040 con 0x7f551c0f7400 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.441+0000 7f551a7fc700 1 --2- 172.21.15.87:0/1065919388 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f551c0fcbb0 0x7f551c109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:24.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f551affd700 1 --2- 172.21.15.87:0/1065919388 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0f7400 0x7f551c109310 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 rx=0x7f550c009eb0 tx=0x7f550c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f5503fff700 10 client.?.objecter ms_handle_connect 0x7f551c0f7400 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f5503fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c00e070 con 0x7f551c0f7400 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f550c013da0 con 0x7f551c0f7400 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551c114ab0 con 0x7f551c0f7400 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c019070 con 0x7f551c0f7400 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f55218ea700 10 client.37259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551c114eb0 con 0x7f551c0f7400 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.442+0000 7f55218ea700 1 librados: init done 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.443+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f551c05fa60 con 0x7f551c0f7400 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.443+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f550c01e420 con 0x7f551c0f7400 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.443+0000 7f5503fff700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5504062c90 0x7f5504065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(251..251 src has 38..251) v4 ==== 8680+0+0 (secure 0 0 0) 0x7f550c07df20 con 0x7f551c0f7400 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 osd_map(251..251 src has 38..251) v4 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f5503fff700 3 client.37259.objecter handle_osd_map got epochs [251,251] > 0 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f5503fff700 3 client.37259.objecter handle_osd_map decoding full epoch 251 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f551a7fc700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5504062c90 0x7f5504065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f5503fff700 20 client.37259.objecter dump_active .. 0 homeless 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v251) v1 ==== 54+0+16 (secure 0 0 0) 0x7f550c0805b0 con 0x7f551c0f7400 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f551a7fc700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5504062c90 0x7f5504065150 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 rx=0x7f551c061720 tx=0x7f551000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f5503fff700 10 client.37259.objecter ms_handle_connect 0x7f5504062c90 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 10 client.37259.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 1 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6801/2153191863 conn(0x7f551c028540 0x7f551c0fd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1060e0 osd=0 3 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1060e0 osd=0 3 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1060e0 osd=0 3 2024-02-23T00:33:24.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 1 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c1060e0 osd=0 3 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 0 1 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1060e0 osd=0 4 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 10 client.37259.objecter _send_command 1 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f551c05ee80 con 0x7f551c028540 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f551b7fe700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6801/2153191863 conn(0x7f551c028540 0x7f551c0fd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f5503fff700 10 client.37259.objecter ms_handle_connect 0x7f551c028540 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.444+0000 7f551b7fe700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6801/2153191863 conn(0x7f551c028540 0x7f551c0fd990 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.445+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f551c05ee80 con 0x7f551c028540 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.445+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.445+0000 7f5503fff700 10 client.37259.objecter _finish_command 1 = system:0 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.445+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c1060e0 osd=0 4 2024-02-23T00:33:24.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.445+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 0 1 2024-02-23T00:33:24.080 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:33:24.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 10 client.37259.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:24.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 2 2024-02-23T00:33:24.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1060e0 osd=0 3 2024-02-23T00:33:24.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1060e0 osd=0 3 2024-02-23T00:33:24.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1060e0 osd=0 3 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 2 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c1060e0 osd=0 3 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 0 2 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1060e0 osd=0 4 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 10 client.37259.objecter _send_command 2 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.460+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f551c05a040 con 0x7f551c028540 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.461+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f551c05a040 con 0x7f551c028540 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.461+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.461+0000 7f5503fff700 10 client.37259.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.461+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c1060e0 osd=0 4 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.461+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 0 2 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 10 client.37259.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 3 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6805/685354171 conn(0x7f551c027b90 0x7f551c027fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c0ff000 osd=1 3 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0ff000 osd=1 3 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c0ff000 osd=1 3 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 3 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c0ff000 osd=1 3 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 1 3 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0ff000 osd=1 4 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 10 client.37259.objecter _send_command 3 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f551c0ff170 con 0x7f551c027b90 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f551affd700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6805/685354171 conn(0x7f551c027b90 0x7f551c027fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f5503fff700 10 client.37259.objecter ms_handle_connect 0x7f551c027b90 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.462+0000 7f551affd700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6805/685354171 conn(0x7f551c027b90 0x7f551c027fb0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.463+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f551c0ff170 con 0x7f551c027b90 2024-02-23T00:33:24.083 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:33:24.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.463+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:33:24.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.463+0000 7f5503fff700 10 client.37259.objecter _finish_command 3 = system:0 2024-02-23T00:33:24.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.463+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c0ff000 osd=1 4 2024-02-23T00:33:24.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.463+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 1 3 2024-02-23T00:33:24.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 10 client.37259.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:24.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 4 2024-02-23T00:33:24.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c0ff000 osd=1 3 2024-02-23T00:33:24.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0ff000 osd=1 3 2024-02-23T00:33:24.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c0ff000 osd=1 3 2024-02-23T00:33:24.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 4 2024-02-23T00:33:24.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c0ff000 osd=1 3 2024-02-23T00:33:24.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 1 4 2024-02-23T00:33:24.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0ff000 osd=1 4 2024-02-23T00:33:24.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 10 client.37259.objecter _send_command 4 2024-02-23T00:33:24.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.478+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f551c027600 con 0x7f551c027b90 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f551c027600 con 0x7f551c027b90 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f5503fff700 10 client.37259.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c0ff000 osd=1 4 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 1 4 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 10 client.37259.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 5 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6809/3605394228 conn(0x7f551c108280 0x7f551c108680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c0fe650 osd=2 3 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0fe650 osd=2 3 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c0fe650 osd=2 3 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 5 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c0fe650 osd=2 3 2024-02-23T00:33:24.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 2 5 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0fe650 osd=2 4 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 10 client.37259.objecter _send_command 5 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f551c11c650 con 0x7f551c108280 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f551a7fc700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6809/3605394228 conn(0x7f551c108280 0x7f551c108680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f5503fff700 10 client.37259.objecter ms_handle_connect 0x7f551c108280 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.480+0000 7f551a7fc700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6809/3605394228 conn(0x7f551c108280 0x7f551c108680 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.481+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f551c11c650 con 0x7f551c108280 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.481+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.481+0000 7f5503fff700 10 client.37259.objecter _finish_command 5 = system:0 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.481+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c0fe650 osd=2 4 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.481+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 2 5 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 10 client.37259.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:24.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 6 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c0fe650 osd=2 3 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0fe650 osd=2 3 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c0fe650 osd=2 3 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 6 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c0fe650 osd=2 3 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 2 6 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0fe650 osd=2 4 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 10 client.37259.objecter _send_command 6 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.496+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f551c108b70 con 0x7f551c108280 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f551c108b70 con 0x7f551c108280 2024-02-23T00:33:24.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:24.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f5503fff700 10 client.37259.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c0fe650 osd=2 4 2024-02-23T00:33:24.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 2 6 2024-02-23T00:33:24.088 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:33:24.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 10 client.37259.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:24.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 7 2024-02-23T00:33:24.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6813/128059790 conn(0x7f551c11b580 0x7f551c11b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c108fb0 osd=3 3 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c108fb0 osd=3 3 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c108fb0 osd=3 3 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 7 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c108fb0 osd=3 3 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 3 7 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c108fb0 osd=3 4 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 10 client.37259.objecter _send_command 7 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f551c109190 con 0x7f551c11b580 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f551b7fe700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6813/128059790 conn(0x7f551c11b580 0x7f551c11b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f5503fff700 10 client.37259.objecter ms_handle_connect 0x7f551c11b580 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.498+0000 7f551b7fe700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6813/128059790 conn(0x7f551c11b580 0x7f551c11b9e0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.499+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5514017490 con 0x7f551c11b580 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.499+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:33:24.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.499+0000 7f5503fff700 10 client.37259.objecter _finish_command 7 = system:0 2024-02-23T00:33:24.090 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:33:24.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.499+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c108fb0 osd=3 4 2024-02-23T00:33:24.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.499+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 3 7 2024-02-23T00:33:24.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 10 client.37259.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 8 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c108fb0 osd=3 3 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c108fb0 osd=3 3 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c108fb0 osd=3 3 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 8 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c108fb0 osd=3 3 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 3 8 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c108fb0 osd=3 4 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 10 client.37259.objecter _send_command 8 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.514+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f551c11c1c0 con 0x7f551c11b580 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.515+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f551c11c1c0 con 0x7f551c11b580 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.515+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.515+0000 7f5503fff700 10 client.37259.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c108fb0 osd=3 4 2024-02-23T00:33:24.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 3 8 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 10 client.37259.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 9 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6800/4175844757 conn(0x7f551c123270 0x7f551c125660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c11c400 osd=4 3 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c11c400 osd=4 3 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c11c400 osd=4 3 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 9 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c11c400 osd=4 3 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 4 9 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c11c400 osd=4 4 2024-02-23T00:33:24.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 10 client.37259.objecter _send_command 9 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f551c1283c0 con 0x7f551c123270 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f551affd700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6800/4175844757 conn(0x7f551c123270 0x7f551c125660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f5503fff700 10 client.37259.objecter ms_handle_connect 0x7f551c123270 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.516+0000 7f551affd700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6800/4175844757 conn(0x7f551c123270 0x7f551c125660 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.517+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f551c1283c0 con 0x7f551c123270 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.517+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.517+0000 7f5503fff700 10 client.37259.objecter _finish_command 9 = system:0 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.517+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c11c400 osd=4 4 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.517+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 4 9 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 10 client.37259.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 10 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c11c400 osd=4 3 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c11c400 osd=4 3 2024-02-23T00:33:24.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c11c400 osd=4 3 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 10 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c11c400 osd=4 3 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 4 10 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c11c400 osd=4 4 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 10 client.37259.objecter _send_command 10 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.532+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f551c1283c0 con 0x7f551c123270 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.534+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f551c1283c0 con 0x7f551c123270 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.534+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.534+0000 7f5503fff700 10 client.37259.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.534+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c11c400 osd=4 4 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.534+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 4 10 2024-02-23T00:33:24.094 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 10 client.37259.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 11 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6804/2550013795 conn(0x7f551c126160 0x7f551c1265e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1283c0 osd=5 3 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1283c0 osd=5 3 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1283c0 osd=5 3 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 11 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c1283c0 osd=5 3 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 5 11 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1283c0 osd=5 4 2024-02-23T00:33:24.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 10 client.37259.objecter _send_command 11 2024-02-23T00:33:24.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f551c1281e0 con 0x7f551c126160 2024-02-23T00:33:24.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f551a7fc700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6804/2550013795 conn(0x7f551c126160 0x7f551c1265e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f551a7fc700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6804/2550013795 conn(0x7f551c126160 0x7f551c1265e0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.535+0000 7f5503fff700 10 client.37259.objecter ms_handle_connect 0x7f551c126160 2024-02-23T00:33:24.096 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.536+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f551c1281e0 con 0x7f551c126160 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.536+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.536+0000 7f5503fff700 10 client.37259.objecter _finish_command 11 = system:0 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.536+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c1283c0 osd=5 4 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.536+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 5 11 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 10 client.37259.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 12 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1283c0 osd=5 3 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1283c0 osd=5 3 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1283c0 osd=5 3 2024-02-23T00:33:24.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 12 2024-02-23T00:33:24.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c1283c0 osd=5 3 2024-02-23T00:33:24.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 5 12 2024-02-23T00:33:24.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1283c0 osd=5 4 2024-02-23T00:33:24.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 10 client.37259.objecter _send_command 12 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.550+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f551c1281e0 con 0x7f551c126160 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f551c1281e0 con 0x7f551c126160 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f5503fff700 10 client.37259.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c1283c0 osd=5 4 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 5 12 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 10 client.37259.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 13 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6808/2509408498 conn(0x7f551c12ac20 0x7f551c12b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1281e0 osd=6 3 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1281e0 osd=6 3 2024-02-23T00:33:24.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1281e0 osd=6 3 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 13 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c1281e0 osd=6 3 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 6 13 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1281e0 osd=6 4 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.552+0000 7f55218ea700 10 client.37259.objecter _send_command 13 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.553+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f551c12e9b0 con 0x7f551c12ac20 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.553+0000 7f551b7fe700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6808/2509408498 conn(0x7f551c12ac20 0x7f551c12b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.553+0000 7f551b7fe700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6808/2509408498 conn(0x7f551c12ac20 0x7f551c12b060 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.553+0000 7f5503fff700 10 client.37259.objecter ms_handle_connect 0x7f551c12ac20 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.554+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f551c12e9b0 con 0x7f551c12ac20 2024-02-23T00:33:24.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.554+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.554+0000 7f5503fff700 10 client.37259.objecter _finish_command 13 = system:0 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.554+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c1281e0 osd=6 4 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.554+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 6 13 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 10 client.37259.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 14 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1281e0 osd=6 3 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1281e0 osd=6 3 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c1281e0 osd=6 3 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 14 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c1281e0 osd=6 3 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 6 14 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1281e0 osd=6 4 2024-02-23T00:33:24.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 10 client.37259.objecter _send_command 14 2024-02-23T00:33:24.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.568+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f551c126b90 con 0x7f551c12ac20 2024-02-23T00:33:24.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f551c126b90 con 0x7f551c12ac20 2024-02-23T00:33:24.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:24.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f5503fff700 10 client.37259.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c1281e0 osd=6 4 2024-02-23T00:33:24.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 6 14 2024-02-23T00:33:24.102 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 10 client.37259.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 15 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6812/373787000 conn(0x7f551c127240 0x7f551c1276a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c126b90 osd=7 3 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c126b90 osd=7 3 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c126b90 osd=7 3 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 15 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c126b90 osd=7 3 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 7 15 2024-02-23T00:33:24.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c126b90 osd=7 4 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 10 client.37259.objecter _send_command 15 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.570+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f551c127d10 con 0x7f551c127240 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.571+0000 7f551affd700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6812/373787000 conn(0x7f551c127240 0x7f551c1276a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.571+0000 7f5503fff700 10 client.37259.objecter ms_handle_connect 0x7f551c127240 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.571+0000 7f551affd700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6812/373787000 conn(0x7f551c127240 0x7f551c1276a0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.571+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f551c127d10 con 0x7f551c127240 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.571+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.571+0000 7f5503fff700 10 client.37259.objecter _finish_command 15 = system:0 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.571+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c126b90 osd=7 4 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.571+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 7 15 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.585+0000 7f55218ea700 10 client.37259.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:24.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.585+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign -1 16 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.585+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c126b90 osd=7 3 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.585+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c126b90 osd=7 3 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.585+0000 7f55218ea700 20 client.37259.objecter _get_session s=0x7f551c126b90 osd=7 3 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.585+0000 7f55218ea700 15 client.37259.objecter _session_command_op_remove -1 16 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.585+0000 7f55218ea700 20 client.37259.objecter get_session s=0x7f551c126b90 osd=7 3 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.586+0000 7f55218ea700 15 client.37259.objecter _session_command_op_assign 7 16 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.586+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c126b90 osd=7 4 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.586+0000 7f55218ea700 10 client.37259.objecter _send_command 16 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.586+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f551c127d10 con 0x7f551c127240 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.587+0000 7f5503fff700 1 -- 172.21.15.87:0/1065919388 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f551c127d10 con 0x7f551c127240 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.587+0000 7f5503fff700 10 client.37259.objecter ms_dispatch 0x7f551c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.587+0000 7f5503fff700 10 client.37259.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.587+0000 7f5503fff700 20 client.37259.objecter put_session s=0x7f551c126b90 osd=7 4 2024-02-23T00:33:24.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.587+0000 7f5503fff700 15 client.37259.objecter _session_command_op_remove 7 16 2024-02-23T00:33:24.106 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:33:24.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 librados: watch_flush enter 2024-02-23T00:33:24.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 librados: watch_flush exit 2024-02-23T00:33:24.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 client.37259.objecter close_session for osd.0 2024-02-23T00:33:24.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6801/2153191863 conn(0x7f551c028540 msgr2=0x7f551c0fd990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6801/2153191863 conn(0x7f551c028540 0x7f551c0fd990 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1060e0 osd=0 1 2024-02-23T00:33:24.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 client.37259.objecter close_session for osd.1 2024-02-23T00:33:24.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6805/685354171 conn(0x7f551c027b90 msgr2=0x7f551c027fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6805/685354171 conn(0x7f551c027b90 0x7f551c027fb0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0ff000 osd=1 1 2024-02-23T00:33:24.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 client.37259.objecter close_session for osd.2 2024-02-23T00:33:24.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6809/3605394228 conn(0x7f551c108280 msgr2=0x7f551c108680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6809/3605394228 conn(0x7f551c108280 0x7f551c108680 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c0fe650 osd=2 1 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 client.37259.objecter close_session for osd.3 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6813/128059790 conn(0x7f551c11b580 msgr2=0x7f551c11b9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6813/128059790 conn(0x7f551c11b580 0x7f551c11b9e0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c108fb0 osd=3 1 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 client.37259.objecter close_session for osd.4 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6800/4175844757 conn(0x7f551c123270 msgr2=0x7f551c125660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6800/4175844757 conn(0x7f551c123270 0x7f551c125660 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c11c400 osd=4 1 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f551b7fe700 1 -- 172.21.15.87:0/1065919388 reap_dead start 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 client.37259.objecter close_session for osd.5 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6804/2550013795 conn(0x7f551c126160 msgr2=0x7f551c1265e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6804/2550013795 conn(0x7f551c126160 0x7f551c1265e0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f551b7fe700 1 -- 172.21.15.87:0/1065919388 reap_dead start 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1283c0 osd=5 1 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 client.37259.objecter close_session for osd.6 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6808/2509408498 conn(0x7f551c12ac20 msgr2=0x7f551c12b060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6808/2509408498 conn(0x7f551c12ac20 0x7f551c12b060 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c1281e0 osd=6 1 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 client.37259.objecter close_session for osd.7 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6812/373787000 conn(0x7f551c127240 msgr2=0x7f551c1276a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.171:6812/373787000 conn(0x7f551c127240 0x7f551c1276a0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 20 client.37259.objecter put_session s=0x7f551c126b90 osd=7 1 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 20 client.37259.objecter shutdown clearing up homeless session... 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 10 client.37259.objecter successfully canceled tick 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f551b7fe700 1 -- 172.21.15.87:0/1065919388 reap_dead start 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5504062c90 msgr2=0x7f5504065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5504062c90 0x7f5504065150 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 rx=0x7f551c061720 tx=0x7f551000b040).stop 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0f7400 msgr2=0x7f551c109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0f7400 0x7f551c109310 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 rx=0x7f550c009eb0 tx=0x7f550c00b040).stop 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 shutdown_connections 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5504062c90 0x7f5504065150 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 --2- 172.21.15.87:0/1065919388 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f551c0f7400 0x7f551c109310 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 >> 172.21.15.87:0/1065919388 conn(0x7f551c0f2820 msgr2=0x7f551c0fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 shutdown_connections 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 -- 172.21.15.87:0/1065919388 wait complete. 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:22.588+0000 7f55218ea700 1 librados: shutdown 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.110 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:33:24.692 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:24.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:24.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:24.796 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:24.827 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:24.849 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:24.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:24.953 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:24.981 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:25.178 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:25.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:23.600899+0000 mgr.y (mgr.24437) 275 : cluster [DBG] pgmap v361: 249 pgs: 32 unknown, 4 active+clean+snaptrim, 213 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:24.036978+0000 mon.a (mon.0) 1798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:24.043122+0000 mon.b (mon.1) 224 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:24.271744+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]': finished 2024-02-23T00:33:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:24.271906+0000 mon.a (mon.0) 1800 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-02-23T00:33:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:24.272518+0000 mon.c (mon.2) 310 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:24.272788+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:24.273196+0000 mon.a (mon.0) 1802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:24.638598+0000 mon.a (mon.0) 1803 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:25.219 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:23.600899+0000 mgr.y (mgr.24437) 275 : cluster [DBG] pgmap v361: 249 pgs: 32 unknown, 4 active+clean+snaptrim, 213 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:24.036978+0000 mon.a (mon.0) 1798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:24.043122+0000 mon.b (mon.1) 224 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:24.271744+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]': finished 2024-02-23T00:33:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:24.271906+0000 mon.a (mon.0) 1800 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-02-23T00:33:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:24.272518+0000 mon.c (mon.2) 310 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:24.272788+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:24.273196+0000 mon.a (mon.0) 1802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:25.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:24.638598+0000 mon.a (mon.0) 1803 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:25.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:23.600899+0000 mgr.y (mgr.24437) 275 : cluster [DBG] pgmap v361: 249 pgs: 32 unknown, 4 active+clean+snaptrim, 213 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:25.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:24.036978+0000 mon.a (mon.0) 1798 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:25.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:24.043122+0000 mon.b (mon.1) 224 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:25.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:24.271744+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-45"}]': finished 2024-02-23T00:33:25.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:24.271906+0000 mon.a (mon.0) 1800 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-02-23T00:33:25.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:24.272518+0000 mon.c (mon.2) 310 : audit [INF] from='client.? 172.21.15.87:0/2628256862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:25.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:24.272788+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:25.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:24.273196+0000 mon.a (mon.0) 1802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]: dispatch 2024-02-23T00:33:25.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:24.638598+0000 mon.a (mon.0) 1803 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:25.240 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-23T00:33:25.283 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:25.315 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:25.336 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:25.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:25.440 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:25.468 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:25.722 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:25.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:25.827 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:25.859 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:25.880 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:25.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:25.983 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:26.012 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:26.032 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:26.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:26.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.961+0000 7f0f67425700 1 -- 172.21.15.87:0/4230925617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f600fd510 msgr2=0x7f0f600f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:26.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.961+0000 7f0f67425700 1 --2- 172.21.15.87:0/4230925617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f600fd510 0x7f0f600f6d40 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 rx=0x7f0f50004870 tx=0x7f0f50009cb0).stop 2024-02-23T00:33:26.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.961+0000 7f0f67425700 1 -- 172.21.15.87:0/4230925617 shutdown_connections 2024-02-23T00:33:26.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.961+0000 7f0f67425700 1 --2- 172.21.15.87:0/4230925617 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f600f7400 0x7f0f60103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:26.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.961+0000 7f0f67425700 1 --2- 172.21.15.87:0/4230925617 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f600fcbb0 0x7f0f600fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:26.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.961+0000 7f0f67425700 1 --2- 172.21.15.87:0/4230925617 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f600fd510 0x7f0f600f6d40 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.961+0000 7f0f67425700 1 -- 172.21.15.87:0/4230925617 >> 172.21.15.87:0/4230925617 conn(0x7f0f600f2820 msgr2=0x7f0f600f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.961+0000 7f0f67425700 1 -- 172.21.15.87:0/4230925617 shutdown_connections 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.961+0000 7f0f67425700 1 -- 172.21.15.87:0/4230925617 wait complete. 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 librados: starting msgr at 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 librados: starting objecter 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 Processor -- start 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 -- start start 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 librados: setting wanted keys 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 librados: calling monclient init 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f600f7400 0x7f0f600f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f600fcbb0 0x7f0f600f8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f600fd510 0x7f0f600f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:26.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f60116660 con 0x7f0f600fd510 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f601167a0 con 0x7f0f600f7400 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f67425700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f60116960 con 0x7f0f600fcbb0 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f649c0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f600fcbb0 0x7f0f600f8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f659c2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f600fd510 0x7f0f600f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f649c0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f600fcbb0 0x7f0f600f8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44666/0 (socket says 172.21.15.87:44666) 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f649c0700 1 -- 172.21.15.87:0/3757428055 learned_addr learned my addr 172.21.15.87:0/3757428055 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f659c2700 1 --2- 172.21.15.87:0/3757428055 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f600fd510 0x7f0f600f92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41806/0 (socket says 172.21.15.87:41806) 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f651c1700 1 --2- 172.21.15.87:0/3757428055 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f600f7400 0x7f0f600f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f649c0700 1 -- 172.21.15.87:0/3757428055 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f600f7400 msgr2=0x7f0f600f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f649c0700 1 --2- 172.21.15.87:0/3757428055 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f600f7400 0x7f0f600f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f649c0700 1 -- 172.21.15.87:0/3757428055 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f600fd510 msgr2=0x7f0f600f92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f649c0700 1 --2- 172.21.15.87:0/3757428055 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f600fd510 0x7f0f600f92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.962+0000 7f0f649c0700 1 -- 172.21.15.87:0/3757428055 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c0097c0 con 0x7f0f600fcbb0 2024-02-23T00:33:26.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f649c0700 1 --2- 172.21.15.87:0/3757428055 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f600fcbb0 0x7f0f600f8db0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7f0f50009eb0 tx=0x7f0f5000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f567fc700 10 client.?.objecter ms_handle_connect 0x7f0f600fcbb0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f567fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f567fc700 1 -- 172.21.15.87:0/3757428055 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f5000e070 con 0x7f0f600fcbb0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f50005040 con 0x7f0f600fcbb0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f567fc700 1 -- 172.21.15.87:0/3757428055 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f50013da0 con 0x7f0f600fcbb0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f567fc700 1 -- 172.21.15.87:0/3757428055 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50019070 con 0x7f0f600fcbb0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f67425700 10 client.37271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f60116e50 con 0x7f0f600fcbb0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.963+0000 7f0f67425700 1 librados: init done 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.964+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6005fa60 con 0x7f0f600fcbb0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.964+0000 7f0f567fc700 1 -- 172.21.15.87:0/3757428055 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0f5001e420 con 0x7f0f600fcbb0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.964+0000 7f0f567fc700 1 --2- 172.21.15.87:0/3757428055 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f4c062c90 0x7f0f4c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.965+0000 7f0f567fc700 1 -- 172.21.15.87:0/3757428055 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(254..254 src has 38..254) v4 ==== 9061+0+0 (secure 0 0 0) 0x7f0f5007e0e0 con 0x7f0f600fcbb0 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.965+0000 7f0f567fc700 10 client.37271.objecter ms_dispatch 0x7f0f60001550 osd_map(254..254 src has 38..254) v4 2024-02-23T00:33:26.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.965+0000 7f0f567fc700 3 client.37271.objecter handle_osd_map got epochs [254,254] > 0 2024-02-23T00:33:26.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.965+0000 7f0f567fc700 3 client.37271.objecter handle_osd_map decoding full epoch 254 2024-02-23T00:33:26.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.965+0000 7f0f651c1700 1 --2- 172.21.15.87:0/3757428055 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f4c062c90 0x7f0f4c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:26.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.965+0000 7f0f567fc700 20 client.37271.objecter dump_active .. 0 homeless 2024-02-23T00:33:26.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.965+0000 7f0f567fc700 10 client.37271.objecter ms_handle_connect 0x7f0f4c062c90 2024-02-23T00:33:26.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.965+0000 7f0f651c1700 1 --2- 172.21.15.87:0/3757428055 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f4c062c90 0x7f0f4c065150 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f0f58004dc0 tx=0x7f0f5800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:26.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:25.969+0000 7f0f567fc700 1 -- 172.21.15.87:0/3757428055 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0f500512a0 con 0x7f0f600fcbb0 2024-02-23T00:33:26.135 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:26.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:26.154+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f0f600f9df0 con 0x7f0f600fcbb0 2024-02-23T00:33:26.162 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:25.040610+0000 mon.a (mon.0) 1804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi087-82011-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi087-82011-37"}]': finished 2024-02-23T00:33:26.162 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:25.040706+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:26.162 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:25.040776+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]': finished 2024-02-23T00:33:26.162 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:25.040829+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-02-23T00:33:26.162 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:25.048275+0000 mon.b (mon.1) 225 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:26.163 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:25.048571+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:26.163 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:25.062917+0000 mon.a (mon.0) 1809 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:26.163 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:25.064397+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:26.163 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:25.064676+0000 mon.a (mon.0) 1811 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:26.164 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:26.183 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:26.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:26.286 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:26.314 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:25.040610+0000 mon.a (mon.0) 1804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi087-82011-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi087-82011-37"}]': finished 2024-02-23T00:33:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:25.040706+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:25.040776+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]': finished 2024-02-23T00:33:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:25.040829+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-02-23T00:33:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:25.048275+0000 mon.b (mon.1) 225 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:25.048571+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:25.062917+0000 mon.a (mon.0) 1809 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:25.064397+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:25.064676+0000 mon.a (mon.0) 1811 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:26.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:25.040610+0000 mon.a (mon.0) 1804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi087-82011-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi087-82011-37"}]': finished 2024-02-23T00:33:26.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:25.040706+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:26.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:25.040776+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-45"}]': finished 2024-02-23T00:33:26.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:25.040829+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-02-23T00:33:26.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:25.048275+0000 mon.b (mon.1) 225 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:26.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:25.048571+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:26.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:25.062917+0000 mon.a (mon.0) 1809 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:26.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:25.064397+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:26.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:25.064676+0000 mon.a (mon.0) 1811 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:26.335 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:26.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:26.353 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:26.436 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:26.464 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:26.662 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:26.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:26.685 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:33:26.766 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:26.797 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:26.819 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:26.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:26.922 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:26.952 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:26.972 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:26.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:27.076 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:27.102 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:27.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.258+0000 7f0f567fc700 1 -- 172.21.15.87:0/3757428055 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v256) v1 ==== 91+0+0 (secure 0 0 0) 0x7f0f50023030 con 0x7f0f600fcbb0 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.260+0000 7f0f67425700 10 librados: watch_flush enter 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 10 librados: watch_flush exit 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 20 client.37271.objecter shutdown clearing up homeless session... 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 10 client.37271.objecter successfully canceled tick 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f4c062c90 msgr2=0x7f0f4c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 --2- 172.21.15.87:0/3757428055 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f4c062c90 0x7f0f4c065150 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 rx=0x7f0f58004dc0 tx=0x7f0f5800b040).stop 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f600fcbb0 msgr2=0x7f0f600f8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 --2- 172.21.15.87:0/3757428055 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f600fcbb0 0x7f0f600f8db0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7f0f50009eb0 tx=0x7f0f5000b040).stop 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 shutdown_connections 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 --2- 172.21.15.87:0/3757428055 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f4c062c90 0x7f0f4c065150 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 --2- 172.21.15.87:0/3757428055 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f600fd510 0x7f0f600f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:27.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 --2- 172.21.15.87:0/3757428055 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f600f7400 0x7f0f600f88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 --2- 172.21.15.87:0/3757428055 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f600fcbb0 0x7f0f600f8db0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 >> 172.21.15.87:0/3757428055 conn(0x7f0f600f2820 msgr2=0x7f0f60060dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 shutdown_connections 2024-02-23T00:33:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.261+0000 7f0f67425700 1 -- 172.21.15.87:0/3757428055 wait complete. 2024-02-23T00:33:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:27.263+0000 7f0f67425700 1 librados: shutdown 2024-02-23T00:33:27.266 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:33:27.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:25.601902+0000 mgr.y (mgr.24437) 276 : cluster [DBG] pgmap v364: 257 pgs: 8 creating+peering, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-23T00:33:27.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:26.047992+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-29"}]': finished 2024-02-23T00:33:27.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:26.048106+0000 mon.a (mon.0) 1813 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:27.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:26.048161+0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-02-23T00:33:27.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:26.052340+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-29", "mode": "writeback"}]: dispatch 2024-02-23T00:33:27.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:26.054845+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-29", "mode": "writeback"}]: dispatch 2024-02-23T00:33:27.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:26.055986+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:27.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:26.156354+0000 mon.c (mon.2) 311 : audit [INF] from='client.? 172.21.15.87:0/3757428055' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:26.156698+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:26.555635+0000 osd.3 (osd.3) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:26.555746+0000 osd.3 (osd.3) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:27.013404+0000 mon.a (mon.0) 1818 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:27.013429+0000 mon.a (mon.0) 1819 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:25.601902+0000 mgr.y (mgr.24437) 276 : cluster [DBG] pgmap v364: 257 pgs: 8 creating+peering, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:26.047992+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-29"}]': finished 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:26.048106+0000 mon.a (mon.0) 1813 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:26.048161+0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-02-23T00:33:27.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:26.052340+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-29", "mode": "writeback"}]: dispatch 2024-02-23T00:33:27.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:26.054845+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-29", "mode": "writeback"}]: dispatch 2024-02-23T00:33:27.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:26.055986+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:27.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:26.156354+0000 mon.c (mon.2) 311 : audit [INF] from='client.? 172.21.15.87:0/3757428055' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:33:27.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:26.156698+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:33:27.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:26.555635+0000 osd.3 (osd.3) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:27.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:26.555746+0000 osd.3 (osd.3) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:27.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:27.013404+0000 mon.a (mon.0) 1818 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:27.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:27.013429+0000 mon.a (mon.0) 1819 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:27.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:25.601902+0000 mgr.y (mgr.24437) 276 : cluster [DBG] pgmap v364: 257 pgs: 8 creating+peering, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-23T00:33:27.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:26.047992+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-29"}]': finished 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:26.048106+0000 mon.a (mon.0) 1813 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi087-82019-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:26.048161+0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:26.052340+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-29", "mode": "writeback"}]: dispatch 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:26.054845+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-29", "mode": "writeback"}]: dispatch 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:26.055986+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:26.156354+0000 mon.c (mon.2) 311 : audit [INF] from='client.? 172.21.15.87:0/3757428055' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:26.156698+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:26.555635+0000 osd.3 (osd.3) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:26.555746+0000 osd.3 (osd.3) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:27.013404+0000 mon.a (mon.0) 1818 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:27.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:27.013429+0000 mon.a (mon.0) 1819 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:26.209776+0000 osd.1 (osd.1) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:26.210049+0000 osd.1 (osd.1) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:27.018126+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-29", "mode": "writeback"}]': finished 2024-02-23T00:33:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:27.259474+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:33:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:27.259574+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-02-23T00:33:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:27.260461+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:27.260899+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:27.321231+0000 mon.b (mon.1) 227 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:27.321517+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:27.424733+0000 osd.7 (osd.7) 125 : cluster [DBG] 22.c scrub starts 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:27.425600+0000 osd.7 (osd.7) 126 : cluster [DBG] 22.c scrub ok 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:27.492441+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.e scrub starts 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:27.495301+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.e scrub ok 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:28.019763+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-46"}]': finished 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:28.019867+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]': finished 2024-02-23T00:33:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:28.019928+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:26.209776+0000 osd.1 (osd.1) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:26.210049+0000 osd.1 (osd.1) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:27.018126+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-29", "mode": "writeback"}]': finished 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:27.259474+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:27.259574+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:27.260461+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:27.260899+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:27.321231+0000 mon.b (mon.1) 227 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:27.321517+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:27.424733+0000 osd.7 (osd.7) 125 : cluster [DBG] 22.c scrub starts 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:27.425600+0000 osd.7 (osd.7) 126 : cluster [DBG] 22.c scrub ok 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:27.492441+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.e scrub starts 2024-02-23T00:33:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:27.495301+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.e scrub ok 2024-02-23T00:33:28.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:28.019763+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-46"}]': finished 2024-02-23T00:33:28.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:28.019867+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]': finished 2024-02-23T00:33:28.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:28.019928+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:28.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:28.019978+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-02-23T00:33:28.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:28.021399+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:28.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:28.021735+0000 mon.a (mon.0) 1829 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:28.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:28.022177+0000 mon.c (mon.2) 313 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:28.022509+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:28.019978+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-02-23T00:33:28.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:28.021399+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:28.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:28.021735+0000 mon.a (mon.0) 1829 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:28.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:28.022177+0000 mon.c (mon.2) 313 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:28.022509+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:26.209776+0000 osd.1 (osd.1) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:26.210049+0000 osd.1 (osd.1) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:27.018126+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-29", "mode": "writeback"}]': finished 2024-02-23T00:33:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:27.259474+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:33:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:27.259574+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-02-23T00:33:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:27.260461+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:27.260899+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:27.321231+0000 mon.b (mon.1) 227 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:27.321517+0000 mon.a (mon.0) 1824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:27.424733+0000 osd.7 (osd.7) 125 : cluster [DBG] 22.c scrub starts 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:27.425600+0000 osd.7 (osd.7) 126 : cluster [DBG] 22.c scrub ok 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:27.492441+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.e scrub starts 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:27.495301+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.e scrub ok 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:28.019763+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi087-82019-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi087-82019-46"}]': finished 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:28.019867+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi087-82011-37"}]': finished 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:28.019928+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:28.019978+0000 mon.a (mon.0) 1828 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:28.021399+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:28.021735+0000 mon.a (mon.0) 1829 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29"}]: dispatch 2024-02-23T00:33:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:28.022177+0000 mon.c (mon.2) 313 : audit [INF] from='client.? 172.21.15.87:0/1057868100' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:28.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:28.022509+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]: dispatch 2024-02-23T00:33:29.484 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:29.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:27.602467+0000 mgr.y (mgr.24437) 277 : cluster [DBG] pgmap v367: 249 pgs: 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:29.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:28.418247+0000 osd.5 (osd.5) 113 : cluster [DBG] 22.e scrub starts 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:28.419369+0000 osd.5 (osd.5) 114 : cluster [DBG] 22.e scrub ok 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:29.017620+0000 mon.a (mon.0) 1831 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.025139+0000 mon.a (mon.0) 1832 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29"}]': finished 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.025227+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]': finished 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:29.025331+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.047049+0000 mon.c (mon.2) 314 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.047436+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.048136+0000 mon.c (mon.2) 315 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.048462+0000 mon.a (mon.0) 1836 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.048929+0000 mon.c (mon.2) 316 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi087-82011-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:29.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.049184+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi087-82011-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:27.602467+0000 mgr.y (mgr.24437) 277 : cluster [DBG] pgmap v367: 249 pgs: 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:28.418247+0000 osd.5 (osd.5) 113 : cluster [DBG] 22.e scrub starts 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:28.419369+0000 osd.5 (osd.5) 114 : cluster [DBG] 22.e scrub ok 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:29.017620+0000 mon.a (mon.0) 1831 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.025139+0000 mon.a (mon.0) 1832 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29"}]': finished 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.025227+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]': finished 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:29.025331+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.047049+0000 mon.c (mon.2) 314 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.047436+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.048136+0000 mon.c (mon.2) 315 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.048462+0000 mon.a (mon.0) 1836 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.048929+0000 mon.c (mon.2) 316 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi087-82011-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:29.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.049184+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi087-82011-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:29.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:29.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:29.591 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:29.622 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:27.602467+0000 mgr.y (mgr.24437) 277 : cluster [DBG] pgmap v367: 249 pgs: 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:28.418247+0000 osd.5 (osd.5) 113 : cluster [DBG] 22.e scrub starts 2024-02-23T00:33:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:28.419369+0000 osd.5 (osd.5) 114 : cluster [DBG] 22.e scrub ok 2024-02-23T00:33:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:29.017620+0000 mon.a (mon.0) 1831 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.025139+0000 mon.a (mon.0) 1832 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-29"}]': finished 2024-02-23T00:33:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.025227+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi087-82011-37"}]': finished 2024-02-23T00:33:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:29.025331+0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-02-23T00:33:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.047049+0000 mon.c (mon.2) 314 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.047436+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.048136+0000 mon.c (mon.2) 315 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.048462+0000 mon.a (mon.0) 1836 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.048929+0000 mon.c (mon.2) 316 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi087-82011-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.049184+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi087-82011-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:29.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 -- 172.21.15.87:0/993436922 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 msgr2=0x7f723406b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 --2- 172.21.15.87:0/993436922 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 0x7f723406b4d0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f7230004870 tx=0x7f7230009cb0).stop 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 -- 172.21.15.87:0/993436922 shutdown_connections 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 --2- 172.21.15.87:0/993436922 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f72341044c0 0x7f72341068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 --2- 172.21.15.87:0/993436922 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 0x7f723406b4d0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 --2- 172.21.15.87:0/993436922 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f723406b9c0 0x7f7234103fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 -- 172.21.15.87:0/993436922 >> 172.21.15.87:0/993436922 conn(0x7f72340f3800 msgr2=0x7f72340f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 -- 172.21.15.87:0/993436922 shutdown_connections 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 -- 172.21.15.87:0/993436922 wait complete. 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 librados: starting msgr at 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 librados: starting objecter 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 Processor -- start 2024-02-23T00:33:29.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 -- start start 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 librados: setting wanted keys 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 librados: calling monclient init 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 0x7f7234110450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f723406b9c0 0x7f7234110940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.781+0000 7f723c054700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72341044c0 0x7f723410a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f723c054700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723410aae0 con 0x7f72341044c0 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f723c054700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723410ac20 con 0x7f723406b9c0 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f723c054700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723410ade0 con 0x7f723406b0f0 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f7239df0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 0x7f7234110450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f7239df0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 0x7f7234110450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44704/0 (socket says 172.21.15.87:44704) 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f7239df0700 1 -- 172.21.15.87:0/3662281903 learned_addr learned my addr 172.21.15.87:0/3662281903 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f723a5f1700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72341044c0 0x7f723410a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:29.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f72395ef700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f723406b9c0 0x7f7234110940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f7239df0700 1 -- 172.21.15.87:0/3662281903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f723406b9c0 msgr2=0x7f7234110940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f7239df0700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f723406b9c0 0x7f7234110940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f7239df0700 1 -- 172.21.15.87:0/3662281903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72341044c0 msgr2=0x7f723410a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f7239df0700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72341044c0 0x7f723410a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f7239df0700 1 -- 172.21.15.87:0/3662281903 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7224005040 con 0x7f723406b0f0 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f723a5f1700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72341044c0 0x7f723410a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f7239df0700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 0x7f7234110450 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f7230009eb0 tx=0x7f723000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f722affd700 10 client.?.objecter ms_handle_connect 0x7f723406b0f0 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f722affd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f722affd700 1 -- 172.21.15.87:0/3662281903 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723000e070 con 0x7f723406b0f0 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f722affd700 1 -- 172.21.15.87:0/3662281903 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7230013da0 con 0x7f723406b0f0 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7230005040 con 0x7f723406b0f0 2024-02-23T00:33:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f722affd700 1 -- 172.21.15.87:0/3662281903 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7230019070 con 0x7f723406b0f0 2024-02-23T00:33:29.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f723c054700 10 client.37283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:29.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723410b2d0 con 0x7f723406b0f0 2024-02-23T00:33:29.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.782+0000 7f723c054700 1 librados: init done 2024-02-23T00:33:29.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.783+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723405ee80 con 0x7f723406b0f0 2024-02-23T00:33:29.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.783+0000 7f722affd700 1 -- 172.21.15.87:0/3662281903 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f723001e420 con 0x7f723406b0f0 2024-02-23T00:33:29.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.784+0000 7f722affd700 1 --2- 172.21.15.87:0/3662281903 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7220062c90 0x7f7220065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:29.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.784+0000 7f722affd700 1 -- 172.21.15.87:0/3662281903 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(258..258 src has 38..258) v4 ==== 9037+0+0 (secure 0 0 0) 0x7f723007e030 con 0x7f723406b0f0 2024-02-23T00:33:29.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.784+0000 7f722affd700 10 client.37283.objecter ms_dispatch 0x7f7234001550 osd_map(258..258 src has 38..258) v4 2024-02-23T00:33:29.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.784+0000 7f722affd700 3 client.37283.objecter handle_osd_map got epochs [258,258] > 0 2024-02-23T00:33:29.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.784+0000 7f722affd700 3 client.37283.objecter handle_osd_map decoding full epoch 258 2024-02-23T00:33:29.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.784+0000 7f72395ef700 1 --2- 172.21.15.87:0/3662281903 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7220062c90 0x7f7220065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:29.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.784+0000 7f722affd700 20 client.37283.objecter dump_active .. 0 homeless 2024-02-23T00:33:29.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.784+0000 7f722affd700 10 client.37283.objecter ms_handle_connect 0x7f7220062c90 2024-02-23T00:33:29.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.784+0000 7f72395ef700 1 --2- 172.21.15.87:0/3662281903 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7220062c90 0x7f7220065150 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 rx=0x7f723410bd90 tx=0x7f722400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:29.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.788+0000 7f722affd700 1 -- 172.21.15.87:0/3662281903 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7230051300 con 0x7f723406b0f0 2024-02-23T00:33:29.931 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:33:29.931 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:33:29.931 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:33:29.931 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:33:29.931 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.931 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.24712.0:21 7.2 e112/94 7:40054bab:test-rados-api-smithi087-82115-6::foo:head v 112'4, mlcod=110'1)", 2024-02-23T00:33:29.931 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:33:29.931 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 147.182137619, 2024-02-23T00:33:29.931 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050194328000000003, 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:29.932 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1158000000000001e-05 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546134+0000", 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8680000000000001e-06 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.933 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546141+0000", 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8170000000000001e-06 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546147+0000", 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0014866250000000001 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.547634+0000", 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0406000000000001e-05 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.934 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.547654+0000", 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001658442 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.549313+0000", 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.047004275999999998 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.596317+0000", 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050194328000000003 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.935 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.25157.0:2 58.15 58:ad551702:::test:head [call hello.record_hello in=17b] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.599190+0000", 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 147.129070195, 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.066902009999999998, 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.25157", 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4218332242", 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:33:29.936 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599190+0000", 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599190+0000", 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.937 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599189+0000", 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4850000000000003e-06 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599196+0000", 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6200000000000005e-06 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599205+0000", 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.934e-06 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.938 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599212+0000", 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062881679999999995 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.662093+0000", 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013823899999999999 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.662232+0000", 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00022222400000000001 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:33:29.939 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.662454+0000", 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0017213930000000001 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.664175+0000", 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001211052 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.665386+0000", 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00061359100000000003 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:29.940 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.666000+0000", 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4636999999999999e-05 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.666054+0000", 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.79e-05 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.666092+0000", 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.066902009999999998 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.941 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:424 7.2 e113/94 7:40054bab:test-rados-api-smithi087-82115-6::foo:9 v 113'6, mlcod=111'3)", 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.583169+0000", 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 146.145091694, 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020999565000000001, 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.583169+0000", 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.942 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.583169+0000", 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.583169+0000", 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3858e-05 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.583182+0000", 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0110000000000006e-06 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.583191+0000", 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9739999999999996e-06 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.583198+0000", 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017407518 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.600606+0000", 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6216000000000001e-05 2024-02-23T00:33:29.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.600632+0000", 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0035042649999999999 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.604136+0000", 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.2008999999999998e-05 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.604168+0000", 2024-02-23T00:33:29.945 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020999565000000001 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:3 63.11 e117/114 63:8d290297:test-rados-api-smithi087-82091-8::foo:head v 117'1, mlcod=0'0)", 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 141.762736244, 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074073193999999995, 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.946 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1930000000000001e-06 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.947 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965529+0000", 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5130000000000002e-06 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965536+0000", 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4860000000000002e-06 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965542+0000", 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002109362 2024-02-23T00:33:29.948 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.967652+0000", 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5451000000000002e-05 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.967667+0000", 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071902888999999998 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.039570+0000", 2024-02-23T00:33:29.949 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7617999999999999e-05 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.039597+0000", 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074073193999999995 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:10 61.1f e118/112 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head v 118'6, mlcod=115'3)", 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 140.54783983999999, 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034719099000000003, 2024-02-23T00:33:29.950 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.951 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180420+0000", 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7989999999999999e-06 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180426+0000", 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8590000000000001e-06 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180432+0000", 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8689999999999998e-06 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.952 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180439+0000", 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8420000000000001e-05 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180477+0000", 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6568000000000001e-05 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180494+0000", 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034614477999999997 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.953 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.215108+0000", 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1405999999999997e-05 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.215140+0000", 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034719099000000003 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:12 61.11 e119/112 61:8c482978:test-rados-api-smithi087-82107-8::foo:head v 119'1, mlcod=0'0)", 2024-02-23T00:33:29.954 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.169560+0000", 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 139.55870022600001, 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025841233000000002, 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169560+0000", 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169560+0000", 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:29.955 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169560+0000", 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1600000000000004e-06 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169569+0000", 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2089999999999994e-06 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169578+0000", 2024-02-23T00:33:29.956 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7379999999999997e-06 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169585+0000", 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0050274330000000004 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.174613+0000", 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9856000000000001e-05 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.174632+0000", 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020737364000000001 2024-02-23T00:33:29.957 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.195370+0000", 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1779999999999997e-05 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.195402+0000", 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025841233000000002 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.958 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.28906.0:2 69.15 69:ad551702:::test:head [create,write 0~4] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 139.54759414200001, 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071209718000000005, 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.28906", 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3024219439", 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.959 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180664+0000", 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8156e-05 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.960 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180683+0000", 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3705e-05 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180696+0000", 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0080000000000002e-06 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180704+0000", 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046747999999999999 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.961 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.181172+0000", 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7989999999999996e-06 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.181179+0000", 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.060883893000000001 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242062+0000", 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012322200000000001 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.962 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242186+0000", 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015854399999999999 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242344+0000", 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0042969569999999997 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:29.963 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.246641+0000", 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020537120000000001 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.248695+0000", 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0030777460000000001 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251773+0000", 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9827999999999997e-05 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.964 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251832+0000", 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3659000000000003e-05 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251876+0000", 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071209718000000005 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.965 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:14 61.11 e120/112 61:8c482978:test-rados-api-smithi087-82107-8::foo:head v 120'3, mlcod=0'0)", 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 138.50239095000001, 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.14463899, 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:33:29.966 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225869+0000", 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2559999999999996e-06 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225877+0000", 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2709999999999998e-06 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225883+0000", 2024-02-23T00:33:29.967 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1999999999999999e-06 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225889+0000", 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.13293687000000001 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358826+0000", 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1993000000000001e-05 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.968 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358848+0000", 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011619638 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.370468+0000", 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0979000000000002e-05 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.370509+0000", 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.14463899 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.969 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:38 66.f 66:f0000000:%01:::head [pgnls start_epoch 120] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e120)", 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.439610+0000", 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 138.28865080200001, 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018524444000000001, 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 38 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.970 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439610+0000", 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439610+0000", 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999967 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439606+0000", 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5622999999999999e-05 2024-02-23T00:33:29.971 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439622+0000", 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.4156e-05 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439646+0000", 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7350000000000008e-06 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.972 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439656+0000", 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015711053999999999 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.455367+0000", 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7255000000000001e-05 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.455414+0000", 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027200750000000002 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.973 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.458134+0000", 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018524444000000001 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.41269.0:2 89.6 89:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e137)", 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 121.705358655, 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.10320085900000001, 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:29.974 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.41269", 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/244424669", 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.975 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022900+0000", 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2230000000000001e-06 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022907+0000", 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026383299999999999 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.023171+0000", 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0040999999999999e-05 2024-02-23T00:33:29.976 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.023181+0000", 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014422166 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.037603+0000", 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5469999999999997e-06 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.037612+0000", 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.064763451999999999 2024-02-23T00:33:29.977 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102375+0000", 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011688600000000001 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102492+0000", 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013874699999999999 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102631+0000", 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00093563499999999998 2024-02-23T00:33:29.978 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.103566+0000", 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001628163 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.105195+0000", 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020814273000000001 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126009+0000", 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0680000000000003e-05 2024-02-23T00:33:29.979 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126059+0000", 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3244999999999997e-05 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126103+0000", 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.10320085900000001 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.980 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:265 86.0 86:00000000:%01:::head [pgnls start_epoch 139] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e139)", 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 119.690849586, 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.048698721, 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 265 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.981 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037409+0000", 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.456e-06 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.982 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037418+0000", 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8989999999999996e-06 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037427+0000", 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0649999999999999e-06 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037435+0000", 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017399350000000001 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.983 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.054835+0000", 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8939999999999993e-06 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.054843+0000", 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.031096629000000001 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085939+0000", 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8654e-05 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.984 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085978+0000", 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013163199999999999 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.086110+0000", 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.048698721 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.36860.0:2 114.16 114:6eb9e51e:::oid_RoundTripWriteSamePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-23T00:33:29.985 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 90.724123575999997, 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036246133, 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.36860", 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4037067096", 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.986 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004135+0000", 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6179999999999995e-06 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004143+0000", 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4696e-05 2024-02-23T00:33:29.987 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004158+0000", 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7260000000000008e-06 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004167+0000", 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5045999999999999e-05 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004242+0000", 2024-02-23T00:33:29.988 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1400000000000002e-06 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004249+0000", 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016930304 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.021179+0000", 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2629999999999997e-06 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.021187+0000", 2024-02-23T00:33:29.989 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016612273 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037799+0000", 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3777999999999997e-05 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037873+0000", 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7861000000000005e-05 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:33:29.990 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037960+0000", 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00035295600000000003 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.038313+0000", 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028716700000000001 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.038601+0000", 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016768 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:29.991 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040277+0000", 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0985000000000001e-05 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040328+0000", 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4675000000000001e-05 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040383+0000", 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036246133 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.992 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.2.0:277 63.1a e182/114 63:5d5f3b80:test-rados-api-smithi087-82091-14::foo78:7 v 182'40, mlcod=148'34)", 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.002983+0000", 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 76.725277425000002, 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016693227000000001, 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002983+0000", 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.993 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002983+0000", 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002983+0000", 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0110000000000003e-06 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.994 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002990+0000", 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.198e-06 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002995+0000", 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8589999999999998e-06 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002999+0000", 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00060515000000000005 2024-02-23T00:33:29.995 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003604+0000", 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8105e-05 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003622+0000", 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016032313999999999 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:29.996 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.019654+0000", 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1886e-05 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.019676+0000", 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016693227000000001 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:537 63.8 e182/114 63:1544ebfe:test-rados-api-smithi087-82091-14::foo17:7 v 182'38, mlcod=148'34)", 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.003082+0000", 2024-02-23T00:33:29.997 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 76.725178663999998, 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026810972999999998, 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003082+0000", 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003082+0000", 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:33:29.998 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003082+0000", 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4889999999999998e-06 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003089+0000", 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1940000000000003e-06 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:29.999 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003095+0000", 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6700000000000002e-06 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003100+0000", 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016500702999999999 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.019601+0000", 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8841000000000001e-05 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.000 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.019620+0000", 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010253505 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.029873+0000", 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9760999999999999e-05 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.029893+0000", 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026810972999999998 2024-02-23T00:33:30.001 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:538 63.8 e182/114 63:1309b075:test-rados-api-smithi087-82091-14::foo36:7 v 182'40, mlcod=148'34)", 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 76.725155712000003, 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027705371999999999, 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:30.002 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003104+0000", 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7220000000000002e-06 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:30.003 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003111+0000", 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5139999999999998e-06 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003118+0000", 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8010000000000001e-06 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003122+0000", 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024802080000000001 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.004 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027924+0000", 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9273999999999999e-05 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027943+0000", 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0028410689999999999 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030784+0000", 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6224000000000002e-05 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:30.005 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030810+0000", 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027705371999999999 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:539 63.f e182/114 63:f4a0536b:test-rados-api-smithi087-82091-14::foo30:7 v 182'38, mlcod=148'34)", 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.003125+0000", 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 76.725135164999998, 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018033995000000001, 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.006 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003125+0000", 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003125+0000", 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003125+0000", 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.517e-06 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.007 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003129+0000", 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5000000000000001e-06 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003133+0000", 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.765e-06 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003137+0000", 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00053606300000000003 2024-02-23T00:33:30.008 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003673+0000", 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9395999999999999e-05 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003692+0000", 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017445268 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021138+0000", 2024-02-23T00:33:30.009 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1705999999999999e-05 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021159+0000", 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018033995000000001 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:540 63.f e182/114 63:f2825405:test-rados-api-smithi087-82091-14::foo80:7 v 182'40, mlcod=148'34)", 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.003149+0000", 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 76.725111193999993, 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021336491999999999, 2024-02-23T00:33:30.010 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003149+0000", 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003149+0000", 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:30.011 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003149+0000", 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6730000000000003e-06 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003156+0000", 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0520000000000001e-06 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003161+0000", 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.703e-06 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.012 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003164+0000", 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0067076920000000003 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.009872+0000", 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0903999999999998e-05 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.009893+0000", 2024-02-23T00:33:30.013 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014570588000000001 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.024464+0000", 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2133e-05 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.024486+0000", 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021336491999999999 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.014 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.7.0:300 63.18 e182/114 63:1a553bd4:test-rados-api-smithi087-82091-14::foo26:7 v 182'34, mlcod=148'30)", 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.012068+0000", 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 76.716192905, 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018374398, 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012068+0000", 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:30.015 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012068+0000", 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012067+0000", 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1919999999999999e-06 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012074+0000", 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2840000000000006e-06 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.016 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012082+0000", 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9789999999999998e-06 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012088+0000", 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015391475999999999 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027480+0000", 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0126e-05 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.017 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027500+0000", 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00292333 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030423+0000", 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8626000000000001e-05 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030442+0000", 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018374398 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:30.018 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37106.0:2 149.1a 149:5ec95afd:::oid_ReadIntoBufferlist_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 42.701093069000002, 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.079034594, 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37106", 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4049231794", 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:30.019 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027166+0000", 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9900000000000002e-06 2024-02-23T00:33:30.020 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027172+0000", 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4880000000000007e-06 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027180+0000", 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4049999999999997e-06 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027187+0000", 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000237202 2024-02-23T00:33:30.021 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027424+0000", 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000115528 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027539+0000", 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00014589299999999999 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,3", 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027685+0000", 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00088195299999999999 2024-02-23T00:33:30.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.028567+0000", 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026987410999999999 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.055555+0000", 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050551275 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106106+0000", 2024-02-23T00:33:30.023 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5090999999999998e-05 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106161+0000", 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0967000000000003e-05 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106202+0000", 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.079034594 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.024 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:2298 169.17 169:ed571587:test-rados-api-smithi087-82091-30::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e256)", 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 2.7028530960000001, 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869, 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2298 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.025 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.026 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025406+0000", 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1740000000000006e-06 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025415+0000", 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1041e-05 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025426+0000", 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0450000000000003e-06 2024-02-23T00:33:30.027 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025433+0000", 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7723999999999998e-05 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025491+0000", 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3479999999999993e-06 2024-02-23T00:33:30.028 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025500+0000", 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.21538338000000001 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.240883+0000", 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021381300000000001 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.241097+0000", 2024-02-23T00:33:30.029 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00573626 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246834+0000", 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6838999999999999e-05 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246930+0000", 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013423800000000001 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,2", 2024-02-23T00:33:30.030 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247065+0000", 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037269000000000001 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247437+0000", 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026355299999999998 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247701+0000", 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011556330000000001 2024-02-23T00:33:30.031 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248856+0000", 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7636999999999998e-05 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248904+0000", 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0179000000000003e-05 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:30.032 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248944+0000", 2024-02-23T00:33:30.033 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869 2024-02-23T00:33:30.033 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.033 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:30.033 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.033 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:30.033 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:30.033 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:33:30.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:29.959+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"} v 0) v1 -- 0x7f72340008d0 con 0x7f723406b0f0 2024-02-23T00:33:30.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.025+0000 7f722affd700 1 -- 172.21.15.87:0/3662281903 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]=0 set pool 4 pgp_num to 32 v259) v1 ==== 142+0+0 (secure 0 0 0) 0x7f7230050ef0 con 0x7f723406b0f0 2024-02-23T00:33:30.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.028+0000 7f723c054700 10 librados: watch_flush enter 2024-02-23T00:33:30.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.028+0000 7f723c054700 10 librados: watch_flush exit 2024-02-23T00:33:30.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.028+0000 7f723c054700 20 client.37283.objecter shutdown clearing up homeless session... 2024-02-23T00:33:30.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.028+0000 7f723c054700 10 client.37283.objecter successfully canceled tick 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.028+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7220062c90 msgr2=0x7f7220065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.028+0000 7f723c054700 1 --2- 172.21.15.87:0/3662281903 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7220062c90 0x7f7220065150 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 rx=0x7f723410bd90 tx=0x7f722400b040).stop 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.028+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 msgr2=0x7f7234110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.028+0000 7f723c054700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 0x7f7234110450 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7f7230009eb0 tx=0x7f723000b040).stop 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.028+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 shutdown_connections 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.029+0000 7f723c054700 1 --2- 172.21.15.87:0/3662281903 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7220062c90 0x7f7220065150 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.029+0000 7f723c054700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f72341044c0 0x7f723410a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.029+0000 7f723c054700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f723406b0f0 0x7f7234110450 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.029+0000 7f723c054700 1 --2- 172.21.15.87:0/3662281903 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f723406b9c0 0x7f7234110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.029+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 >> 172.21.15.87:0/3662281903 conn(0x7f72340f3800 msgr2=0x7f72340f4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.030+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 shutdown_connections 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.030+0000 7f723c054700 1 -- 172.21.15.87:0/3662281903 wait complete. 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:30.030+0000 7f723c054700 1 librados: shutdown 2024-02-23T00:33:30.035 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 pgp_num to 32 2024-02-23T00:33:30.170 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:30.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:30.276 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:30.307 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:30.328 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:30.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:30.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:28.838747+0000 osd.2 (osd.2) 113 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:28.840909+0000 osd.2 (osd.2) 114 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:28.883846+0000 osd.3 (osd.3) 123 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:28.885875+0000 osd.3 (osd.3) 124 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.960735+0000 mon.c (mon.2) 317 : audit [INF] from='client.? 172.21.15.87:0/3662281903' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:29.961038+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:30.023327+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi087-82011-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:30.023444+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:30.023505+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:30.027631+0000 mon.a (mon.0) 1842 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:30.028950+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi087-82011-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:30.029410+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi087-82011-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:28.838747+0000 osd.2 (osd.2) 113 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:28.840909+0000 osd.2 (osd.2) 114 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:28.883846+0000 osd.3 (osd.3) 123 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:28.885875+0000 osd.3 (osd.3) 124 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.960735+0000 mon.c (mon.2) 317 : audit [INF] from='client.? 172.21.15.87:0/3662281903' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:29.961038+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:30.023327+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi087-82011-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:30.023444+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:30.023505+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:30.027631+0000 mon.a (mon.0) 1842 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:30.028950+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi087-82011-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:30.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:30.029410+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi087-82011-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:30.389 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:30.432 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:30.459 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:28.838747+0000 osd.2 (osd.2) 113 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:33:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:28.840909+0000 osd.2 (osd.2) 114 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:33:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:28.883846+0000 osd.3 (osd.3) 123 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:33:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:28.885875+0000 osd.3 (osd.3) 124 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:33:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.960735+0000 mon.c (mon.2) 317 : audit [INF] from='client.? 172.21.15.87:0/3662281903' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:33:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:29.961038+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:33:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:30.023327+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi087-82011-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:30.023444+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:33:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:30.023505+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-02-23T00:33:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:30.027631+0000 mon.a (mon.0) 1842 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:30.028950+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi087-82011-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:30.029410+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi087-82011-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:30.996 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:33:31.008 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:31.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:31.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:31.138 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:31.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:33:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:33:31.493 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:31.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:31.497 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:29.353149+0000 osd.7 (osd.7) 127 : cluster [DBG] 22.a scrub starts 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:29.354231+0000 osd.7 (osd.7) 128 : cluster [DBG] 22.a scrub ok 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:29.602849+0000 mgr.y (mgr.24437) 278 : cluster [DBG] pgmap v370: 257 pgs: 8 unknown, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:30.497058+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:30.498420+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:31.026429+0000 mon.a (mon.0) 1844 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-46"}]': finished 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:31.270364+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:31.271025+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:31.271616+0000 mon.b (mon.1) 229 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:31.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:31.273659+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:29.353149+0000 osd.7 (osd.7) 127 : cluster [DBG] 22.a scrub starts 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:29.354231+0000 osd.7 (osd.7) 128 : cluster [DBG] 22.a scrub ok 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:29.602849+0000 mgr.y (mgr.24437) 278 : cluster [DBG] pgmap v370: 257 pgs: 8 unknown, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:30.497058+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:30.498420+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:31.026429+0000 mon.a (mon.0) 1844 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-46"}]': finished 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:31.270364+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:31.271025+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:31.271616+0000 mon.b (mon.1) 229 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:31.565 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:31.273659+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:31.597 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:31.624 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:29.353149+0000 osd.7 (osd.7) 127 : cluster [DBG] 22.a scrub starts 2024-02-23T00:33:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:29.354231+0000 osd.7 (osd.7) 128 : cluster [DBG] 22.a scrub ok 2024-02-23T00:33:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:29.602849+0000 mgr.y (mgr.24437) 278 : cluster [DBG] pgmap v370: 257 pgs: 8 unknown, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:30.497058+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:33:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:30.498420+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:33:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:31.026429+0000 mon.a (mon.0) 1844 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi087-82019-46"}]': finished 2024-02-23T00:33:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:31.270364+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-02-23T00:33:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:31.271025+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-46"}]: dispatch 2024-02-23T00:33:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:31.271616+0000 mon.b (mon.1) 229 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:31.273659+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:31.772 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:31.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:31.877 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:31.909 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:31.930 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:31.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:32.034 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:32.071 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:32.091 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:32.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:32.196 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:32.226 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:32.324 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:32.326 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:30.764992+0000 mgr.y (mgr.24437) 279 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:32.326 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:30.837719+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:30.840797+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:31.590183+0000 osd.0 (osd.0) 141 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:31.593350+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.022639+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi087-82011-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi087-82011-38"}]': finished 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.022752+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-46"}]': finished 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.022928+0000 mon.a (mon.0) 1850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:32.022996+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.043977+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.045964+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.061003+0000 mon.c (mon.2) 319 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.061335+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.327 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.061976+0000 mon.c (mon.2) 320 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:30.764992+0000 mgr.y (mgr.24437) 279 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:30.837719+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:30.840797+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:31.590183+0000 osd.0 (osd.0) 141 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:31.593350+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.022639+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi087-82011-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi087-82011-38"}]': finished 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.022752+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-46"}]': finished 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.022928+0000 mon.a (mon.0) 1850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:32.022996+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.043977+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.045964+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:32.328 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.061003+0000 mon.c (mon.2) 319 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.329 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.061335+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.329 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.061976+0000 mon.c (mon.2) 320 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.062310+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.062805+0000 mon.c (mon.2) 321 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi087-82019-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:32.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:32.063066+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi087-82019-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:32.329 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.062310+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.329 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.062805+0000 mon.c (mon.2) 321 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi087-82019-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:32.329 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:32.063066+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi087-82019-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:32.330 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:33:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:32] "GET /metrics HTTP/1.1" 200 188737 "" "Prometheus/2.33.4" 2024-02-23T00:33:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:32.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:32.360 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:32.432 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:32.462 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:32.483 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:32.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:32.587 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:32.615 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:30.764992+0000 mgr.y (mgr.24437) 279 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:30.837719+0000 osd.3 (osd.3) 125 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:30.840797+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:31.590183+0000 osd.0 (osd.0) 141 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:31.593350+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.022639+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi087-82011-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi087-82011-38"}]': finished 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.022752+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? 172.21.15.87:0/1131340934' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi087-82019-46"}]': finished 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.022928+0000 mon.a (mon.0) 1850 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:32.022996+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.043977+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.045964+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.061003+0000 mon.c (mon.2) 319 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.061335+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.061976+0000 mon.c (mon.2) 320 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.062310+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.062805+0000 mon.c (mon.2) 321 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi087-82019-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:32.063066+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi087-82019-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:32.872 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:33.163 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:33.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:33.267 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:33.295 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:33.427 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:31.603235+0000 mgr.y (mgr.24437) 280 : cluster [DBG] pgmap v373: 249 pgs: 32 unknown, 217 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:33:33.427 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:32.580522+0000 osd.4 (osd.4) 117 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:32.583501+0000 osd.4 (osd.4) 118 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:33.024302+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:33.024388+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi087-82019-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.024451+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:33.026022+0000 mon.c (mon.2) 322 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi087-82019-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:33.026243+0000 mon.b (mon.1) 231 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:33.026396+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi087-82019-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:33.027071+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:31.603235+0000 mgr.y (mgr.24437) 280 : cluster [DBG] pgmap v373: 249 pgs: 32 unknown, 217 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:33:33.428 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:32.580522+0000 osd.4 (osd.4) 117 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:33:33.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:32.583501+0000 osd.4 (osd.4) 118 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:33:33.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:33.024302+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:33.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:33.024388+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi087-82019-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:33.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.024451+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-02-23T00:33:33.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:33.026022+0000 mon.c (mon.2) 322 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi087-82019-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:33.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:33.026243+0000 mon.b (mon.1) 231 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:33.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:33.026396+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi087-82019-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:33.429 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:33.027071+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:33.442 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:33.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:33.546 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:33.574 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:31.603235+0000 mgr.y (mgr.24437) 280 : cluster [DBG] pgmap v373: 249 pgs: 32 unknown, 217 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:33:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:32.580522+0000 osd.4 (osd.4) 117 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:33:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:32.583501+0000 osd.4 (osd.4) 118 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:33:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:33.024302+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:33.024388+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi087-82019-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.024451+0000 mon.a (mon.0) 1858 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-02-23T00:33:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:33.026022+0000 mon.c (mon.2) 322 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi087-82019-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:33.026243+0000 mon.b (mon.1) 231 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:33.026396+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi087-82019-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:33.027071+0000 mon.a (mon.0) 1860 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:34.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:33:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:33:34.422 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:34.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:34.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:34.529 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:34.560 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:32.790943+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.b scrub starts 2024-02-23T00:33:34.560 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:32.794265+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.b scrub ok 2024-02-23T00:33:34.560 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:32.798671+0000 osd.2 (osd.2) 115 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:33:34.560 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:32.799389+0000 osd.2 (osd.2) 116 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:33:34.560 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.191494+0000 osd.1 (osd.1) 123 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.192478+0000 osd.1 (osd.1) 124 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.381264+0000 osd.5 (osd.5) 115 : cluster [DBG] 22.f scrub starts 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.382505+0000 osd.5 (osd.5) 116 : cluster [DBG] 22.f scrub ok 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.422774+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.425723+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.570966+0000 osd.4 (osd.4) 119 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.574120+0000 osd.4 (osd.4) 120 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.261164+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-31"}]': finished 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:34.261310+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.262489+0000 mon.c (mon.2) 323 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.262760+0000 mon.b (mon.1) 232 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-31", "mode": "writeback"}]: dispatch 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.262869+0000 mon.a (mon.0) 1863 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.263189+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-31", "mode": "writeback"}]: dispatch 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.277185+0000 mon.c (mon.2) 324 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.561 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.277451+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.562 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.278121+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.562 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.278360+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.562 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.278763+0000 mon.c (mon.2) 326 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi087-82051-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:34.562 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:34.278982+0000 mon.a (mon.0) 1867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi087-82051-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:34.562 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:32.790943+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.b scrub starts 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:32.794265+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.b scrub ok 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:32.798671+0000 osd.2 (osd.2) 115 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:32.799389+0000 osd.2 (osd.2) 116 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.191494+0000 osd.1 (osd.1) 123 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.192478+0000 osd.1 (osd.1) 124 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.381264+0000 osd.5 (osd.5) 115 : cluster [DBG] 22.f scrub starts 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.382505+0000 osd.5 (osd.5) 116 : cluster [DBG] 22.f scrub ok 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.422774+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.425723+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:33:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.570966+0000 osd.4 (osd.4) 119 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:33:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.574120+0000 osd.4 (osd.4) 120 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:33:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.261164+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-31"}]': finished 2024-02-23T00:33:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:34.261310+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-02-23T00:33:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.262489+0000 mon.c (mon.2) 323 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.262760+0000 mon.b (mon.1) 232 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-31", "mode": "writeback"}]: dispatch 2024-02-23T00:33:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.262869+0000 mon.a (mon.0) 1863 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.263189+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-31", "mode": "writeback"}]: dispatch 2024-02-23T00:33:34.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:32.790943+0000 osd.3 (osd.3) 127 : cluster [DBG] 6.b scrub starts 2024-02-23T00:33:34.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:32.794265+0000 osd.3 (osd.3) 128 : cluster [DBG] 6.b scrub ok 2024-02-23T00:33:34.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:32.798671+0000 osd.2 (osd.2) 115 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:33:34.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:32.799389+0000 osd.2 (osd.2) 116 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:33:34.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.191494+0000 osd.1 (osd.1) 123 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:33:34.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.192478+0000 osd.1 (osd.1) 124 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:33:34.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.381264+0000 osd.5 (osd.5) 115 : cluster [DBG] 22.f scrub starts 2024-02-23T00:33:34.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.382505+0000 osd.5 (osd.5) 116 : cluster [DBG] 22.f scrub ok 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.422774+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.425723+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.570966+0000 osd.4 (osd.4) 119 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.574120+0000 osd.4 (osd.4) 120 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.261164+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-31"}]': finished 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:34.261310+0000 mon.a (mon.0) 1862 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.262489+0000 mon.c (mon.2) 323 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.262760+0000 mon.b (mon.1) 232 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-31", "mode": "writeback"}]: dispatch 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.262869+0000 mon.a (mon.0) 1863 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:34.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.263189+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-31", "mode": "writeback"}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.277185+0000 mon.c (mon.2) 324 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.277451+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.278121+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.278360+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.278763+0000 mon.c (mon.2) 326 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi087-82051-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:34.278982+0000 mon.a (mon.0) 1867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi087-82051-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.277185+0000 mon.c (mon.2) 324 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.277451+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.278121+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.278360+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:34.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.278763+0000 mon.c (mon.2) 326 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi087-82051-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:34.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:34.278982+0000 mon.a (mon.0) 1867 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi087-82051-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:34.589 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:34.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:34.690 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:34.717 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:34.994 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:34.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:35.097 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:35.127 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:35.147 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:35.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:35.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:35.251 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:35.280 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:35.299 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:35.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:35.403 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:35.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.603682+0000 mgr.y (mgr.24437) 281 : cluster [DBG] pgmap v376: 257 pgs: 40 unknown, 217 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:33:35.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.747769+0000 osd.3 (osd.3) 129 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:33:35.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.750801+0000 osd.3 (osd.3) 130 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.790042+0000 osd.2 (osd.2) 117 : cluster [DBG] 6.f scrub starts 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:33.791133+0000 osd.2 (osd.2) 118 : cluster [DBG] 6.f scrub ok 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:34.196974+0000 osd.1 (osd.1) 125 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:34.199865+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:34.361254+0000 osd.5 (osd.5) 117 : cluster [DBG] 63.a scrub starts 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:34.362539+0000 osd.5 (osd.5) 118 : cluster [DBG] 63.a scrub ok 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:34.544580+0000 osd.4 (osd.4) 121 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:34.545898+0000 osd.4 (osd.4) 122 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:35.023242+0000 mon.a (mon.0) 1868 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.027552+0000 mon.a (mon.0) 1869 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi087-82019-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi087-82019-47"}]': finished 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.027640+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]': finished 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.027716+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-31", "mode": "writeback"}]': finished 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.027784+0000 mon.a (mon.0) 1872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi087-82051-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:35.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:35.027857+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-02-23T00:33:35.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.030516+0000 mon.c (mon.2) 327 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:35.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.030887+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:35.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.031803+0000 mon.c (mon.2) 328 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi087-82051-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:35.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.032391+0000 mon.a (mon.0) 1875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi087-82051-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:35.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:35.065324+0000 osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:35.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:35.065447+0000 osd.7 (osd.7) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:35.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.133172+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:35.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.133384+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:35.431 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:35.450 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:35.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:35.554 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:35.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.603682+0000 mgr.y (mgr.24437) 281 : cluster [DBG] pgmap v376: 257 pgs: 40 unknown, 217 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:33:35.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.747769+0000 osd.3 (osd.3) 129 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:33:35.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.750801+0000 osd.3 (osd.3) 130 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:33:35.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.790042+0000 osd.2 (osd.2) 117 : cluster [DBG] 6.f scrub starts 2024-02-23T00:33:35.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:33.791133+0000 osd.2 (osd.2) 118 : cluster [DBG] 6.f scrub ok 2024-02-23T00:33:35.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:34.196974+0000 osd.1 (osd.1) 125 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:34.199865+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:34.361254+0000 osd.5 (osd.5) 117 : cluster [DBG] 63.a scrub starts 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:34.362539+0000 osd.5 (osd.5) 118 : cluster [DBG] 63.a scrub ok 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:34.544580+0000 osd.4 (osd.4) 121 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:34.545898+0000 osd.4 (osd.4) 122 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:35.023242+0000 mon.a (mon.0) 1868 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.027552+0000 mon.a (mon.0) 1869 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi087-82019-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi087-82019-47"}]': finished 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.027640+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]': finished 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.027716+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-31", "mode": "writeback"}]': finished 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.027784+0000 mon.a (mon.0) 1872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi087-82051-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:35.027857+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-02-23T00:33:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.603682+0000 mgr.y (mgr.24437) 281 : cluster [DBG] pgmap v376: 257 pgs: 40 unknown, 217 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:33:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.747769+0000 osd.3 (osd.3) 129 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:33:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.750801+0000 osd.3 (osd.3) 130 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:33:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.790042+0000 osd.2 (osd.2) 117 : cluster [DBG] 6.f scrub starts 2024-02-23T00:33:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:33.791133+0000 osd.2 (osd.2) 118 : cluster [DBG] 6.f scrub ok 2024-02-23T00:33:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:34.196974+0000 osd.1 (osd.1) 125 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:33:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:34.199865+0000 osd.1 (osd.1) 126 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:33:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:34.361254+0000 osd.5 (osd.5) 117 : cluster [DBG] 63.a scrub starts 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:34.362539+0000 osd.5 (osd.5) 118 : cluster [DBG] 63.a scrub ok 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:34.544580+0000 osd.4 (osd.4) 121 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:34.545898+0000 osd.4 (osd.4) 122 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:35.023242+0000 mon.a (mon.0) 1868 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.027552+0000 mon.a (mon.0) 1869 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi087-82019-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi087-82019-47"}]': finished 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.027640+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi087-82011-38"}]': finished 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.027716+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-31", "mode": "writeback"}]': finished 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.027784+0000 mon.a (mon.0) 1872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi087-82051-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:35.027857+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-02-23T00:33:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.030516+0000 mon.c (mon.2) 327 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.030887+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.031803+0000 mon.c (mon.2) 328 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi087-82051-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.032391+0000 mon.a (mon.0) 1875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi087-82051-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:35.065324+0000 osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:35.065447+0000 osd.7 (osd.7) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.133172+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.133384+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.030516+0000 mon.c (mon.2) 327 : audit [INF] from='client.? 172.21.15.87:0/2105660432' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.030887+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]: dispatch 2024-02-23T00:33:35.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.031803+0000 mon.c (mon.2) 328 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi087-82051-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:35.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.032391+0000 mon.a (mon.0) 1875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi087-82051-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:35.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:35.065324+0000 osd.7 (osd.7) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:35.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:35.065447+0000 osd.7 (osd.7) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:35.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.133172+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:35.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.133384+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.544+0000 7f6a5b967700 1 -- 172.21.15.87:0/2480111959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540fdbb0 msgr2=0x7f6a540fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.544+0000 7f6a5b967700 1 --2- 172.21.15.87:0/2480111959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540fdbb0 0x7f6a540fdf90 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7f6a44004870 tx=0x7f6a44009cb0).stop 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.544+0000 7f6a5b967700 1 -- 172.21.15.87:0/2480111959 shutdown_connections 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.544+0000 7f6a5b967700 1 --2- 172.21.15.87:0/2480111959 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a540f8420 0x7f6a54104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.544+0000 7f6a5b967700 1 --2- 172.21.15.87:0/2480111959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540fdbb0 0x7f6a540fdf90 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.544+0000 7f6a5b967700 1 --2- 172.21.15.87:0/2480111959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a540fe510 0x7f6a540f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.544+0000 7f6a5b967700 1 -- 172.21.15.87:0/2480111959 >> 172.21.15.87:0/2480111959 conn(0x7f6a540f3800 msgr2=0x7f6a540f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.544+0000 7f6a5b967700 1 -- 172.21.15.87:0/2480111959 shutdown_connections 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.544+0000 7f6a5b967700 1 -- 172.21.15.87:0/2480111959 wait complete. 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 librados: starting msgr at 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 librados: starting objecter 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 Processor -- start 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 -- start start 2024-02-23T00:33:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 librados: setting wanted keys 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 librados: calling monclient init 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a540f8420 0x7f6a5410e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540fdbb0 0x7f6a5410e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a540fe510 0x7f6a54108280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a54108770 con 0x7f6a540fdbb0 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a541088e0 con 0x7f6a540f8420 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a5b967700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a54108aa0 con 0x7f6a540fe510 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a58f02700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540fdbb0 0x7f6a5410e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a59f04700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a540fe510 0x7f6a54108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a58f02700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540fdbb0 0x7f6a5410e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60622/0 (socket says 172.21.15.87:60622) 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a58f02700 1 -- 172.21.15.87:0/604464095 learned_addr learned my addr 172.21.15.87:0/604464095 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.545+0000 7f6a59703700 1 --2- 172.21.15.87:0/604464095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a540f8420 0x7f6a5410e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a59f04700 1 -- 172.21.15.87:0/604464095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a540f8420 msgr2=0x7f6a5410e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a59f04700 1 --2- 172.21.15.87:0/604464095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6a540f8420 0x7f6a5410e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a59f04700 1 -- 172.21.15.87:0/604464095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540fdbb0 msgr2=0x7f6a5410e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a59f04700 1 --2- 172.21.15.87:0/604464095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540fdbb0 0x7f6a5410e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a59f04700 1 -- 172.21.15.87:0/604464095 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a50005040 con 0x7f6a540fe510 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a58f02700 1 --2- 172.21.15.87:0/604464095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6a540fdbb0 0x7f6a5410e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a59f04700 1 --2- 172.21.15.87:0/604464095 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a540fe510 0x7f6a54108280 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f6a4c00d940 tx=0x7f6a4c004e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a4a7fc700 10 client.?.objecter ms_handle_connect 0x7f6a540fe510 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a4a7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c00b4f0 con 0x7f6a540fe510 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a44005040 con 0x7f6a540fe510 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a4c00f040 con 0x7f6a540fe510 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c013970 con 0x7f6a540fe510 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a5b967700 10 client.37301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a541090e0 con 0x7f6a540fe510 2024-02-23T00:33:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.546+0000 7f6a5b967700 1 librados: init done 2024-02-23T00:33:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.547+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6a54107060 con 0x7f6a540fe510 2024-02-23T00:33:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.547+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f6a4c00b650 con 0x7f6a540fe510 2024-02-23T00:33:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.547+0000 7f6a4a7fc700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a40062c90 0x7f6a40065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(264..264 src has 38..264) v4 ==== 9059+0+0 (secure 0 0 0) 0x7f6a4c07d0e0 con 0x7f6a540fe510 2024-02-23T00:33:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 osd_map(264..264 src has 38..264) v4 2024-02-23T00:33:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a4a7fc700 3 client.37301.objecter handle_osd_map got epochs [264,264] > 0 2024-02-23T00:33:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a4a7fc700 3 client.37301.objecter handle_osd_map decoding full epoch 264 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a59703700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a40062c90 0x7f6a40065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a4a7fc700 20 client.37301.objecter dump_active .. 0 homeless 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v264) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6a4c084070 con 0x7f6a540fe510 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a4a7fc700 10 client.37301.objecter ms_handle_connect 0x7f6a40062c90 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a59703700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a40062c90 0x7f6a40065150 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7f6a440043d0 tx=0x7f6a4400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 10 client.37301.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 1 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6801/2153191863 conn(0x7f6a540fee60 0x7f6a540ff240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54109710 osd=0 3 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54109710 osd=0 3 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54109710 osd=0 3 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 1 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54109710 osd=0 3 2024-02-23T00:33:35.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 0 1 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54109710 osd=0 4 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 10 client.37301.objecter _send_command 1 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a540ff730 con 0x7f6a540fee60 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a58f02700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6801/2153191863 conn(0x7f6a540fee60 0x7f6a540ff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a4a7fc700 10 client.37301.objecter ms_handle_connect 0x7f6a540fee60 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.548+0000 7f6a58f02700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6801/2153191863 conn(0x7f6a540fee60 0x7f6a540ff240 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.549+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6a540ff730 con 0x7f6a540fee60 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.549+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.549+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 1 = system:0 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.549+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54109710 osd=0 4 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.549+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 0 1 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 10 client.37301.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 2 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54109710 osd=0 3 2024-02-23T00:33:35.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54109710 osd=0 3 2024-02-23T00:33:35.594 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:33:35.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54109710 osd=0 3 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 2 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54109710 osd=0 3 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 0 2 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54109710 osd=0 4 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 10 client.37301.objecter _send_command 2 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.564+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6a540ff950 con 0x7f6a540fee60 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.565+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6a5000f040 con 0x7f6a540fee60 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.565+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.565+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.565+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54109710 osd=0 4 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.565+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 0 2 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 10 client.37301.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 3 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6805/685354171 conn(0x7f6a54028540 0x7f6a54118fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a540fd4a0 osd=1 3 2024-02-23T00:33:35.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a540fd4a0 osd=1 3 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a540fd4a0 osd=1 3 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 3 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a540fd4a0 osd=1 3 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 1 3 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a540fd4a0 osd=1 4 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 10 client.37301.objecter _send_command 3 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a540fd610 con 0x7f6a54028540 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a59703700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6805/685354171 conn(0x7f6a54028540 0x7f6a54118fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a4a7fc700 10 client.37301.objecter ms_handle_connect 0x7f6a54028540 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.566+0000 7f6a59703700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6805/685354171 conn(0x7f6a54028540 0x7f6a54118fa0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.567+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6a540fd610 con 0x7f6a54028540 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.567+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:33:35.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.567+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 3 = system:0 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.567+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a540fd4a0 osd=1 4 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.567+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 1 3 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 10 client.37301.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 4 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a540fd4a0 osd=1 3 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a540fd4a0 osd=1 3 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a540fd4a0 osd=1 3 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 4 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a540fd4a0 osd=1 3 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 1 4 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a540fd4a0 osd=1 4 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 10 client.37301.objecter _send_command 4 2024-02-23T00:33:35.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.583+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6a5405a040 con 0x7f6a54028540 2024-02-23T00:33:35.598 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:35.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.584+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6a5405a040 con 0x7f6a54028540 2024-02-23T00:33:35.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.584+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:35.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.584+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.584+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a540fd4a0 osd=1 4 2024-02-23T00:33:35.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.584+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 1 4 2024-02-23T00:33:35.599 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 10 client.37301.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 5 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6a540594d0 0x7f6a5411b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54059360 osd=2 3 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54059360 osd=2 3 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54059360 osd=2 3 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 5 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54059360 osd=2 3 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 2 5 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54059360 osd=2 4 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 10 client.37301.objecter _send_command 5 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a5411ba20 con 0x7f6a540594d0 2024-02-23T00:33:35.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a58f02700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6a540594d0 0x7f6a5411b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a4a7fc700 10 client.37301.objecter ms_handle_connect 0x7f6a540594d0 2024-02-23T00:33:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.585+0000 7f6a58f02700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6a540594d0 0x7f6a5411b4a0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.586+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6a5411ba20 con 0x7f6a540594d0 2024-02-23T00:33:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.586+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:33:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.586+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 5 = system:0 2024-02-23T00:33:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.586+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54059360 osd=2 4 2024-02-23T00:33:35.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.586+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 2 5 2024-02-23T00:33:35.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 10 client.37301.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:35.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 6 2024-02-23T00:33:35.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54059360 osd=2 3 2024-02-23T00:33:35.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54059360 osd=2 3 2024-02-23T00:33:35.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54059360 osd=2 3 2024-02-23T00:33:35.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 6 2024-02-23T00:33:35.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54059360 osd=2 3 2024-02-23T00:33:35.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 2 6 2024-02-23T00:33:35.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54059360 osd=2 4 2024-02-23T00:33:35.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 10 client.37301.objecter _send_command 6 2024-02-23T00:33:35.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.601+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6a540283a0 con 0x7f6a540594d0 2024-02-23T00:33:35.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6a540283a0 con 0x7f6a540594d0 2024-02-23T00:33:35.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:35.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54059360 osd=2 4 2024-02-23T00:33:35.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 2 6 2024-02-23T00:33:35.604 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:33:35.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 10 client.37301.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 7 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6813/128059790 conn(0x7f6a54027770 0x7f6a54122010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54027600 osd=3 3 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54027600 osd=3 3 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54027600 osd=3 3 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 7 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54027600 osd=3 3 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 3 7 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54027600 osd=3 4 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 10 client.37301.objecter _send_command 7 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a5406cf60 con 0x7f6a54027770 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a59f04700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6813/128059790 conn(0x7f6a54027770 0x7f6a54122010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a4a7fc700 10 client.37301.objecter ms_handle_connect 0x7f6a54027770 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.603+0000 7f6a59f04700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6813/128059790 conn(0x7f6a54027770 0x7f6a54122010 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.604+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6a5406cf60 con 0x7f6a54027770 2024-02-23T00:33:35.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.604+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:33:35.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.604+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 7 = system:0 2024-02-23T00:33:35.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.604+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54027600 osd=3 4 2024-02-23T00:33:35.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.604+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 3 7 2024-02-23T00:33:35.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 10 client.37301.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:35.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 8 2024-02-23T00:33:35.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54027600 osd=3 3 2024-02-23T00:33:35.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54027600 osd=3 3 2024-02-23T00:33:35.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54027600 osd=3 3 2024-02-23T00:33:35.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 8 2024-02-23T00:33:35.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54027600 osd=3 3 2024-02-23T00:33:35.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 3 8 2024-02-23T00:33:35.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54027600 osd=3 4 2024-02-23T00:33:35.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 10 client.37301.objecter _send_command 8 2024-02-23T00:33:35.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.619+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6a5406b920 con 0x7f6a54027770 2024-02-23T00:33:35.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6a5406b920 con 0x7f6a54027770 2024-02-23T00:33:35.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:35.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54027600 osd=3 4 2024-02-23T00:33:35.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 3 8 2024-02-23T00:33:35.622 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 10 client.37301.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 9 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6a5406c160 0x7f6a54123680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a5406bae0 osd=4 3 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406bae0 osd=4 3 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a5406bae0 osd=4 3 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 9 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a5406bae0 osd=4 3 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 4 9 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406bae0 osd=4 4 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 10 client.37301.objecter _send_command 9 2024-02-23T00:33:35.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.620+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a5406c610 con 0x7f6a5406c160 2024-02-23T00:33:35.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.621+0000 7f6a59703700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6a5406c160 0x7f6a54123680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.621+0000 7f6a59703700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6a5406c160 0x7f6a54123680 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.621+0000 7f6a4a7fc700 10 client.37301.objecter ms_handle_connect 0x7f6a5406c160 2024-02-23T00:33:35.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.622+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6a5406c610 con 0x7f6a5406c160 2024-02-23T00:33:35.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.622+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:33:35.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.622+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 9 = system:0 2024-02-23T00:33:35.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.622+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a5406bae0 osd=4 4 2024-02-23T00:33:35.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.622+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 4 9 2024-02-23T00:33:35.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 10 client.37301.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:35.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 10 2024-02-23T00:33:35.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a5406bae0 osd=4 3 2024-02-23T00:33:35.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406bae0 osd=4 3 2024-02-23T00:33:35.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a5406bae0 osd=4 3 2024-02-23T00:33:35.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 10 2024-02-23T00:33:35.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a5406bae0 osd=4 3 2024-02-23T00:33:35.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 4 10 2024-02-23T00:33:35.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406bae0 osd=4 4 2024-02-23T00:33:35.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 10 client.37301.objecter _send_command 10 2024-02-23T00:33:35.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.636+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6a5406c610 con 0x7f6a5406c160 2024-02-23T00:33:35.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6a5406c610 con 0x7f6a5406c160 2024-02-23T00:33:35.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:35.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a5406bae0 osd=4 4 2024-02-23T00:33:35.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 4 10 2024-02-23T00:33:35.640 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:33:35.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 10 client.37301.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 11 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6a54124080 0x7f6a541244e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a5406c610 osd=5 3 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406c610 osd=5 3 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a5406c610 osd=5 3 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 11 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a5406c610 osd=5 3 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 5 11 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406c610 osd=5 4 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 10 client.37301.objecter _send_command 11 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.638+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a54124ac0 con 0x7f6a54124080 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.639+0000 7f6a58f02700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6a54124080 0x7f6a541244e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.639+0000 7f6a4a7fc700 10 client.37301.objecter ms_handle_connect 0x7f6a54124080 2024-02-23T00:33:35.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.639+0000 7f6a58f02700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6a54124080 0x7f6a541244e0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.640+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6a54124ac0 con 0x7f6a54124080 2024-02-23T00:33:35.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.640+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:33:35.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.640+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 11 = system:0 2024-02-23T00:33:35.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.640+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a5406c610 osd=5 4 2024-02-23T00:33:35.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.640+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 5 11 2024-02-23T00:33:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 10 client.37301.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 12 2024-02-23T00:33:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a5406c610 osd=5 3 2024-02-23T00:33:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406c610 osd=5 3 2024-02-23T00:33:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a5406c610 osd=5 3 2024-02-23T00:33:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 12 2024-02-23T00:33:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a5406c610 osd=5 3 2024-02-23T00:33:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 5 12 2024-02-23T00:33:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406c610 osd=5 4 2024-02-23T00:33:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 10 client.37301.objecter _send_command 12 2024-02-23T00:33:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.654+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6a54124ac0 con 0x7f6a54124080 2024-02-23T00:33:35.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6a54124ac0 con 0x7f6a54124080 2024-02-23T00:33:35.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:35.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a5406c610 osd=5 4 2024-02-23T00:33:35.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 5 12 2024-02-23T00:33:35.658 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:33:35.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 10 client.37301.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:35.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 13 2024-02-23T00:33:35.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6a5412a880 0x7f6a5412ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54124ac0 osd=6 3 2024-02-23T00:33:35.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54124ac0 osd=6 3 2024-02-23T00:33:35.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54124ac0 osd=6 3 2024-02-23T00:33:35.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 13 2024-02-23T00:33:35.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54124ac0 osd=6 3 2024-02-23T00:33:35.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 6 13 2024-02-23T00:33:35.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54124ac0 osd=6 4 2024-02-23T00:33:35.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 10 client.37301.objecter _send_command 13 2024-02-23T00:33:35.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a5412e670 con 0x7f6a5412a880 2024-02-23T00:33:35.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.656+0000 7f6a59f04700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6a5412a880 0x7f6a5412ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.657+0000 7f6a4a7fc700 10 client.37301.objecter ms_handle_connect 0x7f6a5412a880 2024-02-23T00:33:35.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.657+0000 7f6a59f04700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6a5412a880 0x7f6a5412ace0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.657+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6a5412e670 con 0x7f6a5412a880 2024-02-23T00:33:35.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.657+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:33:35.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.657+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 13 = system:0 2024-02-23T00:33:35.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.657+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54124ac0 osd=6 4 2024-02-23T00:33:35.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.657+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 6 13 2024-02-23T00:33:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 10 client.37301.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 14 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54124ac0 osd=6 3 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54124ac0 osd=6 3 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54124ac0 osd=6 3 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 14 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54124ac0 osd=6 3 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 6 14 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54124ac0 osd=6 4 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 10 client.37301.objecter _send_command 14 2024-02-23T00:33:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.672+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6a54128d00 con 0x7f6a5412a880 2024-02-23T00:33:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6a54128d00 con 0x7f6a5412a880 2024-02-23T00:33:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54124ac0 osd=6 4 2024-02-23T00:33:35.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 6 14 2024-02-23T00:33:35.675 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 10 client.37301.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 15 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6812/373787000 conn(0x7f6a541293e0 0x7f6a54129840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54128d00 osd=7 3 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54128d00 osd=7 3 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54128d00 osd=7 3 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 15 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54128d00 osd=7 3 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 7 15 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54128d00 osd=7 4 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 10 client.37301.objecter _send_command 15 2024-02-23T00:33:35.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.673+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a54129e60 con 0x7f6a541293e0 2024-02-23T00:33:35.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.674+0000 7f6a59703700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6812/373787000 conn(0x7f6a541293e0 0x7f6a54129840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:35.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.674+0000 7f6a4a7fc700 10 client.37301.objecter ms_handle_connect 0x7f6a541293e0 2024-02-23T00:33:35.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.674+0000 7f6a59703700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6812/373787000 conn(0x7f6a541293e0 0x7f6a54129840 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:35.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.675+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6a54129e60 con 0x7f6a541293e0 2024-02-23T00:33:35.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.675+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:33:35.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.675+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 15 = system:0 2024-02-23T00:33:35.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.675+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54128d00 osd=7 4 2024-02-23T00:33:35.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.675+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 7 15 2024-02-23T00:33:35.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 10 client.37301.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:35.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign -1 16 2024-02-23T00:33:35.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54128d00 osd=7 3 2024-02-23T00:33:35.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54128d00 osd=7 3 2024-02-23T00:33:35.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 20 client.37301.objecter _get_session s=0x7f6a54128d00 osd=7 3 2024-02-23T00:33:35.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_remove -1 16 2024-02-23T00:33:35.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 20 client.37301.objecter get_session s=0x7f6a54128d00 osd=7 3 2024-02-23T00:33:35.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 15 client.37301.objecter _session_command_op_assign 7 16 2024-02-23T00:33:35.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54128d00 osd=7 4 2024-02-23T00:33:35.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 10 client.37301.objecter _send_command 16 2024-02-23T00:33:35.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.689+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6a54129e60 con 0x7f6a541293e0 2024-02-23T00:33:35.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a4a7fc700 1 -- 172.21.15.87:0/604464095 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6a54129e60 con 0x7f6a541293e0 2024-02-23T00:33:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a4a7fc700 10 client.37301.objecter ms_dispatch 0x7f6a54001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a4a7fc700 10 client.37301.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a4a7fc700 20 client.37301.objecter put_session s=0x7f6a54128d00 osd=7 4 2024-02-23T00:33:35.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a4a7fc700 15 client.37301.objecter _session_command_op_remove 7 16 2024-02-23T00:33:35.693 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:33:35.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 librados: watch_flush enter 2024-02-23T00:33:35.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 librados: watch_flush exit 2024-02-23T00:33:35.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 client.37301.objecter close_session for osd.0 2024-02-23T00:33:35.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6801/2153191863 conn(0x7f6a540fee60 msgr2=0x7f6a540ff240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6801/2153191863 conn(0x7f6a540fee60 0x7f6a540ff240 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54109710 osd=0 1 2024-02-23T00:33:35.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 client.37301.objecter close_session for osd.1 2024-02-23T00:33:35.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6805/685354171 conn(0x7f6a54028540 msgr2=0x7f6a54118fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6805/685354171 conn(0x7f6a54028540 0x7f6a54118fa0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a540fd4a0 osd=1 1 2024-02-23T00:33:35.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 client.37301.objecter close_session for osd.2 2024-02-23T00:33:35.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6a540594d0 msgr2=0x7f6a5411b4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6a540594d0 0x7f6a5411b4a0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54059360 osd=2 1 2024-02-23T00:33:35.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 client.37301.objecter close_session for osd.3 2024-02-23T00:33:35.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6813/128059790 conn(0x7f6a54027770 msgr2=0x7f6a54122010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6813/128059790 conn(0x7f6a54027770 0x7f6a54122010 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a59f04700 1 -- 172.21.15.87:0/604464095 reap_dead start 2024-02-23T00:33:35.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54027600 osd=3 1 2024-02-23T00:33:35.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 client.37301.objecter close_session for osd.4 2024-02-23T00:33:35.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6a5406c160 msgr2=0x7f6a54123680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6a5406c160 0x7f6a54123680 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406bae0 osd=4 1 2024-02-23T00:33:35.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 client.37301.objecter close_session for osd.5 2024-02-23T00:33:35.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6a54124080 msgr2=0x7f6a541244e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6a54124080 0x7f6a541244e0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a5406c610 osd=5 1 2024-02-23T00:33:35.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 client.37301.objecter close_session for osd.6 2024-02-23T00:33:35.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6a5412a880 msgr2=0x7f6a5412ace0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6a5412a880 0x7f6a5412ace0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54124ac0 osd=6 1 2024-02-23T00:33:35.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 client.37301.objecter close_session for osd.7 2024-02-23T00:33:35.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6812/373787000 conn(0x7f6a541293e0 msgr2=0x7f6a54129840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.171:6812/373787000 conn(0x7f6a541293e0 0x7f6a54129840 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 20 client.37301.objecter put_session s=0x7f6a54128d00 osd=7 1 2024-02-23T00:33:35.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 20 client.37301.objecter shutdown clearing up homeless session... 2024-02-23T00:33:35.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 10 client.37301.objecter successfully canceled tick 2024-02-23T00:33:35.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a59f04700 1 -- 172.21.15.87:0/604464095 reap_dead start 2024-02-23T00:33:35.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a40062c90 msgr2=0x7f6a40065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a40062c90 0x7f6a40065150 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 rx=0x7f6a440043d0 tx=0x7f6a4400b040).stop 2024-02-23T00:33:35.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a540fe510 msgr2=0x7f6a54108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:35.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a540fe510 0x7f6a54108280 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f6a4c00d940 tx=0x7f6a4c004e30).stop 2024-02-23T00:33:35.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 shutdown_connections 2024-02-23T00:33:35.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6a40062c90 0x7f6a40065150 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 --2- 172.21.15.87:0/604464095 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6a540fe510 0x7f6a54108280 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:35.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 >> 172.21.15.87:0/604464095 conn(0x7f6a540f3800 msgr2=0x7f6a540fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:35.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 shutdown_connections 2024-02-23T00:33:35.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.691+0000 7f6a5b967700 1 -- 172.21.15.87:0/604464095 wait complete. 2024-02-23T00:33:35.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:35.692+0000 7f6a5b967700 1 librados: shutdown 2024-02-23T00:33:35.704 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.704 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.704 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.704 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.704 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.705 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.705 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.705 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:33:35.707 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:35.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:35.819 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:35.849 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:35.872 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:35.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:35.973 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:36.001 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:36.014 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:33:36.014 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:33:36.014 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-23T00:33:36.014 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-23T00:33:36.020 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:36.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:36.124 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:36.151 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:36.170 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:36.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:36.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:36.274 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:36.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:34.776709+0000 osd.2 (osd.2) 119 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:33:36.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:34.777899+0000 osd.2 (osd.2) 120 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:33:36.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:35.381414+0000 osd.5 (osd.5) 119 : cluster [DBG] 63.17 scrub starts 2024-02-23T00:33:36.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:35.383421+0000 osd.5 (osd.5) 120 : cluster [DBG] 63.17 scrub ok 2024-02-23T00:33:36.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:35.548024+0000 mon.c (mon.2) 329 : audit [DBG] from='client.? 172.21.15.87:0/604464095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:36.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.029208+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]': finished 2024-02-23T00:33:36.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.057409+0000 mon.c (mon.2) 330 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.281182+0000 mon.a (mon.0) 1878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:36.299 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:36.281247+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-02-23T00:33:36.303 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:36.426 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:36.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:34.776709+0000 osd.2 (osd.2) 119 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:33:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:34.777899+0000 osd.2 (osd.2) 120 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:33:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:35.381414+0000 osd.5 (osd.5) 119 : cluster [DBG] 63.17 scrub starts 2024-02-23T00:33:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:35.383421+0000 osd.5 (osd.5) 120 : cluster [DBG] 63.17 scrub ok 2024-02-23T00:33:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:35.548024+0000 mon.c (mon.2) 329 : audit [DBG] from='client.? 172.21.15.87:0/604464095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.029208+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]': finished 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.057409+0000 mon.c (mon.2) 330 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.281182+0000 mon.a (mon.0) 1878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:36.281247+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.281871+0000 mon.a (mon.0) 1880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.282058+0000 mon.b (mon.1) 234 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.282478+0000 mon.c (mon.2) 331 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.282916+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.283142+0000 mon.a (mon.0) 1882 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.283406+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi087-82011-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:36.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:36.283656+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi087-82011-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:36.530 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:36.557 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.281871+0000 mon.a (mon.0) 1880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.557 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.282058+0000 mon.b (mon.1) 234 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:36.557 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.282478+0000 mon.c (mon.2) 331 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.557 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.282916+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.283142+0000 mon.a (mon.0) 1882 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.283406+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi087-82011-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:36.283656+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi087-82011-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:34.776709+0000 osd.2 (osd.2) 119 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:34.777899+0000 osd.2 (osd.2) 120 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:35.381414+0000 osd.5 (osd.5) 119 : cluster [DBG] 63.17 scrub starts 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:35.383421+0000 osd.5 (osd.5) 120 : cluster [DBG] 63.17 scrub ok 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:35.548024+0000 mon.c (mon.2) 329 : audit [DBG] from='client.? 172.21.15.87:0/604464095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.029208+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi087-82011-38"}]': finished 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.057409+0000 mon.c (mon.2) 330 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.281182+0000 mon.a (mon.0) 1878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:36.559 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:36.281247+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-02-23T00:33:36.559 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.281871+0000 mon.a (mon.0) 1880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.559 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.282058+0000 mon.b (mon.1) 234 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:36.559 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.282478+0000 mon.c (mon.2) 331 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.559 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.282916+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:36.559 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.283142+0000 mon.a (mon.0) 1882 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31"}]: dispatch 2024-02-23T00:33:36.559 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.283406+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi087-82011-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:36.559 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:36.283656+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi087-82011-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:36.559 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:36.682 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:36.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:36.786 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:36.813 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:36.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:37.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:37.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.046+0000 7f96e8b91700 1 -- 172.21.15.87:0/4033940648 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96e40fcb40 msgr2=0x7f96e40fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:37.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.046+0000 7f96e8b91700 1 --2- 172.21.15.87:0/4033940648 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96e40fcb40 0x7f96e40fcf20 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7f96d8004870 tx=0x7f96d8009cb0).stop 2024-02-23T00:33:37.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.047+0000 7f96e8b91700 1 -- 172.21.15.87:0/4033940648 shutdown_connections 2024-02-23T00:33:37.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.047+0000 7f96e8b91700 1 --2- 172.21.15.87:0/4033940648 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96e40f7ba0 0x7f96e4103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:33:37.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.047+0000 7f96e8b91700 1 --2- 172.21.15.87:0/4033940648 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96e40fcb40 0x7f96e40fcf20 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.047+0000 7f96e8b91700 1 --2- 172.21.15.87:0/4033940648 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40fd4a0 0x7f96e40f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:33:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.047+0000 7f96e8b91700 1 -- 172.21.15.87:0/4033940648 >> 172.21.15.87:0/4033940648 conn(0x7f96e406c370 msgr2=0x7f96e406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.048+0000 7f96e8b91700 1 -- 172.21.15.87:0/4033940648 shutdown_connections 2024-02-23T00:33:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.051+0000 7f96e8b91700 1 -- 172.21.15.87:0/4033940648 wait complete. 2024-02-23T00:33:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.052+0000 7f96e8b91700 1 librados: starting msgr at 2024-02-23T00:33:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.052+0000 7f96e8b91700 1 librados: starting objecter 2024-02-23T00:33:37.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.052+0000 7f96e8b91700 1 Processor -- start 2024-02-23T00:33:37.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.052+0000 7f96e8b91700 1 -- start start 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e8b91700 1 librados: setting wanted keys 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e8b91700 1 librados: calling monclient init 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e8b91700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f7ba0 0x7f96e4102980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e8b91700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96e40fcb40 0x7f96e4102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e8b91700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96e40fd4a0 0x7f96e41230c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e8b91700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4103670 con 0x7f96e40fd4a0 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e8b91700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e41235e0 con 0x7f96e40f7ba0 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e8b91700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e41237a0 con 0x7f96e40fcb40 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e259c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f7ba0 0x7f96e4102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e259c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f7ba0 0x7f96e4102980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.87:34026/0 (socket says 172.21.15.87:34026) 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e259c700 1 -- 172.21.15.87:0/54245078 learned_addr learned my addr 172.21.15.87:0/54245078 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.053+0000 7f96e1d9b700 1 --2- 172.21.15.87:0/54245078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96e40fcb40 0x7f96e4102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:37.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96e259c700 1 -- 172.21.15.87:0/54245078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96e40fcb40 msgr2=0x7f96e4102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96e259c700 1 --2- 172.21.15.87:0/54245078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96e40fcb40 0x7f96e4102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96e259c700 1 -- 172.21.15.87:0/54245078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96e40fd4a0 msgr2=0x7f96e41230c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96e259c700 1 --2- 172.21.15.87:0/54245078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96e40fd4a0 0x7f96e41230c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96e259c700 1 -- 172.21.15.87:0/54245078 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d8005040 con 0x7f96e40f7ba0 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96e259c700 1 --2- 172.21.15.87:0/54245078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f7ba0 0x7f96e4102980 secure :-1 s=READY pgs=4981 cs=0 l=1 rev1=1 rx=0x7f96d800b1a0 tx=0x7f96d8007f90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96d37fe700 10 client.?.objecter ms_handle_connect 0x7f96e40f7ba0 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96d37fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96d37fe700 1 -- 172.21.15.87:0/54245078 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d800e030 con 0x7f96e40f7ba0 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96d37fe700 1 -- 172.21.15.87:0/54245078 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96d80093b0 con 0x7f96e40f7ba0 2024-02-23T00:33:37.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.054+0000 7f96d37fe700 1 -- 172.21.15.87:0/54245078 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8018070 con 0x7f96e40f7ba0 2024-02-23T00:33:37.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.057+0000 7f96e8b91700 1 -- 172.21.15.87:0/54245078 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e4123a20 con 0x7f96e40f7ba0 2024-02-23T00:33:37.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.057+0000 7f96e8b91700 10 client.44764.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:37.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.057+0000 7f96e8b91700 1 -- 172.21.15.87:0/54245078 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e4123dd0 con 0x7f96e40f7ba0 2024-02-23T00:33:37.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.057+0000 7f96e8b91700 1 librados: init done 2024-02-23T00:33:37.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.059+0000 7f96d17fa700 1 -- 172.21.15.87:0/54245078 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c8004f20 con 0x7f96e40f7ba0 2024-02-23T00:33:37.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.061+0000 7f96d37fe700 1 -- 172.21.15.87:0/54245078 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f96d8013ce0 con 0x7f96e40f7ba0 2024-02-23T00:33:37.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.061+0000 7f96d37fe700 1 --2- 172.21.15.87:0/54245078 >> v2:172.21.15.87:6800/1231597463 conn(0x7f96cc062c90 0x7f96cc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:37.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.061+0000 7f96d37fe700 1 -- 172.21.15.87:0/54245078 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(266..266 src has 38..266) v4 ==== 9338+0+0 (secure 0 0 0) 0x7f96d807eb60 con 0x7f96e40f7ba0 2024-02-23T00:33:37.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.061+0000 7f96d37fe700 10 client.44764.objecter ms_dispatch 0x7f96e4001550 osd_map(266..266 src has 38..266) v4 2024-02-23T00:33:37.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.061+0000 7f96d37fe700 3 client.44764.objecter handle_osd_map got epochs [266,266] > 0 2024-02-23T00:33:37.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.061+0000 7f96d37fe700 3 client.44764.objecter handle_osd_map decoding full epoch 266 2024-02-23T00:33:37.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.061+0000 7f96d37fe700 20 client.44764.objecter dump_active .. 0 homeless 2024-02-23T00:33:37.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.062+0000 7f96e1d9b700 1 --2- 172.21.15.87:0/54245078 >> v2:172.21.15.87:6800/1231597463 conn(0x7f96cc062c90 0x7f96cc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:37.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.062+0000 7f96e1d9b700 1 --2- 172.21.15.87:0/54245078 >> v2:172.21.15.87:6800/1231597463 conn(0x7f96cc062c90 0x7f96cc065150 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f96e4107b50 tx=0x7f96d4009060).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:37.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.062+0000 7f96d37fe700 10 client.44764.objecter ms_handle_connect 0x7f96cc062c90 2024-02-23T00:33:37.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.064+0000 7f96d37fe700 1 -- 172.21.15.87:0/54245078 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f96d8051bb0 con 0x7f96e40f7ba0 2024-02-23T00:33:37.070 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:37.095 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:37.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:37.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:37.195 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:33:37.195 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:33:37.198 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:37.227 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:37.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:37.251+0000 7f96d17fa700 1 -- 172.21.15.87:0/54245078 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f96c80054a0 con 0x7f96e40f7ba0 2024-02-23T00:33:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:35.604757+0000 mgr.y (mgr.24437) 282 : cluster [DBG] pgmap v379: 246 pgs: 8 creating+peering, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-23T00:33:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:36.349359+0000 osd.5 (osd.5) 121 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:33:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:36.350884+0000 osd.5 (osd.5) 122 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:36.365688+0000 osd.7 (osd.7) 131 : cluster [DBG] 63.18 scrub starts 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:36.369150+0000 osd.7 (osd.7) 132 : cluster [DBG] 63.18 scrub ok 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:36.599661+0000 osd.4 (osd.4) 123 : cluster [DBG] 4.d scrub starts 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:36.603728+0000 osd.4 (osd.4) 124 : cluster [DBG] 4.d scrub ok 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:37.026541+0000 mon.a (mon.0) 1884 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:37.033071+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi087-82051-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi087-82051-2"}]': finished 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:37.033195+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31"}]': finished 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:37.033268+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi087-82011-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:37.033324+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:37.036335+0000 mon.c (mon.2) 333 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi087-82011-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:37.036632+0000 mon.c (mon.2) 334 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:37.036771+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi087-82011-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:37.037529+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:37.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:37.252591+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.87:0/54245078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:37.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:37.252822+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:35.604757+0000 mgr.y (mgr.24437) 282 : cluster [DBG] pgmap v379: 246 pgs: 8 creating+peering, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-23T00:33:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:36.349359+0000 osd.5 (osd.5) 121 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:33:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:36.350884+0000 osd.5 (osd.5) 122 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:33:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:36.365688+0000 osd.7 (osd.7) 131 : cluster [DBG] 63.18 scrub starts 2024-02-23T00:33:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:36.369150+0000 osd.7 (osd.7) 132 : cluster [DBG] 63.18 scrub ok 2024-02-23T00:33:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:36.599661+0000 osd.4 (osd.4) 123 : cluster [DBG] 4.d scrub starts 2024-02-23T00:33:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:36.603728+0000 osd.4 (osd.4) 124 : cluster [DBG] 4.d scrub ok 2024-02-23T00:33:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:37.026541+0000 mon.a (mon.0) 1884 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:37.033071+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi087-82051-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi087-82051-2"}]': finished 2024-02-23T00:33:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:37.033195+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31"}]': finished 2024-02-23T00:33:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:37.033268+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi087-82011-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:37.033324+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-02-23T00:33:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:37.036335+0000 mon.c (mon.2) 333 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi087-82011-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:37.036632+0000 mon.c (mon.2) 334 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:37.036771+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi087-82011-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:35.604757+0000 mgr.y (mgr.24437) 282 : cluster [DBG] pgmap v379: 246 pgs: 8 creating+peering, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:36.349359+0000 osd.5 (osd.5) 121 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:36.350884+0000 osd.5 (osd.5) 122 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:36.365688+0000 osd.7 (osd.7) 131 : cluster [DBG] 63.18 scrub starts 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:36.369150+0000 osd.7 (osd.7) 132 : cluster [DBG] 63.18 scrub ok 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:36.599661+0000 osd.4 (osd.4) 123 : cluster [DBG] 4.d scrub starts 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:36.603728+0000 osd.4 (osd.4) 124 : cluster [DBG] 4.d scrub ok 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:37.026541+0000 mon.a (mon.0) 1884 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:37.033071+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi087-82051-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi087-82051-2"}]': finished 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:37.033195+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-31"}]': finished 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:37.033268+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi087-82011-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:37.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:37.033324+0000 mon.a (mon.0) 1888 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-02-23T00:33:37.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:37.036335+0000 mon.c (mon.2) 333 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi087-82011-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:37.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:37.036632+0000 mon.c (mon.2) 334 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:37.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:37.036771+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi087-82011-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:37.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:37.037529+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:37.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:37.252591+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.87:0/54245078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:37.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:37.252822+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:37.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:37.037529+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:37.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:37.252591+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.87:0/54245078' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:37.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:37.252822+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:33:38.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.271+0000 7f96d37fe700 1 -- 172.21.15.87:0/54245078 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v267) v1 ==== 101+0+0 (secure 0 0 0) 0x7f96d8004490 con 0x7f96e40f7ba0 2024-02-23T00:33:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 10 librados: watch_flush enter 2024-02-23T00:33:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 10 librados: watch_flush exit 2024-02-23T00:33:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 20 client.44764.objecter shutdown clearing up homeless session... 2024-02-23T00:33:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 10 client.44764.objecter successfully canceled tick 2024-02-23T00:33:38.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 -- 172.21.15.87:0/54245078 >> v2:172.21.15.87:6800/1231597463 conn(0x7f96cc062c90 msgr2=0x7f96cc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 --2- 172.21.15.87:0/54245078 >> v2:172.21.15.87:6800/1231597463 conn(0x7f96cc062c90 0x7f96cc065150 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 rx=0x7f96e4107b50 tx=0x7f96d4009060).stop 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 -- 172.21.15.87:0/54245078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f7ba0 msgr2=0x7f96e4102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 --2- 172.21.15.87:0/54245078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f7ba0 0x7f96e4102980 secure :-1 s=READY pgs=4981 cs=0 l=1 rev1=1 rx=0x7f96d800b1a0 tx=0x7f96d8007f90).stop 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 -- 172.21.15.87:0/54245078 shutdown_connections 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 --2- 172.21.15.87:0/54245078 >> v2:172.21.15.87:6800/1231597463 conn(0x7f96cc062c90 0x7f96cc065150 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 --2- 172.21.15.87:0/54245078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f96e40fd4a0 0x7f96e41230c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 --2- 172.21.15.87:0/54245078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f96e40f7ba0 0x7f96e4102980 unknown :-1 s=CLOSED pgs=4981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 --2- 172.21.15.87:0/54245078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f96e40fcb40 0x7f96e4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 -- 172.21.15.87:0/54245078 >> 172.21.15.87:0/54245078 conn(0x7f96e406c370 msgr2=0x7f96e40f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 -- 172.21.15.87:0/54245078 shutdown_connections 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 -- 172.21.15.87:0/54245078 wait complete. 2024-02-23T00:33:38.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:38.274+0000 7f96d17fa700 1 librados: shutdown 2024-02-23T00:33:38.277 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:33:38.443 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAiontry=3 expected=3 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:c4fdafeb:::6:head 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:c4fdafeb:::6:head expected=19:c4fdafeb:::6:head 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:c4fdafeb:::6:head -> 6 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=6 expected=6 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:bd63b0f1:::8:head 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:bd63b0f1:::8:head expected=19:bd63b0f1:::8:head 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:bd63b0f1:::8:head -> 8 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=8 expected=8 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:b29083e3:::5:head 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:b29083e3:::5:head expected=19:b29083e3:::5:head 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:b29083e3:::5:head -> 5 2024-02-23T00:33:38.444 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=5 expected=5 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:89d3ae78:::11:head 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:89d3ae78:::11:head expected=19:89d3ae78:::11:head 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:89d3ae78:::11:head -> 11 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=11 expected=11 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:863748b0:::15:head 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:863748b0:::15:head expected=19:863748b0:::15:head 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:863748b0:::15:head -> 15 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=15 expected=15 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:6cac518f:::0:head 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:6cac518f:::0:head expected=19:6cac518f:::0:head 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:6cac518f:::0:head -> 0 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=0 expected=0 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:62a1935d:::14:head 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:62a1935d:::14:head expected=19:62a1935d:::14:head 2024-02-23T00:33:38.445 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:62a1935d:::14:head -> 14 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=14 expected=14 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : seek to 19:02547ec2:::1:head 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : cursor()=19:02547ec2:::1:head expected=19:02547ec2:::1:head 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: > 19:02547ec2:::1:head -> 1 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: : entry=1 expected=1 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (217 ms) 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (50017 ms) 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 0/5 -> MIN 19:33333333::::head 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 1/5 -> 19:33333333::::head 19:66666666::::head 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 2/5 -> 19:66666666::::head 19:99999999::::head 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 3/5 -> 19:99999999::::head 19:cccccccc::::head 2024-02-23T00:33:38.446 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: split 4/5 -> 19:cccccccc::::head MAX 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (113827 ms) 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 7 tests from LibRadosList (164616 ms total) 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1372 ms) 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo1,foo2,foo3 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo1 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo2 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo3 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo1,foo4,foo5 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo4 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo5 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo1 2024-02-23T00:33:38.447 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset foo6,foo7 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo7 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: foo6 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo4 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo5 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns2:foo7 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns2:foo6 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: ns1:foo1 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo1 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo2 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: :foo3 2024-02-23T00:33:38.448 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (34 ms) 2024-02-23T00:33:38.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:37.172181+0000 osd.1 (osd.1) 127 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:37.175645+0000 osd.1 (osd.1) 128 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:37.389208+0000 osd.7 (osd.7) 133 : cluster [DBG] 63.b scrub starts 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:37.391198+0000 osd.7 (osd.7) 134 : cluster [DBG] 63.b scrub ok 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.028299+0000 mon.a (mon.0) 1892 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:38.271479+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]': finished 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:38.271637+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.271712+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:38.272101+0000 mon.c (mon.2) 335 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:38.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:38.272639+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:37.172181+0000 osd.1 (osd.1) 127 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:37.175645+0000 osd.1 (osd.1) 128 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:37.389208+0000 osd.7 (osd.7) 133 : cluster [DBG] 63.b scrub starts 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:37.391198+0000 osd.7 (osd.7) 134 : cluster [DBG] 63.b scrub ok 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.028299+0000 mon.a (mon.0) 1892 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:38.271479+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]': finished 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:38.271637+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.271712+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:38.272101+0000 mon.c (mon.2) 335 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:38.272639+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:37.172181+0000 osd.1 (osd.1) 127 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:33:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:37.175645+0000 osd.1 (osd.1) 128 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:33:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:37.389208+0000 osd.7 (osd.7) 133 : cluster [DBG] 63.b scrub starts 2024-02-23T00:33:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:37.391198+0000 osd.7 (osd.7) 134 : cluster [DBG] 63.b scrub ok 2024-02-23T00:33:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.028299+0000 mon.a (mon.0) 1892 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:33:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:38.271479+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi087-82019-47"}]': finished 2024-02-23T00:33:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:38.271637+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:33:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.271712+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-02-23T00:33:38.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:38.272101+0000 mon.c (mon.2) 335 : audit [INF] from='client.? 172.21.15.87:0/180203885' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:38.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:38.272639+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]: dispatch 2024-02-23T00:33:38.978 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:38.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:39.084 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:39.110 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:39.129 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:39.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:39.233 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:39.260 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:39.279 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:39.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:39.385 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:39.412 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:39.432 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:39.434 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:39.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:39.470 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:39.539 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:39.566 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:39.584 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:39.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:39.688 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:39.714 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:39.729 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:37.605166+0000 mgr.y (mgr.24437) 283 : cluster [DBG] pgmap v382: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-23T00:33:39.729 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.162182+0000 osd.1 (osd.1) 129 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.163150+0000 osd.1 (osd.1) 130 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.434366+0000 osd.6 (osd.6) 97 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.437851+0000 osd.6 (osd.6) 98 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.520845+0000 osd.0 (osd.0) 143 : cluster [DBG] 6.16 deep-scrub starts 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.521753+0000 osd.0 (osd.0) 144 : cluster [DBG] 6.16 deep-scrub ok 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.741364+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.6 deep-scrub starts 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:38.742204+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.6 deep-scrub ok 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:38.820207+0000 mon.a (mon.0) 1897 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.036267+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi087-82011-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi087-82011-39"}]': finished 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.036374+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]': finished 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:39.036475+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.040739+0000 mon.c (mon.2) 336 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:39.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.041121+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.046827+0000 mon.b (mon.1) 236 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.047142+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.063039+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.063328+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.064095+0000 mon.c (mon.2) 338 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.064381+0000 mon.a (mon.0) 1904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.064753+0000 mon.c (mon.2) 339 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi087-82019-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.065645+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi087-82019-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.069068+0000 mon.a (mon.0) 1906 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:37.605166+0000 mgr.y (mgr.24437) 283 : cluster [DBG] pgmap v382: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-23T00:33:39.731 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.162182+0000 osd.1 (osd.1) 129 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.163150+0000 osd.1 (osd.1) 130 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.434366+0000 osd.6 (osd.6) 97 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.437851+0000 osd.6 (osd.6) 98 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.520845+0000 osd.0 (osd.0) 143 : cluster [DBG] 6.16 deep-scrub starts 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.521753+0000 osd.0 (osd.0) 144 : cluster [DBG] 6.16 deep-scrub ok 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.741364+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.6 deep-scrub starts 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:38.742204+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.6 deep-scrub ok 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:38.820207+0000 mon.a (mon.0) 1897 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.036267+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi087-82011-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi087-82011-39"}]': finished 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.036374+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]': finished 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:39.036475+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.040739+0000 mon.c (mon.2) 336 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:39.732 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.041121+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:39.733 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.046827+0000 mon.b (mon.1) 236 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:39.733 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.047142+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:39.733 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.063039+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.733 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.063328+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.733 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.064095+0000 mon.c (mon.2) 338 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.733 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.064381+0000 mon.a (mon.0) 1904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.733 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.064753+0000 mon.c (mon.2) 339 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi087-82019-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:39.733 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.065645+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi087-82019-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:39.733 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.069068+0000 mon.a (mon.0) 1906 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:39.733 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:39.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:39.836 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:39.861 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:37.605166+0000 mgr.y (mgr.24437) 283 : cluster [DBG] pgmap v382: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-02-23T00:33:39.861 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.162182+0000 osd.1 (osd.1) 129 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:33:39.861 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.163150+0000 osd.1 (osd.1) 130 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:33:39.861 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.434366+0000 osd.6 (osd.6) 97 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.437851+0000 osd.6 (osd.6) 98 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.520845+0000 osd.0 (osd.0) 143 : cluster [DBG] 6.16 deep-scrub starts 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.521753+0000 osd.0 (osd.0) 144 : cluster [DBG] 6.16 deep-scrub ok 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.741364+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.6 deep-scrub starts 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:38.742204+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.6 deep-scrub ok 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:38.820207+0000 mon.a (mon.0) 1897 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.036267+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi087-82011-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi087-82011-39"}]': finished 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.036374+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi087-82019-47"}]': finished 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:39.036475+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.040739+0000 mon.c (mon.2) 336 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.041121+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.046827+0000 mon.b (mon.1) 236 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:39.862 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.047142+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:39.863 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.063039+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.863 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.063328+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.863 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.064095+0000 mon.c (mon.2) 338 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.863 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.064381+0000 mon.a (mon.0) 1904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:39.863 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.064753+0000 mon.c (mon.2) 339 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi087-82019-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:39.863 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.065645+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi087-82019-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:39.863 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.069068+0000 mon.a (mon.0) 1906 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:39.863 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:39.881 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:39.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:39.985 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:40.011 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:40.029 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:40.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:40.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:40.133 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:40.157 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:33:40.159 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:40.177 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:40.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:40.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:40.281 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:40.312 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:40.410 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:40.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:40.515 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:40.542 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:40.561 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:40.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:40.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:39.137488+0000 osd.1 (osd.1) 131 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:33:40.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:39.140430+0000 osd.1 (osd.1) 132 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:33:40.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:39.349674+0000 osd.5 (osd.5) 123 : cluster [DBG] 63.1f scrub starts 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:39.351567+0000 osd.5 (osd.5) 124 : cluster [DBG] 63.1f scrub ok 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.560716+0000 mon.a (mon.0) 1907 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.561390+0000 mon.a (mon.0) 1908 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.568246+0000 mon.a (mon.0) 1909 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:39.605634+0000 mgr.y (mgr.24437) 284 : cluster [DBG] pgmap v385: 246 pgs: 40 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:39.645123+0000 mon.a (mon.0) 1910 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.038249+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]': finished 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.038378+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.038480+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi087-82019-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:40.038542+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.040141+0000 mon.c (mon.2) 340 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:40.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.040550+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:39.137488+0000 osd.1 (osd.1) 131 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:39.140430+0000 osd.1 (osd.1) 132 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:39.349674+0000 osd.5 (osd.5) 123 : cluster [DBG] 63.1f scrub starts 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:39.351567+0000 osd.5 (osd.5) 124 : cluster [DBG] 63.1f scrub ok 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.560716+0000 mon.a (mon.0) 1907 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.561390+0000 mon.a (mon.0) 1908 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.568246+0000 mon.a (mon.0) 1909 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:39.605634+0000 mgr.y (mgr.24437) 284 : cluster [DBG] pgmap v385: 246 pgs: 40 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:39.645123+0000 mon.a (mon.0) 1910 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.038249+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]': finished 2024-02-23T00:33:40.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.038378+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.038480+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi087-82019-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:40.038542+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-02-23T00:33:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.040141+0000 mon.c (mon.2) 340 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.040550+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.046318+0000 mon.c (mon.2) 341 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi087-82019-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.047083+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi087-82019-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.049316+0000 mon.b (mon.1) 237 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.049957+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.046318+0000 mon.c (mon.2) 341 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi087-82019-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.047083+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi087-82019-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:40.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.049316+0000 mon.b (mon.1) 237 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:40.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.049957+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:40.664 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:40.690 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:40.709 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:40.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:40.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:40.813 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:39.137488+0000 osd.1 (osd.1) 131 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:39.140430+0000 osd.1 (osd.1) 132 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:39.349674+0000 osd.5 (osd.5) 123 : cluster [DBG] 63.1f scrub starts 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:39.351567+0000 osd.5 (osd.5) 124 : cluster [DBG] 63.1f scrub ok 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.560716+0000 mon.a (mon.0) 1907 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.561390+0000 mon.a (mon.0) 1908 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.568246+0000 mon.a (mon.0) 1909 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:39.605634+0000 mgr.y (mgr.24437) 284 : cluster [DBG] pgmap v385: 246 pgs: 40 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:39.645123+0000 mon.a (mon.0) 1910 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.038249+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi087-82051-2"}]': finished 2024-02-23T00:33:40.839 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.038378+0000 mon.a (mon.0) 1912 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:40.840 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.038480+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi087-82019-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:40.840 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:40.038542+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-02-23T00:33:40.840 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.040141+0000 mon.c (mon.2) 340 : audit [INF] from='client.? 172.21.15.87:0/3252933703' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:40.840 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.040550+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]: dispatch 2024-02-23T00:33:40.840 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.046318+0000 mon.c (mon.2) 341 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi087-82019-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:40.840 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.047083+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi087-82019-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:40.840 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.049316+0000 mon.b (mon.1) 237 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:40.840 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.049957+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:40.840 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:33:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:33:40.841 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:40.858 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:40.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:40.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:40.962 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:40.991 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:41.464 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:33:41.918 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:41.941 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:41.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:42.047 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:42.077 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:42.097 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:42.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:40.766213+0000 mgr.y (mgr.24437) 285 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:41.038875+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]': finished 2024-02-23T00:33:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:41.278609+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:41.278809+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-02-23T00:33:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:41.279591+0000 mon.c (mon.2) 342 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:41.279855+0000 mon.b (mon.1) 238 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:41.501407+0000 osd.6 (osd.6) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:42.182 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:41.501564+0000 osd.6 (osd.6) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:41.530858+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:41.531068+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:41.698315+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.030967+0000 mon.a (mon.0) 1923 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi087-82019-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi087-82019-48"}]': finished 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.031090+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]': finished 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.031155+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]': finished 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:42.031209+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.035495+0000 mon.c (mon.2) 343 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.035820+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:40.766213+0000 mgr.y (mgr.24437) 285 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:41.038875+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]': finished 2024-02-23T00:33:42.183 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:41.278609+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:41.278809+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:41.279591+0000 mon.c (mon.2) 342 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:41.279855+0000 mon.b (mon.1) 238 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:41.501407+0000 osd.6 (osd.6) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:41.501564+0000 osd.6 (osd.6) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:41.530858+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:41.531068+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:41.698315+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.030967+0000 mon.a (mon.0) 1923 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi087-82019-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi087-82019-48"}]': finished 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.031090+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]': finished 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.031155+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]': finished 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:42.031209+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-02-23T00:33:42.184 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.035495+0000 mon.c (mon.2) 343 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.185 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.035820+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.201 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:42.231 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:40.766213+0000 mgr.y (mgr.24437) 285 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:41.038875+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi087-82051-2"}]': finished 2024-02-23T00:33:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:41.278609+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:41.278809+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:41.279591+0000 mon.c (mon.2) 342 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:41.279855+0000 mon.b (mon.1) 238 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:41.501407+0000 osd.6 (osd.6) 99 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:41.501564+0000 osd.6 (osd.6) 100 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:41.530858+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:41.531068+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:41.698315+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.030967+0000 mon.a (mon.0) 1923 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi087-82019-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi087-82019-48"}]': finished 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.031090+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]': finished 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.031155+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi087-82011-39"}]': finished 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:42.031209+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-02-23T00:33:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.035495+0000 mon.c (mon.2) 343 : audit [INF] from='client.? 172.21.15.87:0/3413612997' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.035820+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]: dispatch 2024-02-23T00:33:42.516 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:33:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:42] "GET /metrics HTTP/1.1" 200 188967 "" "Prometheus/2.33.4" 2024-02-23T00:33:42.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.766+0000 7f5778cc6700 1 -- 172.21.15.87:0/4174670229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57741022b0 msgr2=0x7f57741046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:42.768 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.766+0000 7f5778cc6700 1 --2- 172.21.15.87:0/4174670229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57741022b0 0x7f57741046a0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7f576c004870 tx=0x7f576c009cb0).stop 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.766+0000 7f5778cc6700 1 -- 172.21.15.87:0/4174670229 shutdown_connections 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.766+0000 7f5778cc6700 1 --2- 172.21.15.87:0/4174670229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57741022b0 0x7f57741046a0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.767+0000 7f5778cc6700 1 --2- 172.21.15.87:0/4174670229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57740f8d50 0x7f57740f9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.767+0000 7f5778cc6700 1 --2- 172.21.15.87:0/4174670229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57740f9620 0x7f5774101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.767+0000 7f5778cc6700 1 -- 172.21.15.87:0/4174670229 >> 172.21.15.87:0/4174670229 conn(0x7f57740f2820 msgr2=0x7f57740f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.767+0000 7f5778cc6700 1 -- 172.21.15.87:0/4174670229 shutdown_connections 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.767+0000 7f5778cc6700 1 -- 172.21.15.87:0/4174670229 wait complete. 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.767+0000 7f5778cc6700 1 librados: starting msgr at 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.767+0000 7f5778cc6700 1 librados: starting objecter 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 Processor -- start 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 -- start start 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 librados: setting wanted keys 2024-02-23T00:33:42.769 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 librados: calling monclient init 2024-02-23T00:33:42.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57740f8d50 0x7f577410df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:42.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57740f9620 0x7f577410e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:42.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57741022b0 0x7f57741080a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:42.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5774108590 con 0x7f57740f9620 2024-02-23T00:33:42.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5774108700 con 0x7f57741022b0 2024-02-23T00:33:42.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5778cc6700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57741088c0 con 0x7f57740f8d50 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f577259c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57740f8d50 0x7f577410df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f577259c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57740f8d50 0x7f577410df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:45648/0 (socket says 172.21.15.87:45648) 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f577259c700 1 -- 172.21.15.87:0/1608508096 learned_addr learned my addr 172.21.15.87:0/1608508096 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5771d9b700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57740f9620 0x7f577410e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f577259c700 1 -- 172.21.15.87:0/1608508096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57741022b0 msgr2=0x7f57741080a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5772d9d700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57741022b0 0x7f57741080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f577259c700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57741022b0 0x7f57741080a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f577259c700 1 -- 172.21.15.87:0/1608508096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57740f9620 msgr2=0x7f577410e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f577259c700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57740f9620 0x7f577410e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f577259c700 1 -- 172.21.15.87:0/1608508096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57640097c0 con 0x7f57740f8d50 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5771d9b700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57740f9620 0x7f577410e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.768+0000 7f5772d9d700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57741022b0 0x7f57741080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f577259c700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57740f8d50 0x7f577410df00 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7f5764001e70 tx=0x7f5764004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:42.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f576b7fe700 10 client.?.objecter ms_handle_connect 0x7f57740f8d50 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f576b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f576b7fe700 1 -- 172.21.15.87:0/1608508096 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57640099f0 con 0x7f57740f8d50 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c005040 con 0x7f57740f8d50 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f576b7fe700 1 -- 172.21.15.87:0/1608508096 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57640095a0 con 0x7f57740f8d50 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f576b7fe700 1 -- 172.21.15.87:0/1608508096 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764007d50 con 0x7f57740f8d50 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f5778cc6700 10 client.37331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5774114d30 con 0x7f57740f8d50 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.769+0000 7f5778cc6700 1 librados: init done 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.770+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577405fa60 con 0x7f57740f8d50 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.770+0000 7f576b7fe700 1 -- 172.21.15.87:0/1608508096 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f576400fb80 con 0x7f57740f8d50 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.770+0000 7f576b7fe700 1 --2- 172.21.15.87:0/1608508096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5760062c90 0x7f5760065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.771+0000 7f576b7fe700 1 -- 172.21.15.87:0/1608508096 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(271..271 src has 38..271) v4 ==== 9131+0+0 (secure 0 0 0) 0x7f576407dfb0 con 0x7f57740f8d50 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.771+0000 7f576b7fe700 10 client.37331.objecter ms_dispatch 0x7f5774001550 osd_map(271..271 src has 38..271) v4 2024-02-23T00:33:42.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.771+0000 7f576b7fe700 3 client.37331.objecter handle_osd_map got epochs [271,271] > 0 2024-02-23T00:33:42.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.771+0000 7f576b7fe700 3 client.37331.objecter handle_osd_map decoding full epoch 271 2024-02-23T00:33:42.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.771+0000 7f5771d9b700 1 --2- 172.21.15.87:0/1608508096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5760062c90 0x7f5760065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:42.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.771+0000 7f576b7fe700 20 client.37331.objecter dump_active .. 0 homeless 2024-02-23T00:33:42.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.771+0000 7f576b7fe700 10 client.37331.objecter ms_handle_connect 0x7f5760062c90 2024-02-23T00:33:42.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.771+0000 7f5771d9b700 1 --2- 172.21.15.87:0/1608508096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5760062c90 0x7f5760065150 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 rx=0x7f5774109840 tx=0x7f575c009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:42.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.775+0000 7f576b7fe700 1 -- 172.21.15.87:0/1608508096 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5764051050 con 0x7f57740f8d50 2024-02-23T00:33:42.778 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:42.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:42.883 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:42.915 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:42.935 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:42.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:42.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:42.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.946+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f577405ee80 con 0x7f57740f8d50 2024-02-23T00:33:42.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.947+0000 7f576b7fe700 1 -- 172.21.15.87:0/1608508096 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v271) v1 ==== 81+0+52657 (secure 0 0 0) 0x7f577405ee80 con 0x7f57740f8d50 2024-02-23T00:33:42.950 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:33:42.950 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:33:42.950 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 271, 2024-02-23T00:33:42.950 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:33:42.950 INFO:teuthology.orchestra.run.smithi087.stdout: "created": "2024-02-23T00:21:28.861318+0000", 2024-02-23T00:33:42.950 INFO:teuthology.orchestra.run.smithi087.stdout: "modified": "2024-02-23T00:33:42.019789+0000", 2024-02-23T00:33:42.950 INFO:teuthology.orchestra.run.smithi087.stdout: "last_up_change": "2024-02-23T00:27:09.633391+0000", 2024-02-23T00:33:42.950 INFO:teuthology.orchestra.run.smithi087.stdout: "last_in_change": "2024-02-23T00:26:52.603467+0000", 2024-02-23T00:33:42.950 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-23T00:33:42.951 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_num": 5799936, 2024-02-23T00:33:42.951 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_set": [ 2024-02-23T00:33:42.951 INFO:teuthology.orchestra.run.smithi087.stdout: "pglog_hardlimit", 2024-02-23T00:33:42.951 INFO:teuthology.orchestra.run.smithi087.stdout: "purged_snapdirs", 2024-02-23T00:33:42.951 INFO:teuthology.orchestra.run.smithi087.stdout: "recovery_deletes", 2024-02-23T00:33:42.951 INFO:teuthology.orchestra.run.smithi087.stdout: "sortbitwise" 2024-02-23T00:33:42.951 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:33:42.951 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_version": 72, 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "full_ratio": 0.94999998807907104, 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_snapshot": "", 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_max": 179, 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd": 8, 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "require_min_compat_client": "luminous", 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "min_compat_client": "luminous", 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "require_osd_release": "pacific", 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: "pools": [ 2024-02-23T00:33:42.952 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "device_health_metrics", 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:25:09.208309+0000", 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:42.953 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 1, 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 1, 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 1, 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 1, 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 1, 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:42.954 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "68", 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:42.955 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:42.956 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:42.957 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_max": 32, 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_min": 1 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_devicehealth": {} 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:42.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 2, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": ".rgw.root", 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.562186+0000", 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:42.959 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:42.960 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "174", 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:42.961 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:42.962 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:42.963 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:42.963 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:42.963 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.963 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:42.963 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:42.963 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:42.963 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:42.963 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 3, 2024-02-23T00:33:42.964 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "iscsi", 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.989206+0000", 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 13, 2024-02-23T00:33:42.965 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 13, 2024-02-23T00:33:42.966 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 13, 2024-02-23T00:33:42.966 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 13, 2024-02-23T00:33:42.966 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 13, 2024-02-23T00:33:42.966 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:42.966 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:42.966 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:42.966 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:42.966 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:42.966 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "137", 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "135", 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:42.967 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:42.968 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:42.969 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd": {} 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 4, 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.log", 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:17.282440+0000", 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:42.970 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:42.971 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "259", 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:33:42.972 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:42.973 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:42.974 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 5, 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.control", 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:19.262164+0000", 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:42.975 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:33:42.976 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "60", 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:33:42.977 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:42.978 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:42.979 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 6, 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.meta", 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:21.319951+0000", 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:42.980 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:33:42.981 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "62", 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:33:42.982 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:42.983 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:42.984 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_bias": 4 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 22, 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-23T00:33:42.985 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:46.071806+0000", 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:42.986 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:42.987 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:42.988 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:42.989 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:42.990 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-6", 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:31:04.375891+0000", 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:42.991 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:42.992 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "271", 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "271", 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "271", 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "271", 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 18, 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 270, 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:42.993 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [ 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: 177 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": 177, 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": 177, 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:42.994 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:42.995 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 177, 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-33", 2024-02-23T00:33:42.996 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:33:38.524165+0000", 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:33:42.997 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "271", 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "271", 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "271", 2024-02-23T00:33:42.998 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "271", 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 18, 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 270, 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": 63, 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:42.999 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:43.000 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.001 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 178, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "RoundTripPP_smithi087-82019-48", 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:33:41.278529+0000", 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 32769, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,creating", 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 3, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 1, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:43.002 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 8, 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 8, 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 8, 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 8, 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 8, 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.003 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "271", 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:43.004 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "testprofile-RoundTripPP_smithi087-82019-48", 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:43.005 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 8192, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.006 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 179, 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82051-3", 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:33:41.530387+0000", 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 32769, 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,creating", 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:33:43.007 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:33:43.008 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "271", 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:33:43.009 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:33:43.010 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.011 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7", 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 8, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 268, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:33:43.012 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6802", 2024-02-23T00:33:43.013 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6804", 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:33:43.014 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6803", 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6801/2153191863", 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6802/2153191863", 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6804/2153191863", 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6803/2153191863", 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:33:43.015 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "85525ce8-38d5-4840-8057-547cb176c859", 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 13, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 268, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:33:43.016 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6805", 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6806", 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:33:43.017 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6808", 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.018 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6807", 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6805/685354171", 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6806/685354171", 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6808/685354171", 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6807/685354171", 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.019 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa", 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 18, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 268, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:33:43.020 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6809", 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.021 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6810", 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6812", 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.022 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6811", 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6809/3605394228", 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6810/3605394228", 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6812/3605394228", 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6811/3605394228", 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:33:43.023 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a", 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 26, 2024-02-23T00:33:43.024 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 268, 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6813", 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.025 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6814", 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6816", 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.026 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6815", 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6813/128059790", 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6814/128059790", 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6816/128059790", 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6815/128059790", 2024-02-23T00:33:43.027 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c", 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 31, 2024-02-23T00:33:43.028 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 268, 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6800", 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6801", 2024-02-23T00:33:43.029 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6803", 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.030 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6802", 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6800/4175844757", 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6801/4175844757", 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4175844757", 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4175844757", 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:33:43.031 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d", 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 38, 2024-02-23T00:33:43.032 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 268, 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6804", 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.033 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6805", 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6807", 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.034 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6806", 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6804/2550013795", 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6805/2550013795", 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2550013795", 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2550013795", 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:33:43.035 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01", 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 44, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 268, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:33:43.036 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6808", 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.037 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6809", 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6811", 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.038 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6810", 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6808/2509408498", 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6809/2509408498", 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6811/2509408498", 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6810/2509408498", 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:33:43.039 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939", 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 51, 2024-02-23T00:33:43.040 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 268, 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6812", 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.041 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6813", 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6815", 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.042 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6814", 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6812/373787000", 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6813/373787000", 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6815/373787000", 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6814/373787000", 2024-02-23T00:33:43.043 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_xinfo": [ 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:33:43.044 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:41.672557+0000", 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:33:43.045 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:33:26.210051+0000", 2024-02-23T00:33:43.047 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:33:43.047 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.047 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.047 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:33:43.047 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:33:43.047 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:30.427929+0000", 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:33:43.048 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:33:26.555748+0000", 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:42.185296+0000", 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:33:43.049 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:32:15.088109+0000", 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:33:43.050 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:33:41.501567+0000", 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:33:35.065450+0000", 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.051 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap": [], 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap_items": [ 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.9", 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.13", 2024-02-23T00:33:43.052 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.17", 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 5 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.053 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_temp": [], 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_temp": [], 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "blocklist": { 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1121018082": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/2138166229": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/27794119": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/1675126043": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/3767016525": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/969918521": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/795095363": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2273151327": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1685406414": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2093733106": "2024-02-24T00:21:58.144574+0000" 2024-02-23T00:33:43.054 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "range_blocklist": {}, 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profiles": { 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "default": { 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van" 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "testprofile-RoundTripPP_smithi087-82019-48": { 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-device-class": "", 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:33:43.055 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-root": "default", 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van", 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "w": "8" 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps_queue": [ 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 7, 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.056 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:33:43.057 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 21, 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.058 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 61, 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:33:43.059 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 78, 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.060 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 86, 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 95, 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 4, 2024-02-23T00:33:43.062 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 4 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 107, 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 112, 2024-02-23T00:33:43.063 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 125, 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 10, 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.065 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 133, 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 154, 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 14, 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.066 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 164, 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 17, 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: "new_removed_snaps": [], 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: "new_purged_snaps": [], 2024-02-23T00:33:43.067 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_node_flags": {}, 2024-02-23T00:33:43.068 INFO:teuthology.orchestra.run.smithi087.stdout: "device_class_flags": {}, 2024-02-23T00:33:43.068 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode": { 2024-02-23T00:33:43.068 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_enabled": false, 2024-02-23T00:33:43.068 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_bucket_count": 0, 2024-02-23T00:33:43.068 INFO:teuthology.orchestra.run.smithi087.stdout: "degraded_stretch_mode": 0, 2024-02-23T00:33:43.068 INFO:teuthology.orchestra.run.smithi087.stdout: "recovering_stretch_mode": 0, 2024-02-23T00:33:43.068 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_bucket": 0 2024-02-23T00:33:43.068 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:43.068 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:33:43.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 10 librados: watch_flush enter 2024-02-23T00:33:43.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 10 librados: watch_flush exit 2024-02-23T00:33:43.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 20 client.37331.objecter shutdown clearing up homeless session... 2024-02-23T00:33:43.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 10 client.37331.objecter successfully canceled tick 2024-02-23T00:33:43.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5760062c90 msgr2=0x7f5760065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:43.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 --2- 172.21.15.87:0/1608508096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5760062c90 0x7f5760065150 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 rx=0x7f5774109840 tx=0x7f575c009040).stop 2024-02-23T00:33:43.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57740f8d50 msgr2=0x7f577410df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57740f8d50 0x7f577410df00 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7f5764001e70 tx=0x7f5764004c50).stop 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 shutdown_connections 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 --2- 172.21.15.87:0/1608508096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5760062c90 0x7f5760065150 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57741022b0 0x7f57741080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57740f8d50 0x7f577410df00 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 --2- 172.21.15.87:0/1608508096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57740f9620 0x7f577410e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 >> 172.21.15.87:0/1608508096 conn(0x7f57740f2820 msgr2=0x7f57740f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 shutdown_connections 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 -- 172.21.15.87:0/1608508096 wait complete. 2024-02-23T00:33:43.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:42.949+0000 7f5778cc6700 1 librados: shutdown 2024-02-23T00:33:43.070 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:43.101 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:41.494815+0000 osd.7 (osd.7) 135 : cluster [DBG] 63.13 scrub starts 2024-02-23T00:33:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:41.498227+0000 osd.7 (osd.7) 136 : cluster [DBG] 63.13 scrub ok 2024-02-23T00:33:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:41.606034+0000 mgr.y (mgr.24437) 286 : cluster [DBG] pgmap v388: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:33:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:41.701628+0000 osd.3 (osd.3) 132 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.036861+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-33", "mode": "writeback"}]: dispatch 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.038003+0000 mon.c (mon.2) 344 : audit [INF] from='client.? 172.21.15.87:0/1081717827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82051-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.038362+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-33", "mode": "writeback"}]: dispatch 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.038655+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82051-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:42.577993+0000 osd.0 (osd.0) 145 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:42.579016+0000 osd.0 (osd.0) 146 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:42.946598+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? 172.21.15.87:0/1608508096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:43.022545+0000 mon.a (mon.0) 1930 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.027522+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]': finished 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.027712+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-33", "mode": "writeback"}]': finished 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.027776+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82051-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:43.027836+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:41.494815+0000 osd.7 (osd.7) 135 : cluster [DBG] 63.13 scrub starts 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:41.498227+0000 osd.7 (osd.7) 136 : cluster [DBG] 63.13 scrub ok 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:41.606034+0000 mgr.y (mgr.24437) 286 : cluster [DBG] pgmap v388: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:41.701628+0000 osd.3 (osd.3) 132 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.036861+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-33", "mode": "writeback"}]: dispatch 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.038003+0000 mon.c (mon.2) 344 : audit [INF] from='client.? 172.21.15.87:0/1081717827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82051-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.038362+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-33", "mode": "writeback"}]: dispatch 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.038655+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82051-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:42.577993+0000 osd.0 (osd.0) 145 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:33:43.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:42.579016+0000 osd.0 (osd.0) 146 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:42.946598+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? 172.21.15.87:0/1608508096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:43.022545+0000 mon.a (mon.0) 1930 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.027522+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]': finished 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.027712+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-33", "mode": "writeback"}]': finished 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.027776+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82051-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:43.027836+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:41.494815+0000 osd.7 (osd.7) 135 : cluster [DBG] 63.13 scrub starts 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:41.498227+0000 osd.7 (osd.7) 136 : cluster [DBG] 63.13 scrub ok 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:41.606034+0000 mgr.y (mgr.24437) 286 : cluster [DBG] pgmap v388: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:41.701628+0000 osd.3 (osd.3) 132 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:33:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.036861+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-33", "mode": "writeback"}]: dispatch 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.038003+0000 mon.c (mon.2) 344 : audit [INF] from='client.? 172.21.15.87:0/1081717827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82051-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.038362+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-33", "mode": "writeback"}]: dispatch 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.038655+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82051-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:42.577993+0000 osd.0 (osd.0) 145 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:42.579016+0000 osd.0 (osd.0) 146 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:42.946598+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? 172.21.15.87:0/1608508096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:43.022545+0000 mon.a (mon.0) 1930 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.027522+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi087-82011-39"}]': finished 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.027712+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-33", "mode": "writeback"}]': finished 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.027776+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82051-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:43.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:43.027836+0000 mon.a (mon.0) 1934 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-02-23T00:33:43.632 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:33:43.632 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:33:44.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-02-23T00:33:44.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 1 0 2024-02-23T00:33:44.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 10 0 2024-02-23T00:33:44.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 13 0 2024-02-23T00:33:44.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 7 0 2024-02-23T00:33:44.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 14 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 0 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 15 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 11 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 5 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 8 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 6 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 3 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 4 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 12 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 9 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2 0 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (45 ms) 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 3 tests from LibRadosListEC (1451 ms total) 2024-02-23T00:33:44.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2024-02-23T00:33:44.042 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-02-23T00:33:44.042 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-02-23T00:33:44.042 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2991 ms) 2024-02-23T00:33:44.042 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] 1 test from LibRadosListNP (2991 ms total) 2024-02-23T00:33:44.042 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: 2024-02-23T00:33:44.042 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [----------] Global test environment tear-down 2024-02-23T00:33:44.042 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [==========] 11 tests from 3 test suites ran. (181916 ms total) 2024-02-23T00:33:44.042 INFO:tasks.workunit.client.0.smithi087.stdout: api_list: [ PASSED ] 11 tests. 2024-02-23T00:33:44.043 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:33:44.043 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82134 2024-02-23T00:33:44.043 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82134 2024-02-23T00:33:44.043 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:33:44.043 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82126 2024-02-23T00:33:44.043 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82126 2024-02-23T00:33:44.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:33:44.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82190 2024-02-23T00:33:44.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82190 2024-02-23T00:33:44.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:33:44.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82014 2024-02-23T00:33:44.044 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82014 2024-02-23T00:33:44.059 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:33:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:33:44.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:42.377276+0000 osd.5 (osd.5) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:44.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:42.377535+0000 osd.5 (osd.5) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:44.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:42.506350+0000 osd.7 (osd.7) 137 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:33:44.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:42.509459+0000 osd.7 (osd.7) 138 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:33:44.061 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:42.777259+0000 osd.4 (osd.4) 125 : cluster [DBG] 63.19 deep-scrub starts 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:42.780893+0000 osd.4 (osd.4) 126 : cluster [DBG] 63.19 deep-scrub ok 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.060624+0000 mon.c (mon.2) 346 : audit [INF] from='client.? 172.21.15.87:0/1081717827' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi087-82051-3","pool2":"test-rados-api-smithi087-82051-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.060991+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi087-82051-3","pool2":"test-rados-api-smithi087-82051-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.065411+0000 mon.c (mon.2) 347 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.065657+0000 mon.a (mon.0) 1936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.066308+0000 mon.c (mon.2) 348 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.066536+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.066944+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi087-82011-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:43.067203+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi087-82011-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:44.028261+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi087-82051-3","pool2":"test-rados-api-smithi087-82051-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:44.028365+0000 mon.a (mon.0) 1940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi087-82011-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:44.028430+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-02-23T00:33:44.062 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:44.032546+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:44.063 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:44.034589+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:44.063 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:44.036716+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi087-82011-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.063 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:44.037275+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi087-82011-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:42.377276+0000 osd.5 (osd.5) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:42.377535+0000 osd.5 (osd.5) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:42.506350+0000 osd.7 (osd.7) 137 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:42.509459+0000 osd.7 (osd.7) 138 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:42.777259+0000 osd.4 (osd.4) 125 : cluster [DBG] 63.19 deep-scrub starts 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:42.780893+0000 osd.4 (osd.4) 126 : cluster [DBG] 63.19 deep-scrub ok 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.060624+0000 mon.c (mon.2) 346 : audit [INF] from='client.? 172.21.15.87:0/1081717827' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi087-82051-3","pool2":"test-rados-api-smithi087-82051-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.060991+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi087-82051-3","pool2":"test-rados-api-smithi087-82051-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.065411+0000 mon.c (mon.2) 347 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.065657+0000 mon.a (mon.0) 1936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.066308+0000 mon.c (mon.2) 348 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.066536+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.066944+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi087-82011-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:43.067203+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi087-82011-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:44.028261+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi087-82051-3","pool2":"test-rados-api-smithi087-82051-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-23T00:33:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:44.028365+0000 mon.a (mon.0) 1940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi087-82011-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:44.028430+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-02-23T00:33:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:44.032546+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:44.034589+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:44.036716+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi087-82011-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:44.037275+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi087-82011-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:42.377276+0000 osd.5 (osd.5) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:42.377535+0000 osd.5 (osd.5) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:42.506350+0000 osd.7 (osd.7) 137 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:42.509459+0000 osd.7 (osd.7) 138 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:42.777259+0000 osd.4 (osd.4) 125 : cluster [DBG] 63.19 deep-scrub starts 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:42.780893+0000 osd.4 (osd.4) 126 : cluster [DBG] 63.19 deep-scrub ok 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.060624+0000 mon.c (mon.2) 346 : audit [INF] from='client.? 172.21.15.87:0/1081717827' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi087-82051-3","pool2":"test-rados-api-smithi087-82051-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.060991+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi087-82051-3","pool2":"test-rados-api-smithi087-82051-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.065411+0000 mon.c (mon.2) 347 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.065657+0000 mon.a (mon.0) 1936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.066308+0000 mon.c (mon.2) 348 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.066536+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.066944+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi087-82011-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:43.067203+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi087-82011-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:44.028261+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi087-82051-3","pool2":"test-rados-api-smithi087-82051-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-23T00:33:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:44.028365+0000 mon.a (mon.0) 1940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi087-82011-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:44.028430+0000 mon.a (mon.0) 1941 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-02-23T00:33:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:44.032546+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:44.034589+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:44.036716+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi087-82011-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:44.037275+0000 mon.a (mon.0) 1943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi087-82011-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:44.513 INFO:tasks.thrashosds.thrasher:j is {'epoch': 271, 'fsid': '66db207c-d1e1-11ee-95c0-87774f69a715', 'created': '2024-02-23T00:21:28.861318+0000', 'modified': '2024-02-23T00:33:42.019789+0000', 'last_up_change': '2024-02-23T00:27:09.633391+0000', 'last_in_change': '2024-02-23T00:26:52.603467+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 72, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 179, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '135', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '259', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 22, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-23T00:30:46.071806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 63, 'pool_name': 'test-rados-api-smithi087-82091-6', 'create_time': '2024-02-23T00:31:04.375891+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '271', 'last_force_op_resend_prenautilus': '271', 'last_force_op_resend_preluminous': '271', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 18, 'snap_epoch': 270, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [177], 'tier_of': -1, 'read_tier': 177, 'write_tier': 177, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 177, 'pool_name': 'test-rados-api-smithi087-82091-33', 'create_time': '2024-02-23T00:33:38.524165+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '271', 'last_force_op_resend_prenautilus': '271', 'last_force_op_resend_preluminous': '271', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 18, 'snap_epoch': 270, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 63, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 178, 'pool_name': 'RoundTripPP_smithi087-82019-48', 'create_time': '2024-02-23T00:33:41.278529+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-RoundTripPP_smithi087-82019-48', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 179, 'pool_name': 'test-rados-api-smithi087-82051-3', 'create_time': '2024-02-23T00:33:41.530387+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': 'e3549bad-2891-47b9-bf77-132cadc20bf7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 268, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6801', 'nonce': 2153191863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6802', 'nonce': 2153191863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6804', 'nonce': 2153191863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6803', 'nonce': 2153191863}]}, 'public_addr': '172.21.15.87:6801/2153191863', 'cluster_addr': '172.21.15.87:6802/2153191863', 'heartbeat_back_addr': '172.21.15.87:6804/2153191863', 'heartbeat_front_addr': '172.21.15.87:6803/2153191863', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '85525ce8-38d5-4840-8057-547cb176c859', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 268, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6805', 'nonce': 685354171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6806', 'nonce': 685354171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6808', 'nonce': 685354171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6807', 'nonce': 685354171}]}, 'public_addr': '172.21.15.87:6805/685354171', 'cluster_addr': '172.21.15.87:6806/685354171', 'heartbeat_back_addr': '172.21.15.87:6808/685354171', 'heartbeat_front_addr': '172.21.15.87:6807/685354171', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 268, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6809', 'nonce': 3605394228}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6810', 'nonce': 3605394228}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6812', 'nonce': 3605394228}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6811', 'nonce': 3605394228}]}, 'public_addr': '172.21.15.87:6809/3605394228', 'cluster_addr': '172.21.15.87:6810/3605394228', 'heartbeat_back_addr': '172.21.15.87:6812/3605394228', 'heartbeat_front_addr': '172.21.15.87:6811/3605394228', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 268, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6813', 'nonce': 128059790}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6814', 'nonce': 128059790}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6816', 'nonce': 128059790}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6815', 'nonce': 128059790}]}, 'public_addr': '172.21.15.87:6813/128059790', 'cluster_addr': '172.21.15.87:6814/128059790', 'heartbeat_back_addr': '172.21.15.87:6816/128059790', 'heartbeat_front_addr': '172.21.15.87:6815/128059790', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c3734a04-7343-4251-9b5b-bedb9e8b882c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 268, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6800', 'nonce': 4175844757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6801', 'nonce': 4175844757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6803', 'nonce': 4175844757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6802', 'nonce': 4175844757}]}, 'public_addr': '172.21.15.171:6800/4175844757', 'cluster_addr': '172.21.15.171:6801/4175844757', 'heartbeat_back_addr': '172.21.15.171:6803/4175844757', 'heartbeat_front_addr': '172.21.15.171:6802/4175844757', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fe86232d-96b9-4cbf-9cac-373286fa103d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 268, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6804', 'nonce': 2550013795}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6805', 'nonce': 2550013795}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6807', 'nonce': 2550013795}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6806', 'nonce': 2550013795}]}, 'public_addr': '172.21.15.171:6804/2550013795', 'cluster_addr': '172.21.15.171:6805/2550013795', 'heartbeat_back_addr': '172.21.15.171:6807/2550013795', 'heartbeat_front_addr': '172.21.15.171:6806/2550013795', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 268, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6808', 'nonce': 2509408498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6809', 'nonce': 2509408498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6811', 'nonce': 2509408498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6810', 'nonce': 2509408498}]}, 'public_addr': '172.21.15.171:6808/2509408498', 'cluster_addr': '172.21.15.171:6809/2509408498', 'heartbeat_back_addr': '172.21.15.171:6811/2509408498', 'heartbeat_front_addr': '172.21.15.171:6810/2509408498', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '29ddcf97-369a-4a2c-b3e2-142e84ccf939', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 268, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6812', 'nonce': 373787000}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6813', 'nonce': 373787000}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6815', 'nonce': 373787000}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6814', 'nonce': 373787000}]}, 'public_addr': '172.21.15.171:6812/373787000', 'cluster_addr': '172.21.15.171:6813/373787000', 'heartbeat_back_addr': '172.21.15.171:6815/373787000', 'heartbeat_front_addr': '172.21.15.171:6814/373787000', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:41.672557+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:33:26.210051+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:30.427929+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:33:26.555748+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:42.185296+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:32:15.088109+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:33:41.501567+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:33:35.065450+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.87:6800/1121018082': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/2138166229': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/27794119': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/1675126043': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/3767016525': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/969918521': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/795095363': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2273151327': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/1685406414': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2093733106': '2024-02-24T00:21:58.144574+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-RoundTripPP_smithi087-82019-48': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 21, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 61, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 78, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 86, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-23T00:33:44.513 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:33:45.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:43.298896+0000 osd.5 (osd.5) 127 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:33:45.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:43.299833+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:33:45.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:43.544966+0000 osd.7 (osd.7) 139 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:33:45.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:43.547896+0000 osd.7 (osd.7) 140 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:33:45.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:43.606468+0000 mgr.y (mgr.24437) 287 : cluster [DBG] pgmap v391: 278 pgs: 40 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:45.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:43.796818+0000 osd.4 (osd.4) 127 : cluster [DBG] 2.f scrub starts 2024-02-23T00:33:45.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:43.800355+0000 osd.4 (osd.4) 128 : cluster [DBG] 2.f scrub ok 2024-02-23T00:33:45.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:44.397784+0000 osd.6 (osd.6) 101 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:33:45.471 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-23T00:33:45.480 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:45.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:45.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:44.400777+0000 osd.6 (osd.6) 102 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:33:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:43.298896+0000 osd.5 (osd.5) 127 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:33:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:43.299833+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:33:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:43.544966+0000 osd.7 (osd.7) 139 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:33:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:43.547896+0000 osd.7 (osd.7) 140 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:33:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:43.606468+0000 mgr.y (mgr.24437) 287 : cluster [DBG] pgmap v391: 278 pgs: 40 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:43.796818+0000 osd.4 (osd.4) 127 : cluster [DBG] 2.f scrub starts 2024-02-23T00:33:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:43.800355+0000 osd.4 (osd.4) 128 : cluster [DBG] 2.f scrub ok 2024-02-23T00:33:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:44.397784+0000 osd.6 (osd.6) 101 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:33:45.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:44.400777+0000 osd.6 (osd.6) 102 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:33:45.585 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:45.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:43.298896+0000 osd.5 (osd.5) 127 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:33:45.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:43.299833+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:33:45.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:43.544966+0000 osd.7 (osd.7) 139 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:33:45.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:43.547896+0000 osd.7 (osd.7) 140 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:33:45.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:43.606468+0000 mgr.y (mgr.24437) 287 : cluster [DBG] pgmap v391: 278 pgs: 40 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:45.614 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:43.796818+0000 osd.4 (osd.4) 127 : cluster [DBG] 2.f scrub starts 2024-02-23T00:33:45.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:43.800355+0000 osd.4 (osd.4) 128 : cluster [DBG] 2.f scrub ok 2024-02-23T00:33:45.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:44.397784+0000 osd.6 (osd.6) 101 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:33:45.615 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:44.400777+0000 osd.6 (osd.6) 102 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:33:45.616 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:45.637 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:45.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:45.741 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:45.771 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:45.791 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:45.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:45.895 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:45.923 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:45.942 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:45.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:45.951 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:45.997 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:46.045 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:46.072 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:46.091 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:46.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:46.194 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:46.221 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:46.239 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:46.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:46.343 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:46.370 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:46.388 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:46.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:46.492 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:46.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:44.149124+0000 osd.1 (osd.1) 133 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:33:46.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:44.152010+0000 osd.1 (osd.1) 134 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:33:46.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:45.030223+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]': finished 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:45.033843+0000 mon.c (mon.2) 352 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:45.263857+0000 osd.5 (osd.5) 129 : cluster [DBG] 63.d scrub starts 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:45.265460+0000 osd.5 (osd.5) 130 : cluster [DBG] 63.d scrub ok 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:45.266645+0000 mon.a (mon.0) 1945 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:45.267569+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:45.303150+0000 mon.b (mon.1) 240 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:45.303374+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:45.585739+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:45.586969+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:45.794300+0000 osd.4 (osd.4) 129 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:45.795273+0000 osd.4 (osd.4) 130 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:46.031902+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi087-82011-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi087-82011-40"}]': finished 2024-02-23T00:33:46.518 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:46.031966+0000 mon.a (mon.0) 1949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]': finished 2024-02-23T00:33:46.519 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:46.032003+0000 mon.a (mon.0) 1950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:46.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:44.149124+0000 osd.1 (osd.1) 133 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:33:46.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:44.152010+0000 osd.1 (osd.1) 134 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:33:46.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:45.030223+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]': finished 2024-02-23T00:33:46.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:45.033843+0000 mon.c (mon.2) 352 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:46.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:45.263857+0000 osd.5 (osd.5) 129 : cluster [DBG] 63.d scrub starts 2024-02-23T00:33:46.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:45.265460+0000 osd.5 (osd.5) 130 : cluster [DBG] 63.d scrub ok 2024-02-23T00:33:46.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:45.266645+0000 mon.a (mon.0) 1945 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-02-23T00:33:46.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:45.267569+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:45.303150+0000 mon.b (mon.1) 240 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:45.303374+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:45.585739+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:45.586969+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:45.794300+0000 osd.4 (osd.4) 129 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:45.795273+0000 osd.4 (osd.4) 130 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:46.031902+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi087-82011-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi087-82011-40"}]': finished 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:46.031966+0000 mon.a (mon.0) 1949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]': finished 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:46.032003+0000 mon.a (mon.0) 1950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.032037+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:46.040968+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:46.041201+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:46.053715+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:46.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:46.054878+0000 mon.a (mon.0) 1954 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:46.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:46.055152+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi087-82019-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:46.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.032037+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-02-23T00:33:46.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:46.040968+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:46.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:46.041201+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:46.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:46.053715+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:46.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:46.054878+0000 mon.a (mon.0) 1954 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:46.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:46.055152+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi087-82019-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:46.521 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:44.149124+0000 osd.1 (osd.1) 133 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:33:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:44.152010+0000 osd.1 (osd.1) 134 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:45.030223+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi087-82019-48"}]': finished 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:45.033843+0000 mon.c (mon.2) 352 : audit [INF] from='client.? 172.21.15.87:0/1544052884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:45.263857+0000 osd.5 (osd.5) 129 : cluster [DBG] 63.d scrub starts 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:45.265460+0000 osd.5 (osd.5) 130 : cluster [DBG] 63.d scrub ok 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:45.266645+0000 mon.a (mon.0) 1945 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:45.267569+0000 mon.a (mon.0) 1946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]: dispatch 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:45.303150+0000 mon.b (mon.1) 240 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:45.303374+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:45.585739+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:45.586969+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:45.794300+0000 osd.4 (osd.4) 129 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:33:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:45.795273+0000 osd.4 (osd.4) 130 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:33:46.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:46.031902+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi087-82011-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi087-82011-40"}]': finished 2024-02-23T00:33:46.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:46.031966+0000 mon.a (mon.0) 1949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi087-82019-48"}]': finished 2024-02-23T00:33:46.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:46.032003+0000 mon.a (mon.0) 1950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:46.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.032037+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-02-23T00:33:46.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:46.040968+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:46.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:46.041201+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:46.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:46.053715+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:46.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:46.054878+0000 mon.a (mon.0) 1954 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:46.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:46.055152+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi087-82019-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:45.607324+0000 mgr.y (mgr.24437) 288 : cluster [DBG] pgmap v394: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:33:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.149028+0000 osd.1 (osd.1) 135 : cluster [DBG] 5.12 deep-scrub starts 2024-02-23T00:33:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.151982+0000 osd.1 (osd.1) 136 : cluster [DBG] 5.12 deep-scrub ok 2024-02-23T00:33:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.272679+0000 osd.5 (osd.5) 131 : cluster [DBG] 63.1b scrub starts 2024-02-23T00:33:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.274395+0000 osd.5 (osd.5) 132 : cluster [DBG] 63.1b scrub ok 2024-02-23T00:33:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.392118+0000 osd.4 (osd.4) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.392250+0000 osd.4 (osd.4) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.396287+0000 osd.6 (osd.6) 103 : cluster [DBG] 63.12 scrub starts 2024-02-23T00:33:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.399776+0000 osd.6 (osd.6) 104 : cluster [DBG] 63.12 scrub ok 2024-02-23T00:33:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:47.034556+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]': finished 2024-02-23T00:33:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:47.034650+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi087-82019-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.034707+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-02-23T00:33:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:47.037794+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi087-82019-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:45.607324+0000 mgr.y (mgr.24437) 288 : cluster [DBG] pgmap v394: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:33:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.149028+0000 osd.1 (osd.1) 135 : cluster [DBG] 5.12 deep-scrub starts 2024-02-23T00:33:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.151982+0000 osd.1 (osd.1) 136 : cluster [DBG] 5.12 deep-scrub ok 2024-02-23T00:33:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.272679+0000 osd.5 (osd.5) 131 : cluster [DBG] 63.1b scrub starts 2024-02-23T00:33:47.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.274395+0000 osd.5 (osd.5) 132 : cluster [DBG] 63.1b scrub ok 2024-02-23T00:33:47.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.392118+0000 osd.4 (osd.4) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:47.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.392250+0000 osd.4 (osd.4) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:47.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.396287+0000 osd.6 (osd.6) 103 : cluster [DBG] 63.12 scrub starts 2024-02-23T00:33:47.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.399776+0000 osd.6 (osd.6) 104 : cluster [DBG] 63.12 scrub ok 2024-02-23T00:33:47.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:47.034556+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]': finished 2024-02-23T00:33:47.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:47.034650+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi087-82019-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:47.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.034707+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-02-23T00:33:47.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:47.037794+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi087-82019-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:45.607324+0000 mgr.y (mgr.24437) 288 : cluster [DBG] pgmap v394: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:33:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.149028+0000 osd.1 (osd.1) 135 : cluster [DBG] 5.12 deep-scrub starts 2024-02-23T00:33:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.151982+0000 osd.1 (osd.1) 136 : cluster [DBG] 5.12 deep-scrub ok 2024-02-23T00:33:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.272679+0000 osd.5 (osd.5) 131 : cluster [DBG] 63.1b scrub starts 2024-02-23T00:33:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.274395+0000 osd.5 (osd.5) 132 : cluster [DBG] 63.1b scrub ok 2024-02-23T00:33:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.392118+0000 osd.4 (osd.4) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.392250+0000 osd.4 (osd.4) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.396287+0000 osd.6 (osd.6) 103 : cluster [DBG] 63.12 scrub starts 2024-02-23T00:33:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.399776+0000 osd.6 (osd.6) 104 : cluster [DBG] 63.12 scrub ok 2024-02-23T00:33:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:47.034556+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-33"}]': finished 2024-02-23T00:33:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:47.034650+0000 mon.a (mon.0) 1957 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi087-82019-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.034707+0000 mon.a (mon.0) 1958 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-02-23T00:33:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:47.037794+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi087-82019-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:47.869 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:47.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:47.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:48.008 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:48.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.281+0000 7f3eaebec700 1 -- 172.21.15.87:0/523150349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fd510 msgr2=0x7f3ea80f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.281+0000 7f3eaebec700 1 --2- 172.21.15.87:0/523150349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fd510 0x7f3ea80f6d40 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 rx=0x7f3e94004870 tx=0x7f3e94009cb0).stop 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.281+0000 7f3eaebec700 1 -- 172.21.15.87:0/523150349 shutdown_connections 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.281+0000 7f3eaebec700 1 --2- 172.21.15.87:0/523150349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80f7400 0x7f3ea8103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.281+0000 7f3eaebec700 1 --2- 172.21.15.87:0/523150349 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80fcbb0 0x7f3ea80fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.281+0000 7f3eaebec700 1 --2- 172.21.15.87:0/523150349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fd510 0x7f3ea80f6d40 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.281+0000 7f3eaebec700 1 -- 172.21.15.87:0/523150349 >> 172.21.15.87:0/523150349 conn(0x7f3ea80f2820 msgr2=0x7f3ea80f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.281+0000 7f3eaebec700 1 -- 172.21.15.87:0/523150349 shutdown_connections 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.281+0000 7f3eaebec700 1 -- 172.21.15.87:0/523150349 wait complete. 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.282+0000 7f3eaebec700 1 librados: starting msgr at 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.282+0000 7f3eaebec700 1 librados: starting objecter 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.282+0000 7f3eaebec700 1 Processor -- start 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.282+0000 7f3eaebec700 1 -- start start 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.282+0000 7f3eaebec700 1 librados: setting wanted keys 2024-02-23T00:33:48.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.282+0000 7f3eaebec700 1 librados: calling monclient init 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3eaebec700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80f7400 0x7f3ea80f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3eaebec700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fcbb0 0x7f3ea80f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3eaebec700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80fd510 0x7f3ea80f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3eaebec700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8116730 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3eaebec700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea81168a0 con 0x7f3ea80f7400 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3eaebec700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8116a60 con 0x7f3ea80fd510 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fcbb0 0x7f3ea80f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fcbb0 0x7f3ea80f8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56572/0 (socket says 172.21.15.87:56572) 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea7fff700 1 -- 172.21.15.87:0/3224185643 learned_addr learned my addr 172.21.15.87:0/3224185643 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ead189700 1 --2- 172.21.15.87:0/3224185643 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80fd510 0x7f3ea80f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3eac988700 1 --2- 172.21.15.87:0/3224185643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80f7400 0x7f3ea80f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea7fff700 1 -- 172.21.15.87:0/3224185643 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80fd510 msgr2=0x7f3ea80f9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/3224185643 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80fd510 0x7f3ea80f9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea7fff700 1 -- 172.21.15.87:0/3224185643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80f7400 msgr2=0x7f3ea80f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/3224185643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80f7400 0x7f3ea80f89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea7fff700 1 -- 172.21.15.87:0/3224185643 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e94005040 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ead189700 1 --2- 172.21.15.87:0/3224185643 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80fd510 0x7f3ea80f9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/3224185643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fcbb0 0x7f3ea80f8ea0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 rx=0x7f3e940043a0 tx=0x7f3e94009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea5ffb700 10 client.?.objecter ms_handle_connect 0x7f3ea80fcbb0 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.283+0000 7f3ea5ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.284+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9400e030 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.284+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e94013da0 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.284+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea8116ce0 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.284+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e94018070 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.285+0000 7f3eaebec700 10 client.26892.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:48.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.285+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea8117090 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.285+0000 7f3eaebec700 1 librados: init done 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.285+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3ea805fa60 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.286+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3e9401d420 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.286+0000 7f3ea5ffb700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 0x7f3e98065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(277..277 src has 38..277) v4 ==== 8753+0+0 (secure 0 0 0) 0x7f3e9407ed60 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 osd_map(277..277 src has 38..277) v4 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ea5ffb700 3 client.26892.objecter handle_osd_map got epochs [277,277] > 0 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ea5ffb700 3 client.26892.objecter handle_osd_map decoding full epoch 277 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eac988700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 0x7f3e98065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ea5ffb700 20 client.26892.objecter dump_active .. 0 homeless 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v277) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3e9407f200 con 0x7f3ea80fcbb0 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ea5ffb700 10 client.26892.objecter ms_handle_connect 0x7f3e98062c90 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eac988700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 0x7f3e98065150 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f3e9c004de0 tx=0x7f3e9c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 10 client.26892.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:48.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 1 2024-02-23T00:33:48.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6801/2153191863 conn(0x7f3ea80ffed0 0x7f3ea8100330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:33:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:33:48.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea805ee80 osd=0 3 2024-02-23T00:33:48.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea805ee80 osd=0 3 2024-02-23T00:33:48.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea805ee80 osd=0 3 2024-02-23T00:33:48.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 1 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea805ee80 osd=0 3 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 0 1 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea805ee80 osd=0 4 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 10 client.26892.objecter _send_command 1 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ea8106170 con 0x7f3ea80ffed0 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ead189700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6801/2153191863 conn(0x7f3ea80ffed0 0x7f3ea8100330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ea5ffb700 10 client.26892.objecter ms_handle_connect 0x7f3ea80ffed0 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.287+0000 7f3ead189700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6801/2153191863 conn(0x7f3ea80ffed0 0x7f3ea8100330 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.288+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3ea8106170 con 0x7f3ea80ffed0 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.288+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.288+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 1 = system:0 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.288+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea805ee80 osd=0 4 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.288+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 0 1 2024-02-23T00:33:48.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 10 client.26892.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 2 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea805ee80 osd=0 3 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea805ee80 osd=0 3 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea805ee80 osd=0 3 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 2 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea805ee80 osd=0 3 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 0 2 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea805ee80 osd=0 4 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 10 client.26892.objecter _send_command 2 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.307+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ea80273c0 con 0x7f3ea80ffed0 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3ea80273c0 con 0x7f3ea80ffed0 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea805ee80 osd=0 4 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 0 2 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 10 client.26892.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 3 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6805/685354171 conn(0x7f3ea80598f0 0x7f3ea8059d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8028710 osd=1 3 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8028710 osd=1 3 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8028710 osd=1 3 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 3 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea8028710 osd=1 3 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 1 3 2024-02-23T00:33:48.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8028710 osd=1 4 2024-02-23T00:33:48.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 10 client.26892.objecter _send_command 3 2024-02-23T00:33:48.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.309+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ea811da00 con 0x7f3ea80598f0 2024-02-23T00:33:48.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.310+0000 7f3eac988700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6805/685354171 conn(0x7f3ea80598f0 0x7f3ea8059d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.310+0000 7f3ea5ffb700 10 client.26892.objecter ms_handle_connect 0x7f3ea80598f0 2024-02-23T00:33:48.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.310+0000 7f3eac988700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6805/685354171 conn(0x7f3ea80598f0 0x7f3ea8059d10 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.310+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3ea811da00 con 0x7f3ea80598f0 2024-02-23T00:33:48.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.310+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:33:48.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.310+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 3 = system:0 2024-02-23T00:33:48.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.310+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea8028710 osd=1 4 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.310+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 1 3 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 10 client.26892.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 4 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8028710 osd=1 3 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8028710 osd=1 3 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8028710 osd=1 3 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 4 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea8028710 osd=1 3 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 1 4 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8028710 osd=1 4 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 10 client.26892.objecter _send_command 4 2024-02-23T00:33:48.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.326+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ea8059360 con 0x7f3ea80598f0 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3ea8059360 con 0x7f3ea80598f0 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea8028710 osd=1 4 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 1 4 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 10 client.26892.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 5 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6809/3605394228 conn(0x7f3ea8028270 0x7f3ea811c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8059770 osd=2 3 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8059770 osd=2 3 2024-02-23T00:33:48.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8059770 osd=2 3 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 5 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea8059770 osd=2 3 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 2 5 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8059770 osd=2 4 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 10 client.26892.objecter _send_command 5 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.327+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ea811c910 con 0x7f3ea8028270 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.328+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6809/3605394228 conn(0x7f3ea8028270 0x7f3ea811c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.328+0000 7f3ea5ffb700 10 client.26892.objecter ms_handle_connect 0x7f3ea8028270 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.328+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6809/3605394228 conn(0x7f3ea8028270 0x7f3ea811c420 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.329+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3ea811c910 con 0x7f3ea8028270 2024-02-23T00:33:48.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.329+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:33:48.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.329+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 5 = system:0 2024-02-23T00:33:48.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.329+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea8059770 osd=2 4 2024-02-23T00:33:48.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.329+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 2 5 2024-02-23T00:33:48.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 10 client.26892.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:48.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 6 2024-02-23T00:33:48.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8059770 osd=2 3 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8059770 osd=2 3 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8059770 osd=2 3 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 6 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea8059770 osd=2 3 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 2 6 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8059770 osd=2 4 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 10 client.26892.objecter _send_command 6 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.346+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ea80277d0 con 0x7f3ea8028270 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3ea80277d0 con 0x7f3ea8028270 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea8059770 osd=2 4 2024-02-23T00:33:48.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 2 6 2024-02-23T00:33:48.352 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:33:48.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 10 client.26892.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:48.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 7 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6813/128059790 conn(0x7f3ea811fb80 0x7f3ea8121f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8027b80 osd=3 3 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8027b80 osd=3 3 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8027b80 osd=3 3 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 7 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea8027b80 osd=3 3 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 3 7 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8027b80 osd=3 4 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 10 client.26892.objecter _send_command 7 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ea8124bc0 con 0x7f3ea811fb80 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.348+0000 7f3ead189700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6813/128059790 conn(0x7f3ea811fb80 0x7f3ea8121f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.349+0000 7f3ea5ffb700 10 client.26892.objecter ms_handle_connect 0x7f3ea811fb80 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.349+0000 7f3ead189700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6813/128059790 conn(0x7f3ea811fb80 0x7f3ea8121f70 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.349+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3ea8124bc0 con 0x7f3ea811fb80 2024-02-23T00:33:48.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.349+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:33:48.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.349+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 7 = system:0 2024-02-23T00:33:48.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.349+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea8027b80 osd=3 4 2024-02-23T00:33:48.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.349+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 3 7 2024-02-23T00:33:48.358 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:48.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:48.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:48.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 10 client.26892.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:48.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 8 2024-02-23T00:33:48.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8027b80 osd=3 3 2024-02-23T00:33:48.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8027b80 osd=3 3 2024-02-23T00:33:48.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea8027b80 osd=3 3 2024-02-23T00:33:48.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 8 2024-02-23T00:33:48.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea8027b80 osd=3 3 2024-02-23T00:33:48.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 3 8 2024-02-23T00:33:48.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8027b80 osd=3 4 2024-02-23T00:33:48.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 10 client.26892.objecter _send_command 8 2024-02-23T00:33:48.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.364+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ea811cd20 con 0x7f3ea811fb80 2024-02-23T00:33:48.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3ea811cd20 con 0x7f3ea811fb80 2024-02-23T00:33:48.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:48.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea8027b80 osd=3 4 2024-02-23T00:33:48.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 3 8 2024-02-23T00:33:48.367 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:33:48.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 10 client.26892.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:48.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 9 2024-02-23T00:33:48.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6800/4175844757 conn(0x7f3ea811ce90 0x7f3ea81235e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea811cd20 osd=4 3 2024-02-23T00:33:48.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811cd20 osd=4 3 2024-02-23T00:33:48.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea811cd20 osd=4 3 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 9 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea811cd20 osd=4 3 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 4 9 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811cd20 osd=4 4 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 10 client.26892.objecter _send_command 9 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ea811c0e0 con 0x7f3ea811ce90 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.366+0000 7f3eac988700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6800/4175844757 conn(0x7f3ea811ce90 0x7f3ea81235e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.367+0000 7f3ea5ffb700 10 client.26892.objecter ms_handle_connect 0x7f3ea811ce90 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.367+0000 7f3eac988700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6800/4175844757 conn(0x7f3ea811ce90 0x7f3ea81235e0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.368+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3ea811c0e0 con 0x7f3ea811ce90 2024-02-23T00:33:48.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.368+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:33:48.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.368+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 9 = system:0 2024-02-23T00:33:48.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.368+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea811cd20 osd=4 4 2024-02-23T00:33:48.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.368+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 4 9 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 10 client.26892.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 10 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea811cd20 osd=4 3 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811cd20 osd=4 3 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea811cd20 osd=4 3 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 10 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea811cd20 osd=4 3 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 4 10 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811cd20 osd=4 4 2024-02-23T00:33:48.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 10 client.26892.objecter _send_command 10 2024-02-23T00:33:48.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.382+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ea811c0e0 con 0x7f3ea811ce90 2024-02-23T00:33:48.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3ea811c0e0 con 0x7f3ea811ce90 2024-02-23T00:33:48.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:48.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea811cd20 osd=4 4 2024-02-23T00:33:48.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 4 10 2024-02-23T00:33:48.387 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:33:48.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 10 client.26892.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:48.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 11 2024-02-23T00:33:48.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3ea8123fe0 0x7f3ea8124420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea811c0e0 osd=5 3 2024-02-23T00:33:48.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811c0e0 osd=5 3 2024-02-23T00:33:48.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea811c0e0 osd=5 3 2024-02-23T00:33:48.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 11 2024-02-23T00:33:48.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea811c0e0 osd=5 3 2024-02-23T00:33:48.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 5 11 2024-02-23T00:33:48.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811c0e0 osd=5 4 2024-02-23T00:33:48.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 10 client.26892.objecter _send_command 11 2024-02-23T00:33:48.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.384+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ea81249f0 con 0x7f3ea8123fe0 2024-02-23T00:33:48.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.385+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3ea8123fe0 0x7f3ea8124420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.385+0000 7f3ea5ffb700 10 client.26892.objecter ms_handle_connect 0x7f3ea8123fe0 2024-02-23T00:33:48.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.385+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3ea8123fe0 0x7f3ea8124420 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.386+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3ea81249f0 con 0x7f3ea8123fe0 2024-02-23T00:33:48.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.386+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:33:48.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.386+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 11 = system:0 2024-02-23T00:33:48.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.386+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea811c0e0 osd=5 4 2024-02-23T00:33:48.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.386+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 5 11 2024-02-23T00:33:48.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 10 client.26892.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:48.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 12 2024-02-23T00:33:48.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea811c0e0 osd=5 3 2024-02-23T00:33:48.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811c0e0 osd=5 3 2024-02-23T00:33:48.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea811c0e0 osd=5 3 2024-02-23T00:33:48.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 12 2024-02-23T00:33:48.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea811c0e0 osd=5 3 2024-02-23T00:33:48.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 5 12 2024-02-23T00:33:48.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811c0e0 osd=5 4 2024-02-23T00:33:48.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 10 client.26892.objecter _send_command 12 2024-02-23T00:33:48.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.400+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ea81249f0 con 0x7f3ea8123fe0 2024-02-23T00:33:48.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3ea81249f0 con 0x7f3ea8123fe0 2024-02-23T00:33:48.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:48.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea811c0e0 osd=5 4 2024-02-23T00:33:48.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 5 12 2024-02-23T00:33:48.404 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:33:48.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 10 client.26892.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 13 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3ea8129260 0x7f3ea81296c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea81249f0 osd=6 3 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea81249f0 osd=6 3 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea81249f0 osd=6 3 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 13 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea81249f0 osd=6 3 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 6 13 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea81249f0 osd=6 4 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 10 client.26892.objecter _send_command 13 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.402+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ea812b310 con 0x7f3ea8129260 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.403+0000 7f3ead189700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3ea8129260 0x7f3ea81296c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.403+0000 7f3ea5ffb700 10 client.26892.objecter ms_handle_connect 0x7f3ea8129260 2024-02-23T00:33:48.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.403+0000 7f3ead189700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3ea8129260 0x7f3ea81296c0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.404+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3ea812b310 con 0x7f3ea8129260 2024-02-23T00:33:48.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.404+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:33:48.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.404+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 13 = system:0 2024-02-23T00:33:48.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.404+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea81249f0 osd=6 4 2024-02-23T00:33:48.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.404+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 6 13 2024-02-23T00:33:48.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 10 client.26892.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:48.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 14 2024-02-23T00:33:48.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea81249f0 osd=6 3 2024-02-23T00:33:48.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea81249f0 osd=6 3 2024-02-23T00:33:48.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea81249f0 osd=6 3 2024-02-23T00:33:48.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 14 2024-02-23T00:33:48.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea81249f0 osd=6 3 2024-02-23T00:33:48.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 6 14 2024-02-23T00:33:48.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea81249f0 osd=6 4 2024-02-23T00:33:48.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 10 client.26892.objecter _send_command 14 2024-02-23T00:33:48.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.418+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ea812b310 con 0x7f3ea8129260 2024-02-23T00:33:48.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3ea812b310 con 0x7f3ea8129260 2024-02-23T00:33:48.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:48.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea81249f0 osd=6 4 2024-02-23T00:33:48.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 6 14 2024-02-23T00:33:48.421 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:33:48.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 10 client.26892.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:48.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 15 2024-02-23T00:33:48.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6812/373787000 conn(0x7f3ea812da90 0x7f3ea812def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:48.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea812b310 osd=7 3 2024-02-23T00:33:48.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea812b310 osd=7 3 2024-02-23T00:33:48.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea812b310 osd=7 3 2024-02-23T00:33:48.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 15 2024-02-23T00:33:48.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea812b310 osd=7 3 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 7 15 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea812b310 osd=7 4 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 10 client.26892.objecter _send_command 15 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ea812e530 con 0x7f3ea812da90 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eac988700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6812/373787000 conn(0x7f3ea812da90 0x7f3ea812def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3ea5ffb700 10 client.26892.objecter ms_handle_connect 0x7f3ea812da90 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.420+0000 7f3eac988700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6812/373787000 conn(0x7f3ea812da90 0x7f3ea812def0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.421+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3ea812e530 con 0x7f3ea812da90 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.421+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.421+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 15 = system:0 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.421+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea812b310 osd=7 4 2024-02-23T00:33:48.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.421+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 7 15 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 10 client.26892.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign -1 16 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea812b310 osd=7 3 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea812b310 osd=7 3 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 20 client.26892.objecter _get_session s=0x7f3ea812b310 osd=7 3 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_remove -1 16 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 20 client.26892.objecter get_session s=0x7f3ea812b310 osd=7 3 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 15 client.26892.objecter _session_command_op_assign 7 16 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea812b310 osd=7 4 2024-02-23T00:33:48.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 10 client.26892.objecter _send_command 16 2024-02-23T00:33:48.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.435+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3ea812e530 con 0x7f3ea812da90 2024-02-23T00:33:48.438 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3ea5ffb700 1 -- 172.21.15.87:0/3224185643 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3ea812e530 con 0x7f3ea812da90 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3ea5ffb700 10 client.26892.objecter ms_dispatch 0x7f3ea8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3ea5ffb700 10 client.26892.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3ea5ffb700 20 client.26892.objecter put_session s=0x7f3ea812b310 osd=7 4 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3ea5ffb700 15 client.26892.objecter _session_command_op_remove 7 16 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 librados: watch_flush enter 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 librados: watch_flush exit 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 client.26892.objecter close_session for osd.0 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6801/2153191863 conn(0x7f3ea80ffed0 msgr2=0x7f3ea8100330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6801/2153191863 conn(0x7f3ea80ffed0 0x7f3ea8100330 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea805ee80 osd=0 1 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 client.26892.objecter close_session for osd.1 2024-02-23T00:33:48.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6805/685354171 conn(0x7f3ea80598f0 msgr2=0x7f3ea8059d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6805/685354171 conn(0x7f3ea80598f0 0x7f3ea8059d10 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8028710 osd=1 1 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 client.26892.objecter close_session for osd.2 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6809/3605394228 conn(0x7f3ea8028270 msgr2=0x7f3ea811c420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6809/3605394228 conn(0x7f3ea8028270 0x7f3ea811c420 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8059770 osd=2 1 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 client.26892.objecter close_session for osd.3 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6813/128059790 conn(0x7f3ea811fb80 msgr2=0x7f3ea8121f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6813/128059790 conn(0x7f3ea811fb80 0x7f3ea8121f70 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3ead189700 1 -- 172.21.15.87:0/3224185643 reap_dead start 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea8027b80 osd=3 1 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 client.26892.objecter close_session for osd.4 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6800/4175844757 conn(0x7f3ea811ce90 msgr2=0x7f3ea81235e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6800/4175844757 conn(0x7f3ea811ce90 0x7f3ea81235e0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3ead189700 1 -- 172.21.15.87:0/3224185643 reap_dead start 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811cd20 osd=4 1 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 client.26892.objecter close_session for osd.5 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3ea8123fe0 msgr2=0x7f3ea8124420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3ea8123fe0 0x7f3ea8124420 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea811c0e0 osd=5 1 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 client.26892.objecter close_session for osd.6 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3ea8129260 msgr2=0x7f3ea81296c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3ea8129260 0x7f3ea81296c0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea81249f0 osd=6 1 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 client.26892.objecter close_session for osd.7 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6812/373787000 conn(0x7f3ea812da90 msgr2=0x7f3ea812def0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.171:6812/373787000 conn(0x7f3ea812da90 0x7f3ea812def0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 20 client.26892.objecter put_session s=0x7f3ea812b310 osd=7 1 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 20 client.26892.objecter shutdown clearing up homeless session... 2024-02-23T00:33:48.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 10 client.26892.objecter successfully canceled tick 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.437+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 msgr2=0x7f3e98065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.438+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 0x7f3e98065150 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 rx=0x7f3e9c004de0 tx=0x7f3e9c00b040).stop 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.438+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fcbb0 msgr2=0x7f3ea80f8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.438+0000 7f3eaebec700 1 --2- 172.21.15.87:0/3224185643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fcbb0 0x7f3ea80f8ea0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 rx=0x7f3e940043a0 tx=0x7f3e94009cb0).stop 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.438+0000 7f3ead189700 1 -- 172.21.15.87:0/3224185643 reap_dead start 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.438+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 shutdown_connections 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.438+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 >> 172.21.15.87:0/3224185643 conn(0x7f3ea80f2820 msgr2=0x7f3ea8060df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.438+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 shutdown_connections 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.438+0000 7f3eaebec700 1 -- 172.21.15.87:0/3224185643 wait complete. 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:48.438+0000 7f3eaebec700 1 librados: shutdown 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:33:48.463 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:48.492 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:48.513 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:48.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:48.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.625734+0000 osd.0 (osd.0) 147 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:46.629198+0000 osd.0 (osd.0) 148 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.361803+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.361972+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.389557+0000 osd.6 (osd.6) 105 : cluster [DBG] 63.c scrub starts 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.391816+0000 osd.6 (osd.6) 106 : cluster [DBG] 63.c scrub ok 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.432042+0000 osd.2 (osd.2) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.432288+0000 osd.2 (osd.2) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.607151+0000 osd.0 (osd.0) 151 : cluster [DBG] 6.9 deep-scrub starts 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.610208+0000 osd.0 (osd.0) 152 : cluster [DBG] 6.9 deep-scrub ok 2024-02-23T00:33:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:48.279935+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.625734+0000 osd.0 (osd.0) 147 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:46.629198+0000 osd.0 (osd.0) 148 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.361803+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.361972+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.389557+0000 osd.6 (osd.6) 105 : cluster [DBG] 63.c scrub starts 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.391816+0000 osd.6 (osd.6) 106 : cluster [DBG] 63.c scrub ok 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.432042+0000 osd.2 (osd.2) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.432288+0000 osd.2 (osd.2) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.607151+0000 osd.0 (osd.0) 151 : cluster [DBG] 6.9 deep-scrub starts 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.610208+0000 osd.0 (osd.0) 152 : cluster [DBG] 6.9 deep-scrub ok 2024-02-23T00:33:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:48.279935+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-02-23T00:33:48.617 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.625734+0000 osd.0 (osd.0) 147 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:46.629198+0000 osd.0 (osd.0) 148 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.361803+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.361972+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.389557+0000 osd.6 (osd.6) 105 : cluster [DBG] 63.c scrub starts 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.391816+0000 osd.6 (osd.6) 106 : cluster [DBG] 63.c scrub ok 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.432042+0000 osd.2 (osd.2) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.432288+0000 osd.2 (osd.2) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.607151+0000 osd.0 (osd.0) 151 : cluster [DBG] 6.9 deep-scrub starts 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.610208+0000 osd.0 (osd.0) 152 : cluster [DBG] 6.9 deep-scrub ok 2024-02-23T00:33:48.644 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:48.279935+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-02-23T00:33:48.645 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:48.665 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:48.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:48.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:48.797 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:47.607735+0000 mgr.y (mgr.24437) 289 : cluster [DBG] pgmap v397: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:48.280808+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:48.281208+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:48.286499+0000 mon.a (mon.0) 1962 : audit [DBG] from='client.? 172.21.15.87:0/3224185643' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:48.307936+0000 mon.b (mon.1) 242 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:48.308256+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:48.355882+0000 osd.6 (osd.6) 107 : cluster [DBG] 63.14 scrub starts 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:48.357366+0000 osd.6 (osd.6) 108 : cluster [DBG] 63.14 scrub ok 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:48.778573+0000 osd.4 (osd.4) 133 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:48.779468+0000 osd.4 (osd.4) 134 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:49.037072+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi087-82019-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi087-82019-49"}]': finished 2024-02-23T00:33:49.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:49.037149+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]': finished 2024-02-23T00:33:49.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:49.037208+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:49.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.037258+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-02-23T00:33:49.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:49.038819+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:49.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:49.039006+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:49.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:49.039698+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:49.040064+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.551 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:47.607735+0000 mgr.y (mgr.24437) 289 : cluster [DBG] pgmap v397: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:48.280808+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:48.281208+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:48.286499+0000 mon.a (mon.0) 1962 : audit [DBG] from='client.? 172.21.15.87:0/3224185643' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:48.307936+0000 mon.b (mon.1) 242 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:48.308256+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:48.355882+0000 osd.6 (osd.6) 107 : cluster [DBG] 63.14 scrub starts 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:48.357366+0000 osd.6 (osd.6) 108 : cluster [DBG] 63.14 scrub ok 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:48.778573+0000 osd.4 (osd.4) 133 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:48.779468+0000 osd.4 (osd.4) 134 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:49.037072+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi087-82019-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi087-82019-49"}]': finished 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:49.037149+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]': finished 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:49.037208+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:49.552 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.037258+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-02-23T00:33:49.553 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:49.038819+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:49.553 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:49.039006+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:49.553 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:49.039698+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.553 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:49.040064+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:49.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:49.655 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:49.681 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:47.607735+0000 mgr.y (mgr.24437) 289 : cluster [DBG] pgmap v397: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:33:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:48.280808+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:48.281208+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:48.286499+0000 mon.a (mon.0) 1962 : audit [DBG] from='client.? 172.21.15.87:0/3224185643' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:48.307936+0000 mon.b (mon.1) 242 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:48.308256+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:48.355882+0000 osd.6 (osd.6) 107 : cluster [DBG] 63.14 scrub starts 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:48.357366+0000 osd.6 (osd.6) 108 : cluster [DBG] 63.14 scrub ok 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:48.778573+0000 osd.4 (osd.4) 133 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:48.779468+0000 osd.4 (osd.4) 134 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:49.037072+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi087-82019-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi087-82019-49"}]': finished 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:49.037149+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi087-82011-40"}]': finished 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:49.037208+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.037258+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:49.038819+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:49.039006+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33"}]: dispatch 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:49.039698+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.87:0/1299735202' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:49.040064+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]: dispatch 2024-02-23T00:33:49.780 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:49.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:49.883 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:49.916 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:49.937 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:49.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:50.041 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:50.077 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:50.130 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:50.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:50.243 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:50.270 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:50.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:50.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:50.422 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.398+0000 7fb2e95a0700 1 -- 172.21.15.87:0/2287343595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 msgr2=0x7fb2e4103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.398+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/2287343595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 0x7fb2e4103740 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 rx=0x7fb2dc009c30 tx=0x7fb2dc004c50).stop 2024-02-23T00:33:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.398+0000 7fb2e95a0700 1 -- 172.21.15.87:0/2287343595 shutdown_connections 2024-02-23T00:33:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.398+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/2287343595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 0x7fb2e4103740 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.398+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/2287343595 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2e40fc920 0x7fb2e40fcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.398+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/2287343595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e40fd280 0x7fb2e40f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:50.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.398+0000 7fb2e95a0700 1 -- 172.21.15.87:0/2287343595 >> 172.21.15.87:0/2287343595 conn(0x7fb2e40f2500 msgr2=0x7fb2e40f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.398+0000 7fb2e95a0700 1 -- 172.21.15.87:0/2287343595 shutdown_connections 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.398+0000 7fb2e95a0700 1 -- 172.21.15.87:0/2287343595 wait complete. 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 librados: starting msgr at 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 librados: starting objecter 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 Processor -- start 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 -- start start 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 librados: setting wanted keys 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 librados: calling monclient init 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 0x7fb2e410dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2e40fc920 0x7fb2e410e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e40fd280 0x7fb2e4107ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e41083c0 con 0x7fb2e40f7120 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e4108530 con 0x7fb2e40fd280 2024-02-23T00:33:50.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e95a0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e41086f0 con 0x7fb2e40fc920 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e37fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2e40fc920 0x7fb2e410e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e3fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 0x7fb2e410dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e3fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 0x7fb2e410dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56598/0 (socket says 172.21.15.87:56598) 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e37fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2e40fc920 0x7fb2e410e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:45750/0 (socket says 172.21.15.87:45750) 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e3fff700 1 -- 172.21.15.87:0/558625001 learned_addr learned my addr 172.21.15.87:0/558625001 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.399+0000 7fb2e8d9f700 1 --2- 172.21.15.87:0/558625001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e40fd280 0x7fb2e4107ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e37fe700 1 -- 172.21.15.87:0/558625001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e40fd280 msgr2=0x7fb2e4107ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e37fe700 1 --2- 172.21.15.87:0/558625001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e40fd280 0x7fb2e4107ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e37fe700 1 -- 172.21.15.87:0/558625001 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 msgr2=0x7fb2e410dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e37fe700 1 --2- 172.21.15.87:0/558625001 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 0x7fb2e410dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e37fe700 1 -- 172.21.15.87:0/558625001 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d4005040 con 0x7fb2e40fc920 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e3fff700 1 --2- 172.21.15.87:0/558625001 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 0x7fb2e410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e37fe700 1 --2- 172.21.15.87:0/558625001 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2e40fc920 0x7fb2e410e220 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7fb2d8002750 tx=0x7fb2d8009e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:50.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e17fa700 10 client.?.objecter ms_handle_connect 0x7fb2e40fc920 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e17fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e17fa700 1 -- 172.21.15.87:0/558625001 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8007b40 con 0x7fb2e40fc920 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc0097c0 con 0x7fb2e40fc920 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e17fa700 1 -- 172.21.15.87:0/558625001 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2d800f040 con 0x7fb2e40fc920 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e17fa700 1 -- 172.21.15.87:0/558625001 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8014690 con 0x7fb2e40fc920 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e95a0700 10 client.37361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e4114b60 con 0x7fb2e40fc920 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.400+0000 7fb2e95a0700 1 librados: init done 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.401+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e4061ec0 con 0x7fb2e40fc920 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.401+0000 7fb2e17fa700 1 -- 172.21.15.87:0/558625001 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb2d8007ca0 con 0x7fb2e40fc920 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.402+0000 7fb2e17fa700 1 --2- 172.21.15.87:0/558625001 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2cc062c90 0x7fb2cc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:50.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.402+0000 7fb2e17fa700 1 -- 172.21.15.87:0/558625001 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(279..279 src has 38..279) v4 ==== 8707+0+0 (secure 0 0 0) 0x7fb2d807cff0 con 0x7fb2e40fc920 2024-02-23T00:33:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.402+0000 7fb2e3fff700 1 --2- 172.21.15.87:0/558625001 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2cc062c90 0x7fb2cc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.402+0000 7fb2e17fa700 10 client.37361.objecter ms_dispatch 0x7fb2e4001550 osd_map(279..279 src has 38..279) v4 2024-02-23T00:33:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.402+0000 7fb2e17fa700 3 client.37361.objecter handle_osd_map got epochs [279,279] > 0 2024-02-23T00:33:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.402+0000 7fb2e17fa700 3 client.37361.objecter handle_osd_map decoding full epoch 279 2024-02-23T00:33:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.402+0000 7fb2e17fa700 20 client.37361.objecter dump_active .. 0 homeless 2024-02-23T00:33:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.402+0000 7fb2e17fa700 10 client.37361.objecter ms_handle_connect 0x7fb2cc062c90 2024-02-23T00:33:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.402+0000 7fb2e3fff700 1 --2- 172.21.15.87:0/558625001 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2cc062c90 0x7fb2cc065150 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7fb2d4004d40 tx=0x7fb2d400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:50.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.406+0000 7fb2e17fa700 1 -- 172.21.15.87:0/558625001 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb2d80502b0 con 0x7fb2e40fc920 2024-02-23T00:33:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.217756+0000 osd.5 (osd.5) 133 : cluster [DBG] 63.1e scrub starts 2024-02-23T00:33:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.219601+0000 osd.5 (osd.5) 134 : cluster [DBG] 63.1e scrub ok 2024-02-23T00:33:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.567858+0000 osd.3 (osd.3) 133 : cluster [DBG] 4.19 deep-scrub starts 2024-02-23T00:33:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.575911+0000 osd.3 (osd.3) 134 : cluster [DBG] 4.19 deep-scrub ok 2024-02-23T00:33:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.585351+0000 osd.0 (osd.0) 153 : cluster [DBG] 4.b scrub starts 2024-02-23T00:33:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.588809+0000 osd.0 (osd.0) 154 : cluster [DBG] 4.b scrub ok 2024-02-23T00:33:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.753877+0000 osd.4 (osd.4) 135 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:33:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.754824+0000 osd.4 (osd.4) 136 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:33:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:50.034476+0000 mon.a (mon.0) 1970 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:50.039243+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33"}]': finished 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:50.039376+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]': finished 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:50.039479+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:50.065798+0000 mon.a (mon.0) 1974 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:50.066919+0000 mon.a (mon.0) 1975 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:50.067247+0000 mon.a (mon.0) 1976 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi087-82011-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.217756+0000 osd.5 (osd.5) 133 : cluster [DBG] 63.1e scrub starts 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.219601+0000 osd.5 (osd.5) 134 : cluster [DBG] 63.1e scrub ok 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.567858+0000 osd.3 (osd.3) 133 : cluster [DBG] 4.19 deep-scrub starts 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.575911+0000 osd.3 (osd.3) 134 : cluster [DBG] 4.19 deep-scrub ok 2024-02-23T00:33:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.585351+0000 osd.0 (osd.0) 153 : cluster [DBG] 4.b scrub starts 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.588809+0000 osd.0 (osd.0) 154 : cluster [DBG] 4.b scrub ok 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.753877+0000 osd.4 (osd.4) 135 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.754824+0000 osd.4 (osd.4) 136 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:50.034476+0000 mon.a (mon.0) 1970 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:50.039243+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33"}]': finished 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:50.039376+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]': finished 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:50.039479+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:50.065798+0000 mon.a (mon.0) 1974 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:50.066919+0000 mon.a (mon.0) 1975 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:50.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:50.067247+0000 mon.a (mon.0) 1976 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi087-82011-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:50.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:50.584+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fb2e4105e30 con 0x7fb2e40fc920 2024-02-23T00:33:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.217756+0000 osd.5 (osd.5) 133 : cluster [DBG] 63.1e scrub starts 2024-02-23T00:33:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.219601+0000 osd.5 (osd.5) 134 : cluster [DBG] 63.1e scrub ok 2024-02-23T00:33:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.567858+0000 osd.3 (osd.3) 133 : cluster [DBG] 4.19 deep-scrub starts 2024-02-23T00:33:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.575911+0000 osd.3 (osd.3) 134 : cluster [DBG] 4.19 deep-scrub ok 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.585351+0000 osd.0 (osd.0) 153 : cluster [DBG] 4.b scrub starts 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.588809+0000 osd.0 (osd.0) 154 : cluster [DBG] 4.b scrub ok 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.753877+0000 osd.4 (osd.4) 135 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.754824+0000 osd.4 (osd.4) 136 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:50.034476+0000 mon.a (mon.0) 1970 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:50.039243+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-33"}]': finished 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:50.039376+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi087-82011-40"}]': finished 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:50.039479+0000 mon.a (mon.0) 1973 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:50.065798+0000 mon.a (mon.0) 1974 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:50.066919+0000 mon.a (mon.0) 1975 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:50.067247+0000 mon.a (mon.0) 1976 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi087-82011-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:51.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.045+0000 7fb2e17fa700 1 -- 172.21.15.87:0/558625001 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v280) v1 ==== 87+0+0 (secure 0 0 0) 0x7fb2d804fea0 con 0x7fb2e40fc920 2024-02-23T00:33:51.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.049+0000 7fb2e95a0700 10 librados: watch_flush enter 2024-02-23T00:33:51.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.049+0000 7fb2e95a0700 10 librados: watch_flush exit 2024-02-23T00:33:51.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.049+0000 7fb2e95a0700 20 client.37361.objecter shutdown clearing up homeless session... 2024-02-23T00:33:51.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.049+0000 7fb2e95a0700 10 client.37361.objecter successfully canceled tick 2024-02-23T00:33:51.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.049+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2cc062c90 msgr2=0x7fb2cc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:51.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.049+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/558625001 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2cc062c90 0x7fb2cc065150 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 rx=0x7fb2d4004d40 tx=0x7fb2d400b040).stop 2024-02-23T00:33:51.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.049+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2e40fc920 msgr2=0x7fb2e410e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:51.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.049+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/558625001 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2e40fc920 0x7fb2e410e220 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7fb2d8002750 tx=0x7fb2d8009e40).stop 2024-02-23T00:33:51.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.050+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 shutdown_connections 2024-02-23T00:33:51.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.050+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/558625001 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2cc062c90 0x7fb2cc065150 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:51.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.050+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/558625001 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2e40fd280 0x7fb2e4107ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:51.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.050+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/558625001 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2e40f7120 0x7fb2e410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:51.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.050+0000 7fb2e95a0700 1 --2- 172.21.15.87:0/558625001 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2e40fc920 0x7fb2e410e220 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:51.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.050+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 >> 172.21.15.87:0/558625001 conn(0x7fb2e40f2500 msgr2=0x7fb2e40f4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:51.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.050+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 shutdown_connections 2024-02-23T00:33:51.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.050+0000 7fb2e95a0700 1 -- 172.21.15.87:0/558625001 wait complete. 2024-02-23T00:33:51.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:51.050+0000 7fb2e95a0700 1 librados: shutdown 2024-02-23T00:33:51.055 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:33:51.172 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:33:51.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:51.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:33:51.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:33:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:33:51.277 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:51.310 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:51.332 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:49.608363+0000 mgr.y (mgr.24437) 290 : cluster [DBG] pgmap v400: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:50.585915+0000 mon.c (mon.2) 355 : audit [INF] from='client.? 172.21.15.87:0/558625001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:50.586205+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:50.597987+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:50.601479+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:51.035957+0000 mon.a (mon.0) 1978 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:51.045498+0000 mon.a (mon.0) 1979 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi087-82011-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:51.045601+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:51.045658+0000 mon.a (mon.0) 1981 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:51.295015+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi087-82011-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:51.295592+0000 mon.a (mon.0) 1983 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:49.608363+0000 mgr.y (mgr.24437) 290 : cluster [DBG] pgmap v400: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:50.585915+0000 mon.c (mon.2) 355 : audit [INF] from='client.? 172.21.15.87:0/558625001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:50.586205+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:50.597987+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:50.601479+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:51.035957+0000 mon.a (mon.0) 1978 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:51.045498+0000 mon.a (mon.0) 1979 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi087-82011-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:51.045601+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:51.045658+0000 mon.a (mon.0) 1981 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:51.295015+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi087-82011-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:51.295592+0000 mon.a (mon.0) 1983 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:51.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:51.436 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:51.465 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:51.613 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:51.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:49.608363+0000 mgr.y (mgr.24437) 290 : cluster [DBG] pgmap v400: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:33:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:50.585915+0000 mon.c (mon.2) 355 : audit [INF] from='client.? 172.21.15.87:0/558625001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:50.586205+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:33:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:50.597987+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:33:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:50.601479+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:33:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:51.035957+0000 mon.a (mon.0) 1978 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:33:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:51.045498+0000 mon.a (mon.0) 1979 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi087-82011-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:51.045601+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:33:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:51.045658+0000 mon.a (mon.0) 1981 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-02-23T00:33:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:51.295015+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi087-82011-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:51.295592+0000 mon.a (mon.0) 1983 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:51.718 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:51.750 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:51.771 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:51.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:51.875 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:51.905 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:51.925 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:51.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:52.029 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:52.061 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:52.079 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:52.081 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:52.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:52.185 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:52.211 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:52.249 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:52.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:52.305 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:52] "GET /metrics HTTP/1.1" 200 188967 "" "Prometheus/2.33.4" 2024-02-23T00:33:52.356 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:52.382 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:50.775542+0000 mgr.y (mgr.24437) 291 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:52.382 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:51.122126+0000 osd.1 (osd.1) 137 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:33:52.382 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:52.035564+0000 mon.b (mon.1) 244 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:52.382 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:52.268605+0000 mon.a (mon.0) 1984 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi087-82019-49"}]': finished 2024-02-23T00:33:52.382 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:52.268699+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-02-23T00:33:52.382 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:52.269284+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:52.383 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:52.269860+0000 mon.a (mon.0) 1987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:52.384 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:52.403 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:52.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:52.506 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:52.534 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:52.553 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:52.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:50.775542+0000 mgr.y (mgr.24437) 291 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:51.122126+0000 osd.1 (osd.1) 137 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:33:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:52.035564+0000 mon.b (mon.1) 244 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:52.268605+0000 mon.a (mon.0) 1984 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi087-82019-49"}]': finished 2024-02-23T00:33:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:52.268699+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-02-23T00:33:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:52.269284+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:52.269860+0000 mon.a (mon.0) 1987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:50.775542+0000 mgr.y (mgr.24437) 291 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:33:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:51.122126+0000 osd.1 (osd.1) 137 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:33:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:52.035564+0000 mon.b (mon.1) 244 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:52.268605+0000 mon.a (mon.0) 1984 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi087-82019-49"}]': finished 2024-02-23T00:33:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:52.268699+0000 mon.a (mon.0) 1985 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-02-23T00:33:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:52.269284+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi087-82019-49"}]: dispatch 2024-02-23T00:33:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:52.269860+0000 mon.a (mon.0) 1987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:52.657 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 -- 172.21.15.87:0/3723351804 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d800fcbb0 msgr2=0x7f4d800fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 --2- 172.21.15.87:0/3723351804 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d800fcbb0 0x7f4d800fcf90 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 rx=0x7f4d740047b0 tx=0x7f4d74009cb0).stop 2024-02-23T00:33:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 -- 172.21.15.87:0/3723351804 shutdown_connections 2024-02-23T00:33:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 --2- 172.21.15.87:0/3723351804 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d800f7400 0x7f4d80103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 --2- 172.21.15.87:0/3723351804 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d800fcbb0 0x7f4d800fcf90 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 --2- 172.21.15.87:0/3723351804 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d800fd510 0x7f4d800f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 -- 172.21.15.87:0/3723351804 >> 172.21.15.87:0/3723351804 conn(0x7f4d800f2820 msgr2=0x7f4d800f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 -- 172.21.15.87:0/3723351804 shutdown_connections 2024-02-23T00:33:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 -- 172.21.15.87:0/3723351804 wait complete. 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 librados: starting msgr at 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 librados: starting objecter 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.631+0000 7f4d86c28700 1 Processor -- start 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 -- start start 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 librados: setting wanted keys 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 librados: calling monclient init 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d800f7400 0x7f4d8010e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d800fcbb0 0x7f4d8010e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d800fd510 0x7f4d801083d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d801088c0 con 0x7f4d800fcbb0 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d80108a30 con 0x7f4d800f7400 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d80108bf0 con 0x7f4d800fd510 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d851c5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d800fd510 0x7f4d801083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d7ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d800fcbb0 0x7f4d8010e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d851c5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d800fd510 0x7f4d801083d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33882/0 (socket says 172.21.15.87:33882) 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d851c5700 1 -- 172.21.15.87:0/2002302697 learned_addr learned my addr 172.21.15.87:0/2002302697 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d849c4700 1 --2- 172.21.15.87:0/2002302697 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d800f7400 0x7f4d8010e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d851c5700 1 -- 172.21.15.87:0/2002302697 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d800f7400 msgr2=0x7f4d8010e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d851c5700 1 --2- 172.21.15.87:0/2002302697 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d800f7400 0x7f4d8010e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d851c5700 1 -- 172.21.15.87:0/2002302697 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d800fcbb0 msgr2=0x7f4d8010e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d851c5700 1 --2- 172.21.15.87:0/2002302697 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d800fcbb0 0x7f4d8010e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d851c5700 1 -- 172.21.15.87:0/2002302697 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780097c0 con 0x7f4d800fd510 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d851c5700 1 --2- 172.21.15.87:0/2002302697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d800fd510 0x7f4d801083d0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7f4d78001df0 tx=0x7f4d78004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d7dffb700 10 client.?.objecter ms_handle_connect 0x7f4d800fd510 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d7dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d7dffb700 1 -- 172.21.15.87:0/2002302697 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78009a10 con 0x7f4d800fd510 2024-02-23T00:33:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d7dffb700 1 -- 172.21.15.87:0/2002302697 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d780090d0 con 0x7f4d800fd510 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d74005040 con 0x7f4d800fd510 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.632+0000 7f4d7dffb700 1 -- 172.21.15.87:0/2002302697 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78007d30 con 0x7f4d800fd510 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.633+0000 7f4d86c28700 10 client.37367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.633+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d801091e0 con 0x7f4d800fd510 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.633+0000 7f4d86c28700 1 librados: init done 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.634+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8005fa60 con 0x7f4d800fd510 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.634+0000 7f4d7dffb700 1 -- 172.21.15.87:0/2002302697 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4d7800fb30 con 0x7f4d800fd510 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.634+0000 7f4d7dffb700 1 --2- 172.21.15.87:0/2002302697 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4d70062c90 0x7f4d70065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.634+0000 7f4d7dffb700 1 -- 172.21.15.87:0/2002302697 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(281..281 src has 38..281) v4 ==== 8384+0+0 (secure 0 0 0) 0x7f4d7807dba0 con 0x7f4d800fd510 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.634+0000 7f4d7dffb700 10 client.37367.objecter ms_dispatch 0x7f4d80001550 osd_map(281..281 src has 38..281) v4 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.634+0000 7f4d7dffb700 3 client.37367.objecter handle_osd_map got epochs [281,281] > 0 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.634+0000 7f4d7dffb700 3 client.37367.objecter handle_osd_map decoding full epoch 281 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.634+0000 7f4d849c4700 1 --2- 172.21.15.87:0/2002302697 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4d70062c90 0x7f4d70065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.635+0000 7f4d7dffb700 20 client.37367.objecter dump_active .. 0 homeless 2024-02-23T00:33:52.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.635+0000 7f4d7dffb700 10 client.37367.objecter ms_handle_connect 0x7f4d70062c90 2024-02-23T00:33:52.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.635+0000 7f4d849c4700 1 --2- 172.21.15.87:0/2002302697 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4d70062c90 0x7f4d70065150 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7f4d74005010 tx=0x7f4d74009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:52.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.639+0000 7f4d7dffb700 1 -- 172.21.15.87:0/2002302697 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4d78050f30 con 0x7f4d800fd510 2024-02-23T00:33:52.684 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:52.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.811+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d80106150 con 0x7f4d70062c90 2024-02-23T00:33:52.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.827+0000 7f4d7dffb700 1 -- 172.21.15.87:0/2002302697 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+599314 (secure 0 0 0) 0x7f4d80106150 con 0x7f4d70062c90 2024-02-23T00:33:52.837 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:33:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 10 librados: watch_flush enter 2024-02-23T00:33:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 10 librados: watch_flush exit 2024-02-23T00:33:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 20 client.37367.objecter shutdown clearing up homeless session... 2024-02-23T00:33:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 10 client.37367.objecter successfully canceled tick 2024-02-23T00:33:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4d70062c90 msgr2=0x7f4d70065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 --2- 172.21.15.87:0/2002302697 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4d70062c90 0x7f4d70065150 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 rx=0x7f4d74005010 tx=0x7f4d74009cb0).stop 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d800fd510 msgr2=0x7f4d801083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 --2- 172.21.15.87:0/2002302697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d800fd510 0x7f4d801083d0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7f4d78001df0 tx=0x7f4d78004d10).stop 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 shutdown_connections 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 --2- 172.21.15.87:0/2002302697 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4d70062c90 0x7f4d70065150 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 --2- 172.21.15.87:0/2002302697 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4d800fd510 0x7f4d801083d0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 --2- 172.21.15.87:0/2002302697 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4d800f7400 0x7f4d8010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 --2- 172.21.15.87:0/2002302697 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4d800fcbb0 0x7f4d8010e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 >> 172.21.15.87:0/2002302697 conn(0x7f4d800f2820 msgr2=0x7f4d800f4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 shutdown_connections 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 -- 172.21.15.87:0/2002302697 wait complete. 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:52.831+0000 7f4d86c28700 1 librados: shutdown 2024-02-23T00:33:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:33:53.191 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:33:53.536 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:53.538 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:51.608707+0000 mgr.y (mgr.24437) 292 : cluster [DBG] pgmap v403: 206 pgs: 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:53.038459+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi087-82011-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi087-82011-41"}]': finished 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:53.038571+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi087-82019-49"}]': finished 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:53.038656+0000 mon.a (mon.0) 1990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:53.038802+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:53.054145+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:53.057200+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:51.608707+0000 mgr.y (mgr.24437) 292 : cluster [DBG] pgmap v403: 206 pgs: 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:53.038459+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi087-82011-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi087-82011-41"}]': finished 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:53.038571+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi087-82019-49"}]': finished 2024-02-23T00:33:53.539 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:53.038656+0000 mon.a (mon.0) 1990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:53.540 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:53.038802+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-02-23T00:33:53.540 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:53.054145+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:53.540 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:53.057200+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:53.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:53.641 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:53.669 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:51.608707+0000 mgr.y (mgr.24437) 292 : cluster [DBG] pgmap v403: 206 pgs: 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:53.038459+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi087-82011-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi087-82011-41"}]': finished 2024-02-23T00:33:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:53.038571+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? 172.21.15.87:0/134760323' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi087-82019-49"}]': finished 2024-02-23T00:33:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:53.038656+0000 mon.a (mon.0) 1990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:53.038802+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-02-23T00:33:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:53.054145+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:53.057200+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:33:54.062 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:33:54.117 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:54.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:54.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:33:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:33:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:33:54.221 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:54.252 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:54.274 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:54.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:54.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:52.812680+0000 mgr.y (mgr.24437) 293 : audit [DBG] from='client.37367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:33:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:54.035392+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:54.035503+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-02-23T00:33:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:54.039411+0000 mon.b (mon.1) 246 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:54.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:54.041193+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:54.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:54.042607+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.87:0/2182945223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:54.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:54.043256+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:54.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:52.812680+0000 mgr.y (mgr.24437) 293 : audit [DBG] from='client.37367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:33:54.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:54.035392+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:54.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:54.035503+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-02-23T00:33:54.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:54.039411+0000 mon.b (mon.1) 246 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:54.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:54.041193+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:54.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:54.042607+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.87:0/2182945223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:54.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:54.043256+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:54.378 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:54.406 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:54.605 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:54.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:54.617 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:33:54.617 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:33:54.617 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:33:54.617 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:33:54.617 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.617 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15162.0:1 57.7s0 57:ea403f26:test-rados-api-smithi087-82139-13::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-23T00:33:54.617 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:01.527880+0000", 2024-02-23T00:33:54.617 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 173.085487035, 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.08591974, 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15162", 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3542609165", 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527880+0000", 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.618 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527880+0000", 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527878+0000", 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2410000000000001e-06 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527887+0000", 2024-02-23T00:33:54.619 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1595e-05 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527898+0000", 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0059999999999992e-06 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527906+0000", 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00104463 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.620 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.528951+0000", 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.6449999999999998e-06 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.528955+0000", 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0732655419999999 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.602220+0000", 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00039692599999999997 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.621 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.602617+0000", 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016094740000000001 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.604227+0000", 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0067402019999999998 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.610967+0000", 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027533340000000001 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.622 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.623 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.613720+0000", 2024-02-23T00:33:54.623 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9392000000000002e-05 2024-02-23T00:33:54.623 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.623 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.623 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.623 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.613800+0000", 2024-02-23T00:33:54.623 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.08591974 2024-02-23T00:33:54.623 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.623 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:424 7.2 e113/94 7:40054bab:test-rados-api-smithi087-82115-6::foo:9 v 113'6, mlcod=111'3)", 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.558545+0000", 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 171.054821484, 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041023802999999998, 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558545+0000", 2024-02-23T00:33:54.624 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558545+0000", 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558545+0000", 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4674e-05 2024-02-23T00:33:54.625 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.626 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.626 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.626 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558560+0000", 2024-02-23T00:33:54.626 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8779999999999997e-06 2024-02-23T00:33:54.626 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.626 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.626 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558565+0000", 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3399999999999997e-06 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558570+0000", 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00173953 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.560310+0000", 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6562000000000001e-05 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.627 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.560336+0000", 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.039195743999999998 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.599532+0000", 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7320000000000002e-05 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.599569+0000", 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041023802999999998 2024-02-23T00:33:54.628 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.5.0:324 56.1 e113/110 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head v 113'1, mlcod=0'0)", 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.675872+0000", 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 170.93749509899999, 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.029613130000000001, 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675872+0000", 2024-02-23T00:33:54.629 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675872+0000", 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675871+0000", 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2730000000000002e-06 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675879+0000", 2024-02-23T00:33:54.630 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4129999999999995e-06 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675889+0000", 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2979999999999997e-06 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675895+0000", 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00054747200000000002 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676442+0000", 2024-02-23T00:33:54.631 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5807999999999998e-05 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676458+0000", 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028979677999999998 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.705438+0000", 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7066e-05 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.632 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.705485+0000", 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.029613130000000001 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15147.0:14 56.1 e113/110 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head v 113'2, mlcod=0'0)", 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.676277+0000", 2024-02-23T00:33:54.633 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 170.937089609, 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.029227679999999999, 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676277+0000", 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676277+0000", 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.634 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676277+0000", 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2530000000000001e-06 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676283+0000", 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2520000000000005e-06 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.635 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.636 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.636 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676292+0000", 2024-02-23T00:33:54.636 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4049999999999999e-06 2024-02-23T00:33:54.636 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.636 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.636 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.636 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676298+0000", 2024-02-23T00:33:54.636 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000294176 2024-02-23T00:33:54.637 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.637 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.637 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.637 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676592+0000", 2024-02-23T00:33:54.637 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6429000000000001e-05 2024-02-23T00:33:54.637 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.638 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.638 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.638 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676608+0000", 2024-02-23T00:33:54.638 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028880586999999999 2024-02-23T00:33:54.638 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.638 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.705489+0000", 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6229999999999999e-05 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.705505+0000", 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.029227679999999999 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.640 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.640 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.640 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.640 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15189.0:3 61.1f 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected e114)", 2024-02-23T00:33:54.640 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:05.220101+0000", 2024-02-23T00:33:54.640 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 169.39326610500001, 2024-02-23T00:33:54.640 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036226108999999999, 2024-02-23T00:33:54.640 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.641 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.641 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.641 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15189", 2024-02-23T00:33:54.641 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3965564315", 2024-02-23T00:33:54.641 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 3 2024-02-23T00:33:54.641 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.641 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.641 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.641 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.642 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220101+0000", 2024-02-23T00:33:54.642 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.642 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.642 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.642 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.642 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220101+0000", 2024-02-23T00:33:54.642 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:54.642 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.643 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.643 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.643 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220099+0000", 2024-02-23T00:33:54.643 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.7810000000000005e-06 2024-02-23T00:33:54.643 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.643 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.643 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.643 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220107+0000", 2024-02-23T00:33:54.643 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.67e-06 2024-02-23T00:33:54.644 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.644 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.644 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.644 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220115+0000", 2024-02-23T00:33:54.644 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3560000000000004e-06 2024-02-23T00:33:54.644 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.644 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.644 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.644 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220123+0000", 2024-02-23T00:33:54.645 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013298133 2024-02-23T00:33:54.645 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.645 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.645 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.645 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.233421+0000", 2024-02-23T00:33:54.645 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9179999999999994e-06 2024-02-23T00:33:54.645 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.645 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.645 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:33:54.646 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.233429+0000", 2024-02-23T00:33:54.646 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0074901580000000002 2024-02-23T00:33:54.646 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.646 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.646 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.646 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.240919+0000", 2024-02-23T00:33:54.646 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8102000000000005e-05 2024-02-23T00:33:54.646 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.646 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.647 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.647 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241007+0000", 2024-02-23T00:33:54.647 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8525999999999998e-05 2024-02-23T00:33:54.647 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.647 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.647 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,1", 2024-02-23T00:33:54.647 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241106+0000", 2024-02-23T00:33:54.647 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.006526647 2024-02-23T00:33:54.647 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.648 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.648 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:54.648 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.247632+0000", 2024-02-23T00:33:54.648 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0074987430000000004 2024-02-23T00:33:54.648 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.648 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.648 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.649 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.255131+0000", 2024-02-23T00:33:54.649 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011306210000000001 2024-02-23T00:33:54.649 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.649 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.649 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.649 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.256262+0000", 2024-02-23T00:33:54.649 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7367999999999999e-05 2024-02-23T00:33:54.649 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.256299+0000", 2024-02-23T00:33:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7931000000000001e-05 2024-02-23T00:33:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.256327+0000", 2024-02-23T00:33:54.650 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036226108999999999 2024-02-23T00:33:54.651 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.651 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.651 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.651 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.651 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.651 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26218.0:2 62.15 e115/114 62:ad551702:::test:head v 115'1, mlcod=0'0)", 2024-02-23T00:33:54.651 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:06.291613+0000", 2024-02-23T00:33:54.651 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 168.32175380999999, 2024-02-23T00:33:54.651 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041167026000000002, 2024-02-23T00:33:54.652 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.652 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.652 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.652 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.652 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.652 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291613+0000", 2024-02-23T00:33:54.652 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.653 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.653 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.653 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.653 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291613+0000", 2024-02-23T00:33:54.653 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:54.653 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.653 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.653 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.653 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291613+0000", 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7430000000000008e-06 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291621+0000", 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7019999999999998e-06 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291627+0000", 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8669999999999996e-06 2024-02-23T00:33:54.654 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291634+0000", 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016782400000000001 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291802+0000", 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6673000000000001e-05 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291819+0000", 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040928475999999998 2024-02-23T00:33:54.655 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.332747+0000", 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3111000000000002e-05 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.332780+0000", 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041167026000000002 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.656 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15189.0:10 61.1f 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e118)", 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.171085+0000", 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 165.442281786, 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065815858000000005, 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15189", 2024-02-23T00:33:54.657 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3965564315", 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 10 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171085+0000", 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.658 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171085+0000", 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171084+0000", 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1025e-05 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.659 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171095+0000", 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0020000000000001e-05 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171105+0000", 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9689999999999999e-06 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171111+0000", 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003718687 2024-02-23T00:33:54.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.174829+0000", 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000100388 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.174930+0000", 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00018267399999999999 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,1", 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.175112+0000", 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.035207443999999997 2024-02-23T00:33:54.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.210320+0000", 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015002176000000001 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.225322+0000", 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011492073 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.236814+0000", 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5346999999999998e-05 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.236859+0000", 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1575000000000001e-05 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.236901+0000", 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065815858000000005 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.5.0:356 56.1 e118/110 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head v 118'13, mlcod=113'11)", 2024-02-23T00:33:54.663 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.929612+0000", 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 164.68375480899999, 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020950686999999999, 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.929612+0000", 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.664 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.929612+0000", 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999971 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.929609+0000", 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8964999999999999e-05 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.929628+0000", 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1889e-05 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.929650+0000", 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0193999999999999e-05 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.929660+0000", 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0065478419999999999 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.936208+0000", 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5116000000000001e-05 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.666 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.936233+0000", 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014273872999999999 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.950507+0000", 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5458000000000003e-05 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.950563+0000", 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020950686999999999 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.667 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.668 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15189.0:12 61.11 61:8c482978:test-rados-api-smithi087-82107-8::foo:head [write 0~128] snapc 6=[6] ondisk+write+known_if_redirected e119)", 2024-02-23T00:33:54.668 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.142541+0000", 2024-02-23T00:33:54.668 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 164.47082552500001, 2024-02-23T00:33:54.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.084312260999999999, 2024-02-23T00:33:54.668 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.668 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.669 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.669 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15189", 2024-02-23T00:33:54.669 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3965564315", 2024-02-23T00:33:54.669 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 12 2024-02-23T00:33:54.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.669 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142541+0000", 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142541+0000", 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142540+0000", 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1822e-05 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142552+0000", 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5166e-05 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142567+0000", 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7140000000000005e-06 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142576+0000", 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4666000000000002e-05 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142621+0000", 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9169999999999997e-06 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142626+0000", 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024531509999999999 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.167158+0000", 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011121200000000001 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.167269+0000", 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000162724 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,6", 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.167431+0000", 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0041109659999999998 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.171542+0000", 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010926302000000001 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.182469+0000", 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044268719999999998 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.226737+0000", 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9410999999999998e-05 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.226797+0000", 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6740999999999998e-05 2024-02-23T00:33:54.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.226854+0000", 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.084312260999999999 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.28903.0:2 68.6 68:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.168576+0000", 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 164.44479042099999, 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.059563420999999998, 2024-02-23T00:33:54.675 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.28903", 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/71734753", 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168576+0000", 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.676 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168576+0000", 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168575+0000", 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9290000000000001e-06 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168582+0000", 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7690000000000002e-06 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.677 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168590+0000", 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9029999999999998e-06 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168597+0000", 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00042019799999999999 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169018+0000", 2024-02-23T00:33:54.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4330000000000001e-06 2024-02-23T00:33:54.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:33:54.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169024+0000", 2024-02-23T00:33:54.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036596376 2024-02-23T00:33:54.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.680 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.680 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.205620+0000", 2024-02-23T00:33:54.680 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000110337 2024-02-23T00:33:54.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.680 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.680 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.205731+0000", 2024-02-23T00:33:54.680 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015317099999999999 2024-02-23T00:33:54.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 5,7", 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.205884+0000", 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0072101079999999998 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.213094+0000", 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6569999999999999e-05 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.213111+0000", 2024-02-23T00:33:54.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014960002999999999 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.228071+0000", 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0018999999999998e-05 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.228101+0000", 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9279999999999997e-05 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.228140+0000", 2024-02-23T00:33:54.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.059563420999999998 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15189.0:14 61.11 61:8c482978:test-rados-api-smithi087-82107-8::foo:head [write 0~128] snapc 7=[7,6] ondisk+write+known_if_redirected e120)", 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.166692+0000", 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 163.44667501699999, 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.21529174600000001, 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15189", 2024-02-23T00:33:54.683 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3965564315", 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 14 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166692+0000", 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166692+0000", 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:33:54.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166690+0000", 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2899999999999997e-06 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166698+0000", 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6489999999999997e-06 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166706+0000", 2024-02-23T00:33:54.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6569999999999997e-06 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166714+0000", 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8881999999999997e-05 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166763+0000", 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0369999999999995e-06 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166771+0000", 2024-02-23T00:33:54.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.037645606999999998 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.204416+0000", 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5403999999999997e-05 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.204512+0000", 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012169938 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,6", 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.216682+0000", 2024-02-23T00:33:54.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0083575160000000006 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225039+0000", 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.13363486999999999 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358674+0000", 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023201506 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.688 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.381876+0000", 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7921999999999999e-05 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.381934+0000", 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0055000000000001e-05 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.689 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.381984+0000", 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.21529174600000001 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15147.0:75 56.1 e122/110 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head v 122'16, mlcod=118'14)", 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:13.004465+0000", 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 161.60890207700001, 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062183177999999999, 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.690 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004465+0000", 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004465+0000", 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004463+0000", 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.925e-05 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004482+0000", 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6399999999999999e-05 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004498+0000", 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8489999999999995e-06 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004507+0000", 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.038591217999999997 2024-02-23T00:33:54.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.043098+0000", 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5066999999999999e-05 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.043123+0000", 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023502034000000002 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.066626+0000", 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2450999999999999e-05 2024-02-23T00:33:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.066648+0000", 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062183177999999999 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.7.0:204 63.13 e139/114 63:cacde8f7:test-rados-api-smithi087-82091-12::bar:2 v 139'5, mlcod=134'1)", 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.059269+0000", 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 144.55409752899999, 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026021935, 2024-02-23T00:33:54.694 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059269+0000", 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059269+0000", 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.695 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059268+0000", 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2863e-05 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059281+0000", 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5006e-05 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059296+0000", 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9260000000000003e-06 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.696 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059303+0000", 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00091365900000000004 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.060217+0000", 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0594999999999998e-05 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.060237+0000", 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025023361000000001 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085261+0000", 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0518999999999997e-05 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085291+0000", 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026021935 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.1.0:298 86.13 e139/133 86:cacde8f7:test-rados-api-smithi087-82091-12::bar:2 v 139'4, mlcod=138'1)", 2024-02-23T00:33:54.698 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 144.52936957899999, 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036243368999999998, 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1370000000000008e-06 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084006+0000", 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4387e-05 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084020+0000", 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5240000000000003e-06 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084028+0000", 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9998e-05 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.701 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084128+0000", 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3402000000000001e-05 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084151+0000", 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036055049999999998 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.120206+0000", 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4238e-05 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.120241+0000", 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036243368999999998 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:369 63.6 63:6271ce74:test-rados-api-smithi087-82091-14::foo0:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:34.984178+0000", 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 139.62918897899999, 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040494046999999998, 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.703 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 369 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984178+0000", 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984178+0000", 2024-02-23T00:33:54.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984176+0000", 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0913e-05 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984187+0000", 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024830016 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.009017+0000", 2024-02-23T00:33:54.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.131e-05 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.009028+0000", 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012722213 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.021750+0000", 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000138596 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.021889+0000", 2024-02-23T00:33:54.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016801399999999999 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,6", 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.022057+0000", 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001482412 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.023539+0000", 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00053109800000000003 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024070+0000", 2024-02-23T00:33:54.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00050785999999999995 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024578+0000", 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1844e-05 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024630+0000", 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1841999999999997e-05 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.708 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024672+0000", 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040494046999999998 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:578 63.a e146/114 63:54d171db:test-rados-api-smithi087-82091-14::foo7:head v 146'5, mlcod=144'1)", 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:37.023058+0000", 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 137.59030872299999, 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044013903, 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.709 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023058+0000", 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023058+0000", 2024-02-23T00:33:54.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023056+0000", 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5184000000000001e-05 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023071+0000", 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8630999999999998e-05 2024-02-23T00:33:54.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023090+0000", 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6339999999999996e-06 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023100+0000", 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1316999999999997e-05 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023151+0000", 2024-02-23T00:33:54.712 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9091999999999999e-05 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023170+0000", 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.043871062000000002 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.067041+0000", 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0898999999999997e-05 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.713 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.067072+0000", 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044013903 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:773 63.6 63:6271ce74:test-rados-api-smithi087-82091-14::foo0:head [writefull 0~11] snapc 7=[7,6,5,4] ondisk+write+known_if_redirected e148)", 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:38.988530+0000", 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 135.62483655299999, 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021984808000000002, 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.714 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 773 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.988530+0000", 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.988530+0000", 2024-02-23T00:33:54.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.988528+0000", 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1435e-05 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.988539+0000", 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1140000000000001e-05 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.988561+0000", 2024-02-23T00:33:54.716 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7089999999999997e-06 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.988570+0000", 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8683000000000001e-05 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.988619+0000", 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000109874 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.988729+0000", 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000202089 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,6", 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.988931+0000", 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0013524139999999999 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.990283+0000", 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011462968 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.001746+0000", 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0086700270000000003 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.010416+0000", 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5074000000000002e-05 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.010471+0000", 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3671999999999998e-05 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.719 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.010515+0000", 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021984808000000002 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:1810 63.8 e216/114 63:15d27fd9:test-rados-api-smithi087-82091-22::foo:head v 216'69, mlcod=198'66)", 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 67.586804778000001, 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.032422279999999998, 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.720 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.721 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:33:54.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1569999999999999e-06 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026567+0000", 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3954000000000001e-05 2024-02-23T00:33:54.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026581+0000", 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1770000000000003e-06 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026587+0000", 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000565939 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027153+0000", 2024-02-23T00:33:54.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0520999999999999e-05 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027173+0000", 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.031782471 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.058956+0000", 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.8521e-05 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.724 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.058984+0000", 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.032422279999999998 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26739.0:1 159.6s0 159:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e233)", 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:04.031498+0000", 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 50.581868555, 2024-02-23T00:33:54.725 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0251979280000001, 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26739", 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1200734777", 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031498+0000", 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.726 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031498+0000", 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031497+0000", 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5536000000000002e-05 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031512+0000", 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0933e-05 2024-02-23T00:33:54.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031523+0000", 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6389999999999992e-06 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031531+0000", 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3112999999999998e-05 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031584+0000", 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.066e-06 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031587+0000", 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0198177390000001 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.051405+0000", 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000104659 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.051509+0000", 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00031744400000000001 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.051827+0000", 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000397279 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.052224+0000", 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0044106010000000001 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.730 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.056635+0000", 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1668000000000007e-05 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.056696+0000", 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0251979280000001 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37253.0:1 170.6s0 170:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e254)", 2024-02-23T00:33:54.731 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:25.048138+0000", 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 29.565228989000001, 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.437999077, 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37253", 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1057868100", 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:33:54.732 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048138+0000", 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048138+0000", 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048136+0000", 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7890000000000006e-06 2024-02-23T00:33:54.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048146+0000", 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7639999999999994e-06 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048155+0000", 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5249999999999996e-06 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:33:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048160+0000", 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9122999999999997e-05 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048209+0000", 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7929999999999999e-06 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048213+0000", 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.436081875 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.484295+0000", 2024-02-23T00:33:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011504499999999999 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.484410+0000", 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00043144499999999997 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.484842+0000", 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00066020200000000003 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:33:54.736 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.485502+0000", 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00059683000000000002 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.486099+0000", 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8315000000000002e-05 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.486137+0000", 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.437999077 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:33:54.737 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:33:54.738 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:33:54.738 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:54.739 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:52.812680+0000 mgr.y (mgr.24437) 293 : audit [DBG] from='client.37367 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:33:54.739 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:54.035392+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:33:54.739 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:54.035503+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-02-23T00:33:54.739 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:54.039411+0000 mon.b (mon.1) 246 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:54.739 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:54.041193+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:54.739 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:54.042607+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.87:0/2182945223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:54.739 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:54.043256+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:33:54.766 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:54.804 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:54.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:54.910 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:54.936 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:55.135 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:55.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:55.157 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-02-23T00:33:52.026771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7005,"num_objects":346,"num_object_clones":2,"num_object_copies":1038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":1282,"num_read_kb":913,"num_write":1165,"num_write_kb":584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2546,"ondisk_log_size":2546,"up":618,"acting":618,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":722,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2480188,"kb_used_data":36804,"kb_used_omap":0,"kb_used_meta":2443264,"kb_avail":747481028,"statfs":{"total":767960285184,"available":765420572672,"internally_reserved":0,"allocated":37687296,"data_stored":20989260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2501902336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4,"num_objects":-4,"num_object_clones":-3,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-3,"num_write_kb":-3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":-3,"num_flush_kb":-3,"num_evict":0,"num_evict_kb":0,"num_promote":-1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.991330"},"pg_stats":[{"pgid":"182.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"182.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:33:52.023935+0000","last_change":"2024-02-23T00:33:52.023935+0000","last_active":"2024-02-23T00:33:52.023935+0000","last_peered":"2024-02-23T00:33:52.023935+0000","last_clean":"2024-02-23T00:33:52.023935+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:33:52.023935+0000","last_undegraded":"2024-02-23T00:33:52.023935+0000","last_fullsized":"2024-02-23T00:33:52.023935+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:52.023935+0000","last_clean_scrub_stamp":"2024-02-23T00:33:52.023935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"63.0","version":"222'68","reported_seq":272,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.041828+0000","last_change":"2024-02-23T00:33:48.242313+0000","last_active":"2024-02-23T00:33:49.041828+0000","last_peered":"2024-02-23T00:33:49.041828+0000","last_clean":"2024-02-23T00:33:49.041828+0000","last_became_active":"2024-02-23T00:31:06.330477+0000","last_became_peered":"2024-02-23T00:31:06.330477+0000","last_unstale":"2024-02-23T00:33:49.041828+0000","last_undegraded":"2024-02-23T00:33:49.041828+0000","last_fullsized":"2024-02-23T00:33:49.041828+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.1","version":"226'85","reported_seq":285,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043806+0000","last_change":"2024-02-23T00:33:48.038349+0000","last_active":"2024-02-23T00:33:49.043806+0000","last_peered":"2024-02-23T00:33:49.043806+0000","last_clean":"2024-02-23T00:33:49.043806+0000","last_became_active":"2024-02-23T00:31:06.258140+0000","last_became_peered":"2024-02-23T00:31:06.258140+0000","last_unstale":"2024-02-23T00:33:49.043806+0000","last_undegraded":"2024-02-23T00:33:49.043806+0000","last_fullsized":"2024-02-23T00:33:49.043806+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.2","version":"264'20","reported_seq":235,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042829+0000","last_change":"2024-02-23T00:33:48.242548+0000","last_active":"2024-02-23T00:33:49.042829+0000","last_peered":"2024-02-23T00:33:49.042829+0000","last_clean":"2024-02-23T00:33:49.042829+0000","last_became_active":"2024-02-23T00:31:06.329726+0000","last_became_peered":"2024-02-23T00:31:06.329726+0000","last_unstale":"2024-02-23T00:33:49.042829+0000","last_undegraded":"2024-02-23T00:33:49.042829+0000","last_fullsized":"2024-02-23T00:33:49.042829+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.3","version":"248'25","reported_seq":238,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037590+0000","last_change":"2024-02-23T00:33:48.237842+0000","last_active":"2024-02-23T00:33:49.037590+0000","last_peered":"2024-02-23T00:33:49.037590+0000","last_clean":"2024-02-23T00:33:49.037590+0000","last_became_active":"2024-02-23T00:31:06.258997+0000","last_became_peered":"2024-02-23T00:31:06.258997+0000","last_unstale":"2024-02-23T00:33:49.037590+0000","last_undegraded":"2024-02-23T00:33:49.037590+0000","last_fullsized":"2024-02-23T00:33:49.037590+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.4","version":"236'39","reported_seq":245,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037466+0000","last_change":"2024-02-23T00:33:48.238511+0000","last_active":"2024-02-23T00:33:49.037466+0000","last_peered":"2024-02-23T00:33:49.037466+0000","last_clean":"2024-02-23T00:33:49.037466+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:33:49.037466+0000","last_undegraded":"2024-02-23T00:33:49.037466+0000","last_fullsized":"2024-02-23T00:33:49.037466+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.5","version":"234'124","reported_seq":323,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044627+0000","last_change":"2024-02-23T00:33:48.037982+0000","last_active":"2024-02-23T00:33:49.044627+0000","last_peered":"2024-02-23T00:33:49.044627+0000","last_clean":"2024-02-23T00:33:49.044627+0000","last_became_active":"2024-02-23T00:31:06.258422+0000","last_became_peered":"2024-02-23T00:31:06.258422+0000","last_unstale":"2024-02-23T00:33:49.044627+0000","last_undegraded":"2024-02-23T00:33:49.044627+0000","last_fullsized":"2024-02-23T00:33:49.044627+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":124,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.6","version":"206'51","reported_seq":264,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037779+0000","last_change":"2024-02-23T00:33:48.043154+0000","last_active":"2024-02-23T00:33:49.037779+0000","last_peered":"2024-02-23T00:33:49.037779+0000","last_clean":"2024-02-23T00:33:49.037779+0000","last_became_active":"2024-02-23T00:31:06.313441+0000","last_became_peered":"2024-02-23T00:31:06.313441+0000","last_unstale":"2024-02-23T00:33:49.037779+0000","last_undegraded":"2024-02-23T00:33:49.037779+0000","last_fullsized":"2024-02-23T00:33:49.037779+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:32:48.841846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.841846+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.7","version":"229'92","reported_seq":291,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043972+0000","last_change":"2024-02-23T00:33:48.247391+0000","last_active":"2024-02-23T00:33:49.043972+0000","last_peered":"2024-02-23T00:33:49.043972+0000","last_clean":"2024-02-23T00:33:49.043972+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:33:49.043972+0000","last_undegraded":"2024-02-23T00:33:49.043972+0000","last_fullsized":"2024-02-23T00:33:49.043972+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.8","version":"219'70","reported_seq":278,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043369+0000","last_change":"2024-02-23T00:33:48.292128+0000","last_active":"2024-02-23T00:33:49.043369+0000","last_peered":"2024-02-23T00:33:49.043369+0000","last_clean":"2024-02-23T00:33:49.043369+0000","last_became_active":"2024-02-23T00:31:06.363442+0000","last_became_peered":"2024-02-23T00:31:06.363442+0000","last_unstale":"2024-02-23T00:33:49.043369+0000","last_undegraded":"2024-02-23T00:33:49.043369+0000","last_fullsized":"2024-02-23T00:33:49.043369+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.9","version":"208'17","reported_seq":229,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042010+0000","last_change":"2024-02-23T00:33:48.243032+0000","last_active":"2024-02-23T00:33:49.042010+0000","last_peered":"2024-02-23T00:33:49.042010+0000","last_clean":"2024-02-23T00:33:49.042010+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:33:49.042010+0000","last_undegraded":"2024-02-23T00:33:49.042010+0000","last_fullsized":"2024-02-23T00:33:49.042010+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.a","version":"248'22","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036954+0000","last_change":"2024-02-23T00:33:48.236273+0000","last_active":"2024-02-23T00:33:49.036954+0000","last_peered":"2024-02-23T00:33:49.036954+0000","last_clean":"2024-02-23T00:33:49.036954+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:33:49.036954+0000","last_undegraded":"2024-02-23T00:33:49.036954+0000","last_fullsized":"2024-02-23T00:33:49.036954+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'22","last_scrub_stamp":"2024-02-23T00:33:34.362545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.362545+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.b","version":"202'34","reported_seq":248,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045664+0000","last_change":"2024-02-23T00:33:48.299421+0000","last_active":"2024-02-23T00:33:49.045664+0000","last_peered":"2024-02-23T00:33:49.045664+0000","last_clean":"2024-02-23T00:33:49.045664+0000","last_became_active":"2024-02-23T00:31:06.294667+0000","last_became_peered":"2024-02-23T00:31:06.294667+0000","last_unstale":"2024-02-23T00:33:49.045664+0000","last_undegraded":"2024-02-23T00:33:49.045664+0000","last_fullsized":"2024-02-23T00:33:49.045664+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-23T00:33:37.391203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.391203+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.c","version":"208'68","reported_seq":274,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.046259+0000","last_change":"2024-02-23T00:33:48.290340+0000","last_active":"2024-02-23T00:33:49.046259+0000","last_peered":"2024-02-23T00:33:49.046259+0000","last_clean":"2024-02-23T00:33:49.046259+0000","last_became_active":"2024-02-23T00:31:06.233844+0000","last_became_peered":"2024-02-23T00:31:06.233844+0000","last_unstale":"2024-02-23T00:33:49.046259+0000","last_undegraded":"2024-02-23T00:33:49.046259+0000","last_fullsized":"2024-02-23T00:33:49.046259+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'68","last_scrub_stamp":"2024-02-23T00:33:47.391824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.391824+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.d","version":"224'34","reported_seq":245,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037616+0000","last_change":"2024-02-23T00:33:48.238674+0000","last_active":"2024-02-23T00:33:49.037616+0000","last_peered":"2024-02-23T00:33:49.037616+0000","last_clean":"2024-02-23T00:33:49.037616+0000","last_became_active":"2024-02-23T00:31:06.262251+0000","last_became_peered":"2024-02-23T00:31:06.262251+0000","last_unstale":"2024-02-23T00:33:49.037616+0000","last_undegraded":"2024-02-23T00:33:49.037616+0000","last_fullsized":"2024-02-23T00:33:49.037616+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-23T00:33:45.265466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.265466+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.e","version":"220'73","reported_seq":279,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042856+0000","last_change":"2024-02-23T00:33:48.246472+0000","last_active":"2024-02-23T00:33:49.042856+0000","last_peered":"2024-02-23T00:33:49.042856+0000","last_clean":"2024-02-23T00:33:49.042856+0000","last_became_active":"2024-02-23T00:31:06.356847+0000","last_became_peered":"2024-02-23T00:31:06.356847+0000","last_unstale":"2024-02-23T00:33:49.042856+0000","last_undegraded":"2024-02-23T00:33:49.042856+0000","last_fullsized":"2024-02-23T00:33:49.042856+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.f","version":"219'69","reported_seq":272,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044678+0000","last_change":"2024-02-23T00:33:48.295325+0000","last_active":"2024-02-23T00:33:49.044678+0000","last_peered":"2024-02-23T00:33:49.044678+0000","last_clean":"2024-02-23T00:33:49.044678+0000","last_became_active":"2024-02-23T00:31:06.364609+0000","last_became_peered":"2024-02-23T00:31:06.364609+0000","last_unstale":"2024-02-23T00:33:49.044678+0000","last_undegraded":"2024-02-23T00:33:49.044678+0000","last_fullsized":"2024-02-23T00:33:49.044678+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.10","version":"218'34","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.041833+0000","last_change":"2024-02-23T00:33:48.242149+0000","last_active":"2024-02-23T00:33:49.041833+0000","last_peered":"2024-02-23T00:33:49.041833+0000","last_clean":"2024-02-23T00:33:49.041833+0000","last_became_active":"2024-02-23T00:31:06.342164+0000","last_became_peered":"2024-02-23T00:31:06.342164+0000","last_unstale":"2024-02-23T00:33:49.041833+0000","last_undegraded":"2024-02-23T00:33:49.041833+0000","last_fullsized":"2024-02-23T00:33:49.041833+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.11","version":"248'45","reported_seq":249,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043686+0000","last_change":"2024-02-23T00:33:48.037299+0000","last_active":"2024-02-23T00:33:49.043686+0000","last_peered":"2024-02-23T00:33:49.043686+0000","last_clean":"2024-02-23T00:33:49.043686+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:33:49.043686+0000","last_undegraded":"2024-02-23T00:33:49.043686+0000","last_fullsized":"2024-02-23T00:33:49.043686+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.12","version":"204'68","reported_seq":269,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044246+0000","last_change":"2024-02-23T00:33:48.289167+0000","last_active":"2024-02-23T00:33:49.044246+0000","last_peered":"2024-02-23T00:33:49.044246+0000","last_clean":"2024-02-23T00:33:49.044246+0000","last_became_active":"2024-02-23T00:31:06.281619+0000","last_became_peered":"2024-02-23T00:31:06.281619+0000","last_unstale":"2024-02-23T00:33:49.044246+0000","last_undegraded":"2024-02-23T00:33:49.044246+0000","last_fullsized":"2024-02-23T00:33:49.044246+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'68","last_scrub_stamp":"2024-02-23T00:33:46.399784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.399784+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.13","version":"248'61","reported_seq":270,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045538+0000","last_change":"2024-02-23T00:33:48.299177+0000","last_active":"2024-02-23T00:33:49.045538+0000","last_peered":"2024-02-23T00:33:49.045538+0000","last_clean":"2024-02-23T00:33:49.045538+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:33:49.045538+0000","last_undegraded":"2024-02-23T00:33:49.045538+0000","last_fullsized":"2024-02-23T00:33:49.045538+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'61","last_scrub_stamp":"2024-02-23T00:33:41.498235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:41.498235+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.14","version":"208'85","reported_seq":287,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.046249+0000","last_change":"2024-02-23T00:33:48.357450+0000","last_active":"2024-02-23T00:33:49.046249+0000","last_peered":"2024-02-23T00:33:49.046249+0000","last_clean":"2024-02-23T00:33:49.046249+0000","last_became_active":"2024-02-23T00:31:06.234521+0000","last_became_peered":"2024-02-23T00:31:06.234521+0000","last_unstale":"2024-02-23T00:33:49.046249+0000","last_undegraded":"2024-02-23T00:33:49.046249+0000","last_fullsized":"2024-02-23T00:33:49.046249+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'85","last_scrub_stamp":"2024-02-23T00:33:48.357371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:48.357371+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.15","version":"220'68","reported_seq":274,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044502+0000","last_change":"2024-02-23T00:33:48.037678+0000","last_active":"2024-02-23T00:33:49.044502+0000","last_peered":"2024-02-23T00:33:49.044502+0000","last_clean":"2024-02-23T00:33:49.044502+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:33:49.044502+0000","last_undegraded":"2024-02-23T00:33:49.044502+0000","last_fullsized":"2024-02-23T00:33:49.044502+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.16","version":"204'51","reported_seq":265,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037680+0000","last_change":"2024-02-23T00:33:48.042926+0000","last_active":"2024-02-23T00:33:49.037680+0000","last_peered":"2024-02-23T00:33:49.037680+0000","last_clean":"2024-02-23T00:33:49.037680+0000","last_became_active":"2024-02-23T00:31:06.367069+0000","last_became_peered":"2024-02-23T00:31:06.367069+0000","last_unstale":"2024-02-23T00:33:49.037680+0000","last_undegraded":"2024-02-23T00:33:49.037680+0000","last_fullsized":"2024-02-23T00:33:49.037680+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:32:57.837196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:57.837196+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.17","version":"277'94","reported_seq":302,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.240110+0000","last_change":"2024-02-23T00:33:50.240380+0000","last_active":"2024-02-23T00:33:50.240110+0000","last_peered":"2024-02-23T00:33:50.240110+0000","last_clean":"2024-02-23T00:33:50.240110+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:33:50.240110+0000","last_undegraded":"2024-02-23T00:33:50.240110+0000","last_fullsized":"2024-02-23T00:33:50.240110+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'86","last_scrub_stamp":"2024-02-23T00:33:35.383425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:35.383425+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.18","version":"211'57","reported_seq":268,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042486+0000","last_change":"2024-02-23T00:33:48.297931+0000","last_active":"2024-02-23T00:33:49.042486+0000","last_peered":"2024-02-23T00:33:49.042486+0000","last_clean":"2024-02-23T00:33:49.042486+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:33:49.042486+0000","last_undegraded":"2024-02-23T00:33:49.042486+0000","last_fullsized":"2024-02-23T00:33:49.042486+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'57","last_scrub_stamp":"2024-02-23T00:33:36.369153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.369153+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.19","version":"271'54","reported_seq":271,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043564+0000","last_change":"2024-02-23T00:33:48.036867+0000","last_active":"2024-02-23T00:33:49.043564+0000","last_peered":"2024-02-23T00:33:49.043564+0000","last_clean":"2024-02-23T00:33:49.043564+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:33:49.043564+0000","last_undegraded":"2024-02-23T00:33:49.043564+0000","last_fullsized":"2024-02-23T00:33:49.043564+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'54","last_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_deep_scrub":"271'54","last_deep_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.780899+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.1a","version":"202'68","reported_seq":275,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038235+0000","last_change":"2024-02-23T00:33:48.042765+0000","last_active":"2024-02-23T00:33:49.038235+0000","last_peered":"2024-02-23T00:33:49.038235+0000","last_clean":"2024-02-23T00:33:49.038235+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:33:49.038235+0000","last_undegraded":"2024-02-23T00:33:49.038235+0000","last_fullsized":"2024-02-23T00:33:49.038235+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'68","last_scrub_stamp":"2024-02-23T00:32:58.808102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.808102+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.1b","version":"236'51","reported_seq":261,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037323+0000","last_change":"2024-02-23T00:33:48.237253+0000","last_active":"2024-02-23T00:33:49.037323+0000","last_peered":"2024-02-23T00:33:49.037323+0000","last_clean":"2024-02-23T00:33:49.037323+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:33:49.037323+0000","last_undegraded":"2024-02-23T00:33:49.037323+0000","last_fullsized":"2024-02-23T00:33:49.037323+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-02-23T00:33:46.274400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.274400+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.1c","version":"216'34","reported_seq":241,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042234+0000","last_change":"2024-02-23T00:33:48.244428+0000","last_active":"2024-02-23T00:33:49.042234+0000","last_peered":"2024-02-23T00:33:49.042234+0000","last_clean":"2024-02-23T00:33:49.042234+0000","last_became_active":"2024-02-23T00:31:06.338540+0000","last_became_peered":"2024-02-23T00:31:06.338540+0000","last_unstale":"2024-02-23T00:33:49.042234+0000","last_undegraded":"2024-02-23T00:33:49.042234+0000","last_fullsized":"2024-02-23T00:33:49.042234+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.1d","version":"229'10","reported_seq":226,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042806+0000","last_change":"2024-02-23T00:33:48.246448+0000","last_active":"2024-02-23T00:33:49.042806+0000","last_peered":"2024-02-23T00:33:49.042806+0000","last_clean":"2024-02-23T00:33:49.042806+0000","last_became_active":"2024-02-23T00:31:06.337635+0000","last_became_peered":"2024-02-23T00:31:06.337635+0000","last_unstale":"2024-02-23T00:33:49.042806+0000","last_undegraded":"2024-02-23T00:33:49.042806+0000","last_fullsized":"2024-02-23T00:33:49.042806+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.1e","version":"232'51","reported_seq":259,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.219744+0000","last_change":"2024-02-23T00:33:49.219707+0000","last_active":"2024-02-23T00:33:49.219744+0000","last_peered":"2024-02-23T00:33:49.219744+0000","last_clean":"2024-02-23T00:33:49.219744+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:33:49.219744+0000","last_undegraded":"2024-02-23T00:33:49.219744+0000","last_fullsized":"2024-02-23T00:33:49.219744+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'51","last_scrub_stamp":"2024-02-23T00:33:49.219607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.219607+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"63.1f","version":"228'68","reported_seq":279,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037986+0000","last_change":"2024-02-23T00:33:48.238796+0000","last_active":"2024-02-23T00:33:49.037986+0000","last_peered":"2024-02-23T00:33:49.037986+0000","last_clean":"2024-02-23T00:33:49.037986+0000","last_became_active":"2024-02-23T00:31:06.355623+0000","last_became_peered":"2024-02-23T00:31:06.355623+0000","last_unstale":"2024-02-23T00:33:49.037986+0000","last_undegraded":"2024-02-23T00:33:49.037986+0000","last_fullsized":"2024-02-23T00:33:49.037986+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-23T00:33:39.351573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.351573+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"1"}]},{"pgid":"5.9","version":"62'8","reported_seq":329,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.046315+0000","last_change":"2024-02-23T00:32:11.743347+0000","last_active":"2024-02-23T00:33:50.046315+0000","last_peered":"2024-02-23T00:33:50.046315+0000","last_clean":"2024-02-23T00:33:50.046315+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:33:50.046315+0000","last_undegraded":"2024-02-23T00:33:50.046315+0000","last_fullsized":"2024-02-23T00:33:50.046315+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:11.743278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.743278+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":306,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036686+0000","last_change":"2024-02-23T00:32:47.376760+0000","last_active":"2024-02-23T00:33:49.036686+0000","last_peered":"2024-02-23T00:33:49.036686+0000","last_clean":"2024-02-23T00:33:49.036686+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:33:49.036686+0000","last_undegraded":"2024-02-23T00:33:49.036686+0000","last_fullsized":"2024-02-23T00:33:49.036686+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:32:47.376691+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.376691+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037923+0000","last_change":"2024-02-23T00:33:00.878453+0000","last_active":"2024-02-23T00:33:49.037923+0000","last_peered":"2024-02-23T00:33:49.037923+0000","last_clean":"2024-02-23T00:33:49.037923+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:33:49.037923+0000","last_undegraded":"2024-02-23T00:33:49.037923+0000","last_fullsized":"2024-02-23T00:33:49.037923+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.878391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.878391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":233,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037101+0000","last_change":"2024-02-23T00:31:31.323024+0000","last_active":"2024-02-23T00:33:49.037101+0000","last_peered":"2024-02-23T00:33:49.037101+0000","last_clean":"2024-02-23T00:33:49.037101+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:33:49.037101+0000","last_undegraded":"2024-02-23T00:33:49.037101+0000","last_fullsized":"2024-02-23T00:33:49.037101+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.322940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.322940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":192,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045175+0000","last_change":"2024-02-23T00:32:48.468225+0000","last_active":"2024-02-23T00:33:49.045175+0000","last_peered":"2024-02-23T00:33:49.045175+0000","last_clean":"2024-02-23T00:33:49.045175+0000","last_became_active":"2024-02-23T00:30:48.777752+0000","last_became_peered":"2024-02-23T00:30:48.777752+0000","last_unstale":"2024-02-23T00:33:49.045175+0000","last_undegraded":"2024-02-23T00:33:49.045175+0000","last_fullsized":"2024-02-23T00:33:49.045175+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":301,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.054471+0000","last_change":"2024-02-23T00:32:37.440685+0000","last_active":"2024-02-23T00:33:50.054471+0000","last_peered":"2024-02-23T00:33:50.054471+0000","last_clean":"2024-02-23T00:33:50.054471+0000","last_became_active":"2024-02-23T00:31:29.077539+0000","last_became_peered":"2024-02-23T00:31:29.077539+0000","last_unstale":"2024-02-23T00:33:50.054471+0000","last_undegraded":"2024-02-23T00:33:50.054471+0000","last_fullsized":"2024-02-23T00:33:50.054471+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:37.440614+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.440614+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":282,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.588918+0000","last_change":"2024-02-23T00:33:49.588891+0000","last_active":"2024-02-23T00:33:49.588918+0000","last_peered":"2024-02-23T00:33:49.588918+0000","last_clean":"2024-02-23T00:33:49.588918+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:33:49.588918+0000","last_undegraded":"2024-02-23T00:33:49.588918+0000","last_fullsized":"2024-02-23T00:33:49.588918+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":245,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.041775+0000","last_change":"2024-02-23T00:32:38.328306+0000","last_active":"2024-02-23T00:33:49.041775+0000","last_peered":"2024-02-23T00:33:49.041775+0000","last_clean":"2024-02-23T00:33:49.041775+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:33:49.041775+0000","last_undegraded":"2024-02-23T00:33:49.041775+0000","last_fullsized":"2024-02-23T00:33:49.041775+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.328237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.328237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038344+0000","last_change":"2024-02-23T00:32:40.855264+0000","last_active":"2024-02-23T00:33:49.038344+0000","last_peered":"2024-02-23T00:33:49.038344+0000","last_clean":"2024-02-23T00:33:49.038344+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:33:49.038344+0000","last_undegraded":"2024-02-23T00:33:49.038344+0000","last_fullsized":"2024-02-23T00:33:49.038344+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.855194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.855194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":247,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036052+0000","last_change":"2024-02-23T00:33:47.610285+0000","last_active":"2024-02-23T00:33:49.036052+0000","last_peered":"2024-02-23T00:33:49.036052+0000","last_clean":"2024-02-23T00:33:49.036052+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:33:49.036052+0000","last_undegraded":"2024-02-23T00:33:49.036052+0000","last_fullsized":"2024-02-23T00:33:49.036052+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":193,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043880+0000","last_change":"2024-02-23T00:33:44.400851+0000","last_active":"2024-02-23T00:33:49.043880+0000","last_peered":"2024-02-23T00:33:49.043880+0000","last_clean":"2024-02-23T00:33:49.043880+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:33:49.043880+0000","last_undegraded":"2024-02-23T00:33:49.043880+0000","last_fullsized":"2024-02-23T00:33:49.043880+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.400781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.400781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":290,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044437+0000","last_change":"2024-02-23T00:31:41.019548+0000","last_active":"2024-02-23T00:33:49.044437+0000","last_peered":"2024-02-23T00:33:49.044437+0000","last_clean":"2024-02-23T00:33:49.044437+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:33:49.044437+0000","last_undegraded":"2024-02-23T00:33:49.044437+0000","last_fullsized":"2024-02-23T00:33:49.044437+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.019441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.019441+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038140+0000","last_change":"2024-02-23T00:32:59.849579+0000","last_active":"2024-02-23T00:33:49.038140+0000","last_peered":"2024-02-23T00:33:49.038140+0000","last_clean":"2024-02-23T00:33:49.038140+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:33:49.038140+0000","last_undegraded":"2024-02-23T00:33:49.038140+0000","last_fullsized":"2024-02-23T00:33:49.038140+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.849503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.849503+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037928+0000","last_change":"2024-02-23T00:32:46.885745+0000","last_active":"2024-02-23T00:33:49.037928+0000","last_peered":"2024-02-23T00:33:49.037928+0000","last_clean":"2024-02-23T00:33:49.037928+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:33:49.037928+0000","last_undegraded":"2024-02-23T00:33:49.037928+0000","last_fullsized":"2024-02-23T00:33:49.037928+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.885681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.885681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042769+0000","last_change":"2024-02-23T00:32:38.683758+0000","last_active":"2024-02-23T00:33:49.042769+0000","last_peered":"2024-02-23T00:33:49.042769+0000","last_clean":"2024-02-23T00:33:49.042769+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:33:49.042769+0000","last_undegraded":"2024-02-23T00:33:49.042769+0000","last_fullsized":"2024-02-23T00:33:49.042769+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.683686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.683686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036061+0000","last_change":"2024-02-23T00:32:45.536091+0000","last_active":"2024-02-23T00:33:49.036061+0000","last_peered":"2024-02-23T00:33:49.036061+0000","last_clean":"2024-02-23T00:33:49.036061+0000","last_became_active":"2024-02-23T00:30:48.673452+0000","last_became_peered":"2024-02-23T00:30:48.673452+0000","last_unstale":"2024-02-23T00:33:49.036061+0000","last_undegraded":"2024-02-23T00:33:49.036061+0000","last_fullsized":"2024-02-23T00:33:49.036061+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.535992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.535992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":255,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044585+0000","last_change":"2024-02-23T00:32:37.445055+0000","last_active":"2024-02-23T00:33:49.044585+0000","last_peered":"2024-02-23T00:33:49.044585+0000","last_clean":"2024-02-23T00:33:49.044585+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:33:49.044585+0000","last_undegraded":"2024-02-23T00:33:49.044585+0000","last_fullsized":"2024-02-23T00:33:49.044585+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.444973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.444973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":274,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045217+0000","last_change":"2024-02-23T00:33:36.603800+0000","last_active":"2024-02-23T00:33:49.045217+0000","last_peered":"2024-02-23T00:33:49.045217+0000","last_clean":"2024-02-23T00:33:49.045217+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:33:49.045217+0000","last_undegraded":"2024-02-23T00:33:49.045217+0000","last_fullsized":"2024-02-23T00:33:49.045217+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:33:36.603732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.603732+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":246,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045814+0000","last_change":"2024-02-23T00:32:56.692114+0000","last_active":"2024-02-23T00:33:49.045814+0000","last_peered":"2024-02-23T00:33:49.045814+0000","last_clean":"2024-02-23T00:33:49.045814+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:33:49.045814+0000","last_undegraded":"2024-02-23T00:33:49.045814+0000","last_fullsized":"2024-02-23T00:33:49.045814+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.692037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.692037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":236,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042382+0000","last_change":"2024-02-23T00:32:09.110225+0000","last_active":"2024-02-23T00:33:49.042382+0000","last_peered":"2024-02-23T00:33:49.042382+0000","last_clean":"2024-02-23T00:33:49.042382+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:33:49.042382+0000","last_undegraded":"2024-02-23T00:33:49.042382+0000","last_fullsized":"2024-02-23T00:33:49.042382+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.110138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":247,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037655+0000","last_change":"2024-02-23T00:33:33.791204+0000","last_active":"2024-02-23T00:33:49.037655+0000","last_peered":"2024-02-23T00:33:49.037655+0000","last_clean":"2024-02-23T00:33:49.037655+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:33:49.037655+0000","last_undegraded":"2024-02-23T00:33:49.037655+0000","last_fullsized":"2024-02-23T00:33:49.037655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.791137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.791137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":194,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.047125+0000","last_change":"2024-02-23T00:33:00.671292+0000","last_active":"2024-02-23T00:33:49.047125+0000","last_peered":"2024-02-23T00:33:49.047125+0000","last_clean":"2024-02-23T00:33:49.047125+0000","last_became_active":"2024-02-23T00:30:48.716552+0000","last_became_peered":"2024-02-23T00:30:48.716552+0000","last_unstale":"2024-02-23T00:33:49.047125+0000","last_undegraded":"2024-02-23T00:33:49.047125+0000","last_fullsized":"2024-02-23T00:33:49.047125+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.671219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.671219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":257,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043923+0000","last_change":"2024-02-23T00:32:53.985225+0000","last_active":"2024-02-23T00:33:49.043923+0000","last_peered":"2024-02-23T00:33:49.043923+0000","last_clean":"2024-02-23T00:33:49.043923+0000","last_became_active":"2024-02-23T00:31:29.064244+0000","last_became_peered":"2024-02-23T00:31:29.064244+0000","last_unstale":"2024-02-23T00:33:49.043923+0000","last_undegraded":"2024-02-23T00:33:49.043923+0000","last_fullsized":"2024-02-23T00:33:49.043923+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.985150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.985150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":256,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044470+0000","last_change":"2024-02-23T00:31:19.982059+0000","last_active":"2024-02-23T00:33:49.044470+0000","last_peered":"2024-02-23T00:33:49.044470+0000","last_clean":"2024-02-23T00:33:49.044470+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:33:49.044470+0000","last_undegraded":"2024-02-23T00:33:49.044470+0000","last_fullsized":"2024-02-23T00:33:49.044470+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:31:19.981985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.981985+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":258,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044832+0000","last_change":"2024-02-23T00:31:49.024710+0000","last_active":"2024-02-23T00:33:49.044832+0000","last_peered":"2024-02-23T00:33:49.044832+0000","last_clean":"2024-02-23T00:33:49.044832+0000","last_became_active":"2024-02-23T00:31:49.024302+0000","last_became_peered":"2024-02-23T00:31:49.024302+0000","last_unstale":"2024-02-23T00:33:49.044832+0000","last_undegraded":"2024-02-23T00:33:49.044832+0000","last_fullsized":"2024-02-23T00:33:49.044832+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.937969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":331,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.040926+0000","last_change":"2024-02-23T00:32:35.929043+0000","last_active":"2024-02-23T00:33:50.040926+0000","last_peered":"2024-02-23T00:33:50.040926+0000","last_clean":"2024-02-23T00:33:50.040926+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:33:50.040926+0000","last_undegraded":"2024-02-23T00:33:50.040926+0000","last_fullsized":"2024-02-23T00:33:50.040926+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.928977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.928977+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044678+0000","last_change":"2024-02-23T00:33:27.495416+0000","last_active":"2024-02-23T00:33:49.044678+0000","last_peered":"2024-02-23T00:33:49.044678+0000","last_clean":"2024-02-23T00:33:49.044678+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:33:49.044678+0000","last_undegraded":"2024-02-23T00:33:49.044678+0000","last_fullsized":"2024-02-23T00:33:49.044678+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:27.495308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:27.495308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044722+0000","last_change":"2024-02-23T00:33:32.583582+0000","last_active":"2024-02-23T00:33:49.044722+0000","last_peered":"2024-02-23T00:33:49.044722+0000","last_clean":"2024-02-23T00:33:49.044722+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:33:49.044722+0000","last_undegraded":"2024-02-23T00:33:49.044722+0000","last_fullsized":"2024-02-23T00:33:49.044722+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:32.583509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:32.583509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":156,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043861+0000","last_change":"2024-02-23T00:32:58.975408+0000","last_active":"2024-02-23T00:33:49.043861+0000","last_peered":"2024-02-23T00:33:49.043861+0000","last_clean":"2024-02-23T00:33:49.043861+0000","last_became_active":"2024-02-23T00:31:29.089319+0000","last_became_peered":"2024-02-23T00:31:29.089319+0000","last_unstale":"2024-02-23T00:33:49.043861+0000","last_undegraded":"2024-02-23T00:33:49.043861+0000","last_fullsized":"2024-02-23T00:33:49.043861+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:58.975334+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.975334+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"87'22","reported_seq":275,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044485+0000","last_change":"2024-02-23T00:32:40.327594+0000","last_active":"2024-02-23T00:33:49.044485+0000","last_peered":"2024-02-23T00:33:49.044485+0000","last_clean":"2024-02-23T00:33:49.044485+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:33:49.044485+0000","last_undegraded":"2024-02-23T00:33:49.044485+0000","last_fullsized":"2024-02-23T00:33:49.044485+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:32:40.327519+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.327519+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042120+0000","last_change":"2024-02-23T00:31:29.920657+0000","last_active":"2024-02-23T00:33:49.042120+0000","last_peered":"2024-02-23T00:33:49.042120+0000","last_clean":"2024-02-23T00:33:49.042120+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:33:49.042120+0000","last_undegraded":"2024-02-23T00:33:49.042120+0000","last_fullsized":"2024-02-23T00:33:49.042120+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:29.920546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:29.920546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":331,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.041736+0000","last_change":"2024-02-23T00:32:35.975973+0000","last_active":"2024-02-23T00:33:50.041736+0000","last_peered":"2024-02-23T00:33:50.041736+0000","last_clean":"2024-02-23T00:33:50.041736+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:33:50.041736+0000","last_undegraded":"2024-02-23T00:33:50.041736+0000","last_fullsized":"2024-02-23T00:33:50.041736+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.975889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.975889+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":234,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038024+0000","last_change":"2024-02-23T00:31:21.366604+0000","last_active":"2024-02-23T00:33:49.038024+0000","last_peered":"2024-02-23T00:33:49.038024+0000","last_clean":"2024-02-23T00:33:49.038024+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:33:49.038024+0000","last_undegraded":"2024-02-23T00:33:49.038024+0000","last_fullsized":"2024-02-23T00:33:49.038024+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:31:21.366535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.366535+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":192,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043025+0000","last_change":"2024-02-23T00:33:39.140496+0000","last_active":"2024-02-23T00:33:49.043025+0000","last_peered":"2024-02-23T00:33:49.043025+0000","last_clean":"2024-02-23T00:33:49.043025+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:33:49.043025+0000","last_undegraded":"2024-02-23T00:33:49.043025+0000","last_fullsized":"2024-02-23T00:33:49.043025+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:39.140434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.140434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":257,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043454+0000","last_change":"2024-02-23T00:32:47.737933+0000","last_active":"2024-02-23T00:33:49.043454+0000","last_peered":"2024-02-23T00:33:49.043454+0000","last_clean":"2024-02-23T00:33:49.043454+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:33:49.043454+0000","last_undegraded":"2024-02-23T00:33:49.043454+0000","last_fullsized":"2024-02-23T00:33:49.043454+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:47.737874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.737874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"86'23","reported_seq":286,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044005+0000","last_change":"2024-02-23T00:32:46.780054+0000","last_active":"2024-02-23T00:33:49.044005+0000","last_peered":"2024-02-23T00:33:49.044005+0000","last_clean":"2024-02-23T00:33:49.044005+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:33:49.044005+0000","last_undegraded":"2024-02-23T00:33:49.044005+0000","last_fullsized":"2024-02-23T00:33:49.044005+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'23","last_scrub_stamp":"2024-02-23T00:32:46.779979+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.779979+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043068+0000","last_change":"2024-02-23T00:31:38.838289+0000","last_active":"2024-02-23T00:33:49.043068+0000","last_peered":"2024-02-23T00:33:49.043068+0000","last_clean":"2024-02-23T00:33:49.043068+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:33:49.043068+0000","last_undegraded":"2024-02-23T00:33:49.043068+0000","last_fullsized":"2024-02-23T00:33:49.043068+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:31:38.838202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.838202+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":235,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038326+0000","last_change":"2024-02-23T00:31:33.381301+0000","last_active":"2024-02-23T00:33:49.038326+0000","last_peered":"2024-02-23T00:33:49.038326+0000","last_clean":"2024-02-23T00:33:49.038326+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:33:49.038326+0000","last_undegraded":"2024-02-23T00:33:49.038326+0000","last_fullsized":"2024-02-23T00:33:49.038326+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.381229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.381229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":237,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045278+0000","last_change":"2024-02-23T00:32:09.012254+0000","last_active":"2024-02-23T00:33:49.045278+0000","last_peered":"2024-02-23T00:33:49.045278+0000","last_clean":"2024-02-23T00:33:49.045278+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:33:49.045278+0000","last_undegraded":"2024-02-23T00:33:49.045278+0000","last_fullsized":"2024-02-23T00:33:49.045278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.012161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.012161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":192,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042439+0000","last_change":"2024-02-23T00:32:53.387558+0000","last_active":"2024-02-23T00:33:49.042439+0000","last_peered":"2024-02-23T00:33:49.042439+0000","last_clean":"2024-02-23T00:33:49.042439+0000","last_became_active":"2024-02-23T00:30:48.740636+0000","last_became_peered":"2024-02-23T00:30:48.740636+0000","last_unstale":"2024-02-23T00:33:49.042439+0000","last_undegraded":"2024-02-23T00:33:49.042439+0000","last_fullsized":"2024-02-23T00:33:49.042439+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.387496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":161,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037781+0000","last_change":"2024-02-23T00:32:56.610957+0000","last_active":"2024-02-23T00:33:49.037781+0000","last_peered":"2024-02-23T00:33:49.037781+0000","last_clean":"2024-02-23T00:33:49.037781+0000","last_became_active":"2024-02-23T00:31:29.108747+0000","last_became_peered":"2024-02-23T00:31:29.108747+0000","last_unstale":"2024-02-23T00:33:49.037781+0000","last_undegraded":"2024-02-23T00:33:49.037781+0000","last_fullsized":"2024-02-23T00:33:49.037781+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.610884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.610884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":330,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.056945+0000","last_change":"2024-02-23T00:31:41.945984+0000","last_active":"2024-02-23T00:33:50.056945+0000","last_peered":"2024-02-23T00:33:50.056945+0000","last_clean":"2024-02-23T00:33:50.056945+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:33:50.056945+0000","last_undegraded":"2024-02-23T00:33:50.056945+0000","last_fullsized":"2024-02-23T00:33:50.056945+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":277,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044029+0000","last_change":"2024-02-23T00:31:38.038523+0000","last_active":"2024-02-23T00:33:49.044029+0000","last_peered":"2024-02-23T00:33:49.044029+0000","last_clean":"2024-02-23T00:33:49.044029+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:33:49.044029+0000","last_undegraded":"2024-02-23T00:33:49.044029+0000","last_fullsized":"2024-02-23T00:33:49.044029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:31:38.038413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.038413+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":245,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.047227+0000","last_change":"2024-02-23T00:32:46.504136+0000","last_active":"2024-02-23T00:33:49.047227+0000","last_peered":"2024-02-23T00:33:49.047227+0000","last_clean":"2024-02-23T00:33:49.047227+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:33:49.047227+0000","last_undegraded":"2024-02-23T00:33:49.047227+0000","last_fullsized":"2024-02-23T00:33:49.047227+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.504045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.504045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":245,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045986+0000","last_change":"2024-02-23T00:33:45.587063+0000","last_active":"2024-02-23T00:33:49.045986+0000","last_peered":"2024-02-23T00:33:49.045986+0000","last_clean":"2024-02-23T00:33:49.045986+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:33:49.045986+0000","last_undegraded":"2024-02-23T00:33:49.045986+0000","last_fullsized":"2024-02-23T00:33:49.045986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:45.586975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.586975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":187,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043034+0000","last_change":"2024-02-23T00:30:48.782732+0000","last_active":"2024-02-23T00:33:49.043034+0000","last_peered":"2024-02-23T00:33:49.043034+0000","last_clean":"2024-02-23T00:33:49.043034+0000","last_became_active":"2024-02-23T00:30:48.778293+0000","last_became_peered":"2024-02-23T00:30:48.778293+0000","last_unstale":"2024-02-23T00:33:49.043034+0000","last_undegraded":"2024-02-23T00:33:49.043034+0000","last_fullsized":"2024-02-23T00:33:49.043034+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":257,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045186+0000","last_change":"2024-02-23T00:32:55.991569+0000","last_active":"2024-02-23T00:33:49.045186+0000","last_peered":"2024-02-23T00:33:49.045186+0000","last_clean":"2024-02-23T00:33:49.045186+0000","last_became_active":"2024-02-23T00:31:29.070521+0000","last_became_peered":"2024-02-23T00:31:29.070521+0000","last_unstale":"2024-02-23T00:33:49.045186+0000","last_undegraded":"2024-02-23T00:33:49.045186+0000","last_fullsized":"2024-02-23T00:33:49.045186+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:55.991514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:55.991514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044368+0000","last_change":"2024-02-23T00:32:41.922931+0000","last_active":"2024-02-23T00:33:49.044368+0000","last_peered":"2024-02-23T00:33:49.044368+0000","last_clean":"2024-02-23T00:33:49.044368+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:33:49.044368+0000","last_undegraded":"2024-02-23T00:33:49.044368+0000","last_fullsized":"2024-02-23T00:33:49.044368+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:41.922849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.922849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":303,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043630+0000","last_change":"2024-02-23T00:31:21.086750+0000","last_active":"2024-02-23T00:33:49.043630+0000","last_peered":"2024-02-23T00:33:49.043630+0000","last_clean":"2024-02-23T00:33:49.043630+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:33:49.043630+0000","last_undegraded":"2024-02-23T00:33:49.043630+0000","last_fullsized":"2024-02-23T00:33:49.043630+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:31:21.086610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.086610+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":245,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042936+0000","last_change":"2024-02-23T00:33:34.199948+0000","last_active":"2024-02-23T00:33:49.042936+0000","last_peered":"2024-02-23T00:33:49.042936+0000","last_clean":"2024-02-23T00:33:49.042936+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:33:49.042936+0000","last_undegraded":"2024-02-23T00:33:49.042936+0000","last_fullsized":"2024-02-23T00:33:49.042936+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.199869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.199869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043251+0000","last_change":"2024-02-23T00:32:17.981331+0000","last_active":"2024-02-23T00:33:49.043251+0000","last_peered":"2024-02-23T00:33:49.043251+0000","last_clean":"2024-02-23T00:33:49.043251+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:33:49.043251+0000","last_undegraded":"2024-02-23T00:33:49.043251+0000","last_fullsized":"2024-02-23T00:33:49.043251+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.981262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.981262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":190,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043371+0000","last_change":"2024-02-23T00:30:48.776316+0000","last_active":"2024-02-23T00:33:49.043371+0000","last_peered":"2024-02-23T00:33:49.043371+0000","last_clean":"2024-02-23T00:33:49.043371+0000","last_became_active":"2024-02-23T00:30:48.769602+0000","last_became_peered":"2024-02-23T00:30:48.769602+0000","last_unstale":"2024-02-23T00:33:49.043371+0000","last_undegraded":"2024-02-23T00:33:49.043371+0000","last_fullsized":"2024-02-23T00:33:49.043371+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":508,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042170+0000","last_change":"2024-02-23T00:32:43.323030+0000","last_active":"2024-02-23T00:33:49.042170+0000","last_peered":"2024-02-23T00:33:49.042170+0000","last_clean":"2024-02-23T00:33:49.042170+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:33:49.042170+0000","last_undegraded":"2024-02-23T00:33:49.042170+0000","last_fullsized":"2024-02-23T00:33:49.042170+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:43.322963+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.322963+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044464+0000","last_change":"2024-02-23T00:32:42.410196+0000","last_active":"2024-02-23T00:33:49.044464+0000","last_peered":"2024-02-23T00:33:49.044464+0000","last_clean":"2024-02-23T00:33:49.044464+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:33:49.044464+0000","last_undegraded":"2024-02-23T00:33:49.044464+0000","last_fullsized":"2024-02-23T00:33:49.044464+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.410112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.410112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":302,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036592+0000","last_change":"2024-02-23T00:32:47.533001+0000","last_active":"2024-02-23T00:33:49.036592+0000","last_peered":"2024-02-23T00:33:49.036592+0000","last_clean":"2024-02-23T00:33:49.036592+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:33:49.036592+0000","last_undegraded":"2024-02-23T00:33:49.036592+0000","last_fullsized":"2024-02-23T00:33:49.036592+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:32:47.532917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.532917+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":246,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.047190+0000","last_change":"2024-02-23T00:31:40.847863+0000","last_active":"2024-02-23T00:33:49.047190+0000","last_peered":"2024-02-23T00:33:49.047190+0000","last_clean":"2024-02-23T00:33:49.047190+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:33:49.047190+0000","last_undegraded":"2024-02-23T00:33:49.047190+0000","last_fullsized":"2024-02-23T00:33:49.047190+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.847784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.847784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042165+0000","last_change":"2024-02-23T00:33:38.163226+0000","last_active":"2024-02-23T00:33:49.042165+0000","last_peered":"2024-02-23T00:33:49.042165+0000","last_clean":"2024-02-23T00:33:49.042165+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:33:49.042165+0000","last_undegraded":"2024-02-23T00:33:49.042165+0000","last_fullsized":"2024-02-23T00:33:49.042165+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.163155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.163155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":187,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042207+0000","last_change":"2024-02-23T00:30:48.707450+0000","last_active":"2024-02-23T00:33:49.042207+0000","last_peered":"2024-02-23T00:33:49.042207+0000","last_clean":"2024-02-23T00:33:49.042207+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:33:49.042207+0000","last_undegraded":"2024-02-23T00:33:49.042207+0000","last_fullsized":"2024-02-23T00:33:49.042207+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038381+0000","last_change":"2024-02-23T00:32:12.386351+0000","last_active":"2024-02-23T00:33:49.038381+0000","last_peered":"2024-02-23T00:33:49.038381+0000","last_clean":"2024-02-23T00:33:49.038381+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:33:49.038381+0000","last_undegraded":"2024-02-23T00:33:49.038381+0000","last_fullsized":"2024-02-23T00:33:49.038381+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":269,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037352+0000","last_change":"2024-02-23T00:32:46.507450+0000","last_active":"2024-02-23T00:33:49.037352+0000","last_peered":"2024-02-23T00:33:49.037352+0000","last_clean":"2024-02-23T00:33:49.037352+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:33:49.037352+0000","last_undegraded":"2024-02-23T00:33:49.037352+0000","last_fullsized":"2024-02-23T00:33:49.037352+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:32:46.507354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.507354+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":246,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042904+0000","last_change":"2024-02-23T00:33:42.509570+0000","last_active":"2024-02-23T00:33:49.042904+0000","last_peered":"2024-02-23T00:33:49.042904+0000","last_clean":"2024-02-23T00:33:49.042904+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:33:49.042904+0000","last_undegraded":"2024-02-23T00:33:49.042904+0000","last_fullsized":"2024-02-23T00:33:49.042904+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.509463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.509463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":262,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.035738+0000","last_change":"2024-02-23T00:33:31.593419+0000","last_active":"2024-02-23T00:33:49.035738+0000","last_peered":"2024-02-23T00:33:49.035738+0000","last_clean":"2024-02-23T00:33:49.035738+0000","last_became_active":"2024-02-23T00:31:27.053776+0000","last_became_peered":"2024-02-23T00:31:27.053776+0000","last_unstale":"2024-02-23T00:33:49.035738+0000","last_undegraded":"2024-02-23T00:33:49.035738+0000","last_fullsized":"2024-02-23T00:33:49.035738+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:31.593354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:31.593354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":234,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042457+0000","last_change":"2024-02-23T00:31:21.959719+0000","last_active":"2024-02-23T00:33:49.042457+0000","last_peered":"2024-02-23T00:33:49.042457+0000","last_clean":"2024-02-23T00:33:49.042457+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:33:49.042457+0000","last_undegraded":"2024-02-23T00:33:49.042457+0000","last_fullsized":"2024-02-23T00:33:49.042457+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.959597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.959597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044548+0000","last_change":"2024-02-23T00:33:34.545991+0000","last_active":"2024-02-23T00:33:49.044548+0000","last_peered":"2024-02-23T00:33:49.044548+0000","last_clean":"2024-02-23T00:33:49.044548+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:33:49.044548+0000","last_undegraded":"2024-02-23T00:33:49.044548+0000","last_fullsized":"2024-02-23T00:33:49.044548+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.545903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.545903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.046557+0000","last_change":"2024-02-23T00:32:17.590019+0000","last_active":"2024-02-23T00:33:49.046557+0000","last_peered":"2024-02-23T00:33:49.046557+0000","last_clean":"2024-02-23T00:33:49.046557+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:33:49.046557+0000","last_undegraded":"2024-02-23T00:33:49.046557+0000","last_fullsized":"2024-02-23T00:33:49.046557+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.589952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.589952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":266,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.046889+0000","last_change":"2024-02-23T00:31:40.030848+0000","last_active":"2024-02-23T00:33:49.046889+0000","last_peered":"2024-02-23T00:33:49.046889+0000","last_clean":"2024-02-23T00:33:49.046889+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:33:49.046889+0000","last_undegraded":"2024-02-23T00:33:49.046889+0000","last_fullsized":"2024-02-23T00:33:49.046889+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:31:40.030776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.030776+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042980+0000","last_change":"2024-02-23T00:33:43.548018+0000","last_active":"2024-02-23T00:33:49.042980+0000","last_peered":"2024-02-23T00:33:49.042980+0000","last_clean":"2024-02-23T00:33:49.042980+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:33:49.042980+0000","last_undegraded":"2024-02-23T00:33:49.042980+0000","last_fullsized":"2024-02-23T00:33:49.042980+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038221+0000","last_change":"2024-02-23T00:32:46.410759+0000","last_active":"2024-02-23T00:33:49.038221+0000","last_peered":"2024-02-23T00:33:49.038221+0000","last_clean":"2024-02-23T00:33:49.038221+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:33:49.038221+0000","last_undegraded":"2024-02-23T00:33:49.038221+0000","last_fullsized":"2024-02-23T00:33:49.038221+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.410635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.410635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":257,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043431+0000","last_change":"2024-02-23T00:32:46.978493+0000","last_active":"2024-02-23T00:33:49.043431+0000","last_peered":"2024-02-23T00:33:49.043431+0000","last_clean":"2024-02-23T00:33:49.043431+0000","last_became_active":"2024-02-23T00:31:27.144976+0000","last_became_peered":"2024-02-23T00:31:27.144976+0000","last_unstale":"2024-02-23T00:33:49.043431+0000","last_undegraded":"2024-02-23T00:33:49.043431+0000","last_fullsized":"2024-02-23T00:33:49.043431+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.978406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.978406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":238,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038433+0000","last_change":"2024-02-23T00:33:00.439952+0000","last_active":"2024-02-23T00:33:49.038433+0000","last_peered":"2024-02-23T00:33:49.038433+0000","last_clean":"2024-02-23T00:33:49.038433+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:33:49.038433+0000","last_undegraded":"2024-02-23T00:33:49.038433+0000","last_fullsized":"2024-02-23T00:33:49.038433+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.439872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044365+0000","last_change":"2024-02-23T00:33:33.574217+0000","last_active":"2024-02-23T00:33:49.044365+0000","last_peered":"2024-02-23T00:33:49.044365+0000","last_clean":"2024-02-23T00:33:49.044365+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:33:49.044365+0000","last_undegraded":"2024-02-23T00:33:49.044365+0000","last_fullsized":"2024-02-23T00:33:49.044365+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.574124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.574124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":249,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038071+0000","last_change":"2024-02-23T00:33:38.742275+0000","last_active":"2024-02-23T00:33:49.038071+0000","last_peered":"2024-02-23T00:33:49.038071+0000","last_clean":"2024-02-23T00:33:49.038071+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:33:49.038071+0000","last_undegraded":"2024-02-23T00:33:49.038071+0000","last_fullsized":"2024-02-23T00:33:49.038071+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":285,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044280+0000","last_change":"2024-02-23T00:32:13.104236+0000","last_active":"2024-02-23T00:33:49.044280+0000","last_peered":"2024-02-23T00:33:49.044280+0000","last_clean":"2024-02-23T00:33:49.044280+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:33:49.044280+0000","last_undegraded":"2024-02-23T00:33:49.044280+0000","last_fullsized":"2024-02-23T00:33:49.044280+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:32:13.104170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.104170+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037032+0000","last_change":"2024-02-23T00:33:32.799453+0000","last_active":"2024-02-23T00:33:49.037032+0000","last_peered":"2024-02-23T00:33:49.037032+0000","last_clean":"2024-02-23T00:33:49.037032+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:33:49.037032+0000","last_undegraded":"2024-02-23T00:33:49.037032+0000","last_fullsized":"2024-02-23T00:33:49.037032+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:32.799394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:32.799394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":501,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042601+0000","last_change":"2024-02-23T00:32:41.360750+0000","last_active":"2024-02-23T00:33:49.042601+0000","last_peered":"2024-02-23T00:33:49.042601+0000","last_clean":"2024-02-23T00:33:49.042601+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:33:49.042601+0000","last_undegraded":"2024-02-23T00:33:49.042601+0000","last_fullsized":"2024-02-23T00:33:49.042601+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:41.360646+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.360646+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.047105+0000","last_change":"2024-02-23T00:32:14.650905+0000","last_active":"2024-02-23T00:33:49.047105+0000","last_peered":"2024-02-23T00:33:49.047105+0000","last_clean":"2024-02-23T00:33:49.047105+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:33:49.047105+0000","last_undegraded":"2024-02-23T00:33:49.047105+0000","last_fullsized":"2024-02-23T00:33:49.047105+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.650827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.650827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037079+0000","last_change":"2024-02-23T00:32:41.861943+0000","last_active":"2024-02-23T00:33:49.037079+0000","last_peered":"2024-02-23T00:33:49.037079+0000","last_clean":"2024-02-23T00:33:49.037079+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:33:49.037079+0000","last_undegraded":"2024-02-23T00:33:49.037079+0000","last_fullsized":"2024-02-23T00:33:49.037079+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:41.861874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.861874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":257,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042751+0000","last_change":"2024-02-23T00:32:50.705430+0000","last_active":"2024-02-23T00:33:49.042751+0000","last_peered":"2024-02-23T00:33:49.042751+0000","last_clean":"2024-02-23T00:33:49.042751+0000","last_became_active":"2024-02-23T00:31:29.065548+0000","last_became_peered":"2024-02-23T00:31:29.065548+0000","last_unstale":"2024-02-23T00:33:49.042751+0000","last_undegraded":"2024-02-23T00:33:49.042751+0000","last_fullsized":"2024-02-23T00:33:49.042751+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.705318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.705318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":251,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037358+0000","last_change":"2024-02-23T00:32:50.596925+0000","last_active":"2024-02-23T00:33:49.037358+0000","last_peered":"2024-02-23T00:33:49.037358+0000","last_clean":"2024-02-23T00:33:49.037358+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:33:49.037358+0000","last_undegraded":"2024-02-23T00:33:49.037358+0000","last_fullsized":"2024-02-23T00:33:49.037358+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":385,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.046447+0000","last_change":"2024-02-23T00:32:45.366480+0000","last_active":"2024-02-23T00:33:50.046447+0000","last_peered":"2024-02-23T00:33:50.046447+0000","last_clean":"2024-02-23T00:33:50.046447+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:33:50.046447+0000","last_undegraded":"2024-02-23T00:33:50.046447+0000","last_fullsized":"2024-02-23T00:33:50.046447+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:32:45.366416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.366416+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":250,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037253+0000","last_change":"2024-02-23T00:33:36.350954+0000","last_active":"2024-02-23T00:33:49.037253+0000","last_peered":"2024-02-23T00:33:49.037253+0000","last_clean":"2024-02-23T00:33:49.037253+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:33:49.037253+0000","last_undegraded":"2024-02-23T00:33:49.037253+0000","last_fullsized":"2024-02-23T00:33:49.037253+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:36.350888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.350888+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044643+0000","last_change":"2024-02-23T00:33:41.701721+0000","last_active":"2024-02-23T00:33:49.044643+0000","last_peered":"2024-02-23T00:33:49.044643+0000","last_clean":"2024-02-23T00:33:49.044643+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:33:49.044643+0000","last_undegraded":"2024-02-23T00:33:49.044643+0000","last_fullsized":"2024-02-23T00:33:49.044643+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:41.701634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:41.701634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038207+0000","last_change":"2024-02-23T00:32:44.885268+0000","last_active":"2024-02-23T00:33:49.038207+0000","last_peered":"2024-02-23T00:33:49.038207+0000","last_clean":"2024-02-23T00:33:49.038207+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:33:49.038207+0000","last_undegraded":"2024-02-23T00:33:49.038207+0000","last_fullsized":"2024-02-23T00:33:49.038207+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.885205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.885205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":154,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037263+0000","last_change":"2024-02-23T00:32:36.447499+0000","last_active":"2024-02-23T00:33:49.037263+0000","last_peered":"2024-02-23T00:33:49.037263+0000","last_clean":"2024-02-23T00:33:49.037263+0000","last_became_active":"2024-02-23T00:31:29.059516+0000","last_became_peered":"2024-02-23T00:31:29.059516+0000","last_unstale":"2024-02-23T00:33:49.037263+0000","last_undegraded":"2024-02-23T00:33:49.037263+0000","last_fullsized":"2024-02-23T00:33:49.037263+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.447439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.447439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.042002+0000","last_change":"2024-02-23T00:33:46.629281+0000","last_active":"2024-02-23T00:33:50.042002+0000","last_peered":"2024-02-23T00:33:50.042002+0000","last_clean":"2024-02-23T00:33:50.042002+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:33:50.042002+0000","last_undegraded":"2024-02-23T00:33:50.042002+0000","last_fullsized":"2024-02-23T00:33:50.042002+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":273,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045077+0000","last_change":"2024-02-23T00:33:37.175714+0000","last_active":"2024-02-23T00:33:49.045077+0000","last_peered":"2024-02-23T00:33:49.045077+0000","last_clean":"2024-02-23T00:33:49.045077+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:33:49.045077+0000","last_undegraded":"2024-02-23T00:33:49.045077+0000","last_fullsized":"2024-02-23T00:33:49.045077+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:33:37.175650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.175650+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042318+0000","last_change":"2024-02-23T00:30:44.543241+0000","last_active":"2024-02-23T00:33:49.042318+0000","last_peered":"2024-02-23T00:33:49.042318+0000","last_clean":"2024-02-23T00:33:49.042318+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:33:49.042318+0000","last_undegraded":"2024-02-23T00:33:49.042318+0000","last_fullsized":"2024-02-23T00:33:49.042318+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:44.543161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:30:44.543161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.035784+0000","last_change":"2024-02-23T00:32:08.596816+0000","last_active":"2024-02-23T00:33:49.035784+0000","last_peered":"2024-02-23T00:33:49.035784+0000","last_clean":"2024-02-23T00:33:49.035784+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:33:49.035784+0000","last_undegraded":"2024-02-23T00:33:49.035784+0000","last_fullsized":"2024-02-23T00:33:49.035784+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:08.596734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:08.596734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.754939+0000","last_change":"2024-02-23T00:33:49.754910+0000","last_active":"2024-02-23T00:33:49.754939+0000","last_peered":"2024-02-23T00:33:49.754939+0000","last_clean":"2024-02-23T00:33:49.754939+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:33:49.754939+0000","last_undegraded":"2024-02-23T00:33:49.754939+0000","last_fullsized":"2024-02-23T00:33:49.754939+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:49.754830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.754830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037398+0000","last_change":"2024-02-23T00:32:42.815796+0000","last_active":"2024-02-23T00:33:49.037398+0000","last_peered":"2024-02-23T00:33:49.037398+0000","last_clean":"2024-02-23T00:33:49.037398+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:33:49.037398+0000","last_undegraded":"2024-02-23T00:33:49.037398+0000","last_fullsized":"2024-02-23T00:33:49.037398+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.815722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.815722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":320,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044278+0000","last_change":"2024-02-23T00:31:41.036909+0000","last_active":"2024-02-23T00:33:49.044278+0000","last_peered":"2024-02-23T00:33:49.044278+0000","last_clean":"2024-02-23T00:33:49.044278+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:33:49.044278+0000","last_undegraded":"2024-02-23T00:33:49.044278+0000","last_fullsized":"2024-02-23T00:33:49.044278+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.036837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.036837+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":261,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044535+0000","last_change":"2024-02-23T00:33:43.800448+0000","last_active":"2024-02-23T00:33:49.044535+0000","last_peered":"2024-02-23T00:33:49.044535+0000","last_clean":"2024-02-23T00:33:49.044535+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:33:49.044535+0000","last_undegraded":"2024-02-23T00:33:49.044535+0000","last_fullsized":"2024-02-23T00:33:49.044535+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:33:43.800360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.800360+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044670+0000","last_change":"2024-02-23T00:33:32.794356+0000","last_active":"2024-02-23T00:33:49.044670+0000","last_peered":"2024-02-23T00:33:49.044670+0000","last_clean":"2024-02-23T00:33:49.044670+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:33:49.044670+0000","last_undegraded":"2024-02-23T00:33:49.044670+0000","last_fullsized":"2024-02-23T00:33:49.044670+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:32.794268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:32.794268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":193,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037724+0000","last_change":"2024-02-23T00:32:59.434164+0000","last_active":"2024-02-23T00:33:49.037724+0000","last_peered":"2024-02-23T00:33:49.037724+0000","last_clean":"2024-02-23T00:33:49.037724+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:33:49.037724+0000","last_undegraded":"2024-02-23T00:33:49.037724+0000","last_fullsized":"2024-02-23T00:33:49.037724+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.434079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.434079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":292,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037431+0000","last_change":"2024-02-23T00:32:44.578143+0000","last_active":"2024-02-23T00:33:49.037431+0000","last_peered":"2024-02-23T00:33:49.037431+0000","last_clean":"2024-02-23T00:33:49.037431+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:33:49.037431+0000","last_undegraded":"2024-02-23T00:33:49.037431+0000","last_fullsized":"2024-02-23T00:33:49.037431+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:32:44.578068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.578068+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":248,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037312+0000","last_change":"2024-02-23T00:32:11.912580+0000","last_active":"2024-02-23T00:33:49.037312+0000","last_peered":"2024-02-23T00:33:49.037312+0000","last_clean":"2024-02-23T00:33:49.037312+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:33:49.037312+0000","last_undegraded":"2024-02-23T00:33:49.037312+0000","last_fullsized":"2024-02-23T00:33:49.037312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:11.912504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.912504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045043+0000","last_change":"2024-02-23T00:30:36.894317+0000","last_active":"2024-02-23T00:33:49.045043+0000","last_peered":"2024-02-23T00:33:49.045043+0000","last_clean":"2024-02-23T00:33:49.045043+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:33:49.045043+0000","last_undegraded":"2024-02-23T00:33:49.045043+0000","last_fullsized":"2024-02-23T00:33:49.045043+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:36.894220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:36.894220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":247,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038030+0000","last_change":"2024-02-23T00:33:34.777997+0000","last_active":"2024-02-23T00:33:49.038030+0000","last_peered":"2024-02-23T00:33:49.038030+0000","last_clean":"2024-02-23T00:33:49.038030+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:33:49.038030+0000","last_undegraded":"2024-02-23T00:33:49.038030+0000","last_fullsized":"2024-02-23T00:33:49.038030+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.777903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.777903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":190,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045500+0000","last_change":"2024-02-23T00:30:48.691960+0000","last_active":"2024-02-23T00:33:49.045500+0000","last_peered":"2024-02-23T00:33:49.045500+0000","last_clean":"2024-02-23T00:33:49.045500+0000","last_became_active":"2024-02-23T00:30:48.691713+0000","last_became_peered":"2024-02-23T00:30:48.691713+0000","last_unstale":"2024-02-23T00:33:49.045500+0000","last_undegraded":"2024-02-23T00:33:49.045500+0000","last_fullsized":"2024-02-23T00:33:49.045500+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":289,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044918+0000","last_change":"2024-02-23T00:33:28.885956+0000","last_active":"2024-02-23T00:33:49.044918+0000","last_peered":"2024-02-23T00:33:49.044918+0000","last_clean":"2024-02-23T00:33:49.044918+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:33:49.044918+0000","last_undegraded":"2024-02-23T00:33:49.044918+0000","last_fullsized":"2024-02-23T00:33:49.044918+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:33:28.885879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:28.885879+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":246,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043484+0000","last_change":"2024-02-23T00:32:14.243873+0000","last_active":"2024-02-23T00:33:49.043484+0000","last_peered":"2024-02-23T00:33:49.043484+0000","last_clean":"2024-02-23T00:33:49.043484+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:33:49.043484+0000","last_undegraded":"2024-02-23T00:33:49.043484+0000","last_fullsized":"2024-02-23T00:33:49.043484+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.243786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.243786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":235,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037630+0000","last_change":"2024-02-23T00:32:07.275896+0000","last_active":"2024-02-23T00:33:49.037630+0000","last_peered":"2024-02-23T00:33:49.037630+0000","last_clean":"2024-02-23T00:33:49.037630+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:33:49.037630+0000","last_undegraded":"2024-02-23T00:33:49.037630+0000","last_fullsized":"2024-02-23T00:33:49.037630+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:07.275816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:07.275816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.046767+0000","last_change":"2024-02-23T00:32:15.651736+0000","last_active":"2024-02-23T00:33:49.046767+0000","last_peered":"2024-02-23T00:33:49.046767+0000","last_clean":"2024-02-23T00:33:49.046767+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:33:49.046767+0000","last_undegraded":"2024-02-23T00:33:49.046767+0000","last_fullsized":"2024-02-23T00:33:49.046767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:15.651651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:32:15.651651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":193,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.047011+0000","last_change":"2024-02-23T00:32:55.433102+0000","last_active":"2024-02-23T00:33:49.047011+0000","last_peered":"2024-02-23T00:33:49.047011+0000","last_clean":"2024-02-23T00:33:49.047011+0000","last_became_active":"2024-02-23T00:30:48.703232+0000","last_became_peered":"2024-02-23T00:30:48.703232+0000","last_unstale":"2024-02-23T00:33:49.047011+0000","last_undegraded":"2024-02-23T00:33:49.047011+0000","last_fullsized":"2024-02-23T00:33:49.047011+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:55.433031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:55.433031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":286,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045321+0000","last_change":"2024-02-23T00:30:37.899931+0000","last_active":"2024-02-23T00:33:49.045321+0000","last_peered":"2024-02-23T00:33:49.045321+0000","last_clean":"2024-02-23T00:33:49.045321+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:33:49.045321+0000","last_undegraded":"2024-02-23T00:33:49.045321+0000","last_fullsized":"2024-02-23T00:33:49.045321+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:30:37.899845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.899845+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037199+0000","last_change":"2024-02-23T00:32:10.319807+0000","last_active":"2024-02-23T00:33:49.037199+0000","last_peered":"2024-02-23T00:33:49.037199+0000","last_clean":"2024-02-23T00:33:49.037199+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:33:49.037199+0000","last_undegraded":"2024-02-23T00:33:49.037199+0000","last_fullsized":"2024-02-23T00:33:49.037199+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.319744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.319744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":328,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.055075+0000","last_change":"2024-02-23T00:32:56.376450+0000","last_active":"2024-02-23T00:33:50.055075+0000","last_peered":"2024-02-23T00:33:50.055075+0000","last_clean":"2024-02-23T00:33:50.055075+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:33:50.055075+0000","last_undegraded":"2024-02-23T00:33:50.055075+0000","last_fullsized":"2024-02-23T00:33:50.055075+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":247,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037270+0000","last_change":"2024-02-23T00:33:38.521859+0000","last_active":"2024-02-23T00:33:49.037270+0000","last_peered":"2024-02-23T00:33:49.037270+0000","last_clean":"2024-02-23T00:33:49.037270+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:33:49.037270+0000","last_undegraded":"2024-02-23T00:33:49.037270+0000","last_fullsized":"2024-02-23T00:33:49.037270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.521760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":193,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037967+0000","last_change":"2024-02-23T00:32:45.409616+0000","last_active":"2024-02-23T00:33:49.037967+0000","last_peered":"2024-02-23T00:33:49.037967+0000","last_clean":"2024-02-23T00:33:49.037967+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:33:49.037967+0000","last_undegraded":"2024-02-23T00:33:49.037967+0000","last_fullsized":"2024-02-23T00:33:49.037967+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.409548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.409548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"144'20","reported_seq":273,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037739+0000","last_change":"2024-02-23T00:31:35.423292+0000","last_active":"2024-02-23T00:33:49.037739+0000","last_peered":"2024-02-23T00:33:49.037739+0000","last_clean":"2024-02-23T00:33:49.037739+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:33:49.037739+0000","last_undegraded":"2024-02-23T00:33:49.037739+0000","last_fullsized":"2024-02-23T00:33:49.037739+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.423227+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036668+0000","last_change":"2024-02-23T00:32:48.536354+0000","last_active":"2024-02-23T00:33:49.036668+0000","last_peered":"2024-02-23T00:33:49.036668+0000","last_clean":"2024-02-23T00:33:49.036668+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:33:49.036668+0000","last_undegraded":"2024-02-23T00:33:49.036668+0000","last_fullsized":"2024-02-23T00:33:49.036668+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.536283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.536283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'9","reported_seq":332,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.059923+0000","last_change":"2024-02-23T00:32:43.990958+0000","last_active":"2024-02-23T00:33:50.059923+0000","last_peered":"2024-02-23T00:33:50.059923+0000","last_clean":"2024-02-23T00:33:50.059923+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:33:50.059923+0000","last_undegraded":"2024-02-23T00:33:50.059923+0000","last_fullsized":"2024-02-23T00:33:50.059923+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:32:43.990886+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.990886+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044772+0000","last_change":"2024-02-23T00:32:56.678465+0000","last_active":"2024-02-23T00:33:49.044772+0000","last_peered":"2024-02-23T00:33:49.044772+0000","last_clean":"2024-02-23T00:33:49.044772+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:33:49.044772+0000","last_undegraded":"2024-02-23T00:33:49.044772+0000","last_fullsized":"2024-02-23T00:33:49.044772+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.678359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.678359+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":190,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045117+0000","last_change":"2024-02-23T00:30:48.717524+0000","last_active":"2024-02-23T00:33:49.045117+0000","last_peered":"2024-02-23T00:33:49.045117+0000","last_clean":"2024-02-23T00:33:49.045117+0000","last_became_active":"2024-02-23T00:30:48.716104+0000","last_became_peered":"2024-02-23T00:30:48.716104+0000","last_unstale":"2024-02-23T00:33:49.045117+0000","last_undegraded":"2024-02-23T00:33:49.045117+0000","last_fullsized":"2024-02-23T00:33:49.045117+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":259,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044923+0000","last_change":"2024-02-23T00:32:50.448701+0000","last_active":"2024-02-23T00:33:49.044923+0000","last_peered":"2024-02-23T00:33:49.044923+0000","last_clean":"2024-02-23T00:33:49.044923+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:33:49.044923+0000","last_undegraded":"2024-02-23T00:33:49.044923+0000","last_fullsized":"2024-02-23T00:33:49.044923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:32:50.448595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.448595+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":246,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.046418+0000","last_change":"2024-02-23T00:32:12.234058+0000","last_active":"2024-02-23T00:33:49.046418+0000","last_peered":"2024-02-23T00:33:49.046418+0000","last_clean":"2024-02-23T00:33:49.046418+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:33:49.046418+0000","last_undegraded":"2024-02-23T00:33:49.046418+0000","last_fullsized":"2024-02-23T00:33:49.046418+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.233960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.233960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044583+0000","last_change":"2024-02-23T00:32:09.978443+0000","last_active":"2024-02-23T00:33:49.044583+0000","last_peered":"2024-02-23T00:33:49.044583+0000","last_clean":"2024-02-23T00:33:49.044583+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:33:49.044583+0000","last_undegraded":"2024-02-23T00:33:49.044583+0000","last_fullsized":"2024-02-23T00:33:49.044583+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.978377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.978377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":247,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.035688+0000","last_change":"2024-02-23T00:32:53.602946+0000","last_active":"2024-02-23T00:33:49.035688+0000","last_peered":"2024-02-23T00:33:49.035688+0000","last_clean":"2024-02-23T00:33:49.035688+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:33:49.035688+0000","last_undegraded":"2024-02-23T00:33:49.035688+0000","last_fullsized":"2024-02-23T00:33:49.035688+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.602880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.602880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042505+0000","last_change":"2024-02-23T00:33:48.779550+0000","last_active":"2024-02-23T00:33:49.042505+0000","last_peered":"2024-02-23T00:33:49.042505+0000","last_clean":"2024-02-23T00:33:49.042505+0000","last_became_active":"2024-02-23T00:30:48.666975+0000","last_became_peered":"2024-02-23T00:30:48.666975+0000","last_unstale":"2024-02-23T00:33:49.042505+0000","last_undegraded":"2024-02-23T00:33:49.042505+0000","last_fullsized":"2024-02-23T00:33:49.042505+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:48.779472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:48.779472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":286,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043477+0000","last_change":"2024-02-23T00:33:30.498502+0000","last_active":"2024-02-23T00:33:49.043477+0000","last_peered":"2024-02-23T00:33:49.043477+0000","last_clean":"2024-02-23T00:33:49.043477+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:33:49.043477+0000","last_undegraded":"2024-02-23T00:33:49.043477+0000","last_fullsized":"2024-02-23T00:33:49.043477+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:33:30.498425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:30.498425+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043253+0000","last_change":"2024-02-23T00:31:32.874749+0000","last_active":"2024-02-23T00:33:49.043253+0000","last_peered":"2024-02-23T00:33:49.043253+0000","last_clean":"2024-02-23T00:33:49.043253+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:33:49.043253+0000","last_undegraded":"2024-02-23T00:33:49.043253+0000","last_fullsized":"2024-02-23T00:33:49.043253+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:32.874678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:31:32.874678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":241,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044999+0000","last_change":"2024-02-23T00:33:46.152048+0000","last_active":"2024-02-23T00:33:49.044999+0000","last_peered":"2024-02-23T00:33:49.044999+0000","last_clean":"2024-02-23T00:33:49.044999+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:33:49.044999+0000","last_undegraded":"2024-02-23T00:33:49.044999+0000","last_fullsized":"2024-02-23T00:33:49.044999+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.151986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":237,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043992+0000","last_change":"2024-02-23T00:30:47.843046+0000","last_active":"2024-02-23T00:33:49.043992+0000","last_peered":"2024-02-23T00:33:49.043992+0000","last_clean":"2024-02-23T00:33:49.043992+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:33:49.043992+0000","last_undegraded":"2024-02-23T00:33:49.043992+0000","last_fullsized":"2024-02-23T00:33:49.043992+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.842931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":206,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043792+0000","last_change":"2024-02-23T00:33:38.437944+0000","last_active":"2024-02-23T00:33:49.043792+0000","last_peered":"2024-02-23T00:33:49.043792+0000","last_clean":"2024-02-23T00:33:49.043792+0000","last_became_active":"2024-02-23T00:30:48.703281+0000","last_became_peered":"2024-02-23T00:30:48.703281+0000","last_unstale":"2024-02-23T00:33:49.043792+0000","last_undegraded":"2024-02-23T00:33:49.043792+0000","last_fullsized":"2024-02-23T00:33:49.043792+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:33:38.437857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.437857+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":254,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043465+0000","last_change":"2024-02-23T00:32:11.989425+0000","last_active":"2024-02-23T00:33:49.043465+0000","last_peered":"2024-02-23T00:33:49.043465+0000","last_clean":"2024-02-23T00:33:49.043465+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:33:49.043465+0000","last_undegraded":"2024-02-23T00:33:49.043465+0000","last_fullsized":"2024-02-23T00:33:49.043465+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:32:11.989333+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.989333+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037765+0000","last_change":"2024-02-23T00:32:40.343941+0000","last_active":"2024-02-23T00:33:49.037765+0000","last_peered":"2024-02-23T00:33:49.037765+0000","last_clean":"2024-02-23T00:33:49.037765+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:33:49.037765+0000","last_undegraded":"2024-02-23T00:33:49.037765+0000","last_fullsized":"2024-02-23T00:33:49.037765+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.343863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.343863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043393+0000","last_change":"2024-02-23T00:32:47.484343+0000","last_active":"2024-02-23T00:33:49.043393+0000","last_peered":"2024-02-23T00:33:49.043393+0000","last_clean":"2024-02-23T00:33:49.043393+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:33:49.043393+0000","last_undegraded":"2024-02-23T00:33:49.043393+0000","last_fullsized":"2024-02-23T00:33:49.043393+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:47.484285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.484285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045312+0000","last_change":"2024-02-23T00:32:12.721784+0000","last_active":"2024-02-23T00:33:49.045312+0000","last_peered":"2024-02-23T00:33:49.045312+0000","last_clean":"2024-02-23T00:33:49.045312+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:33:49.045312+0000","last_undegraded":"2024-02-23T00:33:49.045312+0000","last_fullsized":"2024-02-23T00:33:49.045312+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.721719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.721719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":192,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045234+0000","last_change":"2024-02-23T00:33:44.152089+0000","last_active":"2024-02-23T00:33:49.045234+0000","last_peered":"2024-02-23T00:33:49.045234+0000","last_clean":"2024-02-23T00:33:49.045234+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:33:49.045234+0000","last_undegraded":"2024-02-23T00:33:49.045234+0000","last_fullsized":"2024-02-23T00:33:49.045234+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.152014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.152014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":282,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044726+0000","last_change":"2024-02-23T00:31:27.176319+0000","last_active":"2024-02-23T00:33:49.044726+0000","last_peered":"2024-02-23T00:33:49.044726+0000","last_clean":"2024-02-23T00:33:49.044726+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:33:49.044726+0000","last_undegraded":"2024-02-23T00:33:49.044726+0000","last_fullsized":"2024-02-23T00:33:49.044726+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:31:27.176199+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:31:27.176199+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":245,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.047146+0000","last_change":"2024-02-23T00:32:44.494080+0000","last_active":"2024-02-23T00:33:49.047146+0000","last_peered":"2024-02-23T00:33:49.047146+0000","last_clean":"2024-02-23T00:33:49.047146+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:33:49.047146+0000","last_undegraded":"2024-02-23T00:33:49.047146+0000","last_fullsized":"2024-02-23T00:33:49.047146+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.494004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.494004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043272+0000","last_change":"2024-02-23T00:32:46.655443+0000","last_active":"2024-02-23T00:33:49.043272+0000","last_peered":"2024-02-23T00:33:49.043272+0000","last_clean":"2024-02-23T00:33:49.043272+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:33:49.043272+0000","last_undegraded":"2024-02-23T00:33:49.043272+0000","last_fullsized":"2024-02-23T00:33:49.043272+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.655378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.655378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044852+0000","last_change":"2024-02-23T00:33:30.840877+0000","last_active":"2024-02-23T00:33:49.044852+0000","last_peered":"2024-02-23T00:33:49.044852+0000","last_clean":"2024-02-23T00:33:49.044852+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:33:49.044852+0000","last_undegraded":"2024-02-23T00:33:49.044852+0000","last_fullsized":"2024-02-23T00:33:49.044852+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:30.840800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:30.840800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":195,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043758+0000","last_change":"2024-02-23T00:33:45.795350+0000","last_active":"2024-02-23T00:33:49.043758+0000","last_peered":"2024-02-23T00:33:49.043758+0000","last_clean":"2024-02-23T00:33:49.043758+0000","last_became_active":"2024-02-23T00:30:48.722602+0000","last_became_peered":"2024-02-23T00:30:48.722602+0000","last_unstale":"2024-02-23T00:33:49.043758+0000","last_undegraded":"2024-02-23T00:33:49.043758+0000","last_fullsized":"2024-02-23T00:33:49.043758+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:45.795278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.795278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"92'22","reported_seq":282,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037624+0000","last_change":"2024-02-23T00:32:43.560670+0000","last_active":"2024-02-23T00:33:49.037624+0000","last_peered":"2024-02-23T00:33:49.037624+0000","last_clean":"2024-02-23T00:33:49.037624+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:33:49.037624+0000","last_undegraded":"2024-02-23T00:33:49.037624+0000","last_fullsized":"2024-02-23T00:33:49.037624+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:32:43.560587+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.560587+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036886+0000","last_change":"2024-02-23T00:31:20.333610+0000","last_active":"2024-02-23T00:33:49.036886+0000","last_peered":"2024-02-23T00:33:49.036886+0000","last_clean":"2024-02-23T00:33:49.036886+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:33:49.036886+0000","last_undegraded":"2024-02-23T00:33:49.036886+0000","last_fullsized":"2024-02-23T00:33:49.036886+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:20.333547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:20.333547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":332,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-02-23T00:33:50.047222+0000","last_change":"2024-02-23T00:31:19.971484+0000","last_active":"2024-02-23T00:33:50.047222+0000","last_peered":"2024-02-23T00:33:50.047222+0000","last_clean":"2024-02-23T00:33:50.047222+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:33:50.047222+0000","last_undegraded":"2024-02-23T00:33:50.047222+0000","last_fullsized":"2024-02-23T00:33:50.047222+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:31:19.971419+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.971419+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.046714+0000","last_change":"2024-02-23T00:32:50.706776+0000","last_active":"2024-02-23T00:33:49.046714+0000","last_peered":"2024-02-23T00:33:49.046714+0000","last_clean":"2024-02-23T00:33:49.046714+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:33:49.046714+0000","last_undegraded":"2024-02-23T00:33:49.046714+0000","last_fullsized":"2024-02-23T00:33:49.046714+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:32:50.706693+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.706693+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":194,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.047470+0000","last_change":"2024-02-23T00:33:27.425684+0000","last_active":"2024-02-23T00:33:49.047470+0000","last_peered":"2024-02-23T00:33:49.047470+0000","last_clean":"2024-02-23T00:33:49.047470+0000","last_became_active":"2024-02-23T00:30:48.745982+0000","last_became_peered":"2024-02-23T00:30:48.745982+0000","last_unstale":"2024-02-23T00:33:49.047470+0000","last_undegraded":"2024-02-23T00:33:49.047470+0000","last_fullsized":"2024-02-23T00:33:49.047470+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:27.425604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:27.425604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":278,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.047388+0000","last_change":"2024-02-23T00:32:59.459127+0000","last_active":"2024-02-23T00:33:49.047388+0000","last_peered":"2024-02-23T00:33:49.047388+0000","last_clean":"2024-02-23T00:33:49.047388+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:33:49.047388+0000","last_undegraded":"2024-02-23T00:33:49.047388+0000","last_fullsized":"2024-02-23T00:33:49.047388+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:32:59.459053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.459053+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":248,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044481+0000","last_change":"2024-02-23T00:32:37.932120+0000","last_active":"2024-02-23T00:33:49.044481+0000","last_peered":"2024-02-23T00:33:49.044481+0000","last_clean":"2024-02-23T00:33:49.044481+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:33:49.044481+0000","last_undegraded":"2024-02-23T00:33:49.044481+0000","last_fullsized":"2024-02-23T00:33:49.044481+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.932055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:49.056184+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.932055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":244,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037702+0000","last_change":"2024-02-23T00:32:35.624529+0000","last_active":"2024-02-23T00:33:49.037702+0000","last_peered":"2024-02-23T00:33:49.037702+0000","last_clean":"2024-02-23T00:33:49.037702+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:33:49.037702+0000","last_undegraded":"2024-02-23T00:33:49.037702+0000","last_fullsized":"2024-02-23T00:33:49.037702+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.624461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.624461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043099+0000","last_change":"2024-02-23T00:32:10.145767+0000","last_active":"2024-02-23T00:33:49.043099+0000","last_peered":"2024-02-23T00:33:49.043099+0000","last_clean":"2024-02-23T00:33:49.043099+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:33:49.043099+0000","last_undegraded":"2024-02-23T00:33:49.043099+0000","last_fullsized":"2024-02-23T00:33:49.043099+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.145685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.145685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":194,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.047377+0000","last_change":"2024-02-23T00:32:58.681718+0000","last_active":"2024-02-23T00:33:49.047377+0000","last_peered":"2024-02-23T00:33:49.047377+0000","last_clean":"2024-02-23T00:33:49.047377+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:33:49.047377+0000","last_undegraded":"2024-02-23T00:33:49.047377+0000","last_fullsized":"2024-02-23T00:33:49.047377+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.681626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.681626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":276,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038232+0000","last_change":"2024-02-23T00:33:28.841007+0000","last_active":"2024-02-23T00:33:49.038232+0000","last_peered":"2024-02-23T00:33:49.038232+0000","last_clean":"2024-02-23T00:33:49.038232+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:33:49.038232+0000","last_undegraded":"2024-02-23T00:33:49.038232+0000","last_fullsized":"2024-02-23T00:33:49.038232+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:33:28.840915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:28.840915+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":251,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044532+0000","last_change":"2024-02-23T00:33:33.425803+0000","last_active":"2024-02-23T00:33:49.044532+0000","last_peered":"2024-02-23T00:33:49.044532+0000","last_clean":"2024-02-23T00:33:49.044532+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:33:49.044532+0000","last_undegraded":"2024-02-23T00:33:49.044532+0000","last_fullsized":"2024-02-23T00:33:49.044532+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.425727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.425727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":238,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043476+0000","last_change":"2024-02-23T00:31:38.887671+0000","last_active":"2024-02-23T00:33:49.043476+0000","last_peered":"2024-02-23T00:33:49.043476+0000","last_clean":"2024-02-23T00:33:49.043476+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:33:49.043476+0000","last_undegraded":"2024-02-23T00:33:49.043476+0000","last_fullsized":"2024-02-23T00:33:49.043476+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.887597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":237,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044087+0000","last_change":"2024-02-23T00:32:12.983101+0000","last_active":"2024-02-23T00:33:49.044087+0000","last_peered":"2024-02-23T00:33:49.044087+0000","last_clean":"2024-02-23T00:33:49.044087+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:33:49.044087+0000","last_undegraded":"2024-02-23T00:33:49.044087+0000","last_fullsized":"2024-02-23T00:33:49.044087+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.983023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.983023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":193,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.037966+0000","last_change":"2024-02-23T00:33:28.419457+0000","last_active":"2024-02-23T00:33:49.037966+0000","last_peered":"2024-02-23T00:33:49.037966+0000","last_clean":"2024-02-23T00:33:49.037966+0000","last_became_active":"2024-02-23T00:30:48.710049+0000","last_became_peered":"2024-02-23T00:30:48.710049+0000","last_unstale":"2024-02-23T00:33:49.037966+0000","last_undegraded":"2024-02-23T00:33:49.037966+0000","last_fullsized":"2024-02-23T00:33:49.037966+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:28.419374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:28.419374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":285,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044914+0000","last_change":"2024-02-23T00:30:29.844070+0000","last_active":"2024-02-23T00:33:49.044914+0000","last_peered":"2024-02-23T00:33:49.044914+0000","last_clean":"2024-02-23T00:33:49.044914+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:33:49.044914+0000","last_undegraded":"2024-02-23T00:33:49.044914+0000","last_fullsized":"2024-02-23T00:33:49.044914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:30:29.843954+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:30:29.843954+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":248,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044491+0000","last_change":"2024-02-23T00:33:33.750868+0000","last_active":"2024-02-23T00:33:49.044491+0000","last_peered":"2024-02-23T00:33:49.044491+0000","last_clean":"2024-02-23T00:33:49.044491+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:33:49.044491+0000","last_undegraded":"2024-02-23T00:33:49.044491+0000","last_fullsized":"2024-02-23T00:33:49.044491+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.750804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.750804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044353+0000","last_change":"2024-02-23T00:31:45.961466+0000","last_active":"2024-02-23T00:33:49.044353+0000","last_peered":"2024-02-23T00:33:49.044353+0000","last_clean":"2024-02-23T00:33:49.044353+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:33:49.044353+0000","last_undegraded":"2024-02-23T00:33:49.044353+0000","last_fullsized":"2024-02-23T00:33:49.044353+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:45.961386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:45.961386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044865+0000","last_change":"2024-02-23T00:32:46.021732+0000","last_active":"2024-02-23T00:33:49.044865+0000","last_peered":"2024-02-23T00:33:49.044865+0000","last_clean":"2024-02-23T00:33:49.044865+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:33:49.044865+0000","last_undegraded":"2024-02-23T00:33:49.044865+0000","last_fullsized":"2024-02-23T00:33:49.044865+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.021662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":193,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038128+0000","last_change":"2024-02-23T00:33:33.382591+0000","last_active":"2024-02-23T00:33:49.038128+0000","last_peered":"2024-02-23T00:33:49.038128+0000","last_clean":"2024-02-23T00:33:49.038128+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:33:49.038128+0000","last_undegraded":"2024-02-23T00:33:49.038128+0000","last_fullsized":"2024-02-23T00:33:49.038128+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.382510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.382510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":292,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043068+0000","last_change":"2024-02-23T00:32:44.870419+0000","last_active":"2024-02-23T00:33:49.043068+0000","last_peered":"2024-02-23T00:33:49.043068+0000","last_clean":"2024-02-23T00:33:49.043068+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:33:49.043068+0000","last_undegraded":"2024-02-23T00:33:49.043068+0000","last_fullsized":"2024-02-23T00:33:49.043068+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:32:44.870335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.870335+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":246,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.046712+0000","last_change":"2024-02-23T00:32:45.663968+0000","last_active":"2024-02-23T00:33:49.046712+0000","last_peered":"2024-02-23T00:33:49.046712+0000","last_clean":"2024-02-23T00:33:49.046712+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:33:49.046712+0000","last_undegraded":"2024-02-23T00:33:49.046712+0000","last_fullsized":"2024-02-23T00:33:49.046712+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.663905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.663905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":240,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.038359+0000","last_change":"2024-02-23T00:33:43.299920+0000","last_active":"2024-02-23T00:33:49.038359+0000","last_peered":"2024-02-23T00:33:49.038359+0000","last_clean":"2024-02-23T00:33:49.038359+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:33:49.038359+0000","last_undegraded":"2024-02-23T00:33:49.038359+0000","last_fullsized":"2024-02-23T00:33:49.038359+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036006+0000","last_change":"2024-02-23T00:32:39.644590+0000","last_active":"2024-02-23T00:33:49.036006+0000","last_peered":"2024-02-23T00:33:49.036006+0000","last_clean":"2024-02-23T00:33:49.036006+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:33:49.036006+0000","last_undegraded":"2024-02-23T00:33:49.036006+0000","last_fullsized":"2024-02-23T00:33:49.036006+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:39.644483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:32:39.644483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":194,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042668+0000","last_change":"2024-02-23T00:32:35.725253+0000","last_active":"2024-02-23T00:33:49.042668+0000","last_peered":"2024-02-23T00:33:49.042668+0000","last_clean":"2024-02-23T00:33:49.042668+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:33:49.042668+0000","last_undegraded":"2024-02-23T00:33:49.042668+0000","last_fullsized":"2024-02-23T00:33:49.042668+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.725182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.725182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":274,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043636+0000","last_change":"2024-02-23T00:32:58.673712+0000","last_active":"2024-02-23T00:33:49.043636+0000","last_peered":"2024-02-23T00:33:49.043636+0000","last_clean":"2024-02-23T00:33:49.043636+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:33:49.043636+0000","last_undegraded":"2024-02-23T00:33:49.043636+0000","last_fullsized":"2024-02-23T00:33:49.043636+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:32:58.673628+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.673628+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":246,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.046886+0000","last_change":"2024-02-23T00:32:36.714218+0000","last_active":"2024-02-23T00:33:49.046886+0000","last_peered":"2024-02-23T00:33:49.046886+0000","last_clean":"2024-02-23T00:33:49.046886+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:33:49.046886+0000","last_undegraded":"2024-02-23T00:33:49.046886+0000","last_fullsized":"2024-02-23T00:33:49.046886+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.714139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.714139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045216+0000","last_change":"2024-02-23T00:32:44.640411+0000","last_active":"2024-02-23T00:33:49.045216+0000","last_peered":"2024-02-23T00:33:49.045216+0000","last_clean":"2024-02-23T00:33:49.045216+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:33:49.045216+0000","last_undegraded":"2024-02-23T00:33:49.045216+0000","last_fullsized":"2024-02-23T00:33:49.045216+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.640331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.640331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":233,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036905+0000","last_change":"2024-02-23T00:31:39.494827+0000","last_active":"2024-02-23T00:33:49.036905+0000","last_peered":"2024-02-23T00:33:49.036905+0000","last_clean":"2024-02-23T00:33:49.036905+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:33:49.036905+0000","last_undegraded":"2024-02-23T00:33:49.036905+0000","last_fullsized":"2024-02-23T00:33:49.036905+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:31:39.494737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":192,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042376+0000","last_change":"2024-02-23T00:33:33.192544+0000","last_active":"2024-02-23T00:33:49.042376+0000","last_peered":"2024-02-23T00:33:49.042376+0000","last_clean":"2024-02-23T00:33:49.042376+0000","last_became_active":"2024-02-23T00:30:48.725162+0000","last_became_peered":"2024-02-23T00:30:48.725162+0000","last_unstale":"2024-02-23T00:33:49.042376+0000","last_undegraded":"2024-02-23T00:33:49.042376+0000","last_fullsized":"2024-02-23T00:33:49.042376+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.192483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.192483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":277,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042201+0000","last_change":"2024-02-23T00:32:10.903259+0000","last_active":"2024-02-23T00:33:49.042201+0000","last_peered":"2024-02-23T00:33:49.042201+0000","last_clean":"2024-02-23T00:33:49.042201+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:33:49.042201+0000","last_undegraded":"2024-02-23T00:33:49.042201+0000","last_fullsized":"2024-02-23T00:33:49.042201+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:32:10.903186+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.903186+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":245,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043991+0000","last_change":"2024-02-23T00:31:22.137283+0000","last_active":"2024-02-23T00:33:49.043991+0000","last_peered":"2024-02-23T00:33:49.043991+0000","last_clean":"2024-02-23T00:33:49.043991+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:33:49.043991+0000","last_undegraded":"2024-02-23T00:33:49.043991+0000","last_fullsized":"2024-02-23T00:33:49.043991+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:22.137166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:22.137166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":236,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042089+0000","last_change":"2024-02-23T00:31:26.920222+0000","last_active":"2024-02-23T00:33:49.042089+0000","last_peered":"2024-02-23T00:33:49.042089+0000","last_clean":"2024-02-23T00:33:49.042089+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:33:49.042089+0000","last_undegraded":"2024-02-23T00:33:49.042089+0000","last_fullsized":"2024-02-23T00:33:49.042089+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:26.920133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:26.920133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043014+0000","last_change":"2024-02-23T00:31:37.065941+0000","last_active":"2024-02-23T00:33:49.043014+0000","last_peered":"2024-02-23T00:33:49.043014+0000","last_clean":"2024-02-23T00:33:49.043014+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:33:49.043014+0000","last_undegraded":"2024-02-23T00:33:49.043014+0000","last_fullsized":"2024-02-23T00:33:49.043014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:31:37.065865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":194,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.045455+0000","last_change":"2024-02-23T00:33:29.354309+0000","last_active":"2024-02-23T00:33:49.045455+0000","last_peered":"2024-02-23T00:33:49.045455+0000","last_clean":"2024-02-23T00:33:49.045455+0000","last_became_active":"2024-02-23T00:30:48.697963+0000","last_became_peered":"2024-02-23T00:30:48.697963+0000","last_unstale":"2024-02-23T00:33:49.045455+0000","last_undegraded":"2024-02-23T00:33:49.045455+0000","last_fullsized":"2024-02-23T00:33:49.045455+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:29.354235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:29.354235+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'50","reported_seq":342,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.576071+0000","last_change":"2024-02-23T00:33:49.576025+0000","last_active":"2024-02-23T00:33:49.576071+0000","last_peered":"2024-02-23T00:33:49.576071+0000","last_clean":"2024-02-23T00:33:49.576071+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:33:49.576071+0000","last_undegraded":"2024-02-23T00:33:49.576071+0000","last_fullsized":"2024-02-23T00:33:49.576071+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'50","last_scrub_stamp":"2024-02-23T00:33:49.575922+0000","last_deep_scrub":"278'50","last_deep_scrub_stamp":"2024-02-23T00:33:49.575922+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.575922+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.036699+0000","last_change":"2024-02-23T00:33:42.579086+0000","last_active":"2024-02-23T00:33:49.036699+0000","last_peered":"2024-02-23T00:33:49.036699+0000","last_clean":"2024-02-23T00:33:49.036699+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:33:49.036699+0000","last_undegraded":"2024-02-23T00:33:49.036699+0000","last_fullsized":"2024-02-23T00:33:49.036699+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.042408+0000","last_change":"2024-02-23T00:31:40.071364+0000","last_active":"2024-02-23T00:33:49.042408+0000","last_peered":"2024-02-23T00:33:49.042408+0000","last_clean":"2024-02-23T00:33:49.042408+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:33:49.042408+0000","last_undegraded":"2024-02-23T00:33:49.042408+0000","last_fullsized":"2024-02-23T00:33:49.042408+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.071294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.071294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":237,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.044274+0000","last_change":"2024-02-23T00:30:46.848126+0000","last_active":"2024-02-23T00:33:49.044274+0000","last_peered":"2024-02-23T00:33:49.044274+0000","last_clean":"2024-02-23T00:33:49.044274+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:33:49.044274+0000","last_undegraded":"2024-02-23T00:33:49.044274+0000","last_fullsized":"2024-02-23T00:33:49.044274+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.848063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.848063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":192,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-02-23T00:33:49.043112+0000","last_change":"2024-02-23T00:32:56.396227+0000","last_active":"2024-02-23T00:33:49.043112+0000","last_peered":"2024-02-23T00:33:49.043112+0000","last_clean":"2024-02-23T00:33:49.043112+0000","last_became_active":"2024-02-23T00:30:48.722569+0000","last_became_peered":"2024-02-23T00:30:48.722569+0000","last_unstale":"2024-02-23T00:33:49.043112+0000","last_undegraded":"2024-02-23T00:33:49.043112+0000","last_fullsized":"2024-02-23T00:33:49.043112+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.396150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.396150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":182,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":1204,"num_objects":121,"num_object_clones":2,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":542,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1474560,"data_stored":3612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1790,"ondisk_log_size":1790,"up":96,"acting":96,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":651,"ondisk_log_size":651,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332178,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314036,"kb_used_data":4516,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431116,"statfs":{"total":95995035648,"available":95673462784,"internally_reserved":0,"allocated":4624384,"data_stored":2618851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":2.0800000000000001,"15min":1.0269999999999999},"min":{"1min":0.26300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.371,"5min":61.048999999999999,"15min":61.048999999999999},"last":0.495},{"interface":"front","average":{"1min":0.52400000000000002,"5min":2.653,"15min":1.2390000000000001},"min":{"1min":0.36799999999999999,"5min":0.214,"15min":0.214},"max":{"1min":1.1639999999999999,"5min":73.870999999999995,"15min":73.870999999999995},"last":0.504}]},{"osd":1,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.024,"15min":0.69399999999999995},"min":{"1min":0.377,"5min":0.217,"15min":0.217},"max":{"1min":1.1200000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.47799999999999998,"5min":1.038,"15min":0.68200000000000005},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.60799999999999998,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.95399999999999996,"15min":0.66900000000000004},"min":{"1min":0.30099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.4119999999999999,"5min":16.603999999999999,"15min":16.603999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.748,"15min":0.59799999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.625,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.56999999999999995}]},{"osd":3,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":1.8759999999999999,"15min":0.97299999999999998},"min":{"1min":0.34300000000000003,"5min":0.311,"15min":0.311},"max":{"1min":1.4419999999999999,"5min":57.729999999999997,"15min":57.729999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.609,"15min":0.89300000000000002},"min":{"1min":0.35699999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.80300000000000005,"5min":27.573,"15min":27.573},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.433,"15min":0.41699999999999998},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.46899999999999997,"15min":0.438},"min":{"1min":0.33700000000000002,"5min":0.25700000000000001,"15min":0.254},"max":{"1min":3.294,"5min":3.294,"15min":3.294},"last":0.41299999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.441,"15min":0.43099999999999999},"min":{"1min":0.32400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.85899999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.442},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47599999999999998,"15min":0.442},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.46600000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.503,"15min":0.47999999999999998},"min":{"1min":0.40999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.497,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.83299999999999996,"5min":1.002,"15min":1.002},"last":0.54900000000000004}]}]},{"osd":6,"up_from":44,"seq":188978561111,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318180,"kb_used_data":4564,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426972,"statfs":{"total":95995035648,"available":95669219328,"internally_reserved":0,"allocated":4673536,"data_stored":2632905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":1.7090000000000001,"15min":0.90100000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.63100000000000001,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.71199999999999997,"15min":0.58399999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.70999999999999996,"5min":9.468,"15min":9.468},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":2.0299999999999998,"15min":1.0129999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.70899999999999996,"5min":59.347999999999999,"15min":59.347999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":2.8730000000000002,"15min":1.2889999999999999},"min":{"1min":0.28299999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.63800000000000001,"5min":87.221000000000004,"15min":87.221000000000004},"last":0.42999999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":1.427,"15min":0.80700000000000005},"min":{"1min":0.32900000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78800000000000003,"5min":25.550999999999998,"15min":25.550999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.45400000000000001,"5min":1.5129999999999999,"15min":0.871},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.60799999999999998,"5min":40.914000000000001,"15min":40.914000000000001},"last":0.51700000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":1.387,"15min":0.78800000000000003},"min":{"1min":0.30199999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76000000000000001,"5min":31.309999999999999,"15min":31.309999999999999},"last":0.5},{"interface":"front","average":{"1min":0.495,"5min":2.9129999999999998,"15min":1.3200000000000001},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72199999999999998,"5min":86.311999999999998,"15min":86.311999999999998},"last":0.59699999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.44800000000000001,"15min":0.42399999999999999},"min":{"1min":0.28599999999999998,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":0.68500000000000005,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.65900000000000003,"15min":0.48899999999999999},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.59499999999999997,"5min":17.718,"15min":17.718},"last":0.57199999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.56399999999999995,"15min":0.47299999999999998},"min":{"1min":0.33300000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.73799999999999999,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.54700000000000004,"15min":0.46100000000000002},"min":{"1min":0.316,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.66500000000000004,"5min":8.1370000000000005,"15min":8.1370000000000005},"last":0.66200000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.83499999999999996,"15min":0.58999999999999997},"min":{"1min":0.36899999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.70099999999999996,"5min":17.881,"15min":17.881},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.80700000000000005,"15min":0.58399999999999996},"min":{"1min":0.39600000000000002,"5min":0.28699999999999998,"15min":0.28199999999999997},"max":{"1min":0.72499999999999998,"5min":17.396999999999998,"15min":17.396999999999998},"last":0.68400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301988,"kb_used_data":4756,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93443164,"statfs":{"total":95995035648,"available":95685799936,"internally_reserved":0,"allocated":4870144,"data_stored":2634718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:32:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.3169999999999999,"15min":0.65700000000000003},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":25.129000000000001,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.28199999999999997},{"interface":"front","average":{"1min":1.7,"5min":0.93999999999999995,"15min":0.54900000000000004},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":26.690000000000001,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.33500000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:33:26 2024","interfaces":[{"interface":"back","average":{"1min":2.3879999999999999,"5min":1.3009999999999999,"15min":0.68100000000000005},"min":{"1min":0.22600000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":35.313000000000002,"5min":35.313000000000002,"15min":35.313000000000002},"last":0.26200000000000001},{"interface":"front","average":{"1min":2.0659999999999998,"5min":0.997,"15min":0.58299999999999996},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":27.908999999999999,"5min":27.908999999999999,"15min":27.908999999999999},"last":0.23100000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":1.1499999999999999,"15min":0.66500000000000004},"min":{"1min":0.255,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.3919999999999999,"5min":21.013999999999999,"15min":21.013999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.111,"15min":0.66100000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":10.382999999999999,"5min":29.234999999999999,"15min":29.234999999999999},"last":0.32600000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:33:04 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":0.92100000000000004,"15min":0.67700000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":16.231000000000002,"5min":16.231000000000002,"15min":16.231000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":1.101,"15min":0.73199999999999998},"min":{"1min":0.34599999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72299999999999998,"5min":29.212,"15min":29.212},"last":0.308}]},{"osd":5,"last update":"Fri Feb 23 00:33:32 2024","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.052,"15min":0.73399999999999999},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":16.294,"5min":16.294,"15min":16.294},"last":0.377},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.91600000000000004,"15min":0.69499999999999995},"min":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":8.4030000000000005,"5min":11.105,"15min":11.105},"last":0.36099999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":1.0860000000000001,"15min":0.73999999999999999},"min":{"1min":0.372,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.68100000000000005,"5min":29.187000000000001,"15min":29.187000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.367,"5min":0.98299999999999998,"15min":0.74099999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":16.369,"5min":16.369,"15min":16.369},"last":0.442}]},{"osd":7,"last update":"Fri Feb 23 00:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":1.163,"15min":0.76900000000000002},"min":{"1min":0.41599999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":16.395,"5min":16.395,"15min":16.395},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.81399999999999995,"15min":0.67900000000000005},"min":{"1min":0.40999999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86399999999999999,"5min":11.135999999999999,"15min":11.135999999999999},"last":0.38400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310024,"kb_used_data":4604,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435128,"statfs":{"total":95995035648,"available":95677571072,"internally_reserved":0,"allocated":4714496,"data_stored":2618751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.82499999999999996,"15min":0.49399999999999999},"min":{"1min":0.215,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.7770000000000001,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":1.1779999999999999,"15min":0.61599999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.5469999999999997,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.25600000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.99099999999999999,"15min":0.59299999999999997},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":3.96,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.754,"15min":0.49299999999999999},"min":{"1min":0.27100000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.2770000000000001,"5min":15.965,"15min":15.965},"last":0.28000000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.0980000000000001,"15min":0.66100000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":10.121,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.309},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.0429999999999999,"15min":0.64400000000000002},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":10.378,"5min":16.006,"15min":16.006},"last":0.34100000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.82999999999999996,"15min":0.64600000000000002},"min":{"1min":0.36899999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.4780000000000002,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.432},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.175,"15min":0.76500000000000001},"min":{"1min":0.41499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.399,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.47399999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.091,"15min":0.72999999999999998},"min":{"1min":0.41999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.5779999999999998,"5min":13.311,"15min":13.311},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.1279999999999999,"15min":0.746},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":5.032,"5min":23.744,"15min":23.744},"last":0.44400000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.3740000000000001,"15min":0.80300000000000005},"min":{"1min":0.40100000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.5510000000000002,"5min":33.082999999999998,"15min":33.082999999999998},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0069999999999999,"15min":0.71199999999999997},"min":{"1min":0.38800000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":9.8040000000000003,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.54000000000000004}]},{"osd":7,"last update":"Fri Feb 23 00:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.2310000000000001,"15min":0.78300000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.373},"max":{"1min":5.3369999999999997,"5min":31.251999999999999,"15min":31.251999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.1559999999999999,"15min":0.75800000000000001},"min":{"1min":0.44500000000000001,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":1.998,"5min":23.785,"15min":23.785},"last":0.51100000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309992,"kb_used_data":4576,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435160,"statfs":{"total":95995035648,"available":95677603840,"internally_reserved":0,"allocated":4685824,"data_stored":2618223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.5840000000000001,"15min":0.751},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":10.878,"5min":56.509,"15min":56.509},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.452,"15min":0.70699999999999996},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1859999999999999,"5min":31.498999999999999,"15min":31.498999999999999},"last":0.41799999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.99299999999999999,"15min":0.56000000000000005},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.9649999999999999,"5min":19.324999999999999,"15min":19.324999999999999},"last":0.443},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.4099999999999999,"15min":0.69899999999999995},"min":{"1min":0.23599999999999999,"5min":0.22900000000000001,"15min":0.216},"max":{"1min":13.209,"5min":35.052,"15min":35.052},"last":0.47299999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:32:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":1.25,"15min":0.69299999999999995},"min":{"1min":0.20999999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.54600000000000004,"5min":43.5,"15min":43.5},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.96499999999999997,"15min":0.61599999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57599999999999996,"5min":27.414000000000001,"15min":27.414000000000001},"last":0.46400000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.76700000000000002,"15min":0.61499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71199999999999997,"5min":14.614000000000001,"15min":14.614000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.65200000000000002,"15min":0.58599999999999997},"min":{"1min":0.316,"5min":0.307,"15min":0.307},"max":{"1min":0.73599999999999999,"5min":4.6550000000000002,"15min":4.6550000000000002},"last":0.83099999999999996}]},{"osd":5,"last update":"Fri Feb 23 00:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.90400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34300000000000003,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":3.2879999999999998,"5min":17.765000000000001,"15min":17.765000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.79100000000000004,"15min":0.63300000000000001},"min":{"1min":0.32900000000000001,"5min":0.317,"15min":0.317},"max":{"1min":2.1259999999999999,"5min":14.589,"15min":14.589},"last":0.86099999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.66700000000000004,"15min":0.58799999999999997},"min":{"1min":0.34399999999999997,"5min":0.317,"15min":0.317},"max":{"1min":0.75800000000000001,"5min":4.6120000000000001,"15min":4.6120000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.038,"15min":0.74299999999999999},"min":{"1min":0.39900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69399999999999995,"5min":20.574999999999999,"15min":20.574999999999999},"last":0.80400000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.9059999999999999,"15min":1.0309999999999999},"min":{"1min":0.38200000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3780000000000001,"5min":66.632000000000005,"15min":66.632000000000005},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":1.226,"15min":0.80600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.157,"5min":19.603999999999999,"15min":19.603999999999999},"last":0.83799999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309948,"kb_used_data":4520,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435204,"statfs":{"total":95995035648,"available":95677648896,"internally_reserved":0,"allocated":4628480,"data_stored":2618481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":1.0329999999999999,"15min":0.59099999999999997},"min":{"1min":0.222,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.42399999999999999,"5min":45.390999999999998,"15min":45.390999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.316,"5min":1.0069999999999999,"15min":0.59799999999999998},"min":{"1min":0.25600000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.39900000000000002,"5min":30.186,"15min":30.186},"last":0.498}]},{"osd":1,"last update":"Fri Feb 23 00:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.61899999999999999,"15min":0.48799999999999999},"min":{"1min":0.20799999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.41399999999999998,"5min":18.039999999999999,"15min":18.039999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":1.0129999999999999,"15min":0.61399999999999999},"min":{"1min":0.26000000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.48299999999999998,"5min":45.441000000000003,"15min":45.441000000000003},"last":0.48899999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.96399999999999997,"15min":0.61299999999999999},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51300000000000001,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.90700000000000003,"15min":0.58599999999999997},"min":{"1min":0.30199999999999999,"5min":0.185,"15min":0.185},"max":{"1min":0.436,"5min":24.183,"15min":24.183},"last":0.44500000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":1.071,"15min":0.74199999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73599999999999999,"5min":24.058,"15min":24.058},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.72899999999999998,"15min":0.63400000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66900000000000004,"5min":8.0359999999999996,"15min":8.0359999999999996},"last":0.73499999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":1.056,"15min":0.73099999999999998},"min":{"1min":0.38,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70299999999999996,"5min":45.265000000000001,"15min":45.265000000000001},"last":0.621},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.91700000000000004,"15min":0.69099999999999995},"min":{"1min":0.40200000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71699999999999997,"5min":24.102,"15min":24.102},"last":0.67300000000000004}]},{"osd":6,"last update":"Fri Feb 23 00:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.745,"15min":0.63400000000000001},"min":{"1min":0.35199999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.093,"5min":8.093,"15min":8.093},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":1.179,"15min":0.79900000000000004},"min":{"1min":0.379,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75,"5min":45.232999999999997,"15min":45.232999999999997},"last":0.745}]},{"osd":7,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.71199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38800000000000001,"5min":0.379,"15min":0.34300000000000003},"max":{"1min":0.71399999999999997,"5min":8.1280000000000001,"15min":8.1280000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":1.0840000000000001,"15min":0.76800000000000002},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78700000000000003,"5min":45.287999999999997,"15min":45.287999999999997},"last":0.76700000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986273,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301844,"kb_used_data":4616,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93443308,"statfs":{"total":95995035648,"available":95685947392,"internally_reserved":0,"allocated":4726784,"data_stored":2618325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.4159999999999999,"15min":0.81599999999999995},"min":{"1min":0.27700000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.0009999999999994,"5min":21.419,"15min":21.419},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.85599999999999998,"5min":2.6200000000000001,"15min":1.226},"min":{"1min":0.33900000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.1710000000000003,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.36199999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.9359999999999999,"15min":0.99299999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":13.044,"5min":41.137999999999998,"15min":41.137999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.93100000000000005,"5min":1.444,"15min":0.82699999999999996},"min":{"1min":0.312,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":8.0310000000000006,"5min":26.210000000000001,"15min":26.210000000000001},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":2.1120000000000001,"15min":1.0609999999999999},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.5249999999999999,"5min":67.253,"15min":67.253},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":1.8540000000000001,"15min":0.96499999999999997},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.7280000000000002,"5min":41.927999999999997,"15min":41.927999999999997},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":2.0219999999999998,"15min":1.0229999999999999},"min":{"1min":0.35699999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":3.6949999999999998,"5min":36.301000000000002,"15min":36.301000000000002},"last":0.442},{"interface":"front","average":{"1min":0.626,"5min":2.2389999999999999,"15min":1.101},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.899,"5min":42.064,"15min":42.064},"last":0.46500000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.86599999999999999,"15min":0.53000000000000003},"min":{"1min":0.28799999999999998,"5min":0.217,"15min":0.217},"max":{"1min":8.5299999999999994,"5min":14.242000000000001,"15min":14.242000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.087,"5min":0.83399999999999996,"15min":0.54100000000000004},"min":{"1min":0.38,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":8.3010000000000002,"5min":14.124000000000001,"15min":14.124000000000001},"last":0.57999999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60899999999999999,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":3.6669999999999998,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.95199999999999996,"15min":0.60299999999999998},"min":{"1min":0.26200000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.5640000000000001,"5min":45.814999999999998,"15min":45.814999999999998},"last":0.45200000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.84899999999999998,"15min":0.59099999999999997},"min":{"1min":0.32800000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.165,"5min":17.178999999999998,"15min":17.178999999999998},"last":0.497},{"interface":"front","average":{"1min":1.0569999999999999,"5min":0.88800000000000001,"15min":0.63200000000000001},"min":{"1min":0.38900000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.9690000000000003,"5min":14.212,"15min":14.212},"last":0.52200000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757340,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314176,"kb_used_data":4652,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430976,"statfs":{"total":95995035648,"available":95673319424,"internally_reserved":0,"allocated":4763648,"data_stored":2629006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.123,"15min":0.70999999999999996},"min":{"1min":0.40899999999999997,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.81499999999999995,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":2.1059999999999999,"15min":1.0269999999999999},"min":{"1min":0.39000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68500000000000005,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.48299999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":1.2909999999999999,"15min":0.749},"min":{"1min":0.38600000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.83799999999999997,"5min":38.838000000000001,"15min":38.838000000000001},"last":0.432},{"interface":"front","average":{"1min":0.51700000000000002,"5min":2.5249999999999999,"15min":1.171},"min":{"1min":0.372,"5min":0.251,"15min":0.251},"max":{"1min":0.71099999999999997,"5min":127.794,"15min":127.794},"last":0.45200000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.657,"15min":0.88600000000000001},"min":{"1min":0.40500000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72399999999999998,"5min":37.442999999999998,"15min":37.442999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.071,"15min":0.69399999999999995},"min":{"1min":0.44900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69599999999999995,"5min":23.297999999999998,"15min":23.297999999999998},"last":0.51600000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":1.238,"15min":0.73299999999999998},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.76900000000000002,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.439},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.629,"15min":0.89100000000000001},"min":{"1min":0.41799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.74199999999999999,"5min":40.865000000000002,"15min":40.865000000000002},"last":0.46100000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.5620000000000001,"15min":0.76700000000000002},"min":{"1min":0.29999999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.70899999999999996,"5min":103.13,"15min":103.13},"last":0.31},{"interface":"front","average":{"1min":0.502,"5min":0.58699999999999997,"15min":0.442},"min":{"1min":0.32000000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.74399999999999999,"5min":12.795999999999999,"15min":12.795999999999999},"last":0.35499999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:33:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.57799999999999996,"15min":0.47199999999999998},"min":{"1min":0.35599999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.69099999999999995,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":1.5640000000000001,"15min":0.80800000000000005},"min":{"1min":0.28399999999999997,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73299999999999998,"5min":103.61499999999999,"15min":103.61499999999999},"last":0.41799999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57799999999999996,"15min":0.51600000000000001},"min":{"1min":0.36299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":7.1820000000000004,"15min":7.1820000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52400000000000002,"15min":0.51100000000000001},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.40000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:33:55.204 INFO:tasks.thrashosds.thrasher:Setting 2.c to [5, 0, 1, 4, 6, 3] 2024-02-23T00:33:55.204 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.c', '5', '0', '1', '4', '6', '3'] 2024-02-23T00:33:55.204 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pg-upmap-items 2.c 5 0 1 4 6 3 2024-02-23T00:33:55.240 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:55.273 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:55.395 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:55.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:55.404 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:55.500 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:33:55.531 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:55.553 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:33:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:55.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:33:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:53.609226+0000 mgr.y (mgr.24437) 294 : cluster [DBG] pgmap v406: 246 pgs: 40 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:54.643137+0000 mon.a (mon.0) 1997 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:55.038285+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-35"}]': finished 2024-02-23T00:33:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:55.038383+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:55.038506+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-02-23T00:33:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:55.040353+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:55.047287+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-35", "mode": "writeback"}]: dispatch 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:55.050057+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-35", "mode": "writeback"}]: dispatch 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:53.609226+0000 mgr.y (mgr.24437) 294 : cluster [DBG] pgmap v406: 246 pgs: 40 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:54.643137+0000 mon.a (mon.0) 1997 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:55.038285+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-35"}]': finished 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:55.038383+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:55.038506+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:55.040353+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:55.047287+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-35", "mode": "writeback"}]: dispatch 2024-02-23T00:33:55.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:55.050057+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-35", "mode": "writeback"}]: dispatch 2024-02-23T00:33:55.656 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:55.684 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:53.609226+0000 mgr.y (mgr.24437) 294 : cluster [DBG] pgmap v406: 246 pgs: 40 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:54.643137+0000 mon.a (mon.0) 1997 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:33:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:55.038285+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-35"}]': finished 2024-02-23T00:33:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:55.038383+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi087-82019-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:33:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:55.038506+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-02-23T00:33:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:55.040353+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:55.047287+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-35", "mode": "writeback"}]: dispatch 2024-02-23T00:33:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:55.050057+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-35", "mode": "writeback"}]: dispatch 2024-02-23T00:33:55.831 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:55.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:55.935 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:55.962 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:33:55.985 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:55.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:56.003 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-23T00:33:56.090 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:56.117 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:56.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:56.299 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:56.034512+0000 mon.a (mon.0) 2003 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.038592+0000 mon.a (mon.0) 2004 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi087-82011-41"}]': finished 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.038658+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-35", "mode": "writeback"}]': finished 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:56.038748+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.040086+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.063882+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.064224+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.064800+0000 mon.c (mon.2) 358 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.065050+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.065533+0000 mon.c (mon.2) 359 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:56.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.065854+0000 mon.a (mon.0) 2010 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.097670+0000 mon.b (mon.1) 248 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:56.097910+0000 mon.a (mon.0) 2011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:56.034512+0000 mon.a (mon.0) 2003 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.038592+0000 mon.a (mon.0) 2004 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi087-82011-41"}]': finished 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.038658+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-35", "mode": "writeback"}]': finished 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:56.038748+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.040086+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.063882+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.064224+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.064800+0000 mon.c (mon.2) 358 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.065050+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.065533+0000 mon.c (mon.2) 359 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:56.445 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.065854+0000 mon.a (mon.0) 2010 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:56.445 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.097670+0000 mon.b (mon.1) 248 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:56.445 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:56.097910+0000 mon.a (mon.0) 2011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:56.677 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:56.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:56.034512+0000 mon.a (mon.0) 2003 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:33:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.038592+0000 mon.a (mon.0) 2004 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi087-82011-41"}]': finished 2024-02-23T00:33:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.038658+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-35", "mode": "writeback"}]': finished 2024-02-23T00:33:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:56.038748+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-02-23T00:33:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.040086+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi087-82011-41"}]: dispatch 2024-02-23T00:33:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.063882+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.064224+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.064800+0000 mon.c (mon.2) 358 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.065050+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.065533+0000 mon.c (mon.2) 359 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.065854+0000 mon.a (mon.0) 2010 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:33:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.097670+0000 mon.b (mon.1) 248 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:56.097910+0000 mon.a (mon.0) 2011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:33:56.780 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:56.808 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:57.014 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:57.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:57.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:57.038 INFO:tasks.workunit.client.0.smithi087.stdout:te 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7264 ms) 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6757 ms) 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6993 ms) 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7011 ms) 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7014 ms) 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7011 ms) 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6993 ms) 2024-02-23T00:33:57.039 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6989 ms) 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7008 ms) 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6991 ms) 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7008 ms) 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7007 ms) 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7006 ms) 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7003 ms) 2024-02-23T00:33:57.040 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-02-23T00:33:57.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6991 ms) 2024-02-23T00:33:57.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112024 ms total) 2024-02-23T00:33:57.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: 2024-02-23T00:33:57.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [----------] Global test environment tear-down 2024-02-23T00:33:57.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [==========] 41 tests from 2 test suites ran. (194985 ms total) 2024-02-23T00:33:57.041 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio: [ PASSED ] 41 tests. 2024-02-23T00:33:57.118 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:33:57.148 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:57.169 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:33:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:57.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:33:57.273 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:57.300 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:57.327 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:57.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:57.431 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:55.610380+0000 mgr.y (mgr.24437) 295 : cluster [DBG] pgmap v409: 270 pgs: 270 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:33:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:57.031935+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi087-82011-41"}]': finished 2024-02-23T00:33:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:57.032011+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:57.032078+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:57.032127+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-02-23T00:33:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:57.035083+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:57.035844+0000 mon.c (mon.2) 360 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi087-82019-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:57.041273+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi087-82019-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:57.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:57.041719+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:57.458 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:55.610380+0000 mgr.y (mgr.24437) 295 : cluster [DBG] pgmap v409: 270 pgs: 270 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:33:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:57.031935+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi087-82011-41"}]': finished 2024-02-23T00:33:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:57.032011+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:57.032078+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:57.032127+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-02-23T00:33:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:57.035083+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:57.035844+0000 mon.c (mon.2) 360 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi087-82019-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:57.041273+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi087-82019-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:57.041719+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:55.610380+0000 mgr.y (mgr.24437) 295 : cluster [DBG] pgmap v409: 270 pgs: 270 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-23T00:33:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:57.031935+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? 172.21.15.87:0/3707730563' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi087-82011-41"}]': finished 2024-02-23T00:33:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:57.032011+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:33:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:57.032078+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:33:57.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:57.032127+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-02-23T00:33:57.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:57.035083+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:57.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:57.035844+0000 mon.c (mon.2) 360 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi087-82019-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:57.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:57.041273+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi087-82019-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:33:57.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:57.041719+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35"}]: dispatch 2024-02-23T00:33:57.906 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:57.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:57.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:58.010 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:33:58.037 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:58.188 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:33:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:58.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:33:58.293 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:58.322 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:58.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.492+0000 7f638f11e700 1 -- 172.21.15.87:0/515973005 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f638806b9e0 msgr2=0x7f6388103fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.492+0000 7f638f11e700 1 --2- 172.21.15.87:0/515973005 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f638806b9e0 0x7f6388103fb0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f6370004870 tx=0x7f6370009cb0).stop 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.492+0000 7f638f11e700 1 -- 172.21.15.87:0/515973005 shutdown_connections 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.492+0000 7f638f11e700 1 --2- 172.21.15.87:0/515973005 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63881044a0 0x7f6388106890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.492+0000 7f638f11e700 1 --2- 172.21.15.87:0/515973005 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f638806b110 0x7f638806b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.492+0000 7f638f11e700 1 --2- 172.21.15.87:0/515973005 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f638806b9e0 0x7f6388103fb0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.492+0000 7f638f11e700 1 -- 172.21.15.87:0/515973005 >> 172.21.15.87:0/515973005 conn(0x7f63880f3800 msgr2=0x7f63880f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.492+0000 7f638f11e700 1 -- 172.21.15.87:0/515973005 shutdown_connections 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.492+0000 7f638f11e700 1 -- 172.21.15.87:0/515973005 wait complete. 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 librados: starting msgr at 2024-02-23T00:33:58.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 librados: starting objecter 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 Processor -- start 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 -- start start 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 librados: setting wanted keys 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 librados: calling monclient init 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f638806b110 0x7f63881103b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f638806b9e0 0x7f63881108a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63881044a0 0x7f638810a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638810aa40 con 0x7f638806b9e0 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638810abb0 con 0x7f63881044a0 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f638f11e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638810ad70 con 0x7f638806b110 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.493+0000 7f6387fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f638806b9e0 0x7f63881108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638ceba700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f638806b110 0x7f63881103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f6387fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f638806b9e0 0x7f63881108a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49794/0 (socket says 172.21.15.87:49794) 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f6387fff700 1 -- 172.21.15.87:0/4051263532 learned_addr learned my addr 172.21.15.87:0/4051263532 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:58.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638d6bb700 1 --2- 172.21.15.87:0/4051263532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63881044a0 0x7f638810a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638ceba700 1 -- 172.21.15.87:0/4051263532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63881044a0 msgr2=0x7f638810a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638ceba700 1 --2- 172.21.15.87:0/4051263532 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63881044a0 0x7f638810a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638ceba700 1 -- 172.21.15.87:0/4051263532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f638806b9e0 msgr2=0x7f63881108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638ceba700 1 --2- 172.21.15.87:0/4051263532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f638806b9e0 0x7f63881108a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638ceba700 1 -- 172.21.15.87:0/4051263532 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6378005040 con 0x7f638806b110 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f6387fff700 1 --2- 172.21.15.87:0/4051263532 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f638806b9e0 0x7f63881108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638ceba700 1 --2- 172.21.15.87:0/4051263532 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f638806b110 0x7f63881103b0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f638000c860 tx=0x7f6380007d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f6385ffb700 10 client.?.objecter ms_handle_connect 0x7f638806b110 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f6385ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638000a500 con 0x7f638806b110 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f638000ab30 con 0x7f638806b110 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380012a30 con 0x7f638806b110 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6370005040 con 0x7f638806b110 2024-02-23T00:33:58.534 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638f11e700 10 client.37385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638810b360 con 0x7f638806b110 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.494+0000 7f638f11e700 1 librados: init done 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.495+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6388109320 con 0x7f638806b110 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.495+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f638000aca0 con 0x7f638806b110 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6385ffb700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6374062c90 0x7f6374065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(287..287 src has 38..287) v4 ==== 8284+0+0 (secure 0 0 0) 0x7f638007cdf0 con 0x7f638806b110 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 osd_map(287..287 src has 38..287) v4 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6385ffb700 3 client.37385.objecter handle_osd_map got epochs [287,287] > 0 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6385ffb700 3 client.37385.objecter handle_osd_map decoding full epoch 287 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6387fff700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6374062c90 0x7f6374065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6385ffb700 20 client.37385.objecter dump_active .. 0 homeless 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v287) v1 ==== 54+0+16 (secure 0 0 0) 0x7f638007f480 con 0x7f638806b110 2024-02-23T00:33:58.535 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6385ffb700 10 client.37385.objecter ms_handle_connect 0x7f6374062c90 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f6387fff700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6374062c90 0x7f6374065150 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 rx=0x7f6370009a00 tx=0x7f637000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 10 client.37385.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 1 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6801/2153191863 conn(0x7f63880fe470 0x7f63880fe850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638810ba30 osd=0 3 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638810ba30 osd=0 3 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638810ba30 osd=0 3 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 1 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f638810ba30 osd=0 3 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 0 1 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638810ba30 osd=0 4 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 10 client.37385.objecter _send_command 1 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63880fed40 con 0x7f63880fe470 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.496+0000 7f638d6bb700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6801/2153191863 conn(0x7f63880fe470 0x7f63880fe850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.536 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.497+0000 7f6385ffb700 10 client.37385.objecter ms_handle_connect 0x7f63880fe470 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.497+0000 7f638d6bb700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6801/2153191863 conn(0x7f63880fe470 0x7f63880fe850 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.497+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f63880fed40 con 0x7f63880fe470 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.497+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.497+0000 7f6385ffb700 10 client.37385.objecter _finish_command 1 = system:0 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.497+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f638810ba30 osd=0 4 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.497+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 0 1 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 10 client.37385.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 2 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638810ba30 osd=0 3 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638810ba30 osd=0 3 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638810ba30 osd=0 3 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 2 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f638810ba30 osd=0 3 2024-02-23T00:33:58.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 0 2 2024-02-23T00:33:58.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638810ba30 osd=0 4 2024-02-23T00:33:58.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 10 client.37385.objecter _send_command 2 2024-02-23T00:33:58.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.512+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f63880fef30 con 0x7f63880fe470 2024-02-23T00:33:58.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f63880fef30 con 0x7f63880fe470 2024-02-23T00:33:58.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:58.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f6385ffb700 10 client.37385.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f638810ba30 osd=0 4 2024-02-23T00:33:58.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 0 2 2024-02-23T00:33:58.538 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:33:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 10 client.37385.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 3 2024-02-23T00:33:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6805/685354171 conn(0x7f6388028540 0x7f638811b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f63880f5fa0 osd=1 3 2024-02-23T00:33:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f63880f5fa0 osd=1 3 2024-02-23T00:33:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f63880f5fa0 osd=1 3 2024-02-23T00:33:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 3 2024-02-23T00:33:58.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f63880f5fa0 osd=1 3 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 1 3 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f63880f5fa0 osd=1 4 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 10 client.37385.objecter _send_command 3 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.513+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63880f6110 con 0x7f6388028540 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.514+0000 7f638ceba700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6805/685354171 conn(0x7f6388028540 0x7f638811b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.514+0000 7f638ceba700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6805/685354171 conn(0x7f6388028540 0x7f638811b260 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.514+0000 7f6385ffb700 10 client.37385.objecter ms_handle_connect 0x7f6388028540 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.514+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f63880f6110 con 0x7f6388028540 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.514+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.514+0000 7f6385ffb700 10 client.37385.objecter _finish_command 3 = system:0 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.514+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f63880f5fa0 osd=1 4 2024-02-23T00:33:58.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.514+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 1 3 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 10 client.37385.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 4 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f63880f5fa0 osd=1 3 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f63880f5fa0 osd=1 3 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f63880f5fa0 osd=1 3 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 4 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f63880f5fa0 osd=1 3 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 1 4 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f63880f5fa0 osd=1 4 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 10 client.37385.objecter _send_command 4 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.529+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f638805a040 con 0x7f6388028540 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.530+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f638805a040 con 0x7f6388028540 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.530+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:58.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.530+0000 7f6385ffb700 10 client.37385.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.530+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f63880f5fa0 osd=1 4 2024-02-23T00:33:58.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.530+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 1 4 2024-02-23T00:33:58.542 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 10 client.37385.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 5 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6809/3605394228 conn(0x7f63880594d0 0x7f638811e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388059360 osd=2 3 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388059360 osd=2 3 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388059360 osd=2 3 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 5 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f6388059360 osd=2 3 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 2 5 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388059360 osd=2 4 2024-02-23T00:33:58.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 10 client.37385.objecter _send_command 5 2024-02-23T00:33:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6388028180 con 0x7f63880594d0 2024-02-23T00:33:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f6387fff700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6809/3605394228 conn(0x7f63880594d0 0x7f638811e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f6385ffb700 10 client.37385.objecter ms_handle_connect 0x7f63880594d0 2024-02-23T00:33:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.531+0000 7f6387fff700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6809/3605394228 conn(0x7f63880594d0 0x7f638811e8e0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.532+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6388028180 con 0x7f63880594d0 2024-02-23T00:33:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.532+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:33:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.532+0000 7f6385ffb700 10 client.37385.objecter _finish_command 5 = system:0 2024-02-23T00:33:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.532+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f6388059360 osd=2 4 2024-02-23T00:33:58.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.532+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 2 5 2024-02-23T00:33:58.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 10 client.37385.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:58.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 6 2024-02-23T00:33:58.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388059360 osd=2 3 2024-02-23T00:33:58.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388059360 osd=2 3 2024-02-23T00:33:58.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388059360 osd=2 3 2024-02-23T00:33:58.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 6 2024-02-23T00:33:58.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f6388059360 osd=2 3 2024-02-23T00:33:58.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 2 6 2024-02-23T00:33:58.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388059360 osd=2 4 2024-02-23T00:33:58.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 10 client.37385.objecter _send_command 6 2024-02-23T00:33:58.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.546+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6388027600 con 0x7f63880594d0 2024-02-23T00:33:58.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.547+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6388027600 con 0x7f63880594d0 2024-02-23T00:33:58.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.547+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:58.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.547+0000 7f6385ffb700 10 client.37385.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.547+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f6388059360 osd=2 4 2024-02-23T00:33:58.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.547+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 2 6 2024-02-23T00:33:58.550 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:33:58.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 10 client.37385.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:58.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 7 2024-02-23T00:33:58.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6813/128059790 conn(0x7f638811c430 0x7f6388121f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388027a10 osd=3 3 2024-02-23T00:33:58.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388027a10 osd=3 3 2024-02-23T00:33:58.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388027a10 osd=3 3 2024-02-23T00:33:58.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 7 2024-02-23T00:33:58.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f6388027a10 osd=3 3 2024-02-23T00:33:58.551 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 3 7 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388027a10 osd=3 4 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 10 client.37385.objecter _send_command 7 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6388124bb0 con 0x7f638811c430 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638d6bb700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6813/128059790 conn(0x7f638811c430 0x7f6388121f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f6385ffb700 10 client.37385.objecter ms_handle_connect 0x7f638811c430 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.548+0000 7f638d6bb700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6813/128059790 conn(0x7f638811c430 0x7f6388121f60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.549+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6388124bb0 con 0x7f638811c430 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.549+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.549+0000 7f6385ffb700 10 client.37385.objecter _finish_command 7 = system:0 2024-02-23T00:33:58.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.549+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f6388027a10 osd=3 4 2024-02-23T00:33:58.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.549+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 3 7 2024-02-23T00:33:58.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 10 client.37385.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:58.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 8 2024-02-23T00:33:58.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388027a10 osd=3 3 2024-02-23T00:33:58.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388027a10 osd=3 3 2024-02-23T00:33:58.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388027a10 osd=3 3 2024-02-23T00:33:58.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 8 2024-02-23T00:33:58.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f6388027a10 osd=3 3 2024-02-23T00:33:58.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 3 8 2024-02-23T00:33:58.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388027a10 osd=3 4 2024-02-23T00:33:58.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 10 client.37385.objecter _send_command 8 2024-02-23T00:33:58.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.563+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f638811b990 con 0x7f638811c430 2024-02-23T00:33:58.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f638811b990 con 0x7f638811c430 2024-02-23T00:33:58.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:58.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f6385ffb700 10 client.37385.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f6388027a10 osd=3 4 2024-02-23T00:33:58.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 3 8 2024-02-23T00:33:58.567 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:33:58.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 10 client.37385.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:58.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 9 2024-02-23T00:33:58.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6800/4175844757 conn(0x7f638811bb30 0x7f63881235d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638811b990 osd=4 3 2024-02-23T00:33:58.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811b990 osd=4 3 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638811b990 osd=4 3 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 9 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f638811b990 osd=4 3 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 4 9 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811b990 osd=4 4 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 10 client.37385.objecter _send_command 9 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.564+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f638811fc50 con 0x7f638811bb30 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.565+0000 7f638ceba700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6800/4175844757 conn(0x7f638811bb30 0x7f63881235d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.565+0000 7f638ceba700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6800/4175844757 conn(0x7f638811bb30 0x7f63881235d0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.565+0000 7f6385ffb700 10 client.37385.objecter ms_handle_connect 0x7f638811bb30 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.566+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f638811fc50 con 0x7f638811bb30 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.566+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.566+0000 7f6385ffb700 10 client.37385.objecter _finish_command 9 = system:0 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.566+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f638811b990 osd=4 4 2024-02-23T00:33:58.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.566+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 4 9 2024-02-23T00:33:58.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:57.610943+0000 mgr.y (mgr.24437) 296 : cluster [DBG] pgmap v412: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:58.038680+0000 mon.a (mon.0) 2018 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:58.284958+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35"}]': finished 2024-02-23T00:33:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:58.285066+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-02-23T00:33:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:57.610943+0000 mgr.y (mgr.24437) 296 : cluster [DBG] pgmap v412: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:58.038680+0000 mon.a (mon.0) 2018 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:58.284958+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35"}]': finished 2024-02-23T00:33:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:58.285066+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-02-23T00:33:58.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 10 client.37385.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 10 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638811b990 osd=4 3 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811b990 osd=4 3 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638811b990 osd=4 3 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 10 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f638811b990 osd=4 3 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 4 10 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811b990 osd=4 4 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 10 client.37385.objecter _send_command 10 2024-02-23T00:33:58.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.581+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f638811fc50 con 0x7f638811bb30 2024-02-23T00:33:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f638811fc50 con 0x7f638811bb30 2024-02-23T00:33:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f6385ffb700 10 client.37385.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f638811b990 osd=4 4 2024-02-23T00:33:58.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 4 10 2024-02-23T00:33:58.585 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:33:58.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 10 client.37385.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:58.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 11 2024-02-23T00:33:58.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6388123fd0 0x7f6388124430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638811fc50 osd=5 3 2024-02-23T00:33:58.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811fc50 osd=5 3 2024-02-23T00:33:58.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638811fc50 osd=5 3 2024-02-23T00:33:58.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 11 2024-02-23T00:33:58.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f638811fc50 osd=5 3 2024-02-23T00:33:58.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 5 11 2024-02-23T00:33:58.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811fc50 osd=5 4 2024-02-23T00:33:58.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 10 client.37385.objecter _send_command 11 2024-02-23T00:33:58.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6388124a10 con 0x7f6388123fd0 2024-02-23T00:33:58.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.583+0000 7f6387fff700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6388123fd0 0x7f6388124430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.584+0000 7f6385ffb700 10 client.37385.objecter ms_handle_connect 0x7f6388123fd0 2024-02-23T00:33:58.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.584+0000 7f6387fff700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6388123fd0 0x7f6388124430 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.585+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6388124a10 con 0x7f6388123fd0 2024-02-23T00:33:58.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.585+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:33:58.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.585+0000 7f6385ffb700 10 client.37385.objecter _finish_command 11 = system:0 2024-02-23T00:33:58.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.585+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f638811fc50 osd=5 4 2024-02-23T00:33:58.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.585+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 5 11 2024-02-23T00:33:58.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 10 client.37385.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:58.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 12 2024-02-23T00:33:58.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638811fc50 osd=5 3 2024-02-23T00:33:58.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811fc50 osd=5 3 2024-02-23T00:33:58.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638811fc50 osd=5 3 2024-02-23T00:33:58.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 12 2024-02-23T00:33:58.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f638811fc50 osd=5 3 2024-02-23T00:33:58.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 5 12 2024-02-23T00:33:58.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811fc50 osd=5 4 2024-02-23T00:33:58.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 10 client.37385.objecter _send_command 12 2024-02-23T00:33:58.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.599+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6388124a10 con 0x7f6388123fd0 2024-02-23T00:33:58.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6388124a10 con 0x7f6388123fd0 2024-02-23T00:33:58.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:58.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f6385ffb700 10 client.37385.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f638811fc50 osd=5 4 2024-02-23T00:33:58.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 5 12 2024-02-23T00:33:58.604 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:33:58.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 10 client.37385.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:58.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 13 2024-02-23T00:33:58.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6808/2509408498 conn(0x7f63881292d0 0x7f6388129710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388124a10 osd=6 3 2024-02-23T00:33:58.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388124a10 osd=6 3 2024-02-23T00:33:58.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388124a10 osd=6 3 2024-02-23T00:33:58.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 13 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f6388124a10 osd=6 3 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 6 13 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388124a10 osd=6 4 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 10 client.37385.objecter _send_command 13 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f638812b390 con 0x7f63881292d0 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.601+0000 7f638d6bb700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6808/2509408498 conn(0x7f63881292d0 0x7f6388129710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.602+0000 7f6385ffb700 10 client.37385.objecter ms_handle_connect 0x7f63881292d0 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.602+0000 7f638d6bb700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6808/2509408498 conn(0x7f63881292d0 0x7f6388129710 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.602+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f638812b390 con 0x7f63881292d0 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.602+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.602+0000 7f6385ffb700 10 client.37385.objecter _finish_command 13 = system:0 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.602+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f6388124a10 osd=6 4 2024-02-23T00:33:58.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.602+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 6 13 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 10 client.37385.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 14 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388124a10 osd=6 3 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388124a10 osd=6 3 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f6388124a10 osd=6 3 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 14 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f6388124a10 osd=6 3 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 6 14 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388124a10 osd=6 4 2024-02-23T00:33:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 10 client.37385.objecter _send_command 14 2024-02-23T00:33:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.617+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f638812b390 con 0x7f63881292d0 2024-02-23T00:33:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f638812b390 con 0x7f63881292d0 2024-02-23T00:33:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f6385ffb700 10 client.37385.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f6388124a10 osd=6 4 2024-02-23T00:33:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 6 14 2024-02-23T00:33:58.621 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:33:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 10 client.37385.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:33:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 15 2024-02-23T00:33:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6812/373787000 conn(0x7f638812dc10 0x7f638812e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638812b390 osd=7 3 2024-02-23T00:33:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638812b390 osd=7 3 2024-02-23T00:33:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638812b390 osd=7 3 2024-02-23T00:33:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 15 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f638812b390 osd=7 3 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 7 15 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638812b390 osd=7 4 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 10 client.37385.objecter _send_command 15 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63881319a0 con 0x7f638812dc10 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638ceba700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6812/373787000 conn(0x7f638812dc10 0x7f638812e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f6385ffb700 10 client.37385.objecter ms_handle_connect 0x7f638812dc10 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.619+0000 7f638ceba700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6812/373787000 conn(0x7f638812dc10 0x7f638812e030 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.620+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f63881319a0 con 0x7f638812dc10 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.620+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:33:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.620+0000 7f6385ffb700 10 client.37385.objecter _finish_command 15 = system:0 2024-02-23T00:33:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.620+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f638812b390 osd=7 4 2024-02-23T00:33:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.620+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 7 15 2024-02-23T00:33:58.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 10 client.37385.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:33:58.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign -1 16 2024-02-23T00:33:58.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638812b390 osd=7 3 2024-02-23T00:33:58.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638812b390 osd=7 3 2024-02-23T00:33:58.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 20 client.37385.objecter _get_session s=0x7f638812b390 osd=7 3 2024-02-23T00:33:58.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 15 client.37385.objecter _session_command_op_remove -1 16 2024-02-23T00:33:58.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 20 client.37385.objecter get_session s=0x7f638812b390 osd=7 3 2024-02-23T00:33:58.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 15 client.37385.objecter _session_command_op_assign 7 16 2024-02-23T00:33:58.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638812b390 osd=7 4 2024-02-23T00:33:58.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 10 client.37385.objecter _send_command 16 2024-02-23T00:33:58.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.634+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6388129cf0 con 0x7f638812dc10 2024-02-23T00:33:58.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.636+0000 7f6385ffb700 1 -- 172.21.15.87:0/4051263532 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6388129cf0 con 0x7f638812dc10 2024-02-23T00:33:58.639 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:33:58.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.636+0000 7f6385ffb700 10 client.37385.objecter ms_dispatch 0x7f6388001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:33:58.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.636+0000 7f6385ffb700 10 client.37385.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.636+0000 7f6385ffb700 20 client.37385.objecter put_session s=0x7f638812b390 osd=7 4 2024-02-23T00:33:58.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.636+0000 7f6385ffb700 15 client.37385.objecter _session_command_op_remove 7 16 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 librados: watch_flush enter 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 librados: watch_flush exit 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 client.37385.objecter close_session for osd.0 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6801/2153191863 conn(0x7f63880fe470 msgr2=0x7f63880fe850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6801/2153191863 conn(0x7f63880fe470 0x7f63880fe850 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638810ba30 osd=0 1 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 client.37385.objecter close_session for osd.1 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6805/685354171 conn(0x7f6388028540 msgr2=0x7f638811b260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6805/685354171 conn(0x7f6388028540 0x7f638811b260 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f63880f5fa0 osd=1 1 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 client.37385.objecter close_session for osd.2 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6809/3605394228 conn(0x7f63880594d0 msgr2=0x7f638811e8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6809/3605394228 conn(0x7f63880594d0 0x7f638811e8e0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388059360 osd=2 1 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 client.37385.objecter close_session for osd.3 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6813/128059790 conn(0x7f638811c430 msgr2=0x7f6388121f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638d6bb700 1 -- 172.21.15.87:0/4051263532 reap_dead start 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6813/128059790 conn(0x7f638811c430 0x7f6388121f60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388027a10 osd=3 1 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 client.37385.objecter close_session for osd.4 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6800/4175844757 conn(0x7f638811bb30 msgr2=0x7f63881235d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6800/4175844757 conn(0x7f638811bb30 0x7f63881235d0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811b990 osd=4 1 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 client.37385.objecter close_session for osd.5 2024-02-23T00:33:58.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6388123fd0 msgr2=0x7f6388124430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6388123fd0 0x7f6388124430 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638811fc50 osd=5 1 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 client.37385.objecter close_session for osd.6 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6808/2509408498 conn(0x7f63881292d0 msgr2=0x7f6388129710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6808/2509408498 conn(0x7f63881292d0 0x7f6388129710 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f6388124a10 osd=6 1 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 client.37385.objecter close_session for osd.7 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6812/373787000 conn(0x7f638812dc10 msgr2=0x7f638812e030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.171:6812/373787000 conn(0x7f638812dc10 0x7f638812e030 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 20 client.37385.objecter put_session s=0x7f638812b390 osd=7 1 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 20 client.37385.objecter shutdown clearing up homeless session... 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 10 client.37385.objecter successfully canceled tick 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638d6bb700 1 -- 172.21.15.87:0/4051263532 reap_dead start 2024-02-23T00:33:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6374062c90 msgr2=0x7f6374065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6374062c90 0x7f6374065150 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 rx=0x7f6370009a00 tx=0x7f637000b040).stop 2024-02-23T00:33:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f638806b110 msgr2=0x7f63881103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f638806b110 0x7f63881103b0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f638000c860 tx=0x7f6380007d50).stop 2024-02-23T00:33:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 shutdown_connections 2024-02-23T00:33:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6374062c90 0x7f6374065150 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 --2- 172.21.15.87:0/4051263532 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f638806b110 0x7f63881103b0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 >> 172.21.15.87:0/4051263532 conn(0x7f63880f3800 msgr2=0x7f63880f4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 shutdown_connections 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 -- 172.21.15.87:0/4051263532 wait complete. 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:58.637+0000 7f638f11e700 1 librados: shutdown 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:33:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:57.610943+0000 mgr.y (mgr.24437) 296 : cluster [DBG] pgmap v412: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:33:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:58.038680+0000 mon.a (mon.0) 2018 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:33:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:58.284958+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-35"}]': finished 2024-02-23T00:33:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:58.285066+0000 mon.a (mon.0) 2020 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-02-23T00:33:59.171 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:59.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:59.276 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:33:59.308 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:59.329 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:33:59.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:59.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:33:59.433 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:33:59.449 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:58.497379+0000 mon.c (mon.2) 361 : audit [DBG] from='client.? 172.21.15.87:0/4051263532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:59.450 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:59.044682+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi087-82019-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]': finished 2024-02-23T00:33:59.450 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:33:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:59.044797+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-02-23T00:33:59.450 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:58.497379+0000 mon.c (mon.2) 361 : audit [DBG] from='client.? 172.21.15.87:0/4051263532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:59.450 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:59.044682+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi087-82019-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]': finished 2024-02-23T00:33:59.450 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:33:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:59.044797+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-02-23T00:33:59.470 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:59.489 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:33:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:59.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:33:59.593 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:33:59.623 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:58.497379+0000 mon.c (mon.2) 361 : audit [DBG] from='client.? 172.21.15.87:0/4051263532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:33:59.623 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:59.044682+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi087-82019-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]': finished 2024-02-23T00:33:59.623 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:33:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:59.044797+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-02-23T00:33:59.625 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:59.646 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:33:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:59.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:33:59.752 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/2269276219 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 msgr2=0x7f8ea0103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/2269276219 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 0x7f8ea0103a20 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f8e98009be0 tx=0x7f8e98004d10).stop 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/2269276219 shutdown_connections 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/2269276219 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 0x7f8ea0103a20 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/2269276219 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ea00fcbb0 0x7f8ea00fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/2269276219 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ea00fd510 0x7f8ea00f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/2269276219 >> 172.21.15.87:0/2269276219 conn(0x7f8ea00f2820 msgr2=0x7f8ea00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/2269276219 shutdown_connections 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/2269276219 wait complete. 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 librados: starting msgr at 2024-02-23T00:33:59.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 librados: starting objecter 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 Processor -- start 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 -- start start 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 librados: setting wanted keys 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.712+0000 7f8ea7dbd700 1 librados: calling monclient init 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 0x7f8ea01091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ea00fcbb0 0x7f8ea01096d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ea00fd510 0x7f8ea0109c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea01143d0 con 0x7f8ea00fd510 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea0114540 con 0x7f8ea00fcbb0 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea0114700 con 0x7f8ea00f7400 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5b59700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 0x7f8ea01091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5b59700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 0x7f8ea01091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33930/0 (socket says 172.21.15.87:33930) 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5b59700 1 -- 172.21.15.87:0/61304812 learned_addr learned my addr 172.21.15.87:0/61304812 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:33:59.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea635a700 1 --2- 172.21.15.87:0/61304812 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ea00fd510 0x7f8ea0109c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5358700 1 --2- 172.21.15.87:0/61304812 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ea00fcbb0 0x7f8ea01096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5b59700 1 -- 172.21.15.87:0/61304812 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ea00fcbb0 msgr2=0x7f8ea01096d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5b59700 1 --2- 172.21.15.87:0/61304812 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ea00fcbb0 0x7f8ea01096d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5b59700 1 -- 172.21.15.87:0/61304812 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ea00fd510 msgr2=0x7f8ea0109c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5b59700 1 --2- 172.21.15.87:0/61304812 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ea00fd510 0x7f8ea0109c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5b59700 1 -- 172.21.15.87:0/61304812 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c005040 con 0x7f8ea00f7400 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea5b59700 1 --2- 172.21.15.87:0/61304812 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 0x7f8ea01091e0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 rx=0x7f8e9000b530 tx=0x7f8e900090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8e96ffd700 10 client.?.objecter ms_handle_connect 0x7f8ea00f7400 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8e96ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8e96ffd700 1 -- 172.21.15.87:0/61304812 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90002320 con 0x7f8ea00f7400 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8e96ffd700 1 -- 172.21.15.87:0/61304812 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e90002950 con 0x7f8ea00f7400 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e980097c0 con 0x7f8ea00f7400 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8e96ffd700 1 -- 172.21.15.87:0/61304812 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90012950 con 0x7f8ea00f7400 2024-02-23T00:33:59.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 10 client.37391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea0114cc0 con 0x7f8ea00f7400 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.713+0000 7f8ea7dbd700 1 librados: init done 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.714+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea005fa60 con 0x7f8ea00f7400 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.715+0000 7f8e96ffd700 1 -- 172.21.15.87:0/61304812 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f8e90002480 con 0x7f8ea00f7400 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.715+0000 7f8e96ffd700 1 --2- 172.21.15.87:0/61304812 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e8c062c90 0x7f8e8c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.715+0000 7f8ea5358700 1 --2- 172.21.15.87:0/61304812 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e8c062c90 0x7f8e8c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.715+0000 7f8e96ffd700 1 -- 172.21.15.87:0/61304812 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(288..288 src has 38..288) v4 ==== 8330+0+0 (secure 0 0 0) 0x7f8e9007d3b0 con 0x7f8ea00f7400 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.715+0000 7f8e96ffd700 10 client.37391.objecter ms_dispatch 0x7f8ea0001550 osd_map(288..288 src has 38..288) v4 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.715+0000 7f8e96ffd700 3 client.37391.objecter handle_osd_map got epochs [288,288] > 0 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.715+0000 7f8e96ffd700 3 client.37391.objecter handle_osd_map decoding full epoch 288 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.715+0000 7f8e96ffd700 20 client.37391.objecter dump_active .. 0 homeless 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.716+0000 7f8e96ffd700 10 client.37391.objecter ms_handle_connect 0x7f8e8c062c90 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.716+0000 7f8ea5358700 1 --2- 172.21.15.87:0/61304812 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e8c062c90 0x7f8e8c065150 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f8ea0061720 tx=0x7f8e9c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:33:59.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.719+0000 7f8e96ffd700 1 -- 172.21.15.87:0/61304812 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8e90050770 con 0x7f8ea00f7400 2024-02-23T00:33:59.779 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:33:59.797 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:33:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:33:59.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:33:59.901 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:33:59.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:33:59.899+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f8ea0061460 con 0x7f8ea00f7400 2024-02-23T00:33:59.927 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:00.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.055+0000 7f8e96ffd700 1 -- 172.21.15.87:0/61304812 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v289) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8e90050360 con 0x7f8ea00f7400 2024-02-23T00:34:00.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.058+0000 7f8ea7dbd700 10 librados: watch_flush enter 2024-02-23T00:34:00.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 10 librados: watch_flush exit 2024-02-23T00:34:00.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 20 client.37391.objecter shutdown clearing up homeless session... 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 10 client.37391.objecter successfully canceled tick 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e8c062c90 msgr2=0x7f8e8c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/61304812 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e8c062c90 0x7f8e8c065150 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 rx=0x7f8ea0061720 tx=0x7f8e9c00b040).stop 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 msgr2=0x7f8ea01091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/61304812 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 0x7f8ea01091e0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 rx=0x7f8e9000b530 tx=0x7f8e900090f0).stop 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 shutdown_connections 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/61304812 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8e8c062c90 0x7f8e8c065150 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/61304812 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8ea00fd510 0x7f8ea0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/61304812 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8ea00f7400 0x7f8ea01091e0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 --2- 172.21.15.87:0/61304812 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8ea00fcbb0 0x7f8ea01096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 >> 172.21.15.87:0/61304812 conn(0x7f8ea00f2820 msgr2=0x7f8ea00f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 shutdown_connections 2024-02-23T00:34:00.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 -- 172.21.15.87:0/61304812 wait complete. 2024-02-23T00:34:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:00.059+0000 7f8ea7dbd700 1 librados: shutdown 2024-02-23T00:34:00.063 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:34:00.334 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:00.335 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:00.576 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:00.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:00.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:00.680 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:33:59.611385+0000 mgr.y (mgr.24437) 297 : cluster [DBG] pgmap v415: 214 pgs: 8 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:59.900799+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.87:0/61304812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:33:59.901106+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:00.041867+0000 mon.a (mon.0) 2024 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:00.049492+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:34:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:00.049569+0000 mon.a (mon.0) 2026 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-02-23T00:34:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:00.061143+0000 mon.b (mon.1) 250 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:00.061786+0000 mon.a (mon.0) 2027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:00.715 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:00.737 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:00.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:00.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:33:59.611385+0000 mgr.y (mgr.24437) 297 : cluster [DBG] pgmap v415: 214 pgs: 8 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:00.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:59.900799+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.87:0/61304812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:00.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:33:59.901106+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:00.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:00.041867+0000 mon.a (mon.0) 2024 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:00.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:00.049492+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:34:00.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:00.049569+0000 mon.a (mon.0) 2026 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-02-23T00:34:00.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:00.061143+0000 mon.b (mon.1) 250 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:00.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:00.061786+0000 mon.a (mon.0) 2027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:33:59.611385+0000 mgr.y (mgr.24437) 297 : cluster [DBG] pgmap v415: 214 pgs: 8 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:59.900799+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.87:0/61304812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:33:59.901106+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:00.041867+0000 mon.a (mon.0) 2024 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:00.049492+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:34:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:00.049569+0000 mon.a (mon.0) 2026 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-02-23T00:34:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:00.061143+0000 mon.b (mon.1) 250 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:00.061786+0000 mon.a (mon.0) 2027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:00.841 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:00.870 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:00.890 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:00.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:00.994 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:01.021 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:01.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:34:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:34:01.671 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:01.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:01.775 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:01.805 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:02.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:02] "GET /metrics HTTP/1.1" 200 191752 "" "Prometheus/2.33.4" 2024-02-23T00:34:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:00.783007+0000 mgr.y (mgr.24437) 298 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:01.049805+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:01.049889+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-02-23T00:34:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:01.051552+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:01.051964+0000 mon.a (mon.0) 2030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:01.055125+0000 mon.b (mon.1) 251 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:01.056322+0000 mon.a (mon.0) 2031 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:02.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:02.035798+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]': finished 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:02.035877+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:02.035939+0000 mon.a (mon.0) 2034 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:02.038997+0000 mon.c (mon.2) 364 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:02.039280+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:02.039748+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:02.039862+0000 mon.a (mon.0) 2036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:00.783007+0000 mgr.y (mgr.24437) 298 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:01.049805+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:01.049889+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-02-23T00:34:02.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:01.051552+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:01.051964+0000 mon.a (mon.0) 2030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:01.055125+0000 mon.b (mon.1) 251 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:01.056322+0000 mon.a (mon.0) 2031 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:02.035798+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]': finished 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:02.035877+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:02.035939+0000 mon.a (mon.0) 2034 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:02.038997+0000 mon.c (mon.2) 364 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:02.039280+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:02.039748+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:02.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:02.039862+0000 mon.a (mon.0) 2036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:02.364 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:34:02.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:00.783007+0000 mgr.y (mgr.24437) 298 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:02.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:01.049805+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:02.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:01.049889+0000 mon.a (mon.0) 2029 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-02-23T00:34:02.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:01.051552+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:01.051964+0000 mon.a (mon.0) 2030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:01.055125+0000 mon.b (mon.1) 251 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:01.056322+0000 mon.a (mon.0) 2031 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:02.035798+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi087-82019-51"}]': finished 2024-02-23T00:34:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:02.035877+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:02.035939+0000 mon.a (mon.0) 2034 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-02-23T00:34:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:02.038997+0000 mon.c (mon.2) 364 : audit [INF] from='client.? 172.21.15.87:0/1455470090' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:02.039280+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]: dispatch 2024-02-23T00:34:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:02.039748+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:02.039862+0000 mon.a (mon.0) 2036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:02.754 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:02.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:02.859 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:02.885 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:03.083 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:03.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:03.187 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:03.219 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:03.239 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:03.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:03.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:01.611745+0000 mgr.y (mgr.24437) 299 : cluster [DBG] pgmap v418: 238 pgs: 32 creating+peering, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-23T00:34:03.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.035167+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]': finished 2024-02-23T00:34:03.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.035215+0000 mon.a (mon.0) 2038 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-37"}]': finished 2024-02-23T00:34:03.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:03.035252+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-02-23T00:34:03.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.038821+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-37", "mode": "writeback"}]: dispatch 2024-02-23T00:34:03.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.038989+0000 mon.a (mon.0) 2040 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-37", "mode": "writeback"}]: dispatch 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.052282+0000 mon.c (mon.2) 365 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.052623+0000 mon.a (mon.0) 2041 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.053107+0000 mon.c (mon.2) 366 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.053334+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:01.611745+0000 mgr.y (mgr.24437) 299 : cluster [DBG] pgmap v418: 238 pgs: 32 creating+peering, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.035167+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]': finished 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.035215+0000 mon.a (mon.0) 2038 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-37"}]': finished 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:03.035252+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.038821+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-37", "mode": "writeback"}]: dispatch 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.038989+0000 mon.a (mon.0) 2040 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-37", "mode": "writeback"}]: dispatch 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.052282+0000 mon.c (mon.2) 365 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.052623+0000 mon.a (mon.0) 2041 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.053107+0000 mon.c (mon.2) 366 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.053334+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.344 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:03.371 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:01.611745+0000 mgr.y (mgr.24437) 299 : cluster [DBG] pgmap v418: 238 pgs: 32 creating+peering, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-23T00:34:03.371 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.035167+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi087-82019-51"}]': finished 2024-02-23T00:34:03.371 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.035215+0000 mon.a (mon.0) 2038 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-37"}]': finished 2024-02-23T00:34:03.371 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:03.035252+0000 mon.a (mon.0) 2039 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-02-23T00:34:03.372 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.038821+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-37", "mode": "writeback"}]: dispatch 2024-02-23T00:34:03.372 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.038989+0000 mon.a (mon.0) 2040 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-37", "mode": "writeback"}]: dispatch 2024-02-23T00:34:03.372 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.052282+0000 mon.c (mon.2) 365 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.372 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.052623+0000 mon.a (mon.0) 2041 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.372 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.053107+0000 mon.c (mon.2) 366 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.372 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.053334+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:03.373 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:03.393 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:03.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:03.497 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:03.524 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:03.544 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:03.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:03.648 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:03.663 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:03.675 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:03.693 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:03.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:03.798 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:03.824 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:03.843 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:03.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:03.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:34:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:34:03.947 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:03.976 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:03.994 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:03.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:04.098 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:04.129 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.053713+0000 mon.c (mon.2) 367 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:03.054326+0000 mon.a (mon.0) 2043 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:04.032565+0000 mon.a (mon.0) 2044 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:34:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:04.038106+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-37", "mode": "writeback"}]': finished 2024-02-23T00:34:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:04.038195+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:04.038262+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:04.039737+0000 mon.c (mon.2) 368 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi087-82019-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:04.040119+0000 mon.a (mon.0) 2048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi087-82019-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.053713+0000 mon.c (mon.2) 367 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:03.054326+0000 mon.a (mon.0) 2043 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:04.032565+0000 mon.a (mon.0) 2044 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:04.038106+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-37", "mode": "writeback"}]': finished 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:04.038195+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:04.038262+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:04.039737+0000 mon.c (mon.2) 368 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi087-82019-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:04.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:04.040119+0000 mon.a (mon.0) 2048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi087-82019-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:04.396 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:04.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.053713+0000 mon.c (mon.2) 367 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:03.054326+0000 mon.a (mon.0) 2043 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:04.032565+0000 mon.a (mon.0) 2044 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:34:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:04.038106+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-37", "mode": "writeback"}]': finished 2024-02-23T00:34:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:04.038195+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:04.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:04.038262+0000 mon.a (mon.0) 2047 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-02-23T00:34:04.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:04.039737+0000 mon.c (mon.2) 368 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi087-82019-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:04.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:04.040119+0000 mon.a (mon.0) 2048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi087-82019-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:04.500 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:04.530 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:04.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.276+0000 7f63b7021700 1 -- 172.21.15.87:0/2831801144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63b00f8420 msgr2=0x7f63b0104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:04.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.276+0000 7f63b7021700 1 --2- 172.21.15.87:0/2831801144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63b00f8420 0x7f63b0104a20 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 rx=0x7f63a8009c30 tx=0x7f63a8004c50).stop 2024-02-23T00:34:04.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 -- 172.21.15.87:0/2831801144 shutdown_connections 2024-02-23T00:34:04.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 --2- 172.21.15.87:0/2831801144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63b00f8420 0x7f63b0104a20 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 --2- 172.21.15.87:0/2831801144 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63b00fdbb0 0x7f63b00fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 --2- 172.21.15.87:0/2831801144 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63b00fe510 0x7f63b00f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 -- 172.21.15.87:0/2831801144 >> 172.21.15.87:0/2831801144 conn(0x7f63b00f3800 msgr2=0x7f63b00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 -- 172.21.15.87:0/2831801144 shutdown_connections 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 -- 172.21.15.87:0/2831801144 wait complete. 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 librados: starting msgr at 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 librados: starting objecter 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 Processor -- start 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 -- start start 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 librados: setting wanted keys 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 librados: calling monclient init 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63b00f8420 0x7f63b0110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63b00fdbb0 0x7f63b0110510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63b00fe510 0x7f63b010a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:04.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b010a6f0 con 0x7f63b00fdbb0 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b010a860 con 0x7f63b00fe510 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.277+0000 7f63b7021700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b010a9d0 con 0x7f63b00f8420 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b4dbd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63b00f8420 0x7f63b0110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b4dbd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63b00f8420 0x7f63b0110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:56466/0 (socket says 172.21.15.87:56466) 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b4dbd700 1 -- 172.21.15.87:0/1136149395 learned_addr learned my addr 172.21.15.87:0/1136149395 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63affff700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63b00fdbb0 0x7f63b0110510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b55be700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63b00fe510 0x7f63b010a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b4dbd700 1 -- 172.21.15.87:0/1136149395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63b00fe510 msgr2=0x7f63b010a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b4dbd700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63b00fe510 0x7f63b010a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b4dbd700 1 -- 172.21.15.87:0/1136149395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63b00fdbb0 msgr2=0x7f63b0110510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b4dbd700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63b00fdbb0 0x7f63b0110510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b4dbd700 1 -- 172.21.15.87:0/1136149395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4005040 con 0x7f63b00f8420 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63affff700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63b00fdbb0 0x7f63b0110510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:34:04.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b4dbd700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63b00f8420 0x7f63b0110020 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7f63a4009400 tx=0x7f63a4009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63adffb700 10 client.?.objecter ms_handle_connect 0x7f63b00f8420 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63adffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63adffb700 1 -- 172.21.15.87:0/1136149395 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4007a40 con 0x7f63b00f8420 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63adffb700 1 -- 172.21.15.87:0/1136149395 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63a400f040 con 0x7f63b00f8420 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63adffb700 1 -- 172.21.15.87:0/1136149395 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4014aa0 con 0x7f63b00f8420 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a80097c0 con 0x7f63b00f8420 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b7021700 10 client.37403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b0116e90 con 0x7f63b00f8420 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.278+0000 7f63b7021700 1 librados: init done 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.279+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b005ee80 con 0x7f63b00f8420 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.280+0000 7f63adffb700 1 -- 172.21.15.87:0/1136149395 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f63a400e630 con 0x7f63b00f8420 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.280+0000 7f63adffb700 1 --2- 172.21.15.87:0/1136149395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f63a0062c90 0x7f63a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.280+0000 7f63adffb700 1 -- 172.21.15.87:0/1136149395 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(293..293 src has 38..293) v4 ==== 8172+0+0 (secure 0 0 0) 0x7f63a407cdd0 con 0x7f63b00f8420 2024-02-23T00:34:04.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.280+0000 7f63adffb700 10 client.37403.objecter ms_dispatch 0x7f63b0001550 osd_map(293..293 src has 38..293) v4 2024-02-23T00:34:04.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.280+0000 7f63adffb700 3 client.37403.objecter handle_osd_map got epochs [293,293] > 0 2024-02-23T00:34:04.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.280+0000 7f63adffb700 3 client.37403.objecter handle_osd_map decoding full epoch 293 2024-02-23T00:34:04.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.280+0000 7f63affff700 1 --2- 172.21.15.87:0/1136149395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f63a0062c90 0x7f63a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:04.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.280+0000 7f63adffb700 20 client.37403.objecter dump_active .. 0 homeless 2024-02-23T00:34:04.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.281+0000 7f63adffb700 10 client.37403.objecter ms_handle_connect 0x7f63a0062c90 2024-02-23T00:34:04.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.281+0000 7f63affff700 1 --2- 172.21.15.87:0/1136149395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f63a0062c90 0x7f63a0065150 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 rx=0x7f63b010b9a0 tx=0x7f639c009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:04.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.285+0000 7f63adffb700 1 -- 172.21.15.87:0/1136149395 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f63a4049f60 con 0x7f63b00f8420 2024-02-23T00:34:04.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:04.471+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]} v 0) v1 -- 0x7f63b0107110 con 0x7f63b00f8420 2024-02-23T00:34:04.603 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:04.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:04.706 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:04.724 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:34:04.733 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:04.760 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:04.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:04.864 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:04.890 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:04.895 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:34:04.896 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:34:04.914 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:04.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:05.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:05.177 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:05.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:05.281 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:05.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.281+0000 7f63adffb700 1 -- 172.21.15.87:0/1136149395 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]=0 set 2.c pg_upmap_items mapping to [5->0,1->4,6->3] v294) v1 ==== 157+0+0 (secure 0 0 0) 0x7f63a40500e0 con 0x7f63b00f8420 2024-02-23T00:34:05.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.283+0000 7f63b7021700 10 librados: watch_flush enter 2024-02-23T00:34:05.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.283+0000 7f63b7021700 10 librados: watch_flush exit 2024-02-23T00:34:05.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.283+0000 7f63b7021700 20 client.37403.objecter shutdown clearing up homeless session... 2024-02-23T00:34:05.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.283+0000 7f63b7021700 10 client.37403.objecter successfully canceled tick 2024-02-23T00:34:05.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f63a0062c90 msgr2=0x7f63a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:05.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 --2- 172.21.15.87:0/1136149395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f63a0062c90 0x7f63a0065150 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 rx=0x7f63b010b9a0 tx=0x7f639c009040).stop 2024-02-23T00:34:05.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63b00f8420 msgr2=0x7f63b0110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:05.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63b00f8420 0x7f63b0110020 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7f63a4009400 tx=0x7f63a4009cb0).stop 2024-02-23T00:34:05.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 shutdown_connections 2024-02-23T00:34:05.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 --2- 172.21.15.87:0/1136149395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f63a0062c90 0x7f63a0065150 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:05.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f63b00fe510 0x7f63b010a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:05.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f63b00f8420 0x7f63b0110020 secure :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 rx=0x7f63a4009400 tx=0x7f63a4009cb0).stop 2024-02-23T00:34:05.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 --2- 172.21.15.87:0/1136149395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f63b00fdbb0 0x7f63b0110510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:05.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.284+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 >> 172.21.15.87:0/1136149395 conn(0x7f63b00f3800 msgr2=0x7f63b0103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:05.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.286+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 shutdown_connections 2024-02-23T00:34:05.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.286+0000 7f63b7021700 1 -- 172.21.15.87:0/1136149395 wait complete. 2024-02-23T00:34:05.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:05.286+0000 7f63b7021700 1 librados: shutdown 2024-02-23T00:34:05.288 INFO:teuthology.orchestra.run.smithi087.stderr:set 2.c pg_upmap_items mapping to [5->0,1->4,6->3] 2024-02-23T00:34:05.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:03.612082+0000 mgr.y (mgr.24437) 300 : cluster [DBG] pgmap v421: 238 pgs: 32 creating+peering, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-23T00:34:05.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:04.131083+0000 mon.b (mon.1) 254 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:05.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:04.131293+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:05.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:04.473111+0000 mon.c (mon.2) 369 : audit [INF] from='client.? 172.21.15.87:0/1136149395' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]: dispatch 2024-02-23T00:34:05.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:04.473452+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]: dispatch 2024-02-23T00:34:05.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:03.612082+0000 mgr.y (mgr.24437) 300 : cluster [DBG] pgmap v421: 238 pgs: 32 creating+peering, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-23T00:34:05.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:04.131083+0000 mon.b (mon.1) 254 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:05.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:04.131293+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:05.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:04.473111+0000 mon.c (mon.2) 369 : audit [INF] from='client.? 172.21.15.87:0/1136149395' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]: dispatch 2024-02-23T00:34:05.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:04.473452+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]: dispatch 2024-02-23T00:34:05.311 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:03.612082+0000 mgr.y (mgr.24437) 300 : cluster [DBG] pgmap v421: 238 pgs: 32 creating+peering, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-23T00:34:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:04.131083+0000 mon.b (mon.1) 254 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:04.131293+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:04.473111+0000 mon.c (mon.2) 369 : audit [INF] from='client.? 172.21.15.87:0/1136149395' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]: dispatch 2024-02-23T00:34:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:04.473452+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]: dispatch 2024-02-23T00:34:05.978 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:05.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:06.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:06.110 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:06.130 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:06.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:06.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:06.233 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:06.261 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:06.280 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:06.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:06.383 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:06.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-23T00:34:06.419 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:05.281735+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:34:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:05.281883+0000 mon.a (mon.0) 2052 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]': finished 2024-02-23T00:34:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:05.281993+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-02-23T00:34:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:05.283639+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:05.288842+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:06.035871+0000 mon.a (mon.0) 2055 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:34:06.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:06.040936+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi087-82019-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi087-82019-52"}]': finished 2024-02-23T00:34:06.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:06.041050+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37"}]': finished 2024-02-23T00:34:06.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:06.041121+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-02-23T00:34:06.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:05.281735+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:34:06.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:05.281883+0000 mon.a (mon.0) 2052 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]': finished 2024-02-23T00:34:06.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:05.281993+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-02-23T00:34:06.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:05.283639+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:06.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:05.288842+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:06.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:06.035871+0000 mon.a (mon.0) 2055 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:34:06.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:06.040936+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi087-82019-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi087-82019-52"}]': finished 2024-02-23T00:34:06.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:06.041050+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37"}]': finished 2024-02-23T00:34:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:06.041121+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-02-23T00:34:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:05.281735+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:34:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:05.281883+0000 mon.a (mon.0) 2052 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.c", "id": [5, 0, 1, 4, 6, 3]}]': finished 2024-02-23T00:34:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:05.281993+0000 mon.a (mon.0) 2053 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-02-23T00:34:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:05.283639+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:05.288842+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37"}]: dispatch 2024-02-23T00:34:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:06.035871+0000 mon.a (mon.0) 2055 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:34:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:06.040936+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi087-82019-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi087-82019-52"}]': finished 2024-02-23T00:34:06.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:06.041050+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-37"}]': finished 2024-02-23T00:34:06.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:06.041121+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-02-23T00:34:06.867 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:06.869 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:06.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:06.974 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:07.007 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:07.032 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:07.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:07.133 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:07.161 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:07.309 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:07.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:07.413 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:07.445 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:07.467 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:07.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:07.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:05.613088+0000 mgr.y (mgr.24437) 301 : cluster [DBG] pgmap v424: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-02-23T00:34:07.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:07.041481+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-02-23T00:34:07.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:05.613088+0000 mgr.y (mgr.24437) 301 : cluster [DBG] pgmap v424: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-02-23T00:34:07.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:07.041481+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-02-23T00:34:07.570 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:07.601 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:05.613088+0000 mgr.y (mgr.24437) 301 : cluster [DBG] pgmap v424: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-02-23T00:34:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:07.041481+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.058+0000 7fb493fde700 1 -- 172.21.15.87:0/3958655970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb48c0ffde0 msgr2=0x7fb48c1001c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.058+0000 7fb493fde700 1 --2- 172.21.15.87:0/3958655970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb48c0ffde0 0x7fb48c1001c0 secure :-1 s=READY pgs=4983 cs=0 l=1 rev1=1 rx=0x7fb484009be0 tx=0x7fb484004d10).stop 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.059+0000 7fb493fde700 1 -- 172.21.15.87:0/3958655970 shutdown_connections 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.059+0000 7fb493fde700 1 --2- 172.21.15.87:0/3958655970 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb48c0fa540 0x7fb48c0fa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.059+0000 7fb493fde700 1 --2- 172.21.15.87:0/3958655970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb48c0ffde0 0x7fb48c1001c0 secure :-1 s=CLOSED pgs=4983 cs=0 l=1 rev1=1 rx=0x7fb484009be0 tx=0x7fb484004d10).stop 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.059+0000 7fb493fde700 1 --2- 172.21.15.87:0/3958655970 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb48c1006b0 0x7fb48c0f9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.059+0000 7fb493fde700 1 -- 172.21.15.87:0/3958655970 >> 172.21.15.87:0/3958655970 conn(0x7fb48c0f59f0 msgr2=0x7fb48c0f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.059+0000 7fb493fde700 1 -- 172.21.15.87:0/3958655970 shutdown_connections 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.059+0000 7fb493fde700 1 -- 172.21.15.87:0/3958655970 wait complete. 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.060+0000 7fb493fde700 1 librados: starting msgr at 2024-02-23T00:34:08.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.060+0000 7fb493fde700 1 librados: starting objecter 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.060+0000 7fb493fde700 1 Processor -- start 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.060+0000 7fb493fde700 1 -- start start 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.060+0000 7fb493fde700 1 librados: setting wanted keys 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.060+0000 7fb493fde700 1 librados: calling monclient init 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb493fde700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb48c0fa540 0x7fb48c110660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb493fde700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb48c1006b0 0x7fb48c110b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb493fde700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb48c10a7c0 0x7fb48c10ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb493fde700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c10b130 con 0x7fb48c10a7c0 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb493fde700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c10b2a0 con 0x7fb48c1006b0 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb493fde700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c10b460 con 0x7fb48c0fa540 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491d7a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb48c0fa540 0x7fb48c110660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491d7a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb48c0fa540 0x7fb48c110660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:56492/0 (socket says 172.21.15.87:56492) 2024-02-23T00:34:08.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491d7a700 1 -- 172.21.15.87:0/3015766746 learned_addr learned my addr 172.21.15.87:0/3015766746 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb49257b700 1 --2- 172.21.15.87:0/3015766746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb48c10a7c0 0x7fb48c10ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491579700 1 --2- 172.21.15.87:0/3015766746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb48c1006b0 0x7fb48c110b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491d7a700 1 -- 172.21.15.87:0/3015766746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb48c1006b0 msgr2=0x7fb48c110b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491d7a700 1 --2- 172.21.15.87:0/3015766746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb48c1006b0 0x7fb48c110b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491d7a700 1 -- 172.21.15.87:0/3015766746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb48c10a7c0 msgr2=0x7fb48c10ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491d7a700 1 --2- 172.21.15.87:0/3015766746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb48c10a7c0 0x7fb48c10ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491d7a700 1 -- 172.21.15.87:0/3015766746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4840097c0 con 0x7fb48c0fa540 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb491d7a700 1 --2- 172.21.15.87:0/3015766746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb48c0fa540 0x7fb48c110660 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 rx=0x7fb48400b780 tx=0x7fb484004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb482ffd700 10 client.?.objecter ms_handle_connect 0x7fb48c0fa540 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb482ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.061+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48400e070 con 0x7fb48c0fa540 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.062+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb484018040 con 0x7fb48c0fa540 2024-02-23T00:34:08.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.062+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb484017070 con 0x7fb48c0fa540 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.062+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48c117420 con 0x7fb48c0fa540 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.062+0000 7fb493fde700 10 client.37406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.062+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c117770 con 0x7fb48c0fa540 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.062+0000 7fb493fde700 1 librados: init done 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.062+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb48c05ee80 con 0x7fb48c0fa540 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.063+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb484007b20 con 0x7fb48c0fa540 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.063+0000 7fb482ffd700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb478062ce0 0x7fb4780651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.063+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(297..297 src has 38..297) v4 ==== 8292+0+0 (secure 0 0 0) 0x7fb48407e680 con 0x7fb48c0fa540 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.063+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 osd_map(297..297 src has 38..297) v4 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.063+0000 7fb482ffd700 3 client.37406.objecter handle_osd_map got epochs [297,297] > 0 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.063+0000 7fb482ffd700 3 client.37406.objecter handle_osd_map decoding full epoch 297 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.063+0000 7fb491579700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb478062ce0 0x7fb4780651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.063+0000 7fb482ffd700 20 client.37406.objecter dump_active .. 0 homeless 2024-02-23T00:34:08.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.063+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v297) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb484085070 con 0x7fb48c0fa540 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb482ffd700 10 client.37406.objecter ms_handle_connect 0x7fb478062ce0 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb491579700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb478062ce0 0x7fb4780651a0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7fb48c10bf60 tx=0x7fb47c009060).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 10 client.37406.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 1 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb48c101190 0x7fb48c101570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c109410 osd=0 3 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c109410 osd=0 3 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c109410 osd=0 3 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 1 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c109410 osd=0 3 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 0 1 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c109410 osd=0 4 2024-02-23T00:34:08.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 10 client.37406.objecter _send_command 1 2024-02-23T00:34:08.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb48c10bca0 con 0x7fb48c101190 2024-02-23T00:34:08.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb49257b700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb48c101190 0x7fb48c101570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb49257b700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb48c101190 0x7fb48c101570 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.064+0000 7fb482ffd700 10 client.37406.objecter ms_handle_connect 0x7fb48c101190 2024-02-23T00:34:08.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.065+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb48c10bca0 con 0x7fb48c101190 2024-02-23T00:34:08.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.065+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:34:08.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.065+0000 7fb482ffd700 10 client.37406.objecter _finish_command 1 = system:0 2024-02-23T00:34:08.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.065+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c109410 osd=0 4 2024-02-23T00:34:08.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.065+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 0 1 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 10 client.37406.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 2 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c109410 osd=0 3 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c109410 osd=0 3 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c109410 osd=0 3 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 2 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c109410 osd=0 3 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 0 2 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c109410 osd=0 4 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 10 client.37406.objecter _send_command 2 2024-02-23T00:34:08.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.080+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb48c0273c0 con 0x7fb48c101190 2024-02-23T00:34:08.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.081+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb48c0273c0 con 0x7fb48c101190 2024-02-23T00:34:08.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.081+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:08.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.081+0000 7fb482ffd700 10 client.37406.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.081+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c109410 osd=0 4 2024-02-23T00:34:08.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.081+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 0 2 2024-02-23T00:34:08.087 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 10 client.37406.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 3 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6805/685354171 conn(0x7fb48c0598f0 0x7fb48c059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c028710 osd=1 3 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c028710 osd=1 3 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c028710 osd=1 3 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 3 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c028710 osd=1 3 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 1 3 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c028710 osd=1 4 2024-02-23T00:34:08.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 10 client.37406.objecter _send_command 3 2024-02-23T00:34:08.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb48c11e080 con 0x7fb48c0598f0 2024-02-23T00:34:08.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb491d7a700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6805/685354171 conn(0x7fb48c0598f0 0x7fb48c059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb482ffd700 10 client.37406.objecter ms_handle_connect 0x7fb48c0598f0 2024-02-23T00:34:08.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.082+0000 7fb491d7a700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6805/685354171 conn(0x7fb48c0598f0 0x7fb48c059cd0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.083+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb48c11e080 con 0x7fb48c0598f0 2024-02-23T00:34:08.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.083+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:34:08.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.083+0000 7fb482ffd700 10 client.37406.objecter _finish_command 3 = system:0 2024-02-23T00:34:08.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.083+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c028710 osd=1 4 2024-02-23T00:34:08.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.083+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 1 3 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 10 client.37406.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 4 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c028710 osd=1 3 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c028710 osd=1 3 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c028710 osd=1 3 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 4 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c028710 osd=1 3 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 1 4 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c028710 osd=1 4 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 10 client.37406.objecter _send_command 4 2024-02-23T00:34:08.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.099+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb48c059360 con 0x7fb48c0598f0 2024-02-23T00:34:08.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.101+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb48c059360 con 0x7fb48c0598f0 2024-02-23T00:34:08.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.101+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:08.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.101+0000 7fb482ffd700 10 client.37406.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.101+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c028710 osd=1 4 2024-02-23T00:34:08.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.101+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 1 4 2024-02-23T00:34:08.104 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 10 client.37406.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 5 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb48c11caa0 0x7fb48c11cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c027d60 osd=2 3 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c027d60 osd=2 3 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c027d60 osd=2 3 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 5 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c027d60 osd=2 3 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 2 5 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c027d60 osd=2 4 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 10 client.37406.objecter _send_command 5 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb48c0283e0 con 0x7fb48c11caa0 2024-02-23T00:34:08.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb491579700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb48c11caa0 0x7fb48c11cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb482ffd700 10 client.37406.objecter ms_handle_connect 0x7fb48c11caa0 2024-02-23T00:34:08.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.102+0000 7fb491579700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb48c11caa0 0x7fb48c11cec0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.103+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb48c0283e0 con 0x7fb48c11caa0 2024-02-23T00:34:08.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.103+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:34:08.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.103+0000 7fb482ffd700 10 client.37406.objecter _finish_command 5 = system:0 2024-02-23T00:34:08.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.103+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c027d60 osd=2 4 2024-02-23T00:34:08.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.103+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 2 5 2024-02-23T00:34:08.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 10 client.37406.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 6 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c027d60 osd=2 3 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c027d60 osd=2 3 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c027d60 osd=2 3 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 6 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c027d60 osd=2 3 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 2 6 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c027d60 osd=2 4 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 10 client.37406.objecter _send_command 6 2024-02-23T00:34:08.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.117+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb48c11db80 con 0x7fb48c11caa0 2024-02-23T00:34:08.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb48c11db80 con 0x7fb48c11caa0 2024-02-23T00:34:08.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:08.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb482ffd700 10 client.37406.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c027d60 osd=2 4 2024-02-23T00:34:08.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 2 6 2024-02-23T00:34:08.121 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:34:08.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 10 client.37406.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:08.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 7 2024-02-23T00:34:08.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6813/128059790 conn(0x7fb48c027940 0x7fb48c122720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c0277d0 osd=3 3 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c0277d0 osd=3 3 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c0277d0 osd=3 3 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 7 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c0277d0 osd=3 3 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 3 7 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c0277d0 osd=3 4 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 10 client.37406.objecter _send_command 7 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb48c125370 con 0x7fb48c027940 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.119+0000 7fb49257b700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6813/128059790 conn(0x7fb48c027940 0x7fb48c122720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.120+0000 7fb482ffd700 10 client.37406.objecter ms_handle_connect 0x7fb48c027940 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.120+0000 7fb49257b700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6813/128059790 conn(0x7fb48c027940 0x7fb48c122720 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.120+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb48c125370 con 0x7fb48c027940 2024-02-23T00:34:08.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.120+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:34:08.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.120+0000 7fb482ffd700 10 client.37406.objecter _finish_command 7 = system:0 2024-02-23T00:34:08.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.120+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c0277d0 osd=3 4 2024-02-23T00:34:08.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.120+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 3 7 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 10 client.37406.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 8 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c0277d0 osd=3 3 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c0277d0 osd=3 3 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c0277d0 osd=3 3 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 8 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c0277d0 osd=3 3 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 3 8 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c0277d0 osd=3 4 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 10 client.37406.objecter _send_command 8 2024-02-23T00:34:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.135+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb48c11d5f0 con 0x7fb48c027940 2024-02-23T00:34:08.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.136+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb48c11d5f0 con 0x7fb48c027940 2024-02-23T00:34:08.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.136+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:08.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.136+0000 7fb482ffd700 10 client.37406.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.136+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c0277d0 osd=3 4 2024-02-23T00:34:08.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.136+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 3 8 2024-02-23T00:34:08.139 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:34:08.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 10 client.37406.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 9 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb48c11c600 0x7fb48c123d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c11d5f0 osd=4 3 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d5f0 osd=4 3 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c11d5f0 osd=4 3 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 9 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c11d5f0 osd=4 3 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 4 9 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d5f0 osd=4 4 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 10 client.37406.objecter _send_command 9 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb48c11d7c0 con 0x7fb48c11c600 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb491d7a700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb48c11c600 0x7fb48c123d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb491d7a700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb48c11c600 0x7fb48c123d90 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.137+0000 7fb482ffd700 10 client.37406.objecter ms_handle_connect 0x7fb48c11c600 2024-02-23T00:34:08.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.138+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb48c11d7c0 con 0x7fb48c11c600 2024-02-23T00:34:08.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.138+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:34:08.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.138+0000 7fb482ffd700 10 client.37406.objecter _finish_command 9 = system:0 2024-02-23T00:34:08.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.138+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c11d5f0 osd=4 4 2024-02-23T00:34:08.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.138+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 4 9 2024-02-23T00:34:08.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 10 client.37406.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:08.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 10 2024-02-23T00:34:08.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c11d5f0 osd=4 3 2024-02-23T00:34:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d5f0 osd=4 3 2024-02-23T00:34:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c11d5f0 osd=4 3 2024-02-23T00:34:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 10 2024-02-23T00:34:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c11d5f0 osd=4 3 2024-02-23T00:34:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 4 10 2024-02-23T00:34:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d5f0 osd=4 4 2024-02-23T00:34:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 10 client.37406.objecter _send_command 10 2024-02-23T00:34:08.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.152+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb48c11d7c0 con 0x7fb48c11c600 2024-02-23T00:34:08.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb48c11d7c0 con 0x7fb48c11c600 2024-02-23T00:34:08.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:08.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb482ffd700 10 client.37406.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c11d5f0 osd=4 4 2024-02-23T00:34:08.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 4 10 2024-02-23T00:34:08.157 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 10 client.37406.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 11 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb48c124790 0x7fb48c124bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c11d7c0 osd=5 3 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d7c0 osd=5 3 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c11d7c0 osd=5 3 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 11 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c11d7c0 osd=5 3 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 5 11 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d7c0 osd=5 4 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 10 client.37406.objecter _send_command 11 2024-02-23T00:34:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.154+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb48c1250e0 con 0x7fb48c124790 2024-02-23T00:34:08.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.155+0000 7fb491579700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb48c124790 0x7fb48c124bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.155+0000 7fb482ffd700 10 client.37406.objecter ms_handle_connect 0x7fb48c124790 2024-02-23T00:34:08.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.155+0000 7fb491579700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb48c124790 0x7fb48c124bf0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.156+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb48c1250e0 con 0x7fb48c124790 2024-02-23T00:34:08.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.156+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:34:08.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.156+0000 7fb482ffd700 10 client.37406.objecter _finish_command 11 = system:0 2024-02-23T00:34:08.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.156+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c11d7c0 osd=5 4 2024-02-23T00:34:08.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.156+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 5 11 2024-02-23T00:34:08.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 10 client.37406.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:08.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 12 2024-02-23T00:34:08.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c11d7c0 osd=5 3 2024-02-23T00:34:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d7c0 osd=5 3 2024-02-23T00:34:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c11d7c0 osd=5 3 2024-02-23T00:34:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 12 2024-02-23T00:34:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c11d7c0 osd=5 3 2024-02-23T00:34:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 5 12 2024-02-23T00:34:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d7c0 osd=5 4 2024-02-23T00:34:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 10 client.37406.objecter _send_command 12 2024-02-23T00:34:08.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.169+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb48c1250e0 con 0x7fb48c124790 2024-02-23T00:34:08.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb48c1250e0 con 0x7fb48c124790 2024-02-23T00:34:08.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:08.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb482ffd700 10 client.37406.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c11d7c0 osd=5 4 2024-02-23T00:34:08.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 5 12 2024-02-23T00:34:08.174 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:34:08.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 10 client.37406.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:08.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 13 2024-02-23T00:34:08.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb48c129a10 0x7fb48c12be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c1250e0 osd=6 3 2024-02-23T00:34:08.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c1250e0 osd=6 3 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c1250e0 osd=6 3 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 13 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c1250e0 osd=6 3 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 6 13 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c1250e0 osd=6 4 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 10 client.37406.objecter _send_command 13 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.171+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb48c12eaf0 con 0x7fb48c129a10 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.172+0000 7fb49257b700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb48c129a10 0x7fb48c12be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.172+0000 7fb482ffd700 10 client.37406.objecter ms_handle_connect 0x7fb48c129a10 2024-02-23T00:34:08.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.172+0000 7fb49257b700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb48c129a10 0x7fb48c12be00 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.173+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb48c12eaf0 con 0x7fb48c129a10 2024-02-23T00:34:08.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.173+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:34:08.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.173+0000 7fb482ffd700 10 client.37406.objecter _finish_command 13 = system:0 2024-02-23T00:34:08.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.173+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c1250e0 osd=6 4 2024-02-23T00:34:08.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.173+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 6 13 2024-02-23T00:34:08.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 10 client.37406.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:08.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 14 2024-02-23T00:34:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c1250e0 osd=6 3 2024-02-23T00:34:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c1250e0 osd=6 3 2024-02-23T00:34:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c1250e0 osd=6 3 2024-02-23T00:34:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 14 2024-02-23T00:34:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c1250e0 osd=6 3 2024-02-23T00:34:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 6 14 2024-02-23T00:34:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c1250e0 osd=6 4 2024-02-23T00:34:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 10 client.37406.objecter _send_command 14 2024-02-23T00:34:08.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.187+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb48c12eaf0 con 0x7fb48c129a10 2024-02-23T00:34:08.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb48c12eaf0 con 0x7fb48c129a10 2024-02-23T00:34:08.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:08.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb482ffd700 10 client.37406.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c1250e0 osd=6 4 2024-02-23T00:34:08.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 6 14 2024-02-23T00:34:08.191 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:34:08.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 10 client.37406.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:08.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 15 2024-02-23T00:34:08.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6812/373787000 conn(0x7fb48c12df30 0x7fb48c12e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:08.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c12eaf0 osd=7 3 2024-02-23T00:34:08.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c12eaf0 osd=7 3 2024-02-23T00:34:08.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c12eaf0 osd=7 3 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 15 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c12eaf0 osd=7 3 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 7 15 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c12eaf0 osd=7 4 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 10 client.37406.objecter _send_command 15 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb48c131ea0 con 0x7fb48c12df30 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb491d7a700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6812/373787000 conn(0x7fb48c12df30 0x7fb48c12e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb482ffd700 10 client.37406.objecter ms_handle_connect 0x7fb48c12df30 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.189+0000 7fb491d7a700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6812/373787000 conn(0x7fb48c12df30 0x7fb48c12e370 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.190+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb48c131ea0 con 0x7fb48c12df30 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.190+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.190+0000 7fb482ffd700 10 client.37406.objecter _finish_command 15 = system:0 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.190+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c12eaf0 osd=7 4 2024-02-23T00:34:08.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.190+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 7 15 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 10 client.37406.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign -1 16 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c12eaf0 osd=7 3 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c12eaf0 osd=7 3 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 20 client.37406.objecter _get_session s=0x7fb48c12eaf0 osd=7 3 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 15 client.37406.objecter _session_command_op_remove -1 16 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 20 client.37406.objecter get_session s=0x7fb48c12eaf0 osd=7 3 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 15 client.37406.objecter _session_command_op_assign 7 16 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c12eaf0 osd=7 4 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 10 client.37406.objecter _send_command 16 2024-02-23T00:34:08.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.205+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb48c130d20 con 0x7fb48c12df30 2024-02-23T00:34:08.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb482ffd700 1 -- 172.21.15.87:0/3015766746 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fb48c130d20 con 0x7fb48c12df30 2024-02-23T00:34:08.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb482ffd700 10 client.37406.objecter ms_dispatch 0x7fb48c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:08.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb482ffd700 10 client.37406.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb482ffd700 20 client.37406.objecter put_session s=0x7fb48c12eaf0 osd=7 4 2024-02-23T00:34:08.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb482ffd700 15 client.37406.objecter _session_command_op_remove 7 16 2024-02-23T00:34:08.209 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 librados: watch_flush enter 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 librados: watch_flush exit 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 client.37406.objecter close_session for osd.0 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb48c101190 msgr2=0x7fb48c101570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6801/2153191863 conn(0x7fb48c101190 0x7fb48c101570 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c109410 osd=0 1 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 client.37406.objecter close_session for osd.1 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6805/685354171 conn(0x7fb48c0598f0 msgr2=0x7fb48c059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6805/685354171 conn(0x7fb48c0598f0 0x7fb48c059cd0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c028710 osd=1 1 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 client.37406.objecter close_session for osd.2 2024-02-23T00:34:08.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb48c11caa0 msgr2=0x7fb48c11cec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb48c11caa0 0x7fb48c11cec0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c027d60 osd=2 1 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 client.37406.objecter close_session for osd.3 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6813/128059790 conn(0x7fb48c027940 msgr2=0x7fb48c122720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb49257b700 1 -- 172.21.15.87:0/3015766746 reap_dead start 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6813/128059790 conn(0x7fb48c027940 0x7fb48c122720 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c0277d0 osd=3 1 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 client.37406.objecter close_session for osd.4 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb48c11c600 msgr2=0x7fb48c123d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb48c11c600 0x7fb48c123d90 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d5f0 osd=4 1 2024-02-23T00:34:08.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 client.37406.objecter close_session for osd.5 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb48c124790 msgr2=0x7fb48c124bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb48c124790 0x7fb48c124bf0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c11d7c0 osd=5 1 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 client.37406.objecter close_session for osd.6 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb48c129a10 msgr2=0x7fb48c12be00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb48c129a10 0x7fb48c12be00 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c1250e0 osd=6 1 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 client.37406.objecter close_session for osd.7 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6812/373787000 conn(0x7fb48c12df30 msgr2=0x7fb48c12e370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.171:6812/373787000 conn(0x7fb48c12df30 0x7fb48c12e370 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 20 client.37406.objecter put_session s=0x7fb48c12eaf0 osd=7 1 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 20 client.37406.objecter shutdown clearing up homeless session... 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 10 client.37406.objecter successfully canceled tick 2024-02-23T00:34:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb49257b700 1 -- 172.21.15.87:0/3015766746 reap_dead start 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb478062ce0 msgr2=0x7fb4780651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb478062ce0 0x7fb4780651a0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 rx=0x7fb48c10bf60 tx=0x7fb47c009060).stop 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb48c0fa540 msgr2=0x7fb48c110660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb48c0fa540 0x7fb48c110660 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 rx=0x7fb48400b780 tx=0x7fb484004d10).stop 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 shutdown_connections 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb478062ce0 0x7fb4780651a0 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 --2- 172.21.15.87:0/3015766746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb48c0fa540 0x7fb48c110660 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 >> 172.21.15.87:0/3015766746 conn(0x7fb48c0f59f0 msgr2=0x7fb48c0f7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 shutdown_connections 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.207+0000 7fb493fde700 1 -- 172.21.15.87:0/3015766746 wait complete. 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:08.208+0000 7fb493fde700 1 librados: shutdown 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:34:08.215 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:34:09.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:07.613496+0000 mgr.y (mgr.24437) 302 : cluster [DBG] pgmap v427: 214 pgs: 8 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-23T00:34:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:08.048402+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-02-23T00:34:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:08.051689+0000 mon.c (mon.2) 370 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:08.052411+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:08.055708+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:08.059691+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:08.064768+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? 172.21.15.87:0/3015766746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:09.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:07.613496+0000 mgr.y (mgr.24437) 302 : cluster [DBG] pgmap v427: 214 pgs: 8 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-23T00:34:09.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:08.048402+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-02-23T00:34:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:08.051689+0000 mon.c (mon.2) 370 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:08.052411+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:08.055708+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:08.059691+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:08.064768+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? 172.21.15.87:0/3015766746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:09.353 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:09.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:09.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:07.613496+0000 mgr.y (mgr.24437) 302 : cluster [DBG] pgmap v427: 214 pgs: 8 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-23T00:34:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:08.048402+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-02-23T00:34:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:08.051689+0000 mon.c (mon.2) 370 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:08.052411+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:08.055708+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:08.059691+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:08.064768+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? 172.21.15.87:0/3015766746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:09.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:09.487 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:09.923 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:09.936 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:09.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:10.041 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:10.073 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:10.195 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:10.197 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:09.047490+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]': finished 2024-02-23T00:34:10.197 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:09.047663+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:10.197 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:09.047732+0000 mon.a (mon.0) 2065 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:09.049314+0000 mon.c (mon.2) 372 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:09.051004+0000 mon.b (mon.1) 257 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:09.051094+0000 mon.a (mon.0) 2066 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:09.053662+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:10.047276+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]': finished 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:10.047369+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:10.047430+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:10.049741+0000 mon.b (mon.1) 258 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:10.049993+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:10.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:09.047490+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]': finished 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:09.047663+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:09.047732+0000 mon.a (mon.0) 2065 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:09.049314+0000 mon.c (mon.2) 372 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:09.051004+0000 mon.b (mon.1) 257 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:09.051094+0000 mon.a (mon.0) 2066 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:09.053662+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:10.047276+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]': finished 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:10.047369+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:10.047430+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:10.049741+0000 mon.b (mon.1) 258 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:10.199 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:10.049993+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:10.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:10.301 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:10.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:09.047490+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi087-82019-52"}]': finished 2024-02-23T00:34:10.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:09.047663+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:10.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:09.047732+0000 mon.a (mon.0) 2065 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-02-23T00:34:10.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:09.049314+0000 mon.c (mon.2) 372 : audit [INF] from='client.? 172.21.15.87:0/2829346883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:10.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:09.051004+0000 mon.b (mon.1) 257 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:10.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:09.051094+0000 mon.a (mon.0) 2066 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]: dispatch 2024-02-23T00:34:10.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:09.053662+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:10.333 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:10.047276+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi087-82019-52"}]': finished 2024-02-23T00:34:10.333 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:10.047369+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:10.333 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:10.047430+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-02-23T00:34:10.333 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:10.049741+0000 mon.b (mon.1) 258 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:10.333 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:10.049993+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:10.334 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:10.355 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:10.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:10.459 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:10.487 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:10.560 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:10.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:10.664 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:10.693 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:10.713 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:10.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:10.817 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:10.845 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:10.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:34:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:34:11.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:09.613962+0000 mgr.y (mgr.24437) 303 : cluster [DBG] pgmap v430: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:11.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:10.069738+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:11.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:10.070893+0000 mon.a (mon.0) 2073 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:10.071255+0000 mon.a (mon.0) 2074 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi087-82019-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:11.048765+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:11.048858+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi087-82019-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:11.048918+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:11.051871+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:11.052843+0000 mon.a (mon.0) 2078 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi087-82019-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:11.053387+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:09.613962+0000 mgr.y (mgr.24437) 303 : cluster [DBG] pgmap v430: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:10.069738+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:10.070893+0000 mon.a (mon.0) 2073 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:11.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:10.071255+0000 mon.a (mon.0) 2074 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi087-82019-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:11.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:11.048765+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:34:11.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:11.048858+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi087-82019-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:11.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:11.048918+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-02-23T00:34:11.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:11.051871+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:11.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:11.052843+0000 mon.a (mon.0) 2078 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi087-82019-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:11.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:11.053387+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:09.613962+0000 mgr.y (mgr.24437) 303 : cluster [DBG] pgmap v430: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:10.069738+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:10.070893+0000 mon.a (mon.0) 2073 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:10.071255+0000 mon.a (mon.0) 2074 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi087-82019-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:11.048765+0000 mon.a (mon.0) 2075 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:34:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:11.048858+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi087-82019-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:11.048918+0000 mon.a (mon.0) 2077 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-02-23T00:34:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:11.051871+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:11.052843+0000 mon.a (mon.0) 2078 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi087-82019-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:11.053387+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:11.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.945+0000 7f6da086b700 1 -- 172.21.15.87:0/4066351835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6d9c0fd510 msgr2=0x7f6d9c0f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.945+0000 7f6da086b700 1 --2- 172.21.15.87:0/4066351835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6d9c0fd510 0x7f6d9c0f6d40 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f6d8c0047b0 tx=0x7f6d8c009cb0).stop 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.945+0000 7f6da086b700 1 -- 172.21.15.87:0/4066351835 shutdown_connections 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.945+0000 7f6da086b700 1 --2- 172.21.15.87:0/4066351835 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6d9c0f7400 0x7f6d9c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.945+0000 7f6da086b700 1 --2- 172.21.15.87:0/4066351835 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6d9c0fcbb0 0x7f6d9c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.945+0000 7f6da086b700 1 --2- 172.21.15.87:0/4066351835 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6d9c0fd510 0x7f6d9c0f6d40 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.945+0000 7f6da086b700 1 -- 172.21.15.87:0/4066351835 >> 172.21.15.87:0/4066351835 conn(0x7f6d9c0f2820 msgr2=0x7f6d9c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.945+0000 7f6da086b700 1 -- 172.21.15.87:0/4066351835 shutdown_connections 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.945+0000 7f6da086b700 1 -- 172.21.15.87:0/4066351835 wait complete. 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 librados: starting msgr at 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 librados: starting objecter 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 Processor -- start 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 -- start start 2024-02-23T00:34:11.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 librados: setting wanted keys 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 librados: calling monclient init 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6d9c0f7400 0x7f6d9c109270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6d9c0fcbb0 0x7f6d9c109760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6d9c0fd510 0x7f6d9c109d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c114450 con 0x7f6d9c0fd510 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c1145c0 con 0x7f6d9c0fcbb0 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.946+0000 7f6da086b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c114780 con 0x7f6d9c0f7400 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9a59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6d9c0f7400 0x7f6d9c109270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9ad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6d9c0fd510 0x7f6d9c109d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9a59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6d9c0f7400 0x7f6d9c109270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:56520/0 (socket says 172.21.15.87:56520) 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9a59c700 1 -- 172.21.15.87:0/759896622 learned_addr learned my addr 172.21.15.87:0/759896622 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d99d9b700 1 --2- 172.21.15.87:0/759896622 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6d9c0fcbb0 0x7f6d9c109760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9a59c700 1 -- 172.21.15.87:0/759896622 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6d9c0fcbb0 msgr2=0x7f6d9c109760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:11.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9a59c700 1 --2- 172.21.15.87:0/759896622 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6d9c0fcbb0 0x7f6d9c109760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9a59c700 1 -- 172.21.15.87:0/759896622 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6d9c0fd510 msgr2=0x7f6d9c109d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9a59c700 1 --2- 172.21.15.87:0/759896622 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6d9c0fd510 0x7f6d9c109d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9a59c700 1 -- 172.21.15.87:0/759896622 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d84005040 con 0x7f6d9c0f7400 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9ad9d700 1 --2- 172.21.15.87:0/759896622 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6d9c0fd510 0x7f6d9c109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d9a59c700 1 --2- 172.21.15.87:0/759896622 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6d9c0f7400 0x7f6d9c109270 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f6d84009400 tx=0x7f6d84009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d937fe700 10 client.?.objecter ms_handle_connect 0x7f6d9c0f7400 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d937fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d937fe700 1 -- 172.21.15.87:0/759896622 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84007a40 con 0x7f6d9c0f7400 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c005040 con 0x7f6d9c0f7400 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d937fe700 1 -- 172.21.15.87:0/759896622 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d8400f040 con 0x7f6d9c0f7400 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6d937fe700 1 -- 172.21.15.87:0/759896622 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84014aa0 con 0x7f6d9c0f7400 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6da086b700 10 client.37418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c114e10 con 0x7f6d9c0f7400 2024-02-23T00:34:11.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.947+0000 7f6da086b700 1 librados: init done 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.948+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c05fa60 con 0x7f6d9c0f7400 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.948+0000 7f6d937fe700 1 -- 172.21.15.87:0/759896622 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f6d8400e630 con 0x7f6d9c0f7400 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.949+0000 7f6d937fe700 1 --2- 172.21.15.87:0/759896622 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6d88062c90 0x7f6d88065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.949+0000 7f6d937fe700 1 -- 172.21.15.87:0/759896622 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(299..299 src has 38..299) v4 ==== 7908+0+0 (secure 0 0 0) 0x7f6d8407cda0 con 0x7f6d9c0f7400 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.949+0000 7f6d937fe700 10 client.37418.objecter ms_dispatch 0x7f6d9c001550 osd_map(299..299 src has 38..299) v4 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.949+0000 7f6d937fe700 3 client.37418.objecter handle_osd_map got epochs [299,299] > 0 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.949+0000 7f6d937fe700 3 client.37418.objecter handle_osd_map decoding full epoch 299 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.949+0000 7f6d99d9b700 1 --2- 172.21.15.87:0/759896622 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6d88062c90 0x7f6d88065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.949+0000 7f6d937fe700 20 client.37418.objecter dump_active .. 0 homeless 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.949+0000 7f6d937fe700 10 client.37418.objecter ms_handle_connect 0x7f6d88062c90 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.949+0000 7f6d99d9b700 1 --2- 172.21.15.87:0/759896622 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6d88062c90 0x7f6d88065150 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 rx=0x7f6d8c005010 tx=0x7f6d8c009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:10.953+0000 7f6d937fe700 1 -- 172.21.15.87:0/759896622 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6d84049f60 con 0x7f6d9c0f7400 2024-02-23T00:34:11.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:11.134+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f6d9c061440 con 0x7f6d9c0f7400 2024-02-23T00:34:11.882 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:34:11.882 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:34:11.883 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.log 2024-02-23T00:34:11.883 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.log by 10 min 8 2024-02-23T00:34:11.883 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:34:11.947 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:34:12.101 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:12.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:12.213 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:12.240 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:12] "GET /metrics HTTP/1.1" 200 187054 "" "Prometheus/2.33.4" 2024-02-23T00:34:12.242 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.286+0000 7f6d937fe700 1 -- 172.21.15.87:0/759896622 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v301) v1 ==== 91+0+0 (secure 0 0 0) 0x7f6d84050340 con 0x7f6d9c0f7400 2024-02-23T00:34:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 10 librados: watch_flush enter 2024-02-23T00:34:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 10 librados: watch_flush exit 2024-02-23T00:34:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 20 client.37418.objecter shutdown clearing up homeless session... 2024-02-23T00:34:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 10 client.37418.objecter successfully canceled tick 2024-02-23T00:34:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6d88062c90 msgr2=0x7f6d88065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 --2- 172.21.15.87:0/759896622 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6d88062c90 0x7f6d88065150 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 rx=0x7f6d8c005010 tx=0x7f6d8c009cb0).stop 2024-02-23T00:34:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6d9c0f7400 msgr2=0x7f6d9c109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 --2- 172.21.15.87:0/759896622 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6d9c0f7400 0x7f6d9c109270 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f6d84009400 tx=0x7f6d84009cb0).stop 2024-02-23T00:34:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 shutdown_connections 2024-02-23T00:34:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 --2- 172.21.15.87:0/759896622 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6d88062c90 0x7f6d88065150 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 --2- 172.21.15.87:0/759896622 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6d9c0fd510 0x7f6d9c109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 --2- 172.21.15.87:0/759896622 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6d9c0f7400 0x7f6d9c109270 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 --2- 172.21.15.87:0/759896622 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6d9c0fcbb0 0x7f6d9c109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 >> 172.21.15.87:0/759896622 conn(0x7f6d9c0f2820 msgr2=0x7f6d9c0f42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 shutdown_connections 2024-02-23T00:34:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 -- 172.21.15.87:0/759896622 wait complete. 2024-02-23T00:34:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:12.289+0000 7f6da086b700 1 librados: shutdown 2024-02-23T00:34:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:34:12.369 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:12.372 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:34:12.372 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:34:12.372 INFO:teuthology.orchestra.run.smithi087.stdout: "complaint_time": 30, 2024-02-23T00:34:12.372 INFO:teuthology.orchestra.run.smithi087.stdout: "num_blocked_ops": 0 2024-02-23T00:34:12.372 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:34:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:12.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:12.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:12.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:10.787396+0000 mgr.y (mgr.24437) 304 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:12.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:11.136242+0000 mon.c (mon.2) 373 : audit [INF] from='client.? 172.21.15.87:0/759896622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:12.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:11.136577+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:12.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.031844+0000 mon.a (mon.0) 2081 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:12.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:12.287760+0000 mon.a (mon.0) 2082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:34:12.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:12.287867+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:34:12.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:10.787396+0000 mgr.y (mgr.24437) 304 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:12.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:11.136242+0000 mon.c (mon.2) 373 : audit [INF] from='client.? 172.21.15.87:0/759896622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:12.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:11.136577+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:12.501 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.031844+0000 mon.a (mon.0) 2081 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:12.501 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:12.287760+0000 mon.a (mon.0) 2082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:34:12.501 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:12.287867+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:34:12.502 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:12.651 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:12.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:10.787396+0000 mgr.y (mgr.24437) 304 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:11.136242+0000 mon.c (mon.2) 373 : audit [INF] from='client.? 172.21.15.87:0/759896622' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:11.136577+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.031844+0000 mon.a (mon.0) 2081 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:12.287760+0000 mon.a (mon.0) 2082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:34:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:12.287867+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:34:12.754 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:12.786 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:12.806 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:12.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:12.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:12.911 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:12.938 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:13.525 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:13.530 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:11.614401+0000 mgr.y (mgr.24437) 305 : cluster [DBG] pgmap v433: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.254823+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.258340+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.287943+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:12.288958+0000 mon.b (mon.1) 260 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:12.297992+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.486084+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.488956+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.584402+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:34:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.587916+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:34:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.598914+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:34:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:12.601808+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:34:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:13.037542+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi087-82019-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi087-82019-53"}]': finished 2024-02-23T00:34:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:13.037600+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:34:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:13.037644+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-02-23T00:34:13.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:11.614401+0000 mgr.y (mgr.24437) 305 : cluster [DBG] pgmap v433: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:13.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.254823+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:34:13.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.258340+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:34:13.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.287943+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:12.288958+0000 mon.b (mon.1) 260 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:12.297992+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.486084+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.488956+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.584402+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.587916+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.598914+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:12.601808+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:13.037542+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi087-82019-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi087-82019-53"}]': finished 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:13.037600+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:34:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:13.037644+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-02-23T00:34:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:11.614401+0000 mgr.y (mgr.24437) 305 : cluster [DBG] pgmap v433: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.254823+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:34:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.258340+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.287943+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:12.288958+0000 mon.b (mon.1) 260 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:12.297992+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.486084+0000 osd.0 (osd.0) 155 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.488956+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.584402+0000 osd.3 (osd.3) 135 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.587916+0000 osd.3 (osd.3) 136 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.598914+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:12.601808+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:13.037542+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi087-82019-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi087-82019-53"}]': finished 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:13.037600+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:34:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:13.037644+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-02-23T00:34:14.091 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:14.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:14.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:34:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:34:14.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:14.227 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:14.248 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:14.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:14.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:14.352 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:14.381 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:14.401 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:14.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:14.505 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:14.532 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:14.884 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:14.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:14.916 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:14.916 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:14.988 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:15.015 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:15.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:13.205464+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:34:15.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:13.208471+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:13.614757+0000 mgr.y (mgr.24437) 306 : cluster [DBG] pgmap v436: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:13.743936+0000 osd.4 (osd.4) 137 : cluster [DBG] 63.11 scrub starts 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:13.745359+0000 osd.4 (osd.4) 138 : cluster [DBG] 63.11 scrub ok 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:14.040088+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:14.066603+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:14.066823+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:14.067505+0000 mon.b (mon.1) 262 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39"}]: dispatch 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:14.067643+0000 mon.a (mon.0) 2091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39"}]: dispatch 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:14.585741+0000 osd.3 (osd.3) 137 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:14.586927+0000 osd.3 (osd.3) 138 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:34:15.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:13.205464+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:13.208471+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:13.614757+0000 mgr.y (mgr.24437) 306 : cluster [DBG] pgmap v436: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:13.743936+0000 osd.4 (osd.4) 137 : cluster [DBG] 63.11 scrub starts 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:13.745359+0000 osd.4 (osd.4) 138 : cluster [DBG] 63.11 scrub ok 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:14.040088+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:14.066603+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:14.066823+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:14.067505+0000 mon.b (mon.1) 262 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39"}]: dispatch 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:14.067643+0000 mon.a (mon.0) 2091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39"}]: dispatch 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:14.585741+0000 osd.3 (osd.3) 137 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:34:15.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:14.586927+0000 osd.3 (osd.3) 138 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:34:15.266 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:15.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:15.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:15.398 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:13.205464+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:34:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:13.208471+0000 osd.1 (osd.1) 139 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:34:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:13.614757+0000 mgr.y (mgr.24437) 306 : cluster [DBG] pgmap v436: 246 pgs: 8 unknown, 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:13.743936+0000 osd.4 (osd.4) 137 : cluster [DBG] 63.11 scrub starts 2024-02-23T00:34:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:13.745359+0000 osd.4 (osd.4) 138 : cluster [DBG] 63.11 scrub ok 2024-02-23T00:34:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:14.040088+0000 mon.a (mon.0) 2089 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-02-23T00:34:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:14.066603+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:14.066823+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:14.067505+0000 mon.b (mon.1) 262 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39"}]: dispatch 2024-02-23T00:34:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:14.067643+0000 mon.a (mon.0) 2091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39"}]: dispatch 2024-02-23T00:34:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:14.585741+0000 osd.3 (osd.3) 137 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:34:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:14.586927+0000 osd.3 (osd.3) 138 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:34:15.495 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:15.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:15.599 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:15.630 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:15.652 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:15.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:15.755 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:15.783 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:15.835 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:15.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:15.939 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:15.971 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:15.992 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:15.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:16.048 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [==========] Running 72 tests from 4 test suites. 2024-02-23T00:34:16.048 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-02-23T00:34:16.048 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: seed 82091 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (218 ms) 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (13674 ms) 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (32 ms) 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (13924 ms total) 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 43 tests from LibRadosTwoPoolsPP 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (8797 ms) 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8044 ms) 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-02-23T00:34:16.049 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9957 ms) 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: my_snaps [4] 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: my_snaps [5,4] 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: my_snaps [6,5,4] 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting some heads 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting from clones for snap 7 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting from clones for snap 6 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting from clones for snap 5 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: promoting from clones for snap 4 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: waiting for scrubs... 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: done waiting 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46262 ms) 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9760 ms) 2024-02-23T00:34:16.050 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8000 ms) 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7998 ms) 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8000 ms) 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10245 ms) 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8773 ms) 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10001 ms) 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8003 ms) 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8243 ms) 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-02-23T00:34:16.051 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12777 ms) 2024-02-23T00:34:16.052 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-02-23T00:34:16.052 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8003 ms) 2024-02-23T00:34:16.052 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-02-23T00:34:16.052 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (7992 ms) 2024-02-23T00:34:16.052 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-02-23T00:34:16.052 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-23T00:34:16.052 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: ok, hit_set contains 188:602f83fe:::foo:head 2024-02-23T00:34:16.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:16.126 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:16.145 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:16.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:16.252 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:16.280 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:16.301 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:16.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:16.405 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:16.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:15.040403+0000 mon.a (mon.0) 2092 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39"}]': finished 2024-02-23T00:34:16.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.040454+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-02-23T00:34:16.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:15.043343+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:16.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.293080+0000 osd.6 (osd.6) 111 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:34:16.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.295173+0000 osd.6 (osd.6) 112 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:34:16.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.568002+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:34:16.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.571530+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:34:16.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:16.042353+0000 mon.a (mon.0) 2095 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi087-82019-53"}]': finished 2024-02-23T00:34:16.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:16.042437+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:16.045089+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:15.040403+0000 mon.a (mon.0) 2092 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39"}]': finished 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.040454+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:15.043343+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.293080+0000 osd.6 (osd.6) 111 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.295173+0000 osd.6 (osd.6) 112 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.568002+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.571530+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:16.042353+0000 mon.a (mon.0) 2095 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi087-82019-53"}]': finished 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:16.042437+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-02-23T00:34:16.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:16.045089+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:16.432 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:15.040403+0000 mon.a (mon.0) 2092 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-39"}]': finished 2024-02-23T00:34:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.040454+0000 mon.a (mon.0) 2093 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-02-23T00:34:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:15.043343+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.293080+0000 osd.6 (osd.6) 111 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:34:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.295173+0000 osd.6 (osd.6) 112 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:34:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.568002+0000 osd.3 (osd.3) 139 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:34:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.571530+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:34:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:16.042353+0000 mon.a (mon.0) 2095 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi087-82019-53"}]': finished 2024-02-23T00:34:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:16.042437+0000 mon.a (mon.0) 2096 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-02-23T00:34:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:16.045089+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi087-82019-53"}]: dispatch 2024-02-23T00:34:16.861 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:34:17.181 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:17.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:17.287 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.169319+0000 osd.1 (osd.1) 140 : cluster [DBG] 63.e scrub starts 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.172840+0000 osd.1 (osd.1) 141 : cluster [DBG] 63.e scrub ok 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.395269+0000 osd.5 (osd.5) 135 : cluster [DBG] 63.3 scrub starts 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.396951+0000 osd.5 (osd.5) 136 : cluster [DBG] 63.3 scrub ok 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:15.615896+0000 mgr.y (mgr.24437) 307 : cluster [DBG] pgmap v439: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:16.305105+0000 osd.6 (osd.6) 113 : cluster [DBG] 4.a scrub starts 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:16.310105+0000 osd.6 (osd.6) 114 : cluster [DBG] 4.a scrub ok 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:16.722088+0000 osd.4 (osd.4) 139 : cluster [DBG] 63.1 scrub starts 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:16.724625+0000 osd.4 (osd.4) 140 : cluster [DBG] 63.1 scrub ok 2024-02-23T00:34:17.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:17.050423+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi087-82019-53"}]': finished 2024-02-23T00:34:17.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.050518+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-02-23T00:34:17.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:17.070887+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:17.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:17.071311+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:17.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:17.085907+0000 mon.b (mon.1) 264 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:17.086314+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:17.087072+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:17.087288+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:17.088095+0000 mon.b (mon.1) 266 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi087-82019-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:17.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:17.088414+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi087-82019-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.169319+0000 osd.1 (osd.1) 140 : cluster [DBG] 63.e scrub starts 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.172840+0000 osd.1 (osd.1) 141 : cluster [DBG] 63.e scrub ok 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.395269+0000 osd.5 (osd.5) 135 : cluster [DBG] 63.3 scrub starts 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.396951+0000 osd.5 (osd.5) 136 : cluster [DBG] 63.3 scrub ok 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:15.615896+0000 mgr.y (mgr.24437) 307 : cluster [DBG] pgmap v439: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:16.305105+0000 osd.6 (osd.6) 113 : cluster [DBG] 4.a scrub starts 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:16.310105+0000 osd.6 (osd.6) 114 : cluster [DBG] 4.a scrub ok 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:16.722088+0000 osd.4 (osd.4) 139 : cluster [DBG] 63.1 scrub starts 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:16.724625+0000 osd.4 (osd.4) 140 : cluster [DBG] 63.1 scrub ok 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:17.050423+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi087-82019-53"}]': finished 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.050518+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-02-23T00:34:17.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:17.070887+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:17.318 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:17.071311+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:17.318 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:17.085907+0000 mon.b (mon.1) 264 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.318 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:17.086314+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.318 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:17.087072+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.318 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:17.087288+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.318 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:17.088095+0000 mon.b (mon.1) 266 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi087-82019-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:17.318 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:17.088414+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi087-82019-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:17.344 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.169319+0000 osd.1 (osd.1) 140 : cluster [DBG] 63.e scrub starts 2024-02-23T00:34:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.172840+0000 osd.1 (osd.1) 141 : cluster [DBG] 63.e scrub ok 2024-02-23T00:34:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.395269+0000 osd.5 (osd.5) 135 : cluster [DBG] 63.3 scrub starts 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.396951+0000 osd.5 (osd.5) 136 : cluster [DBG] 63.3 scrub ok 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:15.615896+0000 mgr.y (mgr.24437) 307 : cluster [DBG] pgmap v439: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:16.305105+0000 osd.6 (osd.6) 113 : cluster [DBG] 4.a scrub starts 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:16.310105+0000 osd.6 (osd.6) 114 : cluster [DBG] 4.a scrub ok 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:16.722088+0000 osd.4 (osd.4) 139 : cluster [DBG] 63.1 scrub starts 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:16.724625+0000 osd.4 (osd.4) 140 : cluster [DBG] 63.1 scrub ok 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:17.050423+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? 172.21.15.87:0/3617910981' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi087-82019-53"}]': finished 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.050518+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:17.070887+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:17.071311+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:17.085907+0000 mon.b (mon.1) 264 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:17.086314+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:17.087072+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:17.087288+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:17.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:17.088095+0000 mon.b (mon.1) 266 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi087-82019-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:17.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:17.088414+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi087-82019-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:18.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.033+0000 7f288bb64700 1 -- 172.21.15.87:0/243176794 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28840fdbb0 msgr2=0x7f28840fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.033+0000 7f288bb64700 1 --2- 172.21.15.87:0/243176794 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28840fdbb0 0x7f28840fdf90 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 rx=0x7f28740047b0 tx=0x7f2874009cb0).stop 2024-02-23T00:34:18.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.033+0000 7f288bb64700 1 -- 172.21.15.87:0/243176794 shutdown_connections 2024-02-23T00:34:18.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.033+0000 7f288bb64700 1 --2- 172.21.15.87:0/243176794 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28840f8420 0x7f2884104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.033+0000 7f288bb64700 1 --2- 172.21.15.87:0/243176794 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28840fdbb0 0x7f28840fdf90 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.033+0000 7f288bb64700 1 --2- 172.21.15.87:0/243176794 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28840fe510 0x7f28840f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.033+0000 7f288bb64700 1 -- 172.21.15.87:0/243176794 >> 172.21.15.87:0/243176794 conn(0x7f28840f3800 msgr2=0x7f28840f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:18.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.033+0000 7f288bb64700 1 -- 172.21.15.87:0/243176794 shutdown_connections 2024-02-23T00:34:18.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.033+0000 7f288bb64700 1 -- 172.21.15.87:0/243176794 wait complete. 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 librados: starting msgr at 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 librados: starting objecter 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 Processor -- start 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 -- start start 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 librados: setting wanted keys 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 librados: calling monclient init 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28840f8420 0x7f288410e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28840fdbb0 0x7f288410e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28840fe510 0x7f28841081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28841086b0 con 0x7f28840fdbb0 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884108820 con 0x7f28840fe510 2024-02-23T00:34:18.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f288bb64700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28841089e0 con 0x7f28840f8420 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f2889900700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28840f8420 0x7f288410e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f2889900700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28840f8420 0x7f288410e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:56322/0 (socket says 172.21.15.87:56322) 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.034+0000 7f2889900700 1 -- 172.21.15.87:0/2581813746 learned_addr learned my addr 172.21.15.87:0/2581813746 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f288a101700 1 --2- 172.21.15.87:0/2581813746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28840fe510 0x7f28841081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f28890ff700 1 --2- 172.21.15.87:0/2581813746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28840fdbb0 0x7f288410e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f2889900700 1 -- 172.21.15.87:0/2581813746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28840fe510 msgr2=0x7f28841081c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f2889900700 1 --2- 172.21.15.87:0/2581813746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28840fe510 0x7f28841081c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f2889900700 1 -- 172.21.15.87:0/2581813746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28840fdbb0 msgr2=0x7f288410e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f2889900700 1 --2- 172.21.15.87:0/2581813746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28840fdbb0 0x7f288410e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f2889900700 1 -- 172.21.15.87:0/2581813746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c0097c0 con 0x7f28840f8420 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f2889900700 1 --2- 172.21.15.87:0/2581813746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28840f8420 0x7f288410e020 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f28740044e0 tx=0x7f2874007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f287affd700 10 client.?.objecter ms_handle_connect 0x7f28840f8420 2024-02-23T00:34:18.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f287affd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:18.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874018070 con 0x7f28840f8420 2024-02-23T00:34:18.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28740093b0 con 0x7f28840f8420 2024-02-23T00:34:18.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287401dd90 con 0x7f28840f8420 2024-02-23T00:34:18.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2874005040 con 0x7f28840f8420 2024-02-23T00:34:18.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f288bb64700 10 client.37427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:18.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2884108ea0 con 0x7f28840f8420 2024-02-23T00:34:18.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.035+0000 7f288bb64700 1 librados: init done 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.036+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f288405ee80 con 0x7f28840f8420 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.037+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f287401d420 con 0x7f28840f8420 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.037+0000 7f287affd700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2870062c90 0x7f2870065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.037+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(306..306 src has 173..306) v4 ==== 7887+0+0 (secure 0 0 0) 0x7f287407e480 con 0x7f28840f8420 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.037+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 osd_map(306..306 src has 173..306) v4 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.037+0000 7f287affd700 3 client.37427.objecter handle_osd_map got epochs [306,306] > 0 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.037+0000 7f287affd700 3 client.37427.objecter handle_osd_map decoding full epoch 306 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.037+0000 7f28890ff700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2870062c90 0x7f2870065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.037+0000 7f287affd700 20 client.37427.objecter dump_active .. 0 homeless 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.037+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v306) v1 ==== 54+0+16 (secure 0 0 0) 0x7f2874085070 con 0x7f28840f8420 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f287affd700 10 client.37427.objecter ms_handle_connect 0x7f2870062c90 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f28890ff700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2870062c90 0x7f2870065150 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f2884109960 tx=0x7f288000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 10 client.37427.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 1 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6801/2153191863 conn(0x7f28840fee80 0x7f28840ff260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884107110 osd=0 3 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884107110 osd=0 3 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884107110 osd=0 3 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 1 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f2884107110 osd=0 3 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 0 1 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884107110 osd=0 4 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 10 client.37427.objecter _send_command 1 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f288406cf20 con 0x7f28840fee80 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288a101700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6801/2153191863 conn(0x7f28840fee80 0x7f28840ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f288a101700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6801/2153191863 conn(0x7f28840fee80 0x7f28840ff260 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.038+0000 7f287affd700 10 client.37427.objecter ms_handle_connect 0x7f28840fee80 2024-02-23T00:34:18.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.039+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f288406cf20 con 0x7f28840fee80 2024-02-23T00:34:18.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.039+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:34:18.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.039+0000 7f287affd700 10 client.37427.objecter _finish_command 1 = system:0 2024-02-23T00:34:18.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.039+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f2884107110 osd=0 4 2024-02-23T00:34:18.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.039+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 0 1 2024-02-23T00:34:18.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 10 client.37427.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:18.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 2 2024-02-23T00:34:18.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884107110 osd=0 3 2024-02-23T00:34:18.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884107110 osd=0 3 2024-02-23T00:34:18.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884107110 osd=0 3 2024-02-23T00:34:18.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 2 2024-02-23T00:34:18.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f2884107110 osd=0 3 2024-02-23T00:34:18.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 0 2 2024-02-23T00:34:18.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884107110 osd=0 4 2024-02-23T00:34:18.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 10 client.37427.objecter _send_command 2 2024-02-23T00:34:18.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.055+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f28840ff750 con 0x7f28840fee80 2024-02-23T00:34:18.064 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:34:18.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.062+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f28840ff750 con 0x7f28840fee80 2024-02-23T00:34:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.062+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.062+0000 7f287affd700 10 client.37427.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.062+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f2884107110 osd=0 4 2024-02-23T00:34:18.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.062+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 0 2 2024-02-23T00:34:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 10 client.37427.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 3 2024-02-23T00:34:18.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6805/685354171 conn(0x7f2884028540 0x7f28840598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f28840ff8c0 osd=1 3 2024-02-23T00:34:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f28840ff8c0 osd=1 3 2024-02-23T00:34:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f28840ff8c0 osd=1 3 2024-02-23T00:34:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 3 2024-02-23T00:34:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f28840ff8c0 osd=1 3 2024-02-23T00:34:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 1 3 2024-02-23T00:34:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f28840ff8c0 osd=1 4 2024-02-23T00:34:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 10 client.37427.objecter _send_command 3 2024-02-23T00:34:18.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.066+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2884059de0 con 0x7f2884028540 2024-02-23T00:34:18.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.068+0000 7f2889900700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6805/685354171 conn(0x7f2884028540 0x7f28840598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.069+0000 7f2889900700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6805/685354171 conn(0x7f2884028540 0x7f28840598f0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.074+0000 7f287affd700 10 client.37427.objecter ms_handle_connect 0x7f2884028540 2024-02-23T00:34:18.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.074+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2884059de0 con 0x7f2884028540 2024-02-23T00:34:18.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.075+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:34:18.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.075+0000 7f287affd700 10 client.37427.objecter _finish_command 3 = system:0 2024-02-23T00:34:18.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.075+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f28840ff8c0 osd=1 4 2024-02-23T00:34:18.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.075+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 1 3 2024-02-23T00:34:18.092 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 10 client.37427.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 4 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f28840ff8c0 osd=1 3 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f28840ff8c0 osd=1 3 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f28840ff8c0 osd=1 3 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 4 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f28840ff8c0 osd=1 3 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 1 4 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f28840ff8c0 osd=1 4 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 10 client.37427.objecter _send_command 4 2024-02-23T00:34:18.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.092+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f288411b270 con 0x7f2884028540 2024-02-23T00:34:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:18.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.093+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f288411b270 con 0x7f2884028540 2024-02-23T00:34:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.093+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.093+0000 7f287affd700 10 client.37427.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.093+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f28840ff8c0 osd=1 4 2024-02-23T00:34:18.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.093+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 1 4 2024-02-23T00:34:18.096 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:34:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 10 client.37427.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:18.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 5 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6809/3605394228 conn(0x7f28840594d0 0x7f288411e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884059360 osd=2 3 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884059360 osd=2 3 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884059360 osd=2 3 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 5 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f2884059360 osd=2 3 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 2 5 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884059360 osd=2 4 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 10 client.37427.objecter _send_command 5 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28840280a0 con 0x7f28840594d0 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f28890ff700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6809/3605394228 conn(0x7f28840594d0 0x7f288411e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f287affd700 10 client.37427.objecter ms_handle_connect 0x7f28840594d0 2024-02-23T00:34:18.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.094+0000 7f28890ff700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6809/3605394228 conn(0x7f28840594d0 0x7f288411e5d0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.095+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f28840280a0 con 0x7f28840594d0 2024-02-23T00:34:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.095+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:34:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.095+0000 7f287affd700 10 client.37427.objecter _finish_command 5 = system:0 2024-02-23T00:34:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.095+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f2884059360 osd=2 4 2024-02-23T00:34:18.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.095+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 2 5 2024-02-23T00:34:18.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 10 client.37427.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:18.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 6 2024-02-23T00:34:18.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884059360 osd=2 3 2024-02-23T00:34:18.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884059360 osd=2 3 2024-02-23T00:34:18.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884059360 osd=2 3 2024-02-23T00:34:18.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 6 2024-02-23T00:34:18.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f2884059360 osd=2 3 2024-02-23T00:34:18.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 2 6 2024-02-23T00:34:18.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884059360 osd=2 4 2024-02-23T00:34:18.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 10 client.37427.objecter _send_command 6 2024-02-23T00:34:18.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.109+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2884028380 con 0x7f28840594d0 2024-02-23T00:34:18.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2884028380 con 0x7f28840594d0 2024-02-23T00:34:18.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:18.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f287affd700 10 client.37427.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f2884059360 osd=2 4 2024-02-23T00:34:18.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 2 6 2024-02-23T00:34:18.113 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:34:18.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 10 client.37427.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:18.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 7 2024-02-23T00:34:18.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6813/128059790 conn(0x7f288411c120 0x7f2884121c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884027920 osd=3 3 2024-02-23T00:34:18.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884027920 osd=3 3 2024-02-23T00:34:18.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884027920 osd=3 3 2024-02-23T00:34:18.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 7 2024-02-23T00:34:18.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f2884027920 osd=3 3 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 3 7 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884027920 osd=3 4 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 10 client.37427.objecter _send_command 7 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2884124930 con 0x7f288411c120 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288a101700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6813/128059790 conn(0x7f288411c120 0x7f2884121c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f287affd700 10 client.37427.objecter ms_handle_connect 0x7f288411c120 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.111+0000 7f288a101700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6813/128059790 conn(0x7f288411c120 0x7f2884121c50 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.112+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2884124930 con 0x7f288411c120 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.112+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.112+0000 7f287affd700 10 client.37427.objecter _finish_command 7 = system:0 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.112+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f2884027920 osd=3 4 2024-02-23T00:34:18.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.112+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 3 7 2024-02-23T00:34:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 10 client.37427.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 8 2024-02-23T00:34:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884027920 osd=3 3 2024-02-23T00:34:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884027920 osd=3 3 2024-02-23T00:34:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884027920 osd=3 3 2024-02-23T00:34:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 8 2024-02-23T00:34:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f2884027920 osd=3 3 2024-02-23T00:34:18.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 3 8 2024-02-23T00:34:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884027920 osd=3 4 2024-02-23T00:34:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 10 client.37427.objecter _send_command 8 2024-02-23T00:34:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.129+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f288411b680 con 0x7f288411c120 2024-02-23T00:34:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f288411b680 con 0x7f288411c120 2024-02-23T00:34:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f287affd700 10 client.37427.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f2884027920 osd=3 4 2024-02-23T00:34:18.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 3 8 2024-02-23T00:34:18.133 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 10 client.37427.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 9 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6800/4175844757 conn(0x7f288411f7a0 0x7f28841232c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288411b440 osd=4 3 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b440 osd=4 3 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288411b440 osd=4 3 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 9 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f288411b440 osd=4 3 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 4 9 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b440 osd=4 4 2024-02-23T00:34:18.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 10 client.37427.objecter _send_command 9 2024-02-23T00:34:18.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.130+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f288411b620 con 0x7f288411f7a0 2024-02-23T00:34:18.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.131+0000 7f2889900700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6800/4175844757 conn(0x7f288411f7a0 0x7f28841232c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.131+0000 7f287affd700 10 client.37427.objecter ms_handle_connect 0x7f288411f7a0 2024-02-23T00:34:18.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.131+0000 7f2889900700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6800/4175844757 conn(0x7f288411f7a0 0x7f28841232c0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.132+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f288411b620 con 0x7f288411f7a0 2024-02-23T00:34:18.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.132+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:34:18.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.132+0000 7f287affd700 10 client.37427.objecter _finish_command 9 = system:0 2024-02-23T00:34:18.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.132+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f288411b440 osd=4 4 2024-02-23T00:34:18.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.132+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 4 9 2024-02-23T00:34:18.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 10 client.37427.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:18.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 10 2024-02-23T00:34:18.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288411b440 osd=4 3 2024-02-23T00:34:18.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b440 osd=4 3 2024-02-23T00:34:18.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288411b440 osd=4 3 2024-02-23T00:34:18.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 10 2024-02-23T00:34:18.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f288411b440 osd=4 3 2024-02-23T00:34:18.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 4 10 2024-02-23T00:34:18.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b440 osd=4 4 2024-02-23T00:34:18.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 10 client.37427.objecter _send_command 10 2024-02-23T00:34:18.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.146+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f288411b5e0 con 0x7f288411f7a0 2024-02-23T00:34:18.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.148+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f288411b5e0 con 0x7f288411f7a0 2024-02-23T00:34:18.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.148+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:18.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.148+0000 7f287affd700 10 client.37427.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.148+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f288411b440 osd=4 4 2024-02-23T00:34:18.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.148+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 4 10 2024-02-23T00:34:18.151 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.148+0000 7f288bb64700 10 client.37427.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 11 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2884123d20 0x7f2884124180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288411b5e0 osd=5 3 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b5e0 osd=5 3 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288411b5e0 osd=5 3 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 11 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f288411b5e0 osd=5 3 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 5 11 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b5e0 osd=5 4 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 10 client.37427.objecter _send_command 11 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f288412b2d0 con 0x7f2884123d20 2024-02-23T00:34:18.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f28890ff700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2884123d20 0x7f2884124180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f287affd700 10 client.37427.objecter ms_handle_connect 0x7f2884123d20 2024-02-23T00:34:18.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.149+0000 7f28890ff700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2884123d20 0x7f2884124180 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.150+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f288412b2d0 con 0x7f2884123d20 2024-02-23T00:34:18.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.150+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:34:18.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.150+0000 7f287affd700 10 client.37427.objecter _finish_command 11 = system:0 2024-02-23T00:34:18.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.150+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f288411b5e0 osd=5 4 2024-02-23T00:34:18.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.150+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 5 11 2024-02-23T00:34:18.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 10 client.37427.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:18.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 12 2024-02-23T00:34:18.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288411b5e0 osd=5 3 2024-02-23T00:34:18.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b5e0 osd=5 3 2024-02-23T00:34:18.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288411b5e0 osd=5 3 2024-02-23T00:34:18.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 12 2024-02-23T00:34:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f288411b5e0 osd=5 3 2024-02-23T00:34:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 5 12 2024-02-23T00:34:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b5e0 osd=5 4 2024-02-23T00:34:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 10 client.37427.objecter _send_command 12 2024-02-23T00:34:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.164+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2884128b70 con 0x7f2884123d20 2024-02-23T00:34:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2884128b70 con 0x7f2884123d20 2024-02-23T00:34:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f287affd700 10 client.37427.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f288411b5e0 osd=5 4 2024-02-23T00:34:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 5 12 2024-02-23T00:34:18.169 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:34:18.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 10 client.37427.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:18.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 13 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2884129250 0x7f288412b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884128b70 osd=6 3 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884128b70 osd=6 3 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884128b70 osd=6 3 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 13 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f2884128b70 osd=6 3 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 6 13 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884128b70 osd=6 4 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 10 client.37427.objecter _send_command 13 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.166+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f288412e430 con 0x7f2884129250 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.167+0000 7f288a101700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2884129250 0x7f288412b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.167+0000 7f287affd700 10 client.37427.objecter ms_handle_connect 0x7f2884129250 2024-02-23T00:34:18.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.167+0000 7f288a101700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2884129250 0x7f288412b660 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.168+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f288412e430 con 0x7f2884129250 2024-02-23T00:34:18.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.168+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:34:18.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.168+0000 7f287affd700 10 client.37427.objecter _finish_command 13 = system:0 2024-02-23T00:34:18.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.168+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f2884128b70 osd=6 4 2024-02-23T00:34:18.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.168+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 6 13 2024-02-23T00:34:18.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 10 client.37427.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:18.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 14 2024-02-23T00:34:18.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884128b70 osd=6 3 2024-02-23T00:34:18.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884128b70 osd=6 3 2024-02-23T00:34:18.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f2884128b70 osd=6 3 2024-02-23T00:34:18.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 14 2024-02-23T00:34:18.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f2884128b70 osd=6 3 2024-02-23T00:34:18.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 6 14 2024-02-23T00:34:18.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884128b70 osd=6 4 2024-02-23T00:34:18.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 10 client.37427.objecter _send_command 14 2024-02-23T00:34:18.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.183+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f288412e430 con 0x7f2884129250 2024-02-23T00:34:18.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f288412e430 con 0x7f2884129250 2024-02-23T00:34:18.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:18.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f287affd700 10 client.37427.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f2884128b70 osd=6 4 2024-02-23T00:34:18.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 6 14 2024-02-23T00:34:18.187 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:34:18.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 10 client.37427.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:18.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 15 2024-02-23T00:34:18.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6812/373787000 conn(0x7f288412d7c0 0x7f288412dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:18.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288412e430 osd=7 3 2024-02-23T00:34:18.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288412e430 osd=7 3 2024-02-23T00:34:18.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288412e430 osd=7 3 2024-02-23T00:34:18.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 15 2024-02-23T00:34:18.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f288412e430 osd=7 3 2024-02-23T00:34:18.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 7 15 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288412e430 osd=7 4 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 10 client.37427.objecter _send_command 15 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.184+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28841317e0 con 0x7f288412d7c0 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.185+0000 7f2889900700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6812/373787000 conn(0x7f288412d7c0 0x7f288412dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.185+0000 7f287affd700 10 client.37427.objecter ms_handle_connect 0x7f288412d7c0 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.185+0000 7f2889900700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6812/373787000 conn(0x7f288412d7c0 0x7f288412dc40 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.186+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f28841317e0 con 0x7f288412d7c0 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.186+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.186+0000 7f287affd700 10 client.37427.objecter _finish_command 15 = system:0 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.186+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f288412e430 osd=7 4 2024-02-23T00:34:18.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.186+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 7 15 2024-02-23T00:34:18.197 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:18.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 10 client.37427.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:18.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign -1 16 2024-02-23T00:34:18.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288412e430 osd=7 3 2024-02-23T00:34:18.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288412e430 osd=7 3 2024-02-23T00:34:18.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 20 client.37427.objecter _get_session s=0x7f288412e430 osd=7 3 2024-02-23T00:34:18.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 15 client.37427.objecter _session_command_op_remove -1 16 2024-02-23T00:34:18.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 20 client.37427.objecter get_session s=0x7f288412e430 osd=7 3 2024-02-23T00:34:18.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 15 client.37427.objecter _session_command_op_assign 7 16 2024-02-23T00:34:18.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288412e430 osd=7 4 2024-02-23T00:34:18.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 10 client.37427.objecter _send_command 16 2024-02-23T00:34:18.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.200+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f288412bc10 con 0x7f288412d7c0 2024-02-23T00:34:18.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f287affd700 1 -- 172.21.15.87:0/2581813746 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f288412bc10 con 0x7f288412d7c0 2024-02-23T00:34:18.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f287affd700 10 client.37427.objecter ms_dispatch 0x7f2884001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:18.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f287affd700 10 client.37427.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f287affd700 20 client.37427.objecter put_session s=0x7f288412e430 osd=7 4 2024-02-23T00:34:18.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f287affd700 15 client.37427.objecter _session_command_op_remove 7 16 2024-02-23T00:34:18.205 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:34:18.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 librados: watch_flush enter 2024-02-23T00:34:18.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 librados: watch_flush exit 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 client.37427.objecter close_session for osd.0 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6801/2153191863 conn(0x7f28840fee80 msgr2=0x7f28840ff260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6801/2153191863 conn(0x7f28840fee80 0x7f28840ff260 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884107110 osd=0 1 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 client.37427.objecter close_session for osd.1 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6805/685354171 conn(0x7f2884028540 msgr2=0x7f28840598f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6805/685354171 conn(0x7f2884028540 0x7f28840598f0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f28840ff8c0 osd=1 1 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 client.37427.objecter close_session for osd.2 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6809/3605394228 conn(0x7f28840594d0 msgr2=0x7f288411e5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6809/3605394228 conn(0x7f28840594d0 0x7f288411e5d0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884059360 osd=2 1 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 client.37427.objecter close_session for osd.3 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6813/128059790 conn(0x7f288411c120 msgr2=0x7f2884121c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6813/128059790 conn(0x7f288411c120 0x7f2884121c50 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884027920 osd=3 1 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 client.37427.objecter close_session for osd.4 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6800/4175844757 conn(0x7f288411f7a0 msgr2=0x7f28841232c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6800/4175844757 conn(0x7f288411f7a0 0x7f28841232c0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288a101700 1 -- 172.21.15.87:0/2581813746 reap_dead start 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b440 osd=4 1 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 client.37427.objecter close_session for osd.5 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2884123d20 msgr2=0x7f2884124180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2884123d20 0x7f2884124180 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288a101700 1 -- 172.21.15.87:0/2581813746 reap_dead start 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288411b5e0 osd=5 1 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 client.37427.objecter close_session for osd.6 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2884129250 msgr2=0x7f288412b660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2884129250 0x7f288412b660 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f2884128b70 osd=6 1 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 client.37427.objecter close_session for osd.7 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6812/373787000 conn(0x7f288412d7c0 msgr2=0x7f288412dc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6812/373787000 conn(0x7f288412d7c0 0x7f288412dc40 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288a101700 1 -- 172.21.15.87:0/2581813746 reap_dead start 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 20 client.37427.objecter put_session s=0x7f288412e430 osd=7 1 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 20 client.37427.objecter shutdown clearing up homeless session... 2024-02-23T00:34:18.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 10 client.37427.objecter successfully canceled tick 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2870062c90 msgr2=0x7f2870065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2870062c90 0x7f2870065150 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 rx=0x7f2884109960 tx=0x7f288000b040).stop 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28840f8420 msgr2=0x7f288410e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28840f8420 0x7f288410e020 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f28740044e0 tx=0x7f2874007f90).stop 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 shutdown_connections 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.171:6812/373787000 conn(0x7f288412d7c0 0x7f288412dc40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2870062c90 0x7f2870065150 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 --2- 172.21.15.87:0/2581813746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28840f8420 0x7f288410e020 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 >> 172.21.15.87:0/2581813746 conn(0x7f28840f3800 msgr2=0x7f28840f5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 shutdown_connections 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 -- 172.21.15.87:0/2581813746 wait complete. 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:18.203+0000 7f288bb64700 1 librados: shutdown 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.210 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.211 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.211 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.211 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.211 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.211 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.211 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:34:18.226 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:18.247 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:18.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:18.351 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:18.368 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.098355+0000 osd.1 (osd.1) 142 : cluster [DBG] 63.9 scrub starts 2024-02-23T00:34:18.368 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.107570+0000 osd.1 (osd.1) 143 : cluster [DBG] 63.9 scrub ok 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.309212+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.310101+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.435445+0000 osd.5 (osd.5) 137 : cluster [DBG] 6.d scrub starts 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.437076+0000 osd.5 (osd.5) 138 : cluster [DBG] 6.d scrub ok 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.617102+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.11 scrub starts 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.620103+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.11 scrub ok 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.759820+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.760759+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:18.038736+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? 172.21.15.87:0/2581813746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:18.057567+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:18.057673+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi087-82019-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:18.057734+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-02-23T00:34:18.369 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:18.060264+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi087-82019-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:18.370 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:18.068039+0000 mon.b (mon.1) 268 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi087-82091-6","var": "pg_num","format": "json"}]: dispatch 2024-02-23T00:34:18.370 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:18.069232+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi087-82019-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:18.370 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:18.069429+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:18.370 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:18.070240+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:18.370 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.098355+0000 osd.1 (osd.1) 142 : cluster [DBG] 63.9 scrub starts 2024-02-23T00:34:18.370 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.107570+0000 osd.1 (osd.1) 143 : cluster [DBG] 63.9 scrub ok 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.309212+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.310101+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.435445+0000 osd.5 (osd.5) 137 : cluster [DBG] 6.d scrub starts 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.437076+0000 osd.5 (osd.5) 138 : cluster [DBG] 6.d scrub ok 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.617102+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.11 scrub starts 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.620103+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.11 scrub ok 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.759820+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.760759+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:18.038736+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? 172.21.15.87:0/2581813746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:18.057567+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:18.057673+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi087-82019-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:18.057734+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:18.060264+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi087-82019-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:18.068039+0000 mon.b (mon.1) 268 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi087-82091-6","var": "pg_num","format": "json"}]: dispatch 2024-02-23T00:34:18.371 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:18.069232+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi087-82019-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:18.372 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:18.069429+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:18.372 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:18.070240+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:18.384 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.098355+0000 osd.1 (osd.1) 142 : cluster [DBG] 63.9 scrub starts 2024-02-23T00:34:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.107570+0000 osd.1 (osd.1) 143 : cluster [DBG] 63.9 scrub ok 2024-02-23T00:34:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.309212+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:34:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.310101+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.435445+0000 osd.5 (osd.5) 137 : cluster [DBG] 6.d scrub starts 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.437076+0000 osd.5 (osd.5) 138 : cluster [DBG] 6.d scrub ok 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.617102+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.11 scrub starts 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.620103+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.11 scrub ok 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.759820+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.760759+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:18.038736+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? 172.21.15.87:0/2581813746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:18.057567+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:18.057673+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi087-82019-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:18.057734+0000 mon.a (mon.0) 2106 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:18.060264+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi087-82019-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:18.068039+0000 mon.b (mon.1) 268 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi087-82091-6","var": "pg_num","format": "json"}]: dispatch 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:18.069232+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi087-82019-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:18.069429+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:18.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:18.070240+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:17.616336+0000 mgr.y (mgr.24437) 308 : cluster [DBG] pgmap v442: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:18.120510+0000 osd.1 (osd.1) 144 : cluster [DBG] 63.7 scrub starts 2024-02-23T00:34:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:18.124045+0000 osd.1 (osd.1) 145 : cluster [DBG] 63.7 scrub ok 2024-02-23T00:34:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:18.729377+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:34:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:18.732299+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:34:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:19.288654+0000 mon.a (mon.0) 2109 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:19.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:19.288725+0000 mon.a (mon.0) 2110 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:19.289779+0000 mon.b (mon.1) 270 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:19.289990+0000 mon.a (mon.0) 2111 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:17.616336+0000 mgr.y (mgr.24437) 308 : cluster [DBG] pgmap v442: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:18.120510+0000 osd.1 (osd.1) 144 : cluster [DBG] 63.7 scrub starts 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:18.124045+0000 osd.1 (osd.1) 145 : cluster [DBG] 63.7 scrub ok 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:18.729377+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:18.732299+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:19.288654+0000 mon.a (mon.0) 2109 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:19.288725+0000 mon.a (mon.0) 2110 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-02-23T00:34:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:19.289779+0000 mon.b (mon.1) 270 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-23T00:34:19.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:19.289990+0000 mon.a (mon.0) 2111 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-23T00:34:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:17.616336+0000 mgr.y (mgr.24437) 308 : cluster [DBG] pgmap v442: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:18.120510+0000 osd.1 (osd.1) 144 : cluster [DBG] 63.7 scrub starts 2024-02-23T00:34:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:18.124045+0000 osd.1 (osd.1) 145 : cluster [DBG] 63.7 scrub ok 2024-02-23T00:34:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:18.729377+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:34:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:18.732299+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:34:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:19.288654+0000 mon.a (mon.0) 2109 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:19.288725+0000 mon.a (mon.0) 2110 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-02-23T00:34:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:19.289779+0000 mon.b (mon.1) 270 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-23T00:34:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:19.289990+0000 mon.a (mon.0) 2111 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-23T00:34:20.337 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:20.337 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:20.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:20.442 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:20.473 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:20.494 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:20.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:19.396787+0000 osd.5 (osd.5) 139 : cluster [DBG] 63.4 scrub starts 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:19.398059+0000 osd.5 (osd.5) 140 : cluster [DBG] 63.4 scrub ok 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:19.569756+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:19.572805+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:19.691979+0000 osd.4 (osd.4) 141 : cluster [DBG] 63.5 scrub starts 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:19.695444+0000 osd.4 (osd.4) 142 : cluster [DBG] 63.5 scrub ok 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:20.056279+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi087-82019-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi087-82019-54"}]': finished 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:20.056398+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_count","val": "8"}]': finished 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:20.056471+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-02-23T00:34:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:20.061832+0000 mon.b (mon.1) 271 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:20.064089+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:19.396787+0000 osd.5 (osd.5) 139 : cluster [DBG] 63.4 scrub starts 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:19.398059+0000 osd.5 (osd.5) 140 : cluster [DBG] 63.4 scrub ok 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:19.569756+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:19.572805+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:19.691979+0000 osd.4 (osd.4) 141 : cluster [DBG] 63.5 scrub starts 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:19.695444+0000 osd.4 (osd.4) 142 : cluster [DBG] 63.5 scrub ok 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:20.056279+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi087-82019-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi087-82019-54"}]': finished 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:20.056398+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_count","val": "8"}]': finished 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:20.056471+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-02-23T00:34:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:20.061832+0000 mon.b (mon.1) 271 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:20.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:20.064089+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:20.598 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:19.396787+0000 osd.5 (osd.5) 139 : cluster [DBG] 63.4 scrub starts 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:19.398059+0000 osd.5 (osd.5) 140 : cluster [DBG] 63.4 scrub ok 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:19.569756+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:19.572805+0000 osd.0 (osd.0) 158 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:19.691979+0000 osd.4 (osd.4) 141 : cluster [DBG] 63.5 scrub starts 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:19.695444+0000 osd.4 (osd.4) 142 : cluster [DBG] 63.5 scrub ok 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:20.056279+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi087-82019-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi087-82019-54"}]': finished 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:20.056398+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_count","val": "8"}]': finished 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:20.056471+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:20.061832+0000 mon.b (mon.1) 271 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:20.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:20.064089+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:20.625 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:20.644 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:20.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:20.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.658+0000 7f36ec345700 1 -- 172.21.15.87:0/1558843419 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 msgr2=0x7f36e40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:20.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.658+0000 7f36ec345700 1 --2- 172.21.15.87:0/1558843419 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 0x7f36e40f90f0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f36e0004870 tx=0x7f36e0009cb0).stop 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.658+0000 7f36ec345700 1 -- 172.21.15.87:0/1558843419 shutdown_connections 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.658+0000 7f36ec345700 1 --2- 172.21.15.87:0/1558843419 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36e4102270 0x7f36e4104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.658+0000 7f36ec345700 1 --2- 172.21.15.87:0/1558843419 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 0x7f36e40f90f0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.658+0000 7f36ec345700 1 --2- 172.21.15.87:0/1558843419 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36e40f9670 0x7f36e4101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.658+0000 7f36ec345700 1 -- 172.21.15.87:0/1558843419 >> 172.21.15.87:0/1558843419 conn(0x7f36e40f2820 msgr2=0x7f36e40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 -- 172.21.15.87:0/1558843419 shutdown_connections 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 -- 172.21.15.87:0/1558843419 wait complete. 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 librados: starting msgr at 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 librados: starting objecter 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 Processor -- start 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 -- start start 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 librados: setting wanted keys 2024-02-23T00:34:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 librados: calling monclient init 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 0x7f36e40f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36e40f9670 0x7f36e40f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36e4102270 0x7f36e40f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e4116660 con 0x7f36e4102270 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e41167d0 con 0x7f36e40f9670 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ec345700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e4116990 con 0x7f36e40f8d10 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ea0e1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 0x7f36e40f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ea8e2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36e4102270 0x7f36e40f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ea0e1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 0x7f36e40f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:56334/0 (socket says 172.21.15.87:56334) 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.659+0000 7f36ea0e1700 1 -- 172.21.15.87:0/1188511165 learned_addr learned my addr 172.21.15.87:0/1188511165 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36e98e0700 1 --2- 172.21.15.87:0/1188511165 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36e40f9670 0x7f36e40f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:20.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ea0e1700 1 -- 172.21.15.87:0/1188511165 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36e40f9670 msgr2=0x7f36e40f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ea0e1700 1 --2- 172.21.15.87:0/1188511165 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36e40f9670 0x7f36e40f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ea0e1700 1 -- 172.21.15.87:0/1188511165 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36e4102270 msgr2=0x7f36e40f76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ea0e1700 1 --2- 172.21.15.87:0/1188511165 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36e4102270 0x7f36e40f76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ea0e1700 1 -- 172.21.15.87:0/1188511165 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc0097c0 con 0x7f36e40f8d10 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ea8e2700 1 --2- 172.21.15.87:0/1188511165 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36e4102270 0x7f36e40f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ea0e1700 1 --2- 172.21.15.87:0/1188511165 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 0x7f36e40f6cc0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7f36e0009eb0 tx=0x7f36e000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36db7fe700 10 client.?.objecter ms_handle_connect 0x7f36e40f8d10 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36db7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36db7fe700 1 -- 172.21.15.87:0/1188511165 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e000e070 con 0x7f36e40f8d10 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36db7fe700 1 -- 172.21.15.87:0/1188511165 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36e0013da0 con 0x7f36e40f8d10 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36db7fe700 1 -- 172.21.15.87:0/1188511165 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e0019070 con 0x7f36e40f8d10 2024-02-23T00:34:20.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e0005040 con 0x7f36e40f8d10 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ec345700 10 client.37433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e4116e80 con 0x7f36e40f8d10 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.660+0000 7f36ec345700 1 librados: init done 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.661+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e405fa60 con 0x7f36e40f8d10 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.661+0000 7f36db7fe700 1 -- 172.21.15.87:0/1188511165 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f36e001e420 con 0x7f36e40f8d10 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.662+0000 7f36db7fe700 1 --2- 172.21.15.87:0/1188511165 >> v2:172.21.15.87:6800/1231597463 conn(0x7f36d0062c90 0x7f36d0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.662+0000 7f36db7fe700 1 -- 172.21.15.87:0/1188511165 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(309..309 src has 173..309) v4 ==== 8700+0+0 (secure 0 0 0) 0x7f36e007df90 con 0x7f36e40f8d10 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.662+0000 7f36db7fe700 10 client.37433.objecter ms_dispatch 0x7f36e4001550 osd_map(309..309 src has 173..309) v4 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.662+0000 7f36db7fe700 3 client.37433.objecter handle_osd_map got epochs [309,309] > 0 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.662+0000 7f36db7fe700 3 client.37433.objecter handle_osd_map decoding full epoch 309 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.662+0000 7f36e98e0700 1 --2- 172.21.15.87:0/1188511165 >> v2:172.21.15.87:6800/1231597463 conn(0x7f36d0062c90 0x7f36d0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.662+0000 7f36db7fe700 20 client.37433.objecter dump_active .. 0 homeless 2024-02-23T00:34:20.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.662+0000 7f36db7fe700 10 client.37433.objecter ms_handle_connect 0x7f36d0062c90 2024-02-23T00:34:20.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.662+0000 7f36e98e0700 1 --2- 172.21.15.87:0/1188511165 >> v2:172.21.15.87:6800/1231597463 conn(0x7f36d0062c90 0x7f36d0065150 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7f36e40f8460 tx=0x7f36d400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:20.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.670+0000 7f36db7fe700 1 -- 172.21.15.87:0/1188511165 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f36e00acf60 con 0x7f36e40f8d10 2024-02-23T00:34:20.748 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:20.775 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:20.794 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:20.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:20.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.838+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36e40f81a0 con 0x7f36d0062c90 2024-02-23T00:34:20.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.855+0000 7f36db7fe700 1 -- 172.21.15.87:0/1188511165 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+616784 (secure 0 0 0) 0x7f36e40f81a0 con 0x7f36d0062c90 2024-02-23T00:34:20.864 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:34:20.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.859+0000 7f36ec345700 10 librados: watch_flush enter 2024-02-23T00:34:20.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.859+0000 7f36ec345700 10 librados: watch_flush exit 2024-02-23T00:34:20.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.859+0000 7f36ec345700 20 client.37433.objecter shutdown clearing up homeless session... 2024-02-23T00:34:20.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.859+0000 7f36ec345700 10 client.37433.objecter successfully canceled tick 2024-02-23T00:34:20.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.859+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 >> v2:172.21.15.87:6800/1231597463 conn(0x7f36d0062c90 msgr2=0x7f36d0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:20.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.859+0000 7f36ec345700 1 --2- 172.21.15.87:0/1188511165 >> v2:172.21.15.87:6800/1231597463 conn(0x7f36d0062c90 0x7f36d0065150 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 rx=0x7f36e40f8460 tx=0x7f36d400b040).stop 2024-02-23T00:34:20.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.859+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 msgr2=0x7f36e40f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:20.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.859+0000 7f36ec345700 1 --2- 172.21.15.87:0/1188511165 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 0x7f36e40f6cc0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7f36e0009eb0 tx=0x7f36e000b040).stop 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.860+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 shutdown_connections 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.860+0000 7f36ec345700 1 --2- 172.21.15.87:0/1188511165 >> v2:172.21.15.87:6800/1231597463 conn(0x7f36d0062c90 0x7f36d0065150 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.860+0000 7f36ec345700 1 --2- 172.21.15.87:0/1188511165 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f36e4102270 0x7f36e40f76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.860+0000 7f36ec345700 1 --2- 172.21.15.87:0/1188511165 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f36e40f8d10 0x7f36e40f6cc0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.860+0000 7f36ec345700 1 --2- 172.21.15.87:0/1188511165 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f36e40f9670 0x7f36e40f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.860+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 >> 172.21.15.87:0/1188511165 conn(0x7f36e40f2820 msgr2=0x7f36e40f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.860+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 shutdown_connections 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.860+0000 7f36ec345700 1 -- 172.21.15.87:0/1188511165 wait complete. 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:20.860+0000 7f36ec345700 1 librados: shutdown 2024-02-23T00:34:20.871 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:34:20.898 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:20.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:34:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:34:20.924 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:20.942 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:20.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:21.046 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:21.078 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:21.096 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:21.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:21.199 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:21.228 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:21.492 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:19.616755+0000 mgr.y (mgr.24437) 309 : cluster [DBG] pgmap v445: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:21.492 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:21.056727+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:34:21.492 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:21.056841+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-02-23T00:34:21.492 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:21.061472+0000 mon.b (mon.1) 272 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-23T00:34:21.492 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:21.061868+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-23T00:34:21.492 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:19.616755+0000 mgr.y (mgr.24437) 309 : cluster [DBG] pgmap v445: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:21.493 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:21.056727+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:34:21.493 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:21.056841+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-02-23T00:34:21.493 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:21.061472+0000 mon.b (mon.1) 272 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-23T00:34:21.493 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:21.061868+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-23T00:34:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:19.616755+0000 mgr.y (mgr.24437) 309 : cluster [DBG] pgmap v445: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:21.056727+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:34:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:21.056841+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-02-23T00:34:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:21.061472+0000 mon.b (mon.1) 272 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-23T00:34:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:21.061868+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-23T00:34:22.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9000 ms) 2024-02-23T00:34:22.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-02-23T00:34:22.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg_num = 32 2024-02-23T00:34:22.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 0 ls 1708648463,0 2024-02-23T00:34:22.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 1 ls 1708648463,0 2024-02-23T00:34:22.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 2 ls 1708648463,0 2024-02-23T00:34:22.282 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 3 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 4 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 5 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 6 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 7 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 8 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 9 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 10 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 11 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 12 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 13 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 14 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 15 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 16 ls 1708648463,0 2024-02-23T00:34:22.283 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 17 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 18 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 19 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 20 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 21 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 22 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 23 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 24 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 25 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 26 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 27 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 28 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 29 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 30 ls 1708648463,0 2024-02-23T00:34:22.284 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg 31 ls 1708648463,0 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: pg_num = 32 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:6cac518f:::0:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:02547ec2:::1:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:f905c69b:::2:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:cfc208b3:::3:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d83876eb:::4:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:b29083e3:::5:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c4fdafeb:::6:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:5c6b0b28:::7:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:bd63b0f1:::8:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:e960b815:::9:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:52ea6a34:::10:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:89d3ae78:::11:head 2024-02-23T00:34:22.285 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:de5d7c5f:::12:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:566253c9:::13:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:62a1935d:::14:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:863748b0:::15:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:3958e169:::16:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:4d4dabf9:::17:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:8391935d:::18:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:28883081:::19:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:69259c59:::20:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:4bdb80b7:::21:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:a11c5d71:::22:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:271af37b:::23:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:95b121be:::24:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:58d1031b:::25:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:0a050783:::26:head 2024-02-23T00:34:22.286 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c709704c:::27:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:cbe56eaf:::28:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:86b4b162:::29:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:70d89383:::30:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:dd450c7c:::31:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:6d5729b1:::32:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c388f3fb:::33:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:56cfea31:::34:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:9dbc1bf7:::35:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:40b74ccd:::36:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:4d5aaf42:::37:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:920f362c:::38:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:6cc53222:::39:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:9cad833f:::40:head 2024-02-23T00:34:22.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:1ea84d41:::41:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c4480ef6:::42:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:a694361e:::43:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d1bd33e9:::44:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:ddc2cd5d:::45:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:2b782207:::46:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:7b187fca:::47:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:90ecdf6f:::48:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:a5ed95fe:::49:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:ea0eaa55:::50:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:f33ef17b:::51:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:a0d1b2f6:::52:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:60c5229e:::53:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:edcbc575:::54:head 2024-02-23T00:34:22.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:102cf253:::55:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:efb7fb0b:::56:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:50d0a326:::57:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d4dc5daf:::58:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:3a130462:::59:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:ec87ed71:::60:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d5bc9454:::61:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:3ddfe313:::62:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:7c2816b9:::63:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:47e00e4d:::64:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c6410c18:::65:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:b48ed237:::66:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:cd63ad31:::67:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:b179e92b:::68:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:0d9f741a:::69:head 2024-02-23T00:34:22.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:6d3352ae:::70:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c6d5c19e:::71:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:bc4729c3:::72:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:77e930b9:::73:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:0abeecfd:::74:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:b7c37e15:::75:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:b6378398:::76:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:02bd68de:::77:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:cc795d2d:::78:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:630d4fea:::79:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:e0d29ef5:::80:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:fd6f13d2:::81:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:606461d5:::82:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:eadbdc43:::83:head 2024-02-23T00:34:22.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:8761d0bb:::84:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:9ef0186f:::85:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:e0d41294:::86:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:961de695:::87:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:1423148f:::88:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:633a8fa2:::89:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:a8653809:::90:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:3dac8b33:::91:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:35aad435:::92:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:f6dcc343:::93:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:dbbdad87:::94:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:1cb48ce0:::95:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:03cd461c:::96:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:17a4ea99:::97:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:9993c9a7:::98:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:6394211c:::99:head 2024-02-23T00:34:22.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:94c7ae57:::100:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:6fdee5bb:::101:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:9a477fd1:::102:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:eb850916:::103:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:affc56b9:::104:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:b42dc814:::105:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:f319f8f0:::106:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:9a40b9de:::107:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:8b524f28:::108:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:e3de589f:::109:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:90f90a5b:::110:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:a7b4f1d7:::111:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:af51766e:::112:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:b6f90bd1:::113:head 2024-02-23T00:34:22.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:e0261208:::114:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c9569ef7:::115:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:61bebe50:::116:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:fe93412b:::117:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d3d38bee:::118:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:3100ba0c:::119:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d0560ada:::120:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:f0ea8b35:::121:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:766f231a:::122:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:a07a2582:::123:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:bd7c6b3a:::124:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:fb2ddaff:::125:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:4408e1fe:::126:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:ee1df7a7:::127:head 2024-02-23T00:34:22.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c3002909:::128:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:4f48ffa9:::129:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:edf38733:::130:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c08425c0:::131:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:5f902d98:::132:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:41ea2c93:::133:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:813cee13:::134:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:0131818d:::135:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:26ba5a85:::136:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:381b8a5a:::137:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:28797e47:::138:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:bfca7f22:::139:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:36807075:::140:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:80b03975:::141:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:5c15709b:::142:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:f39ea15e:::143:head 2024-02-23T00:34:22.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:ea992956:::144:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:48887b1c:::145:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:9f24a9dd:::146:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:987f100b:::147:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d2dd3581:::148:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:7fed1808:::149:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c80b70e9:::150:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:85ed90f9:::151:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:36428b24:::152:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d044c34a:::153:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:7c18bf58:::154:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d1c21232:::155:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:a7a3c575:::156:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:87da0633:::157:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d5ac3822:::158:head 2024-02-23T00:34:22.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:3f20522d:::159:head 2024-02-23T00:34:22.296 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:6ca26563:::160:head 2024-02-23T00:34:22.296 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:532ce135:::161:head 2024-02-23T00:34:22.296 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:c78863e6:::162:head 2024-02-23T00:34:22.296 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:7462ddf6:::163:head 2024-02-23T00:34:22.296 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:5d165639:::164:head 2024-02-23T00:34:22.296 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:f43765fc:::165:head 2024-02-23T00:34:22.342 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:34:22.382 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:22.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:22.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:22.483 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:22] "GET /metrics HTTP/1.1" 200 187054 "" "Prometheus/2.33.4" 2024-02-23T00:34:22.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:20.797758+0000 mgr.y (mgr.24437) 310 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:22.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:20.839168+0000 mgr.y (mgr.24437) 311 : audit [DBG] from='client.37433 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:34:22.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:21.441137+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.a scrub starts 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:21.441954+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.a scrub ok 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:21.574227+0000 osd.0 (osd.0) 159 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:21.576285+0000 osd.0 (osd.0) 160 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:21.677553+0000 osd.4 (osd.4) 143 : cluster [DBG] 63.15 scrub starts 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:21.681181+0000 osd.4 (osd.4) 144 : cluster [DBG] 63.15 scrub ok 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:22.016615+0000 osd.1 (osd.1) 146 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:22.019526+0000 osd.1 (osd.1) 147 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:22.047960+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:22.048060+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:22.051146+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:22.051295+0000 mon.a (mon.0) 2121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:22.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:22.281194+0000 mon.b (mon.1) 274 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi087-82091-41","var": "pg_num","format": "json"}]: dispatch 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:20.797758+0000 mgr.y (mgr.24437) 310 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:20.839168+0000 mgr.y (mgr.24437) 311 : audit [DBG] from='client.37433 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:21.441137+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.a scrub starts 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:21.441954+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.a scrub ok 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:21.574227+0000 osd.0 (osd.0) 159 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:21.576285+0000 osd.0 (osd.0) 160 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:21.677553+0000 osd.4 (osd.4) 143 : cluster [DBG] 63.15 scrub starts 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:21.681181+0000 osd.4 (osd.4) 144 : cluster [DBG] 63.15 scrub ok 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:22.016615+0000 osd.1 (osd.1) 146 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:22.019526+0000 osd.1 (osd.1) 147 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:22.047960+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:22.048060+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:22.051146+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:22.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:22.051295+0000 mon.a (mon.0) 2121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:22.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:22.281194+0000 mon.b (mon.1) 274 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi087-82091-41","var": "pg_num","format": "json"}]: dispatch 2024-02-23T00:34:22.487 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:22.513 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:22.531 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:22.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:22.634 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:22.663 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:20.797758+0000 mgr.y (mgr.24437) 310 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:20.839168+0000 mgr.y (mgr.24437) 311 : audit [DBG] from='client.37433 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:34:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:21.441137+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.a scrub starts 2024-02-23T00:34:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:21.441954+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.a scrub ok 2024-02-23T00:34:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:21.574227+0000 osd.0 (osd.0) 159 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:34:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:21.576285+0000 osd.0 (osd.0) 160 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:34:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:21.677553+0000 osd.4 (osd.4) 143 : cluster [DBG] 63.15 scrub starts 2024-02-23T00:34:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:21.681181+0000 osd.4 (osd.4) 144 : cluster [DBG] 63.15 scrub ok 2024-02-23T00:34:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:22.016615+0000 osd.1 (osd.1) 146 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:34:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:22.019526+0000 osd.1 (osd.1) 147 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:34:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:22.047960+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-23T00:34:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:22.048060+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-02-23T00:34:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:22.051146+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:22.051295+0000 mon.a (mon.0) 2121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:22.281194+0000 mon.b (mon.1) 274 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi087-82091-41","var": "pg_num","format": "json"}]: dispatch 2024-02-23T00:34:22.982 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-02-23T00:34:20.053301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007021,"num_objects":348,"num_object_clones":2,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":1307,"num_read_kb":938,"num_write":1171,"num_write_kb":4496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2552,"ondisk_log_size":2552,"up":618,"acting":618,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":718,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2486912,"kb_used_data":51708,"kb_used_omap":0,"kb_used_meta":2435072,"kb_avail":747474304,"statfs":{"total":767960285184,"available":765413687296,"internally_reserved":0,"allocated":52948992,"data_stored":34932631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2493513728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013681"},"pg_stats":[{"pgid":"190.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"191.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:20.050971+0000","last_change":"2024-02-23T00:34:20.050971+0000","last_active":"2024-02-23T00:34:20.050971+0000","last_peered":"2024-02-23T00:34:20.050971+0000","last_clean":"2024-02-23T00:34:20.050971+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:20.050971+0000","last_undegraded":"2024-02-23T00:34:20.050971+0000","last_fullsized":"2024-02-23T00:34:20.050971+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_clean_scrub_stamp":"2024-02-23T00:34:20.050971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"191.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:20.050971+0000","last_change":"2024-02-23T00:34:20.050971+0000","last_active":"2024-02-23T00:34:20.050971+0000","last_peered":"2024-02-23T00:34:20.050971+0000","last_clean":"2024-02-23T00:34:20.050971+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:20.050971+0000","last_undegraded":"2024-02-23T00:34:20.050971+0000","last_fullsized":"2024-02-23T00:34:20.050971+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_clean_scrub_stamp":"2024-02-23T00:34:20.050971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"191.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:20.050971+0000","last_change":"2024-02-23T00:34:20.050971+0000","last_active":"2024-02-23T00:34:20.050971+0000","last_peered":"2024-02-23T00:34:20.050971+0000","last_clean":"2024-02-23T00:34:20.050971+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:20.050971+0000","last_undegraded":"2024-02-23T00:34:20.050971+0000","last_fullsized":"2024-02-23T00:34:20.050971+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_clean_scrub_stamp":"2024-02-23T00:34:20.050971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"191.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:20.050971+0000","last_change":"2024-02-23T00:34:20.050971+0000","last_active":"2024-02-23T00:34:20.050971+0000","last_peered":"2024-02-23T00:34:20.050971+0000","last_clean":"2024-02-23T00:34:20.050971+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:20.050971+0000","last_undegraded":"2024-02-23T00:34:20.050971+0000","last_fullsized":"2024-02-23T00:34:20.050971+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_clean_scrub_stamp":"2024-02-23T00:34:20.050971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"191.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:20.050971+0000","last_change":"2024-02-23T00:34:20.050971+0000","last_active":"2024-02-23T00:34:20.050971+0000","last_peered":"2024-02-23T00:34:20.050971+0000","last_clean":"2024-02-23T00:34:20.050971+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:20.050971+0000","last_undegraded":"2024-02-23T00:34:20.050971+0000","last_fullsized":"2024-02-23T00:34:20.050971+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_clean_scrub_stamp":"2024-02-23T00:34:20.050971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"191.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:20.050971+0000","last_change":"2024-02-23T00:34:20.050971+0000","last_active":"2024-02-23T00:34:20.050971+0000","last_peered":"2024-02-23T00:34:20.050971+0000","last_clean":"2024-02-23T00:34:20.050971+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:20.050971+0000","last_undegraded":"2024-02-23T00:34:20.050971+0000","last_fullsized":"2024-02-23T00:34:20.050971+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_clean_scrub_stamp":"2024-02-23T00:34:20.050971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"191.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:20.050971+0000","last_change":"2024-02-23T00:34:20.050971+0000","last_active":"2024-02-23T00:34:20.050971+0000","last_peered":"2024-02-23T00:34:20.050971+0000","last_clean":"2024-02-23T00:34:20.050971+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:20.050971+0000","last_undegraded":"2024-02-23T00:34:20.050971+0000","last_fullsized":"2024-02-23T00:34:20.050971+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_clean_scrub_stamp":"2024-02-23T00:34:20.050971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"191.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:20.050971+0000","last_change":"2024-02-23T00:34:20.050971+0000","last_active":"2024-02-23T00:34:20.050971+0000","last_peered":"2024-02-23T00:34:20.050971+0000","last_clean":"2024-02-23T00:34:20.050971+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:20.050971+0000","last_undegraded":"2024-02-23T00:34:20.050971+0000","last_fullsized":"2024-02-23T00:34:20.050971+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:20.050971+0000","last_clean_scrub_stamp":"2024-02-23T00:34:20.050971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"190.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:34:17.040457+0000","last_change":"2024-02-23T00:34:17.040457+0000","last_active":"2024-02-23T00:34:17.040457+0000","last_peered":"2024-02-23T00:34:17.040457+0000","last_clean":"2024-02-23T00:34:17.040457+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:34:17.040457+0000","last_undegraded":"2024-02-23T00:34:17.040457+0000","last_fullsized":"2024-02-23T00:34:17.040457+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:17.040457+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.040457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"63.0","version":"222'68","reported_seq":298,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.042701+0000","last_change":"2024-02-23T00:33:48.242313+0000","last_active":"2024-02-23T00:34:14.042701+0000","last_peered":"2024-02-23T00:34:14.042701+0000","last_clean":"2024-02-23T00:34:14.042701+0000","last_became_active":"2024-02-23T00:31:06.330477+0000","last_became_peered":"2024-02-23T00:31:06.330477+0000","last_unstale":"2024-02-23T00:34:14.042701+0000","last_undegraded":"2024-02-23T00:34:14.042701+0000","last_fullsized":"2024-02-23T00:34:14.042701+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1","version":"226'85","reported_seq":311,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045039+0000","last_change":"2024-02-23T00:33:48.038349+0000","last_active":"2024-02-23T00:34:14.045039+0000","last_peered":"2024-02-23T00:34:14.045039+0000","last_clean":"2024-02-23T00:34:14.045039+0000","last_became_active":"2024-02-23T00:31:06.258140+0000","last_became_peered":"2024-02-23T00:31:06.258140+0000","last_unstale":"2024-02-23T00:34:14.045039+0000","last_undegraded":"2024-02-23T00:34:14.045039+0000","last_fullsized":"2024-02-23T00:34:14.045039+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.2","version":"264'20","reported_seq":261,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043737+0000","last_change":"2024-02-23T00:33:48.242548+0000","last_active":"2024-02-23T00:34:14.043737+0000","last_peered":"2024-02-23T00:34:14.043737+0000","last_clean":"2024-02-23T00:34:14.043737+0000","last_became_active":"2024-02-23T00:31:06.329726+0000","last_became_peered":"2024-02-23T00:31:06.329726+0000","last_unstale":"2024-02-23T00:34:14.043737+0000","last_undegraded":"2024-02-23T00:34:14.043737+0000","last_fullsized":"2024-02-23T00:34:14.043737+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.3","version":"248'25","reported_seq":269,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.397037+0000","last_change":"2024-02-23T00:34:15.397037+0000","last_active":"2024-02-23T00:34:15.397037+0000","last_peered":"2024-02-23T00:34:15.397037+0000","last_clean":"2024-02-23T00:34:15.397037+0000","last_became_active":"2024-02-23T00:31:06.258997+0000","last_became_peered":"2024-02-23T00:31:06.258997+0000","last_unstale":"2024-02-23T00:34:15.397037+0000","last_undegraded":"2024-02-23T00:34:15.397037+0000","last_fullsized":"2024-02-23T00:34:15.397037+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'25","last_scrub_stamp":"2024-02-23T00:34:15.396956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.396956+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.4","version":"236'39","reported_seq":271,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045209+0000","last_change":"2024-02-23T00:33:48.238511+0000","last_active":"2024-02-23T00:34:14.045209+0000","last_peered":"2024-02-23T00:34:14.045209+0000","last_clean":"2024-02-23T00:34:14.045209+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:34:14.045209+0000","last_undegraded":"2024-02-23T00:34:14.045209+0000","last_fullsized":"2024-02-23T00:34:14.045209+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.5","version":"234'124","reported_seq":349,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046721+0000","last_change":"2024-02-23T00:33:48.037982+0000","last_active":"2024-02-23T00:34:14.046721+0000","last_peered":"2024-02-23T00:34:14.046721+0000","last_clean":"2024-02-23T00:34:14.046721+0000","last_became_active":"2024-02-23T00:31:06.258422+0000","last_became_peered":"2024-02-23T00:31:06.258422+0000","last_unstale":"2024-02-23T00:34:14.046721+0000","last_undegraded":"2024-02-23T00:34:14.046721+0000","last_fullsized":"2024-02-23T00:34:14.046721+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":124,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.6","version":"206'51","reported_seq":290,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039985+0000","last_change":"2024-02-23T00:33:48.043154+0000","last_active":"2024-02-23T00:34:14.039985+0000","last_peered":"2024-02-23T00:34:14.039985+0000","last_clean":"2024-02-23T00:34:14.039985+0000","last_became_active":"2024-02-23T00:31:06.313441+0000","last_became_peered":"2024-02-23T00:31:06.313441+0000","last_unstale":"2024-02-23T00:34:14.039985+0000","last_undegraded":"2024-02-23T00:34:14.039985+0000","last_fullsized":"2024-02-23T00:34:14.039985+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:32:48.841846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.841846+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.7","version":"229'92","reported_seq":317,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044573+0000","last_change":"2024-02-23T00:33:48.247391+0000","last_active":"2024-02-23T00:34:14.044573+0000","last_peered":"2024-02-23T00:34:14.044573+0000","last_clean":"2024-02-23T00:34:14.044573+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:34:14.044573+0000","last_undegraded":"2024-02-23T00:34:14.044573+0000","last_fullsized":"2024-02-23T00:34:14.044573+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":92,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.8","version":"219'70","reported_seq":304,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039325+0000","last_change":"2024-02-23T00:33:48.292128+0000","last_active":"2024-02-23T00:34:14.039325+0000","last_peered":"2024-02-23T00:34:14.039325+0000","last_clean":"2024-02-23T00:34:14.039325+0000","last_became_active":"2024-02-23T00:31:06.363442+0000","last_became_peered":"2024-02-23T00:31:06.363442+0000","last_unstale":"2024-02-23T00:34:14.039325+0000","last_undegraded":"2024-02-23T00:34:14.039325+0000","last_fullsized":"2024-02-23T00:34:14.039325+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.9","version":"208'17","reported_seq":255,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.042713+0000","last_change":"2024-02-23T00:33:48.243032+0000","last_active":"2024-02-23T00:34:14.042713+0000","last_peered":"2024-02-23T00:34:14.042713+0000","last_clean":"2024-02-23T00:34:14.042713+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:34:14.042713+0000","last_undegraded":"2024-02-23T00:34:14.042713+0000","last_fullsized":"2024-02-23T00:34:14.042713+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.a","version":"248'22","reported_seq":268,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045212+0000","last_change":"2024-02-23T00:33:48.236273+0000","last_active":"2024-02-23T00:34:14.045212+0000","last_peered":"2024-02-23T00:34:14.045212+0000","last_clean":"2024-02-23T00:34:14.045212+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:34:14.045212+0000","last_undegraded":"2024-02-23T00:34:14.045212+0000","last_fullsized":"2024-02-23T00:34:14.045212+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'22","last_scrub_stamp":"2024-02-23T00:33:34.362545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.362545+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.b","version":"202'34","reported_seq":274,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055621+0000","last_change":"2024-02-23T00:33:48.299421+0000","last_active":"2024-02-23T00:34:14.055621+0000","last_peered":"2024-02-23T00:34:14.055621+0000","last_clean":"2024-02-23T00:34:14.055621+0000","last_became_active":"2024-02-23T00:31:06.294667+0000","last_became_peered":"2024-02-23T00:31:06.294667+0000","last_unstale":"2024-02-23T00:34:14.055621+0000","last_undegraded":"2024-02-23T00:34:14.055621+0000","last_fullsized":"2024-02-23T00:34:14.055621+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-23T00:33:37.391203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.391203+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.c","version":"208'68","reported_seq":300,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039978+0000","last_change":"2024-02-23T00:33:48.290340+0000","last_active":"2024-02-23T00:34:14.039978+0000","last_peered":"2024-02-23T00:34:14.039978+0000","last_clean":"2024-02-23T00:34:14.039978+0000","last_became_active":"2024-02-23T00:31:06.233844+0000","last_became_peered":"2024-02-23T00:31:06.233844+0000","last_unstale":"2024-02-23T00:34:14.039978+0000","last_undegraded":"2024-02-23T00:34:14.039978+0000","last_fullsized":"2024-02-23T00:34:14.039978+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'68","last_scrub_stamp":"2024-02-23T00:33:47.391824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.391824+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.d","version":"224'34","reported_seq":271,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045601+0000","last_change":"2024-02-23T00:33:48.238674+0000","last_active":"2024-02-23T00:34:14.045601+0000","last_peered":"2024-02-23T00:34:14.045601+0000","last_clean":"2024-02-23T00:34:14.045601+0000","last_became_active":"2024-02-23T00:31:06.262251+0000","last_became_peered":"2024-02-23T00:31:06.262251+0000","last_unstale":"2024-02-23T00:34:14.045601+0000","last_undegraded":"2024-02-23T00:34:14.045601+0000","last_fullsized":"2024-02-23T00:34:14.045601+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-23T00:33:45.265466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.265466+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.e","version":"293'74","reported_seq":311,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.172949+0000","last_change":"2024-02-23T00:34:15.172949+0000","last_active":"2024-02-23T00:34:15.172949+0000","last_peered":"2024-02-23T00:34:15.172949+0000","last_clean":"2024-02-23T00:34:15.172949+0000","last_became_active":"2024-02-23T00:31:06.356847+0000","last_became_peered":"2024-02-23T00:31:06.356847+0000","last_unstale":"2024-02-23T00:34:15.172949+0000","last_undegraded":"2024-02-23T00:34:15.172949+0000","last_fullsized":"2024-02-23T00:34:15.172949+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'74","last_scrub_stamp":"2024-02-23T00:34:15.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.172848+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000057,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23,"num_write_kb":3929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.f","version":"219'69","reported_seq":298,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039970+0000","last_change":"2024-02-23T00:33:48.295325+0000","last_active":"2024-02-23T00:34:14.039970+0000","last_peered":"2024-02-23T00:34:14.039970+0000","last_clean":"2024-02-23T00:34:14.039970+0000","last_became_active":"2024-02-23T00:31:06.364609+0000","last_became_peered":"2024-02-23T00:31:06.364609+0000","last_unstale":"2024-02-23T00:34:14.039970+0000","last_undegraded":"2024-02-23T00:34:14.039970+0000","last_fullsized":"2024-02-23T00:34:14.039970+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.10","version":"218'34","reported_seq":268,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.042589+0000","last_change":"2024-02-23T00:33:48.242149+0000","last_active":"2024-02-23T00:34:14.042589+0000","last_peered":"2024-02-23T00:34:14.042589+0000","last_clean":"2024-02-23T00:34:14.042589+0000","last_became_active":"2024-02-23T00:31:06.342164+0000","last_became_peered":"2024-02-23T00:31:06.342164+0000","last_unstale":"2024-02-23T00:34:14.042589+0000","last_undegraded":"2024-02-23T00:34:14.042589+0000","last_fullsized":"2024-02-23T00:34:14.042589+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.11","version":"248'45","reported_seq":280,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044949+0000","last_change":"2024-02-23T00:34:13.745456+0000","last_active":"2024-02-23T00:34:14.044949+0000","last_peered":"2024-02-23T00:34:14.044949+0000","last_clean":"2024-02-23T00:34:14.044949+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:34:14.044949+0000","last_undegraded":"2024-02-23T00:34:14.044949+0000","last_fullsized":"2024-02-23T00:34:14.044949+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'45","last_scrub_stamp":"2024-02-23T00:34:13.745366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.745366+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.12","version":"204'68","reported_seq":295,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039363+0000","last_change":"2024-02-23T00:33:48.289167+0000","last_active":"2024-02-23T00:34:14.039363+0000","last_peered":"2024-02-23T00:34:14.039363+0000","last_clean":"2024-02-23T00:34:14.039363+0000","last_became_active":"2024-02-23T00:31:06.281619+0000","last_became_peered":"2024-02-23T00:31:06.281619+0000","last_unstale":"2024-02-23T00:34:14.039363+0000","last_undegraded":"2024-02-23T00:34:14.039363+0000","last_fullsized":"2024-02-23T00:34:14.039363+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'68","last_scrub_stamp":"2024-02-23T00:33:46.399784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.399784+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.13","version":"248'61","reported_seq":296,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055516+0000","last_change":"2024-02-23T00:33:48.299177+0000","last_active":"2024-02-23T00:34:14.055516+0000","last_peered":"2024-02-23T00:34:14.055516+0000","last_clean":"2024-02-23T00:34:14.055516+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:34:14.055516+0000","last_undegraded":"2024-02-23T00:34:14.055516+0000","last_fullsized":"2024-02-23T00:34:14.055516+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'61","last_scrub_stamp":"2024-02-23T00:33:41.498235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:41.498235+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.14","version":"208'85","reported_seq":313,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039758+0000","last_change":"2024-02-23T00:33:48.357450+0000","last_active":"2024-02-23T00:34:14.039758+0000","last_peered":"2024-02-23T00:34:14.039758+0000","last_clean":"2024-02-23T00:34:14.039758+0000","last_became_active":"2024-02-23T00:31:06.234521+0000","last_became_peered":"2024-02-23T00:31:06.234521+0000","last_unstale":"2024-02-23T00:34:14.039758+0000","last_undegraded":"2024-02-23T00:34:14.039758+0000","last_fullsized":"2024-02-23T00:34:14.039758+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'85","last_scrub_stamp":"2024-02-23T00:33:48.357371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:48.357371+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.15","version":"220'68","reported_seq":300,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046610+0000","last_change":"2024-02-23T00:33:48.037678+0000","last_active":"2024-02-23T00:34:14.046610+0000","last_peered":"2024-02-23T00:34:14.046610+0000","last_clean":"2024-02-23T00:34:14.046610+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:34:14.046610+0000","last_undegraded":"2024-02-23T00:34:14.046610+0000","last_fullsized":"2024-02-23T00:34:14.046610+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.16","version":"204'51","reported_seq":291,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039897+0000","last_change":"2024-02-23T00:33:48.042926+0000","last_active":"2024-02-23T00:34:14.039897+0000","last_peered":"2024-02-23T00:34:14.039897+0000","last_clean":"2024-02-23T00:34:14.039897+0000","last_became_active":"2024-02-23T00:31:06.367069+0000","last_became_peered":"2024-02-23T00:31:06.367069+0000","last_unstale":"2024-02-23T00:34:14.039897+0000","last_undegraded":"2024-02-23T00:34:14.039897+0000","last_fullsized":"2024-02-23T00:34:14.039897+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:32:57.837196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:57.837196+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.17","version":"277'94","reported_seq":328,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045214+0000","last_change":"2024-02-23T00:33:50.240380+0000","last_active":"2024-02-23T00:34:14.045214+0000","last_peered":"2024-02-23T00:34:14.045214+0000","last_clean":"2024-02-23T00:34:14.045214+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:34:14.045214+0000","last_undegraded":"2024-02-23T00:34:14.045214+0000","last_fullsized":"2024-02-23T00:34:14.045214+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'86","last_scrub_stamp":"2024-02-23T00:33:35.383425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:35.383425+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.18","version":"211'57","reported_seq":294,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055328+0000","last_change":"2024-02-23T00:33:48.297931+0000","last_active":"2024-02-23T00:34:14.055328+0000","last_peered":"2024-02-23T00:34:14.055328+0000","last_clean":"2024-02-23T00:34:14.055328+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:34:14.055328+0000","last_undegraded":"2024-02-23T00:34:14.055328+0000","last_fullsized":"2024-02-23T00:34:14.055328+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'57","last_scrub_stamp":"2024-02-23T00:33:36.369153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.369153+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.19","version":"285'59","reported_seq":302,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044890+0000","last_change":"2024-02-23T00:33:48.036867+0000","last_active":"2024-02-23T00:34:14.044890+0000","last_peered":"2024-02-23T00:34:14.044890+0000","last_clean":"2024-02-23T00:34:14.044890+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:34:14.044890+0000","last_undegraded":"2024-02-23T00:34:14.044890+0000","last_fullsized":"2024-02-23T00:34:14.044890+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'54","last_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_deep_scrub":"271'54","last_deep_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.780899+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.1a","version":"202'68","reported_seq":301,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039670+0000","last_change":"2024-02-23T00:33:48.042765+0000","last_active":"2024-02-23T00:34:14.039670+0000","last_peered":"2024-02-23T00:34:14.039670+0000","last_clean":"2024-02-23T00:34:14.039670+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:34:14.039670+0000","last_undegraded":"2024-02-23T00:34:14.039670+0000","last_fullsized":"2024-02-23T00:34:14.039670+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'68","last_scrub_stamp":"2024-02-23T00:32:58.808102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.808102+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.1b","version":"236'51","reported_seq":287,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043582+0000","last_change":"2024-02-23T00:33:48.237253+0000","last_active":"2024-02-23T00:34:14.043582+0000","last_peered":"2024-02-23T00:34:14.043582+0000","last_clean":"2024-02-23T00:34:14.043582+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:34:14.043582+0000","last_undegraded":"2024-02-23T00:34:14.043582+0000","last_fullsized":"2024-02-23T00:34:14.043582+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-02-23T00:33:46.274400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.274400+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1c","version":"216'34","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044282+0000","last_change":"2024-02-23T00:33:48.244428+0000","last_active":"2024-02-23T00:34:14.044282+0000","last_peered":"2024-02-23T00:34:14.044282+0000","last_clean":"2024-02-23T00:34:14.044282+0000","last_became_active":"2024-02-23T00:31:06.338540+0000","last_became_peered":"2024-02-23T00:31:06.338540+0000","last_unstale":"2024-02-23T00:34:14.044282+0000","last_undegraded":"2024-02-23T00:34:14.044282+0000","last_fullsized":"2024-02-23T00:34:14.044282+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1d","version":"229'10","reported_seq":252,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045251+0000","last_change":"2024-02-23T00:33:48.246448+0000","last_active":"2024-02-23T00:34:14.045251+0000","last_peered":"2024-02-23T00:34:14.045251+0000","last_clean":"2024-02-23T00:34:14.045251+0000","last_became_active":"2024-02-23T00:31:06.337635+0000","last_became_peered":"2024-02-23T00:31:06.337635+0000","last_unstale":"2024-02-23T00:34:14.045251+0000","last_undegraded":"2024-02-23T00:34:14.045251+0000","last_fullsized":"2024-02-23T00:34:14.045251+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1e","version":"232'51","reported_seq":285,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044099+0000","last_change":"2024-02-23T00:33:49.219707+0000","last_active":"2024-02-23T00:34:14.044099+0000","last_peered":"2024-02-23T00:34:14.044099+0000","last_clean":"2024-02-23T00:34:14.044099+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:34:14.044099+0000","last_undegraded":"2024-02-23T00:34:14.044099+0000","last_fullsized":"2024-02-23T00:34:14.044099+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'51","last_scrub_stamp":"2024-02-23T00:33:49.219607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.219607+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1f","version":"228'68","reported_seq":305,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045117+0000","last_change":"2024-02-23T00:33:48.238796+0000","last_active":"2024-02-23T00:34:14.045117+0000","last_peered":"2024-02-23T00:34:14.045117+0000","last_clean":"2024-02-23T00:34:14.045117+0000","last_became_active":"2024-02-23T00:31:06.355623+0000","last_became_peered":"2024-02-23T00:31:06.355623+0000","last_unstale":"2024-02-23T00:34:14.045117+0000","last_undegraded":"2024-02-23T00:34:14.045117+0000","last_fullsized":"2024-02-23T00:34:14.045117+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-23T00:33:39.351573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.351573+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'8","reported_seq":359,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.062408+0000","last_change":"2024-02-23T00:32:11.743347+0000","last_active":"2024-02-23T00:34:15.062408+0000","last_peered":"2024-02-23T00:34:15.062408+0000","last_clean":"2024-02-23T00:34:15.062408+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:34:15.062408+0000","last_undegraded":"2024-02-23T00:34:15.062408+0000","last_fullsized":"2024-02-23T00:34:15.062408+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:11.743278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.743278+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":331,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043197+0000","last_change":"2024-02-23T00:32:47.376760+0000","last_active":"2024-02-23T00:34:14.043197+0000","last_peered":"2024-02-23T00:34:14.043197+0000","last_clean":"2024-02-23T00:34:14.043197+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:34:14.043197+0000","last_undegraded":"2024-02-23T00:34:14.043197+0000","last_fullsized":"2024-02-23T00:34:14.043197+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:32:47.376691+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.376691+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":278,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040174+0000","last_change":"2024-02-23T00:33:00.878453+0000","last_active":"2024-02-23T00:34:14.040174+0000","last_peered":"2024-02-23T00:34:14.040174+0000","last_clean":"2024-02-23T00:34:14.040174+0000","last_became_active":"2024-02-23T00:27:17.008163+0000","last_became_peered":"2024-02-23T00:27:17.008163+0000","last_unstale":"2024-02-23T00:34:14.040174+0000","last_undegraded":"2024-02-23T00:34:14.040174+0000","last_fullsized":"2024-02-23T00:34:14.040174+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.878391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.878391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":258,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045624+0000","last_change":"2024-02-23T00:31:31.323024+0000","last_active":"2024-02-23T00:34:14.045624+0000","last_peered":"2024-02-23T00:34:14.045624+0000","last_clean":"2024-02-23T00:34:14.045624+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:34:14.045624+0000","last_undegraded":"2024-02-23T00:34:14.045624+0000","last_fullsized":"2024-02-23T00:34:14.045624+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:31.322940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:31.322940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":217,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044602+0000","last_change":"2024-02-23T00:32:48.468225+0000","last_active":"2024-02-23T00:34:14.044602+0000","last_peered":"2024-02-23T00:34:14.044602+0000","last_clean":"2024-02-23T00:34:14.044602+0000","last_became_active":"2024-02-23T00:30:48.777752+0000","last_became_peered":"2024-02-23T00:30:48.777752+0000","last_unstale":"2024-02-23T00:34:14.044602+0000","last_undegraded":"2024-02-23T00:34:14.044602+0000","last_fullsized":"2024-02-23T00:34:14.044602+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":351,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.058885+0000","last_change":"2024-02-23T00:32:37.440685+0000","last_active":"2024-02-23T00:34:15.058885+0000","last_peered":"2024-02-23T00:34:15.058885+0000","last_clean":"2024-02-23T00:34:15.058885+0000","last_became_active":"2024-02-23T00:31:29.077539+0000","last_became_peered":"2024-02-23T00:31:29.077539+0000","last_unstale":"2024-02-23T00:34:15.058885+0000","last_undegraded":"2024-02-23T00:34:15.058885+0000","last_fullsized":"2024-02-23T00:34:15.058885+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:37.440614+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.440614+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"63'15","reported_seq":307,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046548+0000","last_change":"2024-02-23T00:33:49.588891+0000","last_active":"2024-02-23T00:34:14.046548+0000","last_peered":"2024-02-23T00:34:14.046548+0000","last_clean":"2024-02-23T00:34:14.046548+0000","last_became_active":"2024-02-23T00:27:19.023991+0000","last_became_peered":"2024-02-23T00:27:19.023991+0000","last_unstale":"2024-02-23T00:34:14.046548+0000","last_undegraded":"2024-02-23T00:34:14.046548+0000","last_fullsized":"2024-02-23T00:34:14.046548+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":270,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045838+0000","last_change":"2024-02-23T00:32:38.328306+0000","last_active":"2024-02-23T00:34:14.045838+0000","last_peered":"2024-02-23T00:34:14.045838+0000","last_clean":"2024-02-23T00:34:14.045838+0000","last_became_active":"2024-02-23T00:27:17.008371+0000","last_became_peered":"2024-02-23T00:27:17.008371+0000","last_unstale":"2024-02-23T00:34:14.045838+0000","last_undegraded":"2024-02-23T00:34:14.045838+0000","last_fullsized":"2024-02-23T00:34:14.045838+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.328237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.328237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":269,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039797+0000","last_change":"2024-02-23T00:32:40.855264+0000","last_active":"2024-02-23T00:34:14.039797+0000","last_peered":"2024-02-23T00:34:14.039797+0000","last_clean":"2024-02-23T00:34:14.039797+0000","last_became_active":"2024-02-23T00:27:21.017404+0000","last_became_peered":"2024-02-23T00:27:21.017404+0000","last_unstale":"2024-02-23T00:34:14.039797+0000","last_undegraded":"2024-02-23T00:34:14.039797+0000","last_fullsized":"2024-02-23T00:34:14.039797+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.855194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.855194+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":272,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045856+0000","last_change":"2024-02-23T00:33:47.610285+0000","last_active":"2024-02-23T00:34:14.045856+0000","last_peered":"2024-02-23T00:34:14.045856+0000","last_clean":"2024-02-23T00:34:14.045856+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:34:14.045856+0000","last_undegraded":"2024-02-23T00:34:14.045856+0000","last_fullsized":"2024-02-23T00:34:14.045856+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":218,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039975+0000","last_change":"2024-02-23T00:33:44.400851+0000","last_active":"2024-02-23T00:34:14.039975+0000","last_peered":"2024-02-23T00:34:14.039975+0000","last_clean":"2024-02-23T00:34:14.039975+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:34:14.039975+0000","last_undegraded":"2024-02-23T00:34:14.039975+0000","last_fullsized":"2024-02-23T00:34:14.039975+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.400781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.400781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":315,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039856+0000","last_change":"2024-02-23T00:31:41.019548+0000","last_active":"2024-02-23T00:34:14.039856+0000","last_peered":"2024-02-23T00:34:14.039856+0000","last_clean":"2024-02-23T00:34:14.039856+0000","last_became_active":"2024-02-23T00:27:19.022509+0000","last_became_peered":"2024-02-23T00:27:19.022509+0000","last_unstale":"2024-02-23T00:34:14.039856+0000","last_undegraded":"2024-02-23T00:34:14.039856+0000","last_fullsized":"2024-02-23T00:34:14.039856+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.019441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.019441+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":278,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040366+0000","last_change":"2024-02-23T00:32:59.849579+0000","last_active":"2024-02-23T00:34:14.040366+0000","last_peered":"2024-02-23T00:34:14.040366+0000","last_clean":"2024-02-23T00:34:14.040366+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:34:14.040366+0000","last_undegraded":"2024-02-23T00:34:14.040366+0000","last_fullsized":"2024-02-23T00:34:14.040366+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.849503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.849503+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":269,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039376+0000","last_change":"2024-02-23T00:32:46.885745+0000","last_active":"2024-02-23T00:34:14.039376+0000","last_peered":"2024-02-23T00:34:14.039376+0000","last_clean":"2024-02-23T00:34:14.039376+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:34:14.039376+0000","last_undegraded":"2024-02-23T00:34:14.039376+0000","last_fullsized":"2024-02-23T00:34:14.039376+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.885681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.885681+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":265,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055591+0000","last_change":"2024-02-23T00:32:38.683758+0000","last_active":"2024-02-23T00:34:14.055591+0000","last_peered":"2024-02-23T00:34:14.055591+0000","last_clean":"2024-02-23T00:34:14.055591+0000","last_became_active":"2024-02-23T00:27:23.028016+0000","last_became_peered":"2024-02-23T00:27:23.028016+0000","last_unstale":"2024-02-23T00:34:14.055591+0000","last_undegraded":"2024-02-23T00:34:14.055591+0000","last_fullsized":"2024-02-23T00:34:14.055591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.683686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.683686+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046123+0000","last_change":"2024-02-23T00:32:45.536091+0000","last_active":"2024-02-23T00:34:14.046123+0000","last_peered":"2024-02-23T00:34:14.046123+0000","last_clean":"2024-02-23T00:34:14.046123+0000","last_became_active":"2024-02-23T00:30:48.673452+0000","last_became_peered":"2024-02-23T00:30:48.673452+0000","last_unstale":"2024-02-23T00:34:14.046123+0000","last_undegraded":"2024-02-23T00:34:14.046123+0000","last_fullsized":"2024-02-23T00:34:14.046123+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.535992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.535992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":280,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040151+0000","last_change":"2024-02-23T00:32:37.445055+0000","last_active":"2024-02-23T00:34:14.040151+0000","last_peered":"2024-02-23T00:34:14.040151+0000","last_clean":"2024-02-23T00:34:14.040151+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:34:14.040151+0000","last_undegraded":"2024-02-23T00:34:14.040151+0000","last_fullsized":"2024-02-23T00:34:14.040151+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.444973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.444973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"62'17","reported_seq":299,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.047352+0000","last_change":"2024-02-23T00:33:36.603800+0000","last_active":"2024-02-23T00:34:14.047352+0000","last_peered":"2024-02-23T00:34:14.047352+0000","last_clean":"2024-02-23T00:34:14.047352+0000","last_became_active":"2024-02-23T00:27:19.024021+0000","last_became_peered":"2024-02-23T00:27:19.024021+0000","last_unstale":"2024-02-23T00:34:14.047352+0000","last_undegraded":"2024-02-23T00:34:14.047352+0000","last_fullsized":"2024-02-23T00:34:14.047352+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:33:36.603732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.603732+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":271,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055808+0000","last_change":"2024-02-23T00:32:56.692114+0000","last_active":"2024-02-23T00:34:14.055808+0000","last_peered":"2024-02-23T00:34:14.055808+0000","last_clean":"2024-02-23T00:34:14.055808+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:34:14.055808+0000","last_undegraded":"2024-02-23T00:34:14.055808+0000","last_fullsized":"2024-02-23T00:34:14.055808+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.692037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.692037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":261,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044693+0000","last_change":"2024-02-23T00:32:09.110225+0000","last_active":"2024-02-23T00:34:14.044693+0000","last_peered":"2024-02-23T00:34:14.044693+0000","last_clean":"2024-02-23T00:34:14.044693+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:34:14.044693+0000","last_undegraded":"2024-02-23T00:34:14.044693+0000","last_fullsized":"2024-02-23T00:34:14.044693+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.110138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":272,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039645+0000","last_change":"2024-02-23T00:33:33.791204+0000","last_active":"2024-02-23T00:34:14.039645+0000","last_peered":"2024-02-23T00:34:14.039645+0000","last_clean":"2024-02-23T00:34:14.039645+0000","last_became_active":"2024-02-23T00:27:23.029238+0000","last_became_peered":"2024-02-23T00:27:23.029238+0000","last_unstale":"2024-02-23T00:34:14.039645+0000","last_undegraded":"2024-02-23T00:34:14.039645+0000","last_fullsized":"2024-02-23T00:34:14.039645+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.791137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.791137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":219,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.057515+0000","last_change":"2024-02-23T00:33:00.671292+0000","last_active":"2024-02-23T00:34:14.057515+0000","last_peered":"2024-02-23T00:34:14.057515+0000","last_clean":"2024-02-23T00:34:14.057515+0000","last_became_active":"2024-02-23T00:30:48.716552+0000","last_became_peered":"2024-02-23T00:30:48.716552+0000","last_unstale":"2024-02-23T00:34:14.057515+0000","last_undegraded":"2024-02-23T00:34:14.057515+0000","last_fullsized":"2024-02-23T00:34:14.057515+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.671219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.671219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":282,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041449+0000","last_change":"2024-02-23T00:32:53.985225+0000","last_active":"2024-02-23T00:34:14.041449+0000","last_peered":"2024-02-23T00:34:14.041449+0000","last_clean":"2024-02-23T00:34:14.041449+0000","last_became_active":"2024-02-23T00:31:29.064244+0000","last_became_peered":"2024-02-23T00:31:29.064244+0000","last_unstale":"2024-02-23T00:34:14.041449+0000","last_undegraded":"2024-02-23T00:34:14.041449+0000","last_fullsized":"2024-02-23T00:34:14.041449+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.985150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.985150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"62'10","reported_seq":281,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046231+0000","last_change":"2024-02-23T00:31:19.982059+0000","last_active":"2024-02-23T00:34:14.046231+0000","last_peered":"2024-02-23T00:34:14.046231+0000","last_clean":"2024-02-23T00:34:14.046231+0000","last_became_active":"2024-02-23T00:27:19.022564+0000","last_became_peered":"2024-02-23T00:27:19.022564+0000","last_unstale":"2024-02-23T00:34:14.046231+0000","last_undegraded":"2024-02-23T00:34:14.046231+0000","last_fullsized":"2024-02-23T00:34:14.046231+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:31:19.981985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:19.981985+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":283,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044201+0000","last_change":"2024-02-23T00:31:49.024710+0000","last_active":"2024-02-23T00:34:14.044201+0000","last_peered":"2024-02-23T00:34:14.044201+0000","last_clean":"2024-02-23T00:34:14.044201+0000","last_became_active":"2024-02-23T00:31:49.024302+0000","last_became_peered":"2024-02-23T00:31:49.024302+0000","last_unstale":"2024-02-23T00:34:14.044201+0000","last_undegraded":"2024-02-23T00:34:14.044201+0000","last_fullsized":"2024-02-23T00:34:14.044201+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.937969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":361,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.043677+0000","last_change":"2024-02-23T00:32:35.929043+0000","last_active":"2024-02-23T00:34:15.043677+0000","last_peered":"2024-02-23T00:34:15.043677+0000","last_clean":"2024-02-23T00:34:15.043677+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:34:15.043677+0000","last_undegraded":"2024-02-23T00:34:15.043677+0000","last_fullsized":"2024-02-23T00:34:15.043677+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.928977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.928977+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046896+0000","last_change":"2024-02-23T00:33:27.495416+0000","last_active":"2024-02-23T00:34:14.046896+0000","last_peered":"2024-02-23T00:34:14.046896+0000","last_clean":"2024-02-23T00:34:14.046896+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:34:14.046896+0000","last_undegraded":"2024-02-23T00:34:14.046896+0000","last_fullsized":"2024-02-23T00:34:14.046896+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:27.495308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:27.495308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046894+0000","last_change":"2024-02-23T00:33:32.583582+0000","last_active":"2024-02-23T00:34:14.046894+0000","last_peered":"2024-02-23T00:34:14.046894+0000","last_clean":"2024-02-23T00:34:14.046894+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:34:14.046894+0000","last_undegraded":"2024-02-23T00:34:14.046894+0000","last_fullsized":"2024-02-23T00:34:14.046894+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:32.583509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:32.583509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":181,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039740+0000","last_change":"2024-02-23T00:32:58.975408+0000","last_active":"2024-02-23T00:34:14.039740+0000","last_peered":"2024-02-23T00:34:14.039740+0000","last_clean":"2024-02-23T00:34:14.039740+0000","last_became_active":"2024-02-23T00:31:29.089319+0000","last_became_peered":"2024-02-23T00:31:29.089319+0000","last_unstale":"2024-02-23T00:34:14.039740+0000","last_undegraded":"2024-02-23T00:34:14.039740+0000","last_fullsized":"2024-02-23T00:34:14.039740+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:58.975334+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.975334+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"87'22","reported_seq":300,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045079+0000","last_change":"2024-02-23T00:32:40.327594+0000","last_active":"2024-02-23T00:34:14.045079+0000","last_peered":"2024-02-23T00:34:14.045079+0000","last_clean":"2024-02-23T00:34:14.045079+0000","last_became_active":"2024-02-23T00:27:19.024371+0000","last_became_peered":"2024-02-23T00:27:19.024371+0000","last_unstale":"2024-02-23T00:34:14.045079+0000","last_undegraded":"2024-02-23T00:34:14.045079+0000","last_fullsized":"2024-02-23T00:34:14.045079+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:32:40.327519+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.327519+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":265,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.042908+0000","last_change":"2024-02-23T00:31:29.920657+0000","last_active":"2024-02-23T00:34:14.042908+0000","last_peered":"2024-02-23T00:34:14.042908+0000","last_clean":"2024-02-23T00:34:14.042908+0000","last_became_active":"2024-02-23T00:27:17.011773+0000","last_became_peered":"2024-02-23T00:27:17.011773+0000","last_unstale":"2024-02-23T00:34:14.042908+0000","last_undegraded":"2024-02-23T00:34:14.042908+0000","last_fullsized":"2024-02-23T00:34:14.042908+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:29.920546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:29.920546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":361,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.045874+0000","last_change":"2024-02-23T00:32:35.975973+0000","last_active":"2024-02-23T00:34:15.045874+0000","last_peered":"2024-02-23T00:34:15.045874+0000","last_clean":"2024-02-23T00:34:15.045874+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:34:15.045874+0000","last_undegraded":"2024-02-23T00:34:15.045874+0000","last_fullsized":"2024-02-23T00:34:15.045874+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:32:35.975889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.975889+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":259,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046027+0000","last_change":"2024-02-23T00:31:21.366604+0000","last_active":"2024-02-23T00:34:14.046027+0000","last_peered":"2024-02-23T00:34:14.046027+0000","last_clean":"2024-02-23T00:34:14.046027+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:34:14.046027+0000","last_undegraded":"2024-02-23T00:34:14.046027+0000","last_fullsized":"2024-02-23T00:34:14.046027+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:31:21.366535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.366535+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":217,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045411+0000","last_change":"2024-02-23T00:33:39.140496+0000","last_active":"2024-02-23T00:34:14.045411+0000","last_peered":"2024-02-23T00:34:14.045411+0000","last_clean":"2024-02-23T00:34:14.045411+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:34:14.045411+0000","last_undegraded":"2024-02-23T00:34:14.045411+0000","last_fullsized":"2024-02-23T00:34:14.045411+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:39.140434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.140434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":282,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044808+0000","last_change":"2024-02-23T00:32:47.737933+0000","last_active":"2024-02-23T00:34:14.044808+0000","last_peered":"2024-02-23T00:34:14.044808+0000","last_clean":"2024-02-23T00:34:14.044808+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:34:14.044808+0000","last_undegraded":"2024-02-23T00:34:14.044808+0000","last_fullsized":"2024-02-23T00:34:14.044808+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:47.737874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.737874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"86'23","reported_seq":311,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046577+0000","last_change":"2024-02-23T00:32:46.780054+0000","last_active":"2024-02-23T00:34:14.046577+0000","last_peered":"2024-02-23T00:34:14.046577+0000","last_clean":"2024-02-23T00:34:14.046577+0000","last_became_active":"2024-02-23T00:27:19.022586+0000","last_became_peered":"2024-02-23T00:27:19.022586+0000","last_unstale":"2024-02-23T00:34:14.046577+0000","last_undegraded":"2024-02-23T00:34:14.046577+0000","last_fullsized":"2024-02-23T00:34:14.046577+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'23","last_scrub_stamp":"2024-02-23T00:32:46.779979+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.779979+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":283,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056019+0000","last_change":"2024-02-23T00:33:50.601581+0000","last_active":"2024-02-23T00:34:14.056019+0000","last_peered":"2024-02-23T00:34:14.056019+0000","last_clean":"2024-02-23T00:34:14.056019+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:34:14.056019+0000","last_undegraded":"2024-02-23T00:34:14.056019+0000","last_fullsized":"2024-02-23T00:34:14.056019+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":260,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046377+0000","last_change":"2024-02-23T00:31:33.381301+0000","last_active":"2024-02-23T00:34:14.046377+0000","last_peered":"2024-02-23T00:34:14.046377+0000","last_clean":"2024-02-23T00:34:14.046377+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:34:14.046377+0000","last_undegraded":"2024-02-23T00:34:14.046377+0000","last_fullsized":"2024-02-23T00:34:14.046377+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:33.381229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:33.381229+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":262,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041454+0000","last_change":"2024-02-23T00:32:09.012254+0000","last_active":"2024-02-23T00:34:14.041454+0000","last_peered":"2024-02-23T00:34:14.041454+0000","last_clean":"2024-02-23T00:34:14.041454+0000","last_became_active":"2024-02-23T00:27:23.181710+0000","last_became_peered":"2024-02-23T00:27:23.181710+0000","last_unstale":"2024-02-23T00:34:14.041454+0000","last_undegraded":"2024-02-23T00:34:14.041454+0000","last_fullsized":"2024-02-23T00:34:14.041454+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.012161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.012161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":217,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044848+0000","last_change":"2024-02-23T00:32:53.387558+0000","last_active":"2024-02-23T00:34:14.044848+0000","last_peered":"2024-02-23T00:34:14.044848+0000","last_clean":"2024-02-23T00:34:14.044848+0000","last_became_active":"2024-02-23T00:30:48.740636+0000","last_became_peered":"2024-02-23T00:30:48.740636+0000","last_unstale":"2024-02-23T00:34:14.044848+0000","last_undegraded":"2024-02-23T00:34:14.044848+0000","last_fullsized":"2024-02-23T00:34:14.044848+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.387496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":186,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043840+0000","last_change":"2024-02-23T00:32:56.610957+0000","last_active":"2024-02-23T00:34:14.043840+0000","last_peered":"2024-02-23T00:34:14.043840+0000","last_clean":"2024-02-23T00:34:14.043840+0000","last_became_active":"2024-02-23T00:31:29.108747+0000","last_became_peered":"2024-02-23T00:31:29.108747+0000","last_unstale":"2024-02-23T00:34:14.043840+0000","last_undegraded":"2024-02-23T00:34:14.043840+0000","last_fullsized":"2024-02-23T00:34:14.043840+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.610884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.610884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"62'8","reported_seq":360,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.039727+0000","last_change":"2024-02-23T00:31:41.945984+0000","last_active":"2024-02-23T00:34:15.039727+0000","last_peered":"2024-02-23T00:34:15.039727+0000","last_clean":"2024-02-23T00:34:15.039727+0000","last_became_active":"2024-02-23T00:27:21.021613+0000","last_became_peered":"2024-02-23T00:27:21.021613+0000","last_unstale":"2024-02-23T00:34:15.039727+0000","last_undegraded":"2024-02-23T00:34:15.039727+0000","last_fullsized":"2024-02-23T00:34:15.039727+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":302,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045314+0000","last_change":"2024-02-23T00:31:38.038523+0000","last_active":"2024-02-23T00:34:14.045314+0000","last_peered":"2024-02-23T00:34:14.045314+0000","last_clean":"2024-02-23T00:34:14.045314+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:34:14.045314+0000","last_undegraded":"2024-02-23T00:34:14.045314+0000","last_fullsized":"2024-02-23T00:34:14.045314+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:31:38.038413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.038413+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":270,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040454+0000","last_change":"2024-02-23T00:32:46.504136+0000","last_active":"2024-02-23T00:34:14.040454+0000","last_peered":"2024-02-23T00:34:14.040454+0000","last_clean":"2024-02-23T00:34:14.040454+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:34:14.040454+0000","last_undegraded":"2024-02-23T00:34:14.040454+0000","last_fullsized":"2024-02-23T00:34:14.040454+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.504045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.504045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":270,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056243+0000","last_change":"2024-02-23T00:33:45.587063+0000","last_active":"2024-02-23T00:34:14.056243+0000","last_peered":"2024-02-23T00:34:14.056243+0000","last_clean":"2024-02-23T00:34:14.056243+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:34:14.056243+0000","last_undegraded":"2024-02-23T00:34:14.056243+0000","last_fullsized":"2024-02-23T00:34:14.056243+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:45.586975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.586975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":212,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044343+0000","last_change":"2024-02-23T00:30:48.782732+0000","last_active":"2024-02-23T00:34:14.044343+0000","last_peered":"2024-02-23T00:34:14.044343+0000","last_clean":"2024-02-23T00:34:14.044343+0000","last_became_active":"2024-02-23T00:30:48.778293+0000","last_became_peered":"2024-02-23T00:30:48.778293+0000","last_unstale":"2024-02-23T00:34:14.044343+0000","last_undegraded":"2024-02-23T00:34:14.044343+0000","last_fullsized":"2024-02-23T00:34:14.044343+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":282,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040655+0000","last_change":"2024-02-23T00:32:55.991569+0000","last_active":"2024-02-23T00:34:14.040655+0000","last_peered":"2024-02-23T00:34:14.040655+0000","last_clean":"2024-02-23T00:34:14.040655+0000","last_became_active":"2024-02-23T00:31:29.070521+0000","last_became_peered":"2024-02-23T00:31:29.070521+0000","last_unstale":"2024-02-23T00:34:14.040655+0000","last_undegraded":"2024-02-23T00:34:14.040655+0000","last_fullsized":"2024-02-23T00:34:14.040655+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:55.991514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:55.991514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":269,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045619+0000","last_change":"2024-02-23T00:32:41.922931+0000","last_active":"2024-02-23T00:34:14.045619+0000","last_peered":"2024-02-23T00:34:14.045619+0000","last_clean":"2024-02-23T00:34:14.045619+0000","last_became_active":"2024-02-23T00:27:21.025522+0000","last_became_peered":"2024-02-23T00:27:21.025522+0000","last_unstale":"2024-02-23T00:34:14.045619+0000","last_undegraded":"2024-02-23T00:34:14.045619+0000","last_fullsized":"2024-02-23T00:34:14.045619+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:41.922849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.922849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":328,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039536+0000","last_change":"2024-02-23T00:31:21.086750+0000","last_active":"2024-02-23T00:34:14.039536+0000","last_peered":"2024-02-23T00:34:14.039536+0000","last_clean":"2024-02-23T00:34:14.039536+0000","last_became_active":"2024-02-23T00:27:19.019776+0000","last_became_peered":"2024-02-23T00:27:19.019776+0000","last_unstale":"2024-02-23T00:34:14.039536+0000","last_undegraded":"2024-02-23T00:34:14.039536+0000","last_fullsized":"2024-02-23T00:34:14.039536+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:31:21.086610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.086610+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":58,"num_read_kb":44,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":270,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045227+0000","last_change":"2024-02-23T00:33:34.199948+0000","last_active":"2024-02-23T00:34:14.045227+0000","last_peered":"2024-02-23T00:34:14.045227+0000","last_clean":"2024-02-23T00:34:14.045227+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:34:14.045227+0000","last_undegraded":"2024-02-23T00:34:14.045227+0000","last_fullsized":"2024-02-23T00:34:14.045227+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.199869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.199869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045472+0000","last_change":"2024-02-23T00:32:17.981331+0000","last_active":"2024-02-23T00:34:14.045472+0000","last_peered":"2024-02-23T00:34:14.045472+0000","last_clean":"2024-02-23T00:34:14.045472+0000","last_became_active":"2024-02-23T00:27:23.027731+0000","last_became_peered":"2024-02-23T00:27:23.027731+0000","last_unstale":"2024-02-23T00:34:14.045472+0000","last_undegraded":"2024-02-23T00:34:14.045472+0000","last_fullsized":"2024-02-23T00:34:14.045472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.981262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.981262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":215,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041019+0000","last_change":"2024-02-23T00:30:48.776316+0000","last_active":"2024-02-23T00:34:14.041019+0000","last_peered":"2024-02-23T00:34:14.041019+0000","last_clean":"2024-02-23T00:34:14.041019+0000","last_became_active":"2024-02-23T00:30:48.769602+0000","last_became_peered":"2024-02-23T00:30:48.769602+0000","last_unstale":"2024-02-23T00:34:14.041019+0000","last_undegraded":"2024-02-23T00:34:14.041019+0000","last_fullsized":"2024-02-23T00:34:14.041019+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":533,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044169+0000","last_change":"2024-02-23T00:32:43.323030+0000","last_active":"2024-02-23T00:34:14.044169+0000","last_peered":"2024-02-23T00:34:14.044169+0000","last_clean":"2024-02-23T00:34:14.044169+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:34:14.044169+0000","last_undegraded":"2024-02-23T00:34:14.044169+0000","last_fullsized":"2024-02-23T00:34:14.044169+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:43.322963+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.322963+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039879+0000","last_change":"2024-02-23T00:32:42.410196+0000","last_active":"2024-02-23T00:34:14.039879+0000","last_peered":"2024-02-23T00:34:14.039879+0000","last_clean":"2024-02-23T00:34:14.039879+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:34:14.039879+0000","last_undegraded":"2024-02-23T00:34:14.039879+0000","last_fullsized":"2024-02-23T00:34:14.039879+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.410112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.410112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":327,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046459+0000","last_change":"2024-02-23T00:32:47.533001+0000","last_active":"2024-02-23T00:34:14.046459+0000","last_peered":"2024-02-23T00:34:14.046459+0000","last_clean":"2024-02-23T00:34:14.046459+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:34:14.046459+0000","last_undegraded":"2024-02-23T00:34:14.046459+0000","last_fullsized":"2024-02-23T00:34:14.046459+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:32:47.532917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.532917+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":276,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056597+0000","last_change":"2024-02-23T00:34:12.601903+0000","last_active":"2024-02-23T00:34:14.056597+0000","last_peered":"2024-02-23T00:34:14.056597+0000","last_clean":"2024-02-23T00:34:14.056597+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:34:14.056597+0000","last_undegraded":"2024-02-23T00:34:14.056597+0000","last_fullsized":"2024-02-23T00:34:14.056597+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":264,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043058+0000","last_change":"2024-02-23T00:33:38.163226+0000","last_active":"2024-02-23T00:34:14.043058+0000","last_peered":"2024-02-23T00:34:14.043058+0000","last_clean":"2024-02-23T00:34:14.043058+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:34:14.043058+0000","last_undegraded":"2024-02-23T00:34:14.043058+0000","last_fullsized":"2024-02-23T00:34:14.043058+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.163155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.163155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":212,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043119+0000","last_change":"2024-02-23T00:30:48.707450+0000","last_active":"2024-02-23T00:34:14.043119+0000","last_peered":"2024-02-23T00:34:14.043119+0000","last_clean":"2024-02-23T00:34:14.043119+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:34:14.043119+0000","last_undegraded":"2024-02-23T00:34:14.043119+0000","last_fullsized":"2024-02-23T00:34:14.043119+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":265,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046513+0000","last_change":"2024-02-23T00:32:12.386351+0000","last_active":"2024-02-23T00:34:14.046513+0000","last_peered":"2024-02-23T00:34:14.046513+0000","last_clean":"2024-02-23T00:34:14.046513+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:34:14.046513+0000","last_undegraded":"2024-02-23T00:34:14.046513+0000","last_fullsized":"2024-02-23T00:34:14.046513+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":294,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043310+0000","last_change":"2024-02-23T00:32:46.507450+0000","last_active":"2024-02-23T00:34:14.043310+0000","last_peered":"2024-02-23T00:34:14.043310+0000","last_clean":"2024-02-23T00:34:14.043310+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:34:14.043310+0000","last_undegraded":"2024-02-23T00:34:14.043310+0000","last_fullsized":"2024-02-23T00:34:14.043310+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:32:46.507354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.507354+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":271,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055777+0000","last_change":"2024-02-23T00:33:42.509570+0000","last_active":"2024-02-23T00:34:14.055777+0000","last_peered":"2024-02-23T00:34:14.055777+0000","last_clean":"2024-02-23T00:34:14.055777+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:34:14.055777+0000","last_undegraded":"2024-02-23T00:34:14.055777+0000","last_fullsized":"2024-02-23T00:34:14.055777+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.509463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.509463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":287,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045550+0000","last_change":"2024-02-23T00:33:31.593419+0000","last_active":"2024-02-23T00:34:14.045550+0000","last_peered":"2024-02-23T00:34:14.045550+0000","last_clean":"2024-02-23T00:34:14.045550+0000","last_became_active":"2024-02-23T00:31:27.053776+0000","last_became_peered":"2024-02-23T00:31:27.053776+0000","last_unstale":"2024-02-23T00:34:14.045550+0000","last_undegraded":"2024-02-23T00:34:14.045550+0000","last_fullsized":"2024-02-23T00:34:14.045550+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:31.593354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:31.593354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":259,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044774+0000","last_change":"2024-02-23T00:31:21.959719+0000","last_active":"2024-02-23T00:34:14.044774+0000","last_peered":"2024-02-23T00:34:14.044774+0000","last_clean":"2024-02-23T00:34:14.044774+0000","last_became_active":"2024-02-23T00:27:23.032233+0000","last_became_peered":"2024-02-23T00:27:23.032233+0000","last_unstale":"2024-02-23T00:34:14.044774+0000","last_undegraded":"2024-02-23T00:34:14.044774+0000","last_fullsized":"2024-02-23T00:34:14.044774+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:21.959597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:31:21.959597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046356+0000","last_change":"2024-02-23T00:33:34.545991+0000","last_active":"2024-02-23T00:34:14.046356+0000","last_peered":"2024-02-23T00:34:14.046356+0000","last_clean":"2024-02-23T00:34:14.046356+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:34:14.046356+0000","last_undegraded":"2024-02-23T00:34:14.046356+0000","last_fullsized":"2024-02-23T00:34:14.046356+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.545903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.545903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055792+0000","last_change":"2024-02-23T00:32:17.590019+0000","last_active":"2024-02-23T00:34:14.055792+0000","last_peered":"2024-02-23T00:34:14.055792+0000","last_clean":"2024-02-23T00:34:14.055792+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:34:14.055792+0000","last_undegraded":"2024-02-23T00:34:14.055792+0000","last_fullsized":"2024-02-23T00:34:14.055792+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:17.589952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:17.589952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":296,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.295264+0000","last_change":"2024-02-23T00:34:15.295264+0000","last_active":"2024-02-23T00:34:15.295264+0000","last_peered":"2024-02-23T00:34:15.295264+0000","last_clean":"2024-02-23T00:34:15.295264+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:34:15.295264+0000","last_undegraded":"2024-02-23T00:34:15.295264+0000","last_fullsized":"2024-02-23T00:34:15.295264+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:34:15.295178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.295178+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":279,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055879+0000","last_change":"2024-02-23T00:33:43.548018+0000","last_active":"2024-02-23T00:34:14.055879+0000","last_peered":"2024-02-23T00:34:14.055879+0000","last_clean":"2024-02-23T00:34:14.055879+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:34:14.055879+0000","last_undegraded":"2024-02-23T00:34:14.055879+0000","last_fullsized":"2024-02-23T00:34:14.055879+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":269,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044388+0000","last_change":"2024-02-23T00:32:46.410759+0000","last_active":"2024-02-23T00:34:14.044388+0000","last_peered":"2024-02-23T00:34:14.044388+0000","last_clean":"2024-02-23T00:34:14.044388+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:34:14.044388+0000","last_undegraded":"2024-02-23T00:34:14.044388+0000","last_fullsized":"2024-02-23T00:34:14.044388+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.410635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.410635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":282,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041102+0000","last_change":"2024-02-23T00:32:46.978493+0000","last_active":"2024-02-23T00:34:14.041102+0000","last_peered":"2024-02-23T00:34:14.041102+0000","last_clean":"2024-02-23T00:34:14.041102+0000","last_became_active":"2024-02-23T00:31:27.144976+0000","last_became_peered":"2024-02-23T00:31:27.144976+0000","last_unstale":"2024-02-23T00:34:14.041102+0000","last_undegraded":"2024-02-23T00:34:14.041102+0000","last_fullsized":"2024-02-23T00:34:14.041102+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.978406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.978406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":263,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046595+0000","last_change":"2024-02-23T00:33:00.439952+0000","last_active":"2024-02-23T00:34:14.046595+0000","last_peered":"2024-02-23T00:34:14.046595+0000","last_clean":"2024-02-23T00:34:14.046595+0000","last_became_active":"2024-02-23T00:27:23.028348+0000","last_became_peered":"2024-02-23T00:27:23.028348+0000","last_unstale":"2024-02-23T00:34:14.046595+0000","last_undegraded":"2024-02-23T00:34:14.046595+0000","last_fullsized":"2024-02-23T00:34:14.046595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.439872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046218+0000","last_change":"2024-02-23T00:33:33.574217+0000","last_active":"2024-02-23T00:34:14.046218+0000","last_peered":"2024-02-23T00:34:14.046218+0000","last_clean":"2024-02-23T00:34:14.046218+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:34:14.046218+0000","last_undegraded":"2024-02-23T00:34:14.046218+0000","last_fullsized":"2024-02-23T00:34:14.046218+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.574124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.574124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":274,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040302+0000","last_change":"2024-02-23T00:33:38.742275+0000","last_active":"2024-02-23T00:34:14.040302+0000","last_peered":"2024-02-23T00:34:14.040302+0000","last_clean":"2024-02-23T00:34:14.040302+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:34:14.040302+0000","last_undegraded":"2024-02-23T00:34:14.040302+0000","last_fullsized":"2024-02-23T00:34:14.040302+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":310,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044905+0000","last_change":"2024-02-23T00:32:13.104236+0000","last_active":"2024-02-23T00:34:14.044905+0000","last_peered":"2024-02-23T00:34:14.044905+0000","last_clean":"2024-02-23T00:34:14.044905+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:34:14.044905+0000","last_undegraded":"2024-02-23T00:34:14.044905+0000","last_fullsized":"2024-02-23T00:34:14.044905+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:32:13.104170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.104170+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":278,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040869+0000","last_change":"2024-02-23T00:33:32.799453+0000","last_active":"2024-02-23T00:34:14.040869+0000","last_peered":"2024-02-23T00:34:14.040869+0000","last_clean":"2024-02-23T00:34:14.040869+0000","last_became_active":"2024-02-23T00:27:17.012452+0000","last_became_peered":"2024-02-23T00:27:17.012452+0000","last_unstale":"2024-02-23T00:34:14.040869+0000","last_undegraded":"2024-02-23T00:34:14.040869+0000","last_fullsized":"2024-02-23T00:34:14.040869+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:32.799394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:32.799394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":526,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043058+0000","last_change":"2024-02-23T00:32:41.360750+0000","last_active":"2024-02-23T00:34:14.043058+0000","last_peered":"2024-02-23T00:34:14.043058+0000","last_clean":"2024-02-23T00:34:14.043058+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:34:14.043058+0000","last_undegraded":"2024-02-23T00:34:14.043058+0000","last_fullsized":"2024-02-23T00:34:14.043058+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:41.360646+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.360646+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":265,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056589+0000","last_change":"2024-02-23T00:32:14.650905+0000","last_active":"2024-02-23T00:34:14.056589+0000","last_peered":"2024-02-23T00:34:14.056589+0000","last_clean":"2024-02-23T00:34:14.056589+0000","last_became_active":"2024-02-23T00:27:23.180802+0000","last_became_peered":"2024-02-23T00:27:23.180802+0000","last_unstale":"2024-02-23T00:34:14.056589+0000","last_undegraded":"2024-02-23T00:34:14.056589+0000","last_fullsized":"2024-02-23T00:34:14.056589+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.650827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.650827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039589+0000","last_change":"2024-02-23T00:32:41.861943+0000","last_active":"2024-02-23T00:34:14.039589+0000","last_peered":"2024-02-23T00:34:14.039589+0000","last_clean":"2024-02-23T00:34:14.039589+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:34:14.039589+0000","last_undegraded":"2024-02-23T00:34:14.039589+0000","last_fullsized":"2024-02-23T00:34:14.039589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:41.861874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.861874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":282,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045749+0000","last_change":"2024-02-23T00:32:50.705430+0000","last_active":"2024-02-23T00:34:14.045749+0000","last_peered":"2024-02-23T00:34:14.045749+0000","last_clean":"2024-02-23T00:34:14.045749+0000","last_became_active":"2024-02-23T00:31:29.065548+0000","last_became_peered":"2024-02-23T00:31:29.065548+0000","last_unstale":"2024-02-23T00:34:14.045749+0000","last_undegraded":"2024-02-23T00:34:14.045749+0000","last_fullsized":"2024-02-23T00:34:14.045749+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.705318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.705318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":276,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043465+0000","last_change":"2024-02-23T00:32:50.596925+0000","last_active":"2024-02-23T00:34:14.043465+0000","last_peered":"2024-02-23T00:34:14.043465+0000","last_clean":"2024-02-23T00:34:14.043465+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:34:14.043465+0000","last_undegraded":"2024-02-23T00:34:14.043465+0000","last_fullsized":"2024-02-23T00:34:14.043465+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"63'34","reported_seq":415,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.053467+0000","last_change":"2024-02-23T00:32:45.366480+0000","last_active":"2024-02-23T00:34:15.053467+0000","last_peered":"2024-02-23T00:34:15.053467+0000","last_clean":"2024-02-23T00:34:15.053467+0000","last_became_active":"2024-02-23T00:27:19.024869+0000","last_became_peered":"2024-02-23T00:27:19.024869+0000","last_unstale":"2024-02-23T00:34:15.053467+0000","last_undegraded":"2024-02-23T00:34:15.053467+0000","last_fullsized":"2024-02-23T00:34:15.053467+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:32:45.366416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.366416+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":275,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045792+0000","last_change":"2024-02-23T00:33:36.350954+0000","last_active":"2024-02-23T00:34:14.045792+0000","last_peered":"2024-02-23T00:34:14.045792+0000","last_clean":"2024-02-23T00:34:14.045792+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:34:14.045792+0000","last_undegraded":"2024-02-23T00:34:14.045792+0000","last_fullsized":"2024-02-23T00:34:14.045792+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:36.350888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.350888+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041120+0000","last_change":"2024-02-23T00:33:41.701721+0000","last_active":"2024-02-23T00:34:14.041120+0000","last_peered":"2024-02-23T00:34:14.041120+0000","last_clean":"2024-02-23T00:34:14.041120+0000","last_became_active":"2024-02-23T00:27:23.026686+0000","last_became_peered":"2024-02-23T00:27:23.026686+0000","last_unstale":"2024-02-23T00:34:14.041120+0000","last_undegraded":"2024-02-23T00:34:14.041120+0000","last_fullsized":"2024-02-23T00:34:14.041120+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:41.701634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:41.701634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040583+0000","last_change":"2024-02-23T00:32:44.885268+0000","last_active":"2024-02-23T00:34:14.040583+0000","last_peered":"2024-02-23T00:34:14.040583+0000","last_clean":"2024-02-23T00:34:14.040583+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:34:14.040583+0000","last_undegraded":"2024-02-23T00:34:14.040583+0000","last_fullsized":"2024-02-23T00:34:14.040583+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.885205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.885205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":179,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045339+0000","last_change":"2024-02-23T00:32:36.447499+0000","last_active":"2024-02-23T00:34:14.045339+0000","last_peered":"2024-02-23T00:34:14.045339+0000","last_clean":"2024-02-23T00:34:14.045339+0000","last_became_active":"2024-02-23T00:31:29.059516+0000","last_became_peered":"2024-02-23T00:31:29.059516+0000","last_unstale":"2024-02-23T00:34:14.045339+0000","last_undegraded":"2024-02-23T00:34:14.045339+0000","last_fullsized":"2024-02-23T00:34:14.045339+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.447439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.447439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":369,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.066914+0000","last_change":"2024-02-23T00:33:46.629281+0000","last_active":"2024-02-23T00:34:15.066914+0000","last_peered":"2024-02-23T00:34:15.066914+0000","last_clean":"2024-02-23T00:34:15.066914+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:34:15.066914+0000","last_undegraded":"2024-02-23T00:34:15.066914+0000","last_fullsized":"2024-02-23T00:34:15.066914+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":298,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044453+0000","last_change":"2024-02-23T00:33:37.175714+0000","last_active":"2024-02-23T00:34:14.044453+0000","last_peered":"2024-02-23T00:34:14.044453+0000","last_clean":"2024-02-23T00:34:14.044453+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:34:14.044453+0000","last_undegraded":"2024-02-23T00:34:14.044453+0000","last_fullsized":"2024-02-23T00:34:14.044453+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:33:37.175650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.175650+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":273,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044446+0000","last_change":"2024-02-23T00:34:13.208540+0000","last_active":"2024-02-23T00:34:14.044446+0000","last_peered":"2024-02-23T00:34:14.044446+0000","last_clean":"2024-02-23T00:34:14.044446+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:34:14.044446+0000","last_undegraded":"2024-02-23T00:34:14.044446+0000","last_fullsized":"2024-02-23T00:34:14.044446+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":272,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045821+0000","last_change":"2024-02-23T00:34:12.489035+0000","last_active":"2024-02-23T00:34:14.045821+0000","last_peered":"2024-02-23T00:34:14.045821+0000","last_clean":"2024-02-23T00:34:14.045821+0000","last_became_active":"2024-02-23T00:27:23.027557+0000","last_became_peered":"2024-02-23T00:27:23.027557+0000","last_unstale":"2024-02-23T00:34:14.045821+0000","last_undegraded":"2024-02-23T00:34:14.045821+0000","last_fullsized":"2024-02-23T00:34:14.045821+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.488959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.488959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044593+0000","last_change":"2024-02-23T00:33:49.754910+0000","last_active":"2024-02-23T00:34:14.044593+0000","last_peered":"2024-02-23T00:34:14.044593+0000","last_clean":"2024-02-23T00:34:14.044593+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:34:14.044593+0000","last_undegraded":"2024-02-23T00:34:14.044593+0000","last_fullsized":"2024-02-23T00:34:14.044593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:49.754830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.754830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":269,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039055+0000","last_change":"2024-02-23T00:32:42.815796+0000","last_active":"2024-02-23T00:34:14.039055+0000","last_peered":"2024-02-23T00:34:14.039055+0000","last_clean":"2024-02-23T00:34:14.039055+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:34:14.039055+0000","last_undegraded":"2024-02-23T00:34:14.039055+0000","last_fullsized":"2024-02-23T00:34:14.039055+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.815722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.815722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"63'25","reported_seq":345,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045529+0000","last_change":"2024-02-23T00:31:41.036909+0000","last_active":"2024-02-23T00:34:14.045529+0000","last_peered":"2024-02-23T00:34:14.045529+0000","last_clean":"2024-02-23T00:34:14.045529+0000","last_became_active":"2024-02-23T00:28:41.204316+0000","last_became_peered":"2024-02-23T00:28:41.204316+0000","last_unstale":"2024-02-23T00:34:14.045529+0000","last_undegraded":"2024-02-23T00:34:14.045529+0000","last_fullsized":"2024-02-23T00:34:14.045529+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:31:41.036837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.036837+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":38,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":286,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046726+0000","last_change":"2024-02-23T00:33:43.800448+0000","last_active":"2024-02-23T00:34:14.046726+0000","last_peered":"2024-02-23T00:34:14.046726+0000","last_clean":"2024-02-23T00:34:14.046726+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:34:14.046726+0000","last_undegraded":"2024-02-23T00:34:14.046726+0000","last_fullsized":"2024-02-23T00:34:14.046726+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:33:43.800360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.800360+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041939+0000","last_change":"2024-02-23T00:33:32.794356+0000","last_active":"2024-02-23T00:34:14.041939+0000","last_peered":"2024-02-23T00:34:14.041939+0000","last_clean":"2024-02-23T00:34:14.041939+0000","last_became_active":"2024-02-23T00:27:23.032137+0000","last_became_peered":"2024-02-23T00:27:23.032137+0000","last_unstale":"2024-02-23T00:34:14.041939+0000","last_undegraded":"2024-02-23T00:34:14.041939+0000","last_fullsized":"2024-02-23T00:34:14.041939+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:32.794268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:32.794268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":218,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043687+0000","last_change":"2024-02-23T00:32:59.434164+0000","last_active":"2024-02-23T00:34:14.043687+0000","last_peered":"2024-02-23T00:34:14.043687+0000","last_clean":"2024-02-23T00:34:14.043687+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:34:14.043687+0000","last_undegraded":"2024-02-23T00:34:14.043687+0000","last_fullsized":"2024-02-23T00:34:14.043687+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.434079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.434079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":317,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043430+0000","last_change":"2024-02-23T00:32:44.578143+0000","last_active":"2024-02-23T00:34:14.043430+0000","last_peered":"2024-02-23T00:34:14.043430+0000","last_clean":"2024-02-23T00:34:14.043430+0000","last_became_active":"2024-02-23T00:27:19.025812+0000","last_became_peered":"2024-02-23T00:27:19.025812+0000","last_unstale":"2024-02-23T00:34:14.043430+0000","last_undegraded":"2024-02-23T00:34:14.043430+0000","last_fullsized":"2024-02-23T00:34:14.043430+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:32:44.578068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.578068+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":273,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.038994+0000","last_change":"2024-02-23T00:32:11.912580+0000","last_active":"2024-02-23T00:34:14.038994+0000","last_peered":"2024-02-23T00:34:14.038994+0000","last_clean":"2024-02-23T00:34:14.038994+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:34:14.038994+0000","last_undegraded":"2024-02-23T00:34:14.038994+0000","last_fullsized":"2024-02-23T00:34:14.038994+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:11.912504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.912504+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":264,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040427+0000","last_change":"2024-02-23T00:30:36.894317+0000","last_active":"2024-02-23T00:34:14.040427+0000","last_peered":"2024-02-23T00:34:14.040427+0000","last_clean":"2024-02-23T00:34:14.040427+0000","last_became_active":"2024-02-23T00:27:21.021893+0000","last_became_peered":"2024-02-23T00:27:21.021893+0000","last_unstale":"2024-02-23T00:34:14.040427+0000","last_undegraded":"2024-02-23T00:34:14.040427+0000","last_fullsized":"2024-02-23T00:34:14.040427+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:36.894220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:30:36.894220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":272,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040175+0000","last_change":"2024-02-23T00:33:34.777997+0000","last_active":"2024-02-23T00:34:14.040175+0000","last_peered":"2024-02-23T00:34:14.040175+0000","last_clean":"2024-02-23T00:34:14.040175+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:34:14.040175+0000","last_undegraded":"2024-02-23T00:34:14.040175+0000","last_fullsized":"2024-02-23T00:34:14.040175+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.777903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.777903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":215,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041992+0000","last_change":"2024-02-23T00:30:48.691960+0000","last_active":"2024-02-23T00:34:14.041992+0000","last_peered":"2024-02-23T00:34:14.041992+0000","last_clean":"2024-02-23T00:34:14.041992+0000","last_became_active":"2024-02-23T00:30:48.691713+0000","last_became_peered":"2024-02-23T00:30:48.691713+0000","last_unstale":"2024-02-23T00:34:14.041992+0000","last_undegraded":"2024-02-23T00:34:14.041992+0000","last_fullsized":"2024-02-23T00:34:14.041992+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.537238+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":314,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040192+0000","last_change":"2024-02-23T00:33:28.885956+0000","last_active":"2024-02-23T00:34:14.040192+0000","last_peered":"2024-02-23T00:34:14.040192+0000","last_clean":"2024-02-23T00:34:14.040192+0000","last_became_active":"2024-02-23T00:28:41.206327+0000","last_became_peered":"2024-02-23T00:28:41.206327+0000","last_unstale":"2024-02-23T00:34:14.040192+0000","last_undegraded":"2024-02-23T00:34:14.040192+0000","last_fullsized":"2024-02-23T00:34:14.040192+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:33:28.885879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:28.885879+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":271,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039460+0000","last_change":"2024-02-23T00:32:14.243873+0000","last_active":"2024-02-23T00:34:14.039460+0000","last_peered":"2024-02-23T00:34:14.039460+0000","last_clean":"2024-02-23T00:34:14.039460+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:34:14.039460+0000","last_undegraded":"2024-02-23T00:34:14.039460+0000","last_fullsized":"2024-02-23T00:34:14.039460+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:14.243786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:14.243786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":260,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044257+0000","last_change":"2024-02-23T00:32:07.275896+0000","last_active":"2024-02-23T00:34:14.044257+0000","last_peered":"2024-02-23T00:34:14.044257+0000","last_clean":"2024-02-23T00:34:14.044257+0000","last_became_active":"2024-02-23T00:27:21.027174+0000","last_became_peered":"2024-02-23T00:27:21.027174+0000","last_unstale":"2024-02-23T00:34:14.044257+0000","last_undegraded":"2024-02-23T00:34:14.044257+0000","last_fullsized":"2024-02-23T00:34:14.044257+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:07.275816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:07.275816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":265,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056316+0000","last_change":"2024-02-23T00:32:15.651736+0000","last_active":"2024-02-23T00:34:14.056316+0000","last_peered":"2024-02-23T00:34:14.056316+0000","last_clean":"2024-02-23T00:34:14.056316+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:34:14.056316+0000","last_undegraded":"2024-02-23T00:34:14.056316+0000","last_fullsized":"2024-02-23T00:34:14.056316+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:15.651651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:32:15.651651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":218,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040074+0000","last_change":"2024-02-23T00:32:55.433102+0000","last_active":"2024-02-23T00:34:14.040074+0000","last_peered":"2024-02-23T00:34:14.040074+0000","last_clean":"2024-02-23T00:34:14.040074+0000","last_became_active":"2024-02-23T00:30:48.703232+0000","last_became_peered":"2024-02-23T00:30:48.703232+0000","last_unstale":"2024-02-23T00:34:14.040074+0000","last_undegraded":"2024-02-23T00:34:14.040074+0000","last_fullsized":"2024-02-23T00:34:14.040074+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:55.433031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:55.433031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":311,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041593+0000","last_change":"2024-02-23T00:30:37.899931+0000","last_active":"2024-02-23T00:34:14.041593+0000","last_peered":"2024-02-23T00:34:14.041593+0000","last_clean":"2024-02-23T00:34:14.041593+0000","last_became_active":"2024-02-23T00:27:19.021997+0000","last_became_peered":"2024-02-23T00:27:19.021997+0000","last_unstale":"2024-02-23T00:34:14.041593+0000","last_undegraded":"2024-02-23T00:34:14.041593+0000","last_fullsized":"2024-02-23T00:34:14.041593+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:30:37.899845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:30:37.899845+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":39,"num_read_kb":32,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":269,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045689+0000","last_change":"2024-02-23T00:32:10.319807+0000","last_active":"2024-02-23T00:34:14.045689+0000","last_peered":"2024-02-23T00:34:14.045689+0000","last_clean":"2024-02-23T00:34:14.045689+0000","last_became_active":"2024-02-23T00:27:17.012317+0000","last_became_peered":"2024-02-23T00:27:17.012317+0000","last_unstale":"2024-02-23T00:34:14.045689+0000","last_undegraded":"2024-02-23T00:34:14.045689+0000","last_fullsized":"2024-02-23T00:34:14.045689+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.319744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.319744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":358,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.059220+0000","last_change":"2024-02-23T00:32:56.376450+0000","last_active":"2024-02-23T00:34:15.059220+0000","last_peered":"2024-02-23T00:34:15.059220+0000","last_clean":"2024-02-23T00:34:15.059220+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:34:15.059220+0000","last_undegraded":"2024-02-23T00:34:15.059220+0000","last_fullsized":"2024-02-23T00:34:15.059220+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":272,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043230+0000","last_change":"2024-02-23T00:33:38.521859+0000","last_active":"2024-02-23T00:34:14.043230+0000","last_peered":"2024-02-23T00:34:14.043230+0000","last_clean":"2024-02-23T00:34:14.043230+0000","last_became_active":"2024-02-23T00:27:23.028154+0000","last_became_peered":"2024-02-23T00:27:23.028154+0000","last_unstale":"2024-02-23T00:34:14.043230+0000","last_undegraded":"2024-02-23T00:34:14.043230+0000","last_fullsized":"2024-02-23T00:34:14.043230+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.521760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":218,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044857+0000","last_change":"2024-02-23T00:32:45.409616+0000","last_active":"2024-02-23T00:34:14.044857+0000","last_peered":"2024-02-23T00:34:14.044857+0000","last_clean":"2024-02-23T00:34:14.044857+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:34:14.044857+0000","last_undegraded":"2024-02-23T00:34:14.044857+0000","last_fullsized":"2024-02-23T00:34:14.044857+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.409548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.409548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"144'20","reported_seq":298,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045699+0000","last_change":"2024-02-23T00:31:35.423292+0000","last_active":"2024-02-23T00:34:14.045699+0000","last_peered":"2024-02-23T00:34:14.045699+0000","last_clean":"2024-02-23T00:34:14.045699+0000","last_became_active":"2024-02-23T00:27:19.022523+0000","last_became_peered":"2024-02-23T00:27:19.022523+0000","last_unstale":"2024-02-23T00:34:14.045699+0000","last_undegraded":"2024-02-23T00:34:14.045699+0000","last_fullsized":"2024-02-23T00:34:14.045699+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.423227+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":278,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046469+0000","last_change":"2024-02-23T00:32:48.536354+0000","last_active":"2024-02-23T00:34:14.046469+0000","last_peered":"2024-02-23T00:34:14.046469+0000","last_clean":"2024-02-23T00:34:14.046469+0000","last_became_active":"2024-02-23T00:27:17.009600+0000","last_became_peered":"2024-02-23T00:27:17.009600+0000","last_unstale":"2024-02-23T00:34:14.046469+0000","last_undegraded":"2024-02-23T00:34:14.046469+0000","last_fullsized":"2024-02-23T00:34:14.046469+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.536283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.536283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"82'9","reported_seq":362,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.040366+0000","last_change":"2024-02-23T00:32:43.990958+0000","last_active":"2024-02-23T00:34:15.040366+0000","last_peered":"2024-02-23T00:34:15.040366+0000","last_clean":"2024-02-23T00:34:15.040366+0000","last_became_active":"2024-02-23T00:27:21.023325+0000","last_became_peered":"2024-02-23T00:27:21.023325+0000","last_unstale":"2024-02-23T00:34:15.040366+0000","last_undegraded":"2024-02-23T00:34:15.040366+0000","last_fullsized":"2024-02-23T00:34:15.040366+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:32:43.990886+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.990886+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046985+0000","last_change":"2024-02-23T00:32:56.678465+0000","last_active":"2024-02-23T00:34:14.046985+0000","last_peered":"2024-02-23T00:34:14.046985+0000","last_clean":"2024-02-23T00:34:14.046985+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:34:14.046985+0000","last_undegraded":"2024-02-23T00:34:14.046985+0000","last_fullsized":"2024-02-23T00:34:14.046985+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.678359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.678359+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.587055+0000","last_change":"2024-02-23T00:34:14.587023+0000","last_active":"2024-02-23T00:34:14.587055+0000","last_peered":"2024-02-23T00:34:14.587055+0000","last_clean":"2024-02-23T00:34:14.587055+0000","last_became_active":"2024-02-23T00:30:48.716104+0000","last_became_peered":"2024-02-23T00:30:48.716104+0000","last_unstale":"2024-02-23T00:34:14.587055+0000","last_undegraded":"2024-02-23T00:34:14.587055+0000","last_fullsized":"2024-02-23T00:34:14.587055+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:14.586935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:14.586935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":284,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044286+0000","last_change":"2024-02-23T00:32:50.448701+0000","last_active":"2024-02-23T00:34:14.044286+0000","last_peered":"2024-02-23T00:34:14.044286+0000","last_clean":"2024-02-23T00:34:14.044286+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:34:14.044286+0000","last_undegraded":"2024-02-23T00:34:14.044286+0000","last_fullsized":"2024-02-23T00:34:14.044286+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:32:50.448595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.448595+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":271,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040119+0000","last_change":"2024-02-23T00:32:12.234058+0000","last_active":"2024-02-23T00:34:14.040119+0000","last_peered":"2024-02-23T00:34:14.040119+0000","last_clean":"2024-02-23T00:34:14.040119+0000","last_became_active":"2024-02-23T00:27:17.009698+0000","last_became_peered":"2024-02-23T00:27:17.009698+0000","last_unstale":"2024-02-23T00:34:14.040119+0000","last_undegraded":"2024-02-23T00:34:14.040119+0000","last_fullsized":"2024-02-23T00:34:14.040119+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.233960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.233960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":269,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041846+0000","last_change":"2024-02-23T00:32:09.978443+0000","last_active":"2024-02-23T00:34:14.041846+0000","last_peered":"2024-02-23T00:34:14.041846+0000","last_clean":"2024-02-23T00:34:14.041846+0000","last_became_active":"2024-02-23T00:27:21.021628+0000","last_became_peered":"2024-02-23T00:27:21.021628+0000","last_unstale":"2024-02-23T00:34:14.041846+0000","last_undegraded":"2024-02-23T00:34:14.041846+0000","last_fullsized":"2024-02-23T00:34:14.041846+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.978377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.978377+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":272,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045764+0000","last_change":"2024-02-23T00:32:53.602946+0000","last_active":"2024-02-23T00:34:14.045764+0000","last_peered":"2024-02-23T00:34:14.045764+0000","last_clean":"2024-02-23T00:34:14.045764+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:34:14.045764+0000","last_undegraded":"2024-02-23T00:34:14.045764+0000","last_fullsized":"2024-02-23T00:34:14.045764+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.602880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.602880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044509+0000","last_change":"2024-02-23T00:33:48.779550+0000","last_active":"2024-02-23T00:34:14.044509+0000","last_peered":"2024-02-23T00:34:14.044509+0000","last_clean":"2024-02-23T00:34:14.044509+0000","last_became_active":"2024-02-23T00:30:48.666975+0000","last_became_peered":"2024-02-23T00:30:48.666975+0000","last_unstale":"2024-02-23T00:34:14.044509+0000","last_undegraded":"2024-02-23T00:34:14.044509+0000","last_fullsized":"2024-02-23T00:34:14.044509+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:48.779472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:48.779472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":311,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046059+0000","last_change":"2024-02-23T00:33:30.498502+0000","last_active":"2024-02-23T00:34:14.046059+0000","last_peered":"2024-02-23T00:34:14.046059+0000","last_clean":"2024-02-23T00:34:14.046059+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:34:14.046059+0000","last_undegraded":"2024-02-23T00:34:14.046059+0000","last_fullsized":"2024-02-23T00:34:14.046059+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:33:30.498425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:30.498425+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":265,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045706+0000","last_change":"2024-02-23T00:31:32.874749+0000","last_active":"2024-02-23T00:34:14.045706+0000","last_peered":"2024-02-23T00:34:14.045706+0000","last_clean":"2024-02-23T00:34:14.045706+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:34:14.045706+0000","last_undegraded":"2024-02-23T00:34:14.045706+0000","last_fullsized":"2024-02-23T00:34:14.045706+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:32.874678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:31:32.874678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":266,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044376+0000","last_change":"2024-02-23T00:33:46.152048+0000","last_active":"2024-02-23T00:34:14.044376+0000","last_peered":"2024-02-23T00:34:14.044376+0000","last_clean":"2024-02-23T00:34:14.044376+0000","last_became_active":"2024-02-23T00:27:21.027850+0000","last_became_peered":"2024-02-23T00:27:21.027850+0000","last_unstale":"2024-02-23T00:34:14.044376+0000","last_undegraded":"2024-02-23T00:34:14.044376+0000","last_fullsized":"2024-02-23T00:34:14.044376+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.151986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":262,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041832+0000","last_change":"2024-02-23T00:30:47.843046+0000","last_active":"2024-02-23T00:34:14.041832+0000","last_peered":"2024-02-23T00:34:14.041832+0000","last_clean":"2024-02-23T00:34:14.041832+0000","last_became_active":"2024-02-23T00:27:23.029755+0000","last_became_peered":"2024-02-23T00:27:23.029755+0000","last_unstale":"2024-02-23T00:34:14.041832+0000","last_undegraded":"2024-02-23T00:34:14.041832+0000","last_fullsized":"2024-02-23T00:34:14.041832+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:30:47.842931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":231,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039735+0000","last_change":"2024-02-23T00:33:38.437944+0000","last_active":"2024-02-23T00:34:14.039735+0000","last_peered":"2024-02-23T00:34:14.039735+0000","last_clean":"2024-02-23T00:34:14.039735+0000","last_became_active":"2024-02-23T00:30:48.703281+0000","last_became_peered":"2024-02-23T00:30:48.703281+0000","last_unstale":"2024-02-23T00:34:14.039735+0000","last_undegraded":"2024-02-23T00:34:14.039735+0000","last_fullsized":"2024-02-23T00:34:14.039735+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:33:38.437857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.437857+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":279,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039417+0000","last_change":"2024-02-23T00:32:11.989425+0000","last_active":"2024-02-23T00:34:14.039417+0000","last_peered":"2024-02-23T00:34:14.039417+0000","last_clean":"2024-02-23T00:34:14.039417+0000","last_became_active":"2024-02-23T00:27:19.021397+0000","last_became_peered":"2024-02-23T00:27:19.021397+0000","last_unstale":"2024-02-23T00:34:14.039417+0000","last_undegraded":"2024-02-23T00:34:14.039417+0000","last_fullsized":"2024-02-23T00:34:14.039417+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:32:11.989333+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.989333+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":269,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044530+0000","last_change":"2024-02-23T00:32:40.343941+0000","last_active":"2024-02-23T00:34:14.044530+0000","last_peered":"2024-02-23T00:34:14.044530+0000","last_clean":"2024-02-23T00:34:14.044530+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:34:14.044530+0000","last_undegraded":"2024-02-23T00:34:14.044530+0000","last_fullsized":"2024-02-23T00:34:14.044530+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.343863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.343863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039302+0000","last_change":"2024-02-23T00:32:47.484343+0000","last_active":"2024-02-23T00:34:14.039302+0000","last_peered":"2024-02-23T00:34:14.039302+0000","last_clean":"2024-02-23T00:34:14.039302+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:34:14.039302+0000","last_undegraded":"2024-02-23T00:34:14.039302+0000","last_fullsized":"2024-02-23T00:34:14.039302+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:47.484285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.484285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":265,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056139+0000","last_change":"2024-02-23T00:32:12.721784+0000","last_active":"2024-02-23T00:34:14.056139+0000","last_peered":"2024-02-23T00:34:14.056139+0000","last_clean":"2024-02-23T00:34:14.056139+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:34:14.056139+0000","last_undegraded":"2024-02-23T00:34:14.056139+0000","last_fullsized":"2024-02-23T00:34:14.056139+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.721719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.721719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":217,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044532+0000","last_change":"2024-02-23T00:33:44.152089+0000","last_active":"2024-02-23T00:34:14.044532+0000","last_peered":"2024-02-23T00:34:14.044532+0000","last_clean":"2024-02-23T00:34:14.044532+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:34:14.044532+0000","last_undegraded":"2024-02-23T00:34:14.044532+0000","last_fullsized":"2024-02-23T00:34:14.044532+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.152014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.152014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":307,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.042257+0000","last_change":"2024-02-23T00:31:27.176319+0000","last_active":"2024-02-23T00:34:14.042257+0000","last_peered":"2024-02-23T00:34:14.042257+0000","last_clean":"2024-02-23T00:34:14.042257+0000","last_became_active":"2024-02-23T00:27:19.020285+0000","last_became_peered":"2024-02-23T00:27:19.020285+0000","last_unstale":"2024-02-23T00:34:14.042257+0000","last_undegraded":"2024-02-23T00:34:14.042257+0000","last_fullsized":"2024-02-23T00:34:14.042257+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:31:27.176199+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:31:27.176199+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":270,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040422+0000","last_change":"2024-02-23T00:32:44.494080+0000","last_active":"2024-02-23T00:34:14.040422+0000","last_peered":"2024-02-23T00:34:14.040422+0000","last_clean":"2024-02-23T00:34:14.040422+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:34:14.040422+0000","last_undegraded":"2024-02-23T00:34:14.040422+0000","last_fullsized":"2024-02-23T00:34:14.040422+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.494004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.494004+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056261+0000","last_change":"2024-02-23T00:32:46.655443+0000","last_active":"2024-02-23T00:34:14.056261+0000","last_peered":"2024-02-23T00:34:14.056261+0000","last_clean":"2024-02-23T00:34:14.056261+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:34:14.056261+0000","last_undegraded":"2024-02-23T00:34:14.056261+0000","last_fullsized":"2024-02-23T00:34:14.056261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.655378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.655378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.042027+0000","last_change":"2024-02-23T00:33:30.840877+0000","last_active":"2024-02-23T00:34:14.042027+0000","last_peered":"2024-02-23T00:34:14.042027+0000","last_clean":"2024-02-23T00:34:14.042027+0000","last_became_active":"2024-02-23T00:27:23.181174+0000","last_became_peered":"2024-02-23T00:27:23.181174+0000","last_unstale":"2024-02-23T00:34:14.042027+0000","last_undegraded":"2024-02-23T00:34:14.042027+0000","last_fullsized":"2024-02-23T00:34:14.042027+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:30.840800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:30.840800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":220,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046508+0000","last_change":"2024-02-23T00:33:45.795350+0000","last_active":"2024-02-23T00:34:14.046508+0000","last_peered":"2024-02-23T00:34:14.046508+0000","last_clean":"2024-02-23T00:34:14.046508+0000","last_became_active":"2024-02-23T00:30:48.722602+0000","last_became_peered":"2024-02-23T00:30:48.722602+0000","last_unstale":"2024-02-23T00:34:14.046508+0000","last_undegraded":"2024-02-23T00:34:14.046508+0000","last_fullsized":"2024-02-23T00:34:14.046508+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:45.795278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.795278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"92'22","reported_seq":307,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043676+0000","last_change":"2024-02-23T00:32:43.560670+0000","last_active":"2024-02-23T00:34:14.043676+0000","last_peered":"2024-02-23T00:34:14.043676+0000","last_clean":"2024-02-23T00:34:14.043676+0000","last_became_active":"2024-02-23T00:27:19.017247+0000","last_became_peered":"2024-02-23T00:27:19.017247+0000","last_unstale":"2024-02-23T00:34:14.043676+0000","last_undegraded":"2024-02-23T00:34:14.043676+0000","last_fullsized":"2024-02-23T00:34:14.043676+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:32:43.560587+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-02-23T00:30:38.900761+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.560587+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":264,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043212+0000","last_change":"2024-02-23T00:31:20.333610+0000","last_active":"2024-02-23T00:34:14.043212+0000","last_peered":"2024-02-23T00:34:14.043212+0000","last_clean":"2024-02-23T00:34:14.043212+0000","last_became_active":"2024-02-23T00:27:17.010508+0000","last_became_peered":"2024-02-23T00:27:17.010508+0000","last_unstale":"2024-02-23T00:34:14.043212+0000","last_undegraded":"2024-02-23T00:34:14.043212+0000","last_fullsized":"2024-02-23T00:34:14.043212+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:20.333547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:20.333547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":367,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-02-23T00:34:15.052614+0000","last_change":"2024-02-23T00:34:12.258420+0000","last_active":"2024-02-23T00:34:15.052614+0000","last_peered":"2024-02-23T00:34:15.052614+0000","last_clean":"2024-02-23T00:34:15.052614+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:34:15.052614+0000","last_undegraded":"2024-02-23T00:34:15.052614+0000","last_fullsized":"2024-02-23T00:34:15.052614+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:34:12.258344+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.258344+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056294+0000","last_change":"2024-02-23T00:32:50.706776+0000","last_active":"2024-02-23T00:34:14.056294+0000","last_peered":"2024-02-23T00:34:14.056294+0000","last_clean":"2024-02-23T00:34:14.056294+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:34:14.056294+0000","last_undegraded":"2024-02-23T00:34:14.056294+0000","last_fullsized":"2024-02-23T00:34:14.056294+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:32:50.706693+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.706693+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":219,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055909+0000","last_change":"2024-02-23T00:33:27.425684+0000","last_active":"2024-02-23T00:34:14.055909+0000","last_peered":"2024-02-23T00:34:14.055909+0000","last_clean":"2024-02-23T00:34:14.055909+0000","last_became_active":"2024-02-23T00:30:48.745982+0000","last_became_peered":"2024-02-23T00:30:48.745982+0000","last_unstale":"2024-02-23T00:34:14.055909+0000","last_undegraded":"2024-02-23T00:34:14.055909+0000","last_fullsized":"2024-02-23T00:34:14.055909+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:27.425604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:27.425604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"63'17","reported_seq":303,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040517+0000","last_change":"2024-02-23T00:32:59.459127+0000","last_active":"2024-02-23T00:34:14.040517+0000","last_peered":"2024-02-23T00:34:14.040517+0000","last_clean":"2024-02-23T00:34:14.040517+0000","last_became_active":"2024-02-23T00:27:19.024623+0000","last_became_peered":"2024-02-23T00:27:19.024623+0000","last_unstale":"2024-02-23T00:34:14.040517+0000","last_undegraded":"2024-02-23T00:34:14.040517+0000","last_fullsized":"2024-02-23T00:34:14.040517+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'17","last_scrub_stamp":"2024-02-23T00:32:59.459053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.459053+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":273,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.042154+0000","last_change":"2024-02-23T00:32:37.932120+0000","last_active":"2024-02-23T00:34:14.042154+0000","last_peered":"2024-02-23T00:34:14.042154+0000","last_clean":"2024-02-23T00:34:14.042154+0000","last_became_active":"2024-02-23T00:27:17.012290+0000","last_became_peered":"2024-02-23T00:27:17.012290+0000","last_unstale":"2024-02-23T00:34:14.042154+0000","last_undegraded":"2024-02-23T00:34:14.042154+0000","last_fullsized":"2024-02-23T00:34:14.042154+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:37.932055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:49.056184+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.932055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":269,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043757+0000","last_change":"2024-02-23T00:32:35.624529+0000","last_active":"2024-02-23T00:34:14.043757+0000","last_peered":"2024-02-23T00:34:14.043757+0000","last_clean":"2024-02-23T00:34:14.043757+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:34:14.043757+0000","last_undegraded":"2024-02-23T00:34:14.043757+0000","last_fullsized":"2024-02-23T00:34:14.043757+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.624461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.624461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":264,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045552+0000","last_change":"2024-02-23T00:32:10.145767+0000","last_active":"2024-02-23T00:34:14.045552+0000","last_peered":"2024-02-23T00:34:14.045552+0000","last_clean":"2024-02-23T00:34:14.045552+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:34:14.045552+0000","last_undegraded":"2024-02-23T00:34:14.045552+0000","last_fullsized":"2024-02-23T00:34:14.045552+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.145685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.145685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":219,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056873+0000","last_change":"2024-02-23T00:32:58.681718+0000","last_active":"2024-02-23T00:34:14.056873+0000","last_peered":"2024-02-23T00:34:14.056873+0000","last_clean":"2024-02-23T00:34:14.056873+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:34:14.056873+0000","last_undegraded":"2024-02-23T00:34:14.056873+0000","last_fullsized":"2024-02-23T00:34:14.056873+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.681626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.681626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"62'15","reported_seq":301,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040598+0000","last_change":"2024-02-23T00:33:28.841007+0000","last_active":"2024-02-23T00:34:14.040598+0000","last_peered":"2024-02-23T00:34:14.040598+0000","last_clean":"2024-02-23T00:34:14.040598+0000","last_became_active":"2024-02-23T00:27:19.022200+0000","last_became_peered":"2024-02-23T00:27:19.022200+0000","last_unstale":"2024-02-23T00:34:14.040598+0000","last_undegraded":"2024-02-23T00:34:14.040598+0000","last_fullsized":"2024-02-23T00:34:14.040598+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:33:28.840915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:28.840915+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":276,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039988+0000","last_change":"2024-02-23T00:33:33.425803+0000","last_active":"2024-02-23T00:34:14.039988+0000","last_peered":"2024-02-23T00:34:14.039988+0000","last_clean":"2024-02-23T00:34:14.039988+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:34:14.039988+0000","last_undegraded":"2024-02-23T00:34:14.039988+0000","last_fullsized":"2024-02-23T00:34:14.039988+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.425727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.425727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":263,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045967+0000","last_change":"2024-02-23T00:31:38.887671+0000","last_active":"2024-02-23T00:34:14.045967+0000","last_peered":"2024-02-23T00:34:14.045967+0000","last_clean":"2024-02-23T00:34:14.045967+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:34:14.045967+0000","last_undegraded":"2024-02-23T00:34:14.045967+0000","last_fullsized":"2024-02-23T00:34:14.045967+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.887597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":262,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046684+0000","last_change":"2024-02-23T00:32:12.983101+0000","last_active":"2024-02-23T00:34:14.046684+0000","last_peered":"2024-02-23T00:34:14.046684+0000","last_clean":"2024-02-23T00:34:14.046684+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:34:14.046684+0000","last_undegraded":"2024-02-23T00:34:14.046684+0000","last_fullsized":"2024-02-23T00:34:14.046684+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.983023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.983023+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":218,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044766+0000","last_change":"2024-02-23T00:33:28.419457+0000","last_active":"2024-02-23T00:34:14.044766+0000","last_peered":"2024-02-23T00:34:14.044766+0000","last_clean":"2024-02-23T00:34:14.044766+0000","last_became_active":"2024-02-23T00:30:48.710049+0000","last_became_peered":"2024-02-23T00:30:48.710049+0000","last_unstale":"2024-02-23T00:34:14.044766+0000","last_undegraded":"2024-02-23T00:34:14.044766+0000","last_fullsized":"2024-02-23T00:34:14.044766+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:28.419374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:28.419374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"76'24","reported_seq":315,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.039894+0000","last_change":"2024-02-23T00:34:12.587999+0000","last_active":"2024-02-23T00:34:14.039894+0000","last_peered":"2024-02-23T00:34:14.039894+0000","last_clean":"2024-02-23T00:34:14.039894+0000","last_became_active":"2024-02-23T00:27:19.022135+0000","last_became_peered":"2024-02-23T00:27:19.022135+0000","last_unstale":"2024-02-23T00:34:14.039894+0000","last_undegraded":"2024-02-23T00:34:14.039894+0000","last_fullsized":"2024-02-23T00:34:14.039894+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:34:12.587921+0000","last_deep_scrub":"76'24","last_deep_scrub_stamp":"2024-02-23T00:28:33.077848+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.587921+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":27,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":273,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041844+0000","last_change":"2024-02-23T00:33:33.750868+0000","last_active":"2024-02-23T00:34:14.041844+0000","last_peered":"2024-02-23T00:34:14.041844+0000","last_clean":"2024-02-23T00:34:14.041844+0000","last_became_active":"2024-02-23T00:27:17.015983+0000","last_became_peered":"2024-02-23T00:27:17.015983+0000","last_unstale":"2024-02-23T00:34:14.041844+0000","last_undegraded":"2024-02-23T00:34:14.041844+0000","last_fullsized":"2024-02-23T00:34:14.041844+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.750804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.750804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":264,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046105+0000","last_change":"2024-02-23T00:31:45.961466+0000","last_active":"2024-02-23T00:34:14.046105+0000","last_peered":"2024-02-23T00:34:14.046105+0000","last_clean":"2024-02-23T00:34:14.046105+0000","last_became_active":"2024-02-23T00:27:21.024455+0000","last_became_peered":"2024-02-23T00:27:21.024455+0000","last_unstale":"2024-02-23T00:34:14.046105+0000","last_undegraded":"2024-02-23T00:34:14.046105+0000","last_fullsized":"2024-02-23T00:34:14.046105+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:45.961386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:45.961386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.040111+0000","last_change":"2024-02-23T00:32:46.021732+0000","last_active":"2024-02-23T00:34:14.040111+0000","last_peered":"2024-02-23T00:34:14.040111+0000","last_clean":"2024-02-23T00:34:14.040111+0000","last_became_active":"2024-02-23T00:27:23.181083+0000","last_became_peered":"2024-02-23T00:27:23.181083+0000","last_unstale":"2024-02-23T00:34:14.040111+0000","last_undegraded":"2024-02-23T00:34:14.040111+0000","last_fullsized":"2024-02-23T00:34:14.040111+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.021662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.021662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":218,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045367+0000","last_change":"2024-02-23T00:33:33.382591+0000","last_active":"2024-02-23T00:34:14.045367+0000","last_peered":"2024-02-23T00:34:14.045367+0000","last_clean":"2024-02-23T00:34:14.045367+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:34:14.045367+0000","last_undegraded":"2024-02-23T00:34:14.045367+0000","last_fullsized":"2024-02-23T00:34:14.045367+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.382510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.382510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"63'21","reported_seq":317,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.045014+0000","last_change":"2024-02-23T00:32:44.870419+0000","last_active":"2024-02-23T00:34:14.045014+0000","last_peered":"2024-02-23T00:34:14.045014+0000","last_clean":"2024-02-23T00:34:14.045014+0000","last_became_active":"2024-02-23T00:27:19.022440+0000","last_became_peered":"2024-02-23T00:27:19.022440+0000","last_unstale":"2024-02-23T00:34:14.045014+0000","last_undegraded":"2024-02-23T00:34:14.045014+0000","last_fullsized":"2024-02-23T00:34:14.045014+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:32:44.870335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.870335+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":271,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055667+0000","last_change":"2024-02-23T00:32:45.663968+0000","last_active":"2024-02-23T00:34:14.055667+0000","last_peered":"2024-02-23T00:34:14.055667+0000","last_clean":"2024-02-23T00:34:14.055667+0000","last_became_active":"2024-02-23T00:27:17.015960+0000","last_became_peered":"2024-02-23T00:27:17.015960+0000","last_unstale":"2024-02-23T00:34:14.055667+0000","last_undegraded":"2024-02-23T00:34:14.055667+0000","last_fullsized":"2024-02-23T00:34:14.055667+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.663905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.663905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":265,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044482+0000","last_change":"2024-02-23T00:33:43.299920+0000","last_active":"2024-02-23T00:34:14.044482+0000","last_peered":"2024-02-23T00:34:14.044482+0000","last_clean":"2024-02-23T00:34:14.044482+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:34:14.044482+0000","last_undegraded":"2024-02-23T00:34:14.044482+0000","last_fullsized":"2024-02-23T00:34:14.044482+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046049+0000","last_change":"2024-02-23T00:32:39.644590+0000","last_active":"2024-02-23T00:34:14.046049+0000","last_peered":"2024-02-23T00:34:14.046049+0000","last_clean":"2024-02-23T00:34:14.046049+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:34:14.046049+0000","last_undegraded":"2024-02-23T00:34:14.046049+0000","last_fullsized":"2024-02-23T00:34:14.046049+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:39.644483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:32:39.644483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":219,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.055466+0000","last_change":"2024-02-23T00:32:35.725253+0000","last_active":"2024-02-23T00:34:14.055466+0000","last_peered":"2024-02-23T00:34:14.055466+0000","last_clean":"2024-02-23T00:34:14.055466+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:34:14.055466+0000","last_undegraded":"2024-02-23T00:34:14.055466+0000","last_fullsized":"2024-02-23T00:34:14.055466+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:35.725182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:35.725182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"78'14","reported_seq":299,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.046334+0000","last_change":"2024-02-23T00:32:58.673712+0000","last_active":"2024-02-23T00:34:14.046334+0000","last_peered":"2024-02-23T00:34:14.046334+0000","last_clean":"2024-02-23T00:34:14.046334+0000","last_became_active":"2024-02-23T00:27:19.024437+0000","last_became_peered":"2024-02-23T00:27:19.024437+0000","last_unstale":"2024-02-23T00:34:14.046334+0000","last_undegraded":"2024-02-23T00:34:14.046334+0000","last_fullsized":"2024-02-23T00:34:14.046334+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'14","last_scrub_stamp":"2024-02-23T00:32:58.673628+0000","last_deep_scrub":"78'14","last_deep_scrub_stamp":"2024-02-23T00:28:54.657177+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.673628+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":271,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056368+0000","last_change":"2024-02-23T00:32:36.714218+0000","last_active":"2024-02-23T00:34:14.056368+0000","last_peered":"2024-02-23T00:34:14.056368+0000","last_clean":"2024-02-23T00:34:14.056368+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:34:14.056368+0000","last_undegraded":"2024-02-23T00:34:14.056368+0000","last_fullsized":"2024-02-23T00:34:14.056368+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.714139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.714139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":267,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056053+0000","last_change":"2024-02-23T00:32:44.640411+0000","last_active":"2024-02-23T00:34:14.056053+0000","last_peered":"2024-02-23T00:34:14.056053+0000","last_clean":"2024-02-23T00:34:14.056053+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:34:14.056053+0000","last_undegraded":"2024-02-23T00:34:14.056053+0000","last_fullsized":"2024-02-23T00:34:14.056053+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:44.640331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:32:44.640331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":258,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043694+0000","last_change":"2024-02-23T00:31:39.494827+0000","last_active":"2024-02-23T00:34:14.043694+0000","last_peered":"2024-02-23T00:34:14.043694+0000","last_clean":"2024-02-23T00:34:14.043694+0000","last_became_active":"2024-02-23T00:27:23.031224+0000","last_became_peered":"2024-02-23T00:27:23.031224+0000","last_unstale":"2024-02-23T00:34:14.043694+0000","last_undegraded":"2024-02-23T00:34:14.043694+0000","last_fullsized":"2024-02-23T00:34:14.043694+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:31:39.494737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":217,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043260+0000","last_change":"2024-02-23T00:33:33.192544+0000","last_active":"2024-02-23T00:34:14.043260+0000","last_peered":"2024-02-23T00:34:14.043260+0000","last_clean":"2024-02-23T00:34:14.043260+0000","last_became_active":"2024-02-23T00:30:48.725162+0000","last_became_peered":"2024-02-23T00:30:48.725162+0000","last_unstale":"2024-02-23T00:34:14.043260+0000","last_undegraded":"2024-02-23T00:34:14.043260+0000","last_fullsized":"2024-02-23T00:34:14.043260+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.192483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.192483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"81'20","reported_seq":302,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044143+0000","last_change":"2024-02-23T00:32:10.903259+0000","last_active":"2024-02-23T00:34:14.044143+0000","last_peered":"2024-02-23T00:34:14.044143+0000","last_clean":"2024-02-23T00:34:14.044143+0000","last_became_active":"2024-02-23T00:27:19.023248+0000","last_became_peered":"2024-02-23T00:27:19.023248+0000","last_unstale":"2024-02-23T00:34:14.044143+0000","last_undegraded":"2024-02-23T00:34:14.044143+0000","last_fullsized":"2024-02-23T00:34:14.044143+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:32:10.903186+0000","last_deep_scrub":"81'20","last_deep_scrub_stamp":"2024-02-23T00:29:06.822818+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.903186+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":270,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041234+0000","last_change":"2024-02-23T00:31:22.137283+0000","last_active":"2024-02-23T00:34:14.041234+0000","last_peered":"2024-02-23T00:34:14.041234+0000","last_clean":"2024-02-23T00:34:14.041234+0000","last_became_active":"2024-02-23T00:27:17.013279+0000","last_became_peered":"2024-02-23T00:27:17.013279+0000","last_unstale":"2024-02-23T00:34:14.041234+0000","last_undegraded":"2024-02-23T00:34:14.041234+0000","last_fullsized":"2024-02-23T00:34:14.041234+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:22.137166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:31:22.137166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":261,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.042839+0000","last_change":"2024-02-23T00:31:26.920222+0000","last_active":"2024-02-23T00:34:14.042839+0000","last_peered":"2024-02-23T00:34:14.042839+0000","last_clean":"2024-02-23T00:34:14.042839+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:34:14.042839+0000","last_undegraded":"2024-02-23T00:34:14.042839+0000","last_fullsized":"2024-02-23T00:34:14.042839+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:26.920133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:26.920133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":264,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044914+0000","last_change":"2024-02-23T00:31:37.065941+0000","last_active":"2024-02-23T00:34:14.044914+0000","last_peered":"2024-02-23T00:34:14.044914+0000","last_clean":"2024-02-23T00:34:14.044914+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:34:14.044914+0000","last_undegraded":"2024-02-23T00:34:14.044914+0000","last_fullsized":"2024-02-23T00:34:14.044914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:31:37.065865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":219,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.056616+0000","last_change":"2024-02-23T00:33:29.354309+0000","last_active":"2024-02-23T00:34:14.056616+0000","last_peered":"2024-02-23T00:34:14.056616+0000","last_clean":"2024-02-23T00:34:14.056616+0000","last_became_active":"2024-02-23T00:30:48.697963+0000","last_became_peered":"2024-02-23T00:30:48.697963+0000","last_unstale":"2024-02-23T00:34:14.056616+0000","last_undegraded":"2024-02-23T00:34:14.056616+0000","last_fullsized":"2024-02-23T00:34:14.056616+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:29.354235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:29.354235+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"278'50","reported_seq":367,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.042004+0000","last_change":"2024-02-23T00:33:49.576025+0000","last_active":"2024-02-23T00:34:14.042004+0000","last_peered":"2024-02-23T00:34:14.042004+0000","last_clean":"2024-02-23T00:34:14.042004+0000","last_became_active":"2024-02-23T00:27:19.020737+0000","last_became_peered":"2024-02-23T00:27:19.020737+0000","last_unstale":"2024-02-23T00:34:14.042004+0000","last_undegraded":"2024-02-23T00:34:14.042004+0000","last_fullsized":"2024-02-23T00:34:14.042004+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'50","last_scrub_stamp":"2024-02-23T00:33:49.575922+0000","last_deep_scrub":"278'50","last_deep_scrub_stamp":"2024-02-23T00:33:49.575922+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.575922+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":278,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.043474+0000","last_change":"2024-02-23T00:33:42.579086+0000","last_active":"2024-02-23T00:34:14.043474+0000","last_peered":"2024-02-23T00:34:14.043474+0000","last_clean":"2024-02-23T00:34:14.043474+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:34:14.043474+0000","last_undegraded":"2024-02-23T00:34:14.043474+0000","last_fullsized":"2024-02-23T00:34:14.043474+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":264,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044426+0000","last_change":"2024-02-23T00:31:40.071364+0000","last_active":"2024-02-23T00:34:14.044426+0000","last_peered":"2024-02-23T00:34:14.044426+0000","last_clean":"2024-02-23T00:34:14.044426+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:34:14.044426+0000","last_undegraded":"2024-02-23T00:34:14.044426+0000","last_fullsized":"2024-02-23T00:34:14.044426+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:40.071294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:40.071294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":262,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.041677+0000","last_change":"2024-02-23T00:30:46.848126+0000","last_active":"2024-02-23T00:34:14.041677+0000","last_peered":"2024-02-23T00:34:14.041677+0000","last_clean":"2024-02-23T00:34:14.041677+0000","last_became_active":"2024-02-23T00:27:23.182222+0000","last_became_peered":"2024-02-23T00:27:23.182222+0000","last_unstale":"2024-02-23T00:34:14.041677+0000","last_undegraded":"2024-02-23T00:34:14.041677+0000","last_fullsized":"2024-02-23T00:34:14.041677+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:30:46.848063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:30:46.848063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":217,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-02-23T00:34:14.044462+0000","last_change":"2024-02-23T00:32:56.396227+0000","last_active":"2024-02-23T00:34:14.044462+0000","last_peered":"2024-02-23T00:34:14.044462+0000","last_clean":"2024-02-23T00:34:14.044462+0000","last_became_active":"2024-02-23T00:30:48.722569+0000","last_became_peered":"2024-02-23T00:30:48.722569+0000","last_unstale":"2024-02-23T00:34:14.044462+0000","last_undegraded":"2024-02-23T00:34:14.044462+0000","last_fullsized":"2024-02-23T00:34:14.044462+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.396150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.396150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":4001220,"num_objects":123,"num_object_clones":2,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":548,"num_write_kb":4449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13492224,"data_stored":12003660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1796,"ondisk_log_size":1796,"up":96,"acting":96,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":191,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":190,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1082,"num_read_kb":807,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":651,"ondisk_log_size":651,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332183,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314436,"kb_used_data":4916,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430716,"statfs":{"total":95995035648,"available":95673053184,"internally_reserved":0,"allocated":5033984,"data_stored":2865673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":2.0800000000000001,"15min":1.0269999999999999},"min":{"1min":0.26300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.371,"5min":61.048999999999999,"15min":61.048999999999999},"last":0.378},{"interface":"front","average":{"1min":0.52400000000000002,"5min":2.653,"15min":1.2390000000000001},"min":{"1min":0.36799999999999999,"5min":0.214,"15min":0.214},"max":{"1min":1.1639999999999999,"5min":73.870999999999995,"15min":73.870999999999995},"last":0.71299999999999997}]},{"osd":1,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.024,"15min":0.69399999999999995},"min":{"1min":0.377,"5min":0.217,"15min":0.217},"max":{"1min":1.1200000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.624},{"interface":"front","average":{"1min":0.47799999999999998,"5min":1.038,"15min":0.68200000000000005},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.60799999999999998,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.60899999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.95399999999999996,"15min":0.66900000000000004},"min":{"1min":0.30099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.4119999999999999,"5min":16.603999999999999,"15min":16.603999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.748,"15min":0.59799999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.625,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.52900000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":1.8759999999999999,"15min":0.97299999999999998},"min":{"1min":0.34300000000000003,"5min":0.311,"15min":0.311},"max":{"1min":1.4419999999999999,"5min":57.729999999999997,"15min":57.729999999999997},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.609,"15min":0.89300000000000002},"min":{"1min":0.35699999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.80300000000000005,"5min":27.573,"15min":27.573},"last":0.66400000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.433,"15min":0.41699999999999998},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.46899999999999997,"15min":0.438},"min":{"1min":0.33700000000000002,"5min":0.25700000000000001,"15min":0.254},"max":{"1min":3.294,"5min":3.294,"15min":3.294},"last":0.64300000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.441,"15min":0.43099999999999999},"min":{"1min":0.32400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.85899999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47599999999999998,"15min":0.442},"min":{"1min":0.35899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.76700000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.503,"15min":0.47999999999999998},"min":{"1min":0.40999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.497,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.83299999999999996,"5min":1.002,"15min":1.002},"last":0.73099999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561116,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302196,"kb_used_data":4960,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93442956,"statfs":{"total":95995035648,"available":95685586944,"internally_reserved":0,"allocated":5079040,"data_stored":2870683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":1.708,"15min":0.89700000000000002},"min":{"1min":0.34799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.63700000000000001,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.71199999999999997,"15min":0.58099999999999996},"min":{"1min":0.28899999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.57999999999999996,"5min":9.468,"15min":9.468},"last":0.67400000000000004}]},{"osd":1,"last update":"Fri Feb 23 00:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":2.0369999999999999,"15min":1.0109999999999999},"min":{"1min":0.379,"5min":0.245,"15min":0.245},"max":{"1min":0.79100000000000004,"5min":59.347999999999999,"15min":59.347999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":2.8769999999999998,"15min":1.288},"min":{"1min":0.35099999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.58999999999999997,"5min":87.221000000000004,"15min":87.221000000000004},"last":0.42199999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":1.4259999999999999,"15min":0.80500000000000005},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.63300000000000001,"5min":25.550999999999998,"15min":25.550999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":1.5249999999999999,"15min":0.86799999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.64400000000000002,"5min":40.914000000000001,"15min":40.914000000000001},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Feb 23 00:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":1.4219999999999999,"15min":0.79900000000000004},"min":{"1min":0.378,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.2799999999999998,"5min":31.309999999999999,"15min":31.309999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":2.9209999999999998,"15min":1.3200000000000001},"min":{"1min":0.35799999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.69799999999999995,"5min":86.311999999999998,"15min":86.311999999999998},"last":0.47199999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.46500000000000002,"15min":0.42899999999999999},"min":{"1min":0.30599999999999999,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":0.67000000000000004,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":0.63},{"interface":"front","average":{"1min":0.504,"5min":0.67500000000000004,"15min":0.496},"min":{"1min":0.28699999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75600000000000001,"5min":17.718,"15min":17.718},"last":0.58199999999999996}]},{"osd":5,"last update":"Fri Feb 23 00:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.57999999999999996,"15min":0.47699999999999998},"min":{"1min":0.27500000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71299999999999997,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.56299999999999994,"15min":0.46800000000000003},"min":{"1min":0.31900000000000001,"5min":0.309,"15min":0.27500000000000002},"max":{"1min":0.66400000000000003,"5min":8.1370000000000005,"15min":8.1370000000000005},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.83499999999999996,"15min":0.58999999999999997},"min":{"1min":0.36899999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.70099999999999996,"5min":17.881,"15min":17.881},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.80700000000000005,"15min":0.58399999999999996},"min":{"1min":0.39600000000000002,"5min":0.28699999999999998,"15min":0.28199999999999997},"max":{"1min":0.72499999999999998,"5min":17.396999999999998,"15min":17.396999999999998},"last":0.56299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306292,"kb_used_data":9056,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93438860,"statfs":{"total":95995035648,"available":95681392640,"internally_reserved":0,"allocated":9273344,"data_stored":6872496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":1.3200000000000001,"15min":0.66300000000000003},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.2349999999999999,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.93600000000000005,"15min":0.55000000000000004},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.20799999999999999},"max":{"1min":0.93200000000000005,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.67200000000000004}]},{"osd":2,"last update":"Fri Feb 23 00:33:26 2024","interfaces":[{"interface":"back","average":{"1min":2.3879999999999999,"5min":1.3009999999999999,"15min":0.68100000000000005},"min":{"1min":0.22600000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":35.313000000000002,"5min":35.313000000000002,"15min":35.313000000000002},"last":0.84099999999999997},{"interface":"front","average":{"1min":2.0659999999999998,"5min":0.997,"15min":0.58299999999999996},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":27.908999999999999,"5min":27.908999999999999,"15min":27.908999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":1.1499999999999999,"15min":0.66500000000000004},"min":{"1min":0.255,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.3919999999999999,"5min":21.013999999999999,"15min":21.013999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":1.111,"15min":0.66100000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":10.382999999999999,"5min":29.234999999999999,"15min":29.234999999999999},"last":0.71199999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:34:06 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.999,"15min":0.70099999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":8.5559999999999992,"5min":16.231000000000002,"15min":16.231000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.0960000000000001,"15min":0.72999999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":1.135,"5min":29.212,"15min":29.212},"last":0.56000000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:33:32 2024","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.052,"15min":0.73399999999999999},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":16.294,"5min":16.294,"15min":16.294},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.96999999999999997,"5min":0.91600000000000004,"15min":0.69499999999999995},"min":{"1min":0.36599999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":8.4030000000000005,"5min":11.105,"15min":11.105},"last":0.63800000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:34:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.173,"15min":0.77000000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":10.416,"5min":29.187000000000001,"15min":29.187000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.0589999999999999,"15min":0.76300000000000001},"min":{"1min":0.38400000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":8.2929999999999993,"5min":16.369,"15min":16.369},"last":0.65500000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":1.163,"15min":0.76900000000000002},"min":{"1min":0.41599999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":16.395,"5min":16.395,"15min":16.395},"last":0.627},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.81399999999999995,"15min":0.67900000000000005},"min":{"1min":0.40999999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86399999999999999,"5min":11.135999999999999,"15min":11.135999999999999},"last":0.70299999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310416,"kb_used_data":4992,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93434736,"statfs":{"total":95995035648,"available":95677169664,"internally_reserved":0,"allocated":5111808,"data_stored":2861475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.81799999999999995,"15min":0.49399999999999999},"min":{"1min":0.20100000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.495,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.309},{"interface":"front","average":{"1min":0.318,"5min":1.169,"15min":0.61599999999999999},"min":{"1min":0.224,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45200000000000001,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.99099999999999999,"15min":0.59299999999999997},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":3.96,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.754,"15min":0.49299999999999999},"min":{"1min":0.27100000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.2770000000000001,"5min":15.965,"15min":15.965},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.0980000000000001,"15min":0.66100000000000003},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":10.121,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.0429999999999999,"15min":0.64400000000000002},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":10.378,"5min":16.006,"15min":16.006},"last":1.1359999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.82999999999999996,"15min":0.64600000000000002},"min":{"1min":0.36899999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.4780000000000002,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.175,"15min":0.76500000000000001},"min":{"1min":0.41499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.399,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.67200000000000004}]},{"osd":5,"last update":"Fri Feb 23 00:33:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.091,"15min":0.72999999999999998},"min":{"1min":0.41999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.5779999999999998,"5min":13.311,"15min":13.311},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.1279999999999999,"15min":0.746},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":5.032,"5min":23.744,"15min":23.744},"last":0.47999999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.363,"15min":0.80300000000000005},"min":{"1min":0.38400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75800000000000001,"5min":33.082999999999998,"15min":33.082999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.997,"15min":0.71099999999999997},"min":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69899999999999995,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.71199999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.2310000000000001,"15min":0.78300000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.373},"max":{"1min":5.3369999999999997,"5min":31.251999999999999,"15min":31.251999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.1559999999999999,"15min":0.75800000000000001},"min":{"1min":0.44500000000000001,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":1.998,"5min":23.785,"15min":23.785},"last":0.54700000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411439,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318404,"kb_used_data":8888,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426748,"statfs":{"total":95995035648,"available":95668989952,"internally_reserved":0,"allocated":9101312,"data_stored":6865061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.5840000000000001,"15min":0.751},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":10.878,"5min":56.509,"15min":56.509},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":1.452,"15min":0.70699999999999996},"min":{"1min":0.23400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.1859999999999999,"5min":31.498999999999999,"15min":31.498999999999999},"last":0.34699999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.99299999999999999,"15min":0.56000000000000005},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.9649999999999999,"5min":19.324999999999999,"15min":19.324999999999999},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.4099999999999999,"15min":0.69899999999999995},"min":{"1min":0.23599999999999999,"5min":0.22900000000000001,"15min":0.216},"max":{"1min":13.209,"5min":35.052,"15min":35.052},"last":0.32200000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:33:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":1.268,"15min":0.69599999999999995},"min":{"1min":0.24399999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.726,"5min":43.5,"15min":43.5},"last":0.373},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.0489999999999999,"15min":0.64000000000000001},"min":{"1min":0.27600000000000002,"5min":0.25,"15min":0.25},"max":{"1min":10.843,"5min":27.414000000000001,"15min":27.414000000000001},"last":0.28999999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:34:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.78300000000000003,"15min":0.62},"min":{"1min":0.35899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0939999999999999,"5min":14.614000000000001,"15min":14.614000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68500000000000005,"15min":0.59699999999999998},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":3.903,"5min":4.6550000000000002,"15min":4.6550000000000002},"last":0.48199999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.90400000000000003,"15min":0.66400000000000003},"min":{"1min":0.34300000000000003,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":3.2879999999999998,"5min":17.765000000000001,"15min":17.765000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.79100000000000004,"15min":0.63300000000000001},"min":{"1min":0.32900000000000001,"5min":0.317,"15min":0.317},"max":{"1min":2.1259999999999999,"5min":14.589,"15min":14.589},"last":0.59699999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:33:56 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.70399999999999996,"15min":0.60099999999999998},"min":{"1min":0.373,"5min":0.317,"15min":0.317},"max":{"1min":3.9359999999999999,"5min":4.6120000000000001,"15min":4.6120000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":1.0740000000000001,"15min":0.752},"min":{"1min":0.36699999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.3410000000000002,"5min":20.574999999999999,"15min":20.574999999999999},"last":0.55900000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":1.9059999999999999,"15min":1.0309999999999999},"min":{"1min":0.38200000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":3.3780000000000001,"5min":66.632000000000005,"15min":66.632000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":1.226,"15min":0.80600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.157,"5min":19.603999999999999,"15min":19.603999999999999},"last":0.61699999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318340,"kb_used_data":8824,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426812,"statfs":{"total":95995035648,"available":95669055488,"internally_reserved":0,"allocated":9035776,"data_stored":6861215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":1.0329999999999999,"15min":0.59099999999999997},"min":{"1min":0.222,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.42399999999999999,"5min":45.390999999999998,"15min":45.390999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.316,"5min":1.0069999999999999,"15min":0.59799999999999998},"min":{"1min":0.25600000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.39900000000000002,"5min":30.186,"15min":30.186},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.61899999999999999,"15min":0.48799999999999999},"min":{"1min":0.20799999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":0.41399999999999998,"5min":18.039999999999999,"15min":18.039999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":1.0129999999999999,"15min":0.61399999999999999},"min":{"1min":0.26000000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.48299999999999998,"5min":45.441000000000003,"15min":45.441000000000003},"last":0.255}]},{"osd":2,"last update":"Fri Feb 23 00:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.96399999999999997,"15min":0.61299999999999999},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51300000000000001,"5min":45.462000000000003,"15min":45.462000000000003},"last":0.314},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.90700000000000003,"15min":0.58599999999999997},"min":{"1min":0.30199999999999999,"5min":0.185,"15min":0.185},"max":{"1min":0.436,"5min":24.183,"15min":24.183},"last":0.34000000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":1.069,"15min":0.73899999999999999},"min":{"1min":0.35899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.75600000000000001,"5min":24.058,"15min":24.058},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.73899999999999999,"15min":0.63200000000000001},"min":{"1min":0.33700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.81599999999999995,"5min":8.0359999999999996,"15min":8.0359999999999996},"last":0.67200000000000004}]},{"osd":5,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":1.056,"15min":0.73099999999999998},"min":{"1min":0.38,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70299999999999996,"5min":45.265000000000001,"15min":45.265000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.91700000000000004,"15min":0.69099999999999995},"min":{"1min":0.40200000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71699999999999997,"5min":24.102,"15min":24.102},"last":0.61099999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:33:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.752,"15min":0.63400000000000001},"min":{"1min":0.41699999999999998,"5min":0.35199999999999998,"15min":0.34599999999999997},"max":{"1min":0.84999999999999998,"5min":8.093,"15min":8.093},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.1759999999999999,"15min":0.79700000000000004},"min":{"1min":0.373,"5min":0.371,"15min":0.36299999999999999},"max":{"1min":0.76900000000000002,"5min":45.232999999999997,"15min":45.232999999999997},"last":0.69699999999999995}]},{"osd":7,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.71199999999999997,"15min":0.59999999999999998},"min":{"1min":0.38800000000000001,"5min":0.379,"15min":0.34300000000000003},"max":{"1min":0.71399999999999997,"5min":8.1280000000000001,"15min":8.1280000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":1.0840000000000001,"15min":0.76800000000000002},"min":{"1min":0.371,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78700000000000003,"5min":45.287999999999997,"15min":45.287999999999997},"last":0.71799999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986278,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302252,"kb_used_data":5020,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93442900,"statfs":{"total":95995035648,"available":95685529600,"internally_reserved":0,"allocated":5140480,"data_stored":2865155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":1.49,"15min":0.83799999999999997},"min":{"1min":0.23400000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":8.0449999999999999,"5min":21.419,"15min":21.419},"last":0.436},{"interface":"front","average":{"1min":1.105,"5min":2.7389999999999999,"15min":1.264},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.2629999999999999,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.35699999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.9710000000000001,"15min":1},"min":{"1min":0.32900000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.73,"5min":41.137999999999998,"15min":41.137999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.72899999999999998,"5min":1.4910000000000001,"15min":0.84099999999999997},"min":{"1min":0.34599999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.3740000000000001,"5min":26.210000000000001,"15min":26.210000000000001},"last":0.39800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":2.2010000000000001,"15min":1.0860000000000001},"min":{"1min":0.374,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.1959999999999997,"5min":67.253,"15min":67.253},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.9370000000000001,"15min":0.99299999999999999},"min":{"1min":0.38900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.0199999999999996,"5min":41.927999999999997,"15min":41.927999999999997},"last":0.45000000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:34:03 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":2.0680000000000001,"15min":1.0369999999999999},"min":{"1min":0.36099999999999999,"5min":0.35499999999999998,"15min":0.34000000000000002},"max":{"1min":4.6429999999999998,"5min":36.301000000000002,"15min":36.301000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":2.2970000000000002,"15min":1.115},"min":{"1min":0.38900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.8580000000000001,"5min":42.064,"15min":42.064},"last":0.374}]},{"osd":5,"last update":"Fri Feb 23 00:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.86599999999999999,"15min":0.53000000000000003},"min":{"1min":0.28799999999999998,"5min":0.217,"15min":0.217},"max":{"1min":8.5299999999999994,"5min":14.242000000000001,"15min":14.242000000000001},"last":0.621},{"interface":"front","average":{"1min":1.087,"5min":0.83399999999999996,"15min":0.54100000000000004},"min":{"1min":0.38,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":8.3010000000000002,"5min":14.124000000000001,"15min":14.124000000000001},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:33:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.70699999999999996,"15min":0.53400000000000003},"min":{"1min":0.33000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":8.0730000000000004,"5min":8.0730000000000004,"15min":8.0730000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.72599999999999998,"5min":1.0089999999999999,"15min":0.623},"min":{"1min":0.34599999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.71,"5min":45.814999999999998,"15min":45.814999999999998},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.84899999999999998,"15min":0.59099999999999997},"min":{"1min":0.32800000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.165,"5min":17.178999999999998,"15min":17.178999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.0569999999999999,"5min":0.88800000000000001,"15min":0.63200000000000001},"min":{"1min":0.38900000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.9690000000000003,"5min":14.212,"15min":14.212},"last":0.66500000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757345,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314576,"kb_used_data":5052,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430576,"statfs":{"total":95995035648,"available":95672909824,"internally_reserved":0,"allocated":5173248,"data_stored":2870873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.123,"15min":0.70999999999999996},"min":{"1min":0.40899999999999997,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.81499999999999995,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":2.1059999999999999,"15min":1.0269999999999999},"min":{"1min":0.39000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68500000000000005,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.35399999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":1.2909999999999999,"15min":0.749},"min":{"1min":0.38600000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.83799999999999997,"5min":38.838000000000001,"15min":38.838000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":2.5249999999999999,"15min":1.171},"min":{"1min":0.372,"5min":0.251,"15min":0.251},"max":{"1min":0.71099999999999997,"5min":127.794,"15min":127.794},"last":0.43099999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.657,"15min":0.88600000000000001},"min":{"1min":0.40500000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72399999999999998,"5min":37.442999999999998,"15min":37.442999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.071,"15min":0.69399999999999995},"min":{"1min":0.44900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69599999999999995,"5min":23.297999999999998,"15min":23.297999999999998},"last":0.53300000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":1.238,"15min":0.73299999999999998},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.76900000000000002,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.629,"15min":0.89100000000000001},"min":{"1min":0.41799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.74199999999999999,"5min":40.865000000000002,"15min":40.865000000000002},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Feb 23 00:33:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":1.5620000000000001,"15min":0.76700000000000002},"min":{"1min":0.29999999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.70899999999999996,"5min":103.13,"15min":103.13},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.58699999999999997,"15min":0.442},"min":{"1min":0.32000000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.74399999999999999,"5min":12.795999999999999,"15min":12.795999999999999},"last":0.58399999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.58699999999999997,"15min":0.47999999999999998},"min":{"1min":0.32200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72899999999999998,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.63},{"interface":"front","average":{"1min":0.53900000000000003,"5min":1.573,"15min":0.81499999999999995},"min":{"1min":0.40400000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70899999999999996,"5min":103.61499999999999,"15min":103.61499999999999},"last":0.52200000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.57799999999999996,"15min":0.51600000000000001},"min":{"1min":0.36299999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":7.1820000000000004,"15min":7.1820000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52400000000000002,"15min":0.51100000000000001},"min":{"1min":0.33100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4304896,"data_stored":4000739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4186112,"data_stored":4000473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096000,"data_stored":4000243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:34:22.994 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-23T00:34:22.994 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.log pg_num 22 2024-02-23T00:34:23.010 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:23.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:23.115 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:23.144 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:23.172 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:23.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:23.277 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:23.306 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:23.331 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:23.333 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:34:23.333 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:34:23.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:23.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:23.436 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:23.463 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:23.481 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:23.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:23.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:21.617143+0000 mgr.y (mgr.24437) 312 : cluster [DBG] pgmap v448: 246 pgs: 8 creating+peering, 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:23.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:22.393188+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:23.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:22.393400+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:23.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:22.393999+0000 mon.b (mon.1) 276 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41"}]: dispatch 2024-02-23T00:34:23.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:22.394046+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41"}]: dispatch 2024-02-23T00:34:23.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:22.474213+0000 osd.5 (osd.5) 143 : cluster [DBG] 5.f scrub starts 2024-02-23T00:34:23.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:22.474953+0000 osd.5 (osd.5) 144 : cluster [DBG] 5.f scrub ok 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:22.636948+0000 osd.4 (osd.4) 145 : cluster [DBG] 4.c scrub starts 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:22.638636+0000 osd.4 (osd.4) 146 : cluster [DBG] 4.c scrub ok 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:23.043575+0000 osd.1 (osd.1) 148 : cluster [DBG] 63.2 scrub starts 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:23.046567+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]': finished 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:23.046679+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41"}]': finished 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:23.046755+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:23.049390+0000 osd.1 (osd.1) 149 : cluster [DBG] 63.2 scrub ok 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:23.050263+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:23.050610+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:21.617143+0000 mgr.y (mgr.24437) 312 : cluster [DBG] pgmap v448: 246 pgs: 8 creating+peering, 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:22.393188+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:23.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:22.393400+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:22.393999+0000 mon.b (mon.1) 276 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41"}]: dispatch 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:22.394046+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41"}]: dispatch 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:22.474213+0000 osd.5 (osd.5) 143 : cluster [DBG] 5.f scrub starts 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:22.474953+0000 osd.5 (osd.5) 144 : cluster [DBG] 5.f scrub ok 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:22.636948+0000 osd.4 (osd.4) 145 : cluster [DBG] 4.c scrub starts 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:22.638636+0000 osd.4 (osd.4) 146 : cluster [DBG] 4.c scrub ok 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:23.043575+0000 osd.1 (osd.1) 148 : cluster [DBG] 63.2 scrub starts 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:23.046567+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]': finished 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:23.046679+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41"}]': finished 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:23.046755+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:23.049390+0000 osd.1 (osd.1) 149 : cluster [DBG] 63.2 scrub ok 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:23.050263+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:23.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:23.050610+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:23.586 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:21.617143+0000 mgr.y (mgr.24437) 312 : cluster [DBG] pgmap v448: 246 pgs: 8 creating+peering, 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:22.393188+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:22.393400+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:22.393999+0000 mon.b (mon.1) 276 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41"}]: dispatch 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:22.394046+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41"}]: dispatch 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:22.474213+0000 osd.5 (osd.5) 143 : cluster [DBG] 5.f scrub starts 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:22.474953+0000 osd.5 (osd.5) 144 : cluster [DBG] 5.f scrub ok 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:22.636948+0000 osd.4 (osd.4) 145 : cluster [DBG] 4.c scrub starts 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:22.638636+0000 osd.4 (osd.4) 146 : cluster [DBG] 4.c scrub ok 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:23.043575+0000 osd.1 (osd.1) 148 : cluster [DBG] 63.2 scrub starts 2024-02-23T00:34:23.611 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:23.046567+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi087-82019-54"}]': finished 2024-02-23T00:34:23.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:23.046679+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-41"}]': finished 2024-02-23T00:34:23.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:23.046755+0000 mon.a (mon.0) 2126 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-02-23T00:34:23.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:23.049390+0000 osd.1 (osd.1) 149 : cluster [DBG] 63.2 scrub ok 2024-02-23T00:34:23.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:23.050263+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.87:0/2267502642' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:23.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:23.050610+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]: dispatch 2024-02-23T00:34:23.612 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:23.630 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:23.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:23.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:23.733 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:23.760 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:23.779 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:23.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:23.882 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:23.907 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:34:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:34:23.909 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:23.927 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:23.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:24.031 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:24.064 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:24.163 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:24.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:24.267 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:24.295 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:24.317 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:24.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:24.421 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:24.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.396+0000 7f9137fb0700 1 -- 172.21.15.87:0/2269386440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300f8420 msgr2=0x7f9130104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:24.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.396+0000 7f9137fb0700 1 --2- 172.21.15.87:0/2269386440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300f8420 0x7f9130104a20 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 rx=0x7f9128009be0 tx=0x7f9128004d10).stop 2024-02-23T00:34:24.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.396+0000 7f9137fb0700 1 -- 172.21.15.87:0/2269386440 shutdown_connections 2024-02-23T00:34:24.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.396+0000 7f9137fb0700 1 --2- 172.21.15.87:0/2269386440 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300f8420 0x7f9130104a20 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:24.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.396+0000 7f9137fb0700 1 --2- 172.21.15.87:0/2269386440 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91300fdbb0 0x7f91300fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:24.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.396+0000 7f9137fb0700 1 --2- 172.21.15.87:0/2269386440 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91300fe510 0x7f91300f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:24.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.396+0000 7f9137fb0700 1 -- 172.21.15.87:0/2269386440 >> 172.21.15.87:0/2269386440 conn(0x7f91300f3800 msgr2=0x7f91300f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:24.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.396+0000 7f9137fb0700 1 -- 172.21.15.87:0/2269386440 shutdown_connections 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.396+0000 7f9137fb0700 1 -- 172.21.15.87:0/2269386440 wait complete. 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 librados: starting msgr at 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 librados: starting objecter 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 Processor -- start 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 -- start start 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 librados: setting wanted keys 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 librados: calling monclient init 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91300f8420 0x7f913006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91300fdbb0 0x7f913006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300fe510 0x7f913006bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9130116330 con 0x7f91300fe510 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91301164a0 con 0x7f91300fdbb0 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9137fb0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9130116660 con 0x7f91300f8420 2024-02-23T00:34:24.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f913654d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300fe510 0x7f913006bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9135d4c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91300f8420 0x7f913006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f913654d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300fe510 0x7f913006bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59592/0 (socket says 172.21.15.87:59592) 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9135d4c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91300f8420 0x7f913006b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:47658/0 (socket says 172.21.15.87:47658) 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f913654d700 1 -- 172.21.15.87:0/3966745064 learned_addr learned my addr 172.21.15.87:0/3966745064 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f913554b700 1 --2- 172.21.15.87:0/3966745064 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91300fdbb0 0x7f913006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f913654d700 1 -- 172.21.15.87:0/3966745064 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91300f8420 msgr2=0x7f913006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f913654d700 1 --2- 172.21.15.87:0/3966745064 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91300f8420 0x7f913006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f913654d700 1 -- 172.21.15.87:0/3966745064 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91300fdbb0 msgr2=0x7f913006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f913654d700 1 --2- 172.21.15.87:0/3966745064 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91300fdbb0 0x7f913006b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f913654d700 1 -- 172.21.15.87:0/3966745064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c005040 con 0x7f91300fe510 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.397+0000 7f9135d4c700 1 --2- 172.21.15.87:0/3966745064 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91300f8420 0x7f913006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f913654d700 1 --2- 172.21.15.87:0/3966745064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300fe510 0x7f913006bab0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 rx=0x7f9128004a30 tx=0x7f9128004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f9126ffd700 10 client.?.objecter ms_handle_connect 0x7f91300fe510 2024-02-23T00:34:24.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f9126ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f9126ffd700 1 -- 172.21.15.87:0/3966745064 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912800e040 con 0x7f91300fe510 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91280097c0 con 0x7f91300fe510 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f9126ffd700 1 -- 172.21.15.87:0/3966745064 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9128018040 con 0x7f91300fe510 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f9126ffd700 1 -- 172.21.15.87:0/3966745064 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912801c3f0 con 0x7f91300fe510 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f9137fb0700 10 client.26988.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9130116bd0 con 0x7f91300fe510 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.398+0000 7f9137fb0700 1 librados: init done 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.399+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9130107110 con 0x7f91300fe510 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.399+0000 7f9126ffd700 1 -- 172.21.15.87:0/3966745064 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f9128007b20 con 0x7f91300fe510 2024-02-23T00:34:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.399+0000 7f9126ffd700 1 --2- 172.21.15.87:0/3966745064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f911c062c90 0x7f911c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.400+0000 7f9126ffd700 1 -- 172.21.15.87:0/3966745064 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(313..313 src has 173..313) v4 ==== 7515+0+0 (secure 0 0 0) 0x7f912807dfd0 con 0x7f91300fe510 2024-02-23T00:34:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.400+0000 7f9126ffd700 10 client.26988.objecter ms_dispatch 0x7f9130001550 osd_map(313..313 src has 173..313) v4 2024-02-23T00:34:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.400+0000 7f9126ffd700 3 client.26988.objecter handle_osd_map got epochs [313,313] > 0 2024-02-23T00:34:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.400+0000 7f9126ffd700 3 client.26988.objecter handle_osd_map decoding full epoch 313 2024-02-23T00:34:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.400+0000 7f9135d4c700 1 --2- 172.21.15.87:0/3966745064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f911c062c90 0x7f911c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.400+0000 7f9126ffd700 20 client.26988.objecter dump_active .. 0 homeless 2024-02-23T00:34:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.400+0000 7f9135d4c700 1 --2- 172.21.15.87:0/3966745064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f911c062c90 0x7f911c065150 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f91200103c0 tx=0x7f912000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.400+0000 7f9126ffd700 10 client.26988.objecter ms_handle_connect 0x7f911c062c90 2024-02-23T00:34:24.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.404+0000 7f9126ffd700 1 -- 172.21.15.87:0/3966745064 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f91280517b0 con 0x7f91300fe510 2024-02-23T00:34:24.450 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:24.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:24.583+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f913006c5b0 con 0x7f91300fe510 2024-02-23T00:34:24.626 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:24.799 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:24.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:24.903 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:24.930 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:25.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.050+0000 7f9126ffd700 1 -- 172.21.15.87:0/3966745064 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v314) v1 ==== 101+0+0 (secure 0 0 0) 0x7f9128004810 con 0x7f91300fe510 2024-02-23T00:34:25.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.056+0000 7f9137fb0700 10 librados: watch_flush enter 2024-02-23T00:34:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.061+0000 7f9137fb0700 10 librados: watch_flush exit 2024-02-23T00:34:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.061+0000 7f9137fb0700 20 client.26988.objecter shutdown clearing up homeless session... 2024-02-23T00:34:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.061+0000 7f9137fb0700 10 client.26988.objecter successfully canceled tick 2024-02-23T00:34:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.061+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f911c062c90 msgr2=0x7f911c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.061+0000 7f9137fb0700 1 --2- 172.21.15.87:0/3966745064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f911c062c90 0x7f911c065150 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f91200103c0 tx=0x7f912000b040).stop 2024-02-23T00:34:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.061+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300fe510 msgr2=0x7f913006bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:25.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.061+0000 7f9137fb0700 1 --2- 172.21.15.87:0/3966745064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300fe510 0x7f913006bab0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 rx=0x7f9128004a30 tx=0x7f9128004d10).stop 2024-02-23T00:34:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.062+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 shutdown_connections 2024-02-23T00:34:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.062+0000 7f9137fb0700 1 --2- 172.21.15.87:0/3966745064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f911c062c90 0x7f911c065150 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.062+0000 7f9137fb0700 1 --2- 172.21.15.87:0/3966745064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91300fe510 0x7f913006bab0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.062+0000 7f9137fb0700 1 --2- 172.21.15.87:0/3966745064 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91300f8420 0x7f913006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.062+0000 7f9137fb0700 1 --2- 172.21.15.87:0/3966745064 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91300fdbb0 0x7f913006b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.062+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 >> 172.21.15.87:0/3966745064 conn(0x7f91300f3800 msgr2=0x7f91300f5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.062+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 shutdown_connections 2024-02-23T00:34:25.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.063+0000 7f9137fb0700 1 -- 172.21.15.87:0/3966745064 wait complete. 2024-02-23T00:34:25.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:25.063+0000 7f9137fb0700 1 librados: shutdown 2024-02-23T00:34:25.064 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:34:25.129 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:25.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:25.143 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-23T00:34:25.233 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:25.259 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:25.278 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:25.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:23.596880+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:23.597863+0000 osd.4 (osd.4) 148 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:23.617806+0000 mgr.y (mgr.24437) 313 : cluster [DBG] pgmap v451: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:24.046083+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]': finished 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:24.046176+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:24.070657+0000 mon.b (mon.1) 278 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:24.300270+0000 mon.a (mon.0) 2130 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:24.301252+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:24.301403+0000 mon.a (mon.0) 2131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:24.302030+0000 mon.b (mon.1) 280 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi087-82019-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:24.302181+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi087-82019-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:24.583781+0000 mon.a (mon.0) 2133 : audit [INF] from='client.? 172.21.15.87:0/3966745064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:24.614058+0000 osd.3 (osd.3) 143 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:24.614955+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:23.596880+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:23.597863+0000 osd.4 (osd.4) 148 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:23.617806+0000 mgr.y (mgr.24437) 313 : cluster [DBG] pgmap v451: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:24.046083+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]': finished 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:24.046176+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:24.070657+0000 mon.b (mon.1) 278 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:24.300270+0000 mon.a (mon.0) 2130 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:24.301252+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:24.301403+0000 mon.a (mon.0) 2131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:24.302030+0000 mon.b (mon.1) 280 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi087-82019-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:25.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:24.302181+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi087-82019-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:25.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:24.583781+0000 mon.a (mon.0) 2133 : audit [INF] from='client.? 172.21.15.87:0/3966745064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:25.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:24.614058+0000 osd.3 (osd.3) 143 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:34:25.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:24.614955+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:34:25.381 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:25.408 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:23.596880+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:34:25.408 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:23.597863+0000 osd.4 (osd.4) 148 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:34:25.408 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:23.617806+0000 mgr.y (mgr.24437) 313 : cluster [DBG] pgmap v451: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:25.408 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:24.046083+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi087-82019-54"}]': finished 2024-02-23T00:34:25.408 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:24.046176+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-02-23T00:34:25.408 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:24.070657+0000 mon.b (mon.1) 278 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.408 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:24.300270+0000 mon.a (mon.0) 2130 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.409 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:24.301252+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.409 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:24.301403+0000 mon.a (mon.0) 2131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:25.409 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:24.302030+0000 mon.b (mon.1) 280 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi087-82019-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:25.409 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:24.302181+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi087-82019-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:25.409 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:24.583781+0000 mon.a (mon.0) 2133 : audit [INF] from='client.? 172.21.15.87:0/3966745064' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:25.409 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:24.614058+0000 osd.3 (osd.3) 143 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:34:25.409 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:24.614955+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:34:25.410 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:25.414 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:25.429 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:25.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:25.534 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:25.561 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:26.018 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:26.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:26.026 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:26.121 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:26.153 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:24.666593+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:34:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:24.667509+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:34:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:24.672837+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:34:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:24.675725+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:24.994224+0000 osd.1 (osd.1) 150 : cluster [DBG] 63.1d scrub starts 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:24.997690+0000 osd.1 (osd.1) 151 : cluster [DBG] 63.1d scrub ok 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:25.043546+0000 mon.a (mon.0) 2134 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:25.050929+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi087-82019-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:25.051047+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.87:0/3966745064' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:25.051099+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:25.057935+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi087-82019-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:25.060867+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi087-82019-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:25.061715+0000 mon.b (mon.1) 282 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:25.062392+0000 mon.a (mon.0) 2139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:25.592324+0000 osd.0 (osd.0) 161 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:34:26.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:25.593057+0000 osd.0 (osd.0) 162 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:34:26.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:24.666593+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:34:26.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:24.667509+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:34:26.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:24.672837+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:34:26.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:24.675725+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:34:26.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:24.994224+0000 osd.1 (osd.1) 150 : cluster [DBG] 63.1d scrub starts 2024-02-23T00:34:26.290 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:24.997690+0000 osd.1 (osd.1) 151 : cluster [DBG] 63.1d scrub ok 2024-02-23T00:34:26.290 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:25.043546+0000 mon.a (mon.0) 2134 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:34:26.290 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:25.050929+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi087-82019-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:26.290 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:25.051047+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.87:0/3966745064' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:34:26.290 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:25.051099+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-02-23T00:34:26.290 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:25.057935+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi087-82019-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:26.291 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:25.060867+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi087-82019-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:26.291 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:25.061715+0000 mon.b (mon.1) 282 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:26.291 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:25.062392+0000 mon.a (mon.0) 2139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:26.291 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:25.592324+0000 osd.0 (osd.0) 161 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:34:26.291 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:25.593057+0000 osd.0 (osd.0) 162 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:34:26.292 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:24.666593+0000 osd.2 (osd.2) 125 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:34:26.292 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:24.667509+0000 osd.2 (osd.2) 126 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:34:26.292 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:24.672837+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:34:26.292 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:24.675725+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:34:26.292 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:24.994224+0000 osd.1 (osd.1) 150 : cluster [DBG] 63.1d scrub starts 2024-02-23T00:34:26.292 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:24.997690+0000 osd.1 (osd.1) 151 : cluster [DBG] 63.1d scrub ok 2024-02-23T00:34:26.293 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:25.043546+0000 mon.a (mon.0) 2134 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:34:26.293 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:25.050929+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi087-82019-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:26.293 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:25.051047+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.87:0/3966745064' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:34:26.293 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:25.051099+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-02-23T00:34:26.293 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:25.057935+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi087-82019-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:26.293 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:25.060867+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi087-82019-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:26.294 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:25.061715+0000 mon.b (mon.1) 282 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:26.294 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:25.062392+0000 mon.a (mon.0) 2139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:26.294 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:25.592324+0000 osd.0 (osd.0) 161 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:34:26.294 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:25.593057+0000 osd.0 (osd.0) 162 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:34:26.305 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:26.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:26.413 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:26.444 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:26.568 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:26.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:26.673 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:26.701 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:25.618876+0000 mgr.y (mgr.24437) 314 : cluster [DBG] pgmap v454: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:25.678301+0000 osd.2 (osd.2) 127 : cluster [DBG] 5.a deep-scrub starts 2024-02-23T00:34:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:25.679071+0000 osd.2 (osd.2) 128 : cluster [DBG] 5.a deep-scrub ok 2024-02-23T00:34:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:26.306903+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:26.307008+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-02-23T00:34:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:26.309262+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:26.310938+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:25.618876+0000 mgr.y (mgr.24437) 314 : cluster [DBG] pgmap v454: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:25.678301+0000 osd.2 (osd.2) 127 : cluster [DBG] 5.a deep-scrub starts 2024-02-23T00:34:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:25.679071+0000 osd.2 (osd.2) 128 : cluster [DBG] 5.a deep-scrub ok 2024-02-23T00:34:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:26.306903+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:26.307008+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-02-23T00:34:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:26.309262+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:26.310938+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:27.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:25.618876+0000 mgr.y (mgr.24437) 314 : cluster [DBG] pgmap v454: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:27.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:25.678301+0000 osd.2 (osd.2) 127 : cluster [DBG] 5.a deep-scrub starts 2024-02-23T00:34:27.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:25.679071+0000 osd.2 (osd.2) 128 : cluster [DBG] 5.a deep-scrub ok 2024-02-23T00:34:27.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:26.306903+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:27.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:26.307008+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-02-23T00:34:27.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:26.309262+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:27.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:26.310938+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:27.451 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:27.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:27.464 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:34:27.464 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:34:27.555 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:27.585 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:27.683 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:27.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:27.787 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:27.814 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:28.013 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:28.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:28.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:34:28.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:27.064448+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi087-82019-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi087-82019-55"}]': finished 2024-02-23T00:34:28.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:27.064561+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:28.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:27.064637+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-02-23T00:34:28.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:27.065628+0000 mon.b (mon.1) 284 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-23T00:34:28.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:27.066665+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-23T00:34:28.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:27.292920+0000 osd.6 (osd.6) 117 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:34:28.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:27.295974+0000 osd.6 (osd.6) 118 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:34:28.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:27.538091+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:34:28.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:27.539982+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:34:28.090 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-23T00:34:28.118 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:28.159 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:28.179 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:28.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:28.282 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:28.311 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:28.330 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:28.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:27.064448+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi087-82019-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi087-82019-55"}]': finished 2024-02-23T00:34:28.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:27.064561+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:28.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:27.064637+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-02-23T00:34:28.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:27.065628+0000 mon.b (mon.1) 284 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-23T00:34:28.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:27.066665+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-23T00:34:28.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:27.292920+0000 osd.6 (osd.6) 117 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:34:28.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:27.295974+0000 osd.6 (osd.6) 118 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:34:28.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:27.538091+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:34:28.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:27.539982+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:34:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:27.064448+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi087-82019-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi087-82019-55"}]': finished 2024-02-23T00:34:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:27.064561+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:27.064637+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-02-23T00:34:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:27.065628+0000 mon.b (mon.1) 284 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-23T00:34:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:27.066665+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-23T00:34:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:27.292920+0000 osd.6 (osd.6) 117 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:34:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:27.295974+0000 osd.6 (osd.6) 118 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:34:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:27.538091+0000 osd.0 (osd.0) 163 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:34:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:27.539982+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:34:28.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:28.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:28.363 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:28.435 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:28.464 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:28.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.571+0000 7f158ae29700 1 -- 172.21.15.87:0/2719226394 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 msgr2=0x7f15840fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.571+0000 7f158ae29700 1 --2- 172.21.15.87:0/2719226394 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 0x7f15840fdf90 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f15780047b0 tx=0x7f1578009cb0).stop 2024-02-23T00:34:28.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.571+0000 7f158ae29700 1 -- 172.21.15.87:0/2719226394 shutdown_connections 2024-02-23T00:34:28.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.571+0000 7f158ae29700 1 --2- 172.21.15.87:0/2719226394 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15840f8420 0x7f1584104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.571+0000 7f158ae29700 1 --2- 172.21.15.87:0/2719226394 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 0x7f15840fdf90 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.571+0000 7f158ae29700 1 --2- 172.21.15.87:0/2719226394 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15840fe510 0x7f15840f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.571+0000 7f158ae29700 1 -- 172.21.15.87:0/2719226394 >> 172.21.15.87:0/2719226394 conn(0x7f15840f3800 msgr2=0x7f15840f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:28.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.571+0000 7f158ae29700 1 -- 172.21.15.87:0/2719226394 shutdown_connections 2024-02-23T00:34:28.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.571+0000 7f158ae29700 1 -- 172.21.15.87:0/2719226394 wait complete. 2024-02-23T00:34:28.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 librados: starting msgr at 2024-02-23T00:34:28.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 librados: starting objecter 2024-02-23T00:34:28.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 Processor -- start 2024-02-23T00:34:28.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 -- start start 2024-02-23T00:34:28.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 librados: setting wanted keys 2024-02-23T00:34:28.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 librados: calling monclient init 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15840f8420 0x7f1584067f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 0x7f1584068470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15840fe510 0x7f15840631d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1584063750 con 0x7f15840fe510 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15840638c0 con 0x7f15840f8420 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f158ae29700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1584063a80 con 0x7f15840fdbb0 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f1583fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 0x7f1584068470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f1583fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 0x7f1584068470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:47672/0 (socket says 172.21.15.87:47672) 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f1583fff700 1 -- 172.21.15.87:0/532468191 learned_addr learned my addr 172.21.15.87:0/532468191 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.572+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15840f8420 0x7f1584067f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1583fff700 1 -- 172.21.15.87:0/532468191 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15840f8420 msgr2=0x7f1584067f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1583fff700 1 --2- 172.21.15.87:0/532468191 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15840f8420 0x7f1584067f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1583fff700 1 -- 172.21.15.87:0/532468191 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15840fe510 msgr2=0x7f15840631d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1583fff700 1 --2- 172.21.15.87:0/532468191 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f15840fe510 0x7f15840631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1583fff700 1 -- 172.21.15.87:0/532468191 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c0097c0 con 0x7f15840fdbb0 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f15840f8420 0x7f1584067f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1583fff700 1 --2- 172.21.15.87:0/532468191 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 0x7f1584068470 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 rx=0x7f157000e970 tx=0x7f157000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1581ffb700 10 client.?.objecter ms_handle_connect 0x7f15840fdbb0 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1581ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157000ee60 con 0x7f15840fdbb0 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1578005040 con 0x7f15840fdbb0 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f157000f040 con 0x7f15840fdbb0 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1570007ea0 con 0x7f15840fdbb0 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f158ae29700 10 client.37448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158406bd60 con 0x7f15840fdbb0 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.573+0000 7f158ae29700 1 librados: init done 2024-02-23T00:34:28.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.574+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1584064b00 con 0x7f15840fdbb0 2024-02-23T00:34:28.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.574+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f157000cb10 con 0x7f15840fdbb0 2024-02-23T00:34:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1581ffb700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1574062c90 0x7f1574065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(317..317 src has 173..317) v4 ==== 8720+0+0 (secure 0 0 0) 0x7f157007d0f0 con 0x7f15840fdbb0 2024-02-23T00:34:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 osd_map(317..317 src has 173..317) v4 2024-02-23T00:34:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1581ffb700 3 client.37448.objecter handle_osd_map got epochs [317,317] > 0 2024-02-23T00:34:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1581ffb700 3 client.37448.objecter handle_osd_map decoding full epoch 317 2024-02-23T00:34:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1574062c90 0x7f1574065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1581ffb700 20 client.37448.objecter dump_active .. 0 homeless 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v317) v1 ==== 54+0+16 (secure 0 0 0) 0x7f1570084070 con 0x7f15840fdbb0 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 10 client.37448.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 1 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6801/2153191863 conn(0x7f15840fee80 0x7f15840ff260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158405ee80 osd=0 3 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158405ee80 osd=0 3 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158405ee80 osd=0 3 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 1 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158405ee80 osd=0 3 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 0 1 2024-02-23T00:34:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158405ee80 osd=0 4 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 10 client.37448.objecter _send_command 1 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1584107110 con 0x7f15840fee80 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1581ffb700 10 client.37448.objecter ms_handle_connect 0x7f1574062c90 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1574062c90 0x7f1574065150 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 rx=0x7f1578005010 tx=0x7f1578007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.575+0000 7f15893c6700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6801/2153191863 conn(0x7f15840fee80 0x7f15840ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.576+0000 7f15893c6700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6801/2153191863 conn(0x7f15840fee80 0x7f15840ff260 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.576+0000 7f1581ffb700 10 client.37448.objecter ms_handle_connect 0x7f15840fee80 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.576+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1584107110 con 0x7f15840fee80 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.576+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.576+0000 7f1581ffb700 10 client.37448.objecter _finish_command 1 = system:0 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.576+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158405ee80 osd=0 4 2024-02-23T00:34:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.576+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 0 1 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 10 client.37448.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 2 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158405ee80 osd=0 3 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158405ee80 osd=0 3 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158405ee80 osd=0 3 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 2 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158405ee80 osd=0 3 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 0 2 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158405ee80 osd=0 4 2024-02-23T00:34:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 10 client.37448.objecter _send_command 2 2024-02-23T00:34:28.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.596+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f15840ff750 con 0x7f15840fee80 2024-02-23T00:34:28.598 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:34:28.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.597+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f15840ff750 con 0x7f15840fee80 2024-02-23T00:34:28.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.597+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:28.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.597+0000 7f1581ffb700 10 client.37448.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.597+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158405ee80 osd=0 4 2024-02-23T00:34:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.597+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 0 2 2024-02-23T00:34:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.597+0000 7f158ae29700 10 client.37448.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.597+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 3 2024-02-23T00:34:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.597+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6805/685354171 conn(0x7f15840598f0 0x7f1584059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f15840ff920 osd=1 3 2024-02-23T00:34:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f15840ff920 osd=1 3 2024-02-23T00:34:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f15840ff920 osd=1 3 2024-02-23T00:34:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 3 2024-02-23T00:34:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f15840ff920 osd=1 3 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 1 3 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f15840ff920 osd=1 4 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f158ae29700 10 client.37448.objecter _send_command 3 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f158412a1d0 con 0x7f15840598f0 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6805/685354171 conn(0x7f15840598f0 0x7f1584059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f1581ffb700 10 client.37448.objecter ms_handle_connect 0x7f15840598f0 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6805/685354171 conn(0x7f15840598f0 0x7f1584059cd0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f158412a1d0 con 0x7f15840598f0 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f1581ffb700 10 client.37448.objecter _finish_command 3 = system:0 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f15840ff920 osd=1 4 2024-02-23T00:34:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.598+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 1 3 2024-02-23T00:34:28.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 10 client.37448.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 4 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f15840ff920 osd=1 3 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f15840ff920 osd=1 3 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f15840ff920 osd=1 3 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 4 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f15840ff920 osd=1 3 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 1 4 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f15840ff920 osd=1 4 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 10 client.37448.objecter _send_command 4 2024-02-23T00:34:28.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.613+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1584059420 con 0x7f15840598f0 2024-02-23T00:34:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1584059420 con 0x7f15840598f0 2024-02-23T00:34:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f1581ffb700 10 client.37448.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f15840ff920 osd=1 4 2024-02-23T00:34:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 1 4 2024-02-23T00:34:28.617 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 10 client.37448.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 5 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6809/3605394228 conn(0x7f15840273c0 0x7f15840277e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158406c750 osd=2 3 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c750 osd=2 3 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158406c750 osd=2 3 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 5 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158406c750 osd=2 3 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 2 5 2024-02-23T00:34:28.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c750 osd=2 4 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 10 client.37448.objecter _send_command 5 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f158406cf90 con 0x7f15840273c0 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.615+0000 7f1583fff700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6809/3605394228 conn(0x7f15840273c0 0x7f15840277e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.616+0000 7f1581ffb700 10 client.37448.objecter ms_handle_connect 0x7f15840273c0 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.616+0000 7f1583fff700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6809/3605394228 conn(0x7f15840273c0 0x7f15840277e0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.616+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f158406cf90 con 0x7f15840273c0 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.616+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.616+0000 7f1581ffb700 10 client.37448.objecter _finish_command 5 = system:0 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.616+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158406c750 osd=2 4 2024-02-23T00:34:28.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.616+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 2 5 2024-02-23T00:34:28.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.630+0000 7f158ae29700 10 client.37448.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:28.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 6 2024-02-23T00:34:28.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158406c750 osd=2 3 2024-02-23T00:34:28.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c750 osd=2 3 2024-02-23T00:34:28.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158406c750 osd=2 3 2024-02-23T00:34:28.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 6 2024-02-23T00:34:28.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158406c750 osd=2 3 2024-02-23T00:34:28.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 2 6 2024-02-23T00:34:28.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c750 osd=2 4 2024-02-23T00:34:28.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 10 client.37448.objecter _send_command 6 2024-02-23T00:34:28.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.631+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1584028550 con 0x7f15840273c0 2024-02-23T00:34:28.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1584028550 con 0x7f15840273c0 2024-02-23T00:34:28.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:28.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f1581ffb700 10 client.37448.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158406c750 osd=2 4 2024-02-23T00:34:28.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 2 6 2024-02-23T00:34:28.634 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:34:28.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 10 client.37448.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:28.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 7 2024-02-23T00:34:28.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6813/128059790 conn(0x7f158406c330 0x7f158412e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158406c1c0 osd=3 3 2024-02-23T00:34:28.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c1c0 osd=3 3 2024-02-23T00:34:28.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158406c1c0 osd=3 3 2024-02-23T00:34:28.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 7 2024-02-23T00:34:28.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158406c1c0 osd=3 3 2024-02-23T00:34:28.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 3 7 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c1c0 osd=3 4 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 10 client.37448.objecter _send_command 7 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1584131660 con 0x7f158406c330 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.632+0000 7f15893c6700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6813/128059790 conn(0x7f158406c330 0x7f158412e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.633+0000 7f15893c6700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6813/128059790 conn(0x7f158406c330 0x7f158412e890 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.633+0000 7f1581ffb700 10 client.37448.objecter ms_handle_connect 0x7f158406c330 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.633+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1584131660 con 0x7f158406c330 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.633+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.633+0000 7f1581ffb700 10 client.37448.objecter _finish_command 7 = system:0 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.633+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158406c1c0 osd=3 4 2024-02-23T00:34:28.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.633+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 3 7 2024-02-23T00:34:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 10 client.37448.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 8 2024-02-23T00:34:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158406c1c0 osd=3 3 2024-02-23T00:34:28.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c1c0 osd=3 3 2024-02-23T00:34:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158406c1c0 osd=3 3 2024-02-23T00:34:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 8 2024-02-23T00:34:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158406c1c0 osd=3 3 2024-02-23T00:34:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 3 8 2024-02-23T00:34:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c1c0 osd=3 4 2024-02-23T00:34:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 10 client.37448.objecter _send_command 8 2024-02-23T00:34:28.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.647+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1584027fc0 con 0x7f158406c330 2024-02-23T00:34:28.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1584027fc0 con 0x7f158406c330 2024-02-23T00:34:28.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:28.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f1581ffb700 10 client.37448.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158406c1c0 osd=3 4 2024-02-23T00:34:28.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 3 8 2024-02-23T00:34:28.650 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:34:28.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 10 client.37448.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:28.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 9 2024-02-23T00:34:28.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1584129d30 0x7f158412ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f1584027fc0 osd=4 3 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584027fc0 osd=4 3 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f1584027fc0 osd=4 3 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 9 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f1584027fc0 osd=4 3 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 4 9 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584027fc0 osd=4 4 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 10 client.37448.objecter _send_command 9 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f158412f330 con 0x7f1584129d30 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.649+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1584129d30 0x7f158412ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.650+0000 7f1581ffb700 10 client.37448.objecter ms_handle_connect 0x7f1584129d30 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.650+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1584129d30 0x7f158412ee40 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.650+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f158412f330 con 0x7f1584129d30 2024-02-23T00:34:28.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.650+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:34:28.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.650+0000 7f1581ffb700 10 client.37448.objecter _finish_command 9 = system:0 2024-02-23T00:34:28.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.650+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f1584027fc0 osd=4 4 2024-02-23T00:34:28.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.650+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 4 9 2024-02-23T00:34:28.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 10 client.37448.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:28.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 10 2024-02-23T00:34:28.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f1584027fc0 osd=4 3 2024-02-23T00:34:28.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584027fc0 osd=4 3 2024-02-23T00:34:28.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f1584027fc0 osd=4 3 2024-02-23T00:34:28.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 10 2024-02-23T00:34:28.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f1584027fc0 osd=4 3 2024-02-23T00:34:28.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 4 10 2024-02-23T00:34:28.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584027fc0 osd=4 4 2024-02-23T00:34:28.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 10 client.37448.objecter _send_command 10 2024-02-23T00:34:28.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.665+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f158412f330 con 0x7f1584129d30 2024-02-23T00:34:28.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f158412f330 con 0x7f1584129d30 2024-02-23T00:34:28.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:28.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f1581ffb700 10 client.37448.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f1584027fc0 osd=4 4 2024-02-23T00:34:28.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 4 10 2024-02-23T00:34:28.669 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 10 client.37448.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 11 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1584130c60 0x7f1584131080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158412f330 osd=5 3 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158412f330 osd=5 3 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158412f330 osd=5 3 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 11 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158412f330 osd=5 3 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 5 11 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158412f330 osd=5 4 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 10 client.37448.objecter _send_command 11 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.667+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1584138110 con 0x7f1584130c60 2024-02-23T00:34:28.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.668+0000 7f1583fff700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1584130c60 0x7f1584131080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.668+0000 7f1581ffb700 10 client.37448.objecter ms_handle_connect 0x7f1584130c60 2024-02-23T00:34:28.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.668+0000 7f1583fff700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1584130c60 0x7f1584131080 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.669+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1584138110 con 0x7f1584130c60 2024-02-23T00:34:28.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.669+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:34:28.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.669+0000 7f1581ffb700 10 client.37448.objecter _finish_command 11 = system:0 2024-02-23T00:34:28.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.669+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158412f330 osd=5 4 2024-02-23T00:34:28.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.669+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 5 11 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 10 client.37448.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 12 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158412f330 osd=5 3 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158412f330 osd=5 3 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158412f330 osd=5 3 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 12 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158412f330 osd=5 3 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 5 12 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158412f330 osd=5 4 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 10 client.37448.objecter _send_command 12 2024-02-23T00:34:28.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.683+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1584138110 con 0x7f1584130c60 2024-02-23T00:34:28.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.684+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1584138110 con 0x7f1584130c60 2024-02-23T00:34:28.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.684+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:28.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.684+0000 7f1581ffb700 10 client.37448.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.684+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158412f330 osd=5 4 2024-02-23T00:34:28.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.684+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 5 12 2024-02-23T00:34:28.686 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 10 client.37448.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 13 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1584135e50 0x7f15841362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f1584138110 osd=6 3 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584138110 osd=6 3 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f1584138110 osd=6 3 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 13 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f1584138110 osd=6 3 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 6 13 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584138110 osd=6 4 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 10 client.37448.objecter _send_command 13 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1584137f20 con 0x7f1584135e50 2024-02-23T00:34:28.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f15893c6700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1584135e50 0x7f15841362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f15893c6700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1584135e50 0x7f15841362d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.685+0000 7f1581ffb700 10 client.37448.objecter ms_handle_connect 0x7f1584135e50 2024-02-23T00:34:28.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.686+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f157c0202e0 con 0x7f1584135e50 2024-02-23T00:34:28.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.686+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:34:28.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.686+0000 7f1581ffb700 10 client.37448.objecter _finish_command 13 = system:0 2024-02-23T00:34:28.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.686+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f1584138110 osd=6 4 2024-02-23T00:34:28.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.686+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 6 13 2024-02-23T00:34:28.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 10 client.37448.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:28.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 14 2024-02-23T00:34:28.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f1584138110 osd=6 3 2024-02-23T00:34:28.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584138110 osd=6 3 2024-02-23T00:34:28.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f1584138110 osd=6 3 2024-02-23T00:34:28.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 14 2024-02-23T00:34:28.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f1584138110 osd=6 3 2024-02-23T00:34:28.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 6 14 2024-02-23T00:34:28.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584138110 osd=6 4 2024-02-23T00:34:28.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 10 client.37448.objecter _send_command 14 2024-02-23T00:34:28.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.701+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f158413a340 con 0x7f1584135e50 2024-02-23T00:34:28.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f158413a340 con 0x7f1584135e50 2024-02-23T00:34:28.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:28.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f1581ffb700 10 client.37448.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f1584138110 osd=6 4 2024-02-23T00:34:28.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 6 14 2024-02-23T00:34:28.704 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:34:28.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 10 client.37448.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:28.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 15 2024-02-23T00:34:28.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6812/373787000 conn(0x7f158413a9f0 0x7f158413ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:28.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158413a340 osd=7 3 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158413a340 osd=7 3 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158413a340 osd=7 3 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 15 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158413a340 osd=7 3 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 7 15 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158413a340 osd=7 4 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 10 client.37448.objecter _send_command 15 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f158413e700 con 0x7f158413a9f0 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.703+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6812/373787000 conn(0x7f158413a9f0 0x7f158413ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.704+0000 7f1581ffb700 10 client.37448.objecter ms_handle_connect 0x7f158413a9f0 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.704+0000 7f1588bc5700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6812/373787000 conn(0x7f158413a9f0 0x7f158413ae50 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.704+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f158413e700 con 0x7f158413a9f0 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.704+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:34:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.704+0000 7f1581ffb700 10 client.37448.objecter _finish_command 15 = system:0 2024-02-23T00:34:28.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.704+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158413a340 osd=7 4 2024-02-23T00:34:28.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.704+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 7 15 2024-02-23T00:34:28.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 10 client.37448.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign -1 16 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158413a340 osd=7 3 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158413a340 osd=7 3 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 20 client.37448.objecter _get_session s=0x7f158413a340 osd=7 3 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 15 client.37448.objecter _session_command_op_remove -1 16 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 20 client.37448.objecter get_session s=0x7f158413a340 osd=7 3 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 15 client.37448.objecter _session_command_op_assign 7 16 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158413a340 osd=7 4 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 10 client.37448.objecter _send_command 16 2024-02-23T00:34:28.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.719+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f158413d580 con 0x7f158413a9f0 2024-02-23T00:34:28.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f1581ffb700 1 -- 172.21.15.87:0/532468191 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f158413d580 con 0x7f158413a9f0 2024-02-23T00:34:28.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f1581ffb700 10 client.37448.objecter ms_dispatch 0x7f1584001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:28.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f1581ffb700 10 client.37448.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f1581ffb700 20 client.37448.objecter put_session s=0x7f158413a340 osd=7 4 2024-02-23T00:34:28.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f1581ffb700 15 client.37448.objecter _session_command_op_remove 7 16 2024-02-23T00:34:28.723 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 10 librados: watch_flush enter 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 10 librados: watch_flush exit 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 10 client.37448.objecter close_session for osd.0 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6801/2153191863 conn(0x7f15840fee80 msgr2=0x7f15840ff260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6801/2153191863 conn(0x7f15840fee80 0x7f15840ff260 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158405ee80 osd=0 1 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 10 client.37448.objecter close_session for osd.1 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6805/685354171 conn(0x7f15840598f0 msgr2=0x7f1584059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6805/685354171 conn(0x7f15840598f0 0x7f1584059cd0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f15840ff920 osd=1 1 2024-02-23T00:34:28.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 10 client.37448.objecter close_session for osd.2 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6809/3605394228 conn(0x7f15840273c0 msgr2=0x7f15840277e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6809/3605394228 conn(0x7f15840273c0 0x7f15840277e0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c750 osd=2 1 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 10 client.37448.objecter close_session for osd.3 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6813/128059790 conn(0x7f158406c330 msgr2=0x7f158412e890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f15893c6700 1 -- 172.21.15.87:0/532468191 reap_dead start 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6813/128059790 conn(0x7f158406c330 0x7f158412e890 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158406c1c0 osd=3 1 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 10 client.37448.objecter close_session for osd.4 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1584129d30 msgr2=0x7f158412ee40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.721+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6800/4175844757 conn(0x7f1584129d30 0x7f158412ee40 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584027fc0 osd=4 1 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 10 client.37448.objecter close_session for osd.5 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1584130c60 msgr2=0x7f1584131080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1584130c60 0x7f1584131080 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158412f330 osd=5 1 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 10 client.37448.objecter close_session for osd.6 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1584135e50 msgr2=0x7f15841362d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1584135e50 0x7f15841362d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f1584138110 osd=6 1 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 10 client.37448.objecter close_session for osd.7 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6812/373787000 conn(0x7f158413a9f0 msgr2=0x7f158413ae50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.171:6812/373787000 conn(0x7f158413a9f0 0x7f158413ae50 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 20 client.37448.objecter put_session s=0x7f158413a340 osd=7 1 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 20 client.37448.objecter shutdown clearing up homeless session... 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 10 client.37448.objecter successfully canceled tick 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f15893c6700 1 -- 172.21.15.87:0/532468191 reap_dead start 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1574062c90 msgr2=0x7f1574065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1574062c90 0x7f1574065150 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 rx=0x7f1578005010 tx=0x7f1578007f90).stop 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 msgr2=0x7f1584068470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 0x7f1584068470 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 rx=0x7f157000e970 tx=0x7f157000b040).stop 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 shutdown_connections 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1574062c90 0x7f1574065150 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 --2- 172.21.15.87:0/532468191 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f15840fdbb0 0x7f1584068470 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 >> 172.21.15.87:0/532468191 conn(0x7f15840f3800 msgr2=0x7f15840f52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 shutdown_connections 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 -- 172.21.15.87:0/532468191 wait complete. 2024-02-23T00:34:28.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:28.722+0000 7f158ae29700 1 librados: shutdown 2024-02-23T00:34:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:34:28.729 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:34:29.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:27.619533+0000 mgr.y (mgr.24437) 315 : cluster [DBG] pgmap v457: 246 pgs: 8 unknown, 32 creating+peering, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:29.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:27.672014+0000 osd.7 (osd.7) 153 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:34:29.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:27.673477+0000 osd.7 (osd.7) 154 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:34:29.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:28.071382+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_count","val": "3"}]': finished 2024-02-23T00:34:29.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:28.071522+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-02-23T00:34:29.077 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:28.079378+0000 mon.b (mon.1) 285 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-23T00:34:29.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:28.079645+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-23T00:34:29.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:28.496411+0000 osd.0 (osd.0) 165 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:34:29.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:28.498608+0000 osd.0 (osd.0) 166 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:34:29.078 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:28.575248+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.87:0/532468191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:29.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:27.619533+0000 mgr.y (mgr.24437) 315 : cluster [DBG] pgmap v457: 246 pgs: 8 unknown, 32 creating+peering, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:29.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:27.672014+0000 osd.7 (osd.7) 153 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:34:29.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:27.673477+0000 osd.7 (osd.7) 154 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:34:29.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:28.071382+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_count","val": "3"}]': finished 2024-02-23T00:34:29.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:28.071522+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-02-23T00:34:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:28.079378+0000 mon.b (mon.1) 285 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-23T00:34:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:28.079645+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-23T00:34:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:28.496411+0000 osd.0 (osd.0) 165 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:34:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:28.498608+0000 osd.0 (osd.0) 166 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:34:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:28.575248+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.87:0/532468191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:29.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:27.619533+0000 mgr.y (mgr.24437) 315 : cluster [DBG] pgmap v457: 246 pgs: 8 unknown, 32 creating+peering, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:29.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:27.672014+0000 osd.7 (osd.7) 153 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:34:29.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:27.673477+0000 osd.7 (osd.7) 154 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:34:29.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:28.071382+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_count","val": "3"}]': finished 2024-02-23T00:34:29.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:28.071522+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-02-23T00:34:29.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:28.079378+0000 mon.b (mon.1) 285 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-23T00:34:29.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:28.079645+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-23T00:34:29.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:28.496411+0000 osd.0 (osd.0) 165 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:34:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:28.498608+0000 osd.0 (osd.0) 166 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:34:29.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:28.575248+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.87:0/532468191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:29.514 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:29.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:29.619 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:29.646 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:29.720 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:29.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:29.824 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:29.853 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:30.050 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:30.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.962+0000 7f041d46a700 1 -- 172.21.15.87:0/4020525921 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04180f9690 msgr2=0x7f0418101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:30.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.962+0000 7f041d46a700 1 --2- 172.21.15.87:0/4020525921 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04180f9690 0x7f0418101c70 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7f040c0047b0 tx=0x7f040c009cb0).stop 2024-02-23T00:34:30.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.962+0000 7f041d46a700 1 -- 172.21.15.87:0/4020525921 shutdown_connections 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.962+0000 7f041d46a700 1 --2- 172.21.15.87:0/4020525921 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04181021f0 0x7f04181045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.962+0000 7f041d46a700 1 --2- 172.21.15.87:0/4020525921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04180f8dc0 0x7f04180f91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.962+0000 7f041d46a700 1 --2- 172.21.15.87:0/4020525921 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04180f9690 0x7f0418101c70 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.962+0000 7f041d46a700 1 -- 172.21.15.87:0/4020525921 >> 172.21.15.87:0/4020525921 conn(0x7f04180f26f0 msgr2=0x7f04180f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.962+0000 7f041d46a700 1 -- 172.21.15.87:0/4020525921 shutdown_connections 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 -- 172.21.15.87:0/4020525921 wait complete. 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 librados: starting msgr at 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 librados: starting objecter 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 Processor -- start 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 -- start start 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 librados: setting wanted keys 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 librados: calling monclient init 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04180f8dc0 0x7f04180f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:30.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04180f9690 0x7f04180f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04181021f0 0x7f04180f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0418116680 con 0x7f04180f8dc0 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04181167f0 con 0x7f04180f9690 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.963+0000 7f041d46a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04181169b0 con 0x7f04181021f0 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f0416ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04180f8dc0 0x7f04180f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f04177fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04181021f0 0x7f04180f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f0416ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04180f8dc0 0x7f04180f6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59622/0 (socket says 172.21.15.87:59622) 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f0416ffd700 1 -- 172.21.15.87:0/2901060395 learned_addr learned my addr 172.21.15.87:0/2901060395 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f04167fc700 1 --2- 172.21.15.87:0/2901060395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04180f9690 0x7f04180f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f04177fe700 1 -- 172.21.15.87:0/2901060395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04180f9690 msgr2=0x7f04180f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f04177fe700 1 --2- 172.21.15.87:0/2901060395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04180f9690 0x7f04180f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f04177fe700 1 -- 172.21.15.87:0/2901060395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04180f8dc0 msgr2=0x7f04180f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f04177fe700 1 --2- 172.21.15.87:0/2901060395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04180f8dc0 0x7f04180f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:30.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f04177fe700 1 -- 172.21.15.87:0/2901060395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0408005040 con 0x7f04181021f0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f0416ffd700 1 --2- 172.21.15.87:0/2901060395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04180f8dc0 0x7f04180f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f04177fe700 1 --2- 172.21.15.87:0/2901060395 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04181021f0 0x7f04180f7590 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7f041000eb70 tx=0x7f041000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f03fffff700 10 client.?.objecter ms_handle_connect 0x7f04181021f0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f03fffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f03fffff700 1 -- 172.21.15.87:0/2901060395 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410009870 con 0x7f04181021f0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f03fffff700 1 -- 172.21.15.87:0/2901060395 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f041000f040 con 0x7f04181021f0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f03fffff700 1 -- 172.21.15.87:0/2901060395 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410007da0 con 0x7f04181021f0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c005040 con 0x7f04181021f0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f041d46a700 10 client.37454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0418116f40 con 0x7f04181021f0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.964+0000 7f041d46a700 1 librados: init done 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.965+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041805ee80 con 0x7f04181021f0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.965+0000 7f03fffff700 1 -- 172.21.15.87:0/2901060395 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f041000c7c0 con 0x7f04181021f0 2024-02-23T00:34:30.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.966+0000 7f03fffff700 1 --2- 172.21.15.87:0/2901060395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0400062c90 0x7f0400065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.966+0000 7f03fffff700 1 -- 172.21.15.87:0/2901060395 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(318..318 src has 173..318) v4 ==== 8305+0+0 (secure 0 0 0) 0x7f041007d060 con 0x7f04181021f0 2024-02-23T00:34:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.966+0000 7f03fffff700 10 client.37454.objecter ms_dispatch 0x7f0418001550 osd_map(318..318 src has 173..318) v4 2024-02-23T00:34:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.966+0000 7f03fffff700 3 client.37454.objecter handle_osd_map got epochs [318,318] > 0 2024-02-23T00:34:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.966+0000 7f03fffff700 3 client.37454.objecter handle_osd_map decoding full epoch 318 2024-02-23T00:34:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.966+0000 7f0416ffd700 1 --2- 172.21.15.87:0/2901060395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0400062c90 0x7f0400065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.966+0000 7f03fffff700 20 client.37454.objecter dump_active .. 0 homeless 2024-02-23T00:34:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.966+0000 7f03fffff700 10 client.37454.objecter ms_handle_connect 0x7f0400062c90 2024-02-23T00:34:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.966+0000 7f0416ffd700 1 --2- 172.21.15.87:0/2901060395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0400062c90 0x7f0400065150 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 rx=0x7f0408004d40 tx=0x7f040800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:29.970+0000 7f03fffff700 1 -- 172.21.15.87:0/2901060395 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0410049f60 con 0x7f04181021f0 2024-02-23T00:34:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:30.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:30.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:30.139+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"} v 0) v1 -- 0x7f04180f80a0 con 0x7f04181021f0 2024-02-23T00:34:30.158 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:30.186 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:30.300 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:34:30.300 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:34:30.300 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:34:30.300 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:34:30.300 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:34:30.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:28.585790+0000 osd.4 (osd.4) 149 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:34:30.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:28.586552+0000 osd.4 (osd.4) 150 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:34:30.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:28.709311+0000 osd.7 (osd.7) 155 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:34:30.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:28.710169+0000 osd.7 (osd.7) 156 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:34:30.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:28.745075+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.b scrub starts 2024-02-23T00:34:30.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:28.745857+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.b scrub ok 2024-02-23T00:34:30.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:29.071994+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_period","val": "3"}]': finished 2024-02-23T00:34:30.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:29.072101+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-02-23T00:34:30.323 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:29.073585+0000 mon.b (mon.1) 286 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:30.323 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:29.074170+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:30.323 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:29.078991+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:30.323 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:29.079208+0000 mon.a (mon.0) 2153 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:30.323 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:29.230453+0000 osd.6 (osd.6) 119 : cluster [DBG] 3.a scrub starts 2024-02-23T00:34:30.323 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:29.233566+0000 osd.6 (osd.6) 120 : cluster [DBG] 3.a scrub ok 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:28.585790+0000 osd.4 (osd.4) 149 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:28.586552+0000 osd.4 (osd.4) 150 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:28.709311+0000 osd.7 (osd.7) 155 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:28.710169+0000 osd.7 (osd.7) 156 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:28.745075+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.b scrub starts 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:28.745857+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.b scrub ok 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:29.071994+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_period","val": "3"}]': finished 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:29.072101+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:29.073585+0000 mon.b (mon.1) 286 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:29.074170+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:29.078991+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:29.079208+0000 mon.a (mon.0) 2153 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:30.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:29.230453+0000 osd.6 (osd.6) 119 : cluster [DBG] 3.a scrub starts 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:29.233566+0000 osd.6 (osd.6) 120 : cluster [DBG] 3.a scrub ok 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:28.585790+0000 osd.4 (osd.4) 149 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:28.586552+0000 osd.4 (osd.4) 150 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:28.709311+0000 osd.7 (osd.7) 155 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:28.710169+0000 osd.7 (osd.7) 156 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:28.745075+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.b scrub starts 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:28.745857+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.b scrub ok 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:29.071994+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_period","val": "3"}]': finished 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:29.072101+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:29.073585+0000 mon.b (mon.1) 286 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:29.074170+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:30.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:29.078991+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:30.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:29.079208+0000 mon.a (mon.0) 2153 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:30.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:29.230453+0000 osd.6 (osd.6) 119 : cluster [DBG] 3.a scrub starts 2024-02-23T00:34:30.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:29.233566+0000 osd.6 (osd.6) 120 : cluster [DBG] 3.a scrub ok 2024-02-23T00:34:30.497 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:30.554 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:30.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:30.657 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:30.685 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:30.783 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:30.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:30.888 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:30.916 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:30.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:34:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:34:30.968 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-23T00:34:31.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.099+0000 7f03fffff700 1 -- 172.21.15.87:0/2901060395 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]=0 set pool 4 pg_num to 22 v320) v1 ==== 140+0+0 (secure 0 0 0) 0x7f04100505c0 con 0x7f04181021f0 2024-02-23T00:34:31.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.102+0000 7f041d46a700 10 librados: watch_flush enter 2024-02-23T00:34:31.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.102+0000 7f041d46a700 10 librados: watch_flush exit 2024-02-23T00:34:31.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.102+0000 7f041d46a700 20 client.37454.objecter shutdown clearing up homeless session... 2024-02-23T00:34:31.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.102+0000 7f041d46a700 10 client.37454.objecter successfully canceled tick 2024-02-23T00:34:31.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.102+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0400062c90 msgr2=0x7f0400065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:31.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.102+0000 7f041d46a700 1 --2- 172.21.15.87:0/2901060395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0400062c90 0x7f0400065150 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 rx=0x7f0408004d40 tx=0x7f040800b040).stop 2024-02-23T00:34:31.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.102+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04181021f0 msgr2=0x7f04180f7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:31.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.102+0000 7f041d46a700 1 --2- 172.21.15.87:0/2901060395 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04181021f0 0x7f04180f7590 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7f041000eb70 tx=0x7f041000b040).stop 2024-02-23T00:34:31.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.103+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 shutdown_connections 2024-02-23T00:34:31.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.103+0000 7f041d46a700 1 --2- 172.21.15.87:0/2901060395 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0400062c90 0x7f0400065150 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:31.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.103+0000 7f041d46a700 1 --2- 172.21.15.87:0/2901060395 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f04181021f0 0x7f04180f7590 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:31.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.103+0000 7f041d46a700 1 --2- 172.21.15.87:0/2901060395 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f04180f8dc0 0x7f04180f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:31.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.103+0000 7f041d46a700 1 --2- 172.21.15.87:0/2901060395 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f04180f9690 0x7f04180f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:31.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.103+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 >> 172.21.15.87:0/2901060395 conn(0x7f04180f26f0 msgr2=0x7f04180f3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:31.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.107+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 shutdown_connections 2024-02-23T00:34:31.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.107+0000 7f041d46a700 1 -- 172.21.15.87:0/2901060395 wait complete. 2024-02-23T00:34:31.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:31.108+0000 7f041d46a700 1 librados: shutdown 2024-02-23T00:34:31.109 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 pg_num to 22 2024-02-23T00:34:31.241 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:29.619910+0000 mgr.y (mgr.24437) 316 : cluster [DBG] pgmap v460: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:29.673931+0000 osd.7 (osd.7) 157 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:34:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:29.677033+0000 osd.7 (osd.7) 158 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:34:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:30.089099+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:34:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:30.089181+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]': finished 2024-02-23T00:34:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:30.089239+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-02-23T00:34:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:30.091060+0000 mon.b (mon.1) 288 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:30.091212+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:31.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:30.091233+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:30.091691+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:30.140355+0000 mon.c (mon.2) 376 : audit [INF] from='client.? 172.21.15.87:0/2901060395' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:30.140633+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:29.619910+0000 mgr.y (mgr.24437) 316 : cluster [DBG] pgmap v460: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:29.673931+0000 osd.7 (osd.7) 157 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:29.677033+0000 osd.7 (osd.7) 158 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:30.089099+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:30.089181+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]': finished 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:30.089239+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-02-23T00:34:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:30.091060+0000 mon.b (mon.1) 288 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:31.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:30.091212+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:31.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:30.091233+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-23T00:34:31.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:30.091691+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-23T00:34:31.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:30.140355+0000 mon.c (mon.2) 376 : audit [INF] from='client.? 172.21.15.87:0/2901060395' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:34:31.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:30.140633+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:34:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:29.619910+0000 mgr.y (mgr.24437) 316 : cluster [DBG] pgmap v460: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:29.673931+0000 osd.7 (osd.7) 157 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:34:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:29.677033+0000 osd.7 (osd.7) 158 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:34:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:30.089099+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:34:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:30.089181+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi087-82019-55"}]': finished 2024-02-23T00:34:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:30.089239+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-02-23T00:34:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:30.091060+0000 mon.b (mon.1) 288 : audit [INF] from='client.? 172.21.15.87:0/376582802' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:30.091212+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]: dispatch 2024-02-23T00:34:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:30.091233+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-23T00:34:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:30.091691+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-23T00:34:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:30.140355+0000 mon.c (mon.2) 376 : audit [INF] from='client.? 172.21.15.87:0/2901060395' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:34:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:30.140633+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:34:32.065 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:32.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:32.168 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:32.199 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:32.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:32] "GET /metrics HTTP/1.1" 200 184636 "" "Prometheus/2.33.4" 2024-02-23T00:34:32.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:30.722884+0000 osd.7 (osd.7) 159 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:34:32.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:30.726018+0000 osd.7 (osd.7) 160 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:34:32.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:30.802186+0000 mgr.y (mgr.24437) 317 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:32.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:31.094499+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]': finished 2024-02-23T00:34:32.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:31.094557+0000 mon.a (mon.0) 2161 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:31.094614+0000 mon.a (mon.0) 2162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]': finished 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.094663+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:31.117893+0000 mon.a (mon.0) 2164 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:31.118608+0000 mon.a (mon.0) 2165 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:31.118920+0000 mon.a (mon.0) 2166 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi087-82019-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:31.621149+0000 mon.a (mon.0) 2167 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.671429+0000 osd.3 (osd.3) 145 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.674315+0000 osd.3 (osd.3) 146 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:32.046263+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi087-82019-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:32.046336+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:32.046399+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-02-23T00:34:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:32.049009+0000 mon.a (mon.0) 2171 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi087-82019-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:30.722884+0000 osd.7 (osd.7) 159 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:30.726018+0000 osd.7 (osd.7) 160 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:30.802186+0000 mgr.y (mgr.24437) 317 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:31.094499+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]': finished 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:31.094557+0000 mon.a (mon.0) 2161 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:31.094614+0000 mon.a (mon.0) 2162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]': finished 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.094663+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:31.117893+0000 mon.a (mon.0) 2164 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:31.118608+0000 mon.a (mon.0) 2165 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:31.118920+0000 mon.a (mon.0) 2166 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi087-82019-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:31.621149+0000 mon.a (mon.0) 2167 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-23T00:34:32.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.671429+0000 osd.3 (osd.3) 145 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:34:32.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.674315+0000 osd.3 (osd.3) 146 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:34:32.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:32.046263+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi087-82019-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:32.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:32.046336+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-23T00:34:32.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:32.046399+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-02-23T00:34:32.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:32.049009+0000 mon.a (mon.0) 2171 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi087-82019-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:30.722884+0000 osd.7 (osd.7) 159 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:34:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:30.726018+0000 osd.7 (osd.7) 160 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:34:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:30.802186+0000 mgr.y (mgr.24437) 317 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:31.094499+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi087-82019-55"}]': finished 2024-02-23T00:34:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:31.094557+0000 mon.a (mon.0) 2161 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-23T00:34:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:31.094614+0000 mon.a (mon.0) 2162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "22"}]': finished 2024-02-23T00:34:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.094663+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-02-23T00:34:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:31.117893+0000 mon.a (mon.0) 2164 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:32.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:31.118608+0000 mon.a (mon.0) 2165 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:32.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:31.118920+0000 mon.a (mon.0) 2166 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi087-82019-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:32.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:31.621149+0000 mon.a (mon.0) 2167 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-23T00:34:32.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.671429+0000 osd.3 (osd.3) 145 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:34:32.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.674315+0000 osd.3 (osd.3) 146 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:34:32.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:32.046263+0000 mon.a (mon.0) 2168 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi087-82019-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:32.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:32.046336+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-23T00:34:32.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:32.046399+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-02-23T00:34:32.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:32.049009+0000 mon.a (mon.0) 2171 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi087-82019-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:32.449 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:32.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:32.518 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:34:32.550 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:32.577 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:32.725 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:32.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:32.829 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:32.857 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:32.878 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:32.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:32.981 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:32.999 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25070.0:10 44.17 44:e8e7cce9:::oid_PoolQuotaPP_7:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+full_try e111)", 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.464439+0000", 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 209.53218701700001, 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022323852000000002, 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.000 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25070", 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3829298547", 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 10 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464439+0000", 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464439+0000", 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464436+0000", 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2904999999999998e-05 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464459+0000", 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8601e-05 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464478+0000", 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9809999999999998e-06 2024-02-23T00:34:33.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464488+0000", 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1741000000000002e-05 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464570+0000", 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013978800000000001 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464709+0000", 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015174700000000001 2024-02-23T00:34:33.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,3", 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464861+0000", 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000406 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.465267+0000", 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066204979999999998 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:33.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.471888+0000", 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014817204 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.486705+0000", 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.167e-05 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.486736+0000", 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6329000000000002e-05 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.486763+0000", 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022323852000000002 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:3 63.11 63:8d290297:test-rados-api-smithi087-82091-8::foo:head [writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:07.929685+0000", 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 205.06694068600001, 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110693868, 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:34:33.006 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 3 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929685+0000", 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929685+0000", 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929684+0000", 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5640000000000002e-06 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929690+0000", 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9799999999999998e-06 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929698+0000", 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9900000000000002e-06 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929704+0000", 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9026999999999998e-05 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929743+0000", 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0059e-05 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929833+0000", 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6689999999999998e-05 2024-02-23T00:34:33.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,5", 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929930+0000", 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038132900000000002 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.930311+0000", 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018368799999999999 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.930495+0000", 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.109798312 2024-02-23T00:34:33.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:08.040293+0000", 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7428000000000001e-05 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:08.040341+0000", 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8401e-05 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:08.040379+0000", 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110693868 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.011 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24815.0:1113 19.16 19:6cac518f:::0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e148)", 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:39.054396+0000", 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 173.942229879, 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043740108999999999, 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24815", 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3778460189", 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1113 2024-02-23T00:34:33.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054396+0000", 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054396+0000", 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054395+0000", 2024-02-23T00:34:33.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6200000000000001e-06 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054399+0000", 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7519999999999996e-06 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054407+0000", 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8739999999999997e-06 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054411+0000", 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4289999999999999e-05 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054445+0000", 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6217000000000002e-05 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054481+0000", 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042577932999999998 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097059+0000", 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7280000000000005e-05 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097136+0000", 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012174400000000001 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5,7", 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097258+0000", 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036222099999999997 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097620+0000", 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000203802 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097824+0000", 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017177300000000001 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097996+0000", 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6755000000000002e-05 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:33.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.098043+0000", 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3552000000000001e-05 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.098136+0000", 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043740108999999999 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24815.0:1117 19.1d 19:bd63b0f1:::8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e148)", 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:39.103414+0000", 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 173.89321191499999, 2024-02-23T00:34:33.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.25896939400000002, 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24815", 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3778460189", 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1117 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103414+0000", 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103414+0000", 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103413+0000", 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.467e-06 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103417+0000", 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5680000000000002e-06 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103424+0000", 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8120000000000001e-06 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103428+0000", 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.464e-05 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103463+0000", 2024-02-23T00:34:33.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5056999999999996e-05 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for missing object", 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103538+0000", 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00099724100000000006 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.104535+0000", 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3939000000000002e-05 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.104569+0000", 2024-02-23T00:34:33.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.25692525199999999 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.361494+0000", 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6981000000000002e-05 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.361551+0000", 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4629000000000006e-05 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.361636+0000", 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00035837500000000002 2024-02-23T00:34:33.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.361994+0000", 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00025748299999999997 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.362252+0000", 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.371e-05 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.362295+0000", 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9400999999999999e-05 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.362325+0000", 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8569000000000001e-05 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.362383+0000", 2024-02-23T00:34:33.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.25896939400000002 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.7.0:300 63.18 e182/114 63:1a553bd4:test-rados-api-smithi087-82091-14::foo26:7 v 182'34, mlcod=148'30)", 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:13.000057+0000", 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 139.996569024, 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086960830000000003, 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000057+0000", 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000057+0000", 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000057+0000", 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5390000000000001e-06 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000061+0000", 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5890000000000002e-06 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000067+0000", 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3040000000000001e-06 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000073+0000", 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109073 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000182+0000", 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9967999999999999e-05 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000202+0000", 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085273399999999996 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.008729+0000", 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3555999999999999e-05 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.008753+0000", 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086960830000000003 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.7.0:302 63.18 e182/114 63:1e5f7139:test-rados-api-smithi087-82091-14::foo83:7 v 182'36, mlcod=148'30)", 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:13.000084+0000", 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 139.996542318, 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086929699999999995, 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000084+0000", 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000084+0000", 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000083+0000", 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3470000000000004e-06 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000090+0000", 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1109999999999999e-06 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000095+0000", 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.946e-06 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000099+0000", 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0032143689999999999 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.032 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.003313+0000", 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4056000000000001e-05 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.003337+0000", 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054193050000000001 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.008756+0000", 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0129999999999999e-05 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.033 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.008777+0000", 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086929699999999995 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:92 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032803+0000", 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.963823252, 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0081447849999999999, 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.034 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 92 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032803+0000", 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032803+0000", 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032802+0000", 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7379999999999999e-06 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032805+0000", 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9339999999999997e-06 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032810+0000", 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9269999999999998e-06 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032814+0000", 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0080427559999999999 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.040857+0000", 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7802000000000002e-05 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.040894+0000", 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3090000000000002e-05 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.040947+0000", 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0081447849999999999 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:93 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032818+0000", 2024-02-23T00:34:33.038 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.963807559, 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082399150000000004, 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 93 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032818+0000", 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.039 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032818+0000", 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032818+0000", 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.796e-06 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032821+0000", 2024-02-23T00:34:33.040 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.361e-06 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032826+0000", 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.968e-06 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032830+0000", 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0081440739999999994 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.041 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.040974+0000", 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8188000000000001e-05 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041012+0000", 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5927999999999998e-05 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041058+0000", 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082399150000000004 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.042 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:94 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032834+0000", 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.96379161199999, 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0083633389999999991, 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 94 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.043 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032834+0000", 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032834+0000", 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032834+0000", 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7180000000000001e-06 2024-02-23T00:34:33.044 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032837+0000", 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9289999999999997e-06 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032842+0000", 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.782e-06 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032845+0000", 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082468070000000001 2024-02-23T00:34:33.045 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041092+0000", 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7123000000000002e-05 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041149+0000", 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8368000000000002e-05 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041198+0000", 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0083633389999999991 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.046 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:95 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032849+0000", 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.963776524, 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084569580000000005, 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 95 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.047 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032849+0000", 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032849+0000", 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032849+0000", 2024-02-23T00:34:33.048 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.558e-06 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032855+0000", 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.215e-06 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032860+0000", 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9570000000000002e-06 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.049 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032868+0000", 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0083560749999999993 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041224+0000", 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7444999999999998e-05 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041261+0000", 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5086999999999997e-05 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.050 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041306+0000", 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084569580000000005 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:96 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032873+0000", 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.963752782, 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085372469999999995, 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.051 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 96 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032873+0000", 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032873+0000", 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032873+0000", 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2430000000000001e-06 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032876+0000", 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0769999999999998e-06 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032881+0000", 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1320000000000004e-06 2024-02-23T00:34:33.053 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032885+0000", 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084456170000000007 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041331+0000", 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5104999999999997e-05 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041366+0000", 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4536000000000002e-05 2024-02-23T00:34:33.054 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041410+0000", 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085372469999999995 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:97 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032912+0000", 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.96371357300001, 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086066719999999992, 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.055 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 97 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032912+0000", 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.056 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032912+0000", 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032912+0000", 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8670000000000004e-06 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032915+0000", 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.601e-06 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.057 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032922+0000", 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0520000000000001e-06 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032927+0000", 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085083369999999995 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041435+0000", 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8047999999999999e-05 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.058 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041473+0000", 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5596999999999997e-05 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041519+0000", 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086066719999999992 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:98 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032933+0000", 2024-02-23T00:34:33.059 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.96369330100001, 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087421370000000005, 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 98 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032933+0000", 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.060 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032933+0000", 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032932+0000", 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.147e-05 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032944+0000", 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9630000000000003e-06 2024-02-23T00:34:33.061 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032949+0000", 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.262e-06 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032954+0000", 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086076090000000004 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041561+0000", 2024-02-23T00:34:33.062 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8508999999999999e-05 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041610+0000", 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4911000000000002e-05 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041675+0000", 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087421370000000005 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.063 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:99 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032959+0000", 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.96366723600001, 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088395780000000007, 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 99 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.064 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032959+0000", 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032959+0000", 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032958+0000", 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3379e-05 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.065 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032972+0000", 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.443e-06 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032978+0000", 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3379999999999998e-06 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032982+0000", 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087279739999999995 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.066 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041710+0000", 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7415e-05 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041748+0000", 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0489999999999999e-05 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041798+0000", 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088395780000000007 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.067 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:100 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032998+0000", 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.96362748600001, 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0089354540000000007, 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 100 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.068 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032998+0000", 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032998+0000", 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032998+0000", 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.563e-06 2024-02-23T00:34:33.069 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033001+0000", 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1620000000000002e-06 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033009+0000", 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6940000000000003e-06 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033013+0000", 2024-02-23T00:34:33.070 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088147120000000006 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041828+0000", 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4904000000000002e-05 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041883+0000", 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0952000000000003e-05 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041934+0000", 2024-02-23T00:34:33.071 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0089354540000000007 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:101 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.033028+0000", 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.963597839, 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090205070000000005, 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.072 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 101 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033028+0000", 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033028+0000", 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.073 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033027+0000", 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2650000000000001e-06 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033031+0000", 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7240000000000004e-06 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033040+0000", 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.425e-06 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.074 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033045+0000", 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0089160109999999997 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041961+0000", 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7178000000000002e-05 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041998+0000", 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0677999999999998e-05 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.075 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.042049+0000", 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090205070000000005 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:102 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.033051+0000", 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 108.963575299, 2024-02-23T00:34:33.076 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0091115729999999995, 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 102 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033051+0000", 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.077 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033051+0000", 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033050+0000", 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.089e-06 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033053+0000", 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5600000000000001e-06 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.078 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033059+0000", 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3359999999999997e-06 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033063+0000", 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.009004115 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.042067+0000", 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4490000000000003e-05 2024-02-23T00:34:33.079 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.042112+0000", 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0618000000000002e-05 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.042162+0000", 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0091115729999999995 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.080 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:2082 160.3 160:c0773ebc:test-rados-api-smithi087-82091-26::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e238)", 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:09.020460+0000", 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 83.976165945999995, 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018878523000000001, 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2082 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.081 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020460+0000", 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020460+0000", 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020458+0000", 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7559999999999994e-06 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020468+0000", 2024-02-23T00:34:33.082 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3088000000000001e-05 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020481+0000", 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7870000000000002e-06 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020489+0000", 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8291000000000002e-05 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020537+0000", 2024-02-23T00:34:33.083 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00027761399999999997 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020815+0000", 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014981089 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.035796+0000", 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5717999999999994e-05 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:34:33.084 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.035891+0000", 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000521332 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.036413+0000", 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5217999999999997e-05 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.036488+0000", 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028505689999999998 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.085 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.039339+0000", 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018878523000000001 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37277.0:2 173.1s0 173:9294e480:nspace::foo:head [write 0~64] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:33.036033+0000", 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 59.960593295999999, 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13658257500000001, 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37277", 2024-02-23T00:34:33.086 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2105660432", 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036033+0000", 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036033+0000", 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.087 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036030+0000", 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7399999999999993e-06 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036039+0000", 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3021e-05 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036052+0000", 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0880000000000001e-06 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.088 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036061+0000", 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6910000000000003e-05 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036108+0000", 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3079999999999999e-06 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036112+0000", 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.135140337 2024-02-23T00:34:33.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.171252+0000", 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011767799999999999 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.171370+0000", 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041097099999999999 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.171780+0000", 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041141700000000002 2024-02-23T00:34:33.090 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.172192+0000", 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00037785 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.172570+0000", 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5453e-05 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.172615+0000", 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13658257500000001 2024-02-23T00:34:33.091 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26952.0:1 189.6s0 189:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e302)", 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:13.039879+0000", 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 19.956746841000001, 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0195118649999999, 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26952", 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3617910981", 2024-02-23T00:34:33.092 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039879+0000", 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039879+0000", 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:33.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039877+0000", 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8540000000000003e-06 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039886+0000", 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1123e-05 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039897+0000", 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0100000000000001e-06 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039906+0000", 2024-02-23T00:34:33.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9648999999999997e-05 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039976+0000", 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0049999999999998e-06 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039980+0000", 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.017329613 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.057309+0000", 2024-02-23T00:34:33.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000148116 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.057457+0000", 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00072641600000000002 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.058184+0000", 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077851899999999998 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:33.096 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.058962+0000", 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033970199999999999 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.059302+0000", 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8896999999999999e-05 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.059391+0000", 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0195118649999999 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:33.097 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:33.098 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:34:33.098 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 -- 172.21.15.87:0/2793251918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 msgr2=0x7fd1e4104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 --2- 172.21.15.87:0/2793251918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 0x7fd1e4104a20 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 rx=0x7fd1dc009c30 tx=0x7fd1dc004c50).stop 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 -- 172.21.15.87:0/2793251918 shutdown_connections 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 --2- 172.21.15.87:0/2793251918 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 0x7fd1e4104a20 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 --2- 172.21.15.87:0/2793251918 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1e40fdbb0 0x7fd1e40fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 --2- 172.21.15.87:0/2793251918 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1e40fe510 0x7fd1e40f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 -- 172.21.15.87:0/2793251918 >> 172.21.15.87:0/2793251918 conn(0x7fd1e40f3800 msgr2=0x7fd1e40f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 -- 172.21.15.87:0/2793251918 shutdown_connections 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 -- 172.21.15.87:0/2793251918 wait complete. 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 librados: starting msgr at 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.041+0000 7fd1e8860700 1 librados: starting objecter 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 Processor -- start 2024-02-23T00:34:33.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 -- start start 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 librados: setting wanted keys 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 librados: calling monclient init 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 0x7fd1e410df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1e40fdbb0 0x7fd1e410e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1e40fe510 0x7fd1e41080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e41085d0 con 0x7fd1e40f8420 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e4108740 con 0x7fd1e40fdbb0 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e8860700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e4108900 con 0x7fd1e40fe510 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 0x7fd1e410df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e259c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 0x7fd1e410df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48400/0 (socket says 172.21.15.87:48400) 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e259c700 1 -- 172.21.15.87:0/344941687 learned_addr learned my addr 172.21.15.87:0/344941687 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e2d9d700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1e40fe510 0x7fd1e41080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e1d9b700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1e40fdbb0 0x7fd1e410e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:33.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e259c700 1 -- 172.21.15.87:0/344941687 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1e40fe510 msgr2=0x7fd1e41080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e259c700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1e40fe510 0x7fd1e41080e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e259c700 1 -- 172.21.15.87:0/344941687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1e40fdbb0 msgr2=0x7fd1e410e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e259c700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1e40fdbb0 0x7fd1e410e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e259c700 1 -- 172.21.15.87:0/344941687 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8005040 con 0x7fd1e40f8420 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.042+0000 7fd1e2d9d700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1e40fe510 0x7fd1e41080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1e259c700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 0x7fd1e410df40 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 rx=0x7fd1d400e970 tx=0x7fd1d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1d37fe700 10 client.?.objecter ms_handle_connect 0x7fd1e40f8420 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1d37fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1d37fe700 1 -- 172.21.15.87:0/344941687 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4004660 con 0x7fd1e40f8420 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1dc0097c0 con 0x7fd1e40f8420 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1d37fe700 1 -- 172.21.15.87:0/344941687 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1d400f040 con 0x7fd1e40f8420 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1e8860700 10 client.27009.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1d37fe700 1 -- 172.21.15.87:0/344941687 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4014790 con 0x7fd1e40f8420 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e4114d70 con 0x7fd1e40f8420 2024-02-23T00:34:33.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.043+0000 7fd1e8860700 1 librados: init done 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.044+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c4004f50 con 0x7fd1e40f8420 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.045+0000 7fd1d37fe700 1 -- 172.21.15.87:0/344941687 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd1d4007b40 con 0x7fd1e40f8420 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.045+0000 7fd1d37fe700 1 --2- 172.21.15.87:0/344941687 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1cc062c90 0x7fd1cc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.045+0000 7fd1d37fe700 1 -- 172.21.15.87:0/344941687 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(321..321 src has 173..321) v4 ==== 8217+0+0 (secure 0 0 0) 0x7fd1d407cd70 con 0x7fd1e40f8420 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.045+0000 7fd1d37fe700 10 client.27009.objecter ms_dispatch 0x7fd1e4001550 osd_map(321..321 src has 173..321) v4 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.045+0000 7fd1d37fe700 3 client.27009.objecter handle_osd_map got epochs [321,321] > 0 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.045+0000 7fd1d37fe700 3 client.27009.objecter handle_osd_map decoding full epoch 321 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.045+0000 7fd1d37fe700 20 client.27009.objecter dump_active .. 0 homeless 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.046+0000 7fd1e1d9b700 1 --2- 172.21.15.87:0/344941687 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1cc062c90 0x7fd1cc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.046+0000 7fd1e1d9b700 1 --2- 172.21.15.87:0/344941687 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1cc062c90 0x7fd1cc065150 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 rx=0x7fd1e4109880 tx=0x7fd1d800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:33.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.047+0000 7fd1d37fe700 10 client.27009.objecter ms_handle_connect 0x7fd1cc062c90 2024-02-23T00:34:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:33.103 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:33.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:33.117 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:33.204 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:33.233 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:33.258 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:33.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:33.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.308+0000 7fd1d37fe700 1 -- 172.21.15.87:0/344941687 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd1d4050250 con 0x7fd1e40f8420 2024-02-23T00:34:33.364 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:33.388 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.620295+0000 mgr.y (mgr.24437) 318 : cluster [DBG] pgmap v463: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:33.388 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.620844+0000 osd.4 (osd.4) 151 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.622756+0000 osd.4 (osd.4) 152 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.731678+0000 osd.2 (osd.2) 131 : cluster [DBG] 63.6 scrub starts 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.733144+0000 osd.2 (osd.2) 132 : cluster [DBG] 63.6 scrub ok 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.753534+0000 osd.7 (osd.7) 161 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:31.754307+0000 osd.7 (osd.7) 162 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.620295+0000 mgr.y (mgr.24437) 318 : cluster [DBG] pgmap v463: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.620844+0000 osd.4 (osd.4) 151 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.622756+0000 osd.4 (osd.4) 152 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.731678+0000 osd.2 (osd.2) 131 : cluster [DBG] 63.6 scrub starts 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.733144+0000 osd.2 (osd.2) 132 : cluster [DBG] 63.6 scrub ok 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.753534+0000 osd.7 (osd.7) 161 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:34:33.389 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:31.754307+0000 osd.7 (osd.7) 162 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:34:33.391 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:33.416 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:33.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:33.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:33.513+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fd1c4005780 con 0x7fd1e40f8420 2024-02-23T00:34:33.521 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:33.549 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:33.629 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.620295+0000 mgr.y (mgr.24437) 318 : cluster [DBG] pgmap v463: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:33.629 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.620844+0000 osd.4 (osd.4) 151 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:34:33.629 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.622756+0000 osd.4 (osd.4) 152 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:34:33.629 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.731678+0000 osd.2 (osd.2) 131 : cluster [DBG] 63.6 scrub starts 2024-02-23T00:34:33.629 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.733144+0000 osd.2 (osd.2) 132 : cluster [DBG] 63.6 scrub ok 2024-02-23T00:34:33.629 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.753534+0000 osd.7 (osd.7) 161 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:34:33.629 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:31.754307+0000 osd.7 (osd.7) 162 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:34:33.645 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-23T00:34:33.647 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:33.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:33.751 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:33.777 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:33.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:34:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:34:33.943 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:33.975 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:33.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:34.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.052+0000 7fd1d37fe700 1 -- 172.21.15.87:0/344941687 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v323) v1 ==== 87+0+0 (secure 0 0 0) 0x7fd1d4019030 con 0x7fd1e40f8420 2024-02-23T00:34:34.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 10 librados: watch_flush enter 2024-02-23T00:34:34.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 10 librados: watch_flush exit 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 20 client.27009.objecter shutdown clearing up homeless session... 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 10 client.27009.objecter successfully canceled tick 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1cc062c90 msgr2=0x7fd1cc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 --2- 172.21.15.87:0/344941687 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1cc062c90 0x7fd1cc065150 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 rx=0x7fd1e4109880 tx=0x7fd1d800b040).stop 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 msgr2=0x7fd1e410df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 0x7fd1e410df40 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 rx=0x7fd1d400e970 tx=0x7fd1d400b040).stop 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 shutdown_connections 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 --2- 172.21.15.87:0/344941687 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1cc062c90 0x7fd1cc065150 secure :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 rx=0x7fd1e4109880 tx=0x7fd1d800b040).stop 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1e40fe510 0x7fd1e41080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1e40f8420 0x7fd1e410df40 secure :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 rx=0x7fd1d400e970 tx=0x7fd1d400b040).stop 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 --2- 172.21.15.87:0/344941687 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1e40fdbb0 0x7fd1e410e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:34.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.058+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 >> 172.21.15.87:0/344941687 conn(0x7fd1e40f3800 msgr2=0x7fd1e40f5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:34.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.062+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 shutdown_connections 2024-02-23T00:34:34.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.062+0000 7fd1e8860700 1 -- 172.21.15.87:0/344941687 wait complete. 2024-02-23T00:34:34.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:34.062+0000 7fd1e8860700 1 librados: shutdown 2024-02-23T00:34:34.063 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:34:34.080 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:34.109 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:34.458 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:32.519217+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:34:34.458 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:32.520022+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:34:34.458 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:32.719250+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:34:34.458 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:32.720045+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:34:34.458 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:33.303410+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-02-23T00:34:34.458 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:33.513943+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? 172.21.15.87:0/344941687' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:34:34.458 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:34.045359+0000 mon.a (mon.0) 2174 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:34.458 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:34.050711+0000 mon.a (mon.0) 2175 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi087-82019-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi087-82019-56"}]': finished 2024-02-23T00:34:34.458 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:34.050778+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? 172.21.15.87:0/344941687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:34:34.459 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:34.050834+0000 mon.a (mon.0) 2177 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-02-23T00:34:34.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:32.519217+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:34:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:32.520022+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:34:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:32.719250+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:34:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:32.720045+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:34:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:33.303410+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-02-23T00:34:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:33.513943+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? 172.21.15.87:0/344941687' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:34:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:34.045359+0000 mon.a (mon.0) 2174 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:34.050711+0000 mon.a (mon.0) 2175 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi087-82019-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi087-82019-56"}]': finished 2024-02-23T00:34:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:34.050778+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? 172.21.15.87:0/344941687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:34:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:34.050834+0000 mon.a (mon.0) 2177 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:32.519217+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:32.520022+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:32.719250+0000 osd.7 (osd.7) 163 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:32.720045+0000 osd.7 (osd.7) 164 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:33.303410+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:33.513943+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? 172.21.15.87:0/344941687' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:34.045359+0000 mon.a (mon.0) 2174 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:34.050711+0000 mon.a (mon.0) 2175 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi087-82019-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi087-82019-56"}]': finished 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:34.050778+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? 172.21.15.87:0/344941687' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:34:34.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:34.050834+0000 mon.a (mon.0) 2177 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-02-23T00:34:35.061 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:35.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:35.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:35.195 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:35.292 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:35.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:35.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.295+0000 7f73f94c1700 1 -- 172.21.15.87:0/3536040391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 msgr2=0x7f73f40fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.295+0000 7f73f94c1700 1 --2- 172.21.15.87:0/3536040391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 0x7f73f40fcf20 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 rx=0x7f73e4004870 tx=0x7f73e4009cb0).stop 2024-02-23T00:34:35.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.295+0000 7f73f94c1700 1 -- 172.21.15.87:0/3536040391 shutdown_connections 2024-02-23T00:34:35.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.295+0000 7f73f94c1700 1 --2- 172.21.15.87:0/3536040391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73f40f7ba0 0x7f73f4103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.295+0000 7f73f94c1700 1 --2- 172.21.15.87:0/3536040391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 0x7f73f40fcf20 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.295+0000 7f73f94c1700 1 --2- 172.21.15.87:0/3536040391 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73f40fd4a0 0x7f73f40f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.295+0000 7f73f94c1700 1 -- 172.21.15.87:0/3536040391 >> 172.21.15.87:0/3536040391 conn(0x7f73f406c370 msgr2=0x7f73f406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:35.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.295+0000 7f73f94c1700 1 -- 172.21.15.87:0/3536040391 shutdown_connections 2024-02-23T00:34:35.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.295+0000 7f73f94c1700 1 -- 172.21.15.87:0/3536040391 wait complete. 2024-02-23T00:34:35.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.296+0000 7f73f94c1700 1 librados: starting msgr at 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.296+0000 7f73f94c1700 1 librados: starting objecter 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.296+0000 7f73f94c1700 1 Processor -- start 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.296+0000 7f73f94c1700 1 -- start start 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.296+0000 7f73f94c1700 1 librados: setting wanted keys 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.296+0000 7f73f94c1700 1 librados: calling monclient init 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73f40f7ba0 0x7f73f4109260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 0x7f73f4109750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73f40fd4a0 0x7f73f4109d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f4114490 con 0x7f73f40fcb40 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f4114600 con 0x7f73f40fd4a0 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f41147c0 con 0x7f73f40f7ba0 2024-02-23T00:34:35.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f27fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 0x7f73f4109750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f27fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 0x7f73f4109750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48432/0 (socket says 172.21.15.87:48432) 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f27fc700 1 -- 172.21.15.87:0/1817890264 learned_addr learned my addr 172.21.15.87:0/1817890264 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73f40f7ba0 0x7f73f4109260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f37fe700 1 --2- 172.21.15.87:0/1817890264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73f40fd4a0 0x7f73f4109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f27fc700 1 -- 172.21.15.87:0/1817890264 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73f40f7ba0 msgr2=0x7f73f4109260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f27fc700 1 --2- 172.21.15.87:0/1817890264 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73f40f7ba0 0x7f73f4109260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f27fc700 1 -- 172.21.15.87:0/1817890264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73f40fd4a0 msgr2=0x7f73f4109d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f27fc700 1 --2- 172.21.15.87:0/1817890264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73f40fd4a0 0x7f73f4109d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f27fc700 1 -- 172.21.15.87:0/1817890264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8005040 con 0x7f73f40fcb40 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f37fe700 1 --2- 172.21.15.87:0/1817890264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f73f40fd4a0 0x7f73f4109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f73f40f7ba0 0x7f73f4109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f27fc700 1 --2- 172.21.15.87:0/1817890264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 0x7f73f4109750 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f73e80048f0 tx=0x7f73e8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73dbfff700 10 client.?.objecter ms_handle_connect 0x7f73f40fcb40 2024-02-23T00:34:35.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73dbfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:35.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e8007b00 con 0x7f73f40fcb40 2024-02-23T00:34:35.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73e800f040 con 0x7f73f40fcb40 2024-02-23T00:34:35.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e4005040 con 0x7f73f40fcb40 2024-02-23T00:34:35.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e8014a00 con 0x7f73f40fcb40 2024-02-23T00:34:35.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 10 client.27015.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:35.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f4114e00 con 0x7f73f40fcb40 2024-02-23T00:34:35.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.297+0000 7f73f94c1700 1 librados: init done 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.298+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f73f405fa40 con 0x7f73f40fcb40 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.299+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f73e800e580 con 0x7f73f40fcb40 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.299+0000 7f73dbfff700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73dc062c90 0x7f73dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.299+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(324..324 src has 173..324) v4 ==== 8722+0+0 (secure 0 0 0) 0x7f73e807d090 con 0x7f73f40fcb40 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.299+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 osd_map(324..324 src has 173..324) v4 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.299+0000 7f73dbfff700 3 client.27015.objecter handle_osd_map got epochs [324,324] > 0 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.299+0000 7f73dbfff700 3 client.27015.objecter handle_osd_map decoding full epoch 324 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.299+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73dc062c90 0x7f73dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.299+0000 7f73dbfff700 20 client.27015.objecter dump_active .. 0 homeless 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.299+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v324) v1 ==== 54+0+16 (secure 0 0 0) 0x7f73e8084070 con 0x7f73f40fcb40 2024-02-23T00:34:35.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73dbfff700 10 client.27015.objecter ms_handle_connect 0x7f73dc062c90 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73dc062c90 0x7f73dc065150 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 rx=0x7f73e4004ee0 tx=0x7f73e400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 10 client.27015.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 1 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6801/2153191863 conn(0x7f73f4028540 0x7f73f40fd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f405ee80 osd=0 3 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f405ee80 osd=0 3 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f405ee80 osd=0 3 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 1 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f405ee80 osd=0 3 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 0 1 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f405ee80 osd=0 4 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 10 client.27015.objecter _send_command 1 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73f4106070 con 0x7f73f4028540 2024-02-23T00:34:35.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f37fe700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6801/2153191863 conn(0x7f73f4028540 0x7f73f40fd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73dbfff700 10 client.27015.objecter ms_handle_connect 0x7f73f4028540 2024-02-23T00:34:35.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73f37fe700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6801/2153191863 conn(0x7f73f4028540 0x7f73f40fd8e0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f73f4106070 con 0x7f73f4028540 2024-02-23T00:34:35.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:34:35.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73dbfff700 10 client.27015.objecter _finish_command 1 = system:0 2024-02-23T00:34:35.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f405ee80 osd=0 4 2024-02-23T00:34:35.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.300+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 0 1 2024-02-23T00:34:35.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 10 client.27015.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 2 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f405ee80 osd=0 3 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f405ee80 osd=0 3 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f405ee80 osd=0 3 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 2 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f405ee80 osd=0 3 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 0 2 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f405ee80 osd=0 4 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 10 client.27015.objecter _send_command 2 2024-02-23T00:34:35.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.316+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73f40f6af0 con 0x7f73f4028540 2024-02-23T00:34:35.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f73f40f6af0 con 0x7f73f4028540 2024-02-23T00:34:35.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:35.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73dbfff700 10 client.27015.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f405ee80 osd=0 4 2024-02-23T00:34:35.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 0 2 2024-02-23T00:34:35.320 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:34:35.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 10 client.27015.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:35.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 3 2024-02-23T00:34:35.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6805/685354171 conn(0x7f73f4027b90 0x7f73f4027fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fef50 osd=1 3 2024-02-23T00:34:35.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fef50 osd=1 3 2024-02-23T00:34:35.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fef50 osd=1 3 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 3 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f40fef50 osd=1 3 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 1 3 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fef50 osd=1 4 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 10 client.27015.objecter _send_command 3 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73f40ff0c0 con 0x7f73f4027b90 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.318+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6805/685354171 conn(0x7f73f4027b90 0x7f73f4027fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.319+0000 7f73dbfff700 10 client.27015.objecter ms_handle_connect 0x7f73f4027b90 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.319+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6805/685354171 conn(0x7f73f4027b90 0x7f73f4027fe0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.319+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f73f40ff0c0 con 0x7f73f4027b90 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.319+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.319+0000 7f73dbfff700 10 client.27015.objecter _finish_command 3 = system:0 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.319+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f40fef50 osd=1 4 2024-02-23T00:34:35.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.319+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 1 3 2024-02-23T00:34:35.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 10 client.27015.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:35.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 4 2024-02-23T00:34:35.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fef50 osd=1 3 2024-02-23T00:34:35.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fef50 osd=1 3 2024-02-23T00:34:35.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fef50 osd=1 3 2024-02-23T00:34:35.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 4 2024-02-23T00:34:35.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f40fef50 osd=1 3 2024-02-23T00:34:35.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 1 4 2024-02-23T00:34:35.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fef50 osd=1 4 2024-02-23T00:34:35.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 10 client.27015.objecter _send_command 4 2024-02-23T00:34:35.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.336+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73f4027600 con 0x7f73f4027b90 2024-02-23T00:34:35.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f73f4027600 con 0x7f73f4027b90 2024-02-23T00:34:35.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:35.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73dbfff700 10 client.27015.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f40fef50 osd=1 4 2024-02-23T00:34:35.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 1 4 2024-02-23T00:34:35.341 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:34:35.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 10 client.27015.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:35.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 5 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6809/3605394228 conn(0x7f73f411a1c0 0x7f73f411c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fe5a0 osd=2 3 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe5a0 osd=2 3 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fe5a0 osd=2 3 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 5 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f40fe5a0 osd=2 3 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 2 5 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe5a0 osd=2 4 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 10 client.27015.objecter _send_command 5 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73f40fed40 con 0x7f73f411a1c0 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.338+0000 7f73f27fc700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6809/3605394228 conn(0x7f73f411a1c0 0x7f73f411c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.339+0000 7f73dbfff700 10 client.27015.objecter ms_handle_connect 0x7f73f411a1c0 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.339+0000 7f73f27fc700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6809/3605394228 conn(0x7f73f411a1c0 0x7f73f411c5b0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.339+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f73f40fed40 con 0x7f73f411a1c0 2024-02-23T00:34:35.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.339+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:34:35.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.339+0000 7f73dbfff700 10 client.27015.objecter _finish_command 5 = system:0 2024-02-23T00:34:35.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.339+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f40fe5a0 osd=2 4 2024-02-23T00:34:35.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.339+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 2 5 2024-02-23T00:34:35.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 10 client.27015.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 6 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fe5a0 osd=2 3 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe5a0 osd=2 3 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fe5a0 osd=2 3 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 6 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f40fe5a0 osd=2 3 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 2 6 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe5a0 osd=2 4 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 10 client.27015.objecter _send_command 6 2024-02-23T00:34:35.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.354+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73f40fe010 con 0x7f73f411a1c0 2024-02-23T00:34:35.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f73f40fe010 con 0x7f73f411a1c0 2024-02-23T00:34:35.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:35.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73dbfff700 10 client.27015.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f40fe5a0 osd=2 4 2024-02-23T00:34:35.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 2 6 2024-02-23T00:34:35.358 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:34:35.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 10 client.27015.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:35.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 7 2024-02-23T00:34:35.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6813/128059790 conn(0x7f73f4119d20 0x7f73f411dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fe420 osd=3 3 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe420 osd=3 3 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fe420 osd=3 3 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 7 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f40fe420 osd=3 3 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 3 7 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe420 osd=3 4 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 10 client.27015.objecter _send_command 7 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73f411e200 con 0x7f73f4119d20 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.355+0000 7f73f37fe700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6813/128059790 conn(0x7f73f4119d20 0x7f73f411dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.356+0000 7f73dbfff700 10 client.27015.objecter ms_handle_connect 0x7f73f4119d20 2024-02-23T00:34:35.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.356+0000 7f73f37fe700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6813/128059790 conn(0x7f73f4119d20 0x7f73f411dc20 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.356+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f73f411e200 con 0x7f73f4119d20 2024-02-23T00:34:35.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.356+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:34:35.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.356+0000 7f73dbfff700 10 client.27015.objecter _finish_command 7 = system:0 2024-02-23T00:34:35.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.356+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f40fe420 osd=3 4 2024-02-23T00:34:35.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.356+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 3 7 2024-02-23T00:34:35.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 10 client.27015.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:35.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 8 2024-02-23T00:34:35.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fe420 osd=3 3 2024-02-23T00:34:35.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe420 osd=3 3 2024-02-23T00:34:35.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f40fe420 osd=3 3 2024-02-23T00:34:35.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 8 2024-02-23T00:34:35.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f40fe420 osd=3 3 2024-02-23T00:34:35.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 3 8 2024-02-23T00:34:35.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe420 osd=3 4 2024-02-23T00:34:35.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 10 client.27015.objecter _send_command 8 2024-02-23T00:34:35.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.370+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73f4119280 con 0x7f73f4119d20 2024-02-23T00:34:35.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f73f4119280 con 0x7f73f4119d20 2024-02-23T00:34:35.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:35.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73dbfff700 10 client.27015.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f40fe420 osd=3 4 2024-02-23T00:34:35.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 3 8 2024-02-23T00:34:35.375 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 10 client.27015.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 9 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6800/4175844757 conn(0x7f73f4121240 0x7f73f4123630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4119450 osd=4 3 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4119450 osd=4 3 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4119450 osd=4 3 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 9 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f4119450 osd=4 3 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 4 9 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4119450 osd=4 4 2024-02-23T00:34:35.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 10 client.27015.objecter _send_command 9 2024-02-23T00:34:35.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73f41195c0 con 0x7f73f4121240 2024-02-23T00:34:35.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.372+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6800/4175844757 conn(0x7f73f4121240 0x7f73f4123630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.373+0000 7f73dbfff700 10 client.27015.objecter ms_handle_connect 0x7f73f4121240 2024-02-23T00:34:35.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.373+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6800/4175844757 conn(0x7f73f4121240 0x7f73f4123630 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.373+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f73f41195c0 con 0x7f73f4121240 2024-02-23T00:34:35.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.373+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:34:35.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.373+0000 7f73dbfff700 10 client.27015.objecter _finish_command 9 = system:0 2024-02-23T00:34:35.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.373+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f4119450 osd=4 4 2024-02-23T00:34:35.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.373+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 4 9 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 10 client.27015.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 10 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4119450 osd=4 3 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4119450 osd=4 3 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4119450 osd=4 3 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 10 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f4119450 osd=4 3 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 4 10 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4119450 osd=4 4 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 10 client.27015.objecter _send_command 10 2024-02-23T00:34:35.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.388+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73f41195c0 con 0x7f73f4121240 2024-02-23T00:34:35.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f73f41195c0 con 0x7f73f4121240 2024-02-23T00:34:35.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:35.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73dbfff700 10 client.27015.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f4119450 osd=4 4 2024-02-23T00:34:35.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 4 10 2024-02-23T00:34:35.392 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:34:35.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 10 client.27015.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:35.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 11 2024-02-23T00:34:35.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6804/2550013795 conn(0x7f73f4124030 0x7f73f4126420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f41195c0 osd=5 3 2024-02-23T00:34:35.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f41195c0 osd=5 3 2024-02-23T00:34:35.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f41195c0 osd=5 3 2024-02-23T00:34:35.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 11 2024-02-23T00:34:35.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f41195c0 osd=5 3 2024-02-23T00:34:35.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 5 11 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f41195c0 osd=5 4 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 10 client.27015.objecter _send_command 11 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73f41291d0 con 0x7f73f4124030 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f27fc700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6804/2550013795 conn(0x7f73f4124030 0x7f73f4126420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73f27fc700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6804/2550013795 conn(0x7f73f4124030 0x7f73f4126420 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.390+0000 7f73dbfff700 10 client.27015.objecter ms_handle_connect 0x7f73f4124030 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.391+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f73f41291d0 con 0x7f73f4124030 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.391+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.391+0000 7f73dbfff700 10 client.27015.objecter _finish_command 11 = system:0 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.391+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f41195c0 osd=5 4 2024-02-23T00:34:35.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.391+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 5 11 2024-02-23T00:34:35.397 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:35.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 10 client.27015.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:35.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 12 2024-02-23T00:34:35.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f41195c0 osd=5 3 2024-02-23T00:34:35.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f41195c0 osd=5 3 2024-02-23T00:34:35.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f41195c0 osd=5 3 2024-02-23T00:34:35.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 12 2024-02-23T00:34:35.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f41195c0 osd=5 3 2024-02-23T00:34:35.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 5 12 2024-02-23T00:34:35.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f41195c0 osd=5 4 2024-02-23T00:34:35.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 10 client.27015.objecter _send_command 12 2024-02-23T00:34:35.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.406+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73f41291d0 con 0x7f73f4124030 2024-02-23T00:34:35.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f73f41291d0 con 0x7f73f4124030 2024-02-23T00:34:35.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:35.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73dbfff700 10 client.27015.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f41195c0 osd=5 4 2024-02-23T00:34:35.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 5 12 2024-02-23T00:34:35.410 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 10 client.27015.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 13 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6808/2509408498 conn(0x7f73f4128640 0x7f73f4128aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4127fc0 osd=6 3 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4127fc0 osd=6 3 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4127fc0 osd=6 3 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 13 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f4127fc0 osd=6 3 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 6 13 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4127fc0 osd=6 4 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 10 client.27015.objecter _send_command 13 2024-02-23T00:34:35.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.407+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73f4128ff0 con 0x7f73f4128640 2024-02-23T00:34:35.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.408+0000 7f73f37fe700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6808/2509408498 conn(0x7f73f4128640 0x7f73f4128aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.408+0000 7f73dbfff700 10 client.27015.objecter ms_handle_connect 0x7f73f4128640 2024-02-23T00:34:35.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.408+0000 7f73f37fe700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6808/2509408498 conn(0x7f73f4128640 0x7f73f4128aa0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.409+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f73f4128ff0 con 0x7f73f4128640 2024-02-23T00:34:35.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.409+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:34:35.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.409+0000 7f73dbfff700 10 client.27015.objecter _finish_command 13 = system:0 2024-02-23T00:34:35.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.409+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f4127fc0 osd=6 4 2024-02-23T00:34:35.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.409+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 6 13 2024-02-23T00:34:35.423 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:35.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 10 client.27015.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:35.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 14 2024-02-23T00:34:35.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4127fc0 osd=6 3 2024-02-23T00:34:35.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4127fc0 osd=6 3 2024-02-23T00:34:35.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4127fc0 osd=6 3 2024-02-23T00:34:35.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 14 2024-02-23T00:34:35.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f4127fc0 osd=6 3 2024-02-23T00:34:35.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 6 14 2024-02-23T00:34:35.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4127fc0 osd=6 4 2024-02-23T00:34:35.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 10 client.27015.objecter _send_command 14 2024-02-23T00:34:35.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.424+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73f4128ff0 con 0x7f73f4128640 2024-02-23T00:34:35.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f73f4128ff0 con 0x7f73f4128640 2024-02-23T00:34:35.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:35.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73dbfff700 10 client.27015.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f4127fc0 osd=6 4 2024-02-23T00:34:35.428 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 6 14 2024-02-23T00:34:35.428 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:34:35.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 10 client.27015.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:35.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 15 2024-02-23T00:34:35.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6812/373787000 conn(0x7f73f412b950 0x7f73f412bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:35.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4128ff0 osd=7 3 2024-02-23T00:34:35.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4128ff0 osd=7 3 2024-02-23T00:34:35.429 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4128ff0 osd=7 3 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 15 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f4128ff0 osd=7 3 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 7 15 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4128ff0 osd=7 4 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 10 client.27015.objecter _send_command 15 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73f412c400 con 0x7f73f412b950 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.426+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6812/373787000 conn(0x7f73f412b950 0x7f73f412bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.427+0000 7f73dbfff700 10 client.27015.objecter ms_handle_connect 0x7f73f412b950 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.427+0000 7f73f2ffd700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6812/373787000 conn(0x7f73f412b950 0x7f73f412bd90 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.427+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f73e401a040 con 0x7f73f412b950 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.427+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.427+0000 7f73dbfff700 10 client.27015.objecter _finish_command 15 = system:0 2024-02-23T00:34:35.430 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.427+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f4128ff0 osd=7 4 2024-02-23T00:34:35.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.427+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 7 15 2024-02-23T00:34:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:33.620784+0000 mgr.y (mgr.24437) 319 : cluster [DBG] pgmap v466: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:35.049792+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-02-23T00:34:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:35.300723+0000 mon.a (mon.0) 2179 : audit [DBG] from='client.? 172.21.15.87:0/1817890264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:35.442 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:35.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 10 client.27015.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:35.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign -1 16 2024-02-23T00:34:35.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4128ff0 osd=7 3 2024-02-23T00:34:35.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4128ff0 osd=7 3 2024-02-23T00:34:35.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 20 client.27015.objecter _get_session s=0x7f73f4128ff0 osd=7 3 2024-02-23T00:34:35.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_remove -1 16 2024-02-23T00:34:35.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 20 client.27015.objecter get_session s=0x7f73f4128ff0 osd=7 3 2024-02-23T00:34:35.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 15 client.27015.objecter _session_command_op_assign 7 16 2024-02-23T00:34:35.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4128ff0 osd=7 4 2024-02-23T00:34:35.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 10 client.27015.objecter _send_command 16 2024-02-23T00:34:35.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.444+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f73f412e670 con 0x7f73f412b950 2024-02-23T00:34:35.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73dbfff700 1 -- 172.21.15.87:0/1817890264 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f73f412e670 con 0x7f73f412b950 2024-02-23T00:34:35.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73dbfff700 10 client.27015.objecter ms_dispatch 0x7f73f4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:35.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73dbfff700 10 client.27015.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73dbfff700 20 client.27015.objecter put_session s=0x7f73f4128ff0 osd=7 4 2024-02-23T00:34:35.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73dbfff700 15 client.27015.objecter _session_command_op_remove 7 16 2024-02-23T00:34:35.448 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:34:35.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 librados: watch_flush enter 2024-02-23T00:34:35.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 librados: watch_flush exit 2024-02-23T00:34:35.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 client.27015.objecter close_session for osd.0 2024-02-23T00:34:35.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6801/2153191863 conn(0x7f73f4028540 msgr2=0x7f73f40fd8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6801/2153191863 conn(0x7f73f4028540 0x7f73f40fd8e0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f405ee80 osd=0 1 2024-02-23T00:34:35.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 client.27015.objecter close_session for osd.1 2024-02-23T00:34:35.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6805/685354171 conn(0x7f73f4027b90 msgr2=0x7f73f4027fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6805/685354171 conn(0x7f73f4027b90 0x7f73f4027fe0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fef50 osd=1 1 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 client.27015.objecter close_session for osd.2 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6809/3605394228 conn(0x7f73f411a1c0 msgr2=0x7f73f411c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6809/3605394228 conn(0x7f73f411a1c0 0x7f73f411c5b0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe5a0 osd=2 1 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 client.27015.objecter close_session for osd.3 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6813/128059790 conn(0x7f73f4119d20 msgr2=0x7f73f411dc20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6813/128059790 conn(0x7f73f4119d20 0x7f73f411dc20 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f40fe420 osd=3 1 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 client.27015.objecter close_session for osd.4 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f37fe700 1 -- 172.21.15.87:0/1817890264 reap_dead start 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6800/4175844757 conn(0x7f73f4121240 msgr2=0x7f73f4123630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6800/4175844757 conn(0x7f73f4121240 0x7f73f4123630 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f37fe700 1 -- 172.21.15.87:0/1817890264 reap_dead start 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4119450 osd=4 1 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 client.27015.objecter close_session for osd.5 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6804/2550013795 conn(0x7f73f4124030 msgr2=0x7f73f4126420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6804/2550013795 conn(0x7f73f4124030 0x7f73f4126420 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f41195c0 osd=5 1 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 client.27015.objecter close_session for osd.6 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6808/2509408498 conn(0x7f73f4128640 msgr2=0x7f73f4128aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6808/2509408498 conn(0x7f73f4128640 0x7f73f4128aa0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4127fc0 osd=6 1 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 client.27015.objecter close_session for osd.7 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6812/373787000 conn(0x7f73f412b950 msgr2=0x7f73f412bd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.171:6812/373787000 conn(0x7f73f412b950 0x7f73f412bd90 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 20 client.27015.objecter put_session s=0x7f73f4128ff0 osd=7 1 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 20 client.27015.objecter shutdown clearing up homeless session... 2024-02-23T00:34:35.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 10 client.27015.objecter successfully canceled tick 2024-02-23T00:34:35.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73dc062c90 msgr2=0x7f73dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> v2:172.21.15.87:6800/1231597463 conn(0x7f73dc062c90 0x7f73dc065150 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 rx=0x7f73e4004ee0 tx=0x7f73e400b040).stop 2024-02-23T00:34:35.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 msgr2=0x7f73f4109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:35.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f37fe700 1 -- 172.21.15.87:0/1817890264 reap_dead start 2024-02-23T00:34:35.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 0x7f73f4109750 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 rx=0x7f73e80048f0 tx=0x7f73e8009cb0).stop 2024-02-23T00:34:35.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 shutdown_connections 2024-02-23T00:34:35.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 --2- 172.21.15.87:0/1817890264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f73f40fcb40 0x7f73f4109750 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:35.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 >> 172.21.15.87:0/1817890264 conn(0x7f73f406c370 msgr2=0x7f73f40f54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:35.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:35.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 shutdown_connections 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 -- 172.21.15.87:0/1817890264 wait complete. 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:35.446+0000 7f73f94c1700 1 librados: shutdown 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.454 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:34:35.554 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:35.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:33.620784+0000 mgr.y (mgr.24437) 319 : cluster [DBG] pgmap v466: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:35.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:35.049792+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-02-23T00:34:35.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:35.300723+0000 mon.a (mon.0) 2179 : audit [DBG] from='client.? 172.21.15.87:0/1817890264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:35.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:33.620784+0000 mgr.y (mgr.24437) 319 : cluster [DBG] pgmap v466: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:34:35.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:35.049792+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-02-23T00:34:35.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:35.300723+0000 mon.a (mon.0) 2179 : audit [DBG] from='client.? 172.21.15.87:0/1817890264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:35.724 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:34:35.724 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:34:36.030 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:36.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:36.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:36.134 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:36.164 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:36.455 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-23T00:34:36.513 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:36.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:36.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:35.622582+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-23T00:34:36.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:36.054938+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-23T00:34:36.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:36.055005+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-02-23T00:34:36.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:36.060686+0000 mon.a (mon.0) 2183 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:36.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:35.622582+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-23T00:34:36.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:36.054938+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-23T00:34:36.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:36.055005+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-02-23T00:34:36.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:36.060686+0000 mon.a (mon.0) 2183 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:36.616 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:36.640 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:35.622582+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-23T00:34:36.640 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:36.054938+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-23T00:34:36.640 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:36.055005+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-02-23T00:34:36.640 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:36.060686+0000 mon.a (mon.0) 2183 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:36.642 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:36.662 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:36.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:36.766 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:36.766 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:36.794 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:37.040 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:34:37.040 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:34:37.041 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:34:37.041 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd out 3 2024-02-23T00:34:37.043 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:37.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:37.147 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:37.175 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:37.333 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:37.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:35.621686+0000 mgr.y (mgr.24437) 320 : cluster [DBG] pgmap v469: 246 pgs: 246 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 1.7 KiB/s wr, 1 op/s; 247 B/s, 14 objects/s recovering 2024-02-23T00:34:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:36.315111+0000 mon.a (mon.0) 2184 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:34:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:37.064431+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi087-82019-56"}]': finished 2024-02-23T00:34:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:37.064513+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-02-23T00:34:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:37.067506+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:35.621686+0000 mgr.y (mgr.24437) 320 : cluster [DBG] pgmap v469: 246 pgs: 246 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 1.7 KiB/s wr, 1 op/s; 247 B/s, 14 objects/s recovering 2024-02-23T00:34:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:36.315111+0000 mon.a (mon.0) 2184 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:34:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:37.064431+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi087-82019-56"}]': finished 2024-02-23T00:34:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:37.064513+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-02-23T00:34:37.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:37.067506+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:37.623 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:37.625 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:35.621686+0000 mgr.y (mgr.24437) 320 : cluster [DBG] pgmap v469: 246 pgs: 246 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 1.7 KiB/s wr, 1 op/s; 247 B/s, 14 objects/s recovering 2024-02-23T00:34:37.626 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:36.315111+0000 mon.a (mon.0) 2184 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:34:37.626 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:37.064431+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi087-82019-56"}]': finished 2024-02-23T00:34:37.626 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:37.064513+0000 mon.a (mon.0) 2186 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-02-23T00:34:37.626 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:37.067506+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi087-82019-56"}]: dispatch 2024-02-23T00:34:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:37.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:37.728 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:37.755 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:37.829 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:37.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:37.932 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:37.961 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:38.026 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:34:38.084 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:38.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:38.188 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:38.215 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:38.337 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:38.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:38.441 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:38.468 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:38.597 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:34:38.597 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:34:38.868 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:34:38.916 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:38.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:39.020 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:39.047 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:39.135 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:37.622170+0000 mgr.y (mgr.24437) 321 : cluster [DBG] pgmap v472: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s; 247 B/s, 14 objects/s recovering 2024-02-23T00:34:39.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:38.064379+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi087-82019-56"}]': finished 2024-02-23T00:34:39.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:38.064466+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-02-23T00:34:39.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:38.088284+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:39.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:38.089128+0000 mon.a (mon.0) 2191 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:39.136 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:38.089416+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi087-82019-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:39.136 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:37.622170+0000 mgr.y (mgr.24437) 321 : cluster [DBG] pgmap v472: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s; 247 B/s, 14 objects/s recovering 2024-02-23T00:34:39.136 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:38.064379+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi087-82019-56"}]': finished 2024-02-23T00:34:39.136 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:38.064466+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-02-23T00:34:39.136 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:38.088284+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:39.137 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:38.089128+0000 mon.a (mon.0) 2191 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:39.137 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:38.089416+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi087-82019-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:39.147 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:39.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:39.168 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:39.252 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:39.256 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:37.622170+0000 mgr.y (mgr.24437) 321 : cluster [DBG] pgmap v472: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s; 247 B/s, 14 objects/s recovering 2024-02-23T00:34:39.256 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:38.064379+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? 172.21.15.87:0/3963489775' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi087-82019-56"}]': finished 2024-02-23T00:34:39.256 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:38.064466+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-02-23T00:34:39.257 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:38.088284+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:39.257 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:38.089128+0000 mon.a (mon.0) 2191 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:39.257 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:38.089416+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi087-82019-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:39.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-23T00:34:39.284 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:39.304 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:39.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:39.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.372+0000 7fdeb22fc700 1 -- 172.21.15.87:0/2677306959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdeac0fcbb0 msgr2=0x7fdeac0fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:39.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.372+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/2677306959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdeac0fcbb0 0x7fdeac0fcf90 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7fde9c004870 tx=0x7fde9c009cb0).stop 2024-02-23T00:34:39.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.372+0000 7fdeb22fc700 1 -- 172.21.15.87:0/2677306959 shutdown_connections 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.372+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/2677306959 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdeac0f7400 0x7fdeac103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.372+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/2677306959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdeac0fcbb0 0x7fdeac0fcf90 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.372+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/2677306959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0fd510 0x7fdeac0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.372+0000 7fdeb22fc700 1 -- 172.21.15.87:0/2677306959 >> 172.21.15.87:0/2677306959 conn(0x7fdeac0f2820 msgr2=0x7fdeac0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.372+0000 7fdeb22fc700 1 -- 172.21.15.87:0/2677306959 shutdown_connections 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.372+0000 7fdeb22fc700 1 -- 172.21.15.87:0/2677306959 wait complete. 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 librados: starting msgr at 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 librados: starting objecter 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 Processor -- start 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 -- start start 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 librados: setting wanted keys 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 librados: calling monclient init 2024-02-23T00:34:39.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f7400 0x7fdeac1092e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdeac0fcbb0 0x7fdeac1097d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdeac0fd510 0x7fdeac109d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac1144d0 con 0x7fdeac0f7400 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac114640 con 0x7fdeac0fcbb0 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.373+0000 7fdeb22fc700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac114800 con 0x7fdeac0fd510 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeabfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f7400 0x7fdeac1092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeabfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f7400 0x7fdeac1092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48450/0 (socket says 172.21.15.87:48450) 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeabfff700 1 -- 172.21.15.87:0/3895849727 learned_addr learned my addr 172.21.15.87:0/3895849727 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeb0899700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdeac0fd510 0x7fdeac109d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeab7fe700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdeac0fcbb0 0x7fdeac1097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeabfff700 1 -- 172.21.15.87:0/3895849727 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdeac0fd510 msgr2=0x7fdeac109d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeabfff700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdeac0fd510 0x7fdeac109d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:39.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeabfff700 1 -- 172.21.15.87:0/3895849727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdeac0fcbb0 msgr2=0x7fdeac1097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:39.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeabfff700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdeac0fcbb0 0x7fdeac1097d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:39.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeabfff700 1 -- 172.21.15.87:0/3895849727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde94005040 con 0x7fdeac0f7400 2024-02-23T00:34:39.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeb0899700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdeac0fd510 0x7fdeac109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:39.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeabfff700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f7400 0x7fdeac1092e0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 rx=0x7fde9c009eb0 tx=0x7fde9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:39.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdea97fa700 10 client.?.objecter ms_handle_connect 0x7fdeac0f7400 2024-02-23T00:34:39.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdea97fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:39.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdea97fa700 1 -- 172.21.15.87:0/3895849727 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c00e070 con 0x7fdeac0f7400 2024-02-23T00:34:39.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9c005040 con 0x7fdeac0f7400 2024-02-23T00:34:39.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdea97fa700 1 -- 172.21.15.87:0/3895849727 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde9c013da0 con 0x7fdeac0f7400 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdea97fa700 1 -- 172.21.15.87:0/3895849727 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9c019070 con 0x7fdeac0f7400 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeb22fc700 10 client.27027.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeac114cf0 con 0x7fdeac0f7400 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.374+0000 7fdeb22fc700 1 librados: init done 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.375+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeac05fa60 con 0x7fdeac0f7400 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.375+0000 7fdea97fa700 1 -- 172.21.15.87:0/3895849727 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fde9c01e420 con 0x7fdeac0f7400 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.376+0000 7fdea97fa700 1 --2- 172.21.15.87:0/3895849727 >> v2:172.21.15.87:6800/1231597463 conn(0x7fde98062c90 0x7fde98065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.376+0000 7fdea97fa700 1 -- 172.21.15.87:0/3895849727 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(328..328 src has 173..328) v4 ==== 8222+0+0 (secure 0 0 0) 0x7fde9c07dd20 con 0x7fdeac0f7400 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.376+0000 7fdea97fa700 10 client.27027.objecter ms_dispatch 0x7fdeac001550 osd_map(328..328 src has 173..328) v4 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.376+0000 7fdea97fa700 3 client.27027.objecter handle_osd_map got epochs [328,328] > 0 2024-02-23T00:34:39.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.376+0000 7fdea97fa700 3 client.27027.objecter handle_osd_map decoding full epoch 328 2024-02-23T00:34:39.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.376+0000 7fdeab7fe700 1 --2- 172.21.15.87:0/3895849727 >> v2:172.21.15.87:6800/1231597463 conn(0x7fde98062c90 0x7fde98065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:39.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.376+0000 7fdea97fa700 20 client.27027.objecter dump_active .. 0 homeless 2024-02-23T00:34:39.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.376+0000 7fdea97fa700 10 client.27027.objecter ms_handle_connect 0x7fde98062c90 2024-02-23T00:34:39.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.376+0000 7fdeab7fe700 1 --2- 172.21.15.87:0/3895849727 >> v2:172.21.15.87:6800/1231597463 conn(0x7fde98062c90 0x7fde98065150 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7fdeac061720 tx=0x7fde9400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:39.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.380+0000 7fdea97fa700 1 -- 172.21.15.87:0/3895849727 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fde9c051150 con 0x7fdeac0f7400 2024-02-23T00:34:39.409 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:39.432 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:39.435 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:39.454 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:39.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:39.554 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:39.559 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:39.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:39.577+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd out", "ids": ["3"]} v 0) v1 -- 0x7fdeac106170 con 0x7fdeac0f7400 2024-02-23T00:34:39.587 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:39.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:34:39.578+0000 7fe8e2097700 -1 mon.a@0(leader).osd e328 definitely_dead 0 2024-02-23T00:34:40.236 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:40.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:40.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:40.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.320+0000 7fdea97fa700 1 -- 172.21.15.87:0/3895849727 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["3"]}]=0 marked out osd.3. v329) v1 ==== 87+0+0 (secure 0 0 0) 0x7fde9c023030 con 0x7fdeac0f7400 2024-02-23T00:34:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.323+0000 7fdeb22fc700 10 librados: watch_flush enter 2024-02-23T00:34:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.323+0000 7fdeb22fc700 10 librados: watch_flush exit 2024-02-23T00:34:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.323+0000 7fdeb22fc700 20 client.27027.objecter shutdown clearing up homeless session... 2024-02-23T00:34:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.323+0000 7fdeb22fc700 10 client.27027.objecter successfully canceled tick 2024-02-23T00:34:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.323+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 >> v2:172.21.15.87:6800/1231597463 conn(0x7fde98062c90 msgr2=0x7fde98065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.323+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/3895849727 >> v2:172.21.15.87:6800/1231597463 conn(0x7fde98062c90 0x7fde98065150 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 rx=0x7fdeac061720 tx=0x7fde9400b040).stop 2024-02-23T00:34:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.323+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f7400 msgr2=0x7fdeac1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:40.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.323+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f7400 0x7fdeac1092e0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 rx=0x7fde9c009eb0 tx=0x7fde9c00b040).stop 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.324+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 shutdown_connections 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.324+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/3895849727 >> v2:172.21.15.87:6800/1231597463 conn(0x7fde98062c90 0x7fde98065150 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.324+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdeac0fd510 0x7fdeac109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.324+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdeac0f7400 0x7fdeac1092e0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.324+0000 7fdeb22fc700 1 --2- 172.21.15.87:0/3895849727 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdeac0fcbb0 0x7fdeac1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.324+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 >> 172.21.15.87:0/3895849727 conn(0x7fdeac0f2820 msgr2=0x7fdeac0fa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.324+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 shutdown_connections 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.324+0000 7fdeb22fc700 1 -- 172.21.15.87:0/3895849727 wait complete. 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:40.324+0000 7fdeb22fc700 1 librados: shutdown 2024-02-23T00:34:40.327 INFO:teuthology.orchestra.run.smithi087.stderr:marked out osd.3. 2024-02-23T00:34:40.340 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:40.370 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:40.476 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:39.068009+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi087-82019-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:40.476 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:39.068071+0000 mon.a (mon.0) 2194 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-02-23T00:34:40.476 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:39.069119+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi087-82019-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:40.476 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:39.517878+0000 osd.3 (osd.3) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:40.476 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:39.517973+0000 osd.3 (osd.3) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:40.476 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:39.572411+0000 mon.a (mon.0) 2196 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:34:40.476 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:39.579362+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? 172.21.15.87:0/3895849727' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-23T00:34:40.476 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:39.579443+0000 mon.a (mon.0) 2198 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-23T00:34:40.477 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:39.623481+0000 mon.a (mon.0) 2199 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-23T00:34:40.568 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:40.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:39.068009+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi087-82019-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:40.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:39.068071+0000 mon.a (mon.0) 2194 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-02-23T00:34:40.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:39.069119+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi087-82019-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:40.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:39.517878+0000 osd.3 (osd.3) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:40.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:39.517973+0000 osd.3 (osd.3) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:40.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:39.572411+0000 mon.a (mon.0) 2196 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:34:40.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:39.579362+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? 172.21.15.87:0/3895849727' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:39.579443+0000 mon.a (mon.0) 2198 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:39.623481+0000 mon.a (mon.0) 2199 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:39.068009+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi087-82019-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:39.068071+0000 mon.a (mon.0) 2194 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:39.069119+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi087-82019-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:39.517878+0000 osd.3 (osd.3) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:39.517973+0000 osd.3 (osd.3) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:39.572411+0000 mon.a (mon.0) 2196 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:39.579362+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? 172.21.15.87:0/3895849727' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-23T00:34:40.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:39.579443+0000 mon.a (mon.0) 2198 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-23T00:34:40.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:39.623481+0000 mon.a (mon.0) 2199 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-23T00:34:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:40.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:40.673 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:40.700 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:40.798 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:40.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:40.902 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:40.931 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:41.166 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:34:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:34:41.187 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:41.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:41.291 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:41.318 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:41.339 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:41.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:39.622596+0000 mgr.y (mgr.24437) 322 : cluster [DBG] pgmap v475: 237 pgs: 237 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.321833+0000 mon.a (mon.0) 2200 : audit [INF] from='client.? 172.21.15.87:0/3895849727' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-23T00:34:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.321949+0000 mon.a (mon.0) 2201 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-23T00:34:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:40.321999+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-02-23T00:34:41.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.323215+0000 mon.a (mon.0) 2203 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 1]}]: dispatch 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.327167+0000 mon.a (mon.0) 2204 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.327454+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]: dispatch 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.330972+0000 mon.a (mon.0) 2206 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 5]}]: dispatch 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.331179+0000 mon.a (mon.0) 2207 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.0", "id": [1, 7]}]: dispatch 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.331322+0000 mon.a (mon.0) 2208 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1d", "id": [1, 6]}]: dispatch 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:40.352533+0000 osd.1 (osd.1) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:40.352719+0000 osd.1 (osd.1) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.890674+0000 mon.a (mon.0) 2209 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.891328+0000 mon.a (mon.0) 2210 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.895306+0000 mon.a (mon.0) 2211 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:41.070945+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi087-82019-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi087-82019-57"}]': finished 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:41.071040+0000 mon.a (mon.0) 2213 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 1]}]': finished 2024-02-23T00:34:41.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:41.071160+0000 mon.a (mon.0) 2214 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-02-23T00:34:41.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:41.071229+0000 mon.a (mon.0) 2215 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]': finished 2024-02-23T00:34:41.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:41.071298+0000 mon.a (mon.0) 2216 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 5]}]': finished 2024-02-23T00:34:41.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:41.071371+0000 mon.a (mon.0) 2217 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.0", "id": [1, 7]}]': finished 2024-02-23T00:34:41.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:41.071425+0000 mon.a (mon.0) 2218 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1d", "id": [1, 6]}]': finished 2024-02-23T00:34:41.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:41.071474+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-02-23T00:34:41.447 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:41.475 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:39.622596+0000 mgr.y (mgr.24437) 322 : cluster [DBG] pgmap v475: 237 pgs: 237 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.321833+0000 mon.a (mon.0) 2200 : audit [INF] from='client.? 172.21.15.87:0/3895849727' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-23T00:34:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.321949+0000 mon.a (mon.0) 2201 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-23T00:34:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:40.321999+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-02-23T00:34:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.323215+0000 mon.a (mon.0) 2203 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 1]}]: dispatch 2024-02-23T00:34:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.327167+0000 mon.a (mon.0) 2204 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-23T00:34:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.327454+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]: dispatch 2024-02-23T00:34:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.330972+0000 mon.a (mon.0) 2206 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 5]}]: dispatch 2024-02-23T00:34:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.331179+0000 mon.a (mon.0) 2207 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.0", "id": [1, 7]}]: dispatch 2024-02-23T00:34:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.331322+0000 mon.a (mon.0) 2208 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1d", "id": [1, 6]}]: dispatch 2024-02-23T00:34:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:40.352533+0000 osd.1 (osd.1) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:40.352719+0000 osd.1 (osd.1) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.890674+0000 mon.a (mon.0) 2209 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:34:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.891328+0000 mon.a (mon.0) 2210 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:34:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.895306+0000 mon.a (mon.0) 2211 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:34:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:39.622596+0000 mgr.y (mgr.24437) 322 : cluster [DBG] pgmap v475: 237 pgs: 237 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:34:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.321833+0000 mon.a (mon.0) 2200 : audit [INF] from='client.? 172.21.15.87:0/3895849727' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.321949+0000 mon.a (mon.0) 2201 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:40.321999+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.323215+0000 mon.a (mon.0) 2203 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 1]}]: dispatch 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.327167+0000 mon.a (mon.0) 2204 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.327454+0000 mon.a (mon.0) 2205 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]: dispatch 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.330972+0000 mon.a (mon.0) 2206 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 5]}]: dispatch 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.331179+0000 mon.a (mon.0) 2207 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.0", "id": [1, 7]}]: dispatch 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.331322+0000 mon.a (mon.0) 2208 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1d", "id": [1, 6]}]: dispatch 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:40.352533+0000 osd.1 (osd.1) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:40.352719+0000 osd.1 (osd.1) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.890674+0000 mon.a (mon.0) 2209 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.891328+0000 mon.a (mon.0) 2210 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:34:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.895306+0000 mon.a (mon.0) 2211 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:41.070945+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi087-82019-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi087-82019-57"}]': finished 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:41.071040+0000 mon.a (mon.0) 2213 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 1]}]': finished 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:41.071160+0000 mon.a (mon.0) 2214 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:41.071229+0000 mon.a (mon.0) 2215 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]': finished 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:41.071298+0000 mon.a (mon.0) 2216 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 5]}]': finished 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:41.071371+0000 mon.a (mon.0) 2217 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.0", "id": [1, 7]}]': finished 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:41.071425+0000 mon.a (mon.0) 2218 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1d", "id": [1, 6]}]': finished 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:41.071474+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:41.070945+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi087-82019-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi087-82019-57"}]': finished 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:41.071040+0000 mon.a (mon.0) 2213 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 1]}]': finished 2024-02-23T00:34:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:41.071160+0000 mon.a (mon.0) 2214 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-02-23T00:34:41.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:41.071229+0000 mon.a (mon.0) 2215 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.a", "id": [7, 0]}]': finished 2024-02-23T00:34:41.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:41.071298+0000 mon.a (mon.0) 2216 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 5]}]': finished 2024-02-23T00:34:41.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:41.071371+0000 mon.a (mon.0) 2217 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.0", "id": [1, 7]}]': finished 2024-02-23T00:34:41.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:41.071425+0000 mon.a (mon.0) 2218 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1d", "id": [1, 6]}]': finished 2024-02-23T00:34:41.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:41.071474+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-02-23T00:34:41.752 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:34:41.752 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:34:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:40.812841+0000 mgr.y (mgr.24437) 323 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:42.045165+0000 mon.a (mon.0) 2220 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:34:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:42.049992+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-02-23T00:34:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:42.062275+0000 osd.1 (osd.1) 154 : cluster [DBG] 4.f starting backfill to osd.7 from (0'0,0'0] MAX to 87'22 2024-02-23T00:34:42.460 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-23T00:34:42.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:34:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:42] "GET /metrics HTTP/1.1" 200 187225 "" "Prometheus/2.33.4" 2024-02-23T00:34:42.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:40.812841+0000 mgr.y (mgr.24437) 323 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:42.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:42.045165+0000 mon.a (mon.0) 2220 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:34:42.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:42.049992+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-02-23T00:34:42.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:42.062275+0000 osd.1 (osd.1) 154 : cluster [DBG] 4.f starting backfill to osd.7 from (0'0,0'0] MAX to 87'22 2024-02-23T00:34:42.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:40.812841+0000 mgr.y (mgr.24437) 323 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:42.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:42.045165+0000 mon.a (mon.0) 2220 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:34:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:42.049992+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-02-23T00:34:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:42.062275+0000 osd.1 (osd.1) 154 : cluster [DBG] 4.f starting backfill to osd.7 from (0'0,0'0] MAX to 87'22 2024-02-23T00:34:42.631 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:42.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:42.732 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:42.737 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:42.766 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:43.014 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:43.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:43.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.101+0000 7f0b427e2700 1 -- 172.21.15.87:0/4231914059 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0fd510 msgr2=0x7f0b3c0f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.101+0000 7f0b427e2700 1 --2- 172.21.15.87:0/4231914059 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0fd510 0x7f0b3c0f6d40 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7f0b30004870 tx=0x7f0b30009cb0).stop 2024-02-23T00:34:43.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.101+0000 7f0b427e2700 1 -- 172.21.15.87:0/4231914059 shutdown_connections 2024-02-23T00:34:43.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.101+0000 7f0b427e2700 1 --2- 172.21.15.87:0/4231914059 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0b3c0f7400 0x7f0b3c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.101+0000 7f0b427e2700 1 --2- 172.21.15.87:0/4231914059 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b3c0fcbb0 0x7f0b3c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.101+0000 7f0b427e2700 1 --2- 172.21.15.87:0/4231914059 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0fd510 0x7f0b3c0f6d40 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.101+0000 7f0b427e2700 1 -- 172.21.15.87:0/4231914059 >> 172.21.15.87:0/4231914059 conn(0x7f0b3c0f2820 msgr2=0x7f0b3c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.101+0000 7f0b427e2700 1 -- 172.21.15.87:0/4231914059 shutdown_connections 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.102+0000 7f0b427e2700 1 -- 172.21.15.87:0/4231914059 wait complete. 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.102+0000 7f0b427e2700 1 librados: starting msgr at 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.102+0000 7f0b427e2700 1 librados: starting objecter 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.102+0000 7f0b427e2700 1 Processor -- start 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.102+0000 7f0b427e2700 1 -- start start 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.102+0000 7f0b427e2700 1 librados: setting wanted keys 2024-02-23T00:34:43.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.102+0000 7f0b427e2700 1 librados: calling monclient init 2024-02-23T00:34:43.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b427e2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0f7400 0x7f0b3c10d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b427e2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b3c0fcbb0 0x7f0b3c10d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b427e2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0b3c0fd510 0x7f0b3c1072c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b427e2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c1077b0 con 0x7f0b3c0fcbb0 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b427e2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c107920 con 0x7f0b3c0fd510 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b427e2700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c107ae0 con 0x7f0b3c0f7400 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0f7400 0x7f0b3c10d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b3c0fcbb0 0x7f0b3c10d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0f7400 0x7f0b3c10d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34842/0 (socket says 172.21.15.87:34842) 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b3c0fcbb0 0x7f0b3c10d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42084/0 (socket says 172.21.15.87:42084) 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3bfff700 1 -- 172.21.15.87:0/16199064 learned_addr learned my addr 172.21.15.87:0/16199064 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b40d7f700 1 --2- 172.21.15.87:0/16199064 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0b3c0fd510 0x7f0b3c1072c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3bfff700 1 -- 172.21.15.87:0/16199064 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0b3c0fd510 msgr2=0x7f0b3c1072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3bfff700 1 --2- 172.21.15.87:0/16199064 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0b3c0fd510 0x7f0b3c1072c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3bfff700 1 -- 172.21.15.87:0/16199064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b3c0fcbb0 msgr2=0x7f0b3c10d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3bfff700 1 --2- 172.21.15.87:0/16199064 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0b3c0fcbb0 0x7f0b3c10d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3bfff700 1 -- 172.21.15.87:0/16199064 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c005040 con 0x7f0b3c0f7400 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b3bfff700 1 --2- 172.21.15.87:0/16199064 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0f7400 0x7f0b3c10d120 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7f0b2c009400 tx=0x7f0b2c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b397fa700 10 client.?.objecter ms_handle_connect 0x7f0b3c0f7400 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b397fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.103+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c00e030 con 0x7f0b3c0f7400 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.104+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b2c004e10 con 0x7f0b3c0f7400 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.104+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b30005040 con 0x7f0b3c0f7400 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.104+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c01adb0 con 0x7f0b3c0f7400 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.104+0000 7f0b427e2700 10 client.37490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.104+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c1080c0 con 0x7f0b3c0f7400 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.104+0000 7f0b427e2700 1 librados: init done 2024-02-23T00:34:43.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.104+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0b3c05fa60 con 0x7f0b3c0f7400 2024-02-23T00:34:43.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.106+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0b2c004940 con 0x7f0b3c0f7400 2024-02-23T00:34:43.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.106+0000 7f0b397fa700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0b24062c90 0x7f0b24065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.106+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(332..332 src has 173..332) v4 ==== 8536+0+0 (secure 0 0 0) 0x7f0b2c081610 con 0x7f0b3c0f7400 2024-02-23T00:34:43.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.106+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 osd_map(332..332 src has 173..332) v4 2024-02-23T00:34:43.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.106+0000 7f0b397fa700 3 client.37490.objecter handle_osd_map got epochs [332,332] > 0 2024-02-23T00:34:43.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.106+0000 7f0b397fa700 3 client.37490.objecter handle_osd_map decoding full epoch 332 2024-02-23T00:34:43.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.106+0000 7f0b397fa700 20 client.37490.objecter dump_active .. 0 homeless 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.106+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v332) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0b2c081ab0 con 0x7f0b3c0f7400 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 10 client.37490.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 1 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0b3c061380 0x7f0b3c061760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c05ee80 osd=0 3 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c05ee80 osd=0 3 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c05ee80 osd=0 3 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 1 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c05ee80 osd=0 3 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 0 1 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c05ee80 osd=0 4 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 10 client.37490.objecter _send_command 1 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b3c106120 con 0x7f0b3c061380 2024-02-23T00:34:43.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b40d7f700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0b3c061380 0x7f0b3c061760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b397fa700 10 client.37490.objecter ms_handle_connect 0x7f0b3c061380 2024-02-23T00:34:43.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.107+0000 7f0b40d7f700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0b3c061380 0x7f0b3c061760 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.108+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0b3c106120 con 0x7f0b3c061380 2024-02-23T00:34:43.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.108+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:34:43.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.108+0000 7f0b397fa700 10 client.37490.objecter _finish_command 1 = system:0 2024-02-23T00:34:43.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.108+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c05ee80 osd=0 4 2024-02-23T00:34:43.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.108+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 0 1 2024-02-23T00:34:43.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.112+0000 7f0b3b7fe700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0b24062c90 0x7f0b24065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.112+0000 7f0b397fa700 10 client.37490.objecter ms_handle_connect 0x7f0b24062c90 2024-02-23T00:34:43.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.112+0000 7f0b3b7fe700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0b24062c90 0x7f0b24065150 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 rx=0x7f0b30004ee0 tx=0x7f0b3000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.118 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 10 client.37490.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 2 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c05ee80 osd=0 3 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c05ee80 osd=0 3 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c05ee80 osd=0 3 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 2 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c05ee80 osd=0 3 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 0 2 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c05ee80 osd=0 4 2024-02-23T00:34:43.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 10 client.37490.objecter _send_command 2 2024-02-23T00:34:43.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.123+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0b3c0fd9b0 con 0x7f0b3c061380 2024-02-23T00:34:43.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.124+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0b3c0fd9b0 con 0x7f0b3c061380 2024-02-23T00:34:43.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.124+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:43.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.124+0000 7f0b397fa700 10 client.37490.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.124+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c05ee80 osd=0 4 2024-02-23T00:34:43.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.124+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 0 2 2024-02-23T00:34:43.127 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 10 client.37490.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 3 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6805/685354171 conn(0x7f0b3c0f4d50 0x7f0b3c0273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0f4be0 osd=1 3 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0f4be0 osd=1 3 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0f4be0 osd=1 3 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 3 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c0f4be0 osd=1 3 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 1 3 2024-02-23T00:34:43.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0f4be0 osd=1 4 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 10 client.37490.objecter _send_command 3 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b3c0fe860 con 0x7f0b3c0f4d50 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b3bfff700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6805/685354171 conn(0x7f0b3c0f4d50 0x7f0b3c0273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b397fa700 10 client.37490.objecter ms_handle_connect 0x7f0b3c0f4d50 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.125+0000 7f0b3bfff700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6805/685354171 conn(0x7f0b3c0f4d50 0x7f0b3c0273c0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.126+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0b3c0fe860 con 0x7f0b3c0f4d50 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.126+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.126+0000 7f0b397fa700 10 client.37490.objecter _finish_command 3 = system:0 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.126+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c0f4be0 osd=1 4 2024-02-23T00:34:43.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.126+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 1 3 2024-02-23T00:34:43.147 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:43.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 10 client.37490.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:43.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 4 2024-02-23T00:34:43.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0f4be0 osd=1 3 2024-02-23T00:34:43.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0f4be0 osd=1 3 2024-02-23T00:34:43.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0f4be0 osd=1 3 2024-02-23T00:34:43.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 4 2024-02-23T00:34:43.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c0f4be0 osd=1 3 2024-02-23T00:34:43.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 1 4 2024-02-23T00:34:43.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0f4be0 osd=1 4 2024-02-23T00:34:43.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 10 client.37490.objecter _send_command 4 2024-02-23T00:34:43.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.151+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0b3c0fddc0 con 0x7f0b3c0f4d50 2024-02-23T00:34:43.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0b3c0fddc0 con 0x7f0b3c0f4d50 2024-02-23T00:34:43.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:43.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b397fa700 10 client.37490.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c0f4be0 osd=1 4 2024-02-23T00:34:43.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 1 4 2024-02-23T00:34:43.157 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 10 client.37490.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 5 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0b3c028590 0x7f0b3c119f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0fe1d0 osd=2 3 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0fe1d0 osd=2 3 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0fe1d0 osd=2 3 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 5 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c0fe1d0 osd=2 3 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 2 5 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0fe1d0 osd=2 4 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 10 client.37490.objecter _send_command 5 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b3c11cbd0 con 0x7f0b3c028590 2024-02-23T00:34:43.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b3b7fe700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0b3c028590 0x7f0b3c119f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b397fa700 10 client.37490.objecter ms_handle_connect 0x7f0b3c028590 2024-02-23T00:34:43.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.154+0000 7f0b3b7fe700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0b3c028590 0x7f0b3c119f80 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.155+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0b3c11cbd0 con 0x7f0b3c028590 2024-02-23T00:34:43.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.155+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:34:43.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.155+0000 7f0b397fa700 10 client.37490.objecter _finish_command 5 = system:0 2024-02-23T00:34:43.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.155+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c0fe1d0 osd=2 4 2024-02-23T00:34:43.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.155+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 2 5 2024-02-23T00:34:43.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 10 client.37490.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 6 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0fe1d0 osd=2 3 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0fe1d0 osd=2 3 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0fe1d0 osd=2 3 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 6 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c0fe1d0 osd=2 3 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 2 6 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0fe1d0 osd=2 4 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 10 client.37490.objecter _send_command 6 2024-02-23T00:34:43.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.170+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0b3c027b50 con 0x7f0b3c028590 2024-02-23T00:34:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0b3c027b50 con 0x7f0b3c028590 2024-02-23T00:34:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b397fa700 10 client.37490.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c0fe1d0 osd=2 4 2024-02-23T00:34:43.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 2 6 2024-02-23T00:34:43.174 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:34:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 10 client.37490.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 7 2024-02-23T00:34:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6813/128059790 conn(0x7f0b3c11b5f0 0x7f0b3c11ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0598f0 osd=3 3 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0598f0 osd=3 3 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0598f0 osd=3 3 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 7 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c0598f0 osd=3 3 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 3 7 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0598f0 osd=3 4 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 10 client.37490.objecter _send_command 7 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b3c059fa0 con 0x7f0b3c11b5f0 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b40d7f700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6813/128059790 conn(0x7f0b3c11b5f0 0x7f0b3c11ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b397fa700 10 client.37490.objecter ms_handle_connect 0x7f0b3c11b5f0 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.172+0000 7f0b40d7f700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6813/128059790 conn(0x7f0b3c11b5f0 0x7f0b3c11ba30 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.173+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0b3c059fa0 con 0x7f0b3c11b5f0 2024-02-23T00:34:43.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.173+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:34:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.173+0000 7f0b397fa700 10 client.37490.objecter _finish_command 7 = system:0 2024-02-23T00:34:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.173+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c0598f0 osd=3 4 2024-02-23T00:34:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.173+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 3 7 2024-02-23T00:34:43.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 10 client.37490.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 8 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0598f0 osd=3 3 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0598f0 osd=3 3 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c0598f0 osd=3 3 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 8 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c0598f0 osd=3 3 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 3 8 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0598f0 osd=3 4 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 10 client.37490.objecter _send_command 8 2024-02-23T00:34:43.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.188+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0b3c11c6f0 con 0x7f0b3c11b5f0 2024-02-23T00:34:43.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0b3c11c6f0 con 0x7f0b3c11b5f0 2024-02-23T00:34:43.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:43.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b397fa700 10 client.37490.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c0598f0 osd=3 4 2024-02-23T00:34:43.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 3 8 2024-02-23T00:34:43.192 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 10 client.37490.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 9 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0b3c11edd0 0x7f0b3c1211c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c11c6f0 osd=4 3 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c6f0 osd=4 3 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c11c6f0 osd=4 3 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 9 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c11c6f0 osd=4 3 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 4 9 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c6f0 osd=4 4 2024-02-23T00:34:43.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 10 client.37490.objecter _send_command 9 2024-02-23T00:34:43.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b3c11c900 con 0x7f0b3c11edd0 2024-02-23T00:34:43.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.189+0000 7f0b3bfff700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0b3c11edd0 0x7f0b3c1211c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.190+0000 7f0b397fa700 10 client.37490.objecter ms_handle_connect 0x7f0b3c11edd0 2024-02-23T00:34:43.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.190+0000 7f0b3bfff700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0b3c11edd0 0x7f0b3c1211c0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.190+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0b3c11c900 con 0x7f0b3c11edd0 2024-02-23T00:34:43.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.190+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:34:43.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.190+0000 7f0b397fa700 10 client.37490.objecter _finish_command 9 = system:0 2024-02-23T00:34:43.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.190+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c11c6f0 osd=4 4 2024-02-23T00:34:43.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.190+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 4 9 2024-02-23T00:34:43.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 10 client.37490.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:43.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 10 2024-02-23T00:34:43.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c11c6f0 osd=4 3 2024-02-23T00:34:43.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c6f0 osd=4 3 2024-02-23T00:34:43.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c11c6f0 osd=4 3 2024-02-23T00:34:43.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 10 2024-02-23T00:34:43.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c11c6f0 osd=4 3 2024-02-23T00:34:43.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 4 10 2024-02-23T00:34:43.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c6f0 osd=4 4 2024-02-23T00:34:43.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 10 client.37490.objecter _send_command 10 2024-02-23T00:34:43.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.205+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0b3c11c900 con 0x7f0b3c11edd0 2024-02-23T00:34:43.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0b3c11c900 con 0x7f0b3c11edd0 2024-02-23T00:34:43.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:43.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b397fa700 10 client.37490.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c11c6f0 osd=4 4 2024-02-23T00:34:43.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 4 10 2024-02-23T00:34:43.209 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:34:43.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 10 client.37490.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:43.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 11 2024-02-23T00:34:43.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0b3c121bc0 0x7f0b3c122020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c11c900 osd=5 3 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c900 osd=5 3 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c11c900 osd=5 3 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 11 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c11c900 osd=5 3 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 5 11 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c900 osd=5 4 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 10 client.37490.objecter _send_command 11 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b3c123c70 con 0x7f0b3c121bc0 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b3b7fe700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0b3c121bc0 0x7f0b3c122020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b397fa700 10 client.37490.objecter ms_handle_connect 0x7f0b3c121bc0 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.207+0000 7f0b3b7fe700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0b3c121bc0 0x7f0b3c122020 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.208+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0b3c123c70 con 0x7f0b3c121bc0 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.208+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:34:43.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.208+0000 7f0b397fa700 10 client.37490.objecter _finish_command 11 = system:0 2024-02-23T00:34:43.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.208+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c11c900 osd=5 4 2024-02-23T00:34:43.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.208+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 5 11 2024-02-23T00:34:43.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 10 client.37490.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 12 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c11c900 osd=5 3 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c900 osd=5 3 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c11c900 osd=5 3 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 12 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c11c900 osd=5 3 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 5 12 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c900 osd=5 4 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 10 client.37490.objecter _send_command 12 2024-02-23T00:34:43.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.222+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0b3c123c70 con 0x7f0b3c121bc0 2024-02-23T00:34:43.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0b3c123c70 con 0x7f0b3c121bc0 2024-02-23T00:34:43.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:43.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b397fa700 10 client.37490.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c11c900 osd=5 4 2024-02-23T00:34:43.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 5 12 2024-02-23T00:34:43.226 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:34:43.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 10 client.37490.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:43.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 13 2024-02-23T00:34:43.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0b3c126360 0x7f0b3c1267c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c123c70 osd=6 3 2024-02-23T00:34:43.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c123c70 osd=6 3 2024-02-23T00:34:43.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c123c70 osd=6 3 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 13 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c123c70 osd=6 3 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 6 13 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c123c70 osd=6 4 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 10 client.37490.objecter _send_command 13 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b3c126de0 con 0x7f0b3c126360 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.224+0000 7f0b40d7f700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0b3c126360 0x7f0b3c1267c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.225+0000 7f0b397fa700 10 client.37490.objecter ms_handle_connect 0x7f0b3c126360 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.225+0000 7f0b40d7f700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0b3c126360 0x7f0b3c1267c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.225+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0b3c126de0 con 0x7f0b3c126360 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.225+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.225+0000 7f0b397fa700 10 client.37490.objecter _finish_command 13 = system:0 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.225+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c123c70 osd=6 4 2024-02-23T00:34:43.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.225+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 6 13 2024-02-23T00:34:43.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 10 client.37490.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:43.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 14 2024-02-23T00:34:43.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c123c70 osd=6 3 2024-02-23T00:34:43.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c123c70 osd=6 3 2024-02-23T00:34:43.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c123c70 osd=6 3 2024-02-23T00:34:43.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 14 2024-02-23T00:34:43.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c123c70 osd=6 3 2024-02-23T00:34:43.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 6 14 2024-02-23T00:34:43.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c123c70 osd=6 4 2024-02-23T00:34:43.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 10 client.37490.objecter _send_command 14 2024-02-23T00:34:43.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.239+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0b3c126de0 con 0x7f0b3c126360 2024-02-23T00:34:43.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0b3c126de0 con 0x7f0b3c126360 2024-02-23T00:34:43.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:43.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b397fa700 10 client.37490.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c123c70 osd=6 4 2024-02-23T00:34:43.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 6 14 2024-02-23T00:34:43.243 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:34:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 10 client.37490.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 15 2024-02-23T00:34:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6812/373787000 conn(0x7f0b3c129630 0x7f0b3c129a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c126de0 osd=7 3 2024-02-23T00:34:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c126de0 osd=7 3 2024-02-23T00:34:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c126de0 osd=7 3 2024-02-23T00:34:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 15 2024-02-23T00:34:43.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c126de0 osd=7 3 2024-02-23T00:34:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 7 15 2024-02-23T00:34:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c126de0 osd=7 4 2024-02-23T00:34:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 10 client.37490.objecter _send_command 15 2024-02-23T00:34:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b3c12d3d0 con 0x7f0b3c129630 2024-02-23T00:34:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.241+0000 7f0b3bfff700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6812/373787000 conn(0x7f0b3c129630 0x7f0b3c129a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.242+0000 7f0b397fa700 10 client.37490.objecter ms_handle_connect 0x7f0b3c129630 2024-02-23T00:34:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.242+0000 7f0b3bfff700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6812/373787000 conn(0x7f0b3c129630 0x7f0b3c129a90 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.242+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0b3c12d3d0 con 0x7f0b3c129630 2024-02-23T00:34:43.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.243+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:34:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.243+0000 7f0b397fa700 10 client.37490.objecter _finish_command 15 = system:0 2024-02-23T00:34:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.243+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c126de0 osd=7 4 2024-02-23T00:34:43.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.243+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 7 15 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 10 client.37490.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign -1 16 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c126de0 osd=7 3 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c126de0 osd=7 3 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 20 client.37490.objecter _get_session s=0x7f0b3c126de0 osd=7 3 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_remove -1 16 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 20 client.37490.objecter get_session s=0x7f0b3c126de0 osd=7 3 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 15 client.37490.objecter _session_command_op_assign 7 16 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c126de0 osd=7 4 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 10 client.37490.objecter _send_command 16 2024-02-23T00:34:43.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.257+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0b3c1225d0 con 0x7f0b3c129630 2024-02-23T00:34:43.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.259+0000 7f0b397fa700 1 -- 172.21.15.87:0/16199064 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f0b3c1225d0 con 0x7f0b3c129630 2024-02-23T00:34:43.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.259+0000 7f0b397fa700 10 client.37490.objecter ms_dispatch 0x7f0b3c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:43.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.259+0000 7f0b397fa700 10 client.37490.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.259+0000 7f0b397fa700 20 client.37490.objecter put_session s=0x7f0b3c126de0 osd=7 4 2024-02-23T00:34:43.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.259+0000 7f0b397fa700 15 client.37490.objecter _session_command_op_remove 7 16 2024-02-23T00:34:43.262 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:34:43.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 librados: watch_flush enter 2024-02-23T00:34:43.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 librados: watch_flush exit 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 client.37490.objecter close_session for osd.0 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0b3c061380 msgr2=0x7f0b3c061760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6801/2153191863 conn(0x7f0b3c061380 0x7f0b3c061760 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c05ee80 osd=0 1 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 client.37490.objecter close_session for osd.1 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6805/685354171 conn(0x7f0b3c0f4d50 msgr2=0x7f0b3c0273c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6805/685354171 conn(0x7f0b3c0f4d50 0x7f0b3c0273c0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0f4be0 osd=1 1 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 client.37490.objecter close_session for osd.2 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0b3c028590 msgr2=0x7f0b3c119f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0b3c028590 0x7f0b3c119f80 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0fe1d0 osd=2 1 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 client.37490.objecter close_session for osd.3 2024-02-23T00:34:43.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6813/128059790 conn(0x7f0b3c11b5f0 msgr2=0x7f0b3c11ba30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b40d7f700 1 -- 172.21.15.87:0/16199064 reap_dead start 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6813/128059790 conn(0x7f0b3c11b5f0 0x7f0b3c11ba30 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c0598f0 osd=3 1 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 client.37490.objecter close_session for osd.4 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0b3c11edd0 msgr2=0x7f0b3c1211c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0b3c11edd0 0x7f0b3c1211c0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c6f0 osd=4 1 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 client.37490.objecter close_session for osd.5 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0b3c121bc0 msgr2=0x7f0b3c122020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0b3c121bc0 0x7f0b3c122020 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c11c900 osd=5 1 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 client.37490.objecter close_session for osd.6 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0b3c126360 msgr2=0x7f0b3c1267c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0b3c126360 0x7f0b3c1267c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c123c70 osd=6 1 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 client.37490.objecter close_session for osd.7 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6812/373787000 conn(0x7f0b3c129630 msgr2=0x7f0b3c129a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.171:6812/373787000 conn(0x7f0b3c129630 0x7f0b3c129a90 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 20 client.37490.objecter put_session s=0x7f0b3c126de0 osd=7 1 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 20 client.37490.objecter shutdown clearing up homeless session... 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 10 client.37490.objecter successfully canceled tick 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b40d7f700 1 -- 172.21.15.87:0/16199064 reap_dead start 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0b24062c90 msgr2=0x7f0b24065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0b24062c90 0x7f0b24065150 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 rx=0x7f0b30004ee0 tx=0x7f0b3000b040).stop 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0f7400 msgr2=0x7f0b3c10d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0f7400 0x7f0b3c10d120 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7f0b2c009400 tx=0x7f0b2c009cb0).stop 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 shutdown_connections 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0b24062c90 0x7f0b24065150 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 --2- 172.21.15.87:0/16199064 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0b3c0f7400 0x7f0b3c10d120 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:43.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 >> 172.21.15.87:0/16199064 conn(0x7f0b3c0f2820 msgr2=0x7f0b3c0f4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 shutdown_connections 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 -- 172.21.15.87:0/16199064 wait complete. 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:43.260+0000 7f0b427e2700 1 librados: shutdown 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.267 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:34:43.578 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:41.623343+0000 mgr.y (mgr.24437) 324 : cluster [DBG] pgmap v478: 245 pgs: 8 unknown, 90 peering, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 14 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-02-23T00:34:43.578 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:42.329010+0000 mon.a (mon.0) 2222 : cluster [WRN] Health check failed: Reduced data availability: 18 pgs inactive, 27 pgs peering (PG_AVAILABILITY) 2024-02-23T00:34:43.578 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:43.054005+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-02-23T00:34:43.578 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:43.060197+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:43.578 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:43.106603+0000 mon.c (mon.2) 377 : audit [DBG] from='client.? 172.21.15.87:0/16199064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:43.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:41.623343+0000 mgr.y (mgr.24437) 324 : cluster [DBG] pgmap v478: 245 pgs: 8 unknown, 90 peering, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 14 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-02-23T00:34:43.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:42.329010+0000 mon.a (mon.0) 2222 : cluster [WRN] Health check failed: Reduced data availability: 18 pgs inactive, 27 pgs peering (PG_AVAILABILITY) 2024-02-23T00:34:43.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:43.054005+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-02-23T00:34:43.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:43.060197+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:43.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:43.106603+0000 mon.c (mon.2) 377 : audit [DBG] from='client.? 172.21.15.87:0/16199064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:43.595 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:43.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:41.623343+0000 mgr.y (mgr.24437) 324 : cluster [DBG] pgmap v478: 245 pgs: 8 unknown, 90 peering, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 14 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-02-23T00:34:43.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:42.329010+0000 mon.a (mon.0) 2222 : cluster [WRN] Health check failed: Reduced data availability: 18 pgs inactive, 27 pgs peering (PG_AVAILABILITY) 2024-02-23T00:34:43.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:43.054005+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-02-23T00:34:43.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:43.060197+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:43.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:43.106603+0000 mon.c (mon.2) 377 : audit [DBG] from='client.? 172.21.15.87:0/16199064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:43.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:43.699 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:43.726 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:43.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:34:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:34:44.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 -- 172.21.15.87:0/2329959792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f782c0fdbb0 msgr2=0x7f782c0fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2329959792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f782c0fdbb0 0x7f782c0fdf90 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 rx=0x7f781c004870 tx=0x7f781c009cb0).stop 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 -- 172.21.15.87:0/2329959792 shutdown_connections 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2329959792 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f782c0f8420 0x7f782c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2329959792 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f782c0fdbb0 0x7f782c0fdf90 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2329959792 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f782c0fe510 0x7f782c0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 -- 172.21.15.87:0/2329959792 >> 172.21.15.87:0/2329959792 conn(0x7f782c0f3800 msgr2=0x7f782c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 -- 172.21.15.87:0/2329959792 shutdown_connections 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 -- 172.21.15.87:0/2329959792 wait complete. 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 librados: starting msgr at 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 librados: starting objecter 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.084+0000 7f7831cfd700 1 Processor -- start 2024-02-23T00:34:44.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7831cfd700 1 -- start start 2024-02-23T00:34:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7831cfd700 1 librados: setting wanted keys 2024-02-23T00:34:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7831cfd700 1 librados: calling monclient init 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7831cfd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f782c0f8420 0x7f782c06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7831cfd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f782c0fdbb0 0x7f782c06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7831cfd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f782c0fe510 0x7f782c06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7831cfd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c116540 con 0x7f782c0fe510 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7831cfd700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c116680 con 0x7f782c0f8420 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7831cfd700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c116840 con 0x7f782c0fdbb0 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f782c0fdbb0 0x7f782c06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f782c0fdbb0 0x7f782c06b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34866/0 (socket says 172.21.15.87:34866) 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782affd700 1 -- 172.21.15.87:0/2872474221 learned_addr learned my addr 172.21.15.87:0/2872474221 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782b7fe700 1 --2- 172.21.15.87:0/2872474221 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f782c0f8420 0x7f782c06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782bfff700 1 --2- 172.21.15.87:0/2872474221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f782c0fe510 0x7f782c06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782affd700 1 -- 172.21.15.87:0/2872474221 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f782c0f8420 msgr2=0x7f782c06b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782affd700 1 --2- 172.21.15.87:0/2872474221 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f782c0f8420 0x7f782c06b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782affd700 1 -- 172.21.15.87:0/2872474221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f782c0fe510 msgr2=0x7f782c06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782affd700 1 --2- 172.21.15.87:0/2872474221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f782c0fe510 0x7f782c06bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782affd700 1 -- 172.21.15.87:0/2872474221 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7820005040 con 0x7f782c0fdbb0 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f782affd700 1 --2- 172.21.15.87:0/2872474221 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f782c0fdbb0 0x7f782c06b5c0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7f78200048f0 tx=0x7f7820009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7828ff9700 10 client.?.objecter ms_handle_connect 0x7f782c0fdbb0 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.085+0000 7f7828ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.086+0000 7f7828ff9700 1 -- 172.21.15.87:0/2872474221 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820007b00 con 0x7f782c0fdbb0 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.086+0000 7f7828ff9700 1 -- 172.21.15.87:0/2872474221 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f782000f040 con 0x7f782c0fdbb0 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.086+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781c005040 con 0x7f782c0fdbb0 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.086+0000 7f7828ff9700 1 -- 172.21.15.87:0/2872474221 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7820014990 con 0x7f782c0fdbb0 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.086+0000 7f7831cfd700 10 client.37496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.086+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c116f00 con 0x7f782c0fdbb0 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.086+0000 7f7831cfd700 1 librados: init done 2024-02-23T00:34:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.087+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c05ee80 con 0x7f782c0fdbb0 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.087+0000 7f7828ff9700 1 -- 172.21.15.87:0/2872474221 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7820010500 con 0x7f782c0fdbb0 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.087+0000 7f7828ff9700 1 --2- 172.21.15.87:0/2872474221 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7814062c90 0x7f7814065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.087+0000 7f7828ff9700 1 -- 172.21.15.87:0/2872474221 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(333..333 src has 173..333) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f782007ce00 con 0x7f782c0fdbb0 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.087+0000 7f7828ff9700 10 client.37496.objecter ms_dispatch 0x7f782c001550 osd_map(333..333 src has 173..333) v4 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.087+0000 7f782b7fe700 1 --2- 172.21.15.87:0/2872474221 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7814062c90 0x7f7814065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.087+0000 7f7828ff9700 3 client.37496.objecter handle_osd_map got epochs [333,333] > 0 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.087+0000 7f7828ff9700 3 client.37496.objecter handle_osd_map decoding full epoch 333 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.088+0000 7f7828ff9700 20 client.37496.objecter dump_active .. 0 homeless 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.088+0000 7f7828ff9700 10 client.37496.objecter ms_handle_connect 0x7f7814062c90 2024-02-23T00:34:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.088+0000 7f782b7fe700 1 --2- 172.21.15.87:0/2872474221 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7814062c90 0x7f7814065150 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f781c0043d0 tx=0x7f781c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.092+0000 7f7828ff9700 1 -- 172.21.15.87:0/2872474221 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7820049f60 con 0x7f782c0fdbb0 2024-02-23T00:34:44.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:44.274+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f782c05fa60 con 0x7f782c0fdbb0 2024-02-23T00:34:44.376 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:44.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:44.479 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:44.510 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:43.623848+0000 mgr.y (mgr.24437) 325 : cluster [DBG] pgmap v481: 236 pgs: 89 peering, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 15 KiB/s rd, 1.5 KiB/s wr, 17 op/s; 27 B/s, 0 objects/s recovering 2024-02-23T00:34:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:43.624761+0000 mon.a (mon.0) 2225 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-23T00:34:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:44.052765+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi087-82019-57"}]': finished 2024-02-23T00:34:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:44.052836+0000 mon.a (mon.0) 2227 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-23T00:34:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:44.052886+0000 mon.a (mon.0) 2228 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-02-23T00:34:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:44.054216+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:44.276395+0000 mon.c (mon.2) 378 : audit [INF] from='client.? 172.21.15.87:0/2872474221' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:44.276691+0000 mon.a (mon.0) 2230 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:44.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:43.623848+0000 mgr.y (mgr.24437) 325 : cluster [DBG] pgmap v481: 236 pgs: 89 peering, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 15 KiB/s rd, 1.5 KiB/s wr, 17 op/s; 27 B/s, 0 objects/s recovering 2024-02-23T00:34:44.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:43.624761+0000 mon.a (mon.0) 2225 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-23T00:34:44.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:44.052765+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi087-82019-57"}]': finished 2024-02-23T00:34:44.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:44.052836+0000 mon.a (mon.0) 2227 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-23T00:34:44.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:44.052886+0000 mon.a (mon.0) 2228 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-02-23T00:34:44.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:44.054216+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:44.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:44.276395+0000 mon.c (mon.2) 378 : audit [INF] from='client.? 172.21.15.87:0/2872474221' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:44.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:44.276691+0000 mon.a (mon.0) 2230 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:44.593 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:34:44.593 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:34:44.593 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:34:44.593 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:34:44.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:43.623848+0000 mgr.y (mgr.24437) 325 : cluster [DBG] pgmap v481: 236 pgs: 89 peering, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 15 KiB/s rd, 1.5 KiB/s wr, 17 op/s; 27 B/s, 0 objects/s recovering 2024-02-23T00:34:44.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:43.624761+0000 mon.a (mon.0) 2225 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-23T00:34:44.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:44.052765+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi087-82019-57"}]': finished 2024-02-23T00:34:44.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:44.052836+0000 mon.a (mon.0) 2227 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-23T00:34:44.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:44.052886+0000 mon.a (mon.0) 2228 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-02-23T00:34:44.612 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:44.054216+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi087-82019-57"}]: dispatch 2024-02-23T00:34:44.613 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:44.276395+0000 mon.c (mon.2) 378 : audit [INF] from='client.? 172.21.15.87:0/2872474221' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:44.613 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:44.276691+0000 mon.a (mon.0) 2230 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:34:44.632 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:44.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:44.737 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:44.763 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:44.782 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:44.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:44.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:44.886 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:44.912 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:44.930 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:44.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:45.036 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:45.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.053+0000 7f7828ff9700 1 -- 172.21.15.87:0/2872474221 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v334) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7820050210 con 0x7f782c0fdbb0 2024-02-23T00:34:45.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.057+0000 7f7831cfd700 10 librados: watch_flush enter 2024-02-23T00:34:45.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.057+0000 7f7831cfd700 10 librados: watch_flush exit 2024-02-23T00:34:45.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.057+0000 7f7831cfd700 20 client.37496.objecter shutdown clearing up homeless session... 2024-02-23T00:34:45.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.057+0000 7f7831cfd700 10 client.37496.objecter successfully canceled tick 2024-02-23T00:34:45.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.057+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7814062c90 msgr2=0x7f7814065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:45.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.057+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2872474221 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7814062c90 0x7f7814065150 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 rx=0x7f781c0043d0 tx=0x7f781c00b040).stop 2024-02-23T00:34:45.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.057+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f782c0fdbb0 msgr2=0x7f782c06b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:45.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.057+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2872474221 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f782c0fdbb0 0x7f782c06b5c0 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7f78200048f0 tx=0x7f7820009cb0).stop 2024-02-23T00:34:45.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.060+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 shutdown_connections 2024-02-23T00:34:45.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.060+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2872474221 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7814062c90 0x7f7814065150 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:45.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.060+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2872474221 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f782c0fe510 0x7f782c06bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:45.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.060+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2872474221 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f782c0f8420 0x7f782c06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:45.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.060+0000 7f7831cfd700 1 --2- 172.21.15.87:0/2872474221 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f782c0fdbb0 0x7f782c06b5c0 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:45.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.060+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 >> 172.21.15.87:0/2872474221 conn(0x7f782c0f3800 msgr2=0x7f782c0fb940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:45.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.060+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 shutdown_connections 2024-02-23T00:34:45.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.060+0000 7f7831cfd700 1 -- 172.21.15.87:0/2872474221 wait complete. 2024-02-23T00:34:45.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:45.060+0000 7f7831cfd700 1 librados: shutdown 2024-02-23T00:34:45.063 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:34:45.069 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:45.254 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-23T00:34:45.268 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:45.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:45.372 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:45.399 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:45.473 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:45.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:45.538 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:45.581 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:45.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:45.050040+0000 mon.a (mon.0) 2231 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:45.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:45.054029+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi087-82019-57"}]': finished 2024-02-23T00:34:45.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:45.054191+0000 mon.a (mon.0) 2233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:34:45.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:45.054248+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-02-23T00:34:45.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:45.075994+0000 mon.c (mon.2) 379 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:45.076279+0000 mon.a (mon.0) 2235 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:45.076941+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:45.077180+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:45.077654+0000 mon.c (mon.2) 381 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:45.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:45.077872+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:45.607 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:45.807 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:45.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:45.050040+0000 mon.a (mon.0) 2231 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:45.054029+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi087-82019-57"}]': finished 2024-02-23T00:34:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:45.054191+0000 mon.a (mon.0) 2233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:34:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:45.054248+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-02-23T00:34:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:45.075994+0000 mon.c (mon.2) 379 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:45.076279+0000 mon.a (mon.0) 2235 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:45.076941+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:45.077180+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:45.077654+0000 mon.c (mon.2) 381 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:45.077872+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:45.050040+0000 mon.a (mon.0) 2231 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:45.054029+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? 172.21.15.87:0/1129996568' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi087-82019-57"}]': finished 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:45.054191+0000 mon.a (mon.0) 2233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:45.054248+0000 mon.a (mon.0) 2234 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:45.075994+0000 mon.c (mon.2) 379 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:45.076279+0000 mon.a (mon.0) 2235 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:45.076941+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:45.077180+0000 mon.a (mon.0) 2236 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:45.077654+0000 mon.c (mon.2) 381 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:45.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:45.077872+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:45.910 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:45.937 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:46.009 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:46.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:46.113 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:46.141 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:46.339 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:46.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:46.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:46.443 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:46.471 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:46.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:34:46.819 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:46.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:46.922 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:46.924 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:46.952 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:47.200 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:47.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:47.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:47.305 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:47.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:45.624743+0000 mgr.y (mgr.24437) 326 : cluster [DBG] pgmap v484: 236 pgs: 1 active+remapped+backfill_toofull, 2 clean+premerge+peered, 233 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 13/1056 objects misplaced (1.231%); 977 KiB/s, 51 objects/s recovering 2024-02-23T00:34:47.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:46.056289+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:47.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:46.056385+0000 mon.a (mon.0) 2239 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-02-23T00:34:47.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:46.056979+0000 mon.c (mon.2) 382 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi087-82019-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:47.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:46.057279+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi087-82019-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:47.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:46.344899+0000 mon.a (mon.0) 2241 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:34:47.332 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:46.344941+0000 mon.a (mon.0) 2242 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 18 pgs inactive, 27 pgs peering) 2024-02-23T00:34:47.334 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:47.357 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:47.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:47.364 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:34:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:34:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:34:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:34:47.364 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:34:47.460 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:47.489 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:45.624743+0000 mgr.y (mgr.24437) 326 : cluster [DBG] pgmap v484: 236 pgs: 1 active+remapped+backfill_toofull, 2 clean+premerge+peered, 233 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 13/1056 objects misplaced (1.231%); 977 KiB/s, 51 objects/s recovering 2024-02-23T00:34:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:46.056289+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:46.056385+0000 mon.a (mon.0) 2239 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-02-23T00:34:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:46.056979+0000 mon.c (mon.2) 382 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi087-82019-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:46.057279+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi087-82019-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:46.344899+0000 mon.a (mon.0) 2241 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:34:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:46.344941+0000 mon.a (mon.0) 2242 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 18 pgs inactive, 27 pgs peering) 2024-02-23T00:34:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:45.624743+0000 mgr.y (mgr.24437) 326 : cluster [DBG] pgmap v484: 236 pgs: 1 active+remapped+backfill_toofull, 2 clean+premerge+peered, 233 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 13/1056 objects misplaced (1.231%); 977 KiB/s, 51 objects/s recovering 2024-02-23T00:34:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:46.056289+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:46.056385+0000 mon.a (mon.0) 2239 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-02-23T00:34:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:46.056979+0000 mon.c (mon.2) 382 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi087-82019-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:46.057279+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi087-82019-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:46.344899+0000 mon.a (mon.0) 2241 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:34:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:46.344941+0000 mon.a (mon.0) 2242 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 18 pgs inactive, 27 pgs peering) 2024-02-23T00:34:47.586 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:47.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:47.690 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:47.717 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:47.983 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:34:48.048 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-23T00:34:48.069 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:48.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:48.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:48.201 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:48.325 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:48.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:48.336 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:48.429 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:47.159361+0000 mon.b (mon.1) 290 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:47.301745+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:47.302617+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:47.303433+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43"}]: dispatch 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:47.307739+0000 mon.a (mon.0) 2245 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43"}]: dispatch 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:47.626119+0000 mon.a (mon.0) 2246 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:48.060576+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi087-82019-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]': finished 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:48.060677+0000 mon.a (mon.0) 2248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43"}]': finished 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:48.060755+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:48.060818+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-02-23T00:34:48.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:47.159361+0000 mon.b (mon.1) 290 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:48.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:47.301745+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-02-23T00:34:48.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:47.302617+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:48.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:47.303433+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43"}]: dispatch 2024-02-23T00:34:48.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:47.307739+0000 mon.a (mon.0) 2245 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43"}]: dispatch 2024-02-23T00:34:48.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:47.626119+0000 mon.a (mon.0) 2246 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-23T00:34:48.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:48.060576+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi087-82019-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]': finished 2024-02-23T00:34:48.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:48.060677+0000 mon.a (mon.0) 2248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43"}]': finished 2024-02-23T00:34:48.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:48.060755+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-23T00:34:48.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:48.060818+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-02-23T00:34:48.457 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:48.577 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:47.159361+0000 mon.b (mon.1) 290 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:48.577 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:47.301745+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-02-23T00:34:48.577 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:47.302617+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:34:48.577 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:47.303433+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43"}]: dispatch 2024-02-23T00:34:48.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:47.307739+0000 mon.a (mon.0) 2245 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43"}]: dispatch 2024-02-23T00:34:48.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:47.626119+0000 mon.a (mon.0) 2246 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-23T00:34:48.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:48.060576+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi087-82019-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]': finished 2024-02-23T00:34:48.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:48.060677+0000 mon.a (mon.0) 2248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-43"}]': finished 2024-02-23T00:34:48.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:48.060755+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-23T00:34:48.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:48.060818+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-02-23T00:34:48.601 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:34:48.602 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:34:48.602 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-23T00:34:48.602 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-02-23T00:34:48.602 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-23T00:34:48.655 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:34:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:48.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:34:48.758 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:48.786 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:48.803 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:48.805 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:48.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:48.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 -- 172.21.15.87:0/1965559837 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fd420 msgr2=0x7f7acc0f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 --2- 172.21.15.87:0/1965559837 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fd420 0x7f7acc0f6c50 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f7ac0004870 tx=0x7f7ac0009cb0).stop 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 -- 172.21.15.87:0/1965559837 shutdown_connections 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 --2- 172.21.15.87:0/1965559837 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc0f7330 0x7f7acc103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 --2- 172.21.15.87:0/1965559837 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7acc0fcac0 0x7f7acc0fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 --2- 172.21.15.87:0/1965559837 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fd420 0x7f7acc0f6c50 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 -- 172.21.15.87:0/1965559837 >> 172.21.15.87:0/1965559837 conn(0x7f7acc0f26f0 msgr2=0x7f7acc0f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 -- 172.21.15.87:0/1965559837 shutdown_connections 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 -- 172.21.15.87:0/1965559837 wait complete. 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 librados: starting msgr at 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 librados: starting objecter 2024-02-23T00:34:48.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.896+0000 7f7ad1547700 1 Processor -- start 2024-02-23T00:34:48.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7ad1547700 1 -- start start 2024-02-23T00:34:48.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7ad1547700 1 librados: setting wanted keys 2024-02-23T00:34:48.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7ad1547700 1 librados: calling monclient init 2024-02-23T00:34:48.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7ad1547700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7acc0f7330 0x7f7acc109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7ad1547700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fcac0 0x7f7acc109660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7ad1547700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc0fd420 0x7f7acc109c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7ad1547700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc114360 con 0x7f7acc0fd420 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7ad1547700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc1144d0 con 0x7f7acc0f7330 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7ad1547700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc114690 con 0x7f7acc0fcac0 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7aca7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fcac0 0x7f7acc109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7aca7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fcac0 0x7f7acc109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34902/0 (socket says 172.21.15.87:34902) 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7aca7fc700 1 -- 172.21.15.87:0/3263419833 learned_addr learned my addr 172.21.15.87:0/3263419833 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7acb7fe700 1 --2- 172.21.15.87:0/3263419833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc0fd420 0x7f7acc109c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7acaffd700 1 --2- 172.21.15.87:0/3263419833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7acc0f7330 0x7f7acc109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7aca7fc700 1 -- 172.21.15.87:0/3263419833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7acc0f7330 msgr2=0x7f7acc109170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7aca7fc700 1 --2- 172.21.15.87:0/3263419833 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7acc0f7330 0x7f7acc109170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7aca7fc700 1 -- 172.21.15.87:0/3263419833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc0fd420 msgr2=0x7f7acc109c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:48.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7aca7fc700 1 --2- 172.21.15.87:0/3263419833 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7acc0fd420 0x7f7acc109c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.897+0000 7f7aca7fc700 1 -- 172.21.15.87:0/3263419833 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac40097c0 con 0x7f7acc0fcac0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7aca7fc700 1 --2- 172.21.15.87:0/3263419833 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fcac0 0x7f7acc109660 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f7ac00043a0 tx=0x7f7ac0009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7ab3fff700 10 client.?.objecter ms_handle_connect 0x7f7acc0fcac0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7ab3fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac000e030 con 0x7f7acc0fcac0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac0005040 con 0x7f7acc0fcac0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ac0013da0 con 0x7f7acc0fcac0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0018070 con 0x7f7acc0fcac0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7ad1547700 10 client.37508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc114c60 con 0x7f7acc0fcac0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.898+0000 7f7ad1547700 1 librados: init done 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.899+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7acc05f990 con 0x7f7acc0fcac0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.899+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7ac001d420 con 0x7f7acc0fcac0 2024-02-23T00:34:48.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ab3fff700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7ab4062c90 0x7f7ab4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(337..337 src has 173..337) v4 ==== 8947+0+0 (secure 0 0 0) 0x7f7ac007e830 con 0x7f7acc0fcac0 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 osd_map(337..337 src has 173..337) v4 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ab3fff700 3 client.37508.objecter handle_osd_map got epochs [337,337] > 0 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ab3fff700 3 client.37508.objecter handle_osd_map decoding full epoch 337 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7acaffd700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7ab4062c90 0x7f7ab4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ab3fff700 20 client.37508.objecter dump_active .. 0 homeless 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v337) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7ac007eb90 con 0x7f7acc0fcac0 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ab3fff700 10 client.37508.objecter ms_handle_connect 0x7f7ab4062c90 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7acaffd700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7ab4062c90 0x7f7ab4065150 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7f7abc004de0 tx=0x7f7abc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 10 client.37508.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 1 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7acc0fdd10 0x7f7acc0fe0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc05ee80 osd=0 3 2024-02-23T00:34:48.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc05ee80 osd=0 3 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc05ee80 osd=0 3 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 1 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc05ee80 osd=0 3 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 0 1 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc05ee80 osd=0 4 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 10 client.37508.objecter _send_command 1 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7acc106000 con 0x7f7acc0fdd10 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.900+0000 7f7acb7fe700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7acc0fdd10 0x7f7acc0fe0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.901+0000 7f7ab3fff700 10 client.37508.objecter ms_handle_connect 0x7f7acc0fdd10 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.901+0000 7f7acb7fe700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7acc0fdd10 0x7f7acc0fe0f0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:48.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.901+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7acc106000 con 0x7f7acc0fdd10 2024-02-23T00:34:48.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.901+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:34:48.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.901+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 1 = system:0 2024-02-23T00:34:48.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.901+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc05ee80 osd=0 4 2024-02-23T00:34:48.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.901+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 0 1 2024-02-23T00:34:48.908 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 10 client.37508.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 2 2024-02-23T00:34:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc05ee80 osd=0 3 2024-02-23T00:34:48.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc05ee80 osd=0 3 2024-02-23T00:34:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc05ee80 osd=0 3 2024-02-23T00:34:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 2 2024-02-23T00:34:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc05ee80 osd=0 3 2024-02-23T00:34:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 0 2 2024-02-23T00:34:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc05ee80 osd=0 4 2024-02-23T00:34:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 10 client.37508.objecter _send_command 2 2024-02-23T00:34:48.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.917+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7acc0fe820 con 0x7f7acc0fdd10 2024-02-23T00:34:48.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7ac400f040 con 0x7f7acc0fdd10 2024-02-23T00:34:48.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:48.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:48.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc05ee80 osd=0 4 2024-02-23T00:34:48.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 0 2 2024-02-23T00:34:48.922 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:34:48.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 10 client.37508.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:48.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 3 2024-02-23T00:34:48.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6805/685354171 conn(0x7f7acc0598f0 0x7f7acc059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc028540 osd=1 3 2024-02-23T00:34:48.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc028540 osd=1 3 2024-02-23T00:34:48.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc028540 osd=1 3 2024-02-23T00:34:48.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 3 2024-02-23T00:34:48.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc028540 osd=1 3 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 1 3 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc028540 osd=1 4 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 10 client.37508.objecter _send_command 3 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7acc0286b0 con 0x7f7acc0598f0 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7acaffd700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6805/685354171 conn(0x7f7acc0598f0 0x7f7acc059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7ab3fff700 10 client.37508.objecter ms_handle_connect 0x7f7acc0598f0 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.920+0000 7f7acaffd700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6805/685354171 conn(0x7f7acc0598f0 0x7f7acc059cd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.921+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7acc0286b0 con 0x7f7acc0598f0 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.921+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.921+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 3 = system:0 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.921+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc028540 osd=1 4 2024-02-23T00:34:48.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.921+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 1 3 2024-02-23T00:34:48.937 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 10 client.37508.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 4 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc028540 osd=1 3 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc028540 osd=1 3 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc028540 osd=1 3 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 4 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc028540 osd=1 3 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 1 4 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc028540 osd=1 4 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 10 client.37508.objecter _send_command 4 2024-02-23T00:34:48.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.938+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7acc059360 con 0x7f7acc0598f0 2024-02-23T00:34:48.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.939+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7acc059360 con 0x7f7acc0598f0 2024-02-23T00:34:48.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.939+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:48.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.939+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:48.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.939+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc028540 osd=1 4 2024-02-23T00:34:48.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.939+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 1 4 2024-02-23T00:34:48.942 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 10 client.37508.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 5 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7acc11a020 0x7f7acc11c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc027b90 osd=2 3 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc027b90 osd=2 3 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc027b90 osd=2 3 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 5 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc027b90 osd=2 3 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 2 5 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc027b90 osd=2 4 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 10 client.37508.objecter _send_command 5 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7acc0282a0 con 0x7f7acc11a020 2024-02-23T00:34:48.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7aca7fc700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7acc11a020 0x7f7acc11c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7ab3fff700 10 client.37508.objecter ms_handle_connect 0x7f7acc11a020 2024-02-23T00:34:48.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.940+0000 7f7aca7fc700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7acc11a020 0x7f7acc11c410 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:48.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.941+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7acc0282a0 con 0x7f7acc11a020 2024-02-23T00:34:48.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.941+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:34:48.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.941+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 5 = system:0 2024-02-23T00:34:48.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.941+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc027b90 osd=2 4 2024-02-23T00:34:48.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.941+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 2 5 2024-02-23T00:34:48.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 10 client.37508.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:48.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 6 2024-02-23T00:34:48.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc027b90 osd=2 3 2024-02-23T00:34:48.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc027b90 osd=2 3 2024-02-23T00:34:48.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc027b90 osd=2 3 2024-02-23T00:34:48.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 6 2024-02-23T00:34:48.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc027b90 osd=2 3 2024-02-23T00:34:48.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 2 6 2024-02-23T00:34:48.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc027b90 osd=2 4 2024-02-23T00:34:48.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 10 client.37508.objecter _send_command 6 2024-02-23T00:34:48.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.957+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7acc027600 con 0x7f7acc11a020 2024-02-23T00:34:48.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7acc027600 con 0x7f7acc11a020 2024-02-23T00:34:48.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:48.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:48.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc027b90 osd=2 4 2024-02-23T00:34:48.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 2 6 2024-02-23T00:34:48.961 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:34:48.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 10 client.37508.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:48.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 7 2024-02-23T00:34:48.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6813/128059790 conn(0x7f7acc11da80 0x7f7acc11fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc119670 osd=3 3 2024-02-23T00:34:48.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc119670 osd=3 3 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc119670 osd=3 3 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 7 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc119670 osd=3 3 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 3 7 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc119670 osd=3 4 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 10 client.37508.objecter _send_command 7 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7acc119d60 con 0x7f7acc11da80 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7acb7fe700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6813/128059790 conn(0x7f7acc11da80 0x7f7acc11fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7ab3fff700 10 client.37508.objecter ms_handle_connect 0x7f7acc11da80 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.959+0000 7f7acb7fe700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6813/128059790 conn(0x7f7acc11da80 0x7f7acc11fe70 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.960+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7acc119d60 con 0x7f7acc11da80 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.960+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.960+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 7 = system:0 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.960+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc119670 osd=3 4 2024-02-23T00:34:48.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.960+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 3 7 2024-02-23T00:34:48.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 10 client.37508.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:48.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 8 2024-02-23T00:34:48.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc119670 osd=3 3 2024-02-23T00:34:48.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc119670 osd=3 3 2024-02-23T00:34:48.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc119670 osd=3 3 2024-02-23T00:34:48.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 8 2024-02-23T00:34:48.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc119670 osd=3 3 2024-02-23T00:34:48.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 3 8 2024-02-23T00:34:48.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc119670 osd=3 4 2024-02-23T00:34:48.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 10 client.37508.objecter _send_command 8 2024-02-23T00:34:48.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.974+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7acc1190e0 con 0x7f7acc11da80 2024-02-23T00:34:48.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7acc1190e0 con 0x7f7acc11da80 2024-02-23T00:34:48.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:48.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:48.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc119670 osd=3 4 2024-02-23T00:34:48.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 3 8 2024-02-23T00:34:48.978 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:34:48.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 10 client.37508.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:48.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 9 2024-02-23T00:34:48.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7acc119280 0x7f7acc1234f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc1190e0 osd=4 3 2024-02-23T00:34:48.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc1190e0 osd=4 3 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc1190e0 osd=4 3 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 9 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc1190e0 osd=4 3 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 4 9 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc1190e0 osd=4 4 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 10 client.37508.objecter _send_command 9 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7acc11d7b0 con 0x7f7acc119280 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.976+0000 7f7acaffd700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7acc119280 0x7f7acc1234f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.977+0000 7f7ab3fff700 10 client.37508.objecter ms_handle_connect 0x7f7acc119280 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.977+0000 7f7acaffd700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7acc119280 0x7f7acc1234f0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.977+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7acc11d7b0 con 0x7f7acc119280 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.977+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.977+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 9 = system:0 2024-02-23T00:34:48.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.977+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc1190e0 osd=4 4 2024-02-23T00:34:48.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.977+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 4 9 2024-02-23T00:34:48.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 10 client.37508.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:48.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 10 2024-02-23T00:34:48.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc1190e0 osd=4 3 2024-02-23T00:34:48.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc1190e0 osd=4 3 2024-02-23T00:34:48.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc1190e0 osd=4 3 2024-02-23T00:34:48.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 10 2024-02-23T00:34:48.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc1190e0 osd=4 3 2024-02-23T00:34:48.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 4 10 2024-02-23T00:34:48.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc1190e0 osd=4 4 2024-02-23T00:34:48.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 10 client.37508.objecter _send_command 10 2024-02-23T00:34:48.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.991+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7acc11d7b0 con 0x7f7acc119280 2024-02-23T00:34:48.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7acc11d7b0 con 0x7f7acc119280 2024-02-23T00:34:48.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:48.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:48.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc1190e0 osd=4 4 2024-02-23T00:34:48.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 4 10 2024-02-23T00:34:48.996 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:34:48.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 10 client.37508.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:48.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 11 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7acc123ef0 0x7f7acc124330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc11d7b0 osd=5 3 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc11d7b0 osd=5 3 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc11d7b0 osd=5 3 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 11 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc11d7b0 osd=5 3 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 5 11 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc11d7b0 osd=5 4 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 10 client.37508.objecter _send_command 11 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7acc125f80 con 0x7f7acc123ef0 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7aca7fc700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7acc123ef0 0x7f7acc124330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 10 client.37508.objecter ms_handle_connect 0x7f7acc123ef0 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7aca7fc700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7acc123ef0 0x7f7acc124330 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7acc125f80 con 0x7f7acc123ef0 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:34:48.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 11 = system:0 2024-02-23T00:34:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc11d7b0 osd=5 4 2024-02-23T00:34:48.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:48.994+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 5 11 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 10 client.37508.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 12 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc11d7b0 osd=5 3 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc11d7b0 osd=5 3 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc11d7b0 osd=5 3 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 12 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc11d7b0 osd=5 3 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 5 12 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc11d7b0 osd=5 4 2024-02-23T00:34:49.011 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 10 client.37508.objecter _send_command 12 2024-02-23T00:34:49.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.009+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7acc125f80 con 0x7f7acc123ef0 2024-02-23T00:34:49.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.010+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7acc125f80 con 0x7f7acc123ef0 2024-02-23T00:34:49.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.010+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:49.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.010+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.010+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc11d7b0 osd=5 4 2024-02-23T00:34:49.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.010+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 5 12 2024-02-23T00:34:49.013 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.010+0000 7f7ad1547700 10 client.37508.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 13 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7acc128690 0x7f7acc128ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc125f80 osd=6 3 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc125f80 osd=6 3 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc125f80 osd=6 3 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 13 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc125f80 osd=6 3 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 6 13 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc125f80 osd=6 4 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 10 client.37508.objecter _send_command 13 2024-02-23T00:34:49.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7acc129140 con 0x7f7acc128690 2024-02-23T00:34:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7acb7fe700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7acc128690 0x7f7acc128ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7ab3fff700 10 client.37508.objecter ms_handle_connect 0x7f7acc128690 2024-02-23T00:34:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.011+0000 7f7acb7fe700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7acc128690 0x7f7acc128ad0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.012+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7acc129140 con 0x7f7acc128690 2024-02-23T00:34:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.012+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:34:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.012+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 13 = system:0 2024-02-23T00:34:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.012+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc125f80 osd=6 4 2024-02-23T00:34:49.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.012+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 6 13 2024-02-23T00:34:49.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 10 client.37508.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:49.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 14 2024-02-23T00:34:49.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc125f80 osd=6 3 2024-02-23T00:34:49.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc125f80 osd=6 3 2024-02-23T00:34:49.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc125f80 osd=6 3 2024-02-23T00:34:49.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 14 2024-02-23T00:34:49.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc125f80 osd=6 3 2024-02-23T00:34:49.028 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 6 14 2024-02-23T00:34:49.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc125f80 osd=6 4 2024-02-23T00:34:49.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 10 client.37508.objecter _send_command 14 2024-02-23T00:34:49.029 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.026+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7acc129140 con 0x7f7acc128690 2024-02-23T00:34:49.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7acc129140 con 0x7f7acc128690 2024-02-23T00:34:49.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:49.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc125f80 osd=6 4 2024-02-23T00:34:49.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 6 14 2024-02-23T00:34:49.030 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:34:49.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 10 client.37508.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:49.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 15 2024-02-23T00:34:49.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6812/373787000 conn(0x7f7acc12b8d0 0x7f7acc12bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:49.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc129140 osd=7 3 2024-02-23T00:34:49.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc129140 osd=7 3 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc129140 osd=7 3 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 15 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc129140 osd=7 3 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 7 15 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc129140 osd=7 4 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 10 client.37508.objecter _send_command 15 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7acc12c360 con 0x7f7acc12b8d0 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.028+0000 7f7acaffd700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6812/373787000 conn(0x7f7acc12b8d0 0x7f7acc12bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.029+0000 7f7ab3fff700 10 client.37508.objecter ms_handle_connect 0x7f7acc12b8d0 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.029+0000 7f7acaffd700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6812/373787000 conn(0x7f7acc12b8d0 0x7f7acc12bcf0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.029+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7acc12c360 con 0x7f7acc12b8d0 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.029+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.029+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 15 = system:0 2024-02-23T00:34:49.032 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.029+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc129140 osd=7 4 2024-02-23T00:34:49.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.029+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 7 15 2024-02-23T00:34:49.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 10 client.37508.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:34:49.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign -1 16 2024-02-23T00:34:49.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc129140 osd=7 3 2024-02-23T00:34:49.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc129140 osd=7 3 2024-02-23T00:34:49.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 20 client.37508.objecter _get_session s=0x7f7acc129140 osd=7 3 2024-02-23T00:34:49.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_remove -1 16 2024-02-23T00:34:49.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 20 client.37508.objecter get_session s=0x7f7acc129140 osd=7 3 2024-02-23T00:34:49.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 15 client.37508.objecter _session_command_op_assign 7 16 2024-02-23T00:34:49.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc129140 osd=7 4 2024-02-23T00:34:49.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 10 client.37508.objecter _send_command 16 2024-02-23T00:34:49.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.045+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7acc12c360 con 0x7f7acc12b8d0 2024-02-23T00:34:49.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ab3fff700 1 -- 172.21.15.87:0/3263419833 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7acc12c360 con 0x7f7acc12b8d0 2024-02-23T00:34:49.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ab3fff700 10 client.37508.objecter ms_dispatch 0x7f7acc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:34:49.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ab3fff700 10 client.37508.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ab3fff700 20 client.37508.objecter put_session s=0x7f7acc129140 osd=7 4 2024-02-23T00:34:49.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ab3fff700 15 client.37508.objecter _session_command_op_remove 7 16 2024-02-23T00:34:49.049 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:34:49.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 librados: watch_flush enter 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 librados: watch_flush exit 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 client.37508.objecter close_session for osd.0 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7acc0fdd10 msgr2=0x7f7acc0fe0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6801/2153191863 conn(0x7f7acc0fdd10 0x7f7acc0fe0f0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc05ee80 osd=0 1 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 client.37508.objecter close_session for osd.1 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6805/685354171 conn(0x7f7acc0598f0 msgr2=0x7f7acc059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6805/685354171 conn(0x7f7acc0598f0 0x7f7acc059cd0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc028540 osd=1 1 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 client.37508.objecter close_session for osd.2 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7acc11a020 msgr2=0x7f7acc11c410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7acc11a020 0x7f7acc11c410 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc027b90 osd=2 1 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 client.37508.objecter close_session for osd.3 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6813/128059790 conn(0x7f7acc11da80 msgr2=0x7f7acc11fe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6813/128059790 conn(0x7f7acc11da80 0x7f7acc11fe70 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc119670 osd=3 1 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 client.37508.objecter close_session for osd.4 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7acc119280 msgr2=0x7f7acc1234f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7acc119280 0x7f7acc1234f0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7acb7fe700 1 -- 172.21.15.87:0/3263419833 reap_dead start 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc1190e0 osd=4 1 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 client.37508.objecter close_session for osd.5 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7acc123ef0 msgr2=0x7f7acc124330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7acc123ef0 0x7f7acc124330 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7acb7fe700 1 -- 172.21.15.87:0/3263419833 reap_dead start 2024-02-23T00:34:49.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc11d7b0 osd=5 1 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 client.37508.objecter close_session for osd.6 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7acc128690 msgr2=0x7f7acc128ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7acc128690 0x7f7acc128ad0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc125f80 osd=6 1 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7acb7fe700 1 -- 172.21.15.87:0/3263419833 reap_dead start 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 client.37508.objecter close_session for osd.7 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6812/373787000 conn(0x7f7acc12b8d0 msgr2=0x7f7acc12bcf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6812/373787000 conn(0x7f7acc12b8d0 0x7f7acc12bcf0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 20 client.37508.objecter put_session s=0x7f7acc129140 osd=7 1 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 20 client.37508.objecter shutdown clearing up homeless session... 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 10 client.37508.objecter successfully canceled tick 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7ab4062c90 msgr2=0x7f7ab4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7ab4062c90 0x7f7ab4065150 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 rx=0x7f7abc004de0 tx=0x7f7abc00b040).stop 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fcac0 msgr2=0x7f7acc109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:49.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fcac0 0x7f7acc109660 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f7ac00043a0 tx=0x7f7ac0009cb0).stop 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 shutdown_connections 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.171:6812/373787000 conn(0x7f7acc12b8d0 0x7f7acc12bcf0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7ab4062c90 0x7f7ab4065150 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 --2- 172.21.15.87:0/3263419833 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7acc0fcac0 0x7f7acc109660 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 >> 172.21.15.87:0/3263419833 conn(0x7f7acc0f26f0 msgr2=0x7f7acc0fa850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 shutdown_connections 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 -- 172.21.15.87:0/3263419833 wait complete. 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:49.047+0000 7f7ad1547700 1 librados: shutdown 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.054 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:34:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:47.625166+0000 mgr.y (mgr.24437) 327 : cluster [DBG] pgmap v487: 235 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 233 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s wr, 0 op/s; 13/1038 objects misplaced (1.252%); 977 KiB/s, 51 objects/s recovering 2024-02-23T00:34:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:48.901160+0000 mon.c (mon.2) 383 : audit [DBG] from='client.? 172.21.15.87:0/3263419833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:49.061271+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-02-23T00:34:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:47.625166+0000 mgr.y (mgr.24437) 327 : cluster [DBG] pgmap v487: 235 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 233 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s wr, 0 op/s; 13/1038 objects misplaced (1.252%); 977 KiB/s, 51 objects/s recovering 2024-02-23T00:34:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:48.901160+0000 mon.c (mon.2) 383 : audit [DBG] from='client.? 172.21.15.87:0/3263419833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:49.061271+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-02-23T00:34:49.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:47.625166+0000 mgr.y (mgr.24437) 327 : cluster [DBG] pgmap v487: 235 pgs: 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 233 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s wr, 0 op/s; 13/1038 objects misplaced (1.252%); 977 KiB/s, 51 objects/s recovering 2024-02-23T00:34:49.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:48.901160+0000 mon.c (mon.2) 383 : audit [DBG] from='client.? 172.21.15.87:0/3263419833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:49.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:49.061271+0000 mon.a (mon.0) 2251 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-02-23T00:34:50.129 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:34:50.129 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:34:50.985 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-23T00:34:51.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:34:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:34:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:49.625668+0000 mgr.y (mgr.24437) 328 : cluster [DBG] pgmap v490: 211 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 201 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1026 objects misplaced (1.267%) 2024-02-23T00:34:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:50.061821+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-02-23T00:34:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:50.066190+0000 mon.c (mon.2) 384 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:50.066563+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:50.066719+0000 mon.b (mon.1) 292 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:50.067714+0000 mon.a (mon.0) 2254 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:51.274 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:49.625668+0000 mgr.y (mgr.24437) 328 : cluster [DBG] pgmap v490: 211 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 201 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1026 objects misplaced (1.267%) 2024-02-23T00:34:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:50.061821+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-02-23T00:34:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:50.066190+0000 mon.c (mon.2) 384 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:51.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:50.066563+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:51.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:50.066719+0000 mon.b (mon.1) 292 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:51.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:50.067714+0000 mon.a (mon.0) 2254 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:49.625668+0000 mgr.y (mgr.24437) 328 : cluster [DBG] pgmap v490: 211 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 1 clean+premerge+peered, 201 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1026 objects misplaced (1.267%) 2024-02-23T00:34:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:50.061821+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-02-23T00:34:51.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:50.066190+0000 mon.c (mon.2) 384 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:51.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:50.066563+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:51.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:50.066719+0000 mon.b (mon.1) 292 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:51.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:50.067714+0000 mon.a (mon.0) 2254 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:52.175 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:52.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:52.210 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:52.280 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:52.308 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:52.329 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:52.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:52] "GET /metrics HTTP/1.1" 200 187225 "" "Prometheus/2.33.4" 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:50.823570+0000 mgr.y (mgr.24437) 329 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:51.065044+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]': finished 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:51.065127+0000 mon.a (mon.0) 2256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:51.065282+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:51.066231+0000 mon.c (mon.2) 385 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:51.097691+0000 mon.a (mon.0) 2258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:51.128170+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:51.128430+0000 mon.a (mon.0) 2259 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:51.627205+0000 mon.a (mon.0) 2260 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:52.054278+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]': finished 2024-02-23T00:34:52.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:52.054396+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:52.054481+0000 mon.a (mon.0) 2263 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:52.054545+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:50.823570+0000 mgr.y (mgr.24437) 329 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:51.065044+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]': finished 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:51.065127+0000 mon.a (mon.0) 2256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:51.065282+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:51.066231+0000 mon.c (mon.2) 385 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:51.097691+0000 mon.a (mon.0) 2258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:51.128170+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:51.128430+0000 mon.a (mon.0) 2259 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:51.627205+0000 mon.a (mon.0) 2260 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-23T00:34:52.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:52.054278+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]': finished 2024-02-23T00:34:52.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:52.054396+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:52.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:52.054481+0000 mon.a (mon.0) 2263 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-23T00:34:52.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:52.054545+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-02-23T00:34:52.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:52.055884+0000 mon.b (mon.1) 294 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:34:52.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:52.056408+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:34:52.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:52.055884+0000 mon.b (mon.1) 294 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:34:52.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:52.056408+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:34:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:52.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:52.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:34:52.437 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:52.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:50.823570+0000 mgr.y (mgr.24437) 329 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:34:52.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:51.065044+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi087-82019-58"}]': finished 2024-02-23T00:34:52.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:51.065127+0000 mon.a (mon.0) 2256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:52.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:51.065282+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:51.066231+0000 mon.c (mon.2) 385 : audit [INF] from='client.? 172.21.15.87:0/914692870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:51.097691+0000 mon.a (mon.0) 2258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]: dispatch 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:51.128170+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:51.128430+0000 mon.a (mon.0) 2259 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:51.627205+0000 mon.a (mon.0) 2260 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:52.054278+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi087-82019-58"}]': finished 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:52.054396+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:52.054481+0000 mon.a (mon.0) 2263 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:52.054545+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:52.055884+0000 mon.b (mon.1) 294 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:34:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:52.056408+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:34:52.492 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:52.745 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:52.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:52.849 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:52.876 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:52.974 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:52.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:53.078 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.084+0000 7f9231812700 1 -- 172.21.15.87:0/4203552242 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c0fdbb0 msgr2=0x7f922c0fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.084+0000 7f9231812700 1 --2- 172.21.15.87:0/4203552242 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c0fdbb0 0x7f922c0fdf90 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 rx=0x7f921c004870 tx=0x7f921c009cb0).stop 2024-02-23T00:34:53.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.084+0000 7f9231812700 1 -- 172.21.15.87:0/4203552242 shutdown_connections 2024-02-23T00:34:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.084+0000 7f9231812700 1 --2- 172.21.15.87:0/4203552242 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f922c0f8420 0x7f922c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.084+0000 7f9231812700 1 --2- 172.21.15.87:0/4203552242 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c0fdbb0 0x7f922c0fdf90 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.084+0000 7f9231812700 1 --2- 172.21.15.87:0/4203552242 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f922c0fe510 0x7f922c0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.084+0000 7f9231812700 1 -- 172.21.15.87:0/4203552242 >> 172.21.15.87:0/4203552242 conn(0x7f922c0f3800 msgr2=0x7f922c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.084+0000 7f9231812700 1 -- 172.21.15.87:0/4203552242 shutdown_connections 2024-02-23T00:34:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.084+0000 7f9231812700 1 -- 172.21.15.87:0/4203552242 wait complete. 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 librados: starting msgr at 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 librados: starting objecter 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 Processor -- start 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 -- start start 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 librados: setting wanted keys 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 librados: calling monclient init 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f922c0f8420 0x7f922c06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f922c0fdbb0 0x7f922c06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c0fe510 0x7f922c06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c116580 con 0x7f922c0fe510 2024-02-23T00:34:53.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c1166f0 con 0x7f922c0f8420 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f9231812700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c1168b0 con 0x7f922c0fdbb0 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f922a7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f922c0fdbb0 0x7f922c06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f922a7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f922c0fdbb0 0x7f922c06b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:46512/0 (socket says 172.21.15.87:46512) 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f922a7fc700 1 -- 172.21.15.87:0/2486019403 learned_addr learned my addr 172.21.15.87:0/2486019403 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.085+0000 7f922affd700 1 --2- 172.21.15.87:0/2486019403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f922c0f8420 0x7f922c06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f922a7fc700 1 -- 172.21.15.87:0/2486019403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f922c0f8420 msgr2=0x7f922c06b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f922a7fc700 1 --2- 172.21.15.87:0/2486019403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f922c0f8420 0x7f922c06b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f922a7fc700 1 -- 172.21.15.87:0/2486019403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c0fe510 msgr2=0x7f922c06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f922a7fc700 1 --2- 172.21.15.87:0/2486019403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c0fe510 0x7f922c06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f922a7fc700 1 -- 172.21.15.87:0/2486019403 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92240097c0 con 0x7f922c0fdbb0 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f922a7fc700 1 --2- 172.21.15.87:0/2486019403 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f922c0fdbb0 0x7f922c06b5c0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7f922000e970 tx=0x7f922000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f9213fff700 10 client.?.objecter ms_handle_connect 0x7f922c0fdbb0 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f9213fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:53.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f9213fff700 1 -- 172.21.15.87:0/2486019403 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922000ee60 con 0x7f922c0fdbb0 2024-02-23T00:34:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921c005040 con 0x7f922c0fdbb0 2024-02-23T00:34:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f9213fff700 1 -- 172.21.15.87:0/2486019403 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f922000f040 con 0x7f922c0fdbb0 2024-02-23T00:34:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f9213fff700 1 -- 172.21.15.87:0/2486019403 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220007ea0 con 0x7f922c0fdbb0 2024-02-23T00:34:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f9231812700 10 client.37520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c116fc0 con 0x7f922c0fdbb0 2024-02-23T00:34:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.086+0000 7f9231812700 1 librados: init done 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.087+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922c05ee80 con 0x7f922c0fdbb0 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f9213fff700 1 -- 172.21.15.87:0/2486019403 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f922000cb10 con 0x7f922c0fdbb0 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f9213fff700 1 --2- 172.21.15.87:0/2486019403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9214062c90 0x7f9214065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f9213fff700 1 -- 172.21.15.87:0/2486019403 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(342..342 src has 173..342) v4 ==== 8494+0+0 (secure 0 0 0) 0x7f922007cf90 con 0x7f922c0fdbb0 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f9213fff700 10 client.37520.objecter ms_dispatch 0x7f922c001550 osd_map(342..342 src has 173..342) v4 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f9213fff700 3 client.37520.objecter handle_osd_map got epochs [342,342] > 0 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f9213fff700 3 client.37520.objecter handle_osd_map decoding full epoch 342 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f922affd700 1 --2- 172.21.15.87:0/2486019403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9214062c90 0x7f9214065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f9213fff700 20 client.37520.objecter dump_active .. 0 homeless 2024-02-23T00:34:53.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f9213fff700 10 client.37520.objecter ms_handle_connect 0x7f9214062c90 2024-02-23T00:34:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.088+0000 7f922affd700 1 --2- 172.21.15.87:0/2486019403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9214062c90 0x7f9214065150 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 rx=0x7f921c0043d0 tx=0x7f921c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:53.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.092+0000 7f9213fff700 1 -- 172.21.15.87:0/2486019403 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f922004af60 con 0x7f922c0fdbb0 2024-02-23T00:34:53.106 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:53.156 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:34:53.156 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:34:53.159 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:53.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:53.176 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:51.626166+0000 mgr.y (mgr.24437) 330 : cluster [DBG] pgmap v493: 234 pgs: 1 active+remapped+backfill_toofull, 1 peering, 32 creating+peering, 200 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 11 op/s; 13/1044 objects misplaced (1.245%); 12 B/s, 0 objects/s recovering 2024-02-23T00:34:53.176 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:51.878930+0000 osd.7 (osd.7) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:53.176 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:51.879118+0000 osd.7 (osd.7) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:53.176 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:53.056588+0000 mon.a (mon.0) 2266 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-45"}]': finished 2024-02-23T00:34:53.177 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:53.056736+0000 mon.a (mon.0) 2267 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-02-23T00:34:53.177 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:53.059799+0000 mon.c (mon.2) 386 : audit [INF] from='client.? 172.21.15.87:0/2449917586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:53.177 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:53.062663+0000 mon.b (mon.1) 295 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-45", "mode": "writeback"}]: dispatch 2024-02-23T00:34:53.177 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:53.063059+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:53.177 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:53.064803+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-45", "mode": "writeback"}]: dispatch 2024-02-23T00:34:53.263 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:53.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:53.279+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f922c05fa60 con 0x7f922c0fdbb0 2024-02-23T00:34:53.289 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:51.626166+0000 mgr.y (mgr.24437) 330 : cluster [DBG] pgmap v493: 234 pgs: 1 active+remapped+backfill_toofull, 1 peering, 32 creating+peering, 200 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 11 op/s; 13/1044 objects misplaced (1.245%); 12 B/s, 0 objects/s recovering 2024-02-23T00:34:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:51.878930+0000 osd.7 (osd.7) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:51.879118+0000 osd.7 (osd.7) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:53.056588+0000 mon.a (mon.0) 2266 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-45"}]': finished 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:53.056736+0000 mon.a (mon.0) 2267 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:53.059799+0000 mon.c (mon.2) 386 : audit [INF] from='client.? 172.21.15.87:0/2449917586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:53.062663+0000 mon.b (mon.1) 295 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-45", "mode": "writeback"}]: dispatch 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:53.063059+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:53.064803+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-45", "mode": "writeback"}]: dispatch 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:51.626166+0000 mgr.y (mgr.24437) 330 : cluster [DBG] pgmap v493: 234 pgs: 1 active+remapped+backfill_toofull, 1 peering, 32 creating+peering, 200 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s rd, 0 B/s wr, 11 op/s; 13/1044 objects misplaced (1.245%); 12 B/s, 0 objects/s recovering 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:51.878930+0000 osd.7 (osd.7) 165 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:51.879118+0000 osd.7 (osd.7) 166 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:53.056588+0000 mon.a (mon.0) 2266 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-45"}]': finished 2024-02-23T00:34:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:53.056736+0000 mon.a (mon.0) 2267 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-02-23T00:34:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:53.059799+0000 mon.c (mon.2) 386 : audit [INF] from='client.? 172.21.15.87:0/2449917586' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:53.062663+0000 mon.b (mon.1) 295 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-45", "mode": "writeback"}]: dispatch 2024-02-23T00:34:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:53.063059+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:34:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:53.064803+0000 mon.a (mon.0) 2269 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-45", "mode": "writeback"}]: dispatch 2024-02-23T00:34:53.490 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:53.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:53.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:53.621 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:53.766 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:34:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:34:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:34:53.786 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-23T00:34:53.819 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:53.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:53.924 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:53.951 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:53.971 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:53.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:54.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.057+0000 7f9213fff700 1 -- 172.21.15.87:0/2486019403 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v343) v1 ==== 91+0+0 (secure 0 0 0) 0x7f9220019070 con 0x7f922c0fdbb0 2024-02-23T00:34:54.061 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:54.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.062+0000 7f9231812700 10 librados: watch_flush enter 2024-02-23T00:34:54.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.062+0000 7f9231812700 10 librados: watch_flush exit 2024-02-23T00:34:54.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.062+0000 7f9231812700 20 client.37520.objecter shutdown clearing up homeless session... 2024-02-23T00:34:54.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.062+0000 7f9231812700 10 client.37520.objecter successfully canceled tick 2024-02-23T00:34:54.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.062+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9214062c90 msgr2=0x7f9214065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:54.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.062+0000 7f9231812700 1 --2- 172.21.15.87:0/2486019403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9214062c90 0x7f9214065150 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 rx=0x7f921c0043d0 tx=0x7f921c00b040).stop 2024-02-23T00:34:54.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.062+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f922c0fdbb0 msgr2=0x7f922c06b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:54.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.062+0000 7f9231812700 1 --2- 172.21.15.87:0/2486019403 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f922c0fdbb0 0x7f922c06b5c0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7f922000e970 tx=0x7f922000b040).stop 2024-02-23T00:34:54.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.064+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 shutdown_connections 2024-02-23T00:34:54.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.064+0000 7f9231812700 1 --2- 172.21.15.87:0/2486019403 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9214062c90 0x7f9214065150 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.064+0000 7f9231812700 1 --2- 172.21.15.87:0/2486019403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f922c0fe510 0x7f922c06bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:34:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.064+0000 7f9231812700 1 --2- 172.21.15.87:0/2486019403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f922c0f8420 0x7f922c06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.064+0000 7f9231812700 1 --2- 172.21.15.87:0/2486019403 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f922c0fdbb0 0x7f922c06b5c0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.064+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 >> 172.21.15.87:0/2486019403 conn(0x7f922c0f3800 msgr2=0x7f922c0f52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.065+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 shutdown_connections 2024-02-23T00:34:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.065+0000 7f9231812700 1 -- 172.21.15.87:0/2486019403 wait complete. 2024-02-23T00:34:54.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:54.065+0000 7f9231812700 1 librados: shutdown 2024-02-23T00:34:54.071 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:34:54.075 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:54.111 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:54.347 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:53.280945+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.87:0/2486019403' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:54.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:53.281449+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:54.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:54.053154+0000 mon.a (mon.0) 2271 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:34:54.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:54.053182+0000 mon.a (mon.0) 2272 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:54.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:54.057257+0000 mon.a (mon.0) 2273 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:54.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:54.057362+0000 mon.a (mon.0) 2274 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-45", "mode": "writeback"}]': finished 2024-02-23T00:34:54.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:54.057482+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:34:54.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:54.057542+0000 mon.a (mon.0) 2276 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-02-23T00:34:54.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:54.068864+0000 mon.b (mon.1) 296 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:54.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:54.073581+0000 mon.a (mon.0) 2277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:54.358 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:54.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:54.462 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:54.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:53.280945+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.87:0/2486019403' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:54.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:53.281449+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:54.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:54.053154+0000 mon.a (mon.0) 2271 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:34:54.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:54.053182+0000 mon.a (mon.0) 2272 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:54.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:54.057257+0000 mon.a (mon.0) 2273 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:54.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:54.057362+0000 mon.a (mon.0) 2274 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-45", "mode": "writeback"}]': finished 2024-02-23T00:34:54.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:54.057482+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:34:54.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:54.057542+0000 mon.a (mon.0) 2276 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-02-23T00:34:54.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:54.068864+0000 mon.b (mon.1) 296 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:54.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:54.073581+0000 mon.a (mon.0) 2277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:53.280945+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.87:0/2486019403' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:53.281449+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:54.053154+0000 mon.a (mon.0) 2271 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:54.053182+0000 mon.a (mon.0) 2272 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:54.057257+0000 mon.a (mon.0) 2273 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi087-82019-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:54.057362+0000 mon.a (mon.0) 2274 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-45", "mode": "writeback"}]': finished 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:54.057482+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:54.057542+0000 mon.a (mon.0) 2276 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:54.068864+0000 mon.b (mon.1) 296 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:54.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:54.073581+0000 mon.a (mon.0) 2277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:34:54.491 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:54.542 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:54.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:54.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:54.672 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:55.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:53.626653+0000 mgr.y (mgr.24437) 331 : cluster [DBG] pgmap v496: 266 pgs: 1 active+remapped+backfill_toofull, 1 peering, 32 unknown, 32 creating+peering, 200 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s rd, 511 B/s wr, 15 op/s; 13/1044 objects misplaced (1.245%); 54 B/s, 1 objects/s recovering 2024-02-23T00:34:55.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:54.388068+0000 osd.0 (osd.0) 167 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:34:55.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:54.388829+0000 osd.0 (osd.0) 168 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:34:55.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:54.651035+0000 mon.a (mon.0) 2278 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:34:55.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:54.920576+0000 mon.a (mon.0) 2279 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:34:55.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.061053+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:34:55.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:55.061156+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-02-23T00:34:55.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.062958+0000 mon.b (mon.1) 297 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:55.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.063376+0000 mon.a (mon.0) 2282 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:55.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.086662+0000 mon.c (mon.2) 388 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.086980+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.087557+0000 mon.c (mon.2) 389 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.087799+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.088152+0000 mon.c (mon.2) 390 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi087-82019-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:55.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.088390+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi087-82019-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:55.321 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:55.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:55.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:53.626653+0000 mgr.y (mgr.24437) 331 : cluster [DBG] pgmap v496: 266 pgs: 1 active+remapped+backfill_toofull, 1 peering, 32 unknown, 32 creating+peering, 200 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s rd, 511 B/s wr, 15 op/s; 13/1044 objects misplaced (1.245%); 54 B/s, 1 objects/s recovering 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:54.388068+0000 osd.0 (osd.0) 167 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:54.388829+0000 osd.0 (osd.0) 168 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:54.651035+0000 mon.a (mon.0) 2278 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:54.920576+0000 mon.a (mon.0) 2279 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.061053+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:55.061156+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.062958+0000 mon.b (mon.1) 297 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.063376+0000 mon.a (mon.0) 2282 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.086662+0000 mon.c (mon.2) 388 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.086980+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.087557+0000 mon.c (mon.2) 389 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.087799+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.088152+0000 mon.c (mon.2) 390 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi087-82019-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.088390+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi087-82019-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:53.626653+0000 mgr.y (mgr.24437) 331 : cluster [DBG] pgmap v496: 266 pgs: 1 active+remapped+backfill_toofull, 1 peering, 32 unknown, 32 creating+peering, 200 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.0 KiB/s rd, 511 B/s wr, 15 op/s; 13/1044 objects misplaced (1.245%); 54 B/s, 1 objects/s recovering 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:54.388068+0000 osd.0 (osd.0) 167 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:54.388829+0000 osd.0 (osd.0) 168 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:54.651035+0000 mon.a (mon.0) 2278 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:54.920576+0000 mon.a (mon.0) 2279 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:34:55.454 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.061053+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:34:55.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:55.061156+0000 mon.a (mon.0) 2281 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-02-23T00:34:55.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.062958+0000 mon.b (mon.1) 297 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:55.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.063376+0000 mon.a (mon.0) 2282 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:34:55.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.086662+0000 mon.c (mon.2) 388 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.086980+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.087557+0000 mon.c (mon.2) 389 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.087799+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:55.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.088152+0000 mon.c (mon.2) 390 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi087-82019-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:55.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.088390+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi087-82019-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:34:55.457 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:55.508 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:34:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:55.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:34:55.620 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:55.649 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:55.848 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:55.849 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:34:55.849 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:34:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:55.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:55.951 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:55.982 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:56.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:54.689633+0000 osd.4 (osd.4) 153 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:34:56.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:54.691532+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:34:56.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:54.721580+0000 osd.2 (osd.2) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:56.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:54.721748+0000 osd.2 (osd.2) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:56.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:55.628724+0000 mon.a (mon.0) 2286 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-23T00:34:56.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:56.062709+0000 mon.a (mon.0) 2287 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:34:56.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:56.062807+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi087-82019-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:56.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:56.062884+0000 mon.a (mon.0) 2289 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-23T00:34:56.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:56.062934+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-02-23T00:34:56.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:56.063912+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:56.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:56.063958+0000 mon.c (mon.2) 391 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi087-82019-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:56.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:56.064094+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:56.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:56.064448+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi087-82019-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:54.689633+0000 osd.4 (osd.4) 153 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:34:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:54.691532+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:34:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:54.721580+0000 osd.2 (osd.2) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:54.721748+0000 osd.2 (osd.2) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:55.628724+0000 mon.a (mon.0) 2286 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-23T00:34:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:56.062709+0000 mon.a (mon.0) 2287 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:34:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:56.062807+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi087-82019-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:56.062884+0000 mon.a (mon.0) 2289 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-23T00:34:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:56.062934+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-02-23T00:34:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:56.063912+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:56.063958+0000 mon.c (mon.2) 391 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi087-82019-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:56.064094+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:56.064448+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi087-82019-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:54.689633+0000 osd.4 (osd.4) 153 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:54.691532+0000 osd.4 (osd.4) 154 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:54.721580+0000 osd.2 (osd.2) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:54.721748+0000 osd.2 (osd.2) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:55.628724+0000 mon.a (mon.0) 2286 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:56.062709+0000 mon.a (mon.0) 2287 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:56.062807+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi087-82019-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:56.062884+0000 mon.a (mon.0) 2289 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:56.062934+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:56.063912+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:56.063958+0000 mon.c (mon.2) 391 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi087-82019-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:56.064094+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:34:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:56.064448+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi087-82019-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:34:56.431 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:56.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:56.536 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:56.562 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-23T00:34:56.569 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:56.831 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:34:57.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.061+0000 7f776c098700 1 -- 172.21.15.87:0/2447876600 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77640fe510 msgr2=0x7f77640f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.061+0000 7f776c098700 1 --2- 172.21.15.87:0/2447876600 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77640fe510 0x7f77640f7d40 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7f7760004870 tx=0x7f7760009cb0).stop 2024-02-23T00:34:57.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.061+0000 7f776c098700 1 -- 172.21.15.87:0/2447876600 shutdown_connections 2024-02-23T00:34:57.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.061+0000 7f776c098700 1 --2- 172.21.15.87:0/2447876600 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77640f8420 0x7f7764104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.061+0000 7f776c098700 1 --2- 172.21.15.87:0/2447876600 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77640fdbb0 0x7f77640fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:34:57.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.061+0000 7f776c098700 1 --2- 172.21.15.87:0/2447876600 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77640fe510 0x7f77640f7d40 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.061+0000 7f776c098700 1 -- 172.21.15.87:0/2447876600 >> 172.21.15.87:0/2447876600 conn(0x7f77640f3800 msgr2=0x7f77640f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 -- 172.21.15.87:0/2447876600 shutdown_connections 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 -- 172.21.15.87:0/2447876600 wait complete. 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 librados: starting msgr at 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 librados: starting objecter 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 Processor -- start 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 -- start start 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 librados: setting wanted keys 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 librados: calling monclient init 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77640f8420 0x7f776410f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77640fdbb0 0x7f776410f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77640fe510 0x7f776410a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.062+0000 7f776c098700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776410ae00 con 0x7f77640fdbb0 2024-02-23T00:34:57.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776c098700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776410af40 con 0x7f77640f8420 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776c098700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776410b100 con 0x7f77640fe510 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776a635700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77640fe510 0x7f776410a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776a635700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77640fe510 0x7f776410a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:46544/0 (socket says 172.21.15.87:46544) 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776a635700 1 -- 172.21.15.87:0/983994112 learned_addr learned my addr 172.21.15.87:0/983994112 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f7769e34700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77640f8420 0x7f776410f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f7769633700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77640fdbb0 0x7f776410f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776a635700 1 -- 172.21.15.87:0/983994112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77640f8420 msgr2=0x7f776410f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776a635700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77640f8420 0x7f776410f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776a635700 1 -- 172.21.15.87:0/983994112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77640fdbb0 msgr2=0x7f776410f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776a635700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77640fdbb0 0x7f776410f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776a635700 1 -- 172.21.15.87:0/983994112 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7760005040 con 0x7f77640fe510 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f7769633700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77640fdbb0 0x7f776410f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776a635700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77640fe510 0x7f776410a8d0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f775c00ec20 tx=0x7f775c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f775affd700 10 client.?.objecter ms_handle_connect 0x7f77640fe510 2024-02-23T00:34:57.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f775affd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f775affd700 1 -- 172.21.15.87:0/983994112 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c009950 con 0x7f77640fe510 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f775affd700 1 -- 172.21.15.87:0/983994112 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f775c00f040 con 0x7f77640fe510 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776410b380 con 0x7f77640fe510 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f775affd700 1 -- 172.21.15.87:0/983994112 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c007df0 con 0x7f77640fe510 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776c098700 10 client.37529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7764117170 con 0x7f77640fe510 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.063+0000 7f776c098700 1 librados: init done 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.064+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776406bb40 con 0x7f77640fe510 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.065+0000 7f775affd700 1 -- 172.21.15.87:0/983994112 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f775c0092e0 con 0x7f77640fe510 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.065+0000 7f775affd700 1 --2- 172.21.15.87:0/983994112 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7750062c90 0x7f7750065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.065+0000 7f775affd700 1 -- 172.21.15.87:0/983994112 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(345..345 src has 173..345) v4 ==== 8403+0+0 (secure 0 0 0) 0x7f775c07d040 con 0x7f77640fe510 2024-02-23T00:34:57.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.065+0000 7f775affd700 10 client.37529.objecter ms_dispatch 0x7f7764001550 osd_map(345..345 src has 173..345) v4 2024-02-23T00:34:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.065+0000 7f775affd700 3 client.37529.objecter handle_osd_map got epochs [345,345] > 0 2024-02-23T00:34:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.065+0000 7f775affd700 3 client.37529.objecter handle_osd_map decoding full epoch 345 2024-02-23T00:34:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.065+0000 7f7769e34700 1 --2- 172.21.15.87:0/983994112 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7750062c90 0x7f7750065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.066+0000 7f775affd700 20 client.37529.objecter dump_active .. 0 homeless 2024-02-23T00:34:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.066+0000 7f775affd700 10 client.37529.objecter ms_handle_connect 0x7f7750062c90 2024-02-23T00:34:57.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.066+0000 7f7769e34700 1 --2- 172.21.15.87:0/983994112 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7750062c90 0x7f7750065150 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 rx=0x7f7754006fd0 tx=0x7f7754009060).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.076+0000 7f775affd700 1 -- 172.21.15.87:0/983994112 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f775c049f60 con 0x7f77640fe510 2024-02-23T00:34:57.119 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:57.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:57.224 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:34:57.253 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:57.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.275+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f776405fb60 con 0x7f77640fe510 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.396+0000 7fe6bb668700 1 -- 172.21.15.87:0/4274938204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6b40fd510 msgr2=0x7fe6b40f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.396+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4274938204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6b40fd510 0x7fe6b40f6d40 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 rx=0x7fe6b0004870 tx=0x7fe6b0009cb0).stop 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.396+0000 7fe6bb668700 1 -- 172.21.15.87:0/4274938204 shutdown_connections 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.396+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4274938204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6b40f7400 0x7fe6b4103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.396+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4274938204 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6b40fcbb0 0x7fe6b40fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.396+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4274938204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6b40fd510 0x7fe6b40f6d40 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.396+0000 7fe6bb668700 1 -- 172.21.15.87:0/4274938204 >> 172.21.15.87:0/4274938204 conn(0x7fe6b40f2820 msgr2=0x7fe6b40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.397+0000 7fe6bb668700 1 -- 172.21.15.87:0/4274938204 shutdown_connections 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.397+0000 7fe6bb668700 1 -- 172.21.15.87:0/4274938204 wait complete. 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.397+0000 7fe6bb668700 1 librados: starting msgr at 2024-02-23T00:34:57.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.397+0000 7fe6bb668700 1 librados: starting objecter 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.397+0000 7fe6bb668700 1 Processor -- start 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.397+0000 7fe6bb668700 1 -- start start 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.397+0000 7fe6bb668700 1 librados: setting wanted keys 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.397+0000 7fe6bb668700 1 librados: calling monclient init 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6b40f7400 0x7fe6b410d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6b40fcbb0 0x7fe6b410d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6b40fd510 0x7fe6b4107350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4107840 con 0x7fe6b40fd510 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41079b0 con 0x7fe6b40fcbb0 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4107b70 con 0x7fe6b40f7400 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9404700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6b40f7400 0x7fe6b410d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9c05700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6b40fd510 0x7fe6b4107350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9404700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6b40f7400 0x7fe6b410d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:46560/0 (socket says 172.21.15.87:46560) 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9404700 1 -- 172.21.15.87:0/4144670975 learned_addr learned my addr 172.21.15.87:0/4144670975 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b8c03700 1 --2- 172.21.15.87:0/4144670975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6b40fcbb0 0x7fe6b410d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9404700 1 -- 172.21.15.87:0/4144670975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6b40fcbb0 msgr2=0x7fe6b410d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9404700 1 --2- 172.21.15.87:0/4144670975 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe6b40fcbb0 0x7fe6b410d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9404700 1 -- 172.21.15.87:0/4144670975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6b40fd510 msgr2=0x7fe6b4107350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9404700 1 --2- 172.21.15.87:0/4144670975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6b40fd510 0x7fe6b4107350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9404700 1 -- 172.21.15.87:0/4144670975 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a4005040 con 0x7fe6b40f7400 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9c05700 1 --2- 172.21.15.87:0/4144670975 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe6b40fd510 0x7fe6b4107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6b9404700 1 --2- 172.21.15.87:0/4144670975 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6b40f7400 0x7fe6b410d1b0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 rx=0x7fe6a40048f0 tx=0x7fe6a4009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6aa7fc700 10 client.?.objecter ms_handle_connect 0x7fe6b40f7400 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6aa7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:34:57.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4007b00 con 0x7fe6b40f7400 2024-02-23T00:34:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b0005040 con 0x7fe6b40f7400 2024-02-23T00:34:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6a400f040 con 0x7fe6b40f7400 2024-02-23T00:34:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4014990 con 0x7fe6b40f7400 2024-02-23T00:34:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 10 client.37535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:34:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b41081b0 con 0x7fe6b40f7400 2024-02-23T00:34:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.398+0000 7fe6bb668700 1 librados: init done 2024-02-23T00:34:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.399+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe6b405fa60 con 0x7fe6b40f7400 2024-02-23T00:34:57.405 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fe6a4010500 con 0x7fe6b40f7400 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6aa7fc700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe6a0062c90 0x7fe6a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(346..346 src has 173..346) v4 ==== 8536+0+0 (secure 0 0 0) 0x7fe6a407cf00 con 0x7fe6b40f7400 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6b8c03700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe6a0062c90 0x7fe6a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 osd_map(346..346 src has 173..346) v4 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6aa7fc700 3 client.37535.objecter handle_osd_map got epochs [346,346] > 0 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6aa7fc700 3 client.37535.objecter handle_osd_map decoding full epoch 346 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6aa7fc700 20 client.37535.objecter dump_active .. 0 homeless 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v346) v1 ==== 54+0+16 (secure 0 0 0) 0x7fe6a40825c0 con 0x7fe6b40f7400 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6aa7fc700 10 client.37535.objecter ms_handle_connect 0x7fe6a0062c90 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6b8c03700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe6a0062c90 0x7fe6a0065150 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7fe6b0004ee0 tx=0x7fe6b000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 10 client.37535.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 1 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6801/2153191863 conn(0x7fe6b4028540 0x7fe6b40fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b405ee80 osd=0 3 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b405ee80 osd=0 3 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b405ee80 osd=0 3 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 1 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b405ee80 osd=0 3 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 0 1 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b405ee80 osd=0 4 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 10 client.37535.objecter _send_command 1 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.400+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6b4106120 con 0x7fe6b4028540 2024-02-23T00:34:57.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.401+0000 7fe6b9c05700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6801/2153191863 conn(0x7fe6b4028540 0x7fe6b40fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.401+0000 7fe6b9c05700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6801/2153191863 conn(0x7fe6b4028540 0x7fe6b40fd9b0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.401+0000 7fe6aa7fc700 10 client.37535.objecter ms_handle_connect 0x7fe6b4028540 2024-02-23T00:34:57.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.401+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6b4106120 con 0x7fe6b4028540 2024-02-23T00:34:57.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.401+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:34:57.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.401+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 1 = system:0 2024-02-23T00:34:57.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.401+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b405ee80 osd=0 4 2024-02-23T00:34:57.409 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.401+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 0 1 2024-02-23T00:34:57.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 10 client.37535.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 2 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b405ee80 osd=0 3 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b405ee80 osd=0 3 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b405ee80 osd=0 3 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 2 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b405ee80 osd=0 3 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 0 2 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b405ee80 osd=0 4 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 10 client.37535.objecter _send_command 2 2024-02-23T00:34:57.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.416+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6b405a040 con 0x7fe6b4028540 2024-02-23T00:34:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe6b405a040 con 0x7fe6b4028540 2024-02-23T00:34:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b405ee80 osd=0 4 2024-02-23T00:34:57.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 0 2 2024-02-23T00:34:57.424 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:34:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 10 client.37535.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 3 2024-02-23T00:34:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6805/685354171 conn(0x7fe6b40594d0 0x7fe6b40fdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4059360 osd=1 3 2024-02-23T00:34:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4059360 osd=1 3 2024-02-23T00:34:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4059360 osd=1 3 2024-02-23T00:34:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 3 2024-02-23T00:34:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4059360 osd=1 3 2024-02-23T00:34:57.425 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 1 3 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4059360 osd=1 4 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 10 client.37535.objecter _send_command 3 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6b4061a90 con 0x7fe6b40594d0 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.418+0000 7fe6b9404700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6805/685354171 conn(0x7fe6b40594d0 0x7fe6b40fdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.419+0000 7fe6aa7fc700 10 client.37535.objecter ms_handle_connect 0x7fe6b40594d0 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.419+0000 7fe6b9404700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6805/685354171 conn(0x7fe6b40594d0 0x7fe6b40fdea0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.419+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6b4061a90 con 0x7fe6b40594d0 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.419+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.419+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 3 = system:0 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.419+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4059360 osd=1 4 2024-02-23T00:34:57.426 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.419+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 1 3 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 10 client.37535.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 4 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4059360 osd=1 3 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4059360 osd=1 3 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4059360 osd=1 3 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 4 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4059360 osd=1 3 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 1 4 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4059360 osd=1 4 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 10 client.37535.objecter _send_command 4 2024-02-23T00:34:57.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.434+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6b40fe390 con 0x7fe6b40594d0 2024-02-23T00:34:57.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe6b40fe390 con 0x7fe6b40594d0 2024-02-23T00:34:57.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:57.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4059360 osd=1 4 2024-02-23T00:34:57.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 1 4 2024-02-23T00:34:57.438 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:34:57.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 10 client.37535.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:57.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 5 2024-02-23T00:34:57.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe6b4061160 0x7fe6b411c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4060ff0 osd=2 3 2024-02-23T00:34:57.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4060ff0 osd=2 3 2024-02-23T00:34:57.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4060ff0 osd=2 3 2024-02-23T00:34:57.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 5 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4060ff0 osd=2 3 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 2 5 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4060ff0 osd=2 4 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 10 client.37535.objecter _send_command 5 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.435+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6b411efc0 con 0x7fe6b4061160 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.436+0000 7fe6b8c03700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe6b4061160 0x7fe6b411c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.436+0000 7fe6b8c03700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe6b4061160 0x7fe6b411c2e0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.436+0000 7fe6aa7fc700 10 client.37535.objecter ms_handle_connect 0x7fe6b4061160 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.436+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6b411efc0 con 0x7fe6b4061160 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.436+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.436+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 5 = system:0 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.436+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4060ff0 osd=2 4 2024-02-23T00:34:57.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.436+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 2 5 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 10 client.37535.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 6 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4060ff0 osd=2 3 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4060ff0 osd=2 3 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4060ff0 osd=2 3 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 6 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4060ff0 osd=2 3 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 2 6 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4060ff0 osd=2 4 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 10 client.37535.objecter _send_command 6 2024-02-23T00:34:57.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.451+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6b40fe800 con 0x7fe6b4061160 2024-02-23T00:34:57.454 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:34:57.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe6b40fe800 con 0x7fe6b4061160 2024-02-23T00:34:57.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:57.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.455 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4060ff0 osd=2 4 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 2 6 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 10 client.37535.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 7 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6813/128059790 conn(0x7fe6b411d950 0x7fe6b411dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4027b90 osd=3 3 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4027b90 osd=3 3 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4027b90 osd=3 3 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 7 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4027b90 osd=3 3 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 3 7 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4027b90 osd=3 4 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 10 client.37535.objecter _send_command 7 2024-02-23T00:34:57.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.452+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6b4028250 con 0x7fe6b411d950 2024-02-23T00:34:57.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.453+0000 7fe6b9c05700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6813/128059790 conn(0x7fe6b411d950 0x7fe6b411dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.453+0000 7fe6aa7fc700 10 client.37535.objecter ms_handle_connect 0x7fe6b411d950 2024-02-23T00:34:57.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.453+0000 7fe6b9c05700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6813/128059790 conn(0x7fe6b411d950 0x7fe6b411dd70 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.453+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6b4028250 con 0x7fe6b411d950 2024-02-23T00:34:57.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.453+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:34:57.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.453+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 7 = system:0 2024-02-23T00:34:57.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.453+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4027b90 osd=3 4 2024-02-23T00:34:57.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.453+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 3 7 2024-02-23T00:34:57.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 10 client.37535.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:57.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 8 2024-02-23T00:34:57.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4027b90 osd=3 3 2024-02-23T00:34:57.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4027b90 osd=3 3 2024-02-23T00:34:57.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4027b90 osd=3 3 2024-02-23T00:34:57.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 8 2024-02-23T00:34:57.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4027b90 osd=3 3 2024-02-23T00:34:57.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 3 8 2024-02-23T00:34:57.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4027b90 osd=3 4 2024-02-23T00:34:57.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 10 client.37535.objecter _send_command 8 2024-02-23T00:34:57.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.468+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6b411ea30 con 0x7fe6b411d950 2024-02-23T00:34:57.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe6b411ea30 con 0x7fe6b411d950 2024-02-23T00:34:57.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:57.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4027b90 osd=3 4 2024-02-23T00:34:57.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 3 8 2024-02-23T00:34:57.472 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:34:57.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 10 client.37535.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 9 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6800/4175844757 conn(0x7fe6b4121170 0x7fe6b4123560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b411e260 osd=4 3 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e260 osd=4 3 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b411e260 osd=4 3 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 9 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b411e260 osd=4 3 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 4 9 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e260 osd=4 4 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 10 client.37535.objecter _send_command 9 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.469+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6b411e950 con 0x7fe6b4121170 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.470+0000 7fe6b9404700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6800/4175844757 conn(0x7fe6b4121170 0x7fe6b4123560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.470+0000 7fe6aa7fc700 10 client.37535.objecter ms_handle_connect 0x7fe6b4121170 2024-02-23T00:34:57.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.470+0000 7fe6b9404700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6800/4175844757 conn(0x7fe6b4121170 0x7fe6b4123560 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.471+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6b411e950 con 0x7fe6b4121170 2024-02-23T00:34:57.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.471+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:34:57.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.471+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 9 = system:0 2024-02-23T00:34:57.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.471+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b411e260 osd=4 4 2024-02-23T00:34:57.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.471+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 4 9 2024-02-23T00:34:57.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 10 client.37535.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 10 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b411e260 osd=4 3 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e260 osd=4 3 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b411e260 osd=4 3 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 10 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b411e260 osd=4 3 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 4 10 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e260 osd=4 4 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 10 client.37535.objecter _send_command 10 2024-02-23T00:34:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.486+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6b411e950 con 0x7fe6b4121170 2024-02-23T00:34:57.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.487+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe6b411e950 con 0x7fe6b4121170 2024-02-23T00:34:57.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.487+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:57.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.487+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.487+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b411e260 osd=4 4 2024-02-23T00:34:57.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.487+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 4 10 2024-02-23T00:34:57.490 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:34:57.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 10 client.37535.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:57.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 11 2024-02-23T00:34:57.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6804/2550013795 conn(0x7fe6b4123fc0 0x7fe6b4126430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b411e950 osd=5 3 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e950 osd=5 3 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b411e950 osd=5 3 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 11 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b411e950 osd=5 3 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 5 11 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e950 osd=5 4 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 10 client.37535.objecter _send_command 11 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6b4129200 con 0x7fe6b4123fc0 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6b8c03700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6804/2550013795 conn(0x7fe6b4123fc0 0x7fe6b4126430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6aa7fc700 10 client.37535.objecter ms_handle_connect 0x7fe6b4123fc0 2024-02-23T00:34:57.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.488+0000 7fe6b8c03700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6804/2550013795 conn(0x7fe6b4123fc0 0x7fe6b4126430 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.489+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6b4129200 con 0x7fe6b4123fc0 2024-02-23T00:34:57.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.489+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:34:57.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.489+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 11 = system:0 2024-02-23T00:34:57.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.489+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b411e950 osd=5 4 2024-02-23T00:34:57.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.489+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 5 11 2024-02-23T00:34:57.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 10 client.37535.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:57.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 12 2024-02-23T00:34:57.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b411e950 osd=5 3 2024-02-23T00:34:57.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e950 osd=5 3 2024-02-23T00:34:57.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b411e950 osd=5 3 2024-02-23T00:34:57.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 12 2024-02-23T00:34:57.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b411e950 osd=5 3 2024-02-23T00:34:57.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 5 12 2024-02-23T00:34:57.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e950 osd=5 4 2024-02-23T00:34:57.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 10 client.37535.objecter _send_command 12 2024-02-23T00:34:57.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.503+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6b4129200 con 0x7fe6b4123fc0 2024-02-23T00:34:57.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe6b4129200 con 0x7fe6b4123fc0 2024-02-23T00:34:57.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:57.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b411e950 osd=5 4 2024-02-23T00:34:57.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 5 12 2024-02-23T00:34:57.508 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 10 client.37535.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 13 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6808/2509408498 conn(0x7fe6b4128500 0x7fe6b4128980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4129200 osd=6 3 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4129200 osd=6 3 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4129200 osd=6 3 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 13 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4129200 osd=6 3 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 6 13 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4129200 osd=6 4 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 10 client.37535.objecter _send_command 13 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.505+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6b4128fa0 con 0x7fe6b4128500 2024-02-23T00:34:57.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.506+0000 7fe6b9c05700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6808/2509408498 conn(0x7fe6b4128500 0x7fe6b4128980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.506+0000 7fe6aa7fc700 10 client.37535.objecter ms_handle_connect 0x7fe6b4128500 2024-02-23T00:34:57.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.506+0000 7fe6b9c05700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6808/2509408498 conn(0x7fe6b4128500 0x7fe6b4128980 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.506+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6b4128fa0 con 0x7fe6b4128500 2024-02-23T00:34:57.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.506+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:34:57.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.506+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 13 = system:0 2024-02-23T00:34:57.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.506+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4129200 osd=6 4 2024-02-23T00:34:57.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.506+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 6 13 2024-02-23T00:34:57.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 10 client.37535.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:57.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 14 2024-02-23T00:34:57.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4129200 osd=6 3 2024-02-23T00:34:57.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4129200 osd=6 3 2024-02-23T00:34:57.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4129200 osd=6 3 2024-02-23T00:34:57.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 14 2024-02-23T00:34:57.523 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4129200 osd=6 3 2024-02-23T00:34:57.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 6 14 2024-02-23T00:34:57.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4129200 osd=6 4 2024-02-23T00:34:57.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 10 client.37535.objecter _send_command 14 2024-02-23T00:34:57.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.521+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6b4128fa0 con 0x7fe6b4128500 2024-02-23T00:34:57.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe6b4128fa0 con 0x7fe6b4128500 2024-02-23T00:34:57.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:57.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4129200 osd=6 4 2024-02-23T00:34:57.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 6 14 2024-02-23T00:34:57.525 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:34:57.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 10 client.37535.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:34:57.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 15 2024-02-23T00:34:57.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6812/373787000 conn(0x7fe6b412b970 0x7fe6b412bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:34:57.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4128fa0 osd=7 3 2024-02-23T00:34:57.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4128fa0 osd=7 3 2024-02-23T00:34:57.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4128fa0 osd=7 3 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 15 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4128fa0 osd=7 3 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 7 15 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4128fa0 osd=7 4 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 10 client.37535.objecter _send_command 15 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6b412c440 con 0x7fe6b412b970 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.523+0000 7fe6b9404700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6812/373787000 conn(0x7fe6b412b970 0x7fe6b412bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.524+0000 7fe6b9404700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6812/373787000 conn(0x7fe6b412b970 0x7fe6b412bdd0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.524+0000 7fe6aa7fc700 10 client.37535.objecter ms_handle_connect 0x7fe6b412b970 2024-02-23T00:34:57.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.524+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fe6b412c440 con 0x7fe6b412b970 2024-02-23T00:34:57.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.524+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:34:57.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.524+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 15 = system:0 2024-02-23T00:34:57.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.524+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4128fa0 osd=7 4 2024-02-23T00:34:57.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.524+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 7 15 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 10 client.37535.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign -1 16 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4128fa0 osd=7 3 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4128fa0 osd=7 3 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 20 client.37535.objecter _get_session s=0x7fe6b4128fa0 osd=7 3 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_remove -1 16 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 20 client.37535.objecter get_session s=0x7fe6b4128fa0 osd=7 3 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 15 client.37535.objecter _session_command_op_assign 7 16 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4128fa0 osd=7 4 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 10 client.37535.objecter _send_command 16 2024-02-23T00:34:57.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.539+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe6b412c440 con 0x7fe6b412b970 2024-02-23T00:34:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.540+0000 7fe6aa7fc700 1 -- 172.21.15.87:0/4144670975 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fe6b412c440 con 0x7fe6b412b970 2024-02-23T00:34:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.540+0000 7fe6aa7fc700 10 client.37535.objecter ms_dispatch 0x7fe6b4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:34:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.540+0000 7fe6aa7fc700 10 client.37535.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6aa7fc700 20 client.37535.objecter put_session s=0x7fe6b4128fa0 osd=7 4 2024-02-23T00:34:57.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6aa7fc700 15 client.37535.objecter _session_command_op_remove 7 16 2024-02-23T00:34:57.544 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:34:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 librados: watch_flush enter 2024-02-23T00:34:57.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 librados: watch_flush exit 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 client.37535.objecter close_session for osd.0 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6801/2153191863 conn(0x7fe6b4028540 msgr2=0x7fe6b40fd9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6801/2153191863 conn(0x7fe6b4028540 0x7fe6b40fd9b0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b405ee80 osd=0 1 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 client.37535.objecter close_session for osd.1 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6805/685354171 conn(0x7fe6b40594d0 msgr2=0x7fe6b40fdea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6805/685354171 conn(0x7fe6b40594d0 0x7fe6b40fdea0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4059360 osd=1 1 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 client.37535.objecter close_session for osd.2 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe6b4061160 msgr2=0x7fe6b411c2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6809/3605394228 conn(0x7fe6b4061160 0x7fe6b411c2e0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4060ff0 osd=2 1 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 client.37535.objecter close_session for osd.3 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6813/128059790 conn(0x7fe6b411d950 msgr2=0x7fe6b411dd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6b9c05700 1 -- 172.21.15.87:0/4144670975 reap_dead start 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6813/128059790 conn(0x7fe6b411d950 0x7fe6b411dd70 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4027b90 osd=3 1 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 client.37535.objecter close_session for osd.4 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6800/4175844757 conn(0x7fe6b4121170 msgr2=0x7fe6b4123560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6800/4175844757 conn(0x7fe6b4121170 0x7fe6b4123560 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e260 osd=4 1 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 client.37535.objecter close_session for osd.5 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6804/2550013795 conn(0x7fe6b4123fc0 msgr2=0x7fe6b4126430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6804/2550013795 conn(0x7fe6b4123fc0 0x7fe6b4126430 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b411e950 osd=5 1 2024-02-23T00:34:57.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 client.37535.objecter close_session for osd.6 2024-02-23T00:34:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6808/2509408498 conn(0x7fe6b4128500 msgr2=0x7fe6b4128980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6808/2509408498 conn(0x7fe6b4128500 0x7fe6b4128980 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4129200 osd=6 1 2024-02-23T00:34:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 client.37535.objecter close_session for osd.7 2024-02-23T00:34:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6812/373787000 conn(0x7fe6b412b970 msgr2=0x7fe6b412bdd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.171:6812/373787000 conn(0x7fe6b412b970 0x7fe6b412bdd0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 20 client.37535.objecter put_session s=0x7fe6b4128fa0 osd=7 1 2024-02-23T00:34:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 20 client.37535.objecter shutdown clearing up homeless session... 2024-02-23T00:34:57.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 10 client.37535.objecter successfully canceled tick 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6b9c05700 1 -- 172.21.15.87:0/4144670975 reap_dead start 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe6a0062c90 msgr2=0x7fe6a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe6a0062c90 0x7fe6a0065150 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 rx=0x7fe6b0004ee0 tx=0x7fe6b000b040).stop 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6b40f7400 msgr2=0x7fe6b410d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6b40f7400 0x7fe6b410d1b0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 rx=0x7fe6a40048f0 tx=0x7fe6a4009cb0).stop 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 shutdown_connections 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe6a0062c90 0x7fe6a0065150 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 --2- 172.21.15.87:0/4144670975 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe6b40f7400 0x7fe6b410d1b0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 >> 172.21.15.87:0/4144670975 conn(0x7fe6b40f2820 msgr2=0x7fe6b40f4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:57.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 shutdown_connections 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 -- 172.21.15.87:0/4144670975 wait complete. 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:57.541+0000 7fe6bb668700 1 librados: shutdown 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.550 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:34:57.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:55.608002+0000 osd.7 (osd.7) 167 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:34:57.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:55.609244+0000 osd.7 (osd.7) 168 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:34:57.570 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:55.627844+0000 mgr.y (mgr.24437) 332 : cluster [DBG] pgmap v499: 233 pgs: 1 active+remapped+backfill_toofull, 232 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 13/1104 objects misplaced (1.178%) 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:56.231736+0000 osd.6 (osd.6) 121 : cluster [DBG] 6.c scrub starts 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:56.232455+0000 osd.6 (osd.6) 122 : cluster [DBG] 6.c scrub ok 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:56.334726+0000 osd.6 (osd.6) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:56.335039+0000 osd.6 (osd.6) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.060099+0000 mon.a (mon.0) 2293 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:55.608002+0000 osd.7 (osd.7) 167 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:55.609244+0000 osd.7 (osd.7) 168 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:55.627844+0000 mgr.y (mgr.24437) 332 : cluster [DBG] pgmap v499: 233 pgs: 1 active+remapped+backfill_toofull, 232 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 13/1104 objects misplaced (1.178%) 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:56.231736+0000 osd.6 (osd.6) 121 : cluster [DBG] 6.c scrub starts 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:56.232455+0000 osd.6 (osd.6) 122 : cluster [DBG] 6.c scrub ok 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:56.334726+0000 osd.6 (osd.6) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:57.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:56.335039+0000 osd.6 (osd.6) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:57.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.060099+0000 mon.a (mon.0) 2293 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:34:57.602 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:34:57.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:55.608002+0000 osd.7 (osd.7) 167 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:34:57.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:55.609244+0000 osd.7 (osd.7) 168 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:34:57.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:55.627844+0000 mgr.y (mgr.24437) 332 : cluster [DBG] pgmap v499: 233 pgs: 1 active+remapped+backfill_toofull, 232 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 13/1104 objects misplaced (1.178%) 2024-02-23T00:34:57.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:56.231736+0000 osd.6 (osd.6) 121 : cluster [DBG] 6.c scrub starts 2024-02-23T00:34:57.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:56.232455+0000 osd.6 (osd.6) 122 : cluster [DBG] 6.c scrub ok 2024-02-23T00:34:57.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:56.334726+0000 osd.6 (osd.6) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:57.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:56.335039+0000 osd.6 (osd.6) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:57.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.060099+0000 mon.a (mon.0) 2293 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:34:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:57.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:34:57.707 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:57.736 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:57.789 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:57.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:57.892 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:57.921 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:58.070 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:58.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.070+0000 7f775affd700 1 -- 172.21.15.87:0/983994112 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v347) v1 ==== 133+0+0 (secure 0 0 0) 0x7f775c0503b0 con 0x7f77640fe510 2024-02-23T00:34:58.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:58.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:58.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.074+0000 7f776c098700 10 librados: watch_flush enter 2024-02-23T00:34:58.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.074+0000 7f776c098700 10 librados: watch_flush exit 2024-02-23T00:34:58.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.074+0000 7f776c098700 20 client.37529.objecter shutdown clearing up homeless session... 2024-02-23T00:34:58.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.074+0000 7f776c098700 10 client.37529.objecter successfully canceled tick 2024-02-23T00:34:58.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7750062c90 msgr2=0x7f7750065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:58.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 --2- 172.21.15.87:0/983994112 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7750062c90 0x7f7750065150 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 rx=0x7f7754006fd0 tx=0x7f7754009060).stop 2024-02-23T00:34:58.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77640fe510 msgr2=0x7f776410a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:34:58.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77640fe510 0x7f776410a8d0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f775c00ec20 tx=0x7f775c00b040).stop 2024-02-23T00:34:58.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 shutdown_connections 2024-02-23T00:34:58.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 --2- 172.21.15.87:0/983994112 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7750062c90 0x7f7750065150 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:58.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77640fe510 0x7f776410a8d0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:58.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77640f8420 0x7f776410f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:58.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 --2- 172.21.15.87:0/983994112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77640fdbb0 0x7f776410f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:34:58.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 >> 172.21.15.87:0/983994112 conn(0x7f77640f3800 msgr2=0x7f77640f4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:34:58.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 shutdown_connections 2024-02-23T00:34:58.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 -- 172.21.15.87:0/983994112 wait complete. 2024-02-23T00:34:58.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:34:58.075+0000 7f776c098700 1 librados: shutdown 2024-02-23T00:34:58.079 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-23T00:34:58.176 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:58.202 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:58.300 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:58.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:58.404 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:34:58.432 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:34:58.434 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:58.454 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:34:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:58.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:34:58.558 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:34:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:56.401809+0000 osd.5 (osd.5) 147 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:34:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:56.402845+0000 osd.5 (osd.5) 148 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:34:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:56.678339+0000 osd.4 (osd.4) 155 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:34:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:56.679131+0000 osd.4 (osd.4) 156 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:57.072490+0000 mon.a (mon.0) 2294 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:57.077115+0000 mon.b (mon.1) 299 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.096491+0000 osd.5 (osd.5) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.096637+0000 osd.5 (osd.5) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:57.277146+0000 mon.c (mon.2) 392 : audit [INF] from='client.? 172.21.15.87:0/983994112' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.321899+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:57.322900+0000 mon.a (mon.0) 2296 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:57.323155+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:57.401567+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? 172.21.15.87:0/4144670975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.641624+0000 osd.7 (osd.7) 169 : cluster [DBG] 63.0 scrub starts 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.645200+0000 osd.7 (osd.7) 170 : cluster [DBG] 63.0 scrub ok 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.657840+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.660950+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.855546+0000 osd.1 (osd.1) 155 : cluster [DBG] 63.1c scrub starts 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.859042+0000 osd.1 (osd.1) 156 : cluster [DBG] 63.1c scrub ok 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:58.071170+0000 mon.a (mon.0) 2298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi087-82019-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi087-82019-60"}]': finished 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:58.071282+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:58.071361+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:58.071432+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:58.072891+0000 mon.b (mon.1) 300 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-23T00:34:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:58.073178+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:56.401809+0000 osd.5 (osd.5) 147 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:56.402845+0000 osd.5 (osd.5) 148 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:56.678339+0000 osd.4 (osd.4) 155 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:56.679131+0000 osd.4 (osd.4) 156 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:57.072490+0000 mon.a (mon.0) 2294 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:57.077115+0000 mon.b (mon.1) 299 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.096491+0000 osd.5 (osd.5) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.096637+0000 osd.5 (osd.5) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:57.277146+0000 mon.c (mon.2) 392 : audit [INF] from='client.? 172.21.15.87:0/983994112' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.321899+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:57.322900+0000 mon.a (mon.0) 2296 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:34:58.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:57.323155+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:57.401567+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? 172.21.15.87:0/4144670975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.641624+0000 osd.7 (osd.7) 169 : cluster [DBG] 63.0 scrub starts 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.645200+0000 osd.7 (osd.7) 170 : cluster [DBG] 63.0 scrub ok 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.657840+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.660950+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.855546+0000 osd.1 (osd.1) 155 : cluster [DBG] 63.1c scrub starts 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.859042+0000 osd.1 (osd.1) 156 : cluster [DBG] 63.1c scrub ok 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:58.071170+0000 mon.a (mon.0) 2298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi087-82019-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi087-82019-60"}]': finished 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:58.071282+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:58.071361+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:58.071432+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-02-23T00:34:58.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:58.072891+0000 mon.b (mon.1) 300 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-23T00:34:58.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:58.073178+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-23T00:34:58.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:56.401809+0000 osd.5 (osd.5) 147 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:34:58.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:56.402845+0000 osd.5 (osd.5) 148 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:34:58.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:56.678339+0000 osd.4 (osd.4) 155 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:34:58.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:56.679131+0000 osd.4 (osd.4) 156 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:34:58.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:57.072490+0000 mon.a (mon.0) 2294 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:57.077115+0000 mon.b (mon.1) 299 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.096491+0000 osd.5 (osd.5) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.096637+0000 osd.5 (osd.5) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:57.277146+0000 mon.c (mon.2) 392 : audit [INF] from='client.? 172.21.15.87:0/983994112' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.321899+0000 mon.a (mon.0) 2295 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:57.322900+0000 mon.a (mon.0) 2296 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:57.323155+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:57.401567+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? 172.21.15.87:0/4144670975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.641624+0000 osd.7 (osd.7) 169 : cluster [DBG] 63.0 scrub starts 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.645200+0000 osd.7 (osd.7) 170 : cluster [DBG] 63.0 scrub ok 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.657840+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.660950+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.855546+0000 osd.1 (osd.1) 155 : cluster [DBG] 63.1c scrub starts 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.859042+0000 osd.1 (osd.1) 156 : cluster [DBG] 63.1c scrub ok 2024-02-23T00:34:58.589 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:58.071170+0000 mon.a (mon.0) 2298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi087-82019-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi087-82019-60"}]': finished 2024-02-23T00:34:58.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:58.071282+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:34:58.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:58.071361+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-23T00:34:58.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:58.071432+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-02-23T00:34:58.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:58.072891+0000 mon.b (mon.1) 300 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-23T00:34:58.590 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:58.073178+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-23T00:34:58.590 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:58.592 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:34:58.592 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:34:58.592 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:34:58.592 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:34:58.593 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.593 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24712.0:14 7.5 7:a3910452:test-rados-api-smithi087-82115-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-23T00:34:58.593 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:30:58.506107+0000", 2024-02-23T00:34:58.593 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 240.08113244800001, 2024-02-23T00:34:58.593 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021562957000000001, 2024-02-23T00:34:58.593 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.593 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24712", 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2285581802", 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 14 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506107+0000", 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506107+0000", 2024-02-23T00:34:58.594 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:58.595 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.595 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.595 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.595 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506105+0000", 2024-02-23T00:34:58.595 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5230000000000001e-06 2024-02-23T00:34:58.595 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.595 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.595 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.595 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506114+0000", 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1732999999999999e-05 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506126+0000", 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4780000000000002e-06 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506133+0000", 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9349999999999999e-05 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.596 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506193+0000", 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012566300000000001 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506318+0000", 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017116500000000001 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0,2", 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506490+0000", 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072723850000000001 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.597 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.513762+0000", 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046023009999999996 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.518364+0000", 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0092229960000000007 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.527587+0000", 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5426999999999998e-05 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.598 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.527623+0000", 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7034999999999999e-05 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.527670+0000", 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021562957000000001 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.599 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26036.0:2 64.6 64:602f83fe:::foo:head [write 0~128,set-alloc-hint object_size 0 write_size 0] snapc 0=[] ondisk+write+known_if_redirected e115)", 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 232.362363577, 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036711013000000001, 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26036", 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3255892024", 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.600 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:34:58.601 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224873+0000", 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.163e-05 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224885+0000", 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8150000000000001e-05 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.602 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.603 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.603 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224903+0000", 2024-02-23T00:34:58.603 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2343e-05 2024-02-23T00:34:58.603 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.603 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.603 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.603 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224915+0000", 2024-02-23T00:34:58.603 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6192000000000005e-05 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224992+0000", 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4450000000000001e-06 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224998+0000", 2024-02-23T00:34:58.604 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032120923000000003 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257119+0000", 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125234 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257244+0000", 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107636 2024-02-23T00:34:58.605 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,5", 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257352+0000", 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011677390000000001 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.258520+0000", 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014228299999999999 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:58.606 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.259942+0000", 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015894640000000001 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261532+0000", 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.154e-05 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261563+0000", 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3230999999999998e-05 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261587+0000", 2024-02-23T00:34:58.607 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036711013000000001 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDScrubReserve(10.1 RELEASE e139)", 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:30.036961+0000", 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 208.55027816699999, 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011677659, 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.608 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036961+0000", 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036961+0000", 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036960+0000", 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6599999999999998e-06 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.609 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036967+0000", 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3458000000000001e-05 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036980+0000", 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.007e-06 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036987+0000", 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6214e-05 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.610 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037053+0000", 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8050000000000007e-06 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037061+0000", 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011533062 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.048594+0000", 2024-02-23T00:34:58.611 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8111999999999999e-05 2024-02-23T00:34:58.612 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.612 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.612 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.612 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.048612+0000", 2024-02-23T00:34:58.612 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6262999999999999e-05 2024-02-23T00:34:58.612 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.612 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.613 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.613 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.048639+0000", 2024-02-23T00:34:58.613 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011677659 2024-02-23T00:34:58.613 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.613 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.613 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.613 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.613 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:2 93.7s0 93:f24ec45b:test-rados-api-smithi087-82107-15::foo:head [write 0~8192] snapc 2=[2] ondisk+write+known_if_redirected e143)", 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:34.002899+0000", 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 204.584340016, 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079244069999999996, 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.614 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002899+0000", 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002899+0000", 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002897+0000", 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5637e-05 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.615 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.616 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.616 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002913+0000", 2024-02-23T00:34:58.616 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0621e-05 2024-02-23T00:34:58.616 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.616 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.616 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.616 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002923+0000", 2024-02-23T00:34:58.616 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1270000000000003e-06 2024-02-23T00:34:58.616 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002932+0000", 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00029541399999999997 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.003227+0000", 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1740000000000002e-06 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:34:58.617 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.003231+0000", 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034875129999999998 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.006719+0000", 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000150543 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.006869+0000", 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000472211 2024-02-23T00:34:58.618 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.007341+0000", 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00075650800000000003 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.008098+0000", 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00265423 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.619 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.010752+0000", 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1404999999999998e-05 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.010824+0000", 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079244069999999996 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:4 93.7s0 93:f24ec45b:test-rados-api-smithi087-82107-15::foo:head [write 8192~8192] snapc 3=[3,2] ondisk+write+known_if_redirected e144)", 2024-02-23T00:34:58.620 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 203.59106436299999, 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028560524, 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 4 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.621 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996173+0000", 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1501e-05 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.622 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.623 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.623 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996194+0000", 2024-02-23T00:34:58.623 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1659000000000003e-05 2024-02-23T00:34:58.623 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.623 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.623 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.623 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996226+0000", 2024-02-23T00:34:58.623 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.439e-06 2024-02-23T00:34:58.624 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.624 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.624 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.624 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996234+0000", 2024-02-23T00:34:58.624 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013176539999999999 2024-02-23T00:34:58.624 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.624 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.624 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.624 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.997552+0000", 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5900000000000008e-06 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.997560+0000", 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0057013619999999997 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.003262+0000", 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015312100000000001 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.625 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.003415+0000", 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033453049999999998 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.006760+0000", 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038155379999999998 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.010576+0000", 2024-02-23T00:34:58.626 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014109313 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.024685+0000", 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0339e-05 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.024735+0000", 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028560524 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.627 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:9 93.7s0 93:f24ec45b:test-rados-api-smithi087-82107-15::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e146)", 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:36.987620+0000", 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 201.59961902399999, 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017235619000000001, 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:34:58.628 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 9 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987620+0000", 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987620+0000", 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.629 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987618+0000", 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7619999999999995e-06 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987626+0000", 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0876999999999999e-05 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987637+0000", 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4093e-05 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.630 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987651+0000", 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001414867 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.989066+0000", 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5760000000000006e-06 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.989074+0000", 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040788279999999996 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.631 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.993153+0000", 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100617 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.993254+0000", 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038279699999999999 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.993637+0000", 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000640954 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.632 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.994278+0000", 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010509697 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.004787+0000", 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8483000000000001e-05 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.004856+0000", 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017235619000000001 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.633 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.634 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.634 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.634 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:11 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [write 0~8192] snapc 6=[6] ondisk+write+known_if_redirected e147)", 2024-02-23T00:34:58.634 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:37.991540+0000", 2024-02-23T00:34:58.634 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 200.59569889400001, 2024-02-23T00:34:58.634 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020032722999999999, 2024-02-23T00:34:58.634 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.634 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 11 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991540+0000", 2024-02-23T00:34:58.635 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991540+0000", 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991538+0000", 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2162000000000001e-05 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.636 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991570+0000", 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4e-05 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991584+0000", 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1769999999999998e-06 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.637 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991593+0000", 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016113087000000002 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.007706+0000", 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016325899999999999 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.007869+0000", 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000521098 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.638 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.008390+0000", 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0016975200000000001 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.010088+0000", 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001418084 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.011506+0000", 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7416999999999995e-05 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.639 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.640 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.640 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.011573+0000", 2024-02-23T00:34:58.640 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020032722999999999 2024-02-23T00:34:58.640 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.640 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.640 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.640 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.640 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.640 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:13 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [write 8192~8192] snapc 7=[7,6] ondisk+write+known_if_redirected e148)", 2024-02-23T00:34:58.643 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:38.986625+0000", 2024-02-23T00:34:58.643 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 199.60061394499999, 2024-02-23T00:34:58.643 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010875549999999999, 2024-02-23T00:34:58.643 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.643 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.643 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.643 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:34:58.643 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 13 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986625+0000", 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.644 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986625+0000", 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986623+0000", 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1804000000000002e-05 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986665+0000", 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2981e-05 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.645 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986678+0000", 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6139999999999992e-06 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986687+0000", 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001460254 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.988147+0000", 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9940000000000006e-06 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.646 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:34:58.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.988156+0000", 2024-02-23T00:34:58.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020480580000000002 2024-02-23T00:34:58.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.647 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.647 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.990204+0000", 2024-02-23T00:34:58.647 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116233 2024-02-23T00:34:58.647 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.647 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.990320+0000", 2024-02-23T00:34:58.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00062543799999999997 2024-02-23T00:34:58.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.648 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.648 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.648 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.990946+0000", 2024-02-23T00:34:58.648 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001065908 2024-02-23T00:34:58.648 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.992012+0000", 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054126460000000001 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.997424+0000", 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6672e-05 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.997501+0000", 2024-02-23T00:34:58.649 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010875549999999999 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:14 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [rollback 7] snapc 7=[7,6] ondisk+write+known_if_redirected e148)", 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:38.998508+0000", 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 199.58873169, 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010648487, 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:34:58.650 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 14 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998508+0000", 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998508+0000", 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.651 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998506+0000", 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5909999999999998e-06 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998513+0000", 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1420000000000001e-05 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998525+0000", 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7179999999999999e-06 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.652 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998531+0000", 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111612 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998643+0000", 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000123659 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998767+0000", 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000483432 2024-02-23T00:34:58.653 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.999250+0000", 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0069559019999999999 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.006206+0000", 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028810699999999999 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.009087+0000", 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8885000000000002e-05 2024-02-23T00:34:58.654 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.009156+0000", 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010648487 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDECSubOpWriteReply(93.0s0 148/142 ECSubWriteReply(tid=250, last_complete=148'4, committed=1, applied=1))", 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:39.001273+0000", 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 199.58596602200001, 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076685520000000004, 2024-02-23T00:34:58.655 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001273+0000", 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001273+0000", 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.656 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001273+0000", 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2320000000000001e-06 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001276+0000", 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9399999999999999e-06 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001282+0000", 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9769999999999997e-06 2024-02-23T00:34:58.657 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001288+0000", 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00763885 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.008927+0000", 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4881e-05 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.008942+0000", 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076685520000000004 2024-02-23T00:34:58.658 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDECSubOpWriteReply(93.0s0 148/142 ECSubWriteReply(tid=249, last_complete=148'3, committed=1, applied=1))", 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:39.002324+0000", 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 199.584915696, 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066686360000000004, 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.002324+0000", 2024-02-23T00:34:58.659 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.002324+0000", 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.002323+0000", 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.588e-06 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.660 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.002330+0000", 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1884e-05 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.002342+0000", 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7740000000000004e-06 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.002350+0000", 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066125350000000001 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.008962+0000", 2024-02-23T00:34:58.661 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9717e-05 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.008992+0000", 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066686360000000004 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:18 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [delete] snapc 7=[7,6] ondisk+write+peerstat_old+known_if_redirected e150)", 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:34:58.662 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 197.59598812600001, 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026236400999999999, 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 18 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.663 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991249+0000", 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0203999999999999e-05 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991259+0000", 2024-02-23T00:34:58.664 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5091e-05 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991274+0000", 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.729e-06 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991282+0000", 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7206999999999995e-05 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991349+0000", 2024-02-23T00:34:58.665 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134998 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991484+0000", 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00048211400000000001 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991966+0000", 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00039970199999999999 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.992366+0000", 2024-02-23T00:34:58.666 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025049227 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.017415+0000", 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2229000000000003e-05 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.017488+0000", 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026236400999999999 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.667 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:874 95.7 95:e6233af9:test-rados-api-smithi087-82091-14::foo5:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:42.006991+0000", 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 196.58024778399999, 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082505240000000004, 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 874 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.668 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.006991+0000", 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.006991+0000", 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.006989+0000", 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4066999999999999e-05 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.669 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007003+0000", 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4657999999999999e-05 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007018+0000", 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6989999999999997e-06 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007025+0000", 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5380999999999998e-05 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.670 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007070+0000", 2024-02-23T00:34:58.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000206398 2024-02-23T00:34:58.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.671 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.671 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:34:58.671 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007276+0000", 2024-02-23T00:34:58.671 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002540994 2024-02-23T00:34:58.671 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.672 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.009817+0000", 2024-02-23T00:34:58.672 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8937999999999994e-05 2024-02-23T00:34:58.672 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.672 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.672 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:34:58.673 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.009916+0000", 2024-02-23T00:34:58.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050415399999999997 2024-02-23T00:34:58.673 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.673 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.673 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.673 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.014958+0000", 2024-02-23T00:34:58.673 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0696000000000003e-05 2024-02-23T00:34:58.673 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.673 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.674 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.015019+0000", 2024-02-23T00:34:58.674 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000223322 2024-02-23T00:34:58.674 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.674 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.674 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.675 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.015242+0000", 2024-02-23T00:34:58.675 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082505240000000004 2024-02-23T00:34:58.675 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.675 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.675 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.675 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.675 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.676 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26475.0:2 125.4s0 125:3c44b853:test-rados-api-smithi087-82115-21::foo:head [write 0~8192] snapc 2=[2] ondisk+write+known_if_redirected e182)", 2024-02-23T00:34:58.676 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:13.006424+0000", 2024-02-23T00:34:58.676 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 165.580815567, 2024-02-23T00:34:58.676 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018917415999999999, 2024-02-23T00:34:58.676 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.676 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.676 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.677 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26475", 2024-02-23T00:34:58.677 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/1917629916", 2024-02-23T00:34:58.677 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:34:58.677 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.677 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.677 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.678 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006424+0000", 2024-02-23T00:34:58.678 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.678 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.678 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.678 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.678 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006424+0000", 2024-02-23T00:34:58.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:58.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.679 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.679 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.679 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006422+0000", 2024-02-23T00:34:58.679 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4351e-05 2024-02-23T00:34:58.679 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006436+0000", 2024-02-23T00:34:58.680 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2189e-05 2024-02-23T00:34:58.680 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.680 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.680 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.680 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006448+0000", 2024-02-23T00:34:58.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7499999999999997e-06 2024-02-23T00:34:58.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.681 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.681 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.681 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006455+0000", 2024-02-23T00:34:58.681 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0914000000000001e-05 2024-02-23T00:34:58.681 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.682 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.682 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006506+0000", 2024-02-23T00:34:58.682 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.571e-06 2024-02-23T00:34:58.682 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.682 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:34:58.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006509+0000", 2024-02-23T00:34:58.683 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013345159000000001 2024-02-23T00:34:58.683 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.683 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.683 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.683 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.019854+0000", 2024-02-23T00:34:58.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000106708 2024-02-23T00:34:58.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.684 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.684 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.019960+0000", 2024-02-23T00:34:58.684 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036460300000000002 2024-02-23T00:34:58.684 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.684 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.020325+0000", 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00071362200000000002 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.021039+0000", 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042323819999999998 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.025271+0000", 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0007999999999998e-05 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.685 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.025341+0000", 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018917415999999999 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36947.0:2 126.3 126:c1e600bb:test-rados-api-smithi087-82083-31::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:13.009044+0000", 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 165.57819513999999, 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015740071000000001, 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.686 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36947", 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3435478986", 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009044+0000", 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009044+0000", 2024-02-23T00:34:58.687 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009043+0000", 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9830000000000008e-06 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009053+0000", 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3186e-05 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.688 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009066+0000", 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7959999999999998e-06 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009073+0000", 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016034999999999999 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009233+0000", 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5100000000000004e-06 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009240+0000", 2024-02-23T00:34:58.689 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011721904 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.020961+0000", 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8343000000000004e-05 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.021060+0000", 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012799299999999999 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,4", 2024-02-23T00:34:58.690 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.021188+0000", 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00049047399999999997 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.021678+0000", 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00064367000000000005 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.022322+0000", 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023954129999999999 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:34:58.691 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.024717+0000", 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6732e-05 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.024754+0000", 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0110000000000001e-05 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.024784+0000", 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015740071000000001 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.692 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26475.0:4 125.4s0 125:3c44b853:test-rados-api-smithi087-82115-21::foo:head [write 8192~8192] snapc 3=[3,2] ondisk+write+known_if_redirected e183)", 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:14.242083+0000", 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 164.34515638100001, 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014358816999999999, 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26475", 2024-02-23T00:34:58.693 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/1917629916", 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 4 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242083+0000", 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242083+0000", 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:34:58.694 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.695 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.695 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242081+0000", 2024-02-23T00:34:58.695 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2780999999999997e-05 2024-02-23T00:34:58.695 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.695 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.696 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242114+0000", 2024-02-23T00:34:58.696 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0647000000000001e-05 2024-02-23T00:34:58.696 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.696 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.696 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.697 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242135+0000", 2024-02-23T00:34:58.697 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1789999999999996e-06 2024-02-23T00:34:58.697 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.697 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.697 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242141+0000", 2024-02-23T00:34:58.698 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5528000000000002e-05 2024-02-23T00:34:58.698 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.698 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.698 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.698 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242186+0000", 2024-02-23T00:34:58.699 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0899999999999999e-06 2024-02-23T00:34:58.699 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.699 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.699 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:34:58.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242194+0000", 2024-02-23T00:34:58.700 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0089040109999999999 2024-02-23T00:34:58.700 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.700 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.700 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.700 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.251098+0000", 2024-02-23T00:34:58.701 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9748999999999997e-05 2024-02-23T00:34:58.701 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.701 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.701 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.251197+0000", 2024-02-23T00:34:58.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00061586500000000005 2024-02-23T00:34:58.702 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.702 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.702 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.702 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.251813+0000", 2024-02-23T00:34:58.702 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00098538699999999989 2024-02-23T00:34:58.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.703 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.252799+0000", 2024-02-23T00:34:58.703 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035701940000000001 2024-02-23T00:34:58.703 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.703 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.703 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.704 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.256369+0000", 2024-02-23T00:34:58.704 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2904000000000005e-05 2024-02-23T00:34:58.704 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.704 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.705 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.705 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.256442+0000", 2024-02-23T00:34:58.705 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014358816999999999 2024-02-23T00:34:58.705 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.705 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.705 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.705 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.706 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.706 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.44587.0:1 151.6s0 151:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-02-23T00:34:58.706 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:34:58.706 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 128.54892061699999, 2024-02-23T00:34:58.706 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763, 2024-02-23T00:34:58.706 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.706 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.707 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.707 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.44587", 2024-02-23T00:34:58.707 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/843545378", 2024-02-23T00:34:58.707 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:34:58.707 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.707 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.707 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.707 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:34:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:34:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:34:58.708 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038316+0000", 2024-02-23T00:34:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9619999999999999e-06 2024-02-23T00:34:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.709 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038325+0000", 2024-02-23T00:34:58.710 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7385e-05 2024-02-23T00:34:58.710 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.710 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.710 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.710 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038343+0000", 2024-02-23T00:34:58.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0739999999999999e-06 2024-02-23T00:34:58.711 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.711 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.711 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.711 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038352+0000", 2024-02-23T00:34:58.711 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077681699999999996 2024-02-23T00:34:58.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.712 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.712 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.712 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039128+0000", 2024-02-23T00:34:58.712 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0860000000000002e-06 2024-02-23T00:34:58.712 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:34:58.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039132+0000", 2024-02-23T00:34:58.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0036840789999999 2024-02-23T00:34:58.713 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.713 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.713 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.713 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042816+0000", 2024-02-23T00:34:58.713 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000158937 2024-02-23T00:34:58.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.714 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042975+0000", 2024-02-23T00:34:58.714 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000568418 2024-02-23T00:34:58.714 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.714 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.714 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.043543+0000", 2024-02-23T00:34:58.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00061003399999999997 2024-02-23T00:34:58.716 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.716 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.716 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.716 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.044153+0000", 2024-02-23T00:34:58.716 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011287789999999999 2024-02-23T00:34:58.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.717 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.717 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045282+0000", 2024-02-23T00:34:58.717 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9601e-05 2024-02-23T00:34:58.717 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.717 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.718 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.718 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045351+0000", 2024-02-23T00:34:58.718 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763 2024-02-23T00:34:58.718 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.718 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.718 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.719 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.719 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.719 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:2089 160.4 160:20000000:%01:::head [pgnls start_epoch 238] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e238)", 2024-02-23T00:34:58.719 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:09.044476+0000", 2024-02-23T00:34:58.719 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 109.54276311, 2024-02-23T00:34:58.719 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066920149999999999, 2024-02-23T00:34:58.719 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.720 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:34:58.720 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.720 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:34:58.720 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:34:58.720 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2089 2024-02-23T00:34:58.720 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.720 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.720 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.044476+0000", 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.044476+0000", 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.044474+0000", 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9640000000000003e-06 2024-02-23T00:34:58.721 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.044482+0000", 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2336e-05 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.044495+0000", 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7469999999999999e-06 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.044501+0000", 2024-02-23T00:34:58.722 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9223e-05 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.044541+0000", 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1909999999999999e-06 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.044546+0000", 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0065079370000000001 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.723 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.051054+0000", 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0057e-05 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.051094+0000", 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4255e-05 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.051168+0000", 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066920149999999999 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.724 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26886.0:1 181.5s0 181:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e278)", 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 69.540918372999997, 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133, 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26886", 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/134760323", 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.725 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046318+0000", 2024-02-23T00:34:58.726 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9741999999999998e-05 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046338+0000", 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2594e-05 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046351+0000", 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5399999999999996e-06 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046359+0000", 2024-02-23T00:34:58.727 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00098343299999999992 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047343+0000", 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9160000000000003e-06 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047347+0000", 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0029936850000001 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.728 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050340+0000", 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014100300000000001 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050481+0000", 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038932099999999998 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050871+0000", 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00045121199999999999 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.729 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.051322+0000", 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00089874699999999998 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052221+0000", 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1542999999999998e-05 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.730 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052272+0000", 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27003.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:34:58.731 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 24.526764674999999, 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568, 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27003", 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3963489775", 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:34:58.732 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060473+0000", 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.417e-06 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060479+0000", 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8080000000000005e-06 2024-02-23T00:34:58.733 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060486+0000", 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4359999999999999e-06 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060493+0000", 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6943999999999997e-05 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060530+0000", 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4229999999999999e-06 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.734 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060532+0000", 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.99955465700000001 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060087+0000", 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2388999999999999e-05 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.735 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:34:58.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060179+0000", 2024-02-23T00:34:58.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00037435399999999997 2024-02-23T00:34:58.736 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.736 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.736 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:34:58.736 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060554+0000", 2024-02-23T00:34:58.736 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00057048899999999998 2024-02-23T00:34:58.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:34:58.737 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061124+0000", 2024-02-23T00:34:58.737 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00075843999999999998 2024-02-23T00:34:58.737 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.737 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.737 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061883+0000", 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8547999999999998e-05 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061921+0000", 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:34:58.738 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:34:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:58.740 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:34:58.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:34:58.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:58.868 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:59.060 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:34:59.067 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:59.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:34:59.174 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:59.202 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:59.350 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:59.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:59.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:59.362 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-23T00:34:59.454 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:34:59.483 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:59.502 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:34:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:59.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:34:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.628518+0000 mgr.y (mgr.24437) 333 : cluster [DBG] pgmap v502: 233 pgs: 1 active+remapped+backfill_toofull, 232 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 13/1104 objects misplaced (1.178%) 2024-02-23T00:34:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.949062+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.d scrub starts 2024-02-23T00:34:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:57.950445+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.d scrub ok 2024-02-23T00:34:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:58.215750+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:34:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:58.218700+0000 osd.6 (osd.6) 126 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:34:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:58.659222+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:34:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:58.660103+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:34:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:59.073331+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-23T00:34:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:59.073478+0000 mon.a (mon.0) 2304 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:59.077182+0000 mon.b (mon.1) 301 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:59.077528+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.628518+0000 mgr.y (mgr.24437) 333 : cluster [DBG] pgmap v502: 233 pgs: 1 active+remapped+backfill_toofull, 232 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 13/1104 objects misplaced (1.178%) 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.949062+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.d scrub starts 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:57.950445+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.d scrub ok 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:58.215750+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:58.218700+0000 osd.6 (osd.6) 126 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:58.659222+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:58.660103+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:59.073331+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-23T00:34:59.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:59.073478+0000 mon.a (mon.0) 2304 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-02-23T00:34:59.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:59.077182+0000 mon.b (mon.1) 301 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-23T00:34:59.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:34:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:59.077528+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-23T00:34:59.606 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.628518+0000 mgr.y (mgr.24437) 333 : cluster [DBG] pgmap v502: 233 pgs: 1 active+remapped+backfill_toofull, 232 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s; 13/1104 objects misplaced (1.178%) 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.949062+0000 osd.2 (osd.2) 135 : cluster [DBG] 5.d scrub starts 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:57.950445+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.d scrub ok 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:58.215750+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:58.218700+0000 osd.6 (osd.6) 126 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:58.659222+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:58.660103+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:59.073331+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:59.073478+0000 mon.a (mon.0) 2304 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:59.077182+0000 mon.b (mon.1) 301 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-23T00:34:59.632 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:34:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:59.077528+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-23T00:34:59.633 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:59.652 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:59.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:59.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:34:59.783 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:59.800 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:34:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:59.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:34:59.904 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:34:59.933 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:34:59.986 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:34:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:34:59.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:00.089 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:00.116 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:00.134 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:00.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:00.238 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:00.264 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:00.283 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:00.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:00.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:00.389 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:00.417 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:00.442 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:35:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:59.445353+0000 osd.4 (osd.4) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:00.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:59.445482+0000 osd.4 (osd.4) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:00.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:59.626652+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.e scrub starts 2024-02-23T00:35:00.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:59.629823+0000 mon.a (mon.0) 2306 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]: dispatch 2024-02-23T00:35:00.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:59.629990+0000 mon.a (mon.0) 2307 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2024-02-23T00:35:00.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:59.630162+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.e scrub ok 2024-02-23T00:35:00.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:34:59.930237+0000 mon.a (mon.0) 2308 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:00.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:00.076287+0000 mon.a (mon.0) 2309 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-23T00:35:00.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:00.076398+0000 mon.a (mon.0) 2310 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]': finished 2024-02-23T00:35:00.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:00.076461+0000 mon.a (mon.0) 2311 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]': finished 2024-02-23T00:35:00.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:00.076507+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-02-23T00:35:00.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:00.082013+0000 mon.c (mon.2) 394 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:00.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:00.082730+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:00.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:00.109158+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:00.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:00.109362+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:00.600 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:59.445353+0000 osd.4 (osd.4) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:00.600 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:59.445482+0000 osd.4 (osd.4) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:00.600 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:59.626652+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.e scrub starts 2024-02-23T00:35:00.600 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:59.629823+0000 mon.a (mon.0) 2306 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]: dispatch 2024-02-23T00:35:00.600 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:59.629990+0000 mon.a (mon.0) 2307 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2024-02-23T00:35:00.600 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:59.630162+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.e scrub ok 2024-02-23T00:35:00.600 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:34:59.930237+0000 mon.a (mon.0) 2308 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:00.600 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:00.076287+0000 mon.a (mon.0) 2309 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-23T00:35:00.600 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:00.076398+0000 mon.a (mon.0) 2310 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]': finished 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:00.076461+0000 mon.a (mon.0) 2311 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]': finished 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:00.076507+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:00.082013+0000 mon.c (mon.2) 394 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:00.082730+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:00.109158+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:00.109362+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:59.445353+0000 osd.4 (osd.4) 161 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:59.445482+0000 osd.4 (osd.4) 162 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:59.626652+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.e scrub starts 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:59.629823+0000 mon.a (mon.0) 2306 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]: dispatch 2024-02-23T00:35:00.601 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:59.629990+0000 mon.a (mon.0) 2307 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]: dispatch 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:59.630162+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.e scrub ok 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:34:59.930237+0000 mon.a (mon.0) 2308 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:00.076287+0000 mon.a (mon.0) 2309 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:00.076398+0000 mon.a (mon.0) 2310 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.9"}]': finished 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:00.076461+0000 mon.a (mon.0) 2311 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.9"}]': finished 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:00.076507+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:00.082013+0000 mon.c (mon.2) 394 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:00.082730+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:00.109158+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:00.602 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:00.109362+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:00.632 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:00.633 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:00.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:00.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:00.736 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:00.766 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:00.785 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:00.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:00.889 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:00.916 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:00.937 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:00.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:01.040 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:01.068 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:01.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:35:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:35:01.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/236874774 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa2a80f8430 msgr2=0x7fa2a8104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:01.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/236874774 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa2a80f8430 0x7fa2a8104a30 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7fa2a0009c30 tx=0x7fa2a0004c50).stop 2024-02-23T00:35:01.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/236874774 shutdown_connections 2024-02-23T00:35:01.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/236874774 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa2a80f8430 0x7fa2a8104a30 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:01.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/236874774 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa2a80fdbc0 0x7fa2a80fdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:01.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/236874774 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2a80fe520 0x7fa2a80f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:01.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/236874774 >> 172.21.15.87:0/236874774 conn(0x7fa2a80f37d0 msgr2=0x7fa2a80f5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/236874774 shutdown_connections 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/236874774 wait complete. 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 librados: starting msgr at 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 librados: starting objecter 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 Processor -- start 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 -- start start 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 librados: setting wanted keys 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.177+0000 7fa2ad9fa700 1 librados: calling monclient init 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa2a80f8430 0x7fa2a806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:01.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa2a80fdbc0 0x7fa2a806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2a80fe520 0x7fa2a806bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a81163e0 con 0x7fa2a80fe520 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8116550 con 0x7fa2a80f8430 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8116710 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2a80fe520 0x7fa2a806bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a67fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa2a80fdbc0 0x7fa2a806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2a80fe520 0x7fa2a806bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47086/0 (socket says 172.21.15.87:47086) 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a77fe700 1 -- 172.21.15.87:0/2946546126 learned_addr learned my addr 172.21.15.87:0/2946546126 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a6ffd700 1 --2- 172.21.15.87:0/2946546126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa2a80f8430 0x7fa2a806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a67fc700 1 -- 172.21.15.87:0/2946546126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa2a80f8430 msgr2=0x7fa2a806b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a67fc700 1 --2- 172.21.15.87:0/2946546126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa2a80f8430 0x7fa2a806b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a67fc700 1 -- 172.21.15.87:0/2946546126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2a80fe520 msgr2=0x7fa2a806bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a67fc700 1 --2- 172.21.15.87:0/2946546126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2a80fe520 0x7fa2a806bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:01.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a67fc700 1 -- 172.21.15.87:0/2946546126 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c005040 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a77fe700 1 --2- 172.21.15.87:0/2946546126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2a80fe520 0x7fa2a806bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2a67fc700 1 --2- 172.21.15.87:0/2946546126 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa2a80fdbc0 0x7fa2a806b5c0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7fa29c0091f0 tx=0x7fa29c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa28ffff700 10 client.?.objecter ms_handle_connect 0x7fa2a80fdbc0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa28ffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa28ffff700 1 -- 172.21.15.87:0/2946546126 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c0079e0 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa28ffff700 1 -- 172.21.15.87:0/2946546126 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa29c00f040 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a00097c0 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa28ffff700 1 -- 172.21.15.87:0/2946546126 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29c014950 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 10 client.37541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a8116da0 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.178+0000 7fa2ad9fa700 1 librados: init done 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.179+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a805f990 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.180+0000 7fa28ffff700 1 -- 172.21.15.87:0/2946546126 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fa29c0104f0 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.180+0000 7fa28ffff700 1 --2- 172.21.15.87:0/2946546126 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa290062c90 0x7fa290065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:01.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.180+0000 7fa28ffff700 1 -- 172.21.15.87:0/2946546126 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(350..350 src has 173..350) v4 ==== 8251+0+0 (secure 0 0 0) 0x7fa29c07d120 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.180+0000 7fa28ffff700 10 client.37541.objecter ms_dispatch 0x7fa2a8001550 osd_map(350..350 src has 173..350) v4 2024-02-23T00:35:01.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.180+0000 7fa28ffff700 3 client.37541.objecter handle_osd_map got epochs [350,350] > 0 2024-02-23T00:35:01.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.180+0000 7fa28ffff700 3 client.37541.objecter handle_osd_map decoding full epoch 350 2024-02-23T00:35:01.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.180+0000 7fa2a6ffd700 1 --2- 172.21.15.87:0/2946546126 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa290062c90 0x7fa290065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:01.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.181+0000 7fa28ffff700 20 client.37541.objecter dump_active .. 0 homeless 2024-02-23T00:35:01.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.181+0000 7fa28ffff700 10 client.37541.objecter ms_handle_connect 0x7fa290062c90 2024-02-23T00:35:01.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.181+0000 7fa2a6ffd700 1 --2- 172.21.15.87:0/2946546126 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa290062c90 0x7fa290065150 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 rx=0x7fa298009170 tx=0x7fa298009da0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:01.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.184+0000 7fa28ffff700 1 -- 172.21.15.87:0/2946546126 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa29c050550 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.224 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:01.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:01.328 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:01.357 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:01.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:01.372+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa2a805ee80 con 0x7fa2a80fdbc0 2024-02-23T00:35:01.377 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:01.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:01.481 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:34:59.628916+0000 mgr.y (mgr.24437) 334 : cluster [DBG] pgmap v505: 240 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 231 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1086 objects misplaced (1.197%) 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:00.413508+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:00.416488+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:00.617556+0000 osd.7 (osd.7) 171 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:00.621151+0000 osd.7 (osd.7) 172 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:00.831990+0000 osd.1 (osd.1) 157 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:00.832814+0000 osd.1 (osd.1) 158 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:01.078891+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]': finished 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:01.079004+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.079049+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-02-23T00:35:01.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:01.080128+0000 mon.c (mon.2) 395 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:01.081598+0000 mon.b (mon.1) 303 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:01.082151+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:01.082760+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:34:59.628916+0000 mgr.y (mgr.24437) 334 : cluster [DBG] pgmap v505: 240 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 231 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1086 objects misplaced (1.197%) 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:00.413508+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:00.416488+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:00.617556+0000 osd.7 (osd.7) 171 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:00.621151+0000 osd.7 (osd.7) 172 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:00.831990+0000 osd.1 (osd.1) 157 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:00.832814+0000 osd.1 (osd.1) 158 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:35:01.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:01.078891+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]': finished 2024-02-23T00:35:01.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:01.079004+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:35:01.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.079049+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-02-23T00:35:01.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:01.080128+0000 mon.c (mon.2) 395 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:01.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:01.081598+0000 mon.b (mon.1) 303 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:01.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:01.082151+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:01.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:01.082760+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:01.513 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:01.528 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:01.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:01.634 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:01.664 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:01.686 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:01.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:01.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:34:59.628916+0000 mgr.y (mgr.24437) 334 : cluster [DBG] pgmap v505: 240 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 231 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1086 objects misplaced (1.197%) 2024-02-23T00:35:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:00.413508+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:35:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:00.416488+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:35:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:00.617556+0000 osd.7 (osd.7) 171 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:35:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:00.621151+0000 osd.7 (osd.7) 172 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:35:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:00.831990+0000 osd.1 (osd.1) 157 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:35:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:00.832814+0000 osd.1 (osd.1) 158 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:35:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:01.078891+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi087-82019-60"}]': finished 2024-02-23T00:35:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:01.079004+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:35:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.079049+0000 mon.a (mon.0) 2317 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-02-23T00:35:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:01.080128+0000 mon.c (mon.2) 395 : audit [INF] from='client.? 172.21.15.87:0/4086862305' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:01.081598+0000 mon.b (mon.1) 303 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:01.082151+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]: dispatch 2024-02-23T00:35:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:01.082760+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:01.790 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:01.817 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:02.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.060+0000 7fa28ffff700 1 -- 172.21.15.87:0/2946546126 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v351) v1 ==== 101+0+0 (secure 0 0 0) 0x7fa29c022090 con 0x7fa2a80fdbc0 2024-02-23T00:35:02.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.063+0000 7fa2ad9fa700 10 librados: watch_flush enter 2024-02-23T00:35:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:02.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.065+0000 7fa2ad9fa700 10 librados: watch_flush exit 2024-02-23T00:35:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.065+0000 7fa2ad9fa700 20 client.37541.objecter shutdown clearing up homeless session... 2024-02-23T00:35:02.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.065+0000 7fa2ad9fa700 10 client.37541.objecter successfully canceled tick 2024-02-23T00:35:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa290062c90 msgr2=0x7fa290065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/2946546126 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa290062c90 0x7fa290065150 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 rx=0x7fa298009170 tx=0x7fa298009da0).stop 2024-02-23T00:35:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa2a80fdbc0 msgr2=0x7fa2a806b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:02.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/2946546126 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa2a80fdbc0 0x7fa2a806b5c0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7fa29c0091f0 tx=0x7fa29c009cb0).stop 2024-02-23T00:35:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 shutdown_connections 2024-02-23T00:35:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/2946546126 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa290062c90 0x7fa290065150 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/2946546126 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa2a80fe520 0x7fa2a806bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/2946546126 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa2a80f8430 0x7fa2a806b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 --2- 172.21.15.87:0/2946546126 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa2a80fdbc0 0x7fa2a806b5c0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 >> 172.21.15.87:0/2946546126 conn(0x7fa2a80f37d0 msgr2=0x7fa2a80fbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 shutdown_connections 2024-02-23T00:35:02.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.066+0000 7fa2ad9fa700 1 -- 172.21.15.87:0/2946546126 wait complete. 2024-02-23T00:35:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:02.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:02.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:02.068+0000 7fa2ad9fa700 1 librados: shutdown 2024-02-23T00:35:02.073 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:35:02.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:b.RoundTripAppendPP (2981 ms) 2024-02-23T00:35:02.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-02-23T00:35:02.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3013 ms) 2024-02-23T00:35:02.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-02-23T00:35:02.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2999 ms) 2024-02-23T00:35:02.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-02-23T00:35:02.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3223 ms) 2024-02-23T00:35:02.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-02-23T00:35:02.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2771 ms) 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 31 tests from LibRadosAio (122974 ms total) 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3014 ms) 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9014 ms) 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14983 ms) 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3008 ms) 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30019 ms total) 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-02-23T00:35:02.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2997 ms) 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2997 ms total) 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14007 ms) 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7014 ms) 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6990 ms) 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7019 ms) 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2990 ms) 2024-02-23T00:35:02.078 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6990 ms) 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7013 ms) 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7011 ms) 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6990 ms) 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7047 ms) 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6975 ms) 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6982 ms) 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-02-23T00:35:02.079 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7007 ms) 2024-02-23T00:35:02.080 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-02-23T00:35:02.080 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3007 ms) 2024-02-23T00:35:02.080 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-02-23T00:35:02.080 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7005 ms) 2024-02-23T00:35:02.102 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:02.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:02.201 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:02.332 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:02] "GET /metrics HTTP/1.1" 200 189948 "" "Prometheus/2.33.4" 2024-02-23T00:35:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:00.826405+0000 mgr.y (mgr.24437) 335 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:01.373798+0000 mon.c (mon.2) 396 : audit [INF] from='client.? 172.21.15.87:0/2946546126' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:01.374121+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.393398+0000 osd.0 (osd.0) 171 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.396978+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.513771+0000 osd.5 (osd.5) 151 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.515540+0000 osd.5 (osd.5) 152 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.613556+0000 osd.4 (osd.4) 165 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.614336+0000 osd.4 (osd.4) 166 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.629566+0000 mgr.y (mgr.24437) 336 : cluster [DBG] pgmap v508: 232 pgs: 1 active+remapped+backfill_toofull, 1 peering, 230 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.641457+0000 osd.7 (osd.7) 173 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.643108+0000 osd.7 (osd.7) 174 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.790269+0000 osd.1 (osd.1) 159 : cluster [DBG] 63.10 scrub starts 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.791834+0000 osd.1 (osd.1) 160 : cluster [DBG] 63.10 scrub ok 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:02.053003+0000 mon.a (mon.0) 2321 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.058286+0000 mon.a (mon.0) 2322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]': finished 2024-02-23T00:35:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.058360+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]': finished 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.058412+0000 mon.a (mon.0) 2324 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:02.058454+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.099447+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.100198+0000 mon.a (mon.0) 2326 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.101085+0000 mon.b (mon.1) 305 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.101219+0000 mon.a (mon.0) 2327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.105641+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.105845+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.106609+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.106734+0000 mon.a (mon.0) 2329 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.107701+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:02.107952+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:00.826405+0000 mgr.y (mgr.24437) 335 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:01.373798+0000 mon.c (mon.2) 396 : audit [INF] from='client.? 172.21.15.87:0/2946546126' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:01.374121+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.393398+0000 osd.0 (osd.0) 171 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:35:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.396978+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:35:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.513771+0000 osd.5 (osd.5) 151 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:35:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.515540+0000 osd.5 (osd.5) 152 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:35:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.613556+0000 osd.4 (osd.4) 165 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:35:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.614336+0000 osd.4 (osd.4) 166 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:35:02.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.629566+0000 mgr.y (mgr.24437) 336 : cluster [DBG] pgmap v508: 232 pgs: 1 active+remapped+backfill_toofull, 1 peering, 230 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:02.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.641457+0000 osd.7 (osd.7) 173 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:35:02.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.643108+0000 osd.7 (osd.7) 174 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:35:02.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.790269+0000 osd.1 (osd.1) 159 : cluster [DBG] 63.10 scrub starts 2024-02-23T00:35:02.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.791834+0000 osd.1 (osd.1) 160 : cluster [DBG] 63.10 scrub ok 2024-02-23T00:35:02.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:02.053003+0000 mon.a (mon.0) 2321 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:35:02.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.058286+0000 mon.a (mon.0) 2322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]': finished 2024-02-23T00:35:02.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.058360+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]': finished 2024-02-23T00:35:02.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.058412+0000 mon.a (mon.0) 2324 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:00.826405+0000 mgr.y (mgr.24437) 335 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:01.373798+0000 mon.c (mon.2) 396 : audit [INF] from='client.? 172.21.15.87:0/2946546126' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:01.374121+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.393398+0000 osd.0 (osd.0) 171 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.396978+0000 osd.0 (osd.0) 172 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.513771+0000 osd.5 (osd.5) 151 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.515540+0000 osd.5 (osd.5) 152 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.613556+0000 osd.4 (osd.4) 165 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.614336+0000 osd.4 (osd.4) 166 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.629566+0000 mgr.y (mgr.24437) 336 : cluster [DBG] pgmap v508: 232 pgs: 1 active+remapped+backfill_toofull, 1 peering, 230 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.641457+0000 osd.7 (osd.7) 173 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:35:02.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.643108+0000 osd.7 (osd.7) 174 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.790269+0000 osd.1 (osd.1) 159 : cluster [DBG] 63.10 scrub starts 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.791834+0000 osd.1 (osd.1) 160 : cluster [DBG] 63.10 scrub ok 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:02.053003+0000 mon.a (mon.0) 2321 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.058286+0000 mon.a (mon.0) 2322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi087-82019-60"}]': finished 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.058360+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]': finished 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.058412+0000 mon.a (mon.0) 2324 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:02.058454+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.099447+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.100198+0000 mon.a (mon.0) 2326 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:02.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.101085+0000 mon.b (mon.1) 305 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.101219+0000 mon.a (mon.0) 2327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.105641+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.105845+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.106609+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.106734+0000 mon.a (mon.0) 2329 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.107701+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:02.107952+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:02.058454+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.099447+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.100198+0000 mon.a (mon.0) 2326 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:02.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.101085+0000 mon.b (mon.1) 305 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:02.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.101219+0000 mon.a (mon.0) 2327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-45"}]: dispatch 2024-02-23T00:35:02.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.105641+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.105845+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.106609+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.106734+0000 mon.a (mon.0) 2329 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:02.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.107701+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:02.838 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:02.107952+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:03.072 INFO:tasks.workunit.client.0.smithi087.stdout: 4c720e9:::166:head 2024-02-23T00:35:03.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:e694b040:::167:head 2024-02-23T00:35:03.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:afa38db2:::168:head 2024-02-23T00:35:03.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:77ba9f53:::169:head 2024-02-23T00:35:03.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:87495034:::170:head 2024-02-23T00:35:03.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:7c96bf0e:::171:head 2024-02-23T00:35:03.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:dbe346cc:::172:head 2024-02-23T00:35:03.072 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:e943ec24:::173:head 2024-02-23T00:35:03.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:f97a9c0c:::174:head 2024-02-23T00:35:03.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:6f26e74d:::175:head 2024-02-23T00:35:03.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:4f95e106:::176:head 2024-02-23T00:35:03.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:0e6f2f8f:::177:head 2024-02-23T00:35:03.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:05db05f1:::178:head 2024-02-23T00:35:03.073 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:38a78d66:::179:head 2024-02-23T00:35:03.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:d095610b:::180:head 2024-02-23T00:35:03.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:a1a9d709:::181:head 2024-02-23T00:35:03.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:1e5d39db:::182:head 2024-02-23T00:35:03.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:f7df4fb9:::183:head 2024-02-23T00:35:03.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:03a7f161:::184:head 2024-02-23T00:35:03.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:ba70721e:::185:head 2024-02-23T00:35:03.074 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:28e5662d:::186:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:973d52de:::187:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:4303eb1c:::188:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:b990b48e:::189:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:29b8165b:::190:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:3547f197:::191:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:7e260936:::192:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:1abec7b1:::193:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:10fdda93:::194:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:15817eea:::195:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:770bab57:::196:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:ed9e13e7:::197:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:71471a8f:::198:head 2024-02-23T00:35:03.075 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: checking for 190:10fb1d02:::199:head 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8012 ms) 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: first is 1708648472 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,1708648477,1708648478,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,1708648477,1708648478,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,1708648477,1708648478,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,1708648477,1708648482,0 2024-02-23T00:35:03.076 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,1708648477,1708648482,0 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,1708648477,1708648482,0 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,1708648477,1708648482,1708648484,1708648485,0 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,1708648477,1708648482,1708648484,1708648485,0 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648472,1708648474,1708648475,1708648477,1708648482,1708648484,1708648485,0 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648475,1708648477,1708648482,1708648484,1708648485,1708648487,1708648488,0 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: first now 1708648475, trimmed 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (25012 ms) 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: foo0 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-23T00:35:03.077 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14002 ms) 2024-02-23T00:35:03.450 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:03.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:03.554 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:03.582 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.974537+0000 osd.0 (osd.0) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:01.974784+0000 osd.0 (osd.0) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:02.148688+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.19 deep-scrub starts 2024-02-23T00:35:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:02.151722+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.19 deep-scrub ok 2024-02-23T00:35:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:02.740798+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:35:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:02.744080+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:35:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:03.067533+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:03.067659+0000 mon.a (mon.0) 2332 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-02-23T00:35:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:03.073030+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi087-82019-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:03.311438+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi087-82019-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:03.747 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.974537+0000 osd.0 (osd.0) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:03.747 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:01.974784+0000 osd.0 (osd.0) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:03.748 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:02.148688+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.19 deep-scrub starts 2024-02-23T00:35:03.748 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:02.151722+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.19 deep-scrub ok 2024-02-23T00:35:03.748 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:02.740798+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:35:03.748 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:02.744080+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:35:03.748 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:03.067533+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:03.748 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:03.067659+0000 mon.a (mon.0) 2332 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-02-23T00:35:03.748 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:03.073030+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi087-82019-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:03.748 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:03.311438+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi087-82019-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.974537+0000 osd.0 (osd.0) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:01.974784+0000 osd.0 (osd.0) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:02.148688+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.19 deep-scrub starts 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:02.151722+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.19 deep-scrub ok 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:02.740798+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:02.744080+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:03.067533+0000 mon.a (mon.0) 2331 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:03.067659+0000 mon.a (mon.0) 2332 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:03.073030+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi087-82019-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:03.749 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:03.311438+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi087-82019-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:04.088 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:35:04.088 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:35:04.088 INFO:tasks.thrashosds.thrasher:Growing pool default.rgw.control 2024-02-23T00:35:04.089 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:35:04.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:35:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:35:04.317 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:35:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:35:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:35:04.318 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:35:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:03.109495+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:35:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:03.113093+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:35:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:03.594456+0000 osd.4 (osd.4) 167 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:35:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:03.595441+0000 osd.4 (osd.4) 168 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:35:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:03.630214+0000 mgr.y (mgr.24437) 337 : cluster [DBG] pgmap v511: 200 pgs: 1 active+remapped+backfill_toofull, 1 peering, 198 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.5 KiB/s rd, 255 B/s wr, 17 op/s; 13/1104 objects misplaced (1.178%); 27 B/s, 1 objects/s recovering 2024-02-23T00:35:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:04.077994+0000 mon.b (mon.1) 310 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:04.320629+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-02-23T00:35:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:04.321593+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:04.755 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:03.109495+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:35:04.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:03.113093+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:35:04.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:03.594456+0000 osd.4 (osd.4) 167 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:35:04.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:03.595441+0000 osd.4 (osd.4) 168 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:35:04.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:03.630214+0000 mgr.y (mgr.24437) 337 : cluster [DBG] pgmap v511: 200 pgs: 1 active+remapped+backfill_toofull, 1 peering, 198 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.5 KiB/s rd, 255 B/s wr, 17 op/s; 13/1104 objects misplaced (1.178%); 27 B/s, 1 objects/s recovering 2024-02-23T00:35:04.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:04.077994+0000 mon.b (mon.1) 310 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:04.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:04.320629+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-02-23T00:35:04.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:04.321593+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:04.756 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:03.109495+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:35:04.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:03.113093+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:35:04.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:03.594456+0000 osd.4 (osd.4) 167 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:35:04.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:03.595441+0000 osd.4 (osd.4) 168 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:35:04.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:03.630214+0000 mgr.y (mgr.24437) 337 : cluster [DBG] pgmap v511: 200 pgs: 1 active+remapped+backfill_toofull, 1 peering, 198 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.5 KiB/s rd, 255 B/s wr, 17 op/s; 13/1104 objects misplaced (1.178%); 27 B/s, 1 objects/s recovering 2024-02-23T00:35:04.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:04.077994+0000 mon.b (mon.1) 310 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:04.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:04.320629+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-02-23T00:35:04.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:04.321593+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:05.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:35:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:03.996472+0000 osd.2 (osd.2) 137 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:35:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:03.997796+0000 osd.2 (osd.2) 138 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:35:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:04.410966+0000 osd.0 (osd.0) 175 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:35:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:04.413987+0000 osd.0 (osd.0) 176 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:35:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:04.587166+0000 osd.4 (osd.4) 169 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:35:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:04.588127+0000 osd.4 (osd.4) 170 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:35:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:04.670760+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:35:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:04.673829+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:35:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:05.070384+0000 mon.a (mon.0) 2336 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi087-82019-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi087-82019-61"}]': finished 2024-02-23T00:35:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:05.070576+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:05.070705+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-02-23T00:35:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:05.092574+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:05.092850+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 -- 172.21.15.87:0/2977351690 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f8cb0 msgr2=0x7f30d00f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 --2- 172.21.15.87:0/2977351690 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f8cb0 0x7f30d00f9090 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7f30c8009c30 tx=0x7f30c8004c50).stop 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 -- 172.21.15.87:0/2977351690 shutdown_connections 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 --2- 172.21.15.87:0/2977351690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30d0102210 0x7f30d0104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 --2- 172.21.15.87:0/2977351690 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f8cb0 0x7f30d00f9090 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 --2- 172.21.15.87:0/2977351690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30d00f9610 0x7f30d0101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 -- 172.21.15.87:0/2977351690 >> 172.21.15.87:0/2977351690 conn(0x7f30d00f2820 msgr2=0x7f30d00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 -- 172.21.15.87:0/2977351690 shutdown_connections 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 -- 172.21.15.87:0/2977351690 wait complete. 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 librados: starting msgr at 2024-02-23T00:35:05.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 librados: starting objecter 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.805+0000 7f30d6210700 1 Processor -- start 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30d6210700 1 -- start start 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30d6210700 1 librados: setting wanted keys 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30d6210700 1 librados: calling monclient init 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30d6210700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30d00f8cb0 0x7f30d010e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30d6210700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f9610 0x7f30d010e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30d6210700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30d0102210 0x7f30d01083e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30d6210700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d01088d0 con 0x7f30d0102210 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30d6210700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d0108a40 con 0x7f30d00f8cb0 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30d6210700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d0108c00 con 0x7f30d00f9610 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30ceffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f9610 0x7f30d010e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30cffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30d0102210 0x7f30d01083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30ceffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f9610 0x7f30d010e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35116/0 (socket says 172.21.15.87:35116) 2024-02-23T00:35:05.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30ceffd700 1 -- 172.21.15.87:0/1977170018 learned_addr learned my addr 172.21.15.87:0/1977170018 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30cffff700 1 --2- 172.21.15.87:0/1977170018 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30d0102210 0x7f30d01083e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52754/0 (socket says 172.21.15.87:52754) 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30cf7fe700 1 --2- 172.21.15.87:0/1977170018 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30d00f8cb0 0x7f30d010e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30ceffd700 1 -- 172.21.15.87:0/1977170018 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30d00f8cb0 msgr2=0x7f30d010e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30ceffd700 1 --2- 172.21.15.87:0/1977170018 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30d00f8cb0 0x7f30d010e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30ceffd700 1 -- 172.21.15.87:0/1977170018 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30d0102210 msgr2=0x7f30d01083e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30ceffd700 1 --2- 172.21.15.87:0/1977170018 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30d0102210 0x7f30d01083e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30ceffd700 1 -- 172.21.15.87:0/1977170018 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c0005040 con 0x7f30d00f9610 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.806+0000 7f30cffff700 1 --2- 172.21.15.87:0/1977170018 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30d0102210 0x7f30d01083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30ceffd700 1 --2- 172.21.15.87:0/1977170018 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f9610 0x7f30d010e730 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7f30b80041d0 tx=0x7f30b8009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30ccff9700 10 client.?.objecter ms_handle_connect 0x7f30d00f9610 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30ccff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8007b40 con 0x7f30d00f9610 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30b800f040 con 0x7f30d00f9610 2024-02-23T00:35:05.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c80097c0 con 0x7f30d00f9610 2024-02-23T00:35:05.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8014810 con 0x7f30d00f9610 2024-02-23T00:35:05.812 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30d6210700 10 client.37553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:05.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d0109240 con 0x7f30d00f9610 2024-02-23T00:35:05.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.807+0000 7f30d6210700 1 librados: init done 2024-02-23T00:35:05.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.808+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f30d005fa60 con 0x7f30d00f9610 2024-02-23T00:35:05.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.808+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f30b8009110 con 0x7f30d00f9610 2024-02-23T00:35:05.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.808+0000 7f30ccff9700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6800/1231597463 conn(0x7f30bc062c90 0x7f30bc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(354..354 src has 173..354) v4 ==== 8926+0+0 (secure 0 0 0) 0x7f30b807d590 con 0x7f30d00f9610 2024-02-23T00:35:05.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 osd_map(354..354 src has 173..354) v4 2024-02-23T00:35:05.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30ccff9700 3 client.37553.objecter handle_osd_map got epochs [354,354] > 0 2024-02-23T00:35:05.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30ccff9700 3 client.37553.objecter handle_osd_map decoding full epoch 354 2024-02-23T00:35:05.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30cf7fe700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6800/1231597463 conn(0x7f30bc062c90 0x7f30bc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30ccff9700 20 client.37553.objecter dump_active .. 0 homeless 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v354) v1 ==== 54+0+16 (secure 0 0 0) 0x7f30b8084070 con 0x7f30d00f9610 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30ccff9700 10 client.37553.objecter ms_handle_connect 0x7f30bc062c90 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30cf7fe700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6800/1231597463 conn(0x7f30bc062c90 0x7f30bc065150 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7f30c8004890 tx=0x7f30c8004c50).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 10 client.37553.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 1 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6801/2153191863 conn(0x7f30d00fc1e0 0x7f30d00fc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d005ee80 osd=0 3 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d005ee80 osd=0 3 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d005ee80 osd=0 3 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 1 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d005ee80 osd=0 3 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 0 1 2024-02-23T00:35:05.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d005ee80 osd=0 4 2024-02-23T00:35:05.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 10 client.37553.objecter _send_command 1 2024-02-23T00:35:05.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30d0107290 con 0x7f30d00fc1e0 2024-02-23T00:35:05.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30cffff700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6801/2153191863 conn(0x7f30d00fc1e0 0x7f30d00fc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30ccff9700 10 client.37553.objecter ms_handle_connect 0x7f30d00fc1e0 2024-02-23T00:35:05.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.809+0000 7f30cffff700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6801/2153191863 conn(0x7f30d00fc1e0 0x7f30d00fc5c0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.810+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30d0107290 con 0x7f30d00fc1e0 2024-02-23T00:35:05.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.810+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:35:05.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.810+0000 7f30ccff9700 10 client.37553.objecter _finish_command 1 = system:0 2024-02-23T00:35:05.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.810+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d005ee80 osd=0 4 2024-02-23T00:35:05.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.810+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 0 1 2024-02-23T00:35:05.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 10 client.37553.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:05.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 2 2024-02-23T00:35:05.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d005ee80 osd=0 3 2024-02-23T00:35:05.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d005ee80 osd=0 3 2024-02-23T00:35:05.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d005ee80 osd=0 3 2024-02-23T00:35:05.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 2 2024-02-23T00:35:05.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d005ee80 osd=0 3 2024-02-23T00:35:05.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 0 2 2024-02-23T00:35:05.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d005ee80 osd=0 4 2024-02-23T00:35:05.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 10 client.37553.objecter _send_command 2 2024-02-23T00:35:05.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.825+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f30d00273c0 con 0x7f30d00fc1e0 2024-02-23T00:35:05.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f30d00273c0 con 0x7f30d00fc1e0 2024-02-23T00:35:05.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:05.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30ccff9700 10 client.37553.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d005ee80 osd=0 4 2024-02-23T00:35:05.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 0 2 2024-02-23T00:35:05.828 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:35:05.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 10 client.37553.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:05.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 3 2024-02-23T00:35:05.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6805/685354171 conn(0x7f30d00598f0 0x7f30d0059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0028740 osd=1 3 2024-02-23T00:35:05.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0028740 osd=1 3 2024-02-23T00:35:05.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0028740 osd=1 3 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 3 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d0028740 osd=1 3 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 1 3 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0028740 osd=1 4 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 10 client.37553.objecter _send_command 3 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30d011b8b0 con 0x7f30d00598f0 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30cf7fe700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6805/685354171 conn(0x7f30d00598f0 0x7f30d0059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30ccff9700 10 client.37553.objecter ms_handle_connect 0x7f30d00598f0 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.827+0000 7f30cf7fe700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6805/685354171 conn(0x7f30d00598f0 0x7f30d0059cd0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.828+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30d011b8b0 con 0x7f30d00598f0 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.828+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.828+0000 7f30ccff9700 10 client.37553.objecter _finish_command 3 = system:0 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.828+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d0028740 osd=1 4 2024-02-23T00:35:05.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.828+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 1 3 2024-02-23T00:35:05.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:03.996472+0000 osd.2 (osd.2) 137 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:35:05.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:03.997796+0000 osd.2 (osd.2) 138 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:35:05.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:04.410966+0000 osd.0 (osd.0) 175 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:35:05.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:04.413987+0000 osd.0 (osd.0) 176 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:35:05.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:04.587166+0000 osd.4 (osd.4) 169 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:35:05.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:04.588127+0000 osd.4 (osd.4) 170 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:35:05.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:04.670760+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:35:05.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:04.673829+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:35:05.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:05.070384+0000 mon.a (mon.0) 2336 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi087-82019-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi087-82019-61"}]': finished 2024-02-23T00:35:05.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:05.070576+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:05.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:05.070705+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-02-23T00:35:05.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:05.092574+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:05.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:05.092850+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:03.996472+0000 osd.2 (osd.2) 137 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:03.997796+0000 osd.2 (osd.2) 138 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:04.410966+0000 osd.0 (osd.0) 175 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:04.413987+0000 osd.0 (osd.0) 176 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:04.587166+0000 osd.4 (osd.4) 169 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:04.588127+0000 osd.4 (osd.4) 170 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:04.670760+0000 osd.7 (osd.7) 175 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:04.673829+0000 osd.7 (osd.7) 176 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:05.070384+0000 mon.a (mon.0) 2336 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi087-82019-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi087-82019-61"}]': finished 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:05.070576+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:05.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:05.070705+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-02-23T00:35:05.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:05.092574+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:05.837 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:05.092850+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:05.837 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:05.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 10 client.37553.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 4 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0028740 osd=1 3 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0028740 osd=1 3 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0028740 osd=1 3 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 4 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d0028740 osd=1 3 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 1 4 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0028740 osd=1 4 2024-02-23T00:35:05.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 10 client.37553.objecter _send_command 4 2024-02-23T00:35:05.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.849+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f30d0059360 con 0x7f30d00598f0 2024-02-23T00:35:05.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f30d0059360 con 0x7f30d00598f0 2024-02-23T00:35:05.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:05.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30ccff9700 10 client.37553.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d0028740 osd=1 4 2024-02-23T00:35:05.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 1 4 2024-02-23T00:35:05.853 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:35:05.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 10 client.37553.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:05.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 5 2024-02-23T00:35:05.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6809/3605394228 conn(0x7f30d00282a0 0x7f30d011a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0059770 osd=2 3 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0059770 osd=2 3 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0059770 osd=2 3 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 5 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d0059770 osd=2 3 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 2 5 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0059770 osd=2 4 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 10 client.37553.objecter _send_command 5 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30d011a7c0 con 0x7f30d00282a0 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30ceffd700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6809/3605394228 conn(0x7f30d00282a0 0x7f30d011a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30ccff9700 10 client.37553.objecter ms_handle_connect 0x7f30d00282a0 2024-02-23T00:35:05.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.851+0000 7f30ceffd700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6809/3605394228 conn(0x7f30d00282a0 0x7f30d011a2d0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.852+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30d011a7c0 con 0x7f30d00282a0 2024-02-23T00:35:05.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.852+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:35:05.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.852+0000 7f30ccff9700 10 client.37553.objecter _finish_command 5 = system:0 2024-02-23T00:35:05.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.852+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d0059770 osd=2 4 2024-02-23T00:35:05.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.852+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 2 5 2024-02-23T00:35:05.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 10 client.37553.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 6 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0059770 osd=2 3 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0059770 osd=2 3 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0059770 osd=2 3 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 6 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d0059770 osd=2 3 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 2 6 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0059770 osd=2 4 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 10 client.37553.objecter _send_command 6 2024-02-23T00:35:05.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.866+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f30d0027800 con 0x7f30d00282a0 2024-02-23T00:35:05.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f30d0027800 con 0x7f30d00282a0 2024-02-23T00:35:05.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:05.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30ccff9700 10 client.37553.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d0059770 osd=2 4 2024-02-23T00:35:05.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 2 6 2024-02-23T00:35:05.870 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:35:05.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 10 client.37553.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:05.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 7 2024-02-23T00:35:05.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6813/128059790 conn(0x7f30d00f6aa0 0x7f30d00f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0027c10 osd=3 3 2024-02-23T00:35:05.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0027c10 osd=3 3 2024-02-23T00:35:05.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0027c10 osd=3 3 2024-02-23T00:35:05.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 7 2024-02-23T00:35:05.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d0027c10 osd=3 3 2024-02-23T00:35:05.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 3 7 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0027c10 osd=3 4 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 10 client.37553.objecter _send_command 7 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30d00f89f0 con 0x7f30d00f6aa0 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30cffff700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6813/128059790 conn(0x7f30d00f6aa0 0x7f30d00f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30ccff9700 10 client.37553.objecter ms_handle_connect 0x7f30d00f6aa0 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.868+0000 7f30cffff700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6813/128059790 conn(0x7f30d00f6aa0 0x7f30d00f6f20 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.869+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30d00f89f0 con 0x7f30d00f6aa0 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.869+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.869+0000 7f30ccff9700 10 client.37553.objecter _finish_command 7 = system:0 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.869+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d0027c10 osd=3 4 2024-02-23T00:35:05.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.869+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 3 7 2024-02-23T00:35:05.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 10 client.37553.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 8 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0027c10 osd=3 3 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0027c10 osd=3 3 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0027c10 osd=3 3 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 8 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d0027c10 osd=3 3 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 3 8 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0027c10 osd=3 4 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 10 client.37553.objecter _send_command 8 2024-02-23T00:35:05.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.883+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f30d00f8590 con 0x7f30d00f6aa0 2024-02-23T00:35:05.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f30d00f8590 con 0x7f30d00f6aa0 2024-02-23T00:35:05.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:05.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30ccff9700 10 client.37553.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d0027c10 osd=3 4 2024-02-23T00:35:05.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 3 8 2024-02-23T00:35:05.886 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:35:05.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 10 client.37553.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:05.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 9 2024-02-23T00:35:05.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6800/4175844757 conn(0x7f30d00f7a90 0x7f30d00f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d00f7410 osd=4 3 2024-02-23T00:35:05.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f7410 osd=4 3 2024-02-23T00:35:05.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d00f7410 osd=4 3 2024-02-23T00:35:05.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 9 2024-02-23T00:35:05.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d00f7410 osd=4 3 2024-02-23T00:35:05.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 4 9 2024-02-23T00:35:05.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f7410 osd=4 4 2024-02-23T00:35:05.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 10 client.37553.objecter _send_command 9 2024-02-23T00:35:05.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30d00f83e0 con 0x7f30d00f7a90 2024-02-23T00:35:05.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.885+0000 7f30cf7fe700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6800/4175844757 conn(0x7f30d00f7a90 0x7f30d00f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.886+0000 7f30cf7fe700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6800/4175844757 conn(0x7f30d00f7a90 0x7f30d00f7ef0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.886+0000 7f30ccff9700 10 client.37553.objecter ms_handle_connect 0x7f30d00f7a90 2024-02-23T00:35:05.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.887+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30d00f83e0 con 0x7f30d00f7a90 2024-02-23T00:35:05.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.887+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:35:05.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.887+0000 7f30ccff9700 10 client.37553.objecter _finish_command 9 = system:0 2024-02-23T00:35:05.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.887+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d00f7410 osd=4 4 2024-02-23T00:35:05.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.887+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 4 9 2024-02-23T00:35:05.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 10 client.37553.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:05.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 10 2024-02-23T00:35:05.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d00f7410 osd=4 3 2024-02-23T00:35:05.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f7410 osd=4 3 2024-02-23T00:35:05.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d00f7410 osd=4 3 2024-02-23T00:35:05.902 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 10 2024-02-23T00:35:05.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d00f7410 osd=4 3 2024-02-23T00:35:05.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 4 10 2024-02-23T00:35:05.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f7410 osd=4 4 2024-02-23T00:35:05.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 10 client.37553.objecter _send_command 10 2024-02-23T00:35:05.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.901+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f30d00f83e0 con 0x7f30d00f7a90 2024-02-23T00:35:05.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f30d00f83e0 con 0x7f30d00f7a90 2024-02-23T00:35:05.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:05.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30ccff9700 10 client.37553.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d00f7410 osd=4 4 2024-02-23T00:35:05.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 4 10 2024-02-23T00:35:05.905 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 10 client.37553.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 11 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6804/2550013795 conn(0x7f30d0126370 0x7f30d0128760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d00f83e0 osd=5 3 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f83e0 osd=5 3 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d00f83e0 osd=5 3 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 11 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d00f83e0 osd=5 3 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 5 11 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f83e0 osd=5 4 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 10 client.37553.objecter _send_command 11 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.903+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30d012b3b0 con 0x7f30d0126370 2024-02-23T00:35:05.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.904+0000 7f30ceffd700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6804/2550013795 conn(0x7f30d0126370 0x7f30d0128760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.904+0000 7f30ccff9700 10 client.37553.objecter ms_handle_connect 0x7f30d0126370 2024-02-23T00:35:05.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.904+0000 7f30ceffd700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6804/2550013795 conn(0x7f30d0126370 0x7f30d0128760 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.905+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30d012b3b0 con 0x7f30d0126370 2024-02-23T00:35:05.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.905+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:35:05.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.905+0000 7f30ccff9700 10 client.37553.objecter _finish_command 11 = system:0 2024-02-23T00:35:05.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.905+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d00f83e0 osd=5 4 2024-02-23T00:35:05.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.905+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 5 11 2024-02-23T00:35:05.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 10 client.37553.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 12 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d00f83e0 osd=5 3 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f83e0 osd=5 3 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d00f83e0 osd=5 3 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 12 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d00f83e0 osd=5 3 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 5 12 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f83e0 osd=5 4 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 10 client.37553.objecter _send_command 12 2024-02-23T00:35:05.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.918+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f30d0128c50 con 0x7f30d0126370 2024-02-23T00:35:05.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f30d0128c50 con 0x7f30d0126370 2024-02-23T00:35:05.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:05.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30ccff9700 10 client.37553.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d00f83e0 osd=5 4 2024-02-23T00:35:05.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 5 12 2024-02-23T00:35:05.922 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:35:05.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 10 client.37553.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:05.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 13 2024-02-23T00:35:05.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6808/2509408498 conn(0x7f30d012a940 0x7f30d012ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0128c50 osd=6 3 2024-02-23T00:35:05.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0128c50 osd=6 3 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0128c50 osd=6 3 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 13 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d0128c50 osd=6 3 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 6 13 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0128c50 osd=6 4 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 10 client.37553.objecter _send_command 13 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30d012e700 con 0x7f30d012a940 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.921+0000 7f30cffff700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6808/2509408498 conn(0x7f30d012a940 0x7f30d012ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.922+0000 7f30ccff9700 10 client.37553.objecter ms_handle_connect 0x7f30d012a940 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.922+0000 7f30cffff700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6808/2509408498 conn(0x7f30d012a940 0x7f30d012ada0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.922+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30d012e700 con 0x7f30d012a940 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.922+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.922+0000 7f30ccff9700 10 client.37553.objecter _finish_command 13 = system:0 2024-02-23T00:35:05.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.922+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d0128c50 osd=6 4 2024-02-23T00:35:05.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.922+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 6 13 2024-02-23T00:35:05.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 10 client.37553.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:05.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 14 2024-02-23T00:35:05.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0128c50 osd=6 3 2024-02-23T00:35:05.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0128c50 osd=6 3 2024-02-23T00:35:05.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d0128c50 osd=6 3 2024-02-23T00:35:05.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 14 2024-02-23T00:35:05.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d0128c50 osd=6 3 2024-02-23T00:35:05.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 6 14 2024-02-23T00:35:05.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0128c50 osd=6 4 2024-02-23T00:35:05.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 10 client.37553.objecter _send_command 14 2024-02-23T00:35:05.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.936+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f30d0128dc0 con 0x7f30d012a940 2024-02-23T00:35:05.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f30d0128dc0 con 0x7f30d012a940 2024-02-23T00:35:05.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:05.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30ccff9700 10 client.37553.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d0128c50 osd=6 4 2024-02-23T00:35:05.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 6 14 2024-02-23T00:35:05.940 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:35:05.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 10 client.37553.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 15 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6812/373787000 conn(0x7f30d01292d0 0x7f30d0129730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d012a0e0 osd=7 3 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d012a0e0 osd=7 3 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d012a0e0 osd=7 3 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 15 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d012a0e0 osd=7 3 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 7 15 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d012a0e0 osd=7 4 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 10 client.37553.objecter _send_command 15 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.938+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30d0129c20 con 0x7f30d01292d0 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.939+0000 7f30cf7fe700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6812/373787000 conn(0x7f30d01292d0 0x7f30d0129730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:05.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.939+0000 7f30ccff9700 10 client.37553.objecter ms_handle_connect 0x7f30d01292d0 2024-02-23T00:35:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.939+0000 7f30cf7fe700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6812/373787000 conn(0x7f30d01292d0 0x7f30d0129730 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.940+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f30d0129c20 con 0x7f30d01292d0 2024-02-23T00:35:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.940+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:35:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.940+0000 7f30ccff9700 10 client.37553.objecter _finish_command 15 = system:0 2024-02-23T00:35:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.940+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d012a0e0 osd=7 4 2024-02-23T00:35:05.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.940+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 7 15 2024-02-23T00:35:05.942 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:05.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 10 client.37553.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign -1 16 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d012a0e0 osd=7 3 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d012a0e0 osd=7 3 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 20 client.37553.objecter _get_session s=0x7f30d012a0e0 osd=7 3 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 15 client.37553.objecter _session_command_op_remove -1 16 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 20 client.37553.objecter get_session s=0x7f30d012a0e0 osd=7 3 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 15 client.37553.objecter _session_command_op_assign 7 16 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d012a0e0 osd=7 4 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 10 client.37553.objecter _send_command 16 2024-02-23T00:35:05.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.954+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f30d0129c20 con 0x7f30d01292d0 2024-02-23T00:35:05.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.955+0000 7f30ccff9700 1 -- 172.21.15.87:0/1977170018 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f30d0129c20 con 0x7f30d01292d0 2024-02-23T00:35:05.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.955+0000 7f30ccff9700 10 client.37553.objecter ms_dispatch 0x7f30d0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:05.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.955+0000 7f30ccff9700 10 client.37553.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.955+0000 7f30ccff9700 20 client.37553.objecter put_session s=0x7f30d012a0e0 osd=7 4 2024-02-23T00:35:05.957 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.955+0000 7f30ccff9700 15 client.37553.objecter _session_command_op_remove 7 16 2024-02-23T00:35:05.957 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:35:05.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 librados: watch_flush enter 2024-02-23T00:35:05.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 librados: watch_flush exit 2024-02-23T00:35:05.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 client.37553.objecter close_session for osd.0 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6801/2153191863 conn(0x7f30d00fc1e0 msgr2=0x7f30d00fc5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6801/2153191863 conn(0x7f30d00fc1e0 0x7f30d00fc5c0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d005ee80 osd=0 1 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 client.37553.objecter close_session for osd.1 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6805/685354171 conn(0x7f30d00598f0 msgr2=0x7f30d0059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6805/685354171 conn(0x7f30d00598f0 0x7f30d0059cd0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0028740 osd=1 1 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 client.37553.objecter close_session for osd.2 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6809/3605394228 conn(0x7f30d00282a0 msgr2=0x7f30d011a2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6809/3605394228 conn(0x7f30d00282a0 0x7f30d011a2d0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0059770 osd=2 1 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 client.37553.objecter close_session for osd.3 2024-02-23T00:35:05.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6813/128059790 conn(0x7f30d00f6aa0 msgr2=0x7f30d00f6f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6813/128059790 conn(0x7f30d00f6aa0 0x7f30d00f6f20 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30cffff700 1 -- 172.21.15.87:0/1977170018 reap_dead start 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0027c10 osd=3 1 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 client.37553.objecter close_session for osd.4 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6800/4175844757 conn(0x7f30d00f7a90 msgr2=0x7f30d00f7ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6800/4175844757 conn(0x7f30d00f7a90 0x7f30d00f7ef0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f7410 osd=4 1 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 client.37553.objecter close_session for osd.5 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6804/2550013795 conn(0x7f30d0126370 msgr2=0x7f30d0128760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6804/2550013795 conn(0x7f30d0126370 0x7f30d0128760 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d00f83e0 osd=5 1 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 client.37553.objecter close_session for osd.6 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6808/2509408498 conn(0x7f30d012a940 msgr2=0x7f30d012ada0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6808/2509408498 conn(0x7f30d012a940 0x7f30d012ada0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d0128c50 osd=6 1 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 client.37553.objecter close_session for osd.7 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6812/373787000 conn(0x7f30d01292d0 msgr2=0x7f30d0129730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6812/373787000 conn(0x7f30d01292d0 0x7f30d0129730 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30cffff700 1 -- 172.21.15.87:0/1977170018 reap_dead start 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30cffff700 1 -- 172.21.15.87:0/1977170018 reap_dead start 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30cffff700 1 -- 172.21.15.87:0/1977170018 reap_dead start 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 20 client.37553.objecter put_session s=0x7f30d012a0e0 osd=7 1 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 20 client.37553.objecter shutdown clearing up homeless session... 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 10 client.37553.objecter successfully canceled tick 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6800/1231597463 conn(0x7f30bc062c90 msgr2=0x7f30bc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6800/1231597463 conn(0x7f30bc062c90 0x7f30bc065150 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 rx=0x7f30c8004890 tx=0x7f30c8004c50).stop 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f9610 msgr2=0x7f30d010e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:05.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f9610 0x7f30d010e730 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7f30b80041d0 tx=0x7f30b8009f60).stop 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 shutdown_connections 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.171:6812/373787000 conn(0x7f30d01292d0 0x7f30d0129730 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> v2:172.21.15.87:6800/1231597463 conn(0x7f30bc062c90 0x7f30bc065150 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 --2- 172.21.15.87:0/1977170018 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30d00f9610 0x7f30d010e730 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 >> 172.21.15.87:0/1977170018 conn(0x7f30d00f2820 msgr2=0x7f30d00f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 shutdown_connections 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 -- 172.21.15.87:0/1977170018 wait complete. 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:05.956+0000 7f30d6210700 1 librados: shutdown 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.962 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.963 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:35:05.974 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:05.995 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:05.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:05.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:06.099 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:06.129 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:06.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:05.017478+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.e deep-scrub starts 2024-02-23T00:35:06.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:05.018494+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.e deep-scrub ok 2024-02-23T00:35:06.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:05.146339+0000 osd.6 (osd.6) 131 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:35:06.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:05.147244+0000 osd.6 (osd.6) 132 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:35:06.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:05.631138+0000 mgr.y (mgr.24437) 338 : cluster [DBG] pgmap v514: 240 pgs: 1 active+remapped+backfill_toofull, 8 creating+peering, 231 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 13/1107 objects misplaced (1.174%); 54 B/s, 1 objects/s recovering 2024-02-23T00:35:06.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:05.632261+0000 mon.a (mon.0) 2340 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-23T00:35:06.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:05.809121+0000 mon.c (mon.2) 397 : audit [DBG] from='client.? 172.21.15.87:0/1977170018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:06.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:06.072657+0000 mon.a (mon.0) 2341 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:35:06.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:06.072755+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:06.072805+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:06.082709+0000 mon.b (mon.1) 312 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:06.083170+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:05.017478+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.e deep-scrub starts 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:05.018494+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.e deep-scrub ok 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:05.146339+0000 osd.6 (osd.6) 131 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:05.147244+0000 osd.6 (osd.6) 132 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:05.631138+0000 mgr.y (mgr.24437) 338 : cluster [DBG] pgmap v514: 240 pgs: 1 active+remapped+backfill_toofull, 8 creating+peering, 231 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 13/1107 objects misplaced (1.174%); 54 B/s, 1 objects/s recovering 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:05.632261+0000 mon.a (mon.0) 2340 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:05.809121+0000 mon.c (mon.2) 397 : audit [DBG] from='client.? 172.21.15.87:0/1977170018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:06.072657+0000 mon.a (mon.0) 2341 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:06.072755+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-23T00:35:06.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:06.072805+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-02-23T00:35:06.415 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:06.082709+0000 mon.b (mon.1) 312 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:06.415 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:06.083170+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:06.430 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-23T00:35:06.664 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:06.678 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:06.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:05.017478+0000 osd.2 (osd.2) 139 : cluster [DBG] 2.e deep-scrub starts 2024-02-23T00:35:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:05.018494+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.e deep-scrub ok 2024-02-23T00:35:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:05.146339+0000 osd.6 (osd.6) 131 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:35:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:05.147244+0000 osd.6 (osd.6) 132 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:35:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:05.631138+0000 mgr.y (mgr.24437) 338 : cluster [DBG] pgmap v514: 240 pgs: 1 active+remapped+backfill_toofull, 8 creating+peering, 231 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 13/1107 objects misplaced (1.174%); 54 B/s, 1 objects/s recovering 2024-02-23T00:35:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:05.632261+0000 mon.a (mon.0) 2340 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-23T00:35:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:05.809121+0000 mon.c (mon.2) 397 : audit [DBG] from='client.? 172.21.15.87:0/1977170018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:06.072657+0000 mon.a (mon.0) 2341 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:35:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:06.072755+0000 mon.a (mon.0) 2342 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-23T00:35:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:06.072805+0000 mon.a (mon.0) 2343 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-02-23T00:35:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:06.082709+0000 mon.b (mon.1) 312 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:06.083170+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:06.783 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:06.811 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:07.159 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:07.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:07.252 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:07.264 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:07.291 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:07.414 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:07.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:07.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:07.519 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:07.551 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:07.572 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:07.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:07.676 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:07.703 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:08.038 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:08.052 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:08.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:08.156 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:08.186 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:08.208 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:08.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:08.311 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:08.318 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:06.626577+0000 osd.7 (osd.7) 177 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:35:08.318 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:06.628300+0000 osd.7 (osd.7) 178 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:35:08.318 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:06.656585+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.9 deep-scrub starts 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:06.657989+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.9 deep-scrub ok 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:06.982508+0000 osd.2 (osd.2) 141 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:06.983433+0000 osd.2 (osd.2) 142 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:07.073316+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-47"}]': finished 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:07.073415+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:07.076771+0000 mon.b (mon.1) 313 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-47", "mode": "readproxy"}]: dispatch 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:07.077541+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:07.077635+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-47", "mode": "readproxy"}]: dispatch 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:07.078354+0000 mon.a (mon.0) 2348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:08.319 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:06.626577+0000 osd.7 (osd.7) 177 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:06.628300+0000 osd.7 (osd.7) 178 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:06.656585+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.9 deep-scrub starts 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:06.657989+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.9 deep-scrub ok 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:06.982508+0000 osd.2 (osd.2) 141 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:06.983433+0000 osd.2 (osd.2) 142 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:07.073316+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-47"}]': finished 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:07.073415+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:07.076771+0000 mon.b (mon.1) 313 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-47", "mode": "readproxy"}]: dispatch 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:07.077541+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:07.077635+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-47", "mode": "readproxy"}]: dispatch 2024-02-23T00:35:08.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:07.078354+0000 mon.a (mon.0) 2348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:08.340 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:06.626577+0000 osd.7 (osd.7) 177 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:35:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:06.628300+0000 osd.7 (osd.7) 178 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:35:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:06.656585+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.9 deep-scrub starts 2024-02-23T00:35:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:06.657989+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.9 deep-scrub ok 2024-02-23T00:35:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:06.982508+0000 osd.2 (osd.2) 141 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:35:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:06.983433+0000 osd.2 (osd.2) 142 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:35:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:07.073316+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-47"}]': finished 2024-02-23T00:35:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:07.073415+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-02-23T00:35:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:07.076771+0000 mon.b (mon.1) 313 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-47", "mode": "readproxy"}]: dispatch 2024-02-23T00:35:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:07.077541+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:07.077635+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-47", "mode": "readproxy"}]: dispatch 2024-02-23T00:35:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:07.078354+0000 mon.a (mon.0) 2348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:09.178 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:35:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:07.624323+0000 osd.4 (osd.4) 171 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:35:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:07.627616+0000 osd.4 (osd.4) 172 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:35:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:07.631697+0000 mgr.y (mgr.24437) 339 : cluster [DBG] pgmap v517: 232 pgs: 1 active+remapped+backfill_toofull, 231 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 13/1107 objects misplaced (1.174%); 54 B/s, 1 objects/s recovering 2024-02-23T00:35:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:08.071096+0000 mon.a (mon.0) 2349 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:35:09.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:08.083788+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-47", "mode": "readproxy"}]': finished 2024-02-23T00:35:09.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:08.083863+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]': finished 2024-02-23T00:35:09.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:08.083935+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-02-23T00:35:09.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:08.088187+0000 mon.b (mon.1) 315 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:09.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:08.088628+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:09.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:07.624323+0000 osd.4 (osd.4) 171 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:35:09.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:07.627616+0000 osd.4 (osd.4) 172 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:35:09.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:07.631697+0000 mgr.y (mgr.24437) 339 : cluster [DBG] pgmap v517: 232 pgs: 1 active+remapped+backfill_toofull, 231 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 13/1107 objects misplaced (1.174%); 54 B/s, 1 objects/s recovering 2024-02-23T00:35:09.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:08.071096+0000 mon.a (mon.0) 2349 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:35:09.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:08.083788+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-47", "mode": "readproxy"}]': finished 2024-02-23T00:35:09.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:08.083863+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]': finished 2024-02-23T00:35:09.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:08.083935+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-02-23T00:35:09.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:08.088187+0000 mon.b (mon.1) 315 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:09.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:08.088628+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:07.624323+0000 osd.4 (osd.4) 171 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:35:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:07.627616+0000 osd.4 (osd.4) 172 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:35:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:07.631697+0000 mgr.y (mgr.24437) 339 : cluster [DBG] pgmap v517: 232 pgs: 1 active+remapped+backfill_toofull, 231 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 13/1107 objects misplaced (1.174%); 54 B/s, 1 objects/s recovering 2024-02-23T00:35:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:08.071096+0000 mon.a (mon.0) 2349 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:35:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:08.083788+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-47", "mode": "readproxy"}]': finished 2024-02-23T00:35:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:08.083863+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi087-82019-61"}]': finished 2024-02-23T00:35:09.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:08.083935+0000 mon.a (mon.0) 2352 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-02-23T00:35:09.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:08.088187+0000 mon.b (mon.1) 315 : audit [INF] from='client.? 172.21.15.87:0/2432481876' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:09.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:08.088628+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]: dispatch 2024-02-23T00:35:10.090 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:10.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:10.195 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:10.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:09.091972+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]': finished 2024-02-23T00:35:10.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:09.092057+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-02-23T00:35:10.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:09.111635+0000 mon.c (mon.2) 398 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:09.112227+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:09.112721+0000 mon.c (mon.2) 399 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.223 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:09.112965+0000 mon.a (mon.0) 2357 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:09.113416+0000 mon.c (mon.2) 400 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi087-82019-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:09.113682+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi087-82019-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:09.633147+0000 mon.a (mon.0) 2359 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:09.649219+0000 mon.a (mon.0) 2360 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:09.091972+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]': finished 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:09.092057+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:09.111635+0000 mon.c (mon.2) 398 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:09.112227+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:09.112721+0000 mon.c (mon.2) 399 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:09.112965+0000 mon.a (mon.0) 2357 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:09.113416+0000 mon.c (mon.2) 400 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi087-82019-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:10.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:09.113682+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi087-82019-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:10.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:09.633147+0000 mon.a (mon.0) 2359 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-23T00:35:10.225 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:09.649219+0000 mon.a (mon.0) 2360 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:10.225 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:09.091972+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi087-82019-61"}]': finished 2024-02-23T00:35:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:09.092057+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-02-23T00:35:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:09.111635+0000 mon.c (mon.2) 398 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:09.112227+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:09.112721+0000 mon.c (mon.2) 399 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:09.112965+0000 mon.a (mon.0) 2357 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:09.113416+0000 mon.c (mon.2) 400 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi087-82019-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:09.113682+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi087-82019-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:09.633147+0000 mon.a (mon.0) 2359 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-23T00:35:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:09.649219+0000 mon.a (mon.0) 2360 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:11.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:35:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:35:11.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:09.632217+0000 mgr.y (mgr.24437) 340 : cluster [DBG] pgmap v520: 231 pgs: 1 active+remapped+backfill_toofull, 230 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1074 objects misplaced (1.210%) 2024-02-23T00:35:11.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:09.655620+0000 osd.7 (osd.7) 179 : cluster [DBG] 6.1c scrub starts 2024-02-23T00:35:11.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:09.656832+0000 osd.7 (osd.7) 180 : cluster [DBG] 6.1c scrub ok 2024-02-23T00:35:11.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:10.099210+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi087-82019-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:11.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:10.099284+0000 mon.a (mon.0) 2362 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-23T00:35:11.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:10.099345+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-02-23T00:35:11.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:10.099962+0000 mon.c (mon.2) 401 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi087-82019-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:11.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:10.100301+0000 mon.a (mon.0) 2364 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi087-82019-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:10.164786+0000 osd.6 (osd.6) 133 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:10.167825+0000 osd.6 (osd.6) 134 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:10.388958+0000 osd.0 (osd.0) 177 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:10.391990+0000 osd.0 (osd.0) 178 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:10.514679+0000 mon.a (mon.0) 2365 : audit [INF] from='client.? 172.21.15.87:0/2446428378' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:09.632217+0000 mgr.y (mgr.24437) 340 : cluster [DBG] pgmap v520: 231 pgs: 1 active+remapped+backfill_toofull, 230 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1074 objects misplaced (1.210%) 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:09.655620+0000 osd.7 (osd.7) 179 : cluster [DBG] 6.1c scrub starts 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:09.656832+0000 osd.7 (osd.7) 180 : cluster [DBG] 6.1c scrub ok 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:10.099210+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi087-82019-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:10.099284+0000 mon.a (mon.0) 2362 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-23T00:35:11.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:10.099345+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-02-23T00:35:11.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:10.099962+0000 mon.c (mon.2) 401 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi087-82019-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:11.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:10.100301+0000 mon.a (mon.0) 2364 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi087-82019-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:11.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:10.164786+0000 osd.6 (osd.6) 133 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:35:11.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:10.167825+0000 osd.6 (osd.6) 134 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:35:11.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:10.388958+0000 osd.0 (osd.0) 177 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:35:11.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:10.391990+0000 osd.0 (osd.0) 178 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:35:11.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:10.514679+0000 mon.a (mon.0) 2365 : audit [INF] from='client.? 172.21.15.87:0/2446428378' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:35:11.479 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:11.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:11.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:11.586 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:11.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:09.632217+0000 mgr.y (mgr.24437) 340 : cluster [DBG] pgmap v520: 231 pgs: 1 active+remapped+backfill_toofull, 230 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1074 objects misplaced (1.210%) 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:09.655620+0000 osd.7 (osd.7) 179 : cluster [DBG] 6.1c scrub starts 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:09.656832+0000 osd.7 (osd.7) 180 : cluster [DBG] 6.1c scrub ok 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:10.099210+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi087-82019-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:10.099284+0000 mon.a (mon.0) 2362 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:10.099345+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:10.099962+0000 mon.c (mon.2) 401 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi087-82019-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:10.100301+0000 mon.a (mon.0) 2364 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi087-82019-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:10.164786+0000 osd.6 (osd.6) 133 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:10.167825+0000 osd.6 (osd.6) 134 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:10.388958+0000 osd.0 (osd.0) 177 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:10.391990+0000 osd.0 (osd.0) 178 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:35:11.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:10.514679+0000 mon.a (mon.0) 2365 : audit [INF] from='client.? 172.21.15.87:0/2446428378' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:35:11.619 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:11.641 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:11.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:11.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:11.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 -- 172.21.15.87:0/2915435516 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 msgr2=0x7ff2940fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:11.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2915435516 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 0x7ff2940fdf90 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 rx=0x7ff2840047b0 tx=0x7ff284009cb0).stop 2024-02-23T00:35:11.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 -- 172.21.15.87:0/2915435516 shutdown_connections 2024-02-23T00:35:11.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2915435516 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2940f8420 0x7ff294104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2915435516 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 0x7ff2940fdf90 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2915435516 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2940fe510 0x7ff2940f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:35:11.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 -- 172.21.15.87:0/2915435516 >> 172.21.15.87:0/2915435516 conn(0x7ff2940f3800 msgr2=0x7ff2940f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:11.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 -- 172.21.15.87:0/2915435516 shutdown_connections 2024-02-23T00:35:11.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 -- 172.21.15.87:0/2915435516 wait complete. 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 librados: starting msgr at 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.327+0000 7ff2997a1700 1 librados: starting objecter 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 Processor -- start 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 -- start start 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 librados: setting wanted keys 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 librados: calling monclient init 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2940f8420 0x7ff29406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 0x7ff29406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2940fe510 0x7ff29406bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2941165b0 con 0x7ff2940fdbb0 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff294116720 con 0x7ff2940f8420 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2997a1700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2941168e0 con 0x7ff2940fe510 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2937fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2940fe510 0x7ff29406bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:11.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2927fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 0x7ff29406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:11.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2937fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2940fe510 0x7ff29406bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35148/0 (socket says 172.21.15.87:35148) 2024-02-23T00:35:11.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2927fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 0x7ff29406b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52802/0 (socket says 172.21.15.87:52802) 2024-02-23T00:35:11.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2937fe700 1 -- 172.21.15.87:0/2446428378 learned_addr learned my addr 172.21.15.87:0/2446428378 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:11.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff292ffd700 1 --2- 172.21.15.87:0/2446428378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2940f8420 0x7ff29406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:11.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2927fc700 1 -- 172.21.15.87:0/2446428378 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2940fe510 msgr2=0x7ff29406bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:11.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2927fc700 1 --2- 172.21.15.87:0/2446428378 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2940fe510 0x7ff29406bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2927fc700 1 -- 172.21.15.87:0/2446428378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2940f8420 msgr2=0x7ff29406b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:11.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2927fc700 1 --2- 172.21.15.87:0/2446428378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2940f8420 0x7ff29406b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2927fc700 1 -- 172.21.15.87:0/2446428378 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c0097c0 con 0x7ff2940fdbb0 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2937fe700 1 --2- 172.21.15.87:0/2446428378 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2940fe510 0x7ff29406bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff2927fc700 1 --2- 172.21.15.87:0/2446428378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 0x7ff29406b5c0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 rx=0x7ff28800b4e0 tx=0x7ff288009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff27bfff700 10 client.?.objecter ms_handle_connect 0x7ff2940fdbb0 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.328+0000 7ff27bfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.329+0000 7ff27bfff700 1 -- 172.21.15.87:0/2446428378 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff288002320 con 0x7ff2940fdbb0 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.329+0000 7ff27bfff700 1 -- 172.21.15.87:0/2446428378 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff288002950 con 0x7ff2940fdbb0 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.329+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff284005040 con 0x7ff2940fdbb0 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.329+0000 7ff27bfff700 1 -- 172.21.15.87:0/2446428378 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2880129c0 con 0x7ff2940fdbb0 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.329+0000 7ff2997a1700 10 client.27105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.329+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff294116f20 con 0x7ff2940fdbb0 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.329+0000 7ff2997a1700 1 librados: init done 2024-02-23T00:35:11.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.330+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff294107110 con 0x7ff2940fdbb0 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.330+0000 7ff27bfff700 1 -- 172.21.15.87:0/2446428378 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff28801c440 con 0x7ff2940fdbb0 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.330+0000 7ff27bfff700 1 --2- 172.21.15.87:0/2446428378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff27c062c90 0x7ff27c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.330+0000 7ff27bfff700 1 -- 172.21.15.87:0/2446428378 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(359..359 src has 173..359) v4 ==== 8406+0+0 (secure 0 0 0) 0x7ff28807d330 con 0x7ff2940fdbb0 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.330+0000 7ff27bfff700 10 client.27105.objecter ms_dispatch 0x7ff294001550 osd_map(359..359 src has 173..359) v4 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.330+0000 7ff27bfff700 3 client.27105.objecter handle_osd_map got epochs [359,359] > 0 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.330+0000 7ff27bfff700 3 client.27105.objecter handle_osd_map decoding full epoch 359 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.330+0000 7ff292ffd700 1 --2- 172.21.15.87:0/2446428378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff27c062c90 0x7ff27c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.330+0000 7ff27bfff700 20 client.27105.objecter dump_active .. 0 homeless 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.331+0000 7ff27bfff700 10 client.27105.objecter ms_handle_connect 0x7ff27c062c90 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.331+0000 7ff292ffd700 1 --2- 172.21.15.87:0/2446428378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff27c062c90 0x7ff27c065150 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7ff284005010 tx=0x7ff284007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.335+0000 7ff27bfff700 1 -- 172.21.15.87:0/2446428378 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff2880506a0 con 0x7ff2940fdbb0 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:10.514+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff29405fa60 con 0x7ff2940fdbb0 2024-02-23T00:35:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.332+0000 7ff27bfff700 1 -- 172.21.15.87:0/2446428378 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v360) v1 ==== 87+0+0 (secure 0 0 0) 0x7ff288050290 con 0x7ff2940fdbb0 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.335+0000 7ff2997a1700 10 librados: watch_flush enter 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.335+0000 7ff2997a1700 10 librados: watch_flush exit 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.335+0000 7ff2997a1700 20 client.27105.objecter shutdown clearing up homeless session... 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.335+0000 7ff2997a1700 10 client.27105.objecter successfully canceled tick 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff27c062c90 msgr2=0x7ff27c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2446428378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff27c062c90 0x7ff27c065150 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7ff284005010 tx=0x7ff284007f90).stop 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 msgr2=0x7ff29406b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2446428378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 0x7ff29406b5c0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 rx=0x7ff28800b4e0 tx=0x7ff288009060).stop 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 shutdown_connections 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2446428378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff27c062c90 0x7ff27c065150 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2446428378 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff2940fe510 0x7ff29406bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2446428378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff2940f8420 0x7ff29406b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 --2- 172.21.15.87:0/2446428378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff2940fdbb0 0x7ff29406b5c0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:11.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 >> 172.21.15.87:0/2446428378 conn(0x7ff2940f3800 msgr2=0x7ff2940fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:11.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 shutdown_connections 2024-02-23T00:35:11.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 -- 172.21.15.87:0/2446428378 wait complete. 2024-02-23T00:35:11.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:11.338+0000 7ff2997a1700 1 librados: shutdown 2024-02-23T00:35:11.765 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:35:11.774 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:12.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:12] "GET /metrics HTTP/1.1" 200 187630 "" "Prometheus/2.33.4" 2024-02-23T00:35:12.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:10.590011+0000 osd.4 (osd.4) 173 : cluster [DBG] 4.e scrub starts 2024-02-23T00:35:12.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:10.593857+0000 osd.4 (osd.4) 174 : cluster [DBG] 4.e scrub ok 2024-02-23T00:35:12.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:10.701348+0000 osd.7 (osd.7) 181 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:10.704474+0000 osd.7 (osd.7) 182 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:10.837393+0000 mgr.y (mgr.24437) 341 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:11.094696+0000 mon.a (mon.0) 2366 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:11.333093+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.87:0/2446428378' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:11.333153+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:12.060778+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi087-82019-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi087-82019-62"}]': finished 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:12.060889+0000 mon.a (mon.0) 2370 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:10.590011+0000 osd.4 (osd.4) 173 : cluster [DBG] 4.e scrub starts 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:10.593857+0000 osd.4 (osd.4) 174 : cluster [DBG] 4.e scrub ok 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:10.701348+0000 osd.7 (osd.7) 181 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:35:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:10.704474+0000 osd.7 (osd.7) 182 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:35:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:10.837393+0000 mgr.y (mgr.24437) 341 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:11.094696+0000 mon.a (mon.0) 2366 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:11.333093+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.87:0/2446428378' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:35:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:11.333153+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-02-23T00:35:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:12.060778+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi087-82019-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi087-82019-62"}]': finished 2024-02-23T00:35:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:12.060889+0000 mon.a (mon.0) 2370 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-02-23T00:35:12.590 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:35:12.590 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:35:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:10.590011+0000 osd.4 (osd.4) 173 : cluster [DBG] 4.e scrub starts 2024-02-23T00:35:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:10.593857+0000 osd.4 (osd.4) 174 : cluster [DBG] 4.e scrub ok 2024-02-23T00:35:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:10.701348+0000 osd.7 (osd.7) 181 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:35:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:10.704474+0000 osd.7 (osd.7) 182 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:35:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:10.837393+0000 mgr.y (mgr.24437) 341 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:11.094696+0000 mon.a (mon.0) 2366 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:11.333093+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.87:0/2446428378' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:35:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:11.333153+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-02-23T00:35:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:12.060778+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi087-82019-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi087-82019-62"}]': finished 2024-02-23T00:35:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:12.060889+0000 mon.a (mon.0) 2370 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-02-23T00:35:13.317 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-23T00:35:13.454 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:13.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:13.559 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:13.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:11.632693+0000 mgr.y (mgr.24437) 342 : cluster [DBG] pgmap v523: 231 pgs: 1 active+remapped+backfill_toofull, 2 peering, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:13.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:13.063616+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-02-23T00:35:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:11.632693+0000 mgr.y (mgr.24437) 342 : cluster [DBG] pgmap v523: 231 pgs: 1 active+remapped+backfill_toofull, 2 peering, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:13.063616+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-02-23T00:35:13.591 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:13.611 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:13.613 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:11.632693+0000 mgr.y (mgr.24437) 342 : cluster [DBG] pgmap v523: 231 pgs: 1 active+remapped+backfill_toofull, 2 peering, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:13.613 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:13.063616+0000 mon.a (mon.0) 2371 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-02-23T00:35:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:13.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:13.715 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:13.744 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:13.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 -- 172.21.15.87:0/78827387 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 msgr2=0x7f0f9c101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:13.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/78827387 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 0x7f0f9c101c20 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f0f90004870 tx=0x7f0f90009cb0).stop 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 -- 172.21.15.87:0/78827387 shutdown_connections 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/78827387 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f9c102110 0x7f0f9c104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/78827387 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f9c0f8bb0 0x7f0f9c0f8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/78827387 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 0x7f0f9c101c20 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 -- 172.21.15.87:0/78827387 >> 172.21.15.87:0/78827387 conn(0x7f0f9c0f26e0 msgr2=0x7f0f9c0f4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 -- 172.21.15.87:0/78827387 shutdown_connections 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 -- 172.21.15.87:0/78827387 wait complete. 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 librados: starting msgr at 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 librados: starting objecter 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 Processor -- start 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 -- start start 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 librados: setting wanted keys 2024-02-23T00:35:13.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.847+0000 7f0fa1eca700 1 librados: calling monclient init 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f9c0f8bb0 0x7f0f9c0f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 0x7f0f9c0f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f9c102110 0x7f0f9c0f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c116650 con 0x7f0f9c0f8bb0 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c1167c0 con 0x7f0f9c102110 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c116980 con 0x7f0f9c0f9510 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 0x7f0f9c0f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 0x7f0f9c0f7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35704/0 (socket says 172.21.15.87:35704) 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9affd700 1 -- 172.21.15.87:0/3475263435 learned_addr learned my addr 172.21.15.87:0/3475263435 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9b7fe700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f9c0f8bb0 0x7f0f9c0f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9bfff700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f9c102110 0x7f0f9c0f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9affd700 1 -- 172.21.15.87:0/3475263435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f9c102110 msgr2=0x7f0f9c0f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9affd700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f9c102110 0x7f0f9c0f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:13.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9affd700 1 -- 172.21.15.87:0/3475263435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f9c0f8bb0 msgr2=0x7f0f9c0f6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9affd700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f9c0f8bb0 0x7f0f9c0f6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9affd700 1 -- 172.21.15.87:0/3475263435 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c005040 con 0x7f0f9c0f9510 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9bfff700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f9c102110 0x7f0f9c0f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9b7fe700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f9c0f8bb0 0x7f0f9c0f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f9affd700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 0x7f0f9c0f7050 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f0f90009eb0 tx=0x7f0f9000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f98ff9700 10 client.?.objecter ms_handle_connect 0x7f0f9c0f9510 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f98ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f98ff9700 1 -- 172.21.15.87:0/3475263435 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9000e070 con 0x7f0f9c0f9510 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f90005040 con 0x7f0f9c0f9510 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f98ff9700 1 -- 172.21.15.87:0/3475263435 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f90013da0 con 0x7f0f9c0f9510 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0f98ff9700 1 -- 172.21.15.87:0/3475263435 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90019070 con 0x7f0f9c0f9510 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 10 client.37571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c116e70 con 0x7f0f9c0f9510 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.848+0000 7f0fa1eca700 1 librados: init done 2024-02-23T00:35:13.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.849+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c05f950 con 0x7f0f9c0f9510 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f98ff9700 1 -- 172.21.15.87:0/3475263435 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0f9001e420 con 0x7f0f9c0f9510 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f98ff9700 1 --2- 172.21.15.87:0/3475263435 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f84062c90 0x7f0f84065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f98ff9700 1 -- 172.21.15.87:0/3475263435 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(362..362 src has 173..362) v4 ==== 8926+0+0 (secure 0 0 0) 0x7f0f9007e0a0 con 0x7f0f9c0f9510 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f98ff9700 10 client.37571.objecter ms_dispatch 0x7f0f9c001550 osd_map(362..362 src has 173..362) v4 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f98ff9700 3 client.37571.objecter handle_osd_map got epochs [362,362] > 0 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f98ff9700 3 client.37571.objecter handle_osd_map decoding full epoch 362 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f9b7fe700 1 --2- 172.21.15.87:0/3475263435 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f84062c90 0x7f0f84065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f98ff9700 20 client.37571.objecter dump_active .. 0 homeless 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f98ff9700 10 client.37571.objecter ms_handle_connect 0x7f0f84062c90 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.850+0000 7f0f9b7fe700 1 --2- 172.21.15.87:0/3475263435 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f84062c90 0x7f0f84065150 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7f0f8c009760 tx=0x7f0f8c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:13.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:13.854+0000 7f0f98ff9700 1 -- 172.21.15.87:0/3475263435 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0f90051360 con 0x7f0f9c0f9510 2024-02-23T00:35:13.892 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:13.892 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:13.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:13.922 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:13.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:35:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:35:13.997 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:14.029 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.040+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f9c0f80b0 con 0x7f0f84062c90 2024-02-23T00:35:14.051 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:14.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.057+0000 7f0f98ff9700 1 -- 172.21.15.87:0/3475263435 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+598532 (secure 0 0 0) 0x7f0f9c0f80b0 con 0x7f0f84062c90 2024-02-23T00:35:14.066 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:35:14.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 10 librados: watch_flush enter 2024-02-23T00:35:14.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 10 librados: watch_flush exit 2024-02-23T00:35:14.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 20 client.37571.objecter shutdown clearing up homeless session... 2024-02-23T00:35:14.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 10 client.37571.objecter successfully canceled tick 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f84062c90 msgr2=0x7f0f84065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/3475263435 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f84062c90 0x7f0f84065150 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 rx=0x7f0f8c009760 tx=0x7f0f8c00b040).stop 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 msgr2=0x7f0f9c0f7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 0x7f0f9c0f7050 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f0f90009eb0 tx=0x7f0f9000b040).stop 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 shutdown_connections 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/3475263435 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0f84062c90 0x7f0f84065150 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0f9c102110 0x7f0f9c0f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0f9c0f8bb0 0x7f0f9c0f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 --2- 172.21.15.87:0/3475263435 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0f9c0f9510 0x7f0f9c0f7050 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 >> 172.21.15.87:0/3475263435 conn(0x7f0f9c0f26e0 msgr2=0x7f0f9c0f32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 shutdown_connections 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 -- 172.21.15.87:0/3475263435 wait complete. 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:14.061+0000 7f0fa1eca700 1 librados: shutdown 2024-02-23T00:35:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:35:14.155 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:14.184 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:14.205 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:14.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:14.308 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:14.338 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:14.358 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:14.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:14.461 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:14.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:13.633148+0000 mgr.y (mgr.24437) 343 : cluster [DBG] pgmap v526: 238 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 1 peering, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s; 13/1092 objects misplaced (1.190%); 27 B/s, 4 objects/s recovering 2024-02-23T00:35:14.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:13.634068+0000 mon.a (mon.0) 2372 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.17"}]: dispatch 2024-02-23T00:35:14.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:13.634241+0000 mon.a (mon.0) 2373 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.17"}]: dispatch 2024-02-23T00:35:14.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:14.041382+0000 mgr.y (mgr.24437) 344 : audit [DBG] from='client.37571 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:35:14.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:14.063973+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.17"}]': finished 2024-02-23T00:35:14.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:14.064045+0000 mon.a (mon.0) 2375 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.17"}]': finished 2024-02-23T00:35:14.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:14.064098+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:14.068226+0000 mon.c (mon.2) 402 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:14.068571+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:13.633148+0000 mgr.y (mgr.24437) 343 : cluster [DBG] pgmap v526: 238 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 1 peering, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s; 13/1092 objects misplaced (1.190%); 27 B/s, 4 objects/s recovering 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:13.634068+0000 mon.a (mon.0) 2372 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.17"}]: dispatch 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:13.634241+0000 mon.a (mon.0) 2373 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.17"}]: dispatch 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:14.041382+0000 mgr.y (mgr.24437) 344 : audit [DBG] from='client.37571 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:14.063973+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.17"}]': finished 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:14.064045+0000 mon.a (mon.0) 2375 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.17"}]': finished 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:14.064098+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:14.068226+0000 mon.c (mon.2) 402 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:14.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:14.068571+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:14.490 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:14.512 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:14.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:14.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:14.616 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:14.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:13.633148+0000 mgr.y (mgr.24437) 343 : cluster [DBG] pgmap v526: 238 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 1 peering, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 11 KiB/s rd, 0 B/s wr, 18 op/s; 13/1092 objects misplaced (1.190%); 27 B/s, 4 objects/s recovering 2024-02-23T00:35:14.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:13.634068+0000 mon.a (mon.0) 2372 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "4.17"}]: dispatch 2024-02-23T00:35:14.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:13.634241+0000 mon.a (mon.0) 2373 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.17"}]: dispatch 2024-02-23T00:35:14.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:14.041382+0000 mgr.y (mgr.24437) 344 : audit [DBG] from='client.37571 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:35:14.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:14.063973+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "4.17"}]': finished 2024-02-23T00:35:14.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:14.064045+0000 mon.a (mon.0) 2375 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.17"}]': finished 2024-02-23T00:35:14.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:14.064098+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-02-23T00:35:14.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:14.068226+0000 mon.c (mon.2) 402 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:14.642 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:14.068571+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:14.644 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:14.662 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:14.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:14.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:14.713 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:35:14.766 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:14.794 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:14.812 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:14.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:14.916 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:14.943 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:14.961 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:14.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:15.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:15.092 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:15.113 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:15.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:15.217 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:15.245 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:15.368 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:15.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:15.472 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:15.499 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:15.517 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:15.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:15.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:15.621 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:15.650 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:15.723 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:15.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:15.741 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-02-23T00:35:13.632877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007176,"num_objects":364,"num_object_clones":2,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":364,"num_whiteouts":0,"num_read":1385,"num_read_kb":1022,"num_write":1177,"num_write_kb":4516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":4003359,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2256,"ondisk_log_size":2256,"up":690,"acting":690,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":696,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2156828,"kb_used_data":39396,"kb_used_omap":0,"kb_used_meta":2117376,"kb_avail":654059236,"statfs":{"total":671965249536,"available":669756657664,"internally_reserved":0,"allocated":40341504,"data_stored":26077502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2168193024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":45,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000999"},"pg_stats":[{"pgid":"200.1f","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090366+0000","last_change":"2024-02-23T00:35:05.085184+0000","last_active":"2024-02-23T00:35:09.090366+0000","last_peered":"2024-02-23T00:35:09.090366+0000","last_clean":"2024-02-23T00:35:09.090366+0000","last_became_active":"2024-02-23T00:35:05.084890+0000","last_became_peered":"2024-02-23T00:35:05.084890+0000","last_unstale":"2024-02-23T00:35:09.090366+0000","last_undegraded":"2024-02-23T00:35:09.090366+0000","last_fullsized":"2024-02-23T00:35:09.090366+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"200.1e","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092380+0000","last_change":"2024-02-23T00:35:05.088360+0000","last_active":"2024-02-23T00:35:09.092380+0000","last_peered":"2024-02-23T00:35:09.092380+0000","last_clean":"2024-02-23T00:35:09.092380+0000","last_became_active":"2024-02-23T00:35:05.087914+0000","last_became_peered":"2024-02-23T00:35:05.087914+0000","last_unstale":"2024-02-23T00:35:09.092380+0000","last_undegraded":"2024-02-23T00:35:09.092380+0000","last_fullsized":"2024-02-23T00:35:09.092380+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"200.1d","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089936+0000","last_change":"2024-02-23T00:35:05.088631+0000","last_active":"2024-02-23T00:35:09.089936+0000","last_peered":"2024-02-23T00:35:09.089936+0000","last_clean":"2024-02-23T00:35:09.089936+0000","last_became_active":"2024-02-23T00:35:05.088288+0000","last_became_peered":"2024-02-23T00:35:05.088288+0000","last_unstale":"2024-02-23T00:35:09.089936+0000","last_undegraded":"2024-02-23T00:35:09.089936+0000","last_fullsized":"2024-02-23T00:35:09.089936+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"200.1c","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089673+0000","last_change":"2024-02-23T00:35:05.083363+0000","last_active":"2024-02-23T00:35:09.089673+0000","last_peered":"2024-02-23T00:35:09.089673+0000","last_clean":"2024-02-23T00:35:09.089673+0000","last_became_active":"2024-02-23T00:35:05.083028+0000","last_became_peered":"2024-02-23T00:35:05.083028+0000","last_unstale":"2024-02-23T00:35:09.089673+0000","last_undegraded":"2024-02-23T00:35:09.089673+0000","last_fullsized":"2024-02-23T00:35:09.089673+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"200.1b","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092041+0000","last_change":"2024-02-23T00:35:05.085123+0000","last_active":"2024-02-23T00:35:09.092041+0000","last_peered":"2024-02-23T00:35:09.092041+0000","last_clean":"2024-02-23T00:35:09.092041+0000","last_became_active":"2024-02-23T00:35:05.084663+0000","last_became_peered":"2024-02-23T00:35:05.084663+0000","last_unstale":"2024-02-23T00:35:09.092041+0000","last_undegraded":"2024-02-23T00:35:09.092041+0000","last_fullsized":"2024-02-23T00:35:09.092041+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"200.1a","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095792+0000","last_change":"2024-02-23T00:35:05.084260+0000","last_active":"2024-02-23T00:35:09.095792+0000","last_peered":"2024-02-23T00:35:09.095792+0000","last_clean":"2024-02-23T00:35:09.095792+0000","last_became_active":"2024-02-23T00:35:05.083950+0000","last_became_peered":"2024-02-23T00:35:05.083950+0000","last_unstale":"2024-02-23T00:35:09.095792+0000","last_undegraded":"2024-02-23T00:35:09.095792+0000","last_fullsized":"2024-02-23T00:35:09.095792+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"200.19","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090966+0000","last_change":"2024-02-23T00:35:05.082006+0000","last_active":"2024-02-23T00:35:09.090966+0000","last_peered":"2024-02-23T00:35:09.090966+0000","last_clean":"2024-02-23T00:35:09.090966+0000","last_became_active":"2024-02-23T00:35:05.081707+0000","last_became_peered":"2024-02-23T00:35:05.081707+0000","last_unstale":"2024-02-23T00:35:09.090966+0000","last_undegraded":"2024-02-23T00:35:09.090966+0000","last_fullsized":"2024-02-23T00:35:09.090966+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"200.18","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.088767+0000","last_change":"2024-02-23T00:35:05.083379+0000","last_active":"2024-02-23T00:35:09.088767+0000","last_peered":"2024-02-23T00:35:09.088767+0000","last_clean":"2024-02-23T00:35:09.088767+0000","last_became_active":"2024-02-23T00:35:05.082891+0000","last_became_peered":"2024-02-23T00:35:05.082891+0000","last_unstale":"2024-02-23T00:35:09.088767+0000","last_undegraded":"2024-02-23T00:35:09.088767+0000","last_fullsized":"2024-02-23T00:35:09.088767+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"200.17","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092278+0000","last_change":"2024-02-23T00:35:05.088703+0000","last_active":"2024-02-23T00:35:09.092278+0000","last_peered":"2024-02-23T00:35:09.092278+0000","last_clean":"2024-02-23T00:35:09.092278+0000","last_became_active":"2024-02-23T00:35:05.088328+0000","last_became_peered":"2024-02-23T00:35:05.088328+0000","last_unstale":"2024-02-23T00:35:09.092278+0000","last_undegraded":"2024-02-23T00:35:09.092278+0000","last_fullsized":"2024-02-23T00:35:09.092278+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"200.16","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095726+0000","last_change":"2024-02-23T00:35:05.090994+0000","last_active":"2024-02-23T00:35:09.095726+0000","last_peered":"2024-02-23T00:35:09.095726+0000","last_clean":"2024-02-23T00:35:09.095726+0000","last_became_active":"2024-02-23T00:35:05.090598+0000","last_became_peered":"2024-02-23T00:35:05.090598+0000","last_unstale":"2024-02-23T00:35:09.095726+0000","last_undegraded":"2024-02-23T00:35:09.095726+0000","last_fullsized":"2024-02-23T00:35:09.095726+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"200.15","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114430+0000","last_change":"2024-02-23T00:35:05.090892+0000","last_active":"2024-02-23T00:35:09.114430+0000","last_peered":"2024-02-23T00:35:09.114430+0000","last_clean":"2024-02-23T00:35:09.114430+0000","last_became_active":"2024-02-23T00:35:05.090555+0000","last_became_peered":"2024-02-23T00:35:05.090555+0000","last_unstale":"2024-02-23T00:35:09.114430+0000","last_undegraded":"2024-02-23T00:35:09.114430+0000","last_fullsized":"2024-02-23T00:35:09.114430+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"200.14","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096565+0000","last_change":"2024-02-23T00:35:05.090276+0000","last_active":"2024-02-23T00:35:09.096565+0000","last_peered":"2024-02-23T00:35:09.096565+0000","last_clean":"2024-02-23T00:35:09.096565+0000","last_became_active":"2024-02-23T00:35:05.089944+0000","last_became_peered":"2024-02-23T00:35:05.089944+0000","last_unstale":"2024-02-23T00:35:09.096565+0000","last_undegraded":"2024-02-23T00:35:09.096565+0000","last_fullsized":"2024-02-23T00:35:09.096565+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"200.13","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.113783+0000","last_change":"2024-02-23T00:35:05.079590+0000","last_active":"2024-02-23T00:35:09.113783+0000","last_peered":"2024-02-23T00:35:09.113783+0000","last_clean":"2024-02-23T00:35:09.113783+0000","last_became_active":"2024-02-23T00:35:05.079284+0000","last_became_peered":"2024-02-23T00:35:05.079284+0000","last_unstale":"2024-02-23T00:35:09.113783+0000","last_undegraded":"2024-02-23T00:35:09.113783+0000","last_fullsized":"2024-02-23T00:35:09.113783+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"200.12","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091990+0000","last_change":"2024-02-23T00:35:05.086167+0000","last_active":"2024-02-23T00:35:09.091990+0000","last_peered":"2024-02-23T00:35:09.091990+0000","last_clean":"2024-02-23T00:35:09.091990+0000","last_became_active":"2024-02-23T00:35:05.085866+0000","last_became_peered":"2024-02-23T00:35:05.085866+0000","last_unstale":"2024-02-23T00:35:09.091990+0000","last_undegraded":"2024-02-23T00:35:09.091990+0000","last_fullsized":"2024-02-23T00:35:09.091990+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"200.11","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.108842+0000","last_change":"2024-02-23T00:35:05.088848+0000","last_active":"2024-02-23T00:35:09.108842+0000","last_peered":"2024-02-23T00:35:09.108842+0000","last_clean":"2024-02-23T00:35:09.108842+0000","last_became_active":"2024-02-23T00:35:05.088515+0000","last_became_peered":"2024-02-23T00:35:05.088515+0000","last_unstale":"2024-02-23T00:35:09.108842+0000","last_undegraded":"2024-02-23T00:35:09.108842+0000","last_fullsized":"2024-02-23T00:35:09.108842+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"200.10","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093493+0000","last_change":"2024-02-23T00:35:05.085675+0000","last_active":"2024-02-23T00:35:09.093493+0000","last_peered":"2024-02-23T00:35:09.093493+0000","last_clean":"2024-02-23T00:35:09.093493+0000","last_became_active":"2024-02-23T00:35:05.085432+0000","last_became_peered":"2024-02-23T00:35:05.085432+0000","last_unstale":"2024-02-23T00:35:09.093493+0000","last_undegraded":"2024-02-23T00:35:09.093493+0000","last_fullsized":"2024-02-23T00:35:09.093493+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"200.f","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110299+0000","last_change":"2024-02-23T00:35:05.087872+0000","last_active":"2024-02-23T00:35:09.110299+0000","last_peered":"2024-02-23T00:35:09.110299+0000","last_clean":"2024-02-23T00:35:09.110299+0000","last_became_active":"2024-02-23T00:35:05.087533+0000","last_became_peered":"2024-02-23T00:35:05.087533+0000","last_unstale":"2024-02-23T00:35:09.110299+0000","last_undegraded":"2024-02-23T00:35:09.110299+0000","last_fullsized":"2024-02-23T00:35:09.110299+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"200.e","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110040+0000","last_change":"2024-02-23T00:35:05.090438+0000","last_active":"2024-02-23T00:35:09.110040+0000","last_peered":"2024-02-23T00:35:09.110040+0000","last_clean":"2024-02-23T00:35:09.110040+0000","last_became_active":"2024-02-23T00:35:05.090110+0000","last_became_peered":"2024-02-23T00:35:05.090110+0000","last_unstale":"2024-02-23T00:35:09.110040+0000","last_undegraded":"2024-02-23T00:35:09.110040+0000","last_fullsized":"2024-02-23T00:35:09.110040+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"200.d","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092407+0000","last_change":"2024-02-23T00:35:05.088307+0000","last_active":"2024-02-23T00:35:09.092407+0000","last_peered":"2024-02-23T00:35:09.092407+0000","last_clean":"2024-02-23T00:35:09.092407+0000","last_became_active":"2024-02-23T00:35:05.088016+0000","last_became_peered":"2024-02-23T00:35:05.088016+0000","last_unstale":"2024-02-23T00:35:09.092407+0000","last_undegraded":"2024-02-23T00:35:09.092407+0000","last_fullsized":"2024-02-23T00:35:09.092407+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"200.c","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091952+0000","last_change":"2024-02-23T00:35:05.090643+0000","last_active":"2024-02-23T00:35:09.091952+0000","last_peered":"2024-02-23T00:35:09.091952+0000","last_clean":"2024-02-23T00:35:09.091952+0000","last_became_active":"2024-02-23T00:35:05.090231+0000","last_became_peered":"2024-02-23T00:35:05.090231+0000","last_unstale":"2024-02-23T00:35:09.091952+0000","last_undegraded":"2024-02-23T00:35:09.091952+0000","last_fullsized":"2024-02-23T00:35:09.091952+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"200.b","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091193+0000","last_change":"2024-02-23T00:35:05.088771+0000","last_active":"2024-02-23T00:35:09.091193+0000","last_peered":"2024-02-23T00:35:09.091193+0000","last_clean":"2024-02-23T00:35:09.091193+0000","last_became_active":"2024-02-23T00:35:05.088493+0000","last_became_peered":"2024-02-23T00:35:05.088493+0000","last_unstale":"2024-02-23T00:35:09.091193+0000","last_undegraded":"2024-02-23T00:35:09.091193+0000","last_fullsized":"2024-02-23T00:35:09.091193+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"200.a","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.113746+0000","last_change":"2024-02-23T00:35:05.090951+0000","last_active":"2024-02-23T00:35:09.113746+0000","last_peered":"2024-02-23T00:35:09.113746+0000","last_clean":"2024-02-23T00:35:09.113746+0000","last_became_active":"2024-02-23T00:35:05.090643+0000","last_became_peered":"2024-02-23T00:35:05.090643+0000","last_unstale":"2024-02-23T00:35:09.113746+0000","last_undegraded":"2024-02-23T00:35:09.113746+0000","last_fullsized":"2024-02-23T00:35:09.113746+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"200.9","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.108930+0000","last_change":"2024-02-23T00:35:05.084575+0000","last_active":"2024-02-23T00:35:09.108930+0000","last_peered":"2024-02-23T00:35:09.108930+0000","last_clean":"2024-02-23T00:35:09.108930+0000","last_became_active":"2024-02-23T00:35:05.084265+0000","last_became_peered":"2024-02-23T00:35:05.084265+0000","last_unstale":"2024-02-23T00:35:09.108930+0000","last_undegraded":"2024-02-23T00:35:09.108930+0000","last_fullsized":"2024-02-23T00:35:09.108930+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"200.8","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.088775+0000","last_change":"2024-02-23T00:35:05.091775+0000","last_active":"2024-02-23T00:35:09.088775+0000","last_peered":"2024-02-23T00:35:09.088775+0000","last_clean":"2024-02-23T00:35:09.088775+0000","last_became_active":"2024-02-23T00:35:05.091477+0000","last_became_peered":"2024-02-23T00:35:05.091477+0000","last_unstale":"2024-02-23T00:35:09.088775+0000","last_undegraded":"2024-02-23T00:35:09.088775+0000","last_fullsized":"2024-02-23T00:35:09.088775+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"202.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:35:12.056180+0000","last_change":"2024-02-23T00:35:12.056180+0000","last_active":"2024-02-23T00:35:12.056180+0000","last_peered":"2024-02-23T00:35:12.056180+0000","last_clean":"2024-02-23T00:35:12.056180+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:35:12.056180+0000","last_undegraded":"2024-02-23T00:35:12.056180+0000","last_fullsized":"2024-02-23T00:35:12.056180+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_clean_scrub_stamp":"2024-02-23T00:35:12.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"200.7","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095667+0000","last_change":"2024-02-23T00:35:05.090802+0000","last_active":"2024-02-23T00:35:09.095667+0000","last_peered":"2024-02-23T00:35:09.095667+0000","last_clean":"2024-02-23T00:35:09.095667+0000","last_became_active":"2024-02-23T00:35:05.090502+0000","last_became_peered":"2024-02-23T00:35:05.090502+0000","last_unstale":"2024-02-23T00:35:09.095667+0000","last_undegraded":"2024-02-23T00:35:09.095667+0000","last_fullsized":"2024-02-23T00:35:09.095667+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"202.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:35:12.056180+0000","last_change":"2024-02-23T00:35:12.056180+0000","last_active":"2024-02-23T00:35:12.056180+0000","last_peered":"2024-02-23T00:35:12.056180+0000","last_clean":"2024-02-23T00:35:12.056180+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:35:12.056180+0000","last_undegraded":"2024-02-23T00:35:12.056180+0000","last_fullsized":"2024-02-23T00:35:12.056180+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_clean_scrub_stamp":"2024-02-23T00:35:12.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"200.6","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092787+0000","last_change":"2024-02-23T00:35:05.087967+0000","last_active":"2024-02-23T00:35:09.092787+0000","last_peered":"2024-02-23T00:35:09.092787+0000","last_clean":"2024-02-23T00:35:09.092787+0000","last_became_active":"2024-02-23T00:35:05.087633+0000","last_became_peered":"2024-02-23T00:35:05.087633+0000","last_unstale":"2024-02-23T00:35:09.092787+0000","last_undegraded":"2024-02-23T00:35:09.092787+0000","last_fullsized":"2024-02-23T00:35:09.092787+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"202.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:35:12.056180+0000","last_change":"2024-02-23T00:35:12.056180+0000","last_active":"2024-02-23T00:35:12.056180+0000","last_peered":"2024-02-23T00:35:12.056180+0000","last_clean":"2024-02-23T00:35:12.056180+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:35:12.056180+0000","last_undegraded":"2024-02-23T00:35:12.056180+0000","last_fullsized":"2024-02-23T00:35:12.056180+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_clean_scrub_stamp":"2024-02-23T00:35:12.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"200.5","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092068+0000","last_change":"2024-02-23T00:35:05.090986+0000","last_active":"2024-02-23T00:35:09.092068+0000","last_peered":"2024-02-23T00:35:09.092068+0000","last_clean":"2024-02-23T00:35:09.092068+0000","last_became_active":"2024-02-23T00:35:05.090687+0000","last_became_peered":"2024-02-23T00:35:05.090687+0000","last_unstale":"2024-02-23T00:35:09.092068+0000","last_undegraded":"2024-02-23T00:35:09.092068+0000","last_fullsized":"2024-02-23T00:35:09.092068+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"202.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:35:12.056180+0000","last_change":"2024-02-23T00:35:12.056180+0000","last_active":"2024-02-23T00:35:12.056180+0000","last_peered":"2024-02-23T00:35:12.056180+0000","last_clean":"2024-02-23T00:35:12.056180+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:35:12.056180+0000","last_undegraded":"2024-02-23T00:35:12.056180+0000","last_fullsized":"2024-02-23T00:35:12.056180+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_clean_scrub_stamp":"2024-02-23T00:35:12.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"200.4","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109325+0000","last_change":"2024-02-23T00:35:05.087691+0000","last_active":"2024-02-23T00:35:09.109325+0000","last_peered":"2024-02-23T00:35:09.109325+0000","last_clean":"2024-02-23T00:35:09.109325+0000","last_became_active":"2024-02-23T00:35:05.087376+0000","last_became_peered":"2024-02-23T00:35:05.087376+0000","last_unstale":"2024-02-23T00:35:09.109325+0000","last_undegraded":"2024-02-23T00:35:09.109325+0000","last_fullsized":"2024-02-23T00:35:09.109325+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"202.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:35:12.056180+0000","last_change":"2024-02-23T00:35:12.056180+0000","last_active":"2024-02-23T00:35:12.056180+0000","last_peered":"2024-02-23T00:35:12.056180+0000","last_clean":"2024-02-23T00:35:12.056180+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:35:12.056180+0000","last_undegraded":"2024-02-23T00:35:12.056180+0000","last_fullsized":"2024-02-23T00:35:12.056180+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_clean_scrub_stamp":"2024-02-23T00:35:12.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"200.3","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091813+0000","last_change":"2024-02-23T00:35:05.083796+0000","last_active":"2024-02-23T00:35:09.091813+0000","last_peered":"2024-02-23T00:35:09.091813+0000","last_clean":"2024-02-23T00:35:09.091813+0000","last_became_active":"2024-02-23T00:35:05.083577+0000","last_became_peered":"2024-02-23T00:35:05.083577+0000","last_unstale":"2024-02-23T00:35:09.091813+0000","last_undegraded":"2024-02-23T00:35:09.091813+0000","last_fullsized":"2024-02-23T00:35:09.091813+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"202.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:35:12.056180+0000","last_change":"2024-02-23T00:35:12.056180+0000","last_active":"2024-02-23T00:35:12.056180+0000","last_peered":"2024-02-23T00:35:12.056180+0000","last_clean":"2024-02-23T00:35:12.056180+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:35:12.056180+0000","last_undegraded":"2024-02-23T00:35:12.056180+0000","last_fullsized":"2024-02-23T00:35:12.056180+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_clean_scrub_stamp":"2024-02-23T00:35:12.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"200.2","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.113749+0000","last_change":"2024-02-23T00:35:05.086121+0000","last_active":"2024-02-23T00:35:09.113749+0000","last_peered":"2024-02-23T00:35:09.113749+0000","last_clean":"2024-02-23T00:35:09.113749+0000","last_became_active":"2024-02-23T00:35:05.085779+0000","last_became_peered":"2024-02-23T00:35:05.085779+0000","last_unstale":"2024-02-23T00:35:09.113749+0000","last_undegraded":"2024-02-23T00:35:09.113749+0000","last_fullsized":"2024-02-23T00:35:09.113749+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"202.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:35:12.056180+0000","last_change":"2024-02-23T00:35:12.056180+0000","last_active":"2024-02-23T00:35:12.056180+0000","last_peered":"2024-02-23T00:35:12.056180+0000","last_clean":"2024-02-23T00:35:12.056180+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:35:12.056180+0000","last_undegraded":"2024-02-23T00:35:12.056180+0000","last_fullsized":"2024-02-23T00:35:12.056180+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_clean_scrub_stamp":"2024-02-23T00:35:12.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"200.1","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091805+0000","last_change":"2024-02-23T00:35:05.092078+0000","last_active":"2024-02-23T00:35:09.091805+0000","last_peered":"2024-02-23T00:35:09.091805+0000","last_clean":"2024-02-23T00:35:09.091805+0000","last_became_active":"2024-02-23T00:35:05.091748+0000","last_became_peered":"2024-02-23T00:35:05.091748+0000","last_unstale":"2024-02-23T00:35:09.091805+0000","last_undegraded":"2024-02-23T00:35:09.091805+0000","last_fullsized":"2024-02-23T00:35:09.091805+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"202.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:35:12.056180+0000","last_change":"2024-02-23T00:35:12.056180+0000","last_active":"2024-02-23T00:35:12.056180+0000","last_peered":"2024-02-23T00:35:12.056180+0000","last_clean":"2024-02-23T00:35:12.056180+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:35:12.056180+0000","last_undegraded":"2024-02-23T00:35:12.056180+0000","last_fullsized":"2024-02-23T00:35:12.056180+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:12.056180+0000","last_clean_scrub_stamp":"2024-02-23T00:35:12.056180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"200.0","version":"0'0","reported_seq":14,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091695+0000","last_change":"2024-02-23T00:35:05.086449+0000","last_active":"2024-02-23T00:35:09.091695+0000","last_peered":"2024-02-23T00:35:09.091695+0000","last_clean":"2024-02-23T00:35:09.091695+0000","last_became_active":"2024-02-23T00:35:05.085624+0000","last_became_peered":"2024-02-23T00:35:05.085624+0000","last_unstale":"2024-02-23T00:35:09.091695+0000","last_undegraded":"2024-02-23T00:35:09.091695+0000","last_fullsized":"2024-02-23T00:35:09.091695+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":353,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:04.063728+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.063728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.0","version":"222'68","reported_seq":85,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.108753+0000","last_change":"2024-02-23T00:34:57.645284+0000","last_active":"2024-02-23T00:35:09.108753+0000","last_peered":"2024-02-23T00:35:09.108753+0000","last_clean":"2024-02-23T00:35:09.108753+0000","last_became_active":"2024-02-23T00:34:42.057217+0000","last_became_peered":"2024-02-23T00:34:42.057217+0000","last_unstale":"2024-02-23T00:35:09.108753+0000","last_undegraded":"2024-02-23T00:35:09.108753+0000","last_fullsized":"2024-02-23T00:35:09.108753+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'68","last_scrub_stamp":"2024-02-23T00:34:57.645204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.645204+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1","version":"340'86","reported_seq":376,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089078+0000","last_change":"2024-02-23T00:34:16.724717+0000","last_active":"2024-02-23T00:35:09.089078+0000","last_peered":"2024-02-23T00:35:09.089078+0000","last_clean":"2024-02-23T00:35:09.089078+0000","last_became_active":"2024-02-23T00:31:06.258140+0000","last_became_peered":"2024-02-23T00:31:06.258140+0000","last_unstale":"2024-02-23T00:35:09.089078+0000","last_undegraded":"2024-02-23T00:35:09.089078+0000","last_fullsized":"2024-02-23T00:35:09.089078+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'85","last_scrub_stamp":"2024-02-23T00:34:16.724630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:16.724630+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.2","version":"264'20","reported_seq":321,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096059+0000","last_change":"2024-02-23T00:34:23.049466+0000","last_active":"2024-02-23T00:35:09.096059+0000","last_peered":"2024-02-23T00:35:09.096059+0000","last_clean":"2024-02-23T00:35:09.096059+0000","last_became_active":"2024-02-23T00:31:06.329726+0000","last_became_peered":"2024-02-23T00:31:06.329726+0000","last_unstale":"2024-02-23T00:35:09.096059+0000","last_undegraded":"2024-02-23T00:35:09.096059+0000","last_fullsized":"2024-02-23T00:35:09.096059+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'20","last_scrub_stamp":"2024-02-23T00:34:23.049395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.049395+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.3","version":"248'25","reported_seq":353,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.113984+0000","last_change":"2024-02-23T00:34:41.143312+0000","last_active":"2024-02-23T00:35:09.113984+0000","last_peered":"2024-02-23T00:35:09.113984+0000","last_clean":"2024-02-23T00:35:09.113984+0000","last_became_active":"2024-02-23T00:34:41.104249+0000","last_became_peered":"2024-02-23T00:34:41.104249+0000","last_unstale":"2024-02-23T00:35:09.113984+0000","last_undegraded":"2024-02-23T00:35:09.113984+0000","last_fullsized":"2024-02-23T00:35:09.113984+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'25","last_scrub_stamp":"2024-02-23T00:34:15.396956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.396956+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":24,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.4","version":"340'40","reported_seq":332,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114680+0000","last_change":"2024-02-23T00:34:19.398149+0000","last_active":"2024-02-23T00:35:09.114680+0000","last_peered":"2024-02-23T00:35:09.114680+0000","last_clean":"2024-02-23T00:35:09.114680+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:35:09.114680+0000","last_undegraded":"2024-02-23T00:35:09.114680+0000","last_fullsized":"2024-02-23T00:35:09.114680+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'39","last_scrub_stamp":"2024-02-23T00:34:19.398065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.398065+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.5","version":"234'124","reported_seq":409,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090230+0000","last_change":"2024-02-23T00:34:19.695532+0000","last_active":"2024-02-23T00:35:09.090230+0000","last_peered":"2024-02-23T00:35:09.090230+0000","last_clean":"2024-02-23T00:35:09.090230+0000","last_became_active":"2024-02-23T00:31:06.258422+0000","last_became_peered":"2024-02-23T00:31:06.258422+0000","last_unstale":"2024-02-23T00:35:09.090230+0000","last_undegraded":"2024-02-23T00:35:09.090230+0000","last_fullsized":"2024-02-23T00:35:09.090230+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'124","last_scrub_stamp":"2024-02-23T00:34:19.695449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.695449+0000","log_size":124,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.6","version":"340'52","reported_seq":351,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093803+0000","last_change":"2024-02-23T00:34:31.733213+0000","last_active":"2024-02-23T00:35:09.093803+0000","last_peered":"2024-02-23T00:35:09.093803+0000","last_clean":"2024-02-23T00:35:09.093803+0000","last_became_active":"2024-02-23T00:31:06.313441+0000","last_became_peered":"2024-02-23T00:31:06.313441+0000","last_unstale":"2024-02-23T00:35:09.093803+0000","last_undegraded":"2024-02-23T00:35:09.093803+0000","last_fullsized":"2024-02-23T00:35:09.093803+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:34:31.733148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.733148+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.7","version":"340'94","reported_seq":379,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096256+0000","last_change":"2024-02-23T00:34:18.124117+0000","last_active":"2024-02-23T00:35:09.096256+0000","last_peered":"2024-02-23T00:35:09.096256+0000","last_clean":"2024-02-23T00:35:09.096256+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:35:09.096256+0000","last_undegraded":"2024-02-23T00:35:09.096256+0000","last_fullsized":"2024-02-23T00:35:09.096256+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'92","last_scrub_stamp":"2024-02-23T00:34:18.124049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:18.124049+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.8","version":"340'71","reported_seq":287,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.094891+0000","last_change":"2024-02-23T00:34:42.849848+0000","last_active":"2024-02-23T00:35:09.094891+0000","last_peered":"2024-02-23T00:35:09.094891+0000","last_clean":"2024-02-23T00:35:09.094891+0000","last_became_active":"2024-02-23T00:34:41.115222+0000","last_became_peered":"2024-02-23T00:34:41.115222+0000","last_unstale":"2024-02-23T00:35:09.094891+0000","last_undegraded":"2024-02-23T00:35:09.094891+0000","last_fullsized":"2024-02-23T00:35:09.094891+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.9","version":"208'17","reported_seq":315,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095568+0000","last_change":"2024-02-23T00:34:17.107641+0000","last_active":"2024-02-23T00:35:09.095568+0000","last_peered":"2024-02-23T00:35:09.095568+0000","last_clean":"2024-02-23T00:35:09.095568+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:35:09.095568+0000","last_undegraded":"2024-02-23T00:35:09.095568+0000","last_fullsized":"2024-02-23T00:35:09.095568+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'17","last_scrub_stamp":"2024-02-23T00:34:17.107575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.107575+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.a","version":"248'22","reported_seq":323,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.113942+0000","last_change":"2024-02-23T00:33:48.236273+0000","last_active":"2024-02-23T00:35:09.113942+0000","last_peered":"2024-02-23T00:35:09.113942+0000","last_clean":"2024-02-23T00:35:09.113942+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:35:09.113942+0000","last_undegraded":"2024-02-23T00:35:09.113942+0000","last_fullsized":"2024-02-23T00:35:09.113942+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'22","last_scrub_stamp":"2024-02-23T00:33:34.362545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.362545+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.b","version":"340'35","reported_seq":361,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109365+0000","last_change":"2024-02-23T00:34:41.129017+0000","last_active":"2024-02-23T00:35:09.109365+0000","last_peered":"2024-02-23T00:35:09.109365+0000","last_clean":"2024-02-23T00:35:09.109365+0000","last_became_active":"2024-02-23T00:34:41.103580+0000","last_became_peered":"2024-02-23T00:34:41.103580+0000","last_unstale":"2024-02-23T00:35:09.109365+0000","last_undegraded":"2024-02-23T00:35:09.109365+0000","last_fullsized":"2024-02-23T00:35:09.109365+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-23T00:33:37.391203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.391203+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.c","version":"340'70","reported_seq":357,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092766+0000","last_change":"2024-02-23T00:33:48.290340+0000","last_active":"2024-02-23T00:35:09.092766+0000","last_peered":"2024-02-23T00:35:09.092766+0000","last_clean":"2024-02-23T00:35:09.092766+0000","last_became_active":"2024-02-23T00:31:06.233844+0000","last_became_peered":"2024-02-23T00:31:06.233844+0000","last_unstale":"2024-02-23T00:35:09.092766+0000","last_undegraded":"2024-02-23T00:35:09.092766+0000","last_fullsized":"2024-02-23T00:35:09.092766+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'68","last_scrub_stamp":"2024-02-23T00:33:47.391824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.391824+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.d","version":"340'35","reported_seq":358,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114677+0000","last_change":"2024-02-23T00:34:41.146364+0000","last_active":"2024-02-23T00:35:09.114677+0000","last_peered":"2024-02-23T00:35:09.114677+0000","last_clean":"2024-02-23T00:35:09.114677+0000","last_became_active":"2024-02-23T00:34:41.109682+0000","last_became_peered":"2024-02-23T00:34:41.109682+0000","last_unstale":"2024-02-23T00:35:09.114677+0000","last_undegraded":"2024-02-23T00:35:09.114677+0000","last_fullsized":"2024-02-23T00:35:09.114677+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-23T00:33:45.265466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.265466+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.e","version":"340'75","reported_seq":424,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095922+0000","last_change":"2024-02-23T00:34:44.806474+0000","last_active":"2024-02-23T00:35:09.095922+0000","last_peered":"2024-02-23T00:35:09.095922+0000","last_clean":"2024-02-23T00:35:09.095922+0000","last_became_active":"2024-02-23T00:34:41.108381+0000","last_became_peered":"2024-02-23T00:34:41.108381+0000","last_unstale":"2024-02-23T00:35:09.095922+0000","last_undegraded":"2024-02-23T00:35:09.095922+0000","last_fullsized":"2024-02-23T00:35:09.095922+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'74","last_scrub_stamp":"2024-02-23T00:34:15.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.172848+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000065,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":3930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":4000057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.f","version":"219'69","reported_seq":280,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096724+0000","last_change":"2024-02-23T00:34:42.928816+0000","last_active":"2024-02-23T00:35:09.096724+0000","last_peered":"2024-02-23T00:35:09.096724+0000","last_clean":"2024-02-23T00:35:09.096724+0000","last_became_active":"2024-02-23T00:34:41.101718+0000","last_became_peered":"2024-02-23T00:34:41.101718+0000","last_unstale":"2024-02-23T00:35:09.096724+0000","last_undegraded":"2024-02-23T00:35:09.096724+0000","last_fullsized":"2024-02-23T00:35:09.096724+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:31:05.147166+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.10","version":"218'34","reported_seq":328,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.094906+0000","last_change":"2024-02-23T00:35:01.791913+0000","last_active":"2024-02-23T00:35:09.094906+0000","last_peered":"2024-02-23T00:35:09.094906+0000","last_clean":"2024-02-23T00:35:09.094906+0000","last_became_active":"2024-02-23T00:31:06.342164+0000","last_became_peered":"2024-02-23T00:31:06.342164+0000","last_unstale":"2024-02-23T00:35:09.094906+0000","last_undegraded":"2024-02-23T00:35:09.094906+0000","last_fullsized":"2024-02-23T00:35:09.094906+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'34","last_scrub_stamp":"2024-02-23T00:35:01.791840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.791840+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.11","version":"354'47","reported_seq":338,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089050+0000","last_change":"2024-02-23T00:34:13.745456+0000","last_active":"2024-02-23T00:35:09.089050+0000","last_peered":"2024-02-23T00:35:09.089050+0000","last_clean":"2024-02-23T00:35:09.089050+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:35:09.089050+0000","last_undegraded":"2024-02-23T00:35:09.089050+0000","last_fullsized":"2024-02-23T00:35:09.089050+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'45","last_scrub_stamp":"2024-02-23T00:34:13.745366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.745366+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.12","version":"204'68","reported_seq":350,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092611+0000","last_change":"2024-02-23T00:33:48.289167+0000","last_active":"2024-02-23T00:35:09.092611+0000","last_peered":"2024-02-23T00:35:09.092611+0000","last_clean":"2024-02-23T00:35:09.092611+0000","last_became_active":"2024-02-23T00:31:06.281619+0000","last_became_peered":"2024-02-23T00:31:06.281619+0000","last_unstale":"2024-02-23T00:35:09.092611+0000","last_undegraded":"2024-02-23T00:35:09.092611+0000","last_fullsized":"2024-02-23T00:35:09.092611+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'68","last_scrub_stamp":"2024-02-23T00:33:46.399784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.399784+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.13","version":"248'61","reported_seq":351,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109251+0000","last_change":"2024-02-23T00:33:48.299177+0000","last_active":"2024-02-23T00:35:09.109251+0000","last_peered":"2024-02-23T00:35:09.109251+0000","last_clean":"2024-02-23T00:35:09.109251+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:35:09.109251+0000","last_undegraded":"2024-02-23T00:35:09.109251+0000","last_fullsized":"2024-02-23T00:35:09.109251+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'61","last_scrub_stamp":"2024-02-23T00:33:41.498235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:41.498235+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.14","version":"208'85","reported_seq":368,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092656+0000","last_change":"2024-02-23T00:33:48.357450+0000","last_active":"2024-02-23T00:35:09.092656+0000","last_peered":"2024-02-23T00:35:09.092656+0000","last_clean":"2024-02-23T00:35:09.092656+0000","last_became_active":"2024-02-23T00:31:06.234521+0000","last_became_peered":"2024-02-23T00:31:06.234521+0000","last_unstale":"2024-02-23T00:35:09.092656+0000","last_undegraded":"2024-02-23T00:35:09.092656+0000","last_fullsized":"2024-02-23T00:35:09.092656+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'85","last_scrub_stamp":"2024-02-23T00:33:48.357371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:48.357371+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.15","version":"340'70","reported_seq":362,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090065+0000","last_change":"2024-02-23T00:34:21.681266+0000","last_active":"2024-02-23T00:35:09.090065+0000","last_peered":"2024-02-23T00:35:09.090065+0000","last_clean":"2024-02-23T00:35:09.090065+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:35:09.090065+0000","last_undegraded":"2024-02-23T00:35:09.090065+0000","last_fullsized":"2024-02-23T00:35:09.090065+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'68","last_scrub_stamp":"2024-02-23T00:34:21.681187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.681187+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.16","version":"204'51","reported_seq":346,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093752+0000","last_change":"2024-02-23T00:33:48.042926+0000","last_active":"2024-02-23T00:35:09.093752+0000","last_peered":"2024-02-23T00:35:09.093752+0000","last_clean":"2024-02-23T00:35:09.093752+0000","last_became_active":"2024-02-23T00:31:06.367069+0000","last_became_peered":"2024-02-23T00:31:06.367069+0000","last_unstale":"2024-02-23T00:35:09.093752+0000","last_undegraded":"2024-02-23T00:35:09.093752+0000","last_fullsized":"2024-02-23T00:35:09.093752+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:32:57.837196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:57.837196+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.17","version":"340'95","reported_seq":384,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.115008+0000","last_change":"2024-02-23T00:33:50.240380+0000","last_active":"2024-02-23T00:35:09.115008+0000","last_peered":"2024-02-23T00:35:09.115008+0000","last_clean":"2024-02-23T00:35:09.115008+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:35:09.115008+0000","last_undegraded":"2024-02-23T00:35:09.115008+0000","last_fullsized":"2024-02-23T00:35:09.115008+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'86","last_scrub_stamp":"2024-02-23T00:33:35.383425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:35.383425+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":73,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.18","version":"211'57","reported_seq":349,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.108932+0000","last_change":"2024-02-23T00:33:48.297931+0000","last_active":"2024-02-23T00:35:09.108932+0000","last_peered":"2024-02-23T00:35:09.108932+0000","last_clean":"2024-02-23T00:35:09.108932+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:35:09.108932+0000","last_undegraded":"2024-02-23T00:35:09.108932+0000","last_fullsized":"2024-02-23T00:35:09.108932+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'57","last_scrub_stamp":"2024-02-23T00:33:36.369153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.369153+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.19","version":"340'60","reported_seq":358,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.088938+0000","last_change":"2024-02-23T00:33:48.036867+0000","last_active":"2024-02-23T00:35:09.088938+0000","last_peered":"2024-02-23T00:35:09.088938+0000","last_clean":"2024-02-23T00:35:09.088938+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:35:09.088938+0000","last_undegraded":"2024-02-23T00:35:09.088938+0000","last_fullsized":"2024-02-23T00:35:09.088938+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'54","last_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_deep_scrub":"271'54","last_deep_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.780899+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.1a","version":"202'68","reported_seq":356,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093659+0000","last_change":"2024-02-23T00:33:48.042765+0000","last_active":"2024-02-23T00:35:09.093659+0000","last_peered":"2024-02-23T00:35:09.093659+0000","last_clean":"2024-02-23T00:35:09.093659+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:35:09.093659+0000","last_undegraded":"2024-02-23T00:35:09.093659+0000","last_fullsized":"2024-02-23T00:35:09.093659+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'68","last_scrub_stamp":"2024-02-23T00:32:58.808102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.808102+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.1b","version":"236'51","reported_seq":342,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.113939+0000","last_change":"2024-02-23T00:33:48.237253+0000","last_active":"2024-02-23T00:35:09.113939+0000","last_peered":"2024-02-23T00:35:09.113939+0000","last_clean":"2024-02-23T00:35:09.113939+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:35:09.113939+0000","last_undegraded":"2024-02-23T00:35:09.113939+0000","last_fullsized":"2024-02-23T00:35:09.113939+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-02-23T00:33:46.274400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.274400+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1c","version":"216'34","reported_seq":358,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096785+0000","last_change":"2024-02-23T00:34:57.859118+0000","last_active":"2024-02-23T00:35:09.096785+0000","last_peered":"2024-02-23T00:35:09.096785+0000","last_clean":"2024-02-23T00:35:09.096785+0000","last_became_active":"2024-02-23T00:34:41.117846+0000","last_became_peered":"2024-02-23T00:34:41.117846+0000","last_unstale":"2024-02-23T00:35:09.096785+0000","last_undegraded":"2024-02-23T00:35:09.096785+0000","last_fullsized":"2024-02-23T00:35:09.096785+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'34","last_scrub_stamp":"2024-02-23T00:34:57.859047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.859047+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1d","version":"340'13","reported_seq":51,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092805+0000","last_change":"2024-02-23T00:34:42.070971+0000","last_active":"2024-02-23T00:35:09.092805+0000","last_peered":"2024-02-23T00:35:09.092805+0000","last_clean":"2024-02-23T00:35:09.092805+0000","last_became_active":"2024-02-23T00:34:42.058724+0000","last_became_peered":"2024-02-23T00:34:42.058724+0000","last_unstale":"2024-02-23T00:35:09.092805+0000","last_undegraded":"2024-02-23T00:35:09.092805+0000","last_fullsized":"2024-02-23T00:35:09.092805+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-02-23T00:34:24.997695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.997695+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.1e","version":"340'52","reported_seq":341,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114529+0000","last_change":"2024-02-23T00:33:49.219707+0000","last_active":"2024-02-23T00:35:09.114529+0000","last_peered":"2024-02-23T00:35:09.114529+0000","last_clean":"2024-02-23T00:35:09.114529+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:35:09.114529+0000","last_undegraded":"2024-02-23T00:35:09.114529+0000","last_fullsized":"2024-02-23T00:35:09.114529+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'51","last_scrub_stamp":"2024-02-23T00:33:49.219607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.219607+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1f","version":"228'68","reported_seq":360,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114953+0000","last_change":"2024-02-23T00:33:48.238796+0000","last_active":"2024-02-23T00:35:09.114953+0000","last_peered":"2024-02-23T00:35:09.114953+0000","last_clean":"2024-02-23T00:35:09.114953+0000","last_became_active":"2024-02-23T00:31:06.355623+0000","last_became_peered":"2024-02-23T00:31:06.355623+0000","last_unstale":"2024-02-23T00:35:09.114953+0000","last_undegraded":"2024-02-23T00:35:09.114953+0000","last_fullsized":"2024-02-23T00:35:09.114953+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-23T00:33:39.351573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.351573+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"62'8","reported_seq":430,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.129979+0000","last_change":"2024-02-23T00:34:27.673554+0000","last_active":"2024-02-23T00:35:10.129979+0000","last_peered":"2024-02-23T00:35:10.129979+0000","last_clean":"2024-02-23T00:35:10.129979+0000","last_became_active":"2024-02-23T00:27:21.027480+0000","last_became_peered":"2024-02-23T00:27:21.027480+0000","last_unstale":"2024-02-23T00:35:10.129979+0000","last_undegraded":"2024-02-23T00:35:10.129979+0000","last_fullsized":"2024-02-23T00:35:10.129979+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:27.673481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.673481+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"78'36","reported_seq":387,"reported_epoch":359,"state":"peering","last_fresh":"2024-02-23T00:35:10.132601+0000","last_change":"2024-02-23T00:35:10.133605+0000","last_active":"2024-02-23T00:35:10.132601+0000","last_peered":"2024-02-23T00:35:09.113759+0000","last_clean":"2024-02-23T00:35:09.113759+0000","last_became_active":"2024-02-23T00:27:19.015200+0000","last_became_peered":"2024-02-23T00:27:19.015200+0000","last_unstale":"2024-02-23T00:35:10.132601+0000","last_undegraded":"2024-02-23T00:35:10.132601+0000","last_fullsized":"2024-02-23T00:35:10.132601+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:32:47.376691+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.376691+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":39,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":347,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093879+0000","last_change":"2024-02-23T00:35:05.018567+0000","last_active":"2024-02-23T00:35:09.093879+0000","last_peered":"2024-02-23T00:35:09.093879+0000","last_clean":"2024-02-23T00:35:09.093879+0000","last_became_active":"2024-02-23T00:34:41.106070+0000","last_became_peered":"2024-02-23T00:34:41.106070+0000","last_unstale":"2024-02-23T00:35:09.093879+0000","last_undegraded":"2024-02-23T00:35:09.093879+0000","last_fullsized":"2024-02-23T00:35:09.093879+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:35:05.018499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":318,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114098+0000","last_change":"2024-02-23T00:34:21.442023+0000","last_active":"2024-02-23T00:35:09.114098+0000","last_peered":"2024-02-23T00:35:09.114098+0000","last_clean":"2024-02-23T00:35:09.114098+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:35:09.114098+0000","last_undegraded":"2024-02-23T00:35:09.114098+0000","last_fullsized":"2024-02-23T00:35:09.114098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:21.441958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.441958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":281,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096479+0000","last_change":"2024-02-23T00:34:42.061455+0000","last_active":"2024-02-23T00:35:09.096479+0000","last_peered":"2024-02-23T00:35:09.096479+0000","last_clean":"2024-02-23T00:35:09.096479+0000","last_became_active":"2024-02-23T00:34:42.059196+0000","last_became_peered":"2024-02-23T00:34:42.059196+0000","last_unstale":"2024-02-23T00:35:09.096479+0000","last_undegraded":"2024-02-23T00:35:09.096479+0000","last_fullsized":"2024-02-23T00:35:09.096479+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":473,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.132544+0000","last_change":"2024-02-23T00:34:41.145462+0000","last_active":"2024-02-23T00:35:10.132544+0000","last_peered":"2024-02-23T00:35:10.132544+0000","last_clean":"2024-02-23T00:35:10.132544+0000","last_became_active":"2024-02-23T00:34:41.114058+0000","last_became_peered":"2024-02-23T00:34:41.114058+0000","last_unstale":"2024-02-23T00:35:10.132544+0000","last_undegraded":"2024-02-23T00:35:10.132544+0000","last_fullsized":"2024-02-23T00:35:10.132544+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:37.440614+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.440614+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":152,"num_read_kb":152,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"78'14","reported_seq":378,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093118+0000","last_change":"2024-02-23T00:34:55.073114+0000","last_active":"2024-02-23T00:35:09.093118+0000","last_peered":"2024-02-23T00:35:09.093118+0000","last_clean":"2024-02-23T00:35:09.093118+0000","last_became_active":"2024-02-23T00:34:55.072740+0000","last_became_peered":"2024-02-23T00:34:55.072740+0000","last_unstale":"2024-02-23T00:35:09.093118+0000","last_undegraded":"2024-02-23T00:35:09.093118+0000","last_fullsized":"2024-02-23T00:35:09.093118+0000","mapping_epoch":56,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":28,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":334,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095786+0000","last_change":"2024-02-23T00:34:41.100552+0000","last_active":"2024-02-23T00:35:09.095786+0000","last_peered":"2024-02-23T00:35:09.095786+0000","last_clean":"2024-02-23T00:35:09.095786+0000","last_became_active":"2024-02-23T00:34:41.100003+0000","last_became_peered":"2024-02-23T00:34:41.100003+0000","last_unstale":"2024-02-23T00:35:09.095786+0000","last_undegraded":"2024-02-23T00:35:09.095786+0000","last_fullsized":"2024-02-23T00:35:09.095786+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.328237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.328237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":338,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093807+0000","last_change":"2024-02-23T00:34:41.096223+0000","last_active":"2024-02-23T00:35:09.093807+0000","last_peered":"2024-02-23T00:35:09.093807+0000","last_clean":"2024-02-23T00:35:09.093807+0000","last_became_active":"2024-02-23T00:34:41.095851+0000","last_became_peered":"2024-02-23T00:34:41.095851+0000","last_unstale":"2024-02-23T00:35:09.093807+0000","last_undegraded":"2024-02-23T00:35:09.093807+0000","last_fullsized":"2024-02-23T00:35:09.093807+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.679076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":325,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091530+0000","last_change":"2024-02-23T00:33:47.610285+0000","last_active":"2024-02-23T00:35:09.091530+0000","last_peered":"2024-02-23T00:35:09.091530+0000","last_clean":"2024-02-23T00:35:09.091530+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:35:09.091530+0000","last_undegraded":"2024-02-23T00:35:09.091530+0000","last_fullsized":"2024-02-23T00:35:09.091530+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":273,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092743+0000","last_change":"2024-02-23T00:33:44.400851+0000","last_active":"2024-02-23T00:35:09.092743+0000","last_peered":"2024-02-23T00:35:09.092743+0000","last_clean":"2024-02-23T00:35:09.092743+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:35:09.092743+0000","last_undegraded":"2024-02-23T00:35:09.092743+0000","last_fullsized":"2024-02-23T00:35:09.092743+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.400781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.400781+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":393,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093054+0000","last_change":"2024-02-23T00:34:59.084974+0000","last_active":"2024-02-23T00:35:09.093054+0000","last_peered":"2024-02-23T00:35:09.093054+0000","last_clean":"2024-02-23T00:35:09.093054+0000","last_became_active":"2024-02-23T00:34:59.084636+0000","last_became_peered":"2024-02-23T00:34:59.084636+0000","last_unstale":"2024-02-23T00:35:09.093054+0000","last_undegraded":"2024-02-23T00:35:09.093054+0000","last_fullsized":"2024-02-23T00:35:09.093054+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:34:16.310110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:16.310110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":83,"num_read_kb":63,"num_write":48,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":333,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092873+0000","last_change":"2024-02-23T00:32:59.849579+0000","last_active":"2024-02-23T00:35:09.092873+0000","last_peered":"2024-02-23T00:35:09.092873+0000","last_clean":"2024-02-23T00:35:09.092873+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:35:09.092873+0000","last_undegraded":"2024-02-23T00:35:09.092873+0000","last_fullsized":"2024-02-23T00:35:09.092873+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.849503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.849503+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":329,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092044+0000","last_change":"2024-02-23T00:34:28.745964+0000","last_active":"2024-02-23T00:35:09.092044+0000","last_peered":"2024-02-23T00:35:09.092044+0000","last_clean":"2024-02-23T00:35:09.092044+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:35:09.092044+0000","last_undegraded":"2024-02-23T00:35:09.092044+0000","last_fullsized":"2024-02-23T00:35:09.092044+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.745865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.745865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":334,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109101+0000","last_change":"2024-02-23T00:35:04.673936+0000","last_active":"2024-02-23T00:35:09.109101+0000","last_peered":"2024-02-23T00:35:09.109101+0000","last_clean":"2024-02-23T00:35:09.109101+0000","last_became_active":"2024-02-23T00:34:41.105277+0000","last_became_peered":"2024-02-23T00:34:41.105277+0000","last_unstale":"2024-02-23T00:35:09.109101+0000","last_undegraded":"2024-02-23T00:35:09.109101+0000","last_fullsized":"2024-02-23T00:35:09.109101+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.673837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.673837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":273,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091475+0000","last_change":"2024-02-23T00:32:45.536091+0000","last_active":"2024-02-23T00:35:09.091475+0000","last_peered":"2024-02-23T00:35:09.091475+0000","last_clean":"2024-02-23T00:35:09.091475+0000","last_became_active":"2024-02-23T00:30:48.673452+0000","last_became_peered":"2024-02-23T00:30:48.673452+0000","last_unstale":"2024-02-23T00:35:09.091475+0000","last_undegraded":"2024-02-23T00:35:09.091475+0000","last_fullsized":"2024-02-23T00:35:09.091475+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.535992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.535992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":340,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093349+0000","last_change":"2024-02-23T00:34:29.233679+0000","last_active":"2024-02-23T00:35:09.093349+0000","last_peered":"2024-02-23T00:35:09.093349+0000","last_clean":"2024-02-23T00:35:09.093349+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:35:09.093349+0000","last_undegraded":"2024-02-23T00:35:09.093349+0000","last_fullsized":"2024-02-23T00:35:09.093349+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:29.233574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:29.233574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'24","reported_seq":372,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090804+0000","last_change":"2024-02-23T00:34:47.063115+0000","last_active":"2024-02-23T00:35:09.090804+0000","last_peered":"2024-02-23T00:35:09.090804+0000","last_clean":"2024-02-23T00:35:09.090804+0000","last_became_active":"2024-02-23T00:34:47.062796+0000","last_became_peered":"2024-02-23T00:34:47.062796+0000","last_unstale":"2024-02-23T00:35:09.090804+0000","last_undegraded":"2024-02-23T00:35:09.090804+0000","last_fullsized":"2024-02-23T00:35:09.090804+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'17","last_scrub_stamp":"2024-02-23T00:33:36.603732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.603732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":326,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109473+0000","last_change":"2024-02-23T00:32:56.692114+0000","last_active":"2024-02-23T00:35:09.109473+0000","last_peered":"2024-02-23T00:35:09.109473+0000","last_clean":"2024-02-23T00:35:09.109473+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:35:09.109473+0000","last_undegraded":"2024-02-23T00:35:09.109473+0000","last_fullsized":"2024-02-23T00:35:09.109473+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.692037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.692037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":316,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096952+0000","last_change":"2024-02-23T00:32:09.110225+0000","last_active":"2024-02-23T00:35:09.096952+0000","last_peered":"2024-02-23T00:35:09.096952+0000","last_clean":"2024-02-23T00:35:09.096952+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:35:09.096952+0000","last_undegraded":"2024-02-23T00:35:09.096952+0000","last_fullsized":"2024-02-23T00:35:09.096952+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.110138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":336,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092467+0000","last_change":"2024-02-23T00:34:41.091099+0000","last_active":"2024-02-23T00:35:09.092467+0000","last_peered":"2024-02-23T00:35:09.092467+0000","last_clean":"2024-02-23T00:35:09.092467+0000","last_became_active":"2024-02-23T00:34:41.090777+0000","last_became_peered":"2024-02-23T00:34:41.090777+0000","last_unstale":"2024-02-23T00:35:09.092467+0000","last_undegraded":"2024-02-23T00:35:09.092467+0000","last_fullsized":"2024-02-23T00:35:09.092467+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.791137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.791137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":283,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110609+0000","last_change":"2024-02-23T00:34:41.114908+0000","last_active":"2024-02-23T00:35:09.110609+0000","last_peered":"2024-02-23T00:35:09.110609+0000","last_clean":"2024-02-23T00:35:09.110609+0000","last_became_active":"2024-02-23T00:34:41.114555+0000","last_became_peered":"2024-02-23T00:34:41.114555+0000","last_unstale":"2024-02-23T00:35:09.110609+0000","last_undegraded":"2024-02-23T00:35:09.110609+0000","last_fullsized":"2024-02-23T00:35:09.110609+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.671219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.671219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":146,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093409+0000","last_change":"2024-02-23T00:34:41.109231+0000","last_active":"2024-02-23T00:35:09.093409+0000","last_peered":"2024-02-23T00:35:09.093409+0000","last_clean":"2024-02-23T00:35:09.093409+0000","last_became_active":"2024-02-23T00:34:41.108822+0000","last_became_peered":"2024-02-23T00:34:41.108822+0000","last_unstale":"2024-02-23T00:35:09.093409+0000","last_undegraded":"2024-02-23T00:35:09.093409+0000","last_fullsized":"2024-02-23T00:35:09.093409+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.985150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.985150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"62'15","reported_seq":380,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090088+0000","last_change":"2024-02-23T00:34:51.102561+0000","last_active":"2024-02-23T00:35:09.090088+0000","last_peered":"2024-02-23T00:35:09.090088+0000","last_clean":"2024-02-23T00:35:09.090088+0000","last_became_active":"2024-02-23T00:34:51.102216+0000","last_became_peered":"2024-02-23T00:34:51.102216+0000","last_unstale":"2024-02-23T00:35:09.090088+0000","last_undegraded":"2024-02-23T00:35:09.090088+0000","last_fullsized":"2024-02-23T00:35:09.090088+0000","mapping_epoch":329,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:34:22.638641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.638641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":26,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":347,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096247+0000","last_change":"2024-02-23T00:34:41.116172+0000","last_active":"2024-02-23T00:35:09.096247+0000","last_peered":"2024-02-23T00:35:09.096247+0000","last_clean":"2024-02-23T00:35:09.096247+0000","last_became_active":"2024-02-23T00:34:41.115434+0000","last_became_peered":"2024-02-23T00:34:41.115434+0000","last_unstale":"2024-02-23T00:35:09.096247+0000","last_undegraded":"2024-02-23T00:35:09.096247+0000","last_fullsized":"2024-02-23T00:35:09.096247+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:31:35.937969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":432,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.098430+0000","last_change":"2024-02-23T00:34:57.950514+0000","last_active":"2024-02-23T00:35:10.098430+0000","last_peered":"2024-02-23T00:35:10.098430+0000","last_clean":"2024-02-23T00:35:10.098430+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:35:10.098430+0000","last_undegraded":"2024-02-23T00:35:10.098430+0000","last_fullsized":"2024-02-23T00:35:10.098430+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:57.950450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.950450+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":322,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090458+0000","last_change":"2024-02-23T00:33:27.495416+0000","last_active":"2024-02-23T00:35:09.090458+0000","last_peered":"2024-02-23T00:35:09.090458+0000","last_clean":"2024-02-23T00:35:09.090458+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:35:09.090458+0000","last_undegraded":"2024-02-23T00:35:09.090458+0000","last_fullsized":"2024-02-23T00:35:09.090458+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:27.495308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:27.495308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":275,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090480+0000","last_change":"2024-02-23T00:33:32.583582+0000","last_active":"2024-02-23T00:35:09.090480+0000","last_peered":"2024-02-23T00:35:09.090480+0000","last_clean":"2024-02-23T00:35:09.090480+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:35:09.090480+0000","last_undegraded":"2024-02-23T00:35:09.090480+0000","last_fullsized":"2024-02-23T00:35:09.090480+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:32.583509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:32.583509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":395,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.094756+0000","last_change":"2024-02-23T00:34:41.104353+0000","last_active":"2024-02-23T00:35:09.094756+0000","last_peered":"2024-02-23T00:35:09.094756+0000","last_clean":"2024-02-23T00:35:09.094756+0000","last_became_active":"2024-02-23T00:34:41.103369+0000","last_became_peered":"2024-02-23T00:34:41.103369+0000","last_unstale":"2024-02-23T00:35:09.094756+0000","last_undegraded":"2024-02-23T00:35:09.094756+0000","last_fullsized":"2024-02-23T00:35:09.094756+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:58.975334+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.975334+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":59,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"87'22","reported_seq":385,"reported_epoch":358,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:35:09.096839+0000","last_change":"2024-02-23T00:34:43.461182+0000","last_active":"2024-02-23T00:35:09.096839+0000","last_peered":"2024-02-23T00:35:09.096839+0000","last_clean":"2024-02-23T00:34:39.075047+0000","last_became_active":"2024-02-23T00:34:42.068264+0000","last_became_peered":"2024-02-23T00:34:42.068264+0000","last_unstale":"2024-02-23T00:35:09.096839+0000","last_undegraded":"2024-02-23T00:35:09.096839+0000","last_fullsized":"2024-02-23T00:35:09.096839+0000","mapping_epoch":330,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:32:40.327519+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.327519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":56,"num_read_kb":39,"num_write":34,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,3],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":336,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095803+0000","last_change":"2024-02-23T00:35:06.658103+0000","last_active":"2024-02-23T00:35:09.095803+0000","last_peered":"2024-02-23T00:35:09.095803+0000","last_clean":"2024-02-23T00:35:09.095803+0000","last_became_active":"2024-02-23T00:34:41.107670+0000","last_became_peered":"2024-02-23T00:34:41.107670+0000","last_unstale":"2024-02-23T00:35:09.095803+0000","last_undegraded":"2024-02-23T00:35:09.095803+0000","last_fullsized":"2024-02-23T00:35:09.095803+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.657998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":432,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.102298+0000","last_change":"2024-02-23T00:34:59.630249+0000","last_active":"2024-02-23T00:35:10.102298+0000","last_peered":"2024-02-23T00:35:10.102298+0000","last_clean":"2024-02-23T00:35:10.102298+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:35:10.102298+0000","last_undegraded":"2024-02-23T00:35:10.102298+0000","last_fullsized":"2024-02-23T00:35:10.102298+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:59.630171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:59.630171+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":319,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114954+0000","last_change":"2024-02-23T00:34:17.437153+0000","last_active":"2024-02-23T00:35:09.114954+0000","last_peered":"2024-02-23T00:35:09.114954+0000","last_clean":"2024-02-23T00:35:09.114954+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:35:09.114954+0000","last_undegraded":"2024-02-23T00:35:09.114954+0000","last_fullsized":"2024-02-23T00:35:09.114954+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:34:17.437083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.437083+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":272,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095461+0000","last_change":"2024-02-23T00:33:39.140496+0000","last_active":"2024-02-23T00:35:09.095461+0000","last_peered":"2024-02-23T00:35:09.095461+0000","last_clean":"2024-02-23T00:35:09.095461+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:35:09.095461+0000","last_undegraded":"2024-02-23T00:35:09.095461+0000","last_fullsized":"2024-02-23T00:35:09.095461+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:39.140434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.140434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":342,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.088904+0000","last_change":"2024-02-23T00:35:01.614406+0000","last_active":"2024-02-23T00:35:09.088904+0000","last_peered":"2024-02-23T00:35:09.088904+0000","last_clean":"2024-02-23T00:35:09.088904+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:35:09.088904+0000","last_undegraded":"2024-02-23T00:35:09.088904+0000","last_fullsized":"2024-02-23T00:35:09.088904+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:01.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.614340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"92'22","reported_seq":384,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090250+0000","last_change":"2024-02-23T00:34:43.059167+0000","last_active":"2024-02-23T00:35:09.090250+0000","last_peered":"2024-02-23T00:35:09.090250+0000","last_clean":"2024-02-23T00:35:09.090250+0000","last_became_active":"2024-02-23T00:34:43.058886+0000","last_became_peered":"2024-02-23T00:34:43.058886+0000","last_unstale":"2024-02-23T00:35:09.090250+0000","last_undegraded":"2024-02-23T00:35:09.090250+0000","last_fullsized":"2024-02-23T00:35:09.090250+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'23","last_scrub_stamp":"2024-02-23T00:32:46.779979+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.779979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":338,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109352+0000","last_change":"2024-02-23T00:33:50.601581+0000","last_active":"2024-02-23T00:35:09.109352+0000","last_peered":"2024-02-23T00:35:09.109352+0000","last_clean":"2024-02-23T00:35:09.109352+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:35:09.109352+0000","last_undegraded":"2024-02-23T00:35:09.109352+0000","last_fullsized":"2024-02-23T00:35:09.109352+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":320,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.115255+0000","last_change":"2024-02-23T00:34:22.475022+0000","last_active":"2024-02-23T00:35:09.115255+0000","last_peered":"2024-02-23T00:35:09.115255+0000","last_clean":"2024-02-23T00:35:09.115255+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:35:09.115255+0000","last_undegraded":"2024-02-23T00:35:09.115255+0000","last_fullsized":"2024-02-23T00:35:09.115255+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.474956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.474956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":48,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093125+0000","last_change":"2024-02-23T00:34:56.232523+0000","last_active":"2024-02-23T00:35:09.093125+0000","last_peered":"2024-02-23T00:35:09.093125+0000","last_clean":"2024-02-23T00:35:09.093125+0000","last_became_active":"2024-02-23T00:34:41.116969+0000","last_became_peered":"2024-02-23T00:34:41.116969+0000","last_unstale":"2024-02-23T00:35:09.093125+0000","last_undegraded":"2024-02-23T00:35:09.093125+0000","last_fullsized":"2024-02-23T00:35:09.093125+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.232459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.232459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":281,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.097034+0000","last_change":"2024-02-23T00:34:41.112576+0000","last_active":"2024-02-23T00:35:09.097034+0000","last_peered":"2024-02-23T00:35:09.097034+0000","last_clean":"2024-02-23T00:35:09.097034+0000","last_became_active":"2024-02-23T00:34:41.109539+0000","last_became_peered":"2024-02-23T00:34:41.109539+0000","last_unstale":"2024-02-23T00:35:09.097034+0000","last_undegraded":"2024-02-23T00:35:09.097034+0000","last_fullsized":"2024-02-23T00:35:09.097034+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.387496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":239,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092692+0000","last_change":"2024-02-23T00:32:56.610957+0000","last_active":"2024-02-23T00:35:09.092692+0000","last_peered":"2024-02-23T00:35:09.092692+0000","last_clean":"2024-02-23T00:35:09.092692+0000","last_became_active":"2024-02-23T00:31:29.108747+0000","last_became_peered":"2024-02-23T00:31:29.108747+0000","last_unstale":"2024-02-23T00:35:09.092692+0000","last_undegraded":"2024-02-23T00:35:09.092692+0000","last_fullsized":"2024-02-23T00:35:09.092692+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.610884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.610884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":71,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.132973+0000","last_change":"2024-02-23T00:34:41.139457+0000","last_active":"2024-02-23T00:35:10.132973+0000","last_peered":"2024-02-23T00:35:10.132973+0000","last_clean":"2024-02-23T00:35:10.132973+0000","last_became_active":"2024-02-23T00:34:41.104483+0000","last_became_peered":"2024-02-23T00:34:41.104483+0000","last_unstale":"2024-02-23T00:35:10.132973+0000","last_undegraded":"2024-02-23T00:35:10.132973+0000","last_fullsized":"2024-02-23T00:35:10.132973+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":362,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089268+0000","last_change":"2024-02-23T00:34:31.622831+0000","last_active":"2024-02-23T00:35:09.089268+0000","last_peered":"2024-02-23T00:35:09.089268+0000","last_clean":"2024-02-23T00:35:09.089268+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:35:09.089268+0000","last_undegraded":"2024-02-23T00:35:09.089268+0000","last_fullsized":"2024-02-23T00:35:09.089268+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:34:31.622761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.622761+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":325,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093320+0000","last_change":"2024-02-23T00:32:46.504136+0000","last_active":"2024-02-23T00:35:09.093320+0000","last_peered":"2024-02-23T00:35:09.093320+0000","last_clean":"2024-02-23T00:35:09.093320+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:35:09.093320+0000","last_undegraded":"2024-02-23T00:35:09.093320+0000","last_fullsized":"2024-02-23T00:35:09.093320+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.504045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.504045+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":325,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110033+0000","last_change":"2024-02-23T00:33:45.587063+0000","last_active":"2024-02-23T00:35:09.110033+0000","last_peered":"2024-02-23T00:35:09.110033+0000","last_clean":"2024-02-23T00:35:09.110033+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:35:09.110033+0000","last_undegraded":"2024-02-23T00:35:09.110033+0000","last_fullsized":"2024-02-23T00:35:09.110033+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:45.586975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.586975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":281,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095508+0000","last_change":"2024-02-23T00:34:41.104095+0000","last_active":"2024-02-23T00:35:09.095508+0000","last_peered":"2024-02-23T00:35:09.095508+0000","last_clean":"2024-02-23T00:35:09.095508+0000","last_became_active":"2024-02-23T00:34:41.103792+0000","last_became_peered":"2024-02-23T00:34:41.103792+0000","last_unstale":"2024-02-23T00:35:09.095508+0000","last_undegraded":"2024-02-23T00:35:09.095508+0000","last_fullsized":"2024-02-23T00:35:09.095508+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.019530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.019530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":143,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110434+0000","last_change":"2024-02-23T00:34:41.096978+0000","last_active":"2024-02-23T00:35:09.110434+0000","last_peered":"2024-02-23T00:35:09.110434+0000","last_clean":"2024-02-23T00:35:09.110434+0000","last_became_active":"2024-02-23T00:34:41.096644+0000","last_became_peered":"2024-02-23T00:34:41.096644+0000","last_unstale":"2024-02-23T00:35:09.110434+0000","last_undegraded":"2024-02-23T00:35:09.110434+0000","last_fullsized":"2024-02-23T00:35:09.110434+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:55.991514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:55.991514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":338,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089466+0000","last_change":"2024-02-23T00:35:04.588224+0000","last_active":"2024-02-23T00:35:09.089466+0000","last_peered":"2024-02-23T00:35:09.089466+0000","last_clean":"2024-02-23T00:35:09.089466+0000","last_became_active":"2024-02-23T00:34:41.099704+0000","last_became_peered":"2024-02-23T00:34:41.099704+0000","last_unstale":"2024-02-23T00:35:09.089466+0000","last_undegraded":"2024-02-23T00:35:09.089466+0000","last_fullsized":"2024-02-23T00:35:09.089466+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.588132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.588132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":145,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109512+0000","last_change":"2024-02-23T00:35:00.621227+0000","last_active":"2024-02-23T00:35:09.109512+0000","last_peered":"2024-02-23T00:35:09.109512+0000","last_clean":"2024-02-23T00:35:09.109512+0000","last_became_active":"2024-02-23T00:34:41.110156+0000","last_became_peered":"2024-02-23T00:34:41.110156+0000","last_unstale":"2024-02-23T00:35:09.109512+0000","last_undegraded":"2024-02-23T00:35:09.109512+0000","last_fullsized":"2024-02-23T00:35:09.109512+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:35:00.621157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.621157+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":325,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096019+0000","last_change":"2024-02-23T00:33:34.199948+0000","last_active":"2024-02-23T00:35:09.096019+0000","last_peered":"2024-02-23T00:35:09.096019+0000","last_clean":"2024-02-23T00:35:09.096019+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:35:09.096019+0000","last_undegraded":"2024-02-23T00:35:09.096019+0000","last_fullsized":"2024-02-23T00:35:09.096019+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.199869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.199869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":336,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089523+0000","last_change":"2024-02-23T00:34:58.660219+0000","last_active":"2024-02-23T00:35:09.089523+0000","last_peered":"2024-02-23T00:35:09.089523+0000","last_clean":"2024-02-23T00:35:09.089523+0000","last_became_active":"2024-02-23T00:34:41.094428+0000","last_became_peered":"2024-02-23T00:34:41.094428+0000","last_unstale":"2024-02-23T00:35:09.089523+0000","last_undegraded":"2024-02-23T00:35:09.089523+0000","last_fullsized":"2024-02-23T00:35:09.089523+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:58.660111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:58.660111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":41,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.094029+0000","last_change":"2024-02-23T00:34:41.106716+0000","last_active":"2024-02-23T00:35:09.094029+0000","last_peered":"2024-02-23T00:35:09.094029+0000","last_clean":"2024-02-23T00:35:09.094029+0000","last_became_active":"2024-02-23T00:34:41.105998+0000","last_became_peered":"2024-02-23T00:34:41.105998+0000","last_unstale":"2024-02-23T00:35:09.094029+0000","last_undegraded":"2024-02-23T00:35:09.094029+0000","last_fullsized":"2024-02-23T00:35:09.094029+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:31.674319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.674319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":588,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096738+0000","last_change":"2024-02-23T00:32:43.323030+0000","last_active":"2024-02-23T00:35:09.096738+0000","last_peered":"2024-02-23T00:35:09.096738+0000","last_clean":"2024-02-23T00:35:09.096738+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:35:09.096738+0000","last_undegraded":"2024-02-23T00:35:09.096738+0000","last_fullsized":"2024-02-23T00:35:09.096738+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:43.322963+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.322963+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":327,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093174+0000","last_change":"2024-02-23T00:34:58.218781+0000","last_active":"2024-02-23T00:35:09.093174+0000","last_peered":"2024-02-23T00:35:09.093174+0000","last_clean":"2024-02-23T00:35:09.093174+0000","last_became_active":"2024-02-23T00:27:21.020786+0000","last_became_peered":"2024-02-23T00:27:21.020786+0000","last_unstale":"2024-02-23T00:35:09.093174+0000","last_undegraded":"2024-02-23T00:35:09.093174+0000","last_fullsized":"2024-02-23T00:35:09.093174+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:58.218704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:58.218704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":385,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092953+0000","last_change":"2024-02-23T00:35:01.397052+0000","last_active":"2024-02-23T00:35:09.092953+0000","last_peered":"2024-02-23T00:35:09.092953+0000","last_clean":"2024-02-23T00:35:09.092953+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:35:09.092953+0000","last_undegraded":"2024-02-23T00:35:09.092953+0000","last_fullsized":"2024-02-23T00:35:09.092953+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:35:01.396982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.396982+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":331,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110306+0000","last_change":"2024-02-23T00:34:12.601903+0000","last_active":"2024-02-23T00:35:09.110306+0000","last_peered":"2024-02-23T00:35:09.110306+0000","last_clean":"2024-02-23T00:35:09.110306+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:35:09.110306+0000","last_undegraded":"2024-02-23T00:35:09.110306+0000","last_fullsized":"2024-02-23T00:35:09.110306+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":319,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095858+0000","last_change":"2024-02-23T00:33:38.163226+0000","last_active":"2024-02-23T00:35:09.095858+0000","last_peered":"2024-02-23T00:35:09.095858+0000","last_clean":"2024-02-23T00:35:09.095858+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:35:09.095858+0000","last_undegraded":"2024-02-23T00:35:09.095858+0000","last_fullsized":"2024-02-23T00:35:09.095858+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.163155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.163155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":272,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095905+0000","last_change":"2024-02-23T00:35:00.832892+0000","last_active":"2024-02-23T00:35:09.095905+0000","last_peered":"2024-02-23T00:35:09.095905+0000","last_clean":"2024-02-23T00:35:09.095905+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:35:09.095905+0000","last_undegraded":"2024-02-23T00:35:09.095905+0000","last_fullsized":"2024-02-23T00:35:09.095905+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.832818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.832818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":320,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.115327+0000","last_change":"2024-02-23T00:32:12.386351+0000","last_active":"2024-02-23T00:35:09.115327+0000","last_peered":"2024-02-23T00:35:09.115327+0000","last_clean":"2024-02-23T00:35:09.115327+0000","last_became_active":"2024-02-23T00:27:21.028194+0000","last_became_peered":"2024-02-23T00:27:21.028194+0000","last_unstale":"2024-02-23T00:35:09.115327+0000","last_undegraded":"2024-02-23T00:35:09.115327+0000","last_fullsized":"2024-02-23T00:35:09.115327+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"62'12","reported_seq":352,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092469+0000","last_change":"2024-02-23T00:34:21.576393+0000","last_active":"2024-02-23T00:35:09.092469+0000","last_peered":"2024-02-23T00:35:09.092469+0000","last_clean":"2024-02-23T00:35:09.092469+0000","last_became_active":"2024-02-23T00:27:19.024651+0000","last_became_peered":"2024-02-23T00:27:19.024651+0000","last_unstale":"2024-02-23T00:35:09.092469+0000","last_undegraded":"2024-02-23T00:35:09.092469+0000","last_fullsized":"2024-02-23T00:35:09.092469+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":326,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109222+0000","last_change":"2024-02-23T00:33:42.509570+0000","last_active":"2024-02-23T00:35:09.109222+0000","last_peered":"2024-02-23T00:35:09.109222+0000","last_clean":"2024-02-23T00:35:09.109222+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:35:09.109222+0000","last_undegraded":"2024-02-23T00:35:09.109222+0000","last_fullsized":"2024-02-23T00:35:09.109222+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.509463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.509463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":349,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091113+0000","last_change":"2024-02-23T00:34:41.101853+0000","last_active":"2024-02-23T00:35:09.091113+0000","last_peered":"2024-02-23T00:35:09.091113+0000","last_clean":"2024-02-23T00:35:09.091113+0000","last_became_active":"2024-02-23T00:34:41.101472+0000","last_became_peered":"2024-02-23T00:34:41.101472+0000","last_unstale":"2024-02-23T00:35:09.091113+0000","last_undegraded":"2024-02-23T00:35:09.091113+0000","last_fullsized":"2024-02-23T00:35:09.091113+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:31.593354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:31.593354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":328,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096996+0000","last_change":"2024-02-23T00:35:02.744199+0000","last_active":"2024-02-23T00:35:09.096996+0000","last_peered":"2024-02-23T00:35:09.096996+0000","last_clean":"2024-02-23T00:35:09.096996+0000","last_became_active":"2024-02-23T00:34:41.118378+0000","last_became_peered":"2024-02-23T00:34:41.118378+0000","last_unstale":"2024-02-23T00:35:09.096996+0000","last_undegraded":"2024-02-23T00:35:09.096996+0000","last_fullsized":"2024-02-23T00:35:09.096996+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.744088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.744088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":275,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090207+0000","last_change":"2024-02-23T00:33:34.545991+0000","last_active":"2024-02-23T00:35:09.090207+0000","last_peered":"2024-02-23T00:35:09.090207+0000","last_clean":"2024-02-23T00:35:09.090207+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:35:09.090207+0000","last_undegraded":"2024-02-23T00:35:09.090207+0000","last_fullsized":"2024-02-23T00:35:09.090207+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.545903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.545903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":327,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109735+0000","last_change":"2024-02-23T00:34:28.710296+0000","last_active":"2024-02-23T00:35:09.109735+0000","last_peered":"2024-02-23T00:35:09.109735+0000","last_clean":"2024-02-23T00:35:09.109735+0000","last_became_active":"2024-02-23T00:27:21.026101+0000","last_became_peered":"2024-02-23T00:27:21.026101+0000","last_unstale":"2024-02-23T00:35:09.109735+0000","last_undegraded":"2024-02-23T00:35:09.109735+0000","last_fullsized":"2024-02-23T00:35:09.109735+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":351,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093029+0000","last_change":"2024-02-23T00:34:15.295264+0000","last_active":"2024-02-23T00:35:09.093029+0000","last_peered":"2024-02-23T00:35:09.093029+0000","last_clean":"2024-02-23T00:35:09.093029+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:35:09.093029+0000","last_undegraded":"2024-02-23T00:35:09.093029+0000","last_fullsized":"2024-02-23T00:35:09.093029+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:34:15.295178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.295178+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":334,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109300+0000","last_change":"2024-02-23T00:33:43.548018+0000","last_active":"2024-02-23T00:35:09.109300+0000","last_peered":"2024-02-23T00:35:09.109300+0000","last_clean":"2024-02-23T00:35:09.109300+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:35:09.109300+0000","last_undegraded":"2024-02-23T00:35:09.109300+0000","last_fullsized":"2024-02-23T00:35:09.109300+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":324,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114263+0000","last_change":"2024-02-23T00:32:46.410759+0000","last_active":"2024-02-23T00:35:09.114263+0000","last_peered":"2024-02-23T00:35:09.114263+0000","last_clean":"2024-02-23T00:35:09.114263+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:35:09.114263+0000","last_undegraded":"2024-02-23T00:35:09.114263+0000","last_fullsized":"2024-02-23T00:35:09.114263+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.410635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.410635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":141,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114340+0000","last_change":"2024-02-23T00:34:41.099959+0000","last_active":"2024-02-23T00:35:09.114340+0000","last_peered":"2024-02-23T00:35:09.114340+0000","last_clean":"2024-02-23T00:35:09.114340+0000","last_became_active":"2024-02-23T00:34:41.099598+0000","last_became_peered":"2024-02-23T00:34:41.099598+0000","last_unstale":"2024-02-23T00:35:09.114340+0000","last_undegraded":"2024-02-23T00:35:09.114340+0000","last_fullsized":"2024-02-23T00:35:09.114340+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.978406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.978406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":327,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.115374+0000","last_change":"2024-02-23T00:34:41.114523+0000","last_active":"2024-02-23T00:35:09.115374+0000","last_peered":"2024-02-23T00:35:09.115374+0000","last_clean":"2024-02-23T00:35:09.115374+0000","last_became_active":"2024-02-23T00:34:41.114270+0000","last_became_peered":"2024-02-23T00:34:41.114270+0000","last_unstale":"2024-02-23T00:35:09.115374+0000","last_undegraded":"2024-02-23T00:35:09.115374+0000","last_fullsized":"2024-02-23T00:35:09.115374+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.439872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":275,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090643+0000","last_change":"2024-02-23T00:33:33.574217+0000","last_active":"2024-02-23T00:35:09.090643+0000","last_peered":"2024-02-23T00:35:09.090643+0000","last_clean":"2024-02-23T00:35:09.090643+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:35:09.090643+0000","last_undegraded":"2024-02-23T00:35:09.090643+0000","last_fullsized":"2024-02-23T00:35:09.090643+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.574124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.574124+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":329,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093964+0000","last_change":"2024-02-23T00:33:38.742275+0000","last_active":"2024-02-23T00:35:09.093964+0000","last_peered":"2024-02-23T00:35:09.093964+0000","last_clean":"2024-02-23T00:35:09.093964+0000","last_became_active":"2024-02-23T00:27:21.025807+0000","last_became_peered":"2024-02-23T00:27:21.025807+0000","last_unstale":"2024-02-23T00:35:09.093964+0000","last_undegraded":"2024-02-23T00:35:09.093964+0000","last_fullsized":"2024-02-23T00:35:09.093964+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":365,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096569+0000","last_change":"2024-02-23T00:32:13.104236+0000","last_active":"2024-02-23T00:35:09.096569+0000","last_peered":"2024-02-23T00:35:09.096569+0000","last_clean":"2024-02-23T00:35:09.096569+0000","last_became_active":"2024-02-23T00:27:19.026209+0000","last_became_peered":"2024-02-23T00:27:19.026209+0000","last_unstale":"2024-02-23T00:35:09.096569+0000","last_undegraded":"2024-02-23T00:35:09.096569+0000","last_fullsized":"2024-02-23T00:35:09.096569+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:32:13.104170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:13.104170+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":30,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":342,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093599+0000","last_change":"2024-02-23T00:34:41.095033+0000","last_active":"2024-02-23T00:35:09.093599+0000","last_peered":"2024-02-23T00:35:09.093599+0000","last_clean":"2024-02-23T00:35:09.093599+0000","last_became_active":"2024-02-23T00:34:41.094564+0000","last_became_peered":"2024-02-23T00:34:41.094564+0000","last_unstale":"2024-02-23T00:35:09.093599+0000","last_undegraded":"2024-02-23T00:35:09.093599+0000","last_fullsized":"2024-02-23T00:35:09.093599+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:32.799394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:32.799394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":581,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095305+0000","last_change":"2024-02-23T00:32:41.360750+0000","last_active":"2024-02-23T00:35:09.095305+0000","last_peered":"2024-02-23T00:35:09.095305+0000","last_clean":"2024-02-23T00:35:09.095305+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:35:09.095305+0000","last_undegraded":"2024-02-23T00:35:09.095305+0000","last_fullsized":"2024-02-23T00:35:09.095305+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:41.360646+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.360646+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":334,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110301+0000","last_change":"2024-02-23T00:34:41.105901+0000","last_active":"2024-02-23T00:35:09.110301+0000","last_peered":"2024-02-23T00:35:09.110301+0000","last_clean":"2024-02-23T00:35:09.110301+0000","last_became_active":"2024-02-23T00:34:41.105566+0000","last_became_peered":"2024-02-23T00:34:41.105566+0000","last_unstale":"2024-02-23T00:35:09.110301+0000","last_undegraded":"2024-02-23T00:35:09.110301+0000","last_fullsized":"2024-02-23T00:35:09.110301+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:18.732304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:18.732304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":280,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093147+0000","last_change":"2024-02-23T00:35:03.997896+0000","last_active":"2024-02-23T00:35:09.093147+0000","last_peered":"2024-02-23T00:35:09.093147+0000","last_clean":"2024-02-23T00:35:09.093147+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:35:09.093147+0000","last_undegraded":"2024-02-23T00:35:09.093147+0000","last_fullsized":"2024-02-23T00:35:09.093147+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.997803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.997803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":342,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089606+0000","last_change":"2024-02-23T00:35:03.595532+0000","last_active":"2024-02-23T00:35:09.089606+0000","last_peered":"2024-02-23T00:35:09.089606+0000","last_clean":"2024-02-23T00:35:09.089606+0000","last_became_active":"2024-02-23T00:31:29.065548+0000","last_became_peered":"2024-02-23T00:31:29.065548+0000","last_unstale":"2024-02-23T00:35:09.089606+0000","last_undegraded":"2024-02-23T00:35:09.089606+0000","last_fullsized":"2024-02-23T00:35:09.089606+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.595447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.595447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":329,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092528+0000","last_change":"2024-02-23T00:32:50.596925+0000","last_active":"2024-02-23T00:35:09.092528+0000","last_peered":"2024-02-23T00:35:09.092528+0000","last_clean":"2024-02-23T00:35:09.092528+0000","last_became_active":"2024-02-23T00:27:21.029062+0000","last_became_peered":"2024-02-23T00:27:21.029062+0000","last_unstale":"2024-02-23T00:35:09.092528+0000","last_undegraded":"2024-02-23T00:35:09.092528+0000","last_fullsized":"2024-02-23T00:35:09.092528+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"330'35","reported_seq":514,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.133736+0000","last_change":"2024-02-23T00:34:42.829027+0000","last_active":"2024-02-23T00:35:10.133736+0000","last_peered":"2024-02-23T00:35:10.133736+0000","last_clean":"2024-02-23T00:35:10.133736+0000","last_became_active":"2024-02-23T00:34:41.099241+0000","last_became_peered":"2024-02-23T00:34:41.099241+0000","last_unstale":"2024-02-23T00:35:10.133736+0000","last_undegraded":"2024-02-23T00:35:10.133736+0000","last_fullsized":"2024-02-23T00:35:10.133736+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:32:45.366416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.366416+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":330,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114212+0000","last_change":"2024-02-23T00:33:36.350954+0000","last_active":"2024-02-23T00:35:09.114212+0000","last_peered":"2024-02-23T00:35:09.114212+0000","last_clean":"2024-02-23T00:35:09.114212+0000","last_became_active":"2024-02-23T00:27:17.008153+0000","last_became_peered":"2024-02-23T00:27:17.008153+0000","last_unstale":"2024-02-23T00:35:09.114212+0000","last_undegraded":"2024-02-23T00:35:09.114212+0000","last_fullsized":"2024-02-23T00:35:09.114212+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:36.350888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.350888+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":41,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090584+0000","last_change":"2024-02-23T00:34:41.110416+0000","last_active":"2024-02-23T00:35:09.090584+0000","last_peered":"2024-02-23T00:35:09.090584+0000","last_clean":"2024-02-23T00:35:09.090584+0000","last_became_active":"2024-02-23T00:34:41.109972+0000","last_became_peered":"2024-02-23T00:34:41.109972+0000","last_unstale":"2024-02-23T00:35:09.090584+0000","last_undegraded":"2024-02-23T00:35:09.090584+0000","last_fullsized":"2024-02-23T00:35:09.090584+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:41.701634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:41.701634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":280,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.094118+0000","last_change":"2024-02-23T00:35:06.983497+0000","last_active":"2024-02-23T00:35:09.094118+0000","last_peered":"2024-02-23T00:35:09.094118+0000","last_clean":"2024-02-23T00:35:09.094118+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:35:09.094118+0000","last_undegraded":"2024-02-23T00:35:09.094118+0000","last_fullsized":"2024-02-23T00:35:09.094118+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:06.983436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.983436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":243,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114441+0000","last_change":"2024-02-23T00:34:41.110476+0000","last_active":"2024-02-23T00:35:09.114441+0000","last_peered":"2024-02-23T00:35:09.114441+0000","last_clean":"2024-02-23T00:35:09.114441+0000","last_became_active":"2024-02-23T00:34:41.109182+0000","last_became_peered":"2024-02-23T00:34:41.109182+0000","last_unstale":"2024-02-23T00:35:09.114441+0000","last_undegraded":"2024-02-23T00:35:09.114441+0000","last_fullsized":"2024-02-23T00:35:09.114441+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:36.447439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:36.447439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":433,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.136362+0000","last_change":"2024-02-23T00:33:46.629281+0000","last_active":"2024-02-23T00:35:10.136362+0000","last_peered":"2024-02-23T00:35:10.136362+0000","last_clean":"2024-02-23T00:35:10.136362+0000","last_became_active":"2024-02-23T00:27:21.025986+0000","last_became_peered":"2024-02-23T00:27:21.025986+0000","last_unstale":"2024-02-23T00:35:10.136362+0000","last_undegraded":"2024-02-23T00:35:10.136362+0000","last_fullsized":"2024-02-23T00:35:10.136362+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":353,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096394+0000","last_change":"2024-02-23T00:33:37.175714+0000","last_active":"2024-02-23T00:35:09.096394+0000","last_peered":"2024-02-23T00:35:09.096394+0000","last_clean":"2024-02-23T00:35:09.096394+0000","last_became_active":"2024-02-23T00:27:19.022390+0000","last_became_peered":"2024-02-23T00:27:19.022390+0000","last_unstale":"2024-02-23T00:35:09.096394+0000","last_undegraded":"2024-02-23T00:35:09.096394+0000","last_fullsized":"2024-02-23T00:35:09.096394+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:33:37.175650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.175650+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":328,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096826+0000","last_change":"2024-02-23T00:34:13.208540+0000","last_active":"2024-02-23T00:35:09.096826+0000","last_peered":"2024-02-23T00:35:09.096826+0000","last_clean":"2024-02-23T00:35:09.096826+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:35:09.096826+0000","last_undegraded":"2024-02-23T00:35:09.096826+0000","last_fullsized":"2024-02-23T00:35:09.096826+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":334,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091657+0000","last_change":"2024-02-23T00:34:41.116110+0000","last_active":"2024-02-23T00:35:09.091657+0000","last_peered":"2024-02-23T00:35:09.091657+0000","last_clean":"2024-02-23T00:35:09.091657+0000","last_became_active":"2024-02-23T00:34:41.115657+0000","last_became_peered":"2024-02-23T00:34:41.115657+0000","last_unstale":"2024-02-23T00:35:09.091657+0000","last_undegraded":"2024-02-23T00:35:09.091657+0000","last_fullsized":"2024-02-23T00:35:09.091657+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.488959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.488959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":275,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089275+0000","last_change":"2024-02-23T00:33:49.754910+0000","last_active":"2024-02-23T00:35:09.089275+0000","last_peered":"2024-02-23T00:35:09.089275+0000","last_clean":"2024-02-23T00:35:09.089275+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:35:09.089275+0000","last_undegraded":"2024-02-23T00:35:09.089275+0000","last_fullsized":"2024-02-23T00:35:09.089275+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:49.754830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.754830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":324,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091447+0000","last_change":"2024-02-23T00:32:42.815796+0000","last_active":"2024-02-23T00:35:09.091447+0000","last_peered":"2024-02-23T00:35:09.091447+0000","last_clean":"2024-02-23T00:35:09.091447+0000","last_became_active":"2024-02-23T00:27:21.024002+0000","last_became_peered":"2024-02-23T00:27:21.024002+0000","last_unstale":"2024-02-23T00:35:09.091447+0000","last_undegraded":"2024-02-23T00:35:09.091447+0000","last_fullsized":"2024-02-23T00:35:09.091447+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.815722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.815722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"278'50","reported_seq":473,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.098451+0000","last_change":"2024-02-23T00:35:09.098161+0000","last_active":"2024-02-23T00:35:09.098451+0000","last_peered":"2024-02-23T00:35:09.098451+0000","last_clean":"2024-02-23T00:35:09.098451+0000","last_became_active":"2024-02-23T00:35:09.097721+0000","last_became_peered":"2024-02-23T00:35:09.097721+0000","last_unstale":"2024-02-23T00:35:09.098451+0000","last_undegraded":"2024-02-23T00:35:09.098451+0000","last_fullsized":"2024-02-23T00:35:09.098451+0000","mapping_epoch":349,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:34:54.691537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.691537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":123,"num_read_kb":99,"num_write":66,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":341,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090797+0000","last_change":"2024-02-23T00:33:43.800448+0000","last_active":"2024-02-23T00:35:09.090797+0000","last_peered":"2024-02-23T00:35:09.090797+0000","last_clean":"2024-02-23T00:35:09.090797+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:35:09.090797+0000","last_undegraded":"2024-02-23T00:35:09.090797+0000","last_fullsized":"2024-02-23T00:35:09.090797+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:33:43.800360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.800360+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":41,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109665+0000","last_change":"2024-02-23T00:34:41.111021+0000","last_active":"2024-02-23T00:35:09.109665+0000","last_peered":"2024-02-23T00:35:09.109665+0000","last_clean":"2024-02-23T00:35:09.109665+0000","last_became_active":"2024-02-23T00:34:41.110345+0000","last_became_peered":"2024-02-23T00:34:41.110345+0000","last_unstale":"2024-02-23T00:35:09.109665+0000","last_undegraded":"2024-02-23T00:35:09.109665+0000","last_fullsized":"2024-02-23T00:35:09.109665+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:32.794268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:32.794268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":273,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114020+0000","last_change":"2024-02-23T00:32:59.434164+0000","last_active":"2024-02-23T00:35:09.114020+0000","last_peered":"2024-02-23T00:35:09.114020+0000","last_clean":"2024-02-23T00:35:09.114020+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:35:09.114020+0000","last_undegraded":"2024-02-23T00:35:09.114020+0000","last_fullsized":"2024-02-23T00:35:09.114020+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.434079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.434079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"63'21","reported_seq":404,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092549+0000","last_change":"2024-02-23T00:34:33.083331+0000","last_active":"2024-02-23T00:35:09.092549+0000","last_peered":"2024-02-23T00:35:09.092549+0000","last_clean":"2024-02-23T00:35:09.092549+0000","last_became_active":"2024-02-23T00:34:33.061786+0000","last_became_peered":"2024-02-23T00:34:33.061786+0000","last_unstale":"2024-02-23T00:35:09.092549+0000","last_undegraded":"2024-02-23T00:35:09.092549+0000","last_fullsized":"2024-02-23T00:35:09.092549+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'21","last_scrub_stamp":"2024-02-23T00:34:27.539986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.539986+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":333,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091368+0000","last_change":"2024-02-23T00:34:24.667629+0000","last_active":"2024-02-23T00:35:09.091368+0000","last_peered":"2024-02-23T00:35:09.091368+0000","last_clean":"2024-02-23T00:35:09.091368+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:35:09.091368+0000","last_undegraded":"2024-02-23T00:35:09.091368+0000","last_fullsized":"2024-02-23T00:35:09.091368+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.667517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.667517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":41,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096643+0000","last_change":"2024-02-23T00:34:41.111655+0000","last_active":"2024-02-23T00:35:09.096643+0000","last_peered":"2024-02-23T00:35:09.096643+0000","last_clean":"2024-02-23T00:35:09.096643+0000","last_became_active":"2024-02-23T00:34:41.110606+0000","last_became_peered":"2024-02-23T00:34:41.110606+0000","last_unstale":"2024-02-23T00:35:09.096643+0000","last_undegraded":"2024-02-23T00:35:09.096643+0000","last_fullsized":"2024-02-23T00:35:09.096643+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.614960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.614960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":327,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092204+0000","last_change":"2024-02-23T00:33:34.777997+0000","last_active":"2024-02-23T00:35:09.092204+0000","last_peered":"2024-02-23T00:35:09.092204+0000","last_clean":"2024-02-23T00:35:09.092204+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:35:09.092204+0000","last_undegraded":"2024-02-23T00:35:09.092204+0000","last_fullsized":"2024-02-23T00:35:09.092204+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.777903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.777903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":46,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114458+0000","last_change":"2024-02-23T00:34:56.402946+0000","last_active":"2024-02-23T00:35:09.114458+0000","last_peered":"2024-02-23T00:35:09.114458+0000","last_clean":"2024-02-23T00:35:09.114458+0000","last_became_active":"2024-02-23T00:34:41.114943+0000","last_became_peered":"2024-02-23T00:34:41.114943+0000","last_unstale":"2024-02-23T00:35:09.114458+0000","last_undegraded":"2024-02-23T00:35:09.114458+0000","last_fullsized":"2024-02-23T00:35:09.114458+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.402851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.402851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"63'12","reported_seq":61,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096175+0000","last_change":"2024-02-23T00:34:34.457919+0000","last_active":"2024-02-23T00:35:09.096175+0000","last_peered":"2024-02-23T00:35:09.096175+0000","last_clean":"2024-02-23T00:35:09.096175+0000","last_became_active":"2024-02-23T00:34:33.061357+0000","last_became_peered":"2024-02-23T00:34:33.061357+0000","last_unstale":"2024-02-23T00:35:09.096175+0000","last_undegraded":"2024-02-23T00:35:09.096175+0000","last_fullsized":"2024-02-23T00:35:09.096175+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'12","last_scrub_stamp":"2024-02-23T00:33:28.885879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:28.885879+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":331,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092305+0000","last_change":"2024-02-23T00:34:27.296053+0000","last_active":"2024-02-23T00:35:09.092305+0000","last_peered":"2024-02-23T00:35:09.092305+0000","last_clean":"2024-02-23T00:35:09.092305+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:35:09.092305+0000","last_undegraded":"2024-02-23T00:35:09.092305+0000","last_fullsized":"2024-02-23T00:35:09.092305+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:27.295978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.295978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":324,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114651+0000","last_change":"2024-02-23T00:34:41.114079+0000","last_active":"2024-02-23T00:35:09.114651+0000","last_peered":"2024-02-23T00:35:09.114651+0000","last_clean":"2024-02-23T00:35:09.114651+0000","last_became_active":"2024-02-23T00:34:41.113722+0000","last_became_peered":"2024-02-23T00:34:41.113722+0000","last_unstale":"2024-02-23T00:35:09.114651+0000","last_undegraded":"2024-02-23T00:35:09.114651+0000","last_fullsized":"2024-02-23T00:35:09.114651+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:07.275816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:07.275816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":325,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110022+0000","last_change":"2024-02-23T00:34:24.675792+0000","last_active":"2024-02-23T00:35:09.110022+0000","last_peered":"2024-02-23T00:35:09.110022+0000","last_clean":"2024-02-23T00:35:09.110022+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:35:09.110022+0000","last_undegraded":"2024-02-23T00:35:09.110022+0000","last_fullsized":"2024-02-23T00:35:09.110022+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.675729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.675729+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":287,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093283+0000","last_change":"2024-02-23T00:35:05.147321+0000","last_active":"2024-02-23T00:35:09.093283+0000","last_peered":"2024-02-23T00:35:09.093283+0000","last_clean":"2024-02-23T00:35:09.093283+0000","last_became_active":"2024-02-23T00:34:41.101132+0000","last_became_peered":"2024-02-23T00:34:41.101132+0000","last_unstale":"2024-02-23T00:35:09.093283+0000","last_undegraded":"2024-02-23T00:35:09.093283+0000","last_fullsized":"2024-02-23T00:35:09.093283+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:05.147249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:05.147249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":115,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096912+0000","last_change":"2024-02-23T00:34:42.571126+0000","last_active":"2024-02-23T00:35:09.096912+0000","last_peered":"2024-02-23T00:35:09.096912+0000","last_clean":"2024-02-23T00:35:09.096912+0000","last_became_active":"2024-02-23T00:34:41.106795+0000","last_became_peered":"2024-02-23T00:34:41.106795+0000","last_unstale":"2024-02-23T00:35:09.096912+0000","last_undegraded":"2024-02-23T00:35:09.096912+0000","last_fullsized":"2024-02-23T00:35:09.096912+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:34:15.571539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.571539+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":37,"num_read_kb":30,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":333,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114153+0000","last_change":"2024-02-23T00:34:41.104108+0000","last_active":"2024-02-23T00:35:09.114153+0000","last_peered":"2024-02-23T00:35:09.114153+0000","last_clean":"2024-02-23T00:35:09.114153+0000","last_became_active":"2024-02-23T00:34:41.103743+0000","last_became_peered":"2024-02-23T00:34:41.103743+0000","last_unstale":"2024-02-23T00:35:09.114153+0000","last_undegraded":"2024-02-23T00:35:09.114153+0000","last_fullsized":"2024-02-23T00:35:09.114153+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.319744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.319744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":424,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.133513+0000","last_change":"2024-02-23T00:32:56.376450+0000","last_active":"2024-02-23T00:35:10.133513+0000","last_peered":"2024-02-23T00:35:10.133513+0000","last_clean":"2024-02-23T00:35:10.133513+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:35:10.133513+0000","last_undegraded":"2024-02-23T00:35:10.133513+0000","last_fullsized":"2024-02-23T00:35:10.133513+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":334,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092390+0000","last_change":"2024-02-23T00:34:41.101927+0000","last_active":"2024-02-23T00:35:09.092390+0000","last_peered":"2024-02-23T00:35:09.092390+0000","last_clean":"2024-02-23T00:35:09.092390+0000","last_became_active":"2024-02-23T00:34:41.101596+0000","last_became_peered":"2024-02-23T00:34:41.101596+0000","last_unstale":"2024-02-23T00:35:09.092390+0000","last_undegraded":"2024-02-23T00:35:09.092390+0000","last_fullsized":"2024-02-23T00:35:09.092390+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.521760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":273,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114918+0000","last_change":"2024-02-23T00:32:45.409616+0000","last_active":"2024-02-23T00:35:09.114918+0000","last_peered":"2024-02-23T00:35:09.114918+0000","last_clean":"2024-02-23T00:35:09.114918+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:35:09.114918+0000","last_undegraded":"2024-02-23T00:35:09.114918+0000","last_fullsized":"2024-02-23T00:35:09.114918+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.409548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.409548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"144'20","reported_seq":379,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114666+0000","last_change":"2024-02-23T00:35:01.515684+0000","last_active":"2024-02-23T00:35:09.114666+0000","last_peered":"2024-02-23T00:35:09.114666+0000","last_clean":"2024-02-23T00:35:09.114666+0000","last_became_active":"2024-02-23T00:34:41.110140+0000","last_became_peered":"2024-02-23T00:34:41.110140+0000","last_unstale":"2024-02-23T00:35:09.114666+0000","last_undegraded":"2024-02-23T00:35:09.114666+0000","last_fullsized":"2024-02-23T00:35:09.114666+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:35:01.515550+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.515550+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":345,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.392094+0000","last_change":"2024-02-23T00:35:10.392094+0000","last_active":"2024-02-23T00:35:10.392094+0000","last_peered":"2024-02-23T00:35:10.392094+0000","last_clean":"2024-02-23T00:35:10.392094+0000","last_became_active":"2024-02-23T00:34:41.103241+0000","last_became_peered":"2024-02-23T00:34:41.103241+0000","last_unstale":"2024-02-23T00:35:10.392094+0000","last_undegraded":"2024-02-23T00:35:10.392094+0000","last_fullsized":"2024-02-23T00:35:10.392094+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.391997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.391997+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"330'10","reported_seq":131,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.130630+0000","last_change":"2024-02-23T00:35:06.628388+0000","last_active":"2024-02-23T00:35:10.130630+0000","last_peered":"2024-02-23T00:35:10.130630+0000","last_clean":"2024-02-23T00:35:10.130630+0000","last_became_active":"2024-02-23T00:34:41.109074+0000","last_became_peered":"2024-02-23T00:34:41.109074+0000","last_unstale":"2024-02-23T00:35:10.130630+0000","last_undegraded":"2024-02-23T00:35:10.130630+0000","last_fullsized":"2024-02-23T00:35:10.130630+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'10","last_scrub_stamp":"2024-02-23T00:35:06.628306+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.628306+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":327,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090962+0000","last_change":"2024-02-23T00:35:07.627722+0000","last_active":"2024-02-23T00:35:09.090962+0000","last_peered":"2024-02-23T00:35:09.090962+0000","last_clean":"2024-02-23T00:35:09.090962+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:35:09.090962+0000","last_undegraded":"2024-02-23T00:35:09.090962+0000","last_fullsized":"2024-02-23T00:35:09.090962+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:07.627620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:07.627620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":41,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093690+0000","last_change":"2024-02-23T00:34:41.091527+0000","last_active":"2024-02-23T00:35:09.093690+0000","last_peered":"2024-02-23T00:35:09.093690+0000","last_clean":"2024-02-23T00:35:09.093690+0000","last_became_active":"2024-02-23T00:34:41.091189+0000","last_became_peered":"2024-02-23T00:34:41.091189+0000","last_unstale":"2024-02-23T00:35:09.093690+0000","last_undegraded":"2024-02-23T00:35:09.093690+0000","last_fullsized":"2024-02-23T00:35:09.093690+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:14.586935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:14.586935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":339,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096294+0000","last_change":"2024-02-23T00:32:50.448701+0000","last_active":"2024-02-23T00:35:09.096294+0000","last_peered":"2024-02-23T00:35:09.096294+0000","last_clean":"2024-02-23T00:35:09.096294+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:35:09.096294+0000","last_undegraded":"2024-02-23T00:35:09.096294+0000","last_fullsized":"2024-02-23T00:35:09.096294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:32:50.448595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.448595+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":340,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093056+0000","last_change":"2024-02-23T00:34:41.117167+0000","last_active":"2024-02-23T00:35:09.093056+0000","last_peered":"2024-02-23T00:35:09.093056+0000","last_clean":"2024-02-23T00:35:09.093056+0000","last_became_active":"2024-02-23T00:34:41.116840+0000","last_became_peered":"2024-02-23T00:34:41.116840+0000","last_unstale":"2024-02-23T00:35:09.093056+0000","last_undegraded":"2024-02-23T00:35:09.093056+0000","last_fullsized":"2024-02-23T00:35:09.093056+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.310110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.310110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":44,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092783+0000","last_change":"2024-02-23T00:35:00.416568+0000","last_active":"2024-02-23T00:35:09.092783+0000","last_peered":"2024-02-23T00:35:09.092783+0000","last_clean":"2024-02-23T00:35:09.092783+0000","last_became_active":"2024-02-23T00:34:41.115996+0000","last_became_peered":"2024-02-23T00:34:41.115996+0000","last_unstale":"2024-02-23T00:35:09.092783+0000","last_undegraded":"2024-02-23T00:35:09.092783+0000","last_fullsized":"2024-02-23T00:35:09.092783+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.416493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.416493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":330,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090966+0000","last_change":"2024-02-23T00:35:04.414054+0000","last_active":"2024-02-23T00:35:09.090966+0000","last_peered":"2024-02-23T00:35:09.090966+0000","last_clean":"2024-02-23T00:35:09.090966+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:35:09.090966+0000","last_undegraded":"2024-02-23T00:35:09.090966+0000","last_fullsized":"2024-02-23T00:35:09.090966+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.413991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.413991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":275,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089179+0000","last_change":"2024-02-23T00:33:48.779550+0000","last_active":"2024-02-23T00:35:09.089179+0000","last_peered":"2024-02-23T00:35:09.089179+0000","last_clean":"2024-02-23T00:35:09.089179+0000","last_became_active":"2024-02-23T00:30:48.666975+0000","last_became_peered":"2024-02-23T00:30:48.666975+0000","last_unstale":"2024-02-23T00:35:09.089179+0000","last_undegraded":"2024-02-23T00:35:09.089179+0000","last_fullsized":"2024-02-23T00:35:09.089179+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:48.779472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:48.779472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":366,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089828+0000","last_change":"2024-02-23T00:33:30.498502+0000","last_active":"2024-02-23T00:35:09.089828+0000","last_peered":"2024-02-23T00:35:09.089828+0000","last_clean":"2024-02-23T00:35:09.089828+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:35:09.089828+0000","last_undegraded":"2024-02-23T00:35:09.089828+0000","last_fullsized":"2024-02-23T00:35:09.089828+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:33:30.498425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:30.498425+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":320,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095684+0000","last_change":"2024-02-23T00:31:32.874749+0000","last_active":"2024-02-23T00:35:09.095684+0000","last_peered":"2024-02-23T00:35:09.095684+0000","last_clean":"2024-02-23T00:35:09.095684+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:35:09.095684+0000","last_undegraded":"2024-02-23T00:35:09.095684+0000","last_fullsized":"2024-02-23T00:35:09.095684+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:32.874678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:31:32.874678+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":330,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096348+0000","last_change":"2024-02-23T00:34:41.116355+0000","last_active":"2024-02-23T00:35:09.096348+0000","last_peered":"2024-02-23T00:35:09.096348+0000","last_clean":"2024-02-23T00:35:09.096348+0000","last_became_active":"2024-02-23T00:34:41.115862+0000","last_became_peered":"2024-02-23T00:34:41.115862+0000","last_unstale":"2024-02-23T00:35:09.096348+0000","last_undegraded":"2024-02-23T00:35:09.096348+0000","last_fullsized":"2024-02-23T00:35:09.096348+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.151986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":39,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091322+0000","last_change":"2024-02-23T00:34:41.115170+0000","last_active":"2024-02-23T00:35:09.091322+0000","last_peered":"2024-02-23T00:35:09.091322+0000","last_clean":"2024-02-23T00:35:09.091322+0000","last_became_active":"2024-02-23T00:34:41.114829+0000","last_became_peered":"2024-02-23T00:34:41.114829+0000","last_unstale":"2024-02-23T00:35:09.091322+0000","last_undegraded":"2024-02-23T00:35:09.091322+0000","last_fullsized":"2024-02-23T00:35:09.091322+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.620107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":299,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092463+0000","last_change":"2024-02-23T00:34:41.193231+0000","last_active":"2024-02-23T00:35:09.092463+0000","last_peered":"2024-02-23T00:35:09.092463+0000","last_clean":"2024-02-23T00:35:09.092463+0000","last_became_active":"2024-02-23T00:34:41.093430+0000","last_became_peered":"2024-02-23T00:34:41.093430+0000","last_unstale":"2024-02-23T00:35:09.092463+0000","last_undegraded":"2024-02-23T00:35:09.092463+0000","last_fullsized":"2024-02-23T00:35:09.092463+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:33:38.437857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.437857+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":80,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095314+0000","last_change":"2024-02-23T00:34:41.135529+0000","last_active":"2024-02-23T00:35:09.095314+0000","last_peered":"2024-02-23T00:35:09.095314+0000","last_clean":"2024-02-23T00:35:09.095314+0000","last_became_active":"2024-02-23T00:34:41.101219+0000","last_became_peered":"2024-02-23T00:34:41.101219+0000","last_unstale":"2024-02-23T00:35:09.095314+0000","last_undegraded":"2024-02-23T00:35:09.095314+0000","last_fullsized":"2024-02-23T00:35:09.095314+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:32:11.989333+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.989333+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":324,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114751+0000","last_change":"2024-02-23T00:32:40.343941+0000","last_active":"2024-02-23T00:35:09.114751+0000","last_peered":"2024-02-23T00:35:09.114751+0000","last_clean":"2024-02-23T00:35:09.114751+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:35:09.114751+0000","last_undegraded":"2024-02-23T00:35:09.114751+0000","last_fullsized":"2024-02-23T00:35:09.114751+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.343863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.343863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":322,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092262+0000","last_change":"2024-02-23T00:32:47.484343+0000","last_active":"2024-02-23T00:35:09.092262+0000","last_peered":"2024-02-23T00:35:09.092262+0000","last_clean":"2024-02-23T00:35:09.092262+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:35:09.092262+0000","last_undegraded":"2024-02-23T00:35:09.092262+0000","last_fullsized":"2024-02-23T00:35:09.092262+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:47.484285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.484285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":325,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109321+0000","last_change":"2024-02-23T00:34:17.760839+0000","last_active":"2024-02-23T00:35:09.109321+0000","last_peered":"2024-02-23T00:35:09.109321+0000","last_clean":"2024-02-23T00:35:09.109321+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:35:09.109321+0000","last_undegraded":"2024-02-23T00:35:09.109321+0000","last_fullsized":"2024-02-23T00:35:09.109321+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.760765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.760765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":272,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.096437+0000","last_change":"2024-02-23T00:33:44.152089+0000","last_active":"2024-02-23T00:35:09.096437+0000","last_peered":"2024-02-23T00:35:09.096437+0000","last_clean":"2024-02-23T00:35:09.096437+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:35:09.096437+0000","last_undegraded":"2024-02-23T00:35:09.096437+0000","last_fullsized":"2024-02-23T00:35:09.096437+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.152014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.152014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":127,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109185+0000","last_change":"2024-02-23T00:35:01.643176+0000","last_active":"2024-02-23T00:35:09.109185+0000","last_peered":"2024-02-23T00:35:09.109185+0000","last_clean":"2024-02-23T00:35:09.109185+0000","last_became_active":"2024-02-23T00:34:41.114516+0000","last_became_peered":"2024-02-23T00:34:41.114516+0000","last_unstale":"2024-02-23T00:35:09.109185+0000","last_undegraded":"2024-02-23T00:35:09.109185+0000","last_fullsized":"2024-02-23T00:35:09.109185+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:35:01.643114+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.643114+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":330,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093194+0000","last_change":"2024-02-23T00:35:03.113215+0000","last_active":"2024-02-23T00:35:09.093194+0000","last_peered":"2024-02-23T00:35:09.093194+0000","last_clean":"2024-02-23T00:35:09.093194+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:35:09.093194+0000","last_undegraded":"2024-02-23T00:35:09.093194+0000","last_fullsized":"2024-02-23T00:35:09.093194+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.113099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.113099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":322,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109465+0000","last_change":"2024-02-23T00:32:46.655443+0000","last_active":"2024-02-23T00:35:09.109465+0000","last_peered":"2024-02-23T00:35:09.109465+0000","last_clean":"2024-02-23T00:35:09.109465+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:35:09.109465+0000","last_undegraded":"2024-02-23T00:35:09.109465+0000","last_fullsized":"2024-02-23T00:35:09.109465+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.655378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.655378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":39,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091963+0000","last_change":"2024-02-23T00:34:41.097425+0000","last_active":"2024-02-23T00:35:09.091963+0000","last_peered":"2024-02-23T00:35:09.091963+0000","last_clean":"2024-02-23T00:35:09.091963+0000","last_became_active":"2024-02-23T00:34:41.097083+0000","last_became_peered":"2024-02-23T00:34:41.097083+0000","last_unstale":"2024-02-23T00:35:09.091963+0000","last_undegraded":"2024-02-23T00:35:09.091963+0000","last_fullsized":"2024-02-23T00:35:09.091963+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:30.840800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:30.840800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":284,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089971+0000","last_change":"2024-02-23T00:34:41.106123+0000","last_active":"2024-02-23T00:35:09.089971+0000","last_peered":"2024-02-23T00:35:09.089971+0000","last_clean":"2024-02-23T00:35:09.089971+0000","last_became_active":"2024-02-23T00:34:41.105807+0000","last_became_peered":"2024-02-23T00:34:41.105807+0000","last_unstale":"2024-02-23T00:35:09.089971+0000","last_undegraded":"2024-02-23T00:35:09.089971+0000","last_fullsized":"2024-02-23T00:35:09.089971+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:45.795278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.795278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":333,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.113868+0000","last_change":"2024-02-23T00:34:42.064110+0000","last_active":"2024-02-23T00:35:09.113868+0000","last_peered":"2024-02-23T00:35:09.113868+0000","last_clean":"2024-02-23T00:35:09.113868+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:35:09.113868+0000","last_undegraded":"2024-02-23T00:35:09.113868+0000","last_fullsized":"2024-02-23T00:35:09.113868+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:32.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:32.520027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":433,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.137446+0000","last_change":"2024-02-23T00:34:12.258420+0000","last_active":"2024-02-23T00:35:10.137446+0000","last_peered":"2024-02-23T00:35:10.137446+0000","last_clean":"2024-02-23T00:35:10.137446+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:35:10.137446+0000","last_undegraded":"2024-02-23T00:35:10.137446+0000","last_fullsized":"2024-02-23T00:35:10.137446+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:34:12.258344+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.258344+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":327,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.656963+0000","last_change":"2024-02-23T00:35:09.656923+0000","last_active":"2024-02-23T00:35:09.656963+0000","last_peered":"2024-02-23T00:35:09.656963+0000","last_clean":"2024-02-23T00:35:09.656963+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:35:09.656963+0000","last_undegraded":"2024-02-23T00:35:09.656963+0000","last_fullsized":"2024-02-23T00:35:09.656963+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:35:09.656838+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:35:09.656838+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":283,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109883+0000","last_change":"2024-02-23T00:34:41.105563+0000","last_active":"2024-02-23T00:35:09.109883+0000","last_peered":"2024-02-23T00:35:09.109883+0000","last_clean":"2024-02-23T00:35:09.109883+0000","last_became_active":"2024-02-23T00:34:41.105162+0000","last_became_peered":"2024-02-23T00:34:41.105162+0000","last_unstale":"2024-02-23T00:35:09.109883+0000","last_undegraded":"2024-02-23T00:35:09.109883+0000","last_fullsized":"2024-02-23T00:35:09.109883+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:27.425604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:27.425604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":46,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092214+0000","last_change":"2024-02-23T00:35:02.151821+0000","last_active":"2024-02-23T00:35:09.092214+0000","last_peered":"2024-02-23T00:35:09.092214+0000","last_clean":"2024-02-23T00:35:09.092214+0000","last_became_active":"2024-02-23T00:34:41.092835+0000","last_became_peered":"2024-02-23T00:34:41.092835+0000","last_unstale":"2024-02-23T00:35:09.092214+0000","last_undegraded":"2024-02-23T00:35:09.092214+0000","last_fullsized":"2024-02-23T00:35:09.092214+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.151731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":327,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092687+0000","last_change":"2024-02-23T00:34:25.593132+0000","last_active":"2024-02-23T00:35:09.092687+0000","last_peered":"2024-02-23T00:35:09.092687+0000","last_clean":"2024-02-23T00:35:09.092687+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:35:09.092687+0000","last_undegraded":"2024-02-23T00:35:09.092687+0000","last_fullsized":"2024-02-23T00:35:09.092687+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":319,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095534+0000","last_change":"2024-02-23T00:32:10.145767+0000","last_active":"2024-02-23T00:35:09.095534+0000","last_peered":"2024-02-23T00:35:09.095534+0000","last_clean":"2024-02-23T00:35:09.095534+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:35:09.095534+0000","last_undegraded":"2024-02-23T00:35:09.095534+0000","last_fullsized":"2024-02-23T00:35:09.095534+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:10.145685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:32:10.145685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":274,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110479+0000","last_change":"2024-02-23T00:32:58.681718+0000","last_active":"2024-02-23T00:35:09.110479+0000","last_peered":"2024-02-23T00:35:09.110479+0000","last_clean":"2024-02-23T00:35:09.110479+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:35:09.110479+0000","last_undegraded":"2024-02-23T00:35:09.110479+0000","last_fullsized":"2024-02-23T00:35:09.110479+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:58.681626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.681626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":331,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.093224+0000","last_change":"2024-02-23T00:33:33.425803+0000","last_active":"2024-02-23T00:35:09.093224+0000","last_peered":"2024-02-23T00:35:09.093224+0000","last_clean":"2024-02-23T00:35:09.093224+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:35:09.093224+0000","last_undegraded":"2024-02-23T00:35:09.093224+0000","last_fullsized":"2024-02-23T00:35:09.093224+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.425727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.425727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":318,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095789+0000","last_change":"2024-02-23T00:31:38.887671+0000","last_active":"2024-02-23T00:35:09.095789+0000","last_peered":"2024-02-23T00:35:09.095789+0000","last_clean":"2024-02-23T00:35:09.095789+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:35:09.095789+0000","last_undegraded":"2024-02-23T00:35:09.095789+0000","last_fullsized":"2024-02-23T00:35:09.095789+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:31:38.887597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":322,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090260+0000","last_change":"2024-02-23T00:34:57.661029+0000","last_active":"2024-02-23T00:35:09.090260+0000","last_peered":"2024-02-23T00:35:09.090260+0000","last_clean":"2024-02-23T00:35:09.090260+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:35:09.090260+0000","last_undegraded":"2024-02-23T00:35:09.090260+0000","last_fullsized":"2024-02-23T00:35:09.090260+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:57.660954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.660954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":282,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114871+0000","last_change":"2024-02-23T00:34:41.110964+0000","last_active":"2024-02-23T00:35:09.114871+0000","last_peered":"2024-02-23T00:35:09.114871+0000","last_clean":"2024-02-23T00:35:09.114871+0000","last_became_active":"2024-02-23T00:34:41.109162+0000","last_became_peered":"2024-02-23T00:34:41.109162+0000","last_unstale":"2024-02-23T00:35:09.114871+0000","last_undegraded":"2024-02-23T00:35:09.114871+0000","last_fullsized":"2024-02-23T00:35:09.114871+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:28.419374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:28.419374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":41,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109838+0000","last_change":"2024-02-23T00:34:41.103473+0000","last_active":"2024-02-23T00:35:09.109838+0000","last_peered":"2024-02-23T00:35:09.109838+0000","last_clean":"2024-02-23T00:35:09.109838+0000","last_became_active":"2024-02-23T00:34:41.103114+0000","last_became_peered":"2024-02-23T00:34:41.103114+0000","last_unstale":"2024-02-23T00:35:09.109838+0000","last_undegraded":"2024-02-23T00:35:09.109838+0000","last_fullsized":"2024-02-23T00:35:09.109838+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.750804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.750804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":333,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.090042+0000","last_change":"2024-02-23T00:34:56.679204+0000","last_active":"2024-02-23T00:35:09.090042+0000","last_peered":"2024-02-23T00:35:09.090042+0000","last_clean":"2024-02-23T00:35:09.090042+0000","last_became_active":"2024-02-23T00:34:41.106439+0000","last_became_peered":"2024-02-23T00:34:41.106439+0000","last_unstale":"2024-02-23T00:35:09.090042+0000","last_undegraded":"2024-02-23T00:35:09.090042+0000","last_fullsized":"2024-02-23T00:35:09.090042+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.679136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.679136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":46,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-02-23T00:35:10.167898+0000","last_change":"2024-02-23T00:35:10.167898+0000","last_active":"2024-02-23T00:35:10.167898+0000","last_peered":"2024-02-23T00:35:10.167898+0000","last_clean":"2024-02-23T00:35:10.167898+0000","last_became_active":"2024-02-23T00:34:41.113447+0000","last_became_peered":"2024-02-23T00:34:41.113447+0000","last_unstale":"2024-02-23T00:35:10.167898+0000","last_undegraded":"2024-02-23T00:35:10.167898+0000","last_fullsized":"2024-02-23T00:35:10.167898+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.167829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.167829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":273,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.115090+0000","last_change":"2024-02-23T00:33:33.382591+0000","last_active":"2024-02-23T00:35:09.115090+0000","last_peered":"2024-02-23T00:35:09.115090+0000","last_clean":"2024-02-23T00:35:09.115090+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:35:09.115090+0000","last_undegraded":"2024-02-23T00:35:09.115090+0000","last_fullsized":"2024-02-23T00:35:09.115090+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.382510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.382510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":38,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089800+0000","last_change":"2024-02-23T00:34:42.058800+0000","last_active":"2024-02-23T00:35:09.089800+0000","last_peered":"2024-02-23T00:35:09.089800+0000","last_clean":"2024-02-23T00:35:09.089800+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:35:09.089800+0000","last_undegraded":"2024-02-23T00:35:09.089800+0000","last_fullsized":"2024-02-23T00:35:09.089800+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:32.720048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:32.720048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":320,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.114354+0000","last_change":"2024-02-23T00:33:43.299920+0000","last_active":"2024-02-23T00:35:09.114354+0000","last_peered":"2024-02-23T00:35:09.114354+0000","last_clean":"2024-02-23T00:35:09.114354+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:35:09.114354+0000","last_undegraded":"2024-02-23T00:35:09.114354+0000","last_fullsized":"2024-02-23T00:35:09.114354+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":325,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091336+0000","last_change":"2024-02-23T00:34:19.572886+0000","last_active":"2024-02-23T00:35:09.091336+0000","last_peered":"2024-02-23T00:35:09.091336+0000","last_clean":"2024-02-23T00:35:09.091336+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:35:09.091336+0000","last_undegraded":"2024-02-23T00:35:09.091336+0000","last_fullsized":"2024-02-23T00:35:09.091336+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:19.572808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.572808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":279,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109104+0000","last_change":"2024-02-23T00:34:29.677110+0000","last_active":"2024-02-23T00:35:09.109104+0000","last_peered":"2024-02-23T00:35:09.109104+0000","last_clean":"2024-02-23T00:35:09.109104+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:35:09.109104+0000","last_undegraded":"2024-02-23T00:35:09.109104+0000","last_fullsized":"2024-02-23T00:35:09.109104+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:29.677037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:29.677037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":331,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.110074+0000","last_change":"2024-02-23T00:34:30.726107+0000","last_active":"2024-02-23T00:35:09.110074+0000","last_peered":"2024-02-23T00:35:09.110074+0000","last_clean":"2024-02-23T00:35:09.110074+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:35:09.110074+0000","last_undegraded":"2024-02-23T00:35:09.110074+0000","last_fullsized":"2024-02-23T00:35:09.110074+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":327,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109213+0000","last_change":"2024-02-23T00:34:31.754382+0000","last_active":"2024-02-23T00:35:09.109213+0000","last_peered":"2024-02-23T00:35:09.109213+0000","last_clean":"2024-02-23T00:35:09.109213+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:35:09.109213+0000","last_undegraded":"2024-02-23T00:35:09.109213+0000","last_fullsized":"2024-02-23T00:35:09.109213+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:31.754311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.754311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":324,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.113961+0000","last_change":"2024-02-23T00:35:03.519040+0000","last_active":"2024-02-23T00:35:09.113961+0000","last_peered":"2024-02-23T00:35:09.113961+0000","last_clean":"2024-02-23T00:35:09.113961+0000","last_became_active":"2024-02-23T00:34:41.112980+0000","last_became_peered":"2024-02-23T00:34:41.112980+0000","last_unstale":"2024-02-23T00:35:09.113961+0000","last_undegraded":"2024-02-23T00:35:09.113961+0000","last_fullsized":"2024-02-23T00:35:09.113961+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:31:39.494737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":281,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095997+0000","last_change":"2024-02-23T00:34:41.115172+0000","last_active":"2024-02-23T00:35:09.095997+0000","last_peered":"2024-02-23T00:35:09.095997+0000","last_clean":"2024-02-23T00:35:09.095997+0000","last_became_active":"2024-02-23T00:34:41.114758+0000","last_became_peered":"2024-02-23T00:34:41.114758+0000","last_unstale":"2024-02-23T00:35:09.095997+0000","last_undegraded":"2024-02-23T00:35:09.095997+0000","last_fullsized":"2024-02-23T00:35:09.095997+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.192483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.192483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":44,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092617+0000","last_change":"2024-02-23T00:34:54.388897+0000","last_active":"2024-02-23T00:35:09.092617+0000","last_peered":"2024-02-23T00:35:09.092617+0000","last_clean":"2024-02-23T00:35:09.092617+0000","last_became_active":"2024-02-23T00:34:41.115290+0000","last_became_peered":"2024-02-23T00:34:41.115290+0000","last_unstale":"2024-02-23T00:35:09.092617+0000","last_undegraded":"2024-02-23T00:35:09.092617+0000","last_fullsized":"2024-02-23T00:35:09.092617+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.388834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":316,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095720+0000","last_change":"2024-02-23T00:31:26.920222+0000","last_active":"2024-02-23T00:35:09.095720+0000","last_peered":"2024-02-23T00:35:09.095720+0000","last_clean":"2024-02-23T00:35:09.095720+0000","last_became_active":"2024-02-23T00:27:21.026406+0000","last_became_peered":"2024-02-23T00:27:21.026406+0000","last_unstale":"2024-02-23T00:35:09.095720+0000","last_undegraded":"2024-02-23T00:35:09.095720+0000","last_fullsized":"2024-02-23T00:35:09.095720+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:31:26.920133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:26.920133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":324,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089285+0000","last_change":"2024-02-23T00:34:23.597939+0000","last_active":"2024-02-23T00:35:09.089285+0000","last_peered":"2024-02-23T00:35:09.089285+0000","last_clean":"2024-02-23T00:35:09.089285+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:35:09.089285+0000","last_undegraded":"2024-02-23T00:35:09.089285+0000","last_fullsized":"2024-02-23T00:35:09.089285+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:23.597868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.597868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":36,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.091371+0000","last_change":"2024-02-23T00:34:42.056426+0000","last_active":"2024-02-23T00:35:09.091371+0000","last_peered":"2024-02-23T00:35:09.091371+0000","last_clean":"2024-02-23T00:35:09.091371+0000","last_became_active":"2024-02-23T00:34:42.056094+0000","last_became_peered":"2024-02-23T00:34:42.056094+0000","last_unstale":"2024-02-23T00:35:09.091371+0000","last_undegraded":"2024-02-23T00:35:09.091371+0000","last_fullsized":"2024-02-23T00:35:09.091371+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:29.354235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:29.354235+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":331,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.092642+0000","last_change":"2024-02-23T00:33:42.579086+0000","last_active":"2024-02-23T00:35:09.092642+0000","last_peered":"2024-02-23T00:35:09.092642+0000","last_clean":"2024-02-23T00:35:09.092642+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:35:09.092642+0000","last_undegraded":"2024-02-23T00:35:09.092642+0000","last_fullsized":"2024-02-23T00:35:09.092642+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":324,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.089140+0000","last_change":"2024-02-23T00:34:28.586621+0000","last_active":"2024-02-23T00:35:09.089140+0000","last_peered":"2024-02-23T00:35:09.089140+0000","last_clean":"2024-02-23T00:35:09.089140+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:35:09.089140+0000","last_undegraded":"2024-02-23T00:35:09.089140+0000","last_fullsized":"2024-02-23T00:35:09.089140+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.586556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.586556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":46,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.109862+0000","last_change":"2024-02-23T00:34:55.609318+0000","last_active":"2024-02-23T00:35:09.109862+0000","last_peered":"2024-02-23T00:35:09.109862+0000","last_clean":"2024-02-23T00:35:09.109862+0000","last_became_active":"2024-02-23T00:34:41.109129+0000","last_became_peered":"2024-02-23T00:34:41.109129+0000","last_unstale":"2024-02-23T00:35:09.109862+0000","last_undegraded":"2024-02-23T00:35:09.109862+0000","last_fullsized":"2024-02-23T00:35:09.109862+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:55.609248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:55.609248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":281,"reported_epoch":358,"state":"active+clean","last_fresh":"2024-02-23T00:35:09.095628+0000","last_change":"2024-02-23T00:34:41.099947+0000","last_active":"2024-02-23T00:35:09.095628+0000","last_peered":"2024-02-23T00:35:09.095628+0000","last_clean":"2024-02-23T00:35:09.095628+0000","last_became_active":"2024-02-23T00:34:41.099195+0000","last_became_peered":"2024-02-23T00:34:41.099195+0000","last_unstale":"2024-02-23T00:35:09.095628+0000","last_undegraded":"2024-02-23T00:35:09.095628+0000","last_fullsized":"2024-02-23T00:35:09.095628+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.396150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.396150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":202,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":200,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":4001388,"num_objects":144,"num_object_clones":2,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":101,"num_read_kb":7,"num_write":569,"num_write_kb":4470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":4000297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13750272,"data_stored":12004164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1817,"ondisk_log_size":1817,"up":96,"acting":96,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":24,"stat_sum":{"num_bytes":3592,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":0,"num_read":1044,"num_read_kb":776,"num_write":593,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":2668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":332,"ondisk_log_size":332,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":486,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":212,"num_read_kb":211,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332194,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305016,"kb_used_data":7792,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440136,"statfs":{"total":95995035648,"available":95682699264,"internally_reserved":0,"allocated":7979008,"data_stored":6007221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":2.0830000000000002,"15min":1.028},"min":{"1min":0.27100000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.85899999999999999,"5min":61.048999999999999,"15min":61.048999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":2.698,"15min":1.2509999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.214},"max":{"1min":4.7539999999999996,"5min":73.870999999999995,"15min":73.870999999999995},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.0669999999999999,"15min":0.70799999999999996},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":4.6840000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":1.077,"15min":0.69399999999999995},"min":{"1min":0.311,"5min":0.249,"15min":0.249},"max":{"1min":3.8719999999999999,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Feb 23 00:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.98399999999999999,"15min":0.67800000000000005},"min":{"1min":0.314,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.7240000000000002,"5min":16.603999999999999,"15min":16.603999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.79000000000000004,"15min":0.60999999999999999},"min":{"1min":0.34899999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.1369999999999996,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.498}]},{"osd":3,"last update":"Fri Feb 23 00:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.911,"15min":0.98499999999999999},"min":{"1min":0.34999999999999998,"5min":0.316,"15min":0.311},"max":{"1min":3.7679999999999998,"5min":57.729999999999997,"15min":57.729999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.641,"15min":0.90300000000000002},"min":{"1min":0.40600000000000003,"5min":0.31900000000000001,"15min":0.312},"max":{"1min":3.8420000000000001,"5min":27.573,"15min":27.573},"last":0.58599999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.438,"15min":0.42099999999999999},"min":{"1min":0.29899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.2310000000000001,"5min":1.706,"15min":1.706},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.46999999999999997,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.25700000000000001,"15min":0.254},"max":{"1min":0.70199999999999996,"5min":3.294,"15min":3.294},"last":0.55700000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.44700000000000001,"15min":0.434},"min":{"1min":0.315,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70199999999999996,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.436},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48499999999999999,"15min":0.44900000000000001},"min":{"1min":0.32900000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.1919999999999999,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.48599999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51000000000000001,"15min":0.48499999999999999},"min":{"1min":0.34599999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.26,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.5,"15min":0.46999999999999997},"min":{"1min":0.35399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.73099999999999998,"5min":1.002,"15min":1.002},"last":0.60899999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561127,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305248,"kb_used_data":3928,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93439904,"statfs":{"total":95995035648,"available":95682461696,"internally_reserved":0,"allocated":4022272,"data_stored":2016469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":1.6919999999999999,"15min":0.89500000000000002},"min":{"1min":0.26500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.71299999999999997,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.70999999999999996,"15min":0.57999999999999996},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.74199999999999999,"5min":9.468,"15min":9.468},"last":0.55100000000000005}]},{"osd":1,"last update":"Fri Feb 23 00:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":2.0329999999999999,"15min":1.012},"min":{"1min":0.253,"5min":0.245,"15min":0.245},"max":{"1min":0.78900000000000003,"5min":59.347999999999999,"15min":59.347999999999999},"last":0.501},{"interface":"front","average":{"1min":0.52300000000000002,"5min":2.875,"15min":1.29},"min":{"1min":0.30399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.875,"5min":87.221000000000004,"15min":87.221000000000004},"last":0.42999999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.427,"15min":0.80900000000000005},"min":{"1min":0.33000000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.022,"5min":25.550999999999998,"15min":25.550999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":1.524,"15min":0.86599999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.73799999999999999,"5min":40.914000000000001,"15min":40.914000000000001},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":1.385,"15min":0.80200000000000005},"min":{"1min":0.39400000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.82599999999999996,"5min":31.309999999999999,"15min":31.309999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":2.9209999999999998,"15min":1.325},"min":{"1min":0.40899999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.145,"5min":86.311999999999998,"15min":86.311999999999998},"last":0.622}]},{"osd":4,"last update":"Fri Feb 23 00:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45100000000000001,"15min":0.42899999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63900000000000001,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.66000000000000003,"15min":0.499},"min":{"1min":0.27500000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63600000000000001,"5min":17.718,"15min":17.718},"last":0.68799999999999994}]},{"osd":5,"last update":"Fri Feb 23 00:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.56399999999999995,"15min":0.47899999999999998},"min":{"1min":0.27400000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67600000000000005,"5min":8.0990000000000002,"15min":8.0990000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.54800000000000004,"15min":0.47099999999999997},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.75700000000000001,"5min":8.1370000000000005,"15min":8.1370000000000005},"last":0.72099999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.84299999999999997,"15min":0.59499999999999997},"min":{"1min":0.32500000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71299999999999997,"5min":17.881,"15min":17.881},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.81599999999999995,"15min":0.59099999999999997},"min":{"1min":0.315,"5min":0.28699999999999998,"15min":0.28199999999999997},"max":{"1min":0.747,"5min":17.396999999999998,"15min":17.396999999999998},"last":0.67100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574974,"num_pgs":110,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309300,"kb_used_data":7980,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93435852,"statfs":{"total":95995035648,"available":95678312448,"internally_reserved":0,"allocated":8171520,"data_stored":6009159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":1.3160000000000001,"15min":0.66300000000000003},"min":{"1min":0.22800000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.53600000000000003,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.94199999999999995,"15min":0.55100000000000005},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.67200000000000004,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":1.2969999999999999,"15min":0.67900000000000005},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.84099999999999997,"5min":35.313000000000002,"15min":35.313000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.99099999999999999,"15min":0.58099999999999996},"min":{"1min":0.23100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.51900000000000002,"5min":27.908999999999999,"15min":27.908999999999999},"last":0.40100000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":1.149,"15min":0.65900000000000003},"min":{"1min":0.23000000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.57299999999999995,"5min":21.013999999999999,"15min":21.013999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":1.115,"15min":0.65600000000000003},"min":{"1min":0.28799999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.71199999999999997,"5min":29.234999999999999,"15min":29.234999999999999},"last":0.28799999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.98799999999999999,"15min":0.69999999999999996},"min":{"1min":0.34399999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80300000000000005,"5min":16.231000000000002,"15min":16.231000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":1.079,"15min":0.72599999999999998},"min":{"1min":0.34799999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79000000000000004,"5min":29.212,"15min":29.212},"last":0.66900000000000004}]},{"osd":5,"last update":"Fri Feb 23 00:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":1.0449999999999999,"15min":0.72999999999999998},"min":{"1min":0.34699999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.80900000000000005,"5min":16.294,"15min":16.294},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.89900000000000002,"15min":0.69099999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72499999999999998,"5min":11.105,"15min":11.105},"last":0.78600000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.159,"15min":0.76700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82899999999999996,"5min":29.187000000000001,"15min":29.187000000000001},"last":0.625},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.046,"15min":0.75800000000000001},"min":{"1min":0.38300000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.74199999999999999,"5min":16.369,"15min":16.369},"last":0.73199999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":1.155,"15min":0.76700000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39800000000000002},"max":{"1min":0.755,"5min":16.395,"15min":16.395},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.80700000000000005,"15min":0.67400000000000004},"min":{"1min":0.36299999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70299999999999996,"5min":11.135999999999999,"15min":11.135999999999999},"last":0.85899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313428,"kb_used_data":3916,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431724,"statfs":{"total":95995035648,"available":95674085376,"internally_reserved":0,"allocated":4009984,"data_stored":2016023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:35:06 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.81999999999999995,"15min":0.49399999999999999},"min":{"1min":0.20200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.76600000000000001,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":1.1779999999999999,"15min":0.61899999999999999},"min":{"1min":0.26800000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.79600000000000004,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.32300000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:34:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":1.0009999999999999,"15min":0.59299999999999997},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.215},"max":{"1min":1.0549999999999999,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.752,"15min":0.49199999999999999},"min":{"1min":0.23300000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.63700000000000001,"5min":15.965,"15min":15.965},"last":0.30499999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":1.1000000000000001,"15min":0.65600000000000003},"min":{"1min":0.26700000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60799999999999998,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.442,"5min":1.0569999999999999,"15min":0.64300000000000002},"min":{"1min":0.27600000000000002,"5min":0.245,"15min":0.245},"max":{"1min":1.1359999999999999,"5min":16.006,"15min":16.006},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.82699999999999996,"15min":0.64300000000000002},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71199999999999997,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.177,"15min":0.76400000000000001},"min":{"1min":0.41099999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68500000000000005,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:34:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":1.0860000000000001,"15min":0.72899999999999998},"min":{"1min":0.41499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69499999999999995,"5min":13.311,"15min":13.311},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.1279999999999999,"15min":0.746},"min":{"1min":0.40799999999999997,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":1.1020000000000001,"5min":23.744,"15min":23.744},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:35:06 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":1.3540000000000001,"15min":0.80200000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66300000000000003,"5min":33.082999999999998,"15min":33.082999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.98999999999999999,"15min":0.70799999999999996},"min":{"1min":0.38900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71199999999999997,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.66300000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.2310000000000001,"15min":0.78300000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.373},"max":{"1min":0.746,"5min":31.251999999999999,"15min":31.251999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.151,"15min":0.75800000000000001},"min":{"1min":0.42199999999999999,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.72899999999999998,"5min":23.785,"15min":23.785},"last":0.52000000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411450,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317304,"kb_used_data":7792,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93427848,"statfs":{"total":95995035648,"available":95670116352,"internally_reserved":0,"allocated":7979008,"data_stored":6005991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":1.5920000000000001,"15min":0.751},"min":{"1min":0.20599999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.59699999999999998,"5min":56.509,"15min":56.509},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":1.4590000000000001,"15min":0.70799999999999996},"min":{"1min":0.246,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54800000000000004,"5min":31.498999999999999,"15min":31.498999999999999},"last":0.31}]},{"osd":1,"last update":"Fri Feb 23 00:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.996,"15min":0.56100000000000005},"min":{"1min":0.22900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.53600000000000003,"5min":19.324999999999999,"15min":19.324999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":1.4159999999999999,"15min":0.69999999999999996},"min":{"1min":0.23599999999999999,"5min":0.22900000000000001,"15min":0.216},"max":{"1min":0.56000000000000005,"5min":35.052,"15min":35.052},"last":0.29599999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":1.274,"15min":0.69399999999999995},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.83599999999999997,"5min":43.5,"15min":43.5},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":1.054,"15min":0.63900000000000001},"min":{"1min":0.27800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":1.02,"5min":27.414000000000001,"15min":27.414000000000001},"last":0.33500000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:34:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.78300000000000003,"15min":0.62},"min":{"1min":0.35899999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0939999999999999,"5min":14.614000000000001,"15min":14.614000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68500000000000005,"15min":0.59699999999999998},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":3.903,"5min":4.6550000000000002,"15min":4.6550000000000002},"last":0.52200000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:34:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.89900000000000002,"15min":0.66400000000000003},"min":{"1min":0.39900000000000002,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":0.79500000000000004,"5min":17.765000000000001,"15min":17.765000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.79600000000000004,"15min":0.63500000000000001},"min":{"1min":0.39400000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.86099999999999999,"5min":14.589,"15min":14.589},"last":0.495}]},{"osd":6,"last update":"Fri Feb 23 00:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.70199999999999996,"15min":0.60199999999999998},"min":{"1min":0.38500000000000001,"5min":0.32700000000000001,"15min":0.317},"max":{"1min":0.86799999999999999,"5min":4.6120000000000001,"15min":4.6120000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.071,"15min":0.749},"min":{"1min":0.38500000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82999999999999996,"5min":20.574999999999999,"15min":20.574999999999999},"last":0.54400000000000004}]},{"osd":7,"last update":"Fri Feb 23 00:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.9119999999999999,"15min":1.03},"min":{"1min":0.40799999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77000000000000002,"5min":66.632000000000005,"15min":66.632000000000005},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.2290000000000001,"15min":0.80400000000000005},"min":{"1min":0.38500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":19.603999999999999,"15min":19.603999999999999},"last":0.53000000000000003}]}]},{"osd":3,"up_from":0,"seq":111669149814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986289,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305296,"kb_used_data":3976,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93439856,"statfs":{"total":95995035648,"available":95682412544,"internally_reserved":0,"allocated":4071424,"data_stored":2006405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":1.4770000000000001,"15min":0.83199999999999996},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.622,"5min":21.419,"15min":21.419},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":2.7269999999999999,"15min":1.26},"min":{"1min":0.35199999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":1.956,"15min":0.99199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.63500000000000001,"5min":41.137999999999998,"15min":41.137999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":1.478,"15min":0.83599999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.747,"5min":26.210000000000001,"15min":26.210000000000001},"last":0.64000000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":2.1840000000000002,"15min":1.079},"min":{"1min":0.32300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70099999999999996,"5min":67.253,"15min":67.253},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.624,"5min":1.9450000000000001,"15min":0.999},"min":{"1min":0.38900000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.391,"5min":41.927999999999997,"15min":41.927999999999997},"last":0.496}]},{"osd":3,"last update":"Fri Feb 23 00:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":2.0569999999999999,"15min":1.036},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.34000000000000002},"max":{"1min":0.79400000000000004,"5min":36.301000000000002,"15min":36.301000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.441,"5min":2.2789999999999999,"15min":1.1080000000000001},"min":{"1min":0.307,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.68200000000000005,"5min":42.064,"15min":42.064},"last":0.52500000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:34:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.86599999999999999,"15min":0.53700000000000003},"min":{"1min":0.253,"5min":0.217,"15min":0.217},"max":{"1min":0.74299999999999999,"5min":14.242000000000001,"15min":14.242000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.83599999999999997,"15min":0.55000000000000004},"min":{"1min":0.32400000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.84299999999999997,"5min":14.124000000000001,"15min":14.124000000000001},"last":0.56699999999999995}]},{"osd":6,"last update":"Fri Feb 23 00:35:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.69099999999999995,"15min":0.53700000000000003},"min":{"1min":0.317,"5min":0.254,"15min":0.254},"max":{"1min":0.74199999999999999,"5min":8.0730000000000004,"15min":8.0730000000000004},"last":0.624},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.99399999999999999,"15min":0.624},"min":{"1min":0.27900000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67300000000000004,"5min":45.814999999999998,"15min":45.814999999999998},"last":0.45800000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.85499999999999998,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82999999999999996,"5min":17.178999999999998,"15min":17.178999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.89600000000000002,"15min":0.63700000000000001},"min":{"1min":0.35599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.88100000000000001,"5min":14.212,"15min":14.212},"last":0.60399999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757356,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301236,"kb_used_data":4012,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93443916,"statfs":{"total":95995035648,"available":95686569984,"internally_reserved":0,"allocated":4108288,"data_stored":2016234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":1.111,"15min":0.70599999999999996},"min":{"1min":0.33600000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.65100000000000002,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.45100000000000001,"5min":2.097,"15min":1.0249999999999999},"min":{"1min":0.35399999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60399999999999998,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.498}]},{"osd":1,"last update":"Fri Feb 23 00:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.286,"15min":0.748},"min":{"1min":0.35699999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.61799999999999999,"5min":38.838000000000001,"15min":38.838000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.44600000000000001,"5min":2.5139999999999998,"15min":1.1679999999999999},"min":{"1min":0.32200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.56799999999999995,"5min":127.794,"15min":127.794},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":1.6479999999999999,"15min":0.88400000000000001},"min":{"1min":0.36899999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.60799999999999998,"5min":37.442999999999998,"15min":37.442999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":1.0700000000000001,"15min":0.69599999999999995},"min":{"1min":0.32700000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.2490000000000001,"5min":23.297999999999998,"15min":23.297999999999998},"last":0.443}]},{"osd":3,"last update":"Fri Feb 23 00:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.23,"15min":0.73399999999999999},"min":{"1min":0.35599999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63800000000000001,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.48899999999999999,"5min":1.621,"15min":0.88800000000000001},"min":{"1min":0.38,"5min":0.31,"15min":0.31},"max":{"1min":0.57499999999999996,"5min":40.865000000000002,"15min":40.865000000000002},"last":0.50800000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":1.5609999999999999,"15min":0.77000000000000002},"min":{"1min":0.26300000000000001,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":0.55400000000000005,"5min":103.13,"15min":103.13},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.58599999999999997,"15min":0.44700000000000001},"min":{"1min":0.27900000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.58399999999999996,"5min":12.795999999999999,"15min":12.795999999999999},"last":0.33600000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.56599999999999995,"15min":0.48299999999999998},"min":{"1min":0.313,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64400000000000002,"5min":7.1440000000000001,"15min":7.1440000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":1.5580000000000001,"15min":0.81499999999999995},"min":{"1min":0.315,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.55500000000000005,"5min":103.61499999999999,"15min":103.61499999999999},"last":0.46800000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:34:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.58799999999999997,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.75,"5min":7.1820000000000004,"15min":7.1820000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52500000000000002,"15min":0.51200000000000001},"min":{"1min":0.34300000000000003,"5min":0.317,"15min":0.317},"max":{"1min":0.68000000000000005,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4333568,"data_stored":4000789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4214784,"data_stored":4000535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4190208,"data_stored":4000449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":200,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":200,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":200,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":200,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":200,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":200,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":200,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:35:15.751 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-23T00:35:15.752 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.control pg_num 42 2024-02-23T00:35:15.827 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:15.856 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:15.909 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:15.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:16.013 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:16.040 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:16.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:15.067471+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]': finished 2024-02-23T00:35:16.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:15.067538+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-02-23T00:35:16.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:15.069851+0000 mon.c (mon.2) 403 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:16.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:15.070126+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:16.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:15.634936+0000 mon.a (mon.0) 2381 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:35:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:15.067471+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]': finished 2024-02-23T00:35:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:15.067538+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-02-23T00:35:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:15.069851+0000 mon.c (mon.2) 403 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:16.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:15.070126+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:16.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:15.634936+0000 mon.a (mon.0) 2381 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:35:16.388 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:16.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:15.067471+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi087-82019-62"}]': finished 2024-02-23T00:35:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:15.067538+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-02-23T00:35:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:15.069851+0000 mon.c (mon.2) 403 : audit [INF] from='client.? 172.21.15.87:0/1836284438' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:15.070126+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]: dispatch 2024-02-23T00:35:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:15.634936+0000 mon.a (mon.0) 2381 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:35:16.491 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:16.524 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:16.545 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:16.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:16.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:16.650 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:16.681 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:16.702 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:16.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:16.805 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:16.833 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:16.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 -- 172.21.15.87:0/254747435 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d7c0f7400 msgr2=0x7f9d7c103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:16.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 --2- 172.21.15.87:0/254747435 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d7c0f7400 0x7f9d7c103a20 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 rx=0x7f9d740047b0 tx=0x7f9d74009cb0).stop 2024-02-23T00:35:16.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 -- 172.21.15.87:0/254747435 shutdown_connections 2024-02-23T00:35:16.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 --2- 172.21.15.87:0/254747435 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d7c0f7400 0x7f9d7c103a20 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:16.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 --2- 172.21.15.87:0/254747435 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d7c0fcbb0 0x7f9d7c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:16.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 --2- 172.21.15.87:0/254747435 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d7c0fd510 0x7f9d7c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:16.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 -- 172.21.15.87:0/254747435 >> 172.21.15.87:0/254747435 conn(0x7f9d7c0f2820 msgr2=0x7f9d7c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 -- 172.21.15.87:0/254747435 shutdown_connections 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 -- 172.21.15.87:0/254747435 wait complete. 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 librados: starting msgr at 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.885+0000 7f9d816a0700 1 librados: starting objecter 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 Processor -- start 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 -- start start 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 librados: setting wanted keys 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 librados: calling monclient init 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d7c0f7400 0x7f9d7c108fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d7c0fcbb0 0x7f9d7c1094d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d7c0fd510 0x7f9d7c109a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c1141d0 con 0x7f9d7c0f7400 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c114340 con 0x7f9d7c0fcbb0 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d816a0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c114500 con 0x7f9d7c0fd510 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d7affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d7c0f7400 0x7f9d7c108fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d7affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d7c0f7400 0x7f9d7c108fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45794/0 (socket says 172.21.15.87:45794) 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d7b7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d7c0fd510 0x7f9d7c109a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d7affd700 1 -- 172.21.15.87:0/408714771 learned_addr learned my addr 172.21.15.87:0/408714771 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.886+0000 7f9d7a7fc700 1 --2- 172.21.15.87:0/408714771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d7c0fcbb0 0x7f9d7c1094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d7affd700 1 -- 172.21.15.87:0/408714771 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d7c0fd510 msgr2=0x7f9d7c109a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d7affd700 1 --2- 172.21.15.87:0/408714771 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d7c0fd510 0x7f9d7c109a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d7affd700 1 -- 172.21.15.87:0/408714771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d7c0fcbb0 msgr2=0x7f9d7c1094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:16.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d7affd700 1 --2- 172.21.15.87:0/408714771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d7c0fcbb0 0x7f9d7c1094d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d7affd700 1 -- 172.21.15.87:0/408714771 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c0097c0 con 0x7f9d7c0f7400 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d7affd700 1 --2- 172.21.15.87:0/408714771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d7c0f7400 0x7f9d7c108fe0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7f9d6c001e70 tx=0x7f9d6c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d73fff700 10 client.?.objecter ms_handle_connect 0x7f9d7c0f7400 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d73fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0099f0 con 0x7f9d7c0f7400 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d74005040 con 0x7f9d7c0f7400 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d6c0090d0 con 0x7f9d7c0f7400 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c007c60 con 0x7f9d7c0f7400 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d816a0700 10 client.27123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c114b90 con 0x7f9d7c0f7400 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.887+0000 7f9d816a0700 1 librados: init done 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.888+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9d7c05fa60 con 0x7f9d7c0f7400 2024-02-23T00:35:16.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.888+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f9d6c009240 con 0x7f9d7c0f7400 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d73fff700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d68062c90 0x7f9d68065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(365..365 src has 173..365) v4 ==== 8089+0+0 (secure 0 0 0) 0x7f9d6c07dfc0 con 0x7f9d7c0f7400 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 osd_map(365..365 src has 173..365) v4 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d73fff700 3 client.27123.objecter handle_osd_map got epochs [365,365] > 0 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d73fff700 3 client.27123.objecter handle_osd_map decoding full epoch 365 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d7a7fc700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d68062c90 0x7f9d68065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d73fff700 20 client.27123.objecter dump_active .. 0 homeless 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v365) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9d6c07e4a0 con 0x7f9d7c0f7400 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d73fff700 10 client.27123.objecter ms_handle_connect 0x7f9d68062c90 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d7a7fc700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d68062c90 0x7f9d68065150 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7f9d7c061720 tx=0x7f9d6400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 10 client.27123.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:16.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 1 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6801/2153191863 conn(0x7f9d7c0fdea0 0x7f9d7c0fe280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c05ee80 osd=0 3 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c05ee80 osd=0 3 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c05ee80 osd=0 3 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 1 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c05ee80 osd=0 3 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 0 1 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c05ee80 osd=0 4 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 10 client.27123.objecter _send_command 1 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7c1060e0 con 0x7f9d7c0fdea0 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d7b7fe700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6801/2153191863 conn(0x7f9d7c0fdea0 0x7f9d7c0fe280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d73fff700 10 client.27123.objecter ms_handle_connect 0x7f9d7c0fdea0 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.889+0000 7f9d7b7fe700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6801/2153191863 conn(0x7f9d7c0fdea0 0x7f9d7c0fe280 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:16.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.890+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7c1060e0 con 0x7f9d7c0fdea0 2024-02-23T00:35:16.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.890+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:35:16.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.890+0000 7f9d73fff700 10 client.27123.objecter _finish_command 1 = system:0 2024-02-23T00:35:16.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.890+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c05ee80 osd=0 4 2024-02-23T00:35:16.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.890+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 0 1 2024-02-23T00:35:16.906 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:16.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 10 client.27123.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 2 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c05ee80 osd=0 3 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c05ee80 osd=0 3 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c05ee80 osd=0 3 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 2 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c05ee80 osd=0 3 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 0 2 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c05ee80 osd=0 4 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 10 client.27123.objecter _send_command 2 2024-02-23T00:35:16.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.905+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7c0fe9b0 con 0x7f9d7c0fdea0 2024-02-23T00:35:16.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d74019040 con 0x7f9d7c0fdea0 2024-02-23T00:35:16.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:16.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d73fff700 10 client.27123.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:16.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c05ee80 osd=0 4 2024-02-23T00:35:16.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 0 2 2024-02-23T00:35:16.909 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:35:16.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 10 client.27123.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:16.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 3 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d7c0598f0 0x7f9d7c059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c028540 osd=1 3 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c028540 osd=1 3 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c028540 osd=1 3 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 3 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c028540 osd=1 3 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 1 3 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c028540 osd=1 4 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 10 client.27123.objecter _send_command 3 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7c0286b0 con 0x7f9d7c0598f0 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d7affd700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d7c0598f0 0x7f9d7c059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d73fff700 10 client.27123.objecter ms_handle_connect 0x7f9d7c0598f0 2024-02-23T00:35:16.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.907+0000 7f9d7affd700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d7c0598f0 0x7f9d7c059cf0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:16.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.908+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7c0286b0 con 0x7f9d7c0598f0 2024-02-23T00:35:16.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.908+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:35:16.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.908+0000 7f9d73fff700 10 client.27123.objecter _finish_command 3 = system:0 2024-02-23T00:35:16.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.908+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c028540 osd=1 4 2024-02-23T00:35:16.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.908+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 1 3 2024-02-23T00:35:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:16.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:16.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 10 client.27123.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:16.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 4 2024-02-23T00:35:16.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c028540 osd=1 3 2024-02-23T00:35:16.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c028540 osd=1 3 2024-02-23T00:35:16.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c028540 osd=1 3 2024-02-23T00:35:16.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 4 2024-02-23T00:35:16.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c028540 osd=1 3 2024-02-23T00:35:16.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 1 4 2024-02-23T00:35:16.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c028540 osd=1 4 2024-02-23T00:35:16.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 10 client.27123.objecter _send_command 4 2024-02-23T00:35:16.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.923+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7c059360 con 0x7f9d7c0598f0 2024-02-23T00:35:16.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7c059360 con 0x7f9d7c0598f0 2024-02-23T00:35:16.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:16.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d73fff700 10 client.27123.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:16.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c028540 osd=1 4 2024-02-23T00:35:16.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 1 4 2024-02-23T00:35:16.926 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:35:16.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 10 client.27123.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:16.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 5 2024-02-23T00:35:16.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d7c0280a0 0x7f9d7c11bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c059770 osd=2 3 2024-02-23T00:35:16.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c059770 osd=2 3 2024-02-23T00:35:16.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c059770 osd=2 3 2024-02-23T00:35:16.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 5 2024-02-23T00:35:16.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c059770 osd=2 3 2024-02-23T00:35:16.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 2 5 2024-02-23T00:35:16.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c059770 osd=2 4 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 10 client.27123.objecter _send_command 5 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7c11ec10 con 0x7f9d7c0280a0 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d7a7fc700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d7c0280a0 0x7f9d7c11bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d73fff700 10 client.27123.objecter ms_handle_connect 0x7f9d7c0280a0 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.925+0000 7f9d7a7fc700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d7c0280a0 0x7f9d7c11bfc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.926+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7c11ec10 con 0x7f9d7c0280a0 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.926+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.926+0000 7f9d73fff700 10 client.27123.objecter _finish_command 5 = system:0 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.926+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c059770 osd=2 4 2024-02-23T00:35:16.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.926+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 2 5 2024-02-23T00:35:16.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 10 client.27123.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:16.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 6 2024-02-23T00:35:16.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c059770 osd=2 3 2024-02-23T00:35:16.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c059770 osd=2 3 2024-02-23T00:35:16.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c059770 osd=2 3 2024-02-23T00:35:16.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 6 2024-02-23T00:35:16.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c059770 osd=2 3 2024-02-23T00:35:16.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 2 6 2024-02-23T00:35:16.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c059770 osd=2 4 2024-02-23T00:35:16.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 10 client.27123.objecter _send_command 6 2024-02-23T00:35:16.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.940+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7c027600 con 0x7f9d7c0280a0 2024-02-23T00:35:16.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7c027600 con 0x7f9d7c0280a0 2024-02-23T00:35:16.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:16.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d73fff700 10 client.27123.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:16.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c059770 osd=2 4 2024-02-23T00:35:16.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 2 6 2024-02-23T00:35:16.943 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:35:16.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 10 client.27123.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:16.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 7 2024-02-23T00:35:16.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7c119b10 0x7f9d7c11d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c027a10 osd=3 3 2024-02-23T00:35:16.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c027a10 osd=3 3 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c027a10 osd=3 3 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 7 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c027a10 osd=3 3 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 3 7 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c027a10 osd=3 4 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 10 client.27123.objecter _send_command 7 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7c11dc80 con 0x7f9d7c119b10 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d7b7fe700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7c119b10 0x7f9d7c11d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d73fff700 10 client.27123.objecter ms_handle_connect 0x7f9d7c119b10 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.942+0000 7f9d7b7fe700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7c119b10 0x7f9d7c11d630 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.943+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7c11dc80 con 0x7f9d7c119b10 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.943+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.943+0000 7f9d73fff700 10 client.27123.objecter _finish_command 7 = system:0 2024-02-23T00:35:16.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.943+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c027a10 osd=3 4 2024-02-23T00:35:16.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.943+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 3 7 2024-02-23T00:35:16.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 10 client.27123.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:16.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 8 2024-02-23T00:35:16.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c027a10 osd=3 3 2024-02-23T00:35:16.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c027a10 osd=3 3 2024-02-23T00:35:16.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c027a10 osd=3 3 2024-02-23T00:35:16.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 8 2024-02-23T00:35:16.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c027a10 osd=3 3 2024-02-23T00:35:16.958 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 3 8 2024-02-23T00:35:16.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c027a10 osd=3 4 2024-02-23T00:35:16.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 10 client.27123.objecter _send_command 8 2024-02-23T00:35:16.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.957+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7c119070 con 0x7f9d7c119b10 2024-02-23T00:35:16.959 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.958+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7c119070 con 0x7f9d7c119b10 2024-02-23T00:35:16.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.958+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:16.960 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:35:16.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.958+0000 7f9d73fff700 10 client.27123.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.958+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c027a10 osd=3 4 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.958+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 3 8 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 10 client.27123.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 9 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7c1191e0 0x7f9d7c122da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c119070 osd=4 3 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c119070 osd=4 3 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c119070 osd=4 3 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 9 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c119070 osd=4 3 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 4 9 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c119070 osd=4 4 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 10 client.27123.objecter _send_command 9 2024-02-23T00:35:16.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7c125ac0 con 0x7f9d7c1191e0 2024-02-23T00:35:16.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d7affd700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7c1191e0 0x7f9d7c122da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d73fff700 10 client.27123.objecter ms_handle_connect 0x7f9d7c1191e0 2024-02-23T00:35:16.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.959+0000 7f9d7affd700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7c1191e0 0x7f9d7c122da0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:16.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.960+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7c125ac0 con 0x7f9d7c1191e0 2024-02-23T00:35:16.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.960+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:35:16.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.960+0000 7f9d73fff700 10 client.27123.objecter _finish_command 9 = system:0 2024-02-23T00:35:16.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.960+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c119070 osd=4 4 2024-02-23T00:35:16.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.960+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 4 9 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 10 client.27123.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 10 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c119070 osd=4 3 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c119070 osd=4 3 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c119070 osd=4 3 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 10 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c119070 osd=4 3 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 4 10 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c119070 osd=4 4 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 10 client.27123.objecter _send_command 10 2024-02-23T00:35:16.976 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.975+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7c125ac0 con 0x7f9d7c1191e0 2024-02-23T00:35:16.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.976+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7c125ac0 con 0x7f9d7c1191e0 2024-02-23T00:35:16.977 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.976+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:16.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.976+0000 7f9d73fff700 10 client.27123.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:16.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.976+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c119070 osd=4 4 2024-02-23T00:35:16.978 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.976+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 4 10 2024-02-23T00:35:16.978 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 10 client.27123.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 11 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d7c123880 0x7f9d7c123ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c125ac0 osd=5 3 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125ac0 osd=5 3 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c125ac0 osd=5 3 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 11 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c125ac0 osd=5 3 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 5 11 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125ac0 osd=5 4 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 10 client.27123.objecter _send_command 11 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7c128e70 con 0x7f9d7c123880 2024-02-23T00:35:16.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d7a7fc700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d7c123880 0x7f9d7c123ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d7a7fc700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d7c123880 0x7f9d7c123ce0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:16.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.977+0000 7f9d73fff700 10 client.27123.objecter ms_handle_connect 0x7f9d7c123880 2024-02-23T00:35:16.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.978+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7c128e70 con 0x7f9d7c123880 2024-02-23T00:35:16.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.978+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:35:16.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.978+0000 7f9d73fff700 10 client.27123.objecter _finish_command 11 = system:0 2024-02-23T00:35:16.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.978+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c125ac0 osd=5 4 2024-02-23T00:35:16.980 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.978+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 5 11 2024-02-23T00:35:16.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 10 client.27123.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:16.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 12 2024-02-23T00:35:16.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c125ac0 osd=5 3 2024-02-23T00:35:16.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125ac0 osd=5 3 2024-02-23T00:35:16.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c125ac0 osd=5 3 2024-02-23T00:35:16.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 12 2024-02-23T00:35:16.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c125ac0 osd=5 3 2024-02-23T00:35:16.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 5 12 2024-02-23T00:35:16.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125ac0 osd=5 4 2024-02-23T00:35:16.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 10 client.27123.objecter _send_command 12 2024-02-23T00:35:16.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.993+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7c1242c0 con 0x7f9d7c123880 2024-02-23T00:35:16.996 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7c1242c0 con 0x7f9d7c123880 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d73fff700 10 client.27123.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c125ac0 osd=5 4 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 5 12 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 10 client.27123.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 13 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d7c124940 0x7f9d7c124d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c1242c0 osd=6 3 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c1242c0 osd=6 3 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c1242c0 osd=6 3 2024-02-23T00:35:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 13 2024-02-23T00:35:16.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c1242c0 osd=6 3 2024-02-23T00:35:16.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 6 13 2024-02-23T00:35:16.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c1242c0 osd=6 4 2024-02-23T00:35:16.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 10 client.27123.objecter _send_command 13 2024-02-23T00:35:16.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.995+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7c125370 con 0x7f9d7c124940 2024-02-23T00:35:16.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.996+0000 7f9d7b7fe700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d7c124940 0x7f9d7c124d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:16.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.996+0000 7f9d73fff700 10 client.27123.objecter ms_handle_connect 0x7f9d7c124940 2024-02-23T00:35:16.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.996+0000 7f9d7b7fe700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d7c124940 0x7f9d7c124d80 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:16.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.997+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7c125370 con 0x7f9d7c124940 2024-02-23T00:35:16.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.997+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:35:16.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.997+0000 7f9d73fff700 10 client.27123.objecter _finish_command 13 = system:0 2024-02-23T00:35:16.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.997+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c1242c0 osd=6 4 2024-02-23T00:35:16.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:16.997+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 6 13 2024-02-23T00:35:17.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 10 client.27123.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:17.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 14 2024-02-23T00:35:17.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c1242c0 osd=6 3 2024-02-23T00:35:17.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c1242c0 osd=6 3 2024-02-23T00:35:17.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c1242c0 osd=6 3 2024-02-23T00:35:17.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 14 2024-02-23T00:35:17.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c1242c0 osd=6 3 2024-02-23T00:35:17.012 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 6 14 2024-02-23T00:35:17.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c1242c0 osd=6 4 2024-02-23T00:35:17.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 10 client.27123.objecter _send_command 14 2024-02-23T00:35:17.013 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.011+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7c125370 con 0x7f9d7c124940 2024-02-23T00:35:17.013 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:17.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7c125370 con 0x7f9d7c124940 2024-02-23T00:35:17.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:17.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d73fff700 10 client.27123.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c1242c0 osd=6 4 2024-02-23T00:35:17.014 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 6 14 2024-02-23T00:35:17.015 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:35:17.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 10 client.27123.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:17.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 15 2024-02-23T00:35:17.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7c12a210 0x7f9d7c12c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:17.015 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c125370 osd=7 3 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125370 osd=7 3 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c125370 osd=7 3 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 15 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c125370 osd=7 3 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 7 15 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125370 osd=7 4 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 10 client.27123.objecter _send_command 15 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7c12f250 con 0x7f9d7c12a210 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.013+0000 7f9d7affd700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7c12a210 0x7f9d7c12c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.014+0000 7f9d73fff700 10 client.27123.objecter ms_handle_connect 0x7f9d7c12a210 2024-02-23T00:35:17.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.014+0000 7f9d7affd700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7c12a210 0x7f9d7c12c600 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:17.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.015+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7c12f250 con 0x7f9d7c12a210 2024-02-23T00:35:17.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.015+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:35:17.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.015+0000 7f9d73fff700 10 client.27123.objecter _finish_command 15 = system:0 2024-02-23T00:35:17.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.015+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c125370 osd=7 4 2024-02-23T00:35:17.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.015+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 7 15 2024-02-23T00:35:17.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 10 client.27123.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:17.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign -1 16 2024-02-23T00:35:17.030 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c125370 osd=7 3 2024-02-23T00:35:17.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125370 osd=7 3 2024-02-23T00:35:17.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 20 client.27123.objecter _get_session s=0x7f9d7c125370 osd=7 3 2024-02-23T00:35:17.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_remove -1 16 2024-02-23T00:35:17.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 20 client.27123.objecter get_session s=0x7f9d7c125370 osd=7 3 2024-02-23T00:35:17.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 15 client.27123.objecter _session_command_op_assign 7 16 2024-02-23T00:35:17.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125370 osd=7 4 2024-02-23T00:35:17.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 10 client.27123.objecter _send_command 16 2024-02-23T00:35:17.031 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.029+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7c12caf0 con 0x7f9d7c12a210 2024-02-23T00:35:17.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.031+0000 7f9d73fff700 1 -- 172.21.15.87:0/408714771 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7c12caf0 con 0x7f9d7c12a210 2024-02-23T00:35:17.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.031+0000 7f9d73fff700 10 client.27123.objecter ms_dispatch 0x7f9d7c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:17.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.031+0000 7f9d73fff700 10 client.27123.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.031+0000 7f9d73fff700 20 client.27123.objecter put_session s=0x7f9d7c125370 osd=7 4 2024-02-23T00:35:17.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.031+0000 7f9d73fff700 15 client.27123.objecter _session_command_op_remove 7 16 2024-02-23T00:35:17.033 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:35:17.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 librados: watch_flush enter 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 librados: watch_flush exit 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 client.27123.objecter close_session for osd.0 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6801/2153191863 conn(0x7f9d7c0fdea0 msgr2=0x7f9d7c0fe280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6801/2153191863 conn(0x7f9d7c0fdea0 0x7f9d7c0fe280 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c05ee80 osd=0 1 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 client.27123.objecter close_session for osd.1 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d7c0598f0 msgr2=0x7f9d7c059cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d7c0598f0 0x7f9d7c059cf0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c028540 osd=1 1 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 client.27123.objecter close_session for osd.2 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d7c0280a0 msgr2=0x7f9d7c11bfc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d7c0280a0 0x7f9d7c11bfc0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d7b7fe700 1 -- 172.21.15.87:0/408714771 reap_dead start 2024-02-23T00:35:17.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c059770 osd=2 1 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 client.27123.objecter close_session for osd.3 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7c119b10 msgr2=0x7f9d7c11d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7c119b10 0x7f9d7c11d630 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c027a10 osd=3 1 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 client.27123.objecter close_session for osd.4 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7c1191e0 msgr2=0x7f9d7c122da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7c1191e0 0x7f9d7c122da0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c119070 osd=4 1 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 client.27123.objecter close_session for osd.5 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d7c123880 msgr2=0x7f9d7c123ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d7c123880 0x7f9d7c123ce0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125ac0 osd=5 1 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 client.27123.objecter close_session for osd.6 2024-02-23T00:35:17.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d7c124940 msgr2=0x7f9d7c124d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d7c124940 0x7f9d7c124d80 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c1242c0 osd=6 1 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 client.27123.objecter close_session for osd.7 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7c12a210 msgr2=0x7f9d7c12c600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7c12a210 0x7f9d7c12c600 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 20 client.27123.objecter put_session s=0x7f9d7c125370 osd=7 1 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 20 client.27123.objecter shutdown clearing up homeless session... 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 10 client.27123.objecter successfully canceled tick 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d7b7fe700 1 -- 172.21.15.87:0/408714771 reap_dead start 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d68062c90 msgr2=0x7f9d68065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d68062c90 0x7f9d68065150 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 rx=0x7f9d7c061720 tx=0x7f9d6400b040).stop 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d7c0f7400 msgr2=0x7f9d7c108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d7c0f7400 0x7f9d7c108fe0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 rx=0x7f9d6c001e70 tx=0x7f9d6c004d10).stop 2024-02-23T00:35:17.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 shutdown_connections 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d7c0f7400 0x7f9d7c108fe0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 --2- 172.21.15.87:0/408714771 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d68062c90 0x7f9d68065150 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 >> 172.21.15.87:0/408714771 conn(0x7f9d7c0f2820 msgr2=0x7f9d7c0f40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 shutdown_connections 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 -- 172.21.15.87:0/408714771 wait complete. 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:17.032+0000 7f9d816a0700 1 librados: shutdown 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.038 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:35:17.045 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:17.066 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:17.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:17.171 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:17.197 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:15.634060+0000 mgr.y (mgr.24437) 345 : cluster [DBG] pgmap v529: 230 pgs: 1 active+remapped+backfill_toofull, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:16.070005+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]': finished 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:16.070079+0000 mon.a (mon.0) 2383 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:16.070154+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:16.086423+0000 mon.c (mon.2) 404 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:16.086686+0000 mon.a (mon.0) 2385 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:16.087273+0000 mon.c (mon.2) 405 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:16.087484+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:16.087852+0000 mon.c (mon.2) 406 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi087-82019-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:16.088076+0000 mon.a (mon.0) 2387 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi087-82019-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:16.888711+0000 mon.a (mon.0) 2388 : audit [DBG] from='client.? 172.21.15.87:0/408714771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:15.634060+0000 mgr.y (mgr.24437) 345 : cluster [DBG] pgmap v529: 230 pgs: 1 active+remapped+backfill_toofull, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:17.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:16.070005+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]': finished 2024-02-23T00:35:17.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:16.070079+0000 mon.a (mon.0) 2383 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:35:17.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:16.070154+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-02-23T00:35:17.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:16.086423+0000 mon.c (mon.2) 404 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:16.086686+0000 mon.a (mon.0) 2385 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:16.087273+0000 mon.c (mon.2) 405 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:16.087484+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:16.087852+0000 mon.c (mon.2) 406 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi087-82019-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:17.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:16.088076+0000 mon.a (mon.0) 2387 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi087-82019-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:17.199 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:16.888711+0000 mon.a (mon.0) 2388 : audit [DBG] from='client.? 172.21.15.87:0/408714771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:17.268 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:17.398 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:17.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:15.634060+0000 mgr.y (mgr.24437) 345 : cluster [DBG] pgmap v529: 230 pgs: 1 active+remapped+backfill_toofull, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:16.070005+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi087-82019-62"}]': finished 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:16.070079+0000 mon.a (mon.0) 2383 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:16.070154+0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:16.086423+0000 mon.c (mon.2) 404 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:16.086686+0000 mon.a (mon.0) 2385 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:16.087273+0000 mon.c (mon.2) 405 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:16.087484+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:16.087852+0000 mon.c (mon.2) 406 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi087-82019-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:16.088076+0000 mon.a (mon.0) 2387 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi087-82019-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:16.888711+0000 mon.a (mon.0) 2388 : audit [DBG] from='client.? 172.21.15.87:0/408714771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:17.513 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:17.541 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:17.794 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:17.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:17.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:17.898 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:17.931 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:17.952 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:17.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:18.056 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:18.091 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:18.112 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:18.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:17.081451+0000 mon.a (mon.0) 2389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi087-82019-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:17.081532+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-02-23T00:35:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:17.088196+0000 mon.c (mon.2) 407 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi087-82019-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:18.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:17.088792+0000 mon.a (mon.0) 2391 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi087-82019-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:18.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:18.244 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:18.304 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:18.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:18.321 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:35:18.321 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:35:18.321 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:35:18.321 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24757.0:10 21.9 21:928daad9:test-rados-api-smithi087-82107-4::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:30:57.492284+0000", 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 260.82571858599999, 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019562380000000001, 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24757", 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3080165175", 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 10 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.322 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492284+0000", 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492284+0000", 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999971 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492281+0000", 2024-02-23T00:35:18.323 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3389e-05 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492295+0000", 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3093999999999999e-05 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492318+0000", 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7960000000000007e-06 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.324 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492327+0000", 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000104923 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492432+0000", 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9130999999999997e-05 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492531+0000", 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00020787600000000001 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.325 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.492739+0000", 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010449754 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.503189+0000", 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00069305099999999998 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.503882+0000", 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0078638679999999996 2024-02-23T00:35:18.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.511746+0000", 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2425e-05 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.511798+0000", 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7735000000000003e-05 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.511846+0000", 2024-02-23T00:35:18.327 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019562380000000001 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.24712.0:14 7.5 e108/94 7:a3910452:test-rados-api-smithi087-82115-5::foo:head v 108'1, mlcod=0'0)", 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:30:58.510883+0000", 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 259.80711909799999, 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013710257, 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.328 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510883+0000", 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510883+0000", 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510883+0000", 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0350000000000001e-06 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510888+0000", 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.759e-06 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510895+0000", 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6359999999999994e-06 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.510902+0000", 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000288127 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.330 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.511190+0000", 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.7677e-05 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.511208+0000", 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013359852 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.524568+0000", 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5661999999999999e-05 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.331 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:58.524594+0000", 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013710257 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:3 61.1f e114/112 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head v 114'1, mlcod=0'0)", 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:05.241186+0000", 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 253.076816538, 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0078894910000000002, 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.332 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241186+0000", 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241186+0000", 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241184+0000", 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2142e-05 2024-02-23T00:35:18.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241196+0000", 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4297999999999999e-05 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241210+0000", 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3780000000000003e-06 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241218+0000", 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0946999999999998e-05 2024-02-23T00:35:18.334 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241279+0000", 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0699999999999998e-05 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241299+0000", 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0077120080000000002 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.249011+0000", 2024-02-23T00:35:18.335 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3965000000000002e-05 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.249075+0000", 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0078894910000000002 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:10 61.1f e118/112 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head v 118'6, mlcod=115'3)", 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.218637+0000", 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 249.099365541, 2024-02-23T00:35:18.336 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016531127999999999, 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218637+0000", 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218637+0000", 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.337 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218637+0000", 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5999999999999997e-06 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218642+0000", 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6220000000000003e-06 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218649+0000", 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.703e-06 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.338 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.218655+0000", 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010198169 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.228854+0000", 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5311e-05 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.228869+0000", 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0062654290000000003 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.339 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.235134+0000", 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3626e-05 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.235168+0000", 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016531127999999999 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27712.0:3 67.1f 67:f9c13a70:test-rados-api-smithi087-82115-8::foo:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected e119)", 2024-02-23T00:35:18.340 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.170310+0000", 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 248.147692122, 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062446785999999997, 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27712", 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2607672493", 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 3 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170310+0000", 2024-02-23T00:35:18.341 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170310+0000", 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170308+0000", 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.1270000000000003e-06 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.342 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170316+0000", 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3158e-05 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170329+0000", 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2034e-05 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170341+0000", 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0090999999999999e-05 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.343 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170391+0000", 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4679999999999997e-06 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.170398+0000", 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.047525959 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.217924+0000", 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000136811 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.344 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.218061+0000", 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00018028200000000001 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.218241+0000", 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0017470389999999999 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.219988+0000", 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.01262332 2024-02-23T00:35:18.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.232611+0000", 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6244e-05 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.232658+0000", 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6916999999999997e-05 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.232714+0000", 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2565999999999999e-05 2024-02-23T00:35:18.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.232757+0000", 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062446785999999997 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:807 5.3 5:c609908c:::notify.5:head [watch ping cookie 94532396652160] snapc 0=[] ondisk+write+known_if_redirected e114)", 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.382657+0000", 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 247.93534545599999, 2024-02-23T00:35:18.347 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010145018, 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 807 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382657+0000", 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.348 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382657+0000", 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382656+0000", 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0860000000000001e-06 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382661+0000", 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5639999999999999e-06 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.349 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382669+0000", 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5300000000000002e-06 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.382675+0000", 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0099372450000000008 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.392612+0000", 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5774999999999998e-05 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.350 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.392698+0000", 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000103871 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.392802+0000", 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010145018 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27712.0:5 67.1f 67:f9c13a70:test-rados-api-smithi087-82115-8::foo:head [write 0~128] snapc 3=[3,2] ondisk+write+known_if_redirected e120)", 2024-02-23T00:35:18.351 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.171931+0000", 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 247.14607101600001, 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.22139420000000001, 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27712", 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2607672493", 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 5 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.352 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171931+0000", 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171931+0000", 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171930+0000", 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5570000000000001e-06 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.353 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171937+0000", 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0461e-05 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171947+0000", 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9319999999999996e-06 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.171955+0000", 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0064871850000000003 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.354 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.178442+0000", 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1840000000000002e-06 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.178449+0000", 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.18818955100000001 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.366639+0000", 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2639000000000005e-05 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.355 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.366731+0000", 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00018480499999999999 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.366916+0000", 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0048910660000000003 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.371807+0000", 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021056603 2024-02-23T00:35:18.356 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.392864+0000", 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00030953399999999998 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.393173+0000", 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6799000000000006e-05 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.393270+0000", 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5291999999999998e-05 2024-02-23T00:35:18.357 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.393326+0000", 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.22139420000000001 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:37 66.17 66:e8000000:%01:::head [pgnls start_epoch 120] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e120)", 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.430859+0000", 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 246.88714373900001, 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0083297739999999999, 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.358 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 37 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430859+0000", 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430859+0000", 2024-02-23T00:35:18.359 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430856+0000", 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0530000000000001e-05 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430867+0000", 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4768e-05 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430882+0000", 2024-02-23T00:35:18.360 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4930000000000001e-06 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.430889+0000", 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0081343149999999996 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439024+0000", 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5954000000000001e-05 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439069+0000", 2024-02-23T00:35:18.361 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011896399999999999 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.439188+0000", 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0083297739999999999 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27712.0:57 67.3 e138/117 67:c06d7414:test-rados-api-smithi087-82115-12::foo:head v 138'1, mlcod=0'0)", 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:29.024212+0000", 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 229.293790019, 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024530395999999999, 2024-02-23T00:35:18.362 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024212+0000", 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024212+0000", 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.363 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024212+0000", 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7690000000000002e-06 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024221+0000", 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4880000000000004e-06 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024227+0000", 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6460000000000004e-06 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.364 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.024234+0000", 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0069832150000000001 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.031217+0000", 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1175e-05 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.031238+0000", 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017474302000000001 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.365 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.048713+0000", 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0083999999999999e-05 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.048743+0000", 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024530395999999999 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDScrubReserve(10.1 REQUEST e138)", 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.013579+0000", 2024-02-23T00:35:18.366 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 228.30442319100001, 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010003164, 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013579+0000", 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013579+0000", 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.367 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013577+0000", 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4552000000000001e-05 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013592+0000", 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2685999999999999e-05 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013614+0000", 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0902e-05 2024-02-23T00:35:18.368 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.013625+0000", 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0098954390000000007 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.023521+0000", 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2829000000000001e-05 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.023543+0000", 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8949000000000003e-05 2024-02-23T00:35:18.369 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.023582+0000", 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010003164 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.4.0:339 10.1 e138/94 10:81ba0521:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa::a:head v 138'2, mlcod=0'0)", 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.026047+0000", 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 228.29195574100001, 2024-02-23T00:35:18.370 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0074907230000000003, 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026047+0000", 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026047+0000", 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.371 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026045+0000", 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6579999999999997e-06 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026055+0000", 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0818999999999999e-05 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026065+0000", 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7590000000000003e-06 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.372 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026071+0000", 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5367000000000002e-05 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026107+0000", 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6110999999999999e-05 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.026123+0000", 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0073568699999999997 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.373 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.033480+0000", 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7766999999999998e-05 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.033537+0000", 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0074907230000000003 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:672 63.6 e147/114 63:6271ce74:test-rados-api-smithi087-82091-14::foo0:head v 147'17, mlcod=146'13)", 2024-02-23T00:35:18.374 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:37.995603+0000", 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 220.32239941700001, 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012932641, 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995603+0000", 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995603+0000", 2024-02-23T00:35:18.375 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995603+0000", 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4840000000000003e-06 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995608+0000", 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9030000000000001e-06 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.376 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995614+0000", 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5999999999999997e-06 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995620+0000", 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1352000000000005e-05 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995681+0000", 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0752000000000001e-05 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.377 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.995702+0000", 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012792253 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.008494+0000", 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1677000000000001e-05 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.008536+0000", 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012932641 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.378 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.36722.0:2 99.6 99:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e149)", 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:39.997356+0000", 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 218.32064682199999, 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0097361589999999994, 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.36722", 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2157738800", 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.379 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997356+0000", 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997356+0000", 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997354+0000", 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3889999999999999e-06 2024-02-23T00:35:18.380 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997361+0000", 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0131e-05 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997371+0000", 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9399999999999996e-06 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997378+0000", 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0971999999999999e-05 2024-02-23T00:35:18.381 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997429+0000", 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8999999999999995e-06 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.997438+0000", 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003947908 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.001386+0000", 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012212799999999999 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.382 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.001508+0000", 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000146413 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,4", 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.001655+0000", 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00048154200000000001 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.002136+0000", 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000286619 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.383 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.002423+0000", 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0045833820000000004 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.007006+0000", 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4014000000000002e-05 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.007050+0000", 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1578000000000001e-05 2024-02-23T00:35:18.384 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:40.007092+0000", 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0097361589999999994 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.2.0:279 63.16 e182/114 63:689df377:test-rados-api-smithi087-82091-14::foo54:7 v 182'31, mlcod=148'25)", 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.012421+0000", 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 185.30558091, 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014084329, 2024-02-23T00:35:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012421+0000", 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012421+0000", 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.386 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012421+0000", 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9580000000000001e-06 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012425+0000", 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9779999999999996e-06 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012430+0000", 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9790000000000004e-06 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.387 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012434+0000", 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00029118500000000001 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012725+0000", 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6079999999999999e-05 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.012741+0000", 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013727184999999999 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.388 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.026468+0000", 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7320000000000002e-05 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.026506+0000", 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014084329 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1311 63.12 e182/114 63:4961eb90:test-rados-api-smithi087-82091-14::foo67:7 v 182'38, mlcod=148'34)", 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.013199+0000", 2024-02-23T00:35:18.389 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 185.304803542, 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018593278000000001, 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013199+0000", 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013199+0000", 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.390 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013198+0000", 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7669999999999998e-06 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013205+0000", 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5710000000000004e-06 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013215+0000", 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0329999999999997e-06 2024-02-23T00:35:18.391 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013221+0000", 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001101557 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.014322+0000", 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6117000000000001e-05 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.014349+0000", 2024-02-23T00:35:18.392 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017406123999999999 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.031755+0000", 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7462000000000001e-05 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.031792+0000", 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018593278000000001 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.393 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1312 63.12 e182/114 63:49181275:test-rados-api-smithi087-82091-14::foo3:7 v 182'40, mlcod=148'34)", 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.013227+0000", 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 185.304775858, 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019144881999999998, 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013227+0000", 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.394 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013227+0000", 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013226+0000", 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7740000000000002e-06 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013230+0000", 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6500000000000004e-06 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.395 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013235+0000", 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2967e-05 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.013248+0000", 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0072151000000000003 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.020463+0000", 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9933e-05 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.396 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.020483+0000", 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011861216000000001 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.032344+0000", 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.76e-05 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.032371+0000", 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019144881999999998 2024-02-23T00:35:18.397 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1313 63.14 e182/114 63:2ca15095:test-rados-api-smithi087-82091-14::foo41:7 v 182'47, mlcod=148'43)", 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.017065+0000", 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 185.30093689500001, 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015321398999999999, 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017065+0000", 2024-02-23T00:35:18.398 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017065+0000", 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017065+0000", 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2177999999999999e-05 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017077+0000", 2024-02-23T00:35:18.399 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8810000000000002e-06 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017086+0000", 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3049999999999998e-06 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.017092+0000", 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0015861499999999999 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.400 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.018678+0000", 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0152e-05 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.018698+0000", 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013653176 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.032352+0000", 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5281999999999999e-05 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.401 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.032387+0000", 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015321398999999999 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1314 63.14 e182/114 63:2cd9217a:test-rados-api-smithi087-82091-14::foo89:7 v 182'49, mlcod=148'43)", 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.021891+0000", 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 185.29611183399999, 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011822889, 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.402 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021891+0000", 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021891+0000", 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021890+0000", 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0390000000000005e-06 2024-02-23T00:35:18.403 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021898+0000", 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5710000000000003e-06 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021905+0000", 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2120000000000002e-06 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.404 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.021910+0000", 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0060575969999999996 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027967+0000", 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1897999999999999e-05 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027989+0000", 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0056844479999999999 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.405 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.033674+0000", 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9691999999999999e-05 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.033713+0000", 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011822889 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.37106.0:2 149.1a e216/215 149:5ec95afd:::oid_ReadIntoBufferlist_:head v 216'1, mlcod=0'0)", 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.056252+0000", 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 151.26175014500001, 2024-02-23T00:35:18.406 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.049676218000000001, 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056252+0000", 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056252+0000", 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.407 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056252+0000", 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0475e-05 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056272+0000", 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8540000000000002e-06 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056278+0000", 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9689999999999999e-06 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.408 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056284+0000", 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000186269 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056470+0000", 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3541e-05 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.056484+0000", 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.049403124999999999 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.409 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.105887+0000", 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1409999999999998e-05 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.105928+0000", 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.049676218000000001 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:1811 148.8 148:15d27fd9:test-rados-api-smithi087-82091-22::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e219)", 2024-02-23T00:35:18.410 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:50.035563+0000", 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 148.282439079, 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010372207, 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1811 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035563+0000", 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:35:18.411 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035563+0000", 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035561+0000", 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3500000000000002e-06 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035568+0000", 2024-02-23T00:35:18.412 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2699999999999993e-06 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035577+0000", 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.238e-06 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.035584+0000", 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002828269 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.038413+0000", 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021697099999999999 2024-02-23T00:35:18.413 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.038630+0000", 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0024583119999999998 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.041088+0000", 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6991999999999997e-05 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for rw locks", 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.041165+0000", 2024-02-23T00:35:18.414 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0042903780000000001 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.045455+0000", 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7844999999999997e-05 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.045543+0000", 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00039248100000000003 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:50.045936+0000", 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010372207 2024-02-23T00:35:18.415 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.416 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.416 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.416 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:35:18.416 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:35:18.416 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:35:18.416 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:18.417 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:17.081451+0000 mon.a (mon.0) 2389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi087-82019-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:18.417 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:17.081532+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-02-23T00:35:18.417 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:17.088196+0000 mon.c (mon.2) 407 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi087-82019-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:18.417 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:17.088792+0000 mon.a (mon.0) 2391 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi087-82019-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:18.418 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:17.081451+0000 mon.a (mon.0) 2389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi087-82019-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:18.418 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:17.081532+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-02-23T00:35:18.418 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:17.088196+0000 mon.c (mon.2) 407 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi087-82019-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:18.418 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:17.088792+0000 mon.a (mon.0) 2391 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi087-82019-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:18.445 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:19.094 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:19.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:19.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:19.203 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:19.231 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:19.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:17.634782+0000 mgr.y (mgr.24437) 346 : cluster [DBG] pgmap v532: 230 pgs: 1 active+remapped+backfill_toofull, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 6.7 KiB/s rd, 255 B/s wr, 12 op/s; 13/1107 objects misplaced (1.174%); 27 B/s, 1 objects/s recovering 2024-02-23T00:35:19.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:18.332880+0000 mon.a (mon.0) 2392 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-02-23T00:35:19.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:18.430534+0000 mon.b (mon.1) 316 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:19.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:18.430903+0000 mon.a (mon.0) 2393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:19.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:17.634782+0000 mgr.y (mgr.24437) 346 : cluster [DBG] pgmap v532: 230 pgs: 1 active+remapped+backfill_toofull, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 6.7 KiB/s rd, 255 B/s wr, 12 op/s; 13/1107 objects misplaced (1.174%); 27 B/s, 1 objects/s recovering 2024-02-23T00:35:19.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:18.332880+0000 mon.a (mon.0) 2392 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-02-23T00:35:19.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:18.430534+0000 mon.b (mon.1) 316 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:19.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:18.430903+0000 mon.a (mon.0) 2393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:17.634782+0000 mgr.y (mgr.24437) 346 : cluster [DBG] pgmap v532: 230 pgs: 1 active+remapped+backfill_toofull, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 6.7 KiB/s rd, 255 B/s wr, 12 op/s; 13/1107 objects misplaced (1.174%); 27 B/s, 1 objects/s recovering 2024-02-23T00:35:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:18.332880+0000 mon.a (mon.0) 2392 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-02-23T00:35:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:18.430534+0000 mon.b (mon.1) 316 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:18.430903+0000 mon.a (mon.0) 2393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:19.479 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:19.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:19.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:19.589 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:19.616 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:19.944 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-23T00:35:19.964 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:19.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:19.981 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:20.068 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:20.101 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:20.131 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:20.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:20.235 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:20.260 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:19.095845+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi087-82019-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi087-82019-63"}]': finished 2024-02-23T00:35:20.260 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:19.095968+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:35:20.260 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:19.096020+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-02-23T00:35:20.260 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:19.102286+0000 mon.b (mon.1) 317 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:20.260 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:19.107819+0000 mon.a (mon.0) 2397 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:20.260 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:19.636235+0000 mon.a (mon.0) 2398 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-23T00:35:20.260 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:19.095845+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi087-82019-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi087-82019-63"}]': finished 2024-02-23T00:35:20.260 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:19.095968+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:35:20.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:19.096020+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-02-23T00:35:20.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:19.102286+0000 mon.b (mon.1) 317 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:20.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:19.107819+0000 mon.a (mon.0) 2397 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:20.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:19.636235+0000 mon.a (mon.0) 2398 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-23T00:35:20.262 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:20.289 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:20.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:20.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:20.422 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:19.095845+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi087-82019-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi087-82019-63"}]': finished 2024-02-23T00:35:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:19.095968+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:35:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:19.096020+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-02-23T00:35:20.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:19.102286+0000 mon.b (mon.1) 317 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:20.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:19.107819+0000 mon.a (mon.0) 2397 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:20.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:19.636235+0000 mon.a (mon.0) 2398 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-23T00:35:20.569 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:20.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:20.674 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:20.700 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:20.719 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:20.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:20.823 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:20.855 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:20.876 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:20.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:20.981 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:21.009 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:21.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:35:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:35:21.104 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:35:21.140 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:19.635315+0000 mgr.y (mgr.24437) 347 : cluster [DBG] pgmap v535: 237 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1095 objects misplaced (1.187%) 2024-02-23T00:35:21.140 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:20.093145+0000 mon.a (mon.0) 2399 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:35:21.140 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:20.101786+0000 mon.a (mon.0) 2400 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]': finished 2024-02-23T00:35:21.140 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:20.101888+0000 mon.a (mon.0) 2401 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-23T00:35:21.140 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:20.101942+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-02-23T00:35:21.140 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:20.130485+0000 mon.b (mon.1) 318 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:21.140 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:20.130724+0000 mon.a (mon.0) 2403 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:21.140 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:20.131412+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:21.140 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:20.131532+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:21.141 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:19.635315+0000 mgr.y (mgr.24437) 347 : cluster [DBG] pgmap v535: 237 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1095 objects misplaced (1.187%) 2024-02-23T00:35:21.141 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:20.093145+0000 mon.a (mon.0) 2399 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:35:21.141 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:20.101786+0000 mon.a (mon.0) 2400 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]': finished 2024-02-23T00:35:21.141 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:20.101888+0000 mon.a (mon.0) 2401 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-23T00:35:21.141 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:20.101942+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-02-23T00:35:21.141 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:20.130485+0000 mon.b (mon.1) 318 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:21.141 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:20.130724+0000 mon.a (mon.0) 2403 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:21.141 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:20.131412+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:21.141 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:20.131532+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:21.256 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:21.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:21.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:21.361 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:21.388 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:19.635315+0000 mgr.y (mgr.24437) 347 : cluster [DBG] pgmap v535: 237 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1095 objects misplaced (1.187%) 2024-02-23T00:35:21.388 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:20.093145+0000 mon.a (mon.0) 2399 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:35:21.388 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:20.101786+0000 mon.a (mon.0) 2400 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]': finished 2024-02-23T00:35:21.389 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:20.101888+0000 mon.a (mon.0) 2401 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-23T00:35:21.389 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:20.101942+0000 mon.a (mon.0) 2402 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-02-23T00:35:21.389 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:20.130485+0000 mon.b (mon.1) 318 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:21.389 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:20.130724+0000 mon.a (mon.0) 2403 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:21.389 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:20.131412+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:21.389 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:20.131532+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-47"}]: dispatch 2024-02-23T00:35:21.390 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:21.412 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:21.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:21.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:21.516 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:21.544 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:21.563 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:21.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:21.667 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:21.694 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:21.713 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:21.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:21.817 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:21.844 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:22.096 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:22.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:22.200 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:22.228 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:22.247 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:22.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:20.848353+0000 mgr.y (mgr.24437) 348 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:21.111430+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-02-23T00:35:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:21.114273+0000 mon.c (mon.2) 408 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:21.364528+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:22.065433+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]': finished 2024-02-23T00:35:22.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:22.065549+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-02-23T00:35:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:22.068131+0000 mon.b (mon.1) 320 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:22.068422+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:22.070022+0000 mon.c (mon.2) 409 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:22.070330+0000 mon.a (mon.0) 2410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.334 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:22] "GET /metrics HTTP/1.1" 200 187630 "" "Prometheus/2.33.4" 2024-02-23T00:35:22.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:20.848353+0000 mgr.y (mgr.24437) 348 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:22.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:21.111430+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-02-23T00:35:22.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:21.114273+0000 mon.c (mon.2) 408 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:21.364528+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:22.065433+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]': finished 2024-02-23T00:35:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:22.065549+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-02-23T00:35:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:22.068131+0000 mon.b (mon.1) 320 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:22.068422+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:22.070022+0000 mon.c (mon.2) 409 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:22.070330+0000 mon.a (mon.0) 2410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.350 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:22.378 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:20.848353+0000 mgr.y (mgr.24437) 348 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:21.111430+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-02-23T00:35:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:21.114273+0000 mon.c (mon.2) 408 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:21.364528+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:22.065433+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi087-82019-63"}]': finished 2024-02-23T00:35:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:22.065549+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-02-23T00:35:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:22.068131+0000 mon.b (mon.1) 320 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:22.068422+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:22.070022+0000 mon.c (mon.2) 409 : audit [INF] from='client.? 172.21.15.87:0/1620356111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:22.070330+0000 mon.a (mon.0) 2410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]: dispatch 2024-02-23T00:35:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:21.635913+0000 mgr.y (mgr.24437) 349 : cluster [DBG] pgmap v538: 197 pgs: 1 active+remapped+backfill_toofull, 2 peering, 194 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.066802+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.066903+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]': finished 2024-02-23T00:35:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:23.067037+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-02-23T00:35:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.082895+0000 mon.c (mon.2) 410 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.083173+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.083765+0000 mon.c (mon.2) 411 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.083974+0000 mon.a (mon.0) 2415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.084300+0000 mon.c (mon.2) 412 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi087-82019-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.084523+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi087-82019-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:21.635913+0000 mgr.y (mgr.24437) 349 : cluster [DBG] pgmap v538: 197 pgs: 1 active+remapped+backfill_toofull, 2 peering, 194 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.066802+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.066903+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]': finished 2024-02-23T00:35:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:23.067037+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-02-23T00:35:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.082895+0000 mon.c (mon.2) 410 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.083173+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.083765+0000 mon.c (mon.2) 411 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.083974+0000 mon.a (mon.0) 2415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.084300+0000 mon.c (mon.2) 412 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi087-82019-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:23.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.084523+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi087-82019-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:21.635913+0000 mgr.y (mgr.24437) 349 : cluster [DBG] pgmap v538: 197 pgs: 1 active+remapped+backfill_toofull, 2 peering, 194 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 13/1107 objects misplaced (1.174%) 2024-02-23T00:35:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.066802+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.066903+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi087-82019-63"}]': finished 2024-02-23T00:35:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:23.067037+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-02-23T00:35:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.082895+0000 mon.c (mon.2) 410 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.083173+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.083765+0000 mon.c (mon.2) 411 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.083974+0000 mon.a (mon.0) 2415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.084300+0000 mon.c (mon.2) 412 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi087-82019-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:23.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.084523+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi087-82019-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:24.123 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:35:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:35:24.429 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:24.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.118258+0000 mon.b (mon.1) 321 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:24.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:23.118465+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:24.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.071422+0000 mon.a (mon.0) 2418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi087-82019-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:24.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.071519+0000 mon.a (mon.0) 2419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:35:24.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:24.071585+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-02-23T00:35:24.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.073170+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:24.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.074135+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:24.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.074518+0000 mon.c (mon.2) 413 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi087-82019-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:24.432 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.074829+0000 mon.a (mon.0) 2422 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi087-82019-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:24.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.118258+0000 mon.b (mon.1) 321 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:24.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:23.118465+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:24.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.071422+0000 mon.a (mon.0) 2418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi087-82019-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:24.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.071519+0000 mon.a (mon.0) 2419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:35:24.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:24.071585+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-02-23T00:35:24.432 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.073170+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:24.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.074135+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:24.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.074518+0000 mon.c (mon.2) 413 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi087-82019-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:24.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.074829+0000 mon.a (mon.0) 2422 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi087-82019-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:24.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.118258+0000 mon.b (mon.1) 321 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:23.118465+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:35:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.071422+0000 mon.a (mon.0) 2418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi087-82019-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.071519+0000 mon.a (mon.0) 2419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:35:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:24.071585+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-02-23T00:35:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.073170+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.074135+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.074518+0000 mon.c (mon.2) 413 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi087-82019-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.074829+0000 mon.a (mon.0) 2422 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi087-82019-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:24.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:24.566 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:24.587 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:24.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:24.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 -- 172.21.15.87:0/970130078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940fdbb0 msgr2=0x7f02940fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 --2- 172.21.15.87:0/970130078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940fdbb0 0x7f02940fdf90 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 rx=0x7f0288004870 tx=0x7f0288009cb0).stop 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 -- 172.21.15.87:0/970130078 shutdown_connections 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 --2- 172.21.15.87:0/970130078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02940f8420 0x7f0294104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 --2- 172.21.15.87:0/970130078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940fdbb0 0x7f02940fdf90 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 --2- 172.21.15.87:0/970130078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02940fe510 0x7f02940f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 -- 172.21.15.87:0/970130078 >> 172.21.15.87:0/970130078 conn(0x7f02940f3800 msgr2=0x7f02940f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 -- 172.21.15.87:0/970130078 shutdown_connections 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 -- 172.21.15.87:0/970130078 wait complete. 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 librados: starting msgr at 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 librados: starting objecter 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.539+0000 7f0299b2e700 1 Processor -- start 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0299b2e700 1 -- start start 2024-02-23T00:35:24.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0299b2e700 1 librados: setting wanted keys 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0299b2e700 1 librados: calling monclient init 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0299b2e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940f8420 0x7f02941102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0299b2e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02940fdbb0 0x7f0294110790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0299b2e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02940fe510 0x7f029410a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0299b2e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029410a930 con 0x7f02940f8420 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0299b2e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029410aaa0 con 0x7f02940fe510 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0299b2e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029410ac60 con 0x7f02940fdbb0 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f02937fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940f8420 0x7f02941102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0292ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02940fdbb0 0x7f0294110790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f02937fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940f8420 0x7f02941102a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43834/0 (socket says 172.21.15.87:43834) 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f02937fe700 1 -- 172.21.15.87:0/1981431750 learned_addr learned my addr 172.21.15.87:0/1981431750 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0293fff700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02940fe510 0x7f029410a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f02937fe700 1 -- 172.21.15.87:0/1981431750 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02940fdbb0 msgr2=0x7f0294110790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:24.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f02937fe700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02940fdbb0 0x7f0294110790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f02937fe700 1 -- 172.21.15.87:0/1981431750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02940fe510 msgr2=0x7f029410a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f02937fe700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02940fe510 0x7f029410a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f02937fe700 1 -- 172.21.15.87:0/1981431750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0284005040 con 0x7f02940f8420 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0292ffd700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02940fdbb0 0x7f0294110790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0293fff700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02940fe510 0x7f029410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f02937fe700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940f8420 0x7f02941102a0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7f0288009eb0 tx=0x7f028800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0290ff9700 10 client.?.objecter ms_handle_connect 0x7f02940f8420 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.540+0000 7f0290ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.541+0000 7f0290ff9700 1 -- 172.21.15.87:0/1981431750 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028800e070 con 0x7f02940f8420 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.541+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0288005040 con 0x7f02940f8420 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.541+0000 7f0290ff9700 1 -- 172.21.15.87:0/1981431750 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0288013da0 con 0x7f02940f8420 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.541+0000 7f0290ff9700 1 -- 172.21.15.87:0/1981431750 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288019070 con 0x7f02940f8420 2024-02-23T00:35:24.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.541+0000 7f0299b2e700 10 client.27135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.541+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029410b150 con 0x7f02940f8420 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.541+0000 7f0299b2e700 1 librados: init done 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.541+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029405ee80 con 0x7f02940f8420 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.542+0000 7f0290ff9700 1 -- 172.21.15.87:0/1981431750 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f028801e420 con 0x7f02940f8420 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.542+0000 7f0290ff9700 1 --2- 172.21.15.87:0/1981431750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f027c062c90 0x7f027c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.542+0000 7f0290ff9700 1 -- 172.21.15.87:0/1981431750 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(373..373 src has 173..373) v4 ==== 8371+0+0 (secure 0 0 0) 0x7f028807de80 con 0x7f02940f8420 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.542+0000 7f0290ff9700 10 client.27135.objecter ms_dispatch 0x7f0294001550 osd_map(373..373 src has 173..373) v4 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.542+0000 7f0290ff9700 3 client.27135.objecter handle_osd_map got epochs [373,373] > 0 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.542+0000 7f0290ff9700 3 client.27135.objecter handle_osd_map decoding full epoch 373 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.542+0000 7f0292ffd700 1 --2- 172.21.15.87:0/1981431750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f027c062c90 0x7f027c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.543+0000 7f0290ff9700 20 client.27135.objecter dump_active .. 0 homeless 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.543+0000 7f0290ff9700 10 client.27135.objecter ms_handle_connect 0x7f027c062c90 2024-02-23T00:35:24.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.543+0000 7f0292ffd700 1 --2- 172.21.15.87:0/1981431750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f027c062c90 0x7f027c065150 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7f029410bbe0 tx=0x7f028400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:24.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.547+0000 7f0290ff9700 1 -- 172.21.15.87:0/1981431750 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0288051280 con 0x7f02940f8420 2024-02-23T00:35:24.653 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:24.691 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:24.720 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:24.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.738+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f029405fa60 con 0x7f02940f8420 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.686+0000 7ff0cd53a700 1 -- 172.21.15.87:0/2720026951 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0c8102210 msgr2=0x7ff0c8104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.686+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/2720026951 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0c8102210 0x7ff0c8104600 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 rx=0x7ff0c0009c30 tx=0x7ff0c0004c50).stop 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.686+0000 7ff0cd53a700 1 -- 172.21.15.87:0/2720026951 shutdown_connections 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.686+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/2720026951 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0c8102210 0x7ff0c8104600 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.686+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/2720026951 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0c80f8cb0 0x7ff0c80f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.686+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/2720026951 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c80f9610 0x7ff0c8101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.686+0000 7ff0cd53a700 1 -- 172.21.15.87:0/2720026951 >> 172.21.15.87:0/2720026951 conn(0x7ff0c80f2820 msgr2=0x7ff0c80f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.686+0000 7ff0cd53a700 1 -- 172.21.15.87:0/2720026951 shutdown_connections 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.686+0000 7ff0cd53a700 1 -- 172.21.15.87:0/2720026951 wait complete. 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 librados: starting msgr at 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 librados: starting objecter 2024-02-23T00:35:24.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 Processor -- start 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 -- start start 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 librados: setting wanted keys 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 librados: calling monclient init 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0c80f8cb0 0x7ff0c810df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0c80f9610 0x7ff0c810e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c8102210 0x7ff0c81080b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c81085a0 con 0x7ff0c80f9610 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c8108710 con 0x7ff0c8102210 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0cd53a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c81088d0 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0c67fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0c80f9610 0x7ff0c810e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0c6ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0c80f8cb0 0x7ff0c810df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0c67fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0c80f9610 0x7ff0c810e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:43858/0 (socket says 172.21.15.87:43858) 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0c67fc700 1 -- 172.21.15.87:0/3806659717 learned_addr learned my addr 172.21.15.87:0/3806659717 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0c6ffd700 1 -- 172.21.15.87:0/3806659717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c8102210 msgr2=0x7ff0c81080b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0c6ffd700 1 --2- 172.21.15.87:0/3806659717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c8102210 0x7ff0c81080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:35:24.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0c6ffd700 1 -- 172.21.15.87:0/3806659717 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0c80f9610 msgr2=0x7ff0c810e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.687+0000 7ff0c6ffd700 1 --2- 172.21.15.87:0/3806659717 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0c80f9610 0x7ff0c810e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0c6ffd700 1 -- 172.21.15.87:0/3806659717 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b8005040 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0c6ffd700 1 --2- 172.21.15.87:0/3806659717 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0c80f8cb0 0x7ff0c810df10 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7ff0b80091f0 tx=0x7ff0b8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0affff700 10 client.?.objecter ms_handle_connect 0x7ff0c80f8cb0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0affff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0affff700 1 -- 172.21.15.87:0/3806659717 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b80079e0 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0affff700 1 -- 172.21.15.87:0/3806659717 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0b800f040 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c00097c0 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0affff700 1 -- 172.21.15.87:0/3806659717 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8014950 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0cd53a700 10 client.37601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c8114d40 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.688+0000 7ff0cd53a700 1 librados: init done 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.689+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c805fa60 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.689+0000 7ff0affff700 1 -- 172.21.15.87:0/3806659717 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff0b80104f0 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.690+0000 7ff0affff700 1 --2- 172.21.15.87:0/3806659717 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff0b0062c90 0x7ff0b0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:24.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.690+0000 7ff0c67fc700 1 --2- 172.21.15.87:0/3806659717 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff0b0062c90 0x7ff0b0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.690+0000 7ff0affff700 10 client.37601.objecter ms_handle_connect 0x7ff0b0062c90 2024-02-23T00:35:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.690+0000 7ff0c67fc700 1 --2- 172.21.15.87:0/3806659717 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff0b0062c90 0x7ff0b0065150 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 rx=0x7ff0c8109850 tx=0x7ff0bc009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.691+0000 7ff0affff700 1 -- 172.21.15.87:0/3806659717 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(373..373 src has 173..373) v4 ==== 8371+0+0 (secure 0 0 0) 0x7ff0b8050450 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.691+0000 7ff0affff700 10 client.37601.objecter ms_dispatch 0x7ff0c8001550 osd_map(373..373 src has 173..373) v4 2024-02-23T00:35:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.691+0000 7ff0affff700 3 client.37601.objecter handle_osd_map got epochs [373,373] > 0 2024-02-23T00:35:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.691+0000 7ff0affff700 3 client.37601.objecter handle_osd_map decoding full epoch 373 2024-02-23T00:35:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.691+0000 7ff0affff700 20 client.37601.objecter dump_active .. 0 homeless 2024-02-23T00:35:24.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.694+0000 7ff0affff700 1 -- 172.21.15.87:0/3806659717 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff0b8049f60 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.773 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:24.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:24.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:24.864+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"} v 0) v1 -- 0x7ff0c805ee80 con 0x7ff0c80f8cb0 2024-02-23T00:35:24.877 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:24.909 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:24.931 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:24.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:25.035 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:25.063 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:25.087 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:25.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:25.191 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:25.216 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:25.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.312+0000 7f0290ff9700 1 -- 172.21.15.87:0/1981431750 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v374) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0288023030 con 0x7f02940f8420 2024-02-23T00:35:25.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.312+0000 7ff0affff700 1 -- 172.21.15.87:0/3806659717 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]=0 set pool 5 pg_num to 42 v374) v1 ==== 144+0+0 (secure 0 0 0) 0x7ff0b8050220 con 0x7ff0c80f8cb0 2024-02-23T00:35:25.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.314+0000 7f0299b2e700 10 librados: watch_flush enter 2024-02-23T00:35:25.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.314+0000 7f0299b2e700 10 librados: watch_flush exit 2024-02-23T00:35:25.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.314+0000 7f0299b2e700 20 client.27135.objecter shutdown clearing up homeless session... 2024-02-23T00:35:25.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.314+0000 7f0299b2e700 10 client.27135.objecter successfully canceled tick 2024-02-23T00:35:25.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f027c062c90 msgr2=0x7f027c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:25.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 --2- 172.21.15.87:0/1981431750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f027c062c90 0x7f027c065150 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 rx=0x7f029410bbe0 tx=0x7f028400b040).stop 2024-02-23T00:35:25.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940f8420 msgr2=0x7f02941102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:25.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940f8420 0x7f02941102a0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 rx=0x7f0288009eb0 tx=0x7f028800b040).stop 2024-02-23T00:35:25.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 shutdown_connections 2024-02-23T00:35:25.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 --2- 172.21.15.87:0/1981431750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f027c062c90 0x7f027c065150 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:25.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02940fe510 0x7f029410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:25.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02940f8420 0x7f02941102a0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:25.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 --2- 172.21.15.87:0/1981431750 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02940fdbb0 0x7f0294110790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:25.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 >> 172.21.15.87:0/1981431750 conn(0x7f02940f3800 msgr2=0x7f029406af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:25.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 10 librados: watch_flush enter 2024-02-23T00:35:25.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 10 librados: watch_flush exit 2024-02-23T00:35:25.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 20 client.37601.objecter shutdown clearing up homeless session... 2024-02-23T00:35:25.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 10 client.37601.objecter successfully canceled tick 2024-02-23T00:35:25.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 shutdown_connections 2024-02-23T00:35:25.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 -- 172.21.15.87:0/1981431750 wait complete. 2024-02-23T00:35:25.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff0b0062c90 msgr2=0x7ff0b0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:25.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/3806659717 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff0b0062c90 0x7ff0b0065150 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 rx=0x7ff0c8109850 tx=0x7ff0bc009040).stop 2024-02-23T00:35:25.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0c80f8cb0 msgr2=0x7ff0c810df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:25.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/3806659717 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0c80f8cb0 0x7ff0c810df10 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7ff0b80091f0 tx=0x7ff0b8009cb0).stop 2024-02-23T00:35:25.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7f0299b2e700 1 librados: shutdown 2024-02-23T00:35:25.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 shutdown_connections 2024-02-23T00:35:25.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/3806659717 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff0b0062c90 0x7ff0b0065150 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:25.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/3806659717 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0c8102210 0x7ff0c81080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:35:25.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/3806659717 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0c80f8cb0 0x7ff0c810df10 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:25.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 --2- 172.21.15.87:0/3806659717 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0c80f9610 0x7ff0c810e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 >> 172.21.15.87:0/3806659717 conn(0x7ff0c80f2820 msgr2=0x7ff0c80f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 shutdown_connections 2024-02-23T00:35:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 -- 172.21.15.87:0/3806659717 wait complete. 2024-02-23T00:35:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:25.315+0000 7ff0cd53a700 1 librados: shutdown 2024-02-23T00:35:25.321 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:35:25.322 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 5 pg_num to 42 2024-02-23T00:35:25.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:23.636547+0000 mgr.y (mgr.24437) 350 : cluster [DBG] pgmap v541: 228 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 peering, 194 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1089 objects misplaced (1.194%); 0 B/s, 0 objects/s recovering 2024-02-23T00:35:25.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.649379+0000 mon.a (mon.0) 2423 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:25.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.739718+0000 mon.a (mon.0) 2424 : audit [INF] from='client.? 172.21.15.87:0/1981431750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:25.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.866132+0000 mon.c (mon.2) 414 : audit [INF] from='client.? 172.21.15.87:0/3806659717' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]: dispatch 2024-02-23T00:35:25.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:24.866446+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]: dispatch 2024-02-23T00:35:25.325 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:25.065916+0000 mon.a (mon.0) 2426 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:25.325 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:25.070484+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-49"}]': finished 2024-02-23T00:35:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:23.636547+0000 mgr.y (mgr.24437) 350 : cluster [DBG] pgmap v541: 228 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 peering, 194 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1089 objects misplaced (1.194%); 0 B/s, 0 objects/s recovering 2024-02-23T00:35:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.649379+0000 mon.a (mon.0) 2423 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.739718+0000 mon.a (mon.0) 2424 : audit [INF] from='client.? 172.21.15.87:0/1981431750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.866132+0000 mon.c (mon.2) 414 : audit [INF] from='client.? 172.21.15.87:0/3806659717' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]: dispatch 2024-02-23T00:35:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:24.866446+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]: dispatch 2024-02-23T00:35:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:25.065916+0000 mon.a (mon.0) 2426 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:25.070484+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-49"}]': finished 2024-02-23T00:35:25.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:23.636547+0000 mgr.y (mgr.24437) 350 : cluster [DBG] pgmap v541: 228 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 peering, 194 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1089 objects misplaced (1.194%); 0 B/s, 0 objects/s recovering 2024-02-23T00:35:25.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.649379+0000 mon.a (mon.0) 2423 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:25.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.739718+0000 mon.a (mon.0) 2424 : audit [INF] from='client.? 172.21.15.87:0/1981431750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:25.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.866132+0000 mon.c (mon.2) 414 : audit [INF] from='client.? 172.21.15.87:0/3806659717' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]: dispatch 2024-02-23T00:35:25.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:24.866446+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]: dispatch 2024-02-23T00:35:25.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:25.065916+0000 mon.a (mon.0) 2426 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:25.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:25.070484+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-49"}]': finished 2024-02-23T00:35:25.865 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:25.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:25.968 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:26.001 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:26.022 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:26.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:26.126 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:26.154 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:25.071364+0000 mon.b (mon.1) 323 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-49", "mode": "writeback"}]: dispatch 2024-02-23T00:35:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:25.313763+0000 mon.a (mon.0) 2428 : audit [INF] from='client.? 172.21.15.87:0/1981431750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:35:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:25.313828+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]': finished 2024-02-23T00:35:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:25.313882+0000 mon.a (mon.0) 2430 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-02-23T00:35:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:25.315036+0000 mon.a (mon.0) 2431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-49", "mode": "writeback"}]: dispatch 2024-02-23T00:35:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:25.638298+0000 mon.a (mon.0) 2432 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:26.067794+0000 mon.a (mon.0) 2433 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:26.072472+0000 mon.a (mon.0) 2434 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi087-82019-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi087-82019-64"}]': finished 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:26.072571+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-49", "mode": "writeback"}]': finished 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:26.072642+0000 mon.a (mon.0) 2436 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:26.072692+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:26.109821+0000 mon.b (mon.1) 324 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:26.110000+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:25.071364+0000 mon.b (mon.1) 323 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-49", "mode": "writeback"}]: dispatch 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:25.313763+0000 mon.a (mon.0) 2428 : audit [INF] from='client.? 172.21.15.87:0/1981431750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:35:26.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:25.313828+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]': finished 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:25.313882+0000 mon.a (mon.0) 2430 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:25.315036+0000 mon.a (mon.0) 2431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-49", "mode": "writeback"}]: dispatch 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:25.638298+0000 mon.a (mon.0) 2432 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:26.067794+0000 mon.a (mon.0) 2433 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:26.072472+0000 mon.a (mon.0) 2434 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi087-82019-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi087-82019-64"}]': finished 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:26.072571+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-49", "mode": "writeback"}]': finished 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:26.072642+0000 mon.a (mon.0) 2436 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:26.072692+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:26.109821+0000 mon.b (mon.1) 324 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:35:26.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:26.110000+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:35:26.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:25.071364+0000 mon.b (mon.1) 323 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-49", "mode": "writeback"}]: dispatch 2024-02-23T00:35:26.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:25.313763+0000 mon.a (mon.0) 2428 : audit [INF] from='client.? 172.21.15.87:0/1981431750' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:35:26.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:25.313828+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "42"}]': finished 2024-02-23T00:35:26.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:25.313882+0000 mon.a (mon.0) 2430 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-02-23T00:35:26.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:25.315036+0000 mon.a (mon.0) 2431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-49", "mode": "writeback"}]: dispatch 2024-02-23T00:35:26.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:25.638298+0000 mon.a (mon.0) 2432 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-23T00:35:26.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:26.067794+0000 mon.a (mon.0) 2433 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:35:26.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:26.072472+0000 mon.a (mon.0) 2434 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi087-82019-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi087-82019-64"}]': finished 2024-02-23T00:35:26.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:26.072571+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-49", "mode": "writeback"}]': finished 2024-02-23T00:35:26.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:26.072642+0000 mon.a (mon.0) 2436 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-23T00:35:26.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:26.072692+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-02-23T00:35:26.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:26.109821+0000 mon.b (mon.1) 324 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:35:26.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:26.110000+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:35:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:25.637397+0000 mgr.y (mgr.24437) 351 : cluster [DBG] pgmap v544: 228 pgs: 1 active+remapped+backfill_toofull, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 13/1119 objects misplaced (1.162%) 2024-02-23T00:35:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:27.076449+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:35:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:27.076534+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-02-23T00:35:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:27.082212+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:35:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:27.082644+0000 mon.a (mon.0) 2441 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:35:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:25.637397+0000 mgr.y (mgr.24437) 351 : cluster [DBG] pgmap v544: 228 pgs: 1 active+remapped+backfill_toofull, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 13/1119 objects misplaced (1.162%) 2024-02-23T00:35:27.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:27.076449+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:35:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:27.076534+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-02-23T00:35:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:27.082212+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:35:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:27.082644+0000 mon.a (mon.0) 2441 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:35:27.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:25.637397+0000 mgr.y (mgr.24437) 351 : cluster [DBG] pgmap v544: 228 pgs: 1 active+remapped+backfill_toofull, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 13/1119 objects misplaced (1.162%) 2024-02-23T00:35:27.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:27.076449+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:35:27.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:27.076534+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-02-23T00:35:27.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:27.082212+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:35:27.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:27.082644+0000 mon.a (mon.0) 2441 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:35:27.833 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:27.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:27.939 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:27.971 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:27.993 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:27.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:27.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:28.097 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:28.127 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:28.329 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:35:28.329 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:35:28.329 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:35:28.330 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:35:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:27.638982+0000 mon.a (mon.0) 2442 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-23T00:35:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:28.076554+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:35:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:28.076638+0000 mon.a (mon.0) 2444 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-23T00:35:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:28.076694+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-02-23T00:35:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:28.077595+0000 mon.c (mon.2) 415 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:28.077883+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:28.078123+0000 mon.b (mon.1) 326 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:35:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:28.078404+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:35:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:27.638982+0000 mon.a (mon.0) 2442 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-23T00:35:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:28.076554+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:35:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:28.076638+0000 mon.a (mon.0) 2444 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-23T00:35:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:28.076694+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-02-23T00:35:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:28.077595+0000 mon.c (mon.2) 415 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:28.077883+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:28.078123+0000 mon.b (mon.1) 326 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:35:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:28.078404+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:35:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:27.638982+0000 mon.a (mon.0) 2442 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-23T00:35:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:28.076554+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:35:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:28.076638+0000 mon.a (mon.0) 2444 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-23T00:35:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:28.076694+0000 mon.a (mon.0) 2445 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-02-23T00:35:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:28.077595+0000 mon.c (mon.2) 415 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:28.077883+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:28.078123+0000 mon.b (mon.1) 326 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:35:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:28.078404+0000 mon.a (mon.0) 2447 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:35:29.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-23T00:35:29.076 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:29.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:29.154 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:29.182 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:29.209 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:29.228 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:29.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:29.332 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:29.358 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:29.395 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:29.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:29.499 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:29.523 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:27.638012+0000 mgr.y (mgr.24437) 352 : cluster [DBG] pgmap v547: 246 pgs: 18 unknown, 1 active+remapped+backfill_toofull, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.5 KiB/s rd, 1.2 KiB/s wr, 17 op/s; 13/1119 objects misplaced (1.162%); 27 B/s, 1 objects/s recovering 2024-02-23T00:35:29.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:29.073567+0000 mon.a (mon.0) 2448 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:35:29.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:29.083507+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]': finished 2024-02-23T00:35:29.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:29.083582+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:35:29.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:29.083745+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-02-23T00:35:29.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:29.088599+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:35:29.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:29.089791+0000 mon.a (mon.0) 2452 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:35:29.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:29.090452+0000 mon.c (mon.2) 416 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:29.524 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:29.090972+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:29.525 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:29.544 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:29.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:29.648 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:29.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:27.638012+0000 mgr.y (mgr.24437) 352 : cluster [DBG] pgmap v547: 246 pgs: 18 unknown, 1 active+remapped+backfill_toofull, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.5 KiB/s rd, 1.2 KiB/s wr, 17 op/s; 13/1119 objects misplaced (1.162%); 27 B/s, 1 objects/s recovering 2024-02-23T00:35:29.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:29.073567+0000 mon.a (mon.0) 2448 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:35:29.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:29.083507+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]': finished 2024-02-23T00:35:29.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:29.083582+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:35:29.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:29.083745+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-02-23T00:35:29.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:29.088599+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:35:29.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:29.089791+0000 mon.a (mon.0) 2452 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:35:29.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:29.090452+0000 mon.c (mon.2) 416 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:29.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:29.090972+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:29.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:27.638012+0000 mgr.y (mgr.24437) 352 : cluster [DBG] pgmap v547: 246 pgs: 18 unknown, 1 active+remapped+backfill_toofull, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9.5 KiB/s rd, 1.2 KiB/s wr, 17 op/s; 13/1119 objects misplaced (1.162%); 27 B/s, 1 objects/s recovering 2024-02-23T00:35:29.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:29.073567+0000 mon.a (mon.0) 2448 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:35:29.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:29.083507+0000 mon.a (mon.0) 2449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi087-82019-64"}]': finished 2024-02-23T00:35:29.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:29.083582+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:35:29.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:29.083745+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-02-23T00:35:29.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:29.088599+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:35:29.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:29.089791+0000 mon.a (mon.0) 2452 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:35:29.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:29.090452+0000 mon.c (mon.2) 416 : audit [INF] from='client.? 172.21.15.87:0/2818687675' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:29.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:29.090972+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]: dispatch 2024-02-23T00:35:29.674 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:29.692 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:29.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:29.797 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:29.824 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:29.833 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:35:30.016 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:30.022 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:30.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:30.129 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:30.156 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:30.304 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:30.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:30.408 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:30.437 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:30.585 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:30.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:30.692 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:30.719 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:31.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:35:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:35:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:29.638597+0000 mgr.y (mgr.24437) 353 : cluster [DBG] pgmap v550: 238 pgs: 10 unknown, 1 active+remapped+backfill_toofull, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1119 objects misplaced (1.162%) 2024-02-23T00:35:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.078430+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:35:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.078483+0000 mon.a (mon.0) 2455 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]': finished 2024-02-23T00:35:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:30.078520+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-02-23T00:35:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.083977+0000 mon.b (mon.1) 328 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-23T00:35:31.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.084174+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-23T00:35:31.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.098365+0000 mon.c (mon.2) 417 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.098661+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.194 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.099198+0000 mon.c (mon.2) 418 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.195 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.099430+0000 mon.a (mon.0) 2459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.195 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.099803+0000 mon.c (mon.2) 419 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi087-82019-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:31.195 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.100047+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi087-82019-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:29.638597+0000 mgr.y (mgr.24437) 353 : cluster [DBG] pgmap v550: 238 pgs: 10 unknown, 1 active+remapped+backfill_toofull, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1119 objects misplaced (1.162%) 2024-02-23T00:35:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.078430+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:35:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.078483+0000 mon.a (mon.0) 2455 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]': finished 2024-02-23T00:35:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:30.078520+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-02-23T00:35:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.083977+0000 mon.b (mon.1) 328 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-23T00:35:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.084174+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-23T00:35:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.098365+0000 mon.c (mon.2) 417 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.098661+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.099198+0000 mon.c (mon.2) 418 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.099430+0000 mon.a (mon.0) 2459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.099803+0000 mon.c (mon.2) 419 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi087-82019-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.100047+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi087-82019-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:29.638597+0000 mgr.y (mgr.24437) 353 : cluster [DBG] pgmap v550: 238 pgs: 10 unknown, 1 active+remapped+backfill_toofull, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 13/1119 objects misplaced (1.162%) 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.078430+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.078483+0000 mon.a (mon.0) 2455 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi087-82019-64"}]': finished 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:30.078520+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.083977+0000 mon.b (mon.1) 328 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.084174+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-23T00:35:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.098365+0000 mon.c (mon.2) 417 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.098661+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.099198+0000 mon.c (mon.2) 418 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.099430+0000 mon.a (mon.0) 2459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:31.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.099803+0000 mon.c (mon.2) 419 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi087-82019-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:31.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.100047+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi087-82019-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:35:32.304 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:32] "GET /metrics HTTP/1.1" 200 187760 "" "Prometheus/2.33.4" 2024-02-23T00:35:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:30.859110+0000 mgr.y (mgr.24437) 354 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:31.080163+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "target_max_objects","val": "1"}]': finished 2024-02-23T00:35:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:31.080232+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi087-82019-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:31.080301+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-02-23T00:35:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:31.081104+0000 mon.c (mon.2) 420 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi087-82019-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:32.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:31.082108+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi087-82019-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:30.859110+0000 mgr.y (mgr.24437) 354 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:31.080163+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "target_max_objects","val": "1"}]': finished 2024-02-23T00:35:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:31.080232+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi087-82019-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:31.080301+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-02-23T00:35:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:31.081104+0000 mon.c (mon.2) 420 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi087-82019-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:31.082108+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi087-82019-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:32.603 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:35:32.603 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:35:32.603 INFO:teuthology.orchestra.run.smithi087.stdout: "complaint_time": 30, 2024-02-23T00:35:32.604 INFO:teuthology.orchestra.run.smithi087.stdout: "num_blocked_ops": 0 2024-02-23T00:35:32.604 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:35:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:30.859110+0000 mgr.y (mgr.24437) 354 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:31.080163+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-49","var": "target_max_objects","val": "1"}]': finished 2024-02-23T00:35:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:31.080232+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi087-82019-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:35:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:31.080301+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-02-23T00:35:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:31.081104+0000 mon.c (mon.2) 420 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi087-82019-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:31.082108+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi087-82019-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:33.435 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-23T00:35:33.478 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:31.639195+0000 mgr.y (mgr.24437) 355 : cluster [DBG] pgmap v553: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 13/1131 objects misplaced (1.149%); 0 B/s, 0 objects/s recovering 2024-02-23T00:35:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:32.304272+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-02-23T00:35:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:32.304538+0000 mon.a (mon.0) 2466 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-23T00:35:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:33.072873+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi087-82019-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi087-82019-65"}]': finished 2024-02-23T00:35:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:33.072967+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-02-23T00:35:33.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:31.639195+0000 mgr.y (mgr.24437) 355 : cluster [DBG] pgmap v553: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 13/1131 objects misplaced (1.149%); 0 B/s, 0 objects/s recovering 2024-02-23T00:35:33.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:32.304272+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-02-23T00:35:33.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:32.304538+0000 mon.a (mon.0) 2466 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-23T00:35:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:33.072873+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi087-82019-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi087-82019-65"}]': finished 2024-02-23T00:35:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:33.072967+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-02-23T00:35:33.614 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:35:33.614 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:35:33.615 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-23T00:35:33.615 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-02-23T00:35:33.615 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-23T00:35:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:31.639195+0000 mgr.y (mgr.24437) 355 : cluster [DBG] pgmap v553: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 13/1131 objects misplaced (1.149%); 0 B/s, 0 objects/s recovering 2024-02-23T00:35:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:32.304272+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-02-23T00:35:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:32.304538+0000 mon.a (mon.0) 2466 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-23T00:35:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:33.072873+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi087-82019-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi087-82019-65"}]': finished 2024-02-23T00:35:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:33.072967+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-02-23T00:35:33.769 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:33.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:33.873 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:33.906 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:33.928 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:33.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:34.032 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:34.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 -- 172.21.15.87:0/4093691739 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 msgr2=0x7f91ec104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 --2- 172.21.15.87:0/4093691739 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 0x7f91ec104a20 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 rx=0x7f91e4009be0 tx=0x7f91e4004d10).stop 2024-02-23T00:35:34.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 -- 172.21.15.87:0/4093691739 shutdown_connections 2024-02-23T00:35:34.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 --2- 172.21.15.87:0/4093691739 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 0x7f91ec104a20 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 --2- 172.21.15.87:0/4093691739 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91ec0fdbb0 0x7f91ec0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 --2- 172.21.15.87:0/4093691739 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91ec0fe510 0x7f91ec0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 -- 172.21.15.87:0/4093691739 >> 172.21.15.87:0/4093691739 conn(0x7f91ec0f3800 msgr2=0x7f91ec0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:34.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 -- 172.21.15.87:0/4093691739 shutdown_connections 2024-02-23T00:35:34.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 -- 172.21.15.87:0/4093691739 wait complete. 2024-02-23T00:35:34.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 librados: starting msgr at 2024-02-23T00:35:34.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 librados: starting objecter 2024-02-23T00:35:34.063 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 Processor -- start 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 -- start start 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 librados: setting wanted keys 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.039+0000 7f91f428c700 1 librados: calling monclient init 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f428c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 0x7f91ec06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f428c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91ec0fdbb0 0x7f91ec06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f428c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91ec0fe510 0x7f91ec06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f428c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec1163e0 con 0x7f91ec0fe510 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f428c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec116550 con 0x7f91ec0fdbb0 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f428c700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec116710 con 0x7f91ec0f8420 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2028700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 0x7f91ec06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2028700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 0x7f91ec06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33446/0 (socket says 172.21.15.87:33446) 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2028700 1 -- 172.21.15.87:0/2166711042 learned_addr learned my addr 172.21.15.87:0/2166711042 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2829700 1 --2- 172.21.15.87:0/2166711042 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91ec0fe510 0x7f91ec06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f1827700 1 --2- 172.21.15.87:0/2166711042 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91ec0fdbb0 0x7f91ec06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2028700 1 -- 172.21.15.87:0/2166711042 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91ec0fdbb0 msgr2=0x7f91ec06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2028700 1 --2- 172.21.15.87:0/2166711042 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f91ec0fdbb0 0x7f91ec06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2028700 1 -- 172.21.15.87:0/2166711042 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91ec0fe510 msgr2=0x7f91ec06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2028700 1 --2- 172.21.15.87:0/2166711042 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91ec0fe510 0x7f91ec06bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2028700 1 -- 172.21.15.87:0/2166711042 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e8005040 con 0x7f91ec0f8420 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2829700 1 --2- 172.21.15.87:0/2166711042 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f91ec0fe510 0x7f91ec06bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f2028700 1 --2- 172.21.15.87:0/2166711042 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 0x7f91ec06b0d0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f91e80048f0 tx=0x7f91e8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91e3779700 10 client.?.objecter ms_handle_connect 0x7f91ec0f8420 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91e3779700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e8007b00 con 0x7f91ec0f8420 2024-02-23T00:35:34.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e40097c0 con 0x7f91ec0f8420 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.040+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91e800f040 con 0x7f91ec0f8420 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.041+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e8014920 con 0x7f91ec0f8420 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.041+0000 7f91f428c700 10 client.37613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.041+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec116dd0 con 0x7f91ec0f8420 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.041+0000 7f91f428c700 1 librados: init done 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.041+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f91ec0606d0 con 0x7f91ec0f8420 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.042+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f91e8014d50 con 0x7f91ec0f8420 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.042+0000 7f91e3779700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6800/1231597463 conn(0x7f91d8062c90 0x7f91d8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.042+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(382..382 src has 173..382) v4 ==== 8923+0+0 (secure 0 0 0) 0x7f91e807d080 con 0x7f91ec0f8420 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.042+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 osd_map(382..382 src has 173..382) v4 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.042+0000 7f91e3779700 3 client.37613.objecter handle_osd_map got epochs [382,382] > 0 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.042+0000 7f91e3779700 3 client.37613.objecter handle_osd_map decoding full epoch 382 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.042+0000 7f91f1827700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6800/1231597463 conn(0x7f91d8062c90 0x7f91d8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.042+0000 7f91e3779700 20 client.37613.objecter dump_active .. 0 homeless 2024-02-23T00:35:34.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.042+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v382) v1 ==== 54+0+16 (secure 0 0 0) 0x7f91e8084070 con 0x7f91ec0f8420 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 10 client.37613.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91e3779700 10 client.37613.objecter ms_handle_connect 0x7f91d8062c90 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f1827700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6800/1231597463 conn(0x7f91d8062c90 0x7f91d8065150 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 rx=0x7f91ec06c870 tx=0x7f91dc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 1 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6801/2153191863 conn(0x7f91ec0fee80 0x7f91ec0ff2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec05ee80 osd=0 3 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec05ee80 osd=0 3 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec05ee80 osd=0 3 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 1 2024-02-23T00:35:34.067 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec05ee80 osd=0 3 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 0 1 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec05ee80 osd=0 4 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 10 client.37613.objecter _send_command 1 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ec1071b0 con 0x7f91ec0fee80 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f2829700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6801/2153191863 conn(0x7f91ec0fee80 0x7f91ec0ff2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91f2829700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6801/2153191863 conn(0x7f91ec0fee80 0x7f91ec0ff2a0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.043+0000 7f91e3779700 10 client.37613.objecter ms_handle_connect 0x7f91ec0fee80 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.044+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91ec1071b0 con 0x7f91ec0fee80 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.044+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.044+0000 7f91e3779700 10 client.37613.objecter _finish_command 1 = system:0 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.044+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec05ee80 osd=0 4 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.044+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 0 1 2024-02-23T00:35:34.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 10 client.37613.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 2 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec05ee80 osd=0 3 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec05ee80 osd=0 3 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec05ee80 osd=0 3 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 2 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec05ee80 osd=0 3 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 0 2 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec05ee80 osd=0 4 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 10 client.37613.objecter _send_command 2 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.059+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91ec0273c0 con 0x7f91ec0fee80 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91ec0273c0 con 0x7f91ec0fee80 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91e3779700 10 client.37613.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec05ee80 osd=0 4 2024-02-23T00:35:34.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 0 2 2024-02-23T00:35:34.070 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:35:34.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 10 client.37613.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:34.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 3 2024-02-23T00:35:34.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6805/685354171 conn(0x7f91ec0598f0 0x7f91ec059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec028710 osd=1 3 2024-02-23T00:35:34.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec028710 osd=1 3 2024-02-23T00:35:34.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec028710 osd=1 3 2024-02-23T00:35:34.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 3 2024-02-23T00:35:34.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec028710 osd=1 3 2024-02-23T00:35:34.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 1 3 2024-02-23T00:35:34.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec028710 osd=1 4 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 10 client.37613.objecter _send_command 3 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ec11d770 con 0x7f91ec0598f0 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.061+0000 7f91f2028700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6805/685354171 conn(0x7f91ec0598f0 0x7f91ec059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.062+0000 7f91e3779700 10 client.37613.objecter ms_handle_connect 0x7f91ec0598f0 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.062+0000 7f91f2028700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6805/685354171 conn(0x7f91ec0598f0 0x7f91ec059cf0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.062+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91ec11d770 con 0x7f91ec0598f0 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.062+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.062+0000 7f91e3779700 10 client.37613.objecter _finish_command 3 = system:0 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.062+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec028710 osd=1 4 2024-02-23T00:35:34.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.062+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 1 3 2024-02-23T00:35:34.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 10 client.37613.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:34.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 4 2024-02-23T00:35:34.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec028710 osd=1 3 2024-02-23T00:35:34.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec028710 osd=1 3 2024-02-23T00:35:34.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec028710 osd=1 3 2024-02-23T00:35:34.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 4 2024-02-23T00:35:34.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec028710 osd=1 3 2024-02-23T00:35:34.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 1 4 2024-02-23T00:35:34.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec028710 osd=1 4 2024-02-23T00:35:34.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 10 client.37613.objecter _send_command 4 2024-02-23T00:35:34.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.090+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91ec059360 con 0x7f91ec0598f0 2024-02-23T00:35:34.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91ec059360 con 0x7f91ec0598f0 2024-02-23T00:35:34.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:34.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91e3779700 10 client.37613.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec028710 osd=1 4 2024-02-23T00:35:34.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 1 4 2024-02-23T00:35:34.095 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:35:34.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 10 client.37613.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:34.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 5 2024-02-23T00:35:34.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6809/3605394228 conn(0x7f91ec028270 0x7f91ec11c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec059770 osd=2 3 2024-02-23T00:35:34.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec059770 osd=2 3 2024-02-23T00:35:34.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec059770 osd=2 3 2024-02-23T00:35:34.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 5 2024-02-23T00:35:34.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec059770 osd=2 3 2024-02-23T00:35:34.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 2 5 2024-02-23T00:35:34.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec059770 osd=2 4 2024-02-23T00:35:34.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 10 client.37613.objecter _send_command 5 2024-02-23T00:35:34.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.092+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ec11c680 con 0x7f91ec028270 2024-02-23T00:35:34.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.093+0000 7f91f1827700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6809/3605394228 conn(0x7f91ec028270 0x7f91ec11c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.093+0000 7f91f1827700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6809/3605394228 conn(0x7f91ec028270 0x7f91ec11c190 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.093+0000 7f91e3779700 10 client.37613.objecter ms_handle_connect 0x7f91ec028270 2024-02-23T00:35:34.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.094+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91ec11c680 con 0x7f91ec028270 2024-02-23T00:35:34.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.094+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:35:34.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.094+0000 7f91e3779700 10 client.37613.objecter _finish_command 5 = system:0 2024-02-23T00:35:34.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.094+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec059770 osd=2 4 2024-02-23T00:35:34.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.094+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 2 5 2024-02-23T00:35:34.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 10 client.37613.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:34.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 6 2024-02-23T00:35:34.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec059770 osd=2 3 2024-02-23T00:35:34.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec059770 osd=2 3 2024-02-23T00:35:34.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec059770 osd=2 3 2024-02-23T00:35:34.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 6 2024-02-23T00:35:34.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec059770 osd=2 3 2024-02-23T00:35:34.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 2 6 2024-02-23T00:35:34.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec059770 osd=2 4 2024-02-23T00:35:34.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 10 client.37613.objecter _send_command 6 2024-02-23T00:35:34.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.109+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91ec027830 con 0x7f91ec028270 2024-02-23T00:35:34.113 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:35:34.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91ec027830 con 0x7f91ec028270 2024-02-23T00:35:34.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:34.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91e3779700 10 client.37613.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec059770 osd=2 4 2024-02-23T00:35:34.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 2 6 2024-02-23T00:35:34.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 10 client.37613.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:34.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 7 2024-02-23T00:35:34.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6813/128059790 conn(0x7f91ec11f8f0 0x7f91ec121ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec11d070 osd=3 3 2024-02-23T00:35:34.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11d070 osd=3 3 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec11d070 osd=3 3 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 7 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec11d070 osd=3 3 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 3 7 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11d070 osd=3 4 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 10 client.37613.objecter _send_command 7 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ec124a90 con 0x7f91ec11f8f0 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f2829700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6813/128059790 conn(0x7f91ec11f8f0 0x7f91ec121ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91e3779700 10 client.37613.objecter ms_handle_connect 0x7f91ec11f8f0 2024-02-23T00:35:34.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.111+0000 7f91f2829700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6813/128059790 conn(0x7f91ec11f8f0 0x7f91ec121ce0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.112+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91ec124a90 con 0x7f91ec11f8f0 2024-02-23T00:35:34.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.112+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:35:34.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.112+0000 7f91e3779700 10 client.37613.objecter _finish_command 7 = system:0 2024-02-23T00:35:34.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.112+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec11d070 osd=3 4 2024-02-23T00:35:34.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.112+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 3 7 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 10 client.37613.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 8 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec11d070 osd=3 3 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11d070 osd=3 3 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec11d070 osd=3 3 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 8 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec11d070 osd=3 3 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 3 8 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11d070 osd=3 4 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 10 client.37613.objecter _send_command 8 2024-02-23T00:35:34.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.126+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91ec11cae0 con 0x7f91ec11f8f0 2024-02-23T00:35:34.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91ec11cae0 con 0x7f91ec11f8f0 2024-02-23T00:35:34.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:34.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91e3779700 10 client.37613.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec11d070 osd=3 4 2024-02-23T00:35:34.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 3 8 2024-02-23T00:35:34.130 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:35:34.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 10 client.37613.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:34.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 9 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6800/4175844757 conn(0x7f91ec11bcf0 0x7f91ec122270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec11cd20 osd=4 3 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11cd20 osd=4 3 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec11cd20 osd=4 3 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 9 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec11cd20 osd=4 3 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 4 9 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11cd20 osd=4 4 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 10 client.37613.objecter _send_command 9 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ec122760 con 0x7f91ec11bcf0 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.128+0000 7f91f2028700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6800/4175844757 conn(0x7f91ec11bcf0 0x7f91ec122270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.129+0000 7f91e3779700 10 client.37613.objecter ms_handle_connect 0x7f91ec11bcf0 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.129+0000 7f91f2028700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6800/4175844757 conn(0x7f91ec11bcf0 0x7f91ec122270 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.129+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91ec122760 con 0x7f91ec11bcf0 2024-02-23T00:35:34.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.129+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:35:34.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.129+0000 7f91e3779700 10 client.37613.objecter _finish_command 9 = system:0 2024-02-23T00:35:34.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.129+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec11cd20 osd=4 4 2024-02-23T00:35:34.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.129+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 4 9 2024-02-23T00:35:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 10 client.37613.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 10 2024-02-23T00:35:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec11cd20 osd=4 3 2024-02-23T00:35:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11cd20 osd=4 3 2024-02-23T00:35:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec11cd20 osd=4 3 2024-02-23T00:35:34.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 10 2024-02-23T00:35:34.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec11cd20 osd=4 3 2024-02-23T00:35:34.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 4 10 2024-02-23T00:35:34.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11cd20 osd=4 4 2024-02-23T00:35:34.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 10 client.37613.objecter _send_command 10 2024-02-23T00:35:34.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.144+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91ec122760 con 0x7f91ec11bcf0 2024-02-23T00:35:34.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91ec122760 con 0x7f91ec11bcf0 2024-02-23T00:35:34.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:34.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91e3779700 10 client.37613.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec11cd20 osd=4 4 2024-02-23T00:35:34.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 4 10 2024-02-23T00:35:34.148 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:35:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 10 client.37613.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 11 2024-02-23T00:35:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6804/2550013795 conn(0x7f91ec124010 0x7f91ec124450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec122760 osd=5 3 2024-02-23T00:35:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec122760 osd=5 3 2024-02-23T00:35:34.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec122760 osd=5 3 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 11 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec122760 osd=5 3 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 5 11 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec122760 osd=5 4 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 10 client.37613.objecter _send_command 11 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ec12b430 con 0x7f91ec124010 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f1827700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6804/2550013795 conn(0x7f91ec124010 0x7f91ec124450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91e3779700 10 client.37613.objecter ms_handle_connect 0x7f91ec124010 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.146+0000 7f91f1827700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6804/2550013795 conn(0x7f91ec124010 0x7f91ec124450 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.147+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91ec12b430 con 0x7f91ec124010 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.147+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.147+0000 7f91e3779700 10 client.37613.objecter _finish_command 11 = system:0 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.147+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec122760 osd=5 4 2024-02-23T00:35:34.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.147+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 5 11 2024-02-23T00:35:34.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 10 client.37613.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:34.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 12 2024-02-23T00:35:34.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec122760 osd=5 3 2024-02-23T00:35:34.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec122760 osd=5 3 2024-02-23T00:35:34.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec122760 osd=5 3 2024-02-23T00:35:34.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 12 2024-02-23T00:35:34.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec122760 osd=5 3 2024-02-23T00:35:34.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 5 12 2024-02-23T00:35:34.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec122760 osd=5 4 2024-02-23T00:35:34.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 10 client.37613.objecter _send_command 12 2024-02-23T00:35:34.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.161+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91ec128cd0 con 0x7f91ec124010 2024-02-23T00:35:34.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.163+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91ec128cd0 con 0x7f91ec124010 2024-02-23T00:35:34.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.163+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:34.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.163+0000 7f91e3779700 10 client.37613.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.163+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec122760 osd=5 4 2024-02-23T00:35:34.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.163+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 5 12 2024-02-23T00:35:34.166 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:35:34.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 10 client.37613.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:34.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 13 2024-02-23T00:35:34.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6808/2509408498 conn(0x7f91ec1293b0 0x7f91ec129810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec128cd0 osd=6 3 2024-02-23T00:35:34.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec128cd0 osd=6 3 2024-02-23T00:35:34.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec128cd0 osd=6 3 2024-02-23T00:35:34.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 13 2024-02-23T00:35:34.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec128cd0 osd=6 3 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 6 13 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec128cd0 osd=6 4 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 10 client.37613.objecter _send_command 13 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ec12b430 con 0x7f91ec1293b0 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f2829700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6808/2509408498 conn(0x7f91ec1293b0 0x7f91ec129810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91e3779700 10 client.37613.objecter ms_handle_connect 0x7f91ec1293b0 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.164+0000 7f91f2829700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6808/2509408498 conn(0x7f91ec1293b0 0x7f91ec129810 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.165+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91ec12b430 con 0x7f91ec1293b0 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.165+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.165+0000 7f91e3779700 10 client.37613.objecter _finish_command 13 = system:0 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.165+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec128cd0 osd=6 4 2024-02-23T00:35:34.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.165+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 6 13 2024-02-23T00:35:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 10 client.37613.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 14 2024-02-23T00:35:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec128cd0 osd=6 3 2024-02-23T00:35:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec128cd0 osd=6 3 2024-02-23T00:35:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec128cd0 osd=6 3 2024-02-23T00:35:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 14 2024-02-23T00:35:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec128cd0 osd=6 3 2024-02-23T00:35:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 6 14 2024-02-23T00:35:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec128cd0 osd=6 4 2024-02-23T00:35:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 10 client.37613.objecter _send_command 14 2024-02-23T00:35:34.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.179+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91ec12b430 con 0x7f91ec1293b0 2024-02-23T00:35:34.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.181+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91ec12b430 con 0x7f91ec1293b0 2024-02-23T00:35:34.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.181+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:34.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.181+0000 7f91e3779700 10 client.37613.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.181+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec128cd0 osd=6 4 2024-02-23T00:35:34.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.181+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 6 14 2024-02-23T00:35:34.184 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:35:34.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 10 client.37613.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:34.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 15 2024-02-23T00:35:34.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6812/373787000 conn(0x7f91ec12db70 0x7f91ec12dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:34.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec12b430 osd=7 3 2024-02-23T00:35:34.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec12b430 osd=7 3 2024-02-23T00:35:34.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec12b430 osd=7 3 2024-02-23T00:35:34.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 15 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec12b430 osd=7 3 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 7 15 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec12b430 osd=7 4 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 10 client.37613.objecter _send_command 15 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f91ec12e650 con 0x7f91ec12db70 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f2028700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6812/373787000 conn(0x7f91ec12db70 0x7f91ec12dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91e3779700 10 client.37613.objecter ms_handle_connect 0x7f91ec12db70 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.182+0000 7f91f2028700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6812/373787000 conn(0x7f91ec12db70 0x7f91ec12dfd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.183+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f91ec12e650 con 0x7f91ec12db70 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.183+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.183+0000 7f91e3779700 10 client.37613.objecter _finish_command 15 = system:0 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.183+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec12b430 osd=7 4 2024-02-23T00:35:34.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.183+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 7 15 2024-02-23T00:35:34.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:35:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:35:34.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 10 client.37613.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:34.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign -1 16 2024-02-23T00:35:34.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec12b430 osd=7 3 2024-02-23T00:35:34.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec12b430 osd=7 3 2024-02-23T00:35:34.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 20 client.37613.objecter _get_session s=0x7f91ec12b430 osd=7 3 2024-02-23T00:35:34.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 15 client.37613.objecter _session_command_op_remove -1 16 2024-02-23T00:35:34.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 20 client.37613.objecter get_session s=0x7f91ec12b430 osd=7 3 2024-02-23T00:35:34.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 15 client.37613.objecter _session_command_op_assign 7 16 2024-02-23T00:35:34.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec12b430 osd=7 4 2024-02-23T00:35:34.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 10 client.37613.objecter _send_command 16 2024-02-23T00:35:34.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.197+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f91ec12e650 con 0x7f91ec12db70 2024-02-23T00:35:34.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.199+0000 7f91e3779700 1 -- 172.21.15.87:0/2166711042 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f91ec12e650 con 0x7f91ec12db70 2024-02-23T00:35:34.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.199+0000 7f91e3779700 10 client.37613.objecter ms_dispatch 0x7f91ec001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:34.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.199+0000 7f91e3779700 10 client.37613.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.199+0000 7f91e3779700 20 client.37613.objecter put_session s=0x7f91ec12b430 osd=7 4 2024-02-23T00:35:34.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.199+0000 7f91e3779700 15 client.37613.objecter _session_command_op_remove 7 16 2024-02-23T00:35:34.202 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:35:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.199+0000 7f91f428c700 10 librados: watch_flush enter 2024-02-23T00:35:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 librados: watch_flush exit 2024-02-23T00:35:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 client.37613.objecter close_session for osd.0 2024-02-23T00:35:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6801/2153191863 conn(0x7f91ec0fee80 msgr2=0x7f91ec0ff2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6801/2153191863 conn(0x7f91ec0fee80 0x7f91ec0ff2a0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec05ee80 osd=0 1 2024-02-23T00:35:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 client.37613.objecter close_session for osd.1 2024-02-23T00:35:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6805/685354171 conn(0x7f91ec0598f0 msgr2=0x7f91ec059cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6805/685354171 conn(0x7f91ec0598f0 0x7f91ec059cf0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec028710 osd=1 1 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 client.37613.objecter close_session for osd.2 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6809/3605394228 conn(0x7f91ec028270 msgr2=0x7f91ec11c190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6809/3605394228 conn(0x7f91ec028270 0x7f91ec11c190 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec059770 osd=2 1 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 client.37613.objecter close_session for osd.3 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6813/128059790 conn(0x7f91ec11f8f0 msgr2=0x7f91ec121ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f2829700 1 -- 172.21.15.87:0/2166711042 reap_dead start 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6813/128059790 conn(0x7f91ec11f8f0 0x7f91ec121ce0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11d070 osd=3 1 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 client.37613.objecter close_session for osd.4 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6800/4175844757 conn(0x7f91ec11bcf0 msgr2=0x7f91ec122270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6800/4175844757 conn(0x7f91ec11bcf0 0x7f91ec122270 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec11cd20 osd=4 1 2024-02-23T00:35:34.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 client.37613.objecter close_session for osd.5 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6804/2550013795 conn(0x7f91ec124010 msgr2=0x7f91ec124450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6804/2550013795 conn(0x7f91ec124010 0x7f91ec124450 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec122760 osd=5 1 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 client.37613.objecter close_session for osd.6 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6808/2509408498 conn(0x7f91ec1293b0 msgr2=0x7f91ec129810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6808/2509408498 conn(0x7f91ec1293b0 0x7f91ec129810 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec128cd0 osd=6 1 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 client.37613.objecter close_session for osd.7 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6812/373787000 conn(0x7f91ec12db70 msgr2=0x7f91ec12dfd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.171:6812/373787000 conn(0x7f91ec12db70 0x7f91ec12dfd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 20 client.37613.objecter put_session s=0x7f91ec12b430 osd=7 1 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 20 client.37613.objecter shutdown clearing up homeless session... 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 10 client.37613.objecter successfully canceled tick 2024-02-23T00:35:34.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f2829700 1 -- 172.21.15.87:0/2166711042 reap_dead start 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6800/1231597463 conn(0x7f91d8062c90 msgr2=0x7f91d8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6800/1231597463 conn(0x7f91d8062c90 0x7f91d8065150 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 rx=0x7f91ec06c870 tx=0x7f91dc00b040).stop 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 msgr2=0x7f91ec06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 0x7f91ec06b0d0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 rx=0x7f91e80048f0 tx=0x7f91e8009cb0).stop 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 shutdown_connections 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> v2:172.21.15.87:6800/1231597463 conn(0x7f91d8062c90 0x7f91d8065150 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 --2- 172.21.15.87:0/2166711042 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f91ec0f8420 0x7f91ec06b0d0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 >> 172.21.15.87:0/2166711042 conn(0x7f91ec0f3800 msgr2=0x7f91ec0f5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 shutdown_connections 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 -- 172.21.15.87:0/2166711042 wait complete. 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:34.200+0000 7f91f428c700 1 librados: shutdown 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.207 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.208 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.208 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.208 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.208 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:35:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:34.043915+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? 172.21.15.87:0/2166711042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:34.074209+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-02-23T00:35:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:34.043915+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? 172.21.15.87:0/2166711042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:34.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:34.074209+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-02-23T00:35:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:34.043915+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? 172.21.15.87:0/2166711042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:34.074209+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-02-23T00:35:35.125 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:35.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:35.229 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:35.262 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:35.283 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:35.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:35.387 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:35.415 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:35.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:33.639585+0000 mgr.y (mgr.24437) 356 : cluster [DBG] pgmap v556: 246 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 13/1131 objects misplaced (1.149%); 0 B/s, 0 objects/s recovering 2024-02-23T00:35:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:35.073553+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-02-23T00:35:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:35.076250+0000 mon.c (mon.2) 422 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:35.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:35.076625+0000 mon.a (mon.0) 2471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:33.639585+0000 mgr.y (mgr.24437) 356 : cluster [DBG] pgmap v556: 246 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 13/1131 objects misplaced (1.149%); 0 B/s, 0 objects/s recovering 2024-02-23T00:35:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:35.073553+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-02-23T00:35:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:35.076250+0000 mon.c (mon.2) 422 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:35.076625+0000 mon.a (mon.0) 2471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:35.615 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:35.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:33.639585+0000 mgr.y (mgr.24437) 356 : cluster [DBG] pgmap v556: 246 pgs: 8 unknown, 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 13/1131 objects misplaced (1.149%); 0 B/s, 0 objects/s recovering 2024-02-23T00:35:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:35.073553+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-02-23T00:35:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:35.076250+0000 mon.c (mon.2) 422 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:35.076625+0000 mon.a (mon.0) 2471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:35.720 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:35.747 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:35.765 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:36.297 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:36.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:36.401 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:36.433 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:36.455 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:36.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:36.559 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:36.590 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:36.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.786+0000 7fea76d79700 1 -- 172.21.15.87:0/2819969019 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700f7400 msgr2=0x7fea70103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:36.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.786+0000 7fea76d79700 1 --2- 172.21.15.87:0/2819969019 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700f7400 0x7fea70103a20 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 rx=0x7fea68009be0 tx=0x7fea68004d10).stop 2024-02-23T00:35:36.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.786+0000 7fea76d79700 1 -- 172.21.15.87:0/2819969019 shutdown_connections 2024-02-23T00:35:36.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.786+0000 7fea76d79700 1 --2- 172.21.15.87:0/2819969019 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700f7400 0x7fea70103a20 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:36.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.786+0000 7fea76d79700 1 --2- 172.21.15.87:0/2819969019 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea700fcbb0 0x7fea700fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:36.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.786+0000 7fea76d79700 1 --2- 172.21.15.87:0/2819969019 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700fd510 0x7fea700f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:36.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.786+0000 7fea76d79700 1 -- 172.21.15.87:0/2819969019 >> 172.21.15.87:0/2819969019 conn(0x7fea700f2820 msgr2=0x7fea700f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:36.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.786+0000 7fea76d79700 1 -- 172.21.15.87:0/2819969019 shutdown_connections 2024-02-23T00:35:36.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.786+0000 7fea76d79700 1 -- 172.21.15.87:0/2819969019 wait complete. 2024-02-23T00:35:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 librados: starting msgr at 2024-02-23T00:35:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 librados: starting objecter 2024-02-23T00:35:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 Processor -- start 2024-02-23T00:35:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 -- start start 2024-02-23T00:35:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 librados: setting wanted keys 2024-02-23T00:35:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 librados: calling monclient init 2024-02-23T00:35:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea700f7400 0x7fea700fdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700fcbb0 0x7fea700fe0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700fd510 0x7fea700fe590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:36.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea70116410 con 0x7fea700fd510 2024-02-23T00:35:36.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea70116580 con 0x7fea700f7400 2024-02-23T00:35:36.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea76d79700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea70116740 con 0x7fea700fcbb0 2024-02-23T00:35:36.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea6ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700fcbb0 0x7fea700fe0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:36.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea6ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700fcbb0 0x7fea700fe0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33470/0 (socket says 172.21.15.87:33470) 2024-02-23T00:35:36.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea6ffff700 1 -- 172.21.15.87:0/3530316052 learned_addr learned my addr 172.21.15.87:0/3530316052 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:36.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea75316700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700fd510 0x7fea700fe590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:36.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea74b15700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea700f7400 0x7fea700fdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:36.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea6ffff700 1 -- 172.21.15.87:0/3530316052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea700f7400 msgr2=0x7fea700fdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:36.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea6ffff700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea700f7400 0x7fea700fdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:36.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea6ffff700 1 -- 172.21.15.87:0/3530316052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700fd510 msgr2=0x7fea700fe590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:36.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea6ffff700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700fd510 0x7fea700fe590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:36.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea6ffff700 1 -- 172.21.15.87:0/3530316052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea680097c0 con 0x7fea700fcbb0 2024-02-23T00:35:36.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea75316700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700fd510 0x7fea700fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:36.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea74b15700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea700f7400 0x7fea700fdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:35:36.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.787+0000 7fea6ffff700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700fcbb0 0x7fea700fe0a0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 rx=0x7fea5c00e970 tx=0x7fea5c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:36.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.788+0000 7fea6dffb700 10 client.?.objecter ms_handle_connect 0x7fea700fcbb0 2024-02-23T00:35:36.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.788+0000 7fea6dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:36.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.788+0000 7fea6dffb700 1 -- 172.21.15.87:0/3530316052 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00ee60 con 0x7fea700fcbb0 2024-02-23T00:35:36.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.788+0000 7fea6dffb700 1 -- 172.21.15.87:0/3530316052 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea5c00f040 con 0x7fea700fcbb0 2024-02-23T00:35:36.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.788+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea70116a20 con 0x7fea700fcbb0 2024-02-23T00:35:36.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.788+0000 7fea6dffb700 1 -- 172.21.15.87:0/3530316052 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c007ea0 con 0x7fea700fcbb0 2024-02-23T00:35:36.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.788+0000 7fea76d79700 10 client.37619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:36.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.788+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea70116f60 con 0x7fea700fcbb0 2024-02-23T00:35:36.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.788+0000 7fea76d79700 1 librados: init done 2024-02-23T00:35:36.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.789+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7005fa60 con 0x7fea700fcbb0 2024-02-23T00:35:36.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.789+0000 7fea6dffb700 1 -- 172.21.15.87:0/3530316052 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fea5c00cb10 con 0x7fea700fcbb0 2024-02-23T00:35:36.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.790+0000 7fea6dffb700 1 --2- 172.21.15.87:0/3530316052 >> v2:172.21.15.87:6800/1231597463 conn(0x7fea60062c90 0x7fea60065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:36.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.790+0000 7fea6dffb700 1 -- 172.21.15.87:0/3530316052 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(385..385 src has 173..385) v4 ==== 8222+0+0 (secure 0 0 0) 0x7fea5c07cea0 con 0x7fea700fcbb0 2024-02-23T00:35:36.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.790+0000 7fea6dffb700 10 client.37619.objecter ms_dispatch 0x7fea70001550 osd_map(385..385 src has 173..385) v4 2024-02-23T00:35:36.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.790+0000 7fea6dffb700 3 client.37619.objecter handle_osd_map got epochs [385,385] > 0 2024-02-23T00:35:36.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.790+0000 7fea6dffb700 3 client.37619.objecter handle_osd_map decoding full epoch 385 2024-02-23T00:35:36.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.790+0000 7fea74b15700 1 --2- 172.21.15.87:0/3530316052 >> v2:172.21.15.87:6800/1231597463 conn(0x7fea60062c90 0x7fea60065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:36.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.790+0000 7fea6dffb700 20 client.37619.objecter dump_active .. 0 homeless 2024-02-23T00:35:36.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.790+0000 7fea6dffb700 10 client.37619.objecter ms_handle_connect 0x7fea60062c90 2024-02-23T00:35:36.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.790+0000 7fea74b15700 1 --2- 172.21.15.87:0/3530316052 >> v2:172.21.15.87:6800/1231597463 conn(0x7fea60062c90 0x7fea60065150 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 rx=0x7fea64004c80 tx=0x7fea6400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:36.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.794+0000 7fea6dffb700 1 -- 172.21.15.87:0/3530316052 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fea5c050380 con 0x7fea700fcbb0 2024-02-23T00:35:36.914 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:36.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:36.978+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fea7005ee80 con 0x7fea700fcbb0 2024-02-23T00:35:37.079 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:35.640501+0000 mgr.y (mgr.24437) 357 : cluster [DBG] pgmap v559: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:37.079 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:36.075621+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]': finished 2024-02-23T00:35:37.079 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:36.075694+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-02-23T00:35:37.079 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:36.077236+0000 mon.c (mon.2) 423 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:37.079 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:36.077564+0000 mon.a (mon.0) 2474 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:37.079 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:36.980434+0000 mon.c (mon.2) 424 : audit [INF] from='client.? 172.21.15.87:0/3530316052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:35:37.079 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:36.980715+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:35:37.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:35.640501+0000 mgr.y (mgr.24437) 357 : cluster [DBG] pgmap v559: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:37.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:36.075621+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]': finished 2024-02-23T00:35:37.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:36.075694+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-02-23T00:35:37.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:36.077236+0000 mon.c (mon.2) 423 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:37.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:36.077564+0000 mon.a (mon.0) 2474 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:37.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:36.980434+0000 mon.c (mon.2) 424 : audit [INF] from='client.? 172.21.15.87:0/3530316052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:35:37.080 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:36.980715+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:35:37.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.083+0000 7fea6dffb700 1 -- 172.21.15.87:0/3530316052 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v386) v1 ==== 91+0+0 (secure 0 0 0) 0x7fea5c019020 con 0x7fea700fcbb0 2024-02-23T00:35:37.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.085+0000 7fea76d79700 10 librados: watch_flush enter 2024-02-23T00:35:37.088 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:35:37.092 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7020 ms) 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6977 ms) 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6997 ms) 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7015 ms) 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7003 ms) 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139060 ms total) 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-02-23T00:35:37.093 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [==========] 56 tests from 4 test suites ran. (295050 ms total) 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stdout: api_aio_pp: [ PASSED ] 56 tests. 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82030 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82030 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82094 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82094 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82078 2024-02-23T00:35:37.094 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82078 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82054 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82054 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82150 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82150 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82022 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82022 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82166 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82166 2024-02-23T00:35:37.095 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82118 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82118 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82198 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82198 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82158 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82158 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82214 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82214 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82006 2024-02-23T00:35:37.096 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82006 2024-02-23T00:35:37.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82206 2024-02-23T00:35:37.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82206 2024-02-23T00:35:37.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82174 2024-02-23T00:35:37.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82174 2024-02-23T00:35:37.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ for t in "${!pids[@]}" 2024-02-23T00:35:37.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ pid=82086 2024-02-23T00:35:37.097 INFO:tasks.workunit.client.0.smithi087.stderr:+ wait 82086 2024-02-23T00:35:37.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.088+0000 7fea76d79700 10 librados: watch_flush exit 2024-02-23T00:35:37.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.088+0000 7fea76d79700 20 client.37619.objecter shutdown clearing up homeless session... 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.088+0000 7fea76d79700 10 client.37619.objecter successfully canceled tick 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.088+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 >> v2:172.21.15.87:6800/1231597463 conn(0x7fea60062c90 msgr2=0x7fea60065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.088+0000 7fea76d79700 1 --2- 172.21.15.87:0/3530316052 >> v2:172.21.15.87:6800/1231597463 conn(0x7fea60062c90 0x7fea60065150 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 rx=0x7fea64004c80 tx=0x7fea6400b040).stop 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.088+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700fcbb0 msgr2=0x7fea700fe0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700fcbb0 0x7fea700fe0a0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 rx=0x7fea5c00e970 tx=0x7fea5c00b040).stop 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 shutdown_connections 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 --2- 172.21.15.87:0/3530316052 >> v2:172.21.15.87:6800/1231597463 conn(0x7fea60062c90 0x7fea60065150 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fea700fd510 0x7fea700fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fea700f7400 0x7fea700fdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 --2- 172.21.15.87:0/3530316052 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fea700fcbb0 0x7fea700fe0a0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 >> 172.21.15.87:0/3530316052 conn(0x7fea700f2820 msgr2=0x7fea700f2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 shutdown_connections 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 -- 172.21.15.87:0/3530316052 wait complete. 2024-02-23T00:35:37.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:37.089+0000 7fea76d79700 1 librados: shutdown 2024-02-23T00:35:37.099 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:35:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:35.640501+0000 mgr.y (mgr.24437) 357 : cluster [DBG] pgmap v559: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:36.075621+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi087-82019-65"}]': finished 2024-02-23T00:35:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:36.075694+0000 mon.a (mon.0) 2473 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-02-23T00:35:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:36.077236+0000 mon.c (mon.2) 423 : audit [INF] from='client.? 172.21.15.87:0/1110810957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:36.077564+0000 mon.a (mon.0) 2474 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]: dispatch 2024-02-23T00:35:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:36.980434+0000 mon.c (mon.2) 424 : audit [INF] from='client.? 172.21.15.87:0/3530316052' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:35:37.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:36.980715+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:35:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:37.073030+0000 mon.a (mon.0) 2476 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:37.082966+0000 mon.a (mon.0) 2477 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]': finished 2024-02-23T00:35:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:37.083051+0000 mon.a (mon.0) 2478 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:35:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:37.083106+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-02-23T00:35:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:37.073030+0000 mon.a (mon.0) 2476 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:37.082966+0000 mon.a (mon.0) 2477 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]': finished 2024-02-23T00:35:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:37.083051+0000 mon.a (mon.0) 2478 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:35:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:37.083106+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-02-23T00:35:38.438 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:38.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:38.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:37.073030+0000 mon.a (mon.0) 2476 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:37.082966+0000 mon.a (mon.0) 2477 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi087-82019-65"}]': finished 2024-02-23T00:35:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:37.083051+0000 mon.a (mon.0) 2478 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:35:38.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:37.083106+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-02-23T00:35:38.542 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:38.574 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:38.595 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:38.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:38.698 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:38.727 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:39.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:37.640983+0000 mgr.y (mgr.24437) 358 : cluster [DBG] pgmap v562: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:39.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:37.640983+0000 mgr.y (mgr.24437) 358 : cluster [DBG] pgmap v562: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:39.367 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:39.376 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:39.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:37.640983+0000 mgr.y (mgr.24437) 358 : cluster [DBG] pgmap v562: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:39.481 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:39.509 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:39.651 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:39.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:39.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:39.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:39.784 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:39.803 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:39.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:39.910 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:39.938 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:39.946 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:35:39.946 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:35:39.956 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:39.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:40.061 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:40.072 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:35:40.088 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:40.142 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:40.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:40.246 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:40.278 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:40.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:38.616230+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:35:40.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:38.617176+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:35:40.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:39.032825+0000 osd.6 (osd.6) 135 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:35:40.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:39.035762+0000 osd.6 (osd.6) 136 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:35:40.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:39.109176+0000 osd.2 (osd.2) 143 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:35:40.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:39.110945+0000 osd.2 (osd.2) 144 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:35:40.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:39.202191+0000 osd.0 (osd.0) 179 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:35:40.300 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:39.205308+0000 osd.0 (osd.0) 180 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:39.649700+0000 mon.a (mon.0) 2480 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:38.616230+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:38.617176+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:39.032825+0000 osd.6 (osd.6) 135 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:39.035762+0000 osd.6 (osd.6) 136 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:39.109176+0000 osd.2 (osd.2) 143 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:39.110945+0000 osd.2 (osd.2) 144 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:39.202191+0000 osd.0 (osd.0) 179 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:39.205308+0000 osd.0 (osd.0) 180 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:35:40.301 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:39.649700+0000 mon.a (mon.0) 2480 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:40.302 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:40.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:40.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:40.404 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:40.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:38.616230+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:35:40.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:38.617176+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:35:40.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:39.032825+0000 osd.6 (osd.6) 135 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:35:40.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:39.035762+0000 osd.6 (osd.6) 136 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:35:40.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:39.109176+0000 osd.2 (osd.2) 143 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:35:40.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:39.110945+0000 osd.2 (osd.2) 144 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:35:40.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:39.202191+0000 osd.0 (osd.0) 179 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:35:40.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:39.205308+0000 osd.0 (osd.0) 180 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:35:40.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:39.649700+0000 mon.a (mon.0) 2480 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:40.432 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:40.453 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:40.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:40.557 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:40.585 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:40.658 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:40.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:40.667 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-23T00:35:40.762 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:40.789 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:40.988 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:40.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:41.093 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:41.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:35:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:35:41.126 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:41.145 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:41.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:41.251 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:41.277 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:41.296 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:41.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:41.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:39.641414+0000 mgr.y (mgr.24437) 359 : cluster [DBG] pgmap v563: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 919 B/s rd, 0 B/s wr, 1 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:41.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:39.868124+0000 osd.7 (osd.7) 183 : cluster [DBG] 22.d scrub starts 2024-02-23T00:35:41.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:39.871053+0000 osd.7 (osd.7) 184 : cluster [DBG] 22.d scrub ok 2024-02-23T00:35:41.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:40.899546+0000 mon.a (mon.0) 2481 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:35:41.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:41.075837+0000 mon.a (mon.0) 2482 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]}]: dispatch 2024-02-23T00:35:41.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:41.076023+0000 mon.a (mon.0) 2483 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-02-23T00:35:41.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:41.076165+0000 mon.a (mon.0) 2484 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "204.19", "id": [5, 7]}]: dispatch 2024-02-23T00:35:41.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:39.641414+0000 mgr.y (mgr.24437) 359 : cluster [DBG] pgmap v563: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 919 B/s rd, 0 B/s wr, 1 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:41.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:39.868124+0000 osd.7 (osd.7) 183 : cluster [DBG] 22.d scrub starts 2024-02-23T00:35:41.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:39.871053+0000 osd.7 (osd.7) 184 : cluster [DBG] 22.d scrub ok 2024-02-23T00:35:41.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:40.899546+0000 mon.a (mon.0) 2481 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:35:41.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:41.075837+0000 mon.a (mon.0) 2482 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]}]: dispatch 2024-02-23T00:35:41.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:41.076023+0000 mon.a (mon.0) 2483 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-02-23T00:35:41.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:41.076165+0000 mon.a (mon.0) 2484 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "204.19", "id": [5, 7]}]: dispatch 2024-02-23T00:35:41.399 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:41.423 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:39.641414+0000 mgr.y (mgr.24437) 359 : cluster [DBG] pgmap v563: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 919 B/s rd, 0 B/s wr, 1 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:41.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:39.868124+0000 osd.7 (osd.7) 183 : cluster [DBG] 22.d scrub starts 2024-02-23T00:35:41.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:39.871053+0000 osd.7 (osd.7) 184 : cluster [DBG] 22.d scrub ok 2024-02-23T00:35:41.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:40.899546+0000 mon.a (mon.0) 2481 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:35:41.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:41.075837+0000 mon.a (mon.0) 2482 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]}]: dispatch 2024-02-23T00:35:41.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:41.076023+0000 mon.a (mon.0) 2483 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-02-23T00:35:41.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:41.076165+0000 mon.a (mon.0) 2484 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "204.19", "id": [5, 7]}]: dispatch 2024-02-23T00:35:41.425 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:41.444 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:41.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:41.547 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:41.577 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:41.700 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:41.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:41.805 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:41.832 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:42.030 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:42.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:42.135 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:42.173 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:42.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:42] "GET /metrics HTTP/1.1" 200 187865 "" "Prometheus/2.33.4" 2024-02-23T00:35:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:40.823811+0000 osd.7 (osd.7) 185 : cluster [DBG] 2.b scrub starts 2024-02-23T00:35:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:40.824733+0000 osd.7 (osd.7) 186 : cluster [DBG] 2.b scrub ok 2024-02-23T00:35:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:40.861003+0000 mgr.y (mgr.24437) 360 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:41.068356+0000 osd.6 (osd.6) 137 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:41.071246+0000 osd.6 (osd.6) 138 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:41.096405+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:41.099746+0000 mon.a (mon.0) 2485 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]}]': finished 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:41.099825+0000 mon.a (mon.0) 2486 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:41.099877+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "204.19", "id": [5, 7]}]': finished 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:41.099921+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:41.109364+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:41.650062+0000 osd.1 (osd.1) 167 : cluster [DBG] 63.8 scrub starts 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:41.651496+0000 osd.1 (osd.1) 168 : cluster [DBG] 63.8 scrub ok 2024-02-23T00:35:42.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:40.823811+0000 osd.7 (osd.7) 185 : cluster [DBG] 2.b scrub starts 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:40.824733+0000 osd.7 (osd.7) 186 : cluster [DBG] 2.b scrub ok 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:40.861003+0000 mgr.y (mgr.24437) 360 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:41.068356+0000 osd.6 (osd.6) 137 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:41.071246+0000 osd.6 (osd.6) 138 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:41.096405+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:41.099746+0000 mon.a (mon.0) 2485 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]}]': finished 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:41.099825+0000 mon.a (mon.0) 2486 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:41.099877+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "204.19", "id": [5, 7]}]': finished 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:41.099921+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:41.109364+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:41.650062+0000 osd.1 (osd.1) 167 : cluster [DBG] 63.8 scrub starts 2024-02-23T00:35:42.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:41.651496+0000 osd.1 (osd.1) 168 : cluster [DBG] 63.8 scrub ok 2024-02-23T00:35:42.394 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:40.823811+0000 osd.7 (osd.7) 185 : cluster [DBG] 2.b scrub starts 2024-02-23T00:35:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:40.824733+0000 osd.7 (osd.7) 186 : cluster [DBG] 2.b scrub ok 2024-02-23T00:35:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:40.861003+0000 mgr.y (mgr.24437) 360 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:41.068356+0000 osd.6 (osd.6) 137 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:41.071246+0000 osd.6 (osd.6) 138 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:41.096405+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:41.099746+0000 mon.a (mon.0) 2485 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 0]}]': finished 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:41.099825+0000 mon.a (mon.0) 2486 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:41.099877+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "204.19", "id": [5, 7]}]': finished 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:41.099921+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:41.109364+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:41.650062+0000 osd.1 (osd.1) 167 : cluster [DBG] 63.8 scrub starts 2024-02-23T00:35:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:41.651496+0000 osd.1 (osd.1) 168 : cluster [DBG] 63.8 scrub ok 2024-02-23T00:35:42.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.579+0000 7eff5d592700 1 -- 172.21.15.87:0/4107581607 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff5806b9e0 msgr2=0x7eff58103fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:42.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.579+0000 7eff5d592700 1 --2- 172.21.15.87:0/4107581607 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff5806b9e0 0x7eff58103fb0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7eff4c004870 tx=0x7eff4c009cb0).stop 2024-02-23T00:35:42.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.579+0000 7eff5d592700 1 -- 172.21.15.87:0/4107581607 shutdown_connections 2024-02-23T00:35:42.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.579+0000 7eff5d592700 1 --2- 172.21.15.87:0/4107581607 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff581044a0 0x7eff58106890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:42.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.579+0000 7eff5d592700 1 --2- 172.21.15.87:0/4107581607 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff5806b110 0x7eff5806b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:42.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.579+0000 7eff5d592700 1 --2- 172.21.15.87:0/4107581607 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff5806b9e0 0x7eff58103fb0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:42.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.579+0000 7eff5d592700 1 -- 172.21.15.87:0/4107581607 >> 172.21.15.87:0/4107581607 conn(0x7eff580f3800 msgr2=0x7eff580f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:42.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.580+0000 7eff5d592700 1 -- 172.21.15.87:0/4107581607 shutdown_connections 2024-02-23T00:35:42.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.580+0000 7eff5d592700 1 -- 172.21.15.87:0/4107581607 wait complete. 2024-02-23T00:35:42.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.580+0000 7eff5d592700 1 librados: starting msgr at 2024-02-23T00:35:42.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.580+0000 7eff5d592700 1 librados: starting objecter 2024-02-23T00:35:42.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.580+0000 7eff5d592700 1 Processor -- start 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff5d592700 1 -- start start 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff5d592700 1 librados: setting wanted keys 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff5d592700 1 librados: calling monclient init 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff5d592700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff5806b110 0x7eff580f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff5d592700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff5806b9e0 0x7eff580f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff5d592700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff581044a0 0x7eff580f86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff5d592700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff580fbdd0 con 0x7eff5806b110 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff5d592700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5811ab50 con 0x7eff581044a0 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff5d592700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5811ad10 con 0x7eff5806b9e0 2024-02-23T00:35:42.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff567fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff5806b9e0 0x7eff580f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff56ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff5806b110 0x7eff580f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff567fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff5806b9e0 0x7eff580f81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44622/0 (socket says 172.21.15.87:44622) 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff567fc700 1 -- 172.21.15.87:0/1095092528 learned_addr learned my addr 172.21.15.87:0/1095092528 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff577fe700 1 --2- 172.21.15.87:0/1095092528 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff581044a0 0x7eff580f86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff567fc700 1 -- 172.21.15.87:0/1095092528 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff581044a0 msgr2=0x7eff580f86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff567fc700 1 --2- 172.21.15.87:0/1095092528 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff581044a0 0x7eff580f86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff567fc700 1 -- 172.21.15.87:0/1095092528 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff5806b110 msgr2=0x7eff580f7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff567fc700 1 --2- 172.21.15.87:0/1095092528 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff5806b110 0x7eff580f7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff567fc700 1 -- 172.21.15.87:0/1095092528 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff500097c0 con 0x7eff5806b9e0 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.581+0000 7eff56ffd700 1 --2- 172.21.15.87:0/1095092528 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff5806b110 0x7eff580f7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff567fc700 1 --2- 172.21.15.87:0/1095092528 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff5806b9e0 0x7eff580f81f0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 rx=0x7eff4c009ee0 tx=0x7eff4c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff3ffff700 10 client.?.objecter ms_handle_connect 0x7eff5806b9e0 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff3ffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff3ffff700 1 -- 172.21.15.87:0/1095092528 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c00e070 con 0x7eff5806b9e0 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff3ffff700 1 -- 172.21.15.87:0/1095092528 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff4c013d60 con 0x7eff5806b9e0 2024-02-23T00:35:42.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4c005040 con 0x7eff5806b9e0 2024-02-23T00:35:42.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff3ffff700 1 -- 172.21.15.87:0/1095092528 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c019070 con 0x7eff5806b9e0 2024-02-23T00:35:42.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff5d592700 10 client.37625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:42.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5811b1b0 con 0x7eff5806b9e0 2024-02-23T00:35:42.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.582+0000 7eff5d592700 1 librados: init done 2024-02-23T00:35:42.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.583+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff58109320 con 0x7eff5806b9e0 2024-02-23T00:35:42.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.583+0000 7eff3ffff700 1 -- 172.21.15.87:0/1095092528 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7eff4c0090d0 con 0x7eff5806b9e0 2024-02-23T00:35:42.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.583+0000 7eff3ffff700 1 --2- 172.21.15.87:0/1095092528 >> v2:172.21.15.87:6800/1231597463 conn(0x7eff40062c90 0x7eff40065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:42.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.584+0000 7eff3ffff700 1 -- 172.21.15.87:0/1095092528 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(388..388 src has 173..388) v4 ==== 8202+0+0 (secure 0 0 0) 0x7eff4c07ddd0 con 0x7eff5806b9e0 2024-02-23T00:35:42.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.584+0000 7eff3ffff700 10 client.37625.objecter ms_dispatch 0x7eff58001550 osd_map(388..388 src has 173..388) v4 2024-02-23T00:35:42.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.584+0000 7eff3ffff700 3 client.37625.objecter handle_osd_map got epochs [388,388] > 0 2024-02-23T00:35:42.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.584+0000 7eff3ffff700 3 client.37625.objecter handle_osd_map decoding full epoch 388 2024-02-23T00:35:42.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.584+0000 7eff56ffd700 1 --2- 172.21.15.87:0/1095092528 >> v2:172.21.15.87:6800/1231597463 conn(0x7eff40062c90 0x7eff40065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:42.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.584+0000 7eff3ffff700 20 client.37625.objecter dump_active .. 0 homeless 2024-02-23T00:35:42.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.584+0000 7eff3ffff700 10 client.37625.objecter ms_handle_connect 0x7eff40062c90 2024-02-23T00:35:42.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.584+0000 7eff56ffd700 1 --2- 172.21.15.87:0/1095092528 >> v2:172.21.15.87:6800/1231597463 conn(0x7eff40062c90 0x7eff40065150 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 rx=0x7eff480041d0 tx=0x7eff48009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:42.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.588+0000 7eff3ffff700 1 -- 172.21.15.87:0/1095092528 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7eff4c0acf60 con 0x7eff5806b9e0 2024-02-23T00:35:42.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:42.774+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) v1 -- 0x7eff580f91e0 con 0x7eff5806b9e0 2024-02-23T00:35:43.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.172+0000 7eff3ffff700 1 -- 172.21.15.87:0/1095092528 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v389) v1 ==== 133+0+0 (secure 0 0 0) 0x7eff4c051280 con 0x7eff5806b9e0 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 10 librados: watch_flush enter 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 10 librados: watch_flush exit 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 20 client.37625.objecter shutdown clearing up homeless session... 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 10 client.37625.objecter successfully canceled tick 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 >> v2:172.21.15.87:6800/1231597463 conn(0x7eff40062c90 msgr2=0x7eff40065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 --2- 172.21.15.87:0/1095092528 >> v2:172.21.15.87:6800/1231597463 conn(0x7eff40062c90 0x7eff40065150 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 rx=0x7eff480041d0 tx=0x7eff48009040).stop 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff5806b9e0 msgr2=0x7eff580f81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 --2- 172.21.15.87:0/1095092528 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff5806b9e0 0x7eff580f81f0 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 rx=0x7eff4c009ee0 tx=0x7eff4c00b040).stop 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 shutdown_connections 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 --2- 172.21.15.87:0/1095092528 >> v2:172.21.15.87:6800/1231597463 conn(0x7eff40062c90 0x7eff40065150 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 --2- 172.21.15.87:0/1095092528 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7eff581044a0 0x7eff580f86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 --2- 172.21.15.87:0/1095092528 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7eff5806b110 0x7eff580f7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 --2- 172.21.15.87:0/1095092528 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7eff5806b9e0 0x7eff580f81f0 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 >> 172.21.15.87:0/1095092528 conn(0x7eff580f3800 msgr2=0x7eff580f4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 shutdown_connections 2024-02-23T00:35:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 -- 172.21.15.87:0/1095092528 wait complete. 2024-02-23T00:35:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.175+0000 7eff5d592700 1 librados: shutdown 2024-02-23T00:35:43.179 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-23T00:35:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:41.259293+0000 osd.5 (osd.5) 153 : cluster [DBG] 6.19 scrub starts 2024-02-23T00:35:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:41.260228+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.19 scrub ok 2024-02-23T00:35:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:41.641877+0000 mgr.y (mgr.24437) 361 : cluster [DBG] pgmap v565: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:42.113871+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:35:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:42.114803+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:35:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:42.132434+0000 mon.a (mon.0) 2490 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:35:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:42.133093+0000 mon.a (mon.0) 2491 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:42.163646+0000 mon.a (mon.0) 2492 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:42.163728+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:42.167914+0000 mon.b (mon.1) 330 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:42.173805+0000 mon.a (mon.0) 2494 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:42.174252+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:42.775886+0000 mon.c (mon.2) 425 : audit [INF] from='client.? 172.21.15.87:0/1095092528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:42.776197+0000 mon.a (mon.0) 2496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:41.259293+0000 osd.5 (osd.5) 153 : cluster [DBG] 6.19 scrub starts 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:41.260228+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.19 scrub ok 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:41.641877+0000 mgr.y (mgr.24437) 361 : cluster [DBG] pgmap v565: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:42.113871+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:35:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:42.114803+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:35:43.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:42.132434+0000 mon.a (mon.0) 2490 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:35:43.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:42.133093+0000 mon.a (mon.0) 2491 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:35:43.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:42.163646+0000 mon.a (mon.0) 2492 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:35:43.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:42.163728+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-02-23T00:35:43.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:42.167914+0000 mon.b (mon.1) 330 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:43.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:42.173805+0000 mon.a (mon.0) 2494 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:43.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:42.174252+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:43.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:42.775886+0000 mon.c (mon.2) 425 : audit [INF] from='client.? 172.21.15.87:0/1095092528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:35:43.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:42.776197+0000 mon.a (mon.0) 2496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:35:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:41.259293+0000 osd.5 (osd.5) 153 : cluster [DBG] 6.19 scrub starts 2024-02-23T00:35:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:41.260228+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.19 scrub ok 2024-02-23T00:35:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:41.641877+0000 mgr.y (mgr.24437) 361 : cluster [DBG] pgmap v565: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:42.113871+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:42.114803+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:42.132434+0000 mon.a (mon.0) 2490 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:42.133093+0000 mon.a (mon.0) 2491 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:42.163646+0000 mon.a (mon.0) 2492 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:42.163728+0000 mon.a (mon.0) 2493 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:42.167914+0000 mon.b (mon.1) 330 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:42.173805+0000 mon.a (mon.0) 2494 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:42.174252+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:42.775886+0000 mon.c (mon.2) 425 : audit [INF] from='client.? 172.21.15.87:0/1095092528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:35:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:42.776197+0000 mon.a (mon.0) 2496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:35:43.768 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.773+0000 7f42910ab700 1 -- 172.21.15.87:0/785338216 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f428c0fd510 msgr2=0x7f428c0f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.773+0000 7f42910ab700 1 --2- 172.21.15.87:0/785338216 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f428c0fd510 0x7f428c0f6d40 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 rx=0x7f42800047b0 tx=0x7f4280009cb0).stop 2024-02-23T00:35:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.773+0000 7f42910ab700 1 -- 172.21.15.87:0/785338216 shutdown_connections 2024-02-23T00:35:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.773+0000 7f42910ab700 1 --2- 172.21.15.87:0/785338216 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f428c0f7400 0x7f428c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.773+0000 7f42910ab700 1 --2- 172.21.15.87:0/785338216 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f428c0fcbb0 0x7f428c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.773+0000 7f42910ab700 1 --2- 172.21.15.87:0/785338216 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f428c0fd510 0x7f428c0f6d40 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.773+0000 7f42910ab700 1 -- 172.21.15.87:0/785338216 >> 172.21.15.87:0/785338216 conn(0x7f428c0f2820 msgr2=0x7f428c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.774+0000 7f42910ab700 1 -- 172.21.15.87:0/785338216 shutdown_connections 2024-02-23T00:35:43.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.774+0000 7f42910ab700 1 -- 172.21.15.87:0/785338216 wait complete. 2024-02-23T00:35:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.774+0000 7f42910ab700 1 librados: starting msgr at 2024-02-23T00:35:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.774+0000 7f42910ab700 1 librados: starting objecter 2024-02-23T00:35:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 Processor -- start 2024-02-23T00:35:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 -- start start 2024-02-23T00:35:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 librados: setting wanted keys 2024-02-23T00:35:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 librados: calling monclient init 2024-02-23T00:35:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f428c0f7400 0x7f428c109350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f428c0fcbb0 0x7f428c109840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f428c0fd510 0x7f428c109e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c114580 con 0x7f428c0fd510 2024-02-23T00:35:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1146f0 con 0x7f428c0fcbb0 2024-02-23T00:35:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.775+0000 7f42910ab700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428c1148b0 con 0x7f428c0f7400 2024-02-23T00:35:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428b59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f428c0fd510 0x7f428c109e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428ad9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f428c0f7400 0x7f428c109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428b59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f428c0fd510 0x7f428c109e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34740/0 (socket says 172.21.15.87:34740) 2024-02-23T00:35:43.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428b59e700 1 -- 172.21.15.87:0/904292610 learned_addr learned my addr 172.21.15.87:0/904292610 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:43.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428a59c700 1 --2- 172.21.15.87:0/904292610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f428c0fcbb0 0x7f428c109840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428ad9d700 1 -- 172.21.15.87:0/904292610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f428c0fcbb0 msgr2=0x7f428c109840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428ad9d700 1 --2- 172.21.15.87:0/904292610 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f428c0fcbb0 0x7f428c109840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428ad9d700 1 -- 172.21.15.87:0/904292610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f428c0fd510 msgr2=0x7f428c109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428ad9d700 1 --2- 172.21.15.87:0/904292610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f428c0fd510 0x7f428c109e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428ad9d700 1 -- 172.21.15.87:0/904292610 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4280005040 con 0x7f428c0f7400 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428b59e700 1 --2- 172.21.15.87:0/904292610 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f428c0fd510 0x7f428c109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f428ad9d700 1 --2- 172.21.15.87:0/904292610 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f428c0f7400 0x7f428c109350 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f427c009a00 tx=0x7f427c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f4273fff700 10 client.?.objecter ms_handle_connect 0x7f428c0f7400 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f4273fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c004630 con 0x7f428c0f7400 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c1071a0 con 0x7f428c0f7400 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.776+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f427c00f040 con 0x7f428c0f7400 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.777+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c0137f0 con 0x7f428c0f7400 2024-02-23T00:35:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.777+0000 7f42910ab700 10 client.37631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:43.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.777+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c1076e0 con 0x7f428c0f7400 2024-02-23T00:35:43.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.777+0000 7f42910ab700 1 librados: init done 2024-02-23T00:35:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.778+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f428c05fa60 con 0x7f428c0f7400 2024-02-23T00:35:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.778+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f427c019020 con 0x7f428c0f7400 2024-02-23T00:35:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f4273fff700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4274062c90 0x7f4274065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(389..389 src has 173..389) v4 ==== 8168+0+0 (secure 0 0 0) 0x7f427c0804a0 con 0x7f428c0f7400 2024-02-23T00:35:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 osd_map(389..389 src has 173..389) v4 2024-02-23T00:35:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f4273fff700 3 client.37631.objecter handle_osd_map got epochs [389,389] > 0 2024-02-23T00:35:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f4273fff700 3 client.37631.objecter handle_osd_map decoding full epoch 389 2024-02-23T00:35:43.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f4273fff700 20 client.37631.objecter dump_active .. 0 homeless 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v389) v1 ==== 54+0+16 (secure 0 0 0) 0x7f427c087070 con 0x7f428c0f7400 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f428a59c700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4274062c90 0x7f4274065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 10 client.37631.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 1 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6801/2153191863 conn(0x7f428c0fdea0 0x7f428c0fe280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c05ee80 osd=0 3 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c05ee80 osd=0 3 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c05ee80 osd=0 3 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 1 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c05ee80 osd=0 3 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 0 1 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c05ee80 osd=0 4 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 10 client.37631.objecter _send_command 1 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.779+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f428c1060e0 con 0x7f428c0fdea0 2024-02-23T00:35:43.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.780+0000 7f428b59e700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6801/2153191863 conn(0x7f428c0fdea0 0x7f428c0fe280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.780+0000 7f428a59c700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4274062c90 0x7f4274065150 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 rx=0x7f4280005010 tx=0x7f4280009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.780+0000 7f4273fff700 10 client.37631.objecter ms_handle_connect 0x7f4274062c90 2024-02-23T00:35:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.780+0000 7f4273fff700 10 client.37631.objecter ms_handle_connect 0x7f428c0fdea0 2024-02-23T00:35:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.780+0000 7f428b59e700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6801/2153191863 conn(0x7f428c0fdea0 0x7f428c0fe280 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.781+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f428c1060e0 con 0x7f428c0fdea0 2024-02-23T00:35:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.781+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:35:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.781+0000 7f4273fff700 10 client.37631.objecter _finish_command 1 = system:0 2024-02-23T00:35:43.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.781+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c05ee80 osd=0 4 2024-02-23T00:35:43.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.781+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 0 1 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 10 client.37631.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 2 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c05ee80 osd=0 3 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c05ee80 osd=0 3 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c05ee80 osd=0 3 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 2 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c05ee80 osd=0 3 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 0 2 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c05ee80 osd=0 4 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 10 client.37631.objecter _send_command 2 2024-02-23T00:35:43.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.797+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f428c0fe9b0 con 0x7f428c0fdea0 2024-02-23T00:35:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f428400f040 con 0x7f428c0fdea0 2024-02-23T00:35:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f4273fff700 10 client.37631.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c05ee80 osd=0 4 2024-02-23T00:35:43.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 0 2 2024-02-23T00:35:43.801 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:35:43.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 10 client.37631.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:43.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 3 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6805/685354171 conn(0x7f428c0598f0 0x7f428c059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c028540 osd=1 3 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c028540 osd=1 3 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c028540 osd=1 3 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 3 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c028540 osd=1 3 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 1 3 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c028540 osd=1 4 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 10 client.37631.objecter _send_command 3 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.798+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f428c0286b0 con 0x7f428c0598f0 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.799+0000 7f428ad9d700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6805/685354171 conn(0x7f428c0598f0 0x7f428c059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.799+0000 7f4273fff700 10 client.37631.objecter ms_handle_connect 0x7f428c0598f0 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.799+0000 7f428ad9d700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6805/685354171 conn(0x7f428c0598f0 0x7f428c059cd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.799+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f428c0286b0 con 0x7f428c0598f0 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.799+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:35:43.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.799+0000 7f4273fff700 10 client.37631.objecter _finish_command 3 = system:0 2024-02-23T00:35:43.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.799+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c028540 osd=1 4 2024-02-23T00:35:43.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.799+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 1 3 2024-02-23T00:35:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 10 client.37631.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 4 2024-02-23T00:35:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c028540 osd=1 3 2024-02-23T00:35:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c028540 osd=1 3 2024-02-23T00:35:43.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c028540 osd=1 3 2024-02-23T00:35:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 4 2024-02-23T00:35:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c028540 osd=1 3 2024-02-23T00:35:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 1 4 2024-02-23T00:35:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c028540 osd=1 4 2024-02-23T00:35:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 10 client.37631.objecter _send_command 4 2024-02-23T00:35:43.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.814+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f428c059360 con 0x7f428c0598f0 2024-02-23T00:35:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f428c059360 con 0x7f428c0598f0 2024-02-23T00:35:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f4273fff700 10 client.37631.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c028540 osd=1 4 2024-02-23T00:35:43.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 1 4 2024-02-23T00:35:43.818 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:35:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 10 client.37631.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 5 2024-02-23T00:35:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6809/3605394228 conn(0x7f428c108f60 0x7f428c027b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c059770 osd=2 3 2024-02-23T00:35:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c059770 osd=2 3 2024-02-23T00:35:43.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c059770 osd=2 3 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 5 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c059770 osd=2 3 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 2 5 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c059770 osd=2 4 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 10 client.37631.objecter _send_command 5 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f428c0281f0 con 0x7f428c108f60 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f428a59c700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6809/3605394228 conn(0x7f428c108f60 0x7f428c027b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f4273fff700 10 client.37631.objecter ms_handle_connect 0x7f428c108f60 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.816+0000 7f428a59c700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6809/3605394228 conn(0x7f428c108f60 0x7f428c027b90 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.817+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f428c0281f0 con 0x7f428c108f60 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.817+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.817+0000 7f4273fff700 10 client.37631.objecter _finish_command 5 = system:0 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.817+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c059770 osd=2 4 2024-02-23T00:35:43.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.817+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 2 5 2024-02-23T00:35:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 10 client.37631.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 6 2024-02-23T00:35:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c059770 osd=2 3 2024-02-23T00:35:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c059770 osd=2 3 2024-02-23T00:35:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c059770 osd=2 3 2024-02-23T00:35:43.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 6 2024-02-23T00:35:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c059770 osd=2 3 2024-02-23T00:35:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 2 6 2024-02-23T00:35:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c059770 osd=2 4 2024-02-23T00:35:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 10 client.37631.objecter _send_command 6 2024-02-23T00:35:43.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.831+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f428c027600 con 0x7f428c108f60 2024-02-23T00:35:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f428c027600 con 0x7f428c108f60 2024-02-23T00:35:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f4273fff700 10 client.37631.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c059770 osd=2 4 2024-02-23T00:35:43.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 2 6 2024-02-23T00:35:43.835 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:35:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 10 client.37631.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 7 2024-02-23T00:35:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6813/128059790 conn(0x7f428c11bce0 0x7f428c11e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c1080a0 osd=3 3 2024-02-23T00:35:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c1080a0 osd=3 3 2024-02-23T00:35:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c1080a0 osd=3 3 2024-02-23T00:35:43.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 7 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c1080a0 osd=3 3 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 3 7 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c1080a0 osd=3 4 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 10 client.37631.objecter _send_command 7 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f428c108720 con 0x7f428c11bce0 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f428b59e700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6813/128059790 conn(0x7f428c11bce0 0x7f428c11e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f4273fff700 10 client.37631.objecter ms_handle_connect 0x7f428c11bce0 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.833+0000 7f428b59e700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6813/128059790 conn(0x7f428c11bce0 0x7f428c11e0d0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.834+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f428c108720 con 0x7f428c11bce0 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.834+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.834+0000 7f4273fff700 10 client.37631.objecter _finish_command 7 = system:0 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.834+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c1080a0 osd=3 4 2024-02-23T00:35:43.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.834+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 3 7 2024-02-23T00:35:43.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 10 client.37631.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:43.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 8 2024-02-23T00:35:43.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c1080a0 osd=3 3 2024-02-23T00:35:43.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c1080a0 osd=3 3 2024-02-23T00:35:43.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c1080a0 osd=3 3 2024-02-23T00:35:43.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 8 2024-02-23T00:35:43.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c1080a0 osd=3 3 2024-02-23T00:35:43.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 3 8 2024-02-23T00:35:43.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c1080a0 osd=3 4 2024-02-23T00:35:43.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 10 client.37631.objecter _send_command 8 2024-02-23T00:35:43.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.848+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f428c107b10 con 0x7f428c11bce0 2024-02-23T00:35:43.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f428c107b10 con 0x7f428c11bce0 2024-02-23T00:35:43.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:43.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f4273fff700 10 client.37631.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c1080a0 osd=3 4 2024-02-23T00:35:43.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 3 8 2024-02-23T00:35:43.852 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 10 client.37631.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 9 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6800/4175844757 conn(0x7f428c11b840 0x7f428c121750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c107b10 osd=4 3 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107b10 osd=4 3 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c107b10 osd=4 3 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 9 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c107b10 osd=4 3 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 4 9 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107b10 osd=4 4 2024-02-23T00:35:43.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 10 client.37631.objecter _send_command 9 2024-02-23T00:35:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f428c107d20 con 0x7f428c11b840 2024-02-23T00:35:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f428ad9d700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6800/4175844757 conn(0x7f428c11b840 0x7f428c121750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f4273fff700 10 client.37631.objecter ms_handle_connect 0x7f428c11b840 2024-02-23T00:35:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.850+0000 7f428ad9d700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6800/4175844757 conn(0x7f428c11b840 0x7f428c121750 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.851+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f428c107d20 con 0x7f428c11b840 2024-02-23T00:35:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.851+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:35:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.851+0000 7f4273fff700 10 client.37631.objecter _finish_command 9 = system:0 2024-02-23T00:35:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.851+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c107b10 osd=4 4 2024-02-23T00:35:43.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.851+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 4 9 2024-02-23T00:35:43.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 10 client.37631.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:43.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 10 2024-02-23T00:35:43.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c107b10 osd=4 3 2024-02-23T00:35:43.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107b10 osd=4 3 2024-02-23T00:35:43.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c107b10 osd=4 3 2024-02-23T00:35:43.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 10 2024-02-23T00:35:43.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c107b10 osd=4 3 2024-02-23T00:35:43.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 4 10 2024-02-23T00:35:43.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107b10 osd=4 4 2024-02-23T00:35:43.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 10 client.37631.objecter _send_command 10 2024-02-23T00:35:43.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.866+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f428c107d20 con 0x7f428c11b840 2024-02-23T00:35:43.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.867+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f428c107d20 con 0x7f428c11b840 2024-02-23T00:35:43.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:43.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f4273fff700 10 client.37631.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c107b10 osd=4 4 2024-02-23T00:35:43.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 4 10 2024-02-23T00:35:43.870 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:35:43.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 10 client.37631.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:43.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 11 2024-02-23T00:35:43.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6804/2550013795 conn(0x7f428c122150 0x7f428c124540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c107d20 osd=5 3 2024-02-23T00:35:43.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107d20 osd=5 3 2024-02-23T00:35:43.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c107d20 osd=5 3 2024-02-23T00:35:43.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 11 2024-02-23T00:35:43.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c107d20 osd=5 3 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 5 11 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107d20 osd=5 4 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 10 client.37631.objecter _send_command 11 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f428c127270 con 0x7f428c122150 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f428a59c700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6804/2550013795 conn(0x7f428c122150 0x7f428c124540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f428a59c700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6804/2550013795 conn(0x7f428c122150 0x7f428c124540 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.868+0000 7f4273fff700 10 client.37631.objecter ms_handle_connect 0x7f428c122150 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.869+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f428c127270 con 0x7f428c122150 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.869+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.869+0000 7f4273fff700 10 client.37631.objecter _finish_command 11 = system:0 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.869+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c107d20 osd=5 4 2024-02-23T00:35:43.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.869+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 5 11 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 10 client.37631.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 12 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c107d20 osd=5 3 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107d20 osd=5 3 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c107d20 osd=5 3 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 12 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c107d20 osd=5 3 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 5 12 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107d20 osd=5 4 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 10 client.37631.objecter _send_command 12 2024-02-23T00:35:43.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.883+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f428c127270 con 0x7f428c122150 2024-02-23T00:35:43.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f428c127270 con 0x7f428c122150 2024-02-23T00:35:43.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:43.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f4273fff700 10 client.37631.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c107d20 osd=5 4 2024-02-23T00:35:43.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 5 12 2024-02-23T00:35:43.888 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 10 client.37631.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 13 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6808/2509408498 conn(0x7f428c126570 0x7f428c1269f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c127270 osd=6 3 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c127270 osd=6 3 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c127270 osd=6 3 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 13 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c127270 osd=6 3 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 6 13 2024-02-23T00:35:43.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c127270 osd=6 4 2024-02-23T00:35:43.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 10 client.37631.objecter _send_command 13 2024-02-23T00:35:43.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.885+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f428c126fd0 con 0x7f428c126570 2024-02-23T00:35:43.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.886+0000 7f428b59e700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6808/2509408498 conn(0x7f428c126570 0x7f428c1269f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.886+0000 7f4273fff700 10 client.37631.objecter ms_handle_connect 0x7f428c126570 2024-02-23T00:35:43.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.886+0000 7f428b59e700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6808/2509408498 conn(0x7f428c126570 0x7f428c1269f0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.887+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f428c126fd0 con 0x7f428c126570 2024-02-23T00:35:43.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.887+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:35:43.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.887+0000 7f4273fff700 10 client.37631.objecter _finish_command 13 = system:0 2024-02-23T00:35:43.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.887+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c127270 osd=6 4 2024-02-23T00:35:43.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.887+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 6 13 2024-02-23T00:35:43.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 10 client.37631.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:43.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 14 2024-02-23T00:35:43.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c127270 osd=6 3 2024-02-23T00:35:43.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c127270 osd=6 3 2024-02-23T00:35:43.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c127270 osd=6 3 2024-02-23T00:35:43.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 14 2024-02-23T00:35:43.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c127270 osd=6 3 2024-02-23T00:35:43.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 6 14 2024-02-23T00:35:43.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c127270 osd=6 4 2024-02-23T00:35:43.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 10 client.37631.objecter _send_command 14 2024-02-23T00:35:43.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.904+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f428c126fd0 con 0x7f428c126570 2024-02-23T00:35:43.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f428c126fd0 con 0x7f428c126570 2024-02-23T00:35:43.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:43.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f4273fff700 10 client.37631.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c127270 osd=6 4 2024-02-23T00:35:43.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 6 14 2024-02-23T00:35:43.908 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:35:43.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 10 client.37631.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:43.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 15 2024-02-23T00:35:43.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6812/373787000 conn(0x7f428c124fc0 0x7f428c125400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:43.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c126fd0 osd=7 3 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c126fd0 osd=7 3 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c126fd0 osd=7 3 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 15 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c126fd0 osd=7 3 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 7 15 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c126fd0 osd=7 4 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 10 client.37631.objecter _send_command 15 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f428c125a20 con 0x7f428c124fc0 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.906+0000 7f428ad9d700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6812/373787000 conn(0x7f428c124fc0 0x7f428c125400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.907+0000 7f4273fff700 10 client.37631.objecter ms_handle_connect 0x7f428c124fc0 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.907+0000 7f428ad9d700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6812/373787000 conn(0x7f428c124fc0 0x7f428c125400 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.907+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f428c125a20 con 0x7f428c124fc0 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.907+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:35:43.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.907+0000 7f4273fff700 10 client.37631.objecter _finish_command 15 = system:0 2024-02-23T00:35:43.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.907+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c126fd0 osd=7 4 2024-02-23T00:35:43.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.907+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 7 15 2024-02-23T00:35:43.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 10 client.37631.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:35:43.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign -1 16 2024-02-23T00:35:43.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c126fd0 osd=7 3 2024-02-23T00:35:43.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c126fd0 osd=7 3 2024-02-23T00:35:43.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 20 client.37631.objecter _get_session s=0x7f428c126fd0 osd=7 3 2024-02-23T00:35:43.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 15 client.37631.objecter _session_command_op_remove -1 16 2024-02-23T00:35:43.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 20 client.37631.objecter get_session s=0x7f428c126fd0 osd=7 3 2024-02-23T00:35:43.927 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:35:43.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 15 client.37631.objecter _session_command_op_assign 7 16 2024-02-23T00:35:43.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c126fd0 osd=7 4 2024-02-23T00:35:43.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 10 client.37631.objecter _send_command 16 2024-02-23T00:35:43.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.921+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f428c125a20 con 0x7f428c124fc0 2024-02-23T00:35:43.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.923+0000 7f4273fff700 1 -- 172.21.15.87:0/904292610 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f428c125a20 con 0x7f428c124fc0 2024-02-23T00:35:43.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.923+0000 7f4273fff700 10 client.37631.objecter ms_dispatch 0x7f428c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:35:43.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.923+0000 7f4273fff700 10 client.37631.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.923+0000 7f4273fff700 20 client.37631.objecter put_session s=0x7f428c126fd0 osd=7 4 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.923+0000 7f4273fff700 15 client.37631.objecter _session_command_op_remove 7 16 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 librados: watch_flush enter 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 librados: watch_flush exit 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 client.37631.objecter close_session for osd.0 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6801/2153191863 conn(0x7f428c0fdea0 msgr2=0x7f428c0fe280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6801/2153191863 conn(0x7f428c0fdea0 0x7f428c0fe280 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c05ee80 osd=0 1 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 client.37631.objecter close_session for osd.1 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6805/685354171 conn(0x7f428c0598f0 msgr2=0x7f428c059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6805/685354171 conn(0x7f428c0598f0 0x7f428c059cd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c028540 osd=1 1 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 client.37631.objecter close_session for osd.2 2024-02-23T00:35:43.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6809/3605394228 conn(0x7f428c108f60 msgr2=0x7f428c027b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6809/3605394228 conn(0x7f428c108f60 0x7f428c027b90 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c059770 osd=2 1 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 client.37631.objecter close_session for osd.3 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6813/128059790 conn(0x7f428c11bce0 msgr2=0x7f428c11e0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6813/128059790 conn(0x7f428c11bce0 0x7f428c11e0d0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f428b59e700 1 -- 172.21.15.87:0/904292610 reap_dead start 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c1080a0 osd=3 1 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 client.37631.objecter close_session for osd.4 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6800/4175844757 conn(0x7f428c11b840 msgr2=0x7f428c121750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6800/4175844757 conn(0x7f428c11b840 0x7f428c121750 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107b10 osd=4 1 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f428b59e700 1 -- 172.21.15.87:0/904292610 reap_dead start 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 client.37631.objecter close_session for osd.5 2024-02-23T00:35:43.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6804/2550013795 conn(0x7f428c122150 msgr2=0x7f428c124540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6804/2550013795 conn(0x7f428c122150 0x7f428c124540 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c107d20 osd=5 1 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 client.37631.objecter close_session for osd.6 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6808/2509408498 conn(0x7f428c126570 msgr2=0x7f428c1269f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6808/2509408498 conn(0x7f428c126570 0x7f428c1269f0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c127270 osd=6 1 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 client.37631.objecter close_session for osd.7 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6812/373787000 conn(0x7f428c124fc0 msgr2=0x7f428c125400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.171:6812/373787000 conn(0x7f428c124fc0 0x7f428c125400 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 20 client.37631.objecter put_session s=0x7f428c126fd0 osd=7 1 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 20 client.37631.objecter shutdown clearing up homeless session... 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 10 client.37631.objecter successfully canceled tick 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4274062c90 msgr2=0x7f4274065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4274062c90 0x7f4274065150 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 rx=0x7f4280005010 tx=0x7f4280009cb0).stop 2024-02-23T00:35:43.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f428c0f7400 msgr2=0x7f428c109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 --2- 172.21.15.87:0/904292610 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f428c0f7400 0x7f428c109350 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 rx=0x7f427c009a00 tx=0x7f427c00b040).stop 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f428b59e700 1 -- 172.21.15.87:0/904292610 reap_dead start 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 shutdown_connections 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 >> 172.21.15.87:0/904292610 conn(0x7f428c0f2820 msgr2=0x7f428c0f4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 shutdown_connections 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 -- 172.21.15.87:0/904292610 wait complete. 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:43.924+0000 7f42910ab700 1 librados: shutdown 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.932 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:35:43.933 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:35:44.188 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:35:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:35:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:43.173613+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]': finished 2024-02-23T00:35:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:43.173693+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-23T00:35:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:43.173751+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-02-23T00:35:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:43.209090+0000 mon.b (mon.1) 331 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:43.209321+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:43.210035+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:43.210181+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:43.780465+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? 172.21.15.87:0/904292610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:43.173613+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]': finished 2024-02-23T00:35:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:43.173693+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-23T00:35:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:43.173751+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-02-23T00:35:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:43.209090+0000 mon.b (mon.1) 331 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:43.209321+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:44.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:43.210035+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:43.210181+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:44.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:43.780465+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? 172.21.15.87:0/904292610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:43.173613+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]': finished 2024-02-23T00:35:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:43.173693+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-23T00:35:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:43.173751+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-02-23T00:35:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:43.209090+0000 mon.b (mon.1) 331 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:43.209321+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:43.210035+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:43.210181+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-49"}]: dispatch 2024-02-23T00:35:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:43.780465+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? 172.21.15.87:0/904292610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:43.642280+0000 mgr.y (mgr.24437) 362 : cluster [DBG] pgmap v568: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:44.176799+0000 mon.a (mon.0) 2502 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-23T00:35:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:44.190050+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-02-23T00:35:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:44.209301+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:35:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:44.214723+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:35:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:43.642280+0000 mgr.y (mgr.24437) 362 : cluster [DBG] pgmap v568: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:44.176799+0000 mon.a (mon.0) 2502 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-23T00:35:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:44.190050+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-02-23T00:35:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:44.209301+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:35:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:44.214723+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:35:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:43.642280+0000 mgr.y (mgr.24437) 362 : cluster [DBG] pgmap v568: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 13/1125 objects misplaced (1.156%) 2024-02-23T00:35:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:44.176799+0000 mon.a (mon.0) 2502 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-23T00:35:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:44.190050+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-02-23T00:35:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:44.209301+0000 osd.0 (osd.0) 183 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:35:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:44.214723+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:35:45.724 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:45.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:45.829 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:45.861 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:45.883 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:45.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:45.987 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:46.018 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:46.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:45.169839+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:35:46.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:45.171184+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:35:46.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:45.195027+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-02-23T00:35:46.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:45.203825+0000 mon.b (mon.1) 333 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:46.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:45.204137+0000 mon.a (mon.0) 2505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:46.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:45.169839+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:35:46.239 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:45.171184+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:35:46.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:45.195027+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-02-23T00:35:46.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:45.203825+0000 mon.b (mon.1) 333 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:46.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:45.204137+0000 mon.a (mon.0) 2505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:46.367 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:46.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:46.470 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:46.497 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:45.169839+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:35:46.497 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:45.171184+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:35:46.498 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:45.195027+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-02-23T00:35:46.498 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:45.203825+0000 mon.b (mon.1) 333 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:46.498 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:45.204137+0000 mon.a (mon.0) 2505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:46.499 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:46.520 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:46.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:46.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:35:46.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 -- 172.21.15.87:0/1886253564 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 msgr2=0x7fd2f0103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:46.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/1886253564 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 0x7fd2f0103a20 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 rx=0x7fd2e8009be0 tx=0x7fd2e8004d10).stop 2024-02-23T00:35:46.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 -- 172.21.15.87:0/1886253564 shutdown_connections 2024-02-23T00:35:46.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/1886253564 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 0x7fd2f0103a20 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:46.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/1886253564 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f00fcbb0 0x7fd2f00fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:35:46.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/1886253564 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2f00fd510 0x7fd2f00f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:46.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 -- 172.21.15.87:0/1886253564 >> 172.21.15.87:0/1886253564 conn(0x7fd2f00f2820 msgr2=0x7fd2f00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:46.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 -- 172.21.15.87:0/1886253564 shutdown_connections 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 -- 172.21.15.87:0/1886253564 wait complete. 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 librados: starting msgr at 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 librados: starting objecter 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 Processor -- start 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 -- start start 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 librados: setting wanted keys 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.622+0000 7fd2f6c45700 1 librados: calling monclient init 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f6c45700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 0x7fd2f00f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f6c45700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f00fcbb0 0x7fd2f00f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f6c45700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2f00fd510 0x7fd2f00f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f6c45700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0116530 con 0x7fd2f00f7400 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f6c45700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f01166a0 con 0x7fd2f00fcbb0 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f6c45700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0116860 con 0x7fd2f00fd510 2024-02-23T00:35:46.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f51e2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2f00fd510 0x7fd2f00f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f49e1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 0x7fd2f00f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f51e2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2f00fd510 0x7fd2f00f9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44648/0 (socket says 172.21.15.87:44648) 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f49e1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 0x7fd2f00f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34756/0 (socket says 172.21.15.87:34756) 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f51e2700 1 -- 172.21.15.87:0/3288099161 learned_addr learned my addr 172.21.15.87:0/3288099161 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2effff700 1 --2- 172.21.15.87:0/3288099161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f00fcbb0 0x7fd2f00f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f51e2700 1 -- 172.21.15.87:0/3288099161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f00fcbb0 msgr2=0x7fd2f00f8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f51e2700 1 --2- 172.21.15.87:0/3288099161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f00fcbb0 0x7fd2f00f8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f51e2700 1 -- 172.21.15.87:0/3288099161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 msgr2=0x7fd2f00f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f51e2700 1 --2- 172.21.15.87:0/3288099161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 0x7fd2f00f89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:46.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f51e2700 1 -- 172.21.15.87:0/3288099161 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc005040 con 0x7fd2f00fd510 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f49e1700 1 --2- 172.21.15.87:0/3288099161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 0x7fd2f00f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f51e2700 1 --2- 172.21.15.87:0/3288099161 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2f00fd510 0x7fd2f00f9390 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 rx=0x7fd2e8004ed0 tx=0x7fd2e8004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2edffb700 10 client.?.objecter ms_handle_connect 0x7fd2f00fd510 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2edffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2edffb700 1 -- 172.21.15.87:0/3288099161 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e800e040 con 0x7fd2f00fd510 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2edffb700 1 -- 172.21.15.87:0/3288099161 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2e8018040 con 0x7fd2f00fd510 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e80097c0 con 0x7fd2f00fd510 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.623+0000 7fd2edffb700 1 -- 172.21.15.87:0/3288099161 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e801c3f0 con 0x7fd2f00fd510 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.624+0000 7fd2f6c45700 10 client.37637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.624+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f0116d50 con 0x7fd2f00fd510 2024-02-23T00:35:46.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.624+0000 7fd2f6c45700 1 librados: init done 2024-02-23T00:35:46.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.625+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f005fa60 con 0x7fd2f00fd510 2024-02-23T00:35:46.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.625+0000 7fd2edffb700 1 -- 172.21.15.87:0/3288099161 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd2e8007b20 con 0x7fd2f00fd510 2024-02-23T00:35:46.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.625+0000 7fd2edffb700 1 --2- 172.21.15.87:0/3288099161 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2e0062c90 0x7fd2e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:46.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.626+0000 7fd2edffb700 1 -- 172.21.15.87:0/3288099161 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(392..392 src has 173..392) v4 ==== 8106+0+0 (secure 0 0 0) 0x7fd2e807e160 con 0x7fd2f00fd510 2024-02-23T00:35:46.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.626+0000 7fd2edffb700 10 client.37637.objecter ms_dispatch 0x7fd2f0001550 osd_map(392..392 src has 173..392) v4 2024-02-23T00:35:46.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.626+0000 7fd2edffb700 3 client.37637.objecter handle_osd_map got epochs [392,392] > 0 2024-02-23T00:35:46.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.626+0000 7fd2edffb700 3 client.37637.objecter handle_osd_map decoding full epoch 392 2024-02-23T00:35:46.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.626+0000 7fd2f49e1700 1 --2- 172.21.15.87:0/3288099161 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2e0062c90 0x7fd2e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:46.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.626+0000 7fd2edffb700 20 client.37637.objecter dump_active .. 0 homeless 2024-02-23T00:35:46.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.626+0000 7fd2edffb700 10 client.37637.objecter ms_handle_connect 0x7fd2e0062c90 2024-02-23T00:35:46.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.626+0000 7fd2f49e1700 1 --2- 172.21.15.87:0/3288099161 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2e0062c90 0x7fd2e0065150 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 rx=0x7fd2e4006fd0 tx=0x7fd2e4009060).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:46.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.630+0000 7fd2edffb700 1 -- 172.21.15.87:0/3288099161 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd2e80516f0 con 0x7fd2f00fd510 2024-02-23T00:35:46.655 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:46.679 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:35:46.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:46.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:35:46.784 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:46.813 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:46.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:46.823+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd2f005ee80 con 0x7fd2f00fd510 2024-02-23T00:35:46.832 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:46.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:46.936 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:46.963 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:46.982 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:46.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:47.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.074+0000 7fd2edffb700 1 -- 172.21.15.87:0/3288099161 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v393) v1 ==== 101+0+0 (secure 0 0 0) 0x7fd2e8021030 con 0x7fd2f00fd510 2024-02-23T00:35:47.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.076+0000 7fd2f6c45700 10 librados: watch_flush enter 2024-02-23T00:35:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 10 librados: watch_flush exit 2024-02-23T00:35:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 20 client.37637.objecter shutdown clearing up homeless session... 2024-02-23T00:35:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 10 client.37637.objecter successfully canceled tick 2024-02-23T00:35:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2e0062c90 msgr2=0x7fd2e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:47.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/3288099161 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2e0062c90 0x7fd2e0065150 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 rx=0x7fd2e4006fd0 tx=0x7fd2e4009060).stop 2024-02-23T00:35:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2f00fd510 msgr2=0x7fd2f00f9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/3288099161 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2f00fd510 0x7fd2f00f9390 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 rx=0x7fd2e8004ed0 tx=0x7fd2e8004d10).stop 2024-02-23T00:35:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 shutdown_connections 2024-02-23T00:35:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/3288099161 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd2e0062c90 0x7fd2e0065150 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/3288099161 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2f00fd510 0x7fd2f00f9390 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/3288099161 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2f00f7400 0x7fd2f00f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 --2- 172.21.15.87:0/3288099161 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2f00fcbb0 0x7fd2f00f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:47.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.077+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 >> 172.21.15.87:0/3288099161 conn(0x7fd2f00f2820 msgr2=0x7fd2f0060dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:47.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.078+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 shutdown_connections 2024-02-23T00:35:47.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.079+0000 7fd2f6c45700 1 -- 172.21.15.87:0/3288099161 wait complete. 2024-02-23T00:35:47.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:47.080+0000 7fd2f6c45700 1 librados: shutdown 2024-02-23T00:35:47.083 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:35:47.086 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:47.113 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:47.134 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:47.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:47.237 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:47.264 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:47.440 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:45.643418+0000 mgr.y (mgr.24437) 363 : cluster [DBG] pgmap v571: 238 pgs: 1 active+clean+remapped, 32 creating+peering, 205 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s; 54 B/s, 4 objects/s recovering 2024-02-23T00:35:47.440 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.083783+0000 osd.6 (osd.6) 139 : cluster [DBG] 63.14 scrub starts 2024-02-23T00:35:47.440 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.085275+0000 osd.6 (osd.6) 140 : cluster [DBG] 63.14 scrub ok 2024-02-23T00:35:47.440 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.191107+0000 osd.2 (osd.2) 145 : cluster [DBG] 63.16 scrub starts 2024-02-23T00:35:47.440 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.192277+0000 mon.a (mon.0) 2506 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-23T00:35:47.440 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.192701+0000 osd.2 (osd.2) 146 : cluster [DBG] 63.16 scrub ok 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:46.201676+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.201770+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:46.242947+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:46.243171+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:46.243851+0000 mon.b (mon.1) 335 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-51"}]: dispatch 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:46.243937+0000 mon.a (mon.0) 2510 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-51"}]: dispatch 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.746499+0000 osd.1 (osd.1) 169 : cluster [DBG] 63.f scrub starts 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.750018+0000 osd.1 (osd.1) 170 : cluster [DBG] 63.f scrub ok 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:46.825007+0000 mon.c (mon.2) 427 : audit [INF] from='client.? 172.21.15.87:0/3288099161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:46.825308+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:47.068359+0000 mon.a (mon.0) 2512 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:47.072517+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:35:47.441 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:47.072577+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:45.643418+0000 mgr.y (mgr.24437) 363 : cluster [DBG] pgmap v571: 238 pgs: 1 active+clean+remapped, 32 creating+peering, 205 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s; 54 B/s, 4 objects/s recovering 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.083783+0000 osd.6 (osd.6) 139 : cluster [DBG] 63.14 scrub starts 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.085275+0000 osd.6 (osd.6) 140 : cluster [DBG] 63.14 scrub ok 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.191107+0000 osd.2 (osd.2) 145 : cluster [DBG] 63.16 scrub starts 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.192277+0000 mon.a (mon.0) 2506 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.192701+0000 osd.2 (osd.2) 146 : cluster [DBG] 63.16 scrub ok 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:46.201676+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.201770+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:46.242947+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:46.243171+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:46.243851+0000 mon.b (mon.1) 335 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-51"}]: dispatch 2024-02-23T00:35:47.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:46.243937+0000 mon.a (mon.0) 2510 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-51"}]: dispatch 2024-02-23T00:35:47.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.746499+0000 osd.1 (osd.1) 169 : cluster [DBG] 63.f scrub starts 2024-02-23T00:35:47.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.750018+0000 osd.1 (osd.1) 170 : cluster [DBG] 63.f scrub ok 2024-02-23T00:35:47.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:46.825007+0000 mon.c (mon.2) 427 : audit [INF] from='client.? 172.21.15.87:0/3288099161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:47.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:46.825308+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:47.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:47.068359+0000 mon.a (mon.0) 2512 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:35:47.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:47.072517+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:35:47.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:47.072577+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:45.643418+0000 mgr.y (mgr.24437) 363 : cluster [DBG] pgmap v571: 238 pgs: 1 active+clean+remapped, 32 creating+peering, 205 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s; 54 B/s, 4 objects/s recovering 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.083783+0000 osd.6 (osd.6) 139 : cluster [DBG] 63.14 scrub starts 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.085275+0000 osd.6 (osd.6) 140 : cluster [DBG] 63.14 scrub ok 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.191107+0000 osd.2 (osd.2) 145 : cluster [DBG] 63.16 scrub starts 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.192277+0000 mon.a (mon.0) 2506 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.192701+0000 osd.2 (osd.2) 146 : cluster [DBG] 63.16 scrub ok 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:46.201676+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.201770+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:46.242947+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:46.243171+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:46.243851+0000 mon.b (mon.1) 335 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-51"}]: dispatch 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:46.243937+0000 mon.a (mon.0) 2510 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-51"}]: dispatch 2024-02-23T00:35:47.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.746499+0000 osd.1 (osd.1) 169 : cluster [DBG] 63.f scrub starts 2024-02-23T00:35:47.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.750018+0000 osd.1 (osd.1) 170 : cluster [DBG] 63.f scrub ok 2024-02-23T00:35:47.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:46.825007+0000 mon.c (mon.2) 427 : audit [INF] from='client.? 172.21.15.87:0/3288099161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:47.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:46.825308+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:35:47.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:47.068359+0000 mon.a (mon.0) 2512 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:35:47.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:47.072517+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:35:47.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:47.072577+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-02-23T00:35:47.462 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:47.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:47.566 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:47.595 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:47.747 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:35:48.401 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:35:48.401 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:35:48.402 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:35:48.402 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:35:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.172593+0000 osd.4 (osd.4) 177 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:35:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.175472+0000 osd.4 (osd.4) 178 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:35:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.269040+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:35:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:46.269794+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:35:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:47.126045+0000 osd.6 (osd.6) 141 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:35:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:47.130744+0000 osd.6 (osd.6) 142 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:35:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:47.193232+0000 osd.0 (osd.0) 185 : cluster [DBG] 3.b scrub starts 2024-02-23T00:35:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:47.196189+0000 osd.0 (osd.0) 186 : cluster [DBG] 3.b scrub ok 2024-02-23T00:35:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:47.854642+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:35:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:47.857529+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:35:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:48.074320+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-02-23T00:35:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:48.085608+0000 mon.b (mon.1) 336 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:48.085897+0000 mon.a (mon.0) 2516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:48.756 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.172593+0000 osd.4 (osd.4) 177 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.175472+0000 osd.4 (osd.4) 178 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.269040+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:46.269794+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:47.126045+0000 osd.6 (osd.6) 141 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:47.130744+0000 osd.6 (osd.6) 142 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:47.193232+0000 osd.0 (osd.0) 185 : cluster [DBG] 3.b scrub starts 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:47.196189+0000 osd.0 (osd.0) 186 : cluster [DBG] 3.b scrub ok 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:47.854642+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:47.857529+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:48.074320+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:48.085608+0000 mon.b (mon.1) 336 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:48.085897+0000 mon.a (mon.0) 2516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.172593+0000 osd.4 (osd.4) 177 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.175472+0000 osd.4 (osd.4) 178 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.269040+0000 osd.5 (osd.5) 155 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:46.269794+0000 osd.5 (osd.5) 156 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:47.126045+0000 osd.6 (osd.6) 141 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:47.130744+0000 osd.6 (osd.6) 142 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:47.193232+0000 osd.0 (osd.0) 185 : cluster [DBG] 3.b scrub starts 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:47.196189+0000 osd.0 (osd.0) 186 : cluster [DBG] 3.b scrub ok 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:47.854642+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:47.857529+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:48.074320+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:48.085608+0000 mon.b (mon.1) 336 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:48.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:48.085897+0000 mon.a (mon.0) 2516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:49.344 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:49.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:49.448 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:49.478 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:47.643908+0000 mgr.y (mgr.24437) 364 : cluster [DBG] pgmap v574: 206 pgs: 1 active+clean+remapped, 205 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s; 54 B/s, 4 objects/s recovering 2024-02-23T00:35:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:48.864923+0000 osd.7 (osd.7) 189 : cluster [DBG] 6.b scrub starts 2024-02-23T00:35:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:48.866225+0000 osd.7 (osd.7) 190 : cluster [DBG] 6.b scrub ok 2024-02-23T00:35:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:49.076673+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:49.076762+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-02-23T00:35:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:49.091172+0000 mon.b (mon.1) 337 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:35:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:49.125600+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:49.125804+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:49.126396+0000 mon.b (mon.1) 339 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-53"}]: dispatch 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:49.126441+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-53"}]: dispatch 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:47.643908+0000 mgr.y (mgr.24437) 364 : cluster [DBG] pgmap v574: 206 pgs: 1 active+clean+remapped, 205 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s; 54 B/s, 4 objects/s recovering 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:48.864923+0000 osd.7 (osd.7) 189 : cluster [DBG] 6.b scrub starts 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:48.866225+0000 osd.7 (osd.7) 190 : cluster [DBG] 6.b scrub ok 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:49.076673+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:49.076762+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:49.091172+0000 mon.b (mon.1) 337 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:49.125600+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:49.125804+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:49.126396+0000 mon.b (mon.1) 339 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-53"}]: dispatch 2024-02-23T00:35:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:49.126441+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-53"}]: dispatch 2024-02-23T00:35:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:47.643908+0000 mgr.y (mgr.24437) 364 : cluster [DBG] pgmap v574: 206 pgs: 1 active+clean+remapped, 205 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s; 54 B/s, 4 objects/s recovering 2024-02-23T00:35:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:48.864923+0000 osd.7 (osd.7) 189 : cluster [DBG] 6.b scrub starts 2024-02-23T00:35:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:48.866225+0000 osd.7 (osd.7) 190 : cluster [DBG] 6.b scrub ok 2024-02-23T00:35:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:49.076673+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:49.076762+0000 mon.a (mon.0) 2518 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-02-23T00:35:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:49.091172+0000 mon.b (mon.1) 337 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:35:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:49.125600+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:49.125804+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:49.126396+0000 mon.b (mon.1) 339 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-53"}]: dispatch 2024-02-23T00:35:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:49.126441+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-53"}]: dispatch 2024-02-23T00:35:49.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-23T00:35:49.907 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:49.926 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:49.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:50.031 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:50.063 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:50.087 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:50.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:50.190 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:50.220 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:50.240 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:50.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:50.343 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:50.371 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:50.757 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:51.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:35:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:35:51.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:49.644299+0000 mgr.y (mgr.24437) 365 : cluster [DBG] pgmap v577: 238 pgs: 32 unknown, 1 active+clean+remapped, 205 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:35:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:50.078475+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-02-23T00:35:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:50.193772+0000 osd.2 (osd.2) 147 : cluster [DBG] 2.c scrub starts 2024-02-23T00:35:51.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:50.194649+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.c scrub ok 2024-02-23T00:35:51.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:49.644299+0000 mgr.y (mgr.24437) 365 : cluster [DBG] pgmap v577: 238 pgs: 32 unknown, 1 active+clean+remapped, 205 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:35:51.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:50.078475+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-02-23T00:35:51.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:50.193772+0000 osd.2 (osd.2) 147 : cluster [DBG] 2.c scrub starts 2024-02-23T00:35:51.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:50.194649+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.c scrub ok 2024-02-23T00:35:51.271 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:49.644299+0000 mgr.y (mgr.24437) 365 : cluster [DBG] pgmap v577: 238 pgs: 32 unknown, 1 active+clean+remapped, 205 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:35:51.271 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:50.078475+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-02-23T00:35:51.272 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:50.193772+0000 osd.2 (osd.2) 147 : cluster [DBG] 2.c scrub starts 2024-02-23T00:35:51.272 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:50.194649+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.c scrub ok 2024-02-23T00:35:51.327 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:35:51.651 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:51.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:51.755 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:51.787 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:51.808 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:51.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:51.912 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:51.945 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:52.044 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:52.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:52.148 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:52.177 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:52.196 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:52.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:52.223 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:50.871882+0000 mgr.y (mgr.24437) 366 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:52.223 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:51.082036+0000 mon.a (mon.0) 2522 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-02-23T00:35:52.223 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:51.083251+0000 osd.6 (osd.6) 143 : cluster [DBG] 63.c deep-scrub starts 2024-02-23T00:35:52.223 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:51.087858+0000 mon.b (mon.1) 340 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:52.223 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:51.088303+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:52.223 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:51.098340+0000 osd.6 (osd.6) 144 : cluster [DBG] 63.c deep-scrub ok 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:51.184224+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:51.184933+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:50.871882+0000 mgr.y (mgr.24437) 366 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:51.082036+0000 mon.a (mon.0) 2522 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:51.083251+0000 osd.6 (osd.6) 143 : cluster [DBG] 63.c deep-scrub starts 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:51.087858+0000 mon.b (mon.1) 340 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:51.088303+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:51.098340+0000 osd.6 (osd.6) 144 : cluster [DBG] 63.c deep-scrub ok 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:51.184224+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:35:52.224 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:51.184933+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:35:52.300 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:52.327 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:52.346 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:52.347 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:50.871882+0000 mgr.y (mgr.24437) 366 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:35:52.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:51.082036+0000 mon.a (mon.0) 2522 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-02-23T00:35:52.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:51.083251+0000 osd.6 (osd.6) 143 : cluster [DBG] 63.c deep-scrub starts 2024-02-23T00:35:52.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:51.087858+0000 mon.b (mon.1) 340 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:52.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:51.088303+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:52.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:51.098340+0000 osd.6 (osd.6) 144 : cluster [DBG] 63.c deep-scrub ok 2024-02-23T00:35:52.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:51.184224+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:35:52.348 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:51.184933+0000 osd.2 (osd.2) 150 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:35:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:52.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:52.450 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:52.476 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:52.494 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:52.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:52.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.568+0000 7f707e4b8700 1 -- 172.21.15.87:0/2504657306 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70780f6a90 msgr2=0x7f7078104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.568+0000 7f707e4b8700 1 --2- 172.21.15.87:0/2504657306 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70780f6a90 0x7f7078104a20 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 rx=0x7f7070009be0 tx=0x7f7070004d10).stop 2024-02-23T00:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.568+0000 7f707e4b8700 1 -- 172.21.15.87:0/2504657306 shutdown_connections 2024-02-23T00:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.568+0000 7f707e4b8700 1 --2- 172.21.15.87:0/2504657306 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70780f6a90 0x7f7078104a20 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.568+0000 7f707e4b8700 1 --2- 172.21.15.87:0/2504657306 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70780fdbd0 0x7f70780fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.568+0000 7f707e4b8700 1 --2- 172.21.15.87:0/2504657306 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70780fe530 0x7f70780f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.568+0000 7f707e4b8700 1 -- 172.21.15.87:0/2504657306 >> 172.21.15.87:0/2504657306 conn(0x7f707806c370 msgr2=0x7f707806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.568+0000 7f707e4b8700 1 -- 172.21.15.87:0/2504657306 shutdown_connections 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.568+0000 7f707e4b8700 1 -- 172.21.15.87:0/2504657306 wait complete. 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 librados: starting msgr at 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 librados: starting objecter 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 Processor -- start 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 -- start start 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 librados: setting wanted keys 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 librados: calling monclient init 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70780f6a90 0x7f707810e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70780fdbd0 0x7f707810e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70780fe530 0x7f70781082f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70781087e0 con 0x7f70780fdbd0 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7078108950 con 0x7f70780f6a90 2024-02-23T00:35:52.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707e4b8700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7078108b10 con 0x7f70780fe530 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707ca55700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70780fe530 0x7f70781082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f70777fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70780fdbd0 0x7f707810e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707ca55700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70780fe530 0x7f70781082f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50126/0 (socket says 172.21.15.87:50126) 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f70777fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70780fdbd0 0x7f707810e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48916/0 (socket says 172.21.15.87:48916) 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707ca55700 1 -- 172.21.15.87:0/808775384 learned_addr learned my addr 172.21.15.87:0/808775384 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f7077fff700 1 --2- 172.21.15.87:0/808775384 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70780f6a90 0x7f707810e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707ca55700 1 -- 172.21.15.87:0/808775384 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70780f6a90 msgr2=0x7f707810e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707ca55700 1 --2- 172.21.15.87:0/808775384 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f70780f6a90 0x7f707810e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.569+0000 7f707ca55700 1 -- 172.21.15.87:0/808775384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70780fdbd0 msgr2=0x7f707810e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f707ca55700 1 --2- 172.21.15.87:0/808775384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70780fdbd0 0x7f707810e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f707ca55700 1 -- 172.21.15.87:0/808775384 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70700097c0 con 0x7f70780fe530 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f70777fe700 1 --2- 172.21.15.87:0/808775384 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70780fdbd0 0x7f707810e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f707ca55700 1 --2- 172.21.15.87:0/808775384 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70780fe530 0x7f70781082f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7f7070009fd0 tx=0x7f7070004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f70757fa700 10 client.?.objecter ms_handle_connect 0x7f70780fe530 2024-02-23T00:35:52.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f70757fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:52.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707000e040 con 0x7f70780fe530 2024-02-23T00:35:52.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7070018040 con 0x7f70780fe530 2024-02-23T00:35:52.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7078108d90 con 0x7f70780fe530 2024-02-23T00:35:52.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070013e90 con 0x7f70780fe530 2024-02-23T00:35:52.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f707e4b8700 10 client.37643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:52.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7078114ee0 con 0x7f70780fe530 2024-02-23T00:35:52.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.570+0000 7f707e4b8700 1 librados: init done 2024-02-23T00:35:52.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.571+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f70781091d0 con 0x7f70780fe530 2024-02-23T00:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.571+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7070013590 con 0x7f70780fe530 2024-02-23T00:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.571+0000 7f70757fa700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7060062c90 0x7f7060065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(398..398 src has 173..398) v4 ==== 8106+0+0 (secure 0 0 0) 0x7f707007f0b0 con 0x7f70780fe530 2024-02-23T00:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 osd_map(398..398 src has 173..398) v4 2024-02-23T00:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70757fa700 3 client.37643.objecter handle_osd_map got epochs [398,398] > 0 2024-02-23T00:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70757fa700 3 client.37643.objecter handle_osd_map decoding full epoch 398 2024-02-23T00:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f7077fff700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7060062c90 0x7f7060065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70757fa700 20 client.37643.objecter dump_active .. 0 homeless 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v398) v1 ==== 54+0+16 (secure 0 0 0) 0x7f707007f590 con 0x7f70780fe530 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 10 client.37643.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 1 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70757fa700 10 client.37643.objecter ms_handle_connect 0x7f7060062c90 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f7077fff700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7060062c90 0x7f7060065150 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7f7068004ee0 tx=0x7f706800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6801/2153191863 conn(0x7f70780feee0 0x7f70780ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f707805fa60 osd=0 3 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f707805fa60 osd=0 3 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f707805fa60 osd=0 3 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 1 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f707805fa60 osd=0 3 2024-02-23T00:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 0 1 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f707805fa60 osd=0 4 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 10 client.37643.objecter _send_command 1 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f707805ee80 con 0x7f70780feee0 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70777fe700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6801/2153191863 conn(0x7f70780feee0 0x7f70780ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70757fa700 10 client.37643.objecter ms_handle_connect 0x7f70780feee0 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.572+0000 7f70777fe700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6801/2153191863 conn(0x7f70780feee0 0x7f70780ff2c0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.573+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f707805ee80 con 0x7f70780feee0 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.573+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.573+0000 7f70757fa700 10 client.37643.objecter _finish_command 1 = system:0 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.573+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f707805fa60 osd=0 4 2024-02-23T00:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.573+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 0 1 2024-02-23T00:35:52.582 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:35:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:52] "GET /metrics HTTP/1.1" 200 187865 "" "Prometheus/2.33.4" 2024-02-23T00:35:52.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 10 client.37643.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 2 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f707805fa60 osd=0 3 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f707805fa60 osd=0 3 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f707805fa60 osd=0 3 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 2 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f707805fa60 osd=0 3 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 0 2 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f707805fa60 osd=0 4 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 10 client.37643.objecter _send_command 2 2024-02-23T00:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.587+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7078107110 con 0x7f70780feee0 2024-02-23T00:35:52.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7078107110 con 0x7f70780feee0 2024-02-23T00:35:52.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:52.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f70757fa700 10 client.37643.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f707805fa60 osd=0 4 2024-02-23T00:35:52.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 0 2 2024-02-23T00:35:52.590 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:35:52.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 10 client.37643.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:52.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 3 2024-02-23T00:35:52.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6805/685354171 conn(0x7f70780598f0 0x7f7078059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078028540 osd=1 3 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078028540 osd=1 3 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078028540 osd=1 3 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 3 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f7078028540 osd=1 3 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 1 3 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078028540 osd=1 4 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 10 client.37643.objecter _send_command 3 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70780286b0 con 0x7f70780598f0 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.589+0000 7f7077fff700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6805/685354171 conn(0x7f70780598f0 0x7f7078059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.590+0000 7f70757fa700 10 client.37643.objecter ms_handle_connect 0x7f70780598f0 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.590+0000 7f7077fff700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6805/685354171 conn(0x7f70780598f0 0x7f7078059cd0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.590+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f70780286b0 con 0x7f70780598f0 2024-02-23T00:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.590+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:35:52.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.590+0000 7f70757fa700 10 client.37643.objecter _finish_command 3 = system:0 2024-02-23T00:35:52.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.590+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f7078028540 osd=1 4 2024-02-23T00:35:52.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.590+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 1 3 2024-02-23T00:35:52.598 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:52.605 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 10 client.37643.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 4 2024-02-23T00:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078028540 osd=1 3 2024-02-23T00:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078028540 osd=1 3 2024-02-23T00:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078028540 osd=1 3 2024-02-23T00:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 4 2024-02-23T00:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f7078028540 osd=1 3 2024-02-23T00:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 1 4 2024-02-23T00:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078028540 osd=1 4 2024-02-23T00:35:52.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 10 client.37643.objecter _send_command 4 2024-02-23T00:35:52.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.609+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7078059360 con 0x7f70780598f0 2024-02-23T00:35:52.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7078059360 con 0x7f70780598f0 2024-02-23T00:35:52.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:52.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f70757fa700 10 client.37643.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f7078028540 osd=1 4 2024-02-23T00:35:52.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 1 4 2024-02-23T00:35:52.612 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:35:52.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 10 client.37643.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:52.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 5 2024-02-23T00:35:52.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6809/3605394228 conn(0x7f707811a210 0x7f707811c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078027b90 osd=2 3 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027b90 osd=2 3 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078027b90 osd=2 3 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 5 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f7078027b90 osd=2 3 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 2 5 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027b90 osd=2 4 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 10 client.37643.objecter _send_command 5 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7078028210 con 0x7f707811a210 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.611+0000 7f70777fe700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6809/3605394228 conn(0x7f707811a210 0x7f707811c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.612+0000 7f70757fa700 10 client.37643.objecter ms_handle_connect 0x7f707811a210 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.612+0000 7f70777fe700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6809/3605394228 conn(0x7f707811a210 0x7f707811c600 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.612+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7078028210 con 0x7f707811a210 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.612+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.612+0000 7f70757fa700 10 client.37643.objecter _finish_command 5 = system:0 2024-02-23T00:35:52.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.612+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f7078027b90 osd=2 4 2024-02-23T00:35:52.615 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.612+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 2 5 2024-02-23T00:35:52.627 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:52.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 10 client.37643.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:52.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 6 2024-02-23T00:35:52.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078027b90 osd=2 3 2024-02-23T00:35:52.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027b90 osd=2 3 2024-02-23T00:35:52.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078027b90 osd=2 3 2024-02-23T00:35:52.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 6 2024-02-23T00:35:52.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f7078027b90 osd=2 3 2024-02-23T00:35:52.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 2 6 2024-02-23T00:35:52.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027b90 osd=2 4 2024-02-23T00:35:52.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 10 client.37643.objecter _send_command 6 2024-02-23T00:35:52.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.630+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7078027600 con 0x7f707811a210 2024-02-23T00:35:52.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.632+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7078027600 con 0x7f707811a210 2024-02-23T00:35:52.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.632+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:52.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.632+0000 7f70757fa700 10 client.37643.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.632+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f7078027b90 osd=2 4 2024-02-23T00:35:52.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.632+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 2 6 2024-02-23T00:35:52.634 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 10 client.37643.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 7 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6813/128059790 conn(0x7f7078119d70 0x7f707811fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078027a10 osd=3 3 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027a10 osd=3 3 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078027a10 osd=3 3 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 7 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f7078027a10 osd=3 3 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 3 7 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027a10 osd=3 4 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 10 client.37643.objecter _send_command 7 2024-02-23T00:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70781228d0 con 0x7f7078119d70 2024-02-23T00:35:52.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707ca55700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6813/128059790 conn(0x7f7078119d70 0x7f707811fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f707ca55700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6813/128059790 conn(0x7f7078119d70 0x7f707811fc80 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.633+0000 7f70757fa700 10 client.37643.objecter ms_handle_connect 0x7f7078119d70 2024-02-23T00:35:52.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.634+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f70781228d0 con 0x7f7078119d70 2024-02-23T00:35:52.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.634+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:35:52.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.634+0000 7f70757fa700 10 client.37643.objecter _finish_command 7 = system:0 2024-02-23T00:35:52.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.634+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f7078027a10 osd=3 4 2024-02-23T00:35:52.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.634+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 3 7 2024-02-23T00:35:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 10 client.37643.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 8 2024-02-23T00:35:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078027a10 osd=3 3 2024-02-23T00:35:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027a10 osd=3 3 2024-02-23T00:35:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078027a10 osd=3 3 2024-02-23T00:35:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 8 2024-02-23T00:35:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f7078027a10 osd=3 3 2024-02-23T00:35:52.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 3 8 2024-02-23T00:35:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027a10 osd=3 4 2024-02-23T00:35:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 10 client.37643.objecter _send_command 8 2024-02-23T00:35:52.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.650+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f70781192d0 con 0x7f7078119d70 2024-02-23T00:35:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f70781192d0 con 0x7f7078119d70 2024-02-23T00:35:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f70757fa700 10 client.37643.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f7078027a10 osd=3 4 2024-02-23T00:35:52.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 3 8 2024-02-23T00:35:52.653 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:35:52.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 10 client.37643.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:52.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 9 2024-02-23T00:35:52.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7078119470 0x7f70780f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781192d0 osd=4 3 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781192d0 osd=4 3 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781192d0 osd=4 3 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 9 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f70781192d0 osd=4 3 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 4 9 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781192d0 osd=4 4 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 10 client.37643.objecter _send_command 9 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70781237d0 con 0x7f7078119470 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.652+0000 7f7077fff700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7078119470 0x7f70780f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.653+0000 7f70757fa700 10 client.37643.objecter ms_handle_connect 0x7f7078119470 2024-02-23T00:35:52.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.653+0000 7f7077fff700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7078119470 0x7f70780f8640 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.653+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f70781237d0 con 0x7f7078119470 2024-02-23T00:35:52.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.653+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:35:52.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.653+0000 7f70757fa700 10 client.37643.objecter _finish_command 9 = system:0 2024-02-23T00:35:52.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.653+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f70781192d0 osd=4 4 2024-02-23T00:35:52.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.653+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 4 9 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 10 client.37643.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 10 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781192d0 osd=4 3 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781192d0 osd=4 3 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781192d0 osd=4 3 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 10 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f70781192d0 osd=4 3 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 4 10 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781192d0 osd=4 4 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 10 client.37643.objecter _send_command 10 2024-02-23T00:35:52.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.668+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f70781237d0 con 0x7f7078119470 2024-02-23T00:35:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.669+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f70781237d0 con 0x7f7078119470 2024-02-23T00:35:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.669+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.669+0000 7f70757fa700 10 client.37643.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f70781192d0 osd=4 4 2024-02-23T00:35:52.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 4 10 2024-02-23T00:35:52.671 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:35:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 10 client.37643.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 11 2024-02-23T00:35:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6804/2550013795 conn(0x7f707811d800 0x7f70780f9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781237d0 osd=5 3 2024-02-23T00:35:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781237d0 osd=5 3 2024-02-23T00:35:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781237d0 osd=5 3 2024-02-23T00:35:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 11 2024-02-23T00:35:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f70781237d0 osd=5 3 2024-02-23T00:35:52.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 5 11 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781237d0 osd=5 4 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 10 client.37643.objecter _send_command 11 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f707812aff0 con 0x7f707811d800 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.670+0000 7f70777fe700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6804/2550013795 conn(0x7f707811d800 0x7f70780f9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.671+0000 7f70757fa700 10 client.37643.objecter ms_handle_connect 0x7f707811d800 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.671+0000 7f70777fe700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6804/2550013795 conn(0x7f707811d800 0x7f70780f9040 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.671+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f707812aff0 con 0x7f707811d800 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.671+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.671+0000 7f70757fa700 10 client.37643.objecter _finish_command 11 = system:0 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.671+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f70781237d0 osd=5 4 2024-02-23T00:35:52.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.671+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 5 11 2024-02-23T00:35:52.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 10 client.37643.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:52.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 12 2024-02-23T00:35:52.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781237d0 osd=5 3 2024-02-23T00:35:52.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781237d0 osd=5 3 2024-02-23T00:35:52.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781237d0 osd=5 3 2024-02-23T00:35:52.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 12 2024-02-23T00:35:52.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f70781237d0 osd=5 3 2024-02-23T00:35:52.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 5 12 2024-02-23T00:35:52.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781237d0 osd=5 4 2024-02-23T00:35:52.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 10 client.37643.objecter _send_command 12 2024-02-23T00:35:52.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.685+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7078122180 con 0x7f707811d800 2024-02-23T00:35:52.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7078122180 con 0x7f707811d800 2024-02-23T00:35:52.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:52.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f70757fa700 10 client.37643.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f70781237d0 osd=5 4 2024-02-23T00:35:52.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 5 12 2024-02-23T00:35:52.689 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 10 client.37643.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 13 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7078122690 0x7f7078122ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781234f0 osd=6 3 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781234f0 osd=6 3 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781234f0 osd=6 3 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 13 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f70781234f0 osd=6 3 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 6 13 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781234f0 osd=6 4 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 10 client.37643.objecter _send_command 13 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.687+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7078123050 con 0x7f7078122690 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.688+0000 7f707ca55700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7078122690 0x7f7078122ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.688+0000 7f70757fa700 10 client.37643.objecter ms_handle_connect 0x7f7078122690 2024-02-23T00:35:52.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.688+0000 7f707ca55700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7078122690 0x7f7078122ad0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.688+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7078123050 con 0x7f7078122690 2024-02-23T00:35:52.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.688+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:35:52.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.688+0000 7f70757fa700 10 client.37643.objecter _finish_command 13 = system:0 2024-02-23T00:35:52.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.688+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f70781234f0 osd=6 4 2024-02-23T00:35:52.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.688+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 6 13 2024-02-23T00:35:52.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 10 client.37643.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 14 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781234f0 osd=6 3 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781234f0 osd=6 3 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f70781234f0 osd=6 3 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 14 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f70781234f0 osd=6 3 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 6 14 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781234f0 osd=6 4 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 10 client.37643.objecter _send_command 14 2024-02-23T00:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.702+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7078123050 con 0x7f7078122690 2024-02-23T00:35:52.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.704+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7078123050 con 0x7f7078122690 2024-02-23T00:35:52.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.704+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:52.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.704+0000 7f70757fa700 10 client.37643.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.704+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f70781234f0 osd=6 4 2024-02-23T00:35:52.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.704+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 6 14 2024-02-23T00:35:52.706 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 10 client.37643.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 15 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6812/373787000 conn(0x7f707812c390 0x7f707812e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078123050 osd=7 3 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078123050 osd=7 3 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078123050 osd=7 3 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 15 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f7078123050 osd=7 3 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 7 15 2024-02-23T00:35:52.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078123050 osd=7 4 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 10 client.37643.objecter _send_command 15 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7078131440 con 0x7f707812c390 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f7077fff700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6812/373787000 conn(0x7f707812c390 0x7f707812e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f7077fff700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6812/373787000 conn(0x7f707812c390 0x7f707812e780 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.705+0000 7f70757fa700 10 client.37643.objecter ms_handle_connect 0x7f707812c390 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.706+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7078131440 con 0x7f707812c390 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.706+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.706+0000 7f70757fa700 10 client.37643.objecter _finish_command 15 = system:0 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.706+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f7078123050 osd=7 4 2024-02-23T00:35:52.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.706+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 7 15 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 10 client.37643.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign -1 16 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078123050 osd=7 3 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078123050 osd=7 3 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 20 client.37643.objecter _get_session s=0x7f7078123050 osd=7 3 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_remove -1 16 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 20 client.37643.objecter get_session s=0x7f7078123050 osd=7 3 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 15 client.37643.objecter _session_command_op_assign 7 16 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078123050 osd=7 4 2024-02-23T00:35:52.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 10 client.37643.objecter _send_command 16 2024-02-23T00:35:52.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.720+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f7078131440 con 0x7f707812c390 2024-02-23T00:35:52.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f70757fa700 1 -- 172.21.15.87:0/808775384 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f7078131440 con 0x7f707812c390 2024-02-23T00:35:52.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f70757fa700 10 client.37643.objecter ms_dispatch 0x7f7078001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:35:52.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f70757fa700 10 client.37643.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f70757fa700 20 client.37643.objecter put_session s=0x7f7078123050 osd=7 4 2024-02-23T00:35:52.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f70757fa700 15 client.37643.objecter _session_command_op_remove 7 16 2024-02-23T00:35:52.725 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:35:52.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 librados: watch_flush enter 2024-02-23T00:35:52.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 librados: watch_flush exit 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 client.37643.objecter close_session for osd.0 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6801/2153191863 conn(0x7f70780feee0 msgr2=0x7f70780ff2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6801/2153191863 conn(0x7f70780feee0 0x7f70780ff2c0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f707805fa60 osd=0 1 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 client.37643.objecter close_session for osd.1 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6805/685354171 conn(0x7f70780598f0 msgr2=0x7f7078059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6805/685354171 conn(0x7f70780598f0 0x7f7078059cd0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078028540 osd=1 1 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 client.37643.objecter close_session for osd.2 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6809/3605394228 conn(0x7f707811a210 msgr2=0x7f707811c600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6809/3605394228 conn(0x7f707811a210 0x7f707811c600 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027b90 osd=2 1 2024-02-23T00:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 client.37643.objecter close_session for osd.3 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6813/128059790 conn(0x7f7078119d70 msgr2=0x7f707811fc80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6813/128059790 conn(0x7f7078119d70 0x7f707811fc80 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078027a10 osd=3 1 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 client.37643.objecter close_session for osd.4 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7078119470 msgr2=0x7f70780f8640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7078119470 0x7f70780f8640 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707ca55700 1 -- 172.21.15.87:0/808775384 reap_dead start 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707ca55700 1 -- 172.21.15.87:0/808775384 reap_dead start 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781192d0 osd=4 1 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 client.37643.objecter close_session for osd.5 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6804/2550013795 conn(0x7f707811d800 msgr2=0x7f70780f9040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6804/2550013795 conn(0x7f707811d800 0x7f70780f9040 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781237d0 osd=5 1 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 client.37643.objecter close_session for osd.6 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7078122690 msgr2=0x7f7078122ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7078122690 0x7f7078122ad0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f70781234f0 osd=6 1 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 client.37643.objecter close_session for osd.7 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6812/373787000 conn(0x7f707812c390 msgr2=0x7f707812e780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.171:6812/373787000 conn(0x7f707812c390 0x7f707812e780 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 20 client.37643.objecter put_session s=0x7f7078123050 osd=7 1 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 20 client.37643.objecter shutdown clearing up homeless session... 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 10 client.37643.objecter successfully canceled tick 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7060062c90 msgr2=0x7f7060065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7060062c90 0x7f7060065150 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7f7068004ee0 tx=0x7f706800b040).stop 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70780fe530 msgr2=0x7f70781082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 --2- 172.21.15.87:0/808775384 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70780fe530 0x7f70781082f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 rx=0x7f7070009fd0 tx=0x7f7070004d10).stop 2024-02-23T00:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707ca55700 1 -- 172.21.15.87:0/808775384 reap_dead start 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 shutdown_connections 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 >> 172.21.15.87:0/808775384 conn(0x7f707806c370 msgr2=0x7f70780f5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 shutdown_connections 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 -- 172.21.15.87:0/808775384 wait complete. 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:52.722+0000 7f707e4b8700 1 librados: shutdown 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:35:53.180 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:53.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:53.284 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:35:53.313 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:53.331 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:35:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:53.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:35:53.435 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:51.645123+0000 mgr.y (mgr.24437) 367 : cluster [DBG] pgmap v580: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:51.780209+0000 osd.1 (osd.1) 171 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:51.783084+0000 osd.1 (osd.1) 172 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:52.094571+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:52.094695+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:52.138190+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:52.138387+0000 mon.a (mon.0) 2526 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:52.139118+0000 mon.b (mon.1) 342 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-55"}]: dispatch 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:52.139318+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-55"}]: dispatch 2024-02-23T00:35:53.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:52.571637+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.87:0/808775384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:53.461 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:53.479 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:53.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:51.645123+0000 mgr.y (mgr.24437) 367 : cluster [DBG] pgmap v580: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:35:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:51.780209+0000 osd.1 (osd.1) 171 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:35:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:51.783084+0000 osd.1 (osd.1) 172 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:35:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:52.094571+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:52.094695+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-02-23T00:35:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:52.138190+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:52.138387+0000 mon.a (mon.0) 2526 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:53.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:52.139118+0000 mon.b (mon.1) 342 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-55"}]: dispatch 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:52.139318+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-55"}]: dispatch 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:52.571637+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.87:0/808775384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:51.645123+0000 mgr.y (mgr.24437) 367 : cluster [DBG] pgmap v580: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:51.780209+0000 osd.1 (osd.1) 171 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:51.783084+0000 osd.1 (osd.1) 172 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:52.094571+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:52.094695+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:52.138190+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:52.138387+0000 mon.a (mon.0) 2526 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:52.139118+0000 mon.b (mon.1) 342 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-55"}]: dispatch 2024-02-23T00:35:53.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:52.139318+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-55"}]: dispatch 2024-02-23T00:35:53.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:52.571637+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.87:0/808775384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:35:53.585 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:35:53.611 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:53.630 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:35:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:53.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:35:53.734 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:53.764 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:53.776 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:35:53.776 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:35:53.776 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-23T00:35:53.776 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.947396 2024-02-23T00:35:53.777 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 4 0.9473964383286395 2024-02-23T00:35:53.817 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:53.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:35:53.921 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:53.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:35:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:35:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:35:53.948 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:53.966 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:53.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:54.070 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:54.104 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:54.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:54.239 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:35:54.264 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:52.172211+0000 osd.4 (osd.4) 179 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:35:54.264 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:52.175238+0000 osd.4 (osd.4) 180 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:35:54.264 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:52.290051+0000 osd.5 (osd.5) 157 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:35:54.264 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:52.290939+0000 osd.5 (osd.5) 158 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:35:54.264 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:52.788781+0000 osd.1 (osd.1) 173 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:35:54.264 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:52.791741+0000 osd.1 (osd.1) 174 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:35:54.264 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:52.921703+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:35:54.264 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:52.924802+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:35:54.264 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:53.099415+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2024-02-23T00:35:54.265 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:54.284 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:35:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:54.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:35:54.367 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:35:54.368 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:35:54.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:52.172211+0000 osd.4 (osd.4) 179 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:35:54.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:52.175238+0000 osd.4 (osd.4) 180 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:35:54.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:52.290051+0000 osd.5 (osd.5) 157 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:35:54.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:52.290939+0000 osd.5 (osd.5) 158 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:35:54.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:52.788781+0000 osd.1 (osd.1) 173 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:35:54.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:52.791741+0000 osd.1 (osd.1) 174 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:35:54.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:52.921703+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:35:54.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:52.924802+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:35:54.378 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:53.099415+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2024-02-23T00:35:54.379 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:52.172211+0000 osd.4 (osd.4) 179 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:35:54.379 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:52.175238+0000 osd.4 (osd.4) 180 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:35:54.379 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:52.290051+0000 osd.5 (osd.5) 157 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:35:54.379 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:52.290939+0000 osd.5 (osd.5) 158 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:35:54.379 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:52.788781+0000 osd.1 (osd.1) 173 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:35:54.379 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:52.791741+0000 osd.1 (osd.1) 174 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:35:54.379 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:52.921703+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:35:54.379 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:52.924802+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:35:54.379 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:53.099415+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2024-02-23T00:35:54.388 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:54.416 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:54.614 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:54.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:54.717 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:54.744 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:53.152608+0000 osd.4 (osd.4) 181 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:35:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:53.153814+0000 osd.4 (osd.4) 182 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:35:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:53.645663+0000 mgr.y (mgr.24437) 368 : cluster [DBG] pgmap v583: 206 pgs: 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:53.843535+0000 osd.3 (osd.3) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:53.843654+0000 osd.3 (osd.3) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:54.026390+0000 osd.6 (osd.6) 145 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:54.029408+0000 osd.6 (osd.6) 146 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:54.103520+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:54.112954+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:54.117650+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:54.156124+0000 osd.2 (osd.2) 151 : cluster [DBG] 6.f scrub starts 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:54.157204+0000 osd.2 (osd.2) 152 : cluster [DBG] 6.f scrub ok 2024-02-23T00:35:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:54.649644+0000 mon.a (mon.0) 2531 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:53.152608+0000 osd.4 (osd.4) 181 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:35:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:53.153814+0000 osd.4 (osd.4) 182 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:35:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:53.645663+0000 mgr.y (mgr.24437) 368 : cluster [DBG] pgmap v583: 206 pgs: 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:35:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:53.843535+0000 osd.3 (osd.3) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:53.843654+0000 osd.3 (osd.3) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:54.026390+0000 osd.6 (osd.6) 145 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:54.029408+0000 osd.6 (osd.6) 146 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:54.103520+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:54.112954+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:54.117650+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:54.156124+0000 osd.2 (osd.2) 151 : cluster [DBG] 6.f scrub starts 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:54.157204+0000 osd.2 (osd.2) 152 : cluster [DBG] 6.f scrub ok 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:54.649644+0000 mon.a (mon.0) 2531 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:53.152608+0000 osd.4 (osd.4) 181 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:53.153814+0000 osd.4 (osd.4) 182 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:35:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:53.645663+0000 mgr.y (mgr.24437) 368 : cluster [DBG] pgmap v583: 206 pgs: 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:53.843535+0000 osd.3 (osd.3) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:53.843654+0000 osd.3 (osd.3) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:54.026390+0000 osd.6 (osd.6) 145 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:54.029408+0000 osd.6 (osd.6) 146 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:54.103520+0000 mon.a (mon.0) 2529 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:54.112954+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:54.117650+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:54.156124+0000 osd.2 (osd.2) 151 : cluster [DBG] 6.f scrub starts 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:54.157204+0000 osd.2 (osd.2) 152 : cluster [DBG] 6.f scrub ok 2024-02-23T00:35:55.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:54.649644+0000 mon.a (mon.0) 2531 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:35:56.093 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-23T00:35:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:54.853710+0000 osd.7 (osd.7) 193 : cluster [DBG] 63.18 scrub starts 2024-02-23T00:35:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:54.857180+0000 osd.7 (osd.7) 194 : cluster [DBG] 63.18 scrub ok 2024-02-23T00:35:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:55.105084+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:55.105146+0000 mon.a (mon.0) 2533 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2024-02-23T00:35:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:55.124421+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:35:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:55.150002+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:55.150520+0000 mon.a (mon.0) 2534 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:55.151207+0000 mon.b (mon.1) 346 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-57"}]: dispatch 2024-02-23T00:35:56.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:55.151453+0000 mon.a (mon.0) 2535 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-57"}]: dispatch 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:55.173467+0000 osd.0 (osd.0) 187 : cluster [DBG] 22.a scrub starts 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:55.176431+0000 osd.0 (osd.0) 188 : cluster [DBG] 22.a scrub ok 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:54.853710+0000 osd.7 (osd.7) 193 : cluster [DBG] 63.18 scrub starts 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:54.857180+0000 osd.7 (osd.7) 194 : cluster [DBG] 63.18 scrub ok 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:55.105084+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:55.105146+0000 mon.a (mon.0) 2533 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:55.124421+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:55.150002+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:55.150520+0000 mon.a (mon.0) 2534 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:55.151207+0000 mon.b (mon.1) 346 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-57"}]: dispatch 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:55.151453+0000 mon.a (mon.0) 2535 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-57"}]: dispatch 2024-02-23T00:35:56.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:55.173467+0000 osd.0 (osd.0) 187 : cluster [DBG] 22.a scrub starts 2024-02-23T00:35:56.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:55.176431+0000 osd.0 (osd.0) 188 : cluster [DBG] 22.a scrub ok 2024-02-23T00:35:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:54.853710+0000 osd.7 (osd.7) 193 : cluster [DBG] 63.18 scrub starts 2024-02-23T00:35:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:54.857180+0000 osd.7 (osd.7) 194 : cluster [DBG] 63.18 scrub ok 2024-02-23T00:35:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:55.105084+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:55.105146+0000 mon.a (mon.0) 2533 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2024-02-23T00:35:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:55.124421+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:35:56.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:55.150002+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:56.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:55.150520+0000 mon.a (mon.0) 2534 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:56.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:55.151207+0000 mon.b (mon.1) 346 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-57"}]: dispatch 2024-02-23T00:35:56.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:55.151453+0000 mon.a (mon.0) 2535 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-57"}]: dispatch 2024-02-23T00:35:56.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:55.173467+0000 osd.0 (osd.0) 187 : cluster [DBG] 22.a scrub starts 2024-02-23T00:35:56.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:55.176431+0000 osd.0 (osd.0) 188 : cluster [DBG] 22.a scrub ok 2024-02-23T00:35:56.894 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:56.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:56.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:56.913 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:56.999 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:57.032 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:57.054 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:57.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:57.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:57.158 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:57.192 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:57.212 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:57.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:57.316 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:35:57.344 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:55.222122+0000 osd.4 (osd.4) 183 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:35:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:55.222908+0000 osd.4 (osd.4) 184 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:35:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:55.646621+0000 mgr.y (mgr.24437) 369 : cluster [DBG] pgmap v586: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-23T00:35:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:56.122535+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2024-02-23T00:35:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:56.675762+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-02-23T00:35:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:56.678672+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-02-23T00:35:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.076990+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2024-02-23T00:35:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:57.088290+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:57.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:57.088533+0000 mon.a (mon.0) 2538 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:57.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 -- 172.21.15.87:0/2845525826 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0fe510 msgr2=0x7f030c0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:57.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 --2- 172.21.15.87:0/2845525826 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0fe510 0x7f030c0f7d40 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 rx=0x7f03000047b0 tx=0x7f0300009cb0).stop 2024-02-23T00:35:57.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 -- 172.21.15.87:0/2845525826 shutdown_connections 2024-02-23T00:35:57.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 --2- 172.21.15.87:0/2845525826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f030c0f8420 0x7f030c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 --2- 172.21.15.87:0/2845525826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f030c0fdbb0 0x7f030c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 --2- 172.21.15.87:0/2845525826 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0fe510 0x7f030c0f7d40 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 -- 172.21.15.87:0/2845525826 >> 172.21.15.87:0/2845525826 conn(0x7f030c0f3800 msgr2=0x7f030c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 -- 172.21.15.87:0/2845525826 shutdown_connections 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 -- 172.21.15.87:0/2845525826 wait complete. 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 librados: starting msgr at 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 librados: starting objecter 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.416+0000 7f0311299700 1 Processor -- start 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f0311299700 1 -- start start 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f0311299700 1 librados: setting wanted keys 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f0311299700 1 librados: calling monclient init 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f0311299700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0f8420 0x7f030c10e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f0311299700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f030c0fdbb0 0x7f030c10e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:57.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f0311299700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f030c0fe510 0x7f030c1081b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f0311299700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c1086a0 con 0x7f030c0fdbb0 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f0311299700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030c108810 con 0x7f030c0fe510 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f0311299700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030c1089d0 con 0x7f030c0f8420 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0f8420 0x7f030c10e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030a7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f030c0fdbb0 0x7f030c10e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0f8420 0x7f030c10e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50138/0 (socket says 172.21.15.87:50138) 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030affd700 1 -- 172.21.15.87:0/4274659730 learned_addr learned my addr 172.21.15.87:0/4274659730 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030b7fe700 1 --2- 172.21.15.87:0/4274659730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f030c0fe510 0x7f030c1081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030affd700 1 -- 172.21.15.87:0/4274659730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f030c0fe510 msgr2=0x7f030c1081b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030affd700 1 --2- 172.21.15.87:0/4274659730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f030c0fe510 0x7f030c1081b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030affd700 1 -- 172.21.15.87:0/4274659730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f030c0fdbb0 msgr2=0x7f030c10e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030affd700 1 --2- 172.21.15.87:0/4274659730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f030c0fdbb0 0x7f030c10e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030affd700 1 -- 172.21.15.87:0/4274659730 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040097c0 con 0x7f030c0f8420 2024-02-23T00:35:57.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030a7fc700 1 --2- 172.21.15.87:0/4274659730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f030c0fdbb0 0x7f030c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f030affd700 1 --2- 172.21.15.87:0/4274659730 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0f8420 0x7f030c10e010 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7f02fc0041d0 tx=0x7f02fc009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f02f3fff700 10 client.?.objecter ms_handle_connect 0x7f030c0f8420 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f02f3fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.417+0000 7f02f3fff700 1 -- 172.21.15.87:0/4274659730 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc007b40 con 0x7f030c0f8420 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.418+0000 7f02f3fff700 1 -- 172.21.15.87:0/4274659730 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02fc00f040 con 0x7f030c0f8420 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.418+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0300005040 con 0x7f030c0f8420 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.418+0000 7f02f3fff700 1 -- 172.21.15.87:0/4274659730 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc014810 con 0x7f030c0f8420 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.418+0000 7f0311299700 10 client.37649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.418+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030c109010 con 0x7f030c0f8420 2024-02-23T00:35:57.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.418+0000 7f0311299700 1 librados: init done 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.418+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030c107110 con 0x7f030c0f8420 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.419+0000 7f02f3fff700 1 -- 172.21.15.87:0/4274659730 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f02fc009110 con 0x7f030c0f8420 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.419+0000 7f02f3fff700 1 --2- 172.21.15.87:0/4274659730 >> v2:172.21.15.87:6800/1231597463 conn(0x7f02f4062c90 0x7f02f4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.419+0000 7f02f3fff700 1 -- 172.21.15.87:0/4274659730 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(403..403 src has 173..403) v4 ==== 8093+0+0 (secure 0 0 0) 0x7f02fc07d210 con 0x7f030c0f8420 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.419+0000 7f02f3fff700 10 client.37649.objecter ms_dispatch 0x7f030c001550 osd_map(403..403 src has 173..403) v4 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.419+0000 7f02f3fff700 3 client.37649.objecter handle_osd_map got epochs [403,403] > 0 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.419+0000 7f02f3fff700 3 client.37649.objecter handle_osd_map decoding full epoch 403 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.419+0000 7f030a7fc700 1 --2- 172.21.15.87:0/4274659730 >> v2:172.21.15.87:6800/1231597463 conn(0x7f02f4062c90 0x7f02f4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.419+0000 7f02f3fff700 20 client.37649.objecter dump_active .. 0 homeless 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.420+0000 7f02f3fff700 10 client.37649.objecter ms_handle_connect 0x7f02f4062c90 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.420+0000 7f030a7fc700 1 --2- 172.21.15.87:0/4274659730 >> v2:172.21.15.87:6800/1231597463 conn(0x7f02f4062c90 0x7f02f4065150 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7f030c109950 tx=0x7f0300007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:35:57.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.423+0000 7f02f3fff700 1 -- 172.21.15.87:0/4274659730 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f02fc050700 con 0x7f030c0f8420 2024-02-23T00:35:57.467 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:35:57.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:55.222122+0000 osd.4 (osd.4) 183 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:35:57.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:55.222908+0000 osd.4 (osd.4) 184 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:35:57.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:55.646621+0000 mgr.y (mgr.24437) 369 : cluster [DBG] pgmap v586: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-23T00:35:57.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:56.122535+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2024-02-23T00:35:57.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:56.675762+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-02-23T00:35:57.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:56.678672+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-02-23T00:35:57.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.076990+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2024-02-23T00:35:57.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:57.088290+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:57.470 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:57.088533+0000 mon.a (mon.0) 2538 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:57.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:55.222122+0000 osd.4 (osd.4) 183 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:35:57.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:55.222908+0000 osd.4 (osd.4) 184 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:35:57.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:55.646621+0000 mgr.y (mgr.24437) 369 : cluster [DBG] pgmap v586: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-23T00:35:57.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:56.122535+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2024-02-23T00:35:57.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:56.675762+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-02-23T00:35:57.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:56.678672+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-02-23T00:35:57.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.076990+0000 mon.a (mon.0) 2537 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2024-02-23T00:35:57.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:57.088290+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:57.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:57.088533+0000 mon.a (mon.0) 2538 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:35:57.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:57.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:35:57.573 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:35:57.601 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:57.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:57.614+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f030c109690 con 0x7f030c0f8420 2024-02-23T00:35:58.060 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:35:58.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.081+0000 7f02f3fff700 1 -- 172.21.15.87:0/4274659730 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v404) v1 ==== 87+0+0 (secure 0 0 0) 0x7f02fc0502f0 con 0x7f030c0f8420 2024-02-23T00:35:58.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.085+0000 7f0311299700 10 librados: watch_flush enter 2024-02-23T00:35:58.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.085+0000 7f0311299700 10 librados: watch_flush exit 2024-02-23T00:35:58.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.085+0000 7f0311299700 20 client.37649.objecter shutdown clearing up homeless session... 2024-02-23T00:35:58.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.085+0000 7f0311299700 10 client.37649.objecter successfully canceled tick 2024-02-23T00:35:58.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.087+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 >> v2:172.21.15.87:6800/1231597463 conn(0x7f02f4062c90 msgr2=0x7f02f4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:58.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.087+0000 7f0311299700 1 --2- 172.21.15.87:0/4274659730 >> v2:172.21.15.87:6800/1231597463 conn(0x7f02f4062c90 0x7f02f4065150 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 rx=0x7f030c109950 tx=0x7f0300007f90).stop 2024-02-23T00:35:58.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.087+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0f8420 msgr2=0x7f030c10e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:35:58.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.087+0000 7f0311299700 1 --2- 172.21.15.87:0/4274659730 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0f8420 0x7f030c10e010 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 rx=0x7f02fc0041d0 tx=0x7f02fc009f60).stop 2024-02-23T00:35:58.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.090+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 shutdown_connections 2024-02-23T00:35:58.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.090+0000 7f0311299700 1 --2- 172.21.15.87:0/4274659730 >> v2:172.21.15.87:6800/1231597463 conn(0x7f02f4062c90 0x7f02f4065150 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:58.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.090+0000 7f0311299700 1 --2- 172.21.15.87:0/4274659730 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f030c0fe510 0x7f030c1081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:58.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.090+0000 7f0311299700 1 --2- 172.21.15.87:0/4274659730 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f030c0f8420 0x7f030c10e010 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:58.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.090+0000 7f0311299700 1 --2- 172.21.15.87:0/4274659730 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f030c0fdbb0 0x7f030c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:35:58.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.090+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 >> 172.21.15.87:0/4274659730 conn(0x7f030c0f3800 msgr2=0x7f030c0f5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:35:58.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.090+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 shutdown_connections 2024-02-23T00:35:58.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.090+0000 7f0311299700 1 -- 172.21.15.87:0/4274659730 wait complete. 2024-02-23T00:35:58.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:35:58.090+0000 7f0311299700 1 librados: shutdown 2024-02-23T00:35:58.100 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:35:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:56.255163+0000 osd.4 (osd.4) 185 : cluster [DBG] 6.e scrub starts 2024-02-23T00:35:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:56.258051+0000 osd.4 (osd.4) 186 : cluster [DBG] 6.e scrub ok 2024-02-23T00:35:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.130247+0000 osd.2 (osd.2) 153 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:35:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.131134+0000 osd.2 (osd.2) 154 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:35:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.172111+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:35:58.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.175163+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:57.615395+0000 mon.c (mon.2) 429 : audit [INF] from='client.? 172.21.15.87:0/4274659730' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:57.615755+0000 mon.a (mon.0) 2539 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.654929+0000 osd.1 (osd.1) 177 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.658402+0000 osd.1 (osd.1) 178 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:58.074064+0000 mon.a (mon.0) 2540 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:58.079102+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:58.079173+0000 mon.a (mon.0) 2542 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:58.079220+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:58.089725+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:58.094557+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:35:58.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:58.096594+0000 mon.a (mon.0) 2544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:56.255163+0000 osd.4 (osd.4) 185 : cluster [DBG] 6.e scrub starts 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:56.258051+0000 osd.4 (osd.4) 186 : cluster [DBG] 6.e scrub ok 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.130247+0000 osd.2 (osd.2) 153 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.131134+0000 osd.2 (osd.2) 154 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.172111+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.175163+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:57.615395+0000 mon.c (mon.2) 429 : audit [INF] from='client.? 172.21.15.87:0/4274659730' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:57.615755+0000 mon.a (mon.0) 2539 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.654929+0000 osd.1 (osd.1) 177 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.658402+0000 osd.1 (osd.1) 178 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:58.074064+0000 mon.a (mon.0) 2540 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:58.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:58.079102+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:58.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:58.079173+0000 mon.a (mon.0) 2542 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:35:58.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:58.079220+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2024-02-23T00:35:58.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:58.089725+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:35:58.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:58.094557+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:35:58.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:58.096594+0000 mon.a (mon.0) 2544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:35:58.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:56.255163+0000 osd.4 (osd.4) 185 : cluster [DBG] 6.e scrub starts 2024-02-23T00:35:58.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:56.258051+0000 osd.4 (osd.4) 186 : cluster [DBG] 6.e scrub ok 2024-02-23T00:35:58.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.130247+0000 osd.2 (osd.2) 153 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:35:58.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.131134+0000 osd.2 (osd.2) 154 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.172111+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.175163+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:57.615395+0000 mon.c (mon.2) 429 : audit [INF] from='client.? 172.21.15.87:0/4274659730' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:57.615755+0000 mon.a (mon.0) 2539 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.654929+0000 osd.1 (osd.1) 177 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.658402+0000 osd.1 (osd.1) 178 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:58.074064+0000 mon.a (mon.0) 2540 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:58.079102+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:58.079173+0000 mon.a (mon.0) 2542 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:58.079220+0000 mon.a (mon.0) 2543 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:58.089725+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:35:58.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:58.094557+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:35:58.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:58.096594+0000 mon.a (mon.0) 2544 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:35:58.554 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:35:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:58.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:35:58.658 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:35:58.690 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:58.711 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:35:58.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:58.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:35:58.815 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:35:58.843 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:35:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.647079+0000 mgr.y (mgr.24437) 370 : cluster [DBG] pgmap v589: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:35:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.692294+0000 osd.2 (osd.2) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:57.692464+0000 osd.2 (osd.2) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:58.090141+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:35:59.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:59.079756+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:35:59.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:59.079832+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2024-02-23T00:35:59.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:59.111675+0000 mon.b (mon.1) 350 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:59.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:59.111930+0000 mon.a (mon.0) 2547 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:59.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:59.112711+0000 mon.b (mon.1) 351 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-59"}]: dispatch 2024-02-23T00:35:59.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:35:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:35:59.112806+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-59"}]: dispatch 2024-02-23T00:35:59.460 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.647079+0000 mgr.y (mgr.24437) 370 : cluster [DBG] pgmap v589: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.692294+0000 osd.2 (osd.2) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:57.692464+0000 osd.2 (osd.2) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:58.090141+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:59.079756+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:59.079832+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:59.111675+0000 mon.b (mon.1) 350 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:59.111930+0000 mon.a (mon.0) 2547 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:59.112711+0000 mon.b (mon.1) 351 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-59"}]: dispatch 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:35:59.112806+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-59"}]: dispatch 2024-02-23T00:35:59.461 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.647079+0000 mgr.y (mgr.24437) 370 : cluster [DBG] pgmap v589: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-02-23T00:35:59.462 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.692294+0000 osd.2 (osd.2) 155 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:35:59.462 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:57.692464+0000 osd.2 (osd.2) 156 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:35:59.462 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:58.090141+0000 osd.6 (osd.6) 147 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:35:59.462 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:59.079756+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:35:59.462 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:59.079832+0000 mon.a (mon.0) 2546 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2024-02-23T00:35:59.462 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:59.111675+0000 mon.b (mon.1) 350 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:59.462 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:59.111930+0000 mon.a (mon.0) 2547 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:35:59.462 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:59.112711+0000 mon.b (mon.1) 351 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-59"}]: dispatch 2024-02-23T00:35:59.462 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:35:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:35:59.112806+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-59"}]: dispatch 2024-02-23T00:35:59.921 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:35:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:35:59.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:00.025 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:00.053 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:00.084 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:36:00.425 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:59.320679+0000 osd.6 (osd.6) 148 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:00.425 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:59.320883+0000 osd.6 (osd.6) 149 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:00.425 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:00.081314+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2024-02-23T00:36:00.425 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:59.320679+0000 osd.6 (osd.6) 148 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:00.425 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:59.320883+0000 osd.6 (osd.6) 149 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:00.426 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:00.081314+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2024-02-23T00:36:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:59.320679+0000 osd.6 (osd.6) 148 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:59.320883+0000 osd.6 (osd.6) 149 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:00.081314+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2024-02-23T00:36:00.802 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:00.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:00.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:00.907 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:00.938 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:01.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:36:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:36:01.479 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:36:01.614 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:59.422923+0000 osd.5 (osd.5) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:01.614 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:59.423097+0000 osd.5 (osd.5) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:01.614 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:35:59.647705+0000 mgr.y (mgr.24437) 371 : cluster [DBG] pgmap v592: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:36:01.614 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:00.705307+0000 osd.1 (osd.1) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:01.614 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:00.705665+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:01.614 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:00.882920+0000 mgr.y (mgr.24437) 372 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:01.614 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:01.083798+0000 mon.a (mon.0) 2550 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:01.088491+0000 mon.b (mon.1) 352 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:01.089166+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:59.422923+0000 osd.5 (osd.5) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:59.423097+0000 osd.5 (osd.5) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:35:59.647705+0000 mgr.y (mgr.24437) 371 : cluster [DBG] pgmap v592: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:00.705307+0000 osd.1 (osd.1) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:00.705665+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:00.882920+0000 mgr.y (mgr.24437) 372 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:01.083798+0000 mon.a (mon.0) 2550 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:01.088491+0000 mon.b (mon.1) 352 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:01.615 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:01.089166+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:01.618 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:36:01.618 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:36:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:59.422923+0000 osd.5 (osd.5) 159 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:59.423097+0000 osd.5 (osd.5) 160 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:01.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:35:59.647705+0000 mgr.y (mgr.24437) 371 : cluster [DBG] pgmap v592: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:36:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:00.705307+0000 osd.1 (osd.1) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:00.705665+0000 osd.1 (osd.1) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:00.882920+0000 mgr.y (mgr.24437) 372 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:01.083798+0000 mon.a (mon.0) 2550 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2024-02-23T00:36:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:01.088491+0000 mon.b (mon.1) 352 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:01.089166+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:01.887 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:02.115 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:02.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:02.221 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:02.250 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:02.270 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:02.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:02.374 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:02.382 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:36:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:02] "GET /metrics HTTP/1.1" 200 187956 "" "Prometheus/2.33.4" 2024-02-23T00:36:02.404 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:02.853 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:02.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:02.874 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-23T00:36:02.957 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:02.990 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:03.011 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:03.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:03.115 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:36:03.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:01.648095+0000 mgr.y (mgr.24437) 373 : cluster [DBG] pgmap v595: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-23T00:36:03.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:02.086508+0000 mon.a (mon.0) 2552 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:36:03.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:02.086654+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2024-02-23T00:36:03.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:02.097117+0000 mon.b (mon.1) 353 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:36:03.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:02.101166+0000 mon.b (mon.1) 354 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:03.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:02.101870+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:03.122 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:02.767325+0000 mon.a (mon.0) 2555 : audit [INF] from='client.? 172.21.15.87:0/751041035' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9473964383286395}]: dispatch 2024-02-23T00:36:03.123 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:01.648095+0000 mgr.y (mgr.24437) 373 : cluster [DBG] pgmap v595: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-23T00:36:03.123 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:02.086508+0000 mon.a (mon.0) 2552 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:36:03.123 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:02.086654+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2024-02-23T00:36:03.123 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:02.097117+0000 mon.b (mon.1) 353 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:36:03.123 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:02.101166+0000 mon.b (mon.1) 354 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:03.123 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:02.101870+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:03.123 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:02.767325+0000 mon.a (mon.0) 2555 : audit [INF] from='client.? 172.21.15.87:0/751041035' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9473964383286395}]: dispatch 2024-02-23T00:36:03.146 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:03.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.580+0000 7f02b7a12700 1 -- 172.21.15.87:0/4262003563 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00fe510 msgr2=0x7f02b00f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:03.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.580+0000 7f02b7a12700 1 --2- 172.21.15.87:0/4262003563 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00fe510 0x7f02b00f7d40 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 rx=0x7f02ac0047b0 tx=0x7f02ac009cb0).stop 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.580+0000 7f02b7a12700 1 -- 172.21.15.87:0/4262003563 shutdown_connections 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.580+0000 7f02b7a12700 1 --2- 172.21.15.87:0/4262003563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02b00f8420 0x7f02b0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.580+0000 7f02b7a12700 1 --2- 172.21.15.87:0/4262003563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02b00fdbb0 0x7f02b00fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.580+0000 7f02b7a12700 1 --2- 172.21.15.87:0/4262003563 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00fe510 0x7f02b00f7d40 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.580+0000 7f02b7a12700 1 -- 172.21.15.87:0/4262003563 >> 172.21.15.87:0/4262003563 conn(0x7f02b00f3800 msgr2=0x7f02b00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.580+0000 7f02b7a12700 1 -- 172.21.15.87:0/4262003563 shutdown_connections 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.580+0000 7f02b7a12700 1 -- 172.21.15.87:0/4262003563 wait complete. 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.581+0000 7f02b7a12700 1 librados: starting msgr at 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.581+0000 7f02b7a12700 1 librados: starting objecter 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.581+0000 7f02b7a12700 1 Processor -- start 2024-02-23T00:36:03.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.581+0000 7f02b7a12700 1 -- start start 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.581+0000 7f02b7a12700 1 librados: setting wanted keys 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.581+0000 7f02b7a12700 1 librados: calling monclient init 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00f8420 0x7f02b010e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02b00fdbb0 0x7f02b010e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02b00fe510 0x7f02b01081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01086b0 con 0x7f02b00fe510 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0108820 con 0x7f02b00fdbb0 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01089e0 con 0x7f02b00f8420 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b57ae700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00f8420 0x7f02b010e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b5faf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02b00fe510 0x7f02b01081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b57ae700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00f8420 0x7f02b010e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:43208/0 (socket says 172.21.15.87:43208) 2024-02-23T00:36:03.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b57ae700 1 -- 172.21.15.87:0/751041035 learned_addr learned my addr 172.21.15.87:0/751041035 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b4fad700 1 --2- 172.21.15.87:0/751041035 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02b00fdbb0 0x7f02b010e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b5faf700 1 -- 172.21.15.87:0/751041035 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00f8420 msgr2=0x7f02b010e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b5faf700 1 --2- 172.21.15.87:0/751041035 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00f8420 0x7f02b010e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b5faf700 1 -- 172.21.15.87:0/751041035 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02b00fdbb0 msgr2=0x7f02b010e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b5faf700 1 --2- 172.21.15.87:0/751041035 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02b00fdbb0 0x7f02b010e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b5faf700 1 -- 172.21.15.87:0/751041035 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a80097c0 con 0x7f02b00fe510 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b57ae700 1 --2- 172.21.15.87:0/751041035 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00f8420 0x7f02b010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b5faf700 1 --2- 172.21.15.87:0/751041035 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02b00fe510 0x7f02b01081c0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7f02a8001df0 tx=0x7f02a8004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02a67fc700 10 client.?.objecter ms_handle_connect 0x7f02b00fe510 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02a67fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02a67fc700 1 -- 172.21.15.87:0/751041035 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80099f0 con 0x7f02b00fe510 2024-02-23T00:36:03.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac005040 con 0x7f02b00fe510 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02a67fc700 1 -- 172.21.15.87:0/751041035 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02a80095a0 con 0x7f02b00fe510 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02a67fc700 1 -- 172.21.15.87:0/751041035 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8007d20 con 0x7f02b00fe510 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 10 client.27195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b0114e50 con 0x7f02b00fe510 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.582+0000 7f02b7a12700 1 librados: init done 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.583+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b005ee80 con 0x7f02b00fe510 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02a67fc700 1 -- 172.21.15.87:0/751041035 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f02a800fb80 con 0x7f02b00fe510 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02a67fc700 1 --2- 172.21.15.87:0/751041035 >> v2:172.21.15.87:6800/1231597463 conn(0x7f029c062c90 0x7f029c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02a67fc700 1 -- 172.21.15.87:0/751041035 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(408..408 src has 173..408) v4 ==== 8122+0+0 (secure 0 0 0) 0x7f02a807da80 con 0x7f02b00fe510 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02a67fc700 10 client.27195.objecter ms_dispatch 0x7f02b0001550 osd_map(408..408 src has 173..408) v4 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02a67fc700 3 client.27195.objecter handle_osd_map got epochs [408,408] > 0 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02a67fc700 3 client.27195.objecter handle_osd_map decoding full epoch 408 2024-02-23T00:36:03.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02b57ae700 1 --2- 172.21.15.87:0/751041035 >> v2:172.21.15.87:6800/1231597463 conn(0x7f029c062c90 0x7f029c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02a67fc700 20 client.27195.objecter dump_active .. 0 homeless 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02b57ae700 1 --2- 172.21.15.87:0/751041035 >> v2:172.21.15.87:6800/1231597463 conn(0x7f029c062c90 0x7f029c065150 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 rx=0x7f02a00041d0 tx=0x7f02a0009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.584+0000 7f02a67fc700 10 client.27195.objecter ms_handle_connect 0x7f029c062c90 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.588+0000 7f02a67fc700 1 -- 172.21.15.87:0/751041035 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f02a8050f50 con 0x7f02b00fe510 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:02.766+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 0.9473964383286395} v 0) v1 -- 0x7f02b0109770 con 0x7f02b00fe510 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.090+0000 7f02a67fc700 1 -- 172.21.15.87:0/751041035 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9473964383286395}]=0 set osd.4 primary-affinity to 0.947396 (8620882) v409) v1 ==== 155+0+0 (secure 0 0 0) 0x7f02a80191b0 con 0x7f02b00fe510 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 10 librados: watch_flush enter 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 10 librados: watch_flush exit 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 20 client.27195.objecter shutdown clearing up homeless session... 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 10 client.27195.objecter successfully canceled tick 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 >> v2:172.21.15.87:6800/1231597463 conn(0x7f029c062c90 msgr2=0x7f029c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:03.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 --2- 172.21.15.87:0/751041035 >> v2:172.21.15.87:6800/1231597463 conn(0x7f029c062c90 0x7f029c065150 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 rx=0x7f02a00041d0 tx=0x7f02a0009040).stop 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02b00fe510 msgr2=0x7f02b01081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 --2- 172.21.15.87:0/751041035 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02b00fe510 0x7f02b01081c0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 rx=0x7f02a8001df0 tx=0x7f02a8004d10).stop 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 shutdown_connections 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 --2- 172.21.15.87:0/751041035 >> v2:172.21.15.87:6800/1231597463 conn(0x7f029c062c90 0x7f029c065150 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 --2- 172.21.15.87:0/751041035 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f02b00fe510 0x7f02b01081c0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 --2- 172.21.15.87:0/751041035 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f02b00f8420 0x7f02b010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 --2- 172.21.15.87:0/751041035 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f02b00fdbb0 0x7f02b010e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 >> 172.21.15.87:0/751041035 conn(0x7f02b00f3800 msgr2=0x7f02b00fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 shutdown_connections 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 -- 172.21.15.87:0/751041035 wait complete. 2024-02-23T00:36:03.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:03.093+0000 7f02b7a12700 1 librados: shutdown 2024-02-23T00:36:03.294 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.4 primary-affinity to 0.947396 (8620882) 2024-02-23T00:36:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:01.648095+0000 mgr.y (mgr.24437) 373 : cluster [DBG] pgmap v595: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-23T00:36:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:02.086508+0000 mon.a (mon.0) 2552 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:36:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:02.086654+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2024-02-23T00:36:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:02.097117+0000 mon.b (mon.1) 353 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:36:03.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:02.101166+0000 mon.b (mon.1) 354 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:03.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:02.101870+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:03.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:02.767325+0000 mon.a (mon.0) 2555 : audit [INF] from='client.? 172.21.15.87:0/751041035' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9473964383286395}]: dispatch 2024-02-23T00:36:03.495 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:03.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:36:03.599 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:03.631 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:03.652 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:03.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:03.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:03.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:03.785 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:03.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:36:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:36:04.033 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:04.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:04.136 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:04.166 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:04.185 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:04.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:04.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:04.289 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:36:04.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:03.089682+0000 mon.a (mon.0) 2556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:36:04.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:03.089766+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? 172.21.15.87:0/751041035' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9473964383286395}]': finished 2024-02-23T00:36:04.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:03.089811+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2024-02-23T00:36:04.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:04.090725+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2024-02-23T00:36:04.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:03.089682+0000 mon.a (mon.0) 2556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:36:04.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:03.089766+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? 172.21.15.87:0/751041035' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9473964383286395}]': finished 2024-02-23T00:36:04.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:03.089811+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2024-02-23T00:36:04.315 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:04.090725+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2024-02-23T00:36:04.316 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:03.089682+0000 mon.a (mon.0) 2556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:36:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:03.089766+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? 172.21.15.87:0/751041035' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9473964383286395}]': finished 2024-02-23T00:36:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:03.089811+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2024-02-23T00:36:04.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:04.090725+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2024-02-23T00:36:04.525 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:04.564 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:04.565 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:04.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:36:04.669 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:04.695 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:04.792 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:04.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:04.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:04.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:04.926 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:05.024 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:05.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:05.128 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:05.158 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:05.178 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:05.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:05.282 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:05.310 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:05.329 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:05.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:05.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:03.648564+0000 mgr.y (mgr.24437) 374 : cluster [DBG] pgmap v598: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-23T00:36:05.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:03.648564+0000 mgr.y (mgr.24437) 374 : cluster [DBG] pgmap v598: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-23T00:36:05.435 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:03.648564+0000 mgr.y (mgr.24437) 374 : cluster [DBG] pgmap v598: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-23T00:36:05.464 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:05.814 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:05.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:05.918 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:05.950 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:05.971 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:05.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:06.077 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:06.109 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:06.134 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:06.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:06.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:06.240 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:06.267 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:05.110061+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2024-02-23T00:36:06.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:05.110061+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2024-02-23T00:36:06.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:05.110061+0000 mon.a (mon.0) 2560 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2024-02-23T00:36:06.615 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:06.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:06.719 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:06.751 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:06.772 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:06.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.785+0000 7efc055a5700 1 -- 172.21.15.87:0/579361690 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc000fcbb0 msgr2=0x7efc000fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.785+0000 7efc055a5700 1 --2- 172.21.15.87:0/579361690 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc000fcbb0 0x7efc000fcf90 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 rx=0x7efbf4004870 tx=0x7efbf4009cb0).stop 2024-02-23T00:36:06.787 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.785+0000 7efc055a5700 1 -- 172.21.15.87:0/579361690 shutdown_connections 2024-02-23T00:36:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.785+0000 7efc055a5700 1 --2- 172.21.15.87:0/579361690 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc000f7400 0x7efc00103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.785+0000 7efc055a5700 1 --2- 172.21.15.87:0/579361690 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc000fcbb0 0x7efc000fcf90 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.785+0000 7efc055a5700 1 --2- 172.21.15.87:0/579361690 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc000fd510 0x7efc000f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.785+0000 7efc055a5700 1 -- 172.21.15.87:0/579361690 >> 172.21.15.87:0/579361690 conn(0x7efc000f2820 msgr2=0x7efc000f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.785+0000 7efc055a5700 1 -- 172.21.15.87:0/579361690 shutdown_connections 2024-02-23T00:36:06.788 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.785+0000 7efc055a5700 1 -- 172.21.15.87:0/579361690 wait complete. 2024-02-23T00:36:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 librados: starting msgr at 2024-02-23T00:36:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 librados: starting objecter 2024-02-23T00:36:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 Processor -- start 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 -- start start 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 librados: setting wanted keys 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 librados: calling monclient init 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc000f7400 0x7efc000f8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc000fcbb0 0x7efc000f9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc000fd510 0x7efc000f9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc001166d0 con 0x7efc000fcbb0 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc00116840 con 0x7efc000f7400 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efc055a5700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc00116a00 con 0x7efc000fd510 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efbfe7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc000fcbb0 0x7efc000f9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efbfe7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc000fcbb0 0x7efc000f9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41006/0 (socket says 172.21.15.87:41006) 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efbfe7fc700 1 -- 172.21.15.87:0/1256287098 learned_addr learned my addr 172.21.15.87:0/1256287098 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:06.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efbff7fe700 1 --2- 172.21.15.87:0/1256287098 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc000fd510 0x7efc000f9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efbfeffd700 1 --2- 172.21.15.87:0/1256287098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc000f7400 0x7efc000f8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efbfe7fc700 1 -- 172.21.15.87:0/1256287098 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc000fd510 msgr2=0x7efc000f9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.786+0000 7efbfe7fc700 1 --2- 172.21.15.87:0/1256287098 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc000fd510 0x7efc000f9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbfe7fc700 1 -- 172.21.15.87:0/1256287098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc000f7400 msgr2=0x7efc000f8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbfe7fc700 1 --2- 172.21.15.87:0/1256287098 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc000f7400 0x7efc000f8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbfe7fc700 1 -- 172.21.15.87:0/1256287098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0005040 con 0x7efc000fcbb0 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbff7fe700 1 --2- 172.21.15.87:0/1256287098 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc000fd510 0x7efc000f9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbfe7fc700 1 --2- 172.21.15.87:0/1256287098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc000fcbb0 0x7efc000f9460 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7efbf00048f0 tx=0x7efbf0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbe7fff700 10 client.?.objecter ms_handle_connect 0x7efc000fcbb0 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbe7fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0007b00 con 0x7efc000fcbb0 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf4005040 con 0x7efc000fcbb0 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efbf000f040 con 0x7efc000fcbb0 2024-02-23T00:36:06.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0014990 con 0x7efc000fcbb0 2024-02-23T00:36:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efc055a5700 10 client.27201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc00117090 con 0x7efc000fcbb0 2024-02-23T00:36:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.787+0000 7efc055a5700 1 librados: init done 2024-02-23T00:36:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.788+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7efc0005fa60 con 0x7efc000fcbb0 2024-02-23T00:36:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.788+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7efbf0010500 con 0x7efc000fcbb0 2024-02-23T00:36:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.788+0000 7efbe7fff700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6800/1231597463 conn(0x7efbe8062c90 0x7efbe8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(412..412 src has 173..412) v4 ==== 8122+0+0 (secure 0 0 0) 0x7efbf007cd40 con 0x7efc000fcbb0 2024-02-23T00:36:06.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 osd_map(412..412 src has 173..412) v4 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbe7fff700 3 client.27201.objecter handle_osd_map got epochs [412,412] > 0 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbe7fff700 3 client.27201.objecter handle_osd_map decoding full epoch 412 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbfeffd700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6800/1231597463 conn(0x7efbe8062c90 0x7efbe8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbe7fff700 20 client.27201.objecter dump_active .. 0 homeless 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v412) v1 ==== 54+0+16 (secure 0 0 0) 0x7efbf0082450 con 0x7efc000fcbb0 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbe7fff700 10 client.27201.objecter ms_handle_connect 0x7efbe8062c90 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbfeffd700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6800/1231597463 conn(0x7efbe8062c90 0x7efbe8065150 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 rx=0x7efbf40043d0 tx=0x7efbf400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 10 client.27201.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 1 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6801/2153191863 conn(0x7efc000ffed0 0x7efc001002f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0005ee80 osd=0 3 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0005ee80 osd=0 3 2024-02-23T00:36:06.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0005ee80 osd=0 3 2024-02-23T00:36:06.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 1 2024-02-23T00:36:06.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc0005ee80 osd=0 3 2024-02-23T00:36:06.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 0 1 2024-02-23T00:36:06.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0005ee80 osd=0 4 2024-02-23T00:36:06.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 10 client.27201.objecter _send_command 1 2024-02-23T00:36:06.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc001007e0 con 0x7efc000ffed0 2024-02-23T00:36:06.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbff7fe700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6801/2153191863 conn(0x7efc000ffed0 0x7efc001002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbe7fff700 10 client.27201.objecter ms_handle_connect 0x7efc000ffed0 2024-02-23T00:36:06.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.789+0000 7efbff7fe700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6801/2153191863 conn(0x7efc000ffed0 0x7efc001002f0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.790+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efc001007e0 con 0x7efc000ffed0 2024-02-23T00:36:06.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.790+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:36:06.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.790+0000 7efbe7fff700 10 client.27201.objecter _finish_command 1 = system:0 2024-02-23T00:36:06.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.790+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc0005ee80 osd=0 4 2024-02-23T00:36:06.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.790+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 0 1 2024-02-23T00:36:06.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 10 client.27201.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:06.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 2 2024-02-23T00:36:06.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0005ee80 osd=0 3 2024-02-23T00:36:06.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0005ee80 osd=0 3 2024-02-23T00:36:06.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0005ee80 osd=0 3 2024-02-23T00:36:06.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 2 2024-02-23T00:36:06.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc0005ee80 osd=0 3 2024-02-23T00:36:06.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 0 2 2024-02-23T00:36:06.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0005ee80 osd=0 4 2024-02-23T00:36:06.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 10 client.27201.objecter _send_command 2 2024-02-23T00:36:06.807 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.804+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc001009a0 con 0x7efc000ffed0 2024-02-23T00:36:06.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7efc001009a0 con 0x7efc000ffed0 2024-02-23T00:36:06.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:06.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efbe7fff700 10 client.27201.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc0005ee80 osd=0 4 2024-02-23T00:36:06.808 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 0 2 2024-02-23T00:36:06.809 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:36:06.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 10 client.27201.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:06.809 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 3 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6805/685354171 conn(0x7efc00028540 0x7efc0011b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc000f4750 osd=1 3 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc000f4750 osd=1 3 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc000f4750 osd=1 3 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 3 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc000f4750 osd=1 3 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 1 3 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc000f4750 osd=1 4 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 10 client.27201.objecter _send_command 3 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc000f48c0 con 0x7efc00028540 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efbfeffd700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6805/685354171 conn(0x7efc00028540 0x7efc0011b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efbe7fff700 10 client.27201.objecter ms_handle_connect 0x7efc00028540 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.806+0000 7efbfeffd700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6805/685354171 conn(0x7efc00028540 0x7efc0011b2a0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.810 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.807+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efc000f48c0 con 0x7efc00028540 2024-02-23T00:36:06.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.807+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:36:06.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.807+0000 7efbe7fff700 10 client.27201.objecter _finish_command 3 = system:0 2024-02-23T00:36:06.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.807+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc000f4750 osd=1 4 2024-02-23T00:36:06.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.807+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 1 3 2024-02-23T00:36:06.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 10 client.27201.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:06.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 4 2024-02-23T00:36:06.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc000f4750 osd=1 3 2024-02-23T00:36:06.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc000f4750 osd=1 3 2024-02-23T00:36:06.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc000f4750 osd=1 3 2024-02-23T00:36:06.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 4 2024-02-23T00:36:06.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc000f4750 osd=1 3 2024-02-23T00:36:06.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 1 4 2024-02-23T00:36:06.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc000f4750 osd=1 4 2024-02-23T00:36:06.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 10 client.27201.objecter _send_command 4 2024-02-23T00:36:06.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.821+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc0005a040 con 0x7efc00028540 2024-02-23T00:36:06.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7efc0005a040 con 0x7efc00028540 2024-02-23T00:36:06.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:06.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efbe7fff700 10 client.27201.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc000f4750 osd=1 4 2024-02-23T00:36:06.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 1 4 2024-02-23T00:36:06.825 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:36:06.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 10 client.27201.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:06.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 5 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6809/3605394228 conn(0x7efc000594d0 0x7efc0011e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00059360 osd=2 3 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00059360 osd=2 3 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00059360 osd=2 3 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 5 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc00059360 osd=2 3 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 2 5 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00059360 osd=2 4 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 10 client.27201.objecter _send_command 5 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc000280a0 con 0x7efc000594d0 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efbfe7fc700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6809/3605394228 conn(0x7efc000594d0 0x7efc0011e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efbe7fff700 10 client.27201.objecter ms_handle_connect 0x7efc000594d0 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.823+0000 7efbfe7fc700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6809/3605394228 conn(0x7efc000594d0 0x7efc0011e920 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.824+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efc000280a0 con 0x7efc000594d0 2024-02-23T00:36:06.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.824+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:36:06.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.824+0000 7efbe7fff700 10 client.27201.objecter _finish_command 5 = system:0 2024-02-23T00:36:06.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.824+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc00059360 osd=2 4 2024-02-23T00:36:06.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.824+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 2 5 2024-02-23T00:36:06.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 10 client.27201.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:06.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 6 2024-02-23T00:36:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00059360 osd=2 3 2024-02-23T00:36:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00059360 osd=2 3 2024-02-23T00:36:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00059360 osd=2 3 2024-02-23T00:36:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 6 2024-02-23T00:36:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc00059360 osd=2 3 2024-02-23T00:36:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 2 6 2024-02-23T00:36:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00059360 osd=2 4 2024-02-23T00:36:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 10 client.27201.objecter _send_command 6 2024-02-23T00:36:06.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.838+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc0011ee10 con 0x7efc000594d0 2024-02-23T00:36:06.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7efc0011ee10 con 0x7efc000594d0 2024-02-23T00:36:06.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:06.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efbe7fff700 10 client.27201.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc00059360 osd=2 4 2024-02-23T00:36:06.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 2 6 2024-02-23T00:36:06.842 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:36:06.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 10 client.27201.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:06.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 7 2024-02-23T00:36:06.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6813/128059790 conn(0x7efc00027770 0x7efc0011efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00027600 osd=3 3 2024-02-23T00:36:06.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00027600 osd=3 3 2024-02-23T00:36:06.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00027600 osd=3 3 2024-02-23T00:36:06.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 7 2024-02-23T00:36:06.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc00027600 osd=3 3 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 3 7 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00027600 osd=3 4 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 10 client.27201.objecter _send_command 7 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc0011c5e0 con 0x7efc00027770 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efbff7fe700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6813/128059790 conn(0x7efc00027770 0x7efc0011efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efbe7fff700 10 client.27201.objecter ms_handle_connect 0x7efc00027770 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.840+0000 7efbff7fe700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6813/128059790 conn(0x7efc00027770 0x7efc0011efe0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.841+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efc0011c5e0 con 0x7efc00027770 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.841+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.841+0000 7efbe7fff700 10 client.27201.objecter _finish_command 7 = system:0 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.841+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc00027600 osd=3 4 2024-02-23T00:36:06.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.841+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 3 7 2024-02-23T00:36:06.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 10 client.27201.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:06.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 8 2024-02-23T00:36:06.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00027600 osd=3 3 2024-02-23T00:36:06.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00027600 osd=3 3 2024-02-23T00:36:06.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00027600 osd=3 3 2024-02-23T00:36:06.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 8 2024-02-23T00:36:06.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc00027600 osd=3 3 2024-02-23T00:36:06.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 3 8 2024-02-23T00:36:06.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00027600 osd=3 4 2024-02-23T00:36:06.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 10 client.27201.objecter _send_command 8 2024-02-23T00:36:06.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.855+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc0011b9d0 con 0x7efc00027770 2024-02-23T00:36:06.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7efc0011b9d0 con 0x7efc00027770 2024-02-23T00:36:06.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:06.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efbe7fff700 10 client.27201.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc00027600 osd=3 4 2024-02-23T00:36:06.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 3 8 2024-02-23T00:36:06.859 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:36:06.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 10 client.27201.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:06.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 9 2024-02-23T00:36:06.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6800/4175844757 conn(0x7efc0011bb70 0x7efc00125810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0011b9d0 osd=4 3 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0011b9d0 osd=4 3 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0011b9d0 osd=4 3 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 9 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc0011b9d0 osd=4 3 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 4 9 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0011b9d0 osd=4 4 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 10 client.27201.objecter _send_command 9 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc001214b0 con 0x7efc0011bb70 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efbfeffd700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6800/4175844757 conn(0x7efc0011bb70 0x7efc00125810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efbfeffd700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6800/4175844757 conn(0x7efc0011bb70 0x7efc00125810 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.857+0000 7efbe7fff700 10 client.27201.objecter ms_handle_connect 0x7efc0011bb70 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.858+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efc001214b0 con 0x7efc0011bb70 2024-02-23T00:36:06.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.858+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:36:06.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.858+0000 7efbe7fff700 10 client.27201.objecter _finish_command 9 = system:0 2024-02-23T00:36:06.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.858+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc0011b9d0 osd=4 4 2024-02-23T00:36:06.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.858+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 4 9 2024-02-23T00:36:06.875 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:06.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 10 client.27201.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:06.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 10 2024-02-23T00:36:06.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0011b9d0 osd=4 3 2024-02-23T00:36:06.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0011b9d0 osd=4 3 2024-02-23T00:36:06.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0011b9d0 osd=4 3 2024-02-23T00:36:06.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 10 2024-02-23T00:36:06.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc0011b9d0 osd=4 3 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 4 10 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0011b9d0 osd=4 4 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 10 client.27201.objecter _send_command 10 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.873+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc001214b0 con 0x7efc0011bb70 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7efc001214b0 con 0x7efc0011bb70 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efbe7fff700 10 client.27201.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc0011b9d0 osd=4 4 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 4 10 2024-02-23T00:36:06.879 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:36:06.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 10 client.27201.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:06.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 11 2024-02-23T00:36:06.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6804/2550013795 conn(0x7efc00126210 0x7efc00128600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc001214b0 osd=5 3 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc001214b0 osd=5 3 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc001214b0 osd=5 3 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 11 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc001214b0 osd=5 3 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 5 11 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc001214b0 osd=5 4 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 10 client.27201.objecter _send_command 11 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc0012b3b0 con 0x7efc00126210 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.875+0000 7efbfe7fc700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6804/2550013795 conn(0x7efc00126210 0x7efc00128600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.876+0000 7efbe7fff700 10 client.27201.objecter ms_handle_connect 0x7efc00126210 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.876+0000 7efbfe7fc700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6804/2550013795 conn(0x7efc00126210 0x7efc00128600 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.876+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efc0012b3b0 con 0x7efc00126210 2024-02-23T00:36:06.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.876+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:36:06.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.876+0000 7efbe7fff700 10 client.27201.objecter _finish_command 11 = system:0 2024-02-23T00:36:06.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.876+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc001214b0 osd=5 4 2024-02-23T00:36:06.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.876+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 5 11 2024-02-23T00:36:06.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 10 client.27201.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 12 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc001214b0 osd=5 3 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc001214b0 osd=5 3 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc001214b0 osd=5 3 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 12 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc001214b0 osd=5 3 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 5 12 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc001214b0 osd=5 4 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 10 client.27201.objecter _send_command 12 2024-02-23T00:36:06.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.890+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc0012b3b0 con 0x7efc00126210 2024-02-23T00:36:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.892+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7efc0012b3b0 con 0x7efc00126210 2024-02-23T00:36:06.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.892+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.892+0000 7efbe7fff700 10 client.27201.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.892+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc001214b0 osd=5 4 2024-02-23T00:36:06.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.892+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 5 12 2024-02-23T00:36:06.895 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:36:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 10 client.27201.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 13 2024-02-23T00:36:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6808/2509408498 conn(0x7efc00129d70 0x7efc0011f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0012b3b0 osd=6 3 2024-02-23T00:36:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0012b3b0 osd=6 3 2024-02-23T00:36:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0012b3b0 osd=6 3 2024-02-23T00:36:06.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 13 2024-02-23T00:36:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc0012b3b0 osd=6 3 2024-02-23T00:36:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 6 13 2024-02-23T00:36:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0012b3b0 osd=6 4 2024-02-23T00:36:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 10 client.27201.objecter _send_command 13 2024-02-23T00:36:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc00129870 con 0x7efc00129d70 2024-02-23T00:36:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efbff7fe700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6808/2509408498 conn(0x7efc00129d70 0x7efc0011f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efbff7fe700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6808/2509408498 conn(0x7efc00129d70 0x7efc0011f4d0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.893+0000 7efbe7fff700 10 client.27201.objecter ms_handle_connect 0x7efc00129d70 2024-02-23T00:36:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.894+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efc00129870 con 0x7efc00129d70 2024-02-23T00:36:06.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.894+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:36:06.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.894+0000 7efbe7fff700 10 client.27201.objecter _finish_command 13 = system:0 2024-02-23T00:36:06.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.894+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc0012b3b0 osd=6 4 2024-02-23T00:36:06.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.894+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 6 13 2024-02-23T00:36:06.907 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:06.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 10 client.27201.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:06.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 14 2024-02-23T00:36:06.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0012b3b0 osd=6 3 2024-02-23T00:36:06.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0012b3b0 osd=6 3 2024-02-23T00:36:06.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc0012b3b0 osd=6 3 2024-02-23T00:36:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 14 2024-02-23T00:36:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc0012b3b0 osd=6 3 2024-02-23T00:36:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 6 14 2024-02-23T00:36:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0012b3b0 osd=6 4 2024-02-23T00:36:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 10 client.27201.objecter _send_command 14 2024-02-23T00:36:06.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.908+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc00129870 con 0x7efc00129d70 2024-02-23T00:36:06.912 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:36:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7efc00129870 con 0x7efc00129d70 2024-02-23T00:36:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efbe7fff700 10 client.27201.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc0012b3b0 osd=6 4 2024-02-23T00:36:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 6 14 2024-02-23T00:36:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 10 client.27201.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 15 2024-02-23T00:36:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6812/373787000 conn(0x7efc0011fed0 0x7efc00120330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:06.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00129870 osd=7 3 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00129870 osd=7 3 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00129870 osd=7 3 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 15 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc00129870 osd=7 3 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 7 15 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00129870 osd=7 4 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 10 client.27201.objecter _send_command 15 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.910+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc001208e0 con 0x7efc0011fed0 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.911+0000 7efbfeffd700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6812/373787000 conn(0x7efc0011fed0 0x7efc00120330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.911+0000 7efbe7fff700 10 client.27201.objecter ms_handle_connect 0x7efc0011fed0 2024-02-23T00:36:06.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.911+0000 7efbfeffd700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6812/373787000 conn(0x7efc0011fed0 0x7efc00120330 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.912+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7efc001208e0 con 0x7efc0011fed0 2024-02-23T00:36:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.912+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:36:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.912+0000 7efbe7fff700 10 client.27201.objecter _finish_command 15 = system:0 2024-02-23T00:36:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.912+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc00129870 osd=7 4 2024-02-23T00:36:06.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.912+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 7 15 2024-02-23T00:36:06.927 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:06.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 10 client.27201.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:06.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign -1 16 2024-02-23T00:36:06.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00129870 osd=7 3 2024-02-23T00:36:06.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00129870 osd=7 3 2024-02-23T00:36:06.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 20 client.27201.objecter _get_session s=0x7efc00129870 osd=7 3 2024-02-23T00:36:06.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 15 client.27201.objecter _session_command_op_remove -1 16 2024-02-23T00:36:06.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 20 client.27201.objecter get_session s=0x7efc00129870 osd=7 3 2024-02-23T00:36:06.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 15 client.27201.objecter _session_command_op_assign 7 16 2024-02-23T00:36:06.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00129870 osd=7 4 2024-02-23T00:36:06.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 10 client.27201.objecter _send_command 16 2024-02-23T00:36:06.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.926+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7efc001208e0 con 0x7efc0011fed0 2024-02-23T00:36:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:06.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:06.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efbe7fff700 1 -- 172.21.15.87:0/1256287098 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7efc001208e0 con 0x7efc0011fed0 2024-02-23T00:36:06.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efbe7fff700 10 client.27201.objecter ms_dispatch 0x7efc00001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:06.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efbe7fff700 10 client.27201.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efbe7fff700 20 client.27201.objecter put_session s=0x7efc00129870 osd=7 4 2024-02-23T00:36:06.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efbe7fff700 15 client.27201.objecter _session_command_op_remove 7 16 2024-02-23T00:36:06.932 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:36:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 librados: watch_flush enter 2024-02-23T00:36:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 librados: watch_flush exit 2024-02-23T00:36:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 client.27201.objecter close_session for osd.0 2024-02-23T00:36:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6801/2153191863 conn(0x7efc000ffed0 msgr2=0x7efc001002f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6801/2153191863 conn(0x7efc000ffed0 0x7efc001002f0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0005ee80 osd=0 1 2024-02-23T00:36:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 client.27201.objecter close_session for osd.1 2024-02-23T00:36:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6805/685354171 conn(0x7efc00028540 msgr2=0x7efc0011b2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6805/685354171 conn(0x7efc00028540 0x7efc0011b2a0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc000f4750 osd=1 1 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 client.27201.objecter close_session for osd.2 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6809/3605394228 conn(0x7efc000594d0 msgr2=0x7efc0011e920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6809/3605394228 conn(0x7efc000594d0 0x7efc0011e920 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00059360 osd=2 1 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 client.27201.objecter close_session for osd.3 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6813/128059790 conn(0x7efc00027770 msgr2=0x7efc0011efe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6813/128059790 conn(0x7efc00027770 0x7efc0011efe0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efbff7fe700 1 -- 172.21.15.87:0/1256287098 reap_dead start 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00027600 osd=3 1 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 client.27201.objecter close_session for osd.4 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6800/4175844757 conn(0x7efc0011bb70 msgr2=0x7efc00125810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6800/4175844757 conn(0x7efc0011bb70 0x7efc00125810 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0011b9d0 osd=4 1 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 client.27201.objecter close_session for osd.5 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6804/2550013795 conn(0x7efc00126210 msgr2=0x7efc00128600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6804/2550013795 conn(0x7efc00126210 0x7efc00128600 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc001214b0 osd=5 1 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 client.27201.objecter close_session for osd.6 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6808/2509408498 conn(0x7efc00129d70 msgr2=0x7efc0011f4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6808/2509408498 conn(0x7efc00129d70 0x7efc0011f4d0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc0012b3b0 osd=6 1 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 client.27201.objecter close_session for osd.7 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6812/373787000 conn(0x7efc0011fed0 msgr2=0x7efc00120330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.171:6812/373787000 conn(0x7efc0011fed0 0x7efc00120330 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 20 client.27201.objecter put_session s=0x7efc00129870 osd=7 1 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 20 client.27201.objecter shutdown clearing up homeless session... 2024-02-23T00:36:06.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 10 client.27201.objecter successfully canceled tick 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efbff7fe700 1 -- 172.21.15.87:0/1256287098 reap_dead start 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6800/1231597463 conn(0x7efbe8062c90 msgr2=0x7efbe8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6800/1231597463 conn(0x7efbe8062c90 0x7efbe8065150 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 rx=0x7efbf40043d0 tx=0x7efbf400b040).stop 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc000fcbb0 msgr2=0x7efc000f9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc000fcbb0 0x7efc000f9460 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 rx=0x7efbf00048f0 tx=0x7efbf0009cb0).stop 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 shutdown_connections 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc000fcbb0 0x7efc000f9460 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 --2- 172.21.15.87:0/1256287098 >> v2:172.21.15.87:6800/1231597463 conn(0x7efbe8062c90 0x7efbe8065150 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 >> 172.21.15.87:0/1256287098 conn(0x7efc000f2820 msgr2=0x7efc000fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 shutdown_connections 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 -- 172.21.15.87:0/1256287098 wait complete. 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:06.929+0000 7efc055a5700 1 librados: shutdown 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.937 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:36:06.938 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:36:07.031 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:07.060 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:05.649510+0000 mgr.y (mgr.24437) 375 : cluster [DBG] pgmap v601: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-02-23T00:36:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:06.122199+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2024-02-23T00:36:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:06.790088+0000 mon.a (mon.0) 2562 : audit [DBG] from='client.? 172.21.15.87:0/1256287098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:07.080041+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-02-23T00:36:07.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:05.649510+0000 mgr.y (mgr.24437) 375 : cluster [DBG] pgmap v601: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-02-23T00:36:07.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:06.122199+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2024-02-23T00:36:07.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:06.790088+0000 mon.a (mon.0) 2562 : audit [DBG] from='client.? 172.21.15.87:0/1256287098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:07.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:07.080041+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-02-23T00:36:07.409 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:07.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:05.649510+0000 mgr.y (mgr.24437) 375 : cluster [DBG] pgmap v601: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-02-23T00:36:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:06.122199+0000 mon.a (mon.0) 2561 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2024-02-23T00:36:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:06.790088+0000 mon.a (mon.0) 2562 : audit [DBG] from='client.? 172.21.15.87:0/1256287098' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:07.080041+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-02-23T00:36:07.513 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:07.545 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:07.565 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:07.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:07.669 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:07.698 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:08.776 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:08.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:08.882 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:08.914 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:08.936 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:08.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:09.023 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:36:09.024 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:36:09.024 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-23T00:36:09.024 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.851766 2024-02-23T00:36:09.024 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 5 0.8517662085097673 2024-02-23T00:36:09.039 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:09.105 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:09.182 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:09.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 -- 172.21.15.87:0/1883937301 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9f80f9610 msgr2=0x7fa9f8101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/1883937301 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9f80f9610 0x7fa9f8101d20 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 rx=0x7fa9ec004870 tx=0x7fa9ec009cb0).stop 2024-02-23T00:36:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 -- 172.21.15.87:0/1883937301 shutdown_connections 2024-02-23T00:36:09.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/1883937301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9f8102210 0x7fa9f8104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:09.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/1883937301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9f80f8cb0 0x7fa9f80f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:09.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/1883937301 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9f80f9610 0x7fa9f8101d20 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:09.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 -- 172.21.15.87:0/1883937301 >> 172.21.15.87:0/1883937301 conn(0x7fa9f80f2820 msgr2=0x7fa9f80f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:09.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 -- 172.21.15.87:0/1883937301 shutdown_connections 2024-02-23T00:36:09.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 -- 172.21.15.87:0/1883937301 wait complete. 2024-02-23T00:36:09.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:07.649875+0000 mgr.y (mgr.24437) 376 : cluster [DBG] pgmap v604: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-02-23T00:36:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 librados: starting msgr at 2024-02-23T00:36:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.189+0000 7fa9fd30a700 1 librados: starting objecter 2024-02-23T00:36:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 Processor -- start 2024-02-23T00:36:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 -- start start 2024-02-23T00:36:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 librados: setting wanted keys 2024-02-23T00:36:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 librados: calling monclient init 2024-02-23T00:36:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9f80f8cb0 0x7fa9f810e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9f80f9610 0x7fa9f810e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:09.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9f8102210 0x7fa9f8108350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f8108840 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f81089b0 con 0x7fa9f80f9610 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9fd30a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f8108b70 con 0x7fa9f8102210 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9f80f8cb0 0x7fa9f810e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9f80f8cb0 0x7fa9f810e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41032/0 (socket says 172.21.15.87:41032) 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f6ffd700 1 -- 172.21.15.87:0/3116146296 learned_addr learned my addr 172.21.15.87:0/3116146296 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f77fe700 1 --2- 172.21.15.87:0/3116146296 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9f8102210 0x7fa9f8108350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f67fc700 1 --2- 172.21.15.87:0/3116146296 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9f80f9610 0x7fa9f810e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f6ffd700 1 -- 172.21.15.87:0/3116146296 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9f8102210 msgr2=0x7fa9f8108350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f6ffd700 1 --2- 172.21.15.87:0/3116146296 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9f8102210 0x7fa9f8108350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f6ffd700 1 -- 172.21.15.87:0/3116146296 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9f80f9610 msgr2=0x7fa9f810e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:09.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f6ffd700 1 --2- 172.21.15.87:0/3116146296 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9f80f9610 0x7fa9f810e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.190+0000 7fa9f6ffd700 1 -- 172.21.15.87:0/3116146296 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ec005040 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9f6ffd700 1 --2- 172.21.15.87:0/3116146296 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9f80f8cb0 0x7fa9f810e1b0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 rx=0x7fa9e80041d0 tx=0x7fa9e8009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9dffff700 10 client.?.objecter ms_handle_connect 0x7fa9f80f8cb0 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9dffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9dffff700 1 -- 172.21.15.87:0/3116146296 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8007b40 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f8108e50 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9dffff700 1 -- 172.21.15.87:0/3116146296 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa9e800f040 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9dffff700 1 -- 172.21.15.87:0/3116146296 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8014730 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9fd30a700 10 client.27207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f8114fe0 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.191+0000 7fa9fd30a700 1 librados: init done 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.192+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f805fa60 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.192+0000 7fa9dffff700 1 -- 172.21.15.87:0/3116146296 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fa9e8014b60 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.193+0000 7fa9dffff700 1 --2- 172.21.15.87:0/3116146296 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa9e0062c90 0x7fa9e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.193+0000 7fa9dffff700 1 -- 172.21.15.87:0/3116146296 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(413..413 src has 173..413) v4 ==== 8178+0+0 (secure 0 0 0) 0x7fa9e807d690 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.193+0000 7fa9dffff700 10 client.27207.objecter ms_dispatch 0x7fa9f8001550 osd_map(413..413 src has 173..413) v4 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.193+0000 7fa9dffff700 3 client.27207.objecter handle_osd_map got epochs [413,413] > 0 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.193+0000 7fa9dffff700 3 client.27207.objecter handle_osd_map decoding full epoch 413 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.193+0000 7fa9f67fc700 1 --2- 172.21.15.87:0/3116146296 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa9e0062c90 0x7fa9e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.193+0000 7fa9dffff700 20 client.27207.objecter dump_active .. 0 homeless 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.193+0000 7fa9dffff700 10 client.27207.objecter ms_handle_connect 0x7fa9e0062c90 2024-02-23T00:36:09.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.193+0000 7fa9f67fc700 1 --2- 172.21.15.87:0/3116146296 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa9e0062c90 0x7fa9e0065150 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7fa9ec009a00 tx=0x7fa9ec00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:09.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.197+0000 7fa9dffff700 1 -- 172.21.15.87:0/3116146296 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa9e8050b60 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.286 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:09.316 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:07.649875+0000 mgr.y (mgr.24437) 376 : cluster [DBG] pgmap v604: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-02-23T00:36:09.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:07.649875+0000 mgr.y (mgr.24437) 376 : cluster [DBG] pgmap v604: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-02-23T00:36:09.335 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:09.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:09.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:09.378+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa9f805ee80 con 0x7fa9f80f8cb0 2024-02-23T00:36:09.439 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:36:09.466 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:09.814 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:09.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:36:09.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:36:09.945 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:10.043 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:10.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:36:10.147 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:10.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.146+0000 7fa9dffff700 1 -- 172.21.15.87:0/3116146296 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v414) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa9e8050750 con 0x7fa9f80f8cb0 2024-02-23T00:36:10.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.148+0000 7fa9fd30a700 10 librados: watch_flush enter 2024-02-23T00:36:10.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 10 librados: watch_flush exit 2024-02-23T00:36:10.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 20 client.27207.objecter shutdown clearing up homeless session... 2024-02-23T00:36:10.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 10 client.27207.objecter successfully canceled tick 2024-02-23T00:36:10.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa9e0062c90 msgr2=0x7fa9e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:10.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/3116146296 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa9e0062c90 0x7fa9e0065150 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 rx=0x7fa9ec009a00 tx=0x7fa9ec00b040).stop 2024-02-23T00:36:10.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9f80f8cb0 msgr2=0x7fa9f810e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:10.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/3116146296 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9f80f8cb0 0x7fa9f810e1b0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 rx=0x7fa9e80041d0 tx=0x7fa9e8009f60).stop 2024-02-23T00:36:10.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 shutdown_connections 2024-02-23T00:36:10.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/3116146296 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa9e0062c90 0x7fa9e0065150 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:10.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/3116146296 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa9f8102210 0x7fa9f8108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:10.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/3116146296 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa9f80f8cb0 0x7fa9f810e1b0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:10.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 --2- 172.21.15.87:0/3116146296 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa9f80f9610 0x7fa9f810e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:10.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 >> 172.21.15.87:0/3116146296 conn(0x7fa9f80f2820 msgr2=0x7fa9f80f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:10.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 shutdown_connections 2024-02-23T00:36:10.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 -- 172.21.15.87:0/3116146296 wait complete. 2024-02-23T00:36:10.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:10.149+0000 7fa9fd30a700 1 librados: shutdown 2024-02-23T00:36:10.155 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:36:10.175 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:10.194 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:10.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:10.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:10.328 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:10.366 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:36:10.366 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:36:10.366 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:36:10.366 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:36:10.391 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:09.380514+0000 mon.a (mon.0) 2564 : audit [INF] from='client.? 172.21.15.87:0/3116146296' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:10.391 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:09.649980+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:10.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:09.380514+0000 mon.a (mon.0) 2564 : audit [INF] from='client.? 172.21.15.87:0/3116146296' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:10.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:09.649980+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:09.380514+0000 mon.a (mon.0) 2564 : audit [INF] from='client.? 172.21.15.87:0/3116146296' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:09.649980+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:11.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:36:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:36:11.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:09.650221+0000 mgr.y (mgr.24437) 377 : cluster [DBG] pgmap v605: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-02-23T00:36:11.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:10.123917+0000 mon.a (mon.0) 2566 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:11.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:10.146878+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? 172.21.15.87:0/3116146296' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:36:11.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:10.146933+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-02-23T00:36:11.348 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-23T00:36:11.386 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:11.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:09.650221+0000 mgr.y (mgr.24437) 377 : cluster [DBG] pgmap v605: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-02-23T00:36:11.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:10.123917+0000 mon.a (mon.0) 2566 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:11.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:10.146878+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? 172.21.15.87:0/3116146296' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:36:11.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:10.146933+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-02-23T00:36:11.389 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:09.650221+0000 mgr.y (mgr.24437) 377 : cluster [DBG] pgmap v605: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-02-23T00:36:11.389 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:10.123917+0000 mon.a (mon.0) 2566 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:11.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:10.146878+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? 172.21.15.87:0/3116146296' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:36:11.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:10.146933+0000 mon.a (mon.0) 2568 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-02-23T00:36:11.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:11.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:11.391 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:11.492 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:11.519 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:11.822 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:36:11.958 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:12.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:10.893914+0000 mgr.y (mgr.24437) 378 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:12.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:12.080591+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-02-23T00:36:12.456 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:36:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:12] "GET /metrics HTTP/1.1" 200 188111 "" "Prometheus/2.33.4" 2024-02-23T00:36:12.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:10.893914+0000 mgr.y (mgr.24437) 378 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:12.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:12.080591+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-02-23T00:36:12.457 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:10.893914+0000 mgr.y (mgr.24437) 378 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:12.457 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:12.080591+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-02-23T00:36:12.468 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:12.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:12.572 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:12.604 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:12.626 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:12.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:12.729 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:12.764 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:12.785 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:12.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:12.889 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:12.916 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:12.935 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:12.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:13.039 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:13.066 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:13.085 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:13.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:13.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:13.221 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:11.215803+0000 osd.4 (osd.4) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:11.216020+0000 osd.4 (osd.4) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:11.651404+0000 mgr.y (mgr.24437) 379 : cluster [DBG] pgmap v607: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 341 B/s wr, 2 op/s 2024-02-23T00:36:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:11.215803+0000 osd.4 (osd.4) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:11.216020+0000 osd.4 (osd.4) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:11.651404+0000 mgr.y (mgr.24437) 379 : cluster [DBG] pgmap v607: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 341 B/s wr, 2 op/s 2024-02-23T00:36:13.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:11.215803+0000 osd.4 (osd.4) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:13.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:11.216020+0000 osd.4 (osd.4) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:13.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:11.651404+0000 mgr.y (mgr.24437) 379 : cluster [DBG] pgmap v607: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 341 B/s wr, 2 op/s 2024-02-23T00:36:13.896 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:36:14.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:36:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:36:14.201 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:14.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:14.305 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:14.337 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:14.358 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:14.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:14.462 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:14.492 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:14.657 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:15.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:13.651786+0000 mgr.y (mgr.24437) 380 : cluster [DBG] pgmap v609: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 311 B/s wr, 1 op/s 2024-02-23T00:36:15.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:14.965028+0000 mon.a (mon.0) 2570 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:15.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:13.651786+0000 mgr.y (mgr.24437) 380 : cluster [DBG] pgmap v609: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 311 B/s wr, 1 op/s 2024-02-23T00:36:15.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:14.965028+0000 mon.a (mon.0) 2570 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:15.228 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:36:15.228 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:36:15.228 INFO:teuthology.orchestra.run.smithi087.stdout: "complaint_time": 30, 2024-02-23T00:36:15.229 INFO:teuthology.orchestra.run.smithi087.stdout: "num_blocked_ops": 0 2024-02-23T00:36:15.229 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:36:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:15.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:15.360 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:15.388 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:15.426 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:15.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:13.651786+0000 mgr.y (mgr.24437) 380 : cluster [DBG] pgmap v609: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 311 B/s wr, 1 op/s 2024-02-23T00:36:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:14.965028+0000 mon.a (mon.0) 2570 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:15.531 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:15.560 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:15.811 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:15.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:15.916 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:15.943 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 -- 172.21.15.87:0/3587474815 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3ec0fe510 msgr2=0x7ff3ec0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 --2- 172.21.15.87:0/3587474815 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3ec0fe510 0x7ff3ec0f7d40 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7ff3e00047b0 tx=0x7ff3e0009cb0).stop 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 -- 172.21.15.87:0/3587474815 shutdown_connections 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 --2- 172.21.15.87:0/3587474815 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3ec0f8420 0x7ff3ec104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 --2- 172.21.15.87:0/3587474815 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3ec0fdbb0 0x7ff3ec0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 --2- 172.21.15.87:0/3587474815 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3ec0fe510 0x7ff3ec0f7d40 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 -- 172.21.15.87:0/3587474815 >> 172.21.15.87:0/3587474815 conn(0x7ff3ec0f3800 msgr2=0x7ff3ec0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 -- 172.21.15.87:0/3587474815 shutdown_connections 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 -- 172.21.15.87:0/3587474815 wait complete. 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 librados: starting msgr at 2024-02-23T00:36:15.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 librados: starting objecter 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 Processor -- start 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 -- start start 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 librados: setting wanted keys 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.920+0000 7ff3f2295700 1 librados: calling monclient init 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f2295700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3ec0f8420 0x7ff3ec06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f2295700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3ec0fdbb0 0x7ff3ec06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f2295700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3ec0fe510 0x7ff3ec06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f2295700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec1164c0 con 0x7ff3ec0fdbb0 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f2295700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec116630 con 0x7ff3ec0f8420 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f2295700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec1167f0 con 0x7ff3ec0fe510 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3eb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3ec0fdbb0 0x7ff3ec06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f0832700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3ec0fe510 0x7ff3ec06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3eb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3ec0fdbb0 0x7ff3ec06b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:33392/0 (socket says 172.21.15.87:33392) 2024-02-23T00:36:15.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3eb7fe700 1 -- 172.21.15.87:0/783739344 learned_addr learned my addr 172.21.15.87:0/783739344 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3ebfff700 1 --2- 172.21.15.87:0/783739344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3ec0f8420 0x7ff3ec06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f0832700 1 -- 172.21.15.87:0/783739344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3ec0f8420 msgr2=0x7ff3ec06b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f0832700 1 --2- 172.21.15.87:0/783739344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3ec0f8420 0x7ff3ec06b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f0832700 1 -- 172.21.15.87:0/783739344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3ec0fdbb0 msgr2=0x7ff3ec06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f0832700 1 --2- 172.21.15.87:0/783739344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3ec0fdbb0 0x7ff3ec06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f0832700 1 -- 172.21.15.87:0/783739344 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e40097c0 con 0x7ff3ec0fe510 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f0832700 1 --2- 172.21.15.87:0/783739344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3ec0fe510 0x7ff3ec06bab0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 rx=0x7ff3e4001df0 tx=0x7ff3e4004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3e9775700 10 client.?.objecter ms_handle_connect 0x7ff3ec0fe510 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3e9775700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3e9775700 1 -- 172.21.15.87:0/783739344 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e40099f0 con 0x7ff3ec0fe510 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3e9775700 1 -- 172.21.15.87:0/783739344 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3e40090d0 con 0x7ff3ec0fe510 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e0005040 con 0x7ff3ec0fe510 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.921+0000 7ff3e9775700 1 -- 172.21.15.87:0/783739344 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e4007c60 con 0x7ff3ec0fe510 2024-02-23T00:36:15.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.922+0000 7ff3f2295700 10 client.37673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.922+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec116de0 con 0x7ff3ec0fe510 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.922+0000 7ff3f2295700 1 librados: init done 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.922+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec107110 con 0x7ff3ec0fe510 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.923+0000 7ff3e9775700 1 -- 172.21.15.87:0/783739344 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff3e400fad0 con 0x7ff3ec0fe510 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.923+0000 7ff3e9775700 1 --2- 172.21.15.87:0/783739344 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3d4062c90 0x7ff3d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.923+0000 7ff3e9775700 1 -- 172.21.15.87:0/783739344 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(415..415 src has 173..415) v4 ==== 8150+0+0 (secure 0 0 0) 0x7ff3e407da70 con 0x7ff3ec0fe510 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.923+0000 7ff3e9775700 10 client.37673.objecter ms_dispatch 0x7ff3ec001550 osd_map(415..415 src has 173..415) v4 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.923+0000 7ff3e9775700 3 client.37673.objecter handle_osd_map got epochs [415,415] > 0 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.923+0000 7ff3e9775700 3 client.37673.objecter handle_osd_map decoding full epoch 415 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.923+0000 7ff3ebfff700 1 --2- 172.21.15.87:0/783739344 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3d4062c90 0x7ff3d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.923+0000 7ff3e9775700 20 client.37673.objecter dump_active .. 0 homeless 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.924+0000 7ff3e9775700 10 client.37673.objecter ms_handle_connect 0x7ff3d4062c90 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.924+0000 7ff3ebfff700 1 --2- 172.21.15.87:0/783739344 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3d4062c90 0x7ff3d4065150 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7ff3dc004e60 tx=0x7ff3dc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:15.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:15.928+0000 7ff3e9775700 1 -- 172.21.15.87:0/783739344 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff3e4050f20 con 0x7ff3ec0fe510 2024-02-23T00:36:15.959 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:16.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.111+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673} v 0) v1 -- 0x7ff3ec05fa60 con 0x7ff3ec0fe510 2024-02-23T00:36:16.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.186+0000 7ff3e9775700 1 -- 172.21.15.87:0/783739344 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]=0 set osd.5 primary-affinity to 0.851766 (8558212) v416) v1 ==== 155+0+0 (secure 0 0 0) 0x7ff3e4050b10 con 0x7ff3ec0fe510 2024-02-23T00:36:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.188+0000 7ff3f2295700 10 librados: watch_flush enter 2024-02-23T00:36:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.188+0000 7ff3f2295700 10 librados: watch_flush exit 2024-02-23T00:36:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.188+0000 7ff3f2295700 20 client.37673.objecter shutdown clearing up homeless session... 2024-02-23T00:36:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.188+0000 7ff3f2295700 10 client.37673.objecter successfully canceled tick 2024-02-23T00:36:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3d4062c90 msgr2=0x7ff3d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 --2- 172.21.15.87:0/783739344 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3d4062c90 0x7ff3d4065150 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 rx=0x7ff3dc004e60 tx=0x7ff3dc00b040).stop 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3ec0fe510 msgr2=0x7ff3ec06bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 --2- 172.21.15.87:0/783739344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3ec0fe510 0x7ff3ec06bab0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 rx=0x7ff3e4001df0 tx=0x7ff3e4004d10).stop 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 shutdown_connections 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 --2- 172.21.15.87:0/783739344 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff3d4062c90 0x7ff3d4065150 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 --2- 172.21.15.87:0/783739344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3ec0fe510 0x7ff3ec06bab0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 --2- 172.21.15.87:0/783739344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3ec0f8420 0x7ff3ec06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 --2- 172.21.15.87:0/783739344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3ec0fdbb0 0x7ff3ec06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 >> 172.21.15.87:0/783739344 conn(0x7ff3ec0f3800 msgr2=0x7ff3ec0f5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 shutdown_connections 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 -- 172.21.15.87:0/783739344 wait complete. 2024-02-23T00:36:16.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:16.189+0000 7ff3f2295700 1 librados: shutdown 2024-02-23T00:36:16.193 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.5 primary-affinity to 0.851766 (8558212) 2024-02-23T00:36:16.194 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:16.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:16.298 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:16.328 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:16.349 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:16.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:16.113302+0000 mon.c (mon.2) 430 : audit [INF] from='client.? 172.21.15.87:0/783739344' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]: dispatch 2024-02-23T00:36:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:16.113673+0000 mon.a (mon.0) 2571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]: dispatch 2024-02-23T00:36:16.453 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:16.482 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:16.502 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:16.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:16.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:16.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:16.113302+0000 mon.c (mon.2) 430 : audit [INF] from='client.? 172.21.15.87:0/783739344' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]: dispatch 2024-02-23T00:36:16.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:16.113673+0000 mon.a (mon.0) 2571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]: dispatch 2024-02-23T00:36:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:16.113302+0000 mon.c (mon.2) 430 : audit [INF] from='client.? 172.21.15.87:0/783739344' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]: dispatch 2024-02-23T00:36:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:16.113673+0000 mon.a (mon.0) 2571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]: dispatch 2024-02-23T00:36:16.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:16.635 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:16.708 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:16.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:16.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:16.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-23T00:36:16.815 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:16.845 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:16.865 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:16.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:16.968 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:36:16.997 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:17.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:15.652668+0000 mgr.y (mgr.24437) 381 : cluster [DBG] pgmap v610: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-23T00:36:17.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:16.183732+0000 mon.a (mon.0) 2572 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]': finished 2024-02-23T00:36:17.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:16.183808+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-02-23T00:36:17.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:16.490268+0000 osd.7 (osd.7) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:17.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:16.490400+0000 osd.7 (osd.7) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:17.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:17.083250+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-02-23T00:36:17.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:15.652668+0000 mgr.y (mgr.24437) 381 : cluster [DBG] pgmap v610: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-23T00:36:17.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:16.183732+0000 mon.a (mon.0) 2572 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]': finished 2024-02-23T00:36:17.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:16.183808+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-02-23T00:36:17.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:16.490268+0000 osd.7 (osd.7) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:17.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:16.490400+0000 osd.7 (osd.7) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:17.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:17.083250+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-02-23T00:36:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:15.652668+0000 mgr.y (mgr.24437) 381 : cluster [DBG] pgmap v610: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-23T00:36:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:16.183732+0000 mon.a (mon.0) 2572 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.8517662085097673}]': finished 2024-02-23T00:36:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:16.183808+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-02-23T00:36:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:16.490268+0000 osd.7 (osd.7) 195 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:16.490400+0000 osd.7 (osd.7) 196 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:17.083250+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-02-23T00:36:17.946 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:17.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:36:18.050 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:18.079 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:18.630 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:18.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:18.735 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:18.767 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:18.789 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:18.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:18.893 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:18.920 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:19.119 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:19.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:19.224 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:19.249 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:17.653040+0000 mgr.y (mgr.24437) 382 : cluster [DBG] pgmap v613: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-02-23T00:36:19.249 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:18.098505+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-02-23T00:36:19.249 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:17.653040+0000 mgr.y (mgr.24437) 382 : cluster [DBG] pgmap v613: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-02-23T00:36:19.250 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:18.098505+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-02-23T00:36:19.251 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:17.653040+0000 mgr.y (mgr.24437) 382 : cluster [DBG] pgmap v613: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-02-23T00:36:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:18.098505+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-02-23T00:36:19.449 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:19.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:19.554 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:19.586 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:19.608 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:19.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:19.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:19.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.642+0000 7f8c3d61c700 1 -- 172.21.15.87:0/3219541499 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c380fd420 msgr2=0x7f8c380f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.642+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/3219541499 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c380fd420 0x7f8c380f6c50 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 rx=0x7f8c2c0047b0 tx=0x7f8c2c009cb0).stop 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.642+0000 7f8c3d61c700 1 -- 172.21.15.87:0/3219541499 shutdown_connections 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.642+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/3219541499 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c380f7330 0x7f8c38103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.642+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/3219541499 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c380fcac0 0x7f8c380fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/3219541499 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c380fd420 0x7f8c380f6c50 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 -- 172.21.15.87:0/3219541499 >> 172.21.15.87:0/3219541499 conn(0x7f8c380f26f0 msgr2=0x7f8c380f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 -- 172.21.15.87:0/3219541499 shutdown_connections 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 -- 172.21.15.87:0/3219541499 wait complete. 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 librados: starting msgr at 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 librados: starting objecter 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 Processor -- start 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 -- start start 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 librados: setting wanted keys 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 librados: calling monclient init 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.643+0000 7f8c3d61c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c380f7330 0x7f8c380f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c3d61c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c380fcac0 0x7f8c380f8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c3d61c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c380fd420 0x7f8c380f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c3d61c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c381164f0 con 0x7f8c380f7330 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c3d61c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c38116660 con 0x7f8c380fcac0 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c3d61c700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c38116820 con 0x7f8c380fd420 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c377fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c380fd420 0x7f8c380f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c377fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c380fd420 0x7f8c380f92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:45162/0 (socket says 172.21.15.87:45162) 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c377fe700 1 -- 172.21.15.87:0/774203725 learned_addr learned my addr 172.21.15.87:0/774203725 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c36ffd700 1 --2- 172.21.15.87:0/774203725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c380f7330 0x7f8c380f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c367fc700 1 --2- 172.21.15.87:0/774203725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c380fcac0 0x7f8c380f8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c377fe700 1 -- 172.21.15.87:0/774203725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c380fcac0 msgr2=0x7f8c380f8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c377fe700 1 --2- 172.21.15.87:0/774203725 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8c380fcac0 0x7f8c380f8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c377fe700 1 -- 172.21.15.87:0/774203725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c380f7330 msgr2=0x7f8c380f88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c377fe700 1 --2- 172.21.15.87:0/774203725 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8c380f7330 0x7f8c380f88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c377fe700 1 -- 172.21.15.87:0/774203725 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c300097c0 con 0x7f8c380fd420 2024-02-23T00:36:19.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c377fe700 1 --2- 172.21.15.87:0/774203725 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c380fd420 0x7f8c380f92a0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7f8c30001df0 tx=0x7f8c30004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c1ffff700 10 client.?.objecter ms_handle_connect 0x7f8c380fd420 2024-02-23T00:36:19.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c1ffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:19.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30009a10 con 0x7f8c380fd420 2024-02-23T00:36:19.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c2c005040 con 0x7f8c380fd420 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c300095a0 con 0x7f8c380fd420 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30007e30 con 0x7f8c380fd420 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c3d61c700 10 client.37679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c38116ee0 con 0x7f8c380fd420 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.644+0000 7f8c3d61c700 1 librados: init done 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.645+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8c3805f990 con 0x7f8c380fd420 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f8c300090d0 con 0x7f8c380fd420 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c1ffff700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c20062c90 0x7f8c20065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(418..418 src has 173..418) v4 ==== 8178+0+0 (secure 0 0 0) 0x7f8c3007de60 con 0x7f8c380fd420 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 osd_map(418..418 src has 173..418) v4 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c1ffff700 3 client.37679.objecter handle_osd_map got epochs [418,418] > 0 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c1ffff700 3 client.37679.objecter handle_osd_map decoding full epoch 418 2024-02-23T00:36:19.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c36ffd700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c20062c90 0x7f8c20065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c1ffff700 20 client.37679.objecter dump_active .. 0 homeless 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v418) v1 ==== 54+0+16 (secure 0 0 0) 0x7f8c3007e320 con 0x7f8c380fd420 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c1ffff700 10 client.37679.objecter ms_handle_connect 0x7f8c20062c90 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c36ffd700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c20062c90 0x7f8c20065150 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7f8c280041d0 tx=0x7f8c28009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 1 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8c380ffd40 0x7f8c38100120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c381060d0 osd=0 3 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c381060d0 osd=0 3 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c381060d0 osd=0 3 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 1 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c381060d0 osd=0 3 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 0 1 2024-02-23T00:36:19.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c381060d0 osd=0 4 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 10 client.37679.objecter _send_command 1 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.646+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c3805ee80 con 0x7f8c380ffd40 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.647+0000 7f8c367fc700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8c380ffd40 0x7f8c38100120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.647+0000 7f8c1ffff700 10 client.37679.objecter ms_handle_connect 0x7f8c380ffd40 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.647+0000 7f8c367fc700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8c380ffd40 0x7f8c38100120 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.647+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8c3805ee80 con 0x7f8c380ffd40 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.647+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.647+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 1 = system:0 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.647+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c381060d0 osd=0 4 2024-02-23T00:36:19.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.647+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 0 1 2024-02-23T00:36:19.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:19.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 2 2024-02-23T00:36:19.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c381060d0 osd=0 3 2024-02-23T00:36:19.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c381060d0 osd=0 3 2024-02-23T00:36:19.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c381060d0 osd=0 3 2024-02-23T00:36:19.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 2 2024-02-23T00:36:19.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c381060d0 osd=0 3 2024-02-23T00:36:19.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 0 2 2024-02-23T00:36:19.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c381060d0 osd=0 4 2024-02-23T00:36:19.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 10 client.37679.objecter _send_command 2 2024-02-23T00:36:19.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.662+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c380273c0 con 0x7f8c380ffd40 2024-02-23T00:36:19.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8c380273c0 con 0x7f8c380ffd40 2024-02-23T00:36:19.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:19.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c381060d0 osd=0 4 2024-02-23T00:36:19.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 0 2 2024-02-23T00:36:19.666 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:36:19.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:19.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 3 2024-02-23T00:36:19.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6805/685354171 conn(0x7f8c380598f0 0x7f8c38059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c380286b0 osd=1 3 2024-02-23T00:36:19.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c380286b0 osd=1 3 2024-02-23T00:36:19.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c380286b0 osd=1 3 2024-02-23T00:36:19.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 3 2024-02-23T00:36:19.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c380286b0 osd=1 3 2024-02-23T00:36:19.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 1 3 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c380286b0 osd=1 4 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 10 client.37679.objecter _send_command 3 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c38028820 con 0x7f8c380598f0 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c36ffd700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6805/685354171 conn(0x7f8c380598f0 0x7f8c38059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c36ffd700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6805/685354171 conn(0x7f8c380598f0 0x7f8c38059cd0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.664+0000 7f8c1ffff700 10 client.37679.objecter ms_handle_connect 0x7f8c380598f0 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.665+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8c2800fc10 con 0x7f8c380598f0 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.665+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.665+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 3 = system:0 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.665+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c380286b0 osd=1 4 2024-02-23T00:36:19.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.665+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 1 3 2024-02-23T00:36:19.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:19.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 4 2024-02-23T00:36:19.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c380286b0 osd=1 3 2024-02-23T00:36:19.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c380286b0 osd=1 3 2024-02-23T00:36:19.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c380286b0 osd=1 3 2024-02-23T00:36:19.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 4 2024-02-23T00:36:19.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c380286b0 osd=1 3 2024-02-23T00:36:19.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 1 4 2024-02-23T00:36:19.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c380286b0 osd=1 4 2024-02-23T00:36:19.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 10 client.37679.objecter _send_command 4 2024-02-23T00:36:19.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.679+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c38059360 con 0x7f8c380598f0 2024-02-23T00:36:19.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8c38059360 con 0x7f8c380598f0 2024-02-23T00:36:19.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:19.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c380286b0 osd=1 4 2024-02-23T00:36:19.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 1 4 2024-02-23T00:36:19.683 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:36:19.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:19.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 5 2024-02-23T00:36:19.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8c3811c2d0 0x7f8c3811e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c38027d00 osd=2 3 2024-02-23T00:36:19.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c38027d00 osd=2 3 2024-02-23T00:36:19.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c38027d00 osd=2 3 2024-02-23T00:36:19.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 5 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c38027d00 osd=2 3 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 2 5 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c38027d00 osd=2 4 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 10 client.37679.objecter _send_command 5 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c38028380 con 0x7f8c3811c2d0 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c367fc700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8c3811c2d0 0x7f8c3811e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c1ffff700 10 client.37679.objecter ms_handle_connect 0x7f8c3811c2d0 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.681+0000 7f8c367fc700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8c3811c2d0 0x7f8c3811e6c0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.682+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8c38028380 con 0x7f8c3811c2d0 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.682+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.682+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 5 = system:0 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.682+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c38027d00 osd=2 4 2024-02-23T00:36:19.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.682+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 2 5 2024-02-23T00:36:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 6 2024-02-23T00:36:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c38027d00 osd=2 3 2024-02-23T00:36:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c38027d00 osd=2 3 2024-02-23T00:36:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c38027d00 osd=2 3 2024-02-23T00:36:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 6 2024-02-23T00:36:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c38027d00 osd=2 3 2024-02-23T00:36:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 2 6 2024-02-23T00:36:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c38027d00 osd=2 4 2024-02-23T00:36:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 10 client.37679.objecter _send_command 6 2024-02-23T00:36:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.696+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c38027770 con 0x7f8c3811c2d0 2024-02-23T00:36:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.697+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8c38027770 con 0x7f8c3811c2d0 2024-02-23T00:36:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.697+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:19.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.697+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.697+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c38027d00 osd=2 4 2024-02-23T00:36:19.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.697+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 2 6 2024-02-23T00:36:19.700 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 7 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6813/128059790 conn(0x7f8c3811fd30 0x7f8c38122120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b920 osd=3 3 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b920 osd=3 3 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b920 osd=3 3 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 7 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3811b920 osd=3 3 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 3 7 2024-02-23T00:36:19.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b920 osd=3 4 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 10 client.37679.objecter _send_command 7 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c3811bfa0 con 0x7f8c3811fd30 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c377fe700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6813/128059790 conn(0x7f8c3811fd30 0x7f8c38122120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c1ffff700 10 client.37679.objecter ms_handle_connect 0x7f8c3811fd30 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c377fe700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6813/128059790 conn(0x7f8c3811fd30 0x7f8c38122120 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8c3811bfa0 con 0x7f8c3811fd30 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 7 = system:0 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3811b920 osd=3 4 2024-02-23T00:36:19.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.698+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 3 7 2024-02-23T00:36:19.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:36:19.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 8 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b920 osd=3 3 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b920 osd=3 3 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b920 osd=3 3 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 8 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3811b920 osd=3 3 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 3 8 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b920 osd=3 4 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 10 client.37679.objecter _send_command 8 2024-02-23T00:36:19.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.713+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c3811b390 con 0x7f8c3811fd30 2024-02-23T00:36:19.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8c3811b390 con 0x7f8c3811fd30 2024-02-23T00:36:19.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:19.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3811b920 osd=3 4 2024-02-23T00:36:19.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 3 8 2024-02-23T00:36:19.718 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:36:19.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:19.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 9 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c3811f890 0x7f8c381257a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b390 osd=4 3 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b390 osd=4 3 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b390 osd=4 3 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 9 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3811b390 osd=4 3 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 4 9 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b390 osd=4 4 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 10 client.37679.objecter _send_command 9 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.715+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c3811b770 con 0x7f8c3811f890 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.716+0000 7f8c36ffd700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c3811f890 0x7f8c381257a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.716+0000 7f8c1ffff700 10 client.37679.objecter ms_handle_connect 0x7f8c3811f890 2024-02-23T00:36:19.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.716+0000 7f8c36ffd700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c3811f890 0x7f8c381257a0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.717+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8c3811b770 con 0x7f8c3811f890 2024-02-23T00:36:19.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.717+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:36:19.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.717+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 9 = system:0 2024-02-23T00:36:19.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.717+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3811b390 osd=4 4 2024-02-23T00:36:19.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.717+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 4 9 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 10 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b390 osd=4 3 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b390 osd=4 3 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b390 osd=4 3 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 10 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3811b390 osd=4 3 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 4 10 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b390 osd=4 4 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 10 client.37679.objecter _send_command 10 2024-02-23T00:36:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.732+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c3811b770 con 0x7f8c3811f890 2024-02-23T00:36:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8c3811b770 con 0x7f8c3811f890 2024-02-23T00:36:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3811b390 osd=4 4 2024-02-23T00:36:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 4 10 2024-02-23T00:36:19.737 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:36:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 11 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8c381261a0 0x7f8c38128590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b770 osd=5 3 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b770 osd=5 3 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b770 osd=5 3 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 11 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3811b770 osd=5 3 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 5 11 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b770 osd=5 4 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 10 client.37679.objecter _send_command 11 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c3812b3a0 con 0x7f8c381261a0 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.734+0000 7f8c367fc700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8c381261a0 0x7f8c38128590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.735+0000 7f8c1ffff700 10 client.37679.objecter ms_handle_connect 0x7f8c381261a0 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.735+0000 7f8c367fc700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8c381261a0 0x7f8c38128590 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.735+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8c3812b3a0 con 0x7f8c381261a0 2024-02-23T00:36:19.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.735+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:36:19.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.735+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 11 = system:0 2024-02-23T00:36:19.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.735+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3811b770 osd=5 4 2024-02-23T00:36:19.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.735+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 5 11 2024-02-23T00:36:19.742 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:19.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:19.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 12 2024-02-23T00:36:19.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b770 osd=5 3 2024-02-23T00:36:19.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b770 osd=5 3 2024-02-23T00:36:19.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3811b770 osd=5 3 2024-02-23T00:36:19.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 12 2024-02-23T00:36:19.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3811b770 osd=5 3 2024-02-23T00:36:19.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 5 12 2024-02-23T00:36:19.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b770 osd=5 4 2024-02-23T00:36:19.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 10 client.37679.objecter _send_command 12 2024-02-23T00:36:19.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.751+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c3812b3a0 con 0x7f8c381261a0 2024-02-23T00:36:19.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.753+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8c3812b3a0 con 0x7f8c381261a0 2024-02-23T00:36:19.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.753+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:19.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.753+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.753+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3811b770 osd=5 4 2024-02-23T00:36:19.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.753+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 5 12 2024-02-23T00:36:19.756 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 13 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8c3812a6a0 0x7f8c3812ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3812b3a0 osd=6 3 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b3a0 osd=6 3 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3812b3a0 osd=6 3 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 13 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3812b3a0 osd=6 3 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 6 13 2024-02-23T00:36:19.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b3a0 osd=6 4 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 10 client.37679.objecter _send_command 13 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c3812b140 con 0x7f8c3812a6a0 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c377fe700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8c3812a6a0 0x7f8c3812ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c377fe700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8c3812a6a0 0x7f8c3812ab20 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.754+0000 7f8c1ffff700 10 client.37679.objecter ms_handle_connect 0x7f8c3812a6a0 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.755+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8c3812b140 con 0x7f8c3812a6a0 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.755+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.755+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 13 = system:0 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.755+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3812b3a0 osd=6 4 2024-02-23T00:36:19.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.755+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 6 13 2024-02-23T00:36:19.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:19.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 14 2024-02-23T00:36:19.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3812b3a0 osd=6 3 2024-02-23T00:36:19.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b3a0 osd=6 3 2024-02-23T00:36:19.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3812b3a0 osd=6 3 2024-02-23T00:36:19.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 14 2024-02-23T00:36:19.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3812b3a0 osd=6 3 2024-02-23T00:36:19.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 6 14 2024-02-23T00:36:19.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b3a0 osd=6 4 2024-02-23T00:36:19.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 10 client.37679.objecter _send_command 14 2024-02-23T00:36:19.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.770+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c3812b140 con 0x7f8c3812a6a0 2024-02-23T00:36:19.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8c3812b140 con 0x7f8c3812a6a0 2024-02-23T00:36:19.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:19.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3812b3a0 osd=6 4 2024-02-23T00:36:19.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 6 14 2024-02-23T00:36:19.775 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:36:19.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:19.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 15 2024-02-23T00:36:19.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6812/373787000 conn(0x7f8c3812db70 0x7f8c3812dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:19.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3812b140 osd=7 3 2024-02-23T00:36:19.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b140 osd=7 3 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3812b140 osd=7 3 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 15 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3812b140 osd=7 3 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 7 15 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b140 osd=7 4 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 10 client.37679.objecter _send_command 15 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c38131940 con 0x7f8c3812db70 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.772+0000 7f8c36ffd700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6812/373787000 conn(0x7f8c3812db70 0x7f8c3812dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.773+0000 7f8c1ffff700 10 client.37679.objecter ms_handle_connect 0x7f8c3812db70 2024-02-23T00:36:19.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.773+0000 7f8c36ffd700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6812/373787000 conn(0x7f8c3812db70 0x7f8c3812dfd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:19.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.774+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8c38131940 con 0x7f8c3812db70 2024-02-23T00:36:19.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.774+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:36:19.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.774+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 15 = system:0 2024-02-23T00:36:19.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.774+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3812b140 osd=7 4 2024-02-23T00:36:19.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.774+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 7 15 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 10 client.37679.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign -1 16 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3812b140 osd=7 3 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b140 osd=7 3 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 20 client.37679.objecter _get_session s=0x7f8c3812b140 osd=7 3 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_remove -1 16 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 20 client.37679.objecter get_session s=0x7f8c3812b140 osd=7 3 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 15 client.37679.objecter _session_command_op_assign 7 16 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b140 osd=7 4 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 10 client.37679.objecter _send_command 16 2024-02-23T00:36:19.790 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.788+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c38128b80 con 0x7f8c3812db70 2024-02-23T00:36:19.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.789+0000 7f8c1ffff700 1 -- 172.21.15.87:0/774203725 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8c38128b80 con 0x7f8c3812db70 2024-02-23T00:36:19.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.789+0000 7f8c1ffff700 10 client.37679.objecter ms_dispatch 0x7f8c38001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:19.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.789+0000 7f8c1ffff700 10 client.37679.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.789+0000 7f8c1ffff700 20 client.37679.objecter put_session s=0x7f8c3812b140 osd=7 4 2024-02-23T00:36:19.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.789+0000 7f8c1ffff700 15 client.37679.objecter _session_command_op_remove 7 16 2024-02-23T00:36:19.792 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:36:19.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 librados: watch_flush enter 2024-02-23T00:36:19.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 librados: watch_flush exit 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 client.37679.objecter close_session for osd.0 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8c380ffd40 msgr2=0x7f8c38100120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6801/2153191863 conn(0x7f8c380ffd40 0x7f8c38100120 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c381060d0 osd=0 1 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 client.37679.objecter close_session for osd.1 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6805/685354171 conn(0x7f8c380598f0 msgr2=0x7f8c38059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6805/685354171 conn(0x7f8c380598f0 0x7f8c38059cd0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c380286b0 osd=1 1 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 client.37679.objecter close_session for osd.2 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8c3811c2d0 msgr2=0x7f8c3811e6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8c3811c2d0 0x7f8c3811e6c0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c38027d00 osd=2 1 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 client.37679.objecter close_session for osd.3 2024-02-23T00:36:19.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6813/128059790 conn(0x7f8c3811fd30 msgr2=0x7f8c38122120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6813/128059790 conn(0x7f8c3811fd30 0x7f8c38122120 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c377fe700 1 -- 172.21.15.87:0/774203725 reap_dead start 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b920 osd=3 1 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 client.37679.objecter close_session for osd.4 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c3811f890 msgr2=0x7f8c381257a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8c3811f890 0x7f8c381257a0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c377fe700 1 -- 172.21.15.87:0/774203725 reap_dead start 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b390 osd=4 1 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 client.37679.objecter close_session for osd.5 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8c381261a0 msgr2=0x7f8c38128590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8c381261a0 0x7f8c38128590 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3811b770 osd=5 1 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 client.37679.objecter close_session for osd.6 2024-02-23T00:36:19.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8c3812a6a0 msgr2=0x7f8c3812ab20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8c3812a6a0 0x7f8c3812ab20 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b3a0 osd=6 1 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 client.37679.objecter close_session for osd.7 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6812/373787000 conn(0x7f8c3812db70 msgr2=0x7f8c3812dfd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.171:6812/373787000 conn(0x7f8c3812db70 0x7f8c3812dfd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 20 client.37679.objecter put_session s=0x7f8c3812b140 osd=7 1 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 20 client.37679.objecter shutdown clearing up homeless session... 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 10 client.37679.objecter successfully canceled tick 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c20062c90 msgr2=0x7f8c20065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8c20062c90 0x7f8c20065150 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 rx=0x7f8c280041d0 tx=0x7f8c28009040).stop 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c380fd420 msgr2=0x7f8c380f92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 --2- 172.21.15.87:0/774203725 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8c380fd420 0x7f8c380f92a0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 rx=0x7f8c30001df0 tx=0x7f8c30004d10).stop 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c377fe700 1 -- 172.21.15.87:0/774203725 reap_dead start 2024-02-23T00:36:19.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 shutdown_connections 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 >> 172.21.15.87:0/774203725 conn(0x7f8c380f26f0 msgr2=0x7f8c380f41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 shutdown_connections 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 -- 172.21.15.87:0/774203725 wait complete. 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:19.790+0000 7f8c3d61c700 1 librados: shutdown 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:36:19.797 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:36:20.244 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:20.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:36:20.347 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:20.379 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:19.647088+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? 172.21.15.87:0/774203725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:20.380 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:20.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:20.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:20.509 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:20.538 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:20.558 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:20.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:20.567 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:19.647088+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? 172.21.15.87:0/774203725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:20.568 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:19.647088+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? 172.21.15.87:0/774203725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:20.661 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:20.689 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:20.881 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:20.886 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:20.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:20.991 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:21.017 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:21.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:19.653407+0000 mgr.y (mgr.24437) 383 : cluster [DBG] pgmap v615: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:36:21.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:36:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:36:21.215 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:21.216 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:19.653407+0000 mgr.y (mgr.24437) 383 : cluster [DBG] pgmap v615: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:36:21.217 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:19.653407+0000 mgr.y (mgr.24437) 383 : cluster [DBG] pgmap v615: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:36:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:21.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:21.319 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:36:21.345 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:21.549 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:21.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:36:21.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:21.682 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:20.902352+0000 mgr.y (mgr.24437) 384 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:22.059875+0000 mon.a (mon.0) 2576 : audit [INF] from='client.? 172.21.15.87:0/3129626313' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:36:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:22.080647+0000 mon.a (mon.0) 2577 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:22.086994+0000 mon.a (mon.0) 2578 : audit [INF] from='client.? 172.21.15.87:0/3129626313' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:36:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:22.087055+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-02-23T00:36:22.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:20.902352+0000 mgr.y (mgr.24437) 384 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:22.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:22.059875+0000 mon.a (mon.0) 2576 : audit [INF] from='client.? 172.21.15.87:0/3129626313' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:36:22.506 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:22.080647+0000 mon.a (mon.0) 2577 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:22.506 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:22.086994+0000 mon.a (mon.0) 2578 : audit [INF] from='client.? 172.21.15.87:0/3129626313' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:36:22.506 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:22.087055+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-02-23T00:36:22.506 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:22] "GET /metrics HTTP/1.1" 200 188111 "" "Prometheus/2.33.4" 2024-02-23T00:36:22.506 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:20.902352+0000 mgr.y (mgr.24437) 384 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:22.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:22.059875+0000 mon.a (mon.0) 2576 : audit [INF] from='client.? 172.21.15.87:0/3129626313' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:36:22.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:22.080647+0000 mon.a (mon.0) 2577 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:22.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:22.086994+0000 mon.a (mon.0) 2578 : audit [INF] from='client.? 172.21.15.87:0/3129626313' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:36:22.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:22.087055+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-02-23T00:36:22.511 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:22.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:22.616 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:22.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.876+0000 7f497ecb8700 1 -- 172.21.15.87:0/3304436102 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49780f7400 msgr2=0x7f4978103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:22.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.876+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3304436102 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49780f7400 0x7f4978103a20 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 rx=0x7f4970009be0 tx=0x7f4970004d10).stop 2024-02-23T00:36:22.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.876+0000 7f497ecb8700 1 -- 172.21.15.87:0/3304436102 shutdown_connections 2024-02-23T00:36:22.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.876+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3304436102 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49780f7400 0x7f4978103a20 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:22.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.876+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3304436102 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49780fcbb0 0x7f49780fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.876+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3304436102 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49780fd510 0x7f49780f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.876+0000 7f497ecb8700 1 -- 172.21.15.87:0/3304436102 >> 172.21.15.87:0/3304436102 conn(0x7f49780f2820 msgr2=0x7f49780f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 -- 172.21.15.87:0/3304436102 shutdown_connections 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 -- 172.21.15.87:0/3304436102 wait complete. 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 librados: starting msgr at 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 librados: starting objecter 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 Processor -- start 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 -- start start 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 librados: setting wanted keys 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 librados: calling monclient init 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49780f7400 0x7f497810df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49780fcbb0 0x7f497810e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49780fd510 0x7f4978108110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978108600 con 0x7f49780fcbb0 2024-02-23T00:36:22.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978108770 con 0x7f49780fd510 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ecb8700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4978108930 con 0x7f49780f7400 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f4977fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49780fcbb0 0x7f497810e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f4977fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49780fcbb0 0x7f497810e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55540/0 (socket says 172.21.15.87:55540) 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f4977fff700 1 -- 172.21.15.87:0/3129626313 learned_addr learned my addr 172.21.15.87:0/3129626313 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.877+0000 7f497ca54700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49780f7400 0x7f497810df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f497d255700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49780fd510 0x7f4978108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4977fff700 1 -- 172.21.15.87:0/3129626313 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49780f7400 msgr2=0x7f497810df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4977fff700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49780f7400 0x7f497810df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4977fff700 1 -- 172.21.15.87:0/3129626313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49780fd510 msgr2=0x7f4978108110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4977fff700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49780fd510 0x7f4978108110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4977fff700 1 -- 172.21.15.87:0/3129626313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4964005040 con 0x7f49780fcbb0 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f497d255700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49780fd510 0x7f4978108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f497ca54700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49780f7400 0x7f497810df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4977fff700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49780fcbb0 0x7f497810e460 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f49640048f0 tx=0x7f4964009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:22.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4975ffb700 10 client.?.objecter ms_handle_connect 0x7f49780fcbb0 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4975ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4975ffb700 1 -- 172.21.15.87:0/3129626313 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4964007b00 con 0x7f49780fcbb0 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49700097c0 con 0x7f49780fcbb0 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4975ffb700 1 -- 172.21.15.87:0/3129626313 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f496400f040 con 0x7f49780fcbb0 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f4975ffb700 1 -- 172.21.15.87:0/3129626313 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4964014a00 con 0x7f49780fcbb0 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f497ecb8700 10 client.27225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4978114da0 con 0x7f49780fcbb0 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.878+0000 7f497ecb8700 1 librados: init done 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.879+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497805fa60 con 0x7f49780fcbb0 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.879+0000 7f4975ffb700 1 -- 172.21.15.87:0/3129626313 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f496400e580 con 0x7f49780fcbb0 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.880+0000 7f4975ffb700 1 --2- 172.21.15.87:0/3129626313 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4968062c90 0x7f4968065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.880+0000 7f4975ffb700 1 -- 172.21.15.87:0/3129626313 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(418..418 src has 173..418) v4 ==== 8178+0+0 (secure 0 0 0) 0x7f496407ce40 con 0x7f49780fcbb0 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.880+0000 7f4975ffb700 10 client.27225.objecter ms_dispatch 0x7f4978001550 osd_map(418..418 src has 173..418) v4 2024-02-23T00:36:22.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.880+0000 7f4975ffb700 3 client.27225.objecter handle_osd_map got epochs [418,418] > 0 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.880+0000 7f4975ffb700 3 client.27225.objecter handle_osd_map decoding full epoch 418 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.880+0000 7f497ca54700 1 --2- 172.21.15.87:0/3129626313 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4968062c90 0x7f4968065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.880+0000 7f4975ffb700 20 client.27225.objecter dump_active .. 0 homeless 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.880+0000 7f4975ffb700 10 client.27225.objecter ms_handle_connect 0x7f4968062c90 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.880+0000 7f497ca54700 1 --2- 172.21.15.87:0/3129626313 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4968062c90 0x7f4968065150 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 rx=0x7f496c00f2f0 tx=0x7f496c009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:21.884+0000 7f4975ffb700 1 -- 172.21.15.87:0/3129626313 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4964049f60 con 0x7f49780fcbb0 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.058+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f4978106120 con 0x7f49780fcbb0 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.085+0000 7f4975ffb700 1 -- 172.21.15.87:0/3129626313 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v419) v1 ==== 91+0+0 (secure 0 0 0) 0x7f4964050290 con 0x7f49780fcbb0 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.088+0000 7f497ecb8700 10 librados: watch_flush enter 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.089+0000 7f497ecb8700 10 librados: watch_flush exit 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.089+0000 7f497ecb8700 20 client.27225.objecter shutdown clearing up homeless session... 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.089+0000 7f497ecb8700 10 client.27225.objecter successfully canceled tick 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.089+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4968062c90 msgr2=0x7f4968065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.089+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3129626313 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4968062c90 0x7f4968065150 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 rx=0x7f496c00f2f0 tx=0x7f496c009040).stop 2024-02-23T00:36:22.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.089+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49780fcbb0 msgr2=0x7f497810e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.089+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49780fcbb0 0x7f497810e460 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 rx=0x7f49640048f0 tx=0x7f4964009cb0).stop 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.090+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 shutdown_connections 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.090+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3129626313 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4968062c90 0x7f4968065150 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.090+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49780fd510 0x7f4978108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.090+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49780f7400 0x7f497810df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.090+0000 7f497ecb8700 1 --2- 172.21.15.87:0/3129626313 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49780fcbb0 0x7f497810e460 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.090+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 >> 172.21.15.87:0/3129626313 conn(0x7f49780f2820 msgr2=0x7f49780fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.090+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 shutdown_connections 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.090+0000 7f497ecb8700 1 -- 172.21.15.87:0/3129626313 wait complete. 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:22.090+0000 7f497ecb8700 1 librados: shutdown 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:36:22.650 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:22.909 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:36:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:21.654303+0000 mgr.y (mgr.24437) 385 : cluster [DBG] pgmap v616: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2024-02-23T00:36:23.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:21.654303+0000 mgr.y (mgr.24437) 385 : cluster [DBG] pgmap v616: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2024-02-23T00:36:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:23.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:21.654303+0000 mgr.y (mgr.24437) 385 : cluster [DBG] pgmap v616: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 0 op/s 2024-02-23T00:36:23.499 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:23.527 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:23.645 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:36:23.645 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:36:23.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:36:23.651 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:23.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:23.756 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:23.788 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:23.810 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:23.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:23.914 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:23.958 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:36:23.958 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:36:23.961 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:23.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:24.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:24.094 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:36:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:36:24.096 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:24.120 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:24.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:24.223 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:24.255 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:24.276 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:24.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:24.320 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:23.770140+0000 osd.1 (osd.1) 181 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:36:24.321 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:23.773839+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:36:24.321 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:23.770140+0000 osd.1 (osd.1) 181 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:36:24.321 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:23.773839+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:36:24.380 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:24.409 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:23.770140+0000 osd.1 (osd.1) 181 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:36:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:23.773839+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:36:24.787 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-23T00:36:24.826 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:24.861 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:24.861 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:24.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:24.964 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:24.993 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:25.061 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:36:25.142 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:25.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:25.246 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:25.273 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:25.311 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:25.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:25.367 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:23.209832+0000 osd.5 (osd.5) 161 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:36:25.367 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:23.210700+0000 osd.5 (osd.5) 162 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:36:25.367 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:23.654738+0000 mgr.y (mgr.24437) 386 : cluster [DBG] pgmap v618: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 311 B/s rd, 0 B/s wr, 0 op/s 2024-02-23T00:36:25.367 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:24.016034+0000 osd.2 (osd.2) 157 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:36:25.367 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:24.016834+0000 osd.2 (osd.2) 158 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:36:25.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:24.026385+0000 osd.0 (osd.0) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:25.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:24.026591+0000 osd.0 (osd.0) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:25.368 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:24.114868+0000 mon.a (mon.0) 2580 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-02-23T00:36:25.390 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:23.209832+0000 osd.5 (osd.5) 161 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:36:25.390 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:23.210700+0000 osd.5 (osd.5) 162 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:36:25.390 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:23.654738+0000 mgr.y (mgr.24437) 386 : cluster [DBG] pgmap v618: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 311 B/s rd, 0 B/s wr, 0 op/s 2024-02-23T00:36:25.390 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:24.016034+0000 osd.2 (osd.2) 157 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:36:25.390 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:24.016834+0000 osd.2 (osd.2) 158 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:36:25.390 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:24.026385+0000 osd.0 (osd.0) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:25.390 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:24.026591+0000 osd.0 (osd.0) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:25.390 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:24.114868+0000 mon.a (mon.0) 2580 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-02-23T00:36:25.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:23.209832+0000 osd.5 (osd.5) 161 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:36:25.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:23.210700+0000 osd.5 (osd.5) 162 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:36:25.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:23.654738+0000 mgr.y (mgr.24437) 386 : cluster [DBG] pgmap v618: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 311 B/s rd, 0 B/s wr, 0 op/s 2024-02-23T00:36:25.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:24.016034+0000 osd.2 (osd.2) 157 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:36:25.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:24.016834+0000 osd.2 (osd.2) 158 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:36:25.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:24.026385+0000 osd.0 (osd.0) 191 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:36:25.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:24.026591+0000 osd.0 (osd.0) 192 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:36:25.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:24.114868+0000 mon.a (mon.0) 2580 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-02-23T00:36:25.395 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:36:25.414 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:25.443 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:25.541 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:25.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:25.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:25.644 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:25.672 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:25.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:25.974 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:26.002 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:26.075 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:26.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:26.182 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:26.209 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:26.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:25.078823+0000 osd.4 (osd.4) 189 : cluster [DBG] 4.d scrub starts 2024-02-23T00:36:26.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:25.082584+0000 osd.4 (osd.4) 190 : cluster [DBG] 4.d scrub ok 2024-02-23T00:36:26.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:25.097750+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:36:26.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:25.100623+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:36:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:25.078823+0000 osd.4 (osd.4) 189 : cluster [DBG] 4.d scrub starts 2024-02-23T00:36:26.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:25.082584+0000 osd.4 (osd.4) 190 : cluster [DBG] 4.d scrub ok 2024-02-23T00:36:26.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:25.097750+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:36:26.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:25.100623+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:36:26.414 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:25.078823+0000 osd.4 (osd.4) 189 : cluster [DBG] 4.d scrub starts 2024-02-23T00:36:26.414 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:25.082584+0000 osd.4 (osd.4) 190 : cluster [DBG] 4.d scrub ok 2024-02-23T00:36:26.414 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:25.097750+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:36:26.414 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:25.100623+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:36:26.458 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:26.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:26.562 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:26.590 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:27.007 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:36:27.007 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:36:27.407 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:25.655627+0000 mgr.y (mgr.24437) 387 : cluster [DBG] pgmap v620: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 1 op/s 2024-02-23T00:36:27.408 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:27.088376+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-02-23T00:36:27.441 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:27.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:25.655627+0000 mgr.y (mgr.24437) 387 : cluster [DBG] pgmap v620: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 1 op/s 2024-02-23T00:36:27.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:27.088376+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-02-23T00:36:27.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:25.655627+0000 mgr.y (mgr.24437) 387 : cluster [DBG] pgmap v620: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 948 B/s rd, 0 B/s wr, 1 op/s 2024-02-23T00:36:27.444 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:27.088376+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-02-23T00:36:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:27.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:27.546 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:27.577 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:27.705 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:27.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:27.714 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-23T00:36:27.810 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:27.837 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:27.858 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:27.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:27.961 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:27.989 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:28.008 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:28.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:28.029 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:36:28.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:28.143 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:28.216 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:28.218 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:27.075463+0000 osd.6 (osd.6) 152 : cluster [DBG] 63.12 scrub starts 2024-02-23T00:36:28.218 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:27.078986+0000 osd.6 (osd.6) 153 : cluster [DBG] 63.12 scrub ok 2024-02-23T00:36:28.218 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:27.180587+0000 osd.5 (osd.5) 163 : cluster [DBG] 3.c scrub starts 2024-02-23T00:36:28.218 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:27.181967+0000 osd.5 (osd.5) 164 : cluster [DBG] 3.c scrub ok 2024-02-23T00:36:28.218 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:27.948789+0000 osd.7 (osd.7) 197 : cluster [DBG] 22.e scrub starts 2024-02-23T00:36:28.218 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:27.951686+0000 osd.7 (osd.7) 198 : cluster [DBG] 22.e scrub ok 2024-02-23T00:36:28.219 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:27.075463+0000 osd.6 (osd.6) 152 : cluster [DBG] 63.12 scrub starts 2024-02-23T00:36:28.219 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:27.078986+0000 osd.6 (osd.6) 153 : cluster [DBG] 63.12 scrub ok 2024-02-23T00:36:28.219 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:27.180587+0000 osd.5 (osd.5) 163 : cluster [DBG] 3.c scrub starts 2024-02-23T00:36:28.219 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:27.181967+0000 osd.5 (osd.5) 164 : cluster [DBG] 3.c scrub ok 2024-02-23T00:36:28.219 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:27.948789+0000 osd.7 (osd.7) 197 : cluster [DBG] 22.e scrub starts 2024-02-23T00:36:28.219 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:27.951686+0000 osd.7 (osd.7) 198 : cluster [DBG] 22.e scrub ok 2024-02-23T00:36:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:28.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:28.277 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:27.075463+0000 osd.6 (osd.6) 152 : cluster [DBG] 63.12 scrub starts 2024-02-23T00:36:28.277 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:27.078986+0000 osd.6 (osd.6) 153 : cluster [DBG] 63.12 scrub ok 2024-02-23T00:36:28.277 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:27.180587+0000 osd.5 (osd.5) 163 : cluster [DBG] 3.c scrub starts 2024-02-23T00:36:28.277 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:27.181967+0000 osd.5 (osd.5) 164 : cluster [DBG] 3.c scrub ok 2024-02-23T00:36:28.277 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:27.948789+0000 osd.7 (osd.7) 197 : cluster [DBG] 22.e scrub starts 2024-02-23T00:36:28.277 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:27.951686+0000 osd.7 (osd.7) 198 : cluster [DBG] 22.e scrub ok 2024-02-23T00:36:28.321 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:28.349 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:28.448 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.445+0000 7fcf63dc3700 1 -- 172.21.15.87:0/377795349 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf5c0fe510 msgr2=0x7fcf5c0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.445+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/377795349 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf5c0fe510 0x7fcf5c0f7d40 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 rx=0x7fcf4c0047b0 tx=0x7fcf4c009cb0).stop 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.445+0000 7fcf63dc3700 1 -- 172.21.15.87:0/377795349 shutdown_connections 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.445+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/377795349 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf5c0f8420 0x7fcf5c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.445+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/377795349 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf5c0fdbb0 0x7fcf5c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.445+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/377795349 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf5c0fe510 0x7fcf5c0f7d40 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.445+0000 7fcf63dc3700 1 -- 172.21.15.87:0/377795349 >> 172.21.15.87:0/377795349 conn(0x7fcf5c0f3800 msgr2=0x7fcf5c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.445+0000 7fcf63dc3700 1 -- 172.21.15.87:0/377795349 shutdown_connections 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.445+0000 7fcf63dc3700 1 -- 172.21.15.87:0/377795349 wait complete. 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 librados: starting msgr at 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 librados: starting objecter 2024-02-23T00:36:28.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 Processor -- start 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 -- start start 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 librados: setting wanted keys 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 librados: calling monclient init 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf5c0f8420 0x7fcf5c06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf5c0fdbb0 0x7fcf5c06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf5c0fe510 0x7fcf5c06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c116510 con 0x7fcf5c0fe510 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c116680 con 0x7fcf5c0fdbb0 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.446+0000 7fcf63dc3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5c116840 con 0x7fcf5c0f8420 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf61b5f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf5c0f8420 0x7fcf5c06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf62360700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf5c0fe510 0x7fcf5c06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf61b5f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf5c0f8420 0x7fcf5c06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34072/0 (socket says 172.21.15.87:34072) 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf61b5f700 1 -- 172.21.15.87:0/1394414194 learned_addr learned my addr 172.21.15.87:0/1394414194 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:28.451 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf6135e700 1 --2- 172.21.15.87:0/1394414194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf5c0fdbb0 0x7fcf5c06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf61b5f700 1 -- 172.21.15.87:0/1394414194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf5c0fdbb0 msgr2=0x7fcf5c06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf61b5f700 1 --2- 172.21.15.87:0/1394414194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf5c0fdbb0 0x7fcf5c06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf61b5f700 1 -- 172.21.15.87:0/1394414194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf5c0fe510 msgr2=0x7fcf5c06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf61b5f700 1 --2- 172.21.15.87:0/1394414194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf5c0fe510 0x7fcf5c06bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf61b5f700 1 -- 172.21.15.87:0/1394414194 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf540097c0 con 0x7fcf5c0f8420 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf62360700 1 --2- 172.21.15.87:0/1394414194 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf5c0fe510 0x7fcf5c06bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf6135e700 1 --2- 172.21.15.87:0/1394414194 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf5c0fdbb0 0x7fcf5c06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf61b5f700 1 --2- 172.21.15.87:0/1394414194 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf5c0f8420 0x7fcf5c06b0d0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7fcf580027f0 tx=0x7fcf58002b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf52ffd700 10 client.?.objecter ms_handle_connect 0x7fcf5c0f8420 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf52ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58007b40 con 0x7fcf5c0f8420 2024-02-23T00:36:28.452 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf4c005040 con 0x7fcf5c0f8420 2024-02-23T00:36:28.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf5800f040 con 0x7fcf5c0f8420 2024-02-23T00:36:28.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58014730 con 0x7fcf5c0f8420 2024-02-23T00:36:28.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf63dc3700 10 client.37691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:28.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5c116ed0 con 0x7fcf5c0f8420 2024-02-23T00:36:28.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.447+0000 7fcf63dc3700 1 librados: init done 2024-02-23T00:36:28.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.448+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fcf5c0606d0 con 0x7fcf5c0f8420 2024-02-23T00:36:28.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.449+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fcf58014b60 con 0x7fcf5c0f8420 2024-02-23T00:36:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:28.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.449+0000 7fcf52ffd700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcf48062ce0 0x7fcf480651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.449+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(422..422 src has 173..422) v4 ==== 8178+0+0 (secure 0 0 0) 0x7fcf5807cfe0 con 0x7fcf5c0f8420 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.449+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 osd_map(422..422 src has 173..422) v4 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.449+0000 7fcf52ffd700 3 client.37691.objecter handle_osd_map got epochs [422,422] > 0 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.449+0000 7fcf52ffd700 3 client.37691.objecter handle_osd_map decoding full epoch 422 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.449+0000 7fcf6135e700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcf48062ce0 0x7fcf480651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.449+0000 7fcf52ffd700 20 client.37691.objecter dump_active .. 0 homeless 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.449+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v422) v1 ==== 54+0+16 (secure 0 0 0) 0x7fcf58050430 con 0x7fcf5c0f8420 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf52ffd700 10 client.37691.objecter ms_handle_connect 0x7fcf48062ce0 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 1 2024-02-23T00:36:28.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf6135e700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcf48062ce0 0x7fcf480651a0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 rx=0x7fcf5c06c870 tx=0x7fcf4c007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6801/2153191863 conn(0x7fcf5c0fee00 0x7fcf5c0ff200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c05ee80 osd=0 3 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c05ee80 osd=0 3 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c05ee80 osd=0 3 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 1 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c05ee80 osd=0 3 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 0 1 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c05ee80 osd=0 4 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 10 client.37691.objecter _send_command 1 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf5c1071b0 con 0x7fcf5c0fee00 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf62360700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6801/2153191863 conn(0x7fcf5c0fee00 0x7fcf5c0ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf52ffd700 10 client.37691.objecter ms_handle_connect 0x7fcf5c0fee00 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.450+0000 7fcf62360700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6801/2153191863 conn(0x7fcf5c0fee00 0x7fcf5c0ff200 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.451+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf5c1071b0 con 0x7fcf5c0fee00 2024-02-23T00:36:28.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.451+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:36:28.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.451+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 1 = system:0 2024-02-23T00:36:28.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.451+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c05ee80 osd=0 4 2024-02-23T00:36:28.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.451+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 0 1 2024-02-23T00:36:28.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 2 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c05ee80 osd=0 3 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c05ee80 osd=0 3 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c05ee80 osd=0 3 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 2 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c05ee80 osd=0 3 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 0 2 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c05ee80 osd=0 4 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 10 client.37691.objecter _send_command 2 2024-02-23T00:36:28.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.465+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcf5c0273c0 con 0x7fcf5c0fee00 2024-02-23T00:36:28.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcf5c0273c0 con 0x7fcf5c0fee00 2024-02-23T00:36:28.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:28.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c05ee80 osd=0 4 2024-02-23T00:36:28.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 0 2 2024-02-23T00:36:28.470 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:36:28.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 3 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6805/685354171 conn(0x7fcf5c0598f0 0x7fcf5c059d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c028710 osd=1 3 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c028710 osd=1 3 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c028710 osd=1 3 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 3 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c028710 osd=1 3 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 1 3 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c028710 osd=1 4 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 10 client.37691.objecter _send_command 3 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf5c11d840 con 0x7fcf5c0598f0 2024-02-23T00:36:28.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.467+0000 7fcf61b5f700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6805/685354171 conn(0x7fcf5c0598f0 0x7fcf5c059d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.468+0000 7fcf52ffd700 10 client.37691.objecter ms_handle_connect 0x7fcf5c0598f0 2024-02-23T00:36:28.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.468+0000 7fcf61b5f700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6805/685354171 conn(0x7fcf5c0598f0 0x7fcf5c059d10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.468+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf5805cac0 con 0x7fcf5c0598f0 2024-02-23T00:36:28.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.468+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:36:28.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.468+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 3 = system:0 2024-02-23T00:36:28.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.468+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c028710 osd=1 4 2024-02-23T00:36:28.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.468+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 1 3 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 4 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c028710 osd=1 3 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c028710 osd=1 3 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c028710 osd=1 3 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 4 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c028710 osd=1 3 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 1 4 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c028710 osd=1 4 2024-02-23T00:36:28.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 10 client.37691.objecter _send_command 4 2024-02-23T00:36:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.483+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcf5c059360 con 0x7fcf5c0598f0 2024-02-23T00:36:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcf5c059360 con 0x7fcf5c0598f0 2024-02-23T00:36:28.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:28.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c028710 osd=1 4 2024-02-23T00:36:28.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 1 4 2024-02-23T00:36:28.487 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:36:28.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:28.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 5 2024-02-23T00:36:28.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcf5c028270 0x7fcf5c11c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c059710 osd=2 3 2024-02-23T00:36:28.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c059710 osd=2 3 2024-02-23T00:36:28.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c059710 osd=2 3 2024-02-23T00:36:28.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 5 2024-02-23T00:36:28.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c059710 osd=2 3 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 2 5 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c059710 osd=2 4 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 10 client.37691.objecter _send_command 5 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.484+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf5c11c750 con 0x7fcf5c028270 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.485+0000 7fcf6135e700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcf5c028270 0x7fcf5c11c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.485+0000 7fcf52ffd700 10 client.37691.objecter ms_handle_connect 0x7fcf5c028270 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.485+0000 7fcf6135e700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcf5c028270 0x7fcf5c11c260 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.485+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf5c11c750 con 0x7fcf5c028270 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.485+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.485+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 5 = system:0 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.485+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c059710 osd=2 4 2024-02-23T00:36:28.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.485+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 2 5 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 6 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c059710 osd=2 3 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c059710 osd=2 3 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c059710 osd=2 3 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 6 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c059710 osd=2 3 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 2 6 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c059710 osd=2 4 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 10 client.37691.objecter _send_command 6 2024-02-23T00:36:28.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.500+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcf5c0277d0 con 0x7fcf5c028270 2024-02-23T00:36:28.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcf5c0277d0 con 0x7fcf5c028270 2024-02-23T00:36:28.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:28.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c059710 osd=2 4 2024-02-23T00:36:28.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 2 6 2024-02-23T00:36:28.505 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:36:28.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 7 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6813/128059790 conn(0x7fcf5c11f9c0 0x7fcf5c121db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c027be0 osd=3 3 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c027be0 osd=3 3 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c027be0 osd=3 3 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 7 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c027be0 osd=3 3 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 3 7 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c027be0 osd=3 4 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 10 client.37691.objecter _send_command 7 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf5c124a00 con 0x7fcf5c11f9c0 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf62360700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6813/128059790 conn(0x7fcf5c11f9c0 0x7fcf5c121db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf52ffd700 10 client.37691.objecter ms_handle_connect 0x7fcf5c11f9c0 2024-02-23T00:36:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.502+0000 7fcf62360700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6813/128059790 conn(0x7fcf5c11f9c0 0x7fcf5c121db0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.503+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf5c124a00 con 0x7fcf5c11f9c0 2024-02-23T00:36:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.503+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:36:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.503+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 7 = system:0 2024-02-23T00:36:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.503+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c027be0 osd=3 4 2024-02-23T00:36:28.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.503+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 3 7 2024-02-23T00:36:28.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:28.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 8 2024-02-23T00:36:28.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c027be0 osd=3 3 2024-02-23T00:36:28.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c027be0 osd=3 3 2024-02-23T00:36:28.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c027be0 osd=3 3 2024-02-23T00:36:28.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 8 2024-02-23T00:36:28.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c027be0 osd=3 3 2024-02-23T00:36:28.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 3 8 2024-02-23T00:36:28.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c027be0 osd=3 4 2024-02-23T00:36:28.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 10 client.37691.objecter _send_command 8 2024-02-23T00:36:28.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.522+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcf5c11cbb0 con 0x7fcf5c11f9c0 2024-02-23T00:36:28.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcf5c11cbb0 con 0x7fcf5c11f9c0 2024-02-23T00:36:28.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:28.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c027be0 osd=3 4 2024-02-23T00:36:28.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 3 8 2024-02-23T00:36:28.527 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 9 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcf5c11bdc0 0x7fcf5c123420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c11c970 osd=4 3 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11c970 osd=4 3 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c11c970 osd=4 3 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 9 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c11c970 osd=4 3 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 4 9 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11c970 osd=4 4 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 10 client.37691.objecter _send_command 9 2024-02-23T00:36:28.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.524+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf5c11cae0 con 0x7fcf5c11bdc0 2024-02-23T00:36:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.525+0000 7fcf61b5f700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcf5c11bdc0 0x7fcf5c123420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.525+0000 7fcf52ffd700 10 client.37691.objecter ms_handle_connect 0x7fcf5c11bdc0 2024-02-23T00:36:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.525+0000 7fcf61b5f700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcf5c11bdc0 0x7fcf5c123420 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.526+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf5c11cae0 con 0x7fcf5c11bdc0 2024-02-23T00:36:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.526+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:36:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.526+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 9 = system:0 2024-02-23T00:36:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.526+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c11c970 osd=4 4 2024-02-23T00:36:28.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.526+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 4 9 2024-02-23T00:36:28.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:28.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 10 2024-02-23T00:36:28.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c11c970 osd=4 3 2024-02-23T00:36:28.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11c970 osd=4 3 2024-02-23T00:36:28.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c11c970 osd=4 3 2024-02-23T00:36:28.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 10 2024-02-23T00:36:28.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c11c970 osd=4 3 2024-02-23T00:36:28.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 4 10 2024-02-23T00:36:28.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11c970 osd=4 4 2024-02-23T00:36:28.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 10 client.37691.objecter _send_command 10 2024-02-23T00:36:28.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.541+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcf5c11cae0 con 0x7fcf5c11bdc0 2024-02-23T00:36:28.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.543+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcf5c11cae0 con 0x7fcf5c11bdc0 2024-02-23T00:36:28.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.543+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:28.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.543+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.543+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c11c970 osd=4 4 2024-02-23T00:36:28.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.543+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 4 10 2024-02-23T00:36:28.546 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:36:28.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:28.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 11 2024-02-23T00:36:28.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcf5c123e50 0x7fcf5c124290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c11cae0 osd=5 3 2024-02-23T00:36:28.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11cae0 osd=5 3 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c11cae0 osd=5 3 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 11 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c11cae0 osd=5 3 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 5 11 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11cae0 osd=5 4 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 10 client.37691.objecter _send_command 11 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf5c12b2f0 con 0x7fcf5c123e50 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf6135e700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcf5c123e50 0x7fcf5c124290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf52ffd700 10 client.37691.objecter ms_handle_connect 0x7fcf5c123e50 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.544+0000 7fcf6135e700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcf5c123e50 0x7fcf5c124290 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.545+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf5c12b2f0 con 0x7fcf5c123e50 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.545+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:36:28.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.545+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 11 = system:0 2024-02-23T00:36:28.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.545+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c11cae0 osd=5 4 2024-02-23T00:36:28.549 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.545+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 5 11 2024-02-23T00:36:28.556 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:28.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 12 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c11cae0 osd=5 3 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11cae0 osd=5 3 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c11cae0 osd=5 3 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 12 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c11cae0 osd=5 3 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 5 12 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11cae0 osd=5 4 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 10 client.37691.objecter _send_command 12 2024-02-23T00:36:28.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.559+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcf5c128b90 con 0x7fcf5c123e50 2024-02-23T00:36:28.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcf5c128b90 con 0x7fcf5c123e50 2024-02-23T00:36:28.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:28.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c11cae0 osd=5 4 2024-02-23T00:36:28.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 5 12 2024-02-23T00:36:28.564 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:36:28.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 13 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcf5c129270 0x7fcf5c12b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c128b90 osd=6 3 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c128b90 osd=6 3 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c128b90 osd=6 3 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 13 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c128b90 osd=6 3 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 6 13 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c128b90 osd=6 4 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 10 client.37691.objecter _send_command 13 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf5c12e3e0 con 0x7fcf5c129270 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf62360700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcf5c129270 0x7fcf5c12b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf62360700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcf5c129270 0x7fcf5c12b660 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 10 client.37691.objecter ms_handle_connect 0x7fcf5c129270 2024-02-23T00:36:28.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf5c12e3e0 con 0x7fcf5c129270 2024-02-23T00:36:28.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:36:28.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 13 = system:0 2024-02-23T00:36:28.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c128b90 osd=6 4 2024-02-23T00:36:28.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.562+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 6 13 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 14 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c128b90 osd=6 3 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c128b90 osd=6 3 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c128b90 osd=6 3 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 14 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c128b90 osd=6 3 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 6 14 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c128b90 osd=6 4 2024-02-23T00:36:28.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 10 client.37691.objecter _send_command 14 2024-02-23T00:36:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.577+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcf5c12e3e0 con 0x7fcf5c129270 2024-02-23T00:36:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.578+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcf5c12e3e0 con 0x7fcf5c129270 2024-02-23T00:36:28.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.578+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.578+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.578+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c128b90 osd=6 4 2024-02-23T00:36:28.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.578+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 6 14 2024-02-23T00:36:28.581 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.578+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.578+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 15 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.578+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6812/373787000 conn(0x7fcf5c12c100 0x7fcf5c12c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c12e3e0 osd=7 3 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c12e3e0 osd=7 3 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c12e3e0 osd=7 3 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 15 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c12e3e0 osd=7 3 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 7 15 2024-02-23T00:36:28.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c12e3e0 osd=7 4 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf63dc3700 10 client.37691.objecter _send_command 15 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf5c12e1b0 con 0x7fcf5c12c100 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf61b5f700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6812/373787000 conn(0x7fcf5c12c100 0x7fcf5c12c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf52ffd700 10 client.37691.objecter ms_handle_connect 0x7fcf5c12c100 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.579+0000 7fcf61b5f700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6812/373787000 conn(0x7fcf5c12c100 0x7fcf5c12c560 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.580+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcf5c12e1b0 con 0x7fcf5c12c100 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.580+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.580+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 15 = system:0 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.580+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c12e3e0 osd=7 4 2024-02-23T00:36:28.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.580+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 7 15 2024-02-23T00:36:28.585 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:28.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 10 client.37691.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign -1 16 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c12e3e0 osd=7 3 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c12e3e0 osd=7 3 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 20 client.37691.objecter _get_session s=0x7fcf5c12e3e0 osd=7 3 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_remove -1 16 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 20 client.37691.objecter get_session s=0x7fcf5c12e3e0 osd=7 3 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 15 client.37691.objecter _session_command_op_assign 7 16 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c12e3e0 osd=7 4 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 10 client.37691.objecter _send_command 16 2024-02-23T00:36:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.594+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcf5c12e1b0 con 0x7fcf5c12c100 2024-02-23T00:36:28.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf52ffd700 1 -- 172.21.15.87:0/1394414194 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcf5c12e1b0 con 0x7fcf5c12c100 2024-02-23T00:36:28.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf52ffd700 10 client.37691.objecter ms_dispatch 0x7fcf5c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:28.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf52ffd700 10 client.37691.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf52ffd700 20 client.37691.objecter put_session s=0x7fcf5c12e3e0 osd=7 4 2024-02-23T00:36:28.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf52ffd700 15 client.37691.objecter _session_command_op_remove 7 16 2024-02-23T00:36:28.599 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:36:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 librados: watch_flush enter 2024-02-23T00:36:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 librados: watch_flush exit 2024-02-23T00:36:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 client.37691.objecter close_session for osd.0 2024-02-23T00:36:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6801/2153191863 conn(0x7fcf5c0fee00 msgr2=0x7fcf5c0ff200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6801/2153191863 conn(0x7fcf5c0fee00 0x7fcf5c0ff200 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c05ee80 osd=0 1 2024-02-23T00:36:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 client.37691.objecter close_session for osd.1 2024-02-23T00:36:28.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6805/685354171 conn(0x7fcf5c0598f0 msgr2=0x7fcf5c059d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6805/685354171 conn(0x7fcf5c0598f0 0x7fcf5c059d10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c028710 osd=1 1 2024-02-23T00:36:28.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 client.37691.objecter close_session for osd.2 2024-02-23T00:36:28.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcf5c028270 msgr2=0x7fcf5c11c260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcf5c028270 0x7fcf5c11c260 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c059710 osd=2 1 2024-02-23T00:36:28.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 client.37691.objecter close_session for osd.3 2024-02-23T00:36:28.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6813/128059790 conn(0x7fcf5c11f9c0 msgr2=0x7fcf5c121db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6813/128059790 conn(0x7fcf5c11f9c0 0x7fcf5c121db0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf62360700 1 -- 172.21.15.87:0/1394414194 reap_dead start 2024-02-23T00:36:28.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c027be0 osd=3 1 2024-02-23T00:36:28.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 client.37691.objecter close_session for osd.4 2024-02-23T00:36:28.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcf5c11bdc0 msgr2=0x7fcf5c123420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcf5c11bdc0 0x7fcf5c123420 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11c970 osd=4 1 2024-02-23T00:36:28.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 client.37691.objecter close_session for osd.5 2024-02-23T00:36:28.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf62360700 1 -- 172.21.15.87:0/1394414194 reap_dead start 2024-02-23T00:36:28.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcf5c123e50 msgr2=0x7fcf5c124290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcf5c123e50 0x7fcf5c124290 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c11cae0 osd=5 1 2024-02-23T00:36:28.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 client.37691.objecter close_session for osd.6 2024-02-23T00:36:28.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcf5c129270 msgr2=0x7fcf5c12b660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcf5c129270 0x7fcf5c12b660 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c128b90 osd=6 1 2024-02-23T00:36:28.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 client.37691.objecter close_session for osd.7 2024-02-23T00:36:28.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6812/373787000 conn(0x7fcf5c12c100 msgr2=0x7fcf5c12c560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.171:6812/373787000 conn(0x7fcf5c12c100 0x7fcf5c12c560 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:28.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 20 client.37691.objecter put_session s=0x7fcf5c12e3e0 osd=7 1 2024-02-23T00:36:28.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 20 client.37691.objecter shutdown clearing up homeless session... 2024-02-23T00:36:28.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.596+0000 7fcf63dc3700 10 client.37691.objecter successfully canceled tick 2024-02-23T00:36:28.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcf48062ce0 msgr2=0x7fcf480651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcf48062ce0 0x7fcf480651a0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 rx=0x7fcf5c06c870 tx=0x7fcf4c007f90).stop 2024-02-23T00:36:28.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf5c0f8420 msgr2=0x7fcf5c06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:28.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf63dc3700 1 --2- 172.21.15.87:0/1394414194 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf5c0f8420 0x7fcf5c06b0d0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 rx=0x7fcf580027f0 tx=0x7fcf58002b00).stop 2024-02-23T00:36:28.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf62360700 1 -- 172.21.15.87:0/1394414194 reap_dead start 2024-02-23T00:36:28.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 shutdown_connections 2024-02-23T00:36:28.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 >> 172.21.15.87:0/1394414194 conn(0x7fcf5c0f3800 msgr2=0x7fcf5c0f5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:28.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 shutdown_connections 2024-02-23T00:36:28.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf63dc3700 1 -- 172.21.15.87:0/1394414194 wait complete. 2024-02-23T00:36:28.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:28.597+0000 7fcf63dc3700 1 librados: shutdown 2024-02-23T00:36:28.606 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.606 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.607 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.607 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.607 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.607 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.607 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.607 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:36:28.784 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:28.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:28.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:28.887 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:28.916 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:29.013 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:29.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:29.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:29.117 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:29.144 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:29.242 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:29.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:29.329 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:27.656104+0000 mgr.y (mgr.24437) 388 : cluster [DBG] pgmap v622: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:36:29.329 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:28.178379+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-02-23T00:36:29.329 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:28.181623+0000 osd.5 (osd.5) 165 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:36:29.329 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:28.184193+0000 osd.5 (osd.5) 166 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:36:29.329 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:28.450946+0000 mon.c (mon.2) 432 : audit [DBG] from='client.? 172.21.15.87:0/1394414194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:29.329 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:29.100623+0000 osd.4 (osd.4) 191 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:36:29.329 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:29.101496+0000 osd.4 (osd.4) 192 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:36:29.346 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:29.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 -- 172.21.15.87:0/1435589976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 msgr2=0x7f2b080fcec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:29.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/1435589976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 0x7f2b080fcec0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 rx=0x7f2afc004870 tx=0x7f2afc009cb0).stop 2024-02-23T00:36:29.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 -- 172.21.15.87:0/1435589976 shutdown_connections 2024-02-23T00:36:29.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/1435589976 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b080f79d0 0x7f2b08103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:29.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/1435589976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 0x7f2b080fcec0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:29.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/1435589976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b080fd440 0x7f2b080f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:29.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 -- 172.21.15.87:0/1435589976 >> 172.21.15.87:0/1435589976 conn(0x7f2b08070030 msgr2=0x7f2b080f4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:29.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 -- 172.21.15.87:0/1435589976 shutdown_connections 2024-02-23T00:36:29.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 -- 172.21.15.87:0/1435589976 wait complete. 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 librados: starting msgr at 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 librados: starting objecter 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 Processor -- start 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 -- start start 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 librados: setting wanted keys 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.338+0000 7f2b0ee92700 1 librados: calling monclient init 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0ee92700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b080f79d0 0x7f2b08106f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0ee92700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 0x7f2b081073f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0ee92700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b080fd440 0x7f2b081078e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0ee92700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b08116580 con 0x7f2b080fcae0 2024-02-23T00:36:29.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0ee92700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b081166c0 con 0x7f2b080f79d0 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0ee92700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b08116850 con 0x7f2b080fd440 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b07fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 0x7f2b081073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0d42f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b080fd440 0x7f2b081078e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0cc2e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b080f79d0 0x7f2b08106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b07fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 0x7f2b081073f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55576/0 (socket says 172.21.15.87:55576) 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b07fff700 1 -- 172.21.15.87:0/2227703065 learned_addr learned my addr 172.21.15.87:0/2227703065 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b07fff700 1 -- 172.21.15.87:0/2227703065 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b080fd440 msgr2=0x7f2b081078e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b07fff700 1 --2- 172.21.15.87:0/2227703065 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b080fd440 0x7f2b081078e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b07fff700 1 -- 172.21.15.87:0/2227703065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b080f79d0 msgr2=0x7f2b08106f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b07fff700 1 --2- 172.21.15.87:0/2227703065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b080f79d0 0x7f2b08106f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b07fff700 1 -- 172.21.15.87:0/2227703065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc005040 con 0x7f2b080fcae0 2024-02-23T00:36:29.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0d42f700 1 --2- 172.21.15.87:0/2227703065 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b080fd440 0x7f2b081078e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b07fff700 1 --2- 172.21.15.87:0/2227703065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 0x7f2b081073f0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7f2af40041d0 tx=0x7f2af4009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b05ffb700 10 client.?.objecter ms_handle_connect 0x7f2b080fcae0 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b05ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b05ffb700 1 -- 172.21.15.87:0/2227703065 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4007b40 con 0x7f2b080fcae0 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b05ffb700 1 -- 172.21.15.87:0/2227703065 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2af400f040 con 0x7f2b080fcae0 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b05ffb700 1 -- 172.21.15.87:0/2227703065 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4014810 con 0x7f2b080fcae0 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.339+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b08116b30 con 0x7f2b080fcae0 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.340+0000 7f2b0ee92700 10 client.27237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.340+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b08117050 con 0x7f2b080fcae0 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.340+0000 7f2b0ee92700 1 librados: init done 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.340+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0805f990 con 0x7f2b080fcae0 2024-02-23T00:36:29.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b05ffb700 1 -- 172.21.15.87:0/2227703065 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f2af4009110 con 0x7f2b080fcae0 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b05ffb700 1 --2- 172.21.15.87:0/2227703065 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2af8062c90 0x7f2af8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b05ffb700 1 -- 172.21.15.87:0/2227703065 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(422..422 src has 173..422) v4 ==== 8178+0+0 (secure 0 0 0) 0x7f2af407d270 con 0x7f2b080fcae0 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b05ffb700 10 client.27237.objecter ms_dispatch 0x7f2b08001550 osd_map(422..422 src has 173..422) v4 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b05ffb700 3 client.27237.objecter handle_osd_map got epochs [422,422] > 0 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b05ffb700 3 client.27237.objecter handle_osd_map decoding full epoch 422 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b0cc2e700 1 --2- 172.21.15.87:0/2227703065 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2af8062c90 0x7f2af8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b05ffb700 20 client.27237.objecter dump_active .. 0 homeless 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b05ffb700 10 client.27237.objecter ms_handle_connect 0x7f2af8062c90 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.341+0000 7f2b0cc2e700 1 --2- 172.21.15.87:0/2227703065 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2af8062c90 0x7f2af8065150 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7f2afc0043d0 tx=0x7f2afc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.345+0000 7f2b05ffb700 1 -- 172.21.15.87:0/2227703065 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f2af40506c0 con 0x7f2b080fcae0 2024-02-23T00:36:29.375 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:29.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.510+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b080f6890 con 0x7f2af8062c90 2024-02-23T00:36:29.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.526+0000 7f2b05ffb700 1 -- 172.21.15.87:0/2227703065 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+598619 (secure 0 0 0) 0x7f2b080f6890 con 0x7f2af8062c90 2024-02-23T00:36:29.536 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.529+0000 7f2b0ee92700 10 librados: watch_flush enter 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.529+0000 7f2b0ee92700 10 librados: watch_flush exit 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.529+0000 7f2b0ee92700 20 client.27237.objecter shutdown clearing up homeless session... 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.529+0000 7f2b0ee92700 10 client.27237.objecter successfully canceled tick 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2af8062c90 msgr2=0x7f2af8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/2227703065 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2af8062c90 0x7f2af8065150 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7f2afc0043d0 tx=0x7f2afc00b040).stop 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 msgr2=0x7f2b081073f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/2227703065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 0x7f2b081073f0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 rx=0x7f2af40041d0 tx=0x7f2af4009f60).stop 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 shutdown_connections 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/2227703065 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2af8062c90 0x7f2af8065150 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/2227703065 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2b080fd440 0x7f2b081078e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/2227703065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2b080f79d0 0x7f2b08106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:29.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 --2- 172.21.15.87:0/2227703065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2b080fcae0 0x7f2b081073f0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:29.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 >> 172.21.15.87:0/2227703065 conn(0x7f2b08070030 msgr2=0x7f2b0806d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:29.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 shutdown_connections 2024-02-23T00:36:29.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 -- 172.21.15.87:0/2227703065 wait complete. 2024-02-23T00:36:29.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:29.530+0000 7f2b0ee92700 1 librados: shutdown 2024-02-23T00:36:29.544 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:36:29.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:27.656104+0000 mgr.y (mgr.24437) 388 : cluster [DBG] pgmap v622: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:36:29.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:28.178379+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-02-23T00:36:29.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:28.181623+0000 osd.5 (osd.5) 165 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:36:29.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:28.184193+0000 osd.5 (osd.5) 166 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:36:29.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:28.450946+0000 mon.c (mon.2) 432 : audit [DBG] from='client.? 172.21.15.87:0/1394414194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:29.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:29.100623+0000 osd.4 (osd.4) 191 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:36:29.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:29.101496+0000 osd.4 (osd.4) 192 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:36:29.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:27.656104+0000 mgr.y (mgr.24437) 388 : cluster [DBG] pgmap v622: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:36:29.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:28.178379+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-02-23T00:36:29.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:28.181623+0000 osd.5 (osd.5) 165 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:36:29.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:28.184193+0000 osd.5 (osd.5) 166 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:36:29.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:28.450946+0000 mon.c (mon.2) 432 : audit [DBG] from='client.? 172.21.15.87:0/1394414194' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:29.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:29.100623+0000 osd.4 (osd.4) 191 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:36:29.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:29.101496+0000 osd.4 (osd.4) 192 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:36:29.574 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:29.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:29.679 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:29.706 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:29.776 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:36:29.776 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:36:29.776 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:36:29.776 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:36:29.776 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:36:29.778 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:29.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:29.882 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:29.910 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:30.008 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:30.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:30.111 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:30.138 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:29.977142+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:36:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:29.980389+0000 osd.7 (osd.7) 200 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:36:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:29.977142+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:36:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:29.980389+0000 osd.7 (osd.7) 200 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:36:30.408 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":623,"stamp":"2024-02-23T00:36:28.175963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007394,"num_objects":387,"num_object_clones":3,"num_object_copies":1161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":387,"num_whiteouts":0,"num_read":1502,"num_read_kb":1125,"num_write":1232,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":4003689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2251,"ondisk_log_size":2251,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":714,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2167124,"kb_used_data":45596,"kb_used_omap":0,"kb_used_meta":2121472,"kb_avail":654048940,"statfs":{"total":671965249536,"available":669746114560,"internally_reserved":0,"allocated":46690304,"data_stored":29961731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2172387328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.090993"},"pg_stats":[{"pgid":"212.1f","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090186+0000","last_change":"2024-02-23T00:36:02.103306+0000","last_active":"2024-02-23T00:36:22.090186+0000","last_peered":"2024-02-23T00:36:22.090186+0000","last_clean":"2024-02-23T00:36:22.090186+0000","last_became_active":"2024-02-23T00:36:02.102806+0000","last_became_peered":"2024-02-23T00:36:02.102806+0000","last_unstale":"2024-02-23T00:36:22.090186+0000","last_undegraded":"2024-02-23T00:36:22.090186+0000","last_fullsized":"2024-02-23T00:36:22.090186+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.1e","version":"409'1","reported_seq":22,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157911+0000","last_change":"2024-02-23T00:36:02.102101+0000","last_active":"2024-02-23T00:36:23.157911+0000","last_peered":"2024-02-23T00:36:23.157911+0000","last_clean":"2024-02-23T00:36:23.157911+0000","last_became_active":"2024-02-23T00:36:02.101813+0000","last_became_peered":"2024-02-23T00:36:02.101813+0000","last_unstale":"2024-02-23T00:36:23.157911+0000","last_undegraded":"2024-02-23T00:36:23.157911+0000","last_fullsized":"2024-02-23T00:36:23.157911+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.1d","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090752+0000","last_change":"2024-02-23T00:36:02.096801+0000","last_active":"2024-02-23T00:36:22.090752+0000","last_peered":"2024-02-23T00:36:22.090752+0000","last_clean":"2024-02-23T00:36:22.090752+0000","last_became_active":"2024-02-23T00:36:02.096502+0000","last_became_peered":"2024-02-23T00:36:02.096502+0000","last_unstale":"2024-02-23T00:36:22.090752+0000","last_undegraded":"2024-02-23T00:36:22.090752+0000","last_fullsized":"2024-02-23T00:36:22.090752+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.1c","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061362+0000","last_change":"2024-02-23T00:36:02.099432+0000","last_active":"2024-02-23T00:36:23.061362+0000","last_peered":"2024-02-23T00:36:23.061362+0000","last_clean":"2024-02-23T00:36:23.061362+0000","last_became_active":"2024-02-23T00:36:02.099148+0000","last_became_peered":"2024-02-23T00:36:02.099148+0000","last_unstale":"2024-02-23T00:36:23.061362+0000","last_undegraded":"2024-02-23T00:36:23.061362+0000","last_fullsized":"2024-02-23T00:36:23.061362+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.1b","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061150+0000","last_change":"2024-02-23T00:36:02.101530+0000","last_active":"2024-02-23T00:36:23.061150+0000","last_peered":"2024-02-23T00:36:23.061150+0000","last_clean":"2024-02-23T00:36:23.061150+0000","last_became_active":"2024-02-23T00:36:02.101232+0000","last_became_peered":"2024-02-23T00:36:02.101232+0000","last_unstale":"2024-02-23T00:36:23.061150+0000","last_undegraded":"2024-02-23T00:36:23.061150+0000","last_fullsized":"2024-02-23T00:36:23.061150+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.1a","version":"418'4","reported_seq":25,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061043+0000","last_change":"2024-02-23T00:36:02.106399+0000","last_active":"2024-02-23T00:36:23.061043+0000","last_peered":"2024-02-23T00:36:23.061043+0000","last_clean":"2024-02-23T00:36:23.061043+0000","last_became_active":"2024-02-23T00:36:02.105867+0000","last_became_peered":"2024-02-23T00:36:02.105867+0000","last_unstale":"2024-02-23T00:36:23.061043+0000","last_undegraded":"2024-02-23T00:36:23.061043+0000","last_fullsized":"2024-02-23T00:36:23.061043+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.19","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156236+0000","last_change":"2024-02-23T00:36:02.103848+0000","last_active":"2024-02-23T00:36:23.156236+0000","last_peered":"2024-02-23T00:36:23.156236+0000","last_clean":"2024-02-23T00:36:23.156236+0000","last_became_active":"2024-02-23T00:36:02.103563+0000","last_became_peered":"2024-02-23T00:36:02.103563+0000","last_unstale":"2024-02-23T00:36:23.156236+0000","last_undegraded":"2024-02-23T00:36:23.156236+0000","last_fullsized":"2024-02-23T00:36:23.156236+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"212.18","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090001+0000","last_change":"2024-02-23T00:36:02.097839+0000","last_active":"2024-02-23T00:36:22.090001+0000","last_peered":"2024-02-23T00:36:22.090001+0000","last_clean":"2024-02-23T00:36:22.090001+0000","last_became_active":"2024-02-23T00:36:02.097546+0000","last_became_peered":"2024-02-23T00:36:02.097546+0000","last_unstale":"2024-02-23T00:36:22.090001+0000","last_undegraded":"2024-02-23T00:36:22.090001+0000","last_fullsized":"2024-02-23T00:36:22.090001+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.17","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090541+0000","last_change":"2024-02-23T00:36:02.099269+0000","last_active":"2024-02-23T00:36:22.090541+0000","last_peered":"2024-02-23T00:36:22.090541+0000","last_clean":"2024-02-23T00:36:22.090541+0000","last_became_active":"2024-02-23T00:36:02.098705+0000","last_became_peered":"2024-02-23T00:36:02.098705+0000","last_unstale":"2024-02-23T00:36:22.090541+0000","last_undegraded":"2024-02-23T00:36:22.090541+0000","last_fullsized":"2024-02-23T00:36:22.090541+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.16","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159006+0000","last_change":"2024-02-23T00:36:02.107671+0000","last_active":"2024-02-23T00:36:23.159006+0000","last_peered":"2024-02-23T00:36:23.159006+0000","last_clean":"2024-02-23T00:36:23.159006+0000","last_became_active":"2024-02-23T00:36:02.107353+0000","last_became_peered":"2024-02-23T00:36:02.107353+0000","last_unstale":"2024-02-23T00:36:23.159006+0000","last_undegraded":"2024-02-23T00:36:23.159006+0000","last_fullsized":"2024-02-23T00:36:23.159006+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"212.15","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.655654+0000","last_change":"2024-02-23T00:36:02.096840+0000","last_active":"2024-02-23T00:36:22.655654+0000","last_peered":"2024-02-23T00:36:22.655654+0000","last_clean":"2024-02-23T00:36:22.655654+0000","last_became_active":"2024-02-23T00:36:02.096525+0000","last_became_peered":"2024-02-23T00:36:02.096525+0000","last_unstale":"2024-02-23T00:36:22.655654+0000","last_undegraded":"2024-02-23T00:36:22.655654+0000","last_fullsized":"2024-02-23T00:36:22.655654+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"212.14","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089503+0000","last_change":"2024-02-23T00:36:02.106848+0000","last_active":"2024-02-23T00:36:22.089503+0000","last_peered":"2024-02-23T00:36:22.089503+0000","last_clean":"2024-02-23T00:36:22.089503+0000","last_became_active":"2024-02-23T00:36:02.106597+0000","last_became_peered":"2024-02-23T00:36:02.106597+0000","last_unstale":"2024-02-23T00:36:22.089503+0000","last_undegraded":"2024-02-23T00:36:22.089503+0000","last_fullsized":"2024-02-23T00:36:22.089503+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.13","version":"418'5","reported_seq":29,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089282+0000","last_change":"2024-02-23T00:36:02.101840+0000","last_active":"2024-02-23T00:36:22.089282+0000","last_peered":"2024-02-23T00:36:22.089282+0000","last_clean":"2024-02-23T00:36:22.089282+0000","last_became_active":"2024-02-23T00:36:02.101483+0000","last_became_peered":"2024-02-23T00:36:02.101483+0000","last_unstale":"2024-02-23T00:36:22.089282+0000","last_undegraded":"2024-02-23T00:36:22.089282+0000","last_fullsized":"2024-02-23T00:36:22.089282+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.12","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061045+0000","last_change":"2024-02-23T00:36:02.108002+0000","last_active":"2024-02-23T00:36:23.061045+0000","last_peered":"2024-02-23T00:36:23.061045+0000","last_clean":"2024-02-23T00:36:23.061045+0000","last_became_active":"2024-02-23T00:36:02.107703+0000","last_became_peered":"2024-02-23T00:36:02.107703+0000","last_unstale":"2024-02-23T00:36:23.061045+0000","last_undegraded":"2024-02-23T00:36:23.061045+0000","last_fullsized":"2024-02-23T00:36:23.061045+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.11","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156217+0000","last_change":"2024-02-23T00:36:02.100607+0000","last_active":"2024-02-23T00:36:23.156217+0000","last_peered":"2024-02-23T00:36:23.156217+0000","last_clean":"2024-02-23T00:36:23.156217+0000","last_became_active":"2024-02-23T00:36:02.100305+0000","last_became_peered":"2024-02-23T00:36:02.100305+0000","last_unstale":"2024-02-23T00:36:23.156217+0000","last_undegraded":"2024-02-23T00:36:23.156217+0000","last_fullsized":"2024-02-23T00:36:23.156217+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.10","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090080+0000","last_change":"2024-02-23T00:36:02.097517+0000","last_active":"2024-02-23T00:36:22.090080+0000","last_peered":"2024-02-23T00:36:22.090080+0000","last_clean":"2024-02-23T00:36:22.090080+0000","last_became_active":"2024-02-23T00:36:02.097231+0000","last_became_peered":"2024-02-23T00:36:02.097231+0000","last_unstale":"2024-02-23T00:36:22.090080+0000","last_undegraded":"2024-02-23T00:36:22.090080+0000","last_fullsized":"2024-02-23T00:36:22.090080+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.f","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090634+0000","last_change":"2024-02-23T00:36:02.101443+0000","last_active":"2024-02-23T00:36:22.090634+0000","last_peered":"2024-02-23T00:36:22.090634+0000","last_clean":"2024-02-23T00:36:22.090634+0000","last_became_active":"2024-02-23T00:36:02.101142+0000","last_became_peered":"2024-02-23T00:36:02.101142+0000","last_unstale":"2024-02-23T00:36:22.090634+0000","last_undegraded":"2024-02-23T00:36:22.090634+0000","last_fullsized":"2024-02-23T00:36:22.090634+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.e","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157938+0000","last_change":"2024-02-23T00:36:02.107936+0000","last_active":"2024-02-23T00:36:23.157938+0000","last_peered":"2024-02-23T00:36:23.157938+0000","last_clean":"2024-02-23T00:36:23.157938+0000","last_became_active":"2024-02-23T00:36:02.107631+0000","last_became_peered":"2024-02-23T00:36:02.107631+0000","last_unstale":"2024-02-23T00:36:23.157938+0000","last_undegraded":"2024-02-23T00:36:23.157938+0000","last_fullsized":"2024-02-23T00:36:23.157938+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.d","version":"409'2","reported_seq":26,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090663+0000","last_change":"2024-02-23T00:36:17.097162+0000","last_active":"2024-02-23T00:36:22.090663+0000","last_peered":"2024-02-23T00:36:22.090663+0000","last_clean":"2024-02-23T00:36:22.090663+0000","last_became_active":"2024-02-23T00:36:17.096479+0000","last_became_peered":"2024-02-23T00:36:17.096479+0000","last_unstale":"2024-02-23T00:36:22.090663+0000","last_undegraded":"2024-02-23T00:36:22.090663+0000","last_fullsized":"2024-02-23T00:36:22.090663+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.c","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089514+0000","last_change":"2024-02-23T00:36:02.106823+0000","last_active":"2024-02-23T00:36:22.089514+0000","last_peered":"2024-02-23T00:36:22.089514+0000","last_clean":"2024-02-23T00:36:22.089514+0000","last_became_active":"2024-02-23T00:36:02.106317+0000","last_became_peered":"2024-02-23T00:36:02.106317+0000","last_unstale":"2024-02-23T00:36:22.089514+0000","last_undegraded":"2024-02-23T00:36:22.089514+0000","last_fullsized":"2024-02-23T00:36:22.089514+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.b","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157149+0000","last_change":"2024-02-23T00:36:02.103181+0000","last_active":"2024-02-23T00:36:23.157149+0000","last_peered":"2024-02-23T00:36:23.157149+0000","last_clean":"2024-02-23T00:36:23.157149+0000","last_became_active":"2024-02-23T00:36:02.102550+0000","last_became_peered":"2024-02-23T00:36:02.102550+0000","last_unstale":"2024-02-23T00:36:23.157149+0000","last_undegraded":"2024-02-23T00:36:23.157149+0000","last_fullsized":"2024-02-23T00:36:23.157149+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.a","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090998+0000","last_change":"2024-02-23T00:36:02.104378+0000","last_active":"2024-02-23T00:36:22.090998+0000","last_peered":"2024-02-23T00:36:22.090998+0000","last_clean":"2024-02-23T00:36:22.090998+0000","last_became_active":"2024-02-23T00:36:02.104086+0000","last_became_peered":"2024-02-23T00:36:02.104086+0000","last_unstale":"2024-02-23T00:36:22.090998+0000","last_undegraded":"2024-02-23T00:36:22.090998+0000","last_fullsized":"2024-02-23T00:36:22.090998+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.9","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.060746+0000","last_change":"2024-02-23T00:36:02.105278+0000","last_active":"2024-02-23T00:36:23.060746+0000","last_peered":"2024-02-23T00:36:23.060746+0000","last_clean":"2024-02-23T00:36:23.060746+0000","last_became_active":"2024-02-23T00:36:02.104514+0000","last_became_peered":"2024-02-23T00:36:02.104514+0000","last_unstale":"2024-02-23T00:36:23.060746+0000","last_undegraded":"2024-02-23T00:36:23.060746+0000","last_fullsized":"2024-02-23T00:36:23.060746+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.8","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654434+0000","last_change":"2024-02-23T00:36:02.100543+0000","last_active":"2024-02-23T00:36:22.654434+0000","last_peered":"2024-02-23T00:36:22.654434+0000","last_clean":"2024-02-23T00:36:22.654434+0000","last_became_active":"2024-02-23T00:36:02.100227+0000","last_became_peered":"2024-02-23T00:36:02.100227+0000","last_unstale":"2024-02-23T00:36:22.654434+0000","last_undegraded":"2024-02-23T00:36:22.654434+0000","last_fullsized":"2024-02-23T00:36:22.654434+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"212.7","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090118+0000","last_change":"2024-02-23T00:36:02.103116+0000","last_active":"2024-02-23T00:36:22.090118+0000","last_peered":"2024-02-23T00:36:22.090118+0000","last_clean":"2024-02-23T00:36:22.090118+0000","last_became_active":"2024-02-23T00:36:02.102325+0000","last_became_peered":"2024-02-23T00:36:02.102325+0000","last_unstale":"2024-02-23T00:36:22.090118+0000","last_undegraded":"2024-02-23T00:36:22.090118+0000","last_fullsized":"2024-02-23T00:36:22.090118+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"212.6","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158035+0000","last_change":"2024-02-23T00:36:02.104334+0000","last_active":"2024-02-23T00:36:23.158035+0000","last_peered":"2024-02-23T00:36:23.158035+0000","last_clean":"2024-02-23T00:36:23.158035+0000","last_became_active":"2024-02-23T00:36:02.104029+0000","last_became_peered":"2024-02-23T00:36:02.104029+0000","last_unstale":"2024-02-23T00:36:23.158035+0000","last_undegraded":"2024-02-23T00:36:23.158035+0000","last_fullsized":"2024-02-23T00:36:23.158035+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.5","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088843+0000","last_change":"2024-02-23T00:36:02.100942+0000","last_active":"2024-02-23T00:36:22.088843+0000","last_peered":"2024-02-23T00:36:22.088843+0000","last_clean":"2024-02-23T00:36:22.088843+0000","last_became_active":"2024-02-23T00:36:02.100627+0000","last_became_peered":"2024-02-23T00:36:02.100627+0000","last_unstale":"2024-02-23T00:36:22.088843+0000","last_undegraded":"2024-02-23T00:36:22.088843+0000","last_fullsized":"2024-02-23T00:36:22.088843+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.4","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088767+0000","last_change":"2024-02-23T00:36:02.100940+0000","last_active":"2024-02-23T00:36:22.088767+0000","last_peered":"2024-02-23T00:36:22.088767+0000","last_clean":"2024-02-23T00:36:22.088767+0000","last_became_active":"2024-02-23T00:36:02.100661+0000","last_became_peered":"2024-02-23T00:36:02.100661+0000","last_unstale":"2024-02-23T00:36:22.088767+0000","last_undegraded":"2024-02-23T00:36:22.088767+0000","last_fullsized":"2024-02-23T00:36:22.088767+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.3","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157247+0000","last_change":"2024-02-23T00:36:02.103362+0000","last_active":"2024-02-23T00:36:23.157247+0000","last_peered":"2024-02-23T00:36:23.157247+0000","last_clean":"2024-02-23T00:36:23.157247+0000","last_became_active":"2024-02-23T00:36:02.102958+0000","last_became_peered":"2024-02-23T00:36:02.102958+0000","last_unstale":"2024-02-23T00:36:23.157247+0000","last_undegraded":"2024-02-23T00:36:23.157247+0000","last_fullsized":"2024-02-23T00:36:23.157247+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.2","version":"0'0","reported_seq":20,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089491+0000","last_change":"2024-02-23T00:36:02.107432+0000","last_active":"2024-02-23T00:36:22.089491+0000","last_peered":"2024-02-23T00:36:22.089491+0000","last_clean":"2024-02-23T00:36:22.089491+0000","last_became_active":"2024-02-23T00:36:02.107166+0000","last_became_peered":"2024-02-23T00:36:02.107166+0000","last_unstale":"2024-02-23T00:36:22.089491+0000","last_undegraded":"2024-02-23T00:36:22.089491+0000","last_fullsized":"2024-02-23T00:36:22.089491+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.1","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156232+0000","last_change":"2024-02-23T00:36:02.101419+0000","last_active":"2024-02-23T00:36:23.156232+0000","last_peered":"2024-02-23T00:36:23.156232+0000","last_clean":"2024-02-23T00:36:23.156232+0000","last_became_active":"2024-02-23T00:36:02.101143+0000","last_became_peered":"2024-02-23T00:36:02.101143+0000","last_unstale":"2024-02-23T00:36:23.156232+0000","last_undegraded":"2024-02-23T00:36:23.156232+0000","last_fullsized":"2024-02-23T00:36:23.156232+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.0","version":"0'0","reported_seq":21,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654460+0000","last_change":"2024-02-23T00:36:02.107393+0000","last_active":"2024-02-23T00:36:22.654460+0000","last_peered":"2024-02-23T00:36:22.654460+0000","last_clean":"2024-02-23T00:36:22.654460+0000","last_became_active":"2024-02-23T00:36:02.107093+0000","last_became_peered":"2024-02-23T00:36:02.107093+0000","last_unstale":"2024-02-23T00:36:22.654460+0000","last_undegraded":"2024-02-23T00:36:22.654460+0000","last_fullsized":"2024-02-23T00:36:22.654460+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":407,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:01.078658+0000","last_clean_scrub_stamp":"2024-02-23T00:36:01.078658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"63.0","version":"222'68","reported_seq":155,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156186+0000","last_change":"2024-02-23T00:36:18.521201+0000","last_active":"2024-02-23T00:36:23.156186+0000","last_peered":"2024-02-23T00:36:23.156186+0000","last_clean":"2024-02-23T00:36:23.156186+0000","last_became_active":"2024-02-23T00:34:42.057217+0000","last_became_peered":"2024-02-23T00:34:42.057217+0000","last_unstale":"2024-02-23T00:36:23.156186+0000","last_undegraded":"2024-02-23T00:36:23.156186+0000","last_fullsized":"2024-02-23T00:36:23.156186+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'68","last_scrub_stamp":"2024-02-23T00:34:57.645204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.645204+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.1","version":"340'86","reported_seq":446,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088940+0000","last_change":"2024-02-23T00:36:18.099243+0000","last_active":"2024-02-23T00:36:22.088940+0000","last_peered":"2024-02-23T00:36:22.088940+0000","last_clean":"2024-02-23T00:36:22.088940+0000","last_became_active":"2024-02-23T00:31:06.258140+0000","last_became_peered":"2024-02-23T00:31:06.258140+0000","last_unstale":"2024-02-23T00:36:22.088940+0000","last_undegraded":"2024-02-23T00:36:22.088940+0000","last_fullsized":"2024-02-23T00:36:22.088940+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'85","last_scrub_stamp":"2024-02-23T00:34:16.724630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:16.724630+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.2","version":"372'21","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089735+0000","last_change":"2024-02-23T00:36:18.105372+0000","last_active":"2024-02-23T00:36:22.089735+0000","last_peered":"2024-02-23T00:36:22.089735+0000","last_clean":"2024-02-23T00:36:22.089735+0000","last_became_active":"2024-02-23T00:31:06.329726+0000","last_became_peered":"2024-02-23T00:31:06.329726+0000","last_unstale":"2024-02-23T00:36:22.089735+0000","last_undegraded":"2024-02-23T00:36:22.089735+0000","last_fullsized":"2024-02-23T00:36:22.089735+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'20","last_scrub_stamp":"2024-02-23T00:34:23.049395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.049395+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.3","version":"398'27","reported_seq":410,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089768+0000","last_change":"2024-02-23T00:36:18.099944+0000","last_active":"2024-02-23T00:36:22.089768+0000","last_peered":"2024-02-23T00:36:22.089768+0000","last_clean":"2024-02-23T00:36:22.089768+0000","last_became_active":"2024-02-23T00:36:17.093628+0000","last_became_peered":"2024-02-23T00:36:17.093628+0000","last_unstale":"2024-02-23T00:36:22.089768+0000","last_undegraded":"2024-02-23T00:36:22.089768+0000","last_fullsized":"2024-02-23T00:36:22.089768+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'25","last_scrub_stamp":"2024-02-23T00:34:15.396956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.396956+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":24,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.4","version":"395'44","reported_seq":405,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159332+0000","last_change":"2024-02-23T00:36:18.278079+0000","last_active":"2024-02-23T00:36:23.159332+0000","last_peered":"2024-02-23T00:36:23.159332+0000","last_clean":"2024-02-23T00:36:23.159332+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:36:23.159332+0000","last_undegraded":"2024-02-23T00:36:23.159332+0000","last_fullsized":"2024-02-23T00:36:23.159332+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'39","last_scrub_stamp":"2024-02-23T00:34:19.398065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.398065+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.5","version":"234'124","reported_seq":479,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089812+0000","last_change":"2024-02-23T00:36:18.099631+0000","last_active":"2024-02-23T00:36:22.089812+0000","last_peered":"2024-02-23T00:36:22.089812+0000","last_clean":"2024-02-23T00:36:22.089812+0000","last_became_active":"2024-02-23T00:31:06.258422+0000","last_became_peered":"2024-02-23T00:31:06.258422+0000","last_unstale":"2024-02-23T00:36:22.089812+0000","last_undegraded":"2024-02-23T00:36:22.089812+0000","last_fullsized":"2024-02-23T00:36:22.089812+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'124","last_scrub_stamp":"2024-02-23T00:34:19.695449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.695449+0000","log_size":124,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.6","version":"340'52","reported_seq":421,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091704+0000","last_change":"2024-02-23T00:36:18.100071+0000","last_active":"2024-02-23T00:36:22.091704+0000","last_peered":"2024-02-23T00:36:22.091704+0000","last_clean":"2024-02-23T00:36:22.091704+0000","last_became_active":"2024-02-23T00:31:06.313441+0000","last_became_peered":"2024-02-23T00:31:06.313441+0000","last_unstale":"2024-02-23T00:36:22.091704+0000","last_undegraded":"2024-02-23T00:36:22.091704+0000","last_fullsized":"2024-02-23T00:36:22.091704+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:34:31.733148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.733148+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.7","version":"401'97","reported_seq":451,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091270+0000","last_change":"2024-02-23T00:36:18.106066+0000","last_active":"2024-02-23T00:36:22.091270+0000","last_peered":"2024-02-23T00:36:22.091270+0000","last_clean":"2024-02-23T00:36:22.091270+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:36:22.091270+0000","last_undegraded":"2024-02-23T00:36:22.091270+0000","last_fullsized":"2024-02-23T00:36:22.091270+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'92","last_scrub_stamp":"2024-02-23T00:34:18.124049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:18.124049+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":94,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.8","version":"340'71","reported_seq":361,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088587+0000","last_change":"2024-02-23T00:36:18.104508+0000","last_active":"2024-02-23T00:36:22.088587+0000","last_peered":"2024-02-23T00:36:22.088587+0000","last_clean":"2024-02-23T00:36:22.088587+0000","last_became_active":"2024-02-23T00:34:41.115222+0000","last_became_peered":"2024-02-23T00:34:41.115222+0000","last_unstale":"2024-02-23T00:36:22.088587+0000","last_undegraded":"2024-02-23T00:36:22.088587+0000","last_fullsized":"2024-02-23T00:36:22.088587+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'71","last_scrub_stamp":"2024-02-23T00:35:41.651500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.651500+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.9","version":"405'19","reported_seq":386,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088659+0000","last_change":"2024-02-23T00:36:18.104923+0000","last_active":"2024-02-23T00:36:22.088659+0000","last_peered":"2024-02-23T00:36:22.088659+0000","last_clean":"2024-02-23T00:36:22.088659+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:36:22.088659+0000","last_undegraded":"2024-02-23T00:36:22.088659+0000","last_fullsized":"2024-02-23T00:36:22.088659+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'17","last_scrub_stamp":"2024-02-23T00:34:17.107575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.107575+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.a","version":"248'22","reported_seq":391,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157071+0000","last_change":"2024-02-23T00:36:18.279002+0000","last_active":"2024-02-23T00:36:23.157071+0000","last_peered":"2024-02-23T00:36:23.157071+0000","last_clean":"2024-02-23T00:36:23.157071+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:36:23.157071+0000","last_undegraded":"2024-02-23T00:36:23.157071+0000","last_fullsized":"2024-02-23T00:36:23.157071+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'22","last_scrub_stamp":"2024-02-23T00:33:34.362545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.362545+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.b","version":"340'35","reported_seq":431,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157656+0000","last_change":"2024-02-23T00:36:18.521631+0000","last_active":"2024-02-23T00:36:23.157656+0000","last_peered":"2024-02-23T00:36:23.157656+0000","last_clean":"2024-02-23T00:36:23.157656+0000","last_became_active":"2024-02-23T00:34:41.103580+0000","last_became_peered":"2024-02-23T00:34:41.103580+0000","last_unstale":"2024-02-23T00:36:23.157656+0000","last_undegraded":"2024-02-23T00:36:23.157656+0000","last_fullsized":"2024-02-23T00:36:23.157656+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-23T00:33:37.391203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.391203+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.c","version":"418'95","reported_seq":455,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061861+0000","last_change":"2024-02-23T00:36:20.097131+0000","last_active":"2024-02-23T00:36:23.061861+0000","last_peered":"2024-02-23T00:36:23.061861+0000","last_clean":"2024-02-23T00:36:23.061861+0000","last_became_active":"2024-02-23T00:31:06.233844+0000","last_became_peered":"2024-02-23T00:31:06.233844+0000","last_unstale":"2024-02-23T00:36:23.061861+0000","last_undegraded":"2024-02-23T00:36:23.061861+0000","last_fullsized":"2024-02-23T00:36:23.061861+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'76","last_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_deep_scrub":"397'76","last_deep_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.098352+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":73,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.d","version":"395'39","reported_seq":431,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158267+0000","last_change":"2024-02-23T00:36:18.279931+0000","last_active":"2024-02-23T00:36:23.158267+0000","last_peered":"2024-02-23T00:36:23.158267+0000","last_clean":"2024-02-23T00:36:23.158267+0000","last_became_active":"2024-02-23T00:34:41.109682+0000","last_became_peered":"2024-02-23T00:34:41.109682+0000","last_unstale":"2024-02-23T00:36:23.158267+0000","last_undegraded":"2024-02-23T00:36:23.158267+0000","last_fullsized":"2024-02-23T00:36:23.158267+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-23T00:33:45.265466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.265466+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.e","version":"405'77","reported_seq":495,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090665+0000","last_change":"2024-02-23T00:36:18.105686+0000","last_active":"2024-02-23T00:36:22.090665+0000","last_peered":"2024-02-23T00:36:22.090665+0000","last_clean":"2024-02-23T00:36:22.090665+0000","last_became_active":"2024-02-23T00:34:41.108381+0000","last_became_peered":"2024-02-23T00:34:41.108381+0000","last_unstale":"2024-02-23T00:36:22.090665+0000","last_undegraded":"2024-02-23T00:36:22.090665+0000","last_fullsized":"2024-02-23T00:36:22.090665+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'74","last_scrub_stamp":"2024-02-23T00:34:15.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.172848+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000073,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":3931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":4000057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.f","version":"219'69","reported_seq":354,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091599+0000","last_change":"2024-02-23T00:36:18.106107+0000","last_active":"2024-02-23T00:36:22.091599+0000","last_peered":"2024-02-23T00:36:22.091599+0000","last_clean":"2024-02-23T00:36:22.091599+0000","last_became_active":"2024-02-23T00:34:41.101718+0000","last_became_peered":"2024-02-23T00:34:41.101718+0000","last_unstale":"2024-02-23T00:36:22.091599+0000","last_undegraded":"2024-02-23T00:36:22.091599+0000","last_fullsized":"2024-02-23T00:36:22.091599+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'69","last_scrub_stamp":"2024-02-23T00:35:46.750022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.750022+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.10","version":"401'37","reported_seq":400,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088688+0000","last_change":"2024-02-23T00:36:18.104500+0000","last_active":"2024-02-23T00:36:22.088688+0000","last_peered":"2024-02-23T00:36:22.088688+0000","last_clean":"2024-02-23T00:36:22.088688+0000","last_became_active":"2024-02-23T00:31:06.342164+0000","last_became_peered":"2024-02-23T00:31:06.342164+0000","last_unstale":"2024-02-23T00:36:22.088688+0000","last_undegraded":"2024-02-23T00:36:22.088688+0000","last_fullsized":"2024-02-23T00:36:22.088688+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'34","last_scrub_stamp":"2024-02-23T00:35:01.791840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.791840+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.11","version":"392'50","reported_seq":413,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088877+0000","last_change":"2024-02-23T00:36:18.098819+0000","last_active":"2024-02-23T00:36:22.088877+0000","last_peered":"2024-02-23T00:36:22.088877+0000","last_clean":"2024-02-23T00:36:22.088877+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:36:22.088877+0000","last_undegraded":"2024-02-23T00:36:22.088877+0000","last_fullsized":"2024-02-23T00:36:22.088877+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'45","last_scrub_stamp":"2024-02-23T00:34:13.745366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.745366+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.28","version":"0'0","reported_seq":50,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156162+0000","last_change":"2024-02-23T00:35:29.092753+0000","last_active":"2024-02-23T00:36:23.156162+0000","last_peered":"2024-02-23T00:36:23.156162+0000","last_clean":"2024-02-23T00:36:23.156162+0000","last_became_active":"2024-02-23T00:35:29.092396+0000","last_became_peered":"2024-02-23T00:35:29.092396+0000","last_unstale":"2024-02-23T00:36:23.156162+0000","last_undegraded":"2024-02-23T00:36:23.156162+0000","last_fullsized":"2024-02-23T00:36:23.156162+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.815722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.815722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.12","version":"204'68","reported_seq":419,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061677+0000","last_change":"2024-02-23T00:36:18.095584+0000","last_active":"2024-02-23T00:36:23.061677+0000","last_peered":"2024-02-23T00:36:23.061677+0000","last_clean":"2024-02-23T00:36:23.061677+0000","last_became_active":"2024-02-23T00:31:06.281619+0000","last_became_peered":"2024-02-23T00:31:06.281619+0000","last_unstale":"2024-02-23T00:36:23.061677+0000","last_undegraded":"2024-02-23T00:36:23.061677+0000","last_fullsized":"2024-02-23T00:36:23.061677+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'68","last_scrub_stamp":"2024-02-23T00:33:46.399784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.399784+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.29","version":"62'8","reported_seq":52,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654660+0000","last_change":"2024-02-23T00:35:29.089861+0000","last_active":"2024-02-23T00:36:22.654660+0000","last_peered":"2024-02-23T00:36:22.654660+0000","last_clean":"2024-02-23T00:36:22.654660+0000","last_became_active":"2024-02-23T00:35:29.089450+0000","last_became_peered":"2024-02-23T00:35:29.089450+0000","last_unstale":"2024-02-23T00:36:22.654660+0000","last_undegraded":"2024-02-23T00:36:22.654660+0000","last_fullsized":"2024-02-23T00:36:22.654660+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:27.673481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.673481+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"63.13","version":"248'61","reported_seq":421,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157501+0000","last_change":"2024-02-23T00:36:18.519132+0000","last_active":"2024-02-23T00:36:23.157501+0000","last_peered":"2024-02-23T00:36:23.157501+0000","last_clean":"2024-02-23T00:36:23.157501+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:36:23.157501+0000","last_undegraded":"2024-02-23T00:36:23.157501+0000","last_fullsized":"2024-02-23T00:36:23.157501+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'61","last_scrub_stamp":"2024-02-23T00:33:41.498235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:41.498235+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.14","version":"208'85","reported_seq":444,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061770+0000","last_change":"2024-02-23T00:36:18.095179+0000","last_active":"2024-02-23T00:36:23.061770+0000","last_peered":"2024-02-23T00:36:23.061770+0000","last_clean":"2024-02-23T00:36:23.061770+0000","last_became_active":"2024-02-23T00:31:06.234521+0000","last_became_peered":"2024-02-23T00:31:06.234521+0000","last_unstale":"2024-02-23T00:36:23.061770+0000","last_undegraded":"2024-02-23T00:36:23.061770+0000","last_fullsized":"2024-02-23T00:36:23.061770+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'85","last_scrub_stamp":"2024-02-23T00:35:46.085280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.085280+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.15","version":"340'70","reported_seq":432,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089714+0000","last_change":"2024-02-23T00:36:18.099902+0000","last_active":"2024-02-23T00:36:22.089714+0000","last_peered":"2024-02-23T00:36:22.089714+0000","last_clean":"2024-02-23T00:36:22.089714+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:36:22.089714+0000","last_undegraded":"2024-02-23T00:36:22.089714+0000","last_fullsized":"2024-02-23T00:36:22.089714+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'68","last_scrub_stamp":"2024-02-23T00:34:21.681187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.681187+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.16","version":"204'51","reported_seq":421,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091605+0000","last_change":"2024-02-23T00:36:18.099907+0000","last_active":"2024-02-23T00:36:22.091605+0000","last_peered":"2024-02-23T00:36:22.091605+0000","last_clean":"2024-02-23T00:36:22.091605+0000","last_became_active":"2024-02-23T00:31:06.367069+0000","last_became_peered":"2024-02-23T00:31:06.367069+0000","last_unstale":"2024-02-23T00:36:22.091605+0000","last_undegraded":"2024-02-23T00:36:22.091605+0000","last_fullsized":"2024-02-23T00:36:22.091605+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:35:46.192706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.192706+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"63.17","version":"372'96","reported_seq":455,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159318+0000","last_change":"2024-02-23T00:36:18.280177+0000","last_active":"2024-02-23T00:36:23.159318+0000","last_peered":"2024-02-23T00:36:23.159318+0000","last_clean":"2024-02-23T00:36:23.159318+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:36:23.159318+0000","last_undegraded":"2024-02-23T00:36:23.159318+0000","last_fullsized":"2024-02-23T00:36:23.159318+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'86","last_scrub_stamp":"2024-02-23T00:33:35.383425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:35.383425+0000","log_size":96,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":81,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":31,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.22","version":"0'0","reported_seq":52,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156681+0000","last_change":"2024-02-23T00:35:29.089446+0000","last_active":"2024-02-23T00:36:23.156681+0000","last_peered":"2024-02-23T00:36:23.156681+0000","last_clean":"2024-02-23T00:36:23.156681+0000","last_became_active":"2024-02-23T00:35:29.089082+0000","last_became_peered":"2024-02-23T00:35:29.089082+0000","last_unstale":"2024-02-23T00:36:23.156681+0000","last_undegraded":"2024-02-23T00:36:23.156681+0000","last_fullsized":"2024-02-23T00:36:23.156681+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:58.218704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:58.218704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.18","version":"211'57","reported_seq":424,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156385+0000","last_change":"2024-02-23T00:36:18.521068+0000","last_active":"2024-02-23T00:36:23.156385+0000","last_peered":"2024-02-23T00:36:23.156385+0000","last_clean":"2024-02-23T00:36:23.156385+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:36:23.156385+0000","last_undegraded":"2024-02-23T00:36:23.156385+0000","last_fullsized":"2024-02-23T00:36:23.156385+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'57","last_scrub_stamp":"2024-02-23T00:35:54.857184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:54.857184+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.23","version":"379'10","reported_seq":70,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.117185+0000","last_change":"2024-02-23T00:35:29.096415+0000","last_active":"2024-02-23T00:36:24.117185+0000","last_peered":"2024-02-23T00:36:24.117185+0000","last_clean":"2024-02-23T00:36:24.117185+0000","last_became_active":"2024-02-23T00:35:29.086349+0000","last_became_peered":"2024-02-23T00:35:29.086349+0000","last_unstale":"2024-02-23T00:36:24.117185+0000","last_undegraded":"2024-02-23T00:36:24.117185+0000","last_fullsized":"2024-02-23T00:36:24.117185+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.19","version":"340'60","reported_seq":428,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088785+0000","last_change":"2024-02-23T00:36:18.098180+0000","last_active":"2024-02-23T00:36:22.088785+0000","last_peered":"2024-02-23T00:36:22.088785+0000","last_clean":"2024-02-23T00:36:22.088785+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:36:22.088785+0000","last_undegraded":"2024-02-23T00:36:22.088785+0000","last_fullsized":"2024-02-23T00:36:22.088785+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'54","last_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_deep_scrub":"271'54","last_deep_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.780899+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.20","version":"379'11","reported_seq":69,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.421971+0000","last_change":"2024-02-23T00:35:29.267709+0000","last_active":"2024-02-23T00:36:24.421971+0000","last_peered":"2024-02-23T00:36:24.421971+0000","last_clean":"2024-02-23T00:36:24.421971+0000","last_became_active":"2024-02-23T00:35:29.093220+0000","last_became_peered":"2024-02-23T00:35:29.093220+0000","last_unstale":"2024-02-23T00:36:24.421971+0000","last_undegraded":"2024-02-23T00:36:24.421971+0000","last_fullsized":"2024-02-23T00:36:24.421971+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1a","version":"202'68","reported_seq":431,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091160+0000","last_change":"2024-02-23T00:36:18.099251+0000","last_active":"2024-02-23T00:36:22.091160+0000","last_peered":"2024-02-23T00:36:22.091160+0000","last_clean":"2024-02-23T00:36:22.091160+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:36:22.091160+0000","last_undegraded":"2024-02-23T00:36:22.091160+0000","last_fullsized":"2024-02-23T00:36:22.091160+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'68","last_scrub_stamp":"2024-02-23T00:35:39.110950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.110950+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.21","version":"0'0","reported_seq":51,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.060859+0000","last_change":"2024-02-23T00:35:29.093481+0000","last_active":"2024-02-23T00:36:23.060859+0000","last_peered":"2024-02-23T00:36:23.060859+0000","last_clean":"2024-02-23T00:36:23.060859+0000","last_became_active":"2024-02-23T00:35:29.093053+0000","last_became_peered":"2024-02-23T00:35:29.093053+0000","last_unstale":"2024-02-23T00:36:23.060859+0000","last_undegraded":"2024-02-23T00:36:23.060859+0000","last_fullsized":"2024-02-23T00:36:23.060859+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.588132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.588132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.1b","version":"398'54","reported_seq":415,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157605+0000","last_change":"2024-02-23T00:36:18.278918+0000","last_active":"2024-02-23T00:36:23.157605+0000","last_peered":"2024-02-23T00:36:23.157605+0000","last_clean":"2024-02-23T00:36:23.157605+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:36:23.157605+0000","last_undegraded":"2024-02-23T00:36:23.157605+0000","last_fullsized":"2024-02-23T00:36:23.157605+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-02-23T00:33:46.274400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.274400+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.26","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158715+0000","last_change":"2024-02-23T00:35:29.094041+0000","last_active":"2024-02-23T00:36:23.158715+0000","last_peered":"2024-02-23T00:36:23.158715+0000","last_clean":"2024-02-23T00:36:23.158715+0000","last_became_active":"2024-02-23T00:35:29.093626+0000","last_became_peered":"2024-02-23T00:35:29.093626+0000","last_unstale":"2024-02-23T00:36:23.158715+0000","last_undegraded":"2024-02-23T00:36:23.158715+0000","last_fullsized":"2024-02-23T00:36:23.158715+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1c","version":"216'34","reported_seq":427,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089252+0000","last_change":"2024-02-23T00:36:18.105634+0000","last_active":"2024-02-23T00:36:22.089252+0000","last_peered":"2024-02-23T00:36:22.089252+0000","last_clean":"2024-02-23T00:36:22.089252+0000","last_became_active":"2024-02-23T00:34:41.117846+0000","last_became_peered":"2024-02-23T00:34:41.117846+0000","last_unstale":"2024-02-23T00:36:22.089252+0000","last_undegraded":"2024-02-23T00:36:22.089252+0000","last_fullsized":"2024-02-23T00:36:22.089252+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'34","last_scrub_stamp":"2024-02-23T00:34:57.859047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.859047+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.27","version":"0'0","reported_seq":389,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091081+0000","last_change":"2024-02-23T00:35:29.092282+0000","last_active":"2024-02-23T00:36:22.091081+0000","last_peered":"2024-02-23T00:36:22.091081+0000","last_clean":"2024-02-23T00:36:22.091081+0000","last_became_active":"2024-02-23T00:35:29.091970+0000","last_became_peered":"2024-02-23T00:35:29.091970+0000","last_unstale":"2024-02-23T00:36:22.091081+0000","last_undegraded":"2024-02-23T00:36:22.091081+0000","last_fullsized":"2024-02-23T00:36:22.091081+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1d","version":"340'13","reported_seq":120,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061729+0000","last_change":"2024-02-23T00:36:18.095074+0000","last_active":"2024-02-23T00:36:23.061729+0000","last_peered":"2024-02-23T00:36:23.061729+0000","last_clean":"2024-02-23T00:36:23.061729+0000","last_became_active":"2024-02-23T00:34:42.058724+0000","last_became_peered":"2024-02-23T00:34:42.058724+0000","last_unstale":"2024-02-23T00:36:23.061729+0000","last_undegraded":"2024-02-23T00:36:23.061729+0000","last_fullsized":"2024-02-23T00:36:23.061729+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-02-23T00:34:24.997695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.997695+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.24","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158818+0000","last_change":"2024-02-23T00:35:29.090903+0000","last_active":"2024-02-23T00:36:23.158818+0000","last_peered":"2024-02-23T00:36:23.158818+0000","last_clean":"2024-02-23T00:36:23.158818+0000","last_became_active":"2024-02-23T00:35:29.090456+0000","last_became_peered":"2024-02-23T00:35:29.090456+0000","last_unstale":"2024-02-23T00:36:23.158818+0000","last_undegraded":"2024-02-23T00:36:23.158818+0000","last_fullsized":"2024-02-23T00:36:23.158818+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1e","version":"340'52","reported_seq":409,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159103+0000","last_change":"2024-02-23T00:36:18.278128+0000","last_active":"2024-02-23T00:36:23.159103+0000","last_peered":"2024-02-23T00:36:23.159103+0000","last_clean":"2024-02-23T00:36:23.159103+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:36:23.159103+0000","last_undegraded":"2024-02-23T00:36:23.159103+0000","last_fullsized":"2024-02-23T00:36:23.159103+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'51","last_scrub_stamp":"2024-02-23T00:33:49.219607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.219607+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.25","version":"0'0","reported_seq":398,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088926+0000","last_change":"2024-02-23T00:35:29.091129+0000","last_active":"2024-02-23T00:36:22.088926+0000","last_peered":"2024-02-23T00:36:22.088926+0000","last_clean":"2024-02-23T00:36:22.088926+0000","last_became_active":"2024-02-23T00:35:29.090793+0000","last_became_peered":"2024-02-23T00:35:29.090793+0000","last_unstale":"2024-02-23T00:36:22.088926+0000","last_undegraded":"2024-02-23T00:36:22.088926+0000","last_fullsized":"2024-02-23T00:36:22.088926+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1f","version":"228'68","reported_seq":428,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159221+0000","last_change":"2024-02-23T00:36:18.280207+0000","last_active":"2024-02-23T00:36:23.159221+0000","last_peered":"2024-02-23T00:36:23.159221+0000","last_clean":"2024-02-23T00:36:23.159221+0000","last_became_active":"2024-02-23T00:31:06.355623+0000","last_became_peered":"2024-02-23T00:31:06.355623+0000","last_unstale":"2024-02-23T00:36:23.159221+0000","last_undegraded":"2024-02-23T00:36:23.159221+0000","last_fullsized":"2024-02-23T00:36:23.159221+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-23T00:33:39.351573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.351573+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"}]},{"pgid":"5.9","version":"379'9","reported_seq":516,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.420864+0000","last_change":"2024-02-23T00:35:27.095545+0000","last_active":"2024-02-23T00:36:24.420864+0000","last_peered":"2024-02-23T00:36:24.420864+0000","last_clean":"2024-02-23T00:36:24.420864+0000","last_became_active":"2024-02-23T00:35:27.095085+0000","last_became_peered":"2024-02-23T00:35:27.095085+0000","last_unstale":"2024-02-23T00:36:24.420864+0000","last_undegraded":"2024-02-23T00:36:24.420864+0000","last_fullsized":"2024-02-23T00:36:24.420864+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:27.673481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.673481+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"81'20","reported_seq":463,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156461+0000","last_change":"2024-02-23T00:35:13.075104+0000","last_active":"2024-02-23T00:36:23.156461+0000","last_peered":"2024-02-23T00:36:23.156461+0000","last_clean":"2024-02-23T00:36:23.156461+0000","last_became_active":"2024-02-23T00:35:13.074767+0000","last_became_peered":"2024-02-23T00:35:13.074767+0000","last_unstale":"2024-02-23T00:36:23.156461+0000","last_undegraded":"2024-02-23T00:36:23.156461+0000","last_fullsized":"2024-02-23T00:36:23.156461+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'36","last_scrub_stamp":"2024-02-23T00:32:47.376691+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:32:47.376691+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":79,"num_read_kb":61,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":408,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091887+0000","last_change":"2024-02-23T00:35:05.018567+0000","last_active":"2024-02-23T00:36:22.091887+0000","last_peered":"2024-02-23T00:36:22.091887+0000","last_clean":"2024-02-23T00:36:22.091887+0000","last_became_active":"2024-02-23T00:34:41.106070+0000","last_became_peered":"2024-02-23T00:34:41.106070+0000","last_unstale":"2024-02-23T00:36:22.091887+0000","last_undegraded":"2024-02-23T00:36:22.091887+0000","last_fullsized":"2024-02-23T00:36:22.091887+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:35:05.018499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":377,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157195+0000","last_change":"2024-02-23T00:34:21.442023+0000","last_active":"2024-02-23T00:36:23.157195+0000","last_peered":"2024-02-23T00:36:23.157195+0000","last_clean":"2024-02-23T00:36:23.157195+0000","last_became_active":"2024-02-23T00:27:23.180836+0000","last_became_peered":"2024-02-23T00:27:23.180836+0000","last_unstale":"2024-02-23T00:36:23.157195+0000","last_undegraded":"2024-02-23T00:36:23.157195+0000","last_fullsized":"2024-02-23T00:36:23.157195+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:21.441958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.441958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":341,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090331+0000","last_change":"2024-02-23T00:34:42.061455+0000","last_active":"2024-02-23T00:36:22.090331+0000","last_peered":"2024-02-23T00:36:22.090331+0000","last_clean":"2024-02-23T00:36:22.090331+0000","last_became_active":"2024-02-23T00:34:42.059196+0000","last_became_peered":"2024-02-23T00:34:42.059196+0000","last_unstale":"2024-02-23T00:36:22.090331+0000","last_undegraded":"2024-02-23T00:36:22.090331+0000","last_fullsized":"2024-02-23T00:36:22.090331+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":607,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.420302+0000","last_change":"2024-02-23T00:34:41.145462+0000","last_active":"2024-02-23T00:36:24.420302+0000","last_peered":"2024-02-23T00:36:24.420302+0000","last_clean":"2024-02-23T00:36:24.420302+0000","last_became_active":"2024-02-23T00:34:41.114058+0000","last_became_peered":"2024-02-23T00:34:41.114058+0000","last_unstale":"2024-02-23T00:36:24.420302+0000","last_undegraded":"2024-02-23T00:36:24.420302+0000","last_fullsized":"2024-02-23T00:36:24.420302+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:37.440614+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:37.440614+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":227,"num_read_kb":227,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"78'14","reported_seq":439,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.655665+0000","last_change":"2024-02-23T00:34:55.073114+0000","last_active":"2024-02-23T00:36:22.655665+0000","last_peered":"2024-02-23T00:36:22.655665+0000","last_clean":"2024-02-23T00:36:22.655665+0000","last_became_active":"2024-02-23T00:34:55.072740+0000","last_became_peered":"2024-02-23T00:34:55.072740+0000","last_unstale":"2024-02-23T00:36:22.655665+0000","last_undegraded":"2024-02-23T00:36:22.655665+0000","last_fullsized":"2024-02-23T00:36:22.655665+0000","mapping_epoch":56,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":28,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":394,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091698+0000","last_change":"2024-02-23T00:34:41.100552+0000","last_active":"2024-02-23T00:36:22.091698+0000","last_peered":"2024-02-23T00:36:22.091698+0000","last_clean":"2024-02-23T00:36:22.091698+0000","last_became_active":"2024-02-23T00:34:41.100003+0000","last_became_peered":"2024-02-23T00:34:41.100003+0000","last_unstale":"2024-02-23T00:36:22.091698+0000","last_undegraded":"2024-02-23T00:36:22.091698+0000","last_fullsized":"2024-02-23T00:36:22.091698+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:38.328237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:38.328237+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091295+0000","last_change":"2024-02-23T00:34:41.096223+0000","last_active":"2024-02-23T00:36:22.091295+0000","last_peered":"2024-02-23T00:36:22.091295+0000","last_clean":"2024-02-23T00:36:22.091295+0000","last_became_active":"2024-02-23T00:34:41.095851+0000","last_became_peered":"2024-02-23T00:34:41.095851+0000","last_unstale":"2024-02-23T00:36:22.091295+0000","last_undegraded":"2024-02-23T00:36:22.091295+0000","last_fullsized":"2024-02-23T00:36:22.091295+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.679076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":386,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654986+0000","last_change":"2024-02-23T00:33:47.610285+0000","last_active":"2024-02-23T00:36:22.654986+0000","last_peered":"2024-02-23T00:36:22.654986+0000","last_clean":"2024-02-23T00:36:22.654986+0000","last_became_active":"2024-02-23T00:27:23.025608+0000","last_became_peered":"2024-02-23T00:27:23.025608+0000","last_unstale":"2024-02-23T00:36:22.654986+0000","last_undegraded":"2024-02-23T00:36:22.654986+0000","last_fullsized":"2024-02-23T00:36:22.654986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":338,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061412+0000","last_change":"2024-02-23T00:35:54.029484+0000","last_active":"2024-02-23T00:36:23.061412+0000","last_peered":"2024-02-23T00:36:23.061412+0000","last_clean":"2024-02-23T00:36:23.061412+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:36:23.061412+0000","last_undegraded":"2024-02-23T00:36:23.061412+0000","last_fullsized":"2024-02-23T00:36:23.061412+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:54.029412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:54.029412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"63'25","reported_seq":453,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061782+0000","last_change":"2024-02-23T00:34:59.084974+0000","last_active":"2024-02-23T00:36:23.061782+0000","last_peered":"2024-02-23T00:36:23.061782+0000","last_clean":"2024-02-23T00:36:23.061782+0000","last_became_active":"2024-02-23T00:34:59.084636+0000","last_became_peered":"2024-02-23T00:34:59.084636+0000","last_unstale":"2024-02-23T00:36:23.061782+0000","last_undegraded":"2024-02-23T00:36:23.061782+0000","last_fullsized":"2024-02-23T00:36:23.061782+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:34:16.310110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:16.310110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":83,"num_read_kb":63,"num_write":48,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.092171+0000","last_change":"2024-02-23T00:35:50.194747+0000","last_active":"2024-02-23T00:36:22.092171+0000","last_peered":"2024-02-23T00:36:22.092171+0000","last_clean":"2024-02-23T00:36:22.092171+0000","last_became_active":"2024-02-23T00:27:17.011633+0000","last_became_peered":"2024-02-23T00:27:17.011633+0000","last_unstale":"2024-02-23T00:36:22.092171+0000","last_undegraded":"2024-02-23T00:36:22.092171+0000","last_fullsized":"2024-02-23T00:36:22.092171+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:50.194656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:50.194656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":390,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091780+0000","last_change":"2024-02-23T00:34:28.745964+0000","last_active":"2024-02-23T00:36:22.091780+0000","last_peered":"2024-02-23T00:36:22.091780+0000","last_clean":"2024-02-23T00:36:22.091780+0000","last_became_active":"2024-02-23T00:27:21.024317+0000","last_became_peered":"2024-02-23T00:27:21.024317+0000","last_unstale":"2024-02-23T00:36:22.091780+0000","last_undegraded":"2024-02-23T00:36:22.091780+0000","last_fullsized":"2024-02-23T00:36:22.091780+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.745865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.745865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156661+0000","last_change":"2024-02-23T00:35:04.673936+0000","last_active":"2024-02-23T00:36:23.156661+0000","last_peered":"2024-02-23T00:36:23.156661+0000","last_clean":"2024-02-23T00:36:23.156661+0000","last_became_active":"2024-02-23T00:34:41.105277+0000","last_became_peered":"2024-02-23T00:34:41.105277+0000","last_unstale":"2024-02-23T00:36:23.156661+0000","last_undegraded":"2024-02-23T00:36:23.156661+0000","last_fullsized":"2024-02-23T00:36:23.156661+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.673837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.673837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":339,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654960+0000","last_change":"2024-02-23T00:35:39.205378+0000","last_active":"2024-02-23T00:36:22.654960+0000","last_peered":"2024-02-23T00:36:22.654960+0000","last_clean":"2024-02-23T00:36:22.654960+0000","last_became_active":"2024-02-23T00:30:48.673452+0000","last_became_peered":"2024-02-23T00:30:48.673452+0000","last_unstale":"2024-02-23T00:36:22.654960+0000","last_undegraded":"2024-02-23T00:36:22.654960+0000","last_fullsized":"2024-02-23T00:36:22.654960+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.205312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.205312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":400,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061993+0000","last_change":"2024-02-23T00:34:29.233679+0000","last_active":"2024-02-23T00:36:23.061993+0000","last_peered":"2024-02-23T00:36:23.061993+0000","last_clean":"2024-02-23T00:36:23.061993+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:36:23.061993+0000","last_undegraded":"2024-02-23T00:36:23.061993+0000","last_fullsized":"2024-02-23T00:36:23.061993+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:29.233574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:29.233574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"76'24","reported_seq":438,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:25.082682+0000","last_change":"2024-02-23T00:36:25.082682+0000","last_active":"2024-02-23T00:36:25.082682+0000","last_peered":"2024-02-23T00:36:25.082682+0000","last_clean":"2024-02-23T00:36:25.082682+0000","last_became_active":"2024-02-23T00:34:47.062796+0000","last_became_peered":"2024-02-23T00:34:47.062796+0000","last_unstale":"2024-02-23T00:36:25.082682+0000","last_undegraded":"2024-02-23T00:36:25.082682+0000","last_fullsized":"2024-02-23T00:36:25.082682+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":46,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":392,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157781+0000","last_change":"2024-02-23T00:35:40.824848+0000","last_active":"2024-02-23T00:36:23.157781+0000","last_peered":"2024-02-23T00:36:23.157781+0000","last_clean":"2024-02-23T00:36:23.157781+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:36:23.157781+0000","last_undegraded":"2024-02-23T00:36:23.157781+0000","last_fullsized":"2024-02-23T00:36:23.157781+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:40.824741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:40.824741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":376,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089519+0000","last_change":"2024-02-23T00:32:09.110225+0000","last_active":"2024-02-23T00:36:22.089519+0000","last_peered":"2024-02-23T00:36:22.089519+0000","last_clean":"2024-02-23T00:36:22.089519+0000","last_became_active":"2024-02-23T00:27:21.024248+0000","last_became_peered":"2024-02-23T00:27:21.024248+0000","last_unstale":"2024-02-23T00:36:22.089519+0000","last_undegraded":"2024-02-23T00:36:22.089519+0000","last_fullsized":"2024-02-23T00:36:22.089519+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:09.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:09.110138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":404,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090825+0000","last_change":"2024-02-23T00:35:54.157269+0000","last_active":"2024-02-23T00:36:22.090825+0000","last_peered":"2024-02-23T00:36:22.090825+0000","last_clean":"2024-02-23T00:36:22.090825+0000","last_became_active":"2024-02-23T00:34:41.090777+0000","last_became_peered":"2024-02-23T00:34:41.090777+0000","last_unstale":"2024-02-23T00:36:22.090825+0000","last_undegraded":"2024-02-23T00:36:22.090825+0000","last_fullsized":"2024-02-23T00:36:22.090825+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:54.157208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:35:54.157208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":349,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158785+0000","last_change":"2024-02-23T00:35:10.704557+0000","last_active":"2024-02-23T00:36:23.158785+0000","last_peered":"2024-02-23T00:36:23.158785+0000","last_clean":"2024-02-23T00:36:23.158785+0000","last_became_active":"2024-02-23T00:34:41.114555+0000","last_became_peered":"2024-02-23T00:34:41.114555+0000","last_unstale":"2024-02-23T00:36:23.158785+0000","last_undegraded":"2024-02-23T00:36:23.158785+0000","last_fullsized":"2024-02-23T00:36:23.158785+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.704478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.704478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":212,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.655817+0000","last_change":"2024-02-23T00:35:47.196284+0000","last_active":"2024-02-23T00:36:22.655817+0000","last_peered":"2024-02-23T00:36:22.655817+0000","last_clean":"2024-02-23T00:36:22.655817+0000","last_became_active":"2024-02-23T00:34:41.108822+0000","last_became_peered":"2024-02-23T00:34:41.108822+0000","last_unstale":"2024-02-23T00:36:22.655817+0000","last_undegraded":"2024-02-23T00:36:22.655817+0000","last_fullsized":"2024-02-23T00:36:22.655817+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"62'15","reported_seq":441,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089951+0000","last_change":"2024-02-23T00:34:51.102561+0000","last_active":"2024-02-23T00:36:22.089951+0000","last_peered":"2024-02-23T00:36:22.089951+0000","last_clean":"2024-02-23T00:36:22.089951+0000","last_became_active":"2024-02-23T00:34:51.102216+0000","last_became_peered":"2024-02-23T00:34:51.102216+0000","last_unstale":"2024-02-23T00:36:22.089951+0000","last_undegraded":"2024-02-23T00:36:22.089951+0000","last_fullsized":"2024-02-23T00:36:22.089951+0000","mapping_epoch":329,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'10","last_scrub_stamp":"2024-02-23T00:34:22.638641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.638641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":26,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":412,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090039+0000","last_change":"2024-02-23T00:35:56.678738+0000","last_active":"2024-02-23T00:36:22.090039+0000","last_peered":"2024-02-23T00:36:22.090039+0000","last_clean":"2024-02-23T00:36:22.090039+0000","last_became_active":"2024-02-23T00:34:41.115434+0000","last_became_peered":"2024-02-23T00:34:41.115434+0000","last_unstale":"2024-02-23T00:36:22.090039+0000","last_undegraded":"2024-02-23T00:36:22.090039+0000","last_fullsized":"2024-02-23T00:36:22.090039+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.678676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.678676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":508,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.118056+0000","last_change":"2024-02-23T00:34:57.950514+0000","last_active":"2024-02-23T00:36:24.118056+0000","last_peered":"2024-02-23T00:36:24.118056+0000","last_clean":"2024-02-23T00:36:24.118056+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:36:24.118056+0000","last_undegraded":"2024-02-23T00:36:24.118056+0000","last_fullsized":"2024-02-23T00:36:24.118056+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:57.950450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.950450+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090498+0000","last_change":"2024-02-23T00:35:56.258132+0000","last_active":"2024-02-23T00:36:22.090498+0000","last_peered":"2024-02-23T00:36:22.090498+0000","last_clean":"2024-02-23T00:36:22.090498+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:36:22.090498+0000","last_undegraded":"2024-02-23T00:36:22.090498+0000","last_fullsized":"2024-02-23T00:36:22.090498+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.258056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.258056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":341,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090555+0000","last_change":"2024-02-23T00:35:46.175575+0000","last_active":"2024-02-23T00:36:22.090555+0000","last_peered":"2024-02-23T00:36:22.090555+0000","last_clean":"2024-02-23T00:36:22.090555+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:36:22.090555+0000","last_undegraded":"2024-02-23T00:36:22.090555+0000","last_fullsized":"2024-02-23T00:36:22.090555+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:46.175476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.175476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":455,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088579+0000","last_change":"2024-02-23T00:34:41.104353+0000","last_active":"2024-02-23T00:36:22.088579+0000","last_peered":"2024-02-23T00:36:22.088579+0000","last_clean":"2024-02-23T00:36:22.088579+0000","last_became_active":"2024-02-23T00:34:41.103369+0000","last_became_peered":"2024-02-23T00:34:41.103369+0000","last_unstale":"2024-02-23T00:36:22.088579+0000","last_undegraded":"2024-02-23T00:36:22.088579+0000","last_fullsized":"2024-02-23T00:36:22.088579+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:58.975334+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:58.975334+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":59,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"87'22","reported_seq":494,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091644+0000","last_change":"2024-02-23T00:35:47.079899+0000","last_active":"2024-02-23T00:36:22.091644+0000","last_peered":"2024-02-23T00:36:22.091644+0000","last_clean":"2024-02-23T00:36:22.091644+0000","last_became_active":"2024-02-23T00:35:47.079526+0000","last_became_peered":"2024-02-23T00:35:47.079526+0000","last_unstale":"2024-02-23T00:36:22.091644+0000","last_undegraded":"2024-02-23T00:36:22.091644+0000","last_fullsized":"2024-02-23T00:36:22.091644+0000","mapping_epoch":392,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:32:40.327519+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.327519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":56,"num_read_kb":39,"num_write":34,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":396,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088829+0000","last_change":"2024-02-23T00:35:06.658103+0000","last_active":"2024-02-23T00:36:22.088829+0000","last_peered":"2024-02-23T00:36:22.088829+0000","last_clean":"2024-02-23T00:36:22.088829+0000","last_became_active":"2024-02-23T00:34:41.107670+0000","last_became_peered":"2024-02-23T00:34:41.107670+0000","last_unstale":"2024-02-23T00:36:22.088829+0000","last_undegraded":"2024-02-23T00:36:22.088829+0000","last_fullsized":"2024-02-23T00:36:22.088829+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.657998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"62'8","reported_seq":508,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.116861+0000","last_change":"2024-02-23T00:34:59.630249+0000","last_active":"2024-02-23T00:36:24.116861+0000","last_peered":"2024-02-23T00:36:24.116861+0000","last_clean":"2024-02-23T00:36:24.116861+0000","last_became_active":"2024-02-23T00:27:21.020301+0000","last_became_peered":"2024-02-23T00:27:21.020301+0000","last_unstale":"2024-02-23T00:36:24.116861+0000","last_undegraded":"2024-02-23T00:36:24.116861+0000","last_fullsized":"2024-02-23T00:36:24.116861+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:59.630171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:59.630171+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":378,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158578+0000","last_change":"2024-02-23T00:34:17.437153+0000","last_active":"2024-02-23T00:36:23.158578+0000","last_peered":"2024-02-23T00:36:23.158578+0000","last_clean":"2024-02-23T00:36:23.158578+0000","last_became_active":"2024-02-23T00:27:23.033213+0000","last_became_peered":"2024-02-23T00:27:23.033213+0000","last_unstale":"2024-02-23T00:36:23.158578+0000","last_undegraded":"2024-02-23T00:36:23.158578+0000","last_fullsized":"2024-02-23T00:36:23.158578+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:34:17.437083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.437083+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":332,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089184+0000","last_change":"2024-02-23T00:33:39.140496+0000","last_active":"2024-02-23T00:36:22.089184+0000","last_peered":"2024-02-23T00:36:22.089184+0000","last_clean":"2024-02-23T00:36:22.089184+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:36:22.089184+0000","last_undegraded":"2024-02-23T00:36:22.089184+0000","last_fullsized":"2024-02-23T00:36:22.089184+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:39.140434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.140434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":403,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088734+0000","last_change":"2024-02-23T00:35:01.614406+0000","last_active":"2024-02-23T00:36:22.088734+0000","last_peered":"2024-02-23T00:36:22.088734+0000","last_clean":"2024-02-23T00:36:22.088734+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:36:22.088734+0000","last_undegraded":"2024-02-23T00:36:22.088734+0000","last_fullsized":"2024-02-23T00:36:22.088734+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:01.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.614340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"92'22","reported_seq":450,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090291+0000","last_change":"2024-02-23T00:35:10.593944+0000","last_active":"2024-02-23T00:36:22.090291+0000","last_peered":"2024-02-23T00:36:22.090291+0000","last_clean":"2024-02-23T00:36:22.090291+0000","last_became_active":"2024-02-23T00:34:43.058886+0000","last_became_peered":"2024-02-23T00:34:43.058886+0000","last_unstale":"2024-02-23T00:36:22.090291+0000","last_undegraded":"2024-02-23T00:36:22.090291+0000","last_fullsized":"2024-02-23T00:36:22.090291+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:35:10.593862+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.593862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157007+0000","last_change":"2024-02-23T00:33:50.601581+0000","last_active":"2024-02-23T00:36:23.157007+0000","last_peered":"2024-02-23T00:36:23.157007+0000","last_clean":"2024-02-23T00:36:23.157007+0000","last_became_active":"2024-02-23T00:27:17.012500+0000","last_became_peered":"2024-02-23T00:27:17.012500+0000","last_unstale":"2024-02-23T00:36:23.157007+0000","last_undegraded":"2024-02-23T00:36:23.157007+0000","last_fullsized":"2024-02-23T00:36:23.157007+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":379,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159706+0000","last_change":"2024-02-23T00:34:22.475022+0000","last_active":"2024-02-23T00:36:23.159706+0000","last_peered":"2024-02-23T00:36:23.159706+0000","last_clean":"2024-02-23T00:36:23.159706+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:36:23.159706+0000","last_undegraded":"2024-02-23T00:36:23.159706+0000","last_fullsized":"2024-02-23T00:36:23.159706+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.474956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.474956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":108,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.062170+0000","last_change":"2024-02-23T00:34:56.232523+0000","last_active":"2024-02-23T00:36:23.062170+0000","last_peered":"2024-02-23T00:36:23.062170+0000","last_clean":"2024-02-23T00:36:23.062170+0000","last_became_active":"2024-02-23T00:34:41.116969+0000","last_became_peered":"2024-02-23T00:34:41.116969+0000","last_unstale":"2024-02-23T00:36:23.062170+0000","last_undegraded":"2024-02-23T00:36:23.062170+0000","last_fullsized":"2024-02-23T00:36:23.062170+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.232459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.232459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":341,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089663+0000","last_change":"2024-02-23T00:34:41.112576+0000","last_active":"2024-02-23T00:36:22.089663+0000","last_peered":"2024-02-23T00:36:22.089663+0000","last_clean":"2024-02-23T00:36:22.089663+0000","last_became_active":"2024-02-23T00:34:41.109539+0000","last_became_peered":"2024-02-23T00:34:41.109539+0000","last_unstale":"2024-02-23T00:36:22.089663+0000","last_undegraded":"2024-02-23T00:36:22.089663+0000","last_fullsized":"2024-02-23T00:36:22.089663+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:53.387496+0000","last_clean_scrub_stamp":"2024-02-23T00:32:53.387496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":305,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.656325+0000","last_change":"2024-02-23T00:35:42.114874+0000","last_active":"2024-02-23T00:36:22.656325+0000","last_peered":"2024-02-23T00:36:22.656325+0000","last_clean":"2024-02-23T00:36:22.656325+0000","last_became_active":"2024-02-23T00:31:29.108747+0000","last_became_peered":"2024-02-23T00:31:29.108747+0000","last_unstale":"2024-02-23T00:36:22.656325+0000","last_undegraded":"2024-02-23T00:36:22.656325+0000","last_fullsized":"2024-02-23T00:36:22.656325+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:42.114807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:35:42.114807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":143,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088851+0000","last_change":"2024-02-23T00:35:27.096000+0000","last_active":"2024-02-23T00:36:22.088851+0000","last_peered":"2024-02-23T00:36:22.088851+0000","last_clean":"2024-02-23T00:36:22.088851+0000","last_became_active":"2024-02-23T00:35:27.095561+0000","last_became_peered":"2024-02-23T00:35:27.095561+0000","last_unstale":"2024-02-23T00:36:22.088851+0000","last_undegraded":"2024-02-23T00:36:22.088851+0000","last_fullsized":"2024-02-23T00:36:22.088851+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"63'20","reported_seq":423,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089146+0000","last_change":"2024-02-23T00:34:31.622831+0000","last_active":"2024-02-23T00:36:22.089146+0000","last_peered":"2024-02-23T00:36:22.089146+0000","last_clean":"2024-02-23T00:36:22.089146+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:36:22.089146+0000","last_undegraded":"2024-02-23T00:36:22.089146+0000","last_fullsized":"2024-02-23T00:36:22.089146+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:34:31.622761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.622761+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":390,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.062322+0000","last_change":"2024-02-23T00:35:41.071319+0000","last_active":"2024-02-23T00:36:23.062322+0000","last_peered":"2024-02-23T00:36:23.062322+0000","last_clean":"2024-02-23T00:36:23.062322+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:36:23.062322+0000","last_undegraded":"2024-02-23T00:36:23.062322+0000","last_fullsized":"2024-02-23T00:36:23.062322+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:41.071250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.071250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":386,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158289+0000","last_change":"2024-02-23T00:33:45.587063+0000","last_active":"2024-02-23T00:36:23.158289+0000","last_peered":"2024-02-23T00:36:23.158289+0000","last_clean":"2024-02-23T00:36:23.158289+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:36:23.158289+0000","last_undegraded":"2024-02-23T00:36:23.158289+0000","last_fullsized":"2024-02-23T00:36:23.158289+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:45.586975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.586975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":341,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090127+0000","last_change":"2024-02-23T00:34:41.104095+0000","last_active":"2024-02-23T00:36:22.090127+0000","last_peered":"2024-02-23T00:36:22.090127+0000","last_clean":"2024-02-23T00:36:22.090127+0000","last_became_active":"2024-02-23T00:34:41.103792+0000","last_became_peered":"2024-02-23T00:34:41.103792+0000","last_unstale":"2024-02-23T00:36:22.090127+0000","last_undegraded":"2024-02-23T00:36:22.090127+0000","last_fullsized":"2024-02-23T00:36:22.090127+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.019530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.019530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":209,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158262+0000","last_change":"2024-02-23T00:35:52.924868+0000","last_active":"2024-02-23T00:36:23.158262+0000","last_peered":"2024-02-23T00:36:23.158262+0000","last_clean":"2024-02-23T00:36:23.158262+0000","last_became_active":"2024-02-23T00:34:41.096644+0000","last_became_peered":"2024-02-23T00:34:41.096644+0000","last_unstale":"2024-02-23T00:36:23.158262+0000","last_undegraded":"2024-02-23T00:36:23.158262+0000","last_fullsized":"2024-02-23T00:36:23.158262+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.924808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.924808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":408,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089290+0000","last_change":"2024-02-23T00:35:27.094575+0000","last_active":"2024-02-23T00:36:22.089290+0000","last_peered":"2024-02-23T00:36:22.089290+0000","last_clean":"2024-02-23T00:36:22.089290+0000","last_became_active":"2024-02-23T00:35:27.093809+0000","last_became_peered":"2024-02-23T00:35:27.093809+0000","last_unstale":"2024-02-23T00:36:22.089290+0000","last_undegraded":"2024-02-23T00:36:22.089290+0000","last_fullsized":"2024-02-23T00:36:22.089290+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.588132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.588132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"63'30","reported_seq":206,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157308+0000","last_change":"2024-02-23T00:35:00.621227+0000","last_active":"2024-02-23T00:36:23.157308+0000","last_peered":"2024-02-23T00:36:23.157308+0000","last_clean":"2024-02-23T00:36:23.157308+0000","last_became_active":"2024-02-23T00:34:41.110156+0000","last_became_peered":"2024-02-23T00:34:41.110156+0000","last_unstale":"2024-02-23T00:36:23.157308+0000","last_undegraded":"2024-02-23T00:36:23.157308+0000","last_fullsized":"2024-02-23T00:36:23.157308+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:35:00.621157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.621157+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":56,"num_read_kb":42,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":385,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090794+0000","last_change":"2024-02-23T00:33:34.199948+0000","last_active":"2024-02-23T00:36:22.090794+0000","last_peered":"2024-02-23T00:36:22.090794+0000","last_clean":"2024-02-23T00:36:22.090794+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:36:22.090794+0000","last_undegraded":"2024-02-23T00:36:22.090794+0000","last_fullsized":"2024-02-23T00:36:22.090794+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.199869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.199869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":397,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089478+0000","last_change":"2024-02-23T00:34:58.660219+0000","last_active":"2024-02-23T00:36:22.089478+0000","last_peered":"2024-02-23T00:36:22.089478+0000","last_clean":"2024-02-23T00:36:22.089478+0000","last_became_active":"2024-02-23T00:34:41.094428+0000","last_became_peered":"2024-02-23T00:34:41.094428+0000","last_unstale":"2024-02-23T00:36:22.089478+0000","last_undegraded":"2024-02-23T00:36:22.089478+0000","last_fullsized":"2024-02-23T00:36:22.089478+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:58.660111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:58.660111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":102,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091473+0000","last_change":"2024-02-23T00:34:41.106716+0000","last_active":"2024-02-23T00:36:22.091473+0000","last_peered":"2024-02-23T00:36:22.091473+0000","last_clean":"2024-02-23T00:36:22.091473+0000","last_became_active":"2024-02-23T00:34:41.105998+0000","last_became_peered":"2024-02-23T00:34:41.105998+0000","last_unstale":"2024-02-23T00:36:22.091473+0000","last_undegraded":"2024-02-23T00:36:22.091473+0000","last_fullsized":"2024-02-23T00:36:22.091473+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:31.674319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.674319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":648,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089178+0000","last_change":"2024-02-23T00:32:43.323030+0000","last_active":"2024-02-23T00:36:22.089178+0000","last_peered":"2024-02-23T00:36:22.089178+0000","last_clean":"2024-02-23T00:36:22.089178+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:36:22.089178+0000","last_undegraded":"2024-02-23T00:36:22.089178+0000","last_fullsized":"2024-02-23T00:36:22.089178+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:43.322963+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:43.322963+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":396,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061783+0000","last_change":"2024-02-23T00:35:27.098571+0000","last_active":"2024-02-23T00:36:23.061783+0000","last_peered":"2024-02-23T00:36:23.061783+0000","last_clean":"2024-02-23T00:36:23.061783+0000","last_became_active":"2024-02-23T00:35:27.097784+0000","last_became_peered":"2024-02-23T00:35:27.097784+0000","last_unstale":"2024-02-23T00:36:23.061783+0000","last_undegraded":"2024-02-23T00:36:23.061783+0000","last_fullsized":"2024-02-23T00:36:23.061783+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:58.218704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:58.218704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"63'25","reported_seq":446,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.655572+0000","last_change":"2024-02-23T00:35:01.397052+0000","last_active":"2024-02-23T00:36:22.655572+0000","last_peered":"2024-02-23T00:36:22.655572+0000","last_clean":"2024-02-23T00:36:22.655572+0000","last_became_active":"2024-02-23T00:27:19.014701+0000","last_became_peered":"2024-02-23T00:27:19.014701+0000","last_unstale":"2024-02-23T00:36:22.655572+0000","last_undegraded":"2024-02-23T00:36:22.655572+0000","last_fullsized":"2024-02-23T00:36:22.655572+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:35:01.396982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.396982+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":392,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159052+0000","last_change":"2024-02-23T00:34:12.601903+0000","last_active":"2024-02-23T00:36:23.159052+0000","last_peered":"2024-02-23T00:36:23.159052+0000","last_clean":"2024-02-23T00:36:23.159052+0000","last_became_active":"2024-02-23T00:27:17.011409+0000","last_became_peered":"2024-02-23T00:27:17.011409+0000","last_unstale":"2024-02-23T00:36:23.159052+0000","last_undegraded":"2024-02-23T00:36:23.159052+0000","last_fullsized":"2024-02-23T00:36:23.159052+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":379,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088908+0000","last_change":"2024-02-23T00:33:38.163226+0000","last_active":"2024-02-23T00:36:22.088908+0000","last_peered":"2024-02-23T00:36:22.088908+0000","last_clean":"2024-02-23T00:36:22.088908+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:36:22.088908+0000","last_undegraded":"2024-02-23T00:36:22.088908+0000","last_fullsized":"2024-02-23T00:36:22.088908+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.163155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.163155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":332,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088985+0000","last_change":"2024-02-23T00:35:00.832892+0000","last_active":"2024-02-23T00:36:22.088985+0000","last_peered":"2024-02-23T00:36:22.088985+0000","last_clean":"2024-02-23T00:36:22.088985+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:36:22.088985+0000","last_undegraded":"2024-02-23T00:36:22.088985+0000","last_fullsized":"2024-02-23T00:36:22.088985+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.832818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.832818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.155692+0000","last_change":"2024-02-23T00:35:27.099780+0000","last_active":"2024-02-23T00:36:23.155692+0000","last_peered":"2024-02-23T00:36:23.155692+0000","last_clean":"2024-02-23T00:36:23.155692+0000","last_became_active":"2024-02-23T00:35:27.099186+0000","last_became_peered":"2024-02-23T00:35:27.099186+0000","last_unstale":"2024-02-23T00:36:23.155692+0000","last_undegraded":"2024-02-23T00:36:23.155692+0000","last_fullsized":"2024-02-23T00:36:23.155692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"63'21","reported_seq":431,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.656164+0000","last_change":"2024-02-23T00:35:23.112643+0000","last_active":"2024-02-23T00:36:22.656164+0000","last_peered":"2024-02-23T00:36:22.656164+0000","last_clean":"2024-02-23T00:36:22.656164+0000","last_became_active":"2024-02-23T00:35:23.112399+0000","last_became_peered":"2024-02-23T00:35:23.112399+0000","last_unstale":"2024-02-23T00:36:22.656164+0000","last_undegraded":"2024-02-23T00:36:22.656164+0000","last_fullsized":"2024-02-23T00:36:22.656164+0000","mapping_epoch":56,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":59,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":387,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156817+0000","last_change":"2024-02-23T00:33:42.509570+0000","last_active":"2024-02-23T00:36:23.156817+0000","last_peered":"2024-02-23T00:36:23.156817+0000","last_clean":"2024-02-23T00:36:23.156817+0000","last_became_active":"2024-02-23T00:27:17.012870+0000","last_became_peered":"2024-02-23T00:27:17.012870+0000","last_unstale":"2024-02-23T00:36:23.156817+0000","last_undegraded":"2024-02-23T00:36:23.156817+0000","last_fullsized":"2024-02-23T00:36:23.156817+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.509463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.509463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":410,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654812+0000","last_change":"2024-02-23T00:34:41.101853+0000","last_active":"2024-02-23T00:36:22.654812+0000","last_peered":"2024-02-23T00:36:22.654812+0000","last_clean":"2024-02-23T00:36:22.654812+0000","last_became_active":"2024-02-23T00:34:41.101472+0000","last_became_peered":"2024-02-23T00:34:41.101472+0000","last_unstale":"2024-02-23T00:36:22.654812+0000","last_undegraded":"2024-02-23T00:36:22.654812+0000","last_fullsized":"2024-02-23T00:36:22.654812+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:31.593354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:31.593354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089596+0000","last_change":"2024-02-23T00:35:02.744199+0000","last_active":"2024-02-23T00:36:22.089596+0000","last_peered":"2024-02-23T00:36:22.089596+0000","last_clean":"2024-02-23T00:36:22.089596+0000","last_became_active":"2024-02-23T00:34:41.118378+0000","last_became_peered":"2024-02-23T00:34:41.118378+0000","last_unstale":"2024-02-23T00:36:22.089596+0000","last_undegraded":"2024-02-23T00:36:22.089596+0000","last_fullsized":"2024-02-23T00:36:22.089596+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.744088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.744088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":341,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089997+0000","last_change":"2024-02-23T00:35:55.223009+0000","last_active":"2024-02-23T00:36:22.089997+0000","last_peered":"2024-02-23T00:36:22.089997+0000","last_clean":"2024-02-23T00:36:22.089997+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:36:22.089997+0000","last_undegraded":"2024-02-23T00:36:22.089997+0000","last_fullsized":"2024-02-23T00:36:22.089997+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.222912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.222912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":397,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.155887+0000","last_change":"2024-02-23T00:35:27.093871+0000","last_active":"2024-02-23T00:36:23.155887+0000","last_peered":"2024-02-23T00:36:23.155887+0000","last_clean":"2024-02-23T00:36:23.155887+0000","last_became_active":"2024-02-23T00:35:27.093541+0000","last_became_peered":"2024-02-23T00:35:27.093541+0000","last_unstale":"2024-02-23T00:36:23.155887+0000","last_undegraded":"2024-02-23T00:36:23.155887+0000","last_fullsized":"2024-02-23T00:36:23.155887+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"62'14","reported_seq":411,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.060459+0000","last_change":"2024-02-23T00:34:15.295264+0000","last_active":"2024-02-23T00:36:23.060459+0000","last_peered":"2024-02-23T00:36:23.060459+0000","last_clean":"2024-02-23T00:36:23.060459+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:36:23.060459+0000","last_undegraded":"2024-02-23T00:36:23.060459+0000","last_fullsized":"2024-02-23T00:36:23.060459+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:34:15.295178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.295178+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156904+0000","last_change":"2024-02-23T00:33:43.548018+0000","last_active":"2024-02-23T00:36:23.156904+0000","last_peered":"2024-02-23T00:36:23.156904+0000","last_clean":"2024-02-23T00:36:23.156904+0000","last_became_active":"2024-02-23T00:27:11.711943+0000","last_became_peered":"2024-02-23T00:27:11.711943+0000","last_unstale":"2024-02-23T00:36:23.156904+0000","last_undegraded":"2024-02-23T00:36:23.156904+0000","last_fullsized":"2024-02-23T00:36:23.156904+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":383,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158080+0000","last_change":"2024-02-23T00:32:46.410759+0000","last_active":"2024-02-23T00:36:23.158080+0000","last_peered":"2024-02-23T00:36:23.158080+0000","last_clean":"2024-02-23T00:36:23.158080+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:36:23.158080+0000","last_undegraded":"2024-02-23T00:36:23.158080+0000","last_fullsized":"2024-02-23T00:36:23.158080+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.410635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.410635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":200,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158706+0000","last_change":"2024-02-23T00:34:41.099959+0000","last_active":"2024-02-23T00:36:23.158706+0000","last_peered":"2024-02-23T00:36:23.158706+0000","last_clean":"2024-02-23T00:36:23.158706+0000","last_became_active":"2024-02-23T00:34:41.099598+0000","last_became_peered":"2024-02-23T00:34:41.099598+0000","last_unstale":"2024-02-23T00:36:23.158706+0000","last_undegraded":"2024-02-23T00:36:23.158706+0000","last_fullsized":"2024-02-23T00:36:23.158706+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:46.978406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:46.978406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":386,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159759+0000","last_change":"2024-02-23T00:34:41.114523+0000","last_active":"2024-02-23T00:36:23.159759+0000","last_peered":"2024-02-23T00:36:23.159759+0000","last_clean":"2024-02-23T00:36:23.159759+0000","last_became_active":"2024-02-23T00:34:41.114270+0000","last_became_peered":"2024-02-23T00:34:41.114270+0000","last_unstale":"2024-02-23T00:36:23.159759+0000","last_undegraded":"2024-02-23T00:36:23.159759+0000","last_fullsized":"2024-02-23T00:36:23.159759+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:00.439872+0000","last_clean_scrub_stamp":"2024-02-23T00:33:00.439872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":343,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090545+0000","last_change":"2024-02-23T00:35:52.175358+0000","last_active":"2024-02-23T00:36:22.090545+0000","last_peered":"2024-02-23T00:36:22.090545+0000","last_clean":"2024-02-23T00:36:22.090545+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:36:22.090545+0000","last_undegraded":"2024-02-23T00:36:22.090545+0000","last_fullsized":"2024-02-23T00:36:22.090545+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.175246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.175246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.092002+0000","last_change":"2024-02-23T00:35:27.097499+0000","last_active":"2024-02-23T00:36:22.092002+0000","last_peered":"2024-02-23T00:36:22.092002+0000","last_clean":"2024-02-23T00:36:22.092002+0000","last_became_active":"2024-02-23T00:35:27.097186+0000","last_became_peered":"2024-02-23T00:35:27.097186+0000","last_unstale":"2024-02-23T00:36:22.092002+0000","last_undegraded":"2024-02-23T00:36:22.092002+0000","last_fullsized":"2024-02-23T00:36:22.092002+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"63'19","reported_seq":448,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.773991+0000","last_change":"2024-02-23T00:36:23.773954+0000","last_active":"2024-02-23T00:36:23.773991+0000","last_peered":"2024-02-23T00:36:23.773991+0000","last_clean":"2024-02-23T00:36:23.773991+0000","last_became_active":"2024-02-23T00:35:19.106601+0000","last_became_peered":"2024-02-23T00:35:19.106601+0000","last_unstale":"2024-02-23T00:36:23.773991+0000","last_undegraded":"2024-02-23T00:36:23.773991+0000","last_fullsized":"2024-02-23T00:36:23.773991+0000","mapping_epoch":56,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":63,"num_read_kb":47,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":408,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090863+0000","last_change":"2024-02-23T00:35:51.184999+0000","last_active":"2024-02-23T00:36:22.090863+0000","last_peered":"2024-02-23T00:36:22.090863+0000","last_clean":"2024-02-23T00:36:22.090863+0000","last_became_active":"2024-02-23T00:34:41.094564+0000","last_became_peered":"2024-02-23T00:34:41.094564+0000","last_unstale":"2024-02-23T00:36:22.090863+0000","last_undegraded":"2024-02-23T00:36:22.090863+0000","last_fullsized":"2024-02-23T00:36:22.090863+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.184937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.184937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":641,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089135+0000","last_change":"2024-02-23T00:32:41.360750+0000","last_active":"2024-02-23T00:36:22.089135+0000","last_peered":"2024-02-23T00:36:22.089135+0000","last_clean":"2024-02-23T00:36:22.089135+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:36:22.089135+0000","last_undegraded":"2024-02-23T00:36:22.089135+0000","last_fullsized":"2024-02-23T00:36:22.089135+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:32:41.360646+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:32:41.360646+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158957+0000","last_change":"2024-02-23T00:34:41.105901+0000","last_active":"2024-02-23T00:36:23.158957+0000","last_peered":"2024-02-23T00:36:23.158957+0000","last_clean":"2024-02-23T00:36:23.158957+0000","last_became_active":"2024-02-23T00:34:41.105566+0000","last_became_peered":"2024-02-23T00:34:41.105566+0000","last_unstale":"2024-02-23T00:36:23.158957+0000","last_undegraded":"2024-02-23T00:36:23.158957+0000","last_fullsized":"2024-02-23T00:36:23.158957+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:18.732304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:18.732304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":341,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091064+0000","last_change":"2024-02-23T00:35:03.997896+0000","last_active":"2024-02-23T00:36:22.091064+0000","last_peered":"2024-02-23T00:36:22.091064+0000","last_clean":"2024-02-23T00:36:22.091064+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:36:22.091064+0000","last_undegraded":"2024-02-23T00:36:22.091064+0000","last_fullsized":"2024-02-23T00:36:22.091064+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.997803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.997803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":403,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089638+0000","last_change":"2024-02-23T00:35:03.595532+0000","last_active":"2024-02-23T00:36:22.089638+0000","last_peered":"2024-02-23T00:36:22.089638+0000","last_clean":"2024-02-23T00:36:22.089638+0000","last_became_active":"2024-02-23T00:31:29.065548+0000","last_became_peered":"2024-02-23T00:31:29.065548+0000","last_unstale":"2024-02-23T00:36:22.089638+0000","last_undegraded":"2024-02-23T00:36:22.089638+0000","last_fullsized":"2024-02-23T00:36:22.089638+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.595447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.595447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.655997+0000","last_change":"2024-02-23T00:35:27.093071+0000","last_active":"2024-02-23T00:36:22.655997+0000","last_peered":"2024-02-23T00:36:22.655997+0000","last_clean":"2024-02-23T00:36:22.655997+0000","last_became_active":"2024-02-23T00:35:27.092437+0000","last_became_peered":"2024-02-23T00:35:27.092437+0000","last_unstale":"2024-02-23T00:36:22.655997+0000","last_undegraded":"2024-02-23T00:36:22.655997+0000","last_fullsized":"2024-02-23T00:36:22.655997+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"330'35","reported_seq":589,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.115544+0000","last_change":"2024-02-23T00:34:42.829027+0000","last_active":"2024-02-23T00:36:24.115544+0000","last_peered":"2024-02-23T00:36:24.115544+0000","last_clean":"2024-02-23T00:36:24.115544+0000","last_became_active":"2024-02-23T00:34:41.099241+0000","last_became_peered":"2024-02-23T00:34:41.099241+0000","last_unstale":"2024-02-23T00:36:24.115544+0000","last_undegraded":"2024-02-23T00:36:24.115544+0000","last_fullsized":"2024-02-23T00:36:24.115544+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'34","last_scrub_stamp":"2024-02-23T00:32:45.366416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.366416+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":23,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089870+0000","last_change":"2024-02-23T00:36:17.102172+0000","last_active":"2024-02-23T00:36:22.089870+0000","last_peered":"2024-02-23T00:36:22.089870+0000","last_clean":"2024-02-23T00:36:22.089870+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:36:22.089870+0000","last_undegraded":"2024-02-23T00:36:22.089870+0000","last_fullsized":"2024-02-23T00:36:22.089870+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:36.350888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:36.350888+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":102,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090630+0000","last_change":"2024-02-23T00:34:41.110416+0000","last_active":"2024-02-23T00:36:22.090630+0000","last_peered":"2024-02-23T00:36:22.090630+0000","last_clean":"2024-02-23T00:36:22.090630+0000","last_became_active":"2024-02-23T00:34:41.109972+0000","last_became_peered":"2024-02-23T00:34:41.109972+0000","last_unstale":"2024-02-23T00:36:22.090630+0000","last_undegraded":"2024-02-23T00:36:22.090630+0000","last_fullsized":"2024-02-23T00:36:22.090630+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:41.701634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:33:41.701634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":341,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.092092+0000","last_change":"2024-02-23T00:35:06.983497+0000","last_active":"2024-02-23T00:36:22.092092+0000","last_peered":"2024-02-23T00:36:22.092092+0000","last_clean":"2024-02-23T00:36:22.092092+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:36:22.092092+0000","last_undegraded":"2024-02-23T00:36:22.092092+0000","last_fullsized":"2024-02-23T00:36:22.092092+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:06.983436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.983436+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":307,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.210807+0000","last_change":"2024-02-23T00:36:23.210776+0000","last_active":"2024-02-23T00:36:23.210807+0000","last_peered":"2024-02-23T00:36:23.210807+0000","last_clean":"2024-02-23T00:36:23.210807+0000","last_became_active":"2024-02-23T00:34:41.109182+0000","last_became_peered":"2024-02-23T00:34:41.109182+0000","last_unstale":"2024-02-23T00:36:23.210807+0000","last_undegraded":"2024-02-23T00:36:23.210807+0000","last_fullsized":"2024-02-23T00:36:23.210807+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:23.210706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.210706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":506,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.655717+0000","last_change":"2024-02-23T00:35:27.099059+0000","last_active":"2024-02-23T00:36:22.655717+0000","last_peered":"2024-02-23T00:36:22.655717+0000","last_clean":"2024-02-23T00:36:22.655717+0000","last_became_active":"2024-02-23T00:35:27.098681+0000","last_became_peered":"2024-02-23T00:35:27.098681+0000","last_unstale":"2024-02-23T00:36:22.655717+0000","last_undegraded":"2024-02-23T00:36:22.655717+0000","last_fullsized":"2024-02-23T00:36:22.655717+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"63'16","reported_seq":61,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654390+0000","last_change":"2024-02-23T00:35:44.214807+0000","last_active":"2024-02-23T00:36:22.654390+0000","last_peered":"2024-02-23T00:36:22.654390+0000","last_clean":"2024-02-23T00:36:22.654390+0000","last_became_active":"2024-02-23T00:35:42.167547+0000","last_became_peered":"2024-02-23T00:35:42.167547+0000","last_unstale":"2024-02-23T00:36:22.654390+0000","last_undegraded":"2024-02-23T00:36:22.654390+0000","last_fullsized":"2024-02-23T00:36:22.654390+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:35:44.214727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:44.214727+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":21,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089297+0000","last_change":"2024-02-23T00:34:13.208540+0000","last_active":"2024-02-23T00:36:22.089297+0000","last_peered":"2024-02-23T00:36:22.089297+0000","last_clean":"2024-02-23T00:36:22.089297+0000","last_became_active":"2024-02-23T00:27:17.011870+0000","last_became_peered":"2024-02-23T00:27:17.011870+0000","last_unstale":"2024-02-23T00:36:22.089297+0000","last_undegraded":"2024-02-23T00:36:22.089297+0000","last_fullsized":"2024-02-23T00:36:22.089297+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654692+0000","last_change":"2024-02-23T00:34:41.116110+0000","last_active":"2024-02-23T00:36:22.654692+0000","last_peered":"2024-02-23T00:36:22.654692+0000","last_clean":"2024-02-23T00:36:22.654692+0000","last_became_active":"2024-02-23T00:34:41.115657+0000","last_became_peered":"2024-02-23T00:34:41.115657+0000","last_unstale":"2024-02-23T00:36:22.654692+0000","last_undegraded":"2024-02-23T00:36:22.654692+0000","last_fullsized":"2024-02-23T00:36:22.654692+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.488959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.488959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":336,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088994+0000","last_change":"2024-02-23T00:33:49.754910+0000","last_active":"2024-02-23T00:36:22.088994+0000","last_peered":"2024-02-23T00:36:22.088994+0000","last_clean":"2024-02-23T00:36:22.088994+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:36:22.088994+0000","last_undegraded":"2024-02-23T00:36:22.088994+0000","last_fullsized":"2024-02-23T00:36:22.088994+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:49.754830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.754830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.016939+0000","last_change":"2024-02-23T00:36:24.016902+0000","last_active":"2024-02-23T00:36:24.016939+0000","last_peered":"2024-02-23T00:36:24.016939+0000","last_clean":"2024-02-23T00:36:24.016939+0000","last_became_active":"2024-02-23T00:35:27.091010+0000","last_became_peered":"2024-02-23T00:35:27.091010+0000","last_unstale":"2024-02-23T00:36:24.016939+0000","last_undegraded":"2024-02-23T00:36:24.016939+0000","last_fullsized":"2024-02-23T00:36:24.016939+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:24.016837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:24.016837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"278'50","reported_seq":534,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089262+0000","last_change":"2024-02-23T00:35:09.098161+0000","last_active":"2024-02-23T00:36:22.089262+0000","last_peered":"2024-02-23T00:36:22.089262+0000","last_clean":"2024-02-23T00:36:22.089262+0000","last_became_active":"2024-02-23T00:35:09.097721+0000","last_became_peered":"2024-02-23T00:35:09.097721+0000","last_unstale":"2024-02-23T00:36:22.089262+0000","last_undegraded":"2024-02-23T00:36:22.089262+0000","last_fullsized":"2024-02-23T00:36:22.089262+0000","mapping_epoch":349,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:34:54.691537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.691537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":123,"num_read_kb":99,"num_write":66,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":402,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090638+0000","last_change":"2024-02-23T00:33:43.800448+0000","last_active":"2024-02-23T00:36:22.090638+0000","last_peered":"2024-02-23T00:36:22.090638+0000","last_clean":"2024-02-23T00:36:22.090638+0000","last_became_active":"2024-02-23T00:27:17.008872+0000","last_became_peered":"2024-02-23T00:27:17.008872+0000","last_unstale":"2024-02-23T00:36:22.090638+0000","last_undegraded":"2024-02-23T00:36:22.090638+0000","last_fullsized":"2024-02-23T00:36:22.090638+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:33:43.800360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.800360+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":107,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158037+0000","last_change":"2024-02-23T00:35:48.866316+0000","last_active":"2024-02-23T00:36:23.158037+0000","last_peered":"2024-02-23T00:36:23.158037+0000","last_clean":"2024-02-23T00:36:23.158037+0000","last_became_active":"2024-02-23T00:34:41.110345+0000","last_became_peered":"2024-02-23T00:34:41.110345+0000","last_unstale":"2024-02-23T00:36:23.158037+0000","last_undegraded":"2024-02-23T00:36:23.158037+0000","last_fullsized":"2024-02-23T00:36:23.158037+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:48.866230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:48.866230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":332,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157760+0000","last_change":"2024-02-23T00:32:59.434164+0000","last_active":"2024-02-23T00:36:23.157760+0000","last_peered":"2024-02-23T00:36:23.157760+0000","last_clean":"2024-02-23T00:36:23.157760+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:36:23.157760+0000","last_undegraded":"2024-02-23T00:36:23.157760+0000","last_fullsized":"2024-02-23T00:36:23.157760+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:59.434079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:59.434079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":394,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090392+0000","last_change":"2024-02-23T00:34:24.667629+0000","last_active":"2024-02-23T00:36:22.090392+0000","last_peered":"2024-02-23T00:36:22.090392+0000","last_clean":"2024-02-23T00:36:22.090392+0000","last_became_active":"2024-02-23T00:27:17.011606+0000","last_became_peered":"2024-02-23T00:27:17.011606+0000","last_unstale":"2024-02-23T00:36:22.090392+0000","last_undegraded":"2024-02-23T00:36:22.090392+0000","last_fullsized":"2024-02-23T00:36:22.090392+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.667517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.667517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":101,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091554+0000","last_change":"2024-02-23T00:34:41.111655+0000","last_active":"2024-02-23T00:36:22.091554+0000","last_peered":"2024-02-23T00:36:22.091554+0000","last_clean":"2024-02-23T00:36:22.091554+0000","last_became_active":"2024-02-23T00:34:41.110606+0000","last_became_peered":"2024-02-23T00:34:41.110606+0000","last_unstale":"2024-02-23T00:36:22.091554+0000","last_undegraded":"2024-02-23T00:36:22.091554+0000","last_fullsized":"2024-02-23T00:36:22.091554+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.614960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.614960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":393,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.092380+0000","last_change":"2024-02-23T00:35:57.131197+0000","last_active":"2024-02-23T00:36:22.092380+0000","last_peered":"2024-02-23T00:36:22.092380+0000","last_clean":"2024-02-23T00:36:22.092380+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:36:22.092380+0000","last_undegraded":"2024-02-23T00:36:22.092380+0000","last_fullsized":"2024-02-23T00:36:22.092380+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.131138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:16.828490+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.131138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":105,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159070+0000","last_change":"2024-02-23T00:34:56.402946+0000","last_active":"2024-02-23T00:36:23.159070+0000","last_peered":"2024-02-23T00:36:23.159070+0000","last_clean":"2024-02-23T00:36:23.159070+0000","last_became_active":"2024-02-23T00:34:41.114943+0000","last_became_peered":"2024-02-23T00:34:41.114943+0000","last_unstale":"2024-02-23T00:36:23.159070+0000","last_undegraded":"2024-02-23T00:36:23.159070+0000","last_fullsized":"2024-02-23T00:36:23.159070+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.402851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.402851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":391,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061233+0000","last_change":"2024-02-23T00:34:27.296053+0000","last_active":"2024-02-23T00:36:23.061233+0000","last_peered":"2024-02-23T00:36:23.061233+0000","last_clean":"2024-02-23T00:36:23.061233+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:36:23.061233+0000","last_undegraded":"2024-02-23T00:36:23.061233+0000","last_fullsized":"2024-02-23T00:36:23.061233+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:27.295978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.295978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159538+0000","last_change":"2024-02-23T00:35:52.291042+0000","last_active":"2024-02-23T00:36:23.159538+0000","last_peered":"2024-02-23T00:36:23.159538+0000","last_clean":"2024-02-23T00:36:23.159538+0000","last_became_active":"2024-02-23T00:34:41.113722+0000","last_became_peered":"2024-02-23T00:34:41.113722+0000","last_unstale":"2024-02-23T00:36:23.159538+0000","last_undegraded":"2024-02-23T00:36:23.159538+0000","last_fullsized":"2024-02-23T00:36:23.159538+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.290946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.290946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":386,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159087+0000","last_change":"2024-02-23T00:34:24.675792+0000","last_active":"2024-02-23T00:36:23.159087+0000","last_peered":"2024-02-23T00:36:23.159087+0000","last_clean":"2024-02-23T00:36:23.159087+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:36:23.159087+0000","last_undegraded":"2024-02-23T00:36:23.159087+0000","last_fullsized":"2024-02-23T00:36:23.159087+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.675729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.675729+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":347,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061954+0000","last_change":"2024-02-23T00:35:05.147321+0000","last_active":"2024-02-23T00:36:23.061954+0000","last_peered":"2024-02-23T00:36:23.061954+0000","last_clean":"2024-02-23T00:36:23.061954+0000","last_became_active":"2024-02-23T00:34:41.101132+0000","last_became_peered":"2024-02-23T00:34:41.101132+0000","last_unstale":"2024-02-23T00:36:23.061954+0000","last_undegraded":"2024-02-23T00:36:23.061954+0000","last_fullsized":"2024-02-23T00:36:23.061954+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:05.147249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:05.147249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"63'22","reported_seq":175,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089442+0000","last_change":"2024-02-23T00:34:42.571126+0000","last_active":"2024-02-23T00:36:22.089442+0000","last_peered":"2024-02-23T00:36:22.089442+0000","last_clean":"2024-02-23T00:36:22.089442+0000","last_became_active":"2024-02-23T00:34:41.106795+0000","last_became_peered":"2024-02-23T00:34:41.106795+0000","last_unstale":"2024-02-23T00:36:22.089442+0000","last_undegraded":"2024-02-23T00:36:22.089442+0000","last_fullsized":"2024-02-23T00:36:22.089442+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:34:15.571539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.571539+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":37,"num_read_kb":30,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":397,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157226+0000","last_change":"2024-02-23T00:35:46.269869+0000","last_active":"2024-02-23T00:36:23.157226+0000","last_peered":"2024-02-23T00:36:23.157226+0000","last_clean":"2024-02-23T00:36:23.157226+0000","last_became_active":"2024-02-23T00:34:41.103743+0000","last_became_peered":"2024-02-23T00:34:41.103743+0000","last_unstale":"2024-02-23T00:36:23.157226+0000","last_undegraded":"2024-02-23T00:36:23.157226+0000","last_fullsized":"2024-02-23T00:36:23.157226+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:46.269798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.269798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"76'9","reported_seq":498,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.420213+0000","last_change":"2024-02-23T00:32:56.376450+0000","last_active":"2024-02-23T00:36:24.420213+0000","last_peered":"2024-02-23T00:36:24.420213+0000","last_clean":"2024-02-23T00:36:24.420213+0000","last_became_active":"2024-02-23T00:27:21.028361+0000","last_became_peered":"2024-02-23T00:27:21.028361+0000","last_unstale":"2024-02-23T00:36:24.420213+0000","last_undegraded":"2024-02-23T00:36:24.420213+0000","last_fullsized":"2024-02-23T00:36:24.420213+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":395,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.656146+0000","last_change":"2024-02-23T00:34:41.101927+0000","last_active":"2024-02-23T00:36:22.656146+0000","last_peered":"2024-02-23T00:36:22.656146+0000","last_clean":"2024-02-23T00:36:22.656146+0000","last_became_active":"2024-02-23T00:34:41.101596+0000","last_became_peered":"2024-02-23T00:34:41.101596+0000","last_unstale":"2024-02-23T00:36:22.656146+0000","last_undegraded":"2024-02-23T00:36:22.656146+0000","last_fullsized":"2024-02-23T00:36:22.656146+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.521760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":332,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159696+0000","last_change":"2024-02-23T00:32:45.409616+0000","last_active":"2024-02-23T00:36:23.159696+0000","last_peered":"2024-02-23T00:36:23.159696+0000","last_clean":"2024-02-23T00:36:23.159696+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:36:23.159696+0000","last_undegraded":"2024-02-23T00:36:23.159696+0000","last_fullsized":"2024-02-23T00:36:23.159696+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:45.409548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:45.409548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"144'20","reported_seq":438,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158359+0000","last_change":"2024-02-23T00:35:01.515684+0000","last_active":"2024-02-23T00:36:23.158359+0000","last_peered":"2024-02-23T00:36:23.158359+0000","last_clean":"2024-02-23T00:36:23.158359+0000","last_became_active":"2024-02-23T00:34:41.110140+0000","last_became_peered":"2024-02-23T00:34:41.110140+0000","last_unstale":"2024-02-23T00:36:23.158359+0000","last_undegraded":"2024-02-23T00:36:23.158359+0000","last_fullsized":"2024-02-23T00:36:23.158359+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:35:01.515550+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.515550+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":22,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":406,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.655620+0000","last_change":"2024-02-23T00:35:10.392094+0000","last_active":"2024-02-23T00:36:22.655620+0000","last_peered":"2024-02-23T00:36:22.655620+0000","last_clean":"2024-02-23T00:36:22.655620+0000","last_became_active":"2024-02-23T00:34:41.103241+0000","last_became_peered":"2024-02-23T00:34:41.103241+0000","last_unstale":"2024-02-23T00:36:22.655620+0000","last_undegraded":"2024-02-23T00:36:22.655620+0000","last_fullsized":"2024-02-23T00:36:22.655620+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.391997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.391997+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"330'10","reported_seq":207,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.420312+0000","last_change":"2024-02-23T00:35:06.628388+0000","last_active":"2024-02-23T00:36:24.420312+0000","last_peered":"2024-02-23T00:36:24.420312+0000","last_clean":"2024-02-23T00:36:24.420312+0000","last_became_active":"2024-02-23T00:34:41.109074+0000","last_became_peered":"2024-02-23T00:34:41.109074+0000","last_unstale":"2024-02-23T00:36:24.420312+0000","last_undegraded":"2024-02-23T00:36:24.420312+0000","last_fullsized":"2024-02-23T00:36:24.420312+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'10","last_scrub_stamp":"2024-02-23T00:35:06.628306+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.628306+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090740+0000","last_change":"2024-02-23T00:35:07.627722+0000","last_active":"2024-02-23T00:36:22.090740+0000","last_peered":"2024-02-23T00:36:22.090740+0000","last_clean":"2024-02-23T00:36:22.090740+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:36:22.090740+0000","last_undegraded":"2024-02-23T00:36:22.090740+0000","last_fullsized":"2024-02-23T00:36:22.090740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:07.627620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:07.627620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":101,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.062455+0000","last_change":"2024-02-23T00:34:41.091527+0000","last_active":"2024-02-23T00:36:23.062455+0000","last_peered":"2024-02-23T00:36:23.062455+0000","last_clean":"2024-02-23T00:36:23.062455+0000","last_became_active":"2024-02-23T00:34:41.091189+0000","last_became_peered":"2024-02-23T00:34:41.091189+0000","last_unstale":"2024-02-23T00:36:23.062455+0000","last_undegraded":"2024-02-23T00:36:23.062455+0000","last_fullsized":"2024-02-23T00:36:23.062455+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:14.586935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:14.586935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"62'9","reported_seq":399,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090112+0000","last_change":"2024-02-23T00:32:50.448701+0000","last_active":"2024-02-23T00:36:22.090112+0000","last_peered":"2024-02-23T00:36:22.090112+0000","last_clean":"2024-02-23T00:36:22.090112+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:36:22.090112+0000","last_undegraded":"2024-02-23T00:36:22.090112+0000","last_fullsized":"2024-02-23T00:36:22.090112+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:32:50.448595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.448595+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":400,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.062062+0000","last_change":"2024-02-23T00:34:41.117167+0000","last_active":"2024-02-23T00:36:23.062062+0000","last_peered":"2024-02-23T00:36:23.062062+0000","last_clean":"2024-02-23T00:36:23.062062+0000","last_became_active":"2024-02-23T00:34:41.116840+0000","last_became_peered":"2024-02-23T00:34:41.116840+0000","last_unstale":"2024-02-23T00:36:23.062062+0000","last_undegraded":"2024-02-23T00:36:23.062062+0000","last_fullsized":"2024-02-23T00:36:23.062062+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.310110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.310110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":105,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.655527+0000","last_change":"2024-02-23T00:35:00.416568+0000","last_active":"2024-02-23T00:36:22.655527+0000","last_peered":"2024-02-23T00:36:22.655527+0000","last_clean":"2024-02-23T00:36:22.655527+0000","last_became_active":"2024-02-23T00:34:41.115996+0000","last_became_peered":"2024-02-23T00:34:41.115996+0000","last_unstale":"2024-02-23T00:36:22.655527+0000","last_undegraded":"2024-02-23T00:36:22.655527+0000","last_fullsized":"2024-02-23T00:36:22.655527+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.416493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.416493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":391,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654609+0000","last_change":"2024-02-23T00:35:04.414054+0000","last_active":"2024-02-23T00:36:22.654609+0000","last_peered":"2024-02-23T00:36:22.654609+0000","last_clean":"2024-02-23T00:36:22.654609+0000","last_became_active":"2024-02-23T00:27:23.025625+0000","last_became_peered":"2024-02-23T00:27:23.025625+0000","last_unstale":"2024-02-23T00:36:22.654609+0000","last_undegraded":"2024-02-23T00:36:22.654609+0000","last_fullsized":"2024-02-23T00:36:22.654609+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.413991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.413991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":336,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088886+0000","last_change":"2024-02-23T00:33:48.779550+0000","last_active":"2024-02-23T00:36:22.088886+0000","last_peered":"2024-02-23T00:36:22.088886+0000","last_clean":"2024-02-23T00:36:22.088886+0000","last_became_active":"2024-02-23T00:30:48.666975+0000","last_became_peered":"2024-02-23T00:30:48.666975+0000","last_unstale":"2024-02-23T00:36:22.088886+0000","last_undegraded":"2024-02-23T00:36:22.088886+0000","last_fullsized":"2024-02-23T00:36:22.088886+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:48.779472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:48.779472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"62'11","reported_seq":432,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090068+0000","last_change":"2024-02-23T00:35:45.171309+0000","last_active":"2024-02-23T00:36:22.090068+0000","last_peered":"2024-02-23T00:36:22.090068+0000","last_clean":"2024-02-23T00:36:22.090068+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:36:22.090068+0000","last_undegraded":"2024-02-23T00:36:22.090068+0000","last_fullsized":"2024-02-23T00:36:22.090068+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:35:45.171195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:45.171195+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":385,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089472+0000","last_change":"2024-02-23T00:35:51.783163+0000","last_active":"2024-02-23T00:36:22.089472+0000","last_peered":"2024-02-23T00:36:22.089472+0000","last_clean":"2024-02-23T00:36:22.089472+0000","last_became_active":"2024-02-23T00:27:17.010535+0000","last_became_peered":"2024-02-23T00:27:17.010535+0000","last_unstale":"2024-02-23T00:36:22.089472+0000","last_undegraded":"2024-02-23T00:36:22.089472+0000","last_fullsized":"2024-02-23T00:36:22.089472+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.783089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.783089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":390,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090192+0000","last_change":"2024-02-23T00:34:41.116355+0000","last_active":"2024-02-23T00:36:22.090192+0000","last_peered":"2024-02-23T00:36:22.090192+0000","last_clean":"2024-02-23T00:36:22.090192+0000","last_became_active":"2024-02-23T00:34:41.115862+0000","last_became_peered":"2024-02-23T00:34:41.115862+0000","last_unstale":"2024-02-23T00:36:22.090192+0000","last_undegraded":"2024-02-23T00:36:22.090192+0000","last_fullsized":"2024-02-23T00:36:22.090192+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.151986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":100,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654902+0000","last_change":"2024-02-23T00:34:41.115170+0000","last_active":"2024-02-23T00:36:22.654902+0000","last_peered":"2024-02-23T00:36:22.654902+0000","last_clean":"2024-02-23T00:36:22.654902+0000","last_became_active":"2024-02-23T00:34:41.114829+0000","last_became_peered":"2024-02-23T00:34:41.114829+0000","last_unstale":"2024-02-23T00:36:22.654902+0000","last_undegraded":"2024-02-23T00:36:22.654902+0000","last_fullsized":"2024-02-23T00:36:22.654902+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.620107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":364,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061444+0000","last_change":"2024-02-23T00:35:47.130831+0000","last_active":"2024-02-23T00:36:23.061444+0000","last_peered":"2024-02-23T00:36:23.061444+0000","last_clean":"2024-02-23T00:36:23.061444+0000","last_became_active":"2024-02-23T00:34:41.093430+0000","last_became_peered":"2024-02-23T00:34:41.093430+0000","last_unstale":"2024-02-23T00:36:23.061444+0000","last_undegraded":"2024-02-23T00:36:23.061444+0000","last_fullsized":"2024-02-23T00:36:23.061444+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:35:47.130749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.130749+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"74'6","reported_seq":140,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089217+0000","last_change":"2024-02-23T00:34:41.135529+0000","last_active":"2024-02-23T00:36:22.089217+0000","last_peered":"2024-02-23T00:36:22.089217+0000","last_clean":"2024-02-23T00:36:22.089217+0000","last_became_active":"2024-02-23T00:34:41.101219+0000","last_became_peered":"2024-02-23T00:34:41.101219+0000","last_unstale":"2024-02-23T00:36:22.089217+0000","last_undegraded":"2024-02-23T00:36:22.089217+0000","last_fullsized":"2024-02-23T00:36:22.089217+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:32:11.989333+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:32:11.989333+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":383,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159444+0000","last_change":"2024-02-23T00:32:40.343941+0000","last_active":"2024-02-23T00:36:23.159444+0000","last_peered":"2024-02-23T00:36:23.159444+0000","last_clean":"2024-02-23T00:36:23.159444+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:36:23.159444+0000","last_undegraded":"2024-02-23T00:36:23.159444+0000","last_fullsized":"2024-02-23T00:36:23.159444+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:40.343863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:32:40.343863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":387,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061182+0000","last_change":"2024-02-23T00:35:39.035845+0000","last_active":"2024-02-23T00:36:23.061182+0000","last_peered":"2024-02-23T00:36:23.061182+0000","last_clean":"2024-02-23T00:36:23.061182+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:36:23.061182+0000","last_undegraded":"2024-02-23T00:36:23.061182+0000","last_fullsized":"2024-02-23T00:36:23.061182+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.035767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.035767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":386,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157236+0000","last_change":"2024-02-23T00:34:17.760839+0000","last_active":"2024-02-23T00:36:23.157236+0000","last_peered":"2024-02-23T00:36:23.157236+0000","last_clean":"2024-02-23T00:36:23.157236+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:36:23.157236+0000","last_undegraded":"2024-02-23T00:36:23.157236+0000","last_fullsized":"2024-02-23T00:36:23.157236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.760765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.760765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":332,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090261+0000","last_change":"2024-02-23T00:33:44.152089+0000","last_active":"2024-02-23T00:36:22.090261+0000","last_peered":"2024-02-23T00:36:22.090261+0000","last_clean":"2024-02-23T00:36:22.090261+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:36:22.090261+0000","last_undegraded":"2024-02-23T00:36:22.090261+0000","last_fullsized":"2024-02-23T00:36:22.090261+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.152014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.152014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"79'24","reported_seq":188,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156751+0000","last_change":"2024-02-23T00:35:01.643176+0000","last_active":"2024-02-23T00:36:23.156751+0000","last_peered":"2024-02-23T00:36:23.156751+0000","last_clean":"2024-02-23T00:36:23.156751+0000","last_became_active":"2024-02-23T00:34:41.114516+0000","last_became_peered":"2024-02-23T00:34:41.114516+0000","last_unstale":"2024-02-23T00:36:23.156751+0000","last_undegraded":"2024-02-23T00:36:23.156751+0000","last_fullsized":"2024-02-23T00:36:23.156751+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'24","last_scrub_stamp":"2024-02-23T00:35:01.643114+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.643114+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":24,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":390,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.062290+0000","last_change":"2024-02-23T00:35:03.113215+0000","last_active":"2024-02-23T00:36:23.062290+0000","last_peered":"2024-02-23T00:36:23.062290+0000","last_clean":"2024-02-23T00:36:23.062290+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:36:23.062290+0000","last_undegraded":"2024-02-23T00:36:23.062290+0000","last_fullsized":"2024-02-23T00:36:23.062290+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.113099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.113099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157109+0000","last_change":"2024-02-23T00:35:47.857609+0000","last_active":"2024-02-23T00:36:23.157109+0000","last_peered":"2024-02-23T00:36:23.157109+0000","last_clean":"2024-02-23T00:36:23.157109+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:36:23.157109+0000","last_undegraded":"2024-02-23T00:36:23.157109+0000","last_fullsized":"2024-02-23T00:36:23.157109+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.857533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.857533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":105,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.655414+0000","last_change":"2024-02-23T00:35:57.175277+0000","last_active":"2024-02-23T00:36:22.655414+0000","last_peered":"2024-02-23T00:36:22.655414+0000","last_clean":"2024-02-23T00:36:22.655414+0000","last_became_active":"2024-02-23T00:34:41.097083+0000","last_became_peered":"2024-02-23T00:34:41.097083+0000","last_unstale":"2024-02-23T00:36:22.655414+0000","last_undegraded":"2024-02-23T00:36:22.655414+0000","last_fullsized":"2024-02-23T00:36:22.655414+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.175171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.175171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":268,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158329+0000","last_change":"2024-02-23T00:36:04.097119+0000","last_active":"2024-02-23T00:36:23.158329+0000","last_peered":"2024-02-23T00:36:23.158329+0000","last_clean":"2024-02-23T00:36:23.158329+0000","last_became_active":"2024-02-23T00:36:04.096804+0000","last_became_peered":"2024-02-23T00:36:04.096804+0000","last_unstale":"2024-02-23T00:36:23.158329+0000","last_undegraded":"2024-02-23T00:36:23.158329+0000","last_fullsized":"2024-02-23T00:36:23.158329+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:53.153821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:53.153821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":392,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156577+0000","last_change":"2024-02-23T00:34:42.064110+0000","last_active":"2024-02-23T00:36:23.156577+0000","last_peered":"2024-02-23T00:36:23.156577+0000","last_clean":"2024-02-23T00:36:23.156577+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:36:23.156577+0000","last_undegraded":"2024-02-23T00:36:23.156577+0000","last_fullsized":"2024-02-23T00:36:23.156577+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:32.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:32.520027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":508,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:24.420738+0000","last_change":"2024-02-23T00:34:12.258420+0000","last_active":"2024-02-23T00:36:24.420738+0000","last_peered":"2024-02-23T00:36:24.420738+0000","last_clean":"2024-02-23T00:36:24.420738+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:36:24.420738+0000","last_undegraded":"2024-02-23T00:36:24.420738+0000","last_fullsized":"2024-02-23T00:36:24.420738+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:34:12.258344+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.258344+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158416+0000","last_change":"2024-02-23T00:35:09.656923+0000","last_active":"2024-02-23T00:36:23.158416+0000","last_peered":"2024-02-23T00:36:23.158416+0000","last_clean":"2024-02-23T00:36:23.158416+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:36:23.158416+0000","last_undegraded":"2024-02-23T00:36:23.158416+0000","last_fullsized":"2024-02-23T00:36:23.158416+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:35:09.656838+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:35:09.656838+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":344,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158455+0000","last_change":"2024-02-23T00:34:41.105563+0000","last_active":"2024-02-23T00:36:23.158455+0000","last_peered":"2024-02-23T00:36:23.158455+0000","last_clean":"2024-02-23T00:36:23.158455+0000","last_became_active":"2024-02-23T00:34:41.105162+0000","last_became_peered":"2024-02-23T00:34:41.105162+0000","last_unstale":"2024-02-23T00:36:23.158455+0000","last_undegraded":"2024-02-23T00:36:23.158455+0000","last_fullsized":"2024-02-23T00:36:23.158455+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:27.425604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:27.425604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":106,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.061138+0000","last_change":"2024-02-23T00:35:02.151821+0000","last_active":"2024-02-23T00:36:23.061138+0000","last_peered":"2024-02-23T00:36:23.061138+0000","last_clean":"2024-02-23T00:36:23.061138+0000","last_became_active":"2024-02-23T00:34:41.092835+0000","last_became_peered":"2024-02-23T00:34:41.092835+0000","last_unstale":"2024-02-23T00:36:23.061138+0000","last_undegraded":"2024-02-23T00:36:23.061138+0000","last_fullsized":"2024-02-23T00:36:23.061138+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.151731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.656313+0000","last_change":"2024-02-23T00:34:25.593132+0000","last_active":"2024-02-23T00:36:22.656313+0000","last_peered":"2024-02-23T00:36:22.656313+0000","last_clean":"2024-02-23T00:36:22.656313+0000","last_became_active":"2024-02-23T00:27:21.030189+0000","last_became_peered":"2024-02-23T00:27:21.030189+0000","last_unstale":"2024-02-23T00:36:22.656313+0000","last_undegraded":"2024-02-23T00:36:22.656313+0000","last_fullsized":"2024-02-23T00:36:22.656313+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":384,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089272+0000","last_change":"2024-02-23T00:35:57.658486+0000","last_active":"2024-02-23T00:36:22.089272+0000","last_peered":"2024-02-23T00:36:22.089272+0000","last_clean":"2024-02-23T00:36:22.089272+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:36:22.089272+0000","last_undegraded":"2024-02-23T00:36:22.089272+0000","last_fullsized":"2024-02-23T00:36:22.089272+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.658406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.658406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":340,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159196+0000","last_change":"2024-02-23T00:35:39.871154+0000","last_active":"2024-02-23T00:36:23.159196+0000","last_peered":"2024-02-23T00:36:23.159196+0000","last_clean":"2024-02-23T00:36:23.159196+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:36:23.159196+0000","last_undegraded":"2024-02-23T00:36:23.159196+0000","last_fullsized":"2024-02-23T00:36:23.159196+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.871058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.871058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":399,"reported_epoch":420,"state":"active+clean","last_fresh":"2024-02-23T00:36:25.100710+0000","last_change":"2024-02-23T00:36:25.100710+0000","last_active":"2024-02-23T00:36:25.100710+0000","last_peered":"2024-02-23T00:36:25.100710+0000","last_clean":"2024-02-23T00:36:25.100710+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:36:25.100710+0000","last_undegraded":"2024-02-23T00:36:25.100710+0000","last_fullsized":"2024-02-23T00:36:25.100710+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:25.100627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.100627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":383,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.091736+0000","last_change":"2024-02-23T00:35:52.791811+0000","last_active":"2024-02-23T00:36:22.091736+0000","last_peered":"2024-02-23T00:36:22.091736+0000","last_clean":"2024-02-23T00:36:22.091736+0000","last_became_active":"2024-02-23T00:27:21.024750+0000","last_became_peered":"2024-02-23T00:27:21.024750+0000","last_unstale":"2024-02-23T00:36:22.091736+0000","last_undegraded":"2024-02-23T00:36:22.091736+0000","last_fullsized":"2024-02-23T00:36:22.091736+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.791748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.791748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":383,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090315+0000","last_change":"2024-02-23T00:34:57.661029+0000","last_active":"2024-02-23T00:36:22.090315+0000","last_peered":"2024-02-23T00:36:22.090315+0000","last_clean":"2024-02-23T00:36:22.090315+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:36:22.090315+0000","last_undegraded":"2024-02-23T00:36:22.090315+0000","last_fullsized":"2024-02-23T00:36:22.090315+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:57.660954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.660954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":259,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159362+0000","last_change":"2024-02-23T00:36:17.100099+0000","last_active":"2024-02-23T00:36:23.159362+0000","last_peered":"2024-02-23T00:36:23.159362+0000","last_clean":"2024-02-23T00:36:23.159362+0000","last_became_active":"2024-02-23T00:36:17.099716+0000","last_became_peered":"2024-02-23T00:36:17.099716+0000","last_unstale":"2024-02-23T00:36:23.159362+0000","last_undegraded":"2024-02-23T00:36:23.159362+0000","last_fullsized":"2024-02-23T00:36:23.159362+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:28.419374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:28.419374+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":102,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158226+0000","last_change":"2024-02-23T00:34:41.103473+0000","last_active":"2024-02-23T00:36:23.158226+0000","last_peered":"2024-02-23T00:36:23.158226+0000","last_clean":"2024-02-23T00:36:23.158226+0000","last_became_active":"2024-02-23T00:34:41.103114+0000","last_became_peered":"2024-02-23T00:34:41.103114+0000","last_unstale":"2024-02-23T00:36:23.158226+0000","last_undegraded":"2024-02-23T00:36:23.158226+0000","last_fullsized":"2024-02-23T00:36:23.158226+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.750804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.750804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":394,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089847+0000","last_change":"2024-02-23T00:34:56.679204+0000","last_active":"2024-02-23T00:36:22.089847+0000","last_peered":"2024-02-23T00:36:22.089847+0000","last_clean":"2024-02-23T00:36:22.089847+0000","last_became_active":"2024-02-23T00:34:41.106439+0000","last_became_peered":"2024-02-23T00:34:41.106439+0000","last_unstale":"2024-02-23T00:36:22.089847+0000","last_undegraded":"2024-02-23T00:36:22.089847+0000","last_fullsized":"2024-02-23T00:36:22.089847+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.679136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.679136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":106,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.062451+0000","last_change":"2024-02-23T00:35:10.167898+0000","last_active":"2024-02-23T00:36:23.062451+0000","last_peered":"2024-02-23T00:36:23.062451+0000","last_clean":"2024-02-23T00:36:23.062451+0000","last_became_active":"2024-02-23T00:34:41.113447+0000","last_became_peered":"2024-02-23T00:34:41.113447+0000","last_unstale":"2024-02-23T00:36:23.062451+0000","last_undegraded":"2024-02-23T00:36:23.062451+0000","last_fullsized":"2024-02-23T00:36:23.062451+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.167829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.167829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":332,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.159466+0000","last_change":"2024-02-23T00:33:33.382591+0000","last_active":"2024-02-23T00:36:23.159466+0000","last_peered":"2024-02-23T00:36:23.159466+0000","last_clean":"2024-02-23T00:36:23.159466+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:36:23.159466+0000","last_undegraded":"2024-02-23T00:36:23.159466+0000","last_fullsized":"2024-02-23T00:36:23.159466+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.382510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.382510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":99,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089620+0000","last_change":"2024-02-23T00:34:42.058800+0000","last_active":"2024-02-23T00:36:22.089620+0000","last_peered":"2024-02-23T00:36:22.089620+0000","last_clean":"2024-02-23T00:36:22.089620+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:36:22.089620+0000","last_undegraded":"2024-02-23T00:36:22.089620+0000","last_fullsized":"2024-02-23T00:36:22.089620+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:32.720048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:32.720048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":379,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158224+0000","last_change":"2024-02-23T00:33:43.299920+0000","last_active":"2024-02-23T00:36:23.158224+0000","last_peered":"2024-02-23T00:36:23.158224+0000","last_clean":"2024-02-23T00:36:23.158224+0000","last_became_active":"2024-02-23T00:27:21.025975+0000","last_became_peered":"2024-02-23T00:27:21.025975+0000","last_unstale":"2024-02-23T00:36:23.158224+0000","last_undegraded":"2024-02-23T00:36:23.158224+0000","last_fullsized":"2024-02-23T00:36:23.158224+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":386,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654882+0000","last_change":"2024-02-23T00:34:19.572886+0000","last_active":"2024-02-23T00:36:22.654882+0000","last_peered":"2024-02-23T00:36:22.654882+0000","last_clean":"2024-02-23T00:36:22.654882+0000","last_became_active":"2024-02-23T00:27:23.026060+0000","last_became_peered":"2024-02-23T00:27:23.026060+0000","last_unstale":"2024-02-23T00:36:22.654882+0000","last_undegraded":"2024-02-23T00:36:22.654882+0000","last_fullsized":"2024-02-23T00:36:22.654882+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:19.572808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.572808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":340,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156556+0000","last_change":"2024-02-23T00:34:29.677110+0000","last_active":"2024-02-23T00:36:23.156556+0000","last_peered":"2024-02-23T00:36:23.156556+0000","last_clean":"2024-02-23T00:36:23.156556+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:36:23.156556+0000","last_undegraded":"2024-02-23T00:36:23.156556+0000","last_fullsized":"2024-02-23T00:36:23.156556+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:29.677037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:29.677037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":392,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.158727+0000","last_change":"2024-02-23T00:34:30.726107+0000","last_active":"2024-02-23T00:36:23.158727+0000","last_peered":"2024-02-23T00:36:23.158727+0000","last_clean":"2024-02-23T00:36:23.158727+0000","last_became_active":"2024-02-23T00:27:17.012954+0000","last_became_peered":"2024-02-23T00:27:17.012954+0000","last_unstale":"2024-02-23T00:36:23.158727+0000","last_undegraded":"2024-02-23T00:36:23.158727+0000","last_fullsized":"2024-02-23T00:36:23.158727+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157241+0000","last_change":"2024-02-23T00:34:31.754382+0000","last_active":"2024-02-23T00:36:23.157241+0000","last_peered":"2024-02-23T00:36:23.157241+0000","last_clean":"2024-02-23T00:36:23.157241+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:36:23.157241+0000","last_undegraded":"2024-02-23T00:36:23.157241+0000","last_fullsized":"2024-02-23T00:36:23.157241+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:31.754311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.754311+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":388,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.156791+0000","last_change":"2024-02-23T00:35:41.260304+0000","last_active":"2024-02-23T00:36:23.156791+0000","last_peered":"2024-02-23T00:36:23.156791+0000","last_clean":"2024-02-23T00:36:23.156791+0000","last_became_active":"2024-02-23T00:34:41.112980+0000","last_became_peered":"2024-02-23T00:34:41.112980+0000","last_unstale":"2024-02-23T00:36:23.156791+0000","last_undegraded":"2024-02-23T00:36:23.156791+0000","last_fullsized":"2024-02-23T00:36:23.156791+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:41.260231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.260231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":341,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089108+0000","last_change":"2024-02-23T00:34:41.115172+0000","last_active":"2024-02-23T00:36:22.089108+0000","last_peered":"2024-02-23T00:36:22.089108+0000","last_clean":"2024-02-23T00:36:22.089108+0000","last_became_active":"2024-02-23T00:34:41.114758+0000","last_became_peered":"2024-02-23T00:34:41.114758+0000","last_unstale":"2024-02-23T00:36:22.089108+0000","last_undegraded":"2024-02-23T00:36:22.089108+0000","last_fullsized":"2024-02-23T00:36:22.089108+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.192483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.192483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":105,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.656228+0000","last_change":"2024-02-23T00:34:54.388897+0000","last_active":"2024-02-23T00:36:22.656228+0000","last_peered":"2024-02-23T00:36:22.656228+0000","last_clean":"2024-02-23T00:36:22.656228+0000","last_became_active":"2024-02-23T00:34:41.115290+0000","last_became_peered":"2024-02-23T00:34:41.115290+0000","last_unstale":"2024-02-23T00:36:22.656228+0000","last_undegraded":"2024-02-23T00:36:22.656228+0000","last_fullsized":"2024-02-23T00:36:22.656228+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.388834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":42,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.060803+0000","last_change":"2024-02-23T00:35:42.473209+0000","last_active":"2024-02-23T00:36:23.060803+0000","last_peered":"2024-02-23T00:36:23.060803+0000","last_clean":"2024-02-23T00:36:23.060803+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:36:23.060803+0000","last_undegraded":"2024-02-23T00:36:23.060803+0000","last_fullsized":"2024-02-23T00:36:23.060803+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:38.617181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:38.617181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":385,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.089214+0000","last_change":"2024-02-23T00:34:23.597939+0000","last_active":"2024-02-23T00:36:22.089214+0000","last_peered":"2024-02-23T00:36:22.089214+0000","last_clean":"2024-02-23T00:36:22.089214+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:36:22.089214+0000","last_undegraded":"2024-02-23T00:36:22.089214+0000","last_fullsized":"2024-02-23T00:36:22.089214+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:23.597868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.597868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":102,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.654931+0000","last_change":"2024-02-23T00:35:55.176531+0000","last_active":"2024-02-23T00:36:22.654931+0000","last_peered":"2024-02-23T00:36:22.654931+0000","last_clean":"2024-02-23T00:36:22.654931+0000","last_became_active":"2024-02-23T00:34:42.056094+0000","last_became_peered":"2024-02-23T00:34:42.056094+0000","last_unstale":"2024-02-23T00:36:22.654931+0000","last_undegraded":"2024-02-23T00:36:22.654931+0000","last_fullsized":"2024-02-23T00:36:22.654931+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.176441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.176441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":392,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.656188+0000","last_change":"2024-02-23T00:33:42.579086+0000","last_active":"2024-02-23T00:36:22.656188+0000","last_peered":"2024-02-23T00:36:22.656188+0000","last_clean":"2024-02-23T00:36:22.656188+0000","last_became_active":"2024-02-23T00:27:17.008138+0000","last_became_peered":"2024-02-23T00:27:17.008138+0000","last_unstale":"2024-02-23T00:36:22.656188+0000","last_undegraded":"2024-02-23T00:36:22.656188+0000","last_fullsized":"2024-02-23T00:36:22.656188+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":385,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.088891+0000","last_change":"2024-02-23T00:34:28.586621+0000","last_active":"2024-02-23T00:36:22.088891+0000","last_peered":"2024-02-23T00:36:22.088891+0000","last_clean":"2024-02-23T00:36:22.088891+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:36:22.088891+0000","last_undegraded":"2024-02-23T00:36:22.088891+0000","last_fullsized":"2024-02-23T00:36:22.088891+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.586556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.586556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":107,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:23.157684+0000","last_change":"2024-02-23T00:34:55.609318+0000","last_active":"2024-02-23T00:36:23.157684+0000","last_peered":"2024-02-23T00:36:23.157684+0000","last_clean":"2024-02-23T00:36:23.157684+0000","last_became_active":"2024-02-23T00:34:41.109129+0000","last_became_peered":"2024-02-23T00:34:41.109129+0000","last_unstale":"2024-02-23T00:36:23.157684+0000","last_undegraded":"2024-02-23T00:36:23.157684+0000","last_fullsized":"2024-02-23T00:36:23.157684+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:55.609248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:55.609248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":341,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-02-23T00:36:22.090240+0000","last_change":"2024-02-23T00:34:41.099947+0000","last_active":"2024-02-23T00:36:22.090240+0000","last_peered":"2024-02-23T00:36:22.090240+0000","last_clean":"2024-02-23T00:36:22.090240+0000","last_became_active":"2024-02-23T00:34:41.099195+0000","last_became_peered":"2024-02-23T00:34:41.099195+0000","last_unstale":"2024-02-23T00:36:22.090240+0000","last_undegraded":"2024-02-23T00:36:22.090240+0000","last_fullsized":"2024-02-23T00:36:22.090240+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:56.396150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.396150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":212,"num_pg":32,"stat_sum":{"num_bytes":12,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":9,"num_write":12,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":7},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":4001484,"num_objects":159,"num_object_clones":3,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":107,"num_read_kb":9,"num_write":596,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":4000297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13910016,"data_stored":12004461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1870,"ondisk_log_size":1870,"up":96,"acting":96,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":126,"acting":126,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1073,"num_read_kb":798,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":66,"acting":66,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":486,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":287,"num_read_kb":286,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332209,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310000,"kb_used_data":8680,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93435152,"statfs":{"total":95995035648,"available":95677595648,"internally_reserved":0,"allocated":8888320,"data_stored":6566236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:32 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":2.0649999999999999,"15min":1.0249999999999999},"min":{"1min":0.24299999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.746,"5min":61.048999999999999,"15min":61.048999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":2.6909999999999998,"15min":1.248},"min":{"1min":0.29499999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.91400000000000003,"5min":73.870999999999995,"15min":73.870999999999995},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:35:32 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":1.0569999999999999,"15min":0.70399999999999996},"min":{"1min":0.29499999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.72199999999999998,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.501},{"interface":"front","average":{"1min":0.46899999999999997,"5min":1.0580000000000001,"15min":0.69099999999999995},"min":{"1min":0.33000000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.81699999999999995,"5min":29.423999999999999,"15min":29.423999999999999},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:35:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.96499999999999997,"15min":0.67100000000000004},"min":{"1min":0.29799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70599999999999996,"5min":16.603999999999999,"15min":16.603999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.81000000000000005,"15min":0.62},"min":{"1min":0.34799999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":3.1760000000000002,"5min":7.5339999999999998,"15min":7.5339999999999998},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:35:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":1.911,"15min":0.98499999999999999},"min":{"1min":0.34399999999999997,"5min":0.316,"15min":0.311},"max":{"1min":0.95899999999999996,"5min":57.729999999999997,"15min":57.729999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.673,"15min":0.91300000000000003},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.312},"max":{"1min":3.0899999999999999,"5min":27.573,"15min":27.573},"last":0.49299999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:35:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42299999999999999,"15min":0.42199999999999999},"min":{"1min":0.26700000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.75800000000000001,"5min":1.706,"15min":1.706},"last":0.435},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.46200000000000002,"15min":0.442},"min":{"1min":0.28399999999999997,"5min":0.25700000000000001,"15min":0.254},"max":{"1min":0.69799999999999995,"5min":3.294,"15min":3.294},"last":0.42299999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:35:32 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.42999999999999999,"15min":0.434},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66700000000000004,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47199999999999998,"15min":0.45300000000000001},"min":{"1min":0.33300000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.94499999999999995,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.39600000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:35:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.48799999999999999},"min":{"1min":0.32200000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.98199999999999998,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.48999999999999999,"15min":0.47299999999999998},"min":{"1min":0.32300000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.753,"5min":1.002,"15min":1.002},"last":0.41199999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561142,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306124,"kb_used_data":4804,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93439028,"statfs":{"total":95995035648,"available":95681564672,"internally_reserved":0,"allocated":4919296,"data_stored":2566053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.6499999999999999,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.78800000000000003,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64800000000000002,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":9.468,"15min":9.468},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.861,"15min":1.0149999999999999},"min":{"1min":0.34899999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.72799999999999998,"5min":59.347999999999999,"15min":59.347999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":2.7799999999999998,"15min":1.294},"min":{"1min":0.40400000000000003,"5min":0.28299999999999997,"15min":0.23100000000000001},"max":{"1min":0.70599999999999996,"5min":87.221000000000004,"15min":87.221000000000004},"last":0.45300000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.167,"15min":0.81299999999999994},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.745,"5min":25.550999999999998,"15min":25.550999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.302,"15min":0.88300000000000001},"min":{"1min":0.42899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":4.3570000000000002,"5min":40.914000000000001,"15min":40.914000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.2030000000000001,"15min":0.80700000000000005},"min":{"1min":0.35699999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.76800000000000002,"5min":31.309999999999999,"15min":31.309999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.452,"15min":1.3300000000000001},"min":{"1min":0.40200000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78700000000000003,"5min":86.311999999999998,"15min":86.311999999999998},"last":0.499}]},{"osd":4,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46200000000000002,"15min":0.434},"min":{"1min":0.30299999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63600000000000001,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47499999999999998,"15min":0.505},"min":{"1min":0.33100000000000002,"5min":0.26100000000000001,"15min":0.25900000000000001},"max":{"1min":0.73299999999999998,"5min":3.1259999999999999,"15min":17.718},"last":0.439}]},{"osd":5,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48999999999999999,"15min":0.48399999999999999},"min":{"1min":0.34200000000000003,"5min":0.27400000000000002,"15min":0.27200000000000002},"max":{"1min":0.69099999999999995,"5min":4.4210000000000003,"15min":8.0990000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.47399999999999998,"15min":0.47899999999999998},"min":{"1min":0.39400000000000002,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.72099999999999997,"5min":0.75700000000000001,"15min":8.1370000000000005},"last":0.51500000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.83499999999999996,"15min":0.59699999999999998},"min":{"1min":0.32200000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75,"5min":17.881,"15min":17.881},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.81100000000000005,"15min":0.59399999999999997},"min":{"1min":0.32000000000000001,"5min":0.28699999999999998,"15min":0.28199999999999997},"max":{"1min":0.68200000000000005,"5min":17.396999999999998,"15min":17.396999999999998},"last":0.57999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574989,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314292,"kb_used_data":8876,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93430860,"statfs":{"total":95995035648,"available":95673200640,"internally_reserved":0,"allocated":9089024,"data_stored":6567991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.29,"15min":0.66300000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.51700000000000002,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.89600000000000002,"15min":0.55200000000000005},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60499999999999998,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.35399999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":1.2989999999999999,"15min":0.67700000000000005},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.53100000000000003,"5min":35.313000000000002,"15min":35.313000000000002},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.99199999999999999,"15min":0.57999999999999996},"min":{"1min":0.247,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.496,"5min":27.908999999999999,"15min":27.908999999999999},"last":0.33900000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":1.1439999999999999,"15min":0.65400000000000003},"min":{"1min":0.26500000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.55600000000000005,"5min":21.013999999999999,"15min":21.013999999999999},"last":0.309},{"interface":"front","average":{"1min":0.38,"5min":1.1100000000000001,"15min":0.65300000000000002},"min":{"1min":0.28799999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.88900000000000001,"5min":29.234999999999999,"15min":29.234999999999999},"last":0.32600000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.98499999999999999,"15min":0.70399999999999996},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.878,"5min":16.231000000000002,"15min":16.231000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.081,"15min":0.72799999999999998},"min":{"1min":0.36399999999999999,"5min":0.308,"15min":0.308},"max":{"1min":0.872,"5min":29.212,"15min":29.212},"last":0.52600000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":1.034,"15min":0.72699999999999998},"min":{"1min":0.36099999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.68300000000000005,"5min":16.294,"15min":16.294},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.88700000000000001,"15min":0.68799999999999994},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83599999999999997,"5min":11.105,"15min":11.105},"last":0.55400000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.161,"15min":0.76800000000000002},"min":{"1min":0.34699999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94799999999999995,"5min":29.187000000000001,"15min":29.187000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":1.0409999999999999,"15min":0.75900000000000001},"min":{"1min":0.41099999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.80900000000000005,"5min":16.369,"15min":16.369},"last":0.58199999999999996}]},{"osd":7,"last update":"Fri Feb 23 00:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":1.014,"15min":0.77100000000000002},"min":{"1min":0.40799999999999997,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.86099999999999999,"5min":16.395,"15min":16.395},"last":0.627},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.67100000000000004,"15min":0.67000000000000004},"min":{"1min":0.45100000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89500000000000002,"5min":4.4379999999999997,"15min":11.135999999999999},"last":0.49399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738515,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318388,"kb_used_data":4780,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426764,"statfs":{"total":95995035648,"available":95669006336,"internally_reserved":0,"allocated":4894720,"data_stored":2565587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.80600000000000005,"15min":0.495},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47799999999999998,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.165,"15min":0.61899999999999999},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.997,"15min":0.58899999999999997},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.64400000000000002,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.748,"15min":0.48999999999999999},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.22700000000000001},"max":{"1min":0.51100000000000001,"5min":15.965,"15min":15.965},"last":0.312}]},{"osd":3,"last update":"Fri Feb 23 00:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":1.0820000000000001,"15min":0.65000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.432},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.0369999999999999,"15min":0.63700000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.58799999999999997,"5min":16.006,"15min":16.006},"last":0.96299999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.73399999999999999,"15min":0.63800000000000001},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84699999999999998,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.38300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83699999999999997,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.66900000000000004}]},{"osd":5,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.071,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86799999999999999,"5min":13.311,"15min":13.311},"last":0.628},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.113,"15min":0.74399999999999999},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":23.744,"15min":23.744},"last":0.65600000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.0740000000000001,"15min":0.80600000000000005},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78500000000000003,"5min":13.334,"15min":33.082999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.93400000000000005,"15min":0.70999999999999996},"min":{"1min":0.41199999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81699999999999995,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.48099999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.84099999999999997,"15min":0.78500000000000003},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.871,"5min":15.943,"15min":31.251999999999999},"last":0.504},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.091,"15min":0.76100000000000001},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.80000000000000004,"5min":23.785,"15min":23.785},"last":0.61199999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411465,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305904,"kb_used_data":8680,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439248,"statfs":{"total":95995035648,"available":95681789952,"internally_reserved":0,"allocated":8888320,"data_stored":6564788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":1.595,"15min":0.753},"min":{"1min":0.223,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.73699999999999999,"5min":56.509,"15min":56.509},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":1.464,"15min":0.71199999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.4890000000000001,"5min":31.498999999999999,"15min":31.498999999999999},"last":0.36699999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.996,"15min":0.56399999999999995},"min":{"1min":0.252,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.81100000000000005,"5min":19.324999999999999,"15min":19.324999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":1.4199999999999999,"15min":0.70099999999999996},"min":{"1min":0.23400000000000001,"5min":0.22900000000000001,"15min":0.216},"max":{"1min":0.64700000000000002,"5min":35.052,"15min":35.052},"last":0.32900000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:35:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":1.1739999999999999,"15min":0.68899999999999995},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.48499999999999999,"5min":43.5,"15min":43.5},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.91900000000000004,"15min":0.63500000000000001},"min":{"1min":0.26500000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.64200000000000002,"5min":27.414000000000001,"15min":27.414000000000001},"last":0.34300000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:36:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.72399999999999998,"15min":0.622},"min":{"1min":0.40300000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.72099999999999997,"5min":14.614000000000001,"15min":14.614000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63200000000000001,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.71299999999999997,"5min":4.6550000000000002,"15min":4.6550000000000002},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Feb 23 00:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.89700000000000002,"15min":0.66600000000000004},"min":{"1min":0.373,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":0.81000000000000005,"5min":17.765000000000001,"15min":17.765000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.76300000000000001,"15min":0.63600000000000001},"min":{"1min":0.34200000000000003,"5min":0.317,"15min":0.317},"max":{"1min":0.90500000000000003,"5min":14.589,"15min":14.589},"last":0.64800000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:35:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.66300000000000003,"15min":0.60399999999999998},"min":{"1min":0.42899999999999999,"5min":0.32700000000000001,"15min":0.317},"max":{"1min":0.86899999999999999,"5min":4.6120000000000001,"15min":4.6120000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.0760000000000001,"15min":0.748},"min":{"1min":0.39400000000000002,"5min":0.36699999999999999,"15min":0.34899999999999998},"max":{"1min":0.83299999999999996,"5min":20.574999999999999,"15min":20.574999999999999},"last":0.63600000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.752,"15min":1.0289999999999999},"min":{"1min":0.39200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81799999999999995,"5min":11.811999999999999,"15min":66.632000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.72299999999999998,"15min":0.80200000000000005},"min":{"1min":0.42099999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.745,"5min":11.781000000000001,"15min":19.603999999999999},"last":0.55300000000000005}]}]},{"osd":3,"up_from":0,"seq":111669149829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986304,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310296,"kb_used_data":4880,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93434856,"statfs":{"total":95995035648,"available":95677292544,"internally_reserved":0,"allocated":4997120,"data_stored":2565233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1739999999999999,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.68799999999999994,"5min":21.419,"15min":21.419},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":2.1200000000000001,"15min":1.2589999999999999},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.54100000000000004}]},{"osd":1,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":1.294,"15min":0.98999999999999999},"min":{"1min":0.35099999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78000000000000003,"5min":16.318000000000001,"15min":41.137999999999998},"last":0.38},{"interface":"front","average":{"1min":0.52800000000000002,"5min":1.0169999999999999,"15min":0.83599999999999997},"min":{"1min":0.371,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.85299999999999998,"5min":10.491,"15min":26.210000000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":1.2769999999999999,"15min":1.077},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.753,"5min":41.886000000000003,"15min":67.253},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":1.6339999999999999,"15min":1},"min":{"1min":0.35599999999999998,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":0.74099999999999999,"5min":41.927999999999997,"15min":41.927999999999997},"last":0.51200000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.4550000000000001,"15min":1.036},"min":{"1min":0.39800000000000002,"5min":0.35499999999999998,"15min":0.34000000000000002},"max":{"1min":0.67500000000000004,"5min":36.301000000000002,"15min":36.301000000000002},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.8360000000000001,"15min":1.107},"min":{"1min":0.39400000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.72199999999999998,"5min":42.064,"15min":42.064},"last":0.47799999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:35:37 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.85299999999999998,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.54600000000000004,"5min":14.242000000000001,"15min":14.242000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.82899999999999996,"15min":0.55800000000000005},"min":{"1min":0.34999999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.71499999999999997,"5min":14.124000000000001,"15min":14.124000000000001},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:36:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.66000000000000003,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.254,"15min":0.254},"max":{"1min":0.875,"5min":8.0730000000000004,"15min":8.0730000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.504,"5min":0.56799999999999995,"15min":0.629},"min":{"1min":0.34300000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67100000000000004,"5min":4.71,"15min":45.814999999999998},"last":0.629}]},{"osd":7,"last update":"Fri Feb 23 00:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.68600000000000005,"15min":0.60399999999999998},"min":{"1min":0.36899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68700000000000006,"5min":6.165,"15min":17.178999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.76200000000000001,"15min":0.64600000000000002},"min":{"1min":0.45400000000000001,"5min":0.35599999999999998,"15min":0.28299999999999997},"max":{"1min":0.95299999999999996,"5min":7.9690000000000003,"15min":14.212},"last":0.64200000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757371,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302120,"kb_used_data":4896,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93443032,"statfs":{"total":95995035648,"available":95685664768,"internally_reserved":0,"allocated":5013504,"data_stored":2565843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.1200000000000001,"15min":0.70699999999999996},"min":{"1min":0.36699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.115,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":1.028},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.498,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.622}]},{"osd":1,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":1.2969999999999999,"15min":0.752},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.546,"5min":38.838000000000001,"15min":38.838000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":2.524,"15min":1.1719999999999999},"min":{"1min":0.30199999999999999,"5min":0.251,"15min":0.251},"max":{"1min":1.4059999999999999,"5min":127.794,"15min":127.794},"last":0.49299999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.641,"15min":0.88500000000000001},"min":{"1min":0.33400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.248,"5min":37.442999999999998,"15min":37.442999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":1.0860000000000001,"15min":0.69999999999999996},"min":{"1min":0.34399999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.609,"5min":23.297999999999998,"15min":23.297999999999998},"last":0.745}]},{"osd":3,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":1.194,"15min":0.73499999999999999},"min":{"1min":0.36699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65500000000000003,"5min":24.414999999999999,"15min":24.414999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.6080000000000001,"15min":0.89400000000000002},"min":{"1min":0.39400000000000002,"5min":0.31,"15min":0.31},"max":{"1min":1.524,"5min":40.865000000000002,"15min":40.865000000000002},"last":0.69499999999999995}]},{"osd":4,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":1.5580000000000001,"15min":0.77500000000000002},"min":{"1min":0.28199999999999997,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":1.4790000000000001,"5min":103.13,"15min":103.13},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.57899999999999996,"15min":0.45300000000000001},"min":{"1min":0.29399999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.21,"5min":12.795999999999999,"15min":12.795999999999999},"last":0.36899999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.48899999999999999},"min":{"1min":0.32400000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":7.1440000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48199999999999998,"15min":0.81999999999999995},"min":{"1min":0.376,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":103.61499999999999},"last":0.42499999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.58999999999999997,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.379,"5min":7.1820000000000004,"15min":7.1820000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52300000000000002,"15min":0.51500000000000001},"min":{"1min":0.39900000000000002,"5min":0.317,"15min":0.317},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4366336,"data_stored":4000857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4231168,"data_stored":4000564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4000472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:36:30.420 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:30.420 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:29.977142+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:36:30.420 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:29.980389+0000 osd.7 (osd.7) 200 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:36:30.429 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-23T00:36:30.486 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:30.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:30.590 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:30.616 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:30.639 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:30.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:30.743 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:30.745 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:36:30.774 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:30.793 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:30.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:30.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:30.925 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:31.184 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:36:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:36:31.274 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:31.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:31.377 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:31.406 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:29.512015+0000 mgr.y (mgr.24437) 389 : audit [DBG] from='client.27237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:36:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:29.656798+0000 mgr.y (mgr.24437) 390 : cluster [DBG] pgmap v624: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:36:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:31.100934+0000 osd.4 (osd.4) 193 : cluster [DBG] 63.19 scrub starts 2024-02-23T00:36:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:31.102603+0000 osd.4 (osd.4) 194 : cluster [DBG] 63.19 scrub ok 2024-02-23T00:36:31.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:29.512015+0000 mgr.y (mgr.24437) 389 : audit [DBG] from='client.27237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:36:31.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:29.656798+0000 mgr.y (mgr.24437) 390 : cluster [DBG] pgmap v624: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:36:31.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:31.100934+0000 osd.4 (osd.4) 193 : cluster [DBG] 63.19 scrub starts 2024-02-23T00:36:31.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:31.102603+0000 osd.4 (osd.4) 194 : cluster [DBG] 63.19 scrub ok 2024-02-23T00:36:31.443 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:29.512015+0000 mgr.y (mgr.24437) 389 : audit [DBG] from='client.27237 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:36:31.443 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:29.656798+0000 mgr.y (mgr.24437) 390 : cluster [DBG] pgmap v624: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:36:31.443 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:31.100934+0000 osd.4 (osd.4) 193 : cluster [DBG] 63.19 scrub starts 2024-02-23T00:36:31.443 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:31.102603+0000 osd.4 (osd.4) 194 : cluster [DBG] 63.19 scrub ok 2024-02-23T00:36:31.444 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:31.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:31.553 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:31.580 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:31.697 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:36:32.129 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:32.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:32.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:32.233 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:32.264 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:32.291 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:32.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:32.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 -- 172.21.15.87:0/850966773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21400fe4e0 msgr2=0x7f21400f6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 --2- 172.21.15.87:0/850966773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21400fe4e0 0x7f21400f6380 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 rx=0x7f2134004870 tx=0x7f2134009cb0).stop 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 -- 172.21.15.87:0/850966773 shutdown_connections 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 --2- 172.21.15.87:0/850966773 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21400f6a40 0x7f21401049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 --2- 172.21.15.87:0/850966773 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21400fdb80 0x7f21400fdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 --2- 172.21.15.87:0/850966773 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21400fe4e0 0x7f21400f6380 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 -- 172.21.15.87:0/850966773 >> 172.21.15.87:0/850966773 conn(0x7f214006c300 msgr2=0x7f214006c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 -- 172.21.15.87:0/850966773 shutdown_connections 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 -- 172.21.15.87:0/850966773 wait complete. 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 librados: starting msgr at 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 librados: starting objecter 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.382+0000 7f2144a87700 1 Processor -- start 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f2144a87700 1 -- start start 2024-02-23T00:36:32.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f2144a87700 1 librados: setting wanted keys 2024-02-23T00:36:32.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f2144a87700 1 librados: calling monclient init 2024-02-23T00:36:32.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f2144a87700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21400f6a40 0x7f21400f7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:32.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f2144a87700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21400fdb80 0x7f21400f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:32.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f2144a87700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21400fe4e0 0x7f21400f8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f2144a87700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2140116620 con 0x7f21400fdb80 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f2144a87700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2140116790 con 0x7f21400f6a40 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f2144a87700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2140116950 con 0x7f21400fe4e0 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213dd9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21400fdb80 0x7f21400f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213ed9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21400fe4e0 0x7f21400f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213ed9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21400fe4e0 0x7f21400f8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:54022/0 (socket says 172.21.15.87:54022) 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213ed9d700 1 -- 172.21.15.87:0/335653359 learned_addr learned my addr 172.21.15.87:0/335653359 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213e59c700 1 --2- 172.21.15.87:0/335653359 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21400f6a40 0x7f21400f7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213ed9d700 1 -- 172.21.15.87:0/335653359 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21400f6a40 msgr2=0x7f21400f7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213ed9d700 1 --2- 172.21.15.87:0/335653359 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21400f6a40 0x7f21400f7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213ed9d700 1 -- 172.21.15.87:0/335653359 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21400fdb80 msgr2=0x7f21400f7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213ed9d700 1 --2- 172.21.15.87:0/335653359 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21400fdb80 0x7f21400f7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213ed9d700 1 -- 172.21.15.87:0/335653359 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130005040 con 0x7f21400fe4e0 2024-02-23T00:36:32.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213dd9b700 1 --2- 172.21.15.87:0/335653359 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21400fdb80 0x7f21400f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f213ed9d700 1 --2- 172.21.15.87:0/335653359 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21400fe4e0 0x7f21400f8030 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7f213800db50 tx=0x7f2138004f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f212f7fe700 10 client.?.objecter ms_handle_connect 0x7f21400fe4e0 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.383+0000 7f212f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.384+0000 7f212f7fe700 1 -- 172.21.15.87:0/335653359 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213800b750 con 0x7f21400fe4e0 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.384+0000 7f212f7fe700 1 -- 172.21.15.87:0/335653359 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f213800f040 con 0x7f21400fe4e0 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.384+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2134005040 con 0x7f21400fe4e0 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.384+0000 7f212f7fe700 1 -- 172.21.15.87:0/335653359 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138013a40 con 0x7f21400fe4e0 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.384+0000 7f2144a87700 10 client.37703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.384+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2140116f90 con 0x7f21400fe4e0 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.384+0000 7f2144a87700 1 librados: init done 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.385+0000 7f212f7fe700 1 -- 172.21.15.87:0/335653359 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f213800b8b0 con 0x7f21400fe4e0 2024-02-23T00:36:32.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.385+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214005f8d0 con 0x7f21400fe4e0 2024-02-23T00:36:32.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.385+0000 7f212f7fe700 1 --2- 172.21.15.87:0/335653359 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2128062c90 0x7f2128065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:32.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.385+0000 7f212f7fe700 1 -- 172.21.15.87:0/335653359 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(423..423 src has 173..423) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f213807ce60 con 0x7f21400fe4e0 2024-02-23T00:36:32.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.385+0000 7f212f7fe700 10 client.37703.objecter ms_dispatch 0x7f2140001550 osd_map(423..423 src has 173..423) v4 2024-02-23T00:36:32.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.385+0000 7f212f7fe700 3 client.37703.objecter handle_osd_map got epochs [423,423] > 0 2024-02-23T00:36:32.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.385+0000 7f212f7fe700 3 client.37703.objecter handle_osd_map decoding full epoch 423 2024-02-23T00:36:32.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.385+0000 7f213e59c700 1 --2- 172.21.15.87:0/335653359 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2128062c90 0x7f2128065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:32.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.386+0000 7f212f7fe700 20 client.37703.objecter dump_active .. 0 homeless 2024-02-23T00:36:32.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.386+0000 7f212f7fe700 10 client.37703.objecter ms_handle_connect 0x7f2128062c90 2024-02-23T00:36:32.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.386+0000 7f213e59c700 1 --2- 172.21.15.87:0/335653359 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2128062c90 0x7f2128065150 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f2140060950 tx=0x7f213000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:32.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.390+0000 7f212f7fe700 1 -- 172.21.15.87:0/335653359 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f21380502b0 con 0x7f21400fe4e0 2024-02-23T00:36:32.395 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:32.415 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:32.419 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:36:32.419 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:36:32.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:36:32.419 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:36:32.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.419 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25070.0:10 44.17 44:e8e7cce9:::oid_PoolQuotaPP_7:head [writefull 0~4096] snapc 0=[] ondisk+write+known_if_redirected+full_try e111)", 2024-02-23T00:36:32.419 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.464439+0000", 2024-02-23T00:36:32.419 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 328.95165611800002, 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022323852000000002, 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25070", 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3829298547", 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 10 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464439+0000", 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464439+0000", 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464436+0000", 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2904999999999998e-05 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464459+0000", 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8601e-05 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464478+0000", 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9809999999999998e-06 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464488+0000", 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1741000000000002e-05 2024-02-23T00:36:32.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464570+0000", 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013978800000000001 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464709+0000", 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015174700000000001 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,3", 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.464861+0000", 2024-02-23T00:36:32.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000406 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.465267+0000", 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0066204979999999998 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.471888+0000", 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014817204 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.486705+0000", 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.167e-05 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.486736+0000", 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6329000000000002e-05 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.486763+0000", 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.022323852000000002 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:3 63.11 63:8d290297:test-rados-api-smithi087-82091-8::foo:head [writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:07.929685+0000", 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 324.48640978700001, 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110693868, 2024-02-23T00:36:32.426 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 3 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929685+0000", 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929685+0000", 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929684+0000", 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5640000000000002e-06 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929690+0000", 2024-02-23T00:36:32.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9799999999999998e-06 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929698+0000", 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9900000000000002e-06 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929704+0000", 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9026999999999998e-05 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929743+0000", 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0059e-05 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929833+0000", 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6689999999999998e-05 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,5", 2024-02-23T00:36:32.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.929930+0000", 2024-02-23T00:36:32.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038132900000000002 2024-02-23T00:36:32.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:36:32.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.930311+0000", 2024-02-23T00:36:32.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018368799999999999 2024-02-23T00:36:32.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.930495+0000", 2024-02-23T00:36:32.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.109798312 2024-02-23T00:36:32.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:08.040293+0000", 2024-02-23T00:36:32.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7428000000000001e-05 2024-02-23T00:36:32.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:08.040341+0000", 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8401e-05 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:08.040379+0000", 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.110693868 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24815.0:1113 19.16 19:6cac518f:::0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e148)", 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:39.054396+0000", 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 293.36169898000003, 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043740108999999999, 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24815", 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3778460189", 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1113 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054396+0000", 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054396+0000", 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054395+0000", 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6200000000000001e-06 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054399+0000", 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7519999999999996e-06 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054407+0000", 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8739999999999997e-06 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054411+0000", 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4289999999999999e-05 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054445+0000", 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6217000000000002e-05 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.054481+0000", 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.042577932999999998 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097059+0000", 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7280000000000005e-05 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097136+0000", 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012174400000000001 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 5,7", 2024-02-23T00:36:32.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097258+0000", 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036222099999999997 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097620+0000", 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000203802 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097824+0000", 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017177300000000001 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.097996+0000", 2024-02-23T00:36:32.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6755000000000002e-05 2024-02-23T00:36:32.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.098043+0000", 2024-02-23T00:36:32.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3552000000000001e-05 2024-02-23T00:36:32.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.098136+0000", 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043740108999999999 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24815.0:1117 19.1d 19:bd63b0f1:::8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e148)", 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:39.103414+0000", 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 293.312681016, 2024-02-23T00:36:32.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.25896939400000002, 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24815", 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3778460189", 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1117 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103414+0000", 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103414+0000", 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103413+0000", 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.467e-06 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103417+0000", 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5680000000000002e-06 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103424+0000", 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8120000000000001e-06 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103428+0000", 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.464e-05 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103463+0000", 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5056999999999996e-05 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for missing object", 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.103538+0000", 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00099724100000000006 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.104535+0000", 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3939000000000002e-05 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for degraded object", 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.104569+0000", 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.25692525199999999 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.361494+0000", 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6981000000000002e-05 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.361551+0000", 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4629000000000006e-05 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.361636+0000", 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00035837500000000002 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.361994+0000", 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00025748299999999997 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.362252+0000", 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.371e-05 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.362295+0000", 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.9400999999999999e-05 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.362325+0000", 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8569000000000001e-05 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.362383+0000", 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.25896939400000002 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.7.0:300 63.18 e182/114 63:1a553bd4:test-rados-api-smithi087-82091-14::foo26:7 v 182'34, mlcod=148'30)", 2024-02-23T00:36:32.449 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:13.000057+0000", 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 259.416038125, 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086960830000000003, 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000057+0000", 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000057+0000", 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000057+0000", 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5390000000000001e-06 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000061+0000", 2024-02-23T00:36:32.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5890000000000002e-06 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000067+0000", 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3040000000000001e-06 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000073+0000", 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109073 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000182+0000", 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9967999999999999e-05 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000202+0000", 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085273399999999996 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.008729+0000", 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3555999999999999e-05 2024-02-23T00:36:32.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.008753+0000", 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086960830000000003 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.7.0:302 63.18 e182/114 63:1e5f7139:test-rados-api-smithi087-82091-14::foo83:7 v 182'36, mlcod=148'30)", 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:13.000084+0000", 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 259.41601141899997, 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086929699999999995, 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.454 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000084+0000", 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000084+0000", 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000083+0000", 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3470000000000004e-06 2024-02-23T00:36:32.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000090+0000", 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1109999999999999e-06 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000095+0000", 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.946e-06 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.000099+0000", 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0032143689999999999 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.003313+0000", 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4056000000000001e-05 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.003337+0000", 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054193050000000001 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.008756+0000", 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0129999999999999e-05 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.008777+0000", 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086929699999999995 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.458 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:95 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032849+0000", 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 228.383245625, 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084569580000000005, 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 95 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032849+0000", 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032849+0000", 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032849+0000", 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.558e-06 2024-02-23T00:36:32.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032855+0000", 2024-02-23T00:36:32.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.215e-06 2024-02-23T00:36:32.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032860+0000", 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9570000000000002e-06 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032868+0000", 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0083560749999999993 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041224+0000", 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7444999999999998e-05 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041261+0000", 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5086999999999997e-05 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041306+0000", 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084569580000000005 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:96 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:36:32.463 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032873+0000", 2024-02-23T00:36:32.464 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 228.383221883, 2024-02-23T00:36:32.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085372469999999995, 2024-02-23T00:36:32.464 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.464 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.464 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.464 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:36:32.464 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:36:32.464 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 96 2024-02-23T00:36:32.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.465 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032873+0000", 2024-02-23T00:36:32.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032873+0000", 2024-02-23T00:36:32.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:32.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032873+0000", 2024-02-23T00:36:32.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2430000000000001e-06 2024-02-23T00:36:32.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032876+0000", 2024-02-23T00:36:32.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0769999999999998e-06 2024-02-23T00:36:32.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032881+0000", 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1320000000000004e-06 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032885+0000", 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0084456170000000007 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.470 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041331+0000", 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5104999999999997e-05 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041366+0000", 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4536000000000002e-05 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041410+0000", 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085372469999999995 2024-02-23T00:36:32.471 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:97 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032912+0000", 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 228.38318267400001, 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086066719999999992, 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.472 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 97 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032912+0000", 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032912+0000", 2024-02-23T00:36:32.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032912+0000", 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8670000000000004e-06 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032915+0000", 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.601e-06 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032922+0000", 2024-02-23T00:36:32.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0520000000000001e-06 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032927+0000", 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0085083369999999995 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041435+0000", 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8047999999999999e-05 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041473+0000", 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5596999999999997e-05 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041519+0000", 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086066719999999992 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:98 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:36:32.476 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032933+0000", 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 228.38316240200001, 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087421370000000005, 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 98 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032933+0000", 2024-02-23T00:36:32.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032933+0000", 2024-02-23T00:36:32.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:32.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032932+0000", 2024-02-23T00:36:32.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.147e-05 2024-02-23T00:36:32.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032944+0000", 2024-02-23T00:36:32.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9630000000000003e-06 2024-02-23T00:36:32.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032949+0000", 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.262e-06 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032954+0000", 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0086076090000000004 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041561+0000", 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8508999999999999e-05 2024-02-23T00:36:32.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041610+0000", 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4911000000000002e-05 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041675+0000", 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087421370000000005 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.481 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:99 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:36:32.482 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032959+0000", 2024-02-23T00:36:32.482 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 228.383136337, 2024-02-23T00:36:32.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088395780000000007, 2024-02-23T00:36:32.482 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.482 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:36:32.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:36:32.482 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 99 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032959+0000", 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032959+0000", 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032958+0000", 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3379e-05 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032972+0000", 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.443e-06 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032978+0000", 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3379999999999998e-06 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032982+0000", 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0087279739999999995 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041710+0000", 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7415e-05 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041748+0000", 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0489999999999999e-05 2024-02-23T00:36:32.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041798+0000", 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088395780000000007 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:100 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.032998+0000", 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 228.38309658700001, 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0089354540000000007, 2024-02-23T00:36:32.486 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 100 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032998+0000", 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032998+0000", 2024-02-23T00:36:32.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:32.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.032998+0000", 2024-02-23T00:36:32.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.563e-06 2024-02-23T00:36:32.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033001+0000", 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1620000000000002e-06 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033009+0000", 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6940000000000003e-06 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033013+0000", 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088147120000000006 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041828+0000", 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4904000000000002e-05 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041883+0000", 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0952000000000003e-05 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041934+0000", 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0089354540000000007 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:101 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.033028+0000", 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 228.38306693999999, 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090205070000000005, 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.491 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:36:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:36:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 101 2024-02-23T00:36:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033028+0000", 2024-02-23T00:36:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033028+0000", 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033027+0000", 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2650000000000001e-06 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033031+0000", 2024-02-23T00:36:32.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7240000000000004e-06 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033040+0000", 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.425e-06 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033045+0000", 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0089160109999999997 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041961+0000", 2024-02-23T00:36:32.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7178000000000002e-05 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.041998+0000", 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0677999999999998e-05 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.042049+0000", 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0090205070000000005 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37094.0:102 147.6 147:602f83fe:::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e213)", 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:44.033051+0000", 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 228.38304439999999, 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0091115729999999995, 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37094", 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/450042771", 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 102 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033051+0000", 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033051+0000", 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033050+0000", 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.089e-06 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033053+0000", 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5600000000000001e-06 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033059+0000", 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3359999999999997e-06 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.033063+0000", 2024-02-23T00:36:32.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.009004115 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.042067+0000", 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4490000000000003e-05 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.042112+0000", 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0618000000000002e-05 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:44.042162+0000", 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0091115729999999995 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:2082 160.3 160:c0773ebc:test-rados-api-smithi087-82091-26::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e238)", 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:09.020460+0000", 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 203.39563504700001, 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018878523000000001, 2024-02-23T00:36:32.500 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.501 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.501 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.501 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:36:32.501 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:36:32.501 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2082 2024-02-23T00:36:32.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.501 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020460+0000", 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020460+0000", 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020458+0000", 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7559999999999994e-06 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020468+0000", 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3088000000000001e-05 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020481+0000", 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7870000000000002e-06 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020489+0000", 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8291000000000002e-05 2024-02-23T00:36:32.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020537+0000", 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00027761399999999997 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.020815+0000", 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014981089 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.035796+0000", 2024-02-23T00:36:32.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5717999999999994e-05 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.035891+0000", 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000521332 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.036413+0000", 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5217999999999997e-05 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.036488+0000", 2024-02-23T00:36:32.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028505689999999998 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:09.039339+0000", 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018878523000000001 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37277.0:2 173.1s0 173:9294e480:nspace::foo:head [write 0~64] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:33.036033+0000", 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 179.38006239699999, 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13658257500000001, 2024-02-23T00:36:32.506 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37277", 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2105660432", 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036033+0000", 2024-02-23T00:36:32.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036033+0000", 2024-02-23T00:36:32.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:36:32.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036030+0000", 2024-02-23T00:36:32.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7399999999999993e-06 2024-02-23T00:36:32.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036039+0000", 2024-02-23T00:36:32.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3021e-05 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036052+0000", 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0880000000000001e-06 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036061+0000", 2024-02-23T00:36:32.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6910000000000003e-05 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036108+0000", 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3079999999999999e-06 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.036112+0000", 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.135140337 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.171252+0000", 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011767799999999999 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.171370+0000", 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041097099999999999 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:32.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.171780+0000", 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041141700000000002 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.172192+0000", 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00037785 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.172570+0000", 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5453e-05 2024-02-23T00:36:32.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:33.172615+0000", 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.13658257500000001 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26952.0:1 189.6s0 189:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e302)", 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:13.039879+0000", 2024-02-23T00:36:32.514 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 139.37621594199999, 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0195118649999999, 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26952", 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3617910981", 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039879+0000", 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039879+0000", 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039877+0000", 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8540000000000003e-06 2024-02-23T00:36:32.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039886+0000", 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1123e-05 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039897+0000", 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0100000000000001e-06 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039906+0000", 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9648999999999997e-05 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039976+0000", 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0049999999999998e-06 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:13.039980+0000", 2024-02-23T00:36:32.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.017329613 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.057309+0000", 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000148116 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.057457+0000", 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00072641600000000002 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.058184+0000", 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077851899999999998 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.058962+0000", 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033970199999999999 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.059302+0000", 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8896999999999999e-05 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:14.059391+0000", 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0195118649999999 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.521 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:5419 63.11 63:8993aba8:test-rados-api-smithi087-82091-50::bar:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:35:23.102876+0000", 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 69.313219410000002, 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0094787380000000004, 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 5419 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102876+0000", 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102876+0000", 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102873+0000", 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6339999999999996e-06 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102883+0000", 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1302e-05 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102894+0000", 2024-02-23T00:36:32.524 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2899999999999997e-06 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102902+0000", 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7306000000000003e-05 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102959+0000", 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000156081 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.103115+0000", 2024-02-23T00:36:32.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018007400000000001 2024-02-23T00:36:32.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,5", 2024-02-23T00:36:32.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.103295+0000", 2024-02-23T00:36:32.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00051166600000000003 2024-02-23T00:36:32.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.103807+0000", 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036577000000000002 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.104173+0000", 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0081160550000000005 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.112289+0000", 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9344999999999997e-05 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.112328+0000", 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6364e-05 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.112354+0000", 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0094787380000000004 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37589.0:1 205.3s0 205:cbb9903e:::test_obj:head [omap-set-vals,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected e375)", 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:35:26.083964+0000", 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 66.332131211000004, 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.08008338, 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.529 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37589", 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2818687675", 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:26.083964+0000", 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.530 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:26.083964+0000", 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:26.083962+0000", 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9199999999999993e-06 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:26.083971+0000", 2024-02-23T00:36:32.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2459999999999999e-05 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:26.083983+0000", 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0010000000000001e-06 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:26.083991+0000", 2024-02-23T00:36:32.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7405999999999999e-05 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:26.084049+0000", 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3900000000000002e-06 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:26.084052+0000", 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.013749271 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:27.097801+0000", 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015498199999999999 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:27.097956+0000", 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.066091026999999997 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.534 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:27.164047+0000", 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.08008338 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:5597 63.11 63:89e9bdf1:test-rados-api-smithi087-82091-52::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e392)", 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:35:46.208724+0000", 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 46.207370771000001, 2024-02-23T00:36:32.535 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010976323, 2024-02-23T00:36:32.536 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:32.536 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:32.536 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:32.536 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:36:32.536 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:36:32.536 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 5597 2024-02-23T00:36:32.536 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.536 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:32.536 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.208724+0000", 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.208724+0000", 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.208722+0000", 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3005e-05 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.208735+0000", 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.878e-05 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.208754+0000", 2024-02-23T00:36:32.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9029999999999999e-06 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.208764+0000", 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8881000000000001e-05 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.208823+0000", 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015780300000000001 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.208981+0000", 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018199899999999999 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,5", 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.209163+0000", 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0016231730000000001 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:36:32.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.210786+0000", 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014469719999999999 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.212233+0000", 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070164049999999999 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.219249+0000", 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00040338700000000001 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.219653+0000", 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8004000000000003e-05 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:46.219701+0000", 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010976323 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:32.542 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:32.543 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:36:32.543 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:30.913067+0000 mgr.y (mgr.24437) 391 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:32.543 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:32.089107+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-02-23T00:36:32.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:32.095411+0000 osd.4 (osd.4) 195 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:36:32.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:32.098276+0000 osd.4 (osd.4) 196 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:36:32.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:30.913067+0000 mgr.y (mgr.24437) 391 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:32.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:32.089107+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-02-23T00:36:32.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:32.095411+0000 osd.4 (osd.4) 195 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:36:32.544 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:32.098276+0000 osd.4 (osd.4) 196 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:36:32.545 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:36:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:32] "GET /metrics HTTP/1.1" 200 188128 "" "Prometheus/2.33.4" 2024-02-23T00:36:32.545 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:30.913067+0000 mgr.y (mgr.24437) 391 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:32.545 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:32.089107+0000 mon.a (mon.0) 2583 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-02-23T00:36:32.545 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:32.095411+0000 osd.4 (osd.4) 195 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:36:32.545 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:32.098276+0000 osd.4 (osd.4) 196 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:36:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:32.547 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:32.547 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:32.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:32.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:32.586+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f21400f8b30 con 0x7f21400fe4e0 2024-02-23T00:36:32.647 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:32.675 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:32.798 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:32.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:32.915 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:32.942 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:33.136 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-23T00:36:33.140 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:33.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:33.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.193+0000 7f212f7fe700 1 -- 172.21.15.87:0/335653359 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v424) v1 ==== 101+0+0 (secure 0 0 0) 0x7f2138018070 con 0x7f21400fe4e0 2024-02-23T00:36:33.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 10 librados: watch_flush enter 2024-02-23T00:36:33.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 10 librados: watch_flush exit 2024-02-23T00:36:33.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 20 client.37703.objecter shutdown clearing up homeless session... 2024-02-23T00:36:33.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 10 client.37703.objecter successfully canceled tick 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2128062c90 msgr2=0x7f2128065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 --2- 172.21.15.87:0/335653359 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2128062c90 0x7f2128065150 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 rx=0x7f2140060950 tx=0x7f213000b040).stop 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21400fe4e0 msgr2=0x7f21400f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 --2- 172.21.15.87:0/335653359 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21400fe4e0 0x7f21400f8030 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 rx=0x7f213800db50 tx=0x7f2138004f00).stop 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 shutdown_connections 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 --2- 172.21.15.87:0/335653359 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2128062c90 0x7f2128065150 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 --2- 172.21.15.87:0/335653359 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21400fe4e0 0x7f21400f8030 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 --2- 172.21.15.87:0/335653359 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21400f6a40 0x7f21400f7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 --2- 172.21.15.87:0/335653359 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21400fdb80 0x7f21400f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 >> 172.21.15.87:0/335653359 conn(0x7f214006c300 msgr2=0x7f21400f58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 shutdown_connections 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 -- 172.21.15.87:0/335653359 wait complete. 2024-02-23T00:36:33.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:33.196+0000 7f2144a87700 1 librados: shutdown 2024-02-23T00:36:33.200 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:36:33.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:31.657772+0000 mgr.y (mgr.24437) 392 : cluster [DBG] pgmap v625: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:36:33.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:32.147919+0000 osd.6 (osd.6) 154 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:32.150241+0000 osd.6 (osd.6) 155 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:32.198398+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:32.199758+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:32.588540+0000 mon.c (mon.2) 433 : audit [INF] from='client.? 172.21.15.87:0/335653359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:32.589002+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.027485+0000 osd.7 (osd.7) 201 : cluster [DBG] 22.c scrub starts 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.028163+0000 osd.7 (osd.7) 202 : cluster [DBG] 22.c scrub ok 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:31.657772+0000 mgr.y (mgr.24437) 392 : cluster [DBG] pgmap v625: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:32.147919+0000 osd.6 (osd.6) 154 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:32.150241+0000 osd.6 (osd.6) 155 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:36:33.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:32.198398+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:36:33.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:32.199758+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:36:33.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:32.588540+0000 mon.c (mon.2) 433 : audit [INF] from='client.? 172.21.15.87:0/335653359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:33.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:32.589002+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:33.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.027485+0000 osd.7 (osd.7) 201 : cluster [DBG] 22.c scrub starts 2024-02-23T00:36:33.227 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.028163+0000 osd.7 (osd.7) 202 : cluster [DBG] 22.c scrub ok 2024-02-23T00:36:33.244 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:33.270 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:33.289 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:33.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:33.394 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:36:33.420 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:36:33.423 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:31.657772+0000 mgr.y (mgr.24437) 392 : cluster [DBG] pgmap v625: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:36:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:32.147919+0000 osd.6 (osd.6) 154 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:36:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:32.150241+0000 osd.6 (osd.6) 155 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:36:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:32.198398+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:36:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:32.199758+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:36:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:32.588540+0000 mon.c (mon.2) 433 : audit [INF] from='client.? 172.21.15.87:0/335653359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:32.589002+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.027485+0000 osd.7 (osd.7) 201 : cluster [DBG] 22.c scrub starts 2024-02-23T00:36:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.028163+0000 osd.7 (osd.7) 202 : cluster [DBG] 22.c scrub ok 2024-02-23T00:36:33.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:36:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:36:34.172 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:34.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:36:34.276 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:34.306 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:34.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:34.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:34.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.003449+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:36:34.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.004256+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:36:34.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.158952+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:36:34.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.159934+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:36:34.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.188279+0000 mon.a (mon.0) 2585 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:36:34.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:33.191446+0000 mon.a (mon.0) 2586 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:36:34.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.191492+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.193261+0000 osd.6 (osd.6) 156 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.196203+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.003449+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.004256+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.158952+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.159934+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.188279+0000 mon.a (mon.0) 2585 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:33.191446+0000 mon.a (mon.0) 2586 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.191492+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.193261+0000 osd.6 (osd.6) 156 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:36:34.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.196203+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:36:34.523 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:34.551 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:34.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.003449+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:36:34.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.004256+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:36:34.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.158952+0000 osd.5 (osd.5) 169 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:36:34.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.159934+0000 osd.5 (osd.5) 170 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:36:34.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.188279+0000 mon.a (mon.0) 2585 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:36:34.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:33.191446+0000 mon.a (mon.0) 2586 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:36:34.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.191492+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-02-23T00:36:34.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.193261+0000 osd.6 (osd.6) 156 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:36:34.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.196203+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:36:35.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 -- 172.21.15.87:0/162667583 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2080fcbb0 msgr2=0x7fd2080fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/162667583 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2080fcbb0 0x7fd2080fcf90 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 rx=0x7fd1f8004870 tx=0x7fd1f8009cb0).stop 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 -- 172.21.15.87:0/162667583 shutdown_connections 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/162667583 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2080f7400 0x7fd208103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/162667583 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2080fcbb0 0x7fd2080fcf90 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/162667583 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2080fd510 0x7fd2080f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 -- 172.21.15.87:0/162667583 >> 172.21.15.87:0/162667583 conn(0x7fd2080f2820 msgr2=0x7fd2080f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 -- 172.21.15.87:0/162667583 shutdown_connections 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 -- 172.21.15.87:0/162667583 wait complete. 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 librados: starting msgr at 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 librados: starting objecter 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 Processor -- start 2024-02-23T00:36:35.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.067+0000 7fd20e2bf700 1 -- start start 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20e2bf700 1 librados: setting wanted keys 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20e2bf700 1 librados: calling monclient init 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20e2bf700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2080f7400 0x7fd20810e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20e2bf700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2080fcbb0 0x7fd20810e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20e2bf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2080fd510 0x7fd208108350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20e2bf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd208108840 con 0x7fd2080fd510 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20e2bf700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2081089b0 con 0x7fd2080fcbb0 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20e2bf700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd208108b70 con 0x7fd2080f7400 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd207fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2080f7400 0x7fd20810e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd207fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2080f7400 0x7fd20810e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:54054/0 (socket says 172.21.15.87:54054) 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20c85c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2080fd510 0x7fd208108350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd207fff700 1 -- 172.21.15.87:0/2044816031 learned_addr learned my addr 172.21.15.87:0/2044816031 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:35.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd2077fe700 1 --2- 172.21.15.87:0/2044816031 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2080fcbb0 0x7fd20810e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd207fff700 1 -- 172.21.15.87:0/2044816031 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2080fcbb0 msgr2=0x7fd20810e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd207fff700 1 --2- 172.21.15.87:0/2044816031 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd2080fcbb0 0x7fd20810e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd207fff700 1 -- 172.21.15.87:0/2044816031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2080fd510 msgr2=0x7fd208108350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd207fff700 1 --2- 172.21.15.87:0/2044816031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2080fd510 0x7fd208108350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd207fff700 1 -- 172.21.15.87:0/2044816031 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2000097c0 con 0x7fd2080f7400 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd20c85c700 1 --2- 172.21.15.87:0/2044816031 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd2080fd510 0x7fd208108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd207fff700 1 --2- 172.21.15.87:0/2044816031 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2080f7400 0x7fd20810e1b0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7fd1f8009eb0 tx=0x7fd1f800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd205775700 10 client.?.objecter ms_handle_connect 0x7fd2080f7400 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd205775700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.068+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f800e070 con 0x7fd2080f7400 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.069+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1f8013da0 con 0x7fd2080f7400 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.069+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f8005040 con 0x7fd2080f7400 2024-02-23T00:36:35.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.069+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8019070 con 0x7fd2080f7400 2024-02-23T00:36:35.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.069+0000 7fd20e2bf700 10 client.37709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:35.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.069+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd208109010 con 0x7fd2080f7400 2024-02-23T00:36:35.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.069+0000 7fd20e2bf700 1 librados: init done 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.070+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd208109200 con 0x7fd2080f7400 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.070+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd1f801e420 con 0x7fd2080f7400 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd205775700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1f4062c90 0x7fd1f4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(424..424 src has 173..424) v4 ==== 8150+0+0 (secure 0 0 0) 0x7fd1f807dcf0 con 0x7fd2080f7400 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 osd_map(424..424 src has 173..424) v4 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd205775700 3 client.37709.objecter handle_osd_map got epochs [424,424] > 0 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd205775700 3 client.37709.objecter handle_osd_map decoding full epoch 424 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd2077fe700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1f4062c90 0x7fd1f4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd205775700 20 client.37709.objecter dump_active .. 0 homeless 2024-02-23T00:36:35.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v424) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd1f80803c0 con 0x7fd2080f7400 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd205775700 10 client.37709.objecter ms_handle_connect 0x7fd1f4062c90 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd2077fe700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1f4062c90 0x7fd1f4065150 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7fd208109af0 tx=0x7fd1f000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 1 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd2080ffed0 0x7fd2081002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd20805fa60 osd=0 3 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20805fa60 osd=0 3 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd20805fa60 osd=0 3 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 1 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd20805fa60 osd=0 3 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 0 1 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20805fa60 osd=0 4 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 10 client.37709.objecter _send_command 1 2024-02-23T00:36:35.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.87:6801/2153191863 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd20805ee80 con 0x7fd2080ffed0 2024-02-23T00:36:35.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20c85c700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd2080ffed0 0x7fd2081002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd20c85c700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd2080ffed0 0x7fd2081002b0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.071+0000 7fd205775700 10 client.37709.objecter ms_handle_connect 0x7fd2080ffed0 2024-02-23T00:36:35.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.072+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.0 v2:172.21.15.87:6801/2153191863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd20805ee80 con 0x7fd2080ffed0 2024-02-23T00:36:35.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.072+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:36:35.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.072+0000 7fd205775700 10 client.37709.objecter _finish_command 1 = system:0 2024-02-23T00:36:35.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.072+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd20805fa60 osd=0 4 2024-02-23T00:36:35.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.072+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 0 1 2024-02-23T00:36:35.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:35.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 2 2024-02-23T00:36:35.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd20805fa60 osd=0 3 2024-02-23T00:36:35.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20805fa60 osd=0 3 2024-02-23T00:36:35.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd20805fa60 osd=0 3 2024-02-23T00:36:35.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 2 2024-02-23T00:36:35.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd20805fa60 osd=0 3 2024-02-23T00:36:35.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 0 2 2024-02-23T00:36:35.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20805fa60 osd=0 4 2024-02-23T00:36:35.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 10 client.37709.objecter _send_command 2 2024-02-23T00:36:35.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.087+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.87:6801/2153191863 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd2081060e0 con 0x7fd2080ffed0 2024-02-23T00:36:35.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.0 v2:172.21.15.87:6801/2153191863 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd2081060e0 con 0x7fd2080ffed0 2024-02-23T00:36:35.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:35.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd205775700 10 client.37709.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd20805fa60 osd=0 4 2024-02-23T00:36:35.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 0 2 2024-02-23T00:36:35.090 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:36:35.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:35.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 3 2024-02-23T00:36:35.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6805/685354171 conn(0x7fd2080f4dd0 0x7fd2080273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd2080f4c60 osd=1 3 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2080f4c60 osd=1 3 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd2080f4c60 osd=1 3 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 3 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd2080f4c60 osd=1 3 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 1 3 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2080f4c60 osd=1 4 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 10 client.37709.objecter _send_command 3 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2080fa800 con 0x7fd2080f4dd0 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd207fff700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6805/685354171 conn(0x7fd2080f4dd0 0x7fd2080273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd207fff700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6805/685354171 conn(0x7fd2080f4dd0 0x7fd2080273c0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.089+0000 7fd205775700 10 client.37709.objecter ms_handle_connect 0x7fd2080f4dd0 2024-02-23T00:36:35.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.090+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd2080fa800 con 0x7fd2080f4dd0 2024-02-23T00:36:35.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.090+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:36:35.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.090+0000 7fd205775700 10 client.37709.objecter _finish_command 3 = system:0 2024-02-23T00:36:35.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.090+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd2080f4c60 osd=1 4 2024-02-23T00:36:35.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.090+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 1 3 2024-02-23T00:36:35.099 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:35.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:35.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:35.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 4 2024-02-23T00:36:35.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd2080f4c60 osd=1 3 2024-02-23T00:36:35.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2080f4c60 osd=1 3 2024-02-23T00:36:35.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd2080f4c60 osd=1 3 2024-02-23T00:36:35.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 4 2024-02-23T00:36:35.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd2080f4c60 osd=1 3 2024-02-23T00:36:35.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 1 4 2024-02-23T00:36:35.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2080f4c60 osd=1 4 2024-02-23T00:36:35.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 10 client.37709.objecter _send_command 4 2024-02-23T00:36:35.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.105+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd20805a040 con 0x7fd2080f4dd0 2024-02-23T00:36:35.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd20805a040 con 0x7fd2080f4dd0 2024-02-23T00:36:35.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:35.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd205775700 10 client.37709.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd2080f4c60 osd=1 4 2024-02-23T00:36:35.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 1 4 2024-02-23T00:36:35.109 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:36:35.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:35.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 5 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd208028590 0x7fd20811e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208059360 osd=2 3 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208059360 osd=2 3 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208059360 osd=2 3 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 5 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd208059360 osd=2 3 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 2 5 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208059360 osd=2 4 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 10 client.37709.objecter _send_command 5 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd208059560 con 0x7fd208028590 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd2077fe700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd208028590 0x7fd20811e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd205775700 10 client.37709.objecter ms_handle_connect 0x7fd208028590 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.107+0000 7fd2077fe700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd208028590 0x7fd20811e590 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.108+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd208059560 con 0x7fd208028590 2024-02-23T00:36:35.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.108+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:36:35.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.108+0000 7fd205775700 10 client.37709.objecter _finish_command 5 = system:0 2024-02-23T00:36:35.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.108+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd208059360 osd=2 4 2024-02-23T00:36:35.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.108+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 2 5 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 6 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208059360 osd=2 3 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208059360 osd=2 3 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208059360 osd=2 3 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 6 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd208059360 osd=2 3 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 2 6 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208059360 osd=2 4 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 10 client.37709.objecter _send_command 6 2024-02-23T00:36:35.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.123+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd208027af0 con 0x7fd208028590 2024-02-23T00:36:35.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd208027af0 con 0x7fd208028590 2024-02-23T00:36:35.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:35.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd205775700 10 client.37709.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd208059360 osd=2 4 2024-02-23T00:36:35.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 2 6 2024-02-23T00:36:35.126 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:36:35.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 7 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6813/128059790 conn(0x7fd20811fc00 0x7fd208121ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd2081177c0 osd=3 3 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2081177c0 osd=3 3 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd2081177c0 osd=3 3 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 7 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd2081177c0 osd=3 3 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 3 7 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2081177c0 osd=3 4 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 10 client.37709.objecter _send_command 7 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd208117f20 con 0x7fd20811fc00 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.124+0000 7fd20c85c700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6813/128059790 conn(0x7fd20811fc00 0x7fd208121ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.125+0000 7fd205775700 10 client.37709.objecter ms_handle_connect 0x7fd20811fc00 2024-02-23T00:36:35.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.125+0000 7fd20c85c700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6813/128059790 conn(0x7fd20811fc00 0x7fd208121ff0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.125+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd208117f20 con 0x7fd20811fc00 2024-02-23T00:36:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.125+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:36:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.125+0000 7fd205775700 10 client.37709.objecter _finish_command 7 = system:0 2024-02-23T00:36:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.125+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd2081177c0 osd=3 4 2024-02-23T00:36:35.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.125+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 3 7 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 8 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd2081177c0 osd=3 3 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2081177c0 osd=3 3 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd2081177c0 osd=3 3 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 8 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd2081177c0 osd=3 3 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 3 8 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2081177c0 osd=3 4 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 10 client.37709.objecter _send_command 8 2024-02-23T00:36:35.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.140+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd208117230 con 0x7fd20811fc00 2024-02-23T00:36:35.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd208117230 con 0x7fd20811fc00 2024-02-23T00:36:35.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:35.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd205775700 10 client.37709.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd2081177c0 osd=3 4 2024-02-23T00:36:35.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 3 8 2024-02-23T00:36:35.143 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 9 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd20811f760 0x7fd208125670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208117230 osd=4 3 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117230 osd=4 3 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208117230 osd=4 3 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 9 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd208117230 osd=4 3 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 4 9 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117230 osd=4 4 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 10 client.37709.objecter _send_command 9 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.141+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd208117540 con 0x7fd20811f760 2024-02-23T00:36:35.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.142+0000 7fd207fff700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd20811f760 0x7fd208125670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.142+0000 7fd205775700 10 client.37709.objecter ms_handle_connect 0x7fd20811f760 2024-02-23T00:36:35.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.142+0000 7fd207fff700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd20811f760 0x7fd208125670 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.143+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd208117540 con 0x7fd20811f760 2024-02-23T00:36:35.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.143+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:36:35.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.143+0000 7fd205775700 10 client.37709.objecter _finish_command 9 = system:0 2024-02-23T00:36:35.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.143+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd208117230 osd=4 4 2024-02-23T00:36:35.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.143+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 4 9 2024-02-23T00:36:35.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:35.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 10 2024-02-23T00:36:35.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208117230 osd=4 3 2024-02-23T00:36:35.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117230 osd=4 3 2024-02-23T00:36:35.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208117230 osd=4 3 2024-02-23T00:36:35.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 10 2024-02-23T00:36:35.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd208117230 osd=4 3 2024-02-23T00:36:35.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 4 10 2024-02-23T00:36:35.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117230 osd=4 4 2024-02-23T00:36:35.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 10 client.37709.objecter _send_command 10 2024-02-23T00:36:35.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.157+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd208117540 con 0x7fd20811f760 2024-02-23T00:36:35.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd208117540 con 0x7fd20811f760 2024-02-23T00:36:35.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:35.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd205775700 10 client.37709.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd208117230 osd=4 4 2024-02-23T00:36:35.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 4 10 2024-02-23T00:36:35.161 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 11 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd2081260b0 0x7fd208126510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208117540 osd=5 3 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117540 osd=5 3 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208117540 osd=5 3 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 11 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd208117540 osd=5 3 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 5 11 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117540 osd=5 4 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 10 client.37709.objecter _send_command 11 2024-02-23T00:36:35.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd208128130 con 0x7fd2081260b0 2024-02-23T00:36:35.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.159+0000 7fd2077fe700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd2081260b0 0x7fd208126510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.160+0000 7fd205775700 10 client.37709.objecter ms_handle_connect 0x7fd2081260b0 2024-02-23T00:36:35.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.160+0000 7fd2077fe700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd2081260b0 0x7fd208126510 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.160+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd208128130 con 0x7fd2081260b0 2024-02-23T00:36:35.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.160+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:36:35.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.160+0000 7fd205775700 10 client.37709.objecter _finish_command 11 = system:0 2024-02-23T00:36:35.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.160+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd208117540 osd=5 4 2024-02-23T00:36:35.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.160+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 5 11 2024-02-23T00:36:35.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:35.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 12 2024-02-23T00:36:35.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208117540 osd=5 3 2024-02-23T00:36:35.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117540 osd=5 3 2024-02-23T00:36:35.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208117540 osd=5 3 2024-02-23T00:36:35.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 12 2024-02-23T00:36:35.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd208117540 osd=5 3 2024-02-23T00:36:35.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 5 12 2024-02-23T00:36:35.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117540 osd=5 4 2024-02-23T00:36:35.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 10 client.37709.objecter _send_command 12 2024-02-23T00:36:35.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.174+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd208128130 con 0x7fd2081260b0 2024-02-23T00:36:35.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.176+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd208128130 con 0x7fd2081260b0 2024-02-23T00:36:35.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.176+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:35.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.176+0000 7fd205775700 10 client.37709.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.176+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd208117540 osd=5 4 2024-02-23T00:36:35.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.176+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 5 12 2024-02-23T00:36:35.178 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:36:35.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 13 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd20812a830 0x7fd20812ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208128130 osd=6 3 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208128130 osd=6 3 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208128130 osd=6 3 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 13 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd208128130 osd=6 3 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 6 13 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208128130 osd=6 4 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 10 client.37709.objecter _send_command 13 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.177+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd20812b2c0 con 0x7fd20812a830 2024-02-23T00:36:35.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.178+0000 7fd20c85c700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd20812a830 0x7fd20812ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.178+0000 7fd20c85c700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd20812a830 0x7fd20812ac50 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.178+0000 7fd205775700 10 client.37709.objecter ms_handle_connect 0x7fd20812a830 2024-02-23T00:36:35.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.179+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd20812b2c0 con 0x7fd20812a830 2024-02-23T00:36:35.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.179+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:36:35.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.179+0000 7fd205775700 10 client.37709.objecter _finish_command 13 = system:0 2024-02-23T00:36:35.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.179+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd208128130 osd=6 4 2024-02-23T00:36:35.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.179+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 6 13 2024-02-23T00:36:35.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:35.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 14 2024-02-23T00:36:35.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208128130 osd=6 3 2024-02-23T00:36:35.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208128130 osd=6 3 2024-02-23T00:36:35.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd208128130 osd=6 3 2024-02-23T00:36:35.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 14 2024-02-23T00:36:35.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd208128130 osd=6 3 2024-02-23T00:36:35.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 6 14 2024-02-23T00:36:35.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208128130 osd=6 4 2024-02-23T00:36:35.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 10 client.37709.objecter _send_command 14 2024-02-23T00:36:35.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.193+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd20812b2c0 con 0x7fd20812a830 2024-02-23T00:36:35.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.195+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd20812b2c0 con 0x7fd20812a830 2024-02-23T00:36:35.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.195+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:35.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.195+0000 7fd205775700 10 client.37709.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.195+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd208128130 osd=6 4 2024-02-23T00:36:35.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.195+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 6 14 2024-02-23T00:36:35.197 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:36:35.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:35.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 15 2024-02-23T00:36:35.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6812/373787000 conn(0x7fd20812db50 0x7fd20812df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:35.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd20812b2c0 osd=7 3 2024-02-23T00:36:35.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20812b2c0 osd=7 3 2024-02-23T00:36:35.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd20812b2c0 osd=7 3 2024-02-23T00:36:35.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 15 2024-02-23T00:36:35.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd20812b2c0 osd=7 3 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 7 15 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20812b2c0 osd=7 4 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 10 client.37709.objecter _send_command 15 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2081318c0 con 0x7fd20812db50 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd207fff700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6812/373787000 conn(0x7fd20812db50 0x7fd20812df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd205775700 10 client.37709.objecter ms_handle_connect 0x7fd20812db50 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.196+0000 7fd207fff700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6812/373787000 conn(0x7fd20812db50 0x7fd20812df90 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.197+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd2081318c0 con 0x7fd20812db50 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.197+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.197+0000 7fd205775700 10 client.37709.objecter _finish_command 15 = system:0 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.197+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd20812b2c0 osd=7 4 2024-02-23T00:36:35.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.197+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 7 15 2024-02-23T00:36:35.204 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:36:35.209 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:33.658213+0000 mgr.y (mgr.24437) 393 : cluster [DBG] pgmap v628: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:36:35.209 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:34.149703+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.21 deep-scrub starts 2024-02-23T00:36:35.209 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:34.152640+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.21 deep-scrub ok 2024-02-23T00:36:35.209 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:34.602726+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.c scrub starts 2024-02-23T00:36:35.209 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:34.605660+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.c scrub ok 2024-02-23T00:36:35.209 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:35.071170+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? 172.21.15.87:0/2044816031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:35.210 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:33.658213+0000 mgr.y (mgr.24437) 393 : cluster [DBG] pgmap v628: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:36:35.210 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:34.149703+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.21 deep-scrub starts 2024-02-23T00:36:35.210 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:34.152640+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.21 deep-scrub ok 2024-02-23T00:36:35.211 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:34.602726+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.c scrub starts 2024-02-23T00:36:35.211 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:34.605660+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.c scrub ok 2024-02-23T00:36:35.211 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:35.071170+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? 172.21.15.87:0/2044816031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:35.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 10 client.37709.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:35.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign -1 16 2024-02-23T00:36:35.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd20812b2c0 osd=7 3 2024-02-23T00:36:35.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20812b2c0 osd=7 3 2024-02-23T00:36:35.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 20 client.37709.objecter _get_session s=0x7fd20812b2c0 osd=7 3 2024-02-23T00:36:35.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_remove -1 16 2024-02-23T00:36:35.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 20 client.37709.objecter get_session s=0x7fd20812b2c0 osd=7 3 2024-02-23T00:36:35.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 15 client.37709.objecter _session_command_op_assign 7 16 2024-02-23T00:36:35.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20812b2c0 osd=7 4 2024-02-23T00:36:35.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 10 client.37709.objecter _send_command 16 2024-02-23T00:36:35.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.215+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd208126ac0 con 0x7fd20812db50 2024-02-23T00:36:35.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd205775700 1 -- 172.21.15.87:0/2044816031 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd208126ac0 con 0x7fd20812db50 2024-02-23T00:36:35.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd205775700 10 client.37709.objecter ms_dispatch 0x7fd208001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:35.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd205775700 10 client.37709.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd205775700 20 client.37709.objecter put_session s=0x7fd20812b2c0 osd=7 4 2024-02-23T00:36:35.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd205775700 15 client.37709.objecter _session_command_op_remove 7 16 2024-02-23T00:36:35.218 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 librados: watch_flush enter 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 librados: watch_flush exit 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 client.37709.objecter close_session for osd.0 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd2080ffed0 msgr2=0x7fd2081002b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6801/2153191863 conn(0x7fd2080ffed0 0x7fd2081002b0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20805fa60 osd=0 1 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 client.37709.objecter close_session for osd.1 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6805/685354171 conn(0x7fd2080f4dd0 msgr2=0x7fd2080273c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6805/685354171 conn(0x7fd2080f4dd0 0x7fd2080273c0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2080f4c60 osd=1 1 2024-02-23T00:36:35.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 client.37709.objecter close_session for osd.2 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd208028590 msgr2=0x7fd20811e590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd208028590 0x7fd20811e590 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208059360 osd=2 1 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 client.37709.objecter close_session for osd.3 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6813/128059790 conn(0x7fd20811fc00 msgr2=0x7fd208121ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20c85c700 1 -- 172.21.15.87:0/2044816031 reap_dead start 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6813/128059790 conn(0x7fd20811fc00 0x7fd208121ff0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd2081177c0 osd=3 1 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 client.37709.objecter close_session for osd.4 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd20811f760 msgr2=0x7fd208125670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd20811f760 0x7fd208125670 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117230 osd=4 1 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 client.37709.objecter close_session for osd.5 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd2081260b0 msgr2=0x7fd208126510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd2081260b0 0x7fd208126510 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208117540 osd=5 1 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 client.37709.objecter close_session for osd.6 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd20812a830 msgr2=0x7fd20812ac50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd20812a830 0x7fd20812ac50 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd208128130 osd=6 1 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 client.37709.objecter close_session for osd.7 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6812/373787000 conn(0x7fd20812db50 msgr2=0x7fd20812df90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.171:6812/373787000 conn(0x7fd20812db50 0x7fd20812df90 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 20 client.37709.objecter put_session s=0x7fd20812b2c0 osd=7 1 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 20 client.37709.objecter shutdown clearing up homeless session... 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 10 client.37709.objecter successfully canceled tick 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20c85c700 1 -- 172.21.15.87:0/2044816031 reap_dead start 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1f4062c90 msgr2=0x7fd1f4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1f4062c90 0x7fd1f4065150 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 rx=0x7fd208109af0 tx=0x7fd1f000b040).stop 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2080f7400 msgr2=0x7fd20810e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2080f7400 0x7fd20810e1b0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 rx=0x7fd1f8009eb0 tx=0x7fd1f800b040).stop 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 shutdown_connections 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd1f4062c90 0x7fd1f4065150 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 --2- 172.21.15.87:0/2044816031 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd2080f7400 0x7fd20810e1b0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 >> 172.21.15.87:0/2044816031 conn(0x7fd2080f2820 msgr2=0x7fd2080f4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 shutdown_connections 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.217+0000 7fd20e2bf700 1 -- 172.21.15.87:0/2044816031 wait complete. 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:35.218+0000 7fd20e2bf700 1 librados: shutdown 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.223 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.224 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.224 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:36:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:35.233 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:36:35.233 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:36:35.252 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:35.254 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:33.658213+0000 mgr.y (mgr.24437) 393 : cluster [DBG] pgmap v628: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:36:35.254 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:34.149703+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.21 deep-scrub starts 2024-02-23T00:36:35.254 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:34.152640+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.21 deep-scrub ok 2024-02-23T00:36:35.254 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:34.602726+0000 osd.1 (osd.1) 183 : cluster [DBG] 5.c scrub starts 2024-02-23T00:36:35.254 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:34.605660+0000 osd.1 (osd.1) 184 : cluster [DBG] 5.c scrub ok 2024-02-23T00:36:35.254 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:35.071170+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? 172.21.15.87:0/2044816031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:35.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:36:35.356 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:35.383 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:35.420 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:36:35.420 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:36:35.420 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:36:35.421 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-23T00:36:35.421 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.0 2024-02-23T00:36:35.479 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:36:35 smithi087 systemd[1]: Stopping Ceph osd.0 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:36:35.480 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:35.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:35.584 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:35.627 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:35.832 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:36:35 smithi087 bash[189926]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:36:35.968 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-23T00:36:36.184 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:36.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:36.254 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:36:36.289 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:36.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:35.157339+0000 osd.6 (osd.6) 160 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:36:36.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:35.160235+0000 osd.6 (osd.6) 161 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:36:36.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:35.928615+0000 osd.7 (osd.7) 203 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:36:36.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:35.929502+0000 osd.7 (osd.7) 204 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:36:36.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:36.097451+0000 mon.a (mon.0) 2588 : cluster [INF] osd.0 marked itself down and dead 2024-02-23T00:36:36.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:36.127398+0000 osd.4 (osd.4) 197 : cluster [DBG] 2.f scrub starts 2024-02-23T00:36:36.316 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:36.130862+0000 osd.4 (osd.4) 198 : cluster [DBG] 2.f scrub ok 2024-02-23T00:36:36.317 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[48310]: debug 2024-02-23T00:36:36.096+0000 7fc9ca5ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:36:36.317 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[48310]: debug 2024-02-23T00:36:36.096+0000 7fc9ca5ed700 -1 osd.0 424 *** Got signal Terminated *** 2024-02-23T00:36:36.317 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[48310]: debug 2024-02-23T00:36:36.096+0000 7fc9ca5ed700 -1 osd.0 424 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-23T00:36:36.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:35.157339+0000 osd.6 (osd.6) 160 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:36:36.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:35.160235+0000 osd.6 (osd.6) 161 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:36:36.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:35.928615+0000 osd.7 (osd.7) 203 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:36:36.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:35.929502+0000 osd.7 (osd.7) 204 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:36:36.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:36.097451+0000 mon.a (mon.0) 2588 : cluster [INF] osd.0 marked itself down and dead 2024-02-23T00:36:36.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:36.127398+0000 osd.4 (osd.4) 197 : cluster [DBG] 2.f scrub starts 2024-02-23T00:36:36.317 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:36.130862+0000 osd.4 (osd.4) 198 : cluster [DBG] 2.f scrub ok 2024-02-23T00:36:36.318 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:36.503 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:35.157339+0000 osd.6 (osd.6) 160 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:36:36.503 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:35.160235+0000 osd.6 (osd.6) 161 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:36:36.503 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:35.928615+0000 osd.7 (osd.7) 203 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:36:36.503 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:35.929502+0000 osd.7 (osd.7) 204 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:36:36.503 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:36.097451+0000 mon.a (mon.0) 2588 : cluster [INF] osd.0 marked itself down and dead 2024-02-23T00:36:36.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:36.127398+0000 osd.4 (osd.4) 197 : cluster [DBG] 2.f scrub starts 2024-02-23T00:36:36.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:36.130862+0000 osd.4 (osd.4) 198 : cluster [DBG] 2.f scrub ok 2024-02-23T00:36:36.886 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:36:36.913 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:36:36 smithi087 bash[190008]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0 2024-02-23T00:36:36.967 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:36.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:37.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:37.110 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:37.148 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:37.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:37.198 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:37.253 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:37.280 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:37.428 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:37.428 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:35.659166+0000 mgr.y (mgr.24437) 394 : cluster [DBG] pgmap v629: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:36:37.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:36.180021+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:36:37.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:36.180844+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:36:37.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:36.207918+0000 mon.a (mon.0) 2589 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-23T00:36:37.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:36.221316+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e425: 8 total, 7 up, 7 in 2024-02-23T00:36:37.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:36.529711+0000 osd.1 (osd.1) 185 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:36:37.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:36.531418+0000 osd.1 (osd.1) 186 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:36:37.429 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:37.097563+0000 mon.a (mon.0) 2591 : cluster [DBG] osdmap e426: 8 total, 7 up, 7 in 2024-02-23T00:36:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:37.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:37.476 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:35.659166+0000 mgr.y (mgr.24437) 394 : cluster [DBG] pgmap v629: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:36:37.476 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:36.180021+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:36:37.476 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:36.180844+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:36.207918+0000 mon.a (mon.0) 2589 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:36.221316+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e425: 8 total, 7 up, 7 in 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:36.529711+0000 osd.1 (osd.1) 185 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:36.531418+0000 osd.1 (osd.1) 186 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:37.097563+0000 mon.a (mon.0) 2591 : cluster [DBG] osdmap e426: 8 total, 7 up, 7 in 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:35.659166+0000 mgr.y (mgr.24437) 394 : cluster [DBG] pgmap v629: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:36.180021+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:36.180844+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:36.207918+0000 mon.a (mon.0) 2589 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:36.221316+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e425: 8 total, 7 up, 7 in 2024-02-23T00:36:37.477 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:36.529711+0000 osd.1 (osd.1) 185 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:36:37.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:36.531418+0000 osd.1 (osd.1) 186 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:36:37.478 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:37.097563+0000 mon.a (mon.0) 2591 : cluster [DBG] osdmap e426: 8 total, 7 up, 7 in 2024-02-23T00:36:37.531 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:37.560 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:37.660 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:37.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:37.763 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:37.795 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:37.893 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:37.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:37.916 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:36:37.916 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:36:37.999 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:38.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:36:38.030 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:38.085 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:38.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:38.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:38.222 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:37.226566+0000 osd.6 (osd.6) 164 : cluster [DBG] 4.a scrub starts 2024-02-23T00:36:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:37.230347+0000 osd.6 (osd.6) 165 : cluster [DBG] 4.a scrub ok 2024-02-23T00:36:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:37.951500+0000 osd.7 (osd.7) 205 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:36:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:37.952467+0000 osd.7 (osd.7) 206 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:36:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:38.196072+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:38.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:38.196315+0000 mon.a (mon.0) 2592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:38.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:38.197054+0000 mon.b (mon.1) 356 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-61"}]: dispatch 2024-02-23T00:36:38.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:38.197156+0000 mon.a (mon.0) 2593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-61"}]: dispatch 2024-02-23T00:36:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:37.226566+0000 osd.6 (osd.6) 164 : cluster [DBG] 4.a scrub starts 2024-02-23T00:36:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:37.230347+0000 osd.6 (osd.6) 165 : cluster [DBG] 4.a scrub ok 2024-02-23T00:36:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:37.951500+0000 osd.7 (osd.7) 205 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:36:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:37.952467+0000 osd.7 (osd.7) 206 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:36:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:38.196072+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:38.196315+0000 mon.a (mon.0) 2592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:38.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:38.197054+0000 mon.b (mon.1) 356 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-61"}]: dispatch 2024-02-23T00:36:38.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:38.197156+0000 mon.a (mon.0) 2593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-61"}]: dispatch 2024-02-23T00:36:38.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:37.226566+0000 osd.6 (osd.6) 164 : cluster [DBG] 4.a scrub starts 2024-02-23T00:36:38.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:37.230347+0000 osd.6 (osd.6) 165 : cluster [DBG] 4.a scrub ok 2024-02-23T00:36:38.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:37.951500+0000 osd.7 (osd.7) 205 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:36:38.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:37.952467+0000 osd.7 (osd.7) 206 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:36:38.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:38.196072+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:38.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:38.196315+0000 mon.a (mon.0) 2592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:38.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:38.197054+0000 mon.b (mon.1) 356 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-61"}]: dispatch 2024-02-23T00:36:38.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:38.197156+0000 mon.a (mon.0) 2593 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-61"}]: dispatch 2024-02-23T00:36:38.611 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-23T00:36:38.876 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:38.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:38.915 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:36:38.980 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:39.011 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:39.251 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:37.659703+0000 mgr.y (mgr.24437) 395 : cluster [DBG] pgmap v632: 238 pgs: 27 stale+active+clean, 211 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 918 B/s rd, 0 op/s 2024-02-23T00:36:39.251 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:38.943015+0000 osd.7 (osd.7) 207 : cluster [DBG] 63.13 scrub starts 2024-02-23T00:36:39.251 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:38.944612+0000 osd.7 (osd.7) 208 : cluster [DBG] 63.13 scrub ok 2024-02-23T00:36:39.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.0.service' 2024-02-23T00:36:39.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:37.659703+0000 mgr.y (mgr.24437) 395 : cluster [DBG] pgmap v632: 238 pgs: 27 stale+active+clean, 211 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 918 B/s rd, 0 op/s 2024-02-23T00:36:39.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:38.943015+0000 osd.7 (osd.7) 207 : cluster [DBG] 63.13 scrub starts 2024-02-23T00:36:39.433 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:38.944612+0000 osd.7 (osd.7) 208 : cluster [DBG] 63.13 scrub ok 2024-02-23T00:36:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:37.659703+0000 mgr.y (mgr.24437) 395 : cluster [DBG] pgmap v632: 238 pgs: 27 stale+active+clean, 211 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 918 B/s rd, 0 op/s 2024-02-23T00:36:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:38.943015+0000 osd.7 (osd.7) 207 : cluster [DBG] 63.13 scrub starts 2024-02-23T00:36:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:38.944612+0000 osd.7 (osd.7) 208 : cluster [DBG] 63.13 scrub ok 2024-02-23T00:36:39.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:36:39.499 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-23T00:36:39.499 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-02-23T00:36:39.499 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi087.front.sepia.ceph.com 2024-02-23T00:36:39.499 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -n osd.0 --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-02-23T00:36:39.909 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:39.940 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/osd.0/config 2024-02-23T00:36:40.260 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:40.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:40.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:40.398 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:40.553 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:39.229722+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e427: 8 total, 7 up, 7 in 2024-02-23T00:36:40.553 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:39.235385+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:36:40.553 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:39.246265+0000 osd.6 (osd.6) 167 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:36:40.553 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:39.439070+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.f scrub starts 2024-02-23T00:36:40.553 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:39.442668+0000 osd.1 (osd.1) 188 : cluster [DBG] 4.f scrub ok 2024-02-23T00:36:40.553 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:39.651047+0000 mon.a (mon.0) 2595 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:39.229722+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e427: 8 total, 7 up, 7 in 2024-02-23T00:36:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:39.235385+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:36:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:39.246265+0000 osd.6 (osd.6) 167 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:36:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:39.439070+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.f scrub starts 2024-02-23T00:36:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:39.442668+0000 osd.1 (osd.1) 188 : cluster [DBG] 4.f scrub ok 2024-02-23T00:36:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:39.651047+0000 mon.a (mon.0) 2595 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:39.229722+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e427: 8 total, 7 up, 7 in 2024-02-23T00:36:40.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:39.235385+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:36:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:39.246265+0000 osd.6 (osd.6) 167 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:36:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:39.439070+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.f scrub starts 2024-02-23T00:36:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:39.442668+0000 osd.1 (osd.1) 188 : cluster [DBG] 4.f scrub ok 2024-02-23T00:36:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:39.651047+0000 mon.a (mon.0) 2595 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:40.596 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:40.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:40.701 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:40.729 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:40.820 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:36:40.820 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:36:41.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:36:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:36:41.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.186+0000 7fb993533700 1 -- 172.21.15.87:0/4092121852 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c0f8d50 msgr2=0x7fb98c0f9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:41.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.186+0000 7fb993533700 1 --2- 172.21.15.87:0/4092121852 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c0f8d50 0x7fb98c0f9130 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 rx=0x7fb9880047b0 tx=0x7fb988009cb0).stop 2024-02-23T00:36:41.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.186+0000 7fb993533700 1 -- 172.21.15.87:0/4092121852 shutdown_connections 2024-02-23T00:36:41.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.186+0000 7fb993533700 1 --2- 172.21.15.87:0/4092121852 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb98c1022b0 0x7fb98c1046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:41.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.186+0000 7fb993533700 1 --2- 172.21.15.87:0/4092121852 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c0f8d50 0x7fb98c0f9130 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:41.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.186+0000 7fb993533700 1 --2- 172.21.15.87:0/4092121852 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb98c0f9620 0x7fb98c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:41.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.186+0000 7fb993533700 1 -- 172.21.15.87:0/4092121852 >> 172.21.15.87:0/4092121852 conn(0x7fb98c0f2820 msgr2=0x7fb98c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:41.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.186+0000 7fb993533700 1 -- 172.21.15.87:0/4092121852 shutdown_connections 2024-02-23T00:36:41.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.186+0000 7fb993533700 1 -- 172.21.15.87:0/4092121852 wait complete. 2024-02-23T00:36:41.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 librados: starting msgr at 2024-02-23T00:36:41.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 librados: starting objecter 2024-02-23T00:36:41.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 Processor -- start 2024-02-23T00:36:41.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 -- start start 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 librados: setting wanted keys 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 librados: calling monclient init 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb98c0f8d50 0x7fb98c110440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb98c0f9620 0x7fb98c110930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c1022b0 0x7fb98c10af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c10b410 con 0x7fb98c0f9620 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c10b580 con 0x7fb98c0f8d50 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb993533700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c10b740 con 0x7fb98c1022b0 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb991ad0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c1022b0 0x7fb98c10af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb990ace700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb98c0f9620 0x7fb98c110930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb991ad0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c1022b0 0x7fb98c10af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:54074/0 (socket says 172.21.15.87:54074) 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb991ad0700 1 -- 172.21.15.87:0/2903636096 learned_addr learned my addr 172.21.15.87:0/2903636096 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:41.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb990ace700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb98c0f9620 0x7fb98c110930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38632/0 (socket says 172.21.15.87:38632) 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.187+0000 7fb9912cf700 1 --2- 172.21.15.87:0/2903636096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb98c0f8d50 0x7fb98c110440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb991ad0700 1 -- 172.21.15.87:0/2903636096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb98c0f8d50 msgr2=0x7fb98c110440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb991ad0700 1 --2- 172.21.15.87:0/2903636096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb98c0f8d50 0x7fb98c110440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb991ad0700 1 -- 172.21.15.87:0/2903636096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb98c0f9620 msgr2=0x7fb98c110930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb991ad0700 1 --2- 172.21.15.87:0/2903636096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb98c0f9620 0x7fb98c110930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb991ad0700 1 -- 172.21.15.87:0/2903636096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c005040 con 0x7fb98c1022b0 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb991ad0700 1 --2- 172.21.15.87:0/2903636096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c1022b0 0x7fb98c10af20 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7fb98400d940 tx=0x7fb984004e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb9827fc700 10 client.?.objecter ms_handle_connect 0x7fb98c1022b0 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb9827fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb9827fc700 1 -- 172.21.15.87:0/2903636096 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984004020 con 0x7fb98c1022b0 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb9827fc700 1 -- 172.21.15.87:0/2903636096 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb98400f040 con 0x7fb98c1022b0 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb988005040 con 0x7fb98c1022b0 2024-02-23T00:36:41.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb9827fc700 1 -- 172.21.15.87:0/2903636096 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9840036c0 con 0x7fb98c1022b0 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb993533700 10 client.37715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98c117240 con 0x7fb98c1022b0 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.188+0000 7fb993533700 1 librados: init done 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.189+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98c05fa60 con 0x7fb98c1022b0 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.189+0000 7fb9827fc700 1 -- 172.21.15.87:0/2903636096 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb984004180 con 0x7fb98c1022b0 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.190+0000 7fb9827fc700 1 --2- 172.21.15.87:0/2903636096 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb978062c90 0x7fb978065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.190+0000 7fb9827fc700 1 -- 172.21.15.87:0/2903636096 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(428..428 src has 173..428) v4 ==== 8109+0+0 (secure 0 0 0) 0x7fb98407ca70 con 0x7fb98c1022b0 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.190+0000 7fb9827fc700 10 client.37715.objecter ms_dispatch 0x7fb98c001550 osd_map(428..428 src has 173..428) v4 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.190+0000 7fb9827fc700 3 client.37715.objecter handle_osd_map got epochs [428,428] > 0 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.190+0000 7fb9827fc700 3 client.37715.objecter handle_osd_map decoding full epoch 428 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.190+0000 7fb9912cf700 1 --2- 172.21.15.87:0/2903636096 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb978062c90 0x7fb978065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.190+0000 7fb9827fc700 20 client.37715.objecter dump_active .. 0 homeless 2024-02-23T00:36:41.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.190+0000 7fb9827fc700 10 client.37715.objecter ms_handle_connect 0x7fb978062c90 2024-02-23T00:36:41.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.190+0000 7fb9912cf700 1 --2- 172.21.15.87:0/2903636096 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb978062c90 0x7fb978065150 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7fb988005010 tx=0x7fb988009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:41.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.194+0000 7fb9827fc700 1 -- 172.21.15.87:0/2903636096 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb984048f60 con 0x7fb98c1022b0 2024-02-23T00:36:41.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:41.428+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fb98c101e40 con 0x7fb98c1022b0 2024-02-23T00:36:41.567 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:39.659979+0000 mgr.y (mgr.24437) 396 : cluster [DBG] pgmap v634: 206 pgs: 24 stale+active+clean, 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:36:41.567 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:40.248791+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e428: 8 total, 7 up, 7 in 2024-02-23T00:36:41.567 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:40.255701+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:41.567 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:40.256013+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:41.567 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:40.326026+0000 osd.6 (osd.6) 168 : cluster [DBG] 63.1d scrub starts 2024-02-23T00:36:41.567 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:40.328640+0000 osd.6 (osd.6) 169 : cluster [DBG] 63.1d scrub ok 2024-02-23T00:36:41.567 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:40.482394+0000 osd.1 (osd.1) 189 : cluster [DBG] 22.b scrub starts 2024-02-23T00:36:41.567 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:40.485279+0000 osd.1 (osd.1) 190 : cluster [DBG] 22.b scrub ok 2024-02-23T00:36:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:39.659979+0000 mgr.y (mgr.24437) 396 : cluster [DBG] pgmap v634: 206 pgs: 24 stale+active+clean, 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:36:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:40.248791+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e428: 8 total, 7 up, 7 in 2024-02-23T00:36:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:40.255701+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:40.256013+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:40.326026+0000 osd.6 (osd.6) 168 : cluster [DBG] 63.1d scrub starts 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:40.328640+0000 osd.6 (osd.6) 169 : cluster [DBG] 63.1d scrub ok 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:40.482394+0000 osd.1 (osd.1) 189 : cluster [DBG] 22.b scrub starts 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:40.485279+0000 osd.1 (osd.1) 190 : cluster [DBG] 22.b scrub ok 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:39.659979+0000 mgr.y (mgr.24437) 396 : cluster [DBG] pgmap v634: 206 pgs: 24 stale+active+clean, 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:40.248791+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e428: 8 total, 7 up, 7 in 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:40.255701+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:40.256013+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:40.326026+0000 osd.6 (osd.6) 168 : cluster [DBG] 63.1d scrub starts 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:40.328640+0000 osd.6 (osd.6) 169 : cluster [DBG] 63.1d scrub ok 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:40.482394+0000 osd.1 (osd.1) 189 : cluster [DBG] 22.b scrub starts 2024-02-23T00:36:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:40.485279+0000 osd.1 (osd.1) 190 : cluster [DBG] 22.b scrub ok 2024-02-23T00:36:41.586 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-23T00:36:41.851 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:36:42.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.267+0000 7fb9827fc700 1 -- 172.21.15.87:0/2903636096 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v430) v1 ==== 87+0+0 (secure 0 0 0) 0x7fb98407ce60 con 0x7fb98c1022b0 2024-02-23T00:36:42.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.270+0000 7fb993533700 10 librados: watch_flush enter 2024-02-23T00:36:42.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.270+0000 7fb993533700 10 librados: watch_flush exit 2024-02-23T00:36:42.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.270+0000 7fb993533700 20 client.37715.objecter shutdown clearing up homeless session... 2024-02-23T00:36:42.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.270+0000 7fb993533700 10 client.37715.objecter successfully canceled tick 2024-02-23T00:36:42.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.270+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb978062c90 msgr2=0x7fb978065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:42.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.270+0000 7fb993533700 1 --2- 172.21.15.87:0/2903636096 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb978062c90 0x7fb978065150 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 rx=0x7fb988005010 tx=0x7fb988009cb0).stop 2024-02-23T00:36:42.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.270+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c1022b0 msgr2=0x7fb98c10af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:42.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.270+0000 7fb993533700 1 --2- 172.21.15.87:0/2903636096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c1022b0 0x7fb98c10af20 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 rx=0x7fb98400d940 tx=0x7fb984004e30).stop 2024-02-23T00:36:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.271+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 shutdown_connections 2024-02-23T00:36:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.271+0000 7fb993533700 1 --2- 172.21.15.87:0/2903636096 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb978062c90 0x7fb978065150 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.271+0000 7fb993533700 1 --2- 172.21.15.87:0/2903636096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb98c1022b0 0x7fb98c10af20 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.271+0000 7fb993533700 1 --2- 172.21.15.87:0/2903636096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb98c0f8d50 0x7fb98c110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.271+0000 7fb993533700 1 --2- 172.21.15.87:0/2903636096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb98c0f9620 0x7fb98c110930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.271+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 >> 172.21.15.87:0/2903636096 conn(0x7fb98c0f2820 msgr2=0x7fb98c104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.271+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 shutdown_connections 2024-02-23T00:36:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.271+0000 7fb993533700 1 -- 172.21.15.87:0/2903636096 wait complete. 2024-02-23T00:36:42.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:42.278+0000 7fb993533700 1 librados: shutdown 2024-02-23T00:36:42.294 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:36:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:40.923812+0000 mgr.y (mgr.24437) 397 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:41.249662+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:36:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:41.249744+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e429: 8 total, 7 up, 7 in 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:41.269939+0000 mon.b (mon.1) 358 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:41.279996+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:41.281945+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:41.428916+0000 mon.c (mon.2) 435 : audit [INF] from='client.? 172.21.15.87:0/2903636096' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:41.429279+0000 mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:41.529731+0000 osd.1 (osd.1) 191 : cluster [DBG] 2.d scrub starts 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:41.532615+0000 osd.1 (osd.1) 192 : cluster [DBG] 2.d scrub ok 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:41.936541+0000 osd.7 (osd.7) 209 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:41.937339+0000 osd.7 (osd.7) 210 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:42.092275+0000 osd.4 (osd.4) 199 : cluster [DBG] 22.10 deep-scrub starts 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:42.093172+0000 osd.4 (osd.4) 200 : cluster [DBG] 22.10 deep-scrub ok 2024-02-23T00:36:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:42.180006+0000 mon.a (mon.0) 2602 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:36:42.546 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:42] "GET /metrics HTTP/1.1" 200 185739 "" "Prometheus/2.33.4" 2024-02-23T00:36:42.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:40.923812+0000 mgr.y (mgr.24437) 397 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:42.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:41.249662+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:36:42.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:41.249744+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e429: 8 total, 7 up, 7 in 2024-02-23T00:36:42.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:41.269939+0000 mon.b (mon.1) 358 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:36:42.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:41.279996+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:41.281945+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:41.428916+0000 mon.c (mon.2) 435 : audit [INF] from='client.? 172.21.15.87:0/2903636096' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:41.429279+0000 mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:41.529731+0000 osd.1 (osd.1) 191 : cluster [DBG] 2.d scrub starts 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:41.532615+0000 osd.1 (osd.1) 192 : cluster [DBG] 2.d scrub ok 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:41.936541+0000 osd.7 (osd.7) 209 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:41.937339+0000 osd.7 (osd.7) 210 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:42.092275+0000 osd.4 (osd.4) 199 : cluster [DBG] 22.10 deep-scrub starts 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:42.093172+0000 osd.4 (osd.4) 200 : cluster [DBG] 22.10 deep-scrub ok 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:42.180006+0000 mon.a (mon.0) 2602 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:36:42.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:40.923812+0000 mgr.y (mgr.24437) 397 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:41.249662+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:41.249744+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e429: 8 total, 7 up, 7 in 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:41.269939+0000 mon.b (mon.1) 358 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:41.279996+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:41.281945+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:41.428916+0000 mon.c (mon.2) 435 : audit [INF] from='client.? 172.21.15.87:0/2903636096' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:41.429279+0000 mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:41.529731+0000 osd.1 (osd.1) 191 : cluster [DBG] 2.d scrub starts 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:41.532615+0000 osd.1 (osd.1) 192 : cluster [DBG] 2.d scrub ok 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:41.936541+0000 osd.7 (osd.7) 209 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:41.937339+0000 osd.7 (osd.7) 210 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:42.092275+0000 osd.4 (osd.4) 199 : cluster [DBG] 22.10 deep-scrub starts 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:42.093172+0000 osd.4 (osd.4) 200 : cluster [DBG] 22.10 deep-scrub ok 2024-02-23T00:36:42.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:42.180006+0000 mon.a (mon.0) 2602 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:36:42.888 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:42.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:42.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:42.995 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:43.028 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:43.278 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:43.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:43.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:43.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:41.660700+0000 mgr.y (mgr.24437) 398 : cluster [DBG] pgmap v637: 238 pgs: 26 active+undersized+degraded, 52 active+undersized, 32 creating+peering, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1149 objects degraded (10.270%) 2024-02-23T00:36:43.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:41.911830+0000 osd.2 (osd.2) 161 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:36:43.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:41.912661+0000 osd.2 (osd.2) 162 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:36:43.284 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:42.253773+0000 mon.a (mon.0) 2603 : cluster [WRN] Health check failed: Degraded data redundancy: 118/1149 objects degraded (10.270%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:36:43.284 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:41.660700+0000 mgr.y (mgr.24437) 398 : cluster [DBG] pgmap v637: 238 pgs: 26 active+undersized+degraded, 52 active+undersized, 32 creating+peering, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1149 objects degraded (10.270%) 2024-02-23T00:36:43.284 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:41.911830+0000 osd.2 (osd.2) 161 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:36:43.284 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:41.912661+0000 osd.2 (osd.2) 162 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:36:43.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:42.253773+0000 mon.a (mon.0) 2603 : cluster [WRN] Health check failed: Degraded data redundancy: 118/1149 objects degraded (10.270%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:36:43.285 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:42.254533+0000 mon.a (mon.0) 2604 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:43.383 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:43.411 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:43.536 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:41.660700+0000 mgr.y (mgr.24437) 398 : cluster [DBG] pgmap v637: 238 pgs: 26 active+undersized+degraded, 52 active+undersized, 32 creating+peering, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1149 objects degraded (10.270%) 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:41.911830+0000 osd.2 (osd.2) 161 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:41.912661+0000 osd.2 (osd.2) 162 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:42.253773+0000 mon.a (mon.0) 2603 : cluster [WRN] Health check failed: Degraded data redundancy: 118/1149 objects degraded (10.270%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:42.254533+0000 mon.a (mon.0) 2604 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:42.264242+0000 mon.a (mon.0) 2605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:42.264314+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:42.264372+0000 mon.a (mon.0) 2607 : cluster [DBG] osdmap e430: 8 total, 7 up, 7 in 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:42.697546+0000 mon.a (mon.0) 2608 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:42.698166+0000 mon.a (mon.0) 2609 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:36:43.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:42.702418+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:43.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:43.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:43.558 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:36:43.559 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:36:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:42.264242+0000 mon.a (mon.0) 2605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:36:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:42.264314+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:36:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:42.264372+0000 mon.a (mon.0) 2607 : cluster [DBG] osdmap e430: 8 total, 7 up, 7 in 2024-02-23T00:36:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:42.697546+0000 mon.a (mon.0) 2608 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:36:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:42.698166+0000 mon.a (mon.0) 2609 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:36:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:42.702418+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:42.254533+0000 mon.a (mon.0) 2604 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:42.264242+0000 mon.a (mon.0) 2605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:36:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:42.264314+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:36:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:42.264372+0000 mon.a (mon.0) 2607 : cluster [DBG] osdmap e430: 8 total, 7 up, 7 in 2024-02-23T00:36:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:42.697546+0000 mon.a (mon.0) 2608 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:36:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:42.698166+0000 mon.a (mon.0) 2609 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:36:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:42.702418+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:43.641 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:43.669 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:43.818 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:43.821 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:36:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:36:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:43.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:43.923 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:43.954 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:43.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 -- 172.21.15.87:0/3884354676 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88bc0f9620 msgr2=0x7f88bc101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:43.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 --2- 172.21.15.87:0/3884354676 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88bc0f9620 0x7f88bc101dc0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 rx=0x7f88ac004870 tx=0x7f88ac009cb0).stop 2024-02-23T00:36:43.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 -- 172.21.15.87:0/3884354676 shutdown_connections 2024-02-23T00:36:43.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 --2- 172.21.15.87:0/3884354676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88bc1022b0 0x7f88bc1046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:43.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 --2- 172.21.15.87:0/3884354676 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88bc0f8d50 0x7f88bc0f9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:43.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 --2- 172.21.15.87:0/3884354676 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88bc0f9620 0x7f88bc101dc0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:43.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 -- 172.21.15.87:0/3884354676 >> 172.21.15.87:0/3884354676 conn(0x7f88bc0f2820 msgr2=0x7f88bc0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:43.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 -- 172.21.15.87:0/3884354676 shutdown_connections 2024-02-23T00:36:43.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 -- 172.21.15.87:0/3884354676 wait complete. 2024-02-23T00:36:43.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 librados: starting msgr at 2024-02-23T00:36:43.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.980+0000 7f88c3320700 1 librados: starting objecter 2024-02-23T00:36:43.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 Processor -- start 2024-02-23T00:36:43.983 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:43.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 -- start start 2024-02-23T00:36:43.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 librados: setting wanted keys 2024-02-23T00:36:43.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 librados: calling monclient init 2024-02-23T00:36:43.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88bc0f8d50 0x7f88bc10e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88bc0f9620 0x7f88bc10e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88bc1022b0 0x7f88bc108420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc108910 con 0x7f88bc0f8d50 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc108a80 con 0x7f88bc0f9620 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c3320700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc108bf0 con 0x7f88bc1022b0 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c10bc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88bc0f8d50 0x7f88bc10e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c10bc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88bc0f8d50 0x7f88bc10e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39074/0 (socket says 172.21.15.87:39074) 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c10bc700 1 -- 172.21.15.87:0/2983023900 learned_addr learned my addr 172.21.15.87:0/2983023900 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c08bb700 1 --2- 172.21.15.87:0/2983023900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88bc0f9620 0x7f88bc10e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c18bd700 1 --2- 172.21.15.87:0/2983023900 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88bc1022b0 0x7f88bc108420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.981+0000 7f88c10bc700 1 -- 172.21.15.87:0/2983023900 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88bc1022b0 msgr2=0x7f88bc108420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c10bc700 1 --2- 172.21.15.87:0/2983023900 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88bc1022b0 0x7f88bc108420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c10bc700 1 -- 172.21.15.87:0/2983023900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88bc0f9620 msgr2=0x7f88bc10e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:43.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c10bc700 1 --2- 172.21.15.87:0/2983023900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88bc0f9620 0x7f88bc10e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c10bc700 1 -- 172.21.15.87:0/2983023900 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b8005040 con 0x7f88bc0f8d50 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c08bb700 1 --2- 172.21.15.87:0/2983023900 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88bc0f9620 0x7f88bc10e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c10bc700 1 --2- 172.21.15.87:0/2983023900 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88bc0f8d50 0x7f88bc10e240 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 rx=0x7f88b80048f0 tx=0x7f88b8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88b27fc700 10 client.?.objecter ms_handle_connect 0x7f88bc0f8d50 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88b27fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8007b00 con 0x7f88bc0f8d50 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88b800f040 con 0x7f88bc0f8d50 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8014990 con 0x7f88bc0f8d50 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ac005040 con 0x7f88bc0f8d50 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c3320700 10 client.27261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:43.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc1091d0 con 0x7f88bc0f8d50 2024-02-23T00:36:43.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.982+0000 7f88c3320700 1 librados: init done 2024-02-23T00:36:43.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.983+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f88bc05fa60 con 0x7f88bc0f8d50 2024-02-23T00:36:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:43.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:43.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.984+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f88b8010500 con 0x7f88bc0f8d50 2024-02-23T00:36:43.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.984+0000 7f88b27fc700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6800/1231597463 conn(0x7f88a8062c90 0x7f88a8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:43.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.984+0000 7f88c08bb700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6800/1231597463 conn(0x7f88a8062c90 0x7f88a8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:43.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.984+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(431..431 src has 173..431) v4 ==== 8122+0+0 (secure 0 0 0) 0x7f88b807cff0 con 0x7f88bc0f8d50 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.984+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 osd_map(431..431 src has 173..431) v4 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.984+0000 7f88b27fc700 3 client.27261.objecter handle_osd_map got epochs [431,431] > 0 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.984+0000 7f88b27fc700 3 client.27261.objecter handle_osd_map decoding full epoch 431 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88b27fc700 20 client.27261.objecter dump_active .. 0 homeless 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v431) v1 ==== 54+0+16 (secure 0 0 0) 0x7f88b807d4b0 con 0x7f88bc0f8d50 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88b27fc700 10 client.27261.objecter ms_handle_connect 0x7f88a8062c90 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c08bb700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6800/1231597463 conn(0x7f88a8062c90 0x7f88a8065150 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 rx=0x7f88ac004ee0 tx=0x7f88ac00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 10 client.27261.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 1 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 20 client.27261.objecter _get_session osd=-1 returning homeless 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 10 client.27261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=432}) v3 -- 0x7f88bc05ee80 con 0x7f88bc0f8d50 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f88bc107290 con 0x7f88bc0f8d50 2024-02-23T00:36:43.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_get_version_reply(handle=1 version=431) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88b80504e0 con 0x7f88bc0f8d50 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88b37fe700 10 client.27261.objecter _check_command_map_dne tid 1 current 431 map_dne_bound 431 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88b37fe700 10 client.27261.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88b37fe700 15 client.27261.objecter _session_command_op_remove -1 1 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 10 client.27261.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 2 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6805/685354171 conn(0x7f88bc0f4ef0 0x7f88bc0273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc0f4d80 osd=1 3 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc0f4d80 osd=1 3 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc0f4d80 osd=1 3 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 2 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc0f4d80 osd=1 3 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 1 2 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc0f4d80 osd=1 4 2024-02-23T00:36:43.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 10 client.27261.objecter _send_command 2 2024-02-23T00:36:43.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88bc0f8bb0 con 0x7f88bc0f4ef0 2024-02-23T00:36:43.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.985+0000 7f88c18bd700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6805/685354171 conn(0x7f88bc0f4ef0 0x7f88bc0273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:43.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.986+0000 7f88c18bd700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6805/685354171 conn(0x7f88bc0f4ef0 0x7f88bc0273c0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:43.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.986+0000 7f88b27fc700 10 client.27261.objecter ms_handle_connect 0x7f88bc0f4ef0 2024-02-23T00:36:43.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.986+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f88bc0f8bb0 con 0x7f88bc0f4ef0 2024-02-23T00:36:43.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.986+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:36:43.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.986+0000 7f88b27fc700 10 client.27261.objecter _finish_command 2 = system:0 2024-02-23T00:36:43.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.986+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc0f4d80 osd=1 4 2024-02-23T00:36:43.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:43.986+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 1 2 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 10 client.27261.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 3 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc0f4d80 osd=1 3 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc0f4d80 osd=1 3 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc0f4d80 osd=1 3 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 3 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc0f4d80 osd=1 3 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 1 3 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc0f4d80 osd=1 4 2024-02-23T00:36:44.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 10 client.27261.objecter _send_command 3 2024-02-23T00:36:44.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.001+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88bc05a040 con 0x7f88bc0f4ef0 2024-02-23T00:36:44.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f88bc05a040 con 0x7f88bc0f4ef0 2024-02-23T00:36:44.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:44.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88b27fc700 10 client.27261.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc0f4d80 osd=1 4 2024-02-23T00:36:44.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 1 3 2024-02-23T00:36:44.005 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 10 client.27261.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 4 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6809/3605394228 conn(0x7f88bc028590 0x7f88bc064690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc059360 osd=2 3 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc059360 osd=2 3 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc059360 osd=2 3 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 4 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc059360 osd=2 3 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 2 4 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc059360 osd=2 4 2024-02-23T00:36:44.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 10 client.27261.objecter _send_command 4 2024-02-23T00:36:44.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88bc0594d0 con 0x7f88bc028590 2024-02-23T00:36:44.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.002+0000 7f88c10bc700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6809/3605394228 conn(0x7f88bc028590 0x7f88bc064690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:44.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.003+0000 7f88b27fc700 10 client.27261.objecter ms_handle_connect 0x7f88bc028590 2024-02-23T00:36:44.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.003+0000 7f88c10bc700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6809/3605394228 conn(0x7f88bc028590 0x7f88bc064690 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:44.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.003+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f88bc0594d0 con 0x7f88bc028590 2024-02-23T00:36:44.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.003+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:36:44.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.003+0000 7f88b27fc700 10 client.27261.objecter _finish_command 4 = system:0 2024-02-23T00:36:44.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.003+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc059360 osd=2 4 2024-02-23T00:36:44.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.003+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 2 4 2024-02-23T00:36:44.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 10 client.27261.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:44.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 5 2024-02-23T00:36:44.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc059360 osd=2 3 2024-02-23T00:36:44.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc059360 osd=2 3 2024-02-23T00:36:44.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc059360 osd=2 3 2024-02-23T00:36:44.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 5 2024-02-23T00:36:44.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc059360 osd=2 3 2024-02-23T00:36:44.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 2 5 2024-02-23T00:36:44.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc059360 osd=2 4 2024-02-23T00:36:44.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 10 client.27261.objecter _send_command 5 2024-02-23T00:36:44.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.018+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88bc0596f0 con 0x7f88bc028590 2024-02-23T00:36:44.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.019+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f88bc0596f0 con 0x7f88bc028590 2024-02-23T00:36:44.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.019+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:44.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.019+0000 7f88b27fc700 10 client.27261.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.019+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc059360 osd=2 4 2024-02-23T00:36:44.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.019+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 2 5 2024-02-23T00:36:44.021 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:36:44.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 10 client.27261.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:44.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 6 2024-02-23T00:36:44.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6813/128059790 conn(0x7f88bc0fca50 0x7f88bc067d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:44.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc027d30 osd=3 3 2024-02-23T00:36:44.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc027d30 osd=3 3 2024-02-23T00:36:44.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc027d30 osd=3 3 2024-02-23T00:36:44.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 6 2024-02-23T00:36:44.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc027d30 osd=3 3 2024-02-23T00:36:44.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 3 6 2024-02-23T00:36:44.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc027d30 osd=3 4 2024-02-23T00:36:44.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 10 client.27261.objecter _send_command 6 2024-02-23T00:36:44.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88bc06a9f0 con 0x7f88bc0fca50 2024-02-23T00:36:44.023 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c08bb700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6813/128059790 conn(0x7f88bc0fca50 0x7f88bc067d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:44.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88b27fc700 10 client.27261.objecter ms_handle_connect 0x7f88bc0fca50 2024-02-23T00:36:44.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.020+0000 7f88c08bb700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.87:6813/128059790 conn(0x7f88bc0fca50 0x7f88bc067d10 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:44.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.021+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f88bc06a9f0 con 0x7f88bc0fca50 2024-02-23T00:36:44.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.021+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:36:44.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.021+0000 7f88b27fc700 10 client.27261.objecter _finish_command 6 = system:0 2024-02-23T00:36:44.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.021+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc027d30 osd=3 4 2024-02-23T00:36:44.024 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.021+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 3 6 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 10 client.27261.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 7 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc027d30 osd=3 3 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc027d30 osd=3 3 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc027d30 osd=3 3 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 7 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc027d30 osd=3 3 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 3 7 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc027d30 osd=3 4 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 10 client.27261.objecter _send_command 7 2024-02-23T00:36:44.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.035+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88bc068200 con 0x7f88bc0fca50 2024-02-23T00:36:44.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f88bc068200 con 0x7f88bc0fca50 2024-02-23T00:36:44.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:44.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88b27fc700 10 client.27261.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc027d30 osd=3 4 2024-02-23T00:36:44.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 3 7 2024-02-23T00:36:44.038 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:36:44.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 10 client.27261.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:44.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 8 2024-02-23T00:36:44.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6800/4175844757 conn(0x7f88bc0fc120 0x7f88bc0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:44.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc0fbfb0 osd=4 3 2024-02-23T00:36:44.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc0fbfb0 osd=4 3 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc0fbfb0 osd=4 3 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 8 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc0fbfb0 osd=4 3 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 4 8 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc0fbfb0 osd=4 4 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 10 client.27261.objecter _send_command 8 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.036+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88bc065990 con 0x7f88bc0fc120 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.037+0000 7f88c18bd700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6800/4175844757 conn(0x7f88bc0fc120 0x7f88bc0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.037+0000 7f88c18bd700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6800/4175844757 conn(0x7f88bc0fc120 0x7f88bc0694f0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.037+0000 7f88b27fc700 10 client.27261.objecter ms_handle_connect 0x7f88bc0fc120 2024-02-23T00:36:44.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.038+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f88bc065990 con 0x7f88bc0fc120 2024-02-23T00:36:44.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.038+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:36:44.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.038+0000 7f88b27fc700 10 client.27261.objecter _finish_command 8 = system:0 2024-02-23T00:36:44.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.038+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc0fbfb0 osd=4 4 2024-02-23T00:36:44.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.038+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 4 8 2024-02-23T00:36:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 10 client.27261.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 9 2024-02-23T00:36:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc0fbfb0 osd=4 3 2024-02-23T00:36:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc0fbfb0 osd=4 3 2024-02-23T00:36:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc0fbfb0 osd=4 3 2024-02-23T00:36:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 9 2024-02-23T00:36:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc0fbfb0 osd=4 3 2024-02-23T00:36:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 4 9 2024-02-23T00:36:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc0fbfb0 osd=4 4 2024-02-23T00:36:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 10 client.27261.objecter _send_command 9 2024-02-23T00:36:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.052+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88bc064dc0 con 0x7f88bc0fc120 2024-02-23T00:36:44.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f88bc064dc0 con 0x7f88bc0fc120 2024-02-23T00:36:44.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:44.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88b27fc700 10 client.27261.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc0fbfb0 osd=4 4 2024-02-23T00:36:44.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 4 9 2024-02-23T00:36:44.055 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:36:44.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 10 client.27261.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:44.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 10 2024-02-23T00:36:44.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6804/2550013795 conn(0x7f88bc1308e0 0x7f88bc132cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc064b80 osd=5 3 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc064b80 osd=5 3 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc064b80 osd=5 3 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 10 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc064b80 osd=5 3 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 5 10 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc064b80 osd=5 4 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 10 client.27261.objecter _send_command 10 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88bc064cf0 con 0x7f88bc1308e0 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c10bc700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6804/2550013795 conn(0x7f88bc1308e0 0x7f88bc132cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88b27fc700 10 client.27261.objecter ms_handle_connect 0x7f88bc1308e0 2024-02-23T00:36:44.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.054+0000 7f88c10bc700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6804/2550013795 conn(0x7f88bc1308e0 0x7f88bc132cd0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:44.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.055+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f88bc064cf0 con 0x7f88bc1308e0 2024-02-23T00:36:44.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.055+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:36:44.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.055+0000 7f88b27fc700 10 client.27261.objecter _finish_command 10 = system:0 2024-02-23T00:36:44.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.055+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc064b80 osd=5 4 2024-02-23T00:36:44.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.055+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 5 10 2024-02-23T00:36:44.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 10 client.27261.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:44.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 11 2024-02-23T00:36:44.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc064b80 osd=5 3 2024-02-23T00:36:44.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc064b80 osd=5 3 2024-02-23T00:36:44.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc064b80 osd=5 3 2024-02-23T00:36:44.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 11 2024-02-23T00:36:44.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc064b80 osd=5 3 2024-02-23T00:36:44.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 5 11 2024-02-23T00:36:44.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc064b80 osd=5 4 2024-02-23T00:36:44.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 10 client.27261.objecter _send_command 11 2024-02-23T00:36:44.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.069+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88bc064cf0 con 0x7f88bc1308e0 2024-02-23T00:36:44.072 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:36:44.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f88bc064cf0 con 0x7f88bc1308e0 2024-02-23T00:36:44.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:44.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88b27fc700 10 client.27261.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc064b80 osd=5 4 2024-02-23T00:36:44.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 5 11 2024-02-23T00:36:44.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 10 client.27261.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:44.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 12 2024-02-23T00:36:44.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6808/2509408498 conn(0x7f88bc1336d0 0x7f88bc135ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc064cf0 osd=6 3 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc064cf0 osd=6 3 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc064cf0 osd=6 3 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 12 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc064cf0 osd=6 3 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 6 12 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc064cf0 osd=6 4 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 10 client.27261.objecter _send_command 12 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88bc138710 con 0x7f88bc1336d0 2024-02-23T00:36:44.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.071+0000 7f88c08bb700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6808/2509408498 conn(0x7f88bc1336d0 0x7f88bc135ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:44.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.072+0000 7f88c08bb700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6808/2509408498 conn(0x7f88bc1336d0 0x7f88bc135ac0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:44.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.072+0000 7f88b27fc700 10 client.27261.objecter ms_handle_connect 0x7f88bc1336d0 2024-02-23T00:36:44.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.072+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f88bc138710 con 0x7f88bc1336d0 2024-02-23T00:36:44.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.072+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:36:44.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.072+0000 7f88b27fc700 10 client.27261.objecter _finish_command 12 = system:0 2024-02-23T00:36:44.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.072+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc064cf0 osd=6 4 2024-02-23T00:36:44.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.072+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 6 12 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 10 client.27261.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 13 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc064cf0 osd=6 3 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc064cf0 osd=6 3 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc064cf0 osd=6 3 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 13 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc064cf0 osd=6 3 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 6 13 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc064cf0 osd=6 4 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 10 client.27261.objecter _send_command 13 2024-02-23T00:36:44.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.087+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88bc135fb0 con 0x7f88bc1336d0 2024-02-23T00:36:44.089 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f88bc135fb0 con 0x7f88bc1336d0 2024-02-23T00:36:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88b27fc700 10 client.27261.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc064cf0 osd=6 4 2024-02-23T00:36:44.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 6 13 2024-02-23T00:36:44.091 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:36:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 10 client.27261.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 14 2024-02-23T00:36:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6812/373787000 conn(0x7f88bc136630 0x7f88bc138aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc135fb0 osd=7 3 2024-02-23T00:36:44.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc135fb0 osd=7 3 2024-02-23T00:36:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc135fb0 osd=7 3 2024-02-23T00:36:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 14 2024-02-23T00:36:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc135fb0 osd=7 3 2024-02-23T00:36:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 7 14 2024-02-23T00:36:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc135fb0 osd=7 4 2024-02-23T00:36:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 10 client.27261.objecter _send_command 14 2024-02-23T00:36:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88bc13b800 con 0x7f88bc136630 2024-02-23T00:36:44.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c18bd700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6812/373787000 conn(0x7f88bc136630 0x7f88bc138aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:44.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88b27fc700 10 client.27261.objecter ms_handle_connect 0x7f88bc136630 2024-02-23T00:36:44.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.089+0000 7f88c18bd700 1 --2- 172.21.15.87:0/2983023900 >> v2:172.21.15.171:6812/373787000 conn(0x7f88bc136630 0x7f88bc138aa0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:44.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.090+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f88bc13b800 con 0x7f88bc136630 2024-02-23T00:36:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.090+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:36:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.090+0000 7f88b27fc700 10 client.27261.objecter _finish_command 14 = system:0 2024-02-23T00:36:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.090+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc135fb0 osd=7 4 2024-02-23T00:36:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.090+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 7 14 2024-02-23T00:36:44.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 10 client.27261.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:36:44.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign -1 15 2024-02-23T00:36:44.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc135fb0 osd=7 3 2024-02-23T00:36:44.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc135fb0 osd=7 3 2024-02-23T00:36:44.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 20 client.27261.objecter _get_session s=0x7f88bc135fb0 osd=7 3 2024-02-23T00:36:44.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 15 client.27261.objecter _session_command_op_remove -1 15 2024-02-23T00:36:44.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 20 client.27261.objecter get_session s=0x7f88bc135fb0 osd=7 3 2024-02-23T00:36:44.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 15 client.27261.objecter _session_command_op_assign 7 15 2024-02-23T00:36:44.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 20 client.27261.objecter put_session s=0x7f88bc135fb0 osd=7 4 2024-02-23T00:36:44.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 10 client.27261.objecter _send_command 15 2024-02-23T00:36:44.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.105+0000 7f88c3320700 1 -- 172.21.15.87:0/2983023900 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f88bc13a660 con 0x7f88bc136630 2024-02-23T00:36:44.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.106+0000 7f88b27fc700 1 -- 172.21.15.87:0/2983023900 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f88bc13a660 con 0x7f88bc136630 2024-02-23T00:36:44.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.106+0000 7f88b27fc700 10 client.27261.objecter ms_dispatch 0x7f88bc001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:36:44.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.107+0000 7f88b27fc700 10 client.27261.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.107+0000 7f88b27fc700 20 client.27261.objecter put_session s=0x7f88bc135fb0 osd=7 4 2024-02-23T00:36:44.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:44.107+0000 7f88b27fc700 15 client.27261.objecter _session_command_op_remove 7 15 2024-02-23T00:36:44.108 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:36:44.109 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:36:44.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.109 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:36:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:44.169 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:44.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:44.208 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-23T00:36:44.282 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:44.321 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:43.268147+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e431: 8 total, 7 up, 7 in 2024-02-23T00:36:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:43.984308+0000 mon.a (mon.0) 2612 : audit [DBG] from='client.? 172.21.15.87:0/2983023900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:44.514 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:36:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:43.268147+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e431: 8 total, 7 up, 7 in 2024-02-23T00:36:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:43.984308+0000 mon.a (mon.0) 2612 : audit [DBG] from='client.? 172.21.15.87:0/2983023900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:43.268147+0000 mon.a (mon.0) 2611 : cluster [DBG] osdmap e431: 8 total, 7 up, 7 in 2024-02-23T00:36:44.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:43.984308+0000 mon.a (mon.0) 2612 : audit [DBG] from='client.? 172.21.15.87:0/2983023900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:44.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:44.785 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:44.812 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:45.361 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:45.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:45.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:36:45.371 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:36:45.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:36:45.465 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:45.493 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:45.566 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:45.567 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:43.661212+0000 mgr.y (mgr.24437) 399 : cluster [DBG] pgmap v640: 238 pgs: 26 active+undersized+degraded, 52 active+undersized, 32 creating+peering, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1149 objects degraded (10.270%) 2024-02-23T00:36:45.568 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:44.293224+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e432: 8 total, 7 up, 7 in 2024-02-23T00:36:45.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:45.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:43.661212+0000 mgr.y (mgr.24437) 399 : cluster [DBG] pgmap v640: 238 pgs: 26 active+undersized+degraded, 52 active+undersized, 32 creating+peering, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1149 objects degraded (10.270%) 2024-02-23T00:36:45.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:44.293224+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e432: 8 total, 7 up, 7 in 2024-02-23T00:36:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:43.661212+0000 mgr.y (mgr.24437) 399 : cluster [DBG] pgmap v640: 238 pgs: 26 active+undersized+degraded, 52 active+undersized, 32 creating+peering, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1149 objects degraded (10.270%) 2024-02-23T00:36:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:44.293224+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e432: 8 total, 7 up, 7 in 2024-02-23T00:36:45.669 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:45.716 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:45.739 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:45.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:45.843 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:45.869 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:45.896 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:45.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:46.000 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:46.030 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:46.044 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:46.050 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:46.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:46.153 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:46.181 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:46.303 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:46.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:46.409 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:46.437 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:46.511 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:46.512 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:45.294008+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e433: 8 total, 7 up, 7 in 2024-02-23T00:36:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:46.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:46.517 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:36:46.517 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:36:46.517 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:36:46.517 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:36:46.517 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:36:46.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:45.294008+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e433: 8 total, 7 up, 7 in 2024-02-23T00:36:46.549 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:45.294008+0000 mon.a (mon.0) 2614 : cluster [DBG] osdmap e433: 8 total, 7 up, 7 in 2024-02-23T00:36:46.614 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:46.641 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:46.889 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:46.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:46.993 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:36:47.020 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:47.225 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-23T00:36:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:47.226 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:36:47.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:36:47.327 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:47.356 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:47.373 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:36:47.503 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:47.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:47.529 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:36:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:45.662438+0000 mgr.y (mgr.24437) 400 : cluster [DBG] pgmap v643: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:36:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:45.662438+0000 mgr.y (mgr.24437) 400 : cluster [DBG] pgmap v643: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:36:47.608 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:47.634 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:45.662438+0000 mgr.y (mgr.24437) 400 : cluster [DBG] pgmap v643: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 2.5 KiB/s wr, 7 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:36:47.785 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:47.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:47.892 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:47.921 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:48.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.083+0000 7ff44f35b700 1 -- 172.21.15.87:0/1302182253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480fd510 msgr2=0x7ff4480f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:48.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.083+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1302182253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480fd510 0x7ff4480f6d40 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 rx=0x7ff4380047b0 tx=0x7ff438009cb0).stop 2024-02-23T00:36:48.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.083+0000 7ff44f35b700 1 -- 172.21.15.87:0/1302182253 shutdown_connections 2024-02-23T00:36:48.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.083+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1302182253 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4480f7400 0x7ff448103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:48.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.083+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1302182253 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4480fcbb0 0x7ff4480fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:48.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.083+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1302182253 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480fd510 0x7ff4480f6d40 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:48.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.083+0000 7ff44f35b700 1 -- 172.21.15.87:0/1302182253 >> 172.21.15.87:0/1302182253 conn(0x7ff4480f2820 msgr2=0x7ff4480f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:48.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.083+0000 7ff44f35b700 1 -- 172.21.15.87:0/1302182253 shutdown_connections 2024-02-23T00:36:48.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.084+0000 7ff44f35b700 1 -- 172.21.15.87:0/1302182253 wait complete. 2024-02-23T00:36:48.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.084+0000 7ff44f35b700 1 librados: starting msgr at 2024-02-23T00:36:48.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.084+0000 7ff44f35b700 1 librados: starting objecter 2024-02-23T00:36:48.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.084+0000 7ff44f35b700 1 Processor -- start 2024-02-23T00:36:48.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.084+0000 7ff44f35b700 1 -- start start 2024-02-23T00:36:48.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.084+0000 7ff44f35b700 1 librados: setting wanted keys 2024-02-23T00:36:48.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.084+0000 7ff44f35b700 1 librados: calling monclient init 2024-02-23T00:36:48.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44f35b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480f7400 0x7ff4481070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44f35b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4480fcbb0 0x7ff4481075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44f35b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4480fd510 0x7ff448107aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44f35b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448116740 con 0x7ff4480f7400 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44f35b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff448116880 con 0x7ff4480fd510 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44f35b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481169f0 con 0x7ff4480fcbb0 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d0f7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480f7400 0x7ff4481070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d0f7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480f7400 0x7ff4481070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39106/0 (socket says 172.21.15.87:39106) 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d0f7700 1 -- 172.21.15.87:0/1171413715 learned_addr learned my addr 172.21.15.87:0/1171413715 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d8f8700 1 --2- 172.21.15.87:0/1171413715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4480fd510 0x7ff448107aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44c8f6700 1 --2- 172.21.15.87:0/1171413715 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4480fcbb0 0x7ff4481075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d0f7700 1 -- 172.21.15.87:0/1171413715 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4480fcbb0 msgr2=0x7ff4481075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:48.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d0f7700 1 --2- 172.21.15.87:0/1171413715 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4480fcbb0 0x7ff4481075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d0f7700 1 -- 172.21.15.87:0/1171413715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4480fd510 msgr2=0x7ff448107aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d0f7700 1 --2- 172.21.15.87:0/1171413715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4480fd510 0x7ff448107aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d0f7700 1 -- 172.21.15.87:0/1171413715 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff444005040 con 0x7ff4480f7400 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44d0f7700 1 --2- 172.21.15.87:0/1171413715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480f7400 0x7ff4481070c0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7ff444009400 tx=0x7ff444009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff43e7fc700 10 client.?.objecter ms_handle_connect 0x7ff4480f7400 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff43e7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff43e7fc700 1 -- 172.21.15.87:0/1171413715 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444007a40 con 0x7ff4480f7400 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff43e7fc700 1 -- 172.21.15.87:0/1171413715 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff44400f040 con 0x7ff4480f7400 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff438005040 con 0x7ff4480f7400 2024-02-23T00:36:48.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff43e7fc700 1 -- 172.21.15.87:0/1171413715 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff444014a70 con 0x7ff4480f7400 2024-02-23T00:36:48.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.085+0000 7ff44f35b700 10 client.27267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:48.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.086+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff448117030 con 0x7ff4480f7400 2024-02-23T00:36:48.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.086+0000 7ff44f35b700 1 librados: init done 2024-02-23T00:36:48.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.087+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44805fa60 con 0x7ff4480f7400 2024-02-23T00:36:48.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.087+0000 7ff43e7fc700 1 -- 172.21.15.87:0/1171413715 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff444010530 con 0x7ff4480f7400 2024-02-23T00:36:48.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.088+0000 7ff43e7fc700 1 --2- 172.21.15.87:0/1171413715 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff434062c90 0x7ff434065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:48.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.088+0000 7ff43e7fc700 1 -- 172.21.15.87:0/1171413715 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(433..433 src has 173..433) v4 ==== 8178+0+0 (secure 0 0 0) 0x7ff44407ce40 con 0x7ff4480f7400 2024-02-23T00:36:48.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.088+0000 7ff43e7fc700 10 client.27267.objecter ms_dispatch 0x7ff448001550 osd_map(433..433 src has 173..433) v4 2024-02-23T00:36:48.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.088+0000 7ff43e7fc700 3 client.27267.objecter handle_osd_map got epochs [433,433] > 0 2024-02-23T00:36:48.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.088+0000 7ff43e7fc700 3 client.27267.objecter handle_osd_map decoding full epoch 433 2024-02-23T00:36:48.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.088+0000 7ff44c8f6700 1 --2- 172.21.15.87:0/1171413715 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff434062c90 0x7ff434065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:48.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.088+0000 7ff43e7fc700 20 client.27267.objecter dump_active .. 0 homeless 2024-02-23T00:36:48.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.088+0000 7ff43e7fc700 10 client.27267.objecter ms_handle_connect 0x7ff434062c90 2024-02-23T00:36:48.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.088+0000 7ff44c8f6700 1 --2- 172.21.15.87:0/1171413715 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff434062c90 0x7ff434065150 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7ff448061700 tx=0x7ff438009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:48.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.092+0000 7ff43e7fc700 1 -- 172.21.15.87:0/1171413715 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff444049f60 con 0x7ff4480f7400 2024-02-23T00:36:48.110 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.284+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff4481060e0 con 0x7ff4480f7400 2024-02-23T00:36:48.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.310+0000 7ff43e7fc700 1 -- 172.21.15.87:0/1171413715 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v434) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff444050290 con 0x7ff4480f7400 2024-02-23T00:36:48.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 10 librados: watch_flush enter 2024-02-23T00:36:48.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 10 librados: watch_flush exit 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 20 client.27267.objecter shutdown clearing up homeless session... 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 10 client.27267.objecter successfully canceled tick 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff434062c90 msgr2=0x7ff434065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1171413715 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff434062c90 0x7ff434065150 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 rx=0x7ff448061700 tx=0x7ff438009cb0).stop 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480f7400 msgr2=0x7ff4481070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1171413715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480f7400 0x7ff4481070c0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 rx=0x7ff444009400 tx=0x7ff444009cb0).stop 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 shutdown_connections 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1171413715 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff434062c90 0x7ff434065150 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1171413715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff4480fd510 0x7ff448107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1171413715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff4480f7400 0x7ff4481070c0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 --2- 172.21.15.87:0/1171413715 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff4480fcbb0 0x7ff4481075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 >> 172.21.15.87:0/1171413715 conn(0x7ff4480f2820 msgr2=0x7ff4480f43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:48.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 shutdown_connections 2024-02-23T00:36:48.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 -- 172.21.15.87:0/1171413715 wait complete. 2024-02-23T00:36:48.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:48.313+0000 7ff44f35b700 1 librados: shutdown 2024-02-23T00:36:48.316 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:36:48.340 INFO:teuthology.orchestra.run.smithi087.stdout:63.c 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:63.b 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:63.6 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:63.5 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:63.2 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:63.1 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:63.16 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:63.14 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:63.10 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:63.0 2024-02-23T00:36:48.341 INFO:teuthology.orchestra.run.smithi087.stdout:6.d 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:6.a 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:6.9 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:6.18 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:22.13 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:22.0 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:212.1 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:3.6 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:212.8 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:5.5 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:5.b 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:212.1b 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:212.e 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:6.13 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:2.8 2024-02-23T00:36:48.342 INFO:teuthology.orchestra.run.smithi087.stdout:212.15 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:5.1e 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:212.6 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:212.11 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:3.b 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:5.29 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:212.1c 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:212.14 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:5.c 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:212.10 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:212.0 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:5.a 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:2.13 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:2.0 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:2.12 2024-02-23T00:36:48.343 INFO:teuthology.orchestra.run.smithi087.stdout:2.15 2024-02-23T00:36:48.344 INFO:teuthology.orchestra.run.smithi087.stdout:2.1 2024-02-23T00:36:48.344 INFO:teuthology.orchestra.run.smithi087.stdout:2.19 2024-02-23T00:36:48.344 INFO:teuthology.orchestra.run.smithi087.stdout:63.1f 2024-02-23T00:36:48.344 INFO:teuthology.orchestra.run.smithi087.stdout:3.7 2024-02-23T00:36:48.344 INFO:teuthology.orchestra.run.smithi087.stdout:212.5 2024-02-23T00:36:48.344 INFO:teuthology.orchestra.run.smithi087.stdout:2.c 2024-02-23T00:36:48.344 INFO:teuthology.orchestra.run.smithi087.stdout:2.10 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:2.1f 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:212.1d 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:1.0 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:2.f 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:22.1 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:4.7 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:2.1e 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:22.1a 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:22.9 2024-02-23T00:36:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:22.a 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:22.c 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:2.5 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:3.1 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:5.8 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:5.7 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:5.0 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:22.1f 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:212.19 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:4.0 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:63.12 2024-02-23T00:36:48.346 INFO:teuthology.orchestra.run.smithi087.stdout:4.2 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:4.3 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:22.18 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:2.4 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:4.6 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:5.1 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:5.23 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:2.1d 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:5.13 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:6.0 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:5.17 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:4.b 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:5.1b 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:5.1d 2024-02-23T00:36:48.347 INFO:teuthology.orchestra.run.smithi087.stdout:212.4 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:5.15 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:5.2 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:5.24 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:212.f 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:5.26 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:5.3 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:5.e 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:6.11 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:3.3 2024-02-23T00:36:48.348 INFO:teuthology.orchestra.run.smithi087.stdout:6.10 2024-02-23T00:36:48.349 INFO:teuthology.orchestra.run.smithi087.stdout:6.16 2024-02-23T00:36:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:48.285546+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? 172.21.15.87:0/1171413715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:48.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:48.285546+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? 172.21.15.87:0/1171413715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:48.653 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:48.285546+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? 172.21.15.87:0/1171413715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:36:48.670 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:48.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:48.774 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:48.802 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24712.0:14 7.5 7:a3910452:test-rados-api-smithi087-82115-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:30:58.506107+0000", 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 350.70109329100001, 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021562957000000001, 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.213 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24712", 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2285581802", 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 14 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506107+0000", 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506107+0000", 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.214 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506105+0000", 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5230000000000001e-06 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506114+0000", 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1732999999999999e-05 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506126+0000", 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4780000000000002e-06 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.215 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506133+0000", 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9349999999999999e-05 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506193+0000", 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012566300000000001 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506318+0000", 2024-02-23T00:36:49.216 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017116500000000001 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0,2", 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506490+0000", 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072723850000000001 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.513762+0000", 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046023009999999996 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.518364+0000", 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0092229960000000007 2024-02-23T00:36:49.217 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.527587+0000", 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5426999999999998e-05 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.527623+0000", 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7034999999999999e-05 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.218 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.527670+0000", 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021562957000000001 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26036.0:2 64.6 64:602f83fe:::foo:head [write 0~128,set-alloc-hint object_size 0 write_size 0] snapc 0=[] ondisk+write+known_if_redirected e115)", 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 342.98232442, 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036711013000000001, 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.219 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26036", 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3255892024", 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.220 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224873+0000", 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.163e-05 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224885+0000", 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8150000000000001e-05 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224903+0000", 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2343e-05 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.221 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224915+0000", 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6192000000000005e-05 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224992+0000", 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4450000000000001e-06 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224998+0000", 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032120923000000003 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.222 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257119+0000", 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125234 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257244+0000", 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107636 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,5", 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257352+0000", 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011677390000000001 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.223 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.258520+0000", 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014228299999999999 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.259942+0000", 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015894640000000001 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261532+0000", 2024-02-23T00:36:49.224 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.154e-05 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261563+0000", 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3230999999999998e-05 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261587+0000", 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036711013000000001 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.225 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDScrubReserve(10.1 RELEASE e139)", 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:30.036961+0000", 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 319.17023900999999, 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011677659, 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036961+0000", 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.226 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036961+0000", 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036960+0000", 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6599999999999998e-06 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036967+0000", 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3458000000000001e-05 2024-02-23T00:36:49.227 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036980+0000", 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.007e-06 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.036987+0000", 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6214e-05 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037053+0000", 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8050000000000007e-06 2024-02-23T00:36:49.228 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.037061+0000", 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011533062 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.048594+0000", 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8111999999999999e-05 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.048612+0000", 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6262999999999999e-05 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.229 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:30.048639+0000", 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011677659 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:2 93.7s0 93:f24ec45b:test-rados-api-smithi087-82107-15::foo:head [write 0~8192] snapc 2=[2] ondisk+write+known_if_redirected e143)", 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:34.002899+0000", 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 315.204300859, 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079244069999999996, 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.230 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002899+0000", 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002899+0000", 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.231 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002897+0000", 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5637e-05 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002913+0000", 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0621e-05 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002923+0000", 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1270000000000003e-06 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.232 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.002932+0000", 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00029541399999999997 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.003227+0000", 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1740000000000002e-06 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.003231+0000", 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0034875129999999998 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.006719+0000", 2024-02-23T00:36:49.233 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000150543 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.006869+0000", 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000472211 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.007341+0000", 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00075650800000000003 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.008098+0000", 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00265423 2024-02-23T00:36:49.234 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.010752+0000", 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1404999999999998e-05 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.010824+0000", 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0079244069999999996 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.235 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:4 93.7s0 93:f24ec45b:test-rados-api-smithi087-82107-15::foo:head [write 8192~8192] snapc 3=[3,2] ondisk+write+known_if_redirected e144)", 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 314.21102520599999, 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028560524, 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 4 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.236 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996173+0000", 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1501e-05 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996194+0000", 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1659000000000003e-05 2024-02-23T00:36:49.237 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996226+0000", 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.439e-06 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996234+0000", 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013176539999999999 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.997552+0000", 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5900000000000008e-06 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.238 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.997560+0000", 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0057013619999999997 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.003262+0000", 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015312100000000001 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.003415+0000", 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033453049999999998 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.239 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.006760+0000", 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038155379999999998 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.010576+0000", 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014109313 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.024685+0000", 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0339e-05 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.240 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.024735+0000", 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028560524 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:9 93.7s0 93:f24ec45b:test-rados-api-smithi087-82107-15::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e146)", 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:36.987620+0000", 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 312.21957986699999, 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017235619000000001, 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:36:49.241 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 9 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987620+0000", 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987620+0000", 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.242 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987618+0000", 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7619999999999995e-06 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987626+0000", 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0876999999999999e-05 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987637+0000", 2024-02-23T00:36:49.243 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4093e-05 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.987651+0000", 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001414867 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.989066+0000", 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5760000000000006e-06 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.244 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.989074+0000", 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040788279999999996 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.993153+0000", 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100617 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.993254+0000", 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038279699999999999 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.245 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.993637+0000", 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000640954 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:36.994278+0000", 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010509697 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.004787+0000", 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8483000000000001e-05 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.246 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.004856+0000", 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.017235619000000001 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:11 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [write 0~8192] snapc 6=[6] ondisk+write+known_if_redirected e147)", 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:37.991540+0000", 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 311.21565973700001, 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020032722999999999, 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 11 2024-02-23T00:36:49.247 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991540+0000", 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991540+0000", 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.248 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991538+0000", 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2162000000000001e-05 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991570+0000", 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4e-05 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991584+0000", 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1769999999999998e-06 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991593+0000", 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016113087000000002 2024-02-23T00:36:49.249 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.007706+0000", 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016325899999999999 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.007869+0000", 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000521098 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.008390+0000", 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0016975200000000001 2024-02-23T00:36:49.250 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.010088+0000", 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001418084 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.011506+0000", 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7416999999999995e-05 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.011573+0000", 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020032722999999999 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.251 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:13 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [write 8192~8192] snapc 7=[7,6] ondisk+write+known_if_redirected e148)", 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:38.986625+0000", 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 310.22057478800002, 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010875549999999999, 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 13 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.252 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986625+0000", 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986625+0000", 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986623+0000", 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1804000000000002e-05 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.253 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986665+0000", 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2981e-05 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986678+0000", 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6139999999999992e-06 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.986687+0000", 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001460254 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.254 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.988147+0000", 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9940000000000006e-06 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.988156+0000", 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020480580000000002 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.990204+0000", 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000116233 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.990320+0000", 2024-02-23T00:36:49.255 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00062543799999999997 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.990946+0000", 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001065908 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.992012+0000", 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0054126460000000001 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.997424+0000", 2024-02-23T00:36:49.256 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6672e-05 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.997501+0000", 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010875549999999999 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:14 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [rollback 7] snapc 7=[7,6] ondisk+write+known_if_redirected e148)", 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:38.998508+0000", 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 310.20869253299998, 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010648487, 2024-02-23T00:36:49.257 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 14 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998508+0000", 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998508+0000", 2024-02-23T00:36:49.258 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998506+0000", 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5909999999999998e-06 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998513+0000", 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1420000000000001e-05 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998525+0000", 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7179999999999999e-06 2024-02-23T00:36:49.259 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998531+0000", 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111612 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998643+0000", 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000123659 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.998767+0000", 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000483432 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.260 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.999250+0000", 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0069559019999999999 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.006206+0000", 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028810699999999999 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.009087+0000", 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8885000000000002e-05 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.261 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.009156+0000", 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010648487 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDECSubOpWriteReply(93.0s0 148/142 ECSubWriteReply(tid=250, last_complete=148'4, committed=1, applied=1))", 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:39.001273+0000", 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 310.20592686499998, 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076685520000000004, 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.262 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001273+0000", 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001273+0000", 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001273+0000", 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2320000000000001e-06 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001276+0000", 2024-02-23T00:36:49.263 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9399999999999999e-06 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001282+0000", 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9769999999999997e-06 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.001288+0000", 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00763885 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.008927+0000", 2024-02-23T00:36:49.264 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4881e-05 2024-02-23T00:36:49.267 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.267 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:39.008942+0000", 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076685520000000004 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:18 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [delete] snapc 7=[7,6] ondisk+write+peerstat_old+known_if_redirected e150)", 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 308.21594896900001, 2024-02-23T00:36:49.268 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026236400999999999, 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 18 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.269 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991249+0000", 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0203999999999999e-05 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991259+0000", 2024-02-23T00:36:49.270 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5091e-05 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991274+0000", 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.729e-06 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991282+0000", 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7206999999999995e-05 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991349+0000", 2024-02-23T00:36:49.271 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134998 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991484+0000", 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00048211400000000001 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991966+0000", 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00039970199999999999 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.272 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.992366+0000", 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025049227 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.017415+0000", 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2229000000000003e-05 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.017488+0000", 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026236400999999999 2024-02-23T00:36:49.273 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.274 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.274 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.274 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.274 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.274 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:874 95.7 95:e6233af9:test-rados-api-smithi087-82091-14::foo5:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-23T00:36:49.274 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:42.006991+0000", 2024-02-23T00:36:49.274 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 307.20020862699999, 2024-02-23T00:36:49.274 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082505240000000004, 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 874 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.006991+0000", 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.275 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.006991+0000", 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.006989+0000", 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4066999999999999e-05 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.276 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007003+0000", 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4657999999999999e-05 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007018+0000", 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6989999999999997e-06 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007025+0000", 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5380999999999998e-05 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.277 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007070+0000", 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000206398 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.007276+0000", 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002540994 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.278 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.009817+0000", 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8937999999999994e-05 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.009916+0000", 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050415399999999997 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.279 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.014958+0000", 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0696000000000003e-05 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.015019+0000", 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000223322 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.280 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:42.015242+0000", 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0082505240000000004 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26475.0:2 125.4s0 125:3c44b853:test-rados-api-smithi087-82115-21::foo:head [write 0~8192] snapc 2=[2] ondisk+write+known_if_redirected e182)", 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:13.006424+0000", 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 276.20077641, 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018917415999999999, 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.281 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.282 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26475", 2024-02-23T00:36:49.282 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/1917629916", 2024-02-23T00:36:49.282 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:36:49.282 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.282 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.282 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.282 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.282 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006424+0000", 2024-02-23T00:36:49.283 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.283 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.283 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.283 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.283 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006424+0000", 2024-02-23T00:36:49.283 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:49.284 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.284 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.284 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.284 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006422+0000", 2024-02-23T00:36:49.284 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4351e-05 2024-02-23T00:36:49.284 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.285 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.285 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.285 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006436+0000", 2024-02-23T00:36:49.285 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2189e-05 2024-02-23T00:36:49.285 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.285 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.286 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.286 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006448+0000", 2024-02-23T00:36:49.286 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7499999999999997e-06 2024-02-23T00:36:49.286 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.286 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.286 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.287 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006455+0000", 2024-02-23T00:36:49.287 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0914000000000001e-05 2024-02-23T00:36:49.287 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.287 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.287 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.287 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006506+0000", 2024-02-23T00:36:49.287 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.571e-06 2024-02-23T00:36:49.288 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.288 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.288 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:49.288 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.006509+0000", 2024-02-23T00:36:49.288 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013345159000000001 2024-02-23T00:36:49.288 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.289 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.289 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.289 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.019854+0000", 2024-02-23T00:36:49.289 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000106708 2024-02-23T00:36:49.289 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.289 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.290 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.290 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.019960+0000", 2024-02-23T00:36:49.290 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036460300000000002 2024-02-23T00:36:49.290 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.290 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.290 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.291 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.020325+0000", 2024-02-23T00:36:49.291 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00071362200000000002 2024-02-23T00:36:49.291 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.291 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.291 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.291 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.021039+0000", 2024-02-23T00:36:49.292 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0042323819999999998 2024-02-23T00:36:49.292 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.292 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.292 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.292 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.025271+0000", 2024-02-23T00:36:49.292 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0007999999999998e-05 2024-02-23T00:36:49.292 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.292 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.293 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.293 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.025341+0000", 2024-02-23T00:36:49.293 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018917415999999999 2024-02-23T00:36:49.293 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.293 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.293 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.293 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.294 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.294 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36947.0:2 126.3 126:c1e600bb:test-rados-api-smithi087-82083-31::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected e182)", 2024-02-23T00:36:49.294 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:13.009044+0000", 2024-02-23T00:36:49.294 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 276.19815598299999, 2024-02-23T00:36:49.294 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015740071000000001, 2024-02-23T00:36:49.294 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.295 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.295 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.295 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36947", 2024-02-23T00:36:49.295 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3435478986", 2024-02-23T00:36:49.295 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:36:49.295 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.296 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.296 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.296 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.296 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009044+0000", 2024-02-23T00:36:49.296 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.296 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.297 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.297 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.297 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009044+0000", 2024-02-23T00:36:49.297 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:36:49.297 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.297 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.298 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.298 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009043+0000", 2024-02-23T00:36:49.298 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9830000000000008e-06 2024-02-23T00:36:49.298 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.298 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.298 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009053+0000", 2024-02-23T00:36:49.299 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3186e-05 2024-02-23T00:36:49.299 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.299 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.299 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.299 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009066+0000", 2024-02-23T00:36:49.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7959999999999998e-06 2024-02-23T00:36:49.300 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.300 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.300 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.300 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009073+0000", 2024-02-23T00:36:49.300 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016034999999999999 2024-02-23T00:36:49.301 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.301 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.301 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.301 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009233+0000", 2024-02-23T00:36:49.301 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5100000000000004e-06 2024-02-23T00:36:49.301 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.302 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.302 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:49.302 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.009240+0000", 2024-02-23T00:36:49.302 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011721904 2024-02-23T00:36:49.302 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.302 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.302 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.303 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.020961+0000", 2024-02-23T00:36:49.303 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8343000000000004e-05 2024-02-23T00:36:49.303 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.303 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.303 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.303 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.021060+0000", 2024-02-23T00:36:49.303 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012799299999999999 2024-02-23T00:36:49.304 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.304 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.304 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,4", 2024-02-23T00:36:49.304 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.021188+0000", 2024-02-23T00:36:49.304 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00049047399999999997 2024-02-23T00:36:49.304 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.305 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.305 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:36:49.305 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.021678+0000", 2024-02-23T00:36:49.305 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00064367000000000005 2024-02-23T00:36:49.305 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.306 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.306 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:49.306 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.022322+0000", 2024-02-23T00:36:49.306 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023954129999999999 2024-02-23T00:36:49.306 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.306 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.307 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:36:49.307 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.024717+0000", 2024-02-23T00:36:49.307 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6732e-05 2024-02-23T00:36:49.307 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.308 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.308 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.308 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.024754+0000", 2024-02-23T00:36:49.308 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0110000000000001e-05 2024-02-23T00:36:49.308 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.308 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.309 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.309 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:13.024784+0000", 2024-02-23T00:36:49.309 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015740071000000001 2024-02-23T00:36:49.309 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.309 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.309 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.310 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.310 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.310 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26475.0:4 125.4s0 125:3c44b853:test-rados-api-smithi087-82115-21::foo:head [write 8192~8192] snapc 3=[3,2] ondisk+write+known_if_redirected e183)", 2024-02-23T00:36:49.310 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:14.242083+0000", 2024-02-23T00:36:49.310 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 274.96511722399998, 2024-02-23T00:36:49.311 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014358816999999999, 2024-02-23T00:36:49.311 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.311 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.311 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.311 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26475", 2024-02-23T00:36:49.311 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/1917629916", 2024-02-23T00:36:49.311 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 4 2024-02-23T00:36:49.312 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.312 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.312 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.312 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.312 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242083+0000", 2024-02-23T00:36:49.313 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.313 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.313 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.313 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.313 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242083+0000", 2024-02-23T00:36:49.313 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:36:49.313 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.313 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.314 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.314 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242081+0000", 2024-02-23T00:36:49.314 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2780999999999997e-05 2024-02-23T00:36:49.314 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.314 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.314 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.314 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242114+0000", 2024-02-23T00:36:49.314 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0647000000000001e-05 2024-02-23T00:36:49.315 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.315 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.315 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.315 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242135+0000", 2024-02-23T00:36:49.315 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1789999999999996e-06 2024-02-23T00:36:49.315 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.315 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.315 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.315 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242141+0000", 2024-02-23T00:36:49.316 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5528000000000002e-05 2024-02-23T00:36:49.316 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.316 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.316 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.316 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242186+0000", 2024-02-23T00:36:49.316 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0899999999999999e-06 2024-02-23T00:36:49.316 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.316 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.317 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:36:49.317 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.242194+0000", 2024-02-23T00:36:49.317 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0089040109999999999 2024-02-23T00:36:49.317 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.317 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.317 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.251098+0000", 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9748999999999997e-05 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.251197+0000", 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00061586500000000005 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.251813+0000", 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00098538699999999989 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.318 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.252799+0000", 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035701940000000001 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.256369+0000", 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2904000000000005e-05 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:14.256442+0000", 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014358816999999999 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.319 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.44587.0:1 151.6s0 151:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 239.16888145999999, 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763, 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.44587", 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/843545378", 2024-02-23T00:36:49.320 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.321 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038316+0000", 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9619999999999999e-06 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038325+0000", 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7385e-05 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038343+0000", 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0739999999999999e-06 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.322 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038352+0000", 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077681699999999996 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039128+0000", 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0860000000000002e-06 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039132+0000", 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0036840789999999 2024-02-23T00:36:49.323 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042816+0000", 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000158937 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042975+0000", 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000568418 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.043543+0000", 2024-02-23T00:36:49.324 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00061003399999999997 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.044153+0000", 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011287789999999999 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045282+0000", 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9601e-05 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045351+0000", 2024-02-23T00:36:49.325 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26886.0:1 181.5s0 181:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e278)", 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 180.16087921600001, 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133, 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26886", 2024-02-23T00:36:49.326 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/134760323", 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.327 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046318+0000", 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9741999999999998e-05 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046338+0000", 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2594e-05 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046351+0000", 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5399999999999996e-06 2024-02-23T00:36:49.328 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.329 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.329 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046359+0000", 2024-02-23T00:36:49.329 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00098343299999999992 2024-02-23T00:36:49.329 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.329 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047343+0000", 2024-02-23T00:36:49.330 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9160000000000003e-06 2024-02-23T00:36:49.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.330 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.330 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:49.330 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047347+0000", 2024-02-23T00:36:49.330 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0029936850000001 2024-02-23T00:36:49.330 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050340+0000", 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014100300000000001 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050481+0000", 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038932099999999998 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050871+0000", 2024-02-23T00:36:49.331 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00045121199999999999 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.051322+0000", 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00089874699999999998 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052221+0000", 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1542999999999998e-05 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.332 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052272+0000", 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27003.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 135.14672551800001, 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568, 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.333 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27003", 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3963489775", 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.334 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060473+0000", 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.417e-06 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060479+0000", 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8080000000000005e-06 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060486+0000", 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4359999999999999e-06 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.335 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060493+0000", 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6943999999999997e-05 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060530+0000", 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4229999999999999e-06 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060532+0000", 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.99955465700000001 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.336 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060087+0000", 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2388999999999999e-05 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060179+0000", 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00037435399999999997 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060554+0000", 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00057048899999999998 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.337 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061124+0000", 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00075843999999999998 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061883+0000", 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8547999999999998e-05 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061921+0000", 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.338 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37526.0:1 199.2s0 199:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 111.12897766899999, 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.011307347, 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37526", 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/4086862305", 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:36:49.339 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.340 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078221+0000", 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1790000000000008e-06 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078229+0000", 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1657e-05 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078240+0000", 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5909999999999996e-06 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.341 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078249+0000", 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011604219999999999 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.079409+0000", 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0110000000000001e-06 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.079412+0000", 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0080054119999999 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.342 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087418+0000", 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014009999999999999 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087558+0000", 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00028455700000000001 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087842+0000", 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00026429900000000001 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.343 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.088107+0000", 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013807730000000001 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.089488+0000", 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2237000000000002e-05 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.089530+0000", 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.011307347 2024-02-23T00:36:49.344 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:5324 200.4 200:20000000:%01:::head [pgnls start_epoch 369] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e369)", 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:35:20.113022+0000", 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 89.094178080999995, 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0074374920000000004, 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 5324 2024-02-23T00:36:49.345 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.113022+0000", 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.113022+0000", 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:36:49.346 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.113019+0000", 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1982e-05 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.113031+0000", 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0956999999999999e-05 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.113052+0000", 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2930000000000001e-06 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:36:49.347 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.113061+0000", 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5016000000000002e-05 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.113126+0000", 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6510000000000004e-06 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.113132+0000", 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072187709999999997 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.120351+0000", 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7742999999999998e-05 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.120379+0000", 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0673000000000005e-05 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:20.120459+0000", 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0074374920000000004 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.349 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.350 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:36:49.350 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:36:49.350 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:36:49.351 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:49.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:49.455 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:36:49.484 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:49.503 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:36:49.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:47.662910+0000 mgr.y (mgr.24437) 401 : cluster [DBG] pgmap v644: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.9 KiB/s wr, 5 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:36:49.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:48.301388+0000 mon.a (mon.0) 2616 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:49.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:48.310291+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? 172.21.15.87:0/1171413715' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:36:49.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:48.310353+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e434: 8 total, 7 up, 7 in 2024-02-23T00:36:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:49.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:36:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:47.662910+0000 mgr.y (mgr.24437) 401 : cluster [DBG] pgmap v644: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.9 KiB/s wr, 5 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:36:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:48.301388+0000 mon.a (mon.0) 2616 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:48.310291+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? 172.21.15.87:0/1171413715' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:36:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:48.310353+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e434: 8 total, 7 up, 7 in 2024-02-23T00:36:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:47.662910+0000 mgr.y (mgr.24437) 401 : cluster [DBG] pgmap v644: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.9 KiB/s wr, 5 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:36:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:48.301388+0000 mon.a (mon.0) 2616 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:36:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:48.310291+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? 172.21.15.87:0/1171413715' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:36:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:48.310353+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e434: 8 total, 7 up, 7 in 2024-02-23T00:36:49.606 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:49.634 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:49.871 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-23T00:36:49.882 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:49.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:49.923 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -n osd.0 --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 212.4 --file /var/log/ceph/exp.212.4.0 2024-02-23T00:36:49.986 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:50.013 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:50.033 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:50.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:50.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:50.137 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:50.163 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:50.510 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:50.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:50.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:50.614 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:50.641 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:51.189 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:51.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:51.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:36:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:36:51.294 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:51.329 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:51.351 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:51.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:51.455 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:36:51.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.456+0000 7f0fc00d3700 1 -- 172.21.15.87:0/1714883942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fb80f6a90 msgr2=0x7f0fb8104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:51.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.456+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/1714883942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fb80f6a90 0x7f0fb8104a20 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 rx=0x7f0fb0009be0 tx=0x7f0fb0004d10).stop 2024-02-23T00:36:51.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.456+0000 7f0fc00d3700 1 -- 172.21.15.87:0/1714883942 shutdown_connections 2024-02-23T00:36:51.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.456+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/1714883942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fb80f6a90 0x7f0fb8104a20 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:51.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.456+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/1714883942 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fb80fdbd0 0x7f0fb80fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:51.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.456+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/1714883942 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fb80fe530 0x7f0fb80f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:51.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.456+0000 7f0fc00d3700 1 -- 172.21.15.87:0/1714883942 >> 172.21.15.87:0/1714883942 conn(0x7f0fb806c370 msgr2=0x7f0fb806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:36:51.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.456+0000 7f0fc00d3700 1 -- 172.21.15.87:0/1714883942 shutdown_connections 2024-02-23T00:36:51.467 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.456+0000 7f0fc00d3700 1 -- 172.21.15.87:0/1714883942 wait complete. 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 librados: starting msgr at 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 librados: starting objecter 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 Processor -- start 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 -- start start 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 librados: setting wanted keys 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 librados: calling monclient init 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fb80f6a90 0x7f0fb80f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fb80fdbd0 0x7f0fb80f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fb80fe530 0x7f0fb80f8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb81164f0 con 0x7f0fb80fdbd0 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb8116660 con 0x7f0fb80fe530 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fc00d3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb8116820 con 0x7f0fb80f6a90 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbde6f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fb80f6a90 0x7f0fb80f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbd66e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fb80fdbd0 0x7f0fb80f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbde6f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fb80f6a90 0x7f0fb80f7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:59848/0 (socket says 172.21.15.87:59848) 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbde6f700 1 -- 172.21.15.87:0/2287558011 learned_addr learned my addr 172.21.15.87:0/2287558011 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbe670700 1 --2- 172.21.15.87:0/2287558011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fb80fe530 0x7f0fb80f8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbde6f700 1 -- 172.21.15.87:0/2287558011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fb80fe530 msgr2=0x7f0fb80f8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbde6f700 1 --2- 172.21.15.87:0/2287558011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fb80fe530 0x7f0fb80f8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbde6f700 1 -- 172.21.15.87:0/2287558011 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fb80fdbd0 msgr2=0x7f0fb80f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbde6f700 1 --2- 172.21.15.87:0/2287558011 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fb80fdbd0 0x7f0fb80f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbde6f700 1 -- 172.21.15.87:0/2287558011 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb4005040 con 0x7f0fb80f6a90 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.457+0000 7f0fbd66e700 1 --2- 172.21.15.87:0/2287558011 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fb80fdbd0 0x7f0fb80f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0fbde6f700 1 --2- 172.21.15.87:0/2287558011 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fb80f6a90 0x7f0fb80f7680 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 rx=0x7f0fb40048f0 tx=0x7f0fb4009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0faeffd700 10 client.?.objecter ms_handle_connect 0x7f0fb80f6a90 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0faeffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4007b00 con 0x7f0fb80f6a90 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb00097c0 con 0x7f0fb80f6a90 2024-02-23T00:36:51.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fb400f040 con 0x7f0fb80f6a90 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4014a00 con 0x7f0fb80f6a90 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0fc00d3700 10 client.37733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb8116e30 con 0x7f0fb80f6a90 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.458+0000 7f0fc00d3700 1 librados: init done 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.459+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0fb805f930 con 0x7f0fb80f6a90 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.459+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0fb400e580 con 0x7f0fb80f6a90 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.459+0000 7f0faeffd700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0fa4062c90 0x7f0fa4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.459+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(434..434 src has 173..434) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f0fb407cde0 con 0x7f0fb80f6a90 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.459+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 osd_map(434..434 src has 173..434) v4 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.459+0000 7f0faeffd700 3 client.37733.objecter handle_osd_map got epochs [434,434] > 0 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.459+0000 7f0faeffd700 3 client.37733.objecter handle_osd_map decoding full epoch 434 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.459+0000 7f0fbd66e700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0fa4062c90 0x7f0fa4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0faeffd700 20 client.37733.objecter dump_active .. 0 homeless 2024-02-23T00:36:51.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v434) v1 ==== 54+0+16 (secure 0 0 0) 0x7f0fb40824f0 con 0x7f0fb80f6a90 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0faeffd700 10 client.37733.objecter ms_handle_connect 0x7f0fa4062c90 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fbd66e700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0fa4062c90 0x7f0fa4065150 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 rx=0x7f0fb80f8e20 tx=0x7f0fa800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 1 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 20 client.37733.objecter _get_session osd=-1 returning homeless 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 10 client.37733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=435}) v3 -- 0x7f0fb805ee80 con 0x7f0fb80f6a90 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0fb81070d0 con 0x7f0fb80f6a90 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=434) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fb4050290 con 0x7f0fb80f6a90 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0faffff700 10 client.37733.objecter _check_command_map_dne tid 1 current 434 map_dne_bound 434 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0faffff700 10 client.37733.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0faffff700 15 client.37733.objecter _session_command_op_remove -1 1 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 2 2024-02-23T00:36:51.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6805/685354171 conn(0x7f0fb80f60b0 0x7f0fb80273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb80f5f40 osd=1 3 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb80f5f40 osd=1 3 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb80f5f40 osd=1 3 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 2 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb80f5f40 osd=1 3 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 1 2 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb80f5f40 osd=1 4 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 10 client.37733.objecter _send_command 2 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fb8059e00 con 0x7f0fb80f60b0 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fbe670700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6805/685354171 conn(0x7f0fb80f60b0 0x7f0fb80273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0faeffd700 10 client.37733.objecter ms_handle_connect 0x7f0fb80f60b0 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.460+0000 7f0fbe670700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6805/685354171 conn(0x7f0fb80f60b0 0x7f0fb80273c0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.461+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0fb8059e00 con 0x7f0fb80f60b0 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.461+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:36:51.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.461+0000 7f0faeffd700 10 client.37733.objecter _finish_command 2 = system:0 2024-02-23T00:36:51.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.461+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb80f5f40 osd=1 4 2024-02-23T00:36:51.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.461+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 1 2 2024-02-23T00:36:51.474 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/osd.0/config 2024-02-23T00:36:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:51.479 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 3 2024-02-23T00:36:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb80f5f40 osd=1 3 2024-02-23T00:36:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb80f5f40 osd=1 3 2024-02-23T00:36:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb80f5f40 osd=1 3 2024-02-23T00:36:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 3 2024-02-23T00:36:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb80f5f40 osd=1 3 2024-02-23T00:36:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 1 3 2024-02-23T00:36:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb80f5f40 osd=1 4 2024-02-23T00:36:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 10 client.37733.objecter _send_command 3 2024-02-23T00:36:51.480 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.477+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0fb80278b0 con 0x7f0fb80f60b0 2024-02-23T00:36:51.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0fb80278b0 con 0x7f0fb80f60b0 2024-02-23T00:36:51.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:51.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0faeffd700 10 client.37733.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb80f5f40 osd=1 4 2024-02-23T00:36:51.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 1 3 2024-02-23T00:36:51.482 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:36:51.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:51.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 4 2024-02-23T00:36:51.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0fb80594d0 0x7f0fb811b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8059360 osd=2 3 2024-02-23T00:36:51.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8059360 osd=2 3 2024-02-23T00:36:51.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8059360 osd=2 3 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 4 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb8059360 osd=2 3 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 2 4 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8059360 osd=2 4 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 10 client.37733.objecter _send_command 4 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fb8028700 con 0x7f0fb80594d0 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.480+0000 7f0fbde6f700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0fb80594d0 0x7f0fb811b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.481+0000 7f0faeffd700 10 client.37733.objecter ms_handle_connect 0x7f0fb80594d0 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.481+0000 7f0fbde6f700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6809/3605394228 conn(0x7f0fb80594d0 0x7f0fb811b070 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.481+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0fb8028700 con 0x7f0fb80594d0 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.481+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:36:51.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.481+0000 7f0faeffd700 10 client.37733.objecter _finish_command 4 = system:0 2024-02-23T00:36:51.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.481+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb8059360 osd=2 4 2024-02-23T00:36:51.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.481+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 2 4 2024-02-23T00:36:51.485 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:51.505 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:36:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 5 2024-02-23T00:36:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8059360 osd=2 3 2024-02-23T00:36:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8059360 osd=2 3 2024-02-23T00:36:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8059360 osd=2 3 2024-02-23T00:36:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 5 2024-02-23T00:36:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb8059360 osd=2 3 2024-02-23T00:36:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 2 5 2024-02-23T00:36:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8059360 osd=2 4 2024-02-23T00:36:51.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 10 client.37733.objecter _send_command 5 2024-02-23T00:36:51.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.506+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0fb8027c60 con 0x7f0fb80594d0 2024-02-23T00:36:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:51.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:36:51.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.507+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0fb8027c60 con 0x7f0fb80594d0 2024-02-23T00:36:51.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.507+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:51.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.507+0000 7f0faeffd700 10 client.37733.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.507+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb8059360 osd=2 4 2024-02-23T00:36:51.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.507+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 2 5 2024-02-23T00:36:51.511 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 6 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6813/128059790 conn(0x7f0fb811c6e0 0x7f0fb811ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb80ff1a0 osd=3 3 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb80ff1a0 osd=3 3 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb80ff1a0 osd=3 3 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 6 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb80ff1a0 osd=3 3 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 3 6 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb80ff1a0 osd=3 4 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 10 client.37733.objecter _send_command 6 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fb80ff9a0 con 0x7f0fb811c6e0 2024-02-23T00:36:51.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fbd66e700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6813/128059790 conn(0x7f0fb811c6e0 0x7f0fb811ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0faeffd700 10 client.37733.objecter ms_handle_connect 0x7f0fb811c6e0 2024-02-23T00:36:51.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.508+0000 7f0fbd66e700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.87:6813/128059790 conn(0x7f0fb811c6e0 0x7f0fb811ead0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:51.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.509+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0fb80ff9a0 con 0x7f0fb811c6e0 2024-02-23T00:36:51.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.509+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:36:51.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.509+0000 7f0faeffd700 10 client.37733.objecter _finish_command 6 = system:0 2024-02-23T00:36:51.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.509+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb80ff1a0 osd=3 4 2024-02-23T00:36:51.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.509+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 3 6 2024-02-23T00:36:51.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:51.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 7 2024-02-23T00:36:51.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb80ff1a0 osd=3 3 2024-02-23T00:36:51.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb80ff1a0 osd=3 3 2024-02-23T00:36:51.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb80ff1a0 osd=3 3 2024-02-23T00:36:51.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 7 2024-02-23T00:36:51.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb80ff1a0 osd=3 3 2024-02-23T00:36:51.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 3 7 2024-02-23T00:36:51.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb80ff1a0 osd=3 4 2024-02-23T00:36:51.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 10 client.37733.objecter _send_command 7 2024-02-23T00:36:51.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.523+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0fb80fecd0 con 0x7f0fb811c6e0 2024-02-23T00:36:51.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0fb80fecd0 con 0x7f0fb811c6e0 2024-02-23T00:36:51.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:51.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0faeffd700 10 client.37733.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb80ff1a0 osd=3 4 2024-02-23T00:36:51.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 3 7 2024-02-23T00:36:51.528 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 8 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0fb8120140 0x7f0fb8120580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb811bd30 osd=4 3 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb811bd30 osd=4 3 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb811bd30 osd=4 3 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 8 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb811bd30 osd=4 3 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 4 8 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb811bd30 osd=4 4 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 10 client.37733.objecter _send_command 8 2024-02-23T00:36:51.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fb811c550 con 0x7f0fb8120140 2024-02-23T00:36:51.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.525+0000 7f0fbe670700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0fb8120140 0x7f0fb8120580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.526+0000 7f0faeffd700 10 client.37733.objecter ms_handle_connect 0x7f0fb8120140 2024-02-23T00:36:51.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.526+0000 7f0fbe670700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6800/4175844757 conn(0x7f0fb8120140 0x7f0fb8120580 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:51.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.527+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0fb811c550 con 0x7f0fb8120140 2024-02-23T00:36:51.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.527+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:36:51.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.527+0000 7f0faeffd700 10 client.37733.objecter _finish_command 8 = system:0 2024-02-23T00:36:51.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.527+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb811bd30 osd=4 4 2024-02-23T00:36:51.530 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.527+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 4 8 2024-02-23T00:36:51.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:51.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 9 2024-02-23T00:36:51.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb811bd30 osd=4 3 2024-02-23T00:36:51.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb811bd30 osd=4 3 2024-02-23T00:36:51.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb811bd30 osd=4 3 2024-02-23T00:36:51.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 9 2024-02-23T00:36:51.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb811bd30 osd=4 3 2024-02-23T00:36:51.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 4 9 2024-02-23T00:36:51.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb811bd30 osd=4 4 2024-02-23T00:36:51.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 10 client.37733.objecter _send_command 9 2024-02-23T00:36:51.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.541+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0fb81212f0 con 0x7f0fb8120140 2024-02-23T00:36:51.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0fb81212f0 con 0x7f0fb8120140 2024-02-23T00:36:51.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:51.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0faeffd700 10 client.37733.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb811bd30 osd=4 4 2024-02-23T00:36:51.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 4 9 2024-02-23T00:36:51.546 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:36:51.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:51.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 10 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0fb81237b0 0x7f0fb8125ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8120b20 osd=5 3 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8120b20 osd=5 3 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8120b20 osd=5 3 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 10 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb8120b20 osd=5 3 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 5 10 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8120b20 osd=5 4 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 10 client.37733.objecter _send_command 10 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fb81287f0 con 0x7f0fb81237b0 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.543+0000 7f0fbde6f700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0fb81237b0 0x7f0fb8125ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.544+0000 7f0faeffd700 10 client.37733.objecter ms_handle_connect 0x7f0fb81237b0 2024-02-23T00:36:51.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.544+0000 7f0fbde6f700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6804/2550013795 conn(0x7f0fb81237b0 0x7f0fb8125ba0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:51.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.544+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0fb81287f0 con 0x7f0fb81237b0 2024-02-23T00:36:51.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.544+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:36:51.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.544+0000 7f0faeffd700 10 client.37733.objecter _finish_command 10 = system:0 2024-02-23T00:36:51.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.544+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb8120b20 osd=5 4 2024-02-23T00:36:51.548 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.544+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 5 10 2024-02-23T00:36:51.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:51.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 11 2024-02-23T00:36:51.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8120b20 osd=5 3 2024-02-23T00:36:51.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8120b20 osd=5 3 2024-02-23T00:36:51.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8120b20 osd=5 3 2024-02-23T00:36:51.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 11 2024-02-23T00:36:51.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb8120b20 osd=5 3 2024-02-23T00:36:51.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 5 11 2024-02-23T00:36:51.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8120b20 osd=5 4 2024-02-23T00:36:51.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 10 client.37733.objecter _send_command 11 2024-02-23T00:36:51.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.559+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0fb8126090 con 0x7f0fb81237b0 2024-02-23T00:36:51.562 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:36:51.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0fb8126090 con 0x7f0fb81237b0 2024-02-23T00:36:51.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:51.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0faeffd700 10 client.37733.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb8120b20 osd=5 4 2024-02-23T00:36:51.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 5 11 2024-02-23T00:36:51.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:51.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 12 2024-02-23T00:36:51.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0fb8126770 0x7f0fb8128b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8126090 osd=6 3 2024-02-23T00:36:51.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8126090 osd=6 3 2024-02-23T00:36:51.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8126090 osd=6 3 2024-02-23T00:36:51.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 12 2024-02-23T00:36:51.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb8126090 osd=6 3 2024-02-23T00:36:51.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 6 12 2024-02-23T00:36:51.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8126090 osd=6 4 2024-02-23T00:36:51.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 10 client.37733.objecter _send_command 12 2024-02-23T00:36:51.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.560+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fb812b910 con 0x7f0fb8126770 2024-02-23T00:36:51.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.561+0000 7f0fbd66e700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0fb8126770 0x7f0fb8128b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.561+0000 7f0faeffd700 10 client.37733.objecter ms_handle_connect 0x7f0fb8126770 2024-02-23T00:36:51.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.561+0000 7f0fbd66e700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0fb8126770 0x7f0fb8128b60 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:51.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.562+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0fb812b910 con 0x7f0fb8126770 2024-02-23T00:36:51.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.562+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:36:51.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.562+0000 7f0faeffd700 10 client.37733.objecter _finish_command 12 = system:0 2024-02-23T00:36:51.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.562+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb8126090 osd=6 4 2024-02-23T00:36:51.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.562+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 6 12 2024-02-23T00:36:51.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:51.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 13 2024-02-23T00:36:51.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8126090 osd=6 3 2024-02-23T00:36:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8126090 osd=6 3 2024-02-23T00:36:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb8126090 osd=6 3 2024-02-23T00:36:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 13 2024-02-23T00:36:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb8126090 osd=6 3 2024-02-23T00:36:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 6 13 2024-02-23T00:36:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb8126090 osd=6 4 2024-02-23T00:36:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 10 client.37733.objecter _send_command 13 2024-02-23T00:36:51.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.576+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0fb812b910 con 0x7f0fb8126770 2024-02-23T00:36:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0fb812b910 con 0x7f0fb8126770 2024-02-23T00:36:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0faeffd700 10 client.37733.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb8126090 osd=6 4 2024-02-23T00:36:51.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 6 13 2024-02-23T00:36:51.581 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:36:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:36:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 14 2024-02-23T00:36:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6812/373787000 conn(0x7f0fb812ac50 0x7f0fb812b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:36:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb812b910 osd=7 3 2024-02-23T00:36:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb812b910 osd=7 3 2024-02-23T00:36:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb812b910 osd=7 3 2024-02-23T00:36:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 14 2024-02-23T00:36:51.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb812b910 osd=7 3 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 7 14 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb812b910 osd=7 4 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 10 client.37733.objecter _send_command 14 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fb812ed40 con 0x7f0fb812ac50 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.578+0000 7f0fbe670700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6812/373787000 conn(0x7f0fb812ac50 0x7f0fb812b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.579+0000 7f0faeffd700 10 client.37733.objecter ms_handle_connect 0x7f0fb812ac50 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.579+0000 7f0fbe670700 1 --2- 172.21.15.87:0/2287558011 >> v2:172.21.15.171:6812/373787000 conn(0x7f0fb812ac50 0x7f0fb812b0b0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.580+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0fb812ed40 con 0x7f0fb812ac50 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.580+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.580+0000 7f0faeffd700 10 client.37733.objecter _finish_command 14 = system:0 2024-02-23T00:36:51.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.580+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb812b910 osd=7 4 2024-02-23T00:36:51.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.580+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 7 14 2024-02-23T00:36:51.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:49.663401+0000 mgr.y (mgr.24437) 402 : cluster [DBG] pgmap v646: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 5 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:36:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:49.663401+0000 mgr.y (mgr.24437) 402 : cluster [DBG] pgmap v646: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 5 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:36:51.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 10 client.37733.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:36:51.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign -1 15 2024-02-23T00:36:51.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb812b910 osd=7 3 2024-02-23T00:36:51.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb812b910 osd=7 3 2024-02-23T00:36:51.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 20 client.37733.objecter _get_session s=0x7f0fb812b910 osd=7 3 2024-02-23T00:36:51.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_remove -1 15 2024-02-23T00:36:51.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 20 client.37733.objecter get_session s=0x7f0fb812b910 osd=7 3 2024-02-23T00:36:51.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 15 client.37733.objecter _session_command_op_assign 7 15 2024-02-23T00:36:51.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 20 client.37733.objecter put_session s=0x7f0fb812b910 osd=7 4 2024-02-23T00:36:51.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 10 client.37733.objecter _send_command 15 2024-02-23T00:36:51.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.594+0000 7f0fc00d3700 1 -- 172.21.15.87:0/2287558011 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0fb81290f0 con 0x7f0fb812ac50 2024-02-23T00:36:51.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.596+0000 7f0faeffd700 1 -- 172.21.15.87:0/2287558011 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0fb81290f0 con 0x7f0fb812ac50 2024-02-23T00:36:51.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.596+0000 7f0faeffd700 10 client.37733.objecter ms_dispatch 0x7f0fb8001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:36:51.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.596+0000 7f0faeffd700 10 client.37733.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.596+0000 7f0faeffd700 20 client.37733.objecter put_session s=0x7f0fb812b910 osd=7 4 2024-02-23T00:36:51.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:36:51.596+0000 7f0faeffd700 15 client.37733.objecter _session_command_op_remove 7 15 2024-02-23T00:36:51.599 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:36:51.600 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:36:51.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.600 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:36:51.611 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:51.639 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:49.663401+0000 mgr.y (mgr.24437) 402 : cluster [DBG] pgmap v646: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.7 KiB/s wr, 5 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:36:52.563 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:50.935192+0000 mgr.y (mgr.24437) 403 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:52.563 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:51.460991+0000 mon.c (mon.2) 436 : audit [DBG] from='client.? 172.21.15.87:0/2287558011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:52.563 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:52.091387+0000 mon.a (mon.0) 2619 : cluster [WRN] Health check update: Degraded data redundancy: 120/1173 objects degraded (10.230%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:36:52.563 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:52.097620+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e435: 8 total, 7 up, 7 in 2024-02-23T00:36:52.563 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:50.935192+0000 mgr.y (mgr.24437) 403 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:52.564 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:51.460991+0000 mon.c (mon.2) 436 : audit [DBG] from='client.? 172.21.15.87:0/2287558011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:52.564 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:52.091387+0000 mon.a (mon.0) 2619 : cluster [WRN] Health check update: Degraded data redundancy: 120/1173 objects degraded (10.230%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:36:52.564 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:52.097620+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e435: 8 total, 7 up, 7 in 2024-02-23T00:36:52.564 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:36:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:52] "GET /metrics HTTP/1.1" 200 185739 "" "Prometheus/2.33.4" 2024-02-23T00:36:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:36:52.580 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:36:52.588 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:52.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:52.692 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:52.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:50.935192+0000 mgr.y (mgr.24437) 403 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:36:52.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:51.460991+0000 mon.c (mon.2) 436 : audit [DBG] from='client.? 172.21.15.87:0/2287558011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:36:52.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:52.091387+0000 mon.a (mon.0) 2619 : cluster [WRN] Health check update: Degraded data redundancy: 120/1173 objects degraded (10.230%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:36:52.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:52.097620+0000 mon.a (mon.0) 2620 : cluster [DBG] osdmap e435: 8 total, 7 up, 7 in 2024-02-23T00:36:52.721 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:52.741 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:52.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:52.844 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:52.871 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:52.930 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:36:53.069 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:53.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:53.087 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:53.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:53.202 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:53.452 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:53.453 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:51.664356+0000 mgr.y (mgr.24437) 404 : cluster [DBG] pgmap v647: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s; 120/1167 objects degraded (10.283%) 2024-02-23T00:36:53.454 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:51.664356+0000 mgr.y (mgr.24437) 404 : cluster [DBG] pgmap v647: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s; 120/1167 objects degraded (10.283%) 2024-02-23T00:36:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:53.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:53.556 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:53.584 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:51.664356+0000 mgr.y (mgr.24437) 404 : cluster [DBG] pgmap v647: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s; 120/1167 objects degraded (10.283%) 2024-02-23T00:36:54.032 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:54.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:54.136 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:54.163 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:54.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:36:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:36:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:36:54.581 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:36:54.878 INFO:teuthology.orchestra.run.smithi087.stderr:Exporting 212.4 info 212.4( empty local-lis/les=407/408 n=0 ec=407/407 lis/c=407/407 les/c/f=408/408/0 sis=407) 2024-02-23T00:36:54.879 INFO:teuthology.orchestra.run.smithi087.stderr:Export successful 2024-02-23T00:36:55.216 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:55.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:55.326 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:36:55.355 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:53.664972+0000 mgr.y (mgr.24437) 405 : cluster [DBG] pgmap v649: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 120/1167 objects degraded (10.283%) 2024-02-23T00:36:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:54.651545+0000 mon.a (mon.0) 2621 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:55.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:53.664972+0000 mgr.y (mgr.24437) 405 : cluster [DBG] pgmap v649: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 120/1167 objects degraded (10.283%) 2024-02-23T00:36:55.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:54.651545+0000 mon.a (mon.0) 2621 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:53.664972+0000 mgr.y (mgr.24437) 405 : cluster [DBG] pgmap v649: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 120/1167 objects degraded (10.283%) 2024-02-23T00:36:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:54.651545+0000 mon.a (mon.0) 2621 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:36:55.964 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:56.507 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:36:56.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:55.317482+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:56.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:55.317596+0000 mon.a (mon.0) 2622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:56.516 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:55.318310+0000 mon.b (mon.1) 361 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-63"}]: dispatch 2024-02-23T00:36:56.517 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:55.318386+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-63"}]: dispatch 2024-02-23T00:36:56.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:55.317482+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:56.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:55.317596+0000 mon.a (mon.0) 2622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:56.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:55.318310+0000 mon.b (mon.1) 361 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-63"}]: dispatch 2024-02-23T00:36:56.517 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:55.318386+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-63"}]: dispatch 2024-02-23T00:36:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:56.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:36:56.552 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:36:56.552 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:36:56.619 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:36:56.647 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:55.317482+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:56.647 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:55.317596+0000 mon.a (mon.0) 2622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:36:56.647 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:55.318310+0000 mon.b (mon.1) 361 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-63"}]: dispatch 2024-02-23T00:36:56.647 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:55.318386+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-63"}]: dispatch 2024-02-23T00:36:56.649 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:56.667 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:36:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:56.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:36:56.771 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:36:56.801 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:57.339 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -n osd.0 --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-objectstore-tool --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 212.4 2024-02-23T00:36:57.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:55.665922+0000 mgr.y (mgr.24437) 406 : cluster [DBG] pgmap v650: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 118/1155 objects degraded (10.216%) 2024-02-23T00:36:57.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:56.322525+0000 mon.a (mon.0) 2624 : cluster [DBG] osdmap e436: 8 total, 7 up, 7 in 2024-02-23T00:36:57.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:57.093167+0000 mon.a (mon.0) 2625 : cluster [WRN] Health check update: Degraded data redundancy: 118/1155 objects degraded (10.216%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:36:57.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:57.099667+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e437: 8 total, 7 up, 7 in 2024-02-23T00:36:57.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:57.120923+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:57.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:57.140659+0000 mon.a (mon.0) 2627 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:57.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:55.665922+0000 mgr.y (mgr.24437) 406 : cluster [DBG] pgmap v650: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 118/1155 objects degraded (10.216%) 2024-02-23T00:36:57.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:56.322525+0000 mon.a (mon.0) 2624 : cluster [DBG] osdmap e436: 8 total, 7 up, 7 in 2024-02-23T00:36:57.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:57.093167+0000 mon.a (mon.0) 2625 : cluster [WRN] Health check update: Degraded data redundancy: 118/1155 objects degraded (10.216%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:36:57.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:57.099667+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e437: 8 total, 7 up, 7 in 2024-02-23T00:36:57.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:57.120923+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:57.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:57.140659+0000 mon.a (mon.0) 2627 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:55.665922+0000 mgr.y (mgr.24437) 406 : cluster [DBG] pgmap v650: 238 pgs: 28 active+undersized+degraded, 64 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 118/1155 objects degraded (10.216%) 2024-02-23T00:36:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:56.322525+0000 mon.a (mon.0) 2624 : cluster [DBG] osdmap e436: 8 total, 7 up, 7 in 2024-02-23T00:36:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:57.093167+0000 mon.a (mon.0) 2625 : cluster [WRN] Health check update: Degraded data redundancy: 118/1155 objects degraded (10.216%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:36:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:57.099667+0000 mon.a (mon.0) 2626 : cluster [DBG] osdmap e437: 8 total, 7 up, 7 in 2024-02-23T00:36:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:57.120923+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:57.140659+0000 mon.a (mon.0) 2627 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:36:58.052 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:36:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:58.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:36:58.073 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-23T00:36:58.117 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/osd.0/config 2024-02-23T00:36:58.158 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:58.218 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:36:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:57.666441+0000 mgr.y (mgr.24437) 407 : cluster [DBG] pgmap v653: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1155 objects degraded (10.216%) 2024-02-23T00:36:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:58.121526+0000 mon.a (mon.0) 2628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:36:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:58.121606+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e438: 8 total, 7 up, 7 in 2024-02-23T00:36:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:36:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:58.140965+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:36:59.506 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:36:59.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:57.666441+0000 mgr.y (mgr.24437) 407 : cluster [DBG] pgmap v653: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1155 objects degraded (10.216%) 2024-02-23T00:36:59.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:58.121526+0000 mon.a (mon.0) 2628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:36:59.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:58.121606+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e438: 8 total, 7 up, 7 in 2024-02-23T00:36:59.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:36:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:58.140965+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:36:59.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:57.666441+0000 mgr.y (mgr.24437) 407 : cluster [DBG] pgmap v653: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1155 objects degraded (10.216%) 2024-02-23T00:36:59.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:58.121526+0000 mon.a (mon.0) 2628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:36:59.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:58.121606+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e438: 8 total, 7 up, 7 in 2024-02-23T00:36:59.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:36:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:58.140965+0000 mon.b (mon.1) 363 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:36:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:36:59.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:36:59.513 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:36:59.611 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:36:59.639 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:00.287 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:00.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:59.124048+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e439: 8 total, 7 up, 7 in 2024-02-23T00:37:00.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:59.236034+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:00.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:59.236316+0000 mon.a (mon.0) 2631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:00.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:59.237087+0000 mon.b (mon.1) 365 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-65"}]: dispatch 2024-02-23T00:37:00.289 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:36:59.237214+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-65"}]: dispatch 2024-02-23T00:37:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:00.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:00.291 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:59.124048+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e439: 8 total, 7 up, 7 in 2024-02-23T00:37:00.291 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:59.236034+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:00.291 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:59.236316+0000 mon.a (mon.0) 2631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:00.291 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:59.237087+0000 mon.b (mon.1) 365 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-65"}]: dispatch 2024-02-23T00:37:00.291 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:36:59.237214+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-65"}]: dispatch 2024-02-23T00:37:00.391 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:00.416 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:59.124048+0000 mon.a (mon.0) 2630 : cluster [DBG] osdmap e439: 8 total, 7 up, 7 in 2024-02-23T00:37:00.416 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:59.236034+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:00.417 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:59.236316+0000 mon.a (mon.0) 2631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:00.417 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:59.237087+0000 mon.b (mon.1) 365 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-65"}]: dispatch 2024-02-23T00:37:00.417 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:36:59.237214+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-65"}]: dispatch 2024-02-23T00:37:00.418 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:00.441 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:00.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:00.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:00.544 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:00.571 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:00.590 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:00.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:00.694 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:00.720 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:00.739 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:00.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:00.842 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:00.870 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:00.889 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:00.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:00.993 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:01.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:37:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:37:01.021 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:01.039 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:01.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:01.143 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:01.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:36:59.667102+0000 mgr.y (mgr.24437) 408 : cluster [DBG] pgmap v656: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1155 objects degraded (10.216%) 2024-02-23T00:37:01.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:00.129443+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e440: 8 total, 7 up, 7 in 2024-02-23T00:37:01.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:36:59.667102+0000 mgr.y (mgr.24437) 408 : cluster [DBG] pgmap v656: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1155 objects degraded (10.216%) 2024-02-23T00:37:01.198 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:00.129443+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e440: 8 total, 7 up, 7 in 2024-02-23T00:37:01.208 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:01.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:36:59.667102+0000 mgr.y (mgr.24437) 408 : cluster [DBG] pgmap v656: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 118/1155 objects degraded (10.216%) 2024-02-23T00:37:01.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:00.129443+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e440: 8 total, 7 up, 7 in 2024-02-23T00:37:02.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:02] "GET /metrics HTTP/1.1" 200 188382 "" "Prometheus/2.33.4" 2024-02-23T00:37:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:00.940326+0000 mgr.y (mgr.24437) 409 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:01.135738+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e441: 8 total, 7 up, 7 in 2024-02-23T00:37:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:01.142888+0000 mon.b (mon.1) 366 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:01.143373+0000 mon.a (mon.0) 2635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:02.095790+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check update: Degraded data redundancy: 118/1155 objects degraded (10.216%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:00.940326+0000 mgr.y (mgr.24437) 409 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:01.135738+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e441: 8 total, 7 up, 7 in 2024-02-23T00:37:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:01.142888+0000 mon.b (mon.1) 366 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:01.143373+0000 mon.a (mon.0) 2635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:02.095790+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check update: Degraded data redundancy: 118/1155 objects degraded (10.216%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:00.940326+0000 mgr.y (mgr.24437) 409 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:01.135738+0000 mon.a (mon.0) 2634 : cluster [DBG] osdmap e441: 8 total, 7 up, 7 in 2024-02-23T00:37:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:01.142888+0000 mon.b (mon.1) 366 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:01.143373+0000 mon.a (mon.0) 2635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:02.095790+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check update: Degraded data redundancy: 118/1155 objects degraded (10.216%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:02.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.860+0000 7f4e32378700 1 -- 172.21.15.87:0/2724059289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e2c0f8420 msgr2=0x7f4e2c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:02.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.860+0000 7f4e32378700 1 --2- 172.21.15.87:0/2724059289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e2c0f8420 0x7f4e2c104a20 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 rx=0x7f4e24009be0 tx=0x7f4e24004d10).stop 2024-02-23T00:37:02.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.860+0000 7f4e32378700 1 -- 172.21.15.87:0/2724059289 shutdown_connections 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.860+0000 7f4e32378700 1 --2- 172.21.15.87:0/2724059289 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e2c0f8420 0x7f4e2c104a20 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.860+0000 7f4e32378700 1 --2- 172.21.15.87:0/2724059289 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e2c0fdbb0 0x7f4e2c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.860+0000 7f4e32378700 1 --2- 172.21.15.87:0/2724059289 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e2c0fe510 0x7f4e2c0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.860+0000 7f4e32378700 1 -- 172.21.15.87:0/2724059289 >> 172.21.15.87:0/2724059289 conn(0x7f4e2c0f3800 msgr2=0x7f4e2c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.860+0000 7f4e32378700 1 -- 172.21.15.87:0/2724059289 shutdown_connections 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.860+0000 7f4e32378700 1 -- 172.21.15.87:0/2724059289 wait complete. 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 librados: starting msgr at 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 librados: starting objecter 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 Processor -- start 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 -- start start 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 librados: setting wanted keys 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 librados: calling monclient init 2024-02-23T00:37:02.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e2c0f8420 0x7f4e2c10df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e2c0fdbb0 0x7f4e2c10e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e2c0fe510 0x7f4e2c1080e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c1085d0 con 0x7f4e2c0fdbb0 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c108740 con 0x7f4e2c0fe510 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.861+0000 7f4e32378700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c108900 con 0x7f4e2c0f8420 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e2c0f8420 0x7f4e2c10df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e2c0fdbb0 0x7f4e2c10e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e2c0f8420 0x7f4e2c10df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:46336/0 (socket says 172.21.15.87:46336) 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2bfff700 1 -- 172.21.15.87:0/401933598 learned_addr learned my addr 172.21.15.87:0/401933598 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e30915700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e2c0fe510 0x7f4e2c1080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2bfff700 1 -- 172.21.15.87:0/401933598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e2c0fe510 msgr2=0x7f4e2c1080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2bfff700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e2c0fe510 0x7f4e2c1080e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:02.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2bfff700 1 -- 172.21.15.87:0/401933598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e2c0fdbb0 msgr2=0x7f4e2c10e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2bfff700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e2c0fdbb0 0x7f4e2c10e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2bfff700 1 -- 172.21.15.87:0/401933598 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e240097c0 con 0x7f4e2c0f8420 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e30915700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e2c0fe510 0x7f4e2c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2b7fe700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e2c0fdbb0 0x7f4e2c10e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e2bfff700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e2c0f8420 0x7f4e2c10df40 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f4e1c00e970 tx=0x7f4e1c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e297fa700 10 client.?.objecter ms_handle_connect 0x7f4e2c0f8420 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e297fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e297fa700 1 -- 172.21.15.87:0/401933598 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c00ee60 con 0x7f4e2c0f8420 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e297fa700 1 -- 172.21.15.87:0/401933598 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e1c00f040 con 0x7f4e2c0f8420 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c108b80 con 0x7f4e2c0f8420 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e297fa700 1 -- 172.21.15.87:0/401933598 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c007ea0 con 0x7f4e2c0f8420 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e32378700 10 client.37739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:02.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c06af90 con 0x7f4e2c0f8420 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.862+0000 7f4e32378700 1 librados: init done 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.863+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c107110 con 0x7f4e2c0f8420 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.863+0000 7f4e297fa700 1 -- 172.21.15.87:0/401933598 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4e1c00cb10 con 0x7f4e2c0f8420 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.864+0000 7f4e297fa700 1 --2- 172.21.15.87:0/401933598 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e14062c90 0x7f4e14065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.864+0000 7f4e297fa700 1 -- 172.21.15.87:0/401933598 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(442..442 src has 173..442) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f4e1c07ce20 con 0x7f4e2c0f8420 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.864+0000 7f4e2b7fe700 1 --2- 172.21.15.87:0/401933598 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e14062c90 0x7f4e14065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.864+0000 7f4e297fa700 10 client.37739.objecter ms_dispatch 0x7f4e2c001550 osd_map(442..442 src has 173..442) v4 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.864+0000 7f4e297fa700 3 client.37739.objecter handle_osd_map got epochs [442,442] > 0 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.864+0000 7f4e297fa700 3 client.37739.objecter handle_osd_map decoding full epoch 442 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.864+0000 7f4e297fa700 20 client.37739.objecter dump_active .. 0 homeless 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.864+0000 7f4e297fa700 10 client.37739.objecter ms_handle_connect 0x7f4e14062c90 2024-02-23T00:37:02.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.864+0000 7f4e2b7fe700 1 --2- 172.21.15.87:0/401933598 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e14062c90 0x7f4e14065150 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7f4e2c109880 tx=0x7f4e2000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:02.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:02.868+0000 7f4e297fa700 1 -- 172.21.15.87:0/401933598 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4e1c04af60 con 0x7f4e2c0f8420 2024-02-23T00:37:02.873 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:02.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:02.976 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:03.004 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:03.024 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:03.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:03.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.042+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f4e2c05fb30 con 0x7f4e2c0f8420 2024-02-23T00:37:03.127 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:03.157 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:03.176 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:03.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:03.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.195+0000 7f4e297fa700 1 -- 172.21.15.87:0/401933598 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v443) v1 ==== 91+0+0 (secure 0 0 0) 0x7f4e1c019020 con 0x7f4e2c0f8420 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.198+0000 7f4e32378700 10 librados: watch_flush enter 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.203+0000 7f4e32378700 10 librados: watch_flush exit 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.203+0000 7f4e32378700 20 client.37739.objecter shutdown clearing up homeless session... 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.203+0000 7f4e32378700 10 client.37739.objecter successfully canceled tick 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.203+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e14062c90 msgr2=0x7f4e14065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.203+0000 7f4e32378700 1 --2- 172.21.15.87:0/401933598 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e14062c90 0x7f4e14065150 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 rx=0x7f4e2c109880 tx=0x7f4e2000b040).stop 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.203+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e2c0f8420 msgr2=0x7f4e2c10df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.203+0000 7f4e32378700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e2c0f8420 0x7f4e2c10df40 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f4e1c00e970 tx=0x7f4e1c00b040).stop 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.204+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 shutdown_connections 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.204+0000 7f4e32378700 1 --2- 172.21.15.87:0/401933598 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e14062c90 0x7f4e14065150 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:03.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.204+0000 7f4e32378700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e2c0fe510 0x7f4e2c1080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:03.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.204+0000 7f4e32378700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e2c0f8420 0x7f4e2c10df40 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:03.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.204+0000 7f4e32378700 1 --2- 172.21.15.87:0/401933598 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e2c0fdbb0 0x7f4e2c10e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:03.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.204+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 >> 172.21.15.87:0/401933598 conn(0x7f4e2c0f3800 msgr2=0x7f4e2c0f5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:03.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.205+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 shutdown_connections 2024-02-23T00:37:03.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.205+0000 7f4e32378700 1 -- 172.21.15.87:0/401933598 wait complete. 2024-02-23T00:37:03.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:03.205+0000 7f4e32378700 1 librados: shutdown 2024-02-23T00:37:03.212 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:37:03.281 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:03.308 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:03.327 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:03.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:03.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:03.431 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:03.458 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:03.476 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:03.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:03.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:03.580 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:03.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:01.667695+0000 mgr.y (mgr.24437) 410 : cluster [DBG] pgmap v659: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 120/1161 objects degraded (10.336%) 2024-02-23T00:37:03.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:02.187197+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:02.187302+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e442: 8 total, 7 up, 7 in 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:02.228765+0000 mon.b (mon.1) 367 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:02.276961+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:02.277197+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:02.277920+0000 mon.b (mon.1) 369 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-67"}]: dispatch 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:02.278025+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-67"}]: dispatch 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:03.044365+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.87:0/401933598' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:03.044682+0000 mon.a (mon.0) 2641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:03.184782+0000 mon.a (mon.0) 2642 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:03.190621+0000 mon.a (mon.0) 2643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:37:03.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:03.190689+0000 mon.a (mon.0) 2644 : cluster [DBG] osdmap e443: 8 total, 7 up, 7 in 2024-02-23T00:37:03.606 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:03.624 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:03.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:03.728 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:03.754 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:03.775 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:03.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:01.667695+0000 mgr.y (mgr.24437) 410 : cluster [DBG] pgmap v659: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 120/1161 objects degraded (10.336%) 2024-02-23T00:37:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:02.187197+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:02.187302+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e442: 8 total, 7 up, 7 in 2024-02-23T00:37:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:02.228765+0000 mon.b (mon.1) 367 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:02.276961+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:02.277197+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:02.277920+0000 mon.b (mon.1) 369 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-67"}]: dispatch 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:02.278025+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-67"}]: dispatch 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:03.044365+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.87:0/401933598' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:03.044682+0000 mon.a (mon.0) 2641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:03.184782+0000 mon.a (mon.0) 2642 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:03.190621+0000 mon.a (mon.0) 2643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:03.190689+0000 mon.a (mon.0) 2644 : cluster [DBG] osdmap e443: 8 total, 7 up, 7 in 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:01.667695+0000 mgr.y (mgr.24437) 410 : cluster [DBG] pgmap v659: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 120/1161 objects degraded (10.336%) 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:02.187197+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:02.187302+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e442: 8 total, 7 up, 7 in 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:02.228765+0000 mon.b (mon.1) 367 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:02.276961+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:03.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:02.277197+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:03.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:02.277920+0000 mon.b (mon.1) 369 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-67"}]: dispatch 2024-02-23T00:37:03.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:02.278025+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-67"}]: dispatch 2024-02-23T00:37:03.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:03.044365+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.87:0/401933598' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:03.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:03.044682+0000 mon.a (mon.0) 2641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:03.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:03.184782+0000 mon.a (mon.0) 2642 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:03.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:03.190621+0000 mon.a (mon.0) 2643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:37:03.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:03.190689+0000 mon.a (mon.0) 2644 : cluster [DBG] osdmap e443: 8 total, 7 up, 7 in 2024-02-23T00:37:03.884 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:03.909 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:37:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:37:03.911 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:03.937 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:03.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:03.951 INFO:teuthology.orchestra.run.smithi087.stdout: marking collection for removal 2024-02-23T00:37:03.951 INFO:teuthology.orchestra.run.smithi087.stdout:setting '_remove' omap key 2024-02-23T00:37:03.951 INFO:teuthology.orchestra.run.smithi087.stdout:finish_remove_pgs 212.4_head removing 212.4 2024-02-23T00:37:03.951 INFO:teuthology.orchestra.run.smithi087.stdout:Remove successful 2024-02-23T00:37:04.042 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:04.072 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.090+0000 7f600e706700 1 -- 172.21.15.87:0/3335846085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60080f8420 msgr2=0x7f6008104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.090+0000 7f600e706700 1 --2- 172.21.15.87:0/3335846085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60080f8420 0x7f6008104a20 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 rx=0x7f6000009be0 tx=0x7f6000004d10).stop 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.090+0000 7f600e706700 1 -- 172.21.15.87:0/3335846085 shutdown_connections 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.090+0000 7f600e706700 1 --2- 172.21.15.87:0/3335846085 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60080f8420 0x7f6008104a20 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.090+0000 7f600e706700 1 --2- 172.21.15.87:0/3335846085 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60080fdbb0 0x7f60080fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.090+0000 7f600e706700 1 --2- 172.21.15.87:0/3335846085 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60080fe510 0x7f60080f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.090+0000 7f600e706700 1 -- 172.21.15.87:0/3335846085 >> 172.21.15.87:0/3335846085 conn(0x7f60080f3800 msgr2=0x7f60080f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.090+0000 7f600e706700 1 -- 172.21.15.87:0/3335846085 shutdown_connections 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.090+0000 7f600e706700 1 -- 172.21.15.87:0/3335846085 wait complete. 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 librados: starting msgr at 2024-02-23T00:37:04.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 librados: starting objecter 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 Processor -- start 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 -- start start 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 librados: setting wanted keys 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 librados: calling monclient init 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60080f8420 0x7f600810df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60080fdbb0 0x7f600810e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60080fe510 0x7f60081080b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60081085a0 con 0x7f60080f8420 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6008108710 con 0x7f60080fe510 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f600e706700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60081088d0 con 0x7f60080fdbb0 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f6007fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60080f8420 0x7f600810df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f60077fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60080fdbb0 0x7f600810e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.091+0000 7f6007fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60080f8420 0x7f600810df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47892/0 (socket says 172.21.15.87:47892) 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f6007fff700 1 -- 172.21.15.87:0/3604094351 learned_addr learned my addr 172.21.15.87:0/3604094351 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f600cca3700 1 --2- 172.21.15.87:0/3604094351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60080fe510 0x7f60081080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60077fe700 1 -- 172.21.15.87:0/3604094351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60080fe510 msgr2=0x7f60081080b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60077fe700 1 --2- 172.21.15.87:0/3604094351 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f60080fe510 0x7f60081080b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60077fe700 1 -- 172.21.15.87:0/3604094351 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60080f8420 msgr2=0x7f600810df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60077fe700 1 --2- 172.21.15.87:0/3604094351 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60080f8420 0x7f600810df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60077fe700 1 -- 172.21.15.87:0/3604094351 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60000097c0 con 0x7f60080fdbb0 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f6007fff700 1 --2- 172.21.15.87:0/3604094351 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f60080f8420 0x7f600810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60077fe700 1 --2- 172.21.15.87:0/3604094351 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f60080fdbb0 0x7f600810e400 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 rx=0x7f5ffc00e8f0 tx=0x7f5ffc00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60057fa700 10 client.?.objecter ms_handle_connect 0x7f60080fdbb0 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60057fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc004620 con 0x7f60080fdbb0 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ffc00f040 con 0x7f60080fdbb0 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6008108bb0 con 0x7f60080fdbb0 2024-02-23T00:37:04.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc014830 con 0x7f60080fdbb0 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f600e706700 10 client.37745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6008108f30 con 0x7f60080fdbb0 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.092+0000 7f600e706700 1 librados: init done 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.093+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6008107110 con 0x7f60080fdbb0 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.093+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f5ffc0090d0 con 0x7f60080fdbb0 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60057fa700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5ff0062c90 0x7f5ff0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(443..443 src has 173..443) v4 ==== 7765+0+0 (secure 0 0 0) 0x7f5ffc07cbc0 con 0x7f60080fdbb0 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 osd_map(443..443 src has 173..443) v4 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60057fa700 3 client.37745.objecter handle_osd_map got epochs [443,443] > 0 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60057fa700 3 client.37745.objecter handle_osd_map decoding full epoch 443 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f6007fff700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5ff0062c90 0x7f5ff0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60057fa700 20 client.37745.objecter dump_active .. 0 homeless 2024-02-23T00:37:04.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v443) v1 ==== 54+0+16 (secure 0 0 0) 0x7f5ffc081220 con 0x7f60080fdbb0 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f600e706700 10 client.37745.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 1 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f600e706700 20 client.37745.objecter _get_session osd=-1 returning homeless 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f600e706700 10 client.37745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60057fa700 10 client.37745.objecter ms_handle_connect 0x7f5ff0062c90 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=444}) v3 -- 0x7f6008109590 con 0x7f60080fdbb0 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f6007fff700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5ff0062c90 0x7f5ff0065150 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 rx=0x7f5ff8007d30 tx=0x7f5ff800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6008114ff0 con 0x7f60080fdbb0 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=443) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ffc0502d0 con 0x7f60080fdbb0 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60067fc700 10 client.37745.objecter _check_command_map_dne tid 1 current 443 map_dne_bound 443 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60067fc700 10 client.37745.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.094+0000 7f60067fc700 15 client.37745.objecter _session_command_op_remove -1 1 2024-02-23T00:37:04.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 10 client.37745.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 2 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6805/685354171 conn(0x7f60080f5b60 0x7f60080273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60080f59f0 osd=1 3 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60080f59f0 osd=1 3 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60080f59f0 osd=1 3 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 2 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f60080f59f0 osd=1 3 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 1 2 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60080f59f0 osd=1 4 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 10 client.37745.objecter _send_command 2 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6008059e00 con 0x7f60080f5b60 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600cca3700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6805/685354171 conn(0x7f60080f5b60 0x7f60080273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f60057fa700 10 client.37745.objecter ms_handle_connect 0x7f60080f5b60 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f600cca3700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6805/685354171 conn(0x7f60080f5b60 0x7f60080273c0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6008059e00 con 0x7f60080f5b60 2024-02-23T00:37:04.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:37:04.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f60057fa700 10 client.37745.objecter _finish_command 2 = system:0 2024-02-23T00:37:04.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f60080f59f0 osd=1 4 2024-02-23T00:37:04.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.095+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 1 2 2024-02-23T00:37:04.109 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:04.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 10 client.37745.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 3 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60080f59f0 osd=1 3 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60080f59f0 osd=1 3 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60080f59f0 osd=1 3 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 3 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f60080f59f0 osd=1 3 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 1 3 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60080f59f0 osd=1 4 2024-02-23T00:37:04.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 10 client.37745.objecter _send_command 3 2024-02-23T00:37:04.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.111+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f60080278b0 con 0x7f60080f5b60 2024-02-23T00:37:04.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.112+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f60080278b0 con 0x7f60080f5b60 2024-02-23T00:37:04.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.112+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:04.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.112+0000 7f60057fa700 10 client.37745.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.112+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f60080f59f0 osd=1 4 2024-02-23T00:37:04.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.112+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 1 3 2024-02-23T00:37:04.115 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:37:04.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 10 client.37745.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:04.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 4 2024-02-23T00:37:04.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6809/3605394228 conn(0x7f60080594d0 0x7f6008117170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008059360 osd=2 3 2024-02-23T00:37:04.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008059360 osd=2 3 2024-02-23T00:37:04.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008059360 osd=2 3 2024-02-23T00:37:04.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 4 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f6008059360 osd=2 3 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 2 4 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008059360 osd=2 4 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 10 client.37745.objecter _send_command 4 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f600806cf20 con 0x7f60080594d0 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f6007fff700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6809/3605394228 conn(0x7f60080594d0 0x7f6008117170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f60057fa700 10 client.37745.objecter ms_handle_connect 0x7f60080594d0 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.113+0000 7f6007fff700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6809/3605394228 conn(0x7f60080594d0 0x7f6008117170 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.114+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f600806cf20 con 0x7f60080594d0 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.114+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.114+0000 7f60057fa700 10 client.37745.objecter _finish_command 4 = system:0 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.114+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f6008059360 osd=2 4 2024-02-23T00:37:04.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.114+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 2 4 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 10 client.37745.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 5 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008059360 osd=2 3 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008059360 osd=2 3 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008059360 osd=2 3 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 5 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f6008059360 osd=2 3 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 2 5 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008059360 osd=2 4 2024-02-23T00:37:04.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 10 client.37745.objecter _send_command 5 2024-02-23T00:37:04.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.129+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6008117660 con 0x7f60080594d0 2024-02-23T00:37:04.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6008117660 con 0x7f60080594d0 2024-02-23T00:37:04.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:04.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f60057fa700 10 client.37745.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f6008059360 osd=2 4 2024-02-23T00:37:04.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 2 5 2024-02-23T00:37:04.133 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 10 client.37745.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 6 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6813/128059790 conn(0x7f6008027e30 0x7f60081189b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008027cc0 osd=3 3 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008027cc0 osd=3 3 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008027cc0 osd=3 3 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 6 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f6008027cc0 osd=3 3 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 3 6 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008027cc0 osd=3 4 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 10 client.37745.objecter _send_command 6 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60080ff8c0 con 0x7f6008027e30 2024-02-23T00:37:04.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.130+0000 7f60077fe700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6813/128059790 conn(0x7f6008027e30 0x7f60081189b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.131+0000 7f60057fa700 10 client.37745.objecter ms_handle_connect 0x7f6008027e30 2024-02-23T00:37:04.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.131+0000 7f60077fe700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.87:6813/128059790 conn(0x7f6008027e30 0x7f60081189b0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:04.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.131+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f60080ff8c0 con 0x7f6008027e30 2024-02-23T00:37:04.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.131+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:37:04.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.131+0000 7f60057fa700 10 client.37745.objecter _finish_command 6 = system:0 2024-02-23T00:37:04.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.131+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f6008027cc0 osd=3 4 2024-02-23T00:37:04.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.131+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 3 6 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 10 client.37745.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 7 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008027cc0 osd=3 3 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008027cc0 osd=3 3 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008027cc0 osd=3 3 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 7 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f6008027cc0 osd=3 3 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 3 7 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008027cc0 osd=3 4 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 10 client.37745.objecter _send_command 7 2024-02-23T00:37:04.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.146+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6008119670 con 0x7f6008027e30 2024-02-23T00:37:04.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6008119670 con 0x7f6008027e30 2024-02-23T00:37:04.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:04.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f60057fa700 10 client.37745.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f6008027cc0 osd=3 4 2024-02-23T00:37:04.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 3 7 2024-02-23T00:37:04.150 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:37:04.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 10 client.37745.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 8 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6008120200 0x7f60081225f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60080feeb0 osd=4 3 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60080feeb0 osd=4 3 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60080feeb0 osd=4 3 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 8 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f60080feeb0 osd=4 3 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 4 8 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60080feeb0 osd=4 4 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 10 client.37745.objecter _send_command 8 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.147+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6008125240 con 0x7f6008120200 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.148+0000 7f600cca3700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6008120200 0x7f60081225f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.148+0000 7f600cca3700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6008120200 0x7f60081225f0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.148+0000 7f60057fa700 10 client.37745.objecter ms_handle_connect 0x7f6008120200 2024-02-23T00:37:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.149+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6008125240 con 0x7f6008120200 2024-02-23T00:37:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.149+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:37:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.149+0000 7f60057fa700 10 client.37745.objecter _finish_command 8 = system:0 2024-02-23T00:37:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.149+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f60080feeb0 osd=4 4 2024-02-23T00:37:04.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.149+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 4 8 2024-02-23T00:37:04.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 10 client.37745.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:04.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 9 2024-02-23T00:37:04.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60080feeb0 osd=4 3 2024-02-23T00:37:04.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60080feeb0 osd=4 3 2024-02-23T00:37:04.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60080feeb0 osd=4 3 2024-02-23T00:37:04.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 9 2024-02-23T00:37:04.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f60080feeb0 osd=4 3 2024-02-23T00:37:04.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 4 9 2024-02-23T00:37:04.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60080feeb0 osd=4 4 2024-02-23T00:37:04.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 10 client.37745.objecter _send_command 9 2024-02-23T00:37:04.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.163+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f60081190e0 con 0x7f6008120200 2024-02-23T00:37:04.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f60081190e0 con 0x7f6008120200 2024-02-23T00:37:04.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:04.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f60057fa700 10 client.37745.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f60080feeb0 osd=4 4 2024-02-23T00:37:04.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 4 9 2024-02-23T00:37:04.168 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 10 client.37745.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 10 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6008119250 0x7f6008123c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60081190e0 osd=5 3 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60081190e0 osd=5 3 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60081190e0 osd=5 3 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 10 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f60081190e0 osd=5 3 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 5 10 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60081190e0 osd=5 4 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 10 client.37745.objecter _send_command 10 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.165+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6008118600 con 0x7f6008119250 2024-02-23T00:37:04.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.166+0000 7f6007fff700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6008119250 0x7f6008123c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.166+0000 7f60057fa700 10 client.37745.objecter ms_handle_connect 0x7f6008119250 2024-02-23T00:37:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.166+0000 7f6007fff700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6008119250 0x7f6008123c60 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.166+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6008118600 con 0x7f6008119250 2024-02-23T00:37:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.166+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:37:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.166+0000 7f60057fa700 10 client.37745.objecter _finish_command 10 = system:0 2024-02-23T00:37:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.166+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f60081190e0 osd=5 4 2024-02-23T00:37:04.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.166+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 5 10 2024-02-23T00:37:04.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 10 client.37745.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:04.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 11 2024-02-23T00:37:04.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60081190e0 osd=5 3 2024-02-23T00:37:04.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60081190e0 osd=5 3 2024-02-23T00:37:04.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f60081190e0 osd=5 3 2024-02-23T00:37:04.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 11 2024-02-23T00:37:04.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f60081190e0 osd=5 3 2024-02-23T00:37:04.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 5 11 2024-02-23T00:37:04.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f60081190e0 osd=5 4 2024-02-23T00:37:04.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 10 client.37745.objecter _send_command 11 2024-02-23T00:37:04.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.181+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6008118600 con 0x7f6008119250 2024-02-23T00:37:04.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6008118600 con 0x7f6008119250 2024-02-23T00:37:04.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:04.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f60057fa700 10 client.37745.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f60081190e0 osd=5 4 2024-02-23T00:37:04.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 5 11 2024-02-23T00:37:04.185 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:37:04.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 10 client.37745.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:04.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 12 2024-02-23T00:37:04.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6008124660 0x7f6008124a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008118600 osd=6 3 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008118600 osd=6 3 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008118600 osd=6 3 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 12 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f6008118600 osd=6 3 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 6 12 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008118600 osd=6 4 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 10 client.37745.objecter _send_command 12 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6008125000 con 0x7f6008124660 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.183+0000 7f60077fe700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6008124660 0x7f6008124a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.184+0000 7f60057fa700 10 client.37745.objecter ms_handle_connect 0x7f6008124660 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.184+0000 7f60077fe700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6008124660 0x7f6008124a80 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.184+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6008125000 con 0x7f6008124660 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.184+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.184+0000 7f60057fa700 10 client.37745.objecter _finish_command 12 = system:0 2024-02-23T00:37:04.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.184+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f6008118600 osd=6 4 2024-02-23T00:37:04.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.184+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 6 12 2024-02-23T00:37:04.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.186+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== mon.2 v2:172.21.15.87:3301/0 8 ==== osd_map(444..444 src has 173..444) v4 ==== 652+0+0 (secure 0 0 0) 0x7f5ffc0811a0 con 0x7f60080fdbb0 2024-02-23T00:37:04.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.186+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 osd_map(444..444 src has 173..444) v4 2024-02-23T00:37:04.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.186+0000 7f60057fa700 3 client.37745.objecter handle_osd_map got epochs [444,444] > 443 2024-02-23T00:37:04.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.186+0000 7f60057fa700 3 client.37745.objecter handle_osd_map decoding incremental epoch 444 2024-02-23T00:37:04.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.186+0000 7f60057fa700 20 client.37745.objecter dump_active .. 0 homeless 2024-02-23T00:37:04.213 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:04.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 10 client.37745.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:04.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 13 2024-02-23T00:37:04.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008118600 osd=6 3 2024-02-23T00:37:04.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008118600 osd=6 3 2024-02-23T00:37:04.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008118600 osd=6 3 2024-02-23T00:37:04.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 13 2024-02-23T00:37:04.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f6008118600 osd=6 3 2024-02-23T00:37:04.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 6 13 2024-02-23T00:37:04.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008118600 osd=6 4 2024-02-23T00:37:04.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 10 client.37745.objecter _send_command 13 2024-02-23T00:37:04.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.209+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6008125000 con 0x7f6008124660 2024-02-23T00:37:04.222 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:37:04.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.219+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f6008125000 con 0x7f6008124660 2024-02-23T00:37:04.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.219+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.219+0000 7f60057fa700 10 client.37745.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.219+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f6008118600 osd=6 4 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.219+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 6 13 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 10 client.37745.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 14 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6812/373787000 conn(0x7f6008129990 0x7f600812bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008125000 osd=7 3 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008125000 osd=7 3 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008125000 osd=7 3 2024-02-23T00:37:04.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 14 2024-02-23T00:37:04.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f6008125000 osd=7 3 2024-02-23T00:37:04.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 7 14 2024-02-23T00:37:04.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008125000 osd=7 4 2024-02-23T00:37:04.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 10 client.37745.objecter _send_command 14 2024-02-23T00:37:04.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.220+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f600812eae0 con 0x7f6008129990 2024-02-23T00:37:04.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.223+0000 7f600cca3700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6812/373787000 conn(0x7f6008129990 0x7f600812bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:04.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.223+0000 7f60057fa700 10 client.37745.objecter ms_handle_connect 0x7f6008129990 2024-02-23T00:37:04.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.223+0000 7f600cca3700 1 --2- 172.21.15.87:0/3604094351 >> v2:172.21.15.171:6812/373787000 conn(0x7f6008129990 0x7f600812bd80 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:04.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.224+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f600812eae0 con 0x7f6008129990 2024-02-23T00:37:04.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.224+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:37:04.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.224+0000 7f60057fa700 10 client.37745.objecter _finish_command 14 = system:0 2024-02-23T00:37:04.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.224+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f6008125000 osd=7 4 2024-02-23T00:37:04.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.224+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 7 14 2024-02-23T00:37:04.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 10 client.37745.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:04.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign -1 15 2024-02-23T00:37:04.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008125000 osd=7 3 2024-02-23T00:37:04.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008125000 osd=7 3 2024-02-23T00:37:04.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 20 client.37745.objecter _get_session s=0x7f6008125000 osd=7 3 2024-02-23T00:37:04.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 15 client.37745.objecter _session_command_op_remove -1 15 2024-02-23T00:37:04.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 20 client.37745.objecter get_session s=0x7f6008125000 osd=7 3 2024-02-23T00:37:04.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 15 client.37745.objecter _session_command_op_assign 7 15 2024-02-23T00:37:04.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 20 client.37745.objecter put_session s=0x7f6008125000 osd=7 4 2024-02-23T00:37:04.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 10 client.37745.objecter _send_command 15 2024-02-23T00:37:04.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.240+0000 7f600e706700 1 -- 172.21.15.87:0/3604094351 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f600812eae0 con 0x7f6008129990 2024-02-23T00:37:04.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.244+0000 7f60057fa700 1 -- 172.21.15.87:0/3604094351 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f600812eae0 con 0x7f6008129990 2024-02-23T00:37:04.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.244+0000 7f60057fa700 10 client.37745.objecter ms_dispatch 0x7f6008001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:04.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.244+0000 7f60057fa700 10 client.37745.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.244+0000 7f60057fa700 20 client.37745.objecter put_session s=0x7f6008125000 osd=7 4 2024-02-23T00:37:04.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:04.244+0000 7f60057fa700 15 client.37745.objecter _session_command_op_remove 7 15 2024-02-23T00:37:04.247 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:37:04.247 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:37:04.247 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.247 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.247 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.247 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.247 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.247 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.247 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:37:04.269 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:04.292 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:04.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:04.396 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:04.424 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:03.668124+0000 mgr.y (mgr.24437) 411 : cluster [DBG] pgmap v662: 206 pgs: 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 120/1161 objects degraded (10.336%) 2024-02-23T00:37:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:04.028101+0000 osd.4 (osd.4) 201 : cluster [DBG] 63.15 scrub starts 2024-02-23T00:37:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:04.031641+0000 osd.4 (osd.4) 202 : cluster [DBG] 63.15 scrub ok 2024-02-23T00:37:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:04.095375+0000 mon.c (mon.2) 438 : audit [DBG] from='client.? 172.21.15.87:0/3604094351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:04.190122+0000 mon.a (mon.0) 2645 : cluster [DBG] osdmap e444: 8 total, 7 up, 7 in 2024-02-23T00:37:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:04.196566+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:04.196824+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:03.668124+0000 mgr.y (mgr.24437) 411 : cluster [DBG] pgmap v662: 206 pgs: 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 120/1161 objects degraded (10.336%) 2024-02-23T00:37:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:04.028101+0000 osd.4 (osd.4) 201 : cluster [DBG] 63.15 scrub starts 2024-02-23T00:37:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:04.031641+0000 osd.4 (osd.4) 202 : cluster [DBG] 63.15 scrub ok 2024-02-23T00:37:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:04.095375+0000 mon.c (mon.2) 438 : audit [DBG] from='client.? 172.21.15.87:0/3604094351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:04.190122+0000 mon.a (mon.0) 2645 : cluster [DBG] osdmap e444: 8 total, 7 up, 7 in 2024-02-23T00:37:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:04.196566+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:04.196824+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:04.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:03.668124+0000 mgr.y (mgr.24437) 411 : cluster [DBG] pgmap v662: 206 pgs: 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 120/1161 objects degraded (10.336%) 2024-02-23T00:37:04.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:04.028101+0000 osd.4 (osd.4) 201 : cluster [DBG] 63.15 scrub starts 2024-02-23T00:37:04.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:04.031641+0000 osd.4 (osd.4) 202 : cluster [DBG] 63.15 scrub ok 2024-02-23T00:37:04.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:04.095375+0000 mon.c (mon.2) 438 : audit [DBG] from='client.? 172.21.15.87:0/3604094351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:04.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:04.190122+0000 mon.a (mon.0) 2645 : cluster [DBG] osdmap e444: 8 total, 7 up, 7 in 2024-02-23T00:37:04.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:04.196566+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:04.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:04.196824+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:37:05.593 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:37:05.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.077268+0000 osd.4 (osd.4) 203 : cluster [DBG] 63.3 scrub starts 2024-02-23T00:37:05.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.079144+0000 osd.4 (osd.4) 204 : cluster [DBG] 63.3 scrub ok 2024-02-23T00:37:05.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:05.194788+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:05.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.194901+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e445: 8 total, 7 up, 7 in 2024-02-23T00:37:05.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:05.204257+0000 mon.b (mon.1) 371 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:05.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:05.207036+0000 mon.b (mon.1) 372 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:37:05.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:05.207191+0000 mon.a (mon.0) 2649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:37:05.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.077268+0000 osd.4 (osd.4) 203 : cluster [DBG] 63.3 scrub starts 2024-02-23T00:37:05.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.079144+0000 osd.4 (osd.4) 204 : cluster [DBG] 63.3 scrub ok 2024-02-23T00:37:05.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:05.194788+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:05.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.194901+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e445: 8 total, 7 up, 7 in 2024-02-23T00:37:05.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:05.204257+0000 mon.b (mon.1) 371 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:05.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:05.207036+0000 mon.b (mon.1) 372 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:37:05.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:05.207191+0000 mon.a (mon.0) 2649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:37:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.077268+0000 osd.4 (osd.4) 203 : cluster [DBG] 63.3 scrub starts 2024-02-23T00:37:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.079144+0000 osd.4 (osd.4) 204 : cluster [DBG] 63.3 scrub ok 2024-02-23T00:37:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:05.194788+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.194901+0000 mon.a (mon.0) 2648 : cluster [DBG] osdmap e445: 8 total, 7 up, 7 in 2024-02-23T00:37:05.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:05.204257+0000 mon.b (mon.1) 371 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:05.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:05.207036+0000 mon.b (mon.1) 372 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:37:05.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:05.207191+0000 mon.a (mon.0) 2649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:37:06.047 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:06.053 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:06.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:06.072 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -n osd.0 --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.212.4.0 2024-02-23T00:37:06.157 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:06.187 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:06.216 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:06.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:06.320 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:06.342 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/osd.0/config 2024-02-23T00:37:06.349 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:06.374 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:06.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:06.479 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:06.505 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:06.524 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:06.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:06.535 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:37:06.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.352536+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:06.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.352689+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:06.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.669113+0000 mgr.y (mgr.24437) 412 : cluster [DBG] pgmap v665: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 120/1164 objects degraded (10.309%) 2024-02-23T00:37:06.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.954285+0000 osd.7 (osd.7) 211 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:37:06.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.955703+0000 osd.7 (osd.7) 212 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:37:06.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:06.040501+0000 osd.4 (osd.4) 205 : cluster [DBG] 63.11 scrub starts 2024-02-23T00:37:06.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:06.042177+0000 osd.4 (osd.4) 206 : cluster [DBG] 63.11 scrub ok 2024-02-23T00:37:06.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:06.193597+0000 mon.a (mon.0) 2650 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:37:06.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:06.193670+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e446: 8 total, 7 up, 7 in 2024-02-23T00:37:06.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.352536+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:06.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.352689+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:06.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.669113+0000 mgr.y (mgr.24437) 412 : cluster [DBG] pgmap v665: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 120/1164 objects degraded (10.309%) 2024-02-23T00:37:06.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.954285+0000 osd.7 (osd.7) 211 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:37:06.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.955703+0000 osd.7 (osd.7) 212 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:37:06.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:06.040501+0000 osd.4 (osd.4) 205 : cluster [DBG] 63.11 scrub starts 2024-02-23T00:37:06.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:06.042177+0000 osd.4 (osd.4) 206 : cluster [DBG] 63.11 scrub ok 2024-02-23T00:37:06.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:06.193597+0000 mon.a (mon.0) 2650 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:37:06.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:06.193670+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e446: 8 total, 7 up, 7 in 2024-02-23T00:37:06.628 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:06.655 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.352536+0000 osd.2 (osd.2) 163 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.352689+0000 osd.2 (osd.2) 164 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.669113+0000 mgr.y (mgr.24437) 412 : cluster [DBG] pgmap v665: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 120/1164 objects degraded (10.309%) 2024-02-23T00:37:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.954285+0000 osd.7 (osd.7) 211 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:37:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.955703+0000 osd.7 (osd.7) 212 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:37:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:06.040501+0000 osd.4 (osd.4) 205 : cluster [DBG] 63.11 scrub starts 2024-02-23T00:37:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:06.042177+0000 osd.4 (osd.4) 206 : cluster [DBG] 63.11 scrub ok 2024-02-23T00:37:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:06.193597+0000 mon.a (mon.0) 2650 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:37:06.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:06.193670+0000 mon.a (mon.0) 2651 : cluster [DBG] osdmap e446: 8 total, 7 up, 7 in 2024-02-23T00:37:07.232 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:07.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:07.336 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:07.367 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:07.392 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:07.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:07.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:07.522 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:07.542 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:07.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.836758+0000 osd.2 (osd.2) 165 : cluster [DBG] 2.e scrub starts 2024-02-23T00:37:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:05.837662+0000 osd.2 (osd.2) 166 : cluster [DBG] 2.e scrub ok 2024-02-23T00:37:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:07.064017+0000 osd.1 (osd.1) 193 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:07.064233+0000 osd.1 (osd.1) 194 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:07.097649+0000 mon.a (mon.0) 2652 : cluster [WRN] Health check update: Degraded data redundancy: 120/1164 objects degraded (10.309%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:07.102758+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e447: 8 total, 7 up, 7 in 2024-02-23T00:37:07.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.836758+0000 osd.2 (osd.2) 165 : cluster [DBG] 2.e scrub starts 2024-02-23T00:37:07.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:05.837662+0000 osd.2 (osd.2) 166 : cluster [DBG] 2.e scrub ok 2024-02-23T00:37:07.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:07.064017+0000 osd.1 (osd.1) 193 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:07.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:07.064233+0000 osd.1 (osd.1) 194 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:07.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:07.097649+0000 mon.a (mon.0) 2652 : cluster [WRN] Health check update: Degraded data redundancy: 120/1164 objects degraded (10.309%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:07.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:07.102758+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e447: 8 total, 7 up, 7 in 2024-02-23T00:37:07.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:37:07.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:07.674 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.836758+0000 osd.2 (osd.2) 165 : cluster [DBG] 2.e scrub starts 2024-02-23T00:37:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:05.837662+0000 osd.2 (osd.2) 166 : cluster [DBG] 2.e scrub ok 2024-02-23T00:37:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:07.064017+0000 osd.1 (osd.1) 193 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:07.064233+0000 osd.1 (osd.1) 194 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:07.097649+0000 mon.a (mon.0) 2652 : cluster [WRN] Health check update: Degraded data redundancy: 120/1164 objects degraded (10.309%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:07.102758+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e447: 8 total, 7 up, 7 in 2024-02-23T00:37:08.158 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:08.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:08.261 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:08.290 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:08.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:07.669724+0000 mgr.y (mgr.24437) 413 : cluster [DBG] pgmap v668: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 120/1164 objects degraded (10.309%) 2024-02-23T00:37:08.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:08.057151+0000 osd.5 (osd.5) 171 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:37:08.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:08.058066+0000 osd.5 (osd.5) 172 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:37:08.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:08.105155+0000 osd.4 (osd.4) 207 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:37:08.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:08.107101+0000 osd.4 (osd.4) 208 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:37:08.487 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:08.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:08.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:08.591 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:08.623 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:08.645 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:08.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:08.713 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:37:08.713 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:37:08.713 INFO:teuthology.orchestra.run.smithi087.stdout: "complaint_time": 30, 2024-02-23T00:37:08.714 INFO:teuthology.orchestra.run.smithi087.stdout: "num_blocked_ops": 0 2024-02-23T00:37:08.714 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:37:08.736 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:07.669724+0000 mgr.y (mgr.24437) 413 : cluster [DBG] pgmap v668: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 120/1164 objects degraded (10.309%) 2024-02-23T00:37:08.736 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:08.057151+0000 osd.5 (osd.5) 171 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:37:08.736 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:08.058066+0000 osd.5 (osd.5) 172 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:37:08.737 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:08.105155+0000 osd.4 (osd.4) 207 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:37:08.737 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:08.107101+0000 osd.4 (osd.4) 208 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:37:08.749 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:08.778 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:07.669724+0000 mgr.y (mgr.24437) 413 : cluster [DBG] pgmap v668: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s; 120/1164 objects degraded (10.309%) 2024-02-23T00:37:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:08.057151+0000 osd.5 (osd.5) 171 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:37:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:08.058066+0000 osd.5 (osd.5) 172 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:37:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:08.105155+0000 osd.4 (osd.4) 207 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:37:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:08.107101+0000 osd.4 (osd.4) 208 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:37:09.106 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:09.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:09.210 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:09.238 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:09.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:09.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:09.695 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-23T00:37:09.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:07.821037+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.d scrub starts 2024-02-23T00:37:09.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:07.822445+0000 osd.2 (osd.2) 168 : cluster [DBG] 5.d scrub ok 2024-02-23T00:37:09.760 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:08.423734+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:08.426712+0000 osd.6 (osd.6) 171 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:09.025296+0000 osd.7 (osd.7) 213 : cluster [DBG] 5.22 scrub starts 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:09.026378+0000 osd.7 (osd.7) 214 : cluster [DBG] 5.22 scrub ok 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:09.131360+0000 osd.4 (osd.4) 209 : cluster [DBG] 4.c scrub starts 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:09.133465+0000 osd.4 (osd.4) 210 : cluster [DBG] 4.c scrub ok 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:07.821037+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.d scrub starts 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:07.822445+0000 osd.2 (osd.2) 168 : cluster [DBG] 5.d scrub ok 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:08.423734+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:08.426712+0000 osd.6 (osd.6) 171 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:09.025296+0000 osd.7 (osd.7) 213 : cluster [DBG] 5.22 scrub starts 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:09.026378+0000 osd.7 (osd.7) 214 : cluster [DBG] 5.22 scrub ok 2024-02-23T00:37:09.761 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:09.131360+0000 osd.4 (osd.4) 209 : cluster [DBG] 4.c scrub starts 2024-02-23T00:37:09.762 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:09.133465+0000 osd.4 (osd.4) 210 : cluster [DBG] 4.c scrub ok 2024-02-23T00:37:09.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:09.802 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:07.821037+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.d scrub starts 2024-02-23T00:37:09.802 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:07.822445+0000 osd.2 (osd.2) 168 : cluster [DBG] 5.d scrub ok 2024-02-23T00:37:09.802 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:08.423734+0000 osd.6 (osd.6) 170 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:37:09.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:08.426712+0000 osd.6 (osd.6) 171 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:37:09.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:09.025296+0000 osd.7 (osd.7) 213 : cluster [DBG] 5.22 scrub starts 2024-02-23T00:37:09.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:09.026378+0000 osd.7 (osd.7) 214 : cluster [DBG] 5.22 scrub ok 2024-02-23T00:37:09.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:09.131360+0000 osd.4 (osd.4) 209 : cluster [DBG] 4.c scrub starts 2024-02-23T00:37:09.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:09.133465+0000 osd.4 (osd.4) 210 : cluster [DBG] 4.c scrub ok 2024-02-23T00:37:09.804 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:09.826 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:09.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:09.930 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:09.961 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:09.981 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:09.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:10.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:10.114 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:10.133 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:10.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:10.237 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:10.266 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:10.388 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:10.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:10.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:10.492 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:10.519 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:10.537 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:10.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:10.641 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:10.667 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:10.685 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:10.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:10.740 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:10.744 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:10.749 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:10.789 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:10.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:09.589795+0000 osd.6 (osd.6) 172 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:10.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:09.590015+0000 osd.6 (osd.6) 173 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:10.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:09.670196+0000 mgr.y (mgr.24437) 414 : cluster [DBG] pgmap v669: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 560 B/s wr, 1 op/s; 120/1164 objects degraded (10.309%) 2024-02-23T00:37:10.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:10.028831+0000 osd.7 (osd.7) 215 : cluster [DBG] 5.20 scrub starts 2024-02-23T00:37:10.814 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:10.032336+0000 osd.7 (osd.7) 216 : cluster [DBG] 5.20 scrub ok 2024-02-23T00:37:10.816 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:10.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:09.589795+0000 osd.6 (osd.6) 172 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:10.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:09.590015+0000 osd.6 (osd.6) 173 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:10.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:09.670196+0000 mgr.y (mgr.24437) 414 : cluster [DBG] pgmap v669: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 560 B/s wr, 1 op/s; 120/1164 objects degraded (10.309%) 2024-02-23T00:37:10.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:10.028831+0000 osd.7 (osd.7) 215 : cluster [DBG] 5.20 scrub starts 2024-02-23T00:37:10.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:10.032336+0000 osd.7 (osd.7) 216 : cluster [DBG] 5.20 scrub ok 2024-02-23T00:37:10.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:09.589795+0000 osd.6 (osd.6) 172 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:10.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:09.590015+0000 osd.6 (osd.6) 173 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:10.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:09.670196+0000 mgr.y (mgr.24437) 414 : cluster [DBG] pgmap v669: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 560 B/s wr, 1 op/s; 120/1164 objects degraded (10.309%) 2024-02-23T00:37:10.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:10.028831+0000 osd.7 (osd.7) 215 : cluster [DBG] 5.20 scrub starts 2024-02-23T00:37:10.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:10.032336+0000 osd.7 (osd.7) 216 : cluster [DBG] 5.20 scrub ok 2024-02-23T00:37:10.834 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:10.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:10.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:10.887 INFO:teuthology.orchestra.run.smithi087.stdout:get_pg_num_history pg_num_history pg_num_history(e424 pg_nums {1={19=1},2={54=32},3={135=13},4={56=32,327=31,331=30,335=29,339=28,343=27,347=26,357=25,361=24,367=23,371=22},5={58=32,375=42},6={60=32},19={169=27,173=26,177=25,181=24,185=23,189=22,193=21,197=20,201=19,205=18,209=17,213=16,217=15,221=14,225=13,229=12,233=11},22={97=32},63={114=32},95={143=32},112={166=8},116={170=8},118={171=32},119={172=32},120={173=32},121={176=32},122={177=8},123={178=32},124={179=32},125={181=8},126={181=32},127={182=32},128={184=8},129={184=32},130={185=32},131={188=32},132={189=8},133={189=32},134={191=8},135={191=32},136={194=32},137={197=32},138={198=8},139={199=32},140={200=32},141={203=32},142={205=8},143={206=32},144={207=32},145={209=32},146={212=8},147={212=32},148={215=32},149={215=32},150={218=32},151={219=8},152={220=32},153={222=32},154={223=32},155={226=8},156={227=32},157={229=32},158={231=32},159={233=8},160={233=32},161={233=32},162={235=32},163={240=8},164={242=32},165={242=32},166={245=32},167={247=8},168={250=8},169={252=32},170={254=8},171={257=8},172={260=32},173={261=8},174={264=8},175={266=8},176={268=8},177={268=32},178={271=8},179={271=32},180={275=8},181={278=8},182={281=32},183={282=8},184={283=32},185={288=8},186={289=32},187={295=8},188={297=32},189={302=8},190={306=32},191={309=8},192={314=32},193={316=8},194={323=8},195={330=8},196={337=8},197={339=32},198={342=32},199={347=8},200={353=32},201={354=8},202={361=8},203={368=8},204={371=32},205={375=8},206={382=8},207={391=32},208={394=32},209={397=32},210={400=32},211={403=32},212={407=32}} deleted_pools 174,118,174,119,175,116,175,120,176,112,178,121,179,122,180,123,181,124,183,126,184,127,186,128,186,129,187,130,188,95,190,131,191,132,193,134,193,135,194,125,196,136,198,133,199,137,200,138,202,140,205,141,206,139,207,142,208,143,211,145,214,144,214,146,214,147,217,149,221,151,222,148,224,153,225,152,226,150,228,155,232,154,235,159,237,162,238,161,239,158,240,157,241,156,241,160,242,163,244,165,247,166,249,167,251,164,252,168,256,170,259,169,259,171,263,19,263,173,266,174,267,172,268,175,270,176,273,178,273,179,277,180,280,177,280,181,284,183,285,184,288,182,290,185,296,186,297,187,304,189,305,188,311,191,313,190,318,193,325,194,332,195,338,192,339,196,344,198,349,199,352,197,356,201,363,202,370,200,370,203,377,205,384,206,390,204,393,207,396,208,399,209,402,210,406,211) 2024-02-23T00:37:10.887 INFO:teuthology.orchestra.run.smithi087.stdout:Importing pgid 212.4 2024-02-23T00:37:10.887 INFO:teuthology.orchestra.run.smithi087.stdout:write_pg epoch 408 info 212.4( empty local-lis/les=407/408 n=0 ec=407/407 lis/c=407/407 les/c/f=408/408/0 sis=407) 2024-02-23T00:37:10.887 INFO:teuthology.orchestra.run.smithi087.stdout:Import successful 2024-02-23T00:37:10.938 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:10.968 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:11.040 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:11.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:11.145 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:11.169 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:37:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:37:11.171 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:11.190 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:11.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:11.294 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:11.323 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:09.741536+0000 osd.2 (osd.2) 169 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:37:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:09.742930+0000 osd.2 (osd.2) 170 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:37:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:10.478744+0000 mon.a (mon.0) 2654 : cluster [DBG] osdmap e448: 8 total, 7 up, 7 in 2024-02-23T00:37:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:10.950825+0000 mgr.y (mgr.24437) 415 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:09.741536+0000 osd.2 (osd.2) 169 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:37:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:09.742930+0000 osd.2 (osd.2) 170 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:37:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:10.478744+0000 mon.a (mon.0) 2654 : cluster [DBG] osdmap e448: 8 total, 7 up, 7 in 2024-02-23T00:37:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:10.950825+0000 mgr.y (mgr.24437) 415 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:11.772 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:11.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:11.876 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:11.904 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:09.741536+0000 osd.2 (osd.2) 169 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:37:11.904 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:09.742930+0000 osd.2 (osd.2) 170 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:37:11.904 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:10.478744+0000 mon.a (mon.0) 2654 : cluster [DBG] osdmap e448: 8 total, 7 up, 7 in 2024-02-23T00:37:11.904 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:10.950825+0000 mgr.y (mgr.24437) 415 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:11.906 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:11.927 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:11.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:12.031 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:12.061 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:12.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:37:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:12] "GET /metrics HTTP/1.1" 200 188477 "" "Prometheus/2.33.4" 2024-02-23T00:37:12.704 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/exp.212.4.0 2024-02-23T00:37:13.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:11.671302+0000 mgr.y (mgr.24437) 416 : cluster [DBG] pgmap v671: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:13.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:11.765276+0000 osd.2 (osd.2) 171 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:37:13.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:11.766280+0000 osd.2 (osd.2) 172 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:37:13.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:12.020162+0000 osd.7 (osd.7) 217 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:37:13.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:12.023118+0000 osd.7 (osd.7) 218 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:12.107472+0000 mon.a (mon.0) 2655 : cluster [DBG] osdmap e449: 8 total, 7 up, 7 in 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:12.367054+0000 osd.6 (osd.6) 174 : cluster [DBG] 3.a scrub starts 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:12.369958+0000 osd.6 (osd.6) 175 : cluster [DBG] 3.a scrub ok 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:12.473629+0000 mon.a (mon.0) 2656 : cluster [WRN] Health check update: Degraded data redundancy: 122/1194 objects degraded (10.218%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:11.671302+0000 mgr.y (mgr.24437) 416 : cluster [DBG] pgmap v671: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:11.765276+0000 osd.2 (osd.2) 171 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:11.766280+0000 osd.2 (osd.2) 172 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:12.020162+0000 osd.7 (osd.7) 217 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:12.023118+0000 osd.7 (osd.7) 218 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:12.107472+0000 mon.a (mon.0) 2655 : cluster [DBG] osdmap e449: 8 total, 7 up, 7 in 2024-02-23T00:37:13.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:12.367054+0000 osd.6 (osd.6) 174 : cluster [DBG] 3.a scrub starts 2024-02-23T00:37:13.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:12.369958+0000 osd.6 (osd.6) 175 : cluster [DBG] 3.a scrub ok 2024-02-23T00:37:13.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:12.473629+0000 mon.a (mon.0) 2656 : cluster [WRN] Health check update: Degraded data redundancy: 122/1194 objects degraded (10.218%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:11.671302+0000 mgr.y (mgr.24437) 416 : cluster [DBG] pgmap v671: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:11.765276+0000 osd.2 (osd.2) 171 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:37:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:11.766280+0000 osd.2 (osd.2) 172 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:37:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:12.020162+0000 osd.7 (osd.7) 217 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:37:13.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:12.023118+0000 osd.7 (osd.7) 218 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:37:13.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:12.107472+0000 mon.a (mon.0) 2655 : cluster [DBG] osdmap e449: 8 total, 7 up, 7 in 2024-02-23T00:37:13.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:12.367054+0000 osd.6 (osd.6) 174 : cluster [DBG] 3.a scrub starts 2024-02-23T00:37:13.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:12.369958+0000 osd.6 (osd.6) 175 : cluster [DBG] 3.a scrub ok 2024-02-23T00:37:13.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:12.473629+0000 mon.a (mon.0) 2656 : cluster [WRN] Health check update: Degraded data redundancy: 122/1194 objects degraded (10.218%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:13.514 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:13.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:13.619 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:13.648 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:13.668 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:13.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:13.772 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:13.803 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:13.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:37:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:37:14.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:13.404934+0000 osd.6 (osd.6) 176 : cluster [DBG] 6.c scrub starts 2024-02-23T00:37:14.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:13.405798+0000 osd.6 (osd.6) 177 : cluster [DBG] 6.c scrub ok 2024-02-23T00:37:14.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:13.404934+0000 osd.6 (osd.6) 176 : cluster [DBG] 6.c scrub starts 2024-02-23T00:37:14.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:13.405798+0000 osd.6 (osd.6) 177 : cluster [DBG] 6.c scrub ok 2024-02-23T00:37:14.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:13.404934+0000 osd.6 (osd.6) 176 : cluster [DBG] 6.c scrub starts 2024-02-23T00:37:14.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:13.405798+0000 osd.6 (osd.6) 177 : cluster [DBG] 6.c scrub ok 2024-02-23T00:37:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:13.466401+0000 osd.1 (osd.1) 195 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:37:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:13.467424+0000 osd.1 (osd.1) 196 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:37:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:13.629115+0000 osd.5 (osd.5) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:13.629274+0000 osd.5 (osd.5) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:13.671944+0000 mgr.y (mgr.24437) 417 : cluster [DBG] pgmap v673: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 934 B/s rd, 2.0 KiB/s wr, 4 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:14.008707+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:37:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:14.009606+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:37:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:14.113817+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e450: 8 total, 7 up, 7 in 2024-02-23T00:37:15.501 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:13.466401+0000 osd.1 (osd.1) 195 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:37:15.501 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:13.467424+0000 osd.1 (osd.1) 196 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:37:15.501 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:13.629115+0000 osd.5 (osd.5) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:13.629274+0000 osd.5 (osd.5) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:13.671944+0000 mgr.y (mgr.24437) 417 : cluster [DBG] pgmap v673: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 934 B/s rd, 2.0 KiB/s wr, 4 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:14.008707+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:14.009606+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:14.113817+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e450: 8 total, 7 up, 7 in 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:13.466401+0000 osd.1 (osd.1) 195 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:13.467424+0000 osd.1 (osd.1) 196 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:13.629115+0000 osd.5 (osd.5) 173 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:13.629274+0000 osd.5 (osd.5) 174 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:13.671944+0000 mgr.y (mgr.24437) 417 : cluster [DBG] pgmap v673: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 934 B/s rd, 2.0 KiB/s wr, 4 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:15.502 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:14.008707+0000 osd.7 (osd.7) 219 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:37:15.503 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:14.009606+0000 osd.7 (osd.7) 220 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:37:15.503 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:14.113817+0000 mon.a (mon.0) 2657 : cluster [DBG] osdmap e450: 8 total, 7 up, 7 in 2024-02-23T00:37:15.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 -- 172.21.15.87:0/4071099368 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 msgr2=0x7f1fac101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 --2- 172.21.15.87:0/4071099368 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 0x7f1fac101d20 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 rx=0x7f1f9c0047b0 tx=0x7f1f9c009cb0).stop 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 -- 172.21.15.87:0/4071099368 shutdown_connections 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 --2- 172.21.15.87:0/4071099368 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fac102210 0x7f1fac104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 --2- 172.21.15.87:0/4071099368 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fac0f8cb0 0x7f1fac0f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 --2- 172.21.15.87:0/4071099368 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 0x7f1fac101d20 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 -- 172.21.15.87:0/4071099368 >> 172.21.15.87:0/4071099368 conn(0x7f1fac0f2820 msgr2=0x7f1fac0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 -- 172.21.15.87:0/4071099368 shutdown_connections 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 -- 172.21.15.87:0/4071099368 wait complete. 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 librados: starting msgr at 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.356+0000 7f1fb0860700 1 librados: starting objecter 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 Processor -- start 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 -- start start 2024-02-23T00:37:15.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 librados: setting wanted keys 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 librados: calling monclient init 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fac0f8cb0 0x7f1fac10e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 0x7f1fac10e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fac102210 0x7f1fac1083b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac1088a0 con 0x7f1fac102210 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac108a10 con 0x7f1fac0f8cb0 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fb0860700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac108bd0 con 0x7f1fac0f9610 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fa9d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 0x7f1fac10e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1faad9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fac102210 0x7f1fac1083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fa9d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 0x7f1fac10e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:38438/0 (socket says 172.21.15.87:38438) 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fa9d9b700 1 -- 172.21.15.87:0/2982251608 learned_addr learned my addr 172.21.15.87:0/2982251608 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1faa59c700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fac0f8cb0 0x7f1fac10e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fa9d9b700 1 -- 172.21.15.87:0/2982251608 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fac0f8cb0 msgr2=0x7f1fac10e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:15.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fa9d9b700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fac0f8cb0 0x7f1fac10e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fa9d9b700 1 -- 172.21.15.87:0/2982251608 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fac102210 msgr2=0x7f1fac1083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fa9d9b700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fac102210 0x7f1fac1083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1fa9d9b700 1 -- 172.21.15.87:0/2982251608 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa40097c0 con 0x7f1fac0f9610 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1faa59c700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fac0f8cb0 0x7f1fac10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.357+0000 7f1faad9d700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fac102210 0x7f1fac1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1fa9d9b700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 0x7f1fac10e640 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7f1f9c009b10 tx=0x7f1f9c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1f9b7fe700 10 client.?.objecter ms_handle_connect 0x7f1fac0f9610 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1f9b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1f9b7fe700 1 -- 172.21.15.87:0/2982251608 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c018070 con 0x7f1fac0f9610 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9c005040 con 0x7f1fac0f9610 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1f9b7fe700 1 -- 172.21.15.87:0/2982251608 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f9c013e70 con 0x7f1fac0f9610 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1f9b7fe700 1 -- 172.21.15.87:0/2982251608 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c01ddb0 con 0x7f1fac0f9610 2024-02-23T00:37:15.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1fb0860700 10 client.37754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:15.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fac109070 con 0x7f1fac0f9610 2024-02-23T00:37:15.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.358+0000 7f1fb0860700 1 librados: init done 2024-02-23T00:37:15.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.359+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fac05fa60 con 0x7f1fac0f9610 2024-02-23T00:37:15.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.359+0000 7f1f9b7fe700 1 -- 172.21.15.87:0/2982251608 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1f9c01d420 con 0x7f1fac0f9610 2024-02-23T00:37:15.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.359+0000 7f1f9b7fe700 1 --2- 172.21.15.87:0/2982251608 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f94062c90 0x7f1f94065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:15.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.359+0000 7f1f9b7fe700 1 -- 172.21.15.87:0/2982251608 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(450..450 src has 173..450) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f1f9c07e770 con 0x7f1fac0f9610 2024-02-23T00:37:15.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.359+0000 7f1f9b7fe700 10 client.37754.objecter ms_dispatch 0x7f1fac001550 osd_map(450..450 src has 173..450) v4 2024-02-23T00:37:15.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.359+0000 7f1f9b7fe700 3 client.37754.objecter handle_osd_map got epochs [450,450] > 0 2024-02-23T00:37:15.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.359+0000 7f1f9b7fe700 3 client.37754.objecter handle_osd_map decoding full epoch 450 2024-02-23T00:37:15.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.359+0000 7f1faa59c700 1 --2- 172.21.15.87:0/2982251608 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f94062c90 0x7f1f94065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:15.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.360+0000 7f1f9b7fe700 20 client.37754.objecter dump_active .. 0 homeless 2024-02-23T00:37:15.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.360+0000 7f1f9b7fe700 10 client.37754.objecter ms_handle_connect 0x7f1f94062c90 2024-02-23T00:37:15.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.360+0000 7f1faa59c700 1 --2- 172.21.15.87:0/2982251608 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f94062c90 0x7f1f94065150 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 rx=0x7f1fa00041d0 tx=0x7f1fa0009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:15.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.364+0000 7f1f9b7fe700 1 -- 172.21.15.87:0/2982251608 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1f9c051bc0 con 0x7f1fac0f9610 2024-02-23T00:37:15.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:15.552+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f1fac109840 con 0x7f1fac0f9610 2024-02-23T00:37:15.852 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:37:15.852 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:37:15.852 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:37:15.852 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:37:15.852 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.852 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.24712.0:21 7.2 e112/94 7:40054bab:test-rados-api-smithi087-82115-6::foo:head v 112'4, mlcod=110'1)", 2024-02-23T00:37:15.852 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:37:15.852 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 373.27887341399997, 2024-02-23T00:37:15.852 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050194328000000003, 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:37:15.853 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1158000000000001e-05 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546134+0000", 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8680000000000001e-06 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.854 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546141+0000", 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8170000000000001e-06 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546147+0000", 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0014866250000000001 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.855 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.547634+0000", 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0406000000000001e-05 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.547654+0000", 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001658442 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.549313+0000", 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.047004275999999998 2024-02-23T00:37:15.856 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.596317+0000", 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050194328000000003 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.25157.0:2 58.15 58:ad551702:::test:head [call hello.record_hello in=17b] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.599190+0000", 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 373.22580599000003, 2024-02-23T00:37:15.857 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.066902009999999998, 2024-02-23T00:37:15.858 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.858 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.858 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:37:15.858 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.25157", 2024-02-23T00:37:15.858 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4218332242", 2024-02-23T00:37:15.858 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:37:15.858 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.858 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.858 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599190+0000", 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599190+0000", 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599189+0000", 2024-02-23T00:37:15.859 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4850000000000003e-06 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599196+0000", 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6200000000000005e-06 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599205+0000", 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.934e-06 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599212+0000", 2024-02-23T00:37:15.860 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062881679999999995 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.662093+0000", 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013823899999999999 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.662232+0000", 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00022222400000000001 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.662454+0000", 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0017213930000000001 2024-02-23T00:37:15.861 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.664175+0000", 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001211052 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.665386+0000", 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00061359100000000003 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.666000+0000", 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4636999999999999e-05 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.862 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.666054+0000", 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.79e-05 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.666092+0000", 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.066902009999999998 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:3 63.11 e117/114 63:8d290297:test-rados-api-smithi087-82091-8::foo:head v 117'1, mlcod=0'0)", 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 367.85947203900002, 2024-02-23T00:37:15.863 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074073193999999995, 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.864 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1930000000000001e-06 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965529+0000", 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5130000000000002e-06 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965536+0000", 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4860000000000002e-06 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.865 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965542+0000", 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002109362 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.967652+0000", 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5451000000000002e-05 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.967667+0000", 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071902888999999998 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.866 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.039570+0000", 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7617999999999999e-05 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.039597+0000", 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074073193999999995 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.867 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:10 61.1f e118/112 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head v 118'6, mlcod=115'3)", 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 366.64457563500002, 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034719099000000003, 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.868 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180420+0000", 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7989999999999999e-06 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.869 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180426+0000", 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8590000000000001e-06 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180432+0000", 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8689999999999998e-06 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180439+0000", 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8420000000000001e-05 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180477+0000", 2024-02-23T00:37:15.870 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6568000000000001e-05 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180494+0000", 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034614477999999997 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.215108+0000", 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1405999999999997e-05 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.215140+0000", 2024-02-23T00:37:15.871 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034719099000000003 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:12 61.11 e119/112 61:8c482978:test-rados-api-smithi087-82107-8::foo:head v 119'1, mlcod=0'0)", 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.169560+0000", 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 365.65543602100001, 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025841233000000002, 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.872 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169560+0000", 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169560+0000", 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169560+0000", 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1600000000000004e-06 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.873 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169569+0000", 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2089999999999994e-06 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169578+0000", 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7379999999999997e-06 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.874 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169585+0000", 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0050274330000000004 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.174613+0000", 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9856000000000001e-05 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.174632+0000", 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020737364000000001 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.875 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.195370+0000", 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1779999999999997e-05 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.195402+0000", 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025841233000000002 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.876 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.28906.0:2 69.15 69:ad551702:::test:head [create,write 0~4] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 365.64432993700001, 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071209718000000005, 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.28906", 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3024219439", 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180664+0000", 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8156e-05 2024-02-23T00:37:15.878 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180683+0000", 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3705e-05 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180696+0000", 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0080000000000002e-06 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180704+0000", 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046747999999999999 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.879 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.181172+0000", 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7989999999999996e-06 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.181179+0000", 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.060883893000000001 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242062+0000", 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012322200000000001 2024-02-23T00:37:15.880 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242186+0000", 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015854399999999999 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242344+0000", 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0042969569999999997 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.246641+0000", 2024-02-23T00:37:15.881 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020537120000000001 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.248695+0000", 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0030777460000000001 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251773+0000", 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9827999999999997e-05 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.882 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251832+0000", 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3659000000000003e-05 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251876+0000", 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071209718000000005 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:14 61.11 e120/112 61:8c482978:test-rados-api-smithi087-82107-8::foo:head v 120'3, mlcod=0'0)", 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 364.59912674499998, 2024-02-23T00:37:15.883 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.14463899, 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:37:15.884 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225869+0000", 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2559999999999996e-06 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225877+0000", 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2709999999999998e-06 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.885 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225883+0000", 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1999999999999999e-06 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225889+0000", 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.13293687000000001 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358826+0000", 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1993000000000001e-05 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.886 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358848+0000", 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011619638 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.370468+0000", 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0979000000000002e-05 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.370509+0000", 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.14463899 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.887 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.41269.0:2 89.6 89:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e137)", 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 347.80209445000003, 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.10320085900000001, 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.41269", 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/244424669", 2024-02-23T00:37:15.888 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.889 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022900+0000", 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2230000000000001e-06 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022907+0000", 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026383299999999999 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.023171+0000", 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0040999999999999e-05 2024-02-23T00:37:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.023181+0000", 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014422166 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.037603+0000", 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5469999999999997e-06 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.037612+0000", 2024-02-23T00:37:15.891 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.064763451999999999 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102375+0000", 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011688600000000001 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102492+0000", 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013874699999999999 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102631+0000", 2024-02-23T00:37:15.892 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00093563499999999998 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.103566+0000", 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001628163 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.105195+0000", 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020814273000000001 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.893 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126009+0000", 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0680000000000003e-05 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126059+0000", 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3244999999999997e-05 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126103+0000", 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.10320085900000001 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.894 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:265 86.0 86:00000000:%01:::head [pgnls start_epoch 139] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e139)", 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 345.78758538099999, 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.048698721, 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:37:15.895 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 265 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.896 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037409+0000", 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.456e-06 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037418+0000", 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8989999999999996e-06 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037427+0000", 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0649999999999999e-06 2024-02-23T00:37:15.897 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037435+0000", 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017399350000000001 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.054835+0000", 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8939999999999993e-06 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.054843+0000", 2024-02-23T00:37:15.898 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.031096629000000001 2024-02-23T00:37:15.899 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.899 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.899 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.899 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085939+0000", 2024-02-23T00:37:15.899 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8654e-05 2024-02-23T00:37:15.899 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.899 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.899 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.899 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085978+0000", 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013163199999999999 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.086110+0000", 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.048698721 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.36860.0:2 114.16 114:6eb9e51e:::oid_RoundTripWriteSamePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-23T00:37:15.900 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 316.82085937099998, 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036246133, 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.36860", 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4037067096", 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:37:15.901 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004135+0000", 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6179999999999995e-06 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004143+0000", 2024-02-23T00:37:15.902 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4696e-05 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004158+0000", 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7260000000000008e-06 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004167+0000", 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5045999999999999e-05 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004242+0000", 2024-02-23T00:37:15.903 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1400000000000002e-06 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004249+0000", 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016930304 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.021179+0000", 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2629999999999997e-06 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:37:15.904 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.021187+0000", 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016612273 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037799+0000", 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3777999999999997e-05 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037873+0000", 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7861000000000005e-05 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:37:15.905 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037960+0000", 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00035295600000000003 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.038313+0000", 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028716700000000001 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.038601+0000", 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016768 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.906 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040277+0000", 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0985000000000001e-05 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040328+0000", 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4675000000000001e-05 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040383+0000", 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036246133 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.907 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:537 63.8 e182/114 63:1544ebfe:test-rados-api-smithi087-82091-14::foo17:7 v 182'38, mlcod=148'34)", 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.003082+0000", 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 302.82191445900003, 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026810972999999998, 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003082+0000", 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.908 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003082+0000", 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003082+0000", 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4889999999999998e-06 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003089+0000", 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1940000000000003e-06 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.909 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003095+0000", 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6700000000000002e-06 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003100+0000", 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016500702999999999 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.019601+0000", 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8841000000000001e-05 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.910 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.019620+0000", 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010253505 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.029873+0000", 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9760999999999999e-05 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.029893+0000", 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026810972999999998 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.911 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:538 63.8 e182/114 63:1309b075:test-rados-api-smithi087-82091-14::foo36:7 v 182'40, mlcod=148'34)", 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 302.82189150699998, 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027705371999999999, 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.912 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003104+0000", 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7220000000000002e-06 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.913 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003111+0000", 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5139999999999998e-06 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003118+0000", 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8010000000000001e-06 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003122+0000", 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024802080000000001 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.914 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027924+0000", 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9273999999999999e-05 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027943+0000", 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0028410689999999999 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030784+0000", 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6224000000000002e-05 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.915 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030810+0000", 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027705371999999999 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:540 63.f e182/114 63:f2825405:test-rados-api-smithi087-82091-14::foo80:7 v 182'40, mlcod=148'34)", 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.003149+0000", 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 302.82184698899999, 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021336491999999999, 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.916 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003149+0000", 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003149+0000", 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.917 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003149+0000", 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6730000000000003e-06 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003156+0000", 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0520000000000001e-06 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003161+0000", 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.703e-06 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.918 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003164+0000", 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0067076920000000003 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.009872+0000", 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0903999999999998e-05 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.009893+0000", 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014570588000000001 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.919 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.024464+0000", 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2133e-05 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.024486+0000", 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021336491999999999 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37106.0:2 149.1a 149:5ec95afd:::oid_ReadIntoBufferlist_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-23T00:37:15.920 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 268.797828864, 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.079034594, 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37106", 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4049231794", 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:37:15.921 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027166+0000", 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9900000000000002e-06 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.922 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027172+0000", 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4880000000000007e-06 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027180+0000", 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4049999999999997e-06 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027187+0000", 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000237202 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.923 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027424+0000", 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000115528 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027539+0000", 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00014589299999999999 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,3", 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027685+0000", 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00088195299999999999 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.924 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.028567+0000", 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026987410999999999 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.055555+0000", 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050551275 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106106+0000", 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5090999999999998e-05 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.925 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106161+0000", 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0967000000000003e-05 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106202+0000", 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.079034594 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.926 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:2298 169.17 169:ed571587:test-rados-api-smithi087-82091-30::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e256)", 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 228.79958889100001, 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869, 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2298 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:37:15.927 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025406+0000", 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1740000000000006e-06 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.928 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025415+0000", 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1041e-05 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025426+0000", 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0450000000000003e-06 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025433+0000", 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7723999999999998e-05 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.929 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025491+0000", 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3479999999999993e-06 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025500+0000", 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.21538338000000001 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.240883+0000", 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021381300000000001 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.930 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.241097+0000", 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00573626 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246834+0000", 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6838999999999999e-05 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246930+0000", 2024-02-23T00:37:15.931 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013423800000000001 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,2", 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247065+0000", 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037269000000000001 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247437+0000", 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026355299999999998 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.932 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247701+0000", 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011556330000000001 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248856+0000", 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7636999999999998e-05 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248904+0000", 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0179000000000003e-05 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248944+0000", 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 322/321 [PushReplyOp(4:e9187ab8:reshard::reshard.0000000000:head)])", 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 162.501626835, 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089931579999999997, 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.934 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.935 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9330000000000004e-06 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323376+0000", 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1073000000000001e-05 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323387+0000", 2024-02-23T00:37:15.936 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.489e-06 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323393+0000", 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089729134000000002 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.413122+0000", 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00017875200000000001 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.413301+0000", 2024-02-23T00:37:15.937 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089931579999999997 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 322/321 [PushReplyOp(4:ec529b31:::obj_delete_at_hint.0000000050:head)])", 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 162.06737210399999, 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089810938000000007, 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.938 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757623+0000", 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9920000000000007e-06 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.939 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757631+0000", 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1764e-05 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757643+0000", 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4240000000000004e-06 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757650+0000", 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089629418000000002 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.847280+0000", 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000155267 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.847435+0000", 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089810938000000007 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.941 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 323/321 [PushReplyOp(4:ece66ccb:::obj_delete_at_hint.0000000102:head)])", 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 161.719247876, 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089865278000000007, 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.942 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0180000000000001e-06 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105753+0000", 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3409999999999997e-06 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.943 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105760+0000", 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2069999999999997e-06 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105764+0000", 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089713752999999993 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.195478+0000", 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013554999999999999 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.944 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.195614+0000", 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089865278000000007 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 323/321 [PushReplyOp(4:ee62d5d2:lc::lc.1:head)])", 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 161.543000998, 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.090010914999999997, 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.945 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7539999999999998e-06 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.946 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281998+0000", 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8160000000000004e-06 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.282002+0000", 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.794e-06 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.282005+0000", 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089811657000000003 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.947 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.371817+0000", 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000189092 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.372006+0000", 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.090010914999999997 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_update_log_missing(205.3s1 epoch 376/375 rep_tid 700 entries 376'1 (0'0) error 205:cbb9903e:::test_obj:head by client.37589.0:1 0.000000 -95 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-23T00:37:15.948 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 108.72672059600001, 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065042928, 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:37:15.949 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098275+0000", 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2939999999999994e-06 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098285+0000", 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2446000000000001e-05 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098297+0000", 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2310000000000002e-06 2024-02-23T00:37:15.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098306+0000", 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012984100000000001 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098436+0000", 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.064882361999999999 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.163319+0000", 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065042928 2024-02-23T00:37:15.951 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.952 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.952 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.952 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:15.952 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:15.952 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:37:16.084 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:16.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:16.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.125+0000 7f1f9b7fe700 1 -- 172.21.15.87:0/2982251608 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v451) v1 ==== 101+0+0 (secure 0 0 0) 0x7f1f9c0517b0 con 0x7f1fac0f9610 2024-02-23T00:37:16.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 10 librados: watch_flush enter 2024-02-23T00:37:16.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 10 librados: watch_flush exit 2024-02-23T00:37:16.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 20 client.37754.objecter shutdown clearing up homeless session... 2024-02-23T00:37:16.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 10 client.37754.objecter successfully canceled tick 2024-02-23T00:37:16.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f94062c90 msgr2=0x7f1f94065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:16.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 --2- 172.21.15.87:0/2982251608 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f94062c90 0x7f1f94065150 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 rx=0x7f1fa00041d0 tx=0x7f1fa0009040).stop 2024-02-23T00:37:16.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 msgr2=0x7f1fac10e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:16.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 0x7f1fac10e640 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 rx=0x7f1f9c009b10 tx=0x7f1f9c009cb0).stop 2024-02-23T00:37:16.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 shutdown_connections 2024-02-23T00:37:16.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 --2- 172.21.15.87:0/2982251608 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1f94062c90 0x7f1f94065150 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:16.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1fac102210 0x7f1fac1083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:16.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1fac0f8cb0 0x7f1fac10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:16.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 --2- 172.21.15.87:0/2982251608 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1fac0f9610 0x7f1fac10e640 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:16.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.129+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 >> 172.21.15.87:0/2982251608 conn(0x7f1fac0f2820 msgr2=0x7f1fac0f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:16.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.130+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 shutdown_connections 2024-02-23T00:37:16.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.130+0000 7f1fb0860700 1 -- 172.21.15.87:0/2982251608 wait complete. 2024-02-23T00:37:16.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:16.130+0000 7f1fb0860700 1 librados: shutdown 2024-02-23T00:37:16.133 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:37:16.190 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:16.222 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:16.274 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:16.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:16.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:16.377 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:16.403 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:14.197110+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:37:16.403 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:14.200599+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:37:16.404 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:15.554472+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.87:0/2982251608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:16.404 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:15.554756+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:16.404 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:14.197110+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:37:16.404 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:14.200599+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:37:16.404 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:15.554472+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.87:0/2982251608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:16.404 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:15.554756+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:16.406 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:14.197110+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:37:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:14.200599+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:37:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:15.554472+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.87:0/2982251608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:15.554756+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:16.898 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:15.418634+0000 osd.1 (osd.1) 197 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:15.422426+0000 osd.1 (osd.1) 198 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:15.673098+0000 mgr.y (mgr.24437) 418 : cluster [DBG] pgmap v675: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:15.834860+0000 osd.2 (osd.2) 173 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:15.836565+0000 osd.2 (osd.2) 174 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:16.003270+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:16.003463+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:16.008203+0000 osd.7 (osd.7) 221 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:16.011243+0000 osd.7 (osd.7) 222 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:16.112712+0000 mon.a (mon.0) 2659 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:37:17.216 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:16.125239+0000 mon.a (mon.0) 2660 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:16.125308+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e451: 8 total, 7 up, 7 in 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:15.418634+0000 osd.1 (osd.1) 197 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:15.422426+0000 osd.1 (osd.1) 198 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:15.673098+0000 mgr.y (mgr.24437) 418 : cluster [DBG] pgmap v675: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:15.834860+0000 osd.2 (osd.2) 173 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:15.836565+0000 osd.2 (osd.2) 174 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:16.003270+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:16.003463+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:16.008203+0000 osd.7 (osd.7) 221 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:16.011243+0000 osd.7 (osd.7) 222 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:16.112712+0000 mon.a (mon.0) 2659 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:37:17.217 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:16.125239+0000 mon.a (mon.0) 2660 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:37:17.218 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:16.125308+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e451: 8 total, 7 up, 7 in 2024-02-23T00:37:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:15.418634+0000 osd.1 (osd.1) 197 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:37:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:15.422426+0000 osd.1 (osd.1) 198 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:37:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:15.673098+0000 mgr.y (mgr.24437) 418 : cluster [DBG] pgmap v675: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:17.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:15.834860+0000 osd.2 (osd.2) 173 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:37:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:15.836565+0000 osd.2 (osd.2) 174 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:37:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:16.003270+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:16.003463+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:16.008203+0000 osd.7 (osd.7) 221 : cluster [DBG] 6.15 scrub starts 2024-02-23T00:37:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:16.011243+0000 osd.7 (osd.7) 222 : cluster [DBG] 6.15 scrub ok 2024-02-23T00:37:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:16.112712+0000 mon.a (mon.0) 2659 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:37:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:16.125239+0000 mon.a (mon.0) 2660 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:37:17.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:16.125308+0000 mon.a (mon.0) 2661 : cluster [DBG] osdmap e451: 8 total, 7 up, 7 in 2024-02-23T00:37:17.776 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:37:17.777 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:37:17.777 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-02-23T00:37:17.777 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-23T00:37:17.777 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-23T00:37:17.777 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-02-23T00:37:18.156 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:18.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:18.260 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:18.292 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:18.315 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:18.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.425+0000 7f093ca61700 1 -- 172.21.15.87:0/1630571173 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09380fd420 msgr2=0x7f09380f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:18.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.425+0000 7f093ca61700 1 --2- 172.21.15.87:0/1630571173 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09380fd420 0x7f09380f6c50 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 rx=0x7f092c004870 tx=0x7f092c009cb0).stop 2024-02-23T00:37:18.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.425+0000 7f093ca61700 1 -- 172.21.15.87:0/1630571173 shutdown_connections 2024-02-23T00:37:18.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.425+0000 7f093ca61700 1 --2- 172.21.15.87:0/1630571173 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09380f7330 0x7f0938103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.425+0000 7f093ca61700 1 --2- 172.21.15.87:0/1630571173 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09380fcac0 0x7f09380fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.425+0000 7f093ca61700 1 --2- 172.21.15.87:0/1630571173 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09380fd420 0x7f09380f6c50 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.425+0000 7f093ca61700 1 -- 172.21.15.87:0/1630571173 >> 172.21.15.87:0/1630571173 conn(0x7f09380f26f0 msgr2=0x7f09380f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.425+0000 7f093ca61700 1 -- 172.21.15.87:0/1630571173 shutdown_connections 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.425+0000 7f093ca61700 1 -- 172.21.15.87:0/1630571173 wait complete. 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 librados: starting msgr at 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 librados: starting objecter 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 Processor -- start 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 -- start start 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 librados: setting wanted keys 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 librados: calling monclient init 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09380f7330 0x7f093810cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09380fcac0 0x7f093810d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09380fd420 0x7f0938107130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938107620 con 0x7f09380fd420 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938107790 con 0x7f09380fcac0 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.426+0000 7f093ca61700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0938107950 con 0x7f09380f7330 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f093659c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09380f7330 0x7f093810cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f0936d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09380fd420 0x7f0938107130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f093659c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09380f7330 0x7f093810cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:38454/0 (socket says 172.21.15.87:38454) 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f093659c700 1 -- 172.21.15.87:0/2202003160 learned_addr learned my addr 172.21.15.87:0/2202003160 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f0935d9b700 1 --2- 172.21.15.87:0/2202003160 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09380fcac0 0x7f093810d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f0936d9d700 1 -- 172.21.15.87:0/2202003160 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09380f7330 msgr2=0x7f093810cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f0936d9d700 1 --2- 172.21.15.87:0/2202003160 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09380f7330 0x7f093810cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f0936d9d700 1 -- 172.21.15.87:0/2202003160 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09380fcac0 msgr2=0x7f093810d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f0936d9d700 1 --2- 172.21.15.87:0/2202003160 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09380fcac0 0x7f093810d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f0936d9d700 1 -- 172.21.15.87:0/2202003160 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0928005040 con 0x7f09380fd420 2024-02-23T00:37:18.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f0935d9b700 1 --2- 172.21.15.87:0/2202003160 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f09380fcac0 0x7f093810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f093659c700 1 --2- 172.21.15.87:0/2202003160 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f09380f7330 0x7f093810cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f0936d9d700 1 --2- 172.21.15.87:0/2202003160 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f09380fd420 0x7f0938107130 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 rx=0x7f093000d940 tx=0x7f0930004e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f09277fe700 10 client.?.objecter ms_handle_connect 0x7f09380fd420 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f09277fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093000b5a0 con 0x7f09380fd420 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092c005040 con 0x7f09380fd420 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f093000f040 con 0x7f09380fd420 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930013a70 con 0x7f09380fd420 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f093ca61700 10 client.27300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0938107f90 con 0x7f09380fd420 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.427+0000 7f093ca61700 1 librados: init done 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.428+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f09380606d0 con 0x7f09380fd420 2024-02-23T00:37:18.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f093000b700 con 0x7f09380fd420 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0920062c90 0x7f0920065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(451..451 src has 173..451) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f093007d5c0 con 0x7f09380fd420 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 osd_map(451..451 src has 173..451) v4 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f093659c700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0920062c90 0x7f0920065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 3 client.27300.objecter handle_osd_map got epochs [451,451] > 0 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 3 client.27300.objecter handle_osd_map decoding full epoch 451 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 20 client.27300.objecter dump_active .. 0 homeless 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v451) v1 ==== 54+0+16 (secure 0 0 0) 0x7f093007daa0 con 0x7f09380fd420 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 10 client.27300.objecter ms_handle_connect 0x7f0920062c90 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f093659c700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0920062c90 0x7f0920065150 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 rx=0x7f0928009810 tx=0x7f092800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f093ca61700 10 client.27300.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 1 2024-02-23T00:37:18.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f093ca61700 20 client.27300.objecter _get_session osd=-1 returning homeless 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f093ca61700 10 client.27300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=452}) v3 -- 0x7f093805ee80 con 0x7f09380fd420 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0938105f70 con 0x7f09380fd420 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.429+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_get_version_reply(handle=1 version=451) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0930013ea0 con 0x7f09380fd420 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f0934d99700 10 client.27300.objecter _check_command_map_dne tid 1 current 451 map_dne_bound 451 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f0934d99700 10 client.27300.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f0934d99700 15 client.27300.objecter _session_command_op_remove -1 1 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 10 client.27300.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 2 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6805/685354171 conn(0x7f09380f4dc0 0x7f09380273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09380f4c50 osd=1 3 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09380f4c50 osd=1 3 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09380f4c50 osd=1 3 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 2 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f09380f4c50 osd=1 3 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 1 2 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09380f4c50 osd=1 4 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 10 client.27300.objecter _send_command 2 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0938059e00 con 0x7f09380f4dc0 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f0935d9b700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6805/685354171 conn(0x7f09380f4dc0 0x7f09380273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f09277fe700 10 client.27300.objecter ms_handle_connect 0x7f09380f4dc0 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.430+0000 7f0935d9b700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6805/685354171 conn(0x7f09380f4dc0 0x7f09380273c0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.431+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0938059e00 con 0x7f09380f4dc0 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.431+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.431+0000 7f09277fe700 10 client.27300.objecter _finish_command 2 = system:0 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.431+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f09380f4c50 osd=1 4 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.431+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 1 2 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 10 client.27300.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 3 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09380f4c50 osd=1 3 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09380f4c50 osd=1 3 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09380f4c50 osd=1 3 2024-02-23T00:37:18.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 3 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f09380f4c50 osd=1 3 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 1 3 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09380f4c50 osd=1 4 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 10 client.27300.objecter _send_command 3 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.445+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09380278b0 con 0x7f09380f4dc0 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f09380278b0 con 0x7f09380f4dc0 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 10 client.27300.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f09380f4c50 osd=1 4 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 1 3 2024-02-23T00:37:18.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 10 client.27300.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 4 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6809/3605394228 conn(0x7f09380594d0 0x7f0938060e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938059360 osd=2 3 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938059360 osd=2 3 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938059360 osd=2 3 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 4 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f0938059360 osd=2 3 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 2 4 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938059360 osd=2 4 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 10 client.27300.objecter _send_command 4 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0938028760 con 0x7f09380594d0 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093659c700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6809/3605394228 conn(0x7f09380594d0 0x7f0938060e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 10 client.27300.objecter ms_handle_connect 0x7f09380594d0 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f093659c700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6809/3605394228 conn(0x7f09380594d0 0x7f0938060e60 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0938028760 con 0x7f09380594d0 2024-02-23T00:37:18.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:37:18.327 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:37:18.327 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:37:18.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 10 client.27300.objecter _finish_command 4 = system:0 2024-02-23T00:37:18.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f0938059360 osd=2 4 2024-02-23T00:37:18.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.448+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 2 4 2024-02-23T00:37:18.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 10 client.27300.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:18.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 5 2024-02-23T00:37:18.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938059360 osd=2 3 2024-02-23T00:37:18.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938059360 osd=2 3 2024-02-23T00:37:18.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938059360 osd=2 3 2024-02-23T00:37:18.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 5 2024-02-23T00:37:18.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f0938059360 osd=2 3 2024-02-23T00:37:18.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 2 5 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938059360 osd=2 4 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 10 client.27300.objecter _send_command 5 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.462+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0938061b20 con 0x7f09380594d0 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0938061b20 con 0x7f09380594d0 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f09277fe700 10 client.27300.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f0938059360 osd=2 4 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 2 5 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 10 client.27300.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 6 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6813/128059790 conn(0x7f0938027e30 0x7f093811a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938027cc0 osd=3 3 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938027cc0 osd=3 3 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938027cc0 osd=3 3 2024-02-23T00:37:18.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 6 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f0938027cc0 osd=3 3 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 3 6 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938027cc0 osd=3 4 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 10 client.27300.objecter _send_command 6 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f093811d1a0 con 0x7f0938027e30 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f0935d9b700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6813/128059790 conn(0x7f0938027e30 0x7f093811a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f09277fe700 10 client.27300.objecter ms_handle_connect 0x7f0938027e30 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.464+0000 7f0935d9b700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.87:6813/128059790 conn(0x7f0938027e30 0x7f093811a3c0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.465+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f093811d1a0 con 0x7f0938027e30 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.465+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.465+0000 7f09277fe700 10 client.27300.objecter _finish_command 6 = system:0 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.465+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f0938027cc0 osd=3 4 2024-02-23T00:37:18.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.465+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 3 6 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 10 client.27300.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 7 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938027cc0 osd=3 3 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938027cc0 osd=3 3 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938027cc0 osd=3 3 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 7 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f0938027cc0 osd=3 3 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 3 7 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938027cc0 osd=3 4 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 10 client.27300.objecter _send_command 7 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.479+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09380615f0 con 0x7f0938027e30 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f09380615f0 con 0x7f0938027e30 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f09277fe700 10 client.27300.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f0938027cc0 osd=3 4 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 3 7 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 10 client.27300.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 8 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6800/4175844757 conn(0x7f093811a910 0x7f093811ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09380fdff0 osd=4 3 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09380fdff0 osd=4 3 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09380fdff0 osd=4 3 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 8 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f09380fdff0 osd=4 3 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 4 8 2024-02-23T00:37:18.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09380fdff0 osd=4 4 2024-02-23T00:37:18.333 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:37:18.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 10 client.27300.objecter _send_command 8 2024-02-23T00:37:18.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09380fe730 con 0x7f093811a910 2024-02-23T00:37:18.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f0936d9d700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6800/4175844757 conn(0x7f093811a910 0x7f093811ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f09277fe700 10 client.27300.objecter ms_handle_connect 0x7f093811a910 2024-02-23T00:37:18.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.481+0000 7f0936d9d700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6800/4175844757 conn(0x7f093811a910 0x7f093811ad50 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:18.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.482+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f09380fe730 con 0x7f093811a910 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.482+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.482+0000 7f09277fe700 10 client.27300.objecter _finish_command 8 = system:0 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.482+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f09380fdff0 osd=4 4 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.482+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 4 8 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 10 client.27300.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 9 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09380fdff0 osd=4 3 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09380fdff0 osd=4 3 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09380fdff0 osd=4 3 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 9 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f09380fdff0 osd=4 3 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 4 9 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09380fdff0 osd=4 4 2024-02-23T00:37:18.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 10 client.27300.objecter _send_command 9 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.496+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f093811cb90 con 0x7f093811a910 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f093811cb90 con 0x7f093811a910 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f09277fe700 10 client.27300.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f09380fdff0 osd=4 4 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 4 9 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 10 client.27300.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 10 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6804/2550013795 conn(0x7f093811f3d0 0x7f09381217c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f093811cb90 osd=5 3 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f093811cb90 osd=5 3 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f093811cb90 osd=5 3 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 10 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f093811cb90 osd=5 3 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 5 10 2024-02-23T00:37:18.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f093811cb90 osd=5 4 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 10 client.27300.objecter _send_command 10 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09381244f0 con 0x7f093811f3d0 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.498+0000 7f093659c700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6804/2550013795 conn(0x7f093811f3d0 0x7f09381217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.499+0000 7f09277fe700 10 client.27300.objecter ms_handle_connect 0x7f093811f3d0 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.499+0000 7f093659c700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6804/2550013795 conn(0x7f093811f3d0 0x7f09381217c0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.499+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f09381244f0 con 0x7f093811f3d0 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.499+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.499+0000 7f09277fe700 10 client.27300.objecter _finish_command 10 = system:0 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.499+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f093811cb90 osd=5 4 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.499+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 5 10 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 10 client.27300.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 11 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f093811cb90 osd=5 3 2024-02-23T00:37:18.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f093811cb90 osd=5 3 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f093811cb90 osd=5 3 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 11 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f093811cb90 osd=5 3 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 5 11 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f093811cb90 osd=5 4 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 10 client.27300.objecter _send_command 11 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.514+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09381244f0 con 0x7f093811f3d0 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f09381244f0 con 0x7f093811f3d0 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f09277fe700 10 client.27300.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f093811cb90 osd=5 4 2024-02-23T00:37:18.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 5 11 2024-02-23T00:37:18.338 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:37:18.338 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:37:18.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 10 client.27300.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:18.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 12 2024-02-23T00:37:18.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0938122e50 0x7f093811b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09381244f0 osd=6 3 2024-02-23T00:37:18.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09381244f0 osd=6 3 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09381244f0 osd=6 3 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 12 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f09381244f0 osd=6 3 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 6 12 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09381244f0 osd=6 4 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 10 client.27300.objecter _send_command 12 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0938123fc0 con 0x7f0938122e50 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f0935d9b700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0938122e50 0x7f093811b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f0935d9b700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6808/2509408498 conn(0x7f0938122e50 0x7f093811b240 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.516+0000 7f09277fe700 10 client.27300.objecter ms_handle_connect 0x7f0938122e50 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.517+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f0938123fc0 con 0x7f0938122e50 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.517+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.517+0000 7f09277fe700 10 client.27300.objecter _finish_command 12 = system:0 2024-02-23T00:37:18.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.517+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f09381244f0 osd=6 4 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.517+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 6 12 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 10 client.27300.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 13 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09381244f0 osd=6 3 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09381244f0 osd=6 3 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f09381244f0 osd=6 3 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 13 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f09381244f0 osd=6 3 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 6 13 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f09381244f0 osd=6 4 2024-02-23T00:37:18.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 10 client.27300.objecter _send_command 13 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.531+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0938123fc0 con 0x7f0938122e50 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f0938123fc0 con 0x7f0938122e50 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f09277fe700 10 client.27300.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f09381244f0 osd=6 4 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 6 13 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 10 client.27300.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 14 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6812/373787000 conn(0x7f093811bc40 0x7f093811c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938123fc0 osd=7 3 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938123fc0 osd=7 3 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938123fc0 osd=7 3 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 14 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f0938123fc0 osd=7 3 2024-02-23T00:37:18.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 7 14 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938123fc0 osd=7 4 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 10 client.27300.objecter _send_command 14 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09381242c0 con 0x7f093811bc40 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f0936d9d700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6812/373787000 conn(0x7f093811bc40 0x7f093811c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f09277fe700 10 client.27300.objecter ms_handle_connect 0x7f093811bc40 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.533+0000 7f0936d9d700 1 --2- 172.21.15.87:0/2202003160 >> v2:172.21.15.171:6812/373787000 conn(0x7f093811bc40 0x7f093811c0a0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.534+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f09381242c0 con 0x7f093811bc40 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.534+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.534+0000 7f09277fe700 10 client.27300.objecter _finish_command 14 = system:0 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.534+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f0938123fc0 osd=7 4 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.534+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 7 14 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 10 client.27300.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign -1 15 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938123fc0 osd=7 3 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938123fc0 osd=7 3 2024-02-23T00:37:18.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 20 client.27300.objecter _get_session s=0x7f0938123fc0 osd=7 3 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 15 client.27300.objecter _session_command_op_remove -1 15 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 20 client.27300.objecter get_session s=0x7f0938123fc0 osd=7 3 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 15 client.27300.objecter _session_command_op_assign 7 15 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 20 client.27300.objecter put_session s=0x7f0938123fc0 osd=7 4 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 10 client.27300.objecter _send_command 15 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.548+0000 7f093ca61700 1 -- 172.21.15.87:0/2202003160 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f09381242c0 con 0x7f093811bc40 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.550+0000 7f09277fe700 1 -- 172.21.15.87:0/2202003160 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f09381242c0 con 0x7f093811bc40 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.550+0000 7f09277fe700 10 client.27300.objecter ms_dispatch 0x7f0938001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.550+0000 7f09277fe700 10 client.27300.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.550+0000 7f09277fe700 20 client.27300.objecter put_session s=0x7f0938123fc0 osd=7 4 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:17.550+0000 7f09277fe700 15 client.27300.objecter _session_command_op_remove 7 15 2024-02-23T00:37:18.343 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:37:18.344 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:37:18.344 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.344 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.344 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.344 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.344 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.345 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.345 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:37:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:18.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:18.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:17.429988+0000 mon.a (mon.0) 2662 : audit [DBG] from='client.? 172.21.15.87:0/2202003160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:18.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:17.496645+0000 osd.1 (osd.1) 199 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:37:18.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:17.499690+0000 osd.1 (osd.1) 200 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:37:18.447 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:18.476 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:18.497 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:18.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:18.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:18.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:17.429988+0000 mon.a (mon.0) 2662 : audit [DBG] from='client.? 172.21.15.87:0/2202003160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:18.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:17.496645+0000 osd.1 (osd.1) 199 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:37:18.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:17.499690+0000 osd.1 (osd.1) 200 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:37:18.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:17.429988+0000 mon.a (mon.0) 2662 : audit [DBG] from='client.? 172.21.15.87:0/2202003160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:18.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:17.496645+0000 osd.1 (osd.1) 199 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:37:18.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:17.499690+0000 osd.1 (osd.1) 200 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:37:18.600 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:18.627 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:37:18.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:37:18.975 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:18.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:19.018 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:19.019 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:19.079 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:19.107 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:19.281 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:17.673629+0000 mgr.y (mgr.24437) 419 : cluster [DBG] pgmap v677: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:19.282 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:17.821089+0000 osd.2 (osd.2) 175 : cluster [DBG] 6.f scrub starts 2024-02-23T00:37:19.282 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:17.822769+0000 osd.2 (osd.2) 176 : cluster [DBG] 6.f scrub ok 2024-02-23T00:37:19.282 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:18.041809+0000 osd.7 (osd.7) 223 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:37:19.282 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:18.044732+0000 osd.7 (osd.7) 224 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:37:19.282 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:17.673629+0000 mgr.y (mgr.24437) 419 : cluster [DBG] pgmap v677: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:19.282 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:17.821089+0000 osd.2 (osd.2) 175 : cluster [DBG] 6.f scrub starts 2024-02-23T00:37:19.282 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:17.822769+0000 osd.2 (osd.2) 176 : cluster [DBG] 6.f scrub ok 2024-02-23T00:37:19.282 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:18.041809+0000 osd.7 (osd.7) 223 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:37:19.282 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:18.044732+0000 osd.7 (osd.7) 224 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:37:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:17.673629+0000 mgr.y (mgr.24437) 419 : cluster [DBG] pgmap v677: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:19.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:17.821089+0000 osd.2 (osd.2) 175 : cluster [DBG] 6.f scrub starts 2024-02-23T00:37:19.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:17.822769+0000 osd.2 (osd.2) 176 : cluster [DBG] 6.f scrub ok 2024-02-23T00:37:19.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:18.041809+0000 osd.7 (osd.7) 223 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:37:19.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:18.044732+0000 osd.7 (osd.7) 224 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:37:20.134 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:37:20.155 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:20.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:20.261 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:20.294 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:20.315 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:20.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:20.419 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:19.033224+0000 osd.7 (osd.7) 225 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:37:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:19.034127+0000 osd.7 (osd.7) 226 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:37:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:19.325912+0000 osd.6 (osd.6) 178 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:37:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:19.328946+0000 osd.6 (osd.6) 179 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:37:20.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:20.007288+0000 osd.5 (osd.5) 175 : cluster [DBG] 6.7 deep-scrub starts 2024-02-23T00:37:20.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:20.008722+0000 osd.5 (osd.5) 176 : cluster [DBG] 6.7 deep-scrub ok 2024-02-23T00:37:20.448 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:20.467 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:20.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:20.540 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:19.033224+0000 osd.7 (osd.7) 225 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:19.034127+0000 osd.7 (osd.7) 226 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:19.325912+0000 osd.6 (osd.6) 178 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:19.328946+0000 osd.6 (osd.6) 179 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:20.007288+0000 osd.5 (osd.5) 175 : cluster [DBG] 6.7 deep-scrub starts 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:20.008722+0000 osd.5 (osd.5) 176 : cluster [DBG] 6.7 deep-scrub ok 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:19.033224+0000 osd.7 (osd.7) 225 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:19.034127+0000 osd.7 (osd.7) 226 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:19.325912+0000 osd.6 (osd.6) 178 : cluster [DBG] 6.1f scrub starts 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:19.328946+0000 osd.6 (osd.6) 179 : cluster [DBG] 6.1f scrub ok 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:20.007288+0000 osd.5 (osd.5) 175 : cluster [DBG] 6.7 deep-scrub starts 2024-02-23T00:37:20.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:20.008722+0000 osd.5 (osd.5) 176 : cluster [DBG] 6.7 deep-scrub ok 2024-02-23T00:37:20.572 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:20.599 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:20.947 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:20.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:20.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:20.969 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:37:21.051 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:21.077 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:21.297 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:21.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:19.674151+0000 mgr.y (mgr.24437) 420 : cluster [DBG] pgmap v678: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 676 B/s rd, 0 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:21.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:20.315386+0000 osd.6 (osd.6) 180 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:37:21.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:20.316425+0000 osd.6 (osd.6) 181 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:37:21.298 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:20.378433+0000 osd.1 (osd.1) 201 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:37:21.298 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:20.382100+0000 osd.1 (osd.1) 202 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:37:21.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:19.674151+0000 mgr.y (mgr.24437) 420 : cluster [DBG] pgmap v678: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 676 B/s rd, 0 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:21.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:20.315386+0000 osd.6 (osd.6) 180 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:37:21.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:20.316425+0000 osd.6 (osd.6) 181 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:37:21.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:20.378433+0000 osd.1 (osd.1) 201 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:37:21.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:20.382100+0000 osd.1 (osd.1) 202 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:37:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:21.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:21.408 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:21.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:19.674151+0000 mgr.y (mgr.24437) 420 : cluster [DBG] pgmap v678: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 676 B/s rd, 0 op/s; 122/1194 objects degraded (10.218%) 2024-02-23T00:37:21.431 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:20.315386+0000 osd.6 (osd.6) 180 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:37:21.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:20.316425+0000 osd.6 (osd.6) 181 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:37:21.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:20.378433+0000 osd.1 (osd.1) 201 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:37:21.432 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:20.382100+0000 osd.1 (osd.1) 202 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:37:21.432 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:37:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:37:21.433 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:21.452 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:21.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:21.556 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:21.581 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:21.600 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:21.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:21.704 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:21.736 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:21.935 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:21.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:22.039 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:22.049 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:37:22.049 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:37:22.066 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:22.085 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:22.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:22.178 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:20.452700+0000 osd.7 (osd.7) 227 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:22.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:20.452864+0000 osd.7 (osd.7) 228 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:22.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:20.956956+0000 mgr.y (mgr.24437) 421 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:22.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:21.143039+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e452: 8 total, 7 up, 7 in 2024-02-23T00:37:22.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:21.360127+0000 osd.1 (osd.1) 203 : cluster [DBG] 22.1c deep-scrub starts 2024-02-23T00:37:22.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:21.364925+0000 osd.1 (osd.1) 204 : cluster [DBG] 22.1c deep-scrub ok 2024-02-23T00:37:22.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.008919+0000 osd.5 (osd.5) 177 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:37:22.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.011087+0000 osd.5 (osd.5) 178 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:37:22.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.107666+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e453: 8 total, 7 up, 7 in 2024-02-23T00:37:22.188 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:22.214 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:22.233 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:22.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:22.337 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:22.364 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:22.384 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:22.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:20.452700+0000 osd.7 (osd.7) 227 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:20.452864+0000 osd.7 (osd.7) 228 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:20.956956+0000 mgr.y (mgr.24437) 421 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:21.143039+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e452: 8 total, 7 up, 7 in 2024-02-23T00:37:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:21.360127+0000 osd.1 (osd.1) 203 : cluster [DBG] 22.1c deep-scrub starts 2024-02-23T00:37:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:21.364925+0000 osd.1 (osd.1) 204 : cluster [DBG] 22.1c deep-scrub ok 2024-02-23T00:37:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.008919+0000 osd.5 (osd.5) 177 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:37:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.011087+0000 osd.5 (osd.5) 178 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:37:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.107666+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e453: 8 total, 7 up, 7 in 2024-02-23T00:37:22.490 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:22.516 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:22.534 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:22.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:22.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:22] "GET /metrics HTTP/1.1" 200 188477 "" "Prometheus/2.33.4" 2024-02-23T00:37:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:20.452700+0000 osd.7 (osd.7) 227 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:20.452864+0000 osd.7 (osd.7) 228 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:20.956956+0000 mgr.y (mgr.24437) 421 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:21.143039+0000 mon.a (mon.0) 2663 : cluster [DBG] osdmap e452: 8 total, 7 up, 7 in 2024-02-23T00:37:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:21.360127+0000 osd.1 (osd.1) 203 : cluster [DBG] 22.1c deep-scrub starts 2024-02-23T00:37:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:21.364925+0000 osd.1 (osd.1) 204 : cluster [DBG] 22.1c deep-scrub ok 2024-02-23T00:37:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.008919+0000 osd.5 (osd.5) 177 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:37:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.011087+0000 osd.5 (osd.5) 178 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:37:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.107666+0000 mon.a (mon.0) 2664 : cluster [DBG] osdmap e453: 8 total, 7 up, 7 in 2024-02-23T00:37:22.638 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:22.815 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:22.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:22.823 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:22.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-23T00:37:22.864 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:22.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:22.947 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:23.077 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:23.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:23.181 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:23.210 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:23.229 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:23.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:23.332 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:23.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:21.674749+0000 mgr.y (mgr.24437) 422 : cluster [DBG] pgmap v680: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 406 B/s wr, 3 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.165680+0000 mon.a (mon.0) 2665 : cluster [WRN] Health check update: Degraded data redundancy: 120/1182 objects degraded (10.152%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.336284+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:37:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.340121+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:37:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.395263+0000 osd.1 (osd.1) 205 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:37:23.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.396218+0000 osd.1 (osd.1) 206 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:37:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:21.674749+0000 mgr.y (mgr.24437) 422 : cluster [DBG] pgmap v680: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 406 B/s wr, 3 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.165680+0000 mon.a (mon.0) 2665 : cluster [WRN] Health check update: Degraded data redundancy: 120/1182 objects degraded (10.152%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:23.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.336284+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:37:23.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.340121+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:37:23.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.395263+0000 osd.1 (osd.1) 205 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:37:23.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.396218+0000 osd.1 (osd.1) 206 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:37:23.363 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:21.674749+0000 mgr.y (mgr.24437) 422 : cluster [DBG] pgmap v680: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 406 B/s wr, 3 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.165680+0000 mon.a (mon.0) 2665 : cluster [WRN] Health check update: Degraded data redundancy: 120/1182 objects degraded (10.152%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.336284+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:37:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.340121+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:37:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.395263+0000 osd.1 (osd.1) 205 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:37:23.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.396218+0000 osd.1 (osd.1) 206 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:37:23.916 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:23.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:24.020 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:24.033 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:37:24.033 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-23T00:37:24.033 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:37:24.050 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:24.181 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:37:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:37:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.917414+0000 osd.2 (osd.2) 177 : cluster [DBG] 6.14 deep-scrub starts 2024-02-23T00:37:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:22.918243+0000 osd.2 (osd.2) 178 : cluster [DBG] 6.14 deep-scrub ok 2024-02-23T00:37:24.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:23.356700+0000 osd.1 (osd.1) 207 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:37:24.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:23.360221+0000 osd.1 (osd.1) 208 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:37:24.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.917414+0000 osd.2 (osd.2) 177 : cluster [DBG] 6.14 deep-scrub starts 2024-02-23T00:37:24.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:22.918243+0000 osd.2 (osd.2) 178 : cluster [DBG] 6.14 deep-scrub ok 2024-02-23T00:37:24.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:23.356700+0000 osd.1 (osd.1) 207 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:37:24.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:23.360221+0000 osd.1 (osd.1) 208 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:37:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.917414+0000 osd.2 (osd.2) 177 : cluster [DBG] 6.14 deep-scrub starts 2024-02-23T00:37:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:22.918243+0000 osd.2 (osd.2) 178 : cluster [DBG] 6.14 deep-scrub ok 2024-02-23T00:37:24.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:23.356700+0000 osd.1 (osd.1) 207 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:37:24.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:23.360221+0000 osd.1 (osd.1) 208 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:37:25.328 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:25.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:25.432 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:23.675466+0000 mgr.y (mgr.24437) 423 : cluster [DBG] pgmap v682: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 406 B/s wr, 2 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:24.651227+0000 mon.a (mon.0) 2666 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:24.989493+0000 osd.7 (osd.7) 229 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:37:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:24.990667+0000 osd.7 (osd.7) 230 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:37:25.463 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:25.483 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:25.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:25.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:23.675466+0000 mgr.y (mgr.24437) 423 : cluster [DBG] pgmap v682: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 406 B/s wr, 2 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:25.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:24.651227+0000 mon.a (mon.0) 2666 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:25.511 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:24.989493+0000 osd.7 (osd.7) 229 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:37:25.512 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:24.990667+0000 osd.7 (osd.7) 230 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:37:25.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:23.675466+0000 mgr.y (mgr.24437) 423 : cluster [DBG] pgmap v682: 238 pgs: 28 active+undersized+degraded, 62 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 406 B/s wr, 2 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:25.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:24.651227+0000 mon.a (mon.0) 2666 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:25.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:24.989493+0000 osd.7 (osd.7) 229 : cluster [DBG] 6.1b scrub starts 2024-02-23T00:37:25.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:24.990667+0000 osd.7 (osd.7) 230 : cluster [DBG] 6.1b scrub ok 2024-02-23T00:37:25.588 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:25.616 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:25.636 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:25.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:25.740 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:25.767 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:25.787 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:25.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:25.891 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:25.996 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:26.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:26.101 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:26.132 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:26.153 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:26.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:26.257 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:26.286 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:26.305 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:26.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:26.409 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:26.437 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:26.456 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:26.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:26.560 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:26.589 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:26.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:26.815 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:26.842 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:26.861 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:26.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:26.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:26.965 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:26.997 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:27.019 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:27.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:27.123 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:27.153 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:27.173 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:27.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:27.277 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:27.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:25.676300+0000 mgr.y (mgr.24437) 424 : cluster [DBG] pgmap v683: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 48 KiB/s rd, 383 B/s wr, 85 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:27.303 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:26.267418+0000 osd.1 (osd.1) 209 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:26.268274+0000 osd.1 (osd.1) 210 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:26.912793+0000 osd.7 (osd.7) 231 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:26.914692+0000 osd.7 (osd.7) 232 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:26.932394+0000 osd.5 (osd.5) 179 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:26.933284+0000 osd.5 (osd.5) 180 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:27.156521+0000 mon.a (mon.0) 2667 : audit [INF] from='client.? 172.21.15.87:0/3101818036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:25.676300+0000 mgr.y (mgr.24437) 424 : cluster [DBG] pgmap v683: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 48 KiB/s rd, 383 B/s wr, 85 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:26.267418+0000 osd.1 (osd.1) 209 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:26.268274+0000 osd.1 (osd.1) 210 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:37:27.304 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:26.912793+0000 osd.7 (osd.7) 231 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:37:27.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:26.914692+0000 osd.7 (osd.7) 232 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:37:27.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:26.932394+0000 osd.5 (osd.5) 179 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:37:27.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:26.933284+0000 osd.5 (osd.5) 180 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:37:27.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:27.156521+0000 mon.a (mon.0) 2667 : audit [INF] from='client.? 172.21.15.87:0/3101818036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:37:27.306 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:25.676300+0000 mgr.y (mgr.24437) 424 : cluster [DBG] pgmap v683: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 48 KiB/s rd, 383 B/s wr, 85 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:26.267418+0000 osd.1 (osd.1) 209 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:37:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:26.268274+0000 osd.1 (osd.1) 210 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:37:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:26.912793+0000 osd.7 (osd.7) 231 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:37:27.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:26.914692+0000 osd.7 (osd.7) 232 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:37:27.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:26.932394+0000 osd.5 (osd.5) 179 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:37:27.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:26.933284+0000 osd.5 (osd.5) 180 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:37:27.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:27.156521+0000 mon.a (mon.0) 2667 : audit [INF] from='client.? 172.21.15.87:0/3101818036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:37:27.505 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:27.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:27.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 -- 172.21.15.87:0/1966798575 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 msgr2=0x7fbc080f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:27.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/1966798575 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 0x7fbc080f7d40 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 rx=0x7fbbfc004870 tx=0x7fbbfc009cb0).stop 2024-02-23T00:37:27.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 -- 172.21.15.87:0/1966798575 shutdown_connections 2024-02-23T00:37:27.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/1966798575 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc080f8420 0x7fbc08104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:27.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/1966798575 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc080fdbb0 0x7fbc080fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:27.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/1966798575 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 0x7fbc080f7d40 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:27.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 -- 172.21.15.87:0/1966798575 >> 172.21.15.87:0/1966798575 conn(0x7fbc080f3800 msgr2=0x7fbc080f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 -- 172.21.15.87:0/1966798575 shutdown_connections 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 -- 172.21.15.87:0/1966798575 wait complete. 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 librados: starting msgr at 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.970+0000 7fbc0cf73700 1 librados: starting objecter 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 Processor -- start 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 -- start start 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 librados: setting wanted keys 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 librados: calling monclient init 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc080f8420 0x7fbc0810e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:27.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc080fdbb0 0x7fbc0810e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 0x7fbc081082a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc08108790 con 0x7fbc080f8420 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc08108900 con 0x7fbc080fdbb0 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0cf73700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc08108ac0 con 0x7fbc080fe510 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc080f8420 0x7fbc0810e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc06d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 0x7fbc081082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc080f8420 0x7fbc0810e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49738/0 (socket says 172.21.15.87:49738) 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc06d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 0x7fbc081082a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:52364/0 (socket says 172.21.15.87:52364) 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0659c700 1 -- 172.21.15.87:0/3101818036 learned_addr learned my addr 172.21.15.87:0/3101818036 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:27.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc05d9b700 1 --2- 172.21.15.87:0/3101818036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc080fdbb0 0x7fbc0810e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0659c700 1 -- 172.21.15.87:0/3101818036 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 msgr2=0x7fbc081082a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0659c700 1 --2- 172.21.15.87:0/3101818036 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 0x7fbc081082a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0659c700 1 -- 172.21.15.87:0/3101818036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc080fdbb0 msgr2=0x7fbc0810e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0659c700 1 --2- 172.21.15.87:0/3101818036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc080fdbb0 0x7fbc0810e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0659c700 1 -- 172.21.15.87:0/3101818036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf8005040 con 0x7fbc080f8420 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc06d9d700 1 --2- 172.21.15.87:0/3101818036 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 0x7fbc081082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbc0659c700 1 --2- 172.21.15.87:0/3101818036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc080f8420 0x7fbc0810e100 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7fbc0000db80 tx=0x7fbc00004e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbbf77fe700 10 client.?.objecter ms_handle_connect 0x7fbc080f8420 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.971+0000 7fbbf77fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.972+0000 7fbbf77fe700 1 -- 172.21.15.87:0/3101818036 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc00007e70 con 0x7fbc080f8420 2024-02-23T00:37:27.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.972+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbfc005040 con 0x7fbc080f8420 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.972+0000 7fbbf77fe700 1 -- 172.21.15.87:0/3101818036 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc0000f040 con 0x7fbc080f8420 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.972+0000 7fbbf77fe700 1 -- 172.21.15.87:0/3101818036 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc000036c0 con 0x7fbc080f8420 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.972+0000 7fbc0cf73700 10 client.27306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.972+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc08109120 con 0x7fbc080f8420 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.972+0000 7fbc0cf73700 1 librados: init done 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.973+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0805ee80 con 0x7fbc080f8420 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.973+0000 7fbbf77fe700 1 -- 172.21.15.87:0/3101818036 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fbc00003ab0 con 0x7fbc080f8420 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.973+0000 7fbbf77fe700 1 --2- 172.21.15.87:0/3101818036 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbbf0062c90 0x7fbbf0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.973+0000 7fbbf77fe700 1 -- 172.21.15.87:0/3101818036 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(453..453 src has 173..453) v4 ==== 8150+0+0 (secure 0 0 0) 0x7fbc0007cdd0 con 0x7fbc080f8420 2024-02-23T00:37:27.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.973+0000 7fbbf77fe700 10 client.27306.objecter ms_dispatch 0x7fbc08001550 osd_map(453..453 src has 173..453) v4 2024-02-23T00:37:27.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.973+0000 7fbbf77fe700 3 client.27306.objecter handle_osd_map got epochs [453,453] > 0 2024-02-23T00:37:27.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.973+0000 7fbbf77fe700 3 client.27306.objecter handle_osd_map decoding full epoch 453 2024-02-23T00:37:27.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.973+0000 7fbc05d9b700 1 --2- 172.21.15.87:0/3101818036 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbbf0062c90 0x7fbbf0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:27.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.973+0000 7fbbf77fe700 20 client.27306.objecter dump_active .. 0 homeless 2024-02-23T00:37:27.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.974+0000 7fbbf77fe700 10 client.27306.objecter ms_handle_connect 0x7fbbf0062c90 2024-02-23T00:37:27.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.974+0000 7fbc05d9b700 1 --2- 172.21.15.87:0/3101818036 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbbf0062c90 0x7fbbf0065150 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7fbbfc004ee0 tx=0x7fbbfc009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:27.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:26.978+0000 7fbbf77fe700 1 -- 172.21.15.87:0/3101818036 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbc00050280 con 0x7fbc080f8420 2024-02-23T00:37:27.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:27.155+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fbc0805fa60 con 0x7fbc080f8420 2024-02-23T00:37:27.596 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:27.609 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:27.640 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:27.657 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:27.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:27.761 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:27.789 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:27.861 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:27.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:27.965 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:27.995 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:28.093 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:28.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:28.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.110+0000 7fbbf77fe700 1 -- 172.21.15.87:0/3101818036 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v454) v1 ==== 87+0+0 (secure 0 0 0) 0x7fbc0007e030 con 0x7fbc080f8420 2024-02-23T00:37:28.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.114+0000 7fbc0cf73700 10 librados: watch_flush enter 2024-02-23T00:37:28.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.114+0000 7fbc0cf73700 10 librados: watch_flush exit 2024-02-23T00:37:28.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.114+0000 7fbc0cf73700 20 client.27306.objecter shutdown clearing up homeless session... 2024-02-23T00:37:28.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.114+0000 7fbc0cf73700 10 client.27306.objecter successfully canceled tick 2024-02-23T00:37:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.120+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbbf0062c90 msgr2=0x7fbbf0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.120+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/3101818036 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbbf0062c90 0x7fbbf0065150 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7fbbfc004ee0 tx=0x7fbbfc009cb0).stop 2024-02-23T00:37:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.120+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc080f8420 msgr2=0x7fbc0810e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.120+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/3101818036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc080f8420 0x7fbc0810e100 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 rx=0x7fbc0000db80 tx=0x7fbc00004e60).stop 2024-02-23T00:37:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.121+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 shutdown_connections 2024-02-23T00:37:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.121+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/3101818036 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbbf0062c90 0x7fbbf0065150 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.121+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/3101818036 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbc080fe510 0x7fbc081082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:28.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.121+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/3101818036 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbc080f8420 0x7fbc0810e100 secure :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 rx=0x7fbc0000db80 tx=0x7fbc00004e60).stop 2024-02-23T00:37:28.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.121+0000 7fbc0cf73700 1 --2- 172.21.15.87:0/3101818036 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbc080fdbb0 0x7fbc0810e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:28.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.121+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 >> 172.21.15.87:0/3101818036 conn(0x7fbc080f3800 msgr2=0x7fbc080fb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:28.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.123+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 shutdown_connections 2024-02-23T00:37:28.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.131+0000 7fbc0cf73700 1 -- 172.21.15.87:0/3101818036 wait complete. 2024-02-23T00:37:28.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.146+0000 7fbc0cf73700 1 librados: shutdown 2024-02-23T00:37:28.167 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:37:28.198 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:28.227 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:28.247 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:28.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:28.351 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:28.380 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:27.208411+0000 mon.b (mon.1) 373 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:28.380 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:27.208513+0000 mon.a (mon.0) 2668 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:28.380 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:27.209086+0000 mon.b (mon.1) 374 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-69"}]: dispatch 2024-02-23T00:37:28.380 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:27.209094+0000 mon.a (mon.0) 2669 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-69"}]: dispatch 2024-02-23T00:37:28.380 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:28.106835+0000 mon.a (mon.0) 2670 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:28.380 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:28.110863+0000 mon.a (mon.0) 2671 : audit [INF] from='client.? 172.21.15.87:0/3101818036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:37:28.380 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:28.110953+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e454: 8 total, 7 up, 7 in 2024-02-23T00:37:28.380 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:27.208411+0000 mon.b (mon.1) 373 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:28.380 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:27.208513+0000 mon.a (mon.0) 2668 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:28.381 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:27.209086+0000 mon.b (mon.1) 374 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-69"}]: dispatch 2024-02-23T00:37:28.381 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:27.209094+0000 mon.a (mon.0) 2669 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-69"}]: dispatch 2024-02-23T00:37:28.381 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:28.106835+0000 mon.a (mon.0) 2670 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:28.381 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:28.110863+0000 mon.a (mon.0) 2671 : audit [INF] from='client.? 172.21.15.87:0/3101818036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:37:28.381 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:28.110953+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e454: 8 total, 7 up, 7 in 2024-02-23T00:37:28.381 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:28.406 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:28.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:28.510 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:28.539 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:27.208411+0000 mon.b (mon.1) 373 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:27.208513+0000 mon.a (mon.0) 2668 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:27.209086+0000 mon.b (mon.1) 374 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-69"}]: dispatch 2024-02-23T00:37:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:27.209094+0000 mon.a (mon.0) 2669 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-69"}]: dispatch 2024-02-23T00:37:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:28.106835+0000 mon.a (mon.0) 2670 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:28.110863+0000 mon.a (mon.0) 2671 : audit [INF] from='client.? 172.21.15.87:0/3101818036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:37:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:28.110953+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e454: 8 total, 7 up, 7 in 2024-02-23T00:37:28.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.389+0000 7fbde52c9700 1 -- 172.21.15.87:0/602527533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbde00f7400 msgr2=0x7fbde0103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:28.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.389+0000 7fbde52c9700 1 --2- 172.21.15.87:0/602527533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbde00f7400 0x7fbde0103a20 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 rx=0x7fbdd8009be0 tx=0x7fbdd8004d10).stop 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 -- 172.21.15.87:0/602527533 shutdown_connections 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 --2- 172.21.15.87:0/602527533 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbde00f7400 0x7fbde0103a20 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 --2- 172.21.15.87:0/602527533 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbde00fcbb0 0x7fbde00fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 --2- 172.21.15.87:0/602527533 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbde00fd510 0x7fbde00f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 -- 172.21.15.87:0/602527533 >> 172.21.15.87:0/602527533 conn(0x7fbde00f2820 msgr2=0x7fbde00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 -- 172.21.15.87:0/602527533 shutdown_connections 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 -- 172.21.15.87:0/602527533 wait complete. 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 librados: starting msgr at 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 librados: starting objecter 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 Processor -- start 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 -- start start 2024-02-23T00:37:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 librados: setting wanted keys 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.390+0000 7fbde52c9700 1 librados: calling monclient init 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbde52c9700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbde00f7400 0x7fbde00f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbde52c9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbde00fcbb0 0x7fbde00f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbde52c9700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbde00fd510 0x7fbde00f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbde52c9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde0116500 con 0x7fbde00fcbb0 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbde52c9700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde0116670 con 0x7fbde00fd510 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbde52c9700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde0116830 con 0x7fbde00f7400 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddeffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbde00f7400 0x7fbde00f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbdde7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbde00fcbb0 0x7fbde00f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddeffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbde00f7400 0x7fbde00f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:52384/0 (socket says 172.21.15.87:52384) 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbdde7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbde00fcbb0 0x7fbde00f8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49754/0 (socket says 172.21.15.87:49754) 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddeffd700 1 -- 172.21.15.87:0/3784807438 learned_addr learned my addr 172.21.15.87:0/3784807438 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddf7fe700 1 --2- 172.21.15.87:0/3784807438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbde00fd510 0x7fbde00f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddeffd700 1 -- 172.21.15.87:0/3784807438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbde00fd510 msgr2=0x7fbde00f9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddeffd700 1 --2- 172.21.15.87:0/3784807438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbde00fd510 0x7fbde00f9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddeffd700 1 -- 172.21.15.87:0/3784807438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbde00fcbb0 msgr2=0x7fbde00f8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddeffd700 1 --2- 172.21.15.87:0/3784807438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbde00fcbb0 0x7fbde00f8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddeffd700 1 -- 172.21.15.87:0/3784807438 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd80097c0 con 0x7fbde00f7400 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbdde7fc700 1 --2- 172.21.15.87:0/3784807438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbde00fcbb0 0x7fbde00f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbddeffd700 1 --2- 172.21.15.87:0/3784807438 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbde00f7400 0x7fbde00f89b0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 rx=0x7fbdd000e970 tx=0x7fbdd000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbdc7fff700 10 client.?.objecter ms_handle_connect 0x7fbde00f7400 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbdc7fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0004660 con 0x7fbde00f7400 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde0116b10 con 0x7fbde00f7400 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdd000f040 con 0x7fbde00f7400 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.391+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0014790 con 0x7fbde00f7400 2024-02-23T00:37:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.392+0000 7fbde52c9700 10 client.37772.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.392+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde0117050 con 0x7fbde00f7400 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.392+0000 7fbde52c9700 1 librados: init done 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.392+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbde005fa60 con 0x7fbde00f7400 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.393+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fbdd00090d0 con 0x7fbde00f7400 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.393+0000 7fbdc7fff700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbdc8062c90 0x7fbdc8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.393+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(454..454 src has 173..454) v4 ==== 7765+0+0 (secure 0 0 0) 0x7fbdd007cc10 con 0x7fbde00f7400 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.393+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 osd_map(454..454 src has 173..454) v4 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.393+0000 7fbdc7fff700 3 client.37772.objecter handle_osd_map got epochs [454,454] > 0 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.393+0000 7fbdc7fff700 3 client.37772.objecter handle_osd_map decoding full epoch 454 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.393+0000 7fbdde7fc700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbdc8062c90 0x7fbdc8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbdc7fff700 20 client.37772.objecter dump_active .. 0 homeless 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v454) v1 ==== 54+0+16 (secure 0 0 0) 0x7fbdd0082220 con 0x7fbde00f7400 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbdc7fff700 10 client.37772.objecter ms_handle_connect 0x7fbdc8062c90 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbdde7fc700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbdc8062c90 0x7fbdc8065150 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 rx=0x7fbde00fa150 tx=0x7fbdd400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:28.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 10 client.37772.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 1 2024-02-23T00:37:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 20 client.37772.objecter _get_session osd=-1 returning homeless 2024-02-23T00:37:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 10 client.37772.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=455}) v3 -- 0x7fbde005ee80 con 0x7fbde00f7400 2024-02-23T00:37:28.878 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:37:28.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbde0106120 con 0x7fbde00f7400 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=454) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdd00502f0 con 0x7fbde00f7400 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbddd7fa700 10 client.37772.objecter _check_command_map_dne tid 1 current 454 map_dne_bound 454 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbddd7fa700 10 client.37772.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbddd7fa700 15 client.37772.objecter _session_command_op_remove -1 1 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 10 client.37772.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 2 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6805/685354171 conn(0x7fbde0061870 0x7fbde00273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0061700 osd=1 3 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0061700 osd=1 3 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0061700 osd=1 3 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 2 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0061700 osd=1 3 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 1 2 2024-02-23T00:37:28.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0061700 osd=1 4 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 10 client.37772.objecter _send_command 2 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde0059e00 con 0x7fbde0061870 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.394+0000 7fbddf7fe700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6805/685354171 conn(0x7fbde0061870 0x7fbde00273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.395+0000 7fbdc7fff700 10 client.37772.objecter ms_handle_connect 0x7fbde0061870 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.395+0000 7fbddf7fe700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6805/685354171 conn(0x7fbde0061870 0x7fbde00273c0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.395+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde0059e00 con 0x7fbde0061870 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.395+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.395+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 2 = system:0 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.395+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0061700 osd=1 4 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.395+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 1 2 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 10 client.37772.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 3 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0061700 osd=1 3 2024-02-23T00:37:28.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0061700 osd=1 3 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0061700 osd=1 3 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 3 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0061700 osd=1 3 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 1 3 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0061700 osd=1 4 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 10 client.37772.objecter _send_command 3 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.410+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbde00278b0 con 0x7fbde0061870 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbde00278b0 con 0x7fbde0061870 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0061700 osd=1 4 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 1 3 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 10 client.37772.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:28.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 4 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6809/3605394228 conn(0x7fbde00594d0 0x7fbde011b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0059360 osd=2 3 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0059360 osd=2 3 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0059360 osd=2 3 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 4 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0059360 osd=2 3 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 2 4 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0059360 osd=2 4 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 10 client.37772.objecter _send_command 4 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.412+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde0028700 con 0x7fbde00594d0 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.413+0000 7fbddeffd700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6809/3605394228 conn(0x7fbde00594d0 0x7fbde011b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.413+0000 7fbdc7fff700 10 client.37772.objecter ms_handle_connect 0x7fbde00594d0 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.413+0000 7fbddeffd700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6809/3605394228 conn(0x7fbde00594d0 0x7fbde011b260 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:28.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.413+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde0028700 con 0x7fbde00594d0 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.413+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.413+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 4 = system:0 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.413+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0059360 osd=2 4 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.413+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 2 4 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 10 client.37772.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 5 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0059360 osd=2 3 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0059360 osd=2 3 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0059360 osd=2 3 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 5 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0059360 osd=2 3 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 2 5 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0059360 osd=2 4 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 10 client.37772.objecter _send_command 5 2024-02-23T00:37:28.883 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.428+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbde0027cc0 con 0x7fbde00594d0 2024-02-23T00:37:28.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbde0027cc0 con 0x7fbde00594d0 2024-02-23T00:37:28.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:28.884 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:37:28.884 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:37:28.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.884 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0059360 osd=2 4 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 2 5 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 10 client.37772.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 6 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6813/128059790 conn(0x7fbde011c8d0 0x7fbde011ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0100190 osd=3 3 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0100190 osd=3 3 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0100190 osd=3 3 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 6 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0100190 osd=3 3 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 3 6 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0100190 osd=3 4 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 10 client.37772.objecter _send_command 6 2024-02-23T00:37:28.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde0121910 con 0x7fbde011c8d0 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.430+0000 7fbdde7fc700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6813/128059790 conn(0x7fbde011c8d0 0x7fbde011ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.431+0000 7fbdc7fff700 10 client.37772.objecter ms_handle_connect 0x7fbde011c8d0 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.431+0000 7fbdde7fc700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.87:6813/128059790 conn(0x7fbde011c8d0 0x7fbde011ecc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.431+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde0121910 con 0x7fbde011c8d0 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.431+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.431+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 6 = system:0 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.431+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0100190 osd=3 4 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.431+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 3 6 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 10 client.37772.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 7 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0100190 osd=3 3 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0100190 osd=3 3 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0100190 osd=3 3 2024-02-23T00:37:28.886 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 7 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0100190 osd=3 3 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 3 7 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0100190 osd=3 4 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 10 client.37772.objecter _send_command 7 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.446+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbde00ffc60 con 0x7fbde011c8d0 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbde00ffc60 con 0x7fbde011c8d0 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0100190 osd=3 4 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 3 7 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 10 client.37772.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 8 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6800/4175844757 conn(0x7fbde0120330 0x7fbde0120770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde011bf20 osd=4 3 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde011bf20 osd=4 3 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde011bf20 osd=4 3 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 8 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde011bf20 osd=4 3 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 4 8 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde011bf20 osd=4 4 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 10 client.37772.objecter _send_command 8 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde011c6f0 con 0x7fbde0120330 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.448+0000 7fbddf7fe700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6800/4175844757 conn(0x7fbde0120330 0x7fbde0120770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.449+0000 7fbdc7fff700 10 client.37772.objecter ms_handle_connect 0x7fbde0120330 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.449+0000 7fbddf7fe700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6800/4175844757 conn(0x7fbde0120330 0x7fbde0120770 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.449+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde011c6f0 con 0x7fbde0120330 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.449+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.449+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 8 = system:0 2024-02-23T00:37:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.449+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde011bf20 osd=4 4 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.449+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 4 8 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 10 client.37772.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 9 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde011bf20 osd=4 3 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde011bf20 osd=4 3 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde011bf20 osd=4 3 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 9 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde011bf20 osd=4 3 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 4 9 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde011bf20 osd=4 4 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 10 client.37772.objecter _send_command 9 2024-02-23T00:37:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.464+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbde01214b0 con 0x7fbde0120330 2024-02-23T00:37:28.890 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:37:28.890 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:37:28.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbde01214b0 con 0x7fbde0120330 2024-02-23T00:37:28.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:28.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde011bf20 osd=4 4 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 4 9 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 10 client.37772.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 10 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6804/2550013795 conn(0x7fbde0123a90 0x7fbde0125e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0120ce0 osd=5 3 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0120ce0 osd=5 3 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0120ce0 osd=5 3 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 10 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0120ce0 osd=5 3 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 5 10 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0120ce0 osd=5 4 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 10 client.37772.objecter _send_command 10 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.466+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde0121420 con 0x7fbde0123a90 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.467+0000 7fbddeffd700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6804/2550013795 conn(0x7fbde0123a90 0x7fbde0125e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.467+0000 7fbddeffd700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6804/2550013795 conn(0x7fbde0123a90 0x7fbde0125e80 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.467+0000 7fbdc7fff700 10 client.37772.objecter ms_handle_connect 0x7fbde0123a90 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.468+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde0121420 con 0x7fbde0123a90 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.468+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.468+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 10 = system:0 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.468+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0120ce0 osd=5 4 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.468+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 5 10 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 10 client.37772.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 11 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0120ce0 osd=5 3 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0120ce0 osd=5 3 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0120ce0 osd=5 3 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 11 2024-02-23T00:37:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0120ce0 osd=5 3 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 5 11 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0120ce0 osd=5 4 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 10 client.37772.objecter _send_command 11 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.482+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbde0121420 con 0x7fbde0123a90 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbde0121420 con 0x7fbde0123a90 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0120ce0 osd=5 4 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 5 11 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 10 client.37772.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 12 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6808/2509408498 conn(0x7fbde0126880 0x7fbde0126ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0121420 osd=6 3 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0121420 osd=6 3 2024-02-23T00:37:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0121420 osd=6 3 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 12 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0121420 osd=6 3 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 6 12 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0121420 osd=6 4 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 10 client.37772.objecter _send_command 12 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.484+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde0128910 con 0x7fbde0126880 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.485+0000 7fbdde7fc700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6808/2509408498 conn(0x7fbde0126880 0x7fbde0126ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.485+0000 7fbdde7fc700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6808/2509408498 conn(0x7fbde0126880 0x7fbde0126ce0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:37:28.894 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:37:28.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.485+0000 7fbdc7fff700 10 client.37772.objecter ms_handle_connect 0x7fbde0126880 2024-02-23T00:37:28.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.486+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde0128910 con 0x7fbde0126880 2024-02-23T00:37:28.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.486+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:37:28.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.486+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 12 = system:0 2024-02-23T00:37:28.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.486+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0121420 osd=6 4 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.486+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 6 12 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 10 client.37772.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 13 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0121420 osd=6 3 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0121420 osd=6 3 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0121420 osd=6 3 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 13 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0121420 osd=6 3 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 6 13 2024-02-23T00:37:28.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0121420 osd=6 4 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 10 client.37772.objecter _send_command 13 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.500+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbde0128910 con 0x7fbde0126880 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.502+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbde0128910 con 0x7fbde0126880 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.502+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.502+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.502+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0121420 osd=6 4 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.502+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 6 13 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 10 client.37772.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 14 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6812/373787000 conn(0x7fbde012b020 0x7fbde012b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0128910 osd=7 3 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0128910 osd=7 3 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0128910 osd=7 3 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 14 2024-02-23T00:37:28.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0128910 osd=7 3 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 7 14 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0128910 osd=7 4 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 10 client.37772.objecter _send_command 14 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbde012baa0 con 0x7fbde012b020 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbddf7fe700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6812/373787000 conn(0x7fbde012b020 0x7fbde012b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbdc7fff700 10 client.37772.objecter ms_handle_connect 0x7fbde012b020 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.503+0000 7fbddf7fe700 1 --2- 172.21.15.87:0/3784807438 >> v2:172.21.15.171:6812/373787000 conn(0x7fbde012b020 0x7fbde012b460 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.504+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbde012baa0 con 0x7fbde012b020 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.504+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.504+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 14 = system:0 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.504+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0128910 osd=7 4 2024-02-23T00:37:28.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.504+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 7 14 2024-02-23T00:37:28.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 10 client.37772.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:28.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign -1 15 2024-02-23T00:37:28.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0128910 osd=7 3 2024-02-23T00:37:28.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0128910 osd=7 3 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 20 client.37772.objecter _get_session s=0x7fbde0128910 osd=7 3 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_remove -1 15 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 20 client.37772.objecter get_session s=0x7fbde0128910 osd=7 3 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 15 client.37772.objecter _session_command_op_assign 7 15 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 20 client.37772.objecter put_session s=0x7fbde0128910 osd=7 4 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 10 client.37772.objecter _send_command 15 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.518+0000 7fbde52c9700 1 -- 172.21.15.87:0/3784807438 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbde012baa0 con 0x7fbde012b020 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.520+0000 7fbdc7fff700 1 -- 172.21.15.87:0/3784807438 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbde012baa0 con 0x7fbde012b020 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.520+0000 7fbdc7fff700 10 client.37772.objecter ms_dispatch 0x7fbde0001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.520+0000 7fbdc7fff700 10 client.37772.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.520+0000 7fbdc7fff700 20 client.37772.objecter put_session s=0x7fbde0128910 osd=7 4 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:28.520+0000 7fbdc7fff700 15 client.37772.objecter _session_command_op_remove 7 15 2024-02-23T00:37:28.900 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:37:28.901 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.901 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.901 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.901 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.901 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.901 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:37:28.901 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:37:29.087 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:29.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:29.191 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:29.218 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:29.238 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:29.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:29.342 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:29.369 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:29.649 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:27.676992+0000 mgr.y (mgr.24437) 425 : cluster [DBG] pgmap v684: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 48 KiB/s rd, 383 B/s wr, 85 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:29.649 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:27.869247+0000 osd.7 (osd.7) 233 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:37:29.649 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:27.870643+0000 osd.7 (osd.7) 234 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:37:29.649 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:28.350138+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:37:29.649 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:28.355442+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:28.393908+0000 mon.c (mon.2) 440 : audit [DBG] from='client.? 172.21.15.87:0/3784807438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:29.113378+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e455: 8 total, 7 up, 7 in 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:29.120989+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:29.121492+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:27.676992+0000 mgr.y (mgr.24437) 425 : cluster [DBG] pgmap v684: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 48 KiB/s rd, 383 B/s wr, 85 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:27.869247+0000 osd.7 (osd.7) 233 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:27.870643+0000 osd.7 (osd.7) 234 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:28.350138+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:28.355442+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:28.393908+0000 mon.c (mon.2) 440 : audit [DBG] from='client.? 172.21.15.87:0/3784807438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:29.650 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:29.113378+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e455: 8 total, 7 up, 7 in 2024-02-23T00:37:29.651 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:29.120989+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:29.651 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:29.121492+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:27.676992+0000 mgr.y (mgr.24437) 425 : cluster [DBG] pgmap v684: 238 pgs: 26 active+undersized+degraded, 64 active+undersized, 148 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 48 KiB/s rd, 383 B/s wr, 85 op/s; 120/1182 objects degraded (10.152%) 2024-02-23T00:37:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:27.869247+0000 osd.7 (osd.7) 233 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:37:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:27.870643+0000 osd.7 (osd.7) 234 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:37:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:28.350138+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:37:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:28.355442+0000 osd.6 (osd.6) 185 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:37:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:28.393908+0000 mon.c (mon.2) 440 : audit [DBG] from='client.? 172.21.15.87:0/3784807438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:29.113378+0000 mon.a (mon.0) 2673 : cluster [DBG] osdmap e455: 8 total, 7 up, 7 in 2024-02-23T00:37:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:29.120989+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:29.121492+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:29.919 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:29.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:30.022 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:30.050 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:30.350 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:37:30.350 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:37:30.351 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-23T00:37:30.351 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-23T00:37:30.898 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:30.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:31.003 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:31.031 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:31.123 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:29.677687+0000 mgr.y (mgr.24437) 426 : cluster [DBG] pgmap v687: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 49 KiB/s rd, 0 B/s wr, 82 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:37:31.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:30.114223+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:31.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:30.114331+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e456: 8 total, 7 up, 7 in 2024-02-23T00:37:31.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:30.121666+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:31.124 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:30.376519+0000 mon.a (mon.0) 2677 : cluster [WRN] Health check update: Degraded data redundancy: 120/1173 objects degraded (10.230%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:31.158 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:31.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:31.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:31.274 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:31.303 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:37:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:37:31.303 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:29.677687+0000 mgr.y (mgr.24437) 426 : cluster [DBG] pgmap v687: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 49 KiB/s rd, 0 B/s wr, 82 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:37:31.303 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:30.114223+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:31.304 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:30.114331+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e456: 8 total, 7 up, 7 in 2024-02-23T00:37:31.304 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:30.121666+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:31.304 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:30.376519+0000 mon.a (mon.0) 2677 : cluster [WRN] Health check update: Degraded data redundancy: 120/1173 objects degraded (10.230%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:31.305 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:31.323 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:31.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:31.427 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:31.452 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:31.477 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:31.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:31.578 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:29.677687+0000 mgr.y (mgr.24437) 426 : cluster [DBG] pgmap v687: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 49 KiB/s rd, 0 B/s wr, 82 op/s; 120/1173 objects degraded (10.230%) 2024-02-23T00:37:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:30.114223+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:30.114331+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e456: 8 total, 7 up, 7 in 2024-02-23T00:37:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:30.121666+0000 mon.b (mon.1) 376 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:30.376519+0000 mon.a (mon.0) 2677 : cluster [WRN] Health check update: Degraded data redundancy: 120/1173 objects degraded (10.230%), 26 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:31.606 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:31.660 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:31.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:31.764 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:31.792 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:31.796 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:37:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:37:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "complaint_time": 30, 2024-02-23T00:37:31.796 INFO:teuthology.orchestra.run.smithi087.stdout: "num_blocked_ops": 0 2024-02-23T00:37:31.796 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:37:31.811 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:31.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:31.914 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:31.943 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:31.962 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:31.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:32.066 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:32.093 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:32.146 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:32.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:32.266 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:32.291 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:32.318 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:32.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:32.421 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:30.967941+0000 mgr.y (mgr.24437) 427 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:32.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:31.117421+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e457: 8 total, 7 up, 7 in 2024-02-23T00:37:32.446 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:37:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:32] "GET /metrics HTTP/1.1" 200 188509 "" "Prometheus/2.33.4" 2024-02-23T00:37:32.446 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:30.967941+0000 mgr.y (mgr.24437) 427 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:32.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:31.117421+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e457: 8 total, 7 up, 7 in 2024-02-23T00:37:32.447 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:30.967941+0000 mgr.y (mgr.24437) 427 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:32.447 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:31.117421+0000 mon.a (mon.0) 2678 : cluster [DBG] osdmap e457: 8 total, 7 up, 7 in 2024-02-23T00:37:32.448 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:37:32.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:37:32.696 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:32.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:32.808 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:32.835 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:33.033 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:33.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:33.059 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-23T00:37:33.060 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:33.137 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:33.165 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:33.184 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:33.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:33.287 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:33.314 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:33.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:31.678403+0000 mgr.y (mgr.24437) 428 : cluster [DBG] pgmap v690: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123/1188 objects degraded (10.354%) 2024-02-23T00:37:33.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:32.130560+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e458: 8 total, 7 up, 7 in 2024-02-23T00:37:33.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:31.678403+0000 mgr.y (mgr.24437) 428 : cluster [DBG] pgmap v690: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123/1188 objects degraded (10.354%) 2024-02-23T00:37:33.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:32.130560+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e458: 8 total, 7 up, 7 in 2024-02-23T00:37:33.334 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:33.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:33.437 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:31.678403+0000 mgr.y (mgr.24437) 428 : cluster [DBG] pgmap v690: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123/1188 objects degraded (10.354%) 2024-02-23T00:37:33.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:32.130560+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e458: 8 total, 7 up, 7 in 2024-02-23T00:37:33.465 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:33.484 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:33.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:33.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:33.588 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:33.615 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:33.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:37:33.941 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:33.965 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:33.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:34.070 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:34.100 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:37:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:37:34.102 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:34.123 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:34.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:34.228 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:34.255 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:34.577 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:34.604 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:34.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:34.608 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:37:34.708 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:34.734 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:35.225 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:33.641491+0000 osd.4 (osd.4) 213 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:35.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:33.641639+0000 osd.4 (osd.4) 214 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:35.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:33.678838+0000 mgr.y (mgr.24437) 429 : cluster [DBG] pgmap v692: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s; 123/1188 objects degraded (10.354%) 2024-02-23T00:37:35.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:34.140334+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e459: 8 total, 7 up, 7 in 2024-02-23T00:37:35.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:33.641491+0000 osd.4 (osd.4) 213 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:35.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:33.641639+0000 osd.4 (osd.4) 214 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:35.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:33.678838+0000 mgr.y (mgr.24437) 429 : cluster [DBG] pgmap v692: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s; 123/1188 objects degraded (10.354%) 2024-02-23T00:37:35.226 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:34.140334+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e459: 8 total, 7 up, 7 in 2024-02-23T00:37:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:33.641491+0000 osd.4 (osd.4) 213 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:37:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:33.641639+0000 osd.4 (osd.4) 214 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:37:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:33.678838+0000 mgr.y (mgr.24437) 429 : cluster [DBG] pgmap v692: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s; 123/1188 objects degraded (10.354%) 2024-02-23T00:37:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:34.140334+0000 mon.a (mon.0) 2680 : cluster [DBG] osdmap e459: 8 total, 7 up, 7 in 2024-02-23T00:37:35.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.665+0000 7fc214ea6700 1 -- 172.21.15.87:0/3987720581 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 msgr2=0x7fc210101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:35.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.665+0000 7fc214ea6700 1 --2- 172.21.15.87:0/3987720581 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 0x7fc210101c30 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 rx=0x7fc200004870 tx=0x7fc200009cb0).stop 2024-02-23T00:37:35.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.665+0000 7fc214ea6700 1 -- 172.21.15.87:0/3987720581 shutdown_connections 2024-02-23T00:37:35.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.665+0000 7fc214ea6700 1 --2- 172.21.15.87:0/3987720581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc210102120 0x7fc210104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:35.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.665+0000 7fc214ea6700 1 --2- 172.21.15.87:0/3987720581 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2100f8bc0 0x7fc2100f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:35.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.665+0000 7fc214ea6700 1 --2- 172.21.15.87:0/3987720581 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 0x7fc210101c30 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:35.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.665+0000 7fc214ea6700 1 -- 172.21.15.87:0/3987720581 >> 172.21.15.87:0/3987720581 conn(0x7fc2100f2710 msgr2=0x7fc2100f4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:35.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.665+0000 7fc214ea6700 1 -- 172.21.15.87:0/3987720581 shutdown_connections 2024-02-23T00:37:35.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.665+0000 7fc214ea6700 1 -- 172.21.15.87:0/3987720581 wait complete. 2024-02-23T00:37:35.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 librados: starting msgr at 2024-02-23T00:37:35.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 librados: starting objecter 2024-02-23T00:37:35.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 Processor -- start 2024-02-23T00:37:35.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 -- start start 2024-02-23T00:37:35.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 librados: setting wanted keys 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 librados: calling monclient init 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2100f8bc0 0x7fc21010e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 0x7fc21010e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc210102120 0x7fc210108330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc210108820 con 0x7fc210102120 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc210108990 con 0x7fc2100f8bc0 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc214ea6700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc210108b50 con 0x7fc2100f9520 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.666+0000 7fc20dd9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 0x7fc21010e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20ed9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc210102120 0x7fc210108330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20dd9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 0x7fc21010e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44932/0 (socket says 172.21.15.87:44932) 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20dd9b700 1 -- 172.21.15.87:0/351015075 learned_addr learned my addr 172.21.15.87:0/351015075 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:35.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20e59c700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2100f8bc0 0x7fc21010e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20dd9b700 1 -- 172.21.15.87:0/351015075 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2100f8bc0 msgr2=0x7fc21010e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20dd9b700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2100f8bc0 0x7fc21010e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20dd9b700 1 -- 172.21.15.87:0/351015075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc210102120 msgr2=0x7fc210108330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20dd9b700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc210102120 0x7fc210108330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20dd9b700 1 -- 172.21.15.87:0/351015075 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8005040 con 0x7fc2100f9520 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20ed9d700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc210102120 0x7fc210108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20e59c700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2100f8bc0 0x7fc21010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc20dd9b700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 0x7fc21010e680 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7fc2000043a0 tx=0x7fc20000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc2077fe700 10 client.?.objecter ms_handle_connect 0x7fc2100f9520 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc2077fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc2077fe700 1 -- 172.21.15.87:0/351015075 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20000e070 con 0x7fc2100f9520 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc200005040 con 0x7fc2100f9520 2024-02-23T00:37:35.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc2077fe700 1 -- 172.21.15.87:0/351015075 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc200009710 con 0x7fc2100f9520 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc2077fe700 1 -- 172.21.15.87:0/351015075 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc200019070 con 0x7fc2100f9520 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc214ea6700 10 client.37778.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc210108ff0 con 0x7fc2100f9520 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.667+0000 7fc214ea6700 1 librados: init done 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.668+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2101091e0 con 0x7fc2100f9520 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.668+0000 7fc2077fe700 1 -- 172.21.15.87:0/351015075 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc2000283f0 con 0x7fc2100f9520 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.669+0000 7fc2077fe700 1 --2- 172.21.15.87:0/351015075 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc1fc062c90 0x7fc1fc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.669+0000 7fc2077fe700 1 -- 172.21.15.87:0/351015075 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(459..459 src has 173..459) v4 ==== 8150+0+0 (secure 0 0 0) 0x7fc20007dc60 con 0x7fc2100f9520 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.669+0000 7fc2077fe700 10 client.37778.objecter ms_dispatch 0x7fc210001550 osd_map(459..459 src has 173..459) v4 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.669+0000 7fc2077fe700 3 client.37778.objecter handle_osd_map got epochs [459,459] > 0 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.669+0000 7fc2077fe700 3 client.37778.objecter handle_osd_map decoding full epoch 459 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.669+0000 7fc20e59c700 1 --2- 172.21.15.87:0/351015075 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc1fc062c90 0x7fc1fc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:35.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.669+0000 7fc2077fe700 20 client.37778.objecter dump_active .. 0 homeless 2024-02-23T00:37:35.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.669+0000 7fc2077fe700 10 client.37778.objecter ms_handle_connect 0x7fc1fc062c90 2024-02-23T00:37:35.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.669+0000 7fc20e59c700 1 --2- 172.21.15.87:0/351015075 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc1fc062c90 0x7fc1fc065150 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7fc1f80096d0 tx=0x7fc1f800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:35.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.673+0000 7fc2077fe700 1 -- 172.21.15.87:0/351015075 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc200051200 con 0x7fc2100f9520 2024-02-23T00:37:35.685 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:35.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:35.789 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:35.818 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:35.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.851+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fc210107150 con 0x7fc2100f9520 2024-02-23T00:37:35.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.853+0000 7fc2077fe700 1 -- 172.21.15.87:0/351015075 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v459) v1 ==== 81+0+48775 (secure 0 0 0) 0x7fc200050df0 con 0x7fc2100f9520 2024-02-23T00:37:35.856 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:37:35.856 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:37:35.856 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 459, 2024-02-23T00:37:35.856 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:37:35.856 INFO:teuthology.orchestra.run.smithi087.stdout: "created": "2024-02-23T00:21:28.861318+0000", 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "modified": "2024-02-23T00:37:34.128180+0000", 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "last_up_change": "2024-02-23T00:36:36.207346+0000", 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "last_in_change": "2024-02-23T00:34:40.063161+0000", 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_num": 5801984, 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_set": [ 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "noscrub", 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "pglog_hardlimit", 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "purged_snapdirs", 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "recovery_deletes", 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "sortbitwise" 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_version": 108, 2024-02-23T00:37:35.857 INFO:teuthology.orchestra.run.smithi087.stdout: "full_ratio": 0.94999998807907104, 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_snapshot": "", 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_max": 217, 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd": 8, 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "require_min_compat_client": "luminous", 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "min_compat_client": "luminous", 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "require_osd_release": "pacific", 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "pools": [ 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "device_health_metrics", 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:25:09.208309+0000", 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:37:35.858 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 1, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 1, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 1, 2024-02-23T00:37:35.859 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 1, 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 1, 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "68", 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:37:35.860 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:37:35.861 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:37:35.862 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_max": 32, 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_min": 1 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_devicehealth": {} 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.863 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 2, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": ".rgw.root", 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.562186+0000", 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:37:35.864 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "174", 2024-02-23T00:37:35.865 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:37:35.866 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:37:35.867 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 3, 2024-02-23T00:37:35.868 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "iscsi", 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.989206+0000", 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 13, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 13, 2024-02-23T00:37:35.869 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 13, 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 13, 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 13, 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "137", 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "135", 2024-02-23T00:37:35.870 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:37:35.871 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:37:35.872 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd": {} 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 4, 2024-02-23T00:37:35.873 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.log", 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:17.282440+0000", 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 22, 2024-02-23T00:37:35.874 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 22, 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 22, 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 22, 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 22, 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "4.16", 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 371, 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 370, 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 370, 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "63'21", 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "62'12" 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "371", 2024-02-23T00:37:35.875 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "371", 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "369", 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:37:35.876 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:37:35.877 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.878 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 5, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.control", 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:19.262164+0000", 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:37:35.879 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 42, 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 42, 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 42, 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 42, 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 42, 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "377", 2024-02-23T00:37:35.880 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "375", 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:37:35.881 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:37:35.882 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.883 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 6, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.meta", 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:21.319951+0000", 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:37:35.884 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "62", 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:37:35.885 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:37:35.886 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:37:35.887 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_bias": 4 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:37:35.888 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 22, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:46.071806+0000", 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:37:35.889 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:37:35.890 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:37:35.891 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:37:35.892 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:37:35.893 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-6", 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:31:04.375891+0000", 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:37:35.894 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.895 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "458", 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "388", 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "388", 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "388", 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 34, 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 458, 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:37:35.896 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:37:35.897 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "fingerprint_algorithm": "sha1" 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:37:35.898 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 217, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-71", 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:37:28.370450+0000", 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:37:35.899 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:37:35.900 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "457", 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:37:35.901 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:37:35.902 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:37:35.903 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7", 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 0, 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 8, 2024-02-23T00:37:35.904 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 407, 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 425, 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.905 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6802", 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6804", 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.906 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6803", 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6801/2153191863", 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6802/2153191863", 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6804/2153191863", 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6803/2153191863", 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:37:35.907 INFO:teuthology.orchestra.run.smithi087.stdout: "exists" 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "85525ce8-38d5-4840-8057-547cb176c859", 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 13, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 455, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:37:35.908 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6805", 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6806", 2024-02-23T00:37:35.909 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6808", 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.910 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6807", 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6805/685354171", 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6806/685354171", 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6808/685354171", 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6807/685354171", 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:37:35.911 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa", 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 18, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 455, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:37:35.912 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6809", 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6810", 2024-02-23T00:37:35.913 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6812", 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.914 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6811", 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6809/3605394228", 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6810/3605394228", 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6812/3605394228", 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6811/3605394228", 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.915 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a", 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 0, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 0, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 26, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 314, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:37:35.916 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6813", 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6814", 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.917 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6816", 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.918 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6815", 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6813/128059790", 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6814/128059790", 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6816/128059790", 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6815/128059790", 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.919 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c", 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.9473876953125, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 31, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 455, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.920 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6800", 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6801", 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.921 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6803", 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6802", 2024-02-23T00:37:35.922 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6800/4175844757", 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6801/4175844757", 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4175844757", 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4175844757", 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.923 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d", 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.8517608642578125, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 38, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 455, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.924 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6804", 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6805", 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.925 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6807", 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6806", 2024-02-23T00:37:35.926 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6804/2550013795", 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6805/2550013795", 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2550013795", 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2550013795", 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:37:35.927 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01", 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 44, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 455, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.928 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6808", 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6809", 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.929 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6811", 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.930 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6810", 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6808/2509408498", 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6809/2509408498", 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6811/2509408498", 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6810/2509408498", 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.931 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939", 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 51, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 455, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:37:35.932 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6812", 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6813", 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.933 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6815", 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:37:35.934 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6814", 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6812/373787000", 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6813/373787000", 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6815/373787000", 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6814/373787000", 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.935 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_xinfo": [ 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:36:24.026593+0000", 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 424 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:37:35.936 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:37:07.064235+0000", 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:37:05.352692+0000", 2024-02-23T00:37:35.937 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 65536, 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:37:16.003466+0000", 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:37:35.938 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:37:33.641642+0000", 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:37:13.629277+0000", 2024-02-23T00:37:35.939 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:37:09.590018+0000", 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.940 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:37:20.452867+0000", 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap": [], 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap_items": [ 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.c", 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.941 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 5, 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 0 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.18", 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 1 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.942 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.1c", 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 4 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.2", 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 0 2024-02-23T00:37:35.943 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.13", 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "5.19", 2024-02-23T00:37:35.944 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 6 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "22.a", 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 0 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.945 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "22.1a", 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 5 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "63.0", 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:37:35.946 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 7 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "63.1d", 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 6 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_temp": [], 2024-02-23T00:37:35.947 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_temp": [], 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "blocklist": { 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1121018082": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/2138166229": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/27794119": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/1675126043": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/3767016525": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/969918521": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/795095363": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2273151327": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1685406414": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2093733106": "2024-02-24T00:21:58.144574+0000" 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "range_blocklist": {}, 2024-02-23T00:37:35.948 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profiles": { 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "default": { 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van" 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps_queue": [ 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 7, 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.949 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.950 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 21, 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.951 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 61, 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:37:35.952 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 29, 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 78, 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.953 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 86, 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:37:35.954 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.955 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 95, 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 4, 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 4 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 107, 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.956 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 112, 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.957 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 125, 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.958 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 10, 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 133, 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.959 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 154, 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 14, 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 164, 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 17, 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.960 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 177, 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 20, 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: "new_removed_snaps": [], 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: "new_purged_snaps": [], 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_node_flags": {}, 2024-02-23T00:37:35.961 INFO:teuthology.orchestra.run.smithi087.stdout: "device_class_flags": {}, 2024-02-23T00:37:35.962 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode": { 2024-02-23T00:37:35.962 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_enabled": false, 2024-02-23T00:37:35.962 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_bucket_count": 0, 2024-02-23T00:37:35.962 INFO:teuthology.orchestra.run.smithi087.stdout: "degraded_stretch_mode": 0, 2024-02-23T00:37:35.962 INFO:teuthology.orchestra.run.smithi087.stdout: "recovering_stretch_mode": 0, 2024-02-23T00:37:35.962 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_bucket": 0 2024-02-23T00:37:35.962 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:37:35.962 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:37:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 10 librados: watch_flush enter 2024-02-23T00:37:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 10 librados: watch_flush exit 2024-02-23T00:37:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 20 client.37778.objecter shutdown clearing up homeless session... 2024-02-23T00:37:35.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 10 client.37778.objecter successfully canceled tick 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc1fc062c90 msgr2=0x7fc1fc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 --2- 172.21.15.87:0/351015075 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc1fc062c90 0x7fc1fc065150 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 rx=0x7fc1f80096d0 tx=0x7fc1f800b040).stop 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 msgr2=0x7fc21010e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 0x7fc21010e680 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 rx=0x7fc2000043a0 tx=0x7fc20000b040).stop 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 shutdown_connections 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 --2- 172.21.15.87:0/351015075 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc1fc062c90 0x7fc1fc065150 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc210102120 0x7fc210108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2100f8bc0 0x7fc21010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 --2- 172.21.15.87:0/351015075 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2100f9520 0x7fc21010e680 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 >> 172.21.15.87:0/351015075 conn(0x7fc2100f2710 msgr2=0x7fc2100f3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 shutdown_connections 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 -- 172.21.15.87:0/351015075 wait complete. 2024-02-23T00:37:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:35.855+0000 7fc214ea6700 1 librados: shutdown 2024-02-23T00:37:36.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:35.853486+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? 172.21.15.87:0/351015075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:37:36.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:35.853486+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? 172.21.15.87:0/351015075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:37:36.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:35.853486+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? 172.21.15.87:0/351015075' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:37:37.368 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:37.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:37.382 INFO:tasks.thrashosds.thrasher:j is {'epoch': 459, 'fsid': '66db207c-d1e1-11ee-95c0-87774f69a715', 'created': '2024-02-23T00:21:28.861318+0000', 'modified': '2024-02-23T00:37:34.128180+0000', 'last_up_change': '2024-02-23T00:36:36.207346+0000', 'last_in_change': '2024-02-23T00:34:40.063161+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 108, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 217, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '135', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '4.16', 'ready_epoch': 371, 'last_epoch_started': 370, 'last_epoch_clean': 370, 'source_version': "63'21", 'target_version': "62'12"}, 'last_change': '371', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '371', 'last_force_op_resend_preluminous': '369', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '377', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '375', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 22, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-23T00:30:46.071806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 63, 'pool_name': 'test-rados-api-smithi087-82091-6', 'create_time': '2024-02-23T00:31:04.375891+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '458', 'last_force_op_resend': '388', 'last_force_op_resend_prenautilus': '388', 'last_force_op_resend_preluminous': '388', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 34, 'snap_epoch': 458, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}}, {'pool': 217, 'pool_name': 'test-rados-api-smithi087-82091-71', 'create_time': '2024-02-23T00:37:28.370450+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '457', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e3549bad-2891-47b9-bf77-132cadc20bf7', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 407, 'down_at': 425, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6801', 'nonce': 2153191863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6802', 'nonce': 2153191863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6804', 'nonce': 2153191863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6803', 'nonce': 2153191863}]}, 'public_addr': '172.21.15.87:6801/2153191863', 'cluster_addr': '172.21.15.87:6802/2153191863', 'heartbeat_back_addr': '172.21.15.87:6804/2153191863', 'heartbeat_front_addr': '172.21.15.87:6803/2153191863', 'state': ['exists']}, {'osd': 1, 'uuid': '85525ce8-38d5-4840-8057-547cb176c859', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 455, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6805', 'nonce': 685354171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6806', 'nonce': 685354171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6808', 'nonce': 685354171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6807', 'nonce': 685354171}]}, 'public_addr': '172.21.15.87:6805/685354171', 'cluster_addr': '172.21.15.87:6806/685354171', 'heartbeat_back_addr': '172.21.15.87:6808/685354171', 'heartbeat_front_addr': '172.21.15.87:6807/685354171', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 455, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6809', 'nonce': 3605394228}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6810', 'nonce': 3605394228}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6812', 'nonce': 3605394228}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6811', 'nonce': 3605394228}]}, 'public_addr': '172.21.15.87:6809/3605394228', 'cluster_addr': '172.21.15.87:6810/3605394228', 'heartbeat_back_addr': '172.21.15.87:6812/3605394228', 'heartbeat_front_addr': '172.21.15.87:6811/3605394228', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6813', 'nonce': 128059790}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6814', 'nonce': 128059790}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6816', 'nonce': 128059790}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6815', 'nonce': 128059790}]}, 'public_addr': '172.21.15.87:6813/128059790', 'cluster_addr': '172.21.15.87:6814/128059790', 'heartbeat_back_addr': '172.21.15.87:6816/128059790', 'heartbeat_front_addr': '172.21.15.87:6815/128059790', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c3734a04-7343-4251-9b5b-bedb9e8b882c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9473876953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 455, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6800', 'nonce': 4175844757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6801', 'nonce': 4175844757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6803', 'nonce': 4175844757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6802', 'nonce': 4175844757}]}, 'public_addr': '172.21.15.171:6800/4175844757', 'cluster_addr': '172.21.15.171:6801/4175844757', 'heartbeat_back_addr': '172.21.15.171:6803/4175844757', 'heartbeat_front_addr': '172.21.15.171:6802/4175844757', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fe86232d-96b9-4cbf-9cac-373286fa103d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8517608642578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 455, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6804', 'nonce': 2550013795}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6805', 'nonce': 2550013795}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6807', 'nonce': 2550013795}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6806', 'nonce': 2550013795}]}, 'public_addr': '172.21.15.171:6804/2550013795', 'cluster_addr': '172.21.15.171:6805/2550013795', 'heartbeat_back_addr': '172.21.15.171:6807/2550013795', 'heartbeat_front_addr': '172.21.15.171:6806/2550013795', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 455, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6808', 'nonce': 2509408498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6809', 'nonce': 2509408498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6811', 'nonce': 2509408498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6810', 'nonce': 2509408498}]}, 'public_addr': '172.21.15.171:6808/2509408498', 'cluster_addr': '172.21.15.171:6809/2509408498', 'heartbeat_back_addr': '172.21.15.171:6811/2509408498', 'heartbeat_front_addr': '172.21.15.171:6810/2509408498', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '29ddcf97-369a-4a2c-b3e2-142e84ccf939', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 455, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6812', 'nonce': 373787000}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6813', 'nonce': 373787000}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6815', 'nonce': 373787000}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6814', 'nonce': 373787000}]}, 'public_addr': '172.21.15.171:6812/373787000', 'cluster_addr': '172.21.15.171:6813/373787000', 'heartbeat_back_addr': '172.21.15.171:6815/373787000', 'heartbeat_front_addr': '172.21.15.171:6814/373787000', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:36:24.026593+0000', 'dead_epoch': 424}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:37:07.064235+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:37:05.352692+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-23T00:37:16.003466+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:37:33.641642+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:37:13.629277+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:37:09.590018+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:37:20.452867+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.c', 'mappings': [{'from': 5, 'to': 0}]}, {'pgid': '2.18', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '2.1c', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.2', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '22.a', 'mappings': [{'from': 7, 'to': 0}]}, {'pgid': '22.1a', 'mappings': [{'from': 7, 'to': 5}]}, {'pgid': '63.0', 'mappings': [{'from': 1, 'to': 7}]}, {'pgid': '63.1d', 'mappings': [{'from': 1, 'to': 6}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.87:6800/1121018082': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/2138166229': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/27794119': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/1675126043': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/3767016525': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/969918521': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/795095363': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2273151327': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/1685406414': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2093733106': '2024-02-24T00:21:58.144574+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 21, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 61, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 63, 'snaps': [{'begin': 29, 'length': 1}]}, {'pool': 78, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 86, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 177, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-23T00:37:37.382 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:37:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:35.679771+0000 mgr.y (mgr.24437) 430 : cluster [DBG] pgmap v694: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:37.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:36.137970+0000 mon.a (mon.0) 2681 : cluster [WRN] Health check update: Degraded data redundancy: 125/1194 objects degraded (10.469%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:37.449 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:35.679771+0000 mgr.y (mgr.24437) 430 : cluster [DBG] pgmap v694: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:37.449 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:36.137970+0000 mon.a (mon.0) 2681 : cluster [WRN] Health check update: Degraded data redundancy: 125/1194 objects degraded (10.469%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:37.449 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:35.679771+0000 mgr.y (mgr.24437) 430 : cluster [DBG] pgmap v694: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:37.450 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:36.137970+0000 mon.a (mon.0) 2681 : cluster [WRN] Health check update: Degraded data redundancy: 125/1194 objects degraded (10.469%), 28 pgs degraded (PG_DEGRADED) 2024-02-23T00:37:37.473 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:37.505 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:37.526 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:37.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:37.631 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:37.658 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:38.735 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:38.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:38.755 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:38.840 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:38.872 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:38.894 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:38.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:38.997 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:39.027 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:39.047 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:39.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:39.151 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:39.179 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:39.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:37.680633+0000 mgr.y (mgr.24437) 431 : cluster [DBG] pgmap v695: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 311 B/s wr, 3 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:39.240 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:37.680633+0000 mgr.y (mgr.24437) 431 : cluster [DBG] pgmap v695: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 311 B/s wr, 3 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:39.251 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:39.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:39.355 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:39.386 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:39.406 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:39.408 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:37.680633+0000 mgr.y (mgr.24437) 431 : cluster [DBG] pgmap v695: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 311 B/s wr, 3 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:39.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:39.510 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:39.541 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:39.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.465+0000 7efdebc2f700 1 -- 172.21.15.87:0/2024627467 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 msgr2=0x7efde4103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.465+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2024627467 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 0x7efde4103a20 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 rx=0x7efddc009be0 tx=0x7efddc004d10).stop 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.465+0000 7efdebc2f700 1 -- 172.21.15.87:0/2024627467 shutdown_connections 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.465+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2024627467 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 0x7efde4103a20 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.465+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2024627467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efde40fcbb0 0x7efde40fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.465+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2024627467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efde40fd510 0x7efde40f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.465+0000 7efdebc2f700 1 -- 172.21.15.87:0/2024627467 >> 172.21.15.87:0/2024627467 conn(0x7efde40f2820 msgr2=0x7efde40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.465+0000 7efdebc2f700 1 -- 172.21.15.87:0/2024627467 shutdown_connections 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.465+0000 7efdebc2f700 1 -- 172.21.15.87:0/2024627467 wait complete. 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 librados: starting msgr at 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 librados: starting objecter 2024-02-23T00:37:39.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 Processor -- start 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 -- start start 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 librados: setting wanted keys 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 librados: calling monclient init 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 0x7efde41090d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efde40fcbb0 0x7efde41095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efde40fd510 0x7efde4109b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde4114290 con 0x7efde40fd510 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde4114400 con 0x7efde40fcbb0 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdebc2f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde41145c0 con 0x7efde40f7400 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efde99cb700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 0x7efde41090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efdea1cc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efde40fd510 0x7efde4109b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efde99cb700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 0x7efde41090d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44952/0 (socket says 172.21.15.87:44952) 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.466+0000 7efde99cb700 1 -- 172.21.15.87:0/2562740791 learned_addr learned my addr 172.21.15.87:0/2562740791 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efde99cb700 1 -- 172.21.15.87:0/2562740791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efde40fcbb0 msgr2=0x7efde41095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:39.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efde99cb700 1 --2- 172.21.15.87:0/2562740791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efde40fcbb0 0x7efde41095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efde99cb700 1 -- 172.21.15.87:0/2562740791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efde40fd510 msgr2=0x7efde4109b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efde99cb700 1 --2- 172.21.15.87:0/2562740791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efde40fd510 0x7efde4109b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efde99cb700 1 -- 172.21.15.87:0/2562740791 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd4005040 con 0x7efde40f7400 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efdea1cc700 1 --2- 172.21.15.87:0/2562740791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efde40fd510 0x7efde4109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efde99cb700 1 --2- 172.21.15.87:0/2562740791 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 0x7efde41090d0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7efdd40048f0 tx=0x7efdd4009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efddaffd700 10 client.?.objecter ms_handle_connect 0x7efde40f7400 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efddaffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efddaffd700 1 -- 172.21.15.87:0/2562740791 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4007b00 con 0x7efde40f7400 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efddc0097c0 con 0x7efde40f7400 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efddaffd700 1 -- 172.21.15.87:0/2562740791 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdd400f040 con 0x7efde40f7400 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efddaffd700 1 -- 172.21.15.87:0/2562740791 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4014a00 con 0x7efde40f7400 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efdebc2f700 10 client.37784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde4114c20 con 0x7efde40f7400 2024-02-23T00:37:39.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.467+0000 7efdebc2f700 1 librados: init done 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.468+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde405fa60 con 0x7efde40f7400 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.468+0000 7efddaffd700 1 -- 172.21.15.87:0/2562740791 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7efdd400e580 con 0x7efde40f7400 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.468+0000 7efddaffd700 1 --2- 172.21.15.87:0/2562740791 >> v2:172.21.15.87:6800/1231597463 conn(0x7efdd0062c90 0x7efdd0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.469+0000 7efddaffd700 1 -- 172.21.15.87:0/2562740791 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(459..459 src has 173..459) v4 ==== 8150+0+0 (secure 0 0 0) 0x7efdd407cde0 con 0x7efde40f7400 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.469+0000 7efddaffd700 10 client.37784.objecter ms_dispatch 0x7efde4001550 osd_map(459..459 src has 173..459) v4 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.469+0000 7efddaffd700 3 client.37784.objecter handle_osd_map got epochs [459,459] > 0 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.469+0000 7efddaffd700 3 client.37784.objecter handle_osd_map decoding full epoch 459 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.469+0000 7efde91ca700 1 --2- 172.21.15.87:0/2562740791 >> v2:172.21.15.87:6800/1231597463 conn(0x7efdd0062c90 0x7efdd0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.469+0000 7efddaffd700 20 client.37784.objecter dump_active .. 0 homeless 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.469+0000 7efddaffd700 10 client.37784.objecter ms_handle_connect 0x7efdd0062c90 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.469+0000 7efde91ca700 1 --2- 172.21.15.87:0/2562740791 >> v2:172.21.15.87:6800/1231597463 conn(0x7efdd0062c90 0x7efdd0065150 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7efde40616d0 tx=0x7efde0009b60).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:39.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.473+0000 7efddaffd700 1 -- 172.21.15.87:0/2562740791 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efdd4049f60 con 0x7efde40f7400 2024-02-23T00:37:39.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:39.660+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7efde4061410 con 0x7efde40f7400 2024-02-23T00:37:39.992 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:39.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:40.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:40.129 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:40.151 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:40.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:40.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:40.282 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:40.629 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:40.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:40.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.658+0000 7efddaffd700 1 -- 172.21.15.87:0/2562740791 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v460) v1 ==== 97+0+0 (secure 0 0 0) 0x7efdd4050290 con 0x7efde40f7400 2024-02-23T00:37:40.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 10 librados: watch_flush enter 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 10 librados: watch_flush exit 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 20 client.37784.objecter shutdown clearing up homeless session... 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 10 client.37784.objecter successfully canceled tick 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 >> v2:172.21.15.87:6800/1231597463 conn(0x7efdd0062c90 msgr2=0x7efdd0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2562740791 >> v2:172.21.15.87:6800/1231597463 conn(0x7efdd0062c90 0x7efdd0065150 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 rx=0x7efde40616d0 tx=0x7efde0009b60).stop 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 msgr2=0x7efde41090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2562740791 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 0x7efde41090d0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 rx=0x7efdd40048f0 tx=0x7efdd4009cb0).stop 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 shutdown_connections 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2562740791 >> v2:172.21.15.87:6800/1231597463 conn(0x7efdd0062c90 0x7efdd0065150 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2562740791 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efde40fd510 0x7efde4109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2562740791 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efde40f7400 0x7efde41090d0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 --2- 172.21.15.87:0/2562740791 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efde40fcbb0 0x7efde41095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:37:40.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 >> 172.21.15.87:0/2562740791 conn(0x7efde40f2820 msgr2=0x7efde40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:40.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 shutdown_connections 2024-02-23T00:37:40.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 -- 172.21.15.87:0/2562740791 wait complete. 2024-02-23T00:37:40.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:40.660+0000 7efdebc2f700 1 librados: shutdown 2024-02-23T00:37:40.664 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:37:40.734 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:40.765 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:40.786 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:40.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:40.887 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:39.651550+0000 mon.a (mon.0) 2682 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:40.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:39.662550+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.87:0/2562740791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:40.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:39.662880+0000 mon.a (mon.0) 2683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:40.888 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:39.681236+0000 mgr.y (mgr.24437) 432 : cluster [DBG] pgmap v696: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:40.888 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:39.651550+0000 mon.a (mon.0) 2682 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:40.888 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:39.662550+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.87:0/2562740791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:40.888 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:39.662880+0000 mon.a (mon.0) 2683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:40.888 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:39.681236+0000 mgr.y (mgr.24437) 432 : cluster [DBG] pgmap v696: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:40.890 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:40.919 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:39.651550+0000 mon.a (mon.0) 2682 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:39.662550+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.87:0/2562740791' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:39.662880+0000 mon.a (mon.0) 2683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:37:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:39.681236+0000 mgr.y (mgr.24437) 432 : cluster [DBG] pgmap v696: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:41.369 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:41.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:41.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:41.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:37:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:37:41.482 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:41.510 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:41.529 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:41.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:41.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.591+0000 7f51622c3700 1 -- 172.21.15.87:0/3205833257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f515c105c50 msgr2=0x7f515c069570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:41.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.591+0000 7f51622c3700 1 --2- 172.21.15.87:0/3205833257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f515c105c50 0x7f515c069570 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 rx=0x7f5150004870 tx=0x7f5150009cb0).stop 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.591+0000 7f51622c3700 1 -- 172.21.15.87:0/3205833257 shutdown_connections 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.591+0000 7f51622c3700 1 --2- 172.21.15.87:0/3205833257 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f515c069c30 0x7f515c10c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.591+0000 7f51622c3700 1 --2- 172.21.15.87:0/3205833257 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f515c1052f0 0x7f515c1056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.591+0000 7f51622c3700 1 --2- 172.21.15.87:0/3205833257 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f515c105c50 0x7f515c069570 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.591+0000 7f51622c3700 1 -- 172.21.15.87:0/3205833257 >> 172.21.15.87:0/3205833257 conn(0x7f515c063db0 msgr2=0x7f515c067560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.592+0000 7f51622c3700 1 -- 172.21.15.87:0/3205833257 shutdown_connections 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.592+0000 7f51622c3700 1 -- 172.21.15.87:0/3205833257 wait complete. 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.592+0000 7f51622c3700 1 librados: starting msgr at 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.592+0000 7f51622c3700 1 librados: starting objecter 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.592+0000 7f51622c3700 1 Processor -- start 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.592+0000 7f51622c3700 1 -- start start 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.592+0000 7f51622c3700 1 librados: setting wanted keys 2024-02-23T00:37:41.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.592+0000 7f51622c3700 1 librados: calling monclient init 2024-02-23T00:37:41.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f515c069c30 0x7f515c116970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f515c1052f0 0x7f515c116e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f515c105c50 0x7f515c110b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c111000 con 0x7f515c1052f0 2024-02-23T00:37:41.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c111140 con 0x7f515c105c50 2024-02-23T00:37:41.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c111300 con 0x7f515c069c30 2024-02-23T00:37:41.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f515c069c30 0x7f515c116970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f515c1052f0 0x7f515c116e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f515c069c30 0x7f515c116970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35226/0 (socket says 172.21.15.87:35226) 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515bfff700 1 -- 172.21.15.87:0/3848767009 learned_addr learned my addr 172.21.15.87:0/3848767009 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f5160860700 1 --2- 172.21.15.87:0/3848767009 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f515c105c50 0x7f515c110b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515bfff700 1 -- 172.21.15.87:0/3848767009 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f515c105c50 msgr2=0x7f515c110b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515bfff700 1 --2- 172.21.15.87:0/3848767009 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f515c105c50 0x7f515c110b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515bfff700 1 -- 172.21.15.87:0/3848767009 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f515c1052f0 msgr2=0x7f515c116e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515bfff700 1 --2- 172.21.15.87:0/3848767009 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f515c1052f0 0x7f515c116e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515bfff700 1 -- 172.21.15.87:0/3848767009 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c005040 con 0x7f515c069c30 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515b7fe700 1 --2- 172.21.15.87:0/3848767009 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f515c1052f0 0x7f515c116e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f515bfff700 1 --2- 172.21.15.87:0/3848767009 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f515c069c30 0x7f515c116970 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 rx=0x7f514c0048f0 tx=0x7f514c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f5159775700 10 client.?.objecter ms_handle_connect 0x7f515c069c30 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f5159775700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c007b00 con 0x7f515c069c30 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5150005040 con 0x7f515c069c30 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f514c00f040 con 0x7f515c069c30 2024-02-23T00:37:41.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c014a00 con 0x7f515c069c30 2024-02-23T00:37:41.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 10 client.37790.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:41.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c111990 con 0x7f515c069c30 2024-02-23T00:37:41.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.593+0000 7f51622c3700 1 librados: init done 2024-02-23T00:37:41.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.594+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f515c060280 con 0x7f515c069c30 2024-02-23T00:37:41.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.595+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f514c00e580 con 0x7f515c069c30 2024-02-23T00:37:41.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.595+0000 7f5159775700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5144062c90 0x7f5144065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.595+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(460..460 src has 173..460) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f514c07d120 con 0x7f515c069c30 2024-02-23T00:37:41.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.595+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 osd_map(460..460 src has 173..460) v4 2024-02-23T00:37:41.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.595+0000 7f5159775700 3 client.37790.objecter handle_osd_map got epochs [460,460] > 0 2024-02-23T00:37:41.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.595+0000 7f5159775700 3 client.37790.objecter handle_osd_map decoding full epoch 460 2024-02-23T00:37:41.599 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.595+0000 7f515b7fe700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5144062c90 0x7f5144065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.595+0000 7f5159775700 20 client.37790.objecter dump_active .. 0 homeless 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.595+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v460) v1 ==== 54+0+16 (secure 0 0 0) 0x7f514c07d600 con 0x7f515c069c30 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f5159775700 10 client.37790.objecter ms_handle_connect 0x7f5144062c90 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f515b7fe700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5144062c90 0x7f5144065150 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 rx=0x7f5150004ee0 tx=0x7f515000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 10 client.37790.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 1 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 20 client.37790.objecter _get_session osd=-1 returning homeless 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 10 client.37790.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=461}) v3 -- 0x7f515c111ff0 con 0x7f515c069c30 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f515c10e790 con 0x7f515c069c30 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=460) v2 ==== 24+0+0 (secure 0 0 0) 0x7f514c0505d0 con 0x7f515c069c30 2024-02-23T00:37:41.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f515a7fc700 10 client.37790.objecter _check_command_map_dne tid 1 current 460 map_dne_bound 460 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f515a7fc700 10 client.37790.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f515a7fc700 15 client.37790.objecter _session_command_op_remove -1 1 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 10 client.37790.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 2 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6805/685354171 conn(0x7f515c059e00 0x7f515c0273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c065d50 osd=1 3 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c065d50 osd=1 3 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c065d50 osd=1 3 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 2 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c065d50 osd=1 3 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 1 2 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c065d50 osd=1 4 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 10 client.37790.objecter _send_command 2 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f515c065ec0 con 0x7f515c059e00 2024-02-23T00:37:41.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.596+0000 7f5160860700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6805/685354171 conn(0x7f515c059e00 0x7f515c0273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.597+0000 7f5159775700 10 client.37790.objecter ms_handle_connect 0x7f515c059e00 2024-02-23T00:37:41.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.597+0000 7f5160860700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6805/685354171 conn(0x7f515c059e00 0x7f515c0273c0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:41.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.597+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f515c065ec0 con 0x7f515c059e00 2024-02-23T00:37:41.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.597+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:37:41.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.597+0000 7f5159775700 10 client.37790.objecter _finish_command 2 = system:0 2024-02-23T00:37:41.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.597+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c065d50 osd=1 4 2024-02-23T00:37:41.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.597+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 1 2 2024-02-23T00:37:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 10 client.37790.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 3 2024-02-23T00:37:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c065d50 osd=1 3 2024-02-23T00:37:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c065d50 osd=1 3 2024-02-23T00:37:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c065d50 osd=1 3 2024-02-23T00:37:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 3 2024-02-23T00:37:41.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c065d50 osd=1 3 2024-02-23T00:37:41.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 1 3 2024-02-23T00:37:41.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c065d50 osd=1 4 2024-02-23T00:37:41.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 10 client.37790.objecter _send_command 3 2024-02-23T00:37:41.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.616+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f515c059360 con 0x7f515c059e00 2024-02-23T00:37:41.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.617+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f515c059360 con 0x7f515c059e00 2024-02-23T00:37:41.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.617+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:41.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.617+0000 7f5159775700 10 client.37790.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.617+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c065d50 osd=1 4 2024-02-23T00:37:41.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.617+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 1 3 2024-02-23T00:37:41.620 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:37:41.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 10 client.37790.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:41.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 4 2024-02-23T00:37:41.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6809/3605394228 conn(0x7f515c028590 0x7f515c11f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c059770 osd=2 3 2024-02-23T00:37:41.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c059770 osd=2 3 2024-02-23T00:37:41.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c059770 osd=2 3 2024-02-23T00:37:41.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 4 2024-02-23T00:37:41.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c059770 osd=2 3 2024-02-23T00:37:41.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 2 4 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c059770 osd=2 4 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 10 client.37790.objecter _send_command 4 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f515c102f50 con 0x7f515c028590 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f515bfff700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6809/3605394228 conn(0x7f515c028590 0x7f515c11f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f5159775700 10 client.37790.objecter ms_handle_connect 0x7f515c028590 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.618+0000 7f515bfff700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6809/3605394228 conn(0x7f515c028590 0x7f515c11f870 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.619+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f515c102f50 con 0x7f515c028590 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.619+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.619+0000 7f5159775700 10 client.37790.objecter _finish_command 4 = system:0 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.619+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c059770 osd=2 4 2024-02-23T00:37:41.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.619+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 2 4 2024-02-23T00:37:41.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:41.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 10 client.37790.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 5 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c059770 osd=2 3 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c059770 osd=2 3 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c059770 osd=2 3 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 5 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c059770 osd=2 3 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 2 5 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c059770 osd=2 4 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 10 client.37790.objecter _send_command 5 2024-02-23T00:37:41.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.633+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f515c027af0 con 0x7f515c028590 2024-02-23T00:37:41.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f515c027af0 con 0x7f515c028590 2024-02-23T00:37:41.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:41.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f5159775700 10 client.37790.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c059770 osd=2 4 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 2 5 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 10 client.37790.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 6 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6813/128059790 conn(0x7f515c108de0 0x7f515c120ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c027f00 osd=3 3 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c027f00 osd=3 3 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c027f00 osd=3 3 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 6 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c027f00 osd=3 3 2024-02-23T00:37:41.638 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:37:41.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 3 6 2024-02-23T00:37:41.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c027f00 osd=3 4 2024-02-23T00:37:41.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 10 client.37790.objecter _send_command 6 2024-02-23T00:37:41.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f515c121460 con 0x7f515c108de0 2024-02-23T00:37:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.635+0000 7f515b7fe700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6813/128059790 conn(0x7f515c108de0 0x7f515c120ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.636+0000 7f5159775700 10 client.37790.objecter ms_handle_connect 0x7f515c108de0 2024-02-23T00:37:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.636+0000 7f515b7fe700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.87:6813/128059790 conn(0x7f515c108de0 0x7f515c120ee0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.636+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f515c121460 con 0x7f515c108de0 2024-02-23T00:37:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.636+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:37:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.636+0000 7f5159775700 10 client.37790.objecter _finish_command 6 = system:0 2024-02-23T00:37:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.636+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c027f00 osd=3 4 2024-02-23T00:37:41.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.636+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 3 6 2024-02-23T00:37:41.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 10 client.37790.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:41.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 7 2024-02-23T00:37:41.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c027f00 osd=3 3 2024-02-23T00:37:41.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c027f00 osd=3 3 2024-02-23T00:37:41.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c027f00 osd=3 3 2024-02-23T00:37:41.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 7 2024-02-23T00:37:41.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c027f00 osd=3 3 2024-02-23T00:37:41.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 3 7 2024-02-23T00:37:41.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c027f00 osd=3 4 2024-02-23T00:37:41.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 10 client.37790.objecter _send_command 7 2024-02-23T00:37:41.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.651+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f515c121e50 con 0x7f515c108de0 2024-02-23T00:37:41.656 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.653+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f515001a040 con 0x7f515c108de0 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.653+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.653+0000 7f5159775700 10 client.37790.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.653+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c027f00 osd=3 4 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.653+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 3 7 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 10 client.37790.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 8 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6800/4175844757 conn(0x7f515c128900 0x7f515c12acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c108640 osd=4 3 2024-02-23T00:37:41.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c108640 osd=4 3 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c108640 osd=4 3 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 8 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c108640 osd=4 3 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 4 8 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c108640 osd=4 4 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 10 client.37790.objecter _send_command 8 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f515c12d970 con 0x7f515c128900 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f5160860700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6800/4175844757 conn(0x7f515c128900 0x7f515c12acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f5159775700 10 client.37790.objecter ms_handle_connect 0x7f515c128900 2024-02-23T00:37:41.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.654+0000 7f5160860700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6800/4175844757 conn(0x7f515c128900 0x7f515c12acf0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:41.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.655+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f515c12d970 con 0x7f515c128900 2024-02-23T00:37:41.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.655+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:37:41.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.655+0000 7f5159775700 10 client.37790.objecter _finish_command 8 = system:0 2024-02-23T00:37:41.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.655+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c108640 osd=4 4 2024-02-23T00:37:41.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.655+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 4 8 2024-02-23T00:37:41.660 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:40.650237+0000 mon.a (mon.0) 2684 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:41.660 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:40.658419+0000 mon.a (mon.0) 2685 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:37:41.660 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:40.658539+0000 mon.a (mon.0) 2686 : cluster [DBG] osdmap e460: 8 total, 7 up, 7 in 2024-02-23T00:37:41.660 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:40.974931+0000 mgr.y (mgr.24437) 433 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:41.660 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:41.596238+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? 172.21.15.87:0/3848767009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:41.663 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 10 client.37790.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 9 2024-02-23T00:37:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c108640 osd=4 3 2024-02-23T00:37:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c108640 osd=4 3 2024-02-23T00:37:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c108640 osd=4 3 2024-02-23T00:37:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 9 2024-02-23T00:37:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c108640 osd=4 3 2024-02-23T00:37:41.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 4 9 2024-02-23T00:37:41.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c108640 osd=4 4 2024-02-23T00:37:41.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 10 client.37790.objecter _send_command 9 2024-02-23T00:37:41.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.671+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f515c1218c0 con 0x7f515c128900 2024-02-23T00:37:41.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.673+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f515c1218c0 con 0x7f515c128900 2024-02-23T00:37:41.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.673+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:41.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.673+0000 7f5159775700 10 client.37790.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.673+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c108640 osd=4 4 2024-02-23T00:37:41.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.673+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 4 9 2024-02-23T00:37:41.676 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:37:41.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 10 client.37790.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:41.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 10 2024-02-23T00:37:41.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6804/2550013795 conn(0x7f515c120a40 0x7f515c12c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c1218c0 osd=5 3 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c1218c0 osd=5 3 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c1218c0 osd=5 3 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 10 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c1218c0 osd=5 3 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 5 10 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c1218c0 osd=5 4 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 10 client.37790.objecter _send_command 10 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f515c121b90 con 0x7f515c120a40 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f515bfff700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6804/2550013795 conn(0x7f515c120a40 0x7f515c12c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f5159775700 10 client.37790.objecter ms_handle_connect 0x7f515c120a40 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.674+0000 7f515bfff700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6804/2550013795 conn(0x7f515c120a40 0x7f515c12c360 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.675+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f515c121b90 con 0x7f515c120a40 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.675+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.675+0000 7f5159775700 10 client.37790.objecter _finish_command 10 = system:0 2024-02-23T00:37:41.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.675+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c1218c0 osd=5 4 2024-02-23T00:37:41.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.675+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 5 10 2024-02-23T00:37:41.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 10 client.37790.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:41.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 11 2024-02-23T00:37:41.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c1218c0 osd=5 3 2024-02-23T00:37:41.693 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c1218c0 osd=5 3 2024-02-23T00:37:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c1218c0 osd=5 3 2024-02-23T00:37:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 11 2024-02-23T00:37:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c1218c0 osd=5 3 2024-02-23T00:37:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 5 11 2024-02-23T00:37:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c1218c0 osd=5 4 2024-02-23T00:37:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 10 client.37790.objecter _send_command 11 2024-02-23T00:37:41.694 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.691+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f515c121b90 con 0x7f515c120a40 2024-02-23T00:37:41.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f515c121b90 con 0x7f515c120a40 2024-02-23T00:37:41.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:41.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f5159775700 10 client.37790.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c1218c0 osd=5 4 2024-02-23T00:37:41.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 5 11 2024-02-23T00:37:41.696 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:37:41.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 10 client.37790.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:41.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 12 2024-02-23T00:37:41.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6808/2509408498 conn(0x7f515c12cda0 0x7f515c12d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c121b90 osd=6 3 2024-02-23T00:37:41.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c121b90 osd=6 3 2024-02-23T00:37:41.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c121b90 osd=6 3 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 12 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c121b90 osd=6 3 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 6 12 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c121b90 osd=6 4 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 10 client.37790.objecter _send_command 12 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.693+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f515c134310 con 0x7f515c12cda0 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.694+0000 7f515b7fe700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6808/2509408498 conn(0x7f515c12cda0 0x7f515c12d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.694+0000 7f5159775700 10 client.37790.objecter ms_handle_connect 0x7f515c12cda0 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.694+0000 7f515b7fe700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6808/2509408498 conn(0x7f515c12cda0 0x7f515c12d200 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.695+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f515c134310 con 0x7f515c12cda0 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.695+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.695+0000 7f5159775700 10 client.37790.objecter _finish_command 12 = system:0 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.695+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c121b90 osd=6 4 2024-02-23T00:37:41.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.695+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 6 12 2024-02-23T00:37:41.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 10 client.37790.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:41.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 13 2024-02-23T00:37:41.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c121b90 osd=6 3 2024-02-23T00:37:41.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c121b90 osd=6 3 2024-02-23T00:37:41.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c121b90 osd=6 3 2024-02-23T00:37:41.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 13 2024-02-23T00:37:41.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c121b90 osd=6 3 2024-02-23T00:37:41.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 6 13 2024-02-23T00:37:41.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c121b90 osd=6 4 2024-02-23T00:37:41.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 10 client.37790.objecter _send_command 13 2024-02-23T00:37:41.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.709+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f515c131bb0 con 0x7f515c12cda0 2024-02-23T00:37:41.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f515c131bb0 con 0x7f515c12cda0 2024-02-23T00:37:41.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:41.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f5159775700 10 client.37790.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c121b90 osd=6 4 2024-02-23T00:37:41.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 6 13 2024-02-23T00:37:41.713 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:37:41.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 10 client.37790.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:41.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 14 2024-02-23T00:37:41.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6812/373787000 conn(0x7f515c132230 0x7f515c134680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:41.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c131bb0 osd=7 3 2024-02-23T00:37:41.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c131bb0 osd=7 3 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c131bb0 osd=7 3 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 14 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c131bb0 osd=7 3 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 7 14 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c131bb0 osd=7 4 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 10 client.37790.objecter _send_command 14 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f515c137450 con 0x7f515c132230 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f5160860700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6812/373787000 conn(0x7f515c132230 0x7f515c134680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f5159775700 10 client.37790.objecter ms_handle_connect 0x7f515c132230 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.711+0000 7f5160860700 1 --2- 172.21.15.87:0/3848767009 >> v2:172.21.15.171:6812/373787000 conn(0x7f515c132230 0x7f515c134680 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.712+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f515c137450 con 0x7f515c132230 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.712+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:37:41.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.712+0000 7f5159775700 10 client.37790.objecter _finish_command 14 = system:0 2024-02-23T00:37:41.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.712+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c131bb0 osd=7 4 2024-02-23T00:37:41.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.712+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 7 14 2024-02-23T00:37:41.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 10 client.37790.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:37:41.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign -1 15 2024-02-23T00:37:41.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c131bb0 osd=7 3 2024-02-23T00:37:41.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c131bb0 osd=7 3 2024-02-23T00:37:41.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 20 client.37790.objecter _get_session s=0x7f515c131bb0 osd=7 3 2024-02-23T00:37:41.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 15 client.37790.objecter _session_command_op_remove -1 15 2024-02-23T00:37:41.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 20 client.37790.objecter get_session s=0x7f515c131bb0 osd=7 3 2024-02-23T00:37:41.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 15 client.37790.objecter _session_command_op_assign 7 15 2024-02-23T00:37:41.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 20 client.37790.objecter put_session s=0x7f515c131bb0 osd=7 4 2024-02-23T00:37:41.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 10 client.37790.objecter _send_command 15 2024-02-23T00:37:41.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.726+0000 7f51622c3700 1 -- 172.21.15.87:0/3848767009 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f515c137450 con 0x7f515c132230 2024-02-23T00:37:41.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.728+0000 7f5159775700 1 -- 172.21.15.87:0/3848767009 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f515c137450 con 0x7f515c132230 2024-02-23T00:37:41.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.728+0000 7f5159775700 10 client.37790.objecter ms_dispatch 0x7f515c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:37:41.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.728+0000 7f5159775700 10 client.37790.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.728+0000 7f5159775700 20 client.37790.objecter put_session s=0x7f515c131bb0 osd=7 4 2024-02-23T00:37:41.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:41.728+0000 7f5159775700 15 client.37790.objecter _session_command_op_remove 7 15 2024-02-23T00:37:41.730 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:37:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:37:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.731 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:37:41.874 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:37:41.875 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:37:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:40.650237+0000 mon.a (mon.0) 2684 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:40.658419+0000 mon.a (mon.0) 2685 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:37:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:40.658539+0000 mon.a (mon.0) 2686 : cluster [DBG] osdmap e460: 8 total, 7 up, 7 in 2024-02-23T00:37:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:40.974931+0000 mgr.y (mgr.24437) 433 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:41.596238+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? 172.21.15.87:0/3848767009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:42.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:40.650237+0000 mon.a (mon.0) 2684 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:42.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:40.658419+0000 mon.a (mon.0) 2685 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:37:42.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:40.658539+0000 mon.a (mon.0) 2686 : cluster [DBG] osdmap e460: 8 total, 7 up, 7 in 2024-02-23T00:37:42.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:40.974931+0000 mgr.y (mgr.24437) 433 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:42.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:41.596238+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? 172.21.15.87:0/3848767009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:42.112 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:42.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:42.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:42.250 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:42.467 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:42.469 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:42] "GET /metrics HTTP/1.1" 200 188628 "" "Prometheus/2.33.4" 2024-02-23T00:37:42.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:42.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:42.571 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:42.598 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:41.681821+0000 mgr.y (mgr.24437) 434 : cluster [DBG] pgmap v698: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:42.109779+0000 mon.a (mon.0) 2687 : cluster [WRN] Health check update: Degraded data redundancy: 125/1194 objects degraded (10.469%), 28 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:42.115316+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e461: 8 total, 7 up, 7 in 2024-02-23T00:37:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.292895+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.293354+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.294086+0000 mon.b (mon.1) 378 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-71"}]: dispatch 2024-02-23T00:37:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.294398+0000 mon.a (mon.0) 2690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-71"}]: dispatch 2024-02-23T00:37:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:41.681821+0000 mgr.y (mgr.24437) 434 : cluster [DBG] pgmap v698: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:42.109779+0000 mon.a (mon.0) 2687 : cluster [WRN] Health check update: Degraded data redundancy: 125/1194 objects degraded (10.469%), 28 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:42.115316+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e461: 8 total, 7 up, 7 in 2024-02-23T00:37:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.292895+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.293354+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.294086+0000 mon.b (mon.1) 378 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-71"}]: dispatch 2024-02-23T00:37:42.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.294398+0000 mon.a (mon.0) 2690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-71"}]: dispatch 2024-02-23T00:37:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:41.681821+0000 mgr.y (mgr.24437) 434 : cluster [DBG] pgmap v698: 238 pgs: 28 active+undersized+degraded, 67 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 255 B/s wr, 3 op/s; 125/1194 objects degraded (10.469%) 2024-02-23T00:37:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:42.109779+0000 mon.a (mon.0) 2687 : cluster [WRN] Health check update: Degraded data redundancy: 125/1194 objects degraded (10.469%), 28 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:42.115316+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e461: 8 total, 7 up, 7 in 2024-02-23T00:37:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.292895+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.293354+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.294086+0000 mon.b (mon.1) 378 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-71"}]: dispatch 2024-02-23T00:37:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.294398+0000 mon.a (mon.0) 2690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-71"}]: dispatch 2024-02-23T00:37:42.947 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:42.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:42.964 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-23T00:37:43.050 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:43.076 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:43.095 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:43.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:43.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:43.199 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:43.226 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:43.245 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:43.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:43.349 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:43.376 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:43.394 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:43.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:43.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:37:43.447 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:37:43.498 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:43.525 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:43.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.706577+0000 mon.a (mon.0) 2691 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:37:43.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.986553+0000 mon.a (mon.0) 2692 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.987208+0000 mon.a (mon.0) 2693 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.987756+0000 mon.a (mon.0) 2694 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.988293+0000 mon.a (mon.0) 2695 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:37:42.988761+0000 mgr.y (mgr.24437) 435 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 3256M 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:42.992794+0000 mon.a (mon.0) 2696 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:43.049745+0000 mon.a (mon.0) 2697 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:43.050422+0000 mon.a (mon.0) 2698 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:43.050980+0000 mon.a (mon.0) 2699 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:43.051511+0000 mon.a (mon.0) 2700 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cephadm 2024-02-23T00:37:43.051918+0000 mgr.y (mgr.24437) 436 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 3256M 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:43.055808+0000 mon.a (mon.0) 2701 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:43.056764+0000 mon.a (mon.0) 2702 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:43.057410+0000 mon.a (mon.0) 2703 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:37:43.752 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:43.061750+0000 mon.a (mon.0) 2704 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:43.126481+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e462: 8 total, 7 up, 7 in 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.706577+0000 mon.a (mon.0) 2691 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.986553+0000 mon.a (mon.0) 2692 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.987208+0000 mon.a (mon.0) 2693 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.987756+0000 mon.a (mon.0) 2694 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.988293+0000 mon.a (mon.0) 2695 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:37:42.988761+0000 mgr.y (mgr.24437) 435 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 3256M 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:42.992794+0000 mon.a (mon.0) 2696 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:43.049745+0000 mon.a (mon.0) 2697 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:43.050422+0000 mon.a (mon.0) 2698 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.753 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:43.050980+0000 mon.a (mon.0) 2699 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.754 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:43.051511+0000 mon.a (mon.0) 2700 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.754 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cephadm 2024-02-23T00:37:43.051918+0000 mgr.y (mgr.24437) 436 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 3256M 2024-02-23T00:37:43.754 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:43.055808+0000 mon.a (mon.0) 2701 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:43.754 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:43.056764+0000 mon.a (mon.0) 2702 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:37:43.754 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:43.057410+0000 mon.a (mon.0) 2703 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:37:43.754 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:43.061750+0000 mon.a (mon.0) 2704 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:43.754 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:43.126481+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e462: 8 total, 7 up, 7 in 2024-02-23T00:37:43.773 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:43.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:43.876 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:43.905 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:43.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:37:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.706577+0000 mon.a (mon.0) 2691 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:37:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.986553+0000 mon.a (mon.0) 2692 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.987208+0000 mon.a (mon.0) 2693 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.987756+0000 mon.a (mon.0) 2694 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.988293+0000 mon.a (mon.0) 2695 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:37:42.988761+0000 mgr.y (mgr.24437) 435 : cephadm [INF] Adjusting osd_memory_target on smithi171 to 3256M 2024-02-23T00:37:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:42.992794+0000 mon.a (mon.0) 2696 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:43.049745+0000 mon.a (mon.0) 2697 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:43.050422+0000 mon.a (mon.0) 2698 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:43.050980+0000 mon.a (mon.0) 2699 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:43.051511+0000 mon.a (mon.0) 2700 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-23T00:37:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cephadm 2024-02-23T00:37:43.051918+0000 mgr.y (mgr.24437) 436 : cephadm [INF] Adjusting osd_memory_target on smithi087 to 3256M 2024-02-23T00:37:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:43.055808+0000 mon.a (mon.0) 2701 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:43.056764+0000 mon.a (mon.0) 2702 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:37:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:43.057410+0000 mon.a (mon.0) 2703 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:37:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:43.061750+0000 mon.a (mon.0) 2704 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:37:43.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:43.126481+0000 mon.a (mon.0) 2705 : cluster [DBG] osdmap e462: 8 total, 7 up, 7 in 2024-02-23T00:37:44.352 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:44.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:44.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:44.456 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:44.482 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:44.500 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:44.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:44.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:44.631 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:45.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:43.682718+0000 mgr.y (mgr.24437) 437 : cluster [DBG] pgmap v701: 206 pgs: 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123/1182 objects degraded (10.406%) 2024-02-23T00:37:45.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:44.127242+0000 mon.a (mon.0) 2706 : cluster [DBG] osdmap e463: 8 total, 7 up, 7 in 2024-02-23T00:37:45.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:44.137372+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:45.324 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:44.137685+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:45.325 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:43.682718+0000 mgr.y (mgr.24437) 437 : cluster [DBG] pgmap v701: 206 pgs: 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123/1182 objects degraded (10.406%) 2024-02-23T00:37:45.325 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:44.127242+0000 mon.a (mon.0) 2706 : cluster [DBG] osdmap e463: 8 total, 7 up, 7 in 2024-02-23T00:37:45.325 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:44.137372+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:45.325 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:44.137685+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:45.325 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:43.682718+0000 mgr.y (mgr.24437) 437 : cluster [DBG] pgmap v701: 206 pgs: 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123/1182 objects degraded (10.406%) 2024-02-23T00:37:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:44.127242+0000 mon.a (mon.0) 2706 : cluster [DBG] osdmap e463: 8 total, 7 up, 7 in 2024-02-23T00:37:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:44.137372+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:44.137685+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:45.447 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:37:45.688 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:45.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:45.792 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:45.820 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:45.919 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:45.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:46.023 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:46.051 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:46.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:45.132116+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:46.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:45.132214+0000 mon.a (mon.0) 2709 : cluster [DBG] osdmap e464: 8 total, 7 up, 7 in 2024-02-23T00:37:46.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:45.148064+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:46.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:46.135945+0000 mon.a (mon.0) 2710 : cluster [DBG] osdmap e465: 8 total, 7 up, 7 in 2024-02-23T00:37:46.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:45.132116+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:46.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:45.132214+0000 mon.a (mon.0) 2709 : cluster [DBG] osdmap e464: 8 total, 7 up, 7 in 2024-02-23T00:37:46.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:45.148064+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:46.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:46.135945+0000 mon.a (mon.0) 2710 : cluster [DBG] osdmap e465: 8 total, 7 up, 7 in 2024-02-23T00:37:46.401 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:46.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:46.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:45.132116+0000 mon.a (mon.0) 2708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:46.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:45.132214+0000 mon.a (mon.0) 2709 : cluster [DBG] osdmap e464: 8 total, 7 up, 7 in 2024-02-23T00:37:46.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:45.148064+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:46.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:46.135945+0000 mon.a (mon.0) 2710 : cluster [DBG] osdmap e465: 8 total, 7 up, 7 in 2024-02-23T00:37:46.506 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:46.535 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:46.880 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:37:46.984 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:46.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:47.088 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:47.114 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:47.134 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:47.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:47.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:45.683803+0000 mgr.y (mgr.24437) 438 : cluster [DBG] pgmap v704: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1197 objects degraded (10.359%) 2024-02-23T00:37:47.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:47.112247+0000 mon.a (mon.0) 2711 : cluster [WRN] Health check update: Degraded data redundancy: 124/1197 objects degraded (10.359%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:47.181 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:47.140234+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e466: 8 total, 7 up, 7 in 2024-02-23T00:37:47.243 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:47.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.263+0000 7f3923ab3700 1 -- 172.21.15.87:0/943477099 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0fcac0 msgr2=0x7f391c0fcea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:47.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.263+0000 7f3923ab3700 1 --2- 172.21.15.87:0/943477099 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0fcac0 0x7f391c0fcea0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 rx=0x7f39180047b0 tx=0x7f3918009cb0).stop 2024-02-23T00:37:47.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.264+0000 7f3923ab3700 1 -- 172.21.15.87:0/943477099 shutdown_connections 2024-02-23T00:37:47.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.264+0000 7f3923ab3700 1 --2- 172.21.15.87:0/943477099 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f391c0f7330 0x7f391c103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:37:47.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.264+0000 7f3923ab3700 1 --2- 172.21.15.87:0/943477099 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0fcac0 0x7f391c0fcea0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:47.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.264+0000 7f3923ab3700 1 --2- 172.21.15.87:0/943477099 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f391c0fd420 0x7f391c0f6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:47.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.264+0000 7f3923ab3700 1 -- 172.21.15.87:0/943477099 >> 172.21.15.87:0/943477099 conn(0x7f391c0f2710 msgr2=0x7f391c0f4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:47.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.264+0000 7f3923ab3700 1 -- 172.21.15.87:0/943477099 shutdown_connections 2024-02-23T00:37:47.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.264+0000 7f3923ab3700 1 -- 172.21.15.87:0/943477099 wait complete. 2024-02-23T00:37:47.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 librados: starting msgr at 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 librados: starting objecter 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 Processor -- start 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 -- start start 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 librados: setting wanted keys 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 librados: calling monclient init 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0f7330 0x7f391c10e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f391c0fcac0 0x7f391c10e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f391c0fd420 0x7f391c108430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c108920 con 0x7f391c0fd420 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c108a90 con 0x7f391c0fcac0 2024-02-23T00:37:47.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f3923ab3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391c108c50 con 0x7f391c0f7330 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f392184f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0f7330 0x7f391c10e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f392184f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0f7330 0x7f391c10e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35256/0 (socket says 172.21.15.87:35256) 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.265+0000 7f392184f700 1 -- 172.21.15.87:0/3378168663 learned_addr learned my addr 172.21.15.87:0/3378168663 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f392104e700 1 --2- 172.21.15.87:0/3378168663 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f391c0fcac0 0x7f391c10e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f392184f700 1 -- 172.21.15.87:0/3378168663 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f391c0fcac0 msgr2=0x7f391c10e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f392184f700 1 --2- 172.21.15.87:0/3378168663 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f391c0fcac0 0x7f391c10e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f392184f700 1 -- 172.21.15.87:0/3378168663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f391c0fd420 msgr2=0x7f391c108430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f392184f700 1 --2- 172.21.15.87:0/3378168663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f391c0fd420 0x7f391c108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f392184f700 1 -- 172.21.15.87:0/3378168663 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3918005040 con 0x7f391c0f7330 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f392104e700 1 --2- 172.21.15.87:0/3378168663 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f391c0fcac0 0x7f391c10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:47.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f392184f700 1 --2- 172.21.15.87:0/3378168663 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0f7330 0x7f391c10e290 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7f3918005010 tx=0x7f3918007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f3912f78700 10 client.?.objecter ms_handle_connect 0x7f391c0f7330 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f3912f78700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f3912f78700 1 -- 172.21.15.87:0/3378168663 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3918018070 con 0x7f391c0f7330 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f3912f78700 1 -- 172.21.15.87:0/3378168663 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39180093b0 con 0x7f391c0f7330 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391c108ed0 con 0x7f391c0f7330 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f3912f78700 1 -- 172.21.15.87:0/3378168663 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391801dd80 con 0x7f391c0f7330 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f3923ab3700 10 client.37796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391c1150c0 con 0x7f391c0f7330 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.266+0000 7f3923ab3700 1 librados: init done 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.269+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391c0606d0 con 0x7f391c0f7330 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.269+0000 7f3912f78700 1 -- 172.21.15.87:0/3378168663 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f391801d4b0 con 0x7f391c0f7330 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.270+0000 7f3912f78700 1 --2- 172.21.15.87:0/3378168663 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3908062ce0 0x7f39080651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:47.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.270+0000 7f392104e700 1 --2- 172.21.15.87:0/3378168663 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3908062ce0 0x7f39080651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:47.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.271+0000 7f392104e700 1 --2- 172.21.15.87:0/3378168663 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3908062ce0 0x7f39080651a0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f390c004fd0 tx=0x7f390c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:47.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.271+0000 7f3912f78700 10 client.37796.objecter ms_handle_connect 0x7f3908062ce0 2024-02-23T00:37:47.280 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:47.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.278+0000 7f3912f78700 1 -- 172.21.15.87:0/3378168663 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(466..466 src has 173..466) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f391807bb50 con 0x7f391c0f7330 2024-02-23T00:37:47.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.278+0000 7f3912f78700 10 client.37796.objecter ms_dispatch 0x7f391c001550 osd_map(466..466 src has 173..466) v4 2024-02-23T00:37:47.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.278+0000 7f3912f78700 3 client.37796.objecter handle_osd_map got epochs [466,466] > 0 2024-02-23T00:37:47.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.278+0000 7f3912f78700 3 client.37796.objecter handle_osd_map decoding full epoch 466 2024-02-23T00:37:47.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.279+0000 7f3912f78700 20 client.37796.objecter dump_active .. 0 homeless 2024-02-23T00:37:47.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.279+0000 7f3912f78700 1 -- 172.21.15.87:0/3378168663 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3918051330 con 0x7f391c0f7330 2024-02-23T00:37:47.288 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:47.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:47.405 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:47.429 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:37:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-2[57400]: debug 2024-02-23T00:37:47.243+0000 7f6833072700 -1 snap_mapper.add_oid found existing snaps mapped on 63:986b347f:test-rados-api-smithi087-82091-74::foo:24, removing 2024-02-23T00:37:47.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:45.683803+0000 mgr.y (mgr.24437) 438 : cluster [DBG] pgmap v704: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1197 objects degraded (10.359%) 2024-02-23T00:37:47.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:47.112247+0000 mon.a (mon.0) 2711 : cluster [WRN] Health check update: Degraded data redundancy: 124/1197 objects degraded (10.359%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:47.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:47.140234+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e466: 8 total, 7 up, 7 in 2024-02-23T00:37:47.431 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:47.442 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:37:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:37:47.243+0000 7f0bb6e44700 -1 snap_mapper.add_oid found existing snaps mapped on 63:986b347f:test-rados-api-smithi087-82091-74::foo:24, removing 2024-02-23T00:37:47.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:45.683803+0000 mgr.y (mgr.24437) 438 : cluster [DBG] pgmap v704: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1197 objects degraded (10.359%) 2024-02-23T00:37:47.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:47.112247+0000 mon.a (mon.0) 2711 : cluster [WRN] Health check update: Degraded data redundancy: 124/1197 objects degraded (10.359%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:47.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:47.140234+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e466: 8 total, 7 up, 7 in 2024-02-23T00:37:47.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:37:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:37:47.243+0000 7fef39f74700 -1 snap_mapper.add_oid found existing snaps mapped on 63:986b347f:test-rados-api-smithi087-82091-74::foo:24, removing 2024-02-23T00:37:47.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.466+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f391c1099a0 con 0x7f3908062ce0 2024-02-23T00:37:47.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.483+0000 7f3912f78700 1 -- 172.21.15.87:0/3378168663 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+599719 (secure 0 0 0) 0x7f391c1099a0 con 0x7f3908062ce0 2024-02-23T00:37:47.493 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:37:47.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 10 librados: watch_flush enter 2024-02-23T00:37:47.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 10 librados: watch_flush exit 2024-02-23T00:37:47.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 20 client.37796.objecter shutdown clearing up homeless session... 2024-02-23T00:37:47.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 10 client.37796.objecter successfully canceled tick 2024-02-23T00:37:47.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3908062ce0 msgr2=0x7f39080651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:47.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 --2- 172.21.15.87:0/3378168663 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3908062ce0 0x7f39080651a0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 rx=0x7f390c004fd0 tx=0x7f390c00b040).stop 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0f7330 msgr2=0x7f391c10e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 --2- 172.21.15.87:0/3378168663 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0f7330 0x7f391c10e290 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 rx=0x7f3918005010 tx=0x7f3918007f90).stop 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 shutdown_connections 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 --2- 172.21.15.87:0/3378168663 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3908062ce0 0x7f39080651a0 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 --2- 172.21.15.87:0/3378168663 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f391c0fd420 0x7f391c108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 --2- 172.21.15.87:0/3378168663 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f391c0f7330 0x7f391c10e290 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 --2- 172.21.15.87:0/3378168663 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f391c0fcac0 0x7f391c10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 >> 172.21.15.87:0/3378168663 conn(0x7f391c0f2710 msgr2=0x7f391c0f42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 shutdown_connections 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 -- 172.21.15.87:0/3378168663 wait complete. 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:47.487+0000 7f3923ab3700 1 librados: shutdown 2024-02-23T00:37:47.500 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:37:48.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:47.286603+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:48.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:47.291256+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:48.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:47.297891+0000 mon.b (mon.1) 382 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-73"}]: dispatch 2024-02-23T00:37:48.179 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:47.301792+0000 mon.a (mon.0) 2714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-73"}]: dispatch 2024-02-23T00:37:48.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:47.286603+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:48.179 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:47.291256+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:48.180 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:47.297891+0000 mon.b (mon.1) 382 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-73"}]: dispatch 2024-02-23T00:37:48.180 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:47.301792+0000 mon.a (mon.0) 2714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-73"}]: dispatch 2024-02-23T00:37:48.345 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2024-02-23T00:37:47.140719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007419,"num_objects":399,"num_object_clones":7,"num_object_copies":1197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":1934,"num_read_kb":1560,"num_write":1531,"num_write_kb":4568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":4003469,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2591,"ondisk_log_size":2591,"up":619,"acting":619,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":683,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2191000,"kb_used_data":48992,"kb_used_omap":0,"kb_used_meta":2141952,"kb_avail":654025064,"statfs":{"total":671965249536,"available":669721665536,"internally_reserved":0,"allocated":50167808,"data_stored":32242725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2193358848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016942"},"pg_stats":[{"pgid":"218.1f","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.164216+0000","last_change":"2024-02-23T00:37:45.164216+0000","last_active":"2024-02-23T00:37:45.164216+0000","last_peered":"2024-02-23T00:37:45.164216+0000","last_clean":"2024-02-23T00:37:45.164216+0000","last_became_active":"2024-02-23T00:37:45.163565+0000","last_became_peered":"2024-02-23T00:37:45.163565+0000","last_unstale":"2024-02-23T00:37:45.164216+0000","last_undegraded":"2024-02-23T00:37:45.164216+0000","last_fullsized":"2024-02-23T00:37:45.164216+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"218.1e","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.138764+0000","last_change":"2024-02-23T00:37:45.138764+0000","last_active":"2024-02-23T00:37:45.138764+0000","last_peered":"2024-02-23T00:37:45.138764+0000","last_clean":"2024-02-23T00:37:45.138764+0000","last_became_active":"2024-02-23T00:37:45.138483+0000","last_became_peered":"2024-02-23T00:37:45.138483+0000","last_unstale":"2024-02-23T00:37:45.138764+0000","last_undegraded":"2024-02-23T00:37:45.138764+0000","last_fullsized":"2024-02-23T00:37:45.138764+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"218.1d","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.152742+0000","last_change":"2024-02-23T00:37:45.152742+0000","last_active":"2024-02-23T00:37:45.152742+0000","last_peered":"2024-02-23T00:37:45.152742+0000","last_clean":"2024-02-23T00:37:45.152742+0000","last_became_active":"2024-02-23T00:37:45.152448+0000","last_became_peered":"2024-02-23T00:37:45.152448+0000","last_unstale":"2024-02-23T00:37:45.152742+0000","last_undegraded":"2024-02-23T00:37:45.152742+0000","last_fullsized":"2024-02-23T00:37:45.152742+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"218.1c","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.161314+0000","last_change":"2024-02-23T00:37:45.161314+0000","last_active":"2024-02-23T00:37:45.161314+0000","last_peered":"2024-02-23T00:37:45.161314+0000","last_clean":"2024-02-23T00:37:45.161314+0000","last_became_active":"2024-02-23T00:37:45.161039+0000","last_became_peered":"2024-02-23T00:37:45.161039+0000","last_unstale":"2024-02-23T00:37:45.161314+0000","last_undegraded":"2024-02-23T00:37:45.161314+0000","last_fullsized":"2024-02-23T00:37:45.161314+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"218.1b","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.136460+0000","last_change":"2024-02-23T00:37:45.136393+0000","last_active":"2024-02-23T00:37:45.136460+0000","last_peered":"2024-02-23T00:37:45.136460+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.136393+0000","last_became_peered":"2024-02-23T00:37:45.136393+0000","last_unstale":"2024-02-23T00:37:45.136460+0000","last_undegraded":"2024-02-23T00:37:45.136460+0000","last_fullsized":"2024-02-23T00:37:45.130626+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"218.1a","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.161845+0000","last_change":"2024-02-23T00:37:45.161845+0000","last_active":"2024-02-23T00:37:45.161845+0000","last_peered":"2024-02-23T00:37:45.161845+0000","last_clean":"2024-02-23T00:37:45.161845+0000","last_became_active":"2024-02-23T00:37:45.161430+0000","last_became_peered":"2024-02-23T00:37:45.161430+0000","last_unstale":"2024-02-23T00:37:45.161845+0000","last_undegraded":"2024-02-23T00:37:45.161845+0000","last_fullsized":"2024-02-23T00:37:45.161845+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"218.19","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.160988+0000","last_change":"2024-02-23T00:37:45.160988+0000","last_active":"2024-02-23T00:37:45.160988+0000","last_peered":"2024-02-23T00:37:45.160988+0000","last_clean":"2024-02-23T00:37:45.160988+0000","last_became_active":"2024-02-23T00:37:45.160707+0000","last_became_peered":"2024-02-23T00:37:45.160707+0000","last_unstale":"2024-02-23T00:37:45.160988+0000","last_undegraded":"2024-02-23T00:37:45.160988+0000","last_fullsized":"2024-02-23T00:37:45.160988+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"218.18","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.156172+0000","last_change":"2024-02-23T00:37:45.156172+0000","last_active":"2024-02-23T00:37:45.156172+0000","last_peered":"2024-02-23T00:37:45.156172+0000","last_clean":"2024-02-23T00:37:45.156172+0000","last_became_active":"2024-02-23T00:37:45.155919+0000","last_became_peered":"2024-02-23T00:37:45.155919+0000","last_unstale":"2024-02-23T00:37:45.156172+0000","last_undegraded":"2024-02-23T00:37:45.156172+0000","last_fullsized":"2024-02-23T00:37:45.156172+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"218.17","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.152961+0000","last_change":"2024-02-23T00:37:45.152961+0000","last_active":"2024-02-23T00:37:45.152961+0000","last_peered":"2024-02-23T00:37:45.152961+0000","last_clean":"2024-02-23T00:37:45.152961+0000","last_became_active":"2024-02-23T00:37:45.152673+0000","last_became_peered":"2024-02-23T00:37:45.152673+0000","last_unstale":"2024-02-23T00:37:45.152961+0000","last_undegraded":"2024-02-23T00:37:45.152961+0000","last_fullsized":"2024-02-23T00:37:45.152961+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"218.16","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.169473+0000","last_change":"2024-02-23T00:37:45.169407+0000","last_active":"2024-02-23T00:37:45.169473+0000","last_peered":"2024-02-23T00:37:45.169473+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.169407+0000","last_became_peered":"2024-02-23T00:37:45.169407+0000","last_unstale":"2024-02-23T00:37:45.169473+0000","last_undegraded":"2024-02-23T00:37:45.169473+0000","last_fullsized":"2024-02-23T00:37:45.146398+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"218.15","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.163723+0000","last_change":"2024-02-23T00:37:45.163656+0000","last_active":"2024-02-23T00:37:45.163723+0000","last_peered":"2024-02-23T00:37:45.163723+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.163656+0000","last_became_peered":"2024-02-23T00:37:45.163656+0000","last_unstale":"2024-02-23T00:37:45.163723+0000","last_undegraded":"2024-02-23T00:37:45.163723+0000","last_fullsized":"2024-02-23T00:37:45.148002+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"218.14","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.156471+0000","last_change":"2024-02-23T00:37:45.156403+0000","last_active":"2024-02-23T00:37:45.156471+0000","last_peered":"2024-02-23T00:37:45.156471+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.156403+0000","last_became_peered":"2024-02-23T00:37:45.156403+0000","last_unstale":"2024-02-23T00:37:45.156471+0000","last_undegraded":"2024-02-23T00:37:45.156471+0000","last_fullsized":"2024-02-23T00:37:45.130629+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"218.13","version":"464'1","reported_seq":7,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.171827+0000","last_change":"2024-02-23T00:37:45.171617+0000","last_active":"2024-02-23T00:37:45.171827+0000","last_peered":"2024-02-23T00:37:45.171827+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.171617+0000","last_became_peered":"2024-02-23T00:37:45.171617+0000","last_unstale":"2024-02-23T00:37:45.171827+0000","last_undegraded":"2024-02-23T00:37:45.171827+0000","last_fullsized":"2024-02-23T00:37:45.146666+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"218.12","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.159256+0000","last_change":"2024-02-23T00:37:45.159188+0000","last_active":"2024-02-23T00:37:45.159256+0000","last_peered":"2024-02-23T00:37:45.159256+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.159188+0000","last_became_peered":"2024-02-23T00:37:45.159188+0000","last_unstale":"2024-02-23T00:37:45.159256+0000","last_undegraded":"2024-02-23T00:37:45.159256+0000","last_fullsized":"2024-02-23T00:37:45.134519+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"218.11","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.136220+0000","last_change":"2024-02-23T00:37:45.136010+0000","last_active":"2024-02-23T00:37:45.136220+0000","last_peered":"2024-02-23T00:37:45.136220+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.136010+0000","last_became_peered":"2024-02-23T00:37:45.136010+0000","last_unstale":"2024-02-23T00:37:45.136220+0000","last_undegraded":"2024-02-23T00:37:45.136220+0000","last_fullsized":"2024-02-23T00:37:45.134528+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"218.10","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.133743+0000","last_change":"2024-02-23T00:37:45.133665+0000","last_active":"2024-02-23T00:37:45.133743+0000","last_peered":"2024-02-23T00:37:45.133743+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.133665+0000","last_became_peered":"2024-02-23T00:37:45.133665+0000","last_unstale":"2024-02-23T00:37:45.133743+0000","last_undegraded":"2024-02-23T00:37:45.133743+0000","last_fullsized":"2024-02-23T00:37:45.131098+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"218.f","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.164663+0000","last_change":"2024-02-23T00:37:45.164589+0000","last_active":"2024-02-23T00:37:45.164663+0000","last_peered":"2024-02-23T00:37:45.164663+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.164589+0000","last_became_peered":"2024-02-23T00:37:45.164589+0000","last_unstale":"2024-02-23T00:37:45.164663+0000","last_undegraded":"2024-02-23T00:37:45.164663+0000","last_fullsized":"2024-02-23T00:37:45.147143+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"218.e","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.164952+0000","last_change":"2024-02-23T00:37:45.164883+0000","last_active":"2024-02-23T00:37:45.164952+0000","last_peered":"2024-02-23T00:37:45.164952+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.164883+0000","last_became_peered":"2024-02-23T00:37:45.164883+0000","last_unstale":"2024-02-23T00:37:45.164952+0000","last_undegraded":"2024-02-23T00:37:45.164952+0000","last_fullsized":"2024-02-23T00:37:45.132038+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"218.d","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.137720+0000","last_change":"2024-02-23T00:37:45.137652+0000","last_active":"2024-02-23T00:37:45.137720+0000","last_peered":"2024-02-23T00:37:45.137720+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.137652+0000","last_became_peered":"2024-02-23T00:37:45.137652+0000","last_unstale":"2024-02-23T00:37:45.137720+0000","last_undegraded":"2024-02-23T00:37:45.137720+0000","last_fullsized":"2024-02-23T00:37:45.130635+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"218.c","version":"464'1","reported_seq":10,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.166792+0000","last_change":"2024-02-23T00:37:45.165554+0000","last_active":"2024-02-23T00:37:45.166792+0000","last_peered":"2024-02-23T00:37:45.166792+0000","last_clean":"2024-02-23T00:37:45.166792+0000","last_became_active":"2024-02-23T00:37:45.165279+0000","last_became_peered":"2024-02-23T00:37:45.165279+0000","last_unstale":"2024-02-23T00:37:45.166792+0000","last_undegraded":"2024-02-23T00:37:45.166792+0000","last_fullsized":"2024-02-23T00:37:45.166792+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"218.b","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.173233+0000","last_change":"2024-02-23T00:37:45.173233+0000","last_active":"2024-02-23T00:37:45.173233+0000","last_peered":"2024-02-23T00:37:45.173233+0000","last_clean":"2024-02-23T00:37:45.173233+0000","last_became_active":"2024-02-23T00:37:45.172966+0000","last_became_peered":"2024-02-23T00:37:45.172966+0000","last_unstale":"2024-02-23T00:37:45.173233+0000","last_undegraded":"2024-02-23T00:37:45.173233+0000","last_fullsized":"2024-02-23T00:37:45.173233+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"218.a","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.141889+0000","last_change":"2024-02-23T00:37:45.141804+0000","last_active":"2024-02-23T00:37:45.141889+0000","last_peered":"2024-02-23T00:37:45.141889+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.141804+0000","last_became_peered":"2024-02-23T00:37:45.141804+0000","last_unstale":"2024-02-23T00:37:45.141889+0000","last_undegraded":"2024-02-23T00:37:45.141889+0000","last_fullsized":"2024-02-23T00:37:45.138679+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"218.9","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.170250+0000","last_change":"2024-02-23T00:37:45.170164+0000","last_active":"2024-02-23T00:37:45.170250+0000","last_peered":"2024-02-23T00:37:45.170250+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.170164+0000","last_became_peered":"2024-02-23T00:37:45.170164+0000","last_unstale":"2024-02-23T00:37:45.170250+0000","last_undegraded":"2024-02-23T00:37:45.170250+0000","last_fullsized":"2024-02-23T00:37:45.154815+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"218.8","version":"464'1","reported_seq":10,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.163569+0000","last_change":"2024-02-23T00:37:45.163218+0000","last_active":"2024-02-23T00:37:45.163569+0000","last_peered":"2024-02-23T00:37:45.163569+0000","last_clean":"2024-02-23T00:37:45.163569+0000","last_became_active":"2024-02-23T00:37:45.162931+0000","last_became_peered":"2024-02-23T00:37:45.162931+0000","last_unstale":"2024-02-23T00:37:45.163569+0000","last_undegraded":"2024-02-23T00:37:45.163569+0000","last_fullsized":"2024-02-23T00:37:45.163569+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"218.7","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.164374+0000","last_change":"2024-02-23T00:37:45.164374+0000","last_active":"2024-02-23T00:37:45.164374+0000","last_peered":"2024-02-23T00:37:45.164374+0000","last_clean":"2024-02-23T00:37:45.164374+0000","last_became_active":"2024-02-23T00:37:45.163947+0000","last_became_peered":"2024-02-23T00:37:45.163947+0000","last_unstale":"2024-02-23T00:37:45.164374+0000","last_undegraded":"2024-02-23T00:37:45.164374+0000","last_fullsized":"2024-02-23T00:37:45.164374+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"218.6","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.136857+0000","last_change":"2024-02-23T00:37:45.136857+0000","last_active":"2024-02-23T00:37:45.136857+0000","last_peered":"2024-02-23T00:37:45.136857+0000","last_clean":"2024-02-23T00:37:45.136857+0000","last_became_active":"2024-02-23T00:37:45.136586+0000","last_became_peered":"2024-02-23T00:37:45.136586+0000","last_unstale":"2024-02-23T00:37:45.136857+0000","last_undegraded":"2024-02-23T00:37:45.136857+0000","last_fullsized":"2024-02-23T00:37:45.136857+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"218.5","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.167319+0000","last_change":"2024-02-23T00:37:45.167250+0000","last_active":"2024-02-23T00:37:45.167319+0000","last_peered":"2024-02-23T00:37:45.167319+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.167250+0000","last_became_peered":"2024-02-23T00:37:45.167250+0000","last_unstale":"2024-02-23T00:37:45.167319+0000","last_undegraded":"2024-02-23T00:37:45.167319+0000","last_fullsized":"2024-02-23T00:37:45.142627+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"218.4","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.156263+0000","last_change":"2024-02-23T00:37:45.156187+0000","last_active":"2024-02-23T00:37:45.156263+0000","last_peered":"2024-02-23T00:37:45.156263+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.156187+0000","last_became_peered":"2024-02-23T00:37:45.156187+0000","last_unstale":"2024-02-23T00:37:45.156263+0000","last_undegraded":"2024-02-23T00:37:45.156263+0000","last_fullsized":"2024-02-23T00:37:45.139292+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"218.3","version":"0'0","reported_seq":9,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.188877+0000","last_change":"2024-02-23T00:37:45.188877+0000","last_active":"2024-02-23T00:37:45.188877+0000","last_peered":"2024-02-23T00:37:45.188877+0000","last_clean":"2024-02-23T00:37:45.188877+0000","last_became_active":"2024-02-23T00:37:45.188472+0000","last_became_peered":"2024-02-23T00:37:45.188472+0000","last_unstale":"2024-02-23T00:37:45.188877+0000","last_undegraded":"2024-02-23T00:37:45.188877+0000","last_fullsized":"2024-02-23T00:37:45.188877+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"218.2","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.161152+0000","last_change":"2024-02-23T00:37:45.161084+0000","last_active":"2024-02-23T00:37:45.161152+0000","last_peered":"2024-02-23T00:37:45.161152+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.161084+0000","last_became_peered":"2024-02-23T00:37:45.161084+0000","last_unstale":"2024-02-23T00:37:45.161152+0000","last_undegraded":"2024-02-23T00:37:45.161152+0000","last_fullsized":"2024-02-23T00:37:45.138362+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"218.1","version":"0'0","reported_seq":6,"reported_epoch":464,"state":"active+undersized","last_fresh":"2024-02-23T00:37:45.140054+0000","last_change":"2024-02-23T00:37:45.139992+0000","last_active":"2024-02-23T00:37:45.140054+0000","last_peered":"2024-02-23T00:37:45.140054+0000","last_clean":"2024-02-23T00:37:44.123507+0000","last_became_active":"2024-02-23T00:37:45.139992+0000","last_became_peered":"2024-02-23T00:37:45.139992+0000","last_unstale":"2024-02-23T00:37:45.140054+0000","last_undegraded":"2024-02-23T00:37:45.140054+0000","last_fullsized":"2024-02-23T00:37:45.134517+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"218.0","version":"464'1","reported_seq":10,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.175240+0000","last_change":"2024-02-23T00:37:45.173764+0000","last_active":"2024-02-23T00:37:45.175240+0000","last_peered":"2024-02-23T00:37:45.175240+0000","last_clean":"2024-02-23T00:37:45.175240+0000","last_became_active":"2024-02-23T00:37:45.173491+0000","last_became_peered":"2024-02-23T00:37:45.173491+0000","last_unstale":"2024-02-23T00:37:45.175240+0000","last_undegraded":"2024-02-23T00:37:45.175240+0000","last_fullsized":"2024-02-23T00:37:45.175240+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":463,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:44.123507+0000","last_clean_scrub_stamp":"2024-02-23T00:37:44.123507+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.0","version":"458'80","reported_seq":220,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.134151+0000","last_change":"2024-02-23T00:36:37.106755+0000","last_active":"2024-02-23T00:37:44.134151+0000","last_peered":"2024-02-23T00:37:44.134151+0000","last_clean":"2024-02-23T00:36:33.365214+0000","last_became_active":"2024-02-23T00:36:37.106755+0000","last_became_peered":"2024-02-23T00:36:37.106755+0000","last_unstale":"2024-02-23T00:37:44.134151+0000","last_undegraded":"2024-02-23T00:36:37.100498+0000","last_fullsized":"2024-02-23T00:36:37.100377+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'68","last_scrub_stamp":"2024-02-23T00:34:57.645204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.645204+0000","log_size":80,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1","version":"340'86","reported_seq":500,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.131566+0000","last_change":"2024-02-23T00:36:37.109838+0000","last_active":"2024-02-23T00:37:44.131566+0000","last_peered":"2024-02-23T00:37:44.131566+0000","last_clean":"2024-02-23T00:36:33.191340+0000","last_became_active":"2024-02-23T00:36:37.109838+0000","last_became_peered":"2024-02-23T00:36:37.109838+0000","last_unstale":"2024-02-23T00:37:44.131566+0000","last_undegraded":"2024-02-23T00:36:37.093708+0000","last_fullsized":"2024-02-23T00:36:37.093560+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'85","last_scrub_stamp":"2024-02-23T00:34:16.724630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:16.724630+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.2","version":"372'21","reported_seq":447,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.133331+0000","last_change":"2024-02-23T00:36:37.106211+0000","last_active":"2024-02-23T00:37:44.133331+0000","last_peered":"2024-02-23T00:37:44.133331+0000","last_clean":"2024-02-23T00:36:33.365672+0000","last_became_active":"2024-02-23T00:36:37.106211+0000","last_became_peered":"2024-02-23T00:36:37.106211+0000","last_unstale":"2024-02-23T00:37:44.133331+0000","last_undegraded":"2024-02-23T00:36:37.098859+0000","last_fullsized":"2024-02-23T00:36:37.098739+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'20","last_scrub_stamp":"2024-02-23T00:34:23.049395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.049395+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.3","version":"451'44","reported_seq":484,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131735+0000","last_change":"2024-02-23T00:37:05.079232+0000","last_active":"2024-02-23T00:37:44.131735+0000","last_peered":"2024-02-23T00:37:44.131735+0000","last_clean":"2024-02-23T00:37:44.131735+0000","last_became_active":"2024-02-23T00:36:17.093628+0000","last_became_peered":"2024-02-23T00:36:17.093628+0000","last_unstale":"2024-02-23T00:37:44.131735+0000","last_undegraded":"2024-02-23T00:37:44.131735+0000","last_fullsized":"2024-02-23T00:37:44.131735+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'27","last_scrub_stamp":"2024-02-23T00:37:05.079149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.079149+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":24,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.4","version":"442'49","reported_seq":464,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.139440+0000","last_change":"2024-02-23T00:36:45.668442+0000","last_active":"2024-02-23T00:37:44.139440+0000","last_peered":"2024-02-23T00:37:44.139440+0000","last_clean":"2024-02-23T00:37:44.139440+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:37:44.139440+0000","last_undegraded":"2024-02-23T00:37:44.139440+0000","last_fullsized":"2024-02-23T00:37:44.139440+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'39","last_scrub_stamp":"2024-02-23T00:34:19.398065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.398065+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.5","version":"234'124","reported_seq":533,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.132750+0000","last_change":"2024-02-23T00:36:37.105105+0000","last_active":"2024-02-23T00:37:44.132750+0000","last_peered":"2024-02-23T00:37:44.132750+0000","last_clean":"2024-02-23T00:36:33.192144+0000","last_became_active":"2024-02-23T00:36:37.105105+0000","last_became_peered":"2024-02-23T00:36:37.105105+0000","last_unstale":"2024-02-23T00:37:44.132750+0000","last_undegraded":"2024-02-23T00:36:37.095637+0000","last_fullsized":"2024-02-23T00:36:37.094980+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'124","last_scrub_stamp":"2024-02-23T00:34:19.695449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.695449+0000","log_size":124,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.6","version":"340'52","reported_seq":476,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.131174+0000","last_change":"2024-02-23T00:36:37.120734+0000","last_active":"2024-02-23T00:37:44.131174+0000","last_peered":"2024-02-23T00:37:44.131174+0000","last_clean":"2024-02-23T00:36:33.194384+0000","last_became_active":"2024-02-23T00:36:37.120734+0000","last_became_peered":"2024-02-23T00:36:37.120734+0000","last_unstale":"2024-02-23T00:37:44.131174+0000","last_undegraded":"2024-02-23T00:36:37.106912+0000","last_fullsized":"2024-02-23T00:36:37.106799+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:34:31.733148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.733148+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.7","version":"401'97","reported_seq":504,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.144677+0000","last_change":"2024-02-23T00:36:45.610239+0000","last_active":"2024-02-23T00:37:44.144677+0000","last_peered":"2024-02-23T00:37:44.144677+0000","last_clean":"2024-02-23T00:37:44.144677+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:37:44.144677+0000","last_undegraded":"2024-02-23T00:37:44.144677+0000","last_fullsized":"2024-02-23T00:37:44.144677+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'92","last_scrub_stamp":"2024-02-23T00:34:18.124049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:18.124049+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":94,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.8","version":"340'71","reported_seq":414,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133091+0000","last_change":"2024-02-23T00:36:45.607171+0000","last_active":"2024-02-23T00:37:44.133091+0000","last_peered":"2024-02-23T00:37:44.133091+0000","last_clean":"2024-02-23T00:37:44.133091+0000","last_became_active":"2024-02-23T00:34:41.115222+0000","last_became_peered":"2024-02-23T00:34:41.115222+0000","last_unstale":"2024-02-23T00:37:44.133091+0000","last_undegraded":"2024-02-23T00:37:44.133091+0000","last_fullsized":"2024-02-23T00:37:44.133091+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'71","last_scrub_stamp":"2024-02-23T00:35:41.651500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.651500+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.9","version":"405'19","reported_seq":439,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133300+0000","last_change":"2024-02-23T00:36:45.606951+0000","last_active":"2024-02-23T00:37:44.133300+0000","last_peered":"2024-02-23T00:37:44.133300+0000","last_clean":"2024-02-23T00:37:44.133300+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:37:44.133300+0000","last_undegraded":"2024-02-23T00:37:44.133300+0000","last_fullsized":"2024-02-23T00:37:44.133300+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'17","last_scrub_stamp":"2024-02-23T00:34:17.107575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.107575+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.a","version":"248'22","reported_seq":444,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.145566+0000","last_change":"2024-02-23T00:36:45.668030+0000","last_active":"2024-02-23T00:37:44.145566+0000","last_peered":"2024-02-23T00:37:44.145566+0000","last_clean":"2024-02-23T00:37:44.145566+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:37:44.145566+0000","last_undegraded":"2024-02-23T00:37:44.145566+0000","last_fullsized":"2024-02-23T00:37:44.145566+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'22","last_scrub_stamp":"2024-02-23T00:33:34.362545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.362545+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.b","version":"340'35","reported_seq":485,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.136755+0000","last_change":"2024-02-23T00:36:37.120097+0000","last_active":"2024-02-23T00:37:44.136755+0000","last_peered":"2024-02-23T00:37:44.136755+0000","last_clean":"2024-02-23T00:36:33.368807+0000","last_became_active":"2024-02-23T00:36:37.120097+0000","last_became_peered":"2024-02-23T00:36:37.120097+0000","last_unstale":"2024-02-23T00:37:44.136755+0000","last_undegraded":"2024-02-23T00:36:37.107185+0000","last_fullsized":"2024-02-23T00:36:37.107054+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-23T00:33:37.391203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.391203+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.c","version":"422'97","reported_seq":513,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.146140+0000","last_change":"2024-02-23T00:36:37.106629+0000","last_active":"2024-02-23T00:37:44.146140+0000","last_peered":"2024-02-23T00:37:44.146140+0000","last_clean":"2024-02-23T00:36:33.366529+0000","last_became_active":"2024-02-23T00:36:37.106629+0000","last_became_peered":"2024-02-23T00:36:37.106629+0000","last_unstale":"2024-02-23T00:37:44.146140+0000","last_undegraded":"2024-02-23T00:36:37.096711+0000","last_fullsized":"2024-02-23T00:36:37.096581+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'76","last_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_deep_scrub":"397'76","last_deep_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.098352+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.d","version":"395'39","reported_seq":484,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136816+0000","last_change":"2024-02-23T00:36:45.669435+0000","last_active":"2024-02-23T00:37:44.136816+0000","last_peered":"2024-02-23T00:37:44.136816+0000","last_clean":"2024-02-23T00:37:44.136816+0000","last_became_active":"2024-02-23T00:34:41.109682+0000","last_became_peered":"2024-02-23T00:34:41.109682+0000","last_unstale":"2024-02-23T00:37:44.136816+0000","last_undegraded":"2024-02-23T00:37:44.136816+0000","last_fullsized":"2024-02-23T00:37:44.136816+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-23T00:33:45.265466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.265466+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.e","version":"405'77","reported_seq":548,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136046+0000","last_change":"2024-02-23T00:36:45.609915+0000","last_active":"2024-02-23T00:37:44.136046+0000","last_peered":"2024-02-23T00:37:44.136046+0000","last_clean":"2024-02-23T00:37:44.136046+0000","last_became_active":"2024-02-23T00:34:41.108381+0000","last_became_peered":"2024-02-23T00:34:41.108381+0000","last_unstale":"2024-02-23T00:37:44.136046+0000","last_undegraded":"2024-02-23T00:37:44.136046+0000","last_fullsized":"2024-02-23T00:37:44.136046+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'74","last_scrub_stamp":"2024-02-23T00:34:15.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.172848+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000073,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":3931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":4000057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.f","version":"219'69","reported_seq":407,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.145024+0000","last_change":"2024-02-23T00:36:45.610425+0000","last_active":"2024-02-23T00:37:44.145024+0000","last_peered":"2024-02-23T00:37:44.145024+0000","last_clean":"2024-02-23T00:37:44.145024+0000","last_became_active":"2024-02-23T00:34:41.101718+0000","last_became_peered":"2024-02-23T00:34:41.101718+0000","last_unstale":"2024-02-23T00:37:44.145024+0000","last_undegraded":"2024-02-23T00:37:44.145024+0000","last_fullsized":"2024-02-23T00:37:44.145024+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'69","last_scrub_stamp":"2024-02-23T00:35:46.750022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.750022+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.10","version":"401'37","reported_seq":454,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.133223+0000","last_change":"2024-02-23T00:36:37.119458+0000","last_active":"2024-02-23T00:37:44.133223+0000","last_peered":"2024-02-23T00:37:44.133223+0000","last_clean":"2024-02-23T00:36:33.364885+0000","last_became_active":"2024-02-23T00:36:37.119458+0000","last_became_peered":"2024-02-23T00:36:37.119458+0000","last_unstale":"2024-02-23T00:37:44.133223+0000","last_undegraded":"2024-02-23T00:36:37.098703+0000","last_fullsized":"2024-02-23T00:36:37.098584+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'34","last_scrub_stamp":"2024-02-23T00:35:01.791840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.791840+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.11","version":"392'50","reported_seq":471,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131533+0000","last_change":"2024-02-23T00:37:06.042258+0000","last_active":"2024-02-23T00:37:44.131533+0000","last_peered":"2024-02-23T00:37:44.131533+0000","last_clean":"2024-02-23T00:37:44.131533+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:37:44.131533+0000","last_undegraded":"2024-02-23T00:37:44.131533+0000","last_fullsized":"2024-02-23T00:37:44.131533+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'50","last_scrub_stamp":"2024-02-23T00:37:06.042181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:06.042181+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.28","version":"0'0","reported_seq":93,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.135513+0000","last_change":"2024-02-23T00:35:29.092753+0000","last_active":"2024-02-23T00:37:44.135513+0000","last_peered":"2024-02-23T00:37:44.135513+0000","last_clean":"2024-02-23T00:37:44.135513+0000","last_became_active":"2024-02-23T00:35:29.092396+0000","last_became_peered":"2024-02-23T00:35:29.092396+0000","last_unstale":"2024-02-23T00:37:44.135513+0000","last_undegraded":"2024-02-23T00:37:44.135513+0000","last_fullsized":"2024-02-23T00:37:44.135513+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.815722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.815722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.12","version":"204'68","reported_seq":477,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.138123+0000","last_change":"2024-02-23T00:36:37.114447+0000","last_active":"2024-02-23T00:37:44.138123+0000","last_peered":"2024-02-23T00:37:44.138123+0000","last_clean":"2024-02-23T00:36:33.364627+0000","last_became_active":"2024-02-23T00:36:37.114447+0000","last_became_peered":"2024-02-23T00:36:37.114447+0000","last_unstale":"2024-02-23T00:37:44.138123+0000","last_undegraded":"2024-02-23T00:36:37.099733+0000","last_fullsized":"2024-02-23T00:36:37.099613+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'68","last_scrub_stamp":"2024-02-23T00:36:27.078990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.078990+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.29","version":"62'8","reported_seq":503,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.142222+0000","last_change":"2024-02-23T00:36:37.122045+0000","last_active":"2024-02-23T00:37:44.142222+0000","last_peered":"2024-02-23T00:37:44.142222+0000","last_clean":"2024-02-23T00:35:27.100547+0000","last_became_active":"2024-02-23T00:36:37.122045+0000","last_became_peered":"2024-02-23T00:36:37.122045+0000","last_unstale":"2024-02-23T00:37:44.142222+0000","last_undegraded":"2024-02-23T00:36:37.107771+0000","last_fullsized":"2024-02-23T00:36:37.107625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:27.673481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.673481+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.13","version":"248'61","reported_seq":479,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136363+0000","last_change":"2024-02-23T00:36:45.667732+0000","last_active":"2024-02-23T00:37:44.136363+0000","last_peered":"2024-02-23T00:37:44.136363+0000","last_clean":"2024-02-23T00:37:44.136363+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:37:44.136363+0000","last_undegraded":"2024-02-23T00:37:44.136363+0000","last_fullsized":"2024-02-23T00:37:44.136363+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'61","last_scrub_stamp":"2024-02-23T00:36:38.944617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:38.944617+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.14","version":"208'85","reported_seq":497,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.145907+0000","last_change":"2024-02-23T00:36:37.115695+0000","last_active":"2024-02-23T00:37:44.145907+0000","last_peered":"2024-02-23T00:37:44.145907+0000","last_clean":"2024-02-23T00:36:33.366438+0000","last_became_active":"2024-02-23T00:36:37.115695+0000","last_became_peered":"2024-02-23T00:36:37.115695+0000","last_unstale":"2024-02-23T00:37:44.145907+0000","last_undegraded":"2024-02-23T00:36:37.099640+0000","last_fullsized":"2024-02-23T00:36:37.099515+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'85","last_scrub_stamp":"2024-02-23T00:35:46.085280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.085280+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.15","version":"340'70","reported_seq":490,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.132588+0000","last_change":"2024-02-23T00:37:04.031754+0000","last_active":"2024-02-23T00:37:44.132588+0000","last_peered":"2024-02-23T00:37:44.132588+0000","last_clean":"2024-02-23T00:37:44.132588+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:37:44.132588+0000","last_undegraded":"2024-02-23T00:37:44.132588+0000","last_fullsized":"2024-02-23T00:37:44.132588+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'70","last_scrub_stamp":"2024-02-23T00:37:04.031654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:04.031654+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.16","version":"204'51","reported_seq":476,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.131008+0000","last_change":"2024-02-23T00:36:37.111665+0000","last_active":"2024-02-23T00:37:44.131008+0000","last_peered":"2024-02-23T00:37:44.131008+0000","last_clean":"2024-02-23T00:36:33.194334+0000","last_became_active":"2024-02-23T00:36:37.111665+0000","last_became_peered":"2024-02-23T00:36:37.111665+0000","last_unstale":"2024-02-23T00:37:44.131008+0000","last_undegraded":"2024-02-23T00:36:37.100202+0000","last_fullsized":"2024-02-23T00:36:37.100076+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:35:46.192706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.192706+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.17","version":"433'112","reported_seq":522,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143159+0000","last_change":"2024-02-23T00:36:47.692240+0000","last_active":"2024-02-23T00:37:44.143159+0000","last_peered":"2024-02-23T00:37:44.143159+0000","last_clean":"2024-02-23T00:37:44.143159+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:37:44.143159+0000","last_undegraded":"2024-02-23T00:37:44.143159+0000","last_fullsized":"2024-02-23T00:37:44.143159+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'86","last_scrub_stamp":"2024-02-23T00:33:35.383425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:35.383425+0000","log_size":112,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.22","version":"0'0","reported_seq":100,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.150260+0000","last_change":"2024-02-23T00:37:09.026463+0000","last_active":"2024-02-23T00:37:44.150260+0000","last_peered":"2024-02-23T00:37:44.150260+0000","last_clean":"2024-02-23T00:37:44.150260+0000","last_became_active":"2024-02-23T00:35:29.089082+0000","last_became_peered":"2024-02-23T00:35:29.089082+0000","last_unstale":"2024-02-23T00:37:44.150260+0000","last_undegraded":"2024-02-23T00:37:44.150260+0000","last_fullsized":"2024-02-23T00:37:44.150260+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.026383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.026383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.18","version":"211'57","reported_seq":477,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134553+0000","last_change":"2024-02-23T00:36:45.668902+0000","last_active":"2024-02-23T00:37:44.134553+0000","last_peered":"2024-02-23T00:37:44.134553+0000","last_clean":"2024-02-23T00:37:44.134553+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:37:44.134553+0000","last_undegraded":"2024-02-23T00:37:44.134553+0000","last_fullsized":"2024-02-23T00:37:44.134553+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'57","last_scrub_stamp":"2024-02-23T00:35:54.857184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:54.857184+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.23","version":"426'11","reported_seq":136,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:45.150992+0000","last_change":"2024-02-23T00:36:37.112756+0000","last_active":"2024-02-23T00:37:45.150992+0000","last_peered":"2024-02-23T00:37:45.150992+0000","last_clean":"2024-02-23T00:36:35.444860+0000","last_became_active":"2024-02-23T00:36:37.112756+0000","last_became_peered":"2024-02-23T00:36:37.112756+0000","last_unstale":"2024-02-23T00:37:45.150992+0000","last_undegraded":"2024-02-23T00:36:37.094265+0000","last_fullsized":"2024-02-23T00:36:37.094133+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.19","version":"464'61","reported_seq":487,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.141284+0000","last_change":"2024-02-23T00:36:45.297690+0000","last_active":"2024-02-23T00:37:45.141284+0000","last_peered":"2024-02-23T00:37:45.141284+0000","last_clean":"2024-02-23T00:37:45.141284+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:37:45.141284+0000","last_undegraded":"2024-02-23T00:37:45.141284+0000","last_fullsized":"2024-02-23T00:37:45.141284+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'60","last_scrub_stamp":"2024-02-23T00:36:31.102608+0000","last_deep_scrub":"271'54","last_deep_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_clean_scrub_stamp":"2024-02-23T00:36:31.102608+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.20","version":"379'11","reported_seq":133,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.131090+0000","last_change":"2024-02-23T00:37:10.032410+0000","last_active":"2024-02-23T00:37:45.131090+0000","last_peered":"2024-02-23T00:37:45.131090+0000","last_clean":"2024-02-23T00:37:45.131090+0000","last_became_active":"2024-02-23T00:35:29.093220+0000","last_became_peered":"2024-02-23T00:35:29.093220+0000","last_unstale":"2024-02-23T00:37:45.131090+0000","last_undegraded":"2024-02-23T00:37:45.131090+0000","last_fullsized":"2024-02-23T00:37:45.131090+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"379'11","last_scrub_stamp":"2024-02-23T00:37:10.032341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:10.032341+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1a","version":"202'68","reported_seq":490,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.130346+0000","last_change":"2024-02-23T00:37:15.836653+0000","last_active":"2024-02-23T00:37:44.130346+0000","last_peered":"2024-02-23T00:37:44.130346+0000","last_clean":"2024-02-23T00:37:44.130346+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:37:44.130346+0000","last_undegraded":"2024-02-23T00:37:44.130346+0000","last_fullsized":"2024-02-23T00:37:44.130346+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'68","last_scrub_stamp":"2024-02-23T00:37:15.836572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:15.836572+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.21","version":"0'0","reported_seq":98,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.138212+0000","last_change":"2024-02-23T00:36:34.152721+0000","last_active":"2024-02-23T00:37:44.138212+0000","last_peered":"2024-02-23T00:37:44.138212+0000","last_clean":"2024-02-23T00:37:44.138212+0000","last_became_active":"2024-02-23T00:35:29.093053+0000","last_became_peered":"2024-02-23T00:35:29.093053+0000","last_unstale":"2024-02-23T00:37:44.138212+0000","last_undegraded":"2024-02-23T00:37:44.138212+0000","last_fullsized":"2024-02-23T00:37:44.138212+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.152643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:34.152643+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.152643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.1b","version":"398'54","reported_seq":468,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134922+0000","last_change":"2024-02-23T00:36:45.666523+0000","last_active":"2024-02-23T00:37:44.134922+0000","last_peered":"2024-02-23T00:37:44.134922+0000","last_clean":"2024-02-23T00:37:44.134922+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:37:44.134922+0000","last_undegraded":"2024-02-23T00:37:44.134922+0000","last_fullsized":"2024-02-23T00:37:44.134922+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-02-23T00:33:46.274400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.274400+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.26","version":"0'0","reported_seq":448,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.139908+0000","last_change":"2024-02-23T00:36:37.122228+0000","last_active":"2024-02-23T00:37:44.139908+0000","last_peered":"2024-02-23T00:37:44.139908+0000","last_clean":"2024-02-23T00:36:33.367599+0000","last_became_active":"2024-02-23T00:36:37.122228+0000","last_became_peered":"2024-02-23T00:36:37.122228+0000","last_unstale":"2024-02-23T00:37:44.139908+0000","last_undegraded":"2024-02-23T00:37:44.139908+0000","last_fullsized":"2024-02-23T00:36:37.110661+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1c","version":"216'34","reported_seq":480,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134252+0000","last_change":"2024-02-23T00:36:45.608757+0000","last_active":"2024-02-23T00:37:44.134252+0000","last_peered":"2024-02-23T00:37:44.134252+0000","last_clean":"2024-02-23T00:37:44.134252+0000","last_became_active":"2024-02-23T00:34:41.117846+0000","last_became_peered":"2024-02-23T00:34:41.117846+0000","last_unstale":"2024-02-23T00:37:44.134252+0000","last_undegraded":"2024-02-23T00:37:44.134252+0000","last_fullsized":"2024-02-23T00:37:44.134252+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'34","last_scrub_stamp":"2024-02-23T00:34:57.859047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.859047+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.27","version":"0'0","reported_seq":432,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.144456+0000","last_change":"2024-02-23T00:35:29.092282+0000","last_active":"2024-02-23T00:37:44.144456+0000","last_peered":"2024-02-23T00:37:44.144456+0000","last_clean":"2024-02-23T00:37:44.144456+0000","last_became_active":"2024-02-23T00:35:29.091970+0000","last_became_peered":"2024-02-23T00:35:29.091970+0000","last_unstale":"2024-02-23T00:37:44.144456+0000","last_undegraded":"2024-02-23T00:37:44.144456+0000","last_fullsized":"2024-02-23T00:37:44.144456+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1d","version":"340'13","reported_seq":177,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.140935+0000","last_change":"2024-02-23T00:36:45.408437+0000","last_active":"2024-02-23T00:37:44.140935+0000","last_peered":"2024-02-23T00:37:44.140935+0000","last_clean":"2024-02-23T00:37:44.140935+0000","last_became_active":"2024-02-23T00:34:42.058724+0000","last_became_peered":"2024-02-23T00:34:42.058724+0000","last_unstale":"2024-02-23T00:37:44.140935+0000","last_undegraded":"2024-02-23T00:37:44.140935+0000","last_fullsized":"2024-02-23T00:37:44.140935+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'13","last_scrub_stamp":"2024-02-23T00:36:40.328657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:40.328657+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.24","version":"0'0","reported_seq":444,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.136936+0000","last_change":"2024-02-23T00:36:37.117551+0000","last_active":"2024-02-23T00:37:44.136936+0000","last_peered":"2024-02-23T00:37:44.136936+0000","last_clean":"2024-02-23T00:36:33.366613+0000","last_became_active":"2024-02-23T00:36:37.117551+0000","last_became_peered":"2024-02-23T00:36:37.117551+0000","last_unstale":"2024-02-23T00:37:44.136936+0000","last_undegraded":"2024-02-23T00:37:44.136936+0000","last_fullsized":"2024-02-23T00:36:37.104246+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1e","version":"340'52","reported_seq":462,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136802+0000","last_change":"2024-02-23T00:36:45.669526+0000","last_active":"2024-02-23T00:37:44.136802+0000","last_peered":"2024-02-23T00:37:44.136802+0000","last_clean":"2024-02-23T00:37:44.136802+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:37:44.136802+0000","last_undegraded":"2024-02-23T00:37:44.136802+0000","last_fullsized":"2024-02-23T00:37:44.136802+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'51","last_scrub_stamp":"2024-02-23T00:33:49.219607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.219607+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.25","version":"0'0","reported_seq":441,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.135825+0000","last_change":"2024-02-23T00:35:29.091129+0000","last_active":"2024-02-23T00:37:44.135825+0000","last_peered":"2024-02-23T00:37:44.135825+0000","last_clean":"2024-02-23T00:37:44.135825+0000","last_became_active":"2024-02-23T00:35:29.090793+0000","last_became_peered":"2024-02-23T00:35:29.090793+0000","last_unstale":"2024-02-23T00:37:44.135825+0000","last_undegraded":"2024-02-23T00:37:44.135825+0000","last_fullsized":"2024-02-23T00:37:44.135825+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1f","version":"439'74","reported_seq":488,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.142968+0000","last_change":"2024-02-23T00:36:37.117847+0000","last_active":"2024-02-23T00:37:44.142968+0000","last_peered":"2024-02-23T00:37:44.142968+0000","last_clean":"2024-02-23T00:36:33.365701+0000","last_became_active":"2024-02-23T00:36:37.117847+0000","last_became_peered":"2024-02-23T00:36:37.117847+0000","last_unstale":"2024-02-23T00:37:44.142968+0000","last_undegraded":"2024-02-23T00:36:37.102042+0000","last_fullsized":"2024-02-23T00:36:37.101901+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-23T00:33:39.351573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.351573+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"379'9","reported_seq":582,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.131768+0000","last_change":"2024-02-23T00:37:05.955772+0000","last_active":"2024-02-23T00:37:45.131768+0000","last_peered":"2024-02-23T00:37:45.131768+0000","last_clean":"2024-02-23T00:37:45.131768+0000","last_became_active":"2024-02-23T00:35:27.095085+0000","last_became_peered":"2024-02-23T00:35:27.095085+0000","last_unstale":"2024-02-23T00:37:45.131768+0000","last_undegraded":"2024-02-23T00:37:45.131768+0000","last_fullsized":"2024-02-23T00:37:45.131768+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'9","last_scrub_stamp":"2024-02-23T00:37:05.955707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.955707+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":538,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136058+0000","last_change":"2024-02-23T00:37:22.011162+0000","last_active":"2024-02-23T00:37:44.136058+0000","last_peered":"2024-02-23T00:37:44.136058+0000","last_clean":"2024-02-23T00:37:44.136058+0000","last_became_active":"2024-02-23T00:35:13.074767+0000","last_became_peered":"2024-02-23T00:35:13.074767+0000","last_unstale":"2024-02-23T00:37:44.136058+0000","last_undegraded":"2024-02-23T00:37:44.136058+0000","last_fullsized":"2024-02-23T00:37:44.136058+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":106,"num_read_kb":88,"num_write":62,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":457,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131374+0000","last_change":"2024-02-23T00:37:05.837736+0000","last_active":"2024-02-23T00:37:44.131374+0000","last_peered":"2024-02-23T00:37:44.131374+0000","last_clean":"2024-02-23T00:37:44.131374+0000","last_became_active":"2024-02-23T00:34:41.106070+0000","last_became_peered":"2024-02-23T00:34:41.106070+0000","last_unstale":"2024-02-23T00:37:44.131374+0000","last_undegraded":"2024-02-23T00:37:44.131374+0000","last_fullsized":"2024-02-23T00:37:44.131374+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:05.837666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.837666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":426,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.145875+0000","last_change":"2024-02-23T00:36:37.104200+0000","last_active":"2024-02-23T00:37:44.145875+0000","last_peered":"2024-02-23T00:37:44.145875+0000","last_clean":"2024-02-23T00:36:33.366159+0000","last_became_active":"2024-02-23T00:36:37.104200+0000","last_became_peered":"2024-02-23T00:36:37.104200+0000","last_unstale":"2024-02-23T00:37:44.145875+0000","last_undegraded":"2024-02-23T00:37:44.145875+0000","last_fullsized":"2024-02-23T00:36:37.102434+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:21.441958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.441958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":390,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.134053+0000","last_change":"2024-02-23T00:36:37.115879+0000","last_active":"2024-02-23T00:37:44.134053+0000","last_peered":"2024-02-23T00:37:44.134053+0000","last_clean":"2024-02-23T00:36:33.366205+0000","last_became_active":"2024-02-23T00:36:37.115879+0000","last_became_peered":"2024-02-23T00:36:37.115879+0000","last_unstale":"2024-02-23T00:37:44.134053+0000","last_undegraded":"2024-02-23T00:37:44.134053+0000","last_fullsized":"2024-02-23T00:36:37.101967+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":735,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.145677+0000","last_change":"2024-02-23T00:36:27.182035+0000","last_active":"2024-02-23T00:37:45.145677+0000","last_peered":"2024-02-23T00:37:45.145677+0000","last_clean":"2024-02-23T00:37:45.145677+0000","last_became_active":"2024-02-23T00:34:41.114058+0000","last_became_peered":"2024-02-23T00:34:41.114058+0000","last_unstale":"2024-02-23T00:37:45.145677+0000","last_undegraded":"2024-02-23T00:37:45.145677+0000","last_fullsized":"2024-02-23T00:37:45.145677+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:36:27.181972+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.181972+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":307,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":102,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.134356+0000","last_change":"2024-02-23T00:36:37.112695+0000","last_active":"2024-02-23T00:37:44.134356+0000","last_peered":"2024-02-23T00:37:44.134356+0000","last_clean":"2024-02-23T00:27:29.183258+0000","last_became_active":"2024-02-23T00:36:37.112695+0000","last_became_peered":"2024-02-23T00:36:37.112695+0000","last_unstale":"2024-02-23T00:37:44.134356+0000","last_undegraded":"2024-02-23T00:36:37.098637+0000","last_fullsized":"2024-02-23T00:36:37.098502+0000","mapping_epoch":425,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":62,"num_read_kb":50,"num_write":36,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":442,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136438+0000","last_change":"2024-02-23T00:36:41.532683+0000","last_active":"2024-02-23T00:37:44.136438+0000","last_peered":"2024-02-23T00:37:44.136438+0000","last_clean":"2024-02-23T00:37:44.136438+0000","last_became_active":"2024-02-23T00:34:41.100003+0000","last_became_peered":"2024-02-23T00:34:41.100003+0000","last_unstale":"2024-02-23T00:37:44.136438+0000","last_undegraded":"2024-02-23T00:37:44.136438+0000","last_fullsized":"2024-02-23T00:37:44.136438+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.532619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.532619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":449,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.130529+0000","last_change":"2024-02-23T00:36:37.107478+0000","last_active":"2024-02-23T00:37:44.130529+0000","last_peered":"2024-02-23T00:37:44.130529+0000","last_clean":"2024-02-23T00:36:33.193986+0000","last_became_active":"2024-02-23T00:36:37.107478+0000","last_became_peered":"2024-02-23T00:36:37.107478+0000","last_unstale":"2024-02-23T00:37:44.130529+0000","last_undegraded":"2024-02-23T00:37:44.130529+0000","last_fullsized":"2024-02-23T00:36:37.103166+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.679076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":46,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.150424+0000","last_change":"2024-02-23T00:36:37.122229+0000","last_active":"2024-02-23T00:37:44.150424+0000","last_peered":"2024-02-23T00:37:44.150424+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.122229+0000","last_became_peered":"2024-02-23T00:36:37.122229+0000","last_unstale":"2024-02-23T00:37:44.150424+0000","last_undegraded":"2024-02-23T00:37:44.150424+0000","last_fullsized":"2024-02-23T00:36:37.107921+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":380,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.144583+0000","last_change":"2024-02-23T00:35:54.029484+0000","last_active":"2024-02-23T00:37:44.144583+0000","last_peered":"2024-02-23T00:37:44.144583+0000","last_clean":"2024-02-23T00:37:44.144583+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:37:44.144583+0000","last_undegraded":"2024-02-23T00:37:44.144583+0000","last_fullsized":"2024-02-23T00:37:44.144583+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:54.029412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:54.029412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"453'47","reported_seq":533,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.144867+0000","last_change":"2024-02-23T00:36:37.230412+0000","last_active":"2024-02-23T00:37:44.144867+0000","last_peered":"2024-02-23T00:37:44.144867+0000","last_clean":"2024-02-23T00:37:44.144867+0000","last_became_active":"2024-02-23T00:34:59.084636+0000","last_became_peered":"2024-02-23T00:34:59.084636+0000","last_unstale":"2024-02-23T00:37:44.144867+0000","last_undegraded":"2024-02-23T00:37:44.144867+0000","last_fullsized":"2024-02-23T00:37:44.144867+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:36:37.230351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:37.230351+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":116,"num_read_kb":96,"num_write":70,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":449,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.131401+0000","last_change":"2024-02-23T00:36:37.116992+0000","last_active":"2024-02-23T00:37:44.131401+0000","last_peered":"2024-02-23T00:37:44.131401+0000","last_clean":"2024-02-23T00:36:33.194161+0000","last_became_active":"2024-02-23T00:36:37.116992+0000","last_became_peered":"2024-02-23T00:36:37.116992+0000","last_unstale":"2024-02-23T00:37:44.131401+0000","last_undegraded":"2024-02-23T00:37:44.131401+0000","last_fullsized":"2024-02-23T00:36:37.104222+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:50.194656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:50.194656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":440,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.131614+0000","last_change":"2024-02-23T00:36:37.113419+0000","last_active":"2024-02-23T00:37:44.131614+0000","last_peered":"2024-02-23T00:37:44.131614+0000","last_clean":"2024-02-23T00:36:33.194007+0000","last_became_active":"2024-02-23T00:36:37.113419+0000","last_became_peered":"2024-02-23T00:36:37.113419+0000","last_unstale":"2024-02-23T00:37:44.131614+0000","last_undegraded":"2024-02-23T00:37:44.131614+0000","last_fullsized":"2024-02-23T00:36:37.104161+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.745865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.745865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":438,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134909+0000","last_change":"2024-02-23T00:35:04.673936+0000","last_active":"2024-02-23T00:37:44.134909+0000","last_peered":"2024-02-23T00:37:44.134909+0000","last_clean":"2024-02-23T00:37:44.134909+0000","last_became_active":"2024-02-23T00:34:41.105277+0000","last_became_peered":"2024-02-23T00:34:41.105277+0000","last_unstale":"2024-02-23T00:37:44.134909+0000","last_undegraded":"2024-02-23T00:37:44.134909+0000","last_fullsized":"2024-02-23T00:37:44.134909+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.673837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.673837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":47,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.130540+0000","last_change":"2024-02-23T00:36:37.119721+0000","last_active":"2024-02-23T00:37:44.130540+0000","last_peered":"2024-02-23T00:37:44.130540+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119721+0000","last_became_peered":"2024-02-23T00:36:37.119721+0000","last_unstale":"2024-02-23T00:37:44.130540+0000","last_undegraded":"2024-02-23T00:37:44.130540+0000","last_fullsized":"2024-02-23T00:36:37.102163+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.205312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.205312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":447,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.145393+0000","last_change":"2024-02-23T00:37:12.370077+0000","last_active":"2024-02-23T00:37:44.145393+0000","last_peered":"2024-02-23T00:37:44.145393+0000","last_clean":"2024-02-23T00:37:44.145393+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:37:44.145393+0000","last_undegraded":"2024-02-23T00:37:44.145393+0000","last_fullsized":"2024-02-23T00:37:44.145393+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:12.369966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:12.369966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":511,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133371+0000","last_change":"2024-02-23T00:36:25.082682+0000","last_active":"2024-02-23T00:37:44.133371+0000","last_peered":"2024-02-23T00:37:44.133371+0000","last_clean":"2024-02-23T00:37:44.133371+0000","last_became_active":"2024-02-23T00:34:47.062796+0000","last_became_peered":"2024-02-23T00:34:47.062796+0000","last_unstale":"2024-02-23T00:37:44.133371+0000","last_undegraded":"2024-02-23T00:37:44.133371+0000","last_fullsized":"2024-02-23T00:37:44.133371+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":96,"num_read_kb":76,"num_write":58,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":435,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.137067+0000","last_change":"2024-02-23T00:35:40.824848+0000","last_active":"2024-02-23T00:37:44.137067+0000","last_peered":"2024-02-23T00:37:44.137067+0000","last_clean":"2024-02-23T00:37:44.137067+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:37:44.137067+0000","last_undegraded":"2024-02-23T00:37:44.137067+0000","last_fullsized":"2024-02-23T00:37:44.137067+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:40.824741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:40.824741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":430,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.134437+0000","last_change":"2024-02-23T00:36:37.109480+0000","last_active":"2024-02-23T00:37:44.134437+0000","last_peered":"2024-02-23T00:37:44.134437+0000","last_clean":"2024-02-23T00:36:34.605757+0000","last_became_active":"2024-02-23T00:36:37.109480+0000","last_became_peered":"2024-02-23T00:36:37.109480+0000","last_unstale":"2024-02-23T00:37:44.134437+0000","last_undegraded":"2024-02-23T00:37:44.134437+0000","last_fullsized":"2024-02-23T00:36:37.095167+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.605664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.605664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":453,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131183+0000","last_change":"2024-02-23T00:37:17.822884+0000","last_active":"2024-02-23T00:37:44.131183+0000","last_peered":"2024-02-23T00:37:44.131183+0000","last_clean":"2024-02-23T00:37:44.131183+0000","last_became_active":"2024-02-23T00:34:41.090777+0000","last_became_peered":"2024-02-23T00:34:41.090777+0000","last_unstale":"2024-02-23T00:37:44.131183+0000","last_undegraded":"2024-02-23T00:37:44.131183+0000","last_fullsized":"2024-02-23T00:37:44.131183+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:17.822776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.822776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":398,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.149346+0000","last_change":"2024-02-23T00:36:37.124688+0000","last_active":"2024-02-23T00:37:44.149346+0000","last_peered":"2024-02-23T00:37:44.149346+0000","last_clean":"2024-02-23T00:36:33.367188+0000","last_became_active":"2024-02-23T00:36:37.124688+0000","last_became_peered":"2024-02-23T00:36:37.124688+0000","last_unstale":"2024-02-23T00:37:44.149346+0000","last_undegraded":"2024-02-23T00:37:44.149346+0000","last_fullsized":"2024-02-23T00:36:37.113140+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.704478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.704478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":156,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.131842+0000","last_change":"2024-02-23T00:36:37.113121+0000","last_active":"2024-02-23T00:37:44.131842+0000","last_peered":"2024-02-23T00:37:44.131842+0000","last_clean":"2024-02-23T00:31:25.058059+0000","last_became_active":"2024-02-23T00:36:37.113121+0000","last_became_peered":"2024-02-23T00:36:37.113121+0000","last_unstale":"2024-02-23T00:37:44.131842+0000","last_undegraded":"2024-02-23T00:37:44.131842+0000","last_fullsized":"2024-02-23T00:36:37.095087+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":522,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.132553+0000","last_change":"2024-02-23T00:37:09.133547+0000","last_active":"2024-02-23T00:37:44.132553+0000","last_peered":"2024-02-23T00:37:44.132553+0000","last_clean":"2024-02-23T00:37:44.132553+0000","last_became_active":"2024-02-23T00:34:51.102216+0000","last_became_peered":"2024-02-23T00:34:51.102216+0000","last_unstale":"2024-02-23T00:37:44.132553+0000","last_undegraded":"2024-02-23T00:37:44.132553+0000","last_fullsized":"2024-02-23T00:37:44.132553+0000","mapping_epoch":329,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:37:09.133471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.133471+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":48,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":455,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133732+0000","last_change":"2024-02-23T00:35:56.678738+0000","last_active":"2024-02-23T00:37:44.133732+0000","last_peered":"2024-02-23T00:37:44.133732+0000","last_clean":"2024-02-23T00:37:44.133732+0000","last_became_active":"2024-02-23T00:34:41.115434+0000","last_became_peered":"2024-02-23T00:34:41.115434+0000","last_unstale":"2024-02-23T00:37:44.133732+0000","last_undegraded":"2024-02-23T00:37:44.133732+0000","last_fullsized":"2024-02-23T00:37:44.133732+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.678676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.678676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":573,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.132361+0000","last_change":"2024-02-23T00:37:07.822516+0000","last_active":"2024-02-23T00:37:45.132361+0000","last_peered":"2024-02-23T00:37:45.132361+0000","last_clean":"2024-02-23T00:37:45.132361+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:37:45.132361+0000","last_undegraded":"2024-02-23T00:37:45.132361+0000","last_fullsized":"2024-02-23T00:37:45.132361+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:37:07.822449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:07.822449+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143924+0000","last_change":"2024-02-23T00:35:56.258132+0000","last_active":"2024-02-23T00:37:44.143924+0000","last_peered":"2024-02-23T00:37:44.143924+0000","last_clean":"2024-02-23T00:37:44.143924+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:37:44.143924+0000","last_undegraded":"2024-02-23T00:37:44.143924+0000","last_fullsized":"2024-02-23T00:37:44.143924+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.258056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.258056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":384,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.144003+0000","last_change":"2024-02-23T00:35:46.175575+0000","last_active":"2024-02-23T00:37:44.144003+0000","last_peered":"2024-02-23T00:37:44.144003+0000","last_clean":"2024-02-23T00:37:44.144003+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:37:44.144003+0000","last_undegraded":"2024-02-23T00:37:44.144003+0000","last_fullsized":"2024-02-23T00:37:44.144003+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:46.175476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.175476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":503,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133051+0000","last_change":"2024-02-23T00:37:22.396292+0000","last_active":"2024-02-23T00:37:44.133051+0000","last_peered":"2024-02-23T00:37:44.133051+0000","last_clean":"2024-02-23T00:37:44.133051+0000","last_became_active":"2024-02-23T00:34:41.103369+0000","last_became_peered":"2024-02-23T00:34:41.103369+0000","last_unstale":"2024-02-23T00:37:44.133051+0000","last_undegraded":"2024-02-23T00:37:44.133051+0000","last_fullsized":"2024-02-23T00:37:44.133051+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:22.396222+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.396222+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":572,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.145045+0000","last_change":"2024-02-23T00:36:39.442794+0000","last_active":"2024-02-23T00:37:44.145045+0000","last_peered":"2024-02-23T00:37:44.145045+0000","last_clean":"2024-02-23T00:37:44.145045+0000","last_became_active":"2024-02-23T00:35:47.079526+0000","last_became_peered":"2024-02-23T00:35:47.079526+0000","last_unstale":"2024-02-23T00:37:44.145045+0000","last_undegraded":"2024-02-23T00:37:44.145045+0000","last_fullsized":"2024-02-23T00:37:44.145045+0000","mapping_epoch":392,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":86,"num_read_kb":69,"num_write":54,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":439,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133404+0000","last_change":"2024-02-23T00:35:06.658103+0000","last_active":"2024-02-23T00:37:44.133404+0000","last_peered":"2024-02-23T00:37:44.133404+0000","last_clean":"2024-02-23T00:37:44.133404+0000","last_became_active":"2024-02-23T00:34:41.107670+0000","last_became_peered":"2024-02-23T00:34:41.107670+0000","last_unstale":"2024-02-23T00:37:44.133404+0000","last_undegraded":"2024-02-23T00:37:44.133404+0000","last_fullsized":"2024-02-23T00:37:44.133404+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.657998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"426'9","reported_seq":574,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:45.152525+0000","last_change":"2024-02-23T00:36:37.111178+0000","last_active":"2024-02-23T00:37:45.152525+0000","last_peered":"2024-02-23T00:37:45.152525+0000","last_clean":"2024-02-23T00:36:35.444981+0000","last_became_active":"2024-02-23T00:36:37.111178+0000","last_became_peered":"2024-02-23T00:36:37.111178+0000","last_unstale":"2024-02-23T00:37:45.152525+0000","last_undegraded":"2024-02-23T00:36:37.095825+0000","last_fullsized":"2024-02-23T00:36:37.095713+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:59.630171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:59.630171+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":427,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.139305+0000","last_change":"2024-02-23T00:36:37.113156+0000","last_active":"2024-02-23T00:37:44.139305+0000","last_peered":"2024-02-23T00:37:44.139305+0000","last_clean":"2024-02-23T00:36:33.366740+0000","last_became_active":"2024-02-23T00:36:37.113156+0000","last_became_peered":"2024-02-23T00:36:37.113156+0000","last_unstale":"2024-02-23T00:37:44.139305+0000","last_undegraded":"2024-02-23T00:36:37.102218+0000","last_fullsized":"2024-02-23T00:36:37.102103+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:34:17.437083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.437083+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":375,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136026+0000","last_change":"2024-02-23T00:33:39.140496+0000","last_active":"2024-02-23T00:37:44.136026+0000","last_peered":"2024-02-23T00:37:44.136026+0000","last_clean":"2024-02-23T00:37:44.136026+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:37:44.136026+0000","last_undegraded":"2024-02-23T00:37:44.136026+0000","last_fullsized":"2024-02-23T00:37:44.136026+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:39.140434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.140434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":446,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131273+0000","last_change":"2024-02-23T00:35:01.614406+0000","last_active":"2024-02-23T00:37:44.131273+0000","last_peered":"2024-02-23T00:37:44.131273+0000","last_clean":"2024-02-23T00:37:44.131273+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:37:44.131273+0000","last_undegraded":"2024-02-23T00:37:44.131273+0000","last_fullsized":"2024-02-23T00:37:44.131273+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:01.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.614340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":520,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143500+0000","last_change":"2024-02-23T00:35:10.593944+0000","last_active":"2024-02-23T00:37:44.143500+0000","last_peered":"2024-02-23T00:37:44.143500+0000","last_clean":"2024-02-23T00:37:44.143500+0000","last_became_active":"2024-02-23T00:34:43.058886+0000","last_became_peered":"2024-02-23T00:34:43.058886+0000","last_unstale":"2024-02-23T00:37:44.143500+0000","last_undegraded":"2024-02-23T00:37:44.143500+0000","last_fullsized":"2024-02-23T00:37:44.143500+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:35:10.593862+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.593862+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":448,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.135493+0000","last_change":"2024-02-23T00:36:37.126232+0000","last_active":"2024-02-23T00:37:44.135493+0000","last_peered":"2024-02-23T00:37:44.135493+0000","last_clean":"2024-02-23T00:36:33.365949+0000","last_became_active":"2024-02-23T00:36:37.126232+0000","last_became_peered":"2024-02-23T00:36:37.126232+0000","last_unstale":"2024-02-23T00:37:44.135493+0000","last_undegraded":"2024-02-23T00:36:37.112806+0000","last_fullsized":"2024-02-23T00:36:37.112690+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":422,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143765+0000","last_change":"2024-02-23T00:34:22.475022+0000","last_active":"2024-02-23T00:37:44.143765+0000","last_peered":"2024-02-23T00:37:44.143765+0000","last_clean":"2024-02-23T00:37:44.143765+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:37:44.143765+0000","last_undegraded":"2024-02-23T00:37:44.143765+0000","last_fullsized":"2024-02-23T00:37:44.143765+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.474956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.474956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":155,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.146550+0000","last_change":"2024-02-23T00:37:13.405875+0000","last_active":"2024-02-23T00:37:44.146550+0000","last_peered":"2024-02-23T00:37:44.146550+0000","last_clean":"2024-02-23T00:37:44.146550+0000","last_became_active":"2024-02-23T00:34:41.116969+0000","last_became_peered":"2024-02-23T00:34:41.116969+0000","last_unstale":"2024-02-23T00:37:44.146550+0000","last_undegraded":"2024-02-23T00:37:44.146550+0000","last_fullsized":"2024-02-23T00:37:44.146550+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:13.405802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:13.405802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":389,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134556+0000","last_change":"2024-02-23T00:37:21.365047+0000","last_active":"2024-02-23T00:37:44.134556+0000","last_peered":"2024-02-23T00:37:44.134556+0000","last_clean":"2024-02-23T00:37:44.134556+0000","last_became_active":"2024-02-23T00:34:41.109539+0000","last_became_peered":"2024-02-23T00:34:41.109539+0000","last_unstale":"2024-02-23T00:37:44.134556+0000","last_undegraded":"2024-02-23T00:37:44.134556+0000","last_fullsized":"2024-02-23T00:37:44.134556+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_clean_scrub_stamp":"2024-02-23T00:37:21.364934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":47,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.135926+0000","last_change":"2024-02-23T00:36:37.109542+0000","last_active":"2024-02-23T00:37:44.135926+0000","last_peered":"2024-02-23T00:37:44.135926+0000","last_clean":"2024-02-23T00:31:27.149861+0000","last_became_active":"2024-02-23T00:36:37.109542+0000","last_became_peered":"2024-02-23T00:36:37.109542+0000","last_unstale":"2024-02-23T00:37:44.135926+0000","last_undegraded":"2024-02-23T00:37:44.135926+0000","last_fullsized":"2024-02-23T00:36:37.096846+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:42.114807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:35:42.114807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":192,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.133456+0000","last_change":"2024-02-23T00:36:37.114738+0000","last_active":"2024-02-23T00:37:44.133456+0000","last_peered":"2024-02-23T00:37:44.133456+0000","last_clean":"2024-02-23T00:36:33.365051+0000","last_became_active":"2024-02-23T00:36:37.114738+0000","last_became_peered":"2024-02-23T00:36:37.114738+0000","last_unstale":"2024-02-23T00:37:44.133456+0000","last_undegraded":"2024-02-23T00:37:44.133456+0000","last_fullsized":"2024-02-23T00:36:37.094941+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":492,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131793+0000","last_change":"2024-02-23T00:37:08.107220+0000","last_active":"2024-02-23T00:37:44.131793+0000","last_peered":"2024-02-23T00:37:44.131793+0000","last_clean":"2024-02-23T00:37:44.131793+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:37:44.131793+0000","last_undegraded":"2024-02-23T00:37:44.131793+0000","last_fullsized":"2024-02-23T00:37:44.131793+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:37:08.107110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.107110+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":437,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.138516+0000","last_change":"2024-02-23T00:37:28.355537+0000","last_active":"2024-02-23T00:37:44.138516+0000","last_peered":"2024-02-23T00:37:44.138516+0000","last_clean":"2024-02-23T00:37:44.138516+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:37:44.138516+0000","last_undegraded":"2024-02-23T00:37:44.138516+0000","last_fullsized":"2024-02-23T00:37:44.138516+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:28.355449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:28.355449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":434,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.137644+0000","last_change":"2024-02-23T00:36:37.952567+0000","last_active":"2024-02-23T00:37:44.137644+0000","last_peered":"2024-02-23T00:37:44.137644+0000","last_clean":"2024-02-23T00:37:44.137644+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:37:44.137644+0000","last_undegraded":"2024-02-23T00:37:44.137644+0000","last_fullsized":"2024-02-23T00:37:44.137644+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:37.952474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:37.952474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":390,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.134647+0000","last_change":"2024-02-23T00:36:37.116207+0000","last_active":"2024-02-23T00:37:44.134647+0000","last_peered":"2024-02-23T00:37:44.134647+0000","last_clean":"2024-02-23T00:36:33.365474+0000","last_became_active":"2024-02-23T00:36:37.116207+0000","last_became_peered":"2024-02-23T00:36:37.116207+0000","last_unstale":"2024-02-23T00:37:44.134647+0000","last_undegraded":"2024-02-23T00:37:44.134647+0000","last_fullsized":"2024-02-23T00:36:37.098927+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.019530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.019530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":258,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.148677+0000","last_change":"2024-02-23T00:36:37.119099+0000","last_active":"2024-02-23T00:37:44.148677+0000","last_peered":"2024-02-23T00:37:44.148677+0000","last_clean":"2024-02-23T00:36:33.366787+0000","last_became_active":"2024-02-23T00:36:37.119099+0000","last_became_peered":"2024-02-23T00:36:37.119099+0000","last_unstale":"2024-02-23T00:37:44.148677+0000","last_undegraded":"2024-02-23T00:37:44.148677+0000","last_fullsized":"2024-02-23T00:36:37.111585+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.924808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.924808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":457,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.132292+0000","last_change":"2024-02-23T00:36:37.097579+0000","last_active":"2024-02-23T00:37:44.132292+0000","last_peered":"2024-02-23T00:37:44.132292+0000","last_clean":"2024-02-23T00:36:33.191735+0000","last_became_active":"2024-02-23T00:36:37.097579+0000","last_became_peered":"2024-02-23T00:36:37.097579+0000","last_unstale":"2024-02-23T00:37:44.132292+0000","last_undegraded":"2024-02-23T00:37:44.132292+0000","last_fullsized":"2024-02-23T00:36:37.094016+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.588132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.588132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":273,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.135927+0000","last_change":"2024-02-23T00:36:37.119086+0000","last_active":"2024-02-23T00:37:44.135927+0000","last_peered":"2024-02-23T00:37:44.135927+0000","last_clean":"2024-02-23T00:36:33.366171+0000","last_became_active":"2024-02-23T00:36:37.119086+0000","last_became_peered":"2024-02-23T00:36:37.119086+0000","last_unstale":"2024-02-23T00:37:44.135927+0000","last_undegraded":"2024-02-23T00:36:37.101188+0000","last_fullsized":"2024-02-23T00:36:37.101097+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:35:00.621157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.621157+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":428,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136100+0000","last_change":"2024-02-23T00:33:34.199948+0000","last_active":"2024-02-23T00:37:44.136100+0000","last_peered":"2024-02-23T00:37:44.136100+0000","last_clean":"2024-02-23T00:37:44.136100+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:37:44.136100+0000","last_undegraded":"2024-02-23T00:37:44.136100+0000","last_fullsized":"2024-02-23T00:37:44.136100+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.199869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.199869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":440,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.132189+0000","last_change":"2024-02-23T00:34:58.660219+0000","last_active":"2024-02-23T00:37:44.132189+0000","last_peered":"2024-02-23T00:37:44.132189+0000","last_clean":"2024-02-23T00:37:44.132189+0000","last_became_active":"2024-02-23T00:34:41.094428+0000","last_became_peered":"2024-02-23T00:34:41.094428+0000","last_unstale":"2024-02-23T00:37:44.132189+0000","last_undegraded":"2024-02-23T00:37:44.132189+0000","last_fullsized":"2024-02-23T00:37:44.132189+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:58.660111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:58.660111+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":151,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.130648+0000","last_change":"2024-02-23T00:36:41.912734+0000","last_active":"2024-02-23T00:37:44.130648+0000","last_peered":"2024-02-23T00:37:44.130648+0000","last_clean":"2024-02-23T00:37:44.130648+0000","last_became_active":"2024-02-23T00:34:41.105998+0000","last_became_peered":"2024-02-23T00:34:41.105998+0000","last_unstale":"2024-02-23T00:37:44.130648+0000","last_undegraded":"2024-02-23T00:37:44.130648+0000","last_fullsized":"2024-02-23T00:37:44.130648+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.912664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.912664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":696,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134201+0000","last_change":"2024-02-23T00:37:17.499775+0000","last_active":"2024-02-23T00:37:44.134201+0000","last_peered":"2024-02-23T00:37:44.134201+0000","last_clean":"2024-02-23T00:37:44.134201+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:37:44.134201+0000","last_undegraded":"2024-02-23T00:37:44.134201+0000","last_fullsized":"2024-02-23T00:37:44.134201+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:17.499694+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.499694+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":449,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.145011+0000","last_change":"2024-02-23T00:36:37.108691+0000","last_active":"2024-02-23T00:37:44.145011+0000","last_peered":"2024-02-23T00:37:44.145011+0000","last_clean":"2024-02-23T00:36:33.369928+0000","last_became_active":"2024-02-23T00:36:37.108691+0000","last_became_peered":"2024-02-23T00:36:37.108691+0000","last_unstale":"2024-02-23T00:37:44.145011+0000","last_undegraded":"2024-02-23T00:37:44.145011+0000","last_fullsized":"2024-02-23T00:36:37.100335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.196211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.196211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":125,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.135625+0000","last_change":"2024-02-23T00:36:37.123994+0000","last_active":"2024-02-23T00:37:44.135625+0000","last_peered":"2024-02-23T00:37:44.135625+0000","last_clean":"2024-02-23T00:27:29.216066+0000","last_became_active":"2024-02-23T00:36:37.123994+0000","last_became_peered":"2024-02-23T00:36:37.123994+0000","last_unstale":"2024-02-23T00:37:44.135625+0000","last_undegraded":"2024-02-23T00:36:37.103439+0000","last_fullsized":"2024-02-23T00:36:37.103335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:35:01.396982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.396982+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":441,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.147265+0000","last_change":"2024-02-23T00:36:37.120094+0000","last_active":"2024-02-23T00:37:44.147265+0000","last_peered":"2024-02-23T00:37:44.147265+0000","last_clean":"2024-02-23T00:36:33.367908+0000","last_became_active":"2024-02-23T00:36:37.120094+0000","last_became_peered":"2024-02-23T00:36:37.120094+0000","last_unstale":"2024-02-23T00:37:44.147265+0000","last_undegraded":"2024-02-23T00:37:44.147265+0000","last_fullsized":"2024-02-23T00:36:37.107601+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":427,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133491+0000","last_change":"2024-02-23T00:37:26.268389+0000","last_active":"2024-02-23T00:37:44.133491+0000","last_peered":"2024-02-23T00:37:44.133491+0000","last_clean":"2024-02-23T00:37:44.133491+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:37:44.133491+0000","last_undegraded":"2024-02-23T00:37:44.133491+0000","last_fullsized":"2024-02-23T00:37:44.133491+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.268282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":375,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133572+0000","last_change":"2024-02-23T00:35:00.832892+0000","last_active":"2024-02-23T00:37:44.133572+0000","last_peered":"2024-02-23T00:37:44.133572+0000","last_clean":"2024-02-23T00:37:44.133572+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:37:44.133572+0000","last_undegraded":"2024-02-23T00:37:44.133572+0000","last_fullsized":"2024-02-23T00:37:44.133572+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.832818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.832818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":437,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.143876+0000","last_change":"2024-02-23T00:36:37.117616+0000","last_active":"2024-02-23T00:37:44.143876+0000","last_peered":"2024-02-23T00:37:44.143876+0000","last_clean":"2024-02-23T00:36:33.368573+0000","last_became_active":"2024-02-23T00:36:37.117616+0000","last_became_peered":"2024-02-23T00:36:37.117616+0000","last_unstale":"2024-02-23T00:37:44.143876+0000","last_undegraded":"2024-02-23T00:37:44.143876+0000","last_fullsized":"2024-02-23T00:36:37.100549+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":103,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.136145+0000","last_change":"2024-02-23T00:36:37.113785+0000","last_active":"2024-02-23T00:37:44.136145+0000","last_peered":"2024-02-23T00:37:44.136145+0000","last_clean":"2024-02-23T00:27:24.365389+0000","last_became_active":"2024-02-23T00:36:37.113785+0000","last_became_peered":"2024-02-23T00:36:37.113785+0000","last_unstale":"2024-02-23T00:37:44.136145+0000","last_undegraded":"2024-02-23T00:36:37.097517+0000","last_fullsized":"2024-02-23T00:36:37.097457+0000","mapping_epoch":425,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":85,"num_read_kb":71,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":441,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.135125+0000","last_change":"2024-02-23T00:36:37.126102+0000","last_active":"2024-02-23T00:37:44.135125+0000","last_peered":"2024-02-23T00:37:44.135125+0000","last_clean":"2024-02-23T00:36:33.365781+0000","last_became_active":"2024-02-23T00:36:37.126102+0000","last_became_peered":"2024-02-23T00:36:37.126102+0000","last_unstale":"2024-02-23T00:37:44.135125+0000","last_undegraded":"2024-02-23T00:37:44.135125+0000","last_fullsized":"2024-02-23T00:36:37.112617+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.980396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.980396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":359,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.131880+0000","last_change":"2024-02-23T00:36:37.111324+0000","last_active":"2024-02-23T00:37:44.131880+0000","last_peered":"2024-02-23T00:37:44.131880+0000","last_clean":"2024-02-23T00:34:39.448365+0000","last_became_active":"2024-02-23T00:36:37.111324+0000","last_became_peered":"2024-02-23T00:36:37.111324+0000","last_unstale":"2024-02-23T00:37:44.131880+0000","last_undegraded":"2024-02-23T00:37:44.131880+0000","last_fullsized":"2024-02-23T00:36:37.093794+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:31.593354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:31.593354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134514+0000","last_change":"2024-02-23T00:35:02.744199+0000","last_active":"2024-02-23T00:37:44.134514+0000","last_peered":"2024-02-23T00:37:44.134514+0000","last_clean":"2024-02-23T00:37:44.134514+0000","last_became_active":"2024-02-23T00:34:41.118378+0000","last_became_peered":"2024-02-23T00:34:41.118378+0000","last_unstale":"2024-02-23T00:37:44.134514+0000","last_undegraded":"2024-02-23T00:37:44.134514+0000","last_fullsized":"2024-02-23T00:37:44.134514+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.744088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.744088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":384,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133822+0000","last_change":"2024-02-23T00:35:55.223009+0000","last_active":"2024-02-23T00:37:44.133822+0000","last_peered":"2024-02-23T00:37:44.133822+0000","last_clean":"2024-02-23T00:37:44.133822+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:37:44.133822+0000","last_undegraded":"2024-02-23T00:37:44.133822+0000","last_fullsized":"2024-02-23T00:37:44.133822+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.222912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.222912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":445,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.140665+0000","last_change":"2024-02-23T00:36:41.937397+0000","last_active":"2024-02-23T00:37:44.140665+0000","last_peered":"2024-02-23T00:37:44.140665+0000","last_clean":"2024-02-23T00:37:44.140665+0000","last_became_active":"2024-02-23T00:35:27.093541+0000","last_became_peered":"2024-02-23T00:35:27.093541+0000","last_unstale":"2024-02-23T00:37:44.140665+0000","last_undegraded":"2024-02-23T00:37:44.140665+0000","last_fullsized":"2024-02-23T00:37:44.140665+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.937344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.937344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":476,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.141062+0000","last_change":"2024-02-23T00:36:32.150367+0000","last_active":"2024-02-23T00:37:44.141062+0000","last_peered":"2024-02-23T00:37:44.141062+0000","last_clean":"2024-02-23T00:37:44.141062+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:37:44.141062+0000","last_undegraded":"2024-02-23T00:37:44.141062+0000","last_fullsized":"2024-02-23T00:37:44.141062+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'14","last_scrub_stamp":"2024-02-23T00:36:32.150250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.150250+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":444,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.135305+0000","last_change":"2024-02-23T00:36:37.106377+0000","last_active":"2024-02-23T00:37:44.135305+0000","last_peered":"2024-02-23T00:37:44.135305+0000","last_clean":"2024-02-23T00:36:33.365867+0000","last_became_active":"2024-02-23T00:36:37.106377+0000","last_became_peered":"2024-02-23T00:36:37.106377+0000","last_unstale":"2024-02-23T00:37:44.135305+0000","last_undegraded":"2024-02-23T00:37:44.135305+0000","last_fullsized":"2024-02-23T00:36:37.100069+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.135820+0000","last_change":"2024-02-23T00:36:33.160039+0000","last_active":"2024-02-23T00:37:44.135820+0000","last_peered":"2024-02-23T00:37:44.135820+0000","last_clean":"2024-02-23T00:37:44.135820+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:37:44.135820+0000","last_undegraded":"2024-02-23T00:37:44.135820+0000","last_fullsized":"2024-02-23T00:37:44.135820+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.159941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.159941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":248,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.148246+0000","last_change":"2024-02-23T00:37:26.933362+0000","last_active":"2024-02-23T00:37:44.148246+0000","last_peered":"2024-02-23T00:37:44.148246+0000","last_clean":"2024-02-23T00:37:44.148246+0000","last_became_active":"2024-02-23T00:34:41.099598+0000","last_became_peered":"2024-02-23T00:34:41.099598+0000","last_unstale":"2024-02-23T00:37:44.148246+0000","last_undegraded":"2024-02-23T00:37:44.148246+0000","last_fullsized":"2024-02-23T00:37:44.148246+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.933288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.933288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":434,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.144029+0000","last_change":"2024-02-23T00:37:20.008803+0000","last_active":"2024-02-23T00:37:44.144029+0000","last_peered":"2024-02-23T00:37:44.144029+0000","last_clean":"2024-02-23T00:37:44.144029+0000","last_became_active":"2024-02-23T00:34:41.114270+0000","last_became_peered":"2024-02-23T00:34:41.114270+0000","last_unstale":"2024-02-23T00:37:44.144029+0000","last_undegraded":"2024-02-23T00:37:44.144029+0000","last_fullsized":"2024-02-23T00:37:44.144029+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.008726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":386,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143149+0000","last_change":"2024-02-23T00:35:52.175358+0000","last_active":"2024-02-23T00:37:44.143149+0000","last_peered":"2024-02-23T00:37:44.143149+0000","last_clean":"2024-02-23T00:37:44.143149+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:37:44.143149+0000","last_undegraded":"2024-02-23T00:37:44.143149+0000","last_fullsized":"2024-02-23T00:37:44.143149+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.175246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.175246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":448,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131515+0000","last_change":"2024-02-23T00:36:33.004360+0000","last_active":"2024-02-23T00:37:44.131515+0000","last_peered":"2024-02-23T00:37:44.131515+0000","last_clean":"2024-02-23T00:37:44.131515+0000","last_became_active":"2024-02-23T00:35:27.097186+0000","last_became_peered":"2024-02-23T00:35:27.097186+0000","last_unstale":"2024-02-23T00:37:44.131515+0000","last_undegraded":"2024-02-23T00:37:44.131515+0000","last_fullsized":"2024-02-23T00:37:44.131515+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.004264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.004264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":512,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.144888+0000","last_change":"2024-02-23T00:36:37.108623+0000","last_active":"2024-02-23T00:37:44.144888+0000","last_peered":"2024-02-23T00:37:44.144888+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:36:37.108623+0000","last_became_peered":"2024-02-23T00:36:37.108623+0000","last_unstale":"2024-02-23T00:37:44.144888+0000","last_undegraded":"2024-02-23T00:36:37.097062+0000","last_fullsized":"2024-02-23T00:36:37.096949+0000","mapping_epoch":425,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":458,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.130704+0000","last_change":"2024-02-23T00:36:37.119168+0000","last_active":"2024-02-23T00:37:44.130704+0000","last_peered":"2024-02-23T00:37:44.130704+0000","last_clean":"2024-02-23T00:36:33.193740+0000","last_became_active":"2024-02-23T00:36:37.119168+0000","last_became_peered":"2024-02-23T00:36:37.119168+0000","last_unstale":"2024-02-23T00:37:44.130704+0000","last_undegraded":"2024-02-23T00:37:44.130704+0000","last_fullsized":"2024-02-23T00:36:37.101741+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.184937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.184937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":689,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133616+0000","last_change":"2024-02-23T00:37:13.467508+0000","last_active":"2024-02-23T00:37:44.133616+0000","last_peered":"2024-02-23T00:37:44.133616+0000","last_clean":"2024-02-23T00:37:44.133616+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:37:44.133616+0000","last_undegraded":"2024-02-23T00:37:44.133616+0000","last_fullsized":"2024-02-23T00:37:44.133616+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:13.467428+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:13.467428+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":443,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.147149+0000","last_change":"2024-02-23T00:37:12.023243+0000","last_active":"2024-02-23T00:37:44.147149+0000","last_peered":"2024-02-23T00:37:44.147149+0000","last_clean":"2024-02-23T00:37:44.147149+0000","last_became_active":"2024-02-23T00:34:41.105566+0000","last_became_peered":"2024-02-23T00:34:41.105566+0000","last_unstale":"2024-02-23T00:37:44.147149+0000","last_undegraded":"2024-02-23T00:37:44.147149+0000","last_fullsized":"2024-02-23T00:37:44.147149+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:12.023125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:12.023125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":390,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131089+0000","last_change":"2024-02-23T00:37:09.743047+0000","last_active":"2024-02-23T00:37:44.131089+0000","last_peered":"2024-02-23T00:37:44.131089+0000","last_clean":"2024-02-23T00:37:44.131089+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:37:44.131089+0000","last_undegraded":"2024-02-23T00:37:44.131089+0000","last_fullsized":"2024-02-23T00:37:44.131089+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.742938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.742938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":452,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.131822+0000","last_change":"2024-02-23T00:36:37.101373+0000","last_active":"2024-02-23T00:37:44.131822+0000","last_peered":"2024-02-23T00:37:44.131822+0000","last_clean":"2024-02-23T00:36:33.192308+0000","last_became_active":"2024-02-23T00:36:37.101373+0000","last_became_peered":"2024-02-23T00:36:37.101373+0000","last_unstale":"2024-02-23T00:37:44.131822+0000","last_undegraded":"2024-02-23T00:37:44.131822+0000","last_fullsized":"2024-02-23T00:36:37.094754+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.595447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.595447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":392,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.136235+0000","last_change":"2024-02-23T00:36:37.109549+0000","last_active":"2024-02-23T00:37:44.136235+0000","last_peered":"2024-02-23T00:37:44.136235+0000","last_clean":"2024-02-23T00:35:25.318841+0000","last_became_active":"2024-02-23T00:36:37.109549+0000","last_became_peered":"2024-02-23T00:36:37.109549+0000","last_unstale":"2024-02-23T00:37:44.136235+0000","last_undegraded":"2024-02-23T00:37:44.136235+0000","last_fullsized":"2024-02-23T00:36:37.096473+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"453'47","reported_seq":671,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.137144+0000","last_change":"2024-02-23T00:37:15.422544+0000","last_active":"2024-02-23T00:37:45.137144+0000","last_peered":"2024-02-23T00:37:45.137144+0000","last_clean":"2024-02-23T00:37:45.137144+0000","last_became_active":"2024-02-23T00:34:41.099241+0000","last_became_peered":"2024-02-23T00:34:41.099241+0000","last_unstale":"2024-02-23T00:37:45.137144+0000","last_undegraded":"2024-02-23T00:37:45.137144+0000","last_fullsized":"2024-02-23T00:37:45.137144+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'35","last_scrub_stamp":"2024-02-23T00:37:15.422435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:15.422435+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":71,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133451+0000","last_change":"2024-02-23T00:37:23.360353+0000","last_active":"2024-02-23T00:37:44.133451+0000","last_peered":"2024-02-23T00:37:44.133451+0000","last_clean":"2024-02-23T00:37:44.133451+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:37:44.133451+0000","last_undegraded":"2024-02-23T00:37:44.133451+0000","last_fullsized":"2024-02-23T00:37:44.133451+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:37:23.360230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:23.360230+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":150,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.144084+0000","last_change":"2024-02-23T00:36:32.098355+0000","last_active":"2024-02-23T00:37:44.144084+0000","last_peered":"2024-02-23T00:37:44.144084+0000","last_clean":"2024-02-23T00:37:44.144084+0000","last_became_active":"2024-02-23T00:34:41.109972+0000","last_became_peered":"2024-02-23T00:34:41.109972+0000","last_unstale":"2024-02-23T00:37:44.144084+0000","last_undegraded":"2024-02-23T00:37:44.144084+0000","last_fullsized":"2024-02-23T00:37:44.144084+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.098280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.098280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":390,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131711+0000","last_change":"2024-02-23T00:37:11.766364+0000","last_active":"2024-02-23T00:37:44.131711+0000","last_peered":"2024-02-23T00:37:44.131711+0000","last_clean":"2024-02-23T00:37:44.131711+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:37:44.131711+0000","last_undegraded":"2024-02-23T00:37:44.131711+0000","last_fullsized":"2024-02-23T00:37:44.131711+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:11.766284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:11.766284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":350,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136391+0000","last_change":"2024-02-23T00:36:23.210776+0000","last_active":"2024-02-23T00:37:44.136391+0000","last_peered":"2024-02-23T00:37:44.136391+0000","last_clean":"2024-02-23T00:37:44.136391+0000","last_became_active":"2024-02-23T00:34:41.109182+0000","last_became_peered":"2024-02-23T00:34:41.109182+0000","last_unstale":"2024-02-23T00:37:44.136391+0000","last_undegraded":"2024-02-23T00:37:44.136391+0000","last_fullsized":"2024-02-23T00:37:44.136391+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:23.210706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.210706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":67,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.137945+0000","last_change":"2024-02-23T00:36:37.104760+0000","last_active":"2024-02-23T00:37:44.137945+0000","last_peered":"2024-02-23T00:37:44.137945+0000","last_clean":"2024-02-23T00:27:24.172824+0000","last_became_active":"2024-02-23T00:36:37.104760+0000","last_became_peered":"2024-02-23T00:36:37.104760+0000","last_unstale":"2024-02-23T00:37:44.137945+0000","last_undegraded":"2024-02-23T00:37:44.137945+0000","last_fullsized":"2024-02-23T00:36:37.098375+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":104,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.148456+0000","last_change":"2024-02-23T00:36:37.110364+0000","last_active":"2024-02-23T00:37:44.148456+0000","last_peered":"2024-02-23T00:37:44.148456+0000","last_clean":"2024-02-23T00:27:29.173100+0000","last_became_active":"2024-02-23T00:36:37.110364+0000","last_became_peered":"2024-02-23T00:36:37.110364+0000","last_unstale":"2024-02-23T00:37:44.148456+0000","last_undegraded":"2024-02-23T00:36:37.096575+0000","last_fullsized":"2024-02-23T00:36:37.096459+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:35:44.214727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:44.214727+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":437,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.134294+0000","last_change":"2024-02-23T00:36:37.123256+0000","last_active":"2024-02-23T00:37:44.134294+0000","last_peered":"2024-02-23T00:37:44.134294+0000","last_clean":"2024-02-23T00:36:33.366566+0000","last_became_active":"2024-02-23T00:36:37.123256+0000","last_became_peered":"2024-02-23T00:36:37.123256+0000","last_unstale":"2024-02-23T00:37:44.134294+0000","last_undegraded":"2024-02-23T00:37:44.134294+0000","last_fullsized":"2024-02-23T00:36:37.096440+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":345,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.130604+0000","last_change":"2024-02-23T00:36:37.113860+0000","last_active":"2024-02-23T00:37:44.130604+0000","last_peered":"2024-02-23T00:37:44.130604+0000","last_clean":"2024-02-23T00:34:39.448366+0000","last_became_active":"2024-02-23T00:36:37.113860+0000","last_became_peered":"2024-02-23T00:36:37.113860+0000","last_unstale":"2024-02-23T00:37:44.130604+0000","last_undegraded":"2024-02-23T00:37:44.130604+0000","last_fullsized":"2024-02-23T00:36:37.101768+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.488959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.488959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":384,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131593+0000","last_change":"2024-02-23T00:36:42.093272+0000","last_active":"2024-02-23T00:37:44.131593+0000","last_peered":"2024-02-23T00:37:44.131593+0000","last_clean":"2024-02-23T00:37:44.131593+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:37:44.131593+0000","last_undegraded":"2024-02-23T00:37:44.131593+0000","last_fullsized":"2024-02-23T00:37:44.131593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:36:42.093181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":449,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.130212+0000","last_change":"2024-02-23T00:36:37.124382+0000","last_active":"2024-02-23T00:37:44.130212+0000","last_peered":"2024-02-23T00:37:44.130212+0000","last_clean":"2024-02-23T00:36:33.193431+0000","last_became_active":"2024-02-23T00:36:37.124382+0000","last_became_peered":"2024-02-23T00:36:37.124382+0000","last_unstale":"2024-02-23T00:37:44.130212+0000","last_undegraded":"2024-02-23T00:37:44.130212+0000","last_fullsized":"2024-02-23T00:36:37.101474+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:24.016837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:24.016837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":604,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131920+0000","last_change":"2024-02-23T00:35:09.098161+0000","last_active":"2024-02-23T00:37:44.131920+0000","last_peered":"2024-02-23T00:37:44.131920+0000","last_clean":"2024-02-23T00:37:44.131920+0000","last_became_active":"2024-02-23T00:35:09.097721+0000","last_became_peered":"2024-02-23T00:35:09.097721+0000","last_unstale":"2024-02-23T00:37:44.131920+0000","last_undegraded":"2024-02-23T00:37:44.131920+0000","last_fullsized":"2024-02-23T00:37:44.131920+0000","mapping_epoch":349,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:34:54.691537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.691537+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":150,"num_read_kb":126,"num_write":84,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":456,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.143319+0000","last_change":"2024-02-23T00:36:37.118733+0000","last_active":"2024-02-23T00:37:44.143319+0000","last_peered":"2024-02-23T00:37:44.143319+0000","last_clean":"2024-02-23T00:36:36.130983+0000","last_became_active":"2024-02-23T00:36:37.118733+0000","last_became_peered":"2024-02-23T00:36:37.118733+0000","last_unstale":"2024-02-23T00:37:44.143319+0000","last_undegraded":"2024-02-23T00:36:37.095672+0000","last_fullsized":"2024-02-23T00:36:37.095564+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:36:36.130868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.130868+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":150,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.137212+0000","last_change":"2024-02-23T00:35:48.866316+0000","last_active":"2024-02-23T00:37:44.137212+0000","last_peered":"2024-02-23T00:37:44.137212+0000","last_clean":"2024-02-23T00:37:44.137212+0000","last_became_active":"2024-02-23T00:34:41.110345+0000","last_became_peered":"2024-02-23T00:34:41.110345+0000","last_unstale":"2024-02-23T00:37:44.137212+0000","last_undegraded":"2024-02-23T00:37:44.137212+0000","last_fullsized":"2024-02-23T00:37:44.137212+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:48.866230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:48.866230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":380,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.135162+0000","last_change":"2024-02-23T00:36:28.184266+0000","last_active":"2024-02-23T00:37:44.135162+0000","last_peered":"2024-02-23T00:37:44.135162+0000","last_clean":"2024-02-23T00:37:44.135162+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:37:44.135162+0000","last_undegraded":"2024-02-23T00:37:44.135162+0000","last_fullsized":"2024-02-23T00:37:44.135162+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:28.184197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:28.184197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":444,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.130062+0000","last_change":"2024-02-23T00:36:37.111836+0000","last_active":"2024-02-23T00:37:44.130062+0000","last_peered":"2024-02-23T00:37:44.130062+0000","last_clean":"2024-02-23T00:36:33.193391+0000","last_became_active":"2024-02-23T00:36:37.111836+0000","last_became_peered":"2024-02-23T00:36:37.111836+0000","last_unstale":"2024-02-23T00:37:44.130062+0000","last_undegraded":"2024-02-23T00:37:44.130062+0000","last_fullsized":"2024-02-23T00:36:37.101173+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.667517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.667517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":150,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.144918+0000","last_change":"2024-02-23T00:36:37.118505+0000","last_active":"2024-02-23T00:37:44.144918+0000","last_peered":"2024-02-23T00:37:44.144918+0000","last_clean":"2024-02-23T00:36:33.367279+0000","last_became_active":"2024-02-23T00:36:37.118505+0000","last_became_peered":"2024-02-23T00:36:37.118505+0000","last_unstale":"2024-02-23T00:37:44.144918+0000","last_undegraded":"2024-02-23T00:37:44.144918+0000","last_fullsized":"2024-02-23T00:36:37.097249+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.614960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.614960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":442,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131203+0000","last_change":"2024-02-23T00:37:22.918309+0000","last_active":"2024-02-23T00:37:44.131203+0000","last_peered":"2024-02-23T00:37:44.131203+0000","last_clean":"2024-02-23T00:37:44.131203+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:37:44.131203+0000","last_undegraded":"2024-02-23T00:37:44.131203+0000","last_fullsized":"2024-02-23T00:37:44.131203+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.918248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":148,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.139287+0000","last_change":"2024-02-23T00:34:56.402946+0000","last_active":"2024-02-23T00:37:44.139287+0000","last_peered":"2024-02-23T00:37:44.139287+0000","last_clean":"2024-02-23T00:37:44.139287+0000","last_became_active":"2024-02-23T00:34:41.114943+0000","last_became_peered":"2024-02-23T00:34:41.114943+0000","last_unstale":"2024-02-23T00:37:44.139287+0000","last_undegraded":"2024-02-23T00:37:44.139287+0000","last_fullsized":"2024-02-23T00:37:44.139287+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.402851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.402851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":438,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.141292+0000","last_change":"2024-02-23T00:37:08.426793+0000","last_active":"2024-02-23T00:37:44.141292+0000","last_peered":"2024-02-23T00:37:44.141292+0000","last_clean":"2024-02-23T00:37:44.141292+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:37:44.141292+0000","last_undegraded":"2024-02-23T00:37:44.141292+0000","last_fullsized":"2024-02-23T00:37:44.141292+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:08.426716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.426716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.140332+0000","last_change":"2024-02-23T00:35:52.291042+0000","last_active":"2024-02-23T00:37:44.140332+0000","last_peered":"2024-02-23T00:37:44.140332+0000","last_clean":"2024-02-23T00:37:44.140332+0000","last_became_active":"2024-02-23T00:34:41.113722+0000","last_became_peered":"2024-02-23T00:34:41.113722+0000","last_unstale":"2024-02-23T00:37:44.140332+0000","last_undegraded":"2024-02-23T00:37:44.140332+0000","last_fullsized":"2024-02-23T00:37:44.140332+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.290946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.290946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":434,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.146714+0000","last_change":"2024-02-23T00:37:16.011330+0000","last_active":"2024-02-23T00:37:44.146714+0000","last_peered":"2024-02-23T00:37:44.146714+0000","last_clean":"2024-02-23T00:37:44.146714+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:37:44.146714+0000","last_undegraded":"2024-02-23T00:37:44.146714+0000","last_fullsized":"2024-02-23T00:37:44.146714+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:16.011248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:37:16.011248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":394,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.141465+0000","last_change":"2024-02-23T00:37:20.316511+0000","last_active":"2024-02-23T00:37:44.141465+0000","last_peered":"2024-02-23T00:37:44.141465+0000","last_clean":"2024-02-23T00:37:44.141465+0000","last_became_active":"2024-02-23T00:34:41.101132+0000","last_became_peered":"2024-02-23T00:34:41.101132+0000","last_unstale":"2024-02-23T00:37:44.141465+0000","last_undegraded":"2024-02-23T00:37:44.141465+0000","last_fullsized":"2024-02-23T00:37:44.141465+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.316429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.316429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"453'32","reported_seq":233,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134428+0000","last_change":"2024-02-23T00:34:42.571126+0000","last_active":"2024-02-23T00:37:44.134428+0000","last_peered":"2024-02-23T00:37:44.134428+0000","last_clean":"2024-02-23T00:37:44.134428+0000","last_became_active":"2024-02-23T00:34:41.106795+0000","last_became_peered":"2024-02-23T00:34:41.106795+0000","last_unstale":"2024-02-23T00:37:44.134428+0000","last_undegraded":"2024-02-23T00:37:44.134428+0000","last_fullsized":"2024-02-23T00:37:44.134428+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:34:15.571539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.571539+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":446,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.146039+0000","last_change":"2024-02-23T00:36:37.117696+0000","last_active":"2024-02-23T00:37:44.146039+0000","last_peered":"2024-02-23T00:37:44.146039+0000","last_clean":"2024-02-23T00:36:33.366237+0000","last_became_active":"2024-02-23T00:36:37.117696+0000","last_became_peered":"2024-02-23T00:36:37.117696+0000","last_unstale":"2024-02-23T00:37:44.146039+0000","last_undegraded":"2024-02-23T00:37:44.146039+0000","last_fullsized":"2024-02-23T00:36:37.103226+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:46.269798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.269798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"426'10","reported_seq":564,"reported_epoch":464,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:45.148787+0000","last_change":"2024-02-23T00:36:37.113534+0000","last_active":"2024-02-23T00:37:45.148787+0000","last_peered":"2024-02-23T00:37:45.148787+0000","last_clean":"2024-02-23T00:36:35.444798+0000","last_became_active":"2024-02-23T00:36:37.113534+0000","last_became_peered":"2024-02-23T00:36:37.113534+0000","last_unstale":"2024-02-23T00:37:45.148787+0000","last_undegraded":"2024-02-23T00:36:37.102235+0000","last_fullsized":"2024-02-23T00:36:37.102132+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":344,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.140355+0000","last_change":"2024-02-23T00:36:37.120693+0000","last_active":"2024-02-23T00:37:44.140355+0000","last_peered":"2024-02-23T00:37:44.140355+0000","last_clean":"2024-02-23T00:34:39.449328+0000","last_became_active":"2024-02-23T00:36:37.120693+0000","last_became_peered":"2024-02-23T00:36:37.120693+0000","last_unstale":"2024-02-23T00:37:44.140355+0000","last_undegraded":"2024-02-23T00:37:44.140355+0000","last_fullsized":"2024-02-23T00:36:37.107400+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.521760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":380,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.141419+0000","last_change":"2024-02-23T00:37:08.058170+0000","last_active":"2024-02-23T00:37:44.141419+0000","last_peered":"2024-02-23T00:37:44.141419+0000","last_clean":"2024-02-23T00:37:44.141419+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:37:44.141419+0000","last_undegraded":"2024-02-23T00:37:44.141419+0000","last_fullsized":"2024-02-23T00:37:44.141419+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:08.058075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.058075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":490,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136960+0000","last_change":"2024-02-23T00:35:01.515684+0000","last_active":"2024-02-23T00:37:44.136960+0000","last_peered":"2024-02-23T00:37:44.136960+0000","last_clean":"2024-02-23T00:37:44.136960+0000","last_became_active":"2024-02-23T00:34:41.110140+0000","last_became_peered":"2024-02-23T00:34:41.110140+0000","last_unstale":"2024-02-23T00:37:44.136960+0000","last_undegraded":"2024-02-23T00:37:44.136960+0000","last_fullsized":"2024-02-23T00:37:44.136960+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:35:01.515550+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.515550+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":350,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.132177+0000","last_change":"2024-02-23T00:36:37.118542+0000","last_active":"2024-02-23T00:37:44.132177+0000","last_peered":"2024-02-23T00:37:44.132177+0000","last_clean":"2024-02-23T00:34:39.449034+0000","last_became_active":"2024-02-23T00:36:37.118542+0000","last_became_peered":"2024-02-23T00:36:37.118542+0000","last_unstale":"2024-02-23T00:37:44.132177+0000","last_undegraded":"2024-02-23T00:37:44.132177+0000","last_fullsized":"2024-02-23T00:36:37.095362+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.391997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.391997+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"330'10","reported_seq":271,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.132162+0000","last_change":"2024-02-23T00:37:27.870730+0000","last_active":"2024-02-23T00:37:45.132162+0000","last_peered":"2024-02-23T00:37:45.132162+0000","last_clean":"2024-02-23T00:37:45.132162+0000","last_became_active":"2024-02-23T00:34:41.109074+0000","last_became_peered":"2024-02-23T00:34:41.109074+0000","last_unstale":"2024-02-23T00:37:45.132162+0000","last_undegraded":"2024-02-23T00:37:45.132162+0000","last_fullsized":"2024-02-23T00:37:45.132162+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'10","last_scrub_stamp":"2024-02-23T00:37:27.870653+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:37:27.870653+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143584+0000","last_change":"2024-02-23T00:35:07.627722+0000","last_active":"2024-02-23T00:37:44.143584+0000","last_peered":"2024-02-23T00:37:44.143584+0000","last_clean":"2024-02-23T00:37:44.143584+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:37:44.143584+0000","last_undegraded":"2024-02-23T00:37:44.143584+0000","last_fullsized":"2024-02-23T00:37:44.143584+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:07.627620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:07.627620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":148,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.144088+0000","last_change":"2024-02-23T00:36:35.160313+0000","last_active":"2024-02-23T00:37:44.144088+0000","last_peered":"2024-02-23T00:37:44.144088+0000","last_clean":"2024-02-23T00:37:44.144088+0000","last_became_active":"2024-02-23T00:34:41.091189+0000","last_became_peered":"2024-02-23T00:34:41.091189+0000","last_unstale":"2024-02-23T00:37:44.144088+0000","last_undegraded":"2024-02-23T00:37:44.144088+0000","last_fullsized":"2024-02-23T00:37:44.144088+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:35.160239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:35.160239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":456,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133616+0000","last_change":"2024-02-23T00:37:20.382193+0000","last_active":"2024-02-23T00:37:44.133616+0000","last_peered":"2024-02-23T00:37:44.133616+0000","last_clean":"2024-02-23T00:37:44.133616+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:37:44.133616+0000","last_undegraded":"2024-02-23T00:37:44.133616+0000","last_fullsized":"2024-02-23T00:37:44.133616+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:37:20.382108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.382108+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":447,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.146426+0000","last_change":"2024-02-23T00:36:36.180921+0000","last_active":"2024-02-23T00:37:44.146426+0000","last_peered":"2024-02-23T00:37:44.146426+0000","last_clean":"2024-02-23T00:37:44.146426+0000","last_became_active":"2024-02-23T00:34:41.116840+0000","last_became_peered":"2024-02-23T00:34:41.116840+0000","last_unstale":"2024-02-23T00:37:44.146426+0000","last_undegraded":"2024-02-23T00:37:44.146426+0000","last_fullsized":"2024-02-23T00:37:44.146426+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:36.180849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.180849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":49,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.134501+0000","last_change":"2024-02-23T00:36:37.108983+0000","last_active":"2024-02-23T00:37:44.134501+0000","last_peered":"2024-02-23T00:37:44.134501+0000","last_clean":"2024-02-23T00:27:20.002687+0000","last_became_active":"2024-02-23T00:36:37.108983+0000","last_became_peered":"2024-02-23T00:36:37.108983+0000","last_unstale":"2024-02-23T00:37:44.134501+0000","last_undegraded":"2024-02-23T00:37:44.134501+0000","last_fullsized":"2024-02-23T00:36:37.099047+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.416493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.416493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":46,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.136678+0000","last_change":"2024-02-23T00:36:37.112999+0000","last_active":"2024-02-23T00:37:44.136678+0000","last_peered":"2024-02-23T00:37:44.136678+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.112999+0000","last_became_peered":"2024-02-23T00:36:37.112999+0000","last_unstale":"2024-02-23T00:37:44.136678+0000","last_undegraded":"2024-02-23T00:37:44.136678+0000","last_fullsized":"2024-02-23T00:36:37.102073+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.413991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.413991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":390,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.131549+0000","last_change":"2024-02-23T00:36:37.097581+0000","last_active":"2024-02-23T00:37:44.131549+0000","last_peered":"2024-02-23T00:37:44.131549+0000","last_clean":"2024-02-23T00:36:33.191411+0000","last_became_active":"2024-02-23T00:36:37.097581+0000","last_became_peered":"2024-02-23T00:36:37.097581+0000","last_unstale":"2024-02-23T00:37:44.131549+0000","last_undegraded":"2024-02-23T00:37:44.131549+0000","last_fullsized":"2024-02-23T00:36:37.093707+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.101501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.101501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":487,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.132071+0000","last_change":"2024-02-23T00:35:45.171309+0000","last_active":"2024-02-23T00:37:44.132071+0000","last_peered":"2024-02-23T00:37:44.132071+0000","last_clean":"2024-02-23T00:37:44.132071+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:37:44.132071+0000","last_undegraded":"2024-02-23T00:37:44.132071+0000","last_fullsized":"2024-02-23T00:37:44.132071+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:35:45.171195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:45.171195+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":434,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.136310+0000","last_change":"2024-02-23T00:36:37.108194+0000","last_active":"2024-02-23T00:37:44.136310+0000","last_peered":"2024-02-23T00:37:44.136310+0000","last_clean":"2024-02-23T00:36:33.367482+0000","last_became_active":"2024-02-23T00:36:37.108194+0000","last_became_peered":"2024-02-23T00:36:37.108194+0000","last_unstale":"2024-02-23T00:37:44.136310+0000","last_undegraded":"2024-02-23T00:37:44.136310+0000","last_fullsized":"2024-02-23T00:36:37.096730+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.783089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.783089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":433,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133720+0000","last_change":"2024-02-23T00:34:41.116355+0000","last_active":"2024-02-23T00:37:44.133720+0000","last_peered":"2024-02-23T00:37:44.133720+0000","last_clean":"2024-02-23T00:37:44.133720+0000","last_became_active":"2024-02-23T00:34:41.115862+0000","last_became_peered":"2024-02-23T00:34:41.115862+0000","last_unstale":"2024-02-23T00:37:44.133720+0000","last_undegraded":"2024-02-23T00:37:44.133720+0000","last_fullsized":"2024-02-23T00:37:44.133720+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.151986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":49,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.143976+0000","last_change":"2024-02-23T00:36:37.116289+0000","last_active":"2024-02-23T00:37:44.143976+0000","last_peered":"2024-02-23T00:37:44.143976+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.116289+0000","last_became_peered":"2024-02-23T00:36:37.116289+0000","last_unstale":"2024-02-23T00:37:44.143976+0000","last_undegraded":"2024-02-23T00:37:44.143976+0000","last_fullsized":"2024-02-23T00:36:37.106216+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.620107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":412,"reported_epoch":463,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:44.144408+0000","last_change":"2024-02-23T00:36:37.104333+0000","last_active":"2024-02-23T00:37:44.144408+0000","last_peered":"2024-02-23T00:37:44.144408+0000","last_clean":"2024-02-23T00:36:33.365274+0000","last_became_active":"2024-02-23T00:36:37.104333+0000","last_became_peered":"2024-02-23T00:36:37.104333+0000","last_unstale":"2024-02-23T00:37:44.144408+0000","last_undegraded":"2024-02-23T00:36:37.099294+0000","last_fullsized":"2024-02-23T00:36:37.099175+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:35:47.130749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.130749+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":194,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.133655+0000","last_change":"2024-02-23T00:36:36.531491+0000","last_active":"2024-02-23T00:37:44.133655+0000","last_peered":"2024-02-23T00:37:44.133655+0000","last_clean":"2024-02-23T00:37:44.133655+0000","last_became_active":"2024-02-23T00:34:41.101219+0000","last_became_peered":"2024-02-23T00:34:41.101219+0000","last_unstale":"2024-02-23T00:37:44.133655+0000","last_undegraded":"2024-02-23T00:37:44.133655+0000","last_fullsized":"2024-02-23T00:37:44.133655+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:36:36.531424+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.531424+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.141007+0000","last_change":"2024-02-23T00:36:32.199878+0000","last_active":"2024-02-23T00:37:44.141007+0000","last_peered":"2024-02-23T00:37:44.141007+0000","last_clean":"2024-02-23T00:37:44.141007+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:37:44.141007+0000","last_undegraded":"2024-02-23T00:37:44.141007+0000","last_fullsized":"2024-02-23T00:37:44.141007+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.199765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.199765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":429,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.141182+0000","last_change":"2024-02-23T00:35:39.035845+0000","last_active":"2024-02-23T00:37:44.141182+0000","last_peered":"2024-02-23T00:37:44.141182+0000","last_clean":"2024-02-23T00:37:44.141182+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:37:44.141182+0000","last_undegraded":"2024-02-23T00:37:44.141182+0000","last_fullsized":"2024-02-23T00:37:44.141182+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.035767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.035767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":434,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.150409+0000","last_change":"2024-02-23T00:37:14.009706+0000","last_active":"2024-02-23T00:37:44.150409+0000","last_peered":"2024-02-23T00:37:44.150409+0000","last_clean":"2024-02-23T00:37:44.150409+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:37:44.150409+0000","last_undegraded":"2024-02-23T00:37:44.150409+0000","last_fullsized":"2024-02-23T00:37:44.150409+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.009614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.009614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":375,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134065+0000","last_change":"2024-02-23T00:33:44.152089+0000","last_active":"2024-02-23T00:37:44.134065+0000","last_peered":"2024-02-23T00:37:44.134065+0000","last_clean":"2024-02-23T00:37:44.134065+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:37:44.134065+0000","last_undegraded":"2024-02-23T00:37:44.134065+0000","last_fullsized":"2024-02-23T00:37:44.134065+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.152014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.152014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":248,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.149719+0000","last_change":"2024-02-23T00:37:26.914783+0000","last_active":"2024-02-23T00:37:44.149719+0000","last_peered":"2024-02-23T00:37:44.149719+0000","last_clean":"2024-02-23T00:37:44.149719+0000","last_became_active":"2024-02-23T00:34:41.114516+0000","last_became_peered":"2024-02-23T00:34:41.114516+0000","last_unstale":"2024-02-23T00:37:44.149719+0000","last_undegraded":"2024-02-23T00:37:44.149719+0000","last_fullsized":"2024-02-23T00:37:44.149719+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:37:26.914698+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.914698+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":437,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.138375+0000","last_change":"2024-02-23T00:37:22.340222+0000","last_active":"2024-02-23T00:37:44.138375+0000","last_peered":"2024-02-23T00:37:44.138375+0000","last_clean":"2024-02-23T00:37:44.138375+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:37:44.138375+0000","last_undegraded":"2024-02-23T00:37:44.138375+0000","last_fullsized":"2024-02-23T00:37:44.138375+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:22.340128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.340128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.135775+0000","last_change":"2024-02-23T00:35:47.857609+0000","last_active":"2024-02-23T00:37:44.135775+0000","last_peered":"2024-02-23T00:37:44.135775+0000","last_clean":"2024-02-23T00:37:44.135775+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:37:44.135775+0000","last_undegraded":"2024-02-23T00:37:44.135775+0000","last_fullsized":"2024-02-23T00:37:44.135775+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.857533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.857533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":48,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.138162+0000","last_change":"2024-02-23T00:36:37.114573+0000","last_active":"2024-02-23T00:37:44.138162+0000","last_peered":"2024-02-23T00:37:44.138162+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.114573+0000","last_became_peered":"2024-02-23T00:36:37.114573+0000","last_unstale":"2024-02-23T00:37:44.138162+0000","last_undegraded":"2024-02-23T00:37:44.138162+0000","last_fullsized":"2024-02-23T00:36:37.098725+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.175171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.175171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":311,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.137771+0000","last_change":"2024-02-23T00:36:04.097119+0000","last_active":"2024-02-23T00:37:44.137771+0000","last_peered":"2024-02-23T00:37:44.137771+0000","last_clean":"2024-02-23T00:37:44.137771+0000","last_became_active":"2024-02-23T00:36:04.096804+0000","last_became_peered":"2024-02-23T00:36:04.096804+0000","last_unstale":"2024-02-23T00:37:44.137771+0000","last_undegraded":"2024-02-23T00:37:44.137771+0000","last_fullsized":"2024-02-23T00:37:44.137771+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:53.153821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:53.153821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":435,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136186+0000","last_change":"2024-02-23T00:34:42.064110+0000","last_active":"2024-02-23T00:37:44.136186+0000","last_peered":"2024-02-23T00:37:44.136186+0000","last_clean":"2024-02-23T00:37:44.136186+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:37:44.136186+0000","last_undegraded":"2024-02-23T00:37:44.136186+0000","last_fullsized":"2024-02-23T00:37:44.136186+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:32.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:32.520027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":571,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-23T00:37:45.150747+0000","last_change":"2024-02-23T00:36:39.246854+0000","last_active":"2024-02-23T00:37:45.150747+0000","last_peered":"2024-02-23T00:37:45.150747+0000","last_clean":"2024-02-23T00:37:45.150747+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:37:45.150747+0000","last_undegraded":"2024-02-23T00:37:45.150747+0000","last_fullsized":"2024-02-23T00:37:45.150747+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:36:39.246273+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.246273+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.140926+0000","last_change":"2024-02-23T00:35:09.656923+0000","last_active":"2024-02-23T00:37:44.140926+0000","last_peered":"2024-02-23T00:37:44.140926+0000","last_clean":"2024-02-23T00:37:44.140926+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:37:44.140926+0000","last_undegraded":"2024-02-23T00:37:44.140926+0000","last_fullsized":"2024-02-23T00:37:44.140926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:35:09.656838+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:35:09.656838+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":398,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.141047+0000","last_change":"2024-02-23T00:36:37.122220+0000","last_active":"2024-02-23T00:37:44.141047+0000","last_peered":"2024-02-23T00:37:44.141047+0000","last_clean":"2024-02-23T00:36:33.368644+0000","last_became_active":"2024-02-23T00:36:37.122220+0000","last_became_peered":"2024-02-23T00:36:37.122220+0000","last_unstale":"2024-02-23T00:37:44.141047+0000","last_undegraded":"2024-02-23T00:37:44.141047+0000","last_fullsized":"2024-02-23T00:36:37.109811+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.028166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.028166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":154,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.141028+0000","last_change":"2024-02-23T00:36:37.114328+0000","last_active":"2024-02-23T00:37:44.141028+0000","last_peered":"2024-02-23T00:37:44.141028+0000","last_clean":"2024-02-23T00:36:33.365066+0000","last_became_active":"2024-02-23T00:36:37.114328+0000","last_became_peered":"2024-02-23T00:36:37.114328+0000","last_unstale":"2024-02-23T00:37:44.141028+0000","last_undegraded":"2024-02-23T00:37:44.141028+0000","last_fullsized":"2024-02-23T00:36:37.098495+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.151731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":46,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.140769+0000","last_change":"2024-02-23T00:36:37.122400+0000","last_active":"2024-02-23T00:37:44.140769+0000","last_peered":"2024-02-23T00:37:44.140769+0000","last_clean":"2024-02-23T00:27:20.002687+0000","last_became_active":"2024-02-23T00:36:37.122400+0000","last_became_peered":"2024-02-23T00:36:37.122400+0000","last_unstale":"2024-02-23T00:37:44.140769+0000","last_undegraded":"2024-02-23T00:37:44.140769+0000","last_fullsized":"2024-02-23T00:36:37.111280+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":427,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136101+0000","last_change":"2024-02-23T00:35:57.658486+0000","last_active":"2024-02-23T00:37:44.136101+0000","last_peered":"2024-02-23T00:37:44.136101+0000","last_clean":"2024-02-23T00:37:44.136101+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:37:44.136101+0000","last_undegraded":"2024-02-23T00:37:44.136101+0000","last_fullsized":"2024-02-23T00:37:44.136101+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.658406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.658406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":383,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.147606+0000","last_change":"2024-02-23T00:35:39.871154+0000","last_active":"2024-02-23T00:37:44.147606+0000","last_peered":"2024-02-23T00:37:44.147606+0000","last_clean":"2024-02-23T00:37:44.147606+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:37:44.147606+0000","last_undegraded":"2024-02-23T00:37:44.147606+0000","last_fullsized":"2024-02-23T00:37:44.147606+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.871058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.871058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":441,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.145167+0000","last_change":"2024-02-23T00:36:25.100710+0000","last_active":"2024-02-23T00:37:44.145167+0000","last_peered":"2024-02-23T00:37:44.145167+0000","last_clean":"2024-02-23T00:37:44.145167+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:37:44.145167+0000","last_undegraded":"2024-02-23T00:37:44.145167+0000","last_fullsized":"2024-02-23T00:37:44.145167+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:25.100627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.100627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":432,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.136483+0000","last_change":"2024-02-23T00:36:37.113234+0000","last_active":"2024-02-23T00:37:44.136483+0000","last_peered":"2024-02-23T00:37:44.136483+0000","last_clean":"2024-02-23T00:36:33.367528+0000","last_became_active":"2024-02-23T00:36:37.113234+0000","last_became_peered":"2024-02-23T00:36:37.113234+0000","last_unstale":"2024-02-23T00:37:44.136483+0000","last_undegraded":"2024-02-23T00:37:44.136483+0000","last_fullsized":"2024-02-23T00:36:37.097125+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.791748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.791748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":426,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143599+0000","last_change":"2024-02-23T00:34:57.661029+0000","last_active":"2024-02-23T00:37:44.143599+0000","last_peered":"2024-02-23T00:37:44.143599+0000","last_clean":"2024-02-23T00:37:44.143599+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:37:44.143599+0000","last_undegraded":"2024-02-23T00:37:44.143599+0000","last_fullsized":"2024-02-23T00:37:44.143599+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:57.660954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.660954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":307,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.140929+0000","last_change":"2024-02-23T00:36:27.951790+0000","last_active":"2024-02-23T00:37:44.140929+0000","last_peered":"2024-02-23T00:37:44.140929+0000","last_clean":"2024-02-23T00:37:44.140929+0000","last_became_active":"2024-02-23T00:36:17.099716+0000","last_became_peered":"2024-02-23T00:36:17.099716+0000","last_unstale":"2024-02-23T00:37:44.140929+0000","last_undegraded":"2024-02-23T00:37:44.140929+0000","last_fullsized":"2024-02-23T00:37:44.140929+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:27.951690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.951690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":150,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.137424+0000","last_change":"2024-02-23T00:36:35.929592+0000","last_active":"2024-02-23T00:37:44.137424+0000","last_peered":"2024-02-23T00:37:44.137424+0000","last_clean":"2024-02-23T00:37:44.137424+0000","last_became_active":"2024-02-23T00:34:41.103114+0000","last_became_peered":"2024-02-23T00:34:41.103114+0000","last_unstale":"2024-02-23T00:37:44.137424+0000","last_undegraded":"2024-02-23T00:37:44.137424+0000","last_fullsized":"2024-02-23T00:37:44.137424+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:35.929507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:35.929507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":437,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.132451+0000","last_change":"2024-02-23T00:34:56.679204+0000","last_active":"2024-02-23T00:37:44.132451+0000","last_peered":"2024-02-23T00:37:44.132451+0000","last_clean":"2024-02-23T00:37:44.132451+0000","last_became_active":"2024-02-23T00:34:41.106439+0000","last_became_peered":"2024-02-23T00:34:41.106439+0000","last_unstale":"2024-02-23T00:37:44.132451+0000","last_undegraded":"2024-02-23T00:37:44.132451+0000","last_fullsized":"2024-02-23T00:37:44.132451+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.679136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.679136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":153,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143974+0000","last_change":"2024-02-23T00:37:19.329028+0000","last_active":"2024-02-23T00:37:44.143974+0000","last_peered":"2024-02-23T00:37:44.143974+0000","last_clean":"2024-02-23T00:37:44.143974+0000","last_became_active":"2024-02-23T00:34:41.113447+0000","last_became_peered":"2024-02-23T00:34:41.113447+0000","last_unstale":"2024-02-23T00:37:44.143974+0000","last_undegraded":"2024-02-23T00:37:44.143974+0000","last_fullsized":"2024-02-23T00:37:44.143974+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:19.328950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:19.328950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":375,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143364+0000","last_change":"2024-02-23T00:33:33.382591+0000","last_active":"2024-02-23T00:37:44.143364+0000","last_peered":"2024-02-23T00:37:44.143364+0000","last_clean":"2024-02-23T00:37:44.143364+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:37:44.143364+0000","last_undegraded":"2024-02-23T00:37:44.143364+0000","last_fullsized":"2024-02-23T00:37:44.143364+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.382510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.382510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":142,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131768+0000","last_change":"2024-02-23T00:34:42.058800+0000","last_active":"2024-02-23T00:37:44.131768+0000","last_peered":"2024-02-23T00:37:44.131768+0000","last_clean":"2024-02-23T00:37:44.131768+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:37:44.131768+0000","last_undegraded":"2024-02-23T00:37:44.131768+0000","last_fullsized":"2024-02-23T00:37:44.131768+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:32.720048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:32.720048+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":428,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.135948+0000","last_change":"2024-02-23T00:36:37.124006+0000","last_active":"2024-02-23T00:37:44.135948+0000","last_peered":"2024-02-23T00:37:44.135948+0000","last_clean":"2024-02-23T00:36:33.366419+0000","last_became_active":"2024-02-23T00:36:37.124006+0000","last_became_peered":"2024-02-23T00:36:37.124006+0000","last_unstale":"2024-02-23T00:37:44.135948+0000","last_undegraded":"2024-02-23T00:37:44.135948+0000","last_fullsized":"2024-02-23T00:36:37.103625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":46,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.133228+0000","last_change":"2024-02-23T00:36:37.119810+0000","last_active":"2024-02-23T00:37:44.133228+0000","last_peered":"2024-02-23T00:37:44.133228+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.119810+0000","last_became_peered":"2024-02-23T00:36:37.119810+0000","last_unstale":"2024-02-23T00:37:44.133228+0000","last_undegraded":"2024-02-23T00:37:44.133228+0000","last_fullsized":"2024-02-23T00:36:37.098897+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:19.572808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.572808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":388,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134789+0000","last_change":"2024-02-23T00:37:18.044814+0000","last_active":"2024-02-23T00:37:44.134789+0000","last_peered":"2024-02-23T00:37:44.134789+0000","last_clean":"2024-02-23T00:37:44.134789+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:37:44.134789+0000","last_undegraded":"2024-02-23T00:37:44.134789+0000","last_fullsized":"2024-02-23T00:37:44.134789+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:18.044737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:18.044737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":441,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.146837+0000","last_change":"2024-02-23T00:36:37.114846+0000","last_active":"2024-02-23T00:37:44.146837+0000","last_peered":"2024-02-23T00:37:44.146837+0000","last_clean":"2024-02-23T00:36:33.367758+0000","last_became_active":"2024-02-23T00:36:37.114846+0000","last_became_peered":"2024-02-23T00:36:37.114846+0000","last_unstale":"2024-02-23T00:37:44.146837+0000","last_undegraded":"2024-02-23T00:37:44.146837+0000","last_fullsized":"2024-02-23T00:36:37.107709+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":436,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.150249+0000","last_change":"2024-02-23T00:37:19.034254+0000","last_active":"2024-02-23T00:37:44.150249+0000","last_peered":"2024-02-23T00:37:44.150249+0000","last_clean":"2024-02-23T00:37:44.150249+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:37:44.150249+0000","last_undegraded":"2024-02-23T00:37:44.150249+0000","last_fullsized":"2024-02-23T00:37:44.150249+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:19.034136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:37:19.034136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":431,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.143852+0000","last_change":"2024-02-23T00:35:41.260304+0000","last_active":"2024-02-23T00:37:44.143852+0000","last_peered":"2024-02-23T00:37:44.143852+0000","last_clean":"2024-02-23T00:37:44.143852+0000","last_became_active":"2024-02-23T00:34:41.112980+0000","last_became_peered":"2024-02-23T00:34:41.112980+0000","last_unstale":"2024-02-23T00:37:44.143852+0000","last_undegraded":"2024-02-23T00:37:44.143852+0000","last_fullsized":"2024-02-23T00:37:44.143852+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:41.260231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.260231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":390,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.133665+0000","last_change":"2024-02-23T00:36:37.115401+0000","last_active":"2024-02-23T00:37:44.133665+0000","last_peered":"2024-02-23T00:37:44.133665+0000","last_clean":"2024-02-23T00:36:33.365083+0000","last_became_active":"2024-02-23T00:36:37.115401+0000","last_became_peered":"2024-02-23T00:36:37.115401+0000","last_unstale":"2024-02-23T00:37:44.133665+0000","last_undegraded":"2024-02-23T00:37:44.133665+0000","last_fullsized":"2024-02-23T00:36:37.099146+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.192483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.192483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":49,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.140773+0000","last_change":"2024-02-23T00:36:37.123921+0000","last_active":"2024-02-23T00:37:44.140773+0000","last_peered":"2024-02-23T00:37:44.140773+0000","last_clean":"2024-02-23T00:27:15.989319+0000","last_became_active":"2024-02-23T00:36:37.123921+0000","last_became_peered":"2024-02-23T00:36:37.123921+0000","last_unstale":"2024-02-23T00:37:44.140773+0000","last_undegraded":"2024-02-23T00:37:44.140773+0000","last_fullsized":"2024-02-23T00:36:37.106358+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.388834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":84,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.138139+0000","last_change":"2024-02-23T00:35:42.473209+0000","last_active":"2024-02-23T00:37:44.138139+0000","last_peered":"2024-02-23T00:37:44.138139+0000","last_clean":"2024-02-23T00:37:44.138139+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:37:44.138139+0000","last_undegraded":"2024-02-23T00:37:44.138139+0000","last_fullsized":"2024-02-23T00:37:44.138139+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:38.617181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:38.617181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":428,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131744+0000","last_change":"2024-02-23T00:34:23.597939+0000","last_active":"2024-02-23T00:37:44.131744+0000","last_peered":"2024-02-23T00:37:44.131744+0000","last_clean":"2024-02-23T00:37:44.131744+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:37:44.131744+0000","last_undegraded":"2024-02-23T00:37:44.131744+0000","last_fullsized":"2024-02-23T00:37:44.131744+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:23.597868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.597868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":48,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.130387+0000","last_change":"2024-02-23T00:36:37.119114+0000","last_active":"2024-02-23T00:37:44.130387+0000","last_peered":"2024-02-23T00:37:44.130387+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119114+0000","last_became_peered":"2024-02-23T00:36:37.119114+0000","last_unstale":"2024-02-23T00:37:44.130387+0000","last_undegraded":"2024-02-23T00:37:44.130387+0000","last_fullsized":"2024-02-23T00:36:37.106746+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.176441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.176441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":46,"reported_epoch":463,"state":"active+undersized","last_fresh":"2024-02-23T00:37:44.148947+0000","last_change":"2024-02-23T00:36:37.120482+0000","last_active":"2024-02-23T00:37:44.148947+0000","last_peered":"2024-02-23T00:37:44.148947+0000","last_clean":"2024-02-23T00:27:15.989319+0000","last_became_active":"2024-02-23T00:36:37.120482+0000","last_became_peered":"2024-02-23T00:36:37.120482+0000","last_unstale":"2024-02-23T00:37:44.148947+0000","last_undegraded":"2024-02-23T00:37:44.148947+0000","last_fullsized":"2024-02-23T00:36:37.111071+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":433,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.131492+0000","last_change":"2024-02-23T00:37:14.200716+0000","last_active":"2024-02-23T00:37:44.131492+0000","last_peered":"2024-02-23T00:37:44.131492+0000","last_clean":"2024-02-23T00:37:44.131492+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:37:44.131492+0000","last_undegraded":"2024-02-23T00:37:44.131492+0000","last_fullsized":"2024-02-23T00:37:44.131492+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.200606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.200606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":155,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.136949+0000","last_change":"2024-02-23T00:37:24.990742+0000","last_active":"2024-02-23T00:37:44.136949+0000","last_peered":"2024-02-23T00:37:44.136949+0000","last_clean":"2024-02-23T00:37:44.136949+0000","last_became_active":"2024-02-23T00:34:41.109129+0000","last_became_peered":"2024-02-23T00:34:41.109129+0000","last_unstale":"2024-02-23T00:37:44.136949+0000","last_undegraded":"2024-02-23T00:37:44.136949+0000","last_fullsized":"2024-02-23T00:37:44.136949+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:24.990671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:24.990671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":389,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-23T00:37:44.134738+0000","last_change":"2024-02-23T00:36:40.485366+0000","last_active":"2024-02-23T00:37:44.134738+0000","last_peered":"2024-02-23T00:37:44.134738+0000","last_clean":"2024-02-23T00:37:44.134738+0000","last_became_active":"2024-02-23T00:34:41.099195+0000","last_became_peered":"2024-02-23T00:34:41.099195+0000","last_unstale":"2024-02-23T00:37:44.134738+0000","last_undegraded":"2024-02-23T00:37:44.134738+0000","last_fullsized":"2024-02-23T00:37:44.134738+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:40.485283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:40.485283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":218,"num_pg":32,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":79,"acting":79,"num_store_stats":6},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":4001578,"num_objects":169,"num_object_clones":7,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":59,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":109,"num_read_kb":10,"num_write":618,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":4000297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":12004717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1929,"ondisk_log_size":1929,"up":84,"acting":84,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":105,"acting":105,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":55,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1495,"num_read_kb":1220,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":2778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":518,"ondisk_log_size":518,"up":60,"acting":60,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":307,"num_read_kb":307,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":34,"acting":34,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":81,"acting":81,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332225,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310552,"kb_used_data":9232,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93434600,"statfs":{"total":95995035648,"available":95677030400,"internally_reserved":0,"allocated":9453568,"data_stored":6936394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.02,"15min":1.024},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.79800000000000004,"5min":57.401000000000003,"15min":61.048999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.1719999999999999,"15min":1.2450000000000001},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":65.558999999999997,"15min":73.870999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.53400000000000003,"15min":0.69599999999999995},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":0.68100000000000005,"5min":4.6840000000000002,"15min":31.175000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.51300000000000001,"15min":0.68700000000000006},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.59699999999999998,"5min":3.8719999999999999,"15min":29.423999999999999},"last":0.57499999999999996}]},{"osd":2,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.50600000000000001,"15min":0.66200000000000003},"min":{"1min":0.30299999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64300000000000002,"5min":3.7240000000000002,"15min":16.603999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56699999999999995,"15min":0.61499999999999999},"min":{"1min":0.36199999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65400000000000003,"5min":4.1369999999999996,"15min":7.5339999999999998},"last":0.66000000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.54000000000000004,"15min":0.97799999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.59099999999999997,"5min":3.7679999999999998,"15min":57.729999999999997},"last":0.496},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.57699999999999996,"15min":0.90800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.312},"max":{"1min":0.69699999999999995,"5min":3.8420000000000001,"15min":27.573},"last":0.69099999999999995}]},{"osd":4,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42899999999999999,"15min":0.41899999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":0.58499999999999996,"5min":1.706,"15min":1.706},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.45600000000000002,"15min":0.436},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54200000000000004,"5min":3.294,"15min":3.294},"last":0.46000000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.434,"15min":0.43099999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.61399999999999999,"5min":0.85899999999999999,"15min":1.0209999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.48699999999999999,"15min":0.45600000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76400000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.432}]},{"osd":6,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.51200000000000001,"15min":0.48699999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.28000000000000003},"max":{"1min":0.627,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.48899999999999999,"15min":0.47299999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":0.57299999999999995,"5min":0.83299999999999996,"15min":1.002},"last":0.71799999999999997}]}]},{"osd":6,"up_from":44,"seq":188978561158,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310788,"kb_used_data":5372,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93434364,"statfs":{"total":95995035648,"available":95676788736,"internally_reserved":0,"allocated":5500928,"data_stored":2944959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.6499999999999999,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.78800000000000003,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64800000000000002,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":9.468,"15min":9.468},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.5,"15min":1.014},"min":{"1min":0.35699999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.67500000000000004,"5min":0.79100000000000004,"15min":59.347999999999999},"last":0.439},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.497,"15min":1.2929999999999999},"min":{"1min":0.34300000000000003,"5min":0.28299999999999997,"15min":0.23100000000000001},"max":{"1min":0.59899999999999998,"5min":0.875,"15min":87.221000000000004},"last":0.57799999999999996}]},{"osd":2,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.81299999999999994},"min":{"1min":0.372,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74299999999999999,"5min":1.022,"15min":25.550999999999998},"last":0.443},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.88},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.70899999999999996,"5min":4.3570000000000002,"15min":40.914000000000001},"last":0.53100000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.54600000000000004,"15min":0.80800000000000005},"min":{"1min":0.36299999999999999,"5min":0.30199999999999999,"15min":0.23400000000000001},"max":{"1min":0.75900000000000001,"5min":3.2799999999999998,"15min":31.309999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54400000000000004,"15min":1.329},"min":{"1min":0.42599999999999999,"5min":0.35799999999999998,"15min":0.29899999999999999},"max":{"1min":0.69499999999999995,"5min":1.145,"15min":86.311999999999998},"last":0.47599999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.26100000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53100000000000003,"5min":0.68500000000000005,"15min":3.7839999999999998},"last":0.378},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.45200000000000001,"15min":0.504},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":17.718},"last":0.32800000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.45300000000000001,"15min":0.48099999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.47399999999999998,"5min":0.73799999999999999,"15min":8.0990000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.67900000000000005,"5min":0.75700000000000001,"15min":8.1370000000000005},"last":0.38800000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:37:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.48799999999999999,"15min":0.59499999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.50900000000000001,"5min":0.75,"15min":17.881},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.5,"15min":0.58999999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.55400000000000005,"5min":0.80800000000000005,"15min":17.396999999999998},"last":0.35199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318972,"kb_used_data":9460,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426180,"statfs":{"total":95995035648,"available":95668408320,"internally_reserved":0,"allocated":9687040,"data_stored":6946927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.29,"15min":0.66300000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.51700000000000002,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.89600000000000002,"15min":0.55200000000000005},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60499999999999998,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:37:32 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.73999999999999999,"15min":0.67000000000000004},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.45300000000000001,"5min":35.313000000000002,"15min":35.313000000000002},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.68500000000000005,"15min":0.57499999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.20100000000000001},"max":{"1min":0.47699999999999998,"5min":27.908999999999999,"15min":27.908999999999999},"last":0.317}]},{"osd":3,"last update":"Fri Feb 23 00:36:59 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.497,"15min":0.64800000000000002},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.69299999999999995,"5min":10.294,"15min":21.013999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.49299999999999999,"15min":0.64900000000000002},"min":{"1min":0.255,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.72099999999999997,"5min":10.382999999999999,"15min":29.234999999999999},"last":0.29999999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.77200000000000002,"15min":0.69699999999999995},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.79400000000000004,"5min":16.231000000000002,"15min":16.231000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.50900000000000001,"15min":0.72299999999999998},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":0.70999999999999996,"5min":1.135,"15min":29.212},"last":0.496}]},{"osd":5,"last update":"Fri Feb 23 00:37:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.78700000000000003,"15min":0.71999999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.75800000000000001,"5min":16.294,"15min":16.294},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.61599999999999999,"15min":0.67900000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80300000000000005,"5min":8.4030000000000005,"15min":11.105},"last":0.51000000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.627,"15min":0.76400000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.91100000000000003,"5min":10.416,"15min":29.187000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.80200000000000005,"15min":0.752},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.92900000000000005,"5min":16.369,"15min":16.369},"last":0.59699999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:37:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.92200000000000004,"15min":0.76700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73899999999999999,"5min":16.395,"15min":16.395},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.54900000000000004,"15min":0.66200000000000003},"min":{"1min":0.36599999999999999,"5min":0.36299999999999999,"15min":0.35999999999999999},"max":{"1min":0.82399999999999995,"5min":0.89500000000000002,"15min":11.135999999999999},"last":0.56200000000000006}]}]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318432,"kb_used_data":4824,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426720,"statfs":{"total":95995035648,"available":95668961280,"internally_reserved":0,"allocated":4939776,"data_stored":2599387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.80600000000000005,"15min":0.495},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47799999999999998,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.246},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.165,"15min":0.61899999999999999},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.997,"15min":0.58899999999999997},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.64400000000000002,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.748,"15min":0.48999999999999999},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.22700000000000001},"max":{"1min":0.51100000000000001,"5min":15.965,"15min":15.965},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":1.0820000000000001,"15min":0.65000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.0369999999999999,"15min":0.63700000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.58799999999999997,"5min":16.006,"15min":16.006},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.73399999999999999,"15min":0.63800000000000001},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84699999999999998,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.38300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83699999999999997,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.071,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86799999999999999,"5min":13.311,"15min":13.311},"last":0.499},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.113,"15min":0.74399999999999999},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":23.744,"15min":23.744},"last":0.42299999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.0740000000000001,"15min":0.80600000000000005},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78500000000000003,"5min":13.334,"15min":33.082999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.93400000000000005,"15min":0.70999999999999996},"min":{"1min":0.41199999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81699999999999995,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.49199999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.84099999999999997,"15min":0.78500000000000003},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.871,"5min":15.943,"15min":31.251999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.091,"15min":0.76100000000000001},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.80000000000000004,"5min":23.785,"15min":23.785},"last":0.47599999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411481,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310532,"kb_used_data":9212,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93434620,"statfs":{"total":95995035648,"available":95677050880,"internally_reserved":0,"allocated":9433088,"data_stored":6934910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":1.0600000000000001,"15min":0.754},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.65500000000000003,"5min":56.509,"15min":56.509},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.59799999999999998,"15min":0.71199999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56799999999999995,"5min":19.571999999999999,"15min":31.498999999999999},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:37:35 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.46999999999999997,"15min":0.59499999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":7.7110000000000003,"5min":7.7110000000000003,"15min":19.324999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.70999999999999996},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.0249999999999999,"5min":13.209,"15min":35.052},"last":0.44900000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.442,"15min":0.70499999999999996},"min":{"1min":0.23999999999999999,"5min":0.20999999999999999,"15min":0.20899999999999999},"max":{"1min":6.6559999999999997,"5min":6.6559999999999997,"15min":43.5},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.54700000000000004,"15min":0.65400000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.7409999999999997,"5min":10.843,"15min":27.414000000000001},"last":0.28199999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":0.72099999999999997,"15min":0.67700000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":10.106,"5min":10.106,"15min":14.614000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.999,"5min":0.67500000000000004,"15min":0.628},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":7.766,"5min":7.766,"15min":7.766},"last":0.30199999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:36:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56399999999999995,"15min":0.66600000000000004},"min":{"1min":0.40999999999999998,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":0.79400000000000004,"5min":3.2879999999999998,"15min":17.765000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.63900000000000001},"min":{"1min":0.35499999999999998,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":0.79100000000000004,"5min":2.1259999999999999,"15min":14.589},"last":0.55600000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.66000000000000003,"15min":0.627},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.7869999999999999,"5min":7.7869999999999999,"15min":7.7869999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60399999999999998,"15min":0.751},"min":{"1min":0.379,"5min":0.36699999999999999,"15min":0.34899999999999998},"max":{"1min":1.8460000000000001,"5min":3.3410000000000002,"15min":20.574999999999999},"last":0.377}]},{"osd":7,"last update":"Fri Feb 23 00:37:28 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.63,"15min":1.036},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":1.9590000000000001,"5min":3.3780000000000001,"15min":66.632000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.349,"5min":0.748,"15min":0.85199999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":10.163,"5min":10.163,"15min":19.603999999999999},"last":0.443}]}]},{"osd":3,"up_from":0,"seq":111669149845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986320,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314948,"kb_used_data":5436,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430204,"statfs":{"total":95995035648,"available":95672528896,"internally_reserved":0,"allocated":5566464,"data_stored":2944119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1739999999999999,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.68799999999999994,"5min":21.419,"15min":21.419},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":2.1200000000000001,"15min":1.2589999999999999},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":0.83899999999999997,"15min":1.048},"min":{"1min":0.35399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.696000000000002,"5min":16.696000000000002,"15min":41.137999999999998},"last":0.315},{"interface":"front","average":{"1min":1.28,"5min":0.78300000000000003,"15min":0.88700000000000001},"min":{"1min":0.35199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.455,"5min":13.455,"15min":26.210000000000001},"last":0.34599999999999997}]},{"osd":2,"last update":"Fri Feb 23 00:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":0.76200000000000001,"15min":1.1299999999999999},"min":{"1min":0.376,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":13.478,"5min":13.478,"15min":67.253},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.72099999999999997,"15min":1.0209999999999999},"min":{"1min":0.38,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":4.2590000000000003,"5min":8.0199999999999996,"15min":41.927999999999997},"last":0.38600000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.69999999999999996,"15min":1.069},"min":{"1min":0.39000000000000001,"5min":0.35699999999999998,"15min":0.34000000000000002},"max":{"1min":7.774,"5min":7.774,"15min":36.301000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.4419999999999999,"5min":0.75700000000000001,"15min":1.167},"min":{"1min":0.34000000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":16.657,"5min":16.657,"15min":42.064},"last":0.36599999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:37:40 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.219,"5min":0.219,"15min":0.217},"max":{"1min":0.65100000000000002,"5min":8.5299999999999994,"15min":14.242000000000001},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.69399999999999995,"15min":0.59599999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.57799999999999996,"5min":8.3010000000000002,"15min":14.124000000000001},"last":0.40500000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.69399999999999995,"15min":0.56999999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.254},"max":{"1min":7.1189999999999998,"5min":8.0730000000000004,"15min":8.0730000000000004},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55300000000000005,"15min":0.63600000000000001},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.23100000000000001},"max":{"1min":2.2450000000000001,"5min":4.71,"15min":45.814999999999998},"last":0.253}]},{"osd":7,"last update":"Fri Feb 23 00:37:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.29999999999999999},"max":{"1min":5.1280000000000001,"5min":6.165,"15min":17.178999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.288,"5min":0.81399999999999995,"15min":0.69799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28299999999999997},"max":{"1min":15.85,"5min":15.85,"15min":15.85},"last":0.33300000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757387,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306776,"kb_used_data":5456,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438376,"statfs":{"total":95995035648,"available":95680897024,"internally_reserved":0,"allocated":5586944,"data_stored":2936029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.1200000000000001,"15min":0.70699999999999996},"min":{"1min":0.36699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.115,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":1.028},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.498,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48999999999999999,"15min":0.753},"min":{"1min":0.318,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67700000000000005,"5min":1.546,"15min":38.838000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.497,"5min":0.48999999999999999,"15min":1.1719999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.251},"max":{"1min":0.69199999999999995,"5min":1.4059999999999999,"15min":127.794},"last":0.39300000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.504,"15min":0.88700000000000001},"min":{"1min":0.41499999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.68400000000000005,"5min":1.248,"15min":37.442999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52800000000000002,"15min":0.70199999999999996},"min":{"1min":0.35499999999999998,"5min":0.30499999999999999,"15min":0.28499999999999998},"max":{"1min":0.745,"5min":1.609,"15min":23.297999999999998},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.505,"15min":0.73699999999999999},"min":{"1min":0.35899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68000000000000005,"5min":0.76900000000000002,"15min":24.414999999999999},"last":0.435},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53700000000000003,"15min":0.89500000000000002},"min":{"1min":0.38,"5min":0.33300000000000002,"15min":0.31},"max":{"1min":0.76300000000000001,"5min":1.524,"15min":40.865000000000002},"last":0.57899999999999996}]},{"osd":4,"last update":"Fri Feb 23 00:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41899999999999998,"15min":0.77600000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":0.57299999999999995,"5min":1.4790000000000001,"15min":103.13},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.442,"15min":0.45700000000000002},"min":{"1min":0.27200000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.623,"5min":1.21,"15min":12.795999999999999},"last":0.50600000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:37:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51500000000000001,"15min":0.498},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":2.9199999999999999,"5min":2.9199999999999999,"15min":7.1440000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47999999999999998,"15min":0.81999999999999995},"min":{"1min":0.28599999999999998,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.71399999999999997,"5min":1.4490000000000001,"15min":103.61499999999999},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:37:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.9039999999999999,"5min":3.9039999999999999,"15min":7.1820000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.58199999999999996,"15min":0.53200000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.4050000000000002,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.68200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4390912,"data_stored":4000917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4235264,"data_stored":4000574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":218,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":218,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":218,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":218,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":218,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":218,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:37:48.356 INFO:tasks.thrashosds.thrasher:Setting 3.9 to [6, 2, 0, 7, 5, 3] 2024-02-23T00:37:48.356 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.9', '6', '2', '0', '7', '5', '3'] 2024-02-23T00:37:48.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pg-upmap-items 3.9 6 2 0 7 5 3 2024-02-23T00:37:48.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:47.286603+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:48.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:47.291256+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:37:48.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:47.297891+0000 mon.b (mon.1) 382 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-73"}]: dispatch 2024-02-23T00:37:48.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:47.301792+0000 mon.a (mon.0) 2714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-73"}]: dispatch 2024-02-23T00:37:48.786 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:48.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:48.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:48.890 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:48.918 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:47.467823+0000 mgr.y (mgr.24437) 439 : audit [DBG] from='client.37796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:37:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:47.684435+0000 mgr.y (mgr.24437) 440 : cluster [DBG] pgmap v707: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 124/1197 objects degraded (10.359%) 2024-02-23T00:37:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:48.177031+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e467: 8 total, 7 up, 7 in 2024-02-23T00:37:49.567 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:49.569 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:47.467823+0000 mgr.y (mgr.24437) 439 : audit [DBG] from='client.37796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:37:49.569 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:47.684435+0000 mgr.y (mgr.24437) 440 : cluster [DBG] pgmap v707: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 124/1197 objects degraded (10.359%) 2024-02-23T00:37:49.569 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:48.177031+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e467: 8 total, 7 up, 7 in 2024-02-23T00:37:49.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:47.467823+0000 mgr.y (mgr.24437) 439 : audit [DBG] from='client.37796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:37:49.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:47.684435+0000 mgr.y (mgr.24437) 440 : cluster [DBG] pgmap v707: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 124/1197 objects degraded (10.359%) 2024-02-23T00:37:49.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:48.177031+0000 mon.a (mon.0) 2715 : cluster [DBG] osdmap e467: 8 total, 7 up, 7 in 2024-02-23T00:37:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:49.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:49.671 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:49.698 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:49.717 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:49.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:49.821 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:49.847 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:49.866 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:49.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:49.969 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:49.999 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 -- 172.21.15.87:0/658983691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fb01044a0 msgr2=0x7f8fb0106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 --2- 172.21.15.87:0/658983691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fb01044a0 0x7f8fb0106890 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f8fa8009be0 tx=0x7f8fa8004d10).stop 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 -- 172.21.15.87:0/658983691 shutdown_connections 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 --2- 172.21.15.87:0/658983691 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fb01044a0 0x7f8fb0106890 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 --2- 172.21.15.87:0/658983691 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fb006b110 0x7f8fb006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 --2- 172.21.15.87:0/658983691 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8fb006b9e0 0x7f8fb0103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 -- 172.21.15.87:0/658983691 >> 172.21.15.87:0/658983691 conn(0x7f8fb00f3800 msgr2=0x7f8fb00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 -- 172.21.15.87:0/658983691 shutdown_connections 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 -- 172.21.15.87:0/658983691 wait complete. 2024-02-23T00:37:50.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 librados: starting msgr at 2024-02-23T00:37:50.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.092+0000 7f8fb746e700 1 librados: starting objecter 2024-02-23T00:37:50.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 Processor -- start 2024-02-23T00:37:50.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 -- start start 2024-02-23T00:37:50.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 librados: setting wanted keys 2024-02-23T00:37:50.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 librados: calling monclient init 2024-02-23T00:37:50.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fb006b110 0x7f8fb0110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8fb006b9e0 0x7f8fb0110660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fb01044a0 0x7f8fb010a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb010a800 con 0x7f8fb01044a0 2024-02-23T00:37:50.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb010a970 con 0x7f8fb006b110 2024-02-23T00:37:50.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb746e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb010ab30 con 0x7f8fb006b9e0 2024-02-23T00:37:50.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb4a09700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8fb006b9e0 0x7f8fb0110660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb5a0b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fb01044a0 0x7f8fb010a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb4a09700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8fb006b9e0 0x7f8fb0110660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35268/0 (socket says 172.21.15.87:35268) 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb4a09700 1 -- 172.21.15.87:0/1295521052 learned_addr learned my addr 172.21.15.87:0/1295521052 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb520a700 1 --2- 172.21.15.87:0/1295521052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fb006b110 0x7f8fb0110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb4a09700 1 -- 172.21.15.87:0/1295521052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fb006b110 msgr2=0x7f8fb0110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb4a09700 1 --2- 172.21.15.87:0/1295521052 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8fb006b110 0x7f8fb0110170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb4a09700 1 -- 172.21.15.87:0/1295521052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fb01044a0 msgr2=0x7f8fb010a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb4a09700 1 --2- 172.21.15.87:0/1295521052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fb01044a0 0x7f8fb010a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb4a09700 1 -- 172.21.15.87:0/1295521052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac005040 con 0x7f8fb006b9e0 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb5a0b700 1 --2- 172.21.15.87:0/1295521052 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8fb01044a0 0x7f8fb010a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fb4a09700 1 --2- 172.21.15.87:0/1295521052 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8fb006b9e0 0x7f8fb0110660 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 rx=0x7f8fac0048f0 tx=0x7f8fac009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fa27fc700 10 client.?.objecter ms_handle_connect 0x7f8fb006b9e0 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.093+0000 7f8fa27fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:50.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.094+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac007b00 con 0x7f8fb006b9e0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.094+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8fac00f040 con 0x7f8fb006b9e0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.094+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac014990 con 0x7f8fb006b9e0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.094+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa80097c0 con 0x7f8fb006b9e0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.094+0000 7f8fb746e700 10 client.37802.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.094+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb010b120 con 0x7f8fb006b9e0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.094+0000 7f8fb746e700 1 librados: init done 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.094+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8fb0109320 con 0x7f8fb006b9e0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.095+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f8fac010500 con 0x7f8fb006b9e0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.095+0000 7f8fa27fc700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8f9c062c90 0x7f8f9c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.095+0000 7f8fb520a700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8f9c062c90 0x7f8f9c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.095+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(468..468 src has 173..468) v4 ==== 8137+0+0 (secure 0 0 0) 0x7f8fac07cda0 con 0x7f8fb006b9e0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.095+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 osd_map(468..468 src has 173..468) v4 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.095+0000 7f8fa27fc700 3 client.37802.objecter handle_osd_map got epochs [468,468] > 0 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.095+0000 7f8fa27fc700 3 client.37802.objecter handle_osd_map decoding full epoch 468 2024-02-23T00:37:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fa27fc700 20 client.37802.objecter dump_active .. 0 homeless 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v468) v1 ==== 54+0+16 (secure 0 0 0) 0x7f8fac0824b0 con 0x7f8fb006b9e0 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fa27fc700 10 client.37802.objecter ms_handle_connect 0x7f8f9c062c90 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb520a700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8f9c062c90 0x7f8f9c065150 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 rx=0x7f8fa4004dc0 tx=0x7f8fa400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 10 client.37802.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 1 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 20 client.37802.objecter _get_session osd=-1 returning homeless 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 10 client.37802.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=469}) v3 -- 0x7f8fb010b7f0 con 0x7f8fb006b9e0 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8fb0117000 con 0x7f8fb006b9e0 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=468) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fac050260 con 0x7f8fb006b9e0 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fa37fe700 10 client.37802.objecter _check_command_map_dne tid 1 current 468 map_dne_bound 468 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fa37fe700 10 client.37802.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fa37fe700 15 client.37802.objecter _session_command_op_remove -1 1 2024-02-23T00:37:50.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 10 client.37802.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 2 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6805/685354171 conn(0x7f8fb00f5ed0 0x7f8fb00273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb00f5d60 osd=1 3 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb00f5d60 osd=1 3 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb00f5d60 osd=1 3 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 2 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb00f5d60 osd=1 3 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 1 2 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb00f5d60 osd=1 4 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 10 client.37802.objecter _send_command 2 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fb0059e00 con 0x7f8fb00f5ed0 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.096+0000 7f8fb5a0b700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6805/685354171 conn(0x7f8fb00f5ed0 0x7f8fb00273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.097+0000 7f8fb5a0b700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6805/685354171 conn(0x7f8fb00f5ed0 0x7f8fb00273c0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:50.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.097+0000 7f8fa27fc700 10 client.37802.objecter ms_handle_connect 0x7f8fb00f5ed0 2024-02-23T00:37:50.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.097+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8fb0059e00 con 0x7f8fb00f5ed0 2024-02-23T00:37:50.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.097+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:37:50.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.097+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 2 = system:0 2024-02-23T00:37:50.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.097+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb00f5d60 osd=1 4 2024-02-23T00:37:50.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.097+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 1 2 2024-02-23T00:37:50.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:50.105 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:50.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:50.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 10 client.37802.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 3 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb00f5d60 osd=1 3 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb00f5d60 osd=1 3 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb00f5d60 osd=1 3 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 3 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb00f5d60 osd=1 3 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 1 3 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb00f5d60 osd=1 4 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 10 client.37802.objecter _send_command 3 2024-02-23T00:37:50.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.113+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8fb00278b0 con 0x7f8fb00f5ed0 2024-02-23T00:37:50.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8fb00278b0 con 0x7f8fb00f5ed0 2024-02-23T00:37:50.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:50.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb00f5d60 osd=1 4 2024-02-23T00:37:50.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 1 3 2024-02-23T00:37:50.117 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:37:50.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 10 client.37802.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:50.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 4 2024-02-23T00:37:50.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8fb00594d0 0x7f8fb011b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0059360 osd=2 3 2024-02-23T00:37:50.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0059360 osd=2 3 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0059360 osd=2 3 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 4 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb0059360 osd=2 3 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 2 4 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0059360 osd=2 4 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 10 client.37802.objecter _send_command 4 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.114+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fb0028700 con 0x7f8fb00594d0 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.115+0000 7f8fb520a700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8fb00594d0 0x7f8fb011b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.115+0000 7f8fa27fc700 10 client.37802.objecter ms_handle_connect 0x7f8fb00594d0 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.115+0000 7f8fb520a700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8fb00594d0 0x7f8fb011b190 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.115+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8fb0028700 con 0x7f8fb00594d0 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.115+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.115+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 4 = system:0 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.115+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb0059360 osd=2 4 2024-02-23T00:37:50.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.115+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 2 4 2024-02-23T00:37:50.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 10 client.37802.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 5 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0059360 osd=2 3 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0059360 osd=2 3 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0059360 osd=2 3 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 5 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb0059360 osd=2 3 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 2 5 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0059360 osd=2 4 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 10 client.37802.objecter _send_command 5 2024-02-23T00:37:50.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.130+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8fb0027c60 con 0x7f8fb00594d0 2024-02-23T00:37:50.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.131+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8fb0027c60 con 0x7f8fb00594d0 2024-02-23T00:37:50.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.131+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:50.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.131+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.131+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb0059360 osd=2 4 2024-02-23T00:37:50.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.131+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 2 5 2024-02-23T00:37:50.134 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:37:50.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 10 client.37802.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:50.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 6 2024-02-23T00:37:50.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6813/128059790 conn(0x7f8fb00fec40 0x7f8fb011e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0028070 osd=3 3 2024-02-23T00:37:50.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0028070 osd=3 3 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0028070 osd=3 3 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 6 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb0028070 osd=3 3 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 3 6 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0028070 osd=3 4 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 10 client.37802.objecter _send_command 6 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fb01215c0 con 0x7f8fb00fec40 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb4a09700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6813/128059790 conn(0x7f8fb00fec40 0x7f8fb011e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fa27fc700 10 client.37802.objecter ms_handle_connect 0x7f8fb00fec40 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.132+0000 7f8fb4a09700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.87:6813/128059790 conn(0x7f8fb00fec40 0x7f8fb011e810 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.133+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8fb01215c0 con 0x7f8fb00fec40 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.133+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:37:50.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.133+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 6 = system:0 2024-02-23T00:37:50.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.133+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb0028070 osd=3 4 2024-02-23T00:37:50.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.133+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 3 6 2024-02-23T00:37:50.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 10 client.37802.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:50.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 7 2024-02-23T00:37:50.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0028070 osd=3 3 2024-02-23T00:37:50.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0028070 osd=3 3 2024-02-23T00:37:50.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0028070 osd=3 3 2024-02-23T00:37:50.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 7 2024-02-23T00:37:50.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb0028070 osd=3 3 2024-02-23T00:37:50.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 3 7 2024-02-23T00:37:50.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0028070 osd=3 4 2024-02-23T00:37:50.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 10 client.37802.objecter _send_command 7 2024-02-23T00:37:50.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.147+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8fb00fe200 con 0x7f8fb00fec40 2024-02-23T00:37:50.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.148+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8fb00fe200 con 0x7f8fb00fec40 2024-02-23T00:37:50.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.148+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:50.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.148+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.148+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb0028070 osd=3 4 2024-02-23T00:37:50.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.148+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 3 7 2024-02-23T00:37:50.151 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.148+0000 7f8fb746e700 10 client.37802.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.148+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 8 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8fb011ff20 0x7f8fb01203a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb011be50 osd=4 3 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb011be50 osd=4 3 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb011be50 osd=4 3 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 8 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb011be50 osd=4 3 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 4 8 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb011be50 osd=4 4 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 10 client.37802.objecter _send_command 8 2024-02-23T00:37:50.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fb011c560 con 0x7f8fb011ff20 2024-02-23T00:37:50.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb5a0b700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8fb011ff20 0x7f8fb01203a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fa27fc700 10 client.37802.objecter ms_handle_connect 0x7f8fb011ff20 2024-02-23T00:37:50.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.149+0000 7f8fb5a0b700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8fb011ff20 0x7f8fb01203a0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:50.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.150+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8fb011c560 con 0x7f8fb011ff20 2024-02-23T00:37:50.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.150+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:37:50.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.150+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 8 = system:0 2024-02-23T00:37:50.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.150+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb011be50 osd=4 4 2024-02-23T00:37:50.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.150+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 4 8 2024-02-23T00:37:50.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 10 client.37802.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:50.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 9 2024-02-23T00:37:50.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb011be50 osd=4 3 2024-02-23T00:37:50.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb011be50 osd=4 3 2024-02-23T00:37:50.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb011be50 osd=4 3 2024-02-23T00:37:50.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 9 2024-02-23T00:37:50.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb011be50 osd=4 3 2024-02-23T00:37:50.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 4 9 2024-02-23T00:37:50.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb011be50 osd=4 4 2024-02-23T00:37:50.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 10 client.37802.objecter _send_command 9 2024-02-23T00:37:50.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.165+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8fb0121060 con 0x7f8fb011ff20 2024-02-23T00:37:50.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8fb0121060 con 0x7f8fb011ff20 2024-02-23T00:37:50.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:50.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb011be50 osd=4 4 2024-02-23T00:37:50.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 4 9 2024-02-23T00:37:50.170 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 10 client.37802.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 10 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8fb01237f0 0x7f8fb0125be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0121060 osd=5 3 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0121060 osd=5 3 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0121060 osd=5 3 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 10 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb0121060 osd=5 3 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 5 10 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0121060 osd=5 4 2024-02-23T00:37:50.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 10 client.37802.objecter _send_command 10 2024-02-23T00:37:50.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fb0121270 con 0x7f8fb01237f0 2024-02-23T00:37:50.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.167+0000 7f8fb520a700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8fb01237f0 0x7f8fb0125be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.168+0000 7f8fb520a700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8fb01237f0 0x7f8fb0125be0 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:50.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.168+0000 7f8fa27fc700 10 client.37802.objecter ms_handle_connect 0x7f8fb01237f0 2024-02-23T00:37:50.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.168+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8fb0121270 con 0x7f8fb01237f0 2024-02-23T00:37:50.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.168+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:37:50.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.168+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 10 = system:0 2024-02-23T00:37:50.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.168+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb0121060 osd=5 4 2024-02-23T00:37:50.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.168+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 5 10 2024-02-23T00:37:50.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 10 client.37802.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:50.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 11 2024-02-23T00:37:50.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0121060 osd=5 3 2024-02-23T00:37:50.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0121060 osd=5 3 2024-02-23T00:37:50.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0121060 osd=5 3 2024-02-23T00:37:50.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 11 2024-02-23T00:37:50.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb0121060 osd=5 3 2024-02-23T00:37:50.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 5 11 2024-02-23T00:37:50.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0121060 osd=5 4 2024-02-23T00:37:50.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 10 client.37802.objecter _send_command 11 2024-02-23T00:37:50.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.183+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8fb0121270 con 0x7f8fb01237f0 2024-02-23T00:37:50.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.185+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8fb0121270 con 0x7f8fb01237f0 2024-02-23T00:37:50.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.185+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:50.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.185+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.185+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb0121060 osd=5 4 2024-02-23T00:37:50.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.185+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 5 11 2024-02-23T00:37:50.188 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:37:50.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.185+0000 7f8fb746e700 10 client.37802.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 12 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8fb01265e0 0x7f8fb0128a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0121270 osd=6 3 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0121270 osd=6 3 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0121270 osd=6 3 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 12 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb0121270 osd=6 3 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 6 12 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0121270 osd=6 4 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 10 client.37802.objecter _send_command 12 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fb012b800 con 0x7f8fb01265e0 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb4a09700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8fb01265e0 0x7f8fb0128a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fa27fc700 10 client.37802.objecter ms_handle_connect 0x7f8fb01265e0 2024-02-23T00:37:50.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.186+0000 7f8fb4a09700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8fb01265e0 0x7f8fb0128a30 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:50.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.187+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8fb012b800 con 0x7f8fb01265e0 2024-02-23T00:37:50.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.187+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:37:50.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.187+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 12 = system:0 2024-02-23T00:37:50.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.187+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb0121270 osd=6 4 2024-02-23T00:37:50.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.187+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 6 12 2024-02-23T00:37:50.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.192+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== mon.2 v2:172.21.15.87:3301/0 8 ==== osd_map(469..469 src has 173..469) v4 ==== 668+0+0 (secure 0 0 0) 0x7f8fac07e020 con 0x7f8fb006b9e0 2024-02-23T00:37:50.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.192+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 osd_map(469..469 src has 173..469) v4 2024-02-23T00:37:50.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.192+0000 7f8fa27fc700 3 client.37802.objecter handle_osd_map got epochs [469,469] > 468 2024-02-23T00:37:50.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.192+0000 7f8fa27fc700 3 client.37802.objecter handle_osd_map decoding incremental epoch 469 2024-02-23T00:37:50.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.193+0000 7f8fa27fc700 20 client.37802.objecter dump_active .. 0 homeless 2024-02-23T00:37:50.206 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:50.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 10 client.37802.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:50.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 13 2024-02-23T00:37:50.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0121270 osd=6 3 2024-02-23T00:37:50.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0121270 osd=6 3 2024-02-23T00:37:50.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb0121270 osd=6 3 2024-02-23T00:37:50.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 13 2024-02-23T00:37:50.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb0121270 osd=6 3 2024-02-23T00:37:50.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 6 13 2024-02-23T00:37:50.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb0121270 osd=6 4 2024-02-23T00:37:50.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 10 client.37802.objecter _send_command 13 2024-02-23T00:37:50.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.207+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8fb012b800 con 0x7f8fb01265e0 2024-02-23T00:37:50.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.212+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8fb012b800 con 0x7f8fb01265e0 2024-02-23T00:37:50.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.212+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:50.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.212+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.212+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb0121270 osd=6 4 2024-02-23T00:37:50.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.212+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 6 13 2024-02-23T00:37:50.215 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 10 client.37802.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 14 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6812/373787000 conn(0x7f8fb012ab00 0x7f8fb012af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb012b800 osd=7 3 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb012b800 osd=7 3 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb012b800 osd=7 3 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 14 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb012b800 osd=7 3 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 7 14 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb012b800 osd=7 4 2024-02-23T00:37:50.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 10 client.37802.objecter _send_command 14 2024-02-23T00:37:50.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.213+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8fb012b530 con 0x7f8fb012ab00 2024-02-23T00:37:50.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.219+0000 7f8fb5a0b700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6812/373787000 conn(0x7f8fb012ab00 0x7f8fb012af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:50.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.219+0000 7f8fa27fc700 10 client.37802.objecter ms_handle_connect 0x7f8fb012ab00 2024-02-23T00:37:50.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.219+0000 7f8fb5a0b700 1 --2- 172.21.15.87:0/1295521052 >> v2:172.21.15.171:6812/373787000 conn(0x7f8fb012ab00 0x7f8fb012af80 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:50.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.259+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8fb012b530 con 0x7f8fb012ab00 2024-02-23T00:37:50.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.259+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:37:50.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.259+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 14 = system:0 2024-02-23T00:37:50.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.259+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb012b800 osd=7 4 2024-02-23T00:37:50.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.259+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 7 14 2024-02-23T00:37:50.267 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:50.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 10 client.37802.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:37:50.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign -1 15 2024-02-23T00:37:50.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb012b800 osd=7 3 2024-02-23T00:37:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb012b800 osd=7 3 2024-02-23T00:37:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 20 client.37802.objecter _get_session s=0x7f8fb012b800 osd=7 3 2024-02-23T00:37:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_remove -1 15 2024-02-23T00:37:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 20 client.37802.objecter get_session s=0x7f8fb012b800 osd=7 3 2024-02-23T00:37:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 15 client.37802.objecter _session_command_op_assign 7 15 2024-02-23T00:37:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 20 client.37802.objecter put_session s=0x7f8fb012b800 osd=7 4 2024-02-23T00:37:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 10 client.37802.objecter _send_command 15 2024-02-23T00:37:50.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.286+0000 7f8fb746e700 1 -- 172.21.15.87:0/1295521052 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8fb012b530 con 0x7f8fb012ab00 2024-02-23T00:37:50.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.288+0000 7f8fa27fc700 1 -- 172.21.15.87:0/1295521052 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8fb012b530 con 0x7f8fb012ab00 2024-02-23T00:37:50.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.289+0000 7f8fa27fc700 10 client.37802.objecter ms_dispatch 0x7f8fb0001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:37:50.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.289+0000 7f8fa27fc700 10 client.37802.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.289+0000 7f8fa27fc700 20 client.37802.objecter put_session s=0x7f8fb012b800 osd=7 4 2024-02-23T00:37:50.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:50.289+0000 7f8fa27fc700 15 client.37802.objecter _session_command_op_remove 7 15 2024-02-23T00:37:50.291 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:37:50.292 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:37:50.292 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.292 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.292 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.292 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.292 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.292 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.292 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:37:50.303 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:50.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:50.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:50.425 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:37:50.426 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:37:50.426 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:37:50.426 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:37:50.437 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:50.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:49.182964+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e468: 8 total, 7 up, 7 in 2024-02-23T00:37:50.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:49.197302+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:50.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:49.198268+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:50.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:50.096932+0000 mon.c (mon.2) 444 : audit [DBG] from='client.? 172.21.15.87:0/1295521052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:50.448 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:49.182964+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e468: 8 total, 7 up, 7 in 2024-02-23T00:37:50.448 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:49.197302+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:50.448 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:49.198268+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:50.448 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:50.096932+0000 mon.c (mon.2) 444 : audit [DBG] from='client.? 172.21.15.87:0/1295521052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:50.448 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:49.182964+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e468: 8 total, 7 up, 7 in 2024-02-23T00:37:50.448 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:49.197302+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:50.449 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:49.198268+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:37:50.449 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:50.096932+0000 mon.c (mon.2) 444 : audit [DBG] from='client.? 172.21.15.87:0/1295521052' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:37:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:37:51.115 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:37:51.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:37:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:37:51.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:49.685061+0000 mgr.y (mgr.24437) 441 : cluster [DBG] pgmap v710: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123/1185 objects degraded (10.380%) 2024-02-23T00:37:51.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:50.193673+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:50.193749+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e469: 8 total, 7 up, 7 in 2024-02-23T00:37:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:50.246810+0000 mon.b (mon.1) 384 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:51.503 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:49.685061+0000 mgr.y (mgr.24437) 441 : cluster [DBG] pgmap v710: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123/1185 objects degraded (10.380%) 2024-02-23T00:37:51.503 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:50.193673+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:51.504 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:50.193749+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e469: 8 total, 7 up, 7 in 2024-02-23T00:37:51.504 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:50.246810+0000 mon.b (mon.1) 384 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:51.504 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:49.685061+0000 mgr.y (mgr.24437) 441 : cluster [DBG] pgmap v710: 238 pgs: 32 unknown, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 123/1185 objects degraded (10.380%) 2024-02-23T00:37:51.504 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:50.193673+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:37:51.504 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:50.193749+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e469: 8 total, 7 up, 7 in 2024-02-23T00:37:51.504 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:50.246810+0000 mon.b (mon.1) 384 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:37:51.987 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:51.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:51.995 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:51.996 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:51.997 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-23T00:37:52.091 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:52.125 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:52.154 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:52.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:52.258 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:52.291 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:52.312 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:52.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:52.318 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:52] "GET /metrics HTTP/1.1" 200 188628 "" "Prometheus/2.33.4" 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:50.983065+0000 mgr.y (mgr.24437) 442 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:51.192878+0000 mon.a (mon.0) 2720 : cluster [DBG] osdmap e470: 8 total, 7 up, 7 in 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:52.113502+0000 mon.a (mon.0) 2721 : cluster [WRN] Health check update: Degraded data redundancy: 123/1185 objects degraded (10.380%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:52.119430+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e471: 8 total, 7 up, 7 in 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:52.163459+0000 mon.b (mon.1) 385 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.14"}]: dispatch 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:50.983065+0000 mgr.y (mgr.24437) 442 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:51.192878+0000 mon.a (mon.0) 2720 : cluster [DBG] osdmap e470: 8 total, 7 up, 7 in 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:52.113502+0000 mon.a (mon.0) 2721 : cluster [WRN] Health check update: Degraded data redundancy: 123/1185 objects degraded (10.380%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:52.119430+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e471: 8 total, 7 up, 7 in 2024-02-23T00:37:52.319 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:52.163459+0000 mon.b (mon.1) 385 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.14"}]: dispatch 2024-02-23T00:37:52.415 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:52.452 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:50.983065+0000 mgr.y (mgr.24437) 442 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:37:52.452 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:51.192878+0000 mon.a (mon.0) 2720 : cluster [DBG] osdmap e470: 8 total, 7 up, 7 in 2024-02-23T00:37:52.452 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:52.113502+0000 mon.a (mon.0) 2721 : cluster [WRN] Health check update: Degraded data redundancy: 123/1185 objects degraded (10.380%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:52.452 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:52.119430+0000 mon.a (mon.0) 2722 : cluster [DBG] osdmap e471: 8 total, 7 up, 7 in 2024-02-23T00:37:52.452 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:52.163459+0000 mon.b (mon.1) 385 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.14"}]: dispatch 2024-02-23T00:37:52.478 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:52.496 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:52.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:52.600 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:52.628 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:53.077 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:53.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:53.116 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:37:53.183 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:53.213 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:53.233 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:53.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:51.685705+0000 mgr.y (mgr.24437) 443 : cluster [DBG] pgmap v713: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 124/1200 objects degraded (10.333%) 2024-02-23T00:37:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:52.164422+0000 mgr.y (mgr.24437) 444 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.14"}]: dispatch 2024-02-23T00:37:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:51.685705+0000 mgr.y (mgr.24437) 443 : cluster [DBG] pgmap v713: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 124/1200 objects degraded (10.333%) 2024-02-23T00:37:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:52.164422+0000 mgr.y (mgr.24437) 444 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.14"}]: dispatch 2024-02-23T00:37:53.336 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:53.363 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:53.561 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:53.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:53.665 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:51.685705+0000 mgr.y (mgr.24437) 443 : cluster [DBG] pgmap v713: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s; 124/1200 objects degraded (10.333%) 2024-02-23T00:37:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:52.164422+0000 mgr.y (mgr.24437) 444 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.14"}]: dispatch 2024-02-23T00:37:53.696 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:53.718 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:53.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:53.822 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:53.851 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:53.871 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:53.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:53.975 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:54.000 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:37:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:37:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:37:54.002 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:54.021 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:54.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:54.125 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.157+0000 7f46e4c26700 1 -- 172.21.15.87:0/1887090470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fe510 msgr2=0x7f46e00f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.157+0000 7f46e4c26700 1 --2- 172.21.15.87:0/1887090470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fe510 0x7f46e00f7d40 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 rx=0x7f46d4004870 tx=0x7f46d4009cb0).stop 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.157+0000 7f46e4c26700 1 -- 172.21.15.87:0/1887090470 shutdown_connections 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.157+0000 7f46e4c26700 1 --2- 172.21.15.87:0/1887090470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46e00f8420 0x7f46e0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.157+0000 7f46e4c26700 1 --2- 172.21.15.87:0/1887090470 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f46e00fdbb0 0x7f46e00fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.157+0000 7f46e4c26700 1 --2- 172.21.15.87:0/1887090470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fe510 0x7f46e00f7d40 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.157+0000 7f46e4c26700 1 -- 172.21.15.87:0/1887090470 >> 172.21.15.87:0/1887090470 conn(0x7f46e00f3800 msgr2=0x7f46e00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.157+0000 7f46e4c26700 1 -- 172.21.15.87:0/1887090470 shutdown_connections 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.157+0000 7f46e4c26700 1 -- 172.21.15.87:0/1887090470 wait complete. 2024-02-23T00:37:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 librados: starting msgr at 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 librados: starting objecter 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 Processor -- start 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 -- start start 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 librados: setting wanted keys 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 librados: calling monclient init 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f46e00f8420 0x7f46e006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fdbb0 0x7f46e006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46e00fe510 0x7f46e006bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e0116490 con 0x7f46e00fdbb0 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e0116600 con 0x7f46e00fe510 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46e4c26700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e01167c0 con 0x7f46e00f8420 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46ddd9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fdbb0 0x7f46e006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46ddd9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fdbb0 0x7f46e006b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56148/0 (socket says 172.21.15.87:56148) 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.158+0000 7f46ddd9b700 1 -- 172.21.15.87:0/84768616 learned_addr learned my addr 172.21.15.87:0/84768616 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46de59c700 1 --2- 172.21.15.87:0/84768616 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f46e00f8420 0x7f46e006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46ded9d700 1 --2- 172.21.15.87:0/84768616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46e00fe510 0x7f46e006bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46ddd9b700 1 -- 172.21.15.87:0/84768616 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f46e00f8420 msgr2=0x7f46e006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46ddd9b700 1 --2- 172.21.15.87:0/84768616 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f46e00f8420 0x7f46e006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46ddd9b700 1 -- 172.21.15.87:0/84768616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46e00fe510 msgr2=0x7f46e006bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46ddd9b700 1 --2- 172.21.15.87:0/84768616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46e00fe510 0x7f46e006bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46ddd9b700 1 -- 172.21.15.87:0/84768616 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d4005040 con 0x7f46e00fdbb0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46ddd9b700 1 --2- 172.21.15.87:0/84768616 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fdbb0 0x7f46e006b5c0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7f46d4004fc0 tx=0x7f46d4009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46cf7fe700 10 client.?.objecter ms_handle_connect 0x7f46e00fdbb0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46cf7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46cf7fe700 1 -- 172.21.15.87:0/84768616 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d400e030 con 0x7f46e00fdbb0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46cf7fe700 1 -- 172.21.15.87:0/84768616 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46d4013d60 con 0x7f46e00fdbb0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e0116a40 con 0x7f46e00fdbb0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46cf7fe700 1 -- 172.21.15.87:0/84768616 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d4018070 con 0x7f46e00fdbb0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46e4c26700 10 client.27342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e0116df0 con 0x7f46e00fdbb0 2024-02-23T00:37:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.159+0000 7f46e4c26700 1 librados: init done 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.160+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e0107110 con 0x7f46e00fdbb0 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.160+0000 7f46cf7fe700 1 -- 172.21.15.87:0/84768616 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f46d4027440 con 0x7f46e00fdbb0 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.161+0000 7f46cf7fe700 1 --2- 172.21.15.87:0/84768616 >> v2:172.21.15.87:6800/1231597463 conn(0x7f46c8062c90 0x7f46c8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.161+0000 7f46cf7fe700 1 -- 172.21.15.87:0/84768616 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(471..471 src has 173..471) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f46d407e5f0 con 0x7f46e00fdbb0 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.161+0000 7f46de59c700 1 --2- 172.21.15.87:0/84768616 >> v2:172.21.15.87:6800/1231597463 conn(0x7f46c8062c90 0x7f46c8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.161+0000 7f46cf7fe700 10 client.27342.objecter ms_dispatch 0x7f46e0001550 osd_map(471..471 src has 173..471) v4 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.161+0000 7f46cf7fe700 3 client.27342.objecter handle_osd_map got epochs [471,471] > 0 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.161+0000 7f46cf7fe700 3 client.27342.objecter handle_osd_map decoding full epoch 471 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.161+0000 7f46cf7fe700 20 client.27342.objecter dump_active .. 0 homeless 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.161+0000 7f46cf7fe700 10 client.27342.objecter ms_handle_connect 0x7f46c8062c90 2024-02-23T00:37:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.161+0000 7f46de59c700 1 --2- 172.21.15.87:0/84768616 >> v2:172.21.15.87:6800/1231597463 conn(0x7f46c8062c90 0x7f46c8065150 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 rx=0x7f46d0004d30 tx=0x7f46d000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:54.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.166+0000 7f46cf7fe700 1 -- 172.21.15.87:0/84768616 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f46d4051a40 con 0x7f46e00fdbb0 2024-02-23T00:37:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:53.210925+0000 osd.7 (osd.7) 235 : cluster [DBG] 219.14 deep-scrub starts 2024-02-23T00:37:54.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:53.212655+0000 osd.7 (osd.7) 236 : cluster [DBG] 219.14 deep-scrub ok 2024-02-23T00:37:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:53.210925+0000 osd.7 (osd.7) 235 : cluster [DBG] 219.14 deep-scrub starts 2024-02-23T00:37:54.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:53.212655+0000 osd.7 (osd.7) 236 : cluster [DBG] 219.14 deep-scrub ok 2024-02-23T00:37:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:54.346+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "3.9", "id": [6, 2, 0, 7, 5, 3]} v 0) v1 -- 0x7f46e005fac0 con 0x7f46e00fdbb0 2024-02-23T00:37:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:53.210925+0000 osd.7 (osd.7) 235 : cluster [DBG] 219.14 deep-scrub starts 2024-02-23T00:37:54.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:53.212655+0000 osd.7 (osd.7) 236 : cluster [DBG] 219.14 deep-scrub ok 2024-02-23T00:37:54.733 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:55.171 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:37:55.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.221+0000 7f46cf7fe700 1 -- 172.21.15.87:0/84768616 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "3.9", "id": [6, 2, 0, 7, 5, 3]}]=0 set 3.9 pg_upmap_items mapping to [6->2,0->7,5->3] v472) v1 ==== 157+0+0 (secure 0 0 0) 0x7f46d4051630 con 0x7f46e00fdbb0 2024-02-23T00:37:55.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 10 librados: watch_flush enter 2024-02-23T00:37:55.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 10 librados: watch_flush exit 2024-02-23T00:37:55.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 20 client.27342.objecter shutdown clearing up homeless session... 2024-02-23T00:37:55.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 10 client.27342.objecter successfully canceled tick 2024-02-23T00:37:55.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 >> v2:172.21.15.87:6800/1231597463 conn(0x7f46c8062c90 msgr2=0x7f46c8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:55.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 --2- 172.21.15.87:0/84768616 >> v2:172.21.15.87:6800/1231597463 conn(0x7f46c8062c90 0x7f46c8065150 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 rx=0x7f46d0004d30 tx=0x7f46d000b040).stop 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fdbb0 msgr2=0x7f46e006b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 --2- 172.21.15.87:0/84768616 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fdbb0 0x7f46e006b5c0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 rx=0x7f46d4004fc0 tx=0x7f46d4009cb0).stop 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 shutdown_connections 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 --2- 172.21.15.87:0/84768616 >> v2:172.21.15.87:6800/1231597463 conn(0x7f46c8062c90 0x7f46c8065150 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 --2- 172.21.15.87:0/84768616 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f46e00fe510 0x7f46e006bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 --2- 172.21.15.87:0/84768616 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f46e00f8420 0x7f46e006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 --2- 172.21.15.87:0/84768616 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f46e00fdbb0 0x7f46e006b5c0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 >> 172.21.15.87:0/84768616 conn(0x7f46e00f3800 msgr2=0x7f46e00fb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 shutdown_connections 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 -- 172.21.15.87:0/84768616 wait complete. 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:55.223+0000 7f46e4c26700 1 librados: shutdown 2024-02-23T00:37:55.227 INFO:teuthology.orchestra.run.smithi087.stderr:set 3.9 pg_upmap_items mapping to [6->2,0->7,5->3] 2024-02-23T00:37:55.457 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:53.686405+0000 mgr.y (mgr.24437) 445 : cluster [DBG] pgmap v715: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 124/1200 objects degraded (10.333%) 2024-02-23T00:37:55.458 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:54.348530+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? 172.21.15.87:0/84768616' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.9", "id": [6, 2, 0, 7, 5, 3]}]: dispatch 2024-02-23T00:37:55.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:53.686405+0000 mgr.y (mgr.24437) 445 : cluster [DBG] pgmap v715: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 124/1200 objects degraded (10.333%) 2024-02-23T00:37:55.461 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:54.348530+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? 172.21.15.87:0/84768616' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.9", "id": [6, 2, 0, 7, 5, 3]}]: dispatch 2024-02-23T00:37:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:53.686405+0000 mgr.y (mgr.24437) 445 : cluster [DBG] pgmap v715: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 2.2 KiB/s wr, 3 op/s; 124/1200 objects degraded (10.333%) 2024-02-23T00:37:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:54.348530+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? 172.21.15.87:0/84768616' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.9", "id": [6, 2, 0, 7, 5, 3]}]: dispatch 2024-02-23T00:37:55.704 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:55.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:55.808 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:55.837 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:56.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 -- 172.21.15.87:0/1548396218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5dfc0f8420 msgr2=0x7f5dfc104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:56.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1548396218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5dfc0f8420 0x7f5dfc104a20 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 rx=0x7f5dec009c30 tx=0x7f5dec004c50).stop 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 -- 172.21.15.87:0/1548396218 shutdown_connections 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1548396218 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5dfc0f8420 0x7f5dfc104a20 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1548396218 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5dfc0fdbb0 0x7f5dfc0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1548396218 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dfc0fe510 0x7f5dfc0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 -- 172.21.15.87:0/1548396218 >> 172.21.15.87:0/1548396218 conn(0x7f5dfc0f3800 msgr2=0x7f5dfc0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 -- 172.21.15.87:0/1548396218 shutdown_connections 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 -- 172.21.15.87:0/1548396218 wait complete. 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 librados: starting msgr at 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 librados: starting objecter 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 Processor -- start 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 -- start start 2024-02-23T00:37:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 librados: setting wanted keys 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.344+0000 7f5e03f57700 1 librados: calling monclient init 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5dfc0f8420 0x7f5dfc10df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dfc0fdbb0 0x7f5dfc10e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5dfc0fe510 0x7f5dfc108160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc108650 con 0x7f5dfc0fe510 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc1087c0 con 0x7f5dfc0fdbb0 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc108930 con 0x7f5dfc0f8420 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e01cf3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5dfc0f8420 0x7f5dfc10df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e01cf3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5dfc0f8420 0x7f5dfc10df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:55900/0 (socket says 172.21.15.87:55900) 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e01cf3700 1 -- 172.21.15.87:0/1056137555 learned_addr learned my addr 172.21.15.87:0/1056137555 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e024f4700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5dfc0fe510 0x7f5dfc108160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e014f2700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dfc0fdbb0 0x7f5dfc10e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e01cf3700 1 -- 172.21.15.87:0/1056137555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dfc0fdbb0 msgr2=0x7f5dfc10e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e01cf3700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dfc0fdbb0 0x7f5dfc10e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e01cf3700 1 -- 172.21.15.87:0/1056137555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5dfc0fe510 msgr2=0x7f5dfc108160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e01cf3700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5dfc0fe510 0x7f5dfc108160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e01cf3700 1 -- 172.21.15.87:0/1056137555 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec0097c0 con 0x7f5dfc0f8420 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e024f4700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5dfc0fe510 0x7f5dfc108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e01cf3700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5dfc0f8420 0x7f5dfc10df80 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7f5df400e970 tx=0x7f5df400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5dfaffd700 10 client.?.objecter ms_handle_connect 0x7f5dfc0f8420 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5dfaffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5dfaffd700 1 -- 172.21.15.87:0/1056137555 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df40046e0 con 0x7f5dfc0f8420 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5dfaffd700 1 -- 172.21.15.87:0/1056137555 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5df400f040 con 0x7f5dfc0f8420 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dfc108c10 con 0x7f5dfc0f8420 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5dfaffd700 1 -- 172.21.15.87:0/1056137555 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df40146f0 con 0x7f5dfc0f8420 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 10 client.37814.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dfc114e70 con 0x7f5dfc0f8420 2024-02-23T00:37:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.345+0000 7f5e03f57700 1 librados: init done 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.346+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dfc05f950 con 0x7f5dfc0f8420 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5dfaffd700 1 -- 172.21.15.87:0/1056137555 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f5df40090d0 con 0x7f5dfc0f8420 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5dfaffd700 1 --2- 172.21.15.87:0/1056137555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5de4062c90 0x7f5de4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5dfaffd700 1 -- 172.21.15.87:0/1056137555 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(472..472 src has 173..472) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f5df407cc90 con 0x7f5dfc0f8420 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5dfaffd700 10 client.37814.objecter ms_dispatch 0x7f5dfc001550 osd_map(472..472 src has 173..472) v4 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5e014f2700 1 --2- 172.21.15.87:0/1056137555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5de4062c90 0x7f5de4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5dfaffd700 3 client.37814.objecter handle_osd_map got epochs [472,472] > 0 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5dfaffd700 3 client.37814.objecter handle_osd_map decoding full epoch 472 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5dfaffd700 20 client.37814.objecter dump_active .. 0 homeless 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5dfaffd700 10 client.37814.objecter ms_handle_connect 0x7f5de4062c90 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.347+0000 7f5e014f2700 1 --2- 172.21.15.87:0/1056137555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5de4062c90 0x7f5de4065150 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7f5dfc109900 tx=0x7f5df000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:37:56.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.351+0000 7f5dfaffd700 1 -- 172.21.15.87:0/1056137555 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5df40501f0 con 0x7f5dfc0f8420 2024-02-23T00:37:56.384 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:56.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:37:56.488 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:56.513 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:55.222147+0000 mon.a (mon.0) 2724 : audit [INF] from='client.? 172.21.15.87:0/84768616' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.9", "id": [6, 2, 0, 7, 5, 3]}]': finished 2024-02-23T00:37:56.513 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:55.222204+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e472: 8 total, 7 up, 7 in 2024-02-23T00:37:56.514 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:56.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:56.526+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f5dfc05ee80 con 0x7f5dfc0f8420 2024-02-23T00:37:56.532 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:56.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:55.222147+0000 mon.a (mon.0) 2724 : audit [INF] from='client.? 172.21.15.87:0/84768616' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.9", "id": [6, 2, 0, 7, 5, 3]}]': finished 2024-02-23T00:37:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:55.222204+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e472: 8 total, 7 up, 7 in 2024-02-23T00:37:56.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:55.222147+0000 mon.a (mon.0) 2724 : audit [INF] from='client.? 172.21.15.87:0/84768616' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.9", "id": [6, 2, 0, 7, 5, 3]}]': finished 2024-02-23T00:37:56.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:55.222204+0000 mon.a (mon.0) 2725 : cluster [DBG] osdmap e472: 8 total, 7 up, 7 in 2024-02-23T00:37:56.637 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:56.663 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:56.682 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:56.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:56.785 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:56.813 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:56.831 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:56.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:56.935 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:56.964 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:56.991 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:56.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:57.096 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:57.124 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:57.125 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.123+0000 7f5dfaffd700 1 -- 172.21.15.87:0/1056137555 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v473) v1 ==== 91+0+0 (secure 0 0 0) 0x7f5df4019070 con 0x7f5dfc0f8420 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 10 librados: watch_flush enter 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 10 librados: watch_flush exit 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 20 client.37814.objecter shutdown clearing up homeless session... 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 10 client.37814.objecter successfully canceled tick 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5de4062c90 msgr2=0x7f5de4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1056137555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5de4062c90 0x7f5de4065150 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 rx=0x7f5dfc109900 tx=0x7f5df000b040).stop 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5dfc0f8420 msgr2=0x7f5dfc10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5dfc0f8420 0x7f5dfc10df80 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 rx=0x7f5df400e970 tx=0x7f5df400b040).stop 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 shutdown_connections 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1056137555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5de4062c90 0x7f5de4065150 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:57.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5dfc0fe510 0x7f5dfc108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:57.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5dfc0f8420 0x7f5dfc10df80 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:57.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 --2- 172.21.15.87:0/1056137555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5dfc0fdbb0 0x7f5dfc10e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:37:57.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 >> 172.21.15.87:0/1056137555 conn(0x7f5dfc0f3800 msgr2=0x7f5dfc0f4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:37:57.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 shutdown_connections 2024-02-23T00:37:57.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 -- 172.21.15.87:0/1056137555 wait complete. 2024-02-23T00:37:57.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:37:57.127+0000 7f5e03f57700 1 librados: shutdown 2024-02-23T00:37:57.131 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:37:57.150 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:57.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:57.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:57.253 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:57.280 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:57.308 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:57.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:57.412 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:37:57.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:55.687418+0000 mgr.y (mgr.24437) 446 : cluster [DBG] pgmap v717: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:37:57.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:56.528460+0000 mon.c (mon.2) 445 : audit [INF] from='client.? 172.21.15.87:0/1056137555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:57.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:56.528741+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:57.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:57.115359+0000 mon.a (mon.0) 2727 : cluster [WRN] Health check update: Degraded data redundancy: 124/1206 objects degraded (10.282%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:57.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:57.115895+0000 mon.a (mon.0) 2728 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:57.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:37:57.123743+0000 mon.a (mon.0) 2729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:37:57.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:57.123786+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e473: 8 total, 7 up, 7 in 2024-02-23T00:37:57.448 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:57.474 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:37:57.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:57.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:37:57.578 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:55.687418+0000 mgr.y (mgr.24437) 446 : cluster [DBG] pgmap v717: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:56.528460+0000 mon.c (mon.2) 445 : audit [INF] from='client.? 172.21.15.87:0/1056137555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:56.528741+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:57.115359+0000 mon.a (mon.0) 2727 : cluster [WRN] Health check update: Degraded data redundancy: 124/1206 objects degraded (10.282%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:57.115895+0000 mon.a (mon.0) 2728 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:37:57.123743+0000 mon.a (mon.0) 2729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:57.123786+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e473: 8 total, 7 up, 7 in 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:55.687418+0000 mgr.y (mgr.24437) 446 : cluster [DBG] pgmap v717: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.2 KiB/s wr, 5 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:56.528460+0000 mon.c (mon.2) 445 : audit [INF] from='client.? 172.21.15.87:0/1056137555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:56.528741+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:37:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:57.115359+0000 mon.a (mon.0) 2727 : cluster [WRN] Health check update: Degraded data redundancy: 124/1206 objects degraded (10.282%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:37:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:57.115895+0000 mon.a (mon.0) 2728 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:37:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:37:57.123743+0000 mon.a (mon.0) 2729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:37:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:57.123786+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e473: 8 total, 7 up, 7 in 2024-02-23T00:37:57.603 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:57.622 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:57.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:57.725 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:37:57.752 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:57.770 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:37:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:57.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:37:57.873 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:37:57.900 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:37:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:58.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:37:58.252 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:58.281 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:58.300 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:58.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:58.404 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:58.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:58.129570+0000 osd.4 (osd.4) 215 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:37:58.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:58.130475+0000 osd.4 (osd.4) 216 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:37:58.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:58.129570+0000 osd.4 (osd.4) 215 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:37:58.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:58.130475+0000 osd.4 (osd.4) 216 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:37:58.432 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:58.630 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:58.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:58.129570+0000 osd.4 (osd.4) 215 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:37:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:58.130475+0000 osd.4 (osd.4) 216 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:37:58.734 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:37:58.766 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:58.787 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:37:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:58.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:37:58.891 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:37:58.917 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:59.365 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:37:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:59.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:37:59.469 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:59.499 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:57.688239+0000 mgr.y (mgr.24437) 447 : cluster [DBG] pgmap v719: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 341 B/s wr, 1 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:37:59.499 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:59.128323+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:37:59.499 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:37:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:59.129596+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:37:59.501 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:59.523 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:59.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:57.688239+0000 mgr.y (mgr.24437) 447 : cluster [DBG] pgmap v719: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 341 B/s wr, 1 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:37:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:59.128323+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:37:59.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:37:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:59.129596+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:37:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:57.688239+0000 mgr.y (mgr.24437) 447 : cluster [DBG] pgmap v719: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 341 B/s wr, 1 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:37:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:59.128323+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:37:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:37:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:59.129596+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:37:59.627 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:37:59.658 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:59.679 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:37:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:59.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:37:59.783 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:37:59.811 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:37:59.936 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:37:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:37:59.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:00.040 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:00.071 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:00.093 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:00.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:00.141 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:38:00.197 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:00.228 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:00.252 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:00.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:00.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:00.382 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:59.002617+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:38:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:59.003576+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:38:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:00.081012+0000 osd.4 (osd.4) 219 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:38:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:00.081995+0000 osd.4 (osd.4) 220 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:38:00.481 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:00.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:59.002617+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:38:00.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:59.003576+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:38:00.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:00.081012+0000 osd.4 (osd.4) 219 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:38:00.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:00.081995+0000 osd.4 (osd.4) 220 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:38:00.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:59.002617+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:38:00.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:59.003576+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:38:00.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:00.081012+0000 osd.4 (osd.4) 219 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:38:00.484 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:00.081995+0000 osd.4 (osd.4) 220 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:38:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:00.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:00.488 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:38:00.488 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:38:00.488 INFO:teuthology.orchestra.run.smithi087.stdout: "complaint_time": 30, 2024-02-23T00:38:00.488 INFO:teuthology.orchestra.run.smithi087.stdout: "num_blocked_ops": 0 2024-02-23T00:38:00.488 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:38:00.586 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:00.612 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.735+0000 7f88411fb700 1 -- 172.21.15.87:0/1470708679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f883c0fe510 msgr2=0x7f883c0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.735+0000 7f88411fb700 1 --2- 172.21.15.87:0/1470708679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f883c0fe510 0x7f883c0f7d40 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 rx=0x7f8830004870 tx=0x7f8830009cb0).stop 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.735+0000 7f88411fb700 1 -- 172.21.15.87:0/1470708679 shutdown_connections 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.735+0000 7f88411fb700 1 --2- 172.21.15.87:0/1470708679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f883c0f8420 0x7f883c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.735+0000 7f88411fb700 1 --2- 172.21.15.87:0/1470708679 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f883c0fdbb0 0x7f883c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.735+0000 7f88411fb700 1 --2- 172.21.15.87:0/1470708679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f883c0fe510 0x7f883c0f7d40 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.735+0000 7f88411fb700 1 -- 172.21.15.87:0/1470708679 >> 172.21.15.87:0/1470708679 conn(0x7f883c0f3800 msgr2=0x7f883c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.735+0000 7f88411fb700 1 -- 172.21.15.87:0/1470708679 shutdown_connections 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.735+0000 7f88411fb700 1 -- 172.21.15.87:0/1470708679 wait complete. 2024-02-23T00:38:00.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 librados: starting msgr at 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 librados: starting objecter 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 Processor -- start 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 -- start start 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 librados: setting wanted keys 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 librados: calling monclient init 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f883c0f8420 0x7f883c110310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f883c0fdbb0 0x7f883c110800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f883c0fe510 0x7f883c10a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c10a9a0 con 0x7f883c0fdbb0 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c10ab10 con 0x7f883c0f8420 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f88411fb700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883c10acd0 con 0x7f883c0fe510 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f883a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f883c0fdbb0 0x7f883c110800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f883b59e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f883c0fe510 0x7f883c10a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f883a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f883c0fdbb0 0x7f883c110800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56190/0 (socket says 172.21.15.87:56190) 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f883b59e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f883c0fe510 0x7f883c10a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:55928/0 (socket says 172.21.15.87:55928) 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.736+0000 7f883a59c700 1 -- 172.21.15.87:0/1063479808 learned_addr learned my addr 172.21.15.87:0/1063479808 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f883ad9d700 1 --2- 172.21.15.87:0/1063479808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f883c0f8420 0x7f883c110310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f883a59c700 1 -- 172.21.15.87:0/1063479808 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f883c0fe510 msgr2=0x7f883c10a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f883a59c700 1 --2- 172.21.15.87:0/1063479808 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f883c0fe510 0x7f883c10a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f883a59c700 1 -- 172.21.15.87:0/1063479808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f883c0f8420 msgr2=0x7f883c110310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f883a59c700 1 --2- 172.21.15.87:0/1063479808 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f883c0f8420 0x7f883c110310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f883a59c700 1 -- 172.21.15.87:0/1063479808 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c005040 con 0x7f883c0fdbb0 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f883b59e700 1 --2- 172.21.15.87:0/1063479808 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f883c0fe510 0x7f883c10a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f883a59c700 1 --2- 172.21.15.87:0/1063479808 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f883c0fdbb0 0x7f883c110800 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 rx=0x7f8830009eb0 tx=0x7f883000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f8823fff700 10 client.?.objecter ms_handle_connect 0x7f883c0fdbb0 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f8823fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883000e070 con 0x7f883c0fdbb0 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8830013da0 con 0x7f883c0fdbb0 2024-02-23T00:38:00.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8830005040 con 0x7f883c0fdbb0 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830019070 con 0x7f883c0fdbb0 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f88411fb700 10 client.27354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883c10b1c0 con 0x7f883c0fdbb0 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.737+0000 7f88411fb700 1 librados: init done 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.738+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f883001e420 con 0x7f883c0fdbb0 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f883c05ee80 con 0x7f883c0fdbb0 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f8823fff700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8824062c90 0x7f8824065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f883ad9d700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8824062c90 0x7f8824065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(473..473 src has 173..473) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f883007dca0 con 0x7f883c0fdbb0 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 osd_map(473..473 src has 173..473) v4 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f8823fff700 3 client.27354.objecter handle_osd_map got epochs [473,473] > 0 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f8823fff700 3 client.27354.objecter handle_osd_map decoding full epoch 473 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f8823fff700 20 client.27354.objecter dump_active .. 0 homeless 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v473) v1 ==== 54+0+16 (secure 0 0 0) 0x7f8830083070 con 0x7f883c0fdbb0 2024-02-23T00:38:00.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f8823fff700 10 client.27354.objecter ms_handle_connect 0x7f8824062c90 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f883ad9d700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8824062c90 0x7f8824065150 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 rx=0x7f882c0092f0 tx=0x7f882c009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f88411fb700 10 client.27354.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 1 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f88411fb700 20 client.27354.objecter _get_session osd=-1 returning homeless 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f88411fb700 10 client.27354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=474}) v3 -- 0x7f883c107060 con 0x7f883c0fdbb0 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f883c05fa60 con 0x7f883c0fdbb0 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_get_version_reply(handle=1 version=473) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88300511c0 con 0x7f883c0fdbb0 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f883959a700 10 client.27354.objecter _check_command_map_dne tid 1 current 473 map_dne_bound 473 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f883959a700 10 client.27354.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.739+0000 7f883959a700 15 client.27354.objecter _session_command_op_remove -1 1 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 10 client.27354.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 2 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6805/685354171 conn(0x7f883c06c9e0 0x7f883c0273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c06c870 osd=1 3 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c06c870 osd=1 3 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c06c870 osd=1 3 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 2 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c06c870 osd=1 3 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 1 2 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c06c870 osd=1 4 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 10 client.27354.objecter _send_command 2 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f883c059e00 con 0x7f883c06c9e0 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f883b59e700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6805/685354171 conn(0x7f883c06c9e0 0x7f883c0273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f883b59e700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6805/685354171 conn(0x7f883c06c9e0 0x7f883c0273c0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:00.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.740+0000 7f8823fff700 10 client.27354.objecter ms_handle_connect 0x7f883c06c9e0 2024-02-23T00:38:00.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.741+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f883c059e00 con 0x7f883c06c9e0 2024-02-23T00:38:00.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.741+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:38:00.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.741+0000 7f8823fff700 10 client.27354.objecter _finish_command 2 = system:0 2024-02-23T00:38:00.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.741+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c06c870 osd=1 4 2024-02-23T00:38:00.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.741+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 1 2 2024-02-23T00:38:00.753 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:00.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 10 client.27354.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 3 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c06c870 osd=1 3 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c06c870 osd=1 3 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c06c870 osd=1 3 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 3 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c06c870 osd=1 3 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 1 3 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c06c870 osd=1 4 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.760+0000 7f88411fb700 10 client.27354.objecter _send_command 3 2024-02-23T00:38:00.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.761+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f883c0278b0 con 0x7f883c06c9e0 2024-02-23T00:38:00.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f883c0278b0 con 0x7f883c06c9e0 2024-02-23T00:38:00.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:00.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f8823fff700 10 client.27354.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c06c870 osd=1 4 2024-02-23T00:38:00.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 1 3 2024-02-23T00:38:00.765 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:38:00.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 10 client.27354.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:00.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 4 2024-02-23T00:38:00.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6809/3605394228 conn(0x7f883c0594d0 0x7f883c11b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c059360 osd=2 3 2024-02-23T00:38:00.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c059360 osd=2 3 2024-02-23T00:38:00.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c059360 osd=2 3 2024-02-23T00:38:00.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 4 2024-02-23T00:38:00.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c059360 osd=2 3 2024-02-23T00:38:00.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 2 4 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c059360 osd=2 4 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 10 client.27354.objecter _send_command 4 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f883c028760 con 0x7f883c0594d0 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.762+0000 7f883ad9d700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6809/3605394228 conn(0x7f883c0594d0 0x7f883c11b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.763+0000 7f8823fff700 10 client.27354.objecter ms_handle_connect 0x7f883c0594d0 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.763+0000 7f883ad9d700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6809/3605394228 conn(0x7f883c0594d0 0x7f883c11b160 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.763+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f883c028760 con 0x7f883c0594d0 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.763+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.763+0000 7f8823fff700 10 client.27354.objecter _finish_command 4 = system:0 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.763+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c059360 osd=2 4 2024-02-23T00:38:00.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.763+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 2 4 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 10 client.27354.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 5 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c059360 osd=2 3 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c059360 osd=2 3 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c059360 osd=2 3 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 5 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c059360 osd=2 3 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 2 5 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c059360 osd=2 4 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 10 client.27354.objecter _send_command 5 2024-02-23T00:38:00.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.778+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f883c027d50 con 0x7f883c0594d0 2024-02-23T00:38:00.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f883c027d50 con 0x7f883c0594d0 2024-02-23T00:38:00.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:00.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f8823fff700 10 client.27354.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c059360 osd=2 4 2024-02-23T00:38:00.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 2 5 2024-02-23T00:38:00.782 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 10 client.27354.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 6 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6813/128059790 conn(0x7f883c11c7d0 0x7f883c11ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c0f9fc0 osd=3 3 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c0f9fc0 osd=3 3 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c0f9fc0 osd=3 3 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 6 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c0f9fc0 osd=3 3 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 3 6 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c0f9fc0 osd=3 4 2024-02-23T00:38:00.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 10 client.27354.objecter _send_command 6 2024-02-23T00:38:00.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f883c121810 con 0x7f883c11c7d0 2024-02-23T00:38:00.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.779+0000 7f883a59c700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6813/128059790 conn(0x7f883c11c7d0 0x7f883c11ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.780+0000 7f8823fff700 10 client.27354.objecter ms_handle_connect 0x7f883c11c7d0 2024-02-23T00:38:00.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.780+0000 7f883a59c700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.87:6813/128059790 conn(0x7f883c11c7d0 0x7f883c11ebc0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:00.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.780+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f883c121810 con 0x7f883c11c7d0 2024-02-23T00:38:00.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.780+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:38:00.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.780+0000 7f8823fff700 10 client.27354.objecter _finish_command 6 = system:0 2024-02-23T00:38:00.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.780+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c0f9fc0 osd=3 4 2024-02-23T00:38:00.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.780+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 3 6 2024-02-23T00:38:00.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 10 client.27354.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 7 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c0f9fc0 osd=3 3 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c0f9fc0 osd=3 3 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c0f9fc0 osd=3 3 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 7 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c0f9fc0 osd=3 3 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 3 7 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c0f9fc0 osd=3 4 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 10 client.27354.objecter _send_command 7 2024-02-23T00:38:00.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.794+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f883c0f9a90 con 0x7f883c11c7d0 2024-02-23T00:38:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f883c0f9a90 con 0x7f883c11c7d0 2024-02-23T00:38:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f8823fff700 10 client.27354.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c0f9fc0 osd=3 4 2024-02-23T00:38:00.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 3 7 2024-02-23T00:38:00.799 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:38:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 10 client.27354.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 8 2024-02-23T00:38:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6800/4175844757 conn(0x7f883c120230 0x7f883c120670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c11be20 osd=4 3 2024-02-23T00:38:00.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c11be20 osd=4 3 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c11be20 osd=4 3 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 8 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c11be20 osd=4 3 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 4 8 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c11be20 osd=4 4 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 10 client.27354.objecter _send_command 8 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f883c11c550 con 0x7f883c120230 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.796+0000 7f883b59e700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6800/4175844757 conn(0x7f883c120230 0x7f883c120670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.797+0000 7f8823fff700 10 client.27354.objecter ms_handle_connect 0x7f883c120230 2024-02-23T00:38:00.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.797+0000 7f883b59e700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6800/4175844757 conn(0x7f883c120230 0x7f883c120670 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:00.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.797+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f883c11c550 con 0x7f883c120230 2024-02-23T00:38:00.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.797+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:38:00.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.797+0000 7f8823fff700 10 client.27354.objecter _finish_command 8 = system:0 2024-02-23T00:38:00.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.797+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c11be20 osd=4 4 2024-02-23T00:38:00.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.797+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 4 8 2024-02-23T00:38:00.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 10 client.27354.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:00.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 9 2024-02-23T00:38:00.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c11be20 osd=4 3 2024-02-23T00:38:00.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c11be20 osd=4 3 2024-02-23T00:38:00.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c11be20 osd=4 3 2024-02-23T00:38:00.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 9 2024-02-23T00:38:00.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c11be20 osd=4 3 2024-02-23T00:38:00.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 4 9 2024-02-23T00:38:00.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c11be20 osd=4 4 2024-02-23T00:38:00.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 10 client.27354.objecter _send_command 9 2024-02-23T00:38:00.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.812+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f883c121330 con 0x7f883c120230 2024-02-23T00:38:00.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f883c121330 con 0x7f883c120230 2024-02-23T00:38:00.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:00.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f8823fff700 10 client.27354.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c11be20 osd=4 4 2024-02-23T00:38:00.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 4 9 2024-02-23T00:38:00.817 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:38:00.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 10 client.27354.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:00.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 10 2024-02-23T00:38:00.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6804/2550013795 conn(0x7f883c123a10 0x7f883c125e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c121330 osd=5 3 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c121330 osd=5 3 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c121330 osd=5 3 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 10 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c121330 osd=5 3 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 5 10 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c121330 osd=5 4 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 10 client.27354.objecter _send_command 10 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f883c128b40 con 0x7f883c123a10 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.814+0000 7f883ad9d700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6804/2550013795 conn(0x7f883c123a10 0x7f883c125e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.815+0000 7f8823fff700 10 client.27354.objecter ms_handle_connect 0x7f883c123a10 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.815+0000 7f883ad9d700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6804/2550013795 conn(0x7f883c123a10 0x7f883c125e00 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.815+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f883c128b40 con 0x7f883c123a10 2024-02-23T00:38:00.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.815+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:38:00.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.815+0000 7f8823fff700 10 client.27354.objecter _finish_command 10 = system:0 2024-02-23T00:38:00.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.815+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c121330 osd=5 4 2024-02-23T00:38:00.819 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.815+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 5 10 2024-02-23T00:38:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 10 client.27354.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 11 2024-02-23T00:38:00.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c121330 osd=5 3 2024-02-23T00:38:00.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c121330 osd=5 3 2024-02-23T00:38:00.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c121330 osd=5 3 2024-02-23T00:38:00.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 11 2024-02-23T00:38:00.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c121330 osd=5 3 2024-02-23T00:38:00.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 5 11 2024-02-23T00:38:00.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c121330 osd=5 4 2024-02-23T00:38:00.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 10 client.27354.objecter _send_command 11 2024-02-23T00:38:00.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.830+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f883c128b40 con 0x7f883c123a10 2024-02-23T00:38:00.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.832+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f883c128b40 con 0x7f883c123a10 2024-02-23T00:38:00.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.832+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:00.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.832+0000 7f8823fff700 10 client.27354.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.832+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c121330 osd=5 4 2024-02-23T00:38:00.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.832+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 5 11 2024-02-23T00:38:00.835 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:38:00.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 10 client.27354.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:00.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 12 2024-02-23T00:38:00.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6808/2509408498 conn(0x7f883c127e40 0x7f883c1282c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c128b40 osd=6 3 2024-02-23T00:38:00.836 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c128b40 osd=6 3 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c128b40 osd=6 3 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 12 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c128b40 osd=6 3 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 6 12 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c128b40 osd=6 4 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 10 client.27354.objecter _send_command 12 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f883c128930 con 0x7f883c127e40 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f883a59c700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6808/2509408498 conn(0x7f883c127e40 0x7f883c1282c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f8823fff700 10 client.27354.objecter ms_handle_connect 0x7f883c127e40 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.833+0000 7f883a59c700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6808/2509408498 conn(0x7f883c127e40 0x7f883c1282c0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.834+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f883c128930 con 0x7f883c127e40 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.834+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.834+0000 7f8823fff700 10 client.27354.objecter _finish_command 12 = system:0 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.834+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c128b40 osd=6 4 2024-02-23T00:38:00.837 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.834+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 6 12 2024-02-23T00:38:00.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 10 client.27354.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:00.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 13 2024-02-23T00:38:00.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c128b40 osd=6 3 2024-02-23T00:38:00.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c128b40 osd=6 3 2024-02-23T00:38:00.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c128b40 osd=6 3 2024-02-23T00:38:00.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 13 2024-02-23T00:38:00.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c128b40 osd=6 3 2024-02-23T00:38:00.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 6 13 2024-02-23T00:38:00.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c128b40 osd=6 4 2024-02-23T00:38:00.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 10 client.27354.objecter _send_command 13 2024-02-23T00:38:00.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.848+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f883c128930 con 0x7f883c127e40 2024-02-23T00:38:00.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f883c128930 con 0x7f883c127e40 2024-02-23T00:38:00.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:00.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f8823fff700 10 client.27354.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.852 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c128b40 osd=6 4 2024-02-23T00:38:00.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 6 13 2024-02-23T00:38:00.853 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 10 client.27354.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 14 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6812/373787000 conn(0x7f883c12b2b0 0x7f883c12b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c128930 osd=7 3 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c128930 osd=7 3 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c128930 osd=7 3 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 14 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c128930 osd=7 3 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 7 14 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c128930 osd=7 4 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 10 client.27354.objecter _send_command 14 2024-02-23T00:38:00.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f883c12bd80 con 0x7f883c12b2b0 2024-02-23T00:38:00.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.850+0000 7f883b59e700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6812/373787000 conn(0x7f883c12b2b0 0x7f883c12b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:00.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.851+0000 7f8823fff700 10 client.27354.objecter ms_handle_connect 0x7f883c12b2b0 2024-02-23T00:38:00.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.851+0000 7f883b59e700 1 --2- 172.21.15.87:0/1063479808 >> v2:172.21.15.171:6812/373787000 conn(0x7f883c12b2b0 0x7f883c12b710 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:00.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.851+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f883c12bd80 con 0x7f883c12b2b0 2024-02-23T00:38:00.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.851+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:38:00.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.851+0000 7f8823fff700 10 client.27354.objecter _finish_command 14 = system:0 2024-02-23T00:38:00.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.851+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c128930 osd=7 4 2024-02-23T00:38:00.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.852+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 7 14 2024-02-23T00:38:00.858 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:00.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 10 client.27354.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign -1 15 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c128930 osd=7 3 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c128930 osd=7 3 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 20 client.27354.objecter _get_session s=0x7f883c128930 osd=7 3 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 15 client.27354.objecter _session_command_op_remove -1 15 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 20 client.27354.objecter get_session s=0x7f883c128930 osd=7 3 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 15 client.27354.objecter _session_command_op_assign 7 15 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 20 client.27354.objecter put_session s=0x7f883c128930 osd=7 4 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 10 client.27354.objecter _send_command 15 2024-02-23T00:38:00.868 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.865+0000 7f88411fb700 1 -- 172.21.15.87:0/1063479808 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f883c12bd80 con 0x7f883c12b2b0 2024-02-23T00:38:00.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.868+0000 7f8823fff700 1 -- 172.21.15.87:0/1063479808 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f883c12bd80 con 0x7f883c12b2b0 2024-02-23T00:38:00.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.868+0000 7f8823fff700 10 client.27354.objecter ms_dispatch 0x7f883c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:00.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.868+0000 7f8823fff700 10 client.27354.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.868+0000 7f8823fff700 20 client.27354.objecter put_session s=0x7f883c128930 osd=7 4 2024-02-23T00:38:00.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:00.868+0000 7f8823fff700 15 client.27354.objecter _session_command_op_remove 7 15 2024-02-23T00:38:00.870 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:38:00.870 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:38:00.870 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.871 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.871 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.871 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.871 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.871 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.871 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:38:00.888 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:00.909 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:00.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:01.013 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:01.039 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:01.059 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:01.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:01.164 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:38:01.164 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:38:01.165 INFO:tasks.thrashosds.thrasher:Growing pool default.rgw.log 2024-02-23T00:38:01.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:38:01.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:01.191 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:01.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:38:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:38:01.264 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:01.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:37:59.689062+0000 mgr.y (mgr.24437) 448 : cluster [DBG] pgmap v720: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 676 B/s rd, 270 B/s wr, 1 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:00.378007+0000 osd.6 (osd.6) 186 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:38:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:00.380933+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:38:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:00.734003+0000 osd.5 (osd.5) 181 : cluster [DBG] 63.a scrub starts 2024-02-23T00:38:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:00.735235+0000 osd.5 (osd.5) 182 : cluster [DBG] 63.a scrub ok 2024-02-23T00:38:01.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:00.740544+0000 mon.a (mon.0) 2731 : audit [DBG] from='client.? 172.21.15.87:0/1063479808' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:37:59.689062+0000 mgr.y (mgr.24437) 448 : cluster [DBG] pgmap v720: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 676 B/s rd, 270 B/s wr, 1 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:00.378007+0000 osd.6 (osd.6) 186 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:38:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:00.380933+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:38:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:00.734003+0000 osd.5 (osd.5) 181 : cluster [DBG] 63.a scrub starts 2024-02-23T00:38:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:00.735235+0000 osd.5 (osd.5) 182 : cluster [DBG] 63.a scrub ok 2024-02-23T00:38:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:00.740544+0000 mon.a (mon.0) 2731 : audit [DBG] from='client.? 172.21.15.87:0/1063479808' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:01.369 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:01.396 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:01.434 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:01.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:01.548 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:01.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:37:59.689062+0000 mgr.y (mgr.24437) 448 : cluster [DBG] pgmap v720: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 676 B/s rd, 270 B/s wr, 1 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:01.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:00.378007+0000 osd.6 (osd.6) 186 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:38:01.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:00.380933+0000 osd.6 (osd.6) 187 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:38:01.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:00.734003+0000 osd.5 (osd.5) 181 : cluster [DBG] 63.a scrub starts 2024-02-23T00:38:01.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:00.735235+0000 osd.5 (osd.5) 182 : cluster [DBG] 63.a scrub ok 2024-02-23T00:38:01.578 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:00.740544+0000 mon.a (mon.0) 2731 : audit [DBG] from='client.? 172.21.15.87:0/1063479808' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:01.580 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:01.602 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:01.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:01.706 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:01.735 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:01.754 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:01.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:01.858 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:01.886 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:01.906 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:01.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:01.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:02.009 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:02.037 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:02.056 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:02.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:02.159 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:02.186 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:02.208 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:02.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:02.309 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:02.344 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:38:02.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:38:02.371 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:02.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:02.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:00.976212+0000 osd.2 (osd.2) 181 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:38:02.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:00.977274+0000 osd.2 (osd.2) 182 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:38:02.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:00.989858+0000 mgr.y (mgr.24437) 449 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:01.358185+0000 osd.6 (osd.6) 188 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:38:02.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:01.361091+0000 osd.6 (osd.6) 189 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:38:02.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:02.486 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:02] "GET /metrics HTTP/1.1" 200 188724 "" "Prometheus/2.33.4" 2024-02-23T00:38:02.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:00.976212+0000 osd.2 (osd.2) 181 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:38:02.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:00.977274+0000 osd.2 (osd.2) 182 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:38:02.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:00.989858+0000 mgr.y (mgr.24437) 449 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:02.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:01.358185+0000 osd.6 (osd.6) 188 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:38:02.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:01.361091+0000 osd.6 (osd.6) 189 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:38:02.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:00.976212+0000 osd.2 (osd.2) 181 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:38:02.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:00.977274+0000 osd.2 (osd.2) 182 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:38:02.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:00.989858+0000 mgr.y (mgr.24437) 449 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:02.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:01.358185+0000 osd.6 (osd.6) 188 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:38:02.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:01.361091+0000 osd.6 (osd.6) 189 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:38:02.504 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:02.844 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:02.852 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:02.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:02.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:02.870 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-23T00:38:02.958 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:02.987 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:03.008 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:03.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:03.112 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:03.140 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:03.160 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:03.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:03.263 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:03.290 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:03.310 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:03.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:03.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:01.690180+0000 mgr.y (mgr.24437) 450 : cluster [DBG] pgmap v721: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:03.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:02.322361+0000 osd.6 (osd.6) 190 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:38:03.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:02.325250+0000 osd.6 (osd.6) 191 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:38:03.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:03.062615+0000 osd.4 (osd.4) 221 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:38:03.413 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:03.063480+0000 osd.4 (osd.4) 222 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:38:03.413 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:01.690180+0000 mgr.y (mgr.24437) 450 : cluster [DBG] pgmap v721: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:03.413 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:02.322361+0000 osd.6 (osd.6) 190 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:38:03.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:02.325250+0000 osd.6 (osd.6) 191 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:38:03.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:03.062615+0000 osd.4 (osd.4) 221 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:38:03.414 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:03.063480+0000 osd.4 (osd.4) 222 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:38:03.414 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:03.440 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:03.458 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:03.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:03.562 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:03.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:01.690180+0000 mgr.y (mgr.24437) 450 : cluster [DBG] pgmap v721: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:03.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:02.322361+0000 osd.6 (osd.6) 190 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:38:03.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:02.325250+0000 osd.6 (osd.6) 191 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:38:03.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:03.062615+0000 osd.4 (osd.4) 221 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:38:03.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:03.063480+0000 osd.4 (osd.4) 222 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:38:03.589 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:03.607 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:03.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:03.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:03.743 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:03.754 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:03.755 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:03.765 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:03.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:03.870 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:03.894 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:38:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:38:03.896 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:03.915 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:03.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:03.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:04.019 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:04.045 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:04.064 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:04.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:04.167 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:04.195 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:03.325335+0000 osd.6 (osd.6) 192 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:38:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:03.327123+0000 osd.6 (osd.6) 193 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:38:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:03.797786+0000 osd.5 (osd.5) 183 : cluster [DBG] 22.f scrub starts 2024-02-23T00:38:04.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:03.798821+0000 osd.5 (osd.5) 184 : cluster [DBG] 22.f scrub ok 2024-02-23T00:38:04.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:03.325335+0000 osd.6 (osd.6) 192 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:38:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:03.327123+0000 osd.6 (osd.6) 193 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:38:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:03.797786+0000 osd.5 (osd.5) 183 : cluster [DBG] 22.f scrub starts 2024-02-23T00:38:04.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:03.798821+0000 osd.5 (osd.5) 184 : cluster [DBG] 22.f scrub ok 2024-02-23T00:38:04.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:38:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:03.325335+0000 osd.6 (osd.6) 192 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:38:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:03.327123+0000 osd.6 (osd.6) 193 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:38:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:03.797786+0000 osd.5 (osd.5) 183 : cluster [DBG] 22.f scrub starts 2024-02-23T00:38:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:03.798821+0000 osd.5 (osd.5) 184 : cluster [DBG] 22.f scrub ok 2024-02-23T00:38:05.144 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:05.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:05.248 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:05.279 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:03.690787+0000 mgr.y (mgr.24437) 451 : cluster [DBG] pgmap v722: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 241 B/s wr, 2 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:04.816948+0000 osd.5 (osd.5) 185 : cluster [DBG] 63.17 scrub starts 2024-02-23T00:38:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:04.819469+0000 osd.5 (osd.5) 186 : cluster [DBG] 63.17 scrub ok 2024-02-23T00:38:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:05.092420+0000 osd.4 (osd.4) 223 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:38:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:05.095703+0000 osd.4 (osd.4) 224 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:38:05.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:03.690787+0000 mgr.y (mgr.24437) 451 : cluster [DBG] pgmap v722: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 241 B/s wr, 2 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:05.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:04.816948+0000 osd.5 (osd.5) 185 : cluster [DBG] 63.17 scrub starts 2024-02-23T00:38:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:04.819469+0000 osd.5 (osd.5) 186 : cluster [DBG] 63.17 scrub ok 2024-02-23T00:38:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:05.092420+0000 osd.4 (osd.4) 223 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:38:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:05.095703+0000 osd.4 (osd.4) 224 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:38:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:03.690787+0000 mgr.y (mgr.24437) 451 : cluster [DBG] pgmap v722: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 241 B/s wr, 2 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:04.816948+0000 osd.5 (osd.5) 185 : cluster [DBG] 63.17 scrub starts 2024-02-23T00:38:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:04.819469+0000 osd.5 (osd.5) 186 : cluster [DBG] 63.17 scrub ok 2024-02-23T00:38:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:05.092420+0000 osd.4 (osd.4) 223 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:38:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:05.095703+0000 osd.4 (osd.4) 224 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:38:05.727 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:05.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:05.832 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:05.864 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:05.885 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:05.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:05.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:05.992 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:06.023 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:06.043 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:06.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:06.147 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:06.180 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:06.201 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:06.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:06.307 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:06.332 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:06.118828+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:38:06.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:06.122233+0000 osd.4 (osd.4) 226 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:38:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:06.118828+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:38:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:06.122233+0000 osd.4 (osd.4) 226 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:38:06.336 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:06.118828+0000 osd.4 (osd.4) 225 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:38:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:06.122233+0000 osd.4 (osd.4) 226 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 -- 172.21.15.87:0/552117581 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600f6920 msgr2=0x7f0e601047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 --2- 172.21.15.87:0/552117581 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600f6920 0x7f0e601047f0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 rx=0x7f0e58009c60 tx=0x7f0e58004c50).stop 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 -- 172.21.15.87:0/552117581 shutdown_connections 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 --2- 172.21.15.87:0/552117581 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600f6920 0x7f0e601047f0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 --2- 172.21.15.87:0/552117581 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e600fdaf0 0x7f0e600fded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 --2- 172.21.15.87:0/552117581 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0e600f5fb0 0x7f0e600f6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 -- 172.21.15.87:0/552117581 >> 172.21.15.87:0/552117581 conn(0x7f0e6006feb0 msgr2=0x7f0e600f4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 -- 172.21.15.87:0/552117581 shutdown_connections 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 -- 172.21.15.87:0/552117581 wait complete. 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 librados: starting msgr at 2024-02-23T00:38:07.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.230+0000 7f0e6639f700 1 librados: starting objecter 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 Processor -- start 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 -- start start 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 librados: setting wanted keys 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 librados: calling monclient init 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e600f5fb0 0x7f0e6010dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0e600f6920 0x7f0e6010e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600fdaf0 0x7f0e60107f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e60108430 con 0x7f0e600fdaf0 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e601085a0 con 0x7f0e600f5fb0 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6639f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e60108760 con 0x7f0e600f6920 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e65b9e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600fdaf0 0x7f0e60107f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e65b9e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600fdaf0 0x7f0e60107f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55678/0 (socket says 172.21.15.87:55678) 2024-02-23T00:38:07.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e65b9e700 1 -- 172.21.15.87:0/2940740081 learned_addr learned my addr 172.21.15.87:0/2940740081 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e64b9c700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0e600f6920 0x7f0e6010e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6539d700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e600f5fb0 0x7f0e6010dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e65b9e700 1 -- 172.21.15.87:0/2940740081 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0e600f6920 msgr2=0x7f0e6010e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e65b9e700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0e600f6920 0x7f0e6010e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e65b9e700 1 -- 172.21.15.87:0/2940740081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e600f5fb0 msgr2=0x7f0e6010dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e65b9e700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e600f5fb0 0x7f0e6010dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e65b9e700 1 -- 172.21.15.87:0/2940740081 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c005040 con 0x7f0e600fdaf0 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e6539d700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e600f5fb0 0x7f0e6010dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e64b9c700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0e600f6920 0x7f0e6010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e65b9e700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600fdaf0 0x7f0e60107f40 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7f0e58004e10 tx=0x7f0e58004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e567fc700 10 client.?.objecter ms_handle_connect 0x7f0e600fdaf0 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.231+0000 7f0e567fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.232+0000 7f0e567fc700 1 -- 172.21.15.87:0/2940740081 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5800e040 con 0x7f0e600fdaf0 2024-02-23T00:38:07.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.232+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e580097c0 con 0x7f0e600fdaf0 2024-02-23T00:38:07.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.232+0000 7f0e567fc700 1 -- 172.21.15.87:0/2940740081 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e58018040 con 0x7f0e600fdaf0 2024-02-23T00:38:07.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.232+0000 7f0e567fc700 1 -- 172.21.15.87:0/2940740081 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e58013e80 con 0x7f0e600fdaf0 2024-02-23T00:38:07.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.232+0000 7f0e6639f700 10 client.27360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:07.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.232+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e60108c00 con 0x7f0e600fdaf0 2024-02-23T00:38:07.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.232+0000 7f0e6639f700 1 librados: init done 2024-02-23T00:38:07.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.233+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e60108df0 con 0x7f0e600fdaf0 2024-02-23T00:38:07.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.233+0000 7f0e567fc700 1 -- 172.21.15.87:0/2940740081 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0e58007bf0 con 0x7f0e600fdaf0 2024-02-23T00:38:07.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.234+0000 7f0e567fc700 1 --2- 172.21.15.87:0/2940740081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0e4c062c90 0x7f0e4c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:07.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.234+0000 7f0e567fc700 1 -- 172.21.15.87:0/2940740081 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(473..473 src has 173..473) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f0e5807e710 con 0x7f0e600fdaf0 2024-02-23T00:38:07.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.234+0000 7f0e567fc700 10 client.27360.objecter ms_dispatch 0x7f0e60001550 osd_map(473..473 src has 173..473) v4 2024-02-23T00:38:07.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.234+0000 7f0e567fc700 3 client.27360.objecter handle_osd_map got epochs [473,473] > 0 2024-02-23T00:38:07.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.234+0000 7f0e567fc700 3 client.27360.objecter handle_osd_map decoding full epoch 473 2024-02-23T00:38:07.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.234+0000 7f0e6539d700 1 --2- 172.21.15.87:0/2940740081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0e4c062c90 0x7f0e4c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:07.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.234+0000 7f0e567fc700 20 client.27360.objecter dump_active .. 0 homeless 2024-02-23T00:38:07.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.234+0000 7f0e567fc700 10 client.27360.objecter ms_handle_connect 0x7f0e4c062c90 2024-02-23T00:38:07.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.234+0000 7f0e6539d700 1 --2- 172.21.15.87:0/2940740081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0e4c062c90 0x7f0e4c065150 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7f0e50009250 tx=0x7f0e50009d60).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:07.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.238+0000 7f0e567fc700 1 -- 172.21.15.87:0/2940740081 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0e58051c70 con 0x7f0e600fdaf0 2024-02-23T00:38:07.286 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:07.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:07.391 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:07.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.403+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e60107020 con 0x7f0e4c062c90 2024-02-23T00:38:07.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.419+0000 7f0e567fc700 1 -- 172.21.15.87:0/2940740081 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+599653 (secure 0 0 0) 0x7f0e60107020 con 0x7f0e4c062c90 2024-02-23T00:38:07.426 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:07.428 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 10 librados: watch_flush enter 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 10 librados: watch_flush exit 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 20 client.27360.objecter shutdown clearing up homeless session... 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 10 client.27360.objecter successfully canceled tick 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0e4c062c90 msgr2=0x7f0e4c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 --2- 172.21.15.87:0/2940740081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0e4c062c90 0x7f0e4c065150 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 rx=0x7f0e50009250 tx=0x7f0e50009d60).stop 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600fdaf0 msgr2=0x7f0e60107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600fdaf0 0x7f0e60107f40 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 rx=0x7f0e58004e10 tx=0x7f0e58004c50).stop 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 shutdown_connections 2024-02-23T00:38:07.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 --2- 172.21.15.87:0/2940740081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0e4c062c90 0x7f0e4c065150 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:07.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0e600fdaf0 0x7f0e60107f40 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:07.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0e600f5fb0 0x7f0e6010dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:07.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 --2- 172.21.15.87:0/2940740081 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0e600f6920 0x7f0e6010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:07.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 >> 172.21.15.87:0/2940740081 conn(0x7f0e6006feb0 msgr2=0x7f0e600fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:07.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 shutdown_connections 2024-02-23T00:38:07.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 -- 172.21.15.87:0/2940740081 wait complete. 2024-02-23T00:38:07.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.423+0000 7f0e6639f700 1 librados: shutdown 2024-02-23T00:38:07.435 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:38:07.444 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:07.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:07.548 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:07.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:05.692055+0000 mgr.y (mgr.24437) 452 : cluster [DBG] pgmap v723: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:07.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:06.232612+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:38:07.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:06.235605+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:38:07.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:06.373122+0000 osd.6 (osd.6) 194 : cluster [DBG] 5.21 scrub starts 2024-02-23T00:38:07.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:06.376012+0000 osd.6 (osd.6) 195 : cluster [DBG] 5.21 scrub ok 2024-02-23T00:38:07.577 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:07.582 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:05.692055+0000 mgr.y (mgr.24437) 452 : cluster [DBG] pgmap v723: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:07.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:06.232612+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:38:07.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:06.235605+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:38:07.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:06.373122+0000 osd.6 (osd.6) 194 : cluster [DBG] 5.21 scrub starts 2024-02-23T00:38:07.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:06.376012+0000 osd.6 (osd.6) 195 : cluster [DBG] 5.21 scrub ok 2024-02-23T00:38:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:05.692055+0000 mgr.y (mgr.24437) 452 : cluster [DBG] pgmap v723: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:06.232612+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:38:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:06.235605+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:38:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:06.373122+0000 osd.6 (osd.6) 194 : cluster [DBG] 5.21 scrub starts 2024-02-23T00:38:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:06.376012+0000 osd.6 (osd.6) 195 : cluster [DBG] 5.21 scrub ok 2024-02-23T00:38:07.597 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:07.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:07.701 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:07.729 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:07.748 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:07.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:07.853 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:07.879 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:07.898 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:07.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:08.001 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:08.028 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:08.046 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:08.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:08.150 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:08.179 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:08.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.541+0000 7f3e43c52700 1 -- 172.21.15.87:0/2669952865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0f8420 msgr2=0x7f3e3c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.541+0000 7f3e43c52700 1 --2- 172.21.15.87:0/2669952865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0f8420 0x7f3e3c104a20 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 rx=0x7f3e2c004870 tx=0x7f3e2c009cb0).stop 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.541+0000 7f3e43c52700 1 -- 172.21.15.87:0/2669952865 shutdown_connections 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.541+0000 7f3e43c52700 1 --2- 172.21.15.87:0/2669952865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0f8420 0x7f3e3c104a20 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.541+0000 7f3e43c52700 1 --2- 172.21.15.87:0/2669952865 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3e3c0fdbb0 0x7f3e3c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.541+0000 7f3e43c52700 1 --2- 172.21.15.87:0/2669952865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e3c0fe510 0x7f3e3c0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.541+0000 7f3e43c52700 1 -- 172.21.15.87:0/2669952865 >> 172.21.15.87:0/2669952865 conn(0x7f3e3c0f3800 msgr2=0x7f3e3c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.541+0000 7f3e43c52700 1 -- 172.21.15.87:0/2669952865 shutdown_connections 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.541+0000 7f3e43c52700 1 -- 172.21.15.87:0/2669952865 wait complete. 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 librados: starting msgr at 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 librados: starting objecter 2024-02-23T00:38:08.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 Processor -- start 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 -- start start 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 librados: setting wanted keys 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 librados: calling monclient init 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3e3c0f8420 0x7f3e3c10ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e3c0fdbb0 0x7f3e3c10e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0fe510 0x7f3e3c107f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c108440 con 0x7f3e3c0fe510 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1085b0 con 0x7f3e3c0fdbb0 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e43c52700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c108770 con 0x7f3e3c0f8420 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e421ef700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0fe510 0x7f3e3c107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e419ee700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3e3c0f8420 0x7f3e3c10ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e419ee700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3e3c0f8420 0x7f3e3c10ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:46792/0 (socket says 172.21.15.87:46792) 2024-02-23T00:38:08.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e421ef700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0fe510 0x7f3e3c107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55694/0 (socket says 172.21.15.87:55694) 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e419ee700 1 -- 172.21.15.87:0/532197632 learned_addr learned my addr 172.21.15.87:0/532197632 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e411ed700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e3c0fdbb0 0x7f3e3c10e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e421ef700 1 -- 172.21.15.87:0/532197632 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3e3c0f8420 msgr2=0x7f3e3c10ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.542+0000 7f3e421ef700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3e3c0f8420 0x7f3e3c10ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e421ef700 1 -- 172.21.15.87:0/532197632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e3c0fdbb0 msgr2=0x7f3e3c10e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e421ef700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e3c0fdbb0 0x7f3e3c10e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e421ef700 1 -- 172.21.15.87:0/532197632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e340097c0 con 0x7f3e3c0fe510 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e419ee700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3e3c0f8420 0x7f3e3c10ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e411ed700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e3c0fdbb0 0x7f3e3c10e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e421ef700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0fe510 0x7f3e3c107f50 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7f3e2c004870 tx=0x7f3e2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e3affd700 10 client.?.objecter ms_handle_connect 0x7f3e3c0fe510 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e3affd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e3affd700 1 -- 172.21.15.87:0/532197632 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c00e030 con 0x7f3e3c0fe510 2024-02-23T00:38:08.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e3affd700 1 -- 172.21.15.87:0/532197632 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e2c009710 con 0x7f3e3c0fe510 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c005040 con 0x7f3e3c0fe510 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e3affd700 1 -- 172.21.15.87:0/532197632 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c019070 con 0x7f3e3c0fe510 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e43c52700 10 client.27366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c108c60 con 0x7f3e3c0fe510 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.543+0000 7f3e43c52700 1 librados: init done 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.544+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c05ee80 con 0x7f3e3c0fe510 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.544+0000 7f3e3affd700 1 -- 172.21.15.87:0/532197632 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3e2c01e420 con 0x7f3e3c0fe510 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.544+0000 7f3e3affd700 1 --2- 172.21.15.87:0/532197632 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e24062c90 0x7f3e24065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.545+0000 7f3e3affd700 1 -- 172.21.15.87:0/532197632 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(473..473 src has 173..473) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f3e2c07e530 con 0x7f3e3c0fe510 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.545+0000 7f3e419ee700 1 --2- 172.21.15.87:0/532197632 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e24062c90 0x7f3e24065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.545+0000 7f3e3affd700 10 client.27366.objecter ms_dispatch 0x7f3e3c001550 osd_map(473..473 src has 173..473) v4 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.545+0000 7f3e3affd700 3 client.27366.objecter handle_osd_map got epochs [473,473] > 0 2024-02-23T00:38:08.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.545+0000 7f3e3affd700 3 client.27366.objecter handle_osd_map decoding full epoch 473 2024-02-23T00:38:08.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.545+0000 7f3e3affd700 20 client.27366.objecter dump_active .. 0 homeless 2024-02-23T00:38:08.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.545+0000 7f3e3affd700 10 client.27366.objecter ms_handle_connect 0x7f3e24062c90 2024-02-23T00:38:08.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.545+0000 7f3e419ee700 1 --2- 172.21.15.87:0/532197632 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e24062c90 0x7f3e24065150 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f3e34001940 tx=0x7f3e34004c50).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:08.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.549+0000 7f3e3affd700 1 -- 172.21.15.87:0/532197632 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3e2c051a80 con 0x7f3e3c0fe510 2024-02-23T00:38:08.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:07.726+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f3e3c109500 con 0x7f3e3c0fe510 2024-02-23T00:38:08.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.278+0000 7f3e3affd700 1 -- 172.21.15.87:0/532197632 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v474) v1 ==== 101+0+0 (secure 0 0 0) 0x7f3e2c051670 con 0x7f3e3c0fe510 2024-02-23T00:38:08.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.280+0000 7f3e43c52700 10 librados: watch_flush enter 2024-02-23T00:38:08.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.280+0000 7f3e43c52700 10 librados: watch_flush exit 2024-02-23T00:38:08.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.280+0000 7f3e43c52700 20 client.27366.objecter shutdown clearing up homeless session... 2024-02-23T00:38:08.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.280+0000 7f3e43c52700 10 client.27366.objecter successfully canceled tick 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e24062c90 msgr2=0x7f3e24065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 --2- 172.21.15.87:0/532197632 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e24062c90 0x7f3e24065150 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 rx=0x7f3e34001940 tx=0x7f3e34004c50).stop 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0fe510 msgr2=0x7f3e3c107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0fe510 0x7f3e3c107f50 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 rx=0x7f3e2c004870 tx=0x7f3e2c00b040).stop 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 shutdown_connections 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 --2- 172.21.15.87:0/532197632 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e24062c90 0x7f3e24065150 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3e3c0fe510 0x7f3e3c107f50 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3e3c0f8420 0x7f3e3c10ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 --2- 172.21.15.87:0/532197632 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3e3c0fdbb0 0x7f3e3c10e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 >> 172.21.15.87:0/532197632 conn(0x7f3e3c0f3800 msgr2=0x7f3e3c0fbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 shutdown_connections 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 -- 172.21.15.87:0/532197632 wait complete. 2024-02-23T00:38:08.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:08.281+0000 7f3e43c52700 1 librados: shutdown 2024-02-23T00:38:08.284 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:38:08.285 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:08.302 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:08.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:08.406 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:08.433 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:08.451 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:08.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:08.555 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:08.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:07.223473+0000 osd.7 (osd.7) 237 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:38:08.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:07.226375+0000 osd.7 (osd.7) 238 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:38:08.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:07.392250+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:38:08.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:07.395251+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:38:08.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:07.726790+0000 mon.a (mon.0) 2732 : audit [INF] from='client.? 172.21.15.87:0/532197632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:38:08.579 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:07.744242+0000 osd.5 (osd.5) 187 : cluster [DBG] 63.1b scrub starts 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:07.746085+0000 osd.5 (osd.5) 188 : cluster [DBG] 63.1b scrub ok 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:08.067732+0000 osd.4 (osd.4) 227 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:08.068555+0000 osd.4 (osd.4) 228 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:07.223473+0000 osd.7 (osd.7) 237 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:07.226375+0000 osd.7 (osd.7) 238 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:07.392250+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:07.395251+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:07.726790+0000 mon.a (mon.0) 2732 : audit [INF] from='client.? 172.21.15.87:0/532197632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:07.744242+0000 osd.5 (osd.5) 187 : cluster [DBG] 63.1b scrub starts 2024-02-23T00:38:08.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:07.746085+0000 osd.5 (osd.5) 188 : cluster [DBG] 63.1b scrub ok 2024-02-23T00:38:08.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:08.067732+0000 osd.4 (osd.4) 227 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:38:08.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:08.068555+0000 osd.4 (osd.4) 228 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:38:08.581 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:07.223473+0000 osd.7 (osd.7) 237 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:38:08.581 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:07.226375+0000 osd.7 (osd.7) 238 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:38:08.581 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:07.392250+0000 osd.6 (osd.6) 196 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:38:08.582 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:07.395251+0000 osd.6 (osd.6) 197 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:38:08.582 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:07.726790+0000 mon.a (mon.0) 2732 : audit [INF] from='client.? 172.21.15.87:0/532197632' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:38:08.582 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:07.744242+0000 osd.5 (osd.5) 187 : cluster [DBG] 63.1b scrub starts 2024-02-23T00:38:08.582 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:07.746085+0000 osd.5 (osd.5) 188 : cluster [DBG] 63.1b scrub ok 2024-02-23T00:38:08.582 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:08.067732+0000 osd.4 (osd.4) 227 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:38:08.582 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:08.068555+0000 osd.4 (osd.4) 228 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:38:08.582 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:08.600 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:08.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:08.704 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:08.730 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:08.751 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:08.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:08.855 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:08.881 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:08.900 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:08.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:09.003 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:09.044 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:09.069 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:09.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:09.177 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:09.204 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:09.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:07.403784+0000 mgr.y (mgr.24437) 453 : audit [DBG] from='client.27360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:38:09.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:07.692981+0000 mgr.y (mgr.24437) 454 : cluster [DBG] pgmap v724: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 968 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:09.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:08.265088+0000 mon.a (mon.0) 2733 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:38:09.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:08.278461+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? 172.21.15.87:0/532197632' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:38:09.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:08.278522+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e474: 8 total, 7 up, 7 in 2024-02-23T00:38:09.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:07.403784+0000 mgr.y (mgr.24437) 453 : audit [DBG] from='client.27360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:38:09.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:07.692981+0000 mgr.y (mgr.24437) 454 : cluster [DBG] pgmap v724: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 968 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:09.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:08.265088+0000 mon.a (mon.0) 2733 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:38:09.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:08.278461+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? 172.21.15.87:0/532197632' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:38:09.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:08.278522+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e474: 8 total, 7 up, 7 in 2024-02-23T00:38:09.559 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:09.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:09.664 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:09.690 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:07.403784+0000 mgr.y (mgr.24437) 453 : audit [DBG] from='client.27360 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:38:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:07.692981+0000 mgr.y (mgr.24437) 454 : cluster [DBG] pgmap v724: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 968 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:08.265088+0000 mon.a (mon.0) 2733 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:38:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:08.278461+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? 172.21.15.87:0/532197632' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:38:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:08.278522+0000 mon.a (mon.0) 2735 : cluster [DBG] osdmap e474: 8 total, 7 up, 7 in 2024-02-23T00:38:10.442 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:10.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:10.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:10.546 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:10.579 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:10.600 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:10.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:10.704 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:10.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:09.205790+0000 osd.7 (osd.7) 239 : cluster [DBG] 6.1c deep-scrub starts 2024-02-23T00:38:10.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:09.207224+0000 osd.7 (osd.7) 240 : cluster [DBG] 6.1c deep-scrub ok 2024-02-23T00:38:10.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:09.444431+0000 osd.6 (osd.6) 198 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:38:10.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:09.447322+0000 osd.6 (osd.6) 199 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:38:10.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:09.651996+0000 mon.a (mon.0) 2736 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:10.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:09.693630+0000 mgr.y (mgr.24437) 455 : cluster [DBG] pgmap v726: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:09.720999+0000 osd.5 (osd.5) 189 : cluster [DBG] 63.1e scrub starts 2024-02-23T00:38:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:09.722676+0000 osd.5 (osd.5) 190 : cluster [DBG] 63.1e scrub ok 2024-02-23T00:38:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:10.170929+0000 osd.7 (osd.7) 241 : cluster [DBG] 6.b scrub starts 2024-02-23T00:38:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:10.172053+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.b scrub ok 2024-02-23T00:38:11.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:09.205790+0000 osd.7 (osd.7) 239 : cluster [DBG] 6.1c deep-scrub starts 2024-02-23T00:38:11.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:09.207224+0000 osd.7 (osd.7) 240 : cluster [DBG] 6.1c deep-scrub ok 2024-02-23T00:38:11.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:09.444431+0000 osd.6 (osd.6) 198 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:38:11.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:09.447322+0000 osd.6 (osd.6) 199 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:38:11.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:09.651996+0000 mon.a (mon.0) 2736 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:11.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:09.693630+0000 mgr.y (mgr.24437) 455 : cluster [DBG] pgmap v726: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:11.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:09.720999+0000 osd.5 (osd.5) 189 : cluster [DBG] 63.1e scrub starts 2024-02-23T00:38:11.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:09.722676+0000 osd.5 (osd.5) 190 : cluster [DBG] 63.1e scrub ok 2024-02-23T00:38:11.033 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:10.170929+0000 osd.7 (osd.7) 241 : cluster [DBG] 6.b scrub starts 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:10.172053+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.b scrub ok 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:09.205790+0000 osd.7 (osd.7) 239 : cluster [DBG] 6.1c deep-scrub starts 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:09.207224+0000 osd.7 (osd.7) 240 : cluster [DBG] 6.1c deep-scrub ok 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:09.444431+0000 osd.6 (osd.6) 198 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:09.447322+0000 osd.6 (osd.6) 199 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:09.651996+0000 mon.a (mon.0) 2736 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:09.693630+0000 mgr.y (mgr.24437) 455 : cluster [DBG] pgmap v726: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:09.720999+0000 osd.5 (osd.5) 189 : cluster [DBG] 63.1e scrub starts 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:09.722676+0000 osd.5 (osd.5) 190 : cluster [DBG] 63.1e scrub ok 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:10.170929+0000 osd.7 (osd.7) 241 : cluster [DBG] 6.b scrub starts 2024-02-23T00:38:11.034 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:10.172053+0000 osd.7 (osd.7) 242 : cluster [DBG] 6.b scrub ok 2024-02-23T00:38:11.052 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":723,"stamp":"2024-02-23T00:38:05.690999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007449,"num_objects":402,"num_object_clones":11,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":1958,"num_read_kb":1580,"num_write":1543,"num_write_kb":4576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":4003469,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2613,"ondisk_log_size":2613,"up":619,"acting":619,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":713,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2191768,"kb_used_data":49760,"kb_used_omap":0,"kb_used_meta":2141952,"kb_avail":654024296,"statfs":{"total":671965249536,"available":669720879104,"internally_reserved":0,"allocated":50954240,"data_stored":32734485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2193358848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004427"},"pg_stats":[{"pgid":"219.1f","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280973+0000","last_change":"2024-02-23T00:37:50.222327+0000","last_active":"2024-02-23T00:37:55.280973+0000","last_peered":"2024-02-23T00:37:55.280973+0000","last_clean":"2024-02-23T00:37:55.280973+0000","last_became_active":"2024-02-23T00:37:50.222043+0000","last_became_peered":"2024-02-23T00:37:50.222043+0000","last_unstale":"2024-02-23T00:37:55.280973+0000","last_undegraded":"2024-02-23T00:37:55.280973+0000","last_fullsized":"2024-02-23T00:37:55.280973+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.1e","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280942+0000","last_change":"2024-02-23T00:37:50.228634+0000","last_active":"2024-02-23T00:37:55.280942+0000","last_peered":"2024-02-23T00:37:55.280942+0000","last_clean":"2024-02-23T00:37:55.280942+0000","last_became_active":"2024-02-23T00:37:50.228343+0000","last_became_peered":"2024-02-23T00:37:50.228343+0000","last_unstale":"2024-02-23T00:37:55.280942+0000","last_undegraded":"2024-02-23T00:37:55.280942+0000","last_fullsized":"2024-02-23T00:37:55.280942+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.1d","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352161+0000","last_change":"2024-02-23T00:37:50.229896+0000","last_active":"2024-02-23T00:37:55.352161+0000","last_peered":"2024-02-23T00:37:55.352161+0000","last_clean":"2024-02-23T00:37:55.352161+0000","last_became_active":"2024-02-23T00:37:50.229628+0000","last_became_peered":"2024-02-23T00:37:50.229628+0000","last_unstale":"2024-02-23T00:37:55.352161+0000","last_undegraded":"2024-02-23T00:37:55.352161+0000","last_fullsized":"2024-02-23T00:37:55.352161+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.1c","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280309+0000","last_change":"2024-02-23T00:37:50.222341+0000","last_active":"2024-02-23T00:37:55.280309+0000","last_peered":"2024-02-23T00:37:55.280309+0000","last_clean":"2024-02-23T00:37:55.280309+0000","last_became_active":"2024-02-23T00:37:50.222057+0000","last_became_peered":"2024-02-23T00:37:50.222057+0000","last_unstale":"2024-02-23T00:37:55.280309+0000","last_undegraded":"2024-02-23T00:37:55.280309+0000","last_fullsized":"2024-02-23T00:37:55.280309+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.1b","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.352438+0000","last_change":"2024-02-23T00:37:50.211753+0000","last_active":"2024-02-23T00:37:55.352438+0000","last_peered":"2024-02-23T00:37:55.352438+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.211753+0000","last_became_peered":"2024-02-23T00:37:50.211753+0000","last_unstale":"2024-02-23T00:37:55.352438+0000","last_undegraded":"2024-02-23T00:37:55.352438+0000","last_fullsized":"2024-02-23T00:37:50.191671+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.1a","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.075277+0000","last_change":"2024-02-23T00:37:50.233261+0000","last_active":"2024-02-23T00:37:56.075277+0000","last_peered":"2024-02-23T00:37:56.075277+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.233261+0000","last_became_peered":"2024-02-23T00:37:50.233261+0000","last_unstale":"2024-02-23T00:37:56.075277+0000","last_undegraded":"2024-02-23T00:37:56.075277+0000","last_fullsized":"2024-02-23T00:37:50.206815+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.19","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220805+0000","last_change":"2024-02-23T00:37:50.217298+0000","last_active":"2024-02-23T00:37:55.220805+0000","last_peered":"2024-02-23T00:37:55.220805+0000","last_clean":"2024-02-23T00:37:55.220805+0000","last_became_active":"2024-02-23T00:37:50.217021+0000","last_became_peered":"2024-02-23T00:37:50.217021+0000","last_unstale":"2024-02-23T00:37:55.220805+0000","last_undegraded":"2024-02-23T00:37:55.220805+0000","last_fullsized":"2024-02-23T00:37:55.220805+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"219.18","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220736+0000","last_change":"2024-02-23T00:37:50.218018+0000","last_active":"2024-02-23T00:37:55.220736+0000","last_peered":"2024-02-23T00:37:55.220736+0000","last_clean":"2024-02-23T00:37:55.220736+0000","last_became_active":"2024-02-23T00:37:50.217753+0000","last_became_peered":"2024-02-23T00:37:50.217753+0000","last_unstale":"2024-02-23T00:37:55.220736+0000","last_undegraded":"2024-02-23T00:37:55.220736+0000","last_fullsized":"2024-02-23T00:37:55.220736+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"219.17","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.223034+0000","last_change":"2024-02-23T00:37:50.228870+0000","last_active":"2024-02-23T00:37:55.223034+0000","last_peered":"2024-02-23T00:37:55.223034+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.228870+0000","last_became_peered":"2024-02-23T00:37:50.228870+0000","last_unstale":"2024-02-23T00:37:55.223034+0000","last_undegraded":"2024-02-23T00:37:55.223034+0000","last_fullsized":"2024-02-23T00:37:50.203917+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"219.16","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.219998+0000","last_change":"2024-02-23T00:37:50.225076+0000","last_active":"2024-02-23T00:37:55.219998+0000","last_peered":"2024-02-23T00:37:55.219998+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.225076+0000","last_became_peered":"2024-02-23T00:37:50.225076+0000","last_unstale":"2024-02-23T00:37:55.219998+0000","last_undegraded":"2024-02-23T00:37:55.219998+0000","last_fullsized":"2024-02-23T00:37:50.189931+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"219.15","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.352426+0000","last_change":"2024-02-23T00:37:50.217255+0000","last_active":"2024-02-23T00:37:55.352426+0000","last_peered":"2024-02-23T00:37:55.352426+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.217255+0000","last_became_peered":"2024-02-23T00:37:50.217255+0000","last_unstale":"2024-02-23T00:37:55.352426+0000","last_undegraded":"2024-02-23T00:37:55.352426+0000","last_fullsized":"2024-02-23T00:37:50.193198+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.14","version":"471'8","reported_seq":24,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077614+0000","last_change":"2024-02-23T00:37:53.212782+0000","last_active":"2024-02-23T00:37:56.077614+0000","last_peered":"2024-02-23T00:37:56.077614+0000","last_clean":"2024-02-23T00:37:56.077614+0000","last_became_active":"2024-02-23T00:37:50.225703+0000","last_became_peered":"2024-02-23T00:37:50.225703+0000","last_unstale":"2024-02-23T00:37:56.077614+0000","last_undegraded":"2024-02-23T00:37:56.077614+0000","last_fullsized":"2024-02-23T00:37:56.077614+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'8","last_scrub_stamp":"2024-02-23T00:37:53.212664+0000","last_deep_scrub":"471'8","last_deep_scrub_stamp":"2024-02-23T00:37:53.212664+0000","last_clean_scrub_stamp":"2024-02-23T00:37:53.212664+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.13","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.221894+0000","last_change":"2024-02-23T00:37:50.228320+0000","last_active":"2024-02-23T00:37:55.221894+0000","last_peered":"2024-02-23T00:37:55.221894+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.228320+0000","last_became_peered":"2024-02-23T00:37:50.228320+0000","last_unstale":"2024-02-23T00:37:55.221894+0000","last_undegraded":"2024-02-23T00:37:55.221894+0000","last_fullsized":"2024-02-23T00:37:50.203348+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"219.12","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075411+0000","last_change":"2024-02-23T00:37:50.234625+0000","last_active":"2024-02-23T00:37:56.075411+0000","last_peered":"2024-02-23T00:37:56.075411+0000","last_clean":"2024-02-23T00:37:56.075411+0000","last_became_active":"2024-02-23T00:37:50.234050+0000","last_became_peered":"2024-02-23T00:37:50.234050+0000","last_unstale":"2024-02-23T00:37:56.075411+0000","last_undegraded":"2024-02-23T00:37:56.075411+0000","last_fullsized":"2024-02-23T00:37:56.075411+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.11","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.074625+0000","last_change":"2024-02-23T00:37:50.236742+0000","last_active":"2024-02-23T00:37:56.074625+0000","last_peered":"2024-02-23T00:37:56.074625+0000","last_clean":"2024-02-23T00:37:56.074625+0000","last_became_active":"2024-02-23T00:37:50.236351+0000","last_became_peered":"2024-02-23T00:37:50.236351+0000","last_unstale":"2024-02-23T00:37:56.074625+0000","last_undegraded":"2024-02-23T00:37:56.074625+0000","last_fullsized":"2024-02-23T00:37:56.074625+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.10","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.220799+0000","last_change":"2024-02-23T00:37:50.211658+0000","last_active":"2024-02-23T00:37:55.220799+0000","last_peered":"2024-02-23T00:37:55.220799+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.211658+0000","last_became_peered":"2024-02-23T00:37:50.211658+0000","last_unstale":"2024-02-23T00:37:55.220799+0000","last_undegraded":"2024-02-23T00:37:55.220799+0000","last_fullsized":"2024-02-23T00:37:50.198001+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"219.f","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.076631+0000","last_change":"2024-02-23T00:37:50.262049+0000","last_active":"2024-02-23T00:37:56.076631+0000","last_peered":"2024-02-23T00:37:56.076631+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.262049+0000","last_became_peered":"2024-02-23T00:37:50.262049+0000","last_unstale":"2024-02-23T00:37:56.076631+0000","last_undegraded":"2024-02-23T00:37:56.076631+0000","last_fullsized":"2024-02-23T00:37:50.248840+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.e","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.352550+0000","last_change":"2024-02-23T00:37:50.229940+0000","last_active":"2024-02-23T00:37:55.352550+0000","last_peered":"2024-02-23T00:37:55.352550+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.229940+0000","last_became_peered":"2024-02-23T00:37:50.229940+0000","last_unstale":"2024-02-23T00:37:55.352550+0000","last_undegraded":"2024-02-23T00:37:55.352550+0000","last_fullsized":"2024-02-23T00:37:50.194454+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.d","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075018+0000","last_change":"2024-02-23T00:37:50.225121+0000","last_active":"2024-02-23T00:37:56.075018+0000","last_peered":"2024-02-23T00:37:56.075018+0000","last_clean":"2024-02-23T00:37:56.075018+0000","last_became_active":"2024-02-23T00:37:50.224846+0000","last_became_peered":"2024-02-23T00:37:50.224846+0000","last_unstale":"2024-02-23T00:37:56.075018+0000","last_undegraded":"2024-02-23T00:37:56.075018+0000","last_fullsized":"2024-02-23T00:37:56.075018+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"219.c","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077702+0000","last_change":"2024-02-23T00:37:50.226072+0000","last_active":"2024-02-23T00:37:56.077702+0000","last_peered":"2024-02-23T00:37:56.077702+0000","last_clean":"2024-02-23T00:37:56.077702+0000","last_became_active":"2024-02-23T00:37:50.225753+0000","last_became_peered":"2024-02-23T00:37:50.225753+0000","last_unstale":"2024-02-23T00:37:56.077702+0000","last_undegraded":"2024-02-23T00:37:56.077702+0000","last_fullsized":"2024-02-23T00:37:56.077702+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.b","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.351814+0000","last_change":"2024-02-23T00:37:50.229921+0000","last_active":"2024-02-23T00:37:55.351814+0000","last_peered":"2024-02-23T00:37:55.351814+0000","last_clean":"2024-02-23T00:37:55.351814+0000","last_became_active":"2024-02-23T00:37:50.229688+0000","last_became_peered":"2024-02-23T00:37:50.229688+0000","last_unstale":"2024-02-23T00:37:55.351814+0000","last_undegraded":"2024-02-23T00:37:55.351814+0000","last_fullsized":"2024-02-23T00:37:55.351814+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.a","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.221223+0000","last_change":"2024-02-23T00:37:50.217160+0000","last_active":"2024-02-23T00:37:55.221223+0000","last_peered":"2024-02-23T00:37:55.221223+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.217160+0000","last_became_peered":"2024-02-23T00:37:50.217160+0000","last_unstale":"2024-02-23T00:37:55.221223+0000","last_undegraded":"2024-02-23T00:37:55.221223+0000","last_fullsized":"2024-02-23T00:37:50.195283+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"219.9","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.351609+0000","last_change":"2024-02-23T00:37:50.216009+0000","last_active":"2024-02-23T00:37:55.351609+0000","last_peered":"2024-02-23T00:37:55.351609+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.216009+0000","last_became_peered":"2024-02-23T00:37:50.216009+0000","last_unstale":"2024-02-23T00:37:55.351609+0000","last_undegraded":"2024-02-23T00:37:55.351609+0000","last_fullsized":"2024-02-23T00:37:50.191540+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.8","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.218854+0000","last_change":"2024-02-23T00:37:50.225107+0000","last_active":"2024-02-23T00:37:55.218854+0000","last_peered":"2024-02-23T00:37:55.218854+0000","last_clean":"2024-02-23T00:37:55.218854+0000","last_became_active":"2024-02-23T00:37:50.224399+0000","last_became_peered":"2024-02-23T00:37:50.224399+0000","last_unstale":"2024-02-23T00:37:55.218854+0000","last_undegraded":"2024-02-23T00:37:55.218854+0000","last_fullsized":"2024-02-23T00:37:55.218854+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"219.7","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.281090+0000","last_change":"2024-02-23T00:37:50.216445+0000","last_active":"2024-02-23T00:37:55.281090+0000","last_peered":"2024-02-23T00:37:55.281090+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.216445+0000","last_became_peered":"2024-02-23T00:37:50.216445+0000","last_unstale":"2024-02-23T00:37:55.281090+0000","last_undegraded":"2024-02-23T00:37:55.281090+0000","last_fullsized":"2024-02-23T00:37:50.206334+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"219.6","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.222655+0000","last_change":"2024-02-23T00:37:50.237107+0000","last_active":"2024-02-23T00:37:55.222655+0000","last_peered":"2024-02-23T00:37:55.222655+0000","last_clean":"2024-02-23T00:37:55.222655+0000","last_became_active":"2024-02-23T00:37:50.236690+0000","last_became_peered":"2024-02-23T00:37:50.236690+0000","last_unstale":"2024-02-23T00:37:55.222655+0000","last_undegraded":"2024-02-23T00:37:55.222655+0000","last_fullsized":"2024-02-23T00:37:55.222655+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"219.5","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.077273+0000","last_change":"2024-02-23T00:37:50.235098+0000","last_active":"2024-02-23T00:37:56.077273+0000","last_peered":"2024-02-23T00:37:56.077273+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.235098+0000","last_became_peered":"2024-02-23T00:37:50.235098+0000","last_unstale":"2024-02-23T00:37:56.077273+0000","last_undegraded":"2024-02-23T00:37:56.077273+0000","last_fullsized":"2024-02-23T00:37:50.212767+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.4","version":"0'0","reported_seq":13,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077770+0000","last_change":"2024-02-23T00:37:50.234046+0000","last_active":"2024-02-23T00:37:56.077770+0000","last_peered":"2024-02-23T00:37:56.077770+0000","last_clean":"2024-02-23T00:37:56.077770+0000","last_became_active":"2024-02-23T00:37:50.233809+0000","last_became_peered":"2024-02-23T00:37:50.233809+0000","last_unstale":"2024-02-23T00:37:56.077770+0000","last_undegraded":"2024-02-23T00:37:56.077770+0000","last_fullsized":"2024-02-23T00:37:56.077770+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":469,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.3","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.351927+0000","last_change":"2024-02-23T00:37:50.212069+0000","last_active":"2024-02-23T00:37:55.351927+0000","last_peered":"2024-02-23T00:37:55.351927+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.212069+0000","last_became_peered":"2024-02-23T00:37:50.212069+0000","last_unstale":"2024-02-23T00:37:55.351927+0000","last_undegraded":"2024-02-23T00:37:55.351927+0000","last_fullsized":"2024-02-23T00:37:50.193062+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"219.2","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.075512+0000","last_change":"2024-02-23T00:37:50.234370+0000","last_active":"2024-02-23T00:37:56.075512+0000","last_peered":"2024-02-23T00:37:56.075512+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.234370+0000","last_became_peered":"2024-02-23T00:37:50.234370+0000","last_unstale":"2024-02-23T00:37:56.075512+0000","last_undegraded":"2024-02-23T00:37:56.075512+0000","last_fullsized":"2024-02-23T00:37:50.207454+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.1","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.074499+0000","last_change":"2024-02-23T00:37:50.235998+0000","last_active":"2024-02-23T00:37:56.074499+0000","last_peered":"2024-02-23T00:37:56.074499+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.235998+0000","last_became_peered":"2024-02-23T00:37:50.235998+0000","last_unstale":"2024-02-23T00:37:56.074499+0000","last_undegraded":"2024-02-23T00:37:56.074499+0000","last_fullsized":"2024-02-23T00:37:50.206609+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"219.0","version":"0'0","reported_seq":10,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.279543+0000","last_change":"2024-02-23T00:37:50.228438+0000","last_active":"2024-02-23T00:37:55.279543+0000","last_peered":"2024-02-23T00:37:55.279543+0000","last_clean":"2024-02-23T00:37:49.174016+0000","last_became_active":"2024-02-23T00:37:50.228438+0000","last_became_peered":"2024-02-23T00:37:50.228438+0000","last_unstale":"2024-02-23T00:37:55.279543+0000","last_undegraded":"2024-02-23T00:37:55.279543+0000","last_fullsized":"2024-02-23T00:37:50.205562+0000","mapping_epoch":468,"log_start":"0'0","ondisk_log_start":"0'0","created":468,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:49.174016+0000","last_clean_scrub_stamp":"2024-02-23T00:37:49.174016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.0","version":"458'80","reported_seq":229,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:56.074382+0000","last_change":"2024-02-23T00:36:37.106755+0000","last_active":"2024-02-23T00:37:56.074382+0000","last_peered":"2024-02-23T00:37:56.074382+0000","last_clean":"2024-02-23T00:36:33.365214+0000","last_became_active":"2024-02-23T00:36:37.106755+0000","last_became_peered":"2024-02-23T00:36:37.106755+0000","last_unstale":"2024-02-23T00:37:56.074382+0000","last_undegraded":"2024-02-23T00:36:37.100498+0000","last_fullsized":"2024-02-23T00:36:37.100377+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'68","last_scrub_stamp":"2024-02-23T00:34:57.645204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.645204+0000","log_size":80,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1","version":"340'86","reported_seq":509,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.219083+0000","last_change":"2024-02-23T00:36:37.109838+0000","last_active":"2024-02-23T00:37:55.219083+0000","last_peered":"2024-02-23T00:37:55.219083+0000","last_clean":"2024-02-23T00:36:33.191340+0000","last_became_active":"2024-02-23T00:36:37.109838+0000","last_became_peered":"2024-02-23T00:36:37.109838+0000","last_unstale":"2024-02-23T00:37:55.219083+0000","last_undegraded":"2024-02-23T00:36:37.093708+0000","last_fullsized":"2024-02-23T00:36:37.093560+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'85","last_scrub_stamp":"2024-02-23T00:34:16.724630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:16.724630+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.2","version":"372'21","reported_seq":456,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.351805+0000","last_change":"2024-02-23T00:36:37.106211+0000","last_active":"2024-02-23T00:37:55.351805+0000","last_peered":"2024-02-23T00:37:55.351805+0000","last_clean":"2024-02-23T00:36:33.365672+0000","last_became_active":"2024-02-23T00:36:37.106211+0000","last_became_peered":"2024-02-23T00:36:37.106211+0000","last_unstale":"2024-02-23T00:37:55.351805+0000","last_undegraded":"2024-02-23T00:36:37.098859+0000","last_fullsized":"2024-02-23T00:36:37.098739+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'20","last_scrub_stamp":"2024-02-23T00:34:23.049395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.049395+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.3","version":"451'44","reported_seq":493,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.219641+0000","last_change":"2024-02-23T00:37:05.079232+0000","last_active":"2024-02-23T00:37:55.219641+0000","last_peered":"2024-02-23T00:37:55.219641+0000","last_clean":"2024-02-23T00:37:55.219641+0000","last_became_active":"2024-02-23T00:36:17.093628+0000","last_became_peered":"2024-02-23T00:36:17.093628+0000","last_unstale":"2024-02-23T00:37:55.219641+0000","last_undegraded":"2024-02-23T00:37:55.219641+0000","last_fullsized":"2024-02-23T00:37:55.219641+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'27","last_scrub_stamp":"2024-02-23T00:37:05.079149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.079149+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":24,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.4","version":"442'49","reported_seq":473,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077034+0000","last_change":"2024-02-23T00:36:45.668442+0000","last_active":"2024-02-23T00:37:56.077034+0000","last_peered":"2024-02-23T00:37:56.077034+0000","last_clean":"2024-02-23T00:37:56.077034+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:37:56.077034+0000","last_undegraded":"2024-02-23T00:37:56.077034+0000","last_fullsized":"2024-02-23T00:37:56.077034+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'39","last_scrub_stamp":"2024-02-23T00:34:19.398065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.398065+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.5","version":"471'126","reported_seq":544,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.220202+0000","last_change":"2024-02-23T00:36:37.105105+0000","last_active":"2024-02-23T00:37:55.220202+0000","last_peered":"2024-02-23T00:37:55.220202+0000","last_clean":"2024-02-23T00:36:33.192144+0000","last_became_active":"2024-02-23T00:36:37.105105+0000","last_became_peered":"2024-02-23T00:36:37.105105+0000","last_unstale":"2024-02-23T00:37:55.220202+0000","last_undegraded":"2024-02-23T00:36:37.095637+0000","last_fullsized":"2024-02-23T00:36:37.094980+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'124","last_scrub_stamp":"2024-02-23T00:34:19.695449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.695449+0000","log_size":126,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":92,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":39,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.6","version":"340'52","reported_seq":485,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.222932+0000","last_change":"2024-02-23T00:36:37.120734+0000","last_active":"2024-02-23T00:37:55.222932+0000","last_peered":"2024-02-23T00:37:55.222932+0000","last_clean":"2024-02-23T00:36:33.194384+0000","last_became_active":"2024-02-23T00:36:37.120734+0000","last_became_peered":"2024-02-23T00:36:37.120734+0000","last_unstale":"2024-02-23T00:37:55.222932+0000","last_undegraded":"2024-02-23T00:36:37.106912+0000","last_fullsized":"2024-02-23T00:36:37.106799+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:34:31.733148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.733148+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.7","version":"401'97","reported_seq":513,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.353105+0000","last_change":"2024-02-23T00:36:45.610239+0000","last_active":"2024-02-23T00:37:55.353105+0000","last_peered":"2024-02-23T00:37:55.353105+0000","last_clean":"2024-02-23T00:37:55.353105+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:37:55.353105+0000","last_undegraded":"2024-02-23T00:37:55.353105+0000","last_fullsized":"2024-02-23T00:37:55.353105+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'92","last_scrub_stamp":"2024-02-23T00:34:18.124049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:18.124049+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":94,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.8","version":"340'71","reported_seq":423,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.351554+0000","last_change":"2024-02-23T00:36:45.607171+0000","last_active":"2024-02-23T00:37:55.351554+0000","last_peered":"2024-02-23T00:37:55.351554+0000","last_clean":"2024-02-23T00:37:55.351554+0000","last_became_active":"2024-02-23T00:34:41.115222+0000","last_became_peered":"2024-02-23T00:34:41.115222+0000","last_unstale":"2024-02-23T00:37:55.351554+0000","last_undegraded":"2024-02-23T00:37:55.351554+0000","last_fullsized":"2024-02-23T00:37:55.351554+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'71","last_scrub_stamp":"2024-02-23T00:35:41.651500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.651500+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.9","version":"405'19","reported_seq":448,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.351849+0000","last_change":"2024-02-23T00:36:45.606951+0000","last_active":"2024-02-23T00:37:55.351849+0000","last_peered":"2024-02-23T00:37:55.351849+0000","last_clean":"2024-02-23T00:37:55.351849+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:37:55.351849+0000","last_undegraded":"2024-02-23T00:37:55.351849+0000","last_fullsized":"2024-02-23T00:37:55.351849+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'17","last_scrub_stamp":"2024-02-23T00:34:17.107575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.107575+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.a","version":"248'22","reported_seq":458,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.735331+0000","last_change":"2024-02-23T00:38:00.735331+0000","last_active":"2024-02-23T00:38:00.735331+0000","last_peered":"2024-02-23T00:38:00.735331+0000","last_clean":"2024-02-23T00:38:00.735331+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:38:00.735331+0000","last_undegraded":"2024-02-23T00:38:00.735331+0000","last_fullsized":"2024-02-23T00:38:00.735331+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'22","last_scrub_stamp":"2024-02-23T00:38:00.735240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.735240+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.b","version":"340'35","reported_seq":494,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:56.076792+0000","last_change":"2024-02-23T00:36:37.120097+0000","last_active":"2024-02-23T00:37:56.076792+0000","last_peered":"2024-02-23T00:37:56.076792+0000","last_clean":"2024-02-23T00:36:33.368807+0000","last_became_active":"2024-02-23T00:36:37.120097+0000","last_became_peered":"2024-02-23T00:36:37.120097+0000","last_unstale":"2024-02-23T00:37:56.076792+0000","last_undegraded":"2024-02-23T00:36:37.107185+0000","last_fullsized":"2024-02-23T00:36:37.107054+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-23T00:33:37.391203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.391203+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.c","version":"422'97","reported_seq":522,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.280637+0000","last_change":"2024-02-23T00:36:37.106629+0000","last_active":"2024-02-23T00:37:55.280637+0000","last_peered":"2024-02-23T00:37:55.280637+0000","last_clean":"2024-02-23T00:36:33.366529+0000","last_became_active":"2024-02-23T00:36:37.106629+0000","last_became_peered":"2024-02-23T00:36:37.106629+0000","last_unstale":"2024-02-23T00:37:55.280637+0000","last_undegraded":"2024-02-23T00:36:37.096711+0000","last_fullsized":"2024-02-23T00:36:37.096581+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'76","last_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_deep_scrub":"397'76","last_deep_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.098352+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.d","version":"395'39","reported_seq":493,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075572+0000","last_change":"2024-02-23T00:36:45.669435+0000","last_active":"2024-02-23T00:37:56.075572+0000","last_peered":"2024-02-23T00:37:56.075572+0000","last_clean":"2024-02-23T00:37:56.075572+0000","last_became_active":"2024-02-23T00:34:41.109682+0000","last_became_peered":"2024-02-23T00:34:41.109682+0000","last_unstale":"2024-02-23T00:37:56.075572+0000","last_undegraded":"2024-02-23T00:37:56.075572+0000","last_fullsized":"2024-02-23T00:37:56.075572+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-02-23T00:33:45.265466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:45.265466+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.e","version":"405'77","reported_seq":557,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.353029+0000","last_change":"2024-02-23T00:36:45.609915+0000","last_active":"2024-02-23T00:37:55.353029+0000","last_peered":"2024-02-23T00:37:55.353029+0000","last_clean":"2024-02-23T00:37:55.353029+0000","last_became_active":"2024-02-23T00:34:41.108381+0000","last_became_peered":"2024-02-23T00:34:41.108381+0000","last_unstale":"2024-02-23T00:37:55.353029+0000","last_undegraded":"2024-02-23T00:37:55.353029+0000","last_fullsized":"2024-02-23T00:37:55.353029+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'74","last_scrub_stamp":"2024-02-23T00:34:15.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.172848+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000073,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":3931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":4000057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.f","version":"471'71","reported_seq":418,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.353711+0000","last_change":"2024-02-23T00:36:45.610425+0000","last_active":"2024-02-23T00:37:55.353711+0000","last_peered":"2024-02-23T00:37:55.353711+0000","last_clean":"2024-02-23T00:37:55.353711+0000","last_became_active":"2024-02-23T00:34:41.101718+0000","last_became_peered":"2024-02-23T00:34:41.101718+0000","last_unstale":"2024-02-23T00:37:55.353711+0000","last_undegraded":"2024-02-23T00:37:55.353711+0000","last_fullsized":"2024-02-23T00:37:55.353711+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'69","last_scrub_stamp":"2024-02-23T00:35:46.750022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.750022+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.10","version":"401'37","reported_seq":463,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.351657+0000","last_change":"2024-02-23T00:36:37.119458+0000","last_active":"2024-02-23T00:37:55.351657+0000","last_peered":"2024-02-23T00:37:55.351657+0000","last_clean":"2024-02-23T00:36:33.364885+0000","last_became_active":"2024-02-23T00:36:37.119458+0000","last_became_peered":"2024-02-23T00:36:37.119458+0000","last_unstale":"2024-02-23T00:37:55.351657+0000","last_undegraded":"2024-02-23T00:36:37.098703+0000","last_fullsized":"2024-02-23T00:36:37.098584+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'34","last_scrub_stamp":"2024-02-23T00:35:01.791840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.791840+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.11","version":"392'50","reported_seq":480,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.219013+0000","last_change":"2024-02-23T00:37:06.042258+0000","last_active":"2024-02-23T00:37:55.219013+0000","last_peered":"2024-02-23T00:37:55.219013+0000","last_clean":"2024-02-23T00:37:55.219013+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:37:55.219013+0000","last_undegraded":"2024-02-23T00:37:55.219013+0000","last_fullsized":"2024-02-23T00:37:55.219013+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'50","last_scrub_stamp":"2024-02-23T00:37:06.042181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:06.042181+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.28","version":"0'0","reported_seq":102,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.074456+0000","last_change":"2024-02-23T00:35:29.092753+0000","last_active":"2024-02-23T00:37:56.074456+0000","last_peered":"2024-02-23T00:37:56.074456+0000","last_clean":"2024-02-23T00:37:56.074456+0000","last_became_active":"2024-02-23T00:35:29.092396+0000","last_became_peered":"2024-02-23T00:35:29.092396+0000","last_unstale":"2024-02-23T00:37:56.074456+0000","last_undegraded":"2024-02-23T00:37:56.074456+0000","last_fullsized":"2024-02-23T00:37:56.074456+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:42.815722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:42.815722+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.12","version":"204'68","reported_seq":486,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.279878+0000","last_change":"2024-02-23T00:36:37.114447+0000","last_active":"2024-02-23T00:37:55.279878+0000","last_peered":"2024-02-23T00:37:55.279878+0000","last_clean":"2024-02-23T00:36:33.364627+0000","last_became_active":"2024-02-23T00:36:37.114447+0000","last_became_peered":"2024-02-23T00:36:37.114447+0000","last_unstale":"2024-02-23T00:37:55.279878+0000","last_undegraded":"2024-02-23T00:36:37.099733+0000","last_fullsized":"2024-02-23T00:36:37.099613+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'68","last_scrub_stamp":"2024-02-23T00:36:27.078990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.078990+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.29","version":"62'8","reported_seq":512,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:56.077249+0000","last_change":"2024-02-23T00:36:37.122045+0000","last_active":"2024-02-23T00:37:56.077249+0000","last_peered":"2024-02-23T00:37:56.077249+0000","last_clean":"2024-02-23T00:35:27.100547+0000","last_became_active":"2024-02-23T00:36:37.122045+0000","last_became_peered":"2024-02-23T00:36:37.122045+0000","last_unstale":"2024-02-23T00:37:56.077249+0000","last_undegraded":"2024-02-23T00:36:37.107771+0000","last_fullsized":"2024-02-23T00:36:37.107625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:27.673481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.673481+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.13","version":"248'61","reported_seq":488,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.076686+0000","last_change":"2024-02-23T00:36:45.667732+0000","last_active":"2024-02-23T00:37:56.076686+0000","last_peered":"2024-02-23T00:37:56.076686+0000","last_clean":"2024-02-23T00:37:56.076686+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:37:56.076686+0000","last_undegraded":"2024-02-23T00:37:56.076686+0000","last_fullsized":"2024-02-23T00:37:56.076686+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'61","last_scrub_stamp":"2024-02-23T00:36:38.944617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:38.944617+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.14","version":"208'85","reported_seq":506,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.280482+0000","last_change":"2024-02-23T00:36:37.115695+0000","last_active":"2024-02-23T00:37:55.280482+0000","last_peered":"2024-02-23T00:37:55.280482+0000","last_clean":"2024-02-23T00:36:33.366438+0000","last_became_active":"2024-02-23T00:36:37.115695+0000","last_became_peered":"2024-02-23T00:36:37.115695+0000","last_unstale":"2024-02-23T00:37:55.280482+0000","last_undegraded":"2024-02-23T00:36:37.099640+0000","last_fullsized":"2024-02-23T00:36:37.099515+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'85","last_scrub_stamp":"2024-02-23T00:35:46.085280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.085280+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.15","version":"340'70","reported_seq":499,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220077+0000","last_change":"2024-02-23T00:37:04.031754+0000","last_active":"2024-02-23T00:37:55.220077+0000","last_peered":"2024-02-23T00:37:55.220077+0000","last_clean":"2024-02-23T00:37:55.220077+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:37:55.220077+0000","last_undegraded":"2024-02-23T00:37:55.220077+0000","last_fullsized":"2024-02-23T00:37:55.220077+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'70","last_scrub_stamp":"2024-02-23T00:37:04.031654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:04.031654+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.16","version":"204'51","reported_seq":485,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.222814+0000","last_change":"2024-02-23T00:36:37.111665+0000","last_active":"2024-02-23T00:37:55.222814+0000","last_peered":"2024-02-23T00:37:55.222814+0000","last_clean":"2024-02-23T00:36:33.194334+0000","last_became_active":"2024-02-23T00:36:37.111665+0000","last_became_peered":"2024-02-23T00:36:37.111665+0000","last_unstale":"2024-02-23T00:37:55.222814+0000","last_undegraded":"2024-02-23T00:36:37.100202+0000","last_fullsized":"2024-02-23T00:36:37.100076+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:35:46.192706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.192706+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.17","version":"433'112","reported_seq":536,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:04.819561+0000","last_change":"2024-02-23T00:38:04.819561+0000","last_active":"2024-02-23T00:38:04.819561+0000","last_peered":"2024-02-23T00:38:04.819561+0000","last_clean":"2024-02-23T00:38:04.819561+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:38:04.819561+0000","last_undegraded":"2024-02-23T00:38:04.819561+0000","last_fullsized":"2024-02-23T00:38:04.819561+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'112","last_scrub_stamp":"2024-02-23T00:38:04.819475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:04.819475+0000","log_size":112,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.22","version":"0'0","reported_seq":109,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075655+0000","last_change":"2024-02-23T00:37:09.026463+0000","last_active":"2024-02-23T00:37:56.075655+0000","last_peered":"2024-02-23T00:37:56.075655+0000","last_clean":"2024-02-23T00:37:56.075655+0000","last_became_active":"2024-02-23T00:35:29.089082+0000","last_became_peered":"2024-02-23T00:35:29.089082+0000","last_unstale":"2024-02-23T00:37:56.075655+0000","last_undegraded":"2024-02-23T00:37:56.075655+0000","last_fullsized":"2024-02-23T00:37:56.075655+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.026383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.026383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.18","version":"211'57","reported_seq":486,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.074546+0000","last_change":"2024-02-23T00:36:45.668902+0000","last_active":"2024-02-23T00:37:56.074546+0000","last_peered":"2024-02-23T00:37:56.074546+0000","last_clean":"2024-02-23T00:37:56.074546+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:37:56.074546+0000","last_undegraded":"2024-02-23T00:37:56.074546+0000","last_fullsized":"2024-02-23T00:37:56.074546+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'57","last_scrub_stamp":"2024-02-23T00:35:54.857184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:54.857184+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.23","version":"426'11","reported_seq":149,"reported_epoch":473,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:00.460223+0000","last_change":"2024-02-23T00:36:37.112756+0000","last_active":"2024-02-23T00:38:00.460223+0000","last_peered":"2024-02-23T00:38:00.460223+0000","last_clean":"2024-02-23T00:36:35.444860+0000","last_became_active":"2024-02-23T00:36:37.112756+0000","last_became_peered":"2024-02-23T00:36:37.112756+0000","last_unstale":"2024-02-23T00:38:00.460223+0000","last_undegraded":"2024-02-23T00:36:37.094265+0000","last_fullsized":"2024-02-23T00:36:37.094133+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.19","version":"466'75","reported_seq":511,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.218915+0000","last_change":"2024-02-23T00:36:45.297690+0000","last_active":"2024-02-23T00:37:55.218915+0000","last_peered":"2024-02-23T00:37:55.218915+0000","last_clean":"2024-02-23T00:37:55.218915+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:37:55.218915+0000","last_undegraded":"2024-02-23T00:37:55.218915+0000","last_fullsized":"2024-02-23T00:37:55.218915+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'60","last_scrub_stamp":"2024-02-23T00:36:31.102608+0000","last_deep_scrub":"271'54","last_deep_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_clean_scrub_stamp":"2024-02-23T00:36:31.102608+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.20","version":"379'11","reported_seq":146,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.460505+0000","last_change":"2024-02-23T00:37:10.032410+0000","last_active":"2024-02-23T00:38:00.460505+0000","last_peered":"2024-02-23T00:38:00.460505+0000","last_clean":"2024-02-23T00:38:00.460505+0000","last_became_active":"2024-02-23T00:35:29.093220+0000","last_became_peered":"2024-02-23T00:35:29.093220+0000","last_unstale":"2024-02-23T00:38:00.460505+0000","last_undegraded":"2024-02-23T00:38:00.460505+0000","last_fullsized":"2024-02-23T00:38:00.460505+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"379'11","last_scrub_stamp":"2024-02-23T00:37:10.032341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:10.032341+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1a","version":"202'68","reported_seq":499,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.221473+0000","last_change":"2024-02-23T00:37:15.836653+0000","last_active":"2024-02-23T00:37:55.221473+0000","last_peered":"2024-02-23T00:37:55.221473+0000","last_clean":"2024-02-23T00:37:55.221473+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:37:55.221473+0000","last_undegraded":"2024-02-23T00:37:55.221473+0000","last_fullsized":"2024-02-23T00:37:55.221473+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'68","last_scrub_stamp":"2024-02-23T00:37:15.836572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:15.836572+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.21","version":"0'0","reported_seq":107,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.279576+0000","last_change":"2024-02-23T00:36:34.152721+0000","last_active":"2024-02-23T00:37:55.279576+0000","last_peered":"2024-02-23T00:37:55.279576+0000","last_clean":"2024-02-23T00:37:55.279576+0000","last_became_active":"2024-02-23T00:35:29.093053+0000","last_became_peered":"2024-02-23T00:35:29.093053+0000","last_unstale":"2024-02-23T00:37:55.279576+0000","last_undegraded":"2024-02-23T00:37:55.279576+0000","last_fullsized":"2024-02-23T00:37:55.279576+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.152643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:34.152643+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.152643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.1b","version":"398'54","reported_seq":477,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075352+0000","last_change":"2024-02-23T00:36:45.666523+0000","last_active":"2024-02-23T00:37:56.075352+0000","last_peered":"2024-02-23T00:37:56.075352+0000","last_clean":"2024-02-23T00:37:56.075352+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:37:56.075352+0000","last_undegraded":"2024-02-23T00:37:56.075352+0000","last_fullsized":"2024-02-23T00:37:56.075352+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-02-23T00:33:46.274400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.274400+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.26","version":"0'0","reported_seq":457,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.075416+0000","last_change":"2024-02-23T00:36:37.122228+0000","last_active":"2024-02-23T00:37:56.075416+0000","last_peered":"2024-02-23T00:37:56.075416+0000","last_clean":"2024-02-23T00:36:33.367599+0000","last_became_active":"2024-02-23T00:36:37.122228+0000","last_became_peered":"2024-02-23T00:36:37.122228+0000","last_unstale":"2024-02-23T00:37:56.075416+0000","last_undegraded":"2024-02-23T00:37:56.075416+0000","last_fullsized":"2024-02-23T00:36:37.110661+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1c","version":"216'34","reported_seq":489,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352290+0000","last_change":"2024-02-23T00:36:45.608757+0000","last_active":"2024-02-23T00:37:55.352290+0000","last_peered":"2024-02-23T00:37:55.352290+0000","last_clean":"2024-02-23T00:37:55.352290+0000","last_became_active":"2024-02-23T00:34:41.117846+0000","last_became_peered":"2024-02-23T00:34:41.117846+0000","last_unstale":"2024-02-23T00:37:55.352290+0000","last_undegraded":"2024-02-23T00:37:55.352290+0000","last_fullsized":"2024-02-23T00:37:55.352290+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'34","last_scrub_stamp":"2024-02-23T00:34:57.859047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.859047+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.27","version":"0'0","reported_seq":441,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352983+0000","last_change":"2024-02-23T00:35:29.092282+0000","last_active":"2024-02-23T00:37:55.352983+0000","last_peered":"2024-02-23T00:37:55.352983+0000","last_clean":"2024-02-23T00:37:55.352983+0000","last_became_active":"2024-02-23T00:35:29.091970+0000","last_became_peered":"2024-02-23T00:35:29.091970+0000","last_unstale":"2024-02-23T00:37:55.352983+0000","last_undegraded":"2024-02-23T00:37:55.352983+0000","last_fullsized":"2024-02-23T00:37:55.352983+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1d","version":"340'13","reported_seq":186,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280562+0000","last_change":"2024-02-23T00:36:45.408437+0000","last_active":"2024-02-23T00:37:55.280562+0000","last_peered":"2024-02-23T00:37:55.280562+0000","last_clean":"2024-02-23T00:37:55.280562+0000","last_became_active":"2024-02-23T00:34:42.058724+0000","last_became_peered":"2024-02-23T00:34:42.058724+0000","last_unstale":"2024-02-23T00:37:55.280562+0000","last_undegraded":"2024-02-23T00:37:55.280562+0000","last_fullsized":"2024-02-23T00:37:55.280562+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'13","last_scrub_stamp":"2024-02-23T00:36:40.328657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:40.328657+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.24","version":"0'0","reported_seq":453,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.075039+0000","last_change":"2024-02-23T00:36:37.117551+0000","last_active":"2024-02-23T00:37:56.075039+0000","last_peered":"2024-02-23T00:37:56.075039+0000","last_clean":"2024-02-23T00:36:33.366613+0000","last_became_active":"2024-02-23T00:36:37.117551+0000","last_became_peered":"2024-02-23T00:36:37.117551+0000","last_unstale":"2024-02-23T00:37:56.075039+0000","last_undegraded":"2024-02-23T00:37:56.075039+0000","last_fullsized":"2024-02-23T00:36:37.104246+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1e","version":"340'52","reported_seq":471,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075075+0000","last_change":"2024-02-23T00:36:45.669526+0000","last_active":"2024-02-23T00:37:56.075075+0000","last_peered":"2024-02-23T00:37:56.075075+0000","last_clean":"2024-02-23T00:37:56.075075+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:37:56.075075+0000","last_undegraded":"2024-02-23T00:37:56.075075+0000","last_fullsized":"2024-02-23T00:37:56.075075+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"232'51","last_scrub_stamp":"2024-02-23T00:33:49.219607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.219607+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.25","version":"0'0","reported_seq":450,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352505+0000","last_change":"2024-02-23T00:35:29.091129+0000","last_active":"2024-02-23T00:37:55.352505+0000","last_peered":"2024-02-23T00:37:55.352505+0000","last_clean":"2024-02-23T00:37:55.352505+0000","last_became_active":"2024-02-23T00:35:29.090793+0000","last_became_peered":"2024-02-23T00:35:29.090793+0000","last_unstale":"2024-02-23T00:37:55.352505+0000","last_undegraded":"2024-02-23T00:37:55.352505+0000","last_fullsized":"2024-02-23T00:37:55.352505+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1f","version":"439'74","reported_seq":497,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:56.076545+0000","last_change":"2024-02-23T00:36:37.117847+0000","last_active":"2024-02-23T00:37:56.076545+0000","last_peered":"2024-02-23T00:37:56.076545+0000","last_clean":"2024-02-23T00:36:33.365701+0000","last_became_active":"2024-02-23T00:36:37.117847+0000","last_became_peered":"2024-02-23T00:36:37.117847+0000","last_unstale":"2024-02-23T00:37:56.076545+0000","last_undegraded":"2024-02-23T00:36:37.102042+0000","last_fullsized":"2024-02-23T00:36:37.101901+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-23T00:33:39.351573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.351573+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"379'9","reported_seq":595,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.460418+0000","last_change":"2024-02-23T00:37:05.955772+0000","last_active":"2024-02-23T00:38:00.460418+0000","last_peered":"2024-02-23T00:38:00.460418+0000","last_clean":"2024-02-23T00:38:00.460418+0000","last_became_active":"2024-02-23T00:35:27.095085+0000","last_became_peered":"2024-02-23T00:35:27.095085+0000","last_unstale":"2024-02-23T00:38:00.460418+0000","last_undegraded":"2024-02-23T00:38:00.460418+0000","last_fullsized":"2024-02-23T00:38:00.460418+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'9","last_scrub_stamp":"2024-02-23T00:37:05.955707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.955707+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":547,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.074636+0000","last_change":"2024-02-23T00:37:22.011162+0000","last_active":"2024-02-23T00:37:56.074636+0000","last_peered":"2024-02-23T00:37:56.074636+0000","last_clean":"2024-02-23T00:37:56.074636+0000","last_became_active":"2024-02-23T00:35:13.074767+0000","last_became_peered":"2024-02-23T00:35:13.074767+0000","last_unstale":"2024-02-23T00:37:56.074636+0000","last_undegraded":"2024-02-23T00:37:56.074636+0000","last_fullsized":"2024-02-23T00:37:56.074636+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":106,"num_read_kb":88,"num_write":62,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":466,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.223122+0000","last_change":"2024-02-23T00:37:05.837736+0000","last_active":"2024-02-23T00:37:55.223122+0000","last_peered":"2024-02-23T00:37:55.223122+0000","last_clean":"2024-02-23T00:37:55.223122+0000","last_became_active":"2024-02-23T00:34:41.106070+0000","last_became_peered":"2024-02-23T00:34:41.106070+0000","last_unstale":"2024-02-23T00:37:55.223122+0000","last_undegraded":"2024-02-23T00:37:55.223122+0000","last_fullsized":"2024-02-23T00:37:55.223122+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:05.837666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.837666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":435,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.075841+0000","last_change":"2024-02-23T00:36:37.104200+0000","last_active":"2024-02-23T00:37:56.075841+0000","last_peered":"2024-02-23T00:37:56.075841+0000","last_clean":"2024-02-23T00:36:33.366159+0000","last_became_active":"2024-02-23T00:36:37.104200+0000","last_became_peered":"2024-02-23T00:36:37.104200+0000","last_unstale":"2024-02-23T00:37:56.075841+0000","last_undegraded":"2024-02-23T00:37:56.075841+0000","last_fullsized":"2024-02-23T00:36:37.102434+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:21.441958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.441958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":399,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.352463+0000","last_change":"2024-02-23T00:36:37.115879+0000","last_active":"2024-02-23T00:37:55.352463+0000","last_peered":"2024-02-23T00:37:55.352463+0000","last_clean":"2024-02-23T00:36:33.366205+0000","last_became_active":"2024-02-23T00:36:37.115879+0000","last_became_peered":"2024-02-23T00:36:37.115879+0000","last_unstale":"2024-02-23T00:37:55.352463+0000","last_undegraded":"2024-02-23T00:37:55.352463+0000","last_fullsized":"2024-02-23T00:36:37.101967+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":764,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:04.436549+0000","last_change":"2024-02-23T00:36:27.182035+0000","last_active":"2024-02-23T00:38:04.436549+0000","last_peered":"2024-02-23T00:38:04.436549+0000","last_clean":"2024-02-23T00:38:04.436549+0000","last_became_active":"2024-02-23T00:34:41.114058+0000","last_became_peered":"2024-02-23T00:34:41.114058+0000","last_unstale":"2024-02-23T00:38:04.436549+0000","last_undegraded":"2024-02-23T00:38:04.436549+0000","last_fullsized":"2024-02-23T00:38:04.436549+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:36:27.181972+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.181972+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":327,"num_read_kb":327,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":111,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.353526+0000","last_change":"2024-02-23T00:36:37.112695+0000","last_active":"2024-02-23T00:37:55.353526+0000","last_peered":"2024-02-23T00:37:55.353526+0000","last_clean":"2024-02-23T00:27:29.183258+0000","last_became_active":"2024-02-23T00:36:37.112695+0000","last_became_peered":"2024-02-23T00:36:37.112695+0000","last_unstale":"2024-02-23T00:37:55.353526+0000","last_undegraded":"2024-02-23T00:36:37.098637+0000","last_fullsized":"2024-02-23T00:36:37.098502+0000","mapping_epoch":425,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":62,"num_read_kb":50,"num_write":36,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":451,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.353433+0000","last_change":"2024-02-23T00:36:41.532683+0000","last_active":"2024-02-23T00:37:55.353433+0000","last_peered":"2024-02-23T00:37:55.353433+0000","last_clean":"2024-02-23T00:37:55.353433+0000","last_became_active":"2024-02-23T00:34:41.100003+0000","last_became_peered":"2024-02-23T00:34:41.100003+0000","last_unstale":"2024-02-23T00:37:55.353433+0000","last_undegraded":"2024-02-23T00:37:55.353433+0000","last_fullsized":"2024-02-23T00:37:55.353433+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.532619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.532619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":458,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.221595+0000","last_change":"2024-02-23T00:36:37.107478+0000","last_active":"2024-02-23T00:37:55.221595+0000","last_peered":"2024-02-23T00:37:55.221595+0000","last_clean":"2024-02-23T00:36:33.193986+0000","last_became_active":"2024-02-23T00:36:37.107478+0000","last_became_peered":"2024-02-23T00:36:37.107478+0000","last_unstale":"2024-02-23T00:37:55.221595+0000","last_undegraded":"2024-02-23T00:37:55.221595+0000","last_fullsized":"2024-02-23T00:36:37.103166+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.679076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":55,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.077044+0000","last_change":"2024-02-23T00:36:37.122229+0000","last_active":"2024-02-23T00:37:56.077044+0000","last_peered":"2024-02-23T00:37:56.077044+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.122229+0000","last_became_peered":"2024-02-23T00:36:37.122229+0000","last_unstale":"2024-02-23T00:37:56.077044+0000","last_undegraded":"2024-02-23T00:37:56.077044+0000","last_fullsized":"2024-02-23T00:36:37.107921+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":394,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:01.361178+0000","last_change":"2024-02-23T00:38:01.361178+0000","last_active":"2024-02-23T00:38:01.361178+0000","last_peered":"2024-02-23T00:38:01.361178+0000","last_clean":"2024-02-23T00:38:01.361178+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:38:01.361178+0000","last_undegraded":"2024-02-23T00:38:01.361178+0000","last_fullsized":"2024-02-23T00:38:01.361178+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:01.361095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:01.361095+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"453'47","reported_seq":542,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280378+0000","last_change":"2024-02-23T00:36:37.230412+0000","last_active":"2024-02-23T00:37:55.280378+0000","last_peered":"2024-02-23T00:37:55.280378+0000","last_clean":"2024-02-23T00:37:55.280378+0000","last_became_active":"2024-02-23T00:34:59.084636+0000","last_became_peered":"2024-02-23T00:34:59.084636+0000","last_unstale":"2024-02-23T00:37:55.280378+0000","last_undegraded":"2024-02-23T00:37:55.280378+0000","last_fullsized":"2024-02-23T00:37:55.280378+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:36:37.230351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:37.230351+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":116,"num_read_kb":96,"num_write":70,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":458,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.223097+0000","last_change":"2024-02-23T00:36:37.116992+0000","last_active":"2024-02-23T00:37:55.223097+0000","last_peered":"2024-02-23T00:37:55.223097+0000","last_clean":"2024-02-23T00:36:33.194161+0000","last_became_active":"2024-02-23T00:36:37.116992+0000","last_became_peered":"2024-02-23T00:36:37.116992+0000","last_unstale":"2024-02-23T00:37:55.223097+0000","last_undegraded":"2024-02-23T00:37:55.223097+0000","last_fullsized":"2024-02-23T00:36:37.104222+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:50.194656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:50.194656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":449,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.222177+0000","last_change":"2024-02-23T00:36:37.113419+0000","last_active":"2024-02-23T00:37:55.222177+0000","last_peered":"2024-02-23T00:37:55.222177+0000","last_clean":"2024-02-23T00:36:33.194007+0000","last_became_active":"2024-02-23T00:36:37.113419+0000","last_became_peered":"2024-02-23T00:36:37.113419+0000","last_unstale":"2024-02-23T00:37:55.222177+0000","last_undegraded":"2024-02-23T00:37:55.222177+0000","last_fullsized":"2024-02-23T00:36:37.104161+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.745865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.745865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":447,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077360+0000","last_change":"2024-02-23T00:35:04.673936+0000","last_active":"2024-02-23T00:37:56.077360+0000","last_peered":"2024-02-23T00:37:56.077360+0000","last_clean":"2024-02-23T00:37:56.077360+0000","last_became_active":"2024-02-23T00:34:41.105277+0000","last_became_peered":"2024-02-23T00:34:41.105277+0000","last_unstale":"2024-02-23T00:37:56.077360+0000","last_undegraded":"2024-02-23T00:37:56.077360+0000","last_fullsized":"2024-02-23T00:37:56.077360+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.673837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.673837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":56,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.221284+0000","last_change":"2024-02-23T00:36:37.119721+0000","last_active":"2024-02-23T00:37:55.221284+0000","last_peered":"2024-02-23T00:37:55.221284+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119721+0000","last_became_peered":"2024-02-23T00:36:37.119721+0000","last_unstale":"2024-02-23T00:37:55.221284+0000","last_undegraded":"2024-02-23T00:37:55.221284+0000","last_fullsized":"2024-02-23T00:36:37.102163+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.205312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.205312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":456,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280876+0000","last_change":"2024-02-23T00:37:12.370077+0000","last_active":"2024-02-23T00:37:55.280876+0000","last_peered":"2024-02-23T00:37:55.280876+0000","last_clean":"2024-02-23T00:37:55.280876+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:37:55.280876+0000","last_undegraded":"2024-02-23T00:37:55.280876+0000","last_fullsized":"2024-02-23T00:37:55.280876+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:12.369966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:12.369966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":520,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220776+0000","last_change":"2024-02-23T00:36:25.082682+0000","last_active":"2024-02-23T00:37:55.220776+0000","last_peered":"2024-02-23T00:37:55.220776+0000","last_clean":"2024-02-23T00:37:55.220776+0000","last_became_active":"2024-02-23T00:34:47.062796+0000","last_became_peered":"2024-02-23T00:34:47.062796+0000","last_unstale":"2024-02-23T00:37:55.220776+0000","last_undegraded":"2024-02-23T00:37:55.220776+0000","last_fullsized":"2024-02-23T00:37:55.220776+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":96,"num_read_kb":76,"num_write":58,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":444,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077297+0000","last_change":"2024-02-23T00:35:40.824848+0000","last_active":"2024-02-23T00:37:56.077297+0000","last_peered":"2024-02-23T00:37:56.077297+0000","last_clean":"2024-02-23T00:37:56.077297+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:37:56.077297+0000","last_undegraded":"2024-02-23T00:37:56.077297+0000","last_fullsized":"2024-02-23T00:37:56.077297+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:40.824741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:40.824741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":439,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.352540+0000","last_change":"2024-02-23T00:36:37.109480+0000","last_active":"2024-02-23T00:37:55.352540+0000","last_peered":"2024-02-23T00:37:55.352540+0000","last_clean":"2024-02-23T00:36:34.605757+0000","last_became_active":"2024-02-23T00:36:37.109480+0000","last_became_peered":"2024-02-23T00:36:37.109480+0000","last_unstale":"2024-02-23T00:37:55.352540+0000","last_undegraded":"2024-02-23T00:37:55.352540+0000","last_fullsized":"2024-02-23T00:36:37.095167+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.605664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.605664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":462,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.223173+0000","last_change":"2024-02-23T00:37:17.822884+0000","last_active":"2024-02-23T00:37:55.223173+0000","last_peered":"2024-02-23T00:37:55.223173+0000","last_clean":"2024-02-23T00:37:55.223173+0000","last_became_active":"2024-02-23T00:34:41.090777+0000","last_became_peered":"2024-02-23T00:34:41.090777+0000","last_unstale":"2024-02-23T00:37:55.223173+0000","last_undegraded":"2024-02-23T00:37:55.223173+0000","last_fullsized":"2024-02-23T00:37:55.223173+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:17.822776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.822776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":407,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.077983+0000","last_change":"2024-02-23T00:36:37.124688+0000","last_active":"2024-02-23T00:37:56.077983+0000","last_peered":"2024-02-23T00:37:56.077983+0000","last_clean":"2024-02-23T00:36:33.367188+0000","last_became_active":"2024-02-23T00:36:37.124688+0000","last_became_peered":"2024-02-23T00:36:37.124688+0000","last_unstale":"2024-02-23T00:37:56.077983+0000","last_undegraded":"2024-02-23T00:37:56.077983+0000","last_fullsized":"2024-02-23T00:36:37.113140+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.704478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.704478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":165,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.219873+0000","last_change":"2024-02-23T00:36:37.113121+0000","last_active":"2024-02-23T00:37:55.219873+0000","last_peered":"2024-02-23T00:37:55.219873+0000","last_clean":"2024-02-23T00:31:25.058059+0000","last_became_active":"2024-02-23T00:36:37.113121+0000","last_became_peered":"2024-02-23T00:36:37.113121+0000","last_unstale":"2024-02-23T00:37:55.219873+0000","last_undegraded":"2024-02-23T00:37:55.219873+0000","last_fullsized":"2024-02-23T00:36:37.095087+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":531,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.219834+0000","last_change":"2024-02-23T00:37:09.133547+0000","last_active":"2024-02-23T00:37:55.219834+0000","last_peered":"2024-02-23T00:37:55.219834+0000","last_clean":"2024-02-23T00:37:55.219834+0000","last_became_active":"2024-02-23T00:34:51.102216+0000","last_became_peered":"2024-02-23T00:34:51.102216+0000","last_unstale":"2024-02-23T00:37:55.219834+0000","last_undegraded":"2024-02-23T00:37:55.219834+0000","last_fullsized":"2024-02-23T00:37:55.219834+0000","mapping_epoch":329,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":340,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:37:09.133471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.133471+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":48,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":464,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352107+0000","last_change":"2024-02-23T00:35:56.678738+0000","last_active":"2024-02-23T00:37:55.352107+0000","last_peered":"2024-02-23T00:37:55.352107+0000","last_clean":"2024-02-23T00:37:55.352107+0000","last_became_active":"2024-02-23T00:34:41.115434+0000","last_became_peered":"2024-02-23T00:34:41.115434+0000","last_unstale":"2024-02-23T00:37:55.352107+0000","last_undegraded":"2024-02-23T00:37:55.352107+0000","last_fullsized":"2024-02-23T00:37:55.352107+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.678676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.678676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":586,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.459809+0000","last_change":"2024-02-23T00:37:07.822516+0000","last_active":"2024-02-23T00:38:00.459809+0000","last_peered":"2024-02-23T00:38:00.459809+0000","last_clean":"2024-02-23T00:38:00.459809+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:38:00.459809+0000","last_undegraded":"2024-02-23T00:38:00.459809+0000","last_fullsized":"2024-02-23T00:38:00.459809+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:37:07.822449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:07.822449+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":440,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220552+0000","last_change":"2024-02-23T00:35:56.258132+0000","last_active":"2024-02-23T00:37:55.220552+0000","last_peered":"2024-02-23T00:37:55.220552+0000","last_clean":"2024-02-23T00:37:55.220552+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:37:55.220552+0000","last_undegraded":"2024-02-23T00:37:55.220552+0000","last_fullsized":"2024-02-23T00:37:55.220552+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.258056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.258056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":393,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220799+0000","last_change":"2024-02-23T00:35:46.175575+0000","last_active":"2024-02-23T00:37:55.220799+0000","last_peered":"2024-02-23T00:37:55.220799+0000","last_clean":"2024-02-23T00:37:55.220799+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:37:55.220799+0000","last_undegraded":"2024-02-23T00:37:55.220799+0000","last_fullsized":"2024-02-23T00:37:55.220799+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:46.175476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.175476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":512,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.351512+0000","last_change":"2024-02-23T00:37:22.396292+0000","last_active":"2024-02-23T00:37:55.351512+0000","last_peered":"2024-02-23T00:37:55.351512+0000","last_clean":"2024-02-23T00:37:55.351512+0000","last_became_active":"2024-02-23T00:34:41.103369+0000","last_became_peered":"2024-02-23T00:34:41.103369+0000","last_unstale":"2024-02-23T00:37:55.351512+0000","last_undegraded":"2024-02-23T00:37:55.351512+0000","last_fullsized":"2024-02-23T00:37:55.351512+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:22.396222+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.396222+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":581,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.353796+0000","last_change":"2024-02-23T00:36:39.442794+0000","last_active":"2024-02-23T00:37:55.353796+0000","last_peered":"2024-02-23T00:37:55.353796+0000","last_clean":"2024-02-23T00:37:55.353796+0000","last_became_active":"2024-02-23T00:35:47.079526+0000","last_became_peered":"2024-02-23T00:35:47.079526+0000","last_unstale":"2024-02-23T00:37:55.353796+0000","last_undegraded":"2024-02-23T00:37:55.353796+0000","last_fullsized":"2024-02-23T00:37:55.353796+0000","mapping_epoch":392,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":86,"num_read_kb":69,"num_write":54,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":448,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352000+0000","last_change":"2024-02-23T00:35:06.658103+0000","last_active":"2024-02-23T00:37:55.352000+0000","last_peered":"2024-02-23T00:37:55.352000+0000","last_clean":"2024-02-23T00:37:55.352000+0000","last_became_active":"2024-02-23T00:34:41.107670+0000","last_became_peered":"2024-02-23T00:34:41.107670+0000","last_unstale":"2024-02-23T00:37:55.352000+0000","last_undegraded":"2024-02-23T00:37:55.352000+0000","last_fullsized":"2024-02-23T00:37:55.352000+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.657998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"426'9","reported_seq":587,"reported_epoch":473,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:00.460173+0000","last_change":"2024-02-23T00:36:37.111178+0000","last_active":"2024-02-23T00:38:00.460173+0000","last_peered":"2024-02-23T00:38:00.460173+0000","last_clean":"2024-02-23T00:36:35.444981+0000","last_became_active":"2024-02-23T00:36:37.111178+0000","last_became_peered":"2024-02-23T00:36:37.111178+0000","last_unstale":"2024-02-23T00:38:00.460173+0000","last_undegraded":"2024-02-23T00:36:37.095825+0000","last_fullsized":"2024-02-23T00:36:37.095713+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:59.630171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:59.630171+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":436,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:56.076070+0000","last_change":"2024-02-23T00:36:37.113156+0000","last_active":"2024-02-23T00:37:56.076070+0000","last_peered":"2024-02-23T00:37:56.076070+0000","last_clean":"2024-02-23T00:36:33.366740+0000","last_became_active":"2024-02-23T00:36:37.113156+0000","last_became_peered":"2024-02-23T00:36:37.113156+0000","last_unstale":"2024-02-23T00:37:56.076070+0000","last_undegraded":"2024-02-23T00:36:37.102218+0000","last_fullsized":"2024-02-23T00:36:37.102103+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:34:17.437083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.437083+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":384,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352954+0000","last_change":"2024-02-23T00:33:39.140496+0000","last_active":"2024-02-23T00:37:55.352954+0000","last_peered":"2024-02-23T00:37:55.352954+0000","last_clean":"2024-02-23T00:37:55.352954+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:37:55.352954+0000","last_undegraded":"2024-02-23T00:37:55.352954+0000","last_fullsized":"2024-02-23T00:37:55.352954+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:39.140434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.140434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":455,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.218879+0000","last_change":"2024-02-23T00:35:01.614406+0000","last_active":"2024-02-23T00:37:55.218879+0000","last_peered":"2024-02-23T00:37:55.218879+0000","last_clean":"2024-02-23T00:37:55.218879+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:37:55.218879+0000","last_undegraded":"2024-02-23T00:37:55.218879+0000","last_fullsized":"2024-02-23T00:37:55.218879+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:01.614340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.614340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":529,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220317+0000","last_change":"2024-02-23T00:35:10.593944+0000","last_active":"2024-02-23T00:37:55.220317+0000","last_peered":"2024-02-23T00:37:55.220317+0000","last_clean":"2024-02-23T00:37:55.220317+0000","last_became_active":"2024-02-23T00:34:43.058886+0000","last_became_peered":"2024-02-23T00:34:43.058886+0000","last_unstale":"2024-02-23T00:37:55.220317+0000","last_undegraded":"2024-02-23T00:37:55.220317+0000","last_fullsized":"2024-02-23T00:37:55.220317+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-02-23T00:35:10.593862+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.593862+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":457,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:56.076500+0000","last_change":"2024-02-23T00:36:37.126232+0000","last_active":"2024-02-23T00:37:56.076500+0000","last_peered":"2024-02-23T00:37:56.076500+0000","last_clean":"2024-02-23T00:36:33.365949+0000","last_became_active":"2024-02-23T00:36:37.126232+0000","last_became_peered":"2024-02-23T00:36:37.126232+0000","last_unstale":"2024-02-23T00:37:56.076500+0000","last_undegraded":"2024-02-23T00:36:37.112806+0000","last_fullsized":"2024-02-23T00:36:37.112690+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":431,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077028+0000","last_change":"2024-02-23T00:34:22.475022+0000","last_active":"2024-02-23T00:37:56.077028+0000","last_peered":"2024-02-23T00:37:56.077028+0000","last_clean":"2024-02-23T00:37:56.077028+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:37:56.077028+0000","last_undegraded":"2024-02-23T00:37:56.077028+0000","last_fullsized":"2024-02-23T00:37:56.077028+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.474956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.474956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":164,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280904+0000","last_change":"2024-02-23T00:37:13.405875+0000","last_active":"2024-02-23T00:37:55.280904+0000","last_peered":"2024-02-23T00:37:55.280904+0000","last_clean":"2024-02-23T00:37:55.280904+0000","last_became_active":"2024-02-23T00:34:41.116969+0000","last_became_peered":"2024-02-23T00:34:41.116969+0000","last_unstale":"2024-02-23T00:37:55.280904+0000","last_undegraded":"2024-02-23T00:37:55.280904+0000","last_fullsized":"2024-02-23T00:37:55.280904+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:13.405802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:13.405802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":398,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352933+0000","last_change":"2024-02-23T00:37:21.365047+0000","last_active":"2024-02-23T00:37:55.352933+0000","last_peered":"2024-02-23T00:37:55.352933+0000","last_clean":"2024-02-23T00:37:55.352933+0000","last_became_active":"2024-02-23T00:34:41.109539+0000","last_became_peered":"2024-02-23T00:34:41.109539+0000","last_unstale":"2024-02-23T00:37:55.352933+0000","last_undegraded":"2024-02-23T00:37:55.352933+0000","last_fullsized":"2024-02-23T00:37:55.352933+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_clean_scrub_stamp":"2024-02-23T00:37:21.364934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":56,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.353159+0000","last_change":"2024-02-23T00:36:37.109542+0000","last_active":"2024-02-23T00:37:55.353159+0000","last_peered":"2024-02-23T00:37:55.353159+0000","last_clean":"2024-02-23T00:31:27.149861+0000","last_became_active":"2024-02-23T00:36:37.109542+0000","last_became_peered":"2024-02-23T00:36:37.109542+0000","last_unstale":"2024-02-23T00:37:55.353159+0000","last_undegraded":"2024-02-23T00:37:55.353159+0000","last_fullsized":"2024-02-23T00:36:37.096846+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:42.114807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:35:42.114807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":201,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.351989+0000","last_change":"2024-02-23T00:36:37.114738+0000","last_active":"2024-02-23T00:37:55.351989+0000","last_peered":"2024-02-23T00:37:55.351989+0000","last_clean":"2024-02-23T00:36:33.365051+0000","last_became_active":"2024-02-23T00:36:37.114738+0000","last_became_peered":"2024-02-23T00:36:37.114738+0000","last_unstale":"2024-02-23T00:37:55.351989+0000","last_undegraded":"2024-02-23T00:37:55.351989+0000","last_fullsized":"2024-02-23T00:36:37.094941+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":501,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.219294+0000","last_change":"2024-02-23T00:37:08.107220+0000","last_active":"2024-02-23T00:37:55.219294+0000","last_peered":"2024-02-23T00:37:55.219294+0000","last_clean":"2024-02-23T00:37:55.219294+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:37:55.219294+0000","last_undegraded":"2024-02-23T00:37:55.219294+0000","last_fullsized":"2024-02-23T00:37:55.219294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:37:08.107110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.107110+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":446,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.281323+0000","last_change":"2024-02-23T00:37:28.355537+0000","last_active":"2024-02-23T00:37:55.281323+0000","last_peered":"2024-02-23T00:37:55.281323+0000","last_clean":"2024-02-23T00:37:55.281323+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:37:55.281323+0000","last_undegraded":"2024-02-23T00:37:55.281323+0000","last_fullsized":"2024-02-23T00:37:55.281323+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:28.355449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:28.355449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":443,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077788+0000","last_change":"2024-02-23T00:36:37.952567+0000","last_active":"2024-02-23T00:37:56.077788+0000","last_peered":"2024-02-23T00:37:56.077788+0000","last_clean":"2024-02-23T00:37:56.077788+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:37:56.077788+0000","last_undegraded":"2024-02-23T00:37:56.077788+0000","last_fullsized":"2024-02-23T00:37:56.077788+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:37.952474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:37.952474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":399,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.353268+0000","last_change":"2024-02-23T00:36:37.116207+0000","last_active":"2024-02-23T00:37:55.353268+0000","last_peered":"2024-02-23T00:37:55.353268+0000","last_clean":"2024-02-23T00:36:33.365474+0000","last_became_active":"2024-02-23T00:36:37.116207+0000","last_became_peered":"2024-02-23T00:36:37.116207+0000","last_unstale":"2024-02-23T00:37:55.353268+0000","last_undegraded":"2024-02-23T00:37:55.353268+0000","last_fullsized":"2024-02-23T00:36:37.098927+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.019530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.019530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":267,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.076917+0000","last_change":"2024-02-23T00:36:37.119099+0000","last_active":"2024-02-23T00:37:56.076917+0000","last_peered":"2024-02-23T00:37:56.076917+0000","last_clean":"2024-02-23T00:36:33.366787+0000","last_became_active":"2024-02-23T00:36:37.119099+0000","last_became_peered":"2024-02-23T00:36:37.119099+0000","last_unstale":"2024-02-23T00:37:56.076917+0000","last_undegraded":"2024-02-23T00:37:56.076917+0000","last_fullsized":"2024-02-23T00:36:37.111585+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.924808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.924808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":466,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.220155+0000","last_change":"2024-02-23T00:36:37.097579+0000","last_active":"2024-02-23T00:37:55.220155+0000","last_peered":"2024-02-23T00:37:55.220155+0000","last_clean":"2024-02-23T00:36:33.191735+0000","last_became_active":"2024-02-23T00:36:37.097579+0000","last_became_peered":"2024-02-23T00:36:37.097579+0000","last_unstale":"2024-02-23T00:37:55.220155+0000","last_undegraded":"2024-02-23T00:37:55.220155+0000","last_fullsized":"2024-02-23T00:36:37.094016+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.588132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.588132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":282,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:56.077151+0000","last_change":"2024-02-23T00:36:37.119086+0000","last_active":"2024-02-23T00:37:56.077151+0000","last_peered":"2024-02-23T00:37:56.077151+0000","last_clean":"2024-02-23T00:36:33.366171+0000","last_became_active":"2024-02-23T00:36:37.119086+0000","last_became_peered":"2024-02-23T00:36:37.119086+0000","last_unstale":"2024-02-23T00:37:56.077151+0000","last_undegraded":"2024-02-23T00:36:37.101188+0000","last_fullsized":"2024-02-23T00:36:37.101097+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:35:00.621157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.621157+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":437,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.353184+0000","last_change":"2024-02-23T00:33:34.199948+0000","last_active":"2024-02-23T00:37:55.353184+0000","last_peered":"2024-02-23T00:37:55.353184+0000","last_clean":"2024-02-23T00:37:55.353184+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:37:55.353184+0000","last_undegraded":"2024-02-23T00:37:55.353184+0000","last_fullsized":"2024-02-23T00:37:55.353184+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:34.199869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:34.199869+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":454,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.082077+0000","last_change":"2024-02-23T00:38:00.082077+0000","last_active":"2024-02-23T00:38:00.082077+0000","last_peered":"2024-02-23T00:38:00.082077+0000","last_clean":"2024-02-23T00:38:00.082077+0000","last_became_active":"2024-02-23T00:34:41.094428+0000","last_became_peered":"2024-02-23T00:34:41.094428+0000","last_unstale":"2024-02-23T00:38:00.082077+0000","last_undegraded":"2024-02-23T00:38:00.082077+0000","last_fullsized":"2024-02-23T00:38:00.082077+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.082000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.082000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":165,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.977387+0000","last_change":"2024-02-23T00:38:00.977387+0000","last_active":"2024-02-23T00:38:00.977387+0000","last_peered":"2024-02-23T00:38:00.977387+0000","last_clean":"2024-02-23T00:38:00.977387+0000","last_became_active":"2024-02-23T00:34:41.105998+0000","last_became_peered":"2024-02-23T00:34:41.105998+0000","last_unstale":"2024-02-23T00:38:00.977387+0000","last_undegraded":"2024-02-23T00:38:00.977387+0000","last_fullsized":"2024-02-23T00:38:00.977387+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.977282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.977282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":705,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352249+0000","last_change":"2024-02-23T00:37:17.499775+0000","last_active":"2024-02-23T00:37:55.352249+0000","last_peered":"2024-02-23T00:37:55.352249+0000","last_clean":"2024-02-23T00:37:55.352249+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:37:55.352249+0000","last_undegraded":"2024-02-23T00:37:55.352249+0000","last_fullsized":"2024-02-23T00:37:55.352249+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:17.499694+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.499694+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":458,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.280562+0000","last_change":"2024-02-23T00:36:37.108691+0000","last_active":"2024-02-23T00:37:55.280562+0000","last_peered":"2024-02-23T00:37:55.280562+0000","last_clean":"2024-02-23T00:36:33.369928+0000","last_became_active":"2024-02-23T00:36:37.108691+0000","last_became_peered":"2024-02-23T00:36:37.108691+0000","last_unstale":"2024-02-23T00:37:55.280562+0000","last_undegraded":"2024-02-23T00:37:55.280562+0000","last_fullsized":"2024-02-23T00:36:37.100335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.196211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.196211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":134,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:56.075709+0000","last_change":"2024-02-23T00:36:37.123994+0000","last_active":"2024-02-23T00:37:56.075709+0000","last_peered":"2024-02-23T00:37:56.075709+0000","last_clean":"2024-02-23T00:27:29.216066+0000","last_became_active":"2024-02-23T00:36:37.123994+0000","last_became_peered":"2024-02-23T00:36:37.123994+0000","last_unstale":"2024-02-23T00:37:56.075709+0000","last_undegraded":"2024-02-23T00:36:37.103439+0000","last_fullsized":"2024-02-23T00:36:37.103335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:35:01.396982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.396982+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":450,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.078236+0000","last_change":"2024-02-23T00:36:37.120094+0000","last_active":"2024-02-23T00:37:56.078236+0000","last_peered":"2024-02-23T00:37:56.078236+0000","last_clean":"2024-02-23T00:36:33.367908+0000","last_became_active":"2024-02-23T00:36:37.120094+0000","last_became_peered":"2024-02-23T00:36:37.120094+0000","last_unstale":"2024-02-23T00:37:56.078236+0000","last_undegraded":"2024-02-23T00:37:56.078236+0000","last_fullsized":"2024-02-23T00:36:37.107601+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":436,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.351386+0000","last_change":"2024-02-23T00:37:26.268389+0000","last_active":"2024-02-23T00:37:55.351386+0000","last_peered":"2024-02-23T00:37:55.351386+0000","last_clean":"2024-02-23T00:37:55.351386+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:37:55.351386+0000","last_undegraded":"2024-02-23T00:37:55.351386+0000","last_fullsized":"2024-02-23T00:37:55.351386+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.268282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":384,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352100+0000","last_change":"2024-02-23T00:35:00.832892+0000","last_active":"2024-02-23T00:37:55.352100+0000","last_peered":"2024-02-23T00:37:55.352100+0000","last_clean":"2024-02-23T00:37:55.352100+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:37:55.352100+0000","last_undegraded":"2024-02-23T00:37:55.352100+0000","last_fullsized":"2024-02-23T00:37:55.352100+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.832818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.832818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":446,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.077116+0000","last_change":"2024-02-23T00:36:37.117616+0000","last_active":"2024-02-23T00:37:56.077116+0000","last_peered":"2024-02-23T00:37:56.077116+0000","last_clean":"2024-02-23T00:36:33.368573+0000","last_became_active":"2024-02-23T00:36:37.117616+0000","last_became_peered":"2024-02-23T00:36:37.117616+0000","last_unstale":"2024-02-23T00:37:56.077116+0000","last_undegraded":"2024-02-23T00:37:56.077116+0000","last_fullsized":"2024-02-23T00:36:37.100549+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":112,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.353277+0000","last_change":"2024-02-23T00:36:37.113785+0000","last_active":"2024-02-23T00:37:55.353277+0000","last_peered":"2024-02-23T00:37:55.353277+0000","last_clean":"2024-02-23T00:27:24.365389+0000","last_became_active":"2024-02-23T00:36:37.113785+0000","last_became_peered":"2024-02-23T00:36:37.113785+0000","last_unstale":"2024-02-23T00:37:55.353277+0000","last_undegraded":"2024-02-23T00:36:37.097517+0000","last_fullsized":"2024-02-23T00:36:37.097457+0000","mapping_epoch":425,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":85,"num_read_kb":71,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.074869+0000","last_change":"2024-02-23T00:36:37.126102+0000","last_active":"2024-02-23T00:37:56.074869+0000","last_peered":"2024-02-23T00:37:56.074869+0000","last_clean":"2024-02-23T00:36:33.365781+0000","last_became_active":"2024-02-23T00:36:37.126102+0000","last_became_peered":"2024-02-23T00:36:37.126102+0000","last_unstale":"2024-02-23T00:37:56.074869+0000","last_undegraded":"2024-02-23T00:37:56.074869+0000","last_fullsized":"2024-02-23T00:36:37.112617+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.980396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.980396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":368,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.219439+0000","last_change":"2024-02-23T00:36:37.111324+0000","last_active":"2024-02-23T00:37:55.219439+0000","last_peered":"2024-02-23T00:37:55.219439+0000","last_clean":"2024-02-23T00:34:39.448365+0000","last_became_active":"2024-02-23T00:36:37.111324+0000","last_became_peered":"2024-02-23T00:36:37.111324+0000","last_unstale":"2024-02-23T00:37:55.219439+0000","last_undegraded":"2024-02-23T00:37:55.219439+0000","last_fullsized":"2024-02-23T00:36:37.093794+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:31.593354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:31.593354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":440,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352624+0000","last_change":"2024-02-23T00:35:02.744199+0000","last_active":"2024-02-23T00:37:55.352624+0000","last_peered":"2024-02-23T00:37:55.352624+0000","last_clean":"2024-02-23T00:37:55.352624+0000","last_became_active":"2024-02-23T00:34:41.118378+0000","last_became_peered":"2024-02-23T00:34:41.118378+0000","last_unstale":"2024-02-23T00:37:55.352624+0000","last_undegraded":"2024-02-23T00:37:55.352624+0000","last_fullsized":"2024-02-23T00:37:55.352624+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.744088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.744088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":393,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.219991+0000","last_change":"2024-02-23T00:35:55.223009+0000","last_active":"2024-02-23T00:37:55.219991+0000","last_peered":"2024-02-23T00:37:55.219991+0000","last_clean":"2024-02-23T00:37:55.219991+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:37:55.219991+0000","last_undegraded":"2024-02-23T00:37:55.219991+0000","last_fullsized":"2024-02-23T00:37:55.219991+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.222912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.222912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":454,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.078317+0000","last_change":"2024-02-23T00:36:41.937397+0000","last_active":"2024-02-23T00:37:56.078317+0000","last_peered":"2024-02-23T00:37:56.078317+0000","last_clean":"2024-02-23T00:37:56.078317+0000","last_became_active":"2024-02-23T00:35:27.093541+0000","last_became_peered":"2024-02-23T00:35:27.093541+0000","last_unstale":"2024-02-23T00:37:56.078317+0000","last_undegraded":"2024-02-23T00:37:56.078317+0000","last_fullsized":"2024-02-23T00:37:56.078317+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.937344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.937344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":490,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:03.327200+0000","last_change":"2024-02-23T00:38:03.327200+0000","last_active":"2024-02-23T00:38:03.327200+0000","last_peered":"2024-02-23T00:38:03.327200+0000","last_clean":"2024-02-23T00:38:03.327200+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:38:03.327200+0000","last_undegraded":"2024-02-23T00:38:03.327200+0000","last_fullsized":"2024-02-23T00:38:03.327200+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:03.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.327127+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":453,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.076350+0000","last_change":"2024-02-23T00:36:37.106377+0000","last_active":"2024-02-23T00:37:56.076350+0000","last_peered":"2024-02-23T00:37:56.076350+0000","last_clean":"2024-02-23T00:36:33.365867+0000","last_became_active":"2024-02-23T00:36:37.106377+0000","last_became_peered":"2024-02-23T00:36:37.106377+0000","last_unstale":"2024-02-23T00:37:56.076350+0000","last_undegraded":"2024-02-23T00:37:56.076350+0000","last_fullsized":"2024-02-23T00:36:37.100069+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":440,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075993+0000","last_change":"2024-02-23T00:36:33.160039+0000","last_active":"2024-02-23T00:37:56.075993+0000","last_peered":"2024-02-23T00:37:56.075993+0000","last_clean":"2024-02-23T00:37:56.075993+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:37:56.075993+0000","last_undegraded":"2024-02-23T00:37:56.075993+0000","last_fullsized":"2024-02-23T00:37:56.075993+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.159941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.159941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":257,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.076335+0000","last_change":"2024-02-23T00:37:26.933362+0000","last_active":"2024-02-23T00:37:56.076335+0000","last_peered":"2024-02-23T00:37:56.076335+0000","last_clean":"2024-02-23T00:37:56.076335+0000","last_became_active":"2024-02-23T00:34:41.099598+0000","last_became_peered":"2024-02-23T00:34:41.099598+0000","last_unstale":"2024-02-23T00:37:56.076335+0000","last_undegraded":"2024-02-23T00:37:56.076335+0000","last_fullsized":"2024-02-23T00:37:56.076335+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.933288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.933288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":443,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077248+0000","last_change":"2024-02-23T00:37:20.008803+0000","last_active":"2024-02-23T00:37:56.077248+0000","last_peered":"2024-02-23T00:37:56.077248+0000","last_clean":"2024-02-23T00:37:56.077248+0000","last_became_active":"2024-02-23T00:34:41.114270+0000","last_became_peered":"2024-02-23T00:34:41.114270+0000","last_unstale":"2024-02-23T00:37:56.077248+0000","last_undegraded":"2024-02-23T00:37:56.077248+0000","last_fullsized":"2024-02-23T00:37:56.077248+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.008726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":395,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220457+0000","last_change":"2024-02-23T00:35:52.175358+0000","last_active":"2024-02-23T00:37:55.220457+0000","last_peered":"2024-02-23T00:37:55.220457+0000","last_clean":"2024-02-23T00:37:55.220457+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:37:55.220457+0000","last_undegraded":"2024-02-23T00:37:55.220457+0000","last_fullsized":"2024-02-23T00:37:55.220457+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.175246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.175246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":462,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:37:59.003696+0000","last_change":"2024-02-23T00:37:59.003696+0000","last_active":"2024-02-23T00:37:59.003696+0000","last_peered":"2024-02-23T00:37:59.003696+0000","last_clean":"2024-02-23T00:37:59.003696+0000","last_became_active":"2024-02-23T00:35:27.097186+0000","last_became_peered":"2024-02-23T00:35:27.097186+0000","last_unstale":"2024-02-23T00:37:59.003696+0000","last_undegraded":"2024-02-23T00:37:59.003696+0000","last_fullsized":"2024-02-23T00:37:59.003696+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.003583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.003583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":521,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.353426+0000","last_change":"2024-02-23T00:36:37.108623+0000","last_active":"2024-02-23T00:37:55.353426+0000","last_peered":"2024-02-23T00:37:55.353426+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:36:37.108623+0000","last_became_peered":"2024-02-23T00:36:37.108623+0000","last_unstale":"2024-02-23T00:37:55.353426+0000","last_undegraded":"2024-02-23T00:36:37.097062+0000","last_fullsized":"2024-02-23T00:36:37.096949+0000","mapping_epoch":425,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":467,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.221229+0000","last_change":"2024-02-23T00:36:37.119168+0000","last_active":"2024-02-23T00:37:55.221229+0000","last_peered":"2024-02-23T00:37:55.221229+0000","last_clean":"2024-02-23T00:36:33.193740+0000","last_became_active":"2024-02-23T00:36:37.119168+0000","last_became_peered":"2024-02-23T00:36:37.119168+0000","last_unstale":"2024-02-23T00:37:55.221229+0000","last_undegraded":"2024-02-23T00:37:55.221229+0000","last_fullsized":"2024-02-23T00:36:37.101741+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.184937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.184937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":698,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352194+0000","last_change":"2024-02-23T00:37:13.467508+0000","last_active":"2024-02-23T00:37:55.352194+0000","last_peered":"2024-02-23T00:37:55.352194+0000","last_clean":"2024-02-23T00:37:55.352194+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:37:55.352194+0000","last_undegraded":"2024-02-23T00:37:55.352194+0000","last_fullsized":"2024-02-23T00:37:55.352194+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:13.467428+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:13.467428+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":452,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.078155+0000","last_change":"2024-02-23T00:37:12.023243+0000","last_active":"2024-02-23T00:37:56.078155+0000","last_peered":"2024-02-23T00:37:56.078155+0000","last_clean":"2024-02-23T00:37:56.078155+0000","last_became_active":"2024-02-23T00:34:41.105566+0000","last_became_peered":"2024-02-23T00:34:41.105566+0000","last_unstale":"2024-02-23T00:37:56.078155+0000","last_undegraded":"2024-02-23T00:37:56.078155+0000","last_fullsized":"2024-02-23T00:37:56.078155+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:12.023125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:12.023125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":399,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.222492+0000","last_change":"2024-02-23T00:37:09.743047+0000","last_active":"2024-02-23T00:37:55.222492+0000","last_peered":"2024-02-23T00:37:55.222492+0000","last_clean":"2024-02-23T00:37:55.222492+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:37:55.222492+0000","last_undegraded":"2024-02-23T00:37:55.222492+0000","last_fullsized":"2024-02-23T00:37:55.222492+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.742938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.742938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":461,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.219804+0000","last_change":"2024-02-23T00:36:37.101373+0000","last_active":"2024-02-23T00:37:55.219804+0000","last_peered":"2024-02-23T00:37:55.219804+0000","last_clean":"2024-02-23T00:36:33.192308+0000","last_became_active":"2024-02-23T00:36:37.101373+0000","last_became_peered":"2024-02-23T00:36:37.101373+0000","last_unstale":"2024-02-23T00:37:55.219804+0000","last_undegraded":"2024-02-23T00:37:55.219804+0000","last_fullsized":"2024-02-23T00:36:37.094754+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.595447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.595447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":401,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.353221+0000","last_change":"2024-02-23T00:36:37.109549+0000","last_active":"2024-02-23T00:37:55.353221+0000","last_peered":"2024-02-23T00:37:55.353221+0000","last_clean":"2024-02-23T00:35:25.318841+0000","last_became_active":"2024-02-23T00:36:37.109549+0000","last_became_peered":"2024-02-23T00:36:37.109549+0000","last_unstale":"2024-02-23T00:37:55.353221+0000","last_undegraded":"2024-02-23T00:37:55.353221+0000","last_fullsized":"2024-02-23T00:36:37.096473+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"453'47","reported_seq":684,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.459796+0000","last_change":"2024-02-23T00:37:15.422544+0000","last_active":"2024-02-23T00:38:00.459796+0000","last_peered":"2024-02-23T00:38:00.459796+0000","last_clean":"2024-02-23T00:38:00.459796+0000","last_became_active":"2024-02-23T00:34:41.099241+0000","last_became_peered":"2024-02-23T00:34:41.099241+0000","last_unstale":"2024-02-23T00:38:00.459796+0000","last_undegraded":"2024-02-23T00:38:00.459796+0000","last_fullsized":"2024-02-23T00:38:00.459796+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'35","last_scrub_stamp":"2024-02-23T00:37:15.422435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:15.422435+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":80,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352560+0000","last_change":"2024-02-23T00:37:23.360353+0000","last_active":"2024-02-23T00:37:55.352560+0000","last_peered":"2024-02-23T00:37:55.352560+0000","last_clean":"2024-02-23T00:37:55.352560+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:37:55.352560+0000","last_undegraded":"2024-02-23T00:37:55.352560+0000","last_fullsized":"2024-02-23T00:37:55.352560+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:37:23.360230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:23.360230+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":159,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220810+0000","last_change":"2024-02-23T00:36:32.098355+0000","last_active":"2024-02-23T00:37:55.220810+0000","last_peered":"2024-02-23T00:37:55.220810+0000","last_clean":"2024-02-23T00:37:55.220810+0000","last_became_active":"2024-02-23T00:34:41.109972+0000","last_became_peered":"2024-02-23T00:34:41.109972+0000","last_unstale":"2024-02-23T00:37:55.220810+0000","last_undegraded":"2024-02-23T00:37:55.220810+0000","last_fullsized":"2024-02-23T00:37:55.220810+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.098280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.098280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":399,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.223345+0000","last_change":"2024-02-23T00:37:11.766364+0000","last_active":"2024-02-23T00:37:55.223345+0000","last_peered":"2024-02-23T00:37:55.223345+0000","last_clean":"2024-02-23T00:37:55.223345+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:37:55.223345+0000","last_undegraded":"2024-02-23T00:37:55.223345+0000","last_fullsized":"2024-02-23T00:37:55.223345+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:11.766284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:11.766284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":359,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075149+0000","last_change":"2024-02-23T00:36:23.210776+0000","last_active":"2024-02-23T00:37:56.075149+0000","last_peered":"2024-02-23T00:37:56.075149+0000","last_clean":"2024-02-23T00:37:56.075149+0000","last_became_active":"2024-02-23T00:34:41.109182+0000","last_became_peered":"2024-02-23T00:34:41.109182+0000","last_unstale":"2024-02-23T00:37:56.075149+0000","last_undegraded":"2024-02-23T00:37:56.075149+0000","last_fullsized":"2024-02-23T00:37:56.075149+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:23.210706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.210706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":76,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.280178+0000","last_change":"2024-02-23T00:36:37.104760+0000","last_active":"2024-02-23T00:37:55.280178+0000","last_peered":"2024-02-23T00:37:55.280178+0000","last_clean":"2024-02-23T00:27:24.172824+0000","last_became_active":"2024-02-23T00:36:37.104760+0000","last_became_peered":"2024-02-23T00:36:37.104760+0000","last_unstale":"2024-02-23T00:37:55.280178+0000","last_undegraded":"2024-02-23T00:37:55.280178+0000","last_fullsized":"2024-02-23T00:36:37.098375+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":113,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:56.076511+0000","last_change":"2024-02-23T00:36:37.110364+0000","last_active":"2024-02-23T00:37:56.076511+0000","last_peered":"2024-02-23T00:37:56.076511+0000","last_clean":"2024-02-23T00:27:29.173100+0000","last_became_active":"2024-02-23T00:36:37.110364+0000","last_became_peered":"2024-02-23T00:36:37.110364+0000","last_unstale":"2024-02-23T00:37:56.076511+0000","last_undegraded":"2024-02-23T00:36:37.096575+0000","last_fullsized":"2024-02-23T00:36:37.096459+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:35:44.214727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:44.214727+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":446,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.352361+0000","last_change":"2024-02-23T00:36:37.123256+0000","last_active":"2024-02-23T00:37:55.352361+0000","last_peered":"2024-02-23T00:37:55.352361+0000","last_clean":"2024-02-23T00:36:33.366566+0000","last_became_active":"2024-02-23T00:36:37.123256+0000","last_became_peered":"2024-02-23T00:36:37.123256+0000","last_unstale":"2024-02-23T00:37:55.352361+0000","last_undegraded":"2024-02-23T00:37:55.352361+0000","last_fullsized":"2024-02-23T00:36:37.096440+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":354,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.221165+0000","last_change":"2024-02-23T00:36:37.113860+0000","last_active":"2024-02-23T00:37:55.221165+0000","last_peered":"2024-02-23T00:37:55.221165+0000","last_clean":"2024-02-23T00:34:39.448366+0000","last_became_active":"2024-02-23T00:36:37.113860+0000","last_became_peered":"2024-02-23T00:36:37.113860+0000","last_unstale":"2024-02-23T00:37:55.221165+0000","last_undegraded":"2024-02-23T00:37:55.221165+0000","last_fullsized":"2024-02-23T00:36:37.101768+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.488959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.488959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":393,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.219737+0000","last_change":"2024-02-23T00:36:42.093272+0000","last_active":"2024-02-23T00:37:55.219737+0000","last_peered":"2024-02-23T00:37:55.219737+0000","last_clean":"2024-02-23T00:37:55.219737+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:37:55.219737+0000","last_undegraded":"2024-02-23T00:37:55.219737+0000","last_fullsized":"2024-02-23T00:37:55.219737+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:36:42.093181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":458,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.221014+0000","last_change":"2024-02-23T00:36:37.124382+0000","last_active":"2024-02-23T00:37:55.221014+0000","last_peered":"2024-02-23T00:37:55.221014+0000","last_clean":"2024-02-23T00:36:33.193431+0000","last_became_active":"2024-02-23T00:36:37.124382+0000","last_became_peered":"2024-02-23T00:36:37.124382+0000","last_unstale":"2024-02-23T00:37:55.221014+0000","last_undegraded":"2024-02-23T00:37:55.221014+0000","last_fullsized":"2024-02-23T00:36:37.101474+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:24.016837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:24.016837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":613,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.219466+0000","last_change":"2024-02-23T00:35:09.098161+0000","last_active":"2024-02-23T00:37:55.219466+0000","last_peered":"2024-02-23T00:37:55.219466+0000","last_clean":"2024-02-23T00:37:55.219466+0000","last_became_active":"2024-02-23T00:35:09.097721+0000","last_became_peered":"2024-02-23T00:35:09.097721+0000","last_unstale":"2024-02-23T00:37:55.219466+0000","last_undegraded":"2024-02-23T00:37:55.219466+0000","last_fullsized":"2024-02-23T00:37:55.219466+0000","mapping_epoch":349,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:34:54.691537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.691537+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":150,"num_read_kb":126,"num_write":84,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":465,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.220725+0000","last_change":"2024-02-23T00:36:37.118733+0000","last_active":"2024-02-23T00:37:55.220725+0000","last_peered":"2024-02-23T00:37:55.220725+0000","last_clean":"2024-02-23T00:36:36.130983+0000","last_became_active":"2024-02-23T00:36:37.118733+0000","last_became_peered":"2024-02-23T00:36:37.118733+0000","last_unstale":"2024-02-23T00:37:55.220725+0000","last_undegraded":"2024-02-23T00:36:37.095672+0000","last_fullsized":"2024-02-23T00:36:37.095564+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:36:36.130868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.130868+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":159,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077413+0000","last_change":"2024-02-23T00:35:48.866316+0000","last_active":"2024-02-23T00:37:56.077413+0000","last_peered":"2024-02-23T00:37:56.077413+0000","last_clean":"2024-02-23T00:37:56.077413+0000","last_became_active":"2024-02-23T00:34:41.110345+0000","last_became_peered":"2024-02-23T00:34:41.110345+0000","last_unstale":"2024-02-23T00:37:56.077413+0000","last_undegraded":"2024-02-23T00:37:56.077413+0000","last_fullsized":"2024-02-23T00:37:56.077413+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:48.866230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:48.866230+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":389,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.076678+0000","last_change":"2024-02-23T00:36:28.184266+0000","last_active":"2024-02-23T00:37:56.076678+0000","last_peered":"2024-02-23T00:37:56.076678+0000","last_clean":"2024-02-23T00:37:56.076678+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:37:56.076678+0000","last_undegraded":"2024-02-23T00:37:56.076678+0000","last_fullsized":"2024-02-23T00:37:56.076678+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:28.184197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:28.184197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":453,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.221006+0000","last_change":"2024-02-23T00:36:37.111836+0000","last_active":"2024-02-23T00:37:55.221006+0000","last_peered":"2024-02-23T00:37:55.221006+0000","last_clean":"2024-02-23T00:36:33.193391+0000","last_became_active":"2024-02-23T00:36:37.111836+0000","last_became_peered":"2024-02-23T00:36:37.111836+0000","last_unstale":"2024-02-23T00:37:55.221006+0000","last_undegraded":"2024-02-23T00:37:55.221006+0000","last_fullsized":"2024-02-23T00:36:37.101173+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.667517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.667517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":159,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.353578+0000","last_change":"2024-02-23T00:36:37.118505+0000","last_active":"2024-02-23T00:37:55.353578+0000","last_peered":"2024-02-23T00:37:55.353578+0000","last_clean":"2024-02-23T00:36:33.367279+0000","last_became_active":"2024-02-23T00:36:37.118505+0000","last_became_peered":"2024-02-23T00:36:37.118505+0000","last_unstale":"2024-02-23T00:37:55.353578+0000","last_undegraded":"2024-02-23T00:37:55.353578+0000","last_fullsized":"2024-02-23T00:36:37.097249+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.614960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.614960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":451,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.222999+0000","last_change":"2024-02-23T00:37:22.918309+0000","last_active":"2024-02-23T00:37:55.222999+0000","last_peered":"2024-02-23T00:37:55.222999+0000","last_clean":"2024-02-23T00:37:55.222999+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:37:55.222999+0000","last_undegraded":"2024-02-23T00:37:55.222999+0000","last_fullsized":"2024-02-23T00:37:55.222999+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.918248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":157,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.076642+0000","last_change":"2024-02-23T00:34:56.402946+0000","last_active":"2024-02-23T00:37:56.076642+0000","last_peered":"2024-02-23T00:37:56.076642+0000","last_clean":"2024-02-23T00:37:56.076642+0000","last_became_active":"2024-02-23T00:34:41.114943+0000","last_became_peered":"2024-02-23T00:34:41.114943+0000","last_unstale":"2024-02-23T00:37:56.076642+0000","last_undegraded":"2024-02-23T00:37:56.076642+0000","last_fullsized":"2024-02-23T00:37:56.076642+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.402851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.402851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":447,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280223+0000","last_change":"2024-02-23T00:37:08.426793+0000","last_active":"2024-02-23T00:37:55.280223+0000","last_peered":"2024-02-23T00:37:55.280223+0000","last_clean":"2024-02-23T00:37:55.280223+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:37:55.280223+0000","last_undegraded":"2024-02-23T00:37:55.280223+0000","last_fullsized":"2024-02-23T00:37:55.280223+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:08.426716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.426716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":440,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075212+0000","last_change":"2024-02-23T00:35:52.291042+0000","last_active":"2024-02-23T00:37:56.075212+0000","last_peered":"2024-02-23T00:37:56.075212+0000","last_clean":"2024-02-23T00:37:56.075212+0000","last_became_active":"2024-02-23T00:34:41.113722+0000","last_became_peered":"2024-02-23T00:34:41.113722+0000","last_unstale":"2024-02-23T00:37:56.075212+0000","last_undegraded":"2024-02-23T00:37:56.075212+0000","last_fullsized":"2024-02-23T00:37:56.075212+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.290946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.290946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":443,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077952+0000","last_change":"2024-02-23T00:37:16.011330+0000","last_active":"2024-02-23T00:37:56.077952+0000","last_peered":"2024-02-23T00:37:56.077952+0000","last_clean":"2024-02-23T00:37:56.077952+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:37:56.077952+0000","last_undegraded":"2024-02-23T00:37:56.077952+0000","last_fullsized":"2024-02-23T00:37:56.077952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:16.011248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:37:16.011248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":403,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280779+0000","last_change":"2024-02-23T00:37:20.316511+0000","last_active":"2024-02-23T00:37:55.280779+0000","last_peered":"2024-02-23T00:37:55.280779+0000","last_clean":"2024-02-23T00:37:55.280779+0000","last_became_active":"2024-02-23T00:34:41.101132+0000","last_became_peered":"2024-02-23T00:34:41.101132+0000","last_unstale":"2024-02-23T00:37:55.280779+0000","last_undegraded":"2024-02-23T00:37:55.280779+0000","last_fullsized":"2024-02-23T00:37:55.280779+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.316429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.316429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"453'32","reported_seq":242,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352526+0000","last_change":"2024-02-23T00:34:42.571126+0000","last_active":"2024-02-23T00:37:55.352526+0000","last_peered":"2024-02-23T00:37:55.352526+0000","last_clean":"2024-02-23T00:37:55.352526+0000","last_became_active":"2024-02-23T00:34:41.106795+0000","last_became_peered":"2024-02-23T00:34:41.106795+0000","last_unstale":"2024-02-23T00:37:55.352526+0000","last_undegraded":"2024-02-23T00:37:55.352526+0000","last_fullsized":"2024-02-23T00:37:55.352526+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:34:15.571539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.571539+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":455,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.076029+0000","last_change":"2024-02-23T00:36:37.117696+0000","last_active":"2024-02-23T00:37:56.076029+0000","last_peered":"2024-02-23T00:37:56.076029+0000","last_clean":"2024-02-23T00:36:33.366237+0000","last_became_active":"2024-02-23T00:36:37.117696+0000","last_became_peered":"2024-02-23T00:36:37.117696+0000","last_unstale":"2024-02-23T00:37:56.076029+0000","last_undegraded":"2024-02-23T00:37:56.076029+0000","last_fullsized":"2024-02-23T00:36:37.103226+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:46.269798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.269798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"426'10","reported_seq":577,"reported_epoch":473,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:00.460132+0000","last_change":"2024-02-23T00:36:37.113534+0000","last_active":"2024-02-23T00:38:00.460132+0000","last_peered":"2024-02-23T00:38:00.460132+0000","last_clean":"2024-02-23T00:36:35.444798+0000","last_became_active":"2024-02-23T00:36:37.113534+0000","last_became_peered":"2024-02-23T00:36:37.113534+0000","last_unstale":"2024-02-23T00:38:00.460132+0000","last_undegraded":"2024-02-23T00:36:37.102235+0000","last_fullsized":"2024-02-23T00:36:37.102132+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":353,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.075787+0000","last_change":"2024-02-23T00:36:37.120693+0000","last_active":"2024-02-23T00:37:56.075787+0000","last_peered":"2024-02-23T00:37:56.075787+0000","last_clean":"2024-02-23T00:34:39.449328+0000","last_became_active":"2024-02-23T00:36:37.120693+0000","last_became_peered":"2024-02-23T00:36:37.120693+0000","last_unstale":"2024-02-23T00:37:56.075787+0000","last_undegraded":"2024-02-23T00:37:56.075787+0000","last_fullsized":"2024-02-23T00:36:37.107400+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.521760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":389,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.076275+0000","last_change":"2024-02-23T00:37:08.058170+0000","last_active":"2024-02-23T00:37:56.076275+0000","last_peered":"2024-02-23T00:37:56.076275+0000","last_clean":"2024-02-23T00:37:56.076275+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:37:56.076275+0000","last_undegraded":"2024-02-23T00:37:56.076275+0000","last_fullsized":"2024-02-23T00:37:56.076275+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:08.058075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.058075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":499,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075642+0000","last_change":"2024-02-23T00:35:01.515684+0000","last_active":"2024-02-23T00:37:56.075642+0000","last_peered":"2024-02-23T00:37:56.075642+0000","last_clean":"2024-02-23T00:37:56.075642+0000","last_became_active":"2024-02-23T00:34:41.110140+0000","last_became_peered":"2024-02-23T00:34:41.110140+0000","last_unstale":"2024-02-23T00:37:56.075642+0000","last_undegraded":"2024-02-23T00:37:56.075642+0000","last_fullsized":"2024-02-23T00:37:56.075642+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:35:01.515550+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.515550+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":359,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.220108+0000","last_change":"2024-02-23T00:36:37.118542+0000","last_active":"2024-02-23T00:37:55.220108+0000","last_peered":"2024-02-23T00:37:55.220108+0000","last_clean":"2024-02-23T00:34:39.449034+0000","last_became_active":"2024-02-23T00:36:37.118542+0000","last_became_peered":"2024-02-23T00:36:37.118542+0000","last_unstale":"2024-02-23T00:37:55.220108+0000","last_undegraded":"2024-02-23T00:37:55.220108+0000","last_fullsized":"2024-02-23T00:36:37.095362+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.391997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.391997+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"330'10","reported_seq":284,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.460397+0000","last_change":"2024-02-23T00:37:27.870730+0000","last_active":"2024-02-23T00:38:00.460397+0000","last_peered":"2024-02-23T00:38:00.460397+0000","last_clean":"2024-02-23T00:38:00.460397+0000","last_became_active":"2024-02-23T00:34:41.109074+0000","last_became_peered":"2024-02-23T00:34:41.109074+0000","last_unstale":"2024-02-23T00:38:00.460397+0000","last_undegraded":"2024-02-23T00:38:00.460397+0000","last_fullsized":"2024-02-23T00:38:00.460397+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'10","last_scrub_stamp":"2024-02-23T00:37:27.870653+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:37:27.870653+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":445,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:05.095789+0000","last_change":"2024-02-23T00:38:05.095789+0000","last_active":"2024-02-23T00:38:05.095789+0000","last_peered":"2024-02-23T00:38:05.095789+0000","last_clean":"2024-02-23T00:38:05.095789+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:38:05.095789+0000","last_undegraded":"2024-02-23T00:38:05.095789+0000","last_fullsized":"2024-02-23T00:38:05.095789+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:05.095708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:05.095708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":157,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.281501+0000","last_change":"2024-02-23T00:36:35.160313+0000","last_active":"2024-02-23T00:37:55.281501+0000","last_peered":"2024-02-23T00:37:55.281501+0000","last_clean":"2024-02-23T00:37:55.281501+0000","last_became_active":"2024-02-23T00:34:41.091189+0000","last_became_peered":"2024-02-23T00:34:41.091189+0000","last_unstale":"2024-02-23T00:37:55.281501+0000","last_undegraded":"2024-02-23T00:37:55.281501+0000","last_fullsized":"2024-02-23T00:37:55.281501+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:35.160239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:35.160239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":465,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352189+0000","last_change":"2024-02-23T00:37:20.382193+0000","last_active":"2024-02-23T00:37:55.352189+0000","last_peered":"2024-02-23T00:37:55.352189+0000","last_clean":"2024-02-23T00:37:55.352189+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:37:55.352189+0000","last_undegraded":"2024-02-23T00:37:55.352189+0000","last_fullsized":"2024-02-23T00:37:55.352189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:37:20.382108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.382108+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":456,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.281069+0000","last_change":"2024-02-23T00:36:36.180921+0000","last_active":"2024-02-23T00:37:55.281069+0000","last_peered":"2024-02-23T00:37:55.281069+0000","last_clean":"2024-02-23T00:37:55.281069+0000","last_became_active":"2024-02-23T00:34:41.116840+0000","last_became_peered":"2024-02-23T00:34:41.116840+0000","last_unstale":"2024-02-23T00:37:55.281069+0000","last_undegraded":"2024-02-23T00:37:55.281069+0000","last_fullsized":"2024-02-23T00:37:55.281069+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:36.180849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.180849+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":58,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.352977+0000","last_change":"2024-02-23T00:36:37.108983+0000","last_active":"2024-02-23T00:37:55.352977+0000","last_peered":"2024-02-23T00:37:55.352977+0000","last_clean":"2024-02-23T00:27:20.002687+0000","last_became_active":"2024-02-23T00:36:37.108983+0000","last_became_peered":"2024-02-23T00:36:37.108983+0000","last_unstale":"2024-02-23T00:37:55.352977+0000","last_undegraded":"2024-02-23T00:37:55.352977+0000","last_fullsized":"2024-02-23T00:36:37.099047+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.416493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.416493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":55,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.075029+0000","last_change":"2024-02-23T00:36:37.112999+0000","last_active":"2024-02-23T00:37:56.075029+0000","last_peered":"2024-02-23T00:37:56.075029+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.112999+0000","last_became_peered":"2024-02-23T00:36:37.112999+0000","last_unstale":"2024-02-23T00:37:56.075029+0000","last_undegraded":"2024-02-23T00:37:56.075029+0000","last_fullsized":"2024-02-23T00:36:37.102073+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.413991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.413991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":399,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.219518+0000","last_change":"2024-02-23T00:36:37.097581+0000","last_active":"2024-02-23T00:37:55.219518+0000","last_peered":"2024-02-23T00:37:55.219518+0000","last_clean":"2024-02-23T00:36:33.191411+0000","last_became_active":"2024-02-23T00:36:37.097581+0000","last_became_peered":"2024-02-23T00:36:37.097581+0000","last_unstale":"2024-02-23T00:37:55.219518+0000","last_undegraded":"2024-02-23T00:37:55.219518+0000","last_fullsized":"2024-02-23T00:36:37.093707+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.101501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.101501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":496,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220111+0000","last_change":"2024-02-23T00:35:45.171309+0000","last_active":"2024-02-23T00:37:55.220111+0000","last_peered":"2024-02-23T00:37:55.220111+0000","last_clean":"2024-02-23T00:37:55.220111+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:37:55.220111+0000","last_undegraded":"2024-02-23T00:37:55.220111+0000","last_fullsized":"2024-02-23T00:37:55.220111+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'11","last_scrub_stamp":"2024-02-23T00:35:45.171195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:45.171195+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":443,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.353415+0000","last_change":"2024-02-23T00:36:37.108194+0000","last_active":"2024-02-23T00:37:55.353415+0000","last_peered":"2024-02-23T00:37:55.353415+0000","last_clean":"2024-02-23T00:36:33.367482+0000","last_became_active":"2024-02-23T00:36:37.108194+0000","last_became_peered":"2024-02-23T00:36:37.108194+0000","last_unstale":"2024-02-23T00:37:55.353415+0000","last_undegraded":"2024-02-23T00:37:55.353415+0000","last_fullsized":"2024-02-23T00:36:37.096730+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.783089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.783089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":442,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352272+0000","last_change":"2024-02-23T00:34:41.116355+0000","last_active":"2024-02-23T00:37:55.352272+0000","last_peered":"2024-02-23T00:37:55.352272+0000","last_clean":"2024-02-23T00:37:55.352272+0000","last_became_active":"2024-02-23T00:34:41.115862+0000","last_became_peered":"2024-02-23T00:34:41.115862+0000","last_unstale":"2024-02-23T00:37:55.352272+0000","last_undegraded":"2024-02-23T00:37:55.352272+0000","last_fullsized":"2024-02-23T00:37:55.352272+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.151986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":58,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.075391+0000","last_change":"2024-02-23T00:36:37.116289+0000","last_active":"2024-02-23T00:37:56.075391+0000","last_peered":"2024-02-23T00:37:56.075391+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.116289+0000","last_became_peered":"2024-02-23T00:36:37.116289+0000","last_unstale":"2024-02-23T00:37:56.075391+0000","last_undegraded":"2024-02-23T00:37:56.075391+0000","last_fullsized":"2024-02-23T00:36:37.106216+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.620107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":421,"reported_epoch":472,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:37:55.280453+0000","last_change":"2024-02-23T00:36:37.104333+0000","last_active":"2024-02-23T00:37:55.280453+0000","last_peered":"2024-02-23T00:37:55.280453+0000","last_clean":"2024-02-23T00:36:33.365274+0000","last_became_active":"2024-02-23T00:36:37.104333+0000","last_became_peered":"2024-02-23T00:36:37.104333+0000","last_unstale":"2024-02-23T00:37:55.280453+0000","last_undegraded":"2024-02-23T00:36:37.099294+0000","last_fullsized":"2024-02-23T00:36:37.099175+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:35:47.130749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.130749+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":203,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352253+0000","last_change":"2024-02-23T00:36:36.531491+0000","last_active":"2024-02-23T00:37:55.352253+0000","last_peered":"2024-02-23T00:37:55.352253+0000","last_clean":"2024-02-23T00:37:55.352253+0000","last_became_active":"2024-02-23T00:34:41.101219+0000","last_became_peered":"2024-02-23T00:34:41.101219+0000","last_unstale":"2024-02-23T00:37:55.352253+0000","last_undegraded":"2024-02-23T00:37:55.352253+0000","last_fullsized":"2024-02-23T00:37:55.352253+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:36:36.531424+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.531424+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":440,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075405+0000","last_change":"2024-02-23T00:36:32.199878+0000","last_active":"2024-02-23T00:37:56.075405+0000","last_peered":"2024-02-23T00:37:56.075405+0000","last_clean":"2024-02-23T00:37:56.075405+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:37:56.075405+0000","last_undegraded":"2024-02-23T00:37:56.075405+0000","last_fullsized":"2024-02-23T00:37:56.075405+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.199765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.199765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":443,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:02.325317+0000","last_change":"2024-02-23T00:38:02.325317+0000","last_active":"2024-02-23T00:38:02.325317+0000","last_peered":"2024-02-23T00:38:02.325317+0000","last_clean":"2024-02-23T00:38:02.325317+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:38:02.325317+0000","last_undegraded":"2024-02-23T00:38:02.325317+0000","last_fullsized":"2024-02-23T00:38:02.325317+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:02.325254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:02.325254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":443,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.076153+0000","last_change":"2024-02-23T00:37:14.009706+0000","last_active":"2024-02-23T00:37:56.076153+0000","last_peered":"2024-02-23T00:37:56.076153+0000","last_clean":"2024-02-23T00:37:56.076153+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:37:56.076153+0000","last_undegraded":"2024-02-23T00:37:56.076153+0000","last_fullsized":"2024-02-23T00:37:56.076153+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.009614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.009614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":384,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.352336+0000","last_change":"2024-02-23T00:33:44.152089+0000","last_active":"2024-02-23T00:37:55.352336+0000","last_peered":"2024-02-23T00:37:55.352336+0000","last_clean":"2024-02-23T00:37:55.352336+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:37:55.352336+0000","last_undegraded":"2024-02-23T00:37:55.352336+0000","last_fullsized":"2024-02-23T00:37:55.352336+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.152014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.152014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":257,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077371+0000","last_change":"2024-02-23T00:37:26.914783+0000","last_active":"2024-02-23T00:37:56.077371+0000","last_peered":"2024-02-23T00:37:56.077371+0000","last_clean":"2024-02-23T00:37:56.077371+0000","last_became_active":"2024-02-23T00:34:41.114516+0000","last_became_peered":"2024-02-23T00:34:41.114516+0000","last_unstale":"2024-02-23T00:37:56.077371+0000","last_undegraded":"2024-02-23T00:37:56.077371+0000","last_fullsized":"2024-02-23T00:37:56.077371+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:37:26.914698+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.914698+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":446,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.281271+0000","last_change":"2024-02-23T00:37:22.340222+0000","last_active":"2024-02-23T00:37:55.281271+0000","last_peered":"2024-02-23T00:37:55.281271+0000","last_clean":"2024-02-23T00:37:55.281271+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:37:55.281271+0000","last_undegraded":"2024-02-23T00:37:55.281271+0000","last_fullsized":"2024-02-23T00:37:55.281271+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:22.340128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.340128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":440,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077055+0000","last_change":"2024-02-23T00:35:47.857609+0000","last_active":"2024-02-23T00:37:56.077055+0000","last_peered":"2024-02-23T00:37:56.077055+0000","last_clean":"2024-02-23T00:37:56.077055+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:37:56.077055+0000","last_undegraded":"2024-02-23T00:37:56.077055+0000","last_fullsized":"2024-02-23T00:37:56.077055+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.857533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.857533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":57,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.280171+0000","last_change":"2024-02-23T00:36:37.114573+0000","last_active":"2024-02-23T00:37:55.280171+0000","last_peered":"2024-02-23T00:37:55.280171+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.114573+0000","last_became_peered":"2024-02-23T00:36:37.114573+0000","last_unstale":"2024-02-23T00:37:55.280171+0000","last_undegraded":"2024-02-23T00:37:55.280171+0000","last_fullsized":"2024-02-23T00:36:37.098725+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.175171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.175171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":320,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077859+0000","last_change":"2024-02-23T00:36:04.097119+0000","last_active":"2024-02-23T00:37:56.077859+0000","last_peered":"2024-02-23T00:37:56.077859+0000","last_clean":"2024-02-23T00:37:56.077859+0000","last_became_active":"2024-02-23T00:36:04.096804+0000","last_became_peered":"2024-02-23T00:36:04.096804+0000","last_unstale":"2024-02-23T00:37:56.077859+0000","last_undegraded":"2024-02-23T00:37:56.077859+0000","last_fullsized":"2024-02-23T00:37:56.077859+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:53.153821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:53.153821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":444,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.074762+0000","last_change":"2024-02-23T00:34:42.064110+0000","last_active":"2024-02-23T00:37:56.074762+0000","last_peered":"2024-02-23T00:37:56.074762+0000","last_clean":"2024-02-23T00:37:56.074762+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:37:56.074762+0000","last_undegraded":"2024-02-23T00:37:56.074762+0000","last_fullsized":"2024-02-23T00:37:56.074762+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:32.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:32.520027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":584,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.460154+0000","last_change":"2024-02-23T00:36:39.246854+0000","last_active":"2024-02-23T00:38:00.460154+0000","last_peered":"2024-02-23T00:38:00.460154+0000","last_clean":"2024-02-23T00:38:00.460154+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:38:00.460154+0000","last_undegraded":"2024-02-23T00:38:00.460154+0000","last_fullsized":"2024-02-23T00:38:00.460154+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:36:39.246273+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.246273+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":440,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.078470+0000","last_change":"2024-02-23T00:35:09.656923+0000","last_active":"2024-02-23T00:37:56.078470+0000","last_peered":"2024-02-23T00:37:56.078470+0000","last_clean":"2024-02-23T00:37:56.078470+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:37:56.078470+0000","last_undegraded":"2024-02-23T00:37:56.078470+0000","last_fullsized":"2024-02-23T00:37:56.078470+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:35:09.656838+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:28:48.333638+0000","last_clean_scrub_stamp":"2024-02-23T00:35:09.656838+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":407,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.078514+0000","last_change":"2024-02-23T00:36:37.122220+0000","last_active":"2024-02-23T00:37:56.078514+0000","last_peered":"2024-02-23T00:37:56.078514+0000","last_clean":"2024-02-23T00:36:33.368644+0000","last_became_active":"2024-02-23T00:36:37.122220+0000","last_became_peered":"2024-02-23T00:36:37.122220+0000","last_unstale":"2024-02-23T00:37:56.078514+0000","last_undegraded":"2024-02-23T00:37:56.078514+0000","last_fullsized":"2024-02-23T00:36:37.109811+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.028166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.028166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":163,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.280116+0000","last_change":"2024-02-23T00:36:37.114328+0000","last_active":"2024-02-23T00:37:55.280116+0000","last_peered":"2024-02-23T00:37:55.280116+0000","last_clean":"2024-02-23T00:36:33.365066+0000","last_became_active":"2024-02-23T00:36:37.114328+0000","last_became_peered":"2024-02-23T00:36:37.114328+0000","last_unstale":"2024-02-23T00:37:55.280116+0000","last_undegraded":"2024-02-23T00:37:55.280116+0000","last_fullsized":"2024-02-23T00:36:37.098495+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.151731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.076113+0000","last_change":"2024-02-23T00:36:37.122400+0000","last_active":"2024-02-23T00:37:56.076113+0000","last_peered":"2024-02-23T00:37:56.076113+0000","last_clean":"2024-02-23T00:27:20.002687+0000","last_became_active":"2024-02-23T00:36:37.122400+0000","last_became_peered":"2024-02-23T00:36:37.122400+0000","last_unstale":"2024-02-23T00:37:56.076113+0000","last_undegraded":"2024-02-23T00:37:56.076113+0000","last_fullsized":"2024-02-23T00:36:37.111280+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":436,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.353040+0000","last_change":"2024-02-23T00:35:57.658486+0000","last_active":"2024-02-23T00:37:55.353040+0000","last_peered":"2024-02-23T00:37:55.353040+0000","last_clean":"2024-02-23T00:37:55.353040+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:37:55.353040+0000","last_undegraded":"2024-02-23T00:37:55.353040+0000","last_fullsized":"2024-02-23T00:37:55.353040+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.658406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.658406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":392,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.078430+0000","last_change":"2024-02-23T00:35:39.871154+0000","last_active":"2024-02-23T00:37:56.078430+0000","last_peered":"2024-02-23T00:37:56.078430+0000","last_clean":"2024-02-23T00:37:56.078430+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:37:56.078430+0000","last_undegraded":"2024-02-23T00:37:56.078430+0000","last_fullsized":"2024-02-23T00:37:56.078430+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.871058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.871058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":450,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.280712+0000","last_change":"2024-02-23T00:36:25.100710+0000","last_active":"2024-02-23T00:37:55.280712+0000","last_peered":"2024-02-23T00:37:55.280712+0000","last_clean":"2024-02-23T00:37:55.280712+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:37:55.280712+0000","last_undegraded":"2024-02-23T00:37:55.280712+0000","last_fullsized":"2024-02-23T00:37:55.280712+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:25.100627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.100627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":441,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.353515+0000","last_change":"2024-02-23T00:36:37.113234+0000","last_active":"2024-02-23T00:37:55.353515+0000","last_peered":"2024-02-23T00:37:55.353515+0000","last_clean":"2024-02-23T00:36:33.367528+0000","last_became_active":"2024-02-23T00:36:37.113234+0000","last_became_peered":"2024-02-23T00:36:37.113234+0000","last_unstale":"2024-02-23T00:37:55.353515+0000","last_undegraded":"2024-02-23T00:37:55.353515+0000","last_fullsized":"2024-02-23T00:36:37.097125+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.791748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.791748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":435,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.220372+0000","last_change":"2024-02-23T00:34:57.661029+0000","last_active":"2024-02-23T00:37:55.220372+0000","last_peered":"2024-02-23T00:37:55.220372+0000","last_clean":"2024-02-23T00:37:55.220372+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:37:55.220372+0000","last_undegraded":"2024-02-23T00:37:55.220372+0000","last_fullsized":"2024-02-23T00:37:55.220372+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:57.660954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.660954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":316,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.076577+0000","last_change":"2024-02-23T00:36:27.951790+0000","last_active":"2024-02-23T00:37:56.076577+0000","last_peered":"2024-02-23T00:37:56.076577+0000","last_clean":"2024-02-23T00:37:56.076577+0000","last_became_active":"2024-02-23T00:36:17.099716+0000","last_became_peered":"2024-02-23T00:36:17.099716+0000","last_unstale":"2024-02-23T00:37:56.076577+0000","last_undegraded":"2024-02-23T00:37:56.076577+0000","last_fullsized":"2024-02-23T00:37:56.076577+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:27.951690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.951690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":159,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.077615+0000","last_change":"2024-02-23T00:36:35.929592+0000","last_active":"2024-02-23T00:37:56.077615+0000","last_peered":"2024-02-23T00:37:56.077615+0000","last_clean":"2024-02-23T00:37:56.077615+0000","last_became_active":"2024-02-23T00:34:41.103114+0000","last_became_peered":"2024-02-23T00:34:41.103114+0000","last_unstale":"2024-02-23T00:37:56.077615+0000","last_undegraded":"2024-02-23T00:37:56.077615+0000","last_fullsized":"2024-02-23T00:37:56.077615+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:35.929507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:35.929507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":451,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:03.063564+0000","last_change":"2024-02-23T00:38:03.063564+0000","last_active":"2024-02-23T00:38:03.063564+0000","last_peered":"2024-02-23T00:38:03.063564+0000","last_clean":"2024-02-23T00:38:03.063564+0000","last_became_active":"2024-02-23T00:34:41.106439+0000","last_became_peered":"2024-02-23T00:34:41.106439+0000","last_unstale":"2024-02-23T00:38:03.063564+0000","last_undegraded":"2024-02-23T00:38:03.063564+0000","last_fullsized":"2024-02-23T00:38:03.063564+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.063485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.063485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":162,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.281436+0000","last_change":"2024-02-23T00:37:19.329028+0000","last_active":"2024-02-23T00:37:55.281436+0000","last_peered":"2024-02-23T00:37:55.281436+0000","last_clean":"2024-02-23T00:37:55.281436+0000","last_became_active":"2024-02-23T00:34:41.113447+0000","last_became_peered":"2024-02-23T00:34:41.113447+0000","last_unstale":"2024-02-23T00:37:55.281436+0000","last_undegraded":"2024-02-23T00:37:55.281436+0000","last_fullsized":"2024-02-23T00:37:55.281436+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:19.328950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:19.328950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":389,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:03.798894+0000","last_change":"2024-02-23T00:38:03.798894+0000","last_active":"2024-02-23T00:38:03.798894+0000","last_peered":"2024-02-23T00:38:03.798894+0000","last_clean":"2024-02-23T00:38:03.798894+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:38:03.798894+0000","last_undegraded":"2024-02-23T00:38:03.798894+0000","last_fullsized":"2024-02-23T00:38:03.798894+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.798824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.798824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":156,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:37:59.129696+0000","last_change":"2024-02-23T00:37:59.129696+0000","last_active":"2024-02-23T00:37:59.129696+0000","last_peered":"2024-02-23T00:37:59.129696+0000","last_clean":"2024-02-23T00:37:59.129696+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:37:59.129696+0000","last_undegraded":"2024-02-23T00:37:59.129696+0000","last_fullsized":"2024-02-23T00:37:59.129696+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.129604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.129604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":437,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.076100+0000","last_change":"2024-02-23T00:36:37.124006+0000","last_active":"2024-02-23T00:37:56.076100+0000","last_peered":"2024-02-23T00:37:56.076100+0000","last_clean":"2024-02-23T00:36:33.366419+0000","last_became_active":"2024-02-23T00:36:37.124006+0000","last_became_peered":"2024-02-23T00:36:37.124006+0000","last_unstale":"2024-02-23T00:37:56.076100+0000","last_undegraded":"2024-02-23T00:37:56.076100+0000","last_fullsized":"2024-02-23T00:36:37.103625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":55,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.351669+0000","last_change":"2024-02-23T00:36:37.119810+0000","last_active":"2024-02-23T00:37:55.351669+0000","last_peered":"2024-02-23T00:37:55.351669+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.119810+0000","last_became_peered":"2024-02-23T00:36:37.119810+0000","last_unstale":"2024-02-23T00:37:55.351669+0000","last_undegraded":"2024-02-23T00:37:55.351669+0000","last_fullsized":"2024-02-23T00:36:37.098897+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:19.572808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.572808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":397,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.074641+0000","last_change":"2024-02-23T00:37:18.044814+0000","last_active":"2024-02-23T00:37:56.074641+0000","last_peered":"2024-02-23T00:37:56.074641+0000","last_clean":"2024-02-23T00:37:56.074641+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:37:56.074641+0000","last_undegraded":"2024-02-23T00:37:56.074641+0000","last_fullsized":"2024-02-23T00:37:56.074641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:18.044737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:18.044737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":450,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.077993+0000","last_change":"2024-02-23T00:36:37.114846+0000","last_active":"2024-02-23T00:37:56.077993+0000","last_peered":"2024-02-23T00:37:56.077993+0000","last_clean":"2024-02-23T00:36:33.367758+0000","last_became_active":"2024-02-23T00:36:37.114846+0000","last_became_peered":"2024-02-23T00:36:37.114846+0000","last_unstale":"2024-02-23T00:37:56.077993+0000","last_undegraded":"2024-02-23T00:37:56.077993+0000","last_fullsized":"2024-02-23T00:36:37.107709+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":445,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.076061+0000","last_change":"2024-02-23T00:37:19.034254+0000","last_active":"2024-02-23T00:37:56.076061+0000","last_peered":"2024-02-23T00:37:56.076061+0000","last_clean":"2024-02-23T00:37:56.076061+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:37:56.076061+0000","last_undegraded":"2024-02-23T00:37:56.076061+0000","last_fullsized":"2024-02-23T00:37:56.076061+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:19.034136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:37:19.034136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":440,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.075274+0000","last_change":"2024-02-23T00:35:41.260304+0000","last_active":"2024-02-23T00:37:56.075274+0000","last_peered":"2024-02-23T00:37:56.075274+0000","last_clean":"2024-02-23T00:37:56.075274+0000","last_became_active":"2024-02-23T00:34:41.112980+0000","last_became_peered":"2024-02-23T00:34:41.112980+0000","last_unstale":"2024-02-23T00:37:56.075274+0000","last_undegraded":"2024-02-23T00:37:56.075274+0000","last_fullsized":"2024-02-23T00:37:56.075274+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:41.260231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.260231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":399,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.352392+0000","last_change":"2024-02-23T00:36:37.115401+0000","last_active":"2024-02-23T00:37:55.352392+0000","last_peered":"2024-02-23T00:37:55.352392+0000","last_clean":"2024-02-23T00:36:33.365083+0000","last_became_active":"2024-02-23T00:36:37.115401+0000","last_became_peered":"2024-02-23T00:36:37.115401+0000","last_unstale":"2024-02-23T00:37:55.352392+0000","last_undegraded":"2024-02-23T00:37:55.352392+0000","last_fullsized":"2024-02-23T00:36:37.099146+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.192483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.192483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":58,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.075266+0000","last_change":"2024-02-23T00:36:37.123921+0000","last_active":"2024-02-23T00:37:56.075266+0000","last_peered":"2024-02-23T00:37:56.075266+0000","last_clean":"2024-02-23T00:27:15.989319+0000","last_became_active":"2024-02-23T00:36:37.123921+0000","last_became_peered":"2024-02-23T00:36:37.123921+0000","last_unstale":"2024-02-23T00:37:56.075266+0000","last_undegraded":"2024-02-23T00:37:56.075266+0000","last_fullsized":"2024-02-23T00:36:37.106358+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.388834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":98,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:38:00.381029+0000","last_change":"2024-02-23T00:38:00.381029+0000","last_active":"2024-02-23T00:38:00.381029+0000","last_peered":"2024-02-23T00:38:00.381029+0000","last_clean":"2024-02-23T00:38:00.381029+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:38:00.381029+0000","last_undegraded":"2024-02-23T00:38:00.381029+0000","last_fullsized":"2024-02-23T00:38:00.381029+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.380938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.380938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":442,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-02-23T00:37:58.130556+0000","last_change":"2024-02-23T00:37:58.130556+0000","last_active":"2024-02-23T00:37:58.130556+0000","last_peered":"2024-02-23T00:37:58.130556+0000","last_clean":"2024-02-23T00:37:58.130556+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:37:58.130556+0000","last_undegraded":"2024-02-23T00:37:58.130556+0000","last_fullsized":"2024-02-23T00:37:58.130556+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:58.130479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:37:58.130479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":57,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:55.221524+0000","last_change":"2024-02-23T00:36:37.119114+0000","last_active":"2024-02-23T00:37:55.221524+0000","last_peered":"2024-02-23T00:37:55.221524+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119114+0000","last_became_peered":"2024-02-23T00:36:37.119114+0000","last_unstale":"2024-02-23T00:37:55.221524+0000","last_undegraded":"2024-02-23T00:37:55.221524+0000","last_fullsized":"2024-02-23T00:36:37.106746+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.176441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.176441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":55,"reported_epoch":472,"state":"active+undersized","last_fresh":"2024-02-23T00:37:56.077494+0000","last_change":"2024-02-23T00:36:37.120482+0000","last_active":"2024-02-23T00:37:56.077494+0000","last_peered":"2024-02-23T00:37:56.077494+0000","last_clean":"2024-02-23T00:27:15.989319+0000","last_became_active":"2024-02-23T00:36:37.120482+0000","last_became_peered":"2024-02-23T00:36:37.120482+0000","last_unstale":"2024-02-23T00:37:56.077494+0000","last_undegraded":"2024-02-23T00:37:56.077494+0000","last_fullsized":"2024-02-23T00:36:37.111071+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":442,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.219294+0000","last_change":"2024-02-23T00:37:14.200716+0000","last_active":"2024-02-23T00:37:55.219294+0000","last_peered":"2024-02-23T00:37:55.219294+0000","last_clean":"2024-02-23T00:37:55.219294+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:37:55.219294+0000","last_undegraded":"2024-02-23T00:37:55.219294+0000","last_fullsized":"2024-02-23T00:37:55.219294+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.200606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.200606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":164,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:56.076967+0000","last_change":"2024-02-23T00:37:24.990742+0000","last_active":"2024-02-23T00:37:56.076967+0000","last_peered":"2024-02-23T00:37:56.076967+0000","last_clean":"2024-02-23T00:37:56.076967+0000","last_became_active":"2024-02-23T00:34:41.109129+0000","last_became_peered":"2024-02-23T00:34:41.109129+0000","last_unstale":"2024-02-23T00:37:56.076967+0000","last_undegraded":"2024-02-23T00:37:56.076967+0000","last_fullsized":"2024-02-23T00:37:56.076967+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:24.990671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:24.990671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":398,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-23T00:37:55.353455+0000","last_change":"2024-02-23T00:36:40.485366+0000","last_active":"2024-02-23T00:37:55.353455+0000","last_peered":"2024-02-23T00:37:55.353455+0000","last_clean":"2024-02-23T00:37:55.353455+0000","last_became_active":"2024-02-23T00:34:41.099195+0000","last_became_peered":"2024-02-23T00:34:41.099195+0000","last_unstale":"2024-02-23T00:37:55.353455+0000","last_undegraded":"2024-02-23T00:37:55.353455+0000","last_fullsized":"2024-02-23T00:37:55.353455+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:40.485283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:40.485283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":219,"num_pg":32,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":60,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":79,"acting":79,"num_store_stats":6},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":4001618,"num_objects":173,"num_object_clones":9,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":0,"num_read":113,"num_read_kb":10,"num_write":630,"num_write_kb":4526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":4000297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14057472,"data_stored":12004827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1947,"ondisk_log_size":1947,"up":84,"acting":84,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":105,"acting":105,"num_store_stats":8},{"poolid":4,"num_pg":22,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":55,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1495,"num_read_kb":1220,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":2778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":11326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":518,"ondisk_log_size":518,"up":60,"acting":60,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":327,"num_read_kb":327,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":34,"acting":34,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":81,"acting":81,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332229,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310684,"kb_used_data":9364,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93434468,"statfs":{"total":95995035648,"available":95676895232,"internally_reserved":0,"allocated":9588736,"data_stored":7022714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.02,"15min":1.024},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.79800000000000004,"5min":57.401000000000003,"15min":61.048999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.1719999999999999,"15min":1.2450000000000001},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":65.558999999999997,"15min":73.870999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.53400000000000003,"15min":0.69599999999999995},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":0.68100000000000005,"5min":4.6840000000000002,"15min":31.175000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.51300000000000001,"15min":0.68700000000000006},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.59699999999999998,"5min":3.8719999999999999,"15min":29.423999999999999},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.50600000000000001,"15min":0.66200000000000003},"min":{"1min":0.30299999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64300000000000002,"5min":3.7240000000000002,"15min":16.603999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56699999999999995,"15min":0.61499999999999999},"min":{"1min":0.36199999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65400000000000003,"5min":4.1369999999999996,"15min":7.5339999999999998},"last":0.47799999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.54000000000000004,"15min":0.97799999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.59099999999999997,"5min":3.7679999999999998,"15min":57.729999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.57699999999999996,"15min":0.90800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.312},"max":{"1min":0.69699999999999995,"5min":3.8420000000000001,"15min":27.573},"last":0.39900000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42899999999999999,"15min":0.41899999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":0.58499999999999996,"5min":1.706,"15min":1.706},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.37,"5min":0.45600000000000002,"15min":0.436},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54200000000000004,"5min":3.294,"15min":3.294},"last":0.44600000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.434,"15min":0.43099999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.61399999999999999,"5min":0.85899999999999999,"15min":1.0209999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.48699999999999999,"15min":0.45600000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76400000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.623}]},{"osd":6,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.51200000000000001,"15min":0.48699999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.28000000000000003},"max":{"1min":0.627,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.434,"5min":0.48899999999999999,"15min":0.47299999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":0.57299999999999995,"5min":0.83299999999999996,"15min":1.002},"last":0.59499999999999997}]}]},{"osd":6,"up_from":44,"seq":188978561162,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310900,"kb_used_data":5484,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93434252,"statfs":{"total":95995035648,"available":95676674048,"internally_reserved":0,"allocated":5615616,"data_stored":3022529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.6499999999999999,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.78800000000000003,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64800000000000002,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":9.468,"15min":9.468},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.5,"15min":1.014},"min":{"1min":0.35699999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.67500000000000004,"5min":0.79100000000000004,"15min":59.347999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.497,"15min":1.2929999999999999},"min":{"1min":0.34300000000000003,"5min":0.28299999999999997,"15min":0.23100000000000001},"max":{"1min":0.59899999999999998,"5min":0.875,"15min":87.221000000000004},"last":0.42999999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.81299999999999994},"min":{"1min":0.372,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74299999999999999,"5min":1.022,"15min":25.550999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55900000000000005,"15min":0.88},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.70899999999999996,"5min":4.3570000000000002,"15min":40.914000000000001},"last":0.435}]},{"osd":3,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.54600000000000004,"15min":0.80800000000000005},"min":{"1min":0.36299999999999999,"5min":0.30199999999999999,"15min":0.23400000000000001},"max":{"1min":0.75900000000000001,"5min":3.2799999999999998,"15min":31.309999999999999},"last":0.44},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54400000000000004,"15min":1.329},"min":{"1min":0.42599999999999999,"5min":0.35799999999999998,"15min":0.29899999999999999},"max":{"1min":0.69499999999999995,"5min":1.145,"15min":86.311999999999998},"last":0.50800000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.26100000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.53100000000000003,"5min":0.68500000000000005,"15min":3.7839999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.45200000000000001,"15min":0.504},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":17.718},"last":0.34899999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:37:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.45300000000000001,"15min":0.48099999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.47399999999999998,"5min":0.73799999999999999,"15min":8.0990000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.67900000000000005,"5min":0.75700000000000001,"15min":8.1370000000000005},"last":0.51500000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:37:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.48799999999999999,"15min":0.59499999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.50900000000000001,"5min":0.75,"15min":17.881},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.5,"15min":0.58999999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.55400000000000005,"5min":0.80800000000000005,"15min":17.396999999999998},"last":0.40200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575009,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319080,"kb_used_data":9568,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426072,"statfs":{"total":95995035648,"available":95668297728,"internally_reserved":0,"allocated":9797632,"data_stored":7024487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.29,"15min":0.66300000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.51700000000000002,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.89600000000000002,"15min":0.55200000000000005},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60499999999999998,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:37:32 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.73999999999999999,"15min":0.67000000000000004},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.45300000000000001,"5min":35.313000000000002,"15min":35.313000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.68500000000000005,"15min":0.57499999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.20100000000000001},"max":{"1min":0.47699999999999998,"5min":27.908999999999999,"15min":27.908999999999999},"last":0.36799999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:38:00 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.35599999999999998,"15min":0.63900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.46100000000000002,"5min":2.3919999999999999,"15min":21.013999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.318,"5min":0.45700000000000002,"15min":0.64100000000000001},"min":{"1min":0.247,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.42899999999999999,"5min":10.382999999999999,"15min":29.234999999999999},"last":0.45200000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:37:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.77200000000000002,"15min":0.69699999999999995},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.79400000000000004,"5min":16.231000000000002,"15min":16.231000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.50900000000000001,"15min":0.72299999999999998},"min":{"1min":0.313,"5min":0.308,"15min":0.308},"max":{"1min":0.70999999999999996,"5min":1.135,"15min":29.212},"last":0.52500000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:37:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.78700000000000003,"15min":0.71999999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.75800000000000001,"5min":16.294,"15min":16.294},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.61599999999999999,"15min":0.67900000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80300000000000005,"5min":8.4030000000000005,"15min":11.105},"last":0.53400000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:38:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.63100000000000001,"15min":0.76100000000000001},"min":{"1min":0.34899999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70999999999999996,"5min":10.416,"15min":29.187000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63300000000000001,"15min":0.746},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67200000000000004,"5min":8.2929999999999993,"15min":16.369},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:37:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.92200000000000004,"15min":0.76700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73899999999999999,"5min":16.395,"15min":16.395},"last":0.623},{"interface":"front","average":{"1min":0.503,"5min":0.54900000000000004,"15min":0.66200000000000003},"min":{"1min":0.36599999999999999,"5min":0.36299999999999999,"15min":0.35999999999999999},"max":{"1min":0.82399999999999995,"5min":0.89500000000000002,"15min":11.135999999999999},"last":0.56999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318432,"kb_used_data":4824,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426720,"statfs":{"total":95995035648,"available":95668961280,"internally_reserved":0,"allocated":4939776,"data_stored":2599387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.80600000000000005,"15min":0.495},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47799999999999998,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.246},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.165,"15min":0.61899999999999999},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.997,"15min":0.58899999999999997},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.64400000000000002,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.748,"15min":0.48999999999999999},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.22700000000000001},"max":{"1min":0.51100000000000001,"5min":15.965,"15min":15.965},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":1.0820000000000001,"15min":0.65000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.0369999999999999,"15min":0.63700000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.58799999999999997,"5min":16.006,"15min":16.006},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.73399999999999999,"15min":0.63800000000000001},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84699999999999998,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.38300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83699999999999997,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.071,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86799999999999999,"5min":13.311,"15min":13.311},"last":0.499},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.113,"15min":0.74399999999999999},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":23.744,"15min":23.744},"last":0.42299999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.0740000000000001,"15min":0.80600000000000005},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78500000000000003,"5min":13.334,"15min":33.082999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.93400000000000005,"15min":0.70999999999999996},"min":{"1min":0.41199999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81699999999999995,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.49199999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.84099999999999997,"15min":0.78500000000000003},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.871,"5min":15.943,"15min":31.251999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.091,"15min":0.76100000000000001},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.80000000000000004,"5min":23.785,"15min":23.785},"last":0.47599999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411485,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310680,"kb_used_data":9360,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93434472,"statfs":{"total":95995035648,"available":95676899328,"internally_reserved":0,"allocated":9584640,"data_stored":7021270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":1.0600000000000001,"15min":0.754},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.65500000000000003,"5min":56.509,"15min":56.509},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.59799999999999998,"15min":0.71199999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56799999999999995,"5min":19.571999999999999,"15min":31.498999999999999},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:37:35 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.46999999999999997,"15min":0.59499999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":7.7110000000000003,"5min":7.7110000000000003,"15min":19.324999999999999},"last":0.192},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.70999999999999996},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.0249999999999999,"5min":13.209,"15min":35.052},"last":0.25700000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.442,"15min":0.70499999999999996},"min":{"1min":0.23999999999999999,"5min":0.20999999999999999,"15min":0.20899999999999999},"max":{"1min":6.6559999999999997,"5min":6.6559999999999997,"15min":43.5},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.54700000000000004,"15min":0.65400000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":7.7409999999999997,"5min":10.843,"15min":27.414000000000001},"last":0.22500000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:37:16 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":0.72099999999999997,"15min":0.67700000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":10.106,"5min":10.106,"15min":14.614000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.999,"5min":0.67500000000000004,"15min":0.628},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":7.766,"5min":7.766,"15min":7.766},"last":0.61799999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:37:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57899999999999996,"15min":0.66800000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":2.004,"5min":3.2879999999999998,"15min":17.765000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.34499999999999997,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":10.138,"5min":10.138,"15min":14.589},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:37:04 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.66000000000000003,"15min":0.627},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":7.7869999999999999,"5min":7.7869999999999999,"15min":7.7869999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60399999999999998,"15min":0.751},"min":{"1min":0.379,"5min":0.36699999999999999,"15min":0.34899999999999998},"max":{"1min":1.8460000000000001,"5min":3.3410000000000002,"15min":20.574999999999999},"last":0.44600000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:37:28 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.63,"15min":1.036},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":1.9590000000000001,"5min":3.3780000000000001,"15min":66.632000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.349,"5min":0.748,"15min":0.85199999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":10.163,"5min":10.163,"15min":19.603999999999999},"last":0.60199999999999998}]}]},{"osd":3,"up_from":0,"seq":111669149849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986324,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315080,"kb_used_data":5568,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93430072,"statfs":{"total":95995035648,"available":95672393728,"internally_reserved":0,"allocated":5701632,"data_stored":3021739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1739999999999999,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.68799999999999994,"5min":21.419,"15min":21.419},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":2.1200000000000001,"15min":1.2589999999999999},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":0.83899999999999997,"15min":1.048},"min":{"1min":0.35399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":16.696000000000002,"5min":16.696000000000002,"15min":41.137999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.28,"5min":0.78300000000000003,"15min":0.88700000000000001},"min":{"1min":0.35199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":13.455,"5min":13.455,"15min":26.210000000000001},"last":0.42899999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":0.76200000000000001,"15min":1.1299999999999999},"min":{"1min":0.376,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":13.478,"5min":13.478,"15min":67.253},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.72099999999999997,"15min":1.0209999999999999},"min":{"1min":0.38,"5min":0.30499999999999999,"15min":0.29699999999999999},"max":{"1min":4.2590000000000003,"5min":8.0199999999999996,"15min":41.927999999999997},"last":0.46400000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.69999999999999996,"15min":1.069},"min":{"1min":0.39000000000000001,"5min":0.35699999999999998,"15min":0.34000000000000002},"max":{"1min":7.774,"5min":7.774,"15min":36.301000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":0.75700000000000001,"15min":1.167},"min":{"1min":0.34000000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":16.657,"5min":16.657,"15min":42.064},"last":0.40100000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:37:40 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.219,"5min":0.219,"15min":0.217},"max":{"1min":0.65100000000000002,"5min":8.5299999999999994,"15min":14.242000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.69399999999999995,"15min":0.59599999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.57799999999999996,"5min":8.3010000000000002,"15min":14.124000000000001},"last":0.36899999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:38:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.64800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":0.59499999999999997,"5min":8.0730000000000004,"15min":8.0730000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.52500000000000002,"15min":0.63500000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":0.63900000000000001,"5min":4.71,"15min":45.814999999999998},"last":0.34699999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:37:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.29999999999999999},"max":{"1min":5.1280000000000001,"5min":6.165,"15min":17.178999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.288,"5min":0.81399999999999995,"15min":0.69799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28299999999999997},"max":{"1min":15.85,"5min":15.85,"15min":15.85},"last":0.41899999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757391,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306912,"kb_used_data":5592,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438240,"statfs":{"total":95995035648,"available":95680757760,"internally_reserved":0,"allocated":5726208,"data_stored":3022359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.1200000000000001,"15min":0.70699999999999996},"min":{"1min":0.36699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.115,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":1.028},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.498,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51600000000000001,"15min":0.76100000000000001},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.7629999999999999,"5min":3.7629999999999999,"15min":38.838000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.57499999999999996,"15min":1.196},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.7250000000000001,"5min":3.7250000000000001,"15min":127.794},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.56899999999999995,"15min":0.90700000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":37.442999999999998},"last":0.502},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.70899999999999996},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.734,"5min":3.734,"15min":23.297999999999998},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:37:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":0.64700000000000002,"15min":0.78500000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":12.507,"5min":12.507,"15min":24.414999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.873,"5min":0.61599999999999999,"15min":0.91900000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.31},"max":{"1min":3.8119999999999998,"5min":3.8119999999999998,"15min":40.865000000000002},"last":0.51900000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.438,"15min":0.78300000000000003},"min":{"1min":0.255,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":2.8940000000000001,"5min":2.8940000000000001,"15min":103.13},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.47699999999999998,"15min":0.47099999999999997},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.9420000000000002,"5min":2.9420000000000002,"15min":12.795999999999999},"last":0.26500000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:37:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51500000000000001,"15min":0.498},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":2.9199999999999999,"5min":2.9199999999999999,"15min":7.1440000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.47999999999999998,"15min":0.81999999999999995},"min":{"1min":0.28599999999999998,"5min":0.28399999999999997,"15min":0.26300000000000001},"max":{"1min":0.71399999999999997,"5min":1.4490000000000001,"15min":103.61499999999999},"last":0.32000000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:37:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.9039999999999999,"5min":3.9039999999999999,"15min":7.1820000000000004},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.58199999999999996,"15min":0.53200000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.4050000000000002,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.30199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4395008,"data_stored":4000927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4247552,"data_stored":4000604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":219,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:38:11.061 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-23T00:38:11.062 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.log pg_num 32 2024-02-23T00:38:11.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:38:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:38:11.577 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:38:11.782 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:38:11.782 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:38:11.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:10.469406+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:38:11.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:10.470327+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:38:11.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:11.000678+0000 mgr.y (mgr.24437) 456 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:12.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:10.469406+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:38:12.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:10.470327+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:38:12.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:11.000678+0000 mgr.y (mgr.24437) 456 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:12.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:10.469406+0000 osd.6 (osd.6) 200 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:38:12.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:10.470327+0000 osd.6 (osd.6) 201 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:38:12.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:11.000678+0000 mgr.y (mgr.24437) 456 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:12.286 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:38:12.286 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:38:12.286 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:38:12.286 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:38:12.286 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:38:12.286 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:38:12.286 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:38:12.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 -- 172.21.15.87:0/2434521403 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a040fe510 msgr2=0x7f7a040f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:12.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 --2- 172.21.15.87:0/2434521403 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a040fe510 0x7f7a040f7d40 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 rx=0x7f7a000047b0 tx=0x7f7a00009cb0).stop 2024-02-23T00:38:12.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 -- 172.21.15.87:0/2434521403 shutdown_connections 2024-02-23T00:38:12.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 --2- 172.21.15.87:0/2434521403 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a040f8420 0x7f7a04104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:38:12.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 --2- 172.21.15.87:0/2434521403 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a040fdbb0 0x7f7a040fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 --2- 172.21.15.87:0/2434521403 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a040fe510 0x7f7a040f7d40 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 -- 172.21.15.87:0/2434521403 >> 172.21.15.87:0/2434521403 conn(0x7f7a040f3800 msgr2=0x7f7a040f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 -- 172.21.15.87:0/2434521403 shutdown_connections 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 -- 172.21.15.87:0/2434521403 wait complete. 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 librados: starting msgr at 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.009+0000 7f7a0c096700 1 librados: starting objecter 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 Processor -- start 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 -- start start 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 librados: setting wanted keys 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 librados: calling monclient init 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a040f8420 0x7f7a04110400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a040fdbb0 0x7f7a041108f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a040fe510 0x7f7a0410a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0410aa90 con 0x7f7a040f8420 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0410abd0 con 0x7f7a040fdbb0 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0410ad90 con 0x7f7a040fe510 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0a633700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a040fe510 0x7f7a0410a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a09e32700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a040f8420 0x7f7a04110400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0a633700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a040fe510 0x7f7a0410a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:57276/0 (socket says 172.21.15.87:57276) 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a09e32700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a040f8420 0x7f7a04110400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49292/0 (socket says 172.21.15.87:49292) 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0a633700 1 -- 172.21.15.87:0/3898288590 learned_addr learned my addr 172.21.15.87:0/3898288590 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a09631700 1 --2- 172.21.15.87:0/3898288590 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a040fdbb0 0x7f7a041108f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0a633700 1 -- 172.21.15.87:0/3898288590 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a040fdbb0 msgr2=0x7f7a041108f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0a633700 1 --2- 172.21.15.87:0/3898288590 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a040fdbb0 0x7f7a041108f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0a633700 1 -- 172.21.15.87:0/3898288590 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a040f8420 msgr2=0x7f7a04110400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0a633700 1 --2- 172.21.15.87:0/3898288590 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a040f8420 0x7f7a04110400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:12.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0a633700 1 -- 172.21.15.87:0/3898288590 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f4005040 con 0x7f7a040fe510 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a09631700 1 --2- 172.21.15.87:0/3898288590 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a040fdbb0 0x7f7a041108f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a09e32700 1 --2- 172.21.15.87:0/3898288590 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a040f8420 0x7f7a04110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0a633700 1 --2- 172.21.15.87:0/3898288590 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a040fe510 0x7f7a0410a5a0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 rx=0x7f79fc00bc30 tx=0x7f79fc009840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f79faffd700 10 client.?.objecter ms_handle_connect 0x7f7a040fe510 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f79faffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc00c6f0 con 0x7f7a040fe510 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79fc010070 con 0x7f7a040fe510 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79fc007880 con 0x7f7a040fe510 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.010+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a00005040 con 0x7f7a040fe510 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.011+0000 7f7a0c096700 10 client.37838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:12.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.011+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a0410b380 con 0x7f7a040fe510 2024-02-23T00:38:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.011+0000 7f7a0c096700 1 librados: init done 2024-02-23T00:38:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.011+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7a0405ee80 con 0x7f7a040fe510 2024-02-23T00:38:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.012+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f79fc007c30 con 0x7f7a040fe510 2024-02-23T00:38:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.012+0000 7f79faffd700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6800/1231597463 conn(0x7f79f0062c90 0x7f79f0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.012+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(474..474 src has 173..474) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f79fc07cce0 con 0x7f7a040fe510 2024-02-23T00:38:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.012+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 osd_map(474..474 src has 173..474) v4 2024-02-23T00:38:12.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.012+0000 7f79faffd700 3 client.37838.objecter handle_osd_map got epochs [474,474] > 0 2024-02-23T00:38:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.012+0000 7f79faffd700 3 client.37838.objecter handle_osd_map decoding full epoch 474 2024-02-23T00:38:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.012+0000 7f7a09e32700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6800/1231597463 conn(0x7f79f0062c90 0x7f79f0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.012+0000 7f79faffd700 20 client.37838.objecter dump_active .. 0 homeless 2024-02-23T00:38:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.012+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v474) v1 ==== 54+0+16 (secure 0 0 0) 0x7f79fc082230 con 0x7f7a040fe510 2024-02-23T00:38:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f79faffd700 10 client.37838.objecter ms_handle_connect 0x7f79f0062c90 2024-02-23T00:38:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a09e32700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6800/1231597463 conn(0x7f79f0062c90 0x7f79f0065150 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 rx=0x7f79f4004c80 tx=0x7f79f400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 10 client.37838.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:12.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 1 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 20 client.37838.objecter _get_session osd=-1 returning homeless 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 10 client.37838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=475}) v3 -- 0x7f7a04107110 con 0x7f7a040fe510 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7a0410ba80 con 0x7f7a040fe510 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=474) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79fc082070 con 0x7f7a040fe510 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f79fbfff700 10 client.37838.objecter _check_command_map_dne tid 1 current 474 map_dne_bound 474 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f79fbfff700 10 client.37838.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f79fbfff700 15 client.37838.objecter _session_command_op_remove -1 1 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 10 client.37838.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 2 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6805/685354171 conn(0x7f7a0406c9e0 0x7f7a040273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0406c870 osd=1 3 2024-02-23T00:38:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0406c870 osd=1 3 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0406c870 osd=1 3 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 2 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a0406c870 osd=1 3 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 1 2 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0406c870 osd=1 4 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 10 client.37838.objecter _send_command 2 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a04059e00 con 0x7f7a0406c9e0 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a09631700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6805/685354171 conn(0x7f7a0406c9e0 0x7f7a040273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f79faffd700 10 client.37838.objecter ms_handle_connect 0x7f7a0406c9e0 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.013+0000 7f7a09631700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6805/685354171 conn(0x7f7a0406c9e0 0x7f7a040273c0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.014+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7a04059e00 con 0x7f7a0406c9e0 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.014+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.014+0000 7f79faffd700 10 client.37838.objecter _finish_command 2 = system:0 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.014+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a0406c870 osd=1 4 2024-02-23T00:38:12.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.014+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 1 2 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.028+0000 7f7a0c096700 10 client.37838.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.028+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 3 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.029+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0406c870 osd=1 3 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.029+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0406c870 osd=1 3 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.029+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0406c870 osd=1 3 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.029+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 3 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.029+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a0406c870 osd=1 3 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.029+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 1 3 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.029+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0406c870 osd=1 4 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.029+0000 7f7a0c096700 10 client.37838.objecter _send_command 3 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.029+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7a040278b0 con 0x7f7a0406c9e0 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7a040278b0 con 0x7f7a0406c9e0 2024-02-23T00:38:12.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:12.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f79faffd700 10 client.37838.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a0406c870 osd=1 4 2024-02-23T00:38:12.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 1 3 2024-02-23T00:38:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 10 client.37838.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 4 2024-02-23T00:38:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7a040594d0 0x7f7a0411b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a04059360 osd=2 3 2024-02-23T00:38:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a04059360 osd=2 3 2024-02-23T00:38:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a04059360 osd=2 3 2024-02-23T00:38:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 4 2024-02-23T00:38:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a04059360 osd=2 3 2024-02-23T00:38:12.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 2 4 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a04059360 osd=2 4 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 10 client.37838.objecter _send_command 4 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a04028760 con 0x7f7a040594d0 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.030+0000 7f7a09e32700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7a040594d0 0x7f7a0411b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.031+0000 7f79faffd700 10 client.37838.objecter ms_handle_connect 0x7f7a040594d0 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.031+0000 7f7a09e32700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7a040594d0 0x7f7a0411b220 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.031+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7a04028760 con 0x7f7a040594d0 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.031+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.031+0000 7f79faffd700 10 client.37838.objecter _finish_command 4 = system:0 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.031+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a04059360 osd=2 4 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.031+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 2 4 2024-02-23T00:38:12.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 10 client.37838.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 5 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a04059360 osd=2 3 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a04059360 osd=2 3 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a04059360 osd=2 3 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 5 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a04059360 osd=2 3 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 2 5 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a04059360 osd=2 4 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 10 client.37838.objecter _send_command 5 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.046+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7a04027d20 con 0x7f7a040594d0 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7a04027d20 con 0x7f7a040594d0 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:12.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f79faffd700 10 client.37838.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a04059360 osd=2 4 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 2 5 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 10 client.37838.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 6 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6813/128059790 conn(0x7f7a0411c890 0x7f7a0411ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a040f9fc0 osd=3 3 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a040f9fc0 osd=3 3 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a040f9fc0 osd=3 3 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 6 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a040f9fc0 osd=3 3 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 3 6 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a040f9fc0 osd=3 4 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 10 client.37838.objecter _send_command 6 2024-02-23T00:38:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.047+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a041218d0 con 0x7f7a0411c890 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.048+0000 7f7a09631700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6813/128059790 conn(0x7f7a0411c890 0x7f7a0411ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.048+0000 7f79faffd700 10 client.37838.objecter ms_handle_connect 0x7f7a0411c890 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.048+0000 7f7a09631700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.87:6813/128059790 conn(0x7f7a0411c890 0x7f7a0411ec80 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.048+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7a041218d0 con 0x7f7a0411c890 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.048+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.048+0000 7f79faffd700 10 client.37838.objecter _finish_command 6 = system:0 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.048+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a040f9fc0 osd=3 4 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.048+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 3 6 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 10 client.37838.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 7 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a040f9fc0 osd=3 3 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a040f9fc0 osd=3 3 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a040f9fc0 osd=3 3 2024-02-23T00:38:12.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 7 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a040f9fc0 osd=3 3 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 3 7 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a040f9fc0 osd=3 4 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 10 client.37838.objecter _send_command 7 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.062+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7a040f9a90 con 0x7f7a0411c890 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7a040f9a90 con 0x7f7a0411c890 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f79faffd700 10 client.37838.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a040f9fc0 osd=3 4 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 3 7 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 10 client.37838.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 8 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7a041202f0 0x7f7a04120730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0411bee0 osd=4 3 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0411bee0 osd=4 3 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0411bee0 osd=4 3 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 8 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a0411bee0 osd=4 3 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 4 8 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0411bee0 osd=4 4 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 10 client.37838.objecter _send_command 8 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.064+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a0411c5d0 con 0x7f7a041202f0 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.065+0000 7f7a0a633700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7a041202f0 0x7f7a04120730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.065+0000 7f7a0a633700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7a041202f0 0x7f7a04120730 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.065+0000 7f79faffd700 10 client.37838.objecter ms_handle_connect 0x7f7a041202f0 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.066+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7a0411c5d0 con 0x7f7a041202f0 2024-02-23T00:38:12.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.066+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.066+0000 7f79faffd700 10 client.37838.objecter _finish_command 8 = system:0 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.066+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a0411bee0 osd=4 4 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.066+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 4 8 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 10 client.37838.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 9 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0411bee0 osd=4 3 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0411bee0 osd=4 3 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0411bee0 osd=4 3 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 9 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a0411bee0 osd=4 3 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 4 9 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0411bee0 osd=4 4 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 10 client.37838.objecter _send_command 9 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.080+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7a041213f0 con 0x7f7a041202f0 2024-02-23T00:38:12.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7a041213f0 con 0x7f7a041202f0 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f79faffd700 10 client.37838.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a0411bee0 osd=4 4 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 4 9 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 10 client.37838.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 10 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7a04123a50 0x7f7a04125e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a041213f0 osd=5 3 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a041213f0 osd=5 3 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a041213f0 osd=5 3 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 10 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a041213f0 osd=5 3 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 5 10 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a041213f0 osd=5 4 2024-02-23T00:38:12.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 10 client.37838.objecter _send_command 10 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.082+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a04121690 con 0x7f7a04123a50 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.083+0000 7f7a09e32700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7a04123a50 0x7f7a04125e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.083+0000 7f7a09e32700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7a04123a50 0x7f7a04125e40 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.083+0000 7f79faffd700 10 client.37838.objecter ms_handle_connect 0x7f7a04123a50 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.084+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7a04121690 con 0x7f7a04123a50 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.084+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.084+0000 7f79faffd700 10 client.37838.objecter _finish_command 10 = system:0 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.084+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a041213f0 osd=5 4 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.084+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 5 10 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 10 client.37838.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 11 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a041213f0 osd=5 3 2024-02-23T00:38:12.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a041213f0 osd=5 3 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a041213f0 osd=5 3 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 11 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a041213f0 osd=5 3 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 5 11 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a041213f0 osd=5 4 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 10 client.37838.objecter _send_command 11 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.098+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7a04121690 con 0x7f7a04123a50 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.100+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7a04121690 con 0x7f7a04123a50 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.100+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.100+0000 7f79faffd700 10 client.37838.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.100+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a041213f0 osd=5 4 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.100+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 5 11 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 10 client.37838.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:12.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 12 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7a04126840 0x7f7a04128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a04121690 osd=6 3 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a04121690 osd=6 3 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a04121690 osd=6 3 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 12 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a04121690 osd=6 3 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 6 12 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a04121690 osd=6 4 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 10 client.37838.objecter _send_command 12 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a0412ba60 con 0x7f7a04126840 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a09631700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7a04126840 0x7f7a04128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f7a09631700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7a04126840 0x7f7a04128c90 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.101+0000 7f79faffd700 10 client.37838.objecter ms_handle_connect 0x7f7a04126840 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.102+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7a0412ba60 con 0x7f7a04126840 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.102+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.102+0000 7f79faffd700 10 client.37838.objecter _finish_command 12 = system:0 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.102+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a04121690 osd=6 4 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.102+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 6 12 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 10 client.37838.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 13 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a04121690 osd=6 3 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a04121690 osd=6 3 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a04121690 osd=6 3 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 13 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a04121690 osd=6 3 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 6 13 2024-02-23T00:38:12.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a04121690 osd=6 4 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 10 client.37838.objecter _send_command 13 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.116+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7a0412ba60 con 0x7f7a04126840 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.118+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7a0412ba60 con 0x7f7a04126840 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.118+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.118+0000 7f79faffd700 10 client.37838.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.118+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a04121690 osd=6 4 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.118+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 6 13 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.118+0000 7f7a0c096700 10 client.37838.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 14 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6812/373787000 conn(0x7f7a0412adf0 0x7f7a0412b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0412ba60 osd=7 3 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0412ba60 osd=7 3 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0412ba60 osd=7 3 2024-02-23T00:38:12.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 14 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a0412ba60 osd=7 3 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 7 14 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0412ba60 osd=7 4 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 10 client.37838.objecter _send_command 14 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a0412ee10 con 0x7f7a0412adf0 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0a633700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6812/373787000 conn(0x7f7a0412adf0 0x7f7a0412b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f79faffd700 10 client.37838.objecter ms_handle_connect 0x7f7a0412adf0 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.119+0000 7f7a0a633700 1 --2- 172.21.15.87:0/3898288590 >> v2:172.21.15.171:6812/373787000 conn(0x7f7a0412adf0 0x7f7a0412b250 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.120+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7a0412ee10 con 0x7f7a0412adf0 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.120+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.120+0000 7f79faffd700 10 client.37838.objecter _finish_command 14 = system:0 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.120+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a0412ba60 osd=7 4 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.120+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 7 14 2024-02-23T00:38:12.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.127+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== mon.2 v2:172.21.15.87:3301/0 8 ==== osd_map(475..475 src has 173..475) v4 ==== 280+0+0 (secure 0 0 0) 0x7f79fc0098b0 con 0x7f7a040fe510 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.127+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 osd_map(475..475 src has 173..475) v4 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.127+0000 7f79faffd700 3 client.37838.objecter handle_osd_map got epochs [475,475] > 474 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.127+0000 7f79faffd700 3 client.37838.objecter handle_osd_map decoding incremental epoch 475 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.127+0000 7f79faffd700 20 client.37838.objecter dump_active .. 0 homeless 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 10 client.37838.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign -1 15 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0412ba60 osd=7 3 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0412ba60 osd=7 3 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 20 client.37838.objecter _get_session s=0x7f7a0412ba60 osd=7 3 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_remove -1 15 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 20 client.37838.objecter get_session s=0x7f7a0412ba60 osd=7 3 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 15 client.37838.objecter _session_command_op_assign 7 15 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 20 client.37838.objecter put_session s=0x7f7a0412ba60 osd=7 4 2024-02-23T00:38:12.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 10 client.37838.objecter _send_command 15 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.136+0000 7f7a0c096700 1 -- 172.21.15.87:0/3898288590 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7a04129240 con 0x7f7a0412adf0 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.138+0000 7f79faffd700 1 -- 172.21.15.87:0/3898288590 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7a04129240 con 0x7f7a0412adf0 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.138+0000 7f79faffd700 10 client.37838.objecter ms_dispatch 0x7f7a04001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.138+0000 7f79faffd700 10 client.37838.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.138+0000 7f79faffd700 20 client.37838.objecter put_session s=0x7f7a0412ba60 osd=7 4 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:12.138+0000 7f79faffd700 15 client.37838.objecter _session_command_op_remove 7 15 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.313 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.314 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:38:12.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:12] "GET /metrics HTTP/1.1" 200 188726 "" "Prometheus/2.33.4" 2024-02-23T00:38:12.583 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-2[57400]: debug 2024-02-23T00:38:12.235+0000 7f683186f700 -1 snap_mapper.add_oid found existing snaps mapped on 219:2a5c61d9:test-rados-api-smithi087-82091-76::foo:2, removing 2024-02-23T00:38:12.665 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:38:12.230+0000 7f5e9f2f5700 -1 snap_mapper.add_oid found existing snaps mapped on 219:2a5c61d9:test-rados-api-smithi087-82091-76::foo:2, removing 2024-02-23T00:38:12.666 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:38:12.230+0000 7fbec0648700 -1 snap_mapper.add_oid found existing snaps mapped on 219:2a5c61d9:test-rados-api-smithi087-82091-76::foo:2, removing 2024-02-23T00:38:12.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:11.694692+0000 mgr.y (mgr.24437) 457 : cluster [DBG] pgmap v727: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:12.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:12.012734+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.87:0/3898288590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:12.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:12.021456+0000 osd.4 (osd.4) 229 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:38:12.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:12.024465+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:38:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:12.112200+0000 osd.2 (osd.2) 183 : cluster [DBG] 2.e scrub starts 2024-02-23T00:38:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:12.113092+0000 osd.2 (osd.2) 184 : cluster [DBG] 2.e scrub ok 2024-02-23T00:38:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:12.127245+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e475: 8 total, 7 up, 7 in 2024-02-23T00:38:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:12.248805+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:12.256181+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:12.257020+0000 mon.b (mon.1) 387 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-75"}]: dispatch 2024-02-23T00:38:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:12.257160+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-75"}]: dispatch 2024-02-23T00:38:13.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:11.694692+0000 mgr.y (mgr.24437) 457 : cluster [DBG] pgmap v727: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:13.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:12.012734+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.87:0/3898288590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:13.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:12.021456+0000 osd.4 (osd.4) 229 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:38:13.070 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:12.024465+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:38:13.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:12.112200+0000 osd.2 (osd.2) 183 : cluster [DBG] 2.e scrub starts 2024-02-23T00:38:13.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:12.113092+0000 osd.2 (osd.2) 184 : cluster [DBG] 2.e scrub ok 2024-02-23T00:38:13.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:12.127245+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e475: 8 total, 7 up, 7 in 2024-02-23T00:38:13.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:12.248805+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:13.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:12.256181+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:13.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:12.257020+0000 mon.b (mon.1) 387 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-75"}]: dispatch 2024-02-23T00:38:13.071 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:12.257160+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-75"}]: dispatch 2024-02-23T00:38:13.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:11.694692+0000 mgr.y (mgr.24437) 457 : cluster [DBG] pgmap v727: 238 pgs: 26 active+undersized+degraded, 69 active+undersized, 143 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 124/1206 objects degraded (10.282%) 2024-02-23T00:38:13.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:12.012734+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.87:0/3898288590' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:13.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:12.021456+0000 osd.4 (osd.4) 229 : cluster [DBG] 6.1e scrub starts 2024-02-23T00:38:13.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:12.024465+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.1e scrub ok 2024-02-23T00:38:13.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:12.112200+0000 osd.2 (osd.2) 183 : cluster [DBG] 2.e scrub starts 2024-02-23T00:38:13.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:12.113092+0000 osd.2 (osd.2) 184 : cluster [DBG] 2.e scrub ok 2024-02-23T00:38:13.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:12.127245+0000 mon.a (mon.0) 2737 : cluster [DBG] osdmap e475: 8 total, 7 up, 7 in 2024-02-23T00:38:13.072 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:12.248805+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:13.073 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:12.256181+0000 mon.a (mon.0) 2738 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:13.073 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:12.257020+0000 mon.b (mon.1) 387 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-75"}]: dispatch 2024-02-23T00:38:13.073 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:12.257160+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-75"}]: dispatch 2024-02-23T00:38:13.089 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-23T00:38:13.108 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:13.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:13.212 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:13.247 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:13.268 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:13.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:13.372 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:13.392 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:13.400 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:13.659 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:38:13.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:38:13.693 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:13.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:38:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:38:13.952 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:13.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:14.056 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:14.086 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:14.197 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:13.068285+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.e scrub starts 2024-02-23T00:38:14.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:13.072077+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.e scrub ok 2024-02-23T00:38:14.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:13.127043+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e476: 8 total, 7 up, 7 in 2024-02-23T00:38:14.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:13.519470+0000 osd.6 (osd.6) 202 : cluster [DBG] 4.a deep-scrub starts 2024-02-23T00:38:14.198 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:13.528938+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.a deep-scrub ok 2024-02-23T00:38:14.206 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:13.068285+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.e scrub starts 2024-02-23T00:38:14.207 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:13.072077+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.e scrub ok 2024-02-23T00:38:14.207 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:13.127043+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e476: 8 total, 7 up, 7 in 2024-02-23T00:38:14.207 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:13.519470+0000 osd.6 (osd.6) 202 : cluster [DBG] 4.a deep-scrub starts 2024-02-23T00:38:14.207 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:13.528938+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.a deep-scrub ok 2024-02-23T00:38:14.212 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:14.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:14.317 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:14.342 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:13.068285+0000 osd.4 (osd.4) 231 : cluster [DBG] 4.e scrub starts 2024-02-23T00:38:14.342 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:13.072077+0000 osd.4 (osd.4) 232 : cluster [DBG] 4.e scrub ok 2024-02-23T00:38:14.343 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:13.127043+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e476: 8 total, 7 up, 7 in 2024-02-23T00:38:14.343 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:13.519470+0000 osd.6 (osd.6) 202 : cluster [DBG] 4.a deep-scrub starts 2024-02-23T00:38:14.343 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:13.528938+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.a deep-scrub ok 2024-02-23T00:38:14.345 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:14.420 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:14.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:14.530 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:14.558 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:15.107 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:15.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:15.211 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:15.337 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:38:15.337 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:38:15.337 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:38:15.338 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:38:15.360 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:13.695371+0000 mgr.y (mgr.24437) 458 : cluster [DBG] pgmap v730: 206 pgs: 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 124/1197 objects degraded (10.359%) 2024-02-23T00:38:15.360 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:13.722551+0000 osd.5 (osd.5) 191 : cluster [DBG] 63.d scrub starts 2024-02-23T00:38:15.360 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:13.724126+0000 osd.5 (osd.5) 192 : cluster [DBG] 63.d scrub ok 2024-02-23T00:38:15.360 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:14.123855+0000 mon.a (mon.0) 2741 : cluster [WRN] Health check update: Degraded data redundancy: 124/1197 objects degraded (10.359%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:15.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:14.138455+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e477: 8 total, 7 up, 7 in 2024-02-23T00:38:15.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:14.143168+0000 mon.b (mon.1) 388 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:15.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:14.143756+0000 mon.a (mon.0) 2743 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:15.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:14.340867+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:38:15.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:14.343766+0000 osd.1 (osd.1) 214 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:38:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:13.695371+0000 mgr.y (mgr.24437) 458 : cluster [DBG] pgmap v730: 206 pgs: 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 124/1197 objects degraded (10.359%) 2024-02-23T00:38:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:13.722551+0000 osd.5 (osd.5) 191 : cluster [DBG] 63.d scrub starts 2024-02-23T00:38:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:13.724126+0000 osd.5 (osd.5) 192 : cluster [DBG] 63.d scrub ok 2024-02-23T00:38:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:14.123855+0000 mon.a (mon.0) 2741 : cluster [WRN] Health check update: Degraded data redundancy: 124/1197 objects degraded (10.359%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:14.138455+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e477: 8 total, 7 up, 7 in 2024-02-23T00:38:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:14.143168+0000 mon.b (mon.1) 388 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:14.143756+0000 mon.a (mon.0) 2743 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:14.340867+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:14.343766+0000 osd.1 (osd.1) 214 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:13.695371+0000 mgr.y (mgr.24437) 458 : cluster [DBG] pgmap v730: 206 pgs: 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 124/1197 objects degraded (10.359%) 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:13.722551+0000 osd.5 (osd.5) 191 : cluster [DBG] 63.d scrub starts 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:13.724126+0000 osd.5 (osd.5) 192 : cluster [DBG] 63.d scrub ok 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:14.123855+0000 mon.a (mon.0) 2741 : cluster [WRN] Health check update: Degraded data redundancy: 124/1197 objects degraded (10.359%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:14.138455+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e477: 8 total, 7 up, 7 in 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:14.143168+0000 mon.b (mon.1) 388 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:14.143756+0000 mon.a (mon.0) 2743 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:15.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:14.340867+0000 osd.1 (osd.1) 213 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:38:15.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:14.343766+0000 osd.1 (osd.1) 214 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:38:15.686 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:38:16.034 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-23T00:38:16.107 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:16.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:16.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 -- 172.21.15.87:0/2543492392 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40fe510 msgr2=0x7f7af40f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 --2- 172.21.15.87:0/2543492392 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40fe510 0x7f7af40f7d40 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f7ae80047b0 tx=0x7f7ae8009cb0).stop 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 -- 172.21.15.87:0/2543492392 shutdown_connections 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 --2- 172.21.15.87:0/2543492392 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7af40f8420 0x7f7af4104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 --2- 172.21.15.87:0/2543492392 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7af40fdbb0 0x7f7af40fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 --2- 172.21.15.87:0/2543492392 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40fe510 0x7f7af40f7d40 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 -- 172.21.15.87:0/2543492392 >> 172.21.15.87:0/2543492392 conn(0x7f7af40f3800 msgr2=0x7f7af40f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 -- 172.21.15.87:0/2543492392 shutdown_connections 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 -- 172.21.15.87:0/2543492392 wait complete. 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 librados: starting msgr at 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 librados: starting objecter 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 Processor -- start 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 -- start start 2024-02-23T00:38:16.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 librados: setting wanted keys 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.102+0000 7f7af93b7700 1 librados: calling monclient init 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40f8420 0x7f7af410e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7af40fdbb0 0x7f7af410e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7af40fe510 0x7f7af4108230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af4108720 con 0x7f7af40fdbb0 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af4108890 con 0x7f7af40fe510 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af4108a50 con 0x7f7af40f8420 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af27fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7af40fdbb0 0x7f7af410e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af2ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40f8420 0x7f7af410e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af27fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7af40fdbb0 0x7f7af410e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49304/0 (socket says 172.21.15.87:49304) 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af2ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40f8420 0x7f7af410e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:57290/0 (socket says 172.21.15.87:57290) 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af27fc700 1 -- 172.21.15.87:0/195334664 learned_addr learned my addr 172.21.15.87:0/195334664 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af37fe700 1 --2- 172.21.15.87:0/195334664 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7af40fe510 0x7f7af4108230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:16.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af27fc700 1 -- 172.21.15.87:0/195334664 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40f8420 msgr2=0x7f7af410e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af27fc700 1 --2- 172.21.15.87:0/195334664 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40f8420 0x7f7af410e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af27fc700 1 -- 172.21.15.87:0/195334664 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7af40fe510 msgr2=0x7f7af4108230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af27fc700 1 --2- 172.21.15.87:0/195334664 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7af40fe510 0x7f7af4108230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af27fc700 1 -- 172.21.15.87:0/195334664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae4005040 con 0x7f7af40fdbb0 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af2ffd700 1 --2- 172.21.15.87:0/195334664 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40f8420 0x7f7af410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af27fc700 1 --2- 172.21.15.87:0/195334664 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7af40fdbb0 0x7f7af410e580 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f7ae80044e0 tx=0x7f7ae8007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7adbfff700 10 client.?.objecter ms_handle_connect 0x7f7af40fdbb0 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7adbfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7adbfff700 1 -- 172.21.15.87:0/195334664 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8018070 con 0x7f7af40fdbb0 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae8005040 con 0x7f7af40fdbb0 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7adbfff700 1 -- 172.21.15.87:0/195334664 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ae80092e0 con 0x7f7af40fdbb0 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7adbfff700 1 -- 172.21.15.87:0/195334664 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae801dc50 con 0x7f7af40fdbb0 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 10 client.27378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:16.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af4108f70 con 0x7f7af40fdbb0 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.103+0000 7f7af93b7700 1 librados: init done 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.104+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af405ee80 con 0x7f7af40fdbb0 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.105+0000 7f7adbfff700 1 -- 172.21.15.87:0/195334664 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7ae801d420 con 0x7f7af40fdbb0 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.105+0000 7f7adbfff700 1 --2- 172.21.15.87:0/195334664 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7adc062c90 0x7f7adc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.105+0000 7f7adbfff700 1 -- 172.21.15.87:0/195334664 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(478..478 src has 173..478) v4 ==== 8150+0+0 (secure 0 0 0) 0x7f7ae807e510 con 0x7f7af40fdbb0 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.105+0000 7f7adbfff700 10 client.27378.objecter ms_dispatch 0x7f7af4001550 osd_map(478..478 src has 173..478) v4 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.105+0000 7f7adbfff700 3 client.27378.objecter handle_osd_map got epochs [478,478] > 0 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.105+0000 7f7adbfff700 3 client.27378.objecter handle_osd_map decoding full epoch 478 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.105+0000 7f7af2ffd700 1 --2- 172.21.15.87:0/195334664 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7adc062c90 0x7f7adc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.105+0000 7f7adbfff700 20 client.27378.objecter dump_active .. 0 homeless 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.106+0000 7f7af2ffd700 1 --2- 172.21.15.87:0/195334664 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7adc062c90 0x7f7adc065150 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f7ae4004c80 tx=0x7f7ae400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.106+0000 7f7adbfff700 10 client.27378.objecter ms_handle_connect 0x7f7adc062c90 2024-02-23T00:38:16.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.110+0000 7f7adbfff700 1 -- 172.21.15.87:0/195334664 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7ae80519c0 con 0x7f7af40fdbb0 2024-02-23T00:38:16.228 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:16.258 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:16.278 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:16.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:16.277+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "32"} v 0) v1 -- 0x7f7af41097a0 con 0x7f7af40fdbb0 2024-02-23T00:38:16.280 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:15.144255+0000 mon.a (mon.0) 2744 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:16.280 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:15.144372+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e478: 8 total, 7 up, 7 in 2024-02-23T00:38:16.280 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:15.170086+0000 mon.b (mon.1) 389 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:16.280 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:15.180183+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:16.280 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:15.181868+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:16.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:16.325 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:16.382 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:16.408 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:15.144255+0000 mon.a (mon.0) 2744 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:16.408 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:15.144372+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e478: 8 total, 7 up, 7 in 2024-02-23T00:38:16.408 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:15.170086+0000 mon.b (mon.1) 389 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:16.409 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:15.180183+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:16.409 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:15.181868+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:16.409 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:15.144255+0000 mon.a (mon.0) 2744 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:16.409 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:15.144372+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e478: 8 total, 7 up, 7 in 2024-02-23T00:38:16.409 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:15.170086+0000 mon.b (mon.1) 389 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:16.409 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:15.180183+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:16.409 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:15.181868+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:16.410 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:16.611 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:16.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:16.717 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:16.748 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:17.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.150+0000 7f7adbfff700 1 -- 172.21.15.87:0/195334664 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "32"}]=0 set pool 4 pg_num to 32 v480) v1 ==== 140+0+0 (secure 0 0 0) 0x7f7ae80515b0 con 0x7f7af40fdbb0 2024-02-23T00:38:17.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.155+0000 7f7af93b7700 10 librados: watch_flush enter 2024-02-23T00:38:17.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.155+0000 7f7af93b7700 10 librados: watch_flush exit 2024-02-23T00:38:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.155+0000 7f7af93b7700 20 client.27378.objecter shutdown clearing up homeless session... 2024-02-23T00:38:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.155+0000 7f7af93b7700 10 client.27378.objecter successfully canceled tick 2024-02-23T00:38:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.156+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7adc062c90 msgr2=0x7f7adc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.156+0000 7f7af93b7700 1 --2- 172.21.15.87:0/195334664 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7adc062c90 0x7f7adc065150 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 rx=0x7f7ae4004c80 tx=0x7f7ae400b040).stop 2024-02-23T00:38:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.156+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7af40fdbb0 msgr2=0x7f7af410e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:17.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.156+0000 7f7af93b7700 1 --2- 172.21.15.87:0/195334664 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7af40fdbb0 0x7f7af410e580 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 rx=0x7f7ae80044e0 tx=0x7f7ae8007f90).stop 2024-02-23T00:38:17.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 shutdown_connections 2024-02-23T00:38:17.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f7af93b7700 1 --2- 172.21.15.87:0/195334664 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7adc062c90 0x7f7adc065150 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:17.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f7af93b7700 1 --2- 172.21.15.87:0/195334664 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7af40fe510 0x7f7af4108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:17.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f7af93b7700 1 --2- 172.21.15.87:0/195334664 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7af40f8420 0x7f7af410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:17.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f7af93b7700 1 --2- 172.21.15.87:0/195334664 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7af40fdbb0 0x7f7af410e580 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:17.163 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 >> 172.21.15.87:0/195334664 conn(0x7f7af40f3800 msgr2=0x7f7af40f5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:17.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 shutdown_connections 2024-02-23T00:38:17.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f7af93b7700 1 -- 172.21.15.87:0/195334664 wait complete. 2024-02-23T00:38:17.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f7af93b7700 1 librados: shutdown 2024-02-23T00:38:17.164 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 pg_num to 32 2024-02-23T00:38:17.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.143+0000 7f883a742700 1 -- 172.21.15.87:0/499083397 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88340f7400 msgr2=0x7f8834103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:17.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.143+0000 7f883a742700 1 --2- 172.21.15.87:0/499083397 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88340f7400 0x7f8834103a20 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 rx=0x7f882c009be0 tx=0x7f882c004d10).stop 2024-02-23T00:38:17.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.143+0000 7f883a742700 1 -- 172.21.15.87:0/499083397 shutdown_connections 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.143+0000 7f883a742700 1 --2- 172.21.15.87:0/499083397 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88340f7400 0x7f8834103a20 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.143+0000 7f883a742700 1 --2- 172.21.15.87:0/499083397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88340fcbb0 0x7f88340fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.143+0000 7f883a742700 1 --2- 172.21.15.87:0/499083397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88340fd510 0x7f88340f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.143+0000 7f883a742700 1 -- 172.21.15.87:0/499083397 >> 172.21.15.87:0/499083397 conn(0x7f88340f2820 msgr2=0x7f88340f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.143+0000 7f883a742700 1 -- 172.21.15.87:0/499083397 shutdown_connections 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.143+0000 7f883a742700 1 -- 172.21.15.87:0/499083397 wait complete. 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 librados: starting msgr at 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 librados: starting objecter 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 Processor -- start 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 -- start start 2024-02-23T00:38:17.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 librados: setting wanted keys 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 librados: calling monclient init 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88340f7400 0x7f8834106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88340fcbb0 0x7f88341073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88340fd510 0x7f883410bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8834107c00 con 0x7f88340f7400 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8834107d70 con 0x7f88340fcbb0 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f883a742700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883410c480 con 0x7f88340fd510 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f8833fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88340f7400 0x7f8834106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f8833fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88340f7400 0x7f8834106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49322/0 (socket says 172.21.15.87:49322) 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f8833fff700 1 -- 172.21.15.87:0/3609314718 learned_addr learned my addr 172.21.15.87:0/3609314718 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f88337fe700 1 --2- 172.21.15.87:0/3609314718 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88340fcbb0 0x7f88341073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.144+0000 7f8838cdf700 1 --2- 172.21.15.87:0/3609314718 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88340fd510 0x7f883410bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.145+0000 7f8838cdf700 1 -- 172.21.15.87:0/3609314718 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88340fcbb0 msgr2=0x7f88341073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.145+0000 7f8838cdf700 1 --2- 172.21.15.87:0/3609314718 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88340fcbb0 0x7f88341073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.145+0000 7f8838cdf700 1 -- 172.21.15.87:0/3609314718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88340f7400 msgr2=0x7f8834106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.145+0000 7f8838cdf700 1 --2- 172.21.15.87:0/3609314718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88340f7400 0x7f8834106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.145+0000 7f8838cdf700 1 -- 172.21.15.87:0/3609314718 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8824005040 con 0x7f88340fd510 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.145+0000 7f8838cdf700 1 --2- 172.21.15.87:0/3609314718 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88340fd510 0x7f883410bf10 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7f882c004e00 tx=0x7f882c004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.145+0000 7f88317fa700 10 client.?.objecter ms_handle_connect 0x7f88340fd510 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.145+0000 7f88317fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.156+0000 7f88317fa700 1 -- 172.21.15.87:0/3609314718 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c00e040 con 0x7f88340fd510 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.156+0000 7f88317fa700 1 -- 172.21.15.87:0/3609314718 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f882c018040 con 0x7f88340fd510 2024-02-23T00:38:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.156+0000 7f88317fa700 1 -- 172.21.15.87:0/3609314718 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882c013cf0 con 0x7f88340fd510 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.158+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882c0097c0 con 0x7f88340fd510 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.158+0000 7f883a742700 10 client.37850.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.158+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883410c920 con 0x7f88340fd510 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.158+0000 7f883a742700 1 librados: init done 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.160+0000 7f88317fa700 1 -- 172.21.15.87:0/3609314718 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f882c007d30 con 0x7f88340fd510 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.160+0000 7f88317fa700 1 --2- 172.21.15.87:0/3609314718 >> v2:172.21.15.87:6800/1231597463 conn(0x7f881c062ce0 0x7f881c0651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.160+0000 7f88317fa700 1 -- 172.21.15.87:0/3609314718 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(480..480 src has 173..480) v4 ==== 8182+0+0 (secure 0 0 0) 0x7f882c07e170 con 0x7f88340fd510 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.160+0000 7f88317fa700 10 client.37850.objecter ms_dispatch 0x7f8834001550 osd_map(480..480 src has 173..480) v4 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.160+0000 7f88317fa700 3 client.37850.objecter handle_osd_map got epochs [480,480] > 0 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.160+0000 7f88317fa700 3 client.37850.objecter handle_osd_map decoding full epoch 480 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.160+0000 7f8833fff700 1 --2- 172.21.15.87:0/3609314718 >> v2:172.21.15.87:6800/1231597463 conn(0x7f881c062ce0 0x7f881c0651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f88317fa700 20 client.37850.objecter dump_active .. 0 homeless 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f8833fff700 1 --2- 172.21.15.87:0/3609314718 >> v2:172.21.15.87:6800/1231597463 conn(0x7f881c062ce0 0x7f881c0651a0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7f8824009410 tx=0x7f882400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:17.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.161+0000 7f88317fa700 10 client.37850.objecter ms_handle_connect 0x7f881c062ce0 2024-02-23T00:38:17.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.166+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883405fa60 con 0x7f88340fd510 2024-02-23T00:38:17.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.173+0000 7f88317fa700 1 -- 172.21.15.87:0/3609314718 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f882c021070 con 0x7f88340fd510 2024-02-23T00:38:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:17.198 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:17.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:17.299 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:17.328 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:17.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:17.414+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f8834106120 con 0x7f88340fd510 2024-02-23T00:38:17.430 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:17.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:17.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:17.565 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:15.696443+0000 mgr.y (mgr.24437) 459 : cluster [DBG] pgmap v733: 238 pgs: 26 active+undersized+degraded, 65 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s; 124/1194 objects degraded (10.385%) 2024-02-23T00:38:17.565 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.052226+0000 osd.7 (osd.7) 243 : cluster [DBG] 5.10 deep-scrub starts 2024-02-23T00:38:17.565 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.055149+0000 osd.7 (osd.7) 244 : cluster [DBG] 5.10 deep-scrub ok 2024-02-23T00:38:17.565 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:16.147143+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:17.565 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.147198+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e479: 8 total, 7 up, 7 in 2024-02-23T00:38:17.565 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:16.148421+0000 mon.b (mon.1) 391 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:17.565 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:16.148571+0000 mon.a (mon.0) 2749 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:17.566 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.169911+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:38:17.566 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.174518+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:38:17.566 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:16.278489+0000 mon.a (mon.0) 2750 : audit [INF] from='client.? 172.21.15.87:0/195334664' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "32"}]: dispatch 2024-02-23T00:38:17.566 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.393926+0000 osd.1 (osd.1) 215 : cluster [DBG] 22.1d deep-scrub starts 2024-02-23T00:38:17.566 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.396819+0000 osd.1 (osd.1) 216 : cluster [DBG] 22.1d deep-scrub ok 2024-02-23T00:38:17.566 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.419093+0000 osd.6 (osd.6) 204 : cluster [DBG] 63.1d scrub starts 2024-02-23T00:38:17.566 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.420785+0000 osd.6 (osd.6) 205 : cluster [DBG] 63.1d scrub ok 2024-02-23T00:38:17.566 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:15.696443+0000 mgr.y (mgr.24437) 459 : cluster [DBG] pgmap v733: 238 pgs: 26 active+undersized+degraded, 65 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s; 124/1194 objects degraded (10.385%) 2024-02-23T00:38:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.052226+0000 osd.7 (osd.7) 243 : cluster [DBG] 5.10 deep-scrub starts 2024-02-23T00:38:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.055149+0000 osd.7 (osd.7) 244 : cluster [DBG] 5.10 deep-scrub ok 2024-02-23T00:38:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:16.147143+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.147198+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e479: 8 total, 7 up, 7 in 2024-02-23T00:38:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:16.148421+0000 mon.b (mon.1) 391 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:16.148571+0000 mon.a (mon.0) 2749 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.169911+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.174518+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:16.278489+0000 mon.a (mon.0) 2750 : audit [INF] from='client.? 172.21.15.87:0/195334664' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "32"}]: dispatch 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.393926+0000 osd.1 (osd.1) 215 : cluster [DBG] 22.1d deep-scrub starts 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.396819+0000 osd.1 (osd.1) 216 : cluster [DBG] 22.1d deep-scrub ok 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.419093+0000 osd.6 (osd.6) 204 : cluster [DBG] 63.1d scrub starts 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.420785+0000 osd.6 (osd.6) 205 : cluster [DBG] 63.1d scrub ok 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:15.696443+0000 mgr.y (mgr.24437) 459 : cluster [DBG] pgmap v733: 238 pgs: 26 active+undersized+degraded, 65 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s; 124/1194 objects degraded (10.385%) 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.052226+0000 osd.7 (osd.7) 243 : cluster [DBG] 5.10 deep-scrub starts 2024-02-23T00:38:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.055149+0000 osd.7 (osd.7) 244 : cluster [DBG] 5.10 deep-scrub ok 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:16.147143+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.147198+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e479: 8 total, 7 up, 7 in 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:16.148421+0000 mon.b (mon.1) 391 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:16.148571+0000 mon.a (mon.0) 2749 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.169911+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.174518+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:16.278489+0000 mon.a (mon.0) 2750 : audit [INF] from='client.? 172.21.15.87:0/195334664' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "32"}]: dispatch 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.393926+0000 osd.1 (osd.1) 215 : cluster [DBG] 22.1d deep-scrub starts 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.396819+0000 osd.1 (osd.1) 216 : cluster [DBG] 22.1d deep-scrub ok 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.419093+0000 osd.6 (osd.6) 204 : cluster [DBG] 63.1d scrub starts 2024-02-23T00:38:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.420785+0000 osd.6 (osd.6) 205 : cluster [DBG] 63.1d scrub ok 2024-02-23T00:38:17.661 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:17.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:17.764 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:17.793 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:18.028 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:18.043 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:18.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:18.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:18.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.150+0000 7f88317fa700 1 -- 172.21.15.87:0/3609314718 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v481) v1 ==== 87+0+0 (secure 0 0 0) 0x7f882c0516f0 con 0x7f88340fd510 2024-02-23T00:38:18.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 10 librados: watch_flush enter 2024-02-23T00:38:18.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 10 librados: watch_flush exit 2024-02-23T00:38:18.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 20 client.37850.objecter shutdown clearing up homeless session... 2024-02-23T00:38:18.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 10 client.37850.objecter successfully canceled tick 2024-02-23T00:38:18.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 >> v2:172.21.15.87:6800/1231597463 conn(0x7f881c062ce0 msgr2=0x7f881c0651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:18.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 --2- 172.21.15.87:0/3609314718 >> v2:172.21.15.87:6800/1231597463 conn(0x7f881c062ce0 0x7f881c0651a0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7f8824009410 tx=0x7f882400b040).stop 2024-02-23T00:38:18.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88340fd510 msgr2=0x7f883410bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:18.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 --2- 172.21.15.87:0/3609314718 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88340fd510 0x7f883410bf10 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 rx=0x7f882c004e00 tx=0x7f882c004d10).stop 2024-02-23T00:38:18.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 shutdown_connections 2024-02-23T00:38:18.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 --2- 172.21.15.87:0/3609314718 >> v2:172.21.15.87:6800/1231597463 conn(0x7f881c062ce0 0x7f881c0651a0 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:18.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 --2- 172.21.15.87:0/3609314718 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f88340fd510 0x7f883410bf10 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:18.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 --2- 172.21.15.87:0/3609314718 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f88340f7400 0x7f8834106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:18.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 --2- 172.21.15.87:0/3609314718 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f88340fcbb0 0x7f88341073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:18.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.155+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 >> 172.21.15.87:0/3609314718 conn(0x7f88340f2820 msgr2=0x7f88340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.164+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 shutdown_connections 2024-02-23T00:38:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.166+0000 7f883a742700 1 -- 172.21.15.87:0/3609314718 wait complete. 2024-02-23T00:38:18.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:18.166+0000 7f883a742700 1 librados: shutdown 2024-02-23T00:38:18.168 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:38:18.185 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:18.199 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:38:18.199 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:38:18.199 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:38:18.199 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:38:18.199 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:38:18.294 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:18.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.677381+0000 osd.5 (osd.5) 193 : cluster [DBG] 5.28 scrub starts 2024-02-23T00:38:18.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:16.678148+0000 osd.5 (osd.5) 194 : cluster [DBG] 5.28 scrub ok 2024-02-23T00:38:18.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:17.148659+0000 mon.a (mon.0) 2751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:38:18.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:17.148722+0000 mon.a (mon.0) 2752 : audit [INF] from='client.? 172.21.15.87:0/195334664' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "32"}]': finished 2024-02-23T00:38:18.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:17.148764+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e480: 8 total, 7 up, 7 in 2024-02-23T00:38:18.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:17.155779+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:17.181674+0000 mon.a (mon.0) 2754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:17.415418+0000 mon.c (mon.2) 447 : audit [INF] from='client.? 172.21.15.87:0/3609314718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:17.415741+0000 mon.a (mon.0) 2755 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:17.698190+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:18.052839+0000 osd.7 (osd.7) 245 : cluster [DBG] 2.b scrub starts 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:18.055797+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.b scrub ok 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:18.146024+0000 mon.a (mon.0) 2757 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:18.149966+0000 mon.a (mon.0) 2758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:18.150047+0000 mon.a (mon.0) 2759 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:38:18.299 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:18.150115+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "32"}]': finished 2024-02-23T00:38:18.300 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:18.150161+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e481: 8 total, 7 up, 7 in 2024-02-23T00:38:18.300 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:18.158973+0000 mon.b (mon.1) 393 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:18.300 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:18.163536+0000 mon.a (mon.0) 2762 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:18.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:18.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:18.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.677381+0000 osd.5 (osd.5) 193 : cluster [DBG] 5.28 scrub starts 2024-02-23T00:38:18.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:16.678148+0000 osd.5 (osd.5) 194 : cluster [DBG] 5.28 scrub ok 2024-02-23T00:38:18.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:17.148659+0000 mon.a (mon.0) 2751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:38:18.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:17.148722+0000 mon.a (mon.0) 2752 : audit [INF] from='client.? 172.21.15.87:0/195334664' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "32"}]': finished 2024-02-23T00:38:18.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:17.148764+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e480: 8 total, 7 up, 7 in 2024-02-23T00:38:18.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:17.155779+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:18.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:17.181674+0000 mon.a (mon.0) 2754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:18.301 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:17.415418+0000 mon.c (mon.2) 447 : audit [INF] from='client.? 172.21.15.87:0/3609314718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:17.415741+0000 mon.a (mon.0) 2755 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:17.698190+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:18.052839+0000 osd.7 (osd.7) 245 : cluster [DBG] 2.b scrub starts 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:18.055797+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.b scrub ok 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:18.146024+0000 mon.a (mon.0) 2757 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:18.149966+0000 mon.a (mon.0) 2758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:18.150047+0000 mon.a (mon.0) 2759 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:18.150115+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "32"}]': finished 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:18.150161+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e481: 8 total, 7 up, 7 in 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:18.158973+0000 mon.b (mon.1) 393 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:18.302 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:18.163536+0000 mon.a (mon.0) 2762 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:18.402 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:18.429 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:18.457 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:18.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:18.561 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:18.586 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.677381+0000 osd.5 (osd.5) 193 : cluster [DBG] 5.28 scrub starts 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:16.678148+0000 osd.5 (osd.5) 194 : cluster [DBG] 5.28 scrub ok 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:17.148659+0000 mon.a (mon.0) 2751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:17.148722+0000 mon.a (mon.0) 2752 : audit [INF] from='client.? 172.21.15.87:0/195334664' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num", "val": "32"}]': finished 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:17.148764+0000 mon.a (mon.0) 2753 : cluster [DBG] osdmap e480: 8 total, 7 up, 7 in 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:17.155779+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:17.181674+0000 mon.a (mon.0) 2754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:17.415418+0000 mon.c (mon.2) 447 : audit [INF] from='client.? 172.21.15.87:0/3609314718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:17.415741+0000 mon.a (mon.0) 2755 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:17.698190+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:18.052839+0000 osd.7 (osd.7) 245 : cluster [DBG] 2.b scrub starts 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:18.055797+0000 osd.7 (osd.7) 246 : cluster [DBG] 2.b scrub ok 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:18.146024+0000 mon.a (mon.0) 2757 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:18.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:18.149966+0000 mon.a (mon.0) 2758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:18.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:18.150047+0000 mon.a (mon.0) 2759 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:38:18.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:18.150115+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pg_num_actual", "val": "32"}]': finished 2024-02-23T00:38:18.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:18.150161+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e481: 8 total, 7 up, 7 in 2024-02-23T00:38:18.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:18.158973+0000 mon.b (mon.1) 393 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:18.588 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:18.163536+0000 mon.a (mon.0) 2762 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:18.588 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:19.238 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:19.245 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-23T00:38:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:19.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:19.353 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:19.383 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:19.407 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:19.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:19.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:19.511 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:19.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:17.697236+0000 mgr.y (mgr.24437) 460 : cluster [DBG] pgmap v736: 238 pgs: 26 active+undersized+degraded, 65 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 124/1194 objects degraded (10.385%) 2024-02-23T00:38:19.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:19.153046+0000 mon.a (mon.0) 2763 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:19.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:19.153169+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e482: 8 total, 7 up, 7 in 2024-02-23T00:38:19.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:17.697236+0000 mgr.y (mgr.24437) 460 : cluster [DBG] pgmap v736: 238 pgs: 26 active+undersized+degraded, 65 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 124/1194 objects degraded (10.385%) 2024-02-23T00:38:19.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:19.153046+0000 mon.a (mon.0) 2763 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:19.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:19.153169+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e482: 8 total, 7 up, 7 in 2024-02-23T00:38:19.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:17.697236+0000 mgr.y (mgr.24437) 460 : cluster [DBG] pgmap v736: 238 pgs: 26 active+undersized+degraded, 65 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 124/1194 objects degraded (10.385%) 2024-02-23T00:38:19.539 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:19.153046+0000 mon.a (mon.0) 2763 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:19.539 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:19.153169+0000 mon.a (mon.0) 2764 : cluster [DBG] osdmap e482: 8 total, 7 up, 7 in 2024-02-23T00:38:19.540 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:19.561 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:19.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:19.584 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:19.666 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:19.695 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:20.043 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:20.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:20.148 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:20.175 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:20.323 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:20.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:20.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:19.257529+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-23T00:38:20.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:19.257733+0000 mon.a (mon.0) 2765 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-23T00:38:20.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:19.699812+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-23T00:38:20.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:19.257529+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-23T00:38:20.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:19.257733+0000 mon.a (mon.0) 2765 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-23T00:38:20.337 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:19.699812+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-23T00:38:20.437 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:20.471 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:20.564 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:19.257529+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-23T00:38:20.565 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:19.257733+0000 mon.a (mon.0) 2765 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-23T00:38:20.565 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:19.699812+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-23T00:38:20.569 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:20.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.614+0000 7f6acd832700 1 -- 172.21.15.87:0/2268403563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ac80fcbb0 msgr2=0x7f6ac80fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.614+0000 7f6acd832700 1 --2- 172.21.15.87:0/2268403563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ac80fcbb0 0x7f6ac80fcf90 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7f6abc004870 tx=0x7f6abc009cb0).stop 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.614+0000 7f6acd832700 1 -- 172.21.15.87:0/2268403563 shutdown_connections 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.614+0000 7f6acd832700 1 --2- 172.21.15.87:0/2268403563 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6ac80f7400 0x7f6ac8103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.614+0000 7f6acd832700 1 --2- 172.21.15.87:0/2268403563 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ac80fcbb0 0x7f6ac80fcf90 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.614+0000 7f6acd832700 1 --2- 172.21.15.87:0/2268403563 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6ac80fd510 0x7f6ac80f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.614+0000 7f6acd832700 1 -- 172.21.15.87:0/2268403563 >> 172.21.15.87:0/2268403563 conn(0x7f6ac80f2820 msgr2=0x7f6ac80f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.614+0000 7f6acd832700 1 -- 172.21.15.87:0/2268403563 shutdown_connections 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.614+0000 7f6acd832700 1 -- 172.21.15.87:0/2268403563 wait complete. 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 librados: starting msgr at 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 librados: starting objecter 2024-02-23T00:38:20.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 Processor -- start 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 -- start start 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 librados: setting wanted keys 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 librados: calling monclient init 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6ac80f7400 0x7f6ac810d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ac80fcbb0 0x7f6ac810d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6ac80fd510 0x7f6ac81072d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac81077c0 con 0x7f6ac80fcbb0 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac8107930 con 0x7f6ac80f7400 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6acd832700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac8107af0 con 0x7f6ac80fd510 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac67fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ac80fcbb0 0x7f6ac810d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac77fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6ac80fd510 0x7f6ac81072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac67fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ac80fcbb0 0x7f6ac810d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49338/0 (socket says 172.21.15.87:49338) 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac77fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6ac80fd510 0x7f6ac81072d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:57328/0 (socket says 172.21.15.87:57328) 2024-02-23T00:38:20.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac67fc700 1 -- 172.21.15.87:0/1192593071 learned_addr learned my addr 172.21.15.87:0/1192593071 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac6ffd700 1 --2- 172.21.15.87:0/1192593071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6ac80f7400 0x7f6ac810d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac77fe700 1 -- 172.21.15.87:0/1192593071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6ac80f7400 msgr2=0x7f6ac810d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac77fe700 1 --2- 172.21.15.87:0/1192593071 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6ac80f7400 0x7f6ac810d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac77fe700 1 -- 172.21.15.87:0/1192593071 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ac80fcbb0 msgr2=0x7f6ac810d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac77fe700 1 --2- 172.21.15.87:0/1192593071 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ac80fcbb0 0x7f6ac810d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac77fe700 1 -- 172.21.15.87:0/1192593071 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8005040 con 0x7f6ac80fd510 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.615+0000 7f6ac67fc700 1 --2- 172.21.15.87:0/1192593071 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6ac80fcbb0 0x7f6ac810d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6ac77fe700 1 --2- 172.21.15.87:0/1192593071 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6ac80fd510 0x7f6ac81072d0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 rx=0x7f6ac0010d00 tx=0x7f6ac000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6aaffff700 10 client.?.objecter ms_handle_connect 0x7f6ac80fd510 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6aaffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0019070 con 0x7f6ac80fd510 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc005040 con 0x7f6ac80fd510 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ac0009a60 con 0x7f6ac80fd510 2024-02-23T00:38:20.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0003960 con 0x7f6ac80fd510 2024-02-23T00:38:20.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6acd832700 10 client.37856.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:20.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac81080d0 con 0x7f6ac80fd510 2024-02-23T00:38:20.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.616+0000 7f6acd832700 1 librados: init done 2024-02-23T00:38:20.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.617+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6ac805fa60 con 0x7f6ac80fd510 2024-02-23T00:38:20.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.617+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f6ac000eac0 con 0x7f6ac80fd510 2024-02-23T00:38:20.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.617+0000 7f6aaffff700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6ab0062c90 0x7f6ab0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.617+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(483..483 src has 173..483) v4 ==== 8217+0+0 (secure 0 0 0) 0x7f6ac007f620 con 0x7f6ac80fd510 2024-02-23T00:38:20.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.617+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 osd_map(483..483 src has 173..483) v4 2024-02-23T00:38:20.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.617+0000 7f6aaffff700 3 client.37856.objecter handle_osd_map got epochs [483,483] > 0 2024-02-23T00:38:20.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.617+0000 7f6aaffff700 3 client.37856.objecter handle_osd_map decoding full epoch 483 2024-02-23T00:38:20.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.617+0000 7f6ac6ffd700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6ab0062c90 0x7f6ab0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6aaffff700 20 client.37856.objecter dump_active .. 0 homeless 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v483) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6ac0086070 con 0x7f6ac80fd510 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6aaffff700 10 client.37856.objecter ms_handle_connect 0x7f6ab0062c90 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6ac6ffd700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6ab0062c90 0x7f6ab0065150 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 rx=0x7f6abc0043d0 tx=0x7f6abc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 10 client.37856.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 1 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 20 client.37856.objecter _get_session osd=-1 returning homeless 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 10 client.37856.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=484}) v3 -- 0x7f6ac805ee80 con 0x7f6ac80fd510 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ac81060e0 con 0x7f6ac80fd510 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=483) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ac0052a50 con 0x7f6ac80fd510 2024-02-23T00:38:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6ac57fa700 10 client.37856.objecter _check_command_map_dne tid 1 current 483 map_dne_bound 483 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6ac57fa700 10 client.37856.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6ac57fa700 15 client.37856.objecter _session_command_op_remove -1 1 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 10 client.37856.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 2 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6805/685354171 conn(0x7f6ac80fc410 0x7f6ac80fd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac80fc2a0 osd=1 3 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac80fc2a0 osd=1 3 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac80fc2a0 osd=1 3 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 2 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac80fc2a0 osd=1 3 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 1 2 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac80fc2a0 osd=1 4 2024-02-23T00:38:20.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 10 client.37856.objecter _send_command 2 2024-02-23T00:38:20.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ac8061af0 con 0x7f6ac80fc410 2024-02-23T00:38:20.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6ac67fc700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6805/685354171 conn(0x7f6ac80fc410 0x7f6ac80fd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6aaffff700 10 client.37856.objecter ms_handle_connect 0x7f6ac80fc410 2024-02-23T00:38:20.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.618+0000 7f6ac67fc700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6805/685354171 conn(0x7f6ac80fc410 0x7f6ac80fd990 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:20.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.619+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6ac8061af0 con 0x7f6ac80fc410 2024-02-23T00:38:20.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.619+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:38:20.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.619+0000 7f6aaffff700 10 client.37856.objecter _finish_command 2 = system:0 2024-02-23T00:38:20.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.619+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac80fc2a0 osd=1 4 2024-02-23T00:38:20.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.619+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 1 2 2024-02-23T00:38:20.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 10 client.37856.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:20.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 3 2024-02-23T00:38:20.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac80fc2a0 osd=1 3 2024-02-23T00:38:20.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac80fc2a0 osd=1 3 2024-02-23T00:38:20.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac80fc2a0 osd=1 3 2024-02-23T00:38:20.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 3 2024-02-23T00:38:20.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac80fc2a0 osd=1 3 2024-02-23T00:38:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 1 3 2024-02-23T00:38:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac80fc2a0 osd=1 4 2024-02-23T00:38:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 10 client.37856.objecter _send_command 3 2024-02-23T00:38:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.634+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6ac8061d10 con 0x7f6ac80fc410 2024-02-23T00:38:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6ac8061d10 con 0x7f6ac80fc410 2024-02-23T00:38:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6aaffff700 10 client.37856.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac80fc2a0 osd=1 4 2024-02-23T00:38:20.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 1 3 2024-02-23T00:38:20.641 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:38:20.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 10 client.37856.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:20.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 4 2024-02-23T00:38:20.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6ac80fe650 0x7f6ac80278d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8061290 osd=2 3 2024-02-23T00:38:20.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8061290 osd=2 3 2024-02-23T00:38:20.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8061290 osd=2 3 2024-02-23T00:38:20.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 4 2024-02-23T00:38:20.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac8061290 osd=2 3 2024-02-23T00:38:20.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 2 4 2024-02-23T00:38:20.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8061290 osd=2 4 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 10 client.37856.objecter _send_command 4 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ac8061430 con 0x7f6ac80fe650 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6ac6ffd700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6ac80fe650 0x7f6ac80278d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6ac6ffd700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6ac80fe650 0x7f6ac80278d0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.636+0000 7f6aaffff700 10 client.37856.objecter ms_handle_connect 0x7f6ac80fe650 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.637+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6ac8061430 con 0x7f6ac80fe650 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.637+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.637+0000 7f6aaffff700 10 client.37856.objecter _finish_command 4 = system:0 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.637+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac8061290 osd=2 4 2024-02-23T00:38:20.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.637+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 2 4 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 10 client.37856.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 5 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8061290 osd=2 3 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8061290 osd=2 3 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8061290 osd=2 3 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 5 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac8061290 osd=2 3 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 2 5 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8061290 osd=2 4 2024-02-23T00:38:20.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 10 client.37856.objecter _send_command 5 2024-02-23T00:38:20.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.651+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6ac8028610 con 0x7f6ac80fe650 2024-02-23T00:38:20.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6ac8028610 con 0x7f6ac80fe650 2024-02-23T00:38:20.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:20.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6aaffff700 10 client.37856.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac8061290 osd=2 4 2024-02-23T00:38:20.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 2 5 2024-02-23T00:38:20.656 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:38:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 10 client.37856.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 6 2024-02-23T00:38:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6813/128059790 conn(0x7f6ac8118490 0x7f6ac811a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8028810 osd=3 3 2024-02-23T00:38:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8028810 osd=3 3 2024-02-23T00:38:20.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8028810 osd=3 3 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 6 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac8028810 osd=3 3 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 3 6 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8028810 osd=3 4 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 10 client.37856.objecter _send_command 6 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ac811d4d0 con 0x7f6ac8118490 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6ac67fc700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6813/128059790 conn(0x7f6ac8118490 0x7f6ac811a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6aaffff700 10 client.37856.objecter ms_handle_connect 0x7f6ac8118490 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.653+0000 7f6ac67fc700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.87:6813/128059790 conn(0x7f6ac8118490 0x7f6ac811a880 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.654+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6ac811d4d0 con 0x7f6ac8118490 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.654+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.654+0000 7f6aaffff700 10 client.37856.objecter _finish_command 6 = system:0 2024-02-23T00:38:20.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.654+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac8028810 osd=3 4 2024-02-23T00:38:20.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.654+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 3 6 2024-02-23T00:38:20.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 10 client.37856.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:20.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 7 2024-02-23T00:38:20.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8028810 osd=3 3 2024-02-23T00:38:20.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8028810 osd=3 3 2024-02-23T00:38:20.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8028810 osd=3 3 2024-02-23T00:38:20.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 7 2024-02-23T00:38:20.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac8028810 osd=3 3 2024-02-23T00:38:20.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 3 7 2024-02-23T00:38:20.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8028810 osd=3 4 2024-02-23T00:38:20.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 10 client.37856.objecter _send_command 7 2024-02-23T00:38:20.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.668+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6ac80280e0 con 0x7f6ac8118490 2024-02-23T00:38:20.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.669+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6ac80280e0 con 0x7f6ac8118490 2024-02-23T00:38:20.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.669+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:20.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.669+0000 7f6aaffff700 10 client.37856.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.669+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac8028810 osd=3 4 2024-02-23T00:38:20.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.669+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 3 7 2024-02-23T00:38:20.673 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 10 client.37856.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 8 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6ac811bef0 0x7f6ac811c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac80598f0 osd=4 3 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac80598f0 osd=4 3 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac80598f0 osd=4 3 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 8 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac80598f0 osd=4 3 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 4 8 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac80598f0 osd=4 4 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 10 client.37856.objecter _send_command 8 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ac805a000 con 0x7f6ac811bef0 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6ac77fe700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6ac811bef0 0x7f6ac811c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6aaffff700 10 client.37856.objecter ms_handle_connect 0x7f6ac811bef0 2024-02-23T00:38:20.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.670+0000 7f6ac77fe700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6ac811bef0 0x7f6ac811c330 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:20.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.671+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6ac805a000 con 0x7f6ac811bef0 2024-02-23T00:38:20.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.671+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:38:20.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.671+0000 7f6aaffff700 10 client.37856.objecter _finish_command 8 = system:0 2024-02-23T00:38:20.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.671+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac80598f0 osd=4 4 2024-02-23T00:38:20.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.671+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 4 8 2024-02-23T00:38:20.675 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 10 client.37856.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 9 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac80598f0 osd=4 3 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac80598f0 osd=4 3 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac80598f0 osd=4 3 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 9 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac80598f0 osd=4 3 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 4 9 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac80598f0 osd=4 4 2024-02-23T00:38:20.688 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 10 client.37856.objecter _send_command 9 2024-02-23T00:38:20.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.685+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6ac811cff0 con 0x7f6ac811bef0 2024-02-23T00:38:20.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6ac811cff0 con 0x7f6ac811bef0 2024-02-23T00:38:20.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:20.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6aaffff700 10 client.37856.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac80598f0 osd=4 4 2024-02-23T00:38:20.690 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 4 9 2024-02-23T00:38:20.690 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 10 client.37856.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 10 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6ac811f650 0x7f6ac8121a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac811cff0 osd=5 3 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac811cff0 osd=5 3 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac811cff0 osd=5 3 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 10 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac811cff0 osd=5 3 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 5 10 2024-02-23T00:38:20.691 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac811cff0 osd=5 4 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 10 client.37856.objecter _send_command 10 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.687+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ac811d290 con 0x7f6ac811f650 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.688+0000 7f6ac6ffd700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6ac811f650 0x7f6ac8121a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.688+0000 7f6aaffff700 10 client.37856.objecter ms_handle_connect 0x7f6ac811f650 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.688+0000 7f6ac6ffd700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6ac811f650 0x7f6ac8121a40 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.689+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6ac811d290 con 0x7f6ac811f650 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.689+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.689+0000 7f6aaffff700 10 client.37856.objecter _finish_command 10 = system:0 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.689+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac811cff0 osd=5 4 2024-02-23T00:38:20.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.689+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 5 10 2024-02-23T00:38:20.703 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:20.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 10 client.37856.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:20.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 11 2024-02-23T00:38:20.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac811cff0 osd=5 3 2024-02-23T00:38:20.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac811cff0 osd=5 3 2024-02-23T00:38:20.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac811cff0 osd=5 3 2024-02-23T00:38:20.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 11 2024-02-23T00:38:20.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac811cff0 osd=5 3 2024-02-23T00:38:20.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 5 11 2024-02-23T00:38:20.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac811cff0 osd=5 4 2024-02-23T00:38:20.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 10 client.37856.objecter _send_command 11 2024-02-23T00:38:20.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.704+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6ac811d290 con 0x7f6ac811f650 2024-02-23T00:38:20.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6ac811d290 con 0x7f6ac811f650 2024-02-23T00:38:20.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:20.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6aaffff700 10 client.37856.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac811cff0 osd=5 4 2024-02-23T00:38:20.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 5 11 2024-02-23T00:38:20.709 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:38:20.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 10 client.37856.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:20.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 12 2024-02-23T00:38:20.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6ac8122440 0x7f6ac8122880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac811d290 osd=6 3 2024-02-23T00:38:20.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac811d290 osd=6 3 2024-02-23T00:38:20.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac811d290 osd=6 3 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 12 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac811d290 osd=6 3 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 6 12 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac811d290 osd=6 4 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 10 client.37856.objecter _send_command 12 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ac8124480 con 0x7f6ac8122440 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.706+0000 7f6ac67fc700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6ac8122440 0x7f6ac8122880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.707+0000 7f6aaffff700 10 client.37856.objecter ms_handle_connect 0x7f6ac8122440 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.707+0000 7f6ac67fc700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6ac8122440 0x7f6ac8122880 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.707+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6ac8124480 con 0x7f6ac8122440 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.707+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.707+0000 7f6aaffff700 10 client.37856.objecter _finish_command 12 = system:0 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.707+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac811d290 osd=6 4 2024-02-23T00:38:20.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.707+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 6 12 2024-02-23T00:38:20.722 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:20.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:20.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 10 client.37856.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:20.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 13 2024-02-23T00:38:20.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac811d290 osd=6 3 2024-02-23T00:38:20.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac811d290 osd=6 3 2024-02-23T00:38:20.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac811d290 osd=6 3 2024-02-23T00:38:20.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 13 2024-02-23T00:38:20.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac811d290 osd=6 3 2024-02-23T00:38:20.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 6 13 2024-02-23T00:38:20.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac811d290 osd=6 4 2024-02-23T00:38:20.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 10 client.37856.objecter _send_command 13 2024-02-23T00:38:20.726 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.723+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6ac8124480 con 0x7f6ac8122440 2024-02-23T00:38:20.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6ac8124480 con 0x7f6ac8122440 2024-02-23T00:38:20.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:20.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6aaffff700 10 client.37856.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac811d290 osd=6 4 2024-02-23T00:38:20.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 6 13 2024-02-23T00:38:20.728 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:38:20.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 10 client.37856.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:20.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 14 2024-02-23T00:38:20.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6812/373787000 conn(0x7f6ac8126c40 0x7f6ac8127080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:20.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8124480 osd=7 3 2024-02-23T00:38:20.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8124480 osd=7 3 2024-02-23T00:38:20.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8124480 osd=7 3 2024-02-23T00:38:20.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 14 2024-02-23T00:38:20.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac8124480 osd=7 3 2024-02-23T00:38:20.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 7 14 2024-02-23T00:38:20.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8124480 osd=7 4 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 10 client.37856.objecter _send_command 14 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ac812ee50 con 0x7f6ac8126c40 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.725+0000 7f6ac77fe700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6812/373787000 conn(0x7f6ac8126c40 0x7f6ac8127080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.726+0000 7f6aaffff700 10 client.37856.objecter ms_handle_connect 0x7f6ac8126c40 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.726+0000 7f6ac77fe700 1 --2- 172.21.15.87:0/1192593071 >> v2:172.21.15.171:6812/373787000 conn(0x7f6ac8126c40 0x7f6ac8127080 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.726+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6ac812ee50 con 0x7f6ac8126c40 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.726+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.726+0000 7f6aaffff700 10 client.37856.objecter _finish_command 14 = system:0 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.726+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac8124480 osd=7 4 2024-02-23T00:38:20.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.726+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 7 14 2024-02-23T00:38:20.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 10 client.37856.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:20.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign -1 15 2024-02-23T00:38:20.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8124480 osd=7 3 2024-02-23T00:38:20.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8124480 osd=7 3 2024-02-23T00:38:20.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 20 client.37856.objecter _get_session s=0x7f6ac8124480 osd=7 3 2024-02-23T00:38:20.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 15 client.37856.objecter _session_command_op_remove -1 15 2024-02-23T00:38:20.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 20 client.37856.objecter get_session s=0x7f6ac8124480 osd=7 3 2024-02-23T00:38:20.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 15 client.37856.objecter _session_command_op_assign 7 15 2024-02-23T00:38:20.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 20 client.37856.objecter put_session s=0x7f6ac8124480 osd=7 4 2024-02-23T00:38:20.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 10 client.37856.objecter _send_command 15 2024-02-23T00:38:20.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.741+0000 7f6acd832700 1 -- 172.21.15.87:0/1192593071 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6ac8127880 con 0x7f6ac8126c40 2024-02-23T00:38:20.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.743+0000 7f6aaffff700 1 -- 172.21.15.87:0/1192593071 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6ac8127880 con 0x7f6ac8126c40 2024-02-23T00:38:20.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.743+0000 7f6aaffff700 10 client.37856.objecter ms_dispatch 0x7f6ac8001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:20.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.743+0000 7f6aaffff700 10 client.37856.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.743+0000 7f6aaffff700 20 client.37856.objecter put_session s=0x7f6ac8124480 osd=7 4 2024-02-23T00:38:20.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:20.743+0000 7f6aaffff700 15 client.37856.objecter _session_command_op_remove 7 15 2024-02-23T00:38:20.745 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:38:20.745 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:38:20.745 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.746 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.746 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.746 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.746 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.746 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.746 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:38:20.826 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:20.856 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:21.125 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:38:21.390 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:38:21.390 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:38:21.538 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:38:21.566 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:19.522937+0000 osd.2 (osd.2) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:21.566 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:19.523116+0000 osd.2 (osd.2) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:21.566 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:19.698183+0000 mgr.y (mgr.24437) 461 : cluster [DBG] pgmap v739: 248 pgs: 10 unknown, 26 active+undersized+degraded, 65 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1194 objects degraded (10.385%) 2024-02-23T00:38:21.566 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:20.268998+0000 mon.a (mon.0) 2767 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-23T00:38:21.566 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:20.269079+0000 mon.a (mon.0) 2768 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "32"}]': finished 2024-02-23T00:38:21.566 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:20.269124+0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e483: 8 total, 7 up, 7 in 2024-02-23T00:38:21.566 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:20.353571+0000 mon.b (mon.1) 395 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:20.354640+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:20.618998+0000 mon.c (mon.2) 448 : audit [DBG] from='client.? 172.21.15.87:0/1192593071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:19.522937+0000 osd.2 (osd.2) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:19.523116+0000 osd.2 (osd.2) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:19.698183+0000 mgr.y (mgr.24437) 461 : cluster [DBG] pgmap v739: 248 pgs: 10 unknown, 26 active+undersized+degraded, 65 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1194 objects degraded (10.385%) 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:20.268998+0000 mon.a (mon.0) 2767 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:20.269079+0000 mon.a (mon.0) 2768 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "32"}]': finished 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:20.269124+0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e483: 8 total, 7 up, 7 in 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:20.353571+0000 mon.b (mon.1) 395 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:20.354640+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-23T00:38:21.567 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:20.618998+0000 mon.c (mon.2) 448 : audit [DBG] from='client.? 172.21.15.87:0/1192593071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:21.605 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:21.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:38:21.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:38:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:19.522937+0000 osd.2 (osd.2) 185 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:19.523116+0000 osd.2 (osd.2) 186 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:19.698183+0000 mgr.y (mgr.24437) 461 : cluster [DBG] pgmap v739: 248 pgs: 10 unknown, 26 active+undersized+degraded, 65 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1194 objects degraded (10.385%) 2024-02-23T00:38:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:20.268998+0000 mon.a (mon.0) 2767 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-23T00:38:21.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:20.269079+0000 mon.a (mon.0) 2768 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num_actual", "val": "32"}]': finished 2024-02-23T00:38:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:20.269124+0000 mon.a (mon.0) 2769 : cluster [DBG] osdmap e483: 8 total, 7 up, 7 in 2024-02-23T00:38:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:20.353571+0000 mon.b (mon.1) 395 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-23T00:38:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:20.354640+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-23T00:38:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:20.618998+0000 mon.c (mon.2) 448 : audit [DBG] from='client.? 172.21.15.87:0/1192593071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:21.709 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:21.738 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:21.760 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:21.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:21.819 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:21.863 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:21.890 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:22.063 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-23T00:38:22.142 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:22.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:22.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:22.297 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:22.402 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:22.427 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:22.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:21.011637+0000 mgr.y (mgr.24437) 462 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:22.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:21.182548+0000 osd.6 (osd.6) 206 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:22.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:21.182787+0000 osd.6 (osd.6) 207 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:22.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:21.267988+0000 mon.a (mon.0) 2771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-23T00:38:22.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:21.268063+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e484: 8 total, 7 up, 7 in 2024-02-23T00:38:22.430 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:21.381898+0000 mon.b (mon.1) 396 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:22.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:21.382071+0000 mon.a (mon.0) 2773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:22.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.000967+0000 osd.1 (osd.1) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:22.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.001145+0000 osd.1 (osd.1) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:22.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.123582+0000 mon.a (mon.0) 2774 : cluster [WRN] Health check update: Degraded data redundancy: 124/1194 objects degraded (10.385%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:22.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.129166+0000 osd.4 (osd.4) 235 : cluster [DBG] 4.1b starting backfill to osd.2 from (0'0,0'0] MAX to 453'22 2024-02-23T00:38:22.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:22.129573+0000 mon.a (mon.0) 2775 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:22.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.129650+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e485: 8 total, 7 up, 7 in 2024-02-23T00:38:22.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.148454+0000 osd.5 (osd.5) 195 : cluster [DBG] 4.18 starting backfill to osd.1 from (0'0,0'0] MAX to 453'38 2024-02-23T00:38:22.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.148571+0000 osd.5 (osd.5) 196 : cluster [DBG] 4.18 starting backfill to osd.4 from (0'0,0'0] MAX to 453'38 2024-02-23T00:38:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:22.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:22] "GET /metrics HTTP/1.1" 200 188726 "" "Prometheus/2.33.4" 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:21.011637+0000 mgr.y (mgr.24437) 462 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:21.182548+0000 osd.6 (osd.6) 206 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:21.182787+0000 osd.6 (osd.6) 207 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:21.267988+0000 mon.a (mon.0) 2771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:21.268063+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e484: 8 total, 7 up, 7 in 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:21.381898+0000 mon.b (mon.1) 396 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:21.382071+0000 mon.a (mon.0) 2773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.000967+0000 osd.1 (osd.1) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.001145+0000 osd.1 (osd.1) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:22.433 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.123582+0000 mon.a (mon.0) 2774 : cluster [WRN] Health check update: Degraded data redundancy: 124/1194 objects degraded (10.385%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:22.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.129166+0000 osd.4 (osd.4) 235 : cluster [DBG] 4.1b starting backfill to osd.2 from (0'0,0'0] MAX to 453'22 2024-02-23T00:38:22.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:22.129573+0000 mon.a (mon.0) 2775 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:22.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.129650+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e485: 8 total, 7 up, 7 in 2024-02-23T00:38:22.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.148454+0000 osd.5 (osd.5) 195 : cluster [DBG] 4.18 starting backfill to osd.1 from (0'0,0'0] MAX to 453'38 2024-02-23T00:38:22.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.148571+0000 osd.5 (osd.5) 196 : cluster [DBG] 4.18 starting backfill to osd.4 from (0'0,0'0] MAX to 453'38 2024-02-23T00:38:22.534 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:22.560 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:22.648 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:21.011637+0000 mgr.y (mgr.24437) 462 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:22.648 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:21.182548+0000 osd.6 (osd.6) 206 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:22.648 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:21.182787+0000 osd.6 (osd.6) 207 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:22.648 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:21.267988+0000 mon.a (mon.0) 2771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:21.268063+0000 mon.a (mon.0) 2772 : cluster [DBG] osdmap e484: 8 total, 7 up, 7 in 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:21.381898+0000 mon.b (mon.1) 396 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:21.382071+0000 mon.a (mon.0) 2773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.000967+0000 osd.1 (osd.1) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.001145+0000 osd.1 (osd.1) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.123582+0000 mon.a (mon.0) 2774 : cluster [WRN] Health check update: Degraded data redundancy: 124/1194 objects degraded (10.385%), 26 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.129166+0000 osd.4 (osd.4) 235 : cluster [DBG] 4.1b starting backfill to osd.2 from (0'0,0'0] MAX to 453'22 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:22.129573+0000 mon.a (mon.0) 2775 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.129650+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e485: 8 total, 7 up, 7 in 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.148454+0000 osd.5 (osd.5) 195 : cluster [DBG] 4.18 starting backfill to osd.1 from (0'0,0'0] MAX to 453'38 2024-02-23T00:38:22.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.148571+0000 osd.5 (osd.5) 196 : cluster [DBG] 4.18 starting backfill to osd.4 from (0'0,0'0] MAX to 453'38 2024-02-23T00:38:22.760 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:22.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:22.865 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:22.891 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:22.915 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:22.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:23.022 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:23.049 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:38:23.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:38:23.160 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.meta 2024-02-23T00:38:23.160 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.meta by 10 min 8 2024-02-23T00:38:23.161 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:38:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:23.163 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:23.163 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:23.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:23.263 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:23.292 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:23.333 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:23.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:23.443 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:23.470 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:23.620 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:38:23.674 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:23.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:23.778 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:23.802 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:38:23.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:21.698858+0000 mgr.y (mgr.24437) 463 : cluster [DBG] pgmap v742: 248 pgs: 10 peering, 27 active+undersized+degraded, 64 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 114/1074 objects degraded (10.615%) 2024-02-23T00:38:23.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.128540+0000 osd.1 (osd.1) 219 : cluster [DBG] 4.19 starting backfill to osd.2 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:23.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.128560+0000 osd.1 (osd.1) 220 : cluster [DBG] 4.1a starting backfill to osd.2 from (0'0,0'0] MAX to 476'63 2024-02-23T00:38:23.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.128711+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.1a starting backfill to osd.4 from (0'0,0'0] MAX to 476'63 2024-02-23T00:38:23.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.129202+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.1c starting backfill to osd.2 from (0'0,0'0] MAX to 453'37 2024-02-23T00:38:23.803 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.129440+0000 osd.1 (osd.1) 223 : cluster [DBG] 4.1f starting backfill to osd.5 from (0'0,0'0] MAX to 453'42 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.129542+0000 osd.1 (osd.1) 224 : cluster [DBG] 4.1f starting backfill to osd.6 from (0'0,0'0] MAX to 453'42 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.129986+0000 osd.1 (osd.1) 225 : cluster [DBG] 4.19 starting backfill to osd.5 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.130544+0000 osd.1 (osd.1) 226 : cluster [DBG] 4.16 starting backfill to osd.7 from (0'0,0'0] MAX to 453'39 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.130695+0000 osd.1 (osd.1) 227 : cluster [DBG] 4.1d starting backfill to osd.6 from (0'0,0'0] MAX to 453'44 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.132046+0000 osd.1 (osd.1) 228 : cluster [DBG] 4.1e starting backfill to osd.7 from (0'0,0'0] MAX to 453'40 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.132109+0000 osd.1 (osd.1) 229 : cluster [DBG] 4.17 starting backfill to osd.2 from (0'0,0'0] MAX to 453'29 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:22.132226+0000 osd.1 (osd.1) 230 : cluster [DBG] 4.17 starting backfill to osd.7 from (0'0,0'0] MAX to 453'29 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:22.330898+0000 mon.b (mon.1) 397 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:22.331075+0000 mon.a (mon.0) 2777 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:22.331709+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-77"}]: dispatch 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:22.331747+0000 mon.a (mon.0) 2778 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-77"}]: dispatch 2024-02-23T00:38:23.804 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:23.168334+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e486: 8 total, 7 up, 7 in 2024-02-23T00:38:23.805 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:21.698858+0000 mgr.y (mgr.24437) 463 : cluster [DBG] pgmap v742: 248 pgs: 10 peering, 27 active+undersized+degraded, 64 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 114/1074 objects degraded (10.615%) 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.128540+0000 osd.1 (osd.1) 219 : cluster [DBG] 4.19 starting backfill to osd.2 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.128560+0000 osd.1 (osd.1) 220 : cluster [DBG] 4.1a starting backfill to osd.2 from (0'0,0'0] MAX to 476'63 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.128711+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.1a starting backfill to osd.4 from (0'0,0'0] MAX to 476'63 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.129202+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.1c starting backfill to osd.2 from (0'0,0'0] MAX to 453'37 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.129440+0000 osd.1 (osd.1) 223 : cluster [DBG] 4.1f starting backfill to osd.5 from (0'0,0'0] MAX to 453'42 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.129542+0000 osd.1 (osd.1) 224 : cluster [DBG] 4.1f starting backfill to osd.6 from (0'0,0'0] MAX to 453'42 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.129986+0000 osd.1 (osd.1) 225 : cluster [DBG] 4.19 starting backfill to osd.5 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.130544+0000 osd.1 (osd.1) 226 : cluster [DBG] 4.16 starting backfill to osd.7 from (0'0,0'0] MAX to 453'39 2024-02-23T00:38:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.130695+0000 osd.1 (osd.1) 227 : cluster [DBG] 4.1d starting backfill to osd.6 from (0'0,0'0] MAX to 453'44 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.132046+0000 osd.1 (osd.1) 228 : cluster [DBG] 4.1e starting backfill to osd.7 from (0'0,0'0] MAX to 453'40 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.132109+0000 osd.1 (osd.1) 229 : cluster [DBG] 4.17 starting backfill to osd.2 from (0'0,0'0] MAX to 453'29 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:22.132226+0000 osd.1 (osd.1) 230 : cluster [DBG] 4.17 starting backfill to osd.7 from (0'0,0'0] MAX to 453'29 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:22.330898+0000 mon.b (mon.1) 397 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:22.331075+0000 mon.a (mon.0) 2777 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:22.331709+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-77"}]: dispatch 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:22.331747+0000 mon.a (mon.0) 2778 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-77"}]: dispatch 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:23.168334+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e486: 8 total, 7 up, 7 in 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:21.698858+0000 mgr.y (mgr.24437) 463 : cluster [DBG] pgmap v742: 248 pgs: 10 peering, 27 active+undersized+degraded, 64 active+undersized, 147 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 114/1074 objects degraded (10.615%) 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.128540+0000 osd.1 (osd.1) 219 : cluster [DBG] 4.19 starting backfill to osd.2 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.128560+0000 osd.1 (osd.1) 220 : cluster [DBG] 4.1a starting backfill to osd.2 from (0'0,0'0] MAX to 476'63 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.128711+0000 osd.1 (osd.1) 221 : cluster [DBG] 4.1a starting backfill to osd.4 from (0'0,0'0] MAX to 476'63 2024-02-23T00:38:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.129202+0000 osd.1 (osd.1) 222 : cluster [DBG] 4.1c starting backfill to osd.2 from (0'0,0'0] MAX to 453'37 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.129440+0000 osd.1 (osd.1) 223 : cluster [DBG] 4.1f starting backfill to osd.5 from (0'0,0'0] MAX to 453'42 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.129542+0000 osd.1 (osd.1) 224 : cluster [DBG] 4.1f starting backfill to osd.6 from (0'0,0'0] MAX to 453'42 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.129986+0000 osd.1 (osd.1) 225 : cluster [DBG] 4.19 starting backfill to osd.5 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.130544+0000 osd.1 (osd.1) 226 : cluster [DBG] 4.16 starting backfill to osd.7 from (0'0,0'0] MAX to 453'39 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.130695+0000 osd.1 (osd.1) 227 : cluster [DBG] 4.1d starting backfill to osd.6 from (0'0,0'0] MAX to 453'44 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.132046+0000 osd.1 (osd.1) 228 : cluster [DBG] 4.1e starting backfill to osd.7 from (0'0,0'0] MAX to 453'40 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.132109+0000 osd.1 (osd.1) 229 : cluster [DBG] 4.17 starting backfill to osd.2 from (0'0,0'0] MAX to 453'29 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:22.132226+0000 osd.1 (osd.1) 230 : cluster [DBG] 4.17 starting backfill to osd.7 from (0'0,0'0] MAX to 453'29 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:22.330898+0000 mon.b (mon.1) 397 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:22.331075+0000 mon.a (mon.0) 2777 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:22.331709+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-77"}]: dispatch 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:22.331747+0000 mon.a (mon.0) 2778 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-77"}]: dispatch 2024-02-23T00:38:23.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:23.168334+0000 mon.a (mon.0) 2779 : cluster [DBG] osdmap e486: 8 total, 7 up, 7 in 2024-02-23T00:38:23.904 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:23.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:23.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.920+0000 7f508619e700 1 -- 172.21.15.87:0/3961267761 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50800fda80 msgr2=0x7f50800fde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.920+0000 7f508619e700 1 --2- 172.21.15.87:0/3961267761 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50800fda80 0x7f50800fde60 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 rx=0x7f507c004870 tx=0x7f507c009cb0).stop 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 -- 172.21.15.87:0/3961267761 shutdown_connections 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 --2- 172.21.15.87:0/3961267761 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50800f68f0 0x7f5080104860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 --2- 172.21.15.87:0/3961267761 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50800fda80 0x7f50800fde60 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 --2- 172.21.15.87:0/3961267761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50800fe3e0 0x7f50800f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 -- 172.21.15.87:0/3961267761 >> 172.21.15.87:0/3961267761 conn(0x7f508006c1b0 msgr2=0x7f508006c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 -- 172.21.15.87:0/3961267761 shutdown_connections 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 -- 172.21.15.87:0/3961267761 wait complete. 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 librados: starting msgr at 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 librados: starting objecter 2024-02-23T00:38:23.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 Processor -- start 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 -- start start 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 librados: setting wanted keys 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.921+0000 7f508619e700 1 librados: calling monclient init 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50800f68f0 0x7f508010e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50800fda80 0x7f508010e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50800fe3e0 0x7f5080108280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5080108770 con 0x7f50800f68f0 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50801088e0 con 0x7f50800fda80 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5080108aa0 con 0x7f50800fe3e0 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508519c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50800f68f0 0x7f508010e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508519c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50800f68f0 0x7f508010e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59978/0 (socket says 172.21.15.87:59978) 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508519c700 1 -- 172.21.15.87:0/3306146294 learned_addr learned my addr 172.21.15.87:0/3306146294 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:23.940 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508599d700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50800fe3e0 0x7f5080108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508499b700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50800fda80 0x7f508010e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508519c700 1 -- 172.21.15.87:0/3306146294 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50800fe3e0 msgr2=0x7f5080108280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508519c700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50800fe3e0 0x7f5080108280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508519c700 1 -- 172.21.15.87:0/3306146294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50800fda80 msgr2=0x7f508010e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508519c700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50800fda80 0x7f508010e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508519c700 1 -- 172.21.15.87:0/3306146294 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5070005040 con 0x7f50800f68f0 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508599d700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50800fe3e0 0x7f5080108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508519c700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50800f68f0 0x7f508010e0e0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 rx=0x7f507c009eb0 tx=0x7f507c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f50767fc700 10 client.?.objecter ms_handle_connect 0x7f50800f68f0 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f50767fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f50767fc700 1 -- 172.21.15.87:0/3306146294 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c00e070 con 0x7f50800f68f0 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c005040 con 0x7f50800f68f0 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 10 client.27396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5080108f40 con 0x7f50800f68f0 2024-02-23T00:38:23.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f508619e700 1 librados: init done 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f50767fc700 1 -- 172.21.15.87:0/3306146294 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f507c01e420 con 0x7f50800f68f0 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.922+0000 7f50767fc700 1 -- 172.21.15.87:0/3306146294 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c019070 con 0x7f50800f68f0 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.923+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5064004f50 con 0x7f50800f68f0 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f50767fc700 1 -- 172.21.15.87:0/3306146294 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f507c01e850 con 0x7f50800f68f0 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f50767fc700 1 --2- 172.21.15.87:0/3306146294 >> v2:172.21.15.87:6800/1231597463 conn(0x7f506c062c90 0x7f506c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f50767fc700 1 -- 172.21.15.87:0/3306146294 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(486..486 src has 173..486) v4 ==== 8021+0+0 (secure 0 0 0) 0x7f507c07d9f0 con 0x7f50800f68f0 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f50767fc700 10 client.27396.objecter ms_dispatch 0x7f5080001550 osd_map(486..486 src has 173..486) v4 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f50767fc700 3 client.27396.objecter handle_osd_map got epochs [486,486] > 0 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f50767fc700 3 client.27396.objecter handle_osd_map decoding full epoch 486 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f508499b700 1 --2- 172.21.15.87:0/3306146294 >> v2:172.21.15.87:6800/1231597463 conn(0x7f506c062c90 0x7f506c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f50767fc700 20 client.27396.objecter dump_active .. 0 homeless 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f50767fc700 10 client.27396.objecter ms_handle_connect 0x7f506c062c90 2024-02-23T00:38:23.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.924+0000 7f508499b700 1 --2- 172.21.15.87:0/3306146294 >> v2:172.21.15.87:6800/1231597463 conn(0x7f506c062c90 0x7f506c065150 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f5080109a20 tx=0x7f507000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:23.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:23.929+0000 7f50767fc700 1 -- 172.21.15.87:0/3306146294 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f507c050f20 con 0x7f50800f68f0 2024-02-23T00:38:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:24.007 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:24.039 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:24.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.138+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f5064005200 con 0x7f50800f68f0 2024-02-23T00:38:24.142 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:38:24.142 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:38:24.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.171+0000 7f50767fc700 1 -- 172.21.15.87:0/3306146294 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v487) v1 ==== 97+0+0 (secure 0 0 0) 0x7f507c050b10 con 0x7f50800f68f0 2024-02-23T00:38:24.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.174+0000 7f508619e700 10 librados: watch_flush enter 2024-02-23T00:38:24.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.185+0000 7f508619e700 10 librados: watch_flush exit 2024-02-23T00:38:24.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.185+0000 7f508619e700 20 client.27396.objecter shutdown clearing up homeless session... 2024-02-23T00:38:24.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.185+0000 7f508619e700 10 client.27396.objecter successfully canceled tick 2024-02-23T00:38:24.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.186+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 >> v2:172.21.15.87:6800/1231597463 conn(0x7f506c062c90 msgr2=0x7f506c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:24.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.186+0000 7f508619e700 1 --2- 172.21.15.87:0/3306146294 >> v2:172.21.15.87:6800/1231597463 conn(0x7f506c062c90 0x7f506c065150 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 rx=0x7f5080109a20 tx=0x7f507000b040).stop 2024-02-23T00:38:24.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.186+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50800f68f0 msgr2=0x7f508010e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:24.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.186+0000 7f508619e700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50800f68f0 0x7f508010e0e0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 rx=0x7f507c009eb0 tx=0x7f507c00b040).stop 2024-02-23T00:38:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.193+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 shutdown_connections 2024-02-23T00:38:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.193+0000 7f508619e700 1 --2- 172.21.15.87:0/3306146294 >> v2:172.21.15.87:6800/1231597463 conn(0x7f506c062c90 0x7f506c065150 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.193+0000 7f508619e700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f50800fe3e0 0x7f5080108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.193+0000 7f508619e700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f50800f68f0 0x7f508010e0e0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.193+0000 7f508619e700 1 --2- 172.21.15.87:0/3306146294 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f50800fda80 0x7f508010e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:24.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.193+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 >> 172.21.15.87:0/3306146294 conn(0x7f508006c1b0 msgr2=0x7f50800fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:24.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.194+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 shutdown_connections 2024-02-23T00:38:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.197+0000 7f508619e700 1 -- 172.21.15.87:0/3306146294 wait complete. 2024-02-23T00:38:24.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:24.197+0000 7f508619e700 1 librados: shutdown 2024-02-23T00:38:24.218 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:38:24.301 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:24.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:24.415 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:24.442 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:24.497 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:24.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:23.699723+0000 mgr.y (mgr.24437) 464 : cluster [DBG] pgmap v745: 216 pgs: 10 peering, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 113/1071 objects degraded (10.551%) 2024-02-23T00:38:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:24.140206+0000 mon.a (mon.0) 2780 : audit [INF] from='client.? 172.21.15.87:0/3306146294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:38:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:24.165640+0000 mon.a (mon.0) 2781 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:24.171399+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? 172.21.15.87:0/3306146294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:38:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:24.171460+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e487: 8 total, 7 up, 7 in 2024-02-23T00:38:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:24.192127+0000 mon.b (mon.1) 399 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:24.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:24.192553+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:24.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:23.699723+0000 mgr.y (mgr.24437) 464 : cluster [DBG] pgmap v745: 216 pgs: 10 peering, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 113/1071 objects degraded (10.551%) 2024-02-23T00:38:24.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:24.140206+0000 mon.a (mon.0) 2780 : audit [INF] from='client.? 172.21.15.87:0/3306146294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:38:24.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:24.165640+0000 mon.a (mon.0) 2781 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:24.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:24.171399+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? 172.21.15.87:0/3306146294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:38:24.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:24.171460+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e487: 8 total, 7 up, 7 in 2024-02-23T00:38:24.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:24.192127+0000 mon.b (mon.1) 399 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:24.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:24.192553+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:24.600 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:24.628 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:24.866 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:23.699723+0000 mgr.y (mgr.24437) 464 : cluster [DBG] pgmap v745: 216 pgs: 10 peering, 26 active+undersized+degraded, 52 active+undersized, 128 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 113/1071 objects degraded (10.551%) 2024-02-23T00:38:24.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:24.140206+0000 mon.a (mon.0) 2780 : audit [INF] from='client.? 172.21.15.87:0/3306146294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:38:24.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:24.165640+0000 mon.a (mon.0) 2781 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:24.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:24.171399+0000 mon.a (mon.0) 2782 : audit [INF] from='client.? 172.21.15.87:0/3306146294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:38:24.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:24.171460+0000 mon.a (mon.0) 2783 : cluster [DBG] osdmap e487: 8 total, 7 up, 7 in 2024-02-23T00:38:24.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:24.192127+0000 mon.b (mon.1) 399 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:24.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:24.192553+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:24.885 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-23T00:38:24.977 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:24.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:25.080 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:25.110 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:25.153 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:25.658 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:25.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:25.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:24.652251+0000 mon.a (mon.0) 2785 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:25.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:25.174286+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:25.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:25.174365+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e488: 8 total, 7 up, 7 in 2024-02-23T00:38:25.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:25.182336+0000 mon.b (mon.1) 400 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:25.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:25.186516+0000 mon.b (mon.1) 401 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:25.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:25.186691+0000 mon.a (mon.0) 2788 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:25.763 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:25.790 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:26.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:24.652251+0000 mon.a (mon.0) 2785 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:26.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:25.174286+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:25.174365+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e488: 8 total, 7 up, 7 in 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:25.182336+0000 mon.b (mon.1) 400 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:25.186516+0000 mon.b (mon.1) 401 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:25.186691+0000 mon.a (mon.0) 2788 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:24.652251+0000 mon.a (mon.0) 2785 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:25.174286+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:25.174365+0000 mon.a (mon.0) 2787 : cluster [DBG] osdmap e488: 8 total, 7 up, 7 in 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:25.182336+0000 mon.b (mon.1) 400 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:25.186516+0000 mon.b (mon.1) 401 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:26.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:25.186691+0000 mon.a (mon.0) 2788 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:26.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.626+0000 7f07b13c1700 1 -- 172.21.15.87:0/777440244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac0f7330 msgr2=0x7f07ac103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:26.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.626+0000 7f07b13c1700 1 --2- 172.21.15.87:0/777440244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac0f7330 0x7f07ac103a00 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 rx=0x7f07a4009be0 tx=0x7f07a4004d10).stop 2024-02-23T00:38:26.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.626+0000 7f07b13c1700 1 -- 172.21.15.87:0/777440244 shutdown_connections 2024-02-23T00:38:26.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.626+0000 7f07b13c1700 1 --2- 172.21.15.87:0/777440244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac0f7330 0x7f07ac103a00 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.626+0000 7f07b13c1700 1 --2- 172.21.15.87:0/777440244 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f07ac0f89a0 0x7f07ac0f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.626+0000 7f07b13c1700 1 --2- 172.21.15.87:0/777440244 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07ac0f9270 0x7f07ac0f6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.626+0000 7f07b13c1700 1 -- 172.21.15.87:0/777440244 >> 172.21.15.87:0/777440244 conn(0x7f07ac0f26f0 msgr2=0x7f07ac0f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.626+0000 7f07b13c1700 1 -- 172.21.15.87:0/777440244 shutdown_connections 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.626+0000 7f07b13c1700 1 -- 172.21.15.87:0/777440244 wait complete. 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 librados: starting msgr at 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 librados: starting objecter 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 Processor -- start 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 -- start start 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 librados: setting wanted keys 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 librados: calling monclient init 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac0f7330 0x7f07ac1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f07ac0f89a0 0x7f07ac1096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07ac0f9270 0x7f07ac109c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:26.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac114390 con 0x7f07ac0f7330 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac114500 con 0x7f07ac0f9270 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07b13c1700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac1146c0 con 0x7f07ac0f89a0 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07aa7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f07ac0f89a0 0x7f07ac1096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07aaffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac0f7330 0x7f07ac1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07aa7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f07ac0f89a0 0x7f07ac1096c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:36800/0 (socket says 172.21.15.87:36800) 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07aa7fc700 1 -- 172.21.15.87:0/3218974970 learned_addr learned my addr 172.21.15.87:0/3218974970 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.627+0000 7f07ab7fe700 1 --2- 172.21.15.87:0/3218974970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07ac0f9270 0x7f07ac109c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07aaffd700 1 -- 172.21.15.87:0/3218974970 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f07ac0f89a0 msgr2=0x7f07ac1096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07aaffd700 1 --2- 172.21.15.87:0/3218974970 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f07ac0f89a0 0x7f07ac1096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07aaffd700 1 -- 172.21.15.87:0/3218974970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07ac0f9270 msgr2=0x7f07ac109c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07aaffd700 1 --2- 172.21.15.87:0/3218974970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07ac0f9270 0x7f07ac109c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07aaffd700 1 -- 172.21.15.87:0/3218974970 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0794005040 con 0x7f07ac0f7330 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07aa7fc700 1 --2- 172.21.15.87:0/3218974970 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f07ac0f89a0 0x7f07ac1096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:26.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07aaffd700 1 --2- 172.21.15.87:0/3218974970 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac0f7330 0x7f07ac1091d0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7f0794004920 tx=0x7f0794009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07a3fff700 10 client.?.objecter ms_handle_connect 0x7f07ac0f7330 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07a3fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07a3fff700 1 -- 172.21.15.87:0/3218974970 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079400e030 con 0x7f07ac0f7330 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07a3fff700 1 -- 172.21.15.87:0/3218974970 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f079400f040 con 0x7f07ac0f7330 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a40097c0 con 0x7f07ac0f7330 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07a3fff700 1 -- 172.21.15.87:0/3218974970 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794014850 con 0x7f07ac0f7330 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07b13c1700 10 client.27402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ac114dd0 con 0x7f07ac0f7330 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.628+0000 7f07b13c1700 1 librados: init done 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.629+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ac05ee80 con 0x7f07ac0f7330 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.629+0000 7f07a3fff700 1 -- 172.21.15.87:0/3218974970 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0794010500 con 0x7f07ac0f7330 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.630+0000 7f07a3fff700 1 --2- 172.21.15.87:0/3218974970 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0798062c90 0x7f0798065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.630+0000 7f07a3fff700 1 -- 172.21.15.87:0/3218974970 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(489..489 src has 173..489) v4 ==== 8422+0+0 (secure 0 0 0) 0x7f079407d3e0 con 0x7f07ac0f7330 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.630+0000 7f07a3fff700 10 client.27402.objecter ms_dispatch 0x7f07ac001550 osd_map(489..489 src has 173..489) v4 2024-02-23T00:38:26.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.630+0000 7f07a3fff700 3 client.27402.objecter handle_osd_map got epochs [489,489] > 0 2024-02-23T00:38:26.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.630+0000 7f07a3fff700 3 client.27402.objecter handle_osd_map decoding full epoch 489 2024-02-23T00:38:26.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.630+0000 7f07aa7fc700 1 --2- 172.21.15.87:0/3218974970 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0798062c90 0x7f0798065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:26.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.630+0000 7f07a3fff700 20 client.27402.objecter dump_active .. 0 homeless 2024-02-23T00:38:26.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.630+0000 7f07a3fff700 10 client.27402.objecter ms_handle_connect 0x7f0798062c90 2024-02-23T00:38:26.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.630+0000 7f07aa7fc700 1 --2- 172.21.15.87:0/3218974970 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0798062c90 0x7f0798065150 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7f07ac061780 tx=0x7f079c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:26.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.634+0000 7f07a3fff700 1 -- 172.21.15.87:0/3218974970 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f07940507c0 con 0x7f07ac0f7330 2024-02-23T00:38:26.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.800+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07ac0614c0 con 0x7f0798062c90 2024-02-23T00:38:26.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.816+0000 7f07a3fff700 1 -- 172.21.15.87:0/3218974970 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+622717 (secure 0 0 0) 0x7f07ac0614c0 con 0x7f0798062c90 2024-02-23T00:38:26.827 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 10 librados: watch_flush enter 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 10 librados: watch_flush exit 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 20 client.27402.objecter shutdown clearing up homeless session... 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 10 client.27402.objecter successfully canceled tick 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0798062c90 msgr2=0x7f0798065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 --2- 172.21.15.87:0/3218974970 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0798062c90 0x7f0798065150 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 rx=0x7f07ac061780 tx=0x7f079c00b040).stop 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac0f7330 msgr2=0x7f07ac1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 --2- 172.21.15.87:0/3218974970 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac0f7330 0x7f07ac1091d0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 rx=0x7f0794004920 tx=0x7f0794009cb0).stop 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 shutdown_connections 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 --2- 172.21.15.87:0/3218974970 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0798062c90 0x7f0798065150 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 --2- 172.21.15.87:0/3218974970 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f07ac0f9270 0x7f07ac109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:26.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 --2- 172.21.15.87:0/3218974970 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f07ac0f7330 0x7f07ac1091d0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:26.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 --2- 172.21.15.87:0/3218974970 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f07ac0f89a0 0x7f07ac1096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:26.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 >> 172.21.15.87:0/3218974970 conn(0x7f07ac0f26f0 msgr2=0x7f07ac0f2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:26.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 shutdown_connections 2024-02-23T00:38:26.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 -- 172.21.15.87:0/3218974970 wait complete. 2024-02-23T00:38:26.835 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:26.820+0000 7f07b13c1700 1 librados: shutdown 2024-02-23T00:38:26.835 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:38:27.104 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:38:27.104 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:38:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:25.701143+0000 mgr.y (mgr.24437) 465 : cluster [DBG] pgmap v748: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 68 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 88 KiB/s rd, 5.5 KiB/s wr, 174 op/s; 120/1194 objects degraded (10.050%); 84/1194 objects misplaced (7.035%); 343 B/s, 21 objects/s recovering 2024-02-23T00:38:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:25.977965+0000 osd.3 (osd.3) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:25.978198+0000 osd.3 (osd.3) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:26.172429+0000 mon.a (mon.0) 2789 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:26.172509+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e489: 8 total, 7 up, 7 in 2024-02-23T00:38:27.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:26.176899+0000 mon.b (mon.1) 402 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:27.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:26.177344+0000 mon.a (mon.0) 2791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:27.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:26.651899+0000 mon.a (mon.0) 2792 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:38:27.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:27.126263+0000 mon.a (mon.0) 2793 : cluster [WRN] Health check update: Degraded data redundancy: 120/1194 objects degraded (10.050%), 28 pgs degraded, 75 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:25.701143+0000 mgr.y (mgr.24437) 465 : cluster [DBG] pgmap v748: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 68 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 88 KiB/s rd, 5.5 KiB/s wr, 174 op/s; 120/1194 objects degraded (10.050%); 84/1194 objects misplaced (7.035%); 343 B/s, 21 objects/s recovering 2024-02-23T00:38:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:25.977965+0000 osd.3 (osd.3) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:25.978198+0000 osd.3 (osd.3) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:26.172429+0000 mon.a (mon.0) 2789 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:26.172509+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e489: 8 total, 7 up, 7 in 2024-02-23T00:38:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:26.176899+0000 mon.b (mon.1) 402 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:26.177344+0000 mon.a (mon.0) 2791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:26.651899+0000 mon.a (mon.0) 2792 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:38:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:27.126263+0000 mon.a (mon.0) 2793 : cluster [WRN] Health check update: Degraded data redundancy: 120/1194 objects degraded (10.050%), 28 pgs degraded, 75 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:25.701143+0000 mgr.y (mgr.24437) 465 : cluster [DBG] pgmap v748: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 68 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 88 KiB/s rd, 5.5 KiB/s wr, 174 op/s; 120/1194 objects degraded (10.050%); 84/1194 objects misplaced (7.035%); 343 B/s, 21 objects/s recovering 2024-02-23T00:38:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:25.977965+0000 osd.3 (osd.3) 153 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:25.978198+0000 osd.3 (osd.3) 154 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:26.172429+0000 mon.a (mon.0) 2789 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:26.172509+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e489: 8 total, 7 up, 7 in 2024-02-23T00:38:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:26.176899+0000 mon.b (mon.1) 402 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:26.177344+0000 mon.a (mon.0) 2791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:26.651899+0000 mon.a (mon.0) 2792 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:38:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:27.126263+0000 mon.a (mon.0) 2793 : cluster [WRN] Health check update: Degraded data redundancy: 120/1194 objects degraded (10.050%), 28 pgs degraded, 75 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:27.639 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:27.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:27.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:27.743 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:27.773 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:27.794 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:27.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:27.810 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-23T00:38:27.898 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:27.927 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:28.112 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:28.123 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:38:28.185 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:26.801618+0000 mgr.y (mgr.24437) 466 : audit [DBG] from='client.27402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:38:28.186 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:27.176758+0000 mon.a (mon.0) 2794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:38:28.186 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:27.176837+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e490: 8 total, 7 up, 7 in 2024-02-23T00:38:28.186 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:27.179055+0000 mon.b (mon.1) 403 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:28.186 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:27.179500+0000 mon.a (mon.0) 2796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:28.186 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:26.801618+0000 mgr.y (mgr.24437) 466 : audit [DBG] from='client.27402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:38:28.186 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:27.176758+0000 mon.a (mon.0) 2794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:38:28.186 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:27.176837+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e490: 8 total, 7 up, 7 in 2024-02-23T00:38:28.186 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:27.179055+0000 mon.b (mon.1) 403 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:28.186 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:27.179500+0000 mon.a (mon.0) 2796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:28.315 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2024-02-23T00:38:26.170067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007409,"num_objects":398,"num_object_clones":9,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":120,"num_objects_misplaced":84,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":2044,"num_read_kb":1637,"num_write":1629,"num_write_kb":4622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":323,"num_bytes_recovered":4004019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2874,"ondisk_log_size":2874,"up":645,"acting":648,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":732,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2197028,"kb_used_data":50924,"kb_used_omap":0,"kb_used_meta":2146048,"kb_avail":654019036,"statfs":{"total":671965249536,"available":669715492864,"internally_reserved":0,"allocated":52146176,"data_stored":33514451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2197553152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15559,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":84,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":428,"num_read_kb":351,"num_write":271,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":1375,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043209"},"pg_stats":[{"pgid":"221.1f","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.202701+0000","last_change":"2024-02-23T00:38:25.202628+0000","last_active":"2024-02-23T00:38:25.202701+0000","last_peered":"2024-02-23T00:38:25.202701+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.202628+0000","last_became_peered":"2024-02-23T00:38:25.202628+0000","last_unstale":"2024-02-23T00:38:25.202701+0000","last_undegraded":"2024-02-23T00:38:25.202701+0000","last_fullsized":"2024-02-23T00:38:25.176718+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"221.1e","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.197635+0000","last_change":"2024-02-23T00:38:25.197564+0000","last_active":"2024-02-23T00:38:25.197635+0000","last_peered":"2024-02-23T00:38:25.197635+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.197564+0000","last_became_peered":"2024-02-23T00:38:25.197564+0000","last_unstale":"2024-02-23T00:38:25.197635+0000","last_undegraded":"2024-02-23T00:38:25.197635+0000","last_fullsized":"2024-02-23T00:38:25.176310+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"221.1d","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.187616+0000","last_change":"2024-02-23T00:38:25.187547+0000","last_active":"2024-02-23T00:38:25.187616+0000","last_peered":"2024-02-23T00:38:25.187616+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.187547+0000","last_became_peered":"2024-02-23T00:38:25.187547+0000","last_unstale":"2024-02-23T00:38:25.187616+0000","last_undegraded":"2024-02-23T00:38:25.187616+0000","last_fullsized":"2024-02-23T00:38:25.173998+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"221.1c","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.199489+0000","last_change":"2024-02-23T00:38:25.199489+0000","last_active":"2024-02-23T00:38:25.199489+0000","last_peered":"2024-02-23T00:38:25.199489+0000","last_clean":"2024-02-23T00:38:25.199489+0000","last_became_active":"2024-02-23T00:38:25.199167+0000","last_became_peered":"2024-02-23T00:38:25.199167+0000","last_unstale":"2024-02-23T00:38:25.199489+0000","last_undegraded":"2024-02-23T00:38:25.199489+0000","last_fullsized":"2024-02-23T00:38:25.199489+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"221.1b","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.192192+0000","last_change":"2024-02-23T00:38:25.192192+0000","last_active":"2024-02-23T00:38:25.192192+0000","last_peered":"2024-02-23T00:38:25.192192+0000","last_clean":"2024-02-23T00:38:25.192192+0000","last_became_active":"2024-02-23T00:38:25.191934+0000","last_became_peered":"2024-02-23T00:38:25.191934+0000","last_unstale":"2024-02-23T00:38:25.192192+0000","last_undegraded":"2024-02-23T00:38:25.192192+0000","last_fullsized":"2024-02-23T00:38:25.192192+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"221.1a","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.203390+0000","last_change":"2024-02-23T00:38:25.203390+0000","last_active":"2024-02-23T00:38:25.203390+0000","last_peered":"2024-02-23T00:38:25.203390+0000","last_clean":"2024-02-23T00:38:25.203390+0000","last_became_active":"2024-02-23T00:38:25.203128+0000","last_became_peered":"2024-02-23T00:38:25.203128+0000","last_unstale":"2024-02-23T00:38:25.203390+0000","last_undegraded":"2024-02-23T00:38:25.203390+0000","last_fullsized":"2024-02-23T00:38:25.203390+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"221.19","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.196175+0000","last_change":"2024-02-23T00:38:25.196175+0000","last_active":"2024-02-23T00:38:25.196175+0000","last_peered":"2024-02-23T00:38:25.196175+0000","last_clean":"2024-02-23T00:38:25.196175+0000","last_became_active":"2024-02-23T00:38:25.195505+0000","last_became_peered":"2024-02-23T00:38:25.195505+0000","last_unstale":"2024-02-23T00:38:25.196175+0000","last_undegraded":"2024-02-23T00:38:25.196175+0000","last_fullsized":"2024-02-23T00:38:25.196175+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"221.18","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.210300+0000","last_change":"2024-02-23T00:38:25.210300+0000","last_active":"2024-02-23T00:38:25.210300+0000","last_peered":"2024-02-23T00:38:25.210300+0000","last_clean":"2024-02-23T00:38:25.210300+0000","last_became_active":"2024-02-23T00:38:25.210040+0000","last_became_peered":"2024-02-23T00:38:25.210040+0000","last_unstale":"2024-02-23T00:38:25.210300+0000","last_undegraded":"2024-02-23T00:38:25.210300+0000","last_fullsized":"2024-02-23T00:38:25.210300+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"221.17","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.199783+0000","last_change":"2024-02-23T00:38:25.199783+0000","last_active":"2024-02-23T00:38:25.199783+0000","last_peered":"2024-02-23T00:38:25.199783+0000","last_clean":"2024-02-23T00:38:25.199783+0000","last_became_active":"2024-02-23T00:38:25.199502+0000","last_became_peered":"2024-02-23T00:38:25.199502+0000","last_unstale":"2024-02-23T00:38:25.199783+0000","last_undegraded":"2024-02-23T00:38:25.199783+0000","last_fullsized":"2024-02-23T00:38:25.199783+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"221.16","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.192726+0000","last_change":"2024-02-23T00:38:25.192639+0000","last_active":"2024-02-23T00:38:25.192726+0000","last_peered":"2024-02-23T00:38:25.192726+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.192639+0000","last_became_peered":"2024-02-23T00:38:25.192639+0000","last_unstale":"2024-02-23T00:38:25.192726+0000","last_undegraded":"2024-02-23T00:38:25.192726+0000","last_fullsized":"2024-02-23T00:38:25.170519+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"221.15","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.220913+0000","last_change":"2024-02-23T00:38:25.220845+0000","last_active":"2024-02-23T00:38:25.220913+0000","last_peered":"2024-02-23T00:38:25.220913+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.220845+0000","last_became_peered":"2024-02-23T00:38:25.220845+0000","last_unstale":"2024-02-23T00:38:25.220913+0000","last_undegraded":"2024-02-23T00:38:25.220913+0000","last_fullsized":"2024-02-23T00:38:25.176717+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"221.14","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.199552+0000","last_change":"2024-02-23T00:38:25.199489+0000","last_active":"2024-02-23T00:38:25.199552+0000","last_peered":"2024-02-23T00:38:25.199552+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.199489+0000","last_became_peered":"2024-02-23T00:38:25.199489+0000","last_unstale":"2024-02-23T00:38:25.199552+0000","last_undegraded":"2024-02-23T00:38:25.199552+0000","last_fullsized":"2024-02-23T00:38:25.170276+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"221.13","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.185995+0000","last_change":"2024-02-23T00:38:25.185925+0000","last_active":"2024-02-23T00:38:25.185995+0000","last_peered":"2024-02-23T00:38:25.185995+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.185925+0000","last_became_peered":"2024-02-23T00:38:25.185925+0000","last_unstale":"2024-02-23T00:38:25.185995+0000","last_undegraded":"2024-02-23T00:38:25.185995+0000","last_fullsized":"2024-02-23T00:38:25.175748+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"221.12","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.207079+0000","last_change":"2024-02-23T00:38:25.207079+0000","last_active":"2024-02-23T00:38:25.207079+0000","last_peered":"2024-02-23T00:38:25.207079+0000","last_clean":"2024-02-23T00:38:25.207079+0000","last_became_active":"2024-02-23T00:38:25.206661+0000","last_became_peered":"2024-02-23T00:38:25.206661+0000","last_unstale":"2024-02-23T00:38:25.207079+0000","last_undegraded":"2024-02-23T00:38:25.207079+0000","last_fullsized":"2024-02-23T00:38:25.207079+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"221.11","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.195905+0000","last_change":"2024-02-23T00:38:25.195841+0000","last_active":"2024-02-23T00:38:25.195905+0000","last_peered":"2024-02-23T00:38:25.195905+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.195841+0000","last_became_peered":"2024-02-23T00:38:25.195841+0000","last_unstale":"2024-02-23T00:38:25.195905+0000","last_undegraded":"2024-02-23T00:38:25.195905+0000","last_fullsized":"2024-02-23T00:38:25.176677+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"221.10","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.199260+0000","last_change":"2024-02-23T00:38:25.199260+0000","last_active":"2024-02-23T00:38:25.199260+0000","last_peered":"2024-02-23T00:38:25.199260+0000","last_clean":"2024-02-23T00:38:25.199260+0000","last_became_active":"2024-02-23T00:38:25.198982+0000","last_became_peered":"2024-02-23T00:38:25.198982+0000","last_unstale":"2024-02-23T00:38:25.199260+0000","last_undegraded":"2024-02-23T00:38:25.199260+0000","last_fullsized":"2024-02-23T00:38:25.199260+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"221.f","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.220382+0000","last_change":"2024-02-23T00:38:25.220298+0000","last_active":"2024-02-23T00:38:25.220382+0000","last_peered":"2024-02-23T00:38:25.220382+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.220298+0000","last_became_peered":"2024-02-23T00:38:25.220298+0000","last_unstale":"2024-02-23T00:38:25.220382+0000","last_undegraded":"2024-02-23T00:38:25.220382+0000","last_fullsized":"2024-02-23T00:38:25.194723+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"221.e","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.196588+0000","last_change":"2024-02-23T00:38:25.196588+0000","last_active":"2024-02-23T00:38:25.196588+0000","last_peered":"2024-02-23T00:38:25.196588+0000","last_clean":"2024-02-23T00:38:25.196588+0000","last_became_active":"2024-02-23T00:38:25.196290+0000","last_became_peered":"2024-02-23T00:38:25.196290+0000","last_unstale":"2024-02-23T00:38:25.196588+0000","last_undegraded":"2024-02-23T00:38:25.196588+0000","last_fullsized":"2024-02-23T00:38:25.196588+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"221.d","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.198576+0000","last_change":"2024-02-23T00:38:25.198508+0000","last_active":"2024-02-23T00:38:25.198576+0000","last_peered":"2024-02-23T00:38:25.198576+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.198508+0000","last_became_peered":"2024-02-23T00:38:25.198508+0000","last_unstale":"2024-02-23T00:38:25.198576+0000","last_undegraded":"2024-02-23T00:38:25.198576+0000","last_fullsized":"2024-02-23T00:38:25.170878+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"221.c","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.204056+0000","last_change":"2024-02-23T00:38:25.204056+0000","last_active":"2024-02-23T00:38:25.204056+0000","last_peered":"2024-02-23T00:38:25.204056+0000","last_clean":"2024-02-23T00:38:25.204056+0000","last_became_active":"2024-02-23T00:38:25.203749+0000","last_became_peered":"2024-02-23T00:38:25.203749+0000","last_unstale":"2024-02-23T00:38:25.204056+0000","last_undegraded":"2024-02-23T00:38:25.204056+0000","last_fullsized":"2024-02-23T00:38:25.204056+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"221.b","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.185151+0000","last_change":"2024-02-23T00:38:25.185079+0000","last_active":"2024-02-23T00:38:25.185151+0000","last_peered":"2024-02-23T00:38:25.185151+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.185079+0000","last_became_peered":"2024-02-23T00:38:25.185079+0000","last_unstale":"2024-02-23T00:38:25.185151+0000","last_undegraded":"2024-02-23T00:38:25.185151+0000","last_fullsized":"2024-02-23T00:38:25.175454+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"221.a","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.186219+0000","last_change":"2024-02-23T00:38:25.186219+0000","last_active":"2024-02-23T00:38:25.186219+0000","last_peered":"2024-02-23T00:38:25.186219+0000","last_clean":"2024-02-23T00:38:25.186219+0000","last_became_active":"2024-02-23T00:38:25.185937+0000","last_became_peered":"2024-02-23T00:38:25.185937+0000","last_unstale":"2024-02-23T00:38:25.186219+0000","last_undegraded":"2024-02-23T00:38:25.186219+0000","last_fullsized":"2024-02-23T00:38:25.186219+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"221.9","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.200228+0000","last_change":"2024-02-23T00:38:25.200228+0000","last_active":"2024-02-23T00:38:25.200228+0000","last_peered":"2024-02-23T00:38:25.200228+0000","last_clean":"2024-02-23T00:38:25.200228+0000","last_became_active":"2024-02-23T00:38:25.198315+0000","last_became_peered":"2024-02-23T00:38:25.198315+0000","last_unstale":"2024-02-23T00:38:25.200228+0000","last_undegraded":"2024-02-23T00:38:25.200228+0000","last_fullsized":"2024-02-23T00:38:25.200228+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"221.8","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.199884+0000","last_change":"2024-02-23T00:38:25.199819+0000","last_active":"2024-02-23T00:38:25.199884+0000","last_peered":"2024-02-23T00:38:25.199884+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.199819+0000","last_became_peered":"2024-02-23T00:38:25.199819+0000","last_unstale":"2024-02-23T00:38:25.199884+0000","last_undegraded":"2024-02-23T00:38:25.199884+0000","last_fullsized":"2024-02-23T00:38:25.172668+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"221.7","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.177299+0000","last_change":"2024-02-23T00:38:25.177231+0000","last_active":"2024-02-23T00:38:25.177299+0000","last_peered":"2024-02-23T00:38:25.177299+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.177231+0000","last_became_peered":"2024-02-23T00:38:25.177231+0000","last_unstale":"2024-02-23T00:38:25.177299+0000","last_undegraded":"2024-02-23T00:38:25.177299+0000","last_fullsized":"2024-02-23T00:38:25.171492+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"221.6","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.208812+0000","last_change":"2024-02-23T00:38:25.208812+0000","last_active":"2024-02-23T00:38:25.208812+0000","last_peered":"2024-02-23T00:38:25.208812+0000","last_clean":"2024-02-23T00:38:25.208812+0000","last_became_active":"2024-02-23T00:38:25.208522+0000","last_became_peered":"2024-02-23T00:38:25.208522+0000","last_unstale":"2024-02-23T00:38:25.208812+0000","last_undegraded":"2024-02-23T00:38:25.208812+0000","last_fullsized":"2024-02-23T00:38:25.208812+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"221.5","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.201084+0000","last_change":"2024-02-23T00:38:25.201015+0000","last_active":"2024-02-23T00:38:25.201084+0000","last_peered":"2024-02-23T00:38:25.201084+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.201015+0000","last_became_peered":"2024-02-23T00:38:25.201015+0000","last_unstale":"2024-02-23T00:38:25.201084+0000","last_undegraded":"2024-02-23T00:38:25.201084+0000","last_fullsized":"2024-02-23T00:38:25.186396+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"221.4","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.191116+0000","last_change":"2024-02-23T00:38:25.191047+0000","last_active":"2024-02-23T00:38:25.191116+0000","last_peered":"2024-02-23T00:38:25.191116+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.191047+0000","last_became_peered":"2024-02-23T00:38:25.191047+0000","last_unstale":"2024-02-23T00:38:25.191116+0000","last_undegraded":"2024-02-23T00:38:25.191116+0000","last_fullsized":"2024-02-23T00:38:25.184436+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"221.3","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.206217+0000","last_change":"2024-02-23T00:38:25.206217+0000","last_active":"2024-02-23T00:38:25.206217+0000","last_peered":"2024-02-23T00:38:25.206217+0000","last_clean":"2024-02-23T00:38:25.206217+0000","last_became_active":"2024-02-23T00:38:25.205851+0000","last_became_peered":"2024-02-23T00:38:25.205851+0000","last_unstale":"2024-02-23T00:38:25.206217+0000","last_undegraded":"2024-02-23T00:38:25.206217+0000","last_fullsized":"2024-02-23T00:38:25.206217+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"221.2","version":"0'0","reported_seq":6,"reported_epoch":488,"state":"active+undersized","last_fresh":"2024-02-23T00:38:25.186057+0000","last_change":"2024-02-23T00:38:25.185987+0000","last_active":"2024-02-23T00:38:25.186057+0000","last_peered":"2024-02-23T00:38:25.186057+0000","last_clean":"2024-02-23T00:38:24.165166+0000","last_became_active":"2024-02-23T00:38:25.185987+0000","last_became_peered":"2024-02-23T00:38:25.185987+0000","last_unstale":"2024-02-23T00:38:25.186057+0000","last_undegraded":"2024-02-23T00:38:25.186057+0000","last_fullsized":"2024-02-23T00:38:25.170255+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"221.1","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.195154+0000","last_change":"2024-02-23T00:38:25.195154+0000","last_active":"2024-02-23T00:38:25.195154+0000","last_peered":"2024-02-23T00:38:25.195154+0000","last_clean":"2024-02-23T00:38:25.195154+0000","last_became_active":"2024-02-23T00:38:25.194885+0000","last_became_peered":"2024-02-23T00:38:25.194885+0000","last_unstale":"2024-02-23T00:38:25.195154+0000","last_undegraded":"2024-02-23T00:38:25.195154+0000","last_fullsized":"2024-02-23T00:38:25.195154+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"221.0","version":"0'0","reported_seq":9,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.210037+0000","last_change":"2024-02-23T00:38:25.210037+0000","last_active":"2024-02-23T00:38:25.210037+0000","last_peered":"2024-02-23T00:38:25.210037+0000","last_clean":"2024-02-23T00:38:25.210037+0000","last_became_active":"2024-02-23T00:38:25.209832+0000","last_became_peered":"2024-02-23T00:38:25.209832+0000","last_unstale":"2024-02-23T00:38:25.210037+0000","last_undegraded":"2024-02-23T00:38:25.210037+0000","last_fullsized":"2024-02-23T00:38:25.210037+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":487,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:24.165166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:24.165166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.0","version":"485'86","reported_seq":251,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.180354+0000","last_change":"2024-02-23T00:36:37.106755+0000","last_active":"2024-02-23T00:38:24.180354+0000","last_peered":"2024-02-23T00:38:24.180354+0000","last_clean":"2024-02-23T00:36:33.365214+0000","last_became_active":"2024-02-23T00:36:37.106755+0000","last_became_peered":"2024-02-23T00:36:37.106755+0000","last_unstale":"2024-02-23T00:38:24.180354+0000","last_undegraded":"2024-02-23T00:36:37.100498+0000","last_fullsized":"2024-02-23T00:36:37.100377+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'68","last_scrub_stamp":"2024-02-23T00:34:57.645204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.645204+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":32,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1","version":"340'86","reported_seq":524,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.170000+0000","last_change":"2024-02-23T00:36:37.109838+0000","last_active":"2024-02-23T00:38:24.170000+0000","last_peered":"2024-02-23T00:38:24.170000+0000","last_clean":"2024-02-23T00:36:33.191340+0000","last_became_active":"2024-02-23T00:36:37.109838+0000","last_became_peered":"2024-02-23T00:36:37.109838+0000","last_unstale":"2024-02-23T00:38:24.170000+0000","last_undegraded":"2024-02-23T00:36:37.093708+0000","last_fullsized":"2024-02-23T00:36:37.093560+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'85","last_scrub_stamp":"2024-02-23T00:34:16.724630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:16.724630+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.2","version":"485'25","reported_seq":475,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.171605+0000","last_change":"2024-02-23T00:36:37.106211+0000","last_active":"2024-02-23T00:38:24.171605+0000","last_peered":"2024-02-23T00:38:24.171605+0000","last_clean":"2024-02-23T00:36:33.365672+0000","last_became_active":"2024-02-23T00:36:37.106211+0000","last_became_peered":"2024-02-23T00:36:37.106211+0000","last_unstale":"2024-02-23T00:38:24.171605+0000","last_undegraded":"2024-02-23T00:36:37.098859+0000","last_fullsized":"2024-02-23T00:36:37.098739+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'20","last_scrub_stamp":"2024-02-23T00:34:23.049395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.049395+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":15,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.3","version":"451'44","reported_seq":508,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171302+0000","last_change":"2024-02-23T00:37:05.079232+0000","last_active":"2024-02-23T00:38:24.171302+0000","last_peered":"2024-02-23T00:38:24.171302+0000","last_clean":"2024-02-23T00:38:24.171302+0000","last_became_active":"2024-02-23T00:36:17.093628+0000","last_became_peered":"2024-02-23T00:36:17.093628+0000","last_unstale":"2024-02-23T00:38:24.171302+0000","last_undegraded":"2024-02-23T00:38:24.171302+0000","last_fullsized":"2024-02-23T00:38:24.171302+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'27","last_scrub_stamp":"2024-02-23T00:37:05.079149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.079149+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":24,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.4","version":"442'49","reported_seq":488,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.177395+0000","last_change":"2024-02-23T00:36:45.668442+0000","last_active":"2024-02-23T00:38:24.177395+0000","last_peered":"2024-02-23T00:38:24.177395+0000","last_clean":"2024-02-23T00:38:24.177395+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:38:24.177395+0000","last_undegraded":"2024-02-23T00:38:24.177395+0000","last_fullsized":"2024-02-23T00:38:24.177395+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'39","last_scrub_stamp":"2024-02-23T00:34:19.398065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.398065+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.5","version":"471'126","reported_seq":561,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.173314+0000","last_change":"2024-02-23T00:36:37.105105+0000","last_active":"2024-02-23T00:38:24.173314+0000","last_peered":"2024-02-23T00:38:24.173314+0000","last_clean":"2024-02-23T00:36:33.192144+0000","last_became_active":"2024-02-23T00:36:37.105105+0000","last_became_peered":"2024-02-23T00:36:37.105105+0000","last_unstale":"2024-02-23T00:38:24.173314+0000","last_undegraded":"2024-02-23T00:36:37.095637+0000","last_fullsized":"2024-02-23T00:36:37.094980+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'124","last_scrub_stamp":"2024-02-23T00:34:19.695449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.695449+0000","log_size":126,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":92,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":39,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.6","version":"484'56","reported_seq":504,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.172264+0000","last_change":"2024-02-23T00:36:37.120734+0000","last_active":"2024-02-23T00:38:24.172264+0000","last_peered":"2024-02-23T00:38:24.172264+0000","last_clean":"2024-02-23T00:36:33.194384+0000","last_became_active":"2024-02-23T00:36:37.120734+0000","last_became_peered":"2024-02-23T00:36:37.120734+0000","last_unstale":"2024-02-23T00:38:24.172264+0000","last_undegraded":"2024-02-23T00:36:37.106912+0000","last_fullsized":"2024-02-23T00:36:37.106799+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:34:31.733148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.733148+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.7","version":"401'97","reported_seq":528,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176931+0000","last_change":"2024-02-23T00:36:45.610239+0000","last_active":"2024-02-23T00:38:24.176931+0000","last_peered":"2024-02-23T00:38:24.176931+0000","last_clean":"2024-02-23T00:38:24.176931+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:38:24.176931+0000","last_undegraded":"2024-02-23T00:38:24.176931+0000","last_fullsized":"2024-02-23T00:38:24.176931+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'92","last_scrub_stamp":"2024-02-23T00:34:18.124049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:18.124049+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":94,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.8","version":"484'73","reported_seq":440,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171290+0000","last_change":"2024-02-23T00:36:45.607171+0000","last_active":"2024-02-23T00:38:24.171290+0000","last_peered":"2024-02-23T00:38:24.171290+0000","last_clean":"2024-02-23T00:38:24.171290+0000","last_became_active":"2024-02-23T00:34:41.115222+0000","last_became_peered":"2024-02-23T00:34:41.115222+0000","last_unstale":"2024-02-23T00:38:24.171290+0000","last_undegraded":"2024-02-23T00:38:24.171290+0000","last_fullsized":"2024-02-23T00:38:24.171290+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'71","last_scrub_stamp":"2024-02-23T00:35:41.651500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.651500+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.9","version":"485'29","reported_seq":473,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171274+0000","last_change":"2024-02-23T00:36:45.606951+0000","last_active":"2024-02-23T00:38:24.171274+0000","last_peered":"2024-02-23T00:38:24.171274+0000","last_clean":"2024-02-23T00:38:24.171274+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:38:24.171274+0000","last_undegraded":"2024-02-23T00:38:24.171274+0000","last_fullsized":"2024-02-23T00:38:24.171274+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'17","last_scrub_stamp":"2024-02-23T00:34:17.107575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.107575+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.a","version":"248'22","reported_seq":473,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.177677+0000","last_change":"2024-02-23T00:38:00.735331+0000","last_active":"2024-02-23T00:38:24.177677+0000","last_peered":"2024-02-23T00:38:24.177677+0000","last_clean":"2024-02-23T00:38:24.177677+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:38:24.177677+0000","last_undegraded":"2024-02-23T00:38:24.177677+0000","last_fullsized":"2024-02-23T00:38:24.177677+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'22","last_scrub_stamp":"2024-02-23T00:38:00.735240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.735240+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.b","version":"485'37","reported_seq":511,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.178547+0000","last_change":"2024-02-23T00:36:37.120097+0000","last_active":"2024-02-23T00:38:24.178547+0000","last_peered":"2024-02-23T00:38:24.178547+0000","last_clean":"2024-02-23T00:36:33.368807+0000","last_became_active":"2024-02-23T00:36:37.120097+0000","last_became_peered":"2024-02-23T00:36:37.120097+0000","last_unstale":"2024-02-23T00:38:24.178547+0000","last_undegraded":"2024-02-23T00:36:37.107185+0000","last_fullsized":"2024-02-23T00:36:37.107054+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-23T00:33:37.391203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.391203+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":14,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.c","version":"422'97","reported_seq":537,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.178201+0000","last_change":"2024-02-23T00:36:37.106629+0000","last_active":"2024-02-23T00:38:24.178201+0000","last_peered":"2024-02-23T00:38:24.178201+0000","last_clean":"2024-02-23T00:36:33.366529+0000","last_became_active":"2024-02-23T00:36:37.106629+0000","last_became_peered":"2024-02-23T00:36:37.106629+0000","last_unstale":"2024-02-23T00:38:24.178201+0000","last_undegraded":"2024-02-23T00:36:37.096711+0000","last_fullsized":"2024-02-23T00:36:37.096581+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'76","last_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_deep_scrub":"397'76","last_deep_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.098352+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.d","version":"395'39","reported_seq":513,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.186864+0000","last_change":"2024-02-23T00:38:13.724199+0000","last_active":"2024-02-23T00:38:24.186864+0000","last_peered":"2024-02-23T00:38:24.186864+0000","last_clean":"2024-02-23T00:38:24.186864+0000","last_became_active":"2024-02-23T00:34:41.109682+0000","last_became_peered":"2024-02-23T00:34:41.109682+0000","last_unstale":"2024-02-23T00:38:24.186864+0000","last_undegraded":"2024-02-23T00:38:24.186864+0000","last_fullsized":"2024-02-23T00:38:24.186864+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'39","last_scrub_stamp":"2024-02-23T00:38:13.724131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.724131+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.e","version":"405'77","reported_seq":572,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.175416+0000","last_change":"2024-02-23T00:36:45.609915+0000","last_active":"2024-02-23T00:38:24.175416+0000","last_peered":"2024-02-23T00:38:24.175416+0000","last_clean":"2024-02-23T00:38:24.175416+0000","last_became_active":"2024-02-23T00:34:41.108381+0000","last_became_peered":"2024-02-23T00:34:41.108381+0000","last_unstale":"2024-02-23T00:38:24.175416+0000","last_undegraded":"2024-02-23T00:38:24.175416+0000","last_fullsized":"2024-02-23T00:38:24.175416+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'74","last_scrub_stamp":"2024-02-23T00:34:15.172848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.172848+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000073,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":3931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":4000057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.f","version":"485'78","reported_seq":440,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.177486+0000","last_change":"2024-02-23T00:36:45.610425+0000","last_active":"2024-02-23T00:38:24.177486+0000","last_peered":"2024-02-23T00:38:24.177486+0000","last_clean":"2024-02-23T00:38:24.177486+0000","last_became_active":"2024-02-23T00:34:41.101718+0000","last_became_peered":"2024-02-23T00:34:41.101718+0000","last_unstale":"2024-02-23T00:38:24.177486+0000","last_undegraded":"2024-02-23T00:38:24.177486+0000","last_fullsized":"2024-02-23T00:38:24.177486+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'69","last_scrub_stamp":"2024-02-23T00:35:46.750022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.750022+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":33,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.10","version":"401'37","reported_seq":478,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.171434+0000","last_change":"2024-02-23T00:36:37.119458+0000","last_active":"2024-02-23T00:38:24.171434+0000","last_peered":"2024-02-23T00:38:24.171434+0000","last_clean":"2024-02-23T00:36:33.364885+0000","last_became_active":"2024-02-23T00:36:37.119458+0000","last_became_peered":"2024-02-23T00:36:37.119458+0000","last_unstale":"2024-02-23T00:38:24.171434+0000","last_undegraded":"2024-02-23T00:36:37.098703+0000","last_fullsized":"2024-02-23T00:36:37.098584+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'34","last_scrub_stamp":"2024-02-23T00:35:01.791840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.791840+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":13,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.11","version":"484'52","reported_seq":497,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.169896+0000","last_change":"2024-02-23T00:37:06.042258+0000","last_active":"2024-02-23T00:38:24.169896+0000","last_peered":"2024-02-23T00:38:24.169896+0000","last_clean":"2024-02-23T00:38:24.169896+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:38:24.169896+0000","last_undegraded":"2024-02-23T00:38:24.169896+0000","last_fullsized":"2024-02-23T00:38:24.169896+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'50","last_scrub_stamp":"2024-02-23T00:37:06.042181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:06.042181+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":23,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.28","version":"0'0","reported_seq":122,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176680+0000","last_change":"2024-02-23T00:38:16.678215+0000","last_active":"2024-02-23T00:38:24.176680+0000","last_peered":"2024-02-23T00:38:24.176680+0000","last_clean":"2024-02-23T00:38:24.176680+0000","last_became_active":"2024-02-23T00:35:29.092396+0000","last_became_peered":"2024-02-23T00:35:29.092396+0000","last_unstale":"2024-02-23T00:38:24.176680+0000","last_undegraded":"2024-02-23T00:38:24.176680+0000","last_fullsized":"2024-02-23T00:38:24.176680+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.678152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.678152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.12","version":"485'74","reported_seq":507,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.175278+0000","last_change":"2024-02-23T00:36:37.114447+0000","last_active":"2024-02-23T00:38:24.175278+0000","last_peered":"2024-02-23T00:38:24.175278+0000","last_clean":"2024-02-23T00:36:33.364627+0000","last_became_active":"2024-02-23T00:36:37.114447+0000","last_became_peered":"2024-02-23T00:36:37.114447+0000","last_unstale":"2024-02-23T00:38:24.175278+0000","last_undegraded":"2024-02-23T00:36:37.099733+0000","last_fullsized":"2024-02-23T00:36:37.099613+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'68","last_scrub_stamp":"2024-02-23T00:36:27.078990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.078990+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":29,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.29","version":"62'8","reported_seq":527,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.177258+0000","last_change":"2024-02-23T00:36:37.122045+0000","last_active":"2024-02-23T00:38:24.177258+0000","last_peered":"2024-02-23T00:38:24.177258+0000","last_clean":"2024-02-23T00:35:27.100547+0000","last_became_active":"2024-02-23T00:36:37.122045+0000","last_became_peered":"2024-02-23T00:36:37.122045+0000","last_unstale":"2024-02-23T00:38:24.177258+0000","last_undegraded":"2024-02-23T00:36:37.107771+0000","last_fullsized":"2024-02-23T00:36:37.107625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:27.673481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.673481+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.13","version":"484'63","reported_seq":505,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178438+0000","last_change":"2024-02-23T00:36:45.667732+0000","last_active":"2024-02-23T00:38:24.178438+0000","last_peered":"2024-02-23T00:38:24.178438+0000","last_clean":"2024-02-23T00:38:24.178438+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:38:24.178438+0000","last_undegraded":"2024-02-23T00:38:24.178438+0000","last_fullsized":"2024-02-23T00:38:24.178438+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'61","last_scrub_stamp":"2024-02-23T00:36:38.944617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:38.944617+0000","log_size":63,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":22,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.14","version":"208'85","reported_seq":521,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.178042+0000","last_change":"2024-02-23T00:36:37.115695+0000","last_active":"2024-02-23T00:38:24.178042+0000","last_peered":"2024-02-23T00:38:24.178042+0000","last_clean":"2024-02-23T00:36:33.366438+0000","last_became_active":"2024-02-23T00:36:37.115695+0000","last_became_peered":"2024-02-23T00:36:37.115695+0000","last_unstale":"2024-02-23T00:38:24.178042+0000","last_undegraded":"2024-02-23T00:36:37.099640+0000","last_fullsized":"2024-02-23T00:36:37.099515+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'85","last_scrub_stamp":"2024-02-23T00:35:46.085280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.085280+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.15","version":"340'70","reported_seq":514,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.173207+0000","last_change":"2024-02-23T00:37:04.031754+0000","last_active":"2024-02-23T00:38:24.173207+0000","last_peered":"2024-02-23T00:38:24.173207+0000","last_clean":"2024-02-23T00:38:24.173207+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:38:24.173207+0000","last_undegraded":"2024-02-23T00:38:24.173207+0000","last_fullsized":"2024-02-23T00:38:24.173207+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'70","last_scrub_stamp":"2024-02-23T00:37:04.031654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:04.031654+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.16","version":"485'55","reported_seq":504,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.172117+0000","last_change":"2024-02-23T00:36:37.111665+0000","last_active":"2024-02-23T00:38:24.172117+0000","last_peered":"2024-02-23T00:38:24.172117+0000","last_clean":"2024-02-23T00:36:33.194334+0000","last_became_active":"2024-02-23T00:36:37.111665+0000","last_became_peered":"2024-02-23T00:36:37.111665+0000","last_unstale":"2024-02-23T00:38:24.172117+0000","last_undegraded":"2024-02-23T00:36:37.100202+0000","last_fullsized":"2024-02-23T00:36:37.100076+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:35:46.192706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.192706+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":21,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.17","version":"433'112","reported_seq":551,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.185184+0000","last_change":"2024-02-23T00:38:04.819561+0000","last_active":"2024-02-23T00:38:24.185184+0000","last_peered":"2024-02-23T00:38:24.185184+0000","last_clean":"2024-02-23T00:38:24.185184+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:38:24.185184+0000","last_undegraded":"2024-02-23T00:38:24.185184+0000","last_fullsized":"2024-02-23T00:38:24.185184+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'112","last_scrub_stamp":"2024-02-23T00:38:04.819475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:04.819475+0000","log_size":112,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.22","version":"0'0","reported_seq":124,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176578+0000","last_change":"2024-02-23T00:37:09.026463+0000","last_active":"2024-02-23T00:38:24.176578+0000","last_peered":"2024-02-23T00:38:24.176578+0000","last_clean":"2024-02-23T00:38:24.176578+0000","last_became_active":"2024-02-23T00:35:29.089082+0000","last_became_peered":"2024-02-23T00:35:29.089082+0000","last_unstale":"2024-02-23T00:38:24.176578+0000","last_undegraded":"2024-02-23T00:38:24.176578+0000","last_fullsized":"2024-02-23T00:38:24.176578+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.026383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.026383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.18","version":"484'59","reported_seq":504,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.180507+0000","last_change":"2024-02-23T00:36:45.668902+0000","last_active":"2024-02-23T00:38:24.180507+0000","last_peered":"2024-02-23T00:38:24.180507+0000","last_clean":"2024-02-23T00:38:24.180507+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:38:24.180507+0000","last_undegraded":"2024-02-23T00:38:24.180507+0000","last_fullsized":"2024-02-23T00:38:24.180507+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'57","last_scrub_stamp":"2024-02-23T00:35:54.857184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:54.857184+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":21,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.23","version":"426'11","reported_seq":168,"reported_epoch":488,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:25.172012+0000","last_change":"2024-02-23T00:36:37.112756+0000","last_active":"2024-02-23T00:38:25.172012+0000","last_peered":"2024-02-23T00:38:25.172012+0000","last_clean":"2024-02-23T00:36:35.444860+0000","last_became_active":"2024-02-23T00:36:37.112756+0000","last_became_peered":"2024-02-23T00:36:37.112756+0000","last_unstale":"2024-02-23T00:38:25.172012+0000","last_undegraded":"2024-02-23T00:36:37.094265+0000","last_fullsized":"2024-02-23T00:36:37.094133+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.19","version":"485'85","reported_seq":538,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.169790+0000","last_change":"2024-02-23T00:36:45.297690+0000","last_active":"2024-02-23T00:38:24.169790+0000","last_peered":"2024-02-23T00:38:24.169790+0000","last_clean":"2024-02-23T00:38:24.169790+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:38:24.169790+0000","last_undegraded":"2024-02-23T00:38:24.169790+0000","last_fullsized":"2024-02-23T00:38:24.169790+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'60","last_scrub_stamp":"2024-02-23T00:36:31.102608+0000","last_deep_scrub":"271'54","last_deep_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_clean_scrub_stamp":"2024-02-23T00:36:31.102608+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":17,"num_read_kb":8,"num_write":44,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.20","version":"379'11","reported_seq":165,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.190218+0000","last_change":"2024-02-23T00:37:10.032410+0000","last_active":"2024-02-23T00:38:25.190218+0000","last_peered":"2024-02-23T00:38:25.190218+0000","last_clean":"2024-02-23T00:38:25.190218+0000","last_became_active":"2024-02-23T00:35:29.093220+0000","last_became_peered":"2024-02-23T00:35:29.093220+0000","last_unstale":"2024-02-23T00:38:25.190218+0000","last_undegraded":"2024-02-23T00:38:25.190218+0000","last_fullsized":"2024-02-23T00:38:25.190218+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"379'11","last_scrub_stamp":"2024-02-23T00:37:10.032341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:10.032341+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1a","version":"202'68","reported_seq":514,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.170493+0000","last_change":"2024-02-23T00:37:15.836653+0000","last_active":"2024-02-23T00:38:24.170493+0000","last_peered":"2024-02-23T00:38:24.170493+0000","last_clean":"2024-02-23T00:38:24.170493+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:38:24.170493+0000","last_undegraded":"2024-02-23T00:38:24.170493+0000","last_fullsized":"2024-02-23T00:38:24.170493+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'68","last_scrub_stamp":"2024-02-23T00:37:15.836572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:15.836572+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.21","version":"0'0","reported_seq":127,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.175187+0000","last_change":"2024-02-23T00:38:06.376092+0000","last_active":"2024-02-23T00:38:24.175187+0000","last_peered":"2024-02-23T00:38:24.175187+0000","last_clean":"2024-02-23T00:38:24.175187+0000","last_became_active":"2024-02-23T00:35:29.093053+0000","last_became_peered":"2024-02-23T00:35:29.093053+0000","last_unstale":"2024-02-23T00:38:24.175187+0000","last_undegraded":"2024-02-23T00:38:24.175187+0000","last_fullsized":"2024-02-23T00:38:24.175187+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:06.376016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:34.152643+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.376016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.1b","version":"398'54","reported_seq":497,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172303+0000","last_change":"2024-02-23T00:38:07.746173+0000","last_active":"2024-02-23T00:38:24.172303+0000","last_peered":"2024-02-23T00:38:24.172303+0000","last_clean":"2024-02-23T00:38:24.172303+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:38:24.172303+0000","last_undegraded":"2024-02-23T00:38:24.172303+0000","last_fullsized":"2024-02-23T00:38:24.172303+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'54","last_scrub_stamp":"2024-02-23T00:38:07.746090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.746090+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.26","version":"0'0","reported_seq":472,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.181481+0000","last_change":"2024-02-23T00:36:37.122228+0000","last_active":"2024-02-23T00:38:24.181481+0000","last_peered":"2024-02-23T00:38:24.181481+0000","last_clean":"2024-02-23T00:36:33.367599+0000","last_became_active":"2024-02-23T00:36:37.122228+0000","last_became_peered":"2024-02-23T00:36:37.122228+0000","last_unstale":"2024-02-23T00:38:24.181481+0000","last_undegraded":"2024-02-23T00:38:24.181481+0000","last_fullsized":"2024-02-23T00:36:37.110661+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1c","version":"216'34","reported_seq":504,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.174357+0000","last_change":"2024-02-23T00:36:45.608757+0000","last_active":"2024-02-23T00:38:24.174357+0000","last_peered":"2024-02-23T00:38:24.174357+0000","last_clean":"2024-02-23T00:38:24.174357+0000","last_became_active":"2024-02-23T00:34:41.117846+0000","last_became_peered":"2024-02-23T00:34:41.117846+0000","last_unstale":"2024-02-23T00:38:24.174357+0000","last_undegraded":"2024-02-23T00:38:24.174357+0000","last_fullsized":"2024-02-23T00:38:24.174357+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'34","last_scrub_stamp":"2024-02-23T00:34:57.859047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.859047+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.27","version":"0'0","reported_seq":456,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176502+0000","last_change":"2024-02-23T00:35:29.092282+0000","last_active":"2024-02-23T00:38:24.176502+0000","last_peered":"2024-02-23T00:38:24.176502+0000","last_clean":"2024-02-23T00:38:24.176502+0000","last_became_active":"2024-02-23T00:35:29.091970+0000","last_became_peered":"2024-02-23T00:35:29.091970+0000","last_unstale":"2024-02-23T00:38:24.176502+0000","last_undegraded":"2024-02-23T00:38:24.176502+0000","last_fullsized":"2024-02-23T00:38:24.176502+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1d","version":"484'15","reported_seq":208,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178892+0000","last_change":"2024-02-23T00:38:16.420850+0000","last_active":"2024-02-23T00:38:24.178892+0000","last_peered":"2024-02-23T00:38:24.178892+0000","last_clean":"2024-02-23T00:38:24.178892+0000","last_became_active":"2024-02-23T00:34:42.058724+0000","last_became_peered":"2024-02-23T00:34:42.058724+0000","last_unstale":"2024-02-23T00:38:24.178892+0000","last_undegraded":"2024-02-23T00:38:24.178892+0000","last_fullsized":"2024-02-23T00:38:24.178892+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'13","last_scrub_stamp":"2024-02-23T00:38:16.420789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.420789+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.24","version":"0'0","reported_seq":468,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.172580+0000","last_change":"2024-02-23T00:36:37.117551+0000","last_active":"2024-02-23T00:38:24.172580+0000","last_peered":"2024-02-23T00:38:24.172580+0000","last_clean":"2024-02-23T00:36:33.366613+0000","last_became_active":"2024-02-23T00:36:37.117551+0000","last_became_peered":"2024-02-23T00:36:37.117551+0000","last_unstale":"2024-02-23T00:38:24.172580+0000","last_undegraded":"2024-02-23T00:38:24.172580+0000","last_fullsized":"2024-02-23T00:36:37.104246+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1e","version":"340'52","reported_seq":491,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.185977+0000","last_change":"2024-02-23T00:38:09.722750+0000","last_active":"2024-02-23T00:38:24.185977+0000","last_peered":"2024-02-23T00:38:24.185977+0000","last_clean":"2024-02-23T00:38:24.185977+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:38:24.185977+0000","last_undegraded":"2024-02-23T00:38:24.185977+0000","last_fullsized":"2024-02-23T00:38:24.185977+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'52","last_scrub_stamp":"2024-02-23T00:38:09.722680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.722680+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.25","version":"0'0","reported_seq":465,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172663+0000","last_change":"2024-02-23T00:35:29.091129+0000","last_active":"2024-02-23T00:38:24.172663+0000","last_peered":"2024-02-23T00:38:24.172663+0000","last_clean":"2024-02-23T00:38:24.172663+0000","last_became_active":"2024-02-23T00:35:29.090793+0000","last_became_peered":"2024-02-23T00:35:29.090793+0000","last_unstale":"2024-02-23T00:38:24.172663+0000","last_undegraded":"2024-02-23T00:38:24.172663+0000","last_fullsized":"2024-02-23T00:38:24.172663+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1f","version":"439'74","reported_seq":512,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.185041+0000","last_change":"2024-02-23T00:36:37.117847+0000","last_active":"2024-02-23T00:38:24.185041+0000","last_peered":"2024-02-23T00:38:24.185041+0000","last_clean":"2024-02-23T00:36:33.365701+0000","last_became_active":"2024-02-23T00:36:37.117847+0000","last_became_peered":"2024-02-23T00:36:37.117847+0000","last_unstale":"2024-02-23T00:38:24.185041+0000","last_undegraded":"2024-02-23T00:36:37.102042+0000","last_fullsized":"2024-02-23T00:36:37.101901+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-23T00:33:39.351573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.351573+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"379'9","reported_seq":614,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.177622+0000","last_change":"2024-02-23T00:37:05.955772+0000","last_active":"2024-02-23T00:38:25.177622+0000","last_peered":"2024-02-23T00:38:25.177622+0000","last_clean":"2024-02-23T00:38:25.177622+0000","last_became_active":"2024-02-23T00:35:27.095085+0000","last_became_peered":"2024-02-23T00:35:27.095085+0000","last_unstale":"2024-02-23T00:38:25.177622+0000","last_undegraded":"2024-02-23T00:38:25.177622+0000","last_fullsized":"2024-02-23T00:38:25.177622+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'9","last_scrub_stamp":"2024-02-23T00:37:05.955707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.955707+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":572,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172727+0000","last_change":"2024-02-23T00:38:19.164982+0000","last_active":"2024-02-23T00:38:24.172727+0000","last_peered":"2024-02-23T00:38:24.172727+0000","last_clean":"2024-02-23T00:38:24.172727+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:38:24.172727+0000","last_undegraded":"2024-02-23T00:38:24.172727+0000","last_fullsized":"2024-02-23T00:38:24.172727+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":486,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172398+0000","last_change":"2024-02-23T00:38:12.113162+0000","last_active":"2024-02-23T00:38:24.172398+0000","last_peered":"2024-02-23T00:38:24.172398+0000","last_clean":"2024-02-23T00:38:24.172398+0000","last_became_active":"2024-02-23T00:34:41.106070+0000","last_became_peered":"2024-02-23T00:34:41.106070+0000","last_unstale":"2024-02-23T00:38:24.172398+0000","last_undegraded":"2024-02-23T00:38:24.172398+0000","last_fullsized":"2024-02-23T00:38:24.172398+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:12.113097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:38:12.113097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":450,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.176857+0000","last_change":"2024-02-23T00:36:37.104200+0000","last_active":"2024-02-23T00:38:24.176857+0000","last_peered":"2024-02-23T00:38:24.176857+0000","last_clean":"2024-02-23T00:36:33.366159+0000","last_became_active":"2024-02-23T00:36:37.104200+0000","last_became_peered":"2024-02-23T00:36:37.104200+0000","last_unstale":"2024-02-23T00:38:24.176857+0000","last_undegraded":"2024-02-23T00:38:24.176857+0000","last_fullsized":"2024-02-23T00:36:37.102434+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:21.441958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.441958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":414,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.172354+0000","last_change":"2024-02-23T00:36:37.115879+0000","last_active":"2024-02-23T00:38:24.172354+0000","last_peered":"2024-02-23T00:38:24.172354+0000","last_clean":"2024-02-23T00:36:33.366205+0000","last_became_active":"2024-02-23T00:36:37.115879+0000","last_became_peered":"2024-02-23T00:36:37.115879+0000","last_unstale":"2024-02-23T00:38:24.172354+0000","last_undegraded":"2024-02-23T00:38:24.172354+0000","last_fullsized":"2024-02-23T00:36:37.101967+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":799,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.185385+0000","last_change":"2024-02-23T00:36:27.182035+0000","last_active":"2024-02-23T00:38:25.185385+0000","last_peered":"2024-02-23T00:38:25.185385+0000","last_clean":"2024-02-23T00:38:25.185385+0000","last_became_active":"2024-02-23T00:34:41.114058+0000","last_became_peered":"2024-02-23T00:34:41.114058+0000","last_unstale":"2024-02-23T00:38:25.185385+0000","last_undegraded":"2024-02-23T00:38:25.185385+0000","last_fullsized":"2024-02-23T00:38:25.185385+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:36:27.181972+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.181972+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":347,"num_read_kb":347,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":133,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.172883+0000","last_change":"2024-02-23T00:38:19.177071+0000","last_active":"2024-02-23T00:38:24.172883+0000","last_peered":"2024-02-23T00:38:24.172883+0000","last_clean":"2024-02-23T00:27:29.183258+0000","last_became_active":"2024-02-23T00:38:19.177071+0000","last_became_peered":"2024-02-23T00:38:19.177071+0000","last_unstale":"2024-02-23T00:38:24.172883+0000","last_undegraded":"2024-02-23T00:38:19.154601+0000","last_fullsized":"2024-02-23T00:38:19.154480+0000","mapping_epoch":425,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":466,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.173990+0000","last_change":"2024-02-23T00:36:41.532683+0000","last_active":"2024-02-23T00:38:24.173990+0000","last_peered":"2024-02-23T00:38:24.173990+0000","last_clean":"2024-02-23T00:38:24.173990+0000","last_became_active":"2024-02-23T00:34:41.100003+0000","last_became_peered":"2024-02-23T00:34:41.100003+0000","last_unstale":"2024-02-23T00:38:24.173990+0000","last_undegraded":"2024-02-23T00:38:24.173990+0000","last_fullsized":"2024-02-23T00:38:24.173990+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.532619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.532619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":473,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.170729+0000","last_change":"2024-02-23T00:36:37.107478+0000","last_active":"2024-02-23T00:38:24.170729+0000","last_peered":"2024-02-23T00:38:24.170729+0000","last_clean":"2024-02-23T00:36:33.193986+0000","last_became_active":"2024-02-23T00:36:37.107478+0000","last_became_peered":"2024-02-23T00:36:37.107478+0000","last_unstale":"2024-02-23T00:38:24.170729+0000","last_undegraded":"2024-02-23T00:38:24.170729+0000","last_fullsized":"2024-02-23T00:36:37.103166+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.679076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":70,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.177437+0000","last_change":"2024-02-23T00:36:37.122229+0000","last_active":"2024-02-23T00:38:24.177437+0000","last_peered":"2024-02-23T00:38:24.177437+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.122229+0000","last_became_peered":"2024-02-23T00:36:37.122229+0000","last_unstale":"2024-02-23T00:38:24.177437+0000","last_undegraded":"2024-02-23T00:38:24.177437+0000","last_fullsized":"2024-02-23T00:36:37.107921+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":409,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.175719+0000","last_change":"2024-02-23T00:38:01.361178+0000","last_active":"2024-02-23T00:38:24.175719+0000","last_peered":"2024-02-23T00:38:24.175719+0000","last_clean":"2024-02-23T00:38:24.175719+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:38:24.175719+0000","last_undegraded":"2024-02-23T00:38:24.175719+0000","last_fullsized":"2024-02-23T00:38:24.175719+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:01.361095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:01.361095+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":588,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.175913+0000","last_change":"2024-02-23T00:38:19.166424+0000","last_active":"2024-02-23T00:38:24.175913+0000","last_peered":"2024-02-23T00:38:24.175913+0000","last_clean":"2024-02-23T00:38:24.175913+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:38:24.175913+0000","last_undegraded":"2024-02-23T00:38:24.175913+0000","last_fullsized":"2024-02-23T00:38:24.175913+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":473,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.174256+0000","last_change":"2024-02-23T00:36:37.116992+0000","last_active":"2024-02-23T00:38:24.174256+0000","last_peered":"2024-02-23T00:38:24.174256+0000","last_clean":"2024-02-23T00:36:33.194161+0000","last_became_active":"2024-02-23T00:36:37.116992+0000","last_became_peered":"2024-02-23T00:36:37.116992+0000","last_unstale":"2024-02-23T00:38:24.174256+0000","last_undegraded":"2024-02-23T00:38:24.174256+0000","last_fullsized":"2024-02-23T00:36:37.104222+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:50.194656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:50.194656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":464,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.172917+0000","last_change":"2024-02-23T00:36:37.113419+0000","last_active":"2024-02-23T00:38:24.172917+0000","last_peered":"2024-02-23T00:38:24.172917+0000","last_clean":"2024-02-23T00:36:33.194007+0000","last_became_active":"2024-02-23T00:36:37.113419+0000","last_became_peered":"2024-02-23T00:36:37.113419+0000","last_unstale":"2024-02-23T00:38:24.172917+0000","last_undegraded":"2024-02-23T00:38:24.172917+0000","last_fullsized":"2024-02-23T00:36:37.104161+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.745865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.745865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":467,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.180679+0000","last_change":"2024-02-23T00:38:07.226458+0000","last_active":"2024-02-23T00:38:24.180679+0000","last_peered":"2024-02-23T00:38:24.180679+0000","last_clean":"2024-02-23T00:38:24.180679+0000","last_became_active":"2024-02-23T00:34:41.105277+0000","last_became_peered":"2024-02-23T00:34:41.105277+0000","last_unstale":"2024-02-23T00:38:24.180679+0000","last_undegraded":"2024-02-23T00:38:24.180679+0000","last_fullsized":"2024-02-23T00:38:24.180679+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.226380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.226380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":71,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.170067+0000","last_change":"2024-02-23T00:36:37.119721+0000","last_active":"2024-02-23T00:38:24.170067+0000","last_peered":"2024-02-23T00:38:24.170067+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119721+0000","last_became_peered":"2024-02-23T00:36:37.119721+0000","last_unstale":"2024-02-23T00:38:24.170067+0000","last_undegraded":"2024-02-23T00:38:24.170067+0000","last_fullsized":"2024-02-23T00:36:37.102163+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.205312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.205312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":471,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176193+0000","last_change":"2024-02-23T00:37:12.370077+0000","last_active":"2024-02-23T00:38:24.176193+0000","last_peered":"2024-02-23T00:38:24.176193+0000","last_clean":"2024-02-23T00:38:24.176193+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:38:24.176193+0000","last_undegraded":"2024-02-23T00:38:24.176193+0000","last_fullsized":"2024-02-23T00:38:24.176193+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:12.369966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:12.369966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":545,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.173640+0000","last_change":"2024-02-23T00:38:19.177268+0000","last_active":"2024-02-23T00:38:24.173640+0000","last_peered":"2024-02-23T00:38:24.173640+0000","last_clean":"2024-02-23T00:38:24.173640+0000","last_became_active":"2024-02-23T00:38:19.176922+0000","last_became_peered":"2024-02-23T00:38:19.176922+0000","last_unstale":"2024-02-23T00:38:24.173640+0000","last_undegraded":"2024-02-23T00:38:24.173640+0000","last_fullsized":"2024-02-23T00:38:24.173640+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":29,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":464,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178778+0000","last_change":"2024-02-23T00:38:18.055894+0000","last_active":"2024-02-23T00:38:24.178778+0000","last_peered":"2024-02-23T00:38:24.178778+0000","last_clean":"2024-02-23T00:38:24.178778+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:38:24.178778+0000","last_undegraded":"2024-02-23T00:38:24.178778+0000","last_fullsized":"2024-02-23T00:38:24.178778+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:18.055803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:18.055803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":454,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.175858+0000","last_change":"2024-02-23T00:36:37.109480+0000","last_active":"2024-02-23T00:38:24.175858+0000","last_peered":"2024-02-23T00:38:24.175858+0000","last_clean":"2024-02-23T00:36:34.605757+0000","last_became_active":"2024-02-23T00:36:37.109480+0000","last_became_peered":"2024-02-23T00:36:37.109480+0000","last_unstale":"2024-02-23T00:38:24.175858+0000","last_undegraded":"2024-02-23T00:38:24.175858+0000","last_fullsized":"2024-02-23T00:36:37.095167+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.605664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.605664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":477,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171264+0000","last_change":"2024-02-23T00:37:17.822884+0000","last_active":"2024-02-23T00:38:24.171264+0000","last_peered":"2024-02-23T00:38:24.171264+0000","last_clean":"2024-02-23T00:38:24.171264+0000","last_became_active":"2024-02-23T00:34:41.090777+0000","last_became_peered":"2024-02-23T00:34:41.090777+0000","last_unstale":"2024-02-23T00:38:24.171264+0000","last_undegraded":"2024-02-23T00:38:24.171264+0000","last_fullsized":"2024-02-23T00:38:24.171264+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:17.822776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.822776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":422,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.185371+0000","last_change":"2024-02-23T00:36:37.124688+0000","last_active":"2024-02-23T00:38:24.185371+0000","last_peered":"2024-02-23T00:38:24.185371+0000","last_clean":"2024-02-23T00:36:33.367188+0000","last_became_active":"2024-02-23T00:36:37.124688+0000","last_became_peered":"2024-02-23T00:36:37.124688+0000","last_unstale":"2024-02-23T00:38:24.185371+0000","last_undegraded":"2024-02-23T00:38:24.185371+0000","last_fullsized":"2024-02-23T00:36:37.113140+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.704478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.704478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":180,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.171398+0000","last_change":"2024-02-23T00:36:37.113121+0000","last_active":"2024-02-23T00:38:24.171398+0000","last_peered":"2024-02-23T00:38:24.171398+0000","last_clean":"2024-02-23T00:31:25.058059+0000","last_became_active":"2024-02-23T00:36:37.113121+0000","last_became_peered":"2024-02-23T00:36:37.113121+0000","last_unstale":"2024-02-23T00:38:24.171398+0000","last_undegraded":"2024-02-23T00:38:24.171398+0000","last_fullsized":"2024-02-23T00:36:37.095087+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":556,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171058+0000","last_change":"2024-02-23T00:38:19.182992+0000","last_active":"2024-02-23T00:38:24.171058+0000","last_peered":"2024-02-23T00:38:24.171058+0000","last_clean":"2024-02-23T00:38:24.171058+0000","last_became_active":"2024-02-23T00:38:19.182663+0000","last_became_peered":"2024-02-23T00:38:19.182663+0000","last_unstale":"2024-02-23T00:38:24.171058+0000","last_undegraded":"2024-02-23T00:38:24.171058+0000","last_fullsized":"2024-02-23T00:38:24.171058+0000","mapping_epoch":329,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:37:09.133471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.133471+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":479,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172006+0000","last_change":"2024-02-23T00:35:56.678738+0000","last_active":"2024-02-23T00:38:24.172006+0000","last_peered":"2024-02-23T00:38:24.172006+0000","last_clean":"2024-02-23T00:38:24.172006+0000","last_became_active":"2024-02-23T00:34:41.115434+0000","last_became_peered":"2024-02-23T00:34:41.115434+0000","last_unstale":"2024-02-23T00:38:24.172006+0000","last_undegraded":"2024-02-23T00:38:24.172006+0000","last_fullsized":"2024-02-23T00:38:24.172006+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.678676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.678676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":605,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.174549+0000","last_change":"2024-02-23T00:37:07.822516+0000","last_active":"2024-02-23T00:38:25.174549+0000","last_peered":"2024-02-23T00:38:25.174549+0000","last_clean":"2024-02-23T00:38:25.174549+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:38:25.174549+0000","last_undegraded":"2024-02-23T00:38:25.174549+0000","last_fullsized":"2024-02-23T00:38:25.174549+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:37:07.822449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:07.822449+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":455,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171354+0000","last_change":"2024-02-23T00:35:56.258132+0000","last_active":"2024-02-23T00:38:24.171354+0000","last_peered":"2024-02-23T00:38:24.171354+0000","last_clean":"2024-02-23T00:38:24.171354+0000","last_became_active":"2024-02-23T00:27:23.025190+0000","last_became_peered":"2024-02-23T00:27:23.025190+0000","last_unstale":"2024-02-23T00:38:24.171354+0000","last_undegraded":"2024-02-23T00:38:24.171354+0000","last_fullsized":"2024-02-23T00:38:24.171354+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.258056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.258056+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":408,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171429+0000","last_change":"2024-02-23T00:35:46.175575+0000","last_active":"2024-02-23T00:38:24.171429+0000","last_peered":"2024-02-23T00:38:24.171429+0000","last_clean":"2024-02-23T00:38:24.171429+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:38:24.171429+0000","last_undegraded":"2024-02-23T00:38:24.171429+0000","last_fullsized":"2024-02-23T00:38:24.171429+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:46.175476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.175476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":527,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171190+0000","last_change":"2024-02-23T00:37:22.396292+0000","last_active":"2024-02-23T00:38:24.171190+0000","last_peered":"2024-02-23T00:38:24.171190+0000","last_clean":"2024-02-23T00:38:24.171190+0000","last_became_active":"2024-02-23T00:34:41.103369+0000","last_became_peered":"2024-02-23T00:34:41.103369+0000","last_unstale":"2024-02-23T00:38:24.171190+0000","last_undegraded":"2024-02-23T00:38:24.171190+0000","last_fullsized":"2024-02-23T00:38:24.171190+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:22.396222+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.396222+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":606,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.177531+0000","last_change":"2024-02-23T00:38:19.167234+0000","last_active":"2024-02-23T00:38:24.177531+0000","last_peered":"2024-02-23T00:38:24.177531+0000","last_clean":"2024-02-23T00:38:24.177531+0000","last_became_active":"2024-02-23T00:38:19.166944+0000","last_became_peered":"2024-02-23T00:38:19.166944+0000","last_unstale":"2024-02-23T00:38:24.177531+0000","last_undegraded":"2024-02-23T00:38:24.177531+0000","last_fullsized":"2024-02-23T00:38:24.177531+0000","mapping_epoch":392,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":463,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171436+0000","last_change":"2024-02-23T00:35:06.658103+0000","last_active":"2024-02-23T00:38:24.171436+0000","last_peered":"2024-02-23T00:38:24.171436+0000","last_clean":"2024-02-23T00:38:24.171436+0000","last_became_active":"2024-02-23T00:34:41.107670+0000","last_became_peered":"2024-02-23T00:34:41.107670+0000","last_unstale":"2024-02-23T00:38:24.171436+0000","last_undegraded":"2024-02-23T00:38:24.171436+0000","last_fullsized":"2024-02-23T00:38:24.171436+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.657998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"426'9","reported_seq":606,"reported_epoch":488,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:25.174006+0000","last_change":"2024-02-23T00:36:37.111178+0000","last_active":"2024-02-23T00:38:25.174006+0000","last_peered":"2024-02-23T00:38:25.174006+0000","last_clean":"2024-02-23T00:36:35.444981+0000","last_became_active":"2024-02-23T00:36:37.111178+0000","last_became_peered":"2024-02-23T00:36:37.111178+0000","last_unstale":"2024-02-23T00:38:25.174006+0000","last_undegraded":"2024-02-23T00:36:37.095825+0000","last_fullsized":"2024-02-23T00:36:37.095713+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:59.630171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:59.630171+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":451,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.187285+0000","last_change":"2024-02-23T00:36:37.113156+0000","last_active":"2024-02-23T00:38:24.187285+0000","last_peered":"2024-02-23T00:38:24.187285+0000","last_clean":"2024-02-23T00:36:33.366740+0000","last_became_active":"2024-02-23T00:36:37.113156+0000","last_became_peered":"2024-02-23T00:36:37.113156+0000","last_unstale":"2024-02-23T00:38:24.187285+0000","last_undegraded":"2024-02-23T00:36:37.102218+0000","last_fullsized":"2024-02-23T00:36:37.102103+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:34:17.437083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.437083+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":404,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.173452+0000","last_change":"2024-02-23T00:38:16.396902+0000","last_active":"2024-02-23T00:38:24.173452+0000","last_peered":"2024-02-23T00:38:24.173452+0000","last_clean":"2024-02-23T00:38:24.173452+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:38:24.173452+0000","last_undegraded":"2024-02-23T00:38:24.173452+0000","last_fullsized":"2024-02-23T00:38:24.173452+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.396824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":475,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.169711+0000","last_change":"2024-02-23T00:38:08.068624+0000","last_active":"2024-02-23T00:38:24.169711+0000","last_peered":"2024-02-23T00:38:24.169711+0000","last_clean":"2024-02-23T00:38:24.169711+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:38:24.169711+0000","last_undegraded":"2024-02-23T00:38:24.169711+0000","last_fullsized":"2024-02-23T00:38:24.169711+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:08.068559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:08.068559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":559,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171683+0000","last_change":"2024-02-23T00:38:19.182554+0000","last_active":"2024-02-23T00:38:24.171683+0000","last_peered":"2024-02-23T00:38:24.171683+0000","last_clean":"2024-02-23T00:38:24.171683+0000","last_became_active":"2024-02-23T00:38:19.182222+0000","last_became_peered":"2024-02-23T00:38:19.182222+0000","last_unstale":"2024-02-23T00:38:24.171683+0000","last_undegraded":"2024-02-23T00:38:24.171683+0000","last_fullsized":"2024-02-23T00:38:24.171683+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":472,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.181050+0000","last_change":"2024-02-23T00:36:37.126232+0000","last_active":"2024-02-23T00:38:24.181050+0000","last_peered":"2024-02-23T00:38:24.181050+0000","last_clean":"2024-02-23T00:36:33.365949+0000","last_became_active":"2024-02-23T00:36:37.126232+0000","last_became_peered":"2024-02-23T00:36:37.126232+0000","last_unstale":"2024-02-23T00:38:24.181050+0000","last_undegraded":"2024-02-23T00:36:37.112806+0000","last_fullsized":"2024-02-23T00:36:37.112690+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":446,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.185588+0000","last_change":"2024-02-23T00:34:22.475022+0000","last_active":"2024-02-23T00:38:24.185588+0000","last_peered":"2024-02-23T00:38:24.185588+0000","last_clean":"2024-02-23T00:38:24.185588+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:38:24.185588+0000","last_undegraded":"2024-02-23T00:38:24.185588+0000","last_fullsized":"2024-02-23T00:38:24.185588+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.474956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.474956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":179,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178475+0000","last_change":"2024-02-23T00:37:13.405875+0000","last_active":"2024-02-23T00:38:24.178475+0000","last_peered":"2024-02-23T00:38:24.178475+0000","last_clean":"2024-02-23T00:38:24.178475+0000","last_became_active":"2024-02-23T00:34:41.116969+0000","last_became_peered":"2024-02-23T00:34:41.116969+0000","last_unstale":"2024-02-23T00:38:24.178475+0000","last_undegraded":"2024-02-23T00:38:24.178475+0000","last_fullsized":"2024-02-23T00:38:24.178475+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:13.405802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:13.405802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":413,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176051+0000","last_change":"2024-02-23T00:37:21.365047+0000","last_active":"2024-02-23T00:38:24.176051+0000","last_peered":"2024-02-23T00:38:24.176051+0000","last_clean":"2024-02-23T00:38:24.176051+0000","last_became_active":"2024-02-23T00:34:41.109539+0000","last_became_peered":"2024-02-23T00:34:41.109539+0000","last_unstale":"2024-02-23T00:38:24.176051+0000","last_undegraded":"2024-02-23T00:38:24.176051+0000","last_fullsized":"2024-02-23T00:38:24.176051+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_clean_scrub_stamp":"2024-02-23T00:37:21.364934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":71,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.175218+0000","last_change":"2024-02-23T00:36:37.109542+0000","last_active":"2024-02-23T00:38:24.175218+0000","last_peered":"2024-02-23T00:38:24.175218+0000","last_clean":"2024-02-23T00:31:27.149861+0000","last_became_active":"2024-02-23T00:36:37.109542+0000","last_became_peered":"2024-02-23T00:36:37.109542+0000","last_unstale":"2024-02-23T00:38:24.175218+0000","last_undegraded":"2024-02-23T00:38:24.175218+0000","last_fullsized":"2024-02-23T00:36:37.096846+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:42.114807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:35:42.114807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":216,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.171852+0000","last_change":"2024-02-23T00:36:37.114738+0000","last_active":"2024-02-23T00:38:24.171852+0000","last_peered":"2024-02-23T00:38:24.171852+0000","last_clean":"2024-02-23T00:36:33.365051+0000","last_became_active":"2024-02-23T00:36:37.114738+0000","last_became_peered":"2024-02-23T00:36:37.114738+0000","last_unstale":"2024-02-23T00:38:24.171852+0000","last_undegraded":"2024-02-23T00:38:24.171852+0000","last_fullsized":"2024-02-23T00:36:37.094941+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":516,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.170235+0000","last_change":"2024-02-23T00:37:08.107220+0000","last_active":"2024-02-23T00:38:24.170235+0000","last_peered":"2024-02-23T00:38:24.170235+0000","last_clean":"2024-02-23T00:38:24.170235+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:38:24.170235+0000","last_undegraded":"2024-02-23T00:38:24.170235+0000","last_fullsized":"2024-02-23T00:38:24.170235+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:37:08.107110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.107110+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":461,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.182822+0000","last_change":"2024-02-23T00:37:28.355537+0000","last_active":"2024-02-23T00:38:24.182822+0000","last_peered":"2024-02-23T00:38:24.182822+0000","last_clean":"2024-02-23T00:38:24.182822+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:38:24.182822+0000","last_undegraded":"2024-02-23T00:38:24.182822+0000","last_fullsized":"2024-02-23T00:38:24.182822+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:28.355449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:28.355449+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":458,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.179204+0000","last_change":"2024-02-23T00:36:37.952567+0000","last_active":"2024-02-23T00:38:24.179204+0000","last_peered":"2024-02-23T00:38:24.179204+0000","last_clean":"2024-02-23T00:38:24.179204+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:38:24.179204+0000","last_undegraded":"2024-02-23T00:38:24.179204+0000","last_fullsized":"2024-02-23T00:38:24.179204+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:37.952474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:37.952474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":414,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.173224+0000","last_change":"2024-02-23T00:36:37.116207+0000","last_active":"2024-02-23T00:38:24.173224+0000","last_peered":"2024-02-23T00:38:24.173224+0000","last_clean":"2024-02-23T00:36:33.365474+0000","last_became_active":"2024-02-23T00:36:37.116207+0000","last_became_peered":"2024-02-23T00:36:37.116207+0000","last_unstale":"2024-02-23T00:38:24.173224+0000","last_undegraded":"2024-02-23T00:38:24.173224+0000","last_fullsized":"2024-02-23T00:36:37.098927+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.019530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.019530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":282,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.184892+0000","last_change":"2024-02-23T00:36:37.119099+0000","last_active":"2024-02-23T00:38:24.184892+0000","last_peered":"2024-02-23T00:38:24.184892+0000","last_clean":"2024-02-23T00:36:33.366787+0000","last_became_active":"2024-02-23T00:36:37.119099+0000","last_became_peered":"2024-02-23T00:36:37.119099+0000","last_unstale":"2024-02-23T00:38:24.184892+0000","last_undegraded":"2024-02-23T00:38:24.184892+0000","last_fullsized":"2024-02-23T00:36:37.111585+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.924808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.924808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":481,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.170659+0000","last_change":"2024-02-23T00:36:37.097579+0000","last_active":"2024-02-23T00:38:24.170659+0000","last_peered":"2024-02-23T00:38:24.170659+0000","last_clean":"2024-02-23T00:36:33.191735+0000","last_became_active":"2024-02-23T00:36:37.097579+0000","last_became_peered":"2024-02-23T00:36:37.097579+0000","last_unstale":"2024-02-23T00:38:24.170659+0000","last_undegraded":"2024-02-23T00:38:24.170659+0000","last_fullsized":"2024-02-23T00:36:37.094016+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.588132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.588132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":297,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.181204+0000","last_change":"2024-02-23T00:36:37.119086+0000","last_active":"2024-02-23T00:38:24.181204+0000","last_peered":"2024-02-23T00:38:24.181204+0000","last_clean":"2024-02-23T00:36:33.366171+0000","last_became_active":"2024-02-23T00:36:37.119086+0000","last_became_peered":"2024-02-23T00:36:37.119086+0000","last_unstale":"2024-02-23T00:38:24.181204+0000","last_undegraded":"2024-02-23T00:36:37.101188+0000","last_fullsized":"2024-02-23T00:36:37.101097+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:35:00.621157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.621157+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":74,"num_read_kb":60,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":457,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.175545+0000","last_change":"2024-02-23T00:38:14.343844+0000","last_active":"2024-02-23T00:38:24.175545+0000","last_peered":"2024-02-23T00:38:24.175545+0000","last_clean":"2024-02-23T00:38:24.175545+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:38:24.175545+0000","last_undegraded":"2024-02-23T00:38:24.175545+0000","last_fullsized":"2024-02-23T00:38:24.175545+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:14.343770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:14.343770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":469,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.170926+0000","last_change":"2024-02-23T00:38:00.082077+0000","last_active":"2024-02-23T00:38:24.170926+0000","last_peered":"2024-02-23T00:38:24.170926+0000","last_clean":"2024-02-23T00:38:24.170926+0000","last_became_active":"2024-02-23T00:34:41.094428+0000","last_became_peered":"2024-02-23T00:34:41.094428+0000","last_unstale":"2024-02-23T00:38:24.170926+0000","last_undegraded":"2024-02-23T00:38:24.170926+0000","last_fullsized":"2024-02-23T00:38:24.170926+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.082000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.082000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":180,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.170800+0000","last_change":"2024-02-23T00:38:00.977387+0000","last_active":"2024-02-23T00:38:24.170800+0000","last_peered":"2024-02-23T00:38:24.170800+0000","last_clean":"2024-02-23T00:38:24.170800+0000","last_became_active":"2024-02-23T00:34:41.105998+0000","last_became_peered":"2024-02-23T00:34:41.105998+0000","last_unstale":"2024-02-23T00:38:24.170800+0000","last_undegraded":"2024-02-23T00:38:24.170800+0000","last_fullsized":"2024-02-23T00:38:24.170800+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.977282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.977282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":720,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.174259+0000","last_change":"2024-02-23T00:37:17.499775+0000","last_active":"2024-02-23T00:38:24.174259+0000","last_peered":"2024-02-23T00:38:24.174259+0000","last_clean":"2024-02-23T00:38:24.174259+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:38:24.174259+0000","last_undegraded":"2024-02-23T00:38:24.174259+0000","last_fullsized":"2024-02-23T00:38:24.174259+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:17.499694+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.499694+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":473,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.176019+0000","last_change":"2024-02-23T00:36:37.108691+0000","last_active":"2024-02-23T00:38:24.176019+0000","last_peered":"2024-02-23T00:38:24.176019+0000","last_clean":"2024-02-23T00:36:33.369928+0000","last_became_active":"2024-02-23T00:36:37.108691+0000","last_became_peered":"2024-02-23T00:36:37.108691+0000","last_unstale":"2024-02-23T00:38:24.176019+0000","last_undegraded":"2024-02-23T00:38:24.176019+0000","last_fullsized":"2024-02-23T00:36:37.100335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.196211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.196211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":149,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.174552+0000","last_change":"2024-02-23T00:36:37.123994+0000","last_active":"2024-02-23T00:38:24.174552+0000","last_peered":"2024-02-23T00:38:24.174552+0000","last_clean":"2024-02-23T00:27:29.216066+0000","last_became_active":"2024-02-23T00:36:37.123994+0000","last_became_peered":"2024-02-23T00:36:37.123994+0000","last_unstale":"2024-02-23T00:38:24.174552+0000","last_undegraded":"2024-02-23T00:36:37.103439+0000","last_fullsized":"2024-02-23T00:36:37.103335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:35:01.396982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.396982+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":465,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.180708+0000","last_change":"2024-02-23T00:36:37.120094+0000","last_active":"2024-02-23T00:38:24.180708+0000","last_peered":"2024-02-23T00:38:24.180708+0000","last_clean":"2024-02-23T00:36:33.367908+0000","last_became_active":"2024-02-23T00:36:37.120094+0000","last_became_peered":"2024-02-23T00:36:37.120094+0000","last_unstale":"2024-02-23T00:38:24.180708+0000","last_undegraded":"2024-02-23T00:38:24.180708+0000","last_fullsized":"2024-02-23T00:36:37.107601+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":451,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171517+0000","last_change":"2024-02-23T00:37:26.268389+0000","last_active":"2024-02-23T00:38:24.171517+0000","last_peered":"2024-02-23T00:38:24.171517+0000","last_clean":"2024-02-23T00:38:24.171517+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:38:24.171517+0000","last_undegraded":"2024-02-23T00:38:24.171517+0000","last_fullsized":"2024-02-23T00:38:24.171517+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.268282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":399,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171592+0000","last_change":"2024-02-23T00:35:00.832892+0000","last_active":"2024-02-23T00:38:24.171592+0000","last_peered":"2024-02-23T00:38:24.171592+0000","last_clean":"2024-02-23T00:38:24.171592+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:38:24.171592+0000","last_undegraded":"2024-02-23T00:38:24.171592+0000","last_fullsized":"2024-02-23T00:38:24.171592+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.832818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.832818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":461,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.185673+0000","last_change":"2024-02-23T00:36:37.117616+0000","last_active":"2024-02-23T00:38:24.185673+0000","last_peered":"2024-02-23T00:38:24.185673+0000","last_clean":"2024-02-23T00:36:33.368573+0000","last_became_active":"2024-02-23T00:36:37.117616+0000","last_became_peered":"2024-02-23T00:36:37.117616+0000","last_unstale":"2024-02-23T00:38:24.185673+0000","last_undegraded":"2024-02-23T00:38:24.185673+0000","last_fullsized":"2024-02-23T00:36:37.100549+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":134,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.175631+0000","last_change":"2024-02-23T00:38:19.160002+0000","last_active":"2024-02-23T00:38:24.175631+0000","last_peered":"2024-02-23T00:38:24.175631+0000","last_clean":"2024-02-23T00:27:24.365389+0000","last_became_active":"2024-02-23T00:38:19.160002+0000","last_became_peered":"2024-02-23T00:38:19.160002+0000","last_unstale":"2024-02-23T00:38:24.175631+0000","last_undegraded":"2024-02-23T00:38:19.155620+0000","last_fullsized":"2024-02-23T00:38:19.155500+0000","mapping_epoch":425,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.180820+0000","last_change":"2024-02-23T00:36:37.126102+0000","last_active":"2024-02-23T00:38:24.180820+0000","last_peered":"2024-02-23T00:38:24.180820+0000","last_clean":"2024-02-23T00:36:33.365781+0000","last_became_active":"2024-02-23T00:36:37.126102+0000","last_became_peered":"2024-02-23T00:36:37.126102+0000","last_unstale":"2024-02-23T00:38:24.180820+0000","last_undegraded":"2024-02-23T00:38:24.180820+0000","last_fullsized":"2024-02-23T00:36:37.112617+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.980396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.980396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":383,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.170412+0000","last_change":"2024-02-23T00:36:37.111324+0000","last_active":"2024-02-23T00:38:24.170412+0000","last_peered":"2024-02-23T00:38:24.170412+0000","last_clean":"2024-02-23T00:34:39.448365+0000","last_became_active":"2024-02-23T00:36:37.111324+0000","last_became_peered":"2024-02-23T00:36:37.111324+0000","last_unstale":"2024-02-23T00:38:24.170412+0000","last_undegraded":"2024-02-23T00:38:24.170412+0000","last_fullsized":"2024-02-23T00:36:37.093794+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:31.593354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:31.593354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":455,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.175979+0000","last_change":"2024-02-23T00:35:02.744199+0000","last_active":"2024-02-23T00:38:24.175979+0000","last_peered":"2024-02-23T00:38:24.175979+0000","last_clean":"2024-02-23T00:38:24.175979+0000","last_became_active":"2024-02-23T00:34:41.118378+0000","last_became_peered":"2024-02-23T00:34:41.118378+0000","last_unstale":"2024-02-23T00:38:24.175979+0000","last_undegraded":"2024-02-23T00:38:24.175979+0000","last_fullsized":"2024-02-23T00:38:24.175979+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.744088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.744088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":408,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171137+0000","last_change":"2024-02-23T00:35:55.223009+0000","last_active":"2024-02-23T00:38:24.171137+0000","last_peered":"2024-02-23T00:38:24.171137+0000","last_clean":"2024-02-23T00:38:24.171137+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:38:24.171137+0000","last_undegraded":"2024-02-23T00:38:24.171137+0000","last_fullsized":"2024-02-23T00:38:24.171137+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.222912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.222912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":469,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178745+0000","last_change":"2024-02-23T00:36:41.937397+0000","last_active":"2024-02-23T00:38:24.178745+0000","last_peered":"2024-02-23T00:38:24.178745+0000","last_clean":"2024-02-23T00:38:24.178745+0000","last_became_active":"2024-02-23T00:35:27.093541+0000","last_became_peered":"2024-02-23T00:35:27.093541+0000","last_unstale":"2024-02-23T00:38:24.178745+0000","last_undegraded":"2024-02-23T00:38:24.178745+0000","last_fullsized":"2024-02-23T00:38:24.178745+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.937344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.937344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":505,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178981+0000","last_change":"2024-02-23T00:38:03.327200+0000","last_active":"2024-02-23T00:38:24.178981+0000","last_peered":"2024-02-23T00:38:24.178981+0000","last_clean":"2024-02-23T00:38:24.178981+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:38:24.178981+0000","last_undegraded":"2024-02-23T00:38:24.178981+0000","last_fullsized":"2024-02-23T00:38:24.178981+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:03.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.327127+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":468,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.180950+0000","last_change":"2024-02-23T00:36:37.106377+0000","last_active":"2024-02-23T00:38:24.180950+0000","last_peered":"2024-02-23T00:38:24.180950+0000","last_clean":"2024-02-23T00:36:33.365867+0000","last_became_active":"2024-02-23T00:36:37.106377+0000","last_became_peered":"2024-02-23T00:36:37.106377+0000","last_unstale":"2024-02-23T00:38:24.180950+0000","last_undegraded":"2024-02-23T00:38:24.180950+0000","last_fullsized":"2024-02-23T00:36:37.100069+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":455,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.174281+0000","last_change":"2024-02-23T00:36:33.160039+0000","last_active":"2024-02-23T00:38:24.174281+0000","last_peered":"2024-02-23T00:38:24.174281+0000","last_clean":"2024-02-23T00:38:24.174281+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:38:24.174281+0000","last_undegraded":"2024-02-23T00:38:24.174281+0000","last_fullsized":"2024-02-23T00:38:24.174281+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.159941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.159941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":272,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.177247+0000","last_change":"2024-02-23T00:37:26.933362+0000","last_active":"2024-02-23T00:38:24.177247+0000","last_peered":"2024-02-23T00:38:24.177247+0000","last_clean":"2024-02-23T00:38:24.177247+0000","last_became_active":"2024-02-23T00:34:41.099598+0000","last_became_peered":"2024-02-23T00:34:41.099598+0000","last_unstale":"2024-02-23T00:38:24.177247+0000","last_undegraded":"2024-02-23T00:38:24.177247+0000","last_fullsized":"2024-02-23T00:38:24.177247+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.933288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.933288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":458,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.185790+0000","last_change":"2024-02-23T00:37:20.008803+0000","last_active":"2024-02-23T00:38:24.185790+0000","last_peered":"2024-02-23T00:38:24.185790+0000","last_clean":"2024-02-23T00:38:24.185790+0000","last_became_active":"2024-02-23T00:34:41.114270+0000","last_became_peered":"2024-02-23T00:34:41.114270+0000","last_unstale":"2024-02-23T00:38:24.185790+0000","last_undegraded":"2024-02-23T00:38:24.185790+0000","last_fullsized":"2024-02-23T00:38:24.185790+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.008726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":410,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172117+0000","last_change":"2024-02-23T00:35:52.175358+0000","last_active":"2024-02-23T00:38:24.172117+0000","last_peered":"2024-02-23T00:38:24.172117+0000","last_clean":"2024-02-23T00:38:24.172117+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:38:24.172117+0000","last_undegraded":"2024-02-23T00:38:24.172117+0000","last_fullsized":"2024-02-23T00:38:24.172117+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.175246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.175246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":477,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172512+0000","last_change":"2024-02-23T00:37:59.003696+0000","last_active":"2024-02-23T00:38:24.172512+0000","last_peered":"2024-02-23T00:38:24.172512+0000","last_clean":"2024-02-23T00:38:24.172512+0000","last_became_active":"2024-02-23T00:35:27.097186+0000","last_became_peered":"2024-02-23T00:35:27.097186+0000","last_unstale":"2024-02-23T00:38:24.172512+0000","last_undegraded":"2024-02-23T00:38:24.172512+0000","last_fullsized":"2024-02-23T00:38:24.172512+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.003583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.003583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":543,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.177238+0000","last_change":"2024-02-23T00:38:19.164063+0000","last_active":"2024-02-23T00:38:24.177238+0000","last_peered":"2024-02-23T00:38:24.177238+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:38:19.164063+0000","last_became_peered":"2024-02-23T00:38:19.164063+0000","last_unstale":"2024-02-23T00:38:24.177238+0000","last_undegraded":"2024-02-23T00:38:19.153386+0000","last_fullsized":"2024-02-23T00:38:19.153249+0000","mapping_epoch":425,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":482,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.171405+0000","last_change":"2024-02-23T00:36:37.119168+0000","last_active":"2024-02-23T00:38:24.171405+0000","last_peered":"2024-02-23T00:38:24.171405+0000","last_clean":"2024-02-23T00:36:33.193740+0000","last_became_active":"2024-02-23T00:36:37.119168+0000","last_became_peered":"2024-02-23T00:36:37.119168+0000","last_unstale":"2024-02-23T00:38:24.171405+0000","last_undegraded":"2024-02-23T00:38:24.171405+0000","last_fullsized":"2024-02-23T00:36:37.101741+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.184937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.184937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":713,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172209+0000","last_change":"2024-02-23T00:37:13.467508+0000","last_active":"2024-02-23T00:38:24.172209+0000","last_peered":"2024-02-23T00:38:24.172209+0000","last_clean":"2024-02-23T00:38:24.172209+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:38:24.172209+0000","last_undegraded":"2024-02-23T00:38:24.172209+0000","last_fullsized":"2024-02-23T00:38:24.172209+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:13.467428+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:13.467428+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":467,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.180603+0000","last_change":"2024-02-23T00:37:12.023243+0000","last_active":"2024-02-23T00:38:24.180603+0000","last_peered":"2024-02-23T00:38:24.180603+0000","last_clean":"2024-02-23T00:38:24.180603+0000","last_became_active":"2024-02-23T00:34:41.105566+0000","last_became_peered":"2024-02-23T00:34:41.105566+0000","last_unstale":"2024-02-23T00:38:24.180603+0000","last_undegraded":"2024-02-23T00:38:24.180603+0000","last_fullsized":"2024-02-23T00:38:24.180603+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:12.023125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:12.023125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":414,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171707+0000","last_change":"2024-02-23T00:37:09.743047+0000","last_active":"2024-02-23T00:38:24.171707+0000","last_peered":"2024-02-23T00:38:24.171707+0000","last_clean":"2024-02-23T00:38:24.171707+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:38:24.171707+0000","last_undegraded":"2024-02-23T00:38:24.171707+0000","last_fullsized":"2024-02-23T00:38:24.171707+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.742938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.742938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":476,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.171168+0000","last_change":"2024-02-23T00:36:37.101373+0000","last_active":"2024-02-23T00:38:24.171168+0000","last_peered":"2024-02-23T00:38:24.171168+0000","last_clean":"2024-02-23T00:36:33.192308+0000","last_became_active":"2024-02-23T00:36:37.101373+0000","last_became_peered":"2024-02-23T00:36:37.101373+0000","last_unstale":"2024-02-23T00:38:24.171168+0000","last_undegraded":"2024-02-23T00:38:24.171168+0000","last_fullsized":"2024-02-23T00:36:37.094754+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.595447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.595447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":416,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.173680+0000","last_change":"2024-02-23T00:36:37.109549+0000","last_active":"2024-02-23T00:38:24.173680+0000","last_peered":"2024-02-23T00:38:24.173680+0000","last_clean":"2024-02-23T00:35:25.318841+0000","last_became_active":"2024-02-23T00:36:37.109549+0000","last_became_peered":"2024-02-23T00:36:37.109549+0000","last_unstale":"2024-02-23T00:38:24.173680+0000","last_undegraded":"2024-02-23T00:38:24.173680+0000","last_fullsized":"2024-02-23T00:36:37.096473+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"453'47","reported_seq":703,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.176454+0000","last_change":"2024-02-23T00:37:15.422544+0000","last_active":"2024-02-23T00:38:25.176454+0000","last_peered":"2024-02-23T00:38:25.176454+0000","last_clean":"2024-02-23T00:38:25.176454+0000","last_became_active":"2024-02-23T00:34:41.099241+0000","last_became_peered":"2024-02-23T00:34:41.099241+0000","last_unstale":"2024-02-23T00:38:25.176454+0000","last_undegraded":"2024-02-23T00:38:25.176454+0000","last_fullsized":"2024-02-23T00:38:25.176454+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'35","last_scrub_stamp":"2024-02-23T00:37:15.422435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:15.422435+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":95,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171814+0000","last_change":"2024-02-23T00:37:23.360353+0000","last_active":"2024-02-23T00:38:24.171814+0000","last_peered":"2024-02-23T00:38:24.171814+0000","last_clean":"2024-02-23T00:38:24.171814+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:38:24.171814+0000","last_undegraded":"2024-02-23T00:38:24.171814+0000","last_fullsized":"2024-02-23T00:38:24.171814+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:37:23.360230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:23.360230+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":174,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171505+0000","last_change":"2024-02-23T00:36:32.098355+0000","last_active":"2024-02-23T00:38:24.171505+0000","last_peered":"2024-02-23T00:38:24.171505+0000","last_clean":"2024-02-23T00:38:24.171505+0000","last_became_active":"2024-02-23T00:34:41.109972+0000","last_became_peered":"2024-02-23T00:34:41.109972+0000","last_unstale":"2024-02-23T00:38:24.171505+0000","last_undegraded":"2024-02-23T00:38:24.171505+0000","last_fullsized":"2024-02-23T00:38:24.171505+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.098280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.098280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":414,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172703+0000","last_change":"2024-02-23T00:37:11.766364+0000","last_active":"2024-02-23T00:38:24.172703+0000","last_peered":"2024-02-23T00:38:24.172703+0000","last_clean":"2024-02-23T00:38:24.172703+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:38:24.172703+0000","last_undegraded":"2024-02-23T00:38:24.172703+0000","last_fullsized":"2024-02-23T00:38:24.172703+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:11.766284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:11.766284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":374,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.177816+0000","last_change":"2024-02-23T00:36:23.210776+0000","last_active":"2024-02-23T00:38:24.177816+0000","last_peered":"2024-02-23T00:38:24.177816+0000","last_clean":"2024-02-23T00:38:24.177816+0000","last_became_active":"2024-02-23T00:34:41.109182+0000","last_became_peered":"2024-02-23T00:34:41.109182+0000","last_unstale":"2024-02-23T00:38:24.177816+0000","last_undegraded":"2024-02-23T00:38:24.177816+0000","last_fullsized":"2024-02-23T00:38:24.177816+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:23.210706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.210706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":91,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.175935+0000","last_change":"2024-02-23T00:36:37.104760+0000","last_active":"2024-02-23T00:38:24.175935+0000","last_peered":"2024-02-23T00:38:24.175935+0000","last_clean":"2024-02-23T00:27:24.172824+0000","last_became_active":"2024-02-23T00:36:37.104760+0000","last_became_peered":"2024-02-23T00:36:37.104760+0000","last_unstale":"2024-02-23T00:38:24.175935+0000","last_undegraded":"2024-02-23T00:38:24.175935+0000","last_fullsized":"2024-02-23T00:36:37.098375+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":128,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.177396+0000","last_change":"2024-02-23T00:36:37.110364+0000","last_active":"2024-02-23T00:38:24.177396+0000","last_peered":"2024-02-23T00:38:24.177396+0000","last_clean":"2024-02-23T00:27:29.173100+0000","last_became_active":"2024-02-23T00:36:37.110364+0000","last_became_peered":"2024-02-23T00:36:37.110364+0000","last_unstale":"2024-02-23T00:38:24.177396+0000","last_undegraded":"2024-02-23T00:36:37.096575+0000","last_fullsized":"2024-02-23T00:36:37.096459+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:35:44.214727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:44.214727+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":461,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.174449+0000","last_change":"2024-02-23T00:36:37.123256+0000","last_active":"2024-02-23T00:38:24.174449+0000","last_peered":"2024-02-23T00:38:24.174449+0000","last_clean":"2024-02-23T00:36:33.366566+0000","last_became_active":"2024-02-23T00:36:37.123256+0000","last_became_peered":"2024-02-23T00:36:37.123256+0000","last_unstale":"2024-02-23T00:38:24.174449+0000","last_undegraded":"2024-02-23T00:38:24.174449+0000","last_fullsized":"2024-02-23T00:36:37.096440+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":369,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.169931+0000","last_change":"2024-02-23T00:36:37.113860+0000","last_active":"2024-02-23T00:38:24.169931+0000","last_peered":"2024-02-23T00:38:24.169931+0000","last_clean":"2024-02-23T00:34:39.448366+0000","last_became_active":"2024-02-23T00:36:37.113860+0000","last_became_peered":"2024-02-23T00:36:37.113860+0000","last_unstale":"2024-02-23T00:38:24.169931+0000","last_undegraded":"2024-02-23T00:38:24.169931+0000","last_fullsized":"2024-02-23T00:36:37.101768+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.488959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.488959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":408,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.169906+0000","last_change":"2024-02-23T00:36:42.093272+0000","last_active":"2024-02-23T00:38:24.169906+0000","last_peered":"2024-02-23T00:38:24.169906+0000","last_clean":"2024-02-23T00:38:24.169906+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:38:24.169906+0000","last_undegraded":"2024-02-23T00:38:24.169906+0000","last_fullsized":"2024-02-23T00:38:24.169906+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:36:42.093181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":473,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.169854+0000","last_change":"2024-02-23T00:36:37.124382+0000","last_active":"2024-02-23T00:38:24.169854+0000","last_peered":"2024-02-23T00:38:24.169854+0000","last_clean":"2024-02-23T00:36:33.193431+0000","last_became_active":"2024-02-23T00:36:37.124382+0000","last_became_peered":"2024-02-23T00:36:37.124382+0000","last_unstale":"2024-02-23T00:38:24.169854+0000","last_undegraded":"2024-02-23T00:38:24.169854+0000","last_fullsized":"2024-02-23T00:36:37.101474+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:24.016837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:24.016837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":643,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.170553+0000","last_change":"2024-02-23T00:38:19.180258+0000","last_active":"2024-02-23T00:38:24.170553+0000","last_peered":"2024-02-23T00:38:24.170553+0000","last_clean":"2024-02-23T00:38:24.170553+0000","last_became_active":"2024-02-23T00:38:19.179862+0000","last_became_peered":"2024-02-23T00:38:19.179862+0000","last_unstale":"2024-02-23T00:38:24.170553+0000","last_undegraded":"2024-02-23T00:38:24.170553+0000","last_fullsized":"2024-02-23T00:38:24.170553+0000","mapping_epoch":349,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:06.122240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.122240+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":480,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.172269+0000","last_change":"2024-02-23T00:36:37.118733+0000","last_active":"2024-02-23T00:38:24.172269+0000","last_peered":"2024-02-23T00:38:24.172269+0000","last_clean":"2024-02-23T00:36:36.130983+0000","last_became_active":"2024-02-23T00:36:37.118733+0000","last_became_peered":"2024-02-23T00:36:37.118733+0000","last_unstale":"2024-02-23T00:38:24.172269+0000","last_undegraded":"2024-02-23T00:36:37.095672+0000","last_fullsized":"2024-02-23T00:36:37.095564+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:36:36.130868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.130868+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":179,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178917+0000","last_change":"2024-02-23T00:38:10.172149+0000","last_active":"2024-02-23T00:38:24.178917+0000","last_peered":"2024-02-23T00:38:24.178917+0000","last_clean":"2024-02-23T00:38:24.178917+0000","last_became_active":"2024-02-23T00:34:41.110345+0000","last_became_peered":"2024-02-23T00:34:41.110345+0000","last_unstale":"2024-02-23T00:38:24.178917+0000","last_undegraded":"2024-02-23T00:38:24.178917+0000","last_fullsized":"2024-02-23T00:38:24.178917+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.172059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.172059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":404,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172467+0000","last_change":"2024-02-23T00:36:28.184266+0000","last_active":"2024-02-23T00:38:24.172467+0000","last_peered":"2024-02-23T00:38:24.172467+0000","last_clean":"2024-02-23T00:38:24.172467+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:38:24.172467+0000","last_undegraded":"2024-02-23T00:38:24.172467+0000","last_fullsized":"2024-02-23T00:38:24.172467+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:28.184197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:28.184197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":142,"reported_epoch":487,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:24.175055+0000","last_change":"2024-02-23T00:38:22.196082+0000","last_active":"2024-02-23T00:38:24.175055+0000","last_peered":"2024-02-23T00:38:24.175055+0000","last_clean":"2024-02-23T00:27:24.365389+0000","last_became_active":"2024-02-23T00:38:22.170962+0000","last_became_peered":"2024-02-23T00:38:22.170962+0000","last_unstale":"2024-02-23T00:38:24.175055+0000","last_undegraded":"2024-02-23T00:38:22.130658+0000","last_fullsized":"2024-02-23T00:38:22.130434+0000","mapping_epoch":484,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":468,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.169781+0000","last_change":"2024-02-23T00:36:37.111836+0000","last_active":"2024-02-23T00:38:24.169781+0000","last_peered":"2024-02-23T00:38:24.169781+0000","last_clean":"2024-02-23T00:36:33.193391+0000","last_became_active":"2024-02-23T00:36:37.111836+0000","last_became_peered":"2024-02-23T00:36:37.111836+0000","last_unstale":"2024-02-23T00:38:24.169781+0000","last_undegraded":"2024-02-23T00:38:24.169781+0000","last_fullsized":"2024-02-23T00:36:37.101173+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.667517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.667517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":174,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.177410+0000","last_change":"2024-02-23T00:36:37.118505+0000","last_active":"2024-02-23T00:38:24.177410+0000","last_peered":"2024-02-23T00:38:24.177410+0000","last_clean":"2024-02-23T00:36:33.367279+0000","last_became_active":"2024-02-23T00:36:37.118505+0000","last_became_peered":"2024-02-23T00:36:37.118505+0000","last_unstale":"2024-02-23T00:38:24.177410+0000","last_undegraded":"2024-02-23T00:38:24.177410+0000","last_fullsized":"2024-02-23T00:36:37.097249+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.614960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.614960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":466,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.174067+0000","last_change":"2024-02-23T00:37:22.918309+0000","last_active":"2024-02-23T00:38:24.174067+0000","last_peered":"2024-02-23T00:38:24.174067+0000","last_clean":"2024-02-23T00:38:24.174067+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:38:24.174067+0000","last_undegraded":"2024-02-23T00:38:24.174067+0000","last_fullsized":"2024-02-23T00:38:24.174067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.918248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":172,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.177186+0000","last_change":"2024-02-23T00:34:56.402946+0000","last_active":"2024-02-23T00:38:24.177186+0000","last_peered":"2024-02-23T00:38:24.177186+0000","last_clean":"2024-02-23T00:38:24.177186+0000","last_became_active":"2024-02-23T00:34:41.114943+0000","last_became_peered":"2024-02-23T00:34:41.114943+0000","last_unstale":"2024-02-23T00:38:24.177186+0000","last_undegraded":"2024-02-23T00:38:24.177186+0000","last_fullsized":"2024-02-23T00:38:24.177186+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:56.402851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:56.402851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":552,"reported_epoch":487,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:24.176740+0000","last_change":"2024-02-23T00:38:22.186329+0000","last_active":"2024-02-23T00:38:24.176740+0000","last_peered":"2024-02-23T00:38:24.176740+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:38:22.152766+0000","last_became_peered":"2024-02-23T00:38:22.152766+0000","last_unstale":"2024-02-23T00:38:24.176740+0000","last_undegraded":"2024-02-23T00:38:22.132329+0000","last_fullsized":"2024-02-23T00:38:22.132013+0000","mapping_epoch":484,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":462,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.175511+0000","last_change":"2024-02-23T00:37:08.426793+0000","last_active":"2024-02-23T00:38:24.175511+0000","last_peered":"2024-02-23T00:38:24.175511+0000","last_clean":"2024-02-23T00:38:24.175511+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:38:24.175511+0000","last_undegraded":"2024-02-23T00:38:24.175511+0000","last_fullsized":"2024-02-23T00:38:24.175511+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:08.426716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.426716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":455,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.186099+0000","last_change":"2024-02-23T00:35:52.291042+0000","last_active":"2024-02-23T00:38:24.186099+0000","last_peered":"2024-02-23T00:38:24.186099+0000","last_clean":"2024-02-23T00:38:24.186099+0000","last_became_active":"2024-02-23T00:34:41.113722+0000","last_became_peered":"2024-02-23T00:34:41.113722+0000","last_unstale":"2024-02-23T00:38:24.186099+0000","last_undegraded":"2024-02-23T00:38:24.186099+0000","last_fullsized":"2024-02-23T00:38:24.186099+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.290946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.290946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":458,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.180271+0000","last_change":"2024-02-23T00:37:16.011330+0000","last_active":"2024-02-23T00:38:24.180271+0000","last_peered":"2024-02-23T00:38:24.180271+0000","last_clean":"2024-02-23T00:38:24.180271+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:38:24.180271+0000","last_undegraded":"2024-02-23T00:38:24.180271+0000","last_fullsized":"2024-02-23T00:38:24.180271+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:16.011248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:37:16.011248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":418,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.179219+0000","last_change":"2024-02-23T00:37:20.316511+0000","last_active":"2024-02-23T00:38:24.179219+0000","last_peered":"2024-02-23T00:38:24.179219+0000","last_clean":"2024-02-23T00:38:24.179219+0000","last_became_active":"2024-02-23T00:34:41.101132+0000","last_became_peered":"2024-02-23T00:34:41.101132+0000","last_unstale":"2024-02-23T00:38:24.179219+0000","last_undegraded":"2024-02-23T00:38:24.179219+0000","last_fullsized":"2024-02-23T00:38:24.179219+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.316429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.316429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"453'32","reported_seq":257,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.189494+0000","last_change":"2024-02-23T00:34:42.571126+0000","last_active":"2024-02-23T00:38:24.189494+0000","last_peered":"2024-02-23T00:38:24.189494+0000","last_clean":"2024-02-23T00:38:24.189494+0000","last_became_active":"2024-02-23T00:34:41.106795+0000","last_became_peered":"2024-02-23T00:34:41.106795+0000","last_unstale":"2024-02-23T00:38:24.189494+0000","last_undegraded":"2024-02-23T00:38:24.189494+0000","last_fullsized":"2024-02-23T00:38:24.189494+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'22","last_scrub_stamp":"2024-02-23T00:34:15.571539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:15.571539+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":470,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.177007+0000","last_change":"2024-02-23T00:36:37.117696+0000","last_active":"2024-02-23T00:38:24.177007+0000","last_peered":"2024-02-23T00:38:24.177007+0000","last_clean":"2024-02-23T00:36:33.366237+0000","last_became_active":"2024-02-23T00:36:37.117696+0000","last_became_peered":"2024-02-23T00:36:37.117696+0000","last_unstale":"2024-02-23T00:38:24.177007+0000","last_undegraded":"2024-02-23T00:38:24.177007+0000","last_fullsized":"2024-02-23T00:36:37.103226+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:46.269798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.269798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"426'10","reported_seq":596,"reported_epoch":488,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:25.187195+0000","last_change":"2024-02-23T00:36:37.113534+0000","last_active":"2024-02-23T00:38:25.187195+0000","last_peered":"2024-02-23T00:38:25.187195+0000","last_clean":"2024-02-23T00:36:35.444798+0000","last_became_active":"2024-02-23T00:36:37.113534+0000","last_became_peered":"2024-02-23T00:36:37.113534+0000","last_unstale":"2024-02-23T00:38:25.187195+0000","last_undegraded":"2024-02-23T00:36:37.102235+0000","last_fullsized":"2024-02-23T00:36:37.102132+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":368,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.181656+0000","last_change":"2024-02-23T00:36:37.120693+0000","last_active":"2024-02-23T00:38:24.181656+0000","last_peered":"2024-02-23T00:38:24.181656+0000","last_clean":"2024-02-23T00:34:39.449328+0000","last_became_active":"2024-02-23T00:36:37.120693+0000","last_became_peered":"2024-02-23T00:36:37.120693+0000","last_unstale":"2024-02-23T00:38:24.181656+0000","last_undegraded":"2024-02-23T00:38:24.181656+0000","last_fullsized":"2024-02-23T00:36:37.107400+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.521760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":404,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.186557+0000","last_change":"2024-02-23T00:37:08.058170+0000","last_active":"2024-02-23T00:38:24.186557+0000","last_peered":"2024-02-23T00:38:24.186557+0000","last_clean":"2024-02-23T00:38:24.186557+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:38:24.186557+0000","last_undegraded":"2024-02-23T00:38:24.186557+0000","last_fullsized":"2024-02-23T00:38:24.186557+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:08.058075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.058075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":514,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.186947+0000","last_change":"2024-02-23T00:35:01.515684+0000","last_active":"2024-02-23T00:38:24.186947+0000","last_peered":"2024-02-23T00:38:24.186947+0000","last_clean":"2024-02-23T00:38:24.186947+0000","last_became_active":"2024-02-23T00:34:41.110140+0000","last_became_peered":"2024-02-23T00:34:41.110140+0000","last_unstale":"2024-02-23T00:38:24.186947+0000","last_undegraded":"2024-02-23T00:38:24.186947+0000","last_fullsized":"2024-02-23T00:38:24.186947+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'20","last_scrub_stamp":"2024-02-23T00:35:01.515550+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.515550+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":374,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.171680+0000","last_change":"2024-02-23T00:36:37.118542+0000","last_active":"2024-02-23T00:38:24.171680+0000","last_peered":"2024-02-23T00:38:24.171680+0000","last_clean":"2024-02-23T00:34:39.449034+0000","last_became_active":"2024-02-23T00:36:37.118542+0000","last_became_peered":"2024-02-23T00:36:37.118542+0000","last_unstale":"2024-02-23T00:38:24.171680+0000","last_undegraded":"2024-02-23T00:38:24.171680+0000","last_fullsized":"2024-02-23T00:36:37.095362+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.391997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.391997+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"330'10","reported_seq":303,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.182391+0000","last_change":"2024-02-23T00:37:27.870730+0000","last_active":"2024-02-23T00:38:25.182391+0000","last_peered":"2024-02-23T00:38:25.182391+0000","last_clean":"2024-02-23T00:38:25.182391+0000","last_became_active":"2024-02-23T00:34:41.109074+0000","last_became_peered":"2024-02-23T00:34:41.109074+0000","last_unstale":"2024-02-23T00:38:25.182391+0000","last_undegraded":"2024-02-23T00:38:25.182391+0000","last_fullsized":"2024-02-23T00:38:25.182391+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'10","last_scrub_stamp":"2024-02-23T00:37:27.870653+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:37:27.870653+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":460,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172449+0000","last_change":"2024-02-23T00:38:05.095789+0000","last_active":"2024-02-23T00:38:24.172449+0000","last_peered":"2024-02-23T00:38:24.172449+0000","last_clean":"2024-02-23T00:38:24.172449+0000","last_became_active":"2024-02-23T00:27:23.024869+0000","last_became_peered":"2024-02-23T00:27:23.024869+0000","last_unstale":"2024-02-23T00:38:24.172449+0000","last_undegraded":"2024-02-23T00:38:24.172449+0000","last_fullsized":"2024-02-23T00:38:24.172449+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:05.095708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:05.095708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":177,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.183171+0000","last_change":"2024-02-23T00:38:09.447387+0000","last_active":"2024-02-23T00:38:24.183171+0000","last_peered":"2024-02-23T00:38:24.183171+0000","last_clean":"2024-02-23T00:38:24.183171+0000","last_became_active":"2024-02-23T00:34:41.091189+0000","last_became_peered":"2024-02-23T00:34:41.091189+0000","last_unstale":"2024-02-23T00:38:24.183171+0000","last_undegraded":"2024-02-23T00:38:24.183171+0000","last_fullsized":"2024-02-23T00:38:24.183171+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:09.447326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.447326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":480,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172097+0000","last_change":"2024-02-23T00:37:20.382193+0000","last_active":"2024-02-23T00:38:24.172097+0000","last_peered":"2024-02-23T00:38:24.172097+0000","last_clean":"2024-02-23T00:38:24.172097+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:38:24.172097+0000","last_undegraded":"2024-02-23T00:38:24.172097+0000","last_fullsized":"2024-02-23T00:38:24.172097+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:37:20.382108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.382108+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":476,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178395+0000","last_change":"2024-02-23T00:38:10.470410+0000","last_active":"2024-02-23T00:38:24.178395+0000","last_peered":"2024-02-23T00:38:24.178395+0000","last_clean":"2024-02-23T00:38:24.178395+0000","last_became_active":"2024-02-23T00:34:41.116840+0000","last_became_peered":"2024-02-23T00:34:41.116840+0000","last_unstale":"2024-02-23T00:38:24.178395+0000","last_undegraded":"2024-02-23T00:38:24.178395+0000","last_fullsized":"2024-02-23T00:38:24.178395+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.470333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.470333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":73,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.173027+0000","last_change":"2024-02-23T00:36:37.108983+0000","last_active":"2024-02-23T00:38:24.173027+0000","last_peered":"2024-02-23T00:38:24.173027+0000","last_clean":"2024-02-23T00:27:20.002687+0000","last_became_active":"2024-02-23T00:36:37.108983+0000","last_became_peered":"2024-02-23T00:36:37.108983+0000","last_unstale":"2024-02-23T00:38:24.173027+0000","last_undegraded":"2024-02-23T00:38:24.173027+0000","last_fullsized":"2024-02-23T00:36:37.099047+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.416493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.416493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":70,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.176917+0000","last_change":"2024-02-23T00:36:37.112999+0000","last_active":"2024-02-23T00:38:24.176917+0000","last_peered":"2024-02-23T00:38:24.176917+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.112999+0000","last_became_peered":"2024-02-23T00:36:37.112999+0000","last_unstale":"2024-02-23T00:38:24.176917+0000","last_undegraded":"2024-02-23T00:38:24.176917+0000","last_fullsized":"2024-02-23T00:36:37.102073+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.413991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.413991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":414,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.169855+0000","last_change":"2024-02-23T00:36:37.097581+0000","last_active":"2024-02-23T00:38:24.169855+0000","last_peered":"2024-02-23T00:38:24.169855+0000","last_clean":"2024-02-23T00:36:33.191411+0000","last_became_active":"2024-02-23T00:36:37.097581+0000","last_became_peered":"2024-02-23T00:36:37.097581+0000","last_unstale":"2024-02-23T00:38:24.169855+0000","last_undegraded":"2024-02-23T00:38:24.169855+0000","last_fullsized":"2024-02-23T00:36:37.093707+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.101501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.101501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":516,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171581+0000","last_change":"2024-02-23T00:38:16.174640+0000","last_active":"2024-02-23T00:38:24.171581+0000","last_peered":"2024-02-23T00:38:24.171581+0000","last_clean":"2024-02-23T00:38:24.171581+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:38:24.171581+0000","last_undegraded":"2024-02-23T00:38:24.171581+0000","last_fullsized":"2024-02-23T00:38:24.171581+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:38:16.174526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.174526+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":458,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.173810+0000","last_change":"2024-02-23T00:36:37.108194+0000","last_active":"2024-02-23T00:38:24.173810+0000","last_peered":"2024-02-23T00:38:24.173810+0000","last_clean":"2024-02-23T00:36:33.367482+0000","last_became_active":"2024-02-23T00:36:37.108194+0000","last_became_peered":"2024-02-23T00:36:37.108194+0000","last_unstale":"2024-02-23T00:38:24.173810+0000","last_undegraded":"2024-02-23T00:38:24.173810+0000","last_fullsized":"2024-02-23T00:36:37.096730+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.783089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.783089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":462,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172180+0000","last_change":"2024-02-23T00:38:06.235671+0000","last_active":"2024-02-23T00:38:24.172180+0000","last_peered":"2024-02-23T00:38:24.172180+0000","last_clean":"2024-02-23T00:38:24.172180+0000","last_became_active":"2024-02-23T00:34:41.115862+0000","last_became_peered":"2024-02-23T00:34:41.115862+0000","last_unstale":"2024-02-23T00:38:24.172180+0000","last_undegraded":"2024-02-23T00:38:24.172180+0000","last_fullsized":"2024-02-23T00:38:24.172180+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:06.235608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.235608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":73,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.172516+0000","last_change":"2024-02-23T00:36:37.116289+0000","last_active":"2024-02-23T00:38:24.172516+0000","last_peered":"2024-02-23T00:38:24.172516+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.116289+0000","last_became_peered":"2024-02-23T00:36:37.116289+0000","last_unstale":"2024-02-23T00:38:24.172516+0000","last_undegraded":"2024-02-23T00:38:24.172516+0000","last_fullsized":"2024-02-23T00:36:37.106216+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.620107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":436,"reported_epoch":487,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:24.175703+0000","last_change":"2024-02-23T00:36:37.104333+0000","last_active":"2024-02-23T00:38:24.175703+0000","last_peered":"2024-02-23T00:38:24.175703+0000","last_clean":"2024-02-23T00:36:33.365274+0000","last_became_active":"2024-02-23T00:36:37.104333+0000","last_became_peered":"2024-02-23T00:36:37.104333+0000","last_unstale":"2024-02-23T00:38:24.175703+0000","last_undegraded":"2024-02-23T00:36:37.099294+0000","last_fullsized":"2024-02-23T00:36:37.099175+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:35:47.130749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.130749+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":218,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172301+0000","last_change":"2024-02-23T00:36:36.531491+0000","last_active":"2024-02-23T00:38:24.172301+0000","last_peered":"2024-02-23T00:38:24.172301+0000","last_clean":"2024-02-23T00:38:24.172301+0000","last_became_active":"2024-02-23T00:34:41.101219+0000","last_became_peered":"2024-02-23T00:34:41.101219+0000","last_unstale":"2024-02-23T00:38:24.172301+0000","last_undegraded":"2024-02-23T00:38:24.172301+0000","last_fullsized":"2024-02-23T00:38:24.172301+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:36:36.531424+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.531424+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":455,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.186345+0000","last_change":"2024-02-23T00:36:32.199878+0000","last_active":"2024-02-23T00:38:24.186345+0000","last_peered":"2024-02-23T00:38:24.186345+0000","last_clean":"2024-02-23T00:38:24.186345+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:38:24.186345+0000","last_undegraded":"2024-02-23T00:38:24.186345+0000","last_fullsized":"2024-02-23T00:38:24.186345+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.199765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.199765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":458,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.175436+0000","last_change":"2024-02-23T00:38:02.325317+0000","last_active":"2024-02-23T00:38:24.175436+0000","last_peered":"2024-02-23T00:38:24.175436+0000","last_clean":"2024-02-23T00:38:24.175436+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:38:24.175436+0000","last_undegraded":"2024-02-23T00:38:24.175436+0000","last_fullsized":"2024-02-23T00:38:24.175436+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:02.325254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:02.325254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":458,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176781+0000","last_change":"2024-02-23T00:37:14.009706+0000","last_active":"2024-02-23T00:38:24.176781+0000","last_peered":"2024-02-23T00:38:24.176781+0000","last_clean":"2024-02-23T00:38:24.176781+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:38:24.176781+0000","last_undegraded":"2024-02-23T00:38:24.176781+0000","last_fullsized":"2024-02-23T00:38:24.176781+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.009614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.009614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":399,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172257+0000","last_change":"2024-02-23T00:33:44.152089+0000","last_active":"2024-02-23T00:38:24.172257+0000","last_peered":"2024-02-23T00:38:24.172257+0000","last_clean":"2024-02-23T00:38:24.172257+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:38:24.172257+0000","last_undegraded":"2024-02-23T00:38:24.172257+0000","last_fullsized":"2024-02-23T00:38:24.172257+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:44.152014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:44.152014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":272,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.177564+0000","last_change":"2024-02-23T00:37:26.914783+0000","last_active":"2024-02-23T00:38:24.177564+0000","last_peered":"2024-02-23T00:38:24.177564+0000","last_clean":"2024-02-23T00:38:24.177564+0000","last_became_active":"2024-02-23T00:34:41.114516+0000","last_became_peered":"2024-02-23T00:34:41.114516+0000","last_unstale":"2024-02-23T00:38:24.177564+0000","last_undegraded":"2024-02-23T00:38:24.177564+0000","last_fullsized":"2024-02-23T00:38:24.177564+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:37:26.914698+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.914698+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":461,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.182746+0000","last_change":"2024-02-23T00:37:22.340222+0000","last_active":"2024-02-23T00:38:24.182746+0000","last_peered":"2024-02-23T00:38:24.182746+0000","last_clean":"2024-02-23T00:38:24.182746+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:38:24.182746+0000","last_undegraded":"2024-02-23T00:38:24.182746+0000","last_fullsized":"2024-02-23T00:38:24.182746+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:22.340128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.340128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":460,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.181151+0000","last_change":"2024-02-23T00:38:16.055268+0000","last_active":"2024-02-23T00:38:24.181151+0000","last_peered":"2024-02-23T00:38:24.181151+0000","last_clean":"2024-02-23T00:38:24.181151+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:38:24.181151+0000","last_undegraded":"2024-02-23T00:38:24.181151+0000","last_fullsized":"2024-02-23T00:38:24.181151+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.055159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":72,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.175986+0000","last_change":"2024-02-23T00:36:37.114573+0000","last_active":"2024-02-23T00:38:24.175986+0000","last_peered":"2024-02-23T00:38:24.175986+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.114573+0000","last_became_peered":"2024-02-23T00:36:37.114573+0000","last_unstale":"2024-02-23T00:38:24.175986+0000","last_undegraded":"2024-02-23T00:38:24.175986+0000","last_fullsized":"2024-02-23T00:36:37.098725+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.175171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.175171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":335,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.179280+0000","last_change":"2024-02-23T00:36:04.097119+0000","last_active":"2024-02-23T00:38:24.179280+0000","last_peered":"2024-02-23T00:38:24.179280+0000","last_clean":"2024-02-23T00:38:24.179280+0000","last_became_active":"2024-02-23T00:36:04.096804+0000","last_became_peered":"2024-02-23T00:36:04.096804+0000","last_unstale":"2024-02-23T00:38:24.179280+0000","last_undegraded":"2024-02-23T00:38:24.179280+0000","last_fullsized":"2024-02-23T00:38:24.179280+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:53.153821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:53.153821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":544,"reported_epoch":487,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:24.191888+0000","last_change":"2024-02-23T00:38:24.190766+0000","last_active":"2024-02-23T00:38:24.191888+0000","last_peered":"2024-02-23T00:38:24.191888+0000","last_clean":"2024-02-23T00:38:24.191888+0000","last_became_active":"2024-02-23T00:38:24.190269+0000","last_became_peered":"2024-02-23T00:38:24.190269+0000","last_unstale":"2024-02-23T00:38:24.191888+0000","last_undegraded":"2024-02-23T00:38:24.191888+0000","last_fullsized":"2024-02-23T00:38:24.191888+0000","mapping_epoch":486,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":459,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176766+0000","last_change":"2024-02-23T00:34:42.064110+0000","last_active":"2024-02-23T00:38:24.176766+0000","last_peered":"2024-02-23T00:38:24.176766+0000","last_clean":"2024-02-23T00:38:24.176766+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:38:24.176766+0000","last_undegraded":"2024-02-23T00:38:24.176766+0000","last_fullsized":"2024-02-23T00:38:24.176766+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:32.520027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:32.520027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":603,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-02-23T00:38:25.182608+0000","last_change":"2024-02-23T00:36:39.246854+0000","last_active":"2024-02-23T00:38:25.182608+0000","last_peered":"2024-02-23T00:38:25.182608+0000","last_clean":"2024-02-23T00:38:25.182608+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:38:25.182608+0000","last_undegraded":"2024-02-23T00:38:25.182608+0000","last_fullsized":"2024-02-23T00:38:25.182608+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:36:39.246273+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.246273+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":460,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178921+0000","last_change":"2024-02-23T00:38:09.207338+0000","last_active":"2024-02-23T00:38:24.178921+0000","last_peered":"2024-02-23T00:38:24.178921+0000","last_clean":"2024-02-23T00:38:24.178921+0000","last_became_active":"2024-02-23T00:27:23.030147+0000","last_became_peered":"2024-02-23T00:27:23.030147+0000","last_unstale":"2024-02-23T00:38:24.178921+0000","last_undegraded":"2024-02-23T00:38:24.178921+0000","last_fullsized":"2024-02-23T00:38:24.178921+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:38:09.207232+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:38:09.207232+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.207232+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":422,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.179017+0000","last_change":"2024-02-23T00:36:37.122220+0000","last_active":"2024-02-23T00:38:24.179017+0000","last_peered":"2024-02-23T00:38:24.179017+0000","last_clean":"2024-02-23T00:36:33.368644+0000","last_became_active":"2024-02-23T00:36:37.122220+0000","last_became_peered":"2024-02-23T00:36:37.122220+0000","last_unstale":"2024-02-23T00:38:24.179017+0000","last_undegraded":"2024-02-23T00:38:24.179017+0000","last_fullsized":"2024-02-23T00:36:37.109811+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.028166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.028166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":616,"reported_epoch":487,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:24.176311+0000","last_change":"2024-02-23T00:38:22.211888+0000","last_active":"2024-02-23T00:38:24.176311+0000","last_peered":"2024-02-23T00:38:24.176311+0000","last_clean":"2024-02-23T00:38:19.193308+0000","last_became_active":"2024-02-23T00:38:22.159363+0000","last_became_peered":"2024-02-23T00:38:22.159363+0000","last_unstale":"2024-02-23T00:38:24.176311+0000","last_undegraded":"2024-02-23T00:38:24.176311+0000","last_fullsized":"2024-02-23T00:38:24.176311+0000","mapping_epoch":484,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":178,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.175448+0000","last_change":"2024-02-23T00:36:37.114328+0000","last_active":"2024-02-23T00:38:24.175448+0000","last_peered":"2024-02-23T00:38:24.175448+0000","last_clean":"2024-02-23T00:36:33.365066+0000","last_became_active":"2024-02-23T00:36:37.114328+0000","last_became_peered":"2024-02-23T00:36:37.114328+0000","last_unstale":"2024-02-23T00:38:24.175448+0000","last_undegraded":"2024-02-23T00:38:24.175448+0000","last_fullsized":"2024-02-23T00:36:37.098495+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.151731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":70,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.183075+0000","last_change":"2024-02-23T00:36:37.122400+0000","last_active":"2024-02-23T00:38:24.183075+0000","last_peered":"2024-02-23T00:38:24.183075+0000","last_clean":"2024-02-23T00:27:20.002687+0000","last_became_active":"2024-02-23T00:36:37.122400+0000","last_became_peered":"2024-02-23T00:36:37.122400+0000","last_unstale":"2024-02-23T00:38:24.183075+0000","last_undegraded":"2024-02-23T00:38:24.183075+0000","last_fullsized":"2024-02-23T00:36:37.111280+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":451,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.173531+0000","last_change":"2024-02-23T00:35:57.658486+0000","last_active":"2024-02-23T00:38:24.173531+0000","last_peered":"2024-02-23T00:38:24.173531+0000","last_clean":"2024-02-23T00:38:24.173531+0000","last_became_active":"2024-02-23T00:27:23.028456+0000","last_became_peered":"2024-02-23T00:27:23.028456+0000","last_unstale":"2024-02-23T00:38:24.173531+0000","last_undegraded":"2024-02-23T00:38:24.173531+0000","last_fullsized":"2024-02-23T00:38:24.173531+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.658406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.658406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":407,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.180947+0000","last_change":"2024-02-23T00:35:39.871154+0000","last_active":"2024-02-23T00:38:24.180947+0000","last_peered":"2024-02-23T00:38:24.180947+0000","last_clean":"2024-02-23T00:38:24.180947+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:38:24.180947+0000","last_undegraded":"2024-02-23T00:38:24.180947+0000","last_fullsized":"2024-02-23T00:38:24.180947+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.871058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.871058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":551,"reported_epoch":488,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:25.200447+0000","last_change":"2024-02-23T00:38:25.200447+0000","last_active":"2024-02-23T00:38:25.200447+0000","last_peered":"2024-02-23T00:38:25.200447+0000","last_clean":"2024-02-23T00:38:25.200447+0000","last_became_active":"2024-02-23T00:38:25.199958+0000","last_became_peered":"2024-02-23T00:38:25.199958+0000","last_unstale":"2024-02-23T00:38:25.200447+0000","last_undegraded":"2024-02-23T00:38:25.200447+0000","last_fullsized":"2024-02-23T00:38:25.200447+0000","mapping_epoch":487,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:37:09.133471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.133471+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1,4],"avail_no_missing":["2","1","4"],"object_location_counts":[{"shards":"1,2,4","objects":6}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":470,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176139+0000","last_change":"2024-02-23T00:38:07.395392+0000","last_active":"2024-02-23T00:38:24.176139+0000","last_peered":"2024-02-23T00:38:24.176139+0000","last_clean":"2024-02-23T00:38:24.176139+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:38:24.176139+0000","last_undegraded":"2024-02-23T00:38:24.176139+0000","last_fullsized":"2024-02-23T00:38:24.176139+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.395260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.395260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":456,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.174078+0000","last_change":"2024-02-23T00:36:37.113234+0000","last_active":"2024-02-23T00:38:24.174078+0000","last_peered":"2024-02-23T00:38:24.174078+0000","last_clean":"2024-02-23T00:36:33.367528+0000","last_became_active":"2024-02-23T00:36:37.113234+0000","last_became_peered":"2024-02-23T00:36:37.113234+0000","last_unstale":"2024-02-23T00:38:24.174078+0000","last_undegraded":"2024-02-23T00:38:24.174078+0000","last_fullsized":"2024-02-23T00:36:37.097125+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.791748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.791748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":455,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171049+0000","last_change":"2024-02-23T00:38:12.024565+0000","last_active":"2024-02-23T00:38:24.171049+0000","last_peered":"2024-02-23T00:38:24.171049+0000","last_clean":"2024-02-23T00:38:24.171049+0000","last_became_active":"2024-02-23T00:27:23.180816+0000","last_became_peered":"2024-02-23T00:27:23.180816+0000","last_unstale":"2024-02-23T00:38:24.171049+0000","last_undegraded":"2024-02-23T00:38:24.171049+0000","last_fullsized":"2024-02-23T00:38:24.171049+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:12.024470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:12.024470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":331,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.183195+0000","last_change":"2024-02-23T00:36:27.951790+0000","last_active":"2024-02-23T00:38:24.183195+0000","last_peered":"2024-02-23T00:38:24.183195+0000","last_clean":"2024-02-23T00:38:24.183195+0000","last_became_active":"2024-02-23T00:36:17.099716+0000","last_became_peered":"2024-02-23T00:36:17.099716+0000","last_unstale":"2024-02-23T00:38:24.183195+0000","last_undegraded":"2024-02-23T00:38:24.183195+0000","last_fullsized":"2024-02-23T00:38:24.183195+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:27.951690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.951690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":540,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.202498+0000","last_change":"2024-02-23T00:38:24.188459+0000","last_active":"2024-02-23T00:38:24.202498+0000","last_peered":"2024-02-23T00:38:24.202498+0000","last_clean":"2024-02-23T00:38:24.202498+0000","last_became_active":"2024-02-23T00:38:24.188073+0000","last_became_peered":"2024-02-23T00:38:24.188073+0000","last_unstale":"2024-02-23T00:38:24.202498+0000","last_undegraded":"2024-02-23T00:38:24.202498+0000","last_fullsized":"2024-02-23T00:38:24.202498+0000","mapping_epoch":486,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":29,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":174,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.179051+0000","last_change":"2024-02-23T00:36:35.929592+0000","last_active":"2024-02-23T00:38:24.179051+0000","last_peered":"2024-02-23T00:38:24.179051+0000","last_clean":"2024-02-23T00:38:24.179051+0000","last_became_active":"2024-02-23T00:34:41.103114+0000","last_became_peered":"2024-02-23T00:34:41.103114+0000","last_unstale":"2024-02-23T00:38:24.179051+0000","last_undegraded":"2024-02-23T00:38:24.179051+0000","last_fullsized":"2024-02-23T00:38:24.179051+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:35.929507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:35.929507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":466,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.171028+0000","last_change":"2024-02-23T00:38:03.063564+0000","last_active":"2024-02-23T00:38:24.171028+0000","last_peered":"2024-02-23T00:38:24.171028+0000","last_clean":"2024-02-23T00:38:24.171028+0000","last_became_active":"2024-02-23T00:34:41.106439+0000","last_became_peered":"2024-02-23T00:34:41.106439+0000","last_unstale":"2024-02-23T00:38:24.171028+0000","last_undegraded":"2024-02-23T00:38:24.171028+0000","last_fullsized":"2024-02-23T00:38:24.171028+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.063485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.063485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":177,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.183095+0000","last_change":"2024-02-23T00:37:19.329028+0000","last_active":"2024-02-23T00:38:24.183095+0000","last_peered":"2024-02-23T00:38:24.183095+0000","last_clean":"2024-02-23T00:38:24.183095+0000","last_became_active":"2024-02-23T00:34:41.113447+0000","last_became_peered":"2024-02-23T00:34:41.113447+0000","last_unstale":"2024-02-23T00:38:24.183095+0000","last_undegraded":"2024-02-23T00:38:24.183095+0000","last_fullsized":"2024-02-23T00:38:24.183095+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:19.328950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:19.328950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":404,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.185316+0000","last_change":"2024-02-23T00:38:03.798894+0000","last_active":"2024-02-23T00:38:24.185316+0000","last_peered":"2024-02-23T00:38:24.185316+0000","last_clean":"2024-02-23T00:38:24.185316+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:38:24.185316+0000","last_undegraded":"2024-02-23T00:38:24.185316+0000","last_fullsized":"2024-02-23T00:38:24.185316+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.798824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.798824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":564,"reported_epoch":487,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:24.171187+0000","last_change":"2024-02-23T00:38:22.215632+0000","last_active":"2024-02-23T00:38:24.171187+0000","last_peered":"2024-02-23T00:38:24.171187+0000","last_clean":"2024-02-23T00:38:13.519476+0000","last_became_active":"2024-02-23T00:38:22.159516+0000","last_became_peered":"2024-02-23T00:38:22.159516+0000","last_unstale":"2024-02-23T00:38:24.171187+0000","last_undegraded":"2024-02-23T00:38:24.171187+0000","last_fullsized":"2024-02-23T00:38:24.171187+0000","mapping_epoch":484,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[1,6,7],"avail_no_missing":["1","6","7"],"object_location_counts":[{"shards":"1,6,7","objects":8}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":171,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.170766+0000","last_change":"2024-02-23T00:37:59.129696+0000","last_active":"2024-02-23T00:38:24.170766+0000","last_peered":"2024-02-23T00:38:24.170766+0000","last_clean":"2024-02-23T00:38:24.170766+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:38:24.170766+0000","last_undegraded":"2024-02-23T00:38:24.170766+0000","last_fullsized":"2024-02-23T00:38:24.170766+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.129604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.129604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":452,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.174376+0000","last_change":"2024-02-23T00:36:37.124006+0000","last_active":"2024-02-23T00:38:24.174376+0000","last_peered":"2024-02-23T00:38:24.174376+0000","last_clean":"2024-02-23T00:36:33.366419+0000","last_became_active":"2024-02-23T00:36:37.124006+0000","last_became_peered":"2024-02-23T00:36:37.124006+0000","last_unstale":"2024-02-23T00:38:24.174376+0000","last_undegraded":"2024-02-23T00:38:24.174376+0000","last_fullsized":"2024-02-23T00:36:37.103625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":70,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.171587+0000","last_change":"2024-02-23T00:36:37.119810+0000","last_active":"2024-02-23T00:38:24.171587+0000","last_peered":"2024-02-23T00:38:24.171587+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.119810+0000","last_became_peered":"2024-02-23T00:36:37.119810+0000","last_unstale":"2024-02-23T00:38:24.171587+0000","last_undegraded":"2024-02-23T00:38:24.171587+0000","last_fullsized":"2024-02-23T00:36:37.098897+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:19.572808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.572808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":412,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.180596+0000","last_change":"2024-02-23T00:37:18.044814+0000","last_active":"2024-02-23T00:38:24.180596+0000","last_peered":"2024-02-23T00:38:24.180596+0000","last_clean":"2024-02-23T00:38:24.180596+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:38:24.180596+0000","last_undegraded":"2024-02-23T00:38:24.180596+0000","last_fullsized":"2024-02-23T00:38:24.180596+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:18.044737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:18.044737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":123,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.191042+0000","last_change":"2024-02-23T00:38:24.190703+0000","last_active":"2024-02-23T00:38:24.191042+0000","last_peered":"2024-02-23T00:38:24.191042+0000","last_clean":"2024-02-23T00:38:24.191042+0000","last_became_active":"2024-02-23T00:38:24.190374+0000","last_became_peered":"2024-02-23T00:38:24.190374+0000","last_unstale":"2024-02-23T00:38:24.191042+0000","last_undegraded":"2024-02-23T00:38:24.191042+0000","last_fullsized":"2024-02-23T00:38:24.191042+0000","mapping_epoch":486,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":465,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.180364+0000","last_change":"2024-02-23T00:36:37.114846+0000","last_active":"2024-02-23T00:38:24.180364+0000","last_peered":"2024-02-23T00:38:24.180364+0000","last_clean":"2024-02-23T00:36:33.367758+0000","last_became_active":"2024-02-23T00:36:37.114846+0000","last_became_peered":"2024-02-23T00:36:37.114846+0000","last_unstale":"2024-02-23T00:38:24.180364+0000","last_undegraded":"2024-02-23T00:38:24.180364+0000","last_fullsized":"2024-02-23T00:36:37.107709+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":460,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.176970+0000","last_change":"2024-02-23T00:37:19.034254+0000","last_active":"2024-02-23T00:38:24.176970+0000","last_peered":"2024-02-23T00:38:24.176970+0000","last_clean":"2024-02-23T00:38:24.176970+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:38:24.176970+0000","last_undegraded":"2024-02-23T00:38:24.176970+0000","last_fullsized":"2024-02-23T00:38:24.176970+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:19.034136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:37:19.034136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":455,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.172412+0000","last_change":"2024-02-23T00:35:41.260304+0000","last_active":"2024-02-23T00:38:24.172412+0000","last_peered":"2024-02-23T00:38:24.172412+0000","last_clean":"2024-02-23T00:38:24.172412+0000","last_became_active":"2024-02-23T00:34:41.112980+0000","last_became_peered":"2024-02-23T00:34:41.112980+0000","last_unstale":"2024-02-23T00:38:24.172412+0000","last_undegraded":"2024-02-23T00:38:24.172412+0000","last_fullsized":"2024-02-23T00:38:24.172412+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:41.260231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.260231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":414,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.171739+0000","last_change":"2024-02-23T00:36:37.115401+0000","last_active":"2024-02-23T00:38:24.171739+0000","last_peered":"2024-02-23T00:38:24.171739+0000","last_clean":"2024-02-23T00:36:33.365083+0000","last_became_active":"2024-02-23T00:36:37.115401+0000","last_became_peered":"2024-02-23T00:36:37.115401+0000","last_unstale":"2024-02-23T00:38:24.171739+0000","last_undegraded":"2024-02-23T00:38:24.171739+0000","last_fullsized":"2024-02-23T00:36:37.099146+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.192483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.192483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":582,"reported_epoch":487,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:24.172379+0000","last_change":"2024-02-23T00:38:22.188799+0000","last_active":"2024-02-23T00:38:24.172379+0000","last_peered":"2024-02-23T00:38:24.172379+0000","last_clean":"2024-02-23T00:38:19.215057+0000","last_became_active":"2024-02-23T00:38:22.178598+0000","last_became_peered":"2024-02-23T00:38:22.178598+0000","last_unstale":"2024-02-23T00:38:24.172379+0000","last_undegraded":"2024-02-23T00:38:24.172379+0000","last_fullsized":"2024-02-23T00:38:24.172379+0000","mapping_epoch":484,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":73,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.186182+0000","last_change":"2024-02-23T00:36:37.123921+0000","last_active":"2024-02-23T00:38:24.186182+0000","last_peered":"2024-02-23T00:38:24.186182+0000","last_clean":"2024-02-23T00:27:15.989319+0000","last_became_active":"2024-02-23T00:36:37.123921+0000","last_became_peered":"2024-02-23T00:36:37.123921+0000","last_unstale":"2024-02-23T00:38:24.186182+0000","last_undegraded":"2024-02-23T00:38:24.186182+0000","last_fullsized":"2024-02-23T00:36:37.106358+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.388834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":113,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.175102+0000","last_change":"2024-02-23T00:38:00.381029+0000","last_active":"2024-02-23T00:38:24.175102+0000","last_peered":"2024-02-23T00:38:24.175102+0000","last_clean":"2024-02-23T00:38:24.175102+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:38:24.175102+0000","last_undegraded":"2024-02-23T00:38:24.175102+0000","last_fullsized":"2024-02-23T00:38:24.175102+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.380938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.380938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":457,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.170581+0000","last_change":"2024-02-23T00:37:58.130556+0000","last_active":"2024-02-23T00:38:24.170581+0000","last_peered":"2024-02-23T00:38:24.170581+0000","last_clean":"2024-02-23T00:38:24.170581+0000","last_became_active":"2024-02-23T00:27:23.025763+0000","last_became_peered":"2024-02-23T00:27:23.025763+0000","last_unstale":"2024-02-23T00:38:24.170581+0000","last_undegraded":"2024-02-23T00:38:24.170581+0000","last_fullsized":"2024-02-23T00:38:24.170581+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:58.130479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:37:58.130479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":72,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.170553+0000","last_change":"2024-02-23T00:36:37.119114+0000","last_active":"2024-02-23T00:38:24.170553+0000","last_peered":"2024-02-23T00:38:24.170553+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119114+0000","last_became_peered":"2024-02-23T00:36:37.119114+0000","last_unstale":"2024-02-23T00:38:24.170553+0000","last_undegraded":"2024-02-23T00:38:24.170553+0000","last_fullsized":"2024-02-23T00:36:37.106746+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.176441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.176441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":650,"reported_epoch":487,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:24.189225+0000","last_change":"2024-02-23T00:38:24.188838+0000","last_active":"2024-02-23T00:38:24.189225+0000","last_peered":"2024-02-23T00:38:24.189225+0000","last_clean":"2024-02-23T00:38:24.189225+0000","last_became_active":"2024-02-23T00:38:24.188244+0000","last_became_peered":"2024-02-23T00:38:24.188244+0000","last_unstale":"2024-02-23T00:38:24.189225+0000","last_undegraded":"2024-02-23T00:38:24.189225+0000","last_fullsized":"2024-02-23T00:38:24.189225+0000","mapping_epoch":486,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:06.122240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.122240+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":42,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5,1],"avail_no_missing":["2","1","5"],"object_location_counts":[{"shards":"1,2,5","objects":9}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":70,"reported_epoch":487,"state":"active+undersized","last_fresh":"2024-02-23T00:38:24.185083+0000","last_change":"2024-02-23T00:36:37.120482+0000","last_active":"2024-02-23T00:38:24.185083+0000","last_peered":"2024-02-23T00:38:24.185083+0000","last_clean":"2024-02-23T00:27:15.989319+0000","last_became_active":"2024-02-23T00:36:37.120482+0000","last_became_peered":"2024-02-23T00:36:37.120482+0000","last_unstale":"2024-02-23T00:38:24.185083+0000","last_undegraded":"2024-02-23T00:38:24.185083+0000","last_fullsized":"2024-02-23T00:36:37.111071+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":457,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.169829+0000","last_change":"2024-02-23T00:37:14.200716+0000","last_active":"2024-02-23T00:38:24.169829+0000","last_peered":"2024-02-23T00:38:24.169829+0000","last_clean":"2024-02-23T00:38:24.169829+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:38:24.169829+0000","last_undegraded":"2024-02-23T00:38:24.169829+0000","last_fullsized":"2024-02-23T00:38:24.169829+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.200606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.200606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":179,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.178701+0000","last_change":"2024-02-23T00:37:24.990742+0000","last_active":"2024-02-23T00:38:24.178701+0000","last_peered":"2024-02-23T00:38:24.178701+0000","last_clean":"2024-02-23T00:38:24.178701+0000","last_became_active":"2024-02-23T00:34:41.109129+0000","last_became_peered":"2024-02-23T00:34:41.109129+0000","last_unstale":"2024-02-23T00:38:24.178701+0000","last_undegraded":"2024-02-23T00:38:24.178701+0000","last_fullsized":"2024-02-23T00:38:24.178701+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:24.990671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:24.990671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":413,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-02-23T00:38:24.173414+0000","last_change":"2024-02-23T00:36:40.485366+0000","last_active":"2024-02-23T00:38:24.173414+0000","last_peered":"2024-02-23T00:38:24.173414+0000","last_clean":"2024-02-23T00:38:24.173414+0000","last_became_active":"2024-02-23T00:34:41.099195+0000","last_became_peered":"2024-02-23T00:34:41.099195+0000","last_unstale":"2024-02-23T00:38:24.173414+0000","last_undegraded":"2024-02-23T00:38:24.173414+0000","last_fullsized":"2024-02-23T00:38:24.173414+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:40.485283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:40.485283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":221,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":80,"acting":80,"num_store_stats":6},{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":4001608,"num_objects":172,"num_object_clones":9,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":180,"num_read_kb":48,"num_write":720,"num_write_kb":4575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":4000297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14045184,"data_stored":12004797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2010,"ondisk_log_size":2010,"up":84,"acting":84,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":87,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":87,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":105,"acting":105,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51,"num_objects_misplaced":84,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":1494,"num_read_kb":1219,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":3328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":724,"ondisk_log_size":724,"up":85,"acting":88,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":347,"num_read_kb":347,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":34,"acting":34,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":81,"acting":81,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332233,"num_pgs":119,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314976,"kb_used_data":9560,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93430176,"statfs":{"total":95995035648,"available":95672500224,"internally_reserved":0,"allocated":9789440,"data_stored":7155662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.02,"15min":1.024},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.79800000000000004,"5min":57.401000000000003,"15min":61.048999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.1719999999999999,"15min":1.2450000000000001},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":65.558999999999997,"15min":73.870999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.53400000000000003,"15min":0.69599999999999995},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":0.68100000000000005,"5min":4.6840000000000002,"15min":31.175000000000001},"last":0.309},{"interface":"front","average":{"1min":0.442,"5min":0.51300000000000001,"15min":0.68700000000000006},"min":{"1min":0.27900000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.59699999999999998,"5min":3.8719999999999999,"15min":29.423999999999999},"last":0.44600000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.50600000000000001,"15min":0.66200000000000003},"min":{"1min":0.30299999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64300000000000002,"5min":3.7240000000000002,"15min":16.603999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56699999999999995,"15min":0.61499999999999999},"min":{"1min":0.36199999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65400000000000003,"5min":4.1369999999999996,"15min":7.5339999999999998},"last":0.38800000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.54000000000000004,"15min":0.97799999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.59099999999999997,"5min":3.7679999999999998,"15min":57.729999999999997},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.57699999999999996,"15min":0.90800000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.312},"max":{"1min":0.69699999999999995,"5min":3.8420000000000001,"15min":27.573},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42899999999999999,"15min":0.41899999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23100000000000001},"max":{"1min":0.58499999999999996,"5min":1.706,"15min":1.706},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.37,"5min":0.45600000000000002,"15min":0.436},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54200000000000004,"5min":3.294,"15min":3.294},"last":0.33100000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.434,"15min":0.43099999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.61399999999999999,"5min":0.85899999999999999,"15min":1.0209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.48699999999999999,"15min":0.45600000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76400000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.433}]},{"osd":6,"last update":"Fri Feb 23 00:37:33 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.51200000000000001,"15min":0.48699999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.28000000000000003},"max":{"1min":0.627,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.48899999999999999,"15min":0.47299999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":0.57299999999999995,"5min":0.83299999999999996,"15min":1.002},"last":0.51600000000000001}]}]},{"osd":6,"up_from":44,"seq":188978561166,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311092,"kb_used_data":5676,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93434060,"statfs":{"total":95995035648,"available":95676477440,"internally_reserved":0,"allocated":5812224,"data_stored":3146621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.6499999999999999,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.78800000000000003,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64800000000000002,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":9.468,"15min":9.468},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.49399999999999999,"15min":1.0089999999999999},"min":{"1min":0.33000000000000002,"5min":0.253,"15min":0.245},"max":{"1min":0.67500000000000004,"5min":0.79100000000000004,"15min":59.347999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.49399999999999999,"15min":1.288},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":87.221000000000004},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.503,"15min":0.80900000000000005},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.202,"5min":1.202,"15min":25.550999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55000000000000004,"15min":0.871},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66500000000000004,"5min":4.3570000000000002,"15min":40.914000000000001},"last":0.754}]},{"osd":3,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.54200000000000004,"15min":0.80600000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.23400000000000001},"max":{"1min":0.71799999999999997,"5min":3.2799999999999998,"15min":31.309999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.53900000000000003,"15min":1.325},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.29899999999999999},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":86.311999999999998},"last":0.73799999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.42099999999999999,"15min":0.42699999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.626,"5min":0.67000000000000004,"15min":3.7839999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.44500000000000001,"15min":0.503},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57899999999999996,"5min":0.76500000000000001,"15min":17.718},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.441,"15min":0.47699999999999998},"min":{"1min":0.27800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.54600000000000004,"5min":0.71299999999999997,"15min":8.0990000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46899999999999997,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.60899999999999999,"5min":0.75700000000000001,"15min":8.1370000000000005},"last":0.59799999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:37:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.48799999999999999,"15min":0.59499999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.50900000000000001,"5min":0.75,"15min":17.881},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.5,"15min":0.58999999999999997},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.55400000000000005,"5min":0.80800000000000005,"15min":17.396999999999998},"last":0.53400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319280,"kb_used_data":9768,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93425872,"statfs":{"total":95995035648,"available":95668092928,"internally_reserved":0,"allocated":10002432,"data_stored":7157445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.29,"15min":0.66300000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.51700000000000002,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.89600000000000002,"15min":0.55200000000000005},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60499999999999998,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:37:32 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.73999999999999999,"15min":0.67000000000000004},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.45300000000000001,"5min":35.313000000000002,"15min":35.313000000000002},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.30399999999999999,"5min":0.68500000000000005,"15min":0.57499999999999996},"min":{"1min":0.222,"5min":0.222,"15min":0.20100000000000001},"max":{"1min":0.47699999999999998,"5min":27.908999999999999,"15min":27.908999999999999},"last":0.20799999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:38:00 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.35599999999999998,"15min":0.63900000000000001},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":0.46100000000000002,"5min":2.3919999999999999,"15min":21.013999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.318,"5min":0.45700000000000002,"15min":0.64100000000000001},"min":{"1min":0.247,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.42899999999999999,"5min":10.382999999999999,"15min":29.234999999999999},"last":0.25900000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:38:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.61099999999999999,"15min":0.69499999999999995},"min":{"1min":0.35799999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71899999999999997,"5min":8.5559999999999992,"15min":16.231000000000002},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.72099999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.74099999999999999,"5min":1.135,"15min":29.212},"last":0.755}]},{"osd":5,"last update":"Fri Feb 23 00:37:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.78700000000000003,"15min":0.71999999999999997},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.75800000000000001,"5min":16.294,"15min":16.294},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.61599999999999999,"15min":0.67900000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80300000000000005,"5min":8.4030000000000005,"15min":11.105},"last":0.96499999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:38:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.63100000000000001,"15min":0.76100000000000001},"min":{"1min":0.34899999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70999999999999996,"5min":10.416,"15min":29.187000000000001},"last":0.435},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63300000000000001,"15min":0.746},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67200000000000004,"5min":8.2929999999999993,"15min":16.369},"last":0.82999999999999996}]},{"osd":7,"last update":"Fri Feb 23 00:37:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.92200000000000004,"15min":0.76700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73899999999999999,"5min":16.395,"15min":16.395},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.54900000000000004,"15min":0.66200000000000003},"min":{"1min":0.36599999999999999,"5min":0.36299999999999999,"15min":0.35999999999999999},"max":{"1min":0.82399999999999995,"5min":0.89500000000000002,"15min":11.135999999999999},"last":0.93000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318432,"kb_used_data":4824,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426720,"statfs":{"total":95995035648,"available":95668961280,"internally_reserved":0,"allocated":4939776,"data_stored":2599387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.80600000000000005,"15min":0.495},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47799999999999998,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.246},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.165,"15min":0.61899999999999999},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.997,"15min":0.58899999999999997},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.64400000000000002,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.748,"15min":0.48999999999999999},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.22700000000000001},"max":{"1min":0.51100000000000001,"5min":15.965,"15min":15.965},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":1.0820000000000001,"15min":0.65000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.0369999999999999,"15min":0.63700000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.58799999999999997,"5min":16.006,"15min":16.006},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.73399999999999999,"15min":0.63800000000000001},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84699999999999998,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.38300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83699999999999997,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.071,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86799999999999999,"5min":13.311,"15min":13.311},"last":0.499},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.113,"15min":0.74399999999999999},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":23.744,"15min":23.744},"last":0.42299999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.0740000000000001,"15min":0.80600000000000005},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78500000000000003,"5min":13.334,"15min":33.082999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.93400000000000005,"15min":0.70999999999999996},"min":{"1min":0.41199999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81699999999999995,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.49199999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.84099999999999997,"15min":0.78500000000000003},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.871,"5min":15.943,"15min":31.251999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.091,"15min":0.76100000000000001},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.80000000000000004,"5min":23.785,"15min":23.785},"last":0.47599999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411489,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310876,"kb_used_data":9556,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93434276,"statfs":{"total":95995035648,"available":95676698624,"internally_reserved":0,"allocated":9785344,"data_stored":7154318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":1.0600000000000001,"15min":0.754},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.65500000000000003,"5min":56.509,"15min":56.509},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.59799999999999998,"15min":0.71199999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56799999999999995,"5min":19.571999999999999,"15min":31.498999999999999},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:37:35 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.46999999999999997,"15min":0.59499999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":7.7110000000000003,"5min":7.7110000000000003,"15min":19.324999999999999},"last":0.318},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.70999999999999996},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.0249999999999999,"5min":13.209,"15min":35.052},"last":0.28299999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:38:05 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.55700000000000005,"15min":0.73899999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":10.032,"5min":10.032,"15min":43.5},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.66900000000000004,"15min":0.69099999999999995},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":11.815,"5min":11.815,"15min":27.414000000000001},"last":0.37}]},{"osd":4,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.67600000000000005},"min":{"1min":0.32700000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0899999999999999,"5min":10.106,"15min":14.614000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.67900000000000005,"15min":0.627},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.9750000000000001,"5min":7.766,"15min":7.766},"last":0.40200000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:37:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57899999999999996,"15min":0.66800000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":2.004,"5min":3.2879999999999998,"15min":17.765000000000001},"last":0.503},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.34499999999999997,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":10.138,"5min":10.138,"15min":14.589},"last":0.45500000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:38:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.67000000000000004,"15min":0.628},"min":{"1min":0.36299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.2250000000000001,"5min":7.7869999999999999,"15min":7.7869999999999999},"last":0.434},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.62,"15min":0.751},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.34899999999999998},"max":{"1min":1.927,"5min":3.3410000000000002,"15min":20.574999999999999},"last":0.46999999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:37:28 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.63,"15min":1.036},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":1.9590000000000001,"5min":3.3780000000000001,"15min":66.632000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.349,"5min":0.748,"15min":0.85199999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":10.163,"5min":10.163,"15min":19.603999999999999},"last":0.42099999999999999}]}]},{"osd":3,"up_from":0,"seq":111669149853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986328,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315276,"kb_used_data":5764,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93429876,"statfs":{"total":95995035648,"available":95672193024,"internally_reserved":0,"allocated":5902336,"data_stored":3154587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1739999999999999,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.68799999999999994,"5min":21.419,"15min":21.419},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":2.1200000000000001,"15min":1.2589999999999999},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:38:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.75900000000000001,"15min":1.069},"min":{"1min":0.29199999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.1199999999999992,"5min":16.696000000000002,"15min":41.137999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.69199999999999995,"15min":0.88400000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.89100000000000001,"5min":13.455,"15min":26.210000000000001},"last":0.39400000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:38:11 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.81399999999999995,"15min":1.1519999999999999},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.9859999999999998,"5min":13.478,"15min":67.253},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.67800000000000005,"15min":1.0169999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.65300000000000002,"5min":8.0199999999999996,"15min":41.927999999999997},"last":0.34699999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:38:11 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.65800000000000003,"15min":1.0669999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34000000000000002},"max":{"1min":0.67300000000000004,"5min":7.774,"15min":36.301000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":1.1819999999999999},"min":{"1min":0.33100000000000002,"5min":0.307,"15min":0.24299999999999999},"max":{"1min":3.9489999999999998,"5min":16.657,"15min":42.064},"last":0.36299999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:37:40 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.219,"5min":0.219,"15min":0.217},"max":{"1min":0.65100000000000002,"5min":8.5299999999999994,"15min":14.242000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.69399999999999995,"15min":0.59599999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.57799999999999996,"5min":8.3010000000000002,"15min":14.124000000000001},"last":0.29599999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:38:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.64800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":0.59499999999999997,"5min":8.0730000000000004,"15min":8.0730000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.52500000000000002,"15min":0.63500000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":0.63900000000000001,"5min":4.71,"15min":45.814999999999998},"last":0.374}]},{"osd":7,"last update":"Fri Feb 23 00:37:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.29999999999999999},"max":{"1min":5.1280000000000001,"5min":6.165,"15min":17.178999999999998},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.288,"5min":0.81399999999999995,"15min":0.69799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28299999999999997},"max":{"1min":15.85,"5min":15.85,"15min":15.85},"last":0.40500000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757395,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307096,"kb_used_data":5776,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438056,"statfs":{"total":95995035648,"available":95680569344,"internally_reserved":0,"allocated":5914624,"data_stored":3146431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.1200000000000001,"15min":0.70699999999999996},"min":{"1min":0.36699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.115,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":1.028},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.498,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.51600000000000001,"15min":0.76100000000000001},"min":{"1min":0.26200000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.7629999999999999,"5min":3.7629999999999999,"15min":38.838000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.57499999999999996,"15min":1.196},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.7250000000000001,"5min":3.7250000000000001,"15min":127.794},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.56899999999999995,"15min":0.90700000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":37.442999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.70899999999999996},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.734,"5min":3.734,"15min":23.297999999999998},"last":0.39800000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:37:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":0.64700000000000002,"15min":0.78500000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":12.507,"5min":12.507,"15min":24.414999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.873,"5min":0.61599999999999999,"15min":0.91900000000000004},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.31},"max":{"1min":3.8119999999999998,"5min":3.8119999999999998,"15min":40.865000000000002},"last":0.37}]},{"osd":4,"last update":"Fri Feb 23 00:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.438,"15min":0.78300000000000003},"min":{"1min":0.255,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":2.8940000000000001,"5min":2.8940000000000001,"15min":103.13},"last":0.312},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.47699999999999998,"15min":0.47099999999999997},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.9420000000000002,"5min":2.9420000000000002,"15min":12.795999999999999},"last":0.35399999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:38:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.29499999999999998,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":1.6140000000000001,"5min":2.9199999999999999,"15min":7.1440000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47999999999999998,"15min":0.82399999999999995},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":103.61499999999999},"last":0.435}]},{"osd":7,"last update":"Fri Feb 23 00:37:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.55300000000000005,"15min":0.53400000000000003},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.9039999999999999,"5min":3.9039999999999999,"15min":7.1820000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.58199999999999996,"15min":0.53200000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.4050000000000002,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4390912,"data_stored":4000917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4243456,"data_stored":4000594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4000512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":221,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":221,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":221,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":221,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":221,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":221,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:38:28.326 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-23T00:38:28.326 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.meta pg_num 22 2024-02-23T00:38:28.376 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:28.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:26.801618+0000 mgr.y (mgr.24437) 466 : audit [DBG] from='client.27402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:38:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:27.176758+0000 mon.a (mon.0) 2794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:38:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:27.176837+0000 mon.a (mon.0) 2795 : cluster [DBG] osdmap e490: 8 total, 7 up, 7 in 2024-02-23T00:38:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:27.179055+0000 mon.b (mon.1) 403 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:27.179500+0000 mon.a (mon.0) 2796 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:28.479 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:28.506 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:29.355 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:29.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:29.388 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:27.702022+0000 mgr.y (mgr.24437) 467 : cluster [DBG] pgmap v751: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 68 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 88 KiB/s rd, 7.7 KiB/s wr, 175 op/s; 120/1194 objects degraded (10.050%); 84/1194 objects misplaced (7.035%); 343 B/s, 21 objects/s recovering 2024-02-23T00:38:29.388 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:28.178876+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:29.389 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:28.178962+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e491: 8 total, 7 up, 7 in 2024-02-23T00:38:29.389 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:28.180273+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:29.389 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:28.187380+0000 mon.a (mon.0) 2799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:29.462 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:29.484 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.460+0000 7f114e742700 1 -- 172.21.15.87:0/4024164189 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11480fdbb0 msgr2=0x7f11480fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.460+0000 7f114e742700 1 --2- 172.21.15.87:0/4024164189 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11480fdbb0 0x7f11480fdf90 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 rx=0x7f11380047b0 tx=0x7f1138009cb0).stop 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 -- 172.21.15.87:0/4024164189 shutdown_connections 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 --2- 172.21.15.87:0/4024164189 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11480f8420 0x7f1148104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 --2- 172.21.15.87:0/4024164189 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11480fdbb0 0x7f11480fdf90 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 --2- 172.21.15.87:0/4024164189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11480fe510 0x7f11480f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 -- 172.21.15.87:0/4024164189 >> 172.21.15.87:0/4024164189 conn(0x7f11480f3800 msgr2=0x7f11480f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 -- 172.21.15.87:0/4024164189 shutdown_connections 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 -- 172.21.15.87:0/4024164189 wait complete. 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 librados: starting msgr at 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 librados: starting objecter 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 Processor -- start 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 -- start start 2024-02-23T00:38:29.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 librados: setting wanted keys 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.461+0000 7f114e742700 1 librados: calling monclient init 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11480f8420 0x7f114806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11480fdbb0 0x7f114806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11480fe510 0x7f114806bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11481164e0 con 0x7f11480fe510 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1148116650 con 0x7f11480f8420 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1148116810 con 0x7f11480fdbb0 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11477fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11480fdbb0 0x7f114806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114ccdf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11480fe510 0x7f114806bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11477fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11480fdbb0 0x7f114806b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:36812/0 (socket says 172.21.15.87:36812) 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11477fe700 1 -- 172.21.15.87:0/2160524890 learned_addr learned my addr 172.21.15.87:0/2160524890 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f1147fff700 1 --2- 172.21.15.87:0/2160524890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11480f8420 0x7f114806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11477fe700 1 -- 172.21.15.87:0/2160524890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11480f8420 msgr2=0x7f114806b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:29.486 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11477fe700 1 --2- 172.21.15.87:0/2160524890 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11480f8420 0x7f114806b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11477fe700 1 -- 172.21.15.87:0/2160524890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11480fe510 msgr2=0x7f114806bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11477fe700 1 --2- 172.21.15.87:0/2160524890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11480fe510 0x7f114806bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11477fe700 1 -- 172.21.15.87:0/2160524890 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11400097c0 con 0x7f11480fdbb0 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114ccdf700 1 --2- 172.21.15.87:0/2160524890 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11480fe510 0x7f114806bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11477fe700 1 --2- 172.21.15.87:0/2160524890 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11480fdbb0 0x7f114806b5c0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 rx=0x7f113c0041d0 tx=0x7f113c009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11457fa700 10 client.?.objecter ms_handle_connect 0x7f11480fdbb0 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11457fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c007b40 con 0x7f11480fdbb0 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1138005040 con 0x7f11480fdbb0 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f113c00f040 con 0x7f11480fdbb0 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113c014730 con 0x7f11480fdbb0 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 10 client.37874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1148116f20 con 0x7f11480fdbb0 2024-02-23T00:38:29.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.462+0000 7f114e742700 1 librados: init done 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.463+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1148107110 con 0x7f11480fdbb0 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.464+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f113c014b60 con 0x7f11480fdbb0 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.464+0000 7f11457fa700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1130062c90 0x7f1130065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.464+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(492..492 src has 173..492) v4 ==== 8473+0+0 (secure 0 0 0) 0x7f113c07d7d0 con 0x7f11480fdbb0 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.464+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 osd_map(492..492 src has 173..492) v4 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.464+0000 7f11457fa700 3 client.37874.objecter handle_osd_map got epochs [492,492] > 0 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.464+0000 7f11457fa700 3 client.37874.objecter handle_osd_map decoding full epoch 492 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.464+0000 7f1147fff700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1130062c90 0x7f1130065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.464+0000 7f11457fa700 20 client.37874.objecter dump_active .. 0 homeless 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.464+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v492) v1 ==== 54+0+16 (secure 0 0 0) 0x7f113c07e0a0 con 0x7f11480fdbb0 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 10 client.37874.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 1 2024-02-23T00:38:29.488 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 20 client.37874.objecter _get_session osd=-1 returning homeless 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 10 client.37874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=493}) v3 -- 0x7f114806c5b0 con 0x7f11480fdbb0 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f11457fa700 10 client.37874.objecter ms_handle_connect 0x7f1130062c90 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f1147fff700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1130062c90 0x7f1130065150 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 rx=0x7f1138005010 tx=0x7f1138007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1148117110 con 0x7f11480fdbb0 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=492) v2 ==== 24+0+0 (secure 0 0 0) 0x7f113c050b00 con 0x7f11480fdbb0 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f11467fc700 10 client.37874.objecter _check_command_map_dne tid 1 current 492 map_dne_bound 492 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f11467fc700 10 client.37874.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f11467fc700 15 client.37874.objecter _session_command_op_remove -1 1 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 10 client.37874.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 2 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6805/685354171 conn(0x7f11480f5d70 0x7f11480273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11480f5c00 osd=1 3 2024-02-23T00:38:29.489 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11480f5c00 osd=1 3 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11480f5c00 osd=1 3 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 2 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f11480f5c00 osd=1 3 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 1 2 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11480f5c00 osd=1 4 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 10 client.37874.objecter _send_command 2 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1148059e00 con 0x7f11480f5d70 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.465+0000 7f114ccdf700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6805/685354171 conn(0x7f11480f5d70 0x7f11480273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.466+0000 7f11457fa700 10 client.37874.objecter ms_handle_connect 0x7f11480f5d70 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.466+0000 7f114ccdf700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6805/685354171 conn(0x7f11480f5d70 0x7f11480273c0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.466+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1148059e00 con 0x7f11480f5d70 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.466+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.466+0000 7f11457fa700 10 client.37874.objecter _finish_command 2 = system:0 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.466+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f11480f5c00 osd=1 4 2024-02-23T00:38:29.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.466+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 1 2 2024-02-23T00:38:29.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 10 client.37874.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:29.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 3 2024-02-23T00:38:29.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11480f5c00 osd=1 3 2024-02-23T00:38:29.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11480f5c00 osd=1 3 2024-02-23T00:38:29.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11480f5c00 osd=1 3 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 3 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f11480f5c00 osd=1 3 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 1 3 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11480f5c00 osd=1 4 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 10 client.37874.objecter _send_command 3 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.484+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11480278b0 con 0x7f11480f5d70 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.486+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11480278b0 con 0x7f11480f5d70 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.486+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.486+0000 7f11457fa700 10 client.37874.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.486+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f11480f5c00 osd=1 4 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.486+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 1 3 2024-02-23T00:38:29.492 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:38:29.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 10 client.37874.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:29.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 4 2024-02-23T00:38:29.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6809/3605394228 conn(0x7f11480594d0 0x7f114811b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.493 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f1148059360 osd=2 3 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f1148059360 osd=2 3 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f1148059360 osd=2 3 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 4 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f1148059360 osd=2 3 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 2 4 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f1148059360 osd=2 4 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 10 client.37874.objecter _send_command 4 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1148028760 con 0x7f11480594d0 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f1147fff700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6809/3605394228 conn(0x7f11480594d0 0x7f114811b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f11457fa700 10 client.37874.objecter ms_handle_connect 0x7f11480594d0 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.487+0000 7f1147fff700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6809/3605394228 conn(0x7f11480594d0 0x7f114811b2a0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.488+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1148028760 con 0x7f11480594d0 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.488+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:38:29.494 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.488+0000 7f11457fa700 10 client.37874.objecter _finish_command 4 = system:0 2024-02-23T00:38:29.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.488+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f1148059360 osd=2 4 2024-02-23T00:38:29.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.488+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 2 4 2024-02-23T00:38:29.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:27.702022+0000 mgr.y (mgr.24437) 467 : cluster [DBG] pgmap v751: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 68 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 88 KiB/s rd, 7.7 KiB/s wr, 175 op/s; 120/1194 objects degraded (10.050%); 84/1194 objects misplaced (7.035%); 343 B/s, 21 objects/s recovering 2024-02-23T00:38:29.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:28.178876+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:29.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:28.178962+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e491: 8 total, 7 up, 7 in 2024-02-23T00:38:29.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:28.180273+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:29.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:28.187380+0000 mon.a (mon.0) 2799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:29.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:27.702022+0000 mgr.y (mgr.24437) 467 : cluster [DBG] pgmap v751: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 68 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 88 KiB/s rd, 7.7 KiB/s wr, 175 op/s; 120/1194 objects degraded (10.050%); 84/1194 objects misplaced (7.035%); 343 B/s, 21 objects/s recovering 2024-02-23T00:38:29.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:28.178876+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:29.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:28.178962+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e491: 8 total, 7 up, 7 in 2024-02-23T00:38:29.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:28.180273+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:29.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:28.187380+0000 mon.a (mon.0) 2799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:29.496 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:29.496 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:29.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 10 client.37874.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:29.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 5 2024-02-23T00:38:29.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f1148059360 osd=2 3 2024-02-23T00:38:29.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f1148059360 osd=2 3 2024-02-23T00:38:29.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f1148059360 osd=2 3 2024-02-23T00:38:29.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 5 2024-02-23T00:38:29.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f1148059360 osd=2 3 2024-02-23T00:38:29.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 2 5 2024-02-23T00:38:29.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f1148059360 osd=2 4 2024-02-23T00:38:29.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 10 client.37874.objecter _send_command 5 2024-02-23T00:38:29.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.510+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1148027cc0 con 0x7f11480594d0 2024-02-23T00:38:29.514 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:29.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1148027cc0 con 0x7f11480594d0 2024-02-23T00:38:29.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:29.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f11457fa700 10 client.37874.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f1148059360 osd=2 4 2024-02-23T00:38:29.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 2 5 2024-02-23T00:38:29.515 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 10 client.37874.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 6 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6813/128059790 conn(0x7f11480ff650 0x7f114811e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11480280d0 osd=3 3 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11480280d0 osd=3 3 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11480280d0 osd=3 3 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 6 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f11480280d0 osd=3 3 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 3 6 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11480280d0 osd=3 4 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 10 client.37874.objecter _send_command 6 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.512+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1148121630 con 0x7f11480ff650 2024-02-23T00:38:29.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.513+0000 7f11477fe700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6813/128059790 conn(0x7f11480ff650 0x7f114811e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.513+0000 7f11457fa700 10 client.37874.objecter ms_handle_connect 0x7f11480ff650 2024-02-23T00:38:29.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.513+0000 7f11477fe700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.87:6813/128059790 conn(0x7f11480ff650 0x7f114811e920 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:29.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.514+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1148121630 con 0x7f11480ff650 2024-02-23T00:38:29.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.514+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:38:29.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.514+0000 7f11457fa700 10 client.37874.objecter _finish_command 6 = system:0 2024-02-23T00:38:29.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.514+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f11480280d0 osd=3 4 2024-02-23T00:38:29.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.514+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 3 6 2024-02-23T00:38:29.518 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:29.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:29.538 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 10 client.37874.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 7 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11480280d0 osd=3 3 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11480280d0 osd=3 3 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11480280d0 osd=3 3 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 7 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f11480280d0 osd=3 3 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 3 7 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11480280d0 osd=3 4 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 10 client.37874.objecter _send_command 7 2024-02-23T00:38:29.539 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.536+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11480fec70 con 0x7f11480ff650 2024-02-23T00:38:29.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11480fec70 con 0x7f11480ff650 2024-02-23T00:38:29.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:29.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f11457fa700 10 client.37874.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f11480280d0 osd=3 4 2024-02-23T00:38:29.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 3 7 2024-02-23T00:38:29.540 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:38:29.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 10 client.37874.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:29.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 8 2024-02-23T00:38:29.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6800/4175844757 conn(0x7f114811ff90 0x7f11481203b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f114811bf60 osd=4 3 2024-02-23T00:38:29.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f114811bf60 osd=4 3 2024-02-23T00:38:29.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f114811bf60 osd=4 3 2024-02-23T00:38:29.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 8 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f114811bf60 osd=4 3 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 4 8 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f114811bf60 osd=4 4 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 10 client.37874.objecter _send_command 8 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f114811c710 con 0x7f114811ff90 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.538+0000 7f114ccdf700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6800/4175844757 conn(0x7f114811ff90 0x7f11481203b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.539+0000 7f11457fa700 10 client.37874.objecter ms_handle_connect 0x7f114811ff90 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.539+0000 7f114ccdf700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6800/4175844757 conn(0x7f114811ff90 0x7f11481203b0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.539+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f114811c710 con 0x7f114811ff90 2024-02-23T00:38:29.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.539+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:38:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.539+0000 7f11457fa700 10 client.37874.objecter _finish_command 8 = system:0 2024-02-23T00:38:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.539+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f114811bf60 osd=4 4 2024-02-23T00:38:29.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.539+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 4 8 2024-02-23T00:38:29.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 10 client.37874.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:29.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 9 2024-02-23T00:38:29.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f114811bf60 osd=4 3 2024-02-23T00:38:29.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f114811bf60 osd=4 3 2024-02-23T00:38:29.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f114811bf60 osd=4 3 2024-02-23T00:38:29.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 9 2024-02-23T00:38:29.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f114811bf60 osd=4 3 2024-02-23T00:38:29.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 4 9 2024-02-23T00:38:29.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f114811bf60 osd=4 4 2024-02-23T00:38:29.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 10 client.37874.objecter _send_command 9 2024-02-23T00:38:29.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.554+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11481210a0 con 0x7f114811ff90 2024-02-23T00:38:29.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11481210a0 con 0x7f114811ff90 2024-02-23T00:38:29.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:29.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f11457fa700 10 client.37874.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f114811bf60 osd=4 4 2024-02-23T00:38:29.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 4 9 2024-02-23T00:38:29.558 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:38:29.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 10 client.37874.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:29.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 10 2024-02-23T00:38:29.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1148123860 0x7f1148125c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11481210a0 osd=5 3 2024-02-23T00:38:29.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11481210a0 osd=5 3 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11481210a0 osd=5 3 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 10 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f11481210a0 osd=5 3 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 5 10 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11481210a0 osd=5 4 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 10 client.37874.objecter _send_command 10 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1148121370 con 0x7f1148123860 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f1147fff700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1148123860 0x7f1148125c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f11457fa700 10 client.37874.objecter ms_handle_connect 0x7f1148123860 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.556+0000 7f1147fff700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6804/2550013795 conn(0x7f1148123860 0x7f1148125c50 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.557+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1148121370 con 0x7f1148123860 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.557+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.557+0000 7f11457fa700 10 client.37874.objecter _finish_command 10 = system:0 2024-02-23T00:38:29.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.557+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f11481210a0 osd=5 4 2024-02-23T00:38:29.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.557+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 5 10 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 10 client.37874.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 11 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11481210a0 osd=5 3 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11481210a0 osd=5 3 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11481210a0 osd=5 3 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 11 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f11481210a0 osd=5 3 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 5 11 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11481210a0 osd=5 4 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 10 client.37874.objecter _send_command 11 2024-02-23T00:38:29.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.572+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1148121370 con 0x7f1148123860 2024-02-23T00:38:29.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1148121370 con 0x7f1148123860 2024-02-23T00:38:29.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:29.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f11457fa700 10 client.37874.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f11481210a0 osd=5 4 2024-02-23T00:38:29.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 5 11 2024-02-23T00:38:29.576 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 10 client.37874.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 12 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1148126690 0x7f1148126b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f1148121370 osd=6 3 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f1148121370 osd=6 3 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f1148121370 osd=6 3 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 12 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f1148121370 osd=6 3 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 6 12 2024-02-23T00:38:29.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f1148121370 osd=6 4 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 10 client.37874.objecter _send_command 12 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.573+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f114812ba60 con 0x7f1148126690 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.574+0000 7f11477fe700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1148126690 0x7f1148126b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.574+0000 7f11457fa700 10 client.37874.objecter ms_handle_connect 0x7f1148126690 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.574+0000 7f11477fe700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6808/2509408498 conn(0x7f1148126690 0x7f1148126b10 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.575+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f114812ba60 con 0x7f1148126690 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.575+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.575+0000 7f11457fa700 10 client.37874.objecter _finish_command 12 = system:0 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.575+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f1148121370 osd=6 4 2024-02-23T00:38:29.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.575+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 6 12 2024-02-23T00:38:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 10 client.37874.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 13 2024-02-23T00:38:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f1148121370 osd=6 3 2024-02-23T00:38:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f1148121370 osd=6 3 2024-02-23T00:38:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f1148121370 osd=6 3 2024-02-23T00:38:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 13 2024-02-23T00:38:29.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f1148121370 osd=6 3 2024-02-23T00:38:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 6 13 2024-02-23T00:38:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f1148121370 osd=6 4 2024-02-23T00:38:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 10 client.37874.objecter _send_command 13 2024-02-23T00:38:29.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.588+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11481270c0 con 0x7f1148126690 2024-02-23T00:38:29.592 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:38:29.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11481270c0 con 0x7f1148126690 2024-02-23T00:38:29.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:29.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f11457fa700 10 client.37874.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f1148121370 osd=6 4 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 6 13 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 10 client.37874.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 14 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6812/373787000 conn(0x7f1148127740 0x7f1148127ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11481270c0 osd=7 3 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11481270c0 osd=7 3 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11481270c0 osd=7 3 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 14 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f11481270c0 osd=7 3 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 7 14 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11481270c0 osd=7 4 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 10 client.37874.objecter _send_command 14 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1148128120 con 0x7f1148127740 2024-02-23T00:38:29.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.590+0000 7f114ccdf700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6812/373787000 conn(0x7f1148127740 0x7f1148127ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:29.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.591+0000 7f11457fa700 10 client.37874.objecter ms_handle_connect 0x7f1148127740 2024-02-23T00:38:29.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.591+0000 7f114ccdf700 1 --2- 172.21.15.87:0/2160524890 >> v2:172.21.15.171:6812/373787000 conn(0x7f1148127740 0x7f1148127ba0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:29.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.591+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f1148128120 con 0x7f1148127740 2024-02-23T00:38:29.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.591+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:38:29.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.591+0000 7f11457fa700 10 client.37874.objecter _finish_command 14 = system:0 2024-02-23T00:38:29.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.591+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f11481270c0 osd=7 4 2024-02-23T00:38:29.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.591+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 7 14 2024-02-23T00:38:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 10 client.37874.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign -1 15 2024-02-23T00:38:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11481270c0 osd=7 3 2024-02-23T00:38:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11481270c0 osd=7 3 2024-02-23T00:38:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 20 client.37874.objecter _get_session s=0x7f11481270c0 osd=7 3 2024-02-23T00:38:29.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 15 client.37874.objecter _session_command_op_remove -1 15 2024-02-23T00:38:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 20 client.37874.objecter get_session s=0x7f11481270c0 osd=7 3 2024-02-23T00:38:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 15 client.37874.objecter _session_command_op_assign 7 15 2024-02-23T00:38:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 20 client.37874.objecter put_session s=0x7f11481270c0 osd=7 4 2024-02-23T00:38:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 10 client.37874.objecter _send_command 15 2024-02-23T00:38:29.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.605+0000 7f114e742700 1 -- 172.21.15.87:0/2160524890 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1148128120 con 0x7f1148127740 2024-02-23T00:38:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.607+0000 7f11457fa700 1 -- 172.21.15.87:0/2160524890 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f1148128120 con 0x7f1148127740 2024-02-23T00:38:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.607+0000 7f11457fa700 10 client.37874.objecter ms_dispatch 0x7f1148001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.607+0000 7f11457fa700 10 client.37874.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.607+0000 7f11457fa700 20 client.37874.objecter put_session s=0x7f11481270c0 osd=7 4 2024-02-23T00:38:29.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:29.607+0000 7f11457fa700 15 client.37874.objecter _session_command_op_remove 7 15 2024-02-23T00:38:29.609 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:38:29.610 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:38:29.610 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.610 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.610 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.610 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.610 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.610 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.610 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:38:29.625 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:29.654 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:29.726 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:29.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:29.831 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:29.859 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:29.912 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:29.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:30.016 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:30.034 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:38:30.034 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:38:30.034 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:38:30.034 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:38:30.034 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.034 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25166.0:2 59.6 59:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:02.557319+0000", 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 447.47317199000003, 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056150042999999997, 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25166", 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/458030738", 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557319+0000", 2024-02-23T00:38:30.035 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557319+0000", 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557318+0000", 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8050000000000003e-06 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.036 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557324+0000", 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8560000000000007e-06 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557331+0000", 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4490000000000002e-06 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557339+0000", 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7904000000000002e-05 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.037 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557377+0000", 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127668 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557504+0000", 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000131311 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,6", 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557636+0000", 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033497099999999999 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.038 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557971+0000", 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000210115 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.558181+0000", 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055228405000000001 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.613409+0000", 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4125000000000003e-05 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.039 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.040 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.613443+0000", 2024-02-23T00:38:30.040 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5678999999999999e-05 2024-02-23T00:38:30.040 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.040 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.043 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.043 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.613469+0000", 2024-02-23T00:38:30.043 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056150042999999997 2024-02-23T00:38:30.043 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:8 56.2 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.569023+0000", 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 446.46146750299999, 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.061982072999999999, 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.044 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 8 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569023+0000", 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569023+0000", 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.045 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569022+0000", 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9890000000000003e-06 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569029+0000", 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5150000000000006e-06 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569037+0000", 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0450000000000003e-06 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.046 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569045+0000", 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033288000000000002 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569377+0000", 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000205007 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569582+0000", 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070529399999999997 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.576635+0000", 2024-02-23T00:38:30.047 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8623999999999996e-05 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.576734+0000", 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000176076 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.576910+0000", 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00079704099999999996 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.577707+0000", 2024-02-23T00:38:30.048 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9565e-05 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.577807+0000", 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053133055999999998 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.630940+0000", 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2215999999999998e-05 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.630972+0000", 2024-02-23T00:38:30.049 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3599000000000001e-05 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631006+0000", 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.061982072999999999 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:9 56.2 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head [watch watch cookie 93958402142112] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.631729+0000", 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 446.39876169899998, 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018840933000000001, 2024-02-23T00:38:30.050 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 9 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631729+0000", 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.051 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631729+0000", 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631728+0000", 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8719999999999999e-06 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631734+0000", 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1409999999999994e-06 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.052 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631743+0000", 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6979999999999997e-06 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631748+0000", 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5034999999999999e-05 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631783+0000", 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5764000000000002e-05 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.053 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631858+0000", 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104113 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631962+0000", 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00027199 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.632234+0000", 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017308100000000001 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.054 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.632408+0000", 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018079319999999999 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.650487+0000", 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0031000000000001e-05 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.650517+0000", 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3294000000000002e-05 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.055 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.650570+0000", 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018840933000000001 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:14 56.1 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.661664+0000", 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 446.36882714400002, 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044383775, 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.056 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 14 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661664+0000", 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661664+0000", 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.057 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661663+0000", 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3850000000000001e-06 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661668+0000", 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5259999999999999e-06 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661678+0000", 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7389999999999999e-06 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.058 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661683+0000", 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7559000000000002e-05 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661710+0000", 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000168534 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661879+0000", 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014226162000000001 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.059 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676105+0000", 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4277000000000006e-05 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676189+0000", 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100592 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676290+0000", 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00034506400000000002 2024-02-23T00:38:30.060 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676635+0000", 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000797236 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.677432+0000", 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028506070000000001 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.705938+0000", 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3522000000000006e-05 2024-02-23T00:38:30.061 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.706002+0000", 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5997e-05 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.706048+0000", 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044383775 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.062 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:16 56.1 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head [watch unwatch cookie 93958402138000] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.714409+0000", 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 446.316082112, 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021332225999999999, 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 16 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.063 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714409+0000", 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714409+0000", 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714407+0000", 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.015e-06 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.064 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714413+0000", 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7249999999999997e-06 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714421+0000", 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1869999999999999e-06 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714425+0000", 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2351000000000002e-05 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.065 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714468+0000", 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012891899999999999 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714597+0000", 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00019076400000000001 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714787+0000", 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041226000000000001 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.066 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.715200+0000", 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001342099 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.716542+0000", 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019100005 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.735642+0000", 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1798999999999997e-05 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.067 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.735683+0000", 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7598e-05 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.735741+0000", 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021332225999999999 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.068 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:37 56.1 56:972518f1:test-rados-api-smithi087-82147-12::foo:head [watch unwatch cookie 93958402138000] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 446.22728053999998, 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019757686999999999, 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 37 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:38:30.069 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2400000000000001e-06 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803214+0000", 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9640000000000003e-06 2024-02-23T00:38:30.070 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803222+0000", 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7230000000000002e-06 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803226+0000", 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0100999999999999e-05 2024-02-23T00:38:30.071 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803256+0000", 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8650999999999998e-05 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803314+0000", 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0869e-05 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803405+0000", 2024-02-23T00:38:30.072 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00032610700000000001 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803731+0000", 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012946180999999999 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.816678+0000", 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.006224963 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.822902+0000", 2024-02-23T00:38:30.073 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8585000000000002e-05 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.822931+0000", 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7018999999999999e-05 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.822968+0000", 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019757686999999999 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.074 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:56 56.2 56:6c9b5892:test-rados-api-smithi087-82147-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.877562+0000", 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 446.15292944800001, 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015386004999999999, 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 56 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.075 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877562+0000", 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877562+0000", 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877560+0000", 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0372e-05 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.076 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877571+0000", 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1782e-05 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877582+0000", 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0769999999999996e-06 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877588+0000", 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4140999999999997e-05 2024-02-23T00:38:30.077 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877623+0000", 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016143599999999999 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.877784+0000", 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011506860000000001 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.889291+0000", 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2185000000000003e-05 2024-02-23T00:38:30.078 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.889343+0000", 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2941000000000003e-05 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.889436+0000", 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00036122900000000002 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.889797+0000", 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000231883 2024-02-23T00:38:30.079 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.890029+0000", 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028250570000000002 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.892854+0000", 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6941000000000003e-05 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.892911+0000", 2024-02-23T00:38:30.080 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6390999999999998e-05 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.892948+0000", 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015386004999999999 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:67 56.0 56:2858fd9f:test-rados-api-smithi087-82147-17::foo:head [watch unwatch cookie 93958402367200] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:07.920664+0000", 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 442.109826742, 2024-02-23T00:38:30.081 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073703837999999994, 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 67 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920664+0000", 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.082 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920664+0000", 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920663+0000", 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8689999999999998e-06 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920669+0000", 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1250000000000001e-05 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.083 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920681+0000", 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8099999999999998e-06 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920688+0000", 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9345000000000003e-05 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920738+0000", 2024-02-23T00:38:30.084 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109922 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920848+0000", 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016524700000000001 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,6", 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.921013+0000", 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00089610999999999998 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.921909+0000", 2024-02-23T00:38:30.085 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125143 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.922034+0000", 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072257924000000001 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.994292+0000", 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4425000000000003e-05 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.994327+0000", 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1483000000000002e-05 2024-02-23T00:38:30.086 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.994368+0000", 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073703837999999994 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:69 56.1 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e118)", 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:09.916487+0000", 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 440.11400406600001, 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062093785999999998, 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.087 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 69 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916487+0000", 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916487+0000", 2024-02-23T00:38:30.088 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916486+0000", 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7636000000000001e-05 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916514+0000", 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0845999999999999e-05 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916524+0000", 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1185000000000001e-05 2024-02-23T00:38:30.089 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916546+0000", 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3442999999999998e-05 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916579+0000", 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000177116 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916756+0000", 2024-02-23T00:38:30.090 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077845400000000004 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.917535+0000", 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8349999999999994e-05 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.917603+0000", 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108934 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.917712+0000", 2024-02-23T00:38:30.091 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00030745799999999998 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.918019+0000", 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032894684 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.950914+0000", 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027609002000000001 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.978523+0000", 2024-02-23T00:38:30.092 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2388999999999998e-05 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.978555+0000", 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5239e-05 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.978581+0000", 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062093785999999998 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.093 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:75 56.1 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head [watch unwatch cookie 93958402367200] snapc 0=[] ondisk+write+known_if_redirected e122)", 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:12.996174+0000", 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 437.03431695400002, 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.084863057000000006, 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 75 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.094 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996174+0000", 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996174+0000", 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996173+0000", 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1340000000000003e-06 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.095 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996178+0000", 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0209e-05 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996188+0000", 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9609999999999996e-06 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996193+0000", 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0006999999999999e-05 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.096 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996223+0000", 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0087000000000007e-05 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996313+0000", 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000276903 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996590+0000", 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00053075400000000003 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.097 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.997121+0000", 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070072748000000004 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.067194+0000", 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01373247 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.080926+0000", 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.193e-05 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.098 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.080968+0000", 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8838999999999996e-05 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081037+0000", 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.084863057000000006 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:76 56.1 56:a5875266:test-rados-api-smithi087-82147-19::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e122)", 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:13.081838+0000", 2024-02-23T00:38:30.099 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 436.94865265200002, 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028322508, 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 76 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081838+0000", 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.100 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081838+0000", 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081837+0000", 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2859999999999996e-06 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081843+0000", 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0053e-05 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.101 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081853+0000", 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1229999999999999e-06 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081858+0000", 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0298999999999999e-05 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081889+0000", 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015674600000000001 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.102 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.082046+0000", 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011441438 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093487+0000", 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6672000000000002e-05 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093544+0000", 2024-02-23T00:38:30.103 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108148 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093652+0000", 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00028719800000000001 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093939+0000", 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051637150000000001 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.099103+0000", 2024-02-23T00:38:30.104 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011005768000000001 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.110108+0000", 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0571999999999998e-05 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.110139+0000", 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1756999999999999e-05 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.110161+0000", 2024-02-23T00:38:30.105 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028322508 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24413.0:238 3.c 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e137)", 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:28.139198+0000", 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 421.89129258600002, 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94992057799999996, 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24413", 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.171:0/2453883515", 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 238 2024-02-23T00:38:30.106 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139198+0000", 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139198+0000", 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139197+0000", 2024-02-23T00:38:30.107 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0800000000000002e-06 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139203+0000", 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1200000000000008e-06 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139212+0000", 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.344e-06 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139217+0000", 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9641000000000002e-05 2024-02-23T00:38:30.108 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139257+0000", 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1140000000000002e-06 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139263+0000", 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.93888930100000001 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.078152+0000", 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013934400000000001 2024-02-23T00:38:30.109 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for missing object", 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.078292+0000", 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010582111999999999 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.088874+0000", 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7443999999999997e-05 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.088971+0000", 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000147868 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.110 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.089119+0000", 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94992057799999996 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:578 63.a 63:54d171db:test-rados-api-smithi087-82091-14::foo7:head [writefull 0~11] snapc 5=[5,4] ondisk+write+known_if_redirected e146)", 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 413.00791455199999, 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044787916999999997, 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.111 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 578 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.112 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3959999999999997e-06 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022581+0000", 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8709999999999997e-06 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022589+0000", 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2200000000000003e-06 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.113 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022593+0000", 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4632000000000002e-05 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022638+0000", 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012398899999999999 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022762+0000", 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00023933200000000001 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.114 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,4", 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.023001+0000", 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00054081799999999998 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.023542+0000", 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011881500000000001 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.023661+0000", 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043632126 2024-02-23T00:38:30.115 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.067293+0000", 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3884999999999997e-05 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.067337+0000", 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7705e-05 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.067364+0000", 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044787916999999997 2024-02-23T00:38:30.116 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36842.0:1 112.3s0 112:d6e58105:test-rados-api-smithi087-82115-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:57.017766+0000", 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 393.01272480099999, 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.01719962, 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36842", 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3011193832", 2024-02-23T00:38:30.117 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017766+0000", 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017766+0000", 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.118 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017764+0000", 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5610000000000004e-06 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017774+0000", 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1437000000000001e-05 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017785+0000", 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6439999999999995e-06 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.119 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017795+0000", 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020082722000000001 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.037878+0000", 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3429999999999998e-06 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.037882+0000", 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.99038801300000001 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.120 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.028270+0000", 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113621 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.028384+0000", 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00034479999999999998 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.028728+0000", 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011224329999999999 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.121 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.029851+0000", 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050385860000000003 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.034889+0000", 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6290000000000003e-05 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.034966+0000", 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.01719962 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.122 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:2079 63.3 63:c0773ebc:test-rados-api-smithi087-82091-26::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e234)", 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:05.027503+0000", 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 325.00298817200002, 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019673782000000001, 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2079 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.123 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027503+0000", 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027503+0000", 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027501+0000", 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0348e-05 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.124 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027511+0000", 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3602e-05 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027525+0000", 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7789999999999996e-06 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027533+0000", 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0073932110000000002 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.125 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.034926+0000", 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014632299999999999 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.035072+0000", 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000197312 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,4", 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.035269+0000", 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040784150000000002 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.039348+0000", 2024-02-23T00:38:30.126 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035992540000000001 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.042947+0000", 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041140780000000002 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.047061+0000", 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1672999999999998e-05 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.047113+0000", 2024-02-23T00:38:30.127 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3707999999999994e-05 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.047177+0000", 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019673782000000001 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:5418 63.1b 63:d8ef31d3:test-rados-api-smithi087-82091-50::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:35:23.068992+0000", 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 186.96149848900001, 2024-02-23T00:38:30.128 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033546511000000001, 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 5418 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.068992+0000", 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.129 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.068992+0000", 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.068990+0000", 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0651e-05 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069001+0000", 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4694e-05 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.130 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069015+0000", 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1549999999999996e-06 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069025+0000", 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4701999999999999e-05 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069079+0000", 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0494e-05 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:38:30.131 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069090+0000", 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035708100000000002 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.072661+0000", 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014544499999999999 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.072806+0000", 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018308800000000001 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:38:30.132 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.072989+0000", 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036468820000000002 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.076636+0000", 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024542562 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.101179+0000", 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00126581 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102444+0000", 2024-02-23T00:38:30.133 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5225000000000001e-05 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102500+0000", 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9344999999999997e-05 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102539+0000", 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033546511000000001 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.134 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6302 63.17 63:e8b72b4a:test-rados-api-smithi087-82091-64::foo:head [write 0~8] snapc 1d=[1d,1c] ondisk+write+known_if_redirected e432)", 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:36:44.294999+0000", 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 105.73549178099999, 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023072202, 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6302 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.135 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.294999+0000", 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.294999+0000", 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999971 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.294996+0000", 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4647000000000001e-05 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.136 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295011+0000", 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1302e-05 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295032+0000", 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0713999999999999e-05 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295043+0000", 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7822999999999998e-05 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.137 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295101+0000", 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4919999999999993e-06 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295109+0000", 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0058836230000000002 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.300993+0000", 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110759 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.301104+0000", 2024-02-23T00:38:30.138 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00020068000000000001 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.301305+0000", 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00069117699999999996 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.301996+0000", 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00045108700000000003 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.302447+0000", 2024-02-23T00:38:30.139 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015506539999999999 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.317953+0000", 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7464000000000003e-05 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.318021+0000", 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0603000000000003e-05 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.318071+0000", 2024-02-23T00:38:30.140 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023072202 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6304 63.17 63:e8b72b4a:test-rados-api-smithi087-82091-64::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e432)", 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:36:44.326711+0000", 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 105.703779844, 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024147567000000002, 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:30.141 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6304 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326711+0000", 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326711+0000", 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.142 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326710+0000", 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2559999999999996e-06 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326717+0000", 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3875e-05 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326731+0000", 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9949999999999999e-06 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326738+0000", 2024-02-23T00:38:30.143 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7929000000000001e-05 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326786+0000", 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2782999999999996e-05 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326879+0000", 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076338630000000003 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:38:30.144 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.334513+0000", 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011122320000000001 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.335625+0000", 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0062604449999999999 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.341885+0000", 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088814819999999996 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.145 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.350767+0000", 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0455999999999999e-05 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.350817+0000", 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1442000000000001e-05 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.350859+0000", 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024147567000000002 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.146 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6305 63.17 63:e8b72b4a:test-rados-api-smithi087-82091-64::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e432)", 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:36:44.361680+0000", 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 105.668811362, 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026227607999999999, 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6305 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.147 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361680+0000", 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361680+0000", 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361678+0000", 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2040000000000003e-06 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.148 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361687+0000", 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4222999999999999e-05 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361701+0000", 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2899999999999997e-06 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361709+0000", 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7270000000000001e-05 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.149 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361746+0000", 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9739000000000002e-05 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361846+0000", 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.02429605 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.386142+0000", 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00122338 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.150 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387365+0000", 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033618999999999999 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387701+0000", 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000123817 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387825+0000", 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4799000000000003e-05 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:30.151 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387870+0000", 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7274000000000003e-05 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387907+0000", 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026227607999999999 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:7082 219.d 219:b0000000:%01:::head [pgnls start_epoch 475] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e475)", 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:12.189429+0000", 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 17.841062384000001, 2024-02-23T00:38:30.152 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011546450999999999, 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 7082 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.189429+0000", 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.153 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.189429+0000", 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.189426+0000", 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2779000000000001e-05 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.189439+0000", 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1611e-05 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.154 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.189460+0000", 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.054e-06 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.189470+0000", 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4398999999999999e-05 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.189524+0000", 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8070000000000002e-06 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.155 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.189531+0000", 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011257858000000001 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.200789+0000", 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3061000000000001e-05 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.200822+0000", 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015339600000000001 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:30.156 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:12.200975+0000", 2024-02-23T00:38:30.157 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011546450999999999 2024-02-23T00:38:30.157 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.157 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.157 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.157 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:30.157 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:30.157 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:38:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:30.158 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:30.158 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:30.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:30.259 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:30.330 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:30.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:29.192942+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:30.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:29.193014+0000 mon.a (mon.0) 2801 : cluster [DBG] osdmap e492: 8 total, 7 up, 7 in 2024-02-23T00:38:30.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:29.465220+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? 172.21.15.87:0/2160524890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:30.549 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:29.192942+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:30.549 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:29.193014+0000 mon.a (mon.0) 2801 : cluster [DBG] osdmap e492: 8 total, 7 up, 7 in 2024-02-23T00:38:30.549 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:29.465220+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? 172.21.15.87:0/2160524890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:29.192942+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:29.193014+0000 mon.a (mon.0) 2801 : cluster [DBG] osdmap e492: 8 total, 7 up, 7 in 2024-02-23T00:38:30.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:29.465220+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? 172.21.15.87:0/2160524890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:30.757 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-23T00:38:30.810 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:30.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:30.913 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:30.944 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:30.971 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:30.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:38:30.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:38:31.026 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:31.074 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:31.101 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:31.300 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:31.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:31.321 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:29.702668+0000 mgr.y (mgr.24437) 468 : cluster [DBG] pgmap v754: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 68 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 120/1194 objects degraded (10.050%); 84/1194 objects misplaced (7.035%) 2024-02-23T00:38:31.321 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:30.201195+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e493: 8 total, 7 up, 7 in 2024-02-23T00:38:31.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:38:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:38:31.416 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:31.443 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:31.479 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:29.702668+0000 mgr.y (mgr.24437) 468 : cluster [DBG] pgmap v754: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 68 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 120/1194 objects degraded (10.050%); 84/1194 objects misplaced (7.035%) 2024-02-23T00:38:31.479 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:30.201195+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e493: 8 total, 7 up, 7 in 2024-02-23T00:38:31.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:29.702668+0000 mgr.y (mgr.24437) 468 : cluster [DBG] pgmap v754: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 68 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 120/1194 objects degraded (10.050%); 84/1194 objects misplaced (7.035%) 2024-02-23T00:38:31.479 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:30.201195+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e493: 8 total, 7 up, 7 in 2024-02-23T00:38:31.481 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:31.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:31.594 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:31.624 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:31.872 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:31.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:31.976 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:32.002 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:32.351 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:32.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:32.429 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:32] "GET /metrics HTTP/1.1" 200 188826 "" "Prometheus/2.33.4" 2024-02-23T00:38:32.455 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:32.484 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:32.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:31.021985+0000 mgr.y (mgr.24437) 469 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:32.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:31.214316+0000 mon.a (mon.0) 2803 : cluster [DBG] osdmap e494: 8 total, 7 up, 7 in 2024-02-23T00:38:32.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:31.343864+0000 mon.b (mon.1) 405 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:32.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:31.344292+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:32.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:31.346890+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-79"}]: dispatch 2024-02-23T00:38:32.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:31.347052+0000 mon.a (mon.0) 2805 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-79"}]: dispatch 2024-02-23T00:38:32.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:32.132715+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e495: 8 total, 7 up, 7 in 2024-02-23T00:38:32.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:31.021985+0000 mgr.y (mgr.24437) 469 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:32.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:31.214316+0000 mon.a (mon.0) 2803 : cluster [DBG] osdmap e494: 8 total, 7 up, 7 in 2024-02-23T00:38:32.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:31.343864+0000 mon.b (mon.1) 405 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:32.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:31.344292+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:32.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:31.346890+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-79"}]: dispatch 2024-02-23T00:38:32.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:31.347052+0000 mon.a (mon.0) 2805 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-79"}]: dispatch 2024-02-23T00:38:32.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:32.132715+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e495: 8 total, 7 up, 7 in 2024-02-23T00:38:32.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:31.021985+0000 mgr.y (mgr.24437) 469 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:32.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:31.214316+0000 mon.a (mon.0) 2803 : cluster [DBG] osdmap e494: 8 total, 7 up, 7 in 2024-02-23T00:38:32.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:31.343864+0000 mon.b (mon.1) 405 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:32.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:31.344292+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:32.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:31.346890+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-79"}]: dispatch 2024-02-23T00:38:32.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:31.347052+0000 mon.a (mon.0) 2805 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-79"}]: dispatch 2024-02-23T00:38:32.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:32.132715+0000 mon.a (mon.0) 2806 : cluster [DBG] osdmap e495: 8 total, 7 up, 7 in 2024-02-23T00:38:32.887 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.848+0000 7fb7e0042700 1 -- 172.21.15.87:0/472171919 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 msgr2=0x7fb7d80f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.848+0000 7fb7e0042700 1 --2- 172.21.15.87:0/472171919 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 0x7fb7d80f7d40 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 rx=0x7fb7d40047b0 tx=0x7fb7d4009cb0).stop 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.848+0000 7fb7e0042700 1 -- 172.21.15.87:0/472171919 shutdown_connections 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.848+0000 7fb7e0042700 1 --2- 172.21.15.87:0/472171919 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d80f8420 0x7fb7d8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.848+0000 7fb7e0042700 1 --2- 172.21.15.87:0/472171919 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7d80fdbb0 0x7fb7d80fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.848+0000 7fb7e0042700 1 --2- 172.21.15.87:0/472171919 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 0x7fb7d80f7d40 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.848+0000 7fb7e0042700 1 -- 172.21.15.87:0/472171919 >> 172.21.15.87:0/472171919 conn(0x7fb7d80f3800 msgr2=0x7fb7d80f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.848+0000 7fb7e0042700 1 -- 172.21.15.87:0/472171919 shutdown_connections 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.848+0000 7fb7e0042700 1 -- 172.21.15.87:0/472171919 wait complete. 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 librados: starting msgr at 2024-02-23T00:38:32.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 librados: starting objecter 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 Processor -- start 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 -- start start 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 librados: setting wanted keys 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 librados: calling monclient init 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d80f8420 0x7fb7d810e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7d80fdbb0 0x7fb7d810e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 0x7fb7d8108240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8108730 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d81088a0 con 0x7fb7d80f8420 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7e0042700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d8108a60 con 0x7fb7d80fe510 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7de5df700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 0x7fb7d8108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7dd5dd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7d80fdbb0 0x7fb7d810e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7de5df700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 0x7fb7d8108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:55324/0 (socket says 172.21.15.87:55324) 2024-02-23T00:38:32.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7dd5dd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7d80fdbb0 0x7fb7d810e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45166/0 (socket says 172.21.15.87:45166) 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7de5df700 1 -- 172.21.15.87:0/3170182988 learned_addr learned my addr 172.21.15.87:0/3170182988 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.849+0000 7fb7dddde700 1 --2- 172.21.15.87:0/3170182988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d80f8420 0x7fb7d810e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7dd5dd700 1 -- 172.21.15.87:0/3170182988 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 msgr2=0x7fb7d8108240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7dd5dd700 1 --2- 172.21.15.87:0/3170182988 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 0x7fb7d8108240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7dd5dd700 1 -- 172.21.15.87:0/3170182988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d80f8420 msgr2=0x7fb7d810e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7dd5dd700 1 --2- 172.21.15.87:0/3170182988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d80f8420 0x7fb7d810e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7dd5dd700 1 -- 172.21.15.87:0/3170182988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c8005040 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7de5df700 1 --2- 172.21.15.87:0/3170182988 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 0x7fb7d8108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7dd5dd700 1 --2- 172.21.15.87:0/3170182988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7d80fdbb0 0x7fb7d810e590 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7fb7d4004650 tx=0x7fb7d4007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7ceffd700 10 client.?.objecter ms_handle_connect 0x7fb7d80fdbb0 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7ceffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7ceffd700 1 -- 172.21.15.87:0/3170182988 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4018070 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d4005040 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7ceffd700 1 -- 172.21.15.87:0/3170182988 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7d40092e0 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7ceffd700 1 -- 172.21.15.87:0/3170182988 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d401dbd0 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7e0042700 10 client.27414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d8108f80 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.850+0000 7fb7e0042700 1 librados: init done 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.851+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d805ee80 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.851+0000 7fb7ceffd700 1 -- 172.21.15.87:0/3170182988 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb7d401d420 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.852+0000 7fb7ceffd700 1 --2- 172.21.15.87:0/3170182988 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb7c4062c90 0x7fb7c4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.852+0000 7fb7ceffd700 1 -- 172.21.15.87:0/3170182988 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(495..495 src has 173..495) v4 ==== 8021+0+0 (secure 0 0 0) 0x7fb7d407e4b0 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.852+0000 7fb7ceffd700 10 client.27414.objecter ms_dispatch 0x7fb7d8001550 osd_map(495..495 src has 173..495) v4 2024-02-23T00:38:32.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.852+0000 7fb7ceffd700 3 client.27414.objecter handle_osd_map got epochs [495,495] > 0 2024-02-23T00:38:32.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.852+0000 7fb7ceffd700 3 client.27414.objecter handle_osd_map decoding full epoch 495 2024-02-23T00:38:32.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.852+0000 7fb7dddde700 1 --2- 172.21.15.87:0/3170182988 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb7c4062c90 0x7fb7c4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:32.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.852+0000 7fb7ceffd700 20 client.27414.objecter dump_active .. 0 homeless 2024-02-23T00:38:32.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.852+0000 7fb7ceffd700 10 client.27414.objecter ms_handle_connect 0x7fb7c4062c90 2024-02-23T00:38:32.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.852+0000 7fb7dddde700 1 --2- 172.21.15.87:0/3170182988 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb7c4062c90 0x7fb7c4065150 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7fb7c8004870 tx=0x7fb7c800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:32.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:32.856+0000 7fb7ceffd700 1 -- 172.21.15.87:0/3170182988 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb7d40519e0 con 0x7fb7d80fdbb0 2024-02-23T00:38:32.893 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:38:32.894 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:38:32.894 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:38:32.894 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:38:32.933 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:32.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:32.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:32.992 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:38:33.036 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:33.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.042+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fb7d8109770 con 0x7fb7d80fdbb0 2024-02-23T00:38:33.063 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:33.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.135+0000 7fb7ceffd700 1 -- 172.21.15.87:0/3170182988 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v496) v1 ==== 91+0+0 (secure 0 0 0) 0x7fb7d40515d0 con 0x7fb7d80fdbb0 2024-02-23T00:38:33.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.139+0000 7fb7e0042700 10 librados: watch_flush enter 2024-02-23T00:38:33.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.139+0000 7fb7e0042700 10 librados: watch_flush exit 2024-02-23T00:38:33.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.139+0000 7fb7e0042700 20 client.27414.objecter shutdown clearing up homeless session... 2024-02-23T00:38:33.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.139+0000 7fb7e0042700 10 client.27414.objecter successfully canceled tick 2024-02-23T00:38:33.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.143+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb7c4062c90 msgr2=0x7fb7c4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:33.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.143+0000 7fb7e0042700 1 --2- 172.21.15.87:0/3170182988 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb7c4062c90 0x7fb7c4065150 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 rx=0x7fb7c8004870 tx=0x7fb7c800b040).stop 2024-02-23T00:38:33.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.143+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7d80fdbb0 msgr2=0x7fb7d810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:33.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.143+0000 7fb7e0042700 1 --2- 172.21.15.87:0/3170182988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7d80fdbb0 0x7fb7d810e590 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 rx=0x7fb7d4004650 tx=0x7fb7d4007f90).stop 2024-02-23T00:38:33.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.148+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 shutdown_connections 2024-02-23T00:38:33.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.148+0000 7fb7e0042700 1 --2- 172.21.15.87:0/3170182988 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb7c4062c90 0x7fb7c4065150 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:33.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.148+0000 7fb7e0042700 1 --2- 172.21.15.87:0/3170182988 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb7d80fe510 0x7fb7d8108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:33.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.148+0000 7fb7e0042700 1 --2- 172.21.15.87:0/3170182988 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb7d80f8420 0x7fb7d810e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:33.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.148+0000 7fb7e0042700 1 --2- 172.21.15.87:0/3170182988 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb7d80fdbb0 0x7fb7d810e590 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:33.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.148+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 >> 172.21.15.87:0/3170182988 conn(0x7fb7d80f3800 msgr2=0x7fb7d80f52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:33.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.152+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 shutdown_connections 2024-02-23T00:38:33.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.152+0000 7fb7e0042700 1 -- 172.21.15.87:0/3170182988 wait complete. 2024-02-23T00:38:33.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:33.157+0000 7fb7e0042700 1 librados: shutdown 2024-02-23T00:38:33.162 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:38:33.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:31.703292+0000 mgr.y (mgr.24437) 470 : cluster [DBG] pgmap v757: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 27 active+undersized+degraded, 67 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s; 123/1203 objects degraded (10.224%); 84/1203 objects misplaced (6.983%) 2024-02-23T00:38:33.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:32.422201+0000 mon.a (mon.0) 2807 : cluster [WRN] Health check update: Degraded data redundancy: 123/1203 objects degraded (10.224%), 29 pgs degraded, 75 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:33.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:33.044645+0000 mon.a (mon.0) 2808 : audit [INF] from='client.? 172.21.15.87:0/3170182988' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:38:33.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:33.130110+0000 mon.a (mon.0) 2809 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:33.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:33.135985+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? 172.21.15.87:0/3170182988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:38:33.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:33.136047+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e496: 8 total, 7 up, 7 in 2024-02-23T00:38:33.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:33.150851+0000 mon.b (mon.1) 407 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:33.536 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:33.152696+0000 mon.a (mon.0) 2812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:33.553 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-23T00:38:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:31.703292+0000 mgr.y (mgr.24437) 470 : cluster [DBG] pgmap v757: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 27 active+undersized+degraded, 67 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s; 123/1203 objects degraded (10.224%); 84/1203 objects misplaced (6.983%) 2024-02-23T00:38:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:32.422201+0000 mon.a (mon.0) 2807 : cluster [WRN] Health check update: Degraded data redundancy: 123/1203 objects degraded (10.224%), 29 pgs degraded, 75 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:33.044645+0000 mon.a (mon.0) 2808 : audit [INF] from='client.? 172.21.15.87:0/3170182988' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:38:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:33.130110+0000 mon.a (mon.0) 2809 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:33.135985+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? 172.21.15.87:0/3170182988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:38:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:33.136047+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e496: 8 total, 7 up, 7 in 2024-02-23T00:38:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:33.150851+0000 mon.b (mon.1) 407 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:33.152696+0000 mon.a (mon.0) 2812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:31.703292+0000 mgr.y (mgr.24437) 470 : cluster [DBG] pgmap v757: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 27 active+undersized+degraded, 67 active+undersized, 146 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s; 123/1203 objects degraded (10.224%); 84/1203 objects misplaced (6.983%) 2024-02-23T00:38:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:32.422201+0000 mon.a (mon.0) 2807 : cluster [WRN] Health check update: Degraded data redundancy: 123/1203 objects degraded (10.224%), 29 pgs degraded, 75 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:33.044645+0000 mon.a (mon.0) 2808 : audit [INF] from='client.? 172.21.15.87:0/3170182988' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:38:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:33.130110+0000 mon.a (mon.0) 2809 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:33.135985+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? 172.21.15.87:0/3170182988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:38:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:33.136047+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e496: 8 total, 7 up, 7 in 2024-02-23T00:38:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:33.150851+0000 mon.b (mon.1) 407 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:33.152696+0000 mon.a (mon.0) 2812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:33.821 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:33.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:38:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:38:34.412 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:34.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:34.516 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:34.545 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:34.618 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:34.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:33.237667+0000 osd.1 (osd.1) 231 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:38:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:33.242413+0000 osd.1 (osd.1) 232 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:38:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:33.703895+0000 mgr.y (mgr.24437) 471 : cluster [DBG] pgmap v760: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 32 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 52 active+undersized, 130 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 121/1197 objects degraded (10.109%); 84/1197 objects misplaced (7.018%) 2024-02-23T00:38:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:34.094355+0000 osd.4 (osd.4) 236 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:38:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:34.097238+0000 osd.4 (osd.4) 237 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:38:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:34.136156+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:34.136237+0000 mon.a (mon.0) 2814 : cluster [DBG] osdmap e497: 8 total, 7 up, 7 in 2024-02-23T00:38:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:34.140918+0000 mon.b (mon.1) 408 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:34.152429+0000 mon.b (mon.1) 409 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:34.154146+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:34.721 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:34.752 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:34.821 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:34.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:33.237667+0000 osd.1 (osd.1) 231 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:33.242413+0000 osd.1 (osd.1) 232 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:33.703895+0000 mgr.y (mgr.24437) 471 : cluster [DBG] pgmap v760: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 32 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 52 active+undersized, 130 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 121/1197 objects degraded (10.109%); 84/1197 objects misplaced (7.018%) 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:34.094355+0000 osd.4 (osd.4) 236 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:34.097238+0000 osd.4 (osd.4) 237 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:34.136156+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:34.136237+0000 mon.a (mon.0) 2814 : cluster [DBG] osdmap e497: 8 total, 7 up, 7 in 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:34.140918+0000 mon.b (mon.1) 408 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:34.152429+0000 mon.b (mon.1) 409 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:34.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:34.154146+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:33.237667+0000 osd.1 (osd.1) 231 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:33.242413+0000 osd.1 (osd.1) 232 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:33.703895+0000 mgr.y (mgr.24437) 471 : cluster [DBG] pgmap v760: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 32 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 52 active+undersized, 130 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 121/1197 objects degraded (10.109%); 84/1197 objects misplaced (7.018%) 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:34.094355+0000 osd.4 (osd.4) 236 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:34.097238+0000 osd.4 (osd.4) 237 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:34.136156+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:34.136237+0000 mon.a (mon.0) 2814 : cluster [DBG] osdmap e497: 8 total, 7 up, 7 in 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:34.140918+0000 mon.b (mon.1) 408 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:34.152429+0000 mon.b (mon.1) 409 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:34.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:34.154146+0000 mon.a (mon.0) 2815 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:34.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:34.953 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:35.028 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:35.151 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:35.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:35.257 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:35.282 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:35.432 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:35.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:35.533 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:34.285167+0000 osd.2 (osd.2) 187 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:38:35.534 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:34.287808+0000 osd.2 (osd.2) 188 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:38:35.534 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:35.041408+0000 osd.7 (osd.7) 247 : cluster [DBG] 22.d scrub starts 2024-02-23T00:38:35.534 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:35.044293+0000 osd.7 (osd.7) 248 : cluster [DBG] 22.d scrub ok 2024-02-23T00:38:35.534 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:35.137487+0000 mon.a (mon.0) 2816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:35.534 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:35.137558+0000 mon.a (mon.0) 2817 : cluster [DBG] osdmap e498: 8 total, 7 up, 7 in 2024-02-23T00:38:35.534 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:35.148462+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:35.534 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:35.148663+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:35.534 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:34.285167+0000 osd.2 (osd.2) 187 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:38:35.534 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:34.287808+0000 osd.2 (osd.2) 188 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:38:35.535 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:35.041408+0000 osd.7 (osd.7) 247 : cluster [DBG] 22.d scrub starts 2024-02-23T00:38:35.535 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:35.044293+0000 osd.7 (osd.7) 248 : cluster [DBG] 22.d scrub ok 2024-02-23T00:38:35.535 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:35.137487+0000 mon.a (mon.0) 2816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:35.535 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:35.137558+0000 mon.a (mon.0) 2817 : cluster [DBG] osdmap e498: 8 total, 7 up, 7 in 2024-02-23T00:38:35.535 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:35.148462+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:35.535 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:35.148663+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:35.539 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:35.567 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:35.585 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:35.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:35.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:35.689 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:34.285167+0000 osd.2 (osd.2) 187 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:38:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:34.287808+0000 osd.2 (osd.2) 188 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:38:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:35.041408+0000 osd.7 (osd.7) 247 : cluster [DBG] 22.d scrub starts 2024-02-23T00:38:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:35.044293+0000 osd.7 (osd.7) 248 : cluster [DBG] 22.d scrub ok 2024-02-23T00:38:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:35.137487+0000 mon.a (mon.0) 2816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:35.137558+0000 mon.a (mon.0) 2817 : cluster [DBG] osdmap e498: 8 total, 7 up, 7 in 2024-02-23T00:38:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:35.148462+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:35.148663+0000 mon.a (mon.0) 2818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:35.698 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:38:35.698 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:38:35.698 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:38:35.698 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:38:35.698 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:38:35.717 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:35.736 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:35.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:35.840 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:35.866 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:36.345 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-23T00:38:36.416 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:36.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:36.520 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:36.546 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:36.566 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:36.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:36.657 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:36.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:35.330504+0000 osd.2 (osd.2) 189 : cluster [DBG] 5.d scrub starts 2024-02-23T00:38:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:35.337163+0000 osd.2 (osd.2) 190 : cluster [DBG] 5.d scrub ok 2024-02-23T00:38:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:35.705022+0000 mgr.y (mgr.24437) 472 : cluster [DBG] pgmap v763: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 64 active+undersized, 150 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 122/1224 objects degraded (9.967%); 86/1224 objects misplaced (7.026%) 2024-02-23T00:38:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:35.831233+0000 osd.5 (osd.5) 197 : cluster [DBG] 63.4 scrub starts 2024-02-23T00:38:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:35.832546+0000 osd.5 (osd.5) 198 : cluster [DBG] 63.4 scrub ok 2024-02-23T00:38:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:36.005272+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.87:0/1867265486' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:38:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:36.005596+0000 mon.a (mon.0) 2819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:38:36.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:36.139083+0000 mon.a (mon.0) 2820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:38:36.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:36.139156+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]': finished 2024-02-23T00:38:36.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:36.139199+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e499: 8 total, 7 up, 7 in 2024-02-23T00:38:36.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:36.146505+0000 mon.b (mon.1) 411 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:36.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:36.146834+0000 mon.a (mon.0) 2823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:36.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:35.330504+0000 osd.2 (osd.2) 189 : cluster [DBG] 5.d scrub starts 2024-02-23T00:38:36.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:35.337163+0000 osd.2 (osd.2) 190 : cluster [DBG] 5.d scrub ok 2024-02-23T00:38:36.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:35.705022+0000 mgr.y (mgr.24437) 472 : cluster [DBG] pgmap v763: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 64 active+undersized, 150 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 122/1224 objects degraded (9.967%); 86/1224 objects misplaced (7.026%) 2024-02-23T00:38:36.698 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:35.831233+0000 osd.5 (osd.5) 197 : cluster [DBG] 63.4 scrub starts 2024-02-23T00:38:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:35.832546+0000 osd.5 (osd.5) 198 : cluster [DBG] 63.4 scrub ok 2024-02-23T00:38:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:36.005272+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.87:0/1867265486' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:38:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:36.005596+0000 mon.a (mon.0) 2819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:38:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:36.139083+0000 mon.a (mon.0) 2820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:38:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:36.139156+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]': finished 2024-02-23T00:38:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:36.139199+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e499: 8 total, 7 up, 7 in 2024-02-23T00:38:36.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:36.146505+0000 mon.b (mon.1) 411 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:36.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:36.146834+0000 mon.a (mon.0) 2823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:36.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:35.330504+0000 osd.2 (osd.2) 189 : cluster [DBG] 5.d scrub starts 2024-02-23T00:38:36.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:35.337163+0000 osd.2 (osd.2) 190 : cluster [DBG] 5.d scrub ok 2024-02-23T00:38:36.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:35.705022+0000 mgr.y (mgr.24437) 472 : cluster [DBG] pgmap v763: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 64 active+undersized, 150 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 122/1224 objects degraded (9.967%); 86/1224 objects misplaced (7.026%) 2024-02-23T00:38:36.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:35.831233+0000 osd.5 (osd.5) 197 : cluster [DBG] 63.4 scrub starts 2024-02-23T00:38:36.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:35.832546+0000 osd.5 (osd.5) 198 : cluster [DBG] 63.4 scrub ok 2024-02-23T00:38:36.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:36.005272+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.87:0/1867265486' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:38:36.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:36.005596+0000 mon.a (mon.0) 2819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]: dispatch 2024-02-23T00:38:36.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:36.139083+0000 mon.a (mon.0) 2820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_tier","val": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:38:36.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:36.139156+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]': finished 2024-02-23T00:38:36.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:36.139199+0000 mon.a (mon.0) 2822 : cluster [DBG] osdmap e499: 8 total, 7 up, 7 in 2024-02-23T00:38:36.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:36.146505+0000 mon.b (mon.1) 411 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:36.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:36.146834+0000 mon.a (mon.0) 2823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:36.823 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:36.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:36.927 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:36.955 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:36.975 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:36.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:36.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 -- 172.21.15.87:0/1093007140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c0102210 msgr2=0x7f86c0104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:36.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1093007140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c0102210 0x7f86c0104600 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 rx=0x7f86b8009c30 tx=0x7f86b8004c50).stop 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 -- 172.21.15.87:0/1093007140 shutdown_connections 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1093007140 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c0102210 0x7f86c0104600 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1093007140 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86c00f8cb0 0x7f86c00f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1093007140 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86c00f9610 0x7f86c0101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 -- 172.21.15.87:0/1093007140 >> 172.21.15.87:0/1093007140 conn(0x7f86c00f2820 msgr2=0x7f86c00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 -- 172.21.15.87:0/1093007140 shutdown_connections 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 -- 172.21.15.87:0/1093007140 wait complete. 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 librados: starting msgr at 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.824+0000 7f86c7d68700 1 librados: starting objecter 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 Processor -- start 2024-02-23T00:38:36.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 -- start start 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 librados: setting wanted keys 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 librados: calling monclient init 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86c00f8cb0 0x7f86c010dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c00f9610 0x7f86c010e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86c0102210 0x7f86c0108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c0108640 con 0x7f86c00f8cb0 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c01087b0 con 0x7f86c0102210 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c7d68700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c0108970 con 0x7f86c00f9610 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5303700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c00f9610 0x7f86c010e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5b04700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86c00f8cb0 0x7f86c010dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:36.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5303700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c00f9610 0x7f86c010e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:55342/0 (socket says 172.21.15.87:55342) 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5303700 1 -- 172.21.15.87:0/1867265486 learned_addr learned my addr 172.21.15.87:0/1867265486 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5b04700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86c00f8cb0 0x7f86c010dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45170/0 (socket says 172.21.15.87:45170) 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c6305700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86c0102210 0x7f86c0108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5303700 1 -- 172.21.15.87:0/1867265486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86c0102210 msgr2=0x7f86c0108150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5303700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86c0102210 0x7f86c0108150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5303700 1 -- 172.21.15.87:0/1867265486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86c00f8cb0 msgr2=0x7f86c010dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5303700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86c00f8cb0 0x7f86c010dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5303700 1 -- 172.21.15.87:0/1867265486 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86bc005040 con 0x7f86c00f9610 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.825+0000 7f86c5b04700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86c00f8cb0 0x7f86c010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86c5303700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c00f9610 0x7f86c010e4a0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7f86b00041d0 tx=0x7f86b0009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:36.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86b6ffd700 10 client.?.objecter ms_handle_connect 0x7f86c00f9610 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86b6ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86b6ffd700 1 -- 172.21.15.87:0/1867265486 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0007b40 con 0x7f86c00f9610 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b80097c0 con 0x7f86c00f9610 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86b6ffd700 1 -- 172.21.15.87:0/1867265486 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86b000f040 con 0x7f86c00f9610 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86b6ffd700 1 -- 172.21.15.87:0/1867265486 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0014860 con 0x7f86c00f9610 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86c7d68700 10 client.37886.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c0108f50 con 0x7f86c00f9610 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.826+0000 7f86c7d68700 1 librados: init done 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.827+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c005fa60 con 0x7f86c00f9610 2024-02-23T00:38:36.996 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.827+0000 7f86b6ffd700 1 -- 172.21.15.87:0/1867265486 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f86b0014c50 con 0x7f86c00f9610 2024-02-23T00:38:36.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.828+0000 7f86b6ffd700 1 --2- 172.21.15.87:0/1867265486 >> v2:172.21.15.87:6800/1231597463 conn(0x7f86ac062c90 0x7f86ac065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:36.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.828+0000 7f86b6ffd700 1 -- 172.21.15.87:0/1867265486 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(498..498 src has 173..498) v4 ==== 8422+0+0 (secure 0 0 0) 0x7f86b007ce60 con 0x7f86c00f9610 2024-02-23T00:38:36.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.828+0000 7f86b6ffd700 10 client.37886.objecter ms_dispatch 0x7f86c0001550 osd_map(498..498 src has 173..498) v4 2024-02-23T00:38:36.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.828+0000 7f86b6ffd700 3 client.37886.objecter handle_osd_map got epochs [498,498] > 0 2024-02-23T00:38:36.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.828+0000 7f86b6ffd700 3 client.37886.objecter handle_osd_map decoding full epoch 498 2024-02-23T00:38:36.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.828+0000 7f86c5b04700 1 --2- 172.21.15.87:0/1867265486 >> v2:172.21.15.87:6800/1231597463 conn(0x7f86ac062c90 0x7f86ac065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:36.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.828+0000 7f86b6ffd700 20 client.37886.objecter dump_active .. 0 homeless 2024-02-23T00:38:37.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.828+0000 7f86b6ffd700 10 client.37886.objecter ms_handle_connect 0x7f86ac062c90 2024-02-23T00:38:37.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.828+0000 7f86c5b04700 1 --2- 172.21.15.87:0/1867265486 >> v2:172.21.15.87:6800/1231597463 conn(0x7f86ac062c90 0x7f86ac065150 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 rx=0x7f86bc004c80 tx=0x7f86bc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:37.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:35.832+0000 7f86b6ffd700 1 -- 172.21.15.87:0/1867265486 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f86b00501c0 con 0x7f86c00f9610 2024-02-23T00:38:37.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.003+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"} v 0) v1 -- 0x7f86c0107290 con 0x7f86c00f9610 2024-02-23T00:38:37.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.145+0000 7f86b6ffd700 1 -- 172.21.15.87:0/1867265486 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "22"}]=0 set pool 6 pg_num to 22 v499) v1 ==== 141+0+0 (secure 0 0 0) 0x7f86b004fdb0 con 0x7f86c00f9610 2024-02-23T00:38:37.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.148+0000 7f86c7d68700 10 librados: watch_flush enter 2024-02-23T00:38:37.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.149+0000 7f86c7d68700 10 librados: watch_flush exit 2024-02-23T00:38:37.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.149+0000 7f86c7d68700 20 client.37886.objecter shutdown clearing up homeless session... 2024-02-23T00:38:37.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.149+0000 7f86c7d68700 10 client.37886.objecter successfully canceled tick 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.149+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 >> v2:172.21.15.87:6800/1231597463 conn(0x7f86ac062c90 msgr2=0x7f86ac065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.149+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1867265486 >> v2:172.21.15.87:6800/1231597463 conn(0x7f86ac062c90 0x7f86ac065150 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 rx=0x7f86bc004c80 tx=0x7f86bc00b040).stop 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.149+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c00f9610 msgr2=0x7f86c010e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.149+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c00f9610 0x7f86c010e4a0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 rx=0x7f86b00041d0 tx=0x7f86b0009f60).stop 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.151+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 shutdown_connections 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.151+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1867265486 >> v2:172.21.15.87:6800/1231597463 conn(0x7f86ac062c90 0x7f86ac065150 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.151+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f86c0102210 0x7f86c0108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.151+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f86c00f8cb0 0x7f86c010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.151+0000 7f86c7d68700 1 --2- 172.21.15.87:0/1867265486 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f86c00f9610 0x7f86c010e4a0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.151+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 >> 172.21.15.87:0/1867265486 conn(0x7f86c00f2820 msgr2=0x7f86c00f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:37.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.153+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 shutdown_connections 2024-02-23T00:38:37.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.153+0000 7f86c7d68700 1 -- 172.21.15.87:0/1867265486 wait complete. 2024-02-23T00:38:37.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:36.153+0000 7f86c7d68700 1 librados: shutdown 2024-02-23T00:38:37.002 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 6 pg_num to 22 2024-02-23T00:38:37.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:38:37.078 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:37.105 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:38.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 -- 172.21.15.87:0/1358458627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffaac0f9610 msgr2=0x7ffaac101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:38.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/1358458627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffaac0f9610 0x7ffaac101d20 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 rx=0x7ffaa0004870 tx=0x7ffaa0009cb0).stop 2024-02-23T00:38:38.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 -- 172.21.15.87:0/1358458627 shutdown_connections 2024-02-23T00:38:38.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/1358458627 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffaac102210 0x7ffaac104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:38.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/1358458627 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffaac0f8cb0 0x7ffaac0f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:38.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/1358458627 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffaac0f9610 0x7ffaac101d20 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:38.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 -- 172.21.15.87:0/1358458627 >> 172.21.15.87:0/1358458627 conn(0x7ffaac0f2820 msgr2=0x7ffaac0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:38.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 -- 172.21.15.87:0/1358458627 shutdown_connections 2024-02-23T00:38:38.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 -- 172.21.15.87:0/1358458627 wait complete. 2024-02-23T00:38:38.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 librados: starting msgr at 2024-02-23T00:38:38.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.070+0000 7ffab1ecf700 1 librados: starting objecter 2024-02-23T00:38:38.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 Processor -- start 2024-02-23T00:38:38.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 -- start start 2024-02-23T00:38:38.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 librados: setting wanted keys 2024-02-23T00:38:38.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 librados: calling monclient init 2024-02-23T00:38:38.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffaac0f8cb0 0x7ffaac10e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffaac0f9610 0x7ffaac10e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffaac102210 0x7ffaac1083e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac1088d0 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac108a40 con 0x7ffaac102210 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffab1ecf700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac108c00 con 0x7ffaac0f9610 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffaab7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffaac0f8cb0 0x7ffaac10e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffaab7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffaac0f8cb0 0x7ffaac10e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45186/0 (socket says 172.21.15.87:45186) 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffaab7fe700 1 -- 172.21.15.87:0/2215659426 learned_addr learned my addr 172.21.15.87:0/2215659426 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffaaaffd700 1 --2- 172.21.15.87:0/2215659426 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffaac0f9610 0x7ffaac10e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffaabfff700 1 --2- 172.21.15.87:0/2215659426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffaac102210 0x7ffaac1083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffaab7fe700 1 -- 172.21.15.87:0/2215659426 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffaac0f9610 msgr2=0x7ffaac10e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.071+0000 7ffaab7fe700 1 --2- 172.21.15.87:0/2215659426 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffaac0f9610 0x7ffaac10e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaab7fe700 1 -- 172.21.15.87:0/2215659426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffaac102210 msgr2=0x7ffaac1083e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:38.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaab7fe700 1 --2- 172.21.15.87:0/2215659426 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ffaac102210 0x7ffaac1083e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaab7fe700 1 -- 172.21.15.87:0/2215659426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa0005040 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaaaffd700 1 --2- 172.21.15.87:0/2215659426 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ffaac0f9610 0x7ffaac10e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaab7fe700 1 --2- 172.21.15.87:0/2215659426 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ffaac0f8cb0 0x7ffaac10e1d0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 rx=0x7ffa9c00e970 tx=0x7ffa9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaa8ff9700 10 client.?.objecter ms_handle_connect 0x7ffaac0f8cb0 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaa8ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c004660 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa9c00f040 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaac108ee0 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c014790 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffab1ecf700 10 client.27426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaac109260 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.072+0000 7ffab1ecf700 1 librados: init done 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.073+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ffaac05fa60 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.073+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ffa9c0090d0 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa8ff9700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6800/1231597463 conn(0x7ffa94062c90 0x7ffa94065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(500..500 src has 173..500) v4 ==== 8457+0+0 (secure 0 0 0) 0x7ffa9c07cf00 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 osd_map(500..500 src has 173..500) v4 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa8ff9700 3 client.27426.objecter handle_osd_map got epochs [500,500] > 0 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa8ff9700 3 client.27426.objecter handle_osd_map decoding full epoch 500 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaaaffd700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6800/1231597463 conn(0x7ffa94062c90 0x7ffa94065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa8ff9700 20 client.27426.objecter dump_active .. 0 homeless 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v500) v1 ==== 54+0+16 (secure 0 0 0) 0x7ffa9c07f590 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa8ff9700 10 client.27426.objecter ms_handle_connect 0x7ffa94062c90 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaaaffd700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6800/1231597463 conn(0x7ffa94062c90 0x7ffa94065150 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 rx=0x7ffaa0004ee0 tx=0x7ffaa000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 1 2024-02-23T00:38:38.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffab1ecf700 20 client.27426.objecter _get_session osd=-1 returning homeless 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffab1ecf700 10 client.27426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=501}) v3 -- 0x7ffaac1098c0 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffaac05ee80 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_get_version_reply(handle=1 version=500) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa9c0502f0 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa9ffb700 10 client.27426.objecter _check_command_map_dne tid 1 current 500 map_dne_bound 500 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa9ffb700 10 client.27426.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.074+0000 7ffaa9ffb700 15 client.27426.objecter _session_command_op_remove -1 1 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 2 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6805/685354171 conn(0x7ffaac059e00 0x7ffaac0273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac107290 osd=1 3 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac107290 osd=1 3 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac107290 osd=1 3 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 2 2024-02-23T00:38:38.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac107290 osd=1 3 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 1 2 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac107290 osd=1 4 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 10 client.27426.objecter _send_command 2 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffaac0f4fc0 con 0x7ffaac059e00 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffaabfff700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6805/685354171 conn(0x7ffaac059e00 0x7ffaac0273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffaa8ff9700 10 client.27426.objecter ms_handle_connect 0x7ffaac059e00 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.075+0000 7ffaabfff700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6805/685354171 conn(0x7ffaac059e00 0x7ffaac0273c0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.076+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ffaac0f4fc0 con 0x7ffaac059e00 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.076+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.076+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 2 = system:0 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.076+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac107290 osd=1 4 2024-02-23T00:38:38.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.076+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 1 2 2024-02-23T00:38:38.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:38.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 3 2024-02-23T00:38:38.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac107290 osd=1 3 2024-02-23T00:38:38.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac107290 osd=1 3 2024-02-23T00:38:38.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac107290 osd=1 3 2024-02-23T00:38:38.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 3 2024-02-23T00:38:38.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac107290 osd=1 3 2024-02-23T00:38:38.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 1 3 2024-02-23T00:38:38.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac107290 osd=1 4 2024-02-23T00:38:38.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 10 client.27426.objecter _send_command 3 2024-02-23T00:38:38.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.093+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffaac0f5160 con 0x7ffaac059e00 2024-02-23T00:38:38.096 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:38.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ffaac0f5160 con 0x7ffaac059e00 2024-02-23T00:38:38.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:38.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac107290 osd=1 4 2024-02-23T00:38:38.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 1 3 2024-02-23T00:38:38.097 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:38:38.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:38.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 4 2024-02-23T00:38:38.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6809/3605394228 conn(0x7ffaac028590 0x7ffaac117290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac0595a0 osd=2 3 2024-02-23T00:38:38.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac0595a0 osd=2 3 2024-02-23T00:38:38.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac0595a0 osd=2 3 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 4 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac0595a0 osd=2 3 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 2 4 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac0595a0 osd=2 4 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 10 client.27426.objecter _send_command 4 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffaac059710 con 0x7ffaac028590 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffaab7fe700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6809/3605394228 conn(0x7ffaac028590 0x7ffaac117290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffaa8ff9700 10 client.27426.objecter ms_handle_connect 0x7ffaac028590 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.095+0000 7ffaab7fe700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6809/3605394228 conn(0x7ffaac028590 0x7ffaac117290 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.096+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ffaac059710 con 0x7ffaac028590 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.096+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.096+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 4 = system:0 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.096+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac0595a0 osd=2 4 2024-02-23T00:38:38.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.096+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 2 4 2024-02-23T00:38:38.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:38.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 5 2024-02-23T00:38:38.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac0595a0 osd=2 3 2024-02-23T00:38:38.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac0595a0 osd=2 3 2024-02-23T00:38:38.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac0595a0 osd=2 3 2024-02-23T00:38:38.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 5 2024-02-23T00:38:38.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac0595a0 osd=2 3 2024-02-23T00:38:38.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 2 5 2024-02-23T00:38:38.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac0595a0 osd=2 4 2024-02-23T00:38:38.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 10 client.27426.objecter _send_command 5 2024-02-23T00:38:38.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.111+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffaac0f8ab0 con 0x7ffaac028590 2024-02-23T00:38:38.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.112+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ffaac0f8ab0 con 0x7ffaac028590 2024-02-23T00:38:38.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.112+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:38.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.112+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.112+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac0595a0 osd=2 4 2024-02-23T00:38:38.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.112+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 2 5 2024-02-23T00:38:38.115 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:38:38.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:38.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 6 2024-02-23T00:38:38.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6813/128059790 conn(0x7ffaac0fc9b0 0x7ffaac118900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac027d30 osd=3 3 2024-02-23T00:38:38.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac027d30 osd=3 3 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac027d30 osd=3 3 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 6 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac027d30 osd=3 3 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 3 6 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac027d30 osd=3 4 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 10 client.27426.objecter _send_command 6 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffaac118df0 con 0x7ffaac0fc9b0 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffaaaffd700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6813/128059790 conn(0x7ffaac0fc9b0 0x7ffaac118900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffaa8ff9700 10 client.27426.objecter ms_handle_connect 0x7ffaac0fc9b0 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.113+0000 7ffaaaffd700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.87:6813/128059790 conn(0x7ffaac0fc9b0 0x7ffaac118900 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.114+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ffaac118df0 con 0x7ffaac0fc9b0 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.114+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.114+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 6 = system:0 2024-02-23T00:38:38.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.114+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac027d30 osd=3 4 2024-02-23T00:38:38.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.114+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 3 6 2024-02-23T00:38:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 7 2024-02-23T00:38:38.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac027d30 osd=3 3 2024-02-23T00:38:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac027d30 osd=3 3 2024-02-23T00:38:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac027d30 osd=3 3 2024-02-23T00:38:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 7 2024-02-23T00:38:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac027d30 osd=3 3 2024-02-23T00:38:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 3 7 2024-02-23T00:38:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac027d30 osd=3 4 2024-02-23T00:38:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 10 client.27426.objecter _send_command 7 2024-02-23T00:38:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.130+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffaac1197e0 con 0x7ffaac0fc9b0 2024-02-23T00:38:38.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.131+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ffaac1197e0 con 0x7ffaac0fc9b0 2024-02-23T00:38:38.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.131+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:38.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.131+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.131+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac027d30 osd=3 4 2024-02-23T00:38:38.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.131+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 3 7 2024-02-23T00:38:38.134 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:38:38.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:38.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 8 2024-02-23T00:38:38.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6800/4175844757 conn(0x7ffaac120320 0x7ffaac122710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac0fc1e0 osd=4 3 2024-02-23T00:38:38.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac0fc1e0 osd=4 3 2024-02-23T00:38:38.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac0fc1e0 osd=4 3 2024-02-23T00:38:38.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 8 2024-02-23T00:38:38.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac0fc1e0 osd=4 3 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 4 8 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac0fc1e0 osd=4 4 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 10 client.27426.objecter _send_command 8 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffaac125390 con 0x7ffaac120320 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffaabfff700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6800/4175844757 conn(0x7ffaac120320 0x7ffaac122710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffaabfff700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6800/4175844757 conn(0x7ffaac120320 0x7ffaac122710 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.132+0000 7ffaa8ff9700 10 client.27426.objecter ms_handle_connect 0x7ffaac120320 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.133+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ffaac125390 con 0x7ffaac120320 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.133+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.133+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 8 = system:0 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.133+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac0fc1e0 osd=4 4 2024-02-23T00:38:38.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.133+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 4 8 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 9 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac0fc1e0 osd=4 3 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac0fc1e0 osd=4 3 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac0fc1e0 osd=4 3 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 9 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac0fc1e0 osd=4 3 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 4 9 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac0fc1e0 osd=4 4 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 10 client.27426.objecter _send_command 9 2024-02-23T00:38:38.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.150+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffaac1192b0 con 0x7ffaac120320 2024-02-23T00:38:38.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== mon.0 v2:172.21.15.87:3300/0 8 ==== osd_map(501..501 src has 173..501) v4 ==== 1041+0+0 (secure 0 0 0) 0x7ffa9c04e6c0 con 0x7ffaac0f8cb0 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 osd_map(501..501 src has 173..501) v4 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 3 client.27426.objecter handle_osd_map got epochs [501,501] > 500 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 3 client.27426.objecter handle_osd_map decoding incremental epoch 501 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 10 client.27426.objecter checking command 9 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 20 client.27426.objecter _get_session s=0x7ffaac0fc1e0 osd=4 4 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac0fc1e0 osd=4 4 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 20 client.27426.objecter _recalc_command_target 9 no change, 0x7ffaac0fc1e0 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 20 client.27426.objecter dump_active .. 0 homeless 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ffaac1192b0 con 0x7ffaac120320 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac0fc1e0 osd=4 4 2024-02-23T00:38:38.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.152+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 4 9 2024-02-23T00:38:38.156 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:38:38.156 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 10 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6804/2550013795 conn(0x7ffaac118460 0x7ffaac123d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac119070 osd=5 3 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac119070 osd=5 3 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac119070 osd=5 3 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 10 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac119070 osd=5 3 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 5 10 2024-02-23T00:38:38.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac119070 osd=5 4 2024-02-23T00:38:38.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 10 client.27426.objecter _send_command 10 2024-02-23T00:38:38.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.155+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffaac119210 con 0x7ffaac118460 2024-02-23T00:38:38.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.156+0000 7ffaab7fe700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6804/2550013795 conn(0x7ffaac118460 0x7ffaac123d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.156+0000 7ffaa8ff9700 10 client.27426.objecter ms_handle_connect 0x7ffaac118460 2024-02-23T00:38:38.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.156+0000 7ffaab7fe700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6804/2550013795 conn(0x7ffaac118460 0x7ffaac123d80 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:38.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.158+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ffaac119210 con 0x7ffaac118460 2024-02-23T00:38:38.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.158+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:38:38.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.158+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 10 = system:0 2024-02-23T00:38:38.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.158+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac119070 osd=5 4 2024-02-23T00:38:38.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.158+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 5 10 2024-02-23T00:38:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:38.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:38.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:38.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 11 2024-02-23T00:38:38.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac119070 osd=5 3 2024-02-23T00:38:38.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac119070 osd=5 3 2024-02-23T00:38:38.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac119070 osd=5 3 2024-02-23T00:38:38.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 11 2024-02-23T00:38:38.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac119070 osd=5 3 2024-02-23T00:38:38.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 5 11 2024-02-23T00:38:38.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac119070 osd=5 4 2024-02-23T00:38:38.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 10 client.27426.objecter _send_command 11 2024-02-23T00:38:38.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.179+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffaac119210 con 0x7ffaac118460 2024-02-23T00:38:38.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ffaac119210 con 0x7ffaac118460 2024-02-23T00:38:38.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:38.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac119070 osd=5 4 2024-02-23T00:38:38.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 5 11 2024-02-23T00:38:38.184 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:38:38.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:38.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 12 2024-02-23T00:38:38.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6808/2509408498 conn(0x7ffaac1247f0 0x7ffaac124c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac119210 osd=6 3 2024-02-23T00:38:38.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac119210 osd=6 3 2024-02-23T00:38:38.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac119210 osd=6 3 2024-02-23T00:38:38.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 12 2024-02-23T00:38:38.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac119210 osd=6 3 2024-02-23T00:38:38.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 6 12 2024-02-23T00:38:38.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac119210 osd=6 4 2024-02-23T00:38:38.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 10 client.27426.objecter _send_command 12 2024-02-23T00:38:38.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.181+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffaac12bd30 con 0x7ffaac1247f0 2024-02-23T00:38:38.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.183+0000 7ffaaaffd700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6808/2509408498 conn(0x7ffaac1247f0 0x7ffaac124c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.197+0000 7ffaa8ff9700 10 client.27426.objecter ms_handle_connect 0x7ffaac1247f0 2024-02-23T00:38:38.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.197+0000 7ffaaaffd700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6808/2509408498 conn(0x7ffaac1247f0 0x7ffaac124c30 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:38.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.198+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ffaac12bd30 con 0x7ffaac1247f0 2024-02-23T00:38:38.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.198+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:38:38.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.198+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 12 = system:0 2024-02-23T00:38:38.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.198+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac119210 osd=6 4 2024-02-23T00:38:38.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.198+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 6 12 2024-02-23T00:38:38.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 13 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac119210 osd=6 3 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac119210 osd=6 3 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac119210 osd=6 3 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 13 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac119210 osd=6 3 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 6 13 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac119210 osd=6 4 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 10 client.27426.objecter _send_command 13 2024-02-23T00:38:38.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.225+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffaac1295d0 con 0x7ffaac1247f0 2024-02-23T00:38:38.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ffaac1295d0 con 0x7ffaac1247f0 2024-02-23T00:38:38.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:38.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac119210 osd=6 4 2024-02-23T00:38:38.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 6 13 2024-02-23T00:38:38.230 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 14 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6812/373787000 conn(0x7ffaac129c50 0x7ffaac12a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac1295d0 osd=7 3 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac1295d0 osd=7 3 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac1295d0 osd=7 3 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 14 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac1295d0 osd=7 3 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 7 14 2024-02-23T00:38:38.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac1295d0 osd=7 4 2024-02-23T00:38:38.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 10 client.27426.objecter _send_command 14 2024-02-23T00:38:38.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffaac12bca0 con 0x7ffaac129c50 2024-02-23T00:38:38.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffaabfff700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6812/373787000 conn(0x7ffaac129c50 0x7ffaac12a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:38.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffaa8ff9700 10 client.27426.objecter ms_handle_connect 0x7ffaac129c50 2024-02-23T00:38:38.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.228+0000 7ffaabfff700 1 --2- 172.21.15.87:0/2215659426 >> v2:172.21.15.171:6812/373787000 conn(0x7ffaac129c50 0x7ffaac12a0b0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:38.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.229+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ffaac12bca0 con 0x7ffaac129c50 2024-02-23T00:38:38.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.229+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:38:38.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.229+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 14 = system:0 2024-02-23T00:38:38.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.229+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac1295d0 osd=7 4 2024-02-23T00:38:38.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.229+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 7 14 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 10 client.27426.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign -1 15 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac1295d0 osd=7 3 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac1295d0 osd=7 3 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 20 client.27426.objecter _get_session s=0x7ffaac1295d0 osd=7 3 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_remove -1 15 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 20 client.27426.objecter get_session s=0x7ffaac1295d0 osd=7 3 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 15 client.27426.objecter _session_command_op_assign 7 15 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 20 client.27426.objecter put_session s=0x7ffaac1295d0 osd=7 4 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 10 client.27426.objecter _send_command 15 2024-02-23T00:38:38.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.253+0000 7ffab1ecf700 1 -- 172.21.15.87:0/2215659426 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffaac12bca0 con 0x7ffaac129c50 2024-02-23T00:38:38.257 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:38:38.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.255+0000 7ffaa8ff9700 1 -- 172.21.15.87:0/2215659426 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ffaac12bca0 con 0x7ffaac129c50 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.255+0000 7ffaa8ff9700 10 client.27426.objecter ms_dispatch 0x7ffaac001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.255+0000 7ffaa8ff9700 10 client.27426.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.255+0000 7ffaa8ff9700 20 client.27426.objecter put_session s=0x7ffaac1295d0 osd=7 4 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:38.255+0000 7ffaa8ff9700 15 client.27426.objecter _session_command_op_remove 7 15 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.258 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:38:38.269 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:38.301 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:38.378 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:38.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:38.417 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:38:38.417 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:38:38.436 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:36.709292+0000 osd.4 (osd.4) 238 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:38.436 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:36.709434+0000 osd.4 (osd.4) 239 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:38.436 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:36.852731+0000 osd.5 (osd.5) 199 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:38:38.436 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:36.853578+0000 osd.5 (osd.5) 200 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:38:38.436 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:37.147148+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:38.436 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:37.147235+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e500: 8 total, 7 up, 7 in 2024-02-23T00:38:38.436 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:37.148920+0000 mon.b (mon.1) 412 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:38.436 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:37.155717+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:38.437 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:37.446832+0000 osd.6 (osd.6) 208 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:38:38.437 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:37.455271+0000 osd.6 (osd.6) 209 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:38:38.437 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:37.706708+0000 mon.a (mon.0) 2827 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-23T00:38:38.437 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:38.075271+0000 mon.a (mon.0) 2828 : audit [DBG] from='client.? 172.21.15.87:0/2215659426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:38.482 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:38.507 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:36.709292+0000 osd.4 (osd.4) 238 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:36.709434+0000 osd.4 (osd.4) 239 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:36.852731+0000 osd.5 (osd.5) 199 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:36.853578+0000 osd.5 (osd.5) 200 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:37.147148+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:37.147235+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e500: 8 total, 7 up, 7 in 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:37.148920+0000 mon.b (mon.1) 412 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:37.155717+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:37.446832+0000 osd.6 (osd.6) 208 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:37.455271+0000 osd.6 (osd.6) 209 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:37.706708+0000 mon.a (mon.0) 2827 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-23T00:38:38.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:38.075271+0000 mon.a (mon.0) 2828 : audit [DBG] from='client.? 172.21.15.87:0/2215659426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:36.709292+0000 osd.4 (osd.4) 238 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:36.709434+0000 osd.4 (osd.4) 239 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:36.852731+0000 osd.5 (osd.5) 199 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:36.853578+0000 osd.5 (osd.5) 200 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:37.147148+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:37.147235+0000 mon.a (mon.0) 2825 : cluster [DBG] osdmap e500: 8 total, 7 up, 7 in 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:37.148920+0000 mon.b (mon.1) 412 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:37.155717+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:37.446832+0000 osd.6 (osd.6) 208 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:37.455271+0000 osd.6 (osd.6) 209 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:37.706708+0000 mon.a (mon.0) 2827 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]: dispatch 2024-02-23T00:38:38.509 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:38.075271+0000 mon.a (mon.0) 2828 : audit [DBG] from='client.? 172.21.15.87:0/2215659426' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:38.510 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:38.528 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:38.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:38.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:38.659 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:38.713 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:38.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:38.817 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:38.846 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:38.883 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:38.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:38.993 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:39.021 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:39.103 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-23T00:38:39.166 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:37.705722+0000 mgr.y (mgr.24437) 473 : cluster [DBG] pgmap v766: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 64 active+undersized, 150 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 122/1224 objects degraded (9.967%); 86/1224 objects misplaced (7.026%) 2024-02-23T00:38:39.166 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:37.880793+0000 osd.5 (osd.5) 201 : cluster [DBG] 5.f scrub starts 2024-02-23T00:38:39.166 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:37.881614+0000 osd.5 (osd.5) 202 : cluster [DBG] 5.f scrub ok 2024-02-23T00:38:39.166 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:37.978789+0000 osd.7 (osd.7) 249 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:38:39.166 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:37.979781+0000 osd.7 (osd.7) 250 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:38:39.166 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:38.157706+0000 mon.a (mon.0) 2829 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:39.167 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:38.157791+0000 mon.a (mon.0) 2830 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-23T00:38:39.167 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:38.157865+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e501: 8 total, 7 up, 7 in 2024-02-23T00:38:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:38:39.342 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:38:39.368 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:39.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:37.705722+0000 mgr.y (mgr.24437) 473 : cluster [DBG] pgmap v766: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 64 active+undersized, 150 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 122/1224 objects degraded (9.967%); 86/1224 objects misplaced (7.026%) 2024-02-23T00:38:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:37.880793+0000 osd.5 (osd.5) 201 : cluster [DBG] 5.f scrub starts 2024-02-23T00:38:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:37.881614+0000 osd.5 (osd.5) 202 : cluster [DBG] 5.f scrub ok 2024-02-23T00:38:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:37.978789+0000 osd.7 (osd.7) 249 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:38:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:37.979781+0000 osd.7 (osd.7) 250 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:38:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:38.157706+0000 mon.a (mon.0) 2829 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:38.157791+0000 mon.a (mon.0) 2830 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-23T00:38:39.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:38.157865+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e501: 8 total, 7 up, 7 in 2024-02-23T00:38:39.451 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:39.472 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:39.496 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:37.705722+0000 mgr.y (mgr.24437) 473 : cluster [DBG] pgmap v766: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 64 active+undersized, 150 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 122/1224 objects degraded (9.967%); 86/1224 objects misplaced (7.026%) 2024-02-23T00:38:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:37.880793+0000 osd.5 (osd.5) 201 : cluster [DBG] 5.f scrub starts 2024-02-23T00:38:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:37.881614+0000 osd.5 (osd.5) 202 : cluster [DBG] 5.f scrub ok 2024-02-23T00:38:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:37.978789+0000 osd.7 (osd.7) 249 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:38:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:37.979781+0000 osd.7 (osd.7) 250 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:38:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:38.157706+0000 mon.a (mon.0) 2829 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:38.157791+0000 mon.a (mon.0) 2830 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "22"}]': finished 2024-02-23T00:38:39.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:38.157865+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e501: 8 total, 7 up, 7 in 2024-02-23T00:38:39.498 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:39.571 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:39.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:39.674 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:39.701 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:39.728 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:39.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:39.834 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:39.861 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:39.958 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:39.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:39.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:40.062 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:40.089 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:40.177 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.156219+0000 mon.a (mon.0) 2832 : cluster [DBG] osdmap e502: 8 total, 7 up, 7 in 2024-02-23T00:38:40.177 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.202767+0000 osd.1 (osd.1) 233 : cluster [DBG] 63.e scrub starts 2024-02-23T00:38:40.177 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.206365+0000 osd.1 (osd.1) 234 : cluster [DBG] 63.e scrub ok 2024-02-23T00:38:40.177 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.239833+0000 osd.2 (osd.2) 191 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:38:40.178 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.248191+0000 osd.2 (osd.2) 192 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:38:40.178 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:39.652558+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:40.180 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.156219+0000 mon.a (mon.0) 2832 : cluster [DBG] osdmap e502: 8 total, 7 up, 7 in 2024-02-23T00:38:40.180 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.202767+0000 osd.1 (osd.1) 233 : cluster [DBG] 63.e scrub starts 2024-02-23T00:38:40.180 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.206365+0000 osd.1 (osd.1) 234 : cluster [DBG] 63.e scrub ok 2024-02-23T00:38:40.180 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.239833+0000 osd.2 (osd.2) 191 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:38:40.180 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.248191+0000 osd.2 (osd.2) 192 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:38:40.180 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:39.652558+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:40.338 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:40.341 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.156219+0000 mon.a (mon.0) 2832 : cluster [DBG] osdmap e502: 8 total, 7 up, 7 in 2024-02-23T00:38:40.341 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.202767+0000 osd.1 (osd.1) 233 : cluster [DBG] 63.e scrub starts 2024-02-23T00:38:40.341 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.206365+0000 osd.1 (osd.1) 234 : cluster [DBG] 63.e scrub ok 2024-02-23T00:38:40.341 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.239833+0000 osd.2 (osd.2) 191 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:38:40.341 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.248191+0000 osd.2 (osd.2) 192 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:38:40.341 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:39.652558+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:40.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:40.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:40.444 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:40.470 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:40.572 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:40.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:40.676 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:40.702 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:40.850 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:40.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:40.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:40.956 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:40.984 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:41.056 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:41.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:41.160 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:41.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.125+0000 7f7df29d5700 1 -- 172.21.15.87:0/1535799816 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0f7400 msgr2=0x7f7dec103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:41.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.125+0000 7f7df29d5700 1 --2- 172.21.15.87:0/1535799816 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0f7400 0x7f7dec103a20 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f7de4009be0 tx=0x7f7de4004d10).stop 2024-02-23T00:38:41.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.125+0000 7f7df29d5700 1 -- 172.21.15.87:0/1535799816 shutdown_connections 2024-02-23T00:38:41.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.125+0000 7f7df29d5700 1 --2- 172.21.15.87:0/1535799816 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0f7400 0x7f7dec103a20 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:41.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.125+0000 7f7df29d5700 1 --2- 172.21.15.87:0/1535799816 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dec0fcbb0 0x7f7dec0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.125+0000 7f7df29d5700 1 --2- 172.21.15.87:0/1535799816 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7dec0fd510 0x7f7dec0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.125+0000 7f7df29d5700 1 -- 172.21.15.87:0/1535799816 >> 172.21.15.87:0/1535799816 conn(0x7f7dec0f2820 msgr2=0x7f7dec0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.125+0000 7f7df29d5700 1 -- 172.21.15.87:0/1535799816 shutdown_connections 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.125+0000 7f7df29d5700 1 -- 172.21.15.87:0/1535799816 wait complete. 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 librados: starting msgr at 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 librados: starting objecter 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 Processor -- start 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 -- start start 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 librados: setting wanted keys 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 librados: calling monclient init 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dec0f7400 0x7f7dec1090a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0fcbb0 0x7f7dec109590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7dec0fd510 0x7f7dec109ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:41.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec114230 con 0x7f7dec0fd510 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec1143a0 con 0x7f7dec0f7400 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df29d5700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec114560 con 0x7f7dec0fcbb0 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7deb7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0fcbb0 0x7f7dec109590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df0f72700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7dec0fd510 0x7f7dec109ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7deb7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0fcbb0 0x7f7dec109590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:55376/0 (socket says 172.21.15.87:55376) 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7df0f72700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7dec0fd510 0x7f7dec109ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45208/0 (socket says 172.21.15.87:45208) 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7deb7fe700 1 -- 172.21.15.87:0/2662234462 learned_addr learned my addr 172.21.15.87:0/2662234462 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.126+0000 7f7debfff700 1 --2- 172.21.15.87:0/2662234462 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dec0f7400 0x7f7dec1090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df0f72700 1 -- 172.21.15.87:0/2662234462 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0fcbb0 msgr2=0x7f7dec109590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df0f72700 1 --2- 172.21.15.87:0/2662234462 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0fcbb0 0x7f7dec109590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df0f72700 1 -- 172.21.15.87:0/2662234462 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dec0f7400 msgr2=0x7f7dec1090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df0f72700 1 --2- 172.21.15.87:0/2662234462 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dec0f7400 0x7f7dec1090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df0f72700 1 -- 172.21.15.87:0/2662234462 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de40097c0 con 0x7f7dec0fd510 2024-02-23T00:38:41.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7deb7fe700 1 --2- 172.21.15.87:0/2662234462 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0fcbb0 0x7f7dec109590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df0f72700 1 --2- 172.21.15.87:0/2662234462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7dec0fd510 0x7f7dec109ac0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 rx=0x7f7de400b6d0 tx=0x7f7de4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7de97fa700 10 client.?.objecter ms_handle_connect 0x7f7dec0fd510 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7de97fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7de97fa700 1 -- 172.21.15.87:0/2662234462 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de400e040 con 0x7f7dec0fd510 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dec1147e0 con 0x7f7dec0fd510 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7de97fa700 1 -- 172.21.15.87:0/2662234462 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7de4018040 con 0x7f7dec0fd510 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7de97fa700 1 -- 172.21.15.87:0/2662234462 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de401c3f0 con 0x7f7dec0fd510 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df29d5700 10 client.27432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dec114be0 con 0x7f7dec0fd510 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.127+0000 7f7df29d5700 1 librados: init done 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.128+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dec05fa60 con 0x7f7dec0fd510 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.128+0000 7f7de97fa700 1 -- 172.21.15.87:0/2662234462 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7de4013520 con 0x7f7dec0fd510 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.129+0000 7f7de97fa700 1 --2- 172.21.15.87:0/2662234462 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7dd4062c90 0x7f7dd4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:41.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.129+0000 7f7de97fa700 1 -- 172.21.15.87:0/2662234462 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(503..503 src has 173..503) v4 ==== 8572+0+0 (secure 0 0 0) 0x7f7de407dfd0 con 0x7f7dec0fd510 2024-02-23T00:38:41.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.129+0000 7f7de97fa700 10 client.27432.objecter ms_dispatch 0x7f7dec001550 osd_map(503..503 src has 173..503) v4 2024-02-23T00:38:41.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.129+0000 7f7de97fa700 3 client.27432.objecter handle_osd_map got epochs [503,503] > 0 2024-02-23T00:38:41.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.129+0000 7f7de97fa700 3 client.27432.objecter handle_osd_map decoding full epoch 503 2024-02-23T00:38:41.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.129+0000 7f7debfff700 1 --2- 172.21.15.87:0/2662234462 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7dd4062c90 0x7f7dd4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:41.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.129+0000 7f7de97fa700 20 client.27432.objecter dump_active .. 0 homeless 2024-02-23T00:38:41.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.129+0000 7f7de97fa700 10 client.27432.objecter ms_handle_connect 0x7f7dd4062c90 2024-02-23T00:38:41.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.129+0000 7f7debfff700 1 --2- 172.21.15.87:0/2662234462 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7dd4062c90 0x7f7dd4065150 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7f7ddc004dc0 tx=0x7f7ddc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:41.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.133+0000 7f7de97fa700 1 -- 172.21.15.87:0/2662234462 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7de40512d0 con 0x7f7dec0fd510 2024-02-23T00:38:41.188 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:41.313 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:41.321 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.500677+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.500807+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.706568+0000 mgr.y (mgr.24437) 474 : cluster [DBG] pgmap v769: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 64 active+undersized, 150 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 122/1224 objects degraded (9.967%); 86/1224 objects misplaced (7.026%) 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.892499+0000 osd.5 (osd.5) 205 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.893312+0000 osd.5 (osd.5) 206 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.954381+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:39.957268+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:40.166764+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e503: 8 total, 7 up, 7 in 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:40.264390+0000 osd.2 (osd.2) 193 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:40.265379+0000 osd.2 (osd.2) 194 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:40.271617+0000 mon.b (mon.1) 413 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:40.273572+0000 mon.a (mon.0) 2835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:41.322 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:40.274281+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-81"}]: dispatch 2024-02-23T00:38:41.323 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:40.274412+0000 mon.a (mon.0) 2836 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-81"}]: dispatch 2024-02-23T00:38:41.323 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:40.420889+0000 osd.6 (osd.6) 210 : cluster [DBG] 4.a scrub starts 2024-02-23T00:38:41.323 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:40.424486+0000 osd.6 (osd.6) 211 : cluster [DBG] 4.a scrub ok 2024-02-23T00:38:41.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:38:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:38:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:41.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:41.342 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:38:41.407 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:41.404+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f7dec05ee80 con 0x7f7dec0fd510 2024-02-23T00:38:41.425 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:41.452 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:41.477 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:41.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:41.581 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.500677+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.500807+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.706568+0000 mgr.y (mgr.24437) 474 : cluster [DBG] pgmap v769: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 64 active+undersized, 150 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 122/1224 objects degraded (9.967%); 86/1224 objects misplaced (7.026%) 2024-02-23T00:38:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.892499+0000 osd.5 (osd.5) 205 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:38:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.893312+0000 osd.5 (osd.5) 206 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:38:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.954381+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:38:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:39.957268+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:38:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:40.166764+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e503: 8 total, 7 up, 7 in 2024-02-23T00:38:41.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:40.264390+0000 osd.2 (osd.2) 193 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:40.265379+0000 osd.2 (osd.2) 194 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:40.271617+0000 mon.b (mon.1) 413 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:40.273572+0000 mon.a (mon.0) 2835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:40.274281+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-81"}]: dispatch 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:40.274412+0000 mon.a (mon.0) 2836 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-81"}]: dispatch 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:40.420889+0000 osd.6 (osd.6) 210 : cluster [DBG] 4.a scrub starts 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:40.424486+0000 osd.6 (osd.6) 211 : cluster [DBG] 4.a scrub ok 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.500677+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.500807+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.706568+0000 mgr.y (mgr.24437) 474 : cluster [DBG] pgmap v769: 248 pgs: 3 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 64 active+undersized, 150 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 122/1224 objects degraded (9.967%); 86/1224 objects misplaced (7.026%) 2024-02-23T00:38:41.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.892499+0000 osd.5 (osd.5) 205 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.893312+0000 osd.5 (osd.5) 206 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.954381+0000 osd.7 (osd.7) 251 : cluster [DBG] 6.18 scrub starts 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:39.957268+0000 osd.7 (osd.7) 252 : cluster [DBG] 6.18 scrub ok 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:40.166764+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e503: 8 total, 7 up, 7 in 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:40.264390+0000 osd.2 (osd.2) 193 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:40.265379+0000 osd.2 (osd.2) 194 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:40.271617+0000 mon.b (mon.1) 413 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:40.273572+0000 mon.a (mon.0) 2835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:40.274281+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-81"}]: dispatch 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:40.274412+0000 mon.a (mon.0) 2836 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-81"}]: dispatch 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:40.420889+0000 osd.6 (osd.6) 210 : cluster [DBG] 4.a scrub starts 2024-02-23T00:38:41.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:40.424486+0000 osd.6 (osd.6) 211 : cluster [DBG] 4.a scrub ok 2024-02-23T00:38:41.609 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:41.616 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:38:41.616 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:38:41.623 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:41.629 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:41.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:41.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:41.732 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:41.760 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:42.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.141+0000 7f7de97fa700 1 -- 172.21.15.87:0/2662234462 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v505) v1 ==== 101+0+0 (secure 0 0 0) 0x7f7de4021030 con 0x7f7dec0fd510 2024-02-23T00:38:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 10 librados: watch_flush enter 2024-02-23T00:38:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 10 librados: watch_flush exit 2024-02-23T00:38:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 20 client.27432.objecter shutdown clearing up homeless session... 2024-02-23T00:38:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 10 client.27432.objecter successfully canceled tick 2024-02-23T00:38:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7dd4062c90 msgr2=0x7f7dd4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 --2- 172.21.15.87:0/2662234462 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7dd4062c90 0x7f7dd4065150 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 rx=0x7f7ddc004dc0 tx=0x7f7ddc00b040).stop 2024-02-23T00:38:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7dec0fd510 msgr2=0x7f7dec109ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:42.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 --2- 172.21.15.87:0/2662234462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7dec0fd510 0x7f7dec109ac0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 rx=0x7f7de400b6d0 tx=0x7f7de4004d10).stop 2024-02-23T00:38:42.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 shutdown_connections 2024-02-23T00:38:42.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 --2- 172.21.15.87:0/2662234462 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7dd4062c90 0x7f7dd4065150 secure :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 rx=0x7f7ddc004dc0 tx=0x7f7ddc00b040).stop 2024-02-23T00:38:42.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 --2- 172.21.15.87:0/2662234462 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7dec0fd510 0x7f7dec109ac0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:42.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 --2- 172.21.15.87:0/2662234462 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7dec0f7400 0x7f7dec1090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:42.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 --2- 172.21.15.87:0/2662234462 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7dec0fcbb0 0x7f7dec109590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:42.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.144+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 >> 172.21.15.87:0/2662234462 conn(0x7f7dec0f2820 msgr2=0x7f7dec0f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:42.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.147+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 shutdown_connections 2024-02-23T00:38:42.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.149+0000 7f7df29d5700 1 -- 172.21.15.87:0/2662234462 wait complete. 2024-02-23T00:38:42.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:42.150+0000 7f7df29d5700 1 librados: shutdown 2024-02-23T00:38:42.153 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:38:42.312 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:42.315 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:40.857063+0000 osd.5 (osd.5) 207 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:38:42.315 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:40.858795+0000 osd.5 (osd.5) 208 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:38:42.315 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:41.033310+0000 mgr.y (mgr.24437) 475 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:42.315 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:41.174774+0000 osd.4 (osd.4) 240 : cluster [DBG] 6.16 scrub starts 2024-02-23T00:38:42.315 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:41.177670+0000 osd.4 (osd.4) 241 : cluster [DBG] 6.16 scrub ok 2024-02-23T00:38:42.315 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:41.190673+0000 mon.a (mon.0) 2837 : cluster [DBG] osdmap e504: 8 total, 7 up, 7 in 2024-02-23T00:38:42.315 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:41.489976+0000 mon.a (mon.0) 2838 : audit [INF] from='client.? 172.21.15.87:0/2662234462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:38:42.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:41.708335+0000 mon.a (mon.0) 2839 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-23T00:38:42.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.131391+0000 mon.a (mon.0) 2840 : cluster [WRN] Health check update: Degraded data redundancy: 122/1224 objects degraded (9.967%), 28 pgs degraded, 75 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:42.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.132325+0000 mon.a (mon.0) 2841 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:38:42.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:42.138575+0000 mon.a (mon.0) 2842 : audit [INF] from='client.? 172.21.15.87:0/2662234462' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:38:42.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:42.138638+0000 mon.a (mon.0) 2843 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-23T00:38:42.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.138685+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e505: 8 total, 7 up, 7 in 2024-02-23T00:38:42.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:42.143400+0000 mon.b (mon.1) 415 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:42.316 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:42.143647+0000 mon.a (mon.0) 2845 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:42.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:42.338 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-23T00:38:42.418 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:42.444 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:42] "GET /metrics HTTP/1.1" 200 188956 "" "Prometheus/2.33.4" 2024-02-23T00:38:42.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:40.857063+0000 osd.5 (osd.5) 207 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:38:42.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:40.858795+0000 osd.5 (osd.5) 208 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:38:42.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:41.033310+0000 mgr.y (mgr.24437) 475 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:42.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:41.174774+0000 osd.4 (osd.4) 240 : cluster [DBG] 6.16 scrub starts 2024-02-23T00:38:42.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:41.177670+0000 osd.4 (osd.4) 241 : cluster [DBG] 6.16 scrub ok 2024-02-23T00:38:42.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:41.190673+0000 mon.a (mon.0) 2837 : cluster [DBG] osdmap e504: 8 total, 7 up, 7 in 2024-02-23T00:38:42.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:41.489976+0000 mon.a (mon.0) 2838 : audit [INF] from='client.? 172.21.15.87:0/2662234462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:38:42.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:41.708335+0000 mon.a (mon.0) 2839 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-23T00:38:42.445 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.131391+0000 mon.a (mon.0) 2840 : cluster [WRN] Health check update: Degraded data redundancy: 122/1224 objects degraded (9.967%), 28 pgs degraded, 75 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.132325+0000 mon.a (mon.0) 2841 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:42.138575+0000 mon.a (mon.0) 2842 : audit [INF] from='client.? 172.21.15.87:0/2662234462' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:42.138638+0000 mon.a (mon.0) 2843 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.138685+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e505: 8 total, 7 up, 7 in 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:42.143400+0000 mon.b (mon.1) 415 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:42.143647+0000 mon.a (mon.0) 2845 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:40.857063+0000 osd.5 (osd.5) 207 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:40.858795+0000 osd.5 (osd.5) 208 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:41.033310+0000 mgr.y (mgr.24437) 475 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:42.446 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:41.174774+0000 osd.4 (osd.4) 240 : cluster [DBG] 6.16 scrub starts 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:41.177670+0000 osd.4 (osd.4) 241 : cluster [DBG] 6.16 scrub ok 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:41.190673+0000 mon.a (mon.0) 2837 : cluster [DBG] osdmap e504: 8 total, 7 up, 7 in 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:41.489976+0000 mon.a (mon.0) 2838 : audit [INF] from='client.? 172.21.15.87:0/2662234462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:41.708335+0000 mon.a (mon.0) 2839 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.131391+0000 mon.a (mon.0) 2840 : cluster [WRN] Health check update: Degraded data redundancy: 122/1224 objects degraded (9.967%), 28 pgs degraded, 75 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.132325+0000 mon.a (mon.0) 2841 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:42.138575+0000 mon.a (mon.0) 2842 : audit [INF] from='client.? 172.21.15.87:0/2662234462' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:42.138638+0000 mon.a (mon.0) 2843 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.138685+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e505: 8 total, 7 up, 7 in 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:42.143400+0000 mon.b (mon.1) 415 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:42.447 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:42.143647+0000 mon.a (mon.0) 2845 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:38:42.448 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:42.500 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:42.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:42.604 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:42.631 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:42.647 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:42.652 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:42.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:42.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:42.783 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:42.887 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:38:42.887 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:38:42.887 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-02-23T00:38:42.888 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd in 3 2024-02-23T00:38:43.031 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:43.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:43.135 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:43.229 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:43.638 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:43.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:41.707233+0000 mgr.y (mgr.24437) 476 : cluster [DBG] pgmap v772: 216 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 50 active+undersized, 129 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%); 23 B/s, 0 objects/s recovering 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.185455+0000 osd.1 (osd.1) 235 : cluster [DBG] 5.25 deep-scrub starts 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.191215+0000 osd.1 (osd.1) 236 : cluster [DBG] 5.25 deep-scrub ok 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.292810+0000 osd.2 (osd.2) 195 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.293817+0000 osd.2 (osd.2) 196 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.428742+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.11 deep-scrub starts 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.431634+0000 osd.6 (osd.6) 213 : cluster [DBG] 2.11 deep-scrub ok 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:43.066041+0000 mon.a (mon.0) 2846 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:43.138715+0000 mon.a (mon.0) 2847 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:43.697 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:43.138806+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e506: 8 total, 7 up, 7 in 2024-02-23T00:38:43.698 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:43.146056+0000 mon.b (mon.1) 416 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:43.748 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:43.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:41.707233+0000 mgr.y (mgr.24437) 476 : cluster [DBG] pgmap v772: 216 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 50 active+undersized, 129 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%); 23 B/s, 0 objects/s recovering 2024-02-23T00:38:43.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.185455+0000 osd.1 (osd.1) 235 : cluster [DBG] 5.25 deep-scrub starts 2024-02-23T00:38:43.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.191215+0000 osd.1 (osd.1) 236 : cluster [DBG] 5.25 deep-scrub ok 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.292810+0000 osd.2 (osd.2) 195 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.293817+0000 osd.2 (osd.2) 196 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.428742+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.11 deep-scrub starts 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.431634+0000 osd.6 (osd.6) 213 : cluster [DBG] 2.11 deep-scrub ok 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:43.066041+0000 mon.a (mon.0) 2846 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:43.138715+0000 mon.a (mon.0) 2847 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:43.138806+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e506: 8 total, 7 up, 7 in 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:43.146056+0000 mon.b (mon.1) 416 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:41.707233+0000 mgr.y (mgr.24437) 476 : cluster [DBG] pgmap v772: 216 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 50 active+undersized, 129 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%); 23 B/s, 0 objects/s recovering 2024-02-23T00:38:43.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.185455+0000 osd.1 (osd.1) 235 : cluster [DBG] 5.25 deep-scrub starts 2024-02-23T00:38:43.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.191215+0000 osd.1 (osd.1) 236 : cluster [DBG] 5.25 deep-scrub ok 2024-02-23T00:38:43.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.292810+0000 osd.2 (osd.2) 195 : cluster [DBG] 6.14 scrub starts 2024-02-23T00:38:43.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.293817+0000 osd.2 (osd.2) 196 : cluster [DBG] 6.14 scrub ok 2024-02-23T00:38:43.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.428742+0000 osd.6 (osd.6) 212 : cluster [DBG] 2.11 deep-scrub starts 2024-02-23T00:38:43.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.431634+0000 osd.6 (osd.6) 213 : cluster [DBG] 2.11 deep-scrub ok 2024-02-23T00:38:43.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:43.066041+0000 mon.a (mon.0) 2846 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:38:43.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:43.138715+0000 mon.a (mon.0) 2847 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:38:43.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:43.138806+0000 mon.a (mon.0) 2848 : cluster [DBG] osdmap e506: 8 total, 7 up, 7 in 2024-02-23T00:38:43.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:43.146056+0000 mon.b (mon.1) 416 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:38:43.780 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:44.141 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:38:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:38:44.224 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:44.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:44.333 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:44.369 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:44.444 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:38:44.444 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.558+0000 7f3f91070700 1 -- 172.21.15.87:0/2076106213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f8c0f7300 msgr2=0x7f3f8c1040a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.558+0000 7f3f91070700 1 --2- 172.21.15.87:0/2076106213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f8c0f7300 0x7f3f8c1040a0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 rx=0x7f3f800048a0 tx=0x7f3f80009cb0).stop 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.558+0000 7f3f91070700 1 -- 172.21.15.87:0/2076106213 shutdown_connections 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.558+0000 7f3f91070700 1 --2- 172.21.15.87:0/2076106213 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f8c104590 0x7f3f8c106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.558+0000 7f3f91070700 1 --2- 172.21.15.87:0/2076106213 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f8c0f6a30 0x7f3f8c0f6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.558+0000 7f3f91070700 1 --2- 172.21.15.87:0/2076106213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f8c0f7300 0x7f3f8c1040a0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.558+0000 7f3f91070700 1 -- 172.21.15.87:0/2076106213 >> 172.21.15.87:0/2076106213 conn(0x7f3f8c0f26f0 msgr2=0x7f3f8c0f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.558+0000 7f3f91070700 1 -- 172.21.15.87:0/2076106213 shutdown_connections 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.559+0000 7f3f91070700 1 -- 172.21.15.87:0/2076106213 wait complete. 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.559+0000 7f3f91070700 1 librados: starting msgr at 2024-02-23T00:38:44.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.559+0000 7f3f91070700 1 librados: starting objecter 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.559+0000 7f3f91070700 1 Processor -- start 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.559+0000 7f3f91070700 1 -- start start 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.559+0000 7f3f91070700 1 librados: setting wanted keys 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.559+0000 7f3f91070700 1 librados: calling monclient init 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f8c0f6a30 0x7f3f8c110240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f8c0f7300 0x7f3f8c110730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f8c104590 0x7f3f8c10a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c10a8d0 con 0x7f3f8c0f7300 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c10aa40 con 0x7f3f8c104590 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c10ac00 con 0x7f3f8c0f6a30 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f8c0f7300 0x7f3f8c110730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8ad9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f8c0f6a30 0x7f3f8c110240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8a59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f8c0f7300 0x7f3f8c110730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54764/0 (socket says 172.21.15.87:54764) 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8ad9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f8c0f6a30 0x7f3f8c110240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:41920/0 (socket says 172.21.15.87:41920) 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8a59c700 1 -- 172.21.15.87:0/1017401822 learned_addr learned my addr 172.21.15.87:0/1017401822 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8b59e700 1 --2- 172.21.15.87:0/1017401822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f8c104590 0x7f3f8c10a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8a59c700 1 -- 172.21.15.87:0/1017401822 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f8c0f6a30 msgr2=0x7f3f8c110240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8a59c700 1 --2- 172.21.15.87:0/1017401822 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f8c0f6a30 0x7f3f8c110240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8a59c700 1 -- 172.21.15.87:0/1017401822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f8c104590 msgr2=0x7f3f8c10a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8a59c700 1 --2- 172.21.15.87:0/1017401822 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3f8c104590 0x7f3f8c10a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8a59c700 1 -- 172.21.15.87:0/1017401822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f840097c0 con 0x7f3f8c0f7300 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8ad9d700 1 --2- 172.21.15.87:0/1017401822 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3f8c0f6a30 0x7f3f8c110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f8a59c700 1 --2- 172.21.15.87:0/1017401822 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3f8c0f7300 0x7f3f8c110730 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 rx=0x7f3f80004710 tx=0x7f3f80007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:44.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f73fff700 10 client.?.objecter ms_handle_connect 0x7f3f8c0f7300 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f73fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f8000e070 con 0x7f3f8c0f7300 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f80005040 con 0x7f3f8c0f7300 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f800092e0 con 0x7f3f8c0f7300 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f80018070 con 0x7f3f8c0f7300 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 10 client.27438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c10b120 con 0x7f3f8c0f7300 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.560+0000 7f3f91070700 1 librados: init done 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.561+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3f8c0606d0 con 0x7f3f8c0f7300 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.562+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3f8001d420 con 0x7f3f8c0f7300 2024-02-23T00:38:44.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.562+0000 7f3f73fff700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3f74062c90 0x7f3f74065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.562+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(507..507 src has 173..507) v4 ==== 8888+0+0 (secure 0 0 0) 0x7f3f8007e490 con 0x7f3f8c0f7300 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.562+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 osd_map(507..507 src has 173..507) v4 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.562+0000 7f3f73fff700 3 client.27438.objecter handle_osd_map got epochs [507,507] > 0 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.562+0000 7f3f73fff700 3 client.27438.objecter handle_osd_map decoding full epoch 507 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.562+0000 7f3f8ad9d700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3f74062c90 0x7f3f74065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.562+0000 7f3f73fff700 20 client.27438.objecter dump_active .. 0 homeless 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.562+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v507) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3f8007e930 con 0x7f3f8c0f7300 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f73fff700 10 client.27438.objecter ms_handle_connect 0x7f3f74062c90 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f8ad9d700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3f74062c90 0x7f3f74065150 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 rx=0x7f3f7c0103c0 tx=0x7f3f7c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 10 client.27438.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:44.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 1 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 20 client.27438.objecter _get_session osd=-1 returning homeless 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 10 client.27438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=508}) v3 -- 0x7f3f8c05ee80 con 0x7f3f8c0f7300 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f8c109330 con 0x7f3f8c0f7300 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_get_version_reply(handle=1 version=507) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f80051620 con 0x7f3f8c0f7300 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f8959a700 10 client.27438.objecter _check_command_map_dne tid 1 current 507 map_dne_bound 507 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f8959a700 10 client.27438.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f8959a700 15 client.27438.objecter _session_command_op_remove -1 1 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 10 client.27438.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 2 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6805/685354171 conn(0x7f3f8c106030 0x7f3f8c0273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c105ec0 osd=1 3 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c105ec0 osd=1 3 2024-02-23T00:38:44.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c105ec0 osd=1 3 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 2 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c105ec0 osd=1 3 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 1 2 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c105ec0 osd=1 4 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 10 client.27438.objecter _send_command 2 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f8c059e00 con 0x7f3f8c106030 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f8b59e700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6805/685354171 conn(0x7f3f8c106030 0x7f3f8c0273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f73fff700 10 client.27438.objecter ms_handle_connect 0x7f3f8c106030 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.563+0000 7f3f8b59e700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6805/685354171 conn(0x7f3f8c106030 0x7f3f8c0273c0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.564+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3f8c059e00 con 0x7f3f8c106030 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.564+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.564+0000 7f3f73fff700 10 client.27438.objecter _finish_command 2 = system:0 2024-02-23T00:38:44.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.564+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c105ec0 osd=1 4 2024-02-23T00:38:44.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.564+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 1 2 2024-02-23T00:38:44.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 10 client.27438.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 3 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c105ec0 osd=1 3 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c105ec0 osd=1 3 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c105ec0 osd=1 3 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 3 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c105ec0 osd=1 3 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 1 3 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c105ec0 osd=1 4 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 10 client.27438.objecter _send_command 3 2024-02-23T00:38:44.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.579+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3f8c0278b0 con 0x7f3f8c106030 2024-02-23T00:38:44.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3f8c0278b0 con 0x7f3f8c106030 2024-02-23T00:38:44.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:44.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f73fff700 10 client.27438.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c105ec0 osd=1 4 2024-02-23T00:38:44.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 1 3 2024-02-23T00:38:44.584 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 10 client.27438.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 4 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6809/3605394228 conn(0x7f3f8c0594d0 0x7f3f8c11b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c059360 osd=2 3 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c059360 osd=2 3 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c059360 osd=2 3 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 4 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c059360 osd=2 3 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 2 4 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c059360 osd=2 4 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 10 client.27438.objecter _send_command 4 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f8c028790 con 0x7f3f8c0594d0 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.581+0000 7f3f8ad9d700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6809/3605394228 conn(0x7f3f8c0594d0 0x7f3f8c11b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.582+0000 7f3f73fff700 10 client.27438.objecter ms_handle_connect 0x7f3f8c0594d0 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.582+0000 7f3f8ad9d700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6809/3605394228 conn(0x7f3f8c0594d0 0x7f3f8c11b090 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:44.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.582+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3f8c028790 con 0x7f3f8c0594d0 2024-02-23T00:38:44.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.582+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:38:44.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.582+0000 7f3f73fff700 10 client.27438.objecter _finish_command 4 = system:0 2024-02-23T00:38:44.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.582+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c059360 osd=2 4 2024-02-23T00:38:44.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.582+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 2 4 2024-02-23T00:38:44.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.597+0000 7f3f91070700 10 client.27438.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:44.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.597+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 5 2024-02-23T00:38:44.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.598+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c059360 osd=2 3 2024-02-23T00:38:44.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.598+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c059360 osd=2 3 2024-02-23T00:38:44.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.598+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c059360 osd=2 3 2024-02-23T00:38:44.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.598+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 5 2024-02-23T00:38:44.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.598+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c059360 osd=2 3 2024-02-23T00:38:44.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.598+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 2 5 2024-02-23T00:38:44.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.598+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c059360 osd=2 4 2024-02-23T00:38:44.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.598+0000 7f3f91070700 10 client.27438.objecter _send_command 5 2024-02-23T00:38:44.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.598+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3f8c027cf0 con 0x7f3f8c0594d0 2024-02-23T00:38:44.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3f8c027cf0 con 0x7f3f8c0594d0 2024-02-23T00:38:44.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:44.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f73fff700 10 client.27438.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c059360 osd=2 4 2024-02-23T00:38:44.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 2 5 2024-02-23T00:38:44.603 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:38:44.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 10 client.27438.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:44.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 6 2024-02-23T00:38:44.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6813/128059790 conn(0x7f3f8c11c700 0x7f3f8c11eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c0fe830 osd=3 3 2024-02-23T00:38:44.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c0fe830 osd=3 3 2024-02-23T00:38:44.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c0fe830 osd=3 3 2024-02-23T00:38:44.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 6 2024-02-23T00:38:44.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c0fe830 osd=3 3 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 3 6 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c0fe830 osd=3 4 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 10 client.27438.objecter _send_command 6 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f8c0ff040 con 0x7f3f8c11c700 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f8a59c700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6813/128059790 conn(0x7f3f8c11c700 0x7f3f8c11eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f8a59c700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.87:6813/128059790 conn(0x7f3f8c11c700 0x7f3f8c11eaf0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.600+0000 7f3f73fff700 10 client.27438.objecter ms_handle_connect 0x7f3f8c11c700 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.601+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3f8c0ff040 con 0x7f3f8c11c700 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.601+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.601+0000 7f3f73fff700 10 client.27438.objecter _finish_command 6 = system:0 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.601+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c0fe830 osd=3 4 2024-02-23T00:38:44.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.601+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 3 6 2024-02-23T00:38:44.611 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:44.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 10 client.27438.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:44.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 7 2024-02-23T00:38:44.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c0fe830 osd=3 3 2024-02-23T00:38:44.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c0fe830 osd=3 3 2024-02-23T00:38:44.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c0fe830 osd=3 3 2024-02-23T00:38:44.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 7 2024-02-23T00:38:44.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c0fe830 osd=3 3 2024-02-23T00:38:44.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 3 7 2024-02-23T00:38:44.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c0fe830 osd=3 4 2024-02-23T00:38:44.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 10 client.27438.objecter _send_command 7 2024-02-23T00:38:44.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.616+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3f8c0fe360 con 0x7f3f8c11c700 2024-02-23T00:38:44.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3f8c0fe360 con 0x7f3f8c11c700 2024-02-23T00:38:44.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:44.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f73fff700 10 client.27438.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c0fe830 osd=3 4 2024-02-23T00:38:44.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 3 7 2024-02-23T00:38:44.621 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:38:44.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 10 client.27438.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:44.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 8 2024-02-23T00:38:44.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6800/4175844757 conn(0x7f3f8c120160 0x7f3f8c1205a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c11bd50 osd=4 3 2024-02-23T00:38:44.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c11bd50 osd=4 3 2024-02-23T00:38:44.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c11bd50 osd=4 3 2024-02-23T00:38:44.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 8 2024-02-23T00:38:44.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c11bd50 osd=4 3 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 4 8 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c11bd50 osd=4 4 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 10 client.27438.objecter _send_command 8 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f8c11c500 con 0x7f3f8c120160 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.618+0000 7f3f8b59e700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6800/4175844757 conn(0x7f3f8c120160 0x7f3f8c1205a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.619+0000 7f3f73fff700 10 client.27438.objecter ms_handle_connect 0x7f3f8c120160 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.619+0000 7f3f8b59e700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6800/4175844757 conn(0x7f3f8c120160 0x7f3f8c1205a0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.620+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3f8c11c500 con 0x7f3f8c120160 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.620+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.620+0000 7f3f73fff700 10 client.27438.objecter _finish_command 8 = system:0 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.620+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c11bd50 osd=4 4 2024-02-23T00:38:44.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.620+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 4 8 2024-02-23T00:38:44.624 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:44.625 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.786607+0000 osd.5 (osd.5) 209 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:38:44.625 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:42.787566+0000 osd.5 (osd.5) 210 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:38:44.625 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:43.240774+0000 osd.1 (osd.1) 237 : cluster [DBG] 5.27 scrub starts 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:43.246796+0000 osd.1 (osd.1) 238 : cluster [DBG] 5.27 scrub ok 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:43.331902+0000 osd.2 (osd.2) 197 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:43.332900+0000 osd.2 (osd.2) 198 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:43.707901+0000 mgr.y (mgr.24437) 477 : cluster [DBG] pgmap v775: 248 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 32 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 50 active+undersized, 129 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%); 23 B/s, 0 objects/s recovering 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:43.921056+0000 osd.7 (osd.7) 253 : cluster [DBG] 6.19 scrub starts 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:43.922438+0000 osd.7 (osd.7) 254 : cluster [DBG] 6.19 scrub ok 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.140638+0000 mon.a (mon.0) 2849 : cluster [DBG] osdmap e507: 8 total, 7 up, 7 in 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:44.153161+0000 mon.b (mon.1) 417 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:44.157779+0000 mon.a (mon.0) 2850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.159361+0000 osd.4 (osd.4) 242 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.173069+0000 osd.4 (osd.4) 243 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:44.403432+0000 mon.a (mon.0) 2851 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:44.404093+0000 mon.a (mon.0) 2852 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:38:44.626 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:44.408262+0000 mon.a (mon.0) 2853 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:44.627 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.786607+0000 osd.5 (osd.5) 209 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:38:44.627 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:42.787566+0000 osd.5 (osd.5) 210 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:38:44.627 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:43.240774+0000 osd.1 (osd.1) 237 : cluster [DBG] 5.27 scrub starts 2024-02-23T00:38:44.627 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:43.246796+0000 osd.1 (osd.1) 238 : cluster [DBG] 5.27 scrub ok 2024-02-23T00:38:44.627 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:43.331902+0000 osd.2 (osd.2) 197 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:38:44.627 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:43.332900+0000 osd.2 (osd.2) 198 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:38:44.627 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:43.707901+0000 mgr.y (mgr.24437) 477 : cluster [DBG] pgmap v775: 248 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 32 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 50 active+undersized, 129 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%); 23 B/s, 0 objects/s recovering 2024-02-23T00:38:44.627 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:43.921056+0000 osd.7 (osd.7) 253 : cluster [DBG] 6.19 scrub starts 2024-02-23T00:38:44.627 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:43.922438+0000 osd.7 (osd.7) 254 : cluster [DBG] 6.19 scrub ok 2024-02-23T00:38:44.628 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.140638+0000 mon.a (mon.0) 2849 : cluster [DBG] osdmap e507: 8 total, 7 up, 7 in 2024-02-23T00:38:44.628 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:44.153161+0000 mon.b (mon.1) 417 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:44.628 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:44.157779+0000 mon.a (mon.0) 2850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:44.628 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.159361+0000 osd.4 (osd.4) 242 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:38:44.628 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.173069+0000 osd.4 (osd.4) 243 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:38:44.628 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:44.403432+0000 mon.a (mon.0) 2851 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:38:44.628 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:44.404093+0000 mon.a (mon.0) 2852 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:38:44.628 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:44.408262+0000 mon.a (mon.0) 2853 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:44.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:44.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 10 client.27438.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:44.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 9 2024-02-23T00:38:44.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c11bd50 osd=4 3 2024-02-23T00:38:44.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c11bd50 osd=4 3 2024-02-23T00:38:44.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c11bd50 osd=4 3 2024-02-23T00:38:44.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 9 2024-02-23T00:38:44.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c11bd50 osd=4 3 2024-02-23T00:38:44.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 4 9 2024-02-23T00:38:44.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c11bd50 osd=4 4 2024-02-23T00:38:44.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 10 client.27438.objecter _send_command 9 2024-02-23T00:38:44.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.640+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3f8c121290 con 0x7f3f8c120160 2024-02-23T00:38:44.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3f8c121290 con 0x7f3f8c120160 2024-02-23T00:38:44.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:44.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f73fff700 10 client.27438.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c11bd50 osd=4 4 2024-02-23T00:38:44.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 4 9 2024-02-23T00:38:44.645 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:38:44.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 10 client.27438.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:44.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 10 2024-02-23T00:38:44.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3f8c1237d0 0x7f3f8c125bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c121290 osd=5 3 2024-02-23T00:38:44.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c121290 osd=5 3 2024-02-23T00:38:44.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c121290 osd=5 3 2024-02-23T00:38:44.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 10 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c121290 osd=5 3 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 5 10 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c121290 osd=5 4 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 10 client.27438.objecter _send_command 10 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f8c121550 con 0x7f3f8c1237d0 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.642+0000 7f3f8ad9d700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3f8c1237d0 0x7f3f8c125bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.643+0000 7f3f73fff700 10 client.27438.objecter ms_handle_connect 0x7f3f8c1237d0 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.643+0000 7f3f8ad9d700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3f8c1237d0 0x7f3f8c125bc0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.643+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3f8c121550 con 0x7f3f8c1237d0 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.643+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.643+0000 7f3f73fff700 10 client.27438.objecter _finish_command 10 = system:0 2024-02-23T00:38:44.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.643+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c121290 osd=5 4 2024-02-23T00:38:44.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.643+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 5 10 2024-02-23T00:38:44.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 10 client.27438.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 11 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c121290 osd=5 3 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c121290 osd=5 3 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c121290 osd=5 3 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 11 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c121290 osd=5 3 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 5 11 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c121290 osd=5 4 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 10 client.27438.objecter _send_command 11 2024-02-23T00:38:44.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.658+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3f8c121550 con 0x7f3f8c1237d0 2024-02-23T00:38:44.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3f8c121550 con 0x7f3f8c1237d0 2024-02-23T00:38:44.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:44.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f73fff700 10 client.27438.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c121290 osd=5 4 2024-02-23T00:38:44.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 5 11 2024-02-23T00:38:44.663 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:38:44.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 10 client.27438.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:44.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 12 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3f8c126600 0x7f3f8c126a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c121550 osd=6 3 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c121550 osd=6 3 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c121550 osd=6 3 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 12 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c121550 osd=6 3 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 6 12 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c121550 osd=6 4 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 10 client.27438.objecter _send_command 12 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.660+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f8c12ba50 con 0x7f3f8c126600 2024-02-23T00:38:44.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.661+0000 7f3f8a59c700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3f8c126600 0x7f3f8c126a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.661+0000 7f3f73fff700 10 client.27438.objecter ms_handle_connect 0x7f3f8c126600 2024-02-23T00:38:44.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.661+0000 7f3f8a59c700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3f8c126600 0x7f3f8c126a40 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:44.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.662+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3f8c12ba50 con 0x7f3f8c126600 2024-02-23T00:38:44.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.662+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:38:44.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.662+0000 7f3f73fff700 10 client.27438.objecter _finish_command 12 = system:0 2024-02-23T00:38:44.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.662+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c121550 osd=6 4 2024-02-23T00:38:44.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.662+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 6 12 2024-02-23T00:38:44.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 10 client.27438.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:44.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 13 2024-02-23T00:38:44.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c121550 osd=6 3 2024-02-23T00:38:44.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c121550 osd=6 3 2024-02-23T00:38:44.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c121550 osd=6 3 2024-02-23T00:38:44.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 13 2024-02-23T00:38:44.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c121550 osd=6 3 2024-02-23T00:38:44.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 6 13 2024-02-23T00:38:44.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c121550 osd=6 4 2024-02-23T00:38:44.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 10 client.27438.objecter _send_command 13 2024-02-23T00:38:44.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.676+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3f8c127020 con 0x7f3f8c126600 2024-02-23T00:38:44.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3f8c127020 con 0x7f3f8c126600 2024-02-23T00:38:44.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:44.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f73fff700 10 client.27438.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c121550 osd=6 4 2024-02-23T00:38:44.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 6 13 2024-02-23T00:38:44.680 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:38:44.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 10 client.27438.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:44.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 14 2024-02-23T00:38:44.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6812/373787000 conn(0x7f3f8c1276a0 0x7f3f8c127b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:44.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c127020 osd=7 3 2024-02-23T00:38:44.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c127020 osd=7 3 2024-02-23T00:38:44.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c127020 osd=7 3 2024-02-23T00:38:44.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 14 2024-02-23T00:38:44.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c127020 osd=7 3 2024-02-23T00:38:44.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 7 14 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c127020 osd=7 4 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 10 client.27438.objecter _send_command 14 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.677+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f8c128080 con 0x7f3f8c1276a0 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.678+0000 7f3f8b59e700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6812/373787000 conn(0x7f3f8c1276a0 0x7f3f8c127b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.678+0000 7f3f73fff700 10 client.27438.objecter ms_handle_connect 0x7f3f8c1276a0 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.678+0000 7f3f8b59e700 1 --2- 172.21.15.87:0/1017401822 >> v2:172.21.15.171:6812/373787000 conn(0x7f3f8c1276a0 0x7f3f8c127b00 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.679+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3f8c128080 con 0x7f3f8c1276a0 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.679+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.679+0000 7f3f73fff700 10 client.27438.objecter _finish_command 14 = system:0 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.679+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c127020 osd=7 4 2024-02-23T00:38:44.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.679+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 7 14 2024-02-23T00:38:44.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 10 client.27438.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:38:44.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign -1 15 2024-02-23T00:38:44.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c127020 osd=7 3 2024-02-23T00:38:44.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c127020 osd=7 3 2024-02-23T00:38:44.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 20 client.27438.objecter _get_session s=0x7f3f8c127020 osd=7 3 2024-02-23T00:38:44.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 15 client.27438.objecter _session_command_op_remove -1 15 2024-02-23T00:38:44.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 20 client.27438.objecter get_session s=0x7f3f8c127020 osd=7 3 2024-02-23T00:38:44.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 15 client.27438.objecter _session_command_op_assign 7 15 2024-02-23T00:38:44.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 20 client.27438.objecter put_session s=0x7f3f8c127020 osd=7 4 2024-02-23T00:38:44.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 10 client.27438.objecter _send_command 15 2024-02-23T00:38:44.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.693+0000 7f3f91070700 1 -- 172.21.15.87:0/1017401822 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3f8c128080 con 0x7f3f8c1276a0 2024-02-23T00:38:44.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.696+0000 7f3f73fff700 1 -- 172.21.15.87:0/1017401822 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f3f8c128080 con 0x7f3f8c1276a0 2024-02-23T00:38:44.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.696+0000 7f3f73fff700 10 client.27438.objecter ms_dispatch 0x7f3f8c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:38:44.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.696+0000 7f3f73fff700 10 client.27438.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.696+0000 7f3f73fff700 20 client.27438.objecter put_session s=0x7f3f8c127020 osd=7 4 2024-02-23T00:38:44.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:44.696+0000 7f3f73fff700 15 client.27438.objecter _session_command_op_remove 7 15 2024-02-23T00:38:44.698 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:38:44.698 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:38:44.698 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.698 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.698 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.698 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.698 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.698 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.698 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:38:44.730 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:44.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.786607+0000 osd.5 (osd.5) 209 : cluster [DBG] 6.1d scrub starts 2024-02-23T00:38:44.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:42.787566+0000 osd.5 (osd.5) 210 : cluster [DBG] 6.1d scrub ok 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:43.240774+0000 osd.1 (osd.1) 237 : cluster [DBG] 5.27 scrub starts 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:43.246796+0000 osd.1 (osd.1) 238 : cluster [DBG] 5.27 scrub ok 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:43.331902+0000 osd.2 (osd.2) 197 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:43.332900+0000 osd.2 (osd.2) 198 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:43.707901+0000 mgr.y (mgr.24437) 477 : cluster [DBG] pgmap v775: 248 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 32 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 50 active+undersized, 129 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%); 23 B/s, 0 objects/s recovering 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:43.921056+0000 osd.7 (osd.7) 253 : cluster [DBG] 6.19 scrub starts 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:43.922438+0000 osd.7 (osd.7) 254 : cluster [DBG] 6.19 scrub ok 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.140638+0000 mon.a (mon.0) 2849 : cluster [DBG] osdmap e507: 8 total, 7 up, 7 in 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:44.153161+0000 mon.b (mon.1) 417 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:44.157779+0000 mon.a (mon.0) 2850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:38:44.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.159361+0000 osd.4 (osd.4) 242 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:38:44.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.173069+0000 osd.4 (osd.4) 243 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:38:44.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:44.403432+0000 mon.a (mon.0) 2851 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:38:44.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:44.404093+0000 mon.a (mon.0) 2852 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:38:44.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:44.408262+0000 mon.a (mon.0) 2853 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:38:44.761 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:44.859 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:44.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:44.963 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:44.991 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:45.018 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:45.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:45.122 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:45.155 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:45.188 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-23T00:38:45.309 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:45.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:45.421 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:45.449 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:45.469 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:45.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:45.523 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:38:45.574 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.232854+0000 osd.1 (osd.1) 239 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.240428+0000 osd.1 (osd.1) 240 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.333717+0000 osd.2 (osd.2) 199 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.335901+0000 osd.2 (osd.2) 200 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.439845+0000 osd.6 (osd.6) 214 : cluster [DBG] 3.a scrub starts 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.442730+0000 osd.6 (osd.6) 215 : cluster [DBG] 3.a scrub ok 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:44.563283+0000 mon.a (mon.0) 2854 : audit [DBG] from='client.? 172.21.15.87:0/1017401822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.881615+0000 osd.7 (osd.7) 255 : cluster [DBG] 63.18 scrub starts 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:44.885104+0000 osd.7 (osd.7) 256 : cluster [DBG] 63.18 scrub ok 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:45.141964+0000 mon.a (mon.0) 2855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:45.142037+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e508: 8 total, 7 up, 7 in 2024-02-23T00:38:45.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:45.146220+0000 mon.b (mon.1) 418 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_tier","val": "test-rados-api-smithi087-82091-85-test-flush"}]: dispatch 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:45.148043+0000 mon.a (mon.0) 2857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_tier","val": "test-rados-api-smithi087-82091-85-test-flush"}]: dispatch 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.232854+0000 osd.1 (osd.1) 239 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.240428+0000 osd.1 (osd.1) 240 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.333717+0000 osd.2 (osd.2) 199 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.335901+0000 osd.2 (osd.2) 200 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.439845+0000 osd.6 (osd.6) 214 : cluster [DBG] 3.a scrub starts 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.442730+0000 osd.6 (osd.6) 215 : cluster [DBG] 3.a scrub ok 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:44.563283+0000 mon.a (mon.0) 2854 : audit [DBG] from='client.? 172.21.15.87:0/1017401822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.881615+0000 osd.7 (osd.7) 255 : cluster [DBG] 63.18 scrub starts 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:44.885104+0000 osd.7 (osd.7) 256 : cluster [DBG] 63.18 scrub ok 2024-02-23T00:38:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:45.141964+0000 mon.a (mon.0) 2855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:45.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:45.142037+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e508: 8 total, 7 up, 7 in 2024-02-23T00:38:45.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:45.146220+0000 mon.b (mon.1) 418 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_tier","val": "test-rados-api-smithi087-82091-85-test-flush"}]: dispatch 2024-02-23T00:38:45.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:45.148043+0000 mon.a (mon.0) 2857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_tier","val": "test-rados-api-smithi087-82091-85-test-flush"}]: dispatch 2024-02-23T00:38:45.600 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:45.779 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.232854+0000 osd.1 (osd.1) 239 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:38:45.779 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.240428+0000 osd.1 (osd.1) 240 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:38:45.779 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.333717+0000 osd.2 (osd.2) 199 : cluster [DBG] 63.1a scrub starts 2024-02-23T00:38:45.779 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.335901+0000 osd.2 (osd.2) 200 : cluster [DBG] 63.1a scrub ok 2024-02-23T00:38:45.779 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.439845+0000 osd.6 (osd.6) 214 : cluster [DBG] 3.a scrub starts 2024-02-23T00:38:45.779 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.442730+0000 osd.6 (osd.6) 215 : cluster [DBG] 3.a scrub ok 2024-02-23T00:38:45.779 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:44.563283+0000 mon.a (mon.0) 2854 : audit [DBG] from='client.? 172.21.15.87:0/1017401822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:45.780 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.881615+0000 osd.7 (osd.7) 255 : cluster [DBG] 63.18 scrub starts 2024-02-23T00:38:45.780 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:44.885104+0000 osd.7 (osd.7) 256 : cluster [DBG] 63.18 scrub ok 2024-02-23T00:38:45.780 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:45.141964+0000 mon.a (mon.0) 2855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:38:45.780 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:45.142037+0000 mon.a (mon.0) 2856 : cluster [DBG] osdmap e508: 8 total, 7 up, 7 in 2024-02-23T00:38:45.780 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:45.146220+0000 mon.b (mon.1) 418 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_tier","val": "test-rados-api-smithi087-82091-85-test-flush"}]: dispatch 2024-02-23T00:38:45.780 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:45.148043+0000 mon.a (mon.0) 2857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_tier","val": "test-rados-api-smithi087-82091-85-test-flush"}]: dispatch 2024-02-23T00:38:45.797 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:45.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:45.901 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:45.928 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:46.000 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:46.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:38:46.043 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:38:46.104 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:46.129 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:38:46.131 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:46.336 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:46.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:46.440 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:46.468 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:46.492 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:46.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:46.595 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:46.622 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:46.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:45.709158+0000 mgr.y (mgr.24437) 478 : cluster [DBG] pgmap v778: 279 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 82 active+undersized, 160 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%) 2024-02-23T00:38:46.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:45.710233+0000 mon.a (mon.0) 2858 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-23T00:38:46.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:46.132829+0000 osd.4 (osd.4) 244 : cluster [DBG] 6.e scrub starts 2024-02-23T00:38:46.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:46.135792+0000 osd.4 (osd.4) 245 : cluster [DBG] 6.e scrub ok 2024-02-23T00:38:46.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:46.144661+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_tier","val": "test-rados-api-smithi087-82091-85-test-flush"}]': finished 2024-02-23T00:38:46.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:46.144742+0000 mon.a (mon.0) 2860 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-23T00:38:46.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:46.144802+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e509: 8 total, 7 up, 7 in 2024-02-23T00:38:46.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:46.145818+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:46.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:46.146971+0000 mon.a (mon.0) 2862 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:46.820 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:46.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:46.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:45.709158+0000 mgr.y (mgr.24437) 478 : cluster [DBG] pgmap v778: 279 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 82 active+undersized, 160 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%) 2024-02-23T00:38:46.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:45.710233+0000 mon.a (mon.0) 2858 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:46.132829+0000 osd.4 (osd.4) 244 : cluster [DBG] 6.e scrub starts 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:46.135792+0000 osd.4 (osd.4) 245 : cluster [DBG] 6.e scrub ok 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:46.144661+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_tier","val": "test-rados-api-smithi087-82091-85-test-flush"}]': finished 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:46.144742+0000 mon.a (mon.0) 2860 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:46.144802+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e509: 8 total, 7 up, 7 in 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:46.145818+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:46.146971+0000 mon.a (mon.0) 2862 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:45.709158+0000 mgr.y (mgr.24437) 478 : cluster [DBG] pgmap v778: 279 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 82 active+undersized, 160 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%) 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:45.710233+0000 mon.a (mon.0) 2858 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:46.132829+0000 osd.4 (osd.4) 244 : cluster [DBG] 6.e scrub starts 2024-02-23T00:38:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:46.135792+0000 osd.4 (osd.4) 245 : cluster [DBG] 6.e scrub ok 2024-02-23T00:38:46.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:46.144661+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_tier","val": "test-rados-api-smithi087-82091-85-test-flush"}]': finished 2024-02-23T00:38:46.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:46.144742+0000 mon.a (mon.0) 2860 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-23T00:38:46.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:46.144802+0000 mon.a (mon.0) 2861 : cluster [DBG] osdmap e509: 8 total, 7 up, 7 in 2024-02-23T00:38:46.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:46.145818+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:46.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:46.146971+0000 mon.a (mon.0) 2862 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:38:46.923 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:46.951 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:46.999 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:47.024 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:47.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:47.128 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:47.169 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:47.206 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:47.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:47.317 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:47.344 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:38:47.344 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:38:47.344 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24712.0:14 7.5 7:a3910452:test-rados-api-smithi087-82115-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:30:58.506107+0000", 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 468.83455206299999, 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021562957000000001, 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24712", 2024-02-23T00:38:47.345 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2285581802", 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 14 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506107+0000", 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506107+0000", 2024-02-23T00:38:47.346 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506105+0000", 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5230000000000001e-06 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506114+0000", 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1732999999999999e-05 2024-02-23T00:38:47.347 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506126+0000", 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4780000000000002e-06 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506133+0000", 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9349999999999999e-05 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.348 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506193+0000", 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012566300000000001 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506318+0000", 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017116500000000001 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 0,2", 2024-02-23T00:38:47.349 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.506490+0000", 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0072723850000000001 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.513762+0000", 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0046023009999999996 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.518364+0000", 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0092229960000000007 2024-02-23T00:38:47.350 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.527587+0000", 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5426999999999998e-05 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.527623+0000", 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7034999999999999e-05 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.351 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:30:58.527670+0000", 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021562957000000001 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26036.0:2 64.6 64:602f83fe:::foo:head [write 0~128,set-alloc-hint object_size 0 write_size 0] snapc 0=[] ondisk+write+known_if_redirected e115)", 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 461.11578319199998, 2024-02-23T00:38:47.352 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036711013000000001, 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26036", 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3255892024", 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:38:47.353 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224873+0000", 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.163e-05 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224885+0000", 2024-02-23T00:38:47.354 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8150000000000001e-05 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224903+0000", 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2343e-05 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224915+0000", 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6192000000000005e-05 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224992+0000", 2024-02-23T00:38:47.355 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4450000000000001e-06 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224998+0000", 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032120923000000003 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257119+0000", 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125234 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257244+0000", 2024-02-23T00:38:47.356 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107636 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,5", 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257352+0000", 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011677390000000001 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.258520+0000", 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014228299999999999 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.357 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.259942+0000", 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015894640000000001 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261532+0000", 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.154e-05 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261563+0000", 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3230999999999998e-05 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.358 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261587+0000", 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036711013000000001 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:4 93.7s0 93:f24ec45b:test-rados-api-smithi087-82107-15::foo:head [write 8192~8192] snapc 3=[3,2] ondisk+write+known_if_redirected e144)", 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 432.34448397800003, 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028560524, 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.359 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 4 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.360 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996173+0000", 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1501e-05 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996194+0000", 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1659000000000003e-05 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.361 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996226+0000", 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.439e-06 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996234+0000", 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013176539999999999 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.997552+0000", 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5900000000000008e-06 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.362 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.997560+0000", 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0057013619999999997 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.003262+0000", 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015312100000000001 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.003415+0000", 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033453049999999998 2024-02-23T00:38:47.363 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.006760+0000", 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038155379999999998 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.010576+0000", 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014109313 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.364 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.024685+0000", 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0339e-05 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.024735+0000", 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028560524 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:11 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [write 0~8192] snapc 6=[6] ondisk+write+known_if_redirected e147)", 2024-02-23T00:38:47.365 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:37.991540+0000", 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 429.34911850899999, 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020032722999999999, 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 11 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991540+0000", 2024-02-23T00:38:47.366 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991540+0000", 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991538+0000", 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2162000000000001e-05 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.367 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991570+0000", 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4e-05 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991584+0000", 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1769999999999998e-06 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.991593+0000", 2024-02-23T00:38:47.368 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.016113087000000002 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.007706+0000", 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016325899999999999 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.007869+0000", 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000521098 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.369 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.008390+0000", 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0016975200000000001 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.010088+0000", 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001418084 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.011506+0000", 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7416999999999995e-05 2024-02-23T00:38:47.370 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:38.011573+0000", 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020032722999999999 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:18 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [delete] snapc 7=[7,6] ondisk+write+peerstat_old+known_if_redirected e150)", 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 426.34940774099999, 2024-02-23T00:38:47.371 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026236400999999999, 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 18 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.372 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991249+0000", 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0203999999999999e-05 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.373 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991259+0000", 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5091e-05 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991274+0000", 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.729e-06 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991282+0000", 2024-02-23T00:38:47.374 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7206999999999995e-05 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991349+0000", 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134998 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991484+0000", 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00048211400000000001 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.375 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991966+0000", 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00039970199999999999 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.992366+0000", 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025049227 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.017415+0000", 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2229000000000003e-05 2024-02-23T00:38:47.376 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.017488+0000", 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026236400999999999 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.44587.0:1 151.6s0 151:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 357.30234023200001, 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763, 2024-02-23T00:38:47.377 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.44587", 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/843545378", 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:38:47.378 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038316+0000", 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9619999999999999e-06 2024-02-23T00:38:47.379 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038325+0000", 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7385e-05 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038343+0000", 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0739999999999999e-06 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.380 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038352+0000", 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077681699999999996 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039128+0000", 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0860000000000002e-06 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039132+0000", 2024-02-23T00:38:47.381 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0036840789999999 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042816+0000", 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000158937 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042975+0000", 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000568418 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.382 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.043543+0000", 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00061003399999999997 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.044153+0000", 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011287789999999999 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045282+0000", 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9601e-05 2024-02-23T00:38:47.383 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045351+0000", 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26886.0:1 181.5s0 181:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e278)", 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 298.294337988, 2024-02-23T00:38:47.384 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133, 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26886", 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/134760323", 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.385 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046318+0000", 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9741999999999998e-05 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046338+0000", 2024-02-23T00:38:47.386 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2594e-05 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046351+0000", 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5399999999999996e-06 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046359+0000", 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00098343299999999992 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.387 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047343+0000", 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9160000000000003e-06 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047347+0000", 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0029936850000001 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050340+0000", 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014100300000000001 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.388 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050481+0000", 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038932099999999998 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050871+0000", 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00045121199999999999 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.051322+0000", 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00089874699999999998 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.389 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052221+0000", 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1542999999999998e-05 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052272+0000", 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.390 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27003.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 253.28018428999999, 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568, 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27003", 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3963489775", 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:38:47.391 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060473+0000", 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.417e-06 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.392 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060479+0000", 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8080000000000005e-06 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060486+0000", 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4359999999999999e-06 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060493+0000", 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6943999999999997e-05 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.393 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060530+0000", 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4229999999999999e-06 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060532+0000", 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.99955465700000001 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060087+0000", 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2388999999999999e-05 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.394 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060179+0000", 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00037435399999999997 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060554+0000", 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00057048899999999998 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.395 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061124+0000", 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00075843999999999998 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061883+0000", 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8547999999999998e-05 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061921+0000", 2024-02-23T00:38:47.396 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37526.0:1 199.2s0 199:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 229.26243644100001, 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.011307347, 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.397 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37526", 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/4086862305", 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:38:47.398 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078221+0000", 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1790000000000008e-06 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078229+0000", 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1657e-05 2024-02-23T00:38:47.399 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078240+0000", 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5909999999999996e-06 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078249+0000", 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011604219999999999 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.400 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.079409+0000", 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0110000000000001e-06 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.079412+0000", 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0080054119999999 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087418+0000", 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014009999999999999 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.401 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087558+0000", 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00028455700000000001 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087842+0000", 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00026429900000000001 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.088107+0000", 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013807730000000001 2024-02-23T00:38:47.402 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.089488+0000", 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2237000000000002e-05 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.089530+0000", 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.011307347 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.403 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6796 63.0 63:04a528c6:test-rados-api-smithi087-82091-72::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e458)", 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:37:32.149684+0000", 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 75.190974604999994, 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030257736, 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:47.404 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6796 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149684+0000", 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149684+0000", 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:47.405 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149683+0000", 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5489999999999995e-06 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149692+0000", 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3900000000000001e-05 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.406 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149706+0000", 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9720000000000003e-06 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149713+0000", 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2049000000000003e-05 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.407 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149765+0000", 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5885999999999996e-05 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149861+0000", 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025043958000000002 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.174905+0000", 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077345500000000004 2024-02-23T00:38:47.408 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.175678+0000", 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041288080000000003 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.179807+0000", 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6297000000000001e-05 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.409 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.179853+0000", 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8549999999999998e-05 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.179942+0000", 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030257736 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.410 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:92 63.18 63:1d0125d1:test-rados-api-smithi087-82091-78::732be9d7c0f985174f3dbff4d0300cd89f82a6ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309667+0000", 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 27.030992270999999, 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030556142000000001, 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309667+0000", 2024-02-23T00:38:47.411 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309667+0000", 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309665+0000", 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.038e-06 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.412 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309673+0000", 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1586999999999999e-05 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309684+0000", 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0350000000000004e-06 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.413 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309692+0000", 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105009 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309797+0000", 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012772900000000001 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309924+0000", 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039668359999999996 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.414 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.313891+0000", 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014977572 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.328869+0000", 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00024878600000000003 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.329117+0000", 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011034924999999999 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.415 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340152+0000", 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7619e-05 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340190+0000", 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2716000000000003e-05 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340223+0000", 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030556142000000001 2024-02-23T00:38:47.416 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:95 63.0 63:03bea9d3:test-rados-api-smithi087-82091-78::e98f1f7dcd528309122e8d5522249bd2aff3fd83:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 27.030948816999999, 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030136591000000001, 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:38:47.417 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0749999999999999e-06 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.418 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309714+0000", 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0990000000000004e-06 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309720+0000", 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6360000000000002e-06 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309724+0000", 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00097421499999999996 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.419 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.310699+0000", 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012222900000000001 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.310821+0000", 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00020901700000000001 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311030+0000", 2024-02-23T00:38:47.420 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0075810119999999998 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.318611+0000", 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021088788000000001 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339700+0000", 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3208000000000003e-05 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.421 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339753+0000", 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3796999999999997e-05 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339847+0000", 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030136591000000001 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:96 63.13 63:cc844047:test-rados-api-smithi087-82091-78::4b02657dec822853c72fefcaa266fff97efe7078:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309730+0000", 2024-02-23T00:38:47.422 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 27.03092921, 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031035795000000001, 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309730+0000", 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309730+0000", 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.423 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309729+0000", 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2880000000000001e-06 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309732+0000", 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0950000000000003e-06 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.424 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309738+0000", 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2869999999999998e-06 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309742+0000", 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4709000000000001e-05 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309797+0000", 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127417 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.425 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309924+0000", 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000217624 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,5", 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.310142+0000", 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0096070679999999999 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.319749+0000", 2024-02-23T00:38:47.426 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0099226969999999994 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.329671+0000", 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011011036 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340682+0000", 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5581000000000003e-05 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.427 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340728+0000", 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7443e-05 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340765+0000", 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031035795000000001 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.428 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:100 63.0 63:0389d7c9:test-rados-api-smithi087-82091-78::b277a617178f08ca242dc7700d86bf23f0f3ea67:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309754+0000", 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 27.03090525, 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030390868000000001, 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309754+0000", 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.429 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309754+0000", 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309753+0000", 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8859999999999998e-06 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309760+0000", 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1314000000000003e-05 2024-02-23T00:38:47.430 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309821+0000", 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.899e-06 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309826+0000", 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001460601 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.431 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311287+0000", 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011481600000000001 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311402+0000", 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00021134800000000001 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311613+0000", 2024-02-23T00:38:47.432 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070281809999999997 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.318641+0000", 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021369968 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340011+0000", 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6655e-05 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.433 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340058+0000", 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6738999999999997e-05 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340144+0000", 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030390868000000001 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.434 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.6.44:97 63.1d e483/330 63:bb7c65c5:test-rados-api-smithi087-82091-78::c5740e5a8abdfdd24173d1f0746ed7b8f004fd8c:head v 483'14, mlcod=340'12)", 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.311169+0000", 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 27.029489727000001, 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028548314000000002, 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311169+0000", 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.435 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311169+0000", 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311168+0000", 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4179999999999997e-06 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.436 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311177+0000", 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8430000000000005e-06 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311186+0000", 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2770000000000002e-06 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.437 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311193+0000", 2024-02-23T00:38:47.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3401000000000001e-05 2024-02-23T00:38:47.438 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.438 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.438 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311226+0000", 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1882000000000002e-05 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311248+0000", 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028438485999999999 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339687+0000", 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0916e-05 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339717+0000", 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028548314000000002 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:7287 221.f 221:f2233b45:test-rados-api-smithi087-82091-80::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e494)", 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.218861+0000", 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.121798000999998, 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036222770000000001, 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.440 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 7287 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218861+0000", 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218861+0000", 2024-02-23T00:38:47.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218860+0000", 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1700000000000002e-06 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218866+0000", 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0623e-05 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218877+0000", 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5566e-05 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218892+0000", 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5850999999999999e-05 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218938+0000", 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4792999999999999e-05 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.219033+0000", 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036050749999999999 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255084+0000", 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036222770000000001 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.7.51:63 63.10 e494/425 63:0bcb77f5:test-rados-api-smithi087-82091-80::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head v 494'38, mlcod=0'0)", 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.118071472, 2024-02-23T00:38:47.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026196710000000002, 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:38:47.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3339999999999996e-06 2024-02-23T00:38:47.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222594+0000", 2024-02-23T00:38:47.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0679999999999999e-06 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222602+0000", 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6450000000000004e-06 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222606+0000", 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1100999999999997e-05 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222657+0000", 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9842000000000001e-05 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222677+0000", 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026066840000000001 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.248744+0000", 2024-02-23T00:38:47.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0111000000000003e-05 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.248784+0000", 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026196710000000002 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:7288 221.f 221:f2233b45:test-rados-api-smithi087-82091-80::foo:3 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e494)", 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.255388+0000", 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.085270604000002, 2024-02-23T00:38:47.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041802895, 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 7288 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255388+0000", 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255388+0000", 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255387+0000", 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5440000000000004e-06 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255393+0000", 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0249e-05 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255404+0000", 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7889999999999997e-06 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255411+0000", 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2548999999999999e-05 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255483+0000", 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7108000000000006e-05 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255580+0000", 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041610951 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297191+0000", 2024-02-23T00:38:47.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041802895 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.7.0:543 221.f e494/487 ondisk, result = 0)", 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.267731+0000", 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.072928167000001, 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029799866000000001, 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:38:47.455 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267731+0000", 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267731+0000", 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:38:47.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267730+0000", 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7200000000000003e-06 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267736+0000", 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1474000000000001e-05 2024-02-23T00:38:47.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267747+0000", 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7590000000000003e-06 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267753+0000", 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029659342000000002 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297412+0000", 2024-02-23T00:38:47.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.951e-06 2024-02-23T00:38:47.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297419+0000", 2024-02-23T00:38:47.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111462 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297531+0000", 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029799866000000001 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:7394 222.4 222:20e8a91a:test-rados-api-smithi087-82091-82::foo:head [writefull 0~8192] snapc 0=[] ondisk+write+known_if_redirected e497)", 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:38.160767+0000", 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 9.1798918030000003, 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01920902, 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 7394 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:38:47.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.160767+0000", 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.160767+0000", 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.160765+0000", 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4290999999999998e-05 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.160789+0000", 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4141999999999999e-05 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.160803+0000", 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6149999999999997e-06 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.160812+0000", 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8535000000000006e-05 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.160890+0000", 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013841799999999999 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.161029+0000", 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014888499999999999 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.161178+0000", 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00042537 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.161603+0000", 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.006113421 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.167716+0000", 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012168996 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.179885+0000", 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1520000000000001e-05 2024-02-23T00:38:47.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.179937+0000", 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9070999999999999e-05 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:38.179976+0000", 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01920902 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:38:47.466 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:38:47.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:47.468 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:47.468 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:47.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:47.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:46.477618+0000 osd.6 (osd.6) 216 : cluster [DBG] 6.c scrub starts 2024-02-23T00:38:47.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:46.478466+0000 osd.6 (osd.6) 217 : cluster [DBG] 6.c scrub ok 2024-02-23T00:38:47.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:47.133589+0000 mon.a (mon.0) 2863 : cluster [WRN] Health check update: Degraded data redundancy: 124/1257 objects degraded (9.865%), 28 pgs degraded, 73 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:47.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:47.150153+0000 mon.a (mon.0) 2864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:47.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:47.150252+0000 mon.a (mon.0) 2865 : cluster [DBG] osdmap e510: 8 total, 7 up, 7 in 2024-02-23T00:38:47.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:47.154466+0000 mon.b (mon.1) 420 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:47.471 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:47.156913+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:47.569 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:47.597 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:46.477618+0000 osd.6 (osd.6) 216 : cluster [DBG] 6.c scrub starts 2024-02-23T00:38:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:46.478466+0000 osd.6 (osd.6) 217 : cluster [DBG] 6.c scrub ok 2024-02-23T00:38:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:47.133589+0000 mon.a (mon.0) 2863 : cluster [WRN] Health check update: Degraded data redundancy: 124/1257 objects degraded (9.865%), 28 pgs degraded, 73 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:47.150153+0000 mon.a (mon.0) 2864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:47.150252+0000 mon.a (mon.0) 2865 : cluster [DBG] osdmap e510: 8 total, 7 up, 7 in 2024-02-23T00:38:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:47.154466+0000 mon.b (mon.1) 420 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:47.156913+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:47.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:46.477618+0000 osd.6 (osd.6) 216 : cluster [DBG] 6.c scrub starts 2024-02-23T00:38:47.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:46.478466+0000 osd.6 (osd.6) 217 : cluster [DBG] 6.c scrub ok 2024-02-23T00:38:47.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:47.133589+0000 mon.a (mon.0) 2863 : cluster [WRN] Health check update: Degraded data redundancy: 124/1257 objects degraded (9.865%), 28 pgs degraded, 73 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:47.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:47.150153+0000 mon.a (mon.0) 2864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:38:47.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:47.150252+0000 mon.a (mon.0) 2865 : cluster [DBG] osdmap e510: 8 total, 7 up, 7 in 2024-02-23T00:38:47.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:47.154466+0000 mon.b (mon.1) 420 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:47.795 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:47.156913+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:38:48.020 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-23T00:38:48.044 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:38:48.046 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:38:47.920+0000 7fe8e2097700 -1 mon.a@0(leader).osd e510 definitely_dead 0 2024-02-23T00:38:48.154 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:48.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:48.269 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:48.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.728+0000 7f91a443d700 1 -- 172.21.15.87:0/1639851865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f919c0fd510 msgr2=0x7f919c0f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:48.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.728+0000 7f91a443d700 1 --2- 172.21.15.87:0/1639851865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f919c0fd510 0x7f919c0f6d40 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 rx=0x7f9198004870 tx=0x7f9198009cb0).stop 2024-02-23T00:38:48.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.728+0000 7f91a443d700 1 -- 172.21.15.87:0/1639851865 shutdown_connections 2024-02-23T00:38:48.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.728+0000 7f91a443d700 1 --2- 172.21.15.87:0/1639851865 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f919c0f7400 0x7f919c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:48.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.728+0000 7f91a443d700 1 --2- 172.21.15.87:0/1639851865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f919c0fcbb0 0x7f919c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:48.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.728+0000 7f91a443d700 1 --2- 172.21.15.87:0/1639851865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f919c0fd510 0x7f919c0f6d40 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.728+0000 7f91a443d700 1 -- 172.21.15.87:0/1639851865 >> 172.21.15.87:0/1639851865 conn(0x7f919c0f2820 msgr2=0x7f919c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.728+0000 7f91a443d700 1 -- 172.21.15.87:0/1639851865 shutdown_connections 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.728+0000 7f91a443d700 1 -- 172.21.15.87:0/1639851865 wait complete. 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 librados: starting msgr at 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 librados: starting objecter 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 Processor -- start 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 -- start start 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 librados: setting wanted keys 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 librados: calling monclient init 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f919c0f7400 0x7f919c10e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f919c0fcbb0 0x7f919c10e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f919c0fd510 0x7f919c108260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c108750 con 0x7f919c0fcbb0 2024-02-23T00:38:48.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919c1088c0 con 0x7f919c0fd510 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a443d700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919c108a80 con 0x7f919c0f7400 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a21d9700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f919c0f7400 0x7f919c10e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a19d8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f919c0fcbb0 0x7f919c10e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a21d9700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f919c0f7400 0x7f919c10e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:41924/0 (socket says 172.21.15.87:41924) 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a21d9700 1 -- 172.21.15.87:0/2731210946 learned_addr learned my addr 172.21.15.87:0/2731210946 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a29da700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f919c0fd510 0x7f919c108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a19d8700 1 -- 172.21.15.87:0/2731210946 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f919c0f7400 msgr2=0x7f919c10e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a19d8700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f919c0f7400 0x7f919c10e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a19d8700 1 -- 172.21.15.87:0/2731210946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f919c0fd510 msgr2=0x7f919c108260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a19d8700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f919c0fd510 0x7f919c108260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a19d8700 1 -- 172.21.15.87:0/2731210946 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9198005040 con 0x7f919c0fcbb0 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a21d9700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f919c0f7400 0x7f919c10e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a29da700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f919c0fd510 0x7f919c108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f91a19d8700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f919c0fcbb0 0x7f919c10e5b0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f9198004870 tx=0x7f919800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:48.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f918f7fe700 10 client.?.objecter ms_handle_connect 0x7f919c0fcbb0 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.729+0000 7f918f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.730+0000 7f918f7fe700 1 -- 172.21.15.87:0/2731210946 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919800e070 con 0x7f919c0fcbb0 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.730+0000 7f918f7fe700 1 -- 172.21.15.87:0/2731210946 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9198013da0 con 0x7f919c0fcbb0 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.730+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919c108d00 con 0x7f919c0fcbb0 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.730+0000 7f918f7fe700 1 -- 172.21.15.87:0/2731210946 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198019070 con 0x7f919c0fcbb0 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.730+0000 7f91a443d700 10 client.27444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.730+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919c114ef0 con 0x7f919c0fcbb0 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.730+0000 7f91a443d700 1 librados: init done 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.731+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c05fa60 con 0x7f919c0fcbb0 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.731+0000 7f918f7fe700 1 -- 172.21.15.87:0/2731210946 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f919801e420 con 0x7f919c0fcbb0 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.731+0000 7f918f7fe700 1 --2- 172.21.15.87:0/2731210946 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9188062c90 0x7f9188065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.731+0000 7f918f7fe700 1 -- 172.21.15.87:0/2731210946 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(510..510 src has 173..510) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f919807dfa0 con 0x7f919c0fcbb0 2024-02-23T00:38:48.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.731+0000 7f918f7fe700 10 client.27444.objecter ms_dispatch 0x7f919c001550 osd_map(510..510 src has 173..510) v4 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.731+0000 7f918f7fe700 3 client.27444.objecter handle_osd_map got epochs [510,510] > 0 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.731+0000 7f918f7fe700 3 client.27444.objecter handle_osd_map decoding full epoch 510 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.731+0000 7f91a21d9700 1 --2- 172.21.15.87:0/2731210946 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9188062c90 0x7f9188065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.732+0000 7f918f7fe700 20 client.27444.objecter dump_active .. 0 homeless 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.732+0000 7f918f7fe700 10 client.27444.objecter ms_handle_connect 0x7f9188062c90 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.732+0000 7f91a21d9700 1 --2- 172.21.15.87:0/2731210946 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9188062c90 0x7f9188065150 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 rx=0x7f919000f2f0 tx=0x7f9190009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.737+0000 7f918f7fe700 1 -- 172.21.15.87:0/2731210946 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f91980511e0 con 0x7f919c0fcbb0 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:47.919+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd in", "ids": ["3"]} v 0) v1 -- 0x7f919c05eee0 con 0x7f919c0fcbb0 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.144+0000 7f918f7fe700 1 -- 172.21.15.87:0/2731210946 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["3"]}]=0 marked in osd.3. v511) v1 ==== 85+0+0 (secure 0 0 0) 0x7f919804d020 con 0x7f919c0fcbb0 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.149+0000 7f91a443d700 10 librados: watch_flush enter 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.156+0000 7f91a443d700 10 librados: watch_flush exit 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.156+0000 7f91a443d700 20 client.27444.objecter shutdown clearing up homeless session... 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.156+0000 7f91a443d700 10 client.27444.objecter successfully canceled tick 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.161+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9188062c90 msgr2=0x7f9188065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:48.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.161+0000 7f91a443d700 1 --2- 172.21.15.87:0/2731210946 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9188062c90 0x7f9188065150 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 rx=0x7f919000f2f0 tx=0x7f9190009040).stop 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.161+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f919c0fcbb0 msgr2=0x7f919c10e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.161+0000 7f91a443d700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f919c0fcbb0 0x7f919c10e5b0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 rx=0x7f9198004870 tx=0x7f919800b040).stop 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.165+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 shutdown_connections 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.165+0000 7f91a443d700 1 --2- 172.21.15.87:0/2731210946 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9188062c90 0x7f9188065150 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.165+0000 7f91a443d700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f919c0fd510 0x7f919c108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.165+0000 7f91a443d700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f919c0f7400 0x7f919c10e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.165+0000 7f91a443d700 1 --2- 172.21.15.87:0/2731210946 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f919c0fcbb0 0x7f919c10e5b0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.165+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 >> 172.21.15.87:0/2731210946 conn(0x7f919c0f2820 msgr2=0x7f919c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.167+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 shutdown_connections 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.169+0000 7f91a443d700 1 -- 172.21.15.87:0/2731210946 wait complete. 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:48.169+0000 7f91a443d700 1 librados: shutdown 2024-02-23T00:38:48.290 INFO:teuthology.orchestra.run.smithi087.stderr:marked in osd.3. 2024-02-23T00:38:48.306 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:48.459 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:48.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:48.563 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:48.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:47.257054+0000 osd.1 (osd.1) 241 : cluster [DBG] 63.9 scrub starts 2024-02-23T00:38:48.580 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:47.259908+0000 osd.1 (osd.1) 242 : cluster [DBG] 63.9 scrub ok 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:47.709885+0000 mgr.y (mgr.24437) 479 : cluster [DBG] pgmap v781: 279 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 82 active+undersized, 160 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%) 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:47.921424+0000 mon.a (mon.0) 2867 : audit [DBG] from='client.? 172.21.15.87:0/2731210946' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:48.145993+0000 mon.a (mon.0) 2868 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:48.146082+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? 172.21.15.87:0/2731210946' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:48.146128+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e511: 8 total, 7 up, 8 in 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:47.257054+0000 osd.1 (osd.1) 241 : cluster [DBG] 63.9 scrub starts 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:47.259908+0000 osd.1 (osd.1) 242 : cluster [DBG] 63.9 scrub ok 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:47.709885+0000 mgr.y (mgr.24437) 479 : cluster [DBG] pgmap v781: 279 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 82 active+undersized, 160 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%) 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:47.921424+0000 mon.a (mon.0) 2867 : audit [DBG] from='client.? 172.21.15.87:0/2731210946' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-23T00:38:48.581 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:48.145993+0000 mon.a (mon.0) 2868 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:48.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:48.146082+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? 172.21.15.87:0/2731210946' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-23T00:38:48.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:48.146128+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e511: 8 total, 7 up, 8 in 2024-02-23T00:38:48.591 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:47.257054+0000 osd.1 (osd.1) 241 : cluster [DBG] 63.9 scrub starts 2024-02-23T00:38:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:47.259908+0000 osd.1 (osd.1) 242 : cluster [DBG] 63.9 scrub ok 2024-02-23T00:38:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:47.709885+0000 mgr.y (mgr.24437) 479 : cluster [DBG] pgmap v781: 279 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 82 active+undersized, 160 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%) 2024-02-23T00:38:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:47.921424+0000 mon.a (mon.0) 2867 : audit [DBG] from='client.? 172.21.15.87:0/2731210946' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-23T00:38:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:48.145993+0000 mon.a (mon.0) 2868 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:38:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:48.146082+0000 mon.a (mon.0) 2869 : audit [INF] from='client.? 172.21.15.87:0/2731210946' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-23T00:38:48.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:48.146128+0000 mon.a (mon.0) 2870 : cluster [DBG] osdmap e511: 8 total, 7 up, 8 in 2024-02-23T00:38:49.063 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-02-23T00:38:49.140 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:49.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:49.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:49.245 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:49.285 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:49.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:48.307473+0000 osd.7 (osd.7) 257 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:49.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:48.307598+0000 osd.7 (osd.7) 258 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:49.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:48.491323+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:38:49.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:48.494220+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:38:49.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.148016+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e512: 8 total, 7 up, 8 in 2024-02-23T00:38:49.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.161343+0000 osd.7 (osd.7) 259 : cluster [DBG] 4.1e starting backfill to osd.3 from (0'0,0'0] MAX to 453'40 2024-02-23T00:38:49.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:48.307473+0000 osd.7 (osd.7) 257 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:49.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:48.307598+0000 osd.7 (osd.7) 258 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:49.787 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:48.491323+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:38:49.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:48.494220+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:38:49.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.148016+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e512: 8 total, 7 up, 8 in 2024-02-23T00:38:49.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.161343+0000 osd.7 (osd.7) 259 : cluster [DBG] 4.1e starting backfill to osd.3 from (0'0,0'0] MAX to 453'40 2024-02-23T00:38:49.800 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:49.840 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:49.846 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:49.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:48.307473+0000 osd.7 (osd.7) 257 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:38:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:48.307598+0000 osd.7 (osd.7) 258 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:38:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:48.491323+0000 osd.6 (osd.6) 218 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:38:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:48.494220+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:38:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.148016+0000 mon.a (mon.0) 2871 : cluster [DBG] osdmap e512: 8 total, 7 up, 8 in 2024-02-23T00:38:49.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.161343+0000 osd.7 (osd.7) 259 : cluster [DBG] 4.1e starting backfill to osd.3 from (0'0,0'0] MAX to 453'40 2024-02-23T00:38:49.958 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:49.984 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:50.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.154518+0000 osd.2 (osd.2) 201 : cluster [DBG] 4.19 starting backfill to osd.3 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:50.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.158942+0000 osd.1 (osd.1) 243 : cluster [DBG] 4.1a starting backfill to osd.3 from (0'0,0'0] MAX to 476'63 2024-02-23T00:38:50.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.159076+0000 osd.1 (osd.1) 244 : cluster [INF] 4.1a continuing backfill to osd.4 from (63'25,476'63] MIN to 476'63 2024-02-23T00:38:50.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.159276+0000 osd.1 (osd.1) 245 : cluster [DBG] 4.17 starting backfill to osd.3 from (0'0,0'0] MAX to 453'29 2024-02-23T00:38:50.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.159418+0000 osd.1 (osd.1) 246 : cluster [INF] 4.17 continuing backfill to osd.7 from (63'19,453'29] MIN to 453'29 2024-02-23T00:38:50.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.165363+0000 osd.2 (osd.2) 202 : cluster [DBG] 4.1c starting backfill to osd.3 from (0'0,0'0] MAX to 453'37 2024-02-23T00:38:50.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.166140+0000 osd.1 (osd.1) 247 : cluster [DBG] 4.16 starting backfill to osd.3 from (0'0,0'0] MAX to 453'39 2024-02-23T00:38:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.166235+0000 osd.1 (osd.1) 248 : cluster [INF] 4.16 continuing backfill to osd.7 from (63'21,453'39] MIN to 453'39 2024-02-23T00:38:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:49.710754+0000 mgr.y (mgr.24437) 480 : cluster [DBG] pgmap v784: 278 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 82 active+undersized, 159 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%) 2024-02-23T00:38:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:50.150757+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e513: 8 total, 7 up, 8 in 2024-02-23T00:38:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:50.153721+0000 osd.4 (osd.4) 246 : cluster [DBG] 4.9 starting backfill to osd.3 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:50.154532+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.1b starting backfill to osd.3 from (0'0,0'0] MAX to 453'22 2024-02-23T00:38:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:50.156504+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.c starting backfill to osd.3 from (0'0,0'0] MAX to 453'37 2024-02-23T00:38:50.834 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-rgw-foo-a[64469]: debug 2024-02-23T00:38:50.467+0000 7fb181370700 -1 RGWWatcher::handle_error cookie 94532396652160 err (110) Connection timed out 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.154518+0000 osd.2 (osd.2) 201 : cluster [DBG] 4.19 starting backfill to osd.3 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.158942+0000 osd.1 (osd.1) 243 : cluster [DBG] 4.1a starting backfill to osd.3 from (0'0,0'0] MAX to 476'63 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.159076+0000 osd.1 (osd.1) 244 : cluster [INF] 4.1a continuing backfill to osd.4 from (63'25,476'63] MIN to 476'63 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.159276+0000 osd.1 (osd.1) 245 : cluster [DBG] 4.17 starting backfill to osd.3 from (0'0,0'0] MAX to 453'29 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.159418+0000 osd.1 (osd.1) 246 : cluster [INF] 4.17 continuing backfill to osd.7 from (63'19,453'29] MIN to 453'29 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.165363+0000 osd.2 (osd.2) 202 : cluster [DBG] 4.1c starting backfill to osd.3 from (0'0,0'0] MAX to 453'37 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.166140+0000 osd.1 (osd.1) 247 : cluster [DBG] 4.16 starting backfill to osd.3 from (0'0,0'0] MAX to 453'39 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.166235+0000 osd.1 (osd.1) 248 : cluster [INF] 4.16 continuing backfill to osd.7 from (63'21,453'39] MIN to 453'39 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:49.710754+0000 mgr.y (mgr.24437) 480 : cluster [DBG] pgmap v784: 278 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 82 active+undersized, 159 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%) 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:50.150757+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e513: 8 total, 7 up, 8 in 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:50.153721+0000 osd.4 (osd.4) 246 : cluster [DBG] 4.9 starting backfill to osd.3 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:50.154532+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.1b starting backfill to osd.3 from (0'0,0'0] MAX to 453'22 2024-02-23T00:38:50.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:50.156504+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.c starting backfill to osd.3 from (0'0,0'0] MAX to 453'37 2024-02-23T00:38:50.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.154518+0000 osd.2 (osd.2) 201 : cluster [DBG] 4.19 starting backfill to osd.3 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:50.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.158942+0000 osd.1 (osd.1) 243 : cluster [DBG] 4.1a starting backfill to osd.3 from (0'0,0'0] MAX to 476'63 2024-02-23T00:38:50.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.159076+0000 osd.1 (osd.1) 244 : cluster [INF] 4.1a continuing backfill to osd.4 from (63'25,476'63] MIN to 476'63 2024-02-23T00:38:50.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.159276+0000 osd.1 (osd.1) 245 : cluster [DBG] 4.17 starting backfill to osd.3 from (0'0,0'0] MAX to 453'29 2024-02-23T00:38:50.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.159418+0000 osd.1 (osd.1) 246 : cluster [INF] 4.17 continuing backfill to osd.7 from (63'19,453'29] MIN to 453'29 2024-02-23T00:38:50.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.165363+0000 osd.2 (osd.2) 202 : cluster [DBG] 4.1c starting backfill to osd.3 from (0'0,0'0] MAX to 453'37 2024-02-23T00:38:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.166140+0000 osd.1 (osd.1) 247 : cluster [DBG] 4.16 starting backfill to osd.3 from (0'0,0'0] MAX to 453'39 2024-02-23T00:38:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.166235+0000 osd.1 (osd.1) 248 : cluster [INF] 4.16 continuing backfill to osd.7 from (63'21,453'39] MIN to 453'39 2024-02-23T00:38:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:49.710754+0000 mgr.y (mgr.24437) 480 : cluster [DBG] pgmap v784: 278 pgs: 6 active+clean+remapped, 3 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 82 active+undersized, 159 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 124/1257 objects degraded (9.865%); 87/1257 objects misplaced (6.921%) 2024-02-23T00:38:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:50.150757+0000 mon.a (mon.0) 2872 : cluster [DBG] osdmap e513: 8 total, 7 up, 8 in 2024-02-23T00:38:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:50.153721+0000 osd.4 (osd.4) 246 : cluster [DBG] 4.9 starting backfill to osd.3 from (0'0,0'0] MAX to 453'68 2024-02-23T00:38:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:50.154532+0000 osd.4 (osd.4) 247 : cluster [DBG] 4.1b starting backfill to osd.3 from (0'0,0'0] MAX to 453'22 2024-02-23T00:38:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:50.156504+0000 osd.4 (osd.4) 248 : cluster [DBG] 4.c starting backfill to osd.3 from (0'0,0'0] MAX to 453'37 2024-02-23T00:38:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.210+0000 7fbd01b22700 1 -- 172.21.15.87:0/564546090 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0f6b10 msgr2=0x7fbcfc1030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.210+0000 7fbd01b22700 1 --2- 172.21.15.87:0/564546090 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0f6b10 0x7fbcfc1030b0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 rx=0x7fbcf400adc0 tx=0x7fbcf400c040).stop 2024-02-23T00:38:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.210+0000 7fbd01b22700 1 -- 172.21.15.87:0/564546090 shutdown_connections 2024-02-23T00:38:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.210+0000 7fbd01b22700 1 --2- 172.21.15.87:0/564546090 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0f6b10 0x7fbcfc1030b0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.210+0000 7fbd01b22700 1 --2- 172.21.15.87:0/564546090 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbcfc0fc240 0x7fbcfc0fc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.210+0000 7fbd01b22700 1 --2- 172.21.15.87:0/564546090 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbcfc0fcba0 0x7fbcfc0f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.210+0000 7fbd01b22700 1 -- 172.21.15.87:0/564546090 >> 172.21.15.87:0/564546090 conn(0x7fbcfc06c370 msgr2=0x7fbcfc06c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:51.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.210+0000 7fbd01b22700 1 -- 172.21.15.87:0/564546090 shutdown_connections 2024-02-23T00:38:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.210+0000 7fbd01b22700 1 -- 172.21.15.87:0/564546090 wait complete. 2024-02-23T00:38:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 librados: starting msgr at 2024-02-23T00:38:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 librados: starting objecter 2024-02-23T00:38:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 Processor -- start 2024-02-23T00:38:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 -- start start 2024-02-23T00:38:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 librados: setting wanted keys 2024-02-23T00:38:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 librados: calling monclient init 2024-02-23T00:38:51.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbcfc0f6b10 0x7fbcfc106270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbcfc0fc240 0x7fbcfc106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0fcba0 0x7fbcfc122eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc106f40 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc062700 con 0x7fbcfc0f6b10 2024-02-23T00:38:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.211+0000 7fbd01b22700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc0628c0 con 0x7fbcfc0fc240 2024-02-23T00:38:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0fcba0 0x7fbcfc122eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfbfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0fcba0 0x7fbcfc122eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54778/0 (socket says 172.21.15.87:54778) 2024-02-23T00:38:51.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfbfff700 1 -- 172.21.15.87:0/1934243338 learned_addr learned my addr 172.21.15.87:0/1934243338 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfaffd700 1 --2- 172.21.15.87:0/1934243338 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbcfc0fc240 0x7fbcfc106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfbfff700 1 -- 172.21.15.87:0/1934243338 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbcfc0fc240 msgr2=0x7fbcfc106760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfbfff700 1 --2- 172.21.15.87:0/1934243338 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbcfc0fc240 0x7fbcfc106760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfbfff700 1 -- 172.21.15.87:0/1934243338 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbcfc0f6b10 msgr2=0x7fbcfc106270 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-23T00:38:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfbfff700 1 --2- 172.21.15.87:0/1934243338 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbcfc0f6b10 0x7fbcfc106270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:38:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfbfff700 1 -- 172.21.15.87:0/1934243338 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf400a8d0 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcfbfff700 1 --2- 172.21.15.87:0/1934243338 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0fcba0 0x7fbcfc122eb0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7fbcf400ac60 tx=0x7fbcf400c040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcf8ff9700 10 client.?.objecter ms_handle_connect 0x7fbcfc0fcba0 2024-02-23T00:38:51.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcf8ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcf8ff9700 1 -- 172.21.15.87:0/1934243338 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4019070 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcf8ff9700 1 -- 172.21.15.87:0/1934243338 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcf400a530 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.212+0000 7fbcf8ff9700 1 -- 172.21.15.87:0/1934243338 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf401e6f0 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.213+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc062b40 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.213+0000 7fbd01b22700 10 client.27450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.213+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcfc062f80 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.213+0000 7fbd01b22700 1 librados: init done 2024-02-23T00:38:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.214+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcfc05fa40 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.215+0000 7fbcf8ff9700 1 -- 172.21.15.87:0/1934243338 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fbcf401eb20 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.215+0000 7fbcf8ff9700 1 --2- 172.21.15.87:0/1934243338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbce4062c90 0x7fbce4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.215+0000 7fbcf8ff9700 1 -- 172.21.15.87:0/1934243338 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(513..513 src has 173..513) v4 ==== 9912+0+0 (secure 0 0 0) 0x7fbcf407f250 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.215+0000 7fbcf8ff9700 10 client.27450.objecter ms_dispatch 0x7fbcfc001550 osd_map(513..513 src has 173..513) v4 2024-02-23T00:38:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.215+0000 7fbcf8ff9700 3 client.27450.objecter handle_osd_map got epochs [513,513] > 0 2024-02-23T00:38:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.215+0000 7fbcf8ff9700 3 client.27450.objecter handle_osd_map decoding full epoch 513 2024-02-23T00:38:51.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.216+0000 7fbcf8ff9700 20 client.27450.objecter dump_active .. 0 homeless 2024-02-23T00:38:51.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.220+0000 7fbcf8ff9700 1 -- 172.21.15.87:0/1934243338 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbcf40520d0 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.223+0000 7fbcfb7fe700 1 --2- 172.21.15.87:0/1934243338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbce4062c90 0x7fbce4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:51.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.223+0000 7fbcfb7fe700 1 --2- 172.21.15.87:0/1934243338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbce4062c90 0x7fbce4065150 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7fbcec004c60 tx=0x7fbcec00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:51.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.223+0000 7fbcf8ff9700 10 client.27450.objecter ms_handle_connect 0x7fbce4062c90 2024-02-23T00:38:51.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.416+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fbcfc05ee80 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:38:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:38:51.495 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.492+0000 7fbcf8ff9700 1 -- 172.21.15.87:0/1934243338 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v514) v1 ==== 87+0+0 (secure 0 0 0) 0x7fbcf4051cc0 con 0x7fbcfc0fcba0 2024-02-23T00:38:51.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.500+0000 7fbd01b22700 10 librados: watch_flush enter 2024-02-23T00:38:51.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.500+0000 7fbd01b22700 10 librados: watch_flush exit 2024-02-23T00:38:51.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.500+0000 7fbd01b22700 20 client.27450.objecter shutdown clearing up homeless session... 2024-02-23T00:38:51.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.500+0000 7fbd01b22700 10 client.27450.objecter successfully canceled tick 2024-02-23T00:38:51.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.500+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbce4062c90 msgr2=0x7fbce4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:51.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.500+0000 7fbd01b22700 1 --2- 172.21.15.87:0/1934243338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbce4062c90 0x7fbce4065150 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 rx=0x7fbcec004c60 tx=0x7fbcec00b040).stop 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.500+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0fcba0 msgr2=0x7fbcfc122eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.500+0000 7fbd01b22700 1 --2- 172.21.15.87:0/1934243338 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0fcba0 0x7fbcfc122eb0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 rx=0x7fbcf400ac60 tx=0x7fbcf400c040).stop 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.501+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 shutdown_connections 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.501+0000 7fbd01b22700 1 --2- 172.21.15.87:0/1934243338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbce4062c90 0x7fbce4065150 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.501+0000 7fbd01b22700 1 --2- 172.21.15.87:0/1934243338 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbcfc0fcba0 0x7fbcfc122eb0 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.501+0000 7fbd01b22700 1 --2- 172.21.15.87:0/1934243338 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbcfc0f6b10 0x7fbcfc106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.501+0000 7fbd01b22700 1 --2- 172.21.15.87:0/1934243338 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbcfc0fc240 0x7fbcfc106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.501+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 >> 172.21.15.87:0/1934243338 conn(0x7fbcfc06c370 msgr2=0x7fbcfc0f50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.502+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 shutdown_connections 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.502+0000 7fbd01b22700 1 -- 172.21.15.87:0/1934243338 wait complete. 2024-02-23T00:38:51.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:51.502+0000 7fbd01b22700 1 librados: shutdown 2024-02-23T00:38:51.508 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:38:51.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:50.156390+0000 osd.1 (osd.1) 249 : cluster [DBG] 4.1d starting backfill to osd.3 from (0'0,0'0] MAX to 453'44 2024-02-23T00:38:51.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:50.166560+0000 osd.1 (osd.1) 250 : cluster [DBG] 4.f starting backfill to osd.3 from (0'0,0'0] MAX to 453'42 2024-02-23T00:38:51.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:51.039038+0000 mgr.y (mgr.24437) 481 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:51.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:51.417196+0000 mon.a (mon.0) 2873 : audit [INF] from='client.? 172.21.15.87:0/1934243338' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:38:51.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:50.156390+0000 osd.1 (osd.1) 249 : cluster [DBG] 4.1d starting backfill to osd.3 from (0'0,0'0] MAX to 453'44 2024-02-23T00:38:51.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:50.166560+0000 osd.1 (osd.1) 250 : cluster [DBG] 4.f starting backfill to osd.3 from (0'0,0'0] MAX to 453'42 2024-02-23T00:38:51.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:51.039038+0000 mgr.y (mgr.24437) 481 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:51.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:51.417196+0000 mon.a (mon.0) 2873 : audit [INF] from='client.? 172.21.15.87:0/1934243338' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:38:51.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:50.156390+0000 osd.1 (osd.1) 249 : cluster [DBG] 4.1d starting backfill to osd.3 from (0'0,0'0] MAX to 453'44 2024-02-23T00:38:52.055 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:50.166560+0000 osd.1 (osd.1) 250 : cluster [DBG] 4.f starting backfill to osd.3 from (0'0,0'0] MAX to 453'42 2024-02-23T00:38:52.055 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:51.039038+0000 mgr.y (mgr.24437) 481 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:38:52.055 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:51.417196+0000 mon.a (mon.0) 2873 : audit [INF] from='client.? 172.21.15.87:0/1934243338' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:38:52.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:51.486359+0000 mon.a (mon.0) 2874 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:52.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:51.490316+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? 172.21.15.87:0/1934243338' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:38:52.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:51.490368+0000 mon.a (mon.0) 2876 : cluster [DBG] osdmap e514: 8 total, 7 up, 8 in 2024-02-23T00:38:52.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:51.711442+0000 mgr.y (mgr.24437) 482 : cluster [DBG] pgmap v787: 278 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+recovery_wait, 6 active+remapped+backfill_wait, 5 active+recovering, 22 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 163 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 0 op/s; 124/1260 objects degraded (9.841%); 188/1260 objects misplaced (14.921%); 185 B/s, 26 objects/s recovering 2024-02-23T00:38:52.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:52.141928+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e515: 8 total, 7 up, 8 in 2024-02-23T00:38:52.505 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:52] "GET /metrics HTTP/1.1" 200 188956 "" "Prometheus/2.33.4" 2024-02-23T00:38:52.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:51.486359+0000 mon.a (mon.0) 2874 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:52.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:51.490316+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? 172.21.15.87:0/1934243338' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:38:52.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:51.490368+0000 mon.a (mon.0) 2876 : cluster [DBG] osdmap e514: 8 total, 7 up, 8 in 2024-02-23T00:38:52.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:51.711442+0000 mgr.y (mgr.24437) 482 : cluster [DBG] pgmap v787: 278 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+recovery_wait, 6 active+remapped+backfill_wait, 5 active+recovering, 22 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 163 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 0 op/s; 124/1260 objects degraded (9.841%); 188/1260 objects misplaced (14.921%); 185 B/s, 26 objects/s recovering 2024-02-23T00:38:52.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:52.141928+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e515: 8 total, 7 up, 8 in 2024-02-23T00:38:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:51.486359+0000 mon.a (mon.0) 2874 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:38:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:51.490316+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? 172.21.15.87:0/1934243338' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:38:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:51.490368+0000 mon.a (mon.0) 2876 : cluster [DBG] osdmap e514: 8 total, 7 up, 8 in 2024-02-23T00:38:52.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:51.711442+0000 mgr.y (mgr.24437) 482 : cluster [DBG] pgmap v787: 278 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+recovery_wait, 6 active+remapped+backfill_wait, 5 active+recovering, 22 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 163 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 0 op/s; 124/1260 objects degraded (9.841%); 188/1260 objects misplaced (14.921%); 185 B/s, 26 objects/s recovering 2024-02-23T00:38:52.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:52.141928+0000 mon.a (mon.0) 2877 : cluster [DBG] osdmap e515: 8 total, 7 up, 8 in 2024-02-23T00:38:53.550 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:53.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:53.659 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:53.687 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:53.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:52.495906+0000 mon.a (mon.0) 2878 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:38:53.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:52.495945+0000 mon.a (mon.0) 2879 : cluster [WRN] Health check update: Degraded data redundancy: 124/1260 objects degraded (9.841%), 28 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:53.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:53.155067+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e516: 8 total, 7 up, 8 in 2024-02-23T00:38:53.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:52.495906+0000 mon.a (mon.0) 2878 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:38:53.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:52.495945+0000 mon.a (mon.0) 2879 : cluster [WRN] Health check update: Degraded data redundancy: 124/1260 objects degraded (9.841%), 28 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:53.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:53.155067+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e516: 8 total, 7 up, 8 in 2024-02-23T00:38:53.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:52.495906+0000 mon.a (mon.0) 2878 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:38:53.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:52.495945+0000 mon.a (mon.0) 2879 : cluster [WRN] Health check update: Degraded data redundancy: 124/1260 objects degraded (9.841%), 28 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:38:53.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:53.155067+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e516: 8 total, 7 up, 8 in 2024-02-23T00:38:53.943 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:38:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:38:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:38:54.063 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:38:54.064 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:38:54.064 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.log 2024-02-23T00:38:54.064 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:38:54.775 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:54.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:54.879 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:54.896 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:38:54.896 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:38:54.896 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:38:54.897 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:38:54.910 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.116+0000 7f6c62a4e700 1 -- 172.21.15.87:0/3893686123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c5c0fcbb0 msgr2=0x7f6c5c0fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.116+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/3893686123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c5c0fcbb0 0x7f6c5c0fcf90 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 rx=0x7f6c4c004870 tx=0x7f6c4c009cb0).stop 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.116+0000 7f6c62a4e700 1 -- 172.21.15.87:0/3893686123 shutdown_connections 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.116+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/3893686123 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c5c0f7400 0x7f6c5c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.116+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/3893686123 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c5c0fcbb0 0x7f6c5c0fcf90 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.116+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/3893686123 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c5c0fd510 0x7f6c5c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.116+0000 7f6c62a4e700 1 -- 172.21.15.87:0/3893686123 >> 172.21.15.87:0/3893686123 conn(0x7f6c5c0f2820 msgr2=0x7f6c5c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.116+0000 7f6c62a4e700 1 -- 172.21.15.87:0/3893686123 shutdown_connections 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.116+0000 7f6c62a4e700 1 -- 172.21.15.87:0/3893686123 wait complete. 2024-02-23T00:38:55.126 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 librados: starting msgr at 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 librados: starting objecter 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 Processor -- start 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 -- start start 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 librados: setting wanted keys 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 librados: calling monclient init 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c5c0f7400 0x7f6c5c10d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c5c0fcbb0 0x7f6c5c10d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c5c0fd510 0x7f6c5c107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c107770 con 0x7f6c5c0fcbb0 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c1078e0 con 0x7f6c5c0fd510 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c62a4e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c107aa0 con 0x7f6c5c0f7400 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c5c0f7400 0x7f6c5c10d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c5c0fcbb0 0x7f6c5c10d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c5c0f7400 0x7f6c5c10d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:48122/0 (socket says 172.21.15.87:48122) 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5bfff700 1 -- 172.21.15.87:0/1238097181 learned_addr learned my addr 172.21.15.87:0/1238097181 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c60feb700 1 --2- 172.21.15.87:0/1238097181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c5c0fd510 0x7f6c5c107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5bfff700 1 -- 172.21.15.87:0/1238097181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c5c0fd510 msgr2=0x7f6c5c107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5bfff700 1 --2- 172.21.15.87:0/1238097181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c5c0fd510 0x7f6c5c107280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5bfff700 1 -- 172.21.15.87:0/1238097181 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c5c0fcbb0 msgr2=0x7f6c5c10d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5bfff700 1 --2- 172.21.15.87:0/1238097181 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c5c0fcbb0 0x7f6c5c10d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5bfff700 1 -- 172.21.15.87:0/1238097181 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540097c0 con 0x7f6c5c0f7400 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c60feb700 1 --2- 172.21.15.87:0/1238097181 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6c5c0fd510 0x7f6c5c107280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.117+0000 7f6c5b7fe700 1 --2- 172.21.15.87:0/1238097181 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6c5c0fcbb0 0x7f6c5c10d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:38:55.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c5bfff700 1 --2- 172.21.15.87:0/1238097181 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6c5c0f7400 0x7f6c5c10d0e0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 rx=0x7f6c4c009eb0 tx=0x7f6c4c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:55.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c597fa700 10 client.?.objecter ms_handle_connect 0x7f6c5c0f7400 2024-02-23T00:38:55.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c597fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:38:55.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c00e070 con 0x7f6c5c0f7400 2024-02-23T00:38:55.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4c005040 con 0x7f6c5c0f7400 2024-02-23T00:38:55.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c4c013da0 con 0x7f6c5c0f7400 2024-02-23T00:38:55.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c019070 con 0x7f6c5c0f7400 2024-02-23T00:38:55.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c62a4e700 10 client.37922.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:55.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c107f40 con 0x7f6c5c0f7400 2024-02-23T00:38:55.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c62a4e700 1 librados: init done 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.118+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6c5c108130 con 0x7f6c5c0f7400 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.119+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f6c4c01e420 con 0x7f6c5c0f7400 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.119+0000 7f6c597fa700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6c44062c90 0x7f6c44065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.119+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(517..517 src has 173..517) v4 ==== 9784+0+0 (secure 0 0 0) 0x7f6c4c07e3f0 con 0x7f6c5c0f7400 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.119+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 osd_map(517..517 src has 173..517) v4 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.119+0000 7f6c597fa700 3 client.37922.objecter handle_osd_map got epochs [517,517] > 0 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.119+0000 7f6c597fa700 3 client.37922.objecter handle_osd_map decoding full epoch 517 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.119+0000 7f6c5b7fe700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6c44062c90 0x7f6c44065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c597fa700 20 client.37922.objecter dump_active .. 0 homeless 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v517) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6c4c085070 con 0x7f6c5c0f7400 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c597fa700 10 client.37922.objecter ms_handle_connect 0x7f6c44062c90 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c5b7fe700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6c44062c90 0x7f6c44065150 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 rx=0x7f6c5c108a20 tx=0x7f6c5000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:55.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 1 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 20 client.37922.objecter _get_session osd=-1 returning homeless 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 10 client.37922.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=518}) v3 -- 0x7f6c5c05fa60 con 0x7f6c5c0f7400 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6c5c05ee80 con 0x7f6c5c0f7400 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=517) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c4c051270 con 0x7f6c5c0f7400 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c5a7fc700 10 client.37922.objecter _check_command_map_dne tid 1 current 517 map_dne_bound 517 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c5a7fc700 10 client.37922.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c5a7fc700 15 client.37922.objecter _session_command_op_remove -1 1 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 2 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6805/685354171 conn(0x7f6c5c028540 0x7f6c5c0fd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c1060e0 osd=1 3 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c1060e0 osd=1 3 2024-02-23T00:38:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c1060e0 osd=1 3 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 2 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c1060e0 osd=1 3 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 1 2 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c1060e0 osd=1 4 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 10 client.37922.objecter _send_command 2 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c5c05a040 con 0x7f6c5c028540 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.120+0000 7f6c60feb700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6805/685354171 conn(0x7f6c5c028540 0x7f6c5c0fd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.121+0000 7f6c597fa700 10 client.37922.objecter ms_handle_connect 0x7f6c5c028540 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.121+0000 7f6c60feb700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6805/685354171 conn(0x7f6c5c028540 0x7f6c5c0fd990 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.121+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6c5c05a040 con 0x7f6c5c028540 2024-02-23T00:38:55.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.121+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:38:55.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.121+0000 7f6c597fa700 10 client.37922.objecter _finish_command 2 = system:0 2024-02-23T00:38:55.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.121+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c1060e0 osd=1 4 2024-02-23T00:38:55.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.121+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 1 2 2024-02-23T00:38:55.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:55.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 3 2024-02-23T00:38:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c1060e0 osd=1 3 2024-02-23T00:38:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c1060e0 osd=1 3 2024-02-23T00:38:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c1060e0 osd=1 3 2024-02-23T00:38:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 3 2024-02-23T00:38:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c1060e0 osd=1 3 2024-02-23T00:38:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 1 3 2024-02-23T00:38:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c1060e0 osd=1 4 2024-02-23T00:38:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 10 client.37922.objecter _send_command 3 2024-02-23T00:38:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.137+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6c5c0ff000 con 0x7f6c5c028540 2024-02-23T00:38:55.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6c5c0ff000 con 0x7f6c5c028540 2024-02-23T00:38:55.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:55.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c597fa700 10 client.37922.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c1060e0 osd=1 4 2024-02-23T00:38:55.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 1 3 2024-02-23T00:38:55.140 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:38:55.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:55.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 4 2024-02-23T00:38:55.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6c5c0280a0 0x7f6c5c117f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c0597e0 osd=2 3 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c0597e0 osd=2 3 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c0597e0 osd=2 3 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 4 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c0597e0 osd=2 3 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 2 4 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c0597e0 osd=2 4 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 10 client.37922.objecter _send_command 4 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c5c059950 con 0x7f6c5c0280a0 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c5bfff700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6c5c0280a0 0x7f6c5c117f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c597fa700 10 client.37922.objecter ms_handle_connect 0x7f6c5c0280a0 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.139+0000 7f6c5bfff700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6809/3605394228 conn(0x7f6c5c0280a0 0x7f6c5c117f30 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.140+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6c5c059950 con 0x7f6c5c0280a0 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.140+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:38:55.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.140+0000 7f6c597fa700 10 client.37922.objecter _finish_command 4 = system:0 2024-02-23T00:38:55.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.140+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c0597e0 osd=2 4 2024-02-23T00:38:55.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.140+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 2 4 2024-02-23T00:38:55.157 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:53.712382+0000 mgr.y (mgr.24437) 483 : cluster [DBG] pgmap v790: 278 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+recovery_wait, 6 active+remapped+backfill_wait, 5 active+recovering, 22 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 163 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 0 op/s; 124/1260 objects degraded (9.841%); 188/1260 objects misplaced (14.921%); 185 B/s, 26 objects/s recovering 2024-02-23T00:38:55.157 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:54.155215+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e517: 8 total, 7 up, 8 in 2024-02-23T00:38:55.157 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:55.119960+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? 172.21.15.87:0/1238097181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:55.158 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:53.712382+0000 mgr.y (mgr.24437) 483 : cluster [DBG] pgmap v790: 278 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+recovery_wait, 6 active+remapped+backfill_wait, 5 active+recovering, 22 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 163 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 0 op/s; 124/1260 objects degraded (9.841%); 188/1260 objects misplaced (14.921%); 185 B/s, 26 objects/s recovering 2024-02-23T00:38:55.158 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:54.155215+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e517: 8 total, 7 up, 8 in 2024-02-23T00:38:55.158 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:55.119960+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? 172.21.15.87:0/1238097181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:55.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:55.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 5 2024-02-23T00:38:55.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c0597e0 osd=2 3 2024-02-23T00:38:55.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c0597e0 osd=2 3 2024-02-23T00:38:55.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c0597e0 osd=2 3 2024-02-23T00:38:55.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 5 2024-02-23T00:38:55.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c0597e0 osd=2 3 2024-02-23T00:38:55.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 2 5 2024-02-23T00:38:55.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c0597e0 osd=2 4 2024-02-23T00:38:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 10 client.37922.objecter _send_command 5 2024-02-23T00:38:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.156+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6c5c027600 con 0x7f6c5c0280a0 2024-02-23T00:38:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6c5c027600 con 0x7f6c5c0280a0 2024-02-23T00:38:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c597fa700 10 client.37922.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c0597e0 osd=2 4 2024-02-23T00:38:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 2 5 2024-02-23T00:38:55.160 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 6 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6813/128059790 conn(0x7f6c5c061a90 0x7f6c5c11a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c027a10 osd=3 3 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c027a10 osd=3 3 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c027a10 osd=3 3 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 6 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c027a10 osd=3 3 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 3 6 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c027a10 osd=3 4 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 10 client.37922.objecter _send_command 6 2024-02-23T00:38:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c5c11f4f0 con 0x7f6c5c061a90 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.158+0000 7f6c5b7fe700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6813/128059790 conn(0x7f6c5c061a90 0x7f6c5c11a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.159+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== mon.2 v2:172.21.15.87:3301/0 8 ==== osd_map(518..518 src has 173..518) v4 ==== 288+0+0 (secure 0 0 0) 0x7f6c4c04f640 con 0x7f6c5c0f7400 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.159+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 osd_map(518..518 src has 173..518) v4 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.159+0000 7f6c597fa700 3 client.37922.objecter handle_osd_map got epochs [518,518] > 517 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.159+0000 7f6c597fa700 3 client.37922.objecter handle_osd_map decoding incremental epoch 518 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.159+0000 7f6c597fa700 10 client.37922.objecter checking command 6 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.159+0000 7f6c597fa700 20 client.37922.objecter _get_session s=0x7f6c5c027a10 osd=3 4 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.159+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c027a10 osd=3 4 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.159+0000 7f6c597fa700 20 client.37922.objecter _recalc_command_target 6 no change, 0x7f6c5c027a10 2024-02-23T00:38:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.159+0000 7f6c597fa700 20 client.37922.objecter dump_active .. 0 homeless 2024-02-23T00:38:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.171+0000 7f6c597fa700 10 client.37922.objecter ms_handle_connect 0x7f6c5c061a90 2024-02-23T00:38:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.171+0000 7f6c5b7fe700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.87:6813/128059790 conn(0x7f6c5c061a90 0x7f6c5c11a450 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:55.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.176+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6c5c11f4f0 con 0x7f6c5c061a90 2024-02-23T00:38:55.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.176+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:38:55.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.176+0000 7f6c597fa700 10 client.37922.objecter _finish_command 6 = system:0 2024-02-23T00:38:55.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.176+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c027a10 osd=3 4 2024-02-23T00:38:55.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.176+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 3 6 2024-02-23T00:38:55.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:55.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 7 2024-02-23T00:38:55.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c027a10 osd=3 3 2024-02-23T00:38:55.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c027a10 osd=3 3 2024-02-23T00:38:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c027a10 osd=3 3 2024-02-23T00:38:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 7 2024-02-23T00:38:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c027a10 osd=3 3 2024-02-23T00:38:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 3 7 2024-02-23T00:38:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c027a10 osd=3 4 2024-02-23T00:38:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 10 client.37922.objecter _send_command 7 2024-02-23T00:38:55.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.206+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6c5c0610b0 con 0x7f6c5c061a90 2024-02-23T00:38:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.208+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6c5c0610b0 con 0x7f6c5c061a90 2024-02-23T00:38:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.208+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.208+0000 7f6c597fa700 10 client.37922.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.208+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c027a10 osd=3 4 2024-02-23T00:38:55.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.208+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 3 7 2024-02-23T00:38:55.211 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:38:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 8 2024-02-23T00:38:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6c5c11df10 0x7f6c5c11e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c0fe650 osd=4 3 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c0fe650 osd=4 3 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c0fe650 osd=4 3 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 8 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c0fe650 osd=4 3 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 4 8 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c0fe650 osd=4 4 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 10 client.37922.objecter _send_command 8 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c5c0fedf0 con 0x7f6c5c11df10 2024-02-23T00:38:55.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c60feb700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6c5c11df10 0x7f6c5c11e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c597fa700 10 client.37922.objecter ms_handle_connect 0x7f6c5c11df10 2024-02-23T00:38:55.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.209+0000 7f6c60feb700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6800/4175844757 conn(0x7f6c5c11df10 0x7f6c5c11e350 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:55.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.210+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6c5c0fedf0 con 0x7f6c5c11df10 2024-02-23T00:38:55.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.210+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:38:55.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.210+0000 7f6c597fa700 10 client.37922.objecter _finish_command 8 = system:0 2024-02-23T00:38:55.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.210+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c0fe650 osd=4 4 2024-02-23T00:38:55.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.210+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 4 8 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 9 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c0fe650 osd=4 3 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c0fe650 osd=4 3 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c0fe650 osd=4 3 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 9 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c0fe650 osd=4 3 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 4 9 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c0fe650 osd=4 4 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 10 client.37922.objecter _send_command 9 2024-02-23T00:38:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.237+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6c5c11f040 con 0x7f6c5c11df10 2024-02-23T00:38:55.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.239+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6c5c11f040 con 0x7f6c5c11df10 2024-02-23T00:38:55.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.239+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:55.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.239+0000 7f6c597fa700 10 client.37922.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.239+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c0fe650 osd=4 4 2024-02-23T00:38:55.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.239+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 4 9 2024-02-23T00:38:55.241 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.239+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.239+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 10 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.239+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6c5c121720 0x7f6c5c123b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c11e870 osd=5 3 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c11e870 osd=5 3 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c11e870 osd=5 3 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 10 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c11e870 osd=5 3 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 5 10 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c11e870 osd=5 4 2024-02-23T00:38:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c62a4e700 10 client.37922.objecter _send_command 10 2024-02-23T00:38:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c5c11efb0 con 0x7f6c5c121720 2024-02-23T00:38:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c5bfff700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6c5c121720 0x7f6c5c123b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c597fa700 10 client.37922.objecter ms_handle_connect 0x7f6c5c121720 2024-02-23T00:38:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.240+0000 7f6c5bfff700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6804/2550013795 conn(0x7f6c5c121720 0x7f6c5c123b10 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.241+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6c5c11efb0 con 0x7f6c5c121720 2024-02-23T00:38:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.241+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:38:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.241+0000 7f6c597fa700 10 client.37922.objecter _finish_command 10 = system:0 2024-02-23T00:38:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.241+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c11e870 osd=5 4 2024-02-23T00:38:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.241+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 5 10 2024-02-23T00:38:55.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 11 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c11e870 osd=5 3 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c11e870 osd=5 3 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c11e870 osd=5 3 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 11 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c11e870 osd=5 3 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 5 11 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c11e870 osd=5 4 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 10 client.37922.objecter _send_command 11 2024-02-23T00:38:55.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.268+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6c5c11efb0 con 0x7f6c5c121720 2024-02-23T00:38:55.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6c5c11efb0 con 0x7f6c5c121720 2024-02-23T00:38:55.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:55.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c597fa700 10 client.37922.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c11e870 osd=5 4 2024-02-23T00:38:55.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 5 11 2024-02-23T00:38:55.273 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:38:55.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:55.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 12 2024-02-23T00:38:55.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6c5c124550 0x7f6c5c1249b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c11efb0 osd=6 3 2024-02-23T00:38:55.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c11efb0 osd=6 3 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c11efb0 osd=6 3 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 12 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c11efb0 osd=6 3 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 6 12 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c11efb0 osd=6 4 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 10 client.37922.objecter _send_command 12 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c5c129950 con 0x7f6c5c124550 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c5b7fe700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6c5c124550 0x7f6c5c1249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c597fa700 10 client.37922.objecter ms_handle_connect 0x7f6c5c124550 2024-02-23T00:38:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.271+0000 7f6c5b7fe700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6808/2509408498 conn(0x7f6c5c124550 0x7f6c5c1249b0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:55.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.272+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6c5c129950 con 0x7f6c5c124550 2024-02-23T00:38:55.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.272+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:38:55.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.272+0000 7f6c597fa700 10 client.37922.objecter _finish_command 12 = system:0 2024-02-23T00:38:55.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.272+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c11efb0 osd=6 4 2024-02-23T00:38:55.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.272+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 6 12 2024-02-23T00:38:55.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:55.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 13 2024-02-23T00:38:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c11efb0 osd=6 3 2024-02-23T00:38:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c11efb0 osd=6 3 2024-02-23T00:38:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c11efb0 osd=6 3 2024-02-23T00:38:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 13 2024-02-23T00:38:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c11efb0 osd=6 3 2024-02-23T00:38:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 6 13 2024-02-23T00:38:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c11efb0 osd=6 4 2024-02-23T00:38:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 10 client.37922.objecter _send_command 13 2024-02-23T00:38:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.298+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6c5c124f60 con 0x7f6c5c124550 2024-02-23T00:38:55.302 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:38:55.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.300+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6c5c124f60 con 0x7f6c5c124550 2024-02-23T00:38:55.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.300+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:55.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.300+0000 7f6c597fa700 10 client.37922.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.300+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c11efb0 osd=6 4 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.300+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 6 13 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 14 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6812/373787000 conn(0x7f6c5c128ce0 0x7f6c5c129140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c124f60 osd=7 3 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c124f60 osd=7 3 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c124f60 osd=7 3 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 14 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c124f60 osd=7 3 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 7 14 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c124f60 osd=7 4 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 10 client.37922.objecter _send_command 14 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c5c11dd10 con 0x7f6c5c128ce0 2024-02-23T00:38:55.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c60feb700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6812/373787000 conn(0x7f6c5c128ce0 0x7f6c5c129140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:38:55.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c597fa700 10 client.37922.objecter ms_handle_connect 0x7f6c5c128ce0 2024-02-23T00:38:55.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.301+0000 7f6c60feb700 1 --2- 172.21.15.87:0/1238097181 >> v2:172.21.15.171:6812/373787000 conn(0x7f6c5c128ce0 0x7f6c5c129140 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:38:55.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.302+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6c5c11dd10 con 0x7f6c5c128ce0 2024-02-23T00:38:55.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.302+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:38:55.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.302+0000 7f6c597fa700 10 client.37922.objecter _finish_command 14 = system:0 2024-02-23T00:38:55.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.302+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c124f60 osd=7 4 2024-02-23T00:38:55.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.302+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 7 14 2024-02-23T00:38:55.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 10 client.37922.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:38:55.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign -1 15 2024-02-23T00:38:55.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c124f60 osd=7 3 2024-02-23T00:38:55.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c124f60 osd=7 3 2024-02-23T00:38:55.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 20 client.37922.objecter _get_session s=0x7f6c5c124f60 osd=7 3 2024-02-23T00:38:55.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_remove -1 15 2024-02-23T00:38:55.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 20 client.37922.objecter get_session s=0x7f6c5c124f60 osd=7 3 2024-02-23T00:38:55.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 15 client.37922.objecter _session_command_op_assign 7 15 2024-02-23T00:38:55.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 20 client.37922.objecter put_session s=0x7f6c5c124f60 osd=7 4 2024-02-23T00:38:55.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 10 client.37922.objecter _send_command 15 2024-02-23T00:38:55.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.316+0000 7f6c62a4e700 1 -- 172.21.15.87:0/1238097181 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6c5c129630 con 0x7f6c5c128ce0 2024-02-23T00:38:55.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.318+0000 7f6c597fa700 1 -- 172.21.15.87:0/1238097181 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f6c5c129630 con 0x7f6c5c128ce0 2024-02-23T00:38:55.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.318+0000 7f6c597fa700 10 client.37922.objecter ms_dispatch 0x7f6c5c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:38:55.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.318+0000 7f6c597fa700 10 client.37922.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.319 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.318+0000 7f6c597fa700 20 client.37922.objecter put_session s=0x7f6c5c124f60 osd=7 4 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:38:55.318+0000 7f6c597fa700 15 client.37922.objecter _session_command_op_remove 7 15 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.320 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:38:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:53.712382+0000 mgr.y (mgr.24437) 483 : cluster [DBG] pgmap v790: 278 pgs: 1 active+recovery_wait+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 5 active+remapped+backfill_toofull, 1 active+recovery_wait, 6 active+remapped+backfill_wait, 5 active+recovering, 22 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_wait, 46 active+undersized, 1 active+recovering+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 163 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 2.0 KiB/s wr, 0 op/s; 124/1260 objects degraded (9.841%); 188/1260 objects misplaced (14.921%); 185 B/s, 26 objects/s recovering 2024-02-23T00:38:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:54.155215+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e517: 8 total, 7 up, 8 in 2024-02-23T00:38:55.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:55.119960+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? 172.21.15.87:0/1238097181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:38:55.558 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:55.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:55.662 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:38:55.690 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:55.717 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:38:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:55.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:38:55.827 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:55.857 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:55.877 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:55.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:55.980 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:56.008 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:56.027 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:56.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:56.131 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:56.158 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:56.178 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:56.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:56.194 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-23T00:38:56.221 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:56.282 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:56.308 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:56.327 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:56.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:56.430 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:55.162110+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e518: 8 total, 7 up, 8 in 2024-02-23T00:38:56.457 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:56.475 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:56.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:38:56.580 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:55.162110+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e518: 8 total, 7 up, 8 in 2024-02-23T00:38:56.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:55.162110+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e518: 8 total, 7 up, 8 in 2024-02-23T00:38:56.606 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:56.623 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:38:56.624 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:56.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:56.728 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:56.754 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:56.772 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:56.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:38:56.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:38:56.876 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:56.904 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:57.101 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:57.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:57.185 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:55.713958+0000 mgr.y (mgr.24437) 484 : cluster [DBG] pgmap v793: 278 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 29 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 46 active+undersized, 174 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 103/1260 objects degraded (8.175%); 115/1260 objects misplaced (9.127%); 978 KiB/s, 24 objects/s recovering 2024-02-23T00:38:57.185 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:57.141438+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e519: 8 total, 7 up, 8 in 2024-02-23T00:38:57.209 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:57.250 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:57.352 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:57.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:57.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:57.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:55.713958+0000 mgr.y (mgr.24437) 484 : cluster [DBG] pgmap v793: 278 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 29 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 46 active+undersized, 174 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 103/1260 objects degraded (8.175%); 115/1260 objects misplaced (9.127%); 978 KiB/s, 24 objects/s recovering 2024-02-23T00:38:57.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:57.141438+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e519: 8 total, 7 up, 8 in 2024-02-23T00:38:57.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:38:57.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:55.713958+0000 mgr.y (mgr.24437) 484 : cluster [DBG] pgmap v793: 278 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 29 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 46 active+undersized, 174 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 103/1260 objects degraded (8.175%); 115/1260 objects misplaced (9.127%); 978 KiB/s, 24 objects/s recovering 2024-02-23T00:38:57.487 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:57.141438+0000 mon.a (mon.0) 2883 : cluster [DBG] osdmap e519: 8 total, 7 up, 8 in 2024-02-23T00:38:57.488 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:57.696 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:57.696 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:38:58.566 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:38:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:58.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:38:58.672 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:58.705 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:58.725 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:58.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:58.829 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:38:58.855 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:38:58.860 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:58.881 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:38:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:58.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:38:58.985 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:59.017 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:59.059 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:59.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:59.162 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:38:59.205 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:59.366 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:38:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:59.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:38:59.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:38:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:57.714662+0000 mgr.y (mgr.24437) 485 : cluster [DBG] pgmap v795: 278 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 29 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 46 active+undersized, 174 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 103/1260 objects degraded (8.175%); 115/1260 objects misplaced (9.127%); 859 KiB/s, 21 objects/s recovering 2024-02-23T00:38:59.470 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:38:59.495 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:59.514 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:38:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:59.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:38:59.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:38:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:57.714662+0000 mgr.y (mgr.24437) 485 : cluster [DBG] pgmap v795: 278 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 29 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 46 active+undersized, 174 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 103/1260 objects degraded (8.175%); 115/1260 objects misplaced (9.127%); 859 KiB/s, 21 objects/s recovering 2024-02-23T00:38:59.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:38:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:57.714662+0000 mgr.y (mgr.24437) 485 : cluster [DBG] pgmap v795: 278 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 29 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 46 active+undersized, 174 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 103/1260 objects degraded (8.175%); 115/1260 objects misplaced (9.127%); 859 KiB/s, 21 objects/s recovering 2024-02-23T00:38:59.618 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:59.644 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:59.663 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:59.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:59.766 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:38:59.793 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:59.811 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:38:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:59.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:38:59.915 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:38:59.941 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:38:59.959 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:38:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:38:59.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:00.063 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:00.089 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:00.107 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:00.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:00.198 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-02-23T00:39:00.198 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (18042 ms) 2024-02-23T00:39:00.198 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-02-23T00:39:00.198 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23083 ms) 2024-02-23T00:39:00.198 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-02-23T00:39:00.198 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (2881 ms) 2024-02-23T00:39:00.198 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3005 ms) 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3019 ms) 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3026 ms) 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (3960 ms) 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39145 ms) 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-02-23T00:39:00.199 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17092 ms) 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3809 ms) 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3060 ms) 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.200 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24922 ms) 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15016 ms) 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5049 ms) 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: waiting for scrubs... 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: done waiting 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24950 ms) 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10048 ms) 2024-02-23T00:39:00.201 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-02-23T00:39:00.202 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.202 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8970 ms) 2024-02-23T00:39:00.202 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-02-23T00:39:00.202 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.202 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9048 ms) 2024-02-23T00:39:00.202 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-02-23T00:39:00.202 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:39:00.211 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:00.238 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:00.255 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:00.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:00.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:59.181132+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e520: 8 total, 7 up, 8 in 2024-02-23T00:39:00.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:59.249447+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:00.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:59.249763+0000 mon.a (mon.0) 2885 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:00.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:59.250491+0000 mon.b (mon.1) 422 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-83"}]: dispatch 2024-02-23T00:39:00.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:38:59.250657+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-83"}]: dispatch 2024-02-23T00:39:00.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:59.181132+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e520: 8 total, 7 up, 8 in 2024-02-23T00:39:00.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:59.249447+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:00.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:59.249763+0000 mon.a (mon.0) 2885 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:00.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:59.250491+0000 mon.b (mon.1) 422 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-83"}]: dispatch 2024-02-23T00:39:00.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:38:59.250657+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-83"}]: dispatch 2024-02-23T00:39:00.359 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:00.390 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:59.181132+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e520: 8 total, 7 up, 8 in 2024-02-23T00:39:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:59.249447+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:59.249763+0000 mon.a (mon.0) 2885 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:59.250491+0000 mon.b (mon.1) 422 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-83"}]: dispatch 2024-02-23T00:39:00.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:38:59.250657+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-83"}]: dispatch 2024-02-23T00:39:01.139 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:01.142 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:01.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:01.243 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:01.303 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:01.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.375+0000 7f8aa508f700 1 -- 172.21.15.87:0/129600121 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa0104600 msgr2=0x7f8aa01069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:01.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.375+0000 7f8aa508f700 1 --2- 172.21.15.87:0/129600121 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa0104600 0x7f8aa01069f0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 rx=0x7f8a98009be0 tx=0x7f8a98004d10).stop 2024-02-23T00:39:01.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.375+0000 7f8aa508f700 1 -- 172.21.15.87:0/129600121 shutdown_connections 2024-02-23T00:39:01.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.375+0000 7f8aa508f700 1 --2- 172.21.15.87:0/129600121 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa0104600 0x7f8aa01069f0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:01.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.375+0000 7f8aa508f700 1 --2- 172.21.15.87:0/129600121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8aa00f6aa0 0x7f8aa00f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:01.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.375+0000 7f8aa508f700 1 --2- 172.21.15.87:0/129600121 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa00f7370 0x7f8aa0104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:01.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.375+0000 7f8aa508f700 1 -- 172.21.15.87:0/129600121 >> 172.21.15.87:0/129600121 conn(0x7f8aa00f2820 msgr2=0x7f8aa00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:01.386 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.375+0000 7f8aa508f700 1 -- 172.21.15.87:0/129600121 shutdown_connections 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.375+0000 7f8aa508f700 1 -- 172.21.15.87:0/129600121 wait complete. 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 librados: starting msgr at 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 librados: starting objecter 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 Processor -- start 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 -- start start 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 librados: setting wanted keys 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 librados: calling monclient init 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8aa00f6aa0 0x7f8aa01101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa00f7370 0x7f8aa01106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa0104600 0x7f8aa010a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa010a8b0 con 0x7f8aa0104600 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa010aa20 con 0x7f8aa00f6aa0 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.376+0000 7f8aa508f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa010abe0 con 0x7f8aa00f7370 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9f59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa0104600 0x7f8aa010a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:01.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9ed9d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8aa00f6aa0 0x7f8aa01101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9f59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa0104600 0x7f8aa010a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39572/0 (socket says 172.21.15.87:39572) 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9f59e700 1 -- 172.21.15.87:0/2972702340 learned_addr learned my addr 172.21.15.87:0/2972702340 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9e59c700 1 --2- 172.21.15.87:0/2972702340 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa00f7370 0x7f8aa01106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9ed9d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8aa00f6aa0 0x7f8aa01101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.87:38614/0 (socket says 172.21.15.87:38614) 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9e59c700 1 -- 172.21.15.87:0/2972702340 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8aa00f6aa0 msgr2=0x7f8aa01101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9e59c700 1 --2- 172.21.15.87:0/2972702340 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8aa00f6aa0 0x7f8aa01101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9e59c700 1 -- 172.21.15.87:0/2972702340 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa0104600 msgr2=0x7f8aa010a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9e59c700 1 --2- 172.21.15.87:0/2972702340 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa0104600 0x7f8aa010a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9e59c700 1 -- 172.21.15.87:0/2972702340 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a94005040 con 0x7f8aa00f7370 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a9e59c700 1 --2- 172.21.15.87:0/2972702340 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa00f7370 0x7f8aa01106c0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 rx=0x7f8a94004920 tx=0x7f8a94009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a87fff700 10 client.?.objecter ms_handle_connect 0x7f8aa00f7370 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a87fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:01.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a87fff700 1 -- 172.21.15.87:0/2972702340 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400e030 con 0x7f8aa00f7370 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a87fff700 1 -- 172.21.15.87:0/2972702340 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a9400f040 con 0x7f8aa00f7370 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a980097c0 con 0x7f8aa00f7370 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8a87fff700 1 -- 172.21.15.87:0/2972702340 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a94014900 con 0x7f8aa00f7370 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8aa508f700 10 client.37928.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa0117000 con 0x7f8aa00f7370 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.377+0000 7f8aa508f700 1 librados: init done 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.379+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa005fa60 con 0x7f8aa00f7370 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.379+0000 7f8a87fff700 1 -- 172.21.15.87:0/2972702340 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f8a94010500 con 0x7f8aa00f7370 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.379+0000 7f8a87fff700 1 --2- 172.21.15.87:0/2972702340 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a88062c90 0x7f8a88065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.379+0000 7f8a9ed9d700 1 --2- 172.21.15.87:0/2972702340 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a88062c90 0x7f8a88065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.379+0000 7f8a87fff700 1 -- 172.21.15.87:0/2972702340 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(522..522 src has 173..522) v4 ==== 9057+0+0 (secure 0 0 0) 0x7f8a9407d5a0 con 0x7f8aa00f7370 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.379+0000 7f8a87fff700 10 client.37928.objecter ms_dispatch 0x7f8aa0001550 osd_map(522..522 src has 173..522) v4 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.379+0000 7f8a87fff700 3 client.37928.objecter handle_osd_map got epochs [522,522] > 0 2024-02-23T00:39:01.389 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.379+0000 7f8a87fff700 3 client.37928.objecter handle_osd_map decoding full epoch 522 2024-02-23T00:39:01.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.380+0000 7f8a87fff700 20 client.37928.objecter dump_active .. 0 homeless 2024-02-23T00:39:01.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.380+0000 7f8a87fff700 10 client.37928.objecter ms_handle_connect 0x7f8a88062c90 2024-02-23T00:39:01.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.380+0000 7f8a9ed9d700 1 --2- 172.21.15.87:0/2972702340 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a88062c90 0x7f8a88065150 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f8a90004e60 tx=0x7f8a9000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:01.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.384+0000 7f8a87fff700 1 -- 172.21.15.87:0/2972702340 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f8a94050700 con 0x7f8aa00f7370 2024-02-23T00:39:01.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:38:59.715289+0000 mgr.y (mgr.24437) 486 : cluster [DBG] pgmap v797: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 39 active+undersized, 152 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 920 B/s rd, 0 op/s; 103/1260 objects degraded (8.175%); 115/1260 objects misplaced (9.127%); 704 KiB/s, 17 objects/s recovering 2024-02-23T00:39:01.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:00.190861+0000 mon.a (mon.0) 2887 : cluster [DBG] osdmap e521: 8 total, 7 up, 8 in 2024-02-23T00:39:01.443 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:39:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:39:01.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.554+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8aa0109480 con 0x7f8a88062c90 2024-02-23T00:39:01.573 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:39:01.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.570+0000 7f8a87fff700 1 -- 172.21.15.87:0/2972702340 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+619801 (secure 0 0 0) 0x7f8aa0109480 con 0x7f8a88062c90 2024-02-23T00:39:01.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 10 librados: watch_flush enter 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 10 librados: watch_flush exit 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 20 client.37928.objecter shutdown clearing up homeless session... 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 10 client.37928.objecter successfully canceled tick 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a88062c90 msgr2=0x7f8a88065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 --2- 172.21.15.87:0/2972702340 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a88062c90 0x7f8a88065150 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 rx=0x7f8a90004e60 tx=0x7f8a9000b040).stop 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa00f7370 msgr2=0x7f8aa01106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 --2- 172.21.15.87:0/2972702340 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa00f7370 0x7f8aa01106c0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 rx=0x7f8a94004920 tx=0x7f8a94009cb0).stop 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 shutdown_connections 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 --2- 172.21.15.87:0/2972702340 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8a88062c90 0x7f8a88065150 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 --2- 172.21.15.87:0/2972702340 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8aa0104600 0x7f8aa010a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 --2- 172.21.15.87:0/2972702340 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8aa00f6aa0 0x7f8aa01101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:01.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 --2- 172.21.15.87:0/2972702340 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8aa00f7370 0x7f8aa01106c0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:01.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.574+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 >> 172.21.15.87:0/2972702340 conn(0x7f8aa00f2820 msgr2=0x7f8aa00f4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:01.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.575+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 shutdown_connections 2024-02-23T00:39:01.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.575+0000 7f8aa508f700 1 -- 172.21.15.87:0/2972702340 wait complete. 2024-02-23T00:39:01.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:01.575+0000 7f8aa508f700 1 librados: shutdown 2024-02-23T00:39:01.591 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:39:01.591 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:38:59.715289+0000 mgr.y (mgr.24437) 486 : cluster [DBG] pgmap v797: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 39 active+undersized, 152 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 920 B/s rd, 0 op/s; 103/1260 objects degraded (8.175%); 115/1260 objects misplaced (9.127%); 704 KiB/s, 17 objects/s recovering 2024-02-23T00:39:01.591 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:00.190861+0000 mon.a (mon.0) 2887 : cluster [DBG] osdmap e521: 8 total, 7 up, 8 in 2024-02-23T00:39:01.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:38:59.715289+0000 mgr.y (mgr.24437) 486 : cluster [DBG] pgmap v797: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 39 active+undersized, 152 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 920 B/s rd, 0 op/s; 103/1260 objects degraded (8.175%); 115/1260 objects misplaced (9.127%); 704 KiB/s, 17 objects/s recovering 2024-02-23T00:39:01.592 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:00.190861+0000 mon.a (mon.0) 2887 : cluster [DBG] osdmap e521: 8 total, 7 up, 8 in 2024-02-23T00:39:02.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:01.049913+0000 mgr.y (mgr.24437) 487 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:02.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:01.198645+0000 mon.a (mon.0) 2888 : cluster [DBG] osdmap e522: 8 total, 7 up, 8 in 2024-02-23T00:39:02.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:01.201306+0000 mon.b (mon.1) 423 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:02.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:01.203770+0000 mon.a (mon.0) 2889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:02.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:02.138335+0000 mon.a (mon.0) 2890 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:02.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:02.138429+0000 mon.a (mon.0) 2891 : cluster [WRN] Health check update: Degraded data redundancy: 103/1260 objects degraded (8.175%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:02.214 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:02.144484+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:02.215 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:02.144601+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e523: 8 total, 7 up, 8 in 2024-02-23T00:39:02.453 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:39:02.454 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:39:02.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:02] "GET /metrics HTTP/1.1" 200 191689 "" "Prometheus/2.33.4" 2024-02-23T00:39:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:02.151173+0000 mon.b (mon.1) 424 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:39:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:02.159300+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:02.161447+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:02.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:01.049913+0000 mgr.y (mgr.24437) 487 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:01.198645+0000 mon.a (mon.0) 2888 : cluster [DBG] osdmap e522: 8 total, 7 up, 8 in 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:01.201306+0000 mon.b (mon.1) 423 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:01.203770+0000 mon.a (mon.0) 2889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:02.138335+0000 mon.a (mon.0) 2890 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:02.138429+0000 mon.a (mon.0) 2891 : cluster [WRN] Health check update: Degraded data redundancy: 103/1260 objects degraded (8.175%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:02.144484+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:02.144601+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e523: 8 total, 7 up, 8 in 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:02.151173+0000 mon.b (mon.1) 424 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:02.159300+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:02.161447+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:01.049913+0000 mgr.y (mgr.24437) 487 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:01.198645+0000 mon.a (mon.0) 2888 : cluster [DBG] osdmap e522: 8 total, 7 up, 8 in 2024-02-23T00:39:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:01.201306+0000 mon.b (mon.1) 423 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:01.203770+0000 mon.a (mon.0) 2889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:02.138335+0000 mon.a (mon.0) 2890 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:02.138429+0000 mon.a (mon.0) 2891 : cluster [WRN] Health check update: Degraded data redundancy: 103/1260 objects degraded (8.175%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:02.144484+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:02.144601+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e523: 8 total, 7 up, 8 in 2024-02-23T00:39:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:02.151173+0000 mon.b (mon.1) 424 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:39:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:02.159300+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:02.161447+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:02.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.925+0000 7fc3463f3700 1 -- 172.21.15.87:0/2707888170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400f7400 msgr2=0x7fc340103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:02.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.925+0000 7fc3463f3700 1 --2- 172.21.15.87:0/2707888170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400f7400 0x7fc340103a20 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 rx=0x7fc338009be0 tx=0x7fc338004d10).stop 2024-02-23T00:39:02.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.925+0000 7fc3463f3700 1 -- 172.21.15.87:0/2707888170 shutdown_connections 2024-02-23T00:39:02.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.925+0000 7fc3463f3700 1 --2- 172.21.15.87:0/2707888170 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400f7400 0x7fc340103a20 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:02.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.925+0000 7fc3463f3700 1 --2- 172.21.15.87:0/2707888170 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3400fcbb0 0x7fc3400fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:02.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.925+0000 7fc3463f3700 1 --2- 172.21.15.87:0/2707888170 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3400fd510 0x7fc3400f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.925+0000 7fc3463f3700 1 -- 172.21.15.87:0/2707888170 >> 172.21.15.87:0/2707888170 conn(0x7fc3400f2820 msgr2=0x7fc3400f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.925+0000 7fc3463f3700 1 -- 172.21.15.87:0/2707888170 shutdown_connections 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.925+0000 7fc3463f3700 1 -- 172.21.15.87:0/2707888170 wait complete. 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 librados: starting msgr at 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 librados: starting objecter 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 Processor -- start 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 -- start start 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 librados: setting wanted keys 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 librados: calling monclient init 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3400f7400 0x7fc34010ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3400fcbb0 0x7fc34010d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400fd510 0x7fc340107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc340107540 con 0x7fc3400fd510 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3401076b0 con 0x7fc3400f7400 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc3463f3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc340107870 con 0x7fc3400fcbb0 2024-02-23T00:39:02.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc344990700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400fd510 0x7fc340107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc344990700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400fd510 0x7fc340107050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39592/0 (socket says 172.21.15.87:39592) 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc344990700 1 -- 172.21.15.87:0/3398380855 learned_addr learned my addr 172.21.15.87:0/3398380855 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc33f7fe700 1 --2- 172.21.15.87:0/3398380855 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3400fcbb0 0x7fc34010d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.926+0000 7fc33ffff700 1 --2- 172.21.15.87:0/3398380855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3400f7400 0x7fc34010ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc344990700 1 -- 172.21.15.87:0/3398380855 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3400fcbb0 msgr2=0x7fc34010d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc344990700 1 --2- 172.21.15.87:0/3398380855 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3400fcbb0 0x7fc34010d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc344990700 1 -- 172.21.15.87:0/3398380855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3400f7400 msgr2=0x7fc34010ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc344990700 1 --2- 172.21.15.87:0/3398380855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3400f7400 0x7fc34010ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc344990700 1 -- 172.21.15.87:0/3398380855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334005040 con 0x7fc3400fd510 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc344990700 1 --2- 172.21.15.87:0/3398380855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400fd510 0x7fc340107050 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 rx=0x7fc338004ed0 tx=0x7fc338004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc33d7fa700 10 client.?.objecter ms_handle_connect 0x7fc3400fd510 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc33d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc33d7fa700 1 -- 172.21.15.87:0/3398380855 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33800e040 con 0x7fc3400fd510 2024-02-23T00:39:02.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3380097c0 con 0x7fc3400fd510 2024-02-23T00:39:02.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc33d7fa700 1 -- 172.21.15.87:0/3398380855 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc338018040 con 0x7fc3400fd510 2024-02-23T00:39:02.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc33d7fa700 1 -- 172.21.15.87:0/3398380855 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33801c3f0 con 0x7fc3400fd510 2024-02-23T00:39:02.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc3463f3700 10 client.27468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:02.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc340107d10 con 0x7fc3400fd510 2024-02-23T00:39:02.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.927+0000 7fc3463f3700 1 librados: init done 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.928+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc340107f00 con 0x7fc3400fd510 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.928+0000 7fc33d7fa700 1 -- 172.21.15.87:0/3398380855 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc338007b20 con 0x7fc3400fd510 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.929+0000 7fc33d7fa700 1 --2- 172.21.15.87:0/3398380855 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc328062c90 0x7fc328065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.929+0000 7fc33d7fa700 1 -- 172.21.15.87:0/3398380855 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(523..523 src has 173..523) v4 ==== 9070+0+0 (secure 0 0 0) 0x7fc33807e560 con 0x7fc3400fd510 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.929+0000 7fc33d7fa700 10 client.27468.objecter ms_dispatch 0x7fc340001550 osd_map(523..523 src has 173..523) v4 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.929+0000 7fc33d7fa700 3 client.27468.objecter handle_osd_map got epochs [523,523] > 0 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.929+0000 7fc33d7fa700 3 client.27468.objecter handle_osd_map decoding full epoch 523 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.929+0000 7fc33ffff700 1 --2- 172.21.15.87:0/3398380855 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc328062c90 0x7fc328065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.929+0000 7fc33d7fa700 20 client.27468.objecter dump_active .. 0 homeless 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.929+0000 7fc33d7fa700 10 client.27468.objecter ms_handle_connect 0x7fc328062c90 2024-02-23T00:39:02.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.929+0000 7fc33ffff700 1 --2- 172.21.15.87:0/3398380855 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc328062c90 0x7fc328065150 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7fc330004dc0 tx=0x7fc33000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:02.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:02.933+0000 7fc33d7fa700 1 -- 172.21.15.87:0/3398380855 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc3380516f0 con 0x7fc3400fd510 2024-02-23T00:39:03.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.109+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc34005fa60 con 0x7fc3400fd510 2024-02-23T00:39:03.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.147+0000 7fc33d7fa700 1 -- 172.21.15.87:0/3398380855 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v524) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc338021030 con 0x7fc3400fd510 2024-02-23T00:39:03.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.158+0000 7fc3463f3700 10 librados: watch_flush enter 2024-02-23T00:39:03.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.166+0000 7fc3463f3700 10 librados: watch_flush exit 2024-02-23T00:39:03.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.166+0000 7fc3463f3700 20 client.27468.objecter shutdown clearing up homeless session... 2024-02-23T00:39:03.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.166+0000 7fc3463f3700 10 client.27468.objecter successfully canceled tick 2024-02-23T00:39:03.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.167+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc328062c90 msgr2=0x7fc328065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:03.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.167+0000 7fc3463f3700 1 --2- 172.21.15.87:0/3398380855 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc328062c90 0x7fc328065150 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7fc330004dc0 tx=0x7fc33000b040).stop 2024-02-23T00:39:03.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.167+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400fd510 msgr2=0x7fc340107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:03.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.167+0000 7fc3463f3700 1 --2- 172.21.15.87:0/3398380855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400fd510 0x7fc340107050 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 rx=0x7fc338004ed0 tx=0x7fc338004d10).stop 2024-02-23T00:39:03.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.171+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 shutdown_connections 2024-02-23T00:39:03.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.171+0000 7fc3463f3700 1 --2- 172.21.15.87:0/3398380855 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc328062c90 0x7fc328065150 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:03.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.171+0000 7fc3463f3700 1 --2- 172.21.15.87:0/3398380855 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc3400fd510 0x7fc340107050 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:03.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.171+0000 7fc3463f3700 1 --2- 172.21.15.87:0/3398380855 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc3400f7400 0x7fc34010ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:03.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.171+0000 7fc3463f3700 1 --2- 172.21.15.87:0/3398380855 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc3400fcbb0 0x7fc34010d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:03.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.171+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 >> 172.21.15.87:0/3398380855 conn(0x7fc3400f2820 msgr2=0x7fc3400f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:03.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.172+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 shutdown_connections 2024-02-23T00:39:03.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.172+0000 7fc3463f3700 1 -- 172.21.15.87:0/3398380855 wait complete. 2024-02-23T00:39:03.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:03.173+0000 7fc3463f3700 1 librados: shutdown 2024-02-23T00:39:03.175 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:39:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:01.555040+0000 mgr.y (mgr.24437) 488 : audit [DBG] from='client.37928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:39:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:01.716069+0000 mgr.y (mgr.24437) 489 : cluster [DBG] pgmap v800: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s; 103/1257 objects degraded (8.194%); 114/1257 objects misplaced (9.069%) 2024-02-23T00:39:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:03.109691+0000 mon.a (mon.0) 2895 : audit [INF] from='client.? 172.21.15.87:0/3398380855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:03.141418+0000 mon.a (mon.0) 2896 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:03.145880+0000 mon.a (mon.0) 2897 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:39:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:03.145964+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? 172.21.15.87:0/3398380855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:39:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:03.146006+0000 mon.a (mon.0) 2899 : cluster [DBG] osdmap e524: 8 total, 7 up, 8 in 2024-02-23T00:39:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:01.555040+0000 mgr.y (mgr.24437) 488 : audit [DBG] from='client.37928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:39:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:01.716069+0000 mgr.y (mgr.24437) 489 : cluster [DBG] pgmap v800: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s; 103/1257 objects degraded (8.194%); 114/1257 objects misplaced (9.069%) 2024-02-23T00:39:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:03.109691+0000 mon.a (mon.0) 2895 : audit [INF] from='client.? 172.21.15.87:0/3398380855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:03.141418+0000 mon.a (mon.0) 2896 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:03.145880+0000 mon.a (mon.0) 2897 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:39:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:03.145964+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? 172.21.15.87:0/3398380855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:39:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:03.146006+0000 mon.a (mon.0) 2899 : cluster [DBG] osdmap e524: 8 total, 7 up, 8 in 2024-02-23T00:39:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:01.555040+0000 mgr.y (mgr.24437) 488 : audit [DBG] from='client.37928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:39:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:01.716069+0000 mgr.y (mgr.24437) 489 : cluster [DBG] pgmap v800: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s; 103/1257 objects degraded (8.194%); 114/1257 objects misplaced (9.069%) 2024-02-23T00:39:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:03.109691+0000 mon.a (mon.0) 2895 : audit [INF] from='client.? 172.21.15.87:0/3398380855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:03.141418+0000 mon.a (mon.0) 2896 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:03.145880+0000 mon.a (mon.0) 2897 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:39:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:03.145964+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? 172.21.15.87:0/3398380855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:39:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:03.146006+0000 mon.a (mon.0) 2899 : cluster [DBG] osdmap e524: 8 total, 7 up, 8 in 2024-02-23T00:39:04.045 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":799,"stamp":"2024-02-23T00:39:01.196629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4026223,"num_objects":419,"num_object_clones":9,"num_object_copies":1257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":103,"num_objects_misplaced":114,"num_objects_unfound":0,"num_objects_dirty":419,"num_whiteouts":0,"num_read":1938,"num_read_kb":1538,"num_write":1578,"num_write_kb":4655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":476,"num_bytes_recovered":8005733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2909,"ondisk_log_size":2909,"up":568,"acting":590,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":821,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2515828,"kb_used_data":60212,"kb_used_omap":0,"kb_used_meta":2455552,"kb_avail":747445388,"statfs":{"total":767960285184,"available":765384077312,"internally_reserved":0,"allocated":61657088,"data_stored":38921571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2514485248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8192,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":2,"num_write_kb":-7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-8192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.484035"},"pg_stats":[{"pgid":"225.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"225.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:39:01.188552+0000","last_change":"2024-02-23T00:39:01.188552+0000","last_active":"2024-02-23T00:39:01.188552+0000","last_peered":"2024-02-23T00:39:01.188552+0000","last_clean":"2024-02-23T00:39:01.188552+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:39:01.188552+0000","last_undegraded":"2024-02-23T00:39:01.188552+0000","last_fullsized":"2024-02-23T00:39:01.188552+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:01.188552+0000","last_clean_scrub_stamp":"2024-02-23T00:39:01.188552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"63.0","version":"485'86","reported_seq":284,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.230184+0000","last_change":"2024-02-23T00:36:37.106755+0000","last_active":"2024-02-23T00:38:59.230184+0000","last_peered":"2024-02-23T00:38:59.230184+0000","last_clean":"2024-02-23T00:36:33.365214+0000","last_became_active":"2024-02-23T00:36:37.106755+0000","last_became_peered":"2024-02-23T00:36:37.106755+0000","last_unstale":"2024-02-23T00:38:59.230184+0000","last_undegraded":"2024-02-23T00:36:37.100498+0000","last_fullsized":"2024-02-23T00:36:37.100377+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'68","last_scrub_stamp":"2024-02-23T00:34:57.645204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.645204+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":32,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1","version":"340'86","reported_seq":557,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.210218+0000","last_change":"2024-02-23T00:36:37.109838+0000","last_active":"2024-02-23T00:38:59.210218+0000","last_peered":"2024-02-23T00:38:59.210218+0000","last_clean":"2024-02-23T00:36:33.191340+0000","last_became_active":"2024-02-23T00:36:37.109838+0000","last_became_peered":"2024-02-23T00:36:37.109838+0000","last_unstale":"2024-02-23T00:38:59.210218+0000","last_undegraded":"2024-02-23T00:36:37.093708+0000","last_fullsized":"2024-02-23T00:36:37.093560+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'85","last_scrub_stamp":"2024-02-23T00:34:16.724630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:16.724630+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.2","version":"485'25","reported_seq":508,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.181381+0000","last_change":"2024-02-23T00:36:37.106211+0000","last_active":"2024-02-23T00:38:59.181381+0000","last_peered":"2024-02-23T00:38:59.181381+0000","last_clean":"2024-02-23T00:36:33.365672+0000","last_became_active":"2024-02-23T00:36:37.106211+0000","last_became_peered":"2024-02-23T00:36:37.106211+0000","last_unstale":"2024-02-23T00:38:59.181381+0000","last_undegraded":"2024-02-23T00:36:37.098859+0000","last_fullsized":"2024-02-23T00:36:37.098739+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'20","last_scrub_stamp":"2024-02-23T00:34:23.049395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.049395+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":15,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.3","version":"451'44","reported_seq":576,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.212631+0000","last_change":"2024-02-23T00:38:49.270351+0000","last_active":"2024-02-23T00:38:59.212631+0000","last_peered":"2024-02-23T00:38:59.212631+0000","last_clean":"2024-02-23T00:38:59.212631+0000","last_became_active":"2024-02-23T00:38:49.226347+0000","last_became_peered":"2024-02-23T00:38:49.226347+0000","last_unstale":"2024-02-23T00:38:59.212631+0000","last_undegraded":"2024-02-23T00:38:59.212631+0000","last_fullsized":"2024-02-23T00:38:59.212631+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'27","last_scrub_stamp":"2024-02-23T00:37:05.079149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.079149+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":78,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.4","version":"442'49","reported_seq":526,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.186811+0000","last_change":"2024-02-23T00:38:35.832624+0000","last_active":"2024-02-23T00:38:59.186811+0000","last_peered":"2024-02-23T00:38:59.186811+0000","last_clean":"2024-02-23T00:38:59.186811+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:38:59.186811+0000","last_undegraded":"2024-02-23T00:38:59.186811+0000","last_fullsized":"2024-02-23T00:38:59.186811+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'49","last_scrub_stamp":"2024-02-23T00:38:35.832551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.832551+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.5","version":"471'126","reported_seq":594,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.212807+0000","last_change":"2024-02-23T00:36:37.105105+0000","last_active":"2024-02-23T00:38:59.212807+0000","last_peered":"2024-02-23T00:38:59.212807+0000","last_clean":"2024-02-23T00:36:33.192144+0000","last_became_active":"2024-02-23T00:36:37.105105+0000","last_became_peered":"2024-02-23T00:36:37.105105+0000","last_unstale":"2024-02-23T00:38:59.212807+0000","last_undegraded":"2024-02-23T00:36:37.095637+0000","last_fullsized":"2024-02-23T00:36:37.094980+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'124","last_scrub_stamp":"2024-02-23T00:34:19.695449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.695449+0000","log_size":126,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":92,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":39,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.6","version":"484'56","reported_seq":537,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.197323+0000","last_change":"2024-02-23T00:36:37.120734+0000","last_active":"2024-02-23T00:38:59.197323+0000","last_peered":"2024-02-23T00:38:59.197323+0000","last_clean":"2024-02-23T00:36:33.194384+0000","last_became_active":"2024-02-23T00:36:37.120734+0000","last_became_peered":"2024-02-23T00:36:37.120734+0000","last_unstale":"2024-02-23T00:38:59.197323+0000","last_undegraded":"2024-02-23T00:36:37.106912+0000","last_fullsized":"2024-02-23T00:36:37.106799+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:34:31.733148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.733148+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.7","version":"503'99","reported_seq":565,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.186483+0000","last_change":"2024-02-23T00:36:45.610239+0000","last_active":"2024-02-23T00:38:59.186483+0000","last_peered":"2024-02-23T00:38:59.186483+0000","last_clean":"2024-02-23T00:38:59.186483+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:38:59.186483+0000","last_undegraded":"2024-02-23T00:38:59.186483+0000","last_fullsized":"2024-02-23T00:38:59.186483+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'92","last_scrub_stamp":"2024-02-23T00:34:18.124049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:18.124049+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2009,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":37,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.8","version":"484'73","reported_seq":67,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.195496+0000","last_change":"2024-02-23T00:38:53.080986+0000","last_active":"2024-02-23T00:38:59.195496+0000","last_peered":"2024-02-23T00:38:59.195496+0000","last_clean":"2024-02-23T00:38:59.195496+0000","last_became_active":"2024-02-23T00:38:49.222540+0000","last_became_peered":"2024-02-23T00:38:49.222540+0000","last_unstale":"2024-02-23T00:38:59.195496+0000","last_undegraded":"2024-02-23T00:38:59.195496+0000","last_fullsized":"2024-02-23T00:38:59.195496+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'71","last_scrub_stamp":"2024-02-23T00:35:41.651500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.651500+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.9","version":"485'29","reported_seq":511,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.180689+0000","last_change":"2024-02-23T00:38:47.260030+0000","last_active":"2024-02-23T00:38:59.180689+0000","last_peered":"2024-02-23T00:38:59.180689+0000","last_clean":"2024-02-23T00:38:59.180689+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:38:59.180689+0000","last_undegraded":"2024-02-23T00:38:59.180689+0000","last_fullsized":"2024-02-23T00:38:59.180689+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'29","last_scrub_stamp":"2024-02-23T00:38:47.259916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:47.259916+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.a","version":"503'23","reported_seq":507,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.186854+0000","last_change":"2024-02-23T00:38:00.735331+0000","last_active":"2024-02-23T00:38:59.186854+0000","last_peered":"2024-02-23T00:38:59.186854+0000","last_clean":"2024-02-23T00:38:59.186854+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:38:59.186854+0000","last_undegraded":"2024-02-23T00:38:59.186854+0000","last_fullsized":"2024-02-23T00:38:59.186854+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'22","last_scrub_stamp":"2024-02-23T00:38:00.735240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.735240+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1231,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.b","version":"503'42","reported_seq":593,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.232706+0000","last_change":"2024-02-23T00:38:51.156502+0000","last_active":"2024-02-23T00:38:59.232706+0000","last_peered":"2024-02-23T00:38:59.232706+0000","last_clean":"2024-02-23T00:38:59.232706+0000","last_became_active":"2024-02-23T00:38:50.185347+0000","last_became_peered":"2024-02-23T00:38:50.185347+0000","last_unstale":"2024-02-23T00:38:59.232706+0000","last_undegraded":"2024-02-23T00:38:59.232706+0000","last_fullsized":"2024-02-23T00:38:59.232706+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-23T00:33:37.391203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:37.391203+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":20,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,4],"avail_no_missing":["7","3","4"],"object_location_counts":[{"shards":"3,4,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.c","version":"422'97","reported_seq":569,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.225239+0000","last_change":"2024-02-23T00:36:37.106629+0000","last_active":"2024-02-23T00:38:59.225239+0000","last_peered":"2024-02-23T00:38:59.225239+0000","last_clean":"2024-02-23T00:36:33.366529+0000","last_became_active":"2024-02-23T00:36:37.106629+0000","last_became_peered":"2024-02-23T00:36:37.106629+0000","last_unstale":"2024-02-23T00:38:59.225239+0000","last_undegraded":"2024-02-23T00:36:37.096711+0000","last_fullsized":"2024-02-23T00:36:37.096581+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'76","last_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_deep_scrub":"397'76","last_deep_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.098352+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.d","version":"503'46","reported_seq":594,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.187340+0000","last_change":"2024-02-23T00:38:49.272366+0000","last_active":"2024-02-23T00:38:59.187340+0000","last_peered":"2024-02-23T00:38:59.187340+0000","last_clean":"2024-02-23T00:38:59.187340+0000","last_became_active":"2024-02-23T00:38:49.184383+0000","last_became_peered":"2024-02-23T00:38:49.184383+0000","last_unstale":"2024-02-23T00:38:59.187340+0000","last_undegraded":"2024-02-23T00:38:59.187340+0000","last_fullsized":"2024-02-23T00:38:59.187340+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'39","last_scrub_stamp":"2024-02-23T00:38:13.724131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.724131+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":22,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.e","version":"405'77","reported_seq":671,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.184629+0000","last_change":"2024-02-23T00:38:52.604431+0000","last_active":"2024-02-23T00:38:59.184629+0000","last_peered":"2024-02-23T00:38:59.184629+0000","last_clean":"2024-02-23T00:38:59.184629+0000","last_became_active":"2024-02-23T00:38:49.233320+0000","last_became_peered":"2024-02-23T00:38:49.233320+0000","last_unstale":"2024-02-23T00:38:59.184629+0000","last_undegraded":"2024-02-23T00:38:59.184629+0000","last_fullsized":"2024-02-23T00:38:59.184629+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'77","last_scrub_stamp":"2024-02-23T00:38:39.206374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.206374+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000073,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":3931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":8000130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.f","version":"485'78","reported_seq":69,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.199179+0000","last_change":"2024-02-23T00:38:49.405543+0000","last_active":"2024-02-23T00:38:59.199179+0000","last_peered":"2024-02-23T00:38:59.199179+0000","last_clean":"2024-02-23T00:38:59.199179+0000","last_became_active":"2024-02-23T00:38:49.221030+0000","last_became_peered":"2024-02-23T00:38:49.221030+0000","last_unstale":"2024-02-23T00:38:59.199179+0000","last_undegraded":"2024-02-23T00:38:59.199179+0000","last_fullsized":"2024-02-23T00:38:59.199179+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'69","last_scrub_stamp":"2024-02-23T00:35:46.750022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.750022+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":33,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.10","version":"503'39","reported_seq":513,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.180819+0000","last_change":"2024-02-23T00:36:37.119458+0000","last_active":"2024-02-23T00:38:59.180819+0000","last_peered":"2024-02-23T00:38:59.180819+0000","last_clean":"2024-02-23T00:36:33.364885+0000","last_became_active":"2024-02-23T00:36:37.119458+0000","last_became_peered":"2024-02-23T00:36:37.119458+0000","last_unstale":"2024-02-23T00:38:59.180819+0000","last_undegraded":"2024-02-23T00:36:37.098703+0000","last_fullsized":"2024-02-23T00:36:37.098584+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'34","last_scrub_stamp":"2024-02-23T00:35:01.791840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.791840+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":1758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.11","version":"484'52","reported_seq":530,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.210127+0000","last_change":"2024-02-23T00:37:06.042258+0000","last_active":"2024-02-23T00:38:59.210127+0000","last_peered":"2024-02-23T00:38:59.210127+0000","last_clean":"2024-02-23T00:38:59.210127+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:38:59.210127+0000","last_undegraded":"2024-02-23T00:38:59.210127+0000","last_fullsized":"2024-02-23T00:38:59.210127+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'50","last_scrub_stamp":"2024-02-23T00:37:06.042181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:06.042181+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":23,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.28","version":"0'0","reported_seq":164,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.185140+0000","last_change":"2024-02-23T00:38:49.189241+0000","last_active":"2024-02-23T00:38:59.185140+0000","last_peered":"2024-02-23T00:38:59.185140+0000","last_clean":"2024-02-23T00:38:59.185140+0000","last_became_active":"2024-02-23T00:38:49.188918+0000","last_became_peered":"2024-02-23T00:38:49.188918+0000","last_unstale":"2024-02-23T00:38:59.185140+0000","last_undegraded":"2024-02-23T00:38:59.185140+0000","last_fullsized":"2024-02-23T00:38:59.185140+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.678152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.678152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.12","version":"503'75","reported_seq":540,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.225202+0000","last_change":"2024-02-23T00:36:37.114447+0000","last_active":"2024-02-23T00:38:59.225202+0000","last_peered":"2024-02-23T00:38:59.225202+0000","last_clean":"2024-02-23T00:36:33.364627+0000","last_became_active":"2024-02-23T00:36:37.114447+0000","last_became_peered":"2024-02-23T00:36:37.114447+0000","last_unstale":"2024-02-23T00:38:59.225202+0000","last_undegraded":"2024-02-23T00:36:37.099733+0000","last_fullsized":"2024-02-23T00:36:37.099613+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'68","last_scrub_stamp":"2024-02-23T00:36:27.078990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.078990+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":611,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":31,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.29","version":"62'8","reported_seq":560,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.230642+0000","last_change":"2024-02-23T00:36:37.122045+0000","last_active":"2024-02-23T00:38:59.230642+0000","last_peered":"2024-02-23T00:38:59.230642+0000","last_clean":"2024-02-23T00:35:27.100547+0000","last_became_active":"2024-02-23T00:36:37.122045+0000","last_became_peered":"2024-02-23T00:36:37.122045+0000","last_unstale":"2024-02-23T00:38:59.230642+0000","last_undegraded":"2024-02-23T00:36:37.107771+0000","last_fullsized":"2024-02-23T00:36:37.107625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:27.673481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.673481+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.13","version":"494'65","reported_seq":540,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.232682+0000","last_change":"2024-02-23T00:36:45.667732+0000","last_active":"2024-02-23T00:38:59.232682+0000","last_peered":"2024-02-23T00:38:59.232682+0000","last_clean":"2024-02-23T00:38:59.232682+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:38:59.232682+0000","last_undegraded":"2024-02-23T00:38:59.232682+0000","last_fullsized":"2024-02-23T00:38:59.232682+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'61","last_scrub_stamp":"2024-02-23T00:36:38.944617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:38.944617+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2920,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.14","version":"494'86","reported_seq":554,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.225035+0000","last_change":"2024-02-23T00:36:37.115695+0000","last_active":"2024-02-23T00:38:59.225035+0000","last_peered":"2024-02-23T00:38:59.225035+0000","last_clean":"2024-02-23T00:36:33.366438+0000","last_became_active":"2024-02-23T00:36:37.115695+0000","last_became_peered":"2024-02-23T00:36:37.115695+0000","last_unstale":"2024-02-23T00:38:59.225035+0000","last_undegraded":"2024-02-23T00:36:37.099640+0000","last_fullsized":"2024-02-23T00:36:37.099515+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'85","last_scrub_stamp":"2024-02-23T00:35:46.085280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.085280+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":1392,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.15","version":"503'73","reported_seq":550,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.212658+0000","last_change":"2024-02-23T00:37:04.031754+0000","last_active":"2024-02-23T00:38:59.212658+0000","last_peered":"2024-02-23T00:38:59.212658+0000","last_clean":"2024-02-23T00:38:59.212658+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:38:59.212658+0000","last_undegraded":"2024-02-23T00:38:59.212658+0000","last_fullsized":"2024-02-23T00:38:59.212658+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'70","last_scrub_stamp":"2024-02-23T00:37:04.031654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:04.031654+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5454,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.16","version":"485'55","reported_seq":537,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.197294+0000","last_change":"2024-02-23T00:36:37.111665+0000","last_active":"2024-02-23T00:38:59.197294+0000","last_peered":"2024-02-23T00:38:59.197294+0000","last_clean":"2024-02-23T00:36:33.194334+0000","last_became_active":"2024-02-23T00:36:37.111665+0000","last_became_peered":"2024-02-23T00:36:37.111665+0000","last_unstale":"2024-02-23T00:38:59.197294+0000","last_undegraded":"2024-02-23T00:36:37.100202+0000","last_fullsized":"2024-02-23T00:36:37.100076+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:35:46.192706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.192706+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":21,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.17","version":"433'112","reported_seq":584,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.187488+0000","last_change":"2024-02-23T00:38:04.819561+0000","last_active":"2024-02-23T00:38:59.187488+0000","last_peered":"2024-02-23T00:38:59.187488+0000","last_clean":"2024-02-23T00:38:59.187488+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:38:59.187488+0000","last_undegraded":"2024-02-23T00:38:59.187488+0000","last_fullsized":"2024-02-23T00:38:59.187488+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'112","last_scrub_stamp":"2024-02-23T00:38:04.819475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:04.819475+0000","log_size":112,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.22","version":"0'0","reported_seq":157,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.231719+0000","last_change":"2024-02-23T00:37:09.026463+0000","last_active":"2024-02-23T00:38:59.231719+0000","last_peered":"2024-02-23T00:38:59.231719+0000","last_clean":"2024-02-23T00:38:59.231719+0000","last_became_active":"2024-02-23T00:35:29.089082+0000","last_became_peered":"2024-02-23T00:35:29.089082+0000","last_unstale":"2024-02-23T00:38:59.231719+0000","last_undegraded":"2024-02-23T00:38:59.231719+0000","last_fullsized":"2024-02-23T00:38:59.231719+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.026383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.026383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.18","version":"484'59","reported_seq":542,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.230386+0000","last_change":"2024-02-23T00:38:44.885221+0000","last_active":"2024-02-23T00:38:59.230386+0000","last_peered":"2024-02-23T00:38:59.230386+0000","last_clean":"2024-02-23T00:38:59.230386+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:38:59.230386+0000","last_undegraded":"2024-02-23T00:38:59.230386+0000","last_fullsized":"2024-02-23T00:38:59.230386+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'59","last_scrub_stamp":"2024-02-23T00:38:44.885113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.885113+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":21,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.23","version":"513'13","reported_seq":222,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-02-23T00:39:00.398267+0000","last_change":"2024-02-23T00:38:49.290324+0000","last_active":"2024-02-23T00:39:00.398267+0000","last_peered":"2024-02-23T00:39:00.398267+0000","last_clean":"2024-02-23T00:39:00.398267+0000","last_became_active":"2024-02-23T00:38:49.226952+0000","last_became_peered":"2024-02-23T00:38:49.226952+0000","last_unstale":"2024-02-23T00:39:00.398267+0000","last_undegraded":"2024-02-23T00:39:00.398267+0000","last_fullsized":"2024-02-23T00:39:00.398267+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.19","version":"503'86","reported_seq":572,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.210100+0000","last_change":"2024-02-23T00:36:45.297690+0000","last_active":"2024-02-23T00:38:59.210100+0000","last_peered":"2024-02-23T00:38:59.210100+0000","last_clean":"2024-02-23T00:38:59.210100+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:38:59.210100+0000","last_undegraded":"2024-02-23T00:38:59.210100+0000","last_fullsized":"2024-02-23T00:38:59.210100+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'60","last_scrub_stamp":"2024-02-23T00:36:31.102608+0000","last_deep_scrub":"271'54","last_deep_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_clean_scrub_stamp":"2024-02-23T00:36:31.102608+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1416,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":18,"num_read_kb":8,"num_write":46,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.20","version":"379'11","reported_seq":205,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-02-23T00:39:00.394874+0000","last_change":"2024-02-23T00:37:10.032410+0000","last_active":"2024-02-23T00:39:00.394874+0000","last_peered":"2024-02-23T00:39:00.394874+0000","last_clean":"2024-02-23T00:39:00.394874+0000","last_became_active":"2024-02-23T00:35:29.093220+0000","last_became_peered":"2024-02-23T00:35:29.093220+0000","last_unstale":"2024-02-23T00:39:00.394874+0000","last_undegraded":"2024-02-23T00:39:00.394874+0000","last_fullsized":"2024-02-23T00:39:00.394874+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"379'11","last_scrub_stamp":"2024-02-23T00:37:10.032341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:10.032341+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1a","version":"494'70","reported_seq":555,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.188782+0000","last_change":"2024-02-23T00:38:44.336029+0000","last_active":"2024-02-23T00:38:59.188782+0000","last_peered":"2024-02-23T00:38:59.188782+0000","last_clean":"2024-02-23T00:38:59.188782+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:38:59.188782+0000","last_undegraded":"2024-02-23T00:38:59.188782+0000","last_fullsized":"2024-02-23T00:38:59.188782+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'70","last_scrub_stamp":"2024-02-23T00:38:44.335909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.335909+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1265,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":24,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.21","version":"0'0","reported_seq":168,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.224788+0000","last_change":"2024-02-23T00:38:49.195187+0000","last_active":"2024-02-23T00:38:59.224788+0000","last_peered":"2024-02-23T00:38:59.224788+0000","last_clean":"2024-02-23T00:38:59.224788+0000","last_became_active":"2024-02-23T00:38:49.194848+0000","last_became_peered":"2024-02-23T00:38:49.194848+0000","last_unstale":"2024-02-23T00:38:59.224788+0000","last_undegraded":"2024-02-23T00:38:59.224788+0000","last_fullsized":"2024-02-23T00:38:59.224788+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:06.376016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:34.152643+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.376016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.1b","version":"494'55","reported_seq":531,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.185942+0000","last_change":"2024-02-23T00:38:07.746173+0000","last_active":"2024-02-23T00:38:59.185942+0000","last_peered":"2024-02-23T00:38:59.185942+0000","last_clean":"2024-02-23T00:38:59.185942+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:38:59.185942+0000","last_undegraded":"2024-02-23T00:38:59.185942+0000","last_fullsized":"2024-02-23T00:38:59.185942+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'54","last_scrub_stamp":"2024-02-23T00:38:07.746090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.746090+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1256,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.26","version":"0'0","reported_seq":20,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.198562+0000","last_change":"2024-02-23T00:38:50.177811+0000","last_active":"2024-02-23T00:38:59.198562+0000","last_peered":"2024-02-23T00:38:59.198562+0000","last_clean":"2024-02-23T00:38:59.198562+0000","last_became_active":"2024-02-23T00:38:50.177418+0000","last_became_peered":"2024-02-23T00:38:50.177418+0000","last_unstale":"2024-02-23T00:38:59.198562+0000","last_undegraded":"2024-02-23T00:38:59.198562+0000","last_fullsized":"2024-02-23T00:38:59.198562+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,2],"avail_no_missing":["3","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.1c","version":"216'34","reported_seq":568,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.182251+0000","last_change":"2024-02-23T00:38:51.138734+0000","last_active":"2024-02-23T00:38:59.182251+0000","last_peered":"2024-02-23T00:38:59.182251+0000","last_clean":"2024-02-23T00:38:59.182251+0000","last_became_active":"2024-02-23T00:38:49.226791+0000","last_became_peered":"2024-02-23T00:38:49.226791+0000","last_unstale":"2024-02-23T00:38:59.182251+0000","last_undegraded":"2024-02-23T00:38:59.182251+0000","last_fullsized":"2024-02-23T00:38:59.182251+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'34","last_scrub_stamp":"2024-02-23T00:34:57.859047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.859047+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.27","version":"0'0","reported_seq":503,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.186131+0000","last_change":"2024-02-23T00:38:49.227500+0000","last_active":"2024-02-23T00:38:59.186131+0000","last_peered":"2024-02-23T00:38:59.186131+0000","last_clean":"2024-02-23T00:38:59.186131+0000","last_became_active":"2024-02-23T00:38:49.225263+0000","last_became_peered":"2024-02-23T00:38:49.225263+0000","last_unstale":"2024-02-23T00:38:59.186131+0000","last_undegraded":"2024-02-23T00:38:59.186131+0000","last_fullsized":"2024-02-23T00:38:59.186131+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:43.246802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:38:43.246802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1d","version":"520'18","reported_seq":243,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.276730+0000","last_change":"2024-02-23T00:38:16.420850+0000","last_active":"2024-02-23T00:38:59.276730+0000","last_peered":"2024-02-23T00:38:59.276730+0000","last_clean":"2024-02-23T00:38:59.276730+0000","last_became_active":"2024-02-23T00:34:42.058724+0000","last_became_peered":"2024-02-23T00:34:42.058724+0000","last_unstale":"2024-02-23T00:38:59.276730+0000","last_undegraded":"2024-02-23T00:38:59.276730+0000","last_fullsized":"2024-02-23T00:38:59.276730+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'13","last_scrub_stamp":"2024-02-23T00:38:16.420789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.420789+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":13,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.24","version":"0'0","reported_seq":501,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.185385+0000","last_change":"2024-02-23T00:36:37.117551+0000","last_active":"2024-02-23T00:38:59.185385+0000","last_peered":"2024-02-23T00:38:59.185385+0000","last_clean":"2024-02-23T00:36:33.366613+0000","last_became_active":"2024-02-23T00:36:37.117551+0000","last_became_peered":"2024-02-23T00:36:37.117551+0000","last_unstale":"2024-02-23T00:38:59.185385+0000","last_undegraded":"2024-02-23T00:38:59.185385+0000","last_fullsized":"2024-02-23T00:36:37.104246+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1e","version":"340'52","reported_seq":524,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.186176+0000","last_change":"2024-02-23T00:38:09.722750+0000","last_active":"2024-02-23T00:38:59.186176+0000","last_peered":"2024-02-23T00:38:59.186176+0000","last_clean":"2024-02-23T00:38:59.186176+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:38:59.186176+0000","last_undegraded":"2024-02-23T00:38:59.186176+0000","last_fullsized":"2024-02-23T00:38:59.186176+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'52","last_scrub_stamp":"2024-02-23T00:38:09.722680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.722680+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.25","version":"0'0","reported_seq":503,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.183468+0000","last_change":"2024-02-23T00:38:42.191298+0000","last_active":"2024-02-23T00:38:59.183468+0000","last_peered":"2024-02-23T00:38:59.183468+0000","last_clean":"2024-02-23T00:38:59.183468+0000","last_became_active":"2024-02-23T00:35:29.090793+0000","last_became_peered":"2024-02-23T00:35:29.090793+0000","last_unstale":"2024-02-23T00:38:59.183468+0000","last_undegraded":"2024-02-23T00:38:59.183468+0000","last_fullsized":"2024-02-23T00:38:59.183468+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.191221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.191221+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.191221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1f","version":"439'74","reported_seq":545,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.187302+0000","last_change":"2024-02-23T00:36:37.117847+0000","last_active":"2024-02-23T00:38:59.187302+0000","last_peered":"2024-02-23T00:38:59.187302+0000","last_clean":"2024-02-23T00:36:33.365701+0000","last_became_active":"2024-02-23T00:36:37.117847+0000","last_became_peered":"2024-02-23T00:36:37.117847+0000","last_unstale":"2024-02-23T00:38:59.187302+0000","last_undegraded":"2024-02-23T00:36:37.102042+0000","last_fullsized":"2024-02-23T00:36:37.101901+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-23T00:33:39.351573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.351573+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"379'9","reported_seq":654,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-02-23T00:39:00.406887+0000","last_change":"2024-02-23T00:37:05.955772+0000","last_active":"2024-02-23T00:39:00.406887+0000","last_peered":"2024-02-23T00:39:00.406887+0000","last_clean":"2024-02-23T00:39:00.406887+0000","last_became_active":"2024-02-23T00:35:27.095085+0000","last_became_peered":"2024-02-23T00:35:27.095085+0000","last_unstale":"2024-02-23T00:39:00.406887+0000","last_undegraded":"2024-02-23T00:39:00.406887+0000","last_fullsized":"2024-02-23T00:39:00.406887+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'9","last_scrub_stamp":"2024-02-23T00:37:05.955707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.955707+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":605,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.185392+0000","last_change":"2024-02-23T00:38:19.164982+0000","last_active":"2024-02-23T00:38:59.185392+0000","last_peered":"2024-02-23T00:38:59.185392+0000","last_clean":"2024-02-23T00:38:59.185392+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:38:59.185392+0000","last_undegraded":"2024-02-23T00:38:59.185392+0000","last_fullsized":"2024-02-23T00:38:59.185392+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":528,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.197464+0000","last_change":"2024-02-23T00:38:49.199186+0000","last_active":"2024-02-23T00:38:59.197464+0000","last_peered":"2024-02-23T00:38:59.197464+0000","last_clean":"2024-02-23T00:38:59.197464+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:38:59.197464+0000","last_undegraded":"2024-02-23T00:38:59.197464+0000","last_fullsized":"2024-02-23T00:38:59.197464+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:12.113097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:38:12.113097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":483,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.187351+0000","last_change":"2024-02-23T00:36:37.104200+0000","last_active":"2024-02-23T00:38:59.187351+0000","last_peered":"2024-02-23T00:38:59.187351+0000","last_clean":"2024-02-23T00:36:33.366159+0000","last_became_active":"2024-02-23T00:36:37.104200+0000","last_became_peered":"2024-02-23T00:36:37.104200+0000","last_unstale":"2024-02-23T00:38:59.187351+0000","last_undegraded":"2024-02-23T00:38:59.187351+0000","last_fullsized":"2024-02-23T00:36:37.102434+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:21.441958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.441958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":447,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.181980+0000","last_change":"2024-02-23T00:36:37.115879+0000","last_active":"2024-02-23T00:38:59.181980+0000","last_peered":"2024-02-23T00:38:59.181980+0000","last_clean":"2024-02-23T00:36:33.366205+0000","last_became_active":"2024-02-23T00:36:37.115879+0000","last_became_peered":"2024-02-23T00:36:37.115879+0000","last_unstale":"2024-02-23T00:38:59.181980+0000","last_undegraded":"2024-02-23T00:38:59.181980+0000","last_fullsized":"2024-02-23T00:36:37.101967+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":879,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:39:00.205755+0000","last_change":"2024-02-23T00:38:49.283349+0000","last_active":"2024-02-23T00:39:00.205755+0000","last_peered":"2024-02-23T00:39:00.205755+0000","last_clean":"2024-02-23T00:39:00.205755+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:39:00.205755+0000","last_undegraded":"2024-02-23T00:39:00.205755+0000","last_fullsized":"2024-02-23T00:39:00.205755+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:36:27.181972+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.181972+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":381,"num_read_kb":381,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":166,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.182317+0000","last_change":"2024-02-23T00:38:19.177071+0000","last_active":"2024-02-23T00:38:59.182317+0000","last_peered":"2024-02-23T00:38:59.182317+0000","last_clean":"2024-02-23T00:27:29.183258+0000","last_became_active":"2024-02-23T00:38:19.177071+0000","last_became_peered":"2024-02-23T00:38:19.177071+0000","last_unstale":"2024-02-23T00:38:59.182317+0000","last_undegraded":"2024-02-23T00:38:19.154601+0000","last_fullsized":"2024-02-23T00:38:19.154480+0000","mapping_epoch":425,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":508,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.185192+0000","last_change":"2024-02-23T00:38:49.228134+0000","last_active":"2024-02-23T00:38:59.185192+0000","last_peered":"2024-02-23T00:38:59.185192+0000","last_clean":"2024-02-23T00:38:59.185192+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:38:59.185192+0000","last_undegraded":"2024-02-23T00:38:59.185192+0000","last_fullsized":"2024-02-23T00:38:59.185192+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.532619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.532619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":515,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.188991+0000","last_change":"2024-02-23T00:38:49.175746+0000","last_active":"2024-02-23T00:38:59.188991+0000","last_peered":"2024-02-23T00:38:59.188991+0000","last_clean":"2024-02-23T00:38:59.188991+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:38:59.188991+0000","last_undegraded":"2024-02-23T00:38:59.188991+0000","last_fullsized":"2024-02-23T00:38:59.188991+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.679076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":103,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.231027+0000","last_change":"2024-02-23T00:36:37.122229+0000","last_active":"2024-02-23T00:38:59.231027+0000","last_peered":"2024-02-23T00:38:59.231027+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.122229+0000","last_became_peered":"2024-02-23T00:36:37.122229+0000","last_unstale":"2024-02-23T00:38:59.231027+0000","last_undegraded":"2024-02-23T00:38:59.231027+0000","last_fullsized":"2024-02-23T00:36:37.107921+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":441,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.225560+0000","last_change":"2024-02-23T00:38:01.361178+0000","last_active":"2024-02-23T00:38:59.225560+0000","last_peered":"2024-02-23T00:38:59.225560+0000","last_clean":"2024-02-23T00:38:59.225560+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:38:59.225560+0000","last_undegraded":"2024-02-23T00:38:59.225560+0000","last_fullsized":"2024-02-23T00:38:59.225560+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:01.361095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:01.361095+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":625,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.225470+0000","last_change":"2024-02-23T00:38:40.424549+0000","last_active":"2024-02-23T00:38:59.225470+0000","last_peered":"2024-02-23T00:38:59.225470+0000","last_clean":"2024-02-23T00:38:59.225470+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:38:59.225470+0000","last_undegraded":"2024-02-23T00:38:59.225470+0000","last_fullsized":"2024-02-23T00:38:59.225470+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:40.424490+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.424490+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":506,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.190805+0000","last_change":"2024-02-23T00:36:37.116992+0000","last_active":"2024-02-23T00:38:59.190805+0000","last_peered":"2024-02-23T00:38:59.190805+0000","last_clean":"2024-02-23T00:36:33.194161+0000","last_became_active":"2024-02-23T00:36:37.116992+0000","last_became_peered":"2024-02-23T00:36:37.116992+0000","last_unstale":"2024-02-23T00:38:59.190805+0000","last_undegraded":"2024-02-23T00:38:59.190805+0000","last_fullsized":"2024-02-23T00:36:37.104222+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:50.194656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:50.194656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":497,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.195862+0000","last_change":"2024-02-23T00:36:37.113419+0000","last_active":"2024-02-23T00:38:59.195862+0000","last_peered":"2024-02-23T00:38:59.195862+0000","last_clean":"2024-02-23T00:36:33.194007+0000","last_became_active":"2024-02-23T00:36:37.113419+0000","last_became_peered":"2024-02-23T00:36:37.113419+0000","last_unstale":"2024-02-23T00:38:59.195862+0000","last_undegraded":"2024-02-23T00:38:59.195862+0000","last_fullsized":"2024-02-23T00:36:37.104161+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.745865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.745865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":509,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.230491+0000","last_change":"2024-02-23T00:38:49.208110+0000","last_active":"2024-02-23T00:38:59.230491+0000","last_peered":"2024-02-23T00:38:59.230491+0000","last_clean":"2024-02-23T00:38:59.230491+0000","last_became_active":"2024-02-23T00:38:49.207843+0000","last_became_peered":"2024-02-23T00:38:49.207843+0000","last_unstale":"2024-02-23T00:38:59.230491+0000","last_undegraded":"2024-02-23T00:38:59.230491+0000","last_fullsized":"2024-02-23T00:38:59.230491+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.226380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.226380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":104,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.188630+0000","last_change":"2024-02-23T00:36:37.119721+0000","last_active":"2024-02-23T00:38:59.188630+0000","last_peered":"2024-02-23T00:38:59.188630+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119721+0000","last_became_peered":"2024-02-23T00:36:37.119721+0000","last_unstale":"2024-02-23T00:38:59.188630+0000","last_undegraded":"2024-02-23T00:38:59.188630+0000","last_fullsized":"2024-02-23T00:36:37.102163+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.205312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.205312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":508,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.225808+0000","last_change":"2024-02-23T00:38:44.442795+0000","last_active":"2024-02-23T00:38:59.225808+0000","last_peered":"2024-02-23T00:38:59.225808+0000","last_clean":"2024-02-23T00:38:59.225808+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:38:59.225808+0000","last_undegraded":"2024-02-23T00:38:59.225808+0000","last_fullsized":"2024-02-23T00:38:59.225808+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:44.442733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.442733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":578,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.209803+0000","last_change":"2024-02-23T00:38:19.177268+0000","last_active":"2024-02-23T00:38:59.209803+0000","last_peered":"2024-02-23T00:38:59.209803+0000","last_clean":"2024-02-23T00:38:59.209803+0000","last_became_active":"2024-02-23T00:38:19.176922+0000","last_became_peered":"2024-02-23T00:38:19.176922+0000","last_unstale":"2024-02-23T00:38:59.209803+0000","last_undegraded":"2024-02-23T00:38:59.209803+0000","last_fullsized":"2024-02-23T00:38:59.209803+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":29,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":497,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.232828+0000","last_change":"2024-02-23T00:38:18.055894+0000","last_active":"2024-02-23T00:38:59.232828+0000","last_peered":"2024-02-23T00:38:59.232828+0000","last_clean":"2024-02-23T00:38:59.232828+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:38:59.232828+0000","last_undegraded":"2024-02-23T00:38:59.232828+0000","last_fullsized":"2024-02-23T00:38:59.232828+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:18.055803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:18.055803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":487,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.182596+0000","last_change":"2024-02-23T00:36:37.109480+0000","last_active":"2024-02-23T00:38:59.182596+0000","last_peered":"2024-02-23T00:38:59.182596+0000","last_clean":"2024-02-23T00:36:34.605757+0000","last_became_active":"2024-02-23T00:36:37.109480+0000","last_became_peered":"2024-02-23T00:36:37.109480+0000","last_unstale":"2024-02-23T00:38:59.182596+0000","last_undegraded":"2024-02-23T00:38:59.182596+0000","last_fullsized":"2024-02-23T00:36:37.095167+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.605664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.605664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":537,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.191007+0000","last_change":"2024-02-23T00:38:49.182352+0000","last_active":"2024-02-23T00:38:59.191007+0000","last_peered":"2024-02-23T00:38:59.191007+0000","last_clean":"2024-02-23T00:38:59.191007+0000","last_became_active":"2024-02-23T00:38:49.181997+0000","last_became_peered":"2024-02-23T00:38:49.181997+0000","last_unstale":"2024-02-23T00:38:59.191007+0000","last_undegraded":"2024-02-23T00:38:59.191007+0000","last_fullsized":"2024-02-23T00:38:59.191007+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:17.822776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.822776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":464,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.237143+0000","last_change":"2024-02-23T00:38:49.194765+0000","last_active":"2024-02-23T00:38:59.237143+0000","last_peered":"2024-02-23T00:38:59.237143+0000","last_clean":"2024-02-23T00:38:59.237143+0000","last_became_active":"2024-02-23T00:38:49.194450+0000","last_became_peered":"2024-02-23T00:38:49.194450+0000","last_unstale":"2024-02-23T00:38:59.237143+0000","last_undegraded":"2024-02-23T00:38:59.237143+0000","last_fullsized":"2024-02-23T00:38:59.237143+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.704478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.704478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.194752+0000","last_change":"2024-02-23T00:38:50.199088+0000","last_active":"2024-02-23T00:38:59.194752+0000","last_peered":"2024-02-23T00:38:59.194752+0000","last_clean":"2024-02-23T00:38:59.194752+0000","last_became_active":"2024-02-23T00:38:50.198701+0000","last_became_peered":"2024-02-23T00:38:50.198701+0000","last_unstale":"2024-02-23T00:38:59.194752+0000","last_undegraded":"2024-02-23T00:38:59.194752+0000","last_fullsized":"2024-02-23T00:38:59.194752+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,1],"avail_no_missing":["3","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":622,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.213937+0000","last_change":"2024-02-23T00:38:54.172406+0000","last_active":"2024-02-23T00:38:59.213937+0000","last_peered":"2024-02-23T00:38:59.213937+0000","last_clean":"2024-02-23T00:38:59.213937+0000","last_became_active":"2024-02-23T00:38:54.171993+0000","last_became_peered":"2024-02-23T00:38:54.171993+0000","last_unstale":"2024-02-23T00:38:59.213937+0000","last_undegraded":"2024-02-23T00:38:59.213937+0000","last_fullsized":"2024-02-23T00:38:59.213937+0000","mapping_epoch":516,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:37:09.133471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.133471+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":521,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.181735+0000","last_change":"2024-02-23T00:38:49.225600+0000","last_active":"2024-02-23T00:38:59.181735+0000","last_peered":"2024-02-23T00:38:59.181735+0000","last_clean":"2024-02-23T00:38:59.181735+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:38:59.181735+0000","last_undegraded":"2024-02-23T00:38:59.181735+0000","last_fullsized":"2024-02-23T00:38:59.181735+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.678676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.678676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":650,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-02-23T00:39:00.196718+0000","last_change":"2024-02-23T00:38:35.337273+0000","last_active":"2024-02-23T00:39:00.196718+0000","last_peered":"2024-02-23T00:39:00.196718+0000","last_clean":"2024-02-23T00:39:00.196718+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:39:00.196718+0000","last_undegraded":"2024-02-23T00:39:00.196718+0000","last_fullsized":"2024-02-23T00:39:00.196718+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:38:35.337170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.337170+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":511,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.214063+0000","last_change":"2024-02-23T00:38:49.171993+0000","last_active":"2024-02-23T00:38:59.214063+0000","last_peered":"2024-02-23T00:38:59.214063+0000","last_clean":"2024-02-23T00:38:59.214063+0000","last_became_active":"2024-02-23T00:38:49.171696+0000","last_became_peered":"2024-02-23T00:38:49.171696+0000","last_unstale":"2024-02-23T00:38:59.214063+0000","last_undegraded":"2024-02-23T00:38:59.214063+0000","last_fullsized":"2024-02-23T00:38:59.214063+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.135802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.135802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":446,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.213950+0000","last_change":"2024-02-23T00:38:34.097327+0000","last_active":"2024-02-23T00:38:59.213950+0000","last_peered":"2024-02-23T00:38:59.213950+0000","last_clean":"2024-02-23T00:38:59.213950+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:38:59.213950+0000","last_undegraded":"2024-02-23T00:38:59.213950+0000","last_fullsized":"2024-02-23T00:38:59.213950+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:34.097243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.097243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.196352+0000","last_change":"2024-02-23T00:38:49.223162+0000","last_active":"2024-02-23T00:38:59.196352+0000","last_peered":"2024-02-23T00:38:59.196352+0000","last_clean":"2024-02-23T00:38:59.196352+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:38:59.196352+0000","last_undegraded":"2024-02-23T00:38:59.196352+0000","last_fullsized":"2024-02-23T00:38:59.196352+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:22.396222+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.396222+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":651,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:59.187008+0000","last_change":"2024-02-23T00:38:52.605018+0000","last_active":"2024-02-23T00:38:59.187008+0000","last_peered":"2024-02-23T00:38:59.187008+0000","last_clean":"2024-02-23T00:38:47.153110+0000","last_became_active":"2024-02-23T00:38:50.207077+0000","last_became_peered":"2024-02-23T00:38:50.207077+0000","last_unstale":"2024-02-23T00:38:59.187008+0000","last_undegraded":"2024-02-23T00:38:59.187008+0000","last_fullsized":"2024-02-23T00:38:59.187008+0000","mapping_epoch":512,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":505,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.181408+0000","last_change":"2024-02-23T00:38:49.228791+0000","last_active":"2024-02-23T00:38:59.181408+0000","last_peered":"2024-02-23T00:38:59.181408+0000","last_clean":"2024-02-23T00:38:59.181408+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:38:59.181408+0000","last_undegraded":"2024-02-23T00:38:59.181408+0000","last_fullsized":"2024-02-23T00:38:59.181408+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.657998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"426'9","reported_seq":646,"reported_epoch":521,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:00.396153+0000","last_change":"2024-02-23T00:36:37.111178+0000","last_active":"2024-02-23T00:39:00.396153+0000","last_peered":"2024-02-23T00:39:00.396153+0000","last_clean":"2024-02-23T00:36:35.444981+0000","last_became_active":"2024-02-23T00:36:37.111178+0000","last_became_peered":"2024-02-23T00:36:37.111178+0000","last_unstale":"2024-02-23T00:39:00.396153+0000","last_undegraded":"2024-02-23T00:36:37.095825+0000","last_fullsized":"2024-02-23T00:36:37.095713+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:59.630171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:59.630171+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":484,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.188737+0000","last_change":"2024-02-23T00:36:37.113156+0000","last_active":"2024-02-23T00:38:59.188737+0000","last_peered":"2024-02-23T00:38:59.188737+0000","last_clean":"2024-02-23T00:36:33.366740+0000","last_became_active":"2024-02-23T00:36:37.113156+0000","last_became_peered":"2024-02-23T00:36:37.113156+0000","last_unstale":"2024-02-23T00:38:59.188737+0000","last_undegraded":"2024-02-23T00:36:37.102218+0000","last_fullsized":"2024-02-23T00:36:37.102103+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:34:17.437083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.437083+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":437,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.183672+0000","last_change":"2024-02-23T00:38:16.396902+0000","last_active":"2024-02-23T00:38:59.183672+0000","last_peered":"2024-02-23T00:38:59.183672+0000","last_clean":"2024-02-23T00:38:59.183672+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:38:59.183672+0000","last_undegraded":"2024-02-23T00:38:59.183672+0000","last_fullsized":"2024-02-23T00:38:59.183672+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.396824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":508,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.210029+0000","last_change":"2024-02-23T00:38:08.068624+0000","last_active":"2024-02-23T00:38:59.210029+0000","last_peered":"2024-02-23T00:38:59.210029+0000","last_clean":"2024-02-23T00:38:59.210029+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:38:59.210029+0000","last_undegraded":"2024-02-23T00:38:59.210029+0000","last_fullsized":"2024-02-23T00:38:59.210029+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:08.068559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:08.068559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":592,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.209829+0000","last_change":"2024-02-23T00:38:19.182554+0000","last_active":"2024-02-23T00:38:59.209829+0000","last_peered":"2024-02-23T00:38:59.209829+0000","last_clean":"2024-02-23T00:38:59.209829+0000","last_became_active":"2024-02-23T00:38:19.182222+0000","last_became_peered":"2024-02-23T00:38:19.182222+0000","last_unstale":"2024-02-23T00:38:59.209829+0000","last_undegraded":"2024-02-23T00:38:59.209829+0000","last_fullsized":"2024-02-23T00:38:59.209829+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":505,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.230844+0000","last_change":"2024-02-23T00:36:37.126232+0000","last_active":"2024-02-23T00:38:59.230844+0000","last_peered":"2024-02-23T00:38:59.230844+0000","last_clean":"2024-02-23T00:36:33.365949+0000","last_became_active":"2024-02-23T00:36:37.126232+0000","last_became_peered":"2024-02-23T00:36:37.126232+0000","last_unstale":"2024-02-23T00:38:59.230844+0000","last_undegraded":"2024-02-23T00:36:37.112806+0000","last_fullsized":"2024-02-23T00:36:37.112690+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":484,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.187931+0000","last_change":"2024-02-23T00:38:37.881705+0000","last_active":"2024-02-23T00:38:59.187931+0000","last_peered":"2024-02-23T00:38:59.187931+0000","last_clean":"2024-02-23T00:38:59.187931+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:38:59.187931+0000","last_undegraded":"2024-02-23T00:38:59.187931+0000","last_fullsized":"2024-02-23T00:38:59.187931+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:37.881618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.881618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.197306+0000","last_change":"2024-02-23T00:38:49.230667+0000","last_active":"2024-02-23T00:38:59.197306+0000","last_peered":"2024-02-23T00:38:59.197306+0000","last_clean":"2024-02-23T00:38:59.197306+0000","last_became_active":"2024-02-23T00:38:49.226279+0000","last_became_peered":"2024-02-23T00:38:49.226279+0000","last_unstale":"2024-02-23T00:38:59.197306+0000","last_undegraded":"2024-02-23T00:38:59.197306+0000","last_fullsized":"2024-02-23T00:38:59.197306+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.478470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.478470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":455,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.182801+0000","last_change":"2024-02-23T00:38:49.227768+0000","last_active":"2024-02-23T00:38:59.182801+0000","last_peered":"2024-02-23T00:38:59.182801+0000","last_clean":"2024-02-23T00:38:59.182801+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:38:59.182801+0000","last_undegraded":"2024-02-23T00:38:59.182801+0000","last_fullsized":"2024-02-23T00:38:59.182801+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_clean_scrub_stamp":"2024-02-23T00:37:21.364934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":104,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.184514+0000","last_change":"2024-02-23T00:36:37.109542+0000","last_active":"2024-02-23T00:38:59.184514+0000","last_peered":"2024-02-23T00:38:59.184514+0000","last_clean":"2024-02-23T00:31:27.149861+0000","last_became_active":"2024-02-23T00:36:37.109542+0000","last_became_peered":"2024-02-23T00:36:37.109542+0000","last_unstale":"2024-02-23T00:38:59.184514+0000","last_undegraded":"2024-02-23T00:38:59.184514+0000","last_fullsized":"2024-02-23T00:36:37.096846+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:42.114807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:35:42.114807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.196251+0000","last_change":"2024-02-23T00:38:49.222977+0000","last_active":"2024-02-23T00:38:59.196251+0000","last_peered":"2024-02-23T00:38:59.196251+0000","last_clean":"2024-02-23T00:38:59.196251+0000","last_became_active":"2024-02-23T00:38:49.216657+0000","last_became_peered":"2024-02-23T00:38:49.216657+0000","last_unstale":"2024-02-23T00:38:59.196251+0000","last_undegraded":"2024-02-23T00:38:59.196251+0000","last_fullsized":"2024-02-23T00:38:59.196251+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":549,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.210529+0000","last_change":"2024-02-23T00:37:08.107220+0000","last_active":"2024-02-23T00:38:59.210529+0000","last_peered":"2024-02-23T00:38:59.210529+0000","last_clean":"2024-02-23T00:38:59.210529+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:38:59.210529+0000","last_undegraded":"2024-02-23T00:38:59.210529+0000","last_fullsized":"2024-02-23T00:38:59.210529+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:37:08.107110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.107110+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":498,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.227214+0000","last_change":"2024-02-23T00:38:48.494288+0000","last_active":"2024-02-23T00:38:59.227214+0000","last_peered":"2024-02-23T00:38:59.227214+0000","last_clean":"2024-02-23T00:38:59.227214+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:38:59.227214+0000","last_undegraded":"2024-02-23T00:38:59.227214+0000","last_fullsized":"2024-02-23T00:38:59.227214+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:48.494224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:48.494224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":491,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.230024+0000","last_change":"2024-02-23T00:36:37.952567+0000","last_active":"2024-02-23T00:38:59.230024+0000","last_peered":"2024-02-23T00:38:59.230024+0000","last_clean":"2024-02-23T00:38:59.230024+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:38:59.230024+0000","last_undegraded":"2024-02-23T00:38:59.230024+0000","last_fullsized":"2024-02-23T00:38:59.230024+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:37.952474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:37.952474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":457,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.182598+0000","last_change":"2024-02-23T00:38:50.207423+0000","last_active":"2024-02-23T00:38:59.182598+0000","last_peered":"2024-02-23T00:38:59.182598+0000","last_clean":"2024-02-23T00:38:59.182598+0000","last_became_active":"2024-02-23T00:38:50.206899+0000","last_became_peered":"2024-02-23T00:38:50.206899+0000","last_unstale":"2024-02-23T00:38:59.182598+0000","last_undegraded":"2024-02-23T00:38:59.182598+0000","last_fullsized":"2024-02-23T00:38:59.182598+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:22.019530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:34:22.019530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.199032+0000","last_change":"2024-02-23T00:38:50.201494+0000","last_active":"2024-02-23T00:38:59.199032+0000","last_peered":"2024-02-23T00:38:59.199032+0000","last_clean":"2024-02-23T00:38:59.199032+0000","last_became_active":"2024-02-23T00:38:50.201068+0000","last_became_peered":"2024-02-23T00:38:50.201068+0000","last_unstale":"2024-02-23T00:38:59.199032+0000","last_undegraded":"2024-02-23T00:38:59.199032+0000","last_fullsized":"2024-02-23T00:38:59.199032+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.924808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.924808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,5],"avail_no_missing":["3","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":523,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.210957+0000","last_change":"2024-02-23T00:38:49.228711+0000","last_active":"2024-02-23T00:38:59.210957+0000","last_peered":"2024-02-23T00:38:59.210957+0000","last_clean":"2024-02-23T00:38:59.210957+0000","last_became_active":"2024-02-23T00:38:49.228425+0000","last_became_peered":"2024-02-23T00:38:49.228425+0000","last_unstale":"2024-02-23T00:38:59.210957+0000","last_undegraded":"2024-02-23T00:38:59.210957+0000","last_fullsized":"2024-02-23T00:38:59.210957+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.588132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.588132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":42,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.195574+0000","last_change":"2024-02-23T00:38:54.112431+0000","last_active":"2024-02-23T00:38:59.195574+0000","last_peered":"2024-02-23T00:38:59.195574+0000","last_clean":"2024-02-23T00:38:59.195574+0000","last_became_active":"2024-02-23T00:38:50.199619+0000","last_became_peered":"2024-02-23T00:38:50.199619+0000","last_unstale":"2024-02-23T00:38:59.195574+0000","last_undegraded":"2024-02-23T00:38:59.195574+0000","last_fullsized":"2024-02-23T00:38:59.195574+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:35:00.621157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.621157+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":490,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.184705+0000","last_change":"2024-02-23T00:38:14.343844+0000","last_active":"2024-02-23T00:38:59.184705+0000","last_peered":"2024-02-23T00:38:59.184705+0000","last_clean":"2024-02-23T00:38:59.184705+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:38:59.184705+0000","last_undegraded":"2024-02-23T00:38:59.184705+0000","last_fullsized":"2024-02-23T00:38:59.184705+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:14.343770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:14.343770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":511,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.211080+0000","last_change":"2024-02-23T00:38:49.225300+0000","last_active":"2024-02-23T00:38:59.211080+0000","last_peered":"2024-02-23T00:38:59.211080+0000","last_clean":"2024-02-23T00:38:59.211080+0000","last_became_active":"2024-02-23T00:38:49.224892+0000","last_became_peered":"2024-02-23T00:38:49.224892+0000","last_unstale":"2024-02-23T00:38:59.211080+0000","last_undegraded":"2024-02-23T00:38:59.211080+0000","last_fullsized":"2024-02-23T00:38:59.211080+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.082000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.082000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.190718+0000","last_change":"2024-02-23T00:38:49.176838+0000","last_active":"2024-02-23T00:38:59.190718+0000","last_peered":"2024-02-23T00:38:59.190718+0000","last_clean":"2024-02-23T00:38:59.190718+0000","last_became_active":"2024-02-23T00:38:49.175801+0000","last_became_peered":"2024-02-23T00:38:49.175801+0000","last_unstale":"2024-02-23T00:38:59.190718+0000","last_undegraded":"2024-02-23T00:38:59.190718+0000","last_fullsized":"2024-02-23T00:38:59.190718+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.977282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.977282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":753,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.182153+0000","last_change":"2024-02-23T00:37:17.499775+0000","last_active":"2024-02-23T00:38:59.182153+0000","last_peered":"2024-02-23T00:38:59.182153+0000","last_clean":"2024-02-23T00:38:59.182153+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:38:59.182153+0000","last_undegraded":"2024-02-23T00:38:59.182153+0000","last_fullsized":"2024-02-23T00:38:59.182153+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:17.499694+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.499694+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":505,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.225518+0000","last_change":"2024-02-23T00:36:37.108691+0000","last_active":"2024-02-23T00:38:59.225518+0000","last_peered":"2024-02-23T00:38:59.225518+0000","last_clean":"2024-02-23T00:36:33.369928+0000","last_became_active":"2024-02-23T00:36:37.108691+0000","last_became_peered":"2024-02-23T00:36:37.108691+0000","last_unstale":"2024-02-23T00:38:59.225518+0000","last_undegraded":"2024-02-23T00:38:59.225518+0000","last_fullsized":"2024-02-23T00:36:37.100335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.196211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.196211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":182,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.186439+0000","last_change":"2024-02-23T00:36:37.123994+0000","last_active":"2024-02-23T00:38:59.186439+0000","last_peered":"2024-02-23T00:38:59.186439+0000","last_clean":"2024-02-23T00:27:29.216066+0000","last_became_active":"2024-02-23T00:36:37.123994+0000","last_became_peered":"2024-02-23T00:36:37.123994+0000","last_unstale":"2024-02-23T00:38:59.186439+0000","last_undegraded":"2024-02-23T00:36:37.103439+0000","last_fullsized":"2024-02-23T00:36:37.103335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:35:01.396982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.396982+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":498,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.235304+0000","last_change":"2024-02-23T00:36:37.120094+0000","last_active":"2024-02-23T00:38:59.235304+0000","last_peered":"2024-02-23T00:38:59.235304+0000","last_clean":"2024-02-23T00:36:33.367908+0000","last_became_active":"2024-02-23T00:36:37.120094+0000","last_became_peered":"2024-02-23T00:36:37.120094+0000","last_unstale":"2024-02-23T00:38:59.235304+0000","last_undegraded":"2024-02-23T00:38:59.235304+0000","last_fullsized":"2024-02-23T00:36:37.107601+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":484,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.180791+0000","last_change":"2024-02-23T00:37:26.268389+0000","last_active":"2024-02-23T00:38:59.180791+0000","last_peered":"2024-02-23T00:38:59.180791+0000","last_clean":"2024-02-23T00:38:59.180791+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:38:59.180791+0000","last_undegraded":"2024-02-23T00:38:59.180791+0000","last_fullsized":"2024-02-23T00:38:59.180791+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.268282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":432,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.180869+0000","last_change":"2024-02-23T00:35:00.832892+0000","last_active":"2024-02-23T00:38:59.180869+0000","last_peered":"2024-02-23T00:38:59.180869+0000","last_clean":"2024-02-23T00:38:59.180869+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:38:59.180869+0000","last_undegraded":"2024-02-23T00:38:59.180869+0000","last_fullsized":"2024-02-23T00:38:59.180869+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.832818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.832818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":494,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.187964+0000","last_change":"2024-02-23T00:36:37.117616+0000","last_active":"2024-02-23T00:38:59.187964+0000","last_peered":"2024-02-23T00:38:59.187964+0000","last_clean":"2024-02-23T00:36:33.368573+0000","last_became_active":"2024-02-23T00:36:37.117616+0000","last_became_peered":"2024-02-23T00:36:37.117616+0000","last_unstale":"2024-02-23T00:38:59.187964+0000","last_undegraded":"2024-02-23T00:38:59.187964+0000","last_fullsized":"2024-02-23T00:36:37.100549+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":167,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.184781+0000","last_change":"2024-02-23T00:38:19.160002+0000","last_active":"2024-02-23T00:38:59.184781+0000","last_peered":"2024-02-23T00:38:59.184781+0000","last_clean":"2024-02-23T00:27:24.365389+0000","last_became_active":"2024-02-23T00:38:19.160002+0000","last_became_peered":"2024-02-23T00:38:19.160002+0000","last_unstale":"2024-02-23T00:38:59.184781+0000","last_undegraded":"2024-02-23T00:38:19.155620+0000","last_fullsized":"2024-02-23T00:38:19.155500+0000","mapping_epoch":425,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":498,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.230659+0000","last_change":"2024-02-23T00:36:37.126102+0000","last_active":"2024-02-23T00:38:59.230659+0000","last_peered":"2024-02-23T00:38:59.230659+0000","last_clean":"2024-02-23T00:36:33.365781+0000","last_became_active":"2024-02-23T00:36:37.126102+0000","last_became_peered":"2024-02-23T00:36:37.126102+0000","last_unstale":"2024-02-23T00:38:59.230659+0000","last_undegraded":"2024-02-23T00:38:59.230659+0000","last_fullsized":"2024-02-23T00:36:37.112617+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.980396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.980396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":426,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.210789+0000","last_change":"2024-02-23T00:38:50.177994+0000","last_active":"2024-02-23T00:38:59.210789+0000","last_peered":"2024-02-23T00:38:59.210789+0000","last_clean":"2024-02-23T00:38:59.210789+0000","last_became_active":"2024-02-23T00:38:50.177540+0000","last_became_peered":"2024-02-23T00:38:50.177540+0000","last_unstale":"2024-02-23T00:38:59.210789+0000","last_undegraded":"2024-02-23T00:38:59.210789+0000","last_fullsized":"2024-02-23T00:38:59.210789+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:31.593354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:31.593354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,5],"avail_no_missing":["4","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":497,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.182722+0000","last_change":"2024-02-23T00:38:49.224597+0000","last_active":"2024-02-23T00:38:59.182722+0000","last_peered":"2024-02-23T00:38:59.182722+0000","last_clean":"2024-02-23T00:38:59.182722+0000","last_became_active":"2024-02-23T00:38:49.224244+0000","last_became_peered":"2024-02-23T00:38:49.224244+0000","last_unstale":"2024-02-23T00:38:59.182722+0000","last_undegraded":"2024-02-23T00:38:59.182722+0000","last_fullsized":"2024-02-23T00:38:59.182722+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.744088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.744088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":441,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.214032+0000","last_change":"2024-02-23T00:35:55.223009+0000","last_active":"2024-02-23T00:38:59.214032+0000","last_peered":"2024-02-23T00:38:59.214032+0000","last_clean":"2024-02-23T00:38:59.214032+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:38:59.214032+0000","last_undegraded":"2024-02-23T00:38:59.214032+0000","last_fullsized":"2024-02-23T00:38:59.214032+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.222912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.222912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":502,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.233843+0000","last_change":"2024-02-23T00:36:41.937397+0000","last_active":"2024-02-23T00:38:59.233843+0000","last_peered":"2024-02-23T00:38:59.233843+0000","last_clean":"2024-02-23T00:38:59.233843+0000","last_became_active":"2024-02-23T00:35:27.093541+0000","last_became_peered":"2024-02-23T00:35:27.093541+0000","last_unstale":"2024-02-23T00:38:59.233843+0000","last_undegraded":"2024-02-23T00:38:59.233843+0000","last_fullsized":"2024-02-23T00:38:59.233843+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.937344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.937344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":537,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.226424+0000","last_change":"2024-02-23T00:38:03.327200+0000","last_active":"2024-02-23T00:38:59.226424+0000","last_peered":"2024-02-23T00:38:59.226424+0000","last_clean":"2024-02-23T00:38:59.226424+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:38:59.226424+0000","last_undegraded":"2024-02-23T00:38:59.226424+0000","last_fullsized":"2024-02-23T00:38:59.226424+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:03.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.327127+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":501,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.230729+0000","last_change":"2024-02-23T00:36:37.106377+0000","last_active":"2024-02-23T00:38:59.230729+0000","last_peered":"2024-02-23T00:38:59.230729+0000","last_clean":"2024-02-23T00:36:33.365867+0000","last_became_active":"2024-02-23T00:36:37.106377+0000","last_became_peered":"2024-02-23T00:36:37.106377+0000","last_unstale":"2024-02-23T00:38:59.230729+0000","last_undegraded":"2024-02-23T00:38:59.230729+0000","last_fullsized":"2024-02-23T00:36:37.100069+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":488,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.186505+0000","last_change":"2024-02-23T00:36:33.160039+0000","last_active":"2024-02-23T00:38:59.186505+0000","last_peered":"2024-02-23T00:38:59.186505+0000","last_clean":"2024-02-23T00:38:59.186505+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:38:59.186505+0000","last_undegraded":"2024-02-23T00:38:59.186505+0000","last_fullsized":"2024-02-23T00:38:59.186505+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.159941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.159941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.190793+0000","last_change":"2024-02-23T00:38:49.226513+0000","last_active":"2024-02-23T00:38:59.190793+0000","last_peered":"2024-02-23T00:38:59.190793+0000","last_clean":"2024-02-23T00:38:59.190793+0000","last_became_active":"2024-02-23T00:38:49.226188+0000","last_became_peered":"2024-02-23T00:38:49.226188+0000","last_unstale":"2024-02-23T00:38:59.190793+0000","last_undegraded":"2024-02-23T00:38:59.190793+0000","last_fullsized":"2024-02-23T00:38:59.190793+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.933288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.933288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":500,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.188005+0000","last_change":"2024-02-23T00:38:49.184762+0000","last_active":"2024-02-23T00:38:59.188005+0000","last_peered":"2024-02-23T00:38:59.188005+0000","last_clean":"2024-02-23T00:38:59.188005+0000","last_became_active":"2024-02-23T00:38:49.184274+0000","last_became_peered":"2024-02-23T00:38:49.184274+0000","last_unstale":"2024-02-23T00:38:59.188005+0000","last_undegraded":"2024-02-23T00:38:59.188005+0000","last_fullsized":"2024-02-23T00:38:59.188005+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.008726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":448,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.211654+0000","last_change":"2024-02-23T00:38:44.173181+0000","last_active":"2024-02-23T00:38:59.211654+0000","last_peered":"2024-02-23T00:38:59.211654+0000","last_clean":"2024-02-23T00:38:59.211654+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:38:59.211654+0000","last_undegraded":"2024-02-23T00:38:59.211654+0000","last_fullsized":"2024-02-23T00:38:59.211654+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:44.173077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.173077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":510,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.197542+0000","last_change":"2024-02-23T00:37:59.003696+0000","last_active":"2024-02-23T00:38:59.197542+0000","last_peered":"2024-02-23T00:38:59.197542+0000","last_clean":"2024-02-23T00:38:59.197542+0000","last_became_active":"2024-02-23T00:35:27.097186+0000","last_became_peered":"2024-02-23T00:35:27.097186+0000","last_unstale":"2024-02-23T00:38:59.197542+0000","last_undegraded":"2024-02-23T00:38:59.197542+0000","last_fullsized":"2024-02-23T00:38:59.197542+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.003583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.003583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":576,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.186726+0000","last_change":"2024-02-23T00:38:19.164063+0000","last_active":"2024-02-23T00:38:59.186726+0000","last_peered":"2024-02-23T00:38:59.186726+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:38:19.164063+0000","last_became_peered":"2024-02-23T00:38:19.164063+0000","last_unstale":"2024-02-23T00:38:59.186726+0000","last_undegraded":"2024-02-23T00:38:19.153386+0000","last_fullsized":"2024-02-23T00:38:19.153249+0000","mapping_epoch":425,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":525,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.195711+0000","last_change":"2024-02-23T00:38:50.197166+0000","last_active":"2024-02-23T00:38:59.195711+0000","last_peered":"2024-02-23T00:38:59.195711+0000","last_clean":"2024-02-23T00:38:59.195711+0000","last_became_active":"2024-02-23T00:38:50.196782+0000","last_became_peered":"2024-02-23T00:38:50.196782+0000","last_unstale":"2024-02-23T00:38:59.195711+0000","last_undegraded":"2024-02-23T00:38:59.195711+0000","last_fullsized":"2024-02-23T00:38:59.195711+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.184937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.184937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,1],"avail_no_missing":["2","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":746,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.181097+0000","last_change":"2024-02-23T00:37:13.467508+0000","last_active":"2024-02-23T00:38:59.181097+0000","last_peered":"2024-02-23T00:38:59.181097+0000","last_clean":"2024-02-23T00:38:59.181097+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:38:59.181097+0000","last_undegraded":"2024-02-23T00:38:59.181097+0000","last_fullsized":"2024-02-23T00:38:59.181097+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:13.467428+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:13.467428+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":509,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.235202+0000","last_change":"2024-02-23T00:38:49.196976+0000","last_active":"2024-02-23T00:38:59.235202+0000","last_peered":"2024-02-23T00:38:59.235202+0000","last_clean":"2024-02-23T00:38:59.235202+0000","last_became_active":"2024-02-23T00:38:49.195404+0000","last_became_peered":"2024-02-23T00:38:49.195404+0000","last_unstale":"2024-02-23T00:38:59.235202+0000","last_undegraded":"2024-02-23T00:38:59.235202+0000","last_fullsized":"2024-02-23T00:38:59.235202+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:12.023125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:12.023125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":452,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.189839+0000","last_change":"2024-02-23T00:38:40.265472+0000","last_active":"2024-02-23T00:38:59.189839+0000","last_peered":"2024-02-23T00:38:59.189839+0000","last_clean":"2024-02-23T00:38:59.189839+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:38:59.189839+0000","last_undegraded":"2024-02-23T00:38:59.189839+0000","last_fullsized":"2024-02-23T00:38:59.189839+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:40.265385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.265385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":509,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.211932+0000","last_change":"2024-02-23T00:36:37.101373+0000","last_active":"2024-02-23T00:38:59.211932+0000","last_peered":"2024-02-23T00:38:59.211932+0000","last_clean":"2024-02-23T00:36:33.192308+0000","last_became_active":"2024-02-23T00:36:37.101373+0000","last_became_peered":"2024-02-23T00:36:37.101373+0000","last_unstale":"2024-02-23T00:38:59.211932+0000","last_undegraded":"2024-02-23T00:38:59.211932+0000","last_fullsized":"2024-02-23T00:36:37.094754+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.595447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.595447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":449,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.184859+0000","last_change":"2024-02-23T00:36:37.109549+0000","last_active":"2024-02-23T00:38:59.184859+0000","last_peered":"2024-02-23T00:38:59.184859+0000","last_clean":"2024-02-23T00:35:25.318841+0000","last_became_active":"2024-02-23T00:36:37.109549+0000","last_became_peered":"2024-02-23T00:36:37.109549+0000","last_unstale":"2024-02-23T00:38:59.184859+0000","last_undegraded":"2024-02-23T00:38:59.184859+0000","last_fullsized":"2024-02-23T00:36:37.096473+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":776,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-02-23T00:39:00.202996+0000","last_change":"2024-02-23T00:38:51.396520+0000","last_active":"2024-02-23T00:39:00.202996+0000","last_peered":"2024-02-23T00:39:00.202996+0000","last_clean":"2024-02-23T00:39:00.202996+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:39:00.202996+0000","last_undegraded":"2024-02-23T00:39:00.202996+0000","last_fullsized":"2024-02-23T00:39:00.202996+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'35","last_scrub_stamp":"2024-02-23T00:37:15.422435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:15.422435+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":128,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.181457+0000","last_change":"2024-02-23T00:37:23.360353+0000","last_active":"2024-02-23T00:38:59.181457+0000","last_peered":"2024-02-23T00:38:59.181457+0000","last_clean":"2024-02-23T00:38:59.181457+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:38:59.181457+0000","last_undegraded":"2024-02-23T00:38:59.181457+0000","last_fullsized":"2024-02-23T00:38:59.181457+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:37:23.360230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:23.360230+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.199912+0000","last_change":"2024-02-23T00:38:49.190209+0000","last_active":"2024-02-23T00:38:59.199912+0000","last_peered":"2024-02-23T00:38:59.199912+0000","last_clean":"2024-02-23T00:38:59.199912+0000","last_became_active":"2024-02-23T00:38:49.189587+0000","last_became_peered":"2024-02-23T00:38:49.189587+0000","last_unstale":"2024-02-23T00:38:59.199912+0000","last_undegraded":"2024-02-23T00:38:59.199912+0000","last_fullsized":"2024-02-23T00:38:59.199912+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.098280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.098280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":452,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.197700+0000","last_change":"2024-02-23T00:38:43.332992+0000","last_active":"2024-02-23T00:38:59.197700+0000","last_peered":"2024-02-23T00:38:59.197700+0000","last_clean":"2024-02-23T00:38:59.197700+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:38:59.197700+0000","last_undegraded":"2024-02-23T00:38:59.197700+0000","last_fullsized":"2024-02-23T00:38:59.197700+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:43.332907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:43.332907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":416,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.185557+0000","last_change":"2024-02-23T00:38:49.184857+0000","last_active":"2024-02-23T00:38:59.185557+0000","last_peered":"2024-02-23T00:38:59.185557+0000","last_clean":"2024-02-23T00:38:59.185557+0000","last_became_active":"2024-02-23T00:38:49.183777+0000","last_became_peered":"2024-02-23T00:38:49.183777+0000","last_unstale":"2024-02-23T00:38:59.185557+0000","last_undegraded":"2024-02-23T00:38:59.185557+0000","last_fullsized":"2024-02-23T00:38:59.185557+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:23.210706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.210706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":123,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.224857+0000","last_change":"2024-02-23T00:36:37.104760+0000","last_active":"2024-02-23T00:38:59.224857+0000","last_peered":"2024-02-23T00:38:59.224857+0000","last_clean":"2024-02-23T00:27:24.172824+0000","last_became_active":"2024-02-23T00:36:37.104760+0000","last_became_peered":"2024-02-23T00:36:37.104760+0000","last_unstale":"2024-02-23T00:38:59.224857+0000","last_undegraded":"2024-02-23T00:38:59.224857+0000","last_fullsized":"2024-02-23T00:36:37.098375+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":161,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.187591+0000","last_change":"2024-02-23T00:36:37.110364+0000","last_active":"2024-02-23T00:38:59.187591+0000","last_peered":"2024-02-23T00:38:59.187591+0000","last_clean":"2024-02-23T00:27:29.173100+0000","last_became_active":"2024-02-23T00:36:37.110364+0000","last_became_peered":"2024-02-23T00:36:37.110364+0000","last_unstale":"2024-02-23T00:38:59.187591+0000","last_undegraded":"2024-02-23T00:36:37.096575+0000","last_fullsized":"2024-02-23T00:36:37.096459+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:35:44.214727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:44.214727+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":494,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.182364+0000","last_change":"2024-02-23T00:36:37.123256+0000","last_active":"2024-02-23T00:38:59.182364+0000","last_peered":"2024-02-23T00:38:59.182364+0000","last_clean":"2024-02-23T00:36:33.366566+0000","last_became_active":"2024-02-23T00:36:37.123256+0000","last_became_peered":"2024-02-23T00:36:37.123256+0000","last_unstale":"2024-02-23T00:38:59.182364+0000","last_undegraded":"2024-02-23T00:38:59.182364+0000","last_fullsized":"2024-02-23T00:36:37.096440+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":20,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.195344+0000","last_change":"2024-02-23T00:38:50.180582+0000","last_active":"2024-02-23T00:38:59.195344+0000","last_peered":"2024-02-23T00:38:59.195344+0000","last_clean":"2024-02-23T00:38:59.195344+0000","last_became_active":"2024-02-23T00:38:50.177038+0000","last_became_peered":"2024-02-23T00:38:50.177038+0000","last_unstale":"2024-02-23T00:38:59.195344+0000","last_undegraded":"2024-02-23T00:38:59.195344+0000","last_fullsized":"2024-02-23T00:38:59.195344+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.488959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.488959+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,5],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":441,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.210959+0000","last_change":"2024-02-23T00:36:42.093272+0000","last_active":"2024-02-23T00:38:59.210959+0000","last_peered":"2024-02-23T00:38:59.210959+0000","last_clean":"2024-02-23T00:38:59.210959+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:38:59.210959+0000","last_undegraded":"2024-02-23T00:38:59.210959+0000","last_fullsized":"2024-02-23T00:38:59.210959+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:36:42.093181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":506,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.188349+0000","last_change":"2024-02-23T00:36:37.124382+0000","last_active":"2024-02-23T00:38:59.188349+0000","last_peered":"2024-02-23T00:38:59.188349+0000","last_clean":"2024-02-23T00:36:33.193431+0000","last_became_active":"2024-02-23T00:36:37.124382+0000","last_became_peered":"2024-02-23T00:36:37.124382+0000","last_unstale":"2024-02-23T00:38:59.188349+0000","last_undegraded":"2024-02-23T00:38:59.188349+0000","last_fullsized":"2024-02-23T00:36:37.101474+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:24.016837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:24.016837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":713,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.210932+0000","last_change":"2024-02-23T00:38:54.171994+0000","last_active":"2024-02-23T00:38:59.210932+0000","last_peered":"2024-02-23T00:38:59.210932+0000","last_clean":"2024-02-23T00:38:59.210932+0000","last_became_active":"2024-02-23T00:38:54.171563+0000","last_became_peered":"2024-02-23T00:38:54.171563+0000","last_unstale":"2024-02-23T00:38:59.210932+0000","last_undegraded":"2024-02-23T00:38:59.210932+0000","last_fullsized":"2024-02-23T00:38:59.210932+0000","mapping_epoch":516,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:06.122240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.122240+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":513,"reported_epoch":520,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:38:59.211779+0000","last_change":"2024-02-23T00:36:37.118733+0000","last_active":"2024-02-23T00:38:59.211779+0000","last_peered":"2024-02-23T00:38:59.211779+0000","last_clean":"2024-02-23T00:36:36.130983+0000","last_became_active":"2024-02-23T00:36:37.118733+0000","last_became_peered":"2024-02-23T00:36:37.118733+0000","last_unstale":"2024-02-23T00:38:59.211779+0000","last_undegraded":"2024-02-23T00:36:37.095672+0000","last_fullsized":"2024-02-23T00:36:37.095564+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:36:36.130868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.130868+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.194831+0000","last_change":"2024-02-23T00:38:49.220203+0000","last_active":"2024-02-23T00:38:59.194831+0000","last_peered":"2024-02-23T00:38:59.194831+0000","last_clean":"2024-02-23T00:38:59.194831+0000","last_became_active":"2024-02-23T00:38:49.219745+0000","last_became_peered":"2024-02-23T00:38:49.219745+0000","last_unstale":"2024-02-23T00:38:59.194831+0000","last_undegraded":"2024-02-23T00:38:59.194831+0000","last_fullsized":"2024-02-23T00:38:59.194831+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.172059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.172059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":437,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.186107+0000","last_change":"2024-02-23T00:36:28.184266+0000","last_active":"2024-02-23T00:38:59.186107+0000","last_peered":"2024-02-23T00:38:59.186107+0000","last_clean":"2024-02-23T00:38:59.186107+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:38:59.186107+0000","last_undegraded":"2024-02-23T00:38:59.186107+0000","last_fullsized":"2024-02-23T00:38:59.186107+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:28.184197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:28.184197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":212,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.198141+0000","last_change":"2024-02-23T00:38:54.181474+0000","last_active":"2024-02-23T00:38:59.198141+0000","last_peered":"2024-02-23T00:38:59.198141+0000","last_clean":"2024-02-23T00:38:59.198141+0000","last_became_active":"2024-02-23T00:38:54.180971+0000","last_became_peered":"2024-02-23T00:38:54.180971+0000","last_unstale":"2024-02-23T00:38:59.198141+0000","last_undegraded":"2024-02-23T00:38:59.198141+0000","last_fullsized":"2024-02-23T00:38:59.198141+0000","mapping_epoch":516,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":501,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.188210+0000","last_change":"2024-02-23T00:36:37.111836+0000","last_active":"2024-02-23T00:38:59.188210+0000","last_peered":"2024-02-23T00:38:59.188210+0000","last_clean":"2024-02-23T00:36:33.193391+0000","last_became_active":"2024-02-23T00:36:37.111836+0000","last_became_peered":"2024-02-23T00:36:37.111836+0000","last_unstale":"2024-02-23T00:38:59.188210+0000","last_undegraded":"2024-02-23T00:38:59.188210+0000","last_fullsized":"2024-02-23T00:36:37.101173+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.667517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.667517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.199691+0000","last_change":"2024-02-23T00:38:49.222146+0000","last_active":"2024-02-23T00:38:59.199691+0000","last_peered":"2024-02-23T00:38:59.199691+0000","last_clean":"2024-02-23T00:38:59.199691+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:38:59.199691+0000","last_undegraded":"2024-02-23T00:38:59.199691+0000","last_fullsized":"2024-02-23T00:38:59.199691+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.614960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.614960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":506,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.190608+0000","last_change":"2024-02-23T00:38:42.293920+0000","last_active":"2024-02-23T00:38:59.190608+0000","last_peered":"2024-02-23T00:38:59.190608+0000","last_clean":"2024-02-23T00:38:59.190608+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:38:59.190608+0000","last_undegraded":"2024-02-23T00:38:59.190608+0000","last_fullsized":"2024-02-23T00:38:59.190608+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.293824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.293824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.197390+0000","last_change":"2024-02-23T00:38:49.230851+0000","last_active":"2024-02-23T00:38:59.197390+0000","last_peered":"2024-02-23T00:38:59.197390+0000","last_clean":"2024-02-23T00:38:59.197390+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:38:59.197390+0000","last_undegraded":"2024-02-23T00:38:59.197390+0000","last_fullsized":"2024-02-23T00:38:59.197390+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:39.893315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.893315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":596,"reported_epoch":520,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:59.186292+0000","last_change":"2024-02-23T00:38:49.248673+0000","last_active":"2024-02-23T00:38:59.186292+0000","last_peered":"2024-02-23T00:38:59.186292+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:38:49.224732+0000","last_became_peered":"2024-02-23T00:38:49.224732+0000","last_unstale":"2024-02-23T00:38:59.186292+0000","last_undegraded":"2024-02-23T00:38:49.159472+0000","last_fullsized":"2024-02-23T00:38:49.159180+0000","mapping_epoch":511,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":499,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.225273+0000","last_change":"2024-02-23T00:38:42.431748+0000","last_active":"2024-02-23T00:38:59.225273+0000","last_peered":"2024-02-23T00:38:59.225273+0000","last_clean":"2024-02-23T00:38:59.225273+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:38:59.225273+0000","last_undegraded":"2024-02-23T00:38:59.225273+0000","last_fullsized":"2024-02-23T00:38:59.225273+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.431641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":497,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.186379+0000","last_change":"2024-02-23T00:38:49.190282+0000","last_active":"2024-02-23T00:38:59.186379+0000","last_peered":"2024-02-23T00:38:59.186379+0000","last_clean":"2024-02-23T00:38:59.186379+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:38:59.186379+0000","last_undegraded":"2024-02-23T00:38:59.186379+0000","last_fullsized":"2024-02-23T00:38:59.186379+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.290946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.290946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":491,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.230047+0000","last_change":"2024-02-23T00:37:16.011330+0000","last_active":"2024-02-23T00:38:59.230047+0000","last_peered":"2024-02-23T00:38:59.230047+0000","last_clean":"2024-02-23T00:38:59.230047+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:38:59.230047+0000","last_undegraded":"2024-02-23T00:38:59.230047+0000","last_fullsized":"2024-02-23T00:38:59.230047+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:16.011248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:37:16.011248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":459,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.226697+0000","last_change":"2024-02-23T00:38:49.195614+0000","last_active":"2024-02-23T00:38:59.226697+0000","last_peered":"2024-02-23T00:38:59.226697+0000","last_clean":"2024-02-23T00:38:59.226697+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:38:59.226697+0000","last_undegraded":"2024-02-23T00:38:59.226697+0000","last_fullsized":"2024-02-23T00:38:59.226697+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.316429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.316429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"453'32","reported_seq":35,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.197172+0000","last_change":"2024-02-23T00:38:51.793947+0000","last_active":"2024-02-23T00:38:59.197172+0000","last_peered":"2024-02-23T00:38:59.197172+0000","last_clean":"2024-02-23T00:38:59.197172+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:38:59.197172+0000","last_undegraded":"2024-02-23T00:38:59.197172+0000","last_fullsized":"2024-02-23T00:38:59.197172+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:38:44.240437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.240437+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":512,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.187272+0000","last_change":"2024-02-23T00:38:49.186281+0000","last_active":"2024-02-23T00:38:59.187272+0000","last_peered":"2024-02-23T00:38:59.187272+0000","last_clean":"2024-02-23T00:38:59.187272+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:38:59.187272+0000","last_undegraded":"2024-02-23T00:38:59.187272+0000","last_fullsized":"2024-02-23T00:38:59.187272+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:46.269798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.269798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"426'10","reported_seq":636,"reported_epoch":521,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:00.400039+0000","last_change":"2024-02-23T00:36:37.113534+0000","last_active":"2024-02-23T00:39:00.400039+0000","last_peered":"2024-02-23T00:39:00.400039+0000","last_clean":"2024-02-23T00:36:35.444798+0000","last_became_active":"2024-02-23T00:36:37.113534+0000","last_became_peered":"2024-02-23T00:36:37.113534+0000","last_unstale":"2024-02-23T00:39:00.400039+0000","last_undegraded":"2024-02-23T00:36:37.102235+0000","last_fullsized":"2024-02-23T00:36:37.102132+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.199928+0000","last_change":"2024-02-23T00:38:49.190032+0000","last_active":"2024-02-23T00:38:59.199928+0000","last_peered":"2024-02-23T00:38:59.199928+0000","last_clean":"2024-02-23T00:38:59.199928+0000","last_became_active":"2024-02-23T00:38:49.189564+0000","last_became_peered":"2024-02-23T00:38:49.189564+0000","last_unstale":"2024-02-23T00:38:59.199928+0000","last_undegraded":"2024-02-23T00:38:59.199928+0000","last_fullsized":"2024-02-23T00:38:59.199928+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:41.177673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:38:41.177673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":437,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.187524+0000","last_change":"2024-02-23T00:37:08.058170+0000","last_active":"2024-02-23T00:38:59.187524+0000","last_peered":"2024-02-23T00:38:59.187524+0000","last_clean":"2024-02-23T00:38:59.187524+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:38:59.187524+0000","last_undegraded":"2024-02-23T00:38:59.187524+0000","last_fullsized":"2024-02-23T00:38:59.187524+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:08.058075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.058075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":573,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.187442+0000","last_change":"2024-02-23T00:38:49.346222+0000","last_active":"2024-02-23T00:38:59.187442+0000","last_peered":"2024-02-23T00:38:59.187442+0000","last_clean":"2024-02-23T00:38:59.187442+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:38:59.187442+0000","last_undegraded":"2024-02-23T00:38:59.187442+0000","last_fullsized":"2024-02-23T00:38:59.187442+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:40.858800+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.858800+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":417,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.212970+0000","last_change":"2024-02-23T00:38:50.181211+0000","last_active":"2024-02-23T00:38:59.212970+0000","last_peered":"2024-02-23T00:38:59.212970+0000","last_clean":"2024-02-23T00:38:59.212970+0000","last_became_active":"2024-02-23T00:38:50.180740+0000","last_became_peered":"2024-02-23T00:38:50.180740+0000","last_unstale":"2024-02-23T00:38:59.212970+0000","last_undegraded":"2024-02-23T00:38:59.212970+0000","last_fullsized":"2024-02-23T00:38:59.212970+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.391997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.391997+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,7],"avail_no_missing":["4","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":26,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-02-23T00:39:00.195169+0000","last_change":"2024-02-23T00:38:49.384331+0000","last_active":"2024-02-23T00:39:00.195169+0000","last_peered":"2024-02-23T00:39:00.195169+0000","last_clean":"2024-02-23T00:39:00.195169+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:39:00.195169+0000","last_undegraded":"2024-02-23T00:39:00.195169+0000","last_fullsized":"2024-02-23T00:39:00.195169+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'10","last_scrub_stamp":"2024-02-23T00:37:27.870653+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:37:27.870653+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":40,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.187689+0000","last_change":"2024-02-23T00:38:49.184581+0000","last_active":"2024-02-23T00:38:59.187689+0000","last_peered":"2024-02-23T00:38:59.187689+0000","last_clean":"2024-02-23T00:38:59.187689+0000","last_became_active":"2024-02-23T00:38:49.183777+0000","last_became_peered":"2024-02-23T00:38:49.183777+0000","last_unstale":"2024-02-23T00:38:59.187689+0000","last_undegraded":"2024-02-23T00:38:59.187689+0000","last_fullsized":"2024-02-23T00:38:59.187689+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:05.095708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:05.095708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.198946+0000","last_change":"2024-02-23T00:38:49.227700+0000","last_active":"2024-02-23T00:38:59.198946+0000","last_peered":"2024-02-23T00:38:59.198946+0000","last_clean":"2024-02-23T00:38:59.198946+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:38:59.198946+0000","last_undegraded":"2024-02-23T00:38:59.198946+0000","last_fullsized":"2024-02-23T00:38:59.198946+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:09.447326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.447326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":513,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.181845+0000","last_change":"2024-02-23T00:37:20.382193+0000","last_active":"2024-02-23T00:38:59.181845+0000","last_peered":"2024-02-23T00:38:59.181845+0000","last_clean":"2024-02-23T00:38:59.181845+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:38:59.181845+0000","last_undegraded":"2024-02-23T00:38:59.181845+0000","last_fullsized":"2024-02-23T00:38:59.181845+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:37:20.382108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.382108+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":517,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.225418+0000","last_change":"2024-02-23T00:38:49.195750+0000","last_active":"2024-02-23T00:38:59.225418+0000","last_peered":"2024-02-23T00:38:59.225418+0000","last_clean":"2024-02-23T00:38:59.225418+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:38:59.225418+0000","last_undegraded":"2024-02-23T00:38:59.225418+0000","last_fullsized":"2024-02-23T00:38:59.225418+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.470333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.470333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":20,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.194699+0000","last_change":"2024-02-23T00:38:50.200217+0000","last_active":"2024-02-23T00:38:59.194699+0000","last_peered":"2024-02-23T00:38:59.194699+0000","last_clean":"2024-02-23T00:38:59.194699+0000","last_became_active":"2024-02-23T00:38:50.199759+0000","last_became_peered":"2024-02-23T00:38:50.199759+0000","last_unstale":"2024-02-23T00:38:59.194699+0000","last_undegraded":"2024-02-23T00:38:59.194699+0000","last_fullsized":"2024-02-23T00:38:59.194699+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.416493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.416493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":103,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.185604+0000","last_change":"2024-02-23T00:36:37.112999+0000","last_active":"2024-02-23T00:38:59.185604+0000","last_peered":"2024-02-23T00:38:59.185604+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.112999+0000","last_became_peered":"2024-02-23T00:36:37.112999+0000","last_unstale":"2024-02-23T00:38:59.185604+0000","last_undegraded":"2024-02-23T00:38:59.185604+0000","last_fullsized":"2024-02-23T00:36:37.102073+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.413991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.413991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":447,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.210853+0000","last_change":"2024-02-23T00:36:37.097581+0000","last_active":"2024-02-23T00:38:59.210853+0000","last_peered":"2024-02-23T00:38:59.210853+0000","last_clean":"2024-02-23T00:36:33.191411+0000","last_became_active":"2024-02-23T00:36:37.097581+0000","last_became_peered":"2024-02-23T00:36:37.097581+0000","last_unstale":"2024-02-23T00:38:59.210853+0000","last_undegraded":"2024-02-23T00:38:59.210853+0000","last_fullsized":"2024-02-23T00:36:37.093707+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.101501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.101501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":549,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.212837+0000","last_change":"2024-02-23T00:38:16.174640+0000","last_active":"2024-02-23T00:38:59.212837+0000","last_peered":"2024-02-23T00:38:59.212837+0000","last_clean":"2024-02-23T00:38:59.212837+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:38:59.212837+0000","last_undegraded":"2024-02-23T00:38:59.212837+0000","last_fullsized":"2024-02-23T00:38:59.212837+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:38:16.174526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.174526+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":491,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.185002+0000","last_change":"2024-02-23T00:36:37.108194+0000","last_active":"2024-02-23T00:38:59.185002+0000","last_peered":"2024-02-23T00:38:59.185002+0000","last_clean":"2024-02-23T00:36:33.367482+0000","last_became_active":"2024-02-23T00:36:37.108194+0000","last_became_peered":"2024-02-23T00:36:37.108194+0000","last_unstale":"2024-02-23T00:38:59.185002+0000","last_undegraded":"2024-02-23T00:38:59.185002+0000","last_fullsized":"2024-02-23T00:36:37.096730+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.783089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.783089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":504,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.181884+0000","last_change":"2024-02-23T00:38:49.225280+0000","last_active":"2024-02-23T00:38:59.181884+0000","last_peered":"2024-02-23T00:38:59.181884+0000","last_clean":"2024-02-23T00:38:59.181884+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:38:59.181884+0000","last_undegraded":"2024-02-23T00:38:59.181884+0000","last_fullsized":"2024-02-23T00:38:59.181884+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:06.235608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.235608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":20,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.194130+0000","last_change":"2024-02-23T00:38:50.185137+0000","last_active":"2024-02-23T00:38:59.194130+0000","last_peered":"2024-02-23T00:38:59.194130+0000","last_clean":"2024-02-23T00:38:59.194130+0000","last_became_active":"2024-02-23T00:38:50.179247+0000","last_became_peered":"2024-02-23T00:38:50.179247+0000","last_unstale":"2024-02-23T00:38:59.194130+0000","last_undegraded":"2024-02-23T00:38:59.194130+0000","last_fullsized":"2024-02-23T00:38:59.194130+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:17.620107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.620107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,2],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":483,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.225460+0000","last_change":"2024-02-23T00:38:51.510612+0000","last_active":"2024-02-23T00:38:59.225460+0000","last_peered":"2024-02-23T00:38:59.225460+0000","last_clean":"2024-02-23T00:38:59.225460+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:38:59.225460+0000","last_undegraded":"2024-02-23T00:38:59.225460+0000","last_fullsized":"2024-02-23T00:38:59.225460+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:35:47.130749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.130749+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":25,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.196429+0000","last_change":"2024-02-23T00:38:49.987321+0000","last_active":"2024-02-23T00:38:59.196429+0000","last_peered":"2024-02-23T00:38:59.196429+0000","last_clean":"2024-02-23T00:38:59.196429+0000","last_became_active":"2024-02-23T00:38:49.217612+0000","last_became_peered":"2024-02-23T00:38:49.217612+0000","last_unstale":"2024-02-23T00:38:59.196429+0000","last_undegraded":"2024-02-23T00:38:59.196429+0000","last_fullsized":"2024-02-23T00:38:59.196429+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:36:36.531424+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.531424+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":488,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.186622+0000","last_change":"2024-02-23T00:36:32.199878+0000","last_active":"2024-02-23T00:38:59.186622+0000","last_peered":"2024-02-23T00:38:59.186622+0000","last_clean":"2024-02-23T00:38:59.186622+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:38:59.186622+0000","last_undegraded":"2024-02-23T00:38:59.186622+0000","last_fullsized":"2024-02-23T00:38:59.186622+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.199765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.199765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":490,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.225182+0000","last_change":"2024-02-23T00:38:02.325317+0000","last_active":"2024-02-23T00:38:59.225182+0000","last_peered":"2024-02-23T00:38:59.225182+0000","last_clean":"2024-02-23T00:38:59.225182+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:38:59.225182+0000","last_undegraded":"2024-02-23T00:38:59.225182+0000","last_fullsized":"2024-02-23T00:38:59.225182+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:02.325254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:02.325254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":491,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.232090+0000","last_change":"2024-02-23T00:37:14.009706+0000","last_active":"2024-02-23T00:38:59.232090+0000","last_peered":"2024-02-23T00:38:59.232090+0000","last_clean":"2024-02-23T00:38:59.232090+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:38:59.232090+0000","last_undegraded":"2024-02-23T00:38:59.232090+0000","last_fullsized":"2024-02-23T00:38:59.232090+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.009614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.009614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":437,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.181938+0000","last_change":"2024-02-23T00:38:33.242488+0000","last_active":"2024-02-23T00:38:59.181938+0000","last_peered":"2024-02-23T00:38:59.181938+0000","last_clean":"2024-02-23T00:38:59.181938+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:38:59.181938+0000","last_undegraded":"2024-02-23T00:38:59.181938+0000","last_fullsized":"2024-02-23T00:38:59.181938+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:33.242417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:33.242417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":36,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.194202+0000","last_change":"2024-02-23T00:38:51.365185+0000","last_active":"2024-02-23T00:38:59.194202+0000","last_peered":"2024-02-23T00:38:59.194202+0000","last_clean":"2024-02-23T00:38:59.194202+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:38:59.194202+0000","last_undegraded":"2024-02-23T00:38:59.194202+0000","last_fullsized":"2024-02-23T00:38:59.194202+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:37:26.914698+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.914698+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":493,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.227186+0000","last_change":"2024-02-23T00:37:22.340222+0000","last_active":"2024-02-23T00:38:59.227186+0000","last_peered":"2024-02-23T00:38:59.227186+0000","last_clean":"2024-02-23T00:38:59.227186+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:38:59.227186+0000","last_undegraded":"2024-02-23T00:38:59.227186+0000","last_fullsized":"2024-02-23T00:38:59.227186+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:22.340128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.340128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":493,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.231351+0000","last_change":"2024-02-23T00:38:16.055268+0000","last_active":"2024-02-23T00:38:59.231351+0000","last_peered":"2024-02-23T00:38:59.231351+0000","last_clean":"2024-02-23T00:38:59.231351+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:38:59.231351+0000","last_undegraded":"2024-02-23T00:38:59.231351+0000","last_fullsized":"2024-02-23T00:38:59.231351+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.055159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":20,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.194490+0000","last_change":"2024-02-23T00:38:50.184807+0000","last_active":"2024-02-23T00:38:59.194490+0000","last_peered":"2024-02-23T00:38:59.194490+0000","last_clean":"2024-02-23T00:38:59.194490+0000","last_became_active":"2024-02-23T00:38:50.184585+0000","last_became_peered":"2024-02-23T00:38:50.184585+0000","last_unstale":"2024-02-23T00:38:59.194490+0000","last_undegraded":"2024-02-23T00:38:59.194490+0000","last_fullsized":"2024-02-23T00:38:59.194490+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.175171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.175171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.194622+0000","last_change":"2024-02-23T00:38:49.189171+0000","last_active":"2024-02-23T00:38:59.194622+0000","last_peered":"2024-02-23T00:38:59.194622+0000","last_clean":"2024-02-23T00:38:59.194622+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:38:59.194622+0000","last_undegraded":"2024-02-23T00:38:59.194622+0000","last_fullsized":"2024-02-23T00:38:59.194622+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:37.979785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.979785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":614,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.234468+0000","last_change":"2024-02-23T00:38:54.170696+0000","last_active":"2024-02-23T00:38:59.234468+0000","last_peered":"2024-02-23T00:38:59.234468+0000","last_clean":"2024-02-23T00:38:59.234468+0000","last_became_active":"2024-02-23T00:38:54.170041+0000","last_became_peered":"2024-02-23T00:38:54.170041+0000","last_unstale":"2024-02-23T00:38:59.234468+0000","last_undegraded":"2024-02-23T00:38:59.234468+0000","last_fullsized":"2024-02-23T00:38:59.234468+0000","mapping_epoch":516,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,1],"avail_no_missing":["7","1","3"],"object_location_counts":[{"shards":"1,3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":497,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.185487+0000","last_change":"2024-02-23T00:38:36.853659+0000","last_active":"2024-02-23T00:38:59.185487+0000","last_peered":"2024-02-23T00:38:59.185487+0000","last_clean":"2024-02-23T00:38:59.185487+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:38:59.185487+0000","last_undegraded":"2024-02-23T00:38:59.185487+0000","last_fullsized":"2024-02-23T00:38:59.185487+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:36.853582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:36.853582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":647,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-02-23T00:39:00.192347+0000","last_change":"2024-02-23T00:38:37.455385+0000","last_active":"2024-02-23T00:39:00.192347+0000","last_peered":"2024-02-23T00:39:00.192347+0000","last_clean":"2024-02-23T00:39:00.192347+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:39:00.192347+0000","last_undegraded":"2024-02-23T00:39:00.192347+0000","last_fullsized":"2024-02-23T00:39:00.192347+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:38:37.455280+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.455280+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":23,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.197527+0000","last_change":"2024-02-23T00:38:51.364248+0000","last_active":"2024-02-23T00:38:59.197527+0000","last_peered":"2024-02-23T00:38:59.197527+0000","last_clean":"2024-02-23T00:38:59.197527+0000","last_became_active":"2024-02-23T00:38:49.230008+0000","last_became_peered":"2024-02-23T00:38:49.230008+0000","last_unstale":"2024-02-23T00:38:59.197527+0000","last_undegraded":"2024-02-23T00:38:59.197527+0000","last_fullsized":"2024-02-23T00:38:59.197527+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:38:09.207232+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:38:09.207232+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.207232+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":465,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.233900+0000","last_change":"2024-02-23T00:38:50.185706+0000","last_active":"2024-02-23T00:38:59.233900+0000","last_peered":"2024-02-23T00:38:59.233900+0000","last_clean":"2024-02-23T00:38:59.233900+0000","last_became_active":"2024-02-23T00:38:50.185295+0000","last_became_peered":"2024-02-23T00:38:50.185295+0000","last_unstale":"2024-02-23T00:38:59.233900+0000","last_undegraded":"2024-02-23T00:38:59.233900+0000","last_fullsized":"2024-02-23T00:38:59.233900+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.028166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.028166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,2],"avail_no_missing":["7","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":653,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:59.185957+0000","last_change":"2024-02-23T00:38:52.605457+0000","last_active":"2024-02-23T00:38:59.185957+0000","last_peered":"2024-02-23T00:38:59.185957+0000","last_clean":"2024-02-23T00:38:19.193308+0000","last_became_active":"2024-02-23T00:38:22.159363+0000","last_became_peered":"2024-02-23T00:38:22.159363+0000","last_unstale":"2024-02-23T00:38:59.185957+0000","last_undegraded":"2024-02-23T00:38:59.185957+0000","last_fullsized":"2024-02-23T00:38:59.185957+0000","mapping_epoch":484,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":20,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.194058+0000","last_change":"2024-02-23T00:38:50.176938+0000","last_active":"2024-02-23T00:38:59.194058+0000","last_peered":"2024-02-23T00:38:59.194058+0000","last_clean":"2024-02-23T00:38:59.194058+0000","last_became_active":"2024-02-23T00:38:50.176504+0000","last_became_peered":"2024-02-23T00:38:50.176504+0000","last_unstale":"2024-02-23T00:38:59.194058+0000","last_undegraded":"2024-02-23T00:38:59.194058+0000","last_fullsized":"2024-02-23T00:38:59.194058+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.151731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":103,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.235322+0000","last_change":"2024-02-23T00:36:37.122400+0000","last_active":"2024-02-23T00:38:59.235322+0000","last_peered":"2024-02-23T00:38:59.235322+0000","last_clean":"2024-02-23T00:27:20.002687+0000","last_became_active":"2024-02-23T00:36:37.122400+0000","last_became_peered":"2024-02-23T00:36:37.122400+0000","last_unstale":"2024-02-23T00:38:59.235322+0000","last_undegraded":"2024-02-23T00:38:59.235322+0000","last_fullsized":"2024-02-23T00:36:37.111280+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":37,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.188785+0000","last_change":"2024-02-23T00:38:42.787673+0000","last_active":"2024-02-23T00:38:59.188785+0000","last_peered":"2024-02-23T00:38:59.188785+0000","last_clean":"2024-02-23T00:38:59.188785+0000","last_became_active":"2024-02-23T00:38:40.171498+0000","last_became_peered":"2024-02-23T00:38:40.171498+0000","last_unstale":"2024-02-23T00:38:59.188785+0000","last_undegraded":"2024-02-23T00:38:59.188785+0000","last_fullsized":"2024-02-23T00:38:59.188785+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.787571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.787571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,4],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":445,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.235553+0000","last_change":"2024-02-23T00:38:35.044376+0000","last_active":"2024-02-23T00:38:59.235553+0000","last_peered":"2024-02-23T00:38:59.235553+0000","last_clean":"2024-02-23T00:38:59.235553+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:38:59.235553+0000","last_undegraded":"2024-02-23T00:38:59.235553+0000","last_fullsized":"2024-02-23T00:38:59.235553+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:35.044298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.044298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":628,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.190326+0000","last_change":"2024-02-23T00:38:52.159547+0000","last_active":"2024-02-23T00:38:59.190326+0000","last_peered":"2024-02-23T00:38:59.190326+0000","last_clean":"2024-02-23T00:38:59.190326+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:38:59.190326+0000","last_undegraded":"2024-02-23T00:38:59.190326+0000","last_fullsized":"2024-02-23T00:38:59.190326+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:38:39.248198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.248198+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":502,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.225567+0000","last_change":"2024-02-23T00:38:07.395392+0000","last_active":"2024-02-23T00:38:59.225567+0000","last_peered":"2024-02-23T00:38:59.225567+0000","last_clean":"2024-02-23T00:38:59.225567+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:38:59.225567+0000","last_undegraded":"2024-02-23T00:38:59.225567+0000","last_fullsized":"2024-02-23T00:38:59.225567+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.395260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.395260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":489,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.185280+0000","last_change":"2024-02-23T00:36:37.113234+0000","last_active":"2024-02-23T00:38:59.185280+0000","last_peered":"2024-02-23T00:38:59.185280+0000","last_clean":"2024-02-23T00:36:33.367528+0000","last_became_active":"2024-02-23T00:36:37.113234+0000","last_became_peered":"2024-02-23T00:36:37.113234+0000","last_unstale":"2024-02-23T00:38:59.185280+0000","last_undegraded":"2024-02-23T00:38:59.185280+0000","last_fullsized":"2024-02-23T00:36:37.097125+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.791748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.791748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.199208+0000","last_change":"2024-02-23T00:38:49.227843+0000","last_active":"2024-02-23T00:38:59.199208+0000","last_peered":"2024-02-23T00:38:59.199208+0000","last_clean":"2024-02-23T00:38:59.199208+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:38:59.199208+0000","last_undegraded":"2024-02-23T00:38:59.199208+0000","last_fullsized":"2024-02-23T00:38:59.199208+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:27.951690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.951690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":603,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.198322+0000","last_change":"2024-02-23T00:38:55.178058+0000","last_active":"2024-02-23T00:38:59.198322+0000","last_peered":"2024-02-23T00:38:59.198322+0000","last_clean":"2024-02-23T00:38:59.198322+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:38:59.198322+0000","last_undegraded":"2024-02-23T00:38:59.198322+0000","last_fullsized":"2024-02-23T00:38:59.198322+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":29,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.194876+0000","last_change":"2024-02-23T00:38:49.192589+0000","last_active":"2024-02-23T00:38:59.194876+0000","last_peered":"2024-02-23T00:38:59.194876+0000","last_clean":"2024-02-23T00:38:59.194876+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:38:59.194876+0000","last_undegraded":"2024-02-23T00:38:59.194876+0000","last_fullsized":"2024-02-23T00:38:59.194876+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:35.929507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:35.929507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":508,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.213908+0000","last_change":"2024-02-23T00:38:49.224957+0000","last_active":"2024-02-23T00:38:59.213908+0000","last_peered":"2024-02-23T00:38:59.213908+0000","last_clean":"2024-02-23T00:38:59.213908+0000","last_became_active":"2024-02-23T00:38:49.224630+0000","last_became_peered":"2024-02-23T00:38:49.224630+0000","last_unstale":"2024-02-23T00:38:59.213908+0000","last_undegraded":"2024-02-23T00:38:59.213908+0000","last_fullsized":"2024-02-23T00:38:59.213908+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.063485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.063485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":437,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.187582+0000","last_change":"2024-02-23T00:38:03.798894+0000","last_active":"2024-02-23T00:38:59.187582+0000","last_peered":"2024-02-23T00:38:59.187582+0000","last_clean":"2024-02-23T00:38:59.187582+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:38:59.187582+0000","last_undegraded":"2024-02-23T00:38:59.187582+0000","last_fullsized":"2024-02-23T00:38:59.187582+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.798824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.798824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":609,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:59.181097+0000","last_change":"2024-02-23T00:38:49.248692+0000","last_active":"2024-02-23T00:38:59.181097+0000","last_peered":"2024-02-23T00:38:59.181097+0000","last_clean":"2024-02-23T00:38:13.519476+0000","last_became_active":"2024-02-23T00:38:49.225155+0000","last_became_peered":"2024-02-23T00:38:49.225155+0000","last_unstale":"2024-02-23T00:38:59.181097+0000","last_undegraded":"2024-02-23T00:38:59.181097+0000","last_fullsized":"2024-02-23T00:38:59.181097+0000","mapping_epoch":511,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[1,6,7],"avail_no_missing":["1","6","7"],"object_location_counts":[{"shards":"1,6,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":204,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.212950+0000","last_change":"2024-02-23T00:37:59.129696+0000","last_active":"2024-02-23T00:38:59.212950+0000","last_peered":"2024-02-23T00:38:59.212950+0000","last_clean":"2024-02-23T00:38:59.212950+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:38:59.212950+0000","last_undegraded":"2024-02-23T00:38:59.212950+0000","last_fullsized":"2024-02-23T00:38:59.212950+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.129604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.129604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":485,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.186720+0000","last_change":"2024-02-23T00:36:37.124006+0000","last_active":"2024-02-23T00:38:59.186720+0000","last_peered":"2024-02-23T00:38:59.186720+0000","last_clean":"2024-02-23T00:36:33.366419+0000","last_became_active":"2024-02-23T00:36:37.124006+0000","last_became_peered":"2024-02-23T00:36:37.124006+0000","last_unstale":"2024-02-23T00:38:59.186720+0000","last_undegraded":"2024-02-23T00:38:59.186720+0000","last_fullsized":"2024-02-23T00:36:37.103625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":48,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.230405+0000","last_change":"2024-02-23T00:38:49.208012+0000","last_active":"2024-02-23T00:38:59.230405+0000","last_peered":"2024-02-23T00:38:59.230405+0000","last_clean":"2024-02-23T00:38:59.230405+0000","last_became_active":"2024-02-23T00:38:49.207582+0000","last_became_peered":"2024-02-23T00:38:49.207582+0000","last_unstale":"2024-02-23T00:38:59.230405+0000","last_undegraded":"2024-02-23T00:38:59.230405+0000","last_fullsized":"2024-02-23T00:38:59.230405+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:39.957272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.957272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":445,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.230490+0000","last_change":"2024-02-23T00:37:18.044814+0000","last_active":"2024-02-23T00:38:59.230490+0000","last_peered":"2024-02-23T00:38:59.230490+0000","last_clean":"2024-02-23T00:38:59.230490+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:38:59.230490+0000","last_undegraded":"2024-02-23T00:38:59.230490+0000","last_fullsized":"2024-02-23T00:38:59.230490+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:18.044737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:18.044737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":168,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:59.211271+0000","last_change":"2024-02-23T00:38:50.200938+0000","last_active":"2024-02-23T00:38:59.211271+0000","last_peered":"2024-02-23T00:38:59.211271+0000","last_clean":"2024-02-23T00:38:47.152561+0000","last_became_active":"2024-02-23T00:38:50.180708+0000","last_became_peered":"2024-02-23T00:38:50.180708+0000","last_unstale":"2024-02-23T00:38:59.211271+0000","last_undegraded":"2024-02-23T00:38:59.211271+0000","last_fullsized":"2024-02-23T00:38:59.211271+0000","mapping_epoch":512,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":498,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.234957+0000","last_change":"2024-02-23T00:36:37.114846+0000","last_active":"2024-02-23T00:38:59.234957+0000","last_peered":"2024-02-23T00:38:59.234957+0000","last_clean":"2024-02-23T00:36:33.367758+0000","last_became_active":"2024-02-23T00:36:37.114846+0000","last_became_peered":"2024-02-23T00:36:37.114846+0000","last_unstale":"2024-02-23T00:38:59.234957+0000","last_undegraded":"2024-02-23T00:38:59.234957+0000","last_fullsized":"2024-02-23T00:36:37.107709+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":493,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.232011+0000","last_change":"2024-02-23T00:37:19.034254+0000","last_active":"2024-02-23T00:38:59.232011+0000","last_peered":"2024-02-23T00:38:59.232011+0000","last_clean":"2024-02-23T00:38:59.232011+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:38:59.232011+0000","last_undegraded":"2024-02-23T00:38:59.232011+0000","last_fullsized":"2024-02-23T00:38:59.232011+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:19.034136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:37:19.034136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":35,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.230463+0000","last_change":"2024-02-23T00:38:43.922541+0000","last_active":"2024-02-23T00:38:59.230463+0000","last_peered":"2024-02-23T00:38:59.230463+0000","last_clean":"2024-02-23T00:38:59.230463+0000","last_became_active":"2024-02-23T00:38:40.180445+0000","last_became_peered":"2024-02-23T00:38:40.180445+0000","last_unstale":"2024-02-23T00:38:59.230463+0000","last_undegraded":"2024-02-23T00:38:59.230463+0000","last_fullsized":"2024-02-23T00:38:59.230463+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:43.922442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:38:43.922442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,1],"avail_no_missing":["7","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":456,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.181291+0000","last_change":"2024-02-23T00:38:49.225064+0000","last_active":"2024-02-23T00:38:59.181291+0000","last_peered":"2024-02-23T00:38:59.181291+0000","last_clean":"2024-02-23T00:38:59.181291+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:38:59.181291+0000","last_undegraded":"2024-02-23T00:38:59.181291+0000","last_fullsized":"2024-02-23T00:38:59.181291+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:33.192483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:33:33.192483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":619,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:59.185242+0000","last_change":"2024-02-23T00:38:52.224367+0000","last_active":"2024-02-23T00:38:59.185242+0000","last_peered":"2024-02-23T00:38:59.185242+0000","last_clean":"2024-02-23T00:38:19.215057+0000","last_became_active":"2024-02-23T00:38:22.178598+0000","last_became_peered":"2024-02-23T00:38:22.178598+0000","last_unstale":"2024-02-23T00:38:59.185242+0000","last_undegraded":"2024-02-23T00:38:59.185242+0000","last_fullsized":"2024-02-23T00:38:59.185242+0000","mapping_epoch":484,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[5,6,7],"avail_no_missing":["5","6","7"],"object_location_counts":[{"shards":"5,6,7","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":20,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.199287+0000","last_change":"2024-02-23T00:38:50.183535+0000","last_active":"2024-02-23T00:38:59.199287+0000","last_peered":"2024-02-23T00:38:59.199287+0000","last_clean":"2024-02-23T00:38:59.199287+0000","last_became_active":"2024-02-23T00:38:50.183147+0000","last_became_peered":"2024-02-23T00:38:50.183147+0000","last_unstale":"2024-02-23T00:38:59.199287+0000","last_undegraded":"2024-02-23T00:38:59.199287+0000","last_fullsized":"2024-02-23T00:38:59.199287+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.388834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,2],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":145,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.224913+0000","last_change":"2024-02-23T00:38:00.381029+0000","last_active":"2024-02-23T00:38:59.224913+0000","last_peered":"2024-02-23T00:38:59.224913+0000","last_clean":"2024-02-23T00:38:59.224913+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:38:59.224913+0000","last_undegraded":"2024-02-23T00:38:59.224913+0000","last_fullsized":"2024-02-23T00:38:59.224913+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.380938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.380938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":32,"reported_epoch":520,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:38:59.185161+0000","last_change":"2024-02-23T00:38:40.203064+0000","last_active":"2024-02-23T00:38:59.185161+0000","last_peered":"2024-02-23T00:38:59.185161+0000","last_clean":"2024-02-23T00:38:59.185161+0000","last_became_active":"2024-02-23T00:38:40.202564+0000","last_became_peered":"2024-02-23T00:38:40.202564+0000","last_unstale":"2024-02-23T00:38:59.185161+0000","last_undegraded":"2024-02-23T00:38:59.185161+0000","last_fullsized":"2024-02-23T00:38:59.185161+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:58.130479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:37:58.130479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":105,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.188876+0000","last_change":"2024-02-23T00:36:37.119114+0000","last_active":"2024-02-23T00:38:59.188876+0000","last_peered":"2024-02-23T00:38:59.188876+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119114+0000","last_became_peered":"2024-02-23T00:36:37.119114+0000","last_unstale":"2024-02-23T00:38:59.188876+0000","last_undegraded":"2024-02-23T00:38:59.188876+0000","last_fullsized":"2024-02-23T00:36:37.106746+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.176441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.176441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":704,"reported_epoch":520,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:38:59.195187+0000","last_change":"2024-02-23T00:38:49.215579+0000","last_active":"2024-02-23T00:38:59.195187+0000","last_peered":"2024-02-23T00:38:59.195187+0000","last_clean":"2024-02-23T00:38:47.148293+0000","last_became_active":"2024-02-23T00:38:49.185201+0000","last_became_peered":"2024-02-23T00:38:49.185201+0000","last_unstale":"2024-02-23T00:38:59.195187+0000","last_undegraded":"2024-02-23T00:38:59.195187+0000","last_fullsized":"2024-02-23T00:38:59.195187+0000","mapping_epoch":511,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:34.287816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.287816+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[2,5,1],"avail_no_missing":["2","1","5"],"object_location_counts":[{"shards":"1,2,5","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":103,"reported_epoch":520,"state":"active+undersized","last_fresh":"2024-02-23T00:38:59.237014+0000","last_change":"2024-02-23T00:36:37.120482+0000","last_active":"2024-02-23T00:38:59.237014+0000","last_peered":"2024-02-23T00:38:59.237014+0000","last_clean":"2024-02-23T00:27:15.989319+0000","last_became_active":"2024-02-23T00:36:37.120482+0000","last_became_peered":"2024-02-23T00:36:37.120482+0000","last_unstale":"2024-02-23T00:38:59.237014+0000","last_undegraded":"2024-02-23T00:38:59.237014+0000","last_fullsized":"2024-02-23T00:36:37.111071+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":490,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.210738+0000","last_change":"2024-02-23T00:37:14.200716+0000","last_active":"2024-02-23T00:38:59.210738+0000","last_peered":"2024-02-23T00:38:59.210738+0000","last_clean":"2024-02-23T00:38:59.210738+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:38:59.210738+0000","last_undegraded":"2024-02-23T00:38:59.210738+0000","last_fullsized":"2024-02-23T00:38:59.210738+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.200606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.200606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":19,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.194403+0000","last_change":"2024-02-23T00:38:49.220042+0000","last_active":"2024-02-23T00:38:59.194403+0000","last_peered":"2024-02-23T00:38:59.194403+0000","last_clean":"2024-02-23T00:38:59.194403+0000","last_became_active":"2024-02-23T00:38:49.219344+0000","last_became_peered":"2024-02-23T00:38:49.219344+0000","last_unstale":"2024-02-23T00:38:59.194403+0000","last_undegraded":"2024-02-23T00:38:59.194403+0000","last_fullsized":"2024-02-23T00:38:59.194403+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:24.990671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:24.990671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":455,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-02-23T00:38:59.182810+0000","last_change":"2024-02-23T00:38:49.228307+0000","last_active":"2024-02-23T00:38:59.182810+0000","last_peered":"2024-02-23T00:38:59.182810+0000","last_clean":"2024-02-23T00:38:59.182810+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:38:59.182810+0000","last_undegraded":"2024-02-23T00:38:59.182810+0000","last_fullsized":"2024-02-23T00:38:59.182810+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:40.485283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:40.485283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":4020422,"num_objects":188,"num_object_clones":9,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":61,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":211,"num_read_kb":57,"num_write":770,"num_write_kb":4612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8000615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14237696,"data_stored":12057629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2041,"ondisk_log_size":2041,"up":84,"acting":85,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":90,"acting":92,"num_store_stats":8},{"poolid":6,"num_pg":30,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":86,"num_store_stats":8},{"poolid":5,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":110,"acting":112,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":111,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":1323,"num_read_kb":1077,"num_write":794,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":3961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":86,"acting":90,"num_store_stats":8},{"poolid":225,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":381,"num_read_kb":381,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":34,"acting":37,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":82,"acting":86,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332240,"num_pgs":117,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311572,"kb_used_data":6156,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433580,"statfs":{"total":95995035648,"available":95675985920,"internally_reserved":0,"allocated":6303744,"data_stored":3473231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.02,"15min":1.024},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.79800000000000004,"5min":57.401000000000003,"15min":61.048999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.1719999999999999,"15min":1.2450000000000001},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":65.558999999999997,"15min":73.870999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.51600000000000001,"15min":0.68999999999999995},"min":{"1min":0.23999999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":0.81699999999999995,"5min":4.6840000000000002,"15min":31.175000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.50600000000000001,"15min":0.68300000000000005},"min":{"1min":0.28599999999999998,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":0.57499999999999996,"5min":3.8719999999999999,"15min":29.423999999999999},"last":0.36099999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.55900000000000005,"15min":0.68000000000000005},"min":{"1min":0.33100000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.0099999999999998,"5min":8.0099999999999998,"15min":16.603999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.55900000000000005,"15min":0.60999999999999999},"min":{"1min":0.318,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66000000000000003,"5min":4.1369999999999996,"15min":7.5339999999999998},"last":0.38700000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.58999999999999997,"15min":0.996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":8.0449999999999999,"5min":8.0449999999999999,"15min":57.729999999999997},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56999999999999995,"15min":0.90400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69099999999999995,"5min":3.8420000000000001,"15min":27.573},"last":0.432}]},{"osd":4,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.77700000000000002,"5min":1.2310000000000001,"15min":1.706},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.48199999999999998,"15min":0.45800000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":8.1359999999999992,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.32600000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.29299999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":1.0209999999999999},"last":0.31},{"interface":"front","average":{"1min":0.434,"5min":0.46899999999999997,"15min":0.45600000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.84099999999999997,"5min":1.1919999999999999,"15min":1.1950000000000001},"last":0.378}]},{"osd":6,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28000000000000003},"max":{"1min":0.86099999999999999,"5min":1.26,"15min":2.1339999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.51800000000000002,"15min":0.48799999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.194,"5min":5.194,"15min":5.194},"last":0.41399999999999998}]}]},{"osd":6,"up_from":44,"seq":188978561173,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315724,"kb_used_data":6212,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93429428,"statfs":{"total":95995035648,"available":95671734272,"internally_reserved":0,"allocated":6361088,"data_stored":3476136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.6499999999999999,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.78800000000000003,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64800000000000002,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":9.468,"15min":9.468},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.49399999999999999,"15min":1.0089999999999999},"min":{"1min":0.33000000000000002,"5min":0.253,"15min":0.245},"max":{"1min":0.67500000000000004,"5min":0.79100000000000004,"15min":59.347999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.49399999999999999,"15min":1.288},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":87.221000000000004},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.503,"15min":0.80900000000000005},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.202,"5min":1.202,"15min":25.550999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.55000000000000004,"15min":0.871},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66500000000000004,"5min":4.3570000000000002,"15min":40.914000000000001},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.54200000000000004,"15min":0.80600000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.23400000000000001},"max":{"1min":0.71799999999999997,"5min":3.2799999999999998,"15min":31.309999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.53900000000000003,"15min":1.325},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.29899999999999999},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":86.311999999999998},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.42099999999999999,"15min":0.42699999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.626,"5min":0.67000000000000004,"15min":3.7839999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.44500000000000001,"15min":0.503},"min":{"1min":0.25800000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57899999999999996,"5min":0.76500000000000001,"15min":17.718},"last":0.59199999999999997}]},{"osd":5,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.441,"15min":0.47699999999999998},"min":{"1min":0.27800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.54600000000000004,"5min":0.71299999999999997,"15min":8.0990000000000002},"last":0.441},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.46899999999999997,"15min":0.47799999999999998},"min":{"1min":0.313,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.60899999999999999,"5min":0.75700000000000001,"15min":8.1370000000000005},"last":0.40899999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:38:30 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.54000000000000004,"15min":0.61399999999999999},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.4279999999999999,"5min":6.4279999999999999,"15min":17.881},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.401,"5min":0.67500000000000004,"15min":0.65200000000000002},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":18.475000000000001,"5min":18.475000000000001,"15min":18.475000000000001},"last":0.58099999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575020,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319828,"kb_used_data":10316,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93425324,"statfs":{"total":95995035648,"available":95667531776,"internally_reserved":0,"allocated":10563584,"data_stored":7484505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.29,"15min":0.66300000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.51700000000000002,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.89600000000000002,"15min":0.55200000000000005},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60499999999999998,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32700000000000001,"15min":0.66600000000000004},"min":{"1min":0.215,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.44800000000000001,"5min":0.84099999999999997,"15min":35.313000000000002},"last":0.309},{"interface":"front","average":{"1min":0.312,"5min":0.33400000000000002,"15min":0.57099999999999995},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20100000000000001},"max":{"1min":0.54100000000000004,"5min":0.61799999999999999,"15min":27.908999999999999},"last":0.25800000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.41399999999999998,"15min":0.65900000000000003},"min":{"1min":0.26500000000000001,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":3.8119999999999998,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.40000000000000002,"15min":0.65000000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.27300000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:38:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.61099999999999999,"15min":0.69499999999999995},"min":{"1min":0.35799999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.71899999999999997,"5min":8.5559999999999992,"15min":16.231000000000002},"last":0.373},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.72099999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.74099999999999999,"5min":1.135,"15min":29.212},"last":0.35999999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:38:44 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52400000000000002,"15min":0.71499999999999997},"min":{"1min":0.34100000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.71099999999999997,"5min":0.90500000000000003,"15min":16.294},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53100000000000003,"15min":0.67600000000000005},"min":{"1min":0.374,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":11.105},"last":0.41899999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:38:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.63100000000000001,"15min":0.76100000000000001},"min":{"1min":0.34899999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70999999999999996,"5min":10.416,"15min":29.187000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.63300000000000001,"15min":0.746},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67200000000000004,"5min":8.2929999999999993,"15min":16.369},"last":0.47899999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55800000000000005,"15min":0.76400000000000001},"min":{"1min":0.36499999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65400000000000003,"5min":0.86099999999999999,"15min":16.395},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55600000000000005,"15min":0.66000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":11.135999999999999},"last":0.38200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318432,"kb_used_data":4824,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426720,"statfs":{"total":95995035648,"available":95668961280,"internally_reserved":0,"allocated":4939776,"data_stored":2599387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.80600000000000005,"15min":0.495},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47799999999999998,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.246},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.165,"15min":0.61899999999999999},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.997,"15min":0.58899999999999997},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.64400000000000002,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.748,"15min":0.48999999999999999},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.22700000000000001},"max":{"1min":0.51100000000000001,"5min":15.965,"15min":15.965},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":1.0820000000000001,"15min":0.65000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.0369999999999999,"15min":0.63700000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.58799999999999997,"5min":16.006,"15min":16.006},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.73399999999999999,"15min":0.63800000000000001},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84699999999999998,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.38300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83699999999999997,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.071,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86799999999999999,"5min":13.311,"15min":13.311},"last":0.499},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.113,"15min":0.74399999999999999},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":23.744,"15min":23.744},"last":0.42299999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.0740000000000001,"15min":0.80600000000000005},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78500000000000003,"5min":13.334,"15min":33.082999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.93400000000000005,"15min":0.70999999999999996},"min":{"1min":0.41199999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81699999999999995,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.49199999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.84099999999999997,"15min":0.78500000000000003},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.871,"5min":15.943,"15min":31.251999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.091,"15min":0.76100000000000001},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.80000000000000004,"5min":23.785,"15min":23.785},"last":0.47599999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411496,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311428,"kb_used_data":10108,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93433724,"statfs":{"total":95995035648,"available":95676133376,"internally_reserved":0,"allocated":10350592,"data_stored":7478271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":1.0600000000000001,"15min":0.754},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.65500000000000003,"5min":56.509,"15min":56.509},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.59799999999999998,"15min":0.71199999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56799999999999995,"5min":19.571999999999999,"15min":31.498999999999999},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:38:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.45100000000000001,"15min":0.59999999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.8919999999999999,"5min":7.7110000000000003,"15min":19.324999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.51000000000000001,"15min":0.75800000000000001},"min":{"1min":0.221,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":7.9779999999999998,"5min":7.9779999999999998,"15min":35.052},"last":0.27100000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:38:05 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.55700000000000005,"15min":0.73899999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":10.032,"5min":10.032,"15min":43.5},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.66900000000000004,"15min":0.69099999999999995},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":11.815,"5min":11.815,"15min":27.414000000000001},"last":0.31}]},{"osd":4,"last update":"Fri Feb 23 00:38:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.72499999999999998,"15min":0.67600000000000005},"min":{"1min":0.32700000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0899999999999999,"5min":10.106,"15min":14.614000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.67900000000000005,"15min":0.627},"min":{"1min":0.30199999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.9750000000000001,"5min":7.766,"15min":7.766},"last":0.54400000000000004}]},{"osd":5,"last update":"Fri Feb 23 00:37:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57899999999999996,"15min":0.66800000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":2.004,"5min":3.2879999999999998,"15min":17.765000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.68300000000000005,"15min":0.67300000000000004},"min":{"1min":0.34499999999999997,"5min":0.32900000000000001,"15min":0.317},"max":{"1min":10.138,"5min":10.138,"15min":14.589},"last":0.57499999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:38:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.67000000000000004,"15min":0.628},"min":{"1min":0.36299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.2250000000000001,"5min":7.7869999999999999,"15min":7.7869999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.62,"15min":0.751},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.34899999999999998},"max":{"1min":1.927,"5min":3.3410000000000002,"15min":20.574999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:38:29 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":0.73299999999999998,"15min":1.077},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":7.9480000000000004,"5min":7.9480000000000004,"15min":66.632000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.73599999999999999,"15min":0.85199999999999998},"min":{"1min":0.36399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.1789999999999998,"5min":10.163,"15min":19.603999999999999},"last":0.51600000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307228,"kb_used_data":10004,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93437924,"statfs":{"total":95995035648,"available":95680434176,"internally_reserved":0,"allocated":10244096,"data_stored":7470788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":1.506,"15min":0.745},"min":{"1min":0.252,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.76100000000000001,"5min":55.238999999999997,"15min":55.238999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":1.4750000000000001,"15min":0.747},"min":{"1min":0.26100000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.47699999999999998,"5min":54.563000000000002,"15min":54.563000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.82999999999999996,"15min":0.61799999999999999},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.17899999999999999},"max":{"1min":2.718,"5min":54.351999999999997,"15min":54.351999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.85799999999999998,"15min":0.76200000000000001},"min":{"1min":0.23499999999999999,"5min":0.219,"15min":0.219},"max":{"1min":1.3129999999999999,"5min":55.337000000000003,"15min":55.337000000000003},"last":0.314}]},{"osd":2,"last update":"Fri Feb 23 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.85499999999999998,"15min":0.75},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.159,"5min":55.368000000000002,"15min":55.368000000000002},"last":0.374},{"interface":"front","average":{"1min":0.626,"5min":0.88,"15min":0.73499999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.185},"max":{"1min":4.181,"5min":54.383000000000003,"15min":54.383000000000003},"last":0.38600000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":1.0920000000000001,"15min":0.91300000000000003},"min":{"1min":0.34799999999999998,"5min":0.316,"15min":0.27900000000000003},"max":{"1min":0.79100000000000004,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54200000000000004,"15min":0.61799999999999999},"min":{"1min":0.33900000000000002,"5min":0.32600000000000001,"15min":0.24099999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":8.0359999999999996},"last":0.44800000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:38:41 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":1.0089999999999999,"15min":0.877},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.28899999999999998},"max":{"1min":0.83899999999999997,"5min":55.401000000000003,"15min":55.401000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":1.006,"15min":0.83199999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.30299999999999999},"max":{"1min":0.72899999999999998,"5min":54.454999999999998,"15min":54.454999999999998},"last":0.48599999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:38:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.0369999999999999,"15min":0.78700000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.68700000000000006,"5min":54.305,"15min":54.305},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":1.052,"15min":0.94699999999999995},"min":{"1min":0.39800000000000002,"5min":0.373,"15min":0.36299999999999999},"max":{"1min":0.72899999999999998,"5min":55.424999999999997,"15min":55.424999999999997},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Feb 23 00:38:41 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.0720000000000001,"15min":0.77600000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34300000000000003},"max":{"1min":4.9790000000000001,"5min":54.222000000000001,"15min":54.222000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.119,"15min":0.93799999999999994},"min":{"1min":0.40400000000000003,"5min":0.377,"15min":0.36499999999999999},"max":{"1min":6.7279999999999998,"5min":55.460000000000001,"15min":55.460000000000001},"last":0.50900000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986335,"num_pgs":117,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319888,"kb_used_data":6280,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93425264,"statfs":{"total":95995035648,"available":95667470336,"internally_reserved":0,"allocated":6430720,"data_stored":3468971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1739999999999999,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.68799999999999994,"5min":21.419,"15min":21.419},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":2.1200000000000001,"15min":1.2589999999999999},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:38:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.75900000000000001,"15min":1.069},"min":{"1min":0.29199999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.1199999999999992,"5min":16.696000000000002,"15min":41.137999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.69199999999999995,"15min":0.88400000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.89100000000000001,"5min":13.455,"15min":26.210000000000001},"last":0.40400000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:38:11 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.81399999999999995,"15min":1.1519999999999999},"min":{"1min":0.30399999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.9859999999999998,"5min":13.478,"15min":67.253},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.67800000000000005,"15min":1.0169999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.65300000000000002,"5min":8.0199999999999996,"15min":41.927999999999997},"last":0.443}]},{"osd":3,"last update":"Fri Feb 23 00:38:11 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.65800000000000003,"15min":1.0669999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34000000000000002},"max":{"1min":0.67300000000000004,"5min":7.774,"15min":36.301000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.78500000000000003,"15min":1.1819999999999999},"min":{"1min":0.33100000000000002,"5min":0.307,"15min":0.24299999999999999},"max":{"1min":3.9489999999999998,"5min":16.657,"15min":42.064},"last":0.38200000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:38:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44700000000000001,"15min":0.55900000000000005},"min":{"1min":0.22500000000000001,"5min":0.219,"15min":0.217},"max":{"1min":0.73199999999999998,"5min":2.3210000000000002,"15min":14.242000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.56200000000000006,"15min":0.59799999999999998},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.71199999999999997,"5min":7.1459999999999999,"15min":14.124000000000001},"last":0.47299999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:38:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.64800000000000002,"15min":0.56799999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":0.59499999999999997,"5min":8.0730000000000004,"15min":8.0730000000000004},"last":0.495},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.52500000000000002,"15min":0.63500000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.23100000000000001},"max":{"1min":0.63900000000000001,"5min":4.71,"15min":45.814999999999998},"last":0.434}]},{"osd":7,"last update":"Fri Feb 23 00:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.54200000000000004,"15min":0.61799999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70699999999999996,"5min":5.1280000000000001,"15min":17.178999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.69499999999999995,"15min":0.69599999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.28299999999999997},"max":{"1min":0.748,"5min":15.85,"15min":15.85},"last":0.51500000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757402,"num_pgs":110,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311728,"kb_used_data":6312,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433424,"statfs":{"total":95995035648,"available":95675826176,"internally_reserved":0,"allocated":6463488,"data_stored":3470282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.1200000000000001,"15min":0.70699999999999996},"min":{"1min":0.36699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.115,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":1.028},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.498,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.504,"15min":0.75800000000000001},"min":{"1min":0.27500000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67800000000000005,"5min":3.7629999999999999,"15min":38.838000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.55800000000000005,"15min":1.1919999999999999},"min":{"1min":0.33700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.748,"5min":3.7250000000000001,"15min":127.794},"last":0.38700000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.66800000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":11.435,"5min":11.435,"15min":37.442999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.53800000000000003,"15min":0.70399999999999996},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.76800000000000002,"5min":3.734,"15min":23.297999999999998},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.745,"15min":0.82099999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":11.457000000000001,"5min":12.507,"15min":24.414999999999999},"last":1.052},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.59199999999999997,"15min":0.91300000000000003},"min":{"1min":0.34599999999999997,"5min":0.33100000000000002,"15min":0.31},"max":{"1min":0.64600000000000002,"5min":3.8119999999999998,"15min":40.865000000000002},"last":0.434}]},{"osd":4,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40999999999999998,"15min":0.78200000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":0.58199999999999996,"5min":2.8940000000000001,"15min":103.13},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.56899999999999995,"15min":0.51100000000000001},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.381,"5min":11.381,"15min":12.795999999999999},"last":0.46300000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:38:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52200000000000002,"15min":0.50600000000000001},"min":{"1min":0.29499999999999998,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":1.6140000000000001,"5min":2.9199999999999999,"15min":7.1440000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47999999999999998,"15min":0.82399999999999995},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":103.61499999999999},"last":0.33400000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:38:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53300000000000003,"15min":0.53200000000000003},"min":{"1min":0.29999999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81699999999999995,"5min":3.9039999999999999,"15min":7.1820000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.56200000000000006,"15min":0.53000000000000003},"min":{"1min":0.28199999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.79400000000000004,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.40000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4407296,"data_stored":4012344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4268032,"data_stored":4009004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4132864,"data_stored":4000318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":8696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":9234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":5732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":11748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:39:04.056 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.log pgp_num 32 2024-02-23T00:39:04.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:39:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:39:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:03.716711+0000 mgr.y (mgr.24437) 490 : cluster [DBG] pgmap v803: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 103/1257 objects degraded (8.194%); 114/1257 objects misplaced (9.069%) 2024-02-23T00:39:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:04.152263+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e525: 8 total, 7 up, 8 in 2024-02-23T00:39:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:05.039335+0000 mon.a (mon.0) 2901 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:03.716711+0000 mgr.y (mgr.24437) 490 : cluster [DBG] pgmap v803: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 103/1257 objects degraded (8.194%); 114/1257 objects misplaced (9.069%) 2024-02-23T00:39:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:04.152263+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e525: 8 total, 7 up, 8 in 2024-02-23T00:39:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:05.039335+0000 mon.a (mon.0) 2901 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:05.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:03.716711+0000 mgr.y (mgr.24437) 490 : cluster [DBG] pgmap v803: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 103/1257 objects degraded (8.194%); 114/1257 objects misplaced (9.069%) 2024-02-23T00:39:05.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:04.152263+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e525: 8 total, 7 up, 8 in 2024-02-23T00:39:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:05.039335+0000 mon.a (mon.0) 2901 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:05.159324+0000 mon.a (mon.0) 2902 : cluster [DBG] osdmap e526: 8 total, 7 up, 8 in 2024-02-23T00:39:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:05.212722+0000 mon.b (mon.1) 426 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:05.213008+0000 mon.a (mon.0) 2903 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:05.213706+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-86"}]: dispatch 2024-02-23T00:39:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:05.213785+0000 mon.a (mon.0) 2904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-86"}]: dispatch 2024-02-23T00:39:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:05.159324+0000 mon.a (mon.0) 2902 : cluster [DBG] osdmap e526: 8 total, 7 up, 8 in 2024-02-23T00:39:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:05.212722+0000 mon.b (mon.1) 426 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:05.213008+0000 mon.a (mon.0) 2903 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:05.213706+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-86"}]: dispatch 2024-02-23T00:39:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:05.213785+0000 mon.a (mon.0) 2904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-86"}]: dispatch 2024-02-23T00:39:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:05.159324+0000 mon.a (mon.0) 2902 : cluster [DBG] osdmap e526: 8 total, 7 up, 8 in 2024-02-23T00:39:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:05.212722+0000 mon.b (mon.1) 426 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:05.213008+0000 mon.a (mon.0) 2903 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:05.213706+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-86"}]: dispatch 2024-02-23T00:39:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:05.213785+0000 mon.a (mon.0) 2904 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-86"}]: dispatch 2024-02-23T00:39:06.771 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:06.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:06.782 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-23T00:39:06.876 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:06.907 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:06.926 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:06.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:07.030 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:07.057 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:07.075 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:07.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:07.179 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:07.209 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:07.308 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:07.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:07.412 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:07.437 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:05.717884+0000 mgr.y (mgr.24437) 491 : cluster [DBG] pgmap v806: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 25 active+undersized+degraded, 37 active+undersized, 156 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 9 op/s; 103/1266 objects degraded (8.136%); 114/1266 objects misplaced (9.005%) 2024-02-23T00:39:07.437 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:06.165237+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e527: 8 total, 7 up, 8 in 2024-02-23T00:39:07.437 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:07.140539+0000 mon.a (mon.0) 2906 : cluster [WRN] Health check update: Degraded data redundancy: 103/1266 objects degraded (8.136%), 26 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:07.438 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:07.457 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:07.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:07.560 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:07.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:05.717884+0000 mgr.y (mgr.24437) 491 : cluster [DBG] pgmap v806: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 25 active+undersized+degraded, 37 active+undersized, 156 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 9 op/s; 103/1266 objects degraded (8.136%); 114/1266 objects misplaced (9.005%) 2024-02-23T00:39:07.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:06.165237+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e527: 8 total, 7 up, 8 in 2024-02-23T00:39:07.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:07.140539+0000 mon.a (mon.0) 2906 : cluster [WRN] Health check update: Degraded data redundancy: 103/1266 objects degraded (8.136%), 26 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:05.717884+0000 mgr.y (mgr.24437) 491 : cluster [DBG] pgmap v806: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 25 active+undersized+degraded, 37 active+undersized, 156 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 9 op/s; 103/1266 objects degraded (8.136%); 114/1266 objects misplaced (9.005%) 2024-02-23T00:39:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:06.165237+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e527: 8 total, 7 up, 8 in 2024-02-23T00:39:07.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:07.140539+0000 mon.a (mon.0) 2906 : cluster [WRN] Health check update: Degraded data redundancy: 103/1266 objects degraded (8.136%), 26 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:07.587 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:07.605 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:07.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:07.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.607+0000 7fcfc5afd700 1 -- 172.21.15.87:0/3067343824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcfc00fd510 msgr2=0x7fcfc00f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:07.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.607+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/3067343824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcfc00fd510 0x7fcfc00f6d40 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 rx=0x7fcfb4004870 tx=0x7fcfb4009cb0).stop 2024-02-23T00:39:07.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.607+0000 7fcfc5afd700 1 -- 172.21.15.87:0/3067343824 shutdown_connections 2024-02-23T00:39:07.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.607+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/3067343824 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfc00f7400 0x7fcfc0103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:07.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.607+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/3067343824 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcfc00fcbb0 0x7fcfc00fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:07.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.607+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/3067343824 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcfc00fd510 0x7fcfc00f6d40 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:07.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.607+0000 7fcfc5afd700 1 -- 172.21.15.87:0/3067343824 >> 172.21.15.87:0/3067343824 conn(0x7fcfc00f2820 msgr2=0x7fcfc00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:07.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.607+0000 7fcfc5afd700 1 -- 172.21.15.87:0/3067343824 shutdown_connections 2024-02-23T00:39:07.610 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.607+0000 7fcfc5afd700 1 -- 172.21.15.87:0/3067343824 wait complete. 2024-02-23T00:39:07.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 librados: starting msgr at 2024-02-23T00:39:07.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 librados: starting objecter 2024-02-23T00:39:07.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 Processor -- start 2024-02-23T00:39:07.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 -- start start 2024-02-23T00:39:07.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 librados: setting wanted keys 2024-02-23T00:39:07.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 librados: calling monclient init 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcfc00f7400 0x7fcfc010e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfc00fcbb0 0x7fcfc010e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcfc00fd510 0x7fcfc0108360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc0108850 con 0x7fcfc00fd510 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc01089c0 con 0x7fcfc00fcbb0 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfc5afd700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc0108b80 con 0x7fcfc00f7400 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.608+0000 7fcfbf7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcfc00f7400 0x7fcfc010e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbf7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcfc00f7400 0x7fcfc010e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:47746/0 (socket says 172.21.15.87:47746) 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbf7fe700 1 -- 172.21.15.87:0/375144039 learned_addr learned my addr 172.21.15.87:0/375144039 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbeffd700 1 --2- 172.21.15.87:0/375144039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfc00fcbb0 0x7fcfc010e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbf7fe700 1 -- 172.21.15.87:0/375144039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfc00fcbb0 msgr2=0x7fcfc010e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbf7fe700 1 --2- 172.21.15.87:0/375144039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcfc00fcbb0 0x7fcfc010e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbf7fe700 1 -- 172.21.15.87:0/375144039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcfc00fd510 msgr2=0x7fcfc0108360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:07.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbf7fe700 1 --2- 172.21.15.87:0/375144039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcfc00fd510 0x7fcfc0108360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbf7fe700 1 -- 172.21.15.87:0/375144039 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb4005040 con 0x7fcfc00f7400 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbf7fe700 1 --2- 172.21.15.87:0/375144039 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcfc00f7400 0x7fcfc010e1c0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 rx=0x7fcfb000e970 tx=0x7fcfb000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbcff9700 10 client.?.objecter ms_handle_connect 0x7fcfc00f7400 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbcff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0004660 con 0x7fcfc00f7400 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc0108e60 con 0x7fcfc00f7400 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfc5afd700 10 client.37940.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc01091e0 con 0x7fcfc00f7400 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfc5afd700 1 librados: init done 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfb000f040 con 0x7fcfc00f7400 2024-02-23T00:39:07.614 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.609+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb000c840 con 0x7fcfc00f7400 2024-02-23T00:39:07.616 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.610+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fcfac004f30 con 0x7fcfc00f7400 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.611+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fcfb0009630 con 0x7fcfc00f7400 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.611+0000 7fcfbcff9700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcfa8062c90 0x7fcfa8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.611+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(528..528 src has 173..528) v4 ==== 9057+0+0 (secure 0 0 0) 0x7fcfb007d6b0 con 0x7fcfc00f7400 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.611+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 osd_map(528..528 src has 173..528) v4 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.611+0000 7fcfbcff9700 3 client.37940.objecter handle_osd_map got epochs [528,528] > 0 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.611+0000 7fcfbcff9700 3 client.37940.objecter handle_osd_map decoding full epoch 528 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.611+0000 7fcfbeffd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcfa8062c90 0x7fcfa8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfbcff9700 20 client.37940.objecter dump_active .. 0 homeless 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v528) v1 ==== 54+0+16 (secure 0 0 0) 0x7fcfb007db90 con 0x7fcfc00f7400 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfbcff9700 10 client.37940.objecter ms_handle_connect 0x7fcfa8062c90 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfbeffd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcfa8062c90 0x7fcfa8065150 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 rx=0x7fcfb4004ee0 tx=0x7fcfb400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:07.617 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 1 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 20 client.37940.objecter _get_session osd=-1 returning homeless 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 10 client.37940.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=529}) v3 -- 0x7fcfac0051d0 con 0x7fcfc00f7400 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcfac000e50 con 0x7fcfc00f7400 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=528) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfb00508c0 con 0x7fcfc00f7400 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfbdffb700 10 client.37940.objecter _check_command_map_dne tid 1 current 528 map_dne_bound 528 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfbdffb700 10 client.37940.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfbdffb700 15 client.37940.objecter _session_command_op_remove -1 1 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 2 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6805/685354171 conn(0x7fcfac002740 0x7fcfac002bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac002140 osd=1 3 2024-02-23T00:39:07.618 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac002140 osd=1 3 2024-02-23T00:39:07.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac002140 osd=1 3 2024-02-23T00:39:07.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 2 2024-02-23T00:39:07.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac002140 osd=1 3 2024-02-23T00:39:07.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 1 2 2024-02-23T00:39:07.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac002140 osd=1 4 2024-02-23T00:39:07.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 10 client.37940.objecter _send_command 2 2024-02-23T00:39:07.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfac0022b0 con 0x7fcfac002740 2024-02-23T00:39:07.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.612+0000 7fcfbffff700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6805/685354171 conn(0x7fcfac002740 0x7fcfac002bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.613+0000 7fcfbcff9700 10 client.37940.objecter ms_handle_connect 0x7fcfac002740 2024-02-23T00:39:07.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.613+0000 7fcfbffff700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6805/685354171 conn(0x7fcfac002740 0x7fcfac002bb0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:07.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.613+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcfac0022b0 con 0x7fcfac002740 2024-02-23T00:39:07.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.613+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:39:07.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.613+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 2 = system:0 2024-02-23T00:39:07.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.613+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac002140 osd=1 4 2024-02-23T00:39:07.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.613+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 1 2 2024-02-23T00:39:07.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:07.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 3 2024-02-23T00:39:07.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac002140 osd=1 3 2024-02-23T00:39:07.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac002140 osd=1 3 2024-02-23T00:39:07.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac002140 osd=1 3 2024-02-23T00:39:07.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 3 2024-02-23T00:39:07.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac002140 osd=1 3 2024-02-23T00:39:07.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 1 3 2024-02-23T00:39:07.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac002140 osd=1 4 2024-02-23T00:39:07.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 10 client.37940.objecter _send_command 3 2024-02-23T00:39:07.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.630+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcfac003870 con 0x7fcfac002740 2024-02-23T00:39:07.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcfac003870 con 0x7fcfac002740 2024-02-23T00:39:07.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:07.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac002140 osd=1 4 2024-02-23T00:39:07.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 1 3 2024-02-23T00:39:07.634 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:39:07.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:07.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 4 2024-02-23T00:39:07.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcfac004380 0x7fcfac00e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac004210 osd=2 3 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac004210 osd=2 3 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac004210 osd=2 3 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 4 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac004210 osd=2 3 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 2 4 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac004210 osd=2 4 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 10 client.37940.objecter _send_command 4 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfac010fa0 con 0x7fcfac004380 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfbf7fe700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcfac004380 0x7fcfac00e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfbf7fe700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcfac004380 0x7fcfac00e350 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.632+0000 7fcfbcff9700 10 client.37940.objecter ms_handle_connect 0x7fcfac004380 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.633+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcfac010fa0 con 0x7fcfac004380 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.633+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:39:07.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.633+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 4 = system:0 2024-02-23T00:39:07.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.633+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac004210 osd=2 4 2024-02-23T00:39:07.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.633+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 2 4 2024-02-23T00:39:07.649 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 5 2024-02-23T00:39:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac004210 osd=2 3 2024-02-23T00:39:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac004210 osd=2 3 2024-02-23T00:39:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac004210 osd=2 3 2024-02-23T00:39:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 5 2024-02-23T00:39:07.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac004210 osd=2 3 2024-02-23T00:39:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 2 5 2024-02-23T00:39:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac004210 osd=2 4 2024-02-23T00:39:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 10 client.37940.objecter _send_command 5 2024-02-23T00:39:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.649+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcfac003340 con 0x7fcfac004380 2024-02-23T00:39:07.652 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.650+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcfac003340 con 0x7fcfac004380 2024-02-23T00:39:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.650+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.650+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.650+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac004210 osd=2 4 2024-02-23T00:39:07.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.650+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 2 5 2024-02-23T00:39:07.653 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:39:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 6 2024-02-23T00:39:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6813/128059790 conn(0x7fcfac00f9c0 0x7fcfac00fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac001790 osd=3 3 2024-02-23T00:39:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac001790 osd=3 3 2024-02-23T00:39:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac001790 osd=3 3 2024-02-23T00:39:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 6 2024-02-23T00:39:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac001790 osd=3 3 2024-02-23T00:39:07.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 3 6 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac001790 osd=3 4 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 10 client.37940.objecter _send_command 6 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfac0102d0 con 0x7fcfac00f9c0 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfbeffd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6813/128059790 conn(0x7fcfac00f9c0 0x7fcfac00fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfbcff9700 10 client.37940.objecter ms_handle_connect 0x7fcfac00f9c0 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.651+0000 7fcfbeffd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.87:6813/128059790 conn(0x7fcfac00f9c0 0x7fcfac00fde0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.652+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcfac0102d0 con 0x7fcfac00f9c0 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.652+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.652+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 6 = system:0 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.652+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac001790 osd=3 4 2024-02-23T00:39:07.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.652+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 3 6 2024-02-23T00:39:07.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 7 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac001790 osd=3 3 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac001790 osd=3 3 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac001790 osd=3 3 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 7 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac001790 osd=3 3 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 3 7 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac001790 osd=3 4 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 10 client.37940.objecter _send_command 7 2024-02-23T00:39:07.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.675+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcfac010cc0 con 0x7fcfac00f9c0 2024-02-23T00:39:07.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcfac010cc0 con 0x7fcfac00f9c0 2024-02-23T00:39:07.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:07.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac001790 osd=3 4 2024-02-23T00:39:07.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 3 7 2024-02-23T00:39:07.680 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 8 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcfac013680 0x7fcfac015a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac001440 osd=4 3 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac001440 osd=4 3 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac001440 osd=4 3 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 8 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac001440 osd=4 3 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 4 8 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac001440 osd=4 4 2024-02-23T00:39:07.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 10 client.37940.objecter _send_command 8 2024-02-23T00:39:07.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.677+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfac0186c0 con 0x7fcfac013680 2024-02-23T00:39:07.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.678+0000 7fcfbffff700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcfac013680 0x7fcfac015a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.679+0000 7fcfbcff9700 10 client.37940.objecter ms_handle_connect 0x7fcfac013680 2024-02-23T00:39:07.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.679+0000 7fcfbffff700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcfac013680 0x7fcfac015a70 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:07.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.680+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcfac0186c0 con 0x7fcfac013680 2024-02-23T00:39:07.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.680+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:39:07.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.680+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 8 = system:0 2024-02-23T00:39:07.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.680+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac001440 osd=4 4 2024-02-23T00:39:07.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.680+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 4 8 2024-02-23T00:39:07.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:07.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 9 2024-02-23T00:39:07.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac001440 osd=4 3 2024-02-23T00:39:07.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac001440 osd=4 3 2024-02-23T00:39:07.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac001440 osd=4 3 2024-02-23T00:39:07.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 9 2024-02-23T00:39:07.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac001440 osd=4 3 2024-02-23T00:39:07.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 4 9 2024-02-23T00:39:07.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac001440 osd=4 4 2024-02-23T00:39:07.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 10 client.37940.objecter _send_command 9 2024-02-23T00:39:07.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.694+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcfac010730 con 0x7fcfac013680 2024-02-23T00:39:07.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcfac010730 con 0x7fcfac013680 2024-02-23T00:39:07.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:07.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac001440 osd=4 4 2024-02-23T00:39:07.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 4 9 2024-02-23T00:39:07.699 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 10 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcfac0108d0 0x7fcfac0170e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac010730 osd=5 3 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac010730 osd=5 3 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac010730 osd=5 3 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 10 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac010730 osd=5 3 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 5 10 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac010730 osd=5 4 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 10 client.37940.objecter _send_command 10 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.696+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfac00f710 con 0x7fcfac0108d0 2024-02-23T00:39:07.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.697+0000 7fcfbf7fe700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcfac0108d0 0x7fcfac0170e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.697+0000 7fcfbcff9700 10 client.37940.objecter ms_handle_connect 0x7fcfac0108d0 2024-02-23T00:39:07.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.697+0000 7fcfbf7fe700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcfac0108d0 0x7fcfac0170e0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:07.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.698+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcfac00f710 con 0x7fcfac0108d0 2024-02-23T00:39:07.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.698+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:39:07.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.698+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 10 = system:0 2024-02-23T00:39:07.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.698+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac010730 osd=5 4 2024-02-23T00:39:07.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.698+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 5 10 2024-02-23T00:39:07.709 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:07.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:07.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 11 2024-02-23T00:39:07.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac010730 osd=5 3 2024-02-23T00:39:07.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac010730 osd=5 3 2024-02-23T00:39:07.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac010730 osd=5 3 2024-02-23T00:39:07.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 11 2024-02-23T00:39:07.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac010730 osd=5 3 2024-02-23T00:39:07.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 5 11 2024-02-23T00:39:07.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac010730 osd=5 4 2024-02-23T00:39:07.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 10 client.37940.objecter _send_command 11 2024-02-23T00:39:07.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.712+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcfac00f710 con 0x7fcfac0108d0 2024-02-23T00:39:07.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcfac00f710 con 0x7fcfac0108d0 2024-02-23T00:39:07.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:07.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac010730 osd=5 4 2024-02-23T00:39:07.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 5 11 2024-02-23T00:39:07.716 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:39:07.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:07.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 12 2024-02-23T00:39:07.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcfac017ae0 0x7fcfac017f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac00f710 osd=6 3 2024-02-23T00:39:07.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac00f710 osd=6 3 2024-02-23T00:39:07.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac00f710 osd=6 3 2024-02-23T00:39:07.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 12 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac00f710 osd=6 3 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 6 12 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac00f710 osd=6 4 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 10 client.37940.objecter _send_command 12 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfac01efb0 con 0x7fcfac017ae0 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfbeffd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcfac017ae0 0x7fcfac017f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfbcff9700 10 client.37940.objecter ms_handle_connect 0x7fcfac017ae0 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.714+0000 7fcfbeffd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcfac017ae0 0x7fcfac017f40 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.715+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcfac01efb0 con 0x7fcfac017ae0 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.715+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:39:07.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.715+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 12 = system:0 2024-02-23T00:39:07.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.715+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac00f710 osd=6 4 2024-02-23T00:39:07.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.715+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 6 12 2024-02-23T00:39:07.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:07.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 13 2024-02-23T00:39:07.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac00f710 osd=6 3 2024-02-23T00:39:07.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac00f710 osd=6 3 2024-02-23T00:39:07.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac00f710 osd=6 3 2024-02-23T00:39:07.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 13 2024-02-23T00:39:07.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac00f710 osd=6 3 2024-02-23T00:39:07.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 6 13 2024-02-23T00:39:07.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac00f710 osd=6 4 2024-02-23T00:39:07.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 10 client.37940.objecter _send_command 13 2024-02-23T00:39:07.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.729+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcfac01c850 con 0x7fcfac017ae0 2024-02-23T00:39:07.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcfac01c850 con 0x7fcfac017ae0 2024-02-23T00:39:07.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:07.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac00f710 osd=6 4 2024-02-23T00:39:07.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 6 13 2024-02-23T00:39:07.734 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:39:07.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:07.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 14 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6812/373787000 conn(0x7fcfac01ced0 0x7fcfac01f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac01c850 osd=7 3 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac01c850 osd=7 3 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac01c850 osd=7 3 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 14 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac01c850 osd=7 3 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 7 14 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac01c850 osd=7 4 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 10 client.37940.objecter _send_command 14 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.731+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfac0220d0 con 0x7fcfac01ced0 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.732+0000 7fcfbffff700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6812/373787000 conn(0x7fcfac01ced0 0x7fcfac01f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.732+0000 7fcfbcff9700 10 client.37940.objecter ms_handle_connect 0x7fcfac01ced0 2024-02-23T00:39:07.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.732+0000 7fcfbffff700 1 --2- 172.21.15.87:0/375144039 >> v2:172.21.15.171:6812/373787000 conn(0x7fcfac01ced0 0x7fcfac01f300 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:07.736 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:07.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.733+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcfac0220d0 con 0x7fcfac01ced0 2024-02-23T00:39:07.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.733+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:39:07.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.733+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 14 = system:0 2024-02-23T00:39:07.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.733+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac01c850 osd=7 4 2024-02-23T00:39:07.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.733+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 7 14 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 10 client.37940.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign -1 15 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac01c850 osd=7 3 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac01c850 osd=7 3 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 20 client.37940.objecter _get_session s=0x7fcfac01c850 osd=7 3 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_remove -1 15 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 20 client.37940.objecter get_session s=0x7fcfac01c850 osd=7 3 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 15 client.37940.objecter _session_command_op_assign 7 15 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 20 client.37940.objecter put_session s=0x7fcfac01c850 osd=7 4 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 10 client.37940.objecter _send_command 15 2024-02-23T00:39:07.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.747+0000 7fcfc5afd700 1 -- 172.21.15.87:0/375144039 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcfac0220d0 con 0x7fcfac01ced0 2024-02-23T00:39:07.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.748+0000 7fcfbcff9700 1 -- 172.21.15.87:0/375144039 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcfac0220d0 con 0x7fcfac01ced0 2024-02-23T00:39:07.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.748+0000 7fcfbcff9700 10 client.37940.objecter ms_dispatch 0x7fcfc0001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:07.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.748+0000 7fcfbcff9700 10 client.37940.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.748+0000 7fcfbcff9700 20 client.37940.objecter put_session s=0x7fcfac01c850 osd=7 4 2024-02-23T00:39:07.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:07.748+0000 7fcfbcff9700 15 client.37940.objecter _session_command_op_remove 7 15 2024-02-23T00:39:07.751 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:39:07.752 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:39:07.752 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.752 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.752 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.752 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.752 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.752 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.752 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:39:07.753 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:07.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:07.857 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:07.883 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:07.903 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:07.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:08.006 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:08.035 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:08.137 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:08.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:08.239 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:08.281 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:08.290 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:07.173294+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e528: 8 total, 7 up, 8 in 2024-02-23T00:39:08.290 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:07.200904+0000 mon.b (mon.1) 428 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:08.290 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:07.204961+0000 mon.a (mon.0) 2908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:08.290 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:07.612287+0000 mon.c (mon.2) 452 : audit [DBG] from='client.? 172.21.15.87:0/375144039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:08.292 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:07.173294+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e528: 8 total, 7 up, 8 in 2024-02-23T00:39:08.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:07.200904+0000 mon.b (mon.1) 428 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:08.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:07.204961+0000 mon.a (mon.0) 2908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:08.293 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:07.612287+0000 mon.c (mon.2) 452 : audit [DBG] from='client.? 172.21.15.87:0/375144039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:07.173294+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e528: 8 total, 7 up, 8 in 2024-02-23T00:39:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:07.200904+0000 mon.b (mon.1) 428 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:07.204961+0000 mon.a (mon.0) 2908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:07.612287+0000 mon.c (mon.2) 452 : audit [DBG] from='client.? 172.21.15.87:0/375144039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:39:08.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:39:09.188 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:07.718530+0000 mgr.y (mgr.24437) 492 : cluster [DBG] pgmap v809: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 103/1266 objects degraded (8.136%); 114/1266 objects misplaced (9.005%) 2024-02-23T00:39:09.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:08.179829+0000 mon.a (mon.0) 2909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:09.189 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:08.179970+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2024-02-23T00:39:09.330 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:39:09.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:08.202776+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:09.505 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:08.203001+0000 mon.a (mon.0) 2911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:09.505 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:07.718530+0000 mgr.y (mgr.24437) 492 : cluster [DBG] pgmap v809: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 103/1266 objects degraded (8.136%); 114/1266 objects misplaced (9.005%) 2024-02-23T00:39:09.506 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:08.179829+0000 mon.a (mon.0) 2909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:09.506 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:08.179970+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2024-02-23T00:39:09.506 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:08.202776+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:09.506 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:08.203001+0000 mon.a (mon.0) 2911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:09.533 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:09.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:09.638 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:07.718530+0000 mgr.y (mgr.24437) 492 : cluster [DBG] pgmap v809: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 103/1266 objects degraded (8.136%); 114/1266 objects misplaced (9.005%) 2024-02-23T00:39:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:08.179829+0000 mon.a (mon.0) 2909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:08.179970+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e529: 8 total, 7 up, 8 in 2024-02-23T00:39:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:08.202776+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:09.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:08.203001+0000 mon.a (mon.0) 2911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:09.179332+0000 mon.a (mon.0) 2912 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:39:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:09.179394+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2024-02-23T00:39:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:09.183476+0000 mon.b (mon.1) 430 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-88", "mode": "writeback"}]: dispatch 2024-02-23T00:39:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:09.194993+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-88", "mode": "writeback"}]: dispatch 2024-02-23T00:39:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:09.179332+0000 mon.a (mon.0) 2912 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:39:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:09.179394+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2024-02-23T00:39:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:09.183476+0000 mon.b (mon.1) 430 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-88", "mode": "writeback"}]: dispatch 2024-02-23T00:39:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:09.194993+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-88", "mode": "writeback"}]: dispatch 2024-02-23T00:39:10.416 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:10.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:09.179332+0000 mon.a (mon.0) 2912 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:39:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:09.179394+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e530: 8 total, 7 up, 8 in 2024-02-23T00:39:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:09.183476+0000 mon.b (mon.1) 430 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-88", "mode": "writeback"}]: dispatch 2024-02-23T00:39:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:09.194993+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-88", "mode": "writeback"}]: dispatch 2024-02-23T00:39:10.521 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:10.549 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:10.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/2995350078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40f7ba0 msgr2=0x7fb2f4103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:10.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/2995350078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40f7ba0 0x7fb2f4103950 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 rx=0x7fb2ec009be0 tx=0x7fb2ec004d10).stop 2024-02-23T00:39:10.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/2995350078 shutdown_connections 2024-02-23T00:39:10.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/2995350078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40f7ba0 0x7fb2f4103950 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:10.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/2995350078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2f40fcb40 0x7fb2f40fcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:10.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/2995350078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2f40fd4a0 0x7fb2f40f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/2995350078 >> 172.21.15.87:0/2995350078 conn(0x7fb2f406c370 msgr2=0x7fb2f406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/2995350078 shutdown_connections 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/2995350078 wait complete. 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 librados: starting msgr at 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.711+0000 7fb2fa3c0700 1 librados: starting objecter 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 Processor -- start 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 -- start start 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 librados: setting wanted keys 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 librados: calling monclient init 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2f40f7ba0 0x7fb2f410dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40fcb40 0x7fb2f410e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:10.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2f40fd4a0 0x7fb2f4108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f4108e10 con 0x7fb2f40fcb40 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f4108f80 con 0x7fb2f40f7ba0 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2fa3c0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f4109140 con 0x7fb2f40fd4a0 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40fcb40 0x7fb2f410e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f895d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2f40fd4a0 0x7fb2f4108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40fcb40 0x7fb2f410e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39632/0 (socket says 172.21.15.87:39632) 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f37fe700 1 -- 172.21.15.87:0/993102450 learned_addr learned my addr 172.21.15.87:0/993102450 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f895d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2f40fd4a0 0x7fb2f4108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:47758/0 (socket says 172.21.15.87:47758) 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f3fff700 1 --2- 172.21.15.87:0/993102450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2f40f7ba0 0x7fb2f410dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f37fe700 1 -- 172.21.15.87:0/993102450 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2f40fd4a0 msgr2=0x7fb2f4108920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f37fe700 1 --2- 172.21.15.87:0/993102450 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2f40fd4a0 0x7fb2f4108920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:10.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f37fe700 1 -- 172.21.15.87:0/993102450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2f40f7ba0 msgr2=0x7fb2f410dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f37fe700 1 --2- 172.21.15.87:0/993102450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2f40f7ba0 0x7fb2f410dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f37fe700 1 -- 172.21.15.87:0/993102450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec0097c0 con 0x7fb2f40fcb40 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f895d700 1 --2- 172.21.15.87:0/993102450 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2f40fd4a0 0x7fb2f4108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f37fe700 1 --2- 172.21.15.87:0/993102450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40fcb40 0x7fb2f410e390 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 rx=0x7fb2e40027f0 tx=0x7fb2e4002b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.712+0000 7fb2f17fa700 10 client.?.objecter ms_handle_connect 0x7fb2f40fcb40 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.713+0000 7fb2f17fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.713+0000 7fb2f17fa700 1 -- 172.21.15.87:0/993102450 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4007b40 con 0x7fb2f40fcb40 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.713+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f4109420 con 0x7fb2f40fcb40 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.713+0000 7fb2f17fa700 1 -- 172.21.15.87:0/993102450 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2e400f040 con 0x7fb2f40fcb40 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.713+0000 7fb2f17fa700 1 -- 172.21.15.87:0/993102450 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e40146f0 con 0x7fb2f40fcb40 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.713+0000 7fb2fa3c0700 10 client.27477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:10.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.713+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f4114cf0 con 0x7fb2f40fcb40 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.713+0000 7fb2fa3c0700 1 librados: init done 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.714+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f405fa40 con 0x7fb2f40fcb40 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.714+0000 7fb2f17fa700 1 -- 172.21.15.87:0/993102450 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb2e4009110 con 0x7fb2f40fcb40 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.714+0000 7fb2f17fa700 1 --2- 172.21.15.87:0/993102450 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2e0062c90 0x7fb2e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.715+0000 7fb2f3fff700 1 --2- 172.21.15.87:0/993102450 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2e0062c90 0x7fb2e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.715+0000 7fb2f17fa700 1 -- 172.21.15.87:0/993102450 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(531..531 src has 173..531) v4 ==== 9078+0+0 (secure 0 0 0) 0x7fb2e407d300 con 0x7fb2f40fcb40 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.715+0000 7fb2f17fa700 10 client.27477.objecter ms_dispatch 0x7fb2f4001550 osd_map(531..531 src has 173..531) v4 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.715+0000 7fb2f17fa700 3 client.27477.objecter handle_osd_map got epochs [531,531] > 0 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.715+0000 7fb2f17fa700 3 client.27477.objecter handle_osd_map decoding full epoch 531 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.715+0000 7fb2f17fa700 20 client.27477.objecter dump_active .. 0 homeless 2024-02-23T00:39:10.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.715+0000 7fb2f17fa700 10 client.27477.objecter ms_handle_connect 0x7fb2e0062c90 2024-02-23T00:39:10.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.715+0000 7fb2f3fff700 1 --2- 172.21.15.87:0/993102450 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2e0062c90 0x7fb2e0065150 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 rx=0x7fb2dc004e60 tx=0x7fb2dc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:10.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.719+0000 7fb2f17fa700 1 -- 172.21.15.87:0/993102450 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb2e4050370 con 0x7fb2f40fcb40 2024-02-23T00:39:10.747 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:10.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:10.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:10.853 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:10.864 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:39:10.879 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:10.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:10.891+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"} v 0) v1 -- 0x7fb2f41060b0 con 0x7fb2f40fcb40 2024-02-23T00:39:10.897 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:10.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:11.001 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:11.027 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:11.046 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:11.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:11.150 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:11.175 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:39:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:39:11.176 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:11.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.192+0000 7fb2f17fa700 1 -- 172.21.15.87:0/993102450 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]=0 set pool 4 pgp_num to 32 v532) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb2e404ff60 con 0x7fb2f40fcb40 2024-02-23T00:39:11.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.195+0000 7fb2fa3c0700 10 librados: watch_flush enter 2024-02-23T00:39:11.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.195+0000 7fb2fa3c0700 10 librados: watch_flush exit 2024-02-23T00:39:11.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.195+0000 7fb2fa3c0700 20 client.27477.objecter shutdown clearing up homeless session... 2024-02-23T00:39:11.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.195+0000 7fb2fa3c0700 10 client.27477.objecter successfully canceled tick 2024-02-23T00:39:11.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.195+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2e0062c90 msgr2=0x7fb2e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:11.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.195+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/993102450 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2e0062c90 0x7fb2e0065150 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 rx=0x7fb2dc004e60 tx=0x7fb2dc00b040).stop 2024-02-23T00:39:11.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.195+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40fcb40 msgr2=0x7fb2f410e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:11.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.195+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/993102450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40fcb40 0x7fb2f410e390 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 rx=0x7fb2e40027f0 tx=0x7fb2e4002b00).stop 2024-02-23T00:39:11.199 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:11.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.196+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 shutdown_connections 2024-02-23T00:39:11.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.196+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/993102450 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb2e0062c90 0x7fb2e0065150 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:11.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.196+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/993102450 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2f40fd4a0 0x7fb2f4108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:11.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.196+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/993102450 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2f40f7ba0 0x7fb2f410dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:11.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.196+0000 7fb2fa3c0700 1 --2- 172.21.15.87:0/993102450 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2f40fcb40 0x7fb2f410e390 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:11.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.196+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 >> 172.21.15.87:0/993102450 conn(0x7fb2f406c370 msgr2=0x7fb2f40f5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:11.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.196+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 shutdown_connections 2024-02-23T00:39:11.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.196+0000 7fb2fa3c0700 1 -- 172.21.15.87:0/993102450 wait complete. 2024-02-23T00:39:11.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:11.196+0000 7fb2fa3c0700 1 librados: shutdown 2024-02-23T00:39:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:11.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:11.210 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 4 pgp_num to 32 2024-02-23T00:39:11.304 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:11.330 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:09.719140+0000 mgr.y (mgr.24437) 493 : cluster [DBG] pgmap v812: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 103/1266 objects degraded (8.136%); 114/1266 objects misplaced (9.005%) 2024-02-23T00:39:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:10.184626+0000 mon.a (mon.0) 2915 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:39:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:10.191771+0000 mon.a (mon.0) 2916 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-88", "mode": "writeback"}]': finished 2024-02-23T00:39:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:10.191839+0000 mon.a (mon.0) 2917 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2024-02-23T00:39:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:10.192613+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:10.194600+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:10.892993+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? 172.21.15.87:0/993102450' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:39:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:09.719140+0000 mgr.y (mgr.24437) 493 : cluster [DBG] pgmap v812: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 103/1266 objects degraded (8.136%); 114/1266 objects misplaced (9.005%) 2024-02-23T00:39:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:10.184626+0000 mon.a (mon.0) 2915 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:39:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:10.191771+0000 mon.a (mon.0) 2916 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-88", "mode": "writeback"}]': finished 2024-02-23T00:39:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:10.191839+0000 mon.a (mon.0) 2917 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2024-02-23T00:39:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:10.192613+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:10.194600+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:10.892993+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? 172.21.15.87:0/993102450' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:39:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:09.719140+0000 mgr.y (mgr.24437) 493 : cluster [DBG] pgmap v812: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 103/1266 objects degraded (8.136%); 114/1266 objects misplaced (9.005%) 2024-02-23T00:39:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:10.184626+0000 mon.a (mon.0) 2915 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:39:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:10.191771+0000 mon.a (mon.0) 2916 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-88", "mode": "writeback"}]': finished 2024-02-23T00:39:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:10.191839+0000 mon.a (mon.0) 2917 : cluster [DBG] osdmap e531: 8 total, 7 up, 8 in 2024-02-23T00:39:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:10.192613+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:10.194600+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:10.892993+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? 172.21.15.87:0/993102450' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:39:12.060 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:12.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:12.097 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:12.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:12.200 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:12.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:12] "GET /metrics HTTP/1.1" 200 189326 "" "Prometheus/2.33.4" 2024-02-23T00:39:12.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:11.059866+0000 mgr.y (mgr.24437) 494 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:12.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:11.192317+0000 mon.a (mon.0) 2920 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-88"}]': finished 2024-02-23T00:39:12.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:11.192407+0000 mon.a (mon.0) 2921 : audit [INF] from='client.? 172.21.15.87:0/993102450' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:39:12.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:11.192460+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2024-02-23T00:39:12.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:11.194548+0000 mon.b (mon.1) 432 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:39:12.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:11.197382+0000 mon.a (mon.0) 2923 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:12.142681+0000 mon.a (mon.0) 2924 : cluster [WRN] Health check update: Degraded data redundancy: 103/1266 objects degraded (8.136%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:12.143287+0000 mon.a (mon.0) 2925 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:12.148378+0000 mon.a (mon.0) 2926 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:12.148453+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:12.152036+0000 mon.b (mon.1) 433 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:12.152219+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:11.059866+0000 mgr.y (mgr.24437) 494 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:11.192317+0000 mon.a (mon.0) 2920 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-88"}]': finished 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:11.192407+0000 mon.a (mon.0) 2921 : audit [INF] from='client.? 172.21.15.87:0/993102450' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:11.192460+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2024-02-23T00:39:12.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:11.194548+0000 mon.b (mon.1) 432 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:39:12.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:11.197382+0000 mon.a (mon.0) 2923 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:39:12.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:12.142681+0000 mon.a (mon.0) 2924 : cluster [WRN] Health check update: Degraded data redundancy: 103/1266 objects degraded (8.136%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:12.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:12.143287+0000 mon.a (mon.0) 2925 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:39:12.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:12.148378+0000 mon.a (mon.0) 2926 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:39:12.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:12.148453+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2024-02-23T00:39:12.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:12.152036+0000 mon.b (mon.1) 433 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-23T00:39:12.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:12.152219+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-23T00:39:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:11.059866+0000 mgr.y (mgr.24437) 494 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:11.192317+0000 mon.a (mon.0) 2920 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-88"}]': finished 2024-02-23T00:39:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:11.192407+0000 mon.a (mon.0) 2921 : audit [INF] from='client.? 172.21.15.87:0/993102450' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.log", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:39:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:11.192460+0000 mon.a (mon.0) 2922 : cluster [DBG] osdmap e532: 8 total, 7 up, 8 in 2024-02-23T00:39:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:11.194548+0000 mon.b (mon.1) 432 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:39:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:11.197382+0000 mon.a (mon.0) 2923 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:39:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:12.142681+0000 mon.a (mon.0) 2924 : cluster [WRN] Health check update: Degraded data redundancy: 103/1266 objects degraded (8.136%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:12.143287+0000 mon.a (mon.0) 2925 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:39:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:12.148378+0000 mon.a (mon.0) 2926 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:39:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:12.148453+0000 mon.a (mon.0) 2927 : cluster [DBG] osdmap e533: 8 total, 7 up, 8 in 2024-02-23T00:39:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:12.152036+0000 mon.b (mon.1) 433 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-23T00:39:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:12.152219+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-23T00:39:13.433 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.24712.0:21 7.2 e112/94 7:40054bab:test-rados-api-smithi087-82115-6::foo:head v 112'4, mlcod=110'1)", 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 490.90928895600001, 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050194328000000003, 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.459 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.460 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546123+0000", 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1158000000000001e-05 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546134+0000", 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8680000000000001e-06 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546141+0000", 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8170000000000001e-06 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.546147+0000", 2024-02-23T00:39:13.461 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0014866250000000001 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.547634+0000", 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0406000000000001e-05 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.547654+0000", 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001658442 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.462 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.549313+0000", 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.047004275999999998 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.596317+0000", 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050194328000000003 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.25157.0:2 58.15 58:ad551702:::test:head [call hello.record_hello in=17b] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.599190+0000", 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 490.85622153200001, 2024-02-23T00:39:13.463 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.066902009999999998, 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.25157", 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4218332242", 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599190+0000", 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.464 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599190+0000", 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599189+0000", 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4850000000000003e-06 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.465 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599196+0000", 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6200000000000005e-06 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599205+0000", 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.934e-06 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.599212+0000", 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062881679999999995 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.466 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.662093+0000", 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013823899999999999 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.662232+0000", 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00022222400000000001 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.662454+0000", 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0017213930000000001 2024-02-23T00:39:13.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.468 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.468 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.468 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.664175+0000", 2024-02-23T00:39:13.468 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001211052 2024-02-23T00:39:13.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.468 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.468 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.468 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.665386+0000", 2024-02-23T00:39:13.468 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00061359100000000003 2024-02-23T00:39:13.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.469 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.469 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:13.469 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.666000+0000", 2024-02-23T00:39:13.469 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4636999999999999e-05 2024-02-23T00:39:13.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.469 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.469 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.469 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.666054+0000", 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.79e-05 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.666092+0000", 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.066902009999999998 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.470 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.471 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:3 63.11 e117/114 63:8d290297:test-rados-api-smithi087-82091-8::foo:head v 117'1, mlcod=0'0)", 2024-02-23T00:39:13.471 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:39:13.471 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 485.489887581, 2024-02-23T00:39:13.471 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074073193999999995, 2024-02-23T00:39:13.471 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.471 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.471 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.473 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.473 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.473 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1930000000000001e-06 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.474 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965529+0000", 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5130000000000002e-06 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965536+0000", 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4860000000000002e-06 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965542+0000", 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002109362 2024-02-23T00:39:13.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.967652+0000", 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5451000000000002e-05 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.967667+0000", 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071902888999999998 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.476 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.039570+0000", 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7617999999999999e-05 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.039597+0000", 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074073193999999995 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.477 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:10 61.1f e118/112 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head v 118'6, mlcod=115'3)", 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 484.274991177, 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034719099000000003, 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.478 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180420+0000", 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7989999999999999e-06 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.479 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180426+0000", 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8590000000000001e-06 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180432+0000", 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8689999999999998e-06 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180439+0000", 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8420000000000001e-05 2024-02-23T00:39:13.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180477+0000", 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6568000000000001e-05 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180494+0000", 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034614477999999997 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.481 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.215108+0000", 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1405999999999997e-05 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.215140+0000", 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034719099000000003 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.28906.0:2 69.15 69:ad551702:::test:head [create,write 0~4] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:39:13.482 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 483.27474547899999, 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071209718000000005, 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.28906", 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3024219439", 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.483 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180664+0000", 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8156e-05 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180683+0000", 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3705e-05 2024-02-23T00:39:13.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180696+0000", 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0080000000000002e-06 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180704+0000", 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046747999999999999 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.181172+0000", 2024-02-23T00:39:13.485 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7989999999999996e-06 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.181179+0000", 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.060883893000000001 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242062+0000", 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012322200000000001 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242186+0000", 2024-02-23T00:39:13.486 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015854399999999999 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242344+0000", 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0042969569999999997 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.246641+0000", 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020537120000000001 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.248695+0000", 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0030777460000000001 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.487 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251773+0000", 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9827999999999997e-05 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251832+0000", 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3659000000000003e-05 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.488 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251876+0000", 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071209718000000005 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:14 61.11 e120/112 61:8c482978:test-rados-api-smithi087-82107-8::foo:head v 120'3, mlcod=0'0)", 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 482.22954228700002, 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.14463899, 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225869+0000", 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2559999999999996e-06 2024-02-23T00:39:13.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225877+0000", 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2709999999999998e-06 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225883+0000", 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1999999999999999e-06 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225889+0000", 2024-02-23T00:39:13.491 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.13293687000000001 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358826+0000", 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1993000000000001e-05 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358848+0000", 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011619638 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.492 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.370468+0000", 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0979000000000002e-05 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.370509+0000", 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.14463899 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.41269.0:2 89.6 89:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e137)", 2024-02-23T00:39:13.493 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 465.43250999200001, 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.10320085900000001, 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.41269", 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/244424669", 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:39:13.494 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022900+0000", 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2230000000000001e-06 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.495 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022907+0000", 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026383299999999999 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.023171+0000", 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0040999999999999e-05 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.023181+0000", 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014422166 2024-02-23T00:39:13.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.037603+0000", 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5469999999999997e-06 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.037612+0000", 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.064763451999999999 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.497 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102375+0000", 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011688600000000001 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102492+0000", 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013874699999999999 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102631+0000", 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00093563499999999998 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.103566+0000", 2024-02-23T00:39:13.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001628163 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.105195+0000", 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020814273000000001 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126009+0000", 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0680000000000003e-05 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126059+0000", 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3244999999999997e-05 2024-02-23T00:39:13.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.500 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.500 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.500 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126103+0000", 2024-02-23T00:39:13.500 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.10320085900000001 2024-02-23T00:39:13.500 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.500 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:265 86.0 86:00000000:%01:::head [pgnls start_epoch 139] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e139)", 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 463.41800092300002, 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.048698721, 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:39:13.501 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 265 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.502 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037409+0000", 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.456e-06 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037418+0000", 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8989999999999996e-06 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037427+0000", 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0649999999999999e-06 2024-02-23T00:39:13.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037435+0000", 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017399350000000001 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.054835+0000", 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8939999999999993e-06 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:39:13.504 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.054843+0000", 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.031096629000000001 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085939+0000", 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8654e-05 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085978+0000", 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013163199999999999 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.505 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.086110+0000", 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.048698721 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.36860.0:2 114.16 114:6eb9e51e:::oid_RoundTripWriteSamePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 434.45127491300002, 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036246133, 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.506 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.36860", 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4037067096", 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:39:13.507 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004135+0000", 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6179999999999995e-06 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004143+0000", 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4696e-05 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.508 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004158+0000", 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7260000000000008e-06 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004167+0000", 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5045999999999999e-05 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004242+0000", 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1400000000000002e-06 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.509 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004249+0000", 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016930304 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.021179+0000", 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2629999999999997e-06 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.510 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.021187+0000", 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016612273 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037799+0000", 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3777999999999997e-05 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037873+0000", 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7861000000000005e-05 2024-02-23T00:39:13.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037960+0000", 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00035295600000000003 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.038313+0000", 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028716700000000001 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.038601+0000", 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016768 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040277+0000", 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0985000000000001e-05 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040328+0000", 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4675000000000001e-05 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040383+0000", 2024-02-23T00:39:13.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036246133 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:538 63.8 e182/114 63:1309b075:test-rados-api-smithi087-82091-14::foo36:7 v 182'40, mlcod=148'34)", 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 420.45230704900001, 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027705371999999999, 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.514 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003104+0000", 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7220000000000002e-06 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.515 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003111+0000", 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5139999999999998e-06 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003118+0000", 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8010000000000001e-06 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003122+0000", 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024802080000000001 2024-02-23T00:39:13.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027924+0000", 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9273999999999999e-05 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027943+0000", 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0028410689999999999 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030784+0000", 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6224000000000002e-05 2024-02-23T00:39:13.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030810+0000", 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027705371999999999 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37106.0:2 149.1a 149:5ec95afd:::oid_ReadIntoBufferlist_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 386.42824440599998, 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.079034594, 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.518 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37106", 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4049231794", 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027166+0000", 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9900000000000002e-06 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.520 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.521 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027172+0000", 2024-02-23T00:39:13.521 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4880000000000007e-06 2024-02-23T00:39:13.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.521 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.521 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.521 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027180+0000", 2024-02-23T00:39:13.521 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4049999999999997e-06 2024-02-23T00:39:13.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027187+0000", 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000237202 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027424+0000", 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000115528 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.522 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027539+0000", 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00014589299999999999 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,3", 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027685+0000", 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00088195299999999999 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.028567+0000", 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026987410999999999 2024-02-23T00:39:13.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.055555+0000", 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050551275 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106106+0000", 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5090999999999998e-05 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106161+0000", 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0967000000000003e-05 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106202+0000", 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.079034594 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:2298 169.17 169:ed571587:test-rados-api-smithi087-82091-30::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e256)", 2024-02-23T00:39:13.525 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 346.43000443300002, 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869, 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2298 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.526 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025406+0000", 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1740000000000006e-06 2024-02-23T00:39:13.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025415+0000", 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1041e-05 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025426+0000", 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0450000000000003e-06 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025433+0000", 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7723999999999998e-05 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025491+0000", 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3479999999999993e-06 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025500+0000", 2024-02-23T00:39:13.529 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.21538338000000001 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.240883+0000", 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021381300000000001 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.241097+0000", 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00573626 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.530 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246834+0000", 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6838999999999999e-05 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246930+0000", 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013423800000000001 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,2", 2024-02-23T00:39:13.531 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247065+0000", 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037269000000000001 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247437+0000", 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026355299999999998 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247701+0000", 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011556330000000001 2024-02-23T00:39:13.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248856+0000", 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7636999999999998e-05 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248904+0000", 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0179000000000003e-05 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.533 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248944+0000", 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 322/321 [PushReplyOp(4:e9187ab8:reshard::reshard.0000000000:head)])", 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 280.132042377, 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089931579999999997, 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.534 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.535 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9330000000000004e-06 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323376+0000", 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1073000000000001e-05 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.536 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323387+0000", 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.489e-06 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323393+0000", 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089729134000000002 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.413122+0000", 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00017875200000000001 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.537 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.413301+0000", 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089931579999999997 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 322/321 [PushReplyOp(4:ec529b31:::obj_delete_at_hint.0000000050:head)])", 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 279.69778764599999, 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089810938000000007, 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:39:13.538 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757623+0000", 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9920000000000007e-06 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757631+0000", 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1764e-05 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757643+0000", 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4240000000000004e-06 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.540 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757650+0000", 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089629418000000002 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.847280+0000", 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000155267 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.847435+0000", 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089810938000000007 2024-02-23T00:39:13.541 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.542 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.542 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.542 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 323/321 [PushReplyOp(4:ece66ccb:::obj_delete_at_hint.0000000102:head)])", 2024-02-23T00:39:13.542 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:39:13.542 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 279.34966341799998, 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089865278000000007, 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0180000000000001e-06 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105753+0000", 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3409999999999997e-06 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.544 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105760+0000", 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2069999999999997e-06 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105764+0000", 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089713752999999993 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.195478+0000", 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013554999999999999 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.545 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.195614+0000", 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089865278000000007 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 323/321 [PushReplyOp(4:ee62d5d2:lc::lc.1:head)])", 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 279.17341654000001, 2024-02-23T00:39:13.546 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.090010914999999997, 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.547 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7539999999999998e-06 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281998+0000", 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8160000000000004e-06 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.548 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.282002+0000", 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.794e-06 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.282005+0000", 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089811657000000003 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.371817+0000", 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000189092 2024-02-23T00:39:13.549 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.372006+0000", 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.090010914999999997 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_update_log_missing(205.3s1 epoch 376/375 rep_tid 700 entries 376'1 (0'0) error 205:cbb9903e:::test_obj:head by client.37589.0:1 0.000000 -95 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 226.35713613799999, 2024-02-23T00:39:13.550 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065042928, 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.551 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098275+0000", 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2939999999999994e-06 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098285+0000", 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2446000000000001e-05 2024-02-23T00:39:13.552 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098297+0000", 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2310000000000002e-06 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098306+0000", 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012984100000000001 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098436+0000", 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.064882361999999999 2024-02-23T00:39:13.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.163319+0000", 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065042928 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.4.31:55 216.9 e451/444 216:939aa6ec:test-rados-api-smithi087-82091-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head v 451'3, mlcod=446'1)", 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:37:17.130726+0000", 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 116.32468627199999, 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056389038000000002, 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.554 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130726+0000", 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130726+0000", 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.555 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130725+0000", 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5980000000000003e-06 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130734+0000", 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0025e-05 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.556 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130744+0000", 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6889999999999998e-06 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130751+0000", 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5022999999999999e-05 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130796+0000", 2024-02-23T00:39:13.557 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.4711000000000001e-05 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130821+0000", 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056043386000000001 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.186864+0000", 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025056500000000001 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.558 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187115+0000", 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056389038000000002 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.4.31:56 216.17 e451/444 216:ec5c3d73:test-rados-api-smithi087-82091-70::b1a341c59dc4c60bbe00244ac8bba87528936bc7:head v 451'2, mlcod=0'0)", 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:37:17.143192+0000", 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 116.31221960000001, 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044005469999999998, 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.559 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143192+0000", 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143192+0000", 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.560 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143190+0000", 2024-02-23T00:39:13.561 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5917999999999999e-05 2024-02-23T00:39:13.561 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.561 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.561 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.561 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143206+0000", 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5682999999999999e-05 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143222+0000", 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2430000000000005e-06 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143230+0000", 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.672e-05 2024-02-23T00:39:13.562 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143277+0000", 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2272e-05 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143299+0000", 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.043740635999999999 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187040+0000", 2024-02-23T00:39:13.563 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015778499999999999 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187198+0000", 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044005469999999998 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.7.51:63 63.10 63:0bcb77f5:test-rados-api-smithi087-82091-80::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e494)", 2024-02-23T00:39:13.564 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:38:31.219712+0000", 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 42.235699938000003, 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030054408000000001, 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219712+0000", 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219712+0000", 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:13.565 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219710+0000", 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1394e-05 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219721+0000", 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4367e-05 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219736+0000", 2024-02-23T00:39:13.566 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9110000000000004e-06 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219743+0000", 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1088999999999999e-05 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219784+0000", 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3820000000000003e-06 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219789+0000", 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020061689999999999 2024-02-23T00:39:13.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.221795+0000", 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012612300000000001 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.221921+0000", 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000363552 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 7", 2024-02-23T00:39:13.568 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.222285+0000", 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00093142099999999999 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.223216+0000", 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026412662999999999 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.249629+0000", 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9842999999999997e-05 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:13.569 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.249679+0000", 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7428000000000004e-05 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.249766+0000", 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030054408000000001 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:13.570 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:39:13.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:11.719786+0000 mgr.y (mgr.24437) 495 : cluster [DBG] pgmap v815: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:13.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:13.149443+0000 mon.a (mon.0) 2929 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_count","val": "1"}]': finished 2024-02-23T00:39:13.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:13.149515+0000 mon.a (mon.0) 2930 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2024-02-23T00:39:13.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:13.152563+0000 mon.b (mon.1) 434 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:13.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:13.152756+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:13.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:11.719786+0000 mgr.y (mgr.24437) 495 : cluster [DBG] pgmap v815: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:13.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:13.149443+0000 mon.a (mon.0) 2929 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_count","val": "1"}]': finished 2024-02-23T00:39:13.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:13.149515+0000 mon.a (mon.0) 2930 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2024-02-23T00:39:13.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:13.152563+0000 mon.b (mon.1) 434 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:13.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:13.152756+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:11.719786+0000 mgr.y (mgr.24437) 495 : cluster [DBG] pgmap v815: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:13.149443+0000 mon.a (mon.0) 2929 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_count","val": "1"}]': finished 2024-02-23T00:39:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:13.149515+0000 mon.a (mon.0) 2930 : cluster [DBG] osdmap e534: 8 total, 7 up, 8 in 2024-02-23T00:39:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:13.152563+0000 mon.b (mon.1) 434 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:13.152756+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:13.962 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:13.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:14.067 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:14.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:39:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:39:14.204 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:14.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:14.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:14.310 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:14.336 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:14.356 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:14.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:14.459 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:14.486 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:14.510 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:14.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:14.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:14.614 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:14.642 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:13.720589+0000 mgr.y (mgr.24437) 496 : cluster [DBG] pgmap v818: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:15.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:14.150593+0000 mon.a (mon.0) 2932 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:39:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:14.150669+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2024-02-23T00:39:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:14.151482+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-23T00:39:15.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:14.153001+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-23T00:39:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:13.720589+0000 mgr.y (mgr.24437) 496 : cluster [DBG] pgmap v818: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:14.150593+0000 mon.a (mon.0) 2932 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:39:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:14.150669+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2024-02-23T00:39:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:14.151482+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-23T00:39:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:14.153001+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-23T00:39:15.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:13.720589+0000 mgr.y (mgr.24437) 496 : cluster [DBG] pgmap v818: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:15.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:14.150593+0000 mon.a (mon.0) 2932 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:39:15.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:14.150669+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e535: 8 total, 7 up, 8 in 2024-02-23T00:39:15.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:14.151482+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-23T00:39:15.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:14.153001+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-23T00:39:15.814 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-23T00:39:16.191 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:16.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:16.299 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:16.327 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:16.346 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:16.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:15.154028+0000 mon.a (mon.0) 2935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "target_max_objects","val": "250"}]': finished 2024-02-23T00:39:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:15.154104+0000 mon.a (mon.0) 2936 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2024-02-23T00:39:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:15.197282+0000 mon.b (mon.1) 436 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:16.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:15.197535+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:16.450 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:16.477 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:16.495 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:16.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:15.154028+0000 mon.a (mon.0) 2935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "target_max_objects","val": "250"}]': finished 2024-02-23T00:39:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:15.154104+0000 mon.a (mon.0) 2936 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2024-02-23T00:39:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:15.197282+0000 mon.b (mon.1) 436 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:16.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:15.197535+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:15.154028+0000 mon.a (mon.0) 2935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-88","var": "target_max_objects","val": "250"}]': finished 2024-02-23T00:39:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:15.154104+0000 mon.a (mon.0) 2936 : cluster [DBG] osdmap e536: 8 total, 7 up, 8 in 2024-02-23T00:39:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:15.197282+0000 mon.b (mon.1) 436 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:15.197535+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:16.598 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:16.625 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:16.644 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:16.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:16.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:16.747 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:16.775 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:17.022 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:17.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:17.075 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:39:17.076 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:39:17.076 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.control 2024-02-23T00:39:17.076 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.control by 10 min 8 2024-02-23T00:39:17.076 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:39:17.126 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:17.150 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:17.158 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.272+0000 7fbac85dd700 1 -- 172.21.15.87:0/500903493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbac00692d0 msgr2=0x7fbac00696b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.272+0000 7fbac85dd700 1 --2- 172.21.15.87:0/500903493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbac00692d0 0x7fbac00696b0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7fbabc0048a0 tx=0x7fbabc009cb0).stop 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.273+0000 7fbac85dd700 1 -- 172.21.15.87:0/500903493 shutdown_connections 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.273+0000 7fbac85dd700 1 --2- 172.21.15.87:0/500903493 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbac01044a0 0x7fbac0104920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.273+0000 7fbac85dd700 1 --2- 172.21.15.87:0/500903493 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbac00692d0 0x7fbac00696b0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.273+0000 7fbac85dd700 1 --2- 172.21.15.87:0/500903493 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbac0069ba0 0x7fbac0103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.273+0000 7fbac85dd700 1 -- 172.21.15.87:0/500903493 >> 172.21.15.87:0/500903493 conn(0x7fbac0063db0 msgr2=0x7fbac0067560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.273+0000 7fbac85dd700 1 -- 172.21.15.87:0/500903493 shutdown_connections 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.273+0000 7fbac85dd700 1 -- 172.21.15.87:0/500903493 wait complete. 2024-02-23T00:39:17.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 librados: starting msgr at 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 librados: starting objecter 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 Processor -- start 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 -- start start 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 librados: setting wanted keys 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 librados: calling monclient init 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbac00692d0 0x7fbac011d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbac0069ba0 0x7fbac011d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbac01044a0 0x7fbac01171c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac01176b0 con 0x7fbac0069ba0 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0117820 con 0x7fbac01044a0 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac85dd700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac01179e0 con 0x7fbac00692d0 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac5b78700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbac0069ba0 0x7fbac011d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac5b78700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbac0069ba0 0x7fbac011d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60098/0 (socket says 172.21.15.87:60098) 2024-02-23T00:39:17.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac5b78700 1 -- 172.21.15.87:0/3566603870 learned_addr learned my addr 172.21.15.87:0/3566603870 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.274+0000 7fbac6379700 1 --2- 172.21.15.87:0/3566603870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbac00692d0 0x7fbac011d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.275+0000 7fbac6379700 1 -- 172.21.15.87:0/3566603870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbac01044a0 msgr2=0x7fbac01171c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.275+0000 7fbac6379700 1 --2- 172.21.15.87:0/3566603870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbac01044a0 0x7fbac01171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.275+0000 7fbac6379700 1 -- 172.21.15.87:0/3566603870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbac0069ba0 msgr2=0x7fbac011d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.275+0000 7fbac6379700 1 --2- 172.21.15.87:0/3566603870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbac0069ba0 0x7fbac011d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.275+0000 7fbac6379700 1 -- 172.21.15.87:0/3566603870 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbabc005040 con 0x7fbac00692d0 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.275+0000 7fbac6379700 1 --2- 172.21.15.87:0/3566603870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbac00692d0 0x7fbac011d020 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7fbabc004ed0 tx=0x7fbabc007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.275+0000 7fbab77fe700 10 client.?.objecter ms_handle_connect 0x7fbac00692d0 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.275+0000 7fbab77fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.277+0000 7fbab77fe700 1 -- 172.21.15.87:0/3566603870 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc00e030 con 0x7fbac00692d0 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.277+0000 7fbab77fe700 1 -- 172.21.15.87:0/3566603870 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbabc0092e0 con 0x7fbac00692d0 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.277+0000 7fbab77fe700 1 -- 172.21.15.87:0/3566603870 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc018070 con 0x7fbac00692d0 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.278+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac0117c60 con 0x7fbac00692d0 2024-02-23T00:39:17.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.278+0000 7fbac85dd700 10 client.37949.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:17.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.278+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac0107410 con 0x7fbac00692d0 2024-02-23T00:39:17.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.278+0000 7fbac85dd700 1 librados: init done 2024-02-23T00:39:17.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.279+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac0118070 con 0x7fbac00692d0 2024-02-23T00:39:17.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.291+0000 7fbab77fe700 1 -- 172.21.15.87:0/3566603870 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fbabc01d420 con 0x7fbac00692d0 2024-02-23T00:39:17.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.292+0000 7fbab77fe700 1 --2- 172.21.15.87:0/3566603870 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaac062d20 0x7fbaac0651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:17.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.292+0000 7fbab77fe700 1 -- 172.21.15.87:0/3566603870 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(538..538 src has 173..538) v4 ==== 9070+0+0 (secure 0 0 0) 0x7fbabc07eb00 con 0x7fbac00692d0 2024-02-23T00:39:17.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.292+0000 7fbab77fe700 10 client.37949.objecter ms_dispatch 0x7fbac0001550 osd_map(538..538 src has 173..538) v4 2024-02-23T00:39:17.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.292+0000 7fbab77fe700 3 client.37949.objecter handle_osd_map got epochs [538,538] > 0 2024-02-23T00:39:17.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.292+0000 7fbab77fe700 3 client.37949.objecter handle_osd_map decoding full epoch 538 2024-02-23T00:39:17.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.292+0000 7fbab77fe700 20 client.37949.objecter dump_active .. 0 homeless 2024-02-23T00:39:17.296 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.292+0000 7fbab77fe700 1 -- 172.21.15.87:0/3566603870 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fbabc0a7000 con 0x7fbac00692d0 2024-02-23T00:39:17.297 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:17.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.298+0000 7fbac5b78700 1 --2- 172.21.15.87:0/3566603870 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaac062d20 0x7fbaac0651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:17.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.301+0000 7fbac5b78700 1 --2- 172.21.15.87:0/3566603870 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaac062d20 0x7fbaac0651e0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 rx=0x7fbac0118960 tx=0x7fbab800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:17.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.301+0000 7fbab77fe700 10 client.37949.objecter ms_handle_connect 0x7fbaac062d20 2024-02-23T00:39:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:17.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:17.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:15.722026+0000 mgr.y (mgr.24437) 497 : cluster [DBG] pgmap v821: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:17.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:16.172373+0000 mon.a (mon.0) 2938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:39:17.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:16.172431+0000 mon.a (mon.0) 2939 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2024-02-23T00:39:17.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:16.186908+0000 mon.b (mon.1) 437 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:17.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:16.187373+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:17.307 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:17.144410+0000 mon.a (mon.0) 2941 : cluster [WRN] Health check update: Degraded data redundancy: 104/1269 objects degraded (8.195%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:17.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:15.722026+0000 mgr.y (mgr.24437) 497 : cluster [DBG] pgmap v821: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:17.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:16.172373+0000 mon.a (mon.0) 2938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:39:17.307 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:16.172431+0000 mon.a (mon.0) 2939 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2024-02-23T00:39:17.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:16.186908+0000 mon.b (mon.1) 437 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:17.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:16.187373+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:17.308 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:17.144410+0000 mon.a (mon.0) 2941 : cluster [WRN] Health check update: Degraded data redundancy: 104/1269 objects degraded (8.195%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:17.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:17.439 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:17.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:17.505+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fbac0116000 con 0x7fbac00692d0 2024-02-23T00:39:17.541 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:17.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:17.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:17.674 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:15.722026+0000 mgr.y (mgr.24437) 497 : cluster [DBG] pgmap v821: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:16.172373+0000 mon.a (mon.0) 2938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:39:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:16.172431+0000 mon.a (mon.0) 2939 : cluster [DBG] osdmap e537: 8 total, 7 up, 8 in 2024-02-23T00:39:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:16.186908+0000 mon.b (mon.1) 437 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:16.187373+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88"}]: dispatch 2024-02-23T00:39:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:17.144410+0000 mon.a (mon.0) 2941 : cluster [WRN] Health check update: Degraded data redundancy: 104/1269 objects degraded (8.195%), 23 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:17.822 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:17.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:17.926 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:17.955 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:18.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.265+0000 7fbab77fe700 1 -- 172.21.15.87:0/3566603870 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v539) v1 ==== 91+0+0 (secure 0 0 0) 0x7fbabc051be0 con 0x7fbac00692d0 2024-02-23T00:39:18.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.268+0000 7fbac85dd700 10 librados: watch_flush enter 2024-02-23T00:39:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.268+0000 7fbac85dd700 10 librados: watch_flush exit 2024-02-23T00:39:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.268+0000 7fbac85dd700 20 client.37949.objecter shutdown clearing up homeless session... 2024-02-23T00:39:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.268+0000 7fbac85dd700 10 client.37949.objecter successfully canceled tick 2024-02-23T00:39:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.269+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaac062d20 msgr2=0x7fbaac0651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.269+0000 7fbac85dd700 1 --2- 172.21.15.87:0/3566603870 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaac062d20 0x7fbaac0651e0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 rx=0x7fbac0118960 tx=0x7fbab800b040).stop 2024-02-23T00:39:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.269+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbac00692d0 msgr2=0x7fbac011d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:18.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.269+0000 7fbac85dd700 1 --2- 172.21.15.87:0/3566603870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbac00692d0 0x7fbac011d020 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 rx=0x7fbabc004ed0 tx=0x7fbabc007f90).stop 2024-02-23T00:39:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.270+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 shutdown_connections 2024-02-23T00:39:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.270+0000 7fbac85dd700 1 --2- 172.21.15.87:0/3566603870 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbaac062d20 0x7fbaac0651e0 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.270+0000 7fbac85dd700 1 --2- 172.21.15.87:0/3566603870 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbac01044a0 0x7fbac01171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:39:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.270+0000 7fbac85dd700 1 --2- 172.21.15.87:0/3566603870 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbac00692d0 0x7fbac011d020 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.270+0000 7fbac85dd700 1 --2- 172.21.15.87:0/3566603870 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbac0069ba0 0x7fbac011d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.270+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 >> 172.21.15.87:0/3566603870 conn(0x7fbac0063db0 msgr2=0x7fbac010fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.271+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 shutdown_connections 2024-02-23T00:39:18.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.271+0000 7fbac85dd700 1 -- 172.21.15.87:0/3566603870 wait complete. 2024-02-23T00:39:18.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.271+0000 7fbac85dd700 1 librados: shutdown 2024-02-23T00:39:18.288 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:39:18.307 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:18.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:18.416 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:18.442 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:18.461 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:18.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:18.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:18.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.547+0000 7f18defd6700 1 -- 172.21.15.87:0/3362866369 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18d80f7330 msgr2=0x7f18d8103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:18.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.547+0000 7f18defd6700 1 --2- 172.21.15.87:0/3362866369 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18d80f7330 0x7f18d8103930 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 rx=0x7f18d0009be0 tx=0x7f18d0004d10).stop 2024-02-23T00:39:18.552 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.547+0000 7f18defd6700 1 -- 172.21.15.87:0/3362866369 shutdown_connections 2024-02-23T00:39:18.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.547+0000 7f18defd6700 1 --2- 172.21.15.87:0/3362866369 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18d80f7330 0x7f18d8103930 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:18.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.547+0000 7f18defd6700 1 --2- 172.21.15.87:0/3362866369 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d80fcac0 0x7f18d80fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:18.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.547+0000 7f18defd6700 1 --2- 172.21.15.87:0/3362866369 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18d80fd420 0x7f18d80f6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:18.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.547+0000 7f18defd6700 1 -- 172.21.15.87:0/3362866369 >> 172.21.15.87:0/3362866369 conn(0x7f18d80f26f0 msgr2=0x7f18d80f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:18.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.547+0000 7f18defd6700 1 -- 172.21.15.87:0/3362866369 shutdown_connections 2024-02-23T00:39:18.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.547+0000 7f18defd6700 1 -- 172.21.15.87:0/3362866369 wait complete. 2024-02-23T00:39:18.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 librados: starting msgr at 2024-02-23T00:39:18.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 librados: starting objecter 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 Processor -- start 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 -- start start 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 librados: setting wanted keys 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 librados: calling monclient init 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18d80f7330 0x7f18d8108fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18d80fcac0 0x7f18d81094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d80fd420 0x7f18d8109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8114190 con 0x7f18d80fcac0 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8114300 con 0x7f18d80fd420 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18defd6700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d81144c0 con 0x7f18d80f7330 2024-02-23T00:39:18.554 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18d7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18d80fcac0 0x7f18d81094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18dcd72700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18d80f7330 0x7f18d8108fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18d7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18d80fcac0 0x7f18d81094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60110/0 (socket says 172.21.15.87:60110) 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18dcd72700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18d80f7330 0x7f18d8108fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:54826/0 (socket says 172.21.15.87:54826) 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18d7fff700 1 -- 172.21.15.87:0/2008816249 learned_addr learned my addr 172.21.15.87:0/2008816249 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.548+0000 7f18dd573700 1 --2- 172.21.15.87:0/2008816249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d80fd420 0x7f18d8109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18dcd72700 1 -- 172.21.15.87:0/2008816249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d80fd420 msgr2=0x7f18d8109a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18dcd72700 1 --2- 172.21.15.87:0/2008816249 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18d80fd420 0x7f18d8109a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18dcd72700 1 -- 172.21.15.87:0/2008816249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18d80fcac0 msgr2=0x7f18d81094a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18dcd72700 1 --2- 172.21.15.87:0/2008816249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18d80fcac0 0x7f18d81094a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18dcd72700 1 -- 172.21.15.87:0/2008816249 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d00097c0 con 0x7f18d80f7330 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18d7fff700 1 --2- 172.21.15.87:0/2008816249 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18d80fcac0 0x7f18d81094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:18.555 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18dcd72700 1 --2- 172.21.15.87:0/2008816249 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18d80f7330 0x7f18d8108fb0 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 rx=0x7f18c8002760 tx=0x7f18c8009e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18d5ffb700 10 client.?.objecter ms_handle_connect 0x7f18d80f7330 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18d5ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8007b40 con 0x7f18d80f7330 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d81147a0 con 0x7f18d80f7330 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18c800f040 con 0x7f18d80f7330 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8014790 con 0x7f18d80f7330 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18defd6700 10 client.37955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d8114ce0 con 0x7f18d80f7330 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.549+0000 7f18defd6700 1 librados: init done 2024-02-23T00:39:18.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.550+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f18d805f990 con 0x7f18d80f7330 2024-02-23T00:39:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.550+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f18c8007ca0 con 0x7f18d80f7330 2024-02-23T00:39:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d5ffb700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6800/1231597463 conn(0x7f18c0062c90 0x7f18c0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(539..539 src has 173..539) v4 ==== 8685+0+0 (secure 0 0 0) 0x7f18c807ce20 con 0x7f18d80f7330 2024-02-23T00:39:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 osd_map(539..539 src has 173..539) v4 2024-02-23T00:39:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d5ffb700 3 client.37955.objecter handle_osd_map got epochs [539,539] > 0 2024-02-23T00:39:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d5ffb700 3 client.37955.objecter handle_osd_map decoding full epoch 539 2024-02-23T00:39:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d7fff700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6800/1231597463 conn(0x7f18c0062c90 0x7f18c0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d5ffb700 20 client.37955.objecter dump_active .. 0 homeless 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v539) v1 ==== 54+0+16 (secure 0 0 0) 0x7f18c807f4b0 con 0x7f18d80f7330 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d5ffb700 10 client.37955.objecter ms_handle_connect 0x7f18c0062c90 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d7fff700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6800/1231597463 conn(0x7f18c0062c90 0x7f18c0065150 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 rx=0x7f18d80617d0 tx=0x7f18cc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18defd6700 10 client.37955.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 1 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18defd6700 20 client.37955.objecter _get_session osd=-1 returning homeless 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18defd6700 10 client.37955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=540}) v3 -- 0x7f18d805ee80 con 0x7f18d80f7330 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18d8106020 con 0x7f18d80f7330 2024-02-23T00:39:18.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=539) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18c8050080 con 0x7f18d80f7330 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d6ffd700 10 client.37955.objecter _check_command_map_dne tid 1 current 539 map_dne_bound 539 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d6ffd700 10 client.37955.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18d6ffd700 15 client.37955.objecter _session_command_op_remove -1 1 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18defd6700 10 client.37955.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.551+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 2 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6805/685354171 conn(0x7f18d80f4b00 0x7f18d80273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d80f4990 osd=1 3 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d80f4990 osd=1 3 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d80f4990 osd=1 3 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 2 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d80f4990 osd=1 3 2024-02-23T00:39:18.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 1 2 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d80f4990 osd=1 4 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 10 client.37955.objecter _send_command 2 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d8059e00 con 0x7f18d80f4b00 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18dd573700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6805/685354171 conn(0x7f18d80f4b00 0x7f18d80273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18d5ffb700 10 client.37955.objecter ms_handle_connect 0x7f18d80f4b00 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18dd573700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6805/685354171 conn(0x7f18d80f4b00 0x7f18d80273c0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f18d8059e00 con 0x7f18d80f4b00 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 2 = system:0 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d80f4990 osd=1 4 2024-02-23T00:39:18.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.552+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 1 2 2024-02-23T00:39:18.566 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:18.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 10 client.37955.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:18.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 3 2024-02-23T00:39:18.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d80f4990 osd=1 3 2024-02-23T00:39:18.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d80f4990 osd=1 3 2024-02-23T00:39:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d80f4990 osd=1 3 2024-02-23T00:39:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 3 2024-02-23T00:39:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d80f4990 osd=1 3 2024-02-23T00:39:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 1 3 2024-02-23T00:39:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d80f4990 osd=1 4 2024-02-23T00:39:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 10 client.37955.objecter _send_command 3 2024-02-23T00:39:18.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.567+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18d80278b0 con 0x7f18d80f4b00 2024-02-23T00:39:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f18d80278b0 con 0x7f18d80f4b00 2024-02-23T00:39:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d80f4990 osd=1 4 2024-02-23T00:39:18.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 1 3 2024-02-23T00:39:18.571 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:39:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 10 client.37955.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 4 2024-02-23T00:39:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6809/3605394228 conn(0x7f18d80594d0 0x7f18d8118f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8059360 osd=2 3 2024-02-23T00:39:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8059360 osd=2 3 2024-02-23T00:39:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8059360 osd=2 3 2024-02-23T00:39:18.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 4 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d8059360 osd=2 3 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 2 4 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8059360 osd=2 4 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 10 client.37955.objecter _send_command 4 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d8028760 con 0x7f18d80594d0 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18dcd72700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6809/3605394228 conn(0x7f18d80594d0 0x7f18d8118f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18d5ffb700 10 client.37955.objecter ms_handle_connect 0x7f18d80594d0 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.569+0000 7f18dcd72700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6809/3605394228 conn(0x7f18d80594d0 0x7f18d8118f80 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.570+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f18d8028760 con 0x7f18d80594d0 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.570+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.570+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 4 = system:0 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.570+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d8059360 osd=2 4 2024-02-23T00:39:18.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.570+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 2 4 2024-02-23T00:39:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:17.247490+0000 mon.a (mon.0) 2942 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88"}]': finished 2024-02-23T00:39:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:17.247559+0000 mon.a (mon.0) 2943 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2024-02-23T00:39:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:17.507430+0000 mon.c (mon.2) 453 : audit [INF] from='client.? 172.21.15.87:0/3566603870' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:39:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:17.507798+0000 mon.a (mon.0) 2944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:39:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:17.247490+0000 mon.a (mon.0) 2942 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88"}]': finished 2024-02-23T00:39:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:17.247559+0000 mon.a (mon.0) 2943 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2024-02-23T00:39:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:17.507430+0000 mon.c (mon.2) 453 : audit [INF] from='client.? 172.21.15.87:0/3566603870' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:39:18.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:17.507798+0000 mon.a (mon.0) 2944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 10 client.37955.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 5 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8059360 osd=2 3 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8059360 osd=2 3 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8059360 osd=2 3 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 5 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d8059360 osd=2 3 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 2 5 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8059360 osd=2 4 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 10 client.37955.objecter _send_command 5 2024-02-23T00:39:18.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.584+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18d8027d80 con 0x7f18d80594d0 2024-02-23T00:39:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f18d8027d80 con 0x7f18d80594d0 2024-02-23T00:39:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d8059360 osd=2 4 2024-02-23T00:39:18.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 2 5 2024-02-23T00:39:18.589 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:39:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 10 client.37955.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 6 2024-02-23T00:39:18.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6813/128059790 conn(0x7f18d8106da0 0x7f18d81071c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d80fe050 osd=3 3 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d80fe050 osd=3 3 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d80fe050 osd=3 3 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 6 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d80fe050 osd=3 3 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 3 6 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d80fe050 osd=3 4 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 10 client.37955.objecter _send_command 6 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d8108dc0 con 0x7f18d8106da0 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18d7fff700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6813/128059790 conn(0x7f18d8106da0 0x7f18d81071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18d5ffb700 10 client.37955.objecter ms_handle_connect 0x7f18d8106da0 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.586+0000 7f18d7fff700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.87:6813/128059790 conn(0x7f18d8106da0 0x7f18d81071c0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.587+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f18d8108dc0 con 0x7f18d8106da0 2024-02-23T00:39:18.591 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.587+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:39:18.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.587+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 6 = system:0 2024-02-23T00:39:18.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.587+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d80fe050 osd=3 4 2024-02-23T00:39:18.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.587+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 3 6 2024-02-23T00:39:18.592 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:17.247490+0000 mon.a (mon.0) 2942 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-88"}]': finished 2024-02-23T00:39:18.592 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:17.247559+0000 mon.a (mon.0) 2943 : cluster [DBG] osdmap e538: 8 total, 7 up, 8 in 2024-02-23T00:39:18.592 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:17.507430+0000 mon.c (mon.2) 453 : audit [INF] from='client.? 172.21.15.87:0/3566603870' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:39:18.592 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:17.507798+0000 mon.a (mon.0) 2944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:39:18.593 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:18.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 10 client.37955.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:18.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 7 2024-02-23T00:39:18.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d80fe050 osd=3 3 2024-02-23T00:39:18.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d80fe050 osd=3 3 2024-02-23T00:39:18.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d80fe050 osd=3 3 2024-02-23T00:39:18.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 7 2024-02-23T00:39:18.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d80fe050 osd=3 3 2024-02-23T00:39:18.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 3 7 2024-02-23T00:39:18.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d80fe050 osd=3 4 2024-02-23T00:39:18.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 10 client.37955.objecter _send_command 7 2024-02-23T00:39:18.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.601+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18d81076b0 con 0x7f18d8106da0 2024-02-23T00:39:18.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f18d81076b0 con 0x7f18d8106da0 2024-02-23T00:39:18.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:18.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d80fe050 osd=3 4 2024-02-23T00:39:18.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 3 7 2024-02-23T00:39:18.606 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:39:18.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 10 client.37955.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:18.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 8 2024-02-23T00:39:18.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6800/4175844757 conn(0x7f18d8119c40 0x7f18d811a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8108ab0 osd=4 3 2024-02-23T00:39:18.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8108ab0 osd=4 3 2024-02-23T00:39:18.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8108ab0 osd=4 3 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 8 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d8108ab0 osd=4 3 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 4 8 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8108ab0 osd=4 4 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 10 client.37955.objecter _send_command 8 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d8125250 con 0x7f18d8119c40 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.603+0000 7f18dd573700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6800/4175844757 conn(0x7f18d8119c40 0x7f18d811a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.604+0000 7f18d5ffb700 10 client.37955.objecter ms_handle_connect 0x7f18d8119c40 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.604+0000 7f18dd573700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6800/4175844757 conn(0x7f18d8119c40 0x7f18d811a0a0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:18.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.604+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f18d8125250 con 0x7f18d8119c40 2024-02-23T00:39:18.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.604+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:39:18.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.604+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 8 = system:0 2024-02-23T00:39:18.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.604+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d8108ab0 osd=4 4 2024-02-23T00:39:18.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.604+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 4 8 2024-02-23T00:39:18.610 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:18.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:18.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 10 client.37955.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:18.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 9 2024-02-23T00:39:18.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8108ab0 osd=4 3 2024-02-23T00:39:18.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8108ab0 osd=4 3 2024-02-23T00:39:18.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8108ab0 osd=4 3 2024-02-23T00:39:18.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 9 2024-02-23T00:39:18.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d8108ab0 osd=4 3 2024-02-23T00:39:18.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 4 9 2024-02-23T00:39:18.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8108ab0 osd=4 4 2024-02-23T00:39:18.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 10 client.37955.objecter _send_command 9 2024-02-23T00:39:18.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.619+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18d81196b0 con 0x7f18d8119c40 2024-02-23T00:39:18.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f18d81196b0 con 0x7f18d8119c40 2024-02-23T00:39:18.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:18.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d8108ab0 osd=4 4 2024-02-23T00:39:18.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 4 9 2024-02-23T00:39:18.624 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:39:18.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 10 client.37955.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 10 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6804/2550013795 conn(0x7f18d8108530 0x7f18d8123c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d81196b0 osd=5 3 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d81196b0 osd=5 3 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d81196b0 osd=5 3 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 10 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d81196b0 osd=5 3 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 5 10 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d81196b0 osd=5 4 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 10 client.37955.objecter _send_command 10 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d81198e0 con 0x7f18d8108530 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18dcd72700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6804/2550013795 conn(0x7f18d8108530 0x7f18d8123c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18d5ffb700 10 client.37955.objecter ms_handle_connect 0x7f18d8108530 2024-02-23T00:39:18.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.621+0000 7f18dcd72700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6804/2550013795 conn(0x7f18d8108530 0x7f18d8123c40 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:18.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.622+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f18d81198e0 con 0x7f18d8108530 2024-02-23T00:39:18.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.622+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:39:18.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.622+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 10 = system:0 2024-02-23T00:39:18.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.622+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d81196b0 osd=5 4 2024-02-23T00:39:18.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.622+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 5 10 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 10 client.37955.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 11 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d81196b0 osd=5 3 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d81196b0 osd=5 3 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d81196b0 osd=5 3 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 11 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d81196b0 osd=5 3 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 5 11 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d81196b0 osd=5 4 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 10 client.37955.objecter _send_command 11 2024-02-23T00:39:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.636+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18d81198e0 con 0x7f18d8108530 2024-02-23T00:39:18.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f18d81198e0 con 0x7f18d8108530 2024-02-23T00:39:18.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:18.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d81196b0 osd=5 4 2024-02-23T00:39:18.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 5 11 2024-02-23T00:39:18.641 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:39:18.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 10 client.37955.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:18.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 12 2024-02-23T00:39:18.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6808/2509408498 conn(0x7f18d8124640 0x7f18d8124a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d81198e0 osd=6 3 2024-02-23T00:39:18.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d81198e0 osd=6 3 2024-02-23T00:39:18.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d81198e0 osd=6 3 2024-02-23T00:39:18.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 12 2024-02-23T00:39:18.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d81198e0 osd=6 3 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 6 12 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d81198e0 osd=6 4 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 10 client.37955.objecter _send_command 12 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.638+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d8125040 con 0x7f18d8124640 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.639+0000 7f18d7fff700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6808/2509408498 conn(0x7f18d8124640 0x7f18d8124a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.639+0000 7f18d5ffb700 10 client.37955.objecter ms_handle_connect 0x7f18d8124640 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.639+0000 7f18d7fff700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6808/2509408498 conn(0x7f18d8124640 0x7f18d8124a80 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.639+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f18d8125040 con 0x7f18d8124640 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.639+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.639+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 12 = system:0 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.639+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d81198e0 osd=6 4 2024-02-23T00:39:18.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.639+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 6 12 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.653+0000 7f18defd6700 10 client.37955.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.653+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 13 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.653+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d81198e0 osd=6 3 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.653+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d81198e0 osd=6 3 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.653+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d81198e0 osd=6 3 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.653+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 13 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.653+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d81198e0 osd=6 3 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.653+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 6 13 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.653+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d81198e0 osd=6 4 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.654+0000 7f18defd6700 10 client.37955.objecter _send_command 13 2024-02-23T00:39:18.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.654+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18d8125000 con 0x7f18d8124640 2024-02-23T00:39:18.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f18d8125000 con 0x7f18d8124640 2024-02-23T00:39:18.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d81198e0 osd=6 4 2024-02-23T00:39:18.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 6 13 2024-02-23T00:39:18.658 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:39:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 10 client.37955.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 14 2024-02-23T00:39:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6812/373787000 conn(0x7f18d81299a0 0x7f18d8129e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8125000 osd=7 3 2024-02-23T00:39:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8125000 osd=7 3 2024-02-23T00:39:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8125000 osd=7 3 2024-02-23T00:39:18.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 14 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d8125000 osd=7 3 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 7 14 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8125000 osd=7 4 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 10 client.37955.objecter _send_command 14 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.655+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d812ba20 con 0x7f18d81299a0 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.656+0000 7f18dd573700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6812/373787000 conn(0x7f18d81299a0 0x7f18d8129e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.656+0000 7f18dd573700 1 --2- 172.21.15.87:0/2008816249 >> v2:172.21.15.171:6812/373787000 conn(0x7f18d81299a0 0x7f18d8129e30 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.656+0000 7f18d5ffb700 10 client.37955.objecter ms_handle_connect 0x7f18d81299a0 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.657+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f18d812ba20 con 0x7f18d81299a0 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.657+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:39:18.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.657+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 14 = system:0 2024-02-23T00:39:18.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.657+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d8125000 osd=7 4 2024-02-23T00:39:18.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.657+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 7 14 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 10 client.37955.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign -1 15 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8125000 osd=7 3 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8125000 osd=7 3 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 20 client.37955.objecter _get_session s=0x7f18d8125000 osd=7 3 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 15 client.37955.objecter _session_command_op_remove -1 15 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 20 client.37955.objecter get_session s=0x7f18d8125000 osd=7 3 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 15 client.37955.objecter _session_command_op_assign 7 15 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 20 client.37955.objecter put_session s=0x7f18d8125000 osd=7 4 2024-02-23T00:39:18.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 10 client.37955.objecter _send_command 15 2024-02-23T00:39:18.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.671+0000 7f18defd6700 1 -- 172.21.15.87:0/2008816249 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f18d812dc30 con 0x7f18d81299a0 2024-02-23T00:39:18.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.673+0000 7f18d5ffb700 1 -- 172.21.15.87:0/2008816249 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f18d812dc30 con 0x7f18d81299a0 2024-02-23T00:39:18.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.673+0000 7f18d5ffb700 10 client.37955.objecter ms_dispatch 0x7f18d8001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:18.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.673+0000 7f18d5ffb700 10 client.37955.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.673+0000 7f18d5ffb700 20 client.37955.objecter put_session s=0x7f18d8125000 osd=7 4 2024-02-23T00:39:18.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:18.673+0000 7f18d5ffb700 15 client.37955.objecter _session_command_op_remove 7 15 2024-02-23T00:39:18.676 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:39:18.676 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:39:18.676 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.676 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.676 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.677 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.677 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.677 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.677 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:39:18.714 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:18.741 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:18.759 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:18.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:18.863 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:18.889 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:18.907 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:18.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:19.011 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:19.038 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:19.056 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:19.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:19.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:19.160 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:19.187 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:19.336 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:19.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:19.442 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:19.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:17.722878+0000 mgr.y (mgr.24437) 498 : cluster [DBG] pgmap v824: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:19.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:18.245080+0000 mon.a (mon.0) 2945 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:19.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:18.257285+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:39:19.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:18.257345+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2024-02-23T00:39:19.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:18.365979+0000 osd.3 (osd.3) 155 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:39:19.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:18.369576+0000 osd.3 (osd.3) 156 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:39:19.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:18.552216+0000 mon.c (mon.2) 454 : audit [DBG] from='client.? 172.21.15.87:0/2008816249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:19.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:17.722878+0000 mgr.y (mgr.24437) 498 : cluster [DBG] pgmap v824: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:19.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:18.245080+0000 mon.a (mon.0) 2945 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:19.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:18.257285+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:39:19.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:18.257345+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2024-02-23T00:39:19.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:18.365979+0000 osd.3 (osd.3) 155 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:39:19.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:18.369576+0000 osd.3 (osd.3) 156 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:39:19.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:18.552216+0000 mon.c (mon.2) 454 : audit [DBG] from='client.? 172.21.15.87:0/2008816249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:19.467 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:17.722878+0000 mgr.y (mgr.24437) 498 : cluster [DBG] pgmap v824: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:18.245080+0000 mon.a (mon.0) 2945 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:18.257285+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:39:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:18.257345+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e539: 8 total, 7 up, 8 in 2024-02-23T00:39:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:18.365979+0000 osd.3 (osd.3) 155 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:39:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:18.369576+0000 osd.3 (osd.3) 156 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:39:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:18.552216+0000 mon.c (mon.2) 454 : audit [DBG] from='client.? 172.21.15.87:0/2008816249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:20.015 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:20.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:20.120 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:20.151 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:20.172 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:20.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:20.276 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:20.320 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:20.345 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:20.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:20.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:19.262949+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2024-02-23T00:39:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:19.272521+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:20.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:19.272800+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:20.450 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:20.479 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:20.501 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:20.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:19.262949+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2024-02-23T00:39:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:19.272521+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:19.272800+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:20.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:19.262949+0000 mon.a (mon.0) 2948 : cluster [DBG] osdmap e540: 8 total, 7 up, 8 in 2024-02-23T00:39:20.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:19.272521+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:20.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:19.272800+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:20.605 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:20.636 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:20.636 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:39:20.636 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:39:20.653 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:20.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:20.757 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:20.784 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:20.802 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:20.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:20.906 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:20.934 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:21.007 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:21.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:21.111 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:21.137 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:21.289 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:19.723504+0000 mgr.y (mgr.24437) 499 : cluster [DBG] pgmap v827: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:21.289 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:20.267945+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:21.289 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:20.268054+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2024-02-23T00:39:21.289 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:20.283949+0000 mon.b (mon.1) 439 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:21.289 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:20.284219+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:21.290 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:20.395823+0000 osd.2 (osd.2) 203 : cluster [DBG] 5.a scrub starts 2024-02-23T00:39:21.290 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:20.396808+0000 osd.2 (osd.2) 204 : cluster [DBG] 5.a scrub ok 2024-02-23T00:39:21.290 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:20.897143+0000 osd.7 (osd.7) 260 : cluster [DBG] 63.b scrub starts 2024-02-23T00:39:21.290 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:20.899032+0000 osd.7 (osd.7) 261 : cluster [DBG] 63.b scrub ok 2024-02-23T00:39:21.359 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:39:21.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:39:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:19.723504+0000 mgr.y (mgr.24437) 499 : cluster [DBG] pgmap v827: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:21.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:20.267945+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:20.268054+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2024-02-23T00:39:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:20.283949+0000 mon.b (mon.1) 439 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:20.284219+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:20.395823+0000 osd.2 (osd.2) 203 : cluster [DBG] 5.a scrub starts 2024-02-23T00:39:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:20.396808+0000 osd.2 (osd.2) 204 : cluster [DBG] 5.a scrub ok 2024-02-23T00:39:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:20.897143+0000 osd.7 (osd.7) 260 : cluster [DBG] 63.b scrub starts 2024-02-23T00:39:21.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:20.899032+0000 osd.7 (osd.7) 261 : cluster [DBG] 63.b scrub ok 2024-02-23T00:39:21.518 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-23T00:39:21.542 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:19.723504+0000 mgr.y (mgr.24437) 499 : cluster [DBG] pgmap v827: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 6 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 135 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 104/1269 objects degraded (8.195%); 114/1269 objects misplaced (8.983%) 2024-02-23T00:39:21.542 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:20.267945+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:21.542 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:20.268054+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e541: 8 total, 7 up, 8 in 2024-02-23T00:39:21.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:20.283949+0000 mon.b (mon.1) 439 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:21.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:20.284219+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:39:21.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:20.395823+0000 osd.2 (osd.2) 203 : cluster [DBG] 5.a scrub starts 2024-02-23T00:39:21.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:20.396808+0000 osd.2 (osd.2) 204 : cluster [DBG] 5.a scrub ok 2024-02-23T00:39:21.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:20.897143+0000 osd.7 (osd.7) 260 : cluster [DBG] 63.b scrub starts 2024-02-23T00:39:21.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:20.899032+0000 osd.7 (osd.7) 261 : cluster [DBG] 63.b scrub ok 2024-02-23T00:39:21.590 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:21.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:21.599 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:21.694 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:21.720 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:21.740 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:21.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:21.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:21.844 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:21.870 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:21.888 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:21.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:21.992 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:22.019 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:22.071 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:22.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:39:22.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:39:22.111 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:22.139 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:22.175 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:22.202 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:22.222 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:22.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:22.301 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:22] "GET /metrics HTTP/1.1" 200 189326 "" "Prometheus/2.33.4" 2024-02-23T00:39:22.326 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:22.353 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:22.371 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:22.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:22.476 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:22.503 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:22.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:21.068887+0000 mgr.y (mgr.24437) 500 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:22.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:21.272200+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:39:22.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:21.272282+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e542: 8 total, 7 up, 8 in 2024-02-23T00:39:22.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:21.278549+0000 mon.b (mon.1) 440 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:22.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:21.282517+0000 mon.a (mon.0) 2955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:21.367742+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:21.370906+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:21.438677+0000 osd.1 (osd.1) 251 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:21.439774+0000 osd.1 (osd.1) 252 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:22.146200+0000 mon.a (mon.0) 2956 : cluster [WRN] Health check update: Degraded data redundancy: 104/1269 objects degraded (8.195%), 23 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:22.154695+0000 mon.a (mon.0) 2957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-90"}]': finished 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:22.154773+0000 mon.a (mon.0) 2958 : cluster [DBG] osdmap e543: 8 total, 7 up, 8 in 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:22.158108+0000 mon.b (mon.1) 441 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-90", "mode": "writeback"}]: dispatch 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:22.159084+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-90", "mode": "writeback"}]: dispatch 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:21.068887+0000 mgr.y (mgr.24437) 500 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:21.272200+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:21.272282+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e542: 8 total, 7 up, 8 in 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:21.278549+0000 mon.b (mon.1) 440 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:21.282517+0000 mon.a (mon.0) 2955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:21.367742+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:21.370906+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:21.438677+0000 osd.1 (osd.1) 251 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:21.439774+0000 osd.1 (osd.1) 252 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:22.146200+0000 mon.a (mon.0) 2956 : cluster [WRN] Health check update: Degraded data redundancy: 104/1269 objects degraded (8.195%), 23 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:22.154695+0000 mon.a (mon.0) 2957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-90"}]': finished 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:22.154773+0000 mon.a (mon.0) 2958 : cluster [DBG] osdmap e543: 8 total, 7 up, 8 in 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:22.158108+0000 mon.b (mon.1) 441 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-90", "mode": "writeback"}]: dispatch 2024-02-23T00:39:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:22.159084+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-90", "mode": "writeback"}]: dispatch 2024-02-23T00:39:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:21.068887+0000 mgr.y (mgr.24437) 500 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:21.272200+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:39:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:21.272282+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e542: 8 total, 7 up, 8 in 2024-02-23T00:39:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:21.278549+0000 mon.b (mon.1) 440 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:21.282517+0000 mon.a (mon.0) 2955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:21.367742+0000 osd.3 (osd.3) 157 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:39:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:21.370906+0000 osd.3 (osd.3) 158 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:39:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:21.438677+0000 osd.1 (osd.1) 251 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:39:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:21.439774+0000 osd.1 (osd.1) 252 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:39:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:22.146200+0000 mon.a (mon.0) 2956 : cluster [WRN] Health check update: Degraded data redundancy: 104/1269 objects degraded (8.195%), 23 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:22.154695+0000 mon.a (mon.0) 2957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-6", "overlaypool": "test-rados-api-smithi087-82091-90"}]': finished 2024-02-23T00:39:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:22.154773+0000 mon.a (mon.0) 2958 : cluster [DBG] osdmap e543: 8 total, 7 up, 8 in 2024-02-23T00:39:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:22.158108+0000 mon.b (mon.1) 441 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-90", "mode": "writeback"}]: dispatch 2024-02-23T00:39:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:22.159084+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-90", "mode": "writeback"}]: dispatch 2024-02-23T00:39:22.852 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:22.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:22.956 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:22.984 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:23.003 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:23.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:23.108 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:23.136 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:23.577 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:21.724405+0000 mgr.y (mgr.24437) 501 : cluster [DBG] pgmap v830: 246 pgs: 1 peering, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 4 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 86/1269 objects misplaced (6.777%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:23.577 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:22.278324+0000 mon.a (mon.0) 2960 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:22.383090+0000 osd.6 (osd.6) 220 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:22.387082+0000 osd.6 (osd.6) 221 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:22.417445+0000 osd.1 (osd.1) 253 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:22.420376+0000 osd.1 (osd.1) 254 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:22.436329+0000 osd.2 (osd.2) 205 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:22.437099+0000 osd.2 (osd.2) 206 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:23.149512+0000 mon.a (mon.0) 2961 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:23.154224+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-90", "mode": "writeback"}]': finished 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:23.154313+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e544: 8 total, 7 up, 8 in 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:23.164099+0000 mon.b (mon.1) 442 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:39:23.578 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:23.164854+0000 mon.a (mon.0) 2964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:21.724405+0000 mgr.y (mgr.24437) 501 : cluster [DBG] pgmap v830: 246 pgs: 1 peering, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 4 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 86/1269 objects misplaced (6.777%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:22.278324+0000 mon.a (mon.0) 2960 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:22.383090+0000 osd.6 (osd.6) 220 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:22.387082+0000 osd.6 (osd.6) 221 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:22.417445+0000 osd.1 (osd.1) 253 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:22.420376+0000 osd.1 (osd.1) 254 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:22.436329+0000 osd.2 (osd.2) 205 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:22.437099+0000 osd.2 (osd.2) 206 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:23.149512+0000 mon.a (mon.0) 2961 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:39:23.579 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:23.154224+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-90", "mode": "writeback"}]': finished 2024-02-23T00:39:23.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:23.154313+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e544: 8 total, 7 up, 8 in 2024-02-23T00:39:23.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:23.164099+0000 mon.b (mon.1) 442 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:39:23.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:23.164854+0000 mon.a (mon.0) 2964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:39:23.586 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:23.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:23.690 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:21.724405+0000 mgr.y (mgr.24437) 501 : cluster [DBG] pgmap v830: 246 pgs: 1 peering, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 4 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 86/1269 objects misplaced (6.777%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:22.278324+0000 mon.a (mon.0) 2960 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:22.383090+0000 osd.6 (osd.6) 220 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:39:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:22.387082+0000 osd.6 (osd.6) 221 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:39:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:22.417445+0000 osd.1 (osd.1) 253 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:39:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:22.420376+0000 osd.1 (osd.1) 254 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:39:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:22.436329+0000 osd.2 (osd.2) 205 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:39:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:22.437099+0000 osd.2 (osd.2) 206 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:39:23.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:23.149512+0000 mon.a (mon.0) 2961 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:39:23.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:23.154224+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-90", "mode": "writeback"}]': finished 2024-02-23T00:39:23.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:23.154313+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e544: 8 total, 7 up, 8 in 2024-02-23T00:39:23.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:23.164099+0000 mon.b (mon.1) 442 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:39:23.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:23.164854+0000 mon.a (mon.0) 2964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:39:23.718 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:23.965 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:23.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:23.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:24.069 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:24.099 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:39:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:39:24.101 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:39:24.101 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:24.123 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:24.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:24.227 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:24.272 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:25.024 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:25.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:25.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:25.128 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:25.156 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:25.195 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:25.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:25.318 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:23.725040+0000 mgr.y (mgr.24437) 502 : cluster [DBG] pgmap v833: 246 pgs: 1 peering, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 4 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 86/1269 objects misplaced (6.777%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:23.832236+0000 osd.5 (osd.5) 211 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:39:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:23.833054+0000 osd.5 (osd.5) 212 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:39:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:24.160741+0000 mon.a (mon.0) 2965 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:39:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:24.160838+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e545: 8 total, 7 up, 8 in 2024-02-23T00:39:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:24.164517+0000 mon.b (mon.1) 443 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:24.169606+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:24.654162+0000 mon.a (mon.0) 2968 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:23.725040+0000 mgr.y (mgr.24437) 502 : cluster [DBG] pgmap v833: 246 pgs: 1 peering, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 4 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 86/1269 objects misplaced (6.777%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:23.832236+0000 osd.5 (osd.5) 211 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:39:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:23.833054+0000 osd.5 (osd.5) 212 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:39:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:24.160741+0000 mon.a (mon.0) 2965 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:39:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:24.160838+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e545: 8 total, 7 up, 8 in 2024-02-23T00:39:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:24.164517+0000 mon.b (mon.1) 443 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:24.169606+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:25.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:24.654162+0000 mon.a (mon.0) 2968 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:25.344 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:25.366 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:25.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:23.725040+0000 mgr.y (mgr.24437) 502 : cluster [DBG] pgmap v833: 246 pgs: 1 peering, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 4 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 155 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 86/1269 objects misplaced (6.777%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:23.832236+0000 osd.5 (osd.5) 211 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:39:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:23.833054+0000 osd.5 (osd.5) 212 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:39:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:24.160741+0000 mon.a (mon.0) 2965 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:39:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:24.160838+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e545: 8 total, 7 up, 8 in 2024-02-23T00:39:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:24.164517+0000 mon.b (mon.1) 443 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:24.169606+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:39:25.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:24.654162+0000 mon.a (mon.0) 2968 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:25.471 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:25.498 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:25.517 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:25.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:25.620 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:25.647 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:25.665 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:25.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:25.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:25.796 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:26.430 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:24.423924+0000 osd.4 (osd.4) 249 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:24.424934+0000 osd.4 (osd.4) 250 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:25.156403+0000 mon.a (mon.0) 2969 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:25.156483+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e546: 8 total, 7 up, 8 in 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:25.158740+0000 mon.b (mon.1) 444 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:25.170430+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:25.308996+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.11 scrub starts 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:25.311945+0000 osd.3 (osd.3) 160 : cluster [DBG] 6.11 scrub ok 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:24.423924+0000 osd.4 (osd.4) 249 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:24.424934+0000 osd.4 (osd.4) 250 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:39:26.431 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:25.156403+0000 mon.a (mon.0) 2969 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:39:26.432 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:25.156483+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e546: 8 total, 7 up, 8 in 2024-02-23T00:39:26.432 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:25.158740+0000 mon.b (mon.1) 444 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:39:26.432 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:25.170430+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:39:26.432 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:25.308996+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.11 scrub starts 2024-02-23T00:39:26.432 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:25.311945+0000 osd.3 (osd.3) 160 : cluster [DBG] 6.11 scrub ok 2024-02-23T00:39:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:24.423924+0000 osd.4 (osd.4) 249 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:39:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:24.424934+0000 osd.4 (osd.4) 250 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:39:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:25.156403+0000 mon.a (mon.0) 2969 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:39:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:25.156483+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e546: 8 total, 7 up, 8 in 2024-02-23T00:39:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:25.158740+0000 mon.b (mon.1) 444 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:39:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:25.170430+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:39:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:25.308996+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.11 scrub starts 2024-02-23T00:39:26.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:25.311945+0000 osd.3 (osd.3) 160 : cluster [DBG] 6.11 scrub ok 2024-02-23T00:39:26.446 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:26.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:26.551 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:26.578 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:26.865 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:39:26.865 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:39:27.331 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:27.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:27.437 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:25.726234+0000 mgr.y (mgr.24437) 503 : cluster [DBG] pgmap v836: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 158 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 74/1269 objects misplaced (5.831%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:26.164471+0000 mon.a (mon.0) 2972 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:39:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:26.177285+0000 mon.a (mon.0) 2973 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:39:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:26.177351+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e547: 8 total, 7 up, 8 in 2024-02-23T00:39:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:26.185076+0000 mon.b (mon.1) 445 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-23T00:39:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:26.185539+0000 mon.a (mon.0) 2975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-23T00:39:27.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:26.453167+0000 osd.6 (osd.6) 222 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:39:27.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:26.456071+0000 osd.6 (osd.6) 223 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:39:27.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:25.726234+0000 mgr.y (mgr.24437) 503 : cluster [DBG] pgmap v836: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 158 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 74/1269 objects misplaced (5.831%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:27.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:26.164471+0000 mon.a (mon.0) 2972 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:39:27.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:26.177285+0000 mon.a (mon.0) 2973 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:39:27.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:26.177351+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e547: 8 total, 7 up, 8 in 2024-02-23T00:39:27.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:26.185076+0000 mon.b (mon.1) 445 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-23T00:39:27.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:26.185539+0000 mon.a (mon.0) 2975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-23T00:39:27.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:26.453167+0000 osd.6 (osd.6) 222 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:39:27.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:26.456071+0000 osd.6 (osd.6) 223 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:39:27.464 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:25.726234+0000 mgr.y (mgr.24437) 503 : cluster [DBG] pgmap v836: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 158 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 74/1269 objects misplaced (5.831%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:27.464 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:26.164471+0000 mon.a (mon.0) 2972 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:39:27.464 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:26.177285+0000 mon.a (mon.0) 2973 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:39:27.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:26.177351+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e547: 8 total, 7 up, 8 in 2024-02-23T00:39:27.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:26.185076+0000 mon.b (mon.1) 445 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-23T00:39:27.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:26.185539+0000 mon.a (mon.0) 2975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-23T00:39:27.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:26.453167+0000 osd.6 (osd.6) 222 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:39:27.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:26.456071+0000 osd.6 (osd.6) 223 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:39:27.465 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:27.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.766+0000 7f8037e82700 1 -- 172.21.15.87:0/3947010967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f80300fd510 msgr2=0x7f80300f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:27.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.766+0000 7f8037e82700 1 --2- 172.21.15.87:0/3947010967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f80300fd510 0x7f80300f6d40 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 rx=0x7f802c004870 tx=0x7f802c009cb0).stop 2024-02-23T00:39:27.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.766+0000 7f8037e82700 1 -- 172.21.15.87:0/3947010967 shutdown_connections 2024-02-23T00:39:27.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.766+0000 7f8037e82700 1 --2- 172.21.15.87:0/3947010967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80300f7400 0x7f8030103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:27.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.766+0000 7f8037e82700 1 --2- 172.21.15.87:0/3947010967 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80300fcbb0 0x7f80300fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.766+0000 7f8037e82700 1 --2- 172.21.15.87:0/3947010967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f80300fd510 0x7f80300f6d40 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.766+0000 7f8037e82700 1 -- 172.21.15.87:0/3947010967 >> 172.21.15.87:0/3947010967 conn(0x7f80300f2820 msgr2=0x7f80300f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 -- 172.21.15.87:0/3947010967 shutdown_connections 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 -- 172.21.15.87:0/3947010967 wait complete. 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 librados: starting msgr at 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 librados: starting objecter 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 Processor -- start 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 -- start start 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 librados: setting wanted keys 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 librados: calling monclient init 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f80300f7400 0x7f803010d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80300fcbb0 0x7f803010d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:27.779 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80300fd510 0x7f8030107210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8030107700 con 0x7f80300fcbb0 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8030107870 con 0x7f80300fd510 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8037e82700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8030107a30 con 0x7f80300f7400 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f803541d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80300fcbb0 0x7f803010d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8035c1e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f80300f7400 0x7f803010d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f803541d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80300fcbb0 0x7f803010d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51922/0 (socket says 172.21.15.87:51922) 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f8035c1e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f80300f7400 0x7f803010d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:49504/0 (socket says 172.21.15.87:49504) 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.767+0000 7f803541d700 1 -- 172.21.15.87:0/2444109997 learned_addr learned my addr 172.21.15.87:0/2444109997 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f803541d700 1 -- 172.21.15.87:0/2444109997 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f80300f7400 msgr2=0x7f803010d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f803641f700 1 --2- 172.21.15.87:0/2444109997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80300fd510 0x7f8030107210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f803541d700 1 --2- 172.21.15.87:0/2444109997 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f80300f7400 0x7f803010d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f803541d700 1 -- 172.21.15.87:0/2444109997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80300fd510 msgr2=0x7f8030107210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:27.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f803541d700 1 --2- 172.21.15.87:0/2444109997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80300fd510 0x7f8030107210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f803541d700 1 -- 172.21.15.87:0/2444109997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80280097c0 con 0x7f80300fcbb0 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f803641f700 1 --2- 172.21.15.87:0/2444109997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80300fd510 0x7f8030107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f803541d700 1 --2- 172.21.15.87:0/2444109997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80300fcbb0 0x7f803010d560 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7f802c009eb0 tx=0x7f802c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f8026ffd700 10 client.?.objecter ms_handle_connect 0x7f80300fcbb0 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f8026ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f8026ffd700 1 -- 172.21.15.87:0/2444109997 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c00e070 con 0x7f80300fcbb0 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c005040 con 0x7f80300fcbb0 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f8026ffd700 1 -- 172.21.15.87:0/2444109997 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f802c013da0 con 0x7f80300fcbb0 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f8026ffd700 1 -- 172.21.15.87:0/2444109997 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c019070 con 0x7f80300fcbb0 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f8037e82700 10 client.27495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8030107ed0 con 0x7f80300fcbb0 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.768+0000 7f8037e82700 1 librados: init done 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.769+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80301080c0 con 0x7f80300fcbb0 2024-02-23T00:39:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.769+0000 7f8026ffd700 1 -- 172.21.15.87:0/2444109997 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f802c01e420 con 0x7f80300fcbb0 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.770+0000 7f8026ffd700 1 --2- 172.21.15.87:0/2444109997 >> v2:172.21.15.87:6800/1231597463 conn(0x7f801c062c90 0x7f801c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.770+0000 7f8026ffd700 1 -- 172.21.15.87:0/2444109997 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(548..548 src has 173..548) v4 ==== 8979+0+0 (secure 0 0 0) 0x7f802c07e1c0 con 0x7f80300fcbb0 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.770+0000 7f8026ffd700 10 client.27495.objecter ms_dispatch 0x7f8030001550 osd_map(548..548 src has 173..548) v4 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.770+0000 7f8026ffd700 3 client.27495.objecter handle_osd_map got epochs [548,548] > 0 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.770+0000 7f8026ffd700 3 client.27495.objecter handle_osd_map decoding full epoch 548 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.770+0000 7f8035c1e700 1 --2- 172.21.15.87:0/2444109997 >> v2:172.21.15.87:6800/1231597463 conn(0x7f801c062c90 0x7f801c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.770+0000 7f8026ffd700 20 client.27495.objecter dump_active .. 0 homeless 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.770+0000 7f8026ffd700 10 client.27495.objecter ms_handle_connect 0x7f801c062c90 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.770+0000 7f8035c1e700 1 --2- 172.21.15.87:0/2444109997 >> v2:172.21.15.87:6800/1231597463 conn(0x7f801c062c90 0x7f801c065150 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f80200041d0 tx=0x7f8020009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:27.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.774+0000 7f8026ffd700 1 -- 172.21.15.87:0/2444109997 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f802c0512f0 con 0x7f80300fcbb0 2024-02-23T00:39:27.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:27.951+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f8030106120 con 0x7f80300fcbb0 2024-02-23T00:39:28.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.190+0000 7f8026ffd700 1 -- 172.21.15.87:0/2444109997 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v549) v1 ==== 101+0+0 (secure 0 0 0) 0x7f802c023030 con 0x7f80300fcbb0 2024-02-23T00:39:28.207 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.204+0000 7f8037e82700 10 librados: watch_flush enter 2024-02-23T00:39:28.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.206+0000 7f8037e82700 10 librados: watch_flush exit 2024-02-23T00:39:28.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.206+0000 7f8037e82700 20 client.27495.objecter shutdown clearing up homeless session... 2024-02-23T00:39:28.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.206+0000 7f8037e82700 10 client.27495.objecter successfully canceled tick 2024-02-23T00:39:28.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.206+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 >> v2:172.21.15.87:6800/1231597463 conn(0x7f801c062c90 msgr2=0x7f801c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:28.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.206+0000 7f8037e82700 1 --2- 172.21.15.87:0/2444109997 >> v2:172.21.15.87:6800/1231597463 conn(0x7f801c062c90 0x7f801c065150 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 rx=0x7f80200041d0 tx=0x7f8020009040).stop 2024-02-23T00:39:28.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.206+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80300fcbb0 msgr2=0x7f803010d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:28.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.206+0000 7f8037e82700 1 --2- 172.21.15.87:0/2444109997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80300fcbb0 0x7f803010d560 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 rx=0x7f802c009eb0 tx=0x7f802c00b040).stop 2024-02-23T00:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.213+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 shutdown_connections 2024-02-23T00:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.213+0000 7f8037e82700 1 --2- 172.21.15.87:0/2444109997 >> v2:172.21.15.87:6800/1231597463 conn(0x7f801c062c90 0x7f801c065150 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.213+0000 7f8037e82700 1 --2- 172.21.15.87:0/2444109997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f80300fd510 0x7f8030107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.213+0000 7f8037e82700 1 --2- 172.21.15.87:0/2444109997 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f80300f7400 0x7f803010d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.213+0000 7f8037e82700 1 --2- 172.21.15.87:0/2444109997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f80300fcbb0 0x7f803010d560 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:28.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.213+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 >> 172.21.15.87:0/2444109997 conn(0x7f80300f2820 msgr2=0x7f80300f4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:28.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.222+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 shutdown_connections 2024-02-23T00:39:28.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.222+0000 7f8037e82700 1 -- 172.21.15.87:0/2444109997 wait complete. 2024-02-23T00:39:28.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.223+0000 7f8037e82700 1 librados: shutdown 2024-02-23T00:39:28.238 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:39:28.389 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-23T00:39:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:26.482990+0000 osd.4 (osd.4) 251 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:39:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:26.483921+0000 osd.4 (osd.4) 252 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:39:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:27.182619+0000 mon.a (mon.0) 2976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-23T00:39:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:27.182693+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e548: 8 total, 7 up, 8 in 2024-02-23T00:39:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:27.278137+0000 mon.b (mon.1) 446 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:27.278613+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:27.412519+0000 osd.2 (osd.2) 207 : cluster [DBG] 6.f scrub starts 2024-02-23T00:39:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:27.413541+0000 osd.2 (osd.2) 208 : cluster [DBG] 6.f scrub ok 2024-02-23T00:39:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:27.953540+0000 mon.a (mon.0) 2979 : audit [INF] from='client.? 172.21.15.87:0/2444109997' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:28.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:26.482990+0000 osd.4 (osd.4) 251 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:39:28.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:26.483921+0000 osd.4 (osd.4) 252 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:39:28.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:27.182619+0000 mon.a (mon.0) 2976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-23T00:39:28.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:27.182693+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e548: 8 total, 7 up, 8 in 2024-02-23T00:39:28.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:27.278137+0000 mon.b (mon.1) 446 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:28.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:27.278613+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:28.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:27.412519+0000 osd.2 (osd.2) 207 : cluster [DBG] 6.f scrub starts 2024-02-23T00:39:28.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:27.413541+0000 osd.2 (osd.2) 208 : cluster [DBG] 6.f scrub ok 2024-02-23T00:39:28.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:27.953540+0000 mon.a (mon.0) 2979 : audit [INF] from='client.? 172.21.15.87:0/2444109997' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:28.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:26.482990+0000 osd.4 (osd.4) 251 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:39:28.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:26.483921+0000 osd.4 (osd.4) 252 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:39:28.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:27.182619+0000 mon.a (mon.0) 2976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-23T00:39:28.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:27.182693+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e548: 8 total, 7 up, 8 in 2024-02-23T00:39:28.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:27.278137+0000 mon.b (mon.1) 446 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:28.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:27.278613+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:28.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:27.412519+0000 osd.2 (osd.2) 207 : cluster [DBG] 6.f scrub starts 2024-02-23T00:39:28.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:27.413541+0000 osd.2 (osd.2) 208 : cluster [DBG] 6.f scrub ok 2024-02-23T00:39:28.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:27.953540+0000 mon.a (mon.0) 2979 : audit [INF] from='client.? 172.21.15.87:0/2444109997' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:28.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.848+0000 7ff02792c700 1 -- 172.21.15.87:0/4164051979 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 msgr2=0x7ff0200fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:28.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.848+0000 7ff02792c700 1 --2- 172.21.15.87:0/4164051979 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 0x7ff0200fcf90 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 rx=0x7ff010004870 tx=0x7ff010009cb0).stop 2024-02-23T00:39:28.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.848+0000 7ff02792c700 1 -- 172.21.15.87:0/4164051979 shutdown_connections 2024-02-23T00:39:28.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.848+0000 7ff02792c700 1 --2- 172.21.15.87:0/4164051979 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0200f7400 0x7ff020103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:28.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.848+0000 7ff02792c700 1 --2- 172.21.15.87:0/4164051979 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 0x7ff0200fcf90 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:28.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.848+0000 7ff02792c700 1 --2- 172.21.15.87:0/4164051979 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0200fd510 0x7ff0200f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:28.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.848+0000 7ff02792c700 1 -- 172.21.15.87:0/4164051979 >> 172.21.15.87:0/4164051979 conn(0x7ff0200f2820 msgr2=0x7ff0200f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:28.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.848+0000 7ff02792c700 1 -- 172.21.15.87:0/4164051979 shutdown_connections 2024-02-23T00:39:28.960 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.848+0000 7ff02792c700 1 -- 172.21.15.87:0/4164051979 wait complete. 2024-02-23T00:39:28.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 librados: starting msgr at 2024-02-23T00:39:28.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 librados: starting objecter 2024-02-23T00:39:28.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 Processor -- start 2024-02-23T00:39:28.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 -- start start 2024-02-23T00:39:28.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 librados: setting wanted keys 2024-02-23T00:39:28.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 librados: calling monclient init 2024-02-23T00:39:28.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0200f7400 0x7ff0200f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:28.961 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 0x7ff0200f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:28.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0200fd510 0x7ff0200f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:28.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff020116770 con 0x7ff0200fd510 2024-02-23T00:39:28.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0201168e0 con 0x7ff0200f7400 2024-02-23T00:39:28.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff02792c700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff020116aa0 con 0x7ff0200fcbb0 2024-02-23T00:39:28.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff024ec7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 0x7ff0200f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:28.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff025ec9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0200fd510 0x7ff0200f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:28.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff024ec7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 0x7ff0200f8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:49522/0 (socket says 172.21.15.87:49522) 2024-02-23T00:39:28.962 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff024ec7700 1 -- 172.21.15.87:0/3641120378 learned_addr learned my addr 172.21.15.87:0/3641120378 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:28.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff024ec7700 1 -- 172.21.15.87:0/3641120378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0200f7400 msgr2=0x7ff0200f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:28.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff0256c8700 1 --2- 172.21.15.87:0/3641120378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0200f7400 0x7ff0200f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:28.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff024ec7700 1 --2- 172.21.15.87:0/3641120378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0200f7400 0x7ff0200f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:28.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff024ec7700 1 -- 172.21.15.87:0/3641120378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0200fd510 msgr2=0x7ff0200f9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:28.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff024ec7700 1 --2- 172.21.15.87:0/3641120378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0200fd510 0x7ff0200f9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:28.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff024ec7700 1 -- 172.21.15.87:0/3641120378 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c005040 con 0x7ff0200fcbb0 2024-02-23T00:39:28.963 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.849+0000 7ff025ec9700 1 --2- 172.21.15.87:0/3641120378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0200fd510 0x7ff0200f9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:28.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff024ec7700 1 --2- 172.21.15.87:0/3641120378 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 0x7ff0200f8ea0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7ff01c0048f0 tx=0x7ff01c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:28.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff0167fc700 10 client.?.objecter ms_handle_connect 0x7ff0200fcbb0 2024-02-23T00:39:28.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff0167fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:28.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff0167fc700 1 -- 172.21.15.87:0/3641120378 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c007b00 con 0x7ff0200fcbb0 2024-02-23T00:39:28.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff0167fc700 1 -- 172.21.15.87:0/3641120378 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff01c00f040 con 0x7ff0200fcbb0 2024-02-23T00:39:28.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff010005040 con 0x7ff0200fcbb0 2024-02-23T00:39:28.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff0167fc700 1 -- 172.21.15.87:0/3641120378 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c014a00 con 0x7ff0200fcbb0 2024-02-23T00:39:28.964 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff02792c700 10 client.37967.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:28.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0201170e0 con 0x7ff0200fcbb0 2024-02-23T00:39:28.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.850+0000 7ff02792c700 1 librados: init done 2024-02-23T00:39:28.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.851+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff02005fa60 con 0x7ff0200fcbb0 2024-02-23T00:39:28.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.851+0000 7ff0167fc700 1 -- 172.21.15.87:0/3641120378 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff01c00e580 con 0x7ff0200fcbb0 2024-02-23T00:39:28.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.851+0000 7ff0167fc700 1 --2- 172.21.15.87:0/3641120378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff00c062c90 0x7ff00c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:28.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.852+0000 7ff0167fc700 1 -- 172.21.15.87:0/3641120378 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(549..549 src has 173..549) v4 ==== 8979+0+0 (secure 0 0 0) 0x7ff01c07d230 con 0x7ff0200fcbb0 2024-02-23T00:39:28.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.852+0000 7ff0167fc700 10 client.37967.objecter ms_dispatch 0x7ff020001550 osd_map(549..549 src has 173..549) v4 2024-02-23T00:39:28.965 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.852+0000 7ff0167fc700 3 client.37967.objecter handle_osd_map got epochs [549,549] > 0 2024-02-23T00:39:28.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.852+0000 7ff0167fc700 3 client.37967.objecter handle_osd_map decoding full epoch 549 2024-02-23T00:39:28.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.852+0000 7ff0256c8700 1 --2- 172.21.15.87:0/3641120378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff00c062c90 0x7ff00c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:28.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.852+0000 7ff0167fc700 20 client.37967.objecter dump_active .. 0 homeless 2024-02-23T00:39:28.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.852+0000 7ff0167fc700 10 client.37967.objecter ms_handle_connect 0x7ff00c062c90 2024-02-23T00:39:28.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.852+0000 7ff0256c8700 1 --2- 172.21.15.87:0/3641120378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff00c062c90 0x7ff00c065150 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7ff010009a00 tx=0x7ff01000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:28.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:28.856+0000 7ff0167fc700 1 -- 172.21.15.87:0/3641120378 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff01c049f60 con 0x7ff0200fcbb0 2024-02-23T00:39:29.022 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.020+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff020106150 con 0x7ff00c062c90 2024-02-23T00:39:29.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.037+0000 7ff0167fc700 1 -- 172.21.15.87:0/3641120378 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+623258 (secure 0 0 0) 0x7ff020106150 con 0x7ff00c062c90 2024-02-23T00:39:29.048 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.040+0000 7ff02792c700 10 librados: watch_flush enter 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 10 librados: watch_flush exit 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 20 client.37967.objecter shutdown clearing up homeless session... 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 10 client.37967.objecter successfully canceled tick 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff00c062c90 msgr2=0x7ff00c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 --2- 172.21.15.87:0/3641120378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff00c062c90 0x7ff00c065150 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 rx=0x7ff010009a00 tx=0x7ff01000b040).stop 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 msgr2=0x7ff0200f8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 --2- 172.21.15.87:0/3641120378 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 0x7ff0200f8ea0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7ff01c0048f0 tx=0x7ff01c009cb0).stop 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 shutdown_connections 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 --2- 172.21.15.87:0/3641120378 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff00c062c90 0x7ff00c065150 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 --2- 172.21.15.87:0/3641120378 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0200fd510 0x7ff0200f9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:29.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 --2- 172.21.15.87:0/3641120378 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0200f7400 0x7ff0200f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:29.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 --2- 172.21.15.87:0/3641120378 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0200fcbb0 0x7ff0200f8ea0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:29.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 >> 172.21.15.87:0/3641120378 conn(0x7ff0200f2820 msgr2=0x7ff0200f4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:29.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 shutdown_connections 2024-02-23T00:39:29.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 -- 172.21.15.87:0/3641120378 wait complete. 2024-02-23T00:39:29.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:29.041+0000 7ff02792c700 1 librados: shutdown 2024-02-23T00:39:29.056 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:39:29.418 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:29.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:27.726863+0000 mgr.y (mgr.24437) 504 : cluster [DBG] pgmap v839: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 158 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 74/1269 objects misplaced (5.831%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:27.810101+0000 osd.7 (osd.7) 262 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:39:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:27.813000+0000 osd.7 (osd.7) 263 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:39:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:28.180425+0000 mon.a (mon.0) 2980 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:39:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:28.190838+0000 mon.a (mon.0) 2981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:39:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:28.190900+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? 172.21.15.87:0/2444109997' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:39:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:28.190946+0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e549: 8 total, 7 up, 8 in 2024-02-23T00:39:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:28.193680+0000 mon.b (mon.1) 447 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:28.193933+0000 mon.a (mon.0) 2984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:28.320188+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:39:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:28.324175+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:39:29.522 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:29.549 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:29.568 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:29.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:27.726863+0000 mgr.y (mgr.24437) 504 : cluster [DBG] pgmap v839: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 158 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 74/1269 objects misplaced (5.831%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:27.810101+0000 osd.7 (osd.7) 262 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:27.813000+0000 osd.7 (osd.7) 263 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:28.180425+0000 mon.a (mon.0) 2980 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:28.190838+0000 mon.a (mon.0) 2981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:28.190900+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? 172.21.15.87:0/2444109997' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:28.190946+0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e549: 8 total, 7 up, 8 in 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:28.193680+0000 mon.b (mon.1) 447 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:28.193933+0000 mon.a (mon.0) 2984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:29.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:28.320188+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:39:29.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:28.324175+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:39:29.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:27.726863+0000 mgr.y (mgr.24437) 504 : cluster [DBG] pgmap v839: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 158 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 104/1269 objects degraded (8.195%); 74/1269 objects misplaced (5.831%); 54 B/s, 2 objects/s recovering 2024-02-23T00:39:29.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:27.810101+0000 osd.7 (osd.7) 262 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:39:29.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:27.813000+0000 osd.7 (osd.7) 263 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:39:29.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:28.180425+0000 mon.a (mon.0) 2980 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:39:29.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:28.190838+0000 mon.a (mon.0) 2981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]': finished 2024-02-23T00:39:29.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:28.190900+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? 172.21.15.87:0/2444109997' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:39:29.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:28.190946+0000 mon.a (mon.0) 2983 : cluster [DBG] osdmap e549: 8 total, 7 up, 8 in 2024-02-23T00:39:29.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:28.193680+0000 mon.b (mon.1) 447 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:29.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:28.193933+0000 mon.a (mon.0) 2984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90"}]: dispatch 2024-02-23T00:39:29.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:28.320188+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:39:29.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:28.324175+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:39:29.624 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:29.671 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:29.698 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:29.895 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:29.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:30.000 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:30.027 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:30.046 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:30.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:30.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:30.149 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:30.177 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:28.507694+0000 osd.4 (osd.4) 253 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:39:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:28.509666+0000 osd.4 (osd.4) 254 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:39:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:28.780262+0000 osd.5 (osd.5) 213 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:39:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:28.781580+0000 osd.5 (osd.5) 214 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:39:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:29.021831+0000 mgr.y (mgr.24437) 505 : audit [DBG] from='client.37967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:39:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:29.194380+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90"}]': finished 2024-02-23T00:39:30.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:29.194450+0000 mon.a (mon.0) 2986 : cluster [DBG] osdmap e550: 8 total, 7 up, 8 in 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:29.409639+0000 osd.6 (osd.6) 224 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:29.410453+0000 osd.6 (osd.6) 225 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:28.507694+0000 osd.4 (osd.4) 253 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:28.509666+0000 osd.4 (osd.4) 254 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:28.780262+0000 osd.5 (osd.5) 213 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:28.781580+0000 osd.5 (osd.5) 214 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:29.021831+0000 mgr.y (mgr.24437) 505 : audit [DBG] from='client.37967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:29.194380+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90"}]': finished 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:29.194450+0000 mon.a (mon.0) 2986 : cluster [DBG] osdmap e550: 8 total, 7 up, 8 in 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:29.409639+0000 osd.6 (osd.6) 224 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:39:30.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:29.410453+0000 osd.6 (osd.6) 225 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:39:30.436 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":840,"stamp":"2024-02-23T00:39:28.187402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4026236,"num_objects":423,"num_object_clones":11,"num_object_copies":1269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":104,"num_objects_misplaced":74,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":1964,"num_read_kb":1563,"num_write":1585,"num_write_kb":4662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":497,"num_bytes_recovered":8006173,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2925,"ondisk_log_size":2925,"up":652,"acting":674,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":779,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2506464,"kb_used_data":63136,"kb_used_omap":0,"kb_used_meta":2443264,"kb_avail":747454752,"statfs":{"total":767960285184,"available":765393666048,"internally_reserved":0,"allocated":64651264,"data_stored":40629461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2501902336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.029367"},"pg_stats":[{"pgid":"227.1f","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.197979+0000","last_change":"2024-02-23T00:39:20.283756+0000","last_active":"2024-02-23T00:39:24.197979+0000","last_peered":"2024-02-23T00:39:24.197979+0000","last_clean":"2024-02-23T00:39:24.197979+0000","last_became_active":"2024-02-23T00:39:20.283399+0000","last_became_peered":"2024-02-23T00:39:20.283399+0000","last_unstale":"2024-02-23T00:39:24.197979+0000","last_undegraded":"2024-02-23T00:39:24.197979+0000","last_fullsized":"2024-02-23T00:39:24.197979+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"227.1e","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.170279+0000","last_change":"2024-02-23T00:39:20.279596+0000","last_active":"2024-02-23T00:39:24.170279+0000","last_peered":"2024-02-23T00:39:24.170279+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.279596+0000","last_became_peered":"2024-02-23T00:39:20.279596+0000","last_unstale":"2024-02-23T00:39:24.170279+0000","last_undegraded":"2024-02-23T00:39:24.170279+0000","last_fullsized":"2024-02-23T00:39:20.274584+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"227.1d","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.169869+0000","last_change":"2024-02-23T00:39:20.285328+0000","last_active":"2024-02-23T00:39:24.169869+0000","last_peered":"2024-02-23T00:39:24.169869+0000","last_clean":"2024-02-23T00:39:24.169869+0000","last_became_active":"2024-02-23T00:39:20.285015+0000","last_became_peered":"2024-02-23T00:39:20.285015+0000","last_unstale":"2024-02-23T00:39:24.169869+0000","last_undegraded":"2024-02-23T00:39:24.169869+0000","last_fullsized":"2024-02-23T00:39:24.169869+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"227.1c","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.159989+0000","last_change":"2024-02-23T00:39:20.298289+0000","last_active":"2024-02-23T00:39:24.159989+0000","last_peered":"2024-02-23T00:39:24.159989+0000","last_clean":"2024-02-23T00:39:24.159989+0000","last_became_active":"2024-02-23T00:39:20.298001+0000","last_became_peered":"2024-02-23T00:39:20.298001+0000","last_unstale":"2024-02-23T00:39:24.159989+0000","last_undegraded":"2024-02-23T00:39:24.159989+0000","last_fullsized":"2024-02-23T00:39:24.159989+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"227.1b","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.160730+0000","last_change":"2024-02-23T00:39:20.282121+0000","last_active":"2024-02-23T00:39:24.160730+0000","last_peered":"2024-02-23T00:39:24.160730+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.282121+0000","last_became_peered":"2024-02-23T00:39:20.282121+0000","last_unstale":"2024-02-23T00:39:24.160730+0000","last_undegraded":"2024-02-23T00:39:24.160730+0000","last_fullsized":"2024-02-23T00:39:20.272212+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"227.1a","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.196787+0000","last_change":"2024-02-23T00:39:20.283057+0000","last_active":"2024-02-23T00:39:24.196787+0000","last_peered":"2024-02-23T00:39:24.196787+0000","last_clean":"2024-02-23T00:39:24.196787+0000","last_became_active":"2024-02-23T00:39:20.282774+0000","last_became_peered":"2024-02-23T00:39:20.282774+0000","last_unstale":"2024-02-23T00:39:24.196787+0000","last_undegraded":"2024-02-23T00:39:24.196787+0000","last_fullsized":"2024-02-23T00:39:24.196787+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"227.19","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.177107+0000","last_change":"2024-02-23T00:39:20.284875+0000","last_active":"2024-02-23T00:39:24.177107+0000","last_peered":"2024-02-23T00:39:24.177107+0000","last_clean":"2024-02-23T00:39:24.177107+0000","last_became_active":"2024-02-23T00:39:20.284570+0000","last_became_peered":"2024-02-23T00:39:20.284570+0000","last_unstale":"2024-02-23T00:39:24.177107+0000","last_undegraded":"2024-02-23T00:39:24.177107+0000","last_fullsized":"2024-02-23T00:39:24.177107+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"227.18","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161116+0000","last_change":"2024-02-23T00:39:20.289441+0000","last_active":"2024-02-23T00:39:24.161116+0000","last_peered":"2024-02-23T00:39:24.161116+0000","last_clean":"2024-02-23T00:39:24.161116+0000","last_became_active":"2024-02-23T00:39:20.289032+0000","last_became_peered":"2024-02-23T00:39:20.289032+0000","last_unstale":"2024-02-23T00:39:24.161116+0000","last_undegraded":"2024-02-23T00:39:24.161116+0000","last_fullsized":"2024-02-23T00:39:24.161116+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"227.17","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160700+0000","last_change":"2024-02-23T00:39:20.301006+0000","last_active":"2024-02-23T00:39:24.160700+0000","last_peered":"2024-02-23T00:39:24.160700+0000","last_clean":"2024-02-23T00:39:24.160700+0000","last_became_active":"2024-02-23T00:39:20.300696+0000","last_became_peered":"2024-02-23T00:39:20.300696+0000","last_unstale":"2024-02-23T00:39:24.160700+0000","last_undegraded":"2024-02-23T00:39:24.160700+0000","last_fullsized":"2024-02-23T00:39:24.160700+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"227.16","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.181769+0000","last_change":"2024-02-23T00:39:20.289872+0000","last_active":"2024-02-23T00:39:24.181769+0000","last_peered":"2024-02-23T00:39:24.181769+0000","last_clean":"2024-02-23T00:39:24.181769+0000","last_became_active":"2024-02-23T00:39:20.289608+0000","last_became_peered":"2024-02-23T00:39:20.289608+0000","last_unstale":"2024-02-23T00:39:24.181769+0000","last_undegraded":"2024-02-23T00:39:24.181769+0000","last_fullsized":"2024-02-23T00:39:24.181769+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"227.15","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.181407+0000","last_change":"2024-02-23T00:39:20.297386+0000","last_active":"2024-02-23T00:39:24.181407+0000","last_peered":"2024-02-23T00:39:24.181407+0000","last_clean":"2024-02-23T00:39:24.181407+0000","last_became_active":"2024-02-23T00:39:20.297095+0000","last_became_peered":"2024-02-23T00:39:20.297095+0000","last_unstale":"2024-02-23T00:39:24.181407+0000","last_undegraded":"2024-02-23T00:39:24.181407+0000","last_fullsized":"2024-02-23T00:39:24.181407+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"227.14","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.183343+0000","last_change":"2024-02-23T00:39:20.297659+0000","last_active":"2024-02-23T00:39:24.183343+0000","last_peered":"2024-02-23T00:39:24.183343+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.297659+0000","last_became_peered":"2024-02-23T00:39:20.297659+0000","last_unstale":"2024-02-23T00:39:24.183343+0000","last_undegraded":"2024-02-23T00:39:24.183343+0000","last_fullsized":"2024-02-23T00:39:20.269372+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"227.13","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.177900+0000","last_change":"2024-02-23T00:39:20.296548+0000","last_active":"2024-02-23T00:39:24.177900+0000","last_peered":"2024-02-23T00:39:24.177900+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.296548+0000","last_became_peered":"2024-02-23T00:39:20.296548+0000","last_unstale":"2024-02-23T00:39:24.177900+0000","last_undegraded":"2024-02-23T00:39:24.177900+0000","last_fullsized":"2024-02-23T00:39:20.268509+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"227.12","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.157944+0000","last_change":"2024-02-23T00:39:20.280380+0000","last_active":"2024-02-23T00:39:24.157944+0000","last_peered":"2024-02-23T00:39:24.157944+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.280380+0000","last_became_peered":"2024-02-23T00:39:20.280380+0000","last_unstale":"2024-02-23T00:39:24.157944+0000","last_undegraded":"2024-02-23T00:39:24.157944+0000","last_fullsized":"2024-02-23T00:39:20.267023+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"227.11","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157707+0000","last_change":"2024-02-23T00:39:20.275569+0000","last_active":"2024-02-23T00:39:24.157707+0000","last_peered":"2024-02-23T00:39:24.157707+0000","last_clean":"2024-02-23T00:39:24.157707+0000","last_became_active":"2024-02-23T00:39:20.275268+0000","last_became_peered":"2024-02-23T00:39:20.275268+0000","last_unstale":"2024-02-23T00:39:24.157707+0000","last_undegraded":"2024-02-23T00:39:24.157707+0000","last_fullsized":"2024-02-23T00:39:24.157707+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"227.10","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158354+0000","last_change":"2024-02-23T00:39:20.294298+0000","last_active":"2024-02-23T00:39:24.158354+0000","last_peered":"2024-02-23T00:39:24.158354+0000","last_clean":"2024-02-23T00:39:24.158354+0000","last_became_active":"2024-02-23T00:39:20.293946+0000","last_became_peered":"2024-02-23T00:39:20.293946+0000","last_unstale":"2024-02-23T00:39:24.158354+0000","last_undegraded":"2024-02-23T00:39:24.158354+0000","last_fullsized":"2024-02-23T00:39:24.158354+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"227.f","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.160771+0000","last_change":"2024-02-23T00:39:20.295064+0000","last_active":"2024-02-23T00:39:24.160771+0000","last_peered":"2024-02-23T00:39:24.160771+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.295064+0000","last_became_peered":"2024-02-23T00:39:20.295064+0000","last_unstale":"2024-02-23T00:39:24.160771+0000","last_undegraded":"2024-02-23T00:39:24.160771+0000","last_fullsized":"2024-02-23T00:39:20.277594+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"227.e","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.170157+0000","last_change":"2024-02-23T00:39:20.295243+0000","last_active":"2024-02-23T00:39:24.170157+0000","last_peered":"2024-02-23T00:39:24.170157+0000","last_clean":"2024-02-23T00:39:24.170157+0000","last_became_active":"2024-02-23T00:39:20.294957+0000","last_became_peered":"2024-02-23T00:39:20.294957+0000","last_unstale":"2024-02-23T00:39:24.170157+0000","last_undegraded":"2024-02-23T00:39:24.170157+0000","last_fullsized":"2024-02-23T00:39:24.170157+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"227.d","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160786+0000","last_change":"2024-02-23T00:39:20.283108+0000","last_active":"2024-02-23T00:39:24.160786+0000","last_peered":"2024-02-23T00:39:24.160786+0000","last_clean":"2024-02-23T00:39:24.160786+0000","last_became_active":"2024-02-23T00:39:20.282841+0000","last_became_peered":"2024-02-23T00:39:20.282841+0000","last_unstale":"2024-02-23T00:39:24.160786+0000","last_undegraded":"2024-02-23T00:39:24.160786+0000","last_fullsized":"2024-02-23T00:39:24.160786+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"227.c","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.182894+0000","last_change":"2024-02-23T00:39:20.285402+0000","last_active":"2024-02-23T00:39:24.182894+0000","last_peered":"2024-02-23T00:39:24.182894+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.285402+0000","last_became_peered":"2024-02-23T00:39:20.285402+0000","last_unstale":"2024-02-23T00:39:24.182894+0000","last_undegraded":"2024-02-23T00:39:24.182894+0000","last_fullsized":"2024-02-23T00:39:20.268628+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"227.b","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.159812+0000","last_change":"2024-02-23T00:39:20.285956+0000","last_active":"2024-02-23T00:39:24.159812+0000","last_peered":"2024-02-23T00:39:24.159812+0000","last_clean":"2024-02-23T00:39:24.159812+0000","last_became_active":"2024-02-23T00:39:20.285673+0000","last_became_peered":"2024-02-23T00:39:20.285673+0000","last_unstale":"2024-02-23T00:39:24.159812+0000","last_undegraded":"2024-02-23T00:39:24.159812+0000","last_fullsized":"2024-02-23T00:39:24.159812+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"227.a","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.162600+0000","last_change":"2024-02-23T00:39:20.298086+0000","last_active":"2024-02-23T00:39:24.162600+0000","last_peered":"2024-02-23T00:39:24.162600+0000","last_clean":"2024-02-23T00:39:24.162600+0000","last_became_active":"2024-02-23T00:39:20.297811+0000","last_became_peered":"2024-02-23T00:39:20.297811+0000","last_unstale":"2024-02-23T00:39:24.162600+0000","last_undegraded":"2024-02-23T00:39:24.162600+0000","last_fullsized":"2024-02-23T00:39:24.162600+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"227.9","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160289+0000","last_change":"2024-02-23T00:39:20.289930+0000","last_active":"2024-02-23T00:39:24.160289+0000","last_peered":"2024-02-23T00:39:24.160289+0000","last_clean":"2024-02-23T00:39:24.160289+0000","last_became_active":"2024-02-23T00:39:20.289693+0000","last_became_peered":"2024-02-23T00:39:20.289693+0000","last_unstale":"2024-02-23T00:39:24.160289+0000","last_undegraded":"2024-02-23T00:39:24.160289+0000","last_fullsized":"2024-02-23T00:39:24.160289+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"227.8","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.175451+0000","last_change":"2024-02-23T00:39:20.284857+0000","last_active":"2024-02-23T00:39:24.175451+0000","last_peered":"2024-02-23T00:39:24.175451+0000","last_clean":"2024-02-23T00:39:24.175451+0000","last_became_active":"2024-02-23T00:39:20.284554+0000","last_became_peered":"2024-02-23T00:39:20.284554+0000","last_unstale":"2024-02-23T00:39:24.175451+0000","last_undegraded":"2024-02-23T00:39:24.175451+0000","last_fullsized":"2024-02-23T00:39:24.175451+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"227.7","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.198205+0000","last_change":"2024-02-23T00:39:20.280780+0000","last_active":"2024-02-23T00:39:24.198205+0000","last_peered":"2024-02-23T00:39:24.198205+0000","last_clean":"2024-02-23T00:39:24.198205+0000","last_became_active":"2024-02-23T00:39:20.280487+0000","last_became_peered":"2024-02-23T00:39:20.280487+0000","last_unstale":"2024-02-23T00:39:24.198205+0000","last_undegraded":"2024-02-23T00:39:24.198205+0000","last_fullsized":"2024-02-23T00:39:24.198205+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"227.6","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.161138+0000","last_change":"2024-02-23T00:39:20.289395+0000","last_active":"2024-02-23T00:39:24.161138+0000","last_peered":"2024-02-23T00:39:24.161138+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.289395+0000","last_became_peered":"2024-02-23T00:39:20.289395+0000","last_unstale":"2024-02-23T00:39:24.161138+0000","last_undegraded":"2024-02-23T00:39:24.161138+0000","last_fullsized":"2024-02-23T00:39:20.280624+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"227.5","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.160896+0000","last_change":"2024-02-23T00:39:20.275897+0000","last_active":"2024-02-23T00:39:24.160896+0000","last_peered":"2024-02-23T00:39:24.160896+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.275897+0000","last_became_peered":"2024-02-23T00:39:20.275897+0000","last_unstale":"2024-02-23T00:39:24.160896+0000","last_undegraded":"2024-02-23T00:39:24.160896+0000","last_fullsized":"2024-02-23T00:39:20.270693+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"227.4","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157821+0000","last_change":"2024-02-23T00:39:20.296390+0000","last_active":"2024-02-23T00:39:24.157821+0000","last_peered":"2024-02-23T00:39:24.157821+0000","last_clean":"2024-02-23T00:39:24.157821+0000","last_became_active":"2024-02-23T00:39:20.296146+0000","last_became_peered":"2024-02-23T00:39:20.296146+0000","last_unstale":"2024-02-23T00:39:24.157821+0000","last_undegraded":"2024-02-23T00:39:24.157821+0000","last_fullsized":"2024-02-23T00:39:24.157821+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"227.3","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.159931+0000","last_change":"2024-02-23T00:39:20.279725+0000","last_active":"2024-02-23T00:39:24.159931+0000","last_peered":"2024-02-23T00:39:24.159931+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.279725+0000","last_became_peered":"2024-02-23T00:39:20.279725+0000","last_unstale":"2024-02-23T00:39:24.159931+0000","last_undegraded":"2024-02-23T00:39:24.159931+0000","last_fullsized":"2024-02-23T00:39:20.272057+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"227.2","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.196260+0000","last_change":"2024-02-23T00:39:20.276600+0000","last_active":"2024-02-23T00:39:24.196260+0000","last_peered":"2024-02-23T00:39:24.196260+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.276600+0000","last_became_peered":"2024-02-23T00:39:20.276600+0000","last_unstale":"2024-02-23T00:39:24.196260+0000","last_undegraded":"2024-02-23T00:39:24.196260+0000","last_fullsized":"2024-02-23T00:39:20.272928+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"227.1","version":"0'0","reported_seq":11,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.157281+0000","last_change":"2024-02-23T00:39:20.295871+0000","last_active":"2024-02-23T00:39:24.157281+0000","last_peered":"2024-02-23T00:39:24.157281+0000","last_clean":"2024-02-23T00:39:19.253612+0000","last_became_active":"2024-02-23T00:39:20.295871+0000","last_became_peered":"2024-02-23T00:39:20.295871+0000","last_unstale":"2024-02-23T00:39:24.157281+0000","last_undegraded":"2024-02-23T00:39:24.157281+0000","last_fullsized":"2024-02-23T00:39:20.267022+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"227.0","version":"0'0","reported_seq":14,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157104+0000","last_change":"2024-02-23T00:39:20.294578+0000","last_active":"2024-02-23T00:39:24.157104+0000","last_peered":"2024-02-23T00:39:24.157104+0000","last_clean":"2024-02-23T00:39:24.157104+0000","last_became_active":"2024-02-23T00:39:20.294293+0000","last_became_peered":"2024-02-23T00:39:20.294293+0000","last_unstale":"2024-02-23T00:39:24.157104+0000","last_undegraded":"2024-02-23T00:39:24.157104+0000","last_fullsized":"2024-02-23T00:39:24.157104+0000","mapping_epoch":540,"log_start":"0'0","ondisk_log_start":"0'0","created":540,"last_epoch_clean":541,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:19.253612+0000","last_clean_scrub_stamp":"2024-02-23T00:39:19.253612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.0","version":"485'86","reported_seq":309,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.168147+0000","last_change":"2024-02-23T00:36:37.106755+0000","last_active":"2024-02-23T00:39:24.168147+0000","last_peered":"2024-02-23T00:39:24.168147+0000","last_clean":"2024-02-23T00:36:33.365214+0000","last_became_active":"2024-02-23T00:36:37.106755+0000","last_became_peered":"2024-02-23T00:36:37.106755+0000","last_unstale":"2024-02-23T00:39:24.168147+0000","last_undegraded":"2024-02-23T00:36:37.100498+0000","last_fullsized":"2024-02-23T00:36:37.100377+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'68","last_scrub_stamp":"2024-02-23T00:34:57.645204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.645204+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":32,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1","version":"340'86","reported_seq":582,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.157491+0000","last_change":"2024-02-23T00:36:37.109838+0000","last_active":"2024-02-23T00:39:24.157491+0000","last_peered":"2024-02-23T00:39:24.157491+0000","last_clean":"2024-02-23T00:36:33.191340+0000","last_became_active":"2024-02-23T00:36:37.109838+0000","last_became_peered":"2024-02-23T00:36:37.109838+0000","last_unstale":"2024-02-23T00:39:24.157491+0000","last_undegraded":"2024-02-23T00:36:37.093708+0000","last_fullsized":"2024-02-23T00:36:37.093560+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'85","last_scrub_stamp":"2024-02-23T00:34:16.724630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:16.724630+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.2","version":"485'25","reported_seq":533,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.162901+0000","last_change":"2024-02-23T00:36:37.106211+0000","last_active":"2024-02-23T00:39:24.162901+0000","last_peered":"2024-02-23T00:39:24.162901+0000","last_clean":"2024-02-23T00:36:33.365672+0000","last_became_active":"2024-02-23T00:36:37.106211+0000","last_became_peered":"2024-02-23T00:36:37.106211+0000","last_unstale":"2024-02-23T00:39:24.162901+0000","last_undegraded":"2024-02-23T00:36:37.098859+0000","last_fullsized":"2024-02-23T00:36:37.098739+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'20","last_scrub_stamp":"2024-02-23T00:34:23.049395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:23.049395+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":15,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.3","version":"451'44","reported_seq":601,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158576+0000","last_change":"2024-02-23T00:38:49.270351+0000","last_active":"2024-02-23T00:39:24.158576+0000","last_peered":"2024-02-23T00:39:24.158576+0000","last_clean":"2024-02-23T00:39:24.158576+0000","last_became_active":"2024-02-23T00:38:49.226347+0000","last_became_peered":"2024-02-23T00:38:49.226347+0000","last_unstale":"2024-02-23T00:39:24.158576+0000","last_undegraded":"2024-02-23T00:39:24.158576+0000","last_fullsized":"2024-02-23T00:39:24.158576+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'27","last_scrub_stamp":"2024-02-23T00:37:05.079149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.079149+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":78,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.4","version":"442'49","reported_seq":551,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.198379+0000","last_change":"2024-02-23T00:38:35.832624+0000","last_active":"2024-02-23T00:39:24.198379+0000","last_peered":"2024-02-23T00:39:24.198379+0000","last_clean":"2024-02-23T00:39:24.198379+0000","last_became_active":"2024-02-23T00:31:06.212174+0000","last_became_peered":"2024-02-23T00:31:06.212174+0000","last_unstale":"2024-02-23T00:39:24.198379+0000","last_undegraded":"2024-02-23T00:39:24.198379+0000","last_fullsized":"2024-02-23T00:39:24.198379+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'49","last_scrub_stamp":"2024-02-23T00:38:35.832551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.832551+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.5","version":"471'126","reported_seq":619,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.158395+0000","last_change":"2024-02-23T00:36:37.105105+0000","last_active":"2024-02-23T00:39:24.158395+0000","last_peered":"2024-02-23T00:39:24.158395+0000","last_clean":"2024-02-23T00:36:33.192144+0000","last_became_active":"2024-02-23T00:36:37.105105+0000","last_became_peered":"2024-02-23T00:36:37.105105+0000","last_unstale":"2024-02-23T00:39:24.158395+0000","last_undegraded":"2024-02-23T00:36:37.095637+0000","last_fullsized":"2024-02-23T00:36:37.094980+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"234'124","last_scrub_stamp":"2024-02-23T00:34:19.695449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:19.695449+0000","log_size":126,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":92,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":39,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.6","version":"484'56","reported_seq":562,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.163083+0000","last_change":"2024-02-23T00:36:37.120734+0000","last_active":"2024-02-23T00:39:24.163083+0000","last_peered":"2024-02-23T00:39:24.163083+0000","last_clean":"2024-02-23T00:36:33.194384+0000","last_became_active":"2024-02-23T00:36:37.120734+0000","last_became_peered":"2024-02-23T00:36:37.120734+0000","last_unstale":"2024-02-23T00:39:24.163083+0000","last_undegraded":"2024-02-23T00:36:37.106912+0000","last_fullsized":"2024-02-23T00:36:37.106799+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'51","last_scrub_stamp":"2024-02-23T00:34:31.733148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:31.733148+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.7","version":"503'99","reported_seq":590,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.164129+0000","last_change":"2024-02-23T00:36:45.610239+0000","last_active":"2024-02-23T00:39:24.164129+0000","last_peered":"2024-02-23T00:39:24.164129+0000","last_clean":"2024-02-23T00:39:24.164129+0000","last_became_active":"2024-02-23T00:31:06.339410+0000","last_became_peered":"2024-02-23T00:31:06.339410+0000","last_unstale":"2024-02-23T00:39:24.164129+0000","last_undegraded":"2024-02-23T00:39:24.164129+0000","last_fullsized":"2024-02-23T00:39:24.164129+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'92","last_scrub_stamp":"2024-02-23T00:34:18.124049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:18.124049+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2009,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":9,"num_read_kb":2,"num_write":37,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.8","version":"484'73","reported_seq":92,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.175904+0000","last_change":"2024-02-23T00:38:53.080986+0000","last_active":"2024-02-23T00:39:24.175904+0000","last_peered":"2024-02-23T00:39:24.175904+0000","last_clean":"2024-02-23T00:39:24.175904+0000","last_became_active":"2024-02-23T00:38:49.222540+0000","last_became_peered":"2024-02-23T00:38:49.222540+0000","last_unstale":"2024-02-23T00:39:24.175904+0000","last_undegraded":"2024-02-23T00:39:24.175904+0000","last_fullsized":"2024-02-23T00:39:24.175904+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'71","last_scrub_stamp":"2024-02-23T00:35:41.651500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:41.651500+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.9","version":"485'29","reported_seq":536,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.162688+0000","last_change":"2024-02-23T00:38:47.260030+0000","last_active":"2024-02-23T00:39:24.162688+0000","last_peered":"2024-02-23T00:39:24.162688+0000","last_clean":"2024-02-23T00:39:24.162688+0000","last_became_active":"2024-02-23T00:31:06.356156+0000","last_became_peered":"2024-02-23T00:31:06.356156+0000","last_unstale":"2024-02-23T00:39:24.162688+0000","last_undegraded":"2024-02-23T00:39:24.162688+0000","last_fullsized":"2024-02-23T00:39:24.162688+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'29","last_scrub_stamp":"2024-02-23T00:38:47.259916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:47.259916+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.a","version":"503'23","reported_seq":532,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.196522+0000","last_change":"2024-02-23T00:38:00.735331+0000","last_active":"2024-02-23T00:39:24.196522+0000","last_peered":"2024-02-23T00:39:24.196522+0000","last_clean":"2024-02-23T00:39:24.196522+0000","last_became_active":"2024-02-23T00:31:06.313503+0000","last_became_peered":"2024-02-23T00:31:06.313503+0000","last_unstale":"2024-02-23T00:39:24.196522+0000","last_undegraded":"2024-02-23T00:39:24.196522+0000","last_fullsized":"2024-02-23T00:39:24.196522+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'22","last_scrub_stamp":"2024-02-23T00:38:00.735240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.735240+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1231,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.b","version":"503'42","reported_seq":623,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.168886+0000","last_change":"2024-02-23T00:39:20.899150+0000","last_active":"2024-02-23T00:39:24.168886+0000","last_peered":"2024-02-23T00:39:24.168886+0000","last_clean":"2024-02-23T00:39:24.168886+0000","last_became_active":"2024-02-23T00:38:50.185347+0000","last_became_peered":"2024-02-23T00:38:50.185347+0000","last_unstale":"2024-02-23T00:39:24.168886+0000","last_undegraded":"2024-02-23T00:39:24.168886+0000","last_fullsized":"2024-02-23T00:39:24.168886+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'42","last_scrub_stamp":"2024-02-23T00:39:20.899037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:39:20.899037+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":3,"num_write":20,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,4],"avail_no_missing":["7","3","4"],"object_location_counts":[{"shards":"3,4,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.c","version":"536'99","reported_seq":598,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.161793+0000","last_change":"2024-02-23T00:36:37.106629+0000","last_active":"2024-02-23T00:39:24.161793+0000","last_peered":"2024-02-23T00:39:24.161793+0000","last_clean":"2024-02-23T00:36:33.366529+0000","last_became_active":"2024-02-23T00:36:37.106629+0000","last_became_peered":"2024-02-23T00:36:37.106629+0000","last_unstale":"2024-02-23T00:39:24.161793+0000","last_undegraded":"2024-02-23T00:36:37.096711+0000","last_fullsized":"2024-02-23T00:36:37.096581+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'76","last_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_deep_scrub":"397'76","last_deep_scrub_stamp":"2024-02-23T00:35:51.098352+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.098352+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":68,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":8}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.d","version":"503'46","reported_seq":619,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.198082+0000","last_change":"2024-02-23T00:38:49.272366+0000","last_active":"2024-02-23T00:39:24.198082+0000","last_peered":"2024-02-23T00:39:24.198082+0000","last_clean":"2024-02-23T00:39:24.198082+0000","last_became_active":"2024-02-23T00:38:49.184383+0000","last_became_peered":"2024-02-23T00:38:49.184383+0000","last_unstale":"2024-02-23T00:39:24.198082+0000","last_undegraded":"2024-02-23T00:39:24.198082+0000","last_fullsized":"2024-02-23T00:39:24.198082+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'39","last_scrub_stamp":"2024-02-23T00:38:13.724131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.724131+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":22,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.e","version":"405'77","reported_seq":696,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.162006+0000","last_change":"2024-02-23T00:38:52.604431+0000","last_active":"2024-02-23T00:39:24.162006+0000","last_peered":"2024-02-23T00:39:24.162006+0000","last_clean":"2024-02-23T00:39:24.162006+0000","last_became_active":"2024-02-23T00:38:49.233320+0000","last_became_peered":"2024-02-23T00:38:49.233320+0000","last_unstale":"2024-02-23T00:39:24.162006+0000","last_undegraded":"2024-02-23T00:39:24.162006+0000","last_fullsized":"2024-02-23T00:39:24.162006+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'77","last_scrub_stamp":"2024-02-23T00:38:39.206374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.206374+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000073,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":3931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":8000130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.f","version":"485'78","reported_seq":94,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.184390+0000","last_change":"2024-02-23T00:38:49.405543+0000","last_active":"2024-02-23T00:39:24.184390+0000","last_peered":"2024-02-23T00:39:24.184390+0000","last_clean":"2024-02-23T00:39:24.184390+0000","last_became_active":"2024-02-23T00:38:49.221030+0000","last_became_peered":"2024-02-23T00:38:49.221030+0000","last_unstale":"2024-02-23T00:39:24.184390+0000","last_undegraded":"2024-02-23T00:39:24.184390+0000","last_fullsized":"2024-02-23T00:39:24.184390+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'69","last_scrub_stamp":"2024-02-23T00:35:46.750022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.750022+0000","log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":33,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.10","version":"503'39","reported_seq":538,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.159977+0000","last_change":"2024-02-23T00:36:37.119458+0000","last_active":"2024-02-23T00:39:24.159977+0000","last_peered":"2024-02-23T00:39:24.159977+0000","last_clean":"2024-02-23T00:36:33.364885+0000","last_became_active":"2024-02-23T00:36:37.119458+0000","last_became_peered":"2024-02-23T00:36:37.119458+0000","last_unstale":"2024-02-23T00:39:24.159977+0000","last_undegraded":"2024-02-23T00:36:37.098703+0000","last_fullsized":"2024-02-23T00:36:37.098584+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'34","last_scrub_stamp":"2024-02-23T00:35:01.791840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.791840+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":1758,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.11","version":"484'52","reported_seq":555,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157451+0000","last_change":"2024-02-23T00:37:06.042258+0000","last_active":"2024-02-23T00:39:24.157451+0000","last_peered":"2024-02-23T00:39:24.157451+0000","last_clean":"2024-02-23T00:39:24.157451+0000","last_became_active":"2024-02-23T00:31:06.330229+0000","last_became_peered":"2024-02-23T00:31:06.330229+0000","last_unstale":"2024-02-23T00:39:24.157451+0000","last_undegraded":"2024-02-23T00:39:24.157451+0000","last_fullsized":"2024-02-23T00:39:24.157451+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'50","last_scrub_stamp":"2024-02-23T00:37:06.042181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:06.042181+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":23,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.28","version":"0'0","reported_seq":189,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.195553+0000","last_change":"2024-02-23T00:38:49.189241+0000","last_active":"2024-02-23T00:39:24.195553+0000","last_peered":"2024-02-23T00:39:24.195553+0000","last_clean":"2024-02-23T00:39:24.195553+0000","last_became_active":"2024-02-23T00:38:49.188918+0000","last_became_peered":"2024-02-23T00:38:49.188918+0000","last_unstale":"2024-02-23T00:39:24.195553+0000","last_undegraded":"2024-02-23T00:39:24.195553+0000","last_fullsized":"2024-02-23T00:39:24.195553+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.678152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.678152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.12","version":"503'75","reported_seq":565,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.160729+0000","last_change":"2024-02-23T00:36:37.114447+0000","last_active":"2024-02-23T00:39:24.160729+0000","last_peered":"2024-02-23T00:39:24.160729+0000","last_clean":"2024-02-23T00:36:33.364627+0000","last_became_active":"2024-02-23T00:36:37.114447+0000","last_became_peered":"2024-02-23T00:36:37.114447+0000","last_unstale":"2024-02-23T00:39:24.160729+0000","last_undegraded":"2024-02-23T00:36:37.099733+0000","last_fullsized":"2024-02-23T00:36:37.099613+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'68","last_scrub_stamp":"2024-02-23T00:36:27.078990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.078990+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":611,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":31,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.29","version":"62'8","reported_seq":585,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.168297+0000","last_change":"2024-02-23T00:36:37.122045+0000","last_active":"2024-02-23T00:39:24.168297+0000","last_peered":"2024-02-23T00:39:24.168297+0000","last_clean":"2024-02-23T00:35:27.100547+0000","last_became_active":"2024-02-23T00:36:37.122045+0000","last_became_peered":"2024-02-23T00:36:37.122045+0000","last_unstale":"2024-02-23T00:39:24.168297+0000","last_undegraded":"2024-02-23T00:36:37.107771+0000","last_fullsized":"2024-02-23T00:36:37.107625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:27.673481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:27.673481+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.13","version":"494'65","reported_seq":565,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.168775+0000","last_change":"2024-02-23T00:36:45.667732+0000","last_active":"2024-02-23T00:39:24.168775+0000","last_peered":"2024-02-23T00:39:24.168775+0000","last_clean":"2024-02-23T00:39:24.168775+0000","last_became_active":"2024-02-23T00:31:06.231425+0000","last_became_peered":"2024-02-23T00:31:06.231425+0000","last_unstale":"2024-02-23T00:39:24.168775+0000","last_undegraded":"2024-02-23T00:39:24.168775+0000","last_fullsized":"2024-02-23T00:39:24.168775+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'61","last_scrub_stamp":"2024-02-23T00:36:38.944617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:36:38.944617+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2920,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":26,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.14","version":"494'86","reported_seq":579,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.161723+0000","last_change":"2024-02-23T00:36:37.115695+0000","last_active":"2024-02-23T00:39:24.161723+0000","last_peered":"2024-02-23T00:39:24.161723+0000","last_clean":"2024-02-23T00:36:33.366438+0000","last_became_active":"2024-02-23T00:36:37.115695+0000","last_became_peered":"2024-02-23T00:36:37.115695+0000","last_unstale":"2024-02-23T00:39:24.161723+0000","last_undegraded":"2024-02-23T00:36:37.099640+0000","last_fullsized":"2024-02-23T00:36:37.099515+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'85","last_scrub_stamp":"2024-02-23T00:35:46.085280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.085280+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":1392,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":6}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.15","version":"503'73","reported_seq":575,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158114+0000","last_change":"2024-02-23T00:37:04.031754+0000","last_active":"2024-02-23T00:39:24.158114+0000","last_peered":"2024-02-23T00:39:24.158114+0000","last_clean":"2024-02-23T00:39:24.158114+0000","last_became_active":"2024-02-23T00:31:06.229568+0000","last_became_peered":"2024-02-23T00:31:06.229568+0000","last_unstale":"2024-02-23T00:39:24.158114+0000","last_undegraded":"2024-02-23T00:39:24.158114+0000","last_fullsized":"2024-02-23T00:39:24.158114+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'70","last_scrub_stamp":"2024-02-23T00:37:04.031654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:37:04.031654+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5454,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"63.16","version":"485'55","reported_seq":562,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.163006+0000","last_change":"2024-02-23T00:36:37.111665+0000","last_active":"2024-02-23T00:39:24.163006+0000","last_peered":"2024-02-23T00:39:24.163006+0000","last_clean":"2024-02-23T00:36:33.194334+0000","last_became_active":"2024-02-23T00:36:37.111665+0000","last_became_peered":"2024-02-23T00:36:37.111665+0000","last_unstale":"2024-02-23T00:39:24.163006+0000","last_undegraded":"2024-02-23T00:36:37.100202+0000","last_fullsized":"2024-02-23T00:36:37.100076+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'51","last_scrub_stamp":"2024-02-23T00:35:46.192706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:35:46.192706+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":21,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"63.17","version":"433'112","reported_seq":609,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.198597+0000","last_change":"2024-02-23T00:38:04.819561+0000","last_active":"2024-02-23T00:39:24.198597+0000","last_peered":"2024-02-23T00:39:24.198597+0000","last_clean":"2024-02-23T00:39:24.198597+0000","last_became_active":"2024-02-23T00:31:06.320041+0000","last_became_peered":"2024-02-23T00:31:06.320041+0000","last_unstale":"2024-02-23T00:39:24.198597+0000","last_undegraded":"2024-02-23T00:39:24.198597+0000","last_fullsized":"2024-02-23T00:39:24.198597+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'112","last_scrub_stamp":"2024-02-23T00:38:04.819475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:04.819475+0000","log_size":112,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":36,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.22","version":"0'0","reported_seq":182,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.168600+0000","last_change":"2024-02-23T00:37:09.026463+0000","last_active":"2024-02-23T00:39:24.168600+0000","last_peered":"2024-02-23T00:39:24.168600+0000","last_clean":"2024-02-23T00:39:24.168600+0000","last_became_active":"2024-02-23T00:35:29.089082+0000","last_became_peered":"2024-02-23T00:35:29.089082+0000","last_unstale":"2024-02-23T00:39:24.168600+0000","last_undegraded":"2024-02-23T00:39:24.168600+0000","last_fullsized":"2024-02-23T00:39:24.168600+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.026383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.026383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.18","version":"484'59","reported_seq":567,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.168306+0000","last_change":"2024-02-23T00:38:44.885221+0000","last_active":"2024-02-23T00:39:24.168306+0000","last_peered":"2024-02-23T00:39:24.168306+0000","last_clean":"2024-02-23T00:39:24.168306+0000","last_became_active":"2024-02-23T00:31:06.332064+0000","last_became_peered":"2024-02-23T00:31:06.332064+0000","last_unstale":"2024-02-23T00:39:24.168306+0000","last_undegraded":"2024-02-23T00:39:24.168306+0000","last_fullsized":"2024-02-23T00:39:24.168306+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'59","last_scrub_stamp":"2024-02-23T00:38:44.885113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.885113+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":21,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.23","version":"513'13","reported_seq":252,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-02-23T00:39:25.171224+0000","last_change":"2024-02-23T00:38:49.290324+0000","last_active":"2024-02-23T00:39:25.171224+0000","last_peered":"2024-02-23T00:39:25.171224+0000","last_clean":"2024-02-23T00:39:25.171224+0000","last_became_active":"2024-02-23T00:38:49.226952+0000","last_became_peered":"2024-02-23T00:38:49.226952+0000","last_unstale":"2024-02-23T00:39:25.171224+0000","last_undegraded":"2024-02-23T00:39:25.171224+0000","last_fullsized":"2024-02-23T00:39:25.171224+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"63.19","version":"503'86","reported_seq":597,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157306+0000","last_change":"2024-02-23T00:36:45.297690+0000","last_active":"2024-02-23T00:39:24.157306+0000","last_peered":"2024-02-23T00:39:24.157306+0000","last_clean":"2024-02-23T00:39:24.157306+0000","last_became_active":"2024-02-23T00:31:06.253713+0000","last_became_peered":"2024-02-23T00:31:06.253713+0000","last_unstale":"2024-02-23T00:39:24.157306+0000","last_undegraded":"2024-02-23T00:39:24.157306+0000","last_fullsized":"2024-02-23T00:39:24.157306+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'60","last_scrub_stamp":"2024-02-23T00:36:31.102608+0000","last_deep_scrub":"271'54","last_deep_scrub_stamp":"2024-02-23T00:33:42.780899+0000","last_clean_scrub_stamp":"2024-02-23T00:36:31.102608+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1416,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":18,"num_read_kb":8,"num_write":46,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.20","version":"379'11","reported_seq":235,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-02-23T00:39:25.284888+0000","last_change":"2024-02-23T00:37:10.032410+0000","last_active":"2024-02-23T00:39:25.284888+0000","last_peered":"2024-02-23T00:39:25.284888+0000","last_clean":"2024-02-23T00:39:25.284888+0000","last_became_active":"2024-02-23T00:35:29.093220+0000","last_became_peered":"2024-02-23T00:35:29.093220+0000","last_unstale":"2024-02-23T00:39:25.284888+0000","last_undegraded":"2024-02-23T00:39:25.284888+0000","last_fullsized":"2024-02-23T00:39:25.284888+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"379'11","last_scrub_stamp":"2024-02-23T00:37:10.032341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:10.032341+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"63.1a","version":"526'84","reported_seq":596,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161614+0000","last_change":"2024-02-23T00:38:44.336029+0000","last_active":"2024-02-23T00:39:24.161614+0000","last_peered":"2024-02-23T00:39:24.161614+0000","last_clean":"2024-02-23T00:39:24.161614+0000","last_became_active":"2024-02-23T00:31:06.367357+0000","last_became_peered":"2024-02-23T00:31:06.367357+0000","last_unstale":"2024-02-23T00:39:24.161614+0000","last_undegraded":"2024-02-23T00:39:24.161614+0000","last_fullsized":"2024-02-23T00:39:24.161614+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'70","last_scrub_stamp":"2024-02-23T00:38:44.335909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.335909+0000","log_size":84,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1278,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":30,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.21","version":"0'0","reported_seq":193,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160508+0000","last_change":"2024-02-23T00:38:49.195187+0000","last_active":"2024-02-23T00:39:24.160508+0000","last_peered":"2024-02-23T00:39:24.160508+0000","last_clean":"2024-02-23T00:39:24.160508+0000","last_became_active":"2024-02-23T00:38:49.194848+0000","last_became_peered":"2024-02-23T00:38:49.194848+0000","last_unstale":"2024-02-23T00:39:24.160508+0000","last_undegraded":"2024-02-23T00:39:24.160508+0000","last_fullsized":"2024-02-23T00:39:24.160508+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:06.376016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:34.152643+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.376016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"63.1b","version":"494'55","reported_seq":556,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.197069+0000","last_change":"2024-02-23T00:38:07.746173+0000","last_active":"2024-02-23T00:39:24.197069+0000","last_peered":"2024-02-23T00:39:24.197069+0000","last_clean":"2024-02-23T00:39:24.197069+0000","last_became_active":"2024-02-23T00:31:06.280382+0000","last_became_peered":"2024-02-23T00:31:06.280382+0000","last_unstale":"2024-02-23T00:39:24.197069+0000","last_undegraded":"2024-02-23T00:39:24.197069+0000","last_fullsized":"2024-02-23T00:39:24.197069+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'54","last_scrub_stamp":"2024-02-23T00:38:07.746090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.746090+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1256,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.26","version":"0'0","reported_seq":45,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.182205+0000","last_change":"2024-02-23T00:38:50.177811+0000","last_active":"2024-02-23T00:39:24.182205+0000","last_peered":"2024-02-23T00:39:24.182205+0000","last_clean":"2024-02-23T00:39:24.182205+0000","last_became_active":"2024-02-23T00:38:50.177418+0000","last_became_peered":"2024-02-23T00:38:50.177418+0000","last_unstale":"2024-02-23T00:39:24.182205+0000","last_undegraded":"2024-02-23T00:39:24.182205+0000","last_fullsized":"2024-02-23T00:39:24.182205+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,2],"avail_no_missing":["3","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"63.1c","version":"216'34","reported_seq":593,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160289+0000","last_change":"2024-02-23T00:38:51.138734+0000","last_active":"2024-02-23T00:39:24.160289+0000","last_peered":"2024-02-23T00:39:24.160289+0000","last_clean":"2024-02-23T00:39:24.160289+0000","last_became_active":"2024-02-23T00:38:49.226791+0000","last_became_peered":"2024-02-23T00:38:49.226791+0000","last_unstale":"2024-02-23T00:39:24.160289+0000","last_undegraded":"2024-02-23T00:39:24.160289+0000","last_fullsized":"2024-02-23T00:39:24.160289+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'34","last_scrub_stamp":"2024-02-23T00:34:57.859047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:34:57.859047+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.27","version":"0'0","reported_seq":528,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163981+0000","last_change":"2024-02-23T00:38:49.227500+0000","last_active":"2024-02-23T00:39:24.163981+0000","last_peered":"2024-02-23T00:39:24.163981+0000","last_clean":"2024-02-23T00:39:24.163981+0000","last_became_active":"2024-02-23T00:38:49.225263+0000","last_became_peered":"2024-02-23T00:38:49.225263+0000","last_unstale":"2024-02-23T00:39:24.163981+0000","last_undegraded":"2024-02-23T00:39:24.163981+0000","last_fullsized":"2024-02-23T00:39:24.163981+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:43.246802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:38:43.246802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1d","version":"520'18","reported_seq":268,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160865+0000","last_change":"2024-02-23T00:38:16.420850+0000","last_active":"2024-02-23T00:39:24.160865+0000","last_peered":"2024-02-23T00:39:24.160865+0000","last_clean":"2024-02-23T00:39:24.160865+0000","last_became_active":"2024-02-23T00:34:42.058724+0000","last_became_peered":"2024-02-23T00:34:42.058724+0000","last_unstale":"2024-02-23T00:39:24.160865+0000","last_undegraded":"2024-02-23T00:39:24.160865+0000","last_fullsized":"2024-02-23T00:39:24.160865+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'13","last_scrub_stamp":"2024-02-23T00:38:16.420789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.420789+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":13,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.24","version":"0'0","reported_seq":526,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.197849+0000","last_change":"2024-02-23T00:36:37.117551+0000","last_active":"2024-02-23T00:39:24.197849+0000","last_peered":"2024-02-23T00:39:24.197849+0000","last_clean":"2024-02-23T00:36:33.366613+0000","last_became_active":"2024-02-23T00:36:37.117551+0000","last_became_peered":"2024-02-23T00:36:37.117551+0000","last_unstale":"2024-02-23T00:39:24.197849+0000","last_undegraded":"2024-02-23T00:39:24.197849+0000","last_fullsized":"2024-02-23T00:36:37.104246+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"63.1e","version":"340'52","reported_seq":549,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.196526+0000","last_change":"2024-02-23T00:38:09.722750+0000","last_active":"2024-02-23T00:39:24.196526+0000","last_peered":"2024-02-23T00:39:24.196526+0000","last_clean":"2024-02-23T00:39:24.196526+0000","last_became_active":"2024-02-23T00:31:06.208924+0000","last_became_peered":"2024-02-23T00:31:06.208924+0000","last_unstale":"2024-02-23T00:39:24.196526+0000","last_undegraded":"2024-02-23T00:39:24.196526+0000","last_fullsized":"2024-02-23T00:39:24.196526+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"340'52","last_scrub_stamp":"2024-02-23T00:38:09.722680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.722680+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"}]},{"pgid":"5.25","version":"0'0","reported_seq":528,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161172+0000","last_change":"2024-02-23T00:38:42.191298+0000","last_active":"2024-02-23T00:39:24.161172+0000","last_peered":"2024-02-23T00:39:24.161172+0000","last_clean":"2024-02-23T00:39:24.161172+0000","last_became_active":"2024-02-23T00:35:29.090793+0000","last_became_peered":"2024-02-23T00:35:29.090793+0000","last_unstale":"2024-02-23T00:39:24.161172+0000","last_undegraded":"2024-02-23T00:39:24.161172+0000","last_fullsized":"2024-02-23T00:39:24.161172+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.191221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.191221+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.191221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"63.1f","version":"439'74","reported_seq":570,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.198576+0000","last_change":"2024-02-23T00:36:37.117847+0000","last_active":"2024-02-23T00:39:24.198576+0000","last_peered":"2024-02-23T00:39:24.198576+0000","last_clean":"2024-02-23T00:36:33.365701+0000","last_became_active":"2024-02-23T00:36:37.117847+0000","last_became_peered":"2024-02-23T00:36:37.117847+0000","last_unstale":"2024-02-23T00:39:24.198576+0000","last_undegraded":"2024-02-23T00:36:37.102042+0000","last_fullsized":"2024-02-23T00:36:37.101901+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'68","last_scrub_stamp":"2024-02-23T00:33:39.351573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:05.147166+0000","last_clean_scrub_stamp":"2024-02-23T00:33:39.351573+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"379'9","reported_seq":684,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-02-23T00:39:25.287368+0000","last_change":"2024-02-23T00:37:05.955772+0000","last_active":"2024-02-23T00:39:25.287368+0000","last_peered":"2024-02-23T00:39:25.287368+0000","last_clean":"2024-02-23T00:39:25.287368+0000","last_became_active":"2024-02-23T00:35:27.095085+0000","last_became_peered":"2024-02-23T00:35:27.095085+0000","last_unstale":"2024-02-23T00:39:25.287368+0000","last_undegraded":"2024-02-23T00:39:25.287368+0000","last_fullsized":"2024-02-23T00:39:25.287368+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'9","last_scrub_stamp":"2024-02-23T00:37:05.955707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.955707+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":630,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.195738+0000","last_change":"2024-02-23T00:38:19.164982+0000","last_active":"2024-02-23T00:39:24.195738+0000","last_peered":"2024-02-23T00:39:24.195738+0000","last_clean":"2024-02-23T00:39:24.195738+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:39:24.195738+0000","last_undegraded":"2024-02-23T00:39:24.195738+0000","last_fullsized":"2024-02-23T00:39:24.195738+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":553,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163196+0000","last_change":"2024-02-23T00:38:49.199186+0000","last_active":"2024-02-23T00:39:24.163196+0000","last_peered":"2024-02-23T00:39:24.163196+0000","last_clean":"2024-02-23T00:39:24.163196+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:39:24.163196+0000","last_undegraded":"2024-02-23T00:39:24.163196+0000","last_fullsized":"2024-02-23T00:39:24.163196+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:12.113097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:38:12.113097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":508,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.196931+0000","last_change":"2024-02-23T00:36:37.104200+0000","last_active":"2024-02-23T00:39:24.196931+0000","last_peered":"2024-02-23T00:39:24.196931+0000","last_clean":"2024-02-23T00:36:33.366159+0000","last_became_active":"2024-02-23T00:36:37.104200+0000","last_became_peered":"2024-02-23T00:36:37.104200+0000","last_unstale":"2024-02-23T00:39:24.196931+0000","last_undegraded":"2024-02-23T00:39:24.196931+0000","last_fullsized":"2024-02-23T00:36:37.102434+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:21.441958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.441958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":472,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.163610+0000","last_change":"2024-02-23T00:36:37.115879+0000","last_active":"2024-02-23T00:39:24.163610+0000","last_peered":"2024-02-23T00:39:24.163610+0000","last_clean":"2024-02-23T00:36:33.366205+0000","last_became_active":"2024-02-23T00:36:37.115879+0000","last_became_peered":"2024-02-23T00:36:37.115879+0000","last_unstale":"2024-02-23T00:39:24.163610+0000","last_undegraded":"2024-02-23T00:39:24.163610+0000","last_fullsized":"2024-02-23T00:36:37.101967+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":929,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:25.238531+0000","last_change":"2024-02-23T00:38:49.283349+0000","last_active":"2024-02-23T00:39:25.238531+0000","last_peered":"2024-02-23T00:39:25.238531+0000","last_clean":"2024-02-23T00:39:25.238531+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:39:25.238531+0000","last_undegraded":"2024-02-23T00:39:25.238531+0000","last_fullsized":"2024-02-23T00:39:25.238531+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:36:27.181972+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.181972+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":406,"num_read_kb":406,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":191,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.161046+0000","last_change":"2024-02-23T00:38:19.177071+0000","last_active":"2024-02-23T00:39:24.161046+0000","last_peered":"2024-02-23T00:39:24.161046+0000","last_clean":"2024-02-23T00:27:29.183258+0000","last_became_active":"2024-02-23T00:38:19.177071+0000","last_became_peered":"2024-02-23T00:38:19.177071+0000","last_unstale":"2024-02-23T00:39:24.161046+0000","last_undegraded":"2024-02-23T00:38:19.154601+0000","last_fullsized":"2024-02-23T00:38:19.154480+0000","mapping_epoch":425,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":533,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161901+0000","last_change":"2024-02-23T00:38:49.228134+0000","last_active":"2024-02-23T00:39:24.161901+0000","last_peered":"2024-02-23T00:39:24.161901+0000","last_clean":"2024-02-23T00:39:24.161901+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:39:24.161901+0000","last_undegraded":"2024-02-23T00:39:24.161901+0000","last_fullsized":"2024-02-23T00:39:24.161901+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.532619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.532619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":545,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161831+0000","last_change":"2024-02-23T00:39:20.396906+0000","last_active":"2024-02-23T00:39:24.161831+0000","last_peered":"2024-02-23T00:39:24.161831+0000","last_clean":"2024-02-23T00:39:24.161831+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:39:24.161831+0000","last_undegraded":"2024-02-23T00:39:24.161831+0000","last_fullsized":"2024-02-23T00:39:24.161831+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:20.396815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:39:20.396815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":128,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.168489+0000","last_change":"2024-02-23T00:36:37.122229+0000","last_active":"2024-02-23T00:39:24.168489+0000","last_peered":"2024-02-23T00:39:24.168489+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.122229+0000","last_became_peered":"2024-02-23T00:36:37.122229+0000","last_unstale":"2024-02-23T00:39:24.168489+0000","last_undegraded":"2024-02-23T00:39:24.168489+0000","last_fullsized":"2024-02-23T00:36:37.107921+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":466,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161721+0000","last_change":"2024-02-23T00:38:01.361178+0000","last_active":"2024-02-23T00:39:24.161721+0000","last_peered":"2024-02-23T00:39:24.161721+0000","last_clean":"2024-02-23T00:39:24.161721+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:39:24.161721+0000","last_undegraded":"2024-02-23T00:39:24.161721+0000","last_fullsized":"2024-02-23T00:39:24.161721+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:01.361095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:01.361095+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":650,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161578+0000","last_change":"2024-02-23T00:38:40.424549+0000","last_active":"2024-02-23T00:39:24.161578+0000","last_peered":"2024-02-23T00:39:24.161578+0000","last_clean":"2024-02-23T00:39:24.161578+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:39:24.161578+0000","last_undegraded":"2024-02-23T00:39:24.161578+0000","last_fullsized":"2024-02-23T00:39:24.161578+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:40.424490+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.424490+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":531,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.162250+0000","last_change":"2024-02-23T00:36:37.116992+0000","last_active":"2024-02-23T00:39:24.162250+0000","last_peered":"2024-02-23T00:39:24.162250+0000","last_clean":"2024-02-23T00:36:33.194161+0000","last_became_active":"2024-02-23T00:36:37.116992+0000","last_became_peered":"2024-02-23T00:36:37.116992+0000","last_unstale":"2024-02-23T00:39:24.162250+0000","last_undegraded":"2024-02-23T00:39:24.162250+0000","last_fullsized":"2024-02-23T00:36:37.104222+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:50.194656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:50.194656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":522,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.160333+0000","last_change":"2024-02-23T00:36:37.113419+0000","last_active":"2024-02-23T00:39:24.160333+0000","last_peered":"2024-02-23T00:39:24.160333+0000","last_clean":"2024-02-23T00:36:33.194007+0000","last_became_active":"2024-02-23T00:36:37.113419+0000","last_became_peered":"2024-02-23T00:36:37.113419+0000","last_unstale":"2024-02-23T00:39:24.160333+0000","last_undegraded":"2024-02-23T00:39:24.160333+0000","last_fullsized":"2024-02-23T00:36:37.104161+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.745865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.745865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":534,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.168505+0000","last_change":"2024-02-23T00:38:49.208110+0000","last_active":"2024-02-23T00:39:24.168505+0000","last_peered":"2024-02-23T00:39:24.168505+0000","last_clean":"2024-02-23T00:39:24.168505+0000","last_became_active":"2024-02-23T00:38:49.207843+0000","last_became_peered":"2024-02-23T00:38:49.207843+0000","last_unstale":"2024-02-23T00:39:24.168505+0000","last_undegraded":"2024-02-23T00:39:24.168505+0000","last_fullsized":"2024-02-23T00:39:24.168505+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.226380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.226380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":129,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.162233+0000","last_change":"2024-02-23T00:36:37.119721+0000","last_active":"2024-02-23T00:39:24.162233+0000","last_peered":"2024-02-23T00:39:24.162233+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119721+0000","last_became_peered":"2024-02-23T00:36:37.119721+0000","last_unstale":"2024-02-23T00:39:24.162233+0000","last_undegraded":"2024-02-23T00:39:24.162233+0000","last_fullsized":"2024-02-23T00:36:37.102163+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.205312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.205312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":533,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161975+0000","last_change":"2024-02-23T00:38:44.442795+0000","last_active":"2024-02-23T00:39:24.161975+0000","last_peered":"2024-02-23T00:39:24.161975+0000","last_clean":"2024-02-23T00:39:24.161975+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:39:24.161975+0000","last_undegraded":"2024-02-23T00:39:24.161975+0000","last_fullsized":"2024-02-23T00:39:24.161975+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:44.442733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.442733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":603,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158835+0000","last_change":"2024-02-23T00:38:19.177268+0000","last_active":"2024-02-23T00:39:24.158835+0000","last_peered":"2024-02-23T00:39:24.158835+0000","last_clean":"2024-02-23T00:39:24.158835+0000","last_became_active":"2024-02-23T00:38:19.176922+0000","last_became_peered":"2024-02-23T00:38:19.176922+0000","last_unstale":"2024-02-23T00:39:24.158835+0000","last_undegraded":"2024-02-23T00:39:24.158835+0000","last_fullsized":"2024-02-23T00:39:24.158835+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":29,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":522,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.169169+0000","last_change":"2024-02-23T00:38:18.055894+0000","last_active":"2024-02-23T00:39:24.169169+0000","last_peered":"2024-02-23T00:39:24.169169+0000","last_clean":"2024-02-23T00:39:24.169169+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:39:24.169169+0000","last_undegraded":"2024-02-23T00:39:24.169169+0000","last_fullsized":"2024-02-23T00:39:24.169169+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:18.055803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:18.055803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":512,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.160675+0000","last_change":"2024-02-23T00:36:37.109480+0000","last_active":"2024-02-23T00:39:24.160675+0000","last_peered":"2024-02-23T00:39:24.160675+0000","last_clean":"2024-02-23T00:36:34.605757+0000","last_became_active":"2024-02-23T00:36:37.109480+0000","last_became_peered":"2024-02-23T00:36:37.109480+0000","last_unstale":"2024-02-23T00:39:24.160675+0000","last_undegraded":"2024-02-23T00:39:24.160675+0000","last_fullsized":"2024-02-23T00:36:37.095167+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.605664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.605664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":562,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161802+0000","last_change":"2024-02-23T00:38:49.182352+0000","last_active":"2024-02-23T00:39:24.161802+0000","last_peered":"2024-02-23T00:39:24.161802+0000","last_clean":"2024-02-23T00:39:24.161802+0000","last_became_active":"2024-02-23T00:38:49.181997+0000","last_became_peered":"2024-02-23T00:38:49.181997+0000","last_unstale":"2024-02-23T00:39:24.161802+0000","last_undegraded":"2024-02-23T00:39:24.161802+0000","last_fullsized":"2024-02-23T00:39:24.161802+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:17.822776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.822776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":491,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.169965+0000","last_change":"2024-02-23T00:39:22.867678+0000","last_active":"2024-02-23T00:39:24.169965+0000","last_peered":"2024-02-23T00:39:24.169965+0000","last_clean":"2024-02-23T00:39:24.169965+0000","last_became_active":"2024-02-23T00:38:49.194450+0000","last_became_peered":"2024-02-23T00:38:49.194450+0000","last_unstale":"2024-02-23T00:39:24.169965+0000","last_undegraded":"2024-02-23T00:39:24.169965+0000","last_fullsized":"2024-02-23T00:39:24.169965+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.704478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.704478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":45,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.178811+0000","last_change":"2024-02-23T00:38:50.199088+0000","last_active":"2024-02-23T00:39:24.178811+0000","last_peered":"2024-02-23T00:39:24.178811+0000","last_clean":"2024-02-23T00:39:24.178811+0000","last_became_active":"2024-02-23T00:38:50.198701+0000","last_became_peered":"2024-02-23T00:38:50.198701+0000","last_unstale":"2024-02-23T00:39:24.178811+0000","last_undegraded":"2024-02-23T00:39:24.178811+0000","last_fullsized":"2024-02-23T00:39:24.178811+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,1],"avail_no_missing":["3","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":647,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158353+0000","last_change":"2024-02-23T00:38:54.172406+0000","last_active":"2024-02-23T00:39:24.158353+0000","last_peered":"2024-02-23T00:39:24.158353+0000","last_clean":"2024-02-23T00:39:24.158353+0000","last_became_active":"2024-02-23T00:38:54.171993+0000","last_became_peered":"2024-02-23T00:38:54.171993+0000","last_unstale":"2024-02-23T00:39:24.158353+0000","last_undegraded":"2024-02-23T00:39:24.158353+0000","last_fullsized":"2024-02-23T00:39:24.158353+0000","mapping_epoch":516,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:37:09.133471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.133471+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":546,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163336+0000","last_change":"2024-02-23T00:38:49.225600+0000","last_active":"2024-02-23T00:39:24.163336+0000","last_peered":"2024-02-23T00:39:24.163336+0000","last_clean":"2024-02-23T00:39:24.163336+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:39:24.163336+0000","last_undegraded":"2024-02-23T00:39:24.163336+0000","last_fullsized":"2024-02-23T00:39:24.163336+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:56.678676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:35:56.678676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":680,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-02-23T00:39:25.160844+0000","last_change":"2024-02-23T00:38:35.337273+0000","last_active":"2024-02-23T00:39:25.160844+0000","last_peered":"2024-02-23T00:39:25.160844+0000","last_clean":"2024-02-23T00:39:25.160844+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:39:25.160844+0000","last_undegraded":"2024-02-23T00:39:25.160844+0000","last_fullsized":"2024-02-23T00:39:25.160844+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:38:35.337170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.337170+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":536,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158659+0000","last_change":"2024-02-23T00:38:49.171993+0000","last_active":"2024-02-23T00:39:24.158659+0000","last_peered":"2024-02-23T00:39:24.158659+0000","last_clean":"2024-02-23T00:39:24.158659+0000","last_became_active":"2024-02-23T00:38:49.171696+0000","last_became_peered":"2024-02-23T00:38:49.171696+0000","last_unstale":"2024-02-23T00:39:24.158659+0000","last_undegraded":"2024-02-23T00:39:24.158659+0000","last_fullsized":"2024-02-23T00:39:24.158659+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.135802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.135802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":471,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158777+0000","last_change":"2024-02-23T00:38:34.097327+0000","last_active":"2024-02-23T00:39:24.158777+0000","last_peered":"2024-02-23T00:39:24.158777+0000","last_clean":"2024-02-23T00:39:24.158777+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:39:24.158777+0000","last_undegraded":"2024-02-23T00:39:24.158777+0000","last_fullsized":"2024-02-23T00:39:24.158777+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:34.097243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.097243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.176212+0000","last_change":"2024-02-23T00:38:49.223162+0000","last_active":"2024-02-23T00:39:24.176212+0000","last_peered":"2024-02-23T00:39:24.176212+0000","last_clean":"2024-02-23T00:39:24.176212+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:39:24.176212+0000","last_undegraded":"2024-02-23T00:39:24.176212+0000","last_fullsized":"2024-02-23T00:39:24.176212+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:22.396222+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.396222+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":705,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-02-23T00:39:25.200274+0000","last_change":"2024-02-23T00:39:25.200274+0000","last_active":"2024-02-23T00:39:25.200274+0000","last_peered":"2024-02-23T00:39:25.200274+0000","last_clean":"2024-02-23T00:39:25.200274+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:39:25.200274+0000","last_undegraded":"2024-02-23T00:39:25.200274+0000","last_fullsized":"2024-02-23T00:39:25.200274+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":530,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.162823+0000","last_change":"2024-02-23T00:38:49.228791+0000","last_active":"2024-02-23T00:39:24.162823+0000","last_peered":"2024-02-23T00:39:24.162823+0000","last_clean":"2024-02-23T00:39:24.162823+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:39:24.162823+0000","last_undegraded":"2024-02-23T00:39:24.162823+0000","last_fullsized":"2024-02-23T00:39:24.162823+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:35:06.657998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"426'9","reported_seq":676,"reported_epoch":546,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:25.172460+0000","last_change":"2024-02-23T00:36:37.111178+0000","last_active":"2024-02-23T00:39:25.172460+0000","last_peered":"2024-02-23T00:39:25.172460+0000","last_clean":"2024-02-23T00:36:35.444981+0000","last_became_active":"2024-02-23T00:36:37.111178+0000","last_became_peered":"2024-02-23T00:36:37.111178+0000","last_unstale":"2024-02-23T00:39:25.172460+0000","last_undegraded":"2024-02-23T00:36:37.095825+0000","last_fullsized":"2024-02-23T00:36:37.095713+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:59.630171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:59.630171+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":509,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.198676+0000","last_change":"2024-02-23T00:36:37.113156+0000","last_active":"2024-02-23T00:39:24.198676+0000","last_peered":"2024-02-23T00:39:24.198676+0000","last_clean":"2024-02-23T00:36:33.366740+0000","last_became_active":"2024-02-23T00:36:37.113156+0000","last_became_peered":"2024-02-23T00:36:37.113156+0000","last_unstale":"2024-02-23T00:39:24.198676+0000","last_undegraded":"2024-02-23T00:36:37.102218+0000","last_fullsized":"2024-02-23T00:36:37.102103+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:34:17.437083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.437083+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":462,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161377+0000","last_change":"2024-02-23T00:38:16.396902+0000","last_active":"2024-02-23T00:39:24.161377+0000","last_peered":"2024-02-23T00:39:24.161377+0000","last_clean":"2024-02-23T00:39:24.161377+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:39:24.161377+0000","last_undegraded":"2024-02-23T00:39:24.161377+0000","last_fullsized":"2024-02-23T00:39:24.161377+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.396824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":533,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157181+0000","last_change":"2024-02-23T00:38:08.068624+0000","last_active":"2024-02-23T00:39:24.157181+0000","last_peered":"2024-02-23T00:39:24.157181+0000","last_clean":"2024-02-23T00:39:24.157181+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:39:24.157181+0000","last_undegraded":"2024-02-23T00:39:24.157181+0000","last_fullsized":"2024-02-23T00:39:24.157181+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:08.068559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:08.068559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":617,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158142+0000","last_change":"2024-02-23T00:38:19.182554+0000","last_active":"2024-02-23T00:39:24.158142+0000","last_peered":"2024-02-23T00:39:24.158142+0000","last_clean":"2024-02-23T00:39:24.158142+0000","last_became_active":"2024-02-23T00:38:19.182222+0000","last_became_peered":"2024-02-23T00:38:19.182222+0000","last_unstale":"2024-02-23T00:39:24.158142+0000","last_undegraded":"2024-02-23T00:39:24.158142+0000","last_fullsized":"2024-02-23T00:39:24.158142+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":530,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.168972+0000","last_change":"2024-02-23T00:36:37.126232+0000","last_active":"2024-02-23T00:39:24.168972+0000","last_peered":"2024-02-23T00:39:24.168972+0000","last_clean":"2024-02-23T00:36:33.365949+0000","last_became_active":"2024-02-23T00:36:37.126232+0000","last_became_peered":"2024-02-23T00:36:37.126232+0000","last_unstale":"2024-02-23T00:39:24.168972+0000","last_undegraded":"2024-02-23T00:36:37.112806+0000","last_fullsized":"2024-02-23T00:36:37.112690+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":509,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.198886+0000","last_change":"2024-02-23T00:38:37.881705+0000","last_active":"2024-02-23T00:39:24.198886+0000","last_peered":"2024-02-23T00:39:24.198886+0000","last_clean":"2024-02-23T00:39:24.198886+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:39:24.198886+0000","last_undegraded":"2024-02-23T00:39:24.198886+0000","last_fullsized":"2024-02-23T00:39:24.198886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:37.881618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.881618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.183467+0000","last_change":"2024-02-23T00:38:49.230667+0000","last_active":"2024-02-23T00:39:24.183467+0000","last_peered":"2024-02-23T00:39:24.183467+0000","last_clean":"2024-02-23T00:39:24.183467+0000","last_became_active":"2024-02-23T00:38:49.226279+0000","last_became_peered":"2024-02-23T00:38:49.226279+0000","last_unstale":"2024-02-23T00:39:24.183467+0000","last_undegraded":"2024-02-23T00:39:24.183467+0000","last_fullsized":"2024-02-23T00:39:24.183467+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.478470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.478470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":480,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160932+0000","last_change":"2024-02-23T00:38:49.227768+0000","last_active":"2024-02-23T00:39:24.160932+0000","last_peered":"2024-02-23T00:39:24.160932+0000","last_clean":"2024-02-23T00:39:24.160932+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:39:24.160932+0000","last_undegraded":"2024-02-23T00:39:24.160932+0000","last_fullsized":"2024-02-23T00:39:24.160932+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_clean_scrub_stamp":"2024-02-23T00:37:21.364934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":129,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.161792+0000","last_change":"2024-02-23T00:36:37.109542+0000","last_active":"2024-02-23T00:39:24.161792+0000","last_peered":"2024-02-23T00:39:24.161792+0000","last_clean":"2024-02-23T00:31:27.149861+0000","last_became_active":"2024-02-23T00:36:37.109542+0000","last_became_peered":"2024-02-23T00:36:37.109542+0000","last_unstale":"2024-02-23T00:39:24.161792+0000","last_undegraded":"2024-02-23T00:39:24.161792+0000","last_fullsized":"2024-02-23T00:36:37.096846+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:42.114807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:35:42.114807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.176132+0000","last_change":"2024-02-23T00:38:49.222977+0000","last_active":"2024-02-23T00:39:24.176132+0000","last_peered":"2024-02-23T00:39:24.176132+0000","last_clean":"2024-02-23T00:39:24.176132+0000","last_became_active":"2024-02-23T00:38:49.216657+0000","last_became_peered":"2024-02-23T00:38:49.216657+0000","last_unstale":"2024-02-23T00:39:24.176132+0000","last_undegraded":"2024-02-23T00:39:24.176132+0000","last_fullsized":"2024-02-23T00:39:24.176132+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":574,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157709+0000","last_change":"2024-02-23T00:37:08.107220+0000","last_active":"2024-02-23T00:39:24.157709+0000","last_peered":"2024-02-23T00:39:24.157709+0000","last_clean":"2024-02-23T00:39:24.157709+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:39:24.157709+0000","last_undegraded":"2024-02-23T00:39:24.157709+0000","last_fullsized":"2024-02-23T00:39:24.157709+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:37:08.107110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.107110+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":523,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161352+0000","last_change":"2024-02-23T00:38:48.494288+0000","last_active":"2024-02-23T00:39:24.161352+0000","last_peered":"2024-02-23T00:39:24.161352+0000","last_clean":"2024-02-23T00:39:24.161352+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:39:24.161352+0000","last_undegraded":"2024-02-23T00:39:24.161352+0000","last_fullsized":"2024-02-23T00:39:24.161352+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:48.494224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:48.494224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":516,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.169914+0000","last_change":"2024-02-23T00:36:37.952567+0000","last_active":"2024-02-23T00:39:24.169914+0000","last_peered":"2024-02-23T00:39:24.169914+0000","last_clean":"2024-02-23T00:39:24.169914+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:39:24.169914+0000","last_undegraded":"2024-02-23T00:39:24.169914+0000","last_fullsized":"2024-02-23T00:39:24.169914+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:37.952474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:37.952474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":487,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.161304+0000","last_change":"2024-02-23T00:39:22.420483+0000","last_active":"2024-02-23T00:39:24.161304+0000","last_peered":"2024-02-23T00:39:24.161304+0000","last_clean":"2024-02-23T00:39:24.161304+0000","last_became_active":"2024-02-23T00:38:50.206899+0000","last_became_peered":"2024-02-23T00:38:50.206899+0000","last_unstale":"2024-02-23T00:39:24.161304+0000","last_undegraded":"2024-02-23T00:39:24.161304+0000","last_fullsized":"2024-02-23T00:39:24.161304+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:22.420380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.420380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":45,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.184248+0000","last_change":"2024-02-23T00:38:50.201494+0000","last_active":"2024-02-23T00:39:24.184248+0000","last_peered":"2024-02-23T00:39:24.184248+0000","last_clean":"2024-02-23T00:39:24.184248+0000","last_became_active":"2024-02-23T00:38:50.201068+0000","last_became_peered":"2024-02-23T00:38:50.201068+0000","last_unstale":"2024-02-23T00:39:24.184248+0000","last_undegraded":"2024-02-23T00:39:24.184248+0000","last_fullsized":"2024-02-23T00:39:24.184248+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.924808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.924808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,5],"avail_no_missing":["3","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":548,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158102+0000","last_change":"2024-02-23T00:38:49.228711+0000","last_active":"2024-02-23T00:39:24.158102+0000","last_peered":"2024-02-23T00:39:24.158102+0000","last_clean":"2024-02-23T00:39:24.158102+0000","last_became_active":"2024-02-23T00:38:49.228425+0000","last_became_peered":"2024-02-23T00:38:49.228425+0000","last_unstale":"2024-02-23T00:39:24.158102+0000","last_undegraded":"2024-02-23T00:39:24.158102+0000","last_fullsized":"2024-02-23T00:39:24.158102+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.588132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.588132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":67,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.175990+0000","last_change":"2024-02-23T00:38:54.112431+0000","last_active":"2024-02-23T00:39:24.175990+0000","last_peered":"2024-02-23T00:39:24.175990+0000","last_clean":"2024-02-23T00:39:24.175990+0000","last_became_active":"2024-02-23T00:38:50.199619+0000","last_became_peered":"2024-02-23T00:38:50.199619+0000","last_unstale":"2024-02-23T00:39:24.175990+0000","last_undegraded":"2024-02-23T00:39:24.175990+0000","last_fullsized":"2024-02-23T00:39:24.175990+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'30","last_scrub_stamp":"2024-02-23T00:35:00.621157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.621157+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":515,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.162157+0000","last_change":"2024-02-23T00:38:14.343844+0000","last_active":"2024-02-23T00:39:24.162157+0000","last_peered":"2024-02-23T00:39:24.162157+0000","last_clean":"2024-02-23T00:39:24.162157+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:39:24.162157+0000","last_undegraded":"2024-02-23T00:39:24.162157+0000","last_fullsized":"2024-02-23T00:39:24.162157+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:14.343770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:14.343770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":536,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158099+0000","last_change":"2024-02-23T00:38:49.225300+0000","last_active":"2024-02-23T00:39:24.158099+0000","last_peered":"2024-02-23T00:39:24.158099+0000","last_clean":"2024-02-23T00:39:24.158099+0000","last_became_active":"2024-02-23T00:38:49.224892+0000","last_became_peered":"2024-02-23T00:38:49.224892+0000","last_unstale":"2024-02-23T00:39:24.158099+0000","last_undegraded":"2024-02-23T00:39:24.158099+0000","last_fullsized":"2024-02-23T00:39:24.158099+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.082000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.082000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.176876+0000","last_change":"2024-02-23T00:38:49.176838+0000","last_active":"2024-02-23T00:39:24.176876+0000","last_peered":"2024-02-23T00:39:24.176876+0000","last_clean":"2024-02-23T00:39:24.176876+0000","last_became_active":"2024-02-23T00:38:49.175801+0000","last_became_peered":"2024-02-23T00:38:49.175801+0000","last_unstale":"2024-02-23T00:39:24.176876+0000","last_undegraded":"2024-02-23T00:39:24.176876+0000","last_fullsized":"2024-02-23T00:39:24.176876+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.977282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.977282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":778,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160196+0000","last_change":"2024-02-23T00:37:17.499775+0000","last_active":"2024-02-23T00:39:24.160196+0000","last_peered":"2024-02-23T00:39:24.160196+0000","last_clean":"2024-02-23T00:39:24.160196+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:39:24.160196+0000","last_undegraded":"2024-02-23T00:39:24.160196+0000","last_fullsized":"2024-02-23T00:39:24.160196+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:17.499694+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.499694+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":530,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.161760+0000","last_change":"2024-02-23T00:36:37.108691+0000","last_active":"2024-02-23T00:39:24.161760+0000","last_peered":"2024-02-23T00:39:24.161760+0000","last_clean":"2024-02-23T00:36:33.369928+0000","last_became_active":"2024-02-23T00:36:37.108691+0000","last_became_peered":"2024-02-23T00:36:37.108691+0000","last_unstale":"2024-02-23T00:39:24.161760+0000","last_undegraded":"2024-02-23T00:39:24.161760+0000","last_fullsized":"2024-02-23T00:36:37.100335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.196211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.196211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":207,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.197522+0000","last_change":"2024-02-23T00:36:37.123994+0000","last_active":"2024-02-23T00:39:24.197522+0000","last_peered":"2024-02-23T00:39:24.197522+0000","last_clean":"2024-02-23T00:27:29.216066+0000","last_became_active":"2024-02-23T00:36:37.123994+0000","last_became_peered":"2024-02-23T00:36:37.123994+0000","last_unstale":"2024-02-23T00:39:24.197522+0000","last_undegraded":"2024-02-23T00:36:37.103439+0000","last_fullsized":"2024-02-23T00:36:37.103335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:35:01.396982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.396982+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":523,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.170098+0000","last_change":"2024-02-23T00:36:37.120094+0000","last_active":"2024-02-23T00:39:24.170098+0000","last_peered":"2024-02-23T00:39:24.170098+0000","last_clean":"2024-02-23T00:36:33.367908+0000","last_became_active":"2024-02-23T00:36:37.120094+0000","last_became_peered":"2024-02-23T00:36:37.120094+0000","last_unstale":"2024-02-23T00:39:24.170098+0000","last_undegraded":"2024-02-23T00:39:24.170098+0000","last_fullsized":"2024-02-23T00:36:37.107601+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":509,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.162901+0000","last_change":"2024-02-23T00:37:26.268389+0000","last_active":"2024-02-23T00:39:24.162901+0000","last_peered":"2024-02-23T00:39:24.162901+0000","last_clean":"2024-02-23T00:39:24.162901+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:39:24.162901+0000","last_undegraded":"2024-02-23T00:39:24.162901+0000","last_fullsized":"2024-02-23T00:39:24.162901+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.268282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":457,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.162985+0000","last_change":"2024-02-23T00:35:00.832892+0000","last_active":"2024-02-23T00:39:24.162985+0000","last_peered":"2024-02-23T00:39:24.162985+0000","last_clean":"2024-02-23T00:39:24.162985+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:39:24.162985+0000","last_undegraded":"2024-02-23T00:39:24.162985+0000","last_fullsized":"2024-02-23T00:39:24.162985+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.832818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.832818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":519,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.198926+0000","last_change":"2024-02-23T00:36:37.117616+0000","last_active":"2024-02-23T00:39:24.198926+0000","last_peered":"2024-02-23T00:39:24.198926+0000","last_clean":"2024-02-23T00:36:33.368573+0000","last_became_active":"2024-02-23T00:36:37.117616+0000","last_became_peered":"2024-02-23T00:36:37.117616+0000","last_unstale":"2024-02-23T00:39:24.198926+0000","last_undegraded":"2024-02-23T00:39:24.198926+0000","last_fullsized":"2024-02-23T00:36:37.100549+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":192,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.162257+0000","last_change":"2024-02-23T00:38:19.160002+0000","last_active":"2024-02-23T00:39:24.162257+0000","last_peered":"2024-02-23T00:39:24.162257+0000","last_clean":"2024-02-23T00:27:24.365389+0000","last_became_active":"2024-02-23T00:38:19.160002+0000","last_became_peered":"2024-02-23T00:38:19.160002+0000","last_unstale":"2024-02-23T00:39:24.162257+0000","last_undegraded":"2024-02-23T00:38:19.155620+0000","last_fullsized":"2024-02-23T00:38:19.155500+0000","mapping_epoch":425,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":523,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.168834+0000","last_change":"2024-02-23T00:36:37.126102+0000","last_active":"2024-02-23T00:39:24.168834+0000","last_peered":"2024-02-23T00:39:24.168834+0000","last_clean":"2024-02-23T00:36:33.365781+0000","last_became_active":"2024-02-23T00:36:37.126102+0000","last_became_peered":"2024-02-23T00:36:37.126102+0000","last_unstale":"2024-02-23T00:39:24.168834+0000","last_undegraded":"2024-02-23T00:39:24.168834+0000","last_fullsized":"2024-02-23T00:36:37.112617+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.980396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.980396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":460,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.425086+0000","last_change":"2024-02-23T00:39:24.425037+0000","last_active":"2024-02-23T00:39:24.425086+0000","last_peered":"2024-02-23T00:39:24.425086+0000","last_clean":"2024-02-23T00:39:24.425086+0000","last_became_active":"2024-02-23T00:38:50.177540+0000","last_became_peered":"2024-02-23T00:38:50.177540+0000","last_unstale":"2024-02-23T00:39:24.425086+0000","last_undegraded":"2024-02-23T00:39:24.425086+0000","last_fullsized":"2024-02-23T00:39:24.425086+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:24.424938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:24.424938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,5],"avail_no_missing":["4","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":522,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160859+0000","last_change":"2024-02-23T00:38:49.224597+0000","last_active":"2024-02-23T00:39:24.160859+0000","last_peered":"2024-02-23T00:39:24.160859+0000","last_clean":"2024-02-23T00:39:24.160859+0000","last_became_active":"2024-02-23T00:38:49.224244+0000","last_became_peered":"2024-02-23T00:38:49.224244+0000","last_unstale":"2024-02-23T00:39:24.160859+0000","last_undegraded":"2024-02-23T00:39:24.160859+0000","last_fullsized":"2024-02-23T00:39:24.160859+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.744088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.744088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":466,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158460+0000","last_change":"2024-02-23T00:35:55.223009+0000","last_active":"2024-02-23T00:39:24.158460+0000","last_peered":"2024-02-23T00:39:24.158460+0000","last_clean":"2024-02-23T00:39:24.158460+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:39:24.158460+0000","last_undegraded":"2024-02-23T00:39:24.158460+0000","last_fullsized":"2024-02-23T00:39:24.158460+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.222912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.222912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":527,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.169322+0000","last_change":"2024-02-23T00:36:41.937397+0000","last_active":"2024-02-23T00:39:24.169322+0000","last_peered":"2024-02-23T00:39:24.169322+0000","last_clean":"2024-02-23T00:39:24.169322+0000","last_became_active":"2024-02-23T00:35:27.093541+0000","last_became_peered":"2024-02-23T00:35:27.093541+0000","last_unstale":"2024-02-23T00:39:24.169322+0000","last_undegraded":"2024-02-23T00:39:24.169322+0000","last_fullsized":"2024-02-23T00:39:24.169322+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.937344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.937344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":562,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160966+0000","last_change":"2024-02-23T00:38:03.327200+0000","last_active":"2024-02-23T00:39:24.160966+0000","last_peered":"2024-02-23T00:39:24.160966+0000","last_clean":"2024-02-23T00:39:24.160966+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:39:24.160966+0000","last_undegraded":"2024-02-23T00:39:24.160966+0000","last_fullsized":"2024-02-23T00:39:24.160966+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:03.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.327127+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":526,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.169118+0000","last_change":"2024-02-23T00:36:37.106377+0000","last_active":"2024-02-23T00:39:24.169118+0000","last_peered":"2024-02-23T00:39:24.169118+0000","last_clean":"2024-02-23T00:36:33.365867+0000","last_became_active":"2024-02-23T00:36:37.106377+0000","last_became_peered":"2024-02-23T00:36:37.106377+0000","last_unstale":"2024-02-23T00:39:24.169118+0000","last_undegraded":"2024-02-23T00:39:24.169118+0000","last_fullsized":"2024-02-23T00:36:37.100069+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":513,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.197688+0000","last_change":"2024-02-23T00:36:33.160039+0000","last_active":"2024-02-23T00:39:24.197688+0000","last_peered":"2024-02-23T00:39:24.197688+0000","last_clean":"2024-02-23T00:39:24.197688+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:39:24.197688+0000","last_undegraded":"2024-02-23T00:39:24.197688+0000","last_fullsized":"2024-02-23T00:39:24.197688+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.159941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.159941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.176946+0000","last_change":"2024-02-23T00:38:49.226513+0000","last_active":"2024-02-23T00:39:24.176946+0000","last_peered":"2024-02-23T00:39:24.176946+0000","last_clean":"2024-02-23T00:39:24.176946+0000","last_became_active":"2024-02-23T00:38:49.226188+0000","last_became_peered":"2024-02-23T00:38:49.226188+0000","last_unstale":"2024-02-23T00:39:24.176946+0000","last_undegraded":"2024-02-23T00:39:24.176946+0000","last_fullsized":"2024-02-23T00:39:24.176946+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.933288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.933288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":525,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.198970+0000","last_change":"2024-02-23T00:38:49.184762+0000","last_active":"2024-02-23T00:39:24.198970+0000","last_peered":"2024-02-23T00:39:24.198970+0000","last_clean":"2024-02-23T00:39:24.198970+0000","last_became_active":"2024-02-23T00:38:49.184274+0000","last_became_peered":"2024-02-23T00:38:49.184274+0000","last_unstale":"2024-02-23T00:39:24.198970+0000","last_undegraded":"2024-02-23T00:39:24.198970+0000","last_fullsized":"2024-02-23T00:39:24.198970+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.008726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":473,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158926+0000","last_change":"2024-02-23T00:38:44.173181+0000","last_active":"2024-02-23T00:39:24.158926+0000","last_peered":"2024-02-23T00:39:24.158926+0000","last_clean":"2024-02-23T00:39:24.158926+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:39:24.158926+0000","last_undegraded":"2024-02-23T00:39:24.158926+0000","last_fullsized":"2024-02-23T00:39:24.158926+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:44.173077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.173077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":535,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163273+0000","last_change":"2024-02-23T00:37:59.003696+0000","last_active":"2024-02-23T00:39:24.163273+0000","last_peered":"2024-02-23T00:39:24.163273+0000","last_clean":"2024-02-23T00:39:24.163273+0000","last_became_active":"2024-02-23T00:35:27.097186+0000","last_became_peered":"2024-02-23T00:35:27.097186+0000","last_unstale":"2024-02-23T00:39:24.163273+0000","last_undegraded":"2024-02-23T00:39:24.163273+0000","last_fullsized":"2024-02-23T00:39:24.163273+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.003583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.003583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":601,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.164390+0000","last_change":"2024-02-23T00:38:19.164063+0000","last_active":"2024-02-23T00:39:24.164390+0000","last_peered":"2024-02-23T00:39:24.164390+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:38:19.164063+0000","last_became_peered":"2024-02-23T00:38:19.164063+0000","last_unstale":"2024-02-23T00:39:24.164390+0000","last_undegraded":"2024-02-23T00:38:19.153386+0000","last_fullsized":"2024-02-23T00:38:19.153249+0000","mapping_epoch":425,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":555,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.163372+0000","last_change":"2024-02-23T00:39:22.437200+0000","last_active":"2024-02-23T00:39:24.163372+0000","last_peered":"2024-02-23T00:39:24.163372+0000","last_clean":"2024-02-23T00:39:24.163372+0000","last_became_active":"2024-02-23T00:38:50.196782+0000","last_became_peered":"2024-02-23T00:38:50.196782+0000","last_unstale":"2024-02-23T00:39:24.163372+0000","last_undegraded":"2024-02-23T00:39:24.163372+0000","last_fullsized":"2024-02-23T00:39:24.163372+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:22.437104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.437104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,1],"avail_no_missing":["2","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":771,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160515+0000","last_change":"2024-02-23T00:37:13.467508+0000","last_active":"2024-02-23T00:39:24.160515+0000","last_peered":"2024-02-23T00:39:24.160515+0000","last_clean":"2024-02-23T00:39:24.160515+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:39:24.160515+0000","last_undegraded":"2024-02-23T00:39:24.160515+0000","last_fullsized":"2024-02-23T00:39:24.160515+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:13.467428+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:13.467428+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":534,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.170094+0000","last_change":"2024-02-23T00:38:49.196976+0000","last_active":"2024-02-23T00:39:24.170094+0000","last_peered":"2024-02-23T00:39:24.170094+0000","last_clean":"2024-02-23T00:39:24.170094+0000","last_became_active":"2024-02-23T00:38:49.195404+0000","last_became_peered":"2024-02-23T00:38:49.195404+0000","last_unstale":"2024-02-23T00:39:24.170094+0000","last_undegraded":"2024-02-23T00:39:24.170094+0000","last_fullsized":"2024-02-23T00:39:24.170094+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:12.023125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:12.023125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":477,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161073+0000","last_change":"2024-02-23T00:38:40.265472+0000","last_active":"2024-02-23T00:39:24.161073+0000","last_peered":"2024-02-23T00:39:24.161073+0000","last_clean":"2024-02-23T00:39:24.161073+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:39:24.161073+0000","last_undegraded":"2024-02-23T00:39:24.161073+0000","last_fullsized":"2024-02-23T00:39:24.161073+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:40.265385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.265385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":534,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.158582+0000","last_change":"2024-02-23T00:36:37.101373+0000","last_active":"2024-02-23T00:39:24.158582+0000","last_peered":"2024-02-23T00:39:24.158582+0000","last_clean":"2024-02-23T00:36:33.192308+0000","last_became_active":"2024-02-23T00:36:37.101373+0000","last_became_peered":"2024-02-23T00:36:37.101373+0000","last_unstale":"2024-02-23T00:39:24.158582+0000","last_undegraded":"2024-02-23T00:39:24.158582+0000","last_fullsized":"2024-02-23T00:36:37.094754+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.595447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.595447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":474,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.161587+0000","last_change":"2024-02-23T00:36:37.109549+0000","last_active":"2024-02-23T00:39:24.161587+0000","last_peered":"2024-02-23T00:39:24.161587+0000","last_clean":"2024-02-23T00:35:25.318841+0000","last_became_active":"2024-02-23T00:36:37.109549+0000","last_became_peered":"2024-02-23T00:36:37.109549+0000","last_unstale":"2024-02-23T00:39:24.161587+0000","last_undegraded":"2024-02-23T00:39:24.161587+0000","last_fullsized":"2024-02-23T00:36:37.096473+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":806,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-02-23T00:39:25.163507+0000","last_change":"2024-02-23T00:38:51.396520+0000","last_active":"2024-02-23T00:39:25.163507+0000","last_peered":"2024-02-23T00:39:25.163507+0000","last_clean":"2024-02-23T00:39:25.163507+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:39:25.163507+0000","last_undegraded":"2024-02-23T00:39:25.163507+0000","last_fullsized":"2024-02-23T00:39:25.163507+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'35","last_scrub_stamp":"2024-02-23T00:37:15.422435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:15.422435+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":153,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163143+0000","last_change":"2024-02-23T00:37:23.360353+0000","last_active":"2024-02-23T00:39:24.163143+0000","last_peered":"2024-02-23T00:39:24.163143+0000","last_clean":"2024-02-23T00:39:24.163143+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:39:24.163143+0000","last_undegraded":"2024-02-23T00:39:24.163143+0000","last_fullsized":"2024-02-23T00:39:24.163143+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:37:23.360230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:23.360230+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.182533+0000","last_change":"2024-02-23T00:38:49.190209+0000","last_active":"2024-02-23T00:39:24.182533+0000","last_peered":"2024-02-23T00:39:24.182533+0000","last_clean":"2024-02-23T00:39:24.182533+0000","last_became_active":"2024-02-23T00:38:49.189587+0000","last_became_peered":"2024-02-23T00:38:49.189587+0000","last_unstale":"2024-02-23T00:39:24.182533+0000","last_undegraded":"2024-02-23T00:39:24.182533+0000","last_fullsized":"2024-02-23T00:39:24.182533+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.098280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.098280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":477,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163426+0000","last_change":"2024-02-23T00:38:43.332992+0000","last_active":"2024-02-23T00:39:24.163426+0000","last_peered":"2024-02-23T00:39:24.163426+0000","last_clean":"2024-02-23T00:39:24.163426+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:39:24.163426+0000","last_undegraded":"2024-02-23T00:39:24.163426+0000","last_fullsized":"2024-02-23T00:39:24.163426+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:43.332907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:43.332907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":441,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.197857+0000","last_change":"2024-02-23T00:38:49.184857+0000","last_active":"2024-02-23T00:39:24.197857+0000","last_peered":"2024-02-23T00:39:24.197857+0000","last_clean":"2024-02-23T00:39:24.197857+0000","last_became_active":"2024-02-23T00:38:49.183777+0000","last_became_peered":"2024-02-23T00:38:49.183777+0000","last_unstale":"2024-02-23T00:39:24.197857+0000","last_undegraded":"2024-02-23T00:39:24.197857+0000","last_fullsized":"2024-02-23T00:39:24.197857+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:23.210706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.210706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":148,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.161021+0000","last_change":"2024-02-23T00:36:37.104760+0000","last_active":"2024-02-23T00:39:24.161021+0000","last_peered":"2024-02-23T00:39:24.161021+0000","last_clean":"2024-02-23T00:27:24.172824+0000","last_became_active":"2024-02-23T00:36:37.104760+0000","last_became_peered":"2024-02-23T00:36:37.104760+0000","last_unstale":"2024-02-23T00:39:24.161021+0000","last_undegraded":"2024-02-23T00:39:24.161021+0000","last_fullsized":"2024-02-23T00:36:37.098375+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":186,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.197349+0000","last_change":"2024-02-23T00:36:37.110364+0000","last_active":"2024-02-23T00:39:24.197349+0000","last_peered":"2024-02-23T00:39:24.197349+0000","last_clean":"2024-02-23T00:27:29.173100+0000","last_became_active":"2024-02-23T00:36:37.110364+0000","last_became_peered":"2024-02-23T00:36:37.110364+0000","last_unstale":"2024-02-23T00:39:24.197349+0000","last_undegraded":"2024-02-23T00:36:37.096575+0000","last_fullsized":"2024-02-23T00:36:37.096459+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:35:44.214727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:44.214727+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":519,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.160392+0000","last_change":"2024-02-23T00:36:37.123256+0000","last_active":"2024-02-23T00:39:24.160392+0000","last_peered":"2024-02-23T00:39:24.160392+0000","last_clean":"2024-02-23T00:36:33.366566+0000","last_became_active":"2024-02-23T00:36:37.123256+0000","last_became_peered":"2024-02-23T00:36:37.123256+0000","last_unstale":"2024-02-23T00:39:24.160392+0000","last_undegraded":"2024-02-23T00:39:24.160392+0000","last_fullsized":"2024-02-23T00:36:37.096440+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":50,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.175754+0000","last_change":"2024-02-23T00:39:21.371091+0000","last_active":"2024-02-23T00:39:24.175754+0000","last_peered":"2024-02-23T00:39:24.175754+0000","last_clean":"2024-02-23T00:39:24.175754+0000","last_became_active":"2024-02-23T00:38:50.177038+0000","last_became_peered":"2024-02-23T00:38:50.177038+0000","last_unstale":"2024-02-23T00:39:24.175754+0000","last_undegraded":"2024-02-23T00:39:24.175754+0000","last_fullsized":"2024-02-23T00:39:24.175754+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:21.370914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:21.370914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,5],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":466,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157836+0000","last_change":"2024-02-23T00:36:42.093272+0000","last_active":"2024-02-23T00:39:24.157836+0000","last_peered":"2024-02-23T00:39:24.157836+0000","last_clean":"2024-02-23T00:39:24.157836+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:39:24.157836+0000","last_undegraded":"2024-02-23T00:39:24.157836+0000","last_fullsized":"2024-02-23T00:39:24.157836+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:36:42.093181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":531,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.162655+0000","last_change":"2024-02-23T00:36:37.124382+0000","last_active":"2024-02-23T00:39:24.162655+0000","last_peered":"2024-02-23T00:39:24.162655+0000","last_clean":"2024-02-23T00:36:33.193431+0000","last_became_active":"2024-02-23T00:36:37.124382+0000","last_became_peered":"2024-02-23T00:36:37.124382+0000","last_unstale":"2024-02-23T00:39:24.162655+0000","last_undegraded":"2024-02-23T00:39:24.162655+0000","last_fullsized":"2024-02-23T00:36:37.101474+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:24.016837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:24.016837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":738,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158014+0000","last_change":"2024-02-23T00:38:54.171994+0000","last_active":"2024-02-23T00:39:24.158014+0000","last_peered":"2024-02-23T00:39:24.158014+0000","last_clean":"2024-02-23T00:39:24.158014+0000","last_became_active":"2024-02-23T00:38:54.171563+0000","last_became_peered":"2024-02-23T00:38:54.171563+0000","last_unstale":"2024-02-23T00:39:24.158014+0000","last_undegraded":"2024-02-23T00:39:24.158014+0000","last_fullsized":"2024-02-23T00:39:24.158014+0000","mapping_epoch":516,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:06.122240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.122240+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":538,"reported_epoch":545,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:24.158970+0000","last_change":"2024-02-23T00:36:37.118733+0000","last_active":"2024-02-23T00:39:24.158970+0000","last_peered":"2024-02-23T00:39:24.158970+0000","last_clean":"2024-02-23T00:36:36.130983+0000","last_became_active":"2024-02-23T00:36:37.118733+0000","last_became_peered":"2024-02-23T00:36:37.118733+0000","last_unstale":"2024-02-23T00:39:24.158970+0000","last_undegraded":"2024-02-23T00:36:37.095672+0000","last_fullsized":"2024-02-23T00:36:37.095564+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:36:36.130868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.130868+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.181066+0000","last_change":"2024-02-23T00:38:49.220203+0000","last_active":"2024-02-23T00:39:24.181066+0000","last_peered":"2024-02-23T00:39:24.181066+0000","last_clean":"2024-02-23T00:39:24.181066+0000","last_became_active":"2024-02-23T00:38:49.219745+0000","last_became_peered":"2024-02-23T00:38:49.219745+0000","last_unstale":"2024-02-23T00:39:24.181066+0000","last_undegraded":"2024-02-23T00:39:24.181066+0000","last_fullsized":"2024-02-23T00:39:24.181066+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.172059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.172059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":462,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.197216+0000","last_change":"2024-02-23T00:36:28.184266+0000","last_active":"2024-02-23T00:39:24.197216+0000","last_peered":"2024-02-23T00:39:24.197216+0000","last_clean":"2024-02-23T00:39:24.197216+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:39:24.197216+0000","last_undegraded":"2024-02-23T00:39:24.197216+0000","last_fullsized":"2024-02-23T00:39:24.197216+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:28.184197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:28.184197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":237,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.181910+0000","last_change":"2024-02-23T00:38:54.181474+0000","last_active":"2024-02-23T00:39:24.181910+0000","last_peered":"2024-02-23T00:39:24.181910+0000","last_clean":"2024-02-23T00:39:24.181910+0000","last_became_active":"2024-02-23T00:38:54.180971+0000","last_became_peered":"2024-02-23T00:38:54.180971+0000","last_unstale":"2024-02-23T00:39:24.181910+0000","last_undegraded":"2024-02-23T00:39:24.181910+0000","last_fullsized":"2024-02-23T00:39:24.181910+0000","mapping_epoch":516,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":526,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.161778+0000","last_change":"2024-02-23T00:36:37.111836+0000","last_active":"2024-02-23T00:39:24.161778+0000","last_peered":"2024-02-23T00:39:24.161778+0000","last_clean":"2024-02-23T00:36:33.193391+0000","last_became_active":"2024-02-23T00:36:37.111836+0000","last_became_peered":"2024-02-23T00:36:37.111836+0000","last_unstale":"2024-02-23T00:39:24.161778+0000","last_undegraded":"2024-02-23T00:39:24.161778+0000","last_fullsized":"2024-02-23T00:36:37.101173+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.667517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.667517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":49,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.184467+0000","last_change":"2024-02-23T00:39:18.369649+0000","last_active":"2024-02-23T00:39:24.184467+0000","last_peered":"2024-02-23T00:39:24.184467+0000","last_clean":"2024-02-23T00:39:24.184467+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:39:24.184467+0000","last_undegraded":"2024-02-23T00:39:24.184467+0000","last_fullsized":"2024-02-23T00:39:24.184467+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:18.369579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:18.369579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":531,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.162069+0000","last_change":"2024-02-23T00:38:42.293920+0000","last_active":"2024-02-23T00:39:24.162069+0000","last_peered":"2024-02-23T00:39:24.162069+0000","last_clean":"2024-02-23T00:39:24.162069+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:39:24.162069+0000","last_undegraded":"2024-02-23T00:39:24.162069+0000","last_fullsized":"2024-02-23T00:39:24.162069+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.293824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.293824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.183491+0000","last_change":"2024-02-23T00:38:49.230851+0000","last_active":"2024-02-23T00:39:24.183491+0000","last_peered":"2024-02-23T00:39:24.183491+0000","last_clean":"2024-02-23T00:39:24.183491+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:39:24.183491+0000","last_undegraded":"2024-02-23T00:39:24.183491+0000","last_fullsized":"2024-02-23T00:39:24.183491+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:39.893315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.893315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":625,"reported_epoch":545,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-23T00:39:24.164145+0000","last_change":"2024-02-23T00:39:19.250055+0000","last_active":"2024-02-23T00:39:24.164145+0000","last_peered":"2024-02-23T00:39:24.164145+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:38:49.224732+0000","last_became_peered":"2024-02-23T00:38:49.224732+0000","last_unstale":"2024-02-23T00:39:24.164145+0000","last_undegraded":"2024-02-23T00:38:49.159472+0000","last_fullsized":"2024-02-23T00:38:49.159180+0000","mapping_epoch":511,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":524,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161435+0000","last_change":"2024-02-23T00:38:42.431748+0000","last_active":"2024-02-23T00:39:24.161435+0000","last_peered":"2024-02-23T00:39:24.161435+0000","last_clean":"2024-02-23T00:39:24.161435+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:39:24.161435+0000","last_undegraded":"2024-02-23T00:39:24.161435+0000","last_fullsized":"2024-02-23T00:39:24.161435+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.431641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":522,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.196677+0000","last_change":"2024-02-23T00:38:49.190282+0000","last_active":"2024-02-23T00:39:24.196677+0000","last_peered":"2024-02-23T00:39:24.196677+0000","last_clean":"2024-02-23T00:39:24.196677+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:39:24.196677+0000","last_undegraded":"2024-02-23T00:39:24.196677+0000","last_fullsized":"2024-02-23T00:39:24.196677+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.290946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.290946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":516,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.169542+0000","last_change":"2024-02-23T00:37:16.011330+0000","last_active":"2024-02-23T00:39:24.169542+0000","last_peered":"2024-02-23T00:39:24.169542+0000","last_clean":"2024-02-23T00:39:24.169542+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:39:24.169542+0000","last_undegraded":"2024-02-23T00:39:24.169542+0000","last_fullsized":"2024-02-23T00:39:24.169542+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:16.011248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:37:16.011248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":484,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161436+0000","last_change":"2024-02-23T00:38:49.195614+0000","last_active":"2024-02-23T00:39:24.161436+0000","last_peered":"2024-02-23T00:39:24.161436+0000","last_clean":"2024-02-23T00:39:24.161436+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:39:24.161436+0000","last_undegraded":"2024-02-23T00:39:24.161436+0000","last_fullsized":"2024-02-23T00:39:24.161436+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.316429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.316429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"453'32","reported_seq":60,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.183231+0000","last_change":"2024-02-23T00:38:51.793947+0000","last_active":"2024-02-23T00:39:24.183231+0000","last_peered":"2024-02-23T00:39:24.183231+0000","last_clean":"2024-02-23T00:39:24.183231+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:39:24.183231+0000","last_undegraded":"2024-02-23T00:39:24.183231+0000","last_fullsized":"2024-02-23T00:39:24.183231+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:38:44.240437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.240437+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":544,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.197051+0000","last_change":"2024-02-23T00:39:23.833122+0000","last_active":"2024-02-23T00:39:24.197051+0000","last_peered":"2024-02-23T00:39:24.197051+0000","last_clean":"2024-02-23T00:39:24.197051+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:39:24.197051+0000","last_undegraded":"2024-02-23T00:39:24.197051+0000","last_fullsized":"2024-02-23T00:39:24.197051+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:23.833059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:39:23.833059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"426'10","reported_seq":666,"reported_epoch":546,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:25.286059+0000","last_change":"2024-02-23T00:36:37.113534+0000","last_active":"2024-02-23T00:39:25.286059+0000","last_peered":"2024-02-23T00:39:25.286059+0000","last_clean":"2024-02-23T00:36:35.444798+0000","last_became_active":"2024-02-23T00:36:37.113534+0000","last_became_peered":"2024-02-23T00:36:37.113534+0000","last_unstale":"2024-02-23T00:39:25.286059+0000","last_undegraded":"2024-02-23T00:36:37.102235+0000","last_fullsized":"2024-02-23T00:36:37.102132+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.182663+0000","last_change":"2024-02-23T00:38:49.190032+0000","last_active":"2024-02-23T00:39:24.182663+0000","last_peered":"2024-02-23T00:39:24.182663+0000","last_clean":"2024-02-23T00:39:24.182663+0000","last_became_active":"2024-02-23T00:38:49.189564+0000","last_became_peered":"2024-02-23T00:38:49.189564+0000","last_unstale":"2024-02-23T00:39:24.182663+0000","last_undegraded":"2024-02-23T00:39:24.182663+0000","last_fullsized":"2024-02-23T00:39:24.182663+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:41.177673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:38:41.177673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":462,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.196916+0000","last_change":"2024-02-23T00:37:08.058170+0000","last_active":"2024-02-23T00:39:24.196916+0000","last_peered":"2024-02-23T00:39:24.196916+0000","last_clean":"2024-02-23T00:39:24.196916+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:39:24.196916+0000","last_undegraded":"2024-02-23T00:39:24.196916+0000","last_fullsized":"2024-02-23T00:39:24.196916+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:08.058075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.058075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":598,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.198243+0000","last_change":"2024-02-23T00:38:49.346222+0000","last_active":"2024-02-23T00:39:24.198243+0000","last_peered":"2024-02-23T00:39:24.198243+0000","last_clean":"2024-02-23T00:39:24.198243+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:39:24.198243+0000","last_undegraded":"2024-02-23T00:39:24.198243+0000","last_fullsized":"2024-02-23T00:39:24.198243+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:40.858800+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.858800+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":442,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.158857+0000","last_change":"2024-02-23T00:38:50.181211+0000","last_active":"2024-02-23T00:39:24.158857+0000","last_peered":"2024-02-23T00:39:24.158857+0000","last_clean":"2024-02-23T00:39:24.158857+0000","last_became_active":"2024-02-23T00:38:50.180740+0000","last_became_peered":"2024-02-23T00:38:50.180740+0000","last_unstale":"2024-02-23T00:39:24.158857+0000","last_undegraded":"2024-02-23T00:39:24.158857+0000","last_fullsized":"2024-02-23T00:39:24.158857+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:10.391997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:10.391997+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,7],"avail_no_missing":["4","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":56,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-02-23T00:39:25.159022+0000","last_change":"2024-02-23T00:38:49.384331+0000","last_active":"2024-02-23T00:39:25.159022+0000","last_peered":"2024-02-23T00:39:25.159022+0000","last_clean":"2024-02-23T00:39:25.159022+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:39:25.159022+0000","last_undegraded":"2024-02-23T00:39:25.159022+0000","last_fullsized":"2024-02-23T00:39:25.159022+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'10","last_scrub_stamp":"2024-02-23T00:37:27.870653+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:37:27.870653+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":65,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.198740+0000","last_change":"2024-02-23T00:38:49.184581+0000","last_active":"2024-02-23T00:39:24.198740+0000","last_peered":"2024-02-23T00:39:24.198740+0000","last_clean":"2024-02-23T00:39:24.198740+0000","last_became_active":"2024-02-23T00:38:49.183777+0000","last_became_peered":"2024-02-23T00:38:49.183777+0000","last_unstale":"2024-02-23T00:39:24.198740+0000","last_undegraded":"2024-02-23T00:39:24.198740+0000","last_fullsized":"2024-02-23T00:39:24.198740+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:05.095708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:05.095708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.184172+0000","last_change":"2024-02-23T00:38:49.227700+0000","last_active":"2024-02-23T00:39:24.184172+0000","last_peered":"2024-02-23T00:39:24.184172+0000","last_clean":"2024-02-23T00:39:24.184172+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:39:24.184172+0000","last_undegraded":"2024-02-23T00:39:24.184172+0000","last_fullsized":"2024-02-23T00:39:24.184172+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:09.447326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.447326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":538,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163430+0000","last_change":"2024-02-23T00:37:20.382193+0000","last_active":"2024-02-23T00:39:24.163430+0000","last_peered":"2024-02-23T00:39:24.163430+0000","last_clean":"2024-02-23T00:39:24.163430+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:39:24.163430+0000","last_undegraded":"2024-02-23T00:39:24.163430+0000","last_fullsized":"2024-02-23T00:39:24.163430+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:37:20.382108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.382108+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":542,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161873+0000","last_change":"2024-02-23T00:38:49.195750+0000","last_active":"2024-02-23T00:39:24.161873+0000","last_peered":"2024-02-23T00:39:24.161873+0000","last_clean":"2024-02-23T00:39:24.161873+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:39:24.161873+0000","last_undegraded":"2024-02-23T00:39:24.161873+0000","last_fullsized":"2024-02-23T00:39:24.161873+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.470333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.470333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":45,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.178678+0000","last_change":"2024-02-23T00:38:50.200217+0000","last_active":"2024-02-23T00:39:24.178678+0000","last_peered":"2024-02-23T00:39:24.178678+0000","last_clean":"2024-02-23T00:39:24.178678+0000","last_became_active":"2024-02-23T00:38:50.199759+0000","last_became_peered":"2024-02-23T00:38:50.199759+0000","last_unstale":"2024-02-23T00:39:24.178678+0000","last_undegraded":"2024-02-23T00:39:24.178678+0000","last_fullsized":"2024-02-23T00:39:24.178678+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:00.416493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:00.416493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":128,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.195908+0000","last_change":"2024-02-23T00:36:37.112999+0000","last_active":"2024-02-23T00:39:24.195908+0000","last_peered":"2024-02-23T00:39:24.195908+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.112999+0000","last_became_peered":"2024-02-23T00:36:37.112999+0000","last_unstale":"2024-02-23T00:39:24.195908+0000","last_undegraded":"2024-02-23T00:39:24.195908+0000","last_fullsized":"2024-02-23T00:36:37.102073+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.413991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.413991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":472,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.157708+0000","last_change":"2024-02-23T00:36:37.097581+0000","last_active":"2024-02-23T00:39:24.157708+0000","last_peered":"2024-02-23T00:39:24.157708+0000","last_clean":"2024-02-23T00:36:33.191411+0000","last_became_active":"2024-02-23T00:36:37.097581+0000","last_became_peered":"2024-02-23T00:36:37.097581+0000","last_unstale":"2024-02-23T00:39:24.157708+0000","last_undegraded":"2024-02-23T00:39:24.157708+0000","last_fullsized":"2024-02-23T00:36:37.093707+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.101501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.101501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":574,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158808+0000","last_change":"2024-02-23T00:38:16.174640+0000","last_active":"2024-02-23T00:39:24.158808+0000","last_peered":"2024-02-23T00:39:24.158808+0000","last_clean":"2024-02-23T00:39:24.158808+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:39:24.158808+0000","last_undegraded":"2024-02-23T00:39:24.158808+0000","last_fullsized":"2024-02-23T00:39:24.158808+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:38:16.174526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.174526+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":516,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.161720+0000","last_change":"2024-02-23T00:36:37.108194+0000","last_active":"2024-02-23T00:39:24.161720+0000","last_peered":"2024-02-23T00:39:24.161720+0000","last_clean":"2024-02-23T00:36:33.367482+0000","last_became_active":"2024-02-23T00:36:37.108194+0000","last_became_peered":"2024-02-23T00:36:37.108194+0000","last_unstale":"2024-02-23T00:39:24.161720+0000","last_undegraded":"2024-02-23T00:39:24.161720+0000","last_fullsized":"2024-02-23T00:36:37.096730+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.783089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.783089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":529,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163494+0000","last_change":"2024-02-23T00:38:49.225280+0000","last_active":"2024-02-23T00:39:24.163494+0000","last_peered":"2024-02-23T00:39:24.163494+0000","last_clean":"2024-02-23T00:39:24.163494+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:39:24.163494+0000","last_undegraded":"2024-02-23T00:39:24.163494+0000","last_fullsized":"2024-02-23T00:39:24.163494+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:06.235608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.235608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":50,"reported_epoch":546,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:25.312118+0000","last_change":"2024-02-23T00:39:25.312118+0000","last_active":"2024-02-23T00:39:25.312118+0000","last_peered":"2024-02-23T00:39:25.312118+0000","last_clean":"2024-02-23T00:39:25.312118+0000","last_became_active":"2024-02-23T00:38:50.179247+0000","last_became_peered":"2024-02-23T00:38:50.179247+0000","last_unstale":"2024-02-23T00:39:25.312118+0000","last_undegraded":"2024-02-23T00:39:25.312118+0000","last_fullsized":"2024-02-23T00:39:25.312118+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:25.311952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:39:25.311952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,2],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":515,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161721+0000","last_change":"2024-02-23T00:39:22.387191+0000","last_active":"2024-02-23T00:39:24.161721+0000","last_peered":"2024-02-23T00:39:24.161721+0000","last_clean":"2024-02-23T00:39:24.161721+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:39:24.161721+0000","last_undegraded":"2024-02-23T00:39:24.161721+0000","last_fullsized":"2024-02-23T00:39:24.161721+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:39:22.387089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.387089+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":50,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.176290+0000","last_change":"2024-02-23T00:38:49.987321+0000","last_active":"2024-02-23T00:39:24.176290+0000","last_peered":"2024-02-23T00:39:24.176290+0000","last_clean":"2024-02-23T00:39:24.176290+0000","last_became_active":"2024-02-23T00:38:49.217612+0000","last_became_peered":"2024-02-23T00:38:49.217612+0000","last_unstale":"2024-02-23T00:39:24.176290+0000","last_undegraded":"2024-02-23T00:39:24.176290+0000","last_fullsized":"2024-02-23T00:39:24.176290+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:36:36.531424+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.531424+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":513,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.196794+0000","last_change":"2024-02-23T00:36:32.199878+0000","last_active":"2024-02-23T00:39:24.196794+0000","last_peered":"2024-02-23T00:39:24.196794+0000","last_clean":"2024-02-23T00:39:24.196794+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:39:24.196794+0000","last_undegraded":"2024-02-23T00:39:24.196794+0000","last_fullsized":"2024-02-23T00:39:24.196794+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.199765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.199765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":515,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161335+0000","last_change":"2024-02-23T00:38:02.325317+0000","last_active":"2024-02-23T00:39:24.161335+0000","last_peered":"2024-02-23T00:39:24.161335+0000","last_clean":"2024-02-23T00:39:24.161335+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:39:24.161335+0000","last_undegraded":"2024-02-23T00:39:24.161335+0000","last_fullsized":"2024-02-23T00:39:24.161335+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:02.325254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:02.325254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":516,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.169075+0000","last_change":"2024-02-23T00:37:14.009706+0000","last_active":"2024-02-23T00:39:24.169075+0000","last_peered":"2024-02-23T00:39:24.169075+0000","last_clean":"2024-02-23T00:39:24.169075+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:39:24.169075+0000","last_undegraded":"2024-02-23T00:39:24.169075+0000","last_fullsized":"2024-02-23T00:39:24.169075+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.009614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.009614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":462,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163509+0000","last_change":"2024-02-23T00:38:33.242488+0000","last_active":"2024-02-23T00:39:24.163509+0000","last_peered":"2024-02-23T00:39:24.163509+0000","last_clean":"2024-02-23T00:39:24.163509+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:39:24.163509+0000","last_undegraded":"2024-02-23T00:39:24.163509+0000","last_fullsized":"2024-02-23T00:39:24.163509+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:33.242417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:33.242417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":61,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.177740+0000","last_change":"2024-02-23T00:38:51.365185+0000","last_active":"2024-02-23T00:39:24.177740+0000","last_peered":"2024-02-23T00:39:24.177740+0000","last_clean":"2024-02-23T00:39:24.177740+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:39:24.177740+0000","last_undegraded":"2024-02-23T00:39:24.177740+0000","last_fullsized":"2024-02-23T00:39:24.177740+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:37:26.914698+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.914698+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":518,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161248+0000","last_change":"2024-02-23T00:37:22.340222+0000","last_active":"2024-02-23T00:39:24.161248+0000","last_peered":"2024-02-23T00:39:24.161248+0000","last_clean":"2024-02-23T00:39:24.161248+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:39:24.161248+0000","last_undegraded":"2024-02-23T00:39:24.161248+0000","last_fullsized":"2024-02-23T00:39:24.161248+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:22.340128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.340128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":518,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.169263+0000","last_change":"2024-02-23T00:38:16.055268+0000","last_active":"2024-02-23T00:39:24.169263+0000","last_peered":"2024-02-23T00:39:24.169263+0000","last_clean":"2024-02-23T00:39:24.169263+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:39:24.169263+0000","last_undegraded":"2024-02-23T00:39:24.169263+0000","last_fullsized":"2024-02-23T00:39:24.169263+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.055159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.178474+0000","last_change":"2024-02-23T00:38:50.184807+0000","last_active":"2024-02-23T00:39:24.178474+0000","last_peered":"2024-02-23T00:39:24.178474+0000","last_clean":"2024-02-23T00:39:24.178474+0000","last_became_active":"2024-02-23T00:38:50.184585+0000","last_became_peered":"2024-02-23T00:38:50.184585+0000","last_unstale":"2024-02-23T00:39:24.178474+0000","last_undegraded":"2024-02-23T00:39:24.178474+0000","last_fullsized":"2024-02-23T00:39:24.178474+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:57.175171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:57.175171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.178602+0000","last_change":"2024-02-23T00:38:49.189171+0000","last_active":"2024-02-23T00:39:24.178602+0000","last_peered":"2024-02-23T00:39:24.178602+0000","last_clean":"2024-02-23T00:39:24.178602+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:39:24.178602+0000","last_undegraded":"2024-02-23T00:39:24.178602+0000","last_fullsized":"2024-02-23T00:39:24.178602+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:37.979785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.979785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":639,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.169973+0000","last_change":"2024-02-23T00:38:54.170696+0000","last_active":"2024-02-23T00:39:24.169973+0000","last_peered":"2024-02-23T00:39:24.169973+0000","last_clean":"2024-02-23T00:39:24.169973+0000","last_became_active":"2024-02-23T00:38:54.170041+0000","last_became_peered":"2024-02-23T00:38:54.170041+0000","last_unstale":"2024-02-23T00:39:24.169973+0000","last_undegraded":"2024-02-23T00:39:24.169973+0000","last_fullsized":"2024-02-23T00:39:24.169973+0000","mapping_epoch":516,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,1],"avail_no_missing":["7","1","3"],"object_location_counts":[{"shards":"1,3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":522,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.195782+0000","last_change":"2024-02-23T00:38:36.853659+0000","last_active":"2024-02-23T00:39:24.195782+0000","last_peered":"2024-02-23T00:39:24.195782+0000","last_clean":"2024-02-23T00:39:24.195782+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:39:24.195782+0000","last_undegraded":"2024-02-23T00:39:24.195782+0000","last_fullsized":"2024-02-23T00:39:24.195782+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:36.853582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:36.853582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":677,"reported_epoch":546,"state":"active+clean","last_fresh":"2024-02-23T00:39:25.158111+0000","last_change":"2024-02-23T00:38:37.455385+0000","last_active":"2024-02-23T00:39:25.158111+0000","last_peered":"2024-02-23T00:39:25.158111+0000","last_clean":"2024-02-23T00:39:25.158111+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:39:25.158111+0000","last_undegraded":"2024-02-23T00:39:25.158111+0000","last_fullsized":"2024-02-23T00:39:25.158111+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:38:37.455280+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.455280+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":48,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.183583+0000","last_change":"2024-02-23T00:38:51.364248+0000","last_active":"2024-02-23T00:39:24.183583+0000","last_peered":"2024-02-23T00:39:24.183583+0000","last_clean":"2024-02-23T00:39:24.183583+0000","last_became_active":"2024-02-23T00:38:49.230008+0000","last_became_peered":"2024-02-23T00:38:49.230008+0000","last_unstale":"2024-02-23T00:39:24.183583+0000","last_undegraded":"2024-02-23T00:39:24.183583+0000","last_fullsized":"2024-02-23T00:39:24.183583+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:38:09.207232+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:38:09.207232+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.207232+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":490,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.169328+0000","last_change":"2024-02-23T00:38:50.185706+0000","last_active":"2024-02-23T00:39:24.169328+0000","last_peered":"2024-02-23T00:39:24.169328+0000","last_clean":"2024-02-23T00:39:24.169328+0000","last_became_active":"2024-02-23T00:38:50.185295+0000","last_became_peered":"2024-02-23T00:38:50.185295+0000","last_unstale":"2024-02-23T00:39:24.169328+0000","last_undegraded":"2024-02-23T00:39:24.169328+0000","last_fullsized":"2024-02-23T00:39:24.169328+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.028166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.028166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,2],"avail_no_missing":["7","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":682,"reported_epoch":545,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:39:24.163871+0000","last_change":"2024-02-23T00:39:22.606140+0000","last_active":"2024-02-23T00:39:24.163871+0000","last_peered":"2024-02-23T00:39:24.163871+0000","last_clean":"2024-02-23T00:38:19.193308+0000","last_became_active":"2024-02-23T00:38:22.159363+0000","last_became_peered":"2024-02-23T00:38:22.159363+0000","last_unstale":"2024-02-23T00:39:24.163871+0000","last_undegraded":"2024-02-23T00:39:24.163871+0000","last_fullsized":"2024-02-23T00:39:24.163871+0000","mapping_epoch":484,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":45,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.177591+0000","last_change":"2024-02-23T00:38:50.176938+0000","last_active":"2024-02-23T00:39:24.177591+0000","last_peered":"2024-02-23T00:39:24.177591+0000","last_clean":"2024-02-23T00:39:24.177591+0000","last_became_active":"2024-02-23T00:38:50.176504+0000","last_became_peered":"2024-02-23T00:38:50.176504+0000","last_unstale":"2024-02-23T00:39:24.177591+0000","last_undegraded":"2024-02-23T00:39:24.177591+0000","last_fullsized":"2024-02-23T00:39:24.177591+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:35:02.151731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":128,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.170541+0000","last_change":"2024-02-23T00:36:37.122400+0000","last_active":"2024-02-23T00:39:24.170541+0000","last_peered":"2024-02-23T00:39:24.170541+0000","last_clean":"2024-02-23T00:27:20.002687+0000","last_became_active":"2024-02-23T00:36:37.122400+0000","last_became_peered":"2024-02-23T00:36:37.122400+0000","last_unstale":"2024-02-23T00:39:24.170541+0000","last_undegraded":"2024-02-23T00:39:24.170541+0000","last_fullsized":"2024-02-23T00:36:37.111280+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":62,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.198825+0000","last_change":"2024-02-23T00:38:42.787673+0000","last_active":"2024-02-23T00:39:24.198825+0000","last_peered":"2024-02-23T00:39:24.198825+0000","last_clean":"2024-02-23T00:39:24.198825+0000","last_became_active":"2024-02-23T00:38:40.171498+0000","last_became_peered":"2024-02-23T00:38:40.171498+0000","last_unstale":"2024-02-23T00:39:24.198825+0000","last_undegraded":"2024-02-23T00:39:24.198825+0000","last_fullsized":"2024-02-23T00:39:24.198825+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.787571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.787571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,4],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":470,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.170271+0000","last_change":"2024-02-23T00:38:35.044376+0000","last_active":"2024-02-23T00:39:24.170271+0000","last_peered":"2024-02-23T00:39:24.170271+0000","last_clean":"2024-02-23T00:39:24.170271+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:39:24.170271+0000","last_undegraded":"2024-02-23T00:39:24.170271+0000","last_fullsized":"2024-02-23T00:39:24.170271+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:35.044298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.044298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":653,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161445+0000","last_change":"2024-02-23T00:38:52.159547+0000","last_active":"2024-02-23T00:39:24.161445+0000","last_peered":"2024-02-23T00:39:24.161445+0000","last_clean":"2024-02-23T00:39:24.161445+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:39:24.161445+0000","last_undegraded":"2024-02-23T00:39:24.161445+0000","last_fullsized":"2024-02-23T00:39:24.161445+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:38:39.248198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.248198+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":527,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161898+0000","last_change":"2024-02-23T00:38:07.395392+0000","last_active":"2024-02-23T00:39:24.161898+0000","last_peered":"2024-02-23T00:39:24.161898+0000","last_clean":"2024-02-23T00:39:24.161898+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:39:24.161898+0000","last_undegraded":"2024-02-23T00:39:24.161898+0000","last_fullsized":"2024-02-23T00:39:24.161898+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.395260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.395260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":514,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.161989+0000","last_change":"2024-02-23T00:36:37.113234+0000","last_active":"2024-02-23T00:39:24.161989+0000","last_peered":"2024-02-23T00:39:24.161989+0000","last_clean":"2024-02-23T00:36:33.367528+0000","last_became_active":"2024-02-23T00:36:37.113234+0000","last_became_peered":"2024-02-23T00:36:37.113234+0000","last_unstale":"2024-02-23T00:39:24.161989+0000","last_undegraded":"2024-02-23T00:39:24.161989+0000","last_fullsized":"2024-02-23T00:36:37.097125+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.791748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.791748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.182327+0000","last_change":"2024-02-23T00:38:49.227843+0000","last_active":"2024-02-23T00:39:24.182327+0000","last_peered":"2024-02-23T00:39:24.182327+0000","last_clean":"2024-02-23T00:39:24.182327+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:39:24.182327+0000","last_undegraded":"2024-02-23T00:39:24.182327+0000","last_fullsized":"2024-02-23T00:39:24.182327+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:27.951690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.951690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":628,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.181654+0000","last_change":"2024-02-23T00:38:55.178058+0000","last_active":"2024-02-23T00:39:24.181654+0000","last_peered":"2024-02-23T00:39:24.181654+0000","last_clean":"2024-02-23T00:39:24.181654+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:39:24.181654+0000","last_undegraded":"2024-02-23T00:39:24.181654+0000","last_fullsized":"2024-02-23T00:39:24.181654+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":29,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.181004+0000","last_change":"2024-02-23T00:38:49.192589+0000","last_active":"2024-02-23T00:39:24.181004+0000","last_peered":"2024-02-23T00:39:24.181004+0000","last_clean":"2024-02-23T00:39:24.181004+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:39:24.181004+0000","last_undegraded":"2024-02-23T00:39:24.181004+0000","last_fullsized":"2024-02-23T00:39:24.181004+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:35.929507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:35.929507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":533,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158344+0000","last_change":"2024-02-23T00:38:49.224957+0000","last_active":"2024-02-23T00:39:24.158344+0000","last_peered":"2024-02-23T00:39:24.158344+0000","last_clean":"2024-02-23T00:39:24.158344+0000","last_became_active":"2024-02-23T00:38:49.224630+0000","last_became_peered":"2024-02-23T00:38:49.224630+0000","last_unstale":"2024-02-23T00:39:24.158344+0000","last_undegraded":"2024-02-23T00:39:24.158344+0000","last_fullsized":"2024-02-23T00:39:24.158344+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.063485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.063485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":462,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.198706+0000","last_change":"2024-02-23T00:38:03.798894+0000","last_active":"2024-02-23T00:39:24.198706+0000","last_peered":"2024-02-23T00:39:24.198706+0000","last_clean":"2024-02-23T00:39:24.198706+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:39:24.198706+0000","last_undegraded":"2024-02-23T00:39:24.198706+0000","last_fullsized":"2024-02-23T00:39:24.198706+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.798824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.798824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":664,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.176505+0000","last_change":"2024-02-23T00:39:21.295448+0000","last_active":"2024-02-23T00:39:24.176505+0000","last_peered":"2024-02-23T00:39:24.176505+0000","last_clean":"2024-02-23T00:39:24.176505+0000","last_became_active":"2024-02-23T00:39:21.294883+0000","last_became_peered":"2024-02-23T00:39:21.294883+0000","last_unstale":"2024-02-23T00:39:24.176505+0000","last_undegraded":"2024-02-23T00:39:24.176505+0000","last_fullsized":"2024-02-23T00:39:24.176505+0000","mapping_epoch":541,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,1],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":229,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157827+0000","last_change":"2024-02-23T00:37:59.129696+0000","last_active":"2024-02-23T00:39:24.157827+0000","last_peered":"2024-02-23T00:39:24.157827+0000","last_clean":"2024-02-23T00:39:24.157827+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:39:24.157827+0000","last_undegraded":"2024-02-23T00:39:24.157827+0000","last_fullsized":"2024-02-23T00:39:24.157827+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.129604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.129604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":510,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.197755+0000","last_change":"2024-02-23T00:36:37.124006+0000","last_active":"2024-02-23T00:39:24.197755+0000","last_peered":"2024-02-23T00:39:24.197755+0000","last_clean":"2024-02-23T00:36:33.366419+0000","last_became_active":"2024-02-23T00:36:37.124006+0000","last_became_peered":"2024-02-23T00:36:37.124006+0000","last_unstale":"2024-02-23T00:39:24.197755+0000","last_undegraded":"2024-02-23T00:39:24.197755+0000","last_fullsized":"2024-02-23T00:36:37.103625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":73,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.168361+0000","last_change":"2024-02-23T00:38:49.208012+0000","last_active":"2024-02-23T00:39:24.168361+0000","last_peered":"2024-02-23T00:39:24.168361+0000","last_clean":"2024-02-23T00:39:24.168361+0000","last_became_active":"2024-02-23T00:38:49.207582+0000","last_became_peered":"2024-02-23T00:38:49.207582+0000","last_unstale":"2024-02-23T00:39:24.168361+0000","last_undegraded":"2024-02-23T00:39:24.168361+0000","last_fullsized":"2024-02-23T00:39:24.168361+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:39.957272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.957272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":470,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.168397+0000","last_change":"2024-02-23T00:37:18.044814+0000","last_active":"2024-02-23T00:39:24.168397+0000","last_peered":"2024-02-23T00:39:24.168397+0000","last_clean":"2024-02-23T00:39:24.168397+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:39:24.168397+0000","last_undegraded":"2024-02-23T00:39:24.168397+0000","last_fullsized":"2024-02-23T00:39:24.168397+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:18.044737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:18.044737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":214,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.158228+0000","last_change":"2024-02-23T00:39:21.278362+0000","last_active":"2024-02-23T00:39:24.158228+0000","last_peered":"2024-02-23T00:39:24.158228+0000","last_clean":"2024-02-23T00:39:24.158228+0000","last_became_active":"2024-02-23T00:39:21.278024+0000","last_became_peered":"2024-02-23T00:39:21.278024+0000","last_unstale":"2024-02-23T00:39:24.158228+0000","last_undegraded":"2024-02-23T00:39:24.158228+0000","last_fullsized":"2024-02-23T00:39:24.158228+0000","mapping_epoch":541,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":523,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.169634+0000","last_change":"2024-02-23T00:36:37.114846+0000","last_active":"2024-02-23T00:39:24.169634+0000","last_peered":"2024-02-23T00:39:24.169634+0000","last_clean":"2024-02-23T00:36:33.367758+0000","last_became_active":"2024-02-23T00:36:37.114846+0000","last_became_peered":"2024-02-23T00:36:37.114846+0000","last_unstale":"2024-02-23T00:39:24.169634+0000","last_undegraded":"2024-02-23T00:39:24.169634+0000","last_fullsized":"2024-02-23T00:36:37.107709+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":518,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.168967+0000","last_change":"2024-02-23T00:37:19.034254+0000","last_active":"2024-02-23T00:39:24.168967+0000","last_peered":"2024-02-23T00:39:24.168967+0000","last_clean":"2024-02-23T00:39:24.168967+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:39:24.168967+0000","last_undegraded":"2024-02-23T00:39:24.168967+0000","last_fullsized":"2024-02-23T00:39:24.168967+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:19.034136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:37:19.034136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":60,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.168211+0000","last_change":"2024-02-23T00:38:43.922541+0000","last_active":"2024-02-23T00:39:24.168211+0000","last_peered":"2024-02-23T00:39:24.168211+0000","last_clean":"2024-02-23T00:39:24.168211+0000","last_became_active":"2024-02-23T00:38:40.180445+0000","last_became_peered":"2024-02-23T00:38:40.180445+0000","last_unstale":"2024-02-23T00:39:24.168211+0000","last_undegraded":"2024-02-23T00:39:24.168211+0000","last_fullsized":"2024-02-23T00:39:24.168211+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:43.922442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:38:43.922442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,1],"avail_no_missing":["7","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":486,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.163120+0000","last_change":"2024-02-23T00:39:21.439880+0000","last_active":"2024-02-23T00:39:24.163120+0000","last_peered":"2024-02-23T00:39:24.163120+0000","last_clean":"2024-02-23T00:39:24.163120+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:39:24.163120+0000","last_undegraded":"2024-02-23T00:39:24.163120+0000","last_fullsized":"2024-02-23T00:39:24.163120+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:21.439783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:21.439783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":671,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.165373+0000","last_change":"2024-02-23T00:39:24.165093+0000","last_active":"2024-02-23T00:39:24.165373+0000","last_peered":"2024-02-23T00:39:24.165373+0000","last_clean":"2024-02-23T00:39:24.165373+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:39:24.165373+0000","last_undegraded":"2024-02-23T00:39:24.165373+0000","last_fullsized":"2024-02-23T00:39:24.165373+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":45,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.182408+0000","last_change":"2024-02-23T00:38:50.183535+0000","last_active":"2024-02-23T00:39:24.182408+0000","last_peered":"2024-02-23T00:39:24.182408+0000","last_clean":"2024-02-23T00:39:24.182408+0000","last_became_active":"2024-02-23T00:38:50.183147+0000","last_became_peered":"2024-02-23T00:38:50.183147+0000","last_unstale":"2024-02-23T00:39:24.182408+0000","last_undegraded":"2024-02-23T00:39:24.182408+0000","last_fullsized":"2024-02-23T00:39:24.182408+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:54.388834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:54.388834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,2],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":170,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.160392+0000","last_change":"2024-02-23T00:38:00.381029+0000","last_active":"2024-02-23T00:39:24.160392+0000","last_peered":"2024-02-23T00:39:24.160392+0000","last_clean":"2024-02-23T00:39:24.160392+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:39:24.160392+0000","last_undegraded":"2024-02-23T00:39:24.160392+0000","last_fullsized":"2024-02-23T00:39:24.160392+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.380938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.380938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":57,"reported_epoch":545,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:24.196615+0000","last_change":"2024-02-23T00:38:40.203064+0000","last_active":"2024-02-23T00:39:24.196615+0000","last_peered":"2024-02-23T00:39:24.196615+0000","last_clean":"2024-02-23T00:39:24.196615+0000","last_became_active":"2024-02-23T00:38:40.202564+0000","last_became_peered":"2024-02-23T00:38:40.202564+0000","last_unstale":"2024-02-23T00:39:24.196615+0000","last_undegraded":"2024-02-23T00:39:24.196615+0000","last_fullsized":"2024-02-23T00:39:24.196615+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:58.130479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:37:58.130479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":130,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.161709+0000","last_change":"2024-02-23T00:36:37.119114+0000","last_active":"2024-02-23T00:39:24.161709+0000","last_peered":"2024-02-23T00:39:24.161709+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119114+0000","last_became_peered":"2024-02-23T00:36:37.119114+0000","last_unstale":"2024-02-23T00:39:24.161709+0000","last_undegraded":"2024-02-23T00:39:24.161709+0000","last_fullsized":"2024-02-23T00:36:37.106746+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.176441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.176441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":733,"reported_epoch":545,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:39:24.162586+0000","last_change":"2024-02-23T00:39:19.216599+0000","last_active":"2024-02-23T00:39:24.162586+0000","last_peered":"2024-02-23T00:39:24.162586+0000","last_clean":"2024-02-23T00:38:47.148293+0000","last_became_active":"2024-02-23T00:38:49.185201+0000","last_became_peered":"2024-02-23T00:38:49.185201+0000","last_unstale":"2024-02-23T00:39:24.162586+0000","last_undegraded":"2024-02-23T00:39:24.162586+0000","last_fullsized":"2024-02-23T00:39:24.162586+0000","mapping_epoch":511,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:34.287816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.287816+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[2,5,1],"avail_no_missing":["2","1","5"],"object_location_counts":[{"shards":"1,2,5","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":128,"reported_epoch":545,"state":"active+undersized","last_fresh":"2024-02-23T00:39:24.169698+0000","last_change":"2024-02-23T00:36:37.120482+0000","last_active":"2024-02-23T00:39:24.169698+0000","last_peered":"2024-02-23T00:39:24.169698+0000","last_clean":"2024-02-23T00:27:15.989319+0000","last_became_active":"2024-02-23T00:36:37.120482+0000","last_became_peered":"2024-02-23T00:36:37.120482+0000","last_unstale":"2024-02-23T00:39:24.169698+0000","last_undegraded":"2024-02-23T00:39:24.169698+0000","last_fullsized":"2024-02-23T00:36:37.111071+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":515,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.157577+0000","last_change":"2024-02-23T00:37:14.200716+0000","last_active":"2024-02-23T00:39:24.157577+0000","last_peered":"2024-02-23T00:39:24.157577+0000","last_clean":"2024-02-23T00:39:24.157577+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:39:24.157577+0000","last_undegraded":"2024-02-23T00:39:24.157577+0000","last_fullsized":"2024-02-23T00:39:24.157577+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.200606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.200606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":44,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.178393+0000","last_change":"2024-02-23T00:38:49.220042+0000","last_active":"2024-02-23T00:39:24.178393+0000","last_peered":"2024-02-23T00:39:24.178393+0000","last_clean":"2024-02-23T00:39:24.178393+0000","last_became_active":"2024-02-23T00:38:49.219344+0000","last_became_peered":"2024-02-23T00:38:49.219344+0000","last_unstale":"2024-02-23T00:39:24.178393+0000","last_undegraded":"2024-02-23T00:39:24.178393+0000","last_fullsized":"2024-02-23T00:39:24.178393+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:24.990671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:24.990671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":480,"reported_epoch":545,"state":"active+clean","last_fresh":"2024-02-23T00:39:24.161513+0000","last_change":"2024-02-23T00:38:49.228307+0000","last_active":"2024-02-23T00:39:24.161513+0000","last_peered":"2024-02-23T00:39:24.161513+0000","last_clean":"2024-02-23T00:39:24.161513+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:39:24.161513+0000","last_undegraded":"2024-02-23T00:39:24.161513+0000","last_fullsized":"2024-02-23T00:39:24.161513+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:40.485283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:40.485283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":63,"num_pg":32,"stat_sum":{"num_bytes":4020435,"num_objects":192,"num_object_clones":11,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":62,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":212,"num_read_kb":57,"num_write":777,"num_write_kb":4619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8000615,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14274560,"data_stored":12057719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2057,"ondisk_log_size":2057,"up":84,"acting":85,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":90,"acting":92,"num_store_stats":8},{"poolid":6,"num_pg":30,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":86,"num_store_stats":8},{"poolid":227,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":7},{"poolid":5,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":110,"acting":112,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":71,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":1323,"num_read_kb":1077,"num_write":794,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":4401,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":86,"acting":90,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":406,"num_read_kb":406,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":34,"acting":37,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":82,"acting":86,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332245,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311976,"kb_used_data":6560,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433176,"statfs":{"total":95995035648,"available":95675572224,"internally_reserved":0,"allocated":6717440,"data_stored":3715556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.02,"15min":1.024},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.79800000000000004,"5min":57.401000000000003,"15min":61.048999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.1719999999999999,"15min":1.2450000000000001},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":65.558999999999997,"15min":73.870999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.51600000000000001,"15min":0.68999999999999995},"min":{"1min":0.23999999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":0.81699999999999995,"5min":4.6840000000000002,"15min":31.175000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.50600000000000001,"15min":0.68300000000000005},"min":{"1min":0.28599999999999998,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":0.57499999999999996,"5min":3.8719999999999999,"15min":29.423999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.55900000000000005,"15min":0.68000000000000005},"min":{"1min":0.33100000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.0099999999999998,"5min":8.0099999999999998,"15min":16.603999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.55900000000000005,"15min":0.60999999999999999},"min":{"1min":0.318,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66000000000000003,"5min":4.1369999999999996,"15min":7.5339999999999998},"last":0.47599999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.58999999999999997,"15min":0.996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":8.0449999999999999,"5min":8.0449999999999999,"15min":57.729999999999997},"last":0.501},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.56999999999999995,"15min":0.90400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69099999999999995,"5min":3.8420000000000001,"15min":27.573},"last":0.57999999999999996}]},{"osd":4,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.77700000000000002,"5min":1.2310000000000001,"15min":1.706},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.48199999999999998,"15min":0.45800000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":8.1359999999999992,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.33900000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42499999999999999,"15min":0.43099999999999999},"min":{"1min":0.29299999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":1.0209999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.46899999999999997,"15min":0.45600000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.84099999999999997,"5min":1.1919999999999999,"15min":1.1950000000000001},"last":0.36699999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.28000000000000003},"max":{"1min":0.86099999999999999,"5min":1.26,"15min":2.1339999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.51800000000000002,"15min":0.48799999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.194,"5min":5.194,"15min":5.194},"last":0.35999999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561178,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316128,"kb_used_data":6616,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93429024,"statfs":{"total":95995035648,"available":95671320576,"internally_reserved":0,"allocated":6774784,"data_stored":3718660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.6499999999999999,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.78800000000000003,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64800000000000002,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":9.468,"15min":9.468},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.56299999999999994,"15min":1.0309999999999999},"min":{"1min":0.28699999999999998,"5min":0.253,"15min":0.245},"max":{"1min":6.056,"5min":6.056,"15min":59.347999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.4279999999999999,"5min":0.68300000000000005,"15min":1.3500000000000001},"min":{"1min":0.30299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":18.423999999999999,"5min":18.423999999999999,"15min":87.221000000000004},"last":0.35499999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":0.70599999999999996,"15min":0.875},"min":{"1min":0.34000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":18.381,"5min":18.381,"15min":25.550999999999998},"last":3.3330000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.60499999999999998,"15min":0.88600000000000001},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.1749999999999998,"5min":6.1749999999999998,"15min":40.914000000000001},"last":2.5190000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.57799999999999996,"15min":0.82799999999999996},"min":{"1min":0.373,"5min":0.317,"15min":0.23400000000000001},"max":{"1min":6.1200000000000001,"5min":6.1200000000000001,"15min":31.309999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.476,"5min":0.72799999999999998,"15min":1.3879999999999999},"min":{"1min":0.378,"5min":0.30099999999999999,"15min":0.29899999999999999},"max":{"1min":18.448,"5min":18.448,"15min":86.311999999999998},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.47199999999999998,"15min":0.44900000000000001},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.1500000000000004,"5min":6.1500000000000004,"15min":6.1500000000000004},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.496,"15min":0.52700000000000002},"min":{"1min":0.247,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":17.718},"last":0.33200000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.49199999999999999,"15min":0.499},"min":{"1min":0.29599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":8.0990000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.51600000000000001,"15min":0.5},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":6.0780000000000003,"5min":6.0780000000000003,"15min":8.1370000000000005},"last":0.26800000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:38:30 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.54000000000000004,"15min":0.61399999999999999},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.4279999999999999,"5min":6.4279999999999999,"15min":17.881},"last":0.378},{"interface":"front","average":{"1min":1.401,"5min":0.67500000000000004,"15min":0.65200000000000002},"min":{"1min":0.30099999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":18.475000000000001,"5min":18.475000000000001,"15min":18.475000000000001},"last":0.29799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575025,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320252,"kb_used_data":10740,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93424900,"statfs":{"total":95995035648,"available":95667097600,"internally_reserved":0,"allocated":10997760,"data_stored":7727279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.29,"15min":0.66300000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.51700000000000002,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.89600000000000002,"15min":0.55200000000000005},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60499999999999998,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32700000000000001,"15min":0.66600000000000004},"min":{"1min":0.215,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.44800000000000001,"5min":0.84099999999999997,"15min":35.313000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.312,"5min":0.33400000000000002,"15min":0.57099999999999995},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20100000000000001},"max":{"1min":0.54100000000000004,"5min":0.61799999999999999,"15min":27.908999999999999},"last":0.307}]},{"osd":3,"last update":"Fri Feb 23 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.41399999999999998,"15min":0.65900000000000003},"min":{"1min":0.26500000000000001,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":3.8119999999999998,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.40000000000000002,"15min":0.65000000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.38600000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:39:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.56299999999999994,"15min":0.70399999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.4900000000000002,"5min":3.4900000000000002,"15min":16.231000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52100000000000002,"15min":0.72099999999999997},"min":{"1min":0.29599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.518,"5min":1.518,"15min":29.212},"last":0.42699999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:38:44 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52400000000000002,"15min":0.71499999999999997},"min":{"1min":0.34100000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.71099999999999997,"5min":0.90500000000000003,"15min":16.294},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53100000000000003,"15min":0.67600000000000005},"min":{"1min":0.374,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":11.105},"last":0.40999999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:39:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.52100000000000002,"15min":0.755},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.622,"5min":0.94799999999999995,"15min":29.187000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.57799999999999996,"15min":0.749},"min":{"1min":0.39200000000000002,"5min":0.312,"15min":0.312},"max":{"1min":2.528,"5min":2.528,"15min":16.369},"last":0.48399999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:38:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55800000000000005,"15min":0.76400000000000001},"min":{"1min":0.36499999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.65400000000000003,"5min":0.86099999999999999,"15min":16.395},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55600000000000005,"15min":0.66000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":11.135999999999999},"last":0.36399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318432,"kb_used_data":4824,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426720,"statfs":{"total":95995035648,"available":95668961280,"internally_reserved":0,"allocated":4939776,"data_stored":2599387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.80600000000000005,"15min":0.495},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47799999999999998,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.246},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.165,"15min":0.61899999999999999},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.997,"15min":0.58899999999999997},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.64400000000000002,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.748,"15min":0.48999999999999999},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.22700000000000001},"max":{"1min":0.51100000000000001,"5min":15.965,"15min":15.965},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":1.0820000000000001,"15min":0.65000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.0369999999999999,"15min":0.63700000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.58799999999999997,"5min":16.006,"15min":16.006},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.73399999999999999,"15min":0.63800000000000001},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84699999999999998,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.38300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83699999999999997,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.071,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86799999999999999,"5min":13.311,"15min":13.311},"last":0.499},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.113,"15min":0.74399999999999999},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":23.744,"15min":23.744},"last":0.42299999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.0740000000000001,"15min":0.80600000000000005},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78500000000000003,"5min":13.334,"15min":33.082999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.93400000000000005,"15min":0.70999999999999996},"min":{"1min":0.41199999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81699999999999995,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.49199999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.84099999999999997,"15min":0.78500000000000003},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.871,"5min":15.943,"15min":31.251999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.091,"15min":0.76100000000000001},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.80000000000000004,"5min":23.785,"15min":23.785},"last":0.47599999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411501,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315940,"kb_used_data":10524,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93429212,"statfs":{"total":95995035648,"available":95671513088,"internally_reserved":0,"allocated":10776576,"data_stored":7720825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":1.0600000000000001,"15min":0.754},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.65500000000000003,"5min":56.509,"15min":56.509},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.59799999999999998,"15min":0.71199999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56799999999999995,"5min":19.571999999999999,"15min":31.498999999999999},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:38:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.45100000000000001,"15min":0.59999999999999998},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.8919999999999999,"5min":7.7110000000000003,"15min":19.324999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.51000000000000001,"15min":0.75800000000000001},"min":{"1min":0.221,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":7.9779999999999998,"5min":7.9779999999999998,"15min":35.052},"last":0.32300000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:39:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.748},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.944,"5min":10.032,"15min":43.5},"last":0.497},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.60899999999999999,"15min":0.69599999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.0230000000000001,"5min":11.815,"15min":27.414000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.69999999999999996,"15min":0.67300000000000004},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.93999999999999995,"5min":10.106,"15min":14.614000000000001},"last":0.437},{"interface":"front","average":{"1min":0.495,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.33000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73599999999999999,"5min":7.766,"15min":7.766},"last":0.38300000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:39:01 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.67900000000000005,"15min":0.70999999999999996},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":7.915,"5min":7.915,"15min":17.765000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.68100000000000005,"15min":0.67600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.317},"max":{"1min":2.129,"5min":10.138,"15min":14.589},"last":0.40500000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:39:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.35399999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.944,"5min":7.7869999999999999,"15min":7.7869999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.288,"5min":0.73099999999999998,"15min":0.79700000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.9589999999999996,"5min":7.9589999999999996,"15min":20.574999999999999},"last":0.42699999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:38:29 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":0.73299999999999998,"15min":1.077},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":7.9480000000000004,"5min":7.9480000000000004,"15min":66.632000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.73599999999999999,"15min":0.85199999999999998},"min":{"1min":0.36399999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.1789999999999998,"5min":10.163,"15min":19.603999999999999},"last":0.46000000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307648,"kb_used_data":10424,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93437504,"statfs":{"total":95995035648,"available":95680004096,"internally_reserved":0,"allocated":10674176,"data_stored":7713752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":1.506,"15min":0.745},"min":{"1min":0.252,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.76100000000000001,"5min":55.238999999999997,"15min":55.238999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":1.4750000000000001,"15min":0.747},"min":{"1min":0.26100000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.47699999999999998,"5min":54.563000000000002,"15min":54.563000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.82999999999999996,"15min":0.61799999999999999},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.17899999999999999},"max":{"1min":2.718,"5min":54.351999999999997,"15min":54.351999999999997},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.85799999999999998,"15min":0.76200000000000001},"min":{"1min":0.23499999999999999,"5min":0.219,"15min":0.219},"max":{"1min":1.3129999999999999,"5min":55.337000000000003,"15min":55.337000000000003},"last":0.32800000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.85499999999999998,"15min":0.75},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.159,"5min":55.368000000000002,"15min":55.368000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.88,"15min":0.73499999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.185},"max":{"1min":4.181,"5min":54.383000000000003,"15min":54.383000000000003},"last":0.32000000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:39:20 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.1850000000000001,"15min":0.94099999999999995},"min":{"1min":0.35999999999999999,"5min":0.316,"15min":0.27900000000000003},"max":{"1min":6.8799999999999999,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.58899999999999997,"15min":0.63200000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.24099999999999999},"max":{"1min":4.9829999999999997,"5min":4.9829999999999997,"15min":8.0359999999999996},"last":0.48299999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:38:41 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":1.0089999999999999,"15min":0.877},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.28899999999999998},"max":{"1min":0.83899999999999997,"5min":55.401000000000003,"15min":55.401000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":1.006,"15min":0.83199999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.30299999999999999},"max":{"1min":0.72899999999999998,"5min":54.454999999999998,"15min":54.454999999999998},"last":0.495}]},{"osd":6,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.2330000000000001,"15min":0.85199999999999998},"min":{"1min":0.38100000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":9.7490000000000006,"5min":54.305,"15min":54.305},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.25,"15min":1.01},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":9.6820000000000004,"5min":55.424999999999997,"15min":55.424999999999997},"last":0.52600000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:38:41 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.0720000000000001,"15min":0.77600000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34300000000000003},"max":{"1min":4.9790000000000001,"5min":54.222000000000001,"15min":54.222000000000001},"last":0.437},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.119,"15min":0.93799999999999994},"min":{"1min":0.40400000000000003,"5min":0.377,"15min":0.36499999999999999},"max":{"1min":6.7279999999999998,"5min":55.460000000000001,"15min":55.460000000000001},"last":0.50600000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986340,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303940,"kb_used_data":6716,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441212,"statfs":{"total":95995035648,"available":95683801088,"internally_reserved":0,"allocated":6877184,"data_stored":3721255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1739999999999999,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.68799999999999994,"5min":21.419,"15min":21.419},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":2.1200000000000001,"15min":1.2589999999999999},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.81999999999999995,"15min":1.097},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.0540000000000003,"5min":16.696000000000002,"15min":41.137999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.88,"5min":0.72199999999999998,"15min":0.90800000000000003},"min":{"1min":0.27500000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":8.4809999999999999,"5min":13.455,"15min":26.210000000000001},"last":0.372}]},{"osd":2,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.73699999999999999,"15min":1.1519999999999999},"min":{"1min":0.31900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.0510000000000002,"5min":13.478,"15min":67.253},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.67200000000000004,"15min":1.042},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.4980000000000002,"5min":7.4980000000000002,"15min":41.927999999999997},"last":0.41399999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.68899999999999995,"15min":1.0920000000000001},"min":{"1min":0.38400000000000001,"5min":0.34799999999999998,"15min":0.34000000000000002},"max":{"1min":7.5309999999999997,"5min":7.774,"15min":36.301000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.751,"15min":1.1850000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.24299999999999999},"max":{"1min":2.726,"5min":16.657,"15min":42.064},"last":0.35399999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:38:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44700000000000001,"15min":0.55900000000000005},"min":{"1min":0.22500000000000001,"5min":0.219,"15min":0.217},"max":{"1min":0.73199999999999998,"5min":2.3210000000000002,"15min":14.242000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.56200000000000006,"15min":0.59799999999999998},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.71199999999999997,"5min":7.1459999999999999,"15min":14.124000000000001},"last":0.442}]},{"osd":6,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.55700000000000005,"15min":0.57099999999999995},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":1.734,"5min":7.1189999999999998,"15min":8.0730000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.46000000000000002,"15min":0.63300000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.76300000000000001,"5min":2.2450000000000001,"15min":45.814999999999998},"last":0.26400000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:38:27 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.54200000000000004,"15min":0.61799999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70699999999999996,"5min":5.1280000000000001,"15min":17.178999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.69499999999999995,"15min":0.69599999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.28299999999999997},"max":{"1min":0.748,"5min":15.85,"15min":15.85},"last":0.38300000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757407,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312148,"kb_used_data":6732,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433004,"statfs":{"total":95995035648,"available":95675396096,"internally_reserved":0,"allocated":6893568,"data_stored":3712747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.1200000000000001,"15min":0.70699999999999996},"min":{"1min":0.36699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.115,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":1.028},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.498,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.504,"15min":0.75800000000000001},"min":{"1min":0.27500000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67800000000000005,"5min":3.7629999999999999,"15min":38.838000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.55800000000000005,"15min":1.1919999999999999},"min":{"1min":0.33700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.748,"5min":3.7250000000000001,"15min":127.794},"last":0.28999999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.66800000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":11.435,"5min":11.435,"15min":37.442999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.53800000000000003,"15min":0.70399999999999996},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.76800000000000002,"5min":3.734,"15min":23.297999999999998},"last":0.32200000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.745,"15min":0.82099999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":11.457000000000001,"5min":12.507,"15min":24.414999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.59199999999999997,"15min":0.91300000000000003},"min":{"1min":0.34599999999999997,"5min":0.33100000000000002,"15min":0.31},"max":{"1min":0.64600000000000002,"5min":3.8119999999999998,"15min":40.865000000000002},"last":0.39300000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40999999999999998,"15min":0.78200000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":0.58199999999999996,"5min":2.8940000000000001,"15min":103.13},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.56899999999999995,"15min":0.51100000000000001},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.381,"5min":11.381,"15min":12.795999999999999},"last":0.29999999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.61699999999999999,"15min":0.54600000000000004},"min":{"1min":0.29699999999999999,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45500000000000002,"15min":0.82299999999999995},"min":{"1min":0.27000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57799999999999996,"5min":1.5229999999999999,"15min":103.61499999999999},"last":0.33100000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:38:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.53300000000000003,"15min":0.53200000000000003},"min":{"1min":0.29999999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81699999999999995,"5min":3.9039999999999999,"15min":7.1820000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.56200000000000006,"15min":0.53000000000000003},"min":{"1min":0.28199999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.79400000000000004,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4419584,"data_stored":4012374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4280320,"data_stored":4009034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4132864,"data_stored":4000318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":8696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":9264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":5732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":63,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":11748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":227,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":227,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":227,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":227,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":227,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":227,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":227,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:39:30.447 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-23T00:39:30.448 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.control pg_num 32 2024-02-23T00:39:30.449 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:28.507694+0000 osd.4 (osd.4) 253 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:39:30.449 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:28.509666+0000 osd.4 (osd.4) 254 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:39:30.449 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:28.780262+0000 osd.5 (osd.5) 213 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:39:30.449 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:28.781580+0000 osd.5 (osd.5) 214 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:39:30.449 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:29.021831+0000 mgr.y (mgr.24437) 505 : audit [DBG] from='client.37967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:39:30.449 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:29.194380+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-90"}]': finished 2024-02-23T00:39:30.449 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:29.194450+0000 mon.a (mon.0) 2986 : cluster [DBG] osdmap e550: 8 total, 7 up, 8 in 2024-02-23T00:39:30.449 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:29.409639+0000 osd.6 (osd.6) 224 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:39:30.449 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:29.410453+0000 osd.6 (osd.6) 225 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:39:31.428 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:39:31.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:39:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:39:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:29.727570+0000 mgr.y (mgr.24437) 506 : cluster [DBG] pgmap v842: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 158 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 104/1269 objects degraded (8.195%); 74/1269 objects misplaced (5.831%) 2024-02-23T00:39:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:30.202550+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e551: 8 total, 7 up, 8 in 2024-02-23T00:39:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:30.367740+0000 osd.3 (osd.3) 163 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:39:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:30.370616+0000 osd.3 (osd.3) 164 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:39:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:30.413003+0000 osd.2 (osd.2) 209 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:39:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:30.413873+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:39:31.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:29.727570+0000 mgr.y (mgr.24437) 506 : cluster [DBG] pgmap v842: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 158 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 104/1269 objects degraded (8.195%); 74/1269 objects misplaced (5.831%) 2024-02-23T00:39:31.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:30.202550+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e551: 8 total, 7 up, 8 in 2024-02-23T00:39:31.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:30.367740+0000 osd.3 (osd.3) 163 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:39:31.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:30.370616+0000 osd.3 (osd.3) 164 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:39:31.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:30.413003+0000 osd.2 (osd.2) 209 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:39:31.454 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:30.413873+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:39:31.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:29.727570+0000 mgr.y (mgr.24437) 506 : cluster [DBG] pgmap v842: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 158 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 104/1269 objects degraded (8.195%); 74/1269 objects misplaced (5.831%) 2024-02-23T00:39:31.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:30.202550+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e551: 8 total, 7 up, 8 in 2024-02-23T00:39:31.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:30.367740+0000 osd.3 (osd.3) 163 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:39:31.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:30.370616+0000 osd.3 (osd.3) 164 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:39:31.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:30.413003+0000 osd.2 (osd.2) 209 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:39:31.455 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:30.413873+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:39:31.527 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:31.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:31.632 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:31.658 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:31.677 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:31.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:31.780 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:31.808 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:31.828 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:31.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:31.934 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:31.962 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:32.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 -- 172.21.15.87:0/1619181479 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5918102120 msgr2=0x7f5918104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1619181479 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5918102120 0x7f5918104510 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 rx=0x7f5910009c30 tx=0x7f5910004c50).stop 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 -- 172.21.15.87:0/1619181479 shutdown_connections 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1619181479 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f5918102120 0x7f5918104510 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1619181479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59180f8bc0 0x7f59180f8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1619181479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59180f9520 0x7f5918101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 -- 172.21.15.87:0/1619181479 >> 172.21.15.87:0/1619181479 conn(0x7f59180f26f0 msgr2=0x7f59180f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 -- 172.21.15.87:0/1619181479 shutdown_connections 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 -- 172.21.15.87:0/1619181479 wait complete. 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 librados: starting msgr at 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.104+0000 7f591c9a9700 1 librados: starting objecter 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 Processor -- start 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 -- start start 2024-02-23T00:39:32.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 librados: setting wanted keys 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 librados: calling monclient init 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59180f8bc0 0x7f59180f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59180f9520 0x7f59180f7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5918102120 0x7f59180f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5918116470 con 0x7f59180f8bc0 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59181165e0 con 0x7f5918102120 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591c9a9700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59181167a0 con 0x7f59180f9520 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5915d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59180f9520 0x7f59180f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59180f8bc0 0x7f59180f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5915d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59180f9520 0x7f59180f7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33806/0 (socket says 172.21.15.87:33806) 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5915d9b700 1 -- 172.21.15.87:0/1333189601 learned_addr learned my addr 172.21.15.87:0/1333189601 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59180f8bc0 0x7f59180f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53520/0 (socket says 172.21.15.87:53520) 2024-02-23T00:39:32.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5916d9d700 1 --2- 172.21.15.87:0/1333189601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5918102120 0x7f59180f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5915d9b700 1 -- 172.21.15.87:0/1333189601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5918102120 msgr2=0x7f59180f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5915d9b700 1 --2- 172.21.15.87:0/1333189601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5918102120 0x7f59180f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5915d9b700 1 -- 172.21.15.87:0/1333189601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59180f8bc0 msgr2=0x7f59180f6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5915d9b700 1 --2- 172.21.15.87:0/1333189601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59180f8bc0 0x7f59180f6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5915d9b700 1 -- 172.21.15.87:0/1333189601 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5908005040 con 0x7f59180f9520 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f591659c700 1 --2- 172.21.15.87:0/1333189601 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59180f8bc0 0x7f59180f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.105+0000 7f5916d9d700 1 --2- 172.21.15.87:0/1333189601 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5918102120 0x7f59180f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f59077fe700 10 client.?.objecter ms_handle_connect 0x7f59180f9520 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f59077fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f5915d9b700 1 --2- 172.21.15.87:0/1333189601 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59180f9520 0x7f59180f7060 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 rx=0x7f5908009400 tx=0x7f5908009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908007a40 con 0x7f59180f9520 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f590800f040 con 0x7f59180f9520 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59100097c0 con 0x7f59180f9520 2024-02-23T00:39:32.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5908014aa0 con 0x7f59180f9520 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f591c9a9700 10 client.37973.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5918116de0 con 0x7f59180f9520 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f591c9a9700 1 librados: init done 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.106+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f591805f990 con 0x7f59180f9520 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.107+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f590800e630 con 0x7f59180f9520 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.107+0000 7f59077fe700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5900062c90 0x7f5900065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.107+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(552..552 src has 173..552) v4 ==== 8958+0+0 (secure 0 0 0) 0x7f590807d1f0 con 0x7f59180f9520 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.107+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 osd_map(552..552 src has 173..552) v4 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.107+0000 7f59077fe700 3 client.37973.objecter handle_osd_map got epochs [552,552] > 0 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.107+0000 7f59077fe700 3 client.37973.objecter handle_osd_map decoding full epoch 552 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.107+0000 7f591659c700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5900062c90 0x7f5900065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f59077fe700 20 client.37973.objecter dump_active .. 0 homeless 2024-02-23T00:39:32.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v552) v1 ==== 54+0+16 (secure 0 0 0) 0x7f5908084070 con 0x7f59180f9520 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f59077fe700 10 client.37973.objecter ms_handle_connect 0x7f5900062c90 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591659c700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5900062c90 0x7f5900065150 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 rx=0x7f590c0103c0 tx=0x7f590c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 10 client.37973.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 1 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 20 client.37973.objecter _get_session osd=-1 returning homeless 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 10 client.37973.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=553}) v3 -- 0x7f5918107150 con 0x7f59180f9520 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f591805ee80 con 0x7f59180f9520 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=552) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5908050340 con 0x7f59180f9520 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f5914d99700 10 client.37973.objecter _check_command_map_dne tid 1 current 552 map_dne_bound 552 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f5914d99700 10 client.37973.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f5914d99700 15 client.37973.objecter _session_command_op_remove -1 1 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 10 client.37973.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 2 2024-02-23T00:39:32.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6805/685354171 conn(0x7f59180f4dc0 0x7f59180273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59180f4c50 osd=1 3 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59180f4c50 osd=1 3 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59180f4c50 osd=1 3 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 2 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f59180f4c50 osd=1 3 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 1 2 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59180f4c50 osd=1 4 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 10 client.37973.objecter _send_command 2 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5918059e00 con 0x7f59180f4dc0 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f5916d9d700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6805/685354171 conn(0x7f59180f4dc0 0x7f59180273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f59077fe700 10 client.37973.objecter ms_handle_connect 0x7f59180f4dc0 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.108+0000 7f5916d9d700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6805/685354171 conn(0x7f59180f4dc0 0x7f59180273c0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.109+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5918059e00 con 0x7f59180f4dc0 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.109+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.109+0000 7f59077fe700 10 client.37973.objecter _finish_command 2 = system:0 2024-02-23T00:39:32.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.109+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f59180f4c50 osd=1 4 2024-02-23T00:39:32.179 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:39:32.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.109+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 1 2 2024-02-23T00:39:32.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 10 client.37973.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:32.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 3 2024-02-23T00:39:32.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59180f4c50 osd=1 3 2024-02-23T00:39:32.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59180f4c50 osd=1 3 2024-02-23T00:39:32.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59180f4c50 osd=1 3 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 3 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f59180f4c50 osd=1 3 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 1 3 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59180f4c50 osd=1 4 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 10 client.37973.objecter _send_command 3 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.124+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f59180278b0 con 0x7f59180f4dc0 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f59180278b0 con 0x7f59180f4dc0 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f59077fe700 10 client.37973.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f59180f4c50 osd=1 4 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 1 3 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 10 client.37973.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 4 2024-02-23T00:39:32.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6809/3605394228 conn(0x7f59180594d0 0x7f591811af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f5918059360 osd=2 3 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f5918059360 osd=2 3 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f5918059360 osd=2 3 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 4 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f5918059360 osd=2 3 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 2 4 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f5918059360 osd=2 4 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 10 client.37973.objecter _send_command 4 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59180287c0 con 0x7f59180594d0 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591659c700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6809/3605394228 conn(0x7f59180594d0 0x7f591811af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f59077fe700 10 client.37973.objecter ms_handle_connect 0x7f59180594d0 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.126+0000 7f591659c700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6809/3605394228 conn(0x7f59180594d0 0x7f591811af90 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.127+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f59180287c0 con 0x7f59180594d0 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.127+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:39:32.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.127+0000 7f59077fe700 10 client.37973.objecter _finish_command 4 = system:0 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.127+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f5918059360 osd=2 4 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.127+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 2 4 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 10 client.37973.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 5 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f5918059360 osd=2 3 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f5918059360 osd=2 3 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f5918059360 osd=2 3 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 5 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f5918059360 osd=2 3 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 2 5 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f5918059360 osd=2 4 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 10 client.37973.objecter _send_command 5 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.142+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5918027cc0 con 0x7f59180594d0 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.143+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f5918027cc0 con 0x7f59180594d0 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.143+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:32.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.143+0000 7f59077fe700 10 client.37973.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.143+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f5918059360 osd=2 4 2024-02-23T00:39:32.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.143+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 2 5 2024-02-23T00:39:32.183 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:39:32.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.143+0000 7f591c9a9700 10 client.37973.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:32.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.143+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 6 2024-02-23T00:39:32.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.143+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6813/128059790 conn(0x7f591811c600 0x7f591811e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59180fc370 osd=3 3 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59180fc370 osd=3 3 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59180fc370 osd=3 3 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 6 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f59180fc370 osd=3 3 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 3 6 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59180fc370 osd=3 4 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f591c9a9700 10 client.37973.objecter _send_command 6 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5918121640 con 0x7f591811c600 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f5915d9b700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6813/128059790 conn(0x7f591811c600 0x7f591811e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f59077fe700 10 client.37973.objecter ms_handle_connect 0x7f591811c600 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f5915d9b700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.87:6813/128059790 conn(0x7f591811c600 0x7f591811e9f0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:32.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5918121640 con 0x7f591811c600 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f59077fe700 10 client.37973.objecter _finish_command 6 = system:0 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f59180fc370 osd=3 4 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.144+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 3 6 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.153+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== mon.2 v2:172.21.15.87:3301/0 8 ==== osd_map(553..553 src has 173..553) v4 ==== 676+0+0 (secure 0 0 0) 0x7f5908050570 con 0x7f59180f9520 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.154+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 osd_map(553..553 src has 173..553) v4 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.154+0000 7f59077fe700 3 client.37973.objecter handle_osd_map got epochs [553,553] > 552 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.154+0000 7f59077fe700 3 client.37973.objecter handle_osd_map decoding incremental epoch 553 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.154+0000 7f59077fe700 20 client.37973.objecter dump_active .. 0 homeless 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 10 client.37973.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 7 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59180fc370 osd=3 3 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59180fc370 osd=3 3 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59180fc370 osd=3 3 2024-02-23T00:39:32.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 7 2024-02-23T00:39:32.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f59180fc370 osd=3 3 2024-02-23T00:39:32.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 3 7 2024-02-23T00:39:32.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59180fc370 osd=3 4 2024-02-23T00:39:32.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 10 client.37973.objecter _send_command 7 2024-02-23T00:39:32.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.175+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f59180fbe40 con 0x7f591811c600 2024-02-23T00:39:32.186 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.180+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f59180fbe40 con 0x7f591811c600 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.180+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.180+0000 7f59077fe700 10 client.37973.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.180+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f59180fc370 osd=3 4 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.180+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 3 7 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 10 client.37973.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 8 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6800/4175844757 conn(0x7f5918120060 0x7f59181204a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f591811bc50 osd=4 3 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f591811bc50 osd=4 3 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f591811bc50 osd=4 3 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 8 2024-02-23T00:39:32.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f591811bc50 osd=4 3 2024-02-23T00:39:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 4 8 2024-02-23T00:39:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f591811bc50 osd=4 4 2024-02-23T00:39:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 10 client.37973.objecter _send_command 8 2024-02-23T00:39:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f591811c2d0 con 0x7f5918120060 2024-02-23T00:39:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f5916d9d700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6800/4175844757 conn(0x7f5918120060 0x7f59181204a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f5916d9d700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6800/4175844757 conn(0x7f5918120060 0x7f59181204a0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:32.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.181+0000 7f59077fe700 10 client.37973.objecter ms_handle_connect 0x7f5918120060 2024-02-23T00:39:32.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.182+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f591811c2d0 con 0x7f5918120060 2024-02-23T00:39:32.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.182+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:39:32.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.182+0000 7f59077fe700 10 client.37973.objecter _finish_command 8 = system:0 2024-02-23T00:39:32.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.182+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f591811bc50 osd=4 4 2024-02-23T00:39:32.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.182+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 4 8 2024-02-23T00:39:32.194 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:32.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:32.230 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:39:32.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 10 client.37973.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:32.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 9 2024-02-23T00:39:32.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f591811bc50 osd=4 3 2024-02-23T00:39:32.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f591811bc50 osd=4 3 2024-02-23T00:39:32.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f591811bc50 osd=4 3 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 9 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f591811bc50 osd=4 3 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 4 9 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f591811bc50 osd=4 4 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 10 client.37973.objecter _send_command 9 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.214+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5918121160 con 0x7f5918120060 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.217+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f5918121160 con 0x7f5918120060 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.217+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.217+0000 7f59077fe700 10 client.37973.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.217+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f591811bc50 osd=4 4 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.217+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 4 9 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 10 client.37973.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 10 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6804/2550013795 conn(0x7f59181237c0 0x7f5918125bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f5918121160 osd=5 3 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f5918121160 osd=5 3 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f5918121160 osd=5 3 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 10 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f5918121160 osd=5 3 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 5 10 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f5918121160 osd=5 4 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 10 client.37973.objecter _send_command 10 2024-02-23T00:39:32.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.227+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59181212d0 con 0x7f59181237c0 2024-02-23T00:39:32.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.239+0000 7f591659c700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6804/2550013795 conn(0x7f59181237c0 0x7f5918125bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.247 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:32.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.250+0000 7f591659c700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6804/2550013795 conn(0x7f59181237c0 0x7f5918125bb0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:32.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.250+0000 7f59077fe700 10 client.37973.objecter ms_handle_connect 0x7f59181237c0 2024-02-23T00:39:32.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.250+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f59181212d0 con 0x7f59181237c0 2024-02-23T00:39:32.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.250+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:39:32.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.250+0000 7f59077fe700 10 client.37973.objecter _finish_command 10 = system:0 2024-02-23T00:39:32.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.250+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f5918121160 osd=5 4 2024-02-23T00:39:32.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.250+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 5 10 2024-02-23T00:39:32.263 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:32.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 10 client.37973.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 11 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f5918121160 osd=5 3 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f5918121160 osd=5 3 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f5918121160 osd=5 3 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 11 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f5918121160 osd=5 3 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 5 11 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f5918121160 osd=5 4 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 10 client.37973.objecter _send_command 11 2024-02-23T00:39:32.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.278+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f59181212d0 con 0x7f59181237c0 2024-02-23T00:39:32.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f59181212d0 con 0x7f59181237c0 2024-02-23T00:39:32.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:32.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f59077fe700 10 client.37973.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f5918121160 osd=5 4 2024-02-23T00:39:32.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 5 11 2024-02-23T00:39:32.283 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:39:32.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 10 client.37973.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:32.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 12 2024-02-23T00:39:32.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6808/2509408498 conn(0x7f59181265b0 0x7f59181289a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59181212d0 osd=6 3 2024-02-23T00:39:32.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59181212d0 osd=6 3 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59181212d0 osd=6 3 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 12 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f59181212d0 osd=6 3 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 6 12 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59181212d0 osd=6 4 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 10 client.37973.objecter _send_command 12 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f591812b780 con 0x7f59181265b0 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.280+0000 7f5915d9b700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6808/2509408498 conn(0x7f59181265b0 0x7f59181289a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.281+0000 7f59077fe700 10 client.37973.objecter ms_handle_connect 0x7f59181265b0 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.281+0000 7f5915d9b700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6808/2509408498 conn(0x7f59181265b0 0x7f59181289a0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.281+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f591812b780 con 0x7f59181265b0 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.281+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.281+0000 7f59077fe700 10 client.37973.objecter _finish_command 12 = system:0 2024-02-23T00:39:32.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.281+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f59181212d0 osd=6 4 2024-02-23T00:39:32.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.281+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 6 12 2024-02-23T00:39:32.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 10 client.37973.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 13 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59181212d0 osd=6 3 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59181212d0 osd=6 3 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f59181212d0 osd=6 3 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 13 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f59181212d0 osd=6 3 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 6 13 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f59181212d0 osd=6 4 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 10 client.37973.objecter _send_command 13 2024-02-23T00:39:32.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.297+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f591812b780 con 0x7f59181265b0 2024-02-23T00:39:32.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f591812b780 con 0x7f59181265b0 2024-02-23T00:39:32.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:32.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f59077fe700 10 client.37973.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f59181212d0 osd=6 4 2024-02-23T00:39:32.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 6 13 2024-02-23T00:39:32.302 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:39:32.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 10 client.37973.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:32.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 14 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6812/373787000 conn(0x7f591812aa80 0x7f591812af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f591812b780 osd=7 3 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f591812b780 osd=7 3 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f591812b780 osd=7 3 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 14 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f591812b780 osd=7 3 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 7 14 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f591812b780 osd=7 4 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 10 client.37973.objecter _send_command 14 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f591812b5a0 con 0x7f591812aa80 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.299+0000 7f5916d9d700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6812/373787000 conn(0x7f591812aa80 0x7f591812af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.300+0000 7f59077fe700 10 client.37973.objecter ms_handle_connect 0x7f591812aa80 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.300+0000 7f5916d9d700 1 --2- 172.21.15.87:0/1333189601 >> v2:172.21.15.171:6812/373787000 conn(0x7f591812aa80 0x7f591812af00 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:32.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.300+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f591812b5a0 con 0x7f591812aa80 2024-02-23T00:39:32.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.300+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:39:32.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.300+0000 7f59077fe700 10 client.37973.objecter _finish_command 14 = system:0 2024-02-23T00:39:32.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.300+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f591812b780 osd=7 4 2024-02-23T00:39:32.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.300+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 7 14 2024-02-23T00:39:32.310 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:32.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 10 client.37973.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:39:32.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign -1 15 2024-02-23T00:39:32.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f591812b780 osd=7 3 2024-02-23T00:39:32.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f591812b780 osd=7 3 2024-02-23T00:39:32.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 20 client.37973.objecter _get_session s=0x7f591812b780 osd=7 3 2024-02-23T00:39:32.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_remove -1 15 2024-02-23T00:39:32.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 20 client.37973.objecter get_session s=0x7f591812b780 osd=7 3 2024-02-23T00:39:32.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 15 client.37973.objecter _session_command_op_assign 7 15 2024-02-23T00:39:32.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 20 client.37973.objecter put_session s=0x7f591812b780 osd=7 4 2024-02-23T00:39:32.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 10 client.37973.objecter _send_command 15 2024-02-23T00:39:32.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.316+0000 7f591c9a9700 1 -- 172.21.15.87:0/1333189601 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f591812b5a0 con 0x7f591812aa80 2024-02-23T00:39:32.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.318+0000 7f59077fe700 1 -- 172.21.15.87:0/1333189601 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f591812b5a0 con 0x7f591812aa80 2024-02-23T00:39:32.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.318+0000 7f59077fe700 10 client.37973.objecter ms_dispatch 0x7f5918001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:39:32.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.318+0000 7f59077fe700 10 client.37973.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.318+0000 7f59077fe700 20 client.37973.objecter put_session s=0x7f591812b780 osd=7 4 2024-02-23T00:39:32.321 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:39:32.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:32.318+0000 7f59077fe700 15 client.37973.objecter _session_command_op_remove 7 15 2024-02-23T00:39:32.322 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:39:32.322 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.322 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.322 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.322 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.322 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.322 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.322 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:39:32.335 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:32.355 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:32.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:32.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:32.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:32.483 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:32] "GET /metrics HTTP/1.1" 200 189442 "" "Prometheus/2.33.4" 2024-02-23T00:39:32.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:30.472764+0000 osd.4 (osd.4) 255 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:30.473599+0000 osd.4 (osd.4) 256 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:31.079593+0000 mgr.y (mgr.24437) 507 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.208156+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e552: 8 total, 7 up, 8 in 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:31.216417+0000 mon.b (mon.1) 448 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:31.216712+0000 mon.a (mon.0) 2989 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.343345+0000 osd.3 (osd.3) 165 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.346252+0000 osd.3 (osd.3) 166 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.409453+0000 osd.6 (osd.6) 226 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.412341+0000 osd.6 (osd.6) 227 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.886733+0000 osd.7 (osd.7) 264 : cluster [DBG] 22.c scrub starts 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.887580+0000 osd.7 (osd.7) 265 : cluster [DBG] 22.c scrub ok 2024-02-23T00:39:32.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:32.108916+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? 172.21.15.87:0/1333189601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:32.150237+0000 mon.a (mon.0) 2990 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:32.154853+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:32.154935+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e553: 8 total, 7 up, 8 in 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:32.168305+0000 mon.b (mon.1) 449 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:32.173449+0000 mon.b (mon.1) 450 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:32.179034+0000 mon.a (mon.0) 2993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:30.472764+0000 osd.4 (osd.4) 255 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:30.473599+0000 osd.4 (osd.4) 256 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:31.079593+0000 mgr.y (mgr.24437) 507 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:32.485 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.208156+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e552: 8 total, 7 up, 8 in 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:31.216417+0000 mon.b (mon.1) 448 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:31.216712+0000 mon.a (mon.0) 2989 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.343345+0000 osd.3 (osd.3) 165 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.346252+0000 osd.3 (osd.3) 166 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.409453+0000 osd.6 (osd.6) 226 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.412341+0000 osd.6 (osd.6) 227 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.886733+0000 osd.7 (osd.7) 264 : cluster [DBG] 22.c scrub starts 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.887580+0000 osd.7 (osd.7) 265 : cluster [DBG] 22.c scrub ok 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:32.108916+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? 172.21.15.87:0/1333189601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:32.150237+0000 mon.a (mon.0) 2990 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:32.154853+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:32.154935+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e553: 8 total, 7 up, 8 in 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:32.168305+0000 mon.b (mon.1) 449 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:32.173449+0000 mon.b (mon.1) 450 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:32.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:32.179034+0000 mon.a (mon.0) 2993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:32.487 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:30.472764+0000 osd.4 (osd.4) 255 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:30.473599+0000 osd.4 (osd.4) 256 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:31.079593+0000 mgr.y (mgr.24437) 507 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.208156+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e552: 8 total, 7 up, 8 in 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:31.216417+0000 mon.b (mon.1) 448 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:31.216712+0000 mon.a (mon.0) 2989 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.343345+0000 osd.3 (osd.3) 165 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.346252+0000 osd.3 (osd.3) 166 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.409453+0000 osd.6 (osd.6) 226 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.412341+0000 osd.6 (osd.6) 227 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.886733+0000 osd.7 (osd.7) 264 : cluster [DBG] 22.c scrub starts 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.887580+0000 osd.7 (osd.7) 265 : cluster [DBG] 22.c scrub ok 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:32.108916+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? 172.21.15.87:0/1333189601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:32.150237+0000 mon.a (mon.0) 2990 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:32.154853+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:32.154935+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e553: 8 total, 7 up, 8 in 2024-02-23T00:39:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:32.168305+0000 mon.b (mon.1) 449 : audit [DBG] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-23T00:39:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:32.173449+0000 mon.b (mon.1) 450 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:32.179034+0000 mon.a (mon.0) 2993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-23T00:39:33.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.446495+0000 osd.4 (osd.4) 257 : cluster [DBG] 63.19 scrub starts 2024-02-23T00:39:33.271 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.448432+0000 osd.4 (osd.4) 258 : cluster [DBG] 63.19 scrub ok 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:31.728183+0000 mgr.y (mgr.24437) 508 : cluster [DBG] pgmap v845: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 138 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1272 objects degraded (8.176%); 74/1272 objects misplaced (5.818%) 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:32.208492+0000 mon.a (mon.0) 2994 : cluster [WRN] Health check update: Degraded data redundancy: 104/1272 objects degraded (8.176%), 23 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:32.307789+0000 osd.3 (osd.3) 167 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:32.310675+0000 osd.3 (osd.3) 168 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:32.808335+0000 osd.5 (osd.5) 215 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:32.809130+0000 osd.5 (osd.5) 216 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:33.157020+0000 mon.a (mon.0) 2995 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:33.157093+0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e554: 8 total, 7 up, 8 in 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.446495+0000 osd.4 (osd.4) 257 : cluster [DBG] 63.19 scrub starts 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.448432+0000 osd.4 (osd.4) 258 : cluster [DBG] 63.19 scrub ok 2024-02-23T00:39:33.272 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:31.728183+0000 mgr.y (mgr.24437) 508 : cluster [DBG] pgmap v845: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 138 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1272 objects degraded (8.176%); 74/1272 objects misplaced (5.818%) 2024-02-23T00:39:33.273 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:32.208492+0000 mon.a (mon.0) 2994 : cluster [WRN] Health check update: Degraded data redundancy: 104/1272 objects degraded (8.176%), 23 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:33.273 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:32.307789+0000 osd.3 (osd.3) 167 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:39:33.273 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:32.310675+0000 osd.3 (osd.3) 168 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:39:33.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:33.159503+0000 mon.b (mon.1) 451 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:39:33.548 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:33.163833+0000 mon.a (mon.0) 2997 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:39:33.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:32.808335+0000 osd.5 (osd.5) 215 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:39:33.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:32.809130+0000 osd.5 (osd.5) 216 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:39:33.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:33.157020+0000 mon.a (mon.0) 2995 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:39:33.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:33.157093+0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e554: 8 total, 7 up, 8 in 2024-02-23T00:39:33.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:33.159503+0000 mon.b (mon.1) 451 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:39:33.548 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:33.163833+0000 mon.a (mon.0) 2997 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:39:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.446495+0000 osd.4 (osd.4) 257 : cluster [DBG] 63.19 scrub starts 2024-02-23T00:39:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.448432+0000 osd.4 (osd.4) 258 : cluster [DBG] 63.19 scrub ok 2024-02-23T00:39:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:31.728183+0000 mgr.y (mgr.24437) 508 : cluster [DBG] pgmap v845: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 138 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1272 objects degraded (8.176%); 74/1272 objects misplaced (5.818%) 2024-02-23T00:39:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:32.208492+0000 mon.a (mon.0) 2994 : cluster [WRN] Health check update: Degraded data redundancy: 104/1272 objects degraded (8.176%), 23 pgs degraded, 51 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:32.307789+0000 osd.3 (osd.3) 167 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:39:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:32.310675+0000 osd.3 (osd.3) 168 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:39:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:32.808335+0000 osd.5 (osd.5) 215 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:39:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:32.809130+0000 osd.5 (osd.5) 216 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:39:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:33.157020+0000 mon.a (mon.0) 2995 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-23T00:39:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:33.157093+0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e554: 8 total, 7 up, 8 in 2024-02-23T00:39:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:33.159503+0000 mon.b (mon.1) 451 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:39:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:33.163833+0000 mon.a (mon.0) 2997 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-23T00:39:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:39:34.014 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:39:34.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:39:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:39:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:33.343784+0000 osd.3 (osd.3) 169 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:39:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:33.344862+0000 osd.3 (osd.3) 170 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:39:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:34.159089+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:39:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:34.159170+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e555: 8 total, 7 up, 8 in 2024-02-23T00:39:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:34.162560+0000 mon.b (mon.1) 452 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:39:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:34.163296+0000 mon.a (mon.0) 3000 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:39:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:33.343784+0000 osd.3 (osd.3) 169 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:39:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:33.344862+0000 osd.3 (osd.3) 170 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:39:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:34.159089+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:39:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:34.159170+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e555: 8 total, 7 up, 8 in 2024-02-23T00:39:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:34.162560+0000 mon.b (mon.1) 452 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:39:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:34.163296+0000 mon.a (mon.0) 3000 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:39:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:33.343784+0000 osd.3 (osd.3) 169 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:39:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:33.344862+0000 osd.3 (osd.3) 170 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:39:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:34.159089+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-23T00:39:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:34.159170+0000 mon.a (mon.0) 2999 : cluster [DBG] osdmap e555: 8 total, 7 up, 8 in 2024-02-23T00:39:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:34.162560+0000 mon.b (mon.1) 452 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:39:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:34.163296+0000 mon.a (mon.0) 3000 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-23T00:39:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:33.729027+0000 mgr.y (mgr.24437) 509 : cluster [DBG] pgmap v848: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 138 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1272 objects degraded (8.176%); 74/1272 objects misplaced (5.818%) 2024-02-23T00:39:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:35.162483+0000 mon.a (mon.0) 3001 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:39:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:35.162542+0000 mon.a (mon.0) 3002 : cluster [DBG] osdmap e556: 8 total, 7 up, 8 in 2024-02-23T00:39:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:35.243751+0000 mon.b (mon.1) 453 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:35.244217+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:35.246464+0000 mon.b (mon.1) 454 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-92"}]: dispatch 2024-02-23T00:39:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:35.246758+0000 mon.a (mon.0) 3004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-92"}]: dispatch 2024-02-23T00:39:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:33.729027+0000 mgr.y (mgr.24437) 509 : cluster [DBG] pgmap v848: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 138 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1272 objects degraded (8.176%); 74/1272 objects misplaced (5.818%) 2024-02-23T00:39:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:35.162483+0000 mon.a (mon.0) 3001 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:39:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:35.162542+0000 mon.a (mon.0) 3002 : cluster [DBG] osdmap e556: 8 total, 7 up, 8 in 2024-02-23T00:39:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:35.243751+0000 mon.b (mon.1) 453 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:35.244217+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:35.246464+0000 mon.b (mon.1) 454 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-92"}]: dispatch 2024-02-23T00:39:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:35.246758+0000 mon.a (mon.0) 3004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-92"}]: dispatch 2024-02-23T00:39:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:33.729027+0000 mgr.y (mgr.24437) 509 : cluster [DBG] pgmap v848: 246 pgs: 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 22 active+undersized+degraded, 29 active+undersized, 138 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 104/1272 objects degraded (8.176%); 74/1272 objects misplaced (5.818%) 2024-02-23T00:39:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:35.162483+0000 mon.a (mon.0) 3001 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-23T00:39:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:35.162542+0000 mon.a (mon.0) 3002 : cluster [DBG] osdmap e556: 8 total, 7 up, 8 in 2024-02-23T00:39:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:35.243751+0000 mon.b (mon.1) 453 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:35.244217+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-6"}]: dispatch 2024-02-23T00:39:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:35.246464+0000 mon.b (mon.1) 454 : audit [INF] from='client.? 172.21.15.87:0/2563268269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-92"}]: dispatch 2024-02-23T00:39:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:35.246758+0000 mon.a (mon.0) 3004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-6", "tierpool": "test-rados-api-smithi087-82091-92"}]: dispatch 2024-02-23T00:39:35.737 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:35.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:35.842 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:35.869 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:35.888 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:35.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:35.991 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:36.015 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:39:36.021 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:36.553 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:35.339489+0000 osd.6 (osd.6) 228 : cluster [DBG] 22.19 deep-scrub starts 2024-02-23T00:39:36.553 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:35.342421+0000 osd.6 (osd.6) 229 : cluster [DBG] 22.19 deep-scrub ok 2024-02-23T00:39:36.553 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:35.383538+0000 osd.3 (osd.3) 171 : cluster [DBG] 63.8 scrub starts 2024-02-23T00:39:36.553 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:35.386985+0000 osd.3 (osd.3) 172 : cluster [DBG] 63.8 scrub ok 2024-02-23T00:39:36.553 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:36.162164+0000 mon.a (mon.0) 3005 : cluster [DBG] osdmap e557: 8 total, 7 up, 8 in 2024-02-23T00:39:36.553 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:35.339489+0000 osd.6 (osd.6) 228 : cluster [DBG] 22.19 deep-scrub starts 2024-02-23T00:39:36.553 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:35.342421+0000 osd.6 (osd.6) 229 : cluster [DBG] 22.19 deep-scrub ok 2024-02-23T00:39:36.553 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:35.383538+0000 osd.3 (osd.3) 171 : cluster [DBG] 63.8 scrub starts 2024-02-23T00:39:36.553 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:35.386985+0000 osd.3 (osd.3) 172 : cluster [DBG] 63.8 scrub ok 2024-02-23T00:39:36.554 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:36.162164+0000 mon.a (mon.0) 3005 : cluster [DBG] osdmap e557: 8 total, 7 up, 8 in 2024-02-23T00:39:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:35.339489+0000 osd.6 (osd.6) 228 : cluster [DBG] 22.19 deep-scrub starts 2024-02-23T00:39:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:35.342421+0000 osd.6 (osd.6) 229 : cluster [DBG] 22.19 deep-scrub ok 2024-02-23T00:39:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:35.383538+0000 osd.3 (osd.3) 171 : cluster [DBG] 63.8 scrub starts 2024-02-23T00:39:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:35.386985+0000 osd.3 (osd.3) 172 : cluster [DBG] 63.8 scrub ok 2024-02-23T00:39:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:36.162164+0000 mon.a (mon.0) 3005 : cluster [DBG] osdmap e557: 8 total, 7 up, 8 in 2024-02-23T00:39:36.772 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:36.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:36.894 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:36.921 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15162.0:1 57.7s0 57:ea403f26:test-rados-api-smithi087-82139-13::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:01.527880+0000", 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 515.90349367700003, 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.08591974, 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:37.435 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15162", 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3542609165", 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527880+0000", 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527880+0000", 2024-02-23T00:39:37.436 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527878+0000", 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2410000000000001e-06 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527887+0000", 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1595e-05 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527898+0000", 2024-02-23T00:39:37.437 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0059999999999992e-06 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.527906+0000", 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00104463 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.528951+0000", 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.6449999999999998e-06 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:39:37.438 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.528955+0000", 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0732655419999999 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.602220+0000", 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00039692599999999997 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.602617+0000", 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016094740000000001 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:39:37.439 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.604227+0000", 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0067402019999999998 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.610967+0000", 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0027533340000000001 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.613720+0000", 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9392000000000002e-05 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.440 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.613800+0000", 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.08591974 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:424 7.2 e113/94 7:40054bab:test-rados-api-smithi087-82115-6::foo:9 v 113'6, mlcod=111'3)", 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.558545+0000", 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 513.87282812599994, 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041023802999999998, 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.441 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558545+0000", 2024-02-23T00:39:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558545+0000", 2024-02-23T00:39:37.442 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558545+0000", 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4674e-05 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558560+0000", 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8779999999999997e-06 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.443 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558565+0000", 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3399999999999997e-06 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.558570+0000", 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00173953 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.444 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.560310+0000", 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6562000000000001e-05 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.560336+0000", 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.039195743999999998 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.599532+0000", 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7320000000000002e-05 2024-02-23T00:39:37.445 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.599569+0000", 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041023802999999998 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.5.0:324 56.1 e113/110 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head v 113'1, mlcod=0'0)", 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.675872+0000", 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 513.75550174099999, 2024-02-23T00:39:37.446 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.029613130000000001, 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675872+0000", 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675872+0000", 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.447 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675871+0000", 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2730000000000002e-06 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675879+0000", 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4129999999999995e-06 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675889+0000", 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2979999999999997e-06 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.448 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.675895+0000", 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00054747200000000002 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676442+0000", 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5807999999999998e-05 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676458+0000", 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028979677999999998 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.449 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.705438+0000", 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7066e-05 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.705485+0000", 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.029613130000000001 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15147.0:14 56.1 e113/110 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head v 113'2, mlcod=0'0)", 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.676277+0000", 2024-02-23T00:39:37.450 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 513.75509625100005, 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.029227679999999999, 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676277+0000", 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676277+0000", 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.451 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676277+0000", 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2530000000000001e-06 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676283+0000", 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2520000000000005e-06 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676292+0000", 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4049999999999999e-06 2024-02-23T00:39:37.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676298+0000", 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000294176 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676592+0000", 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6429000000000001e-05 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.676608+0000", 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028880586999999999 2024-02-23T00:39:37.453 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.705489+0000", 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6229999999999999e-05 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.705505+0000", 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.029227679999999999 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.454 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15189.0:3 61.1f 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected e114)", 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:05.220101+0000", 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 512.21127274699995, 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036226108999999999, 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15189", 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3965564315", 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 3 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.455 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220101+0000", 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220101+0000", 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220099+0000", 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.7810000000000005e-06 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.456 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220107+0000", 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.67e-06 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220115+0000", 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3560000000000004e-06 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.220123+0000", 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013298133 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.457 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.233421+0000", 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9179999999999994e-06 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.233429+0000", 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0074901580000000002 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.240919+0000", 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8102000000000005e-05 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.458 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241007+0000", 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8525999999999998e-05 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,1", 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.241106+0000", 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.006526647 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.247632+0000", 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0074987430000000004 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.459 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.255131+0000", 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011306210000000001 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.256262+0000", 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7367999999999999e-05 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.256299+0000", 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7931000000000001e-05 2024-02-23T00:39:37.460 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:05.256327+0000", 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036226108999999999 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26218.0:2 62.15 e115/114 62:ad551702:::test:head v 115'1, mlcod=0'0)", 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:06.291613+0000", 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 511.13976045200002, 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041167026000000002, 2024-02-23T00:39:37.461 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291613+0000", 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291613+0000", 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.462 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291613+0000", 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7430000000000008e-06 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291621+0000", 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7019999999999998e-06 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291627+0000", 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8669999999999996e-06 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.463 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.464 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291634+0000", 2024-02-23T00:39:37.464 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016782400000000001 2024-02-23T00:39:37.464 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.464 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.464 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.464 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291802+0000", 2024-02-23T00:39:37.464 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6673000000000001e-05 2024-02-23T00:39:37.464 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.464 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.291819+0000", 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040928475999999998 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.332747+0000", 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3111000000000002e-05 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.465 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.332780+0000", 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041167026000000002 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15189.0:10 61.1f 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head [delete] snapc 3=[3,2] ondisk+write+peerstat_old+known_if_redirected e118)", 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.171085+0000", 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 508.26028842800002, 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065815858000000005, 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.466 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15189", 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3965564315", 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 10 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171085+0000", 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.467 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171085+0000", 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171084+0000", 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1025e-05 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171095+0000", 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0020000000000001e-05 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.468 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171105+0000", 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9689999999999999e-06 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.171111+0000", 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003718687 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.174829+0000", 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000100388 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.469 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.174930+0000", 2024-02-23T00:39:37.470 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00018267399999999999 2024-02-23T00:39:37.470 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.470 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.470 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,1", 2024-02-23T00:39:37.470 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.175112+0000", 2024-02-23T00:39:37.470 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.035207443999999997 2024-02-23T00:39:37.470 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.470 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.470 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.210320+0000", 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015002176000000001 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.225322+0000", 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011492073 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.236814+0000", 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5346999999999998e-05 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.471 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.236859+0000", 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1575000000000001e-05 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.236901+0000", 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065815858000000005 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.472 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15189.0:12 61.11 61:8c482978:test-rados-api-smithi087-82107-8::foo:head [write 0~128] snapc 6=[6] ondisk+write+known_if_redirected e119)", 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.142541+0000", 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 507.28883216700001, 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.084312260999999999, 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15189", 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3965564315", 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 12 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142541+0000", 2024-02-23T00:39:37.473 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142541+0000", 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142540+0000", 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1822e-05 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.474 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142552+0000", 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5166e-05 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142567+0000", 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7140000000000005e-06 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142576+0000", 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4666000000000002e-05 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.475 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142621+0000", 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9169999999999997e-06 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.142626+0000", 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024531509999999999 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.167158+0000", 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011121200000000001 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.476 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.167269+0000", 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000162724 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,6", 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.167431+0000", 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0041109659999999998 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.171542+0000", 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010926302000000001 2024-02-23T00:39:37.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.182469+0000", 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044268719999999998 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.226737+0000", 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9410999999999998e-05 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.226797+0000", 2024-02-23T00:39:37.478 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6740999999999998e-05 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.226854+0000", 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.084312260999999999 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.28903.0:2 68.6 68:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.168576+0000", 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 507.26279706299999, 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.059563420999999998, 2024-02-23T00:39:37.479 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.28903", 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/71734753", 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168576+0000", 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.480 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168576+0000", 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168575+0000", 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9290000000000001e-06 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168582+0000", 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7690000000000002e-06 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.481 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168590+0000", 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9029999999999998e-06 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.168597+0000", 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00042019799999999999 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169018+0000", 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4330000000000001e-06 2024-02-23T00:39:37.482 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.169024+0000", 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036596376 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.205620+0000", 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000110337 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.205731+0000", 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015317099999999999 2024-02-23T00:39:37.483 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 5,7", 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.205884+0000", 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0072101079999999998 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.213094+0000", 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6569999999999999e-05 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.213111+0000", 2024-02-23T00:39:37.484 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014960002999999999 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.228071+0000", 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0018999999999998e-05 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.228101+0000", 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9279999999999997e-05 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.485 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.228140+0000", 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.059563420999999998 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.15189.0:14 61.11 61:8c482978:test-rados-api-smithi087-82107-8::foo:head [write 0~128] snapc 7=[7,6] ondisk+write+known_if_redirected e120)", 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.166692+0000", 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 506.26468165900002, 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.21529174600000001, 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:37.486 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.15189", 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3965564315", 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 14 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166692+0000", 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166692+0000", 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:39:37.487 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166690+0000", 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2899999999999997e-06 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166698+0000", 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6489999999999997e-06 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166706+0000", 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6569999999999997e-06 2024-02-23T00:39:37.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166714+0000", 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8881999999999997e-05 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166763+0000", 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0369999999999995e-06 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.166771+0000", 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.037645606999999998 2024-02-23T00:39:37.489 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.204416+0000", 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5403999999999997e-05 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.204512+0000", 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012169938 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,6", 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.216682+0000", 2024-02-23T00:39:37.490 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0083575160000000006 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225039+0000", 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.13363486999999999 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358674+0000", 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023201506 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.491 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.381876+0000", 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7921999999999999e-05 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.381934+0000", 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0055000000000001e-05 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.381984+0000", 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.21529174600000001 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.492 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15147.0:75 56.1 e122/110 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head v 122'16, mlcod=118'14)", 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:13.004465+0000", 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 504.42690871899998, 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062183177999999999, 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004465+0000", 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.493 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004465+0000", 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004463+0000", 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.925e-05 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004482+0000", 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6399999999999999e-05 2024-02-23T00:39:37.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004498+0000", 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8489999999999995e-06 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.004507+0000", 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.038591217999999997 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.043098+0000", 2024-02-23T00:39:37.495 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5066999999999999e-05 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.043123+0000", 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023502034000000002 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.066626+0000", 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2450999999999999e-05 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.496 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:13.066648+0000", 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.062183177999999999 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.7.0:204 63.13 e139/114 63:cacde8f7:test-rados-api-smithi087-82091-12::bar:2 v 139'5, mlcod=134'1)", 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.059269+0000", 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 487.37210417099999, 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026021935, 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.497 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059269+0000", 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059269+0000", 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059268+0000", 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2863e-05 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.498 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059281+0000", 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5006e-05 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059296+0000", 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9260000000000003e-06 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059303+0000", 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00091365900000000004 2024-02-23T00:39:37.499 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.060217+0000", 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0594999999999998e-05 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.060237+0000", 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025023361000000001 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085261+0000", 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0518999999999997e-05 2024-02-23T00:39:37.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085291+0000", 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026021935 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.1.0:298 86.13 e139/133 86:cacde8f7:test-rados-api-smithi087-82091-12::bar:2 v 139'4, mlcod=138'1)", 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 487.34737622099999, 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036243368999999998, 2024-02-23T00:39:37.501 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:39:37.502 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1370000000000008e-06 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084006+0000", 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4387e-05 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084020+0000", 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5240000000000003e-06 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.503 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084028+0000", 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9998e-05 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084128+0000", 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3402000000000001e-05 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084151+0000", 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036055049999999998 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.504 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.120206+0000", 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4238e-05 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.120241+0000", 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036243368999999998 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:369 63.6 63:6271ce74:test-rados-api-smithi087-82091-14::foo0:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-23T00:39:37.505 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:34.984178+0000", 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 482.44719562099999, 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040494046999999998, 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 369 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984178+0000", 2024-02-23T00:39:37.506 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984178+0000", 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984176+0000", 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0913e-05 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984187+0000", 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024830016 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.009017+0000", 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.131e-05 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.009028+0000", 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012722213 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.021750+0000", 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000138596 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.021889+0000", 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016801399999999999 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,6", 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.022057+0000", 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001482412 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.023539+0000", 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00053109800000000003 2024-02-23T00:39:37.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024070+0000", 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00050785999999999995 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024578+0000", 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1844e-05 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024630+0000", 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1841999999999997e-05 2024-02-23T00:39:37.510 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024672+0000", 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040494046999999998 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:578 63.a e146/114 63:54d171db:test-rados-api-smithi087-82091-14::foo7:head v 146'5, mlcod=144'1)", 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:37.023058+0000", 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 480.40831536500002, 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044013903, 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.511 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023058+0000", 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023058+0000", 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023056+0000", 2024-02-23T00:39:37.512 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5184000000000001e-05 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023071+0000", 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8630999999999998e-05 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023090+0000", 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6339999999999996e-06 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023100+0000", 2024-02-23T00:39:37.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1316999999999997e-05 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023151+0000", 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9091999999999999e-05 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023170+0000", 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.043871062000000002 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.067041+0000", 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0898999999999997e-05 2024-02-23T00:39:37.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.067072+0000", 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044013903 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:1810 63.8 e216/114 63:15d27fd9:test-rados-api-smithi087-82091-22::foo:head v 216'69, mlcod=198'66)", 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 410.40481141999999, 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.032422279999999998, 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.515 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.516 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1569999999999999e-06 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026567+0000", 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3954000000000001e-05 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026581+0000", 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1770000000000003e-06 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.517 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026587+0000", 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000565939 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027153+0000", 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0520999999999999e-05 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027173+0000", 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.031782471 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.518 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.058956+0000", 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.8521e-05 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.058984+0000", 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.032422279999999998 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26739.0:1 159.6s0 159:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e233)", 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:04.031498+0000", 2024-02-23T00:39:37.519 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 393.39987519699997, 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0251979280000001, 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26739", 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1200734777", 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031498+0000", 2024-02-23T00:39:37.520 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031498+0000", 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031497+0000", 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5536000000000002e-05 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.521 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031512+0000", 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0933e-05 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031523+0000", 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6389999999999992e-06 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031531+0000", 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3112999999999998e-05 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.522 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031584+0000", 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.066e-06 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031587+0000", 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0198177390000001 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.051405+0000", 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000104659 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.051509+0000", 2024-02-23T00:39:37.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00031744400000000001 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.051827+0000", 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000397279 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.052224+0000", 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0044106010000000001 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.524 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.056635+0000", 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1668000000000007e-05 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.056696+0000", 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0251979280000001 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37253.0:1 170.6s0 170:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e254)", 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:25.048138+0000", 2024-02-23T00:39:37.525 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 372.38323563099999, 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.437999077, 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37253", 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1057868100", 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048138+0000", 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.526 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048138+0000", 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048136+0000", 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7890000000000006e-06 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048146+0000", 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7639999999999994e-06 2024-02-23T00:39:37.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048155+0000", 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5249999999999996e-06 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048160+0000", 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9122999999999997e-05 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.528 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048209+0000", 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7929999999999999e-06 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048213+0000", 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.436081875 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.484295+0000", 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011504499999999999 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.529 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.484410+0000", 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00043144499999999997 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.484842+0000", 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00066020200000000003 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.485502+0000", 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00059683000000000002 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.530 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.486099+0000", 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8315000000000002e-05 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.486137+0000", 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.437999077 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.4.31:55 216.9 216:939aa6ec:test-rados-api-smithi087-82091-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e451)", 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:37:17.129723+0000", 2024-02-23T00:39:37.531 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 140.30165093599999, 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.057627593999999997, 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129723+0000", 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129723+0000", 2024-02-23T00:39:37.532 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129720+0000", 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2629999999999999e-05 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129733+0000", 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9596e-05 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129752+0000", 2024-02-23T00:39:37.533 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8230000000000006e-06 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129762+0000", 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4041000000000005e-05 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129836+0000", 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000126854 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129963+0000", 2024-02-23T00:39:37.534 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00019699 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130160+0000", 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00052119500000000001 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130681+0000", 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00031204600000000001 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130993+0000", 2024-02-23T00:39:37.535 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056278791000000002 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187272+0000", 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3816000000000002e-05 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187316+0000", 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4229e-05 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.536 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187350+0000", 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.057627593999999997 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.6.44:88 63.6 63:6735b1f6:test-rados-api-smithi087-82091-78::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:38:20.305065+0000", 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 77.126308530000003, 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024404019999999998, 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:39:37.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305065+0000", 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305065+0000", 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305063+0000", 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6510000000000001e-06 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.538 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305071+0000", 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6728e-05 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305088+0000", 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5950000000000001e-06 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305095+0000", 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1520000000000002e-05 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:39:37.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305137+0000", 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4927999999999996e-05 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305232+0000", 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00017876200000000001 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 6", 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305410+0000", 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000352796 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305763+0000", 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023542632000000001 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.329306+0000", 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4443000000000006e-05 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.329370+0000", 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8653999999999994e-05 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:39:37.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:39:37.542 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:39:37.542 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.329469+0000", 2024-02-23T00:39:37.542 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024404019999999998 2024-02-23T00:39:37.542 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.542 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.542 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.542 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:39:37.542 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:39:37.542 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:39:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:37.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:35.730183+0000 mgr.y (mgr.24437) 510 : cluster [DBG] pgmap v851: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 23 active+undersized+degraded, 47 active+undersized, 151 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s; 104/1272 objects degraded (8.176%); 74/1272 objects misplaced (5.818%) 2024-02-23T00:39:37.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:36.340402+0000 osd.3 (osd.3) 173 : cluster [DBG] 63.f scrub starts 2024-02-23T00:39:37.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:36.342821+0000 osd.3 (osd.3) 174 : cluster [DBG] 63.f scrub ok 2024-02-23T00:39:37.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:36.348876+0000 osd.1 (osd.1) 255 : cluster [DBG] 63.7 scrub starts 2024-02-23T00:39:37.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:36.352496+0000 osd.1 (osd.1) 256 : cluster [DBG] 63.7 scrub ok 2024-02-23T00:39:37.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:36.785997+0000 osd.5 (osd.5) 217 : cluster [DBG] 3.c scrub starts 2024-02-23T00:39:37.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:36.787445+0000 osd.5 (osd.5) 218 : cluster [DBG] 3.c scrub ok 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:36.900914+0000 osd.7 (osd.7) 266 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:36.902009+0000 osd.7 (osd.7) 267 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.159291+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e558: 8 total, 7 up, 8 in 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:37.223550+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:37.225635+0000 mon.a (mon.0) 3007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:37.228163+0000 mon.c (mon.2) 457 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:37.229716+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:37.230455+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:37.230921+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:39:37.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:35.730183+0000 mgr.y (mgr.24437) 510 : cluster [DBG] pgmap v851: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 23 active+undersized+degraded, 47 active+undersized, 151 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s; 104/1272 objects degraded (8.176%); 74/1272 objects misplaced (5.818%) 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:36.340402+0000 osd.3 (osd.3) 173 : cluster [DBG] 63.f scrub starts 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:36.342821+0000 osd.3 (osd.3) 174 : cluster [DBG] 63.f scrub ok 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:36.348876+0000 osd.1 (osd.1) 255 : cluster [DBG] 63.7 scrub starts 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:36.352496+0000 osd.1 (osd.1) 256 : cluster [DBG] 63.7 scrub ok 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:36.785997+0000 osd.5 (osd.5) 217 : cluster [DBG] 3.c scrub starts 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:36.787445+0000 osd.5 (osd.5) 218 : cluster [DBG] 3.c scrub ok 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:36.900914+0000 osd.7 (osd.7) 266 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:36.902009+0000 osd.7 (osd.7) 267 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.159291+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e558: 8 total, 7 up, 8 in 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:37.223550+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:37.225635+0000 mon.a (mon.0) 3007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:37.228163+0000 mon.c (mon.2) 457 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:37.229716+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:37.230455+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:39:37.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:37.230921+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:39:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:37.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:37.647 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:37.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:35.730183+0000 mgr.y (mgr.24437) 510 : cluster [DBG] pgmap v851: 246 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 22 active+clean+remapped, 2 active+remapped+backfill_toofull, 23 active+undersized+degraded, 47 active+undersized, 151 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s; 104/1272 objects degraded (8.176%); 74/1272 objects misplaced (5.818%) 2024-02-23T00:39:37.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:36.340402+0000 osd.3 (osd.3) 173 : cluster [DBG] 63.f scrub starts 2024-02-23T00:39:37.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:36.342821+0000 osd.3 (osd.3) 174 : cluster [DBG] 63.f scrub ok 2024-02-23T00:39:37.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:36.348876+0000 osd.1 (osd.1) 255 : cluster [DBG] 63.7 scrub starts 2024-02-23T00:39:37.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:36.352496+0000 osd.1 (osd.1) 256 : cluster [DBG] 63.7 scrub ok 2024-02-23T00:39:37.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:36.785997+0000 osd.5 (osd.5) 217 : cluster [DBG] 3.c scrub starts 2024-02-23T00:39:37.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:36.787445+0000 osd.5 (osd.5) 218 : cluster [DBG] 3.c scrub ok 2024-02-23T00:39:37.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:36.900914+0000 osd.7 (osd.7) 266 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:39:37.673 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:36.902009+0000 osd.7 (osd.7) 267 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:39:37.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.159291+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e558: 8 total, 7 up, 8 in 2024-02-23T00:39:37.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:37.223550+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:37.225635+0000 mon.a (mon.0) 3007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:37.228163+0000 mon.c (mon.2) 457 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:37.229716+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:37.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:37.230455+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:39:37.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:37.230921+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:39:37.674 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:37.693 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:37.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:37.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:37.797 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:37.827 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:38.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.307321+0000 osd.6 (osd.6) 230 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:38.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.307453+0000 osd.6 (osd.6) 231 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:38.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.336310+0000 osd.1 (osd.1) 257 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:39:38.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.339871+0000 osd.1 (osd.1) 258 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:39:38.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.342175+0000 osd.4 (osd.4) 259 : cluster [DBG] 4.d scrub starts 2024-02-23T00:39:38.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.345913+0000 osd.4 (osd.4) 260 : cluster [DBG] 4.d scrub ok 2024-02-23T00:39:38.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:38.172442+0000 mon.a (mon.0) 3010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:39:38.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:38.172501+0000 mon.a (mon.0) 3011 : cluster [DBG] osdmap e559: 8 total, 7 up, 8 in 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:38.181625+0000 mon.c (mon.2) 459 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi087-82091-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:38.181993+0000 mon.a (mon.0) 3012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi087-82091-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.307321+0000 osd.6 (osd.6) 230 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.307453+0000 osd.6 (osd.6) 231 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.336310+0000 osd.1 (osd.1) 257 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.339871+0000 osd.1 (osd.1) 258 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.342175+0000 osd.4 (osd.4) 259 : cluster [DBG] 4.d scrub starts 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.345913+0000 osd.4 (osd.4) 260 : cluster [DBG] 4.d scrub ok 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:38.172442+0000 mon.a (mon.0) 3010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:38.172501+0000 mon.a (mon.0) 3011 : cluster [DBG] osdmap e559: 8 total, 7 up, 8 in 2024-02-23T00:39:38.562 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:38.181625+0000 mon.c (mon.2) 459 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi087-82091-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:38.563 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:38.181993+0000 mon.a (mon.0) 3012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi087-82091-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:38.584 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:38.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:38.689 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.307321+0000 osd.6 (osd.6) 230 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.307453+0000 osd.6 (osd.6) 231 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.336310+0000 osd.1 (osd.1) 257 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:39:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.339871+0000 osd.1 (osd.1) 258 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:39:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.342175+0000 osd.4 (osd.4) 259 : cluster [DBG] 4.d scrub starts 2024-02-23T00:39:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.345913+0000 osd.4 (osd.4) 260 : cluster [DBG] 4.d scrub ok 2024-02-23T00:39:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:38.172442+0000 mon.a (mon.0) 3010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:39:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:38.172501+0000 mon.a (mon.0) 3011 : cluster [DBG] osdmap e559: 8 total, 7 up, 8 in 2024-02-23T00:39:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:38.181625+0000 mon.c (mon.2) 459 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi087-82091-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:38.181993+0000 mon.a (mon.0) 3012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi087-82091-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:39:38.717 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:38.840 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:38.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:38.953 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:38.979 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:39.239 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-23T00:39:39.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 -- 172.21.15.87:0/3480262108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840fcbb0 msgr2=0x7fe0840fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3480262108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840fcbb0 0x7fe0840fcf90 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 rx=0x7fe074004870 tx=0x7fe074009cb0).stop 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 -- 172.21.15.87:0/3480262108 shutdown_connections 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3480262108 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0840f7400 0x7fe084103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3480262108 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840fcbb0 0x7fe0840fcf90 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3480262108 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0840fd510 0x7fe0840f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 -- 172.21.15.87:0/3480262108 >> 172.21.15.87:0/3480262108 conn(0x7fe0840f2820 msgr2=0x7fe0840f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 -- 172.21.15.87:0/3480262108 shutdown_connections 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 -- 172.21.15.87:0/3480262108 wait complete. 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 librados: starting msgr at 2024-02-23T00:39:39.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 librados: starting objecter 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.786+0000 7fe08c42f700 1 Processor -- start 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 -- start start 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 librados: setting wanted keys 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 librados: calling monclient init 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840f7400 0x7fe0841092e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0840fcbb0 0x7fe0841097d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0840fd510 0x7fe084109d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0841144d0 con 0x7fe0840f7400 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe084114640 con 0x7fe0840fd510 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe084114800 con 0x7fe0840fcbb0 2024-02-23T00:39:39.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe0899ca700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0840fcbb0 0x7fe0841097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08a1cb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840f7400 0x7fe0841092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe0899ca700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0840fcbb0 0x7fe0841097d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33842/0 (socket says 172.21.15.87:33842) 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe0899ca700 1 -- 172.21.15.87:0/3041017746 learned_addr learned my addr 172.21.15.87:0/3041017746 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08a1cb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840f7400 0x7fe0841092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53540/0 (socket says 172.21.15.87:53540) 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08a9cc700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0840fd510 0x7fe084109d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe0899ca700 1 -- 172.21.15.87:0/3041017746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0840fd510 msgr2=0x7fe084109d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe0899ca700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0840fd510 0x7fe084109d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe0899ca700 1 -- 172.21.15.87:0/3041017746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840f7400 msgr2=0x7fe0841092e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe0899ca700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840f7400 0x7fe0841092e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe0899ca700 1 -- 172.21.15.87:0/3041017746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe080005040 con 0x7fe0840fcbb0 2024-02-23T00:39:39.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08a1cb700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840f7400 0x7fe0841092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08a9cc700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0840fd510 0x7fe084109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe0899ca700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0840fcbb0 0x7fe0841097d0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7fe0800048f0 tx=0x7fe080009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe07b7fe700 10 client.?.objecter ms_handle_connect 0x7fe0840fcbb0 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe07b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe07b7fe700 1 -- 172.21.15.87:0/3041017746 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080015070 con 0x7fe0840fcbb0 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe07b7fe700 1 -- 172.21.15.87:0/3041017746 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe080007b00 con 0x7fe0840fcbb0 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe074005040 con 0x7fe0840fcbb0 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.787+0000 7fe07b7fe700 1 -- 172.21.15.87:0/3041017746 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080019cd0 con 0x7fe0840fcbb0 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.788+0000 7fe08c42f700 10 client.37985.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.788+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe084107490 con 0x7fe0840fcbb0 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.788+0000 7fe08c42f700 1 librados: init done 2024-02-23T00:39:39.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.788+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08405fa60 con 0x7fe0840fcbb0 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe07b7fe700 1 -- 172.21.15.87:0/3041017746 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fe0800108f0 con 0x7fe0840fcbb0 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe07b7fe700 1 --2- 172.21.15.87:0/3041017746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe070062c90 0x7fe070065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe07b7fe700 1 -- 172.21.15.87:0/3041017746 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(559..559 src has 173..559) v4 ==== 8387+0+0 (secure 0 0 0) 0x7fe080081390 con 0x7fe0840fcbb0 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe07b7fe700 10 client.37985.objecter ms_dispatch 0x7fe084001550 osd_map(559..559 src has 173..559) v4 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe07b7fe700 3 client.37985.objecter handle_osd_map got epochs [559,559] > 0 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe07b7fe700 3 client.37985.objecter handle_osd_map decoding full epoch 559 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe08a1cb700 1 --2- 172.21.15.87:0/3041017746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe070062c90 0x7fe070065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe07b7fe700 20 client.37985.objecter dump_active .. 0 homeless 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe08a1cb700 1 --2- 172.21.15.87:0/3041017746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe070062c90 0x7fe070065150 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 rx=0x7fe074004ee0 tx=0x7fe07400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.789+0000 7fe07b7fe700 10 client.37985.objecter ms_handle_connect 0x7fe070062c90 2024-02-23T00:39:39.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.793+0000 7fe07b7fe700 1 -- 172.21.15.87:0/3041017746 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe080054710 con 0x7fe0840fcbb0 2024-02-23T00:39:39.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:38.963+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"} v 0) v1 -- 0x7fe0841060e0 con 0x7fe0840fcbb0 2024-02-23T00:39:39.321 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:39.431 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.430+0000 7fe07b7fe700 1 -- 172.21.15.87:0/3041017746 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]=0 set pool 5 pg_num to 32 v560) v1 ==== 144+0+0 (secure 0 0 0) 0x7fe08001f070 con 0x7fe0840fcbb0 2024-02-23T00:39:39.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.432+0000 7fe08c42f700 10 librados: watch_flush enter 2024-02-23T00:39:39.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.432+0000 7fe08c42f700 10 librados: watch_flush exit 2024-02-23T00:39:39.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.432+0000 7fe08c42f700 20 client.37985.objecter shutdown clearing up homeless session... 2024-02-23T00:39:39.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.432+0000 7fe08c42f700 10 client.37985.objecter successfully canceled tick 2024-02-23T00:39:39.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.432+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe070062c90 msgr2=0x7fe070065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:39.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.432+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3041017746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe070062c90 0x7fe070065150 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 rx=0x7fe074004ee0 tx=0x7fe07400b040).stop 2024-02-23T00:39:39.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.432+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0840fcbb0 msgr2=0x7fe0841097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:39.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.432+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0840fcbb0 0x7fe0841097d0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 rx=0x7fe0800048f0 tx=0x7fe080009cb0).stop 2024-02-23T00:39:39.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.434+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 shutdown_connections 2024-02-23T00:39:39.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.434+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3041017746 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe070062c90 0x7fe070065150 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:39.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.434+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0840fd510 0x7fe084109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:39.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.434+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0840f7400 0x7fe0841092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:39.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.434+0000 7fe08c42f700 1 --2- 172.21.15.87:0/3041017746 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0840fcbb0 0x7fe0841097d0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:39.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.434+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 >> 172.21.15.87:0/3041017746 conn(0x7fe0840f2820 msgr2=0x7fe0840fa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:39.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.434+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 shutdown_connections 2024-02-23T00:39:39.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.434+0000 7fe08c42f700 1 -- 172.21.15.87:0/3041017746 wait complete. 2024-02-23T00:39:39.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:39.434+0000 7fe08c42f700 1 librados: shutdown 2024-02-23T00:39:39.436 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 5 pg_num to 32 2024-02-23T00:39:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.730747+0000 mgr.y (mgr.24437) 511 : cluster [DBG] pgmap v854: 182 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 2 active+remapped+backfill_toofull, 11 active+undersized+degraded, 29 active+undersized, 118 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42/693 objects degraded (6.061%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.736175+0000 osd.2 (osd.2) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:37.736433+0000 osd.2 (osd.2) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:38.285403+0000 mon.a (mon.0) 3013 : cluster [WRN] Health check update: Degraded data redundancy: 42/693 objects degraded (6.061%), 12 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:38.333708+0000 osd.6 (osd.6) 232 : cluster [DBG] 5.21 scrub starts 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:38.337248+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.21 scrub ok 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:38.345531+0000 osd.1 (osd.1) 259 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:38.350882+0000 osd.1 (osd.1) 260 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:38.964227+0000 mon.c (mon.2) 460 : audit [INF] from='client.? 172.21.15.87:0/3041017746' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]: dispatch 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:38.964558+0000 mon.a (mon.0) 3014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]: dispatch 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.730747+0000 mgr.y (mgr.24437) 511 : cluster [DBG] pgmap v854: 182 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 2 active+remapped+backfill_toofull, 11 active+undersized+degraded, 29 active+undersized, 118 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42/693 objects degraded (6.061%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.736175+0000 osd.2 (osd.2) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:37.736433+0000 osd.2 (osd.2) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:38.285403+0000 mon.a (mon.0) 3013 : cluster [WRN] Health check update: Degraded data redundancy: 42/693 objects degraded (6.061%), 12 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:38.333708+0000 osd.6 (osd.6) 232 : cluster [DBG] 5.21 scrub starts 2024-02-23T00:39:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:38.337248+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.21 scrub ok 2024-02-23T00:39:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:38.345531+0000 osd.1 (osd.1) 259 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:39:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:38.350882+0000 osd.1 (osd.1) 260 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:39:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:38.964227+0000 mon.c (mon.2) 460 : audit [INF] from='client.? 172.21.15.87:0/3041017746' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]: dispatch 2024-02-23T00:39:39.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:38.964558+0000 mon.a (mon.0) 3014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]: dispatch 2024-02-23T00:39:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.730747+0000 mgr.y (mgr.24437) 511 : cluster [DBG] pgmap v854: 182 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 2 active+remapped+backfill_toofull, 11 active+undersized+degraded, 29 active+undersized, 118 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42/693 objects degraded (6.061%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.736175+0000 osd.2 (osd.2) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:37.736433+0000 osd.2 (osd.2) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:38.285403+0000 mon.a (mon.0) 3013 : cluster [WRN] Health check update: Degraded data redundancy: 42/693 objects degraded (6.061%), 12 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:38.333708+0000 osd.6 (osd.6) 232 : cluster [DBG] 5.21 scrub starts 2024-02-23T00:39:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:38.337248+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.21 scrub ok 2024-02-23T00:39:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:38.345531+0000 osd.1 (osd.1) 259 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:39:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:38.350882+0000 osd.1 (osd.1) 260 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:39:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:38.964227+0000 mon.c (mon.2) 460 : audit [INF] from='client.? 172.21.15.87:0/3041017746' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]: dispatch 2024-02-23T00:39:39.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:38.964558+0000 mon.a (mon.0) 3014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]: dispatch 2024-02-23T00:39:40.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 -- 172.21.15.87:0/1431964460 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fdbb0 msgr2=0x7fd1500fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:40.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 --2- 172.21.15.87:0/1431964460 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fdbb0 0x7fd1500fdf90 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 rx=0x7fd1400047b0 tx=0x7fd140009cb0).stop 2024-02-23T00:39:40.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 -- 172.21.15.87:0/1431964460 shutdown_connections 2024-02-23T00:39:40.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 --2- 172.21.15.87:0/1431964460 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1500f8420 0x7fd150104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:40.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 --2- 172.21.15.87:0/1431964460 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fdbb0 0x7fd1500fdf90 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 --2- 172.21.15.87:0/1431964460 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1500fe510 0x7fd1500f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 -- 172.21.15.87:0/1431964460 >> 172.21.15.87:0/1431964460 conn(0x7fd1500f3800 msgr2=0x7fd1500f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 -- 172.21.15.87:0/1431964460 shutdown_connections 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 -- 172.21.15.87:0/1431964460 wait complete. 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 librados: starting msgr at 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 librados: starting objecter 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 Processor -- start 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 -- start start 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 librados: setting wanted keys 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.112+0000 7fd156a45700 1 librados: calling monclient init 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1500f8420 0x7fd1501102c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:40.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1500fdbb0 0x7fd1501107b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fe510 0x7fd15010a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15010a950 con 0x7fd1500f8420 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15010aac0 con 0x7fd1500fdbb0 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15010ac80 con 0x7fd1500fe510 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1500f8420 0x7fd1501102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd154fe2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fe510 0x7fd15010a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1500f8420 0x7fd1501102c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53552/0 (socket says 172.21.15.87:53552) 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14ffff700 1 -- 172.21.15.87:0/3037683166 learned_addr learned my addr 172.21.15.87:0/3037683166 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd154fe2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fe510 0x7fd15010a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33846/0 (socket says 172.21.15.87:33846) 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14f7fe700 1 --2- 172.21.15.87:0/3037683166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1500fdbb0 0x7fd1501107b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14ffff700 1 -- 172.21.15.87:0/3037683166 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fe510 msgr2=0x7fd15010a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14ffff700 1 --2- 172.21.15.87:0/3037683166 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fe510 0x7fd15010a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14ffff700 1 -- 172.21.15.87:0/3037683166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1500fdbb0 msgr2=0x7fd1501107b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:40.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14ffff700 1 --2- 172.21.15.87:0/3037683166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1500fdbb0 0x7fd1501107b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14ffff700 1 -- 172.21.15.87:0/3037683166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144005040 con 0x7fd1500f8420 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd154fe2700 1 --2- 172.21.15.87:0/3037683166 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fe510 0x7fd15010a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14ffff700 1 --2- 172.21.15.87:0/3037683166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1500f8420 0x7fd1501102c0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7fd1400044e0 tx=0x7fd140007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14d7fa700 10 client.?.objecter ms_handle_connect 0x7fd1500f8420 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14d7fa700 1 -- 172.21.15.87:0/3037683166 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140018070 con 0x7fd1500f8420 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd140005040 con 0x7fd1500f8420 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14d7fa700 1 -- 172.21.15.87:0/3037683166 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1400093b0 con 0x7fd1500f8420 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd14d7fa700 1 -- 172.21.15.87:0/3037683166 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14001dd90 con 0x7fd1500f8420 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 10 client.27519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15010b1a0 con 0x7fd1500f8420 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.113+0000 7fd156a45700 1 librados: init done 2024-02-23T00:39:40.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.114+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15005ee80 con 0x7fd1500f8420 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14d7fa700 1 -- 172.21.15.87:0/3037683166 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd14001d420 con 0x7fd1500f8420 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14d7fa700 1 --2- 172.21.15.87:0/3037683166 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd138062c90 0x7fd138065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14d7fa700 1 -- 172.21.15.87:0/3037683166 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(560..560 src has 173..560) v4 ==== 8498+0+0 (secure 0 0 0) 0x7fd14007e7d0 con 0x7fd1500f8420 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14d7fa700 10 client.27519.objecter ms_dispatch 0x7fd150001550 osd_map(560..560 src has 173..560) v4 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14d7fa700 3 client.27519.objecter handle_osd_map got epochs [560,560] > 0 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14d7fa700 3 client.27519.objecter handle_osd_map decoding full epoch 560 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14f7fe700 1 --2- 172.21.15.87:0/3037683166 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd138062c90 0x7fd138065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14d7fa700 20 client.27519.objecter dump_active .. 0 homeless 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14d7fa700 10 client.27519.objecter ms_handle_connect 0x7fd138062c90 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.115+0000 7fd14f7fe700 1 --2- 172.21.15.87:0/3037683166 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd138062c90 0x7fd138065150 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7fd15010bc00 tx=0x7fd14400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:40.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.120+0000 7fd14d7fa700 1 -- 172.21.15.87:0/3037683166 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd140051ae0 con 0x7fd1500f8420 2024-02-23T00:39:40.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:40.347+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fd15010ba50 con 0x7fd1500f8420 2024-02-23T00:39:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.310925+0000 osd.1 (osd.1) 261 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:39:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.315755+0000 osd.1 (osd.1) 262 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:39:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.335844+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.e scrub starts 2024-02-23T00:39:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.337295+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.e scrub ok 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.378362+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.381443+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:39.430000+0000 mon.a (mon.0) 3015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]': finished 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.430086+0000 mon.a (mon.0) 3016 : cluster [DBG] osdmap e560: 8 total, 7 up, 8 in 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:39.654367+0000 mon.a (mon.0) 3017 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.731344+0000 mgr.y (mgr.24437) 512 : cluster [DBG] pgmap v857: 182 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 2 active+remapped+backfill_toofull, 11 active+undersized+degraded, 29 active+undersized, 118 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42/693 objects degraded (6.061%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:39.732245+0000 mon.a (mon.0) 3018 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.962679+0000 osd.7 (osd.7) 268 : cluster [DBG] 5.4 deep-scrub starts 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:39.963435+0000 osd.7 (osd.7) 269 : cluster [DBG] 5.4 deep-scrub ok 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:40.174049+0000 mon.a (mon.0) 3019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi087-82091-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi087-82091-94"}]': finished 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:40.174150+0000 mon.a (mon.0) 3020 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]': finished 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:40.174207+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e561: 8 total, 7 up, 8 in 2024-02-23T00:39:40.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:40.347896+0000 mon.a (mon.0) 3022 : audit [INF] from='client.? 172.21.15.87:0/3037683166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:39:40.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.310925+0000 osd.1 (osd.1) 261 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:39:40.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.315755+0000 osd.1 (osd.1) 262 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:39:40.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.335844+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.e scrub starts 2024-02-23T00:39:40.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.337295+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.e scrub ok 2024-02-23T00:39:40.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.378362+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:39:40.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.381443+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:39.430000+0000 mon.a (mon.0) 3015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]': finished 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.430086+0000 mon.a (mon.0) 3016 : cluster [DBG] osdmap e560: 8 total, 7 up, 8 in 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:39.654367+0000 mon.a (mon.0) 3017 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.731344+0000 mgr.y (mgr.24437) 512 : cluster [DBG] pgmap v857: 182 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 2 active+remapped+backfill_toofull, 11 active+undersized+degraded, 29 active+undersized, 118 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42/693 objects degraded (6.061%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:39.732245+0000 mon.a (mon.0) 3018 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.962679+0000 osd.7 (osd.7) 268 : cluster [DBG] 5.4 deep-scrub starts 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:39.963435+0000 osd.7 (osd.7) 269 : cluster [DBG] 5.4 deep-scrub ok 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:40.174049+0000 mon.a (mon.0) 3019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi087-82091-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi087-82091-94"}]': finished 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:40.174150+0000 mon.a (mon.0) 3020 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]': finished 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:40.174207+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e561: 8 total, 7 up, 8 in 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:40.347896+0000 mon.a (mon.0) 3022 : audit [INF] from='client.? 172.21.15.87:0/3037683166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:39:40.700 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.310925+0000 osd.1 (osd.1) 261 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.315755+0000 osd.1 (osd.1) 262 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.335844+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.e scrub starts 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.337295+0000 osd.2 (osd.2) 214 : cluster [DBG] 2.e scrub ok 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.378362+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.13 scrub starts 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.381443+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.13 scrub ok 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:39.430000+0000 mon.a (mon.0) 3015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]': finished 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.430086+0000 mon.a (mon.0) 3016 : cluster [DBG] osdmap e560: 8 total, 7 up, 8 in 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:39.654367+0000 mon.a (mon.0) 3017 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.731344+0000 mgr.y (mgr.24437) 512 : cluster [DBG] pgmap v857: 182 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 21 active+clean+remapped, 2 active+remapped+backfill_toofull, 11 active+undersized+degraded, 29 active+undersized, 118 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 42/693 objects degraded (6.061%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:39.732245+0000 mon.a (mon.0) 3018 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.962679+0000 osd.7 (osd.7) 268 : cluster [DBG] 5.4 deep-scrub starts 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:39.963435+0000 osd.7 (osd.7) 269 : cluster [DBG] 5.4 deep-scrub ok 2024-02-23T00:39:40.701 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:40.174049+0000 mon.a (mon.0) 3019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi087-82091-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi087-82091-94"}]': finished 2024-02-23T00:39:40.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:40.174150+0000 mon.a (mon.0) 3020 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]': finished 2024-02-23T00:39:40.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:40.174207+0000 mon.a (mon.0) 3021 : cluster [DBG] osdmap e561: 8 total, 7 up, 8 in 2024-02-23T00:39:40.702 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:40.347896+0000 mon.a (mon.0) 3022 : audit [INF] from='client.? 172.21.15.87:0/3037683166' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:39:40.721 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:40.983 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:40.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:41.087 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:41.115 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:41.133 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:41.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:41.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:41.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.183+0000 7fd14d7fa700 1 -- 172.21.15.87:0/3037683166 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v562) v1 ==== 87+0+0 (secure 0 0 0) 0x7fd1400516d0 con 0x7fd1500f8420 2024-02-23T00:39:41.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.186+0000 7fd156a45700 10 librados: watch_flush enter 2024-02-23T00:39:41.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.189+0000 7fd156a45700 10 librados: watch_flush exit 2024-02-23T00:39:41.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.189+0000 7fd156a45700 20 client.27519.objecter shutdown clearing up homeless session... 2024-02-23T00:39:41.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.189+0000 7fd156a45700 10 client.27519.objecter successfully canceled tick 2024-02-23T00:39:41.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.200+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd138062c90 msgr2=0x7fd138065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:41.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.200+0000 7fd156a45700 1 --2- 172.21.15.87:0/3037683166 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd138062c90 0x7fd138065150 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 rx=0x7fd15010bc00 tx=0x7fd14400b040).stop 2024-02-23T00:39:41.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.200+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1500f8420 msgr2=0x7fd1501102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:41.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.200+0000 7fd156a45700 1 --2- 172.21.15.87:0/3037683166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1500f8420 0x7fd1501102c0 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 rx=0x7fd1400044e0 tx=0x7fd140007f90).stop 2024-02-23T00:39:41.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.201+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 shutdown_connections 2024-02-23T00:39:41.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.201+0000 7fd156a45700 1 --2- 172.21.15.87:0/3037683166 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd138062c90 0x7fd138065150 secure :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 rx=0x7fd15010bc00 tx=0x7fd14400b040).stop 2024-02-23T00:39:41.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.201+0000 7fd156a45700 1 --2- 172.21.15.87:0/3037683166 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1500fe510 0x7fd15010a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:41.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.201+0000 7fd156a45700 1 --2- 172.21.15.87:0/3037683166 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1500f8420 0x7fd1501102c0 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:41.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.201+0000 7fd156a45700 1 --2- 172.21.15.87:0/3037683166 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1500fdbb0 0x7fd1501107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:41.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.201+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 >> 172.21.15.87:0/3037683166 conn(0x7fd1500f3800 msgr2=0x7fd1500fb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:41.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.206+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 shutdown_connections 2024-02-23T00:39:41.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.206+0000 7fd156a45700 1 -- 172.21.15.87:0/3037683166 wait complete. 2024-02-23T00:39:41.208 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:41.207+0000 7fd156a45700 1 librados: shutdown 2024-02-23T00:39:41.209 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:39:41.237 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:41.266 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:41.285 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:41.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:41.389 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:41.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:39:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:40.284700+0000 osd.1 (osd.1) 263 : cluster [DBG] 2.a scrub starts 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:40.287602+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.a scrub ok 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:40.366947+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:40.370087+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:40.689523+0000 osd.5 (osd.5) 219 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:40.690530+0000 osd.5 (osd.5) 220 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:40.836264+0000 osd.1 (osd.1) 265 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:40.836523+0000 osd.1 (osd.1) 266 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:41.082288+0000 mgr.y (mgr.24437) 513 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:41.170530+0000 mon.a (mon.0) 3023 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:41.180306+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? 172.21.15.87:0/3037683166' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:39:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:41.180372+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e562: 8 total, 7 up, 8 in 2024-02-23T00:39:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:40.284700+0000 osd.1 (osd.1) 263 : cluster [DBG] 2.a scrub starts 2024-02-23T00:39:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:40.287602+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.a scrub ok 2024-02-23T00:39:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:40.366947+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:39:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:40.370087+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:39:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:40.689523+0000 osd.5 (osd.5) 219 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:39:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:40.690530+0000 osd.5 (osd.5) 220 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:39:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:40.836264+0000 osd.1 (osd.1) 265 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:40.836523+0000 osd.1 (osd.1) 266 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:41.082288+0000 mgr.y (mgr.24437) 513 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:41.170530+0000 mon.a (mon.0) 3023 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:41.180306+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? 172.21.15.87:0/3037683166' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:41.180372+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e562: 8 total, 7 up, 8 in 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:40.284700+0000 osd.1 (osd.1) 263 : cluster [DBG] 2.a scrub starts 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:40.287602+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.a scrub ok 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:40.366947+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:40.370087+0000 osd.3 (osd.3) 178 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:40.689523+0000 osd.5 (osd.5) 219 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:39:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:40.690530+0000 osd.5 (osd.5) 220 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:39:42.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:40.836264+0000 osd.1 (osd.1) 265 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:42.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:40.836523+0000 osd.1 (osd.1) 266 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:42.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:41.082288+0000 mgr.y (mgr.24437) 513 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:42.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:41.170530+0000 mon.a (mon.0) 3023 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:42.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:41.180306+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? 172.21.15.87:0/3037683166' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:39:42.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:41.180372+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e562: 8 total, 7 up, 8 in 2024-02-23T00:39:42.527 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:39:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:42] "GET /metrics HTTP/1.1" 200 184596 "" "Prometheus/2.33.4" 2024-02-23T00:39:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:41.732205+0000 mgr.y (mgr.24437) 514 : cluster [DBG] pgmap v860: 190 pgs: 1 remapped+peering, 7 peering, 8 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 20 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 28 active+undersized, 113 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/693 objects degraded (5.916%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:41.934298+0000 osd.5 (osd.5) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:41.934580+0000 osd.5 (osd.5) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:42.160653+0000 mon.a (mon.0) 3026 : cluster [DBG] osdmap e563: 8 total, 7 up, 8 in 2024-02-23T00:39:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:42.163911+0000 mon.c (mon.2) 461 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:42.164470+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:43.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:42.654888+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:39:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:41.732205+0000 mgr.y (mgr.24437) 514 : cluster [DBG] pgmap v860: 190 pgs: 1 remapped+peering, 7 peering, 8 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 20 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 28 active+undersized, 113 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/693 objects degraded (5.916%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:41.934298+0000 osd.5 (osd.5) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:41.934580+0000 osd.5 (osd.5) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:42.160653+0000 mon.a (mon.0) 3026 : cluster [DBG] osdmap e563: 8 total, 7 up, 8 in 2024-02-23T00:39:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:42.163911+0000 mon.c (mon.2) 461 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:42.164470+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:43.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:42.654888+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:39:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:41.732205+0000 mgr.y (mgr.24437) 514 : cluster [DBG] pgmap v860: 190 pgs: 1 remapped+peering, 7 peering, 8 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 20 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 28 active+undersized, 113 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/693 objects degraded (5.916%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:41.934298+0000 osd.5 (osd.5) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:41.934580+0000 osd.5 (osd.5) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:42.160653+0000 mon.a (mon.0) 3026 : cluster [DBG] osdmap e563: 8 total, 7 up, 8 in 2024-02-23T00:39:43.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:42.163911+0000 mon.c (mon.2) 461 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:42.164470+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:43.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:42.654888+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:39:43.477 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:43.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:43.581 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:43.608 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:43.626 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:43.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:43.730 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:43.759 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:43.832 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:43.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:43.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:39:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:39:43.952 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:43.980 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:44.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.037+0000 7f54dbd42700 1 -- 172.21.15.87:0/4030919853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54d40f96e0 msgr2=0x7f54d4101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:44.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.037+0000 7f54dbd42700 1 --2- 172.21.15.87:0/4030919853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54d40f96e0 0x7f54d4101c30 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 rx=0x7f54c8004870 tx=0x7f54c8009cb0).stop 2024-02-23T00:39:44.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.037+0000 7f54dbd42700 1 -- 172.21.15.87:0/4030919853 shutdown_connections 2024-02-23T00:39:44.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.037+0000 7f54dbd42700 1 --2- 172.21.15.87:0/4030919853 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54d4102250 0x7f54d4104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:44.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.037+0000 7f54dbd42700 1 --2- 172.21.15.87:0/4030919853 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54d40f8d80 0x7f54d40f9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:44.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.037+0000 7f54dbd42700 1 --2- 172.21.15.87:0/4030919853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54d40f96e0 0x7f54d4101c30 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:44.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.037+0000 7f54dbd42700 1 -- 172.21.15.87:0/4030919853 >> 172.21.15.87:0/4030919853 conn(0x7f54d40f26f0 msgr2=0x7f54d40f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.038+0000 7f54dbd42700 1 -- 172.21.15.87:0/4030919853 shutdown_connections 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.038+0000 7f54dbd42700 1 -- 172.21.15.87:0/4030919853 wait complete. 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.038+0000 7f54dbd42700 1 librados: starting msgr at 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.038+0000 7f54dbd42700 1 librados: starting objecter 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.038+0000 7f54dbd42700 1 Processor -- start 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.038+0000 7f54dbd42700 1 -- start start 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.038+0000 7f54dbd42700 1 librados: setting wanted keys 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.038+0000 7f54dbd42700 1 librados: calling monclient init 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54d40f8d80 0x7f54d410e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54d40f96e0 0x7f54d410e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54d4102250 0x7f54d4108330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4108820 con 0x7f54d40f96e0 2024-02-23T00:39:44.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4108990 con 0x7f54d40f8d80 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d4108b50 con 0x7f54d4102250 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d92dd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54d40f96e0 0x7f54d410e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d92dd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54d40f96e0 0x7f54d410e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58776/0 (socket says 172.21.15.87:58776) 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d92dd700 1 -- 172.21.15.87:0/1693079246 learned_addr learned my addr 172.21.15.87:0/1693079246 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54da2df700 1 --2- 172.21.15.87:0/1693079246 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54d4102250 0x7f54d4108330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d9ade700 1 --2- 172.21.15.87:0/1693079246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54d40f8d80 0x7f54d410e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d92dd700 1 -- 172.21.15.87:0/1693079246 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54d4102250 msgr2=0x7f54d4108330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d92dd700 1 --2- 172.21.15.87:0/1693079246 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54d4102250 0x7f54d4108330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d92dd700 1 -- 172.21.15.87:0/1693079246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54d40f8d80 msgr2=0x7f54d410e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d92dd700 1 --2- 172.21.15.87:0/1693079246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54d40f8d80 0x7f54d410e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d92dd700 1 -- 172.21.15.87:0/1693079246 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc005040 con 0x7f54d40f96e0 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d9ade700 1 --2- 172.21.15.87:0/1693079246 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54d40f8d80 0x7f54d410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54da2df700 1 --2- 172.21.15.87:0/1693079246 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54d4102250 0x7f54d4108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:44.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d92dd700 1 --2- 172.21.15.87:0/1693079246 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54d40f96e0 0x7f54d410e680 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 rx=0x7f54c8009ee0 tx=0x7f54c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d2ffd700 10 client.?.objecter ms_handle_connect 0x7f54d40f96e0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d2ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c800e070 con 0x7f54d40f96e0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c8005040 con 0x7f54d40f96e0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54c8013da0 con 0x7f54d40f96e0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8019070 con 0x7f54d40f96e0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 10 client.27528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d4109070 con 0x7f54d40f96e0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.039+0000 7f54dbd42700 1 librados: init done 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.040+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f54d405ee80 con 0x7f54d40f96e0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.041+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f54c80090d0 con 0x7f54d40f96e0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.041+0000 7f54d2ffd700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6800/1231597463 conn(0x7f54bc062c90 0x7f54bc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(564..564 src has 173..564) v4 ==== 9436+0+0 (secure 0 0 0) 0x7f54c807e1f0 con 0x7f54d40f96e0 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 osd_map(564..564 src has 173..564) v4 2024-02-23T00:39:44.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d2ffd700 3 client.27528.objecter handle_osd_map got epochs [564,564] > 0 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d2ffd700 3 client.27528.objecter handle_osd_map decoding full epoch 564 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d9ade700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6800/1231597463 conn(0x7f54bc062c90 0x7f54bc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d2ffd700 20 client.27528.objecter dump_active .. 0 homeless 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v564) v1 ==== 54+0+16 (secure 0 0 0) 0x7f54c807e690 con 0x7f54d40f96e0 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d2ffd700 10 client.27528.objecter ms_handle_connect 0x7f54bc062c90 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d9ade700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6800/1231597463 conn(0x7f54bc062c90 0x7f54bc065150 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 rx=0x7f54c4007e00 tx=0x7f54c4009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 10 client.27528.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 1 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 20 client.27528.objecter _get_session osd=-1 returning homeless 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 10 client.27528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=565}) v3 -- 0x7f54d4107250 con 0x7f54d40f96e0 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f54d4114fc0 con 0x7f54d40f96e0 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_get_version_reply(handle=1 version=564) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54c804d020 con 0x7f54d40f96e0 2024-02-23T00:39:44.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d3fff700 10 client.27528.objecter _check_command_map_dne tid 1 current 564 map_dne_bound 564 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d3fff700 10 client.27528.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54d3fff700 15 client.27528.objecter _session_command_op_remove -1 1 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 10 client.27528.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 2 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6805/685354171 conn(0x7f54d40f4dc0 0x7f54d40273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40f4c50 osd=1 3 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40f4c50 osd=1 3 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40f4c50 osd=1 3 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 2 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d40f4c50 osd=1 3 2024-02-23T00:39:44.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 1 2 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40f4c50 osd=1 4 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 10 client.27528.objecter _send_command 2 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d4059e00 con 0x7f54d40f4dc0 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.042+0000 7f54da2df700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6805/685354171 conn(0x7f54d40f4dc0 0x7f54d40273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.043+0000 7f54d2ffd700 10 client.27528.objecter ms_handle_connect 0x7f54d40f4dc0 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.043+0000 7f54da2df700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6805/685354171 conn(0x7f54d40f4dc0 0x7f54d40273c0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.043+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f54d4059e00 con 0x7f54d40f4dc0 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.043+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.043+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 2 = system:0 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.043+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d40f4c50 osd=1 4 2024-02-23T00:39:44.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.043+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 1 2 2024-02-23T00:39:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:44.055 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:44.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 10 client.27528.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 3 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40f4c50 osd=1 3 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40f4c50 osd=1 3 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40f4c50 osd=1 3 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 3 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d40f4c50 osd=1 3 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 1 3 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40f4c50 osd=1 4 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 10 client.27528.objecter _send_command 3 2024-02-23T00:39:44.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.059+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f54d40278b0 con 0x7f54d40f4dc0 2024-02-23T00:39:44.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f54d40278b0 con 0x7f54d40f4dc0 2024-02-23T00:39:44.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:44.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d40f4c50 osd=1 4 2024-02-23T00:39:44.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 1 3 2024-02-23T00:39:44.062 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 10 client.27528.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 4 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6809/3605394228 conn(0x7f54d40594d0 0x7f54d4119150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4059360 osd=2 3 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4059360 osd=2 3 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4059360 osd=2 3 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 4 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d4059360 osd=2 3 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 2 4 2024-02-23T00:39:44.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4059360 osd=2 4 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 10 client.27528.objecter _send_command 4 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.060+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d4028760 con 0x7f54d40594d0 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.061+0000 7f54d9ade700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6809/3605394228 conn(0x7f54d40594d0 0x7f54d4119150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.061+0000 7f54d2ffd700 10 client.27528.objecter ms_handle_connect 0x7f54d40594d0 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.061+0000 7f54d9ade700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6809/3605394228 conn(0x7f54d40594d0 0x7f54d4119150 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.061+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f54d4028760 con 0x7f54d40594d0 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.061+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.061+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 4 = system:0 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.061+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d4059360 osd=2 4 2024-02-23T00:39:44.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.061+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 2 4 2024-02-23T00:39:44.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 10 client.27528.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:44.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 5 2024-02-23T00:39:44.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4059360 osd=2 3 2024-02-23T00:39:44.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4059360 osd=2 3 2024-02-23T00:39:44.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4059360 osd=2 3 2024-02-23T00:39:44.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 5 2024-02-23T00:39:44.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d4059360 osd=2 3 2024-02-23T00:39:44.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 2 5 2024-02-23T00:39:44.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4059360 osd=2 4 2024-02-23T00:39:44.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 10 client.27528.objecter _send_command 5 2024-02-23T00:39:44.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.076+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f54d4027cc0 con 0x7f54d40594d0 2024-02-23T00:39:44.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.077+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f54d4027cc0 con 0x7f54d40594d0 2024-02-23T00:39:44.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.077+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:44.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.077+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.077+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d4059360 osd=2 4 2024-02-23T00:39:44.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.077+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 2 5 2024-02-23T00:39:44.079 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:39:44.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 10 client.27528.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:44.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 6 2024-02-23T00:39:44.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6813/128059790 conn(0x7f54d411a7c0 0x7f54d411cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40fc3b0 osd=3 3 2024-02-23T00:39:44.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40fc3b0 osd=3 3 2024-02-23T00:39:44.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40fc3b0 osd=3 3 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 6 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d40fc3b0 osd=3 3 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 3 6 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40fc3b0 osd=3 4 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 10 client.27528.objecter _send_command 6 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d40fca30 con 0x7f54d411a7c0 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54d92dd700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6813/128059790 conn(0x7f54d411a7c0 0x7f54d411cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54d92dd700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.87:6813/128059790 conn(0x7f54d411a7c0 0x7f54d411cbb0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.078+0000 7f54d2ffd700 10 client.27528.objecter ms_handle_connect 0x7f54d411a7c0 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.079+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f54d40fca30 con 0x7f54d411a7c0 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.079+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.079+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 6 = system:0 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.079+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d40fc3b0 osd=3 4 2024-02-23T00:39:44.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.079+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 3 6 2024-02-23T00:39:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 10 client.27528.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 7 2024-02-23T00:39:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40fc3b0 osd=3 3 2024-02-23T00:39:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40fc3b0 osd=3 3 2024-02-23T00:39:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40fc3b0 osd=3 3 2024-02-23T00:39:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 7 2024-02-23T00:39:44.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d40fc3b0 osd=3 3 2024-02-23T00:39:44.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 3 7 2024-02-23T00:39:44.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40fc3b0 osd=3 4 2024-02-23T00:39:44.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 10 client.27528.objecter _send_command 7 2024-02-23T00:39:44.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.093+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f54d40fbe20 con 0x7f54d411a7c0 2024-02-23T00:39:44.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f54d40fbe20 con 0x7f54d411a7c0 2024-02-23T00:39:44.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:44.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d40fc3b0 osd=3 4 2024-02-23T00:39:44.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 3 7 2024-02-23T00:39:44.096 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:39:44.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 10 client.27528.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:44.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 8 2024-02-23T00:39:44.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6800/4175844757 conn(0x7f54d411a320 0x7f54d4120230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40fc230 osd=4 3 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40fc230 osd=4 3 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40fc230 osd=4 3 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 8 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d40fc230 osd=4 3 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 4 8 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40fc230 osd=4 4 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 10 client.27528.objecter _send_command 8 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.094+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d4122f20 con 0x7f54d411a320 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.095+0000 7f54da2df700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6800/4175844757 conn(0x7f54d411a320 0x7f54d4120230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.095+0000 7f54d2ffd700 10 client.27528.objecter ms_handle_connect 0x7f54d411a320 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.095+0000 7f54da2df700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6800/4175844757 conn(0x7f54d411a320 0x7f54d4120230 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.096+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f54d4122f20 con 0x7f54d411a320 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.096+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.096+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 8 = system:0 2024-02-23T00:39:44.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.096+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d40fc230 osd=4 4 2024-02-23T00:39:44.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.096+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 4 8 2024-02-23T00:39:44.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 10 client.27528.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 9 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40fc230 osd=4 3 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40fc230 osd=4 3 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d40fc230 osd=4 3 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 9 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d40fc230 osd=4 3 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 4 9 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d40fc230 osd=4 4 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 10 client.27528.objecter _send_command 9 2024-02-23T00:39:44.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.110+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f54d4119880 con 0x7f54d411a320 2024-02-23T00:39:44.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.112+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f54d4119880 con 0x7f54d411a320 2024-02-23T00:39:44.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.112+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:44.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.112+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.112+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d40fc230 osd=4 4 2024-02-23T00:39:44.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.112+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 4 9 2024-02-23T00:39:44.114 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 10 client.27528.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 10 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6804/2550013795 conn(0x7f54d411dd80 0x7f54d4120720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4119a50 osd=5 3 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4119a50 osd=5 3 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4119a50 osd=5 3 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 10 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d4119a50 osd=5 3 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 5 10 2024-02-23T00:39:44.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4119a50 osd=5 4 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 10 client.27528.objecter _send_command 10 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d4119bc0 con 0x7f54d411dd80 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54d9ade700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6804/2550013795 conn(0x7f54d411dd80 0x7f54d4120720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54d2ffd700 10 client.27528.objecter ms_handle_connect 0x7f54d411dd80 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.113+0000 7f54d9ade700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6804/2550013795 conn(0x7f54d411dd80 0x7f54d4120720 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.114+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f54d4119bc0 con 0x7f54d411dd80 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.114+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.114+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 10 = system:0 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.114+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d4119a50 osd=5 4 2024-02-23T00:39:44.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.114+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 5 10 2024-02-23T00:39:44.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 10 client.27528.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:44.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 11 2024-02-23T00:39:44.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4119a50 osd=5 3 2024-02-23T00:39:44.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4119a50 osd=5 3 2024-02-23T00:39:44.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4119a50 osd=5 3 2024-02-23T00:39:44.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 11 2024-02-23T00:39:44.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d4119a50 osd=5 3 2024-02-23T00:39:44.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 5 11 2024-02-23T00:39:44.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4119a50 osd=5 4 2024-02-23T00:39:44.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 10 client.27528.objecter _send_command 11 2024-02-23T00:39:44.131 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.129+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f54d4119bc0 con 0x7f54d411dd80 2024-02-23T00:39:44.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f54d4119bc0 con 0x7f54d411dd80 2024-02-23T00:39:44.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:44.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d4119a50 osd=5 4 2024-02-23T00:39:44.132 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 5 11 2024-02-23T00:39:44.132 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:39:44.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 10 client.27528.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:44.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 12 2024-02-23T00:39:44.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6808/2509408498 conn(0x7f54d41211b0 0x7f54d41215f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4119bc0 osd=6 3 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4119bc0 osd=6 3 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4119bc0 osd=6 3 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 12 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d4119bc0 osd=6 3 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 6 12 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4119bc0 osd=6 4 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 10 client.27528.objecter _send_command 12 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d41298c0 con 0x7f54d41211b0 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54d92dd700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6808/2509408498 conn(0x7f54d41211b0 0x7f54d41215f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54d92dd700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6808/2509408498 conn(0x7f54d41211b0 0x7f54d41215f0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.131+0000 7f54d2ffd700 10 client.27528.objecter ms_handle_connect 0x7f54d41211b0 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.132+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f54d41298c0 con 0x7f54d41211b0 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.132+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.132+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 12 = system:0 2024-02-23T00:39:44.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.132+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d4119bc0 osd=6 4 2024-02-23T00:39:44.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.132+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 6 12 2024-02-23T00:39:44.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 10 client.27528.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:44.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 13 2024-02-23T00:39:44.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4119bc0 osd=6 3 2024-02-23T00:39:44.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4119bc0 osd=6 3 2024-02-23T00:39:44.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4119bc0 osd=6 3 2024-02-23T00:39:44.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 13 2024-02-23T00:39:44.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d4119bc0 osd=6 3 2024-02-23T00:39:44.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 6 13 2024-02-23T00:39:44.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4119bc0 osd=6 4 2024-02-23T00:39:44.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 10 client.27528.objecter _send_command 13 2024-02-23T00:39:44.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.146+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f54d4127160 con 0x7f54d41211b0 2024-02-23T00:39:44.149 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:39:44.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f54d4127160 con 0x7f54d41211b0 2024-02-23T00:39:44.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:44.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d4119bc0 osd=6 4 2024-02-23T00:39:44.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 6 13 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 10 client.27528.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 14 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6812/373787000 conn(0x7f54d4127820 0x7f54d4127c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4127160 osd=7 3 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4127160 osd=7 3 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4127160 osd=7 3 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 14 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d4127160 osd=7 3 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 7 14 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4127160 osd=7 4 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 10 client.27528.objecter _send_command 14 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54d4129890 con 0x7f54d4127820 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.148+0000 7f54da2df700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6812/373787000 conn(0x7f54d4127820 0x7f54d4127c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.149+0000 7f54d2ffd700 10 client.27528.objecter ms_handle_connect 0x7f54d4127820 2024-02-23T00:39:44.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.149+0000 7f54da2df700 1 --2- 172.21.15.87:0/1693079246 >> v2:172.21.15.171:6812/373787000 conn(0x7f54d4127820 0x7f54d4127c40 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:44.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.149+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f54d4129890 con 0x7f54d4127820 2024-02-23T00:39:44.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.149+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:39:44.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.149+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 14 = system:0 2024-02-23T00:39:44.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.149+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d4127160 osd=7 4 2024-02-23T00:39:44.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.149+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 7 14 2024-02-23T00:39:44.156 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 10 client.27528.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:39:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign -1 15 2024-02-23T00:39:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4127160 osd=7 3 2024-02-23T00:39:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4127160 osd=7 3 2024-02-23T00:39:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 20 client.27528.objecter _get_session s=0x7f54d4127160 osd=7 3 2024-02-23T00:39:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_remove -1 15 2024-02-23T00:39:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 20 client.27528.objecter get_session s=0x7f54d4127160 osd=7 3 2024-02-23T00:39:44.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 15 client.27528.objecter _session_command_op_assign 7 15 2024-02-23T00:39:44.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 20 client.27528.objecter put_session s=0x7f54d4127160 osd=7 4 2024-02-23T00:39:44.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 10 client.27528.objecter _send_command 15 2024-02-23T00:39:44.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.164+0000 7f54dbd42700 1 -- 172.21.15.87:0/1693079246 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f54d4129890 con 0x7f54d4127820 2024-02-23T00:39:44.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.166+0000 7f54d2ffd700 1 -- 172.21.15.87:0/1693079246 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f54d4129890 con 0x7f54d4127820 2024-02-23T00:39:44.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.166+0000 7f54d2ffd700 10 client.27528.objecter ms_dispatch 0x7f54d4001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:39:44.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.166+0000 7f54d2ffd700 10 client.27528.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.166+0000 7f54d2ffd700 20 client.27528.objecter put_session s=0x7f54d4127160 osd=7 4 2024-02-23T00:39:44.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:44.166+0000 7f54d2ffd700 15 client.27528.objecter _session_command_op_remove 7 15 2024-02-23T00:39:44.167 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:39:44.168 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:39:44.168 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.168 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.168 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.168 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.168 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.168 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.168 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:39:44.210 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:43.161427+0000 mon.a (mon.0) 3029 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:43.161587+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e564: 8 total, 7 up, 8 in 2024-02-23T00:39:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:44.041267+0000 mon.a (mon.0) 3031 : audit [DBG] from='client.? 172.21.15.87:0/1693079246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:43.161427+0000 mon.a (mon.0) 3029 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:43.161587+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e564: 8 total, 7 up, 8 in 2024-02-23T00:39:44.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:44.041267+0000 mon.a (mon.0) 3031 : audit [DBG] from='client.? 172.21.15.87:0/1693079246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:43.161427+0000 mon.a (mon.0) 3029 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:43.161587+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e564: 8 total, 7 up, 8 in 2024-02-23T00:39:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:44.041267+0000 mon.a (mon.0) 3031 : audit [DBG] from='client.? 172.21.15.87:0/1693079246' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:39:44.545 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:39:44.866 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:44.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:44.971 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:44.999 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:45.097 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:45.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:45.124 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:39:45.124 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:39:45.201 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:45.235 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:45.358 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:45.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:43.732800+0000 mgr.y (mgr.24437) 515 : cluster [DBG] pgmap v863: 222 pgs: 1 remapped+peering, 7 peering, 32 unknown, 8 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 20 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 28 active+undersized, 113 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/693 objects degraded (5.916%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:45.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:44.178316+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e565: 8 total, 7 up, 8 in 2024-02-23T00:39:45.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:44.181569+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:45.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:44.182705+0000 mon.a (mon.0) 3033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:45.361 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:44.412464+0000 mon.a (mon.0) 3034 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:39:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:45.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:45.463 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:45.489 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:43.732800+0000 mgr.y (mgr.24437) 515 : cluster [DBG] pgmap v863: 222 pgs: 1 remapped+peering, 7 peering, 32 unknown, 8 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 20 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 28 active+undersized, 113 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/693 objects degraded (5.916%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:45.490 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:44.178316+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e565: 8 total, 7 up, 8 in 2024-02-23T00:39:45.490 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:44.181569+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:45.490 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:44.182705+0000 mon.a (mon.0) 3033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:45.490 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:44.412464+0000 mon.a (mon.0) 3034 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:39:45.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:43.732800+0000 mgr.y (mgr.24437) 515 : cluster [DBG] pgmap v863: 222 pgs: 1 remapped+peering, 7 peering, 32 unknown, 8 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 20 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 28 active+undersized, 113 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/693 objects degraded (5.916%); 71/693 objects misplaced (10.245%) 2024-02-23T00:39:45.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:44.178316+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e565: 8 total, 7 up, 8 in 2024-02-23T00:39:45.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:44.181569+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:45.490 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:44.182705+0000 mon.a (mon.0) 3033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:45.491 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:44.412464+0000 mon.a (mon.0) 3034 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:39:45.492 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:45.722 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:39:45.723 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:39:45.723 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.meta 2024-02-23T00:39:45.723 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.meta by 10 min 8 2024-02-23T00:39:45.723 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:39:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:39:46.337 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:39:46.341 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:46.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:46.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:45.178616+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:45.178756+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e566: 8 total, 7 up, 8 in 2024-02-23T00:39:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:45.181816+0000 mon.c (mon.2) 463 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:45.182938+0000 mon.a (mon.0) 3037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:45.735056+0000 mon.a (mon.0) 3038 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "41"}]: dispatch 2024-02-23T00:39:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:46.179953+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]': finished 2024-02-23T00:39:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:46.180008+0000 mon.a (mon.0) 3040 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "41"}]': finished 2024-02-23T00:39:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:46.180051+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e567: 8 total, 7 up, 8 in 2024-02-23T00:39:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:46.183869+0000 mon.c (mon.2) 464 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-97", "overlaypool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:46.185595+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-97", "overlaypool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.446 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:46.473 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:46.510 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:46.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:45.178616+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:45.178756+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e566: 8 total, 7 up, 8 in 2024-02-23T00:39:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:45.181816+0000 mon.c (mon.2) 463 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:45.182938+0000 mon.a (mon.0) 3037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:45.735056+0000 mon.a (mon.0) 3038 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "41"}]: dispatch 2024-02-23T00:39:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:46.179953+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]': finished 2024-02-23T00:39:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:46.180008+0000 mon.a (mon.0) 3040 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "41"}]': finished 2024-02-23T00:39:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:46.180051+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e567: 8 total, 7 up, 8 in 2024-02-23T00:39:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:46.183869+0000 mon.c (mon.2) 464 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-97", "overlaypool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:46.185595+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-97", "overlaypool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:45.178616+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:46.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:45.178756+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e566: 8 total, 7 up, 8 in 2024-02-23T00:39:46.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:45.181816+0000 mon.c (mon.2) 463 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:45.182938+0000 mon.a (mon.0) 3037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:45.735056+0000 mon.a (mon.0) 3038 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "41"}]: dispatch 2024-02-23T00:39:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:46.179953+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]': finished 2024-02-23T00:39:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:46.180008+0000 mon.a (mon.0) 3040 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "41"}]': finished 2024-02-23T00:39:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:46.180051+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e567: 8 total, 7 up, 8 in 2024-02-23T00:39:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:46.183869+0000 mon.c (mon.2) 464 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-97", "overlaypool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:46.185595+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-97", "overlaypool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:46.614 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:46.641 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:46.693 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:46.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:46.758 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-23T00:39:46.771 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:46.772 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:46.797 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:46.832 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:47.081 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:47.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:47.186 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:47.216 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:47.338 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:47.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:47.400 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:45.733927+0000 mgr.y (mgr.24437) 516 : cluster [DBG] pgmap v866: 254 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 58 active+undersized, 158 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 41/699 objects degraded (5.866%); 72/699 objects misplaced (10.300%); 0 B/s, 0 objects/s recovering 2024-02-23T00:39:47.400 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:46.186679+0000 mon.a (mon.0) 3043 : cluster [WRN] Health check update: Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:47.400 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:46.186697+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:39:47.400 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:46.186708+0000 mon.a (mon.0) 3045 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-23T00:39:47.400 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:47.182292+0000 mon.a (mon.0) 3046 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-97", "overlaypool": "test-rados-api-smithi087-82091-97-cache"}]': finished 2024-02-23T00:39:47.400 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:47.182371+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e568: 8 total, 7 up, 8 in 2024-02-23T00:39:47.400 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:47.185777+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-97-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:39:47.400 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:47.186158+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-97-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:39:47.400 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:45.733927+0000 mgr.y (mgr.24437) 516 : cluster [DBG] pgmap v866: 254 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 58 active+undersized, 158 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 41/699 objects degraded (5.866%); 72/699 objects misplaced (10.300%); 0 B/s, 0 objects/s recovering 2024-02-23T00:39:47.401 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:46.186679+0000 mon.a (mon.0) 3043 : cluster [WRN] Health check update: Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:47.401 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:46.186697+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:39:47.401 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:46.186708+0000 mon.a (mon.0) 3045 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-23T00:39:47.401 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:47.182292+0000 mon.a (mon.0) 3046 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-97", "overlaypool": "test-rados-api-smithi087-82091-97-cache"}]': finished 2024-02-23T00:39:47.401 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:47.182371+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e568: 8 total, 7 up, 8 in 2024-02-23T00:39:47.401 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:47.185777+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-97-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:39:47.401 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:47.186158+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-97-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:39:47.444 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:47.471 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:47.545 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:47.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:47.649 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:47.674 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:45.733927+0000 mgr.y (mgr.24437) 516 : cluster [DBG] pgmap v866: 254 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 58 active+undersized, 158 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 41/699 objects degraded (5.866%); 72/699 objects misplaced (10.300%); 0 B/s, 0 objects/s recovering 2024-02-23T00:39:47.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:46.186679+0000 mon.a (mon.0) 3043 : cluster [WRN] Health check update: Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:47.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:46.186697+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:39:47.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:46.186708+0000 mon.a (mon.0) 3045 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-23T00:39:47.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:47.182292+0000 mon.a (mon.0) 3046 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-97", "overlaypool": "test-rados-api-smithi087-82091-97-cache"}]': finished 2024-02-23T00:39:47.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:47.182371+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e568: 8 total, 7 up, 8 in 2024-02-23T00:39:47.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:47.185777+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-97-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:39:47.675 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:47.186158+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-97-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:39:47.676 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:47.817 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:47.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:47.921 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:47.947 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:48.144 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:48.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:48.211 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:48.179277+0000 mon.a (mon.0) 3049 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:39:48.211 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:48.184189+0000 mon.a (mon.0) 3050 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-97-cache", "mode": "writeback"}]': finished 2024-02-23T00:39:48.211 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:48.184279+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e569: 8 total, 7 up, 8 in 2024-02-23T00:39:48.248 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:48.274 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:48.292 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:48.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:48.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:48.179277+0000 mon.a (mon.0) 3049 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:39:48.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:48.184189+0000 mon.a (mon.0) 3050 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-97-cache", "mode": "writeback"}]': finished 2024-02-23T00:39:48.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:48.184279+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e569: 8 total, 7 up, 8 in 2024-02-23T00:39:48.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:48.179277+0000 mon.a (mon.0) 3049 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:39:48.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:48.184189+0000 mon.a (mon.0) 3050 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-97-cache", "mode": "writeback"}]': finished 2024-02-23T00:39:48.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:48.184279+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e569: 8 total, 7 up, 8 in 2024-02-23T00:39:48.338 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:39:48.396 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:48.424 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:48.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:48.554 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:48.580 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:48.599 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:48.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:48.703 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:48.729 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:48.748 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:48.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:48.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:48.852 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:48.881 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:48.898 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:48.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:49.002 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:49.028 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:49.048 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:49.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:49.152 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:49.180 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:49.524 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:47.734842+0000 mgr.y (mgr.24437) 517 : cluster [DBG] pgmap v869: 254 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 58 active+undersized, 158 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 41/699 objects degraded (5.866%); 72/699 objects misplaced (10.300%); 0 B/s, 0 objects/s recovering 2024-02-23T00:39:49.524 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:48.209231+0000 mon.c (mon.2) 466 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-97"}]: dispatch 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:48.212721+0000 mon.a (mon.0) 3052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-97"}]: dispatch 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:48.580877+0000 mon.a (mon.0) 3053 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:49.185831+0000 mon.a (mon.0) 3054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-97"}]': finished 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:49.185903+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e570: 8 total, 7 up, 8 in 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:49.190864+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:49.191619+0000 mon.a (mon.0) 3056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:47.734842+0000 mgr.y (mgr.24437) 517 : cluster [DBG] pgmap v869: 254 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 58 active+undersized, 158 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 41/699 objects degraded (5.866%); 72/699 objects misplaced (10.300%); 0 B/s, 0 objects/s recovering 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:48.209231+0000 mon.c (mon.2) 466 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-97"}]: dispatch 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:48.212721+0000 mon.a (mon.0) 3052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-97"}]: dispatch 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:48.580877+0000 mon.a (mon.0) 3053 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:49.185831+0000 mon.a (mon.0) 3054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-97"}]': finished 2024-02-23T00:39:49.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:49.185903+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e570: 8 total, 7 up, 8 in 2024-02-23T00:39:49.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:49.190864+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:49.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:49.191619+0000 mon.a (mon.0) 3056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:49.541 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:49.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:49.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:49.672 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:47.734842+0000 mgr.y (mgr.24437) 517 : cluster [DBG] pgmap v869: 254 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 58 active+undersized, 158 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 41/699 objects degraded (5.866%); 72/699 objects misplaced (10.300%); 0 B/s, 0 objects/s recovering 2024-02-23T00:39:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:48.209231+0000 mon.c (mon.2) 466 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-97"}]: dispatch 2024-02-23T00:39:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:48.212721+0000 mon.a (mon.0) 3052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-97"}]: dispatch 2024-02-23T00:39:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:48.580877+0000 mon.a (mon.0) 3053 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:49.185831+0000 mon.a (mon.0) 3054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-97"}]': finished 2024-02-23T00:39:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:49.185903+0000 mon.a (mon.0) 3055 : cluster [DBG] osdmap e570: 8 total, 7 up, 8 in 2024-02-23T00:39:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:49.190864+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.87:0/522664846' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:49.191619+0000 mon.a (mon.0) 3056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]: dispatch 2024-02-23T00:39:49.724 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:49.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:49.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:49.827 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:49.855 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:49.873 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:49.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:49.977 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:50.003 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:50.022 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:50.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:50.121 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:50.125 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:50.151 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:50.170 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:50.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:50.274 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:50.304 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:50.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:50.182970+0000 mon.a (mon.0) 3057 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:39:50.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:50.187682+0000 mon.a (mon.0) 3058 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]': finished 2024-02-23T00:39:50.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:50.187783+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e571: 8 total, 7 up, 8 in 2024-02-23T00:39:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:50.182970+0000 mon.a (mon.0) 3057 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:39:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:50.187682+0000 mon.a (mon.0) 3058 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]': finished 2024-02-23T00:39:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:50.187783+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e571: 8 total, 7 up, 8 in 2024-02-23T00:39:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:50.182970+0000 mon.a (mon.0) 3057 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:39:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:50.187682+0000 mon.a (mon.0) 3058 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-97", "tierpool": "test-rados-api-smithi087-82091-97-cache"}]': finished 2024-02-23T00:39:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:50.187783+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e571: 8 total, 7 up, 8 in 2024-02-23T00:39:50.954 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:50.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:50.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:51.058 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:51.085 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:51.211 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:51.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:51.316 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:51.343 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:51.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:49.735405+0000 mgr.y (mgr.24437) 518 : cluster [DBG] pgmap v872: 253 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 58 active+undersized, 157 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/696 objects degraded (5.891%); 72/696 objects misplaced (10.345%) 2024-02-23T00:39:51.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:51.187691+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e572: 8 total, 7 up, 8 in 2024-02-23T00:39:51.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:39:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:39:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:49.735405+0000 mgr.y (mgr.24437) 518 : cluster [DBG] pgmap v872: 253 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 58 active+undersized, 157 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/696 objects degraded (5.891%); 72/696 objects misplaced (10.345%) 2024-02-23T00:39:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:51.187691+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e572: 8 total, 7 up, 8 in 2024-02-23T00:39:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:49.735405+0000 mgr.y (mgr.24437) 518 : cluster [DBG] pgmap v872: 253 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 2 active+remapped+backfill_toofull, 10 active+undersized+degraded, 58 active+undersized, 157 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/696 objects degraded (5.891%); 72/696 objects misplaced (10.345%) 2024-02-23T00:39:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:51.187691+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e572: 8 total, 7 up, 8 in 2024-02-23T00:39:51.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 -- 172.21.15.87:0/3291283661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 msgr2=0x7fc76c103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:51.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 --2- 172.21.15.87:0/3291283661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 0x7fc76c103740 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 rx=0x7fc764009be0 tx=0x7fc764004d10).stop 2024-02-23T00:39:51.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 -- 172.21.15.87:0/3291283661 shutdown_connections 2024-02-23T00:39:51.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 --2- 172.21.15.87:0/3291283661 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 0x7fc76c103740 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:51.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 --2- 172.21.15.87:0/3291283661 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fc920 0x7fc76c0fcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:39:51.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 --2- 172.21.15.87:0/3291283661 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fd280 0x7fc76c0f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 -- 172.21.15.87:0/3291283661 >> 172.21.15.87:0/3291283661 conn(0x7fc76c0f2500 msgr2=0x7fc76c0f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 -- 172.21.15.87:0/3291283661 shutdown_connections 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 -- 172.21.15.87:0/3291283661 wait complete. 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 librados: starting msgr at 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 librados: starting objecter 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 Processor -- start 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 -- start start 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 librados: setting wanted keys 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.563+0000 7fc7717a2700 1 librados: calling monclient init 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7717a2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 0x7fc76c0f8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7717a2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fc920 0x7fc76c0f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:51.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7717a2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fd280 0x7fc76c0f9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7717a2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c1162a0 con 0x7fc76c0fc920 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7717a2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c116410 con 0x7fc76c0fd280 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7717a2700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c1165d0 con 0x7fc76c0f7120 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 0x7fc76c0f8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fc920 0x7fc76c0f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 0x7fc76c0f8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:38186/0 (socket says 172.21.15.87:38186) 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76bfff700 1 -- 172.21.15.87:0/2228851065 learned_addr learned my addr 172.21.15.87:0/2228851065 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc770fa1700 1 --2- 172.21.15.87:0/2228851065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fd280 0x7fc76c0f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76bfff700 1 -- 172.21.15.87:0/2228851065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fd280 msgr2=0x7fc76c0f9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76bfff700 1 --2- 172.21.15.87:0/2228851065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fd280 0x7fc76c0f9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76bfff700 1 -- 172.21.15.87:0/2228851065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fc920 msgr2=0x7fc76c0f8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:51.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76bfff700 1 --2- 172.21.15.87:0/2228851065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fc920 0x7fc76c0f8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76bfff700 1 -- 172.21.15.87:0/2228851065 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c005040 con 0x7fc76c0f7120 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76b7fe700 1 --2- 172.21.15.87:0/2228851065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fc920 0x7fc76c0f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc76bfff700 1 --2- 172.21.15.87:0/2228851065 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 0x7fc76c0f8670 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7fc76000b530 tx=0x7fc7600090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7697fa700 10 client.?.objecter ms_handle_connect 0x7fc76c0f7120 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7697fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7697fa700 1 -- 172.21.15.87:0/2228851065 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760002320 con 0x7fc76c0f7120 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7640097c0 con 0x7fc76c0f7120 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7697fa700 1 -- 172.21.15.87:0/2228851065 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc760002950 con 0x7fc76c0f7120 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.564+0000 7fc7697fa700 1 -- 172.21.15.87:0/2228851065 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760012950 con 0x7fc76c0f7120 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.565+0000 7fc7717a2700 10 client.38012.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.565+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76c116ce0 con 0x7fc76c0f7120 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.565+0000 7fc7717a2700 1 librados: init done 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.565+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76c061ec0 con 0x7fc76c0f7120 2024-02-23T00:39:51.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.566+0000 7fc7697fa700 1 -- 172.21.15.87:0/2228851065 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc760002480 con 0x7fc76c0f7120 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.566+0000 7fc7697fa700 1 --2- 172.21.15.87:0/2228851065 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc754062c90 0x7fc754065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.566+0000 7fc76b7fe700 1 --2- 172.21.15.87:0/2228851065 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc754062c90 0x7fc754065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.566+0000 7fc7697fa700 1 -- 172.21.15.87:0/2228851065 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(572..572 src has 173..572) v4 ==== 9436+0+0 (secure 0 0 0) 0x7fc76007d790 con 0x7fc76c0f7120 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.566+0000 7fc7697fa700 10 client.38012.objecter ms_dispatch 0x7fc76c001550 osd_map(572..572 src has 173..572) v4 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.566+0000 7fc7697fa700 3 client.38012.objecter handle_osd_map got epochs [572,572] > 0 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.566+0000 7fc7697fa700 3 client.38012.objecter handle_osd_map decoding full epoch 572 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.567+0000 7fc7697fa700 20 client.38012.objecter dump_active .. 0 homeless 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.567+0000 7fc7697fa700 10 client.38012.objecter ms_handle_connect 0x7fc754062c90 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.567+0000 7fc76b7fe700 1 --2- 172.21.15.87:0/2228851065 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc754062c90 0x7fc754065150 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 rx=0x7fc76c0f9e10 tx=0x7fc75c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:51.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.571+0000 7fc7697fa700 1 -- 172.21.15.87:0/2228851065 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc760050700 con 0x7fc76c0f7120 2024-02-23T00:39:51.651 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:51.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:51.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:51.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:51.757+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc76c0f9b50 con 0x7fc76c0f7120 2024-02-23T00:39:51.785 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:52.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.166+0000 7fc7697fa700 1 -- 172.21.15.87:0/2228851065 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v573) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc7600502f0 con 0x7fc76c0f7120 2024-02-23T00:39:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 10 librados: watch_flush enter 2024-02-23T00:39:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 10 librados: watch_flush exit 2024-02-23T00:39:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 20 client.38012.objecter shutdown clearing up homeless session... 2024-02-23T00:39:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 10 client.38012.objecter successfully canceled tick 2024-02-23T00:39:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc754062c90 msgr2=0x7fc754065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 --2- 172.21.15.87:0/2228851065 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc754062c90 0x7fc754065150 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 rx=0x7fc76c0f9e10 tx=0x7fc75c00b040).stop 2024-02-23T00:39:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 msgr2=0x7fc76c0f8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 --2- 172.21.15.87:0/2228851065 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 0x7fc76c0f8670 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 rx=0x7fc76000b530 tx=0x7fc7600090f0).stop 2024-02-23T00:39:52.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 shutdown_connections 2024-02-23T00:39:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 --2- 172.21.15.87:0/2228851065 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc754062c90 0x7fc754065150 secure :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 rx=0x7fc76c0f9e10 tx=0x7fc75c00b040).stop 2024-02-23T00:39:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 --2- 172.21.15.87:0/2228851065 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fd280 0x7fc76c0f9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 --2- 172.21.15.87:0/2228851065 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7120 0x7fc76c0f8670 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 --2- 172.21.15.87:0/2228851065 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fc920 0x7fc76c0f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:52.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.174+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 >> 172.21.15.87:0/2228851065 conn(0x7fc76c0f2500 msgr2=0x7fc76c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:52.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.179+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 shutdown_connections 2024-02-23T00:39:52.182 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.180+0000 7fc7717a2700 1 -- 172.21.15.87:0/2228851065 wait complete. 2024-02-23T00:39:52.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.184+0000 7fc7717a2700 1 librados: shutdown 2024-02-23T00:39:52.198 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:39:52.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:52] "GET /metrics HTTP/1.1" 200 184596 "" "Prometheus/2.33.4" 2024-02-23T00:39:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:50.562431+0000 mgr.y (mgr.24437) 519 : audit [DBG] from='client.27534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:39:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:51.083893+0000 mgr.y (mgr.24437) 520 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:51.759442+0000 mon.c (mon.2) 468 : audit [INF] from='client.? 172.21.15.87:0/2228851065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:51.759702+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:52.157171+0000 mon.a (mon.0) 3062 : cluster [WRN] Health check update: Degraded data redundancy: 41/696 objects degraded (5.891%), 11 pgs degraded, 39 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:52.157774+0000 mon.a (mon.0) 3063 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:52.165447+0000 mon.a (mon.0) 3064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:52.165494+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e573: 8 total, 7 up, 8 in 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:52.216527+0000 mon.a (mon.0) 3066 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:52.220652+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:52.221221+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:50.562431+0000 mgr.y (mgr.24437) 519 : audit [DBG] from='client.27534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:51.083893+0000 mgr.y (mgr.24437) 520 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:51.759442+0000 mon.c (mon.2) 468 : audit [INF] from='client.? 172.21.15.87:0/2228851065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:51.759702+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:52.157171+0000 mon.a (mon.0) 3062 : cluster [WRN] Health check update: Degraded data redundancy: 41/696 objects degraded (5.891%), 11 pgs degraded, 39 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:52.157774+0000 mon.a (mon.0) 3063 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:52.165447+0000 mon.a (mon.0) 3064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:39:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:52.165494+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e573: 8 total, 7 up, 8 in 2024-02-23T00:39:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:52.216527+0000 mon.a (mon.0) 3066 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:39:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:52.220652+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:39:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:52.221221+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:39:52.665 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:52.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:50.562431+0000 mgr.y (mgr.24437) 519 : audit [DBG] from='client.27534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:39:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:51.083893+0000 mgr.y (mgr.24437) 520 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:39:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:51.759442+0000 mon.c (mon.2) 468 : audit [INF] from='client.? 172.21.15.87:0/2228851065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:51.759702+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:39:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:52.157171+0000 mon.a (mon.0) 3062 : cluster [WRN] Health check update: Degraded data redundancy: 41/696 objects degraded (5.891%), 11 pgs degraded, 39 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:52.157774+0000 mon.a (mon.0) 3063 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:39:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:52.165447+0000 mon.a (mon.0) 3064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:39:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:52.165494+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e573: 8 total, 7 up, 8 in 2024-02-23T00:39:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:52.216527+0000 mon.a (mon.0) 3066 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:39:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:52.220652+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:39:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:52.221221+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:39:52.769 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:52.797 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:52.815 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:52.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:52.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.386+0000 7f64c81b3700 1 -- 172.21.15.87:0/4149218097 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f64c00f9610 msgr2=0x7f64c0101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:52.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.386+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4149218097 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f64c00f9610 0x7f64c0101d20 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 rx=0x7f64b0004870 tx=0x7f64b0009cb0).stop 2024-02-23T00:39:52.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.386+0000 7f64c81b3700 1 -- 172.21.15.87:0/4149218097 shutdown_connections 2024-02-23T00:39:52.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.386+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4149218097 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64c0102210 0x7f64c0104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:39:52.838 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.386+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4149218097 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64c00f8cb0 0x7f64c00f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.386+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4149218097 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f64c00f9610 0x7f64c0101d20 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.386+0000 7f64c81b3700 1 -- 172.21.15.87:0/4149218097 >> 172.21.15.87:0/4149218097 conn(0x7f64c00f2820 msgr2=0x7f64c00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.386+0000 7f64c81b3700 1 -- 172.21.15.87:0/4149218097 shutdown_connections 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.386+0000 7f64c81b3700 1 -- 172.21.15.87:0/4149218097 wait complete. 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 librados: starting msgr at 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 librados: starting objecter 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 Processor -- start 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 -- start start 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 librados: setting wanted keys 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 librados: calling monclient init 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64c00f8cb0 0x7f64c010e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64c00f9610 0x7f64c010e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f64c0102210 0x7f64c01084a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c0108990 con 0x7f64c00f8cb0 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c0108b00 con 0x7f64c00f9610 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c81b3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c0108cc0 con 0x7f64c0102210 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c5f4f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64c00f8cb0 0x7f64c010e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c6750700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f64c0102210 0x7f64c01084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c5f4f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64c00f8cb0 0x7f64c010e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58798/0 (socket says 172.21.15.87:58798) 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.387+0000 7f64c5f4f700 1 -- 172.21.15.87:0/4190733767 learned_addr learned my addr 172.21.15.87:0/4190733767 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c5f4f700 1 -- 172.21.15.87:0/4190733767 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f64c0102210 msgr2=0x7f64c01084a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c5f4f700 1 --2- 172.21.15.87:0/4190733767 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f64c0102210 0x7f64c01084a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c5f4f700 1 -- 172.21.15.87:0/4190733767 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64c00f9610 msgr2=0x7f64c010e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:52.841 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:39:52.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c5f4f700 1 --2- 172.21.15.87:0/4190733767 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64c00f9610 0x7f64c010e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:39:52.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c5f4f700 1 -- 172.21.15.87:0/4190733767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0005040 con 0x7f64c00f8cb0 2024-02-23T00:39:52.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c6750700 1 --2- 172.21.15.87:0/4190733767 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f64c0102210 0x7f64c01084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c5f4f700 1 --2- 172.21.15.87:0/4190733767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64c00f8cb0 0x7f64c010e300 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f64bc00e970 tx=0x7f64bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64b6ffd700 10 client.?.objecter ms_handle_connect 0x7f64c00f8cb0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64b6ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64b6ffd700 1 -- 172.21.15.87:0/4190733767 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc00ee60 con 0x7f64c00f8cb0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64b6ffd700 1 -- 172.21.15.87:0/4190733767 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f64bc00f040 con 0x7f64c00f8cb0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c0108fa0 con 0x7f64c00f8cb0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64b6ffd700 1 -- 172.21.15.87:0/4190733767 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc007ea0 con 0x7f64c00f8cb0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c81b3700 10 client.27534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c0109320 con 0x7f64c00f8cb0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.388+0000 7f64c81b3700 1 librados: init done 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.389+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c005fa60 con 0x7f64c00f8cb0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.390+0000 7f64b6ffd700 1 -- 172.21.15.87:0/4190733767 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f64bc00cb10 con 0x7f64c00f8cb0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.390+0000 7f64b6ffd700 1 --2- 172.21.15.87:0/4190733767 >> v2:172.21.15.87:6800/1231597463 conn(0x7f64ac062c90 0x7f64ac065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.390+0000 7f64b6ffd700 1 -- 172.21.15.87:0/4190733767 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(571..571 src has 173..571) v4 ==== 9827+0+0 (secure 0 0 0) 0x7f64bc07d470 con 0x7f64c00f8cb0 2024-02-23T00:39:52.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.390+0000 7f64b6ffd700 10 client.27534.objecter ms_dispatch 0x7f64c0001550 osd_map(571..571 src has 173..571) v4 2024-02-23T00:39:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.390+0000 7f64b6ffd700 3 client.27534.objecter handle_osd_map got epochs [571,571] > 0 2024-02-23T00:39:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.390+0000 7f64b6ffd700 3 client.27534.objecter handle_osd_map decoding full epoch 571 2024-02-23T00:39:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.390+0000 7f64b6ffd700 20 client.27534.objecter dump_active .. 0 homeless 2024-02-23T00:39:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.391+0000 7f64c574e700 1 --2- 172.21.15.87:0/4190733767 >> v2:172.21.15.87:6800/1231597463 conn(0x7f64ac062c90 0x7f64ac065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.391+0000 7f64b6ffd700 10 client.27534.objecter ms_handle_connect 0x7f64ac062c90 2024-02-23T00:39:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.391+0000 7f64c574e700 1 --2- 172.21.15.87:0/4190733767 >> v2:172.21.15.87:6800/1231597463 conn(0x7f64ac062c90 0x7f64ac065150 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f64b0009da0 tx=0x7f64b000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.395+0000 7f64b6ffd700 1 -- 172.21.15.87:0/4190733767 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f64bc04af60 con 0x7f64c00f8cb0 2024-02-23T00:39:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.560+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64c0109980 con 0x7f64ac062c90 2024-02-23T00:39:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:50.577+0000 7f64b6ffd700 1 -- 172.21.15.87:0/4190733767 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+639371 (secure 0 0 0) 0x7f64c0109980 con 0x7f64ac062c90 2024-02-23T00:39:52.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 10 librados: watch_flush enter 2024-02-23T00:39:52.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 10 librados: watch_flush exit 2024-02-23T00:39:52.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 20 client.27534.objecter shutdown clearing up homeless session... 2024-02-23T00:39:52.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 10 client.27534.objecter successfully canceled tick 2024-02-23T00:39:52.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 >> v2:172.21.15.87:6800/1231597463 conn(0x7f64ac062c90 msgr2=0x7f64ac065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4190733767 >> v2:172.21.15.87:6800/1231597463 conn(0x7f64ac062c90 0x7f64ac065150 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f64b0009da0 tx=0x7f64b000b040).stop 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64c00f8cb0 msgr2=0x7f64c010e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4190733767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64c00f8cb0 0x7f64c010e300 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 rx=0x7f64bc00e970 tx=0x7f64bc00b040).stop 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 shutdown_connections 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4190733767 >> v2:172.21.15.87:6800/1231597463 conn(0x7f64ac062c90 0x7f64ac065150 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4190733767 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f64c0102210 0x7f64c01084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4190733767 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f64c00f8cb0 0x7f64c010e300 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 --2- 172.21.15.87:0/4190733767 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f64c00f9610 0x7f64c010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 >> 172.21.15.87:0/4190733767 conn(0x7f64c00f2820 msgr2=0x7f64c00f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 shutdown_connections 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 -- 172.21.15.87:0/4190733767 wait complete. 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:52.840+0000 7f64c81b3700 1 librados: shutdown 2024-02-23T00:39:52.858 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:39:52.919 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:52.945 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:52.964 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:52.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:53.068 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:53.097 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:53.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:51.735987+0000 mgr.y (mgr.24437) 521 : cluster [DBG] pgmap v875: 221 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 11 active+undersized+degraded, 44 active+undersized, 138 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 42/702 objects degraded (5.983%); 50/702 objects misplaced (7.123%); 27 B/s, 2 objects/s recovering 2024-02-23T00:39:53.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:52.053908+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:53.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:52.054031+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:53.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:52.241955+0000 mon.a (mon.0) 3069 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:53.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:53.179192+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:39:53.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:53.179266+0000 mon.a (mon.0) 3071 : cluster [DBG] osdmap e574: 8 total, 7 up, 8 in 2024-02-23T00:39:53.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:53.180381+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:39:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:51.735987+0000 mgr.y (mgr.24437) 521 : cluster [DBG] pgmap v875: 221 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 11 active+undersized+degraded, 44 active+undersized, 138 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 42/702 objects degraded (5.983%); 50/702 objects misplaced (7.123%); 27 B/s, 2 objects/s recovering 2024-02-23T00:39:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:52.053908+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:52.054031+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:52.241955+0000 mon.a (mon.0) 3069 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:53.179192+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:39:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:53.179266+0000 mon.a (mon.0) 3071 : cluster [DBG] osdmap e574: 8 total, 7 up, 8 in 2024-02-23T00:39:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:53.180381+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:39:53.346 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:53.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:53.452 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:53.479 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:51.735987+0000 mgr.y (mgr.24437) 521 : cluster [DBG] pgmap v875: 221 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 11 active+undersized+degraded, 44 active+undersized, 138 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 42/702 objects degraded (5.983%); 50/702 objects misplaced (7.123%); 27 B/s, 2 objects/s recovering 2024-02-23T00:39:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:52.053908+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:52.054031+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:52.241955+0000 mon.a (mon.0) 3069 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:39:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:53.179192+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:39:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:53.179266+0000 mon.a (mon.0) 3071 : cluster [DBG] osdmap e574: 8 total, 7 up, 8 in 2024-02-23T00:39:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:53.180381+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:39:53.890 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":873,"stamp":"2024-02-23T00:39:50.185520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5746,"num_objects":232,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41,"num_objects_misplaced":72,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":1726,"num_read_kb":1488,"num_write":780,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":5448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":867,"ondisk_log_size":867,"up":669,"acting":694,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":650,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2499184,"kb_used_data":51704,"kb_used_omap":0,"kb_used_meta":2447360,"kb_avail":747462032,"statfs":{"total":767960285184,"available":765401120768,"internally_reserved":0,"allocated":52944896,"data_stored":29926601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2506096640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.007557"},"pg_stats":[{"pgid":"231.1e","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.193784+0000","last_change":"2024-02-23T00:39:45.193784+0000","last_active":"2024-02-23T00:39:45.193784+0000","last_peered":"2024-02-23T00:39:45.193784+0000","last_clean":"2024-02-23T00:39:45.193784+0000","last_became_active":"2024-02-23T00:39:45.193490+0000","last_became_peered":"2024-02-23T00:39:45.193490+0000","last_unstale":"2024-02-23T00:39:45.193784+0000","last_undegraded":"2024-02-23T00:39:45.193784+0000","last_fullsized":"2024-02-23T00:39:45.193784+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"230.1f","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.188557+0000","last_change":"2024-02-23T00:39:43.172202+0000","last_active":"2024-02-23T00:39:44.188557+0000","last_peered":"2024-02-23T00:39:44.188557+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.172202+0000","last_became_peered":"2024-02-23T00:39:43.172202+0000","last_unstale":"2024-02-23T00:39:44.188557+0000","last_undegraded":"2024-02-23T00:39:44.188557+0000","last_fullsized":"2024-02-23T00:39:43.160286+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"231.1f","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.193771+0000","last_change":"2024-02-23T00:39:45.193710+0000","last_active":"2024-02-23T00:39:45.193771+0000","last_peered":"2024-02-23T00:39:45.193771+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.193710+0000","last_became_peered":"2024-02-23T00:39:45.193710+0000","last_unstale":"2024-02-23T00:39:45.193771+0000","last_undegraded":"2024-02-23T00:39:45.193771+0000","last_fullsized":"2024-02-23T00:39:45.184788+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"230.1e","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.184832+0000","last_change":"2024-02-23T00:39:43.171386+0000","last_active":"2024-02-23T00:39:44.184832+0000","last_peered":"2024-02-23T00:39:44.184832+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.171386+0000","last_became_peered":"2024-02-23T00:39:43.171386+0000","last_unstale":"2024-02-23T00:39:44.184832+0000","last_undegraded":"2024-02-23T00:39:44.184832+0000","last_fullsized":"2024-02-23T00:39:43.167314+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"231.1c","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.189425+0000","last_change":"2024-02-23T00:39:45.189353+0000","last_active":"2024-02-23T00:39:45.189425+0000","last_peered":"2024-02-23T00:39:45.189425+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.189353+0000","last_became_peered":"2024-02-23T00:39:45.189353+0000","last_unstale":"2024-02-23T00:39:45.189425+0000","last_undegraded":"2024-02-23T00:39:45.189425+0000","last_fullsized":"2024-02-23T00:39:45.178912+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"230.1d","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179096+0000","last_change":"2024-02-23T00:39:43.183117+0000","last_active":"2024-02-23T00:39:44.179096+0000","last_peered":"2024-02-23T00:39:44.179096+0000","last_clean":"2024-02-23T00:39:44.179096+0000","last_became_active":"2024-02-23T00:39:43.182760+0000","last_became_peered":"2024-02-23T00:39:43.182760+0000","last_unstale":"2024-02-23T00:39:44.179096+0000","last_undegraded":"2024-02-23T00:39:44.179096+0000","last_fullsized":"2024-02-23T00:39:44.179096+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.1d","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.207537+0000","last_change":"2024-02-23T00:39:45.207537+0000","last_active":"2024-02-23T00:39:45.207537+0000","last_peered":"2024-02-23T00:39:45.207537+0000","last_clean":"2024-02-23T00:39:45.207537+0000","last_became_active":"2024-02-23T00:39:45.207227+0000","last_became_peered":"2024-02-23T00:39:45.207227+0000","last_unstale":"2024-02-23T00:39:45.207537+0000","last_undegraded":"2024-02-23T00:39:45.207537+0000","last_fullsized":"2024-02-23T00:39:45.207537+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"230.1c","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181605+0000","last_change":"2024-02-23T00:39:43.176513+0000","last_active":"2024-02-23T00:39:44.181605+0000","last_peered":"2024-02-23T00:39:44.181605+0000","last_clean":"2024-02-23T00:39:44.181605+0000","last_became_active":"2024-02-23T00:39:43.176190+0000","last_became_peered":"2024-02-23T00:39:43.176190+0000","last_unstale":"2024-02-23T00:39:44.181605+0000","last_undegraded":"2024-02-23T00:39:44.181605+0000","last_fullsized":"2024-02-23T00:39:44.181605+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"231.1a","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.202249+0000","last_change":"2024-02-23T00:39:45.202249+0000","last_active":"2024-02-23T00:39:45.202249+0000","last_peered":"2024-02-23T00:39:45.202249+0000","last_clean":"2024-02-23T00:39:45.202249+0000","last_became_active":"2024-02-23T00:39:45.201958+0000","last_became_peered":"2024-02-23T00:39:45.201958+0000","last_unstale":"2024-02-23T00:39:45.202249+0000","last_undegraded":"2024-02-23T00:39:45.202249+0000","last_fullsized":"2024-02-23T00:39:45.202249+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.1b","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179908+0000","last_change":"2024-02-23T00:39:43.173468+0000","last_active":"2024-02-23T00:39:44.179908+0000","last_peered":"2024-02-23T00:39:44.179908+0000","last_clean":"2024-02-23T00:39:44.179908+0000","last_became_active":"2024-02-23T00:39:43.173221+0000","last_became_peered":"2024-02-23T00:39:43.173221+0000","last_unstale":"2024-02-23T00:39:44.179908+0000","last_undegraded":"2024-02-23T00:39:44.179908+0000","last_fullsized":"2024-02-23T00:39:44.179908+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.1b","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.207570+0000","last_change":"2024-02-23T00:39:45.207570+0000","last_active":"2024-02-23T00:39:45.207570+0000","last_peered":"2024-02-23T00:39:45.207570+0000","last_clean":"2024-02-23T00:39:45.207570+0000","last_became_active":"2024-02-23T00:39:45.207242+0000","last_became_peered":"2024-02-23T00:39:45.207242+0000","last_unstale":"2024-02-23T00:39:45.207570+0000","last_undegraded":"2024-02-23T00:39:45.207570+0000","last_fullsized":"2024-02-23T00:39:45.207570+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"230.1a","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179142+0000","last_change":"2024-02-23T00:39:43.171160+0000","last_active":"2024-02-23T00:39:44.179142+0000","last_peered":"2024-02-23T00:39:44.179142+0000","last_clean":"2024-02-23T00:39:44.179142+0000","last_became_active":"2024-02-23T00:39:43.170833+0000","last_became_peered":"2024-02-23T00:39:43.170833+0000","last_unstale":"2024-02-23T00:39:44.179142+0000","last_undegraded":"2024-02-23T00:39:44.179142+0000","last_fullsized":"2024-02-23T00:39:44.179142+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"231.18","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.200110+0000","last_change":"2024-02-23T00:39:45.200024+0000","last_active":"2024-02-23T00:39:45.200110+0000","last_peered":"2024-02-23T00:39:45.200110+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.200024+0000","last_became_peered":"2024-02-23T00:39:45.200024+0000","last_unstale":"2024-02-23T00:39:45.200110+0000","last_undegraded":"2024-02-23T00:39:45.200110+0000","last_fullsized":"2024-02-23T00:39:45.191598+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"230.19","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.188409+0000","last_change":"2024-02-23T00:39:43.183234+0000","last_active":"2024-02-23T00:39:44.188409+0000","last_peered":"2024-02-23T00:39:44.188409+0000","last_clean":"2024-02-23T00:39:44.188409+0000","last_became_active":"2024-02-23T00:39:43.182954+0000","last_became_peered":"2024-02-23T00:39:43.182954+0000","last_unstale":"2024-02-23T00:39:44.188409+0000","last_undegraded":"2024-02-23T00:39:44.188409+0000","last_fullsized":"2024-02-23T00:39:44.188409+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"231.19","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.205467+0000","last_change":"2024-02-23T00:39:45.205402+0000","last_active":"2024-02-23T00:39:45.205467+0000","last_peered":"2024-02-23T00:39:45.205467+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.205402+0000","last_became_peered":"2024-02-23T00:39:45.205402+0000","last_unstale":"2024-02-23T00:39:45.205467+0000","last_undegraded":"2024-02-23T00:39:45.205467+0000","last_fullsized":"2024-02-23T00:39:45.196770+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"230.18","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.176902+0000","last_change":"2024-02-23T00:39:43.173236+0000","last_active":"2024-02-23T00:39:44.176902+0000","last_peered":"2024-02-23T00:39:44.176902+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.173236+0000","last_became_peered":"2024-02-23T00:39:43.173236+0000","last_unstale":"2024-02-23T00:39:44.176902+0000","last_undegraded":"2024-02-23T00:39:44.176902+0000","last_fullsized":"2024-02-23T00:39:43.159907+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.16","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.205952+0000","last_change":"2024-02-23T00:39:45.205881+0000","last_active":"2024-02-23T00:39:45.205952+0000","last_peered":"2024-02-23T00:39:45.205952+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.205881+0000","last_became_peered":"2024-02-23T00:39:45.205881+0000","last_unstale":"2024-02-23T00:39:45.205952+0000","last_undegraded":"2024-02-23T00:39:45.205952+0000","last_fullsized":"2024-02-23T00:39:45.194958+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"230.17","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.207134+0000","last_change":"2024-02-23T00:39:43.175725+0000","last_active":"2024-02-23T00:39:44.207134+0000","last_peered":"2024-02-23T00:39:44.207134+0000","last_clean":"2024-02-23T00:39:44.207134+0000","last_became_active":"2024-02-23T00:39:43.175424+0000","last_became_peered":"2024-02-23T00:39:43.175424+0000","last_unstale":"2024-02-23T00:39:44.207134+0000","last_undegraded":"2024-02-23T00:39:44.207134+0000","last_fullsized":"2024-02-23T00:39:44.207134+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"231.17","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.199857+0000","last_change":"2024-02-23T00:39:45.199857+0000","last_active":"2024-02-23T00:39:45.199857+0000","last_peered":"2024-02-23T00:39:45.199857+0000","last_clean":"2024-02-23T00:39:45.199857+0000","last_became_active":"2024-02-23T00:39:45.197848+0000","last_became_peered":"2024-02-23T00:39:45.197848+0000","last_unstale":"2024-02-23T00:39:45.199857+0000","last_undegraded":"2024-02-23T00:39:45.199857+0000","last_fullsized":"2024-02-23T00:39:45.199857+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.16","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178619+0000","last_change":"2024-02-23T00:39:43.177607+0000","last_active":"2024-02-23T00:39:44.178619+0000","last_peered":"2024-02-23T00:39:44.178619+0000","last_clean":"2024-02-23T00:39:44.178619+0000","last_became_active":"2024-02-23T00:39:43.177206+0000","last_became_peered":"2024-02-23T00:39:43.177206+0000","last_unstale":"2024-02-23T00:39:44.178619+0000","last_undegraded":"2024-02-23T00:39:44.178619+0000","last_fullsized":"2024-02-23T00:39:44.178619+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.14","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.196925+0000","last_change":"2024-02-23T00:39:45.196854+0000","last_active":"2024-02-23T00:39:45.196925+0000","last_peered":"2024-02-23T00:39:45.196925+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.196854+0000","last_became_peered":"2024-02-23T00:39:45.196854+0000","last_unstale":"2024-02-23T00:39:45.196925+0000","last_undegraded":"2024-02-23T00:39:45.196925+0000","last_fullsized":"2024-02-23T00:39:45.179043+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"230.15","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.179191+0000","last_change":"2024-02-23T00:39:43.174956+0000","last_active":"2024-02-23T00:39:44.179191+0000","last_peered":"2024-02-23T00:39:44.179191+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.174956+0000","last_became_peered":"2024-02-23T00:39:43.174956+0000","last_unstale":"2024-02-23T00:39:44.179191+0000","last_undegraded":"2024-02-23T00:39:44.179191+0000","last_fullsized":"2024-02-23T00:39:43.164172+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.15","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.189561+0000","last_change":"2024-02-23T00:39:45.189496+0000","last_active":"2024-02-23T00:39:45.189561+0000","last_peered":"2024-02-23T00:39:45.189561+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.189496+0000","last_became_peered":"2024-02-23T00:39:45.189496+0000","last_unstale":"2024-02-23T00:39:45.189561+0000","last_undegraded":"2024-02-23T00:39:45.189561+0000","last_fullsized":"2024-02-23T00:39:45.185534+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"230.14","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.180229+0000","last_change":"2024-02-23T00:39:43.177301+0000","last_active":"2024-02-23T00:39:44.180229+0000","last_peered":"2024-02-23T00:39:44.180229+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.177301+0000","last_became_peered":"2024-02-23T00:39:43.177301+0000","last_unstale":"2024-02-23T00:39:44.180229+0000","last_undegraded":"2024-02-23T00:39:44.180229+0000","last_fullsized":"2024-02-23T00:39:43.170583+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.12","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.192333+0000","last_change":"2024-02-23T00:39:45.192333+0000","last_active":"2024-02-23T00:39:45.192333+0000","last_peered":"2024-02-23T00:39:45.192333+0000","last_clean":"2024-02-23T00:39:45.192333+0000","last_became_active":"2024-02-23T00:39:45.192057+0000","last_became_peered":"2024-02-23T00:39:45.192057+0000","last_unstale":"2024-02-23T00:39:45.192333+0000","last_undegraded":"2024-02-23T00:39:45.192333+0000","last_fullsized":"2024-02-23T00:39:45.192333+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"230.13","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.185104+0000","last_change":"2024-02-23T00:39:43.177937+0000","last_active":"2024-02-23T00:39:44.185104+0000","last_peered":"2024-02-23T00:39:44.185104+0000","last_clean":"2024-02-23T00:39:44.185104+0000","last_became_active":"2024-02-23T00:39:43.177597+0000","last_became_peered":"2024-02-23T00:39:43.177597+0000","last_unstale":"2024-02-23T00:39:44.185104+0000","last_undegraded":"2024-02-23T00:39:44.185104+0000","last_fullsized":"2024-02-23T00:39:44.185104+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.13","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.197185+0000","last_change":"2024-02-23T00:39:45.197185+0000","last_active":"2024-02-23T00:39:45.197185+0000","last_peered":"2024-02-23T00:39:45.197185+0000","last_clean":"2024-02-23T00:39:45.197185+0000","last_became_active":"2024-02-23T00:39:45.196898+0000","last_became_peered":"2024-02-23T00:39:45.196898+0000","last_unstale":"2024-02-23T00:39:45.197185+0000","last_undegraded":"2024-02-23T00:39:45.197185+0000","last_fullsized":"2024-02-23T00:39:45.197185+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"230.12","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.178788+0000","last_change":"2024-02-23T00:39:43.173752+0000","last_active":"2024-02-23T00:39:44.178788+0000","last_peered":"2024-02-23T00:39:44.178788+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.173752+0000","last_became_peered":"2024-02-23T00:39:43.173752+0000","last_unstale":"2024-02-23T00:39:44.178788+0000","last_undegraded":"2024-02-23T00:39:44.178788+0000","last_fullsized":"2024-02-23T00:39:43.161836+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.10","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.190385+0000","last_change":"2024-02-23T00:39:45.190385+0000","last_active":"2024-02-23T00:39:45.190385+0000","last_peered":"2024-02-23T00:39:45.190385+0000","last_clean":"2024-02-23T00:39:45.190385+0000","last_became_active":"2024-02-23T00:39:45.190099+0000","last_became_peered":"2024-02-23T00:39:45.190099+0000","last_unstale":"2024-02-23T00:39:45.190385+0000","last_undegraded":"2024-02-23T00:39:45.190385+0000","last_fullsized":"2024-02-23T00:39:45.190385+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"230.11","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.186548+0000","last_change":"2024-02-23T00:39:43.185516+0000","last_active":"2024-02-23T00:39:44.186548+0000","last_peered":"2024-02-23T00:39:44.186548+0000","last_clean":"2024-02-23T00:39:44.186548+0000","last_became_active":"2024-02-23T00:39:43.184941+0000","last_became_peered":"2024-02-23T00:39:43.184941+0000","last_unstale":"2024-02-23T00:39:44.186548+0000","last_undegraded":"2024-02-23T00:39:44.186548+0000","last_fullsized":"2024-02-23T00:39:44.186548+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"231.11","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.193320+0000","last_change":"2024-02-23T00:39:45.193240+0000","last_active":"2024-02-23T00:39:45.193320+0000","last_peered":"2024-02-23T00:39:45.193320+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.193240+0000","last_became_peered":"2024-02-23T00:39:45.193240+0000","last_unstale":"2024-02-23T00:39:45.193320+0000","last_undegraded":"2024-02-23T00:39:45.193320+0000","last_fullsized":"2024-02-23T00:39:45.187215+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.10","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.204498+0000","last_change":"2024-02-23T00:39:43.179023+0000","last_active":"2024-02-23T00:39:44.204498+0000","last_peered":"2024-02-23T00:39:44.204498+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.179023+0000","last_became_peered":"2024-02-23T00:39:43.179023+0000","last_unstale":"2024-02-23T00:39:44.204498+0000","last_undegraded":"2024-02-23T00:39:44.204498+0000","last_fullsized":"2024-02-23T00:39:43.166692+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"231.e","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.200479+0000","last_change":"2024-02-23T00:39:45.200479+0000","last_active":"2024-02-23T00:39:45.200479+0000","last_peered":"2024-02-23T00:39:45.200479+0000","last_clean":"2024-02-23T00:39:45.200479+0000","last_became_active":"2024-02-23T00:39:45.200257+0000","last_became_peered":"2024-02-23T00:39:45.200257+0000","last_unstale":"2024-02-23T00:39:45.200479+0000","last_undegraded":"2024-02-23T00:39:45.200479+0000","last_fullsized":"2024-02-23T00:39:45.200479+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.f","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.188685+0000","last_change":"2024-02-23T00:39:43.167861+0000","last_active":"2024-02-23T00:39:44.188685+0000","last_peered":"2024-02-23T00:39:44.188685+0000","last_clean":"2024-02-23T00:39:44.188685+0000","last_became_active":"2024-02-23T00:39:43.167540+0000","last_became_peered":"2024-02-23T00:39:43.167540+0000","last_unstale":"2024-02-23T00:39:44.188685+0000","last_undegraded":"2024-02-23T00:39:44.188685+0000","last_fullsized":"2024-02-23T00:39:44.188685+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"231.f","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.205265+0000","last_change":"2024-02-23T00:39:45.205265+0000","last_active":"2024-02-23T00:39:45.205265+0000","last_peered":"2024-02-23T00:39:45.205265+0000","last_clean":"2024-02-23T00:39:45.205265+0000","last_became_active":"2024-02-23T00:39:45.205049+0000","last_became_peered":"2024-02-23T00:39:45.205049+0000","last_unstale":"2024-02-23T00:39:45.205265+0000","last_undegraded":"2024-02-23T00:39:45.205265+0000","last_fullsized":"2024-02-23T00:39:45.205265+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"230.e","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.191914+0000","last_change":"2024-02-23T00:39:43.163875+0000","last_active":"2024-02-23T00:39:44.191914+0000","last_peered":"2024-02-23T00:39:44.191914+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.163875+0000","last_became_peered":"2024-02-23T00:39:43.163875+0000","last_unstale":"2024-02-23T00:39:44.191914+0000","last_undegraded":"2024-02-23T00:39:44.191914+0000","last_fullsized":"2024-02-23T00:39:43.159918+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"231.c","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.200266+0000","last_change":"2024-02-23T00:39:45.200191+0000","last_active":"2024-02-23T00:39:45.200266+0000","last_peered":"2024-02-23T00:39:45.200266+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.200191+0000","last_became_peered":"2024-02-23T00:39:45.200191+0000","last_unstale":"2024-02-23T00:39:45.200266+0000","last_undegraded":"2024-02-23T00:39:45.200266+0000","last_fullsized":"2024-02-23T00:39:45.190967+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"230.d","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.180786+0000","last_change":"2024-02-23T00:39:43.171430+0000","last_active":"2024-02-23T00:39:44.180786+0000","last_peered":"2024-02-23T00:39:44.180786+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.171430+0000","last_became_peered":"2024-02-23T00:39:43.171430+0000","last_unstale":"2024-02-23T00:39:44.180786+0000","last_undegraded":"2024-02-23T00:39:44.180786+0000","last_fullsized":"2024-02-23T00:39:43.160829+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"231.d","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.202352+0000","last_change":"2024-02-23T00:39:45.202352+0000","last_active":"2024-02-23T00:39:45.202352+0000","last_peered":"2024-02-23T00:39:45.202352+0000","last_clean":"2024-02-23T00:39:45.202352+0000","last_became_active":"2024-02-23T00:39:45.202062+0000","last_became_peered":"2024-02-23T00:39:45.202062+0000","last_unstale":"2024-02-23T00:39:45.202352+0000","last_undegraded":"2024-02-23T00:39:45.202352+0000","last_fullsized":"2024-02-23T00:39:45.202352+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.c","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.197442+0000","last_change":"2024-02-23T00:39:43.179468+0000","last_active":"2024-02-23T00:39:44.197442+0000","last_peered":"2024-02-23T00:39:44.197442+0000","last_clean":"2024-02-23T00:39:44.197442+0000","last_became_active":"2024-02-23T00:39:43.179187+0000","last_became_peered":"2024-02-23T00:39:43.179187+0000","last_unstale":"2024-02-23T00:39:44.197442+0000","last_undegraded":"2024-02-23T00:39:44.197442+0000","last_fullsized":"2024-02-23T00:39:44.197442+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"231.a","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.198906+0000","last_change":"2024-02-23T00:39:45.198849+0000","last_active":"2024-02-23T00:39:45.198906+0000","last_peered":"2024-02-23T00:39:45.198906+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.198849+0000","last_became_peered":"2024-02-23T00:39:45.198849+0000","last_unstale":"2024-02-23T00:39:45.198906+0000","last_undegraded":"2024-02-23T00:39:45.198906+0000","last_fullsized":"2024-02-23T00:39:45.183829+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"230.b","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.195858+0000","last_change":"2024-02-23T00:39:43.178184+0000","last_active":"2024-02-23T00:39:44.195858+0000","last_peered":"2024-02-23T00:39:44.195858+0000","last_clean":"2024-02-23T00:39:44.195858+0000","last_became_active":"2024-02-23T00:39:43.177887+0000","last_became_peered":"2024-02-23T00:39:43.177887+0000","last_unstale":"2024-02-23T00:39:44.195858+0000","last_undegraded":"2024-02-23T00:39:44.195858+0000","last_fullsized":"2024-02-23T00:39:44.195858+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"231.b","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.202632+0000","last_change":"2024-02-23T00:39:45.202632+0000","last_active":"2024-02-23T00:39:45.202632+0000","last_peered":"2024-02-23T00:39:45.202632+0000","last_clean":"2024-02-23T00:39:45.202632+0000","last_became_active":"2024-02-23T00:39:45.202325+0000","last_became_peered":"2024-02-23T00:39:45.202325+0000","last_unstale":"2024-02-23T00:39:45.202632+0000","last_undegraded":"2024-02-23T00:39:45.202632+0000","last_fullsized":"2024-02-23T00:39:45.202632+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"230.a","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179667+0000","last_change":"2024-02-23T00:39:43.179625+0000","last_active":"2024-02-23T00:39:44.179667+0000","last_peered":"2024-02-23T00:39:44.179667+0000","last_clean":"2024-02-23T00:39:44.179667+0000","last_became_active":"2024-02-23T00:39:43.179324+0000","last_became_peered":"2024-02-23T00:39:43.179324+0000","last_unstale":"2024-02-23T00:39:44.179667+0000","last_undegraded":"2024-02-23T00:39:44.179667+0000","last_fullsized":"2024-02-23T00:39:44.179667+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"231.8","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.198497+0000","last_change":"2024-02-23T00:39:45.198422+0000","last_active":"2024-02-23T00:39:45.198497+0000","last_peered":"2024-02-23T00:39:45.198497+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.198422+0000","last_became_peered":"2024-02-23T00:39:45.198422+0000","last_unstale":"2024-02-23T00:39:45.198497+0000","last_undegraded":"2024-02-23T00:39:45.198497+0000","last_fullsized":"2024-02-23T00:39:45.191288+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"230.9","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.186132+0000","last_change":"2024-02-23T00:39:43.185712+0000","last_active":"2024-02-23T00:39:44.186132+0000","last_peered":"2024-02-23T00:39:44.186132+0000","last_clean":"2024-02-23T00:39:44.186132+0000","last_became_active":"2024-02-23T00:39:43.185300+0000","last_became_peered":"2024-02-23T00:39:43.185300+0000","last_unstale":"2024-02-23T00:39:44.186132+0000","last_undegraded":"2024-02-23T00:39:44.186132+0000","last_fullsized":"2024-02-23T00:39:44.186132+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"231.9","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.193264+0000","last_change":"2024-02-23T00:39:45.193264+0000","last_active":"2024-02-23T00:39:45.193264+0000","last_peered":"2024-02-23T00:39:45.193264+0000","last_clean":"2024-02-23T00:39:45.193264+0000","last_became_active":"2024-02-23T00:39:45.192983+0000","last_became_peered":"2024-02-23T00:39:45.192983+0000","last_unstale":"2024-02-23T00:39:45.193264+0000","last_undegraded":"2024-02-23T00:39:45.193264+0000","last_fullsized":"2024-02-23T00:39:45.193264+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"230.8","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179133+0000","last_change":"2024-02-23T00:39:43.184273+0000","last_active":"2024-02-23T00:39:44.179133+0000","last_peered":"2024-02-23T00:39:44.179133+0000","last_clean":"2024-02-23T00:39:44.179133+0000","last_became_active":"2024-02-23T00:39:43.183964+0000","last_became_peered":"2024-02-23T00:39:43.183964+0000","last_unstale":"2024-02-23T00:39:44.179133+0000","last_undegraded":"2024-02-23T00:39:44.179133+0000","last_fullsized":"2024-02-23T00:39:44.179133+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"231.5","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.206207+0000","last_change":"2024-02-23T00:39:45.206207+0000","last_active":"2024-02-23T00:39:45.206207+0000","last_peered":"2024-02-23T00:39:45.206207+0000","last_clean":"2024-02-23T00:39:45.206207+0000","last_became_active":"2024-02-23T00:39:45.205949+0000","last_became_peered":"2024-02-23T00:39:45.205949+0000","last_unstale":"2024-02-23T00:39:45.206207+0000","last_undegraded":"2024-02-23T00:39:45.206207+0000","last_fullsized":"2024-02-23T00:39:45.206207+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"230.4","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.180366+0000","last_change":"2024-02-23T00:39:43.183577+0000","last_active":"2024-02-23T00:39:44.180366+0000","last_peered":"2024-02-23T00:39:44.180366+0000","last_clean":"2024-02-23T00:39:44.180366+0000","last_became_active":"2024-02-23T00:39:43.183082+0000","last_became_peered":"2024-02-23T00:39:43.183082+0000","last_unstale":"2024-02-23T00:39:44.180366+0000","last_undegraded":"2024-02-23T00:39:44.180366+0000","last_fullsized":"2024-02-23T00:39:44.180366+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"229.7","version":"0'0","reported_seq":10,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.179630+0000","last_change":"2024-02-23T00:39:41.188513+0000","last_active":"2024-02-23T00:39:44.179630+0000","last_peered":"2024-02-23T00:39:44.179630+0000","last_clean":"2024-02-23T00:39:40.168986+0000","last_became_active":"2024-02-23T00:39:41.188513+0000","last_became_peered":"2024-02-23T00:39:41.188513+0000","last_unstale":"2024-02-23T00:39:44.179630+0000","last_undegraded":"2024-02-23T00:39:44.179630+0000","last_fullsized":"2024-02-23T00:39:41.184390+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":561,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.168986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2147483647,5],"acting":[1,2147483647,5],"avail_no_missing":["1(0)","5(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.4","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.193104+0000","last_change":"2024-02-23T00:39:45.193104+0000","last_active":"2024-02-23T00:39:45.193104+0000","last_peered":"2024-02-23T00:39:45.193104+0000","last_clean":"2024-02-23T00:39:45.193104+0000","last_became_active":"2024-02-23T00:39:45.192824+0000","last_became_peered":"2024-02-23T00:39:45.192824+0000","last_unstale":"2024-02-23T00:39:45.193104+0000","last_undegraded":"2024-02-23T00:39:45.193104+0000","last_fullsized":"2024-02-23T00:39:45.193104+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"230.5","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179306+0000","last_change":"2024-02-23T00:39:43.179492+0000","last_active":"2024-02-23T00:39:44.179306+0000","last_peered":"2024-02-23T00:39:44.179306+0000","last_clean":"2024-02-23T00:39:44.179306+0000","last_became_active":"2024-02-23T00:39:43.179182+0000","last_became_peered":"2024-02-23T00:39:43.179182+0000","last_unstale":"2024-02-23T00:39:44.179306+0000","last_undegraded":"2024-02-23T00:39:44.179306+0000","last_fullsized":"2024-02-23T00:39:44.179306+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.6","version":"0'0","reported_seq":10,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.206473+0000","last_change":"2024-02-23T00:39:41.182181+0000","last_active":"2024-02-23T00:39:44.206473+0000","last_peered":"2024-02-23T00:39:44.206473+0000","last_clean":"2024-02-23T00:39:40.168986+0000","last_became_active":"2024-02-23T00:39:41.182181+0000","last_became_peered":"2024-02-23T00:39:41.182181+0000","last_unstale":"2024-02-23T00:39:44.206473+0000","last_undegraded":"2024-02-23T00:39:44.206473+0000","last_fullsized":"2024-02-23T00:39:41.178236+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":561,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.168986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,5],"acting":[2147483647,3,5],"avail_no_missing":["3(1)","5(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"231.7","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.191025+0000","last_change":"2024-02-23T00:39:45.191025+0000","last_active":"2024-02-23T00:39:45.191025+0000","last_peered":"2024-02-23T00:39:45.191025+0000","last_clean":"2024-02-23T00:39:45.191025+0000","last_became_active":"2024-02-23T00:39:45.190753+0000","last_became_peered":"2024-02-23T00:39:45.190753+0000","last_unstale":"2024-02-23T00:39:45.191025+0000","last_undegraded":"2024-02-23T00:39:45.191025+0000","last_fullsized":"2024-02-23T00:39:45.191025+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"230.6","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.178718+0000","last_change":"2024-02-23T00:39:43.166429+0000","last_active":"2024-02-23T00:39:44.178718+0000","last_peered":"2024-02-23T00:39:44.178718+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.166429+0000","last_became_peered":"2024-02-23T00:39:43.166429+0000","last_unstale":"2024-02-23T00:39:44.178718+0000","last_undegraded":"2024-02-23T00:39:44.178718+0000","last_fullsized":"2024-02-23T00:39:43.163849+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.5","version":"0'0","reported_seq":10,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.184266+0000","last_change":"2024-02-23T00:39:41.182497+0000","last_active":"2024-02-23T00:39:44.184266+0000","last_peered":"2024-02-23T00:39:44.184266+0000","last_clean":"2024-02-23T00:39:40.168986+0000","last_became_active":"2024-02-23T00:39:41.182497+0000","last_became_peered":"2024-02-23T00:39:41.182497+0000","last_unstale":"2024-02-23T00:39:44.184266+0000","last_undegraded":"2024-02-23T00:39:44.184266+0000","last_fullsized":"2024-02-23T00:39:41.177054+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":561,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.168986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,2],"acting":[5,2147483647,2],"avail_no_missing":["5(0)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"231.6","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.203084+0000","last_change":"2024-02-23T00:39:45.203017+0000","last_active":"2024-02-23T00:39:45.203084+0000","last_peered":"2024-02-23T00:39:45.203084+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.203017+0000","last_became_peered":"2024-02-23T00:39:45.203017+0000","last_unstale":"2024-02-23T00:39:45.203084+0000","last_undegraded":"2024-02-23T00:39:45.203084+0000","last_fullsized":"2024-02-23T00:39:45.187871+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"230.7","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.206690+0000","last_change":"2024-02-23T00:39:43.174842+0000","last_active":"2024-02-23T00:39:44.206690+0000","last_peered":"2024-02-23T00:39:44.206690+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.174842+0000","last_became_peered":"2024-02-23T00:39:43.174842+0000","last_unstale":"2024-02-23T00:39:44.206690+0000","last_undegraded":"2024-02-23T00:39:44.206690+0000","last_fullsized":"2024-02-23T00:39:43.166254+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"229.4","version":"0'0","reported_seq":10,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.187583+0000","last_change":"2024-02-23T00:39:41.190995+0000","last_active":"2024-02-23T00:39:44.187583+0000","last_peered":"2024-02-23T00:39:44.187583+0000","last_clean":"2024-02-23T00:39:40.168986+0000","last_became_active":"2024-02-23T00:39:41.190995+0000","last_became_peered":"2024-02-23T00:39:41.190995+0000","last_unstale":"2024-02-23T00:39:44.187583+0000","last_undegraded":"2024-02-23T00:39:44.187583+0000","last_fullsized":"2024-02-23T00:39:41.180069+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":561,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.168986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,2147483647],"acting":[2,3,2147483647],"avail_no_missing":["2(0)","3(1)"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"231.1","version":"0'0","reported_seq":6,"reported_epoch":566,"state":"active+undersized","last_fresh":"2024-02-23T00:39:45.185962+0000","last_change":"2024-02-23T00:39:45.185878+0000","last_active":"2024-02-23T00:39:45.185962+0000","last_peered":"2024-02-23T00:39:45.185962+0000","last_clean":"2024-02-23T00:39:44.168834+0000","last_became_active":"2024-02-23T00:39:45.185878+0000","last_became_peered":"2024-02-23T00:39:45.185878+0000","last_unstale":"2024-02-23T00:39:45.185962+0000","last_undegraded":"2024-02-23T00:39:45.185962+0000","last_fullsized":"2024-02-23T00:39:45.178368+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"230.0","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.186636+0000","last_change":"2024-02-23T00:39:43.181631+0000","last_active":"2024-02-23T00:39:44.186636+0000","last_peered":"2024-02-23T00:39:44.186636+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.181631+0000","last_became_peered":"2024-02-23T00:39:43.181631+0000","last_unstale":"2024-02-23T00:39:44.186636+0000","last_undegraded":"2024-02-23T00:39:44.186636+0000","last_fullsized":"2024-02-23T00:39:43.165928+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"229.3","version":"0'0","reported_seq":13,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.185088+0000","last_change":"2024-02-23T00:39:41.188920+0000","last_active":"2024-02-23T00:39:44.185088+0000","last_peered":"2024-02-23T00:39:44.185088+0000","last_clean":"2024-02-23T00:39:44.185088+0000","last_became_active":"2024-02-23T00:39:41.188603+0000","last_became_peered":"2024-02-23T00:39:41.188603+0000","last_unstale":"2024-02-23T00:39:44.185088+0000","last_undegraded":"2024-02-23T00:39:44.185088+0000","last_fullsized":"2024-02-23T00:39:44.185088+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":561,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.168986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"231.0","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.197971+0000","last_change":"2024-02-23T00:39:45.197971+0000","last_active":"2024-02-23T00:39:45.197971+0000","last_peered":"2024-02-23T00:39:45.197971+0000","last_clean":"2024-02-23T00:39:45.197971+0000","last_became_active":"2024-02-23T00:39:45.197676+0000","last_became_peered":"2024-02-23T00:39:45.197676+0000","last_unstale":"2024-02-23T00:39:45.197971+0000","last_undegraded":"2024-02-23T00:39:45.197971+0000","last_fullsized":"2024-02-23T00:39:45.197971+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"230.1","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178555+0000","last_change":"2024-02-23T00:39:43.181224+0000","last_active":"2024-02-23T00:39:44.178555+0000","last_peered":"2024-02-23T00:39:44.178555+0000","last_clean":"2024-02-23T00:39:44.178555+0000","last_became_active":"2024-02-23T00:39:43.180928+0000","last_became_peered":"2024-02-23T00:39:43.180928+0000","last_unstale":"2024-02-23T00:39:44.178555+0000","last_undegraded":"2024-02-23T00:39:44.178555+0000","last_fullsized":"2024-02-23T00:39:44.178555+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"229.2","version":"562'4","reported_seq":19,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179779+0000","last_change":"2024-02-23T00:39:41.189099+0000","last_active":"2024-02-23T00:39:44.179779+0000","last_peered":"2024-02-23T00:39:44.179779+0000","last_clean":"2024-02-23T00:39:44.179779+0000","last_became_active":"2024-02-23T00:39:41.188377+0000","last_became_peered":"2024-02-23T00:39:41.188377+0000","last_unstale":"2024-02-23T00:39:44.179779+0000","last_undegraded":"2024-02-23T00:39:44.179779+0000","last_fullsized":"2024-02-23T00:39:44.179779+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":561,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.168986+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"231.3","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.190123+0000","last_change":"2024-02-23T00:39:45.190123+0000","last_active":"2024-02-23T00:39:45.190123+0000","last_peered":"2024-02-23T00:39:45.190123+0000","last_clean":"2024-02-23T00:39:45.190123+0000","last_became_active":"2024-02-23T00:39:45.189830+0000","last_became_peered":"2024-02-23T00:39:45.189830+0000","last_unstale":"2024-02-23T00:39:45.190123+0000","last_undegraded":"2024-02-23T00:39:45.190123+0000","last_fullsized":"2024-02-23T00:39:45.190123+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"230.2","version":"0'0","reported_seq":11,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178938+0000","last_change":"2024-02-23T00:39:43.181128+0000","last_active":"2024-02-23T00:39:44.178938+0000","last_peered":"2024-02-23T00:39:44.178938+0000","last_clean":"2024-02-23T00:39:44.178938+0000","last_became_active":"2024-02-23T00:39:43.180695+0000","last_became_peered":"2024-02-23T00:39:43.180695+0000","last_unstale":"2024-02-23T00:39:44.178938+0000","last_undegraded":"2024-02-23T00:39:44.178938+0000","last_fullsized":"2024-02-23T00:39:44.178938+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"229.1","version":"0'0","reported_seq":13,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.188477+0000","last_change":"2024-02-23T00:39:41.196579+0000","last_active":"2024-02-23T00:39:44.188477+0000","last_peered":"2024-02-23T00:39:44.188477+0000","last_clean":"2024-02-23T00:39:44.188477+0000","last_became_active":"2024-02-23T00:39:41.196180+0000","last_became_peered":"2024-02-23T00:39:41.196180+0000","last_unstale":"2024-02-23T00:39:44.188477+0000","last_undegraded":"2024-02-23T00:39:44.188477+0000","last_fullsized":"2024-02-23T00:39:44.188477+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":561,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.168986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"231.2","version":"0'0","reported_seq":9,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.204520+0000","last_change":"2024-02-23T00:39:45.204520+0000","last_active":"2024-02-23T00:39:45.204520+0000","last_peered":"2024-02-23T00:39:45.204520+0000","last_clean":"2024-02-23T00:39:45.204520+0000","last_became_active":"2024-02-23T00:39:45.204163+0000","last_became_peered":"2024-02-23T00:39:45.204163+0000","last_unstale":"2024-02-23T00:39:45.204520+0000","last_undegraded":"2024-02-23T00:39:45.204520+0000","last_fullsized":"2024-02-23T00:39:45.204520+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":565,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:44.168834+0000","last_clean_scrub_stamp":"2024-02-23T00:39:44.168834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"230.3","version":"0'0","reported_seq":8,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.181050+0000","last_change":"2024-02-23T00:39:43.170457+0000","last_active":"2024-02-23T00:39:44.181050+0000","last_peered":"2024-02-23T00:39:44.181050+0000","last_clean":"2024-02-23T00:39:42.154665+0000","last_became_active":"2024-02-23T00:39:43.170457+0000","last_became_peered":"2024-02-23T00:39:43.170457+0000","last_unstale":"2024-02-23T00:39:44.181050+0000","last_undegraded":"2024-02-23T00:39:44.181050+0000","last_fullsized":"2024-02-23T00:39:43.165582+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":563,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:42.154665+0000","last_clean_scrub_stamp":"2024-02-23T00:39:42.154665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"229.0","version":"0'0","reported_seq":13,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.204628+0000","last_change":"2024-02-23T00:39:41.189531+0000","last_active":"2024-02-23T00:39:44.204628+0000","last_peered":"2024-02-23T00:39:44.204628+0000","last_clean":"2024-02-23T00:39:44.204628+0000","last_became_active":"2024-02-23T00:39:41.188582+0000","last_became_peered":"2024-02-23T00:39:41.188582+0000","last_unstale":"2024-02-23T00:39:44.204628+0000","last_undegraded":"2024-02-23T00:39:44.204628+0000","last_fullsized":"2024-02-23T00:39:44.204628+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":561,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:40.168986+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.168986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":365,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.180411+0000","last_change":"2024-02-23T00:39:42.167336+0000","last_active":"2024-02-23T00:39:44.180411+0000","last_peered":"2024-02-23T00:39:44.180411+0000","last_clean":"2024-02-23T00:39:44.180411+0000","last_became_active":"2024-02-23T00:39:42.166798+0000","last_became_peered":"2024-02-23T00:39:42.166798+0000","last_unstale":"2024-02-23T00:39:44.180411+0000","last_undegraded":"2024-02-23T00:39:44.180411+0000","last_fullsized":"2024-02-23T00:39:44.180411+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.678152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.678152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1,3],"avail_no_missing":["2","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":368,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.182831+0000","last_change":"2024-02-23T00:39:42.179625+0000","last_active":"2024-02-23T00:39:44.182831+0000","last_peered":"2024-02-23T00:39:44.182831+0000","last_clean":"2024-02-23T00:39:44.182831+0000","last_became_active":"2024-02-23T00:39:42.179186+0000","last_became_peered":"2024-02-23T00:39:42.179186+0000","last_unstale":"2024-02-23T00:39:44.182831+0000","last_undegraded":"2024-02-23T00:39:44.182831+0000","last_fullsized":"2024-02-23T00:39:44.182831+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.026383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.026383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5,7],"avail_no_missing":["6","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"563'15","reported_seq":23,"reported_epoch":566,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:45.183326+0000","last_change":"2024-02-23T00:39:42.185217+0000","last_active":"2024-02-23T00:39:45.183326+0000","last_peered":"2024-02-23T00:39:45.183326+0000","last_clean":"2024-02-23T00:39:45.183326+0000","last_became_active":"2024-02-23T00:39:42.181209+0000","last_became_peered":"2024-02-23T00:39:42.181209+0000","last_unstale":"2024-02-23T00:39:45.183326+0000","last_undegraded":"2024-02-23T00:39:45.183326+0000","last_fullsized":"2024-02-23T00:39:45.183326+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5,2],"avail_no_missing":["6","2","5"],"object_location_counts":[{"shards":"2,5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"562'12","reported_seq":20,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.180052+0000","last_change":"2024-02-23T00:39:41.275972+0000","last_active":"2024-02-23T00:39:45.180052+0000","last_peered":"2024-02-23T00:39:45.180052+0000","last_clean":"2024-02-23T00:39:45.180052+0000","last_became_active":"2024-02-23T00:39:41.185268+0000","last_became_peered":"2024-02-23T00:39:41.185268+0000","last_unstale":"2024-02-23T00:39:45.180052+0000","last_undegraded":"2024-02-23T00:39:45.180052+0000","last_fullsized":"2024-02-23T00:39:45.180052+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":6,"last_scrub":"379'11","last_scrub_stamp":"2024-02-23T00:37:10.032341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:10.032341+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":14,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178698+0000","last_change":"2024-02-23T00:39:41.189853+0000","last_active":"2024-02-23T00:39:44.178698+0000","last_peered":"2024-02-23T00:39:44.178698+0000","last_clean":"2024-02-23T00:39:44.178698+0000","last_became_active":"2024-02-23T00:39:41.189429+0000","last_became_peered":"2024-02-23T00:39:41.189429+0000","last_unstale":"2024-02-23T00:39:44.178698+0000","last_undegraded":"2024-02-23T00:39:44.178698+0000","last_fullsized":"2024-02-23T00:39:44.178698+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:38.337256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:34.152643+0000","last_clean_scrub_stamp":"2024-02-23T00:39:38.337256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":420,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.192029+0000","last_change":"2024-02-23T00:39:42.175134+0000","last_active":"2024-02-23T00:39:44.192029+0000","last_peered":"2024-02-23T00:39:44.192029+0000","last_clean":"2024-02-23T00:39:44.192029+0000","last_became_active":"2024-02-23T00:39:42.174812+0000","last_became_peered":"2024-02-23T00:39:42.174812+0000","last_unstale":"2024-02-23T00:39:44.192029+0000","last_undegraded":"2024-02-23T00:39:44.192029+0000","last_fullsized":"2024-02-23T00:39:44.192029+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:33:38.742210+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":15,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.185215+0000","last_change":"2024-02-23T00:39:42.180977+0000","last_active":"2024-02-23T00:39:44.185215+0000","last_peered":"2024-02-23T00:39:44.185215+0000","last_clean":"2024-02-23T00:39:44.185215+0000","last_became_active":"2024-02-23T00:39:42.180358+0000","last_became_peered":"2024-02-23T00:39:42.180358+0000","last_unstale":"2024-02-23T00:39:44.185215+0000","last_undegraded":"2024-02-23T00:39:44.185215+0000","last_fullsized":"2024-02-23T00:39:44.185215+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:43.246802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:38:43.246802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":14,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181697+0000","last_change":"2024-02-23T00:39:41.189014+0000","last_active":"2024-02-23T00:39:44.181697+0000","last_peered":"2024-02-23T00:39:44.181697+0000","last_clean":"2024-02-23T00:39:44.181697+0000","last_became_active":"2024-02-23T00:39:41.188508+0000","last_became_peered":"2024-02-23T00:39:41.188508+0000","last_unstale":"2024-02-23T00:39:44.181697+0000","last_undegraded":"2024-02-23T00:39:44.181697+0000","last_fullsized":"2024-02-23T00:39:44.181697+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":558,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.179446+0000","last_change":"2024-02-23T00:39:42.176054+0000","last_active":"2024-02-23T00:39:44.179446+0000","last_peered":"2024-02-23T00:39:44.179446+0000","last_clean":"2024-02-23T00:39:44.179446+0000","last_became_active":"2024-02-23T00:39:42.175618+0000","last_became_peered":"2024-02-23T00:39:42.175618+0000","last_unstale":"2024-02-23T00:39:44.179446+0000","last_undegraded":"2024-02-23T00:39:44.179446+0000","last_fullsized":"2024-02-23T00:39:44.179446+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.191221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.191221+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.191221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"379'9","reported_seq":708,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.179341+0000","last_change":"2024-02-23T00:37:05.955772+0000","last_active":"2024-02-23T00:39:45.179341+0000","last_peered":"2024-02-23T00:39:45.179341+0000","last_clean":"2024-02-23T00:39:45.179341+0000","last_became_active":"2024-02-23T00:35:27.095085+0000","last_became_peered":"2024-02-23T00:35:27.095085+0000","last_unstale":"2024-02-23T00:39:45.179341+0000","last_undegraded":"2024-02-23T00:39:45.179341+0000","last_fullsized":"2024-02-23T00:39:45.179341+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'9","last_scrub_stamp":"2024-02-23T00:37:05.955707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.955707+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":650,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179695+0000","last_change":"2024-02-23T00:38:19.164982+0000","last_active":"2024-02-23T00:39:44.179695+0000","last_peered":"2024-02-23T00:39:44.179695+0000","last_clean":"2024-02-23T00:39:44.179695+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:39:44.179695+0000","last_undegraded":"2024-02-23T00:39:44.179695+0000","last_fullsized":"2024-02-23T00:39:44.179695+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":578,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.192201+0000","last_change":"2024-02-23T00:39:39.337422+0000","last_active":"2024-02-23T00:39:44.192201+0000","last_peered":"2024-02-23T00:39:44.192201+0000","last_clean":"2024-02-23T00:39:44.192201+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:39:44.192201+0000","last_undegraded":"2024-02-23T00:39:44.192201+0000","last_fullsized":"2024-02-23T00:39:44.192201+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.337303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.337303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":528,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.182671+0000","last_change":"2024-02-23T00:36:37.104200+0000","last_active":"2024-02-23T00:39:44.182671+0000","last_peered":"2024-02-23T00:39:44.182671+0000","last_clean":"2024-02-23T00:36:33.366159+0000","last_became_active":"2024-02-23T00:36:37.104200+0000","last_became_peered":"2024-02-23T00:36:37.104200+0000","last_unstale":"2024-02-23T00:39:44.182671+0000","last_undegraded":"2024-02-23T00:39:44.182671+0000","last_fullsized":"2024-02-23T00:36:37.102434+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:21.441958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.441958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":492,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.179662+0000","last_change":"2024-02-23T00:36:37.115879+0000","last_active":"2024-02-23T00:39:44.179662+0000","last_peered":"2024-02-23T00:39:44.179662+0000","last_clean":"2024-02-23T00:36:33.366205+0000","last_became_active":"2024-02-23T00:36:37.115879+0000","last_became_peered":"2024-02-23T00:36:37.115879+0000","last_unstale":"2024-02-23T00:39:44.179662+0000","last_undegraded":"2024-02-23T00:39:44.179662+0000","last_fullsized":"2024-02-23T00:36:37.101967+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":974,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.184170+0000","last_change":"2024-02-23T00:39:36.787515+0000","last_active":"2024-02-23T00:39:45.184170+0000","last_peered":"2024-02-23T00:39:45.184170+0000","last_clean":"2024-02-23T00:39:45.184170+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:39:45.184170+0000","last_undegraded":"2024-02-23T00:39:45.184170+0000","last_fullsized":"2024-02-23T00:39:45.184170+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:39:36.787450+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:39:36.787450+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":426,"num_read_kb":426,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":211,"reported_epoch":565,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:44.185628+0000","last_change":"2024-02-23T00:38:19.177071+0000","last_active":"2024-02-23T00:39:44.185628+0000","last_peered":"2024-02-23T00:39:44.185628+0000","last_clean":"2024-02-23T00:27:29.183258+0000","last_became_active":"2024-02-23T00:38:19.177071+0000","last_became_peered":"2024-02-23T00:38:19.177071+0000","last_unstale":"2024-02-23T00:39:44.185628+0000","last_undegraded":"2024-02-23T00:38:19.154601+0000","last_fullsized":"2024-02-23T00:38:19.154480+0000","mapping_epoch":425,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":553,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.180034+0000","last_change":"2024-02-23T00:38:49.228134+0000","last_active":"2024-02-23T00:39:44.180034+0000","last_peered":"2024-02-23T00:39:44.180034+0000","last_clean":"2024-02-23T00:39:44.180034+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:39:44.180034+0000","last_undegraded":"2024-02-23T00:39:44.180034+0000","last_fullsized":"2024-02-23T00:39:44.180034+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:41.532619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:41.532619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":565,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.188114+0000","last_change":"2024-02-23T00:39:20.396906+0000","last_active":"2024-02-23T00:39:44.188114+0000","last_peered":"2024-02-23T00:39:44.188114+0000","last_clean":"2024-02-23T00:39:44.188114+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:39:44.188114+0000","last_undegraded":"2024-02-23T00:39:44.188114+0000","last_fullsized":"2024-02-23T00:39:44.188114+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:20.396815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:39:20.396815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":148,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.186751+0000","last_change":"2024-02-23T00:36:37.122229+0000","last_active":"2024-02-23T00:39:44.186751+0000","last_peered":"2024-02-23T00:39:44.186751+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.122229+0000","last_became_peered":"2024-02-23T00:36:37.122229+0000","last_unstale":"2024-02-23T00:39:44.186751+0000","last_undegraded":"2024-02-23T00:39:44.186751+0000","last_fullsized":"2024-02-23T00:36:37.107921+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:33:47.610212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":491,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187427+0000","last_change":"2024-02-23T00:39:35.342488+0000","last_active":"2024-02-23T00:39:44.187427+0000","last_peered":"2024-02-23T00:39:44.187427+0000","last_clean":"2024-02-23T00:39:44.187427+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:39:44.187427+0000","last_undegraded":"2024-02-23T00:39:44.187427+0000","last_fullsized":"2024-02-23T00:39:44.187427+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:39:35.342427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":670,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.183228+0000","last_change":"2024-02-23T00:38:40.424549+0000","last_active":"2024-02-23T00:39:44.183228+0000","last_peered":"2024-02-23T00:39:44.183228+0000","last_clean":"2024-02-23T00:39:44.183228+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:39:44.183228+0000","last_undegraded":"2024-02-23T00:39:44.183228+0000","last_fullsized":"2024-02-23T00:39:44.183228+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:40.424490+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.424490+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":551,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.188173+0000","last_change":"2024-02-23T00:36:37.116992+0000","last_active":"2024-02-23T00:39:44.188173+0000","last_peered":"2024-02-23T00:39:44.188173+0000","last_clean":"2024-02-23T00:36:33.194161+0000","last_became_active":"2024-02-23T00:36:37.116992+0000","last_became_peered":"2024-02-23T00:36:37.116992+0000","last_unstale":"2024-02-23T00:39:44.188173+0000","last_undegraded":"2024-02-23T00:39:44.188173+0000","last_fullsized":"2024-02-23T00:36:37.104222+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:50.194656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:50.194656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":542,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.191813+0000","last_change":"2024-02-23T00:36:37.113419+0000","last_active":"2024-02-23T00:39:44.191813+0000","last_peered":"2024-02-23T00:39:44.191813+0000","last_clean":"2024-02-23T00:36:33.194007+0000","last_became_active":"2024-02-23T00:36:37.113419+0000","last_became_peered":"2024-02-23T00:36:37.113419+0000","last_unstale":"2024-02-23T00:39:44.191813+0000","last_undegraded":"2024-02-23T00:39:44.191813+0000","last_fullsized":"2024-02-23T00:36:37.104161+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.745865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.745865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":554,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187367+0000","last_change":"2024-02-23T00:38:49.208110+0000","last_active":"2024-02-23T00:39:44.187367+0000","last_peered":"2024-02-23T00:39:44.187367+0000","last_clean":"2024-02-23T00:39:44.187367+0000","last_became_active":"2024-02-23T00:38:49.207843+0000","last_became_peered":"2024-02-23T00:38:49.207843+0000","last_unstale":"2024-02-23T00:39:44.187367+0000","last_undegraded":"2024-02-23T00:39:44.187367+0000","last_fullsized":"2024-02-23T00:39:44.187367+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.226380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.226380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":149,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.181537+0000","last_change":"2024-02-23T00:36:37.119721+0000","last_active":"2024-02-23T00:39:44.181537+0000","last_peered":"2024-02-23T00:39:44.181537+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119721+0000","last_became_peered":"2024-02-23T00:36:37.119721+0000","last_unstale":"2024-02-23T00:39:44.181537+0000","last_undegraded":"2024-02-23T00:39:44.181537+0000","last_fullsized":"2024-02-23T00:36:37.102163+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.205312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.205312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":553,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.183668+0000","last_change":"2024-02-23T00:38:44.442795+0000","last_active":"2024-02-23T00:39:44.183668+0000","last_peered":"2024-02-23T00:39:44.183668+0000","last_clean":"2024-02-23T00:39:44.183668+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:39:44.183668+0000","last_undegraded":"2024-02-23T00:39:44.183668+0000","last_fullsized":"2024-02-23T00:39:44.183668+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:44.442733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.442733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":628,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181446+0000","last_change":"2024-02-23T00:39:37.346021+0000","last_active":"2024-02-23T00:39:44.181446+0000","last_peered":"2024-02-23T00:39:44.181446+0000","last_clean":"2024-02-23T00:39:44.181446+0000","last_became_active":"2024-02-23T00:38:19.176922+0000","last_became_peered":"2024-02-23T00:38:19.176922+0000","last_unstale":"2024-02-23T00:39:44.181446+0000","last_undegraded":"2024-02-23T00:39:44.181446+0000","last_fullsized":"2024-02-23T00:39:44.181446+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:39:37.345922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:39:37.345922+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":542,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.185296+0000","last_change":"2024-02-23T00:38:18.055894+0000","last_active":"2024-02-23T00:39:44.185296+0000","last_peered":"2024-02-23T00:39:44.185296+0000","last_clean":"2024-02-23T00:39:44.185296+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:39:44.185296+0000","last_undegraded":"2024-02-23T00:39:44.185296+0000","last_fullsized":"2024-02-23T00:39:44.185296+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:18.055803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:18.055803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":532,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.178907+0000","last_change":"2024-02-23T00:36:37.109480+0000","last_active":"2024-02-23T00:39:44.178907+0000","last_peered":"2024-02-23T00:39:44.178907+0000","last_clean":"2024-02-23T00:36:34.605757+0000","last_became_active":"2024-02-23T00:36:37.109480+0000","last_became_peered":"2024-02-23T00:36:37.109480+0000","last_unstale":"2024-02-23T00:39:44.178907+0000","last_undegraded":"2024-02-23T00:39:44.178907+0000","last_fullsized":"2024-02-23T00:36:37.095167+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.605664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.605664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":587,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.190528+0000","last_change":"2024-02-23T00:39:27.413638+0000","last_active":"2024-02-23T00:39:44.190528+0000","last_peered":"2024-02-23T00:39:44.190528+0000","last_clean":"2024-02-23T00:39:44.190528+0000","last_became_active":"2024-02-23T00:38:49.181997+0000","last_became_peered":"2024-02-23T00:38:49.181997+0000","last_unstale":"2024-02-23T00:39:44.190528+0000","last_undegraded":"2024-02-23T00:39:44.190528+0000","last_fullsized":"2024-02-23T00:39:44.190528+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:27.413549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:39:27.413549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":516,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.183016+0000","last_change":"2024-02-23T00:39:27.813080+0000","last_active":"2024-02-23T00:39:44.183016+0000","last_peered":"2024-02-23T00:39:44.183016+0000","last_clean":"2024-02-23T00:39:44.183016+0000","last_became_active":"2024-02-23T00:38:49.194450+0000","last_became_peered":"2024-02-23T00:38:49.194450+0000","last_unstale":"2024-02-23T00:39:44.183016+0000","last_undegraded":"2024-02-23T00:39:44.183016+0000","last_fullsized":"2024-02-23T00:39:44.183016+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:27.813005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:27.813005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":65,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.197090+0000","last_change":"2024-02-23T00:38:50.199088+0000","last_active":"2024-02-23T00:39:44.197090+0000","last_peered":"2024-02-23T00:39:44.197090+0000","last_clean":"2024-02-23T00:39:44.197090+0000","last_became_active":"2024-02-23T00:38:50.198701+0000","last_became_peered":"2024-02-23T00:38:50.198701+0000","last_unstale":"2024-02-23T00:39:44.197090+0000","last_undegraded":"2024-02-23T00:39:44.197090+0000","last_fullsized":"2024-02-23T00:39:44.197090+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,1],"avail_no_missing":["3","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":667,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.182964+0000","last_change":"2024-02-23T00:38:54.172406+0000","last_active":"2024-02-23T00:39:44.182964+0000","last_peered":"2024-02-23T00:39:44.182964+0000","last_clean":"2024-02-23T00:39:44.182964+0000","last_became_active":"2024-02-23T00:38:54.171993+0000","last_became_peered":"2024-02-23T00:38:54.171993+0000","last_unstale":"2024-02-23T00:39:44.182964+0000","last_undegraded":"2024-02-23T00:39:44.182964+0000","last_fullsized":"2024-02-23T00:39:44.182964+0000","mapping_epoch":516,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:37:09.133471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.133471+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":571,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179344+0000","last_change":"2024-02-23T00:39:40.287714+0000","last_active":"2024-02-23T00:39:44.179344+0000","last_peered":"2024-02-23T00:39:44.179344+0000","last_clean":"2024-02-23T00:39:44.179344+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:39:44.179344+0000","last_undegraded":"2024-02-23T00:39:44.179344+0000","last_fullsized":"2024-02-23T00:39:44.179344+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.287611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.287611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":704,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.188686+0000","last_change":"2024-02-23T00:38:35.337273+0000","last_active":"2024-02-23T00:39:45.188686+0000","last_peered":"2024-02-23T00:39:45.188686+0000","last_clean":"2024-02-23T00:39:45.188686+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:39:45.188686+0000","last_undegraded":"2024-02-23T00:39:45.188686+0000","last_fullsized":"2024-02-23T00:39:45.188686+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:38:35.337170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.337170+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":556,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.182093+0000","last_change":"2024-02-23T00:38:49.171993+0000","last_active":"2024-02-23T00:39:44.182093+0000","last_peered":"2024-02-23T00:39:44.182093+0000","last_clean":"2024-02-23T00:39:44.182093+0000","last_became_active":"2024-02-23T00:38:49.171696+0000","last_became_peered":"2024-02-23T00:38:49.171696+0000","last_unstale":"2024-02-23T00:39:44.182093+0000","last_undegraded":"2024-02-23T00:39:44.182093+0000","last_fullsized":"2024-02-23T00:39:44.182093+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.135802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.135802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":491,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.182146+0000","last_change":"2024-02-23T00:38:34.097327+0000","last_active":"2024-02-23T00:39:44.182146+0000","last_peered":"2024-02-23T00:39:44.182146+0000","last_clean":"2024-02-23T00:39:44.182146+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:39:44.182146+0000","last_undegraded":"2024-02-23T00:39:44.182146+0000","last_fullsized":"2024-02-23T00:39:44.182146+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:34.097243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.097243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.205323+0000","last_change":"2024-02-23T00:38:49.223162+0000","last_active":"2024-02-23T00:39:44.205323+0000","last_peered":"2024-02-23T00:39:44.205323+0000","last_clean":"2024-02-23T00:39:44.205323+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:39:44.205323+0000","last_undegraded":"2024-02-23T00:39:44.205323+0000","last_fullsized":"2024-02-23T00:39:44.205323+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:22.396222+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.396222+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":725,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179973+0000","last_change":"2024-02-23T00:39:25.200274+0000","last_active":"2024-02-23T00:39:44.179973+0000","last_peered":"2024-02-23T00:39:44.179973+0000","last_clean":"2024-02-23T00:39:44.179973+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:39:44.179973+0000","last_undegraded":"2024-02-23T00:39:44.179973+0000","last_fullsized":"2024-02-23T00:39:44.179973+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":555,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178131+0000","last_change":"2024-02-23T00:39:37.339984+0000","last_active":"2024-02-23T00:39:44.178131+0000","last_peered":"2024-02-23T00:39:44.178131+0000","last_clean":"2024-02-23T00:39:44.178131+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:39:44.178131+0000","last_undegraded":"2024-02-23T00:39:44.178131+0000","last_fullsized":"2024-02-23T00:39:44.178131+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:37.339877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:39:37.339877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"426'9","reported_seq":700,"reported_epoch":566,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:45.197778+0000","last_change":"2024-02-23T00:36:37.111178+0000","last_active":"2024-02-23T00:39:45.197778+0000","last_peered":"2024-02-23T00:39:45.197778+0000","last_clean":"2024-02-23T00:36:35.444981+0000","last_became_active":"2024-02-23T00:36:37.111178+0000","last_became_peered":"2024-02-23T00:36:37.111178+0000","last_unstale":"2024-02-23T00:39:45.197778+0000","last_undegraded":"2024-02-23T00:36:37.095825+0000","last_fullsized":"2024-02-23T00:36:37.095713+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:59.630171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:59.630171+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":529,"reported_epoch":565,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:44.184656+0000","last_change":"2024-02-23T00:36:37.113156+0000","last_active":"2024-02-23T00:39:44.184656+0000","last_peered":"2024-02-23T00:39:44.184656+0000","last_clean":"2024-02-23T00:36:33.366740+0000","last_became_active":"2024-02-23T00:36:37.113156+0000","last_became_peered":"2024-02-23T00:36:37.113156+0000","last_unstale":"2024-02-23T00:39:44.184656+0000","last_undegraded":"2024-02-23T00:36:37.102218+0000","last_fullsized":"2024-02-23T00:36:37.102103+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:34:17.437083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:17.437083+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":482,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179641+0000","last_change":"2024-02-23T00:38:16.396902+0000","last_active":"2024-02-23T00:39:44.179641+0000","last_peered":"2024-02-23T00:39:44.179641+0000","last_clean":"2024-02-23T00:39:44.179641+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:39:44.179641+0000","last_undegraded":"2024-02-23T00:39:44.179641+0000","last_fullsized":"2024-02-23T00:39:44.179641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.396824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":553,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178788+0000","last_change":"2024-02-23T00:38:08.068624+0000","last_active":"2024-02-23T00:39:44.178788+0000","last_peered":"2024-02-23T00:39:44.178788+0000","last_clean":"2024-02-23T00:39:44.178788+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:39:44.178788+0000","last_undegraded":"2024-02-23T00:39:44.178788+0000","last_fullsized":"2024-02-23T00:39:44.178788+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:08.068559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:08.068559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":637,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181904+0000","last_change":"2024-02-23T00:38:19.182554+0000","last_active":"2024-02-23T00:39:44.181904+0000","last_peered":"2024-02-23T00:39:44.181904+0000","last_clean":"2024-02-23T00:39:44.181904+0000","last_became_active":"2024-02-23T00:38:19.182222+0000","last_became_peered":"2024-02-23T00:38:19.182222+0000","last_unstale":"2024-02-23T00:39:44.181904+0000","last_undegraded":"2024-02-23T00:39:44.181904+0000","last_fullsized":"2024-02-23T00:39:44.181904+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":550,"reported_epoch":565,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:44.187785+0000","last_change":"2024-02-23T00:36:37.126232+0000","last_active":"2024-02-23T00:39:44.187785+0000","last_peered":"2024-02-23T00:39:44.187785+0000","last_clean":"2024-02-23T00:36:33.365949+0000","last_became_active":"2024-02-23T00:36:37.126232+0000","last_became_peered":"2024-02-23T00:36:37.126232+0000","last_unstale":"2024-02-23T00:39:44.187785+0000","last_undegraded":"2024-02-23T00:36:37.112806+0000","last_fullsized":"2024-02-23T00:36:37.112690+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":529,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.185908+0000","last_change":"2024-02-23T00:38:37.881705+0000","last_active":"2024-02-23T00:39:44.185908+0000","last_peered":"2024-02-23T00:39:44.185908+0000","last_clean":"2024-02-23T00:39:44.185908+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:39:44.185908+0000","last_undegraded":"2024-02-23T00:39:44.185908+0000","last_fullsized":"2024-02-23T00:39:44.185908+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:37.881618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.881618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.198039+0000","last_change":"2024-02-23T00:38:49.230667+0000","last_active":"2024-02-23T00:39:44.198039+0000","last_peered":"2024-02-23T00:39:44.198039+0000","last_clean":"2024-02-23T00:39:44.198039+0000","last_became_active":"2024-02-23T00:38:49.226279+0000","last_became_peered":"2024-02-23T00:38:49.226279+0000","last_unstale":"2024-02-23T00:39:44.198039+0000","last_undegraded":"2024-02-23T00:39:44.198039+0000","last_fullsized":"2024-02-23T00:39:44.198039+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.478470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.478470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":500,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179036+0000","last_change":"2024-02-23T00:38:49.227768+0000","last_active":"2024-02-23T00:39:44.179036+0000","last_peered":"2024-02-23T00:39:44.179036+0000","last_clean":"2024-02-23T00:39:44.179036+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:39:44.179036+0000","last_undegraded":"2024-02-23T00:39:44.179036+0000","last_fullsized":"2024-02-23T00:39:44.179036+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:21.364934+0000","last_clean_scrub_stamp":"2024-02-23T00:37:21.364934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":149,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.186210+0000","last_change":"2024-02-23T00:36:37.109542+0000","last_active":"2024-02-23T00:39:44.186210+0000","last_peered":"2024-02-23T00:39:44.186210+0000","last_clean":"2024-02-23T00:31:27.149861+0000","last_became_active":"2024-02-23T00:36:37.109542+0000","last_became_peered":"2024-02-23T00:36:37.109542+0000","last_unstale":"2024-02-23T00:39:44.186210+0000","last_undegraded":"2024-02-23T00:39:44.186210+0000","last_fullsized":"2024-02-23T00:36:37.096846+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:42.114807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:35:42.114807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.205243+0000","last_change":"2024-02-23T00:38:49.222977+0000","last_active":"2024-02-23T00:39:44.205243+0000","last_peered":"2024-02-23T00:39:44.205243+0000","last_clean":"2024-02-23T00:39:44.205243+0000","last_became_active":"2024-02-23T00:38:49.216657+0000","last_became_peered":"2024-02-23T00:38:49.216657+0000","last_unstale":"2024-02-23T00:39:44.205243+0000","last_undegraded":"2024-02-23T00:39:44.205243+0000","last_fullsized":"2024-02-23T00:39:44.205243+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":594,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179324+0000","last_change":"2024-02-23T00:37:08.107220+0000","last_active":"2024-02-23T00:39:44.179324+0000","last_peered":"2024-02-23T00:39:44.179324+0000","last_clean":"2024-02-23T00:39:44.179324+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:39:44.179324+0000","last_undegraded":"2024-02-23T00:39:44.179324+0000","last_fullsized":"2024-02-23T00:39:44.179324+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'20","last_scrub_stamp":"2024-02-23T00:37:08.107110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.107110+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":543,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181917+0000","last_change":"2024-02-23T00:38:48.494288+0000","last_active":"2024-02-23T00:39:44.181917+0000","last_peered":"2024-02-23T00:39:44.181917+0000","last_clean":"2024-02-23T00:39:44.181917+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:39:44.181917+0000","last_undegraded":"2024-02-23T00:39:44.181917+0000","last_fullsized":"2024-02-23T00:39:44.181917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:48.494224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:48.494224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":541,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.185518+0000","last_change":"2024-02-23T00:39:36.902074+0000","last_active":"2024-02-23T00:39:44.185518+0000","last_peered":"2024-02-23T00:39:44.185518+0000","last_clean":"2024-02-23T00:39:44.185518+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:39:44.185518+0000","last_undegraded":"2024-02-23T00:39:44.185518+0000","last_fullsized":"2024-02-23T00:39:44.185518+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:36.902015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:36.902015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":507,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.185870+0000","last_change":"2024-02-23T00:39:22.420483+0000","last_active":"2024-02-23T00:39:44.185870+0000","last_peered":"2024-02-23T00:39:44.185870+0000","last_clean":"2024-02-23T00:39:44.185870+0000","last_became_active":"2024-02-23T00:38:50.206899+0000","last_became_peered":"2024-02-23T00:38:50.206899+0000","last_unstale":"2024-02-23T00:39:44.185870+0000","last_undegraded":"2024-02-23T00:39:44.185870+0000","last_fullsized":"2024-02-23T00:39:44.185870+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:22.420380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.420380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":70,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.207213+0000","last_change":"2024-02-23T00:39:33.345047+0000","last_active":"2024-02-23T00:39:44.207213+0000","last_peered":"2024-02-23T00:39:44.207213+0000","last_clean":"2024-02-23T00:39:44.207213+0000","last_became_active":"2024-02-23T00:38:50.201068+0000","last_became_peered":"2024-02-23T00:38:50.201068+0000","last_unstale":"2024-02-23T00:39:44.207213+0000","last_undegraded":"2024-02-23T00:39:44.207213+0000","last_fullsized":"2024-02-23T00:39:44.207213+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:33.344870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:33.344870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,5],"avail_no_missing":["3","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":568,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179351+0000","last_change":"2024-02-23T00:38:49.228711+0000","last_active":"2024-02-23T00:39:44.179351+0000","last_peered":"2024-02-23T00:39:44.179351+0000","last_clean":"2024-02-23T00:39:44.179351+0000","last_became_active":"2024-02-23T00:38:49.228425+0000","last_became_peered":"2024-02-23T00:38:49.228425+0000","last_unstale":"2024-02-23T00:39:44.179351+0000","last_undegraded":"2024-02-23T00:39:44.179351+0000","last_fullsized":"2024-02-23T00:39:44.179351+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.588132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.588132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":92,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.205095+0000","last_change":"2024-02-23T00:39:28.324374+0000","last_active":"2024-02-23T00:39:44.205095+0000","last_peered":"2024-02-23T00:39:44.205095+0000","last_clean":"2024-02-23T00:39:44.205095+0000","last_became_active":"2024-02-23T00:38:50.199619+0000","last_became_peered":"2024-02-23T00:38:50.199619+0000","last_unstale":"2024-02-23T00:39:44.205095+0000","last_undegraded":"2024-02-23T00:39:44.205095+0000","last_fullsized":"2024-02-23T00:39:44.205095+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:39:28.324185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.324185+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":535,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.186268+0000","last_change":"2024-02-23T00:38:14.343844+0000","last_active":"2024-02-23T00:39:44.186268+0000","last_peered":"2024-02-23T00:39:44.186268+0000","last_clean":"2024-02-23T00:39:44.186268+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:39:44.186268+0000","last_undegraded":"2024-02-23T00:39:44.186268+0000","last_fullsized":"2024-02-23T00:39:44.186268+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:14.343770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:14.343770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":556,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179235+0000","last_change":"2024-02-23T00:38:49.225300+0000","last_active":"2024-02-23T00:39:44.179235+0000","last_peered":"2024-02-23T00:39:44.179235+0000","last_clean":"2024-02-23T00:39:44.179235+0000","last_became_active":"2024-02-23T00:38:49.224892+0000","last_became_peered":"2024-02-23T00:38:49.224892+0000","last_unstale":"2024-02-23T00:39:44.179235+0000","last_undegraded":"2024-02-23T00:39:44.179235+0000","last_fullsized":"2024-02-23T00:39:44.179235+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.082000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.082000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.196260+0000","last_change":"2024-02-23T00:38:49.176838+0000","last_active":"2024-02-23T00:39:44.196260+0000","last_peered":"2024-02-23T00:39:44.196260+0000","last_clean":"2024-02-23T00:39:44.196260+0000","last_became_active":"2024-02-23T00:38:49.175801+0000","last_became_peered":"2024-02-23T00:38:49.175801+0000","last_unstale":"2024-02-23T00:39:44.196260+0000","last_undegraded":"2024-02-23T00:39:44.196260+0000","last_fullsized":"2024-02-23T00:39:44.196260+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.977282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.977282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":798,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178507+0000","last_change":"2024-02-23T00:37:17.499775+0000","last_active":"2024-02-23T00:39:44.178507+0000","last_peered":"2024-02-23T00:39:44.178507+0000","last_clean":"2024-02-23T00:39:44.178507+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:39:44.178507+0000","last_undegraded":"2024-02-23T00:39:44.178507+0000","last_fullsized":"2024-02-23T00:39:44.178507+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:17.499694+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:17.499694+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":550,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.183331+0000","last_change":"2024-02-23T00:36:37.108691+0000","last_active":"2024-02-23T00:39:44.183331+0000","last_peered":"2024-02-23T00:39:44.183331+0000","last_clean":"2024-02-23T00:36:33.369928+0000","last_became_active":"2024-02-23T00:36:37.108691+0000","last_became_peered":"2024-02-23T00:36:37.108691+0000","last_unstale":"2024-02-23T00:39:44.183331+0000","last_undegraded":"2024-02-23T00:39:44.183331+0000","last_fullsized":"2024-02-23T00:36:37.100335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:33.196211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:33.196211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":227,"reported_epoch":565,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:44.181206+0000","last_change":"2024-02-23T00:36:37.123994+0000","last_active":"2024-02-23T00:39:44.181206+0000","last_peered":"2024-02-23T00:39:44.181206+0000","last_clean":"2024-02-23T00:27:29.216066+0000","last_became_active":"2024-02-23T00:36:37.123994+0000","last_became_peered":"2024-02-23T00:36:37.123994+0000","last_unstale":"2024-02-23T00:39:44.181206+0000","last_undegraded":"2024-02-23T00:36:37.103439+0000","last_fullsized":"2024-02-23T00:36:37.103335+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:35:01.396982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.396982+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":543,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.187739+0000","last_change":"2024-02-23T00:36:37.120094+0000","last_active":"2024-02-23T00:39:44.187739+0000","last_peered":"2024-02-23T00:39:44.187739+0000","last_clean":"2024-02-23T00:36:33.367908+0000","last_became_active":"2024-02-23T00:36:37.120094+0000","last_became_peered":"2024-02-23T00:36:37.120094+0000","last_unstale":"2024-02-23T00:39:44.187739+0000","last_undegraded":"2024-02-23T00:39:44.187739+0000","last_fullsized":"2024-02-23T00:36:37.107601+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":529,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178215+0000","last_change":"2024-02-23T00:37:26.268389+0000","last_active":"2024-02-23T00:39:44.178215+0000","last_peered":"2024-02-23T00:39:44.178215+0000","last_clean":"2024-02-23T00:39:44.178215+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:39:44.178215+0000","last_undegraded":"2024-02-23T00:39:44.178215+0000","last_fullsized":"2024-02-23T00:39:44.178215+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.268282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":482,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178289+0000","last_change":"2024-02-23T00:39:38.350980+0000","last_active":"2024-02-23T00:39:44.178289+0000","last_peered":"2024-02-23T00:39:44.178289+0000","last_clean":"2024-02-23T00:39:44.178289+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:39:44.178289+0000","last_undegraded":"2024-02-23T00:39:44.178289+0000","last_fullsized":"2024-02-23T00:39:44.178289+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:38.350890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:38.350890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":539,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.185975+0000","last_change":"2024-02-23T00:36:37.117616+0000","last_active":"2024-02-23T00:39:44.185975+0000","last_peered":"2024-02-23T00:39:44.185975+0000","last_clean":"2024-02-23T00:36:33.368573+0000","last_became_active":"2024-02-23T00:36:37.117616+0000","last_became_peered":"2024-02-23T00:36:37.117616+0000","last_unstale":"2024-02-23T00:39:44.185975+0000","last_undegraded":"2024-02-23T00:39:44.185975+0000","last_fullsized":"2024-02-23T00:36:37.100549+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":212,"reported_epoch":565,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:44.186414+0000","last_change":"2024-02-23T00:38:19.160002+0000","last_active":"2024-02-23T00:39:44.186414+0000","last_peered":"2024-02-23T00:39:44.186414+0000","last_clean":"2024-02-23T00:27:24.365389+0000","last_became_active":"2024-02-23T00:38:19.160002+0000","last_became_peered":"2024-02-23T00:38:19.160002+0000","last_unstale":"2024-02-23T00:39:44.186414+0000","last_undegraded":"2024-02-23T00:38:19.155620+0000","last_fullsized":"2024-02-23T00:38:19.155500+0000","mapping_epoch":425,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":543,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.187533+0000","last_change":"2024-02-23T00:36:37.126102+0000","last_active":"2024-02-23T00:39:44.187533+0000","last_peered":"2024-02-23T00:39:44.187533+0000","last_clean":"2024-02-23T00:36:33.365781+0000","last_became_active":"2024-02-23T00:36:37.126102+0000","last_became_peered":"2024-02-23T00:36:37.126102+0000","last_unstale":"2024-02-23T00:39:44.187533+0000","last_undegraded":"2024-02-23T00:39:44.187533+0000","last_fullsized":"2024-02-23T00:36:37.112617+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.980396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.980396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":480,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.179110+0000","last_change":"2024-02-23T00:39:24.425037+0000","last_active":"2024-02-23T00:39:44.179110+0000","last_peered":"2024-02-23T00:39:44.179110+0000","last_clean":"2024-02-23T00:39:44.179110+0000","last_became_active":"2024-02-23T00:38:50.177540+0000","last_became_peered":"2024-02-23T00:38:50.177540+0000","last_unstale":"2024-02-23T00:39:44.179110+0000","last_undegraded":"2024-02-23T00:39:44.179110+0000","last_fullsized":"2024-02-23T00:39:44.179110+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:24.424938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:24.424938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,5],"avail_no_missing":["4","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":547,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178942+0000","last_change":"2024-02-23T00:39:39.315882+0000","last_active":"2024-02-23T00:39:44.178942+0000","last_peered":"2024-02-23T00:39:44.178942+0000","last_clean":"2024-02-23T00:39:44.178942+0000","last_became_active":"2024-02-23T00:38:49.224244+0000","last_became_peered":"2024-02-23T00:38:49.224244+0000","last_unstale":"2024-02-23T00:39:44.178942+0000","last_undegraded":"2024-02-23T00:39:44.178942+0000","last_fullsized":"2024-02-23T00:39:44.178942+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.315765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.315765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":491,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.183098+0000","last_change":"2024-02-23T00:39:28.509774+0000","last_active":"2024-02-23T00:39:44.183098+0000","last_peered":"2024-02-23T00:39:44.183098+0000","last_clean":"2024-02-23T00:39:44.183098+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:39:44.183098+0000","last_undegraded":"2024-02-23T00:39:44.183098+0000","last_fullsized":"2024-02-23T00:39:44.183098+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:28.509675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.509675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":552,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181957+0000","last_change":"2024-02-23T00:39:39.963503+0000","last_active":"2024-02-23T00:39:44.181957+0000","last_peered":"2024-02-23T00:39:44.181957+0000","last_clean":"2024-02-23T00:39:44.181957+0000","last_became_active":"2024-02-23T00:35:27.093541+0000","last_became_peered":"2024-02-23T00:35:27.093541+0000","last_unstale":"2024-02-23T00:39:44.181957+0000","last_undegraded":"2024-02-23T00:39:44.181957+0000","last_fullsized":"2024-02-23T00:39:44.181957+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.963439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":582,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.180910+0000","last_change":"2024-02-23T00:38:03.327200+0000","last_active":"2024-02-23T00:39:44.180910+0000","last_peered":"2024-02-23T00:39:44.180910+0000","last_clean":"2024-02-23T00:39:44.180910+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:39:44.180910+0000","last_undegraded":"2024-02-23T00:39:44.180910+0000","last_fullsized":"2024-02-23T00:39:44.180910+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:03.327127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.327127+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":546,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.187764+0000","last_change":"2024-02-23T00:36:37.106377+0000","last_active":"2024-02-23T00:39:44.187764+0000","last_peered":"2024-02-23T00:39:44.187764+0000","last_clean":"2024-02-23T00:36:33.365867+0000","last_became_active":"2024-02-23T00:36:37.106377+0000","last_became_peered":"2024-02-23T00:36:37.106377+0000","last_unstale":"2024-02-23T00:39:44.187764+0000","last_undegraded":"2024-02-23T00:39:44.187764+0000","last_fullsized":"2024-02-23T00:36:37.100069+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":538,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181413+0000","last_change":"2024-02-23T00:39:40.690606+0000","last_active":"2024-02-23T00:39:44.181413+0000","last_peered":"2024-02-23T00:39:44.181413+0000","last_clean":"2024-02-23T00:39:44.181413+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:39:44.181413+0000","last_undegraded":"2024-02-23T00:39:44.181413+0000","last_fullsized":"2024-02-23T00:39:44.181413+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.690534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.690534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.196330+0000","last_change":"2024-02-23T00:38:49.226513+0000","last_active":"2024-02-23T00:39:44.196330+0000","last_peered":"2024-02-23T00:39:44.196330+0000","last_clean":"2024-02-23T00:39:44.196330+0000","last_became_active":"2024-02-23T00:38:49.226188+0000","last_became_peered":"2024-02-23T00:38:49.226188+0000","last_unstale":"2024-02-23T00:39:44.196330+0000","last_undegraded":"2024-02-23T00:39:44.196330+0000","last_fullsized":"2024-02-23T00:39:44.196330+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.933288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.933288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":545,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.186056+0000","last_change":"2024-02-23T00:38:49.184762+0000","last_active":"2024-02-23T00:39:44.186056+0000","last_peered":"2024-02-23T00:39:44.186056+0000","last_clean":"2024-02-23T00:39:44.186056+0000","last_became_active":"2024-02-23T00:38:49.184274+0000","last_became_peered":"2024-02-23T00:38:49.184274+0000","last_unstale":"2024-02-23T00:39:44.186056+0000","last_undegraded":"2024-02-23T00:39:44.186056+0000","last_fullsized":"2024-02-23T00:39:44.186056+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.008726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":493,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181247+0000","last_change":"2024-02-23T00:38:44.173181+0000","last_active":"2024-02-23T00:39:44.181247+0000","last_peered":"2024-02-23T00:39:44.181247+0000","last_clean":"2024-02-23T00:39:44.181247+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:39:44.181247+0000","last_undegraded":"2024-02-23T00:39:44.181247+0000","last_fullsized":"2024-02-23T00:39:44.181247+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:44.173077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.173077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":560,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.192349+0000","last_change":"2024-02-23T00:39:30.413946+0000","last_active":"2024-02-23T00:39:44.192349+0000","last_peered":"2024-02-23T00:39:44.192349+0000","last_clean":"2024-02-23T00:39:44.192349+0000","last_became_active":"2024-02-23T00:35:27.097186+0000","last_became_peered":"2024-02-23T00:35:27.097186+0000","last_unstale":"2024-02-23T00:39:44.192349+0000","last_undegraded":"2024-02-23T00:39:44.192349+0000","last_fullsized":"2024-02-23T00:39:44.192349+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.413877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.413877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":621,"reported_epoch":565,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:44.179823+0000","last_change":"2024-02-23T00:38:19.164063+0000","last_active":"2024-02-23T00:39:44.179823+0000","last_peered":"2024-02-23T00:39:44.179823+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:38:19.164063+0000","last_became_peered":"2024-02-23T00:38:19.164063+0000","last_unstale":"2024-02-23T00:39:44.179823+0000","last_undegraded":"2024-02-23T00:38:19.153386+0000","last_fullsized":"2024-02-23T00:38:19.153249+0000","mapping_epoch":425,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":575,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.188895+0000","last_change":"2024-02-23T00:39:22.437200+0000","last_active":"2024-02-23T00:39:44.188895+0000","last_peered":"2024-02-23T00:39:44.188895+0000","last_clean":"2024-02-23T00:39:44.188895+0000","last_became_active":"2024-02-23T00:38:50.196782+0000","last_became_peered":"2024-02-23T00:38:50.196782+0000","last_unstale":"2024-02-23T00:39:44.188895+0000","last_undegraded":"2024-02-23T00:39:44.188895+0000","last_fullsized":"2024-02-23T00:39:44.188895+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:22.437104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.437104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,1],"avail_no_missing":["2","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":791,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.177735+0000","last_change":"2024-02-23T00:37:13.467508+0000","last_active":"2024-02-23T00:39:44.177735+0000","last_peered":"2024-02-23T00:39:44.177735+0000","last_clean":"2024-02-23T00:39:44.177735+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:39:44.177735+0000","last_undegraded":"2024-02-23T00:39:44.177735+0000","last_fullsized":"2024-02-23T00:39:44.177735+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:37:13.467428+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:37:13.467428+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":554,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187627+0000","last_change":"2024-02-23T00:38:49.196976+0000","last_active":"2024-02-23T00:39:44.187627+0000","last_peered":"2024-02-23T00:39:44.187627+0000","last_clean":"2024-02-23T00:39:44.187627+0000","last_became_active":"2024-02-23T00:38:49.195404+0000","last_became_peered":"2024-02-23T00:38:49.195404+0000","last_unstale":"2024-02-23T00:39:44.187627+0000","last_undegraded":"2024-02-23T00:39:44.187627+0000","last_fullsized":"2024-02-23T00:39:44.187627+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:12.023125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:12.023125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":497,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187725+0000","last_change":"2024-02-23T00:38:40.265472+0000","last_active":"2024-02-23T00:39:44.187725+0000","last_peered":"2024-02-23T00:39:44.187725+0000","last_clean":"2024-02-23T00:39:44.187725+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:39:44.187725+0000","last_undegraded":"2024-02-23T00:39:44.187725+0000","last_fullsized":"2024-02-23T00:39:44.187725+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:40.265385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.265385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":554,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.179638+0000","last_change":"2024-02-23T00:36:37.101373+0000","last_active":"2024-02-23T00:39:44.179638+0000","last_peered":"2024-02-23T00:39:44.179638+0000","last_clean":"2024-02-23T00:36:33.192308+0000","last_became_active":"2024-02-23T00:36:37.101373+0000","last_became_peered":"2024-02-23T00:36:37.101373+0000","last_unstale":"2024-02-23T00:39:44.179638+0000","last_undegraded":"2024-02-23T00:39:44.179638+0000","last_fullsized":"2024-02-23T00:36:37.094754+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.595447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.595447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":494,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.179842+0000","last_change":"2024-02-23T00:36:37.109549+0000","last_active":"2024-02-23T00:39:44.179842+0000","last_peered":"2024-02-23T00:39:44.179842+0000","last_clean":"2024-02-23T00:35:25.318841+0000","last_became_active":"2024-02-23T00:36:37.109549+0000","last_became_peered":"2024-02-23T00:36:37.109549+0000","last_unstale":"2024-02-23T00:39:44.179842+0000","last_undegraded":"2024-02-23T00:39:44.179842+0000","last_fullsized":"2024-02-23T00:36:37.096473+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":830,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.189809+0000","last_change":"2024-02-23T00:38:51.396520+0000","last_active":"2024-02-23T00:39:45.189809+0000","last_peered":"2024-02-23T00:39:45.189809+0000","last_clean":"2024-02-23T00:39:45.189809+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:39:45.189809+0000","last_undegraded":"2024-02-23T00:39:45.189809+0000","last_fullsized":"2024-02-23T00:39:45.189809+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'35","last_scrub_stamp":"2024-02-23T00:37:15.422435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:15.422435+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"55'1","reported_seq":173,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179154+0000","last_change":"2024-02-23T00:37:23.360353+0000","last_active":"2024-02-23T00:39:44.179154+0000","last_peered":"2024-02-23T00:39:44.179154+0000","last_clean":"2024-02-23T00:39:44.179154+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:39:44.179154+0000","last_undegraded":"2024-02-23T00:39:44.179154+0000","last_fullsized":"2024-02-23T00:39:44.179154+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:37:23.360230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:23.360230+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":69,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.207111+0000","last_change":"2024-02-23T00:39:40.370151+0000","last_active":"2024-02-23T00:39:44.207111+0000","last_peered":"2024-02-23T00:39:44.207111+0000","last_clean":"2024-02-23T00:39:44.207111+0000","last_became_active":"2024-02-23T00:38:49.189587+0000","last_became_peered":"2024-02-23T00:38:49.189587+0000","last_unstale":"2024-02-23T00:39:44.207111+0000","last_undegraded":"2024-02-23T00:39:44.207111+0000","last_fullsized":"2024-02-23T00:39:44.207111+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.370090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.370090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":497,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.192585+0000","last_change":"2024-02-23T00:38:43.332992+0000","last_active":"2024-02-23T00:39:44.192585+0000","last_peered":"2024-02-23T00:39:44.192585+0000","last_clean":"2024-02-23T00:39:44.192585+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:39:44.192585+0000","last_undegraded":"2024-02-23T00:39:44.192585+0000","last_fullsized":"2024-02-23T00:39:44.192585+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:43.332907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:43.332907+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":466,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.184305+0000","last_change":"2024-02-23T00:39:32.809209+0000","last_active":"2024-02-23T00:39:44.184305+0000","last_peered":"2024-02-23T00:39:44.184305+0000","last_clean":"2024-02-23T00:39:44.184305+0000","last_became_active":"2024-02-23T00:38:49.183777+0000","last_became_peered":"2024-02-23T00:38:49.183777+0000","last_unstale":"2024-02-23T00:39:44.184305+0000","last_undegraded":"2024-02-23T00:39:44.184305+0000","last_fullsized":"2024-02-23T00:39:44.184305+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:32.809133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:32.809133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":168,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.185627+0000","last_change":"2024-02-23T00:36:37.104760+0000","last_active":"2024-02-23T00:39:44.185627+0000","last_peered":"2024-02-23T00:39:44.185627+0000","last_clean":"2024-02-23T00:27:24.172824+0000","last_became_active":"2024-02-23T00:36:37.104760+0000","last_became_peered":"2024-02-23T00:36:37.104760+0000","last_unstale":"2024-02-23T00:39:44.185627+0000","last_undegraded":"2024-02-23T00:39:44.185627+0000","last_fullsized":"2024-02-23T00:36:37.098375+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":206,"reported_epoch":565,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:44.183057+0000","last_change":"2024-02-23T00:36:37.110364+0000","last_active":"2024-02-23T00:39:44.183057+0000","last_peered":"2024-02-23T00:39:44.183057+0000","last_clean":"2024-02-23T00:27:29.173100+0000","last_became_active":"2024-02-23T00:36:37.110364+0000","last_became_peered":"2024-02-23T00:36:37.110364+0000","last_unstale":"2024-02-23T00:39:44.183057+0000","last_undegraded":"2024-02-23T00:36:37.096575+0000","last_fullsized":"2024-02-23T00:36:37.096459+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:35:44.214727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:44.214727+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":539,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.178603+0000","last_change":"2024-02-23T00:36:37.123256+0000","last_active":"2024-02-23T00:39:44.178603+0000","last_peered":"2024-02-23T00:39:44.178603+0000","last_clean":"2024-02-23T00:36:33.366566+0000","last_became_active":"2024-02-23T00:36:37.123256+0000","last_became_peered":"2024-02-23T00:36:37.123256+0000","last_unstale":"2024-02-23T00:39:44.178603+0000","last_undegraded":"2024-02-23T00:39:44.178603+0000","last_fullsized":"2024-02-23T00:36:37.096440+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":70,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.204955+0000","last_change":"2024-02-23T00:39:21.371091+0000","last_active":"2024-02-23T00:39:44.204955+0000","last_peered":"2024-02-23T00:39:44.204955+0000","last_clean":"2024-02-23T00:39:44.204955+0000","last_became_active":"2024-02-23T00:38:50.177038+0000","last_became_peered":"2024-02-23T00:38:50.177038+0000","last_unstale":"2024-02-23T00:39:44.204955+0000","last_undegraded":"2024-02-23T00:39:44.204955+0000","last_fullsized":"2024-02-23T00:39:44.204955+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:21.370914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:21.370914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,5],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":491,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.183244+0000","last_change":"2024-02-23T00:39:30.473677+0000","last_active":"2024-02-23T00:39:44.183244+0000","last_peered":"2024-02-23T00:39:44.183244+0000","last_clean":"2024-02-23T00:39:44.183244+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:39:44.183244+0000","last_undegraded":"2024-02-23T00:39:44.183244+0000","last_fullsized":"2024-02-23T00:39:44.183244+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.473602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.473602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":551,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.181269+0000","last_change":"2024-02-23T00:36:37.124382+0000","last_active":"2024-02-23T00:39:44.181269+0000","last_peered":"2024-02-23T00:39:44.181269+0000","last_clean":"2024-02-23T00:36:33.193431+0000","last_became_active":"2024-02-23T00:36:37.124382+0000","last_became_peered":"2024-02-23T00:36:37.124382+0000","last_unstale":"2024-02-23T00:39:44.181269+0000","last_undegraded":"2024-02-23T00:39:44.181269+0000","last_fullsized":"2024-02-23T00:36:37.101474+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:24.016837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:24.016837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":758,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179264+0000","last_change":"2024-02-23T00:38:54.171994+0000","last_active":"2024-02-23T00:39:44.179264+0000","last_peered":"2024-02-23T00:39:44.179264+0000","last_clean":"2024-02-23T00:39:44.179264+0000","last_became_active":"2024-02-23T00:38:54.171563+0000","last_became_peered":"2024-02-23T00:38:54.171563+0000","last_unstale":"2024-02-23T00:39:44.179264+0000","last_undegraded":"2024-02-23T00:39:44.179264+0000","last_fullsized":"2024-02-23T00:39:44.179264+0000","mapping_epoch":516,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:06.122240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.122240+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":558,"reported_epoch":565,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:44.181400+0000","last_change":"2024-02-23T00:36:37.118733+0000","last_active":"2024-02-23T00:39:44.181400+0000","last_peered":"2024-02-23T00:39:44.181400+0000","last_clean":"2024-02-23T00:36:36.130983+0000","last_became_active":"2024-02-23T00:36:37.118733+0000","last_became_peered":"2024-02-23T00:36:37.118733+0000","last_unstale":"2024-02-23T00:39:44.181400+0000","last_undegraded":"2024-02-23T00:36:37.095672+0000","last_fullsized":"2024-02-23T00:36:37.095564+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:36:36.130868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.130868+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.197215+0000","last_change":"2024-02-23T00:38:49.220203+0000","last_active":"2024-02-23T00:39:44.197215+0000","last_peered":"2024-02-23T00:39:44.197215+0000","last_clean":"2024-02-23T00:39:44.197215+0000","last_became_active":"2024-02-23T00:38:49.219745+0000","last_became_peered":"2024-02-23T00:38:49.219745+0000","last_unstale":"2024-02-23T00:39:44.197215+0000","last_undegraded":"2024-02-23T00:39:44.197215+0000","last_fullsized":"2024-02-23T00:39:44.197215+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.172059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.172059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":482,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.180700+0000","last_change":"2024-02-23T00:36:28.184266+0000","last_active":"2024-02-23T00:39:44.180700+0000","last_peered":"2024-02-23T00:39:44.180700+0000","last_clean":"2024-02-23T00:39:44.180700+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:39:44.180700+0000","last_undegraded":"2024-02-23T00:39:44.180700+0000","last_fullsized":"2024-02-23T00:39:44.180700+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:28.184197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:28.184197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":257,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.206608+0000","last_change":"2024-02-23T00:38:54.181474+0000","last_active":"2024-02-23T00:39:44.206608+0000","last_peered":"2024-02-23T00:39:44.206608+0000","last_clean":"2024-02-23T00:39:44.206608+0000","last_became_active":"2024-02-23T00:38:54.180971+0000","last_became_peered":"2024-02-23T00:38:54.180971+0000","last_unstale":"2024-02-23T00:39:44.206608+0000","last_undegraded":"2024-02-23T00:39:44.206608+0000","last_fullsized":"2024-02-23T00:39:44.206608+0000","mapping_epoch":516,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":546,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.181144+0000","last_change":"2024-02-23T00:36:37.111836+0000","last_active":"2024-02-23T00:39:44.181144+0000","last_peered":"2024-02-23T00:39:44.181144+0000","last_clean":"2024-02-23T00:36:33.193391+0000","last_became_active":"2024-02-23T00:36:37.111836+0000","last_became_peered":"2024-02-23T00:36:37.111836+0000","last_unstale":"2024-02-23T00:39:44.181144+0000","last_undegraded":"2024-02-23T00:39:44.181144+0000","last_fullsized":"2024-02-23T00:36:37.101173+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.667517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.667517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":69,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.207352+0000","last_change":"2024-02-23T00:39:18.369649+0000","last_active":"2024-02-23T00:39:44.207352+0000","last_peered":"2024-02-23T00:39:44.207352+0000","last_clean":"2024-02-23T00:39:44.207352+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:39:44.207352+0000","last_undegraded":"2024-02-23T00:39:44.207352+0000","last_fullsized":"2024-02-23T00:39:44.207352+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:18.369579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:18.369579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":551,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187992+0000","last_change":"2024-02-23T00:38:42.293920+0000","last_active":"2024-02-23T00:39:44.187992+0000","last_peered":"2024-02-23T00:39:44.187992+0000","last_clean":"2024-02-23T00:39:44.187992+0000","last_became_active":"2024-02-23T00:27:23.022032+0000","last_became_peered":"2024-02-23T00:27:23.022032+0000","last_unstale":"2024-02-23T00:39:44.187992+0000","last_undegraded":"2024-02-23T00:39:44.187992+0000","last_fullsized":"2024-02-23T00:39:44.187992+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.293824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.293824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.198108+0000","last_change":"2024-02-23T00:38:49.230851+0000","last_active":"2024-02-23T00:39:44.198108+0000","last_peered":"2024-02-23T00:39:44.198108+0000","last_clean":"2024-02-23T00:39:44.198108+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:39:44.198108+0000","last_undegraded":"2024-02-23T00:39:44.198108+0000","last_fullsized":"2024-02-23T00:39:44.198108+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:39.893315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.893315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":645,"reported_epoch":565,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-23T00:39:44.179392+0000","last_change":"2024-02-23T00:39:19.250055+0000","last_active":"2024-02-23T00:39:44.179392+0000","last_peered":"2024-02-23T00:39:44.179392+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:38:49.224732+0000","last_became_peered":"2024-02-23T00:38:49.224732+0000","last_unstale":"2024-02-23T00:39:44.179392+0000","last_undegraded":"2024-02-23T00:38:49.159472+0000","last_fullsized":"2024-02-23T00:38:49.159180+0000","mapping_epoch":511,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":544,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187026+0000","last_change":"2024-02-23T00:38:42.431748+0000","last_active":"2024-02-23T00:39:44.187026+0000","last_peered":"2024-02-23T00:39:44.187026+0000","last_clean":"2024-02-23T00:39:44.187026+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:39:44.187026+0000","last_undegraded":"2024-02-23T00:39:44.187026+0000","last_fullsized":"2024-02-23T00:39:44.187026+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.431641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":547,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.184746+0000","last_change":"2024-02-23T00:39:28.781689+0000","last_active":"2024-02-23T00:39:44.184746+0000","last_peered":"2024-02-23T00:39:44.184746+0000","last_clean":"2024-02-23T00:39:44.184746+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:39:44.184746+0000","last_undegraded":"2024-02-23T00:39:44.184746+0000","last_fullsized":"2024-02-23T00:39:44.184746+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:28.781587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.781587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":536,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187373+0000","last_change":"2024-02-23T00:37:16.011330+0000","last_active":"2024-02-23T00:39:44.187373+0000","last_peered":"2024-02-23T00:39:44.187373+0000","last_clean":"2024-02-23T00:39:44.187373+0000","last_became_active":"2024-02-23T00:27:23.182136+0000","last_became_peered":"2024-02-23T00:27:23.182136+0000","last_unstale":"2024-02-23T00:39:44.187373+0000","last_undegraded":"2024-02-23T00:39:44.187373+0000","last_fullsized":"2024-02-23T00:39:44.187373+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:16.011248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:58.383642+0000","last_clean_scrub_stamp":"2024-02-23T00:37:16.011248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":509,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181168+0000","last_change":"2024-02-23T00:39:29.410514+0000","last_active":"2024-02-23T00:39:44.181168+0000","last_peered":"2024-02-23T00:39:44.181168+0000","last_clean":"2024-02-23T00:39:44.181168+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:39:44.181168+0000","last_undegraded":"2024-02-23T00:39:44.181168+0000","last_fullsized":"2024-02-23T00:39:44.181168+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:29.410456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:29.410456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"453'32","reported_seq":80,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.197919+0000","last_change":"2024-02-23T00:38:51.793947+0000","last_active":"2024-02-23T00:39:44.197919+0000","last_peered":"2024-02-23T00:39:44.197919+0000","last_clean":"2024-02-23T00:39:44.197919+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:39:44.197919+0000","last_undegraded":"2024-02-23T00:39:44.197919+0000","last_fullsized":"2024-02-23T00:39:44.197919+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:38:44.240437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.240437+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":564,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.182801+0000","last_change":"2024-02-23T00:39:23.833122+0000","last_active":"2024-02-23T00:39:44.182801+0000","last_peered":"2024-02-23T00:39:44.182801+0000","last_clean":"2024-02-23T00:39:44.182801+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:39:44.182801+0000","last_undegraded":"2024-02-23T00:39:44.182801+0000","last_fullsized":"2024-02-23T00:39:44.182801+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:23.833059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:39:23.833059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"426'10","reported_seq":690,"reported_epoch":566,"state":"active+undersized+degraded","last_fresh":"2024-02-23T00:39:45.186595+0000","last_change":"2024-02-23T00:36:37.113534+0000","last_active":"2024-02-23T00:39:45.186595+0000","last_peered":"2024-02-23T00:39:45.186595+0000","last_clean":"2024-02-23T00:36:35.444798+0000","last_became_active":"2024-02-23T00:36:37.113534+0000","last_became_peered":"2024-02-23T00:36:37.113534+0000","last_unstale":"2024-02-23T00:39:45.186595+0000","last_undegraded":"2024-02-23T00:36:37.102235+0000","last_fullsized":"2024-02-23T00:36:37.102132+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.207184+0000","last_change":"2024-02-23T00:38:49.190032+0000","last_active":"2024-02-23T00:39:44.207184+0000","last_peered":"2024-02-23T00:39:44.207184+0000","last_clean":"2024-02-23T00:39:44.207184+0000","last_became_active":"2024-02-23T00:38:49.189564+0000","last_became_peered":"2024-02-23T00:38:49.189564+0000","last_unstale":"2024-02-23T00:39:44.207184+0000","last_undegraded":"2024-02-23T00:39:44.207184+0000","last_fullsized":"2024-02-23T00:39:44.207184+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:41.177673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:38:41.177673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":482,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.185057+0000","last_change":"2024-02-23T00:37:08.058170+0000","last_active":"2024-02-23T00:39:44.185057+0000","last_peered":"2024-02-23T00:39:44.185057+0000","last_clean":"2024-02-23T00:39:44.185057+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:39:44.185057+0000","last_undegraded":"2024-02-23T00:39:44.185057+0000","last_fullsized":"2024-02-23T00:39:44.185057+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:08.058075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:08.058075+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":618,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.184414+0000","last_change":"2024-02-23T00:38:49.346222+0000","last_active":"2024-02-23T00:39:44.184414+0000","last_peered":"2024-02-23T00:39:44.184414+0000","last_clean":"2024-02-23T00:39:44.184414+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:39:44.184414+0000","last_undegraded":"2024-02-23T00:39:44.184414+0000","last_fullsized":"2024-02-23T00:39:44.184414+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:40.858800+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.858800+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":467,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.180330+0000","last_change":"2024-02-23T00:39:26.484091+0000","last_active":"2024-02-23T00:39:44.180330+0000","last_peered":"2024-02-23T00:39:44.180330+0000","last_clean":"2024-02-23T00:39:44.180330+0000","last_became_active":"2024-02-23T00:38:50.180740+0000","last_became_peered":"2024-02-23T00:38:50.180740+0000","last_unstale":"2024-02-23T00:39:44.180330+0000","last_undegraded":"2024-02-23T00:39:44.180330+0000","last_fullsized":"2024-02-23T00:39:44.180330+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:26.483929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:26.483929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,7],"avail_no_missing":["4","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":80,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.182003+0000","last_change":"2024-02-23T00:38:49.384331+0000","last_active":"2024-02-23T00:39:45.182003+0000","last_peered":"2024-02-23T00:39:45.182003+0000","last_clean":"2024-02-23T00:39:45.182003+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:39:45.182003+0000","last_undegraded":"2024-02-23T00:39:45.182003+0000","last_fullsized":"2024-02-23T00:39:45.182003+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'10","last_scrub_stamp":"2024-02-23T00:37:27.870653+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:37:27.870653+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":85,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.185620+0000","last_change":"2024-02-23T00:38:49.184581+0000","last_active":"2024-02-23T00:39:44.185620+0000","last_peered":"2024-02-23T00:39:44.185620+0000","last_clean":"2024-02-23T00:39:44.185620+0000","last_became_active":"2024-02-23T00:38:49.183777+0000","last_became_peered":"2024-02-23T00:38:49.183777+0000","last_unstale":"2024-02-23T00:39:44.185620+0000","last_undegraded":"2024-02-23T00:39:44.185620+0000","last_fullsized":"2024-02-23T00:39:44.185620+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:05.095708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:05.095708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.208401+0000","last_change":"2024-02-23T00:38:49.227700+0000","last_active":"2024-02-23T00:39:44.208401+0000","last_peered":"2024-02-23T00:39:44.208401+0000","last_clean":"2024-02-23T00:39:44.208401+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:39:44.208401+0000","last_undegraded":"2024-02-23T00:39:44.208401+0000","last_fullsized":"2024-02-23T00:39:44.208401+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:09.447326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.447326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":558,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179424+0000","last_change":"2024-02-23T00:37:20.382193+0000","last_active":"2024-02-23T00:39:44.179424+0000","last_peered":"2024-02-23T00:39:44.179424+0000","last_clean":"2024-02-23T00:39:44.179424+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:39:44.179424+0000","last_undegraded":"2024-02-23T00:39:44.179424+0000","last_fullsized":"2024-02-23T00:39:44.179424+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:37:20.382108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.382108+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":562,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.180672+0000","last_change":"2024-02-23T00:38:49.195750+0000","last_active":"2024-02-23T00:39:44.180672+0000","last_peered":"2024-02-23T00:39:44.180672+0000","last_clean":"2024-02-23T00:39:44.180672+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:39:44.180672+0000","last_undegraded":"2024-02-23T00:39:44.180672+0000","last_fullsized":"2024-02-23T00:39:44.180672+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.470333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.470333+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":70,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.196963+0000","last_change":"2024-02-23T00:39:30.370724+0000","last_active":"2024-02-23T00:39:44.196963+0000","last_peered":"2024-02-23T00:39:44.196963+0000","last_clean":"2024-02-23T00:39:44.196963+0000","last_became_active":"2024-02-23T00:38:50.199759+0000","last_became_peered":"2024-02-23T00:38:50.199759+0000","last_unstale":"2024-02-23T00:39:44.196963+0000","last_undegraded":"2024-02-23T00:39:44.196963+0000","last_fullsized":"2024-02-23T00:39:44.196963+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.370620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.370620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":148,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.179967+0000","last_change":"2024-02-23T00:36:37.112999+0000","last_active":"2024-02-23T00:39:44.179967+0000","last_peered":"2024-02-23T00:39:44.179967+0000","last_clean":"2024-02-23T00:27:22.009070+0000","last_became_active":"2024-02-23T00:36:37.112999+0000","last_became_peered":"2024-02-23T00:36:37.112999+0000","last_unstale":"2024-02-23T00:39:44.179967+0000","last_undegraded":"2024-02-23T00:39:44.179967+0000","last_fullsized":"2024-02-23T00:36:37.102073+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:04.413991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:35:04.413991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":492,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.179206+0000","last_change":"2024-02-23T00:36:37.097581+0000","last_active":"2024-02-23T00:39:44.179206+0000","last_peered":"2024-02-23T00:39:44.179206+0000","last_clean":"2024-02-23T00:36:33.191411+0000","last_became_active":"2024-02-23T00:36:37.097581+0000","last_became_peered":"2024-02-23T00:36:37.097581+0000","last_unstale":"2024-02-23T00:39:44.179206+0000","last_undegraded":"2024-02-23T00:39:44.179206+0000","last_fullsized":"2024-02-23T00:36:37.093707+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.101501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.101501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":594,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.180241+0000","last_change":"2024-02-23T00:38:16.174640+0000","last_active":"2024-02-23T00:39:44.180241+0000","last_peered":"2024-02-23T00:39:44.180241+0000","last_clean":"2024-02-23T00:39:44.180241+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:39:44.180241+0000","last_undegraded":"2024-02-23T00:39:44.180241+0000","last_fullsized":"2024-02-23T00:39:44.180241+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:38:16.174526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.174526+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":536,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.179979+0000","last_change":"2024-02-23T00:36:37.108194+0000","last_active":"2024-02-23T00:39:44.179979+0000","last_peered":"2024-02-23T00:39:44.179979+0000","last_clean":"2024-02-23T00:36:33.367482+0000","last_became_active":"2024-02-23T00:36:37.108194+0000","last_became_peered":"2024-02-23T00:36:37.108194+0000","last_unstale":"2024-02-23T00:39:44.179979+0000","last_undegraded":"2024-02-23T00:39:44.179979+0000","last_fullsized":"2024-02-23T00:36:37.096730+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.783089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.783089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":549,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179501+0000","last_change":"2024-02-23T00:38:49.225280+0000","last_active":"2024-02-23T00:39:44.179501+0000","last_peered":"2024-02-23T00:39:44.179501+0000","last_clean":"2024-02-23T00:39:44.179501+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:39:44.179501+0000","last_undegraded":"2024-02-23T00:39:44.179501+0000","last_fullsized":"2024-02-23T00:39:44.179501+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:06.235608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.235608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":70,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.194628+0000","last_change":"2024-02-23T00:39:25.312118+0000","last_active":"2024-02-23T00:39:44.194628+0000","last_peered":"2024-02-23T00:39:44.194628+0000","last_clean":"2024-02-23T00:39:44.194628+0000","last_became_active":"2024-02-23T00:38:50.179247+0000","last_became_peered":"2024-02-23T00:38:50.179247+0000","last_unstale":"2024-02-23T00:39:44.194628+0000","last_undegraded":"2024-02-23T00:39:44.194628+0000","last_fullsized":"2024-02-23T00:39:44.194628+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:25.311952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:39:25.311952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,2],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":535,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187332+0000","last_change":"2024-02-23T00:39:22.387191+0000","last_active":"2024-02-23T00:39:44.187332+0000","last_peered":"2024-02-23T00:39:44.187332+0000","last_clean":"2024-02-23T00:39:44.187332+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:39:44.187332+0000","last_undegraded":"2024-02-23T00:39:44.187332+0000","last_fullsized":"2024-02-23T00:39:44.187332+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:39:22.387089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.387089+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":70,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.205412+0000","last_change":"2024-02-23T00:38:49.987321+0000","last_active":"2024-02-23T00:39:44.205412+0000","last_peered":"2024-02-23T00:39:44.205412+0000","last_clean":"2024-02-23T00:39:44.205412+0000","last_became_active":"2024-02-23T00:38:49.217612+0000","last_became_peered":"2024-02-23T00:38:49.217612+0000","last_unstale":"2024-02-23T00:39:44.205412+0000","last_undegraded":"2024-02-23T00:39:44.205412+0000","last_fullsized":"2024-02-23T00:39:44.205412+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-23T00:36:36.531424+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.531424+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":533,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.184907+0000","last_change":"2024-02-23T00:36:32.199878+0000","last_active":"2024-02-23T00:39:44.184907+0000","last_peered":"2024-02-23T00:39:44.184907+0000","last_clean":"2024-02-23T00:39:44.184907+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:39:44.184907+0000","last_undegraded":"2024-02-23T00:39:44.184907+0000","last_fullsized":"2024-02-23T00:39:44.184907+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:32.199765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:32.199765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":535,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.186940+0000","last_change":"2024-02-23T00:38:02.325317+0000","last_active":"2024-02-23T00:39:44.186940+0000","last_peered":"2024-02-23T00:39:44.186940+0000","last_clean":"2024-02-23T00:39:44.186940+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:39:44.186940+0000","last_undegraded":"2024-02-23T00:39:44.186940+0000","last_fullsized":"2024-02-23T00:39:44.186940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:02.325254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:02.325254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":536,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.186361+0000","last_change":"2024-02-23T00:37:14.009706+0000","last_active":"2024-02-23T00:39:44.186361+0000","last_peered":"2024-02-23T00:39:44.186361+0000","last_clean":"2024-02-23T00:39:44.186361+0000","last_became_active":"2024-02-23T00:27:23.030112+0000","last_became_peered":"2024-02-23T00:27:23.030112+0000","last_unstale":"2024-02-23T00:39:44.186361+0000","last_undegraded":"2024-02-23T00:39:44.186361+0000","last_fullsized":"2024-02-23T00:39:44.186361+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.009614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.009614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":482,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179578+0000","last_change":"2024-02-23T00:38:33.242488+0000","last_active":"2024-02-23T00:39:44.179578+0000","last_peered":"2024-02-23T00:39:44.179578+0000","last_clean":"2024-02-23T00:39:44.179578+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:39:44.179578+0000","last_undegraded":"2024-02-23T00:39:44.179578+0000","last_fullsized":"2024-02-23T00:39:44.179578+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:33.242417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:33.242417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":81,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.194758+0000","last_change":"2024-02-23T00:38:51.365185+0000","last_active":"2024-02-23T00:39:44.194758+0000","last_peered":"2024-02-23T00:39:44.194758+0000","last_clean":"2024-02-23T00:39:44.194758+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:39:44.194758+0000","last_undegraded":"2024-02-23T00:39:44.194758+0000","last_fullsized":"2024-02-23T00:39:44.194758+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:37:26.914698+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.914698+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":543,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.181829+0000","last_change":"2024-02-23T00:39:26.456134+0000","last_active":"2024-02-23T00:39:44.181829+0000","last_peered":"2024-02-23T00:39:44.181829+0000","last_clean":"2024-02-23T00:39:44.181829+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:39:44.181829+0000","last_undegraded":"2024-02-23T00:39:44.181829+0000","last_fullsized":"2024-02-23T00:39:44.181829+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:26.456074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:26.456074+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":538,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187990+0000","last_change":"2024-02-23T00:38:16.055268+0000","last_active":"2024-02-23T00:39:44.187990+0000","last_peered":"2024-02-23T00:39:44.187990+0000","last_clean":"2024-02-23T00:39:44.187990+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:39:44.187990+0000","last_undegraded":"2024-02-23T00:39:44.187990+0000","last_fullsized":"2024-02-23T00:39:44.187990+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.055159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":70,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.196767+0000","last_change":"2024-02-23T00:39:39.381543+0000","last_active":"2024-02-23T00:39:44.196767+0000","last_peered":"2024-02-23T00:39:44.196767+0000","last_clean":"2024-02-23T00:39:44.196767+0000","last_became_active":"2024-02-23T00:38:50.184585+0000","last_became_peered":"2024-02-23T00:38:50.184585+0000","last_unstale":"2024-02-23T00:39:44.196767+0000","last_undegraded":"2024-02-23T00:39:44.196767+0000","last_fullsized":"2024-02-23T00:39:44.196767+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.381447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.381447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.196890+0000","last_change":"2024-02-23T00:38:49.189171+0000","last_active":"2024-02-23T00:39:44.196890+0000","last_peered":"2024-02-23T00:39:44.196890+0000","last_clean":"2024-02-23T00:39:44.196890+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:39:44.196890+0000","last_undegraded":"2024-02-23T00:39:44.196890+0000","last_fullsized":"2024-02-23T00:39:44.196890+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:37.979785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.979785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":659,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.183711+0000","last_change":"2024-02-23T00:38:54.170696+0000","last_active":"2024-02-23T00:39:44.183711+0000","last_peered":"2024-02-23T00:39:44.183711+0000","last_clean":"2024-02-23T00:39:44.183711+0000","last_became_active":"2024-02-23T00:38:54.170041+0000","last_became_peered":"2024-02-23T00:38:54.170041+0000","last_unstale":"2024-02-23T00:39:44.183711+0000","last_undegraded":"2024-02-23T00:39:44.183711+0000","last_fullsized":"2024-02-23T00:39:44.183711+0000","mapping_epoch":516,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,1],"avail_no_missing":["7","1","3"],"object_location_counts":[{"shards":"1,3,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":542,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179787+0000","last_change":"2024-02-23T00:38:36.853659+0000","last_active":"2024-02-23T00:39:44.179787+0000","last_peered":"2024-02-23T00:39:44.179787+0000","last_clean":"2024-02-23T00:39:44.179787+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:39:44.179787+0000","last_undegraded":"2024-02-23T00:39:44.179787+0000","last_fullsized":"2024-02-23T00:39:44.179787+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:36.853582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:36.853582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":701,"reported_epoch":566,"state":"active+clean","last_fresh":"2024-02-23T00:39:45.184641+0000","last_change":"2024-02-23T00:38:37.455385+0000","last_active":"2024-02-23T00:39:45.184641+0000","last_peered":"2024-02-23T00:39:45.184641+0000","last_clean":"2024-02-23T00:39:45.184641+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:39:45.184641+0000","last_undegraded":"2024-02-23T00:39:45.184641+0000","last_fullsized":"2024-02-23T00:39:45.184641+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:38:37.455280+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.455280+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"78'2","reported_seq":68,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.198234+0000","last_change":"2024-02-23T00:38:51.364248+0000","last_active":"2024-02-23T00:39:44.198234+0000","last_peered":"2024-02-23T00:39:44.198234+0000","last_clean":"2024-02-23T00:39:44.198234+0000","last_became_active":"2024-02-23T00:38:49.230008+0000","last_became_peered":"2024-02-23T00:38:49.230008+0000","last_unstale":"2024-02-23T00:39:44.198234+0000","last_undegraded":"2024-02-23T00:39:44.198234+0000","last_fullsized":"2024-02-23T00:39:44.198234+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:38:09.207232+0000","last_deep_scrub":"78'2","last_deep_scrub_stamp":"2024-02-23T00:38:09.207232+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.207232+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":515,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.182100+0000","last_change":"2024-02-23T00:39:31.887689+0000","last_active":"2024-02-23T00:39:44.182100+0000","last_peered":"2024-02-23T00:39:44.182100+0000","last_clean":"2024-02-23T00:39:44.182100+0000","last_became_active":"2024-02-23T00:38:50.185295+0000","last_became_peered":"2024-02-23T00:38:50.185295+0000","last_unstale":"2024-02-23T00:39:44.182100+0000","last_undegraded":"2024-02-23T00:39:44.182100+0000","last_fullsized":"2024-02-23T00:39:44.182100+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.887584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.887584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,2],"avail_no_missing":["7","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":702,"reported_epoch":565,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:39:44.179120+0000","last_change":"2024-02-23T00:39:22.606140+0000","last_active":"2024-02-23T00:39:44.179120+0000","last_peered":"2024-02-23T00:39:44.179120+0000","last_clean":"2024-02-23T00:38:19.193308+0000","last_became_active":"2024-02-23T00:38:22.159363+0000","last_became_peered":"2024-02-23T00:38:22.159363+0000","last_unstale":"2024-02-23T00:39:44.179120+0000","last_undegraded":"2024-02-23T00:39:44.179120+0000","last_fullsized":"2024-02-23T00:39:44.179120+0000","mapping_epoch":484,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":70,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.194438+0000","last_change":"2024-02-23T00:39:32.310781+0000","last_active":"2024-02-23T00:39:44.194438+0000","last_peered":"2024-02-23T00:39:44.194438+0000","last_clean":"2024-02-23T00:39:44.194438+0000","last_became_active":"2024-02-23T00:38:50.176504+0000","last_became_peered":"2024-02-23T00:38:50.176504+0000","last_unstale":"2024-02-23T00:39:44.194438+0000","last_undegraded":"2024-02-23T00:39:44.194438+0000","last_fullsized":"2024-02-23T00:39:44.194438+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:32.310679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:39:32.310679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,2],"avail_no_missing":["3","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":148,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.184099+0000","last_change":"2024-02-23T00:36:37.122400+0000","last_active":"2024-02-23T00:39:44.184099+0000","last_peered":"2024-02-23T00:39:44.184099+0000","last_clean":"2024-02-23T00:27:20.002687+0000","last_became_active":"2024-02-23T00:36:37.122400+0000","last_became_peered":"2024-02-23T00:36:37.122400+0000","last_unstale":"2024-02-23T00:39:44.184099+0000","last_undegraded":"2024-02-23T00:39:44.184099+0000","last_fullsized":"2024-02-23T00:36:37.111280+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":82,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.184693+0000","last_change":"2024-02-23T00:38:42.787673+0000","last_active":"2024-02-23T00:39:44.184693+0000","last_peered":"2024-02-23T00:39:44.184693+0000","last_clean":"2024-02-23T00:39:44.184693+0000","last_became_active":"2024-02-23T00:38:40.171498+0000","last_became_peered":"2024-02-23T00:38:40.171498+0000","last_unstale":"2024-02-23T00:39:44.184693+0000","last_undegraded":"2024-02-23T00:39:44.184693+0000","last_fullsized":"2024-02-23T00:39:44.184693+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.787571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.787571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,4],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":490,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.188052+0000","last_change":"2024-02-23T00:38:35.044376+0000","last_active":"2024-02-23T00:39:44.188052+0000","last_peered":"2024-02-23T00:39:44.188052+0000","last_clean":"2024-02-23T00:39:44.188052+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:39:44.188052+0000","last_undegraded":"2024-02-23T00:39:44.188052+0000","last_fullsized":"2024-02-23T00:39:44.188052+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:35.044298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.044298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":673,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187731+0000","last_change":"2024-02-23T00:38:52.159547+0000","last_active":"2024-02-23T00:39:44.187731+0000","last_peered":"2024-02-23T00:39:44.187731+0000","last_clean":"2024-02-23T00:39:44.187731+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:39:44.187731+0000","last_undegraded":"2024-02-23T00:39:44.187731+0000","last_fullsized":"2024-02-23T00:39:44.187731+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:38:39.248198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.248198+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":547,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.183457+0000","last_change":"2024-02-23T00:38:07.395392+0000","last_active":"2024-02-23T00:39:44.183457+0000","last_peered":"2024-02-23T00:39:44.183457+0000","last_clean":"2024-02-23T00:39:44.183457+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:39:44.183457+0000","last_undegraded":"2024-02-23T00:39:44.183457+0000","last_fullsized":"2024-02-23T00:39:44.183457+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.395260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.395260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":534,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.180133+0000","last_change":"2024-02-23T00:36:37.113234+0000","last_active":"2024-02-23T00:39:44.180133+0000","last_peered":"2024-02-23T00:39:44.180133+0000","last_clean":"2024-02-23T00:36:33.367528+0000","last_became_active":"2024-02-23T00:36:37.113234+0000","last_became_peered":"2024-02-23T00:36:37.113234+0000","last_unstale":"2024-02-23T00:39:44.180133+0000","last_undegraded":"2024-02-23T00:39:44.180133+0000","last_fullsized":"2024-02-23T00:36:37.097125+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.791748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.791748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.206906+0000","last_change":"2024-02-23T00:38:49.227843+0000","last_active":"2024-02-23T00:39:44.206906+0000","last_peered":"2024-02-23T00:39:44.206906+0000","last_clean":"2024-02-23T00:39:44.206906+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:39:44.206906+0000","last_undegraded":"2024-02-23T00:39:44.206906+0000","last_fullsized":"2024-02-23T00:39:44.206906+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:27.951690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:27.951690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":648,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.204842+0000","last_change":"2024-02-23T00:38:55.178058+0000","last_active":"2024-02-23T00:39:44.204842+0000","last_peered":"2024-02-23T00:39:44.204842+0000","last_clean":"2024-02-23T00:39:44.204842+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:39:44.204842+0000","last_undegraded":"2024-02-23T00:39:44.204842+0000","last_fullsized":"2024-02-23T00:39:44.204842+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":29,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.197285+0000","last_change":"2024-02-23T00:38:49.192589+0000","last_active":"2024-02-23T00:39:44.197285+0000","last_peered":"2024-02-23T00:39:44.197285+0000","last_clean":"2024-02-23T00:39:44.197285+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:39:44.197285+0000","last_undegraded":"2024-02-23T00:39:44.197285+0000","last_fullsized":"2024-02-23T00:39:44.197285+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:35.929507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:35.929507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":553,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.182877+0000","last_change":"2024-02-23T00:38:49.224957+0000","last_active":"2024-02-23T00:39:44.182877+0000","last_peered":"2024-02-23T00:39:44.182877+0000","last_clean":"2024-02-23T00:39:44.182877+0000","last_became_active":"2024-02-23T00:38:49.224630+0000","last_became_peered":"2024-02-23T00:38:49.224630+0000","last_unstale":"2024-02-23T00:39:44.182877+0000","last_undegraded":"2024-02-23T00:39:44.182877+0000","last_fullsized":"2024-02-23T00:39:44.182877+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.063485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.063485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":482,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.185537+0000","last_change":"2024-02-23T00:38:03.798894+0000","last_active":"2024-02-23T00:39:44.185537+0000","last_peered":"2024-02-23T00:39:44.185537+0000","last_clean":"2024-02-23T00:39:44.185537+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:39:44.185537+0000","last_undegraded":"2024-02-23T00:39:44.185537+0000","last_fullsized":"2024-02-23T00:39:44.185537+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.798824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.798824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":684,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.195827+0000","last_change":"2024-02-23T00:39:21.295448+0000","last_active":"2024-02-23T00:39:44.195827+0000","last_peered":"2024-02-23T00:39:44.195827+0000","last_clean":"2024-02-23T00:39:44.195827+0000","last_became_active":"2024-02-23T00:39:21.294883+0000","last_became_peered":"2024-02-23T00:39:21.294883+0000","last_unstale":"2024-02-23T00:39:44.195827+0000","last_undegraded":"2024-02-23T00:39:44.195827+0000","last_fullsized":"2024-02-23T00:39:44.195827+0000","mapping_epoch":541,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,1],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":249,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.182629+0000","last_change":"2024-02-23T00:37:59.129696+0000","last_active":"2024-02-23T00:39:44.182629+0000","last_peered":"2024-02-23T00:39:44.182629+0000","last_clean":"2024-02-23T00:39:44.182629+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:39:44.182629+0000","last_undegraded":"2024-02-23T00:39:44.182629+0000","last_fullsized":"2024-02-23T00:39:44.182629+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.129604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.129604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":530,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.181494+0000","last_change":"2024-02-23T00:36:37.124006+0000","last_active":"2024-02-23T00:39:44.181494+0000","last_peered":"2024-02-23T00:39:44.181494+0000","last_clean":"2024-02-23T00:36:33.366419+0000","last_became_active":"2024-02-23T00:36:37.124006+0000","last_became_peered":"2024-02-23T00:36:37.124006+0000","last_unstale":"2024-02-23T00:39:44.181494+0000","last_undegraded":"2024-02-23T00:39:44.181494+0000","last_fullsized":"2024-02-23T00:36:37.103625+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":93,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187639+0000","last_change":"2024-02-23T00:38:49.208012+0000","last_active":"2024-02-23T00:39:44.187639+0000","last_peered":"2024-02-23T00:39:44.187639+0000","last_clean":"2024-02-23T00:39:44.187639+0000","last_became_active":"2024-02-23T00:38:49.207582+0000","last_became_peered":"2024-02-23T00:38:49.207582+0000","last_unstale":"2024-02-23T00:39:44.187639+0000","last_undegraded":"2024-02-23T00:39:44.187639+0000","last_fullsized":"2024-02-23T00:39:44.187639+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:39.957272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.957272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":490,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.187264+0000","last_change":"2024-02-23T00:37:18.044814+0000","last_active":"2024-02-23T00:39:44.187264+0000","last_peered":"2024-02-23T00:39:44.187264+0000","last_clean":"2024-02-23T00:39:44.187264+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:39:44.187264+0000","last_undegraded":"2024-02-23T00:39:44.187264+0000","last_fullsized":"2024-02-23T00:39:44.187264+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:18.044737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:37:18.044737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":234,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.179498+0000","last_change":"2024-02-23T00:39:21.278362+0000","last_active":"2024-02-23T00:39:44.179498+0000","last_peered":"2024-02-23T00:39:44.179498+0000","last_clean":"2024-02-23T00:39:44.179498+0000","last_became_active":"2024-02-23T00:39:21.278024+0000","last_became_peered":"2024-02-23T00:39:21.278024+0000","last_unstale":"2024-02-23T00:39:44.179498+0000","last_undegraded":"2024-02-23T00:39:44.179498+0000","last_fullsized":"2024-02-23T00:39:44.179498+0000","mapping_epoch":541,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":543,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.187478+0000","last_change":"2024-02-23T00:36:37.114846+0000","last_active":"2024-02-23T00:39:44.187478+0000","last_peered":"2024-02-23T00:39:44.187478+0000","last_clean":"2024-02-23T00:36:33.367758+0000","last_became_active":"2024-02-23T00:36:37.114846+0000","last_became_peered":"2024-02-23T00:36:37.114846+0000","last_unstale":"2024-02-23T00:39:44.187478+0000","last_undegraded":"2024-02-23T00:39:44.187478+0000","last_fullsized":"2024-02-23T00:36:37.107709+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":538,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.186282+0000","last_change":"2024-02-23T00:37:19.034254+0000","last_active":"2024-02-23T00:39:44.186282+0000","last_peered":"2024-02-23T00:39:44.186282+0000","last_clean":"2024-02-23T00:39:44.186282+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:39:44.186282+0000","last_undegraded":"2024-02-23T00:39:44.186282+0000","last_fullsized":"2024-02-23T00:39:44.186282+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:19.034136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:37:19.034136+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":80,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.186353+0000","last_change":"2024-02-23T00:38:43.922541+0000","last_active":"2024-02-23T00:39:44.186353+0000","last_peered":"2024-02-23T00:39:44.186353+0000","last_clean":"2024-02-23T00:39:44.186353+0000","last_became_active":"2024-02-23T00:38:40.180445+0000","last_became_peered":"2024-02-23T00:38:40.180445+0000","last_unstale":"2024-02-23T00:39:44.186353+0000","last_undegraded":"2024-02-23T00:39:44.186353+0000","last_fullsized":"2024-02-23T00:39:44.186353+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:43.922442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.494737+0000","last_clean_scrub_stamp":"2024-02-23T00:38:43.922442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,1],"avail_no_missing":["7","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":506,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178457+0000","last_change":"2024-02-23T00:39:21.439880+0000","last_active":"2024-02-23T00:39:44.178457+0000","last_peered":"2024-02-23T00:39:44.178457+0000","last_clean":"2024-02-23T00:39:44.178457+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:39:44.178457+0000","last_undegraded":"2024-02-23T00:39:44.178457+0000","last_fullsized":"2024-02-23T00:39:44.178457+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:21.439783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:21.439783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":691,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178538+0000","last_change":"2024-02-23T00:39:24.165093+0000","last_active":"2024-02-23T00:39:44.178538+0000","last_peered":"2024-02-23T00:39:44.178538+0000","last_clean":"2024-02-23T00:39:44.178538+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:39:44.178538+0000","last_undegraded":"2024-02-23T00:39:44.178538+0000","last_fullsized":"2024-02-23T00:39:44.178538+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":70,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.206981+0000","last_change":"2024-02-23T00:39:31.346410+0000","last_active":"2024-02-23T00:39:44.206981+0000","last_peered":"2024-02-23T00:39:44.206981+0000","last_clean":"2024-02-23T00:39:44.206981+0000","last_became_active":"2024-02-23T00:38:50.183147+0000","last_became_peered":"2024-02-23T00:38:50.183147+0000","last_unstale":"2024-02-23T00:39:44.206981+0000","last_undegraded":"2024-02-23T00:39:44.206981+0000","last_fullsized":"2024-02-23T00:39:44.206981+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.346259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.346259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,2],"avail_no_missing":["3","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":195,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.186255+0000","last_change":"2024-02-23T00:39:31.412402+0000","last_active":"2024-02-23T00:39:44.186255+0000","last_peered":"2024-02-23T00:39:44.186255+0000","last_clean":"2024-02-23T00:39:44.186255+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:39:44.186255+0000","last_undegraded":"2024-02-23T00:39:44.186255+0000","last_fullsized":"2024-02-23T00:39:44.186255+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.412345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.412345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":77,"reported_epoch":565,"state":"active+clean+remapped","last_fresh":"2024-02-23T00:39:44.179955+0000","last_change":"2024-02-23T00:38:40.203064+0000","last_active":"2024-02-23T00:39:44.179955+0000","last_peered":"2024-02-23T00:39:44.179955+0000","last_clean":"2024-02-23T00:39:44.179955+0000","last_became_active":"2024-02-23T00:38:40.202564+0000","last_became_peered":"2024-02-23T00:38:40.202564+0000","last_unstale":"2024-02-23T00:39:44.179955+0000","last_undegraded":"2024-02-23T00:39:44.179955+0000","last_fullsized":"2024-02-23T00:39:44.179955+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:58.130479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:37.065865+0000","last_clean_scrub_stamp":"2024-02-23T00:37:58.130479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6,1],"avail_no_missing":["5","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":150,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.188003+0000","last_change":"2024-02-23T00:36:37.119114+0000","last_active":"2024-02-23T00:39:44.188003+0000","last_peered":"2024-02-23T00:39:44.188003+0000","last_clean":"2024-02-23T00:30:47.537238+0000","last_became_active":"2024-02-23T00:36:37.119114+0000","last_became_peered":"2024-02-23T00:36:37.119114+0000","last_unstale":"2024-02-23T00:39:44.188003+0000","last_undegraded":"2024-02-23T00:39:44.188003+0000","last_fullsized":"2024-02-23T00:36:37.106746+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.176441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.176441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":753,"reported_epoch":565,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:39:44.188539+0000","last_change":"2024-02-23T00:39:19.216599+0000","last_active":"2024-02-23T00:39:44.188539+0000","last_peered":"2024-02-23T00:39:44.188539+0000","last_clean":"2024-02-23T00:38:47.148293+0000","last_became_active":"2024-02-23T00:38:49.185201+0000","last_became_peered":"2024-02-23T00:38:49.185201+0000","last_unstale":"2024-02-23T00:39:44.188539+0000","last_undegraded":"2024-02-23T00:39:44.188539+0000","last_fullsized":"2024-02-23T00:39:44.188539+0000","mapping_epoch":511,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":487,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:34.287816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.287816+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[2,5,1],"avail_no_missing":["2","1","5"],"object_location_counts":[{"shards":"1,2,5","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":148,"reported_epoch":565,"state":"active+undersized","last_fresh":"2024-02-23T00:39:44.182706+0000","last_change":"2024-02-23T00:36:37.120482+0000","last_active":"2024-02-23T00:39:44.182706+0000","last_peered":"2024-02-23T00:39:44.182706+0000","last_clean":"2024-02-23T00:27:15.989319+0000","last_became_active":"2024-02-23T00:36:37.120482+0000","last_became_peered":"2024-02-23T00:36:37.120482+0000","last_unstale":"2024-02-23T00:39:44.182706+0000","last_undegraded":"2024-02-23T00:39:44.182706+0000","last_fullsized":"2024-02-23T00:36:37.111071+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":535,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.178801+0000","last_change":"2024-02-23T00:37:14.200716+0000","last_active":"2024-02-23T00:39:44.178801+0000","last_peered":"2024-02-23T00:39:44.178801+0000","last_clean":"2024-02-23T00:39:44.178801+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:39:44.178801+0000","last_undegraded":"2024-02-23T00:39:44.178801+0000","last_fullsized":"2024-02-23T00:39:44.178801+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:14.200606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:14.200606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":64,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.196690+0000","last_change":"2024-02-23T00:38:49.220042+0000","last_active":"2024-02-23T00:39:44.196690+0000","last_peered":"2024-02-23T00:39:44.196690+0000","last_clean":"2024-02-23T00:39:44.196690+0000","last_became_active":"2024-02-23T00:38:49.219344+0000","last_became_peered":"2024-02-23T00:38:49.219344+0000","last_unstale":"2024-02-23T00:39:44.196690+0000","last_undegraded":"2024-02-23T00:39:44.196690+0000","last_fullsized":"2024-02-23T00:39:44.196690+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:24.990671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:24.990671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":500,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-02-23T00:39:44.186054+0000","last_change":"2024-02-23T00:38:49.228307+0000","last_active":"2024-02-23T00:39:44.186054+0000","last_peered":"2024-02-23T00:39:44.186054+0000","last_clean":"2024-02-23T00:39:44.186054+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:39:44.186054+0000","last_undegraded":"2024-02-23T00:39:44.186054+0000","last_fullsized":"2024-02-23T00:39:44.186054+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:40.485283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:40.485283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":231,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":83,"acting":83,"num_store_stats":7},{"poolid":230,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":83,"acting":83,"num_store_stats":7},{"poolid":229,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":24,"acting":24,"num_store_stats":6},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":90,"acting":92,"num_store_stats":8},{"poolid":6,"num_pg":30,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":80,"acting":86,"num_store_stats":8},{"poolid":5,"num_pg":41,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":105,"acting":111,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3647,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":71,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1275,"num_read_kb":1039,"num_write":765,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":4291,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":434176,"data_stored":11546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":86,"acting":90,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":426,"num_read_kb":426,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":34,"acting":37,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":82,"acting":86,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332249,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312108,"kb_used_data":6684,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433044,"statfs":{"total":95995035648,"available":95675437056,"internally_reserved":0,"allocated":6844416,"data_stored":3896546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.02,"15min":1.024},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.79800000000000004,"5min":57.401000000000003,"15min":61.048999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.1719999999999999,"15min":1.2450000000000001},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":65.558999999999997,"15min":73.870999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45900000000000002,"15min":0.68400000000000005},"min":{"1min":0.28100000000000003,"5min":0.23999999999999999,"15min":0.217},"max":{"1min":1.23,"5min":1.23,"15min":31.175000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.022,"5min":0.57599999999999996,"15min":0.71699999999999997},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":11.066000000000001,"5min":11.066000000000001,"15min":29.423999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.64500000000000002,"15min":0.71699999999999997},"min":{"1min":0.28799999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":6.9729999999999999,"5min":8.0099999999999998,"15min":16.603999999999999},"last":0.44},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.56100000000000005,"15min":0.622},"min":{"1min":0.30599999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.375,"5min":4.375,"15min":7.5339999999999998},"last":0.45800000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.60199999999999998,"15min":1.0109999999999999},"min":{"1min":0.32100000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.9900000000000002,"5min":8.0449999999999999,"15min":57.729999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.57599999999999996,"15min":0.91700000000000004},"min":{"1min":0.35099999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.8479999999999999,"5min":3.8479999999999999,"15min":27.573},"last":0.41999999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.41599999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.706},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.46600000000000003,"15min":0.45400000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.52600000000000002,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.35199999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40500000000000003,"15min":0.42699999999999999},"min":{"1min":0.26900000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.58899999999999997,"5min":0.72099999999999997,"15min":1.0209999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.36599999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46800000000000003,"15min":0.48499999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":2.1339999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.498,"15min":0.48599999999999999},"min":{"1min":0.28199999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70999999999999996,"5min":5.194,"15min":5.194},"last":0.38}]}]},{"osd":6,"up_from":44,"seq":188978561182,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316204,"kb_used_data":6684,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428948,"statfs":{"total":95995035648,"available":95671242752,"internally_reserved":0,"allocated":6844416,"data_stored":3906672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.6499999999999999,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.78800000000000003,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64800000000000002,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":9.468,"15min":9.468},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.56299999999999994,"15min":1.0309999999999999},"min":{"1min":0.28699999999999998,"5min":0.253,"15min":0.245},"max":{"1min":6.056,"5min":6.056,"15min":59.347999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.4279999999999999,"5min":0.68300000000000005,"15min":1.3500000000000001},"min":{"1min":0.30299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":18.423999999999999,"5min":18.423999999999999,"15min":87.221000000000004},"last":0.49099999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":0.70599999999999996,"15min":0.875},"min":{"1min":0.34000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":18.381,"5min":18.381,"15min":25.550999999999998},"last":0.44},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.60499999999999998,"15min":0.88600000000000001},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.1749999999999998,"5min":6.1749999999999998,"15min":40.914000000000001},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.57799999999999996,"15min":0.82799999999999996},"min":{"1min":0.373,"5min":0.317,"15min":0.23400000000000001},"max":{"1min":6.1200000000000001,"5min":6.1200000000000001,"15min":31.309999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":1.476,"5min":0.72799999999999998,"15min":1.3879999999999999},"min":{"1min":0.378,"5min":0.30099999999999999,"15min":0.29899999999999999},"max":{"1min":18.448,"5min":18.448,"15min":86.311999999999998},"last":0.46800000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.47199999999999998,"15min":0.44900000000000001},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.1500000000000004,"5min":6.1500000000000004,"15min":6.1500000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.496,"15min":0.52700000000000002},"min":{"1min":0.247,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":17.718},"last":0.496}]},{"osd":5,"last update":"Fri Feb 23 00:39:16 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.49199999999999999,"15min":0.499},"min":{"1min":0.29599999999999999,"5min":0.254,"15min":0.254},"max":{"1min":6.3559999999999999,"5min":6.3559999999999999,"15min":8.0990000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.51600000000000001,"15min":0.5},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27500000000000002},"max":{"1min":6.0780000000000003,"5min":6.0780000000000003,"15min":8.1370000000000005},"last":0.40799999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:39:32 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52900000000000003,"15min":0.61599999999999999},"min":{"1min":0.308,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5629999999999999,"5min":6.4279999999999999,"15min":17.881},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.29799999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.66900000000000004,"5min":18.475000000000001,"15min":18.475000000000001},"last":0.36399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320368,"kb_used_data":6752,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93424784,"statfs":{"total":95995035648,"available":95666978816,"internally_reserved":0,"allocated":6914048,"data_stored":3908649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.29,"15min":0.66300000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.51700000000000002,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.89600000000000002,"15min":0.55200000000000005},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60499999999999998,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.35599999999999998,"15min":0.67400000000000004},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.9980000000000002,"5min":3.9980000000000002,"15min":35.313000000000002},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.57799999999999996},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.625,"5min":2.625,"15min":27.908999999999999},"last":0.26800000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.41399999999999998,"15min":0.65900000000000003},"min":{"1min":0.26500000000000001,"5min":0.23000000000000001,"15min":0.21299999999999999},"max":{"1min":3.8119999999999998,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.254},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.40000000000000002,"15min":0.65000000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.0259999999999998,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.30199999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:39:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.56299999999999994,"15min":0.70399999999999996},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.4900000000000002,"5min":3.4900000000000002,"15min":16.231000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52100000000000002,"15min":0.72099999999999997},"min":{"1min":0.29599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.518,"5min":1.518,"15min":29.212},"last":0.377}]},{"osd":5,"last update":"Fri Feb 23 00:38:44 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.52400000000000002,"15min":0.71499999999999997},"min":{"1min":0.34100000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.71099999999999997,"5min":0.90500000000000003,"15min":16.294},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53100000000000003,"15min":0.67600000000000005},"min":{"1min":0.374,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":11.105},"last":0.438}]},{"osd":6,"last update":"Fri Feb 23 00:39:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.52100000000000002,"15min":0.755},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.622,"5min":0.94799999999999995,"15min":29.187000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.57799999999999996,"15min":0.749},"min":{"1min":0.39200000000000002,"5min":0.312,"15min":0.312},"max":{"1min":2.528,"5min":2.528,"15min":16.369},"last":0.36699999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:39:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.61399999999999999,"15min":0.78100000000000003},"min":{"1min":0.40500000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":5.2119999999999997,"5min":5.2119999999999997,"15min":16.395},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.61699999999999999,"15min":0.67600000000000005},"min":{"1min":0.35599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":3.7669999999999999,"5min":3.7669999999999999,"15min":11.135999999999999},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318432,"kb_used_data":4824,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93426720,"statfs":{"total":95995035648,"available":95668961280,"internally_reserved":0,"allocated":4939776,"data_stored":2599387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.80600000000000005,"15min":0.495},"min":{"1min":0.217,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47799999999999998,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.246},{"interface":"front","average":{"1min":0.32800000000000001,"5min":1.165,"15min":0.61899999999999999},"min":{"1min":0.215,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.497,"5min":23.832999999999998,"15min":23.832999999999998},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.997,"15min":0.58899999999999997},"min":{"1min":0.23400000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.64400000000000002,"5min":23.809999999999999,"15min":23.809999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.748,"15min":0.48999999999999999},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.22700000000000001},"max":{"1min":0.51100000000000001,"5min":15.965,"15min":15.965},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:35:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":1.0820000000000001,"15min":0.65000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":24.164999999999999,"15min":24.164999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.0369999999999999,"15min":0.63700000000000001},"min":{"1min":0.29599999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.58799999999999997,"5min":16.006,"15min":16.006},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.73399999999999999,"15min":0.63800000000000001},"min":{"1min":0.36299999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84699999999999998,"5min":15.917999999999999,"15min":15.917999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.038,"15min":0.76200000000000001},"min":{"1min":0.38300000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83699999999999997,"5min":13.287000000000001,"15min":13.287000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":1.071,"15min":0.72799999999999998},"min":{"1min":0.39100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86799999999999999,"5min":13.311,"15min":13.311},"last":0.499},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.113,"15min":0.74399999999999999},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.34699999999999998},"max":{"1min":0.69299999999999995,"5min":23.744,"15min":23.744},"last":0.42299999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:36:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":1.0740000000000001,"15min":0.80600000000000005},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78500000000000003,"5min":13.334,"15min":33.082999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.93400000000000005,"15min":0.70999999999999996},"min":{"1min":0.41199999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81699999999999995,"5min":15.874000000000001,"15min":15.874000000000001},"last":0.49199999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.84099999999999997,"15min":0.78500000000000003},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.871,"5min":15.943,"15min":31.251999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":1.091,"15min":0.76100000000000001},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.36799999999999999},"max":{"1min":0.80000000000000004,"5min":23.785,"15min":23.785},"last":0.47599999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411505,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312096,"kb_used_data":6672,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433056,"statfs":{"total":95995035648,"available":95675449344,"internally_reserved":0,"allocated":6832128,"data_stored":3905535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":1.0600000000000001,"15min":0.754},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.65500000000000003,"5min":56.509,"15min":56.509},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.59799999999999998,"15min":0.71199999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56799999999999995,"5min":19.571999999999999,"15min":31.498999999999999},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:39:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.45100000000000001,"15min":0.60099999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.54100000000000004,"5min":7.7110000000000003,"15min":19.324999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.318,"5min":0.504,"15min":0.75700000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.66100000000000003,"5min":7.9779999999999998,"15min":35.052},"last":0.32000000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:39:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.748},"min":{"1min":0.21199999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.944,"5min":10.032,"15min":43.5},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.60899999999999999,"15min":0.69599999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.0230000000000001,"5min":11.815,"15min":27.414000000000001},"last":0.33600000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:39:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.69999999999999996,"15min":0.67300000000000004},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.93999999999999995,"5min":10.106,"15min":14.614000000000001},"last":0.503},{"interface":"front","average":{"1min":0.495,"5min":0.63500000000000001,"15min":0.623},"min":{"1min":0.33000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73599999999999999,"5min":7.766,"15min":7.766},"last":0.41499999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:39:01 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":0.67900000000000005,"15min":0.70999999999999996},"min":{"1min":0.35199999999999998,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":7.915,"5min":7.915,"15min":17.765000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.68100000000000005,"15min":0.67600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.317},"max":{"1min":2.129,"5min":10.138,"15min":14.589},"last":0.46600000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:39:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.63400000000000001,"15min":0.63},"min":{"1min":0.35399999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.944,"5min":7.7869999999999999,"15min":7.7869999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.288,"5min":0.73099999999999998,"15min":0.79700000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.9589999999999996,"5min":7.9589999999999996,"15min":20.574999999999999},"last":0.48799999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:39:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.73299999999999998,"15min":1.0760000000000001},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.88100000000000001,"5min":7.9480000000000004,"15min":66.632000000000005},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.73299999999999998,"15min":0.84999999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.878,"5min":10.163,"15min":19.603999999999999},"last":0.52800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303948,"kb_used_data":6716,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441204,"statfs":{"total":95995035648,"available":95683792896,"internally_reserved":0,"allocated":6877184,"data_stored":3907178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":1.506,"15min":0.745},"min":{"1min":0.252,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.76100000000000001,"5min":55.238999999999997,"15min":55.238999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":1.4750000000000001,"15min":0.747},"min":{"1min":0.26100000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.47699999999999998,"5min":54.563000000000002,"15min":54.563000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.82999999999999996,"15min":0.61799999999999999},"min":{"1min":0.21299999999999999,"5min":0.21099999999999999,"15min":0.17899999999999999},"max":{"1min":2.718,"5min":54.351999999999997,"15min":54.351999999999997},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.85799999999999998,"15min":0.76200000000000001},"min":{"1min":0.23499999999999999,"5min":0.219,"15min":0.219},"max":{"1min":1.3129999999999999,"5min":55.337000000000003,"15min":55.337000000000003},"last":0.253}]},{"osd":2,"last update":"Fri Feb 23 00:39:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.85499999999999998,"15min":0.75},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.159,"5min":55.368000000000002,"15min":55.368000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.88,"15min":0.73499999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.185},"max":{"1min":4.181,"5min":54.383000000000003,"15min":54.383000000000003},"last":0.26800000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:39:20 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.1850000000000001,"15min":0.94099999999999995},"min":{"1min":0.35999999999999999,"5min":0.316,"15min":0.27900000000000003},"max":{"1min":6.8799999999999999,"5min":54.530000000000001,"15min":54.530000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.58899999999999997,"15min":0.63200000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.24099999999999999},"max":{"1min":4.9829999999999997,"5min":4.9829999999999997,"15min":8.0359999999999996},"last":0.56299999999999994}]},{"osd":5,"last update":"Fri Feb 23 00:38:41 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":1.0089999999999999,"15min":0.877},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.28899999999999998},"max":{"1min":0.83899999999999997,"5min":55.401000000000003,"15min":55.401000000000003},"last":0.436},{"interface":"front","average":{"1min":0.49399999999999999,"5min":1.006,"15min":0.83199999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.30299999999999999},"max":{"1min":0.72899999999999998,"5min":54.454999999999998,"15min":54.454999999999998},"last":0.628}]},{"osd":6,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.2330000000000001,"15min":0.85199999999999998},"min":{"1min":0.38100000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":9.7490000000000006,"5min":54.305,"15min":54.305},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.25,"15min":1.01},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":9.6820000000000004,"5min":55.424999999999997,"15min":55.424999999999997},"last":0.59099999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:38:41 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.0720000000000001,"15min":0.77600000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34300000000000003},"max":{"1min":4.9790000000000001,"5min":54.222000000000001,"15min":54.222000000000001},"last":0.499},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.119,"15min":0.93799999999999994},"min":{"1min":0.40400000000000003,"5min":0.377,"15min":0.36499999999999999},"max":{"1min":6.7279999999999998,"5min":55.460000000000001,"15min":55.460000000000001},"last":0.61099999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986344,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303932,"kb_used_data":6700,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441220,"statfs":{"total":95995035648,"available":95683809280,"internally_reserved":0,"allocated":6860800,"data_stored":3906303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1739999999999999,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.68799999999999994,"5min":21.419,"15min":21.419},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":2.1200000000000001,"15min":1.2589999999999999},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.81999999999999995,"15min":1.097},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.0540000000000003,"5min":16.696000000000002,"15min":41.137999999999998},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.88,"5min":0.72199999999999998,"15min":0.90800000000000003},"min":{"1min":0.27500000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":8.4809999999999999,"5min":13.455,"15min":26.210000000000001},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.73699999999999999,"15min":1.1519999999999999},"min":{"1min":0.31900000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.0510000000000002,"5min":13.478,"15min":67.253},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.67200000000000004,"15min":1.042},"min":{"1min":0.318,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.4980000000000002,"5min":7.4980000000000002,"15min":41.927999999999997},"last":0.34499999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.68899999999999995,"15min":1.0920000000000001},"min":{"1min":0.38400000000000001,"5min":0.34799999999999998,"15min":0.34000000000000002},"max":{"1min":7.5309999999999997,"5min":7.774,"15min":36.301000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.751,"15min":1.1850000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.24299999999999999},"max":{"1min":2.726,"5min":16.657,"15min":42.064},"last":0.372}]},{"osd":5,"last update":"Fri Feb 23 00:38:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44700000000000001,"15min":0.55900000000000005},"min":{"1min":0.22500000000000001,"5min":0.219,"15min":0.217},"max":{"1min":0.73199999999999998,"5min":2.3210000000000002,"15min":14.242000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.56200000000000006,"15min":0.59799999999999998},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.71199999999999997,"5min":7.1459999999999999,"15min":14.124000000000001},"last":0.29399999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:39:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.55700000000000005,"15min":0.57099999999999995},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":1.734,"5min":7.1189999999999998,"15min":8.0730000000000004},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.46000000000000002,"15min":0.63300000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.76300000000000001,"5min":2.2450000000000001,"15min":45.814999999999998},"last":0.35199999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:39:31 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.52400000000000002,"15min":0.61599999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70699999999999996,"5min":5.1280000000000001,"15min":17.178999999999998},"last":0.378},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.68400000000000005,"15min":0.69699999999999995},"min":{"1min":0.35399999999999998,"5min":0.307,"15min":0.28299999999999997},"max":{"1min":1.891,"5min":15.85,"15min":15.85},"last":0.40699999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757411,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312096,"kb_used_data":6672,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433056,"statfs":{"total":95995035648,"available":95675449344,"internally_reserved":0,"allocated":6832128,"data_stored":3896331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.1200000000000001,"15min":0.70699999999999996},"min":{"1min":0.36699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.115,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":1.028},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.498,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.504,"15min":0.75800000000000001},"min":{"1min":0.27500000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.67800000000000005,"5min":3.7629999999999999,"15min":38.838000000000001},"last":0.307},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.55800000000000005,"15min":1.1919999999999999},"min":{"1min":0.33700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.748,"5min":3.7250000000000001,"15min":127.794},"last":0.41099999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":0.66800000000000004,"15min":0.94099999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":11.435,"5min":11.435,"15min":37.442999999999998},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.53800000000000003,"15min":0.70399999999999996},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.76800000000000002,"5min":3.734,"15min":23.297999999999998},"last":0.81899999999999995}]},{"osd":3,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.745,"15min":0.82099999999999995},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":11.457000000000001,"5min":12.507,"15min":24.414999999999999},"last":0.501},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.59199999999999997,"15min":0.91300000000000003},"min":{"1min":0.34599999999999997,"5min":0.33100000000000002,"15min":0.31},"max":{"1min":0.64600000000000002,"5min":3.8119999999999998,"15min":40.865000000000002},"last":0.96299999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:38:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40999999999999998,"15min":0.78200000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":0.58199999999999996,"5min":2.8940000000000001,"15min":103.13},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.56899999999999995,"15min":0.51100000000000001},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.381,"5min":11.381,"15min":12.795999999999999},"last":0.33600000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:39:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.61699999999999999,"15min":0.54600000000000004},"min":{"1min":0.29699999999999999,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45500000000000002,"15min":0.82299999999999995},"min":{"1min":0.27000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57799999999999996,"5min":1.5229999999999999,"15min":103.61499999999999},"last":0.42799999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:39:33 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.62,"15min":0.56599999999999995},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":11.478,"5min":11.478,"15min":11.478},"last":0.376},{"interface":"front","average":{"1min":0.443,"5min":0.54700000000000004,"15min":0.52600000000000002},"min":{"1min":0.309,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60499999999999998,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.35199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":230,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":231,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:39:53.902 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-23T00:39:53.902 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.meta pg_num 12 2024-02-23T00:39:53.929 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:53.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:54.092 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:54.117 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:39:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:39:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:39:54.119 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:54.137 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:54.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:54.241 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:54.284 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:54.640 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:54.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:54.744 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:54.772 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:54.791 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:39:54.791 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:39:55.019 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:55.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:55.027 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:55.124 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:55.150 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:55.169 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:55.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:55.272 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:55.298 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:53.736692+0000 mgr.y (mgr.24437) 522 : cluster [DBG] pgmap v878: 189 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/699 objects degraded (5.866%); 50/699 objects misplaced (7.153%); 27 B/s, 2 objects/s recovering 2024-02-23T00:39:55.298 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:54.427221+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e575: 8 total, 7 up, 8 in 2024-02-23T00:39:55.298 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:54.655733+0000 mon.a (mon.0) 3074 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:55.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:53.736692+0000 mgr.y (mgr.24437) 522 : cluster [DBG] pgmap v878: 189 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/699 objects degraded (5.866%); 50/699 objects misplaced (7.153%); 27 B/s, 2 objects/s recovering 2024-02-23T00:39:55.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:54.427221+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e575: 8 total, 7 up, 8 in 2024-02-23T00:39:55.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:54.655733+0000 mon.a (mon.0) 3074 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:53.736692+0000 mgr.y (mgr.24437) 522 : cluster [DBG] pgmap v878: 189 pgs: 1 active+remapped+backfill_toofull, 1 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 25 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/699 objects degraded (5.866%); 50/699 objects misplaced (7.153%); 27 B/s, 2 objects/s recovering 2024-02-23T00:39:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:54.427221+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e575: 8 total, 7 up, 8 in 2024-02-23T00:39:55.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:54.655733+0000 mon.a (mon.0) 3074 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:39:55.592 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:55.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-23T00:39:55.853 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:39:55.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:55.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:39:55.960 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:55.986 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:56.005 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:56.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:56.108 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:56.372 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:54.305638+0000 osd.4 (osd.4) 261 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:56.372 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:54.305857+0000 osd.4 (osd.4) 262 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:56.373 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:55.183174+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:39:56.373 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:55.183255+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e576: 8 total, 7 up, 8 in 2024-02-23T00:39:56.373 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:54.305638+0000 osd.4 (osd.4) 261 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:56.373 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:54.305857+0000 osd.4 (osd.4) 262 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:56.373 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:55.183174+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:39:56.373 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:55.183255+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e576: 8 total, 7 up, 8 in 2024-02-23T00:39:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:54.305638+0000 osd.4 (osd.4) 261 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:54.305857+0000 osd.4 (osd.4) 262 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:55.183174+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:39:56.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:55.183255+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e576: 8 total, 7 up, 8 in 2024-02-23T00:39:56.668 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:39:57.469 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:57.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:57.573 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:55.737733+0000 mgr.y (mgr.24437) 523 : cluster [DBG] pgmap v881: 197 pgs: 1 active+remapped+backfill_toofull, 6 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 2 unknown, 26 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/699 objects degraded (5.866%); 61/699 objects misplaced (8.727%) 2024-02-23T00:39:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:56.189742+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e577: 8 total, 7 up, 8 in 2024-02-23T00:39:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:56.193051+0000 mon.a (mon.0) 3078 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:57.159143+0000 mon.a (mon.0) 3079 : cluster [WRN] Health check update: Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 39 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:55.737733+0000 mgr.y (mgr.24437) 523 : cluster [DBG] pgmap v881: 197 pgs: 1 active+remapped+backfill_toofull, 6 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 2 unknown, 26 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/699 objects degraded (5.866%); 61/699 objects misplaced (8.727%) 2024-02-23T00:39:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:56.189742+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e577: 8 total, 7 up, 8 in 2024-02-23T00:39:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:56.193051+0000 mon.a (mon.0) 3078 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:57.159143+0000 mon.a (mon.0) 3079 : cluster [WRN] Health check update: Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 39 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:57.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:55.737733+0000 mgr.y (mgr.24437) 523 : cluster [DBG] pgmap v881: 197 pgs: 1 active+remapped+backfill_toofull, 6 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 2 unknown, 26 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/699 objects degraded (5.866%); 61/699 objects misplaced (8.727%) 2024-02-23T00:39:57.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:56.189742+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e577: 8 total, 7 up, 8 in 2024-02-23T00:39:57.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:56.193051+0000 mon.a (mon.0) 3078 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:39:57.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:57.159143+0000 mon.a (mon.0) 3079 : cluster [WRN] Health check update: Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 39 pgs undersized (PG_DEGRADED) 2024-02-23T00:39:57.599 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:57.617 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:57.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:57.721 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:39:57.748 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:57.997 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:39:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:58.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:39:58.103 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:58.131 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:58.150 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:58.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:58.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:58.254 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:39:58.281 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:58.300 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:39:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:58.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:39:58.404 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:39:58.426 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:56.938111+0000 osd.7 (osd.7) 270 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:58.427 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:56.938355+0000 osd.7 (osd.7) 271 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:58.427 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:57.246297+0000 mon.a (mon.0) 3080 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:58.427 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:57.246412+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e578: 8 total, 7 up, 8 in 2024-02-23T00:39:58.427 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:57.263673+0000 mon.a (mon.0) 3082 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]: dispatch 2024-02-23T00:39:58.429 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:58.452 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:39:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:58.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:39:58.556 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:39:58.582 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:58.582 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:56.938111+0000 osd.7 (osd.7) 270 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:56.938355+0000 osd.7 (osd.7) 271 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:57.246297+0000 mon.a (mon.0) 3080 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:57.246412+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e578: 8 total, 7 up, 8 in 2024-02-23T00:39:58.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:57.263673+0000 mon.a (mon.0) 3082 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]: dispatch 2024-02-23T00:39:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:56.938111+0000 osd.7 (osd.7) 270 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:39:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:56.938355+0000 osd.7 (osd.7) 271 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:39:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:57.246297+0000 mon.a (mon.0) 3080 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:39:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:57.246412+0000 mon.a (mon.0) 3081 : cluster [DBG] osdmap e578: 8 total, 7 up, 8 in 2024-02-23T00:39:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:57.263673+0000 mon.a (mon.0) 3082 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]: dispatch 2024-02-23T00:39:58.601 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:39:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:58.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:39:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.673+0000 7f0032a75700 1 -- 172.21.15.87:0/2137230508 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fd510 msgr2=0x7f002c0f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.673+0000 7f0032a75700 1 --2- 172.21.15.87:0/2137230508 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fd510 0x7f002c0f6d40 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 rx=0x7f00180047b0 tx=0x7f0018009cb0).stop 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.673+0000 7f0032a75700 1 -- 172.21.15.87:0/2137230508 shutdown_connections 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.673+0000 7f0032a75700 1 --2- 172.21.15.87:0/2137230508 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f002c0f7400 0x7f002c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 --2- 172.21.15.87:0/2137230508 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f002c0fcbb0 0x7f002c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 --2- 172.21.15.87:0/2137230508 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fd510 0x7f002c0f6d40 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 -- 172.21.15.87:0/2137230508 >> 172.21.15.87:0/2137230508 conn(0x7f002c0f2820 msgr2=0x7f002c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 -- 172.21.15.87:0/2137230508 shutdown_connections 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 -- 172.21.15.87:0/2137230508 wait complete. 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 librados: starting msgr at 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 librados: starting objecter 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 Processor -- start 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 -- start start 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 librados: setting wanted keys 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.674+0000 7f0032a75700 1 librados: calling monclient init 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f002c0f7400 0x7f002c1103b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fcbb0 0x7f002c1108a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f002c0fd510 0x7f002c10a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c10aa40 con 0x7f002c0fd510 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c10abb0 con 0x7f002c0f7400 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c10ad70 con 0x7f002c0fcbb0 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f002bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fcbb0 0x7f002c1108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0031012700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f002c0fd510 0x7f002c10a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f002bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fcbb0 0x7f002c1108a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:38242/0 (socket says 172.21.15.87:38242) 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f002bfff700 1 -- 172.21.15.87:0/624350809 learned_addr learned my addr 172.21.15.87:0/624350809 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0031012700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f002c0fd510 0x7f002c10a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42460/0 (socket says 172.21.15.87:42460) 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0030811700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f002c0f7400 0x7f002c1103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f002bfff700 1 -- 172.21.15.87:0/624350809 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f002c0f7400 msgr2=0x7f002c1103b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f002bfff700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f002c0f7400 0x7f002c1103b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:58.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f002bfff700 1 -- 172.21.15.87:0/624350809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f002c0fd510 msgr2=0x7f002c10a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f002bfff700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f002c0fd510 0x7f002c10a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f002bfff700 1 -- 172.21.15.87:0/624350809 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020005040 con 0x7f002c0fcbb0 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0031012700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f002c0fd510 0x7f002c10a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f002bfff700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fcbb0 0x7f002c1108a0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7f0018009b10 tx=0x7f0018009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0029f76700 10 client.?.objecter ms_handle_connect 0x7f002c0fcbb0 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0029f76700 10 client.?.objecter resend_mon_ops 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0029f76700 1 -- 172.21.15.87:0/624350809 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018018070 con 0x7f002c0fcbb0 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0029f76700 1 -- 172.21.15.87:0/624350809 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0018013da0 con 0x7f002c0fcbb0 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0018005040 con 0x7f002c0fcbb0 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0029f76700 1 -- 172.21.15.87:0/624350809 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001801dcc0 con 0x7f002c0fcbb0 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 10 client.38024.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002c10b210 con 0x7f002c0fcbb0 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.675+0000 7f0032a75700 1 librados: init done 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.676+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002c10b400 con 0x7f002c0fcbb0 2024-02-23T00:39:58.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.677+0000 7f0029f76700 1 -- 172.21.15.87:0/624350809 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f001801d420 con 0x7f002c0fcbb0 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.677+0000 7f0029f76700 1 --2- 172.21.15.87:0/624350809 >> v2:172.21.15.87:6800/1231597463 conn(0x7f001c062c90 0x7f001c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.677+0000 7f0029f76700 1 -- 172.21.15.87:0/624350809 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(579..579 src has 173..579) v4 ==== 10253+0+0 (secure 0 0 0) 0x7f001807efd0 con 0x7f002c0fcbb0 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.677+0000 7f0029f76700 10 client.38024.objecter ms_dispatch 0x7f002c001550 osd_map(579..579 src has 173..579) v4 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.677+0000 7f0029f76700 3 client.38024.objecter handle_osd_map got epochs [579,579] > 0 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.677+0000 7f0029f76700 3 client.38024.objecter handle_osd_map decoding full epoch 579 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.677+0000 7f0030811700 1 --2- 172.21.15.87:0/624350809 >> v2:172.21.15.87:6800/1231597463 conn(0x7f001c062c90 0x7f001c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.677+0000 7f0029f76700 20 client.38024.objecter dump_active .. 0 homeless 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.678+0000 7f0029f76700 10 client.38024.objecter ms_handle_connect 0x7f001c062c90 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.678+0000 7f0030811700 1 --2- 172.21.15.87:0/624350809 >> v2:172.21.15.87:6800/1231597463 conn(0x7f001c062c90 0x7f001c065150 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 rx=0x7f00200047e0 tx=0x7f002000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:39:58.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.681+0000 7f0029f76700 1 -- 172.21.15.87:0/624350809 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f00180adf60 con 0x7f002c0fcbb0 2024-02-23T00:39:58.705 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:39:58.731 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:58.749 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:39:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:58.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:39:58.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:58.847+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"} v 0) v1 -- 0x7f002c106120 con 0x7f002c0fcbb0 2024-02-23T00:39:58.853 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:39:58.882 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:59.081 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:39:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:39:59.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:39:59.129 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:59.135 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:39:59.187 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:39:59.214 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:39:59.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.274+0000 7f0029f76700 1 -- 172.21.15.87:0/624350809 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]=0 set pool 6 pg_num to 12 v580) v1 ==== 141+0+0 (secure 0 0 0) 0x7f00180519e0 con 0x7f002c0fcbb0 2024-02-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.277+0000 7f0032a75700 10 librados: watch_flush enter 2024-02-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.277+0000 7f0032a75700 10 librados: watch_flush exit 2024-02-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.277+0000 7f0032a75700 20 client.38024.objecter shutdown clearing up homeless session... 2024-02-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.277+0000 7f0032a75700 10 client.38024.objecter successfully canceled tick 2024-02-23T00:39:59.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.277+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 >> v2:172.21.15.87:6800/1231597463 conn(0x7f001c062c90 msgr2=0x7f001c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:59.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.277+0000 7f0032a75700 1 --2- 172.21.15.87:0/624350809 >> v2:172.21.15.87:6800/1231597463 conn(0x7f001c062c90 0x7f001c065150 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 rx=0x7f00200047e0 tx=0x7f002000b040).stop 2024-02-23T00:39:59.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.277+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fcbb0 msgr2=0x7f002c1108a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:39:59.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.277+0000 7f0032a75700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fcbb0 0x7f002c1108a0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 rx=0x7f0018009b10 tx=0x7f0018009cb0).stop 2024-02-23T00:39:59.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.284+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 shutdown_connections 2024-02-23T00:39:59.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.284+0000 7f0032a75700 1 --2- 172.21.15.87:0/624350809 >> v2:172.21.15.87:6800/1231597463 conn(0x7f001c062c90 0x7f001c065150 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:59.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.284+0000 7f0032a75700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f002c0fd510 0x7f002c10a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:59.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.284+0000 7f0032a75700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f002c0f7400 0x7f002c1103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:59.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.284+0000 7f0032a75700 1 --2- 172.21.15.87:0/624350809 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f002c0fcbb0 0x7f002c1108a0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:39:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.284+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 >> 172.21.15.87:0/624350809 conn(0x7f002c0f2820 msgr2=0x7f002c060df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:39:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.288+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 shutdown_connections 2024-02-23T00:39:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.288+0000 7f0032a75700 1 -- 172.21.15.87:0/624350809 wait complete. 2024-02-23T00:39:59.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:39:59.288+0000 7f0032a75700 1 librados: shutdown 2024-02-23T00:39:59.292 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 6 pg_num to 12 2024-02-23T00:39:59.464 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:57.738298+0000 mgr.y (mgr.24437) 524 : cluster [DBG] pgmap v884: 229 pgs: 1 active+remapped+backfill_toofull, 6 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 34 unknown, 26 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/699 objects degraded (5.866%); 61/699 objects misplaced (8.727%) 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:58.263709+0000 mon.a (mon.0) 3083 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]': finished 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:58.263769+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e579: 8 total, 7 up, 8 in 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:58.265183+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-99", "overlaypool": "test-rados-api-smithi087-82091-99-cache"}]: dispatch 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:58.849243+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.87:0/624350809' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]: dispatch 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:58.849545+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]: dispatch 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:57.738298+0000 mgr.y (mgr.24437) 524 : cluster [DBG] pgmap v884: 229 pgs: 1 active+remapped+backfill_toofull, 6 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 34 unknown, 26 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/699 objects degraded (5.866%); 61/699 objects misplaced (8.727%) 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:58.263709+0000 mon.a (mon.0) 3083 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]': finished 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:58.263769+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e579: 8 total, 7 up, 8 in 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:58.265183+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-99", "overlaypool": "test-rados-api-smithi087-82091-99-cache"}]: dispatch 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:58.849243+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.87:0/624350809' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]: dispatch 2024-02-23T00:39:59.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:39:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:58.849545+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]: dispatch 2024-02-23T00:39:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:57.738298+0000 mgr.y (mgr.24437) 524 : cluster [DBG] pgmap v884: 229 pgs: 1 active+remapped+backfill_toofull, 6 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 34 unknown, 26 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/699 objects degraded (5.866%); 61/699 objects misplaced (8.727%) 2024-02-23T00:39:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:58.263709+0000 mon.a (mon.0) 3083 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]': finished 2024-02-23T00:39:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:58.263769+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e579: 8 total, 7 up, 8 in 2024-02-23T00:39:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:58.265183+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-99", "overlaypool": "test-rados-api-smithi087-82091-99-cache"}]: dispatch 2024-02-23T00:39:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:58.849243+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.87:0/624350809' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]: dispatch 2024-02-23T00:39:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:39:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:58.849545+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]: dispatch 2024-02-23T00:40:00.263 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:00.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:00.367 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:00.397 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:00.417 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:00.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:00.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:59.272165+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-99", "overlaypool": "test-rados-api-smithi087-82091-99-cache"}]': finished 2024-02-23T00:40:00.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:59.272215+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]': finished 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:59.272272+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e580: 8 total, 7 up, 8 in 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:59.273106+0000 mon.a (mon.0) 3090 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-99-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:39:59.739955+0000 mon.a (mon.0) 3091 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "12"}]: dispatch 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000237+0000 mon.a (mon.0) 3092 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull; Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 39 pgs undersized; 2 pool(s) do not have an application enabled 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000282+0000 mon.a (mon.0) 3093 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000301+0000 mon.a (mon.0) 3094 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000324+0000 mon.a (mon.0) 3095 : cluster [WRN] osd.0 (root=default,host=smithi087) is down 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000377+0000 mon.a (mon.0) 3096 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000404+0000 mon.a (mon.0) 3097 : cluster [WRN] pg 4.17 is active+undersized+degraded+remapped+backfill_toofull, acting [1,5] 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000426+0000 mon.a (mon.0) 3098 : cluster [WRN] pg 4.1f is active+remapped+backfill_toofull, acting [1,4,7] 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000458+0000 mon.a (mon.0) 3099 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 39 pgs undersized 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000469+0000 mon.a (mon.0) 3100 : cluster [WRN] pg 1.0 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-02-23T00:40:00.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000478+0000 mon.a (mon.0) 3101 : cluster [WRN] pg 2.0 is stuck undersized for 3m, current state active+undersized, last acting [7,1] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000486+0000 mon.a (mon.0) 3102 : cluster [WRN] pg 2.4 is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000496+0000 mon.a (mon.0) 3103 : cluster [WRN] pg 2.5 is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000504+0000 mon.a (mon.0) 3104 : cluster [WRN] pg 2.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,1] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000514+0000 mon.a (mon.0) 3105 : cluster [WRN] pg 2.c is stuck undersized for 3m, current state active+undersized, last acting [2,5] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000523+0000 mon.a (mon.0) 3106 : cluster [WRN] pg 2.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,7] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000533+0000 mon.a (mon.0) 3107 : cluster [WRN] pg 2.10 is stuck undersized for 3m, current state active+undersized, last acting [2,1] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000540+0000 mon.a (mon.0) 3108 : cluster [WRN] pg 2.15 is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000549+0000 mon.a (mon.0) 3109 : cluster [WRN] pg 2.1d is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000559+0000 mon.a (mon.0) 3110 : cluster [WRN] pg 2.1f is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000568+0000 mon.a (mon.0) 3111 : cluster [WRN] pg 3.3 is stuck undersized for 3m, current state active+undersized, last acting [4,6] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000577+0000 mon.a (mon.0) 3112 : cluster [WRN] pg 3.6 is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000586+0000 mon.a (mon.0) 3113 : cluster [WRN] pg 4.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,4] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000595+0000 mon.a (mon.0) 3114 : cluster [WRN] pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,7] 2024-02-23T00:40:00.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000602+0000 mon.a (mon.0) 3115 : cluster [WRN] pg 4.6 is stuck undersized for 98s, current state active+undersized+degraded, last acting [1,2] 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000610+0000 mon.a (mon.0) 3116 : cluster [WRN] pg 4.7 is stuck undersized for 98s, current state active+undersized+degraded, last acting [1,5] 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000618+0000 mon.a (mon.0) 3117 : cluster [WRN] pg 4.b is stuck undersized for 98s, current state active+undersized+degraded, last acting [1,4] 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:59.272165+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-99", "overlaypool": "test-rados-api-smithi087-82091-99-cache"}]': finished 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:59.272215+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]': finished 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:59.272272+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e580: 8 total, 7 up, 8 in 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:59.273106+0000 mon.a (mon.0) 3090 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-99-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:39:59.739955+0000 mon.a (mon.0) 3091 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "12"}]: dispatch 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000237+0000 mon.a (mon.0) 3092 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull; Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 39 pgs undersized; 2 pool(s) do not have an application enabled 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000282+0000 mon.a (mon.0) 3093 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-23T00:40:00.523 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000301+0000 mon.a (mon.0) 3094 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000324+0000 mon.a (mon.0) 3095 : cluster [WRN] osd.0 (root=default,host=smithi087) is down 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000377+0000 mon.a (mon.0) 3096 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000404+0000 mon.a (mon.0) 3097 : cluster [WRN] pg 4.17 is active+undersized+degraded+remapped+backfill_toofull, acting [1,5] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000426+0000 mon.a (mon.0) 3098 : cluster [WRN] pg 4.1f is active+remapped+backfill_toofull, acting [1,4,7] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000458+0000 mon.a (mon.0) 3099 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 39 pgs undersized 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000469+0000 mon.a (mon.0) 3100 : cluster [WRN] pg 1.0 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000478+0000 mon.a (mon.0) 3101 : cluster [WRN] pg 2.0 is stuck undersized for 3m, current state active+undersized, last acting [7,1] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000486+0000 mon.a (mon.0) 3102 : cluster [WRN] pg 2.4 is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000496+0000 mon.a (mon.0) 3103 : cluster [WRN] pg 2.5 is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000504+0000 mon.a (mon.0) 3104 : cluster [WRN] pg 2.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,1] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000514+0000 mon.a (mon.0) 3105 : cluster [WRN] pg 2.c is stuck undersized for 3m, current state active+undersized, last acting [2,5] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000523+0000 mon.a (mon.0) 3106 : cluster [WRN] pg 2.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,7] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000533+0000 mon.a (mon.0) 3107 : cluster [WRN] pg 2.10 is stuck undersized for 3m, current state active+undersized, last acting [2,1] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000540+0000 mon.a (mon.0) 3108 : cluster [WRN] pg 2.15 is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-02-23T00:40:00.524 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000549+0000 mon.a (mon.0) 3109 : cluster [WRN] pg 2.1d is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000559+0000 mon.a (mon.0) 3110 : cluster [WRN] pg 2.1f is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000568+0000 mon.a (mon.0) 3111 : cluster [WRN] pg 3.3 is stuck undersized for 3m, current state active+undersized, last acting [4,6] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000577+0000 mon.a (mon.0) 3112 : cluster [WRN] pg 3.6 is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000586+0000 mon.a (mon.0) 3113 : cluster [WRN] pg 4.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,4] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000595+0000 mon.a (mon.0) 3114 : cluster [WRN] pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,7] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000602+0000 mon.a (mon.0) 3115 : cluster [WRN] pg 4.6 is stuck undersized for 98s, current state active+undersized+degraded, last acting [1,2] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000610+0000 mon.a (mon.0) 3116 : cluster [WRN] pg 4.7 is stuck undersized for 98s, current state active+undersized+degraded, last acting [1,5] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000618+0000 mon.a (mon.0) 3117 : cluster [WRN] pg 4.b is stuck undersized for 98s, current state active+undersized+degraded, last acting [1,4] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000626+0000 mon.a (mon.0) 3118 : cluster [WRN] pg 4.17 is stuck undersized for 68s, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,5] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000634+0000 mon.a (mon.0) 3119 : cluster [WRN] pg 5.2 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000644+0000 mon.a (mon.0) 3120 : cluster [WRN] pg 5.3 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000653+0000 mon.a (mon.0) 3121 : cluster [WRN] pg 5.5 is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000663+0000 mon.a (mon.0) 3122 : cluster [WRN] pg 5.7 is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000671+0000 mon.a (mon.0) 3123 : cluster [WRN] pg 5.8 is stuck undersized for 3m, current state active+undersized, last acting [2,1] 2024-02-23T00:40:00.525 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000682+0000 mon.a (mon.0) 3124 : cluster [WRN] pg 5.b is stuck undersized for 3m, current state active+undersized, last acting [2,5] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000691+0000 mon.a (mon.0) 3125 : cluster [WRN] pg 5.c is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000698+0000 mon.a (mon.0) 3126 : cluster [WRN] pg 5.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,5] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000709+0000 mon.a (mon.0) 3127 : cluster [WRN] pg 5.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,1] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000718+0000 mon.a (mon.0) 3128 : cluster [WRN] pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000726+0000 mon.a (mon.0) 3129 : cluster [WRN] pg 5.1d is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000736+0000 mon.a (mon.0) 3130 : cluster [WRN] pg 5.1e is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000744+0000 mon.a (mon.0) 3131 : cluster [WRN] pg 6.9 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000752+0000 mon.a (mon.0) 3132 : cluster [WRN] pg 6.a is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000761+0000 mon.a (mon.0) 3133 : cluster [WRN] pg 6.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,1] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000771+0000 mon.a (mon.0) 3134 : cluster [WRN] pg 6.10 is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000779+0000 mon.a (mon.0) 3135 : cluster [WRN] pg 22.0 is stuck undersized for 3m, current state active+undersized, last acting [4,5] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000789+0000 mon.a (mon.0) 3136 : cluster [WRN] pg 22.a is stuck undersized for 3m, current state active+undersized, last acting [2,6] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000797+0000 mon.a (mon.0) 3137 : cluster [WRN] pg 22.18 is stuck undersized for 3m, current state active+undersized, last acting [2,7] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000804+0000 mon.a (mon.0) 3138 : cluster [WRN] pg 22.1a is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-02-23T00:40:00.526 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000813+0000 mon.a (mon.0) 3139 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000821+0000 mon.a (mon.0) 3140 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000833+0000 mon.a (mon.0) 3141 : cluster [WRN] application not enabled on pool 'LibRadosTierECPP_smithi087-82091-94' 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.000842+0000 mon.a (mon.0) 3142 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000626+0000 mon.a (mon.0) 3118 : cluster [WRN] pg 4.17 is stuck undersized for 68s, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,5] 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000634+0000 mon.a (mon.0) 3119 : cluster [WRN] pg 5.2 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000644+0000 mon.a (mon.0) 3120 : cluster [WRN] pg 5.3 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000653+0000 mon.a (mon.0) 3121 : cluster [WRN] pg 5.5 is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000663+0000 mon.a (mon.0) 3122 : cluster [WRN] pg 5.7 is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000671+0000 mon.a (mon.0) 3123 : cluster [WRN] pg 5.8 is stuck undersized for 3m, current state active+undersized, last acting [2,1] 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000682+0000 mon.a (mon.0) 3124 : cluster [WRN] pg 5.b is stuck undersized for 3m, current state active+undersized, last acting [2,5] 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000691+0000 mon.a (mon.0) 3125 : cluster [WRN] pg 5.c is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000698+0000 mon.a (mon.0) 3126 : cluster [WRN] pg 5.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,5] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000709+0000 mon.a (mon.0) 3127 : cluster [WRN] pg 5.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,1] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000718+0000 mon.a (mon.0) 3128 : cluster [WRN] pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000726+0000 mon.a (mon.0) 3129 : cluster [WRN] pg 5.1d is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000736+0000 mon.a (mon.0) 3130 : cluster [WRN] pg 5.1e is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000744+0000 mon.a (mon.0) 3131 : cluster [WRN] pg 6.9 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000752+0000 mon.a (mon.0) 3132 : cluster [WRN] pg 6.a is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000761+0000 mon.a (mon.0) 3133 : cluster [WRN] pg 6.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,1] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000771+0000 mon.a (mon.0) 3134 : cluster [WRN] pg 6.10 is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000779+0000 mon.a (mon.0) 3135 : cluster [WRN] pg 22.0 is stuck undersized for 3m, current state active+undersized, last acting [4,5] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000789+0000 mon.a (mon.0) 3136 : cluster [WRN] pg 22.a is stuck undersized for 3m, current state active+undersized, last acting [2,6] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000797+0000 mon.a (mon.0) 3137 : cluster [WRN] pg 22.18 is stuck undersized for 3m, current state active+undersized, last acting [2,7] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000804+0000 mon.a (mon.0) 3138 : cluster [WRN] pg 22.1a is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-02-23T00:40:00.528 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000813+0000 mon.a (mon.0) 3139 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-02-23T00:40:00.529 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000821+0000 mon.a (mon.0) 3140 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-23T00:40:00.529 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000833+0000 mon.a (mon.0) 3141 : cluster [WRN] application not enabled on pool 'LibRadosTierECPP_smithi087-82091-94' 2024-02-23T00:40:00.529 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.000842+0000 mon.a (mon.0) 3142 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-23T00:40:00.529 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:00.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:59.272165+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-99", "overlaypool": "test-rados-api-smithi087-82091-99-cache"}]': finished 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:59.272215+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "12"}]': finished 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:59.272272+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e580: 8 total, 7 up, 8 in 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:59.273106+0000 mon.a (mon.0) 3090 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-99-cache", "mode": "writeback"}]: dispatch 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:39:59.739955+0000 mon.a (mon.0) 3091 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "12"}]: dispatch 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000237+0000 mon.a (mon.0) 3092 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull; Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 39 pgs undersized; 2 pool(s) do not have an application enabled 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000282+0000 mon.a (mon.0) 3093 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000301+0000 mon.a (mon.0) 3094 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000324+0000 mon.a (mon.0) 3095 : cluster [WRN] osd.0 (root=default,host=smithi087) is down 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000377+0000 mon.a (mon.0) 3096 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000404+0000 mon.a (mon.0) 3097 : cluster [WRN] pg 4.17 is active+undersized+degraded+remapped+backfill_toofull, acting [1,5] 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000426+0000 mon.a (mon.0) 3098 : cluster [WRN] pg 4.1f is active+remapped+backfill_toofull, acting [1,4,7] 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000458+0000 mon.a (mon.0) 3099 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 41/699 objects degraded (5.866%), 11 pgs degraded, 39 pgs undersized 2024-02-23T00:40:00.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000469+0000 mon.a (mon.0) 3100 : cluster [WRN] pg 1.0 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000478+0000 mon.a (mon.0) 3101 : cluster [WRN] pg 2.0 is stuck undersized for 3m, current state active+undersized, last acting [7,1] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000486+0000 mon.a (mon.0) 3102 : cluster [WRN] pg 2.4 is stuck undersized for 3m, current state active+undersized, last acting [1,7] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000496+0000 mon.a (mon.0) 3103 : cluster [WRN] pg 2.5 is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000504+0000 mon.a (mon.0) 3104 : cluster [WRN] pg 2.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,1] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000514+0000 mon.a (mon.0) 3105 : cluster [WRN] pg 2.c is stuck undersized for 3m, current state active+undersized, last acting [2,5] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000523+0000 mon.a (mon.0) 3106 : cluster [WRN] pg 2.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,7] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000533+0000 mon.a (mon.0) 3107 : cluster [WRN] pg 2.10 is stuck undersized for 3m, current state active+undersized, last acting [2,1] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000540+0000 mon.a (mon.0) 3108 : cluster [WRN] pg 2.15 is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000549+0000 mon.a (mon.0) 3109 : cluster [WRN] pg 2.1d is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000559+0000 mon.a (mon.0) 3110 : cluster [WRN] pg 2.1f is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000568+0000 mon.a (mon.0) 3111 : cluster [WRN] pg 3.3 is stuck undersized for 3m, current state active+undersized, last acting [4,6] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000577+0000 mon.a (mon.0) 3112 : cluster [WRN] pg 3.6 is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000586+0000 mon.a (mon.0) 3113 : cluster [WRN] pg 4.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,4] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000595+0000 mon.a (mon.0) 3114 : cluster [WRN] pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,7] 2024-02-23T00:40:00.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000602+0000 mon.a (mon.0) 3115 : cluster [WRN] pg 4.6 is stuck undersized for 98s, current state active+undersized+degraded, last acting [1,2] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000610+0000 mon.a (mon.0) 3116 : cluster [WRN] pg 4.7 is stuck undersized for 98s, current state active+undersized+degraded, last acting [1,5] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000618+0000 mon.a (mon.0) 3117 : cluster [WRN] pg 4.b is stuck undersized for 98s, current state active+undersized+degraded, last acting [1,4] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000626+0000 mon.a (mon.0) 3118 : cluster [WRN] pg 4.17 is stuck undersized for 68s, current state active+undersized+degraded+remapped+backfill_toofull, last acting [1,5] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000634+0000 mon.a (mon.0) 3119 : cluster [WRN] pg 5.2 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000644+0000 mon.a (mon.0) 3120 : cluster [WRN] pg 5.3 is stuck undersized for 3m, current state active+undersized, last acting [6,5] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000653+0000 mon.a (mon.0) 3121 : cluster [WRN] pg 5.5 is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000663+0000 mon.a (mon.0) 3122 : cluster [WRN] pg 5.7 is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000671+0000 mon.a (mon.0) 3123 : cluster [WRN] pg 5.8 is stuck undersized for 3m, current state active+undersized, last acting [2,1] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000682+0000 mon.a (mon.0) 3124 : cluster [WRN] pg 5.b is stuck undersized for 3m, current state active+undersized, last acting [2,5] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000691+0000 mon.a (mon.0) 3125 : cluster [WRN] pg 5.c is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000698+0000 mon.a (mon.0) 3126 : cluster [WRN] pg 5.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,5] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000709+0000 mon.a (mon.0) 3127 : cluster [WRN] pg 5.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,1] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000718+0000 mon.a (mon.0) 3128 : cluster [WRN] pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [5,7] 2024-02-23T00:40:00.558 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000726+0000 mon.a (mon.0) 3129 : cluster [WRN] pg 5.1d is stuck undersized for 3m, current state active+undersized, last acting [1,4] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000736+0000 mon.a (mon.0) 3130 : cluster [WRN] pg 5.1e is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000744+0000 mon.a (mon.0) 3131 : cluster [WRN] pg 6.9 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000752+0000 mon.a (mon.0) 3132 : cluster [WRN] pg 6.a is stuck undersized for 3m, current state active+undersized, last acting [5,6] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000761+0000 mon.a (mon.0) 3133 : cluster [WRN] pg 6.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [5,1] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000771+0000 mon.a (mon.0) 3134 : cluster [WRN] pg 6.10 is stuck undersized for 3m, current state active+undersized, last acting [5,1] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000779+0000 mon.a (mon.0) 3135 : cluster [WRN] pg 22.0 is stuck undersized for 3m, current state active+undersized, last acting [4,5] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000789+0000 mon.a (mon.0) 3136 : cluster [WRN] pg 22.a is stuck undersized for 3m, current state active+undersized, last acting [2,6] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000797+0000 mon.a (mon.0) 3137 : cluster [WRN] pg 22.18 is stuck undersized for 3m, current state active+undersized, last acting [2,7] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000804+0000 mon.a (mon.0) 3138 : cluster [WRN] pg 22.1a is stuck undersized for 3m, current state active+undersized, last acting [1,5] 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000813+0000 mon.a (mon.0) 3139 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000821+0000 mon.a (mon.0) 3140 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000833+0000 mon.a (mon.0) 3141 : cluster [WRN] application not enabled on pool 'LibRadosTierECPP_smithi087-82091-94' 2024-02-23T00:40:00.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.000842+0000 mon.a (mon.0) 3142 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-23T00:40:00.560 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:00.576 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:00.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:00.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:00.680 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:00.711 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:00.909 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:00.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:01.013 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:01.042 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:01.062 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:01.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:01.165 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:01.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 -- 172.21.15.87:0/564328330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59000f8d70 msgr2=0x7f59000f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:01.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 --2- 172.21.15.87:0/564328330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59000f8d70 0x7f59000f9150 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 rx=0x7f58f00047b0 tx=0x7f58f0009cb0).stop 2024-02-23T00:40:01.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 -- 172.21.15.87:0/564328330 shutdown_connections 2024-02-23T00:40:01.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 --2- 172.21.15.87:0/564328330 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59001022d0 0x7f59001046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:01.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 --2- 172.21.15.87:0/564328330 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59000f8d70 0x7f59000f9150 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 --2- 172.21.15.87:0/564328330 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59000f9640 0x7f5900101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 -- 172.21.15.87:0/564328330 >> 172.21.15.87:0/564328330 conn(0x7f59000f2820 msgr2=0x7f59000f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 -- 172.21.15.87:0/564328330 shutdown_connections 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 -- 172.21.15.87:0/564328330 wait complete. 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 librados: starting msgr at 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.173+0000 7f5905818700 1 librados: starting objecter 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 Processor -- start 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 -- start start 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 librados: setting wanted keys 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 librados: calling monclient init 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59000f8d70 0x7f59000f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59000f9640 0x7f59000f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59001022d0 0x7f59000f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5900116740 con 0x7f59001022d0 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59001168b0 con 0x7f59000f8d70 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f5905818700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5900116a70 con 0x7f59000f9640 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f58ff7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59001022d0 0x7f59000f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f58fe7fc700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59000f9640 0x7f59000f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f58ff7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59001022d0 0x7f59000f7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42468/0 (socket says 172.21.15.87:42468) 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f58ff7fe700 1 -- 172.21.15.87:0/3845210887 learned_addr learned my addr 172.21.15.87:0/3845210887 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f58feffd700 1 --2- 172.21.15.87:0/3845210887 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59000f8d70 0x7f59000f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f58fe7fc700 1 -- 172.21.15.87:0/3845210887 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59000f8d70 msgr2=0x7f59000f6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f58fe7fc700 1 --2- 172.21.15.87:0/3845210887 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59000f8d70 0x7f59000f6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f58fe7fc700 1 -- 172.21.15.87:0/3845210887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59001022d0 msgr2=0x7f59000f7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.174+0000 7f58fe7fc700 1 --2- 172.21.15.87:0/3845210887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59001022d0 0x7f59000f7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f58fe7fc700 1 -- 172.21.15.87:0/3845210887 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f4005040 con 0x7f59000f9640 2024-02-23T00:40:01.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f58ff7fe700 1 --2- 172.21.15.87:0/3845210887 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59001022d0 0x7f59000f7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f58fe7fc700 1 --2- 172.21.15.87:0/3845210887 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59000f9640 0x7f59000f7210 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 rx=0x7f58f40091f0 tx=0x7f58f4009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f58e7fff700 10 client.?.objecter ms_handle_connect 0x7f59000f9640 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f58e7fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f400e030 con 0x7f59000f9640 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f0005040 con 0x7f59000f9640 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58f400f040 con 0x7f59000f9640 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40147e0 con 0x7f59000f9640 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f5905818700 10 client.38030.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5900117100 con 0x7f59000f9640 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.175+0000 7f5905818700 1 librados: init done 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.176+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f590005fa60 con 0x7f59000f9640 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.177+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f58f4014b90 con 0x7f59000f9640 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.177+0000 7f58e7fff700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6800/1231597463 conn(0x7f58e8062c90 0x7f58e8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58feffd700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6800/1231597463 conn(0x7f58e8062c90 0x7f58e8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(581..581 src has 173..581) v4 ==== 10253+0+0 (secure 0 0 0) 0x7f58f407df10 con 0x7f59000f9640 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 osd_map(581..581 src has 173..581) v4 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58e7fff700 3 client.38030.objecter handle_osd_map got epochs [581,581] > 0 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58e7fff700 3 client.38030.objecter handle_osd_map decoding full epoch 581 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58e7fff700 20 client.38030.objecter dump_active .. 0 homeless 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v581) v1 ==== 54+0+16 (secure 0 0 0) 0x7f58f407e3b0 con 0x7f59000f9640 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58e7fff700 10 client.38030.objecter ms_handle_connect 0x7f58e8062c90 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58feffd700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6800/1231597463 conn(0x7f58e8062c90 0x7f58e8065150 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 rx=0x7f58f0004e20 tx=0x7f58f0009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 10 client.38030.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 1 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 20 client.38030.objecter _get_session osd=-1 returning homeless 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 10 client.38030.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=582}) v3 -- 0x7f590005ee80 con 0x7f59000f9640 2024-02-23T00:40:01.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f59001072e0 con 0x7f59000f9640 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=581) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58f4050b50 con 0x7f59000f9640 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58fd7fa700 10 client.38030.objecter _check_command_map_dne tid 1 current 581 map_dne_bound 581 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58fd7fa700 10 client.38030.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f58fd7fa700 15 client.38030.objecter _session_command_op_remove -1 1 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 10 client.38030.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 2 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6805/685354171 conn(0x7f59000f4ef0 0x7f59000273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59000f4d80 osd=1 3 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59000f4d80 osd=1 3 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59000f4d80 osd=1 3 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 2 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f59000f4d80 osd=1 3 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 1 2 2024-02-23T00:40:01.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59000f4d80 osd=1 4 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 10 client.38030.objecter _send_command 2 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.178+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5900059e00 con 0x7f59000f4ef0 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.179+0000 7f58ff7fe700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6805/685354171 conn(0x7f59000f4ef0 0x7f59000273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.179+0000 7f58e7fff700 10 client.38030.objecter ms_handle_connect 0x7f59000f4ef0 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.179+0000 7f58ff7fe700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6805/685354171 conn(0x7f59000f4ef0 0x7f59000273c0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.179+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5900059e00 con 0x7f59000f4ef0 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.179+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.179+0000 7f58e7fff700 10 client.38030.objecter _finish_command 2 = system:0 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.179+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f59000f4d80 osd=1 4 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.179+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 1 2 2024-02-23T00:40:01.197 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:01.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 10 client.38030.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:01.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 3 2024-02-23T00:40:01.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59000f4d80 osd=1 3 2024-02-23T00:40:01.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59000f4d80 osd=1 3 2024-02-23T00:40:01.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59000f4d80 osd=1 3 2024-02-23T00:40:01.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 3 2024-02-23T00:40:01.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f59000f4d80 osd=1 3 2024-02-23T00:40:01.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 1 3 2024-02-23T00:40:01.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59000f4d80 osd=1 4 2024-02-23T00:40:01.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 10 client.38030.objecter _send_command 3 2024-02-23T00:40:01.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.196+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f59000278b0 con 0x7f59000f4ef0 2024-02-23T00:40:01.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f59000278b0 con 0x7f59000f4ef0 2024-02-23T00:40:01.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:01.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f58e7fff700 10 client.38030.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f59000f4d80 osd=1 4 2024-02-23T00:40:01.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 1 3 2024-02-23T00:40:01.200 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:40:01.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 10 client.38030.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:01.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 4 2024-02-23T00:40:01.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6809/3605394228 conn(0x7f59000594d0 0x7f590011b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900059360 osd=2 3 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900059360 osd=2 3 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900059360 osd=2 3 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 4 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f5900059360 osd=2 3 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 2 4 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900059360 osd=2 4 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 10 client.38030.objecter _send_command 4 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5900028760 con 0x7f59000594d0 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f58feffd700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6809/3605394228 conn(0x7f59000594d0 0x7f590011b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f58e7fff700 10 client.38030.objecter ms_handle_connect 0x7f59000594d0 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.198+0000 7f58feffd700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6809/3605394228 conn(0x7f59000594d0 0x7f590011b310 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.199+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5900028760 con 0x7f59000594d0 2024-02-23T00:40:01.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.199+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:40:01.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.199+0000 7f58e7fff700 10 client.38030.objecter _finish_command 4 = system:0 2024-02-23T00:40:01.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.199+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f5900059360 osd=2 4 2024-02-23T00:40:01.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.199+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 2 4 2024-02-23T00:40:01.219 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:01.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 10 client.38030.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:01.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 5 2024-02-23T00:40:01.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900059360 osd=2 3 2024-02-23T00:40:01.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900059360 osd=2 3 2024-02-23T00:40:01.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900059360 osd=2 3 2024-02-23T00:40:01.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 5 2024-02-23T00:40:01.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f5900059360 osd=2 3 2024-02-23T00:40:01.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 2 5 2024-02-23T00:40:01.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900059360 osd=2 4 2024-02-23T00:40:01.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 10 client.38030.objecter _send_command 5 2024-02-23T00:40:01.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.220+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5900027cc0 con 0x7f59000594d0 2024-02-23T00:40:01.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:01.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f5900027cc0 con 0x7f59000594d0 2024-02-23T00:40:01.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:01.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f58e7fff700 10 client.38030.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f5900059360 osd=2 4 2024-02-23T00:40:01.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 2 5 2024-02-23T00:40:01.227 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:40:01.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 10 client.38030.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 6 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6813/128059790 conn(0x7f59000fca70 0x7f590011e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59000280d0 osd=3 3 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59000280d0 osd=3 3 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59000280d0 osd=3 3 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 6 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f59000280d0 osd=3 3 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 3 6 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59000280d0 osd=3 4 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 10 client.38030.objecter _send_command 6 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.224+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5900121730 con 0x7f59000fca70 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.225+0000 7f58fe7fc700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6813/128059790 conn(0x7f59000fca70 0x7f590011e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.225+0000 7f58e7fff700 10 client.38030.objecter ms_handle_connect 0x7f59000fca70 2024-02-23T00:40:01.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.225+0000 7f58fe7fc700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.87:6813/128059790 conn(0x7f59000fca70 0x7f590011e990 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:01.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.226+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5900121730 con 0x7f59000fca70 2024-02-23T00:40:01.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.226+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:40:01.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.226+0000 7f58e7fff700 10 client.38030.objecter _finish_command 6 = system:0 2024-02-23T00:40:01.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.226+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f59000280d0 osd=3 4 2024-02-23T00:40:01.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.226+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 3 6 2024-02-23T00:40:01.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 10 client.38030.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:01.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 7 2024-02-23T00:40:01.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59000280d0 osd=3 3 2024-02-23T00:40:01.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59000280d0 osd=3 3 2024-02-23T00:40:01.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59000280d0 osd=3 3 2024-02-23T00:40:01.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 7 2024-02-23T00:40:01.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f59000280d0 osd=3 3 2024-02-23T00:40:01.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 3 7 2024-02-23T00:40:01.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59000280d0 osd=3 4 2024-02-23T00:40:01.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 10 client.38030.objecter _send_command 7 2024-02-23T00:40:01.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.246+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f59000fc090 con 0x7f59000fca70 2024-02-23T00:40:01.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f59000fc090 con 0x7f59000fca70 2024-02-23T00:40:01.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:01.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f58e7fff700 10 client.38030.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f59000280d0 osd=3 4 2024-02-23T00:40:01.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 3 7 2024-02-23T00:40:01.250 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:40:01.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 10 client.38030.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:01.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 8 2024-02-23T00:40:01.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6800/4175844757 conn(0x7f590011ee80 0x7f590011f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f590011bfd0 osd=4 3 2024-02-23T00:40:01.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f590011bfd0 osd=4 3 2024-02-23T00:40:01.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f590011bfd0 osd=4 3 2024-02-23T00:40:01.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 8 2024-02-23T00:40:01.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f590011bfd0 osd=4 3 2024-02-23T00:40:01.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 4 8 2024-02-23T00:40:01.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f590011bfd0 osd=4 4 2024-02-23T00:40:01.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 10 client.38030.objecter _send_command 8 2024-02-23T00:40:01.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590011c7b0 con 0x7f590011ee80 2024-02-23T00:40:01.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.248+0000 7f58ff7fe700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6800/4175844757 conn(0x7f590011ee80 0x7f590011f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.249+0000 7f58e7fff700 10 client.38030.objecter ms_handle_connect 0x7f590011ee80 2024-02-23T00:40:01.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.249+0000 7f58ff7fe700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6800/4175844757 conn(0x7f590011ee80 0x7f590011f2c0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:01.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.249+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f590011c7b0 con 0x7f590011ee80 2024-02-23T00:40:01.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.249+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:40:01.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.249+0000 7f58e7fff700 10 client.38030.objecter _finish_command 8 = system:0 2024-02-23T00:40:01.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.249+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f590011bfd0 osd=4 4 2024-02-23T00:40:01.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.249+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 4 8 2024-02-23T00:40:01.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 10 client.38030.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:01.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 9 2024-02-23T00:40:01.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f590011bfd0 osd=4 3 2024-02-23T00:40:01.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f590011bfd0 osd=4 3 2024-02-23T00:40:01.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f590011bfd0 osd=4 3 2024-02-23T00:40:01.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 9 2024-02-23T00:40:01.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f590011bfd0 osd=4 3 2024-02-23T00:40:01.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 4 9 2024-02-23T00:40:01.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f590011bfd0 osd=4 4 2024-02-23T00:40:01.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 10 client.38030.objecter _send_command 9 2024-02-23T00:40:01.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.265+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5900121100 con 0x7f590011ee80 2024-02-23T00:40:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.266+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f5900121100 con 0x7f590011ee80 2024-02-23T00:40:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.266+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.266+0000 7f58e7fff700 10 client.38030.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.266+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f590011bfd0 osd=4 4 2024-02-23T00:40:01.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.266+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 4 9 2024-02-23T00:40:01.269 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:40:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 10 client.38030.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 10 2024-02-23T00:40:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6804/2550013795 conn(0x7f5900123960 0x7f5900125d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900120930 osd=5 3 2024-02-23T00:40:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900120930 osd=5 3 2024-02-23T00:40:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900120930 osd=5 3 2024-02-23T00:40:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 10 2024-02-23T00:40:01.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f5900120930 osd=5 3 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 5 10 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900120930 osd=5 4 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 10 client.38030.objecter _send_command 10 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5900121100 con 0x7f5900123960 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f58feffd700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6804/2550013795 conn(0x7f5900123960 0x7f5900125d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f58e7fff700 10 client.38030.objecter ms_handle_connect 0x7f5900123960 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.267+0000 7f58feffd700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6804/2550013795 conn(0x7f5900123960 0x7f5900125d50 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.268+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5900121100 con 0x7f5900123960 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.268+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.268+0000 7f58e7fff700 10 client.38030.objecter _finish_command 10 = system:0 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.268+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f5900120930 osd=5 4 2024-02-23T00:40:01.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.268+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 5 10 2024-02-23T00:40:01.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.281+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== mon.2 v2:172.21.15.87:3301/0 8 ==== osd_map(582..582 src has 173..582) v4 ==== 743+0+0 (secure 0 0 0) 0x7f58f4014e40 con 0x7f59000f9640 2024-02-23T00:40:01.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.281+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 osd_map(582..582 src has 173..582) v4 2024-02-23T00:40:01.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.281+0000 7f58e7fff700 3 client.38030.objecter handle_osd_map got epochs [582,582] > 581 2024-02-23T00:40:01.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.281+0000 7f58e7fff700 3 client.38030.objecter handle_osd_map decoding incremental epoch 582 2024-02-23T00:40:01.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.281+0000 7f58e7fff700 20 client.38030.objecter dump_active .. 0 homeless 2024-02-23T00:40:01.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 10 client.38030.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 11 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900120930 osd=5 3 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900120930 osd=5 3 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900120930 osd=5 3 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 11 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f5900120930 osd=5 3 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 5 11 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900120930 osd=5 4 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 10 client.38030.objecter _send_command 11 2024-02-23T00:40:01.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.287+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5900121100 con 0x7f5900123960 2024-02-23T00:40:01.299 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.295+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f5900121100 con 0x7f5900123960 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.295+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.295+0000 7f58e7fff700 10 client.38030.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.295+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f5900120930 osd=5 4 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.295+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 5 11 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 10 client.38030.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 12 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6808/2509408498 conn(0x7f5900126750 0x7f5900126b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900121100 osd=6 3 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900121100 osd=6 3 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900121100 osd=6 3 2024-02-23T00:40:01.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 12 2024-02-23T00:40:01.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f5900121100 osd=6 3 2024-02-23T00:40:01.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 6 12 2024-02-23T00:40:01.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900121100 osd=6 4 2024-02-23T00:40:01.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 10 client.38030.objecter _send_command 12 2024-02-23T00:40:01.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.297+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5900128730 con 0x7f5900126750 2024-02-23T00:40:01.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.301+0000 7f58fe7fc700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6808/2509408498 conn(0x7f5900126750 0x7f5900126b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.305+0000 7f58e7fff700 10 client.38030.objecter ms_handle_connect 0x7f5900126750 2024-02-23T00:40:01.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.305+0000 7f58fe7fc700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6808/2509408498 conn(0x7f5900126750 0x7f5900126b70 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:01.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.305+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5900128730 con 0x7f5900126750 2024-02-23T00:40:01.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.305+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:40:01.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.305+0000 7f58e7fff700 10 client.38030.objecter _finish_command 12 = system:0 2024-02-23T00:40:01.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.305+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f5900121100 osd=6 4 2024-02-23T00:40:01.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.305+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 6 12 2024-02-23T00:40:01.325 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:01.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 10 client.38030.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:01.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 13 2024-02-23T00:40:01.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900121100 osd=6 3 2024-02-23T00:40:01.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900121100 osd=6 3 2024-02-23T00:40:01.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f5900121100 osd=6 3 2024-02-23T00:40:01.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 13 2024-02-23T00:40:01.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f5900121100 osd=6 3 2024-02-23T00:40:01.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 6 13 2024-02-23T00:40:01.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f5900121100 osd=6 4 2024-02-23T00:40:01.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 10 client.38030.objecter _send_command 13 2024-02-23T00:40:01.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.323+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5900127150 con 0x7f5900126750 2024-02-23T00:40:01.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.330+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f58f405a020 con 0x7f5900126750 2024-02-23T00:40:01.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.330+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:01.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.330+0000 7f58e7fff700 10 client.38030.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.330+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f5900121100 osd=6 4 2024-02-23T00:40:01.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.330+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 6 13 2024-02-23T00:40:01.333 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:40:01.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 10 client.38030.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:01.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 14 2024-02-23T00:40:01.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6812/373787000 conn(0x7f5900127940 0x7f5900127da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:01.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59001272c0 osd=7 3 2024-02-23T00:40:01.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59001272c0 osd=7 3 2024-02-23T00:40:01.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59001272c0 osd=7 3 2024-02-23T00:40:01.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 14 2024-02-23T00:40:01.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f59001272c0 osd=7 3 2024-02-23T00:40:01.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 7 14 2024-02-23T00:40:01.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59001272c0 osd=7 4 2024-02-23T00:40:01.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 10 client.38030.objecter _send_command 14 2024-02-23T00:40:01.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5900128370 con 0x7f5900127940 2024-02-23T00:40:01.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f58ff7fe700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6812/373787000 conn(0x7f5900127940 0x7f5900127da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:01.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f58e7fff700 10 client.38030.objecter ms_handle_connect 0x7f5900127940 2024-02-23T00:40:01.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.331+0000 7f58ff7fe700 1 --2- 172.21.15.87:0/3845210887 >> v2:172.21.15.171:6812/373787000 conn(0x7f5900127940 0x7f5900127da0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:01.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.332+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5900128370 con 0x7f5900127940 2024-02-23T00:40:01.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.332+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:40:01.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.332+0000 7f58e7fff700 10 client.38030.objecter _finish_command 14 = system:0 2024-02-23T00:40:01.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.332+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f59001272c0 osd=7 4 2024-02-23T00:40:01.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.332+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 7 14 2024-02-23T00:40:01.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 10 client.38030.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:01.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign -1 15 2024-02-23T00:40:01.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59001272c0 osd=7 3 2024-02-23T00:40:01.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59001272c0 osd=7 3 2024-02-23T00:40:01.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 20 client.38030.objecter _get_session s=0x7f59001272c0 osd=7 3 2024-02-23T00:40:01.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 15 client.38030.objecter _session_command_op_remove -1 15 2024-02-23T00:40:01.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 20 client.38030.objecter get_session s=0x7f59001272c0 osd=7 3 2024-02-23T00:40:01.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 15 client.38030.objecter _session_command_op_assign 7 15 2024-02-23T00:40:01.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 20 client.38030.objecter put_session s=0x7f59001272c0 osd=7 4 2024-02-23T00:40:01.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 10 client.38030.objecter _send_command 15 2024-02-23T00:40:01.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.347+0000 7f5905818700 1 -- 172.21.15.87:0/3845210887 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5900128370 con 0x7f5900127940 2024-02-23T00:40:01.352 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:40:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:40:01.352 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:39:59.738929+0000 mgr.y (mgr.24437) 525 : cluster [DBG] pgmap v887: 229 pgs: 1 active+remapped+backfill_toofull, 6 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 34 unknown, 26 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/699 objects degraded (5.866%); 61/699 objects misplaced (8.727%) 2024-02-23T00:40:01.352 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.270762+0000 mon.a (mon.0) 3143 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:40:01.352 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:00.282654+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-99-cache", "mode": "writeback"}]': finished 2024-02-23T00:40:01.353 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:00.282723+0000 mon.a (mon.0) 3145 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "12"}]': finished 2024-02-23T00:40:01.353 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:00.282770+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e581: 8 total, 7 up, 8 in 2024-02-23T00:40:01.353 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:00.290362+0000 mon.a (mon.0) 3147 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:40:01.353 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:01.177947+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? 172.21.15.87:0/3845210887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:01.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.349+0000 7f58e7fff700 1 -- 172.21.15.87:0/3845210887 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f5900128370 con 0x7f5900127940 2024-02-23T00:40:01.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.349+0000 7f58e7fff700 10 client.38030.objecter ms_dispatch 0x7f5900001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:01.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.349+0000 7f58e7fff700 10 client.38030.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.349+0000 7f58e7fff700 20 client.38030.objecter put_session s=0x7f59001272c0 osd=7 4 2024-02-23T00:40:01.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:01.349+0000 7f58e7fff700 15 client.38030.objecter _session_command_op_remove 7 15 2024-02-23T00:40:01.354 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:40:01.354 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:40:01.354 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.354 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.354 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.354 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.354 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.355 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.355 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:40:01.355 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:01.377 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:01.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:01.481 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:39:59.738929+0000 mgr.y (mgr.24437) 525 : cluster [DBG] pgmap v887: 229 pgs: 1 active+remapped+backfill_toofull, 6 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 34 unknown, 26 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/699 objects degraded (5.866%); 61/699 objects misplaced (8.727%) 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.270762+0000 mon.a (mon.0) 3143 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:00.282654+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-99-cache", "mode": "writeback"}]': finished 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:00.282723+0000 mon.a (mon.0) 3145 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "12"}]': finished 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:00.282770+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e581: 8 total, 7 up, 8 in 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:00.290362+0000 mon.a (mon.0) 3147 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:01.177947+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? 172.21.15.87:0/3845210887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:39:59.738929+0000 mgr.y (mgr.24437) 525 : cluster [DBG] pgmap v887: 229 pgs: 1 active+remapped+backfill_toofull, 6 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 34 unknown, 26 active+clean+remapped, 10 active+undersized+degraded, 32 active+undersized, 119 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 41/699 objects degraded (5.866%); 61/699 objects misplaced (8.727%) 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.270762+0000 mon.a (mon.0) 3143 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:40:01.507 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:00.282654+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-99-cache", "mode": "writeback"}]': finished 2024-02-23T00:40:01.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:00.282723+0000 mon.a (mon.0) 3145 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "12"}]': finished 2024-02-23T00:40:01.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:00.282770+0000 mon.a (mon.0) 3146 : cluster [DBG] osdmap e581: 8 total, 7 up, 8 in 2024-02-23T00:40:01.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:00.290362+0000 mon.a (mon.0) 3147 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:40:01.508 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:01.177947+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? 172.21.15.87:0/3845210887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:01.510 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:02.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:02] "GET /metrics HTTP/1.1" 200 187142 "" "Prometheus/2.33.4" 2024-02-23T00:40:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:01.092996+0000 mgr.y (mgr.24437) 526 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:01.284468+0000 mon.a (mon.0) 3148 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:01.284604+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e582: 8 total, 7 up, 8 in 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:01.285551+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:02.166402+0000 mon.a (mon.0) 3151 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:02.166529+0000 mon.a (mon.0) 3152 : cluster [DBG] osdmap e583: 8 total, 7 up, 8 in 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:02.167158+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:01.092996+0000 mgr.y (mgr.24437) 526 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:01.284468+0000 mon.a (mon.0) 3148 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:01.284604+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e582: 8 total, 7 up, 8 in 2024-02-23T00:40:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:01.285551+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:40:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:02.166402+0000 mon.a (mon.0) 3151 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:40:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:02.166529+0000 mon.a (mon.0) 3152 : cluster [DBG] osdmap e583: 8 total, 7 up, 8 in 2024-02-23T00:40:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:02.167158+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:40:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:01.092996+0000 mgr.y (mgr.24437) 526 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:01.284468+0000 mon.a (mon.0) 3148 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:40:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:01.284604+0000 mon.a (mon.0) 3149 : cluster [DBG] osdmap e582: 8 total, 7 up, 8 in 2024-02-23T00:40:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:01.285551+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:40:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:02.166402+0000 mon.a (mon.0) 3151 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:40:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:02.166529+0000 mon.a (mon.0) 3152 : cluster [DBG] osdmap e583: 8 total, 7 up, 8 in 2024-02-23T00:40:02.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:02.167158+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:40:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:01.739750+0000 mgr.y (mgr.24437) 527 : cluster [DBG] pgmap v890: 229 pgs: 3 remapped+peering, 8 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 9 active+undersized+degraded, 50 active+undersized, 134 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 40/693 objects degraded (5.772%); 61/693 objects misplaced (8.802%) 2024-02-23T00:40:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:02.281981+0000 mon.a (mon.0) 3154 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:40:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:02.282010+0000 mon.a (mon.0) 3155 : cluster [WRN] Health check update: Degraded data redundancy: 40/693 objects degraded (5.772%), 10 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:03.163101+0000 mon.a (mon.0) 3156 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:40:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:03.167763+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:40:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:03.167865+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e584: 8 total, 7 up, 8 in 2024-02-23T00:40:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:03.175818+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-23T00:40:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:01.739750+0000 mgr.y (mgr.24437) 527 : cluster [DBG] pgmap v890: 229 pgs: 3 remapped+peering, 8 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 9 active+undersized+degraded, 50 active+undersized, 134 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 40/693 objects degraded (5.772%); 61/693 objects misplaced (8.802%) 2024-02-23T00:40:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:02.281981+0000 mon.a (mon.0) 3154 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:40:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:02.282010+0000 mon.a (mon.0) 3155 : cluster [WRN] Health check update: Degraded data redundancy: 40/693 objects degraded (5.772%), 10 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:03.163101+0000 mon.a (mon.0) 3156 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:40:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:03.167763+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:40:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:03.167865+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e584: 8 total, 7 up, 8 in 2024-02-23T00:40:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:03.175818+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-23T00:40:03.666 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:03.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:01.739750+0000 mgr.y (mgr.24437) 527 : cluster [DBG] pgmap v890: 229 pgs: 3 remapped+peering, 8 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 9 active+undersized+degraded, 50 active+undersized, 134 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 40/693 objects degraded (5.772%); 61/693 objects misplaced (8.802%) 2024-02-23T00:40:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:02.281981+0000 mon.a (mon.0) 3154 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:40:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:02.282010+0000 mon.a (mon.0) 3155 : cluster [WRN] Health check update: Degraded data redundancy: 40/693 objects degraded (5.772%), 10 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:03.163101+0000 mon.a (mon.0) 3156 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:40:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:03.167763+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:40:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:03.167865+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e584: 8 total, 7 up, 8 in 2024-02-23T00:40:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:03.175818+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-23T00:40:03.771 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:03.800 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:04.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:40:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.342+0000 7f714fe54700 1 -- 172.21.15.87:0/4146570259 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f73f0 msgr2=0x7f7148104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.342+0000 7f714fe54700 1 --2- 172.21.15.87:0/4146570259 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f73f0 0x7f7148104190 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f7138004870 tx=0x7f7138009cb0).stop 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.342+0000 7f714fe54700 1 -- 172.21.15.87:0/4146570259 shutdown_connections 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.342+0000 7f714fe54700 1 --2- 172.21.15.87:0/4146570259 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7148104680 0x7f7148106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.342+0000 7f714fe54700 1 --2- 172.21.15.87:0/4146570259 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f71480f6b20 0x7f71480f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.342+0000 7f714fe54700 1 --2- 172.21.15.87:0/4146570259 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f73f0 0x7f7148104190 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.342+0000 7f714fe54700 1 -- 172.21.15.87:0/4146570259 >> 172.21.15.87:0/4146570259 conn(0x7f71480f2820 msgr2=0x7f71480f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.342+0000 7f714fe54700 1 -- 172.21.15.87:0/4146570259 shutdown_connections 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.342+0000 7f714fe54700 1 -- 172.21.15.87:0/4146570259 wait complete. 2024-02-23T00:40:04.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 librados: starting msgr at 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 librados: starting objecter 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 Processor -- start 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 -- start start 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 librados: setting wanted keys 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 librados: calling monclient init 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f6b20 0x7f71481103f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f71480f73f0 0x7f71481108e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7148104680 0x7f714810a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714810aa80 con 0x7f71480f73f0 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714810abf0 con 0x7f7148104680 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714fe54700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714810adb0 con 0x7f71480f6b20 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714d3ef700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f71480f73f0 0x7f71481108e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714dbf0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f6b20 0x7f71481103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:04.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714dbf0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f6b20 0x7f71481103f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:49854/0 (socket says 172.21.15.87:49854) 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714d3ef700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f71480f73f0 0x7f71481108e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54406/0 (socket says 172.21.15.87:54406) 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714dbf0700 1 -- 172.21.15.87:0/476650199 learned_addr learned my addr 172.21.15.87:0/476650199 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.343+0000 7f714e3f1700 1 --2- 172.21.15.87:0/476650199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7148104680 0x7f714810a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714dbf0700 1 -- 172.21.15.87:0/476650199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7148104680 msgr2=0x7f714810a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714dbf0700 1 --2- 172.21.15.87:0/476650199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7148104680 0x7f714810a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714dbf0700 1 -- 172.21.15.87:0/476650199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f71480f73f0 msgr2=0x7f71481108e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714dbf0700 1 --2- 172.21.15.87:0/476650199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f71480f73f0 0x7f71481108e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714dbf0700 1 -- 172.21.15.87:0/476650199 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144005040 con 0x7f71480f6b20 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714d3ef700 1 --2- 172.21.15.87:0/476650199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f71480f73f0 0x7f71481108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714dbf0700 1 --2- 172.21.15.87:0/476650199 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f6b20 0x7f71481103f0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f71440048f0 tx=0x7f7144009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f713effd700 10 client.?.objecter ms_handle_connect 0x7f71480f6b20 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f713effd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:04.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f713effd700 1 -- 172.21.15.87:0/476650199 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144007b00 con 0x7f71480f6b20 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f713effd700 1 -- 172.21.15.87:0/476650199 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f714400f040 con 0x7f71480f6b20 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71400097c0 con 0x7f71480f6b20 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f713effd700 1 -- 172.21.15.87:0/476650199 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144014990 con 0x7f71480f6b20 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714fe54700 10 client.38036.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7138005040 con 0x7f71480f6b20 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.344+0000 7f714fe54700 1 librados: init done 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.345+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714805fa60 con 0x7f71480f6b20 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.345+0000 7f713effd700 1 -- 172.21.15.87:0/476650199 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7144010500 con 0x7f71480f6b20 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.346+0000 7f713effd700 1 --2- 172.21.15.87:0/476650199 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7134062c90 0x7f7134065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.346+0000 7f713effd700 1 -- 172.21.15.87:0/476650199 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(585..585 src has 173..585) v4 ==== 10187+0+0 (secure 0 0 0) 0x7f714407d5e0 con 0x7f71480f6b20 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.346+0000 7f713effd700 10 client.38036.objecter ms_dispatch 0x7f7148001550 osd_map(585..585 src has 173..585) v4 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.346+0000 7f713effd700 3 client.38036.objecter handle_osd_map got epochs [585,585] > 0 2024-02-23T00:40:04.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.346+0000 7f713effd700 3 client.38036.objecter handle_osd_map decoding full epoch 585 2024-02-23T00:40:04.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.346+0000 7f714d3ef700 1 --2- 172.21.15.87:0/476650199 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7134062c90 0x7f7134065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:04.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.346+0000 7f713effd700 20 client.38036.objecter dump_active .. 0 homeless 2024-02-23T00:40:04.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.346+0000 7f713effd700 10 client.38036.objecter ms_handle_connect 0x7f7134062c90 2024-02-23T00:40:04.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.346+0000 7f714d3ef700 1 --2- 172.21.15.87:0/476650199 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7134062c90 0x7f7134065150 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 rx=0x7f7138004ee0 tx=0x7f713800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:04.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.354+0000 7f713effd700 1 -- 172.21.15.87:0/476650199 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7144049f60 con 0x7f71480f6b20 2024-02-23T00:40:04.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:04.534+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f71481093f0 con 0x7f71480f6b20 2024-02-23T00:40:05.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.176+0000 7f713effd700 1 -- 172.21.15.87:0/476650199 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v586) v1 ==== 91+0+0 (secure 0 0 0) 0x7f7144050260 con 0x7f71480f6b20 2024-02-23T00:40:05.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.179+0000 7f714fe54700 10 librados: watch_flush enter 2024-02-23T00:40:05.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.179+0000 7f714fe54700 10 librados: watch_flush exit 2024-02-23T00:40:05.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.179+0000 7f714fe54700 20 client.38036.objecter shutdown clearing up homeless session... 2024-02-23T00:40:05.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.179+0000 7f714fe54700 10 client.38036.objecter successfully canceled tick 2024-02-23T00:40:05.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7134062c90 msgr2=0x7f7134065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:05.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 --2- 172.21.15.87:0/476650199 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7134062c90 0x7f7134065150 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 rx=0x7f7138004ee0 tx=0x7f713800b040).stop 2024-02-23T00:40:05.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f6b20 msgr2=0x7f71481103f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:05.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 --2- 172.21.15.87:0/476650199 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f6b20 0x7f71481103f0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 rx=0x7f71440048f0 tx=0x7f7144009cb0).stop 2024-02-23T00:40:05.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 shutdown_connections 2024-02-23T00:40:05.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 --2- 172.21.15.87:0/476650199 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7134062c90 0x7f7134065150 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:05.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 --2- 172.21.15.87:0/476650199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7148104680 0x7f714810a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:05.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 --2- 172.21.15.87:0/476650199 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f71480f6b20 0x7f71481103f0 secure :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 rx=0x7f71440048f0 tx=0x7f7144009cb0).stop 2024-02-23T00:40:05.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 --2- 172.21.15.87:0/476650199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f71480f73f0 0x7f71481108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:05.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.184+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 >> 172.21.15.87:0/476650199 conn(0x7f71480f2820 msgr2=0x7f71480f4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:05.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.185+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 shutdown_connections 2024-02-23T00:40:05.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.185+0000 7f714fe54700 1 -- 172.21.15.87:0/476650199 wait complete. 2024-02-23T00:40:05.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:05.185+0000 7f714fe54700 1 librados: shutdown 2024-02-23T00:40:05.202 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:40:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:40:05.206 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:40:05.271 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:40:05.271 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:40:05.271 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-02-23T00:40:05.271 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-23T00:40:05.271 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-23T00:40:05.272 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-02-23T00:40:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:03.740498+0000 mgr.y (mgr.24437) 528 : cluster [DBG] pgmap v893: 229 pgs: 3 remapped+peering, 8 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 9 active+undersized+degraded, 50 active+undersized, 134 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 40/693 objects degraded (5.772%); 61/693 objects misplaced (8.802%) 2024-02-23T00:40:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:04.168817+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-23T00:40:05.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:04.168878+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e585: 8 total, 7 up, 8 in 2024-02-23T00:40:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:04.220080+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:40:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:04.535658+0000 mon.c (mon.2) 471 : audit [INF] from='client.? 172.21.15.87:0/476650199' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:40:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:04.535950+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:40:05.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:03.740498+0000 mgr.y (mgr.24437) 528 : cluster [DBG] pgmap v893: 229 pgs: 3 remapped+peering, 8 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 9 active+undersized+degraded, 50 active+undersized, 134 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 40/693 objects degraded (5.772%); 61/693 objects misplaced (8.802%) 2024-02-23T00:40:05.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:04.168817+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-23T00:40:05.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:04.168878+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e585: 8 total, 7 up, 8 in 2024-02-23T00:40:05.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:04.220080+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:40:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:04.535658+0000 mon.c (mon.2) 471 : audit [INF] from='client.? 172.21.15.87:0/476650199' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:40:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:04.535950+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:40:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:03.740498+0000 mgr.y (mgr.24437) 528 : cluster [DBG] pgmap v893: 229 pgs: 3 remapped+peering, 8 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 9 active+undersized+degraded, 50 active+undersized, 134 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 40/693 objects degraded (5.772%); 61/693 objects misplaced (8.802%) 2024-02-23T00:40:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:04.168817+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-23T00:40:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:04.168878+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e585: 8 total, 7 up, 8 in 2024-02-23T00:40:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:04.220080+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:40:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:04.535658+0000 mon.c (mon.2) 471 : audit [INF] from='client.? 172.21.15.87:0/476650199' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:40:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:04.535950+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:40:06.051 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:06.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:06.156 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:06.184 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:06.205 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:06.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:06.309 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:05.166309+0000 mon.a (mon.0) 3164 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:05.176113+0000 mon.a (mon.0) 3165 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:40:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:05.176160+0000 mon.a (mon.0) 3166 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:40:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:05.176202+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e586: 8 total, 7 up, 8 in 2024-02-23T00:40:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:05.183475+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]: dispatch 2024-02-23T00:40:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:05.742671+0000 mon.a (mon.0) 3169 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-23T00:40:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:06.083511+0000 osd.1 (osd.1) 267 : cluster [DBG] 22.b scrub starts 2024-02-23T00:40:06.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:06.086631+0000 osd.1 (osd.1) 268 : cluster [DBG] 22.b scrub ok 2024-02-23T00:40:06.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:05.166309+0000 mon.a (mon.0) 3164 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:06.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:05.176113+0000 mon.a (mon.0) 3165 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:40:06.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:05.176160+0000 mon.a (mon.0) 3166 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:40:06.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:05.176202+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e586: 8 total, 7 up, 8 in 2024-02-23T00:40:06.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:05.183475+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]: dispatch 2024-02-23T00:40:06.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:05.742671+0000 mon.a (mon.0) 3169 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-23T00:40:06.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:06.083511+0000 osd.1 (osd.1) 267 : cluster [DBG] 22.b scrub starts 2024-02-23T00:40:06.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:06.086631+0000 osd.1 (osd.1) 268 : cluster [DBG] 22.b scrub ok 2024-02-23T00:40:06.335 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:06.358 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:06.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:05.166309+0000 mon.a (mon.0) 3164 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:05.176113+0000 mon.a (mon.0) 3165 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:40:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:05.176160+0000 mon.a (mon.0) 3166 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:40:06.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:05.176202+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e586: 8 total, 7 up, 8 in 2024-02-23T00:40:06.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:05.183475+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]: dispatch 2024-02-23T00:40:06.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:05.742671+0000 mon.a (mon.0) 3169 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-23T00:40:06.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:06.083511+0000 osd.1 (osd.1) 267 : cluster [DBG] 22.b scrub starts 2024-02-23T00:40:06.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:06.086631+0000 osd.1 (osd.1) 268 : cluster [DBG] 22.b scrub ok 2024-02-23T00:40:06.462 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:06.488 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:06.510 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:06.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:06.613 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:06.643 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:06.842 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:06.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:06.847 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24712.0:10 7.a 7:54825363:test-rados-api-smithi087-82115-4::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:30:57.476130+0000", 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 549.36824294600001, 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030623026000000001, 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.848 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24712", 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2285581802", 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 10 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.476130+0000", 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.849 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.476130+0000", 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.476128+0000", 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.047e-05 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.476138+0000", 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0051071739999999999 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.850 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.481245+0000", 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1459e-05 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.481257+0000", 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0026739429999999998 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.483931+0000", 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1996999999999993e-05 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.851 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.484023+0000", 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000192447 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 2,5", 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.484215+0000", 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012288584999999999 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.496504+0000", 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0049043990000000003 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.852 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.501408+0000", 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0052414000000000002 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.506650+0000", 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2675e-05 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.506702+0000", 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0937999999999999e-05 2024-02-23T00:40:06.853 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:30:57.506753+0000", 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030623026000000001 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24712.0:19 7.2 7:40054bab:test-rados-api-smithi087-82115-6::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e111)", 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:01.502859+0000", 2024-02-23T00:40:06.854 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 545.34151458500003, 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040250123999999998, 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24712", 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2285581802", 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 19 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502859+0000", 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.855 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502859+0000", 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502857+0000", 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.7179999999999996e-06 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502865+0000", 2024-02-23T00:40:06.856 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0523000000000001e-05 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502875+0000", 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8630000000000002e-06 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.502882+0000", 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00075966 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.503642+0000", 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2692e-05 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.857 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.503734+0000", 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021787099999999999 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.503952+0000", 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0062092650000000003 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.510162+0000", 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.022996850999999999 2024-02-23T00:40:06.858 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.533158+0000", 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0098446820000000004 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.543003+0000", 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5009000000000001e-05 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.859 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.543058+0000", 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0720999999999998e-05 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:01.543109+0000", 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040250123999999998 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24712.0:21 7.2 7:40054bab:test-rados-api-smithi087-82115-6::foo:head [create,delete] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-23T00:40:06.860 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.518065+0000", 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 544.32630829899995, 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.083348715000000004, 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24712", 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2285581802", 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 21 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518065+0000", 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.861 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518065+0000", 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518063+0000", 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5610000000000004e-06 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518070+0000", 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0468e-05 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.862 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518080+0000", 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8329999999999996e-06 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.518087+0000", 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0055818969999999997 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.523669+0000", 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1359999999999999e-06 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.863 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.523676+0000", 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0062083249999999998 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.529885+0000", 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3505999999999995e-05 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.864 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.529978+0000", 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00020373200000000001 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.530182+0000", 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010326180000000001 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.540508+0000", 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.052153245000000001 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.865 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.592661+0000", 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0086538880000000002 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.601315+0000", 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4023000000000001e-05 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.601369+0000", 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4456000000000003e-05 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.866 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.601414+0000", 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.083348715000000004 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.25166.0:2 59.6 e112/111 59:602f83fe:::foo:head v 112'1, mlcod=0'0)", 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:02.593159+0000", 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 544.25121388100001, 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020005182999999999, 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.867 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593159+0000", 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593159+0000", 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593159+0000", 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8780000000000004e-06 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.868 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593168+0000", 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7429999999999999e-06 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593174+0000", 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7609999999999998e-06 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.869 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593178+0000", 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7953000000000005e-05 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593246+0000", 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4884e-05 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.593261+0000", 2024-02-23T00:40:06.870 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019871346000000002 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.613133+0000", 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1909000000000002e-05 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:02.613165+0000", 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020005182999999999 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.871 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.5.0:320 56.2 e113/110 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head v 113'1, mlcod=0'0)", 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.596901+0000", 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 543.24747252999998, 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028157662, 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596901+0000", 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.872 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596901+0000", 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596900+0000", 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3729999999999998e-06 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596907+0000", 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7179999999999999e-06 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.873 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596913+0000", 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0419999999999997e-06 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596920+0000", 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8403999999999999e-05 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596969+0000", 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1588000000000001e-05 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.874 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596990+0000", 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028040088000000001 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.625030+0000", 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7909999999999999e-05 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.625058+0000", 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028157662 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.875 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15147.0:8 56.2 e113/110 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head v 113'2, mlcod=0'0)", 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:03.596928+0000", 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 543.247445062, 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028148210999999999, 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596928+0000", 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.876 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596928+0000", 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596928+0000", 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4100000000000008e-06 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596936+0000", 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9109999999999998e-06 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.877 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596942+0000", 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2209999999999999e-06 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.596946+0000", 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00096380100000000002 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.597910+0000", 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9411000000000001e-05 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.597930+0000", 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027132234000000002 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.625062+0000", 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4535999999999999e-05 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.879 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:03.625076+0000", 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.028148210999999999 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26218.0:2 62.15 62:ad551702:::test:head [create,write-same] snapc 0=[] ondisk+write+known_if_redirected e115)", 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:06.197041+0000", 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 540.64733234599998, 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.135993164, 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.880 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26218", 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2330528609", 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197041+0000", 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.881 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197041+0000", 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197038+0000", 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3602e-05 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197052+0000", 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00051941599999999997 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.882 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197571+0000", 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1411e-05 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.197583+0000", 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015026843 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.212610+0000", 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8179999999999998e-06 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.883 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.212616+0000", 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074955025999999994 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.287571+0000", 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9249000000000001e-05 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.287641+0000", 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001974618 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.884 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 2,7", 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.289615+0000", 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020824579999999999 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.310440+0000", 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00060312300000000001 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.311043+0000", 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021897838999999999 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.885 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.332941+0000", 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2295e-05 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.332993+0000", 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0945999999999997e-05 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:06.333034+0000", 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.135993164 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.886 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15147.0:69 56.1 e118/110 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head v 118'14, mlcod=113'11)", 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.954974+0000", 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 536.88939943800005, 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023167466000000001, 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954974+0000", 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.887 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954974+0000", 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954973+0000", 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2839999999999997e-06 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954979+0000", 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.463e-06 2024-02-23T00:40:06.888 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954984+0000", 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8449999999999996e-06 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.954988+0000", 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0044694829999999998 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.959457+0000", 2024-02-23T00:40:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7171000000000001e-05 2024-02-23T00:40:06.890 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.890 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.890 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.890 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.959485+0000", 2024-02-23T00:40:06.890 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018617076 2024-02-23T00:40:06.890 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.890 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.890 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.890 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.978102+0000", 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9558000000000003e-05 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.978141+0000", 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023167466000000001 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:5 66.11 66:8d290297:test-rados-api-smithi087-82091-8::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e120)", 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.184196+0000", 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 535.66017751000004, 2024-02-23T00:40:06.891 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.078409890999999995, 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 5 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184196+0000", 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.892 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184196+0000", 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999971 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184193+0000", 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4197e-05 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.893 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184207+0000", 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1573000000000002e-05 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184229+0000", 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1409999999999994e-06 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.184238+0000", 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014631701 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.894 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.198870+0000", 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5130000000000002e-06 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.198879+0000", 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019356801999999999 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.218236+0000", 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5295000000000002e-05 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.895 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.218321+0000", 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044284507000000001 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.262606+0000", 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.078409890999999995 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27712.0:57 67.3 e138/117 67:c06d7414:test-rados-api-smithi087-82115-12::foo:head v 138'1, mlcod=0'0)", 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:29.022463+0000", 2024-02-23T00:40:06.896 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 517.82191029299997, 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020343938999999998, 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022463+0000", 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022463+0000", 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.897 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022462+0000", 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4499e-05 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022477+0000", 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5664000000000001e-05 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022492+0000", 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3310000000000004e-06 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.898 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022501+0000", 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7853000000000002e-05 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022568+0000", 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1843000000000001e-05 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.022590+0000", 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020078691999999999 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.899 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.900 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.042669+0000", 2024-02-23T00:40:06.900 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000137921 2024-02-23T00:40:06.900 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.900 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.900 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.900 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.042807+0000", 2024-02-23T00:40:06.900 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020343938999999998 2024-02-23T00:40:06.900 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:256 86.e 86:70f8162d:test-rados-api-smithi087-82091-12::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:29.038511+0000", 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 517.80586216200004, 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.055373620999999998, 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 256 2024-02-23T00:40:06.901 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038511+0000", 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038511+0000", 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.902 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038509+0000", 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4069999999999993e-06 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038518+0000", 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1795e-05 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038529+0000", 2024-02-23T00:40:06.903 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9810000000000003e-06 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038537+0000", 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6368000000000001e-05 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038584+0000", 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.959e-06 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.904 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.038590+0000", 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012529669 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.051119+0000", 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021244600000000001 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.051332+0000", 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.039488665999999999 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.905 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.090821+0000", 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5575000000000007e-05 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for rw locks", 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.090906+0000", 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3004000000000006e-05 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.090999+0000", 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9804e-05 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.906 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.091039+0000", 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0028457880000000001 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:29.093885+0000", 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.055373620999999998 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27712.0:63 67.d 67:b74fee6d:test-rados-api-smithi087-82115-13::foo:head [write 0~128] snapc 16=[16] ondisk+write+known_if_redirected e140)", 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:31.046764+0000", 2024-02-23T00:40:06.907 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 515.79760917800002, 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021316562000000001, 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27712", 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2607672493", 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 63 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046764+0000", 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.908 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046764+0000", 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046762+0000", 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5170000000000005e-06 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046772+0000", 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1219e-05 2024-02-23T00:40:06.909 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046783+0000", 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4849999999999992e-06 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.046791+0000", 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000425838 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.910 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.047217+0000", 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0990000000000001e-06 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.047224+0000", 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011776207 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.911 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.059000+0000", 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012097800000000001 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.059121+0000", 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000135659 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 6,7", 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.059257+0000", 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.007406035 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.912 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.066663+0000", 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1823999999999999e-05 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.066695+0000", 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00128031 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.067975+0000", 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8869000000000002e-05 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.913 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.068034+0000", 2024-02-23T00:40:06.914 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6480000000000002e-05 2024-02-23T00:40:06.914 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.914 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.914 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.914 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:31.068081+0000", 2024-02-23T00:40:06.916 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.021316562000000001 2024-02-23T00:40:06.916 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.916 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.916 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.916 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.916 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.36707.0:2 96.18 96:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:34.989654+0000", 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 511.854718746, 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.038403145, 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.36707", 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1873510578", 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.917 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989654+0000", 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989654+0000", 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.918 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989652+0000", 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1190000000000003e-06 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989661+0000", 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5359999999999999e-05 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989677+0000", 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0920000000000004e-06 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.919 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989686+0000", 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8553e-05 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989744+0000", 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.653e-06 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.989753+0000", 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025073980999999999 2024-02-23T00:40:06.920 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.921 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.921 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.921 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.014827+0000", 2024-02-23T00:40:06.921 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000121211 2024-02-23T00:40:06.921 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.921 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.921 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.921 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.014948+0000", 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015277699999999999 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.015101+0000", 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011323990000000001 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.016233+0000", 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0086185819999999996 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.922 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024852+0000", 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0031188610000000001 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.027971+0000", 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7979000000000003e-05 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.028019+0000", 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8761999999999999e-05 2024-02-23T00:40:06.923 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.028058+0000", 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.038403145 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.24815.0:1045 19.10 19:08000000::::head [pgnls start_epoch 98] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_overlay+known_if_redirected e147)", 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:38.022061+0000", 2024-02-23T00:40:06.924 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 508.82231229899998, 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.984081753, 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.24815", 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3778460189", 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1045 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022061+0000", 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.925 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022061+0000", 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022059+0000", 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6950000000000003e-06 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022066+0000", 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3033999999999999e-05 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.926 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022079+0000", 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8210000000000001e-06 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022084+0000", 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00033546699999999999 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022419+0000", 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.7959999999999996e-06 2024-02-23T00:40:06.927 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:38.022427+0000", 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.983542099 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.005969+0000", 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5502e-05 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.928 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.006005+0000", 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013795699999999999 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:39.006143+0000", 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.984081753 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.5.0:551 63.d e182/114 63:b6ad5569:test-rados-api-smithi087-82091-14::foo18:7 v 182'22, mlcod=148'16)", 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.002771+0000", 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 473.84160240599999, 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019809093999999999, 2024-02-23T00:40:06.929 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002771+0000", 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002771+0000", 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.930 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002771+0000", 2024-02-23T00:40:06.931 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1500000000000002e-06 2024-02-23T00:40:06.931 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.931 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.931 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.931 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002778+0000", 2024-02-23T00:40:06.931 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0109999999999997e-06 2024-02-23T00:40:06.931 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.931 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.931 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002784+0000", 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5360000000000001e-06 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.002787+0000", 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0046298449999999996 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.007417+0000", 2024-02-23T00:40:06.932 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8009999999999999e-05 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.007435+0000", 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.015111853999999999 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.022547+0000", 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3019999999999999e-05 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.933 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.022580+0000", 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.019809093999999999 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37088.0:1 146.6s0 146:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:43.010737+0000", 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 443.83363611200002, 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0162247929999999, 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.934 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37088", 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/758004491", 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010737+0000", 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010737+0000", 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.935 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010735+0000", 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8709999999999992e-06 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010745+0000", 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3915e-05 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010759+0000", 2024-02-23T00:40:06.936 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4579000000000001e-05 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010774+0000", 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0025273800000000001 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.013301+0000", 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3150000000000002e-06 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:40:06.937 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.013305+0000", 2024-02-23T00:40:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0005927610000001 2024-02-23T00:40:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.013898+0000", 2024-02-23T00:40:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011305499999999999 2024-02-23T00:40:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.938 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.014011+0000", 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028954399999999998 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.014301+0000", 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0018669419999999999 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.016168+0000", 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.01075132 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.939 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.026919+0000", 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3027e-05 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.026962+0000", 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0162247929999999 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.940 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:1810 63.8 63:15d27fd9:test-rados-api-smithi087-82091-22::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-23T00:40:06.941 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.023992+0000", 2024-02-23T00:40:06.941 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 439.82038086199998, 2024-02-23T00:40:06.941 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041033304999999999, 2024-02-23T00:40:06.941 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.941 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.941 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.941 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:40:06.942 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:06.942 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1810 2024-02-23T00:40:06.942 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.942 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.942 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.942 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.943 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.023992+0000", 2024-02-23T00:40:06.943 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.943 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.943 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.943 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.943 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.023992+0000", 2024-02-23T00:40:06.943 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:06.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.944 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.944 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.944 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.023990+0000", 2024-02-23T00:40:06.944 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4919999999999999e-06 2024-02-23T00:40:06.944 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.945 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.945 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024000+0000", 2024-02-23T00:40:06.945 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1005000000000001e-05 2024-02-23T00:40:06.945 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.945 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.946 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.946 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024011+0000", 2024-02-23T00:40:06.946 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4859999999999997e-06 2024-02-23T00:40:06.946 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.946 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.946 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.947 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024019+0000", 2024-02-23T00:40:06.947 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4852999999999998e-05 2024-02-23T00:40:06.947 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.947 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.947 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.947 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024074+0000", 2024-02-23T00:40:06.948 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013309699999999999 2024-02-23T00:40:06.948 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.948 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.948 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.948 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024207+0000", 2024-02-23T00:40:06.948 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00018933000000000001 2024-02-23T00:40:06.949 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.949 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.949 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:40:06.949 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024397+0000", 2024-02-23T00:40:06.949 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0012107419999999999 2024-02-23T00:40:06.949 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.950 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:40:06.950 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.025607+0000", 2024-02-23T00:40:06.950 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013983776 2024-02-23T00:40:06.950 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.950 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.951 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.951 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.039591+0000", 2024-02-23T00:40:06.951 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025340995000000002 2024-02-23T00:40:06.951 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.951 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.952 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:06.952 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.064932+0000", 2024-02-23T00:40:06.952 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7939000000000003e-05 2024-02-23T00:40:06.952 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.952 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.952 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.953 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.064980+0000", 2024-02-23T00:40:06.953 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5509000000000001e-05 2024-02-23T00:40:06.953 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.953 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.953 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.953 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.065026+0000", 2024-02-23T00:40:06.954 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041033304999999999 2024-02-23T00:40:06.954 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.954 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.954 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.954 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.954 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.955 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37277.0:1 173.6s0 173:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e261)", 2024-02-23T00:40:06.955 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:32.043764+0000", 2024-02-23T00:40:06.955 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 394.80060882700002, 2024-02-23T00:40:06.955 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.99129025299999995, 2024-02-23T00:40:06.955 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.955 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.956 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.956 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37277", 2024-02-23T00:40:06.956 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2105660432", 2024-02-23T00:40:06.956 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:40:06.956 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.956 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.957 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.957 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.957 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043764+0000", 2024-02-23T00:40:06.957 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.957 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.957 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.957 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.957 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043764+0000", 2024-02-23T00:40:06.957 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:06.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.958 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.958 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.958 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043762+0000", 2024-02-23T00:40:06.958 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8310000000000001e-06 2024-02-23T00:40:06.958 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.958 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.959 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.959 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043770+0000", 2024-02-23T00:40:06.959 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2220000000000008e-06 2024-02-23T00:40:06.959 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.959 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.959 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.959 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043779+0000", 2024-02-23T00:40:06.959 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3200000000000002e-06 2024-02-23T00:40:06.960 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.960 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.960 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.960 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043787+0000", 2024-02-23T00:40:06.960 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0408999999999994e-05 2024-02-23T00:40:06.960 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043877+0000", 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.213e-06 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043880+0000", 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.98460219900000001 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.961 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.028483+0000", 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012271999999999999 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.028605+0000", 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028859799999999999 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.028894+0000", 2024-02-23T00:40:06.962 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00041391600000000001 2024-02-23T00:40:06.963 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.963 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.963 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:40:06.963 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.029308+0000", 2024-02-23T00:40:06.963 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0056940439999999997 2024-02-23T00:40:06.963 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.963 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.963 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.035002+0000", 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2791999999999999e-05 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.035055+0000", 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.99129025299999995 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.964 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26898.0:1 183.6s0 183:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:53.056908+0000", 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 373.78746566199999, 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.99571678900000005, 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26898", 2024-02-23T00:40:06.965 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3707730563", 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056908+0000", 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056908+0000", 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.966 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056906+0000", 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2740000000000004e-06 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056912+0000", 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6000000000000007e-06 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056921+0000", 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1380000000000001e-06 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.967 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056928+0000", 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025497999999999997 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.057183+0000", 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4829999999999999e-06 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.057187+0000", 2024-02-23T00:40:06.968 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.98844382500000005 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.045630+0000", 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000127668 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.045758+0000", 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00040348400000000001 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:40:06.969 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.046162+0000", 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000493244 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.046655+0000", 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.005911449 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.052566+0000", 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8118999999999997e-05 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.970 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.052624+0000", 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.99571678900000005 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:1877 5.20 5:05bf5b68:::notify.1:head [watch reconnect cookie 94532396646400 gen 4] snapc 0=[] ondisk+write+known_if_redirected e561)", 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:39:40.477301+0000", 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 26.367072426, 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.73050895699999996, 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:40:06.971 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1877 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477301+0000", 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:40:06.972 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477301+0000", 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477300+0000", 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9739999999999999e-06 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477306+0000", 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1486e-05 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.973 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477317+0000", 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5060000000000001e-06 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477324+0000", 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2507000000000003e-05 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477386+0000", 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4099999999999997e-06 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.974 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477396+0000", 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.70815977299999999 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.185555+0000", 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013567700000000001 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for missing object", 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.185691+0000", 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003571552 2024-02-23T00:40:06.975 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.189263+0000", 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000101983 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for rw locks", 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.189365+0000", 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0061286350000000003 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:40:06.976 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.195493+0000", 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5148000000000005e-05 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.195578+0000", 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012231357999999999 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.207810+0000", 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.73050895699999996 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:06.977 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:06.978 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:40:06.978 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:07.004 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:07.102 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:07.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:07.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:07.206 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:07.207 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:40:07.232 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:07.256 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:07.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:05.741591+0000 mgr.y (mgr.24437) 529 : cluster [DBG] pgmap v896: 229 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 49 active+undersized, 145 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 40/705 objects degraded (5.674%); 61/705 objects misplaced (8.652%); 187 B/s, 0 objects/s recovering 2024-02-23T00:40:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:05.935361+0000 osd.5 (osd.5) 223 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:40:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:05.936756+0000 osd.5 (osd.5) 224 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:40:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:06.171955+0000 mon.a (mon.0) 3170 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:06.171983+0000 mon.a (mon.0) 3171 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-23T00:40:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:06.178945+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]': finished 2024-02-23T00:40:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:06.179020+0000 mon.a (mon.0) 3173 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-23T00:40:07.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:06.179083+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e587: 8 total, 7 up, 8 in 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:06.281687+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.28 scrub starts 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:06.284374+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.28 scrub ok 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:06.289798+0000 osd.3 (osd.3) 181 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:06.292717+0000 osd.3 (osd.3) 182 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:05.741591+0000 mgr.y (mgr.24437) 529 : cluster [DBG] pgmap v896: 229 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 49 active+undersized, 145 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 40/705 objects degraded (5.674%); 61/705 objects misplaced (8.652%); 187 B/s, 0 objects/s recovering 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:05.935361+0000 osd.5 (osd.5) 223 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:05.936756+0000 osd.5 (osd.5) 224 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:06.171955+0000 mon.a (mon.0) 3170 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:06.171983+0000 mon.a (mon.0) 3171 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:06.178945+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]': finished 2024-02-23T00:40:07.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:06.179020+0000 mon.a (mon.0) 3173 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-23T00:40:07.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:06.179083+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e587: 8 total, 7 up, 8 in 2024-02-23T00:40:07.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:06.281687+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.28 scrub starts 2024-02-23T00:40:07.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:06.284374+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.28 scrub ok 2024-02-23T00:40:07.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:06.289798+0000 osd.3 (osd.3) 181 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:40:07.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:06.292717+0000 osd.3 (osd.3) 182 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:40:07.361 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:07.387 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:07.405 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:07.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:05.741591+0000 mgr.y (mgr.24437) 529 : cluster [DBG] pgmap v896: 229 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 49 active+undersized, 145 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 40/705 objects degraded (5.674%); 61/705 objects misplaced (8.652%); 187 B/s, 0 objects/s recovering 2024-02-23T00:40:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:05.935361+0000 osd.5 (osd.5) 223 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:40:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:05.936756+0000 osd.5 (osd.5) 224 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:40:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:06.171955+0000 mon.a (mon.0) 3170 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:06.171983+0000 mon.a (mon.0) 3171 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-23T00:40:07.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:06.178945+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-99", "tierpool": "test-rados-api-smithi087-82091-99-cache"}]': finished 2024-02-23T00:40:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:06.179020+0000 mon.a (mon.0) 3173 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-23T00:40:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:06.179083+0000 mon.a (mon.0) 3174 : cluster [DBG] osdmap e587: 8 total, 7 up, 8 in 2024-02-23T00:40:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:06.281687+0000 osd.2 (osd.2) 215 : cluster [DBG] 5.28 scrub starts 2024-02-23T00:40:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:06.284374+0000 osd.2 (osd.2) 216 : cluster [DBG] 5.28 scrub ok 2024-02-23T00:40:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:06.289798+0000 osd.3 (osd.3) 181 : cluster [DBG] 6.10 scrub starts 2024-02-23T00:40:07.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:06.292717+0000 osd.3 (osd.3) 182 : cluster [DBG] 6.10 scrub ok 2024-02-23T00:40:07.510 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:07.536 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:07.685 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:07.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:07.789 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:07.815 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:07.834 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:07.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:07.937 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:07.963 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:07.982 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:07.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:08.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:08.112 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:08.130 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:08.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:08.148 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:08.234 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:08.262 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:08.284 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:08.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:08.304 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:07.182941+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e588: 8 total, 7 up, 8 in 2024-02-23T00:40:08.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:07.246874+0000 osd.3 (osd.3) 183 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:40:08.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:07.255342+0000 osd.3 (osd.3) 184 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:40:08.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:08.073911+0000 osd.1 (osd.1) 269 : cluster [DBG] 2.d scrub starts 2024-02-23T00:40:08.305 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:08.076815+0000 osd.1 (osd.1) 270 : cluster [DBG] 2.d scrub ok 2024-02-23T00:40:08.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:07.182941+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e588: 8 total, 7 up, 8 in 2024-02-23T00:40:08.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:07.246874+0000 osd.3 (osd.3) 183 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:40:08.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:07.255342+0000 osd.3 (osd.3) 184 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:40:08.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:08.073911+0000 osd.1 (osd.1) 269 : cluster [DBG] 2.d scrub starts 2024-02-23T00:40:08.305 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:08.076815+0000 osd.1 (osd.1) 270 : cluster [DBG] 2.d scrub ok 2024-02-23T00:40:08.388 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:08.412 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:07.182941+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e588: 8 total, 7 up, 8 in 2024-02-23T00:40:08.412 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:07.246874+0000 osd.3 (osd.3) 183 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:40:08.412 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:07.255342+0000 osd.3 (osd.3) 184 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:40:08.412 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:08.073911+0000 osd.1 (osd.1) 269 : cluster [DBG] 2.d scrub starts 2024-02-23T00:40:08.412 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:08.076815+0000 osd.1 (osd.1) 270 : cluster [DBG] 2.d scrub ok 2024-02-23T00:40:08.414 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:08.432 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:08.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:08.536 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:08.565 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:08.805 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-23T00:40:08.812 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:08.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:08.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:08.916 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:08.943 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:08.962 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:08.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:09.065 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:09.076 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:09.094 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:09.145 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:09.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:07.355972+0000 osd.4 (osd.4) 263 : cluster [DBG] 4.e scrub starts 2024-02-23T00:40:09.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:07.359514+0000 osd.4 (osd.4) 264 : cluster [DBG] 4.e scrub ok 2024-02-23T00:40:09.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:07.742233+0000 mgr.y (mgr.24437) 530 : cluster [DBG] pgmap v899: 197 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 36 active+undersized, 126 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%); 187 B/s, 0 objects/s recovering 2024-02-23T00:40:09.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:08.179907+0000 mon.a (mon.0) 3176 : cluster [WRN] Health check update: Degraded data redundancy: 40/702 objects degraded (5.698%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:09.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:08.188582+0000 mon.a (mon.0) 3177 : cluster [DBG] osdmap e589: 8 total, 7 up, 8 in 2024-02-23T00:40:09.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:08.189847+0000 mon.a (mon.0) 3178 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:40:09.355 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:08.273886+0000 osd.6 (osd.6) 234 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:40:09.356 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:08.274718+0000 osd.6 (osd.6) 235 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:40:09.356 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:09.176310+0000 mon.a (mon.0) 3179 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:09.445 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:09.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:09.549 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:09.577 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:09.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:07.355972+0000 osd.4 (osd.4) 263 : cluster [DBG] 4.e scrub starts 2024-02-23T00:40:09.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:07.359514+0000 osd.4 (osd.4) 264 : cluster [DBG] 4.e scrub ok 2024-02-23T00:40:09.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:07.742233+0000 mgr.y (mgr.24437) 530 : cluster [DBG] pgmap v899: 197 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 36 active+undersized, 126 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%); 187 B/s, 0 objects/s recovering 2024-02-23T00:40:09.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:08.179907+0000 mon.a (mon.0) 3176 : cluster [WRN] Health check update: Degraded data redundancy: 40/702 objects degraded (5.698%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:08.188582+0000 mon.a (mon.0) 3177 : cluster [DBG] osdmap e589: 8 total, 7 up, 8 in 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:08.189847+0000 mon.a (mon.0) 3178 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:08.273886+0000 osd.6 (osd.6) 234 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:08.274718+0000 osd.6 (osd.6) 235 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:09.176310+0000 mon.a (mon.0) 3179 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:07.355972+0000 osd.4 (osd.4) 263 : cluster [DBG] 4.e scrub starts 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:07.359514+0000 osd.4 (osd.4) 264 : cluster [DBG] 4.e scrub ok 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:07.742233+0000 mgr.y (mgr.24437) 530 : cluster [DBG] pgmap v899: 197 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 36 active+undersized, 126 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%); 187 B/s, 0 objects/s recovering 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:08.179907+0000 mon.a (mon.0) 3176 : cluster [WRN] Health check update: Degraded data redundancy: 40/702 objects degraded (5.698%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:08.188582+0000 mon.a (mon.0) 3177 : cluster [DBG] osdmap e589: 8 total, 7 up, 8 in 2024-02-23T00:40:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:08.189847+0000 mon.a (mon.0) 3178 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:40:09.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:08.273886+0000 osd.6 (osd.6) 234 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:40:09.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:08.274718+0000 osd.6 (osd.6) 235 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:40:09.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:09.176310+0000 mon.a (mon.0) 3179 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:09.597 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:09.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:09.692 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:40:09.701 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:09.728 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:09.926 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:09.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:10.029 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:10.056 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:09.192150+0000 mon.a (mon.0) 3180 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:40:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:09.192235+0000 mon.a (mon.0) 3181 : cluster [DBG] osdmap e590: 8 total, 7 up, 8 in 2024-02-23T00:40:10.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:09.193435+0000 mon.a (mon.0) 3182 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:40:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:09.739859+0000 mon.a (mon.0) 3183 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:40:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:09.740535+0000 mon.a (mon.0) 3184 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:40:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:09.744929+0000 mon.a (mon.0) 3185 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:09.746371+0000 mon.a (mon.0) 3186 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-23T00:40:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:09.192150+0000 mon.a (mon.0) 3180 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:40:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:09.192235+0000 mon.a (mon.0) 3181 : cluster [DBG] osdmap e590: 8 total, 7 up, 8 in 2024-02-23T00:40:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:09.193435+0000 mon.a (mon.0) 3182 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:40:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:09.739859+0000 mon.a (mon.0) 3183 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:40:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:09.740535+0000 mon.a (mon.0) 3184 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:40:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:09.744929+0000 mon.a (mon.0) 3185 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:10.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:09.746371+0000 mon.a (mon.0) 3186 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-23T00:40:10.513 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:09.192150+0000 mon.a (mon.0) 3180 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:40:10.513 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:09.192235+0000 mon.a (mon.0) 3181 : cluster [DBG] osdmap e590: 8 total, 7 up, 8 in 2024-02-23T00:40:10.513 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:09.193435+0000 mon.a (mon.0) 3182 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-99"}]: dispatch 2024-02-23T00:40:10.513 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:09.739859+0000 mon.a (mon.0) 3183 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:40:10.513 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:09.740535+0000 mon.a (mon.0) 3184 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:40:10.514 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:09.744929+0000 mon.a (mon.0) 3185 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:10.514 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:09.746371+0000 mon.a (mon.0) 3186 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-23T00:40:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:11.009 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:11.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:11.011 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:40:11.011 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:40:11.011 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:40:11.011 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:40:11.110 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:11.138 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:11.157 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:11.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:11.262 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:11.290 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:40:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:40:11.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:09.659865+0000 osd.7 (osd.7) 272 : cluster [DBG] 6.d scrub starts 2024-02-23T00:40:11.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:09.663356+0000 osd.7 (osd.7) 273 : cluster [DBG] 6.d scrub ok 2024-02-23T00:40:11.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:09.742961+0000 mgr.y (mgr.24437) 531 : cluster [DBG] pgmap v902: 188 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 31 active+undersized, 122 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/699 objects degraded (5.722%); 61/699 objects misplaced (8.727%) 2024-02-23T00:40:11.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:10.193497+0000 mon.a (mon.0) 3187 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:40:11.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:10.193565+0000 mon.a (mon.0) 3188 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-23T00:40:11.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:10.193618+0000 mon.a (mon.0) 3189 : cluster [DBG] osdmap e591: 8 total, 7 up, 8 in 2024-02-23T00:40:11.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.200634+0000 mon.a (mon.0) 3190 : cluster [DBG] osdmap e592: 8 total, 7 up, 8 in 2024-02-23T00:40:11.291 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:11.203967+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:11.292 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:11.390 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:11.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:11.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:11.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:09.659865+0000 osd.7 (osd.7) 272 : cluster [DBG] 6.d scrub starts 2024-02-23T00:40:11.474 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:09.663356+0000 osd.7 (osd.7) 273 : cluster [DBG] 6.d scrub ok 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:09.742961+0000 mgr.y (mgr.24437) 531 : cluster [DBG] pgmap v902: 188 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 31 active+undersized, 122 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/699 objects degraded (5.722%); 61/699 objects misplaced (8.727%) 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:10.193497+0000 mon.a (mon.0) 3187 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:10.193565+0000 mon.a (mon.0) 3188 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:10.193618+0000 mon.a (mon.0) 3189 : cluster [DBG] osdmap e591: 8 total, 7 up, 8 in 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.200634+0000 mon.a (mon.0) 3190 : cluster [DBG] osdmap e592: 8 total, 7 up, 8 in 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:11.203967+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:09.659865+0000 osd.7 (osd.7) 272 : cluster [DBG] 6.d scrub starts 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:09.663356+0000 osd.7 (osd.7) 273 : cluster [DBG] 6.d scrub ok 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:09.742961+0000 mgr.y (mgr.24437) 531 : cluster [DBG] pgmap v902: 188 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 31 active+undersized, 122 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/699 objects degraded (5.722%); 61/699 objects misplaced (8.727%) 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:10.193497+0000 mon.a (mon.0) 3187 : audit [INF] from='client.? 172.21.15.87:0/4281879676' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-99"}]': finished 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:10.193565+0000 mon.a (mon.0) 3188 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-23T00:40:11.475 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:10.193618+0000 mon.a (mon.0) 3189 : cluster [DBG] osdmap e591: 8 total, 7 up, 8 in 2024-02-23T00:40:11.476 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.200634+0000 mon.a (mon.0) 3190 : cluster [DBG] osdmap e592: 8 total, 7 up, 8 in 2024-02-23T00:40:11.476 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:11.203967+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:11.506 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:11.535 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:11.673 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-23T00:40:11.763 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:11.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:11.767 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:40:11.767 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "" 2024-02-23T00:40:11.768 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:40:11.867 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:11.895 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:11.937 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:10.334954+0000 osd.4 (osd.4) 265 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:10.335841+0000 osd.4 (osd.4) 266 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:11.095028+0000 mgr.y (mgr.24437) 532 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.169018+0000 osd.3 (osd.3) 185 : cluster [DBG] 22.e scrub starts 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.172062+0000 osd.3 (osd.3) 186 : cluster [DBG] 22.e scrub ok 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:11.203071+0000 mon.c (mon.2) 472 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.258293+0000 osd.6 (osd.6) 236 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.261095+0000 osd.6 (osd.6) 237 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.668488+0000 osd.7 (osd.7) 274 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.671389+0000 osd.7 (osd.7) 275 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.888569+0000 osd.5 (osd.5) 225 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:40:12.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.889324+0000 osd.5 (osd.5) 226 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:40:12.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:12.164325+0000 mon.a (mon.0) 3192 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:12.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:12.169329+0000 mon.a (mon.0) 3193 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]': finished 2024-02-23T00:40:12.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:12.169436+0000 mon.a (mon.0) 3194 : cluster [DBG] osdmap e593: 8 total, 7 up, 8 in 2024-02-23T00:40:12.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:12.171108+0000 mon.c (mon.2) 473 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:12.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:12.176677+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:12.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:12] "GET /metrics HTTP/1.1" 200 187132 "" "Prometheus/2.33.4" 2024-02-23T00:40:12.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:10.334954+0000 osd.4 (osd.4) 265 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:40:12.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:10.335841+0000 osd.4 (osd.4) 266 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:11.095028+0000 mgr.y (mgr.24437) 532 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.169018+0000 osd.3 (osd.3) 185 : cluster [DBG] 22.e scrub starts 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.172062+0000 osd.3 (osd.3) 186 : cluster [DBG] 22.e scrub ok 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:11.203071+0000 mon.c (mon.2) 472 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.258293+0000 osd.6 (osd.6) 236 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.261095+0000 osd.6 (osd.6) 237 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.668488+0000 osd.7 (osd.7) 274 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.671389+0000 osd.7 (osd.7) 275 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.888569+0000 osd.5 (osd.5) 225 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.889324+0000 osd.5 (osd.5) 226 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:40:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:12.164325+0000 mon.a (mon.0) 3192 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:12.169329+0000 mon.a (mon.0) 3193 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]': finished 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:12.169436+0000 mon.a (mon.0) 3194 : cluster [DBG] osdmap e593: 8 total, 7 up, 8 in 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:12.171108+0000 mon.c (mon.2) 473 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:12.176677+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:10.334954+0000 osd.4 (osd.4) 265 : cluster [DBG] 6.12 scrub starts 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:10.335841+0000 osd.4 (osd.4) 266 : cluster [DBG] 6.12 scrub ok 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:11.095028+0000 mgr.y (mgr.24437) 532 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.169018+0000 osd.3 (osd.3) 185 : cluster [DBG] 22.e scrub starts 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.172062+0000 osd.3 (osd.3) 186 : cluster [DBG] 22.e scrub ok 2024-02-23T00:40:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:11.203071+0000 mon.c (mon.2) 472 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.258293+0000 osd.6 (osd.6) 236 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.261095+0000 osd.6 (osd.6) 237 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.668488+0000 osd.7 (osd.7) 274 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.671389+0000 osd.7 (osd.7) 275 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.888569+0000 osd.5 (osd.5) 225 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.889324+0000 osd.5 (osd.5) 226 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:12.164325+0000 mon.a (mon.0) 3192 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:12.169329+0000 mon.a (mon.0) 3193 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi087-82091-94"}]': finished 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:12.169436+0000 mon.a (mon.0) 3194 : cluster [DBG] osdmap e593: 8 total, 7 up, 8 in 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:12.171108+0000 mon.c (mon.2) 473 : audit [INF] from='client.? 172.21.15.87:0/403995334' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:12.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:12.176677+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]: dispatch 2024-02-23T00:40:12.644 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:12.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:12.748 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:12.774 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:13.115 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:13.121 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:13.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:13.226 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:13.255 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:13.378 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:11.743584+0000 mgr.y (mgr.24437) 533 : cluster [DBG] pgmap v905: 180 pgs: 2 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 27 active+undersized, 116 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/696 objects degraded (5.747%); 61/696 objects misplaced (8.764%) 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:12.125175+0000 osd.3 (osd.3) 187 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:12.128681+0000 osd.3 (osd.3) 188 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:12.983216+0000 osd.1 (osd.1) 271 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:12.984036+0000 osd.1 (osd.1) 272 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:13.171421+0000 mon.a (mon.0) 3196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]': finished 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:13.171485+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e594: 8 total, 7 up, 8 in 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:13.191313+0000 mon.a (mon.0) 3198 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:13.193035+0000 mon.a (mon.0) 3199 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:13.381 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:13.193332+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:40:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:13.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:11.743584+0000 mgr.y (mgr.24437) 533 : cluster [DBG] pgmap v905: 180 pgs: 2 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 27 active+undersized, 116 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/696 objects degraded (5.747%); 61/696 objects misplaced (8.764%) 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:12.125175+0000 osd.3 (osd.3) 187 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:12.128681+0000 osd.3 (osd.3) 188 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:12.983216+0000 osd.1 (osd.1) 271 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:12.984036+0000 osd.1 (osd.1) 272 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:13.171421+0000 mon.a (mon.0) 3196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]': finished 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:13.171485+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e594: 8 total, 7 up, 8 in 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:13.191313+0000 mon.a (mon.0) 3198 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:13.193035+0000 mon.a (mon.0) 3199 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:13.193332+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:40:13.385 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:11.743584+0000 mgr.y (mgr.24437) 533 : cluster [DBG] pgmap v905: 180 pgs: 2 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 27 active+undersized, 116 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/696 objects degraded (5.747%); 61/696 objects misplaced (8.764%) 2024-02-23T00:40:13.386 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:12.125175+0000 osd.3 (osd.3) 187 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:40:13.386 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:12.128681+0000 osd.3 (osd.3) 188 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:40:13.386 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:12.983216+0000 osd.1 (osd.1) 271 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:40:13.386 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:12.984036+0000 osd.1 (osd.1) 272 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:40:13.386 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:13.171421+0000 mon.a (mon.0) 3196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi087-82091-94"}]': finished 2024-02-23T00:40:13.386 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:13.171485+0000 mon.a (mon.0) 3197 : cluster [DBG] osdmap e594: 8 total, 7 up, 8 in 2024-02-23T00:40:13.386 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:13.191313+0000 mon.a (mon.0) 3198 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:13.386 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:13.193035+0000 mon.a (mon.0) 3199 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:13.386 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:13.193332+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-23T00:40:13.484 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:13.512 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:13.531 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:13.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:13.634 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:13.654 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:40:13.654 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:40:13.654 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:40:13.654 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:40:13.654 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:40:13.662 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:13.943 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:40:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:40:14.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 -- 172.21.15.87:0/2214274778 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f66a40fcbb0 msgr2=0x7f66a40fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:14.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 --2- 172.21.15.87:0/2214274778 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f66a40fcbb0 0x7f66a40fcf90 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 rx=0x7f6698004870 tx=0x7f6698009cb0).stop 2024-02-23T00:40:14.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 -- 172.21.15.87:0/2214274778 shutdown_connections 2024-02-23T00:40:14.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 --2- 172.21.15.87:0/2214274778 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66a40f7400 0x7f66a4103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:14.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 --2- 172.21.15.87:0/2214274778 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f66a40fcbb0 0x7f66a40fcf90 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 --2- 172.21.15.87:0/2214274778 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66a40fd510 0x7f66a40f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 -- 172.21.15.87:0/2214274778 >> 172.21.15.87:0/2214274778 conn(0x7f66a40f2820 msgr2=0x7f66a40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 -- 172.21.15.87:0/2214274778 shutdown_connections 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 -- 172.21.15.87:0/2214274778 wait complete. 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 librados: starting msgr at 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 librados: starting objecter 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 Processor -- start 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 -- start start 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 librados: setting wanted keys 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.981+0000 7f66aaae5700 1 librados: calling monclient init 2024-02-23T00:40:14.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66aaae5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66a40f7400 0x7f66a4109250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66aaae5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f66a40fcbb0 0x7f66a4109740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66aaae5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66a40fd510 0x7f66a4109d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66aaae5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a4114440 con 0x7f66a40f7400 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66aaae5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a41145b0 con 0x7f66a40fd510 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66aaae5700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a4114770 con 0x7f66a40fcbb0 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a8881700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66a40f7400 0x7f66a4109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a8881700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66a40f7400 0x7f66a4109250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41184/0 (socket says 172.21.15.87:41184) 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a8881700 1 -- 172.21.15.87:0/762347543 learned_addr learned my addr 172.21.15.87:0/762347543 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a3fff700 1 --2- 172.21.15.87:0/762347543 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f66a40fcbb0 0x7f66a4109740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a9082700 1 --2- 172.21.15.87:0/762347543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66a40fd510 0x7f66a4109d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a8881700 1 -- 172.21.15.87:0/762347543 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f66a40fcbb0 msgr2=0x7f66a4109740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a8881700 1 --2- 172.21.15.87:0/762347543 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f66a40fcbb0 0x7f66a4109740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a8881700 1 -- 172.21.15.87:0/762347543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66a40fd510 msgr2=0x7f66a4109d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a8881700 1 --2- 172.21.15.87:0/762347543 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f66a40fd510 0x7f66a4109d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a8881700 1 -- 172.21.15.87:0/762347543 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c0097c0 con 0x7f66a40f7400 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a3fff700 1 --2- 172.21.15.87:0/762347543 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f66a40fcbb0 0x7f66a4109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a8881700 1 --2- 172.21.15.87:0/762347543 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f66a40f7400 0x7f66a4109250 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 rx=0x7f6698009eb0 tx=0x7f669800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a1f76700 10 client.?.objecter ms_handle_connect 0x7f66a40f7400 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a1f76700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669800e070 con 0x7f66a40f7400 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6698005040 con 0x7f66a40f7400 2024-02-23T00:40:14.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6698013da0 con 0x7f66a40f7400 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.982+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698019070 con 0x7f66a40f7400 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.983+0000 7f66aaae5700 10 client.27573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.983+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a4114c60 con 0x7f66a40f7400 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.983+0000 7f66aaae5700 1 librados: init done 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.983+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f66a405fa60 con 0x7f66a40f7400 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.984+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f669801e420 con 0x7f66a40f7400 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.984+0000 7f66a1f76700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6694062c90 0x7f6694065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.984+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(594..594 src has 173..594) v4 ==== 8161+0+0 (secure 0 0 0) 0x7f669807dd30 con 0x7f66a40f7400 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.984+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 osd_map(594..594 src has 173..594) v4 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.984+0000 7f66a1f76700 3 client.27573.objecter handle_osd_map got epochs [594,594] > 0 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.984+0000 7f66a1f76700 3 client.27573.objecter handle_osd_map decoding full epoch 594 2024-02-23T00:40:14.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.984+0000 7f66a3fff700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6694062c90 0x7f6694065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.984+0000 7f66a1f76700 20 client.27573.objecter dump_active .. 0 homeless 2024-02-23T00:40:14.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v594) v1 ==== 54+0+16 (secure 0 0 0) 0x7f6698080400 con 0x7f66a40f7400 2024-02-23T00:40:14.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 10 client.27573.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:14.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 1 2024-02-23T00:40:14.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 20 client.27573.objecter _get_session osd=-1 returning homeless 2024-02-23T00:40:14.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 10 client.27573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:14.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=595}) v3 -- 0x7f66a405ee80 con 0x7f66a40f7400 2024-02-23T00:40:14.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f66a4106120 con 0x7f66a40f7400 2024-02-23T00:40:14.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a1f76700 10 client.27573.objecter ms_handle_connect 0x7f6694062c90 2024-02-23T00:40:14.045 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:40:14.045 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:40:14.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a3fff700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6694062c90 0x7f6694065150 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 rx=0x7f66a4061720 tx=0x7f669000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_get_version_reply(handle=1 version=594) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66980512b0 con 0x7f66a40f7400 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a2ffd700 10 client.27573.objecter _check_command_map_dne tid 1 current 594 map_dne_bound 594 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a2ffd700 10 client.27573.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a2ffd700 15 client.27573.objecter _session_command_op_remove -1 1 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 10 client.27573.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 2 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6805/685354171 conn(0x7f66a40fd990 0x7f66a40fdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a4028540 osd=1 3 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a4028540 osd=1 3 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a4028540 osd=1 3 2024-02-23T00:40:14.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 2 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a4028540 osd=1 3 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 1 2 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a4028540 osd=1 4 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 10 client.27573.objecter _send_command 2 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a405a040 con 0x7f66a40fd990 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a9082700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6805/685354171 conn(0x7f66a40fd990 0x7f66a40fdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a1f76700 10 client.27573.objecter ms_handle_connect 0x7f66a40fd990 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.985+0000 7f66a9082700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6805/685354171 conn(0x7f66a40fd990 0x7f66a40fdd70 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.986+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f66a405a040 con 0x7f66a40fd990 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.986+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.986+0000 7f66a1f76700 10 client.27573.objecter _finish_command 2 = system:0 2024-02-23T00:40:14.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.986+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a4028540 osd=1 4 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:13.986+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 1 2 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 10 client.27573.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 3 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a4028540 osd=1 3 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a4028540 osd=1 3 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a4028540 osd=1 3 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 3 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a4028540 osd=1 3 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 1 3 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a4028540 osd=1 4 2024-02-23T00:40:14.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 10 client.27573.objecter _send_command 3 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.001+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a40fe260 con 0x7f66a40fd990 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.002+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f66a40fe260 con 0x7f66a40fd990 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.002+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.002+0000 7f66a1f76700 10 client.27573.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.002+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a4028540 osd=1 4 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.002+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 1 3 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 10 client.27573.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 4 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6809/3605394228 conn(0x7f66a4059710 0x7f66a411b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a40595a0 osd=2 3 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a40595a0 osd=2 3 2024-02-23T00:40:14.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a40595a0 osd=2 3 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 4 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a40595a0 osd=2 3 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 2 4 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a40595a0 osd=2 4 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 10 client.27573.objecter _send_command 4 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a4109050 con 0x7f66a4059710 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66a8881700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6809/3605394228 conn(0x7f66a4059710 0x7f66a411b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66a8881700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6809/3605394228 conn(0x7f66a4059710 0x7f66a411b360 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66a1f76700 10 client.27573.objecter ms_handle_connect 0x7f66a4059710 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f66a4109050 con 0x7f66a4059710 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.003+0000 7f66a1f76700 10 client.27573.objecter _finish_command 4 = system:0 2024-02-23T00:40:14.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.004+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a40595a0 osd=2 4 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.004+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 2 4 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 10 client.27573.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 5 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a40595a0 osd=2 3 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a40595a0 osd=2 3 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a40595a0 osd=2 3 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 5 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a40595a0 osd=2 3 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 2 5 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a40595a0 osd=2 4 2024-02-23T00:40:14.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 10 client.27573.objecter _send_command 5 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.018+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a40fe610 con 0x7f66a4059710 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f66a40fe610 con 0x7f66a4059710 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66a1f76700 10 client.27573.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a40595a0 osd=2 4 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 2 5 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 10 client.27573.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 6 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6813/128059790 conn(0x7f66a40280a0 0x7f66a411e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a40fea20 osd=3 3 2024-02-23T00:40:14.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a40fea20 osd=3 3 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a40fea20 osd=3 3 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 6 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a40fea20 osd=3 3 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 3 6 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a40fea20 osd=3 4 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 10 client.27573.objecter _send_command 6 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a41216c0 con 0x7f66a40280a0 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66a3fff700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6813/128059790 conn(0x7f66a40280a0 0x7f66a411e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66a1f76700 10 client.27573.objecter ms_handle_connect 0x7f66a40280a0 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.020+0000 7f66a3fff700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.87:6813/128059790 conn(0x7f66a40280a0 0x7f66a411e9e0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.021+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f66a41216c0 con 0x7f66a40280a0 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.021+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:40:14.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.021+0000 7f66a1f76700 10 client.27573.objecter _finish_command 6 = system:0 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.021+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a40fea20 osd=3 4 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.021+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 3 6 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 10 client.27573.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 7 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a40fea20 osd=3 3 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a40fea20 osd=3 3 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a40fea20 osd=3 3 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 7 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a40fea20 osd=3 3 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 3 7 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a40fea20 osd=3 4 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 10 client.27573.objecter _send_command 7 2024-02-23T00:40:14.054 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.035+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a4027660 con 0x7f66a40280a0 2024-02-23T00:40:14.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f66a4027660 con 0x7f66a40280a0 2024-02-23T00:40:14.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:14.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66a1f76700 10 client.27573.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a40fea20 osd=3 4 2024-02-23T00:40:14.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 3 7 2024-02-23T00:40:14.055 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 10 client.27573.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 8 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6800/4175844757 conn(0x7f66a4120050 0x7f66a4120490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a411c020 osd=4 3 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a411c020 osd=4 3 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a411c020 osd=4 3 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 8 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a411c020 osd=4 3 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 4 8 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a411c020 osd=4 4 2024-02-23T00:40:14.057 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 10 client.27573.objecter _send_command 8 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a411c750 con 0x7f66a4120050 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66a9082700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6800/4175844757 conn(0x7f66a4120050 0x7f66a4120490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66a1f76700 10 client.27573.objecter ms_handle_connect 0x7f66a4120050 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.037+0000 7f66a9082700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6800/4175844757 conn(0x7f66a4120050 0x7f66a4120490 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.038+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f66a411c750 con 0x7f66a4120050 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.038+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.038+0000 7f66a1f76700 10 client.27573.objecter _finish_command 8 = system:0 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.038+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a411c020 osd=4 4 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.038+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 4 8 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 10 client.27573.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 9 2024-02-23T00:40:14.058 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a411c020 osd=4 3 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a411c020 osd=4 3 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a411c020 osd=4 3 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 9 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a411c020 osd=4 3 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 4 9 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a411c020 osd=4 4 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 10 client.27573.objecter _send_command 9 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.053+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a4121150 con 0x7f66a4120050 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.054+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f66a4121150 con 0x7f66a4120050 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.054+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.054+0000 7f66a1f76700 10 client.27573.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.054+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a411c020 osd=4 4 2024-02-23T00:40:14.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.054+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 4 9 2024-02-23T00:40:14.060 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:40:14.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 10 client.27573.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:14.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 10 2024-02-23T00:40:14.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6804/2550013795 conn(0x7f66a41238c0 0x7f66a4125cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a4120980 osd=5 3 2024-02-23T00:40:14.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a4120980 osd=5 3 2024-02-23T00:40:14.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a4120980 osd=5 3 2024-02-23T00:40:14.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 10 2024-02-23T00:40:14.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a4120980 osd=5 3 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 5 10 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a4120980 osd=5 4 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 10 client.27573.objecter _send_command 10 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a4128900 con 0x7f66a41238c0 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66a8881700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6804/2550013795 conn(0x7f66a41238c0 0x7f66a4125cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66a1f76700 10 client.27573.objecter ms_handle_connect 0x7f66a41238c0 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.055+0000 7f66a8881700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6804/2550013795 conn(0x7f66a41238c0 0x7f66a4125cb0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.056+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f66a4128900 con 0x7f66a41238c0 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.056+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.056+0000 7f66a1f76700 10 client.27573.objecter _finish_command 10 = system:0 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.056+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a4120980 osd=5 4 2024-02-23T00:40:14.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.056+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 5 10 2024-02-23T00:40:14.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 10 client.27573.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 11 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a4120980 osd=5 3 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a4120980 osd=5 3 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a4120980 osd=5 3 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 11 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a4120980 osd=5 3 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 5 11 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a4120980 osd=5 4 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 10 client.27573.objecter _send_command 11 2024-02-23T00:40:14.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.070+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a41261a0 con 0x7f66a41238c0 2024-02-23T00:40:14.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f66a41261a0 con 0x7f66a41238c0 2024-02-23T00:40:14.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:14.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66a1f76700 10 client.27573.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a4120980 osd=5 4 2024-02-23T00:40:14.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 5 11 2024-02-23T00:40:14.075 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 10 client.27573.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 12 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6808/2509408498 conn(0x7f66a4126820 0x7f66a4128c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a41261a0 osd=6 3 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a41261a0 osd=6 3 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a41261a0 osd=6 3 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 12 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a41261a0 osd=6 3 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 6 12 2024-02-23T00:40:14.076 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a41261a0 osd=6 4 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 10 client.27573.objecter _send_command 12 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.072+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a412b950 con 0x7f66a4126820 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.073+0000 7f66a3fff700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6808/2509408498 conn(0x7f66a4126820 0x7f66a4128c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.073+0000 7f66a1f76700 10 client.27573.objecter ms_handle_connect 0x7f66a4126820 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.073+0000 7f66a3fff700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6808/2509408498 conn(0x7f66a4126820 0x7f66a4128c10 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.074+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f66a412b950 con 0x7f66a4126820 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.074+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.074+0000 7f66a1f76700 10 client.27573.objecter _finish_command 12 = system:0 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.074+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a41261a0 osd=6 4 2024-02-23T00:40:14.077 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.074+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 6 12 2024-02-23T00:40:14.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 10 client.27573.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:14.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 13 2024-02-23T00:40:14.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a41261a0 osd=6 3 2024-02-23T00:40:14.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a41261a0 osd=6 3 2024-02-23T00:40:14.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a41261a0 osd=6 3 2024-02-23T00:40:14.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 13 2024-02-23T00:40:14.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a41261a0 osd=6 3 2024-02-23T00:40:14.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 6 13 2024-02-23T00:40:14.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a41261a0 osd=6 4 2024-02-23T00:40:14.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 10 client.27573.objecter _send_command 13 2024-02-23T00:40:14.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.088+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a412b950 con 0x7f66a4126820 2024-02-23T00:40:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f66a412b950 con 0x7f66a4126820 2024-02-23T00:40:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66a1f76700 10 client.27573.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a41261a0 osd=6 4 2024-02-23T00:40:14.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 6 13 2024-02-23T00:40:14.093 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 10 client.27573.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 14 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6812/373787000 conn(0x7f66a412ac90 0x7f66a412b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a412b950 osd=7 3 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a412b950 osd=7 3 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a412b950 osd=7 3 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 14 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a412b950 osd=7 3 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 7 14 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a412b950 osd=7 4 2024-02-23T00:40:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 10 client.27573.objecter _send_command 14 2024-02-23T00:40:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66a412ed80 con 0x7f66a412ac90 2024-02-23T00:40:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.090+0000 7f66a9082700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6812/373787000 conn(0x7f66a412ac90 0x7f66a412b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.091+0000 7f66a1f76700 10 client.27573.objecter ms_handle_connect 0x7f66a412ac90 2024-02-23T00:40:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.091+0000 7f66a9082700 1 --2- 172.21.15.87:0/762347543 >> v2:172.21.15.171:6812/373787000 conn(0x7f66a412ac90 0x7f66a412b0f0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.091+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f66a412ed80 con 0x7f66a412ac90 2024-02-23T00:40:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.091+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:40:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.091+0000 7f66a1f76700 10 client.27573.objecter _finish_command 14 = system:0 2024-02-23T00:40:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.091+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a412b950 osd=7 4 2024-02-23T00:40:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.091+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 7 14 2024-02-23T00:40:14.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 10 client.27573.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:14.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign -1 15 2024-02-23T00:40:14.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a412b950 osd=7 3 2024-02-23T00:40:14.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a412b950 osd=7 3 2024-02-23T00:40:14.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 20 client.27573.objecter _get_session s=0x7f66a412b950 osd=7 3 2024-02-23T00:40:14.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_remove -1 15 2024-02-23T00:40:14.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 20 client.27573.objecter get_session s=0x7f66a412b950 osd=7 3 2024-02-23T00:40:14.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 15 client.27573.objecter _session_command_op_assign 7 15 2024-02-23T00:40:14.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 20 client.27573.objecter put_session s=0x7f66a412b950 osd=7 4 2024-02-23T00:40:14.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 10 client.27573.objecter _send_command 15 2024-02-23T00:40:14.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.106+0000 7f66aaae5700 1 -- 172.21.15.87:0/762347543 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f66a4129130 con 0x7f66a412ac90 2024-02-23T00:40:14.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.107+0000 7f66a1f76700 1 -- 172.21.15.87:0/762347543 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f66a4129130 con 0x7f66a412ac90 2024-02-23T00:40:14.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.107+0000 7f66a1f76700 10 client.27573.objecter ms_dispatch 0x7f66a4001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:14.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.107+0000 7f66a1f76700 10 client.27573.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.107+0000 7f66a1f76700 20 client.27573.objecter put_session s=0x7f66a412b950 osd=7 4 2024-02-23T00:40:14.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:14.107+0000 7f66a1f76700 15 client.27573.objecter _session_command_op_remove 7 15 2024-02-23T00:40:14.110 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:40:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:40:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:13.310682+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:13.313563+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:13.745041+0000 mon.a (mon.0) 3201 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:13.946913+0000 osd.5 (osd.5) 227 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:13.948911+0000 osd.5 (osd.5) 228 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:13.985923+0000 mon.a (mon.0) 3202 : audit [DBG] from='client.? 172.21.15.87:0/762347543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:14.172548+0000 mon.a (mon.0) 3203 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:14.172614+0000 mon.a (mon.0) 3204 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:14.172650+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e595: 8 total, 7 up, 8 in 2024-02-23T00:40:14.369 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:14.174060+0000 mon.a (mon.0) 3206 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:14.386 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-23T00:40:14.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:13.310682+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:40:14.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:13.313563+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:40:14.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:13.745041+0000 mon.a (mon.0) 3201 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-23T00:40:14.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:13.946913+0000 osd.5 (osd.5) 227 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:40:14.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:13.948911+0000 osd.5 (osd.5) 228 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:13.985923+0000 mon.a (mon.0) 3202 : audit [DBG] from='client.? 172.21.15.87:0/762347543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:14.172548+0000 mon.a (mon.0) 3203 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:14.172614+0000 mon.a (mon.0) 3204 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:14.172650+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e595: 8 total, 7 up, 8 in 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:14.174060+0000 mon.a (mon.0) 3206 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:13.310682+0000 osd.6 (osd.6) 238 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:13.313563+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:13.745041+0000 mon.a (mon.0) 3201 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:13.946913+0000 osd.5 (osd.5) 227 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:40:14.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:13.948911+0000 osd.5 (osd.5) 228 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:40:14.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:13.985923+0000 mon.a (mon.0) 3202 : audit [DBG] from='client.? 172.21.15.87:0/762347543' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:14.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:14.172548+0000 mon.a (mon.0) 3203 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi087-82091-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-23T00:40:14.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:14.172614+0000 mon.a (mon.0) 3204 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-23T00:40:14.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:14.172650+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e595: 8 total, 7 up, 8 in 2024-02-23T00:40:14.521 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:14.174060+0000 mon.a (mon.0) 3206 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:14.651 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:13.236476+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:40:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:13.237412+0000 osd.4 (osd.4) 268 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:40:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:13.744092+0000 mgr.y (mgr.24437) 534 : cluster [DBG] pgmap v908: 179 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 27 active+undersized, 116 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/693 objects degraded (5.772%); 61/693 objects misplaced (8.802%) 2024-02-23T00:40:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:14.220293+0000 mon.a (mon.0) 3207 : cluster [WRN] Health check update: Degraded data redundancy: 40/693 objects degraded (5.772%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.039865+0000 osd.1 (osd.1) 273 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:40:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.043409+0000 osd.1 (osd.1) 274 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:40:15.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:13.236476+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:40:15.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:13.237412+0000 osd.4 (osd.4) 268 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:40:15.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:13.744092+0000 mgr.y (mgr.24437) 534 : cluster [DBG] pgmap v908: 179 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 27 active+undersized, 116 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/693 objects degraded (5.772%); 61/693 objects misplaced (8.802%) 2024-02-23T00:40:15.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:14.220293+0000 mon.a (mon.0) 3207 : cluster [WRN] Health check update: Degraded data redundancy: 40/693 objects degraded (5.772%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:15.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.039865+0000 osd.1 (osd.1) 273 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:40:15.720 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.043409+0000 osd.1 (osd.1) 274 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:40:15.720 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:13.236476+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:40:15.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:13.237412+0000 osd.4 (osd.4) 268 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:40:15.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:13.744092+0000 mgr.y (mgr.24437) 534 : cluster [DBG] pgmap v908: 179 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 27 active+undersized, 116 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/693 objects degraded (5.772%); 61/693 objects misplaced (8.802%) 2024-02-23T00:40:15.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:14.220293+0000 mon.a (mon.0) 3207 : cluster [WRN] Health check update: Degraded data redundancy: 40/693 objects degraded (5.772%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:15.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.039865+0000 osd.1 (osd.1) 273 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:40:15.721 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.043409+0000 osd.1 (osd.1) 274 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:40:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:40:15.740 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:40:16.214 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:16.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.212+0000 7fb8c21bf700 1 -- 172.21.15.87:0/1421249637 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 msgr2=0x7fb8bc0f6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:16.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.212+0000 7fb8c21bf700 1 --2- 172.21.15.87:0/1421249637 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 0x7fb8bc0f6790 secure :-1 s=READY pgs=4991 cs=0 l=1 rev1=1 rx=0x7fb8b0004870 tx=0x7fb8b0009cb0).stop 2024-02-23T00:40:16.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.212+0000 7fb8c21bf700 1 -- 172.21.15.87:0/1421249637 shutdown_connections 2024-02-23T00:40:16.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.212+0000 7fb8c21bf700 1 --2- 172.21.15.87:0/1421249637 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8bc1087f0 0x7fb8bc10abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:16.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.212+0000 7fb8c21bf700 1 --2- 172.21.15.87:0/1421249637 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 0x7fb8bc0f6790 unknown :-1 s=CLOSED pgs=4991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:16.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.212+0000 7fb8c21bf700 1 --2- 172.21.15.87:0/1421249637 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8bc0f6c80 0x7fb8bc108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:16.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.212+0000 7fb8c21bf700 1 -- 172.21.15.87:0/1421249637 >> 172.21.15.87:0/1421249637 conn(0x7fb8bc06c370 msgr2=0x7fb8bc06c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:16.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.212+0000 7fb8c21bf700 1 -- 172.21.15.87:0/1421249637 shutdown_connections 2024-02-23T00:40:16.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.212+0000 7fb8c21bf700 1 -- 172.21.15.87:0/1421249637 wait complete. 2024-02-23T00:40:16.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.213+0000 7fb8c21bf700 1 librados: starting msgr at 2024-02-23T00:40:16.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.213+0000 7fb8c21bf700 1 librados: starting objecter 2024-02-23T00:40:16.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.213+0000 7fb8c21bf700 1 Processor -- start 2024-02-23T00:40:16.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.213+0000 7fb8c21bf700 1 -- start start 2024-02-23T00:40:16.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.213+0000 7fb8c21bf700 1 librados: setting wanted keys 2024-02-23T00:40:16.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.213+0000 7fb8c21bf700 1 librados: calling monclient init 2024-02-23T00:40:16.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8c21bf700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 0x7fb8bc0fe6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:16.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8c21bf700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8bc0f6c80 0x7fb8bc0febb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:16.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8c21bf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8bc1087f0 0x7fb8bc122e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:16.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8c21bf700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0ff370 con 0x7fb8bc1087f0 2024-02-23T00:40:16.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8c21bf700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc062700 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8c21bf700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc062890 con 0x7fb8bc0f6c80 2024-02-23T00:40:16.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8bb7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 0x7fb8bc0fe6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:16.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8bb7fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 0x7fb8bc0fe6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.87:50606/0 (socket says 172.21.15.87:50606) 2024-02-23T00:40:16.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8bb7fe700 1 -- 172.21.15.87:0/2425268595 learned_addr learned my addr 172.21.15.87:0/2425268595 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:16.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.214+0000 7fb8baffd700 1 --2- 172.21.15.87:0/2425268595 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8bc0f6c80 0x7fb8bc0febb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8bb7fe700 1 -- 172.21.15.87:0/2425268595 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8bc0f6c80 msgr2=0x7fb8bc0febb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8bb7fe700 1 --2- 172.21.15.87:0/2425268595 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8bc0f6c80 0x7fb8bc0febb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8bb7fe700 1 -- 172.21.15.87:0/2425268595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8bc1087f0 msgr2=0x7fb8bc122e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8bb7fe700 1 --2- 172.21.15.87:0/2425268595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8bc1087f0 0x7fb8bc122e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8bb7fe700 1 -- 172.21.15.87:0/2425268595 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b0005040 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8bb7fe700 1 --2- 172.21.15.87:0/2425268595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 0x7fb8bc0fe6c0 secure :-1 s=READY pgs=4992 cs=0 l=1 rev1=1 rx=0x7fb8b0009da0 tx=0x7fb8b0007f90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8b8ff9700 10 client.?.objecter ms_handle_connect 0x7fb8bc0f63b0 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8b8ff9700 1 -- 172.21.15.87:0/2425268595 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000e030 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8b8ff9700 1 -- 172.21.15.87:0/2425268595 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8b00093b0 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8c21bf700 1 -- 172.21.15.87:0/2425268595 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc062b10 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8baffd700 1 --2- 172.21.15.87:0/2425268595 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8bc0f6c80 0x7fb8bc0febb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:40:16.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8c21bf700 10 client.45385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:16.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8c21bf700 1 -- 172.21.15.87:0/2425268595 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc062fa0 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8c21bf700 1 librados: init done 2024-02-23T00:40:16.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.215+0000 7fb8b8ff9700 1 -- 172.21.15.87:0/2425268595 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0018070 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.216+0000 7fb8a27fc700 1 -- 172.21.15.87:0/2425268595 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc05fa40 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.218+0000 7fb8b8ff9700 1 -- 172.21.15.87:0/2425268595 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb8b0013ad0 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.218+0000 7fb8b8ff9700 1 --2- 172.21.15.87:0/2425268595 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb8a4062ce0 0x7fb8a40651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:16.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.219+0000 7fb8b8ff9700 1 -- 172.21.15.87:0/2425268595 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(597..597 src has 173..597) v4 ==== 8981+0+0 (secure 0 0 0) 0x7fb8b007eb80 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.219+0000 7fb8b8ff9700 10 client.45385.objecter ms_dispatch 0x7fb8bc001550 osd_map(597..597 src has 173..597) v4 2024-02-23T00:40:16.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.219+0000 7fb8b8ff9700 3 client.45385.objecter handle_osd_map got epochs [597,597] > 0 2024-02-23T00:40:16.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.219+0000 7fb8b8ff9700 3 client.45385.objecter handle_osd_map decoding full epoch 597 2024-02-23T00:40:16.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.219+0000 7fb8b8ff9700 20 client.45385.objecter dump_active .. 0 homeless 2024-02-23T00:40:16.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.219+0000 7fb8baffd700 1 --2- 172.21.15.87:0/2425268595 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb8a4062ce0 0x7fb8a40651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:16.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.219+0000 7fb8baffd700 1 --2- 172.21.15.87:0/2425268595 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb8a4062ce0 0x7fb8a40651a0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 rx=0x7fb8bc0ffca0 tx=0x7fb8ac009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:16.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.219+0000 7fb8b8ff9700 10 client.45385.objecter ms_handle_connect 0x7fb8a4062ce0 2024-02-23T00:40:16.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.222+0000 7fb8b8ff9700 1 -- 172.21.15.87:0/2425268595 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb8b0051d30 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.319 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:16.356 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:16.380 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:16.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:16.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:16.431+0000 7fb8a27fc700 1 -- 172.21.15.87:0/2425268595 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fb8bc10d6e0 con 0x7fb8bc0f63b0 2024-02-23T00:40:16.485 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:16.515 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:16.543 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:16.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:16.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:16.574 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:40:16.574 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:40:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.200822+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:40:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.208116+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:40:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.215028+0000 osd.4 (osd.4) 269 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:40:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.217218+0000 osd.4 (osd.4) 270 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:40:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.259057+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.26 scrub starts 2024-02-23T00:40:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.260200+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.26 scrub ok 2024-02-23T00:40:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.435253+0000 mon.a (mon.0) 3208 : cluster [DBG] osdmap e596: 8 total, 7 up, 8 in 2024-02-23T00:40:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:16.176985+0000 mon.a (mon.0) 3209 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:16.177103+0000 mon.a (mon.0) 3210 : cluster [DBG] osdmap e597: 8 total, 7 up, 8 in 2024-02-23T00:40:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.200822+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:40:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.208116+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:40:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.215028+0000 osd.4 (osd.4) 269 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:40:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.217218+0000 osd.4 (osd.4) 270 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:40:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.259057+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.26 scrub starts 2024-02-23T00:40:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.260200+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.26 scrub ok 2024-02-23T00:40:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.435253+0000 mon.a (mon.0) 3208 : cluster [DBG] osdmap e596: 8 total, 7 up, 8 in 2024-02-23T00:40:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:16.176985+0000 mon.a (mon.0) 3209 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:16.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:16.177103+0000 mon.a (mon.0) 3210 : cluster [DBG] osdmap e597: 8 total, 7 up, 8 in 2024-02-23T00:40:16.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.200822+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:40:16.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.208116+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:40:16.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.215028+0000 osd.4 (osd.4) 269 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:40:16.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.217218+0000 osd.4 (osd.4) 270 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:40:16.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.259057+0000 osd.2 (osd.2) 217 : cluster [DBG] 5.26 scrub starts 2024-02-23T00:40:16.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.260200+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.26 scrub ok 2024-02-23T00:40:16.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.435253+0000 mon.a (mon.0) 3208 : cluster [DBG] osdmap e596: 8 total, 7 up, 8 in 2024-02-23T00:40:16.594 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:16.176985+0000 mon.a (mon.0) 3209 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi087-82091-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:16.595 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:16.177103+0000 mon.a (mon.0) 3210 : cluster [DBG] osdmap e597: 8 total, 7 up, 8 in 2024-02-23T00:40:16.647 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:16.675 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:16.874 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:16.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:16.977 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:17.004 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:17.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.188+0000 7fb8b8ff9700 1 -- 172.21.15.87:0/2425268595 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v598) v1 ==== 101+0+0 (secure 0 0 0) 0x7fb8b0004490 con 0x7fb8bc0f63b0 2024-02-23T00:40:17.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.193+0000 7fb8a27fc700 10 librados: watch_flush enter 2024-02-23T00:40:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.197+0000 7fb8a27fc700 10 librados: watch_flush exit 2024-02-23T00:40:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.197+0000 7fb8a27fc700 20 client.45385.objecter shutdown clearing up homeless session... 2024-02-23T00:40:17.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.197+0000 7fb8a27fc700 10 client.45385.objecter successfully canceled tick 2024-02-23T00:40:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.198+0000 7fb8a27fc700 1 -- 172.21.15.87:0/2425268595 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb8a4062ce0 msgr2=0x7fb8a40651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.198+0000 7fb8a27fc700 1 --2- 172.21.15.87:0/2425268595 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb8a4062ce0 0x7fb8a40651a0 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 rx=0x7fb8bc0ffca0 tx=0x7fb8ac009040).stop 2024-02-23T00:40:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.198+0000 7fb8a27fc700 1 -- 172.21.15.87:0/2425268595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 msgr2=0x7fb8bc0fe6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:17.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.198+0000 7fb8a27fc700 1 --2- 172.21.15.87:0/2425268595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 0x7fb8bc0fe6c0 secure :-1 s=READY pgs=4992 cs=0 l=1 rev1=1 rx=0x7fb8b0009da0 tx=0x7fb8b0007f90).stop 2024-02-23T00:40:17.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.202+0000 7fb8a27fc700 1 -- 172.21.15.87:0/2425268595 shutdown_connections 2024-02-23T00:40:17.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.202+0000 7fb8a27fc700 1 --2- 172.21.15.87:0/2425268595 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb8a4062ce0 0x7fb8a40651a0 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:17.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.202+0000 7fb8a27fc700 1 --2- 172.21.15.87:0/2425268595 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8bc1087f0 0x7fb8bc122e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:40:17.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.202+0000 7fb8a27fc700 1 --2- 172.21.15.87:0/2425268595 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8bc0f63b0 0x7fb8bc0fe6c0 unknown :-1 s=CLOSED pgs=4992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:17.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.202+0000 7fb8a27fc700 1 --2- 172.21.15.87:0/2425268595 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8bc0f6c80 0x7fb8bc0febb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:17.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.202+0000 7fb8a27fc700 1 -- 172.21.15.87:0/2425268595 >> 172.21.15.87:0/2425268595 conn(0x7fb8bc06c370 msgr2=0x7fb8bc106b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:17.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.213+0000 7fb8a27fc700 1 -- 172.21.15.87:0/2425268595 shutdown_connections 2024-02-23T00:40:17.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.214+0000 7fb8a27fc700 1 -- 172.21.15.87:0/2425268595 wait complete. 2024-02-23T00:40:17.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:17.214+0000 7fb8a27fc700 1 librados: shutdown 2024-02-23T00:40:17.220 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:40:17.335 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-23T00:40:17.353 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:17.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:17.457 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:17.485 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:17.504 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:17.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:15.745121+0000 mgr.y (mgr.24437) 535 : cluster [DBG] pgmap v911: 179 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 2 clean+premerge+peered, 9 active+undersized+degraded, 27 active+undersized, 115 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%) 2024-02-23T00:40:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:16.433839+0000 mon.b (mon.1) 455 : audit [INF] from='client.? 172.21.15.87:0/2425268595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:16.439977+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:17.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.040416+0000 osd.1 (osd.1) 275 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.043317+0000 osd.1 (osd.1) 276 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.166260+0000 mon.a (mon.0) 3212 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.175020+0000 mon.a (mon.0) 3213 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:17.186734+0000 mon.a (mon.0) 3214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.186812+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e598: 8 total, 7 up, 8 in 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:17.188639+0000 mon.a (mon.0) 3216 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:15.745121+0000 mgr.y (mgr.24437) 535 : cluster [DBG] pgmap v911: 179 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 2 clean+premerge+peered, 9 active+undersized+degraded, 27 active+undersized, 115 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%) 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:16.433839+0000 mon.b (mon.1) 455 : audit [INF] from='client.? 172.21.15.87:0/2425268595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:16.439977+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:17.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.040416+0000 osd.1 (osd.1) 275 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:40:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.043317+0000 osd.1 (osd.1) 276 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:40:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.166260+0000 mon.a (mon.0) 3212 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.175020+0000 mon.a (mon.0) 3213 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:40:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:17.186734+0000 mon.a (mon.0) 3214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:40:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.186812+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e598: 8 total, 7 up, 8 in 2024-02-23T00:40:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:17.188639+0000 mon.a (mon.0) 3216 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:17.608 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:17.624 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:17.632 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:40:17.632 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:40:17.633 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 4, 5, 6, 7, 3] 2024-02-23T00:40:17.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd out 0 2024-02-23T00:40:17.633 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:15.745121+0000 mgr.y (mgr.24437) 535 : cluster [DBG] pgmap v911: 179 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 2 clean+premerge+peered, 9 active+undersized+degraded, 27 active+undersized, 115 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%) 2024-02-23T00:40:17.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:16.433839+0000 mon.b (mon.1) 455 : audit [INF] from='client.? 172.21.15.87:0/2425268595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:17.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:16.439977+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:17.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.040416+0000 osd.1 (osd.1) 275 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:40:17.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.043317+0000 osd.1 (osd.1) 276 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:40:17.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.166260+0000 mon.a (mon.0) 3212 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:17.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.175020+0000 mon.a (mon.0) 3213 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:40:17.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:17.186734+0000 mon.a (mon.0) 3214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:40:17.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.186812+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e598: 8 total, 7 up, 8 in 2024-02-23T00:40:17.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:17.188639+0000 mon.a (mon.0) 3216 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:17.635 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:17.657 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:17.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:17.741 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:40:17.760 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:17.811 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:18.160 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:18.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:18.197 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:18.264 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:18.292 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:18.495 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:18.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:18.599 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:18.627 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:18.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.236009+0000 osd.2 (osd.2) 219 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:40:18.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.268560+0000 osd.2 (osd.2) 220 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:40:18.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.686340+0000 osd.7 (osd.7) 276 : cluster [DBG] 6.15 deep-scrub starts 2024-02-23T00:40:18.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.689236+0000 osd.7 (osd.7) 277 : cluster [DBG] 6.15 deep-scrub ok 2024-02-23T00:40:18.713 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.745704+0000 mgr.y (mgr.24437) 536 : cluster [DBG] pgmap v914: 218 pgs: 1 active+remapped+backfill_toofull, 40 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 1 clean+premerge+peered, 9 active+undersized+degraded, 27 active+undersized, 115 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%); 170 B/s, 0 objects/s recovering 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.977562+0000 osd.5 (osd.5) 229 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:17.978556+0000 osd.5 (osd.5) 230 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:18.189306+0000 mon.a (mon.0) 3217 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:18.189488+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e599: 8 total, 7 up, 8 in 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:18.245555+0000 mon.a (mon.0) 3219 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.236009+0000 osd.2 (osd.2) 219 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.268560+0000 osd.2 (osd.2) 220 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.686340+0000 osd.7 (osd.7) 276 : cluster [DBG] 6.15 deep-scrub starts 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.689236+0000 osd.7 (osd.7) 277 : cluster [DBG] 6.15 deep-scrub ok 2024-02-23T00:40:18.714 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.745704+0000 mgr.y (mgr.24437) 536 : cluster [DBG] pgmap v914: 218 pgs: 1 active+remapped+backfill_toofull, 40 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 1 clean+premerge+peered, 9 active+undersized+degraded, 27 active+undersized, 115 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%); 170 B/s, 0 objects/s recovering 2024-02-23T00:40:18.715 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.977562+0000 osd.5 (osd.5) 229 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:40:18.715 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:17.978556+0000 osd.5 (osd.5) 230 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:40:18.715 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:18.189306+0000 mon.a (mon.0) 3217 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:18.715 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:18.189488+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e599: 8 total, 7 up, 8 in 2024-02-23T00:40:18.715 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:18.245555+0000 mon.a (mon.0) 3219 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:40:18.891 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.236009+0000 osd.2 (osd.2) 219 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:40:18.892 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.268560+0000 osd.2 (osd.2) 220 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:40:18.892 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.686340+0000 osd.7 (osd.7) 276 : cluster [DBG] 6.15 deep-scrub starts 2024-02-23T00:40:18.892 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.689236+0000 osd.7 (osd.7) 277 : cluster [DBG] 6.15 deep-scrub ok 2024-02-23T00:40:18.892 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.745704+0000 mgr.y (mgr.24437) 536 : cluster [DBG] pgmap v914: 218 pgs: 1 active+remapped+backfill_toofull, 40 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 1 clean+premerge+peered, 9 active+undersized+degraded, 27 active+undersized, 115 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%); 170 B/s, 0 objects/s recovering 2024-02-23T00:40:18.892 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.977562+0000 osd.5 (osd.5) 229 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:40:18.892 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:17.978556+0000 osd.5 (osd.5) 230 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:40:18.892 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:18.189306+0000 mon.a (mon.0) 3217 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:18.892 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:18.189488+0000 mon.a (mon.0) 3218 : cluster [DBG] osdmap e599: 8 total, 7 up, 8 in 2024-02-23T00:40:18.892 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:18.245555+0000 mon.a (mon.0) 3219 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:40:19.025 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:19.178 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:19.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:19.282 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:19.316 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:19.445 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:40:19.445 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:40:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.123482+0000 osd.1 (osd.1) 277 : cluster [DBG] 22.1c deep-scrub starts 2024-02-23T00:40:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.126371+0000 osd.1 (osd.1) 278 : cluster [DBG] 22.1c deep-scrub ok 2024-02-23T00:40:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:19.190574+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:40:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.190646+0000 mon.a (mon.0) 3221 : cluster [DBG] osdmap e600: 8 total, 7 up, 8 in 2024-02-23T00:40:19.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:19.192801+0000 mon.a (mon.0) 3222 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-102"}]: dispatch 2024-02-23T00:40:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.204303+0000 osd.4 (osd.4) 271 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:40:19.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.205734+0000 osd.4 (osd.4) 272 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:40:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.123482+0000 osd.1 (osd.1) 277 : cluster [DBG] 22.1c deep-scrub starts 2024-02-23T00:40:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.126371+0000 osd.1 (osd.1) 278 : cluster [DBG] 22.1c deep-scrub ok 2024-02-23T00:40:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:19.190574+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:40:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.190646+0000 mon.a (mon.0) 3221 : cluster [DBG] osdmap e600: 8 total, 7 up, 8 in 2024-02-23T00:40:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:19.192801+0000 mon.a (mon.0) 3222 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-102"}]: dispatch 2024-02-23T00:40:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.204303+0000 osd.4 (osd.4) 271 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:40:19.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.205734+0000 osd.4 (osd.4) 272 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:40:19.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.123482+0000 osd.1 (osd.1) 277 : cluster [DBG] 22.1c deep-scrub starts 2024-02-23T00:40:19.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.126371+0000 osd.1 (osd.1) 278 : cluster [DBG] 22.1c deep-scrub ok 2024-02-23T00:40:19.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:19.190574+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:40:19.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.190646+0000 mon.a (mon.0) 3221 : cluster [DBG] osdmap e600: 8 total, 7 up, 8 in 2024-02-23T00:40:19.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:19.192801+0000 mon.a (mon.0) 3222 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-102"}]: dispatch 2024-02-23T00:40:19.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.204303+0000 osd.4 (osd.4) 271 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:40:19.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.205734+0000 osd.4 (osd.4) 272 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:40:20.092 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-23T00:40:20.125 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:40:20.273 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:20.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:20.377 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:20.410 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:20.415 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:20.430 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:20.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:20.467 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.372634+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:40:20.467 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.377279+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:40:20.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.659192+0000 osd.7 (osd.7) 278 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:40:20.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.659965+0000 osd.7 (osd.7) 279 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:40:20.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:19.746271+0000 mgr.y (mgr.24437) 537 : cluster [DBG] pgmap v917: 218 pgs: 1 active+remapped+backfill_toofull, 40 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 1 clean+premerge+peered, 9 active+undersized+degraded, 27 active+undersized, 115 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%) 2024-02-23T00:40:20.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:20.032045+0000 osd.5 (osd.5) 231 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:40:20.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:20.033050+0000 osd.5 (osd.5) 232 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:40:20.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:20.192618+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-102"}]': finished 2024-02-23T00:40:20.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:20.192688+0000 mon.a (mon.0) 3224 : cluster [DBG] osdmap e601: 8 total, 7 up, 8 in 2024-02-23T00:40:20.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:20.233126+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:20.534 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:20.560 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.372634+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.377279+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.659192+0000 osd.7 (osd.7) 278 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.659965+0000 osd.7 (osd.7) 279 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:19.746271+0000 mgr.y (mgr.24437) 537 : cluster [DBG] pgmap v917: 218 pgs: 1 active+remapped+backfill_toofull, 40 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 1 clean+premerge+peered, 9 active+undersized+degraded, 27 active+undersized, 115 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%) 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:20.032045+0000 osd.5 (osd.5) 231 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:20.033050+0000 osd.5 (osd.5) 232 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:20.192618+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-102"}]': finished 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:20.192688+0000 mon.a (mon.0) 3224 : cluster [DBG] osdmap e601: 8 total, 7 up, 8 in 2024-02-23T00:40:20.660 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:20.233126+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:20.661 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:20.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:20.765 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:20.792 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:20.819 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.372634+0000 osd.6 (osd.6) 240 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:40:20.819 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.377279+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:40:20.819 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.659192+0000 osd.7 (osd.7) 278 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:40:20.820 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.659965+0000 osd.7 (osd.7) 279 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:40:20.820 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:19.746271+0000 mgr.y (mgr.24437) 537 : cluster [DBG] pgmap v917: 218 pgs: 1 active+remapped+backfill_toofull, 40 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 1 clean+premerge+peered, 9 active+undersized+degraded, 27 active+undersized, 115 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40/702 objects degraded (5.698%); 61/702 objects misplaced (8.689%) 2024-02-23T00:40:20.820 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:20.032045+0000 osd.5 (osd.5) 231 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:40:20.820 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:20.033050+0000 osd.5 (osd.5) 232 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:40:20.820 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:20.192618+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-102"}]': finished 2024-02-23T00:40:20.820 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:20.192688+0000 mon.a (mon.0) 3224 : cluster [DBG] osdmap e601: 8 total, 7 up, 8 in 2024-02-23T00:40:20.820 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:20.233126+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:21.241 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:21.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:21.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:21.345 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:21.359 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:40:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:40:21.381 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:21.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 -- 172.21.15.87:0/3810338204 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 msgr2=0x7f42fc104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:21.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 --2- 172.21.15.87:0/3810338204 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 0x7f42fc104a20 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 rx=0x7f42f4009be0 tx=0x7f42f4004d10).stop 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 -- 172.21.15.87:0/3810338204 shutdown_connections 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 --2- 172.21.15.87:0/3810338204 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 0x7f42fc104a20 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 --2- 172.21.15.87:0/3810338204 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42fc0fdbb0 0x7f42fc0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 --2- 172.21.15.87:0/3810338204 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42fc0fe510 0x7f42fc0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 -- 172.21.15.87:0/3810338204 >> 172.21.15.87:0/3810338204 conn(0x7f42fc0f3800 msgr2=0x7f42fc0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 -- 172.21.15.87:0/3810338204 shutdown_connections 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 -- 172.21.15.87:0/3810338204 wait complete. 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 librados: starting msgr at 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.794+0000 7f43035ae700 1 librados: starting objecter 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 Processor -- start 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 -- start start 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 librados: setting wanted keys 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 librados: calling monclient init 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 0x7f42fc10df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:21.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42fc0fdbb0 0x7f42fc10e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42fc0fe510 0x7f42fc108110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc108600 con 0x7f42fc0fe510 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc108770 con 0x7f42fc0fdbb0 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f43035ae700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc108930 con 0x7f42fc0f8420 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f430134a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 0x7f42fc10df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f4301b4b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42fc0fe510 0x7f42fc108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f430134a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 0x7f42fc10df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:54002/0 (socket says 172.21.15.87:54002) 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f430134a700 1 -- 172.21.15.87:0/463382080 learned_addr learned my addr 172.21.15.87:0/463382080 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f4300b49700 1 --2- 172.21.15.87:0/463382080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42fc0fdbb0 0x7f42fc10e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f430134a700 1 -- 172.21.15.87:0/463382080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42fc0fdbb0 msgr2=0x7f42fc10e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f430134a700 1 --2- 172.21.15.87:0/463382080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42fc0fdbb0 0x7f42fc10e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f430134a700 1 -- 172.21.15.87:0/463382080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42fc0fe510 msgr2=0x7f42fc108110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f430134a700 1 --2- 172.21.15.87:0/463382080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42fc0fe510 0x7f42fc108110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f430134a700 1 -- 172.21.15.87:0/463382080 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f8005040 con 0x7f42fc0f8420 2024-02-23T00:40:21.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f4301b4b700 1 --2- 172.21.15.87:0/463382080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42fc0fe510 0x7f42fc108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f430134a700 1 --2- 172.21.15.87:0/463382080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 0x7f42fc10df70 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7f42ec00e990 tx=0x7f42ec00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f42f27fc700 10 client.?.objecter ms_handle_connect 0x7f42fc0f8420 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.795+0000 7f42f27fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.796+0000 7f42f27fc700 1 -- 172.21.15.87:0/463382080 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec00ee80 con 0x7f42fc0f8420 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.796+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f40097c0 con 0x7f42fc0f8420 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.796+0000 7f42f27fc700 1 -- 172.21.15.87:0/463382080 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42ec00f040 con 0x7f42fc0f8420 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.796+0000 7f42f27fc700 1 -- 172.21.15.87:0/463382080 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec00c840 con 0x7f42fc0f8420 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.796+0000 7f43035ae700 10 client.38054.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.796+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42fc108ec0 con 0x7f42fc0f8420 2024-02-23T00:40:21.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.796+0000 7f43035ae700 1 librados: init done 2024-02-23T00:40:21.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.797+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42fc107110 con 0x7f42fc0f8420 2024-02-23T00:40:21.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.797+0000 7f42f27fc700 1 -- 172.21.15.87:0/463382080 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f42ec00cc70 con 0x7f42fc0f8420 2024-02-23T00:40:21.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.797+0000 7f42f27fc700 1 --2- 172.21.15.87:0/463382080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f42e8062c90 0x7f42e8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:21.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.797+0000 7f42f27fc700 1 -- 172.21.15.87:0/463382080 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(602..602 src has 173..602) v4 ==== 9362+0+0 (secure 0 0 0) 0x7f42ec07d320 con 0x7f42fc0f8420 2024-02-23T00:40:21.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.797+0000 7f42f27fc700 10 client.38054.objecter ms_dispatch 0x7f42fc001550 osd_map(602..602 src has 173..602) v4 2024-02-23T00:40:21.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.797+0000 7f4300b49700 1 --2- 172.21.15.87:0/463382080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f42e8062c90 0x7f42e8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:21.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.797+0000 7f42f27fc700 3 client.38054.objecter handle_osd_map got epochs [602,602] > 0 2024-02-23T00:40:21.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.797+0000 7f42f27fc700 3 client.38054.objecter handle_osd_map decoding full epoch 602 2024-02-23T00:40:21.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.798+0000 7f42f27fc700 20 client.38054.objecter dump_active .. 0 homeless 2024-02-23T00:40:21.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.798+0000 7f42f27fc700 10 client.38054.objecter ms_handle_connect 0x7f42e8062c90 2024-02-23T00:40:21.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.798+0000 7f4300b49700 1 --2- 172.21.15.87:0/463382080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f42e8062c90 0x7f42e8065150 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7f42fc1098b0 tx=0x7f42f8009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:21.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.802+0000 7f42f27fc700 1 -- 172.21.15.87:0/463382080 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f42ec049f60 con 0x7f42fc0f8420 2024-02-23T00:40:21.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:21.978+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd out", "ids": ["0"]} v 0) v1 -- 0x7f42fc1095f0 con 0x7f42fc0f8420 2024-02-23T00:40:22.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.032+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/2541216806 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcbb00fe510 msgr2=0x7fcbb00f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:22.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.032+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/2541216806 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcbb00fe510 0x7fcbb00f7d40 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 rx=0x7fcba40047b0 tx=0x7fcba4009cb0).stop 2024-02-23T00:40:22.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.033+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/2541216806 shutdown_connections 2024-02-23T00:40:22.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.033+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/2541216806 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcbb00f8420 0x7fcbb0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:22.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.033+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/2541216806 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbb00fdbb0 0x7fcbb00fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.033+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/2541216806 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcbb00fe510 0x7fcbb00f7d40 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.033+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/2541216806 >> 172.21.15.87:0/2541216806 conn(0x7fcbb00f3800 msgr2=0x7fcbb00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.033+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/2541216806 shutdown_connections 2024-02-23T00:40:22.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.033+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/2541216806 wait complete. 2024-02-23T00:40:22.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.033+0000 7fcbb7dc7700 1 librados: starting msgr at 2024-02-23T00:40:22.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.033+0000 7fcbb7dc7700 1 librados: starting objecter 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 Processor -- start 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 -- start start 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 librados: setting wanted keys 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 librados: calling monclient init 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbb00f8420 0x7fcbb0110310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcbb00fdbb0 0x7fcbb0110800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcbb00fe510 0x7fcbb010a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb010a9a0 con 0x7fcbb00f8420 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb010ab10 con 0x7fcbb00fe510 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb7dc7700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb010acd0 con 0x7fcbb00fdbb0 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb5b63700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbb00f8420 0x7fcbb0110310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb5b63700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbb00f8420 0x7fcbb0110310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58262/0 (socket says 172.21.15.87:58262) 2024-02-23T00:40:22.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb5b63700 1 -- 172.21.15.87:0/4195866527 learned_addr learned my addr 172.21.15.87:0/4195866527 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb5362700 1 --2- 172.21.15.87:0/4195866527 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcbb00fdbb0 0x7fcbb0110800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb6364700 1 --2- 172.21.15.87:0/4195866527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcbb00fe510 0x7fcbb010a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb6364700 1 -- 172.21.15.87:0/4195866527 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcbb00fdbb0 msgr2=0x7fcbb0110800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb6364700 1 --2- 172.21.15.87:0/4195866527 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcbb00fdbb0 0x7fcbb0110800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb6364700 1 -- 172.21.15.87:0/4195866527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbb00f8420 msgr2=0x7fcbb0110310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb6364700 1 --2- 172.21.15.87:0/4195866527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbb00f8420 0x7fcbb0110310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb6364700 1 -- 172.21.15.87:0/4195866527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba00097c0 con 0x7fcbb00fe510 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb5b63700 1 --2- 172.21.15.87:0/4195866527 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcbb00f8420 0x7fcbb0110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.034+0000 7fcbb5362700 1 --2- 172.21.15.87:0/4195866527 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcbb00fdbb0 0x7fcbb0110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbb6364700 1 --2- 172.21.15.87:0/4195866527 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcbb00fe510 0x7fcbb010a4b0 secure :-1 s=READY pgs=4993 cs=0 l=1 rev1=1 rx=0x7fcba0001df0 tx=0x7fcba0004d10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbaeffd700 10 client.?.objecter ms_handle_connect 0x7fcbb00fe510 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbaeffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba00099f0 con 0x7fcbb00fe510 2024-02-23T00:40:22.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba8005040 con 0x7fcbb00fe510 2024-02-23T00:40:22.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== mon.1 v2:172.21.15.171:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcba00090d0 con 0x7fcbb00fe510 2024-02-23T00:40:22.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== mon.1 v2:172.21.15.171:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba0007c60 con 0x7fcbb00fe510 2024-02-23T00:40:22.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbb7dc7700 10 client.45397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:22.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba4005040 con 0x7fcbb00fe510 2024-02-23T00:40:22.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.035+0000 7fcbb7dc7700 1 librados: init done 2024-02-23T00:40:22.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.036+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fcbb005ee80 con 0x7fcbb00fe510 2024-02-23T00:40:22.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.036+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== mon.1 v2:172.21.15.171:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fcba0015630 con 0x7fcbb00fe510 2024-02-23T00:40:22.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.036+0000 7fcbaeffd700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcb98062c90 0x7fcb98065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== mon.1 v2:172.21.15.171:3300/0 5 ==== osd_map(602..602 src has 173..602) v4 ==== 9362+0+0 (secure 0 0 0) 0x7fcba007e050 con 0x7fcbb00fe510 2024-02-23T00:40:22.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 osd_map(602..602 src has 173..602) v4 2024-02-23T00:40:22.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaeffd700 3 client.45397.objecter handle_osd_map got epochs [602,602] > 0 2024-02-23T00:40:22.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaeffd700 3 client.45397.objecter handle_osd_map decoding full epoch 602 2024-02-23T00:40:22.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaeffd700 20 client.45397.objecter dump_active .. 0 homeless 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== mon.1 v2:172.21.15.171:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v602) v1 ==== 54+0+16 (secure 0 0 0) 0x7fcba0085070 con 0x7fcbb00fe510 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 1 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 20 client.45397.objecter _get_session osd=-1 returning homeless 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 10 client.45397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_subscribe({osdmap=603}) v3 -- 0x7fcbb0107120 con 0x7fcbb00fe510 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcbb010b990 con 0x7fcbb00fe510 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== mon.1 v2:172.21.15.171:3300/0 7 ==== mon_get_version_reply(handle=1 version=602) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcba0051150 con 0x7fcbb00fe510 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaffff700 10 client.45397.objecter _check_command_map_dne tid 1 current 602 map_dne_bound 602 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaffff700 10 client.45397.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbaffff700 15 client.45397.objecter _session_command_op_remove -1 1 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 2 2024-02-23T00:40:22.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6805/685354171 conn(0x7fcbb00f97f0 0x7fcbb00f9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0028540 osd=1 3 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0028540 osd=1 3 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0028540 osd=1 3 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 2 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb0028540 osd=1 3 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 1 2 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0028540 osd=1 4 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 2 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.037+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbb005a040 con 0x7fcbb00f97f0 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbb5362700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6805/685354171 conn(0x7fcbb00f97f0 0x7fcbb00f9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbb5b63700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcb98062c90 0x7fcb98065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbaeffd700 10 client.45397.objecter ms_handle_connect 0x7fcbb00f97f0 2024-02-23T00:40:22.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbb5362700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6805/685354171 conn(0x7fcbb00f97f0 0x7fcbb00f9bd0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:22.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbb5b63700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcb98062c90 0x7fcb98065150 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 rx=0x7fcba8009810 tx=0x7fcba800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:22.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbaeffd700 10 client.45397.objecter ms_handle_connect 0x7fcb98062c90 2024-02-23T00:40:22.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbb005a040 con 0x7fcbb00f97f0 2024-02-23T00:40:22.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:40:22.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 2 = system:0 2024-02-23T00:40:22.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb0028540 osd=1 4 2024-02-23T00:40:22.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.038+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 1 2 2024-02-23T00:40:22.044 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:22.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:22.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 3 2024-02-23T00:40:22.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0028540 osd=1 3 2024-02-23T00:40:22.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0028540 osd=1 3 2024-02-23T00:40:22.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0028540 osd=1 3 2024-02-23T00:40:22.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 3 2024-02-23T00:40:22.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb0028540 osd=1 3 2024-02-23T00:40:22.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 1 3 2024-02-23T00:40:22.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0028540 osd=1 4 2024-02-23T00:40:22.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 3 2024-02-23T00:40:22.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.060+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbb00fa0c0 con 0x7fcbb00f97f0 2024-02-23T00:40:22.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcbb00fa0c0 con 0x7fcbb00f97f0 2024-02-23T00:40:22.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:22.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb0028540 osd=1 4 2024-02-23T00:40:22.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 1 3 2024-02-23T00:40:22.063 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:40:22.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:22.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 4 2024-02-23T00:40:22.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcbb0059710 0x7fcbb011b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.064 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb00595a0 osd=2 3 2024-02-23T00:40:22.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb00595a0 osd=2 3 2024-02-23T00:40:22.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb00595a0 osd=2 3 2024-02-23T00:40:22.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 4 2024-02-23T00:40:22.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb00595a0 osd=2 3 2024-02-23T00:40:22.065 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 2 4 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb00595a0 osd=2 4 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 4 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbb00faf10 con 0x7fcbb0059710 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb5b63700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcbb0059710 0x7fcbb011b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbaeffd700 10 client.45397.objecter ms_handle_connect 0x7fcbb0059710 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.062+0000 7fcbb5b63700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6809/3605394228 conn(0x7fcbb0059710 0x7fcbb011b160 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.063+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbb00faf10 con 0x7fcbb0059710 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.063+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.063+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 4 = system:0 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.063+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb00595a0 osd=2 4 2024-02-23T00:40:22.066 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.063+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 2 4 2024-02-23T00:40:22.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 5 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb00595a0 osd=2 3 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb00595a0 osd=2 3 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb00595a0 osd=2 3 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 5 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb00595a0 osd=2 3 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 2 5 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb00595a0 osd=2 4 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 5 2024-02-23T00:40:22.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.079+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbb00fa470 con 0x7fcbb0059710 2024-02-23T00:40:22.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.080+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcbb00fa470 con 0x7fcbb0059710 2024-02-23T00:40:22.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.080+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:22.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.080+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.080+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb00595a0 osd=2 4 2024-02-23T00:40:22.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.080+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 2 5 2024-02-23T00:40:22.082 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:40:22.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:22.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 6 2024-02-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6813/128059790 conn(0x7fcbb00280a0 0x7fcbb011e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb00fa880 osd=3 3 2024-02-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb00fa880 osd=3 3 2024-02-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb00fa880 osd=3 3 2024-02-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 6 2024-02-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb00fa880 osd=3 3 2024-02-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 3 6 2024-02-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb00fa880 osd=3 4 2024-02-23T00:40:22.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 6 2024-02-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbb0121620 con 0x7fcbb00280a0 2024-02-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb5362700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6813/128059790 conn(0x7fcbb00280a0 0x7fcbb011e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbaeffd700 10 client.45397.objecter ms_handle_connect 0x7fcbb00280a0 2024-02-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.081+0000 7fcbb5362700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.87:6813/128059790 conn(0x7fcbb00280a0 0x7fcbb011e7e0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.082+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbb0121620 con 0x7fcbb00280a0 2024-02-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.082+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.082+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 6 = system:0 2024-02-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.082+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb00fa880 osd=3 4 2024-02-23T00:40:22.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.082+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 3 6 2024-02-23T00:40:22.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:22.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 7 2024-02-23T00:40:22.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb00fa880 osd=3 3 2024-02-23T00:40:22.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb00fa880 osd=3 3 2024-02-23T00:40:22.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb00fa880 osd=3 3 2024-02-23T00:40:22.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 7 2024-02-23T00:40:22.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb00fa880 osd=3 3 2024-02-23T00:40:22.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 3 7 2024-02-23T00:40:22.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb00fa880 osd=3 4 2024-02-23T00:40:22.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 7 2024-02-23T00:40:22.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.096+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbb00276c0 con 0x7fcbb00280a0 2024-02-23T00:40:22.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.097+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcbb00276c0 con 0x7fcbb00280a0 2024-02-23T00:40:22.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.097+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:22.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.097+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.097+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb00fa880 osd=3 4 2024-02-23T00:40:22.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.097+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 3 7 2024-02-23T00:40:22.102 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:40:22.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:22.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 8 2024-02-23T00:40:22.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcbb011edd0 0x7fcbb011f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb011be20 osd=4 3 2024-02-23T00:40:22.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb011be20 osd=4 3 2024-02-23T00:40:22.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb011be20 osd=4 3 2024-02-23T00:40:22.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 8 2024-02-23T00:40:22.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb011be20 osd=4 3 2024-02-23T00:40:22.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 4 8 2024-02-23T00:40:22.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb011be20 osd=4 4 2024-02-23T00:40:22.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 8 2024-02-23T00:40:22.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbb011c560 con 0x7fcbb011edd0 2024-02-23T00:40:22.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb6364700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcbb011edd0 0x7fcbb011f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbaeffd700 10 client.45397.objecter ms_handle_connect 0x7fcbb011edd0 2024-02-23T00:40:22.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.098+0000 7fcbb6364700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6800/4175844757 conn(0x7fcbb011edd0 0x7fcbb011f210 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:22.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.099+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbb011c560 con 0x7fcbb011edd0 2024-02-23T00:40:22.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.099+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:40:22.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.099+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 8 = system:0 2024-02-23T00:40:22.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.099+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb011be20 osd=4 4 2024-02-23T00:40:22.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.099+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 4 8 2024-02-23T00:40:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 9 2024-02-23T00:40:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb011be20 osd=4 3 2024-02-23T00:40:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb011be20 osd=4 3 2024-02-23T00:40:22.115 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb011be20 osd=4 3 2024-02-23T00:40:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 9 2024-02-23T00:40:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb011be20 osd=4 3 2024-02-23T00:40:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 4 9 2024-02-23T00:40:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb011be20 osd=4 4 2024-02-23T00:40:22.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 9 2024-02-23T00:40:22.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.113+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbb0121050 con 0x7fcbb011edd0 2024-02-23T00:40:22.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcbb0121050 con 0x7fcbb011edd0 2024-02-23T00:40:22.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:22.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb011be20 osd=4 4 2024-02-23T00:40:22.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 4 9 2024-02-23T00:40:22.119 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:40:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 10 2024-02-23T00:40:22.120 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcbb0123850 0x7fcbb0125c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0120880 osd=5 3 2024-02-23T00:40:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0120880 osd=5 3 2024-02-23T00:40:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0120880 osd=5 3 2024-02-23T00:40:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 10 2024-02-23T00:40:22.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb0120880 osd=5 3 2024-02-23T00:40:22.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 5 10 2024-02-23T00:40:22.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0120880 osd=5 4 2024-02-23T00:40:22.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 10 2024-02-23T00:40:22.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.115+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbb0120fc0 con 0x7fcbb0123850 2024-02-23T00:40:22.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.116+0000 7fcbb5b63700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcbb0123850 0x7fcbb0125c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.116+0000 7fcbb5b63700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6804/2550013795 conn(0x7fcbb0123850 0x7fcbb0125c40 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:22.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.116+0000 7fcbaeffd700 10 client.45397.objecter ms_handle_connect 0x7fcbb0123850 2024-02-23T00:40:22.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.117+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbb0120fc0 con 0x7fcbb0123850 2024-02-23T00:40:22.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.117+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:40:22.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.117+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 10 = system:0 2024-02-23T00:40:22.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.117+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb0120880 osd=5 4 2024-02-23T00:40:22.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.117+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 5 10 2024-02-23T00:40:22.130 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:40:22.130 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:40:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 11 2024-02-23T00:40:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0120880 osd=5 3 2024-02-23T00:40:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0120880 osd=5 3 2024-02-23T00:40:22.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0120880 osd=5 3 2024-02-23T00:40:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 11 2024-02-23T00:40:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb0120880 osd=5 3 2024-02-23T00:40:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 5 11 2024-02-23T00:40:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0120880 osd=5 4 2024-02-23T00:40:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 11 2024-02-23T00:40:22.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.131+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbb0120fc0 con 0x7fcbb0123850 2024-02-23T00:40:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcbb0120fc0 con 0x7fcbb0123850 2024-02-23T00:40:22.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:22.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb0120880 osd=5 4 2024-02-23T00:40:22.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 5 11 2024-02-23T00:40:22.137 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:40:22.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:22.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 12 2024-02-23T00:40:22.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcbb0126640 0x7fcbb0126ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0120fc0 osd=6 3 2024-02-23T00:40:22.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0120fc0 osd=6 3 2024-02-23T00:40:22.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0120fc0 osd=6 3 2024-02-23T00:40:22.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 12 2024-02-23T00:40:22.139 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb0120fc0 osd=6 3 2024-02-23T00:40:22.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 6 12 2024-02-23T00:40:22.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0120fc0 osd=6 4 2024-02-23T00:40:22.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 12 2024-02-23T00:40:22.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbb012ba80 con 0x7fcbb0126640 2024-02-23T00:40:22.140 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb5362700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcbb0126640 0x7fcbb0126ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbaeffd700 10 client.45397.objecter ms_handle_connect 0x7fcbb0126640 2024-02-23T00:40:22.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.133+0000 7fcbb5362700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6808/2509408498 conn(0x7fcbb0126640 0x7fcbb0126ad0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:22.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.134+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbb012ba80 con 0x7fcbb0126640 2024-02-23T00:40:22.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.134+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:40:22.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.134+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 12 = system:0 2024-02-23T00:40:22.141 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.134+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb0120fc0 osd=6 4 2024-02-23T00:40:22.142 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.134+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 6 12 2024-02-23T00:40:22.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 13 2024-02-23T00:40:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0120fc0 osd=6 3 2024-02-23T00:40:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0120fc0 osd=6 3 2024-02-23T00:40:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb0120fc0 osd=6 3 2024-02-23T00:40:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 13 2024-02-23T00:40:22.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb0120fc0 osd=6 3 2024-02-23T00:40:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 6 13 2024-02-23T00:40:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb0120fc0 osd=6 4 2024-02-23T00:40:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 13 2024-02-23T00:40:22.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.148+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbb01270e0 con 0x7fcbb0126640 2024-02-23T00:40:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcbb01270e0 con 0x7fcbb0126640 2024-02-23T00:40:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb0120fc0 osd=6 4 2024-02-23T00:40:22.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 6 13 2024-02-23T00:40:22.153 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:40:22.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:22.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 14 2024-02-23T00:40:22.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6812/373787000 conn(0x7fcbb0127760 0x7fcbb0127bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:22.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb01270e0 osd=7 3 2024-02-23T00:40:22.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb01270e0 osd=7 3 2024-02-23T00:40:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb01270e0 osd=7 3 2024-02-23T00:40:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 14 2024-02-23T00:40:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb01270e0 osd=7 3 2024-02-23T00:40:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 7 14 2024-02-23T00:40:22.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb01270e0 osd=7 4 2024-02-23T00:40:22.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 14 2024-02-23T00:40:22.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.150+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcbb01281b0 con 0x7fcbb0127760 2024-02-23T00:40:22.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.151+0000 7fcbb6364700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6812/373787000 conn(0x7fcbb0127760 0x7fcbb0127bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:22.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.151+0000 7fcbaeffd700 10 client.45397.objecter ms_handle_connect 0x7fcbb0127760 2024-02-23T00:40:22.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.151+0000 7fcbb6364700 1 --2- 172.21.15.87:0/4195866527 >> v2:172.21.15.171:6812/373787000 conn(0x7fcbb0127760 0x7fcbb0127bc0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:22.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.151+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fcbb01281b0 con 0x7fcbb0127760 2024-02-23T00:40:22.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.151+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:40:22.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.151+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 14 = system:0 2024-02-23T00:40:22.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.151+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb01270e0 osd=7 4 2024-02-23T00:40:22.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.151+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 7 14 2024-02-23T00:40:22.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 10 client.45397.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:22.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign -1 15 2024-02-23T00:40:22.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb01270e0 osd=7 3 2024-02-23T00:40:22.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb01270e0 osd=7 3 2024-02-23T00:40:22.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 20 client.45397.objecter _get_session s=0x7fcbb01270e0 osd=7 3 2024-02-23T00:40:22.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_remove -1 15 2024-02-23T00:40:22.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 20 client.45397.objecter get_session s=0x7fcbb01270e0 osd=7 3 2024-02-23T00:40:22.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 15 client.45397.objecter _session_command_op_assign 7 15 2024-02-23T00:40:22.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 20 client.45397.objecter put_session s=0x7fcbb01270e0 osd=7 4 2024-02-23T00:40:22.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 10 client.45397.objecter _send_command 15 2024-02-23T00:40:22.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.166+0000 7fcbb7dc7700 1 -- 172.21.15.87:0/4195866527 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fcbb01281b0 con 0x7fcbb0127760 2024-02-23T00:40:22.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.167+0000 7fcbaeffd700 1 -- 172.21.15.87:0/4195866527 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fcbb01281b0 con 0x7fcbb0127760 2024-02-23T00:40:22.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.167+0000 7fcbaeffd700 10 client.45397.objecter ms_dispatch 0x7fcbb0001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:22.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.167+0000 7fcbaeffd700 10 client.45397.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.167+0000 7fcbaeffd700 20 client.45397.objecter put_session s=0x7fcbb01270e0 osd=7 4 2024-02-23T00:40:22.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.167+0000 7fcbaeffd700 15 client.45397.objecter _session_command_op_remove 7 15 2024-02-23T00:40:22.171 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:40:22.171 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:40:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.172 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.173 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:40:22.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.180+0000 7f42f27fc700 1 -- 172.21.15.87:0/463382080 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["0"]}]=0 marked out osd.0. v603) v1 ==== 87+0+0 (secure 0 0 0) 0x7f42ec0502d0 con 0x7f42fc0f8420 2024-02-23T00:40:22.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.187+0000 7f43035ae700 10 librados: watch_flush enter 2024-02-23T00:40:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.188+0000 7f43035ae700 10 librados: watch_flush exit 2024-02-23T00:40:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.188+0000 7f43035ae700 20 client.38054.objecter shutdown clearing up homeless session... 2024-02-23T00:40:22.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.188+0000 7f43035ae700 10 client.38054.objecter successfully canceled tick 2024-02-23T00:40:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f42e8062c90 msgr2=0x7f42e8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 --2- 172.21.15.87:0/463382080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f42e8062c90 0x7f42e8065150 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 rx=0x7f42fc1098b0 tx=0x7f42f8009cb0).stop 2024-02-23T00:40:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 msgr2=0x7f42fc10df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 --2- 172.21.15.87:0/463382080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 0x7f42fc10df70 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 rx=0x7f42ec00e990 tx=0x7f42ec00b040).stop 2024-02-23T00:40:22.198 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 shutdown_connections 2024-02-23T00:40:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 --2- 172.21.15.87:0/463382080 >> v2:172.21.15.87:6800/1231597463 conn(0x7f42e8062c90 0x7f42e8065150 secure :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 rx=0x7f42fc1098b0 tx=0x7f42f8009cb0).stop 2024-02-23T00:40:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 --2- 172.21.15.87:0/463382080 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42fc0fe510 0x7f42fc108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 --2- 172.21.15.87:0/463382080 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42fc0f8420 0x7f42fc10df70 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 --2- 172.21.15.87:0/463382080 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42fc0fdbb0 0x7f42fc10e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:22.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.190+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 >> 172.21.15.87:0/463382080 conn(0x7f42fc0f3800 msgr2=0x7f42fc0f50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.199+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 shutdown_connections 2024-02-23T00:40:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.202+0000 7f43035ae700 1 -- 172.21.15.87:0/463382080 wait complete. 2024-02-23T00:40:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:22.203+0000 7f43035ae700 1 librados: shutdown 2024-02-23T00:40:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:marked out osd.0. 2024-02-23T00:40:22.234 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:22] "GET /metrics HTTP/1.1" 200 187132 "" "Prometheus/2.33.4" 2024-02-23T00:40:22.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:20.639417+0000 osd.7 (osd.7) 280 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:40:22.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:20.642332+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:40:22.235 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:21.018426+0000 osd.5 (osd.5) 233 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:40:22.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:21.019381+0000 osd.5 (osd.5) 234 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:40:22.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:21.104871+0000 mgr.y (mgr.24437) 538 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:22.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:21.182008+0000 osd.1 (osd.1) 279 : cluster [DBG] 2.2 deep-scrub starts 2024-02-23T00:40:22.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:21.186211+0000 osd.1 (osd.1) 280 : cluster [DBG] 2.2 deep-scrub ok 2024-02-23T00:40:22.236 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:21.193713+0000 mon.a (mon.0) 3226 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:22.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:21.193782+0000 mon.a (mon.0) 3227 : cluster [DBG] osdmap e602: 8 total, 7 up, 8 in 2024-02-23T00:40:22.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:21.194539+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102"}]: dispatch 2024-02-23T00:40:22.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:21.978944+0000 mon.c (mon.2) 474 : audit [INF] from='client.? 172.21.15.87:0/463382080' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-23T00:40:22.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:21.979287+0000 mon.a (mon.0) 3229 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-23T00:40:22.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:21.979361+0000 mon.a (mon.0) 3230 : cluster [INF] Client client.admin marked osd.0 out, after it was down for 225 seconds 2024-02-23T00:40:22.237 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:22.037261+0000 mon.b (mon.1) 456 : audit [DBG] from='client.? 172.21.15.87:0/4195866527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:22.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:22.168244+0000 mon.a (mon.0) 3231 : cluster [WRN] Health check update: Degraded data redundancy: 40/702 objects degraded (5.698%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:22.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:22.169025+0000 mon.a (mon.0) 3232 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-23T00:40:22.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:22.175849+0000 mon.a (mon.0) 3233 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102"}]': finished 2024-02-23T00:40:22.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:22.175919+0000 mon.a (mon.0) 3234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-23T00:40:22.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:22.175976+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e603: 8 total, 7 up, 7 in 2024-02-23T00:40:22.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:40:21.979+0000 7fe8e2097700 -1 mon.a@0(leader).osd e602 definitely_dead 0 2024-02-23T00:40:22.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:20.639417+0000 osd.7 (osd.7) 280 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:40:22.239 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:20.642332+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:40:22.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:21.018426+0000 osd.5 (osd.5) 233 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:40:22.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:21.019381+0000 osd.5 (osd.5) 234 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:40:22.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:21.104871+0000 mgr.y (mgr.24437) 538 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:22.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:21.182008+0000 osd.1 (osd.1) 279 : cluster [DBG] 2.2 deep-scrub starts 2024-02-23T00:40:22.240 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:21.186211+0000 osd.1 (osd.1) 280 : cluster [DBG] 2.2 deep-scrub ok 2024-02-23T00:40:22.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:21.193713+0000 mon.a (mon.0) 3226 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:22.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:21.193782+0000 mon.a (mon.0) 3227 : cluster [DBG] osdmap e602: 8 total, 7 up, 8 in 2024-02-23T00:40:22.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:21.194539+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102"}]: dispatch 2024-02-23T00:40:22.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:21.978944+0000 mon.c (mon.2) 474 : audit [INF] from='client.? 172.21.15.87:0/463382080' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-23T00:40:22.241 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:21.979287+0000 mon.a (mon.0) 3229 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-23T00:40:22.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:21.979361+0000 mon.a (mon.0) 3230 : cluster [INF] Client client.admin marked osd.0 out, after it was down for 225 seconds 2024-02-23T00:40:22.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:22.037261+0000 mon.b (mon.1) 456 : audit [DBG] from='client.? 172.21.15.87:0/4195866527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:22.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:22.168244+0000 mon.a (mon.0) 3231 : cluster [WRN] Health check update: Degraded data redundancy: 40/702 objects degraded (5.698%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:22.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:22.169025+0000 mon.a (mon.0) 3232 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-23T00:40:22.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:22.175849+0000 mon.a (mon.0) 3233 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102"}]': finished 2024-02-23T00:40:22.242 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:22.175919+0000 mon.a (mon.0) 3234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-23T00:40:22.243 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:22.175976+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e603: 8 total, 7 up, 7 in 2024-02-23T00:40:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:20.639417+0000 osd.7 (osd.7) 280 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:40:22.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:20.642332+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:21.018426+0000 osd.5 (osd.5) 233 : cluster [DBG] 6.1a scrub starts 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:21.019381+0000 osd.5 (osd.5) 234 : cluster [DBG] 6.1a scrub ok 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:21.104871+0000 mgr.y (mgr.24437) 538 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:21.182008+0000 osd.1 (osd.1) 279 : cluster [DBG] 2.2 deep-scrub starts 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:21.186211+0000 osd.1 (osd.1) 280 : cluster [DBG] 2.2 deep-scrub ok 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:21.193713+0000 mon.a (mon.0) 3226 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:21.193782+0000 mon.a (mon.0) 3227 : cluster [DBG] osdmap e602: 8 total, 7 up, 8 in 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:21.194539+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102"}]: dispatch 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:21.978944+0000 mon.c (mon.2) 474 : audit [INF] from='client.? 172.21.15.87:0/463382080' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:21.979287+0000 mon.a (mon.0) 3229 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:21.979361+0000 mon.a (mon.0) 3230 : cluster [INF] Client client.admin marked osd.0 out, after it was down for 225 seconds 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:22.037261+0000 mon.b (mon.1) 456 : audit [DBG] from='client.? 172.21.15.87:0/4195866527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:22.168244+0000 mon.a (mon.0) 3231 : cluster [WRN] Health check update: Degraded data redundancy: 40/702 objects degraded (5.698%), 10 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-02-23T00:40:22.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:22.169025+0000 mon.a (mon.0) 3232 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-23T00:40:22.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:22.175849+0000 mon.a (mon.0) 3233 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-102"}]': finished 2024-02-23T00:40:22.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:22.175919+0000 mon.a (mon.0) 3234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-23T00:40:22.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:22.175976+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e603: 8 total, 7 up, 7 in 2024-02-23T00:40:22.730 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:22.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:22.801 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-23T00:40:22.833 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:22.864 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:23.070 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:23.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:21.746845+0000 mgr.y (mgr.24437) 539 : cluster [DBG] pgmap v920: 218 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 43 active+undersized, 140 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 40/708 objects degraded (5.650%); 61/708 objects misplaced (8.616%) 2024-02-23T00:40:23.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:22.047987+0000 osd.5 (osd.5) 235 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:40:23.551 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:22.049072+0000 osd.5 (osd.5) 236 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:40:23.551 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:22.190109+0000 mon.a (mon.0) 3236 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:23.551 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:22.199946+0000 osd.2 (osd.2) 221 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:40:23.551 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:22.203037+0000 osd.2 (osd.2) 222 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:40:23.551 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:23.178861+0000 mon.a (mon.0) 3237 : cluster [DBG] osdmap e604: 8 total, 7 up, 7 in 2024-02-23T00:40:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:21.746845+0000 mgr.y (mgr.24437) 539 : cluster [DBG] pgmap v920: 218 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 43 active+undersized, 140 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 40/708 objects degraded (5.650%); 61/708 objects misplaced (8.616%) 2024-02-23T00:40:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:22.047987+0000 osd.5 (osd.5) 235 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:40:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:22.049072+0000 osd.5 (osd.5) 236 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:40:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:22.190109+0000 mon.a (mon.0) 3236 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:22.199946+0000 osd.2 (osd.2) 221 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:40:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:22.203037+0000 osd.2 (osd.2) 222 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:40:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:23.178861+0000 mon.a (mon.0) 3237 : cluster [DBG] osdmap e604: 8 total, 7 up, 7 in 2024-02-23T00:40:23.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:21.746845+0000 mgr.y (mgr.24437) 539 : cluster [DBG] pgmap v920: 218 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 43 active+undersized, 140 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 40/708 objects degraded (5.650%); 61/708 objects misplaced (8.616%) 2024-02-23T00:40:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:22.047987+0000 osd.5 (osd.5) 235 : cluster [DBG] 6.17 scrub starts 2024-02-23T00:40:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:22.049072+0000 osd.5 (osd.5) 236 : cluster [DBG] 6.17 scrub ok 2024-02-23T00:40:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:22.190109+0000 mon.a (mon.0) 3236 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:40:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:22.199946+0000 osd.2 (osd.2) 221 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:40:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:22.203037+0000 osd.2 (osd.2) 222 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:40:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:23.178861+0000 mon.a (mon.0) 3237 : cluster [DBG] osdmap e604: 8 total, 7 up, 7 in 2024-02-23T00:40:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:40:23.847 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:40:23.940 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:40:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:40:24.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:23.183847+0000 osd.3 (osd.3) 191 : cluster [DBG] 4.1a starting backfill to osd.2 from (0'0,0'0] MAX to 476'63 2024-02-23T00:40:24.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:23.187199+0000 osd.3 (osd.3) 192 : cluster [DBG] 4.16 starting backfill to osd.2 from (0'0,0'0] MAX to 453'39 2024-02-23T00:40:24.487 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:24.177776+0000 mon.a (mon.0) 3238 : cluster [DBG] osdmap e605: 8 total, 7 up, 7 in 2024-02-23T00:40:24.663 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:23.183847+0000 osd.3 (osd.3) 191 : cluster [DBG] 4.1a starting backfill to osd.2 from (0'0,0'0] MAX to 476'63 2024-02-23T00:40:24.663 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:23.187199+0000 osd.3 (osd.3) 192 : cluster [DBG] 4.16 starting backfill to osd.2 from (0'0,0'0] MAX to 453'39 2024-02-23T00:40:24.663 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:24.177776+0000 mon.a (mon.0) 3238 : cluster [DBG] osdmap e605: 8 total, 7 up, 7 in 2024-02-23T00:40:24.663 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:23.183847+0000 osd.3 (osd.3) 191 : cluster [DBG] 4.1a starting backfill to osd.2 from (0'0,0'0] MAX to 476'63 2024-02-23T00:40:24.663 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:23.187199+0000 osd.3 (osd.3) 192 : cluster [DBG] 4.16 starting backfill to osd.2 from (0'0,0'0] MAX to 453'39 2024-02-23T00:40:24.664 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:24.177776+0000 mon.a (mon.0) 3238 : cluster [DBG] osdmap e605: 8 total, 7 up, 7 in 2024-02-23T00:40:24.766 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:40:24.766 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.25166.0:2 59.6 59:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:02.557319+0000", 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 562.20531481499995, 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056150042999999997, 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.767 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.25166", 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/458030738", 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557319+0000", 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.768 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557319+0000", 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557318+0000", 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8050000000000003e-06 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.769 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557324+0000", 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8560000000000007e-06 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557331+0000", 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.4490000000000002e-06 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557339+0000", 2024-02-23T00:40:24.770 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7904000000000002e-05 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557377+0000", 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127668 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557504+0000", 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000131311 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.771 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,6", 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557636+0000", 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033497099999999999 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.557971+0000", 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000210115 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.772 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.558181+0000", 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.055228405000000001 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.613409+0000", 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4125000000000003e-05 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.773 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.613443+0000", 2024-02-23T00:40:24.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5678999999999999e-05 2024-02-23T00:40:24.774 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.774 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.774 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.774 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:02.613469+0000", 2024-02-23T00:40:24.774 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.056150042999999997 2024-02-23T00:40:24.774 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.774 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:8 56.2 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:40:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.569023+0000", 2024-02-23T00:40:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 561.19361032799998, 2024-02-23T00:40:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.061982072999999999, 2024-02-23T00:40:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.775 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.776 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.776 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:40:24.776 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:40:24.776 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 8 2024-02-23T00:40:24.776 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.776 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.776 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.776 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.776 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569023+0000", 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569023+0000", 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569022+0000", 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9890000000000003e-06 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.777 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569029+0000", 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5150000000000006e-06 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569037+0000", 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0450000000000003e-06 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569045+0000", 2024-02-23T00:40:24.778 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033288000000000002 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569377+0000", 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000205007 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.569582+0000", 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070529399999999997 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.779 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.576635+0000", 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8623999999999996e-05 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.576734+0000", 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000176076 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.576910+0000", 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00079704099999999996 2024-02-23T00:40:24.780 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.577707+0000", 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9565e-05 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.577807+0000", 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.053133055999999998 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.630940+0000", 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2215999999999998e-05 2024-02-23T00:40:24.781 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.630972+0000", 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3599000000000001e-05 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631006+0000", 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.061982072999999999 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.782 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:9 56.2 56:74c9cfff:test-rados-api-smithi087-82147-7::foo:head [watch watch cookie 93958402142112] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.631729+0000", 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 561.13090452400002, 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018840933000000001, 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 9 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.783 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631729+0000", 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631729+0000", 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631728+0000", 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8719999999999999e-06 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.784 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631734+0000", 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1409999999999994e-06 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631743+0000", 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6979999999999997e-06 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631748+0000", 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.5034999999999999e-05 2024-02-23T00:40:24.785 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631783+0000", 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5764000000000002e-05 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631858+0000", 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000104113 2024-02-23T00:40:24.786 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.631962+0000", 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00027199 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.632234+0000", 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00017308100000000001 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.632408+0000", 2024-02-23T00:40:24.787 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018079319999999999 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.650487+0000", 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0031000000000001e-05 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.650517+0000", 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3294000000000002e-05 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.788 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.650570+0000", 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.018840933000000001 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:14 56.1 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.661664+0000", 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 561.10096996899995, 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044383775, 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:40:24.789 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 14 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661664+0000", 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661664+0000", 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.790 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661663+0000", 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3850000000000001e-06 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661668+0000", 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5259999999999999e-06 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661678+0000", 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7389999999999999e-06 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.791 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661683+0000", 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7559000000000002e-05 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661710+0000", 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000168534 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.661879+0000", 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014226162000000001 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.792 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676105+0000", 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4277000000000006e-05 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676189+0000", 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000100592 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676290+0000", 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00034506400000000002 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.793 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.676635+0000", 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000797236 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.677432+0000", 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028506070000000001 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.705938+0000", 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3522000000000006e-05 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.794 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.706002+0000", 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5997e-05 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.706048+0000", 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044383775 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.795 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:16 56.1 56:b6b94b45:test-rados-api-smithi087-82147-8::foo:head [watch unwatch cookie 93958402138000] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.714409+0000", 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 561.04822493699999, 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021332225999999999, 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 16 2024-02-23T00:40:24.796 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714409+0000", 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714409+0000", 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.797 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714407+0000", 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.015e-06 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714413+0000", 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7249999999999997e-06 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714421+0000", 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1869999999999999e-06 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.798 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714425+0000", 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2351000000000002e-05 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714468+0000", 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012891899999999999 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714597+0000", 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00019076400000000001 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.799 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.714787+0000", 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00041226000000000001 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.715200+0000", 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001342099 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.716542+0000", 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019100005 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.800 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.735642+0000", 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1798999999999997e-05 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.735683+0000", 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7598e-05 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.735741+0000", 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021332225999999999 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.801 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:37 56.1 56:972518f1:test-rados-api-smithi087-82147-12::foo:head [watch unwatch cookie 93958402138000] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 560.95942336500002, 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019757686999999999, 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 37 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.802 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.803 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803210+0000", 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2400000000000001e-06 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803214+0000", 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9640000000000003e-06 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.804 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803222+0000", 2024-02-23T00:40:24.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7230000000000002e-06 2024-02-23T00:40:24.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.805 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803226+0000", 2024-02-23T00:40:24.805 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0100999999999999e-05 2024-02-23T00:40:24.805 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.805 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.805 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803256+0000", 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8650999999999998e-05 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803314+0000", 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0869e-05 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803405+0000", 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00032610700000000001 2024-02-23T00:40:24.806 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.803731+0000", 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.012946180999999999 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.816678+0000", 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.006224963 2024-02-23T00:40:24.807 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.822902+0000", 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.8585000000000002e-05 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.822931+0000", 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7018999999999999e-05 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:03.822968+0000", 2024-02-23T00:40:24.808 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019757686999999999 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:67 56.0 56:2858fd9f:test-rados-api-smithi087-82147-17::foo:head [watch unwatch cookie 93958402367200] snapc 0=[] ondisk+write+known_if_redirected e117)", 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:07.920664+0000", 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 556.84196956699998, 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073703837999999994, 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:40:24.809 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 67 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920664+0000", 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920664+0000", 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.810 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920663+0000", 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8689999999999998e-06 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920669+0000", 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1250000000000001e-05 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.811 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920681+0000", 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8099999999999998e-06 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920688+0000", 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9345000000000003e-05 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920738+0000", 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000109922 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.812 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.920848+0000", 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00016524700000000001 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,6", 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.921013+0000", 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00089610999999999998 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.921909+0000", 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125143 2024-02-23T00:40:24.813 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.922034+0000", 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.072257924000000001 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.994292+0000", 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4425000000000003e-05 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.814 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.994327+0000", 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1483000000000002e-05 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:07.994368+0000", 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.073703837999999994 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.815 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:69 56.1 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e118)", 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:09.916487+0000", 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 554.84614689099999, 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062093785999999998, 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 69 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.816 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916487+0000", 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916487+0000", 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.817 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916486+0000", 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7636000000000001e-05 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916514+0000", 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0845999999999999e-05 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916524+0000", 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1185000000000001e-05 2024-02-23T00:40:24.818 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916546+0000", 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3442999999999998e-05 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916579+0000", 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000177116 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.819 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.916756+0000", 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077845400000000004 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.917535+0000", 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8349999999999994e-05 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.820 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.917603+0000", 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108934 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.917712+0000", 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00030745799999999998 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.918019+0000", 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032894684 2024-02-23T00:40:24.821 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.950914+0000", 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.027609002000000001 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.978523+0000", 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2388999999999998e-05 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.822 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.978555+0000", 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5239e-05 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:09.978581+0000", 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.062093785999999998 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.823 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:75 56.1 56:9b9f7999:test-rados-api-smithi087-82147-18::foo:head [watch unwatch cookie 93958402367200] snapc 0=[] ondisk+write+known_if_redirected e122)", 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:12.996174+0000", 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 551.766459779, 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.084863057000000006, 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 75 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.824 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996174+0000", 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996174+0000", 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996173+0000", 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1340000000000003e-06 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.825 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996178+0000", 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0209e-05 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996188+0000", 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9609999999999996e-06 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996193+0000", 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0006999999999999e-05 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.826 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996223+0000", 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0087000000000007e-05 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996313+0000", 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000276903 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.996590+0000", 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00053075400000000003 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.827 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:12.997121+0000", 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.070072748000000004 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.067194+0000", 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.01373247 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.080926+0000", 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.193e-05 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.828 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.080968+0000", 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8838999999999996e-05 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081037+0000", 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.084863057000000006 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.15147.0:76 56.1 56:a5875266:test-rados-api-smithi087-82147-19::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e122)", 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:13.081838+0000", 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 551.68079547699995, 2024-02-23T00:40:24.829 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028322508, 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.15147", 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2103098061", 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 76 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081838+0000", 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081838+0000", 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081837+0000", 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2859999999999996e-06 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081843+0000", 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0053e-05 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.831 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081853+0000", 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1229999999999999e-06 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081858+0000", 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0298999999999999e-05 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.081889+0000", 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015674600000000001 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:40:24.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.082046+0000", 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011441438 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093487+0000", 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6672000000000002e-05 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093544+0000", 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000108148 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,3", 2024-02-23T00:40:24.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093652+0000", 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00028719800000000001 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.093939+0000", 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0051637150000000001 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.099103+0000", 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011005768000000001 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.110108+0000", 2024-02-23T00:40:24.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0571999999999998e-05 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.110139+0000", 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1756999999999999e-05 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:13.110161+0000", 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028322508 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24413.0:238 3.c 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e137)", 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:28.139198+0000", 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 536.62343541099995, 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94992057799999996, 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24413", 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.171:0/2453883515", 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 238 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139198+0000", 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139198+0000", 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139197+0000", 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0800000000000002e-06 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139203+0000", 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1200000000000008e-06 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139212+0000", 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.344e-06 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139217+0000", 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9641000000000002e-05 2024-02-23T00:40:24.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139257+0000", 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1140000000000002e-06 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:28.139263+0000", 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.93888930100000001 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.078152+0000", 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013934400000000001 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for missing object", 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.078292+0000", 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010582111999999999 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.088874+0000", 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7443999999999997e-05 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.088971+0000", 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000147868 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:29.089119+0000", 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.94992057799999996 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:578 63.a 63:54d171db:test-rados-api-smithi087-82091-14::foo7:head [writefull 0~11] snapc 5=[5,4] ondisk+write+known_if_redirected e146)", 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 527.74005737699997, 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044787916999999997, 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 578 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022576+0000", 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3959999999999997e-06 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022581+0000", 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8709999999999997e-06 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022589+0000", 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2200000000000003e-06 2024-02-23T00:40:24.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022593+0000", 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4632000000000002e-05 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022638+0000", 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012398899999999999 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.022762+0000", 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00023933200000000001 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,4", 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.023001+0000", 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00054081799999999998 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.023542+0000", 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011881500000000001 2024-02-23T00:40:24.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.023661+0000", 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.043632126 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.067293+0000", 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3884999999999997e-05 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.847 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.067337+0000", 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.7705e-05 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:37.067364+0000", 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.044787916999999997 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36842.0:1 112.3s0 112:d6e58105:test-rados-api-smithi087-82115-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:57.017766+0000", 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 507.74486762599997, 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.01719962, 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36842", 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3011193832", 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017766+0000", 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017766+0000", 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017764+0000", 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5610000000000004e-06 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017774+0000", 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1437000000000001e-05 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017785+0000", 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.6439999999999995e-06 2024-02-23T00:40:24.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.017795+0000", 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.020082722000000001 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.037878+0000", 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.3429999999999998e-06 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:57.037882+0000", 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.99038801300000001 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.028270+0000", 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000113621 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.028384+0000", 2024-02-23T00:40:24.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00034479999999999998 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.028728+0000", 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011224329999999999 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.029851+0000", 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0050385860000000003 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.854 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.034889+0000", 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6290000000000003e-05 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:58.034966+0000", 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.01719962 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.855 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:2079 63.3 63:c0773ebc:test-rados-api-smithi087-82091-26::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e234)", 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:05.027503+0000", 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 439.735130997, 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019673782000000001, 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.856 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2079 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027503+0000", 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027503+0000", 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027501+0000", 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0348e-05 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027511+0000", 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3602e-05 2024-02-23T00:40:24.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027525+0000", 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7789999999999996e-06 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027533+0000", 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0073932110000000002 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.034926+0000", 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014632299999999999 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.035072+0000", 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000197312 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,4", 2024-02-23T00:40:24.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.035269+0000", 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040784150000000002 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.039348+0000", 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035992540000000001 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.042947+0000", 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041140780000000002 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.047061+0000", 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1672999999999998e-05 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.047113+0000", 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3707999999999994e-05 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.047177+0000", 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019673782000000001 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:5418 63.1b 63:d8ef31d3:test-rados-api-smithi087-82091-50::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:35:23.068992+0000", 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 301.69364131399999, 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033546511000000001, 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 5418 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.068992+0000", 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.068992+0000", 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.068990+0000", 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0651e-05 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069001+0000", 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4694e-05 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069015+0000", 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1549999999999996e-06 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069025+0000", 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4701999999999999e-05 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069079+0000", 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0494e-05 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069090+0000", 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035708100000000002 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.072661+0000", 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014544499999999999 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.072806+0000", 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018308800000000001 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.072989+0000", 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036468820000000002 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.076636+0000", 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024542562 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.101179+0000", 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00126581 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102444+0000", 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5225000000000001e-05 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102500+0000", 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9344999999999997e-05 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.868 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102539+0000", 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033546511000000001 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6302 63.17 63:e8b72b4a:test-rados-api-smithi087-82091-64::foo:head [write 0~8] snapc 1d=[1d,1c] ondisk+write+known_if_redirected e432)", 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:36:44.294999+0000", 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 220.46763460599999, 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023072202, 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:24.869 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6302 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.294999+0000", 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.294999+0000", 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999971 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.294996+0000", 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4647000000000001e-05 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295011+0000", 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1302e-05 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295032+0000", 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0713999999999999e-05 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295043+0000", 2024-02-23T00:40:24.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7822999999999998e-05 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295101+0000", 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4919999999999993e-06 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295109+0000", 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0058836230000000002 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.300993+0000", 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110759 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.301104+0000", 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00020068000000000001 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.301305+0000", 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00069117699999999996 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.301996+0000", 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00045108700000000003 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.302447+0000", 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015506539999999999 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.317953+0000", 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7464000000000003e-05 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.318021+0000", 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0603000000000003e-05 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.318071+0000", 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023072202 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6304 63.17 63:e8b72b4a:test-rados-api-smithi087-82091-64::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e432)", 2024-02-23T00:40:24.875 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:36:44.326711+0000", 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 220.43592266900001, 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024147567000000002, 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6304 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.876 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326711+0000", 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326711+0000", 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326710+0000", 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2559999999999996e-06 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326717+0000", 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3875e-05 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326731+0000", 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9949999999999999e-06 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326738+0000", 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7929000000000001e-05 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326786+0000", 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2782999999999996e-05 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326879+0000", 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076338630000000003 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.334513+0000", 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011122320000000001 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.335625+0000", 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0062604449999999999 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.341885+0000", 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088814819999999996 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.350767+0000", 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0455999999999999e-05 2024-02-23T00:40:24.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.350817+0000", 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1442000000000001e-05 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.350859+0000", 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024147567000000002 2024-02-23T00:40:24.881 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6305 63.17 63:e8b72b4a:test-rados-api-smithi087-82091-64::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e432)", 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:36:44.361680+0000", 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 220.400954187, 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026227607999999999, 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.882 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6305 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361680+0000", 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.883 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361680+0000", 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361678+0000", 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2040000000000003e-06 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361687+0000", 2024-02-23T00:40:24.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4222999999999999e-05 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361701+0000", 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2899999999999997e-06 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361709+0000", 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7270000000000001e-05 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361746+0000", 2024-02-23T00:40:24.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9739000000000002e-05 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361846+0000", 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.02429605 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.386142+0000", 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00122338 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387365+0000", 2024-02-23T00:40:24.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033618999999999999 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387701+0000", 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000123817 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387825+0000", 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4799000000000003e-05 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387870+0000", 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7274000000000003e-05 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387907+0000", 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026227607999999999 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24413.0:676 3.c 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e509)", 2024-02-23T00:40:24.888 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:48.490504+0000", 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 96.272129376999999, 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.69719402699999999, 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24413", 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.171:0/2453883515", 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 676 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490504+0000", 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490504+0000", 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490502+0000", 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0909999999999999e-06 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490511+0000", 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.396e-05 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490525+0000", 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3749999999999992e-06 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490535+0000", 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6785000000000004e-05 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490611+0000", 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8079999999999999e-06 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:40:24.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490621+0000", 2024-02-23T00:40:24.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.69679959300000005 2024-02-23T00:40:24.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:49.187421+0000", 2024-02-23T00:40:24.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013327599999999999 2024-02-23T00:40:24.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:49.187554+0000", 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000144265 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:49.187698+0000", 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.69719402699999999 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37979.0:1 229.2s0 229:48558768:test-rados-api-smithi087-82091-95::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected e561)", 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:39:40.184380+0000", 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 44.578253725000003, 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0235315789999999, 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37979", 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/403995334", 2024-02-23T00:40:24.895 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184380+0000", 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184380+0000", 2024-02-23T00:40:24.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184379+0000", 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8969999999999999e-06 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184384+0000", 2024-02-23T00:40:24.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2439999999999993e-06 2024-02-23T00:40:24.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:24.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184392+0000", 2024-02-23T00:40:24.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8159999999999999e-06 2024-02-23T00:40:24.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:24.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184398+0000", 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4269e-05 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184432+0000", 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5000000000000002e-06 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184435+0000", 2024-02-23T00:40:24.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.004191152 2024-02-23T00:40:24.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:24.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:41.188626+0000", 2024-02-23T00:40:24.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012520899999999999 2024-02-23T00:40:24.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:24.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:41.188751+0000", 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019160613999999999 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:41.207912+0000", 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0235315789999999 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:24.901 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:40:25.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:40:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:40:24.661+0000 7f6e819c8700 -1 mgr get_metadata_python Requested missing service osd.0 2024-02-23T00:40:25.414 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:25.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:25.517 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:25.548 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:25.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.047+0000 7f3d46f99700 1 -- 172.21.15.87:0/1886395733 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400f8420 msgr2=0x7f3d40104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:25.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.047+0000 7f3d46f99700 1 --2- 172.21.15.87:0/1886395733 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400f8420 0x7f3d40104a20 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 rx=0x7f3d300047b0 tx=0x7f3d30009cb0).stop 2024-02-23T00:40:25.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.047+0000 7f3d46f99700 1 -- 172.21.15.87:0/1886395733 shutdown_connections 2024-02-23T00:40:25.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.047+0000 7f3d46f99700 1 --2- 172.21.15.87:0/1886395733 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400f8420 0x7f3d40104a20 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:25.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.047+0000 7f3d46f99700 1 --2- 172.21.15.87:0/1886395733 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400fdbb0 0x7f3d400fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:25.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.047+0000 7f3d46f99700 1 --2- 172.21.15.87:0/1886395733 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fe510 0x7f3d400f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:25.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.047+0000 7f3d46f99700 1 -- 172.21.15.87:0/1886395733 >> 172.21.15.87:0/1886395733 conn(0x7f3d400f3800 msgr2=0x7f3d400f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:25.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.047+0000 7f3d46f99700 1 -- 172.21.15.87:0/1886395733 shutdown_connections 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.047+0000 7f3d46f99700 1 -- 172.21.15.87:0/1886395733 wait complete. 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 librados: starting msgr at 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 librados: starting objecter 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 Processor -- start 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 -- start start 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 librados: setting wanted keys 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 librados: calling monclient init 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400f8420 0x7f3d4010de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40107fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d401084c0 con 0x7f3d400fdbb0 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d40108630 con 0x7f3d400f8420 2024-02-23T00:40:25.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d46f99700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d401087f0 con 0x7f3d400fe510 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d45536700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40107fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d3ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d45536700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40107fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:54040/0 (socket says 172.21.15.87:54040) 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d45536700 1 -- 172.21.15.87:0/2447939284 learned_addr learned my addr 172.21.15.87:0/2447939284 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.048+0000 7f3d3ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58282/0 (socket says 172.21.15.87:58282) 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d44d35700 1 --2- 172.21.15.87:0/2447939284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400f8420 0x7f3d4010de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d45536700 1 -- 172.21.15.87:0/2447939284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400f8420 msgr2=0x7f3d4010de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d45536700 1 --2- 172.21.15.87:0/2447939284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400f8420 0x7f3d4010de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d45536700 1 -- 172.21.15.87:0/2447939284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 msgr2=0x7f3d4010e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d45536700 1 --2- 172.21.15.87:0/2447939284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d45536700 1 -- 172.21.15.87:0/2447939284 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d28005040 con 0x7f3d400fe510 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d3ffff700 1 --2- 172.21.15.87:0/2447939284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d45536700 1 --2- 172.21.15.87:0/2447939284 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40107fd0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f3d300044e0 tx=0x7f3d30007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:25.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d3dffb700 10 client.?.objecter ms_handle_connect 0x7f3d400fe510 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d3dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d3dffb700 1 -- 172.21.15.87:0/2447939284 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30018070 con 0x7f3d400fe510 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d30005040 con 0x7f3d400fe510 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d3dffb700 1 -- 172.21.15.87:0/2447939284 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d30013ce0 con 0x7f3d400fe510 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d3dffb700 1 -- 172.21.15.87:0/2447939284 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3001c660 con 0x7f3d400fe510 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d46f99700 10 client.38066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d40108d10 con 0x7f3d400fe510 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.049+0000 7f3d46f99700 1 librados: init done 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.050+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4005ee80 con 0x7f3d400fe510 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.050+0000 7f3d3dffb700 1 -- 172.21.15.87:0/2447939284 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3d3001d470 con 0x7f3d400fe510 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.051+0000 7f3d3dffb700 1 --2- 172.21.15.87:0/2447939284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.051+0000 7f3d3dffb700 1 -- 172.21.15.87:0/2447939284 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(605..605 src has 173..605) v4 ==== 8222+0+0 (secure 0 0 0) 0x7f3d3007e370 con 0x7f3d400fe510 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.051+0000 7f3d3dffb700 10 client.38066.objecter ms_dispatch 0x7f3d40001550 osd_map(605..605 src has 173..605) v4 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.051+0000 7f3d3dffb700 3 client.38066.objecter handle_osd_map got epochs [605,605] > 0 2024-02-23T00:40:25.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.051+0000 7f3d3dffb700 3 client.38066.objecter handle_osd_map decoding full epoch 605 2024-02-23T00:40:25.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.051+0000 7f3d44d35700 1 --2- 172.21.15.87:0/2447939284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:25.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.051+0000 7f3d3dffb700 20 client.38066.objecter dump_active .. 0 homeless 2024-02-23T00:40:25.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.051+0000 7f3d3dffb700 10 client.38066.objecter ms_handle_connect 0x7f3d2c062c90 2024-02-23T00:40:25.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.051+0000 7f3d44d35700 1 --2- 172.21.15.87:0/2447939284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 rx=0x7f3d38007d70 tx=0x7f3d38004bb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:25.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.055+0000 7f3d3dffb700 1 -- 172.21.15.87:0/2447939284 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3d300517a0 con 0x7f3d400fe510 2024-02-23T00:40:25.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:25.262+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f3d40109550 con 0x7f3d400fe510 2024-02-23T00:40:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:23.747395+0000 mgr.y (mgr.24437) 540 : cluster [DBG] pgmap v923: 218 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 43 active+undersized, 140 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 40/708 objects degraded (5.650%); 61/708 objects misplaced (8.616%) 2024-02-23T00:40:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:24.179198+0000 osd.1 (osd.1) 281 : cluster [DBG] 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 453'39 2024-02-23T00:40:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:24.179780+0000 osd.1 (osd.1) 282 : cluster [DBG] 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 453'29 2024-02-23T00:40:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:24.182633+0000 osd.1 (osd.1) 283 : cluster [DBG] 4.b starting backfill to osd.3 from (0'0,0'0] MAX to 453'22 2024-02-23T00:40:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:24.655098+0000 mon.a (mon.0) 3239 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:25.179101+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e606: 8 total, 7 up, 7 in 2024-02-23T00:40:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:25.186789+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:25.263110+0000 mon.c (mon.2) 475 : audit [INF] from='client.? 172.21.15.87:0/2447939284' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:25.263427+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:23.747395+0000 mgr.y (mgr.24437) 540 : cluster [DBG] pgmap v923: 218 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 43 active+undersized, 140 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 40/708 objects degraded (5.650%); 61/708 objects misplaced (8.616%) 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:24.179198+0000 osd.1 (osd.1) 281 : cluster [DBG] 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 453'39 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:24.179780+0000 osd.1 (osd.1) 282 : cluster [DBG] 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 453'29 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:24.182633+0000 osd.1 (osd.1) 283 : cluster [DBG] 4.b starting backfill to osd.3 from (0'0,0'0] MAX to 453'22 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:24.655098+0000 mon.a (mon.0) 3239 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:25.179101+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e606: 8 total, 7 up, 7 in 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:25.186789+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:25.263110+0000 mon.c (mon.2) 475 : audit [INF] from='client.? 172.21.15.87:0/2447939284' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:40:25.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:25.263427+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:40:25.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:40:25.912 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:23.747395+0000 mgr.y (mgr.24437) 540 : cluster [DBG] pgmap v923: 218 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 24 active+clean+remapped, 9 active+undersized+degraded, 43 active+undersized, 140 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s; 40/708 objects degraded (5.650%); 61/708 objects misplaced (8.616%) 2024-02-23T00:40:25.912 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:24.179198+0000 osd.1 (osd.1) 281 : cluster [DBG] 4.6 starting backfill to osd.7 from (0'0,0'0] MAX to 453'39 2024-02-23T00:40:25.912 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:24.179780+0000 osd.1 (osd.1) 282 : cluster [DBG] 4.7 starting backfill to osd.7 from (0'0,0'0] MAX to 453'29 2024-02-23T00:40:25.912 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:24.182633+0000 osd.1 (osd.1) 283 : cluster [DBG] 4.b starting backfill to osd.3 from (0'0,0'0] MAX to 453'22 2024-02-23T00:40:25.912 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:24.655098+0000 mon.a (mon.0) 3239 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:25.913 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:25.179101+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e606: 8 total, 7 up, 7 in 2024-02-23T00:40:25.913 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:25.186789+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:25.913 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:25.263110+0000 mon.c (mon.2) 475 : audit [INF] from='client.? 172.21.15.87:0/2447939284' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:40:25.913 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:25.263427+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:40:25.933 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-23T00:40:25.997 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:26.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:26.101 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:26.112 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:26.128 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:26.147 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:26.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:26.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.182+0000 7f3d3dffb700 1 -- 172.21.15.87:0/2447939284 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v607) v1 ==== 87+0+0 (secure 0 0 0) 0x7f3d30022030 con 0x7f3d400fe510 2024-02-23T00:40:26.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.185+0000 7f3d46f99700 10 librados: watch_flush enter 2024-02-23T00:40:26.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.185+0000 7f3d46f99700 10 librados: watch_flush exit 2024-02-23T00:40:26.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.185+0000 7f3d46f99700 20 client.38066.objecter shutdown clearing up homeless session... 2024-02-23T00:40:26.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.185+0000 7f3d46f99700 10 client.38066.objecter successfully canceled tick 2024-02-23T00:40:26.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.191+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 msgr2=0x7f3d2c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:26.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.191+0000 7f3d46f99700 1 --2- 172.21.15.87:0/2447939284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 rx=0x7f3d38007d70 tx=0x7f3d38004bb0).stop 2024-02-23T00:40:26.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.191+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 msgr2=0x7f3d40107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:26.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.191+0000 7f3d46f99700 1 --2- 172.21.15.87:0/2447939284 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40107fd0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 rx=0x7f3d300044e0 tx=0x7f3d30007f90).stop 2024-02-23T00:40:26.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.193+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 shutdown_connections 2024-02-23T00:40:26.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.193+0000 7f3d46f99700 1 --2- 172.21.15.87:0/2447939284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:26.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.193+0000 7f3d46f99700 1 --2- 172.21.15.87:0/2447939284 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40107fd0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:26.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.193+0000 7f3d46f99700 1 --2- 172.21.15.87:0/2447939284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400f8420 0x7f3d4010de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:26.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.193+0000 7f3d46f99700 1 --2- 172.21.15.87:0/2447939284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:26.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.193+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 >> 172.21.15.87:0/2447939284 conn(0x7f3d400f3800 msgr2=0x7f3d400f5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:26.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.193+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 shutdown_connections 2024-02-23T00:40:26.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.193+0000 7f3d46f99700 1 -- 172.21.15.87:0/2447939284 wait complete. 2024-02-23T00:40:26.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:26.193+0000 7f3d46f99700 1 librados: shutdown 2024-02-23T00:40:26.195 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:40:26.251 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:26.262 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:26.277 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:25.148825+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.b scrub starts 2024-02-23T00:40:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:25.149868+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.b scrub ok 2024-02-23T00:40:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:25.448997+0000 osd.6 (osd.6) 242 : cluster [DBG] 5.2 deep-scrub starts 2024-02-23T00:40:26.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:25.452065+0000 osd.6 (osd.6) 243 : cluster [DBG] 5.2 deep-scrub ok 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:25.631710+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:25.632720+0000 osd.7 (osd.7) 283 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:25.748250+0000 mgr.y (mgr.24437) 541 : cluster [DBG] pgmap v926: 218 pgs: 3 peering, 2 active+remapped+backfill_toofull, 32 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 180 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 5/705 objects degraded (0.709%); 25/705 objects misplaced (3.546%); 461 B/s, 10 objects/s recovering 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:25.749241+0000 mon.a (mon.0) 3243 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "40"}]: dispatch 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:25.749432+0000 mon.a (mon.0) 3244 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:26.176406+0000 mon.a (mon.0) 3245 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:26.181462+0000 mon.a (mon.0) 3246 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:26.181540+0000 mon.a (mon.0) 3247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:26.181606+0000 mon.a (mon.0) 3248 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "40"}]': finished 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:26.181660+0000 mon.a (mon.0) 3249 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-23T00:40:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:26.181709+0000 mon.a (mon.0) 3250 : cluster [DBG] osdmap e607: 8 total, 7 up, 7 in 2024-02-23T00:40:26.587 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:26.206350+0000 mon.a (mon.0) 3251 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:40:26.587 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:25.148825+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.b scrub starts 2024-02-23T00:40:26.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:25.149868+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.b scrub ok 2024-02-23T00:40:26.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:25.448997+0000 osd.6 (osd.6) 242 : cluster [DBG] 5.2 deep-scrub starts 2024-02-23T00:40:26.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:25.452065+0000 osd.6 (osd.6) 243 : cluster [DBG] 5.2 deep-scrub ok 2024-02-23T00:40:26.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:25.631710+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:40:26.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:25.632720+0000 osd.7 (osd.7) 283 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:40:26.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:25.748250+0000 mgr.y (mgr.24437) 541 : cluster [DBG] pgmap v926: 218 pgs: 3 peering, 2 active+remapped+backfill_toofull, 32 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 180 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 5/705 objects degraded (0.709%); 25/705 objects misplaced (3.546%); 461 B/s, 10 objects/s recovering 2024-02-23T00:40:26.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:25.749241+0000 mon.a (mon.0) 3243 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "40"}]: dispatch 2024-02-23T00:40:26.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:25.749432+0000 mon.a (mon.0) 3244 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-23T00:40:26.588 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:26.176406+0000 mon.a (mon.0) 3245 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:26.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:26.181462+0000 mon.a (mon.0) 3246 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:26.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:26.181540+0000 mon.a (mon.0) 3247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:40:26.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:26.181606+0000 mon.a (mon.0) 3248 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "40"}]': finished 2024-02-23T00:40:26.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:26.181660+0000 mon.a (mon.0) 3249 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-23T00:40:26.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:26.181709+0000 mon.a (mon.0) 3250 : cluster [DBG] osdmap e607: 8 total, 7 up, 7 in 2024-02-23T00:40:26.589 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:26.206350+0000 mon.a (mon.0) 3251 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:40:26.757 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:25.148825+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.b scrub starts 2024-02-23T00:40:26.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:25.149868+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.b scrub ok 2024-02-23T00:40:26.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:25.448997+0000 osd.6 (osd.6) 242 : cluster [DBG] 5.2 deep-scrub starts 2024-02-23T00:40:26.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:25.452065+0000 osd.6 (osd.6) 243 : cluster [DBG] 5.2 deep-scrub ok 2024-02-23T00:40:26.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:25.631710+0000 osd.7 (osd.7) 282 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:40:26.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:25.632720+0000 osd.7 (osd.7) 283 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:40:26.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:25.748250+0000 mgr.y (mgr.24437) 541 : cluster [DBG] pgmap v926: 218 pgs: 3 peering, 2 active+remapped+backfill_toofull, 32 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 180 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 5/705 objects degraded (0.709%); 25/705 objects misplaced (3.546%); 461 B/s, 10 objects/s recovering 2024-02-23T00:40:26.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:25.749241+0000 mon.a (mon.0) 3243 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "40"}]: dispatch 2024-02-23T00:40:26.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:25.749432+0000 mon.a (mon.0) 3244 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-23T00:40:26.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:26.176406+0000 mon.a (mon.0) 3245 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:26.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:26.181462+0000 mon.a (mon.0) 3246 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:26.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:26.181540+0000 mon.a (mon.0) 3247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:40:26.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:26.181606+0000 mon.a (mon.0) 3248 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "40"}]': finished 2024-02-23T00:40:26.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:26.181660+0000 mon.a (mon.0) 3249 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-23T00:40:26.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:26.181709+0000 mon.a (mon.0) 3250 : cluster [DBG] osdmap e607: 8 total, 7 up, 7 in 2024-02-23T00:40:26.760 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:26.206350+0000 mon.a (mon.0) 3251 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:40:27.494 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:26.455921+0000 mon.a (mon.0) 3252 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:40:27.495 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:27.171480+0000 mon.a (mon.0) 3253 : cluster [WRN] Health check update: Degraded data redundancy: 5/705 objects degraded (0.709%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:27.495 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:27.182434+0000 mon.a (mon.0) 3254 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:40:27.495 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:27.182485+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e608: 8 total, 7 up, 7 in 2024-02-23T00:40:27.495 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:27.183316+0000 mon.a (mon.0) 3256 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-104"}]: dispatch 2024-02-23T00:40:27.724 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:26.455921+0000 mon.a (mon.0) 3252 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:40:27.724 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:27.171480+0000 mon.a (mon.0) 3253 : cluster [WRN] Health check update: Degraded data redundancy: 5/705 objects degraded (0.709%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:27.724 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:27.182434+0000 mon.a (mon.0) 3254 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:40:27.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:27.182485+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e608: 8 total, 7 up, 7 in 2024-02-23T00:40:27.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:27.183316+0000 mon.a (mon.0) 3256 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-104"}]: dispatch 2024-02-23T00:40:27.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:26.455921+0000 mon.a (mon.0) 3252 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:40:27.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:27.171480+0000 mon.a (mon.0) 3253 : cluster [WRN] Health check update: Degraded data redundancy: 5/705 objects degraded (0.709%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:27.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:27.182434+0000 mon.a (mon.0) 3254 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:40:27.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:27.182485+0000 mon.a (mon.0) 3255 : cluster [DBG] osdmap e608: 8 total, 7 up, 7 in 2024-02-23T00:40:27.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:27.183316+0000 mon.a (mon.0) 3256 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-104"}]: dispatch 2024-02-23T00:40:27.729 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:27.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:27.834 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:27.864 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:27.885 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:27.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:27.989 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:28.013 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:40:28.014 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:40:28.014 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:40:28.014 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:40:28.017 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:28.266 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:28.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:28.370 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:28.399 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:28.636 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-23T00:40:28.648 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:28.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:28.751 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:28.777 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:28.795 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:28.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:28.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.815+0000 7facb4ea4700 1 -- 172.21.15.87:0/2076088868 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7facb00f7ba0 msgr2=0x7facb0103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.815+0000 7facb4ea4700 1 --2- 172.21.15.87:0/2076088868 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7facb00f7ba0 0x7facb0103950 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 rx=0x7faca8009c30 tx=0x7faca8004c50).stop 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.815+0000 7facb4ea4700 1 -- 172.21.15.87:0/2076088868 shutdown_connections 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.815+0000 7facb4ea4700 1 --2- 172.21.15.87:0/2076088868 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7facb00f7ba0 0x7facb0103950 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.815+0000 7facb4ea4700 1 --2- 172.21.15.87:0/2076088868 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7facb00fcb40 0x7facb00fcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.815+0000 7facb4ea4700 1 --2- 172.21.15.87:0/2076088868 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7facb00fd4a0 0x7facb00f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.815+0000 7facb4ea4700 1 -- 172.21.15.87:0/2076088868 >> 172.21.15.87:0/2076088868 conn(0x7facb006c370 msgr2=0x7facb006c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.815+0000 7facb4ea4700 1 -- 172.21.15.87:0/2076088868 shutdown_connections 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.815+0000 7facb4ea4700 1 -- 172.21.15.87:0/2076088868 wait complete. 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.816+0000 7facb4ea4700 1 librados: starting msgr at 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.816+0000 7facb4ea4700 1 librados: starting objecter 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.816+0000 7facb4ea4700 1 Processor -- start 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.816+0000 7facb4ea4700 1 -- start start 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.816+0000 7facb4ea4700 1 librados: setting wanted keys 2024-02-23T00:40:28.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.816+0000 7facb4ea4700 1 librados: calling monclient init 2024-02-23T00:40:28.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facb4ea4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7facb00f7ba0 0x7facb0104280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facb4ea4700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7facb00fcb40 0x7facb0104770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facb4ea4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7facb00fd4a0 0x7facb0104c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facb4ea4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb0116450 con 0x7facb00f7ba0 2024-02-23T00:40:28.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facb4ea4700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb01165c0 con 0x7facb00fcb40 2024-02-23T00:40:28.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facb4ea4700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb0116780 con 0x7facb00fd4a0 2024-02-23T00:40:28.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facae59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7facb00f7ba0 0x7facb0104280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facae59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7facb00f7ba0 0x7facb0104280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58304/0 (socket says 172.21.15.87:58304) 2024-02-23T00:40:28.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facae59c700 1 -- 172.21.15.87:0/3744880438 learned_addr learned my addr 172.21.15.87:0/3744880438 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facaed9d700 1 --2- 172.21.15.87:0/3744880438 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7facb00fd4a0 0x7facb0104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facadd9b700 1 --2- 172.21.15.87:0/3744880438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7facb00fcb40 0x7facb0104770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facae59c700 1 -- 172.21.15.87:0/3744880438 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7facb00fd4a0 msgr2=0x7facb0104c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facae59c700 1 --2- 172.21.15.87:0/3744880438 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7facb00fd4a0 0x7facb0104c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facae59c700 1 -- 172.21.15.87:0/3744880438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7facb00fcb40 msgr2=0x7facb0104770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facae59c700 1 --2- 172.21.15.87:0/3744880438 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7facb00fcb40 0x7facb0104770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facae59c700 1 -- 172.21.15.87:0/3744880438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca4005040 con 0x7facb00f7ba0 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facaed9d700 1 --2- 172.21.15.87:0/3744880438 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7facb00fd4a0 0x7facb0104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facae59c700 1 --2- 172.21.15.87:0/3744880438 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7facb00f7ba0 0x7facb0104280 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 rx=0x7faca00027f0 tx=0x7faca0002b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7fac9f7fe700 10 client.?.objecter ms_handle_connect 0x7facb00f7ba0 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7fac9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca0007b40 con 0x7facb00f7ba0 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.817+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca80097c0 con 0x7facb00f7ba0 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.818+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faca000f040 con 0x7facb00f7ba0 2024-02-23T00:40:28.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.818+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca00146f0 con 0x7facb00f7ba0 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.818+0000 7facb4ea4700 10 client.27600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.818+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb0116de0 con 0x7facb00f7ba0 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.818+0000 7facb4ea4700 1 librados: init done 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.818+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7facb005fa40 con 0x7facb00f7ba0 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.819+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7faca0009110 con 0x7facb00f7ba0 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.819+0000 7fac9f7fe700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6800/1231597463 conn(0x7fac98062c90 0x7fac98065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.819+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(609..609 src has 173..609) v4 ==== 8516+0+0 (secure 0 0 0) 0x7faca007d510 con 0x7facb00f7ba0 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.819+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 osd_map(609..609 src has 173..609) v4 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.819+0000 7fac9f7fe700 3 client.27600.objecter handle_osd_map got epochs [609,609] > 0 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.819+0000 7fac9f7fe700 3 client.27600.objecter handle_osd_map decoding full epoch 609 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.819+0000 7facadd9b700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6800/1231597463 conn(0x7fac98062c90 0x7fac98065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7fac9f7fe700 20 client.27600.objecter dump_active .. 0 homeless 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v609) v1 ==== 54+0+16 (secure 0 0 0) 0x7faca007d9b0 con 0x7facb00f7ba0 2024-02-23T00:40:28.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facadd9b700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6800/1231597463 conn(0x7fac98062c90 0x7fac98065150 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 rx=0x7facb0105a20 tx=0x7faca400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7fac9f7fe700 10 client.27600.objecter ms_handle_connect 0x7fac98062c90 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 10 client.27600.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 1 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 20 client.27600.objecter _get_session osd=-1 returning homeless 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 10 client.27600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=610}) v3 -- 0x7facb005ee80 con 0x7facb00f7ba0 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7facb01071b0 con 0x7facb00f7ba0 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_get_version_reply(handle=1 version=609) v2 ==== 24+0+0 (secure 0 0 0) 0x7faca00507b0 con 0x7facb00f7ba0 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facacd99700 10 client.27600.objecter _check_command_map_dne tid 1 current 609 map_dne_bound 609 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facacd99700 10 client.27600.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facacd99700 15 client.27600.objecter _session_command_op_remove -1 1 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 10 client.27600.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:28.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 2 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6805/685354171 conn(0x7facb00f5c70 0x7facb00273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb00f5b00 osd=1 3 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb00f5b00 osd=1 3 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb00f5b00 osd=1 3 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 2 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb00f5b00 osd=1 3 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 1 2 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb00f5b00 osd=1 4 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 10 client.27600.objecter _send_command 2 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7facb0059e00 con 0x7facb00f5c70 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.820+0000 7facaed9d700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6805/685354171 conn(0x7facb00f5c70 0x7facb00273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.821+0000 7facaed9d700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6805/685354171 conn(0x7facb00f5c70 0x7facb00273c0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:28.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.821+0000 7fac9f7fe700 10 client.27600.objecter ms_handle_connect 0x7facb00f5c70 2024-02-23T00:40:28.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.821+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facb0059e00 con 0x7facb00f5c70 2024-02-23T00:40:28.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.821+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:40:28.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.821+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 2 = system:0 2024-02-23T00:40:28.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.821+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb00f5b00 osd=1 4 2024-02-23T00:40:28.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.821+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 1 2 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 10 client.27600.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 3 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb00f5b00 osd=1 3 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb00f5b00 osd=1 3 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb00f5b00 osd=1 3 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 3 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb00f5b00 osd=1 3 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 1 3 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb00f5b00 osd=1 4 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 10 client.27600.objecter _send_command 3 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.836+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facb00278b0 con 0x7facb00f5c70 2024-02-23T00:40:28.840 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facb00278b0 con 0x7facb00f5c70 2024-02-23T00:40:28.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:28.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb00f5b00 osd=1 4 2024-02-23T00:40:28.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 1 3 2024-02-23T00:40:28.841 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:40:28.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 10 client.27600.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:28.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 4 2024-02-23T00:40:28.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6809/3605394228 conn(0x7facb00594d0 0x7facb011b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0059360 osd=2 3 2024-02-23T00:40:28.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0059360 osd=2 3 2024-02-23T00:40:28.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0059360 osd=2 3 2024-02-23T00:40:28.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 4 2024-02-23T00:40:28.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb0059360 osd=2 3 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 2 4 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0059360 osd=2 4 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 10 client.27600.objecter _send_command 4 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7facb0028700 con 0x7facb00594d0 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facae59c700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6809/3605394228 conn(0x7facb00594d0 0x7facb011b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7fac9f7fe700 10 client.27600.objecter ms_handle_connect 0x7facb00594d0 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.838+0000 7facae59c700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6809/3605394228 conn(0x7facb00594d0 0x7facb011b020 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.839+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facb0028700 con 0x7facb00594d0 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.839+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.839+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 4 = system:0 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.839+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb0059360 osd=2 4 2024-02-23T00:40:28.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.839+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 2 4 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 10 client.27600.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 5 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0059360 osd=2 3 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0059360 osd=2 3 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0059360 osd=2 3 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 5 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb0059360 osd=2 3 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 2 5 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0059360 osd=2 4 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 10 client.27600.objecter _send_command 5 2024-02-23T00:40:28.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.853+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facb0027c60 con 0x7facb00594d0 2024-02-23T00:40:28.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facb0027c60 con 0x7facb00594d0 2024-02-23T00:40:28.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:28.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb0059360 osd=2 4 2024-02-23T00:40:28.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 2 5 2024-02-23T00:40:28.857 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:40:28.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 10 client.27600.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:28.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 6 2024-02-23T00:40:28.857 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6813/128059790 conn(0x7facb00fe5c0 0x7facb011e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0028070 osd=3 3 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0028070 osd=3 3 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0028070 osd=3 3 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 6 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb0028070 osd=3 3 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 3 6 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0028070 osd=3 4 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 10 client.27600.objecter _send_command 6 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7facb0121450 con 0x7facb00fe5c0 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facadd9b700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6813/128059790 conn(0x7facb00fe5c0 0x7facb011e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7fac9f7fe700 10 client.27600.objecter ms_handle_connect 0x7facb00fe5c0 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.855+0000 7facadd9b700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.87:6813/128059790 conn(0x7facb00fe5c0 0x7facb011e6a0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.856+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facb0121450 con 0x7facb00fe5c0 2024-02-23T00:40:28.858 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.856+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:40:28.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.856+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 6 = system:0 2024-02-23T00:40:28.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.856+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb0028070 osd=3 4 2024-02-23T00:40:28.859 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.856+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 3 6 2024-02-23T00:40:28.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 10 client.27600.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:28.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 7 2024-02-23T00:40:28.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0028070 osd=3 3 2024-02-23T00:40:28.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0028070 osd=3 3 2024-02-23T00:40:28.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0028070 osd=3 3 2024-02-23T00:40:28.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 7 2024-02-23T00:40:28.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb0028070 osd=3 3 2024-02-23T00:40:28.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 3 7 2024-02-23T00:40:28.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0028070 osd=3 4 2024-02-23T00:40:28.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 10 client.27600.objecter _send_command 7 2024-02-23T00:40:28.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.870+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facb00fdbe0 con 0x7facb00fe5c0 2024-02-23T00:40:28.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facb00fdbe0 con 0x7facb00fe5c0 2024-02-23T00:40:28.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:28.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb0028070 osd=3 4 2024-02-23T00:40:28.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 3 7 2024-02-23T00:40:28.873 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:40:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 10 client.27600.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 8 2024-02-23T00:40:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6800/4175844757 conn(0x7facb011fdb0 0x7facb0120210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb00fde60 osd=4 3 2024-02-23T00:40:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb00fde60 osd=4 3 2024-02-23T00:40:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb00fde60 osd=4 3 2024-02-23T00:40:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 8 2024-02-23T00:40:28.874 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb00fde60 osd=4 3 2024-02-23T00:40:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 4 8 2024-02-23T00:40:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb00fde60 osd=4 4 2024-02-23T00:40:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 10 client.27600.objecter _send_command 8 2024-02-23T00:40:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7facb0120700 con 0x7facb011fdb0 2024-02-23T00:40:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facaed9d700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6800/4175844757 conn(0x7facb011fdb0 0x7facb0120210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7fac9f7fe700 10 client.27600.objecter ms_handle_connect 0x7facb011fdb0 2024-02-23T00:40:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.872+0000 7facaed9d700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6800/4175844757 conn(0x7facb011fdb0 0x7facb0120210 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:28.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.873+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facb0120700 con 0x7facb011fdb0 2024-02-23T00:40:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.873+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:40:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.873+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 8 = system:0 2024-02-23T00:40:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.873+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb00fde60 osd=4 4 2024-02-23T00:40:28.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.873+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 4 8 2024-02-23T00:40:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 10 client.27600.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 9 2024-02-23T00:40:28.888 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb00fde60 osd=4 3 2024-02-23T00:40:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb00fde60 osd=4 3 2024-02-23T00:40:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb00fde60 osd=4 3 2024-02-23T00:40:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 9 2024-02-23T00:40:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb00fde60 osd=4 3 2024-02-23T00:40:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 4 9 2024-02-23T00:40:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb00fde60 osd=4 4 2024-02-23T00:40:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 10 client.27600.objecter _send_command 9 2024-02-23T00:40:28.889 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.887+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facb01210f0 con 0x7facb011fdb0 2024-02-23T00:40:28.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facb01210f0 con 0x7facb011fdb0 2024-02-23T00:40:28.890 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb00fde60 osd=4 4 2024-02-23T00:40:28.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 4 9 2024-02-23T00:40:28.891 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:40:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 10 client.27600.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 10 2024-02-23T00:40:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6804/2550013795 conn(0x7facb0123680 0x7facb0125a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb01210f0 osd=5 3 2024-02-23T00:40:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb01210f0 osd=5 3 2024-02-23T00:40:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb01210f0 osd=5 3 2024-02-23T00:40:28.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 10 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb01210f0 osd=5 3 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 5 10 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb01210f0 osd=5 4 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 10 client.27600.objecter _send_command 10 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.889+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7facb01286c0 con 0x7facb0123680 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.890+0000 7facae59c700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6804/2550013795 conn(0x7facb0123680 0x7facb0125a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.890+0000 7fac9f7fe700 10 client.27600.objecter ms_handle_connect 0x7facb0123680 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.890+0000 7facae59c700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6804/2550013795 conn(0x7facb0123680 0x7facb0125a70 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.890+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facb01286c0 con 0x7facb0123680 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.890+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.890+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 10 = system:0 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.890+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb01210f0 osd=5 4 2024-02-23T00:40:28.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.890+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 5 10 2024-02-23T00:40:28.899 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 10 client.27600.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 11 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb01210f0 osd=5 3 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb01210f0 osd=5 3 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb01210f0 osd=5 3 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 11 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb01210f0 osd=5 3 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 5 11 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb01210f0 osd=5 4 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 10 client.27600.objecter _send_command 11 2024-02-23T00:40:28.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.906+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facb0125f60 con 0x7facb0123680 2024-02-23T00:40:28.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.908+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facb0125f60 con 0x7facb0123680 2024-02-23T00:40:28.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.908+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:28.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.908+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.908+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb01210f0 osd=5 4 2024-02-23T00:40:28.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.908+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 5 11 2024-02-23T00:40:28.910 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:40:28.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 10 client.27600.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:28.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 12 2024-02-23T00:40:28.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6808/2509408498 conn(0x7facb01265e0 0x7facb01289d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0125f60 osd=6 3 2024-02-23T00:40:28.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0125f60 osd=6 3 2024-02-23T00:40:28.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0125f60 osd=6 3 2024-02-23T00:40:28.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 12 2024-02-23T00:40:28.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb0125f60 osd=6 3 2024-02-23T00:40:28.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 6 12 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0125f60 osd=6 4 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 10 client.27600.objecter _send_command 12 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7facb012b780 con 0x7facb01265e0 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facadd9b700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6808/2509408498 conn(0x7facb01265e0 0x7facb01289d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7fac9f7fe700 10 client.27600.objecter ms_handle_connect 0x7facb01265e0 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.909+0000 7facadd9b700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6808/2509408498 conn(0x7facb01265e0 0x7facb01289d0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.910+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facb012b780 con 0x7facb01265e0 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.910+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.910+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 12 = system:0 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.910+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb0125f60 osd=6 4 2024-02-23T00:40:28.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.910+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 6 12 2024-02-23T00:40:28.915 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:28.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 10 client.27600.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:28.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 13 2024-02-23T00:40:28.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0125f60 osd=6 3 2024-02-23T00:40:28.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0125f60 osd=6 3 2024-02-23T00:40:28.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb0125f60 osd=6 3 2024-02-23T00:40:28.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 13 2024-02-23T00:40:28.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb0125f60 osd=6 3 2024-02-23T00:40:28.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 6 13 2024-02-23T00:40:28.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb0125f60 osd=6 4 2024-02-23T00:40:28.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 10 client.27600.objecter _send_command 13 2024-02-23T00:40:28.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.926+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facb012b780 con 0x7facb01265e0 2024-02-23T00:40:28.928 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:28.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facb012b780 con 0x7facb01265e0 2024-02-23T00:40:28.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:28.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb0125f60 osd=6 4 2024-02-23T00:40:28.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 6 13 2024-02-23T00:40:28.930 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:40:28.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 10 client.27600.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:28.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 14 2024-02-23T00:40:28.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6812/373787000 conn(0x7facb012aac0 0x7facb012af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:28.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb012b780 osd=7 3 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb012b780 osd=7 3 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb012b780 osd=7 3 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 14 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb012b780 osd=7 3 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 7 14 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb012b780 osd=7 4 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 10 client.27600.objecter _send_command 14 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.928+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7facb012eb30 con 0x7facb012aac0 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.929+0000 7facaed9d700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6812/373787000 conn(0x7facb012aac0 0x7facb012af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.929+0000 7fac9f7fe700 10 client.27600.objecter ms_handle_connect 0x7facb012aac0 2024-02-23T00:40:28.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.929+0000 7facaed9d700 1 --2- 172.21.15.87:0/3744880438 >> v2:172.21.15.171:6812/373787000 conn(0x7facb012aac0 0x7facb012af40 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:28.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.930+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7facb012eb30 con 0x7facb012aac0 2024-02-23T00:40:28.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.930+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:40:28.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.930+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 14 = system:0 2024-02-23T00:40:28.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.930+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb012b780 osd=7 4 2024-02-23T00:40:28.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.930+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 7 14 2024-02-23T00:40:28.946 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 10 client.27600.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign -1 15 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb012b780 osd=7 3 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb012b780 osd=7 3 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 20 client.27600.objecter _get_session s=0x7facb012b780 osd=7 3 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_remove -1 15 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 20 client.27600.objecter get_session s=0x7facb012b780 osd=7 3 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 15 client.27600.objecter _session_command_op_assign 7 15 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 20 client.27600.objecter put_session s=0x7facb012b780 osd=7 4 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 10 client.27600.objecter _send_command 15 2024-02-23T00:40:28.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.945+0000 7facb4ea4700 1 -- 172.21.15.87:0/3744880438 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7facb0128f60 con 0x7facb012aac0 2024-02-23T00:40:28.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.947+0000 7fac9f7fe700 1 -- 172.21.15.87:0/3744880438 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7facb0128f60 con 0x7facb012aac0 2024-02-23T00:40:28.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.947+0000 7fac9f7fe700 10 client.27600.objecter ms_dispatch 0x7facb0001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:28.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.947+0000 7fac9f7fe700 10 client.27600.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.947+0000 7fac9f7fe700 20 client.27600.objecter put_session s=0x7facb012b780 osd=7 4 2024-02-23T00:40:28.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:28.947+0000 7fac9f7fe700 15 client.27600.objecter _session_command_op_remove 7 15 2024-02-23T00:40:28.949 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:40:28.949 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:40:28.949 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.949 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.949 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.949 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.949 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.949 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.949 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:40:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:28.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:29.051 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:29.078 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:29.177 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:29.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:29.281 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:29.308 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:27.748721+0000 mgr.y (mgr.24437) 542 : cluster [DBG] pgmap v929: 218 pgs: 3 peering, 2 active+remapped+backfill_toofull, 32 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 180 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 5/705 objects degraded (0.709%); 25/705 objects misplaced (3.546%); 461 B/s, 10 objects/s recovering 2024-02-23T00:40:29.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:28.183389+0000 mon.a (mon.0) 3257 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-104"}]': finished 2024-02-23T00:40:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:28.183485+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e609: 8 total, 7 up, 7 in 2024-02-23T00:40:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:28.184056+0000 mon.a (mon.0) 3259 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-104", "mode": "writeback"}]: dispatch 2024-02-23T00:40:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:28.819374+0000 mon.a (mon.0) 3260 : audit [DBG] from='client.? 172.21.15.87:0/3744880438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:29.564 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:27.748721+0000 mgr.y (mgr.24437) 542 : cluster [DBG] pgmap v929: 218 pgs: 3 peering, 2 active+remapped+backfill_toofull, 32 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 180 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 5/705 objects degraded (0.709%); 25/705 objects misplaced (3.546%); 461 B/s, 10 objects/s recovering 2024-02-23T00:40:29.564 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:28.183389+0000 mon.a (mon.0) 3257 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-104"}]': finished 2024-02-23T00:40:29.564 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:28.183485+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e609: 8 total, 7 up, 7 in 2024-02-23T00:40:29.564 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:28.184056+0000 mon.a (mon.0) 3259 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-104", "mode": "writeback"}]: dispatch 2024-02-23T00:40:29.564 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:28.819374+0000 mon.a (mon.0) 3260 : audit [DBG] from='client.? 172.21.15.87:0/3744880438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:29.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:27.748721+0000 mgr.y (mgr.24437) 542 : cluster [DBG] pgmap v929: 218 pgs: 3 peering, 2 active+remapped+backfill_toofull, 32 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 180 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 5/705 objects degraded (0.709%); 25/705 objects misplaced (3.546%); 461 B/s, 10 objects/s recovering 2024-02-23T00:40:29.564 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:28.183389+0000 mon.a (mon.0) 3257 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-104"}]': finished 2024-02-23T00:40:29.565 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:28.183485+0000 mon.a (mon.0) 3258 : cluster [DBG] osdmap e609: 8 total, 7 up, 7 in 2024-02-23T00:40:29.565 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:28.184056+0000 mon.a (mon.0) 3259 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-104", "mode": "writeback"}]: dispatch 2024-02-23T00:40:29.565 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:28.819374+0000 mon.a (mon.0) 3260 : audit [DBG] from='client.? 172.21.15.87:0/3744880438' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:29.743 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:40:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:40:30.006 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:40:30.056 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:30.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:30.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:30.060 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:30.159 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:30.187 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:30.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:29.180879+0000 mon.a (mon.0) 3261 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:40:30.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:29.184940+0000 mon.a (mon.0) 3262 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-104", "mode": "writeback"}]': finished 2024-02-23T00:40:30.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:29.185041+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e610: 8 total, 7 up, 7 in 2024-02-23T00:40:30.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:29.227639+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:30.309 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:29.750204+0000 mon.a (mon.0) 3265 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "39"}]: dispatch 2024-02-23T00:40:30.310 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:29.750459+0000 mon.a (mon.0) 3266 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-23T00:40:30.310 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:29.180879+0000 mon.a (mon.0) 3261 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:40:30.310 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:29.184940+0000 mon.a (mon.0) 3262 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-104", "mode": "writeback"}]': finished 2024-02-23T00:40:30.310 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:29.185041+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e610: 8 total, 7 up, 7 in 2024-02-23T00:40:30.310 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:29.227639+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:30.310 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:29.750204+0000 mon.a (mon.0) 3265 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "39"}]: dispatch 2024-02-23T00:40:30.310 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:29.750459+0000 mon.a (mon.0) 3266 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-23T00:40:30.314 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:30.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:30.318 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:29.180879+0000 mon.a (mon.0) 3261 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:40:30.318 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:29.184940+0000 mon.a (mon.0) 3262 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-104", "mode": "writeback"}]': finished 2024-02-23T00:40:30.318 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:29.185041+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e610: 8 total, 7 up, 7 in 2024-02-23T00:40:30.318 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:29.227639+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:30.318 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:29.750204+0000 mon.a (mon.0) 3265 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "39"}]: dispatch 2024-02-23T00:40:30.319 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:29.750459+0000 mon.a (mon.0) 3266 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-23T00:40:30.416 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:40:30.416 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:40:30.416 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-23T00:40:30.417 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-23T00:40:30.418 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:30.447 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:30.545 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:30.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:30.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:30.586 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:40:30.586 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:40:30.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:30.677 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:30.876 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:30.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:30.920 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:30.981 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:31.012 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:31.200 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:40:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:40:31.201 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:29.749138+0000 mgr.y (mgr.24437) 543 : cluster [DBG] pgmap v932: 216 pgs: 3 peering, 2 active+remapped+backfill_toofull, 32 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 178 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5/705 objects degraded (0.709%); 25/705 objects misplaced (3.546%) 2024-02-23T00:40:31.201 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:30.201501+0000 mon.a (mon.0) 3267 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:31.201 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:30.201571+0000 mon.a (mon.0) 3268 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "39"}]': finished 2024-02-23T00:40:31.201 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:30.201631+0000 mon.a (mon.0) 3269 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-23T00:40:31.201 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:30.201673+0000 mon.a (mon.0) 3270 : cluster [DBG] osdmap e611: 8 total, 7 up, 7 in 2024-02-23T00:40:31.201 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:30.205230+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104"}]: dispatch 2024-02-23T00:40:31.244 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-23T00:40:31.261 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:31.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:31.365 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:31.393 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:31.528 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:29.749138+0000 mgr.y (mgr.24437) 543 : cluster [DBG] pgmap v932: 216 pgs: 3 peering, 2 active+remapped+backfill_toofull, 32 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 178 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5/705 objects degraded (0.709%); 25/705 objects misplaced (3.546%) 2024-02-23T00:40:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:30.201501+0000 mon.a (mon.0) 3267 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:30.201571+0000 mon.a (mon.0) 3268 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "39"}]': finished 2024-02-23T00:40:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:30.201631+0000 mon.a (mon.0) 3269 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-23T00:40:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:30.201673+0000 mon.a (mon.0) 3270 : cluster [DBG] osdmap e611: 8 total, 7 up, 7 in 2024-02-23T00:40:31.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:30.205230+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104"}]: dispatch 2024-02-23T00:40:31.641 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:31.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:29.749138+0000 mgr.y (mgr.24437) 543 : cluster [DBG] pgmap v932: 216 pgs: 3 peering, 2 active+remapped+backfill_toofull, 32 creating+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 178 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5/705 objects degraded (0.709%); 25/705 objects misplaced (3.546%) 2024-02-23T00:40:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:30.201501+0000 mon.a (mon.0) 3267 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:30.201571+0000 mon.a (mon.0) 3268 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "39"}]': finished 2024-02-23T00:40:31.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:30.201631+0000 mon.a (mon.0) 3269 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-23T00:40:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:30.201673+0000 mon.a (mon.0) 3270 : cluster [DBG] osdmap e611: 8 total, 7 up, 7 in 2024-02-23T00:40:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:30.205230+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104"}]: dispatch 2024-02-23T00:40:31.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:31.772 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:31.871 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:31.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:31.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:32.003 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:32.007 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:40:32.025 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:32.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:32.130 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:32.161 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:32.259 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:32] "GET /metrics HTTP/1.1" 200 176966 "" "Prometheus/2.33.4" 2024-02-23T00:40:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:31.116094+0000 mgr.y (mgr.24437) 544 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:31.198329+0000 mon.a (mon.0) 3272 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:40:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:31.206613+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104"}]': finished 2024-02-23T00:40:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:31.206688+0000 mon.a (mon.0) 3274 : cluster [DBG] osdmap e612: 8 total, 7 up, 7 in 2024-02-23T00:40:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:32.180081+0000 mon.a (mon.0) 3275 : cluster [DBG] osdmap e613: 8 total, 7 up, 7 in 2024-02-23T00:40:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:31.116094+0000 mgr.y (mgr.24437) 544 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:31.198329+0000 mon.a (mon.0) 3272 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:40:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:31.206613+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104"}]': finished 2024-02-23T00:40:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:31.206688+0000 mon.a (mon.0) 3274 : cluster [DBG] osdmap e612: 8 total, 7 up, 7 in 2024-02-23T00:40:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:32.180081+0000 mon.a (mon.0) 3275 : cluster [DBG] osdmap e613: 8 total, 7 up, 7 in 2024-02-23T00:40:32.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 -- 172.21.15.87:0/2755817205 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 msgr2=0x7f03d4103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:32.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 --2- 172.21.15.87:0/2755817205 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 0x7f03d4103a20 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 rx=0x7f03cc009be0 tx=0x7f03cc004d10).stop 2024-02-23T00:40:32.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 -- 172.21.15.87:0/2755817205 shutdown_connections 2024-02-23T00:40:32.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 --2- 172.21.15.87:0/2755817205 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 0x7f03d4103a20 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:32.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 --2- 172.21.15.87:0/2755817205 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d40fcbb0 0x7f03d40fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 --2- 172.21.15.87:0/2755817205 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03d40fd510 0x7f03d40f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 -- 172.21.15.87:0/2755817205 >> 172.21.15.87:0/2755817205 conn(0x7f03d40f2820 msgr2=0x7f03d40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 -- 172.21.15.87:0/2755817205 shutdown_connections 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 -- 172.21.15.87:0/2755817205 wait complete. 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 librados: starting msgr at 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 librados: starting objecter 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 Processor -- start 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 -- start start 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 librados: setting wanted keys 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.264+0000 7f03db908700 1 librados: calling monclient init 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03db908700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 0x7f03d4105040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:32.674 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03db908700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d40fcbb0 0x7f03d4105530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03db908700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03d40fd510 0x7f03d4105a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03db908700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41163a0 con 0x7f03d40fd510 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03db908700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d4116510 con 0x7f03d40fcbb0 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03db908700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d41166d0 con 0x7f03d40f7400 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d96a4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 0x7f03d4105040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d96a4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 0x7f03d4105040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40730/0 (socket says 172.21.15.87:40730) 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d96a4700 1 -- 172.21.15.87:0/3925224228 learned_addr learned my addr 172.21.15.87:0/3925224228 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d9ea5700 1 --2- 172.21.15.87:0/3925224228 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03d40fd510 0x7f03d4105a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d8ea3700 1 --2- 172.21.15.87:0/3925224228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d40fcbb0 0x7f03d4105530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d96a4700 1 -- 172.21.15.87:0/3925224228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d40fcbb0 msgr2=0x7f03d4105530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d96a4700 1 --2- 172.21.15.87:0/3925224228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d40fcbb0 0x7f03d4105530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:32.675 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d96a4700 1 -- 172.21.15.87:0/3925224228 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03d40fd510 msgr2=0x7f03d4105a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d96a4700 1 --2- 172.21.15.87:0/3925224228 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03d40fd510 0x7f03d4105a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d96a4700 1 -- 172.21.15.87:0/3925224228 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc0097c0 con 0x7f03d40f7400 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03d96a4700 1 --2- 172.21.15.87:0/3925224228 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 0x7f03d4105040 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7f03d000c810 tx=0x7f03d000cb20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03ca7fc700 10 client.?.objecter ms_handle_connect 0x7f03d40f7400 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.265+0000 7f03ca7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.266+0000 7f03ca7fc700 1 -- 172.21.15.87:0/3925224228 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0010510 con 0x7f03d40f7400 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.266+0000 7f03ca7fc700 1 -- 172.21.15.87:0/3925224228 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03d0010b40 con 0x7f03d40f7400 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.266+0000 7f03ca7fc700 1 -- 172.21.15.87:0/3925224228 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d0012a90 con 0x7f03d40f7400 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.266+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d41169b0 con 0x7f03d40f7400 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.266+0000 7f03db908700 10 client.38078.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.266+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d4116e50 con 0x7f03d40f7400 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.266+0000 7f03db908700 1 librados: init done 2024-02-23T00:40:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.267+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d405fa60 con 0x7f03d40f7400 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.267+0000 7f03ca7fc700 1 -- 172.21.15.87:0/3925224228 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f03d0010cb0 con 0x7f03d40f7400 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.267+0000 7f03ca7fc700 1 --2- 172.21.15.87:0/3925224228 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03c0062c90 0x7f03c0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.268+0000 7f03ca7fc700 1 -- 172.21.15.87:0/3925224228 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(613..613 src has 173..613) v4 ==== 8135+0+0 (secure 0 0 0) 0x7f03d007d190 con 0x7f03d40f7400 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.268+0000 7f03ca7fc700 10 client.38078.objecter ms_dispatch 0x7f03d4001550 osd_map(613..613 src has 173..613) v4 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.268+0000 7f03ca7fc700 3 client.38078.objecter handle_osd_map got epochs [613,613] > 0 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.268+0000 7f03ca7fc700 3 client.38078.objecter handle_osd_map decoding full epoch 613 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.268+0000 7f03ca7fc700 20 client.38078.objecter dump_active .. 0 homeless 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.268+0000 7f03d8ea3700 1 --2- 172.21.15.87:0/3925224228 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03c0062c90 0x7f03c0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.268+0000 7f03d8ea3700 1 --2- 172.21.15.87:0/3925224228 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03c0062c90 0x7f03c0065150 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 rx=0x7f03d4106790 tx=0x7f03c4009de0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.268+0000 7f03ca7fc700 10 client.38078.objecter ms_handle_connect 0x7f03c0062c90 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.272+0000 7f03ca7fc700 1 -- 172.21.15.87:0/3925224228 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f03d0050610 con 0x7f03d40f7400 2024-02-23T00:40:32.677 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:32.457+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f03d41064d0 con 0x7f03d40f7400 2024-02-23T00:40:32.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:31.116094+0000 mgr.y (mgr.24437) 544 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:32.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:31.198329+0000 mon.a (mon.0) 3272 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:40:32.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:31.206613+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-104"}]': finished 2024-02-23T00:40:32.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:31.206688+0000 mon.a (mon.0) 3274 : cluster [DBG] osdmap e612: 8 total, 7 up, 7 in 2024-02-23T00:40:32.798 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:32.180081+0000 mon.a (mon.0) 3275 : cluster [DBG] osdmap e613: 8 total, 7 up, 7 in 2024-02-23T00:40:33.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.182+0000 7f03ca7fc700 1 -- 172.21.15.87:0/3925224228 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v614) v1 ==== 97+0+0 (secure 0 0 0) 0x7f03d0050200 con 0x7f03d40f7400 2024-02-23T00:40:33.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.186+0000 7f03db908700 10 librados: watch_flush enter 2024-02-23T00:40:33.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.186+0000 7f03db908700 10 librados: watch_flush exit 2024-02-23T00:40:33.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.186+0000 7f03db908700 20 client.38078.objecter shutdown clearing up homeless session... 2024-02-23T00:40:33.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.186+0000 7f03db908700 10 client.38078.objecter successfully canceled tick 2024-02-23T00:40:33.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03c0062c90 msgr2=0x7f03c0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:33.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 --2- 172.21.15.87:0/3925224228 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03c0062c90 0x7f03c0065150 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 rx=0x7f03d4106790 tx=0x7f03c4009de0).stop 2024-02-23T00:40:33.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 msgr2=0x7f03d4105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:33.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 --2- 172.21.15.87:0/3925224228 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 0x7f03d4105040 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 rx=0x7f03d000c810 tx=0x7f03d000cb20).stop 2024-02-23T00:40:33.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 shutdown_connections 2024-02-23T00:40:33.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 --2- 172.21.15.87:0/3925224228 >> v2:172.21.15.87:6800/1231597463 conn(0x7f03c0062c90 0x7f03c0065150 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:33.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 --2- 172.21.15.87:0/3925224228 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f03d40fd510 0x7f03d4105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:33.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 --2- 172.21.15.87:0/3925224228 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f03d40f7400 0x7f03d4105040 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:33.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 --2- 172.21.15.87:0/3925224228 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f03d40fcbb0 0x7f03d4105530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:33.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 >> 172.21.15.87:0/3925224228 conn(0x7f03d40f2820 msgr2=0x7f03d40f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:33.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 shutdown_connections 2024-02-23T00:40:33.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.187+0000 7f03db908700 1 -- 172.21.15.87:0/3925224228 wait complete. 2024-02-23T00:40:33.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:33.188+0000 7f03db908700 1 librados: shutdown 2024-02-23T00:40:33.190 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:40:33.311 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:40:33.312 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:40:33.460 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:31.749603+0000 mgr.y (mgr.24437) 545 : cluster [DBG] pgmap v935: 216 pgs: 4 peering, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 209 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 5/708 objects degraded (0.706%); 25/708 objects misplaced (3.531%) 2024-02-23T00:40:33.514 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:33.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:33.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:31.749603+0000 mgr.y (mgr.24437) 545 : cluster [DBG] pgmap v935: 216 pgs: 4 peering, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 209 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 5/708 objects degraded (0.706%); 25/708 objects misplaced (3.531%) 2024-02-23T00:40:33.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:32.450808+0000 mon.a (mon.0) 3276 : cluster [WRN] Health check update: Degraded data redundancy: 5/708 objects degraded (0.706%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:33.616 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:32.460232+0000 mon.c (mon.2) 476 : audit [INF] from='client.? 172.21.15.87:0/3925224228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:33.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:32.461093+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:33.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:33.176432+0000 mon.a (mon.0) 3278 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:33.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:33.181105+0000 mon.a (mon.0) 3279 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:40:33.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:33.181172+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e614: 8 total, 7 up, 7 in 2024-02-23T00:40:33.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:33.184559+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:33.617 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:33.646 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:32.450808+0000 mon.a (mon.0) 3276 : cluster [WRN] Health check update: Degraded data redundancy: 5/708 objects degraded (0.706%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:32.460232+0000 mon.c (mon.2) 476 : audit [INF] from='client.? 172.21.15.87:0/3925224228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:32.461093+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:33.176432+0000 mon.a (mon.0) 3278 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:33.181105+0000 mon.a (mon.0) 3279 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:33.181172+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e614: 8 total, 7 up, 7 in 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:33.184559+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:31.749603+0000 mgr.y (mgr.24437) 545 : cluster [DBG] pgmap v935: 216 pgs: 4 peering, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 209 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 5/708 objects degraded (0.706%); 25/708 objects misplaced (3.531%) 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:32.450808+0000 mon.a (mon.0) 3276 : cluster [WRN] Health check update: Degraded data redundancy: 5/708 objects degraded (0.706%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:32.460232+0000 mon.c (mon.2) 476 : audit [INF] from='client.? 172.21.15.87:0/3925224228' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:32.461093+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:33.176432+0000 mon.a (mon.0) 3278 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:33.181105+0000 mon.a (mon.0) 3279 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:33.181172+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e614: 8 total, 7 up, 7 in 2024-02-23T00:40:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:33.184559+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:33.905 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:40:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:40:33.923 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-23T00:40:33.995 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:33.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:34.064 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:34.099 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:34.125 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:34.145 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:34.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:34.248 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:34.249 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:34.276 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:34.525 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:34.527 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:33.750262+0000 mgr.y (mgr.24437) 546 : cluster [DBG] pgmap v938: 214 pgs: 2 peering, 2 active+remapped+backfill_toofull, 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 177 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 5/708 objects degraded (0.706%); 25/708 objects misplaced (3.531%) 2024-02-23T00:40:34.527 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:33.751145+0000 mon.a (mon.0) 3282 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "38"}]: dispatch 2024-02-23T00:40:34.527 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:33.751328+0000 mon.a (mon.0) 3283 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-23T00:40:34.527 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:34.182946+0000 mon.a (mon.0) 3284 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:34.527 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:34.183088+0000 mon.a (mon.0) 3285 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "38"}]': finished 2024-02-23T00:40:34.527 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:34.183154+0000 mon.a (mon.0) 3286 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-23T00:40:34.527 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:34.183200+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e615: 8 total, 7 up, 7 in 2024-02-23T00:40:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:34.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:33.750262+0000 mgr.y (mgr.24437) 546 : cluster [DBG] pgmap v938: 214 pgs: 2 peering, 2 active+remapped+backfill_toofull, 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 177 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 5/708 objects degraded (0.706%); 25/708 objects misplaced (3.531%) 2024-02-23T00:40:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:33.751145+0000 mon.a (mon.0) 3282 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "38"}]: dispatch 2024-02-23T00:40:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:33.751328+0000 mon.a (mon.0) 3283 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-23T00:40:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:34.182946+0000 mon.a (mon.0) 3284 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:34.183088+0000 mon.a (mon.0) 3285 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "38"}]': finished 2024-02-23T00:40:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:34.183154+0000 mon.a (mon.0) 3286 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-23T00:40:34.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:34.183200+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e615: 8 total, 7 up, 7 in 2024-02-23T00:40:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:33.750262+0000 mgr.y (mgr.24437) 546 : cluster [DBG] pgmap v938: 214 pgs: 2 peering, 2 active+remapped+backfill_toofull, 32 unknown, 1 active+undersized+degraded+remapped+backfill_toofull, 177 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 5/708 objects degraded (0.706%); 25/708 objects misplaced (3.531%) 2024-02-23T00:40:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:33.751145+0000 mon.a (mon.0) 3282 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "38"}]: dispatch 2024-02-23T00:40:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:33.751328+0000 mon.a (mon.0) 3283 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-23T00:40:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:34.182946+0000 mon.a (mon.0) 3284 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:34.183088+0000 mon.a (mon.0) 3285 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "38"}]': finished 2024-02-23T00:40:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:34.183154+0000 mon.a (mon.0) 3286 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-23T00:40:34.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:34.183200+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e615: 8 total, 7 up, 7 in 2024-02-23T00:40:34.629 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:34.657 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:34.711 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:34.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:34.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:34.816 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:34.844 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:35.043 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:35.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:35.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.937+0000 7fb63d056700 1 -- 172.21.15.87:0/2953722088 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb6380f9610 msgr2=0x7fb638101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:35.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.937+0000 7fb63d056700 1 --2- 172.21.15.87:0/2953722088 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb6380f9610 0x7fb638101d20 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 rx=0x7fb62c0047b0 tx=0x7fb62c009cb0).stop 2024-02-23T00:40:35.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.937+0000 7fb63d056700 1 -- 172.21.15.87:0/2953722088 shutdown_connections 2024-02-23T00:40:35.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.937+0000 7fb63d056700 1 --2- 172.21.15.87:0/2953722088 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb638102210 0x7fb638104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:35.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.937+0000 7fb63d056700 1 --2- 172.21.15.87:0/2953722088 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb6380f8cb0 0x7fb6380f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:35.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.937+0000 7fb63d056700 1 --2- 172.21.15.87:0/2953722088 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb6380f9610 0x7fb638101d20 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.937+0000 7fb63d056700 1 -- 172.21.15.87:0/2953722088 >> 172.21.15.87:0/2953722088 conn(0x7fb6380f2820 msgr2=0x7fb6380f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.938+0000 7fb63d056700 1 -- 172.21.15.87:0/2953722088 shutdown_connections 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.938+0000 7fb63d056700 1 -- 172.21.15.87:0/2953722088 wait complete. 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.938+0000 7fb63d056700 1 librados: starting msgr at 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.938+0000 7fb63d056700 1 librados: starting objecter 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.938+0000 7fb63d056700 1 Processor -- start 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.938+0000 7fb63d056700 1 -- start start 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.938+0000 7fb63d056700 1 librados: setting wanted keys 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.938+0000 7fb63d056700 1 librados: calling monclient init 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63d056700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb6380f8cb0 0x7fb63810e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63d056700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6380f9610 0x7fb63810e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63d056700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb638102210 0x7fb638108430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63d056700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb638108920 con 0x7fb638102210 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63d056700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb638108a90 con 0x7fb6380f9610 2024-02-23T00:40:35.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63d056700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb638108c50 con 0x7fb6380f8cb0 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb636d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb6380f8cb0 0x7fb63810e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63759e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb638102210 0x7fb638108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63659c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6380f9610 0x7fb63810e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb636d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb6380f8cb0 0x7fb63810e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40754/0 (socket says 172.21.15.87:40754) 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63659c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6380f9610 0x7fb63810e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.171:3300/0 says I am v2:172.21.15.87:53830/0 (socket says 172.21.15.87:53830) 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63759e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb638102210 0x7fb638108430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47844/0 (socket says 172.21.15.87:47844) 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb636d9d700 1 -- 172.21.15.87:0/1430584344 learned_addr learned my addr 172.21.15.87:0/1430584344 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63759e700 1 -- 172.21.15.87:0/1430584344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb6380f8cb0 msgr2=0x7fb63810e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63759e700 1 --2- 172.21.15.87:0/1430584344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb6380f8cb0 0x7fb63810e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63759e700 1 -- 172.21.15.87:0/1430584344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6380f9610 msgr2=0x7fb63810e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63759e700 1 --2- 172.21.15.87:0/1430584344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6380f9610 0x7fb63810e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63759e700 1 -- 172.21.15.87:0/1430584344 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb628005040 con 0x7fb638102210 2024-02-23T00:40:35.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63659c700 1 --2- 172.21.15.87:0/1430584344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6380f9610 0x7fb63810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb636d9d700 1 --2- 172.21.15.87:0/1430584344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb6380f8cb0 0x7fb63810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63759e700 1 --2- 172.21.15.87:0/1430584344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb638102210 0x7fb638108430 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fb630010e50 tx=0x7fb63000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb61ffff700 10 client.?.objecter ms_handle_connect 0x7fb638102210 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb61ffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb61ffff700 1 -- 172.21.15.87:0/1430584344 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb630015070 con 0x7fb638102210 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.939+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62c005040 con 0x7fb638102210 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.940+0000 7fb61ffff700 1 -- 172.21.15.87:0/1430584344 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb630009870 con 0x7fb638102210 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.940+0000 7fb61ffff700 1 -- 172.21.15.87:0/1430584344 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6300077a0 con 0x7fb638102210 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.940+0000 7fb63d056700 10 client.27612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.940+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb638109280 con 0x7fb638102210 2024-02-23T00:40:35.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.940+0000 7fb63d056700 1 librados: init done 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.941+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63805fa60 con 0x7fb638102210 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.941+0000 7fb61ffff700 1 -- 172.21.15.87:0/1430584344 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb630004020 con 0x7fb638102210 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.942+0000 7fb61ffff700 1 --2- 172.21.15.87:0/1430584344 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb620062c90 0x7fb620065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.942+0000 7fb61ffff700 1 -- 172.21.15.87:0/1430584344 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(615..615 src has 173..615) v4 ==== 8521+0+0 (secure 0 0 0) 0x7fb6300815a0 con 0x7fb638102210 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.942+0000 7fb61ffff700 10 client.27612.objecter ms_dispatch 0x7fb638001550 osd_map(615..615 src has 173..615) v4 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.942+0000 7fb61ffff700 3 client.27612.objecter handle_osd_map got epochs [615,615] > 0 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.942+0000 7fb61ffff700 3 client.27612.objecter handle_osd_map decoding full epoch 615 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.942+0000 7fb636d9d700 1 --2- 172.21.15.87:0/1430584344 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb620062c90 0x7fb620065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.942+0000 7fb61ffff700 20 client.27612.objecter dump_active .. 0 homeless 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.942+0000 7fb61ffff700 10 client.27612.objecter ms_handle_connect 0x7fb620062c90 2024-02-23T00:40:35.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.942+0000 7fb636d9d700 1 --2- 172.21.15.87:0/1430584344 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb620062c90 0x7fb620065150 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 rx=0x7fb628004d40 tx=0x7fb62800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:35.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:34.946+0000 7fb61ffff700 1 -- 172.21.15.87:0/1430584344 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb630054920 con 0x7fb638102210 2024-02-23T00:40:35.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.114+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fb638107290 con 0x7fb638102210 2024-02-23T00:40:35.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.116+0000 7fb61ffff700 1 -- 172.21.15.87:0/1430584344 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v615) v1 ==== 81+0+48686 (secure 0 0 0) 0x7fb63001f030 con 0x7fb638102210 2024-02-23T00:40:35.119 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:40:35.119 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:40:35.119 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 615, 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "created": "2024-02-23T00:21:28.861318+0000", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "modified": "2024-02-23T00:40:34.177660+0000", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "last_up_change": "2024-02-23T00:36:36.207346+0000", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "last_in_change": "2024-02-23T00:40:22.168751+0000", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_num": 5806080, 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_set": [ 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "nodeep-scrub", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "noscrub", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "pglog_hardlimit", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "purged_snapdirs", 2024-02-23T00:40:35.120 INFO:teuthology.orchestra.run.smithi087.stdout: "recovery_deletes", 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "sortbitwise" 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_version": 113, 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "full_ratio": 0.94999998807907104, 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_snapshot": "", 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_max": 237, 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd": 8, 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "require_min_compat_client": "luminous", 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "min_compat_client": "luminous", 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "require_osd_release": "pacific", 2024-02-23T00:40:35.122 INFO:teuthology.orchestra.run.smithi087.stdout: "pools": [ 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "device_health_metrics", 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:25:09.208309+0000", 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:40:35.123 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 1, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 1, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 1, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 1, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 1, 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:40:35.124 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "68", 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:40:35.125 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:40:35.126 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:40:35.127 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_max": 32, 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_min": 1 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_devicehealth": {} 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.128 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 2, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": ".rgw.root", 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.562186+0000", 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:40:35.129 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "174", 2024-02-23T00:40:35.130 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:40:35.131 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:40:35.132 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.133 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 3, 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "iscsi", 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.989206+0000", 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:40:35.134 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 13, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 13, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 13, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 13, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 13, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:40:35.135 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "137", 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "135", 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:40:35.136 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:40:35.137 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:40:35.138 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd": {} 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 4, 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.log", 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:17.282440+0000", 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:40:35.139 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "4.16", 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 371, 2024-02-23T00:40:35.140 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 370, 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 370, 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "63'21", 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "62'12" 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "532", 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "371", 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "481", 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:40:35.141 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:40:35.142 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:40:35.143 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 5, 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.control", 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:19.262164+0000", 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:40:35.144 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 39, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 38, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "5.27", 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 613, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 612, 2024-02-23T00:40:35.145 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 612, 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "615", 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "615", 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "615", 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:40:35.146 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:40:35.147 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:40:35.148 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 6, 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.meta", 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:21.319951+0000", 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:40:35.149 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 25, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 12, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 12, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 12, 2024-02-23T00:40:35.150 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 24, 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "6.19", 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 613, 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 612, 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 612, 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "615", 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "615", 2024-02-23T00:40:35.151 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "615", 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:40:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:40:35.153 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_bias": 4 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.154 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 22, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:46.071806+0000", 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:40:35.155 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:40:35.156 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:40:35.157 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:40:35.158 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 234, 2024-02-23T00:40:35.159 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-101", 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:40:15.435168+0000", 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 3, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 1, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:40:35.160 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 8, 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 8, 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 8, 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 8, 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 8, 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "612", 2024-02-23T00:40:35.161 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "611", 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "611", 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "611", 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 608, 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:40:35.162 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi087-82091-101", 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:40:35.163 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 8192, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 237, 2024-02-23T00:40:35.164 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-106", 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:40:32.450545+0000", 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 32769, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,creating", 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:40:35.165 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "615", 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:40:35.166 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:40:35.167 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:40:35.168 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "rados": {} 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:40:35.169 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7", 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 0, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 0, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 0, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 8, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 407, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 425, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.170 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6802", 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.171 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6804", 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.172 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6803", 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2153191863 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6801/2153191863", 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6802/2153191863", 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6804/2153191863", 2024-02-23T00:40:35.173 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6803/2153191863", 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "exists" 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "85525ce8-38d5-4840-8057-547cb176c859", 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 13, 2024-02-23T00:40:35.174 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 614, 2024-02-23T00:40:35.175 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:40:35.175 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:40:35.175 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:40:35.175 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.175 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.175 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6805", 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6806", 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.176 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6808", 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.177 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6807", 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6805/685354171", 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6806/685354171", 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6808/685354171", 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6807/685354171", 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.178 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa", 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 18, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 614, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.179 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6809", 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6810", 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.180 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6812", 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6811", 2024-02-23T00:40:35.181 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6809/3605394228", 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6810/3605394228", 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6812/3605394228", 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6811/3605394228", 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.182 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a", 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 26, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 614, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.183 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6813", 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6814", 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.184 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6816", 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6815", 2024-02-23T00:40:35.185 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:40:35.186 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.186 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.186 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.186 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6813/128059790", 2024-02-23T00:40:35.186 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6814/128059790", 2024-02-23T00:40:35.187 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6816/128059790", 2024-02-23T00:40:35.187 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6815/128059790", 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c", 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:40:35.188 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.9473876953125, 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 31, 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 614, 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6800", 2024-02-23T00:40:35.189 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6801", 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:40:35.190 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6803", 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6802", 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:40:35.191 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6800/4175844757", 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6801/4175844757", 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4175844757", 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4175844757", 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:40:35.192 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d", 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.8517608642578125, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 38, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 614, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.193 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6804", 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6805", 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.194 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6807", 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6806", 2024-02-23T00:40:35.195 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6804/2550013795", 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6805/2550013795", 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2550013795", 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2550013795", 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:40:35.196 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01", 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 44, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 614, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.197 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6808", 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6809", 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.198 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6811", 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.199 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6810", 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6808/2509408498", 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6809/2509408498", 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6811/2509408498", 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6810/2509408498", 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.200 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939", 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 51, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 614, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.201 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6812", 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6813", 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.202 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6815", 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6814", 2024-02-23T00:40:35.203 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6812/373787000", 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6813/373787000", 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6815/373787000", 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6814/373787000", 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_xinfo": [ 2024-02-23T00:40:35.204 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 65536, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:36:24.026593+0000", 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 424 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:40:35.205 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:39:40.836525+0000", 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:39:37.736435+0000", 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.206 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:39:52.054033+0000", 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:40:35.207 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:39:54.305859+0000", 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:39:41.934582+0000", 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.208 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:39:37.307455+0000", 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:40:35.209 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:39:56.938360+0000", 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap": [], 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap_items": [ 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.18", 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 1 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.210 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.1c", 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 4 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.13", 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:40:35.211 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "5.19", 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 6 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "22.1a", 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:40:35.212 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 5 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_temp": [ 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.16", 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: 3, 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: 7, 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: 1 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.213 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.17", 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: 1, 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: 5 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.1f", 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: 1, 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: 4, 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: 7 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.214 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_temp": [], 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "blocklist": { 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1121018082": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/2138166229": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/27794119": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/1675126043": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/3767016525": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/969918521": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/795095363": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2273151327": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1685406414": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2093733106": "2024-02-24T00:21:58.144574+0000" 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.215 INFO:teuthology.orchestra.run.smithi087.stdout: "range_blocklist": {}, 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profiles": { 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "default": { 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van" 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "testprofile-test-rados-api-smithi087-82091-101": { 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-device-class": "", 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-root": "default", 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-23T00:40:35.216 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van", 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "w": "8" 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps_queue": [ 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 7, 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.217 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.218 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 21, 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.219 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 61, 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:40:35.220 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 29, 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 78, 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.221 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 86, 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.222 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 95, 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.223 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 4, 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 4 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 107, 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.224 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 112, 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 125, 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.225 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 10, 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 133, 2024-02-23T00:40:35.226 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 154, 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 14, 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.227 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 164, 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 17, 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 177, 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:40:35.228 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 20, 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "new_removed_snaps": [], 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "new_purged_snaps": [], 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_node_flags": {}, 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "device_class_flags": {}, 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode": { 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_enabled": false, 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_bucket_count": 0, 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "degraded_stretch_mode": 0, 2024-02-23T00:40:35.229 INFO:teuthology.orchestra.run.smithi087.stdout: "recovering_stretch_mode": 0, 2024-02-23T00:40:35.230 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_bucket": 0 2024-02-23T00:40:35.230 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:40:35.230 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:40:35.230 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 10 librados: watch_flush enter 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 10 librados: watch_flush exit 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 20 client.27612.objecter shutdown clearing up homeless session... 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 10 client.27612.objecter successfully canceled tick 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb620062c90 msgr2=0x7fb620065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 --2- 172.21.15.87:0/1430584344 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb620062c90 0x7fb620065150 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 rx=0x7fb628004d40 tx=0x7fb62800b040).stop 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb638102210 msgr2=0x7fb638108430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 --2- 172.21.15.87:0/1430584344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb638102210 0x7fb638108430 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 rx=0x7fb630010e50 tx=0x7fb63000b040).stop 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 shutdown_connections 2024-02-23T00:40:35.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 --2- 172.21.15.87:0/1430584344 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb620062c90 0x7fb620065150 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:35.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 --2- 172.21.15.87:0/1430584344 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb638102210 0x7fb638108430 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:35.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 --2- 172.21.15.87:0/1430584344 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb6380f8cb0 0x7fb63810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:35.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 --2- 172.21.15.87:0/1430584344 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb6380f9610 0x7fb63810e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:35.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 >> 172.21.15.87:0/1430584344 conn(0x7fb6380f2820 msgr2=0x7fb6380f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:35.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 shutdown_connections 2024-02-23T00:40:35.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 -- 172.21.15.87:0/1430584344 wait complete. 2024-02-23T00:40:35.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:35.118+0000 7fb63d056700 1 librados: shutdown 2024-02-23T00:40:35.257 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:35.562 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:35.116554+0000 mon.a (mon.0) 3288 : audit [DBG] from='client.? 172.21.15.87:0/1430584344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:40:35.562 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:35.183175+0000 mon.a (mon.0) 3289 : cluster [DBG] osdmap e616: 8 total, 7 up, 7 in 2024-02-23T00:40:35.563 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:35.201699+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:40:35.606 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:35.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:35.710 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:35.737 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:35.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:35.116554+0000 mon.a (mon.0) 3288 : audit [DBG] from='client.? 172.21.15.87:0/1430584344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:40:35.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:35.183175+0000 mon.a (mon.0) 3289 : cluster [DBG] osdmap e616: 8 total, 7 up, 7 in 2024-02-23T00:40:35.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:35.201699+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:40:35.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:35.116554+0000 mon.a (mon.0) 3288 : audit [DBG] from='client.? 172.21.15.87:0/1430584344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:40:35.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:35.183175+0000 mon.a (mon.0) 3289 : cluster [DBG] osdmap e616: 8 total, 7 up, 7 in 2024-02-23T00:40:35.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:35.201699+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:40:35.834 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:35.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:35.940 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:35.971 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:36.028 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:40:36.077 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:40:36.077 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:40:36.094 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:36.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:36.198 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:36.226 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:36.246 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:36.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:36.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:36.349 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:36.375 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:36.497 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:36.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:36.601 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:36.628 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:36.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.626+0000 7ff118bc7700 1 -- 172.21.15.87:0/2052836771 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff1140f8420 msgr2=0x7ff114104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.626+0000 7ff118bc7700 1 --2- 172.21.15.87:0/2052836771 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff1140f8420 0x7ff114104a20 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 rx=0x7ff10c009be0 tx=0x7ff10c004d10).stop 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.626+0000 7ff118bc7700 1 -- 172.21.15.87:0/2052836771 shutdown_connections 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.626+0000 7ff118bc7700 1 --2- 172.21.15.87:0/2052836771 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff1140f8420 0x7ff114104a20 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.626+0000 7ff118bc7700 1 --2- 172.21.15.87:0/2052836771 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1140fdbb0 0x7ff1140fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.626+0000 7ff118bc7700 1 --2- 172.21.15.87:0/2052836771 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1140fe510 0x7ff1140f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.626+0000 7ff118bc7700 1 -- 172.21.15.87:0/2052836771 >> 172.21.15.87:0/2052836771 conn(0x7ff1140f3800 msgr2=0x7ff1140f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.626+0000 7ff118bc7700 1 -- 172.21.15.87:0/2052836771 shutdown_connections 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.626+0000 7ff118bc7700 1 -- 172.21.15.87:0/2052836771 wait complete. 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 librados: starting msgr at 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 librados: starting objecter 2024-02-23T00:40:36.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 Processor -- start 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 -- start start 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 librados: setting wanted keys 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 librados: calling monclient init 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1140f8420 0x7ff11410df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1140fdbb0 0x7ff11410e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff1140fe510 0x7ff1141080f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1141085e0 con 0x7ff1140fdbb0 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff114108750 con 0x7ff1140f8420 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff118bc7700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1141088c0 con 0x7ff1140fe510 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff111d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1140fdbb0 0x7ff11410e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff112d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff1140fe510 0x7ff1141080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff111d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1140fdbb0 0x7ff11410e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47848/0 (socket says 172.21.15.87:47848) 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff111d9b700 1 -- 172.21.15.87:0/401133062 learned_addr learned my addr 172.21.15.87:0/401133062 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:36.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff112d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff1140fe510 0x7ff1141080f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40764/0 (socket says 172.21.15.87:40764) 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.627+0000 7ff11259c700 1 --2- 172.21.15.87:0/401133062 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1140f8420 0x7ff11410df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff112d9d700 1 -- 172.21.15.87:0/401133062 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1140f8420 msgr2=0x7ff11410df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff112d9d700 1 --2- 172.21.15.87:0/401133062 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff1140f8420 0x7ff11410df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff112d9d700 1 -- 172.21.15.87:0/401133062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1140fdbb0 msgr2=0x7ff11410e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff112d9d700 1 --2- 172.21.15.87:0/401133062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1140fdbb0 0x7ff11410e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff112d9d700 1 -- 172.21.15.87:0/401133062 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff104005040 con 0x7ff1140fe510 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff111d9b700 1 --2- 172.21.15.87:0/401133062 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff1140fdbb0 0x7ff11410e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff112d9d700 1 --2- 172.21.15.87:0/401133062 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff1140fe510 0x7ff1141080f0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 rx=0x7ff10c009bb0 tx=0x7ff10c004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff10b7fe700 10 client.?.objecter ms_handle_connect 0x7ff1140fe510 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff10b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00e040 con 0x7ff1140fe510 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff10c018040 con 0x7ff1140fe510 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10c0097c0 con 0x7ff1140fe510 2024-02-23T00:40:36.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c013de0 con 0x7ff1140fe510 2024-02-23T00:40:36.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff118bc7700 10 client.38090.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:36.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff114108e30 con 0x7ff1140fe510 2024-02-23T00:40:36.634 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.628+0000 7ff118bc7700 1 librados: init done 2024-02-23T00:40:36.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.629+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff11405ee80 con 0x7ff1140fe510 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.629+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff10c0134b0 con 0x7ff1140fe510 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff10b7fe700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff100062c90 0x7ff100065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(617..617 src has 173..617) v4 ==== 8516+0+0 (secure 0 0 0) 0x7ff10c07e1c0 con 0x7ff1140fe510 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 osd_map(617..617 src has 173..617) v4 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff11259c700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff100062c90 0x7ff100065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff10b7fe700 3 client.38090.objecter handle_osd_map got epochs [617,617] > 0 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff10b7fe700 3 client.38090.objecter handle_osd_map decoding full epoch 617 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff10b7fe700 20 client.38090.objecter dump_active .. 0 homeless 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v617) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff10c085070 con 0x7ff1140fe510 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff10b7fe700 10 client.38090.objecter ms_handle_connect 0x7ff100062c90 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff11259c700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff100062c90 0x7ff100065150 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 rx=0x7ff104009350 tx=0x7ff10400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff118bc7700 10 client.38090.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:36.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 1 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff118bc7700 20 client.38090.objecter _get_session osd=-1 returning homeless 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff118bc7700 10 client.38090.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=618}) v3 -- 0x7ff114107110 con 0x7ff1140fe510 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff114114d80 con 0x7ff1140fe510 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=617) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff10c0515f0 con 0x7ff1140fe510 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff110d99700 10 client.38090.objecter _check_command_map_dne tid 1 current 617 map_dne_bound 617 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff110d99700 10 client.38090.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.630+0000 7ff110d99700 15 client.38090.objecter _session_command_op_remove -1 1 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 10 client.38090.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 2 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6805/685354171 conn(0x7ff1140f5b70 0x7ff1140273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff1140f5a00 osd=1 3 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff1140f5a00 osd=1 3 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff1140f5a00 osd=1 3 2024-02-23T00:40:36.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 2 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff1140f5a00 osd=1 3 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 1 2 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff1140f5a00 osd=1 4 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 10 client.38090.objecter _send_command 2 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff114059e00 con 0x7ff1140f5b70 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff111d9b700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6805/685354171 conn(0x7ff1140f5b70 0x7ff1140273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff10b7fe700 10 client.38090.objecter ms_handle_connect 0x7ff1140f5b70 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.631+0000 7ff111d9b700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6805/685354171 conn(0x7ff1140f5b70 0x7ff1140273c0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.632+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff114059e00 con 0x7ff1140f5b70 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.632+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.632+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 2 = system:0 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.632+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff1140f5a00 osd=1 4 2024-02-23T00:40:36.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.632+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 1 2 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 10 client.38090.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 3 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff1140f5a00 osd=1 3 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff1140f5a00 osd=1 3 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff1140f5a00 osd=1 3 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 3 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff1140f5a00 osd=1 3 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 1 3 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff1140f5a00 osd=1 4 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 10 client.38090.objecter _send_command 3 2024-02-23T00:40:36.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.646+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1140278b0 con 0x7ff1140f5b70 2024-02-23T00:40:36.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff1140278b0 con 0x7ff1140f5b70 2024-02-23T00:40:36.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:36.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff1140f5a00 osd=1 4 2024-02-23T00:40:36.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 1 3 2024-02-23T00:40:36.651 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:40:36.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 10 client.38090.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:36.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 4 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff1140594d0 0x7ff114118f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114059360 osd=2 3 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114059360 osd=2 3 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114059360 osd=2 3 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 4 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff114059360 osd=2 3 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 2 4 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114059360 osd=2 4 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 10 client.38090.objecter _send_command 4 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff114028760 con 0x7ff1140594d0 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff11259c700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff1140594d0 0x7ff114118f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff11259c700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff1140594d0 0x7ff114118f10 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:36.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.648+0000 7ff10b7fe700 10 client.38090.objecter ms_handle_connect 0x7ff1140594d0 2024-02-23T00:40:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.649+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff114028760 con 0x7ff1140594d0 2024-02-23T00:40:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.649+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:40:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.649+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 4 = system:0 2024-02-23T00:40:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.649+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff114059360 osd=2 4 2024-02-23T00:40:36.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.649+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 2 4 2024-02-23T00:40:36.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 10 client.38090.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 5 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114059360 osd=2 3 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114059360 osd=2 3 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114059360 osd=2 3 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 5 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff114059360 osd=2 3 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 2 5 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114059360 osd=2 4 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 10 client.38090.objecter _send_command 5 2024-02-23T00:40:36.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.663+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff114027cc0 con 0x7ff1140594d0 2024-02-23T00:40:36.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff114027cc0 con 0x7ff1140594d0 2024-02-23T00:40:36.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:36.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff114059360 osd=2 4 2024-02-23T00:40:36.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 2 5 2024-02-23T00:40:36.668 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:40:36.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 10 client.38090.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:36.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 6 2024-02-23T00:40:36.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6813/128059790 conn(0x7ff11411a580 0x7ff11411c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff1140ff140 osd=3 3 2024-02-23T00:40:36.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff1140ff140 osd=3 3 2024-02-23T00:40:36.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff1140ff140 osd=3 3 2024-02-23T00:40:36.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 6 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff1140ff140 osd=3 3 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 3 6 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff1140ff140 osd=3 4 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 10 client.38090.objecter _send_command 6 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1140ff940 con 0x7ff11411a580 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff111d9b700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6813/128059790 conn(0x7ff11411a580 0x7ff11411c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff111d9b700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.87:6813/128059790 conn(0x7ff11411a580 0x7ff11411c970 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.665+0000 7ff10b7fe700 10 client.38090.objecter ms_handle_connect 0x7ff11411a580 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.666+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff1140ff940 con 0x7ff11411a580 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.666+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.666+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 6 = system:0 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.666+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff1140ff140 osd=3 4 2024-02-23T00:40:36.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.666+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 3 6 2024-02-23T00:40:36.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 10 client.38090.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:36.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 7 2024-02-23T00:40:36.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff1140ff140 osd=3 3 2024-02-23T00:40:36.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff1140ff140 osd=3 3 2024-02-23T00:40:36.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff1140ff140 osd=3 3 2024-02-23T00:40:36.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 7 2024-02-23T00:40:36.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff1140ff140 osd=3 3 2024-02-23T00:40:36.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 3 7 2024-02-23T00:40:36.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff1140ff140 osd=3 4 2024-02-23T00:40:36.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 10 client.38090.objecter _send_command 7 2024-02-23T00:40:36.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.680+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff1140febb0 con 0x7ff11411a580 2024-02-23T00:40:36.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff1140febb0 con 0x7ff11411a580 2024-02-23T00:40:36.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:36.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff1140ff140 osd=3 4 2024-02-23T00:40:36.684 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 3 7 2024-02-23T00:40:36.684 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:40:36.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 10 client.38090.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:36.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 8 2024-02-23T00:40:36.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff11406c090 0x7ff11411ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114119bd0 osd=4 3 2024-02-23T00:40:36.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114119bd0 osd=4 3 2024-02-23T00:40:36.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114119bd0 osd=4 3 2024-02-23T00:40:36.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 8 2024-02-23T00:40:36.685 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff114119bd0 osd=4 3 2024-02-23T00:40:36.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 4 8 2024-02-23T00:40:36.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114119bd0 osd=4 4 2024-02-23T00:40:36.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 10 client.38090.objecter _send_command 8 2024-02-23T00:40:36.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff11411f360 con 0x7ff11406c090 2024-02-23T00:40:36.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff112d9d700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff11406c090 0x7ff11411ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff10b7fe700 10 client.38090.objecter ms_handle_connect 0x7ff11406c090 2024-02-23T00:40:36.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.682+0000 7ff112d9d700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff11406c090 0x7ff11411ee70 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:36.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.683+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff11411f360 con 0x7ff11406c090 2024-02-23T00:40:36.686 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.683+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:40:36.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.683+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 8 = system:0 2024-02-23T00:40:36.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.683+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff114119bd0 osd=4 4 2024-02-23T00:40:36.687 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.683+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 4 8 2024-02-23T00:40:36.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 10 client.38090.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 9 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114119bd0 osd=4 3 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114119bd0 osd=4 3 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114119bd0 osd=4 3 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 9 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff114119bd0 osd=4 3 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 4 9 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114119bd0 osd=4 4 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 10 client.38090.objecter _send_command 9 2024-02-23T00:40:36.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.697+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff11406c4d0 con 0x7ff11406c090 2024-02-23T00:40:36.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff11406c4d0 con 0x7ff11406c090 2024-02-23T00:40:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff114119bd0 osd=4 4 2024-02-23T00:40:36.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 4 9 2024-02-23T00:40:36.702 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:40:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 10 client.38090.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 10 2024-02-23T00:40:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff11406cb50 0x7ff114125a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff11406c4d0 osd=5 3 2024-02-23T00:40:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff11406c4d0 osd=5 3 2024-02-23T00:40:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff11406c4d0 osd=5 3 2024-02-23T00:40:36.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 10 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff11406c4d0 osd=5 3 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 5 10 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff11406c4d0 osd=5 4 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 10 client.38090.objecter _send_command 10 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.699+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1141286b0 con 0x7ff11406cb50 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.700+0000 7ff11259c700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff11406cb50 0x7ff114125a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.700+0000 7ff10b7fe700 10 client.38090.objecter ms_handle_connect 0x7ff11406cb50 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.700+0000 7ff11259c700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff11406cb50 0x7ff114125a60 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.701+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff1141286b0 con 0x7ff11406cb50 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.701+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.701+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 10 = system:0 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.701+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff11406c4d0 osd=5 4 2024-02-23T00:40:36.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.701+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 5 10 2024-02-23T00:40:36.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 10 client.38090.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:36.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 11 2024-02-23T00:40:36.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff11406c4d0 osd=5 3 2024-02-23T00:40:36.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff11406c4d0 osd=5 3 2024-02-23T00:40:36.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff11406c4d0 osd=5 3 2024-02-23T00:40:36.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 11 2024-02-23T00:40:36.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff11406c4d0 osd=5 3 2024-02-23T00:40:36.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 5 11 2024-02-23T00:40:36.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff11406c4d0 osd=5 4 2024-02-23T00:40:36.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 10 client.38090.objecter _send_command 11 2024-02-23T00:40:36.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.715+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff114125f50 con 0x7ff11406cb50 2024-02-23T00:40:36.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff114125f50 con 0x7ff11406cb50 2024-02-23T00:40:36.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:36.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff11406c4d0 osd=5 4 2024-02-23T00:40:36.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 5 11 2024-02-23T00:40:36.720 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 10 client.38090.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 12 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff1141265d0 0x7ff114126a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114125f50 osd=6 3 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114125f50 osd=6 3 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114125f50 osd=6 3 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 12 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff114125f50 osd=6 3 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 6 12 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114125f50 osd=6 4 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 10 client.38090.objecter _send_command 12 2024-02-23T00:40:36.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff114128690 con 0x7ff1141265d0 2024-02-23T00:40:36.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.717+0000 7ff111d9b700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff1141265d0 0x7ff114126a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.718+0000 7ff10b7fe700 10 client.38090.objecter ms_handle_connect 0x7ff1141265d0 2024-02-23T00:40:36.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.718+0000 7ff111d9b700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff1141265d0 0x7ff114126a10 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:36.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.718+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff114128690 con 0x7ff1141265d0 2024-02-23T00:40:36.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.718+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:40:36.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.718+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 12 = system:0 2024-02-23T00:40:36.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.718+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff114125f50 osd=6 4 2024-02-23T00:40:36.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.718+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 6 12 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 10 client.38090.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 13 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114125f50 osd=6 3 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114125f50 osd=6 3 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114125f50 osd=6 3 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 13 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff114125f50 osd=6 3 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 6 13 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114125f50 osd=6 4 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 10 client.38090.objecter _send_command 13 2024-02-23T00:40:36.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.733+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff114128690 con 0x7ff1141265d0 2024-02-23T00:40:36.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.734+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff114128690 con 0x7ff1141265d0 2024-02-23T00:40:36.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.734+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:36.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.734+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.734+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff114125f50 osd=6 4 2024-02-23T00:40:36.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.734+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 6 13 2024-02-23T00:40:36.737 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:40:36.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 10 client.38090.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:36.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 14 2024-02-23T00:40:36.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6812/373787000 conn(0x7ff11412adc0 0x7ff11412b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:36.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114128690 osd=7 3 2024-02-23T00:40:36.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114128690 osd=7 3 2024-02-23T00:40:36.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114128690 osd=7 3 2024-02-23T00:40:36.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 14 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff114128690 osd=7 3 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 7 14 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114128690 osd=7 4 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 10 client.38090.objecter _send_command 14 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff11412b850 con 0x7ff11412adc0 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff112d9d700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6812/373787000 conn(0x7ff11412adc0 0x7ff11412b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff10b7fe700 10 client.38090.objecter ms_handle_connect 0x7ff11412adc0 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.735+0000 7ff112d9d700 1 --2- 172.21.15.87:0/401133062 >> v2:172.21.15.171:6812/373787000 conn(0x7ff11412adc0 0x7ff11412b200 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.736+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff11412b850 con 0x7ff11412adc0 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.736+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.736+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 14 = system:0 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.736+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff114128690 osd=7 4 2024-02-23T00:40:36.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.736+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 7 14 2024-02-23T00:40:36.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 10 client.38090.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:36.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign -1 15 2024-02-23T00:40:36.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114128690 osd=7 3 2024-02-23T00:40:36.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114128690 osd=7 3 2024-02-23T00:40:36.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 20 client.38090.objecter _get_session s=0x7ff114128690 osd=7 3 2024-02-23T00:40:36.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_remove -1 15 2024-02-23T00:40:36.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 20 client.38090.objecter get_session s=0x7ff114128690 osd=7 3 2024-02-23T00:40:36.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 15 client.38090.objecter _session_command_op_assign 7 15 2024-02-23T00:40:36.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 20 client.38090.objecter put_session s=0x7ff114128690 osd=7 4 2024-02-23T00:40:36.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 10 client.38090.objecter _send_command 15 2024-02-23T00:40:36.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.750+0000 7ff118bc7700 1 -- 172.21.15.87:0/401133062 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff11412b850 con 0x7ff11412adc0 2024-02-23T00:40:36.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.752+0000 7ff10b7fe700 1 -- 172.21.15.87:0/401133062 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff11412b850 con 0x7ff11412adc0 2024-02-23T00:40:36.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.752+0000 7ff10b7fe700 10 client.38090.objecter ms_dispatch 0x7ff114001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:36.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.752+0000 7ff10b7fe700 10 client.38090.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.752+0000 7ff10b7fe700 20 client.38090.objecter put_session s=0x7ff114128690 osd=7 4 2024-02-23T00:40:36.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:36.752+0000 7ff10b7fe700 15 client.38090.objecter _session_command_op_remove 7 15 2024-02-23T00:40:36.755 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:40:36.755 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:40:36.755 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.755 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.755 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.755 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.755 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.755 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.756 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:40:36.963 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-23T00:40:36.975 INFO:tasks.thrashosds.thrasher:j is {'epoch': 615, 'fsid': '66db207c-d1e1-11ee-95c0-87774f69a715', 'created': '2024-02-23T00:21:28.861318+0000', 'modified': '2024-02-23T00:40:34.177660+0000', 'last_up_change': '2024-02-23T00:36:36.207346+0000', 'last_in_change': '2024-02-23T00:40:22.168751+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 113, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 237, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '135', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '4.16', 'ready_epoch': 371, 'last_epoch_started': 370, 'last_epoch_clean': 370, 'source_version': "63'21", 'target_version': "62'12"}, 'last_change': '532', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '371', 'last_force_op_resend_preluminous': '481', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 39, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 38, 'last_pg_merge_meta': {'source_pgid': '5.27', 'ready_epoch': 613, 'last_epoch_started': 612, 'last_epoch_clean': 612, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '615', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '615', 'last_force_op_resend_preluminous': '615', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 25, 'pg_placement_num': 12, 'pg_placement_num_target': 12, 'pg_num_target': 12, 'pg_num_pending': 24, 'last_pg_merge_meta': {'source_pgid': '6.19', 'ready_epoch': 613, 'last_epoch_started': 612, 'last_epoch_clean': 612, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '615', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '615', 'last_force_op_resend_preluminous': '615', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 22, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-23T00:30:46.071806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi087-82091-101', 'create_time': '2024-02-23T00:40:15.435168+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '612', 'last_force_op_resend': '611', 'last_force_op_resend_prenautilus': '611', 'last_force_op_resend_preluminous': '611', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 608, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi087-82091-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 237, 'pool_name': 'test-rados-api-smithi087-82091-106', 'create_time': '2024-02-23T00:40:32.450545+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '615', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e3549bad-2891-47b9-bf77-132cadc20bf7', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 407, 'down_at': 425, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6801', 'nonce': 2153191863}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6802', 'nonce': 2153191863}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6804', 'nonce': 2153191863}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6803', 'nonce': 2153191863}]}, 'public_addr': '172.21.15.87:6801/2153191863', 'cluster_addr': '172.21.15.87:6802/2153191863', 'heartbeat_back_addr': '172.21.15.87:6804/2153191863', 'heartbeat_front_addr': '172.21.15.87:6803/2153191863', 'state': ['exists']}, {'osd': 1, 'uuid': '85525ce8-38d5-4840-8057-547cb176c859', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 614, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6805', 'nonce': 685354171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6806', 'nonce': 685354171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6808', 'nonce': 685354171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6807', 'nonce': 685354171}]}, 'public_addr': '172.21.15.87:6805/685354171', 'cluster_addr': '172.21.15.87:6806/685354171', 'heartbeat_back_addr': '172.21.15.87:6808/685354171', 'heartbeat_front_addr': '172.21.15.87:6807/685354171', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 614, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6809', 'nonce': 3605394228}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6810', 'nonce': 3605394228}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6812', 'nonce': 3605394228}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6811', 'nonce': 3605394228}]}, 'public_addr': '172.21.15.87:6809/3605394228', 'cluster_addr': '172.21.15.87:6810/3605394228', 'heartbeat_back_addr': '172.21.15.87:6812/3605394228', 'heartbeat_front_addr': '172.21.15.87:6811/3605394228', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 614, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6813', 'nonce': 128059790}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6814', 'nonce': 128059790}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6816', 'nonce': 128059790}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6815', 'nonce': 128059790}]}, 'public_addr': '172.21.15.87:6813/128059790', 'cluster_addr': '172.21.15.87:6814/128059790', 'heartbeat_back_addr': '172.21.15.87:6816/128059790', 'heartbeat_front_addr': '172.21.15.87:6815/128059790', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c3734a04-7343-4251-9b5b-bedb9e8b882c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9473876953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 614, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6800', 'nonce': 4175844757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6801', 'nonce': 4175844757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6803', 'nonce': 4175844757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6802', 'nonce': 4175844757}]}, 'public_addr': '172.21.15.171:6800/4175844757', 'cluster_addr': '172.21.15.171:6801/4175844757', 'heartbeat_back_addr': '172.21.15.171:6803/4175844757', 'heartbeat_front_addr': '172.21.15.171:6802/4175844757', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fe86232d-96b9-4cbf-9cac-373286fa103d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8517608642578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 614, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6804', 'nonce': 2550013795}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6805', 'nonce': 2550013795}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6807', 'nonce': 2550013795}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6806', 'nonce': 2550013795}]}, 'public_addr': '172.21.15.171:6804/2550013795', 'cluster_addr': '172.21.15.171:6805/2550013795', 'heartbeat_back_addr': '172.21.15.171:6807/2550013795', 'heartbeat_front_addr': '172.21.15.171:6806/2550013795', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 614, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6808', 'nonce': 2509408498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6809', 'nonce': 2509408498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6811', 'nonce': 2509408498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6810', 'nonce': 2509408498}]}, 'public_addr': '172.21.15.171:6808/2509408498', 'cluster_addr': '172.21.15.171:6809/2509408498', 'heartbeat_back_addr': '172.21.15.171:6811/2509408498', 'heartbeat_front_addr': '172.21.15.171:6810/2509408498', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '29ddcf97-369a-4a2c-b3e2-142e84ccf939', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 614, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6812', 'nonce': 373787000}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6813', 'nonce': 373787000}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6815', 'nonce': 373787000}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6814', 'nonce': 373787000}]}, 'public_addr': '172.21.15.171:6812/373787000', 'cluster_addr': '172.21.15.171:6813/373787000', 'heartbeat_back_addr': '172.21.15.171:6815/373787000', 'heartbeat_front_addr': '172.21.15.171:6814/373787000', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-23T00:36:24.026593+0000', 'dead_epoch': 424}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:39:40.836525+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:39:37.736435+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:39:52.054033+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:39:54.305859+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:39:41.934582+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:39:37.307455+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:39:56.938360+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '2.1c', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '22.1a', 'mappings': [{'from': 7, 'to': 5}]}], 'pg_temp': [{'pgid': '4.16', 'osds': [3, 7, 1]}, {'pgid': '4.17', 'osds': [1, 5]}, {'pgid': '4.1f', 'osds': [1, 4, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.87:6800/1121018082': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/2138166229': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/27794119': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/1675126043': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/3767016525': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/969918521': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/795095363': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2273151327': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/1685406414': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2093733106': '2024-02-24T00:21:58.144574+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi087-82091-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 21, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 61, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 63, 'snaps': [{'begin': 29, 'length': 1}]}, {'pool': 78, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 86, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 177, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-23T00:40:36.975 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:40:36.976 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:36.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:37.080 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:37.106 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:37.125 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:37.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:37.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:37.198 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:35.751168+0000 mgr.y (mgr.24437) 547 : cluster [DBG] pgmap v941: 214 pgs: 4 clean+premerge+peered, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 207 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s; 5/732 objects degraded (0.683%); 25/732 objects misplaced (3.415%) 2024-02-23T00:40:37.198 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:36.185533+0000 mon.a (mon.0) 3291 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:40:37.198 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:36.185607+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e617: 8 total, 7 up, 7 in 2024-02-23T00:40:37.198 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:36.188403+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-106"}]: dispatch 2024-02-23T00:40:37.198 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:36.631414+0000 mon.c (mon.2) 477 : audit [DBG] from='client.? 172.21.15.87:0/401133062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:37.231 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:37.250 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:37.257 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:35.751168+0000 mgr.y (mgr.24437) 547 : cluster [DBG] pgmap v941: 214 pgs: 4 clean+premerge+peered, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 207 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s; 5/732 objects degraded (0.683%); 25/732 objects misplaced (3.415%) 2024-02-23T00:40:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:36.185533+0000 mon.a (mon.0) 3291 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:40:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:36.185607+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e617: 8 total, 7 up, 7 in 2024-02-23T00:40:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:36.188403+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-106"}]: dispatch 2024-02-23T00:40:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:36.631414+0000 mon.c (mon.2) 477 : audit [DBG] from='client.? 172.21.15.87:0/401133062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:35.751168+0000 mgr.y (mgr.24437) 547 : cluster [DBG] pgmap v941: 214 pgs: 4 clean+premerge+peered, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 207 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s; 5/732 objects degraded (0.683%); 25/732 objects misplaced (3.415%) 2024-02-23T00:40:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:36.185533+0000 mon.a (mon.0) 3291 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:40:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:36.185607+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e617: 8 total, 7 up, 7 in 2024-02-23T00:40:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:36.188403+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-106"}]: dispatch 2024-02-23T00:40:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:36.631414+0000 mon.c (mon.2) 477 : audit [DBG] from='client.? 172.21.15.87:0/401133062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:40:37.745 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:40:37.777 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:37.805 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:38.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:37.186958+0000 mon.a (mon.0) 3294 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-106"}]': finished 2024-02-23T00:40:38.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:37.187059+0000 mon.a (mon.0) 3295 : cluster [DBG] osdmap e618: 8 total, 7 up, 7 in 2024-02-23T00:40:38.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:37.196767+0000 mon.a (mon.0) 3296 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-106", "mode": "writeback"}]: dispatch 2024-02-23T00:40:38.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:37.752528+0000 mon.a (mon.0) 3297 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "37"}]: dispatch 2024-02-23T00:40:38.278 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:37.752693+0000 mon.a (mon.0) 3298 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:40:38.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:37.186958+0000 mon.a (mon.0) 3294 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-106"}]': finished 2024-02-23T00:40:38.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:37.187059+0000 mon.a (mon.0) 3295 : cluster [DBG] osdmap e618: 8 total, 7 up, 7 in 2024-02-23T00:40:38.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:37.196767+0000 mon.a (mon.0) 3296 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-106", "mode": "writeback"}]: dispatch 2024-02-23T00:40:38.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:37.752528+0000 mon.a (mon.0) 3297 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "37"}]: dispatch 2024-02-23T00:40:38.279 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:37.752693+0000 mon.a (mon.0) 3298 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:40:38.512 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:37.186958+0000 mon.a (mon.0) 3294 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-106"}]': finished 2024-02-23T00:40:38.512 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:37.187059+0000 mon.a (mon.0) 3295 : cluster [DBG] osdmap e618: 8 total, 7 up, 7 in 2024-02-23T00:40:38.512 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:37.196767+0000 mon.a (mon.0) 3296 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-106", "mode": "writeback"}]: dispatch 2024-02-23T00:40:38.512 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:37.752528+0000 mon.a (mon.0) 3297 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "37"}]: dispatch 2024-02-23T00:40:38.512 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:37.752693+0000 mon.a (mon.0) 3298 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:40:39.066 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:40:39.066 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:40:39.066 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:40:39.066 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:40:39.066 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:40:39.161 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:39.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:39.267 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:39.295 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:39.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:37.751607+0000 mgr.y (mgr.24437) 548 : cluster [DBG] pgmap v944: 212 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 207 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 5/732 objects degraded (0.683%); 25/732 objects misplaced (3.415%) 2024-02-23T00:40:39.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:38.192014+0000 mon.a (mon.0) 3299 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:40:39.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:38.199289+0000 mon.a (mon.0) 3300 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-106", "mode": "writeback"}]': finished 2024-02-23T00:40:39.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:38.199382+0000 mon.a (mon.0) 3301 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "37"}]': finished 2024-02-23T00:40:39.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:38.199433+0000 mon.a (mon.0) 3302 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:40:39.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:38.199472+0000 mon.a (mon.0) 3303 : cluster [DBG] osdmap e619: 8 total, 7 up, 7 in 2024-02-23T00:40:39.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:38.211002+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-23T00:40:39.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:38.211163+0000 mgr.y (mgr.24437) 549 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-23T00:40:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:37.751607+0000 mgr.y (mgr.24437) 548 : cluster [DBG] pgmap v944: 212 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 207 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 5/732 objects degraded (0.683%); 25/732 objects misplaced (3.415%) 2024-02-23T00:40:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:38.192014+0000 mon.a (mon.0) 3299 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:40:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:38.199289+0000 mon.a (mon.0) 3300 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-106", "mode": "writeback"}]': finished 2024-02-23T00:40:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:38.199382+0000 mon.a (mon.0) 3301 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "37"}]': finished 2024-02-23T00:40:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:38.199433+0000 mon.a (mon.0) 3302 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:38.199472+0000 mon.a (mon.0) 3303 : cluster [DBG] osdmap e619: 8 total, 7 up, 7 in 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:38.211002+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:38.211163+0000 mgr.y (mgr.24437) 549 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:37.751607+0000 mgr.y (mgr.24437) 548 : cluster [DBG] pgmap v944: 212 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 207 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 5/732 objects degraded (0.683%); 25/732 objects misplaced (3.415%) 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:38.192014+0000 mon.a (mon.0) 3299 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:38.199289+0000 mon.a (mon.0) 3300 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-106", "mode": "writeback"}]': finished 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:38.199382+0000 mon.a (mon.0) 3301 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "37"}]': finished 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:38.199433+0000 mon.a (mon.0) 3302 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:38.199472+0000 mon.a (mon.0) 3303 : cluster [DBG] osdmap e619: 8 total, 7 up, 7 in 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:38.211002+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-23T00:40:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:38.211163+0000 mgr.y (mgr.24437) 549 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.4"}]: dispatch 2024-02-23T00:40:39.744 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:39.746 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:40:39.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:39.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:39.829 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-23T00:40:39.849 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:39.898 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:39.911 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:40:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:40:39.657+0000 7f6e819c8700 -1 mgr get_metadata_python Requested missing service osd.0 2024-02-23T00:40:39.921 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:39.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:40.025 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:40.052 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:40.104 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:40:40.500 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:40.502 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:39.203751+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e620: 8 total, 7 up, 7 in 2024-02-23T00:40:40.502 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:39.654839+0000 mon.a (mon.0) 3306 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:40.503 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:39.203751+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e620: 8 total, 7 up, 7 in 2024-02-23T00:40:40.503 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:39.654839+0000 mon.a (mon.0) 3306 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:40.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:40.596 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:39.203751+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e620: 8 total, 7 up, 7 in 2024-02-23T00:40:40.597 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:39.654839+0000 mon.a (mon.0) 3306 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:40.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:40.632 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:40.830 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:40.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:40.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:40.934 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:40.961 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:40.997 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:41.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:41.102 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:41.131 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:41.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.090+0000 7f7ce274d700 1 -- 172.21.15.87:0/2887470199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7cdc0fdbb0 msgr2=0x7f7cdc0fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:41.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.090+0000 7f7ce274d700 1 --2- 172.21.15.87:0/2887470199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7cdc0fdbb0 0x7f7cdc0fdf90 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 rx=0x7f7ccc004870 tx=0x7f7ccc009cb0).stop 2024-02-23T00:40:41.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.090+0000 7f7ce274d700 1 -- 172.21.15.87:0/2887470199 shutdown_connections 2024-02-23T00:40:41.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.090+0000 7f7ce274d700 1 --2- 172.21.15.87:0/2887470199 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cdc0f8420 0x7f7cdc104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.090+0000 7f7ce274d700 1 --2- 172.21.15.87:0/2887470199 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7cdc0fdbb0 0x7f7cdc0fdf90 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.090+0000 7f7ce274d700 1 --2- 172.21.15.87:0/2887470199 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7cdc0fe510 0x7f7cdc0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.090+0000 7f7ce274d700 1 -- 172.21.15.87:0/2887470199 >> 172.21.15.87:0/2887470199 conn(0x7f7cdc0f3800 msgr2=0x7f7cdc0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:41.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.090+0000 7f7ce274d700 1 -- 172.21.15.87:0/2887470199 shutdown_connections 2024-02-23T00:40:41.143 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.090+0000 7f7ce274d700 1 -- 172.21.15.87:0/2887470199 wait complete. 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 librados: starting msgr at 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 librados: starting objecter 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 Processor -- start 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 -- start start 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 librados: setting wanted keys 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 librados: calling monclient init 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7cdc0f8420 0x7f7cdc06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cdc0fdbb0 0x7f7cdc06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7cdc0fe510 0x7f7cdc06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc116540 con 0x7f7cdc0fe510 2024-02-23T00:40:41.144 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc1166b0 con 0x7f7cdc0fdbb0 2024-02-23T00:40:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce274d700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cdc116870 con 0x7f7cdc0f8420 2024-02-23T00:40:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7cdbfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7cdc0f8420 0x7f7cdc06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7ce0cea700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7cdc0fe510 0x7f7cdc06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7cdbfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7cdc0f8420 0x7f7cdc06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40786/0 (socket says 172.21.15.87:40786) 2024-02-23T00:40:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.091+0000 7f7cdbfff700 1 -- 172.21.15.87:0/3768220606 learned_addr learned my addr 172.21.15.87:0/3768220606 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cdb7fe700 1 --2- 172.21.15.87:0/3768220606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cdc0fdbb0 0x7f7cdc06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cdbfff700 1 -- 172.21.15.87:0/3768220606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cdc0fdbb0 msgr2=0x7f7cdc06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cdbfff700 1 --2- 172.21.15.87:0/3768220606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cdc0fdbb0 0x7f7cdc06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.145 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cdbfff700 1 -- 172.21.15.87:0/3768220606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7cdc0fe510 msgr2=0x7f7cdc06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cdbfff700 1 --2- 172.21.15.87:0/3768220606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7cdc0fe510 0x7f7cdc06bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cdbfff700 1 -- 172.21.15.87:0/3768220606 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd0005040 con 0x7f7cdc0f8420 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7ce0cea700 1 --2- 172.21.15.87:0/3768220606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7cdc0fe510 0x7f7cdc06bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cdbfff700 1 --2- 172.21.15.87:0/3768220606 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7cdc0f8420 0x7f7cdc06b0d0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7f7ccc009eb0 tx=0x7f7ccc00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cd97fa700 10 client.?.objecter ms_handle_connect 0x7f7cdc0f8420 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cd97fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cd97fa700 1 -- 172.21.15.87:0/3768220606 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc00e070 con 0x7f7cdc0f8420 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc005040 con 0x7f7cdc0f8420 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cd97fa700 1 -- 172.21.15.87:0/3768220606 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ccc013da0 con 0x7f7cdc0f8420 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7cd97fa700 1 -- 172.21.15.87:0/3768220606 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc019070 con 0x7f7cdc0f8420 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7ce274d700 10 client.38096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cdc116dc0 con 0x7f7cdc0f8420 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.092+0000 7f7ce274d700 1 librados: init done 2024-02-23T00:40:41.146 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.093+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cdc107060 con 0x7f7cdc0f8420 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.093+0000 7f7cd97fa700 1 -- 172.21.15.87:0/3768220606 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7ccc01e420 con 0x7f7cdc0f8420 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.094+0000 7f7cd97fa700 1 --2- 172.21.15.87:0/3768220606 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7cc4062c90 0x7f7cc4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.094+0000 7f7cd97fa700 1 -- 172.21.15.87:0/3768220606 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(621..621 src has 173..621) v4 ==== 8516+0+0 (secure 0 0 0) 0x7f7ccc07de50 con 0x7f7cdc0f8420 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.094+0000 7f7cd97fa700 10 client.38096.objecter ms_dispatch 0x7f7cdc001550 osd_map(621..621 src has 173..621) v4 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.094+0000 7f7cd97fa700 3 client.38096.objecter handle_osd_map got epochs [621,621] > 0 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.094+0000 7f7cd97fa700 3 client.38096.objecter handle_osd_map decoding full epoch 621 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.094+0000 7f7cdb7fe700 1 --2- 172.21.15.87:0/3768220606 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7cc4062c90 0x7f7cc4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.094+0000 7f7cd97fa700 20 client.38096.objecter dump_active .. 0 homeless 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.094+0000 7f7cd97fa700 10 client.38096.objecter ms_handle_connect 0x7f7cc4062c90 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.094+0000 7f7cdb7fe700 1 --2- 172.21.15.87:0/3768220606 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7cc4062c90 0x7f7cc4065150 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7f7cdc06c820 tx=0x7f7cd000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:41.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.098+0000 7f7cd97fa700 1 -- 172.21.15.87:0/3768220606 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7ccc051150 con 0x7f7cdc0f8420 2024-02-23T00:40:41.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.270+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7cdc05fac0 con 0x7f7cc4062c90 2024-02-23T00:40:41.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.285+0000 7f7cd97fa700 1 -- 172.21.15.87:0/3768220606 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+535345 (secure 0 0 0) 0x7f7cdc05fac0 con 0x7f7cc4062c90 2024-02-23T00:40:41.295 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 10 librados: watch_flush enter 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 10 librados: watch_flush exit 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 20 client.38096.objecter shutdown clearing up homeless session... 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 10 client.38096.objecter successfully canceled tick 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7cc4062c90 msgr2=0x7f7cc4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 --2- 172.21.15.87:0/3768220606 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7cc4062c90 0x7f7cc4065150 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 rx=0x7f7cdc06c820 tx=0x7f7cd000b040).stop 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7cdc0f8420 msgr2=0x7f7cdc06b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 --2- 172.21.15.87:0/3768220606 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7cdc0f8420 0x7f7cdc06b0d0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7f7ccc009eb0 tx=0x7f7ccc00b040).stop 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 shutdown_connections 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 --2- 172.21.15.87:0/3768220606 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7cc4062c90 0x7f7cc4065150 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 --2- 172.21.15.87:0/3768220606 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7cdc0fe510 0x7f7cdc06bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 --2- 172.21.15.87:0/3768220606 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7cdc0f8420 0x7f7cdc06b0d0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 --2- 172.21.15.87:0/3768220606 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7cdc0fdbb0 0x7f7cdc06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 >> 172.21.15.87:0/3768220606 conn(0x7f7cdc0f3800 msgr2=0x7f7cdc0fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:41.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 shutdown_connections 2024-02-23T00:40:41.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 -- 172.21.15.87:0/3768220606 wait complete. 2024-02-23T00:40:41.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.289+0000 7f7ce274d700 1 librados: shutdown 2024-02-23T00:40:41.301 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:40:41.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:39.687728+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:40:41.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:39.687824+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:40:41.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:39.752088+0000 mgr.y (mgr.24437) 550 : cluster [DBG] pgmap v947: 212 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 207 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5/732 objects degraded (0.683%); 25/732 objects misplaced (3.415%) 2024-02-23T00:40:41.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:40.208166+0000 mon.a (mon.0) 3307 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2024-02-23T00:40:41.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:41.208781+0000 mon.a (mon.0) 3308 : cluster [DBG] osdmap e622: 8 total, 7 up, 7 in 2024-02-23T00:40:41.326 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:40:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:40:41.334 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:41.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:41.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 -- 172.21.15.87:0/2155480712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 msgr2=0x7f42ac101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/2155480712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 0x7f42ac101d20 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f42a00047b0 tx=0x7f42a0009cb0).stop 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 -- 172.21.15.87:0/2155480712 shutdown_connections 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/2155480712 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42ac102210 0x7f42ac104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/2155480712 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42ac0f8cb0 0x7f42ac0f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/2155480712 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 0x7f42ac101d20 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 -- 172.21.15.87:0/2155480712 >> 172.21.15.87:0/2155480712 conn(0x7f42ac0f2820 msgr2=0x7f42ac0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 -- 172.21.15.87:0/2155480712 shutdown_connections 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 -- 172.21.15.87:0/2155480712 wait complete. 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 librados: starting msgr at 2024-02-23T00:40:41.420 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 librados: starting objecter 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 Processor -- start 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 -- start start 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 librados: setting wanted keys 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.405+0000 7f42b2a0d700 1 librados: calling monclient init 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42ac0f8cb0 0x7f42ac10e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 0x7f42ac10e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42ac102210 0x7f42ac108350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac108840 con 0x7f42ac0f9610 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac1089b0 con 0x7f42ac0f8cb0 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac108b70 con 0x7f42ac102210 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42ab7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 0x7f42ac10e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:41.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42ab7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 0x7f42ac10e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48046/0 (socket says 172.21.15.87:48046) 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42ab7fe700 1 -- 172.21.15.87:0/3407417391 learned_addr learned my addr 172.21.15.87:0/3407417391 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b0faa700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42ac102210 0x7f42ac108350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42abfff700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42ac0f8cb0 0x7f42ac10e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42ab7fe700 1 -- 172.21.15.87:0/3407417391 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42ac102210 msgr2=0x7f42ac108350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42ab7fe700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42ac102210 0x7f42ac108350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42ab7fe700 1 -- 172.21.15.87:0/3407417391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42ac0f8cb0 msgr2=0x7f42ac10e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42ab7fe700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42ac0f8cb0 0x7f42ac10e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42ab7fe700 1 -- 172.21.15.87:0/3407417391 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a40097c0 con 0x7f42ac0f9610 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b0faa700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42ac102210 0x7f42ac108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42ab7fe700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 0x7f42ac10e630 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f42a0009b10 tx=0x7f42a0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:41.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42a97fa700 10 client.?.objecter ms_handle_connect 0x7f42ac0f9610 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42a97fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42a97fa700 1 -- 172.21.15.87:0/3407417391 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0018070 con 0x7f42ac0f9610 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a0005040 con 0x7f42ac0f9610 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42a97fa700 1 -- 172.21.15.87:0/3407417391 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42a0013e70 con 0x7f42ac0f9610 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42a97fa700 1 -- 172.21.15.87:0/3407417391 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a001ddb0 con 0x7f42ac0f9610 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 10 client.27630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ac109010 con 0x7f42ac0f9610 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.406+0000 7f42b2a0d700 1 librados: init done 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.407+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ac109200 con 0x7f42ac0f9610 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42a97fa700 1 -- 172.21.15.87:0/3407417391 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f42a001d420 con 0x7f42ac0f9610 2024-02-23T00:40:41.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42a97fa700 1 --2- 172.21.15.87:0/3407417391 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4294062c90 0x7f4294065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:41.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42a97fa700 1 -- 172.21.15.87:0/3407417391 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(622..622 src has 173..622) v4 ==== 8516+0+0 (secure 0 0 0) 0x7f42a007e920 con 0x7f42ac0f9610 2024-02-23T00:40:41.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42a97fa700 10 client.27630.objecter ms_dispatch 0x7f42ac001550 osd_map(622..622 src has 173..622) v4 2024-02-23T00:40:41.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42a97fa700 3 client.27630.objecter handle_osd_map got epochs [622,622] > 0 2024-02-23T00:40:41.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42a97fa700 3 client.27630.objecter handle_osd_map decoding full epoch 622 2024-02-23T00:40:41.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42abfff700 1 --2- 172.21.15.87:0/3407417391 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4294062c90 0x7f4294065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:41.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42a97fa700 20 client.27630.objecter dump_active .. 0 homeless 2024-02-23T00:40:41.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42abfff700 1 --2- 172.21.15.87:0/3407417391 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4294062c90 0x7f4294065150 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 rx=0x7f429c0041d0 tx=0x7f429c009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:41.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.408+0000 7f42a97fa700 10 client.27630.objecter ms_handle_connect 0x7f4294062c90 2024-02-23T00:40:41.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.412+0000 7f42a97fa700 1 -- 172.21.15.87:0/3407417391 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f42a0051bc0 con 0x7f42ac0f9610 2024-02-23T00:40:41.439 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:41.466 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:41.565 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:41.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:39.687728+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:40:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:39.687824+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:40:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:39.752088+0000 mgr.y (mgr.24437) 550 : cluster [DBG] pgmap v947: 212 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 207 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5/732 objects degraded (0.683%); 25/732 objects misplaced (3.415%) 2024-02-23T00:40:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:40.208166+0000 mon.a (mon.0) 3307 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2024-02-23T00:40:41.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:41.208781+0000 mon.a (mon.0) 3308 : cluster [DBG] osdmap e622: 8 total, 7 up, 7 in 2024-02-23T00:40:41.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:39.687728+0000 osd.2 (osd.2) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:40:41.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:39.687824+0000 osd.2 (osd.2) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:40:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:39.752088+0000 mgr.y (mgr.24437) 550 : cluster [DBG] pgmap v947: 212 pgs: 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 207 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5/732 objects degraded (0.683%); 25/732 objects misplaced (3.415%) 2024-02-23T00:40:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:40.208166+0000 mon.a (mon.0) 3307 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2024-02-23T00:40:41.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:41.208781+0000 mon.a (mon.0) 3308 : cluster [DBG] osdmap e622: 8 total, 7 up, 7 in 2024-02-23T00:40:41.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:41.597+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f42ac05ee80 con 0x7f42ac0f9610 2024-02-23T00:40:41.670 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:41.699 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:41.823 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:41.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:41.830 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:40:41.830 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:40:41.830 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:40:41.830 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:40:41.830 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26036.0:2 64.6 64:602f83fe:::foo:head [write 0~128,set-alloc-hint object_size 0 write_size 0] snapc 0=[] ondisk+write+known_if_redirected e115)", 2024-02-23T00:40:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:40:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 575.601653801, 2024-02-23T00:40:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036711013000000001, 2024-02-23T00:40:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26036", 2024-02-23T00:40:41.831 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3255892024", 2024-02-23T00:40:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2 2024-02-23T00:40:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:40:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.832 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224876+0000", 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224873+0000", 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.163e-05 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224885+0000", 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.8150000000000001e-05 2024-02-23T00:40:41.833 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224903+0000", 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2343e-05 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224915+0000", 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6192000000000005e-05 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224992+0000", 2024-02-23T00:40:41.834 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4450000000000001e-06 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.224998+0000", 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.032120923000000003 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257119+0000", 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000125234 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.835 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257244+0000", 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000107636 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,5", 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.257352+0000", 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011677390000000001 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.258520+0000", 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014228299999999999 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.836 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.259942+0000", 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015894640000000001 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261532+0000", 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.154e-05 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261563+0000", 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.3230999999999998e-05 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.837 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:06.261587+0000", 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036711013000000001 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:4 93.7s0 93:f24ec45b:test-rados-api-smithi087-82107-15::foo:head [write 8192~8192] snapc 3=[3,2] ondisk+write+known_if_redirected e144)", 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 546.83035458699999, 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028560524, 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.838 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 4 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.839 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996175+0000", 2024-02-23T00:40:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:40:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.840 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996173+0000", 2024-02-23T00:40:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1501e-05 2024-02-23T00:40:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996194+0000", 2024-02-23T00:40:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.1659000000000003e-05 2024-02-23T00:40:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.841 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996226+0000", 2024-02-23T00:40:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.439e-06 2024-02-23T00:40:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.996234+0000", 2024-02-23T00:40:41.842 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013176539999999999 2024-02-23T00:40:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.997552+0000", 2024-02-23T00:40:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5900000000000008e-06 2024-02-23T00:40:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:40:41.843 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:34.997560+0000", 2024-02-23T00:40:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0057013619999999997 2024-02-23T00:40:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.003262+0000", 2024-02-23T00:40:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015312100000000001 2024-02-23T00:40:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.844 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.003415+0000", 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0033453049999999998 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.006760+0000", 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038155379999999998 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.010576+0000", 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014109313 2024-02-23T00:40:41.845 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.024685+0000", 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0339e-05 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:35.024735+0000", 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028560524 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.846 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.36632.0:18 93.0s0 93:1b316dc0:test-rados-api-smithi087-82107-16::foo:head [delete] snapc 7=[7,6] ondisk+write+peerstat_old+known_if_redirected e150)", 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 540.83527834999995, 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026236400999999999, 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.36632", 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3246607396", 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 18 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.847 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:40:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991251+0000", 2024-02-23T00:40:41.848 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991249+0000", 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0203999999999999e-05 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991259+0000", 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5091e-05 2024-02-23T00:40:41.849 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991274+0000", 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.729e-06 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991282+0000", 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7206999999999995e-05 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991349+0000", 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000134998 2024-02-23T00:40:41.850 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991484+0000", 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00048211400000000001 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.991966+0000", 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00039970199999999999 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.851 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:40:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:40.992366+0000", 2024-02-23T00:40:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025049227 2024-02-23T00:40:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.852 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.017415+0000", 2024-02-23T00:40:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2229000000000003e-05 2024-02-23T00:40:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:31:41.017488+0000", 2024-02-23T00:40:41.853 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026236400999999999 2024-02-23T00:40:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.854 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.44587.0:1 151.6s0 151:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-02-23T00:40:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:40:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 471.78821084100002, 2024-02-23T00:40:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763, 2024-02-23T00:40:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.855 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.44587", 2024-02-23T00:40:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/843545378", 2024-02-23T00:40:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:40:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.856 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:40:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.857 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:40:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:40:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038316+0000", 2024-02-23T00:40:41.858 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9619999999999999e-06 2024-02-23T00:40:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038325+0000", 2024-02-23T00:40:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7385e-05 2024-02-23T00:40:41.859 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038343+0000", 2024-02-23T00:40:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0739999999999999e-06 2024-02-23T00:40:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.860 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038352+0000", 2024-02-23T00:40:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077681699999999996 2024-02-23T00:40:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.861 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039128+0000", 2024-02-23T00:40:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0860000000000002e-06 2024-02-23T00:40:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:40:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039132+0000", 2024-02-23T00:40:41.862 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0036840789999999 2024-02-23T00:40:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042816+0000", 2024-02-23T00:40:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000158937 2024-02-23T00:40:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.863 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042975+0000", 2024-02-23T00:40:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000568418 2024-02-23T00:40:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.864 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:40:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.043543+0000", 2024-02-23T00:40:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00061003399999999997 2024-02-23T00:40:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:40:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.044153+0000", 2024-02-23T00:40:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011287789999999999 2024-02-23T00:40:41.865 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045282+0000", 2024-02-23T00:40:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9601e-05 2024-02-23T00:40:41.866 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045351+0000", 2024-02-23T00:40:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763 2024-02-23T00:40:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.867 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26886.0:1 181.5s0 181:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e278)", 2024-02-23T00:40:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:40:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 412.78020859700001, 2024-02-23T00:40:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133, 2024-02-23T00:40:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.868 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26886", 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/134760323", 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.869 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:40:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:40:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.870 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046318+0000", 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9741999999999998e-05 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046338+0000", 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2594e-05 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046351+0000", 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5399999999999996e-06 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.871 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046359+0000", 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00098343299999999992 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047343+0000", 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9160000000000003e-06 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047347+0000", 2024-02-23T00:40:41.872 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0029936850000001 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050340+0000", 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014100300000000001 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050481+0000", 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038932099999999998 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050871+0000", 2024-02-23T00:40:41.873 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00045121199999999999 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.051322+0000", 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00089874699999999998 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052221+0000", 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1542999999999998e-05 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.874 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052272+0000", 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27003.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 367.76605489899998, 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568, 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.875 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27003", 2024-02-23T00:40:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3963489775", 2024-02-23T00:40:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:40:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.876 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:40:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.877 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:40:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060473+0000", 2024-02-23T00:40:41.878 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.417e-06 2024-02-23T00:40:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060479+0000", 2024-02-23T00:40:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8080000000000005e-06 2024-02-23T00:40:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.879 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060486+0000", 2024-02-23T00:40:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4359999999999999e-06 2024-02-23T00:40:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.880 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060493+0000", 2024-02-23T00:40:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6943999999999997e-05 2024-02-23T00:40:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060530+0000", 2024-02-23T00:40:41.881 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4229999999999999e-06 2024-02-23T00:40:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:40:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060532+0000", 2024-02-23T00:40:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.99955465700000001 2024-02-23T00:40:41.882 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060087+0000", 2024-02-23T00:40:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2388999999999999e-05 2024-02-23T00:40:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.883 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060179+0000", 2024-02-23T00:40:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00037435399999999997 2024-02-23T00:40:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:40:41.884 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060554+0000", 2024-02-23T00:40:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00057048899999999998 2024-02-23T00:40:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:40:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061124+0000", 2024-02-23T00:40:41.885 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00075843999999999998 2024-02-23T00:40:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061883+0000", 2024-02-23T00:40:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8547999999999998e-05 2024-02-23T00:40:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.886 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061921+0000", 2024-02-23T00:40:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568 2024-02-23T00:40:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.887 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37526.0:1 199.2s0 199:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-23T00:40:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:40:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 343.74830704999999, 2024-02-23T00:40:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.011307347, 2024-02-23T00:40:41.888 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37526", 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/4086862305", 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.889 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078221+0000", 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1790000000000008e-06 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078229+0000", 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1657e-05 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.890 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078240+0000", 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5909999999999996e-06 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078249+0000", 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011604219999999999 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.079409+0000", 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0110000000000001e-06 2024-02-23T00:40:41.891 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.079412+0000", 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0080054119999999 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087418+0000", 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014009999999999999 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.892 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087558+0000", 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00028455700000000001 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087842+0000", 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00026429900000000001 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.088107+0000", 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013807730000000001 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.893 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.089488+0000", 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2237000000000002e-05 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.089530+0000", 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.011307347 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.894 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6796 63.0 63:04a528c6:test-rados-api-smithi087-82091-72::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e458)", 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:37:32.149684+0000", 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 189.676845214, 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030257736, 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6796 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.895 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149684+0000", 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149684+0000", 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149683+0000", 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5489999999999995e-06 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.896 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149692+0000", 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3900000000000001e-05 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149706+0000", 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9720000000000003e-06 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149713+0000", 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2049000000000003e-05 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.897 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149765+0000", 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5885999999999996e-05 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149861+0000", 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025043958000000002 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.174905+0000", 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077345500000000004 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.898 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.175678+0000", 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041288080000000003 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.179807+0000", 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6297000000000001e-05 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.179853+0000", 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8549999999999998e-05 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.899 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.179942+0000", 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030257736 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:92 63.18 63:1d0125d1:test-rados-api-smithi087-82091-78::732be9d7c0f985174f3dbff4d0300cd89f82a6ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309667+0000", 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 141.51686287999999, 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030556142000000001, 2024-02-23T00:40:41.900 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309667+0000", 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309667+0000", 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.901 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309665+0000", 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.038e-06 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309673+0000", 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1586999999999999e-05 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309684+0000", 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0350000000000004e-06 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.902 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309692+0000", 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105009 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309797+0000", 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012772900000000001 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309924+0000", 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039668359999999996 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.903 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.313891+0000", 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014977572 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.328869+0000", 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00024878600000000003 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.329117+0000", 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011034924999999999 2024-02-23T00:40:41.904 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340152+0000", 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7619e-05 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340190+0000", 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2716000000000003e-05 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340223+0000", 2024-02-23T00:40:41.905 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030556142000000001 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:95 63.0 63:03bea9d3:test-rados-api-smithi087-82091-78::e98f1f7dcd528309122e8d5522249bd2aff3fd83:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 141.51681942600001, 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030136591000000001, 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.906 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0749999999999999e-06 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.907 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309714+0000", 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0990000000000004e-06 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309720+0000", 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6360000000000002e-06 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309724+0000", 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00097421499999999996 2024-02-23T00:40:41.908 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.310699+0000", 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012222900000000001 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.310821+0000", 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00020901700000000001 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311030+0000", 2024-02-23T00:40:41.909 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0075810119999999998 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.318611+0000", 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021088788000000001 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339700+0000", 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3208000000000003e-05 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.910 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339753+0000", 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3796999999999997e-05 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339847+0000", 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030136591000000001 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:96 63.13 63:cc844047:test-rados-api-smithi087-82091-78::4b02657dec822853c72fefcaa266fff97efe7078:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:40:41.911 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309730+0000", 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 141.516799819, 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031035795000000001, 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309730+0000", 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309730+0000", 2024-02-23T00:40:41.912 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309729+0000", 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2880000000000001e-06 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309732+0000", 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0950000000000003e-06 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309738+0000", 2024-02-23T00:40:41.913 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2869999999999998e-06 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309742+0000", 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4709000000000001e-05 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309797+0000", 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127417 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.914 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309924+0000", 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000217624 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,5", 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.310142+0000", 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0096070679999999999 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.319749+0000", 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0099226969999999994 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.915 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.329671+0000", 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011011036 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340682+0000", 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5581000000000003e-05 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340728+0000", 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7443e-05 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.916 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340765+0000", 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031035795000000001 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:100 63.0 63:0389d7c9:test-rados-api-smithi087-82091-78::b277a617178f08ca242dc7700d86bf23f0f3ea67:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309754+0000", 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 141.51677585900001, 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030390868000000001, 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.917 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309754+0000", 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309754+0000", 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309753+0000", 2024-02-23T00:40:41.918 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8859999999999998e-06 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309760+0000", 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1314000000000003e-05 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309821+0000", 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.899e-06 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.919 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309826+0000", 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001460601 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311287+0000", 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011481600000000001 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311402+0000", 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00021134800000000001 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.920 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311613+0000", 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070281809999999997 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.318641+0000", 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021369968 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340011+0000", 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6655e-05 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.921 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340058+0000", 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6738999999999997e-05 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340144+0000", 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030390868000000001 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.6.44:97 63.1d e483/330 63:bb7c65c5:test-rados-api-smithi087-82091-78::c5740e5a8abdfdd24173d1f0746ed7b8f004fd8c:head v 483'14, mlcod=340'12)", 2024-02-23T00:40:41.922 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.311169+0000", 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 141.51536033599999, 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028548314000000002, 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311169+0000", 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311169+0000", 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.923 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311168+0000", 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4179999999999997e-06 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311177+0000", 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8430000000000005e-06 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311186+0000", 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2770000000000002e-06 2024-02-23T00:40:41.924 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311193+0000", 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3401000000000001e-05 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311226+0000", 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1882000000000002e-05 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.925 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311248+0000", 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028438485999999999 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339687+0000", 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0916e-05 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339717+0000", 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028548314000000002 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.926 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:7287 221.f 221:f2233b45:test-rados-api-smithi087-82091-80::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e494)", 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.218861+0000", 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 130.60766860999999, 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036222770000000001, 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 7287 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.927 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218861+0000", 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218861+0000", 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218860+0000", 2024-02-23T00:40:41.928 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1700000000000002e-06 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218866+0000", 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0623e-05 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218877+0000", 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5566e-05 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.929 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218892+0000", 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5850999999999999e-05 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218938+0000", 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4792999999999999e-05 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.219033+0000", 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036050749999999999 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.930 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255084+0000", 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036222770000000001 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.7.51:63 63.10 e494/425 63:0bcb77f5:test-rados-api-smithi087-82091-80::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head v 494'38, mlcod=0'0)", 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 130.60394208100001, 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026196710000000002, 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3339999999999996e-06 2024-02-23T00:40:41.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222594+0000", 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0679999999999999e-06 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222602+0000", 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6450000000000004e-06 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222606+0000", 2024-02-23T00:40:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1100999999999997e-05 2024-02-23T00:40:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222657+0000", 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9842000000000001e-05 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222677+0000", 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026066840000000001 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.248744+0000", 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0111000000000003e-05 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.248784+0000", 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026196710000000002 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:7288 221.f 221:f2233b45:test-rados-api-smithi087-82091-80::foo:3 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e494)", 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.255388+0000", 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 130.571141213, 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041802895, 2024-02-23T00:40:41.937 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 7288 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255388+0000", 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255388+0000", 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255387+0000", 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5440000000000004e-06 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255393+0000", 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0249e-05 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255404+0000", 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7889999999999997e-06 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255411+0000", 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2548999999999999e-05 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255483+0000", 2024-02-23T00:40:41.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7108000000000006e-05 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255580+0000", 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041610951 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297191+0000", 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041802895 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.7.0:543 221.f e494/487 ondisk, result = 0)", 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.267731+0000", 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 130.558798776, 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029799866000000001, 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267731+0000", 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.942 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267731+0000", 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267730+0000", 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7200000000000003e-06 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267736+0000", 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1474000000000001e-05 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267747+0000", 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7590000000000003e-06 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267753+0000", 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029659342000000002 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297412+0000", 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.951e-06 2024-02-23T00:40:41.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297419+0000", 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111462 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297531+0000", 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029799866000000001 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "pg_backfill(finish_ack 4.1e e 515/515 lb MIN)", 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:52.183084+0000", 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 109.643445371, 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024878091000000001, 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183084+0000", 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183084+0000", 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183084+0000", 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.081e-06 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183093+0000", 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5926999999999998e-05 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183109+0000", 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8450000000000003e-06 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183117+0000", 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4298000000000002e-05 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183161+0000", 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1062e-05 2024-02-23T00:40:41.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183172+0000", 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024756987000000001 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207929+0000", 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3018e-05 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207942+0000", 2024-02-23T00:40:41.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0004999999999999e-05 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207962+0000", 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024878091000000001 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "pg_backfill(progress 4.16 e 515/515 lb 4:6b0d4d95:lc::lc.20:head)", 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:52.183360+0000", 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 109.64316980300001, 2024-02-23T00:40:41.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024472534000000001, 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183360+0000", 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183360+0000", 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183359+0000", 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6789999999999994e-06 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183368+0000", 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0594000000000001e-05 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183379+0000", 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2260000000000002e-06 2024-02-23T00:40:41.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183385+0000", 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9660000000000003e-05 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183425+0000", 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.695e-06 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183432+0000", 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024336048999999998 2024-02-23T00:40:41.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207768+0000", 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4369000000000001e-05 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207783+0000", 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9452e-05 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.954 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207832+0000", 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024472534000000001 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDPGPush(4.16 515/511 [PushOp(4:6d4431ed:lc::lc.7:head, version: 63'21, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d4431ed:lc::lc.7:head@63'21, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:52.183392+0000", 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 109.64313733900001, 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024578755000000001, 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:40:41.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183392+0000", 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183392+0000", 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183392+0000", 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0757999999999999e-05 2024-02-23T00:40:41.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183403+0000", 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7390000000000005e-06 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183411+0000", 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9799999999999998e-06 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183416+0000", 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6529999999999997e-05 2024-02-23T00:40:41.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183463+0000", 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5279999999999997e-06 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting_for_map not empty", 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183467+0000", 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024400908999999998 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207868+0000", 2024-02-23T00:40:41.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7630000000000003e-06 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207875+0000", 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5723000000000005e-05 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207971+0000", 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024578755000000001 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:40:41.959 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:40:41.960 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:40:41.960 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:41.989 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:42.113 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:42.116 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":949,"stamp":"2024-02-23T00:40:41.206742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5870,"num_objects":246,"num_object_clones":4,"num_object_copies":738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":25,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":1,"num_read":1688,"num_read_kb":1465,"num_write":732,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":7730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":874,"ondisk_log_size":874,"up":630,"acting":629,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":638,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2187180,"kb_used_data":53296,"kb_used_omap":0,"kb_used_meta":2133760,"kb_avail":654028884,"statfs":{"total":671965249536,"available":669725577216,"internally_reserved":0,"allocated":54575104,"data_stored":31308874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2184970240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.022688"},"pg_stats":[{"pgid":"237.1f","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396609+0000","last_change":"2024-02-23T00:40:34.207051+0000","last_active":"2024-02-23T00:40:39.396609+0000","last_peered":"2024-02-23T00:40:39.396609+0000","last_clean":"2024-02-23T00:40:39.396609+0000","last_became_active":"2024-02-23T00:40:34.206627+0000","last_became_peered":"2024-02-23T00:40:34.206627+0000","last_unstale":"2024-02-23T00:40:39.396609+0000","last_undegraded":"2024-02-23T00:40:39.396609+0000","last_fullsized":"2024-02-23T00:40:39.396609+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.1e","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204697+0000","last_change":"2024-02-23T00:40:34.205032+0000","last_active":"2024-02-23T00:40:39.204697+0000","last_peered":"2024-02-23T00:40:39.204697+0000","last_clean":"2024-02-23T00:40:39.204697+0000","last_became_active":"2024-02-23T00:40:34.204313+0000","last_became_peered":"2024-02-23T00:40:34.204313+0000","last_unstale":"2024-02-23T00:40:39.204697+0000","last_undegraded":"2024-02-23T00:40:39.204697+0000","last_fullsized":"2024-02-23T00:40:39.204697+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"237.1d","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204443+0000","last_change":"2024-02-23T00:40:34.204611+0000","last_active":"2024-02-23T00:40:39.204443+0000","last_peered":"2024-02-23T00:40:39.204443+0000","last_clean":"2024-02-23T00:40:39.204443+0000","last_became_active":"2024-02-23T00:40:34.204304+0000","last_became_peered":"2024-02-23T00:40:34.204304+0000","last_unstale":"2024-02-23T00:40:39.204443+0000","last_undegraded":"2024-02-23T00:40:39.204443+0000","last_fullsized":"2024-02-23T00:40:39.204443+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"237.1c","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203323+0000","last_change":"2024-02-23T00:40:34.194618+0000","last_active":"2024-02-23T00:40:39.203323+0000","last_peered":"2024-02-23T00:40:39.203323+0000","last_clean":"2024-02-23T00:40:39.203323+0000","last_became_active":"2024-02-23T00:40:34.194277+0000","last_became_peered":"2024-02-23T00:40:34.194277+0000","last_unstale":"2024-02-23T00:40:39.203323+0000","last_undegraded":"2024-02-23T00:40:39.203323+0000","last_fullsized":"2024-02-23T00:40:39.203323+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"237.1b","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207144+0000","last_change":"2024-02-23T00:40:34.208381+0000","last_active":"2024-02-23T00:40:39.207144+0000","last_peered":"2024-02-23T00:40:39.207144+0000","last_clean":"2024-02-23T00:40:39.207144+0000","last_became_active":"2024-02-23T00:40:34.208070+0000","last_became_peered":"2024-02-23T00:40:34.208070+0000","last_unstale":"2024-02-23T00:40:39.207144+0000","last_undegraded":"2024-02-23T00:40:39.207144+0000","last_fullsized":"2024-02-23T00:40:39.207144+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.1a","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395542+0000","last_change":"2024-02-23T00:40:34.203021+0000","last_active":"2024-02-23T00:40:39.395542+0000","last_peered":"2024-02-23T00:40:39.395542+0000","last_clean":"2024-02-23T00:40:39.395542+0000","last_became_active":"2024-02-23T00:40:34.202735+0000","last_became_peered":"2024-02-23T00:40:34.202735+0000","last_unstale":"2024-02-23T00:40:39.395542+0000","last_undegraded":"2024-02-23T00:40:39.395542+0000","last_fullsized":"2024-02-23T00:40:39.395542+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"237.19","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205241+0000","last_change":"2024-02-23T00:40:34.201929+0000","last_active":"2024-02-23T00:40:39.205241+0000","last_peered":"2024-02-23T00:40:39.205241+0000","last_clean":"2024-02-23T00:40:39.205241+0000","last_became_active":"2024-02-23T00:40:34.201633+0000","last_became_peered":"2024-02-23T00:40:34.201633+0000","last_unstale":"2024-02-23T00:40:39.205241+0000","last_undegraded":"2024-02-23T00:40:39.205241+0000","last_fullsized":"2024-02-23T00:40:39.205241+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"237.18","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394504+0000","last_change":"2024-02-23T00:40:34.205996+0000","last_active":"2024-02-23T00:40:39.394504+0000","last_peered":"2024-02-23T00:40:39.394504+0000","last_clean":"2024-02-23T00:40:39.394504+0000","last_became_active":"2024-02-23T00:40:34.205672+0000","last_became_peered":"2024-02-23T00:40:34.205672+0000","last_unstale":"2024-02-23T00:40:39.394504+0000","last_undegraded":"2024-02-23T00:40:39.394504+0000","last_fullsized":"2024-02-23T00:40:39.394504+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"237.17","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.397117+0000","last_change":"2024-02-23T00:40:34.203253+0000","last_active":"2024-02-23T00:40:39.397117+0000","last_peered":"2024-02-23T00:40:39.397117+0000","last_clean":"2024-02-23T00:40:39.397117+0000","last_became_active":"2024-02-23T00:40:34.202690+0000","last_became_peered":"2024-02-23T00:40:34.202690+0000","last_unstale":"2024-02-23T00:40:39.397117+0000","last_undegraded":"2024-02-23T00:40:39.397117+0000","last_fullsized":"2024-02-23T00:40:39.397117+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"237.16","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207990+0000","last_change":"2024-02-23T00:40:34.208438+0000","last_active":"2024-02-23T00:40:39.207990+0000","last_peered":"2024-02-23T00:40:39.207990+0000","last_clean":"2024-02-23T00:40:39.207990+0000","last_became_active":"2024-02-23T00:40:34.208151+0000","last_became_peered":"2024-02-23T00:40:34.208151+0000","last_unstale":"2024-02-23T00:40:39.207990+0000","last_undegraded":"2024-02-23T00:40:39.207990+0000","last_fullsized":"2024-02-23T00:40:39.207990+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.15","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203397+0000","last_change":"2024-02-23T00:40:34.189134+0000","last_active":"2024-02-23T00:40:39.203397+0000","last_peered":"2024-02-23T00:40:39.203397+0000","last_clean":"2024-02-23T00:40:39.203397+0000","last_became_active":"2024-02-23T00:40:34.188832+0000","last_became_peered":"2024-02-23T00:40:34.188832+0000","last_unstale":"2024-02-23T00:40:39.203397+0000","last_undegraded":"2024-02-23T00:40:39.203397+0000","last_fullsized":"2024-02-23T00:40:39.203397+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"237.14","version":"619'1","reported_seq":17,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205153+0000","last_change":"2024-02-23T00:40:34.204956+0000","last_active":"2024-02-23T00:40:39.205153+0000","last_peered":"2024-02-23T00:40:39.205153+0000","last_clean":"2024-02-23T00:40:39.205153+0000","last_became_active":"2024-02-23T00:40:34.204486+0000","last_became_peered":"2024-02-23T00:40:34.204486+0000","last_unstale":"2024-02-23T00:40:39.205153+0000","last_undegraded":"2024-02-23T00:40:39.205153+0000","last_fullsized":"2024-02-23T00:40:39.205153+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"237.13","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205336+0000","last_change":"2024-02-23T00:40:34.196688+0000","last_active":"2024-02-23T00:40:39.205336+0000","last_peered":"2024-02-23T00:40:39.205336+0000","last_clean":"2024-02-23T00:40:39.205336+0000","last_became_active":"2024-02-23T00:40:34.196411+0000","last_became_peered":"2024-02-23T00:40:34.196411+0000","last_unstale":"2024-02-23T00:40:39.205336+0000","last_undegraded":"2024-02-23T00:40:39.205336+0000","last_fullsized":"2024-02-23T00:40:39.205336+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"237.12","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206622+0000","last_change":"2024-02-23T00:40:34.204643+0000","last_active":"2024-02-23T00:40:39.206622+0000","last_peered":"2024-02-23T00:40:39.206622+0000","last_clean":"2024-02-23T00:40:39.206622+0000","last_became_active":"2024-02-23T00:40:34.204367+0000","last_became_peered":"2024-02-23T00:40:34.204367+0000","last_unstale":"2024-02-23T00:40:39.206622+0000","last_undegraded":"2024-02-23T00:40:39.206622+0000","last_fullsized":"2024-02-23T00:40:39.206622+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.11","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205317+0000","last_change":"2024-02-23T00:40:34.201818+0000","last_active":"2024-02-23T00:40:39.205317+0000","last_peered":"2024-02-23T00:40:39.205317+0000","last_clean":"2024-02-23T00:40:39.205317+0000","last_became_active":"2024-02-23T00:40:34.201327+0000","last_became_peered":"2024-02-23T00:40:34.201327+0000","last_unstale":"2024-02-23T00:40:39.205317+0000","last_undegraded":"2024-02-23T00:40:39.205317+0000","last_fullsized":"2024-02-23T00:40:39.205317+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"237.10","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.202949+0000","last_change":"2024-02-23T00:40:34.204615+0000","last_active":"2024-02-23T00:40:39.202949+0000","last_peered":"2024-02-23T00:40:39.202949+0000","last_clean":"2024-02-23T00:40:39.202949+0000","last_became_active":"2024-02-23T00:40:34.204329+0000","last_became_peered":"2024-02-23T00:40:34.204329+0000","last_unstale":"2024-02-23T00:40:39.202949+0000","last_undegraded":"2024-02-23T00:40:39.202949+0000","last_fullsized":"2024-02-23T00:40:39.202949+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"237.f","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.397710+0000","last_change":"2024-02-23T00:40:34.203421+0000","last_active":"2024-02-23T00:40:39.397710+0000","last_peered":"2024-02-23T00:40:39.397710+0000","last_clean":"2024-02-23T00:40:39.397710+0000","last_became_active":"2024-02-23T00:40:34.202988+0000","last_became_peered":"2024-02-23T00:40:34.202988+0000","last_unstale":"2024-02-23T00:40:39.397710+0000","last_undegraded":"2024-02-23T00:40:39.397710+0000","last_fullsized":"2024-02-23T00:40:39.397710+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"237.e","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206390+0000","last_change":"2024-02-23T00:40:34.207112+0000","last_active":"2024-02-23T00:40:39.206390+0000","last_peered":"2024-02-23T00:40:39.206390+0000","last_clean":"2024-02-23T00:40:39.206390+0000","last_became_active":"2024-02-23T00:40:34.206834+0000","last_became_peered":"2024-02-23T00:40:34.206834+0000","last_unstale":"2024-02-23T00:40:39.206390+0000","last_undegraded":"2024-02-23T00:40:39.206390+0000","last_fullsized":"2024-02-23T00:40:39.206390+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"237.d","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207455+0000","last_change":"2024-02-23T00:40:34.193396+0000","last_active":"2024-02-23T00:40:39.207455+0000","last_peered":"2024-02-23T00:40:39.207455+0000","last_clean":"2024-02-23T00:40:39.207455+0000","last_became_active":"2024-02-23T00:40:34.193096+0000","last_became_peered":"2024-02-23T00:40:34.193096+0000","last_unstale":"2024-02-23T00:40:39.207455+0000","last_undegraded":"2024-02-23T00:40:39.207455+0000","last_fullsized":"2024-02-23T00:40:39.207455+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.c","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203413+0000","last_change":"2024-02-23T00:40:34.189840+0000","last_active":"2024-02-23T00:40:39.203413+0000","last_peered":"2024-02-23T00:40:39.203413+0000","last_clean":"2024-02-23T00:40:39.203413+0000","last_became_active":"2024-02-23T00:40:34.189245+0000","last_became_peered":"2024-02-23T00:40:34.189245+0000","last_unstale":"2024-02-23T00:40:39.203413+0000","last_undegraded":"2024-02-23T00:40:39.203413+0000","last_fullsized":"2024-02-23T00:40:39.203413+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"237.b","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203955+0000","last_change":"2024-02-23T00:40:34.193906+0000","last_active":"2024-02-23T00:40:39.203955+0000","last_peered":"2024-02-23T00:40:39.203955+0000","last_clean":"2024-02-23T00:40:39.203955+0000","last_became_active":"2024-02-23T00:40:34.193580+0000","last_became_peered":"2024-02-23T00:40:34.193580+0000","last_unstale":"2024-02-23T00:40:39.203955+0000","last_undegraded":"2024-02-23T00:40:39.203955+0000","last_fullsized":"2024-02-23T00:40:39.203955+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.a","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394591+0000","last_change":"2024-02-23T00:40:34.195486+0000","last_active":"2024-02-23T00:40:39.394591+0000","last_peered":"2024-02-23T00:40:39.394591+0000","last_clean":"2024-02-23T00:40:39.394591+0000","last_became_active":"2024-02-23T00:40:34.195282+0000","last_became_peered":"2024-02-23T00:40:34.195282+0000","last_unstale":"2024-02-23T00:40:39.394591+0000","last_undegraded":"2024-02-23T00:40:39.394591+0000","last_fullsized":"2024-02-23T00:40:39.394591+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.9","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394278+0000","last_change":"2024-02-23T00:40:34.202542+0000","last_active":"2024-02-23T00:40:39.394278+0000","last_peered":"2024-02-23T00:40:39.394278+0000","last_clean":"2024-02-23T00:40:39.394278+0000","last_became_active":"2024-02-23T00:40:34.202245+0000","last_became_peered":"2024-02-23T00:40:34.202245+0000","last_unstale":"2024-02-23T00:40:39.394278+0000","last_undegraded":"2024-02-23T00:40:39.394278+0000","last_fullsized":"2024-02-23T00:40:39.394278+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"237.8","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.202714+0000","last_change":"2024-02-23T00:40:34.195591+0000","last_active":"2024-02-23T00:40:39.202714+0000","last_peered":"2024-02-23T00:40:39.202714+0000","last_clean":"2024-02-23T00:40:39.202714+0000","last_became_active":"2024-02-23T00:40:34.195282+0000","last_became_peered":"2024-02-23T00:40:34.195282+0000","last_unstale":"2024-02-23T00:40:39.202714+0000","last_undegraded":"2024-02-23T00:40:39.202714+0000","last_fullsized":"2024-02-23T00:40:39.202714+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.0","version":"619'1","reported_seq":17,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204520+0000","last_change":"2024-02-23T00:40:34.208408+0000","last_active":"2024-02-23T00:40:39.204520+0000","last_peered":"2024-02-23T00:40:39.204520+0000","last_clean":"2024-02-23T00:40:39.204520+0000","last_became_active":"2024-02-23T00:40:34.208106+0000","last_became_peered":"2024-02-23T00:40:34.208106+0000","last_unstale":"2024-02-23T00:40:39.204520+0000","last_undegraded":"2024-02-23T00:40:39.204520+0000","last_fullsized":"2024-02-23T00:40:39.204520+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.7","version":"0'0","reported_seq":32,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396935+0000","last_change":"2024-02-23T00:40:17.211770+0000","last_active":"2024-02-23T00:40:39.396935+0000","last_peered":"2024-02-23T00:40:39.396935+0000","last_clean":"2024-02-23T00:40:39.396935+0000","last_became_active":"2024-02-23T00:40:17.211405+0000","last_became_peered":"2024-02-23T00:40:17.211405+0000","last_unstale":"2024-02-23T00:40:39.396935+0000","last_undegraded":"2024-02-23T00:40:39.396935+0000","last_fullsized":"2024-02-23T00:40:39.396935+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"237.1","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205386+0000","last_change":"2024-02-23T00:40:34.195586+0000","last_active":"2024-02-23T00:40:39.205386+0000","last_peered":"2024-02-23T00:40:39.205386+0000","last_clean":"2024-02-23T00:40:39.205386+0000","last_became_active":"2024-02-23T00:40:34.195308+0000","last_became_peered":"2024-02-23T00:40:34.195308+0000","last_unstale":"2024-02-23T00:40:39.205386+0000","last_undegraded":"2024-02-23T00:40:39.205386+0000","last_fullsized":"2024-02-23T00:40:39.205386+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.6","version":"616'4","reported_seq":36,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395864+0000","last_change":"2024-02-23T00:40:17.215555+0000","last_active":"2024-02-23T00:40:39.395864+0000","last_peered":"2024-02-23T00:40:39.395864+0000","last_clean":"2024-02-23T00:40:39.395864+0000","last_became_active":"2024-02-23T00:40:17.215263+0000","last_became_peered":"2024-02-23T00:40:17.215263+0000","last_unstale":"2024-02-23T00:40:39.395864+0000","last_undegraded":"2024-02-23T00:40:39.395864+0000","last_fullsized":"2024-02-23T00:40:39.395864+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.2","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203107+0000","last_change":"2024-02-23T00:40:34.195809+0000","last_active":"2024-02-23T00:40:39.203107+0000","last_peered":"2024-02-23T00:40:39.203107+0000","last_clean":"2024-02-23T00:40:39.203107+0000","last_became_active":"2024-02-23T00:40:34.195306+0000","last_became_peered":"2024-02-23T00:40:34.195306+0000","last_unstale":"2024-02-23T00:40:39.203107+0000","last_undegraded":"2024-02-23T00:40:39.203107+0000","last_fullsized":"2024-02-23T00:40:39.203107+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.5","version":"616'3","reported_seq":40,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203492+0000","last_change":"2024-02-23T00:40:23.191317+0000","last_active":"2024-02-23T00:40:39.203492+0000","last_peered":"2024-02-23T00:40:39.203492+0000","last_clean":"2024-02-23T00:40:39.203492+0000","last_became_active":"2024-02-23T00:40:23.191006+0000","last_became_peered":"2024-02-23T00:40:23.191006+0000","last_unstale":"2024-02-23T00:40:39.203492+0000","last_undegraded":"2024-02-23T00:40:39.203492+0000","last_fullsized":"2024-02-23T00:40:39.203492+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"237.3","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207242+0000","last_change":"2024-02-23T00:40:34.193971+0000","last_active":"2024-02-23T00:40:39.207242+0000","last_peered":"2024-02-23T00:40:39.207242+0000","last_clean":"2024-02-23T00:40:39.207242+0000","last_became_active":"2024-02-23T00:40:34.193701+0000","last_became_peered":"2024-02-23T00:40:34.193701+0000","last_unstale":"2024-02-23T00:40:39.207242+0000","last_undegraded":"2024-02-23T00:40:39.207242+0000","last_fullsized":"2024-02-23T00:40:39.207242+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.4","version":"616'3","reported_seq":36,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204491+0000","last_change":"2024-02-23T00:40:17.215582+0000","last_active":"2024-02-23T00:40:39.204491+0000","last_peered":"2024-02-23T00:40:39.204491+0000","last_clean":"2024-02-23T00:40:39.204491+0000","last_became_active":"2024-02-23T00:40:17.215235+0000","last_became_peered":"2024-02-23T00:40:17.215235+0000","last_unstale":"2024-02-23T00:40:39.204491+0000","last_undegraded":"2024-02-23T00:40:39.204491+0000","last_fullsized":"2024-02-23T00:40:39.204491+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.4","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205204+0000","last_change":"2024-02-23T00:40:34.204774+0000","last_active":"2024-02-23T00:40:39.205204+0000","last_peered":"2024-02-23T00:40:39.205204+0000","last_clean":"2024-02-23T00:40:39.205204+0000","last_became_active":"2024-02-23T00:40:34.204076+0000","last_became_peered":"2024-02-23T00:40:34.204076+0000","last_unstale":"2024-02-23T00:40:39.205204+0000","last_undegraded":"2024-02-23T00:40:39.205204+0000","last_fullsized":"2024-02-23T00:40:39.205204+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":32,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205449+0000","last_change":"2024-02-23T00:40:17.202591+0000","last_active":"2024-02-23T00:40:39.205449+0000","last_peered":"2024-02-23T00:40:39.205449+0000","last_clean":"2024-02-23T00:40:39.205449+0000","last_became_active":"2024-02-23T00:40:17.202244+0000","last_became_peered":"2024-02-23T00:40:17.202244+0000","last_unstale":"2024-02-23T00:40:39.205449+0000","last_undegraded":"2024-02-23T00:40:39.205449+0000","last_fullsized":"2024-02-23T00:40:39.205449+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"237.5","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396205+0000","last_change":"2024-02-23T00:40:34.208269+0000","last_active":"2024-02-23T00:40:39.396205+0000","last_peered":"2024-02-23T00:40:39.396205+0000","last_clean":"2024-02-23T00:40:39.396205+0000","last_became_active":"2024-02-23T00:40:34.207980+0000","last_became_peered":"2024-02-23T00:40:34.207980+0000","last_unstale":"2024-02-23T00:40:39.396205+0000","last_undegraded":"2024-02-23T00:40:39.396205+0000","last_fullsized":"2024-02-23T00:40:39.396205+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.2","version":"0'0","reported_seq":38,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394681+0000","last_change":"2024-02-23T00:40:23.203813+0000","last_active":"2024-02-23T00:40:39.394681+0000","last_peered":"2024-02-23T00:40:39.394681+0000","last_clean":"2024-02-23T00:40:39.394681+0000","last_became_active":"2024-02-23T00:40:23.203522+0000","last_became_peered":"2024-02-23T00:40:23.203522+0000","last_unstale":"2024-02-23T00:40:39.394681+0000","last_undegraded":"2024-02-23T00:40:39.394681+0000","last_fullsized":"2024-02-23T00:40:39.394681+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.6","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395773+0000","last_change":"2024-02-23T00:40:34.206261+0000","last_active":"2024-02-23T00:40:39.395773+0000","last_peered":"2024-02-23T00:40:39.395773+0000","last_clean":"2024-02-23T00:40:39.395773+0000","last_became_active":"2024-02-23T00:40:34.206031+0000","last_became_peered":"2024-02-23T00:40:34.206031+0000","last_unstale":"2024-02-23T00:40:39.395773+0000","last_undegraded":"2024-02-23T00:40:39.395773+0000","last_fullsized":"2024-02-23T00:40:39.395773+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.1","version":"607'1","reported_seq":41,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204098+0000","last_change":"2024-02-23T00:40:23.203151+0000","last_active":"2024-02-23T00:40:39.204098+0000","last_peered":"2024-02-23T00:40:39.204098+0000","last_clean":"2024-02-23T00:40:39.204098+0000","last_became_active":"2024-02-23T00:40:23.202822+0000","last_became_peered":"2024-02-23T00:40:23.202822+0000","last_unstale":"2024-02-23T00:40:39.204098+0000","last_undegraded":"2024-02-23T00:40:39.204098+0000","last_fullsized":"2024-02-23T00:40:39.204098+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.7","version":"0'0","reported_seq":15,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207878+0000","last_change":"2024-02-23T00:40:34.197282+0000","last_active":"2024-02-23T00:40:39.207878+0000","last_peered":"2024-02-23T00:40:39.207878+0000","last_clean":"2024-02-23T00:40:39.207878+0000","last_became_active":"2024-02-23T00:40:34.196987+0000","last_became_peered":"2024-02-23T00:40:34.196987+0000","last_unstale":"2024-02-23T00:40:39.207878+0000","last_undegraded":"2024-02-23T00:40:39.207878+0000","last_fullsized":"2024-02-23T00:40:39.207878+0000","mapping_epoch":614,"log_start":"0'0","ondisk_log_start":"0'0","created":614,"last_epoch_clean":615,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:33.175971+0000","last_clean_scrub_stamp":"2024-02-23T00:40:33.175971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.0","version":"616'3","reported_seq":36,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203069+0000","last_change":"2024-02-23T00:40:17.201445+0000","last_active":"2024-02-23T00:40:39.203069+0000","last_peered":"2024-02-23T00:40:39.203069+0000","last_clean":"2024-02-23T00:40:39.203069+0000","last_became_active":"2024-02-23T00:40:17.201123+0000","last_became_peered":"2024-02-23T00:40:17.201123+0000","last_unstale":"2024-02-23T00:40:39.203069+0000","last_undegraded":"2024-02-23T00:40:39.203069+0000","last_fullsized":"2024-02-23T00:40:39.203069+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":433,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394762+0000","last_change":"2024-02-23T00:40:24.194411+0000","last_active":"2024-02-23T00:40:39.394762+0000","last_peered":"2024-02-23T00:40:39.394762+0000","last_clean":"2024-02-23T00:40:39.394762+0000","last_became_active":"2024-02-23T00:40:24.194076+0000","last_became_peered":"2024-02-23T00:40:24.194076+0000","last_unstale":"2024-02-23T00:40:39.394762+0000","last_undegraded":"2024-02-23T00:40:39.394762+0000","last_fullsized":"2024-02-23T00:40:39.394762+0000","mapping_epoch":604,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:09.026383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.026383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"606'17","reported_seq":104,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.215072+0000","last_change":"2024-02-23T00:40:24.201934+0000","last_active":"2024-02-23T00:40:40.215072+0000","last_peered":"2024-02-23T00:40:40.215072+0000","last_clean":"2024-02-23T00:40:40.215072+0000","last_became_active":"2024-02-23T00:40:24.197925+0000","last_became_peered":"2024-02-23T00:40:24.197925+0000","last_unstale":"2024-02-23T00:40:40.215072+0000","last_undegraded":"2024-02-23T00:40:40.215072+0000","last_fullsized":"2024-02-23T00:40:40.215072+0000","mapping_epoch":604,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"562'12","reported_seq":86,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.209712+0000","last_change":"2024-02-23T00:39:41.275972+0000","last_active":"2024-02-23T00:40:40.209712+0000","last_peered":"2024-02-23T00:40:40.209712+0000","last_clean":"2024-02-23T00:40:40.209712+0000","last_became_active":"2024-02-23T00:39:41.185268+0000","last_became_peered":"2024-02-23T00:39:41.185268+0000","last_unstale":"2024-02-23T00:40:40.209712+0000","last_undegraded":"2024-02-23T00:40:40.209712+0000","last_fullsized":"2024-02-23T00:40:40.209712+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":6,"last_scrub":"379'11","last_scrub_stamp":"2024-02-23T00:37:10.032341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:10.032341+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":69,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203061+0000","last_change":"2024-02-23T00:39:41.189853+0000","last_active":"2024-02-23T00:40:39.203061+0000","last_peered":"2024-02-23T00:40:39.203061+0000","last_clean":"2024-02-23T00:40:39.203061+0000","last_became_active":"2024-02-23T00:39:41.189429+0000","last_became_peered":"2024-02-23T00:39:41.189429+0000","last_unstale":"2024-02-23T00:40:39.203061+0000","last_undegraded":"2024-02-23T00:40:39.203061+0000","last_fullsized":"2024-02-23T00:40:39.203061+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:38.337256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:34.152643+0000","last_clean_scrub_stamp":"2024-02-23T00:39:38.337256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":69,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395786+0000","last_change":"2024-02-23T00:39:41.189014+0000","last_active":"2024-02-23T00:40:39.395786+0000","last_peered":"2024-02-23T00:40:39.395786+0000","last_clean":"2024-02-23T00:40:39.395786+0000","last_became_active":"2024-02-23T00:39:41.188508+0000","last_became_peered":"2024-02-23T00:39:41.188508+0000","last_unstale":"2024-02-23T00:40:39.395786+0000","last_undegraded":"2024-02-23T00:40:39.395786+0000","last_fullsized":"2024-02-23T00:40:39.395786+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:28.710178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:28.710178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"571'11","reported_seq":793,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.263442+0000","last_change":"2024-02-23T00:39:49.195375+0000","last_active":"2024-02-23T00:40:40.263442+0000","last_peered":"2024-02-23T00:40:40.263442+0000","last_clean":"2024-02-23T00:40:40.263442+0000","last_became_active":"2024-02-23T00:39:49.195036+0000","last_became_peered":"2024-02-23T00:39:49.195036+0000","last_unstale":"2024-02-23T00:40:40.263442+0000","last_undegraded":"2024-02-23T00:40:40.263442+0000","last_fullsized":"2024-02-23T00:40:40.263442+0000","mapping_epoch":58,"log_start":"379'9","ondisk_log_start":"379'9","created":58,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'9","last_scrub_stamp":"2024-02-23T00:37:05.955707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.955707+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":709,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205124+0000","last_change":"2024-02-23T00:40:13.948977+0000","last_active":"2024-02-23T00:40:39.205124+0000","last_peered":"2024-02-23T00:40:39.205124+0000","last_clean":"2024-02-23T00:40:39.205124+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:40:39.205124+0000","last_undegraded":"2024-02-23T00:40:39.205124+0000","last_fullsized":"2024-02-23T00:40:39.205124+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:40:13.948915+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:40:13.948915+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":633,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204155+0000","last_change":"2024-02-23T00:39:39.337422+0000","last_active":"2024-02-23T00:40:39.204155+0000","last_peered":"2024-02-23T00:40:39.204155+0000","last_clean":"2024-02-23T00:40:39.204155+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:40:39.204155+0000","last_undegraded":"2024-02-23T00:40:39.204155+0000","last_fullsized":"2024-02-23T00:40:39.204155+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.337303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.337303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":609,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205810+0000","last_change":"2024-02-23T00:40:29.205670+0000","last_active":"2024-02-23T00:40:39.205810+0000","last_peered":"2024-02-23T00:40:39.205810+0000","last_clean":"2024-02-23T00:40:39.205810+0000","last_became_active":"2024-02-23T00:40:29.205312+0000","last_became_peered":"2024-02-23T00:40:29.205312+0000","last_unstale":"2024-02-23T00:40:39.205810+0000","last_undegraded":"2024-02-23T00:40:39.205810+0000","last_fullsized":"2024-02-23T00:40:39.205810+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:21.441958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.441958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":556,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207984+0000","last_change":"2024-02-23T00:40:23.194839+0000","last_active":"2024-02-23T00:40:39.207984+0000","last_peered":"2024-02-23T00:40:39.207984+0000","last_clean":"2024-02-23T00:40:39.207984+0000","last_became_active":"2024-02-23T00:40:23.194357+0000","last_became_peered":"2024-02-23T00:40:23.194357+0000","last_unstale":"2024-02-23T00:40:39.207984+0000","last_undegraded":"2024-02-23T00:40:39.207984+0000","last_fullsized":"2024-02-23T00:40:39.207984+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:48.468159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:32:48.468159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":1083,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.208525+0000","last_change":"2024-02-23T00:39:36.787515+0000","last_active":"2024-02-23T00:40:40.208525+0000","last_peered":"2024-02-23T00:40:40.208525+0000","last_clean":"2024-02-23T00:40:40.208525+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:40:40.208525+0000","last_undegraded":"2024-02-23T00:40:40.208525+0000","last_fullsized":"2024-02-23T00:40:40.208525+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:39:36.787450+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:39:36.787450+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":481,"num_read_kb":481,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":297,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207418+0000","last_change":"2024-02-23T00:40:26.207420+0000","last_active":"2024-02-23T00:40:39.207418+0000","last_peered":"2024-02-23T00:40:39.207418+0000","last_clean":"2024-02-23T00:40:39.207418+0000","last_became_active":"2024-02-23T00:40:26.207084+0000","last_became_peered":"2024-02-23T00:40:26.207084+0000","last_unstale":"2024-02-23T00:40:39.207418+0000","last_undegraded":"2024-02-23T00:40:39.207418+0000","last_fullsized":"2024-02-23T00:40:39.207418+0000","mapping_epoch":606,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":613,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208152+0000","last_change":"2024-02-23T00:40:08.076879+0000","last_active":"2024-02-23T00:40:39.208152+0000","last_peered":"2024-02-23T00:40:39.208152+0000","last_clean":"2024-02-23T00:40:39.208152+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:40:39.208152+0000","last_undegraded":"2024-02-23T00:40:39.208152+0000","last_fullsized":"2024-02-23T00:40:39.208152+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:08.076820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:08.076820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":620,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204505+0000","last_change":"2024-02-23T00:39:20.396906+0000","last_active":"2024-02-23T00:40:39.204505+0000","last_peered":"2024-02-23T00:40:39.204505+0000","last_clean":"2024-02-23T00:40:39.204505+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:40:39.204505+0000","last_undegraded":"2024-02-23T00:40:39.204505+0000","last_fullsized":"2024-02-23T00:40:39.204505+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:20.396815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:39:20.396815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":235,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394485+0000","last_change":"2024-02-23T00:40:33.189003+0000","last_active":"2024-02-23T00:40:39.394485+0000","last_peered":"2024-02-23T00:40:39.394485+0000","last_clean":"2024-02-23T00:40:39.394485+0000","last_became_active":"2024-02-23T00:40:33.188712+0000","last_became_peered":"2024-02-23T00:40:33.188712+0000","last_unstale":"2024-02-23T00:40:39.394485+0000","last_undegraded":"2024-02-23T00:40:39.394485+0000","last_fullsized":"2024-02-23T00:40:39.394485+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":614,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:25.632725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:40:25.632725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":546,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395203+0000","last_change":"2024-02-23T00:39:35.342488+0000","last_active":"2024-02-23T00:40:39.395203+0000","last_peered":"2024-02-23T00:40:39.395203+0000","last_clean":"2024-02-23T00:40:39.395203+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:40:39.395203+0000","last_undegraded":"2024-02-23T00:40:39.395203+0000","last_fullsized":"2024-02-23T00:40:39.395203+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:39:35.342427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":725,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395143+0000","last_change":"2024-02-23T00:38:40.424549+0000","last_active":"2024-02-23T00:40:39.395143+0000","last_peered":"2024-02-23T00:40:39.395143+0000","last_clean":"2024-02-23T00:40:39.395143+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:40:39.395143+0000","last_undegraded":"2024-02-23T00:40:39.395143+0000","last_fullsized":"2024-02-23T00:40:39.395143+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:40.424490+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.424490+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":615,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205646+0000","last_change":"2024-02-23T00:40:23.196078+0000","last_active":"2024-02-23T00:40:39.205646+0000","last_peered":"2024-02-23T00:40:39.205646+0000","last_clean":"2024-02-23T00:40:39.205646+0000","last_became_active":"2024-02-23T00:40:23.195689+0000","last_became_peered":"2024-02-23T00:40:23.195689+0000","last_unstale":"2024-02-23T00:40:39.205646+0000","last_undegraded":"2024-02-23T00:40:39.205646+0000","last_fullsized":"2024-02-23T00:40:39.205646+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:50.194656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:50.194656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":611,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203244+0000","last_change":"2024-02-23T00:40:25.149933+0000","last_active":"2024-02-23T00:40:39.203244+0000","last_peered":"2024-02-23T00:40:39.203244+0000","last_clean":"2024-02-23T00:40:39.203244+0000","last_became_active":"2024-02-23T00:40:23.194895+0000","last_became_peered":"2024-02-23T00:40:23.194895+0000","last_unstale":"2024-02-23T00:40:39.203244+0000","last_undegraded":"2024-02-23T00:40:39.203244+0000","last_fullsized":"2024-02-23T00:40:39.203244+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:25.149871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:40:25.149871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":627,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394748+0000","last_change":"2024-02-23T00:40:37.197309+0000","last_active":"2024-02-23T00:40:39.394748+0000","last_peered":"2024-02-23T00:40:39.394748+0000","last_clean":"2024-02-23T00:40:39.394748+0000","last_became_active":"2024-02-23T00:40:37.197010+0000","last_became_peered":"2024-02-23T00:40:37.197010+0000","last_unstale":"2024-02-23T00:40:39.394748+0000","last_undegraded":"2024-02-23T00:40:39.394748+0000","last_fullsized":"2024-02-23T00:40:39.394748+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.226380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.226380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":213,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205121+0000","last_change":"2024-02-23T00:40:23.197061+0000","last_active":"2024-02-23T00:40:39.205121+0000","last_peered":"2024-02-23T00:40:39.205121+0000","last_clean":"2024-02-23T00:40:39.205121+0000","last_became_active":"2024-02-23T00:40:23.196188+0000","last_became_peered":"2024-02-23T00:40:23.196188+0000","last_unstale":"2024-02-23T00:40:39.205121+0000","last_undegraded":"2024-02-23T00:40:39.205121+0000","last_fullsized":"2024-02-23T00:40:39.205121+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.205312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.205312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":608,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395449+0000","last_change":"2024-02-23T00:38:44.442795+0000","last_active":"2024-02-23T00:40:39.395449+0000","last_peered":"2024-02-23T00:40:39.395449+0000","last_clean":"2024-02-23T00:40:39.395449+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:40:39.395449+0000","last_undegraded":"2024-02-23T00:40:39.395449+0000","last_fullsized":"2024-02-23T00:40:39.395449+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:44.442733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.442733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":683,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204271+0000","last_change":"2024-02-23T00:39:37.346021+0000","last_active":"2024-02-23T00:40:39.204271+0000","last_peered":"2024-02-23T00:40:39.204271+0000","last_clean":"2024-02-23T00:40:39.204271+0000","last_became_active":"2024-02-23T00:38:19.176922+0000","last_became_peered":"2024-02-23T00:38:19.176922+0000","last_unstale":"2024-02-23T00:40:39.204271+0000","last_undegraded":"2024-02-23T00:40:39.204271+0000","last_fullsized":"2024-02-23T00:40:39.204271+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:39:37.345922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:39:37.345922+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":597,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395699+0000","last_change":"2024-02-23T00:38:18.055894+0000","last_active":"2024-02-23T00:40:39.395699+0000","last_peered":"2024-02-23T00:40:39.395699+0000","last_clean":"2024-02-23T00:40:39.395699+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:40:39.395699+0000","last_undegraded":"2024-02-23T00:40:39.395699+0000","last_fullsized":"2024-02-23T00:40:39.395699+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:18.055803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:18.055803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":596,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208180+0000","last_change":"2024-02-23T00:40:23.200141+0000","last_active":"2024-02-23T00:40:39.208180+0000","last_peered":"2024-02-23T00:40:39.208180+0000","last_clean":"2024-02-23T00:40:39.208180+0000","last_became_active":"2024-02-23T00:40:23.199844+0000","last_became_peered":"2024-02-23T00:40:23.199844+0000","last_unstale":"2024-02-23T00:40:39.208180+0000","last_undegraded":"2024-02-23T00:40:39.208180+0000","last_fullsized":"2024-02-23T00:40:39.208180+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.605664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.605664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":49,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206597+0000","last_change":"2024-02-23T00:40:01.300115+0000","last_active":"2024-02-23T00:40:39.206597+0000","last_peered":"2024-02-23T00:40:39.206597+0000","last_clean":"2024-02-23T00:40:39.206597+0000","last_became_active":"2024-02-23T00:40:01.299780+0000","last_became_peered":"2024-02-23T00:40:01.299780+0000","last_unstale":"2024-02-23T00:40:39.206597+0000","last_undegraded":"2024-02-23T00:40:39.206597+0000","last_fullsized":"2024-02-23T00:40:39.206597+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:27.413549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:39:27.413549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":571,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.397354+0000","last_change":"2024-02-23T00:39:27.813080+0000","last_active":"2024-02-23T00:40:39.397354+0000","last_peered":"2024-02-23T00:40:39.397354+0000","last_clean":"2024-02-23T00:40:39.397354+0000","last_became_active":"2024-02-23T00:38:49.194450+0000","last_became_peered":"2024-02-23T00:38:49.194450+0000","last_unstale":"2024-02-23T00:40:39.397354+0000","last_undegraded":"2024-02-23T00:40:39.397354+0000","last_fullsized":"2024-02-23T00:40:39.397354+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:27.813005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:27.813005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":129,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204469+0000","last_change":"2024-02-23T00:40:23.206602+0000","last_active":"2024-02-23T00:40:39.204469+0000","last_peered":"2024-02-23T00:40:39.204469+0000","last_clean":"2024-02-23T00:40:39.204469+0000","last_became_active":"2024-02-23T00:40:23.205794+0000","last_became_peered":"2024-02-23T00:40:23.205794+0000","last_unstale":"2024-02-23T00:40:39.204469+0000","last_undegraded":"2024-02-23T00:40:39.204469+0000","last_fullsized":"2024-02-23T00:40:39.204469+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":722,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203610+0000","last_change":"2024-02-23T00:38:54.172406+0000","last_active":"2024-02-23T00:40:39.203610+0000","last_peered":"2024-02-23T00:40:39.203610+0000","last_clean":"2024-02-23T00:40:39.203610+0000","last_became_active":"2024-02-23T00:38:54.171993+0000","last_became_peered":"2024-02-23T00:38:54.171993+0000","last_unstale":"2024-02-23T00:40:39.203610+0000","last_undegraded":"2024-02-23T00:40:39.203610+0000","last_fullsized":"2024-02-23T00:40:39.203610+0000","mapping_epoch":516,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'15","last_scrub_stamp":"2024-02-23T00:37:09.133471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:09.133471+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":626,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207575+0000","last_change":"2024-02-23T00:39:40.287714+0000","last_active":"2024-02-23T00:40:39.207575+0000","last_peered":"2024-02-23T00:40:39.207575+0000","last_clean":"2024-02-23T00:40:39.207575+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:40:39.207575+0000","last_undegraded":"2024-02-23T00:40:39.207575+0000","last_fullsized":"2024-02-23T00:40:39.207575+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.287611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.287611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":770,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.208273+0000","last_change":"2024-02-23T00:38:35.337273+0000","last_active":"2024-02-23T00:40:40.208273+0000","last_peered":"2024-02-23T00:40:40.208273+0000","last_clean":"2024-02-23T00:40:40.208273+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:40:40.208273+0000","last_undegraded":"2024-02-23T00:40:40.208273+0000","last_fullsized":"2024-02-23T00:40:40.208273+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:38:35.337170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.337170+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":49,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203076+0000","last_change":"2024-02-23T00:40:01.299027+0000","last_active":"2024-02-23T00:40:39.203076+0000","last_peered":"2024-02-23T00:40:39.203076+0000","last_clean":"2024-02-23T00:40:39.203076+0000","last_became_active":"2024-02-23T00:40:01.298668+0000","last_became_peered":"2024-02-23T00:40:01.298668+0000","last_unstale":"2024-02-23T00:40:39.203076+0000","last_undegraded":"2024-02-23T00:40:39.203076+0000","last_fullsized":"2024-02-23T00:40:39.203076+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.135802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.135802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":546,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205104+0000","last_change":"2024-02-23T00:38:34.097327+0000","last_active":"2024-02-23T00:40:39.205104+0000","last_peered":"2024-02-23T00:40:39.205104+0000","last_clean":"2024-02-23T00:40:39.205104+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:40:39.205104+0000","last_undegraded":"2024-02-23T00:40:39.205104+0000","last_fullsized":"2024-02-23T00:40:39.205104+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:34.097243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.097243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":124,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205155+0000","last_change":"2024-02-23T00:40:15.208236+0000","last_active":"2024-02-23T00:40:39.205155+0000","last_peered":"2024-02-23T00:40:39.205155+0000","last_clean":"2024-02-23T00:40:39.205155+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:40:39.205155+0000","last_undegraded":"2024-02-23T00:40:39.205155+0000","last_fullsized":"2024-02-23T00:40:39.205155+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:15.208124+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.208124+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":780,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208180+0000","last_change":"2024-02-23T00:39:25.200274+0000","last_active":"2024-02-23T00:40:39.208180+0000","last_peered":"2024-02-23T00:40:39.208180+0000","last_clean":"2024-02-23T00:40:39.208180+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:40:39.208180+0000","last_undegraded":"2024-02-23T00:40:39.208180+0000","last_fullsized":"2024-02-23T00:40:39.208180+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":610,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206984+0000","last_change":"2024-02-23T00:39:37.339984+0000","last_active":"2024-02-23T00:40:39.206984+0000","last_peered":"2024-02-23T00:40:39.206984+0000","last_clean":"2024-02-23T00:40:39.206984+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:40:39.206984+0000","last_undegraded":"2024-02-23T00:40:39.206984+0000","last_fullsized":"2024-02-23T00:40:39.206984+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:37.339877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:39:37.339877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"606'10","reported_seq":779,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.206093+0000","last_change":"2024-02-23T00:40:23.280090+0000","last_active":"2024-02-23T00:40:40.206093+0000","last_peered":"2024-02-23T00:40:40.206093+0000","last_clean":"2024-02-23T00:40:40.206093+0000","last_became_active":"2024-02-23T00:40:23.191923+0000","last_became_peered":"2024-02-23T00:40:23.191923+0000","last_unstale":"2024-02-23T00:40:40.206093+0000","last_undegraded":"2024-02-23T00:40:40.206093+0000","last_fullsized":"2024-02-23T00:40:40.206093+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:34:59.630171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:59.630171+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":79,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396245+0000","last_change":"2024-02-23T00:40:09.663488+0000","last_active":"2024-02-23T00:40:39.396245+0000","last_peered":"2024-02-23T00:40:39.396245+0000","last_clean":"2024-02-23T00:40:39.396245+0000","last_became_active":"2024-02-23T00:40:09.200996+0000","last_became_peered":"2024-02-23T00:40:09.200996+0000","last_unstale":"2024-02-23T00:40:39.396245+0000","last_undegraded":"2024-02-23T00:40:39.396245+0000","last_fullsized":"2024-02-23T00:40:39.396245+0000","mapping_epoch":581,"log_start":"62'1","ondisk_log_start":"62'1","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:40:09.663366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:09.663366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":537,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207940+0000","last_change":"2024-02-23T00:38:16.396902+0000","last_active":"2024-02-23T00:40:39.207940+0000","last_peered":"2024-02-23T00:40:39.207940+0000","last_clean":"2024-02-23T00:40:39.207940+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:40:39.207940+0000","last_undegraded":"2024-02-23T00:40:39.207940+0000","last_fullsized":"2024-02-23T00:40:39.207940+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.396824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":608,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203110+0000","last_change":"2024-02-23T00:38:08.068624+0000","last_active":"2024-02-23T00:40:39.203110+0000","last_peered":"2024-02-23T00:40:39.203110+0000","last_clean":"2024-02-23T00:40:39.203110+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:40:39.203110+0000","last_undegraded":"2024-02-23T00:40:39.203110+0000","last_fullsized":"2024-02-23T00:40:39.203110+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:08.068559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:08.068559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":697,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204833+0000","last_change":"2024-02-23T00:40:07.359576+0000","last_active":"2024-02-23T00:40:39.204833+0000","last_peered":"2024-02-23T00:40:39.204833+0000","last_clean":"2024-02-23T00:40:39.204833+0000","last_became_active":"2024-02-23T00:38:19.182222+0000","last_became_peered":"2024-02-23T00:38:19.182222+0000","last_unstale":"2024-02-23T00:40:39.204833+0000","last_undegraded":"2024-02-23T00:40:39.204833+0000","last_fullsized":"2024-02-23T00:40:39.204833+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:40:07.359519+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:40:07.359519+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":618,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395030+0000","last_change":"2024-02-23T00:40:23.295585+0000","last_active":"2024-02-23T00:40:39.395030+0000","last_peered":"2024-02-23T00:40:39.395030+0000","last_clean":"2024-02-23T00:40:39.395030+0000","last_became_active":"2024-02-23T00:40:23.196664+0000","last_became_peered":"2024-02-23T00:40:23.196664+0000","last_unstale":"2024-02-23T00:40:39.395030+0000","last_undegraded":"2024-02-23T00:40:39.395030+0000","last_fullsized":"2024-02-23T00:40:39.395030+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:33:50.601484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:50.601484+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":583,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206458+0000","last_change":"2024-02-23T00:38:37.881705+0000","last_active":"2024-02-23T00:40:39.206458+0000","last_peered":"2024-02-23T00:40:39.206458+0000","last_clean":"2024-02-23T00:40:39.206458+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:40:39.206458+0000","last_undegraded":"2024-02-23T00:40:39.206458+0000","last_fullsized":"2024-02-23T00:40:39.206458+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:37.881618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.881618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"78'2","reported_seq":67,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208360+0000","last_change":"2024-02-23T00:40:13.179766+0000","last_active":"2024-02-23T00:40:39.208360+0000","last_peered":"2024-02-23T00:40:39.208360+0000","last_clean":"2024-02-23T00:40:39.208360+0000","last_became_active":"2024-02-23T00:40:13.179322+0000","last_became_peered":"2024-02-23T00:40:13.179322+0000","last_unstale":"2024-02-23T00:40:39.208360+0000","last_undegraded":"2024-02-23T00:40:39.208360+0000","last_fullsized":"2024-02-23T00:40:39.208360+0000","mapping_epoch":581,"log_start":"78'2","ondisk_log_start":"78'2","created":60,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.478470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.478470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":560,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208346+0000","last_change":"2024-02-23T00:40:19.126448+0000","last_active":"2024-02-23T00:40:39.208346+0000","last_peered":"2024-02-23T00:40:39.208346+0000","last_clean":"2024-02-23T00:40:39.208346+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:40:39.208346+0000","last_undegraded":"2024-02-23T00:40:39.208346+0000","last_fullsized":"2024-02-23T00:40:39.208346+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.126375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":213,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208305+0000","last_change":"2024-02-23T00:40:23.194620+0000","last_active":"2024-02-23T00:40:39.208305+0000","last_peered":"2024-02-23T00:40:39.208305+0000","last_clean":"2024-02-23T00:40:39.208305+0000","last_became_active":"2024-02-23T00:40:23.194164+0000","last_became_peered":"2024-02-23T00:40:23.194164+0000","last_unstale":"2024-02-23T00:40:39.208305+0000","last_undegraded":"2024-02-23T00:40:39.208305+0000","last_fullsized":"2024-02-23T00:40:39.208305+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:42.114807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:35:42.114807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":119,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203358+0000","last_change":"2024-02-23T00:38:49.222977+0000","last_active":"2024-02-23T00:40:39.203358+0000","last_peered":"2024-02-23T00:40:39.203358+0000","last_clean":"2024-02-23T00:40:39.203358+0000","last_became_active":"2024-02-23T00:38:49.216657+0000","last_became_peered":"2024-02-23T00:38:49.216657+0000","last_unstale":"2024-02-23T00:40:39.203358+0000","last_undegraded":"2024-02-23T00:40:39.203358+0000","last_fullsized":"2024-02-23T00:40:39.203358+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":654,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203723+0000","last_change":"2024-02-23T00:40:15.217300+0000","last_active":"2024-02-23T00:40:39.203723+0000","last_peered":"2024-02-23T00:40:39.203723+0000","last_clean":"2024-02-23T00:40:39.203723+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:40:39.203723+0000","last_undegraded":"2024-02-23T00:40:39.203723+0000","last_fullsized":"2024-02-23T00:40:39.203723+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'34","last_scrub_stamp":"2024-02-23T00:40:15.217223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.217223+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":598,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396637+0000","last_change":"2024-02-23T00:38:48.494288+0000","last_active":"2024-02-23T00:40:39.396637+0000","last_peered":"2024-02-23T00:40:39.396637+0000","last_clean":"2024-02-23T00:40:39.396637+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:40:39.396637+0000","last_undegraded":"2024-02-23T00:40:39.396637+0000","last_fullsized":"2024-02-23T00:40:39.396637+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:48.494224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:48.494224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":596,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396034+0000","last_change":"2024-02-23T00:39:36.902074+0000","last_active":"2024-02-23T00:40:39.396034+0000","last_peered":"2024-02-23T00:40:39.396034+0000","last_clean":"2024-02-23T00:40:39.396034+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:27:23.182229+0000","last_unstale":"2024-02-23T00:40:39.396034+0000","last_undegraded":"2024-02-23T00:40:39.396034+0000","last_fullsized":"2024-02-23T00:40:39.396034+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:36.902015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:36.902015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":571,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207516+0000","last_change":"2024-02-23T00:40:23.198210+0000","last_active":"2024-02-23T00:40:39.207516+0000","last_peered":"2024-02-23T00:40:39.207516+0000","last_clean":"2024-02-23T00:40:39.207516+0000","last_became_active":"2024-02-23T00:40:23.197778+0000","last_became_peered":"2024-02-23T00:40:23.197778+0000","last_unstale":"2024-02-23T00:40:39.207516+0000","last_undegraded":"2024-02-23T00:40:39.207516+0000","last_fullsized":"2024-02-23T00:40:39.207516+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:22.420380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.420380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":134,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204059+0000","last_change":"2024-02-23T00:40:23.200251+0000","last_active":"2024-02-23T00:40:39.204059+0000","last_peered":"2024-02-23T00:40:39.204059+0000","last_clean":"2024-02-23T00:40:39.204059+0000","last_became_active":"2024-02-23T00:40:23.199949+0000","last_became_peered":"2024-02-23T00:40:23.199949+0000","last_unstale":"2024-02-23T00:40:39.204059+0000","last_undegraded":"2024-02-23T00:40:39.204059+0000","last_fullsized":"2024-02-23T00:40:39.204059+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:33.344870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:33.344870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":628,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203989+0000","last_change":"2024-02-23T00:40:19.205802+0000","last_active":"2024-02-23T00:40:39.203989+0000","last_peered":"2024-02-23T00:40:39.203989+0000","last_clean":"2024-02-23T00:40:39.203989+0000","last_became_active":"2024-02-23T00:38:49.228425+0000","last_became_peered":"2024-02-23T00:38:49.228425+0000","last_unstale":"2024-02-23T00:40:39.203989+0000","last_undegraded":"2024-02-23T00:40:39.203989+0000","last_fullsized":"2024-02-23T00:40:39.203989+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.205739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.205739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":156,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203268+0000","last_change":"2024-02-23T00:40:23.208576+0000","last_active":"2024-02-23T00:40:39.203268+0000","last_peered":"2024-02-23T00:40:39.203268+0000","last_clean":"2024-02-23T00:40:39.203268+0000","last_became_active":"2024-02-23T00:40:23.208225+0000","last_became_peered":"2024-02-23T00:40:23.208225+0000","last_unstale":"2024-02-23T00:40:39.203268+0000","last_undegraded":"2024-02-23T00:40:39.203268+0000","last_fullsized":"2024-02-23T00:40:39.203268+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:39:28.324185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.324185+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":590,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208375+0000","last_change":"2024-02-23T00:38:14.343844+0000","last_active":"2024-02-23T00:40:39.208375+0000","last_peered":"2024-02-23T00:40:39.208375+0000","last_clean":"2024-02-23T00:40:39.208375+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:40:39.208375+0000","last_undegraded":"2024-02-23T00:40:39.208375+0000","last_fullsized":"2024-02-23T00:40:39.208375+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:14.343770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:14.343770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":611,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203628+0000","last_change":"2024-02-23T00:38:49.225300+0000","last_active":"2024-02-23T00:40:39.203628+0000","last_peered":"2024-02-23T00:40:39.203628+0000","last_clean":"2024-02-23T00:40:39.203628+0000","last_became_active":"2024-02-23T00:38:49.224892+0000","last_became_peered":"2024-02-23T00:38:49.224892+0000","last_unstale":"2024-02-23T00:40:39.203628+0000","last_undegraded":"2024-02-23T00:40:39.203628+0000","last_fullsized":"2024-02-23T00:40:39.203628+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.082000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.082000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":119,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203756+0000","last_change":"2024-02-23T00:38:49.176838+0000","last_active":"2024-02-23T00:40:39.203756+0000","last_peered":"2024-02-23T00:40:39.203756+0000","last_clean":"2024-02-23T00:40:39.203756+0000","last_became_active":"2024-02-23T00:38:49.175801+0000","last_became_peered":"2024-02-23T00:38:49.175801+0000","last_unstale":"2024-02-23T00:40:39.203756+0000","last_undegraded":"2024-02-23T00:40:39.203756+0000","last_fullsized":"2024-02-23T00:40:39.203756+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.977282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.977282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":858,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207849+0000","last_change":"2024-02-23T00:40:17.043400+0000","last_active":"2024-02-23T00:40:39.207849+0000","last_peered":"2024-02-23T00:40:39.207849+0000","last_clean":"2024-02-23T00:40:39.207849+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:40:39.207849+0000","last_undegraded":"2024-02-23T00:40:39.207849+0000","last_fullsized":"2024-02-23T00:40:39.207849+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:17.043322+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.043322+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":619,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.393827+0000","last_change":"2024-02-23T00:40:25.452177+0000","last_active":"2024-02-23T00:40:39.393827+0000","last_peered":"2024-02-23T00:40:39.393827+0000","last_clean":"2024-02-23T00:40:39.393827+0000","last_became_active":"2024-02-23T00:40:23.205302+0000","last_became_peered":"2024-02-23T00:40:23.205302+0000","last_unstale":"2024-02-23T00:40:39.393827+0000","last_undegraded":"2024-02-23T00:40:39.393827+0000","last_fullsized":"2024-02-23T00:40:39.393827+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_clean_scrub_stamp":"2024-02-23T00:40:25.452086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":310,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206125+0000","last_change":"2024-02-23T00:40:23.222856+0000","last_active":"2024-02-23T00:40:39.206125+0000","last_peered":"2024-02-23T00:40:39.206125+0000","last_clean":"2024-02-23T00:40:39.206125+0000","last_became_active":"2024-02-23T00:40:23.194343+0000","last_became_peered":"2024-02-23T00:40:23.194343+0000","last_unstale":"2024-02-23T00:40:39.206125+0000","last_undegraded":"2024-02-23T00:40:39.206125+0000","last_fullsized":"2024-02-23T00:40:39.206125+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'25","last_scrub_stamp":"2024-02-23T00:35:01.396982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:01.396982+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":607,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396455+0000","last_change":"2024-02-23T00:40:23.196111+0000","last_active":"2024-02-23T00:40:39.396455+0000","last_peered":"2024-02-23T00:40:39.396455+0000","last_clean":"2024-02-23T00:40:39.396455+0000","last_became_active":"2024-02-23T00:40:23.195814+0000","last_became_peered":"2024-02-23T00:40:23.195814+0000","last_unstale":"2024-02-23T00:40:39.396455+0000","last_undegraded":"2024-02-23T00:40:39.396455+0000","last_fullsized":"2024-02-23T00:40:39.396455+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":584,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207071+0000","last_change":"2024-02-23T00:37:26.268389+0000","last_active":"2024-02-23T00:40:39.207071+0000","last_peered":"2024-02-23T00:40:39.207071+0000","last_clean":"2024-02-23T00:40:39.207071+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:40:39.207071+0000","last_undegraded":"2024-02-23T00:40:39.207071+0000","last_fullsized":"2024-02-23T00:40:39.207071+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.268282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":537,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207110+0000","last_change":"2024-02-23T00:39:38.350980+0000","last_active":"2024-02-23T00:40:39.207110+0000","last_peered":"2024-02-23T00:40:39.207110+0000","last_clean":"2024-02-23T00:40:39.207110+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:40:39.207110+0000","last_undegraded":"2024-02-23T00:40:39.207110+0000","last_fullsized":"2024-02-23T00:40:39.207110+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:38.350890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:38.350890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":620,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206524+0000","last_change":"2024-02-23T00:40:33.192004+0000","last_active":"2024-02-23T00:40:39.206524+0000","last_peered":"2024-02-23T00:40:39.206524+0000","last_clean":"2024-02-23T00:40:39.206524+0000","last_became_active":"2024-02-23T00:40:33.191698+0000","last_became_peered":"2024-02-23T00:40:33.191698+0000","last_unstale":"2024-02-23T00:40:39.206524+0000","last_undegraded":"2024-02-23T00:40:39.206524+0000","last_fullsized":"2024-02-23T00:40:39.206524+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":614,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":300,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208404+0000","last_change":"2024-02-23T00:40:26.193541+0000","last_active":"2024-02-23T00:40:39.208404+0000","last_peered":"2024-02-23T00:40:39.208404+0000","last_clean":"2024-02-23T00:40:39.208404+0000","last_became_active":"2024-02-23T00:40:26.193180+0000","last_became_peered":"2024-02-23T00:40:26.193180+0000","last_unstale":"2024-02-23T00:40:39.208404+0000","last_undegraded":"2024-02-23T00:40:39.208404+0000","last_fullsized":"2024-02-23T00:40:39.208404+0000","mapping_epoch":606,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":607,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394875+0000","last_change":"2024-02-23T00:40:23.196965+0000","last_active":"2024-02-23T00:40:39.394875+0000","last_peered":"2024-02-23T00:40:39.394875+0000","last_clean":"2024-02-23T00:40:39.394875+0000","last_became_active":"2024-02-23T00:40:23.196560+0000","last_became_peered":"2024-02-23T00:40:23.196560+0000","last_unstale":"2024-02-23T00:40:39.394875+0000","last_undegraded":"2024-02-23T00:40:39.394875+0000","last_fullsized":"2024-02-23T00:40:39.394875+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.980396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.980396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":544,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203864+0000","last_change":"2024-02-23T00:40:23.195425+0000","last_active":"2024-02-23T00:40:39.203864+0000","last_peered":"2024-02-23T00:40:39.203864+0000","last_clean":"2024-02-23T00:40:39.203864+0000","last_became_active":"2024-02-23T00:40:23.195137+0000","last_became_peered":"2024-02-23T00:40:23.195137+0000","last_unstale":"2024-02-23T00:40:39.203864+0000","last_undegraded":"2024-02-23T00:40:39.203864+0000","last_fullsized":"2024-02-23T00:40:39.203864+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:24.424938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:24.424938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":602,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208261+0000","last_change":"2024-02-23T00:39:39.315882+0000","last_active":"2024-02-23T00:40:39.208261+0000","last_peered":"2024-02-23T00:40:39.208261+0000","last_clean":"2024-02-23T00:40:39.208261+0000","last_became_active":"2024-02-23T00:38:49.224244+0000","last_became_peered":"2024-02-23T00:38:49.224244+0000","last_unstale":"2024-02-23T00:40:39.208261+0000","last_undegraded":"2024-02-23T00:40:39.208261+0000","last_fullsized":"2024-02-23T00:40:39.208261+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.315765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.315765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":546,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203785+0000","last_change":"2024-02-23T00:39:28.509774+0000","last_active":"2024-02-23T00:40:39.203785+0000","last_peered":"2024-02-23T00:40:39.203785+0000","last_clean":"2024-02-23T00:40:39.203785+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:40:39.203785+0000","last_undegraded":"2024-02-23T00:40:39.203785+0000","last_fullsized":"2024-02-23T00:40:39.203785+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:28.509675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.509675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":607,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395994+0000","last_change":"2024-02-23T00:39:39.963503+0000","last_active":"2024-02-23T00:40:39.395994+0000","last_peered":"2024-02-23T00:40:39.395994+0000","last_clean":"2024-02-23T00:40:39.395994+0000","last_became_active":"2024-02-23T00:35:27.093541+0000","last_became_peered":"2024-02-23T00:35:27.093541+0000","last_unstale":"2024-02-23T00:40:39.395994+0000","last_undegraded":"2024-02-23T00:40:39.395994+0000","last_fullsized":"2024-02-23T00:40:39.395994+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.963439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":642,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395613+0000","last_change":"2024-02-23T00:40:11.261185+0000","last_active":"2024-02-23T00:40:39.395613+0000","last_peered":"2024-02-23T00:40:39.395613+0000","last_clean":"2024-02-23T00:40:39.395613+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:40:39.395613+0000","last_undegraded":"2024-02-23T00:40:39.395613+0000","last_fullsized":"2024-02-23T00:40:39.395613+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:40:11.261102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.261102+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":610,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394948+0000","last_change":"2024-02-23T00:40:23.198426+0000","last_active":"2024-02-23T00:40:39.394948+0000","last_peered":"2024-02-23T00:40:39.394948+0000","last_clean":"2024-02-23T00:40:39.394948+0000","last_became_active":"2024-02-23T00:40:23.198114+0000","last_became_peered":"2024-02-23T00:40:23.198114+0000","last_unstale":"2024-02-23T00:40:39.394948+0000","last_undegraded":"2024-02-23T00:40:39.394948+0000","last_fullsized":"2024-02-23T00:40:39.394948+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.547900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.547900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":592,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205892+0000","last_change":"2024-02-23T00:39:40.690606+0000","last_active":"2024-02-23T00:40:39.205892+0000","last_peered":"2024-02-23T00:40:39.205892+0000","last_clean":"2024-02-23T00:40:39.205892+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:40:39.205892+0000","last_undegraded":"2024-02-23T00:40:39.205892+0000","last_fullsized":"2024-02-23T00:40:39.205892+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.690534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.690534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":119,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203844+0000","last_change":"2024-02-23T00:38:49.226513+0000","last_active":"2024-02-23T00:40:39.203844+0000","last_peered":"2024-02-23T00:40:39.203844+0000","last_clean":"2024-02-23T00:40:39.203844+0000","last_became_active":"2024-02-23T00:38:49.226188+0000","last_became_peered":"2024-02-23T00:38:49.226188+0000","last_unstale":"2024-02-23T00:40:39.203844+0000","last_undegraded":"2024-02-23T00:40:39.203844+0000","last_fullsized":"2024-02-23T00:40:39.203844+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.933288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.933288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":614,"reported_epoch":621,"state":"peering","last_fresh":"2024-02-23T00:40:40.208304+0000","last_change":"2024-02-23T00:40:40.208465+0000","last_active":"2024-02-23T00:40:38.201364+0000","last_peered":"2024-02-23T00:40:39.209692+0000","last_clean":"2024-02-23T00:40:39.209692+0000","last_became_active":"2024-02-23T00:38:49.184274+0000","last_became_peered":"2024-02-23T00:40:39.209132+0000","last_unstale":"2024-02-23T00:40:40.208304+0000","last_undegraded":"2024-02-23T00:40:40.208304+0000","last_fullsized":"2024-02-23T00:40:40.208304+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:20.033055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:40:20.033055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":548,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203671+0000","last_change":"2024-02-23T00:38:44.173181+0000","last_active":"2024-02-23T00:40:39.203671+0000","last_peered":"2024-02-23T00:40:39.203671+0000","last_clean":"2024-02-23T00:40:39.203671+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:40:39.203671+0000","last_undegraded":"2024-02-23T00:40:39.203671+0000","last_fullsized":"2024-02-23T00:40:39.203671+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:44.173077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.173077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":633,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204234+0000","last_change":"2024-02-23T00:40:38.061984+0000","last_active":"2024-02-23T00:40:39.204234+0000","last_peered":"2024-02-23T00:40:39.204234+0000","last_clean":"2024-02-23T00:40:39.204234+0000","last_became_active":"2024-02-23T00:40:38.061696+0000","last_became_peered":"2024-02-23T00:40:38.061696+0000","last_unstale":"2024-02-23T00:40:39.204234+0000","last_undegraded":"2024-02-23T00:40:39.204234+0000","last_fullsized":"2024-02-23T00:40:39.204234+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.413877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.413877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":709,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208097+0000","last_change":"2024-02-23T00:40:26.191175+0000","last_active":"2024-02-23T00:40:39.208097+0000","last_peered":"2024-02-23T00:40:39.208097+0000","last_clean":"2024-02-23T00:40:39.208097+0000","last_became_active":"2024-02-23T00:40:26.190861+0000","last_became_peered":"2024-02-23T00:40:26.190861+0000","last_unstale":"2024-02-23T00:40:39.208097+0000","last_undegraded":"2024-02-23T00:40:39.208097+0000","last_fullsized":"2024-02-23T00:40:39.208097+0000","mapping_epoch":606,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":639,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203914+0000","last_change":"2024-02-23T00:40:23.196884+0000","last_active":"2024-02-23T00:40:39.203914+0000","last_peered":"2024-02-23T00:40:39.203914+0000","last_clean":"2024-02-23T00:40:39.203914+0000","last_became_active":"2024-02-23T00:40:23.196485+0000","last_became_peered":"2024-02-23T00:40:23.196485+0000","last_unstale":"2024-02-23T00:40:39.203914+0000","last_undegraded":"2024-02-23T00:40:39.203914+0000","last_fullsized":"2024-02-23T00:40:39.203914+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:22.437104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.437104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":851,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207504+0000","last_change":"2024-02-23T00:40:12.984115+0000","last_active":"2024-02-23T00:40:39.207504+0000","last_peered":"2024-02-23T00:40:39.207504+0000","last_clean":"2024-02-23T00:40:39.207504+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:40:39.207504+0000","last_undegraded":"2024-02-23T00:40:39.207504+0000","last_fullsized":"2024-02-23T00:40:39.207504+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:12.984040+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.984040+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":614,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396415+0000","last_change":"2024-02-23T00:40:20.642447+0000","last_active":"2024-02-23T00:40:39.396415+0000","last_peered":"2024-02-23T00:40:39.396415+0000","last_clean":"2024-02-23T00:40:39.396415+0000","last_became_active":"2024-02-23T00:38:49.195404+0000","last_became_peered":"2024-02-23T00:38:49.195404+0000","last_unstale":"2024-02-23T00:40:39.396415+0000","last_undegraded":"2024-02-23T00:40:39.396415+0000","last_fullsized":"2024-02-23T00:40:39.396415+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:20.642339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:20.642339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":552,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203577+0000","last_change":"2024-02-23T00:38:40.265472+0000","last_active":"2024-02-23T00:40:39.203577+0000","last_peered":"2024-02-23T00:40:39.203577+0000","last_clean":"2024-02-23T00:40:39.203577+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:40:39.203577+0000","last_undegraded":"2024-02-23T00:40:39.203577+0000","last_fullsized":"2024-02-23T00:40:39.203577+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:40.265385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.265385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":618,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203542+0000","last_change":"2024-02-23T00:40:23.204324+0000","last_active":"2024-02-23T00:40:39.203542+0000","last_peered":"2024-02-23T00:40:39.203542+0000","last_clean":"2024-02-23T00:40:39.203542+0000","last_became_active":"2024-02-23T00:40:23.203665+0000","last_became_peered":"2024-02-23T00:40:23.203665+0000","last_unstale":"2024-02-23T00:40:39.203542+0000","last_undegraded":"2024-02-23T00:40:39.203542+0000","last_fullsized":"2024-02-23T00:40:39.203542+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:03.595447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:03.595447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":568,"reported_epoch":621,"state":"peering","last_fresh":"2024-02-23T00:40:40.211896+0000","last_change":"2024-02-23T00:40:40.212691+0000","last_active":"2024-02-23T00:40:38.202708+0000","last_peered":"2024-02-23T00:40:39.395976+0000","last_clean":"2024-02-23T00:40:39.395976+0000","last_became_active":"2024-02-23T00:40:23.200579+0000","last_became_peered":"2024-02-23T00:40:39.395539+0000","last_unstale":"2024-02-23T00:40:40.211896+0000","last_undegraded":"2024-02-23T00:40:40.211896+0000","last_fullsized":"2024-02-23T00:40:40.211896+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":901,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.212214+0000","last_change":"2024-02-23T00:40:15.043521+0000","last_active":"2024-02-23T00:40:40.212214+0000","last_peered":"2024-02-23T00:40:40.212214+0000","last_clean":"2024-02-23T00:40:40.212214+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:40:40.212214+0000","last_undegraded":"2024-02-23T00:40:40.212214+0000","last_fullsized":"2024-02-23T00:40:40.212214+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'48","last_scrub_stamp":"2024-02-23T00:40:15.043419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.043419+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"601'2","reported_seq":234,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207381+0000","last_change":"2024-02-23T00:40:21.186280+0000","last_active":"2024-02-23T00:40:39.207381+0000","last_peered":"2024-02-23T00:40:39.207381+0000","last_clean":"2024-02-23T00:40:39.207381+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:40:39.207381+0000","last_undegraded":"2024-02-23T00:40:39.207381+0000","last_fullsized":"2024-02-23T00:40:39.207381+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'2","last_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_deep_scrub":"601'2","last_deep_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_clean_scrub_stamp":"2024-02-23T00:40:21.186215+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":124,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204660+0000","last_change":"2024-02-23T00:39:40.370151+0000","last_active":"2024-02-23T00:40:39.204660+0000","last_peered":"2024-02-23T00:40:39.204660+0000","last_clean":"2024-02-23T00:40:39.204660+0000","last_became_active":"2024-02-23T00:38:49.189587+0000","last_became_peered":"2024-02-23T00:38:49.189587+0000","last_unstale":"2024-02-23T00:40:39.204660+0000","last_undegraded":"2024-02-23T00:40:39.204660+0000","last_fullsized":"2024-02-23T00:40:39.204660+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.370090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.370090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":557,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204332+0000","last_change":"2024-02-23T00:40:22.203106+0000","last_active":"2024-02-23T00:40:39.204332+0000","last_peered":"2024-02-23T00:40:39.204332+0000","last_clean":"2024-02-23T00:40:39.204332+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:40:39.204332+0000","last_undegraded":"2024-02-23T00:40:39.204332+0000","last_fullsized":"2024-02-23T00:40:39.204332+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:22.203041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:22.203041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":520,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205999+0000","last_change":"2024-02-23T00:39:32.809209+0000","last_active":"2024-02-23T00:40:39.205999+0000","last_peered":"2024-02-23T00:40:39.205999+0000","last_clean":"2024-02-23T00:40:39.205999+0000","last_became_active":"2024-02-23T00:38:49.183777+0000","last_became_peered":"2024-02-23T00:38:49.183777+0000","last_unstale":"2024-02-23T00:40:39.205999+0000","last_undegraded":"2024-02-23T00:40:39.205999+0000","last_fullsized":"2024-02-23T00:40:39.205999+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:32.809133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:32.809133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"62'8","reported_seq":232,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395373+0000","last_change":"2024-02-23T00:40:23.198554+0000","last_active":"2024-02-23T00:40:39.395373+0000","last_peered":"2024-02-23T00:40:39.395373+0000","last_clean":"2024-02-23T00:40:39.395373+0000","last_became_active":"2024-02-23T00:40:23.197914+0000","last_became_peered":"2024-02-23T00:40:23.197914+0000","last_unstale":"2024-02-23T00:40:39.395373+0000","last_undegraded":"2024-02-23T00:40:39.395373+0000","last_fullsized":"2024-02-23T00:40:39.395373+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":41,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207236+0000","last_change":"2024-02-23T00:40:24.241074+0000","last_active":"2024-02-23T00:40:39.207236+0000","last_peered":"2024-02-23T00:40:39.207236+0000","last_clean":"2024-02-23T00:40:39.207236+0000","last_became_active":"2024-02-23T00:40:23.194277+0000","last_became_peered":"2024-02-23T00:40:23.194277+0000","last_unstale":"2024-02-23T00:40:39.207236+0000","last_undegraded":"2024-02-23T00:40:39.207236+0000","last_fullsized":"2024-02-23T00:40:39.207236+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'16","last_scrub_stamp":"2024-02-23T00:35:44.214727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:35:44.214727+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":603,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207937+0000","last_change":"2024-02-23T00:40:23.198662+0000","last_active":"2024-02-23T00:40:39.207937+0000","last_peered":"2024-02-23T00:40:39.207937+0000","last_clean":"2024-02-23T00:40:39.207937+0000","last_became_active":"2024-02-23T00:40:23.198153+0000","last_became_peered":"2024-02-23T00:40:23.198153+0000","last_unstale":"2024-02-23T00:40:39.207937+0000","last_undegraded":"2024-02-23T00:40:39.207937+0000","last_fullsized":"2024-02-23T00:40:39.207937+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:13.208475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:34:13.208475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":134,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203189+0000","last_change":"2024-02-23T00:40:23.200282+0000","last_active":"2024-02-23T00:40:39.203189+0000","last_peered":"2024-02-23T00:40:39.203189+0000","last_clean":"2024-02-23T00:40:39.203189+0000","last_became_active":"2024-02-23T00:40:23.199290+0000","last_became_peered":"2024-02-23T00:40:23.199290+0000","last_unstale":"2024-02-23T00:40:39.203189+0000","last_undegraded":"2024-02-23T00:40:39.203189+0000","last_fullsized":"2024-02-23T00:40:39.203189+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:21.370914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:21.370914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":546,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203408+0000","last_change":"2024-02-23T00:39:30.473677+0000","last_active":"2024-02-23T00:40:39.203408+0000","last_peered":"2024-02-23T00:40:39.203408+0000","last_clean":"2024-02-23T00:40:39.203408+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:40:39.203408+0000","last_undegraded":"2024-02-23T00:40:39.203408+0000","last_fullsized":"2024-02-23T00:40:39.203408+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.473602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.473602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":633,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204875+0000","last_change":"2024-02-23T00:40:29.192688+0000","last_active":"2024-02-23T00:40:39.204875+0000","last_peered":"2024-02-23T00:40:39.204875+0000","last_clean":"2024-02-23T00:40:39.204875+0000","last_became_active":"2024-02-23T00:40:29.192428+0000","last_became_peered":"2024-02-23T00:40:29.192428+0000","last_unstale":"2024-02-23T00:40:39.204875+0000","last_undegraded":"2024-02-23T00:40:39.204875+0000","last_fullsized":"2024-02-23T00:40:39.204875+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:24.016837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:24.016837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":813,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203927+0000","last_change":"2024-02-23T00:38:54.171994+0000","last_active":"2024-02-23T00:40:39.203927+0000","last_peered":"2024-02-23T00:40:39.203927+0000","last_clean":"2024-02-23T00:40:39.203927+0000","last_became_active":"2024-02-23T00:38:54.171563+0000","last_became_peered":"2024-02-23T00:38:54.171563+0000","last_unstale":"2024-02-23T00:40:39.203927+0000","last_undegraded":"2024-02-23T00:40:39.203927+0000","last_fullsized":"2024-02-23T00:40:39.203927+0000","mapping_epoch":516,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:06.122240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.122240+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":628,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203824+0000","last_change":"2024-02-23T00:40:23.373198+0000","last_active":"2024-02-23T00:40:39.203824+0000","last_peered":"2024-02-23T00:40:39.203824+0000","last_clean":"2024-02-23T00:40:39.203824+0000","last_became_active":"2024-02-23T00:40:23.191879+0000","last_became_peered":"2024-02-23T00:40:23.191879+0000","last_unstale":"2024-02-23T00:40:39.203824+0000","last_undegraded":"2024-02-23T00:40:39.203824+0000","last_fullsized":"2024-02-23T00:40:39.203824+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:36:36.130868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:36.130868+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":137,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204551+0000","last_change":"2024-02-23T00:40:17.214527+0000","last_active":"2024-02-23T00:40:39.204551+0000","last_peered":"2024-02-23T00:40:39.204551+0000","last_clean":"2024-02-23T00:40:39.204551+0000","last_became_active":"2024-02-23T00:40:17.214288+0000","last_became_peered":"2024-02-23T00:40:17.214288+0000","last_unstale":"2024-02-23T00:40:39.204551+0000","last_undegraded":"2024-02-23T00:40:39.204551+0000","last_fullsized":"2024-02-23T00:40:39.204551+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.172059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.172059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":541,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205607+0000","last_change":"2024-02-23T00:40:05.936821+0000","last_active":"2024-02-23T00:40:39.205607+0000","last_peered":"2024-02-23T00:40:39.205607+0000","last_clean":"2024-02-23T00:40:39.205607+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:40:39.205607+0000","last_undegraded":"2024-02-23T00:40:39.205607+0000","last_fullsized":"2024-02-23T00:40:39.205607+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:05.936761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:05.936761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":323,"reported_epoch":620,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:40:39.202762+0000","last_change":"2024-02-23T00:40:23.205071+0000","last_active":"2024-02-23T00:40:39.202762+0000","last_peered":"2024-02-23T00:40:39.202762+0000","last_clean":"2024-02-23T00:40:21.193234+0000","last_became_active":"2024-02-23T00:40:23.202555+0000","last_became_peered":"2024-02-23T00:40:23.202555+0000","last_unstale":"2024-02-23T00:40:39.202762+0000","last_undegraded":"2024-02-23T00:40:39.202762+0000","last_fullsized":"2024-02-23T00:40:39.202762+0000","mapping_epoch":603,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":610,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204800+0000","last_change":"2024-02-23T00:40:23.197180+0000","last_active":"2024-02-23T00:40:39.204800+0000","last_peered":"2024-02-23T00:40:39.204800+0000","last_clean":"2024-02-23T00:40:39.204800+0000","last_became_active":"2024-02-23T00:40:23.196558+0000","last_became_peered":"2024-02-23T00:40:23.196558+0000","last_unstale":"2024-02-23T00:40:39.204800+0000","last_undegraded":"2024-02-23T00:40:39.204800+0000","last_fullsized":"2024-02-23T00:40:39.204800+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:24.667517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:24.667517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":124,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204138+0000","last_change":"2024-02-23T00:39:18.369649+0000","last_active":"2024-02-23T00:40:39.204138+0000","last_peered":"2024-02-23T00:40:39.204138+0000","last_clean":"2024-02-23T00:40:39.204138+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:40:39.204138+0000","last_undegraded":"2024-02-23T00:40:39.204138+0000","last_fullsized":"2024-02-23T00:40:39.204138+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:18.369579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:18.369579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":49,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208238+0000","last_change":"2024-02-23T00:40:01.313675+0000","last_active":"2024-02-23T00:40:39.208238+0000","last_peered":"2024-02-23T00:40:39.208238+0000","last_clean":"2024-02-23T00:40:39.208238+0000","last_became_active":"2024-02-23T00:40:01.313237+0000","last_became_peered":"2024-02-23T00:40:01.313237+0000","last_unstale":"2024-02-23T00:40:39.208238+0000","last_undegraded":"2024-02-23T00:40:39.208238+0000","last_fullsized":"2024-02-23T00:40:39.208238+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.293824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:22.918248+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.293824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":119,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205014+0000","last_change":"2024-02-23T00:38:49.230851+0000","last_active":"2024-02-23T00:40:39.205014+0000","last_peered":"2024-02-23T00:40:39.205014+0000","last_clean":"2024-02-23T00:40:39.205014+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:40:39.205014+0000","last_undegraded":"2024-02-23T00:40:39.205014+0000","last_fullsized":"2024-02-23T00:40:39.205014+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:39.893315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.893315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":708,"reported_epoch":620,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-23T00:40:39.207020+0000","last_change":"2024-02-23T00:40:19.252581+0000","last_active":"2024-02-23T00:40:39.207020+0000","last_peered":"2024-02-23T00:40:39.207020+0000","last_clean":"2024-02-23T00:36:33.367236+0000","last_became_active":"2024-02-23T00:38:49.224732+0000","last_became_peered":"2024-02-23T00:38:49.224732+0000","last_unstale":"2024-02-23T00:40:39.207020+0000","last_undegraded":"2024-02-23T00:38:49.159472+0000","last_fullsized":"2024-02-23T00:38:49.159180+0000","mapping_epoch":511,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":599,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394814+0000","last_change":"2024-02-23T00:38:42.431748+0000","last_active":"2024-02-23T00:40:39.394814+0000","last_peered":"2024-02-23T00:40:39.394814+0000","last_clean":"2024-02-23T00:40:39.394814+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:40:39.394814+0000","last_undegraded":"2024-02-23T00:40:39.394814+0000","last_fullsized":"2024-02-23T00:40:39.394814+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:38:42.431641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":601,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206507+0000","last_change":"2024-02-23T00:39:28.781689+0000","last_active":"2024-02-23T00:40:39.206507+0000","last_peered":"2024-02-23T00:40:39.206507+0000","last_clean":"2024-02-23T00:40:39.206507+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:40:39.206507+0000","last_undegraded":"2024-02-23T00:40:39.206507+0000","last_fullsized":"2024-02-23T00:40:39.206507+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:28.781587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.781587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":605,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396319+0000","last_change":"2024-02-23T00:40:17.689322+0000","last_active":"2024-02-23T00:40:39.396319+0000","last_peered":"2024-02-23T00:40:39.396319+0000","last_clean":"2024-02-23T00:40:39.396319+0000","last_became_active":"2024-02-23T00:40:01.297296+0000","last_became_peered":"2024-02-23T00:40:01.297296+0000","last_unstale":"2024-02-23T00:40:39.396319+0000","last_undegraded":"2024-02-23T00:40:39.396319+0000","last_fullsized":"2024-02-23T00:40:39.396319+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:17.689240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:17.689240+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.689240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":564,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395809+0000","last_change":"2024-02-23T00:39:29.410514+0000","last_active":"2024-02-23T00:40:39.395809+0000","last_peered":"2024-02-23T00:40:39.395809+0000","last_clean":"2024-02-23T00:40:39.395809+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:40:39.395809+0000","last_undegraded":"2024-02-23T00:40:39.395809+0000","last_fullsized":"2024-02-23T00:40:39.395809+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:29.410456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:29.410456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"453'32","reported_seq":135,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204826+0000","last_change":"2024-02-23T00:38:51.793947+0000","last_active":"2024-02-23T00:40:39.204826+0000","last_peered":"2024-02-23T00:40:39.204826+0000","last_clean":"2024-02-23T00:40:39.204826+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:40:39.204826+0000","last_undegraded":"2024-02-23T00:40:39.204826+0000","last_fullsized":"2024-02-23T00:40:39.204826+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:38:44.240437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.240437+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":618,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206788+0000","last_change":"2024-02-23T00:39:23.833122+0000","last_active":"2024-02-23T00:40:39.206788+0000","last_peered":"2024-02-23T00:40:39.206788+0000","last_clean":"2024-02-23T00:40:39.206788+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:40:39.206788+0000","last_undegraded":"2024-02-23T00:40:39.206788+0000","last_fullsized":"2024-02-23T00:40:39.206788+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:23.833059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:39:23.833059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"606'11","reported_seq":768,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.208847+0000","last_change":"2024-02-23T00:40:23.211815+0000","last_active":"2024-02-23T00:40:40.208847+0000","last_peered":"2024-02-23T00:40:40.208847+0000","last_clean":"2024-02-23T00:40:40.208847+0000","last_became_active":"2024-02-23T00:40:23.202761+0000","last_became_peered":"2024-02-23T00:40:23.202761+0000","last_unstale":"2024-02-23T00:40:40.208847+0000","last_undegraded":"2024-02-23T00:40:40.208847+0000","last_fullsized":"2024-02-23T00:40:40.208847+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'9","last_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:32:56.376370+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":119,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204738+0000","last_change":"2024-02-23T00:38:49.190032+0000","last_active":"2024-02-23T00:40:39.204738+0000","last_peered":"2024-02-23T00:40:39.204738+0000","last_clean":"2024-02-23T00:40:39.204738+0000","last_became_active":"2024-02-23T00:38:49.189564+0000","last_became_peered":"2024-02-23T00:38:49.189564+0000","last_unstale":"2024-02-23T00:40:39.204738+0000","last_undegraded":"2024-02-23T00:40:39.204738+0000","last_fullsized":"2024-02-23T00:40:39.204738+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:41.177673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:38:41.177673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":541,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206802+0000","last_change":"2024-02-23T00:40:17.978624+0000","last_active":"2024-02-23T00:40:39.206802+0000","last_peered":"2024-02-23T00:40:39.206802+0000","last_clean":"2024-02-23T00:40:39.206802+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:40:39.206802+0000","last_undegraded":"2024-02-23T00:40:39.206802+0000","last_fullsized":"2024-02-23T00:40:39.206802+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:17.978560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.978560+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":672,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206099+0000","last_change":"2024-02-23T00:38:49.346222+0000","last_active":"2024-02-23T00:40:39.206099+0000","last_peered":"2024-02-23T00:40:39.206099+0000","last_clean":"2024-02-23T00:40:39.206099+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:40:39.206099+0000","last_undegraded":"2024-02-23T00:40:39.206099+0000","last_fullsized":"2024-02-23T00:40:39.206099+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:40.858800+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.858800+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":531,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204192+0000","last_change":"2024-02-23T00:40:23.196005+0000","last_active":"2024-02-23T00:40:39.204192+0000","last_peered":"2024-02-23T00:40:39.204192+0000","last_clean":"2024-02-23T00:40:39.204192+0000","last_became_active":"2024-02-23T00:40:23.195711+0000","last_became_peered":"2024-02-23T00:40:23.195711+0000","last_unstale":"2024-02-23T00:40:39.204192+0000","last_undegraded":"2024-02-23T00:40:39.204192+0000","last_fullsized":"2024-02-23T00:40:39.204192+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:26.483929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:26.483929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":146,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.209629+0000","last_change":"2024-02-23T00:38:49.384331+0000","last_active":"2024-02-23T00:40:40.209629+0000","last_peered":"2024-02-23T00:40:40.209629+0000","last_clean":"2024-02-23T00:40:40.209629+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:40:40.209629+0000","last_undegraded":"2024-02-23T00:40:40.209629+0000","last_fullsized":"2024-02-23T00:40:40.209629+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'10","last_scrub_stamp":"2024-02-23T00:37:27.870653+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:37:27.870653+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":119,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203974+0000","last_change":"2024-02-23T00:38:49.227700+0000","last_active":"2024-02-23T00:40:39.203974+0000","last_peered":"2024-02-23T00:40:39.203974+0000","last_clean":"2024-02-23T00:40:39.203974+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:40:39.203974+0000","last_undegraded":"2024-02-23T00:40:39.203974+0000","last_fullsized":"2024-02-23T00:40:39.203974+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:09.447326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.447326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":613,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207888+0000","last_change":"2024-02-23T00:37:20.382193+0000","last_active":"2024-02-23T00:40:39.207888+0000","last_peered":"2024-02-23T00:40:39.207888+0000","last_clean":"2024-02-23T00:40:39.207888+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:40:39.207888+0000","last_undegraded":"2024-02-23T00:40:39.207888+0000","last_fullsized":"2024-02-23T00:40:39.207888+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:37:20.382108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.382108+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":622,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395443+0000","last_change":"2024-02-23T00:40:08.274788+0000","last_active":"2024-02-23T00:40:39.395443+0000","last_peered":"2024-02-23T00:40:39.395443+0000","last_clean":"2024-02-23T00:40:39.395443+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:40:39.395443+0000","last_undegraded":"2024-02-23T00:40:39.395443+0000","last_fullsized":"2024-02-23T00:40:39.395443+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:08.274721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:08.274721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":134,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204392+0000","last_change":"2024-02-23T00:40:23.203466+0000","last_active":"2024-02-23T00:40:39.204392+0000","last_peered":"2024-02-23T00:40:39.204392+0000","last_clean":"2024-02-23T00:40:39.204392+0000","last_became_active":"2024-02-23T00:40:23.203017+0000","last_became_peered":"2024-02-23T00:40:23.203017+0000","last_unstale":"2024-02-23T00:40:39.204392+0000","last_undegraded":"2024-02-23T00:40:39.204392+0000","last_fullsized":"2024-02-23T00:40:39.204392+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.370620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.370620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":65,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.202920+0000","last_change":"2024-02-23T00:40:24.195853+0000","last_active":"2024-02-23T00:40:39.202920+0000","last_peered":"2024-02-23T00:40:39.202920+0000","last_clean":"2024-02-23T00:40:39.202920+0000","last_became_active":"2024-02-23T00:40:24.195542+0000","last_became_peered":"2024-02-23T00:40:24.195542+0000","last_unstale":"2024-02-23T00:40:39.202920+0000","last_undegraded":"2024-02-23T00:40:39.202920+0000","last_fullsized":"2024-02-23T00:40:39.202920+0000","mapping_epoch":604,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:06.292724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:06.292724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":556,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203341+0000","last_change":"2024-02-23T00:40:23.187111+0000","last_active":"2024-02-23T00:40:39.203341+0000","last_peered":"2024-02-23T00:40:39.203341+0000","last_clean":"2024-02-23T00:40:39.203341+0000","last_became_active":"2024-02-23T00:40:23.186760+0000","last_became_peered":"2024-02-23T00:40:23.186760+0000","last_unstale":"2024-02-23T00:40:39.203341+0000","last_undegraded":"2024-02-23T00:40:39.203341+0000","last_fullsized":"2024-02-23T00:40:39.203341+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.101501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.101501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":649,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204124+0000","last_change":"2024-02-23T00:38:16.174640+0000","last_active":"2024-02-23T00:40:39.204124+0000","last_peered":"2024-02-23T00:40:39.204124+0000","last_clean":"2024-02-23T00:40:39.204124+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:40:39.204124+0000","last_undegraded":"2024-02-23T00:40:39.204124+0000","last_fullsized":"2024-02-23T00:40:39.204124+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:38:16.174526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.174526+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":600,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208039+0000","last_change":"2024-02-23T00:40:23.201115+0000","last_active":"2024-02-23T00:40:39.208039+0000","last_peered":"2024-02-23T00:40:39.208039+0000","last_clean":"2024-02-23T00:40:39.208039+0000","last_became_active":"2024-02-23T00:40:23.200826+0000","last_became_peered":"2024-02-23T00:40:23.200826+0000","last_unstale":"2024-02-23T00:40:39.208039+0000","last_undegraded":"2024-02-23T00:40:39.208039+0000","last_fullsized":"2024-02-23T00:40:39.208039+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.783089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.783089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":604,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207812+0000","last_change":"2024-02-23T00:38:49.225280+0000","last_active":"2024-02-23T00:40:39.207812+0000","last_peered":"2024-02-23T00:40:39.207812+0000","last_clean":"2024-02-23T00:40:39.207812+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:40:39.207812+0000","last_undegraded":"2024-02-23T00:40:39.207812+0000","last_fullsized":"2024-02-23T00:40:39.207812+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:06.235608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.235608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":48,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206814+0000","last_change":"2024-02-23T00:40:02.301118+0000","last_active":"2024-02-23T00:40:39.206814+0000","last_peered":"2024-02-23T00:40:39.206814+0000","last_clean":"2024-02-23T00:40:39.206814+0000","last_became_active":"2024-02-23T00:40:02.300737+0000","last_became_peered":"2024-02-23T00:40:02.300737+0000","last_unstale":"2024-02-23T00:40:39.206814+0000","last_undegraded":"2024-02-23T00:40:39.206814+0000","last_fullsized":"2024-02-23T00:40:39.206814+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:25.311952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:39:25.311952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":590,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395051+0000","last_change":"2024-02-23T00:39:22.387191+0000","last_active":"2024-02-23T00:40:39.395051+0000","last_peered":"2024-02-23T00:40:39.395051+0000","last_clean":"2024-02-23T00:40:39.395051+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:40:39.395051+0000","last_undegraded":"2024-02-23T00:40:39.395051+0000","last_fullsized":"2024-02-23T00:40:39.395051+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:39:22.387089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.387089+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":130,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204882+0000","last_change":"2024-02-23T00:40:12.128771+0000","last_active":"2024-02-23T00:40:39.204882+0000","last_peered":"2024-02-23T00:40:39.204882+0000","last_clean":"2024-02-23T00:40:39.204882+0000","last_became_active":"2024-02-23T00:38:49.217612+0000","last_became_peered":"2024-02-23T00:38:49.217612+0000","last_unstale":"2024-02-23T00:40:39.204882+0000","last_undegraded":"2024-02-23T00:40:39.204882+0000","last_fullsized":"2024-02-23T00:40:39.204882+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'10","last_scrub_stamp":"2024-02-23T00:40:12.128684+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.128684+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":592,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206621+0000","last_change":"2024-02-23T00:40:11.889392+0000","last_active":"2024-02-23T00:40:39.206621+0000","last_peered":"2024-02-23T00:40:39.206621+0000","last_clean":"2024-02-23T00:40:39.206621+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:40:39.206621+0000","last_undegraded":"2024-02-23T00:40:39.206621+0000","last_fullsized":"2024-02-23T00:40:39.206621+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:11.889327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.889327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":595,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394737+0000","last_change":"2024-02-23T00:40:13.313672+0000","last_active":"2024-02-23T00:40:39.394737+0000","last_peered":"2024-02-23T00:40:39.394737+0000","last_clean":"2024-02-23T00:40:39.394737+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:40:39.394737+0000","last_undegraded":"2024-02-23T00:40:39.394737+0000","last_fullsized":"2024-02-23T00:40:39.394737+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:13.313569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:13.313569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":56,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203309+0000","last_change":"2024-02-23T00:40:10.335909+0000","last_active":"2024-02-23T00:40:39.203309+0000","last_peered":"2024-02-23T00:40:39.203309+0000","last_clean":"2024-02-23T00:40:39.203309+0000","last_became_active":"2024-02-23T00:40:01.298496+0000","last_became_peered":"2024-02-23T00:40:01.298496+0000","last_unstale":"2024-02-23T00:40:39.203309+0000","last_undegraded":"2024-02-23T00:40:39.203309+0000","last_fullsized":"2024-02-23T00:40:39.203309+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:10.335845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:10.335845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":537,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207899+0000","last_change":"2024-02-23T00:38:33.242488+0000","last_active":"2024-02-23T00:40:39.207899+0000","last_peered":"2024-02-23T00:40:39.207899+0000","last_clean":"2024-02-23T00:40:39.207899+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:40:39.207899+0000","last_undegraded":"2024-02-23T00:40:39.207899+0000","last_fullsized":"2024-02-23T00:40:39.207899+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:33.242417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:33.242417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":136,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204850+0000","last_change":"2024-02-23T00:38:51.365185+0000","last_active":"2024-02-23T00:40:39.204850+0000","last_peered":"2024-02-23T00:40:39.204850+0000","last_clean":"2024-02-23T00:40:39.204850+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:40:39.204850+0000","last_undegraded":"2024-02-23T00:40:39.204850+0000","last_fullsized":"2024-02-23T00:40:39.204850+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:37:26.914698+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.914698+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":598,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396559+0000","last_change":"2024-02-23T00:39:26.456134+0000","last_active":"2024-02-23T00:40:39.396559+0000","last_peered":"2024-02-23T00:40:39.396559+0000","last_clean":"2024-02-23T00:40:39.396559+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:40:39.396559+0000","last_undegraded":"2024-02-23T00:40:39.396559+0000","last_fullsized":"2024-02-23T00:40:39.396559+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:26.456074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:26.456074+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":593,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395165+0000","last_change":"2024-02-23T00:38:16.055268+0000","last_active":"2024-02-23T00:40:39.395165+0000","last_peered":"2024-02-23T00:40:39.395165+0000","last_clean":"2024-02-23T00:40:39.395165+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:40:39.395165+0000","last_undegraded":"2024-02-23T00:40:39.395165+0000","last_fullsized":"2024-02-23T00:40:39.395165+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.055159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":48,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396113+0000","last_change":"2024-02-23T00:40:02.301143+0000","last_active":"2024-02-23T00:40:39.396113+0000","last_peered":"2024-02-23T00:40:39.396113+0000","last_clean":"2024-02-23T00:40:39.396113+0000","last_became_active":"2024-02-23T00:40:02.300797+0000","last_became_peered":"2024-02-23T00:40:02.300797+0000","last_unstale":"2024-02-23T00:40:39.396113+0000","last_undegraded":"2024-02-23T00:40:39.396113+0000","last_fullsized":"2024-02-23T00:40:39.396113+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.381447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.381447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":119,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204285+0000","last_change":"2024-02-23T00:38:49.189171+0000","last_active":"2024-02-23T00:40:39.204285+0000","last_peered":"2024-02-23T00:40:39.204285+0000","last_clean":"2024-02-23T00:40:39.204285+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:40:39.204285+0000","last_undegraded":"2024-02-23T00:40:39.204285+0000","last_fullsized":"2024-02-23T00:40:39.204285+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:37.979785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.979785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":723,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396986+0000","last_change":"2024-02-23T00:40:23.194464+0000","last_active":"2024-02-23T00:40:39.396986+0000","last_peered":"2024-02-23T00:40:39.396986+0000","last_clean":"2024-02-23T00:40:39.396986+0000","last_became_active":"2024-02-23T00:40:23.193752+0000","last_became_peered":"2024-02-23T00:40:23.193752+0000","last_unstale":"2024-02-23T00:40:39.396986+0000","last_undegraded":"2024-02-23T00:40:39.396986+0000","last_fullsized":"2024-02-23T00:40:39.396986+0000","mapping_epoch":603,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":596,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205157+0000","last_change":"2024-02-23T00:38:36.853659+0000","last_active":"2024-02-23T00:40:39.205157+0000","last_peered":"2024-02-23T00:40:39.205157+0000","last_clean":"2024-02-23T00:40:39.205157+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:40:39.205157+0000","last_undegraded":"2024-02-23T00:40:39.205157+0000","last_fullsized":"2024-02-23T00:40:39.205157+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:36.853582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:36.853582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":767,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-02-23T00:40:40.215464+0000","last_change":"2024-02-23T00:38:37.455385+0000","last_active":"2024-02-23T00:40:40.215464+0000","last_peered":"2024-02-23T00:40:40.215464+0000","last_clean":"2024-02-23T00:40:40.215464+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:40:40.215464+0000","last_undegraded":"2024-02-23T00:40:40.215464+0000","last_fullsized":"2024-02-23T00:40:40.215464+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:38:37.455280+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.455280+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":579,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396135+0000","last_change":"2024-02-23T00:40:23.194034+0000","last_active":"2024-02-23T00:40:39.396135+0000","last_peered":"2024-02-23T00:40:39.396135+0000","last_clean":"2024-02-23T00:40:39.396135+0000","last_became_active":"2024-02-23T00:40:23.193733+0000","last_became_peered":"2024-02-23T00:40:23.193733+0000","last_unstale":"2024-02-23T00:40:39.396135+0000","last_undegraded":"2024-02-23T00:40:39.396135+0000","last_fullsized":"2024-02-23T00:40:39.396135+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.887584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.887584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":765,"reported_epoch":620,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:40:39.207968+0000","last_change":"2024-02-23T00:40:22.608402+0000","last_active":"2024-02-23T00:40:39.207968+0000","last_peered":"2024-02-23T00:40:39.207968+0000","last_clean":"2024-02-23T00:38:19.193308+0000","last_became_active":"2024-02-23T00:38:22.159363+0000","last_became_peered":"2024-02-23T00:38:22.159363+0000","last_unstale":"2024-02-23T00:40:39.207968+0000","last_undegraded":"2024-02-23T00:40:39.207968+0000","last_fullsized":"2024-02-23T00:40:39.207968+0000","mapping_epoch":484,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":134,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204721+0000","last_change":"2024-02-23T00:40:23.205225+0000","last_active":"2024-02-23T00:40:39.204721+0000","last_peered":"2024-02-23T00:40:39.204721+0000","last_clean":"2024-02-23T00:40:39.204721+0000","last_became_active":"2024-02-23T00:40:23.203999+0000","last_became_peered":"2024-02-23T00:40:23.203999+0000","last_unstale":"2024-02-23T00:40:39.204721+0000","last_undegraded":"2024-02-23T00:40:39.204721+0000","last_fullsized":"2024-02-23T00:40:39.204721+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:32.310679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:39:32.310679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":212,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.397161+0000","last_change":"2024-02-23T00:40:23.194631+0000","last_active":"2024-02-23T00:40:39.397161+0000","last_peered":"2024-02-23T00:40:39.397161+0000","last_clean":"2024-02-23T00:40:39.397161+0000","last_became_active":"2024-02-23T00:40:23.194155+0000","last_became_peered":"2024-02-23T00:40:23.194155+0000","last_unstale":"2024-02-23T00:40:39.397161+0000","last_undegraded":"2024-02-23T00:40:39.397161+0000","last_fullsized":"2024-02-23T00:40:39.397161+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":545,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396556+0000","last_change":"2024-02-23T00:38:35.044376+0000","last_active":"2024-02-23T00:40:39.396556+0000","last_peered":"2024-02-23T00:40:39.396556+0000","last_clean":"2024-02-23T00:40:39.396556+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:40:39.396556+0000","last_undegraded":"2024-02-23T00:40:39.396556+0000","last_fullsized":"2024-02-23T00:40:39.396556+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:35.044298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.044298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":733,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205393+0000","last_change":"2024-02-23T00:40:17.268690+0000","last_active":"2024-02-23T00:40:39.205393+0000","last_peered":"2024-02-23T00:40:39.205393+0000","last_clean":"2024-02-23T00:40:39.205393+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:40:39.205393+0000","last_undegraded":"2024-02-23T00:40:39.205393+0000","last_fullsized":"2024-02-23T00:40:39.205393+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:40:17.268569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.268569+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":607,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395313+0000","last_change":"2024-02-23T00:40:19.377388+0000","last_active":"2024-02-23T00:40:39.395313+0000","last_peered":"2024-02-23T00:40:39.395313+0000","last_clean":"2024-02-23T00:40:39.395313+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:40:39.395313+0000","last_undegraded":"2024-02-23T00:40:39.395313+0000","last_fullsized":"2024-02-23T00:40:39.395313+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.377289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.377289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":598,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.208139+0000","last_change":"2024-02-23T00:40:23.194154+0000","last_active":"2024-02-23T00:40:39.208139+0000","last_peered":"2024-02-23T00:40:39.208139+0000","last_clean":"2024-02-23T00:40:39.208139+0000","last_became_active":"2024-02-23T00:40:23.193867+0000","last_became_peered":"2024-02-23T00:40:23.193867+0000","last_unstale":"2024-02-23T00:40:39.208139+0000","last_undegraded":"2024-02-23T00:40:39.208139+0000","last_fullsized":"2024-02-23T00:40:39.208139+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.791748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.791748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":126,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204488+0000","last_change":"2024-02-23T00:40:11.172134+0000","last_active":"2024-02-23T00:40:39.204488+0000","last_peered":"2024-02-23T00:40:39.204488+0000","last_clean":"2024-02-23T00:40:39.204488+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:40:39.204488+0000","last_undegraded":"2024-02-23T00:40:39.204488+0000","last_fullsized":"2024-02-23T00:40:39.204488+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:11.172065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.172065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":703,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203824+0000","last_change":"2024-02-23T00:38:55.178058+0000","last_active":"2024-02-23T00:40:39.203824+0000","last_peered":"2024-02-23T00:40:39.203824+0000","last_clean":"2024-02-23T00:40:39.203824+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:40:39.203824+0000","last_undegraded":"2024-02-23T00:40:39.203824+0000","last_fullsized":"2024-02-23T00:40:39.203824+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":29,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":124,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204623+0000","last_change":"2024-02-23T00:40:07.255442+0000","last_active":"2024-02-23T00:40:39.204623+0000","last_peered":"2024-02-23T00:40:39.204623+0000","last_clean":"2024-02-23T00:40:39.204623+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:40:39.204623+0000","last_undegraded":"2024-02-23T00:40:39.204623+0000","last_fullsized":"2024-02-23T00:40:39.204623+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:07.255349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:07.255349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":608,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204541+0000","last_change":"2024-02-23T00:38:49.224957+0000","last_active":"2024-02-23T00:40:39.204541+0000","last_peered":"2024-02-23T00:40:39.204541+0000","last_clean":"2024-02-23T00:40:39.204541+0000","last_became_active":"2024-02-23T00:38:49.224630+0000","last_became_peered":"2024-02-23T00:38:49.224630+0000","last_unstale":"2024-02-23T00:40:39.204541+0000","last_undegraded":"2024-02-23T00:40:39.204541+0000","last_fullsized":"2024-02-23T00:40:39.204541+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.063485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.063485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":536,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.206102+0000","last_change":"2024-02-23T00:38:03.798894+0000","last_active":"2024-02-23T00:40:39.206102+0000","last_peered":"2024-02-23T00:40:39.206102+0000","last_clean":"2024-02-23T00:40:39.206102+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:40:39.206102+0000","last_undegraded":"2024-02-23T00:40:39.206102+0000","last_fullsized":"2024-02-23T00:40:39.206102+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.798824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.798824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":773,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204200+0000","last_change":"2024-02-23T00:40:25.188808+0000","last_active":"2024-02-23T00:40:39.204200+0000","last_peered":"2024-02-23T00:40:39.204200+0000","last_clean":"2024-02-23T00:40:39.204200+0000","last_became_active":"2024-02-23T00:40:25.188309+0000","last_became_peered":"2024-02-23T00:40:25.188309+0000","last_unstale":"2024-02-23T00:40:39.204200+0000","last_undegraded":"2024-02-23T00:40:39.204200+0000","last_fullsized":"2024-02-23T00:40:39.204200+0000","mapping_epoch":605,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":304,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203505+0000","last_change":"2024-02-23T00:37:59.129696+0000","last_active":"2024-02-23T00:40:39.203505+0000","last_peered":"2024-02-23T00:40:39.203505+0000","last_clean":"2024-02-23T00:40:39.203505+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:40:39.203505+0000","last_undegraded":"2024-02-23T00:40:39.203505+0000","last_fullsized":"2024-02-23T00:40:39.203505+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:59.129604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:37:59.129604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":593,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.205976+0000","last_change":"2024-02-23T00:40:23.206928+0000","last_active":"2024-02-23T00:40:39.205976+0000","last_peered":"2024-02-23T00:40:39.205976+0000","last_clean":"2024-02-23T00:40:39.205976+0000","last_became_active":"2024-02-23T00:40:23.206276+0000","last_became_peered":"2024-02-23T00:40:23.206276+0000","last_unstale":"2024-02-23T00:40:39.205976+0000","last_undegraded":"2024-02-23T00:40:39.205976+0000","last_fullsized":"2024-02-23T00:40:39.205976+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:43.299839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:43.299839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":550,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394663+0000","last_change":"2024-02-23T00:40:11.671468+0000","last_active":"2024-02-23T00:40:39.394663+0000","last_peered":"2024-02-23T00:40:39.394663+0000","last_clean":"2024-02-23T00:40:39.394663+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:40:39.394663+0000","last_undegraded":"2024-02-23T00:40:39.394663+0000","last_fullsized":"2024-02-23T00:40:39.394663+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:11.671393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.671393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":289,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203381+0000","last_change":"2024-02-23T00:39:21.278362+0000","last_active":"2024-02-23T00:40:39.203381+0000","last_peered":"2024-02-23T00:40:39.203381+0000","last_clean":"2024-02-23T00:40:39.203381+0000","last_became_active":"2024-02-23T00:39:21.278024+0000","last_became_peered":"2024-02-23T00:39:21.278024+0000","last_unstale":"2024-02-23T00:40:39.203381+0000","last_undegraded":"2024-02-23T00:40:39.203381+0000","last_fullsized":"2024-02-23T00:40:39.203381+0000","mapping_epoch":541,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":607,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394008+0000","last_change":"2024-02-23T00:40:23.206017+0000","last_active":"2024-02-23T00:40:39.394008+0000","last_peered":"2024-02-23T00:40:39.394008+0000","last_clean":"2024-02-23T00:40:39.394008+0000","last_became_active":"2024-02-23T00:40:23.205326+0000","last_became_peered":"2024-02-23T00:40:23.205326+0000","last_unstale":"2024-02-23T00:40:39.394008+0000","last_undegraded":"2024-02-23T00:40:39.394008+0000","last_fullsized":"2024-02-23T00:40:39.394008+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":598,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.396062+0000","last_change":"2024-02-23T00:40:19.660039+0000","last_active":"2024-02-23T00:40:39.396062+0000","last_peered":"2024-02-23T00:40:39.396062+0000","last_clean":"2024-02-23T00:40:39.396062+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:40:39.396062+0000","last_undegraded":"2024-02-23T00:40:39.396062+0000","last_fullsized":"2024-02-23T00:40:39.396062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.659970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.659970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":561,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207154+0000","last_change":"2024-02-23T00:39:21.439880+0000","last_active":"2024-02-23T00:40:39.207154+0000","last_peered":"2024-02-23T00:40:39.207154+0000","last_clean":"2024-02-23T00:40:39.207154+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:40:39.207154+0000","last_undegraded":"2024-02-23T00:40:39.207154+0000","last_fullsized":"2024-02-23T00:40:39.207154+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:21.439783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:21.439783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":746,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203126+0000","last_change":"2024-02-23T00:39:24.165093+0000","last_active":"2024-02-23T00:40:39.203126+0000","last_peered":"2024-02-23T00:40:39.203126+0000","last_clean":"2024-02-23T00:40:39.203126+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:40:39.203126+0000","last_undegraded":"2024-02-23T00:40:39.203126+0000","last_fullsized":"2024-02-23T00:40:39.203126+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":134,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204578+0000","last_change":"2024-02-23T00:40:23.199788+0000","last_active":"2024-02-23T00:40:39.204578+0000","last_peered":"2024-02-23T00:40:39.204578+0000","last_clean":"2024-02-23T00:40:39.204578+0000","last_became_active":"2024-02-23T00:40:23.199323+0000","last_became_peered":"2024-02-23T00:40:23.199323+0000","last_unstale":"2024-02-23T00:40:39.204578+0000","last_undegraded":"2024-02-23T00:40:39.204578+0000","last_fullsized":"2024-02-23T00:40:39.204578+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.346259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.346259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":250,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.394577+0000","last_change":"2024-02-23T00:39:31.412402+0000","last_active":"2024-02-23T00:40:39.394577+0000","last_peered":"2024-02-23T00:40:39.394577+0000","last_clean":"2024-02-23T00:40:39.394577+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:40:39.394577+0000","last_undegraded":"2024-02-23T00:40:39.394577+0000","last_fullsized":"2024-02-23T00:40:39.394577+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.412345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.412345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":27,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.395628+0000","last_change":"2024-02-23T00:40:23.199367+0000","last_active":"2024-02-23T00:40:39.395628+0000","last_peered":"2024-02-23T00:40:39.395628+0000","last_clean":"2024-02-23T00:40:39.395628+0000","last_became_active":"2024-02-23T00:40:23.199068+0000","last_became_peered":"2024-02-23T00:40:23.199068+0000","last_unstale":"2024-02-23T00:40:39.395628+0000","last_undegraded":"2024-02-23T00:40:39.395628+0000","last_fullsized":"2024-02-23T00:40:39.395628+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.176441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.176441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":849,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.204644+0000","last_change":"2024-02-23T00:40:23.204704+0000","last_active":"2024-02-23T00:40:39.204644+0000","last_peered":"2024-02-23T00:40:39.204644+0000","last_clean":"2024-02-23T00:40:39.204644+0000","last_became_active":"2024-02-23T00:40:23.203940+0000","last_became_peered":"2024-02-23T00:40:23.203940+0000","last_unstale":"2024-02-23T00:40:39.204644+0000","last_undegraded":"2024-02-23T00:40:39.204644+0000","last_fullsized":"2024-02-23T00:40:39.204644+0000","mapping_epoch":603,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:34.287816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.287816+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":212,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.397198+0000","last_change":"2024-02-23T00:40:23.197829+0000","last_active":"2024-02-23T00:40:39.397198+0000","last_peered":"2024-02-23T00:40:39.397198+0000","last_clean":"2024-02-23T00:40:39.397198+0000","last_became_active":"2024-02-23T00:40:23.197454+0000","last_became_peered":"2024-02-23T00:40:23.197454+0000","last_unstale":"2024-02-23T00:40:39.397198+0000","last_undegraded":"2024-02-23T00:40:39.397198+0000","last_fullsized":"2024-02-23T00:40:39.397198+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:33:42.579021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:33:42.579021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":595,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.203301+0000","last_change":"2024-02-23T00:40:13.237519+0000","last_active":"2024-02-23T00:40:39.203301+0000","last_peered":"2024-02-23T00:40:39.203301+0000","last_clean":"2024-02-23T00:40:39.203301+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:40:39.203301+0000","last_undegraded":"2024-02-23T00:40:39.203301+0000","last_fullsized":"2024-02-23T00:40:39.203301+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:13.237419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:13.237419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":560,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-02-23T00:40:39.207522+0000","last_change":"2024-02-23T00:40:06.086703+0000","last_active":"2024-02-23T00:40:39.207522+0000","last_peered":"2024-02-23T00:40:39.207522+0000","last_clean":"2024-02-23T00:40:39.207522+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:40:39.207522+0000","last_undegraded":"2024-02-23T00:40:39.207522+0000","last_fullsized":"2024-02-23T00:40:39.207522+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:06.086635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:06.086635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":237,"num_pg":32,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":7},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":59,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":24,"acting":24,"num_store_stats":7},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":23,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":69,"acting":69,"num_store_stats":8},{"poolid":5,"num_pg":37,"stat_sum":{"num_bytes":0,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":111,"acting":111,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":217,"num_object_clones":0,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":25,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":1178,"num_read_kb":958,"num_write":708,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":5226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":95,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":481,"num_read_kb":481,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332260,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317132,"kb_used_data":7608,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428020,"statfs":{"total":95995035648,"available":95670292480,"internally_reserved":0,"allocated":7790592,"data_stored":4457140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.02,"15min":1.024},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.79800000000000004,"5min":57.401000000000003,"15min":61.048999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.1719999999999999,"15min":1.2450000000000001},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":65.558999999999997,"15min":73.870999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45900000000000002,"15min":0.68000000000000005},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.217},"max":{"1min":0.69799999999999995,"5min":1.23,"15min":31.175000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.64800000000000002,"15min":0.73899999999999999},"min":{"1min":0.38300000000000001,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":6.742,"5min":11.066000000000001,"15min":29.423999999999999},"last":0.34300000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.68400000000000005,"15min":0.72399999999999998},"min":{"1min":0.34000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.0110000000000001,"5min":8.0099999999999998,"15min":16.603999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.58199999999999996,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":5.7229999999999999,"5min":5.7229999999999999,"15min":7.5339999999999998},"last":0.32100000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.59799999999999998,"15min":1.01},"min":{"1min":0.39200000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72999999999999998,"5min":8.0449999999999999,"15min":57.729999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.53500000000000003,"15min":0.91400000000000003},"min":{"1min":0.35999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68600000000000005,"5min":3.8479999999999999,"15min":27.573},"last":0.35899999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38400000000000001,"15min":0.41399999999999998},"min":{"1min":0.218,"5min":0.214,"15min":0.214},"max":{"1min":0.86299999999999999,"5min":1.2589999999999999,"15min":1.706},"last":0.308},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.45400000000000001,"15min":0.45200000000000001},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66000000000000003,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.24299999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.39400000000000002,"15min":0.42299999999999999},"min":{"1min":0.26600000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.746,"5min":0.746,"15min":1.0209999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.443,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89200000000000002,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.33500000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45500000000000002,"15min":0.48399999999999999},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.91500000000000004,"5min":1.3129999999999999,"15min":2.1339999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.311,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77000000000000002,"5min":5.194,"15min":5.194},"last":0.26000000000000001}]}]},{"osd":6,"up_from":44,"seq":188978561193,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317124,"kb_used_data":7600,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428028,"statfs":{"total":95995035648,"available":95670300672,"internally_reserved":0,"allocated":7782400,"data_stored":4481976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.6499999999999999,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.78800000000000003,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64800000000000002,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":9.468,"15min":9.468},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.54700000000000004,"15min":1.026},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.77200000000000002,"5min":6.056,"15min":59.347999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.76700000000000002,"15min":1.3799999999999999},"min":{"1min":0.24299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.3520000000000003,"5min":18.423999999999999,"15min":87.221000000000004},"last":0.44}]},{"osd":2,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.78100000000000003,"15min":0.90400000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":7.2560000000000002,"5min":18.381,"15min":25.550999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.69499999999999995,"15min":0.91500000000000004},"min":{"1min":0.31,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.0099999999999998,"5min":6.1749999999999998,"15min":40.914000000000001},"last":0.49299999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.626,"15min":0.84799999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":7.1630000000000003,"5min":7.1630000000000003,"15min":31.309999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.77000000000000002,"15min":1.407},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.0940000000000003,"5min":18.448,"15min":86.311999999999998},"last":0.40200000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.53500000000000003,"15min":0.46999999999999997},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.8769999999999998,"5min":7.8769999999999998,"15min":7.8769999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.47999999999999998,"15min":0.52400000000000002},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57999999999999996,"5min":6.0300000000000002,"15min":17.718},"last":0.314}]},{"osd":5,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.54500000000000004,"15min":0.51800000000000002},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":6.7999999999999998,"5min":6.7999999999999998,"15min":8.0990000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.376,"5min":0.498,"15min":0.498},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.54800000000000004,"5min":6.0780000000000003,"15min":8.1370000000000005},"last":0.41599999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.51100000000000001,"15min":0.61199999999999999},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59399999999999997,"5min":6.4279999999999999,"15min":17.881},"last":0.501},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.27700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58599999999999997,"5min":18.475000000000001,"15min":18.475000000000001},"last":0.48199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304892,"kb_used_data":7660,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440260,"statfs":{"total":95995035648,"available":95682826240,"internally_reserved":0,"allocated":7843840,"data_stored":4475330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.29,"15min":0.66300000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.51700000000000002,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.89600000000000002,"15min":0.55200000000000005},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60499999999999998,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:39:35 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.35599999999999998,"15min":0.67400000000000004},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.9980000000000002,"5min":3.9980000000000002,"15min":35.313000000000002},"last":0.187},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.36099999999999999,"15min":0.57799999999999996},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.625,"5min":2.625,"15min":27.908999999999999},"last":0.252}]},{"osd":3,"last update":"Fri Feb 23 00:40:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.40899999999999997,"15min":0.65100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.21299999999999999},"max":{"1min":0.46700000000000003,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.39400000000000002,"15min":0.64300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.48699999999999999,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.26500000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.70299999999999996},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.0230000000000001,"5min":3.4900000000000002,"15min":16.231000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51900000000000002,"15min":0.71599999999999997},"min":{"1min":0.317,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.629,"5min":1.518,"15min":29.212},"last":0.60899999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:39:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.55200000000000005,"15min":0.71999999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":16.294},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.58899999999999997,"15min":0.69099999999999995},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.3380000000000001,"5min":4.3380000000000001,"15min":11.105},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:40:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.754},"min":{"1min":0.33400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":29.187000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58999999999999997,"15min":0.748},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":2.0499999999999998,"5min":2.528,"15min":16.369},"last":0.503}]},{"osd":7,"last update":"Fri Feb 23 00:40:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.63500000000000001,"15min":0.78800000000000003},"min":{"1min":0.39100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.895,"5min":5.2119999999999997,"15min":16.395},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.60599999999999998,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70199999999999996,"5min":3.7669999999999999,"15min":11.135999999999999},"last":0.53900000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411516,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313000,"kb_used_data":7580,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432152,"statfs":{"total":95995035648,"available":95674523648,"internally_reserved":0,"allocated":7761920,"data_stored":4467769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":1.0600000000000001,"15min":0.754},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.65500000000000003,"5min":56.509,"15min":56.509},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.59799999999999998,"15min":0.71199999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56799999999999995,"5min":19.571999999999999,"15min":31.498999999999999},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:39:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.45100000000000001,"15min":0.60099999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.54100000000000004,"5min":7.7110000000000003,"15min":19.324999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.318,"5min":0.504,"15min":0.75700000000000001},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.66100000000000003,"5min":7.9779999999999998,"15min":35.052},"last":0.56299999999999994}]},{"osd":3,"last update":"Fri Feb 23 00:40:09 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.56899999999999995,"15min":0.74399999999999999},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.88400000000000001,"5min":10.032,"15min":43.5},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.61599999999999999,"15min":0.69699999999999995},"min":{"1min":0.245,"5min":0.22,"15min":0.22},"max":{"1min":1.7490000000000001,"5min":11.815,"15min":27.414000000000001},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.68600000000000005,"15min":0.67000000000000004},"min":{"1min":0.33000000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.84299999999999997,"5min":10.106,"15min":14.614000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.34899999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81100000000000005,"5min":7.766,"15min":7.766},"last":0.94399999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68500000000000005,"15min":0.71099999999999997},"min":{"1min":0.39800000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.83099999999999996,"5min":7.915,"15min":17.765000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.66700000000000004,"15min":0.67300000000000004},"min":{"1min":0.35799999999999998,"5min":0.33400000000000002,"15min":0.317},"max":{"1min":0.67600000000000005,"5min":10.138,"15min":14.589},"last":0.99399999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:40:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.628,"15min":0.628},"min":{"1min":0.36299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77500000000000002,"5min":7.7869999999999999,"15min":7.7869999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.73699999999999999,"15min":0.79600000000000004},"min":{"1min":0.38800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86299999999999999,"5min":7.9589999999999996,"15min":20.574999999999999},"last":0.878}]},{"osd":7,"last update":"Fri Feb 23 00:40:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.72999999999999998,"15min":1.075},"min":{"1min":0.38400000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":1.1359999999999999,"5min":7.9480000000000004,"15min":66.632000000000005},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.72899999999999998,"15min":0.84699999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.3660000000000001,"5min":10.163,"15min":19.603999999999999},"last":1.0149999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149880,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308976,"kb_used_data":7648,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436176,"statfs":{"total":95995035648,"available":95678644224,"internally_reserved":0,"allocated":7831552,"data_stored":4483076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":1.506,"15min":0.745},"min":{"1min":0.252,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.76100000000000001,"5min":55.238999999999997,"15min":55.238999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":1.4750000000000001,"15min":0.747},"min":{"1min":0.26100000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.47699999999999998,"5min":54.563000000000002,"15min":54.563000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.375,"15min":0.61599999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":1.778,"5min":2.718,"15min":54.351999999999997},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.46500000000000002,"15min":0.78700000000000003},"min":{"1min":0.253,"5min":0.219,"15min":0.219},"max":{"1min":3.7120000000000002,"5min":3.7120000000000002,"15min":55.337000000000003},"last":0.30199999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.45700000000000002,"15min":0.77100000000000002},"min":{"1min":0.23899999999999999,"5min":0.223,"15min":0.223},"max":{"1min":3.7570000000000001,"5min":3.7570000000000001,"15min":55.368000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.45200000000000001,"15min":0.74399999999999999},"min":{"1min":0.249,"5min":0.214,"15min":0.185},"max":{"1min":2.4710000000000001,"5min":4.181,"15min":54.383000000000003},"last":0.26900000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63400000000000001,"15min":0.94199999999999995},"min":{"1min":0.34000000000000002,"5min":0.316,"15min":0.27900000000000003},"max":{"1min":1.6220000000000001,"5min":6.8799999999999999,"15min":54.530000000000001},"last":0.443},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.59299999999999997,"15min":0.629},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.24099999999999999},"max":{"1min":1.5369999999999999,"5min":4.9829999999999997,"15min":8.0359999999999996},"last":0.46100000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.55600000000000005,"15min":0.88300000000000001},"min":{"1min":0.34699999999999998,"5min":0.33700000000000002,"15min":0.28899999999999998},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":55.401000000000003},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.212,"5min":0.66700000000000004,"15min":0.873},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30299999999999999},"max":{"1min":7.0179999999999998,"5min":7.0179999999999998,"15min":54.454999999999998},"last":0.41799999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.73899999999999999,"15min":0.84799999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.5660000000000001,"5min":9.7490000000000006,"15min":54.305},"last":0.433},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79800000000000004,"15min":1.022},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.8420000000000001,"5min":9.6820000000000004,"15min":55.424999999999997},"last":0.39200000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.64100000000000001,"15min":0.79000000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.1859999999999999,"5min":4.9790000000000001,"15min":54.222000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":2.113,"5min":0.94499999999999995,"15min":1.038},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":23.928000000000001,"5min":23.928000000000001,"15min":55.460000000000001},"last":0.45200000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986355,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308948,"kb_used_data":7616,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436204,"statfs":{"total":95995035648,"available":95678672896,"internally_reserved":0,"allocated":7798784,"data_stored":4478740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1739999999999999,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.68799999999999994,"5min":21.419,"15min":21.419},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":2.1200000000000001,"15min":1.2589999999999999},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.83699999999999997,"15min":1.095},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.242,"5min":16.696000000000002,"15min":41.137999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.71699999999999997,"15min":0.90100000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.627,"5min":13.455,"15min":26.210000000000001},"last":0.45100000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.83799999999999997,"15min":1.179},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":9.3490000000000002,"5min":13.478,"15min":67.253},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.7070000000000001,"5min":0.88900000000000001,"15min":1.121},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":14.442,"5min":14.442,"15min":41.927999999999997},"last":0.45700000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.77900000000000003,"15min":1.121},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":36.301000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.78400000000000003,"15min":1.1899999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.24299999999999999},"max":{"1min":3.6819999999999999,"5min":16.657,"15min":42.064},"last":0.60599999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:39:46 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.42199999999999999,"15min":0.55700000000000005},"min":{"1min":0.23699999999999999,"5min":0.219,"15min":0.217},"max":{"1min":0.44500000000000001,"5min":2.3210000000000002,"15min":14.242000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.53500000000000003,"15min":0.59699999999999998},"min":{"1min":0.27200000000000002,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.55700000000000005,"5min":7.1459999999999999,"15min":14.124000000000001},"last":0.32300000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.54000000000000004,"15min":0.56799999999999995},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":0.58899999999999997,"5min":7.1189999999999998,"15min":8.0730000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.44700000000000001,"15min":0.63},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.54300000000000004,"5min":2.2450000000000001,"15min":45.814999999999998},"last":0.313}]},{"osd":7,"last update":"Fri Feb 23 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.58299999999999996,"15min":0.63800000000000001},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.9210000000000003,"5min":7.9210000000000003,"15min":17.178999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72599999999999998,"15min":0.71299999999999997},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.28299999999999997},"max":{"1min":6.3220000000000001,"5min":15.85,"15min":15.85},"last":0.38200000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757422,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317108,"kb_used_data":7584,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428044,"statfs":{"total":95995035648,"available":95670317056,"internally_reserved":0,"allocated":7766016,"data_stored":4464843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.1200000000000001,"15min":0.70699999999999996},"min":{"1min":0.36699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.115,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":1.028},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.498,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.49199999999999999,"15min":0.753},"min":{"1min":0.27700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58499999999999996,"5min":3.7629999999999999,"15min":38.838000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.54600000000000004,"15min":1.1850000000000001},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53000000000000003,"5min":3.7250000000000001,"15min":127.794},"last":0.41899999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.72499999999999998,"15min":0.95899999999999996},"min":{"1min":0.26800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":6.7869999999999999,"5min":11.435,"15min":37.442999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.52200000000000002,"15min":0.70099999999999996},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.81899999999999995,"5min":3.734,"15min":23.297999999999998},"last":0.438}]},{"osd":3,"last update":"Fri Feb 23 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.74399999999999999,"15min":0.82099999999999995},"min":{"1min":0.34899999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.052,"5min":12.507,"15min":24.414999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.59199999999999997,"15min":0.91100000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.31},"max":{"1min":0.96299999999999997,"5min":3.8119999999999998,"15min":40.865000000000002},"last":0.40300000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:39:52 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.39400000000000002,"15min":0.78000000000000003},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.219},"max":{"1min":0.54400000000000004,"5min":2.8940000000000001,"15min":103.13},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.55400000000000005,"15min":0.51100000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.65100000000000002,"5min":11.381,"15min":12.795999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.59899999999999998,"15min":0.54300000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26200000000000001},"max":{"1min":0.58199999999999996,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.44700000000000001,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64200000000000002,"5min":1.5229999999999999,"15min":103.61499999999999},"last":0.497}]},{"osd":7,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.59799999999999998,"15min":0.56100000000000005},"min":{"1min":0.29899999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.625,"5min":11.478,"15min":11.478},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.58499999999999996,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:40:42.125 INFO:tasks.thrashosds.thrasher:Setting 4.c to [7, 3, 1, 2, 4, 0] 2024-02-23T00:40:42.125 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.c', '7', '3', '1', '2', '4', '0'] 2024-02-23T00:40:42.126 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pg-upmap-items 4.c 7 3 1 2 4 0 2024-02-23T00:40:42.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:42.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.185+0000 7f42a97fa700 1 -- 172.21.15.87:0/3407417391 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v623) v1 ==== 91+0+0 (secure 0 0 0) 0x7f42a00517b0 con 0x7f42ac0f9610 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 10 librados: watch_flush enter 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 10 librados: watch_flush exit 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 20 client.27630.objecter shutdown clearing up homeless session... 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 10 client.27630.objecter successfully canceled tick 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4294062c90 msgr2=0x7f4294065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/3407417391 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4294062c90 0x7f4294065150 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 rx=0x7f429c0041d0 tx=0x7f429c009040).stop 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 msgr2=0x7f42ac10e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 0x7f42ac10e630 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 rx=0x7f42a0009b10 tx=0x7f42a0009cb0).stop 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 shutdown_connections 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/3407417391 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4294062c90 0x7f4294065150 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:42.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f42ac102210 0x7f42ac108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:42.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f42ac0f8cb0 0x7f42ac10e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:42.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 --2- 172.21.15.87:0/3407417391 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f42ac0f9610 0x7f42ac10e630 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:42.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.188+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 >> 172.21.15.87:0/3407417391 conn(0x7f42ac0f2820 msgr2=0x7f42ac0f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:42.192 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:42.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.194+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 shutdown_connections 2024-02-23T00:40:42.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.194+0000 7f42b2a0d700 1 -- 172.21.15.87:0/3407417391 wait complete. 2024-02-23T00:40:42.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:42.194+0000 7f42b2a0d700 1 librados: shutdown 2024-02-23T00:40:42.197 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:40:42.227 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:42.255 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:42.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:42] "GET /metrics HTTP/1.1" 200 179522 "" "Prometheus/2.33.4" 2024-02-23T00:40:42.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:41.123836+0000 mgr.y (mgr.24437) 551 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:42.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:41.174752+0000 osd.2 (osd.2) 227 : cluster [DBG] 237.4 scrub starts 2024-02-23T00:40:42.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:41.175918+0000 osd.2 (osd.2) 228 : cluster [DBG] 237.4 scrub ok 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:41.272485+0000 mgr.y (mgr.24437) 552 : audit [DBG] from='client.38096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:41.599373+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? 172.21.15.87:0/3407417391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:41.753593+0000 mon.a (mon.0) 3310 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:41.753764+0000 mon.a (mon.0) 3311 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:42.177332+0000 mon.a (mon.0) 3312 : cluster [WRN] Health check update: Degraded data redundancy: 5/732 objects degraded (0.683%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:42.178096+0000 mon.a (mon.0) 3313 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:42.183475+0000 mon.a (mon.0) 3314 : audit [INF] from='client.? 172.21.15.87:0/3407417391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:42.183543+0000 mon.a (mon.0) 3315 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:42.183603+0000 mon.a (mon.0) 3316 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-23T00:40:42.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:42.183657+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e623: 8 total, 7 up, 7 in 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:41.123836+0000 mgr.y (mgr.24437) 551 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:41.174752+0000 osd.2 (osd.2) 227 : cluster [DBG] 237.4 scrub starts 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:41.175918+0000 osd.2 (osd.2) 228 : cluster [DBG] 237.4 scrub ok 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:41.272485+0000 mgr.y (mgr.24437) 552 : audit [DBG] from='client.38096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:41.599373+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? 172.21.15.87:0/3407417391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:41.753593+0000 mon.a (mon.0) 3310 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:41.753764+0000 mon.a (mon.0) 3311 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:42.177332+0000 mon.a (mon.0) 3312 : cluster [WRN] Health check update: Degraded data redundancy: 5/732 objects degraded (0.683%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:42.178096+0000 mon.a (mon.0) 3313 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:42.183475+0000 mon.a (mon.0) 3314 : audit [INF] from='client.? 172.21.15.87:0/3407417391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:42.183543+0000 mon.a (mon.0) 3315 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:42.183603+0000 mon.a (mon.0) 3316 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-23T00:40:42.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:42.183657+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e623: 8 total, 7 up, 7 in 2024-02-23T00:40:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:41.123836+0000 mgr.y (mgr.24437) 551 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:41.174752+0000 osd.2 (osd.2) 227 : cluster [DBG] 237.4 scrub starts 2024-02-23T00:40:42.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:41.175918+0000 osd.2 (osd.2) 228 : cluster [DBG] 237.4 scrub ok 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:41.272485+0000 mgr.y (mgr.24437) 552 : audit [DBG] from='client.38096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:41.599373+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? 172.21.15.87:0/3407417391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:41.753593+0000 mon.a (mon.0) 3310 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:41.753764+0000 mon.a (mon.0) 3311 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:42.177332+0000 mon.a (mon.0) 3312 : cluster [WRN] Health check update: Degraded data redundancy: 5/732 objects degraded (0.683%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:42.178096+0000 mon.a (mon.0) 3313 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:42.183475+0000 mon.a (mon.0) 3314 : audit [INF] from='client.? 172.21.15.87:0/3407417391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:42.183543+0000 mon.a (mon.0) 3315 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:42.183603+0000 mon.a (mon.0) 3316 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-23T00:40:42.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:42.183657+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e623: 8 total, 7 up, 7 in 2024-02-23T00:40:42.459 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:42.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:42.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:42.468 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-23T00:40:42.563 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:42.589 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:42.608 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:42.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:42.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:42.739 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:43.480 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:41.752738+0000 mgr.y (mgr.24437) 553 : cluster [DBG] pgmap v950: 210 pgs: 2 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 2 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:43.480 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:42.780316+0000 osd.7 (osd.7) 284 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:40:43.480 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:42.783203+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:40:43.480 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:43.184403+0000 mon.a (mon.0) 3318 : cluster [DBG] osdmap e624: 8 total, 7 up, 7 in 2024-02-23T00:40:43.480 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:41.752738+0000 mgr.y (mgr.24437) 553 : cluster [DBG] pgmap v950: 210 pgs: 2 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 2 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:43.480 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:42.780316+0000 osd.7 (osd.7) 284 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:40:43.480 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:42.783203+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:40:43.480 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:43.184403+0000 mon.a (mon.0) 3318 : cluster [DBG] osdmap e624: 8 total, 7 up, 7 in 2024-02-23T00:40:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:41.752738+0000 mgr.y (mgr.24437) 553 : cluster [DBG] pgmap v950: 210 pgs: 2 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 2 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:42.780316+0000 osd.7 (osd.7) 284 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:40:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:42.783203+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:40:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:43.184403+0000 mon.a (mon.0) 3318 : cluster [DBG] osdmap e624: 8 total, 7 up, 7 in 2024-02-23T00:40:43.790 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:43.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:43.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:43.828 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:43.832 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:43.893 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:43.921 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:44.120 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:44.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:44.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:40:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:40:44.224 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:44.254 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:44.275 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:44.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:44.379 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:44.396 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:42.566581+0000 osd.4 (osd.4) 273 : cluster [DBG] 5.e scrub starts 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:42.570048+0000 osd.4 (osd.4) 274 : cluster [DBG] 5.e scrub ok 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:43.230099+0000 osd.3 (osd.3) 193 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:43.233601+0000 osd.3 (osd.3) 194 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:43.964633+0000 osd.1 (osd.1) 284 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:43.964707+0000 osd.1 (osd.1) 285 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:44.187464+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e625: 8 total, 7 up, 7 in 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:42.566581+0000 osd.4 (osd.4) 273 : cluster [DBG] 5.e scrub starts 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:42.570048+0000 osd.4 (osd.4) 274 : cluster [DBG] 5.e scrub ok 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:43.230099+0000 osd.3 (osd.3) 193 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:40:44.397 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:43.233601+0000 osd.3 (osd.3) 194 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:40:44.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:43.964633+0000 osd.1 (osd.1) 284 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:40:44.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:43.964707+0000 osd.1 (osd.1) 285 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:40:44.398 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:44.187464+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e625: 8 total, 7 up, 7 in 2024-02-23T00:40:44.408 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:44.427 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:44.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:44.531 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:44.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:42.566581+0000 osd.4 (osd.4) 273 : cluster [DBG] 5.e scrub starts 2024-02-23T00:40:44.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:42.570048+0000 osd.4 (osd.4) 274 : cluster [DBG] 5.e scrub ok 2024-02-23T00:40:44.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:43.230099+0000 osd.3 (osd.3) 193 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:40:44.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:43.233601+0000 osd.3 (osd.3) 194 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:40:44.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:43.964633+0000 osd.1 (osd.1) 284 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:40:44.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:43.964707+0000 osd.1 (osd.1) 285 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:40:44.557 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:44.187464+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e625: 8 total, 7 up, 7 in 2024-02-23T00:40:44.558 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:44.577 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:44.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:44.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:44.681 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:44.708 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:45.059 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:45.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:45.164 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:45.204 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:43.521874+0000 osd.4 (osd.4) 275 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:40:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:43.522841+0000 osd.4 (osd.4) 276 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:40:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:43.753426+0000 mgr.y (mgr.24437) 554 : cluster [DBG] pgmap v953: 210 pgs: 2 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 2 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.002838+0000 osd.5 (osd.5) 237 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:40:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.004423+0000 osd.5 (osd.5) 238 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:40:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.187970+0000 mon.a (mon.0) 3320 : cluster [DBG] osdmap e626: 8 total, 7 up, 7 in 2024-02-23T00:40:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:43.521874+0000 osd.4 (osd.4) 275 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:40:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:43.522841+0000 osd.4 (osd.4) 276 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:40:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:43.753426+0000 mgr.y (mgr.24437) 554 : cluster [DBG] pgmap v953: 210 pgs: 2 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 2 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.002838+0000 osd.5 (osd.5) 237 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:40:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.004423+0000 osd.5 (osd.5) 238 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:40:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.187970+0000 mon.a (mon.0) 3320 : cluster [DBG] osdmap e626: 8 total, 7 up, 7 in 2024-02-23T00:40:45.337 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:40:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:43.521874+0000 osd.4 (osd.4) 275 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:40:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:43.522841+0000 osd.4 (osd.4) 276 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:40:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:43.753426+0000 mgr.y (mgr.24437) 554 : cluster [DBG] pgmap v953: 210 pgs: 2 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 2 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:45.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.002838+0000 osd.5 (osd.5) 237 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:40:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.004423+0000 osd.5 (osd.5) 238 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:40:45.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.187970+0000 mon.a (mon.0) 3320 : cluster [DBG] osdmap e626: 8 total, 7 up, 7 in 2024-02-23T00:40:45.756 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:45.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:45.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:45.888 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:46.236 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:46.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:46.250 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.169613+0000 osd.1 (osd.1) 286 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:40:46.250 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.172538+0000 osd.1 (osd.1) 287 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:40:46.250 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.428330+0000 osd.6 (osd.6) 244 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:40:46.250 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.431808+0000 osd.6 (osd.6) 245 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:40:46.250 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.578426+0000 osd.4 (osd.4) 277 : cluster [DBG] 2.f scrub starts 2024-02-23T00:40:46.250 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.581895+0000 osd.4 (osd.4) 278 : cluster [DBG] 2.f scrub ok 2024-02-23T00:40:46.250 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:45.755575+0000 mon.a (mon.0) 3321 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "35"}]: dispatch 2024-02-23T00:40:46.250 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:45.755854+0000 mon.a (mon.0) 3322 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:46.003370+0000 osd.5 (osd.5) 239 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:46.004302+0000 osd.5 (osd.5) 240 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.169613+0000 osd.1 (osd.1) 286 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.172538+0000 osd.1 (osd.1) 287 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.428330+0000 osd.6 (osd.6) 244 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.431808+0000 osd.6 (osd.6) 245 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.578426+0000 osd.4 (osd.4) 277 : cluster [DBG] 2.f scrub starts 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.581895+0000 osd.4 (osd.4) 278 : cluster [DBG] 2.f scrub ok 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:45.755575+0000 mon.a (mon.0) 3321 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "35"}]: dispatch 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:45.755854+0000 mon.a (mon.0) 3322 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-23T00:40:46.251 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:46.003370+0000 osd.5 (osd.5) 239 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:40:46.252 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:46.004302+0000 osd.5 (osd.5) 240 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:40:46.340 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:46.368 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:46.566 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:46.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:46.670 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.169613+0000 osd.1 (osd.1) 286 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:40:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.172538+0000 osd.1 (osd.1) 287 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:40:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.428330+0000 osd.6 (osd.6) 244 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:40:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.431808+0000 osd.6 (osd.6) 245 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:40:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.578426+0000 osd.4 (osd.4) 277 : cluster [DBG] 2.f scrub starts 2024-02-23T00:40:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.581895+0000 osd.4 (osd.4) 278 : cluster [DBG] 2.f scrub ok 2024-02-23T00:40:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:45.755575+0000 mon.a (mon.0) 3321 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "35"}]: dispatch 2024-02-23T00:40:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:45.755854+0000 mon.a (mon.0) 3322 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-23T00:40:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:46.003370+0000 osd.5 (osd.5) 239 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:40:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:46.004302+0000 osd.5 (osd.5) 240 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:40:46.698 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:46.797 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:46.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:46.901 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:46.934 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:46.956 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:46.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:47.060 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:47.089 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:47.162 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:47.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:47.266 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:47.300 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:47.320 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:47.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:47.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:47.452 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:45.754536+0000 mgr.y (mgr.24437) 555 : cluster [DBG] pgmap v956: 208 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:46.248389+0000 mon.a (mon.0) 3323 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "35"}]': finished 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:46.248458+0000 mon.a (mon.0) 3324 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:46.248505+0000 mon.a (mon.0) 3325 : cluster [DBG] osdmap e627: 8 total, 7 up, 7 in 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:46.424697+0000 osd.6 (osd.6) 246 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:46.427684+0000 osd.6 (osd.6) 247 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:46.620132+0000 osd.4 (osd.4) 279 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:46.621720+0000 osd.4 (osd.4) 280 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:46.768418+0000 osd.7 (osd.7) 286 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:46.771460+0000 osd.7 (osd.7) 287 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:47.179755+0000 mon.a (mon.0) 3326 : cluster [WRN] Health check update: Degraded data redundancy: 5/738 objects degraded (0.678%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:45.754536+0000 mgr.y (mgr.24437) 555 : cluster [DBG] pgmap v956: 208 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:46.248389+0000 mon.a (mon.0) 3323 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "35"}]': finished 2024-02-23T00:40:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:46.248458+0000 mon.a (mon.0) 3324 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-23T00:40:47.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:46.248505+0000 mon.a (mon.0) 3325 : cluster [DBG] osdmap e627: 8 total, 7 up, 7 in 2024-02-23T00:40:47.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:46.424697+0000 osd.6 (osd.6) 246 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:40:47.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:46.427684+0000 osd.6 (osd.6) 247 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:40:47.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:46.620132+0000 osd.4 (osd.4) 279 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:40:47.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:46.621720+0000 osd.4 (osd.4) 280 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:40:47.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:46.768418+0000 osd.7 (osd.7) 286 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:40:47.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:46.771460+0000 osd.7 (osd.7) 287 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:40:47.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:47.179755+0000 mon.a (mon.0) 3326 : cluster [WRN] Health check update: Degraded data redundancy: 5/738 objects degraded (0.678%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:45.754536+0000 mgr.y (mgr.24437) 555 : cluster [DBG] pgmap v956: 208 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:46.248389+0000 mon.a (mon.0) 3323 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "35"}]': finished 2024-02-23T00:40:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:46.248458+0000 mon.a (mon.0) 3324 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-23T00:40:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:46.248505+0000 mon.a (mon.0) 3325 : cluster [DBG] osdmap e627: 8 total, 7 up, 7 in 2024-02-23T00:40:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:46.424697+0000 osd.6 (osd.6) 246 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:40:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:46.427684+0000 osd.6 (osd.6) 247 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:40:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:46.620132+0000 osd.4 (osd.4) 279 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:40:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:46.621720+0000 osd.4 (osd.4) 280 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:40:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:46.768418+0000 osd.7 (osd.7) 286 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:40:47.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:46.771460+0000 osd.7 (osd.7) 287 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:40:47.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:47.179755+0000 mon.a (mon.0) 3326 : cluster [WRN] Health check update: Degraded data redundancy: 5/738 objects degraded (0.678%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:48.101 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:48.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:48.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:48.205 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:48.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.959+0000 7f49e2c6a700 1 -- 172.21.15.87:0/3460691523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49dc0fcbb0 msgr2=0x7f49dc0fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:48.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.959+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/3460691523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49dc0fcbb0 0x7f49dc0fcf90 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 rx=0x7f49d0004870 tx=0x7f49d0009cb0).stop 2024-02-23T00:40:48.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.959+0000 7f49e2c6a700 1 -- 172.21.15.87:0/3460691523 shutdown_connections 2024-02-23T00:40:48.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.959+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/3460691523 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49dc0f7400 0x7f49dc103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:48.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.959+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/3460691523 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49dc0fcbb0 0x7f49dc0fcf90 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:48.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.959+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/3460691523 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49dc0fd510 0x7f49dc0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:48.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.959+0000 7f49e2c6a700 1 -- 172.21.15.87:0/3460691523 >> 172.21.15.87:0/3460691523 conn(0x7f49dc0f2820 msgr2=0x7f49dc0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:48.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 -- 172.21.15.87:0/3460691523 shutdown_connections 2024-02-23T00:40:48.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 -- 172.21.15.87:0/3460691523 wait complete. 2024-02-23T00:40:48.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 librados: starting msgr at 2024-02-23T00:40:48.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 librados: starting objecter 2024-02-23T00:40:48.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 Processor -- start 2024-02-23T00:40:48.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 -- start start 2024-02-23T00:40:48.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 librados: setting wanted keys 2024-02-23T00:40:48.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 librados: calling monclient init 2024-02-23T00:40:48.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49dc0f7400 0x7f49dc10d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49dc0fcbb0 0x7f49dc10d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49dc0fd510 0x7f49dc107280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc107770 con 0x7f49dc0f7400 2024-02-23T00:40:48.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc1078e0 con 0x7f49dc0fcbb0 2024-02-23T00:40:48.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e2c6a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc107aa0 con 0x7f49dc0fd510 2024-02-23T00:40:48.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e0a06700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49dc0f7400 0x7f49dc10d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e0a06700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49dc0f7400 0x7f49dc10d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:48062/0 (socket says 172.21.15.87:48062) 2024-02-23T00:40:48.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e0a06700 1 -- 172.21.15.87:0/2250174298 learned_addr learned my addr 172.21.15.87:0/2250174298 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:48.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.960+0000 7f49e1207700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49dc0fd510 0x7f49dc107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e0a06700 1 -- 172.21.15.87:0/2250174298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49dc0fd510 msgr2=0x7f49dc107280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:48.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e0a06700 1 --2- 172.21.15.87:0/2250174298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f49dc0fd510 0x7f49dc107280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:48.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e0a06700 1 -- 172.21.15.87:0/2250174298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49dc0fcbb0 msgr2=0x7f49dc10d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:48.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49dbfff700 1 --2- 172.21.15.87:0/2250174298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49dc0fcbb0 0x7f49dc10d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e0a06700 1 --2- 172.21.15.87:0/2250174298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f49dc0fcbb0 0x7f49dc10d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:48.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e0a06700 1 -- 172.21.15.87:0/2250174298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d40097c0 con 0x7f49dc0f7400 2024-02-23T00:40:48.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e0a06700 1 --2- 172.21.15.87:0/2250174298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f49dc0f7400 0x7f49dc10d0e0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 rx=0x7f49d0009eb0 tx=0x7f49d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:48.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49d9ffb700 10 client.?.objecter ms_handle_connect 0x7f49dc0f7400 2024-02-23T00:40:48.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49d9ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:48.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d000e070 con 0x7f49dc0f7400 2024-02-23T00:40:48.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49d0013da0 con 0x7f49dc0f7400 2024-02-23T00:40:48.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d0005040 con 0x7f49dc0f7400 2024-02-23T00:40:48.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0019070 con 0x7f49dc0f7400 2024-02-23T00:40:48.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e2c6a700 10 client.27636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:48.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49dc107f40 con 0x7f49dc0f7400 2024-02-23T00:40:48.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.961+0000 7f49e2c6a700 1 librados: init done 2024-02-23T00:40:48.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.962+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f49dc108130 con 0x7f49dc0f7400 2024-02-23T00:40:48.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f49d001e420 con 0x7f49dc0f7400 2024-02-23T00:40:48.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6800/1231597463 conn(0x7f49cc062c90 0x7f49cc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(628..628 src has 173..628) v4 ==== 8516+0+0 (secure 0 0 0) 0x7f49d007dff0 con 0x7f49dc0f7400 2024-02-23T00:40:48.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 osd_map(628..628 src has 173..628) v4 2024-02-23T00:40:48.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 3 client.27636.objecter handle_osd_map got epochs [628,628] > 0 2024-02-23T00:40:48.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 3 client.27636.objecter handle_osd_map decoding full epoch 628 2024-02-23T00:40:48.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49dbfff700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6800/1231597463 conn(0x7f49cc062c90 0x7f49cc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 20 client.27636.objecter dump_active .. 0 homeless 2024-02-23T00:40:48.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v628) v1 ==== 54+0+16 (secure 0 0 0) 0x7f49d0080620 con 0x7f49dc0f7400 2024-02-23T00:40:48.240 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:40:48.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 10 client.27636.objecter ms_handle_connect 0x7f49cc062c90 2024-02-23T00:40:48.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49dbfff700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6800/1231597463 conn(0x7f49cc062c90 0x7f49cc065150 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 rx=0x7f49dc108a20 tx=0x7f49c8009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:48.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:48.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 1 2024-02-23T00:40:48.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49e2c6a700 20 client.27636.objecter _get_session osd=-1 returning homeless 2024-02-23T00:40:48.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49e2c6a700 10 client.27636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:48.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=629}) v3 -- 0x7f49dc05fa60 con 0x7f49dc0f7400 2024-02-23T00:40:48.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f49dc05ee80 con 0x7f49dc0f7400 2024-02-23T00:40:48.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_get_version_reply(handle=1 version=628) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49d00512f0 con 0x7f49dc0f7400 2024-02-23T00:40:48.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49daffd700 10 client.27636.objecter _check_command_map_dne tid 1 current 628 map_dne_bound 628 2024-02-23T00:40:48.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49daffd700 10 client.27636.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:40:48.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.963+0000 7f49daffd700 15 client.27636.objecter _session_command_op_remove -1 1 2024-02-23T00:40:48.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:48.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 2 2024-02-23T00:40:48.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6805/685354171 conn(0x7f49dc028540 0x7f49dc0fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc1060e0 osd=1 3 2024-02-23T00:40:48.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc1060e0 osd=1 3 2024-02-23T00:40:48.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc1060e0 osd=1 3 2024-02-23T00:40:48.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 2 2024-02-23T00:40:48.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc1060e0 osd=1 3 2024-02-23T00:40:48.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 1 2 2024-02-23T00:40:48.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc1060e0 osd=1 4 2024-02-23T00:40:48.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 10 client.27636.objecter _send_command 2 2024-02-23T00:40:48.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49dc05a040 con 0x7f49dc028540 2024-02-23T00:40:48.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e1207700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6805/685354171 conn(0x7f49dc028540 0x7f49dc0fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49d9ffb700 10 client.27636.objecter ms_handle_connect 0x7f49dc028540 2024-02-23T00:40:48.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.964+0000 7f49e1207700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6805/685354171 conn(0x7f49dc028540 0x7f49dc0fd9b0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:48.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.965+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f49dc05a040 con 0x7f49dc028540 2024-02-23T00:40:48.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.965+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:40:48.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.965+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 2 = system:0 2024-02-23T00:40:48.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.965+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc1060e0 osd=1 4 2024-02-23T00:40:48.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.965+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 1 2 2024-02-23T00:40:48.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:48.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 3 2024-02-23T00:40:48.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc1060e0 osd=1 3 2024-02-23T00:40:48.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc1060e0 osd=1 3 2024-02-23T00:40:48.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc1060e0 osd=1 3 2024-02-23T00:40:48.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 3 2024-02-23T00:40:48.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc1060e0 osd=1 3 2024-02-23T00:40:48.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 1 3 2024-02-23T00:40:48.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc1060e0 osd=1 4 2024-02-23T00:40:48.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 10 client.27636.objecter _send_command 3 2024-02-23T00:40:48.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.980+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f49dc0ff020 con 0x7f49dc028540 2024-02-23T00:40:48.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f49dc0ff020 con 0x7f49dc028540 2024-02-23T00:40:48.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:48.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc1060e0 osd=1 4 2024-02-23T00:40:48.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 1 3 2024-02-23T00:40:48.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:48.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 4 2024-02-23T00:40:48.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6809/3605394228 conn(0x7f49dc0280a0 0x7f49dc117f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0ff190 osd=2 3 2024-02-23T00:40:48.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0ff190 osd=2 3 2024-02-23T00:40:48.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0ff190 osd=2 3 2024-02-23T00:40:48.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 4 2024-02-23T00:40:48.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc0ff190 osd=2 3 2024-02-23T00:40:48.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 2 4 2024-02-23T00:40:48.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0ff190 osd=2 4 2024-02-23T00:40:48.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 10 client.27636.objecter _send_command 4 2024-02-23T00:40:48.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49dc0597e0 con 0x7f49dc0280a0 2024-02-23T00:40:48.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.982+0000 7f49e0a06700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6809/3605394228 conn(0x7f49dc0280a0 0x7f49dc117f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.983+0000 7f49d9ffb700 10 client.27636.objecter ms_handle_connect 0x7f49dc0280a0 2024-02-23T00:40:48.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.983+0000 7f49e0a06700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6809/3605394228 conn(0x7f49dc0280a0 0x7f49dc117f30 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:48.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.983+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f49dc0597e0 con 0x7f49dc0280a0 2024-02-23T00:40:48.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.983+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:40:48.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.983+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 4 = system:0 2024-02-23T00:40:48.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.983+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc0ff190 osd=2 4 2024-02-23T00:40:48.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.983+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 2 4 2024-02-23T00:40:48.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:48.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 5 2024-02-23T00:40:48.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0ff190 osd=2 3 2024-02-23T00:40:48.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0ff190 osd=2 3 2024-02-23T00:40:48.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0ff190 osd=2 3 2024-02-23T00:40:48.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 5 2024-02-23T00:40:48.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc0ff190 osd=2 3 2024-02-23T00:40:48.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 2 5 2024-02-23T00:40:48.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0ff190 osd=2 4 2024-02-23T00:40:48.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 10 client.27636.objecter _send_command 5 2024-02-23T00:40:48.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.998+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f49dc027600 con 0x7f49dc0280a0 2024-02-23T00:40:48.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.999+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f49dc027600 con 0x7f49dc0280a0 2024-02-23T00:40:48.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.999+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:48.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.999+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.999+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc0ff190 osd=2 4 2024-02-23T00:40:48.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.999+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 2 5 2024-02-23T00:40:48.254 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:40:48.254 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:40:48.254 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:40:48.254 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:47.999+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 6 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6813/128059790 conn(0x7f49dc0feb80 0x7f49dc11b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc027a10 osd=3 3 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc027a10 osd=3 3 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc027a10 osd=3 3 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 6 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc027a10 osd=3 3 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 3 6 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc027a10 osd=3 4 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 10 client.27636.objecter _send_command 6 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49dc060db0 con 0x7f49dc0feb80 2024-02-23T00:40:48.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49dbfff700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6813/128059790 conn(0x7f49dc0feb80 0x7f49dc11b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49d9ffb700 10 client.27636.objecter ms_handle_connect 0x7f49dc0feb80 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49dbfff700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.87:6813/128059790 conn(0x7f49dc0feb80 0x7f49dc11b5b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f49dc060db0 con 0x7f49dc0feb80 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 6 = system:0 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc027a10 osd=3 4 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.000+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 3 6 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 7 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc027a10 osd=3 3 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc027a10 osd=3 3 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc027a10 osd=3 3 2024-02-23T00:40:48.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 7 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc027a10 osd=3 3 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 3 7 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc027a10 osd=3 4 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 10 client.27636.objecter _send_command 7 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.015+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f49dc0fe1a0 con 0x7f49dc0feb80 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f49dc0fe1a0 con 0x7f49dc0feb80 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc027a10 osd=3 4 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 3 7 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 8 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6800/4175844757 conn(0x7f49dc11baa0 0x7f49dc120530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0617a0 osd=4 3 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0617a0 osd=4 3 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0617a0 osd=4 3 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 8 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc0617a0 osd=4 3 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 4 8 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0617a0 osd=4 4 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 10 client.27636.objecter _send_command 8 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.016+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49dc11dfe0 con 0x7f49dc11baa0 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.017+0000 7f49e1207700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6800/4175844757 conn(0x7f49dc11baa0 0x7f49dc120530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.017+0000 7f49e1207700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6800/4175844757 conn(0x7f49dc11baa0 0x7f49dc120530 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.017+0000 7f49d9ffb700 10 client.27636.objecter ms_handle_connect 0x7f49dc11baa0 2024-02-23T00:40:48.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.018+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f49dc11dfe0 con 0x7f49dc11baa0 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.018+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.018+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 8 = system:0 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.018+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc0617a0 osd=4 4 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.018+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 4 8 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 9 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0617a0 osd=4 3 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0617a0 osd=4 3 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0617a0 osd=4 3 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 9 2024-02-23T00:40:48.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc0617a0 osd=4 3 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 4 9 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0617a0 osd=4 4 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 10 client.27636.objecter _send_command 9 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.032+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f49dc061210 con 0x7f49dc11baa0 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f49dc061210 con 0x7f49dc11baa0 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc0617a0 osd=4 4 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 4 9 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 10 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6804/2550013795 conn(0x7f49dc119100 0x7f49dc123bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc061210 osd=5 3 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc061210 osd=5 3 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc061210 osd=5 3 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 10 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc061210 osd=5 3 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 5 10 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc061210 osd=5 4 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 10 client.27636.objecter _send_command 10 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49dc0614f0 con 0x7f49dc119100 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.034+0000 7f49e0a06700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6804/2550013795 conn(0x7f49dc119100 0x7f49dc123bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.035+0000 7f49e0a06700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6804/2550013795 conn(0x7f49dc119100 0x7f49dc123bb0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.035+0000 7f49d9ffb700 10 client.27636.objecter ms_handle_connect 0x7f49dc119100 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.036+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f49dc0614f0 con 0x7f49dc119100 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.036+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.036+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 10 = system:0 2024-02-23T00:40:48.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.036+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc061210 osd=5 4 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.036+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 5 10 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 11 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc061210 osd=5 3 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc061210 osd=5 3 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc061210 osd=5 3 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 11 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc061210 osd=5 3 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 5 11 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc061210 osd=5 4 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 10 client.27636.objecter _send_command 11 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.050+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f49dc0614f0 con 0x7f49dc119100 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f49dc0614f0 con 0x7f49dc119100 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:48.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc061210 osd=5 4 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 5 11 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 12 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6808/2509408498 conn(0x7f49dc1245b0 0x7f49dc1269a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0614f0 osd=6 3 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0614f0 osd=6 3 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0614f0 osd=6 3 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 12 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc0614f0 osd=6 3 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 6 12 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0614f0 osd=6 4 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 10 client.27636.objecter _send_command 12 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49dc129750 con 0x7f49dc1245b0 2024-02-23T00:40:48.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.052+0000 7f49dbfff700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6808/2509408498 conn(0x7f49dc1245b0 0x7f49dc1269a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.053+0000 7f49dbfff700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6808/2509408498 conn(0x7f49dc1245b0 0x7f49dc1269a0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:48.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.053+0000 7f49d9ffb700 10 client.27636.objecter ms_handle_connect 0x7f49dc1245b0 2024-02-23T00:40:48.264 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:40:48.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.054+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f49dc129750 con 0x7f49dc1245b0 2024-02-23T00:40:48.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.054+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:40:48.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.054+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 12 = system:0 2024-02-23T00:40:48.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.054+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc0614f0 osd=6 4 2024-02-23T00:40:48.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.054+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 6 12 2024-02-23T00:40:48.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:48.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 13 2024-02-23T00:40:48.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0614f0 osd=6 3 2024-02-23T00:40:48.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0614f0 osd=6 3 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc0614f0 osd=6 3 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 13 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc0614f0 osd=6 3 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 6 13 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc0614f0 osd=6 4 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 10 client.27636.objecter _send_command 13 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.068+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f49dc129750 con 0x7f49dc1245b0 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f49dc129750 con 0x7f49dc1245b0 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc0614f0 osd=6 4 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 6 13 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:48.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 14 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6812/373787000 conn(0x7f49dc128a90 0x7f49dc128ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc129750 osd=7 3 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc129750 osd=7 3 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc129750 osd=7 3 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 14 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc129750 osd=7 3 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 7 14 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc129750 osd=7 4 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 10 client.27636.objecter _send_command 14 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49dc12cb80 con 0x7f49dc128a90 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.070+0000 7f49e1207700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6812/373787000 conn(0x7f49dc128a90 0x7f49dc128ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.071+0000 7f49d9ffb700 10 client.27636.objecter ms_handle_connect 0x7f49dc128a90 2024-02-23T00:40:48.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.071+0000 7f49e1207700 1 --2- 172.21.15.87:0/2250174298 >> v2:172.21.15.171:6812/373787000 conn(0x7f49dc128a90 0x7f49dc128ef0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:48.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.071+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f49dc12cb80 con 0x7f49dc128a90 2024-02-23T00:40:48.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.071+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:40:48.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.071+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 14 = system:0 2024-02-23T00:40:48.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.071+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc129750 osd=7 4 2024-02-23T00:40:48.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.071+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 7 14 2024-02-23T00:40:48.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 10 client.27636.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:40:48.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign -1 15 2024-02-23T00:40:48.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc129750 osd=7 3 2024-02-23T00:40:48.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc129750 osd=7 3 2024-02-23T00:40:48.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 20 client.27636.objecter _get_session s=0x7f49dc129750 osd=7 3 2024-02-23T00:40:48.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_remove -1 15 2024-02-23T00:40:48.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 20 client.27636.objecter get_session s=0x7f49dc129750 osd=7 3 2024-02-23T00:40:48.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 15 client.27636.objecter _session_command_op_assign 7 15 2024-02-23T00:40:48.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 20 client.27636.objecter put_session s=0x7f49dc129750 osd=7 4 2024-02-23T00:40:48.270 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:40:48.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 10 client.27636.objecter _send_command 15 2024-02-23T00:40:48.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.088+0000 7f49e2c6a700 1 -- 172.21.15.87:0/2250174298 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f49dc126f30 con 0x7f49dc128a90 2024-02-23T00:40:48.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.090+0000 7f49d9ffb700 1 -- 172.21.15.87:0/2250174298 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f49dc126f30 con 0x7f49dc128a90 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.090+0000 7f49d9ffb700 10 client.27636.objecter ms_dispatch 0x7f49dc001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.090+0000 7f49d9ffb700 10 client.27636.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.090+0000 7f49d9ffb700 20 client.27636.objecter put_session s=0x7f49dc129750 osd=7 4 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:48.090+0000 7f49d9ffb700 15 client.27636.objecter _session_command_op_remove 7 15 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.271 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:40:48.272 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:48.524 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:40:48.524 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:40:48.524 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:40:48.524 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:40:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:47.278556+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e628: 8 total, 7 up, 7 in 2024-02-23T00:40:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:47.425680+0000 osd.6 (osd.6) 248 : cluster [DBG] 4.a scrub starts 2024-02-23T00:40:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:47.429404+0000 osd.6 (osd.6) 249 : cluster [DBG] 4.a scrub ok 2024-02-23T00:40:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:47.964496+0000 mon.a (mon.0) 3328 : audit [DBG] from='client.? 172.21.15.87:0/2250174298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:47.990070+0000 osd.5 (osd.5) 241 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:40:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:47.990147+0000 osd.5 (osd.5) 242 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:40:48.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:47.278556+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e628: 8 total, 7 up, 7 in 2024-02-23T00:40:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:47.425680+0000 osd.6 (osd.6) 248 : cluster [DBG] 4.a scrub starts 2024-02-23T00:40:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:47.429404+0000 osd.6 (osd.6) 249 : cluster [DBG] 4.a scrub ok 2024-02-23T00:40:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:47.964496+0000 mon.a (mon.0) 3328 : audit [DBG] from='client.? 172.21.15.87:0/2250174298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:47.990070+0000 osd.5 (osd.5) 241 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:40:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:47.990147+0000 osd.5 (osd.5) 242 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:40:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:47.278556+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e628: 8 total, 7 up, 7 in 2024-02-23T00:40:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:47.425680+0000 osd.6 (osd.6) 248 : cluster [DBG] 4.a scrub starts 2024-02-23T00:40:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:47.429404+0000 osd.6 (osd.6) 249 : cluster [DBG] 4.a scrub ok 2024-02-23T00:40:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:47.964496+0000 mon.a (mon.0) 3328 : audit [DBG] from='client.? 172.21.15.87:0/2250174298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:47.990070+0000 osd.5 (osd.5) 241 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:40:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:47.990147+0000 osd.5 (osd.5) 242 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:40:49.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.044+0000 7f21d7f10700 1 -- 172.21.15.87:0/3250053189 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00fd510 msgr2=0x7f21d00f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:49.068 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.044+0000 7f21d7f10700 1 --2- 172.21.15.87:0/3250053189 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00fd510 0x7f21d00f6d40 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 rx=0x7f21cc004870 tx=0x7f21cc009cb0).stop 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.044+0000 7f21d7f10700 1 -- 172.21.15.87:0/3250053189 shutdown_connections 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.044+0000 7f21d7f10700 1 --2- 172.21.15.87:0/3250053189 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21d00f7400 0x7f21d0103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.044+0000 7f21d7f10700 1 --2- 172.21.15.87:0/3250053189 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21d00fcbb0 0x7f21d00fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.044+0000 7f21d7f10700 1 --2- 172.21.15.87:0/3250053189 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00fd510 0x7f21d00f6d40 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.044+0000 7f21d7f10700 1 -- 172.21.15.87:0/3250053189 >> 172.21.15.87:0/3250053189 conn(0x7f21d00f2820 msgr2=0x7f21d00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.044+0000 7f21d7f10700 1 -- 172.21.15.87:0/3250053189 shutdown_connections 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.044+0000 7f21d7f10700 1 -- 172.21.15.87:0/3250053189 wait complete. 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 librados: starting msgr at 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 librados: starting objecter 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 Processor -- start 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 -- start start 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 librados: setting wanted keys 2024-02-23T00:40:49.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 librados: calling monclient init 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00f7400 0x7f21d01092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21d00fcbb0 0x7f21d01097e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21d00fd510 0x7f21d0109da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01144e0 con 0x7f21d00fcbb0 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d0114650 con 0x7f21d00fd510 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d7f10700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d0114810 con 0x7f21d00f7400 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d5cac700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00f7400 0x7f21d01092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d5cac700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00f7400 0x7f21d01092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44544/0 (socket says 172.21.15.87:44544) 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d5cac700 1 -- 172.21.15.87:0/1104084871 learned_addr learned my addr 172.21.15.87:0/1104084871 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.045+0000 7f21d54ab700 1 --2- 172.21.15.87:0/1104084871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21d00fcbb0 0x7f21d01097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d5cac700 1 -- 172.21.15.87:0/1104084871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21d00fd510 msgr2=0x7f21d0109da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d5cac700 1 --2- 172.21.15.87:0/1104084871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21d00fd510 0x7f21d0109da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:40:49.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d5cac700 1 -- 172.21.15.87:0/1104084871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21d00fcbb0 msgr2=0x7f21d01097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d5cac700 1 --2- 172.21.15.87:0/1104084871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21d00fcbb0 0x7f21d01097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d5cac700 1 -- 172.21.15.87:0/1104084871 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c0005040 con 0x7f21d00f7400 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d5cac700 1 --2- 172.21.15.87:0/1104084871 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00f7400 0x7f21d01092f0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f21c00048f0 tx=0x7f21c0009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21c6ffd700 10 client.?.objecter ms_handle_connect 0x7f21d00f7400 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21c6ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21c6ffd700 1 -- 172.21.15.87:0/1104084871 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0007b00 con 0x7f21d00f7400 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21c6ffd700 1 -- 172.21.15.87:0/1104084871 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f21c000f040 con 0x7f21d00f7400 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc005040 con 0x7f21d00f7400 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21c6ffd700 1 -- 172.21.15.87:0/1104084871 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0014a00 con 0x7f21d00f7400 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d7f10700 10 client.38114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d0114e50 con 0x7f21d00f7400 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.046+0000 7f21d7f10700 1 librados: init done 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.047+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d005fa60 con 0x7f21d00f7400 2024-02-23T00:40:49.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21c6ffd700 1 -- 172.21.15.87:0/1104084871 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f21c000e580 con 0x7f21d00f7400 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21c6ffd700 1 --2- 172.21.15.87:0/1104084871 >> v2:172.21.15.87:6800/1231597463 conn(0x7f21bc062c90 0x7f21bc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21c6ffd700 1 -- 172.21.15.87:0/1104084871 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(629..629 src has 173..629) v4 ==== 8516+0+0 (secure 0 0 0) 0x7f21c007d2b0 con 0x7f21d00f7400 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21c6ffd700 10 client.38114.objecter ms_dispatch 0x7f21d0001550 osd_map(629..629 src has 173..629) v4 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21d54ab700 1 --2- 172.21.15.87:0/1104084871 >> v2:172.21.15.87:6800/1231597463 conn(0x7f21bc062c90 0x7f21bc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21c6ffd700 3 client.38114.objecter handle_osd_map got epochs [629,629] > 0 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21c6ffd700 3 client.38114.objecter handle_osd_map decoding full epoch 629 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21c6ffd700 20 client.38114.objecter dump_active .. 0 homeless 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21c6ffd700 10 client.38114.objecter ms_handle_connect 0x7f21bc062c90 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.048+0000 7f21d54ab700 1 --2- 172.21.15.87:0/1104084871 >> v2:172.21.15.87:6800/1231597463 conn(0x7f21bc062c90 0x7f21bc065150 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f21cc004ee0 tx=0x7f21cc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:49.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.052+0000 7f21c6ffd700 1 -- 172.21.15.87:0/1104084871 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f21c0050630 con 0x7f21d00f7400 2024-02-23T00:40:49.187 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:49.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:49.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.236+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]} v 0) v1 -- 0x7f21d005ee80 con 0x7f21d00f7400 2024-02-23T00:40:49.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.292+0000 7f21c6ffd700 1 -- 172.21.15.87:0/1104084871 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]=0 set 4.c pg_upmap_items mapping to [7->3,1->2,4->0] v630) v1 ==== 157+0+0 (secure 0 0 0) 0x7f21c00220b0 con 0x7f21d00f7400 2024-02-23T00:40:49.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.295+0000 7f21d7f10700 10 librados: watch_flush enter 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 10 librados: watch_flush exit 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 20 client.38114.objecter shutdown clearing up homeless session... 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 10 client.38114.objecter successfully canceled tick 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 >> v2:172.21.15.87:6800/1231597463 conn(0x7f21bc062c90 msgr2=0x7f21bc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 --2- 172.21.15.87:0/1104084871 >> v2:172.21.15.87:6800/1231597463 conn(0x7f21bc062c90 0x7f21bc065150 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f21cc004ee0 tx=0x7f21cc00b040).stop 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00f7400 msgr2=0x7f21d01092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 --2- 172.21.15.87:0/1104084871 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00f7400 0x7f21d01092f0 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 rx=0x7f21c00048f0 tx=0x7f21c0009cb0).stop 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 shutdown_connections 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 --2- 172.21.15.87:0/1104084871 >> v2:172.21.15.87:6800/1231597463 conn(0x7f21bc062c90 0x7f21bc065150 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 --2- 172.21.15.87:0/1104084871 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f21d00fd510 0x7f21d0109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:40:49.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 --2- 172.21.15.87:0/1104084871 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f21d00f7400 0x7f21d01092f0 secure :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 rx=0x7f21c00048f0 tx=0x7f21c0009cb0).stop 2024-02-23T00:40:49.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 --2- 172.21.15.87:0/1104084871 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f21d00fcbb0 0x7f21d01097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:49.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.296+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 >> 172.21.15.87:0/1104084871 conn(0x7f21d00f2820 msgr2=0x7f21d00f4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:49.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.298+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 shutdown_connections 2024-02-23T00:40:49.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.298+0000 7f21d7f10700 1 -- 172.21.15.87:0/1104084871 wait complete. 2024-02-23T00:40:49.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:49.298+0000 7f21d7f10700 1 librados: shutdown 2024-02-23T00:40:49.303 INFO:teuthology.orchestra.run.smithi087.stderr:set 4.c pg_upmap_items mapping to [7->3,1->2,4->0] 2024-02-23T00:40:49.307 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:49.334 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:47.755029+0000 mgr.y (mgr.24437) 556 : cluster [DBG] pgmap v959: 208 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:48.291645+0000 mon.a (mon.0) 3329 : cluster [DBG] osdmap e629: 8 total, 7 up, 7 in 2024-02-23T00:40:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:48.416158+0000 osd.6 (osd.6) 250 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:40:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:48.419094+0000 osd.6 (osd.6) 251 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:40:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:49.238549+0000 mon.c (mon.2) 478 : audit [INF] from='client.? 172.21.15.87:0/1104084871' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]: dispatch 2024-02-23T00:40:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:49.238884+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]: dispatch 2024-02-23T00:40:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:47.755029+0000 mgr.y (mgr.24437) 556 : cluster [DBG] pgmap v959: 208 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:48.291645+0000 mon.a (mon.0) 3329 : cluster [DBG] osdmap e629: 8 total, 7 up, 7 in 2024-02-23T00:40:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:48.416158+0000 osd.6 (osd.6) 250 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:40:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:48.419094+0000 osd.6 (osd.6) 251 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:40:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:49.238549+0000 mon.c (mon.2) 478 : audit [INF] from='client.? 172.21.15.87:0/1104084871' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]: dispatch 2024-02-23T00:40:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:49.238884+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]: dispatch 2024-02-23T00:40:49.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-23T00:40:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:47.755029+0000 mgr.y (mgr.24437) 556 : cluster [DBG] pgmap v959: 208 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 205 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 5/738 objects degraded (0.678%); 25/738 objects misplaced (3.388%) 2024-02-23T00:40:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:48.291645+0000 mon.a (mon.0) 3329 : cluster [DBG] osdmap e629: 8 total, 7 up, 7 in 2024-02-23T00:40:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:48.416158+0000 osd.6 (osd.6) 250 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:40:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:48.419094+0000 osd.6 (osd.6) 251 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:40:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:49.238549+0000 mon.c (mon.2) 478 : audit [INF] from='client.? 172.21.15.87:0/1104084871' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]: dispatch 2024-02-23T00:40:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:49.238884+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]: dispatch 2024-02-23T00:40:49.989 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:49.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:50.093 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:50.121 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:49.292726+0000 mon.a (mon.0) 3331 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]': finished 2024-02-23T00:40:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:49.292805+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e630: 8 total, 7 up, 7 in 2024-02-23T00:40:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:49.756902+0000 mon.a (mon.0) 3333 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "34"}]: dispatch 2024-02-23T00:40:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:49.757168+0000 mon.a (mon.0) 3334 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-23T00:40:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:49.853559+0000 osd.5 (osd.5) 243 : cluster [DBG] 6.a scrub starts 2024-02-23T00:40:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:49.854630+0000 osd.5 (osd.5) 244 : cluster [DBG] 6.a scrub ok 2024-02-23T00:40:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:49.292726+0000 mon.a (mon.0) 3331 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]': finished 2024-02-23T00:40:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:49.292805+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e630: 8 total, 7 up, 7 in 2024-02-23T00:40:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:49.756902+0000 mon.a (mon.0) 3333 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "34"}]: dispatch 2024-02-23T00:40:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:49.757168+0000 mon.a (mon.0) 3334 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-23T00:40:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:49.853559+0000 osd.5 (osd.5) 243 : cluster [DBG] 6.a scrub starts 2024-02-23T00:40:50.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:49.854630+0000 osd.5 (osd.5) 244 : cluster [DBG] 6.a scrub ok 2024-02-23T00:40:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:49.292726+0000 mon.a (mon.0) 3331 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.c", "id": [7, 3, 1, 2, 4, 0]}]': finished 2024-02-23T00:40:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:49.292805+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e630: 8 total, 7 up, 7 in 2024-02-23T00:40:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:49.756902+0000 mon.a (mon.0) 3333 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "34"}]: dispatch 2024-02-23T00:40:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:49.757168+0000 mon.a (mon.0) 3334 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-23T00:40:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:49.853559+0000 osd.5 (osd.5) 243 : cluster [DBG] 6.a scrub starts 2024-02-23T00:40:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:49.854630+0000 osd.5 (osd.5) 244 : cluster [DBG] 6.a scrub ok 2024-02-23T00:40:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:40:50.714 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:40:51.003 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:51.040 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:51.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:49.755417+0000 mgr.y (mgr.24437) 557 : cluster [DBG] pgmap v962: 206 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 203 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5/735 objects degraded (0.680%); 25/735 objects misplaced (3.401%) 2024-02-23T00:40:51.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:50.310106+0000 mon.a (mon.0) 3335 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "34"}]': finished 2024-02-23T00:40:51.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:50.310152+0000 mon.a (mon.0) 3336 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-23T00:40:51.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:50.310222+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e631: 8 total, 7 up, 7 in 2024-02-23T00:40:51.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:50.418349+0000 osd.6 (osd.6) 252 : cluster [DBG] 3.a scrub starts 2024-02-23T00:40:51.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:50.421220+0000 osd.6 (osd.6) 253 : cluster [DBG] 3.a scrub ok 2024-02-23T00:40:51.315 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:50.450106+0000 osd.3 (osd.3) 195 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:40:51.316 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:50.453576+0000 osd.3 (osd.3) 196 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:40:51.316 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:50.674456+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:40:51.316 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:50.676374+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:40:51.316 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:50.811548+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:40:51.316 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:50.813705+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:40:51.370 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:51.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:51.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:49.755417+0000 mgr.y (mgr.24437) 557 : cluster [DBG] pgmap v962: 206 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 203 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5/735 objects degraded (0.680%); 25/735 objects misplaced (3.401%) 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:50.310106+0000 mon.a (mon.0) 3335 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "34"}]': finished 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:50.310152+0000 mon.a (mon.0) 3336 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:50.310222+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e631: 8 total, 7 up, 7 in 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:50.418349+0000 osd.6 (osd.6) 252 : cluster [DBG] 3.a scrub starts 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:50.421220+0000 osd.6 (osd.6) 253 : cluster [DBG] 3.a scrub ok 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:50.450106+0000 osd.3 (osd.3) 195 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:50.453576+0000 osd.3 (osd.3) 196 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:50.674456+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:50.676374+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:50.811548+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:40:51.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:50.813705+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:40:51.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:51.506 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:51.529 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:51.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:49.755417+0000 mgr.y (mgr.24437) 557 : cluster [DBG] pgmap v962: 206 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 203 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 5/735 objects degraded (0.680%); 25/735 objects misplaced (3.401%) 2024-02-23T00:40:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:50.310106+0000 mon.a (mon.0) 3335 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "34"}]': finished 2024-02-23T00:40:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:50.310152+0000 mon.a (mon.0) 3336 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-23T00:40:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:50.310222+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e631: 8 total, 7 up, 7 in 2024-02-23T00:40:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:50.418349+0000 osd.6 (osd.6) 252 : cluster [DBG] 3.a scrub starts 2024-02-23T00:40:51.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:50.421220+0000 osd.6 (osd.6) 253 : cluster [DBG] 3.a scrub ok 2024-02-23T00:40:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:50.450106+0000 osd.3 (osd.3) 195 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:40:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:50.453576+0000 osd.3 (osd.3) 196 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:40:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:50.674456+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:40:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:50.676374+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:40:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:50.811548+0000 osd.5 (osd.5) 245 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:40:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:50.813705+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:40:51.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:51.665 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:51.685 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:51.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:51.789 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:40:51.821 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:51.842 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:40:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:51.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:40:51.946 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:51.976 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:52.173 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:52.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:52.277 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:52.310 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:52.332 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:52.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:52.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:52.436 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:40:52.463 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:52.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:52] "GET /metrics HTTP/1.1" 200 179522 "" "Prometheus/2.33.4" 2024-02-23T00:40:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:51.135078+0000 mgr.y (mgr.24437) 558 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:51.168003+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:40:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:51.169141+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:51.309964+0000 mon.a (mon.0) 3338 : cluster [DBG] osdmap e632: 8 total, 7 up, 7 in 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:51.447983+0000 osd.6 (osd.6) 254 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:51.451468+0000 osd.6 (osd.6) 255 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:51.631211+0000 osd.4 (osd.4) 283 : cluster [DBG] 4.c scrub starts 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:51.632941+0000 osd.4 (osd.4) 284 : cluster [DBG] 4.c scrub ok 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.181478+0000 mon.a (mon.0) 3339 : cluster [WRN] Health check update: Degraded data redundancy: 5/735 objects degraded (0.680%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.186667+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e633: 8 total, 7 up, 7 in 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:51.135078+0000 mgr.y (mgr.24437) 558 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:51.168003+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:51.169141+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:40:52.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:51.309964+0000 mon.a (mon.0) 3338 : cluster [DBG] osdmap e632: 8 total, 7 up, 7 in 2024-02-23T00:40:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:51.447983+0000 osd.6 (osd.6) 254 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:40:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:51.451468+0000 osd.6 (osd.6) 255 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:40:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:51.631211+0000 osd.4 (osd.4) 283 : cluster [DBG] 4.c scrub starts 2024-02-23T00:40:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:51.632941+0000 osd.4 (osd.4) 284 : cluster [DBG] 4.c scrub ok 2024-02-23T00:40:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.181478+0000 mon.a (mon.0) 3339 : cluster [WRN] Health check update: Degraded data redundancy: 5/735 objects degraded (0.680%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.186667+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e633: 8 total, 7 up, 7 in 2024-02-23T00:40:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:51.135078+0000 mgr.y (mgr.24437) 558 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:40:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:51.168003+0000 osd.2 (osd.2) 229 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:40:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:51.169141+0000 osd.2 (osd.2) 230 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:40:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:51.309964+0000 mon.a (mon.0) 3338 : cluster [DBG] osdmap e632: 8 total, 7 up, 7 in 2024-02-23T00:40:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:51.447983+0000 osd.6 (osd.6) 254 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:40:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:51.451468+0000 osd.6 (osd.6) 255 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:40:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:51.631211+0000 osd.4 (osd.4) 283 : cluster [DBG] 4.c scrub starts 2024-02-23T00:40:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:51.632941+0000 osd.4 (osd.4) 284 : cluster [DBG] 4.c scrub ok 2024-02-23T00:40:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.181478+0000 mon.a (mon.0) 3339 : cluster [WRN] Health check update: Degraded data redundancy: 5/735 objects degraded (0.680%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-23T00:40:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.186667+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e633: 8 total, 7 up, 7 in 2024-02-23T00:40:52.714 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:40:53.091 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:40:53.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:53.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:40:53.195 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:53.226 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:53.248 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:53.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:51.755849+0000 mgr.y (mgr.24437) 559 : cluster [DBG] pgmap v965: 206 pgs: 5 peering, 2 active+remapped+backfill_toofull, 199 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/738 objects misplaced (2.710%); 0 B/s, 0 objects/s recovering 2024-02-23T00:40:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.205897+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:40:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.208778+0000 osd.1 (osd.1) 289 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:40:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.312689+0000 mon.a (mon.0) 3341 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.312732+0000 mon.a (mon.0) 3342 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/735 objects degraded (0.680%), 1 pg degraded, 1 pg undersized) 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.430606+0000 osd.6 (osd.6) 256 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.433494+0000 osd.6 (osd.6) 257 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.471343+0000 osd.3 (osd.3) 197 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.474487+0000 osd.3 (osd.3) 198 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.653633+0000 osd.4 (osd.4) 285 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.654610+0000 osd.4 (osd.4) 286 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.822585+0000 osd.7 (osd.7) 288 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:52.825476+0000 osd.7 (osd.7) 289 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:53.187699+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e634: 8 total, 7 up, 7 in 2024-02-23T00:40:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:51.755849+0000 mgr.y (mgr.24437) 559 : cluster [DBG] pgmap v965: 206 pgs: 5 peering, 2 active+remapped+backfill_toofull, 199 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/738 objects misplaced (2.710%); 0 B/s, 0 objects/s recovering 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.205897+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.208778+0000 osd.1 (osd.1) 289 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.312689+0000 mon.a (mon.0) 3341 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.312732+0000 mon.a (mon.0) 3342 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/735 objects degraded (0.680%), 1 pg degraded, 1 pg undersized) 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.430606+0000 osd.6 (osd.6) 256 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.433494+0000 osd.6 (osd.6) 257 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.471343+0000 osd.3 (osd.3) 197 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.474487+0000 osd.3 (osd.3) 198 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.653633+0000 osd.4 (osd.4) 285 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.654610+0000 osd.4 (osd.4) 286 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.822585+0000 osd.7 (osd.7) 288 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:40:53.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:52.825476+0000 osd.7 (osd.7) 289 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:40:53.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:53.187699+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e634: 8 total, 7 up, 7 in 2024-02-23T00:40:53.352 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:53.381 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:53.400 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:53.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:53.504 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:53.533 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:51.755849+0000 mgr.y (mgr.24437) 559 : cluster [DBG] pgmap v965: 206 pgs: 5 peering, 2 active+remapped+backfill_toofull, 199 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/738 objects misplaced (2.710%); 0 B/s, 0 objects/s recovering 2024-02-23T00:40:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.205897+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:40:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.208778+0000 osd.1 (osd.1) 289 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:40:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.312689+0000 mon.a (mon.0) 3341 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.312732+0000 mon.a (mon.0) 3342 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/735 objects degraded (0.680%), 1 pg degraded, 1 pg undersized) 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.430606+0000 osd.6 (osd.6) 256 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.433494+0000 osd.6 (osd.6) 257 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.471343+0000 osd.3 (osd.3) 197 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.474487+0000 osd.3 (osd.3) 198 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.653633+0000 osd.4 (osd.4) 285 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.654610+0000 osd.4 (osd.4) 286 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.822585+0000 osd.7 (osd.7) 288 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:52.825476+0000 osd.7 (osd.7) 289 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:40:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:53.187699+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e634: 8 total, 7 up, 7 in 2024-02-23T00:40:53.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2479766930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9c800fcbb0 msgr2=0x7f9c800fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:53.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2479766930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9c800fcbb0 0x7f9c800fcf90 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 rx=0x7f9c74004870 tx=0x7f9c74009cb0).stop 2024-02-23T00:40:53.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2479766930 shutdown_connections 2024-02-23T00:40:53.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2479766930 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9c800f7400 0x7f9c80103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2479766930 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9c800fcbb0 0x7f9c800fcf90 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2479766930 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c800fd510 0x7f9c800f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2479766930 >> 172.21.15.87:0/2479766930 conn(0x7f9c800f2820 msgr2=0x7f9c800f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2479766930 shutdown_connections 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2479766930 wait complete. 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 librados: starting msgr at 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 librados: starting objecter 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 Processor -- start 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 -- start start 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 librados: setting wanted keys 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.707+0000 7f9c86ca8700 1 librados: calling monclient init 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c86ca8700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c800f7400 0x7f9c8010e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:53.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c86ca8700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9c800fcbb0 0x7f9c8010e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c86ca8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9c800fd510 0x7f9c80108330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c86ca8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c80108820 con 0x7f9c800fd510 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c86ca8700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c80108990 con 0x7f9c800f7400 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c86ca8700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c80108b50 con 0x7f9c800fcbb0 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c7ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9c800fcbb0 0x7f9c8010e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c85245700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9c800fd510 0x7f9c80108330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c7ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9c800fcbb0 0x7f9c8010e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50456/0 (socket says 172.21.15.87:50456) 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c7ffff700 1 -- 172.21.15.87:0/2338216967 learned_addr learned my addr 172.21.15.87:0/2338216967 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c84a44700 1 --2- 172.21.15.87:0/2338216967 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c800f7400 0x7f9c8010e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c85245700 1 -- 172.21.15.87:0/2338216967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9c800fcbb0 msgr2=0x7f9c8010e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c85245700 1 --2- 172.21.15.87:0/2338216967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9c800fcbb0 0x7f9c8010e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c85245700 1 -- 172.21.15.87:0/2338216967 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c800f7400 msgr2=0x7f9c8010e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c85245700 1 --2- 172.21.15.87:0/2338216967 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c800f7400 0x7f9c8010e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:53.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c85245700 1 -- 172.21.15.87:0/2338216967 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c780097c0 con 0x7f9c800fd510 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c7ffff700 1 --2- 172.21.15.87:0/2338216967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9c800fcbb0 0x7f9c8010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c85245700 1 --2- 172.21.15.87:0/2338216967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9c800fd510 0x7f9c80108330 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f9c78001cb0 tx=0x7f9c78004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c7dffb700 10 client.?.objecter ms_handle_connect 0x7f9c800fd510 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c7dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c7dffb700 1 -- 172.21.15.87:0/2338216967 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c780079b0 con 0x7f9c800fd510 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c74005040 con 0x7f9c800fd510 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c7dffb700 1 -- 172.21.15.87:0/2338216967 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c780090d0 con 0x7f9c800fd510 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.708+0000 7f9c7dffb700 1 -- 172.21.15.87:0/2338216967 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7800fc20 con 0x7f9c800fd510 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.709+0000 7f9c86ca8700 10 client.27648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.709+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c80109190 con 0x7f9c800fd510 2024-02-23T00:40:53.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.709+0000 7f9c86ca8700 1 librados: init done 2024-02-23T00:40:53.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.710+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8005fa60 con 0x7f9c800fd510 2024-02-23T00:40:53.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.710+0000 7f9c7dffb700 1 -- 172.21.15.87:0/2338216967 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f9c7800fd80 con 0x7f9c800fd510 2024-02-23T00:40:53.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.710+0000 7f9c7dffb700 1 --2- 172.21.15.87:0/2338216967 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9c70062c90 0x7f9c70065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:53.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.710+0000 7f9c7dffb700 1 -- 172.21.15.87:0/2338216967 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(634..634 src has 173..634) v4 ==== 8487+0+0 (secure 0 0 0) 0x7f9c7807dc30 con 0x7f9c800fd510 2024-02-23T00:40:53.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.710+0000 7f9c7dffb700 10 client.27648.objecter ms_dispatch 0x7f9c80001550 osd_map(634..634 src has 173..634) v4 2024-02-23T00:40:53.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.710+0000 7f9c7dffb700 3 client.27648.objecter handle_osd_map got epochs [634,634] > 0 2024-02-23T00:40:53.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.710+0000 7f9c7dffb700 3 client.27648.objecter handle_osd_map decoding full epoch 634 2024-02-23T00:40:53.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.710+0000 7f9c84a44700 1 --2- 172.21.15.87:0/2338216967 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9c70062c90 0x7f9c70065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:53.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.710+0000 7f9c7dffb700 20 client.27648.objecter dump_active .. 0 homeless 2024-02-23T00:40:53.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.711+0000 7f9c7dffb700 10 client.27648.objecter ms_handle_connect 0x7f9c70062c90 2024-02-23T00:40:53.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.711+0000 7f9c84a44700 1 --2- 172.21.15.87:0/2338216967 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9c70062c90 0x7f9c70065150 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 rx=0x7f9c740043d0 tx=0x7f9c7400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:53.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.714+0000 7f9c7dffb700 1 -- 172.21.15.87:0/2338216967 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9c78050f50 con 0x7f9c800fd510 2024-02-23T00:40:53.732 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:53.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:53.836 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:53.865 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:53.884 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:53.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:53.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:53.894+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f9c801060e0 con 0x7f9c800fd510 2024-02-23T00:40:53.987 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:54.014 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:40:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:40:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:40:54.016 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:54.035 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:54.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:54.139 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:54.167 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:54.186 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:54.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:54.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:54.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.194+0000 7f9c7dffb700 1 -- 172.21.15.87:0/2338216967 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v635) v1 ==== 101+0+0 (secure 0 0 0) 0x7f9c78050b40 con 0x7f9c800fd510 2024-02-23T00:40:54.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.197+0000 7f9c86ca8700 10 librados: watch_flush enter 2024-02-23T00:40:54.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.197+0000 7f9c86ca8700 10 librados: watch_flush exit 2024-02-23T00:40:54.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.197+0000 7f9c86ca8700 20 client.27648.objecter shutdown clearing up homeless session... 2024-02-23T00:40:54.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.197+0000 7f9c86ca8700 10 client.27648.objecter successfully canceled tick 2024-02-23T00:40:54.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.197+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9c70062c90 msgr2=0x7f9c70065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:54.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.197+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2338216967 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9c70062c90 0x7f9c70065150 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 rx=0x7f9c740043d0 tx=0x7f9c7400b040).stop 2024-02-23T00:40:54.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.197+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9c800fd510 msgr2=0x7f9c80108330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:54.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.197+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2338216967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9c800fd510 0x7f9c80108330 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 rx=0x7f9c78001cb0 tx=0x7f9c78004c50).stop 2024-02-23T00:40:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.200+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 shutdown_connections 2024-02-23T00:40:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.200+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2338216967 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9c70062c90 0x7f9c70065150 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.200+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2338216967 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9c800fd510 0x7f9c80108330 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.200+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2338216967 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9c800f7400 0x7f9c8010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.200+0000 7f9c86ca8700 1 --2- 172.21.15.87:0/2338216967 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9c800fcbb0 0x7f9c8010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.200+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 >> 172.21.15.87:0/2338216967 conn(0x7f9c800f2820 msgr2=0x7f9c80060e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:54.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.200+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 shutdown_connections 2024-02-23T00:40:54.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.200+0000 7f9c86ca8700 1 -- 172.21.15.87:0/2338216967 wait complete. 2024-02-23T00:40:54.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:54.200+0000 7f9c86ca8700 1 librados: shutdown 2024-02-23T00:40:54.204 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:40:54.289 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:54.317 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:54.392 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:54.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:54.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:54.522 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:54.540 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:54.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:53.233317+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:40:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:53.236787+0000 osd.1 (osd.1) 291 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:40:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:53.459116+0000 osd.6 (osd.6) 258 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:40:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:53.462014+0000 osd.6 (osd.6) 259 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:40:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:53.757098+0000 mon.a (mon.0) 3344 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "33"}]: dispatch 2024-02-23T00:40:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:53.757269+0000 mon.a (mon.0) 3345 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-23T00:40:54.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:53.780672+0000 osd.5 (osd.5) 247 : cluster [DBG] 22.f scrub starts 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:53.782665+0000 osd.5 (osd.5) 248 : cluster [DBG] 22.f scrub ok 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:53.895621+0000 mon.a (mon.0) 3346 : audit [INF] from='client.? 172.21.15.87:0/2338216967' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:54.185146+0000 mon.a (mon.0) 3347 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:54.194878+0000 mon.a (mon.0) 3348 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "33"}]': finished 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:54.194944+0000 mon.a (mon.0) 3349 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:54.194993+0000 mon.a (mon.0) 3350 : audit [INF] from='client.? 172.21.15.87:0/2338216967' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:54.195038+0000 mon.a (mon.0) 3351 : cluster [DBG] osdmap e635: 8 total, 7 up, 7 in 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:54.218332+0000 mon.a (mon.0) 3352 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:53.233317+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:40:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:53.236787+0000 osd.1 (osd.1) 291 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:53.459116+0000 osd.6 (osd.6) 258 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:53.462014+0000 osd.6 (osd.6) 259 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:53.757098+0000 mon.a (mon.0) 3344 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "33"}]: dispatch 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:53.757269+0000 mon.a (mon.0) 3345 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:53.780672+0000 osd.5 (osd.5) 247 : cluster [DBG] 22.f scrub starts 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:53.782665+0000 osd.5 (osd.5) 248 : cluster [DBG] 22.f scrub ok 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:53.895621+0000 mon.a (mon.0) 3346 : audit [INF] from='client.? 172.21.15.87:0/2338216967' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:54.185146+0000 mon.a (mon.0) 3347 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:54.194878+0000 mon.a (mon.0) 3348 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "33"}]': finished 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:54.194944+0000 mon.a (mon.0) 3349 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:54.194993+0000 mon.a (mon.0) 3350 : audit [INF] from='client.? 172.21.15.87:0/2338216967' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:54.195038+0000 mon.a (mon.0) 3351 : cluster [DBG] osdmap e635: 8 total, 7 up, 7 in 2024-02-23T00:40:54.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:54.218332+0000 mon.a (mon.0) 3352 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:54.607 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:54.644 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:54.671 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:53.233317+0000 osd.1 (osd.1) 290 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:40:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:53.236787+0000 osd.1 (osd.1) 291 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:40:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:53.459116+0000 osd.6 (osd.6) 258 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:40:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:53.462014+0000 osd.6 (osd.6) 259 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:40:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:53.757098+0000 mon.a (mon.0) 3344 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "33"}]: dispatch 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:53.757269+0000 mon.a (mon.0) 3345 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:53.780672+0000 osd.5 (osd.5) 247 : cluster [DBG] 22.f scrub starts 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:53.782665+0000 osd.5 (osd.5) 248 : cluster [DBG] 22.f scrub ok 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:53.895621+0000 mon.a (mon.0) 3346 : audit [INF] from='client.? 172.21.15.87:0/2338216967' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:54.185146+0000 mon.a (mon.0) 3347 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:54.194878+0000 mon.a (mon.0) 3348 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "33"}]': finished 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:54.194944+0000 mon.a (mon.0) 3349 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:54.194993+0000 mon.a (mon.0) 3350 : audit [INF] from='client.? 172.21.15.87:0/2338216967' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:54.195038+0000 mon.a (mon.0) 3351 : cluster [DBG] osdmap e635: 8 total, 7 up, 7 in 2024-02-23T00:40:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:54.218332+0000 mon.a (mon.0) 3352 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:40:54.795 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:54.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:54.899 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:54.925 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:54.939 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:40:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:40:54.658+0000 7f6e819c8700 -1 mgr get_metadata_python Requested missing service osd.0 2024-02-23T00:40:54.944 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:54.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:55.048 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:55.074 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:55.093 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:55.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:55.196 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:55.224 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:55.244 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:55.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:55.348 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:55.377 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:55.414 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:55.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:55.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:55.518 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:55.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:53.756277+0000 mgr.y (mgr.24437) 560 : cluster [DBG] pgmap v968: 204 pgs: 3 peering, 2 active+remapped+backfill_toofull, 199 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/738 objects misplaced (2.710%); 0 B/s, 1 objects/s recovering 2024-02-23T00:40:55.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:54.655082+0000 mon.a (mon.0) 3353 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:55.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:54.788058+0000 osd.5 (osd.5) 249 : cluster [DBG] 5.f deep-scrub starts 2024-02-23T00:40:55.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:54.788858+0000 osd.5 (osd.5) 250 : cluster [DBG] 5.f deep-scrub ok 2024-02-23T00:40:55.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:55.191405+0000 mon.a (mon.0) 3354 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:55.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:55.191470+0000 mon.a (mon.0) 3355 : cluster [DBG] osdmap e636: 8 total, 7 up, 7 in 2024-02-23T00:40:55.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:55.193406+0000 mon.a (mon.0) 3356 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106"}]: dispatch 2024-02-23T00:40:55.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:53.756277+0000 mgr.y (mgr.24437) 560 : cluster [DBG] pgmap v968: 204 pgs: 3 peering, 2 active+remapped+backfill_toofull, 199 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/738 objects misplaced (2.710%); 0 B/s, 1 objects/s recovering 2024-02-23T00:40:55.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:54.655082+0000 mon.a (mon.0) 3353 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:55.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:54.788058+0000 osd.5 (osd.5) 249 : cluster [DBG] 5.f deep-scrub starts 2024-02-23T00:40:55.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:54.788858+0000 osd.5 (osd.5) 250 : cluster [DBG] 5.f deep-scrub ok 2024-02-23T00:40:55.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:55.191405+0000 mon.a (mon.0) 3354 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:55.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:55.191470+0000 mon.a (mon.0) 3355 : cluster [DBG] osdmap e636: 8 total, 7 up, 7 in 2024-02-23T00:40:55.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:55.193406+0000 mon.a (mon.0) 3356 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106"}]: dispatch 2024-02-23T00:40:55.545 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:55.616 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:40:55.617 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:40:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:53.756277+0000 mgr.y (mgr.24437) 560 : cluster [DBG] pgmap v968: 204 pgs: 3 peering, 2 active+remapped+backfill_toofull, 199 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 20/738 objects misplaced (2.710%); 0 B/s, 1 objects/s recovering 2024-02-23T00:40:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:54.655082+0000 mon.a (mon.0) 3353 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:40:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:54.788058+0000 osd.5 (osd.5) 249 : cluster [DBG] 5.f deep-scrub starts 2024-02-23T00:40:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:54.788858+0000 osd.5 (osd.5) 250 : cluster [DBG] 5.f deep-scrub ok 2024-02-23T00:40:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:55.191405+0000 mon.a (mon.0) 3354 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:40:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:55.191470+0000 mon.a (mon.0) 3355 : cluster [DBG] osdmap e636: 8 total, 7 up, 7 in 2024-02-23T00:40:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:55.193406+0000 mon.a (mon.0) 3356 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106"}]: dispatch 2024-02-23T00:40:56.033 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:40:56.033 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:40:56.033 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.meta 2024-02-23T00:40:56.033 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.meta by 10 min 8 2024-02-23T00:40:56.034 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:40:56.363 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-23T00:40:56.394 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:56.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:56.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:56.499 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:56.524 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:56.682 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:55.815369+0000 osd.5 (osd.5) 251 : cluster [DBG] 2.18 deep-scrub starts 2024-02-23T00:40:56.682 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:55.816151+0000 osd.5 (osd.5) 252 : cluster [DBG] 2.18 deep-scrub ok 2024-02-23T00:40:56.682 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:56.188111+0000 mon.a (mon.0) 3357 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:40:56.682 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:56.192545+0000 mon.a (mon.0) 3358 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106"}]': finished 2024-02-23T00:40:56.682 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:56.192645+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e637: 8 total, 7 up, 7 in 2024-02-23T00:40:56.682 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:55.815369+0000 osd.5 (osd.5) 251 : cluster [DBG] 2.18 deep-scrub starts 2024-02-23T00:40:56.683 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:55.816151+0000 osd.5 (osd.5) 252 : cluster [DBG] 2.18 deep-scrub ok 2024-02-23T00:40:56.683 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:56.188111+0000 mon.a (mon.0) 3357 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:40:56.683 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:56.192545+0000 mon.a (mon.0) 3358 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106"}]': finished 2024-02-23T00:40:56.683 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:56.192645+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e637: 8 total, 7 up, 7 in 2024-02-23T00:40:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:55.815369+0000 osd.5 (osd.5) 251 : cluster [DBG] 2.18 deep-scrub starts 2024-02-23T00:40:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:55.816151+0000 osd.5 (osd.5) 252 : cluster [DBG] 2.18 deep-scrub ok 2024-02-23T00:40:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:56.188111+0000 mon.a (mon.0) 3357 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:40:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:56.192545+0000 mon.a (mon.0) 3358 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-106"}]': finished 2024-02-23T00:40:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:56.192645+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e637: 8 total, 7 up, 7 in 2024-02-23T00:40:56.722 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:56.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:56.759 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:56.826 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:56.853 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:57.281 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:57.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:57.385 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:57.413 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:57.431 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:57.431 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:40:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:57.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:57.535 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:57.562 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:57.580 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:55.757264+0000 mgr.y (mgr.24437) 561 : cluster [DBG] pgmap v971: 204 pgs: 7 active+clean+snaptrim, 4 clean+premerge+peered, 2 active+remapped+backfill_toofull, 191 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/708 objects misplaced (2.825%) 2024-02-23T00:40:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:56.198999+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:40:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:56.203194+0000 osd.1 (osd.1) 293 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:40:57.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:55.757264+0000 mgr.y (mgr.24437) 561 : cluster [DBG] pgmap v971: 204 pgs: 7 active+clean+snaptrim, 4 clean+premerge+peered, 2 active+remapped+backfill_toofull, 191 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/708 objects misplaced (2.825%) 2024-02-23T00:40:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:56.198999+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:40:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:56.203194+0000 osd.1 (osd.1) 293 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:40:57.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:57.685 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:55.757264+0000 mgr.y (mgr.24437) 561 : cluster [DBG] pgmap v971: 204 pgs: 7 active+clean+snaptrim, 4 clean+premerge+peered, 2 active+remapped+backfill_toofull, 191 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/708 objects misplaced (2.825%) 2024-02-23T00:40:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:56.198999+0000 osd.1 (osd.1) 292 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:40:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:56.203194+0000 osd.1 (osd.1) 293 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:40:57.711 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:57.729 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:57.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:57.834 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:57.860 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:57.879 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:57.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:57.982 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:58.009 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:58.027 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:58.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:58.131 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:58.157 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:58.405 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:58.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:40:58.509 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:40:58.536 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:58.555 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:40:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:58.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 -- 172.21.15.87:0/146454636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c00fe510 msgr2=0x7f11c00f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/146454636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c00fe510 0x7f11c00f7d40 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 rx=0x7f11bc004870 tx=0x7f11bc009cb0).stop 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 -- 172.21.15.87:0/146454636 shutdown_connections 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/146454636 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c00f8420 0x7f11c0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/146454636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c00fdbb0 0x7f11c00fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/146454636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c00fe510 0x7f11c00f7d40 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 -- 172.21.15.87:0/146454636 >> 172.21.15.87:0/146454636 conn(0x7f11c00f3800 msgr2=0x7f11c00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 -- 172.21.15.87:0/146454636 shutdown_connections 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 -- 172.21.15.87:0/146454636 wait complete. 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 librados: starting msgr at 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.588+0000 7f11c7c9f700 1 librados: starting objecter 2024-02-23T00:40:58.619 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 Processor -- start 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 -- start start 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 librados: setting wanted keys 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 librados: calling monclient init 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c00f8420 0x7f11c010dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c00fdbb0 0x7f11c010e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c00fe510 0x7f11c0108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c0108640 con 0x7f11c00f8420 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c01087b0 con 0x7f11c00fe510 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c7c9f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c0108970 con 0x7f11c00fdbb0 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c523a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c00fdbb0 0x7f11c010e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c5a3b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c00f8420 0x7f11c010dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.620 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c5a3b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c00f8420 0x7f11c010dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45652/0 (socket says 172.21.15.87:45652) 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c5a3b700 1 -- 172.21.15.87:0/3221767621 learned_addr learned my addr 172.21.15.87:0/3221767621 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c523a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c00fdbb0 0x7f11c010e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50466/0 (socket says 172.21.15.87:50466) 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c623c700 1 --2- 172.21.15.87:0/3221767621 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c00fe510 0x7f11c0108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c523a700 1 -- 172.21.15.87:0/3221767621 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c00fe510 msgr2=0x7f11c0108150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c523a700 1 --2- 172.21.15.87:0/3221767621 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f11c00fe510 0x7f11c0108150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c523a700 1 -- 172.21.15.87:0/3221767621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c00f8420 msgr2=0x7f11c010dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c523a700 1 --2- 172.21.15.87:0/3221767621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c00f8420 0x7f11c010dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c523a700 1 -- 172.21.15.87:0/3221767621 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b80097c0 con 0x7f11c00fdbb0 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c5a3b700 1 --2- 172.21.15.87:0/3221767621 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f11c00f8420 0x7f11c010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11c523a700 1 --2- 172.21.15.87:0/3221767621 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f11c00fdbb0 0x7f11c010e4a0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 rx=0x7f11bc004870 tx=0x7f11bc00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11b6ffd700 10 client.?.objecter ms_handle_connect 0x7f11c00fdbb0 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11b6ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.589+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc00e070 con 0x7f11c00fdbb0 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.590+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11bc013da0 con 0x7f11c00fdbb0 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.590+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc005040 con 0x7f11c00fdbb0 2024-02-23T00:40:58.621 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.590+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc019070 con 0x7f11c00fdbb0 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.590+0000 7f11c7c9f700 10 client.38126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.590+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c0108ee0 con 0x7f11c00fdbb0 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.590+0000 7f11c7c9f700 1 librados: init done 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.590+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f11c01070a0 con 0x7f11c00fdbb0 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f11bc01e420 con 0x7f11c00fdbb0 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11b6ffd700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6800/1231597463 conn(0x7f11ac062c90 0x7f11ac065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(639..639 src has 173..639) v4 ==== 8535+0+0 (secure 0 0 0) 0x7f11bc07dd30 con 0x7f11c00fdbb0 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 osd_map(639..639 src has 173..639) v4 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11b6ffd700 3 client.38126.objecter handle_osd_map got epochs [639,639] > 0 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11b6ffd700 3 client.38126.objecter handle_osd_map decoding full epoch 639 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11c5a3b700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6800/1231597463 conn(0x7f11ac062c90 0x7f11ac065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11b6ffd700 20 client.38126.objecter dump_active .. 0 homeless 2024-02-23T00:40:58.622 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v639) v1 ==== 54+0+16 (secure 0 0 0) 0x7f11bc0803c0 con 0x7f11c00fdbb0 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11b6ffd700 10 client.38126.objecter ms_handle_connect 0x7f11ac062c90 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11c5a3b700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6800/1231597463 conn(0x7f11ac062c90 0x7f11ac065150 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 rx=0x7f11b00041d0 tx=0x7f11b0009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 1 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.591+0000 7f11c7c9f700 20 client.38126.objecter _get_session osd=-1 returning homeless 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 10 client.38126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=640}) v3 -- 0x7f11c0109630 con 0x7f11c00fdbb0 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f11c0114de0 con 0x7f11c00fdbb0 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=639) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11bc051170 con 0x7f11c00fdbb0 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11b7fff700 10 client.38126.objecter _check_command_map_dne tid 1 current 639 map_dne_bound 639 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11b7fff700 10 client.38126.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11b7fff700 15 client.38126.objecter _session_command_op_remove -1 1 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 2 2024-02-23T00:40:58.623 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6805/685354171 conn(0x7f11c00fd3d0 0x7f11c00273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c00fd260 osd=1 3 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c00fd260 osd=1 3 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c00fd260 osd=1 3 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 2 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c00fd260 osd=1 3 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 1 2 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c00fd260 osd=1 4 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 10 client.38126.objecter _send_command 2 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11c0059e00 con 0x7f11c00fd3d0 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c623c700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6805/685354171 conn(0x7f11c00fd3d0 0x7f11c00273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11b6ffd700 10 client.38126.objecter ms_handle_connect 0x7f11c00fd3d0 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.592+0000 7f11c623c700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6805/685354171 conn(0x7f11c00fd3d0 0x7f11c00273c0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.593+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f11c0059e00 con 0x7f11c00fd3d0 2024-02-23T00:40:58.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.593+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.593+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 2 = system:0 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.593+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c00fd260 osd=1 4 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.593+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 1 2 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 3 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c00fd260 osd=1 3 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c00fd260 osd=1 3 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c00fd260 osd=1 3 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 3 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c00fd260 osd=1 3 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 1 3 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c00fd260 osd=1 4 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 10 client.38126.objecter _send_command 3 2024-02-23T00:40:58.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.607+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11c00278b0 con 0x7f11c00fd3d0 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11c00278b0 con 0x7f11c00fd3d0 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c00fd260 osd=1 4 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 1 3 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 4 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6809/3605394228 conn(0x7f11c00594d0 0x7f11c0118f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c0059360 osd=2 3 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c0059360 osd=2 3 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c0059360 osd=2 3 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 4 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c0059360 osd=2 3 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 2 4 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c0059360 osd=2 4 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 10 client.38126.objecter _send_command 4 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11c0028730 con 0x7f11c00594d0 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c5a3b700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6809/3605394228 conn(0x7f11c00594d0 0x7f11c0118f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11b6ffd700 10 client.38126.objecter ms_handle_connect 0x7f11c00594d0 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.609+0000 7f11c5a3b700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6809/3605394228 conn(0x7f11c00594d0 0x7f11c0118f70 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.610+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f11c0028730 con 0x7f11c00594d0 2024-02-23T00:40:58.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.610+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:40:58.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.610+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 4 = system:0 2024-02-23T00:40:58.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.610+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c0059360 osd=2 4 2024-02-23T00:40:58.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.610+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 2 4 2024-02-23T00:40:58.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:58.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 5 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c0059360 osd=2 3 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c0059360 osd=2 3 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c0059360 osd=2 3 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 5 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c0059360 osd=2 3 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 2 5 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c0059360 osd=2 4 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 10 client.38126.objecter _send_command 5 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.625+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11c0027c90 con 0x7f11c00594d0 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11c0027c90 con 0x7f11c00594d0 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c0059360 osd=2 4 2024-02-23T00:40:58.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 2 5 2024-02-23T00:40:58.630 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 6 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6813/128059790 conn(0x7f11c011a5e0 0x7f11c011c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c00ff0e0 osd=3 3 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c00ff0e0 osd=3 3 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c00ff0e0 osd=3 3 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 6 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c00ff0e0 osd=3 3 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 3 6 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c00ff0e0 osd=3 4 2024-02-23T00:40:58.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 10 client.38126.objecter _send_command 6 2024-02-23T00:40:58.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11c00ff8c0 con 0x7f11c011a5e0 2024-02-23T00:40:58.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.626+0000 7f11c523a700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6813/128059790 conn(0x7f11c011a5e0 0x7f11c011c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.627+0000 7f11b6ffd700 10 client.38126.objecter ms_handle_connect 0x7f11c011a5e0 2024-02-23T00:40:58.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.627+0000 7f11c523a700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.87:6813/128059790 conn(0x7f11c011a5e0 0x7f11c011c9d0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:58.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.627+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f11c00ff8c0 con 0x7f11c011a5e0 2024-02-23T00:40:58.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.627+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:40:58.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.627+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 6 = system:0 2024-02-23T00:40:58.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.627+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c00ff0e0 osd=3 4 2024-02-23T00:40:58.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.627+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 3 6 2024-02-23T00:40:58.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 7 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c00ff0e0 osd=3 3 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c00ff0e0 osd=3 3 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c00ff0e0 osd=3 3 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 7 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c00ff0e0 osd=3 3 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 3 7 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c00ff0e0 osd=3 4 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 10 client.38126.objecter _send_command 7 2024-02-23T00:40:58.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.641+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11c00fec10 con 0x7f11c011a5e0 2024-02-23T00:40:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11c00fec10 con 0x7f11c011a5e0 2024-02-23T00:40:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c00ff0e0 osd=3 4 2024-02-23T00:40:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 3 7 2024-02-23T00:40:58.646 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 8 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6800/4175844757 conn(0x7f11c011e040 0x7f11c011e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c0119c30 osd=4 3 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c0119c30 osd=4 3 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c0119c30 osd=4 3 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 8 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c0119c30 osd=4 3 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 4 8 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c0119c30 osd=4 4 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 10 client.38126.objecter _send_command 8 2024-02-23T00:40:58.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11c011a370 con 0x7f11c011e040 2024-02-23T00:40:58.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.643+0000 7f11c623c700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6800/4175844757 conn(0x7f11c011e040 0x7f11c011e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.644+0000 7f11b6ffd700 10 client.38126.objecter ms_handle_connect 0x7f11c011e040 2024-02-23T00:40:58.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.644+0000 7f11c623c700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6800/4175844757 conn(0x7f11c011e040 0x7f11c011e480 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:58.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.644+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f11c011a370 con 0x7f11c011e040 2024-02-23T00:40:58.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.644+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:40:58.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.644+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 8 = system:0 2024-02-23T00:40:58.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.644+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c0119c30 osd=4 4 2024-02-23T00:40:58.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.644+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 4 8 2024-02-23T00:40:58.659 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:58.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 9 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c0119c30 osd=4 3 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c0119c30 osd=4 3 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c0119c30 osd=4 3 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 9 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c0119c30 osd=4 3 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 4 9 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c0119c30 osd=4 4 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 10 client.38126.objecter _send_command 9 2024-02-23T00:40:58.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.659+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11c011f140 con 0x7f11c011e040 2024-02-23T00:40:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.660+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11c011f140 con 0x7f11c011e040 2024-02-23T00:40:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.660+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.660+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.660+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c0119c30 osd=4 4 2024-02-23T00:40:58.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.660+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 4 9 2024-02-23T00:40:58.663 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:40:58.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:58.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 10 2024-02-23T00:40:58.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6804/2550013795 conn(0x7f11c0121680 0x7f11c0123a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c011e970 osd=5 3 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c011e970 osd=5 3 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c011e970 osd=5 3 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 10 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c011e970 osd=5 3 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 5 10 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c011e970 osd=5 4 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 10 client.38126.objecter _send_command 10 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11c011f110 con 0x7f11c0121680 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c5a3b700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6804/2550013795 conn(0x7f11c0121680 0x7f11c0123a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11b6ffd700 10 client.38126.objecter ms_handle_connect 0x7f11c0121680 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.661+0000 7f11c5a3b700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6804/2550013795 conn(0x7f11c0121680 0x7f11c0123a70 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.662+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f11c011f110 con 0x7f11c0121680 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.662+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:40:58.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.662+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 10 = system:0 2024-02-23T00:40:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.662+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c011e970 osd=5 4 2024-02-23T00:40:58.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.662+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 5 10 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 11 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c011e970 osd=5 3 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c011e970 osd=5 3 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c011e970 osd=5 3 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 11 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c011e970 osd=5 3 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 5 11 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c011e970 osd=5 4 2024-02-23T00:40:58.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 10 client.38126.objecter _send_command 11 2024-02-23T00:40:58.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.677+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11c011f110 con 0x7f11c0121680 2024-02-23T00:40:58.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11c011f110 con 0x7f11c0121680 2024-02-23T00:40:58.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:58.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c011e970 osd=5 4 2024-02-23T00:40:58.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 5 11 2024-02-23T00:40:58.681 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:40:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 12 2024-02-23T00:40:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6808/2509408498 conn(0x7f11c006b420 0x7f11c0125f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c011f110 osd=6 3 2024-02-23T00:40:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c011f110 osd=6 3 2024-02-23T00:40:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c011f110 osd=6 3 2024-02-23T00:40:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 12 2024-02-23T00:40:58.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c011f110 osd=6 3 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 6 12 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c011f110 osd=6 4 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 10 client.38126.objecter _send_command 12 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.678+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11c006ce40 con 0x7f11c006b420 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.679+0000 7f11c523a700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6808/2509408498 conn(0x7f11c006b420 0x7f11c0125f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.679+0000 7f11c523a700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6808/2509408498 conn(0x7f11c006b420 0x7f11c0125f70 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.679+0000 7f11b6ffd700 10 client.38126.objecter ms_handle_connect 0x7f11c006b420 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.680+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f11c006ce40 con 0x7f11c006b420 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.680+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.680+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 12 = system:0 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.680+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c011f110 osd=6 4 2024-02-23T00:40:58.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.680+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 6 12 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:56.826890+0000 osd.7 (osd.7) 290 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:56.829787+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:57.171723+0000 osd.1 (osd.1) 294 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:57.174941+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:57.205238+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e638: 8 total, 7 up, 7 in 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:57.486661+0000 osd.6 (osd.6) 260 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:57.491316+0000 osd.6 (osd.6) 261 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:57.757792+0000 mgr.y (mgr.24437) 562 : cluster [DBG] pgmap v974: 170 pgs: 4 active+clean+snaptrim, 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/708 objects misplaced (2.825%) 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:57.758636+0000 mon.a (mon.0) 3361 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-23T00:40:58.686 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:57.758820+0000 mon.a (mon.0) 3362 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:58.204331+0000 mon.a (mon.0) 3363 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]': finished 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:58.204403+0000 mon.a (mon.0) 3364 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:58.204450+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e639: 8 total, 7 up, 7 in 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:58.207305+0000 mon.a (mon.0) 3366 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:56.826890+0000 osd.7 (osd.7) 290 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:56.829787+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:57.171723+0000 osd.1 (osd.1) 294 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:57.174941+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:57.205238+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e638: 8 total, 7 up, 7 in 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:57.486661+0000 osd.6 (osd.6) 260 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:40:58.687 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:57.491316+0000 osd.6 (osd.6) 261 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:40:58.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:57.757792+0000 mgr.y (mgr.24437) 562 : cluster [DBG] pgmap v974: 170 pgs: 4 active+clean+snaptrim, 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/708 objects misplaced (2.825%) 2024-02-23T00:40:58.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:57.758636+0000 mon.a (mon.0) 3361 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-23T00:40:58.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:57.758820+0000 mon.a (mon.0) 3362 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-23T00:40:58.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:58.204331+0000 mon.a (mon.0) 3363 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]': finished 2024-02-23T00:40:58.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:58.204403+0000 mon.a (mon.0) 3364 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-23T00:40:58.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:58.204450+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e639: 8 total, 7 up, 7 in 2024-02-23T00:40:58.688 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:58.207305+0000 mon.a (mon.0) 3366 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:58.688 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:58.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:58.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 13 2024-02-23T00:40:58.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c011f110 osd=6 3 2024-02-23T00:40:58.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c011f110 osd=6 3 2024-02-23T00:40:58.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c011f110 osd=6 3 2024-02-23T00:40:58.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 13 2024-02-23T00:40:58.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c011f110 osd=6 3 2024-02-23T00:40:58.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 6 13 2024-02-23T00:40:58.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c011f110 osd=6 4 2024-02-23T00:40:58.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 10 client.38126.objecter _send_command 13 2024-02-23T00:40:58.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.699+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11c006b860 con 0x7f11c006b420 2024-02-23T00:40:58.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11c006b860 con 0x7f11c006b420 2024-02-23T00:40:58.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:58.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c011f110 osd=6 4 2024-02-23T00:40:58.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 6 13 2024-02-23T00:40:58.704 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 14 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6812/373787000 conn(0x7f11c006bf40 0x7f11c006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c006b860 osd=7 3 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c006b860 osd=7 3 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c006b860 osd=7 3 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 14 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c006b860 osd=7 3 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 7 14 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c006b860 osd=7 4 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 10 client.38126.objecter _send_command 14 2024-02-23T00:40:58.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11c006c8c0 con 0x7f11c006bf40 2024-02-23T00:40:58.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.701+0000 7f11c623c700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6812/373787000 conn(0x7f11c006bf40 0x7f11c006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:40:58.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.702+0000 7f11b6ffd700 10 client.38126.objecter ms_handle_connect 0x7f11c006bf40 2024-02-23T00:40:58.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.702+0000 7f11c623c700 1 --2- 172.21.15.87:0/3221767621 >> v2:172.21.15.171:6812/373787000 conn(0x7f11c006bf40 0x7f11c006c3a0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:40:58.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.702+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f11c006c8c0 con 0x7f11c006bf40 2024-02-23T00:40:58.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.702+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:40:58.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.702+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 14 = system:0 2024-02-23T00:40:58.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.702+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c006b860 osd=7 4 2024-02-23T00:40:58.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.702+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 7 14 2024-02-23T00:40:58.713 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:58.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:58.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 10 client.38126.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign -1 15 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c006b860 osd=7 3 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c006b860 osd=7 3 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 20 client.38126.objecter _get_session s=0x7f11c006b860 osd=7 3 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_remove -1 15 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 20 client.38126.objecter get_session s=0x7f11c006b860 osd=7 3 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 15 client.38126.objecter _session_command_op_assign 7 15 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 20 client.38126.objecter put_session s=0x7f11c006b860 osd=7 4 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 10 client.38126.objecter _send_command 15 2024-02-23T00:40:58.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.720+0000 7f11c7c9f700 1 -- 172.21.15.87:0/3221767621 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f11c006c8c0 con 0x7f11c006bf40 2024-02-23T00:40:58.724 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.722+0000 7f11b6ffd700 1 -- 172.21.15.87:0/3221767621 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f11c006c8c0 con 0x7f11c006bf40 2024-02-23T00:40:58.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.722+0000 7f11b6ffd700 10 client.38126.objecter ms_dispatch 0x7f11c0001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:40:58.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.722+0000 7f11b6ffd700 10 client.38126.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.722+0000 7f11b6ffd700 20 client.38126.objecter put_session s=0x7f11c006b860 osd=7 4 2024-02-23T00:40:58.725 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:40:58.722+0000 7f11b6ffd700 15 client.38126.objecter _session_command_op_remove 7 15 2024-02-23T00:40:58.725 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:40:58.725 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:40:58.725 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.726 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.726 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.726 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.726 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.726 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.726 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:40:58.818 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:40:58.843 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:56.826890+0000 osd.7 (osd.7) 290 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:40:58.843 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:56.829787+0000 osd.7 (osd.7) 291 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:40:58.843 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:57.171723+0000 osd.1 (osd.1) 294 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:57.174941+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:57.205238+0000 mon.a (mon.0) 3360 : cluster [DBG] osdmap e638: 8 total, 7 up, 7 in 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:57.486661+0000 osd.6 (osd.6) 260 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:57.491316+0000 osd.6 (osd.6) 261 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:57.757792+0000 mgr.y (mgr.24437) 562 : cluster [DBG] pgmap v974: 170 pgs: 4 active+clean+snaptrim, 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/708 objects misplaced (2.825%) 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:57.758636+0000 mon.a (mon.0) 3361 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:57.758820+0000 mon.a (mon.0) 3362 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:58.204331+0000 mon.a (mon.0) 3363 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]': finished 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:58.204403+0000 mon.a (mon.0) 3364 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:58.204450+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e639: 8 total, 7 up, 7 in 2024-02-23T00:40:58.844 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:58.207305+0000 mon.a (mon.0) 3366 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:40:58.845 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:58.863 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:40:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:58.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:40:58.967 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:40:58.993 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:59.012 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:40:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:59.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:40:59.044 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:40:59.116 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:59.143 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:59.165 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:59.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:59.277 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:40:59.304 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:59.653 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:40:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:59.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:40:59.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:40:59.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:58.395358+0000 osd.3 (osd.3) 199 : cluster [DBG] 4.1d deep-scrub starts 2024-02-23T00:40:59.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:58.399359+0000 osd.3 (osd.3) 200 : cluster [DBG] 4.1d deep-scrub ok 2024-02-23T00:40:59.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:58.497277+0000 osd.6 (osd.6) 262 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:40:59.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:58.498747+0000 osd.6 (osd.6) 263 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:40:59.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:58.592701+0000 mon.c (mon.2) 479 : audit [DBG] from='client.? 172.21.15.87:0/3221767621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:59.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:40:59.206264+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:59.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:59.206445+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e640: 8 total, 7 up, 7 in 2024-02-23T00:40:59.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:58.395358+0000 osd.3 (osd.3) 199 : cluster [DBG] 4.1d deep-scrub starts 2024-02-23T00:40:59.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:58.399359+0000 osd.3 (osd.3) 200 : cluster [DBG] 4.1d deep-scrub ok 2024-02-23T00:40:59.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:58.497277+0000 osd.6 (osd.6) 262 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:40:59.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:58.498747+0000 osd.6 (osd.6) 263 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:40:59.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:58.592701+0000 mon.c (mon.2) 479 : audit [DBG] from='client.? 172.21.15.87:0/3221767621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:59.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:40:59.206264+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:59.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:40:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:59.206445+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e640: 8 total, 7 up, 7 in 2024-02-23T00:40:59.785 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:40:59.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:58.395358+0000 osd.3 (osd.3) 199 : cluster [DBG] 4.1d deep-scrub starts 2024-02-23T00:40:59.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:58.399359+0000 osd.3 (osd.3) 200 : cluster [DBG] 4.1d deep-scrub ok 2024-02-23T00:40:59.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:58.497277+0000 osd.6 (osd.6) 262 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:40:59.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:58.498747+0000 osd.6 (osd.6) 263 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:40:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:58.592701+0000 mon.c (mon.2) 479 : audit [DBG] from='client.? 172.21.15.87:0/3221767621' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:40:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:40:59.206264+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:40:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:40:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:59.206445+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e640: 8 total, 7 up, 7 in 2024-02-23T00:40:59.982 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:40:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:40:59.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:00.086 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:00.113 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:59.124281+0000 osd.2 (osd.2) 231 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:41:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:59.125240+0000 osd.2 (osd.2) 232 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:41:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:59.668197+0000 osd.4 (osd.4) 287 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:59.669019+0000 osd.4 (osd.4) 288 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:59.758376+0000 mgr.y (mgr.24437) 563 : cluster [DBG] pgmap v977: 202 pgs: 32 unknown, 4 active+clean+snaptrim, 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/708 objects misplaced (2.825%) 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:59.774481+0000 osd.7 (osd.7) 292 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:40:59.777387+0000 osd.7 (osd.7) 293 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:00.206707+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e641: 8 total, 7 up, 7 in 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:00.216107+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:59.124281+0000 osd.2 (osd.2) 231 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:59.125240+0000 osd.2 (osd.2) 232 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:59.668197+0000 osd.4 (osd.4) 287 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:41:00.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:59.669019+0000 osd.4 (osd.4) 288 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:41:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:59.758376+0000 mgr.y (mgr.24437) 563 : cluster [DBG] pgmap v977: 202 pgs: 32 unknown, 4 active+clean+snaptrim, 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/708 objects misplaced (2.825%) 2024-02-23T00:41:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:59.774481+0000 osd.7 (osd.7) 292 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:41:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:40:59.777387+0000 osd.7 (osd.7) 293 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:41:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:00.206707+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e641: 8 total, 7 up, 7 in 2024-02-23T00:41:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:00.216107+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:59.124281+0000 osd.2 (osd.2) 231 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:41:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:59.125240+0000 osd.2 (osd.2) 232 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:41:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:59.668197+0000 osd.4 (osd.4) 287 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:41:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:59.669019+0000 osd.4 (osd.4) 288 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:41:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:59.758376+0000 mgr.y (mgr.24437) 563 : cluster [DBG] pgmap v977: 202 pgs: 32 unknown, 4 active+clean+snaptrim, 2 clean+premerge+peered, 2 active+remapped+backfill_toofull, 162 active+clean; 5.7 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/708 objects misplaced (2.825%) 2024-02-23T00:41:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:59.774481+0000 osd.7 (osd.7) 292 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:41:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:40:59.777387+0000 osd.7 (osd.7) 293 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:41:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:00.206707+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e641: 8 total, 7 up, 7 in 2024-02-23T00:41:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:00.216107+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:00.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 -- 172.21.15.87:0/476792451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 msgr2=0x7f4f640fdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:00.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 --2- 172.21.15.87:0/476792451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 0x7f4f640fdfb0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 rx=0x7f4f60004870 tx=0x7f4f60009cb0).stop 2024-02-23T00:41:00.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 -- 172.21.15.87:0/476792451 shutdown_connections 2024-02-23T00:41:00.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 --2- 172.21.15.87:0/476792451 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f640f6a90 0x7f4f64104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:00.981 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 --2- 172.21.15.87:0/476792451 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 0x7f4f640fdfb0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 --2- 172.21.15.87:0/476792451 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f640fe530 0x7f4f640f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 -- 172.21.15.87:0/476792451 >> 172.21.15.87:0/476792451 conn(0x7f4f6406c370 msgr2=0x7f4f6406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 -- 172.21.15.87:0/476792451 shutdown_connections 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 -- 172.21.15.87:0/476792451 wait complete. 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 librados: starting msgr at 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 librados: starting objecter 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 Processor -- start 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 -- start start 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 librados: setting wanted keys 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 librados: calling monclient init 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f640f6a90 0x7f4f640f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.973+0000 7f4f6b312700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 0x7f4f640f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f6b312700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f640fe530 0x7f4f640f8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:00.982 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f6b312700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f641165a0 con 0x7f4f640fdbd0 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f6b312700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f64116710 con 0x7f4f640f6a90 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f6b312700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f641168d0 con 0x7f4f640fe530 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f688ad700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 0x7f4f640f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f698af700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f640fe530 0x7f4f640f8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f688ad700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 0x7f4f640f7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:45676/0 (socket says 172.21.15.87:45676) 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f688ad700 1 -- 172.21.15.87:0/2247994363 learned_addr learned my addr 172.21.15.87:0/2247994363 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f690ae700 1 --2- 172.21.15.87:0/2247994363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f640f6a90 0x7f4f640f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f688ad700 1 -- 172.21.15.87:0/2247994363 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f640fe530 msgr2=0x7f4f640f8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f688ad700 1 --2- 172.21.15.87:0/2247994363 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f640fe530 0x7f4f640f8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f688ad700 1 -- 172.21.15.87:0/2247994363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f640f6a90 msgr2=0x7f4f640f7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f688ad700 1 --2- 172.21.15.87:0/2247994363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f640f6a90 0x7f4f640f7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f688ad700 1 -- 172.21.15.87:0/2247994363 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c0097c0 con 0x7f4f640fdbd0 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f698af700 1 --2- 172.21.15.87:0/2247994363 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f640fe530 0x7f4f640f8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:00.983 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f688ad700 1 --2- 172.21.15.87:0/2247994363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 0x7f4f640f7b70 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7f4f540041d0 tx=0x7f4f54009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f5a7fc700 10 client.?.objecter ms_handle_connect 0x7f4f640fdbd0 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f5a7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f5a7fc700 1 -- 172.21.15.87:0/2247994363 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54007b40 con 0x7f4f640fdbd0 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f5a7fc700 1 -- 172.21.15.87:0/2247994363 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f5400f040 con 0x7f4f640fdbd0 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f5a7fc700 1 -- 172.21.15.87:0/2247994363 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54014810 con 0x7f4f640fdbd0 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f60005040 con 0x7f4f640fdbd0 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f6b312700 10 client.27660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f64116ec0 con 0x7f4f640fdbd0 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.974+0000 7f4f6b312700 1 librados: init done 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.975+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6405f910 con 0x7f4f640fdbd0 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.975+0000 7f4f5a7fc700 1 -- 172.21.15.87:0/2247994363 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4f54009110 con 0x7f4f640fdbd0 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.976+0000 7f4f5a7fc700 1 --2- 172.21.15.87:0/2247994363 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f50062c90 0x7f4f50065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:00.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.976+0000 7f4f5a7fc700 1 -- 172.21.15.87:0/2247994363 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(641..641 src has 173..641) v4 ==== 8548+0+0 (secure 0 0 0) 0x7f4f5407d3e0 con 0x7f4f640fdbd0 2024-02-23T00:41:00.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.976+0000 7f4f690ae700 1 --2- 172.21.15.87:0/2247994363 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f50062c90 0x7f4f50065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:00.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.976+0000 7f4f5a7fc700 10 client.27660.objecter ms_dispatch 0x7f4f64001550 osd_map(641..641 src has 173..641) v4 2024-02-23T00:41:00.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.976+0000 7f4f5a7fc700 3 client.27660.objecter handle_osd_map got epochs [641,641] > 0 2024-02-23T00:41:00.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.976+0000 7f4f5a7fc700 3 client.27660.objecter handle_osd_map decoding full epoch 641 2024-02-23T00:41:00.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.976+0000 7f4f5a7fc700 20 client.27660.objecter dump_active .. 0 homeless 2024-02-23T00:41:00.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.976+0000 7f4f5a7fc700 10 client.27660.objecter ms_handle_connect 0x7f4f50062c90 2024-02-23T00:41:00.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.976+0000 7f4f690ae700 1 --2- 172.21.15.87:0/2247994363 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f50062c90 0x7f4f50065150 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 rx=0x7f4f600043d0 tx=0x7f4f6000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:00.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:00.981+0000 7f4f5a7fc700 1 -- 172.21.15.87:0/2247994363 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4f540506c0 con 0x7f4f640fdbd0 2024-02-23T00:41:00.988 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:01.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.149+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f640f8b60 con 0x7f4f50062c90 2024-02-23T00:41:01.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.163+0000 7f4f5a7fc700 1 -- 172.21.15.87:0/2247994363 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+509284 (secure 0 0 0) 0x7f4f640f8b60 con 0x7f4f50062c90 2024-02-23T00:41:01.173 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:41:01.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 10 librados: watch_flush enter 2024-02-23T00:41:01.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 10 librados: watch_flush exit 2024-02-23T00:41:01.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 20 client.27660.objecter shutdown clearing up homeless session... 2024-02-23T00:41:01.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 10 client.27660.objecter successfully canceled tick 2024-02-23T00:41:01.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f50062c90 msgr2=0x7f4f50065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:01.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 --2- 172.21.15.87:0/2247994363 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f50062c90 0x7f4f50065150 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 rx=0x7f4f600043d0 tx=0x7f4f6000b040).stop 2024-02-23T00:41:01.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 msgr2=0x7f4f640f7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 --2- 172.21.15.87:0/2247994363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 0x7f4f640f7b70 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 rx=0x7f4f540041d0 tx=0x7f4f54009f60).stop 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 shutdown_connections 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 --2- 172.21.15.87:0/2247994363 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4f50062c90 0x7f4f50065150 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 --2- 172.21.15.87:0/2247994363 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4f640fe530 0x7f4f640f8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 --2- 172.21.15.87:0/2247994363 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4f640f6a90 0x7f4f640f7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 --2- 172.21.15.87:0/2247994363 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4f640fdbd0 0x7f4f640f7b70 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 >> 172.21.15.87:0/2247994363 conn(0x7f4f6406c370 msgr2=0x7f4f640fb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 shutdown_connections 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 -- 172.21.15.87:0/2247994363 wait complete. 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:01.166+0000 7f4f6b312700 1 librados: shutdown 2024-02-23T00:41:01.179 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:41:01.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:41:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:00.139143+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.c scrub starts 2024-02-23T00:41:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:00.144509+0000 osd.2 (osd.2) 234 : cluster [DBG] 2.c scrub ok 2024-02-23T00:41:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:00.147967+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.c scrub starts 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:00.151027+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.c scrub ok 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:00.641238+0000 osd.4 (osd.4) 289 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:00.643130+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:01.146261+0000 mgr.y (mgr.24437) 564 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:01.151418+0000 mgr.y (mgr.24437) 565 : audit [DBG] from='client.27660 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:01.208379+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.208456+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e642: 8 total, 7 up, 7 in 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:01.208870+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-108"}]: dispatch 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:00.139143+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.c scrub starts 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:00.144509+0000 osd.2 (osd.2) 234 : cluster [DBG] 2.c scrub ok 2024-02-23T00:41:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:00.147967+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.c scrub starts 2024-02-23T00:41:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:00.151027+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.c scrub ok 2024-02-23T00:41:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:00.641238+0000 osd.4 (osd.4) 289 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:41:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:00.643130+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:41:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:01.146261+0000 mgr.y (mgr.24437) 564 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:01.151418+0000 mgr.y (mgr.24437) 565 : audit [DBG] from='client.27660 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:41:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:01.208379+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.208456+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e642: 8 total, 7 up, 7 in 2024-02-23T00:41:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:01.208870+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-108"}]: dispatch 2024-02-23T00:41:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:00.139143+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.c scrub starts 2024-02-23T00:41:01.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:00.144509+0000 osd.2 (osd.2) 234 : cluster [DBG] 2.c scrub ok 2024-02-23T00:41:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:00.147967+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.c scrub starts 2024-02-23T00:41:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:00.151027+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.c scrub ok 2024-02-23T00:41:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:00.641238+0000 osd.4 (osd.4) 289 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:41:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:00.643130+0000 osd.4 (osd.4) 290 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:41:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:01.146261+0000 mgr.y (mgr.24437) 564 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:01.151418+0000 mgr.y (mgr.24437) 565 : audit [DBG] from='client.27660 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:41:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:01.208379+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.208456+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e642: 8 total, 7 up, 7 in 2024-02-23T00:41:01.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:01.208870+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-108"}]: dispatch 2024-02-23T00:41:02.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:02] "GET /metrics HTTP/1.1" 200 179543 "" "Prometheus/2.33.4" 2024-02-23T00:41:02.765 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.090944+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:41:02.765 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.091901+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:41:02.765 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.178586+0000 osd.1 (osd.1) 298 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:41:02.765 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.179663+0000 osd.1 (osd.1) 299 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:41:02.765 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.637502+0000 osd.4 (osd.4) 291 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.638549+0000 osd.4 (osd.4) 292 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.758820+0000 mgr.y (mgr.24437) 566 : cluster [DBG] pgmap v980: 200 pgs: 2 peering, 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:01.759736+0000 mon.a (mon.0) 3374 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:02.131201+0000 osd.1 (osd.1) 300 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:02.134894+0000 osd.1 (osd.1) 301 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:02.189878+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-108"}]': finished 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:02.189942+0000 mon.a (mon.0) 3376 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:02.189981+0000 mon.a (mon.0) 3377 : cluster [DBG] osdmap e643: 8 total, 7 up, 7 in 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:02.192913+0000 mon.a (mon.0) 3378 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-108", "mode": "writeback"}]: dispatch 2024-02-23T00:41:02.766 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.090944+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:41:02.767 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.091901+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:41:02.767 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.178586+0000 osd.1 (osd.1) 298 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:41:02.767 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.179663+0000 osd.1 (osd.1) 299 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:41:02.767 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.637502+0000 osd.4 (osd.4) 291 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:41:02.767 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.638549+0000 osd.4 (osd.4) 292 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:41:02.767 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.758820+0000 mgr.y (mgr.24437) 566 : cluster [DBG] pgmap v980: 200 pgs: 2 peering, 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:02.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:01.759736+0000 mon.a (mon.0) 3374 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-23T00:41:02.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:02.131201+0000 osd.1 (osd.1) 300 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:41:02.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:02.134894+0000 osd.1 (osd.1) 301 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:41:02.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:02.189878+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-108"}]': finished 2024-02-23T00:41:02.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:02.189942+0000 mon.a (mon.0) 3376 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-23T00:41:02.768 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:02.189981+0000 mon.a (mon.0) 3377 : cluster [DBG] osdmap e643: 8 total, 7 up, 7 in 2024-02-23T00:41:02.769 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:02.192913+0000 mon.a (mon.0) 3378 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-108", "mode": "writeback"}]: dispatch 2024-02-23T00:41:02.784 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":978,"stamp":"2024-02-23T00:41:00.204597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5828,"num_objects":235,"num_object_clones":0,"num_object_copies":705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":0,"num_read":1592,"num_read_kb":1386,"num_write":666,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":6960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":845,"ondisk_log_size":845,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":616,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2192864,"kb_used_data":54888,"kb_used_omap":0,"kb_used_meta":2137856,"kb_avail":654023200,"statfs":{"total":671965249536,"available":669719756800,"internally_reserved":0,"allocated":56205312,"data_stored":32210681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2189164544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-7,"num_read_kb":-7,"num_write":-3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":-7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.014924"},"pg_stats":[{"pgid":"238.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"238.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.7","version":"0'0","reported_seq":51,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198006+0000","last_change":"2024-02-23T00:40:54.198006+0000","last_active":"2024-02-23T00:40:54.198006+0000","last_peered":"2024-02-23T00:40:54.198006+0000","last_clean":"2024-02-23T00:40:54.198006+0000","last_became_active":"2024-02-23T00:40:17.211405+0000","last_became_peered":"2024-02-23T00:40:17.211405+0000","last_unstale":"2024-02-23T00:40:54.198006+0000","last_undegraded":"2024-02-23T00:40:54.198006+0000","last_fullsized":"2024-02-23T00:40:54.198006+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"238.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.6","version":"635'6","reported_seq":58,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:40:54.204270+0000","last_change":"2024-02-23T00:40:54.194840+0000","last_active":"2024-02-23T00:40:54.204270+0000","last_peered":"2024-02-23T00:40:54.204270+0000","last_clean":"2024-02-23T00:40:54.204270+0000","last_became_active":"2024-02-23T00:40:17.215263+0000","last_became_peered":"2024-02-23T00:40:17.215263+0000","last_unstale":"2024-02-23T00:40:54.204270+0000","last_undegraded":"2024-02-23T00:40:54.204270+0000","last_fullsized":"2024-02-23T00:40:54.204270+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":9,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"238.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.5","version":"635'5","reported_seq":61,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:40:54.203644+0000","last_change":"2024-02-23T00:40:54.201978+0000","last_active":"2024-02-23T00:40:54.203644+0000","last_peered":"2024-02-23T00:40:54.203644+0000","last_clean":"2024-02-23T00:40:54.203644+0000","last_became_active":"2024-02-23T00:40:23.191006+0000","last_became_peered":"2024-02-23T00:40:23.191006+0000","last_unstale":"2024-02-23T00:40:54.203644+0000","last_undegraded":"2024-02-23T00:40:54.203644+0000","last_fullsized":"2024-02-23T00:40:54.203644+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"238.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.4","version":"635'5","reported_seq":56,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:40:54.203595+0000","last_change":"2024-02-23T00:40:54.194792+0000","last_active":"2024-02-23T00:40:54.203595+0000","last_peered":"2024-02-23T00:40:54.203595+0000","last_clean":"2024-02-23T00:40:54.203595+0000","last_became_active":"2024-02-23T00:40:17.215235+0000","last_became_peered":"2024-02-23T00:40:17.215235+0000","last_unstale":"2024-02-23T00:40:54.203595+0000","last_undegraded":"2024-02-23T00:40:54.203595+0000","last_fullsized":"2024-02-23T00:40:54.203595+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"238.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":51,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.201616+0000","last_change":"2024-02-23T00:40:54.201616+0000","last_active":"2024-02-23T00:40:54.201616+0000","last_peered":"2024-02-23T00:40:54.201616+0000","last_clean":"2024-02-23T00:40:54.201616+0000","last_became_active":"2024-02-23T00:40:17.202244+0000","last_became_peered":"2024-02-23T00:40:17.202244+0000","last_unstale":"2024-02-23T00:40:54.201616+0000","last_undegraded":"2024-02-23T00:40:54.201616+0000","last_fullsized":"2024-02-23T00:40:54.201616+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"238.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.2","version":"0'0","reported_seq":57,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194061+0000","last_change":"2024-02-23T00:40:54.194061+0000","last_active":"2024-02-23T00:40:54.194061+0000","last_peered":"2024-02-23T00:40:54.194061+0000","last_clean":"2024-02-23T00:40:54.194061+0000","last_became_active":"2024-02-23T00:40:23.203522+0000","last_became_peered":"2024-02-23T00:40:23.203522+0000","last_unstale":"2024-02-23T00:40:54.194061+0000","last_undegraded":"2024-02-23T00:40:54.194061+0000","last_fullsized":"2024-02-23T00:40:54.194061+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"238.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.1","version":"607'1","reported_seq":60,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.195386+0000","last_change":"2024-02-23T00:40:54.195386+0000","last_active":"2024-02-23T00:40:54.195386+0000","last_peered":"2024-02-23T00:40:54.195386+0000","last_clean":"2024-02-23T00:40:54.195386+0000","last_became_active":"2024-02-23T00:40:23.202822+0000","last_became_peered":"2024-02-23T00:40:23.202822+0000","last_unstale":"2024-02-23T00:40:54.195386+0000","last_undegraded":"2024-02-23T00:40:54.195386+0000","last_fullsized":"2024-02-23T00:40:54.195386+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"238.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:40:58.199712+0000","last_change":"2024-02-23T00:40:58.199712+0000","last_active":"2024-02-23T00:40:58.199712+0000","last_peered":"2024-02-23T00:40:58.199712+0000","last_clean":"2024-02-23T00:40:58.199712+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:40:58.199712+0000","last_undegraded":"2024-02-23T00:40:58.199712+0000","last_fullsized":"2024-02-23T00:40:58.199712+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:58.199712+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.199712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.0","version":"635'5","reported_seq":55,"reported_epoch":635,"state":"active+clean+snaptrim","last_fresh":"2024-02-23T00:40:54.195179+0000","last_change":"2024-02-23T00:40:54.188486+0000","last_active":"2024-02-23T00:40:54.195179+0000","last_peered":"2024-02-23T00:40:54.195179+0000","last_clean":"2024-02-23T00:40:54.195179+0000","last_became_active":"2024-02-23T00:40:17.201123+0000","last_became_peered":"2024-02-23T00:40:17.201123+0000","last_unstale":"2024-02-23T00:40:54.195179+0000","last_undegraded":"2024-02-23T00:40:54.195179+0000","last_fullsized":"2024-02-23T00:40:54.195179+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"571'11","reported_seq":811,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-23T00:40:55.195614+0000","last_change":"2024-02-23T00:39:49.195375+0000","last_active":"2024-02-23T00:40:55.195614+0000","last_peered":"2024-02-23T00:40:55.195614+0000","last_clean":"2024-02-23T00:40:55.195614+0000","last_became_active":"2024-02-23T00:39:49.195036+0000","last_became_peered":"2024-02-23T00:39:49.195036+0000","last_unstale":"2024-02-23T00:40:55.195614+0000","last_undegraded":"2024-02-23T00:40:55.195614+0000","last_fullsized":"2024-02-23T00:40:55.195614+0000","mapping_epoch":58,"log_start":"379'9","ondisk_log_start":"379'9","created":58,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'9","last_scrub_stamp":"2024-02-23T00:37:05.955707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:37:05.955707+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":724,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199667+0000","last_change":"2024-02-23T00:40:13.948977+0000","last_active":"2024-02-23T00:40:54.199667+0000","last_peered":"2024-02-23T00:40:54.199667+0000","last_clean":"2024-02-23T00:40:54.199667+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:40:54.199667+0000","last_undegraded":"2024-02-23T00:40:54.199667+0000","last_fullsized":"2024-02-23T00:40:54.199667+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:40:13.948915+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:40:13.948915+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":648,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199406+0000","last_change":"2024-02-23T00:39:39.337422+0000","last_active":"2024-02-23T00:40:54.199406+0000","last_peered":"2024-02-23T00:40:54.199406+0000","last_clean":"2024-02-23T00:40:54.199406+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:40:54.199406+0000","last_undegraded":"2024-02-23T00:40:54.199406+0000","last_fullsized":"2024-02-23T00:40:54.199406+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.337303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.337303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":629,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200182+0000","last_change":"2024-02-23T00:40:49.854729+0000","last_active":"2024-02-23T00:40:54.200182+0000","last_peered":"2024-02-23T00:40:54.200182+0000","last_clean":"2024-02-23T00:40:54.200182+0000","last_became_active":"2024-02-23T00:40:29.205312+0000","last_became_peered":"2024-02-23T00:40:29.205312+0000","last_unstale":"2024-02-23T00:40:54.200182+0000","last_undegraded":"2024-02-23T00:40:54.200182+0000","last_fullsized":"2024-02-23T00:40:54.200182+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:49.854635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:49.854635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":576,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199121+0000","last_change":"2024-02-23T00:40:45.172608+0000","last_active":"2024-02-23T00:40:54.199121+0000","last_peered":"2024-02-23T00:40:54.199121+0000","last_clean":"2024-02-23T00:40:54.199121+0000","last_became_active":"2024-02-23T00:40:23.194357+0000","last_became_peered":"2024-02-23T00:40:23.194357+0000","last_unstale":"2024-02-23T00:40:54.199121+0000","last_undegraded":"2024-02-23T00:40:54.199121+0000","last_fullsized":"2024-02-23T00:40:54.199121+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:45.172542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.172542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":1113,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-23T00:40:55.198347+0000","last_change":"2024-02-23T00:39:36.787515+0000","last_active":"2024-02-23T00:40:55.198347+0000","last_peered":"2024-02-23T00:40:55.198347+0000","last_clean":"2024-02-23T00:40:55.198347+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:40:55.198347+0000","last_undegraded":"2024-02-23T00:40:55.198347+0000","last_fullsized":"2024-02-23T00:40:55.198347+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:39:36.787450+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:39:36.787450+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":496,"num_read_kb":496,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":312,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198993+0000","last_change":"2024-02-23T00:40:26.207420+0000","last_active":"2024-02-23T00:40:54.198993+0000","last_peered":"2024-02-23T00:40:54.198993+0000","last_clean":"2024-02-23T00:40:54.198993+0000","last_became_active":"2024-02-23T00:40:26.207084+0000","last_became_peered":"2024-02-23T00:40:26.207084+0000","last_unstale":"2024-02-23T00:40:54.198993+0000","last_undegraded":"2024-02-23T00:40:54.198993+0000","last_fullsized":"2024-02-23T00:40:54.198993+0000","mapping_epoch":606,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":628,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198322+0000","last_change":"2024-02-23T00:40:08.076879+0000","last_active":"2024-02-23T00:40:54.198322+0000","last_peered":"2024-02-23T00:40:54.198322+0000","last_clean":"2024-02-23T00:40:54.198322+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:40:54.198322+0000","last_undegraded":"2024-02-23T00:40:54.198322+0000","last_fullsized":"2024-02-23T00:40:54.198322+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:08.076820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:08.076820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":635,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197544+0000","last_change":"2024-02-23T00:39:20.396906+0000","last_active":"2024-02-23T00:40:54.197544+0000","last_peered":"2024-02-23T00:40:54.197544+0000","last_clean":"2024-02-23T00:40:54.197544+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:40:54.197544+0000","last_undegraded":"2024-02-23T00:40:54.197544+0000","last_fullsized":"2024-02-23T00:40:54.197544+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:20.396815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:39:20.396815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":250,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193347+0000","last_change":"2024-02-23T00:40:33.189003+0000","last_active":"2024-02-23T00:40:54.193347+0000","last_peered":"2024-02-23T00:40:54.193347+0000","last_clean":"2024-02-23T00:40:54.193347+0000","last_became_active":"2024-02-23T00:40:33.188712+0000","last_became_peered":"2024-02-23T00:40:33.188712+0000","last_unstale":"2024-02-23T00:40:54.193347+0000","last_undegraded":"2024-02-23T00:40:54.193347+0000","last_fullsized":"2024-02-23T00:40:54.193347+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":614,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:25.632725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:40:25.632725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":561,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192889+0000","last_change":"2024-02-23T00:39:35.342488+0000","last_active":"2024-02-23T00:40:54.192889+0000","last_peered":"2024-02-23T00:40:54.192889+0000","last_clean":"2024-02-23T00:40:54.192889+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:40:54.192889+0000","last_undegraded":"2024-02-23T00:40:54.192889+0000","last_fullsized":"2024-02-23T00:40:54.192889+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:39:35.342427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":745,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193285+0000","last_change":"2024-02-23T00:40:47.429480+0000","last_active":"2024-02-23T00:40:54.193285+0000","last_peered":"2024-02-23T00:40:54.193285+0000","last_clean":"2024-02-23T00:40:54.193285+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:40:54.193285+0000","last_undegraded":"2024-02-23T00:40:54.193285+0000","last_fullsized":"2024-02-23T00:40:54.193285+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:40:47.429409+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:40:47.429409+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":630,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200847+0000","last_change":"2024-02-23T00:40:23.196078+0000","last_active":"2024-02-23T00:40:54.200847+0000","last_peered":"2024-02-23T00:40:54.200847+0000","last_clean":"2024-02-23T00:40:54.200847+0000","last_became_active":"2024-02-23T00:40:23.195689+0000","last_became_peered":"2024-02-23T00:40:23.195689+0000","last_unstale":"2024-02-23T00:40:54.200847+0000","last_undegraded":"2024-02-23T00:40:54.200847+0000","last_fullsized":"2024-02-23T00:40:54.200847+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:50.194656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:50.194656+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":626,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198564+0000","last_change":"2024-02-23T00:40:25.149933+0000","last_active":"2024-02-23T00:40:54.198564+0000","last_peered":"2024-02-23T00:40:54.198564+0000","last_clean":"2024-02-23T00:40:54.198564+0000","last_became_active":"2024-02-23T00:40:23.194895+0000","last_became_peered":"2024-02-23T00:40:23.194895+0000","last_unstale":"2024-02-23T00:40:54.198564+0000","last_undegraded":"2024-02-23T00:40:54.198564+0000","last_fullsized":"2024-02-23T00:40:54.198564+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:25.149871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:40:25.149871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":642,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194833+0000","last_change":"2024-02-23T00:40:37.197309+0000","last_active":"2024-02-23T00:40:54.194833+0000","last_peered":"2024-02-23T00:40:54.194833+0000","last_clean":"2024-02-23T00:40:54.194833+0000","last_became_active":"2024-02-23T00:40:37.197010+0000","last_became_peered":"2024-02-23T00:40:37.197010+0000","last_unstale":"2024-02-23T00:40:54.194833+0000","last_undegraded":"2024-02-23T00:40:54.194833+0000","last_fullsized":"2024-02-23T00:40:54.194833+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:07.226380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:07.226380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":228,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200170+0000","last_change":"2024-02-23T00:40:23.197061+0000","last_active":"2024-02-23T00:40:54.200170+0000","last_peered":"2024-02-23T00:40:54.200170+0000","last_clean":"2024-02-23T00:40:54.200170+0000","last_became_active":"2024-02-23T00:40:23.196188+0000","last_became_peered":"2024-02-23T00:40:23.196188+0000","last_unstale":"2024-02-23T00:40:54.200170+0000","last_undegraded":"2024-02-23T00:40:54.200170+0000","last_fullsized":"2024-02-23T00:40:54.200170+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:39.205312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:39.205312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":630,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193604+0000","last_change":"2024-02-23T00:40:50.421297+0000","last_active":"2024-02-23T00:40:54.193604+0000","last_peered":"2024-02-23T00:40:54.193604+0000","last_clean":"2024-02-23T00:40:54.193604+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:40:54.193604+0000","last_undegraded":"2024-02-23T00:40:54.193604+0000","last_fullsized":"2024-02-23T00:40:54.193604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:50.421224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.421224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":698,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.190228+0000","last_change":"2024-02-23T00:39:37.346021+0000","last_active":"2024-02-23T00:40:54.190228+0000","last_peered":"2024-02-23T00:40:54.190228+0000","last_clean":"2024-02-23T00:40:54.190228+0000","last_became_active":"2024-02-23T00:38:19.176922+0000","last_became_peered":"2024-02-23T00:38:19.176922+0000","last_unstale":"2024-02-23T00:40:54.190228+0000","last_undegraded":"2024-02-23T00:40:54.190228+0000","last_fullsized":"2024-02-23T00:40:54.190228+0000","mapping_epoch":56,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:39:37.345922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:39:37.345922+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":612,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194904+0000","last_change":"2024-02-23T00:38:18.055894+0000","last_active":"2024-02-23T00:40:54.194904+0000","last_peered":"2024-02-23T00:40:54.194904+0000","last_clean":"2024-02-23T00:40:54.194904+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:40:54.194904+0000","last_undegraded":"2024-02-23T00:40:54.194904+0000","last_fullsized":"2024-02-23T00:40:54.194904+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:18.055803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:18.055803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":611,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200668+0000","last_change":"2024-02-23T00:40:23.200141+0000","last_active":"2024-02-23T00:40:54.200668+0000","last_peered":"2024-02-23T00:40:54.200668+0000","last_clean":"2024-02-23T00:40:54.200668+0000","last_became_active":"2024-02-23T00:40:23.199844+0000","last_became_peered":"2024-02-23T00:40:23.199844+0000","last_unstale":"2024-02-23T00:40:54.200668+0000","last_undegraded":"2024-02-23T00:40:54.200668+0000","last_fullsized":"2024-02-23T00:40:54.200668+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:34.605664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:34.605664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":64,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.201104+0000","last_change":"2024-02-23T00:40:01.300115+0000","last_active":"2024-02-23T00:40:54.201104+0000","last_peered":"2024-02-23T00:40:54.201104+0000","last_clean":"2024-02-23T00:40:54.201104+0000","last_became_active":"2024-02-23T00:40:01.299780+0000","last_became_peered":"2024-02-23T00:40:01.299780+0000","last_unstale":"2024-02-23T00:40:54.201104+0000","last_undegraded":"2024-02-23T00:40:54.201104+0000","last_fullsized":"2024-02-23T00:40:54.201104+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:27.413549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:32:15.855580+0000","last_clean_scrub_stamp":"2024-02-23T00:39:27.413549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":586,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197581+0000","last_change":"2024-02-23T00:39:27.813080+0000","last_active":"2024-02-23T00:40:54.197581+0000","last_peered":"2024-02-23T00:40:54.197581+0000","last_clean":"2024-02-23T00:40:54.197581+0000","last_became_active":"2024-02-23T00:38:49.194450+0000","last_became_peered":"2024-02-23T00:38:49.194450+0000","last_unstale":"2024-02-23T00:40:54.197581+0000","last_undegraded":"2024-02-23T00:40:54.197581+0000","last_fullsized":"2024-02-23T00:40:54.197581+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:27.813005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:27.813005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":144,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193744+0000","last_change":"2024-02-23T00:40:23.206602+0000","last_active":"2024-02-23T00:40:54.193744+0000","last_peered":"2024-02-23T00:40:54.193744+0000","last_clean":"2024-02-23T00:40:54.193744+0000","last_became_active":"2024-02-23T00:40:23.205794+0000","last_became_peered":"2024-02-23T00:40:23.205794+0000","last_unstale":"2024-02-23T00:40:54.193744+0000","last_undegraded":"2024-02-23T00:40:54.193744+0000","last_fullsized":"2024-02-23T00:40:54.193744+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":742,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189149+0000","last_change":"2024-02-23T00:40:51.633022+0000","last_active":"2024-02-23T00:40:54.189149+0000","last_peered":"2024-02-23T00:40:54.189149+0000","last_clean":"2024-02-23T00:40:54.189149+0000","last_became_active":"2024-02-23T00:38:54.171993+0000","last_became_peered":"2024-02-23T00:38:54.171993+0000","last_unstale":"2024-02-23T00:40:54.189149+0000","last_undegraded":"2024-02-23T00:40:54.189149+0000","last_fullsized":"2024-02-23T00:40:54.189149+0000","mapping_epoch":516,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:40:51.632948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:51.632948+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":641,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198801+0000","last_change":"2024-02-23T00:39:40.287714+0000","last_active":"2024-02-23T00:40:54.198801+0000","last_peered":"2024-02-23T00:40:54.198801+0000","last_clean":"2024-02-23T00:40:54.198801+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:40:54.198801+0000","last_undegraded":"2024-02-23T00:40:54.198801+0000","last_fullsized":"2024-02-23T00:40:54.198801+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.287611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.287611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":788,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-23T00:40:55.191069+0000","last_change":"2024-02-23T00:38:35.337273+0000","last_active":"2024-02-23T00:40:55.191069+0000","last_peered":"2024-02-23T00:40:55.191069+0000","last_clean":"2024-02-23T00:40:55.191069+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:40:55.191069+0000","last_undegraded":"2024-02-23T00:40:55.191069+0000","last_fullsized":"2024-02-23T00:40:55.191069+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:38:35.337170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.337170+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":64,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193155+0000","last_change":"2024-02-23T00:40:01.299027+0000","last_active":"2024-02-23T00:40:54.193155+0000","last_peered":"2024-02-23T00:40:54.193155+0000","last_clean":"2024-02-23T00:40:54.193155+0000","last_became_active":"2024-02-23T00:40:01.298668+0000","last_became_peered":"2024-02-23T00:40:01.298668+0000","last_unstale":"2024-02-23T00:40:54.193155+0000","last_undegraded":"2024-02-23T00:40:54.193155+0000","last_fullsized":"2024-02-23T00:40:54.193155+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.135802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.135802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":561,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189787+0000","last_change":"2024-02-23T00:38:34.097327+0000","last_active":"2024-02-23T00:40:54.189787+0000","last_peered":"2024-02-23T00:40:54.189787+0000","last_clean":"2024-02-23T00:40:54.189787+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:40:54.189787+0000","last_undegraded":"2024-02-23T00:40:54.189787+0000","last_fullsized":"2024-02-23T00:40:54.189787+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:34.097243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.097243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":139,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193663+0000","last_change":"2024-02-23T00:40:15.208236+0000","last_active":"2024-02-23T00:40:54.193663+0000","last_peered":"2024-02-23T00:40:54.193663+0000","last_clean":"2024-02-23T00:40:54.193663+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:40:54.193663+0000","last_undegraded":"2024-02-23T00:40:54.193663+0000","last_fullsized":"2024-02-23T00:40:54.193663+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:15.208124+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.208124+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":795,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199969+0000","last_change":"2024-02-23T00:39:25.200274+0000","last_active":"2024-02-23T00:40:54.199969+0000","last_peered":"2024-02-23T00:40:54.199969+0000","last_clean":"2024-02-23T00:40:54.199969+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:40:54.199969+0000","last_undegraded":"2024-02-23T00:40:54.199969+0000","last_fullsized":"2024-02-23T00:40:54.199969+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":625,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198656+0000","last_change":"2024-02-23T00:39:37.339984+0000","last_active":"2024-02-23T00:40:54.198656+0000","last_peered":"2024-02-23T00:40:54.198656+0000","last_clean":"2024-02-23T00:40:54.198656+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:40:54.198656+0000","last_undegraded":"2024-02-23T00:40:54.198656+0000","last_fullsized":"2024-02-23T00:40:54.198656+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:37.339877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:39:37.339877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"606'10","reported_seq":802,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-23T00:40:55.195482+0000","last_change":"2024-02-23T00:40:42.570131+0000","last_active":"2024-02-23T00:40:55.195482+0000","last_peered":"2024-02-23T00:40:55.195482+0000","last_clean":"2024-02-23T00:40:55.195482+0000","last_became_active":"2024-02-23T00:40:23.191923+0000","last_became_peered":"2024-02-23T00:40:23.191923+0000","last_unstale":"2024-02-23T00:40:55.195482+0000","last_undegraded":"2024-02-23T00:40:55.195482+0000","last_fullsized":"2024-02-23T00:40:55.195482+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"606'10","last_scrub_stamp":"2024-02-23T00:40:42.570053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:42.570053+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":94,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194277+0000","last_change":"2024-02-23T00:40:09.663488+0000","last_active":"2024-02-23T00:40:54.194277+0000","last_peered":"2024-02-23T00:40:54.194277+0000","last_clean":"2024-02-23T00:40:54.194277+0000","last_became_active":"2024-02-23T00:40:09.200996+0000","last_became_peered":"2024-02-23T00:40:09.200996+0000","last_unstale":"2024-02-23T00:40:54.194277+0000","last_undegraded":"2024-02-23T00:40:54.194277+0000","last_fullsized":"2024-02-23T00:40:54.194277+0000","mapping_epoch":581,"log_start":"62'1","ondisk_log_start":"62'1","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:40:09.663366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:09.663366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":552,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198062+0000","last_change":"2024-02-23T00:38:16.396902+0000","last_active":"2024-02-23T00:40:54.198062+0000","last_peered":"2024-02-23T00:40:54.198062+0000","last_clean":"2024-02-23T00:40:54.198062+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:40:54.198062+0000","last_undegraded":"2024-02-23T00:40:54.198062+0000","last_fullsized":"2024-02-23T00:40:54.198062+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.396824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":623,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.188385+0000","last_change":"2024-02-23T00:38:08.068624+0000","last_active":"2024-02-23T00:40:54.188385+0000","last_peered":"2024-02-23T00:40:54.188385+0000","last_clean":"2024-02-23T00:40:54.188385+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:40:54.188385+0000","last_undegraded":"2024-02-23T00:40:54.188385+0000","last_fullsized":"2024-02-23T00:40:54.188385+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:08.068559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:08.068559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":712,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189327+0000","last_change":"2024-02-23T00:40:07.359576+0000","last_active":"2024-02-23T00:40:54.189327+0000","last_peered":"2024-02-23T00:40:54.189327+0000","last_clean":"2024-02-23T00:40:54.189327+0000","last_became_active":"2024-02-23T00:38:19.182222+0000","last_became_peered":"2024-02-23T00:38:19.182222+0000","last_unstale":"2024-02-23T00:40:54.189327+0000","last_undegraded":"2024-02-23T00:40:54.189327+0000","last_fullsized":"2024-02-23T00:40:54.189327+0000","mapping_epoch":56,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:40:07.359519+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:40:07.359519+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":640,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.195631+0000","last_change":"2024-02-23T00:40:52.825568+0000","last_active":"2024-02-23T00:40:54.195631+0000","last_peered":"2024-02-23T00:40:54.195631+0000","last_clean":"2024-02-23T00:40:54.195631+0000","last_became_active":"2024-02-23T00:40:23.196664+0000","last_became_peered":"2024-02-23T00:40:23.196664+0000","last_unstale":"2024-02-23T00:40:54.195631+0000","last_undegraded":"2024-02-23T00:40:54.195631+0000","last_fullsized":"2024-02-23T00:40:54.195631+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:40:52.825480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.825480+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":603,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.788996+0000","last_change":"2024-02-23T00:40:54.788960+0000","last_active":"2024-02-23T00:40:54.788996+0000","last_peered":"2024-02-23T00:40:54.788996+0000","last_clean":"2024-02-23T00:40:54.788996+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:40:54.788996+0000","last_undegraded":"2024-02-23T00:40:54.788996+0000","last_fullsized":"2024-02-23T00:40:54.788996+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_clean_scrub_stamp":"2024-02-23T00:40:54.788865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"78'2","reported_seq":82,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200853+0000","last_change":"2024-02-23T00:40:13.179766+0000","last_active":"2024-02-23T00:40:54.200853+0000","last_peered":"2024-02-23T00:40:54.200853+0000","last_clean":"2024-02-23T00:40:54.200853+0000","last_became_active":"2024-02-23T00:40:13.179322+0000","last_became_peered":"2024-02-23T00:40:13.179322+0000","last_unstale":"2024-02-23T00:40:54.200853+0000","last_undegraded":"2024-02-23T00:40:54.200853+0000","last_fullsized":"2024-02-23T00:40:54.200853+0000","mapping_epoch":581,"log_start":"78'2","ondisk_log_start":"78'2","created":60,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:46.478470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:46.478470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":575,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200955+0000","last_change":"2024-02-23T00:40:19.126448+0000","last_active":"2024-02-23T00:40:54.200955+0000","last_peered":"2024-02-23T00:40:54.200955+0000","last_clean":"2024-02-23T00:40:54.200955+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:40:54.200955+0000","last_undegraded":"2024-02-23T00:40:54.200955+0000","last_fullsized":"2024-02-23T00:40:54.200955+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.126375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":228,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197576+0000","last_change":"2024-02-23T00:40:23.194620+0000","last_active":"2024-02-23T00:40:54.197576+0000","last_peered":"2024-02-23T00:40:54.197576+0000","last_clean":"2024-02-23T00:40:54.197576+0000","last_became_active":"2024-02-23T00:40:23.194164+0000","last_became_peered":"2024-02-23T00:40:23.194164+0000","last_unstale":"2024-02-23T00:40:54.197576+0000","last_undegraded":"2024-02-23T00:40:54.197576+0000","last_fullsized":"2024-02-23T00:40:54.197576+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:42.114807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:35:42.114807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"330'9","reported_seq":134,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193549+0000","last_change":"2024-02-23T00:38:49.222977+0000","last_active":"2024-02-23T00:40:54.193549+0000","last_peered":"2024-02-23T00:40:54.193549+0000","last_clean":"2024-02-23T00:40:54.193549+0000","last_became_active":"2024-02-23T00:38:49.216657+0000","last_became_peered":"2024-02-23T00:38:49.216657+0000","last_unstale":"2024-02-23T00:40:54.193549+0000","last_undegraded":"2024-02-23T00:40:54.193549+0000","last_fullsized":"2024-02-23T00:40:54.193549+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":669,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189868+0000","last_change":"2024-02-23T00:40:15.217300+0000","last_active":"2024-02-23T00:40:54.189868+0000","last_peered":"2024-02-23T00:40:54.189868+0000","last_clean":"2024-02-23T00:40:54.189868+0000","last_became_active":"2024-02-23T00:27:19.015374+0000","last_became_peered":"2024-02-23T00:27:19.015374+0000","last_unstale":"2024-02-23T00:40:54.189868+0000","last_undegraded":"2024-02-23T00:40:54.189868+0000","last_fullsized":"2024-02-23T00:40:54.189868+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'34","last_scrub_stamp":"2024-02-23T00:40:15.217223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.217223+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":618,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193658+0000","last_change":"2024-02-23T00:40:46.427758+0000","last_active":"2024-02-23T00:40:54.193658+0000","last_peered":"2024-02-23T00:40:54.193658+0000","last_clean":"2024-02-23T00:40:54.193658+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:40:54.193658+0000","last_undegraded":"2024-02-23T00:40:54.193658+0000","last_fullsized":"2024-02-23T00:40:54.193658+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:46.427689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:46.427689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":620,"reported_epoch":636,"state":"clean+premerge+peered","last_fresh":"2024-02-23T00:40:55.397849+0000","last_change":"2024-02-23T00:40:55.397849+0000","last_active":"2024-02-23T00:40:54.195480+0000","last_peered":"2024-02-23T00:40:55.397849+0000","last_clean":"2024-02-23T00:40:55.397849+0000","last_became_active":"2024-02-23T00:27:23.182229+0000","last_became_peered":"2024-02-23T00:40:55.397500+0000","last_unstale":"2024-02-23T00:40:55.397849+0000","last_undegraded":"2024-02-23T00:40:55.397849+0000","last_fullsized":"2024-02-23T00:40:55.397849+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:36.902015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:36.902015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":586,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199192+0000","last_change":"2024-02-23T00:40:23.198210+0000","last_active":"2024-02-23T00:40:54.199192+0000","last_peered":"2024-02-23T00:40:54.199192+0000","last_clean":"2024-02-23T00:40:54.199192+0000","last_became_active":"2024-02-23T00:40:23.197778+0000","last_became_peered":"2024-02-23T00:40:23.197778+0000","last_unstale":"2024-02-23T00:40:54.199192+0000","last_undegraded":"2024-02-23T00:40:54.199192+0000","last_fullsized":"2024-02-23T00:40:54.199192+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:22.420380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.420380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":149,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194217+0000","last_change":"2024-02-23T00:40:23.200251+0000","last_active":"2024-02-23T00:40:54.194217+0000","last_peered":"2024-02-23T00:40:54.194217+0000","last_clean":"2024-02-23T00:40:54.194217+0000","last_became_active":"2024-02-23T00:40:23.199949+0000","last_became_peered":"2024-02-23T00:40:23.199949+0000","last_unstale":"2024-02-23T00:40:54.194217+0000","last_undegraded":"2024-02-23T00:40:54.194217+0000","last_fullsized":"2024-02-23T00:40:54.194217+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:33.344870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:33.344870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":652,"reported_epoch":636,"state":"clean+premerge+peered","last_fresh":"2024-02-23T00:40:55.202063+0000","last_change":"2024-02-23T00:40:55.202063+0000","last_active":"2024-02-23T00:40:54.190122+0000","last_peered":"2024-02-23T00:40:55.202063+0000","last_clean":"2024-02-23T00:40:55.202063+0000","last_became_active":"2024-02-23T00:38:49.228425+0000","last_became_peered":"2024-02-23T00:40:55.201688+0000","last_unstale":"2024-02-23T00:40:55.202063+0000","last_undegraded":"2024-02-23T00:40:55.202063+0000","last_fullsized":"2024-02-23T00:40:55.202063+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.205739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.205739+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":171,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193451+0000","last_change":"2024-02-23T00:40:23.208576+0000","last_active":"2024-02-23T00:40:54.193451+0000","last_peered":"2024-02-23T00:40:54.193451+0000","last_clean":"2024-02-23T00:40:54.193451+0000","last_became_active":"2024-02-23T00:40:23.208225+0000","last_became_peered":"2024-02-23T00:40:23.208225+0000","last_unstale":"2024-02-23T00:40:54.193451+0000","last_undegraded":"2024-02-23T00:40:54.193451+0000","last_fullsized":"2024-02-23T00:40:54.193451+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:39:28.324185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.324185+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":605,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197723+0000","last_change":"2024-02-23T00:38:14.343844+0000","last_active":"2024-02-23T00:40:54.197723+0000","last_peered":"2024-02-23T00:40:54.197723+0000","last_clean":"2024-02-23T00:40:54.197723+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:40:54.197723+0000","last_undegraded":"2024-02-23T00:40:54.197723+0000","last_fullsized":"2024-02-23T00:40:54.197723+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:14.343770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:14.343770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":626,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189293+0000","last_change":"2024-02-23T00:38:49.225300+0000","last_active":"2024-02-23T00:40:54.189293+0000","last_peered":"2024-02-23T00:40:54.189293+0000","last_clean":"2024-02-23T00:40:54.189293+0000","last_became_active":"2024-02-23T00:38:49.224892+0000","last_became_peered":"2024-02-23T00:38:49.224892+0000","last_unstale":"2024-02-23T00:40:54.189293+0000","last_undegraded":"2024-02-23T00:40:54.189293+0000","last_fullsized":"2024-02-23T00:40:54.189293+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.082000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.082000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":134,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194224+0000","last_change":"2024-02-23T00:38:49.176838+0000","last_active":"2024-02-23T00:40:54.194224+0000","last_peered":"2024-02-23T00:40:54.194224+0000","last_clean":"2024-02-23T00:40:54.194224+0000","last_became_active":"2024-02-23T00:38:49.175801+0000","last_became_peered":"2024-02-23T00:38:49.175801+0000","last_unstale":"2024-02-23T00:40:54.194224+0000","last_undegraded":"2024-02-23T00:40:54.194224+0000","last_fullsized":"2024-02-23T00:40:54.194224+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:00.977282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:00.977282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":873,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200421+0000","last_change":"2024-02-23T00:40:17.043400+0000","last_active":"2024-02-23T00:40:54.200421+0000","last_peered":"2024-02-23T00:40:54.200421+0000","last_clean":"2024-02-23T00:40:54.200421+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:40:54.200421+0000","last_undegraded":"2024-02-23T00:40:54.200421+0000","last_fullsized":"2024-02-23T00:40:54.200421+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:17.043322+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.043322+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":652,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193381+0000","last_change":"2024-02-23T00:40:53.234078+0000","last_active":"2024-02-23T00:40:54.193381+0000","last_peered":"2024-02-23T00:40:54.193381+0000","last_clean":"2024-02-23T00:40:54.193381+0000","last_became_active":"2024-02-23T00:40:53.233711+0000","last_became_peered":"2024-02-23T00:40:53.233711+0000","last_unstale":"2024-02-23T00:40:54.193381+0000","last_undegraded":"2024-02-23T00:40:54.193381+0000","last_fullsized":"2024-02-23T00:40:54.193381+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_clean_scrub_stamp":"2024-02-23T00:40:25.452086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":330,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200344+0000","last_change":"2024-02-23T00:40:50.813789+0000","last_active":"2024-02-23T00:40:54.200344+0000","last_peered":"2024-02-23T00:40:54.200344+0000","last_clean":"2024-02-23T00:40:54.200344+0000","last_became_active":"2024-02-23T00:40:23.194343+0000","last_became_peered":"2024-02-23T00:40:23.194343+0000","last_unstale":"2024-02-23T00:40:54.200344+0000","last_undegraded":"2024-02-23T00:40:54.200344+0000","last_fullsized":"2024-02-23T00:40:54.200344+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'35","last_scrub_stamp":"2024-02-23T00:40:50.813709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.813709+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":622,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194680+0000","last_change":"2024-02-23T00:40:23.196111+0000","last_active":"2024-02-23T00:40:54.194680+0000","last_peered":"2024-02-23T00:40:54.194680+0000","last_clean":"2024-02-23T00:40:54.194680+0000","last_became_active":"2024-02-23T00:40:23.195814+0000","last_became_peered":"2024-02-23T00:40:23.195814+0000","last_unstale":"2024-02-23T00:40:54.194680+0000","last_undegraded":"2024-02-23T00:40:54.194680+0000","last_fullsized":"2024-02-23T00:40:54.194680+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:12.601813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:34:12.601813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":599,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198811+0000","last_change":"2024-02-23T00:37:26.268389+0000","last_active":"2024-02-23T00:40:54.198811+0000","last_peered":"2024-02-23T00:40:54.198811+0000","last_clean":"2024-02-23T00:40:54.198811+0000","last_became_active":"2024-02-23T00:27:23.181003+0000","last_became_peered":"2024-02-23T00:27:23.181003+0000","last_unstale":"2024-02-23T00:40:54.198811+0000","last_undegraded":"2024-02-23T00:40:54.198811+0000","last_fullsized":"2024-02-23T00:40:54.198811+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:37:26.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:37:26.268282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":552,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198901+0000","last_change":"2024-02-23T00:39:38.350980+0000","last_active":"2024-02-23T00:40:54.198901+0000","last_peered":"2024-02-23T00:40:54.198901+0000","last_clean":"2024-02-23T00:40:54.198901+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:40:54.198901+0000","last_undegraded":"2024-02-23T00:40:54.198901+0000","last_fullsized":"2024-02-23T00:40:54.198901+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:38.350890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:38.350890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":635,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200994+0000","last_change":"2024-02-23T00:40:33.192004+0000","last_active":"2024-02-23T00:40:54.200994+0000","last_peered":"2024-02-23T00:40:54.200994+0000","last_clean":"2024-02-23T00:40:54.200994+0000","last_became_active":"2024-02-23T00:40:33.191698+0000","last_became_peered":"2024-02-23T00:40:33.191698+0000","last_unstale":"2024-02-23T00:40:54.200994+0000","last_undegraded":"2024-02-23T00:40:54.200994+0000","last_fullsized":"2024-02-23T00:40:54.200994+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":614,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:12.386288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:32:12.386288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":315,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197804+0000","last_change":"2024-02-23T00:40:26.193541+0000","last_active":"2024-02-23T00:40:54.197804+0000","last_peered":"2024-02-23T00:40:54.197804+0000","last_clean":"2024-02-23T00:40:54.197804+0000","last_became_active":"2024-02-23T00:40:26.193180+0000","last_became_peered":"2024-02-23T00:40:26.193180+0000","last_unstale":"2024-02-23T00:40:54.197804+0000","last_undegraded":"2024-02-23T00:40:54.197804+0000","last_fullsized":"2024-02-23T00:40:54.197804+0000","mapping_epoch":606,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":622,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194963+0000","last_change":"2024-02-23T00:40:23.196965+0000","last_active":"2024-02-23T00:40:54.194963+0000","last_peered":"2024-02-23T00:40:54.194963+0000","last_clean":"2024-02-23T00:40:54.194963+0000","last_became_active":"2024-02-23T00:40:23.196560+0000","last_became_peered":"2024-02-23T00:40:23.196560+0000","last_unstale":"2024-02-23T00:40:54.194963+0000","last_undegraded":"2024-02-23T00:40:54.194963+0000","last_fullsized":"2024-02-23T00:40:54.194963+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:29.980396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:36:29.980396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":559,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189432+0000","last_change":"2024-02-23T00:40:23.195425+0000","last_active":"2024-02-23T00:40:54.189432+0000","last_peered":"2024-02-23T00:40:54.189432+0000","last_clean":"2024-02-23T00:40:54.189432+0000","last_became_active":"2024-02-23T00:40:23.195137+0000","last_became_peered":"2024-02-23T00:40:23.195137+0000","last_unstale":"2024-02-23T00:40:54.189432+0000","last_undegraded":"2024-02-23T00:40:54.189432+0000","last_fullsized":"2024-02-23T00:40:54.189432+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:24.424938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:24.424938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":635,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200755+0000","last_change":"2024-02-23T00:40:53.233358+0000","last_active":"2024-02-23T00:40:54.200755+0000","last_peered":"2024-02-23T00:40:54.200755+0000","last_clean":"2024-02-23T00:40:54.200755+0000","last_became_active":"2024-02-23T00:40:53.233065+0000","last_became_peered":"2024-02-23T00:40:53.233065+0000","last_unstale":"2024-02-23T00:40:54.200755+0000","last_undegraded":"2024-02-23T00:40:54.200755+0000","last_fullsized":"2024-02-23T00:40:54.200755+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":634,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.315765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.315765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":561,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189561+0000","last_change":"2024-02-23T00:39:28.509774+0000","last_active":"2024-02-23T00:40:54.189561+0000","last_peered":"2024-02-23T00:40:54.189561+0000","last_clean":"2024-02-23T00:40:54.189561+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:40:54.189561+0000","last_undegraded":"2024-02-23T00:40:54.189561+0000","last_fullsized":"2024-02-23T00:40:54.189561+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:28.509675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.509675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":640,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193770+0000","last_change":"2024-02-23T00:40:45.191842+0000","last_active":"2024-02-23T00:40:54.193770+0000","last_peered":"2024-02-23T00:40:54.193770+0000","last_clean":"2024-02-23T00:40:54.193770+0000","last_became_active":"2024-02-23T00:40:45.191453+0000","last_became_peered":"2024-02-23T00:40:45.191453+0000","last_unstale":"2024-02-23T00:40:54.193770+0000","last_undegraded":"2024-02-23T00:40:54.193770+0000","last_fullsized":"2024-02-23T00:40:54.193770+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:39:39.963439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":657,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192581+0000","last_change":"2024-02-23T00:40:11.261185+0000","last_active":"2024-02-23T00:40:54.192581+0000","last_peered":"2024-02-23T00:40:54.192581+0000","last_clean":"2024-02-23T00:40:54.192581+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:40:54.192581+0000","last_undegraded":"2024-02-23T00:40:54.192581+0000","last_fullsized":"2024-02-23T00:40:54.192581+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:40:11.261102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.261102+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":630,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.195536+0000","last_change":"2024-02-23T00:40:46.771540+0000","last_active":"2024-02-23T00:40:54.195536+0000","last_peered":"2024-02-23T00:40:54.195536+0000","last_clean":"2024-02-23T00:40:54.195536+0000","last_became_active":"2024-02-23T00:40:23.198114+0000","last_became_peered":"2024-02-23T00:40:23.198114+0000","last_unstale":"2024-02-23T00:40:54.195536+0000","last_undegraded":"2024-02-23T00:40:54.195536+0000","last_fullsized":"2024-02-23T00:40:54.195536+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:46.771465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:40:46.771465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":607,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200443+0000","last_change":"2024-02-23T00:39:40.690606+0000","last_active":"2024-02-23T00:40:54.200443+0000","last_peered":"2024-02-23T00:40:54.200443+0000","last_clean":"2024-02-23T00:40:54.200443+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:40:54.200443+0000","last_undegraded":"2024-02-23T00:40:54.200443+0000","last_fullsized":"2024-02-23T00:40:54.200443+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.690534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.690534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":139,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194287+0000","last_change":"2024-02-23T00:40:52.474608+0000","last_active":"2024-02-23T00:40:54.194287+0000","last_peered":"2024-02-23T00:40:54.194287+0000","last_clean":"2024-02-23T00:40:54.194287+0000","last_became_active":"2024-02-23T00:38:49.226188+0000","last_became_peered":"2024-02-23T00:38:49.226188+0000","last_unstale":"2024-02-23T00:40:54.194287+0000","last_undegraded":"2024-02-23T00:40:54.194287+0000","last_fullsized":"2024-02-23T00:40:54.194287+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.474495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.474495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":637,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.201046+0000","last_change":"2024-02-23T00:40:41.298338+0000","last_active":"2024-02-23T00:40:54.201046+0000","last_peered":"2024-02-23T00:40:54.201046+0000","last_clean":"2024-02-23T00:40:54.201046+0000","last_became_active":"2024-02-23T00:40:41.297993+0000","last_became_peered":"2024-02-23T00:40:41.297993+0000","last_unstale":"2024-02-23T00:40:54.201046+0000","last_undegraded":"2024-02-23T00:40:54.201046+0000","last_fullsized":"2024-02-23T00:40:54.201046+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:20.033055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:40:20.033055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":563,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189436+0000","last_change":"2024-02-23T00:38:44.173181+0000","last_active":"2024-02-23T00:40:54.189436+0000","last_peered":"2024-02-23T00:40:54.189436+0000","last_clean":"2024-02-23T00:40:54.189436+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:40:54.189436+0000","last_undegraded":"2024-02-23T00:40:54.189436+0000","last_fullsized":"2024-02-23T00:40:54.189436+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:44.173077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.173077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":648,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199563+0000","last_change":"2024-02-23T00:40:38.061984+0000","last_active":"2024-02-23T00:40:54.199563+0000","last_peered":"2024-02-23T00:40:54.199563+0000","last_clean":"2024-02-23T00:40:54.199563+0000","last_became_active":"2024-02-23T00:40:38.061696+0000","last_became_peered":"2024-02-23T00:40:38.061696+0000","last_unstale":"2024-02-23T00:40:54.199563+0000","last_undegraded":"2024-02-23T00:40:54.199563+0000","last_fullsized":"2024-02-23T00:40:54.199563+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.413877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.413877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":724,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199826+0000","last_change":"2024-02-23T00:40:26.191175+0000","last_active":"2024-02-23T00:40:54.199826+0000","last_peered":"2024-02-23T00:40:54.199826+0000","last_clean":"2024-02-23T00:40:54.199826+0000","last_became_active":"2024-02-23T00:40:26.190861+0000","last_became_peered":"2024-02-23T00:40:26.190861+0000","last_unstale":"2024-02-23T00:40:54.199826+0000","last_undegraded":"2024-02-23T00:40:54.199826+0000","last_fullsized":"2024-02-23T00:40:54.199826+0000","mapping_epoch":606,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":607,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":654,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.201061+0000","last_change":"2024-02-23T00:40:23.196884+0000","last_active":"2024-02-23T00:40:54.201061+0000","last_peered":"2024-02-23T00:40:54.201061+0000","last_clean":"2024-02-23T00:40:54.201061+0000","last_became_active":"2024-02-23T00:40:23.196485+0000","last_became_peered":"2024-02-23T00:40:23.196485+0000","last_unstale":"2024-02-23T00:40:54.201061+0000","last_undegraded":"2024-02-23T00:40:54.201061+0000","last_fullsized":"2024-02-23T00:40:54.201061+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:22.437104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:22.437104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":866,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197614+0000","last_change":"2024-02-23T00:40:12.984115+0000","last_active":"2024-02-23T00:40:54.197614+0000","last_peered":"2024-02-23T00:40:54.197614+0000","last_clean":"2024-02-23T00:40:54.197614+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:40:54.197614+0000","last_undegraded":"2024-02-23T00:40:54.197614+0000","last_fullsized":"2024-02-23T00:40:54.197614+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:12.984040+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.984040+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":647,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194586+0000","last_change":"2024-02-23T00:40:49.298781+0000","last_active":"2024-02-23T00:40:54.194586+0000","last_peered":"2024-02-23T00:40:54.194586+0000","last_clean":"2024-02-23T00:40:54.194586+0000","last_became_active":"2024-02-23T00:40:49.298465+0000","last_became_peered":"2024-02-23T00:40:49.298465+0000","last_unstale":"2024-02-23T00:40:54.194586+0000","last_undegraded":"2024-02-23T00:40:54.194586+0000","last_fullsized":"2024-02-23T00:40:54.194586+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:20.642339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:20.642339+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":567,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200242+0000","last_change":"2024-02-23T00:38:40.265472+0000","last_active":"2024-02-23T00:40:54.200242+0000","last_peered":"2024-02-23T00:40:54.200242+0000","last_clean":"2024-02-23T00:40:54.200242+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:40:54.200242+0000","last_undegraded":"2024-02-23T00:40:54.200242+0000","last_fullsized":"2024-02-23T00:40:54.200242+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:40.265385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.265385+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":638,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189140+0000","last_change":"2024-02-23T00:40:43.522958+0000","last_active":"2024-02-23T00:40:54.189140+0000","last_peered":"2024-02-23T00:40:54.189140+0000","last_clean":"2024-02-23T00:40:54.189140+0000","last_became_active":"2024-02-23T00:40:23.203665+0000","last_became_peered":"2024-02-23T00:40:23.203665+0000","last_unstale":"2024-02-23T00:40:54.189140+0000","last_undegraded":"2024-02-23T00:40:54.189140+0000","last_fullsized":"2024-02-23T00:40:54.189140+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:43.522859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:43.522859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":591,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198164+0000","last_change":"2024-02-23T00:40:41.493939+0000","last_active":"2024-02-23T00:40:54.198164+0000","last_peered":"2024-02-23T00:40:54.198164+0000","last_clean":"2024-02-23T00:40:54.198164+0000","last_became_active":"2024-02-23T00:40:41.493640+0000","last_became_peered":"2024-02-23T00:40:41.493640+0000","last_unstale":"2024-02-23T00:40:54.198164+0000","last_undegraded":"2024-02-23T00:40:54.198164+0000","last_fullsized":"2024-02-23T00:40:54.198164+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":622,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:32:50.596856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:32:50.596856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":919,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-23T00:40:55.398313+0000","last_change":"2024-02-23T00:40:15.043521+0000","last_active":"2024-02-23T00:40:55.398313+0000","last_peered":"2024-02-23T00:40:55.398313+0000","last_clean":"2024-02-23T00:40:55.398313+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:40:55.398313+0000","last_undegraded":"2024-02-23T00:40:55.398313+0000","last_fullsized":"2024-02-23T00:40:55.398313+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'48","last_scrub_stamp":"2024-02-23T00:40:15.043419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.043419+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"601'2","reported_seq":249,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198614+0000","last_change":"2024-02-23T00:40:21.186280+0000","last_active":"2024-02-23T00:40:54.198614+0000","last_peered":"2024-02-23T00:40:54.198614+0000","last_clean":"2024-02-23T00:40:54.198614+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:40:54.198614+0000","last_undegraded":"2024-02-23T00:40:54.198614+0000","last_fullsized":"2024-02-23T00:40:54.198614+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'2","last_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_deep_scrub":"601'2","last_deep_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_clean_scrub_stamp":"2024-02-23T00:40:21.186215+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":157,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194827+0000","last_change":"2024-02-23T00:40:45.365100+0000","last_active":"2024-02-23T00:40:54.194827+0000","last_peered":"2024-02-23T00:40:54.194827+0000","last_clean":"2024-02-23T00:40:54.194827+0000","last_became_active":"2024-02-23T00:40:45.364819+0000","last_became_peered":"2024-02-23T00:40:45.364819+0000","last_unstale":"2024-02-23T00:40:54.194827+0000","last_undegraded":"2024-02-23T00:40:54.194827+0000","last_fullsized":"2024-02-23T00:40:54.194827+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.370090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.370090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":572,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200612+0000","last_change":"2024-02-23T00:40:22.203106+0000","last_active":"2024-02-23T00:40:54.200612+0000","last_peered":"2024-02-23T00:40:54.200612+0000","last_clean":"2024-02-23T00:40:54.200612+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:40:54.200612+0000","last_undegraded":"2024-02-23T00:40:54.200612+0000","last_fullsized":"2024-02-23T00:40:54.200612+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:22.203041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:22.203041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":535,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200200+0000","last_change":"2024-02-23T00:39:32.809209+0000","last_active":"2024-02-23T00:40:54.200200+0000","last_peered":"2024-02-23T00:40:54.200200+0000","last_clean":"2024-02-23T00:40:54.200200+0000","last_became_active":"2024-02-23T00:38:49.183777+0000","last_became_peered":"2024-02-23T00:38:49.183777+0000","last_unstale":"2024-02-23T00:40:54.200200+0000","last_undegraded":"2024-02-23T00:40:54.200200+0000","last_fullsized":"2024-02-23T00:40:54.200200+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:32.809133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:32.809133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"631'19","reported_seq":268,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-23T00:40:55.398349+0000","last_change":"2024-02-23T00:40:49.301418+0000","last_active":"2024-02-23T00:40:55.398349+0000","last_peered":"2024-02-23T00:40:55.398349+0000","last_clean":"2024-02-23T00:40:55.398349+0000","last_became_active":"2024-02-23T00:40:49.300985+0000","last_became_peered":"2024-02-23T00:40:49.300985+0000","last_unstale":"2024-02-23T00:40:55.398349+0000","last_undegraded":"2024-02-23T00:40:55.398349+0000","last_fullsized":"2024-02-23T00:40:55.398349+0000","mapping_epoch":603,"log_start":"606'17","ondisk_log_start":"606'17","created":58,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:33:46.629202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:33:46.629202+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":61,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198479+0000","last_change":"2024-02-23T00:40:53.236854+0000","last_active":"2024-02-23T00:40:54.198479+0000","last_peered":"2024-02-23T00:40:54.198479+0000","last_clean":"2024-02-23T00:40:54.198479+0000","last_became_active":"2024-02-23T00:40:23.194277+0000","last_became_peered":"2024-02-23T00:40:23.194277+0000","last_unstale":"2024-02-23T00:40:54.198479+0000","last_undegraded":"2024-02-23T00:40:54.198479+0000","last_fullsized":"2024-02-23T00:40:54.198479+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:40:53.236791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.236791+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":623,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200471+0000","last_change":"2024-02-23T00:40:52.208850+0000","last_active":"2024-02-23T00:40:54.200471+0000","last_peered":"2024-02-23T00:40:54.200471+0000","last_clean":"2024-02-23T00:40:54.200471+0000","last_became_active":"2024-02-23T00:40:23.198153+0000","last_became_peered":"2024-02-23T00:40:23.198153+0000","last_unstale":"2024-02-23T00:40:54.200471+0000","last_undegraded":"2024-02-23T00:40:54.200471+0000","last_fullsized":"2024-02-23T00:40:54.200471+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.208784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.208784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":149,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193254+0000","last_change":"2024-02-23T00:40:23.200282+0000","last_active":"2024-02-23T00:40:54.193254+0000","last_peered":"2024-02-23T00:40:54.193254+0000","last_clean":"2024-02-23T00:40:54.193254+0000","last_became_active":"2024-02-23T00:40:23.199290+0000","last_became_peered":"2024-02-23T00:40:23.199290+0000","last_unstale":"2024-02-23T00:40:54.193254+0000","last_undegraded":"2024-02-23T00:40:54.193254+0000","last_fullsized":"2024-02-23T00:40:54.193254+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:21.370914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:39:21.370914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":561,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.188843+0000","last_change":"2024-02-23T00:39:30.473677+0000","last_active":"2024-02-23T00:40:54.188843+0000","last_peered":"2024-02-23T00:40:54.188843+0000","last_clean":"2024-02-23T00:40:54.188843+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:40:54.188843+0000","last_undegraded":"2024-02-23T00:40:54.188843+0000","last_fullsized":"2024-02-23T00:40:54.188843+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.473602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.473602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":648,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197688+0000","last_change":"2024-02-23T00:40:29.192688+0000","last_active":"2024-02-23T00:40:54.197688+0000","last_peered":"2024-02-23T00:40:54.197688+0000","last_clean":"2024-02-23T00:40:54.197688+0000","last_became_active":"2024-02-23T00:40:29.192428+0000","last_became_peered":"2024-02-23T00:40:29.192428+0000","last_unstale":"2024-02-23T00:40:54.197688+0000","last_undegraded":"2024-02-23T00:40:54.197688+0000","last_fullsized":"2024-02-23T00:40:54.197688+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":610,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:36:24.016837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:36:24.016837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":835,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189885+0000","last_change":"2024-02-23T00:40:50.676459+0000","last_active":"2024-02-23T00:40:54.189885+0000","last_peered":"2024-02-23T00:40:54.189885+0000","last_clean":"2024-02-23T00:40:54.189885+0000","last_became_active":"2024-02-23T00:38:54.171563+0000","last_became_peered":"2024-02-23T00:38:54.171563+0000","last_unstale":"2024-02-23T00:40:54.189885+0000","last_undegraded":"2024-02-23T00:40:54.189885+0000","last_fullsized":"2024-02-23T00:40:54.189885+0000","mapping_epoch":516,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:40:50.676380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.676380+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":648,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189377+0000","last_change":"2024-02-23T00:40:45.582021+0000","last_active":"2024-02-23T00:40:54.189377+0000","last_peered":"2024-02-23T00:40:54.189377+0000","last_clean":"2024-02-23T00:40:54.189377+0000","last_became_active":"2024-02-23T00:40:23.191879+0000","last_became_peered":"2024-02-23T00:40:23.191879+0000","last_unstale":"2024-02-23T00:40:54.189377+0000","last_undegraded":"2024-02-23T00:40:54.189377+0000","last_fullsized":"2024-02-23T00:40:54.189377+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:40:45.581901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.581901+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":152,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193820+0000","last_change":"2024-02-23T00:40:17.214527+0000","last_active":"2024-02-23T00:40:54.193820+0000","last_peered":"2024-02-23T00:40:54.193820+0000","last_clean":"2024-02-23T00:40:54.193820+0000","last_became_active":"2024-02-23T00:40:17.214288+0000","last_became_peered":"2024-02-23T00:40:17.214288+0000","last_unstale":"2024-02-23T00:40:54.193820+0000","last_undegraded":"2024-02-23T00:40:54.193820+0000","last_fullsized":"2024-02-23T00:40:54.193820+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:10.172059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:38:10.172059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":556,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200072+0000","last_change":"2024-02-23T00:40:05.936821+0000","last_active":"2024-02-23T00:40:54.200072+0000","last_peered":"2024-02-23T00:40:54.200072+0000","last_clean":"2024-02-23T00:40:54.200072+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:40:54.200072+0000","last_undegraded":"2024-02-23T00:40:54.200072+0000","last_fullsized":"2024-02-23T00:40:54.200072+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:05.936761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:05.936761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":342,"reported_epoch":635,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:40:54.193273+0000","last_change":"2024-02-23T00:40:53.213834+0000","last_active":"2024-02-23T00:40:54.193273+0000","last_peered":"2024-02-23T00:40:54.193273+0000","last_clean":"2024-02-23T00:40:21.193234+0000","last_became_active":"2024-02-23T00:40:23.202555+0000","last_became_peered":"2024-02-23T00:40:23.202555+0000","last_unstale":"2024-02-23T00:40:54.193273+0000","last_undegraded":"2024-02-23T00:40:54.193273+0000","last_fullsized":"2024-02-23T00:40:54.193273+0000","mapping_epoch":603,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":517,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'12","last_scrub_stamp":"2024-02-23T00:34:21.576290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:34:21.576290+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":630,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197593+0000","last_change":"2024-02-23T00:40:51.169237+0000","last_active":"2024-02-23T00:40:54.197593+0000","last_peered":"2024-02-23T00:40:54.197593+0000","last_clean":"2024-02-23T00:40:54.197593+0000","last_became_active":"2024-02-23T00:40:23.196558+0000","last_became_peered":"2024-02-23T00:40:23.196558+0000","last_unstale":"2024-02-23T00:40:54.197593+0000","last_undegraded":"2024-02-23T00:40:54.197593+0000","last_fullsized":"2024-02-23T00:40:54.197593+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:51.169147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:51.169147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":139,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194306+0000","last_change":"2024-02-23T00:39:18.369649+0000","last_active":"2024-02-23T00:40:54.194306+0000","last_peered":"2024-02-23T00:40:54.194306+0000","last_clean":"2024-02-23T00:40:54.194306+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:40:54.194306+0000","last_undegraded":"2024-02-23T00:40:54.194306+0000","last_fullsized":"2024-02-23T00:40:54.194306+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:18.369579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:18.369579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":134,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.195535+0000","last_change":"2024-02-23T00:38:49.230851+0000","last_active":"2024-02-23T00:40:54.195535+0000","last_peered":"2024-02-23T00:40:54.195535+0000","last_clean":"2024-02-23T00:40:54.195535+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:40:54.195535+0000","last_undegraded":"2024-02-23T00:40:54.195535+0000","last_fullsized":"2024-02-23T00:40:54.195535+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:39.893315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:39.893315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":747,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193550+0000","last_change":"2024-02-23T00:40:51.316847+0000","last_active":"2024-02-23T00:40:54.193550+0000","last_peered":"2024-02-23T00:40:54.193550+0000","last_clean":"2024-02-23T00:40:54.193550+0000","last_became_active":"2024-02-23T00:40:51.316452+0000","last_became_peered":"2024-02-23T00:40:51.316452+0000","last_unstale":"2024-02-23T00:40:54.193550+0000","last_undegraded":"2024-02-23T00:40:54.193550+0000","last_fullsized":"2024-02-23T00:40:54.193550+0000","mapping_epoch":631,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":619,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192401+0000","last_change":"2024-02-23T00:40:48.419187+0000","last_active":"2024-02-23T00:40:54.192401+0000","last_peered":"2024-02-23T00:40:54.192401+0000","last_clean":"2024-02-23T00:40:54.192401+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:40:54.192401+0000","last_undegraded":"2024-02-23T00:40:54.192401+0000","last_fullsized":"2024-02-23T00:40:54.192401+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:48.419098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:40:48.419098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":616,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200675+0000","last_change":"2024-02-23T00:39:28.781689+0000","last_active":"2024-02-23T00:40:54.200675+0000","last_peered":"2024-02-23T00:40:54.200675+0000","last_clean":"2024-02-23T00:40:54.200675+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:40:54.200675+0000","last_undegraded":"2024-02-23T00:40:54.200675+0000","last_fullsized":"2024-02-23T00:40:54.200675+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:28.781587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.781587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":579,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192759+0000","last_change":"2024-02-23T00:39:29.410514+0000","last_active":"2024-02-23T00:40:54.192759+0000","last_peered":"2024-02-23T00:40:54.192759+0000","last_clean":"2024-02-23T00:40:54.192759+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:40:54.192759+0000","last_undegraded":"2024-02-23T00:40:54.192759+0000","last_fullsized":"2024-02-23T00:40:54.192759+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:29.410456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:29.410456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"453'32","reported_seq":150,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.195285+0000","last_change":"2024-02-23T00:38:51.793947+0000","last_active":"2024-02-23T00:40:54.195285+0000","last_peered":"2024-02-23T00:40:54.195285+0000","last_clean":"2024-02-23T00:40:54.195285+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:40:54.195285+0000","last_undegraded":"2024-02-23T00:40:54.195285+0000","last_fullsized":"2024-02-23T00:40:54.195285+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:38:44.240437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:44.240437+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":633,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200080+0000","last_change":"2024-02-23T00:39:23.833122+0000","last_active":"2024-02-23T00:40:54.200080+0000","last_peered":"2024-02-23T00:40:54.200080+0000","last_clean":"2024-02-23T00:40:54.200080+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:40:54.200080+0000","last_undegraded":"2024-02-23T00:40:54.200080+0000","last_fullsized":"2024-02-23T00:40:54.200080+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:23.833059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:39:23.833059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"606'11","reported_seq":791,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-23T00:40:55.199953+0000","last_change":"2024-02-23T00:40:45.004502+0000","last_active":"2024-02-23T00:40:55.199953+0000","last_peered":"2024-02-23T00:40:55.199953+0000","last_clean":"2024-02-23T00:40:55.199953+0000","last_became_active":"2024-02-23T00:40:23.202761+0000","last_became_peered":"2024-02-23T00:40:23.202761+0000","last_unstale":"2024-02-23T00:40:55.199953+0000","last_undegraded":"2024-02-23T00:40:55.199953+0000","last_fullsized":"2024-02-23T00:40:55.199953+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"606'11","last_scrub_stamp":"2024-02-23T00:40:45.004428+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.004428+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":556,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.201003+0000","last_change":"2024-02-23T00:40:17.978624+0000","last_active":"2024-02-23T00:40:54.201003+0000","last_peered":"2024-02-23T00:40:54.201003+0000","last_clean":"2024-02-23T00:40:54.201003+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:40:54.201003+0000","last_undegraded":"2024-02-23T00:40:54.201003+0000","last_fullsized":"2024-02-23T00:40:54.201003+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:17.978560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.978560+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":687,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200236+0000","last_change":"2024-02-23T00:38:49.346222+0000","last_active":"2024-02-23T00:40:54.200236+0000","last_peered":"2024-02-23T00:40:54.200236+0000","last_clean":"2024-02-23T00:40:54.200236+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:40:54.200236+0000","last_undegraded":"2024-02-23T00:40:54.200236+0000","last_fullsized":"2024-02-23T00:40:54.200236+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:38:40.858800+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:38:40.858800+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":546,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189524+0000","last_change":"2024-02-23T00:40:23.196005+0000","last_active":"2024-02-23T00:40:54.189524+0000","last_peered":"2024-02-23T00:40:54.189524+0000","last_clean":"2024-02-23T00:40:54.189524+0000","last_became_active":"2024-02-23T00:40:23.195711+0000","last_became_peered":"2024-02-23T00:40:23.195711+0000","last_unstale":"2024-02-23T00:40:54.189524+0000","last_undegraded":"2024-02-23T00:40:54.189524+0000","last_fullsized":"2024-02-23T00:40:54.189524+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:26.483929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:26.483929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":169,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-23T00:40:55.193094+0000","last_change":"2024-02-23T00:40:50.453642+0000","last_active":"2024-02-23T00:40:55.193094+0000","last_peered":"2024-02-23T00:40:55.193094+0000","last_clean":"2024-02-23T00:40:55.193094+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:40:55.193094+0000","last_undegraded":"2024-02-23T00:40:55.193094+0000","last_fullsized":"2024-02-23T00:40:55.193094+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'11","last_scrub_stamp":"2024-02-23T00:40:50.453579+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.453579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":134,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194102+0000","last_change":"2024-02-23T00:38:49.227700+0000","last_active":"2024-02-23T00:40:54.194102+0000","last_peered":"2024-02-23T00:40:54.194102+0000","last_clean":"2024-02-23T00:40:54.194102+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:40:54.194102+0000","last_undegraded":"2024-02-23T00:40:54.194102+0000","last_fullsized":"2024-02-23T00:40:54.194102+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:09.447326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:09.447326+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":628,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198879+0000","last_change":"2024-02-23T00:37:20.382193+0000","last_active":"2024-02-23T00:40:54.198879+0000","last_peered":"2024-02-23T00:40:54.198879+0000","last_clean":"2024-02-23T00:40:54.198879+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:40:54.198879+0000","last_undegraded":"2024-02-23T00:40:54.198879+0000","last_fullsized":"2024-02-23T00:40:54.198879+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'9","last_scrub_stamp":"2024-02-23T00:37:20.382108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:37:20.382108+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":637,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192304+0000","last_change":"2024-02-23T00:40:08.274788+0000","last_active":"2024-02-23T00:40:54.192304+0000","last_peered":"2024-02-23T00:40:54.192304+0000","last_clean":"2024-02-23T00:40:54.192304+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:40:54.192304+0000","last_undegraded":"2024-02-23T00:40:54.192304+0000","last_fullsized":"2024-02-23T00:40:54.192304+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:08.274721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:08.274721+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":149,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193695+0000","last_change":"2024-02-23T00:40:23.203466+0000","last_active":"2024-02-23T00:40:54.193695+0000","last_peered":"2024-02-23T00:40:54.193695+0000","last_clean":"2024-02-23T00:40:54.193695+0000","last_became_active":"2024-02-23T00:40:23.203017+0000","last_became_peered":"2024-02-23T00:40:23.203017+0000","last_unstale":"2024-02-23T00:40:54.193695+0000","last_undegraded":"2024-02-23T00:40:54.193695+0000","last_fullsized":"2024-02-23T00:40:54.193695+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.370620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.370620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":80,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194957+0000","last_change":"2024-02-23T00:40:24.195853+0000","last_active":"2024-02-23T00:40:54.194957+0000","last_peered":"2024-02-23T00:40:54.194957+0000","last_clean":"2024-02-23T00:40:54.194957+0000","last_became_active":"2024-02-23T00:40:24.195542+0000","last_became_peered":"2024-02-23T00:40:24.195542+0000","last_unstale":"2024-02-23T00:40:54.194957+0000","last_undegraded":"2024-02-23T00:40:54.194957+0000","last_fullsized":"2024-02-23T00:40:54.194957+0000","mapping_epoch":604,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:06.292724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:06.292724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":576,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.188736+0000","last_change":"2024-02-23T00:40:46.621846+0000","last_active":"2024-02-23T00:40:54.188736+0000","last_peered":"2024-02-23T00:40:54.188736+0000","last_clean":"2024-02-23T00:40:54.188736+0000","last_became_active":"2024-02-23T00:40:23.186760+0000","last_became_peered":"2024-02-23T00:40:23.186760+0000","last_unstale":"2024-02-23T00:40:54.188736+0000","last_undegraded":"2024-02-23T00:40:54.188736+0000","last_fullsized":"2024-02-23T00:40:54.188736+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:46.621733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:46.621733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":664,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189410+0000","last_change":"2024-02-23T00:38:16.174640+0000","last_active":"2024-02-23T00:40:54.189410+0000","last_peered":"2024-02-23T00:40:54.189410+0000","last_clean":"2024-02-23T00:40:54.189410+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:40:54.189410+0000","last_undegraded":"2024-02-23T00:40:54.189410+0000","last_fullsized":"2024-02-23T00:40:54.189410+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:38:16.174526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.174526+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":615,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198218+0000","last_change":"2024-02-23T00:40:23.201115+0000","last_active":"2024-02-23T00:40:54.198218+0000","last_peered":"2024-02-23T00:40:54.198218+0000","last_clean":"2024-02-23T00:40:54.198218+0000","last_became_active":"2024-02-23T00:40:23.200826+0000","last_became_peered":"2024-02-23T00:40:23.200826+0000","last_unstale":"2024-02-23T00:40:54.198218+0000","last_undegraded":"2024-02-23T00:40:54.198218+0000","last_fullsized":"2024-02-23T00:40:54.198218+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:51.783089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:35:51.783089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":619,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198967+0000","last_change":"2024-02-23T00:38:49.225280+0000","last_active":"2024-02-23T00:40:54.198967+0000","last_peered":"2024-02-23T00:40:54.198967+0000","last_clean":"2024-02-23T00:40:54.198967+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:40:54.198967+0000","last_undegraded":"2024-02-23T00:40:54.198967+0000","last_fullsized":"2024-02-23T00:40:54.198967+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:06.235608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:38:06.235608+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":63,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.196607+0000","last_change":"2024-02-23T00:40:02.301118+0000","last_active":"2024-02-23T00:40:54.196607+0000","last_peered":"2024-02-23T00:40:54.196607+0000","last_clean":"2024-02-23T00:40:54.196607+0000","last_became_active":"2024-02-23T00:40:02.300737+0000","last_became_peered":"2024-02-23T00:40:02.300737+0000","last_unstale":"2024-02-23T00:40:54.196607+0000","last_undegraded":"2024-02-23T00:40:54.196607+0000","last_fullsized":"2024-02-23T00:40:54.196607+0000","mapping_epoch":582,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":583,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:25.311952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:39:25.311952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":610,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192769+0000","last_change":"2024-02-23T00:40:51.451607+0000","last_active":"2024-02-23T00:40:54.192769+0000","last_peered":"2024-02-23T00:40:54.192769+0000","last_clean":"2024-02-23T00:40:54.192769+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:40:54.192769+0000","last_undegraded":"2024-02-23T00:40:54.192769+0000","last_fullsized":"2024-02-23T00:40:54.192769+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:40:51.451477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:51.451477+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":145,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193742+0000","last_change":"2024-02-23T00:40:12.128771+0000","last_active":"2024-02-23T00:40:54.193742+0000","last_peered":"2024-02-23T00:40:54.193742+0000","last_clean":"2024-02-23T00:40:54.193742+0000","last_became_active":"2024-02-23T00:38:49.217612+0000","last_became_peered":"2024-02-23T00:38:49.217612+0000","last_unstale":"2024-02-23T00:40:54.193742+0000","last_undegraded":"2024-02-23T00:40:54.193742+0000","last_fullsized":"2024-02-23T00:40:54.193742+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'10","last_scrub_stamp":"2024-02-23T00:40:12.128684+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.128684+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":607,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200915+0000","last_change":"2024-02-23T00:40:11.889392+0000","last_active":"2024-02-23T00:40:54.200915+0000","last_peered":"2024-02-23T00:40:54.200915+0000","last_clean":"2024-02-23T00:40:54.200915+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:40:54.200915+0000","last_undegraded":"2024-02-23T00:40:54.200915+0000","last_fullsized":"2024-02-23T00:40:54.200915+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:11.889327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.889327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":610,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192282+0000","last_change":"2024-02-23T00:40:13.313672+0000","last_active":"2024-02-23T00:40:54.192282+0000","last_peered":"2024-02-23T00:40:54.192282+0000","last_clean":"2024-02-23T00:40:54.192282+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:40:54.192282+0000","last_undegraded":"2024-02-23T00:40:54.192282+0000","last_fullsized":"2024-02-23T00:40:54.192282+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:13.313569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:13.313569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":552,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199072+0000","last_change":"2024-02-23T00:38:33.242488+0000","last_active":"2024-02-23T00:40:54.199072+0000","last_peered":"2024-02-23T00:40:54.199072+0000","last_clean":"2024-02-23T00:40:54.199072+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:40:54.199072+0000","last_undegraded":"2024-02-23T00:40:54.199072+0000","last_fullsized":"2024-02-23T00:40:54.199072+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:33.242417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:33.242417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":156,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.195144+0000","last_change":"2024-02-23T00:40:43.233713+0000","last_active":"2024-02-23T00:40:54.195144+0000","last_peered":"2024-02-23T00:40:54.195144+0000","last_clean":"2024-02-23T00:40:54.195144+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:40:54.195144+0000","last_undegraded":"2024-02-23T00:40:54.195144+0000","last_fullsized":"2024-02-23T00:40:54.195144+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:40:43.233609+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:40:43.233609+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":618,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193432+0000","last_change":"2024-02-23T00:40:53.462082+0000","last_active":"2024-02-23T00:40:54.193432+0000","last_peered":"2024-02-23T00:40:54.193432+0000","last_clean":"2024-02-23T00:40:54.193432+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:40:54.193432+0000","last_undegraded":"2024-02-23T00:40:54.193432+0000","last_fullsized":"2024-02-23T00:40:54.193432+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:53.462018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.462018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":608,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.195826+0000","last_change":"2024-02-23T00:38:16.055268+0000","last_active":"2024-02-23T00:40:54.195826+0000","last_peered":"2024-02-23T00:40:54.195826+0000","last_clean":"2024-02-23T00:40:54.195826+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:40:54.195826+0000","last_undegraded":"2024-02-23T00:40:54.195826+0000","last_fullsized":"2024-02-23T00:40:54.195826+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:38:16.055159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":134,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193578+0000","last_change":"2024-02-23T00:38:49.189171+0000","last_active":"2024-02-23T00:40:54.193578+0000","last_peered":"2024-02-23T00:40:54.193578+0000","last_clean":"2024-02-23T00:40:54.193578+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:40:54.193578+0000","last_undegraded":"2024-02-23T00:40:54.193578+0000","last_fullsized":"2024-02-23T00:40:54.193578+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:37.979785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:37.979785+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":738,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.196611+0000","last_change":"2024-02-23T00:40:23.194464+0000","last_active":"2024-02-23T00:40:54.196611+0000","last_peered":"2024-02-23T00:40:54.196611+0000","last_clean":"2024-02-23T00:40:54.196611+0000","last_became_active":"2024-02-23T00:40:23.193752+0000","last_became_peered":"2024-02-23T00:40:23.193752+0000","last_unstale":"2024-02-23T00:40:54.196611+0000","last_undegraded":"2024-02-23T00:40:54.196611+0000","last_fullsized":"2024-02-23T00:40:54.196611+0000","mapping_epoch":603,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":611,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199688+0000","last_change":"2024-02-23T00:38:36.853659+0000","last_active":"2024-02-23T00:40:54.199688+0000","last_peered":"2024-02-23T00:40:54.199688+0000","last_clean":"2024-02-23T00:40:54.199688+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:40:54.199688+0000","last_undegraded":"2024-02-23T00:40:54.199688+0000","last_fullsized":"2024-02-23T00:40:54.199688+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:36.853582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:38:36.853582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":790,"reported_epoch":636,"state":"active+clean","last_fresh":"2024-02-23T00:40:55.399562+0000","last_change":"2024-02-23T00:40:45.431895+0000","last_active":"2024-02-23T00:40:55.399562+0000","last_peered":"2024-02-23T00:40:55.399562+0000","last_clean":"2024-02-23T00:40:55.399562+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:40:55.399562+0000","last_undegraded":"2024-02-23T00:40:55.399562+0000","last_fullsized":"2024-02-23T00:40:55.399562+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:40:45.431814+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.431814+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":594,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193911+0000","last_change":"2024-02-23T00:40:23.194034+0000","last_active":"2024-02-23T00:40:54.193911+0000","last_peered":"2024-02-23T00:40:54.193911+0000","last_clean":"2024-02-23T00:40:54.193911+0000","last_became_active":"2024-02-23T00:40:23.193733+0000","last_became_peered":"2024-02-23T00:40:23.193733+0000","last_unstale":"2024-02-23T00:40:54.193911+0000","last_undegraded":"2024-02-23T00:40:54.193911+0000","last_fullsized":"2024-02-23T00:40:54.193911+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.887584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.887584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":784,"reported_epoch":635,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:40:54.199479+0000","last_change":"2024-02-23T00:40:52.610115+0000","last_active":"2024-02-23T00:40:54.199479+0000","last_peered":"2024-02-23T00:40:54.199479+0000","last_clean":"2024-02-23T00:38:19.193308+0000","last_became_active":"2024-02-23T00:38:22.159363+0000","last_became_peered":"2024-02-23T00:38:22.159363+0000","last_unstale":"2024-02-23T00:40:54.199479+0000","last_undegraded":"2024-02-23T00:40:54.199479+0000","last_fullsized":"2024-02-23T00:40:54.199479+0000","mapping_epoch":484,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":5,"last_scrub":"87'22","last_scrub_stamp":"2024-02-23T00:36:39.442676+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:36:39.442676+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":27,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[1,4,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":149,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.195075+0000","last_change":"2024-02-23T00:40:23.205225+0000","last_active":"2024-02-23T00:40:54.195075+0000","last_peered":"2024-02-23T00:40:54.195075+0000","last_clean":"2024-02-23T00:40:54.195075+0000","last_became_active":"2024-02-23T00:40:23.203999+0000","last_became_peered":"2024-02-23T00:40:23.203999+0000","last_unstale":"2024-02-23T00:40:54.195075+0000","last_undegraded":"2024-02-23T00:40:54.195075+0000","last_fullsized":"2024-02-23T00:40:54.195075+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:32.310679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:39:32.310679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":227,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197044+0000","last_change":"2024-02-23T00:40:23.194631+0000","last_active":"2024-02-23T00:40:54.197044+0000","last_peered":"2024-02-23T00:40:54.197044+0000","last_clean":"2024-02-23T00:40:54.197044+0000","last_became_active":"2024-02-23T00:40:23.194155+0000","last_became_peered":"2024-02-23T00:40:23.194155+0000","last_unstale":"2024-02-23T00:40:54.197044+0000","last_undegraded":"2024-02-23T00:40:54.197044+0000","last_fullsized":"2024-02-23T00:40:54.197044+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:25.593062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:34:25.593062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":560,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194905+0000","last_change":"2024-02-23T00:38:35.044376+0000","last_active":"2024-02-23T00:40:54.194905+0000","last_peered":"2024-02-23T00:40:54.194905+0000","last_clean":"2024-02-23T00:40:54.194905+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:40:54.194905+0000","last_undegraded":"2024-02-23T00:40:54.194905+0000","last_fullsized":"2024-02-23T00:40:54.194905+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:35.044298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:38:35.044298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":748,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199730+0000","last_change":"2024-02-23T00:40:17.268690+0000","last_active":"2024-02-23T00:40:54.199730+0000","last_peered":"2024-02-23T00:40:54.199730+0000","last_clean":"2024-02-23T00:40:54.199730+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:40:54.199730+0000","last_undegraded":"2024-02-23T00:40:54.199730+0000","last_fullsized":"2024-02-23T00:40:54.199730+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:40:17.268569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.268569+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":622,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193466+0000","last_change":"2024-02-23T00:40:19.377388+0000","last_active":"2024-02-23T00:40:54.193466+0000","last_peered":"2024-02-23T00:40:54.193466+0000","last_clean":"2024-02-23T00:40:54.193466+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:40:54.193466+0000","last_undegraded":"2024-02-23T00:40:54.193466+0000","last_fullsized":"2024-02-23T00:40:54.193466+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.377289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.377289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":613,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198327+0000","last_change":"2024-02-23T00:40:23.194154+0000","last_active":"2024-02-23T00:40:54.198327+0000","last_peered":"2024-02-23T00:40:54.198327+0000","last_clean":"2024-02-23T00:40:54.198327+0000","last_became_active":"2024-02-23T00:40:23.193867+0000","last_became_peered":"2024-02-23T00:40:23.193867+0000","last_unstale":"2024-02-23T00:40:54.198327+0000","last_undegraded":"2024-02-23T00:40:54.198327+0000","last_fullsized":"2024-02-23T00:40:54.198327+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:52.791748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:35:52.791748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":141,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193624+0000","last_change":"2024-02-23T00:40:11.172134+0000","last_active":"2024-02-23T00:40:54.193624+0000","last_peered":"2024-02-23T00:40:54.193624+0000","last_clean":"2024-02-23T00:40:54.193624+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:40:54.193624+0000","last_undegraded":"2024-02-23T00:40:54.193624+0000","last_fullsized":"2024-02-23T00:40:54.193624+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:11.172065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.172065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":718,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193866+0000","last_change":"2024-02-23T00:38:55.178058+0000","last_active":"2024-02-23T00:40:54.193866+0000","last_peered":"2024-02-23T00:40:54.193866+0000","last_clean":"2024-02-23T00:40:54.193866+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:40:54.193866+0000","last_undegraded":"2024-02-23T00:40:54.193866+0000","last_fullsized":"2024-02-23T00:40:54.193866+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"76'24","last_scrub_stamp":"2024-02-23T00:36:25.082591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:25.082591+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":29,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":139,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193902+0000","last_change":"2024-02-23T00:40:07.255442+0000","last_active":"2024-02-23T00:40:54.193902+0000","last_peered":"2024-02-23T00:40:54.193902+0000","last_clean":"2024-02-23T00:40:54.193902+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:40:54.193902+0000","last_undegraded":"2024-02-23T00:40:54.193902+0000","last_fullsized":"2024-02-23T00:40:54.193902+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:07.255349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:07.255349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":623,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.189039+0000","last_change":"2024-02-23T00:38:49.224957+0000","last_active":"2024-02-23T00:40:54.189039+0000","last_peered":"2024-02-23T00:40:54.189039+0000","last_clean":"2024-02-23T00:40:54.189039+0000","last_became_active":"2024-02-23T00:38:49.224630+0000","last_became_peered":"2024-02-23T00:38:49.224630+0000","last_unstale":"2024-02-23T00:40:54.189039+0000","last_undegraded":"2024-02-23T00:40:54.189039+0000","last_fullsized":"2024-02-23T00:40:54.189039+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:38:03.063485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:38:03.063485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":556,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200645+0000","last_change":"2024-02-23T00:40:53.782744+0000","last_active":"2024-02-23T00:40:54.200645+0000","last_peered":"2024-02-23T00:40:54.200645+0000","last_clean":"2024-02-23T00:40:54.200645+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:40:54.200645+0000","last_undegraded":"2024-02-23T00:40:54.200645+0000","last_fullsized":"2024-02-23T00:40:54.200645+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:53.782669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.782669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":788,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.193207+0000","last_change":"2024-02-23T00:40:25.188808+0000","last_active":"2024-02-23T00:40:54.193207+0000","last_peered":"2024-02-23T00:40:54.193207+0000","last_clean":"2024-02-23T00:40:54.193207+0000","last_became_active":"2024-02-23T00:40:25.188309+0000","last_became_peered":"2024-02-23T00:40:25.188309+0000","last_unstale":"2024-02-23T00:40:54.193207+0000","last_undegraded":"2024-02-23T00:40:54.193207+0000","last_fullsized":"2024-02-23T00:40:54.193207+0000","mapping_epoch":605,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":324,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.188860+0000","last_change":"2024-02-23T00:40:52.654719+0000","last_active":"2024-02-23T00:40:54.188860+0000","last_peered":"2024-02-23T00:40:54.188860+0000","last_clean":"2024-02-23T00:40:54.188860+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:40:54.188860+0000","last_undegraded":"2024-02-23T00:40:54.188860+0000","last_fullsized":"2024-02-23T00:40:54.188860+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.654614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.654614+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":613,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.200536+0000","last_change":"2024-02-23T00:40:46.004386+0000","last_active":"2024-02-23T00:40:54.200536+0000","last_peered":"2024-02-23T00:40:54.200536+0000","last_clean":"2024-02-23T00:40:54.200536+0000","last_became_active":"2024-02-23T00:40:23.206276+0000","last_became_peered":"2024-02-23T00:40:23.206276+0000","last_unstale":"2024-02-23T00:40:54.200536+0000","last_undegraded":"2024-02-23T00:40:54.200536+0000","last_fullsized":"2024-02-23T00:40:54.200536+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:46.004307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:46.004307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":565,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194737+0000","last_change":"2024-02-23T00:40:11.671468+0000","last_active":"2024-02-23T00:40:54.194737+0000","last_peered":"2024-02-23T00:40:54.194737+0000","last_clean":"2024-02-23T00:40:54.194737+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:40:54.194737+0000","last_undegraded":"2024-02-23T00:40:54.194737+0000","last_fullsized":"2024-02-23T00:40:54.194737+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:11.671393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.671393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":304,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.188989+0000","last_change":"2024-02-23T00:39:21.278362+0000","last_active":"2024-02-23T00:40:54.188989+0000","last_peered":"2024-02-23T00:40:54.188989+0000","last_clean":"2024-02-23T00:40:54.188989+0000","last_became_active":"2024-02-23T00:39:21.278024+0000","last_became_peered":"2024-02-23T00:39:21.278024+0000","last_unstale":"2024-02-23T00:40:54.188989+0000","last_undegraded":"2024-02-23T00:40:54.188989+0000","last_fullsized":"2024-02-23T00:40:54.188989+0000","mapping_epoch":541,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":542,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'15","last_scrub_stamp":"2024-02-23T00:33:49.588815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:33:49.588815+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":622,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194358+0000","last_change":"2024-02-23T00:40:23.206017+0000","last_active":"2024-02-23T00:40:54.194358+0000","last_peered":"2024-02-23T00:40:54.194358+0000","last_clean":"2024-02-23T00:40:54.194358+0000","last_became_active":"2024-02-23T00:40:23.205326+0000","last_became_peered":"2024-02-23T00:40:23.205326+0000","last_unstale":"2024-02-23T00:40:54.194358+0000","last_undegraded":"2024-02-23T00:40:54.194358+0000","last_fullsized":"2024-02-23T00:40:54.194358+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:34:30.726025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:34:30.726025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":613,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192924+0000","last_change":"2024-02-23T00:40:19.660039+0000","last_active":"2024-02-23T00:40:54.192924+0000","last_peered":"2024-02-23T00:40:54.192924+0000","last_clean":"2024-02-23T00:40:54.192924+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:40:54.192924+0000","last_undegraded":"2024-02-23T00:40:54.192924+0000","last_fullsized":"2024-02-23T00:40:54.192924+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.659970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.659970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":576,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.198970+0000","last_change":"2024-02-23T00:39:21.439880+0000","last_active":"2024-02-23T00:40:54.198970+0000","last_peered":"2024-02-23T00:40:54.198970+0000","last_clean":"2024-02-23T00:40:54.198970+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:40:54.198970+0000","last_undegraded":"2024-02-23T00:40:54.198970+0000","last_fullsized":"2024-02-23T00:40:54.198970+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:21.439783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:39:21.439783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":761,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.188407+0000","last_change":"2024-02-23T00:39:24.165093+0000","last_active":"2024-02-23T00:40:54.188407+0000","last_peered":"2024-02-23T00:40:54.188407+0000","last_clean":"2024-02-23T00:40:54.188407+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:40:54.188407+0000","last_undegraded":"2024-02-23T00:40:54.188407+0000","last_fullsized":"2024-02-23T00:40:54.188407+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"81'20","last_scrub_stamp":"2024-02-23T00:37:22.011093+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:37:22.011093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":53,"num_read_kb":44,"num_write":31,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":149,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.194869+0000","last_change":"2024-02-23T00:40:23.199788+0000","last_active":"2024-02-23T00:40:54.194869+0000","last_peered":"2024-02-23T00:40:54.194869+0000","last_clean":"2024-02-23T00:40:54.194869+0000","last_became_active":"2024-02-23T00:40:23.199323+0000","last_became_peered":"2024-02-23T00:40:23.199323+0000","last_unstale":"2024-02-23T00:40:54.194869+0000","last_undegraded":"2024-02-23T00:40:54.194869+0000","last_fullsized":"2024-02-23T00:40:54.194869+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.346259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.346259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":270,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192000+0000","last_change":"2024-02-23T00:40:52.433576+0000","last_active":"2024-02-23T00:40:54.192000+0000","last_peered":"2024-02-23T00:40:54.192000+0000","last_clean":"2024-02-23T00:40:54.192000+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:40:54.192000+0000","last_undegraded":"2024-02-23T00:40:54.192000+0000","last_fullsized":"2024-02-23T00:40:54.192000+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.433499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.433499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":42,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.192543+0000","last_change":"2024-02-23T00:40:23.199367+0000","last_active":"2024-02-23T00:40:54.192543+0000","last_peered":"2024-02-23T00:40:54.192543+0000","last_clean":"2024-02-23T00:40:54.192543+0000","last_became_active":"2024-02-23T00:40:23.199068+0000","last_became_peered":"2024-02-23T00:40:23.199068+0000","last_unstale":"2024-02-23T00:40:54.192543+0000","last_undegraded":"2024-02-23T00:40:54.192543+0000","last_fullsized":"2024-02-23T00:40:54.192543+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:55.176441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:35:55.176441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":864,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.195005+0000","last_change":"2024-02-23T00:40:23.204704+0000","last_active":"2024-02-23T00:40:54.195005+0000","last_peered":"2024-02-23T00:40:54.195005+0000","last_clean":"2024-02-23T00:40:54.195005+0000","last_became_active":"2024-02-23T00:40:23.203940+0000","last_became_peered":"2024-02-23T00:40:23.203940+0000","last_unstale":"2024-02-23T00:40:54.195005+0000","last_undegraded":"2024-02-23T00:40:54.195005+0000","last_fullsized":"2024-02-23T00:40:54.195005+0000","mapping_epoch":603,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:38:34.287816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:38:34.287816+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":232,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.197428+0000","last_change":"2024-02-23T00:40:42.783271+0000","last_active":"2024-02-23T00:40:54.197428+0000","last_peered":"2024-02-23T00:40:54.197428+0000","last_clean":"2024-02-23T00:40:54.197428+0000","last_became_active":"2024-02-23T00:40:23.197454+0000","last_became_peered":"2024-02-23T00:40:23.197454+0000","last_unstale":"2024-02-23T00:40:54.197428+0000","last_undegraded":"2024-02-23T00:40:54.197428+0000","last_fullsized":"2024-02-23T00:40:54.197428+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:42.783207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:42.783207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":610,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.188666+0000","last_change":"2024-02-23T00:40:13.237519+0000","last_active":"2024-02-23T00:40:54.188666+0000","last_peered":"2024-02-23T00:40:54.188666+0000","last_clean":"2024-02-23T00:40:54.188666+0000","last_became_active":"2024-02-23T00:27:21.024036+0000","last_became_peered":"2024-02-23T00:27:21.024036+0000","last_unstale":"2024-02-23T00:40:54.188666+0000","last_undegraded":"2024-02-23T00:40:54.188666+0000","last_fullsized":"2024-02-23T00:40:54.188666+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:13.237419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:13.237419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":575,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-02-23T00:40:54.199220+0000","last_change":"2024-02-23T00:40:06.086703+0000","last_active":"2024-02-23T00:40:54.199220+0000","last_peered":"2024-02-23T00:40:54.199220+0000","last_clean":"2024-02-23T00:40:54.199220+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:40:54.199220+0000","last_undegraded":"2024-02-23T00:40:54.199220+0000","last_fullsized":"2024-02-23T00:40:54.199220+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:06.086635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:06.086635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":238,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":27,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22,"ondisk_log_size":22,"up":24,"acting":24,"num_store_stats":7},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":18,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62,"ondisk_log_size":62,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":1067,"num_read_kb":866,"num_write":642,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":4456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":496,"num_read_kb":496,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":965,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332263,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317364,"kb_used_data":7840,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93427788,"statfs":{"total":95995035648,"available":95670054912,"internally_reserved":0,"allocated":8028160,"data_stored":4594333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":1.02,"15min":1.024},"min":{"1min":0.223,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.79800000000000004,"5min":57.401000000000003,"15min":61.048999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.1719999999999999,"15min":1.2450000000000001},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":65.558999999999997,"15min":73.870999999999995},"last":0.63500000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45900000000000002,"15min":0.68000000000000005},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.217},"max":{"1min":0.69799999999999995,"5min":1.23,"15min":31.175000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.64800000000000002,"15min":0.73899999999999999},"min":{"1min":0.38300000000000001,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":6.742,"5min":11.066000000000001,"15min":29.423999999999999},"last":0.436}]},{"osd":2,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.68400000000000005,"15min":0.72399999999999998},"min":{"1min":0.34000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.0110000000000001,"5min":8.0099999999999998,"15min":16.603999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.58199999999999996,"15min":0.63900000000000001},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":5.7229999999999999,"5min":5.7229999999999999,"15min":7.5339999999999998},"last":0.53100000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.59799999999999998,"15min":1.01},"min":{"1min":0.39200000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72999999999999998,"5min":8.0449999999999999,"15min":57.729999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.53500000000000003,"15min":0.91400000000000003},"min":{"1min":0.35999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68600000000000005,"5min":3.8479999999999999,"15min":27.573},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38400000000000001,"15min":0.41399999999999998},"min":{"1min":0.218,"5min":0.214,"15min":0.214},"max":{"1min":0.86299999999999999,"5min":1.2589999999999999,"15min":1.706},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.45400000000000001,"15min":0.45200000000000001},"min":{"1min":0.26700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66000000000000003,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.39400000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.39400000000000002,"15min":0.42299999999999999},"min":{"1min":0.26600000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.746,"5min":0.746,"15min":1.0209999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.443,"15min":0.45900000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89200000000000002,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.374}]},{"osd":6,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45500000000000002,"15min":0.48399999999999999},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":0.91500000000000004,"5min":1.3129999999999999,"15min":2.1339999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.435,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.311,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77000000000000002,"5min":5.194,"15min":5.194},"last":0.50600000000000001}]}]},{"osd":6,"up_from":44,"seq":188978561196,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321452,"kb_used_data":7832,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93423700,"statfs":{"total":95995035648,"available":95665868800,"internally_reserved":0,"allocated":8019968,"data_stored":4610996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.6499999999999999,"15min":0.89600000000000002},"min":{"1min":0.28799999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.78800000000000003,"5min":94.906000000000006,"15min":94.906000000000006},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.64800000000000002,"15min":0.58199999999999996},"min":{"1min":0.37,"5min":0.252,"15min":0.252},"max":{"1min":0.67000000000000004,"5min":9.468,"15min":9.468},"last":0.42599999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.54700000000000004,"15min":1.026},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.77200000000000002,"5min":6.056,"15min":59.347999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.76700000000000002,"15min":1.3799999999999999},"min":{"1min":0.24299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.3520000000000003,"5min":18.423999999999999,"15min":87.221000000000004},"last":0.496}]},{"osd":2,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.78100000000000003,"15min":0.90400000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":7.2560000000000002,"5min":18.381,"15min":25.550999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.69499999999999995,"15min":0.91500000000000004},"min":{"1min":0.31,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.0099999999999998,"5min":6.1749999999999998,"15min":40.914000000000001},"last":0.47599999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.626,"15min":0.84799999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":7.1630000000000003,"5min":7.1630000000000003,"15min":31.309999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.77000000000000002,"15min":1.407},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.0940000000000003,"5min":18.448,"15min":86.311999999999998},"last":0.42999999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.53500000000000003,"15min":0.46999999999999997},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.8769999999999998,"5min":7.8769999999999998,"15min":7.8769999999999998},"last":0.442},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.47999999999999998,"15min":0.52400000000000002},"min":{"1min":0.23899999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57999999999999996,"5min":6.0300000000000002,"15min":17.718},"last":0.33700000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.54500000000000004,"15min":0.51800000000000002},"min":{"1min":0.28499999999999998,"5min":0.254,"15min":0.254},"max":{"1min":6.7999999999999998,"5min":6.7999999999999998,"15min":8.0990000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.498,"15min":0.498},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.54800000000000004,"5min":6.0780000000000003,"15min":8.1370000000000005},"last":0.41999999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:40:37 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.51100000000000001,"15min":0.61199999999999999},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59399999999999997,"5min":6.4279999999999999,"15min":17.881},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.623,"15min":0.64400000000000002},"min":{"1min":0.27700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.58599999999999997,"5min":18.475000000000001,"15min":18.475000000000001},"last":0.373}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305120,"kb_used_data":7888,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440032,"statfs":{"total":95995035648,"available":95682592768,"internally_reserved":0,"allocated":8077312,"data_stored":4604336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:02 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.29,"15min":0.66300000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.51700000000000002,"5min":37.133000000000003,"15min":37.133000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.89600000000000002,"15min":0.55200000000000005},"min":{"1min":0.23499999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.60499999999999998,"5min":26.690000000000001,"15min":26.690000000000001},"last":0.32400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:40:40 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.34899999999999998,"15min":0.66900000000000004},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.48299999999999998,"5min":3.9980000000000002,"15min":35.313000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.372,"15min":0.58099999999999996},"min":{"1min":0.20799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.7999999999999998,"5min":2.7999999999999998,"15min":27.908999999999999},"last":0.28100000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:40:00 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.40899999999999997,"15min":0.65100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.21299999999999999},"max":{"1min":0.46700000000000003,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.39400000000000002,"15min":0.64300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.48699999999999999,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.29899999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.70299999999999996},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.0230000000000001,"5min":3.4900000000000002,"15min":16.231000000000002},"last":0.5},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51900000000000002,"15min":0.71599999999999997},"min":{"1min":0.317,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.629,"5min":1.518,"15min":29.212},"last":0.441}]},{"osd":5,"last update":"Fri Feb 23 00:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.56899999999999995,"15min":0.72299999999999998},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.8410000000000002,"5min":2.8410000000000002,"15min":16.294},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.57899999999999996,"15min":0.68400000000000005},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67800000000000005,"5min":4.3380000000000001,"15min":11.105},"last":0.44500000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:40:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.754},"min":{"1min":0.33400000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.4049999999999998,"5min":2.4049999999999998,"15min":29.187000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58999999999999997,"15min":0.748},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":2.0499999999999998,"5min":2.528,"15min":16.369},"last":0.47499999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:40:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.63500000000000001,"15min":0.78800000000000003},"min":{"1min":0.39100000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":2.895,"5min":5.2119999999999997,"15min":16.395},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.60599999999999998,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70199999999999996,"5min":3.7669999999999999,"15min":11.135999999999999},"last":0.49199999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411519,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313224,"kb_used_data":7804,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431928,"statfs":{"total":95995035648,"available":95674294272,"internally_reserved":0,"allocated":7991296,"data_stored":4596770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":1.0600000000000001,"15min":0.754},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.65500000000000003,"5min":56.509,"15min":56.509},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.59799999999999998,"15min":0.71199999999999997},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.56799999999999995,"5min":19.571999999999999,"15min":31.498999999999999},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:40:41 2024","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.44,"15min":0.59899999999999998},"min":{"1min":0.22500000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":7.7110000000000003,"15min":19.324999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.5,"15min":0.75700000000000001},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56299999999999994,"5min":7.9779999999999998,"15min":35.052},"last":0.30399999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:40:09 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.56899999999999995,"15min":0.74399999999999999},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.88400000000000001,"5min":10.032,"15min":43.5},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.61599999999999999,"15min":0.69699999999999995},"min":{"1min":0.245,"5min":0.22,"15min":0.22},"max":{"1min":1.7490000000000001,"5min":11.815,"15min":27.414000000000001},"last":0.372}]},{"osd":4,"last update":"Fri Feb 23 00:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.68600000000000005,"15min":0.67000000000000004},"min":{"1min":0.33000000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.84299999999999997,"5min":10.106,"15min":14.614000000000001},"last":0.443},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.34899999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81100000000000005,"5min":7.766,"15min":7.766},"last":0.40300000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68500000000000005,"15min":0.71099999999999997},"min":{"1min":0.39800000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":0.83099999999999996,"5min":7.915,"15min":17.765000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.66700000000000004,"15min":0.67300000000000004},"min":{"1min":0.35799999999999998,"5min":0.33400000000000002,"15min":0.317},"max":{"1min":0.67600000000000005,"5min":10.138,"15min":14.589},"last":0.46300000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:40:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.628,"15min":0.628},"min":{"1min":0.36299999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77500000000000002,"5min":7.7869999999999999,"15min":7.7869999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.73699999999999999,"15min":0.79600000000000004},"min":{"1min":0.38800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.86299999999999999,"5min":7.9589999999999996,"15min":20.574999999999999},"last":0.53300000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:40:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.72999999999999998,"15min":1.075},"min":{"1min":0.38400000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":1.1359999999999999,"5min":7.9480000000000004,"15min":66.632000000000005},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.72899999999999998,"15min":0.84699999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.3660000000000001,"5min":10.163,"15min":19.603999999999999},"last":0.497}]}]},{"osd":3,"up_from":26,"seq":111669149883,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309204,"kb_used_data":7876,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93435948,"statfs":{"total":95995035648,"available":95678410752,"internally_reserved":0,"allocated":8065024,"data_stored":4612091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":1.506,"15min":0.745},"min":{"1min":0.252,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.76100000000000001,"5min":55.238999999999997,"15min":55.238999999999997},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":1.4750000000000001,"15min":0.747},"min":{"1min":0.26100000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.47699999999999998,"5min":54.563000000000002,"15min":54.563000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.375,"15min":0.61599999999999999},"min":{"1min":0.193,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":1.778,"5min":2.718,"15min":54.351999999999997},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.46500000000000002,"15min":0.78700000000000003},"min":{"1min":0.253,"5min":0.219,"15min":0.219},"max":{"1min":3.7120000000000002,"5min":3.7120000000000002,"15min":55.337000000000003},"last":0.24199999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.45700000000000002,"15min":0.77100000000000002},"min":{"1min":0.23899999999999999,"5min":0.223,"15min":0.223},"max":{"1min":3.7570000000000001,"5min":3.7570000000000001,"15min":55.368000000000002},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.45200000000000001,"15min":0.74399999999999999},"min":{"1min":0.249,"5min":0.214,"15min":0.185},"max":{"1min":2.4710000000000001,"5min":4.181,"15min":54.383000000000003},"last":0.26200000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63400000000000001,"15min":0.94199999999999995},"min":{"1min":0.34000000000000002,"5min":0.316,"15min":0.27900000000000003},"max":{"1min":1.6220000000000001,"5min":6.8799999999999999,"15min":54.530000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.59299999999999997,"15min":0.629},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.24099999999999999},"max":{"1min":1.5369999999999999,"5min":4.9829999999999997,"15min":8.0359999999999996},"last":0.41699999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.55200000000000005,"15min":0.878},"min":{"1min":0.35799999999999998,"5min":0.34699999999999998,"15min":0.28899999999999998},"max":{"1min":0.81100000000000005,"5min":3.8180000000000001,"15min":55.401000000000003},"last":0.443},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.66500000000000004,"15min":0.86899999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.30299999999999999},"max":{"1min":0.78200000000000003,"5min":7.0179999999999998,"15min":54.454999999999998},"last":0.51300000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.73899999999999999,"15min":0.84799999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.5660000000000001,"5min":9.7490000000000006,"15min":54.305},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79800000000000004,"15min":1.022},"min":{"1min":0.375,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.8420000000000001,"5min":9.6820000000000004,"15min":55.424999999999997},"last":0.42499999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.63200000000000001,"15min":0.78700000000000003},"min":{"1min":0.33000000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73899999999999999,"5min":4.9790000000000001,"15min":54.222000000000001},"last":0.437},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.93899999999999995,"15min":1.0309999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78500000000000003,"5min":23.928000000000001,"15min":55.460000000000001},"last":0.497}]}]},{"osd":4,"up_from":31,"seq":133143986358,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309160,"kb_used_data":7832,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93435992,"statfs":{"total":95995035648,"available":95678455808,"internally_reserved":0,"allocated":8019968,"data_stored":4598301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:36:08 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":1.1739999999999999,"15min":0.83099999999999996},"min":{"1min":0.28699999999999998,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.68799999999999994,"5min":21.419,"15min":21.419},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":2.1200000000000001,"15min":1.2589999999999999},"min":{"1min":0.40400000000000003,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":0.69699999999999995,"5min":63.462000000000003,"15min":63.462000000000003},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.83699999999999997,"15min":1.095},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.242,"5min":16.696000000000002,"15min":41.137999999999998},"last":0.435},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.71699999999999997,"15min":0.90100000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.627,"5min":13.455,"15min":26.210000000000001},"last":0.46400000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.83799999999999997,"15min":1.179},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":9.3490000000000002,"5min":13.478,"15min":67.253},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.7070000000000001,"5min":0.88900000000000001,"15min":1.121},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":14.442,"5min":14.442,"15min":41.927999999999997},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.77900000000000003,"15min":1.121},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":9.5579999999999998,"5min":9.5579999999999998,"15min":36.301000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.78400000000000003,"15min":1.1899999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.24299999999999999},"max":{"1min":3.6819999999999999,"5min":16.657,"15min":42.064},"last":0.379}]},{"osd":5,"last update":"Fri Feb 23 00:40:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.42899999999999999,"15min":0.56399999999999995},"min":{"1min":0.254,"5min":0.219,"15min":0.217},"max":{"1min":1.4279999999999999,"5min":2.3210000000000002,"15min":14.242000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.53100000000000003,"15min":0.60399999999999998},"min":{"1min":0.255,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":1.4079999999999999,"5min":7.1459999999999999,"15min":14.124000000000001},"last":0.308}]},{"osd":6,"last update":"Fri Feb 23 00:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.54000000000000004,"15min":0.56799999999999995},"min":{"1min":0.28999999999999998,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":0.58899999999999997,"5min":7.1189999999999998,"15min":8.0730000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.44700000000000001,"15min":0.63},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.54300000000000004,"5min":2.2450000000000001,"15min":45.814999999999998},"last":0.41399999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:40:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.58299999999999996,"15min":0.63800000000000001},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.9210000000000003,"5min":7.9210000000000003,"15min":17.178999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.72599999999999998,"15min":0.71299999999999997},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.28299999999999997},"max":{"1min":6.3220000000000001,"5min":15.85,"15min":15.85},"last":0.39300000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757425,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317340,"kb_used_data":7816,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93427812,"statfs":{"total":95995035648,"available":95670079488,"internally_reserved":0,"allocated":8003584,"data_stored":4593854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.1200000000000001,"15min":0.70699999999999996},"min":{"1min":0.36699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.115,"5min":27.411000000000001,"15min":27.411000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":2.0750000000000002,"15min":1.028},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.498,"5min":103.57599999999999,"15min":103.57599999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.46899999999999997,"15min":0.749},"min":{"1min":0.316,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":0.53600000000000003,"5min":3.7629999999999999,"15min":38.838000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.51600000000000001,"15min":1.179},"min":{"1min":0.252,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.56100000000000005,"5min":3.7250000000000001,"15min":127.794},"last":0.505}]},{"osd":2,"last update":"Fri Feb 23 00:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.70999999999999996,"15min":0.95499999999999996},"min":{"1min":0.34399999999999997,"5min":0.22,"15min":0.22},"max":{"1min":0.59199999999999997,"5min":11.435,"15min":37.442999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.438,"5min":0.495,"15min":0.69699999999999995},"min":{"1min":0.27600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.58599999999999997,"5min":3.734,"15min":23.297999999999998},"last":0.58599999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.73599999999999999,"15min":0.82099999999999995},"min":{"1min":0.33300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67100000000000004,"5min":12.507,"15min":24.414999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.56299999999999994,"15min":0.90700000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.60999999999999999,"5min":3.8119999999999998,"15min":40.865000000000002},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:40:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.378,"15min":0.77900000000000003},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.219},"max":{"1min":0.5,"5min":2.8940000000000001,"15min":103.13},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.54300000000000004,"15min":0.51300000000000001},"min":{"1min":0.26600000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.60099999999999998,"5min":11.381,"15min":12.795999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:40:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.59899999999999998,"15min":0.54300000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.26200000000000001},"max":{"1min":0.58199999999999996,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.497},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.44700000000000001,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64200000000000002,"5min":1.5229999999999999,"15min":103.61499999999999},"last":0.55800000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:40:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.59799999999999998,"15min":0.56100000000000005},"min":{"1min":0.29899999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.625,"5min":11.478,"15min":11.478},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.58499999999999996,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.62}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:41:02.792 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 2, currently 12 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.090944+0000 osd.2 (osd.2) 235 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.091901+0000 osd.2 (osd.2) 236 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.178586+0000 osd.1 (osd.1) 298 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.179663+0000 osd.1 (osd.1) 299 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.637502+0000 osd.4 (osd.4) 291 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.638549+0000 osd.4 (osd.4) 292 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.758820+0000 mgr.y (mgr.24437) 566 : cluster [DBG] pgmap v980: 200 pgs: 2 peering, 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:01.759736+0000 mon.a (mon.0) 3374 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:02.131201+0000 osd.1 (osd.1) 300 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:02.134894+0000 osd.1 (osd.1) 301 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:02.189878+0000 mon.a (mon.0) 3375 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-108"}]': finished 2024-02-23T00:41:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:02.189942+0000 mon.a (mon.0) 3376 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-23T00:41:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:02.189981+0000 mon.a (mon.0) 3377 : cluster [DBG] osdmap e643: 8 total, 7 up, 7 in 2024-02-23T00:41:02.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:02.192913+0000 mon.a (mon.0) 3378 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-108", "mode": "writeback"}]: dispatch 2024-02-23T00:41:02.965 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:02.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:03.069 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:03.104 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:03.125 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:03.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:03.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:03.229 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:03.259 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:3 63.11 e117/114 63:8d290297:test-rados-api-smithi087-82091-8::foo:head v 117'1, mlcod=0'0)", 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 595.29607199300006, 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074073193999999995, 2024-02-23T00:41:03.265 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.266 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965524+0000", 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1930000000000001e-06 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965529+0000", 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5130000000000002e-06 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965536+0000", 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4860000000000002e-06 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.965542+0000", 2024-02-23T00:41:03.267 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002109362 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.967652+0000", 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5451000000000002e-05 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:07.967667+0000", 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071902888999999998 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.039570+0000", 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7617999999999999e-05 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:08.039597+0000", 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.074073193999999995 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:10 61.1f e118/112 61:ff3dbdbd:test-rados-api-smithi087-82107-7::foo:head v 118'6, mlcod=115'3)", 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:41:03.269 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 594.08117558900005, 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034719099000000003, 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180421+0000", 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.270 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180420+0000", 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7989999999999999e-06 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180426+0000", 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8590000000000001e-06 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180432+0000", 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8689999999999998e-06 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.271 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180439+0000", 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8420000000000001e-05 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180477+0000", 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6568000000000001e-05 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.180494+0000", 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034614477999999997 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.272 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.215108+0000", 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1405999999999997e-05 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:09.215140+0000", 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.034719099000000003 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.28906.0:2 69.15 69:ad551702:::test:head [create,write 0~4] snapc 0=[] ondisk+write+known_if_redirected e119)", 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:41:03.273 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 593.08092989099998, 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071209718000000005, 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.28906", 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3024219439", 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.274 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180666+0000", 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180664+0000", 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8156e-05 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180683+0000", 2024-02-23T00:41:03.275 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3705e-05 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180696+0000", 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0080000000000002e-06 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.180704+0000", 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046747999999999999 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.181172+0000", 2024-02-23T00:41:03.276 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7989999999999996e-06 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.181179+0000", 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.060883893000000001 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242062+0000", 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012322200000000001 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242186+0000", 2024-02-23T00:41:03.277 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015854399999999999 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.242344+0000", 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0042969569999999997 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.246641+0000", 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020537120000000001 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.278 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.248695+0000", 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0030777460000000001 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251773+0000", 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9827999999999997e-05 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251832+0000", 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3659000000000003e-05 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.279 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:10.251876+0000", 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.071209718000000005 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.15189.0:14 61.11 e120/112 61:8c482978:test-rados-api-smithi087-82107-8::foo:head v 120'3, mlcod=0'0)", 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 592.03572669899995, 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.14463899, 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.280 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225870+0000", 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225869+0000", 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2559999999999996e-06 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.281 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225877+0000", 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2709999999999998e-06 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225883+0000", 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1999999999999999e-06 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.225889+0000", 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.13293687000000001 2024-02-23T00:41:03.282 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358826+0000", 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1993000000000001e-05 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.358848+0000", 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.011619638 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.370468+0000", 2024-02-23T00:41:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0979000000000002e-05 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:11.370509+0000", 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.14463899 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.41269.0:2 89.6 89:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e137)", 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:41:03.284 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 575.23869440399994, 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.10320085900000001, 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.41269", 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/244424669", 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.285 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022902+0000", 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022900+0000", 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2230000000000001e-06 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.022907+0000", 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026383299999999999 2024-02-23T00:41:03.286 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.023171+0000", 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0040999999999999e-05 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.023181+0000", 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.014422166 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.037603+0000", 2024-02-23T00:41:03.287 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5469999999999997e-06 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.037612+0000", 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.064763451999999999 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102375+0000", 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011688600000000001 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.288 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102492+0000", 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013874699999999999 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.102631+0000", 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00093563499999999998 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.103566+0000", 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001628163 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.289 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.105195+0000", 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.020814273000000001 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126009+0000", 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0680000000000003e-05 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126059+0000", 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3244999999999997e-05 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.290 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:28.126103+0000", 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.10320085900000001 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:265 86.0 86:00000000:%01:::head [pgnls start_epoch 139] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e139)", 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 573.22418533500002, 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.048698721, 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:41:03.291 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 265 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037411+0000", 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:03.292 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037409+0000", 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.456e-06 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037418+0000", 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8989999999999996e-06 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037427+0000", 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0649999999999999e-06 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.293 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.037435+0000", 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017399350000000001 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.054835+0000", 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8939999999999993e-06 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.054843+0000", 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.031096629000000001 2024-02-23T00:41:03.294 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085939+0000", 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8654e-05 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085978+0000", 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013163199999999999 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.086110+0000", 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.048698721 2024-02-23T00:41:03.295 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.36860.0:2 114.16 114:6eb9e51e:::oid_RoundTripWriteSamePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e168)", 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 544.25745932500001, 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036246133, 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.36860", 2024-02-23T00:41:03.296 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4037067096", 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004137+0000", 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.297 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004135+0000", 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6179999999999995e-06 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004143+0000", 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4696e-05 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004158+0000", 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7260000000000008e-06 2024-02-23T00:41:03.298 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004167+0000", 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5045999999999999e-05 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004242+0000", 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1400000000000002e-06 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:41:03.299 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.004249+0000", 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016930304 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.021179+0000", 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2629999999999997e-06 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.021187+0000", 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.016612273 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.300 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037799+0000", 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3777999999999997e-05 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037873+0000", 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7861000000000005e-05 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 4,5", 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.037960+0000", 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00035295600000000003 2024-02-23T00:41:03.301 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.038313+0000", 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028716700000000001 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.038601+0000", 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016768 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040277+0000", 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0985000000000001e-05 2024-02-23T00:41:03.302 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040328+0000", 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4675000000000001e-05 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:59.040383+0000", 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036246133 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.303 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.3.0:538 63.8 e182/114 63:1309b075:test-rados-api-smithi087-82091-14::foo36:7 v 182'40, mlcod=148'34)", 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 530.25849146099995, 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027705371999999999, 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.304 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003105+0000", 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003104+0000", 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7220000000000002e-06 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003111+0000", 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5139999999999998e-06 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.305 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003118+0000", 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8010000000000001e-06 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.003122+0000", 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024802080000000001 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027924+0000", 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9273999999999999e-05 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.306 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.027943+0000", 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0028410689999999999 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030784+0000", 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6224000000000002e-05 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:13.030810+0000", 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027705371999999999 2024-02-23T00:41:03.307 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37106.0:2 149.1a 149:5ec95afd:::oid_ReadIntoBufferlist_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 496.23442881800003, 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.079034594, 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37106", 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/4049231794", 2024-02-23T00:41:03.308 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027167+0000", 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.309 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027166+0000", 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9900000000000002e-06 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027172+0000", 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4880000000000007e-06 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027180+0000", 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4049999999999997e-06 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.310 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027187+0000", 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000237202 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027424+0000", 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000115528 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027539+0000", 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00014589299999999999 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.311 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,3", 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027685+0000", 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00088195299999999999 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.028567+0000", 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026987410999999999 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.055555+0000", 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.050551275 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.312 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106106+0000", 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5090999999999998e-05 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106161+0000", 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0967000000000003e-05 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.106202+0000", 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.079034594 2024-02-23T00:41:03.313 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:2298 169.17 169:ed571587:test-rados-api-smithi087-82091-30::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e256)", 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 456.23618884500002, 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869, 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2298 2024-02-23T00:41:03.314 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025406+0000", 2024-02-23T00:41:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1740000000000006e-06 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025415+0000", 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1041e-05 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025426+0000", 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0450000000000003e-06 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025433+0000", 2024-02-23T00:41:03.316 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7723999999999998e-05 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025491+0000", 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3479999999999993e-06 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025500+0000", 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.21538338000000001 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.317 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.240883+0000", 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021381300000000001 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.241097+0000", 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00573626 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246834+0000", 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6838999999999999e-05 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.318 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246930+0000", 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013423800000000001 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,2", 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247065+0000", 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037269000000000001 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247437+0000", 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026355299999999998 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.319 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247701+0000", 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011556330000000001 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248856+0000", 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7636999999999998e-05 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248904+0000", 2024-02-23T00:41:03.320 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0179000000000003e-05 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248944+0000", 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 322/321 [PushReplyOp(4:e9187ab8:reshard::reshard.0000000000:head)])", 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 389.938226789, 2024-02-23T00:41:03.321 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089931579999999997, 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.322 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9330000000000004e-06 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323376+0000", 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1073000000000001e-05 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323387+0000", 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.489e-06 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.323 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323393+0000", 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089729134000000002 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.413122+0000", 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00017875200000000001 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.413301+0000", 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089931579999999997 2024-02-23T00:41:03.324 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 322/321 [PushReplyOp(4:ec529b31:::obj_delete_at_hint.0000000050:head)])", 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 389.50397205799999, 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089810938000000007, 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:41:03.325 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757623+0000", 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9920000000000007e-06 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.326 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757631+0000", 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1764e-05 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757643+0000", 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4240000000000004e-06 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757650+0000", 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089629418000000002 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.327 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.847280+0000", 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000155267 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.847435+0000", 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089810938000000007 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 323/321 [PushReplyOp(4:ece66ccb:::obj_delete_at_hint.0000000102:head)])", 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:41:03.328 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 389.15584783000003, 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089865278000000007, 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0180000000000001e-06 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105753+0000", 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3409999999999997e-06 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105760+0000", 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2069999999999997e-06 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105764+0000", 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089713752999999993 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.195478+0000", 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013554999999999999 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.195614+0000", 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089865278000000007 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.331 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 323/321 [PushReplyOp(4:ee62d5d2:lc::lc.1:head)])", 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 388.979600952, 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.090010914999999997, 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:41:03.332 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7539999999999998e-06 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281998+0000", 2024-02-23T00:41:03.333 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8160000000000004e-06 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.282002+0000", 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.794e-06 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.282005+0000", 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089811657000000003 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.371817+0000", 2024-02-23T00:41:03.334 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000189092 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.372006+0000", 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.090010914999999997 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_update_log_missing(205.3s1 epoch 376/375 rep_tid 700 entries 376'1 (0'0) error 205:cbb9903e:::test_obj:head by client.37589.0:1 0.000000 -95 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 336.16332054999998, 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065042928, 2024-02-23T00:41:03.335 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.336 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098275+0000", 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2939999999999994e-06 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098285+0000", 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2446000000000001e-05 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098297+0000", 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2310000000000002e-06 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.337 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098306+0000", 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012984100000000001 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098436+0000", 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.064882361999999999 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.163319+0000", 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065042928 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.338 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.4.31:55 216.9 e451/444 216:939aa6ec:test-rados-api-smithi087-82091-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head v 451'3, mlcod=446'1)", 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:37:17.130726+0000", 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 226.130870684, 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056389038000000002, 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130726+0000", 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.339 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130726+0000", 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130725+0000", 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5980000000000003e-06 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.340 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130734+0000", 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0025e-05 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130744+0000", 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6889999999999998e-06 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130751+0000", 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5022999999999999e-05 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.341 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130796+0000", 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.4711000000000001e-05 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130821+0000", 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056043386000000001 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.186864+0000", 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025056500000000001 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.342 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187115+0000", 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056389038000000002 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.4.31:56 216.17 e451/444 216:ec5c3d73:test-rados-api-smithi087-82091-70::b1a341c59dc4c60bbe00244ac8bba87528936bc7:head v 451'2, mlcod=0'0)", 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:37:17.143192+0000", 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 226.11840401200001, 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044005469999999998, 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.343 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143192+0000", 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143192+0000", 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143190+0000", 2024-02-23T00:41:03.344 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5917999999999999e-05 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143206+0000", 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5682999999999999e-05 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143222+0000", 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2430000000000005e-06 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.345 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143230+0000", 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.672e-05 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143277+0000", 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2272e-05 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143299+0000", 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.043740635999999999 2024-02-23T00:41:03.346 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187040+0000", 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015778499999999999 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187198+0000", 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044005469999999998 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.347 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.7.51:63 63.10 63:0bcb77f5:test-rados-api-smithi087-82091-80::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e494)", 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:38:31.219712+0000", 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 152.04188435, 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030054408000000001, 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219712+0000", 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.348 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219712+0000", 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219710+0000", 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1394e-05 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219721+0000", 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4367e-05 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.349 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219736+0000", 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9110000000000004e-06 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219743+0000", 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1088999999999999e-05 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219784+0000", 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3820000000000003e-06 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:41:03.350 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219789+0000", 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020061689999999999 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.221795+0000", 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012612300000000001 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.221921+0000", 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000363552 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 7", 2024-02-23T00:41:03.351 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.222285+0000", 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00093142099999999999 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.223216+0000", 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026412662999999999 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.249629+0000", 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9842999999999997e-05 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:03.352 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.249679+0000", 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7428000000000004e-05 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.249766+0000", 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030054408000000001 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:03.353 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:41:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:03.355 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:03.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:03.455 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:03.484 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:03.503 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:03.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:03.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.921819+0000 osd.6 (osd.6) 264 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:01.921979+0000 osd.6 (osd.6) 265 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:02.303841+0000 osd.3 (osd.3) 201 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:02.306852+0000 osd.3 (osd.3) 202 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:02.450398+0000 osd.6 (osd.6) 266 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:02.451875+0000 osd.6 (osd.6) 267 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:02.797363+0000 osd.5 (osd.5) 253 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:02.799401+0000 osd.5 (osd.5) 254 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:03.187266+0000 mon.a (mon.0) 3379 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:03.191187+0000 mon.a (mon.0) 3380 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-108", "mode": "writeback"}]': finished 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:03.191273+0000 mon.a (mon.0) 3381 : cluster [DBG] osdmap e644: 8 total, 7 up, 7 in 2024-02-23T00:41:03.545 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.921819+0000 osd.6 (osd.6) 264 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:01.921979+0000 osd.6 (osd.6) 265 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:02.303841+0000 osd.3 (osd.3) 201 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:02.306852+0000 osd.3 (osd.3) 202 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:02.450398+0000 osd.6 (osd.6) 266 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:02.451875+0000 osd.6 (osd.6) 267 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:02.797363+0000 osd.5 (osd.5) 253 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:02.799401+0000 osd.5 (osd.5) 254 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:03.187266+0000 mon.a (mon.0) 3379 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:03.191187+0000 mon.a (mon.0) 3380 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-108", "mode": "writeback"}]': finished 2024-02-23T00:41:03.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:03.191273+0000 mon.a (mon.0) 3381 : cluster [DBG] osdmap e644: 8 total, 7 up, 7 in 2024-02-23T00:41:03.608 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:03.636 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:03.655 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:03.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:03.759 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:03.783 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.921819+0000 osd.6 (osd.6) 264 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:01.921979+0000 osd.6 (osd.6) 265 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:02.303841+0000 osd.3 (osd.3) 201 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:02.306852+0000 osd.3 (osd.3) 202 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:02.450398+0000 osd.6 (osd.6) 266 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:02.451875+0000 osd.6 (osd.6) 267 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:02.797363+0000 osd.5 (osd.5) 253 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:02.799401+0000 osd.5 (osd.5) 254 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:03.187266+0000 mon.a (mon.0) 3379 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:03.191187+0000 mon.a (mon.0) 3380 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-108", "mode": "writeback"}]': finished 2024-02-23T00:41:03.784 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:03.191273+0000 mon.a (mon.0) 3381 : cluster [DBG] osdmap e644: 8 total, 7 up, 7 in 2024-02-23T00:41:03.785 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:03.805 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:03.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:03.909 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:03.935 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:03.954 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:03.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:04.057 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:04.088 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:04.162 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:04.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:04.266 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:04.294 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:04.744 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:04.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:04.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:04.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:03.318851+0000 osd.3 (osd.3) 203 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:41:04.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:03.320380+0000 osd.3 (osd.3) 204 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:41:04.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:03.629814+0000 osd.4 (osd.4) 293 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:41:04.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:03.631908+0000 osd.4 (osd.4) 294 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:41:04.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:03.759537+0000 mgr.y (mgr.24437) 567 : cluster [DBG] pgmap v983: 200 pgs: 2 peering, 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:04.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:04.193178+0000 mon.a (mon.0) 3382 : cluster [DBG] osdmap e645: 8 total, 7 up, 7 in 2024-02-23T00:41:04.751 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:04.220333+0000 mon.a (mon.0) 3383 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:04.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:03.318851+0000 osd.3 (osd.3) 203 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:41:04.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:03.320380+0000 osd.3 (osd.3) 204 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:41:04.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:03.629814+0000 osd.4 (osd.4) 293 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:41:04.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:03.631908+0000 osd.4 (osd.4) 294 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:41:04.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:03.759537+0000 mgr.y (mgr.24437) 567 : cluster [DBG] pgmap v983: 200 pgs: 2 peering, 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:04.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:04.193178+0000 mon.a (mon.0) 3382 : cluster [DBG] osdmap e645: 8 total, 7 up, 7 in 2024-02-23T00:41:04.752 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:04.220333+0000 mon.a (mon.0) 3383 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:04.753 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-23T00:41:04.848 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:04.876 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:03.318851+0000 osd.3 (osd.3) 203 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:41:04.876 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:03.320380+0000 osd.3 (osd.3) 204 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:41:04.876 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:03.629814+0000 osd.4 (osd.4) 293 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:41:04.876 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:03.631908+0000 osd.4 (osd.4) 294 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:41:04.876 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:03.759537+0000 mgr.y (mgr.24437) 567 : cluster [DBG] pgmap v983: 200 pgs: 2 peering, 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:04.877 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:04.193178+0000 mon.a (mon.0) 3382 : cluster [DBG] osdmap e645: 8 total, 7 up, 7 in 2024-02-23T00:41:04.877 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:04.220333+0000 mon.a (mon.0) 3383 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:04.878 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:04.898 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:04.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:05.002 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:05.031 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:05.050 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:05.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:05.154 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:05.181 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:05.203 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:05.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:05.307 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:05.333 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:04.067784+0000 osd.2 (osd.2) 237 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:41:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:04.069268+0000 osd.2 (osd.2) 238 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:41:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:04.154048+0000 osd.3 (osd.3) 205 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:04.154186+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:05.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:05.197008+0000 mon.a (mon.0) 3384 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:05.197071+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e646: 8 total, 7 up, 7 in 2024-02-23T00:41:05.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:05.199895+0000 mon.a (mon.0) 3386 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108"}]: dispatch 2024-02-23T00:41:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:04.067784+0000 osd.2 (osd.2) 237 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:41:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:04.069268+0000 osd.2 (osd.2) 238 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:41:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:04.154048+0000 osd.3 (osd.3) 205 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:04.154186+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:05.197008+0000 mon.a (mon.0) 3384 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:05.197071+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e646: 8 total, 7 up, 7 in 2024-02-23T00:41:05.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:05.199895+0000 mon.a (mon.0) 3386 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108"}]: dispatch 2024-02-23T00:41:05.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:04.067784+0000 osd.2 (osd.2) 237 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:41:05.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:04.069268+0000 osd.2 (osd.2) 238 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:41:05.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:04.154048+0000 osd.3 (osd.3) 205 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:05.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:04.154186+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:05.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:05.197008+0000 mon.a (mon.0) 3384 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:05.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:05.197071+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e646: 8 total, 7 up, 7 in 2024-02-23T00:41:05.945 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:05.199895+0000 mon.a (mon.0) 3386 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108"}]: dispatch 2024-02-23T00:41:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:41:06.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:41:06.774 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:05.760483+0000 mgr.y (mgr.24437) 568 : cluster [DBG] pgmap v986: 199 pgs: 1 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/699 objects misplaced (2.861%) 2024-02-23T00:41:06.774 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:05.761362+0000 mon.a (mon.0) 3387 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-23T00:41:06.774 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:06.191447+0000 mon.a (mon.0) 3388 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:06.774 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:06.199178+0000 mon.a (mon.0) 3389 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108"}]': finished 2024-02-23T00:41:06.774 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:06.199273+0000 mon.a (mon.0) 3390 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-23T00:41:06.774 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:06.199327+0000 mon.a (mon.0) 3391 : cluster [DBG] osdmap e647: 8 total, 7 up, 7 in 2024-02-23T00:41:06.774 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:05.760483+0000 mgr.y (mgr.24437) 568 : cluster [DBG] pgmap v986: 199 pgs: 1 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/699 objects misplaced (2.861%) 2024-02-23T00:41:06.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:05.761362+0000 mon.a (mon.0) 3387 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-23T00:41:06.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:06.191447+0000 mon.a (mon.0) 3388 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:06.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:06.199178+0000 mon.a (mon.0) 3389 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108"}]': finished 2024-02-23T00:41:06.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:06.199273+0000 mon.a (mon.0) 3390 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-23T00:41:06.775 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:06.199327+0000 mon.a (mon.0) 3391 : cluster [DBG] osdmap e647: 8 total, 7 up, 7 in 2024-02-23T00:41:06.832 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:06.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:06.936 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:05.760483+0000 mgr.y (mgr.24437) 568 : cluster [DBG] pgmap v986: 199 pgs: 1 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/699 objects misplaced (2.861%) 2024-02-23T00:41:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:05.761362+0000 mon.a (mon.0) 3387 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-23T00:41:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:06.191447+0000 mon.a (mon.0) 3388 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:06.199178+0000 mon.a (mon.0) 3389 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-108"}]': finished 2024-02-23T00:41:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:06.199273+0000 mon.a (mon.0) 3390 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-23T00:41:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:06.199327+0000 mon.a (mon.0) 3391 : cluster [DBG] osdmap e647: 8 total, 7 up, 7 in 2024-02-23T00:41:06.968 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:06.990 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:06.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:07.093 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:07.124 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:07.145 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:07.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:07.249 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:07.277 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:07.296 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:07.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:07.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:07.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.373+0000 7f4b23a84700 1 -- 172.21.15.87:0/541134266 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 msgr2=0x7f4b1c0fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:07.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.373+0000 7f4b23a84700 1 --2- 172.21.15.87:0/541134266 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 0x7f4b1c0fcf90 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 rx=0x7f4b18004870 tx=0x7f4b18009cb0).stop 2024-02-23T00:41:07.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.373+0000 7f4b23a84700 1 -- 172.21.15.87:0/541134266 shutdown_connections 2024-02-23T00:41:07.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.373+0000 7f4b23a84700 1 --2- 172.21.15.87:0/541134266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b1c0f7400 0x7f4b1c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:07.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.373+0000 7f4b23a84700 1 --2- 172.21.15.87:0/541134266 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 0x7f4b1c0fcf90 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:07.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.373+0000 7f4b23a84700 1 --2- 172.21.15.87:0/541134266 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b1c0fd510 0x7f4b1c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:07.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.374+0000 7f4b23a84700 1 -- 172.21.15.87:0/541134266 >> 172.21.15.87:0/541134266 conn(0x7f4b1c0f2820 msgr2=0x7f4b1c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:07.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.374+0000 7f4b23a84700 1 -- 172.21.15.87:0/541134266 shutdown_connections 2024-02-23T00:41:07.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.374+0000 7f4b23a84700 1 -- 172.21.15.87:0/541134266 wait complete. 2024-02-23T00:41:07.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.374+0000 7f4b23a84700 1 librados: starting msgr at 2024-02-23T00:41:07.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.374+0000 7f4b23a84700 1 librados: starting objecter 2024-02-23T00:41:07.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.374+0000 7f4b23a84700 1 Processor -- start 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.374+0000 7f4b23a84700 1 -- start start 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.374+0000 7f4b23a84700 1 librados: setting wanted keys 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.374+0000 7f4b23a84700 1 librados: calling monclient init 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b23a84700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b1c0f7400 0x7f4b1c107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b23a84700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 0x7f4b1c107540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b23a84700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b1c0fd510 0x7f4b1c107b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b23a84700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c10c240 con 0x7f4b1c0fd510 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b23a84700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c10c3b0 con 0x7f4b1c0f7400 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b23a84700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c10c570 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b2101f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 0x7f4b1c107540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b2101f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 0x7f4b1c107540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:34176/0 (socket says 172.21.15.87:34176) 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b2101f700 1 -- 172.21.15.87:0/1933792211 learned_addr learned my addr 172.21.15.87:0/1933792211 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:07.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b22021700 1 --2- 172.21.15.87:0/1933792211 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b1c0fd510 0x7f4b1c107b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b21820700 1 --2- 172.21.15.87:0/1933792211 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b1c0f7400 0x7f4b1c107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b2101f700 1 -- 172.21.15.87:0/1933792211 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b1c0f7400 msgr2=0x7f4b1c107050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b2101f700 1 --2- 172.21.15.87:0/1933792211 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b1c0f7400 0x7f4b1c107050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b2101f700 1 -- 172.21.15.87:0/1933792211 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b1c0fd510 msgr2=0x7f4b1c107b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b2101f700 1 --2- 172.21.15.87:0/1933792211 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b1c0fd510 0x7f4b1c107b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b2101f700 1 -- 172.21.15.87:0/1933792211 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18005040 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b2101f700 1 --2- 172.21.15.87:0/1933792211 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 0x7f4b1c107540 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7f4b0c0041d0 tx=0x7f4b0c009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b12f78700 10 client.?.objecter ms_handle_connect 0x7f4b1c0fcbb0 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.375+0000 7f4b12f78700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.376+0000 7f4b12f78700 1 -- 172.21.15.87:0/1933792211 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c007b40 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.376+0000 7f4b12f78700 1 -- 172.21.15.87:0/1933792211 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b0c00f040 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.376+0000 7f4b12f78700 1 -- 172.21.15.87:0/1933792211 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c014810 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.376+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1c10c850 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.376+0000 7f4b23a84700 10 client.38138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:07.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.376+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1c10ccf0 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.376+0000 7f4b23a84700 1 librados: init done 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.377+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1c05fa60 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.377+0000 7f4b12f78700 1 -- 172.21.15.87:0/1933792211 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4b0c014c00 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.377+0000 7f4b12f78700 1 --2- 172.21.15.87:0/1933792211 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4b08062ce0 0x7f4b080651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.378+0000 7f4b12f78700 1 -- 172.21.15.87:0/1933792211 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(648..648 src has 173..648) v4 ==== 8218+0+0 (secure 0 0 0) 0x7f4b0c07d280 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.378+0000 7f4b12f78700 10 client.38138.objecter ms_dispatch 0x7f4b1c001550 osd_map(648..648 src has 173..648) v4 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.378+0000 7f4b12f78700 3 client.38138.objecter handle_osd_map got epochs [648,648] > 0 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.378+0000 7f4b12f78700 3 client.38138.objecter handle_osd_map decoding full epoch 648 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.378+0000 7f4b21820700 1 --2- 172.21.15.87:0/1933792211 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4b08062ce0 0x7f4b080651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.378+0000 7f4b12f78700 20 client.38138.objecter dump_active .. 0 homeless 2024-02-23T00:41:07.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.378+0000 7f4b12f78700 10 client.38138.objecter ms_handle_connect 0x7f4b08062ce0 2024-02-23T00:41:07.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.378+0000 7f4b21820700 1 --2- 172.21.15.87:0/1933792211 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4b08062ce0 0x7f4b080651a0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7f4b18004710 tx=0x7f4b18007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:07.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.382+0000 7f4b12f78700 1 -- 172.21.15.87:0/1933792211 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4b0c0506b0 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.398 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:07.401 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:07.428 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:07.447 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:07.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:07.550 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:07.576 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:07.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:07.577+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f4b1c05ee80 con 0x7f4b1c0fcbb0 2024-02-23T00:41:07.595 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:07.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:07.698 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:07.725 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:07.743 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:07.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:07.794 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-02-23T00:41:07.794 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:41:07.794 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-02-23T00:41:07.794 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-02-23T00:41:07.794 DEBUG:teuthology.orchestra.run.smithi087:osd.0> sudo journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.0.service 2024-02-23T00:41:07.797 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl start ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.0 2024-02-23T00:41:07.826 INFO:journalctl@ceph.osd.0.smithi087.stdout:-- Logs begin at Fri 2024-02-23 00:01:42 UTC. -- 2024-02-23T00:41:07.848 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:07.874 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:07.893 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:07.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:07.984 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:07 smithi087 systemd[1]: Starting Ceph osd.0 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:41:07.997 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:08.025 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:08.043 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:08.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:08.147 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:08.173 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:08.192 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:08.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:08.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:08.209 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.207+0000 7f4b12f78700 1 -- 172.21.15.87:0/1933792211 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v649) v1 ==== 87+0+0 (secure 0 0 0) 0x7f4b0c0181b0 con 0x7f4b1c0fcbb0 2024-02-23T00:41:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.211+0000 7f4b23a84700 10 librados: watch_flush enter 2024-02-23T00:41:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.211+0000 7f4b23a84700 10 librados: watch_flush exit 2024-02-23T00:41:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.211+0000 7f4b23a84700 20 client.38138.objecter shutdown clearing up homeless session... 2024-02-23T00:41:08.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.211+0000 7f4b23a84700 10 client.38138.objecter successfully canceled tick 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.211+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4b08062ce0 msgr2=0x7f4b080651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.211+0000 7f4b23a84700 1 --2- 172.21.15.87:0/1933792211 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4b08062ce0 0x7f4b080651a0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 rx=0x7f4b18004710 tx=0x7f4b18007f90).stop 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 msgr2=0x7f4b1c107540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 --2- 172.21.15.87:0/1933792211 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 0x7f4b1c107540 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 rx=0x7f4b0c0041d0 tx=0x7f4b0c009f60).stop 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 shutdown_connections 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 --2- 172.21.15.87:0/1933792211 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4b08062ce0 0x7f4b080651a0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 --2- 172.21.15.87:0/1933792211 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4b1c0fd510 0x7f4b1c107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 --2- 172.21.15.87:0/1933792211 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4b1c0f7400 0x7f4b1c107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 --2- 172.21.15.87:0/1933792211 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4b1c0fcbb0 0x7f4b1c107540 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:08.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 >> 172.21.15.87:0/1933792211 conn(0x7f4b1c0f2820 msgr2=0x7f4b1c0f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:08.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 shutdown_connections 2024-02-23T00:41:08.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 -- 172.21.15.87:0/1933792211 wait complete. 2024-02-23T00:41:08.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:08.212+0000 7f4b23a84700 1 librados: shutdown 2024-02-23T00:41:08.215 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:07.193363+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e648: 8 total, 7 up, 7 in 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:07.578384+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.87:0/1933792211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:07.578676+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:08.113069+0000 osd.1 (osd.1) 302 : cluster [DBG] 4.f scrub starts 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:08.116758+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.f scrub ok 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:07.193363+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e648: 8 total, 7 up, 7 in 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:07.578384+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.87:0/1933792211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:07.578676+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:08.113069+0000 osd.1 (osd.1) 302 : cluster [DBG] 4.f scrub starts 2024-02-23T00:41:08.238 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:08.116758+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.f scrub ok 2024-02-23T00:41:08.296 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:08.327 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:08.403 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:41:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:07.193363+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e648: 8 total, 7 up, 7 in 2024-02-23T00:41:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:07.578384+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.87:0/1933792211' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:07.578676+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:08.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:08.113069+0000 osd.1 (osd.1) 302 : cluster [DBG] 4.f scrub starts 2024-02-23T00:41:08.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:08.116758+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.f scrub ok 2024-02-23T00:41:08.576 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:08.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:08.680 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:08.708 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:09.157 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:09.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:09.261 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:09.291 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:09.312 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:09.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:09.416 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:09.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:07.609531+0000 osd.4 (osd.4) 295 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:41:09.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:07.610324+0000 osd.4 (osd.4) 296 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:41:09.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:07.747868+0000 osd.7 (osd.7) 294 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:41:09.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:07.753673+0000 osd.7 (osd.7) 295 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:41:09.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:07.760926+0000 mgr.y (mgr.24437) 569 : cluster [DBG] pgmap v989: 167 pgs: 1 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 164 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/699 objects misplaced (2.861%) 2024-02-23T00:41:09.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:07.802419+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:41:09.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:07.803502+0000 osd.5 (osd.5) 256 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:41:09.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:08.190638+0000 mon.a (mon.0) 3394 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:09.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:08.201474+0000 mon.a (mon.0) 3395 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:41:09.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:08.201532+0000 mon.a (mon.0) 3396 : cluster [DBG] osdmap e649: 8 total, 7 up, 7 in 2024-02-23T00:41:09.445 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:08.203191+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:09.446 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:09.464 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:09.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:09.467 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:07.609531+0000 osd.4 (osd.4) 295 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:41:09.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:07.610324+0000 osd.4 (osd.4) 296 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:41:09.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:07.747868+0000 osd.7 (osd.7) 294 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:41:09.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:07.753673+0000 osd.7 (osd.7) 295 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:41:09.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:07.760926+0000 mgr.y (mgr.24437) 569 : cluster [DBG] pgmap v989: 167 pgs: 1 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 164 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/699 objects misplaced (2.861%) 2024-02-23T00:41:09.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:07.802419+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:41:09.468 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:07.803502+0000 osd.5 (osd.5) 256 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:08.190638+0000 mon.a (mon.0) 3394 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:08.201474+0000 mon.a (mon.0) 3395 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:08.201532+0000 mon.a (mon.0) 3396 : cluster [DBG] osdmap e649: 8 total, 7 up, 7 in 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:08.203191+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:07.609531+0000 osd.4 (osd.4) 295 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:07.610324+0000 osd.4 (osd.4) 296 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:07.747868+0000 osd.7 (osd.7) 294 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:07.753673+0000 osd.7 (osd.7) 295 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:07.760926+0000 mgr.y (mgr.24437) 569 : cluster [DBG] pgmap v989: 167 pgs: 1 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 164 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/699 objects misplaced (2.861%) 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:07.802419+0000 osd.5 (osd.5) 255 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:07.803502+0000 osd.5 (osd.5) 256 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:08.190638+0000 mon.a (mon.0) 3394 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:09.469 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:08.201474+0000 mon.a (mon.0) 3395 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:41:09.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:08.201532+0000 mon.a (mon.0) 3396 : cluster [DBG] osdmap e649: 8 total, 7 up, 7 in 2024-02-23T00:41:09.470 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:08.203191+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:09.568 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:09.599 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:09.749 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:09.752 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:41:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:41:09.658+0000 7f6e819c8700 -1 mgr get_metadata_python Requested missing service osd.0 2024-02-23T00:41:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:09.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:09.854 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:09.882 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:09.901 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:09.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:09.977 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:41:09.977 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:41:09.977 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:41:09.977 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:41:10.005 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:10.032 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:10.052 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:10.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:10.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:10.155 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:10.182 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:10.201 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:10.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:10.304 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:09.189825+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:09.190048+0000 osd.4 (osd.4) 298 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:10.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:09.203724+0000 mon.a (mon.0) 3398 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:09.203896+0000 mon.a (mon.0) 3399 : cluster [DBG] osdmap e650: 8 total, 7 up, 7 in 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:09.225189+0000 mon.a (mon.0) 3400 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:09.655198+0000 mon.a (mon.0) 3401 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:09.749952+0000 mon.a (mon.0) 3402 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:09.762251+0000 mon.a (mon.0) 3403 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:10.057465+0000 mon.a (mon.0) 3404 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:10.058191+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:10.063189+0000 mon.a (mon.0) 3406 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:09.189825+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:10.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:09.190048+0000 osd.4 (osd.4) 298 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:10.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:09.203724+0000 mon.a (mon.0) 3398 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:10.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:09.203896+0000 mon.a (mon.0) 3399 : cluster [DBG] osdmap e650: 8 total, 7 up, 7 in 2024-02-23T00:41:10.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:09.225189+0000 mon.a (mon.0) 3400 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:10.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:09.655198+0000 mon.a (mon.0) 3401 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:10.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:09.749952+0000 mon.a (mon.0) 3402 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:41:10.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:09.762251+0000 mon.a (mon.0) 3403 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-23T00:41:10.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:10.057465+0000 mon.a (mon.0) 3404 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:41:10.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:10.058191+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:41:10.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:10.063189+0000 mon.a (mon.0) 3406 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:10.341 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:10.421 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:10.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:10.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:09.189825+0000 osd.4 (osd.4) 297 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:09.190048+0000 osd.4 (osd.4) 298 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:09.203724+0000 mon.a (mon.0) 3398 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:09.203896+0000 mon.a (mon.0) 3399 : cluster [DBG] osdmap e650: 8 total, 7 up, 7 in 2024-02-23T00:41:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:09.225189+0000 mon.a (mon.0) 3400 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:09.655198+0000 mon.a (mon.0) 3401 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:09.749952+0000 mon.a (mon.0) 3402 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:41:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:09.762251+0000 mon.a (mon.0) 3403 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-23T00:41:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:10.057465+0000 mon.a (mon.0) 3404 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:41:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:10.058191+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:41:10.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:10.063189+0000 mon.a (mon.0) 3406 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:10.528 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:10.555 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:10.575 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:10.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:10.678 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:10.689 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:10 smithi087 bash[224079]: --> Failed to activate via raw: did not find any matching OSD to activate 2024-02-23T00:41:10.689 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:10 smithi087 bash[224079]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-23T00:41:10.689 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:10 smithi087 bash[224079]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-02-23T00:41:10.689 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:10 smithi087 bash[224079]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-23T00:41:10.690 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:10 smithi087 bash[224079]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-02-23T00:41:10.690 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:10 smithi087 bash[224079]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-23T00:41:10.690 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:10 smithi087 bash[224079]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-23T00:41:10.690 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:10 smithi087 bash[224079]: --> ceph-volume lvm activate successful for osd ID: 0 2024-02-23T00:41:10.705 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:10.726 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:10.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:10.829 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:10.859 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:11.197 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-23T00:41:11.208 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:11.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:11.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:11.338 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:09.761334+0000 mgr.y (mgr.24437) 570 : cluster [DBG] pgmap v992: 198 pgs: 32 unknown, 1 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 163 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/699 objects misplaced (2.861%) 2024-02-23T00:41:11.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:10.211582+0000 mon.a (mon.0) 3407 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:11.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:10.211663+0000 mon.a (mon.0) 3408 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-23T00:41:11.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:10.211712+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e651: 8 total, 7 up, 7 in 2024-02-23T00:41:11.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:10.214712+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-110"}]: dispatch 2024-02-23T00:41:11.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:41:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:41:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:09.761334+0000 mgr.y (mgr.24437) 570 : cluster [DBG] pgmap v992: 198 pgs: 32 unknown, 1 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 163 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/699 objects misplaced (2.861%) 2024-02-23T00:41:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:10.211582+0000 mon.a (mon.0) 3407 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:10.211663+0000 mon.a (mon.0) 3408 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-23T00:41:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:10.211712+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e651: 8 total, 7 up, 7 in 2024-02-23T00:41:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:10.214712+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-110"}]: dispatch 2024-02-23T00:41:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:09.761334+0000 mgr.y (mgr.24437) 570 : cluster [DBG] pgmap v992: 198 pgs: 32 unknown, 1 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 163 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/699 objects misplaced (2.861%) 2024-02-23T00:41:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:10.211582+0000 mon.a (mon.0) 3407 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:10.211663+0000 mon.a (mon.0) 3408 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-23T00:41:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:10.211712+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e651: 8 total, 7 up, 7 in 2024-02-23T00:41:11.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:10.214712+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-110"}]: dispatch 2024-02-23T00:41:11.841 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:12.225 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:12] "GET /metrics HTTP/1.1" 200 179641 "" "Prometheus/2.33.4" 2024-02-23T00:41:12.491 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:12.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:10.945236+0000 osd.7 (osd.7) 296 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:12.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:10.945469+0000 osd.7 (osd.7) 297 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:12.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:11.157282+0000 mgr.y (mgr.24437) 571 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:12.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:11.215666+0000 mon.a (mon.0) 3411 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-110"}]': finished 2024-02-23T00:41:12.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:11.215753+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e652: 8 total, 7 up, 7 in 2024-02-23T00:41:12.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:11.217802+0000 mon.a (mon.0) 3413 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-110", "mode": "writeback"}]: dispatch 2024-02-23T00:41:12.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:12.189684+0000 mon.a (mon.0) 3414 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:12.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:12.196893+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-110", "mode": "writeback"}]': finished 2024-02-23T00:41:12.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:12.196973+0000 mon.a (mon.0) 3416 : cluster [DBG] osdmap e653: 8 total, 7 up, 7 in 2024-02-23T00:41:12.495 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:10.945236+0000 osd.7 (osd.7) 296 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:12.495 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:10.945469+0000 osd.7 (osd.7) 297 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:12.495 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:11.157282+0000 mgr.y (mgr.24437) 571 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:12.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:11.215666+0000 mon.a (mon.0) 3411 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-110"}]': finished 2024-02-23T00:41:12.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:11.215753+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e652: 8 total, 7 up, 7 in 2024-02-23T00:41:12.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:11.217802+0000 mon.a (mon.0) 3413 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-110", "mode": "writeback"}]: dispatch 2024-02-23T00:41:12.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:12.189684+0000 mon.a (mon.0) 3414 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:12.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:12.196893+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-110", "mode": "writeback"}]': finished 2024-02-23T00:41:12.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:12.196973+0000 mon.a (mon.0) 3416 : cluster [DBG] osdmap e653: 8 total, 7 up, 7 in 2024-02-23T00:41:12.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:12.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:12.598 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:12.625 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:10.945236+0000 osd.7 (osd.7) 296 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:10.945469+0000 osd.7 (osd.7) 297 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:11.157282+0000 mgr.y (mgr.24437) 571 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:11.215666+0000 mon.a (mon.0) 3411 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-110"}]': finished 2024-02-23T00:41:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:11.215753+0000 mon.a (mon.0) 3412 : cluster [DBG] osdmap e652: 8 total, 7 up, 7 in 2024-02-23T00:41:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:11.217802+0000 mon.a (mon.0) 3413 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-110", "mode": "writeback"}]: dispatch 2024-02-23T00:41:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:12.189684+0000 mon.a (mon.0) 3414 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:12.196893+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-110", "mode": "writeback"}]': finished 2024-02-23T00:41:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:12.196973+0000 mon.a (mon.0) 3416 : cluster [DBG] osdmap e653: 8 total, 7 up, 7 in 2024-02-23T00:41:12.731 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:12.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:12.835 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:12.868 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:12.890 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:12.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:12.995 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:13.024 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:13.045 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:13.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:13.149 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:13.180 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:13.284 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:13.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:13.386 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:13.415 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:13.427 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-23T00:41:13.436 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:13.438 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[224705]: debug 2024-02-23T00:41:13.303+0000 7fc02f099700 1 -- 172.21.15.87:0/2917304268 <== mon.2 v2:172.21.15.87:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5613232a6680 con 0x56132326d400 2024-02-23T00:41:13.438 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:13 smithi087 bash[224679]: 1c15912054b5413d3066125dddf21c47139f0eb1199140dd6f0f0ce0c04d7c93 2024-02-23T00:41:13.439 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:13 smithi087 systemd[1]: Started Ceph osd.0 for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:41:13.439 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:11.761877+0000 mgr.y (mgr.24437) 572 : cluster [DBG] pgmap v995: 198 pgs: 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:13.439 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:12.323085+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:13.439 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:13.195547+0000 mon.a (mon.0) 3418 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:13.439 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:13.195604+0000 mon.a (mon.0) 3419 : cluster [DBG] osdmap e654: 8 total, 7 up, 7 in 2024-02-23T00:41:13.439 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:13.197389+0000 mon.a (mon.0) 3420 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110"}]: dispatch 2024-02-23T00:41:13.439 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:11.761877+0000 mgr.y (mgr.24437) 572 : cluster [DBG] pgmap v995: 198 pgs: 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:13.439 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:12.323085+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:13.440 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:13.195547+0000 mon.a (mon.0) 3418 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:13.440 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:13.195604+0000 mon.a (mon.0) 3419 : cluster [DBG] osdmap e654: 8 total, 7 up, 7 in 2024-02-23T00:41:13.440 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:13.197389+0000 mon.a (mon.0) 3420 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110"}]: dispatch 2024-02-23T00:41:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:13.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:13.541 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:13.568 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:13.625 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:41:13.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:11.761877+0000 mgr.y (mgr.24437) 572 : cluster [DBG] pgmap v995: 198 pgs: 2 active+remapped+backfill_toofull, 196 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:13.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:12.323085+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:13.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:13.195547+0000 mon.a (mon.0) 3418 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:13.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:13.195604+0000 mon.a (mon.0) 3419 : cluster [DBG] osdmap e654: 8 total, 7 up, 7 in 2024-02-23T00:41:13.696 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:13.197389+0000 mon.a (mon.0) 3420 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110"}]: dispatch 2024-02-23T00:41:13.738 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:14.083 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[224705]: debug 2024-02-23T00:41:13.830+0000 7fc036b27380 -1 Falling back to public interface 2024-02-23T00:41:14.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:41:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:41:14.318 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:14.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:14.421 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:14.454 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:14.475 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:14.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:14.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:13.763766+0000 mon.a (mon.0) 3421 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-23T00:41:14.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:14.192579+0000 mon.a (mon.0) 3422 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:14.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:14.202601+0000 mon.a (mon.0) 3423 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110"}]': finished 2024-02-23T00:41:14.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:14.202668+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-23T00:41:14.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:14.202713+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e655: 8 total, 7 up, 7 in 2024-02-23T00:41:14.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:13.763766+0000 mon.a (mon.0) 3421 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-23T00:41:14.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:14.192579+0000 mon.a (mon.0) 3422 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:14.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:14.202601+0000 mon.a (mon.0) 3423 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110"}]': finished 2024-02-23T00:41:14.571 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:14.202668+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-23T00:41:14.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:14.202713+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e655: 8 total, 7 up, 7 in 2024-02-23T00:41:14.578 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:14.612 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:14.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:13.763766+0000 mon.a (mon.0) 3421 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-23T00:41:14.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:14.192579+0000 mon.a (mon.0) 3422 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:14.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:14.202601+0000 mon.a (mon.0) 3423 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-110"}]': finished 2024-02-23T00:41:14.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:14.202668+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-23T00:41:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:14.202713+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e655: 8 total, 7 up, 7 in 2024-02-23T00:41:14.864 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:14.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:14.969 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:15.001 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:15.022 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:15.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:15.126 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:15.156 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:15.176 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:15.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.247+0000 7fd093890700 1 -- 172.21.15.87:0/2785245165 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd08c0fe510 msgr2=0x7fd08c0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.247+0000 7fd093890700 1 --2- 172.21.15.87:0/2785245165 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd08c0fe510 0x7fd08c0f7d40 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 rx=0x7fd07c004870 tx=0x7fd07c009cb0).stop 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.247+0000 7fd093890700 1 -- 172.21.15.87:0/2785245165 shutdown_connections 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.247+0000 7fd093890700 1 --2- 172.21.15.87:0/2785245165 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd08c0f8420 0x7fd08c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.247+0000 7fd093890700 1 --2- 172.21.15.87:0/2785245165 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd08c0fdbb0 0x7fd08c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.247+0000 7fd093890700 1 --2- 172.21.15.87:0/2785245165 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd08c0fe510 0x7fd08c0f7d40 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.247+0000 7fd093890700 1 -- 172.21.15.87:0/2785245165 >> 172.21.15.87:0/2785245165 conn(0x7fd08c0f3800 msgr2=0x7fd08c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.247+0000 7fd093890700 1 -- 172.21.15.87:0/2785245165 shutdown_connections 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.247+0000 7fd093890700 1 -- 172.21.15.87:0/2785245165 wait complete. 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 librados: starting msgr at 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 librados: starting objecter 2024-02-23T00:41:15.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 Processor -- start 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 -- start start 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 librados: setting wanted keys 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 librados: calling monclient init 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd08c0f8420 0x7fd08c10e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd08c0fdbb0 0x7fd08c10e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd08c0fe510 0x7fd08c1081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c1086b0 con 0x7fd08c0f8420 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c108820 con 0x7fd08c0fdbb0 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd093890700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c1089e0 con 0x7fd08c0fe510 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd09162c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd08c0f8420 0x7fd08c10e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd09162c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd08c0f8420 0x7fd08c10e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53192/0 (socket says 172.21.15.87:53192) 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd091e2d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd08c0fe510 0x7fd08c1081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd09162c700 1 -- 172.21.15.87:0/2678214715 learned_addr learned my addr 172.21.15.87:0/2678214715 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd090e2b700 1 --2- 172.21.15.87:0/2678214715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd08c0fdbb0 0x7fd08c10e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd091e2d700 1 -- 172.21.15.87:0/2678214715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd08c0fdbb0 msgr2=0x7fd08c10e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd091e2d700 1 --2- 172.21.15.87:0/2678214715 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd08c0fdbb0 0x7fd08c10e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd091e2d700 1 -- 172.21.15.87:0/2678214715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd08c0f8420 msgr2=0x7fd08c10e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd091e2d700 1 --2- 172.21.15.87:0/2678214715 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd08c0f8420 0x7fd08c10e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.248+0000 7fd091e2d700 1 -- 172.21.15.87:0/2678214715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c005040 con 0x7fd08c0fe510 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd091e2d700 1 --2- 172.21.15.87:0/2678214715 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd08c0fe510 0x7fd08c1081c0 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 rx=0x7fd084010e80 tx=0x7fd08400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd0827fc700 10 client.?.objecter ms_handle_connect 0x7fd08c0fe510 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd0827fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084015070 con 0x7fd08c0fe510 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd084009870 con 0x7fd08c0fe510 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08c108c60 con 0x7fd08c0fe510 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd093890700 10 client.38147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0840077a0 con 0x7fd08c0fe510 2024-02-23T00:41:15.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08c06af10 con 0x7fd08c0fe510 2024-02-23T00:41:15.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.249+0000 7fd093890700 1 librados: init done 2024-02-23T00:41:15.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.250+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd08c05ee80 con 0x7fd08c0fe510 2024-02-23T00:41:15.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.250+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd084004020 con 0x7fd08c0fe510 2024-02-23T00:41:15.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0827fc700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd078062c90 0x7fd078065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd09162c700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd078062c90 0x7fd078065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(656..656 src has 173..656) v4 ==== 8162+0+0 (secure 0 0 0) 0x7fd084081640 con 0x7fd08c0fe510 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 osd_map(656..656 src has 173..656) v4 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0827fc700 3 client.38147.objecter handle_osd_map got epochs [656,656] > 0 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0827fc700 3 client.38147.objecter handle_osd_map decoding full epoch 656 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0827fc700 20 client.38147.objecter dump_active .. 0 homeless 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v656) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd084081b20 con 0x7fd08c0fe510 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd093890700 10 client.38147.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 1 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd093890700 20 client.38147.objecter _get_session osd=-1 returning homeless 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd093890700 10 client.38147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0827fc700 10 client.38147.objecter ms_handle_connect 0x7fd078062c90 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd09162c700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd078062c90 0x7fd078065150 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 rx=0x7fd088004f30 tx=0x7fd08800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:15.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=657}) v3 -- 0x7fd08c1070a0 con 0x7fd08c0fe510 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd08c1096a0 con 0x7fd08c0fe510 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_get_version_reply(handle=1 version=656) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd084054ae0 con 0x7fd08c0fe510 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0837fe700 10 client.38147.objecter _check_command_map_dne tid 1 current 656 map_dne_bound 656 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0837fe700 10 client.38147.objecter _finish_command 1 = osd:6 osd down 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.251+0000 7fd0837fe700 15 client.38147.objecter _session_command_op_remove -1 1 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 10 client.38147.objecter _submit_command 2 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 2 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6805/685354171 conn(0x7fd08c0fe8f0 0x7fd08c0fecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c028540 osd=1 3 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c028540 osd=1 3 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c028540 osd=1 3 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 2 2024-02-23T00:41:15.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c028540 osd=1 3 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 1 2 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c028540 osd=1 4 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 10 client.38147.objecter _send_command 2 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd08c05a040 con 0x7fd08c0fe8f0 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd090e2b700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6805/685354171 conn(0x7fd08c0fe8f0 0x7fd08c0fecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd0827fc700 10 client.38147.objecter ms_handle_connect 0x7fd08c0fe8f0 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd090e2b700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6805/685354171 conn(0x7fd08c0fe8f0 0x7fd08c0fecd0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.252+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd08c05a040 con 0x7fd08c0fe8f0 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.253+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.253+0000 7fd0827fc700 10 client.38147.objecter _finish_command 2 = system:0 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.253+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c028540 osd=1 4 2024-02-23T00:41:15.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.253+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 1 2 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 10 client.38147.objecter _submit_command 3 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 3 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c028540 osd=1 3 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c028540 osd=1 3 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c028540 osd=1 3 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 3 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c028540 osd=1 3 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 1 3 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c028540 osd=1 4 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 10 client.38147.objecter _send_command 3 2024-02-23T00:41:15.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.267+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd08c0ff1c0 con 0x7fd08c0fe8f0 2024-02-23T00:41:15.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd08c0ff1c0 con 0x7fd08c0fe8f0 2024-02-23T00:41:15.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 3: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:15.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd0827fc700 10 client.38147.objecter _finish_command 3 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c028540 osd=1 4 2024-02-23T00:41:15.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 1 3 2024-02-23T00:41:15.271 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:41:15.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 10 client.38147.objecter _submit_command 4 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:15.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 4 2024-02-23T00:41:15.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd08c059710 0x7fd08c06b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c0595a0 osd=2 3 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c0595a0 osd=2 3 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c0595a0 osd=2 3 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 4 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c0595a0 osd=2 3 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 2 4 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c0595a0 osd=2 4 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 10 client.38147.objecter _send_command 4 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.87:6809/3605394228 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd08c100070 con 0x7fd08c059710 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd09162c700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd08c059710 0x7fd08c06b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd0827fc700 10 client.38147.objecter ms_handle_connect 0x7fd08c059710 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.269+0000 7fd09162c700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd08c059710 0x7fd08c06b190 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.270+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd08c100070 con 0x7fd08c059710 2024-02-23T00:41:15.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.270+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 4: 0 ) v1 2024-02-23T00:41:15.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.270+0000 7fd0827fc700 10 client.38147.objecter _finish_command 4 = system:0 2024-02-23T00:41:15.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.270+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c0595a0 osd=2 4 2024-02-23T00:41:15.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.270+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 2 4 2024-02-23T00:41:15.281 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:15.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 10 client.38147.objecter _submit_command 5 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:15.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 5 2024-02-23T00:41:15.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c0595a0 osd=2 3 2024-02-23T00:41:15.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c0595a0 osd=2 3 2024-02-23T00:41:15.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c0595a0 osd=2 3 2024-02-23T00:41:15.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 5 2024-02-23T00:41:15.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c0595a0 osd=2 3 2024-02-23T00:41:15.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 2 5 2024-02-23T00:41:15.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c0595a0 osd=2 4 2024-02-23T00:41:15.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 10 client.38147.objecter _send_command 5 2024-02-23T00:41:15.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.285+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd08c0ff5d0 con 0x7fd08c059710 2024-02-23T00:41:15.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd08c0ff5d0 con 0x7fd08c059710 2024-02-23T00:41:15.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 5: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:15.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd0827fc700 10 client.38147.objecter _finish_command 5 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c0595a0 osd=2 4 2024-02-23T00:41:15.288 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 2 5 2024-02-23T00:41:15.289 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:41:15.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 10 client.38147.objecter _submit_command 6 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:15.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 6 2024-02-23T00:41:15.289 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6813/128059790 conn(0x7fd08c06cd10 0x7fd08c027b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c0ff9e0 osd=3 3 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c0ff9e0 osd=3 3 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c0ff9e0 osd=3 3 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 6 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c0ff9e0 osd=3 3 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 3 6 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c0ff9e0 osd=3 4 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 10 client.38147.objecter _send_command 6 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.87:6813/128059790 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd08c028230 con 0x7fd08c06cd10 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd090e2b700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6813/128059790 conn(0x7fd08c06cd10 0x7fd08c027b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd0827fc700 10 client.38147.objecter ms_handle_connect 0x7fd08c06cd10 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.287+0000 7fd090e2b700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.87:6813/128059790 conn(0x7fd08c06cd10 0x7fd08c027b90 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.288+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd08c028230 con 0x7fd08c06cd10 2024-02-23T00:41:15.290 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.288+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 6: 0 ) v1 2024-02-23T00:41:15.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.288+0000 7fd0827fc700 10 client.38147.objecter _finish_command 6 = system:0 2024-02-23T00:41:15.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.288+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c0ff9e0 osd=3 4 2024-02-23T00:41:15.291 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.288+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 3 6 2024-02-23T00:41:15.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 10 client.38147.objecter _submit_command 7 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:15.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 7 2024-02-23T00:41:15.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c0ff9e0 osd=3 3 2024-02-23T00:41:15.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c0ff9e0 osd=3 3 2024-02-23T00:41:15.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c0ff9e0 osd=3 3 2024-02-23T00:41:15.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 7 2024-02-23T00:41:15.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c0ff9e0 osd=3 3 2024-02-23T00:41:15.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 3 7 2024-02-23T00:41:15.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c0ff9e0 osd=3 4 2024-02-23T00:41:15.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 10 client.38147.objecter _send_command 7 2024-02-23T00:41:15.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.303+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd08c027600 con 0x7fd08c06cd10 2024-02-23T00:41:15.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd08c027600 con 0x7fd08c06cd10 2024-02-23T00:41:15.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 7: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:15.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd0827fc700 10 client.38147.objecter _finish_command 7 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c0ff9e0 osd=3 4 2024-02-23T00:41:15.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 3 7 2024-02-23T00:41:15.306 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:41:15.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 10 client.38147.objecter _submit_command 8 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:15.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 8 2024-02-23T00:41:15.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd08c11c000 0x7fd08c11e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06be50 osd=4 3 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06be50 osd=4 3 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06be50 osd=4 3 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 8 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c06be50 osd=4 3 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 4 8 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06be50 osd=4 4 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 10 client.38147.objecter _send_command 8 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.171:6800/4175844757 -- command(tid 8: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd08c06c510 con 0x7fd08c11c000 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.305+0000 7fd091e2d700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd08c11c000 0x7fd08c11e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.306+0000 7fd091e2d700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd08c11c000 0x7fd08c11e3f0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:15.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.306+0000 7fd0827fc700 10 client.38147.objecter ms_handle_connect 0x7fd08c11c000 2024-02-23T00:41:15.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.306+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 8: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd08c06c510 con 0x7fd08c11c000 2024-02-23T00:41:15.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.306+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 8: 0 ) v1 2024-02-23T00:41:15.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.306+0000 7fd0827fc700 10 client.38147.objecter _finish_command 8 = system:0 2024-02-23T00:41:15.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.306+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c06be50 osd=4 4 2024-02-23T00:41:15.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.306+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 4 8 2024-02-23T00:41:15.310 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:15.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.321+0000 7fd093890700 10 client.38147.objecter _submit_command 9 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:15.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 9 2024-02-23T00:41:15.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06be50 osd=4 3 2024-02-23T00:41:15.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06be50 osd=4 3 2024-02-23T00:41:15.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06be50 osd=4 3 2024-02-23T00:41:15.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 9 2024-02-23T00:41:15.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c06be50 osd=4 3 2024-02-23T00:41:15.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 4 9 2024-02-23T00:41:15.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06be50 osd=4 4 2024-02-23T00:41:15.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 10 client.38147.objecter _send_command 9 2024-02-23T00:41:15.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.322+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd08c06b8c0 con 0x7fd08c11c000 2024-02-23T00:41:15.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.323+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd08c06b8c0 con 0x7fd08c11c000 2024-02-23T00:41:15.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.323+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 9: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:15.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.323+0000 7fd0827fc700 10 client.38147.objecter _finish_command 9 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.323+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c06be50 osd=4 4 2024-02-23T00:41:15.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.323+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 4 9 2024-02-23T00:41:15.326 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 10 client.38147.objecter _submit_command 10 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 10 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd08c11bb60 0x7fd08c121a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06b8c0 osd=5 3 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06b8c0 osd=5 3 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06b8c0 osd=5 3 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 10 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c06b8c0 osd=5 3 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 5 10 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06b8c0 osd=5 4 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 10 client.38147.objecter _send_command 10 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.171:6804/2550013795 -- command(tid 10: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd08c06bb50 con 0x7fd08c11bb60 2024-02-23T00:41:15.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.324+0000 7fd09162c700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd08c11bb60 0x7fd08c121a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.325+0000 7fd0827fc700 10 client.38147.objecter ms_handle_connect 0x7fd08c11bb60 2024-02-23T00:41:15.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.325+0000 7fd09162c700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd08c11bb60 0x7fd08c121a70 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:15.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.325+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 10: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd08c06bb50 con 0x7fd08c11bb60 2024-02-23T00:41:15.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.325+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 10: 0 ) v1 2024-02-23T00:41:15.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.325+0000 7fd0827fc700 10 client.38147.objecter _finish_command 10 = system:0 2024-02-23T00:41:15.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.325+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c06b8c0 osd=5 4 2024-02-23T00:41:15.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.325+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 5 10 2024-02-23T00:41:15.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 10 client.38147.objecter _submit_command 11 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:15.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 11 2024-02-23T00:41:15.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06b8c0 osd=5 3 2024-02-23T00:41:15.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06b8c0 osd=5 3 2024-02-23T00:41:15.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06b8c0 osd=5 3 2024-02-23T00:41:15.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 11 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c06b8c0 osd=5 3 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 5 11 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06b8c0 osd=5 4 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 10 client.38147.objecter _send_command 11 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.340+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd08c06bb50 con 0x7fd08c11bb60 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.342+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd08c06bb50 con 0x7fd08c11bb60 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.342+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 11: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.342+0000 7fd0827fc700 10 client.38147.objecter _finish_command 11 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.342+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c06b8c0 osd=5 4 2024-02-23T00:41:15.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.342+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 5 11 2024-02-23T00:41:15.346 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:41:15.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 10 client.38147.objecter _submit_command 12 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 12 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd08c122470 0x7fd08c124860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06bb50 osd=6 3 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06bb50 osd=6 3 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06bb50 osd=6 3 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 12 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c06bb50 osd=6 3 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 6 12 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06bb50 osd=6 4 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 10 client.38147.objecter _send_command 12 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.171:6808/2509408498 -- command(tid 12: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd08c127670 con 0x7fd08c122470 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd090e2b700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd08c122470 0x7fd08c124860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd0827fc700 10 client.38147.objecter ms_handle_connect 0x7fd08c122470 2024-02-23T00:41:15.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.343+0000 7fd090e2b700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd08c122470 0x7fd08c124860 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:15.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.344+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 12: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd08c127670 con 0x7fd08c122470 2024-02-23T00:41:15.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.344+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 12: 0 ) v1 2024-02-23T00:41:15.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.344+0000 7fd0827fc700 10 client.38147.objecter _finish_command 12 = system:0 2024-02-23T00:41:15.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.344+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c06bb50 osd=6 4 2024-02-23T00:41:15.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.344+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 6 12 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 10 client.38147.objecter _submit_command 13 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 13 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06bb50 osd=6 3 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06bb50 osd=6 3 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c06bb50 osd=6 3 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 13 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c06bb50 osd=6 3 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 6 13 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c06bb50 osd=6 4 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 10 client.38147.objecter _send_command 13 2024-02-23T00:41:15.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.358+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd08c127670 con 0x7fd08c122470 2024-02-23T00:41:15.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd08c127670 con 0x7fd08c122470 2024-02-23T00:41:15.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 13: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:15.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd0827fc700 10 client.38147.objecter _finish_command 13 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c06bb50 osd=6 4 2024-02-23T00:41:15.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 6 13 2024-02-23T00:41:15.362 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:41:15.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 10 client.38147.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:15.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 14 2024-02-23T00:41:15.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6812/373787000 conn(0x7fd08c126970 0x7fd08c126dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:15.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c127670 osd=7 3 2024-02-23T00:41:15.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c127670 osd=7 3 2024-02-23T00:41:15.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c127670 osd=7 3 2024-02-23T00:41:15.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 14 2024-02-23T00:41:15.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c127670 osd=7 3 2024-02-23T00:41:15.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 7 14 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c127670 osd=7 4 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 10 client.38147.objecter _send_command 14 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.360+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.171:6812/373787000 -- command(tid 14: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd08c12aa50 con 0x7fd08c126970 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.361+0000 7fd091e2d700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6812/373787000 conn(0x7fd08c126970 0x7fd08c126dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.361+0000 7fd0827fc700 10 client.38147.objecter ms_handle_connect 0x7fd08c126970 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.361+0000 7fd091e2d700 1 --2- 172.21.15.87:0/2678214715 >> v2:172.21.15.171:6812/373787000 conn(0x7fd08c126970 0x7fd08c126dd0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.362+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 14: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd08c12aa50 con 0x7fd08c126970 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.362+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 14: 0 ) v1 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.362+0000 7fd0827fc700 10 client.38147.objecter _finish_command 14 = system:0 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.362+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c127670 osd=7 4 2024-02-23T00:41:15.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.362+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 7 14 2024-02-23T00:41:15.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 10 client.38147.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:15.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign -1 15 2024-02-23T00:41:15.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c127670 osd=7 3 2024-02-23T00:41:15.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c127670 osd=7 3 2024-02-23T00:41:15.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 20 client.38147.objecter _get_session s=0x7fd08c127670 osd=7 3 2024-02-23T00:41:15.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 15 client.38147.objecter _session_command_op_remove -1 15 2024-02-23T00:41:15.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 20 client.38147.objecter get_session s=0x7fd08c127670 osd=7 3 2024-02-23T00:41:15.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 15 client.38147.objecter _session_command_op_assign 7 15 2024-02-23T00:41:15.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 20 client.38147.objecter put_session s=0x7fd08c127670 osd=7 4 2024-02-23T00:41:15.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 10 client.38147.objecter _send_command 15 2024-02-23T00:41:15.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.376+0000 7fd093890700 1 -- 172.21.15.87:0/2678214715 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd08c124e50 con 0x7fd08c126970 2024-02-23T00:41:15.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.378+0000 7fd0827fc700 1 -- 172.21.15.87:0/2678214715 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd08c124e50 con 0x7fd08c126970 2024-02-23T00:41:15.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.378+0000 7fd0827fc700 10 client.38147.objecter ms_dispatch 0x7fd08c001550 command_reply(tid 15: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:15.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.378+0000 7fd0827fc700 10 client.38147.objecter _finish_command 15 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.378+0000 7fd0827fc700 20 client.38147.objecter put_session s=0x7fd08c127670 osd=7 4 2024-02-23T00:41:15.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:15.378+0000 7fd0827fc700 15 client.38147.objecter _session_command_op_remove 7 15 2024-02-23T00:41:15.380 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:41:15.380 INFO:teuthology.orchestra.run.smithi087.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-02-23T00:41:15.380 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.380 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.380 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.380 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:41:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:13.762266+0000 mgr.y (mgr.24437) 573 : cluster [DBG] pgmap v998: 197 pgs: 2 active+remapped+backfill_toofull, 195 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:15.198477+0000 mon.a (mon.0) 3426 : cluster [DBG] osdmap e656: 8 total, 7 up, 7 in 2024-02-23T00:41:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:13.762266+0000 mgr.y (mgr.24437) 573 : cluster [DBG] pgmap v998: 197 pgs: 2 active+remapped+backfill_toofull, 195 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:15.198477+0000 mon.a (mon.0) 3426 : cluster [DBG] osdmap e656: 8 total, 7 up, 7 in 2024-02-23T00:41:15.686 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:13.762266+0000 mgr.y (mgr.24437) 573 : cluster [DBG] pgmap v998: 197 pgs: 2 active+remapped+backfill_toofull, 195 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:15.198477+0000 mon.a (mon.0) 3426 : cluster [DBG] osdmap e656: 8 total, 7 up, 7 in 2024-02-23T00:41:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:16.833 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[224705]: debug 2024-02-23T00:41:16.603+0000 7fc036b27380 -1 osd.0 424 log_to_monitors {default=true} 2024-02-23T00:41:16.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:15.250700+0000 mon.c (mon.2) 481 : audit [DBG] from='client.? 172.21.15.87:0/2678214715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:16.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:15.763065+0000 mgr.y (mgr.24437) 574 : cluster [DBG] pgmap v1001: 165 pgs: 2 activating, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:16.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:16.201258+0000 mon.a (mon.0) 3427 : cluster [DBG] osdmap e657: 8 total, 7 up, 7 in 2024-02-23T00:41:16.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:16.206696+0000 mon.a (mon.0) 3428 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:15.250700+0000 mon.c (mon.2) 481 : audit [DBG] from='client.? 172.21.15.87:0/2678214715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:15.763065+0000 mgr.y (mgr.24437) 574 : cluster [DBG] pgmap v1001: 165 pgs: 2 activating, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:16.201258+0000 mon.a (mon.0) 3427 : cluster [DBG] osdmap e657: 8 total, 7 up, 7 in 2024-02-23T00:41:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:16.206696+0000 mon.a (mon.0) 3428 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:15.250700+0000 mon.c (mon.2) 481 : audit [DBG] from='client.? 172.21.15.87:0/2678214715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:15.763065+0000 mgr.y (mgr.24437) 574 : cluster [DBG] pgmap v1001: 165 pgs: 2 activating, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:16.201258+0000 mon.a (mon.0) 3427 : cluster [DBG] osdmap e657: 8 total, 7 up, 7 in 2024-02-23T00:41:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:16.206696+0000 mon.a (mon.0) 3428 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-23T00:41:16.961 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-23T00:41:17.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:16.606841+0000 mon.a (mon.0) 3429 : audit [INF] from='osd.0 v2:172.21.15.87:6801/484081831' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-23T00:41:17.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:17.198732+0000 mon.a (mon.0) 3430 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:17.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:17.198882+0000 mon.a (mon.0) 3431 : audit [INF] from='osd.0 v2:172.21.15.87:6801/484081831' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-23T00:41:17.521 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:17.198938+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e658: 8 total, 7 up, 7 in 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:17.203601+0000 mon.a (mon.0) 3433 : audit [INF] from='osd.0 v2:172.21.15.87:6801/484081831' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:17.212377+0000 mon.a (mon.0) 3434 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:17.243119+0000 mon.a (mon.0) 3435 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:16.606841+0000 mon.a (mon.0) 3429 : audit [INF] from='osd.0 v2:172.21.15.87:6801/484081831' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:17.198732+0000 mon.a (mon.0) 3430 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:17.198882+0000 mon.a (mon.0) 3431 : audit [INF] from='osd.0 v2:172.21.15.87:6801/484081831' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:17.198938+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e658: 8 total, 7 up, 7 in 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:17.203601+0000 mon.a (mon.0) 3433 : audit [INF] from='osd.0 v2:172.21.15.87:6801/484081831' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:17.212377+0000 mon.a (mon.0) 3434 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:41:17.522 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:17.243119+0000 mon.a (mon.0) 3435 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:17.833 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:41:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[224705]: debug 2024-02-23T00:41:17.740+0000 7fc02700c700 -1 osd.0 658 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-23T00:41:17.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:16.606841+0000 mon.a (mon.0) 3429 : audit [INF] from='osd.0 v2:172.21.15.87:6801/484081831' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-23T00:41:17.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:17.198732+0000 mon.a (mon.0) 3430 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:17.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:17.198882+0000 mon.a (mon.0) 3431 : audit [INF] from='osd.0 v2:172.21.15.87:6801/484081831' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-23T00:41:17.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:17.198938+0000 mon.a (mon.0) 3432 : cluster [DBG] osdmap e658: 8 total, 7 up, 7 in 2024-02-23T00:41:17.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:17.203601+0000 mon.a (mon.0) 3433 : audit [INF] from='osd.0 v2:172.21.15.87:6801/484081831' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi087", "root=default"]}]: dispatch 2024-02-23T00:41:17.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:17.212377+0000 mon.a (mon.0) 3434 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:41:17.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:17.243119+0000 mon.a (mon.0) 3435 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:18.165 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:18.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:18.270 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:18.301 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:18.321 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:18.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:18.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:18.458 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:17.763617+0000 mgr.y (mgr.24437) 575 : cluster [DBG] pgmap v1004: 196 pgs: 32 unknown, 1 activating, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:17.764459+0000 mon.a (mon.0) 3436 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-23T00:41:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:18.214675+0000 mon.a (mon.0) 3437 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:18.214753+0000 mon.a (mon.0) 3438 : cluster [INF] osd.0 v2:172.21.15.87:6801/484081831 boot 2024-02-23T00:41:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:18.214807+0000 mon.a (mon.0) 3439 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-23T00:41:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:18.214850+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e659: 8 total, 8 up, 7 in 2024-02-23T00:41:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:18.215718+0000 mon.a (mon.0) 3441 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:41:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:18.216864+0000 mon.a (mon.0) 3442 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-112"}]: dispatch 2024-02-23T00:41:18.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:17.763617+0000 mgr.y (mgr.24437) 575 : cluster [DBG] pgmap v1004: 196 pgs: 32 unknown, 1 activating, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:18.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:17.764459+0000 mon.a (mon.0) 3436 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-23T00:41:18.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:18.214675+0000 mon.a (mon.0) 3437 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:18.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:18.214753+0000 mon.a (mon.0) 3438 : cluster [INF] osd.0 v2:172.21.15.87:6801/484081831 boot 2024-02-23T00:41:18.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:18.214807+0000 mon.a (mon.0) 3439 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-23T00:41:18.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:18.214850+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e659: 8 total, 8 up, 7 in 2024-02-23T00:41:18.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:18.215718+0000 mon.a (mon.0) 3441 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:41:18.787 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:18.216864+0000 mon.a (mon.0) 3442 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-112"}]: dispatch 2024-02-23T00:41:18.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:17.763617+0000 mgr.y (mgr.24437) 575 : cluster [DBG] pgmap v1004: 196 pgs: 32 unknown, 1 activating, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:18.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:17.764459+0000 mon.a (mon.0) 3436 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-23T00:41:18.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:18.214675+0000 mon.a (mon.0) 3437 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:18.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:18.214753+0000 mon.a (mon.0) 3438 : cluster [INF] osd.0 v2:172.21.15.87:6801/484081831 boot 2024-02-23T00:41:18.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:18.214807+0000 mon.a (mon.0) 3439 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-23T00:41:18.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:18.214850+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e659: 8 total, 8 up, 7 in 2024-02-23T00:41:18.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:18.215718+0000 mon.a (mon.0) 3441 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-23T00:41:18.788 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:18.216864+0000 mon.a (mon.0) 3442 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-112"}]: dispatch 2024-02-23T00:41:18.962 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:41:19.410 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:19.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:19.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:19.515 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:19.548 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:19.569 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:19.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:19.673 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:19.703 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:19.723 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:19.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:19.827 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:19.855 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:19.874 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:19.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:19.978 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:19.996 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:41:19.997 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:41:20.009 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:20.029 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:20.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:20.034 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:20.133 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:20.160 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:20.259 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:20.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:20.364 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:20.393 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:17.571627+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:20.393 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:17.571821+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:20.393 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:19.217426+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-112"}]': finished 2024-02-23T00:41:20.393 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:19.217511+0000 mon.a (mon.0) 3444 : cluster [DBG] osdmap e660: 8 total, 8 up, 7 in 2024-02-23T00:41:20.393 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:19.220164+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-112", "mode": "writeback"}]: dispatch 2024-02-23T00:41:20.395 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:20.415 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:20.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:20.520 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:20.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:17.571627+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:20.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:17.571821+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:20.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:19.217426+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-112"}]': finished 2024-02-23T00:41:20.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:19.217511+0000 mon.a (mon.0) 3444 : cluster [DBG] osdmap e660: 8 total, 8 up, 7 in 2024-02-23T00:41:20.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:19.220164+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-112", "mode": "writeback"}]: dispatch 2024-02-23T00:41:20.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:17.571627+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:20.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:17.571821+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:20.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:19.217426+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-112"}]': finished 2024-02-23T00:41:20.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:19.217511+0000 mon.a (mon.0) 3444 : cluster [DBG] osdmap e660: 8 total, 8 up, 7 in 2024-02-23T00:41:20.547 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:19.220164+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-112", "mode": "writeback"}]: dispatch 2024-02-23T00:41:20.549 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:20.569 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:20.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:20.673 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:20.700 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:20.719 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:20.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:20.823 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:20.849 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:20.874 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:20.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:20.975 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:21.001 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:21.019 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:21.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:21.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:21.125 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:21.152 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:21.170 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:21.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:21.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:19.764188+0000 mgr.y (mgr.24437) 576 : cluster [DBG] pgmap v1007: 196 pgs: 32 unknown, 1 activating, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:21.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:20.212784+0000 mon.a (mon.0) 3446 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:21.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:20.220008+0000 mon.a (mon.0) 3447 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-112", "mode": "writeback"}]': finished 2024-02-23T00:41:21.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:20.220085+0000 mon.a (mon.0) 3448 : cluster [DBG] osdmap e661: 8 total, 8 up, 7 in 2024-02-23T00:41:21.226 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:20.266859+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:21.274 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:21.307 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:19.764188+0000 mgr.y (mgr.24437) 576 : cluster [DBG] pgmap v1007: 196 pgs: 32 unknown, 1 activating, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:21.308 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:20.212784+0000 mon.a (mon.0) 3446 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:21.308 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:20.220008+0000 mon.a (mon.0) 3447 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-112", "mode": "writeback"}]': finished 2024-02-23T00:41:21.308 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:20.220085+0000 mon.a (mon.0) 3448 : cluster [DBG] osdmap e661: 8 total, 8 up, 7 in 2024-02-23T00:41:21.308 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:20.266859+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:21.309 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:41:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:41:21.309 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:21.328 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:21.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:21.432 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:21.461 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:21.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:19.764188+0000 mgr.y (mgr.24437) 576 : cluster [DBG] pgmap v1007: 196 pgs: 32 unknown, 1 activating, 2 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20/705 objects misplaced (2.837%) 2024-02-23T00:41:21.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:20.212784+0000 mon.a (mon.0) 3446 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:21.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:20.220008+0000 mon.a (mon.0) 3447 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-112", "mode": "writeback"}]': finished 2024-02-23T00:41:21.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:20.220085+0000 mon.a (mon.0) 3448 : cluster [DBG] osdmap e661: 8 total, 8 up, 7 in 2024-02-23T00:41:21.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:20.266859+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:21.649 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.846+0000 7f95ba1a4700 1 -- 172.21.15.87:0/4054046196 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b40f9610 msgr2=0x7f95b4101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.846+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/4054046196 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b40f9610 0x7f95b4101d20 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 rx=0x7f95a8004870 tx=0x7f95a8009cb0).stop 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.846+0000 7f95ba1a4700 1 -- 172.21.15.87:0/4054046196 shutdown_connections 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.846+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/4054046196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95b4102210 0x7f95b4104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.846+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/4054046196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b40f8cb0 0x7f95b40f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.846+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/4054046196 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b40f9610 0x7f95b4101d20 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.846+0000 7f95ba1a4700 1 -- 172.21.15.87:0/4054046196 >> 172.21.15.87:0/4054046196 conn(0x7f95b40f2820 msgr2=0x7f95b40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.846+0000 7f95ba1a4700 1 -- 172.21.15.87:0/4054046196 shutdown_connections 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.846+0000 7f95ba1a4700 1 -- 172.21.15.87:0/4054046196 wait complete. 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 librados: starting msgr at 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 librados: starting objecter 2024-02-23T00:41:21.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 Processor -- start 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 -- start start 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 librados: setting wanted keys 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 librados: calling monclient init 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95b40f8cb0 0x7f95b40f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b40f9610 0x7f95b40f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b4102210 0x7f95b40f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b4116770 con 0x7f95b40f8cb0 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b41168e0 con 0x7f95b40f9610 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95ba1a4700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b4116aa0 con 0x7f95b4102210 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95b3fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b4102210 0x7f95b40f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95b37fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95b40f8cb0 0x7f95b40f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95b3fff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b4102210 0x7f95b40f7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40242/0 (socket says 172.21.15.87:40242) 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95b3fff700 1 -- 172.21.15.87:0/2265305250 learned_addr learned my addr 172.21.15.87:0/2265305250 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:21.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.847+0000 7f95b2ffd700 1 --2- 172.21.15.87:0/2265305250 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b40f9610 0x7f95b40f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:21.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b3fff700 1 -- 172.21.15.87:0/2265305250 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b40f9610 msgr2=0x7f95b40f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:21.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b3fff700 1 --2- 172.21.15.87:0/2265305250 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b40f9610 0x7f95b40f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:21.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b3fff700 1 -- 172.21.15.87:0/2265305250 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95b40f8cb0 msgr2=0x7f95b40f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:21.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b3fff700 1 --2- 172.21.15.87:0/2265305250 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95b40f8cb0 0x7f95b40f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:21.853 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b3fff700 1 -- 172.21.15.87:0/2265305250 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a8005040 con 0x7f95b4102210 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b37fe700 1 --2- 172.21.15.87:0/2265305250 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95b40f8cb0 0x7f95b40f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b3fff700 1 --2- 172.21.15.87:0/2265305250 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b4102210 0x7f95b40f7660 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f95ac00eb70 tx=0x7f95ac00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b0ff9700 10 client.?.objecter ms_handle_connect 0x7f95b4102210 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b0ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b0ff9700 1 -- 172.21.15.87:0/2265305250 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac009910 con 0x7f95b4102210 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b0ff9700 1 -- 172.21.15.87:0/2265305250 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95ac00f040 con 0x7f95b4102210 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95b0ff9700 1 -- 172.21.15.87:0/2265305250 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac007e10 con 0x7f95b4102210 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b4116d20 con 0x7f95b4102210 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95ba1a4700 10 client.38156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b41171c0 con 0x7f95b4102210 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.848+0000 7f95ba1a4700 1 librados: init done 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.849+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b405fa60 con 0x7f95b4102210 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.849+0000 7f95b0ff9700 1 -- 172.21.15.87:0/2265305250 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f95ac004020 con 0x7f95b4102210 2024-02-23T00:41:21.854 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.850+0000 7f95b0ff9700 1 --2- 172.21.15.87:0/2265305250 >> v2:172.21.15.87:6800/1231597463 conn(0x7f959c062c90 0x7f959c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:21.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.850+0000 7f95b37fe700 1 --2- 172.21.15.87:0/2265305250 >> v2:172.21.15.87:6800/1231597463 conn(0x7f959c062c90 0x7f959c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:21.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.850+0000 7f95b0ff9700 1 -- 172.21.15.87:0/2265305250 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(662..662 src has 173..662) v4 ==== 8515+0+0 (secure 0 0 0) 0x7f95ac07d110 con 0x7f95b4102210 2024-02-23T00:41:21.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.850+0000 7f95b0ff9700 10 client.38156.objecter ms_dispatch 0x7f95b4001550 osd_map(662..662 src has 173..662) v4 2024-02-23T00:41:21.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.850+0000 7f95b0ff9700 3 client.38156.objecter handle_osd_map got epochs [662,662] > 0 2024-02-23T00:41:21.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.850+0000 7f95b0ff9700 3 client.38156.objecter handle_osd_map decoding full epoch 662 2024-02-23T00:41:21.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.850+0000 7f95b0ff9700 20 client.38156.objecter dump_active .. 0 homeless 2024-02-23T00:41:21.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.850+0000 7f95b0ff9700 10 client.38156.objecter ms_handle_connect 0x7f959c062c90 2024-02-23T00:41:21.855 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.850+0000 7f95b37fe700 1 --2- 172.21.15.87:0/2265305250 >> v2:172.21.15.87:6800/1231597463 conn(0x7f959c062c90 0x7f959c065150 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f95a40103c0 tx=0x7f95a400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:21.856 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:21.854+0000 7f95b0ff9700 1 -- 172.21.15.87:0/2265305250 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f95ac049f60 con 0x7f95b4102210 2024-02-23T00:41:22.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.034+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f95b4107290 con 0x7f95b4102210 2024-02-23T00:41:22.120 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:41:22.121 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:41:22.121 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:41:22.121 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:41:22.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.198+0000 7f95b0ff9700 1 -- 172.21.15.87:0/2265305250 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v663) v1 ==== 97+0+0 (secure 0 0 0) 0x7f95ac050410 con 0x7f95b4102210 2024-02-23T00:41:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 10 librados: watch_flush enter 2024-02-23T00:41:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 10 librados: watch_flush exit 2024-02-23T00:41:22.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 20 client.38156.objecter shutdown clearing up homeless session... 2024-02-23T00:41:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 10 client.38156.objecter successfully canceled tick 2024-02-23T00:41:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 >> v2:172.21.15.87:6800/1231597463 conn(0x7f959c062c90 msgr2=0x7f959c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/2265305250 >> v2:172.21.15.87:6800/1231597463 conn(0x7f959c062c90 0x7f959c065150 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 rx=0x7f95a40103c0 tx=0x7f95a400b040).stop 2024-02-23T00:41:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b4102210 msgr2=0x7f95b40f7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:22.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/2265305250 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b4102210 0x7f95b40f7660 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 rx=0x7f95ac00eb70 tx=0x7f95ac00b040).stop 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 shutdown_connections 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/2265305250 >> v2:172.21.15.87:6800/1231597463 conn(0x7f959c062c90 0x7f959c065150 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/2265305250 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95b4102210 0x7f95b40f7660 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/2265305250 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95b40f8cb0 0x7f95b40f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 --2- 172.21.15.87:0/2265305250 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95b40f9610 0x7f95b40f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 >> 172.21.15.87:0/2265305250 conn(0x7f95b40f2820 msgr2=0x7f95b40f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.201+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 shutdown_connections 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.202+0000 7f95ba1a4700 1 -- 172.21.15.87:0/2265305250 wait complete. 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:22.202+0000 7f95ba1a4700 1 librados: shutdown 2024-02-23T00:41:22.205 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:41:22.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:22] "GET /metrics HTTP/1.1" 200 179641 "" "Prometheus/2.33.4" 2024-02-23T00:41:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:21.168008+0000 mgr.y (mgr.24437) 577 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:21.220698+0000 mon.a (mon.0) 3450 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:21.220763+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e662: 8 total, 8 up, 7 in 2024-02-23T00:41:22.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:21.221802+0000 mon.a (mon.0) 3452 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112"}]: dispatch 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:21.766131+0000 mon.a (mon.0) 3453 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:22.035968+0000 mon.c (mon.2) 482 : audit [INF] from='client.? 172.21.15.87:0/2265305250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:22.036344+0000 mon.a (mon.0) 3454 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:22.193251+0000 mon.a (mon.0) 3455 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:22.193273+0000 mon.a (mon.0) 3456 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:22.199019+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112"}]': finished 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:22.199091+0000 mon.a (mon.0) 3458 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:22.199144+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:22.199200+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e663: 8 total, 8 up, 7 in 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:21.168008+0000 mgr.y (mgr.24437) 577 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:21.220698+0000 mon.a (mon.0) 3450 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:21.220763+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e662: 8 total, 8 up, 7 in 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:21.221802+0000 mon.a (mon.0) 3452 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112"}]: dispatch 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:21.766131+0000 mon.a (mon.0) 3453 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:22.035968+0000 mon.c (mon.2) 482 : audit [INF] from='client.? 172.21.15.87:0/2265305250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:22.036344+0000 mon.a (mon.0) 3454 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:22.193251+0000 mon.a (mon.0) 3455 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:22.193273+0000 mon.a (mon.0) 3456 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:22.199019+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112"}]': finished 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:22.199091+0000 mon.a (mon.0) 3458 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:22.199144+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:41:22.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:22.199200+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e663: 8 total, 8 up, 7 in 2024-02-23T00:41:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:21.168008+0000 mgr.y (mgr.24437) 577 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:21.220698+0000 mon.a (mon.0) 3450 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:22.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:21.220763+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e662: 8 total, 8 up, 7 in 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:21.221802+0000 mon.a (mon.0) 3452 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112"}]: dispatch 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:21.766131+0000 mon.a (mon.0) 3453 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:22.035968+0000 mon.c (mon.2) 482 : audit [INF] from='client.? 172.21.15.87:0/2265305250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:22.036344+0000 mon.a (mon.0) 3454 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:22.193251+0000 mon.a (mon.0) 3455 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:22.193273+0000 mon.a (mon.0) 3456 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:22.199019+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-112"}]': finished 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:22.199091+0000 mon.a (mon.0) 3458 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:22.199144+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:41:22.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:22.199200+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e663: 8 total, 8 up, 7 in 2024-02-23T00:41:23.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:21.764702+0000 mgr.y (mgr.24437) 578 : cluster [DBG] pgmap v1010: 195 pgs: 1 peering, 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:23.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:23.205666+0000 mon.a (mon.0) 3461 : cluster [DBG] osdmap e664: 8 total, 8 up, 7 in 2024-02-23T00:41:23.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:21.764702+0000 mgr.y (mgr.24437) 578 : cluster [DBG] pgmap v1010: 195 pgs: 1 peering, 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:23.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:23.205666+0000 mon.a (mon.0) 3461 : cluster [DBG] osdmap e664: 8 total, 8 up, 7 in 2024-02-23T00:41:23.943 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:41:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:41:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:21.764702+0000 mgr.y (mgr.24437) 578 : cluster [DBG] pgmap v1010: 195 pgs: 1 peering, 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:23.205666+0000 mon.a (mon.0) 3461 : cluster [DBG] osdmap e664: 8 total, 8 up, 7 in 2024-02-23T00:41:24.152 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:24.613 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:24.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:24.717 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:24.749 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:24.847 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:24.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:24.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:24.951 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:24.983 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:25.004 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:25.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:25.108 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:25.136 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:25.156 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:25.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:25.260 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:25.291 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:23.765400+0000 mgr.y (mgr.24437) 579 : cluster [DBG] pgmap v1013: 163 pgs: 1 peering, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:24.209577+0000 mon.a (mon.0) 3462 : cluster [DBG] osdmap e665: 8 total, 8 up, 7 in 2024-02-23T00:41:25.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:24.212151+0000 mon.a (mon.0) 3463 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:25.543 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:23.765400+0000 mgr.y (mgr.24437) 579 : cluster [DBG] pgmap v1013: 163 pgs: 1 peering, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:25.543 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:24.209577+0000 mon.a (mon.0) 3462 : cluster [DBG] osdmap e665: 8 total, 8 up, 7 in 2024-02-23T00:41:25.544 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:24.212151+0000 mon.a (mon.0) 3463 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:25.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:23.765400+0000 mgr.y (mgr.24437) 579 : cluster [DBG] pgmap v1013: 163 pgs: 1 peering, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s; 20/711 objects misplaced (2.813%) 2024-02-23T00:41:25.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:24.209577+0000 mon.a (mon.0) 3462 : cluster [DBG] osdmap e665: 8 total, 8 up, 7 in 2024-02-23T00:41:25.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:24.212151+0000 mon.a (mon.0) 3463 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:25.640 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:25.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:25.744 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:25.774 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:25.793 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:25.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:25.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:41:25.925 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:26.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.375+0000 7ff009852700 1 -- 172.21.15.87:0/345330978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0040f6a90 msgr2=0x7ff004104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.375+0000 7ff009852700 1 --2- 172.21.15.87:0/345330978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0040f6a90 0x7ff004104a20 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 rx=0x7feffc009c30 tx=0x7feffc004c50).stop 2024-02-23T00:41:26.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.375+0000 7ff009852700 1 -- 172.21.15.87:0/345330978 shutdown_connections 2024-02-23T00:41:26.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.375+0000 7ff009852700 1 --2- 172.21.15.87:0/345330978 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0040f6a90 0x7ff004104a20 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.375+0000 7ff009852700 1 --2- 172.21.15.87:0/345330978 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0040fdbd0 0x7ff0040fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.375+0000 7ff009852700 1 --2- 172.21.15.87:0/345330978 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0040fe530 0x7ff0040f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.375+0000 7ff009852700 1 -- 172.21.15.87:0/345330978 >> 172.21.15.87:0/345330978 conn(0x7ff00406c370 msgr2=0x7ff00406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:26.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.375+0000 7ff009852700 1 -- 172.21.15.87:0/345330978 shutdown_connections 2024-02-23T00:41:26.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.375+0000 7ff009852700 1 -- 172.21.15.87:0/345330978 wait complete. 2024-02-23T00:41:26.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 librados: starting msgr at 2024-02-23T00:41:26.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 librados: starting objecter 2024-02-23T00:41:26.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 Processor -- start 2024-02-23T00:41:26.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 -- start start 2024-02-23T00:41:26.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 librados: setting wanted keys 2024-02-23T00:41:26.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 librados: calling monclient init 2024-02-23T00:41:26.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0040f6a90 0x7ff00410df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0040fdbd0 0x7ff00410e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0040fe530 0x7ff004108120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff004108610 con 0x7ff0040f6a90 2024-02-23T00:41:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff004108780 con 0x7ff0040fdbd0 2024-02-23T00:41:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff009852700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff004108940 con 0x7ff0040fe530 2024-02-23T00:41:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff002ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0040f6a90 0x7ff00410df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff002ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0040f6a90 0x7ff00410df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51162/0 (socket says 172.21.15.87:51162) 2024-02-23T00:41:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff002ffd700 1 -- 172.21.15.87:0/3896029720 learned_addr learned my addr 172.21.15.87:0/3896029720 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff0037fe700 1 --2- 172.21.15.87:0/3896029720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0040fe530 0x7ff004108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.376+0000 7ff0027fc700 1 --2- 172.21.15.87:0/3896029720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0040fdbd0 0x7ff00410e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff002ffd700 1 -- 172.21.15.87:0/3896029720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0040fe530 msgr2=0x7ff004108120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff002ffd700 1 --2- 172.21.15.87:0/3896029720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0040fe530 0x7ff004108120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff002ffd700 1 -- 172.21.15.87:0/3896029720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0040fdbd0 msgr2=0x7ff00410e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff002ffd700 1 --2- 172.21.15.87:0/3896029720 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff0040fdbd0 0x7ff00410e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff002ffd700 1 -- 172.21.15.87:0/3896029720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff4005040 con 0x7ff0040f6a90 2024-02-23T00:41:26.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff0037fe700 1 --2- 172.21.15.87:0/3896029720 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff0040fe530 0x7ff004108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:26.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff002ffd700 1 --2- 172.21.15.87:0/3896029720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0040f6a90 0x7ff00410df80 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7feff80041d0 tx=0x7feff8009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7fefebfff700 10 client.?.objecter ms_handle_connect 0x7ff0040f6a90 2024-02-23T00:41:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7fefebfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff8007b40 con 0x7ff0040f6a90 2024-02-23T00:41:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc0097c0 con 0x7ff0040f6a90 2024-02-23T00:41:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feff800f040 con 0x7ff0040f6a90 2024-02-23T00:41:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff8014730 con 0x7ff0040f6a90 2024-02-23T00:41:26.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff009852700 10 client.27690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff004114db0 con 0x7ff0040f6a90 2024-02-23T00:41:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.377+0000 7ff009852700 1 librados: init done 2024-02-23T00:41:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.378+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff00405f970 con 0x7ff0040f6a90 2024-02-23T00:41:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.378+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7feff8009110 con 0x7ff0040f6a90 2024-02-23T00:41:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7fefebfff700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefec062c90 0x7fefec065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.315 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(666..666 src has 173..666) v4 ==== 8520+0+0 (secure 0 0 0) 0x7feff807d400 con 0x7ff0040f6a90 2024-02-23T00:41:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 osd_map(666..666 src has 173..666) v4 2024-02-23T00:41:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7fefebfff700 3 client.27690.objecter handle_osd_map got epochs [666,666] > 0 2024-02-23T00:41:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7fefebfff700 3 client.27690.objecter handle_osd_map decoding full epoch 666 2024-02-23T00:41:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff0027fc700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefec062c90 0x7fefec065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7fefebfff700 20 client.27690.objecter dump_active .. 0 homeless 2024-02-23T00:41:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v666) v1 ==== 54+0+16 (secure 0 0 0) 0x7feff8084070 con 0x7ff0040f6a90 2024-02-23T00:41:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7fefebfff700 10 client.27690.objecter ms_handle_connect 0x7fefec062c90 2024-02-23T00:41:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff0027fc700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefec062c90 0x7fefec065150 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7ff0041098c0 tx=0x7feff400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 10 client.27690.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 1 2024-02-23T00:41:26.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6801/484081831 conn(0x7ff0040feee0 0x7ff0040ff2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff0041070d0 osd=0 3 2024-02-23T00:41:26.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0041070d0 osd=0 3 2024-02-23T00:41:26.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff0041070d0 osd=0 3 2024-02-23T00:41:26.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 1 2024-02-23T00:41:26.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff0041070d0 osd=0 3 2024-02-23T00:41:26.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 0 1 2024-02-23T00:41:26.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0041070d0 osd=0 4 2024-02-23T00:41:26.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 10 client.27690.objecter _send_command 1 2024-02-23T00:41:26.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0040ff7b0 con 0x7ff0040feee0 2024-02-23T00:41:26.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff0037fe700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6801/484081831 conn(0x7ff0040feee0 0x7ff0040ff2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7fefebfff700 10 client.27690.objecter ms_handle_connect 0x7ff0040feee0 2024-02-23T00:41:26.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.379+0000 7ff0037fe700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6801/484081831 conn(0x7ff0040feee0 0x7ff0040ff2c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.380+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff0040ff7b0 con 0x7ff0040feee0 2024-02-23T00:41:26.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.380+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:41:26.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.380+0000 7fefebfff700 10 client.27690.objecter _finish_command 1 = system:0 2024-02-23T00:41:26.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.380+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff0041070d0 osd=0 4 2024-02-23T00:41:26.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.380+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 0 1 2024-02-23T00:41:26.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 10 client.27690.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:26.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 2 2024-02-23T00:41:26.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff0041070d0 osd=0 3 2024-02-23T00:41:26.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0041070d0 osd=0 3 2024-02-23T00:41:26.322 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:41:26.322 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:41:26.322 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:41:26.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff0041070d0 osd=0 3 2024-02-23T00:41:26.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 2 2024-02-23T00:41:26.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff0041070d0 osd=0 3 2024-02-23T00:41:26.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 0 2 2024-02-23T00:41:26.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0041070d0 osd=0 4 2024-02-23T00:41:26.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 10 client.27690.objecter _send_command 2 2024-02-23T00:41:26.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.395+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff0040ff980 con 0x7ff0040feee0 2024-02-23T00:41:26.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.396+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff0040ff980 con 0x7ff0040feee0 2024-02-23T00:41:26.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.396+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:26.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.396+0000 7fefebfff700 10 client.27690.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.396+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff0041070d0 osd=0 4 2024-02-23T00:41:26.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.396+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 0 2 2024-02-23T00:41:26.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 10 client.27690.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:26.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 3 2024-02-23T00:41:26.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6805/685354171 conn(0x7ff004028540 0x7ff004118f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff0040f6290 osd=1 3 2024-02-23T00:41:26.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0040f6290 osd=1 3 2024-02-23T00:41:26.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff0040f6290 osd=1 3 2024-02-23T00:41:26.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 3 2024-02-23T00:41:26.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff0040f6290 osd=1 3 2024-02-23T00:41:26.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 1 3 2024-02-23T00:41:26.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0040f6290 osd=1 4 2024-02-23T00:41:26.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 10 client.27690.objecter _send_command 3 2024-02-23T00:41:26.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0040f6400 con 0x7ff004028540 2024-02-23T00:41:26.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff002ffd700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6805/685354171 conn(0x7ff004028540 0x7ff004118f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7ff002ffd700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6805/685354171 conn(0x7ff004028540 0x7ff004118f60 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7fefebfff700 10 client.27690.objecter ms_handle_connect 0x7ff004028540 2024-02-23T00:41:26.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff0040f6400 con 0x7ff004028540 2024-02-23T00:41:26.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:41:26.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7fefebfff700 10 client.27690.objecter _finish_command 3 = system:0 2024-02-23T00:41:26.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff0040f6290 osd=1 4 2024-02-23T00:41:26.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.397+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 1 3 2024-02-23T00:41:26.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 10 client.27690.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:26.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 4 2024-02-23T00:41:26.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff0040f6290 osd=1 3 2024-02-23T00:41:26.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0040f6290 osd=1 3 2024-02-23T00:41:26.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff0040f6290 osd=1 3 2024-02-23T00:41:26.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 4 2024-02-23T00:41:26.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff0040f6290 osd=1 3 2024-02-23T00:41:26.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 1 4 2024-02-23T00:41:26.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0040f6290 osd=1 4 2024-02-23T00:41:26.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 10 client.27690.objecter _send_command 4 2024-02-23T00:41:26.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.412+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff00405a040 con 0x7ff004028540 2024-02-23T00:41:26.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff00405a040 con 0x7ff004028540 2024-02-23T00:41:26.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:26.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7fefebfff700 10 client.27690.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff0040f6290 osd=1 4 2024-02-23T00:41:26.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 1 4 2024-02-23T00:41:26.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 10 client.27690.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:26.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 5 2024-02-23T00:41:26.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff004027b90 0x7ff004027f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00405ee80 osd=2 3 2024-02-23T00:41:26.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00405ee80 osd=2 3 2024-02-23T00:41:26.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00405ee80 osd=2 3 2024-02-23T00:41:26.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 5 2024-02-23T00:41:26.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff00405ee80 osd=2 3 2024-02-23T00:41:26.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 2 5 2024-02-23T00:41:26.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00405ee80 osd=2 4 2024-02-23T00:41:26.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 10 client.27690.objecter _send_command 5 2024-02-23T00:41:26.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff00411ed40 con 0x7ff004027b90 2024-02-23T00:41:26.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff0027fc700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff004027b90 0x7ff004027f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7fefebfff700 10 client.27690.objecter ms_handle_connect 0x7ff004027b90 2024-02-23T00:41:26.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.414+0000 7ff0027fc700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff004027b90 0x7ff004027f90 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.415+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff00411ed40 con 0x7ff004027b90 2024-02-23T00:41:26.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.415+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:41:26.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.415+0000 7fefebfff700 10 client.27690.objecter _finish_command 5 = system:0 2024-02-23T00:41:26.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.415+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff00405ee80 osd=2 4 2024-02-23T00:41:26.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.415+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 2 5 2024-02-23T00:41:26.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 10 client.27690.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:26.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 6 2024-02-23T00:41:26.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00405ee80 osd=2 3 2024-02-23T00:41:26.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00405ee80 osd=2 3 2024-02-23T00:41:26.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00405ee80 osd=2 3 2024-02-23T00:41:26.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 6 2024-02-23T00:41:26.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff00405ee80 osd=2 3 2024-02-23T00:41:26.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 2 6 2024-02-23T00:41:26.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00405ee80 osd=2 4 2024-02-23T00:41:26.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 10 client.27690.objecter _send_command 6 2024-02-23T00:41:26.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.429+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff004027600 con 0x7ff004027b90 2024-02-23T00:41:26.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.430+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff004027600 con 0x7ff004027b90 2024-02-23T00:41:26.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:26.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7fefebfff700 10 client.27690.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff00405ee80 osd=2 4 2024-02-23T00:41:26.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 2 6 2024-02-23T00:41:26.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 10 client.27690.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:26.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 7 2024-02-23T00:41:26.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6813/128059790 conn(0x7ff00411d760 0x7ff00411dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004119c20 osd=3 3 2024-02-23T00:41:26.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004119c20 osd=3 3 2024-02-23T00:41:26.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004119c20 osd=3 3 2024-02-23T00:41:26.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 7 2024-02-23T00:41:26.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff004119c20 osd=3 3 2024-02-23T00:41:26.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 3 7 2024-02-23T00:41:26.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004119c20 osd=3 4 2024-02-23T00:41:26.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 10 client.27690.objecter _send_command 7 2024-02-23T00:41:26.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff00411a3c0 con 0x7ff00411d760 2024-02-23T00:41:26.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff0037fe700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6813/128059790 conn(0x7ff00411d760 0x7ff00411dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7fefebfff700 10 client.27690.objecter ms_handle_connect 0x7ff00411d760 2024-02-23T00:41:26.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.431+0000 7ff0037fe700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6813/128059790 conn(0x7ff00411d760 0x7ff00411dba0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.432+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff00411a3c0 con 0x7ff00411d760 2024-02-23T00:41:26.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.432+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:41:26.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.432+0000 7fefebfff700 10 client.27690.objecter _finish_command 7 = system:0 2024-02-23T00:41:26.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.432+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff004119c20 osd=3 4 2024-02-23T00:41:26.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.432+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 3 7 2024-02-23T00:41:26.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 10 client.27690.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:26.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 8 2024-02-23T00:41:26.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004119c20 osd=3 3 2024-02-23T00:41:26.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004119c20 osd=3 3 2024-02-23T00:41:26.343 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:41:26.343 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:41:26.343 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:41:26.344 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:41:26.344 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:41:26.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004119c20 osd=3 3 2024-02-23T00:41:26.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 8 2024-02-23T00:41:26.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff004119c20 osd=3 3 2024-02-23T00:41:26.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 3 8 2024-02-23T00:41:26.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004119c20 osd=3 4 2024-02-23T00:41:26.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 10 client.27690.objecter _send_command 8 2024-02-23T00:41:26.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.446+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff00411e890 con 0x7ff00411d760 2024-02-23T00:41:26.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.447+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff00411e890 con 0x7ff00411d760 2024-02-23T00:41:26.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.447+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:26.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.447+0000 7fefebfff700 10 client.27690.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.447+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff004119c20 osd=3 4 2024-02-23T00:41:26.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.447+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 3 8 2024-02-23T00:41:26.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 10 client.27690.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:26.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 9 2024-02-23T00:41:26.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff004120ec0 0x7ff0041232b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00411e890 osd=4 3 2024-02-23T00:41:26.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411e890 osd=4 3 2024-02-23T00:41:26.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00411e890 osd=4 3 2024-02-23T00:41:26.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 9 2024-02-23T00:41:26.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff00411e890 osd=4 3 2024-02-23T00:41:26.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 4 9 2024-02-23T00:41:26.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411e890 osd=4 4 2024-02-23T00:41:26.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 10 client.27690.objecter _send_command 9 2024-02-23T00:41:26.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff00411eb30 con 0x7ff004120ec0 2024-02-23T00:41:26.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff002ffd700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff004120ec0 0x7ff0041232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7fefebfff700 10 client.27690.objecter ms_handle_connect 0x7ff004120ec0 2024-02-23T00:41:26.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.448+0000 7ff002ffd700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff004120ec0 0x7ff0041232b0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.449+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff00411eb30 con 0x7ff004120ec0 2024-02-23T00:41:26.350 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.449+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:41:26.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.449+0000 7fefebfff700 10 client.27690.objecter _finish_command 9 = system:0 2024-02-23T00:41:26.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.449+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff00411e890 osd=4 4 2024-02-23T00:41:26.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.449+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 4 9 2024-02-23T00:41:26.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 10 client.27690.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:26.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 10 2024-02-23T00:41:26.351 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00411e890 osd=4 3 2024-02-23T00:41:26.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411e890 osd=4 3 2024-02-23T00:41:26.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00411e890 osd=4 3 2024-02-23T00:41:26.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 10 2024-02-23T00:41:26.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff00411e890 osd=4 3 2024-02-23T00:41:26.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 4 10 2024-02-23T00:41:26.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411e890 osd=4 4 2024-02-23T00:41:26.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 10 client.27690.objecter _send_command 10 2024-02-23T00:41:26.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.464+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff00411eb30 con 0x7ff004120ec0 2024-02-23T00:41:26.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.465+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff00411eb30 con 0x7ff004120ec0 2024-02-23T00:41:26.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.465+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:26.353 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.465+0000 7fefebfff700 10 client.27690.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.465+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff00411e890 osd=4 4 2024-02-23T00:41:26.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.465+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 4 10 2024-02-23T00:41:26.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 10 client.27690.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:26.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 11 2024-02-23T00:41:26.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff004123cb0 0x7ff004124110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00411eb30 osd=5 3 2024-02-23T00:41:26.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411eb30 osd=5 3 2024-02-23T00:41:26.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00411eb30 osd=5 3 2024-02-23T00:41:26.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 11 2024-02-23T00:41:26.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff00411eb30 osd=5 3 2024-02-23T00:41:26.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 5 11 2024-02-23T00:41:26.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411eb30 osd=5 4 2024-02-23T00:41:26.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 10 client.27690.objecter _send_command 11 2024-02-23T00:41:26.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff004125d60 con 0x7ff004123cb0 2024-02-23T00:41:26.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff0027fc700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff004123cb0 0x7ff004124110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7fefebfff700 10 client.27690.objecter ms_handle_connect 0x7ff004123cb0 2024-02-23T00:41:26.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.466+0000 7ff0027fc700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff004123cb0 0x7ff004124110 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.467+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff004125d60 con 0x7ff004123cb0 2024-02-23T00:41:26.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.467+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:41:26.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.467+0000 7fefebfff700 10 client.27690.objecter _finish_command 11 = system:0 2024-02-23T00:41:26.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.467+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff00411eb30 osd=5 4 2024-02-23T00:41:26.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.467+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 5 11 2024-02-23T00:41:26.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 10 client.27690.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:26.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 12 2024-02-23T00:41:26.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00411eb30 osd=5 3 2024-02-23T00:41:26.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411eb30 osd=5 3 2024-02-23T00:41:26.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff00411eb30 osd=5 3 2024-02-23T00:41:26.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 12 2024-02-23T00:41:26.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff00411eb30 osd=5 3 2024-02-23T00:41:26.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 5 12 2024-02-23T00:41:26.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411eb30 osd=5 4 2024-02-23T00:41:26.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 10 client.27690.objecter _send_command 12 2024-02-23T00:41:26.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.481+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff004125d60 con 0x7ff004123cb0 2024-02-23T00:41:26.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff004125d60 con 0x7ff004123cb0 2024-02-23T00:41:26.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:26.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7fefebfff700 10 client.27690.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff00411eb30 osd=5 4 2024-02-23T00:41:26.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 5 12 2024-02-23T00:41:26.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 10 client.27690.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:26.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 13 2024-02-23T00:41:26.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff0041284a0 0x7ff004128900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004125d60 osd=6 3 2024-02-23T00:41:26.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004125d60 osd=6 3 2024-02-23T00:41:26.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004125d60 osd=6 3 2024-02-23T00:41:26.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 13 2024-02-23T00:41:26.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff004125d60 osd=6 3 2024-02-23T00:41:26.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 6 13 2024-02-23T00:41:26.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004125d60 osd=6 4 2024-02-23T00:41:26.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 10 client.27690.objecter _send_command 13 2024-02-23T00:41:26.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff004128fa0 con 0x7ff0041284a0 2024-02-23T00:41:26.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.483+0000 7ff0037fe700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff0041284a0 0x7ff004128900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.484+0000 7fefebfff700 10 client.27690.objecter ms_handle_connect 0x7ff0041284a0 2024-02-23T00:41:26.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.484+0000 7ff0037fe700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff0041284a0 0x7ff004128900 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.485+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff004128fa0 con 0x7ff0041284a0 2024-02-23T00:41:26.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.485+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:41:26.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.485+0000 7fefebfff700 10 client.27690.objecter _finish_command 13 = system:0 2024-02-23T00:41:26.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.485+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff004125d60 osd=6 4 2024-02-23T00:41:26.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.485+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 6 13 2024-02-23T00:41:26.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 10 client.27690.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:26.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 14 2024-02-23T00:41:26.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004125d60 osd=6 3 2024-02-23T00:41:26.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004125d60 osd=6 3 2024-02-23T00:41:26.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004125d60 osd=6 3 2024-02-23T00:41:26.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 14 2024-02-23T00:41:26.365 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff004125d60 osd=6 3 2024-02-23T00:41:26.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 6 14 2024-02-23T00:41:26.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004125d60 osd=6 4 2024-02-23T00:41:26.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 10 client.27690.objecter _send_command 14 2024-02-23T00:41:26.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.499+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff004128fa0 con 0x7ff0041284a0 2024-02-23T00:41:26.366 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff004128fa0 con 0x7ff0041284a0 2024-02-23T00:41:26.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:26.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7fefebfff700 10 client.27690.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff004125d60 osd=6 4 2024-02-23T00:41:26.367 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 6 14 2024-02-23T00:41:26.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 10 client.27690.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:26.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 15 2024-02-23T00:41:26.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6812/373787000 conn(0x7ff00412b6b0 0x7ff00412bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:26.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004128fa0 osd=7 3 2024-02-23T00:41:26.368 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004128fa0 osd=7 3 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004128fa0 osd=7 3 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 15 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff004128fa0 osd=7 3 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 7 15 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004128fa0 osd=7 4 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 10 client.27690.objecter _send_command 15 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff00412c130 con 0x7ff00412b6b0 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff002ffd700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6812/373787000 conn(0x7ff00412b6b0 0x7ff00412bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7fefebfff700 10 client.27690.objecter ms_handle_connect 0x7ff00412b6b0 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.502+0000 7ff002ffd700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6812/373787000 conn(0x7ff00412b6b0 0x7ff00412bb10 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.503+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff00412c130 con 0x7ff00412b6b0 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.503+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:41:26.369 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.503+0000 7fefebfff700 10 client.27690.objecter _finish_command 15 = system:0 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.503+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff004128fa0 osd=7 4 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.503+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 7 15 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 10 client.27690.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign -1 16 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004128fa0 osd=7 3 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004128fa0 osd=7 3 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 20 client.27690.objecter _get_session s=0x7ff004128fa0 osd=7 3 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 15 client.27690.objecter _session_command_op_remove -1 16 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 20 client.27690.objecter get_session s=0x7ff004128fa0 osd=7 3 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 15 client.27690.objecter _session_command_op_assign 7 16 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004128fa0 osd=7 4 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 10 client.27690.objecter _send_command 16 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.517+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff00412c130 con 0x7ff00412b6b0 2024-02-23T00:41:26.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7fefebfff700 1 -- 172.21.15.87:0/3896029720 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7ff00412c130 con 0x7ff00412b6b0 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7fefebfff700 10 client.27690.objecter ms_dispatch 0x7ff004001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7fefebfff700 10 client.27690.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7fefebfff700 20 client.27690.objecter put_session s=0x7ff004128fa0 osd=7 4 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7fefebfff700 15 client.27690.objecter _session_command_op_remove 7 16 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 librados: watch_flush enter 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 librados: watch_flush exit 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 client.27690.objecter close_session for osd.0 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6801/484081831 conn(0x7ff0040feee0 msgr2=0x7ff0040ff2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6801/484081831 conn(0x7ff0040feee0 0x7ff0040ff2c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0041070d0 osd=0 1 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 client.27690.objecter close_session for osd.1 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6805/685354171 conn(0x7ff004028540 msgr2=0x7ff004118f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6805/685354171 conn(0x7ff004028540 0x7ff004118f60 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff0040f6290 osd=1 1 2024-02-23T00:41:26.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 client.27690.objecter close_session for osd.2 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff004027b90 msgr2=0x7ff004027f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff004027b90 0x7ff004027f90 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00405ee80 osd=2 1 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 client.27690.objecter close_session for osd.3 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6813/128059790 conn(0x7ff00411d760 msgr2=0x7ff00411dba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6813/128059790 conn(0x7ff00411d760 0x7ff00411dba0 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff0037fe700 1 -- 172.21.15.87:0/3896029720 reap_dead start 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004119c20 osd=3 1 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 client.27690.objecter close_session for osd.4 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff004120ec0 msgr2=0x7ff0041232b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff004120ec0 0x7ff0041232b0 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411e890 osd=4 1 2024-02-23T00:41:26.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 client.27690.objecter close_session for osd.5 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff0037fe700 1 -- 172.21.15.87:0/3896029720 reap_dead start 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff004123cb0 msgr2=0x7ff004124110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff004123cb0 0x7ff004124110 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff00411eb30 osd=5 1 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 client.27690.objecter close_session for osd.6 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff0041284a0 msgr2=0x7ff004128900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff0041284a0 0x7ff004128900 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004125d60 osd=6 1 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 client.27690.objecter close_session for osd.7 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6812/373787000 conn(0x7ff00412b6b0 msgr2=0x7ff00412bb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.171:6812/373787000 conn(0x7ff00412b6b0 0x7ff00412bb10 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 20 client.27690.objecter put_session s=0x7ff004128fa0 osd=7 1 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 20 client.27690.objecter shutdown clearing up homeless session... 2024-02-23T00:41:26.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 10 client.27690.objecter successfully canceled tick 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefec062c90 msgr2=0x7fefec065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> v2:172.21.15.87:6800/1231597463 conn(0x7fefec062c90 0x7fefec065150 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 rx=0x7ff0041098c0 tx=0x7feff400b040).stop 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0040f6a90 msgr2=0x7ff00410df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 --2- 172.21.15.87:0/3896029720 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff0040f6a90 0x7ff00410df80 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 rx=0x7feff80041d0 tx=0x7feff8009f60).stop 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff0037fe700 1 -- 172.21.15.87:0/3896029720 reap_dead start 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 shutdown_connections 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 >> 172.21.15.87:0/3896029720 conn(0x7ff00406c370 msgr2=0x7ff0040f5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 shutdown_connections 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 -- 172.21.15.87:0/3896029720 wait complete. 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:25.519+0000 7ff009852700 1 librados: shutdown 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.374 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.375 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.375 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.375 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:41:26.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:25.216164+0000 mon.a (mon.0) 3464 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:26.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:25.216354+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e666: 8 total, 8 up, 7 in 2024-02-23T00:41:26.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:25.380175+0000 mon.a (mon.0) 3466 : audit [DBG] from='client.? 172.21.15.87:0/3896029720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:26.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:25.216164+0000 mon.a (mon.0) 3464 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:26.573 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:25.216354+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e666: 8 total, 8 up, 7 in 2024-02-23T00:41:26.573 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:25.380175+0000 mon.a (mon.0) 3466 : audit [DBG] from='client.? 172.21.15.87:0/3896029720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:26.573 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:41:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:26.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:41:26.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:41:26.686 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:26.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:25.216164+0000 mon.a (mon.0) 3464 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:26.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:25.216354+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e666: 8 total, 8 up, 7 in 2024-02-23T00:41:26.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:25.380175+0000 mon.a (mon.0) 3466 : audit [DBG] from='client.? 172.21.15.87:0/3896029720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:26.712 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:27.391 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:25.766281+0000 mgr.y (mgr.24437) 580 : cluster [DBG] pgmap v1016: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 10/723 objects misplaced (1.383%); 27 B/s, 1 objects/s recovering 2024-02-23T00:41:27.391 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:26.224342+0000 mon.a (mon.0) 3467 : cluster [DBG] osdmap e667: 8 total, 8 up, 7 in 2024-02-23T00:41:27.391 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:26.237987+0000 mon.a (mon.0) 3468 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:27.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:25.766281+0000 mgr.y (mgr.24437) 580 : cluster [DBG] pgmap v1016: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 10/723 objects misplaced (1.383%); 27 B/s, 1 objects/s recovering 2024-02-23T00:41:27.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:26.224342+0000 mon.a (mon.0) 3467 : cluster [DBG] osdmap e667: 8 total, 8 up, 7 in 2024-02-23T00:41:27.391 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:26.237987+0000 mon.a (mon.0) 3468 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:27.440 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:27.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:27.544 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:27.573 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:25.766281+0000 mgr.y (mgr.24437) 580 : cluster [DBG] pgmap v1016: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 10/723 objects misplaced (1.383%); 27 B/s, 1 objects/s recovering 2024-02-23T00:41:27.574 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:26.224342+0000 mon.a (mon.0) 3467 : cluster [DBG] osdmap e667: 8 total, 8 up, 7 in 2024-02-23T00:41:27.574 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:26.237987+0000 mon.a (mon.0) 3468 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:27.576 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:27.598 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:27.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:27.643 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:41:27.643 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:41:27.643 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:41:27.643 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:41:27.643 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.643 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.7.0:204 63.13 e139/114 63:cacde8f7:test-rados-api-smithi087-82091-12::bar:2 v 139'5, mlcod=134'1)", 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.059269+0000", 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 597.58024844399995, 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026021935, 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059269+0000", 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.644 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059269+0000", 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059268+0000", 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2863e-05 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059281+0000", 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5006e-05 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.645 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059296+0000", 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9260000000000003e-06 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.059303+0000", 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00091365900000000004 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.060217+0000", 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0594999999999998e-05 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.646 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.060237+0000", 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025023361000000001 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085261+0000", 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0518999999999997e-05 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.085291+0000", 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026021935 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.647 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.1.0:298 86.13 e139/133 86:cacde8f7:test-rados-api-smithi087-82091-12::bar:2 v 139'4, mlcod=138'1)", 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 597.55552049400001, 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036243368999999998, 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.648 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.083997+0000", 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1370000000000008e-06 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084006+0000", 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4387e-05 2024-02-23T00:41:27.649 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084020+0000", 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5240000000000003e-06 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084028+0000", 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9998e-05 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084128+0000", 2024-02-23T00:41:27.650 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3402000000000001e-05 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.084151+0000", 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036055049999999998 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.120206+0000", 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4238e-05 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:30.120241+0000", 2024-02-23T00:41:27.651 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.036243368999999998 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:369 63.6 63:6271ce74:test-rados-api-smithi087-82091-14::foo0:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e144)", 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:34.984178+0000", 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 592.65533989400001, 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040494046999999998, 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:27.652 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 369 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984178+0000", 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984178+0000", 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.653 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984176+0000", 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0913e-05 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:34.984187+0000", 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024830016 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.009017+0000", 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.131e-05 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.654 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.009028+0000", 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012722213 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.021750+0000", 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000138596 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.021889+0000", 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016801399999999999 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,6", 2024-02-23T00:41:27.655 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.022057+0000", 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001482412 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.023539+0000", 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00053109800000000003 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024070+0000", 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00050785999999999995 2024-02-23T00:41:27.656 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024578+0000", 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1844e-05 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024630+0000", 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1841999999999997e-05 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.657 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:35.024672+0000", 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.040494046999999998 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:578 63.a e146/114 63:54d171db:test-rados-api-smithi087-82091-14::foo7:head v 146'5, mlcod=144'1)", 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:31:37.023058+0000", 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 590.61645963800004, 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044013903, 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.658 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023058+0000", 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023058+0000", 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023056+0000", 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5184000000000001e-05 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.659 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023071+0000", 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8630999999999998e-05 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023090+0000", 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6339999999999996e-06 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023100+0000", 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1316999999999997e-05 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.660 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023151+0000", 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9091999999999999e-05 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.023170+0000", 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.043871062000000002 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.067041+0000", 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0898999999999997e-05 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:31:37.067072+0000", 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044013903 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.26224.0:1810 63.8 e216/114 63:15d27fd9:test-rados-api-smithi087-82091-22::foo:head v 216'69, mlcod=198'66)", 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 520.612955693, 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.032422279999999998, 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026562+0000", 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1569999999999999e-06 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.663 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026567+0000", 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3954000000000001e-05 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026581+0000", 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1770000000000003e-06 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.026587+0000", 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000565939 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.664 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027153+0000", 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0520999999999999e-05 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.027173+0000", 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.031782471 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.058956+0000", 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.8521e-05 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.665 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.058984+0000", 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.032422279999999998 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26739.0:1 159.6s0 159:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e233)", 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:04.031498+0000", 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 503.60801946999999, 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0251979280000001, 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:27.666 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26739", 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1200734777", 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031498+0000", 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031498+0000", 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031497+0000", 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5536000000000002e-05 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031512+0000", 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0933e-05 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031523+0000", 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6389999999999992e-06 2024-02-23T00:41:27.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031531+0000", 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3112999999999998e-05 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031584+0000", 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.066e-06 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:04.031587+0000", 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0198177390000001 2024-02-23T00:41:27.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.051405+0000", 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000104659 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.051509+0000", 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00031744400000000001 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.051827+0000", 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000397279 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.052224+0000", 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0044106010000000001 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.056635+0000", 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1668000000000007e-05 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:05.056696+0000", 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0251979280000001 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.671 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37253.0:1 170.6s0 170:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e254)", 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:25.048138+0000", 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 482.59137990400001, 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.437999077, 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37253", 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/1057868100", 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.672 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048138+0000", 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048138+0000", 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048136+0000", 2024-02-23T00:41:27.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7890000000000006e-06 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048146+0000", 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7639999999999994e-06 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048155+0000", 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5249999999999996e-06 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048160+0000", 2024-02-23T00:41:27.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9122999999999997e-05 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048209+0000", 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7929999999999999e-06 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:25.048213+0000", 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.436081875 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.484295+0000", 2024-02-23T00:41:27.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011504499999999999 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.484410+0000", 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00043144499999999997 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.484842+0000", 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00066020200000000003 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.485502+0000", 2024-02-23T00:41:27.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00059683000000000002 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.486099+0000", 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8315000000000002e-05 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:26.486137+0000", 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.437999077 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.677 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.4.31:55 216.9 216:939aa6ec:test-rados-api-smithi087-82091-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e451)", 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:37:17.129723+0000", 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 250.509795209, 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.057627593999999997, 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129723+0000", 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129723+0000", 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:41:27.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129720+0000", 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2629999999999999e-05 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129733+0000", 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9596e-05 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129752+0000", 2024-02-23T00:41:27.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8230000000000006e-06 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129762+0000", 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4041000000000005e-05 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129836+0000", 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000126854 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.680 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129963+0000", 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00019699 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130160+0000", 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00052119500000000001 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130681+0000", 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00031204600000000001 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130993+0000", 2024-02-23T00:41:27.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056278791000000002 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187272+0000", 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3816000000000002e-05 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187316+0000", 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4229e-05 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187350+0000", 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.057627593999999997 2024-02-23T00:41:27.682 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.6.44:88 63.6 63:6735b1f6:test-rados-api-smithi087-82091-78::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:38:20.305065+0000", 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 187.334452803, 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024404019999999998, 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305065+0000", 2024-02-23T00:41:27.683 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305065+0000", 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305063+0000", 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6510000000000001e-06 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305071+0000", 2024-02-23T00:41:27.684 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6728e-05 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305088+0000", 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5950000000000001e-06 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305095+0000", 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1520000000000002e-05 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305137+0000", 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4927999999999996e-05 2024-02-23T00:41:27.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305232+0000", 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00017876200000000001 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 6", 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305410+0000", 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000352796 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305763+0000", 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023542632000000001 2024-02-23T00:41:27.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.329306+0000", 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4443000000000006e-05 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.329370+0000", 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8653999999999994e-05 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.329469+0000", 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024404019999999998 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.687 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2034 5.d 5:b4812045:::notify.4:head [watch ping cookie 94532396651008] snapc 0=[] ondisk+write+known_if_redirected e642)", 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:05.492045+0000", 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 22.147472797999999, 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00043935799999999999, 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2034 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.688 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:05.492045+0000", 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:05.492045+0000", 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999967 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:05.492041+0000", 2024-02-23T00:41:27.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8609000000000001e-05 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:05.492060+0000", 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8583e-05 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:05.492099+0000", 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9519000000000001e-05 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:05.492118+0000", 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016777799999999999 2024-02-23T00:41:27.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:05.492286+0000", 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2163999999999994e-05 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:05.492378+0000", 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00010623100000000001 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:05.492484+0000", 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00043935799999999999 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2043 5.d 5:b4812045:::notify.4:head [watch ping cookie 94532396651008] snapc 0=[] ondisk+write+known_if_redirected e646)", 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:10.492537+0000", 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 17.146980945999999, 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00033707700000000003, 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:27.692 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2043 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:10.492537+0000", 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:10.492537+0000", 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:10.492535+0000", 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1143e-05 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:10.492546+0000", 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.389e-05 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:10.492570+0000", 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6970000000000005e-06 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:10.492579+0000", 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000102821 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:10.492682+0000", 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3998000000000005e-05 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:10.492766+0000", 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00010785699999999999 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.695 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:10.492874+0000", 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00033707700000000003 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1636 239.1f e653/649 239:fe98049f:test-rados-api-smithi087-82091-111::foo:head v 653'1, mlcod=0'0)", 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:12.209723+0000", 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 15.429794382000001, 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00048164300000000002, 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.696 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.209723+0000", 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.209723+0000", 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.209723+0000", 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5673e-05 2024-02-23T00:41:27.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.209749+0000", 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3973e-05 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.209763+0000", 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.9390000000000003e-06 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.209772+0000", 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0163000000000001e-05 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.698 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.209822+0000", 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.7844999999999999e-05 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.209840+0000", 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00033821099999999999 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210178+0000", 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7044999999999999e-05 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210205+0000", 2024-02-23T00:41:27.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00048164300000000002 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27567.0:475 239.1f e653/649 239:fe98049f:test-rados-api-smithi087-82091-111::foo:head v 653'2, mlcod=0'0)", 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:12.210383+0000", 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 15.429135236, 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028215599999999999, 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210383+0000", 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210383+0000", 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210382+0000", 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7439999999999998e-06 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210387+0000", 2024-02-23T00:41:27.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9289999999999998e-06 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210395+0000", 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9169999999999999e-06 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210399+0000", 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.8277e-05 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210427+0000", 2024-02-23T00:41:27.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5241e-05 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210442+0000", 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000203403 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210646+0000", 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9171e-05 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.210665+0000", 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028215599999999999 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:478 239.0 239:00000000:test-rados-api-smithi087-82091-111:::head [pgnls start_epoch 653] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e653)", 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:12.215665+0000", 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 15.423853155, 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025609299999999999, 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:27.704 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 478 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.215665+0000", 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.215665+0000", 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.215662+0000", 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0356e-05 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.215673+0000", 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6914000000000001e-05 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.215690+0000", 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8349999999999998e-06 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.215696+0000", 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6409999999999997e-05 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.215753+0000", 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1307000000000003e-05 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.215794+0000", 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012657900000000001 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.707 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.215921+0000", 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025609299999999999 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_update_log_missing(239.1f epoch 653/649 rep_tid 1638 entries 653'3 (0'0) error 239:fe98049f:test-rados-api-smithi087-82091-111::foo:head by client.27567.0:510 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 653'1)", 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:12.230030+0000", 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 15.409488172, 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00032090299999999999, 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:27.708 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.230030+0000", 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.230030+0000", 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.230028+0000", 2024-02-23T00:41:27.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4144999999999999e-05 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.230042+0000", 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.519e-05 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.230058+0000", 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9179999999999997e-06 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.230064+0000", 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8062999999999997e-05 2024-02-23T00:41:27.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.230112+0000", 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00023803799999999999 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.230351+0000", 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00032090299999999999 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27567.0:544 239.1f e653/649 239:fe98049f:test-rados-api-smithi087-82091-111::foo:head v 653'4, mlcod=653'2)", 2024-02-23T00:41:27.711 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:12.243408+0000", 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 15.396109885, 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.063953162999999993, 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243408+0000", 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.712 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243408+0000", 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243407+0000", 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9549999999999992e-06 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243415+0000", 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5590000000000006e-06 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.713 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243424+0000", 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.929e-06 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243428+0000", 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3080999999999997e-05 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243461+0000", 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6864999999999999e-05 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.714 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243478+0000", 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.063840279999999999 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.307318+0000", 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3244999999999997e-05 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.307361+0000", 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.063953162999999993 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:546 239.0 239:00000000:%01:::head [pgnls start_epoch 653] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e653)", 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:12.308445+0000", 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 15.331072376, 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00027451100000000001, 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 546 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.716 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.308445+0000", 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.308445+0000", 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.308444+0000", 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9160000000000004e-06 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.308451+0000", 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1538e-05 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.308463+0000", 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.596e-06 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.308469+0000", 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0423999999999997e-05 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.308520+0000", 2024-02-23T00:41:27.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4944000000000002e-05 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.308565+0000", 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000155413 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.308720+0000", 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00027451100000000001 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1640 239.1f e653/649 239:fe98049f:test-rados-api-smithi087-82091-111::foo:head v 653'5, mlcod=653'3)", 2024-02-23T00:41:27.719 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:12.321437+0000", 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 15.318080863, 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00036128400000000002, 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321437+0000", 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321437+0000", 2024-02-23T00:41:27.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321436+0000", 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8530000000000003e-06 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321443+0000", 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3310000000000004e-06 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321451+0000", 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9949999999999999e-06 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321455+0000", 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.9708000000000001e-05 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321485+0000", 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8913999999999999e-05 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321504+0000", 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025946399999999999 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321763+0000", 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4851999999999998e-05 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.321798+0000", 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00036128400000000002 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27567.0:579 239.1f e653/649 239:fe98049f:test-rados-api-smithi087-82091-111::foo:head v 653'6, mlcod=653'4)", 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:12.322258+0000", 2024-02-23T00:41:27.723 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 15.317259464999999, 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00030353999999999998, 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322258+0000", 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322258+0000", 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322258+0000", 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6500000000000004e-06 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322262+0000", 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4569999999999994e-06 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322272+0000", 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4380000000000003e-06 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.725 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322277+0000", 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.2498e-05 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322310+0000", 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3061000000000001e-05 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322323+0000", 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021769700000000001 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322541+0000", 2024-02-23T00:41:27.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1362e-05 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.322562+0000", 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00030353999999999998 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2070 5.d 5:b4812045:::notify.4:head [watch ping cookie 94532396651008] snapc 0=[] ondisk+write+known_if_redirected e661)", 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:25.495549+0000", 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 2.143968385, 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00031447700000000002, 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:27.727 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2070 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:25.495549+0000", 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:25.495549+0000", 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:27.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:25.495548+0000", 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.9460000000000004e-06 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:25.495557+0000", 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.7864e-05 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:25.495575+0000", 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7830000000000004e-06 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:25.495583+0000", 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2258999999999999e-05 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:25.495676+0000", 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0397999999999997e-05 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:25.495766+0000", 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7932999999999999e-05 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:27.730 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:25.495864+0000", 2024-02-23T00:41:27.731 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00031447700000000002 2024-02-23T00:41:27.731 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.731 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.731 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.731 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:27.731 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:27.731 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:41:27.731 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:27.760 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:27.936 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:28.062 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:28.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:28.120 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-23T00:41:28.120 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:41:28.120 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-02-23T00:41:28.120 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd in 0 2024-02-23T00:41:28.168 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:28.194 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:28.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:27.229993+0000 mon.a (mon.0) 3469 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:28.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:27.230036+0000 mon.a (mon.0) 3470 : cluster [DBG] osdmap e668: 8 total, 8 up, 7 in 2024-02-23T00:41:28.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:27.230826+0000 mon.a (mon.0) 3471 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-114"}]: dispatch 2024-02-23T00:41:28.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:27.229993+0000 mon.a (mon.0) 3469 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:28.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:27.230036+0000 mon.a (mon.0) 3470 : cluster [DBG] osdmap e668: 8 total, 8 up, 7 in 2024-02-23T00:41:28.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:27.230826+0000 mon.a (mon.0) 3471 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-114"}]: dispatch 2024-02-23T00:41:28.487 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-23T00:41:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:27.229993+0000 mon.a (mon.0) 3469 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:27.230036+0000 mon.a (mon.0) 3470 : cluster [DBG] osdmap e668: 8 total, 8 up, 7 in 2024-02-23T00:41:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:27.230826+0000 mon.a (mon.0) 3471 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-114"}]: dispatch 2024-02-23T00:41:28.844 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:28.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:28.948 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:28.975 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:29.130 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:41:29.323 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:29.324 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:29.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:29.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:29.430 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:29.455 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:27.766640+0000 mgr.y (mgr.24437) 581 : cluster [DBG] pgmap v1019: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 10/723 objects misplaced (1.383%); 27 B/s, 1 objects/s recovering 2024-02-23T00:41:29.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:28.239830+0000 mon.a (mon.0) 3472 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-114"}]': finished 2024-02-23T00:41:29.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:28.239888+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e669: 8 total, 8 up, 7 in 2024-02-23T00:41:29.456 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:28.240561+0000 mon.a (mon.0) 3474 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-114", "mode": "writeback"}]: dispatch 2024-02-23T00:41:29.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:27.766640+0000 mgr.y (mgr.24437) 581 : cluster [DBG] pgmap v1019: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 10/723 objects misplaced (1.383%); 27 B/s, 1 objects/s recovering 2024-02-23T00:41:29.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:28.239830+0000 mon.a (mon.0) 3472 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-114"}]': finished 2024-02-23T00:41:29.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:28.239888+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e669: 8 total, 8 up, 7 in 2024-02-23T00:41:29.456 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:28.240561+0000 mon.a (mon.0) 3474 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-114", "mode": "writeback"}]: dispatch 2024-02-23T00:41:29.457 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:29.495 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:29.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:29.605 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:29.635 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:27.766640+0000 mgr.y (mgr.24437) 581 : cluster [DBG] pgmap v1019: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 10/723 objects misplaced (1.383%); 27 B/s, 1 objects/s recovering 2024-02-23T00:41:29.635 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:28.239830+0000 mon.a (mon.0) 3472 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-114"}]': finished 2024-02-23T00:41:29.635 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:28.239888+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e669: 8 total, 8 up, 7 in 2024-02-23T00:41:29.635 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:28.240561+0000 mon.a (mon.0) 3474 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-114", "mode": "writeback"}]: dispatch 2024-02-23T00:41:29.637 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:29.658 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:29.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:29.762 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:29.793 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:29.814 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:29.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:29.917 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:29.947 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:29.967 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:29.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:30.070 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:30.098 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:30.222 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:30.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:30.326 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:30.354 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:30.374 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:30.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:30.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.466+0000 7f3a1427f700 1 -- 172.21.15.87:0/553637011 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c102210 msgr2=0x7f3a0c104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.466+0000 7f3a1427f700 1 --2- 172.21.15.87:0/553637011 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c102210 0x7f3a0c104600 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7f3a04009be0 tx=0x7f3a04004d10).stop 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 -- 172.21.15.87:0/553637011 shutdown_connections 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 --2- 172.21.15.87:0/553637011 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c102210 0x7f3a0c104600 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 --2- 172.21.15.87:0/553637011 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a0c0f8cb0 0x7f3a0c0f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 --2- 172.21.15.87:0/553637011 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a0c0f9610 0x7f3a0c101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 -- 172.21.15.87:0/553637011 >> 172.21.15.87:0/553637011 conn(0x7f3a0c0f2820 msgr2=0x7f3a0c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 -- 172.21.15.87:0/553637011 shutdown_connections 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 -- 172.21.15.87:0/553637011 wait complete. 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 librados: starting msgr at 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 librados: starting objecter 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 Processor -- start 2024-02-23T00:41:30.471 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 -- start start 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 librados: setting wanted keys 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 librados: calling monclient init 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c0f8cb0 0x7f3a0c0f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a0c0f9610 0x7f3a0c0f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a0c102210 0x7f3a0c0f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c116440 con 0x7f3a0c0f8cb0 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c1165b0 con 0x7f3a0c102210 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.467+0000 7f3a1427f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c116770 con 0x7f3a0c0f9610 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1181a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a0c0f9610 0x7f3a0c0f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1201b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c0f8cb0 0x7f3a0c0f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1181a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a0c0f9610 0x7f3a0c0f7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40274/0 (socket says 172.21.15.87:40274) 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1201b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c0f8cb0 0x7f3a0c0f6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51186/0 (socket says 172.21.15.87:51186) 2024-02-23T00:41:30.472 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1181a700 1 -- 172.21.15.87:0/1737172944 learned_addr learned my addr 172.21.15.87:0/1737172944 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1281c700 1 --2- 172.21.15.87:0/1737172944 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a0c102210 0x7f3a0c0f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1181a700 1 -- 172.21.15.87:0/1737172944 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a0c102210 msgr2=0x7f3a0c0f7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1181a700 1 --2- 172.21.15.87:0/1737172944 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a0c102210 0x7f3a0c0f7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1181a700 1 -- 172.21.15.87:0/1737172944 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c0f8cb0 msgr2=0x7f3a0c0f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1181a700 1 --2- 172.21.15.87:0/1737172944 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c0f8cb0 0x7f3a0c0f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1181a700 1 -- 172.21.15.87:0/1737172944 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a08005040 con 0x7f3a0c0f9610 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1201b700 1 --2- 172.21.15.87:0/1737172944 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c0f8cb0 0x7f3a0c0f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1181a700 1 --2- 172.21.15.87:0/1737172944 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a0c0f9610 0x7f3a0c0f7150 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f39fc0041d0 tx=0x7f39fc009f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a03779700 10 client.?.objecter ms_handle_connect 0x7f3a0c0f9610 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a03779700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a03779700 1 -- 172.21.15.87:0/1737172944 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc007b40 con 0x7f3a0c0f9610 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a03779700 1 -- 172.21.15.87:0/1737172944 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39fc00f040 con 0x7f3a0c0f9610 2024-02-23T00:41:30.473 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a040097c0 con 0x7f3a0c0f9610 2024-02-23T00:41:30.474 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a03779700 1 -- 172.21.15.87:0/1737172944 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc014810 con 0x7f3a0c0f9610 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1427f700 10 client.38168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c116db0 con 0x7f3a0c0f9610 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.468+0000 7f3a1427f700 1 librados: init done 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.469+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c05fa60 con 0x7f3a0c0f9610 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a03779700 1 -- 172.21.15.87:0/1737172944 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f39fc009110 con 0x7f3a0c0f9610 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a03779700 1 --2- 172.21.15.87:0/1737172944 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39f8062c90 0x7f39f8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a03779700 1 -- 172.21.15.87:0/1737172944 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(671..671 src has 173..671) v4 ==== 8627+0+0 (secure 0 0 0) 0x7f39fc07d470 con 0x7f3a0c0f9610 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a03779700 10 client.38168.objecter ms_dispatch 0x7f3a0c001550 osd_map(671..671 src has 173..671) v4 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a03779700 3 client.38168.objecter handle_osd_map got epochs [671,671] > 0 2024-02-23T00:41:30.475 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a03779700 3 client.38168.objecter handle_osd_map decoding full epoch 671 2024-02-23T00:41:30.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a1201b700 1 --2- 172.21.15.87:0/1737172944 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39f8062c90 0x7f39f8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:30.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a03779700 20 client.38168.objecter dump_active .. 0 homeless 2024-02-23T00:41:30.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a03779700 10 client.38168.objecter ms_handle_connect 0x7f39f8062c90 2024-02-23T00:41:30.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.470+0000 7f3a1201b700 1 --2- 172.21.15.87:0/1737172944 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39f8062c90 0x7f39f8065150 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7f3a08009760 tx=0x7f3a0800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:30.476 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.474+0000 7f3a03779700 1 -- 172.21.15.87:0/1737172944 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f39fc050700 con 0x7f3a0c0f9610 2024-02-23T00:41:30.478 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:41:30.494 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:30.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:29.235237+0000 mon.a (mon.0) 3475 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:30.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:29.239399+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-114", "mode": "writeback"}]': finished 2024-02-23T00:41:30.512 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:29.239465+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e670: 8 total, 8 up, 7 in 2024-02-23T00:41:30.513 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:29.235237+0000 mon.a (mon.0) 3475 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:30.513 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:29.239399+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-114", "mode": "writeback"}]': finished 2024-02-23T00:41:30.513 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:29.239465+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e670: 8 total, 8 up, 7 in 2024-02-23T00:41:30.515 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:30.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:30.664+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f3a0c05ee80 con 0x7f3a0c0f9610 2024-02-23T00:41:30.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:29.235237+0000 mon.a (mon.0) 3475 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:30.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:29.239399+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-114", "mode": "writeback"}]': finished 2024-02-23T00:41:30.695 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:29.239465+0000 mon.a (mon.0) 3477 : cluster [DBG] osdmap e670: 8 total, 8 up, 7 in 2024-02-23T00:41:30.765 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:41:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:30.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:41:30.869 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:30.897 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:30.916 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:30.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:31.019 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:31.048 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:31.084 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:31.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:31.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:31.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:31.215 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:31.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.259+0000 7f3a03779700 1 -- 172.21.15.87:0/1737172944 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v672) v1 ==== 91+0+0 (secure 0 0 0) 0x7f39fc0502f0 con 0x7f3a0c0f9610 2024-02-23T00:41:31.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 10 librados: watch_flush enter 2024-02-23T00:41:31.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 10 librados: watch_flush exit 2024-02-23T00:41:31.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 20 client.38168.objecter shutdown clearing up homeless session... 2024-02-23T00:41:31.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 10 client.38168.objecter successfully canceled tick 2024-02-23T00:41:31.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39f8062c90 msgr2=0x7f39f8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:31.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 --2- 172.21.15.87:0/1737172944 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39f8062c90 0x7f39f8065150 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 rx=0x7f3a08009760 tx=0x7f3a0800b040).stop 2024-02-23T00:41:31.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a0c0f9610 msgr2=0x7f3a0c0f7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:31.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 --2- 172.21.15.87:0/1737172944 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a0c0f9610 0x7f3a0c0f7150 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 rx=0x7f39fc0041d0 tx=0x7f39fc009f60).stop 2024-02-23T00:41:31.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 shutdown_connections 2024-02-23T00:41:31.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 --2- 172.21.15.87:0/1737172944 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39f8062c90 0x7f39f8065150 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:31.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 --2- 172.21.15.87:0/1737172944 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a0c102210 0x7f3a0c0f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:31.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 --2- 172.21.15.87:0/1737172944 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a0c0f8cb0 0x7f3a0c0f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:31.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 --2- 172.21.15.87:0/1737172944 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a0c0f9610 0x7f3a0c0f7150 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:31.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 >> 172.21.15.87:0/1737172944 conn(0x7f3a0c0f2820 msgr2=0x7f3a0c0f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:31.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 shutdown_connections 2024-02-23T00:41:31.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 -- 172.21.15.87:0/1737172944 wait complete. 2024-02-23T00:41:31.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:31.261+0000 7f3a1427f700 1 librados: shutdown 2024-02-23T00:41:31.266 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:41:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:29.767063+0000 mgr.y (mgr.24437) 582 : cluster [DBG] pgmap v1022: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 10/723 objects misplaced (1.383%) 2024-02-23T00:41:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:30.251045+0000 mon.a (mon.0) 3478 : cluster [DBG] osdmap e671: 8 total, 8 up, 7 in 2024-02-23T00:41:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:30.278843+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:30.665659+0000 mon.c (mon.2) 483 : audit [INF] from='client.? 172.21.15.87:0/1737172944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:41:31.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:30.665958+0000 mon.a (mon.0) 3480 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:41:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:29.767063+0000 mgr.y (mgr.24437) 582 : cluster [DBG] pgmap v1022: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 10/723 objects misplaced (1.383%) 2024-02-23T00:41:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:30.251045+0000 mon.a (mon.0) 3478 : cluster [DBG] osdmap e671: 8 total, 8 up, 7 in 2024-02-23T00:41:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:30.278843+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:30.665659+0000 mon.c (mon.2) 483 : audit [INF] from='client.? 172.21.15.87:0/1737172944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:41:31.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:30.665958+0000 mon.a (mon.0) 3480 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:41:31.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:41:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:41:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:29.767063+0000 mgr.y (mgr.24437) 582 : cluster [DBG] pgmap v1022: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 10/723 objects misplaced (1.383%) 2024-02-23T00:41:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:30.251045+0000 mon.a (mon.0) 3478 : cluster [DBG] osdmap e671: 8 total, 8 up, 7 in 2024-02-23T00:41:31.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:30.278843+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:30.665659+0000 mon.c (mon.2) 483 : audit [INF] from='client.? 172.21.15.87:0/1737172944' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:41:31.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:30.665958+0000 mon.a (mon.0) 3480 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:41:32.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:32] "GET /metrics HTTP/1.1" 200 189899 "" "Prometheus/2.33.4" 2024-02-23T00:41:32.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:31.177807+0000 mgr.y (mgr.24437) 583 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:32.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:31.247750+0000 mon.a (mon.0) 3481 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:32.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:31.258960+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:32.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:31.259023+0000 mon.a (mon.0) 3483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:31.259077+0000 mon.a (mon.0) 3484 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:31.260274+0000 mon.a (mon.0) 3485 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114"}]: dispatch 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:32.196228+0000 mon.a (mon.0) 3486 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:32.201094+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114"}]': finished 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:32.201164+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:31.177807+0000 mgr.y (mgr.24437) 583 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:31.247750+0000 mon.a (mon.0) 3481 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:31.258960+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:31.259023+0000 mon.a (mon.0) 3483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:41:32.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:31.259077+0000 mon.a (mon.0) 3484 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2024-02-23T00:41:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:31.260274+0000 mon.a (mon.0) 3485 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114"}]: dispatch 2024-02-23T00:41:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:32.196228+0000 mon.a (mon.0) 3486 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:32.201094+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114"}]': finished 2024-02-23T00:41:32.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:32.201164+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2024-02-23T00:41:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:31.177807+0000 mgr.y (mgr.24437) 583 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:31.247750+0000 mon.a (mon.0) 3481 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:31.258960+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:31.259023+0000 mon.a (mon.0) 3483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:41:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:31.259077+0000 mon.a (mon.0) 3484 : cluster [DBG] osdmap e672: 8 total, 8 up, 7 in 2024-02-23T00:41:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:31.260274+0000 mon.a (mon.0) 3485 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114"}]: dispatch 2024-02-23T00:41:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:32.196228+0000 mon.a (mon.0) 3486 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:32.201094+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-114"}]': finished 2024-02-23T00:41:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:32.201164+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e673: 8 total, 8 up, 7 in 2024-02-23T00:41:33.548 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:31.767472+0000 mgr.y (mgr.24437) 584 : cluster [DBG] pgmap v1025: 194 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 189 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:33.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.208730+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2024-02-23T00:41:33.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:31.767472+0000 mgr.y (mgr.24437) 584 : cluster [DBG] pgmap v1025: 194 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 189 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:33.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.208730+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2024-02-23T00:41:33.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:41:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:41:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:31.767472+0000 mgr.y (mgr.24437) 584 : cluster [DBG] pgmap v1025: 194 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 189 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.208730+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e674: 8 total, 8 up, 7 in 2024-02-23T00:41:34.495 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:41:34.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.039255+0000 osd.3 (osd.3) 207 : cluster [DBG] 6.b scrub starts 2024-02-23T00:41:34.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.042516+0000 osd.3 (osd.3) 208 : cluster [DBG] 6.b scrub ok 2024-02-23T00:41:34.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.045370+0000 osd.2 (osd.2) 239 : cluster [DBG] 5.d scrub starts 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.047149+0000 osd.2 (osd.2) 240 : cluster [DBG] 5.d scrub ok 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.589602+0000 osd.6 (osd.6) 268 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.595260+0000 osd.6 (osd.6) 269 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.713869+0000 osd.4 (osd.4) 299 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.715372+0000 osd.4 (osd.4) 300 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:33.768095+0000 mgr.y (mgr.24437) 585 : cluster [DBG] pgmap v1028: 162 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 157 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:34.210642+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:34.214804+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.039255+0000 osd.3 (osd.3) 207 : cluster [DBG] 6.b scrub starts 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.042516+0000 osd.3 (osd.3) 208 : cluster [DBG] 6.b scrub ok 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.045370+0000 osd.2 (osd.2) 239 : cluster [DBG] 5.d scrub starts 2024-02-23T00:41:34.738 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.047149+0000 osd.2 (osd.2) 240 : cluster [DBG] 5.d scrub ok 2024-02-23T00:41:34.739 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.589602+0000 osd.6 (osd.6) 268 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:41:34.739 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.595260+0000 osd.6 (osd.6) 269 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:41:34.739 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.713869+0000 osd.4 (osd.4) 299 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:41:34.739 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.715372+0000 osd.4 (osd.4) 300 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:41:34.739 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:33.768095+0000 mgr.y (mgr.24437) 585 : cluster [DBG] pgmap v1028: 162 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 157 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:34.739 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:34.210642+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2024-02-23T00:41:34.739 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:34.214804+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.039255+0000 osd.3 (osd.3) 207 : cluster [DBG] 6.b scrub starts 2024-02-23T00:41:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.042516+0000 osd.3 (osd.3) 208 : cluster [DBG] 6.b scrub ok 2024-02-23T00:41:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.045370+0000 osd.2 (osd.2) 239 : cluster [DBG] 5.d scrub starts 2024-02-23T00:41:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.047149+0000 osd.2 (osd.2) 240 : cluster [DBG] 5.d scrub ok 2024-02-23T00:41:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.589602+0000 osd.6 (osd.6) 268 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:41:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.595260+0000 osd.6 (osd.6) 269 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:41:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.713869+0000 osd.4 (osd.4) 299 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:41:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.715372+0000 osd.4 (osd.4) 300 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:41:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:33.768095+0000 mgr.y (mgr.24437) 585 : cluster [DBG] pgmap v1028: 162 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 157 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:34.210642+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e675: 8 total, 8 up, 7 in 2024-02-23T00:41:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:34.214804+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:34.987 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:41:34.987 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:41:34.987 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:41:34.987 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:41:35.467 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:35.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:35.572 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:34.072855+0000 osd.3 (osd.3) 209 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:41:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:34.076317+0000 osd.3 (osd.3) 210 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:41:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:34.560935+0000 osd.6 (osd.6) 270 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:41:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:34.563958+0000 osd.6 (osd.6) 271 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:41:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:34.716182+0000 osd.4 (osd.4) 301 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:41:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:34.719383+0000 osd.4 (osd.4) 302 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:41:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:34.729352+0000 osd.7 (osd.7) 298 : cluster [DBG] 22.a scrub starts 2024-02-23T00:41:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:34.730398+0000 osd.7 (osd.7) 299 : cluster [DBG] 22.a scrub ok 2024-02-23T00:41:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:35.135320+0000 osd.1 (osd.1) 304 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:35.138312+0000 osd.1 (osd.1) 305 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:35.210798+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:35.210936+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:35.212408+0000 mon.a (mon.0) 3494 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:34.072855+0000 osd.3 (osd.3) 209 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:34.076317+0000 osd.3 (osd.3) 210 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:34.560935+0000 osd.6 (osd.6) 270 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:34.563958+0000 osd.6 (osd.6) 271 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:34.716182+0000 osd.4 (osd.4) 301 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:34.719383+0000 osd.4 (osd.4) 302 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:41:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:34.729352+0000 osd.7 (osd.7) 298 : cluster [DBG] 22.a scrub starts 2024-02-23T00:41:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:34.730398+0000 osd.7 (osd.7) 299 : cluster [DBG] 22.a scrub ok 2024-02-23T00:41:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:35.135320+0000 osd.1 (osd.1) 304 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:41:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:35.138312+0000 osd.1 (osd.1) 305 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:41:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:35.210798+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:35.210936+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2024-02-23T00:41:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:35.212408+0000 mon.a (mon.0) 3494 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:35.602 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:35.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 -- 172.21.15.87:0/2628116917 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 msgr2=0x7ff34c0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 --2- 172.21.15.87:0/2628116917 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 0x7ff34c0f7d40 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 rx=0x7ff3340047b0 tx=0x7ff334009cb0).stop 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 -- 172.21.15.87:0/2628116917 shutdown_connections 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 --2- 172.21.15.87:0/2628116917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff34c0f8420 0x7ff34c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 --2- 172.21.15.87:0/2628116917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff34c0fdbb0 0x7ff34c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 --2- 172.21.15.87:0/2628116917 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 0x7ff34c0f7d40 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 -- 172.21.15.87:0/2628116917 >> 172.21.15.87:0/2628116917 conn(0x7ff34c0f3800 msgr2=0x7ff34c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 -- 172.21.15.87:0/2628116917 shutdown_connections 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 -- 172.21.15.87:0/2628116917 wait complete. 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 librados: starting msgr at 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.674+0000 7ff351b35700 1 librados: starting objecter 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 Processor -- start 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 -- start start 2024-02-23T00:41:35.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 librados: setting wanted keys 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 librados: calling monclient init 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff34c0f8420 0x7ff34c06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff34c0fdbb0 0x7ff34c06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 0x7ff34c06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c1165a0 con 0x7ff34c0fdbb0 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c116710 con 0x7ff34c0f8420 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff351b35700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c1168d0 con 0x7ff34c0fe510 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff34c0fdbb0 0x7ff34c06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 0x7ff34c06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34bfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 0x7ff34c06bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50798/0 (socket says 172.21.15.87:50798) 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34bfff700 1 -- 172.21.15.87:0/3369802215 learned_addr learned my addr 172.21.15.87:0/3369802215 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34b7fe700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff34c0f8420 0x7ff34c06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34bfff700 1 -- 172.21.15.87:0/3369802215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff34c0f8420 msgr2=0x7ff34c06b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:35.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34bfff700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff34c0f8420 0x7ff34c06b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34bfff700 1 -- 172.21.15.87:0/3369802215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff34c0fdbb0 msgr2=0x7ff34c06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34bfff700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff34c0fdbb0 0x7ff34c06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34bfff700 1 -- 172.21.15.87:0/3369802215 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c0097c0 con 0x7ff34c0fe510 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34b7fe700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff34c0f8420 0x7ff34c06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34affd700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff34c0fdbb0 0x7ff34c06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff34bfff700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 0x7ff34c06bab0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7ff33c001df0 tx=0x7ff33c004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff348ff9700 10 client.?.objecter ms_handle_connect 0x7ff34c0fe510 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.675+0000 7ff348ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.676+0000 7ff348ff9700 1 -- 172.21.15.87:0/3369802215 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c009a10 con 0x7ff34c0fe510 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.676+0000 7ff348ff9700 1 -- 172.21.15.87:0/3369802215 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff33c0095a0 con 0x7ff34c0fe510 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.676+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff334005040 con 0x7ff34c0fe510 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.676+0000 7ff348ff9700 1 -- 172.21.15.87:0/3369802215 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c007e30 con 0x7ff34c0fe510 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.676+0000 7ff351b35700 10 client.38174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:35.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.676+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c116ec0 con 0x7ff34c0fe510 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.676+0000 7ff351b35700 1 librados: init done 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.677+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34c0606d0 con 0x7ff34c0fe510 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.677+0000 7ff348ff9700 1 -- 172.21.15.87:0/3369802215 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff33c0090d0 con 0x7ff34c0fe510 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.677+0000 7ff348ff9700 1 --2- 172.21.15.87:0/3369802215 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff338062c90 0x7ff338065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.677+0000 7ff348ff9700 1 -- 172.21.15.87:0/3369802215 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(676..676 src has 173..676) v4 ==== 8576+0+0 (secure 0 0 0) 0x7ff33c07dcf0 con 0x7ff34c0fe510 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.677+0000 7ff348ff9700 10 client.38174.objecter ms_dispatch 0x7ff34c001550 osd_map(676..676 src has 173..676) v4 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.677+0000 7ff348ff9700 3 client.38174.objecter handle_osd_map got epochs [676,676] > 0 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.677+0000 7ff348ff9700 3 client.38174.objecter handle_osd_map decoding full epoch 676 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.677+0000 7ff34b7fe700 1 --2- 172.21.15.87:0/3369802215 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff338062c90 0x7ff338065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.678+0000 7ff348ff9700 20 client.38174.objecter dump_active .. 0 homeless 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.678+0000 7ff348ff9700 10 client.38174.objecter ms_handle_connect 0x7ff338062c90 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.678+0000 7ff34b7fe700 1 --2- 172.21.15.87:0/3369802215 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff338062c90 0x7ff338065150 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 rx=0x7ff34400a900 tx=0x7ff344008040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:35.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.682+0000 7ff348ff9700 1 -- 172.21.15.87:0/3369802215 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7ff33c050fc0 con 0x7ff34c0fe510 2024-02-23T00:41:35.750 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:35.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:35.853 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:35.860+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd in", "ids": ["0"]} v 0) v1 -- 0x7ff34c06c560 con 0x7ff34c0fe510 2024-02-23T00:41:35.880 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:35.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:34.072855+0000 osd.3 (osd.3) 209 : cluster [DBG] 4.19 scrub starts 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:34.076317+0000 osd.3 (osd.3) 210 : cluster [DBG] 4.19 scrub ok 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:34.560935+0000 osd.6 (osd.6) 270 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:34.563958+0000 osd.6 (osd.6) 271 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:34.716182+0000 osd.4 (osd.4) 301 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:34.719383+0000 osd.4 (osd.4) 302 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:34.729352+0000 osd.7 (osd.7) 298 : cluster [DBG] 22.a scrub starts 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:34.730398+0000 osd.7 (osd.7) 299 : cluster [DBG] 22.a scrub ok 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:35.135320+0000 osd.1 (osd.1) 304 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:35.138312+0000 osd.1 (osd.1) 305 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:35.210798+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:35.210936+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e676: 8 total, 8 up, 7 in 2024-02-23T00:41:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:35.212408+0000 mon.a (mon.0) 3494 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:35.977 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:35.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:36.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:36.113 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:36.133 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:36.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:36.170 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:41:35.860+0000 7fe8e2097700 -1 mon.a@0(leader).osd e676 definitely_dead 0 2024-02-23T00:41:36.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.222+0000 7ff348ff9700 1 -- 172.21.15.87:0/3369802215 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["0"]}]=0 marked in osd.0. v677) v1 ==== 85+0+0 (secure 0 0 0) 0x7ff33c050bb0 con 0x7ff34c0fe510 2024-02-23T00:41:36.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 10 librados: watch_flush enter 2024-02-23T00:41:36.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 10 librados: watch_flush exit 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 20 client.38174.objecter shutdown clearing up homeless session... 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 10 client.38174.objecter successfully canceled tick 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff338062c90 msgr2=0x7ff338065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 --2- 172.21.15.87:0/3369802215 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff338062c90 0x7ff338065150 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 rx=0x7ff34400a900 tx=0x7ff344008040).stop 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 msgr2=0x7ff34c06bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 0x7ff34c06bab0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 rx=0x7ff33c001df0 tx=0x7ff33c004d10).stop 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 shutdown_connections 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 --2- 172.21.15.87:0/3369802215 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff338062c90 0x7ff338065150 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff34c0fe510 0x7ff34c06bab0 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff34c0f8420 0x7ff34c06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 --2- 172.21.15.87:0/3369802215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff34c0fdbb0 0x7ff34c06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.225+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 >> 172.21.15.87:0/3369802215 conn(0x7ff34c0f3800 msgr2=0x7ff34c0fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.226+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 shutdown_connections 2024-02-23T00:41:36.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.226+0000 7ff351b35700 1 -- 172.21.15.87:0/3369802215 wait complete. 2024-02-23T00:41:36.230 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:36.226+0000 7ff351b35700 1 librados: shutdown 2024-02-23T00:41:36.235 INFO:teuthology.orchestra.run.smithi087.stderr:marked in osd.0. 2024-02-23T00:41:36.238 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:36.266 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:36.286 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:36.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:36.390 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:36.418 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:36.437 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:36.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:36.540 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:36.566 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:36.585 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:36.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:36.602 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-23T00:41:36.689 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:35.600600+0000 osd.6 (osd.6) 272 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:41:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:35.603553+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:41:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:35.761138+0000 osd.4 (osd.4) 303 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:41:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:35.764855+0000 osd.4 (osd.4) 304 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:41:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:35.768939+0000 mgr.y (mgr.24437) 586 : cluster [DBG] pgmap v1031: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:35.861724+0000 mon.c (mon.2) 484 : audit [DBG] from='client.? 172.21.15.87:0/3369802215' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-23T00:41:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:35.862046+0000 mon.a (mon.0) 3495 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-23T00:41:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:36.214640+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:36.214723+0000 mon.a (mon.0) 3497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-23T00:41:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:36.214787+0000 mon.a (mon.0) 3498 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-02-23T00:41:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:36.215277+0000 mon.a (mon.0) 3499 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-116"}]: dispatch 2024-02-23T00:41:36.716 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:36.735 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:36.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:35.600600+0000 osd.6 (osd.6) 272 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:41:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:35.603553+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:41:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:35.761138+0000 osd.4 (osd.4) 303 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:41:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:35.764855+0000 osd.4 (osd.4) 304 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:41:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:35.768939+0000 mgr.y (mgr.24437) 586 : cluster [DBG] pgmap v1031: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:35.861724+0000 mon.c (mon.2) 484 : audit [DBG] from='client.? 172.21.15.87:0/3369802215' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-23T00:41:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:35.862046+0000 mon.a (mon.0) 3495 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:36.214640+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:36.214723+0000 mon.a (mon.0) 3497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:36.214787+0000 mon.a (mon.0) 3498 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:36.215277+0000 mon.a (mon.0) 3499 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-116"}]: dispatch 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:35.600600+0000 osd.6 (osd.6) 272 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:35.603553+0000 osd.6 (osd.6) 273 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:35.761138+0000 osd.4 (osd.4) 303 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:35.764855+0000 osd.4 (osd.4) 304 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:35.768939+0000 mgr.y (mgr.24437) 586 : cluster [DBG] pgmap v1031: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:35.861724+0000 mon.c (mon.2) 484 : audit [DBG] from='client.? 172.21.15.87:0/3369802215' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-23T00:41:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:35.862046+0000 mon.a (mon.0) 3495 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-23T00:41:36.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:36.214640+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:36.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:36.214723+0000 mon.a (mon.0) 3497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-23T00:41:36.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:36.214787+0000 mon.a (mon.0) 3498 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-02-23T00:41:36.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:36.215277+0000 mon.a (mon.0) 3499 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-116"}]: dispatch 2024-02-23T00:41:36.838 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:36.864 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:36.986 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:36.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:37.100 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:37.127 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:37.145 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:37.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:37.248 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:37.274 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:37.293 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:37.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:37.397 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:37.423 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:37.503 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:37.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:37.517 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-02-23T00:41:37.607 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:37.634 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:37.652 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:37.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:37.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:37.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:36.044687+0000 osd.2 (osd.2) 241 : cluster [DBG] 5.a scrub starts 2024-02-23T00:41:37.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:36.045578+0000 osd.2 (osd.2) 242 : cluster [DBG] 5.a scrub ok 2024-02-23T00:41:37.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:36.602317+0000 osd.6 (osd.6) 274 : cluster [DBG] 234.6 scrub starts 2024-02-23T00:41:37.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:36.605795+0000 osd.6 (osd.6) 275 : cluster [DBG] 234.6 scrub ok 2024-02-23T00:41:37.780 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:36.714203+0000 osd.4 (osd.4) 305 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:36.715007+0000 osd.4 (osd.4) 306 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:37.204284+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-116"}]': finished 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:37.204564+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:37.205819+0000 mon.a (mon.0) 3502 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-116", "mode": "writeback"}]: dispatch 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:36.044687+0000 osd.2 (osd.2) 241 : cluster [DBG] 5.a scrub starts 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:36.045578+0000 osd.2 (osd.2) 242 : cluster [DBG] 5.a scrub ok 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:36.602317+0000 osd.6 (osd.6) 274 : cluster [DBG] 234.6 scrub starts 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:36.605795+0000 osd.6 (osd.6) 275 : cluster [DBG] 234.6 scrub ok 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:36.714203+0000 osd.4 (osd.4) 305 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:36.715007+0000 osd.4 (osd.4) 306 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:37.204284+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-116"}]': finished 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:37.204564+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-02-23T00:41:37.781 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:37.205819+0000 mon.a (mon.0) 3502 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-116", "mode": "writeback"}]: dispatch 2024-02-23T00:41:37.782 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:36.044687+0000 osd.2 (osd.2) 241 : cluster [DBG] 5.a scrub starts 2024-02-23T00:41:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:36.045578+0000 osd.2 (osd.2) 242 : cluster [DBG] 5.a scrub ok 2024-02-23T00:41:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:36.602317+0000 osd.6 (osd.6) 274 : cluster [DBG] 234.6 scrub starts 2024-02-23T00:41:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:36.605795+0000 osd.6 (osd.6) 275 : cluster [DBG] 234.6 scrub ok 2024-02-23T00:41:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:36.714203+0000 osd.4 (osd.4) 305 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:41:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:36.715007+0000 osd.4 (osd.4) 306 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:41:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:37.204284+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-116"}]': finished 2024-02-23T00:41:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:37.204564+0000 mon.a (mon.0) 3501 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-02-23T00:41:37.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:37.205819+0000 mon.a (mon.0) 3502 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-116", "mode": "writeback"}]: dispatch 2024-02-23T00:41:38.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:37.093529+0000 osd.3 (osd.3) 211 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:41:38.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:37.096902+0000 osd.3 (osd.3) 212 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:41:38.487 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:37.769344+0000 mgr.y (mgr.24437) 587 : cluster [DBG] pgmap v1034: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:38.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:38.201258+0000 mon.a (mon.0) 3503 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:38.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:38.208156+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-116", "mode": "writeback"}]': finished 2024-02-23T00:41:38.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:38.208248+0000 mon.a (mon.0) 3505 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-02-23T00:41:38.488 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:38.279115+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:38.488 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:37.093529+0000 osd.3 (osd.3) 211 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:41:38.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:37.096902+0000 osd.3 (osd.3) 212 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:41:38.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:37.769344+0000 mgr.y (mgr.24437) 587 : cluster [DBG] pgmap v1034: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:38.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:38.201258+0000 mon.a (mon.0) 3503 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:38.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:38.208156+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-116", "mode": "writeback"}]': finished 2024-02-23T00:41:38.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:38.208248+0000 mon.a (mon.0) 3505 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-02-23T00:41:38.489 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:38.279115+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:38.621 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:38.662 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:38.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:38.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:38.674 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:38.769 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:38.797 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:37.093529+0000 osd.3 (osd.3) 211 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:41:38.797 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:37.096902+0000 osd.3 (osd.3) 212 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:41:38.797 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:37.769344+0000 mgr.y (mgr.24437) 587 : cluster [DBG] pgmap v1034: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 10/720 objects misplaced (1.389%) 2024-02-23T00:41:38.797 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:38.201258+0000 mon.a (mon.0) 3503 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:38.797 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:38.208156+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-116", "mode": "writeback"}]': finished 2024-02-23T00:41:38.797 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:38.208248+0000 mon.a (mon.0) 3505 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-02-23T00:41:38.797 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:38.279115+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:38.799 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:38.819 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:38.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:38.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:38.923 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:38.951 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:38.971 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:38.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:39.074 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:39.102 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:39.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:38.207411+0000 osd.3 (osd.3) 213 : cluster [DBG] 5.3 starting backfill to osd.0 from (0'0,0'0] MAX to 631'19 2024-02-23T00:41:39.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:38.207468+0000 osd.3 (osd.3) 214 : cluster [DBG] 4.19 starting backfill to osd.0 from (0'0,0'0] MAX to 453'68 2024-02-23T00:41:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:38.208414+0000 osd.3 (osd.3) 215 : cluster [DBG] 4.1a starting backfill to osd.0 from (0'0,0'0] MAX to 476'63 2024-02-23T00:41:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:38.216493+0000 osd.1 (osd.1) 306 : cluster [DBG] 4.b starting backfill to osd.0 from (0'0,0'0] MAX to 453'22 2024-02-23T00:41:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:38.216509+0000 osd.1 (osd.1) 307 : cluster [DBG] 4.6 starting backfill to osd.0 from (0'0,0'0] MAX to 453'39 2024-02-23T00:41:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:38.216577+0000 osd.1 (osd.1) 308 : cluster [DBG] 4.7 starting backfill to osd.0 from (0'0,0'0] MAX to 453'29 2024-02-23T00:41:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:38.217187+0000 osd.1 (osd.1) 309 : cluster [DBG] 4.1e starting backfill to osd.0 from (0'0,0'0] MAX to 453'40 2024-02-23T00:41:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:39.206513+0000 mon.a (mon.0) 3507 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:39.206589+0000 mon.a (mon.0) 3508 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-02-23T00:41:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:39.211681+0000 mon.a (mon.0) 3509 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116"}]: dispatch 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:38.207411+0000 osd.3 (osd.3) 213 : cluster [DBG] 5.3 starting backfill to osd.0 from (0'0,0'0] MAX to 631'19 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:38.207468+0000 osd.3 (osd.3) 214 : cluster [DBG] 4.19 starting backfill to osd.0 from (0'0,0'0] MAX to 453'68 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:38.208414+0000 osd.3 (osd.3) 215 : cluster [DBG] 4.1a starting backfill to osd.0 from (0'0,0'0] MAX to 476'63 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:38.216493+0000 osd.1 (osd.1) 306 : cluster [DBG] 4.b starting backfill to osd.0 from (0'0,0'0] MAX to 453'22 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:38.216509+0000 osd.1 (osd.1) 307 : cluster [DBG] 4.6 starting backfill to osd.0 from (0'0,0'0] MAX to 453'39 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:38.216577+0000 osd.1 (osd.1) 308 : cluster [DBG] 4.7 starting backfill to osd.0 from (0'0,0'0] MAX to 453'29 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:38.217187+0000 osd.1 (osd.1) 309 : cluster [DBG] 4.1e starting backfill to osd.0 from (0'0,0'0] MAX to 453'40 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:39.206513+0000 mon.a (mon.0) 3507 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:39.206589+0000 mon.a (mon.0) 3508 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-02-23T00:41:39.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:39.211681+0000 mon.a (mon.0) 3509 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116"}]: dispatch 2024-02-23T00:41:39.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:38.207411+0000 osd.3 (osd.3) 213 : cluster [DBG] 5.3 starting backfill to osd.0 from (0'0,0'0] MAX to 631'19 2024-02-23T00:41:39.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:38.207468+0000 osd.3 (osd.3) 214 : cluster [DBG] 4.19 starting backfill to osd.0 from (0'0,0'0] MAX to 453'68 2024-02-23T00:41:39.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:38.208414+0000 osd.3 (osd.3) 215 : cluster [DBG] 4.1a starting backfill to osd.0 from (0'0,0'0] MAX to 476'63 2024-02-23T00:41:39.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:38.216493+0000 osd.1 (osd.1) 306 : cluster [DBG] 4.b starting backfill to osd.0 from (0'0,0'0] MAX to 453'22 2024-02-23T00:41:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:38.216509+0000 osd.1 (osd.1) 307 : cluster [DBG] 4.6 starting backfill to osd.0 from (0'0,0'0] MAX to 453'39 2024-02-23T00:41:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:38.216577+0000 osd.1 (osd.1) 308 : cluster [DBG] 4.7 starting backfill to osd.0 from (0'0,0'0] MAX to 453'29 2024-02-23T00:41:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:38.217187+0000 osd.1 (osd.1) 309 : cluster [DBG] 4.1e starting backfill to osd.0 from (0'0,0'0] MAX to 453'40 2024-02-23T00:41:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:39.206513+0000 mon.a (mon.0) 3507 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:39.206589+0000 mon.a (mon.0) 3508 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-02-23T00:41:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:39.211681+0000 mon.a (mon.0) 3509 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116"}]: dispatch 2024-02-23T00:41:40.237 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:40.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:40.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:41:40.369 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:40.493 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:41:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:40.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:41:40.598 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:40.630 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:40.651 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:40.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:40.755 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:40.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:39.658248+0000 mon.a (mon.0) 3510 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:40.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:39.769805+0000 mgr.y (mgr.24437) 588 : cluster [DBG] pgmap v1037: 194 pgs: 1 activating+degraded, 1 active+recovering, 1 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3/720 objects degraded (0.417%); 7/720 objects misplaced (0.972%) 2024-02-23T00:41:40.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:39.896593+0000 osd.5 (osd.5) 257 : cluster [DBG] 3.c scrub starts 2024-02-23T00:41:40.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:39.898167+0000 osd.5 (osd.5) 258 : cluster [DBG] 3.c scrub ok 2024-02-23T00:41:40.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:40.203941+0000 mon.a (mon.0) 3511 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:40.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:40.210644+0000 mon.a (mon.0) 3512 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116"}]': finished 2024-02-23T00:41:40.783 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:40.210764+0000 mon.a (mon.0) 3513 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-02-23T00:41:40.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:39.658248+0000 mon.a (mon.0) 3510 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:40.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:39.769805+0000 mgr.y (mgr.24437) 588 : cluster [DBG] pgmap v1037: 194 pgs: 1 activating+degraded, 1 active+recovering, 1 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3/720 objects degraded (0.417%); 7/720 objects misplaced (0.972%) 2024-02-23T00:41:40.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:39.896593+0000 osd.5 (osd.5) 257 : cluster [DBG] 3.c scrub starts 2024-02-23T00:41:40.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:39.898167+0000 osd.5 (osd.5) 258 : cluster [DBG] 3.c scrub ok 2024-02-23T00:41:40.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:40.203941+0000 mon.a (mon.0) 3511 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:40.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:40.210644+0000 mon.a (mon.0) 3512 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116"}]': finished 2024-02-23T00:41:40.784 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:40.210764+0000 mon.a (mon.0) 3513 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-02-23T00:41:40.786 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:39.658248+0000 mon.a (mon.0) 3510 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:39.769805+0000 mgr.y (mgr.24437) 588 : cluster [DBG] pgmap v1037: 194 pgs: 1 activating+degraded, 1 active+recovering, 1 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3/720 objects degraded (0.417%); 7/720 objects misplaced (0.972%) 2024-02-23T00:41:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:39.896593+0000 osd.5 (osd.5) 257 : cluster [DBG] 3.c scrub starts 2024-02-23T00:41:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:39.898167+0000 osd.5 (osd.5) 258 : cluster [DBG] 3.c scrub ok 2024-02-23T00:41:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:40.203941+0000 mon.a (mon.0) 3511 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:40.210644+0000 mon.a (mon.0) 3512 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-116"}]': finished 2024-02-23T00:41:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:40.210764+0000 mon.a (mon.0) 3513 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-02-23T00:41:41.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.873+0000 7ff34faef700 1 -- 172.21.15.87:0/1532675397 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fcbb0 msgr2=0x7ff3480fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.873+0000 7ff34faef700 1 --2- 172.21.15.87:0/1532675397 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fcbb0 0x7ff3480fcf90 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7ff3440047b0 tx=0x7ff344009cb0).stop 2024-02-23T00:41:41.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.873+0000 7ff34faef700 1 -- 172.21.15.87:0/1532675397 shutdown_connections 2024-02-23T00:41:41.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.873+0000 7ff34faef700 1 --2- 172.21.15.87:0/1532675397 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3480f7400 0x7ff348103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.873+0000 7ff34faef700 1 --2- 172.21.15.87:0/1532675397 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fcbb0 0x7ff3480fcf90 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.873+0000 7ff34faef700 1 --2- 172.21.15.87:0/1532675397 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3480fd510 0x7ff3480f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.873+0000 7ff34faef700 1 -- 172.21.15.87:0/1532675397 >> 172.21.15.87:0/1532675397 conn(0x7ff3480f2820 msgr2=0x7ff3480f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 -- 172.21.15.87:0/1532675397 shutdown_connections 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 -- 172.21.15.87:0/1532675397 wait complete. 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 librados: starting msgr at 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 librados: starting objecter 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 Processor -- start 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 -- start start 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 librados: setting wanted keys 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 librados: calling monclient init 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3480f7400 0x7ff348107120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3480fcbb0 0x7ff348107610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fd510 0x7ff348107b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.241 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3481167a0 con 0x7ff3480fcbb0 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3481168e0 con 0x7ff3480f7400 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34faef700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff348116a70 con 0x7ff3480fd510 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34e08c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fd510 0x7ff348107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.874+0000 7ff34d08a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3480fcbb0 0x7ff348107610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34e08c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fd510 0x7ff348107b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50804/0 (socket says 172.21.15.87:50804) 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34e08c700 1 -- 172.21.15.87:0/4040164517 learned_addr learned my addr 172.21.15.87:0/4040164517 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34d08a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3480fcbb0 0x7ff348107610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49318/0 (socket says 172.21.15.87:49318) 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34d88b700 1 --2- 172.21.15.87:0/4040164517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3480f7400 0x7ff348107120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34e08c700 1 -- 172.21.15.87:0/4040164517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3480f7400 msgr2=0x7ff348107120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34e08c700 1 --2- 172.21.15.87:0/4040164517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7ff3480f7400 0x7ff348107120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34e08c700 1 -- 172.21.15.87:0/4040164517 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3480fcbb0 msgr2=0x7ff348107610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34e08c700 1 --2- 172.21.15.87:0/4040164517 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3480fcbb0 0x7ff348107610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34e08c700 1 -- 172.21.15.87:0/4040164517 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3400097c0 con 0x7ff3480fd510 2024-02-23T00:41:41.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34d08a700 1 --2- 172.21.15.87:0/4040164517 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7ff3480fcbb0 0x7ff348107610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34e08c700 1 --2- 172.21.15.87:0/4040164517 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fd510 0x7ff348107b00 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7ff340001df0 tx=0x7ff340004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff33effd700 10 client.?.objecter ms_handle_connect 0x7ff3480fd510 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff33effd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340009a10 con 0x7ff3480fd510 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff344005040 con 0x7ff3480fd510 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3400095a0 con 0x7ff3480fd510 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340007e30 con 0x7ff3480fd510 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34faef700 10 client.38180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff348117060 con 0x7ff3480fd510 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.875+0000 7ff34faef700 1 librados: init done 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.876+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff34805fa60 con 0x7ff3480fd510 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.876+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7ff3400090d0 con 0x7ff3480fd510 2024-02-23T00:41:41.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff33effd700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff334062c90 0x7ff334065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.244 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:41:41.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(681..681 src has 173..681) v4 ==== 8568+0+0 (secure 0 0 0) 0x7ff34007dcf0 con 0x7ff3480fd510 2024-02-23T00:41:41.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 osd_map(681..681 src has 173..681) v4 2024-02-23T00:41:41.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff33effd700 3 client.38180.objecter handle_osd_map got epochs [681,681] > 0 2024-02-23T00:41:41.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff33effd700 3 client.38180.objecter handle_osd_map decoding full epoch 681 2024-02-23T00:41:41.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34d88b700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff334062c90 0x7ff334065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff33effd700 20 client.38180.objecter dump_active .. 0 homeless 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v681) v1 ==== 54+0+16 (secure 0 0 0) 0x7ff340080380 con 0x7ff3480fd510 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 10 client.38180.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 1 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6801/484081831 conn(0x7ff348028540 0x7ff3480fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff34805ee80 osd=0 3 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34805ee80 osd=0 3 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff34805ee80 osd=0 3 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 1 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff34805ee80 osd=0 3 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 0 1 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34805ee80 osd=0 4 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 10 client.38180.objecter _send_command 1 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff348106120 con 0x7ff348028540 2024-02-23T00:41:41.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff33effd700 10 client.38180.objecter ms_handle_connect 0x7ff334062c90 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34d88b700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff334062c90 0x7ff334065150 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7ff344005010 tx=0x7ff344009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.877+0000 7ff34d08a700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6801/484081831 conn(0x7ff348028540 0x7ff3480fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.878+0000 7ff33effd700 10 client.38180.objecter ms_handle_connect 0x7ff348028540 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.878+0000 7ff34d08a700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6801/484081831 conn(0x7ff348028540 0x7ff3480fd9b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.878+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff348106120 con 0x7ff348028540 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.878+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.878+0000 7ff33effd700 10 client.38180.objecter _finish_command 1 = system:0 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.878+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff34805ee80 osd=0 4 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.878+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 0 1 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 10 client.38180.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 2 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff34805ee80 osd=0 3 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34805ee80 osd=0 3 2024-02-23T00:41:41.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff34805ee80 osd=0 3 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 2 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff34805ee80 osd=0 3 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 0 2 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34805ee80 osd=0 4 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 10 client.38180.objecter _send_command 2 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.893+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff34805a040 con 0x7ff348028540 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.894+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff34805a040 con 0x7ff348028540 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.894+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.894+0000 7ff33effd700 10 client.38180.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.894+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff34805ee80 osd=0 4 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.894+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 0 2 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 10 client.38180.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 3 2024-02-23T00:41:41.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6805/685354171 conn(0x7ff348027b90 0x7ff348027f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff3480ff020 osd=1 3 2024-02-23T00:41:41.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff3480ff020 osd=1 3 2024-02-23T00:41:41.248 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:41:41.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff3480ff020 osd=1 3 2024-02-23T00:41:41.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 3 2024-02-23T00:41:41.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff3480ff020 osd=1 3 2024-02-23T00:41:41.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 1 3 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff3480ff020 osd=1 4 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 10 client.38180.objecter _send_command 3 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3480ff190 con 0x7ff348027b90 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34d88b700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6805/685354171 conn(0x7ff348027b90 0x7ff348027f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff34d88b700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6805/685354171 conn(0x7ff348027b90 0x7ff348027f70 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff33effd700 10 client.38180.objecter ms_handle_connect 0x7ff348027b90 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff3480ff190 con 0x7ff348027b90 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff33effd700 10 client.38180.objecter _finish_command 3 = system:0 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff3480ff020 osd=1 4 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.895+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 1 3 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 10 client.38180.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 4 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff3480ff020 osd=1 3 2024-02-23T00:41:41.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff3480ff020 osd=1 3 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff3480ff020 osd=1 3 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 4 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff3480ff020 osd=1 3 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 1 4 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff3480ff020 osd=1 4 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 10 client.38180.objecter _send_command 4 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.910+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff348027600 con 0x7ff348027b90 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff348027600 con 0x7ff348027b90 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff33effd700 10 client.38180.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff3480ff020 osd=1 4 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 1 4 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 10 client.38180.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 5 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff3480feb80 0x7ff34811e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348027a10 osd=2 3 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348027a10 osd=2 3 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348027a10 osd=2 3 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 5 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348027a10 osd=2 3 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 2 5 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348027a10 osd=2 4 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 10 client.38180.objecter _send_command 5 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff348121140 con 0x7ff3480feb80 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34d08a700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff3480feb80 0x7ff34811e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff34d08a700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff3480feb80 0x7ff34811e460 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.912+0000 7ff33effd700 10 client.38180.objecter ms_handle_connect 0x7ff3480feb80 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.913+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff348121140 con 0x7ff3480feb80 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.913+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:41:41.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.913+0000 7ff33effd700 10 client.38180.objecter _finish_command 5 = system:0 2024-02-23T00:41:41.252 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:41:41.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.913+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348027a10 osd=2 4 2024-02-23T00:41:41.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.913+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 2 5 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 10 client.38180.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 6 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348027a10 osd=2 3 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348027a10 osd=2 3 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348027a10 osd=2 3 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 6 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348027a10 osd=2 3 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 2 6 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348027a10 osd=2 4 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 10 client.38180.objecter _send_command 6 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.927+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff3480fe1a0 con 0x7ff3480feb80 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.928+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff3480fe1a0 con 0x7ff3480feb80 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.928+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.928+0000 7ff33effd700 10 client.38180.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.928+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348027a10 osd=2 4 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.928+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 2 6 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 10 client.38180.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 7 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6813/128059790 conn(0x7ff34811fad0 0x7ff34811ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff34811baa0 osd=3 3 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34811baa0 osd=3 3 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff34811baa0 osd=3 3 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 7 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff34811baa0 osd=3 3 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 3 7 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34811baa0 osd=3 4 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 10 client.38180.objecter _send_command 7 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff34811c1e0 con 0x7ff34811fad0 2024-02-23T00:41:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34e08c700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6813/128059790 conn(0x7ff34811fad0 0x7ff34811ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff33effd700 10 client.38180.objecter ms_handle_connect 0x7ff34811fad0 2024-02-23T00:41:41.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.929+0000 7ff34e08c700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6813/128059790 conn(0x7ff34811fad0 0x7ff34811ff10 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.930+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff34811c1e0 con 0x7ff34811fad0 2024-02-23T00:41:41.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.930+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:41:41.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.930+0000 7ff33effd700 10 client.38180.objecter _finish_command 7 = system:0 2024-02-23T00:41:41.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.930+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff34811baa0 osd=3 4 2024-02-23T00:41:41.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.930+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 3 7 2024-02-23T00:41:41.255 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:41:41.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 10 client.38180.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 8 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff34811baa0 osd=3 3 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34811baa0 osd=3 3 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff34811baa0 osd=3 3 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 8 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff34811baa0 osd=3 3 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 3 8 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34811baa0 osd=3 4 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 10 client.38180.objecter _send_command 8 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.944+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff348120bd0 con 0x7ff34811fad0 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.945+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff348120bd0 con 0x7ff34811fad0 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.945+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:41.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.945+0000 7ff33effd700 10 client.38180.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.945+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff34811baa0 osd=3 4 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.945+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 3 8 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 10 client.38180.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 9 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff348123340 0x7ff348125730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348120400 osd=4 3 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120400 osd=4 3 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348120400 osd=4 3 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 9 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348120400 osd=4 3 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 4 9 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120400 osd=4 4 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 10 client.38180.objecter _send_command 9 2024-02-23T00:41:41.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff348120b40 con 0x7ff348123340 2024-02-23T00:41:41.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34d88b700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff348123340 0x7ff348125730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff33effd700 10 client.38180.objecter ms_handle_connect 0x7ff348123340 2024-02-23T00:41:41.258 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:41:41.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.946+0000 7ff34d88b700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff348123340 0x7ff348125730 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.947+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff348120b40 con 0x7ff348123340 2024-02-23T00:41:41.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.947+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.947+0000 7ff33effd700 10 client.38180.objecter _finish_command 9 = system:0 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.947+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348120400 osd=4 4 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.947+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 4 9 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 10 client.38180.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 10 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348120400 osd=4 3 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120400 osd=4 3 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348120400 osd=4 3 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 10 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348120400 osd=4 3 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 4 10 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120400 osd=4 4 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 10 client.38180.objecter _send_command 10 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.962+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff348120b40 con 0x7ff348123340 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.963+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff348120b40 con 0x7ff348123340 2024-02-23T00:41:41.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.963+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.963+0000 7ff33effd700 10 client.38180.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.963+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348120400 osd=4 4 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.963+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 4 10 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 10 client.38180.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 11 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff348126130 0x7ff348126590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348120b40 osd=5 3 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120b40 osd=5 3 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348120b40 osd=5 3 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 11 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348120b40 osd=5 3 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 5 11 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120b40 osd=5 4 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 10 client.38180.objecter _send_command 11 2024-02-23T00:41:41.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff348128160 con 0x7ff348126130 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34d08a700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff348126130 0x7ff348126590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff33effd700 10 client.38180.objecter ms_handle_connect 0x7ff348126130 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.964+0000 7ff34d08a700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff348126130 0x7ff348126590 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.965+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff348128160 con 0x7ff348126130 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.965+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.965+0000 7ff33effd700 10 client.38180.objecter _finish_command 11 = system:0 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.965+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348120b40 osd=5 4 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.965+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 5 11 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 10 client.38180.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 12 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348120b40 osd=5 3 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120b40 osd=5 3 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348120b40 osd=5 3 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 12 2024-02-23T00:41:41.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348120b40 osd=5 3 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 5 12 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120b40 osd=5 4 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 10 client.38180.objecter _send_command 12 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.980+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff348128160 con 0x7ff348126130 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.982+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff348128160 con 0x7ff348126130 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.982+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.982+0000 7ff33effd700 10 client.38180.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.982+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348120b40 osd=5 4 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.982+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 5 12 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:41:41.262 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 10 client.38180.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 13 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff34812a910 0x7ff34812ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348128160 osd=6 3 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348128160 osd=6 3 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348128160 osd=6 3 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 13 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348128160 osd=6 3 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 6 13 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348128160 osd=6 4 2024-02-23T00:41:41.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 10 client.38180.objecter _send_command 13 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff34812e700 con 0x7ff34812a910 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34e08c700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff34812a910 0x7ff34812ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff33effd700 10 client.38180.objecter ms_handle_connect 0x7ff34812a910 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.983+0000 7ff34e08c700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff34812a910 0x7ff34812ad70 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.984+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff34812e700 con 0x7ff34812a910 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.984+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.984+0000 7ff33effd700 10 client.38180.objecter _finish_command 13 = system:0 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.984+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348128160 osd=6 4 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.984+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 6 13 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 10 client.38180.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 14 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348128160 osd=6 3 2024-02-23T00:41:41.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348128160 osd=6 3 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348128160 osd=6 3 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 14 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348128160 osd=6 3 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 6 14 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348128160 osd=6 4 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 10 client.38180.objecter _send_command 14 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:40.998+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff348126b40 con 0x7ff34812a910 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.000+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff348126b40 con 0x7ff34812a910 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.000+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.000+0000 7ff33effd700 10 client.38180.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.000+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348128160 osd=6 4 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.000+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 6 14 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 10 client.38180.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 15 2024-02-23T00:41:41.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6812/373787000 conn(0x7ff3481271c0 0x7ff348127620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348126b40 osd=7 3 2024-02-23T00:41:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348126b40 osd=7 3 2024-02-23T00:41:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348126b40 osd=7 3 2024-02-23T00:41:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 15 2024-02-23T00:41:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348126b40 osd=7 3 2024-02-23T00:41:41.266 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 7 15 2024-02-23T00:41:41.266 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348126b40 osd=7 4 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 10 client.38180.objecter _send_command 15 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff348127c10 con 0x7ff3481271c0 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34d88b700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6812/373787000 conn(0x7ff3481271c0 0x7ff348127620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff34d88b700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6812/373787000 conn(0x7ff3481271c0 0x7ff348127620 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.001+0000 7ff33effd700 10 client.38180.objecter ms_handle_connect 0x7ff3481271c0 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.002+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7ff348127c10 con 0x7ff3481271c0 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.002+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.002+0000 7ff33effd700 10 client.38180.objecter _finish_command 15 = system:0 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.002+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348126b40 osd=7 4 2024-02-23T00:41:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.002+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 7 15 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 10 client.38180.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign -1 16 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348126b40 osd=7 3 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348126b40 osd=7 3 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 20 client.38180.objecter _get_session s=0x7ff348126b40 osd=7 3 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 15 client.38180.objecter _session_command_op_remove -1 16 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 20 client.38180.objecter get_session s=0x7ff348126b40 osd=7 3 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 15 client.38180.objecter _session_command_op_assign 7 16 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348126b40 osd=7 4 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 10 client.38180.objecter _send_command 16 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.016+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ff348127c10 con 0x7ff3481271c0 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff33effd700 1 -- 172.21.15.87:0/4040164517 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7ff348127c10 con 0x7ff3481271c0 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff33effd700 10 client.38180.objecter ms_dispatch 0x7ff348001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:41.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff33effd700 10 client.38180.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff33effd700 20 client.38180.objecter put_session s=0x7ff348126b40 osd=7 4 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff33effd700 15 client.38180.objecter _session_command_op_remove 7 16 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 10 librados: watch_flush enter 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 10 librados: watch_flush exit 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 10 client.38180.objecter close_session for osd.0 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6801/484081831 conn(0x7ff348028540 msgr2=0x7ff3480fd9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6801/484081831 conn(0x7ff348028540 0x7ff3480fd9b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34805ee80 osd=0 1 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 10 client.38180.objecter close_session for osd.1 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6805/685354171 conn(0x7ff348027b90 msgr2=0x7ff348027f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6805/685354171 conn(0x7ff348027b90 0x7ff348027f70 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff3480ff020 osd=1 1 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 10 client.38180.objecter close_session for osd.2 2024-02-23T00:41:41.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff3480feb80 msgr2=0x7ff34811e460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6809/3605394228 conn(0x7ff3480feb80 0x7ff34811e460 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348027a10 osd=2 1 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 10 client.38180.objecter close_session for osd.3 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6813/128059790 conn(0x7ff34811fad0 msgr2=0x7ff34811ff10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34e08c700 1 -- 172.21.15.87:0/4040164517 reap_dead start 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.018+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6813/128059790 conn(0x7ff34811fad0 0x7ff34811ff10 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff34811baa0 osd=3 1 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 10 client.38180.objecter close_session for osd.4 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff348123340 msgr2=0x7ff348125730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6800/4175844757 conn(0x7ff348123340 0x7ff348125730 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120400 osd=4 1 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 10 client.38180.objecter close_session for osd.5 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff348126130 msgr2=0x7ff348126590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6804/2550013795 conn(0x7ff348126130 0x7ff348126590 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348120b40 osd=5 1 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 10 client.38180.objecter close_session for osd.6 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff34812a910 msgr2=0x7ff34812ad70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6808/2509408498 conn(0x7ff34812a910 0x7ff34812ad70 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348128160 osd=6 1 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 10 client.38180.objecter close_session for osd.7 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6812/373787000 conn(0x7ff3481271c0 msgr2=0x7ff348127620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.171:6812/373787000 conn(0x7ff3481271c0 0x7ff348127620 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 20 client.38180.objecter put_session s=0x7ff348126b40 osd=7 1 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 20 client.38180.objecter shutdown clearing up homeless session... 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 10 client.38180.objecter successfully canceled tick 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34e08c700 1 -- 172.21.15.87:0/4040164517 reap_dead start 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff334062c90 msgr2=0x7ff334065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff334062c90 0x7ff334065150 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 rx=0x7ff344005010 tx=0x7ff344009cb0).stop 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fd510 msgr2=0x7ff348107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:41.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fd510 0x7ff348107b00 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 rx=0x7ff340001df0 tx=0x7ff340004d10).stop 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 shutdown_connections 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> v2:172.21.15.87:6800/1231597463 conn(0x7ff334062c90 0x7ff334065150 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 --2- 172.21.15.87:0/4040164517 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7ff3480fd510 0x7ff348107b00 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 >> 172.21.15.87:0/4040164517 conn(0x7ff3480f2820 msgr2=0x7ff3480fb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 shutdown_connections 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 -- 172.21.15.87:0/4040164517 wait complete. 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:41.019+0000 7ff34faef700 1 librados: shutdown 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.272 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:41:41.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:41:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:40.168525+0000 osd.1 (osd.1) 310 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:41:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:40.171622+0000 osd.1 (osd.1) 311 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:41:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:40.656363+0000 mon.a (mon.0) 3514 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:40.656416+0000 mon.a (mon.0) 3515 : cluster [WRN] Health check failed: Degraded data redundancy: 3/720 objects degraded (0.417%), 1 pg degraded (PG_DEGRADED) 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:40.760477+0000 osd.4 (osd.4) 307 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:40.761461+0000 osd.4 (osd.4) 308 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:40.878373+0000 mon.c (mon.2) 485 : audit [DBG] from='client.? 172.21.15.87:0/4040164517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:41.181121+0000 mgr.y (mgr.24437) 589 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:41.209435+0000 mon.a (mon.0) 3516 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:40.168525+0000 osd.1 (osd.1) 310 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:40.171622+0000 osd.1 (osd.1) 311 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:40.656363+0000 mon.a (mon.0) 3514 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:40.656416+0000 mon.a (mon.0) 3515 : cluster [WRN] Health check failed: Degraded data redundancy: 3/720 objects degraded (0.417%), 1 pg degraded (PG_DEGRADED) 2024-02-23T00:41:41.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:40.760477+0000 osd.4 (osd.4) 307 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:41:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:40.761461+0000 osd.4 (osd.4) 308 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:41:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:40.878373+0000 mon.c (mon.2) 485 : audit [DBG] from='client.? 172.21.15.87:0/4040164517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:41.181121+0000 mgr.y (mgr.24437) 589 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:41.209435+0000 mon.a (mon.0) 3516 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-02-23T00:41:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:40.168525+0000 osd.1 (osd.1) 310 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:41:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:40.171622+0000 osd.1 (osd.1) 311 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:41:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:40.656363+0000 mon.a (mon.0) 3514 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:41:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:40.656416+0000 mon.a (mon.0) 3515 : cluster [WRN] Health check failed: Degraded data redundancy: 3/720 objects degraded (0.417%), 1 pg degraded (PG_DEGRADED) 2024-02-23T00:41:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:40.760477+0000 osd.4 (osd.4) 307 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:41:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:40.761461+0000 osd.4 (osd.4) 308 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:41:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:40.878373+0000 mon.c (mon.2) 485 : audit [DBG] from='client.? 172.21.15.87:0/4040164517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:41.181121+0000 mgr.y (mgr.24437) 589 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:41.209435+0000 mon.a (mon.0) 3516 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-02-23T00:41:42.041 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:42.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:42.145 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:42.177 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:42.198 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:42.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:42.302 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:42.330 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:41:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:42] "GET /metrics HTTP/1.1" 200 190040 "" "Prometheus/2.33.4" 2024-02-23T00:41:42.332 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:42.517 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-23T00:41:42.518 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:41:42.518 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-23T00:41:42.518 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.577575 2024-02-23T00:41:42.518 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 4 0.5775746836831981 2024-02-23T00:41:42.580 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:42.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:42.684 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:42.710 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:43.138 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:43.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:43.242 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:43.271 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:43.293 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:43.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:43.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.393+0000 7f6e564f1700 1 -- 172.21.15.87:0/2007838895 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 msgr2=0x7f6e50104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:43.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.393+0000 7f6e564f1700 1 --2- 172.21.15.87:0/2007838895 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 0x7f6e50104a20 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 rx=0x7f6e48009c30 tx=0x7f6e48004c50).stop 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.393+0000 7f6e564f1700 1 -- 172.21.15.87:0/2007838895 shutdown_connections 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.393+0000 7f6e564f1700 1 --2- 172.21.15.87:0/2007838895 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 0x7f6e50104a20 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.393+0000 7f6e564f1700 1 --2- 172.21.15.87:0/2007838895 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e500fdbb0 0x7f6e500fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.393+0000 7f6e564f1700 1 --2- 172.21.15.87:0/2007838895 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e500fe510 0x7f6e500f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.393+0000 7f6e564f1700 1 -- 172.21.15.87:0/2007838895 >> 172.21.15.87:0/2007838895 conn(0x7f6e500f3800 msgr2=0x7f6e500f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.393+0000 7f6e564f1700 1 -- 172.21.15.87:0/2007838895 shutdown_connections 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.393+0000 7f6e564f1700 1 -- 172.21.15.87:0/2007838895 wait complete. 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 librados: starting msgr at 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 librados: starting objecter 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 Processor -- start 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 -- start start 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 librados: setting wanted keys 2024-02-23T00:41:43.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 librados: calling monclient init 2024-02-23T00:41:43.398 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:43.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 0x7f6e5010ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e500fdbb0 0x7f6e5010e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e500fe510 0x7f6e50107f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e50108480 con 0x7f6e500fe510 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e501085f0 con 0x7f6e500fdbb0 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e564f1700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e501087b0 con 0x7f6e500f8420 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e4ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 0x7f6e5010ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e54a8e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e500fe510 0x7f6e50107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e4ffff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 0x7f6e5010ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:45386/0 (socket says 172.21.15.87:45386) 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4ffff700 1 -- 172.21.15.87:0/4043619876 learned_addr learned my addr 172.21.15.87:0/4043619876 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.394+0000 7f6e54a8e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e500fe510 0x7f6e50107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54138/0 (socket says 172.21.15.87:54138) 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4f7fe700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e500fdbb0 0x7f6e5010e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:43.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4ffff700 1 -- 172.21.15.87:0/4043619876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e500fdbb0 msgr2=0x7f6e5010e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4ffff700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e500fdbb0 0x7f6e5010e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4ffff700 1 -- 172.21.15.87:0/4043619876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e500fe510 msgr2=0x7f6e50107f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4ffff700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e500fe510 0x7f6e50107f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4ffff700 1 -- 172.21.15.87:0/4043619876 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e44005040 con 0x7f6e500f8420 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4f7fe700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e500fdbb0 0x7f6e5010e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e54a8e700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e500fe510 0x7f6e50107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4ffff700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 0x7f6e5010ddf0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f6e4000c7c0 tx=0x7f6e40007ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4d7fa700 10 client.?.objecter ms_handle_connect 0x7f6e500f8420 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4d7fa700 1 -- 172.21.15.87:0/4043619876 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e40004430 con 0x7f6e500f8420 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4d7fa700 1 -- 172.21.15.87:0/4043619876 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4000e970 con 0x7f6e500f8420 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e480097c0 con 0x7f6e500f8420 2024-02-23T00:41:43.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e4d7fa700 1 -- 172.21.15.87:0/4043619876 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e40013a50 con 0x7f6e500f8420 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e564f1700 10 client.38186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e50108df0 con 0x7f6e500f8420 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.395+0000 7f6e564f1700 1 librados: init done 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.396+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e50107110 con 0x7f6e500f8420 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.396+0000 7f6e4d7fa700 1 -- 172.21.15.87:0/4043619876 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f6e4000eae0 con 0x7f6e500f8420 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.397+0000 7f6e4d7fa700 1 --2- 172.21.15.87:0/4043619876 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6e38062c90 0x7f6e38065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.397+0000 7f6e4d7fa700 1 -- 172.21.15.87:0/4043619876 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(684..684 src has 173..684) v4 ==== 8581+0+0 (secure 0 0 0) 0x7f6e4007d670 con 0x7f6e500f8420 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.397+0000 7f6e4d7fa700 10 client.38186.objecter ms_dispatch 0x7f6e50001550 osd_map(684..684 src has 173..684) v4 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.397+0000 7f6e4d7fa700 3 client.38186.objecter handle_osd_map got epochs [684,684] > 0 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.397+0000 7f6e4d7fa700 3 client.38186.objecter handle_osd_map decoding full epoch 684 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.397+0000 7f6e4f7fe700 1 --2- 172.21.15.87:0/4043619876 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6e38062c90 0x7f6e38065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.397+0000 7f6e4d7fa700 20 client.38186.objecter dump_active .. 0 homeless 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.397+0000 7f6e4f7fe700 1 --2- 172.21.15.87:0/4043619876 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6e38062c90 0x7f6e38065150 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 rx=0x7f6e44009f20 tx=0x7f6e44009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:43.403 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.397+0000 7f6e4d7fa700 10 client.38186.objecter ms_handle_connect 0x7f6e38062c90 2024-02-23T00:41:43.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.401+0000 7f6e4d7fa700 1 -- 172.21.15.87:0/4043619876 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6e40050930 con 0x7f6e500f8420 2024-02-23T00:41:43.426 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:43.446 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:43.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:43.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:43.549 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:43.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:41.770231+0000 mgr.y (mgr.24437) 590 : cluster [DBG] pgmap v1040: 162 pgs: 1 peering, 1 activating+degraded, 1 active+recovering, 3 active+remapped+backfill_toofull, 156 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 3/723 objects degraded (0.415%); 14/723 objects misplaced (1.936%); 436 B/s, 9 objects/s recovering 2024-02-23T00:41:43.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:41.954591+0000 osd.5 (osd.5) 259 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:41:43.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:41.955580+0000 osd.5 (osd.5) 260 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:41:43.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:42.157096+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:41:43.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:42.160182+0000 osd.1 (osd.1) 313 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:41:43.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:42.208247+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-02-23T00:41:43.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:42.212146+0000 mon.a (mon.0) 3518 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:43.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:42.606594+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:41:43.575 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:42.607634+0000 osd.6 (osd.6) 277 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:41:43.576 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:43.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:43.577+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f6e5005fb30 con 0x7f6e500f8420 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:41.770231+0000 mgr.y (mgr.24437) 590 : cluster [DBG] pgmap v1040: 162 pgs: 1 peering, 1 activating+degraded, 1 active+recovering, 3 active+remapped+backfill_toofull, 156 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 3/723 objects degraded (0.415%); 14/723 objects misplaced (1.936%); 436 B/s, 9 objects/s recovering 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:41.954591+0000 osd.5 (osd.5) 259 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:41.955580+0000 osd.5 (osd.5) 260 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:42.157096+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:42.160182+0000 osd.1 (osd.1) 313 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:42.208247+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:42.212146+0000 mon.a (mon.0) 3518 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:42.606594+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:42.607634+0000 osd.6 (osd.6) 277 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:41:43.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:41.770231+0000 mgr.y (mgr.24437) 590 : cluster [DBG] pgmap v1040: 162 pgs: 1 peering, 1 activating+degraded, 1 active+recovering, 3 active+remapped+backfill_toofull, 156 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 3/723 objects degraded (0.415%); 14/723 objects misplaced (1.936%); 436 B/s, 9 objects/s recovering 2024-02-23T00:41:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:41.954591+0000 osd.5 (osd.5) 259 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:41:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:41.955580+0000 osd.5 (osd.5) 260 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:41:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:42.157096+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:41:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:42.160182+0000 osd.1 (osd.1) 313 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:41:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:42.208247+0000 mon.a (mon.0) 3517 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-02-23T00:41:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:42.212146+0000 mon.a (mon.0) 3518 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:42.606594+0000 osd.6 (osd.6) 276 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:41:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:42.607634+0000 osd.6 (osd.6) 277 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:41:43.596 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:43.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:43.699 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:43.726 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:43.745 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:43.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:43.849 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:43.874 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:41:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:41:43.875 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:43.894 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:43.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:43.999 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:44.002 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:41:44.002 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:41:44.013 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:41:44.025 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:44.043 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:44.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:44.147 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:44.173 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:44.192 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:44.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:44.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:44.210 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.208+0000 7f6e4d7fa700 1 -- 172.21.15.87:0/4043619876 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v685) v1 ==== 101+0+0 (secure 0 0 0) 0x7f6e40050520 con 0x7f6e500f8420 2024-02-23T00:41:44.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.210+0000 7f6e564f1700 10 librados: watch_flush enter 2024-02-23T00:41:44.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.210+0000 7f6e564f1700 10 librados: watch_flush exit 2024-02-23T00:41:44.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.210+0000 7f6e564f1700 20 client.38186.objecter shutdown clearing up homeless session... 2024-02-23T00:41:44.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.210+0000 7f6e564f1700 10 client.38186.objecter successfully canceled tick 2024-02-23T00:41:44.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.210+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6e38062c90 msgr2=0x7f6e38065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:44.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.210+0000 7f6e564f1700 1 --2- 172.21.15.87:0/4043619876 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6e38062c90 0x7f6e38065150 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 rx=0x7f6e44009f20 tx=0x7f6e44009cb0).stop 2024-02-23T00:41:44.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.210+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 msgr2=0x7f6e5010ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:44.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.210+0000 7f6e564f1700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 0x7f6e5010ddf0 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 rx=0x7f6e4000c7c0 tx=0x7f6e40007ef0).stop 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.211+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 shutdown_connections 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.211+0000 7f6e564f1700 1 --2- 172.21.15.87:0/4043619876 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6e38062c90 0x7f6e38065150 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.211+0000 7f6e564f1700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6e500fe510 0x7f6e50107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.211+0000 7f6e564f1700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6e500f8420 0x7f6e5010ddf0 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.211+0000 7f6e564f1700 1 --2- 172.21.15.87:0/4043619876 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f6e500fdbb0 0x7f6e5010e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.211+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 >> 172.21.15.87:0/4043619876 conn(0x7f6e500f3800 msgr2=0x7f6e500f4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.211+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 shutdown_connections 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.211+0000 7f6e564f1700 1 -- 172.21.15.87:0/4043619876 wait complete. 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:44.211+0000 7f6e564f1700 1 librados: shutdown 2024-02-23T00:41:44.214 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:41:44.296 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:44.322 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:42.603537+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:41:44.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:42.606630+0000 osd.7 (osd.7) 301 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:41:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:43.210169+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:43.210274+0000 mon.a (mon.0) 3520 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-02-23T00:41:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:43.212315+0000 mon.a (mon.0) 3521 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:43.490976+0000 osd.2 (osd.2) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:43.491266+0000 osd.2 (osd.2) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:43.579119+0000 mon.c (mon.2) 486 : audit [INF] from='client.? 172.21.15.87:0/4043619876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:44.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:43.579406+0000 mon.a (mon.0) 3522 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:44.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:43.615673+0000 osd.6 (osd.6) 278 : cluster [DBG] 234.2 scrub starts 2024-02-23T00:41:44.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:43.619115+0000 osd.6 (osd.6) 279 : cluster [DBG] 234.2 scrub ok 2024-02-23T00:41:44.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.204493+0000 mon.a (mon.0) 3523 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:41:44.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:44.209086+0000 mon.a (mon.0) 3524 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:44.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:44.209147+0000 mon.a (mon.0) 3525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:41:44.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.209209+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-02-23T00:41:44.336 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:44.210666+0000 mon.a (mon.0) 3527 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-118"}]: dispatch 2024-02-23T00:41:44.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:42.603537+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:41:44.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:42.606630+0000 osd.7 (osd.7) 301 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:43.210169+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:43.210274+0000 mon.a (mon.0) 3520 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:43.212315+0000 mon.a (mon.0) 3521 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:43.490976+0000 osd.2 (osd.2) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:43.491266+0000 osd.2 (osd.2) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:43.579119+0000 mon.c (mon.2) 486 : audit [INF] from='client.? 172.21.15.87:0/4043619876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:43.579406+0000 mon.a (mon.0) 3522 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:43.615673+0000 osd.6 (osd.6) 278 : cluster [DBG] 234.2 scrub starts 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:43.619115+0000 osd.6 (osd.6) 279 : cluster [DBG] 234.2 scrub ok 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.204493+0000 mon.a (mon.0) 3523 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:44.209086+0000 mon.a (mon.0) 3524 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:44.209147+0000 mon.a (mon.0) 3525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:41:44.337 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.209209+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-02-23T00:41:44.338 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:44.210666+0000 mon.a (mon.0) 3527 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-118"}]: dispatch 2024-02-23T00:41:44.340 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:44.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:42.603537+0000 osd.7 (osd.7) 300 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:41:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:42.606630+0000 osd.7 (osd.7) 301 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:41:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:43.210169+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:44.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:43.210274+0000 mon.a (mon.0) 3520 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:43.212315+0000 mon.a (mon.0) 3521 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:43.490976+0000 osd.2 (osd.2) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:43.491266+0000 osd.2 (osd.2) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:43.579119+0000 mon.c (mon.2) 486 : audit [INF] from='client.? 172.21.15.87:0/4043619876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:43.579406+0000 mon.a (mon.0) 3522 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:43.615673+0000 osd.6 (osd.6) 278 : cluster [DBG] 234.2 scrub starts 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:43.619115+0000 osd.6 (osd.6) 279 : cluster [DBG] 234.2 scrub ok 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.204493+0000 mon.a (mon.0) 3523 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:44.209086+0000 mon.a (mon.0) 3524 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:44.209147+0000 mon.a (mon.0) 3525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.209209+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-02-23T00:41:44.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:44.210666+0000 mon.a (mon.0) 3527 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-118"}]: dispatch 2024-02-23T00:41:44.444 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:44.469 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:44.488 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:44.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:44.592 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:44.620 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:45.069 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:45.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:45.106 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:45.106 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:45.174 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:45.203 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:45.226 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:45.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:45.331 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:43.770728+0000 mgr.y (mgr.24437) 591 : cluster [DBG] pgmap v1043: 194 pgs: 5 creating+activating, 27 unknown, 1 peering, 3 active+remapped+backfill_toofull, 158 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 11/723 objects misplaced (1.521%); 464 B/s, 14 objects/s recovering 2024-02-23T00:41:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:43.983707+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:41:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:43.984486+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:41:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.193174+0000 osd.1 (osd.1) 314 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:41:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.196084+0000 osd.1 (osd.1) 315 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:41:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.219690+0000 mon.a (mon.0) 3528 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/723 objects degraded (0.415%), 1 pg degraded) 2024-02-23T00:41:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.256755+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:41:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.257800+0000 osd.2 (osd.2) 246 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:45.210411+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-118"}]': finished 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:45.210466+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:45.211062+0000 mon.a (mon.0) 3531 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-118", "mode": "writeback"}]: dispatch 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:43.770728+0000 mgr.y (mgr.24437) 591 : cluster [DBG] pgmap v1043: 194 pgs: 5 creating+activating, 27 unknown, 1 peering, 3 active+remapped+backfill_toofull, 158 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 11/723 objects misplaced (1.521%); 464 B/s, 14 objects/s recovering 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:43.983707+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:43.984486+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.193174+0000 osd.1 (osd.1) 314 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.196084+0000 osd.1 (osd.1) 315 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.219690+0000 mon.a (mon.0) 3528 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/723 objects degraded (0.415%), 1 pg degraded) 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.256755+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:41:45.335 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.257800+0000 osd.2 (osd.2) 246 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:41:45.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:45.210411+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-118"}]': finished 2024-02-23T00:41:45.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:45.210466+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-02-23T00:41:45.336 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:45.211062+0000 mon.a (mon.0) 3531 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-118", "mode": "writeback"}]: dispatch 2024-02-23T00:41:45.357 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:45.556 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:45.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:45.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:45.577 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-23T00:41:45.663 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:45.689 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:43.770728+0000 mgr.y (mgr.24437) 591 : cluster [DBG] pgmap v1043: 194 pgs: 5 creating+activating, 27 unknown, 1 peering, 3 active+remapped+backfill_toofull, 158 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 11/723 objects misplaced (1.521%); 464 B/s, 14 objects/s recovering 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:43.983707+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:43.984486+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.193174+0000 osd.1 (osd.1) 314 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.196084+0000 osd.1 (osd.1) 315 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.219690+0000 mon.a (mon.0) 3528 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/723 objects degraded (0.415%), 1 pg degraded) 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.256755+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.257800+0000 osd.2 (osd.2) 246 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:45.210411+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-118"}]': finished 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:45.210466+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-02-23T00:41:45.690 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:45.211062+0000 mon.a (mon.0) 3531 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-118", "mode": "writeback"}]: dispatch 2024-02-23T00:41:45.691 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:45.710 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:45.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:45.814 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:45.842 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:45.861 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:45.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:45.965 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:45.993 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:46.012 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:46.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:46.116 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:46.142 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:46.161 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:46.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:46.265 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:46.289 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.516621+0000 osd.7 (osd.7) 302 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:41:46.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.520408+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:41:46.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.732170+0000 osd.0 (osd.0) 3 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:41:46.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.733021+0000 osd.0 (osd.0) 4 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:41:46.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.787734+0000 osd.4 (osd.4) 309 : cluster [DBG] 4.d scrub starts 2024-02-23T00:41:46.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:44.791359+0000 osd.4 (osd.4) 310 : cluster [DBG] 4.d scrub ok 2024-02-23T00:41:46.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:46.207723+0000 mon.a (mon.0) 3532 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:46.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:46.211251+0000 mon.a (mon.0) 3533 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-118", "mode": "writeback"}]': finished 2024-02-23T00:41:46.290 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:46.211291+0000 mon.a (mon.0) 3534 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-02-23T00:41:46.291 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:46.310 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:46.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:46.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.516621+0000 osd.7 (osd.7) 302 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:41:46.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.520408+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:41:46.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.732170+0000 osd.0 (osd.0) 3 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:41:46.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.733021+0000 osd.0 (osd.0) 4 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:41:46.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.787734+0000 osd.4 (osd.4) 309 : cluster [DBG] 4.d scrub starts 2024-02-23T00:41:46.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:44.791359+0000 osd.4 (osd.4) 310 : cluster [DBG] 4.d scrub ok 2024-02-23T00:41:46.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:46.207723+0000 mon.a (mon.0) 3532 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:46.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:46.211251+0000 mon.a (mon.0) 3533 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-118", "mode": "writeback"}]': finished 2024-02-23T00:41:46.329 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:46.211291+0000 mon.a (mon.0) 3534 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-02-23T00:41:46.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.516621+0000 osd.7 (osd.7) 302 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:41:46.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.520408+0000 osd.7 (osd.7) 303 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:41:46.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.732170+0000 osd.0 (osd.0) 3 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:41:46.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.733021+0000 osd.0 (osd.0) 4 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:41:46.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.787734+0000 osd.4 (osd.4) 309 : cluster [DBG] 4.d scrub starts 2024-02-23T00:41:46.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:44.791359+0000 osd.4 (osd.4) 310 : cluster [DBG] 4.d scrub ok 2024-02-23T00:41:46.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:46.207723+0000 mon.a (mon.0) 3532 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:46.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:46.211251+0000 mon.a (mon.0) 3533 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-118", "mode": "writeback"}]': finished 2024-02-23T00:41:46.330 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:46.211291+0000 mon.a (mon.0) 3534 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-02-23T00:41:46.414 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:46.440 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:46.460 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:46.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:46.563 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:46.590 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:46.608 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:46.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:46.712 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:46.741 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:47.064 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:41:47.089 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:47.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:47.194 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:47.222 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:45.771580+0000 mgr.y (mgr.24437) 592 : cluster [DBG] pgmap v1046: 194 pgs: 5 creating+activating, 3 active+remapped+backfill_toofull, 186 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11/723 objects misplaced (1.521%); 27 B/s, 4 objects/s recovering 2024-02-23T00:41:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:46.864996+0000 osd.4 (osd.4) 311 : cluster [DBG] 4.e scrub starts 2024-02-23T00:41:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:46.868724+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.e scrub ok 2024-02-23T00:41:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:47.202088+0000 mon.a (mon.0) 3535 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:41:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:45.771580+0000 mgr.y (mgr.24437) 592 : cluster [DBG] pgmap v1046: 194 pgs: 5 creating+activating, 3 active+remapped+backfill_toofull, 186 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11/723 objects misplaced (1.521%); 27 B/s, 4 objects/s recovering 2024-02-23T00:41:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:46.864996+0000 osd.4 (osd.4) 311 : cluster [DBG] 4.e scrub starts 2024-02-23T00:41:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:46.868724+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.e scrub ok 2024-02-23T00:41:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:47.202088+0000 mon.a (mon.0) 3535 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:41:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:45.771580+0000 mgr.y (mgr.24437) 592 : cluster [DBG] pgmap v1046: 194 pgs: 5 creating+activating, 3 active+remapped+backfill_toofull, 186 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11/723 objects misplaced (1.521%); 27 B/s, 4 objects/s recovering 2024-02-23T00:41:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:46.864996+0000 osd.4 (osd.4) 311 : cluster [DBG] 4.e scrub starts 2024-02-23T00:41:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:46.868724+0000 osd.4 (osd.4) 312 : cluster [DBG] 4.e scrub ok 2024-02-23T00:41:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:47.202088+0000 mon.a (mon.0) 3535 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:41:48.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:47.252366+0000 osd.1 (osd.1) 316 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:41:48.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:47.254324+0000 osd.1 (osd.1) 317 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:41:48.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:47.543961+0000 osd.7 (osd.7) 304 : cluster [DBG] 2.b scrub starts 2024-02-23T00:41:48.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:47.544948+0000 osd.7 (osd.7) 305 : cluster [DBG] 2.b scrub ok 2024-02-23T00:41:48.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:47.252366+0000 osd.1 (osd.1) 316 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:41:48.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:47.254324+0000 osd.1 (osd.1) 317 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:41:48.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:47.543961+0000 osd.7 (osd.7) 304 : cluster [DBG] 2.b scrub starts 2024-02-23T00:41:48.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:47.544948+0000 osd.7 (osd.7) 305 : cluster [DBG] 2.b scrub ok 2024-02-23T00:41:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 -- 172.21.15.87:0/3855353385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f151c0f7400 msgr2=0x7f151c103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:48.443 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 --2- 172.21.15.87:0/3855353385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f151c0f7400 0x7f151c103a20 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 rx=0x7f1514009c30 tx=0x7f1514004c50).stop 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 -- 172.21.15.87:0/3855353385 shutdown_connections 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 --2- 172.21.15.87:0/3855353385 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f151c0f7400 0x7f151c103a20 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 --2- 172.21.15.87:0/3855353385 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f151c0fcbb0 0x7f151c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 --2- 172.21.15.87:0/3855353385 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0fd510 0x7f151c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 -- 172.21.15.87:0/3855353385 >> 172.21.15.87:0/3855353385 conn(0x7f151c0f2820 msgr2=0x7f151c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 -- 172.21.15.87:0/3855353385 shutdown_connections 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 -- 172.21.15.87:0/3855353385 wait complete. 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 librados: starting msgr at 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.387+0000 7f1520d0a700 1 librados: starting objecter 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 Processor -- start 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 -- start start 2024-02-23T00:41:48.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 librados: setting wanted keys 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 librados: calling monclient init 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f7400 0x7f151c0f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f151c0fcbb0 0x7f151c0f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f151c0fd510 0x7f151c0f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c116430 con 0x7f151c0fcbb0 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c1165a0 con 0x7f151c0f7400 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1520d0a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c116760 con 0x7f151c0fd510 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151ad9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f151c0fd510 0x7f151c0f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1519d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f151c0fcbb0 0x7f151c0f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151ad9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f151c0fd510 0x7f151c0f9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:45408/0 (socket says 172.21.15.87:45408) 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151ad9d700 1 -- 172.21.15.87:0/1248489275 learned_addr learned my addr 172.21.15.87:0/1248489275 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151a59c700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f7400 0x7f151c0f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151ad9d700 1 -- 172.21.15.87:0/1248489275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f7400 msgr2=0x7f151c0f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:48.445 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151ad9d700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f7400 0x7f151c0f89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151ad9d700 1 -- 172.21.15.87:0/1248489275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f151c0fcbb0 msgr2=0x7f151c0f8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151ad9d700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f151c0fcbb0 0x7f151c0f8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151ad9d700 1 -- 172.21.15.87:0/1248489275 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1510005040 con 0x7f151c0fd510 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f151a59c700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f7400 0x7f151c0f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.388+0000 7f1519d9b700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f151c0fcbb0 0x7f151c0f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f151ad9d700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f151c0fd510 0x7f151c0f9390 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f1514004e10 tx=0x7f1514004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f150b7fe700 10 client.?.objecter ms_handle_connect 0x7f151c0fd510 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f150b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f150b7fe700 1 -- 172.21.15.87:0/1248489275 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151400e040 con 0x7f151c0fd510 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f150b7fe700 1 -- 172.21.15.87:0/1248489275 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1514018040 con 0x7f151c0fd510 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15140097c0 con 0x7f151c0fd510 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f150b7fe700 1 -- 172.21.15.87:0/1248489275 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514003b30 con 0x7f151c0fd510 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f1520d0a700 10 client.38192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:48.446 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c116ba0 con 0x7f151c0fd510 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.389+0000 7f1520d0a700 1 librados: init done 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.390+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c05fa60 con 0x7f151c0fd510 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.390+0000 7f150b7fe700 1 -- 172.21.15.87:0/1248489275 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1514007bf0 con 0x7f151c0fd510 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.390+0000 7f150b7fe700 1 --2- 172.21.15.87:0/1248489275 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1504062c90 0x7f1504065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.390+0000 7f150b7fe700 1 -- 172.21.15.87:0/1248489275 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(687..687 src has 173..687) v4 ==== 8576+0+0 (secure 0 0 0) 0x7f151407e550 con 0x7f151c0fd510 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.390+0000 7f150b7fe700 10 client.38192.objecter ms_dispatch 0x7f151c001550 osd_map(687..687 src has 173..687) v4 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.390+0000 7f150b7fe700 3 client.38192.objecter handle_osd_map got epochs [687,687] > 0 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.390+0000 7f150b7fe700 3 client.38192.objecter handle_osd_map decoding full epoch 687 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.391+0000 7f151a59c700 1 --2- 172.21.15.87:0/1248489275 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1504062c90 0x7f1504065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.391+0000 7f150b7fe700 20 client.38192.objecter dump_active .. 0 homeless 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.391+0000 7f150b7fe700 10 client.38192.objecter ms_handle_connect 0x7f1504062c90 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.391+0000 7f151a59c700 1 --2- 172.21.15.87:0/1248489275 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1504062c90 0x7f1504065150 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7f1510009f20 tx=0x7f1510009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:48.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.395+0000 7f150b7fe700 1 -- 172.21.15.87:0/1248489275 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1514051820 con 0x7f151c0fd510 2024-02-23T00:41:48.451 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:48.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:48.482 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:48.557 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:48.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:47.252366+0000 osd.1 (osd.1) 316 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:41:48.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:47.254324+0000 osd.1 (osd.1) 317 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:41:48.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:47.543961+0000 osd.7 (osd.7) 304 : cluster [DBG] 2.b scrub starts 2024-02-23T00:41:48.587 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:47.544948+0000 osd.7 (osd.7) 305 : cluster [DBG] 2.b scrub ok 2024-02-23T00:41:48.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:48.585+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981} v 0) v1 -- 0x7f151c05ee80 con 0x7f151c0fd510 2024-02-23T00:41:48.589 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:48.610 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:48.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:48.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:48.714 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:48.752 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:49.004 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:49.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:49.108 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:41:49.135 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:49.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.248+0000 7f150b7fe700 1 -- 172.21.15.87:0/1248489275 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]=0 set osd.4 primary-affinity to 0.577575 (8378512) v688) v1 ==== 155+0+0 (secure 0 0 0) 0x7f1514051410 con 0x7f151c0fd510 2024-02-23T00:41:49.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.251+0000 7f1520d0a700 10 librados: watch_flush enter 2024-02-23T00:41:49.254 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:47.716618+0000 osd.0 (osd.0) 5 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:41:49.254 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:47.719586+0000 osd.0 (osd.0) 6 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:41:49.254 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:47.771986+0000 mgr.y (mgr.24437) 593 : cluster [DBG] pgmap v1048: 194 pgs: 5 creating+activating, 3 active+remapped+backfill_toofull, 186 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11/723 objects misplaced (1.521%); 24 B/s, 3 objects/s recovering 2024-02-23T00:41:49.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:48.294303+0000 osd.3 (osd.3) 216 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:41:49.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:48.297216+0000 osd.3 (osd.3) 217 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:41:49.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:48.587194+0000 mon.c (mon.2) 487 : audit [INF] from='client.? 172.21.15.87:0/1248489275' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]: dispatch 2024-02-23T00:41:49.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:48.587511+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]: dispatch 2024-02-23T00:41:49.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:48.629187+0000 osd.6 (osd.6) 280 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:41:49.255 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:48.632095+0000 osd.6 (osd.6) 281 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:41:49.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.251+0000 7f1520d0a700 10 librados: watch_flush exit 2024-02-23T00:41:49.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.251+0000 7f1520d0a700 20 client.38192.objecter shutdown clearing up homeless session... 2024-02-23T00:41:49.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.251+0000 7f1520d0a700 10 client.38192.objecter successfully canceled tick 2024-02-23T00:41:49.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1504062c90 msgr2=0x7f1504065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 --2- 172.21.15.87:0/1248489275 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1504062c90 0x7f1504065150 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 rx=0x7f1510009f20 tx=0x7f1510009cb0).stop 2024-02-23T00:41:49.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f151c0fd510 msgr2=0x7f151c0f9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f151c0fd510 0x7f151c0f9390 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 rx=0x7f1514004e10 tx=0x7f1514004c50).stop 2024-02-23T00:41:49.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 shutdown_connections 2024-02-23T00:41:49.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 --2- 172.21.15.87:0/1248489275 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1504062c90 0x7f1504065150 secure :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 rx=0x7f1510009f20 tx=0x7f1510009cb0).stop 2024-02-23T00:41:49.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f151c0fd510 0x7f151c0f9390 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f151c0f7400 0x7f151c0f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 --2- 172.21.15.87:0/1248489275 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f151c0fcbb0 0x7f151c0f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 >> 172.21.15.87:0/1248489275 conn(0x7f151c0f2820 msgr2=0x7f151c060dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:49.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 shutdown_connections 2024-02-23T00:41:49.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 -- 172.21.15.87:0/1248489275 wait complete. 2024-02-23T00:41:49.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.253+0000 7f1520d0a700 1 librados: shutdown 2024-02-23T00:41:49.257 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.4 primary-affinity to 0.577575 (8378512) 2024-02-23T00:41:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:47.716618+0000 osd.0 (osd.0) 5 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:41:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:47.719586+0000 osd.0 (osd.0) 6 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:41:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:47.771986+0000 mgr.y (mgr.24437) 593 : cluster [DBG] pgmap v1048: 194 pgs: 5 creating+activating, 3 active+remapped+backfill_toofull, 186 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11/723 objects misplaced (1.521%); 24 B/s, 3 objects/s recovering 2024-02-23T00:41:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:48.294303+0000 osd.3 (osd.3) 216 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:41:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:48.297216+0000 osd.3 (osd.3) 217 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:41:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:48.587194+0000 mon.c (mon.2) 487 : audit [INF] from='client.? 172.21.15.87:0/1248489275' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]: dispatch 2024-02-23T00:41:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:48.587511+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]: dispatch 2024-02-23T00:41:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:48.629187+0000 osd.6 (osd.6) 280 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:41:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:48.632095+0000 osd.6 (osd.6) 281 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:41:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:47.716618+0000 osd.0 (osd.0) 5 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:41:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:47.719586+0000 osd.0 (osd.0) 6 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:41:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:47.771986+0000 mgr.y (mgr.24437) 593 : cluster [DBG] pgmap v1048: 194 pgs: 5 creating+activating, 3 active+remapped+backfill_toofull, 186 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11/723 objects misplaced (1.521%); 24 B/s, 3 objects/s recovering 2024-02-23T00:41:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:48.294303+0000 osd.3 (osd.3) 216 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:41:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:48.297216+0000 osd.3 (osd.3) 217 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:41:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:48.587194+0000 mon.c (mon.2) 487 : audit [INF] from='client.? 172.21.15.87:0/1248489275' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]: dispatch 2024-02-23T00:41:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:48.587511+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]: dispatch 2024-02-23T00:41:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:48.629187+0000 osd.6 (osd.6) 280 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:41:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:48.632095+0000 osd.6 (osd.6) 281 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:41:49.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 -- 172.21.15.87:0/408680986 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c4102210 msgr2=0x7f05c4104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 --2- 172.21.15.87:0/408680986 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c4102210 0x7f05c4104600 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 rx=0x7f05bc009c30 tx=0x7f05bc004c50).stop 2024-02-23T00:41:49.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 -- 172.21.15.87:0/408680986 shutdown_connections 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 --2- 172.21.15.87:0/408680986 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c4102210 0x7f05c4104600 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 --2- 172.21.15.87:0/408680986 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05c40f8cb0 0x7f05c40f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 --2- 172.21.15.87:0/408680986 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05c40f9610 0x7f05c4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 -- 172.21.15.87:0/408680986 >> 172.21.15.87:0/408680986 conn(0x7f05c40f2820 msgr2=0x7f05c40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 -- 172.21.15.87:0/408680986 shutdown_connections 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 -- 172.21.15.87:0/408680986 wait complete. 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 librados: starting msgr at 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.308+0000 7f05cb467700 1 librados: starting objecter 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 Processor -- start 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 -- start start 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 librados: setting wanted keys 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 librados: calling monclient init 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c40f8cb0 0x7f05c41101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05c40f9610 0x7f05c41106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05c4102210 0x7f05c410a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c410a880 con 0x7f05c40f9610 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c410a9f0 con 0x7f05c4102210 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05cb467700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c410abb0 con 0x7f05c40f8cb0 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c8a02700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05c40f9610 0x7f05c41106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c9203700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c40f8cb0 0x7f05c41101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c8a02700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05c40f9610 0x7f05c41106e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54166/0 (socket says 172.21.15.87:54166) 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c9203700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c40f8cb0 0x7f05c41101f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:45432/0 (socket says 172.21.15.87:45432) 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c8a02700 1 -- 172.21.15.87:0/3318851458 learned_addr learned my addr 172.21.15.87:0/3318851458 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c9a04700 1 --2- 172.21.15.87:0/3318851458 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05c4102210 0x7f05c410a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c9203700 1 -- 172.21.15.87:0/3318851458 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05c4102210 msgr2=0x7f05c410a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c9203700 1 --2- 172.21.15.87:0/3318851458 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f05c4102210 0x7f05c410a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c9203700 1 -- 172.21.15.87:0/3318851458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05c40f9610 msgr2=0x7f05c41106e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c9203700 1 --2- 172.21.15.87:0/3318851458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05c40f9610 0x7f05c41106e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c9203700 1 -- 172.21.15.87:0/3318851458 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b4005040 con 0x7f05c40f8cb0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c8a02700 1 --2- 172.21.15.87:0/3318851458 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f05c40f9610 0x7f05c41106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05c9203700 1 --2- 172.21.15.87:0/3318851458 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c40f8cb0 0x7f05c41101f0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f05c0002760 tx=0x7f05c0009e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05ba7fc700 10 client.?.objecter ms_handle_connect 0x7f05c40f8cb0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.309+0000 7f05ba7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.310+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0007b40 con 0x7f05c40f8cb0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.310+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05c000f040 con 0x7f05c40f8cb0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.310+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0014790 con 0x7f05c40f8cb0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.310+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc0097c0 con 0x7f05c40f8cb0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.310+0000 7f05cb467700 10 client.38198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.310+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c410b1f0 con 0x7f05c40f8cb0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.310+0000 7f05cb467700 1 librados: init done 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.310+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f05c405fa60 con 0x7f05c40f8cb0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.311+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f05c0007ca0 con 0x7f05c40f8cb0 2024-02-23T00:41:49.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.311+0000 7f05ba7fc700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6800/1231597463 conn(0x7f05b0062c90 0x7f05b0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.311+0000 7f05c8a02700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6800/1231597463 conn(0x7f05b0062c90 0x7f05b0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.311+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(688..688 src has 173..688) v4 ==== 8576+0+0 (secure 0 0 0) 0x7f05c007cd60 con 0x7f05c40f8cb0 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.311+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 osd_map(688..688 src has 173..688) v4 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.311+0000 7f05ba7fc700 3 client.38198.objecter handle_osd_map got epochs [688,688] > 0 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.311+0000 7f05ba7fc700 3 client.38198.objecter handle_osd_map decoding full epoch 688 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05ba7fc700 20 client.38198.objecter dump_active .. 0 homeless 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v688) v1 ==== 54+0+16 (secure 0 0 0) 0x7f05c0082420 con 0x7f05c40f8cb0 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05ba7fc700 10 client.38198.objecter ms_handle_connect 0x7f05b0062c90 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05c8a02700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6800/1231597463 conn(0x7f05b0062c90 0x7f05b0065150 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7f05c410bb30 tx=0x7f05b400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 10 client.38198.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 1 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6801/484081831 conn(0x7f05c40fe5a0 0x7f05c40fe980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c405ee80 osd=0 3 2024-02-23T00:41:49.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c405ee80 osd=0 3 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c405ee80 osd=0 3 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 1 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c405ee80 osd=0 3 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 0 1 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c405ee80 osd=0 4 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 10 client.38198.objecter _send_command 1 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05c4107290 con 0x7f05c40fe5a0 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05c9a04700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6801/484081831 conn(0x7f05c40fe5a0 0x7f05c40fe980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05ba7fc700 10 client.38198.objecter ms_handle_connect 0x7f05c40fe5a0 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.312+0000 7f05c9a04700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6801/484081831 conn(0x7f05c40fe5a0 0x7f05c40fe980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.313+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f05c4107290 con 0x7f05c40fe5a0 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.313+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.313+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 1 = system:0 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.313+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c405ee80 osd=0 4 2024-02-23T00:41:49.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.313+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 0 1 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 10 client.38198.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 2 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c405ee80 osd=0 3 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c405ee80 osd=0 3 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c405ee80 osd=0 3 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 2 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c405ee80 osd=0 3 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 0 2 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c405ee80 osd=0 4 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 10 client.38198.objecter _send_command 2 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.328+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f05c40273c0 con 0x7f05c40fe5a0 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.330+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f05c40273c0 con 0x7f05c40fe5a0 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.330+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.330+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.734 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:41:49.734 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:41:49.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.330+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c405ee80 osd=0 4 2024-02-23T00:41:49.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.330+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 0 2 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 10 client.38198.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 3 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6805/685354171 conn(0x7f05c40598f0 0x7f05c4059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4028710 osd=1 3 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4028710 osd=1 3 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4028710 osd=1 3 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 3 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c4028710 osd=1 3 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 1 3 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4028710 osd=1 4 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 10 client.38198.objecter _send_command 3 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05c411d860 con 0x7f05c40598f0 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05c9203700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6805/685354171 conn(0x7f05c40598f0 0x7f05c4059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05ba7fc700 10 client.38198.objecter ms_handle_connect 0x7f05c40598f0 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.331+0000 7f05c9203700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6805/685354171 conn(0x7f05c40598f0 0x7f05c4059cd0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.332+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f05c411d860 con 0x7f05c40598f0 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.332+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.332+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 3 = system:0 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.332+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c4028710 osd=1 4 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.332+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 1 3 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 10 client.38198.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 4 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4028710 osd=1 3 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4028710 osd=1 3 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4028710 osd=1 3 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 4 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c4028710 osd=1 3 2024-02-23T00:41:49.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 1 4 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4028710 osd=1 4 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 10 client.38198.objecter _send_command 4 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.349+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f05c4059360 con 0x7f05c40598f0 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f05c4059360 con 0x7f05c40598f0 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c4028710 osd=1 4 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 1 4 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 10 client.38198.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 5 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6809/3605394228 conn(0x7f05c4028270 0x7f05c411c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4059770 osd=2 3 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4059770 osd=2 3 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4059770 osd=2 3 2024-02-23T00:41:49.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 5 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c4059770 osd=2 3 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 2 5 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4059770 osd=2 4 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 10 client.38198.objecter _send_command 5 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05c411c770 con 0x7f05c4028270 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05c8a02700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6809/3605394228 conn(0x7f05c4028270 0x7f05c411c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05ba7fc700 10 client.38198.objecter ms_handle_connect 0x7f05c4028270 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.351+0000 7f05c8a02700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6809/3605394228 conn(0x7f05c4028270 0x7f05c411c280 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.352+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f05c411c770 con 0x7f05c4028270 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.352+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.352+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 5 = system:0 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.352+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c4059770 osd=2 4 2024-02-23T00:41:49.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.352+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 2 5 2024-02-23T00:41:49.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 10 client.38198.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:49.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 6 2024-02-23T00:41:49.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4059770 osd=2 3 2024-02-23T00:41:49.739 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:41:49.739 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:41:49.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4059770 osd=2 3 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4059770 osd=2 3 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 6 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c4059770 osd=2 3 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 2 6 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4059770 osd=2 4 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 10 client.38198.objecter _send_command 6 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.367+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f05c40277d0 con 0x7f05c4028270 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f05c40277d0 con 0x7f05c4028270 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c4059770 osd=2 4 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 2 6 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 10 client.38198.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:49.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 7 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6813/128059790 conn(0x7f05c411f9e0 0x7f05c4121dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4027be0 osd=3 3 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4027be0 osd=3 3 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4027be0 osd=3 3 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 7 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c4027be0 osd=3 3 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 3 7 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4027be0 osd=3 4 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 10 client.38198.objecter _send_command 7 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05c4124a20 con 0x7f05c411f9e0 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.368+0000 7f05c9a04700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6813/128059790 conn(0x7f05c411f9e0 0x7f05c4121dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.369+0000 7f05ba7fc700 10 client.38198.objecter ms_handle_connect 0x7f05c411f9e0 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.369+0000 7f05c9a04700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6813/128059790 conn(0x7f05c411f9e0 0x7f05c4121dd0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.369+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f05c4124a20 con 0x7f05c411f9e0 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.369+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.369+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 7 = system:0 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.369+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c4027be0 osd=3 4 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.369+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 3 7 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 10 client.38198.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 8 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4027be0 osd=3 3 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4027be0 osd=3 3 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4027be0 osd=3 3 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 8 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c4027be0 osd=3 3 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 3 8 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4027be0 osd=3 4 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 10 client.38198.objecter _send_command 8 2024-02-23T00:41:49.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.384+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f05c411cb80 con 0x7f05c411f9e0 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f05c411cb80 con 0x7f05c411f9e0 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c4027be0 osd=3 4 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 3 8 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 10 client.38198.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 9 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6800/4175844757 conn(0x7f05c411ccf0 0x7f05c4123440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c411cb80 osd=4 3 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411cb80 osd=4 3 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c411cb80 osd=4 3 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 9 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c411cb80 osd=4 3 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 4 9 2024-02-23T00:41:49.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411cb80 osd=4 4 2024-02-23T00:41:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 10 client.38198.objecter _send_command 9 2024-02-23T00:41:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.385+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05c411bf60 con 0x7f05c411ccf0 2024-02-23T00:41:49.744 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:41:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.386+0000 7f05c9203700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6800/4175844757 conn(0x7f05c411ccf0 0x7f05c4123440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.386+0000 7f05ba7fc700 10 client.38198.objecter ms_handle_connect 0x7f05c411ccf0 2024-02-23T00:41:49.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.386+0000 7f05c9203700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6800/4175844757 conn(0x7f05c411ccf0 0x7f05c4123440 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.387+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f05c411bf60 con 0x7f05c411ccf0 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.387+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.387+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 9 = system:0 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.387+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c411cb80 osd=4 4 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.387+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 4 9 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 10 client.38198.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 10 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c411cb80 osd=4 3 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411cb80 osd=4 3 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c411cb80 osd=4 3 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 10 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c411cb80 osd=4 3 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 4 10 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411cb80 osd=4 4 2024-02-23T00:41:49.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 10 client.38198.objecter _send_command 10 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.402+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f05c411bf60 con 0x7f05c411ccf0 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f05c411bf60 con 0x7f05c411ccf0 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c411cb80 osd=4 4 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 4 10 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 10 client.38198.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 11 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6804/2550013795 conn(0x7f05c4123e40 0x7f05c41242a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c411bf60 osd=5 3 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411bf60 osd=5 3 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c411bf60 osd=5 3 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 11 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c411bf60 osd=5 3 2024-02-23T00:41:49.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 5 11 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411bf60 osd=5 4 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 10 client.38198.objecter _send_command 11 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05c4124850 con 0x7f05c4123e40 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05c8a02700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6804/2550013795 conn(0x7f05c4123e40 0x7f05c41242a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 10 client.38198.objecter ms_handle_connect 0x7f05c4123e40 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05c8a02700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6804/2550013795 conn(0x7f05c4123e40 0x7f05c41242a0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f05c4124850 con 0x7f05c4123e40 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 11 = system:0 2024-02-23T00:41:49.747 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:41:49.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c411bf60 osd=5 4 2024-02-23T00:41:49.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.405+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 5 11 2024-02-23T00:41:49.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 10 client.38198.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:49.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 12 2024-02-23T00:41:49.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c411bf60 osd=5 3 2024-02-23T00:41:49.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411bf60 osd=5 3 2024-02-23T00:41:49.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c411bf60 osd=5 3 2024-02-23T00:41:49.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 12 2024-02-23T00:41:49.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c411bf60 osd=5 3 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 5 12 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411bf60 osd=5 4 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 10 client.38198.objecter _send_command 12 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.431+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f05c4124850 con 0x7f05c4123e40 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.432+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f05c4124850 con 0x7f05c4123e40 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.432+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.432+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.432+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c411bf60 osd=5 4 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.432+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 5 12 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 10 client.38198.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 13 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6808/2509408498 conn(0x7f05c4129110 0x7f05c4129550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4124850 osd=6 3 2024-02-23T00:41:49.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4124850 osd=6 3 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4124850 osd=6 3 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 13 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c4124850 osd=6 3 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 6 13 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4124850 osd=6 4 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 10 client.38198.objecter _send_command 13 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05c412b1a0 con 0x7f05c4129110 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05c9a04700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6808/2509408498 conn(0x7f05c4129110 0x7f05c4129550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05ba7fc700 10 client.38198.objecter ms_handle_connect 0x7f05c4129110 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.433+0000 7f05c9a04700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6808/2509408498 conn(0x7f05c4129110 0x7f05c4129550 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.434+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f05c412b1a0 con 0x7f05c4129110 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.434+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.434+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 13 = system:0 2024-02-23T00:41:49.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.434+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c4124850 osd=6 4 2024-02-23T00:41:49.751 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.434+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 6 13 2024-02-23T00:41:49.751 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:41:49.751 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 10 client.38198.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 14 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4124850 osd=6 3 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4124850 osd=6 3 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c4124850 osd=6 3 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 14 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c4124850 osd=6 3 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 6 14 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4124850 osd=6 4 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 10 client.38198.objecter _send_command 14 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.450+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f05c412b1a0 con 0x7f05c4129110 2024-02-23T00:41:49.752 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f05c412b1a0 con 0x7f05c4129110 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c4124850 osd=6 4 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 6 14 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 10 client.38198.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 15 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6812/373787000 conn(0x7f05c412d990 0x7f05c412ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c412b1a0 osd=7 3 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c412b1a0 osd=7 3 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c412b1a0 osd=7 3 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 15 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c412b1a0 osd=7 3 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 7 15 2024-02-23T00:41:49.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c412b1a0 osd=7 4 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 10 client.38198.objecter _send_command 15 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.452+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05c412e3f0 con 0x7f05c412d990 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.453+0000 7f05c9203700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6812/373787000 conn(0x7f05c412d990 0x7f05c412ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.453+0000 7f05ba7fc700 10 client.38198.objecter ms_handle_connect 0x7f05c412d990 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.453+0000 7f05c9203700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6812/373787000 conn(0x7f05c412d990 0x7f05c412ddd0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.454+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f05c004e4e0 con 0x7f05c412d990 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.454+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.454+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 15 = system:0 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.454+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c412b1a0 osd=7 4 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.454+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 7 15 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 10 client.38198.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign -1 16 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c412b1a0 osd=7 3 2024-02-23T00:41:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c412b1a0 osd=7 3 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 20 client.38198.objecter _get_session s=0x7f05c412b1a0 osd=7 3 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 15 client.38198.objecter _session_command_op_remove -1 16 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 20 client.38198.objecter get_session s=0x7f05c412b1a0 osd=7 3 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 15 client.38198.objecter _session_command_op_assign 7 16 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c412b1a0 osd=7 4 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 10 client.38198.objecter _send_command 16 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.468+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f05c4130600 con 0x7f05c412d990 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.470+0000 7f05ba7fc700 1 -- 172.21.15.87:0/3318851458 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f05c4130600 con 0x7f05c412d990 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.470+0000 7f05ba7fc700 10 client.38198.objecter ms_dispatch 0x7f05c4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.470+0000 7f05ba7fc700 10 client.38198.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.470+0000 7f05ba7fc700 20 client.38198.objecter put_session s=0x7f05c412b1a0 osd=7 4 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.470+0000 7f05ba7fc700 15 client.38198.objecter _session_command_op_remove 7 16 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 librados: watch_flush enter 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 librados: watch_flush exit 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 client.38198.objecter close_session for osd.0 2024-02-23T00:41:49.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6801/484081831 conn(0x7f05c40fe5a0 msgr2=0x7f05c40fe980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6801/484081831 conn(0x7f05c40fe5a0 0x7f05c40fe980 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c405ee80 osd=0 1 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 client.38198.objecter close_session for osd.1 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6805/685354171 conn(0x7f05c40598f0 msgr2=0x7f05c4059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6805/685354171 conn(0x7f05c40598f0 0x7f05c4059cd0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4028710 osd=1 1 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 client.38198.objecter close_session for osd.2 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6809/3605394228 conn(0x7f05c4028270 msgr2=0x7f05c411c280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6809/3605394228 conn(0x7f05c4028270 0x7f05c411c280 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4059770 osd=2 1 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 client.38198.objecter close_session for osd.3 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6813/128059790 conn(0x7f05c411f9e0 msgr2=0x7f05c4121dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6813/128059790 conn(0x7f05c411f9e0 0x7f05c4121dd0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4027be0 osd=3 1 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 client.38198.objecter close_session for osd.4 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6800/4175844757 conn(0x7f05c411ccf0 msgr2=0x7f05c4123440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6800/4175844757 conn(0x7f05c411ccf0 0x7f05c4123440 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411cb80 osd=4 1 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05c9a04700 1 -- 172.21.15.87:0/3318851458 reap_dead start 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 client.38198.objecter close_session for osd.5 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6804/2550013795 conn(0x7f05c4123e40 msgr2=0x7f05c41242a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6804/2550013795 conn(0x7f05c4123e40 0x7f05c41242a0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05c9a04700 1 -- 172.21.15.87:0/3318851458 reap_dead start 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c411bf60 osd=5 1 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 client.38198.objecter close_session for osd.6 2024-02-23T00:41:49.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6808/2509408498 conn(0x7f05c4129110 msgr2=0x7f05c4129550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6808/2509408498 conn(0x7f05c4129110 0x7f05c4129550 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c4124850 osd=6 1 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 client.38198.objecter close_session for osd.7 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6812/373787000 conn(0x7f05c412d990 msgr2=0x7f05c412ddd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6812/373787000 conn(0x7f05c412d990 0x7f05c412ddd0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05c9a04700 1 -- 172.21.15.87:0/3318851458 reap_dead start 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 20 client.38198.objecter put_session s=0x7f05c412b1a0 osd=7 1 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 20 client.38198.objecter shutdown clearing up homeless session... 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 10 client.38198.objecter successfully canceled tick 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6800/1231597463 conn(0x7f05b0062c90 msgr2=0x7f05b0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6800/1231597463 conn(0x7f05b0062c90 0x7f05b0065150 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 rx=0x7f05c410bb30 tx=0x7f05b400b040).stop 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c40f8cb0 msgr2=0x7f05c41101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c40f8cb0 0x7f05c41101f0 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 rx=0x7f05c0002760 tx=0x7f05c0009e80).stop 2024-02-23T00:41:49.758 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 shutdown_connections 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.171:6812/373787000 conn(0x7f05c412d990 0x7f05c412ddd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> v2:172.21.15.87:6800/1231597463 conn(0x7f05b0062c90 0x7f05b0065150 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 --2- 172.21.15.87:0/3318851458 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f05c40f8cb0 0x7f05c41101f0 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 >> 172.21.15.87:0/3318851458 conn(0x7f05c40f2820 msgr2=0x7f05c40f4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 shutdown_connections 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 -- 172.21.15.87:0/3318851458 wait complete. 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:49.471+0000 7f05cb467700 1 librados: shutdown 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.759 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.760 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.760 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:41:49.760 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:41:50.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:49.244833+0000 mon.a (mon.0) 3537 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]': finished 2024-02-23T00:41:50.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:49.244906+0000 mon.a (mon.0) 3538 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-02-23T00:41:50.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:49.312981+0000 mon.c (mon.2) 488 : audit [DBG] from='client.? 172.21.15.87:0/3318851458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:50.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:49.331684+0000 osd.3 (osd.3) 218 : cluster [DBG] 4.14 deep-scrub starts 2024-02-23T00:41:50.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:49.337964+0000 osd.3 (osd.3) 219 : cluster [DBG] 4.14 deep-scrub ok 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:49.585404+0000 osd.7 (osd.7) 306 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:49.588285+0000 osd.7 (osd.7) 307 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:50.123613+0000 mon.a (mon.0) 3539 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:50.247031+0000 mon.a (mon.0) 3540 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:49.244833+0000 mon.a (mon.0) 3537 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]': finished 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:49.244906+0000 mon.a (mon.0) 3538 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:49.312981+0000 mon.c (mon.2) 488 : audit [DBG] from='client.? 172.21.15.87:0/3318851458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:49.331684+0000 osd.3 (osd.3) 218 : cluster [DBG] 4.14 deep-scrub starts 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:49.337964+0000 osd.3 (osd.3) 219 : cluster [DBG] 4.14 deep-scrub ok 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:49.585404+0000 osd.7 (osd.7) 306 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:49.588285+0000 osd.7 (osd.7) 307 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:50.123613+0000 mon.a (mon.0) 3539 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:50.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:50.247031+0000 mon.a (mon.0) 3540 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-02-23T00:41:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:49.244833+0000 mon.a (mon.0) 3537 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.5775746836831981}]': finished 2024-02-23T00:41:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:49.244906+0000 mon.a (mon.0) 3538 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-02-23T00:41:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:49.312981+0000 mon.c (mon.2) 488 : audit [DBG] from='client.? 172.21.15.87:0/3318851458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:49.331684+0000 osd.3 (osd.3) 218 : cluster [DBG] 4.14 deep-scrub starts 2024-02-23T00:41:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:49.337964+0000 osd.3 (osd.3) 219 : cluster [DBG] 4.14 deep-scrub ok 2024-02-23T00:41:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:49.585404+0000 osd.7 (osd.7) 306 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:41:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:49.588285+0000 osd.7 (osd.7) 307 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:41:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:50.123613+0000 mon.a (mon.0) 3539 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:50.247031+0000 mon.a (mon.0) 3540 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-02-23T00:41:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:49.736947+0000 osd.0 (osd.0) 7 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:41:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:49.737923+0000 osd.0 (osd.0) 8 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:41:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:49.772606+0000 mgr.y (mgr.24437) 594 : cluster [DBG] pgmap v1050: 194 pgs: 3 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 919 B/s rd, 0 op/s; 11/723 objects misplaced (1.521%) 2024-02-23T00:41:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:50.263290+0000 osd.2 (osd.2) 247 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:41:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:50.264235+0000 osd.2 (osd.2) 248 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:41:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:50.661420+0000 osd.6 (osd.6) 282 : cluster [DBG] 234.0 scrub starts 2024-02-23T00:41:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:50.664965+0000 osd.6 (osd.6) 283 : cluster [DBG] 234.0 scrub ok 2024-02-23T00:41:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:49.736947+0000 osd.0 (osd.0) 7 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:41:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:49.737923+0000 osd.0 (osd.0) 8 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:41:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:49.772606+0000 mgr.y (mgr.24437) 594 : cluster [DBG] pgmap v1050: 194 pgs: 3 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 919 B/s rd, 0 op/s; 11/723 objects misplaced (1.521%) 2024-02-23T00:41:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:50.263290+0000 osd.2 (osd.2) 247 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:41:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:50.264235+0000 osd.2 (osd.2) 248 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:41:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:50.661420+0000 osd.6 (osd.6) 282 : cluster [DBG] 234.0 scrub starts 2024-02-23T00:41:51.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:50.664965+0000 osd.6 (osd.6) 283 : cluster [DBG] 234.0 scrub ok 2024-02-23T00:41:51.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:41:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:41:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:49.736947+0000 osd.0 (osd.0) 7 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:41:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:49.737923+0000 osd.0 (osd.0) 8 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:41:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:49.772606+0000 mgr.y (mgr.24437) 594 : cluster [DBG] pgmap v1050: 194 pgs: 3 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 919 B/s rd, 0 op/s; 11/723 objects misplaced (1.521%) 2024-02-23T00:41:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:50.263290+0000 osd.2 (osd.2) 247 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:41:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:50.264235+0000 osd.2 (osd.2) 248 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:41:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:50.661420+0000 osd.6 (osd.6) 282 : cluster [DBG] 234.0 scrub starts 2024-02-23T00:41:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:50.664965+0000 osd.6 (osd.6) 283 : cluster [DBG] 234.0 scrub ok 2024-02-23T00:41:51.880 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:51.986 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:41:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:51.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37088.0:1 146.6s0 146:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e212)", 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:43.010737+0000", 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 548.98498709299997, 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0162247929999999, 2024-02-23T00:41:51.999 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37088", 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/758004491", 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010737+0000", 2024-02-23T00:41:52.000 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010737+0000", 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010735+0000", 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8709999999999992e-06 2024-02-23T00:41:52.001 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010745+0000", 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3915e-05 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010759+0000", 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4579000000000001e-05 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.002 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.010774+0000", 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0025273800000000001 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.013301+0000", 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3150000000000002e-06 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:43.013305+0000", 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0005927610000001 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.003 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.013898+0000", 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00011305499999999999 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.014011+0000", 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028954399999999998 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:41:52.004 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.014301+0000", 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0018669419999999999 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.016168+0000", 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.01075132 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.026919+0000", 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3027e-05 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.005 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:44.026962+0000", 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0162247929999999 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:1810 63.8 63:15d27fd9:test-rados-api-smithi087-82091-22::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:32:47.023992+0000", 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 544.97173184300004, 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041033304999999999, 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.006 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:52.007 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:52.007 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:41:52.007 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:41:52.007 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1810 2024-02-23T00:41:52.007 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.007 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.007 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.007 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.007 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.023992+0000", 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.023992+0000", 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.023990+0000", 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4919999999999999e-06 2024-02-23T00:41:52.008 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024000+0000", 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1005000000000001e-05 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024011+0000", 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4859999999999997e-06 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.009 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024019+0000", 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4852999999999998e-05 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024074+0000", 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013309699999999999 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.010 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024207+0000", 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00018933000000000001 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.024397+0000", 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0012107419999999999 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.011 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.025607+0000", 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013983776 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.039591+0000", 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.025340995000000002 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.012 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.064932+0000", 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7939000000000003e-05 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.064980+0000", 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5509000000000001e-05 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:32:47.065026+0000", 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.041033304999999999 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.013 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.37277.0:1 173.6s0 173:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e261)", 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:32.043764+0000", 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 499.95195980800003, 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.99129025299999995, 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.37277", 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2105660432", 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:41:52.014 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043764+0000", 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043764+0000", 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.015 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043762+0000", 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8310000000000001e-06 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043770+0000", 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2220000000000008e-06 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043779+0000", 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3200000000000002e-06 2024-02-23T00:41:52.016 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043787+0000", 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0408999999999994e-05 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043877+0000", 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.213e-06 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.017 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:32.043880+0000", 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.98460219900000001 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.028483+0000", 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012271999999999999 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.028605+0000", 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028859799999999999 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.018 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.028894+0000", 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00041391600000000001 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.029308+0000", 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0056940439999999997 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.035002+0000", 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2791999999999999e-05 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.019 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:33.035055+0000", 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.99129025299999995 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26898.0:1 183.6s0 183:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e282)", 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:53.056908+0000", 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 478.938816643, 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.99571678900000005, 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.020 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26898", 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3707730563", 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056908+0000", 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.021 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.022 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.022 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056908+0000", 2024-02-23T00:41:52.022 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:41:52.022 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.022 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.022 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.023 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056906+0000", 2024-02-23T00:41:52.023 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2740000000000004e-06 2024-02-23T00:41:52.023 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.023 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.023 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.023 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056912+0000", 2024-02-23T00:41:52.024 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6000000000000007e-06 2024-02-23T00:41:52.024 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.024 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.024 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.024 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056921+0000", 2024-02-23T00:41:52.024 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1380000000000001e-06 2024-02-23T00:41:52.025 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.025 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.025 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.025 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.056928+0000", 2024-02-23T00:41:52.025 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025497999999999997 2024-02-23T00:41:52.025 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.026 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.026 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.026 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.057183+0000", 2024-02-23T00:41:52.026 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4829999999999999e-06 2024-02-23T00:41:52.026 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.026 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.027 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:52.027 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:53.057187+0000", 2024-02-23T00:41:52.027 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.98844382500000005 2024-02-23T00:41:52.027 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.027 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.027 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.028 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.045630+0000", 2024-02-23T00:41:52.028 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000127668 2024-02-23T00:41:52.028 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.028 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.028 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:52.029 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.045758+0000", 2024-02-23T00:41:52.029 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00040348400000000001 2024-02-23T00:41:52.029 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.029 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.029 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:41:52.029 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.046162+0000", 2024-02-23T00:41:52.030 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000493244 2024-02-23T00:41:52.030 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.030 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.030 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:41:52.030 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.046655+0000", 2024-02-23T00:41:52.030 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.005911449 2024-02-23T00:41:52.030 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.031 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.031 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:41:52.031 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.052566+0000", 2024-02-23T00:41:52.031 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8118999999999997e-05 2024-02-23T00:41:52.031 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.031 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.032 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.032 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:54.052624+0000", 2024-02-23T00:41:52.032 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.99571678900000005 2024-02-23T00:41:52.032 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.032 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.033 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.033 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.033 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.033 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:1877 5.20 5:05bf5b68:::notify.1:head [watch reconnect cookie 94532396646400 gen 4] snapc 0=[] ondisk+write+known_if_redirected e561)", 2024-02-23T00:41:52.033 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:39:40.477301+0000", 2024-02-23T00:41:52.033 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 131.518423407, 2024-02-23T00:41:52.034 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.73050895699999996, 2024-02-23T00:41:52.034 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.034 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:52.034 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:52.034 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:41:52.034 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:41:52.034 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1877 2024-02-23T00:41:52.035 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.035 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.035 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.035 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.035 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477301+0000", 2024-02-23T00:41:52.035 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.036 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.036 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.036 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.036 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477301+0000", 2024-02-23T00:41:52.036 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:52.036 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.037 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.037 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.037 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477300+0000", 2024-02-23T00:41:52.037 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9739999999999999e-06 2024-02-23T00:41:52.037 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.037 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.037 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.038 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477306+0000", 2024-02-23T00:41:52.038 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1486e-05 2024-02-23T00:41:52.038 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.038 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.038 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.038 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477317+0000", 2024-02-23T00:41:52.038 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5060000000000001e-06 2024-02-23T00:41:52.039 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.039 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.039 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.039 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477324+0000", 2024-02-23T00:41:52.039 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2507000000000003e-05 2024-02-23T00:41:52.039 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.039 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.039 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.040 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477386+0000", 2024-02-23T00:41:52.040 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4099999999999997e-06 2024-02-23T00:41:52.040 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.040 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.040 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:52.040 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477396+0000", 2024-02-23T00:41:52.041 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.70815977299999999 2024-02-23T00:41:52.041 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.041 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.041 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.041 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.185555+0000", 2024-02-23T00:41:52.041 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013567700000000001 2024-02-23T00:41:52.042 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.042 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.042 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for missing object", 2024-02-23T00:41:52.042 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.185691+0000", 2024-02-23T00:41:52.042 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003571552 2024-02-23T00:41:52.043 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.043 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.043 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.043 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.189263+0000", 2024-02-23T00:41:52.043 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000101983 2024-02-23T00:41:52.044 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.044 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.044 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for rw locks", 2024-02-23T00:41:52.044 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.189365+0000", 2024-02-23T00:41:52.044 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0061286350000000003 2024-02-23T00:41:52.044 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.045 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.045 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.045 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.195493+0000", 2024-02-23T00:41:52.045 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5148000000000005e-05 2024-02-23T00:41:52.045 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.045 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.046 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:52.046 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.195578+0000", 2024-02-23T00:41:52.046 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012231357999999999 2024-02-23T00:41:52.046 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.046 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.046 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.207810+0000", 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.73050895699999996 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:41 235.0 235:00000000::::head [pgnls start_epoch 603] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e603)", 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:40:22.179696+0000", 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 89.816027954000006, 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0204403200000001, 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.047 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:52.048 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:52.048 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:41:52.048 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:41:52.048 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 41 2024-02-23T00:41:52.048 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.048 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.048 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.049 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.049 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179696+0000", 2024-02-23T00:41:52.049 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.049 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.049 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.049 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.049 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179696+0000", 2024-02-23T00:41:52.049 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179695+0000", 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9789999999999998e-06 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179701+0000", 2024-02-23T00:41:52.050 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.9279999999999999e-06 2024-02-23T00:41:52.051 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.051 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.051 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.051 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179710+0000", 2024-02-23T00:41:52.051 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6839999999999996e-06 2024-02-23T00:41:52.051 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.051 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.051 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.051 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179717+0000", 2024-02-23T00:41:52.052 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0094283470000000001 2024-02-23T00:41:52.052 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.052 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.052 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.052 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.189146+0000", 2024-02-23T00:41:52.052 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8720000000000008e-06 2024-02-23T00:41:52.053 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.053 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.053 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:52.053 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.189155+0000", 2024-02-23T00:41:52.053 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.010297435 2024-02-23T00:41:52.053 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.053 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.054 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.054 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:23.199452+0000", 2024-02-23T00:41:52.054 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7379000000000002e-05 2024-02-23T00:41:52.054 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.054 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.054 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:52.055 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:23.199489+0000", 2024-02-23T00:41:52.055 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00064718399999999998 2024-02-23T00:41:52.055 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.055 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.055 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.055 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:23.200137+0000", 2024-02-23T00:41:52.055 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0204403200000001 2024-02-23T00:41:52.056 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.056 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.056 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.056 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.056 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.057 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2031 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94532396646400 gen 5] snapc 0=[] ondisk+write+known_if_redirected e641)", 2024-02-23T00:41:52.057 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:00.492645+0000", 2024-02-23T00:41:52.057 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 51.503079401000001, 2024-02-23T00:41:52.057 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72160946599999998, 2024-02-23T00:41:52.057 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.057 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:52.058 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:52.058 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:41:52.058 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:41:52.058 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2031 2024-02-23T00:41:52.058 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.058 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.059 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.059 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.059 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492645+0000", 2024-02-23T00:41:52.059 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.059 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.059 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.059 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492645+0000", 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492644+0000", 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9739999999999999e-06 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492650+0000", 2024-02-23T00:41:52.060 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3244000000000001e-05 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492663+0000", 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7290000000000003e-06 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492670+0000", 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3585999999999997e-05 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.061 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492733+0000", 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0555e-05 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492744+0000", 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.720318397 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.213062+0000", 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016163399999999999 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.062 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.213224+0000", 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0010303999999999999 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.214254+0000", 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72160946599999998 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2032 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94532396646400 gen 6] snapc 0=[] ondisk+write+known_if_redirected e641)", 2024-02-23T00:41:52.063 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:00.492681+0000", 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 51.503043394999999, 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72175932899999995, 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2032 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492681+0000", 2024-02-23T00:41:52.064 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492681+0000", 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492680+0000", 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.988e-06 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.065 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492683+0000", 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5169999999999996e-06 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492690+0000", 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.51e-06 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492694+0000", 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4622000000000005e-05 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.066 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492769+0000", 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4109999999999996e-06 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492777+0000", 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72074307900000001 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.213520+0000", 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4667999999999997e-05 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:52.067 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.213595+0000", 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00084507899999999999 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.214440+0000", 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72175932899999995 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:05588ce8:lc::lc.6:head)])", 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:37.648001+0000", 2024-02-23T00:41:52.068 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 14.347723558, 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040277480000000003, 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648001+0000", 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648001+0000", 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.070 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.070 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648000+0000", 2024-02-23T00:41:52.070 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3129999999999999e-06 2024-02-23T00:41:52.070 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.070 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.070 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.071 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648008+0000", 2024-02-23T00:41:52.071 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3129999999999999e-06 2024-02-23T00:41:52.071 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.071 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.071 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.071 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648017+0000", 2024-02-23T00:41:52.072 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.296e-06 2024-02-23T00:41:52.072 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.072 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.072 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.072 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648021+0000", 2024-02-23T00:41:52.072 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038703909999999999 2024-02-23T00:41:52.073 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.073 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.073 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.073 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.651891+0000", 2024-02-23T00:41:52.073 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013709599999999999 2024-02-23T00:41:52.074 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.074 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.074 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.074 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.652028+0000", 2024-02-23T00:41:52.074 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040277480000000003 2024-02-23T00:41:52.074 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.074 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.075 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.075 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.075 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.075 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:02683654:reshard::reshard.0000000010:head)])", 2024-02-23T00:41:52.075 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:37.738432+0000", 2024-02-23T00:41:52.076 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 14.257292243, 2024-02-23T00:41:52.076 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039951580000000004, 2024-02-23T00:41:52.076 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.076 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.076 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.076 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.077 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.077 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738432+0000", 2024-02-23T00:41:52.077 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.077 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.077 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.077 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.078 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738432+0000", 2024-02-23T00:41:52.078 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:52.078 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.078 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.078 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.078 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738431+0000", 2024-02-23T00:41:52.079 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5740000000000001e-06 2024-02-23T00:41:52.079 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.079 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.079 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.079 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738438+0000", 2024-02-23T00:41:52.079 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1375999999999999e-05 2024-02-23T00:41:52.080 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.080 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.080 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.080 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738449+0000", 2024-02-23T00:41:52.080 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4710000000000001e-06 2024-02-23T00:41:52.080 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.080 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738454+0000", 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038411320000000001 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.742295+0000", 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000132378 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.742427+0000", 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039951580000000004 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.081 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:0705c68b:::obj_delete_at_hint.0000000028:head)])", 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:37.919259+0000", 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 14.076464943, 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003982579, 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919259+0000", 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.082 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919259+0000", 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919258+0000", 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7479999999999996e-06 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919265+0000", 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2201999999999999e-05 2024-02-23T00:41:52.083 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919277+0000", 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4779999999999996e-06 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919284+0000", 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038323099999999998 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.084 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.923116+0000", 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000125709 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.923242+0000", 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003982579 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:0216b5b6:::obj_delete_at_hint.0000000070:head)])", 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.009599+0000", 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 13.986125407999999, 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040234959999999997, 2024-02-23T00:41:52.085 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009599+0000", 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009599+0000", 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.086 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009598+0000", 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6440000000000003e-06 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009605+0000", 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2456000000000001e-05 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009617+0000", 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5989999999999998e-06 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.087 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009624+0000", 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038776380000000001 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.013501+0000", 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012103 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.013622+0000", 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040234959999999997 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.088 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:00569cb6:::obj_delete_at_hint.0000000078:head)])", 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.100151+0000", 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 13.895573367000001, 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038904009999999999, 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.100151+0000", 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.089 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.100151+0000", 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.100150+0000", 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.78e-06 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.100155+0000", 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6080000000000002e-06 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.090 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.100165+0000", 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9800000000000003e-06 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.100171+0000", 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037482330000000001 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.103919+0000", 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012234 2024-02-23T00:41:52.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.104041+0000", 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038904009999999999 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:0255b7ba:::obj_delete_at_hint.0000000104:head)])", 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.190491+0000", 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 13.805232907000001, 2024-02-23T00:41:52.092 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039407130000000002, 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190491+0000", 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190491+0000", 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.093 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190491+0000", 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.698e-06 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190495+0000", 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2920000000000001e-06 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190501+0000", 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.908e-06 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.094 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190507+0000", 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037958549999999999 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.194303+0000", 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012923700000000001 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.194432+0000", 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039407130000000002 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.095 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_scan(digest 4.1a MIN-MAX e 679/678)", 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.216534+0000", 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 13.779190706, 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039114390000000001, 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216534+0000", 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.096 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216534+0000", 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216533+0000", 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8890000000000004e-06 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216537+0000", 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6380000000000001e-06 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.097 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216543+0000", 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8800000000000002e-06 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216550+0000", 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038478010000000001 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.220397+0000", 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5129e-05 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.098 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.220413+0000", 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.2404999999999999e-05 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.220445+0000", 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039114390000000001 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.1a 679/678 [PushReplyOp(4:5909e671:gc::gc.27:head)])", 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.302205+0000", 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 13.693519336, 2024-02-23T00:41:52.099 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039522969999999996, 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302205+0000", 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302205+0000", 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.100 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302205+0000", 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8999999999999999e-06 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302208+0000", 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0290000000000002e-06 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.101 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.102 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302215+0000", 2024-02-23T00:41:52.102 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1099999999999999e-06 2024-02-23T00:41:52.102 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.102 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.102 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.102 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302221+0000", 2024-02-23T00:41:52.102 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0036789309999999999 2024-02-23T00:41:52.102 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.102 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.305900+0000", 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025763700000000001 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.306157+0000", 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039522969999999996 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.103 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.19 679/678 [PushReplyOp(4:994d36d1:lc::lc.9:head)])", 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.393809+0000", 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 13.601915413, 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039211339999999997, 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393809+0000", 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393809+0000", 2024-02-23T00:41:52.104 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393808+0000", 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9199999999999997e-06 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393812+0000", 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4679999999999997e-06 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393819+0000", 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3820000000000003e-06 2024-02-23T00:41:52.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393824+0000", 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003639775 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.397464+0000", 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026597399999999998 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.397730+0000", 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039211339999999997 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.106 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.19 679/678 [PushReplyOp(4:9dc32cef:lc::lc.19:head)])", 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.757352+0000", 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 13.238372585, 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038861659999999999, 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.757352+0000", 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.107 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.757352+0000", 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.757351+0000", 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2649999999999998e-06 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.757356+0000", 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2450000000000003e-06 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.108 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.757362+0000", 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.161e-06 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.757368+0000", 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037013359999999999 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.761069+0000", 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016847800000000001 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.761238+0000", 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038861659999999999 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.1a 679/678 [PushReplyOp(4:5f2e3d26:lc::lc.18:head)])", 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.763294+0000", 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 13.232430431999999, 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038946779999999999, 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.110 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763294+0000", 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763294+0000", 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763293+0000", 2024-02-23T00:41:52.111 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1159999999999999e-06 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763297+0000", 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4370000000000004e-06 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763303+0000", 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4269999999999999e-06 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763308+0000", 2024-02-23T00:41:52.112 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037399030000000002 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.767048+0000", 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00014011199999999999 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.767188+0000", 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038946779999999999 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.113 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.19 679/678 [PushReplyOp(4:9fecf831:::obj_delete_at_hint.0000000057:head)])", 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.216473+0000", 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 12.779251010999999, 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.082714981000000007, 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216473+0000", 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:41:52.114 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216473+0000", 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216473+0000", 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0299999999999999e-06 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216479+0000", 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0036e-05 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.115 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216489+0000", 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8669999999999996e-06 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216496+0000", 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.082519156999999996 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.299015+0000", 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000173456 2024-02-23T00:41:52.116 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.299188+0000", 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.082714981000000007 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:41:52.117 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:41:52.117 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:41:52.150 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:52.171 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:41:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:52.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:41:52.275 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:52.294 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:52] "GET /metrics HTTP/1.1" 200 190040 "" "Prometheus/2.33.4" 2024-02-23T00:41:52.308 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:52.362 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:52.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:52.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:50.711423+0000 osd.0 (osd.0) 9 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:41:52.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:50.712406+0000 osd.0 (osd.0) 10 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:41:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:51.192311+0000 mgr.y (mgr.24437) 595 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:51.277956+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:51.304374+0000 osd.1 (osd.1) 318 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:41:52.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:51.307438+0000 osd.1 (osd.1) 319 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:41:52.466 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:52.493 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:52.530 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:41:52.554 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:50.711423+0000 osd.0 (osd.0) 9 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:41:52.554 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:50.712406+0000 osd.0 (osd.0) 10 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:41:52.554 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:51.192311+0000 mgr.y (mgr.24437) 595 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:52.554 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:51.277956+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:52.554 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:51.304374+0000 osd.1 (osd.1) 318 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:41:52.555 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:51.307438+0000 osd.1 (osd.1) 319 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:41:52.555 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:50.711423+0000 osd.0 (osd.0) 9 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:41:52.555 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:50.712406+0000 osd.0 (osd.0) 10 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:41:52.555 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:51.192311+0000 mgr.y (mgr.24437) 595 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:41:52.555 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:51.277956+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:52.555 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:51.304374+0000 osd.1 (osd.1) 318 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:41:52.555 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:51.307438+0000 osd.1 (osd.1) 319 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:41:52.741 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:52.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:52.846 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:52.877 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:52.898 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:52.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:53.002 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:53.030 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:53.308 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:51.773594+0000 mgr.y (mgr.24437) 596 : cluster [DBG] pgmap v1052: 194 pgs: 3 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 32 KiB/s rd, 853 B/s wr, 62 op/s; 11/735 objects misplaced (1.497%); 146 B/s, 7 objects/s recovering 2024-02-23T00:41:53.308 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:52.302287+0000 mon.a (mon.0) 3542 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:53.308 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:52.302355+0000 mon.a (mon.0) 3543 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-02-23T00:41:53.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:51.773594+0000 mgr.y (mgr.24437) 596 : cluster [DBG] pgmap v1052: 194 pgs: 3 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 32 KiB/s rd, 853 B/s wr, 62 op/s; 11/735 objects misplaced (1.497%); 146 B/s, 7 objects/s recovering 2024-02-23T00:41:53.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:52.302287+0000 mon.a (mon.0) 3542 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:53.309 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:52.302355+0000 mon.a (mon.0) 3543 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-02-23T00:41:53.381 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-23T00:41:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:52.303952+0000 mon.a (mon.0) 3544 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118"}]: dispatch 2024-02-23T00:41:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:52.312689+0000 osd.2 (osd.2) 249 : cluster [DBG] 2.e scrub starts 2024-02-23T00:41:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:52.313928+0000 osd.2 (osd.2) 250 : cluster [DBG] 2.e scrub ok 2024-02-23T00:41:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:52.303952+0000 mon.a (mon.0) 3544 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118"}]: dispatch 2024-02-23T00:41:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:52.312689+0000 osd.2 (osd.2) 249 : cluster [DBG] 2.e scrub starts 2024-02-23T00:41:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:52.313928+0000 osd.2 (osd.2) 250 : cluster [DBG] 2.e scrub ok 2024-02-23T00:41:53.650 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:41:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:51.773594+0000 mgr.y (mgr.24437) 596 : cluster [DBG] pgmap v1052: 194 pgs: 3 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 32 KiB/s rd, 853 B/s wr, 62 op/s; 11/735 objects misplaced (1.497%); 146 B/s, 7 objects/s recovering 2024-02-23T00:41:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:52.302287+0000 mon.a (mon.0) 3542 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:41:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:52.302355+0000 mon.a (mon.0) 3543 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-02-23T00:41:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:52.303952+0000 mon.a (mon.0) 3544 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118"}]: dispatch 2024-02-23T00:41:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:52.312689+0000 osd.2 (osd.2) 249 : cluster [DBG] 2.e scrub starts 2024-02-23T00:41:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:52.313928+0000 osd.2 (osd.2) 250 : cluster [DBG] 2.e scrub ok 2024-02-23T00:41:53.878 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:53.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:53.970 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:41:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:41:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:41:53.982 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:54.012 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:54.031 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:54.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:54.135 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:54.161 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:54.180 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:54.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:54.284 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:54.313 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:54.391 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:54.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:54.495 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:54.521 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.299604+0000 mon.a (mon.0) 3545 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:53.304448+0000 mon.a (mon.0) 3546 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118"}]': finished 2024-02-23T00:41:54.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.304521+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-02-23T00:41:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.368394+0000 osd.1 (osd.1) 320 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:41:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.371848+0000 osd.1 (osd.1) 321 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:41:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.494896+0000 osd.7 (osd.7) 308 : cluster [DBG] 22.d scrub starts 2024-02-23T00:41:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.497788+0000 osd.7 (osd.7) 309 : cluster [DBG] 22.d scrub ok 2024-02-23T00:41:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.921806+0000 osd.4 (osd.4) 313 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:41:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.922588+0000 osd.4 (osd.4) 314 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:41:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.983801+0000 osd.5 (osd.5) 263 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:41:54.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.984675+0000 osd.5 (osd.5) 264 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:41:54.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.299604+0000 mon.a (mon.0) 3545 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:53.304448+0000 mon.a (mon.0) 3546 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118"}]': finished 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.304521+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.368394+0000 osd.1 (osd.1) 320 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.371848+0000 osd.1 (osd.1) 321 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.494896+0000 osd.7 (osd.7) 308 : cluster [DBG] 22.d scrub starts 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.497788+0000 osd.7 (osd.7) 309 : cluster [DBG] 22.d scrub ok 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.921806+0000 osd.4 (osd.4) 313 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.922588+0000 osd.4 (osd.4) 314 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.983801+0000 osd.5 (osd.5) 263 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:41:54.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.984675+0000 osd.5 (osd.5) 264 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:41:54.608 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.299604+0000 mon.a (mon.0) 3545 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:41:54.608 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:53.304448+0000 mon.a (mon.0) 3546 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-118"}]': finished 2024-02-23T00:41:54.608 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.304521+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-02-23T00:41:54.608 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.368394+0000 osd.1 (osd.1) 320 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:41:54.608 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.371848+0000 osd.1 (osd.1) 321 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:41:54.609 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.494896+0000 osd.7 (osd.7) 308 : cluster [DBG] 22.d scrub starts 2024-02-23T00:41:54.609 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.497788+0000 osd.7 (osd.7) 309 : cluster [DBG] 22.d scrub ok 2024-02-23T00:41:54.609 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.921806+0000 osd.4 (osd.4) 313 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:41:54.609 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.922588+0000 osd.4 (osd.4) 314 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:41:54.609 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.983801+0000 osd.5 (osd.5) 263 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:41:54.609 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.984675+0000 osd.5 (osd.5) 264 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:41:54.770 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:54.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:54.875 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:54.904 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:55.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.334+0000 7f97f8891700 1 -- 172.21.15.87:0/479997872 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 msgr2=0x7f97f40fdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:55.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.334+0000 7f97f8891700 1 --2- 172.21.15.87:0/479997872 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 0x7f97f40fdfb0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 rx=0x7f97e4004870 tx=0x7f97e4009cb0).stop 2024-02-23T00:41:55.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.335+0000 7f97f8891700 1 -- 172.21.15.87:0/479997872 shutdown_connections 2024-02-23T00:41:55.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.335+0000 7f97f8891700 1 --2- 172.21.15.87:0/479997872 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97f40f6a90 0x7f97f4104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:55.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.335+0000 7f97f8891700 1 --2- 172.21.15.87:0/479997872 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 0x7f97f40fdfb0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:55.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.335+0000 7f97f8891700 1 --2- 172.21.15.87:0/479997872 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97f40fe530 0x7f97f40f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.335+0000 7f97f8891700 1 -- 172.21.15.87:0/479997872 >> 172.21.15.87:0/479997872 conn(0x7f97f406c370 msgr2=0x7f97f406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.335+0000 7f97f8891700 1 -- 172.21.15.87:0/479997872 shutdown_connections 2024-02-23T00:41:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.335+0000 7f97f8891700 1 -- 172.21.15.87:0/479997872 wait complete. 2024-02-23T00:41:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.335+0000 7f97f8891700 1 librados: starting msgr at 2024-02-23T00:41:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.335+0000 7f97f8891700 1 librados: starting objecter 2024-02-23T00:41:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 Processor -- start 2024-02-23T00:41:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 -- start start 2024-02-23T00:41:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 librados: setting wanted keys 2024-02-23T00:41:55.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 librados: calling monclient init 2024-02-23T00:41:55.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97f40f6a90 0x7f97f410e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 0x7f97f410e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97f40fe530 0x7f97f4108300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f41087f0 con 0x7f97f40f6a90 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f4108960 con 0x7f97f40fe530 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f8891700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f4108b20 con 0x7f97f40fdbd0 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f1d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 0x7f97f410e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f1d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 0x7f97f410e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:58834/0 (socket says 172.21.15.87:58834) 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f1d9b700 1 -- 172.21.15.87:0/2607645750 learned_addr learned my addr 172.21.15.87:0/2607645750 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f2d9d700 1 --2- 172.21.15.87:0/2607645750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97f40fe530 0x7f97f4108300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f1d9b700 1 -- 172.21.15.87:0/2607645750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97f40fe530 msgr2=0x7f97f4108300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f1d9b700 1 --2- 172.21.15.87:0/2607645750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97f40fe530 0x7f97f4108300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f1d9b700 1 -- 172.21.15.87:0/2607645750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97f40f6a90 msgr2=0x7f97f410e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f1d9b700 1 --2- 172.21.15.87:0/2607645750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97f40f6a90 0x7f97f410e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:41:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f1d9b700 1 -- 172.21.15.87:0/2607645750 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e8005040 con 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.336+0000 7f97f2d9d700 1 --2- 172.21.15.87:0/2607645750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97f40fe530 0x7f97f4108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97f1d9b700 1 --2- 172.21.15.87:0/2607645750 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 0x7f97f410e650 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7f97e80048f0 tx=0x7f97e8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97e37fe700 10 client.?.objecter ms_handle_connect 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97e37fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97e37fe700 1 -- 172.21.15.87:0/2607645750 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8007b00 con 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e4005040 con 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97e37fe700 1 -- 172.21.15.87:0/2607645750 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97e800f040 con 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97e37fe700 1 -- 172.21.15.87:0/2607645750 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8014aa0 con 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97f8891700 10 client.38204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f4109160 con 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.337+0000 7f97f8891700 1 librados: init done 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.338+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f405f950 con 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.338+0000 7f97e37fe700 1 -- 172.21.15.87:0/2607645750 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f97e80133c0 con 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.339+0000 7f97e37fe700 1 --2- 172.21.15.87:0/2607645750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f97dc062c90 0x7f97dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.339+0000 7f97e37fe700 1 -- 172.21.15.87:0/2607645750 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(693..693 src has 173..693) v4 ==== 8535+0+0 (secure 0 0 0) 0x7f97e807d0f0 con 0x7f97f40fdbd0 2024-02-23T00:41:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.339+0000 7f97e37fe700 10 client.38204.objecter ms_dispatch 0x7f97f4001550 osd_map(693..693 src has 173..693) v4 2024-02-23T00:41:55.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.339+0000 7f97e37fe700 3 client.38204.objecter handle_osd_map got epochs [693,693] > 0 2024-02-23T00:41:55.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.339+0000 7f97e37fe700 3 client.38204.objecter handle_osd_map decoding full epoch 693 2024-02-23T00:41:55.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.339+0000 7f97e37fe700 20 client.38204.objecter dump_active .. 0 homeless 2024-02-23T00:41:55.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.339+0000 7f97f259c700 1 --2- 172.21.15.87:0/2607645750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f97dc062c90 0x7f97dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:55.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.339+0000 7f97e37fe700 10 client.38204.objecter ms_handle_connect 0x7f97dc062c90 2024-02-23T00:41:55.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.339+0000 7f97f259c700 1 --2- 172.21.15.87:0/2607645750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f97dc062c90 0x7f97dc065150 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 rx=0x7f97e4009da0 tx=0x7f97e400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:55.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.343+0000 7f97e37fe700 1 -- 172.21.15.87:0/2607645750 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f97e8049f60 con 0x7f97f40fdbd0 2024-02-23T00:41:55.353 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:55.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:55.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:55.371 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:55.458 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:41:55.482 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:53.773975+0000 mgr.y (mgr.24437) 597 : cluster [DBG] pgmap v1055: 194 pgs: 3 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 1.1 KiB/s wr, 82 op/s; 11/735 objects misplaced (1.497%); 194 B/s, 10 objects/s recovering 2024-02-23T00:41:55.482 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:54.305892+0000 mon.a (mon.0) 3548 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:54.315702+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:54.657089+0000 mon.a (mon.0) 3550 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:54.919732+0000 osd.4 (osd.4) 315 : cluster [DBG] 2.f scrub starts 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:54.923209+0000 osd.4 (osd.4) 316 : cluster [DBG] 2.f scrub ok 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.023918+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.5 deep-scrub starts 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.024700+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.5 deep-scrub ok 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:53.773975+0000 mgr.y (mgr.24437) 597 : cluster [DBG] pgmap v1055: 194 pgs: 3 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 1.1 KiB/s wr, 82 op/s; 11/735 objects misplaced (1.497%); 194 B/s, 10 objects/s recovering 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:54.305892+0000 mon.a (mon.0) 3548 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:54.315702+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-02-23T00:41:55.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:54.657089+0000 mon.a (mon.0) 3550 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:55.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:54.919732+0000 osd.4 (osd.4) 315 : cluster [DBG] 2.f scrub starts 2024-02-23T00:41:55.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:54.923209+0000 osd.4 (osd.4) 316 : cluster [DBG] 2.f scrub ok 2024-02-23T00:41:55.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.023918+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.5 deep-scrub starts 2024-02-23T00:41:55.484 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.024700+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.5 deep-scrub ok 2024-02-23T00:41:55.484 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:55.528 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:41:55.528 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:41:55.529 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:41:55.529 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:41:55.537 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:55.536+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f97f4107140 con 0x7f97f40fdbd0 2024-02-23T00:41:55.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:53.773975+0000 mgr.y (mgr.24437) 597 : cluster [DBG] pgmap v1055: 194 pgs: 3 active+remapped+backfill_toofull, 191 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 1.1 KiB/s wr, 82 op/s; 11/735 objects misplaced (1.497%); 194 B/s, 10 objects/s recovering 2024-02-23T00:41:55.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:54.305892+0000 mon.a (mon.0) 3548 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:41:55.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:54.315702+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-02-23T00:41:55.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:54.657089+0000 mon.a (mon.0) 3550 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:41:55.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:54.919732+0000 osd.4 (osd.4) 315 : cluster [DBG] 2.f scrub starts 2024-02-23T00:41:55.554 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:54.923209+0000 osd.4 (osd.4) 316 : cluster [DBG] 2.f scrub ok 2024-02-23T00:41:55.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.023918+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.5 deep-scrub starts 2024-02-23T00:41:55.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.024700+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.5 deep-scrub ok 2024-02-23T00:41:55.632 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:41:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:55.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:41:55.736 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:41:55.763 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:55.886 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:41:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:55.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:41:55.990 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:41:56.021 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:56.119 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:41:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:56.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:41:56.132 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-23T00:41:56.224 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:56.252 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:56.271 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:56.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:56.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.335+0000 7f97e37fe700 1 -- 172.21.15.87:0/2607645750 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v694) v1 ==== 87+0+0 (secure 0 0 0) 0x7f97e80503e0 con 0x7f97f40fdbd0 2024-02-23T00:41:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.341+0000 7f97f8891700 10 librados: watch_flush enter 2024-02-23T00:41:56.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.341+0000 7f97f8891700 10 librados: watch_flush exit 2024-02-23T00:41:56.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.341+0000 7f97f8891700 20 client.38204.objecter shutdown clearing up homeless session... 2024-02-23T00:41:56.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.341+0000 7f97f8891700 10 client.38204.objecter successfully canceled tick 2024-02-23T00:41:56.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.343+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f97dc062c90 msgr2=0x7f97dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:56.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.343+0000 7f97f8891700 1 --2- 172.21.15.87:0/2607645750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f97dc062c90 0x7f97dc065150 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 rx=0x7f97e4009da0 tx=0x7f97e400b040).stop 2024-02-23T00:41:56.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.344+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 msgr2=0x7f97f410e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:56.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.344+0000 7f97f8891700 1 --2- 172.21.15.87:0/2607645750 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 0x7f97f410e650 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 rx=0x7f97e80048f0 tx=0x7f97e8009cb0).stop 2024-02-23T00:41:56.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.344+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 shutdown_connections 2024-02-23T00:41:56.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.344+0000 7f97f8891700 1 --2- 172.21.15.87:0/2607645750 >> v2:172.21.15.87:6800/1231597463 conn(0x7f97dc062c90 0x7f97dc065150 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:56.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.344+0000 7f97f8891700 1 --2- 172.21.15.87:0/2607645750 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f97f40fe530 0x7f97f4108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:56.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.344+0000 7f97f8891700 1 --2- 172.21.15.87:0/2607645750 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f97f40f6a90 0x7f97f410e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:41:56.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.344+0000 7f97f8891700 1 --2- 172.21.15.87:0/2607645750 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f97f40fdbd0 0x7f97f410e650 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:56.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.344+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 >> 172.21.15.87:0/2607645750 conn(0x7f97f406c370 msgr2=0x7f97f40fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:56.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.344+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 shutdown_connections 2024-02-23T00:41:56.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.345+0000 7f97f8891700 1 -- 172.21.15.87:0/2607645750 wait complete. 2024-02-23T00:41:56.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:56.345+0000 7f97f8891700 1 librados: shutdown 2024-02-23T00:41:56.350 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:41:56.375 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:41:56.402 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:56.445 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:41:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:54.687496+0000 osd.0 (osd.0) 11 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:41:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:54.690404+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:41:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.273859+0000 osd.2 (osd.2) 251 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:41:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.275095+0000 osd.2 (osd.2) 252 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:41:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.318797+0000 mon.a (mon.0) 3551 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-02-23T00:41:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:55.323599+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.354506+0000 osd.3 (osd.3) 220 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.355546+0000 osd.3 (osd.3) 221 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.510729+0000 osd.7 (osd.7) 310 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.514024+0000 osd.7 (osd.7) 311 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:55.536841+0000 mon.c (mon.2) 489 : audit [INF] from='client.? 172.21.15.87:0/2607645750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:55.537183+0000 mon.a (mon.0) 3553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:54.687496+0000 osd.0 (osd.0) 11 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:54.690404+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.273859+0000 osd.2 (osd.2) 251 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:41:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.275095+0000 osd.2 (osd.2) 252 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:41:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.318797+0000 mon.a (mon.0) 3551 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-02-23T00:41:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:55.323599+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.354506+0000 osd.3 (osd.3) 220 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:41:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.355546+0000 osd.3 (osd.3) 221 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:41:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.510729+0000 osd.7 (osd.7) 310 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:41:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.514024+0000 osd.7 (osd.7) 311 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:41:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:55.536841+0000 mon.c (mon.2) 489 : audit [INF] from='client.? 172.21.15.87:0/2607645750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:55.537183+0000 mon.a (mon.0) 3553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:54.687496+0000 osd.0 (osd.0) 11 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:54.690404+0000 osd.0 (osd.0) 12 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.273859+0000 osd.2 (osd.2) 251 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.275095+0000 osd.2 (osd.2) 252 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.318797+0000 mon.a (mon.0) 3551 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:55.323599+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.354506+0000 osd.3 (osd.3) 220 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.355546+0000 osd.3 (osd.3) 221 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.510729+0000 osd.7 (osd.7) 310 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:41:56.685 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.514024+0000 osd.7 (osd.7) 311 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:41:56.686 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:55.536841+0000 mon.c (mon.2) 489 : audit [INF] from='client.? 172.21.15.87:0/2607645750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:56.686 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:55.537183+0000 mon.a (mon.0) 3553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:41:57.251 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:41:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:57.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:41:57.355 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:41:57.383 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:55.774783+0000 mgr.y (mgr.24437) 598 : cluster [DBG] pgmap v1058: 194 pgs: 29 creating+peering, 3 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:41:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:56.316036+0000 mon.a (mon.0) 3554 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:41:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:56.316631+0000 mon.a (mon.0) 3555 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:56.333615+0000 mon.a (mon.0) 3556 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:56.333756+0000 mon.a (mon.0) 3557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:56.333810+0000 mon.a (mon.0) 3558 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:56.344542+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:57.212027+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:57.212102+0000 mon.a (mon.0) 3561 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:57.213448+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-120"}]: dispatch 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:55.774783+0000 mgr.y (mgr.24437) 598 : cluster [DBG] pgmap v1058: 194 pgs: 29 creating+peering, 3 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:56.316036+0000 mon.a (mon.0) 3554 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:56.316631+0000 mon.a (mon.0) 3555 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:56.333615+0000 mon.a (mon.0) 3556 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:56.333756+0000 mon.a (mon.0) 3557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:41:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:56.333810+0000 mon.a (mon.0) 3558 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-02-23T00:41:57.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:56.344542+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:57.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:57.212027+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:57.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:57.212102+0000 mon.a (mon.0) 3561 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-02-23T00:41:57.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:57.213448+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-120"}]: dispatch 2024-02-23T00:41:57.619 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:55.774783+0000 mgr.y (mgr.24437) 598 : cluster [DBG] pgmap v1058: 194 pgs: 29 creating+peering, 3 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:41:57.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:56.316036+0000 mon.a (mon.0) 3554 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:41:57.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:56.316631+0000 mon.a (mon.0) 3555 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:41:57.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:56.333615+0000 mon.a (mon.0) 3556 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:41:57.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:56.333756+0000 mon.a (mon.0) 3557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:41:57.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:56.333810+0000 mon.a (mon.0) 3558 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-02-23T00:41:57.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:56.344542+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:41:57.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:57.212027+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:41:57.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:57.212102+0000 mon.a (mon.0) 3561 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-02-23T00:41:57.620 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:57.213448+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-120"}]: dispatch 2024-02-23T00:41:57.631 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:41:57.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:57.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:41:57.735 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:57.763 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:57.912 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:41:57.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:57.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:41:58.016 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:41:58.044 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.143+0000 7f33285ec700 1 -- 172.21.15.87:0/189981212 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33200fdbb0 msgr2=0x7f33200fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.143+0000 7f33285ec700 1 --2- 172.21.15.87:0/189981212 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33200fdbb0 0x7f33200fdf90 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 rx=0x7f331c004870 tx=0x7f331c009cb0).stop 2024-02-23T00:41:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.143+0000 7f33285ec700 1 -- 172.21.15.87:0/189981212 shutdown_connections 2024-02-23T00:41:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.143+0000 7f33285ec700 1 --2- 172.21.15.87:0/189981212 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33200f8420 0x7f3320104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.143+0000 7f33285ec700 1 --2- 172.21.15.87:0/189981212 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33200fdbb0 0x7f33200fdf90 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.143+0000 7f33285ec700 1 --2- 172.21.15.87:0/189981212 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33200fe510 0x7f33200f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.143+0000 7f33285ec700 1 -- 172.21.15.87:0/189981212 >> 172.21.15.87:0/189981212 conn(0x7f33200f3800 msgr2=0x7f33200f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:58.147 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.143+0000 7f33285ec700 1 -- 172.21.15.87:0/189981212 shutdown_connections 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 -- 172.21.15.87:0/189981212 wait complete. 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 librados: starting msgr at 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 librados: starting objecter 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 Processor -- start 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 -- start start 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 librados: setting wanted keys 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 librados: calling monclient init 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33200f8420 0x7f332006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33200fdbb0 0x7f332006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33200fe510 0x7f332006bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3320116580 con 0x7f33200fdbb0 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33201166f0 con 0x7f33200f8420 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f33285ec700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33201168b0 con 0x7f33200fe510 2024-02-23T00:41:58.148 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f3325b87700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33200fdbb0 0x7f332006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f3326b89700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33200fe510 0x7f332006bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f3325b87700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33200fdbb0 0x7f332006b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47364/0 (socket says 172.21.15.87:47364) 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f3325b87700 1 -- 172.21.15.87:0/2847108917 learned_addr learned my addr 172.21.15.87:0/2847108917 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.144+0000 7f3326388700 1 --2- 172.21.15.87:0/2847108917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33200f8420 0x7f332006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f3326b89700 1 -- 172.21.15.87:0/2847108917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33200f8420 msgr2=0x7f332006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f3326b89700 1 --2- 172.21.15.87:0/2847108917 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f33200f8420 0x7f332006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f3326b89700 1 -- 172.21.15.87:0/2847108917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33200fdbb0 msgr2=0x7f332006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f3326b89700 1 --2- 172.21.15.87:0/2847108917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33200fdbb0 0x7f332006b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f3326b89700 1 -- 172.21.15.87:0/2847108917 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180097c0 con 0x7f33200fe510 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f3325b87700 1 --2- 172.21.15.87:0/2847108917 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f33200fdbb0 0x7f332006b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f3326b89700 1 --2- 172.21.15.87:0/2847108917 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33200fe510 0x7f332006bab0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 rx=0x7f3318001cb0 tx=0x7f3318004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f33177fe700 10 client.?.objecter ms_handle_connect 0x7f33200fe510 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f33177fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:41:58.149 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33180079b0 con 0x7f33200fe510 2024-02-23T00:41:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331c005040 con 0x7f33200fe510 2024-02-23T00:41:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33180090d0 con 0x7f33200fe510 2024-02-23T00:41:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331800fc20 con 0x7f33200fe510 2024-02-23T00:41:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f33285ec700 10 client.38210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:41:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3320116ea0 con 0x7f33200fe510 2024-02-23T00:41:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.145+0000 7f33285ec700 1 librados: init done 2024-02-23T00:41:58.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.146+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f33201070a0 con 0x7f33200fe510 2024-02-23T00:41:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.146+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f331800fd80 con 0x7f33200fe510 2024-02-23T00:41:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33177fe700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6800/1231597463 conn(0x7f330c062c90 0x7f330c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(695..695 src has 173..695) v4 ==== 8543+0+0 (secure 0 0 0) 0x7f331807dc40 con 0x7f33200fe510 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 osd_map(695..695 src has 173..695) v4 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33177fe700 3 client.38210.objecter handle_osd_map got epochs [695,695] > 0 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33177fe700 3 client.38210.objecter handle_osd_map decoding full epoch 695 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f3326388700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6800/1231597463 conn(0x7f330c062c90 0x7f330c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33177fe700 20 client.38210.objecter dump_active .. 0 homeless 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v695) v1 ==== 54+0+16 (secure 0 0 0) 0x7f3318084020 con 0x7f33200fe510 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33177fe700 10 client.38210.objecter ms_handle_connect 0x7f330c062c90 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f3326388700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6800/1231597463 conn(0x7f330c062c90 0x7f330c065150 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7f331c0043d0 tx=0x7f331c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 10 client.38210.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 1 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6801/484081831 conn(0x7f3320028540 0x7f33200fe970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332006c5b0 osd=0 3 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332006c5b0 osd=0 3 2024-02-23T00:41:58.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332006c5b0 osd=0 3 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 1 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f332006c5b0 osd=0 3 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 0 1 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332006c5b0 osd=0 4 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 10 client.38210.objecter _send_command 1 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332005a040 con 0x7f3320028540 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f3325b87700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6801/484081831 conn(0x7f3320028540 0x7f33200fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f33177fe700 10 client.38210.objecter ms_handle_connect 0x7f3320028540 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.147+0000 7f3325b87700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6801/484081831 conn(0x7f3320028540 0x7f33200fe970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.148+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f332005a040 con 0x7f3320028540 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.148+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.148+0000 7f33177fe700 10 client.38210.objecter _finish_command 1 = system:0 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.148+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f332006c5b0 osd=0 4 2024-02-23T00:41:58.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.148+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 0 1 2024-02-23T00:41:58.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 10 client.38210.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:58.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 2 2024-02-23T00:41:58.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332006c5b0 osd=0 3 2024-02-23T00:41:58.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332006c5b0 osd=0 3 2024-02-23T00:41:58.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332006c5b0 osd=0 3 2024-02-23T00:41:58.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 2 2024-02-23T00:41:58.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f332006c5b0 osd=0 3 2024-02-23T00:41:58.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 0 2 2024-02-23T00:41:58.164 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332006c5b0 osd=0 4 2024-02-23T00:41:58.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 10 client.38210.objecter _send_command 2 2024-02-23T00:41:58.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.163+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33200fee60 con 0x7f3320028540 2024-02-23T00:41:58.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f33200fee60 con 0x7f3320028540 2024-02-23T00:41:58.165 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:58.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33177fe700 10 client.38210.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f332006c5b0 osd=0 4 2024-02-23T00:41:58.166 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 0 2 2024-02-23T00:41:58.166 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:41:58.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 10 client.38210.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:58.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 3 2024-02-23T00:41:58.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6805/685354171 conn(0x7f3320027b90 0x7f3320027f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320100150 osd=1 3 2024-02-23T00:41:58.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320100150 osd=1 3 2024-02-23T00:41:58.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320100150 osd=1 3 2024-02-23T00:41:58.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 3 2024-02-23T00:41:58.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f3320100150 osd=1 3 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 1 3 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320100150 osd=1 4 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 10 client.38210.objecter _send_command 3 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.164+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332011d870 con 0x7f3320027b90 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.165+0000 7f3326388700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6805/685354171 conn(0x7f3320027b90 0x7f3320027f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.165+0000 7f33177fe700 10 client.38210.objecter ms_handle_connect 0x7f3320027b90 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.165+0000 7f3326388700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6805/685354171 conn(0x7f3320027b90 0x7f3320027f90 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.165+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f332011d870 con 0x7f3320027b90 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.165+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.165+0000 7f33177fe700 10 client.38210.objecter _finish_command 3 = system:0 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.165+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f3320100150 osd=1 4 2024-02-23T00:41:58.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.165+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 1 3 2024-02-23T00:41:58.169 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:41:58.170 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:41:58.170 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:41:58.170 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:41:58.170 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:41:58.170 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:41:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:58.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 10 client.38210.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 4 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320100150 osd=1 3 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320100150 osd=1 3 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320100150 osd=1 3 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 4 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f3320100150 osd=1 3 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 1 4 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320100150 osd=1 4 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 10 client.38210.objecter _send_command 4 2024-02-23T00:41:58.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.181+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3320027600 con 0x7f3320027b90 2024-02-23T00:41:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f3320027600 con 0x7f3320027b90 2024-02-23T00:41:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33177fe700 10 client.38210.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f3320100150 osd=1 4 2024-02-23T00:41:58.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 1 4 2024-02-23T00:41:58.184 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:41:58.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 10 client.38210.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:58.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 5 2024-02-23T00:41:58.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6809/3605394228 conn(0x7f33200ffcb0 0x7f332011c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320027a10 osd=2 3 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320027a10 osd=2 3 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320027a10 osd=2 3 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 5 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f3320027a10 osd=2 3 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 2 5 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320027a10 osd=2 4 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 10 client.38210.objecter _send_command 5 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332011c780 con 0x7f33200ffcb0 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f3325b87700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6809/3605394228 conn(0x7f33200ffcb0 0x7f332011c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f33177fe700 10 client.38210.objecter ms_handle_connect 0x7f33200ffcb0 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.183+0000 7f3325b87700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6809/3605394228 conn(0x7f33200ffcb0 0x7f332011c290 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.184+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f332011c780 con 0x7f33200ffcb0 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.184+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.184+0000 7f33177fe700 10 client.38210.objecter _finish_command 5 = system:0 2024-02-23T00:41:58.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.184+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f3320027a10 osd=2 4 2024-02-23T00:41:58.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.184+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 2 5 2024-02-23T00:41:58.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 10 client.38210.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:58.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 6 2024-02-23T00:41:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320027a10 osd=2 3 2024-02-23T00:41:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320027a10 osd=2 3 2024-02-23T00:41:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320027a10 osd=2 3 2024-02-23T00:41:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 6 2024-02-23T00:41:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f3320027a10 osd=2 3 2024-02-23T00:41:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 2 6 2024-02-23T00:41:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320027a10 osd=2 4 2024-02-23T00:41:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 10 client.38210.objecter _send_command 6 2024-02-23T00:41:58.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.198+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f33200ff210 con 0x7f33200ffcb0 2024-02-23T00:41:58.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f33200ff210 con 0x7f33200ffcb0 2024-02-23T00:41:58.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:58.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33177fe700 10 client.38210.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f3320027a10 osd=2 4 2024-02-23T00:41:58.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 2 6 2024-02-23T00:41:58.201 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:41:58.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 10 client.38210.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:58.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 7 2024-02-23T00:41:58.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6813/128059790 conn(0x7f332011f9f0 0x7f3320121de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f33200ff620 osd=3 3 2024-02-23T00:41:58.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f33200ff620 osd=3 3 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f33200ff620 osd=3 3 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 7 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f33200ff620 osd=3 3 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 3 7 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f33200ff620 osd=3 4 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 10 client.38210.objecter _send_command 7 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3320124a30 con 0x7f332011f9f0 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f3326b89700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6813/128059790 conn(0x7f332011f9f0 0x7f3320121de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f33177fe700 10 client.38210.objecter ms_handle_connect 0x7f332011f9f0 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.200+0000 7f3326b89700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6813/128059790 conn(0x7f332011f9f0 0x7f3320121de0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.201+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3320124a30 con 0x7f332011f9f0 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.201+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.201+0000 7f33177fe700 10 client.38210.objecter _finish_command 7 = system:0 2024-02-23T00:41:58.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.201+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f33200ff620 osd=3 4 2024-02-23T00:41:58.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.201+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 3 7 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 10 client.38210.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 8 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f33200ff620 osd=3 3 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f33200ff620 osd=3 3 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f33200ff620 osd=3 3 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 8 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f33200ff620 osd=3 3 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 3 8 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f33200ff620 osd=3 4 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 10 client.38210.objecter _send_command 8 2024-02-23T00:41:58.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.217+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f332011cb60 con 0x7f332011f9f0 2024-02-23T00:41:58.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f332011cb60 con 0x7f332011f9f0 2024-02-23T00:41:58.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:58.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33177fe700 10 client.38210.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f33200ff620 osd=3 4 2024-02-23T00:41:58.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 3 8 2024-02-23T00:41:58.221 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:41:58.221 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 10 client.38210.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 9 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6800/4175844757 conn(0x7f332011bdf0 0x7f3320123450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332011c920 osd=4 3 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011c920 osd=4 3 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332011c920 osd=4 3 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 9 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f332011c920 osd=4 3 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 4 9 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011c920 osd=4 4 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 10 client.38210.objecter _send_command 9 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332011cac0 con 0x7f332011bdf0 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.219+0000 7f3326388700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6800/4175844757 conn(0x7f332011bdf0 0x7f3320123450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.220+0000 7f33177fe700 10 client.38210.objecter ms_handle_connect 0x7f332011bdf0 2024-02-23T00:41:58.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.220+0000 7f3326388700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6800/4175844757 conn(0x7f332011bdf0 0x7f3320123450 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.220+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f332011cac0 con 0x7f332011bdf0 2024-02-23T00:41:58.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.220+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:41:58.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.220+0000 7f33177fe700 10 client.38210.objecter _finish_command 9 = system:0 2024-02-23T00:41:58.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.220+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f332011c920 osd=4 4 2024-02-23T00:41:58.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.220+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 4 9 2024-02-23T00:41:58.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 10 client.38210.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:58.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 10 2024-02-23T00:41:58.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332011c920 osd=4 3 2024-02-23T00:41:58.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011c920 osd=4 3 2024-02-23T00:41:58.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332011c920 osd=4 3 2024-02-23T00:41:58.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 10 2024-02-23T00:41:58.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f332011c920 osd=4 3 2024-02-23T00:41:58.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 4 10 2024-02-23T00:41:58.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011c920 osd=4 4 2024-02-23T00:41:58.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 10 client.38210.objecter _send_command 10 2024-02-23T00:41:58.237 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.235+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f332011cac0 con 0x7f332011bdf0 2024-02-23T00:41:58.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.236+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f332011cac0 con 0x7f332011bdf0 2024-02-23T00:41:58.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.236+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:58.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.236+0000 7f33177fe700 10 client.38210.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f332011c920 osd=4 4 2024-02-23T00:41:58.238 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 4 10 2024-02-23T00:41:58.238 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:41:58.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 10 client.38210.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:58.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 11 2024-02-23T00:41:58.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3320123e80 0x7f33201242e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332011cac0 osd=5 3 2024-02-23T00:41:58.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011cac0 osd=5 3 2024-02-23T00:41:58.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332011cac0 osd=5 3 2024-02-23T00:41:58.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 11 2024-02-23T00:41:58.239 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f332011cac0 osd=5 3 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 5 11 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011cac0 osd=5 4 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 10 client.38210.objecter _send_command 11 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332012b320 con 0x7f3320123e80 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f3325b87700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3320123e80 0x7f33201242e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f33177fe700 10 client.38210.objecter ms_handle_connect 0x7f3320123e80 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.237+0000 7f3325b87700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3320123e80 0x7f33201242e0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.238+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f332012b320 con 0x7f3320123e80 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.238+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.238+0000 7f33177fe700 10 client.38210.objecter _finish_command 11 = system:0 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.238+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f332011cac0 osd=5 4 2024-02-23T00:41:58.240 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.238+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 5 11 2024-02-23T00:41:58.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 10 client.38210.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:58.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 12 2024-02-23T00:41:58.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332011cac0 osd=5 3 2024-02-23T00:41:58.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011cac0 osd=5 3 2024-02-23T00:41:58.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332011cac0 osd=5 3 2024-02-23T00:41:58.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 12 2024-02-23T00:41:58.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f332011cac0 osd=5 3 2024-02-23T00:41:58.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 5 12 2024-02-23T00:41:58.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011cac0 osd=5 4 2024-02-23T00:41:58.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 10 client.38210.objecter _send_command 12 2024-02-23T00:41:58.254 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.252+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3320128bc0 con 0x7f3320123e80 2024-02-23T00:41:58.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f3320128bc0 con 0x7f3320123e80 2024-02-23T00:41:58.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:58.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33177fe700 10 client.38210.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f332011cac0 osd=5 4 2024-02-23T00:41:58.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 5 12 2024-02-23T00:41:58.256 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 10 client.38210.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 13 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3320129240 0x7f332012b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320128bc0 osd=6 3 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320128bc0 osd=6 3 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320128bc0 osd=6 3 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 13 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f3320128bc0 osd=6 3 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 6 13 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320128bc0 osd=6 4 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 10 client.38210.objecter _send_command 13 2024-02-23T00:41:58.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332012e470 con 0x7f3320129240 2024-02-23T00:41:58.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f3326b89700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3320129240 0x7f332012b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f33177fe700 10 client.38210.objecter ms_handle_connect 0x7f3320129240 2024-02-23T00:41:58.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.254+0000 7f3326b89700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3320129240 0x7f332012b6b0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.255+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f332012e470 con 0x7f3320129240 2024-02-23T00:41:58.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.255+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:41:58.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.255+0000 7f33177fe700 10 client.38210.objecter _finish_command 13 = system:0 2024-02-23T00:41:58.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.255+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f3320128bc0 osd=6 4 2024-02-23T00:41:58.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.255+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 6 13 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 10 client.38210.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 14 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320128bc0 osd=6 3 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320128bc0 osd=6 3 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f3320128bc0 osd=6 3 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 14 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f3320128bc0 osd=6 3 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 6 14 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320128bc0 osd=6 4 2024-02-23T00:41:58.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 10 client.38210.objecter _send_command 14 2024-02-23T00:41:58.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.269+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f332012e470 con 0x7f3320129240 2024-02-23T00:41:58.272 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:41:58.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.271+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f332012e470 con 0x7f3320129240 2024-02-23T00:41:58.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.271+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:58.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.271+0000 7f33177fe700 10 client.38210.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.271+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f3320128bc0 osd=6 4 2024-02-23T00:41:58.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.271+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 6 14 2024-02-23T00:41:58.274 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:41:58.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 10 client.38210.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:41:58.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 15 2024-02-23T00:41:58.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6812/373787000 conn(0x7f332012d7b0 0x7f332012dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:41:58.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332012e470 osd=7 3 2024-02-23T00:41:58.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332012e470 osd=7 3 2024-02-23T00:41:58.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332012e470 osd=7 3 2024-02-23T00:41:58.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 15 2024-02-23T00:41:58.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f332012e470 osd=7 3 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 7 15 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332012e470 osd=7 4 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 10 client.38210.objecter _send_command 15 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f332012e240 con 0x7f332012d7b0 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f3326388700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6812/373787000 conn(0x7f332012d7b0 0x7f332012dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f3326388700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6812/373787000 conn(0x7f332012d7b0 0x7f332012dbf0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.272+0000 7f33177fe700 10 client.38210.objecter ms_handle_connect 0x7f332012d7b0 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.273+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f332012e240 con 0x7f332012d7b0 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.273+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.273+0000 7f33177fe700 10 client.38210.objecter _finish_command 15 = system:0 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.273+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f332012e470 osd=7 4 2024-02-23T00:41:58.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.273+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 7 15 2024-02-23T00:41:58.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 10 client.38210.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:41:58.292 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign -1 16 2024-02-23T00:41:58.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332012e470 osd=7 3 2024-02-23T00:41:58.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332012e470 osd=7 3 2024-02-23T00:41:58.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 20 client.38210.objecter _get_session s=0x7f332012e470 osd=7 3 2024-02-23T00:41:58.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 15 client.38210.objecter _session_command_op_remove -1 16 2024-02-23T00:41:58.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 20 client.38210.objecter get_session s=0x7f332012e470 osd=7 3 2024-02-23T00:41:58.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 15 client.38210.objecter _session_command_op_assign 7 16 2024-02-23T00:41:58.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332012e470 osd=7 4 2024-02-23T00:41:58.293 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 10 client.38210.objecter _send_command 16 2024-02-23T00:41:58.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.291+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f332012e240 con 0x7f332012d7b0 2024-02-23T00:41:58.294 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33177fe700 1 -- 172.21.15.87:0/2847108917 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f332012e240 con 0x7f332012d7b0 2024-02-23T00:41:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33177fe700 10 client.38210.objecter ms_dispatch 0x7f3320001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:41:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33177fe700 10 client.38210.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33177fe700 20 client.38210.objecter put_session s=0x7f332012e470 osd=7 4 2024-02-23T00:41:58.295 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33177fe700 15 client.38210.objecter _session_command_op_remove 7 16 2024-02-23T00:41:58.296 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:41:58.297 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 librados: watch_flush enter 2024-02-23T00:41:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 librados: watch_flush exit 2024-02-23T00:41:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 client.38210.objecter close_session for osd.0 2024-02-23T00:41:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6801/484081831 conn(0x7f3320028540 msgr2=0x7f33200fe970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6801/484081831 conn(0x7f3320028540 0x7f33200fe970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332006c5b0 osd=0 1 2024-02-23T00:41:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 client.38210.objecter close_session for osd.1 2024-02-23T00:41:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6805/685354171 conn(0x7f3320027b90 msgr2=0x7f3320027f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6805/685354171 conn(0x7f3320027b90 0x7f3320027f90 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320100150 osd=1 1 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 client.38210.objecter close_session for osd.2 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6809/3605394228 conn(0x7f33200ffcb0 msgr2=0x7f332011c290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6809/3605394228 conn(0x7f33200ffcb0 0x7f332011c290 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320027a10 osd=2 1 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 client.38210.objecter close_session for osd.3 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6813/128059790 conn(0x7f332011f9f0 msgr2=0x7f3320121de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6813/128059790 conn(0x7f332011f9f0 0x7f3320121de0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f33200ff620 osd=3 1 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 client.38210.objecter close_session for osd.4 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6800/4175844757 conn(0x7f332011bdf0 msgr2=0x7f3320123450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6800/4175844757 conn(0x7f332011bdf0 0x7f3320123450 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011c920 osd=4 1 2024-02-23T00:41:58.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 client.38210.objecter close_session for osd.5 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3320123e80 msgr2=0x7f33201242e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f3326b89700 1 -- 172.21.15.87:0/2847108917 reap_dead start 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6804/2550013795 conn(0x7f3320123e80 0x7f33201242e0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f3326b89700 1 -- 172.21.15.87:0/2847108917 reap_dead start 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f3326b89700 1 -- 172.21.15.87:0/2847108917 reap_dead start 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332011cac0 osd=5 1 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 client.38210.objecter close_session for osd.6 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3320129240 msgr2=0x7f332012b6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6808/2509408498 conn(0x7f3320129240 0x7f332012b6b0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f3320128bc0 osd=6 1 2024-02-23T00:41:58.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 client.38210.objecter close_session for osd.7 2024-02-23T00:41:58.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6812/373787000 conn(0x7f332012d7b0 msgr2=0x7f332012dbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.171:6812/373787000 conn(0x7f332012d7b0 0x7f332012dbf0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:41:58.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 20 client.38210.objecter put_session s=0x7f332012e470 osd=7 1 2024-02-23T00:41:58.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 20 client.38210.objecter shutdown clearing up homeless session... 2024-02-23T00:41:58.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 10 client.38210.objecter successfully canceled tick 2024-02-23T00:41:58.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6800/1231597463 conn(0x7f330c062c90 msgr2=0x7f330c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> v2:172.21.15.87:6800/1231597463 conn(0x7f330c062c90 0x7f330c065150 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 rx=0x7f331c0043d0 tx=0x7f331c00b040).stop 2024-02-23T00:41:58.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33200fe510 msgr2=0x7f332006bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:41:58.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 --2- 172.21.15.87:0/2847108917 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f33200fe510 0x7f332006bab0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 rx=0x7f3318001cb0 tx=0x7f3318004c50).stop 2024-02-23T00:41:58.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f3326b89700 1 -- 172.21.15.87:0/2847108917 reap_dead start 2024-02-23T00:41:58.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 shutdown_connections 2024-02-23T00:41:58.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 >> 172.21.15.87:0/2847108917 conn(0x7f33200f3800 msgr2=0x7f33200f53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:41:58.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 shutdown_connections 2024-02-23T00:41:58.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 -- 172.21.15.87:0/2847108917 wait complete. 2024-02-23T00:41:58.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:41:58.293+0000 7f33285ec700 1 librados: shutdown 2024-02-23T00:41:58.303 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.303 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.303 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.303 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.303 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.303 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.303 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.303 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:41:58.303 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:58.323 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:41:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:58.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:41:58.427 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:41:58.452 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:58.147324+0000 mon.c (mon.2) 490 : audit [DBG] from='client.? 172.21.15.87:0/2847108917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:58.452 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:58.212014+0000 mon.a (mon.0) 3563 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-120"}]': finished 2024-02-23T00:41:58.452 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:58.212069+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-02-23T00:41:58.452 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:58.213469+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-120", "mode": "writeback"}]: dispatch 2024-02-23T00:41:58.452 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:58.147324+0000 mon.c (mon.2) 490 : audit [DBG] from='client.? 172.21.15.87:0/2847108917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:58.453 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:58.212014+0000 mon.a (mon.0) 3563 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-120"}]': finished 2024-02-23T00:41:58.453 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:58.212069+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-02-23T00:41:58.453 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:58.213469+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-120", "mode": "writeback"}]: dispatch 2024-02-23T00:41:58.454 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:58.579 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:58.147324+0000 mon.c (mon.2) 490 : audit [DBG] from='client.? 172.21.15.87:0/2847108917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:41:58.580 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:58.212014+0000 mon.a (mon.0) 3563 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-120"}]': finished 2024-02-23T00:41:58.580 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:58.212069+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-02-23T00:41:58.580 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:58.213469+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-120", "mode": "writeback"}]: dispatch 2024-02-23T00:41:58.861 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-23T00:41:59.126 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:41:59.132 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:41:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:59.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:41:59.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:41:59.234 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:41:59.263 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:59.463 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:41:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:59.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:41:59.501 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:41:59.567 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:41:59.591 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:57.775202+0000 mgr.y (mgr.24437) 599 : cluster [DBG] pgmap v1061: 194 pgs: 29 creating+peering, 3 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:41:59.591 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:59.209167+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:59.591 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:59.213970+0000 mon.a (mon.0) 3567 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-120", "mode": "writeback"}]': finished 2024-02-23T00:41:59.591 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:59.214042+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-02-23T00:41:59.591 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:41:59.296301+0000 mon.a (mon.0) 3569 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:59.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:57.775202+0000 mgr.y (mgr.24437) 599 : cluster [DBG] pgmap v1061: 194 pgs: 29 creating+peering, 3 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:41:59.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:59.209167+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:59.591 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:59.213970+0000 mon.a (mon.0) 3567 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-120", "mode": "writeback"}]': finished 2024-02-23T00:41:59.592 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:59.214042+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-02-23T00:41:59.592 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:41:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:41:59.296301+0000 mon.a (mon.0) 3569 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:59.593 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:41:59.676 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:57.775202+0000 mgr.y (mgr.24437) 599 : cluster [DBG] pgmap v1061: 194 pgs: 29 creating+peering, 3 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:41:59.676 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:59.209167+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:41:59.676 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:59.213970+0000 mon.a (mon.0) 3567 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-120", "mode": "writeback"}]': finished 2024-02-23T00:41:59.677 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:59.214042+0000 mon.a (mon.0) 3568 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-02-23T00:41:59.677 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:41:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:41:59.296301+0000 mon.a (mon.0) 3569 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:41:59.791 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:41:59.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:41:59.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:41:59.895 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:41:59.923 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:00.071 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:00.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:00.149 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-23T00:42:00.149 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:42:00.150 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-02-23T00:42:00.150 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-23T00:42:00.150 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-23T00:42:00.150 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-02-23T00:42:00.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:00.223 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:00.421 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:00.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:00.475 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:00.525 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:00.553 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:01.067 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:42:01.102 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:01.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:01.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:01.182 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:42:01.182 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:42:01.207 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:01.243 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:41:59.775607+0000 mgr.y (mgr.24437) 600 : cluster [DBG] pgmap v1064: 194 pgs: 29 creating+peering, 2 active+remapped+backfill_toofull, 163 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:00.215375+0000 mon.a (mon.0) 3570 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:00.215433+0000 mon.a (mon.0) 3571 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-02-23T00:42:01.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:00.216325+0000 mon.a (mon.0) 3572 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120"}]: dispatch 2024-02-23T00:42:01.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:41:59.775607+0000 mgr.y (mgr.24437) 600 : cluster [DBG] pgmap v1064: 194 pgs: 29 creating+peering, 2 active+remapped+backfill_toofull, 163 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:00.215375+0000 mon.a (mon.0) 3570 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:00.215433+0000 mon.a (mon.0) 3571 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-02-23T00:42:01.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:00.216325+0000 mon.a (mon.0) 3572 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120"}]: dispatch 2024-02-23T00:42:01.444 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:01.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:01.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:41:59.775607+0000 mgr.y (mgr.24437) 600 : cluster [DBG] pgmap v1064: 194 pgs: 29 creating+peering, 2 active+remapped+backfill_toofull, 163 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:01.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:00.215375+0000 mon.a (mon.0) 3570 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:01.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:00.215433+0000 mon.a (mon.0) 3571 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-02-23T00:42:01.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:00.216325+0000 mon.a (mon.0) 3572 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120"}]: dispatch 2024-02-23T00:42:01.521 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:42:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:42:01.547 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:01.579 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:01.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.679+0000 7fed9deee700 1 -- 172.21.15.87:0/4186431864 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 msgr2=0x7fed98104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.679+0000 7fed9deee700 1 --2- 172.21.15.87:0/4186431864 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 0x7fed98104a20 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 rx=0x7fed90009be0 tx=0x7fed90004d10).stop 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 -- 172.21.15.87:0/4186431864 shutdown_connections 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 --2- 172.21.15.87:0/4186431864 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 0x7fed98104a20 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 --2- 172.21.15.87:0/4186431864 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed980fdbb0 0x7fed980fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 --2- 172.21.15.87:0/4186431864 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fed980fe510 0x7fed980f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 -- 172.21.15.87:0/4186431864 >> 172.21.15.87:0/4186431864 conn(0x7fed980f3800 msgr2=0x7fed980f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 -- 172.21.15.87:0/4186431864 shutdown_connections 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 -- 172.21.15.87:0/4186431864 wait complete. 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 librados: starting msgr at 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 librados: starting objecter 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.680+0000 7fed9deee700 1 Processor -- start 2024-02-23T00:42:01.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed9deee700 1 -- start start 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed9deee700 1 librados: setting wanted keys 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed9deee700 1 librados: calling monclient init 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed9deee700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 0x7fed9810de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed9deee700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed980fdbb0 0x7fed9810e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed9deee700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fed980fe510 0x7fed98108030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed9deee700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed98108520 con 0x7fed980fdbb0 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed9deee700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed98108690 con 0x7fed980fe510 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed9deee700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed98108850 con 0x7fed980f8420 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed977fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 0x7fed9810de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed977fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 0x7fed9810de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:46270/0 (socket says 172.21.15.87:46270) 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed977fe700 1 -- 172.21.15.87:0/335085347 learned_addr learned my addr 172.21.15.87:0/335085347 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed96ffd700 1 --2- 172.21.15.87:0/335085347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed980fdbb0 0x7fed9810e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed977fe700 1 -- 172.21.15.87:0/335085347 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fed980fe510 msgr2=0x7fed98108030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed977fe700 1 --2- 172.21.15.87:0/335085347 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fed980fe510 0x7fed98108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:42:01.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed977fe700 1 -- 172.21.15.87:0/335085347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed980fdbb0 msgr2=0x7fed9810e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed977fe700 1 --2- 172.21.15.87:0/335085347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed980fdbb0 0x7fed9810e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed977fe700 1 -- 172.21.15.87:0/335085347 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed88005040 con 0x7fed980f8420 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed977fe700 1 --2- 172.21.15.87:0/335085347 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 0x7fed9810de90 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7fed880048f0 tx=0x7fed88009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed94ff9700 10 client.?.objecter ms_handle_connect 0x7fed980f8420 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed94ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.681+0000 7fed94ff9700 1 -- 172.21.15.87:0/335085347 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed88007b00 con 0x7fed980f8420 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.682+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed900097c0 con 0x7fed980f8420 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.682+0000 7fed94ff9700 1 -- 172.21.15.87:0/335085347 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed8800f040 con 0x7fed980f8420 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.682+0000 7fed94ff9700 1 -- 172.21.15.87:0/335085347 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed88014a00 con 0x7fed980f8420 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.682+0000 7fed9deee700 10 client.38216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.682+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed98108e90 con 0x7fed980f8420 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.682+0000 7fed9deee700 1 librados: init done 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.683+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed98107110 con 0x7fed980f8420 2024-02-23T00:42:01.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.683+0000 7fed94ff9700 1 -- 172.21.15.87:0/335085347 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fed8800e580 con 0x7fed980f8420 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.683+0000 7fed94ff9700 1 --2- 172.21.15.87:0/335085347 >> v2:172.21.15.87:6800/1231597463 conn(0x7fed80062c90 0x7fed80065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.683+0000 7fed94ff9700 1 -- 172.21.15.87:0/335085347 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(699..699 src has 173..699) v4 ==== 8535+0+0 (secure 0 0 0) 0x7fed8807cfa0 con 0x7fed980f8420 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.683+0000 7fed94ff9700 10 client.38216.objecter ms_dispatch 0x7fed98001550 osd_map(699..699 src has 173..699) v4 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.683+0000 7fed94ff9700 3 client.38216.objecter handle_osd_map got epochs [699,699] > 0 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.683+0000 7fed94ff9700 3 client.38216.objecter handle_osd_map decoding full epoch 699 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.683+0000 7fed96ffd700 1 --2- 172.21.15.87:0/335085347 >> v2:172.21.15.87:6800/1231597463 conn(0x7fed80062c90 0x7fed80065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.683+0000 7fed94ff9700 20 client.38216.objecter dump_active .. 0 homeless 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.684+0000 7fed94ff9700 10 client.38216.objecter ms_handle_connect 0x7fed80062c90 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.684+0000 7fed96ffd700 1 --2- 172.21.15.87:0/335085347 >> v2:172.21.15.87:6800/1231597463 conn(0x7fed80062c90 0x7fed80065150 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7fed981097d0 tx=0x7fed8c009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:01.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.688+0000 7fed94ff9700 1 -- 172.21.15.87:0/335085347 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fed88049f60 con 0x7fed980f8420 2024-02-23T00:42:01.774 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:01.778 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:01.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:01.786 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-23T00:42:01.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:01.879+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fed9805fb70 con 0x7fed980f8420 2024-02-23T00:42:01.883 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:01.911 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:01.986 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:01.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:02.089 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:02.091 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:42:02.117 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:02.242 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:02.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.240+0000 7fed94ff9700 1 -- 172.21.15.87:0/335085347 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v700) v1 ==== 97+0+0 (secure 0 0 0) 0x7fed88050290 con 0x7fed980f8420 2024-02-23T00:42:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:02.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 10 librados: watch_flush enter 2024-02-23T00:42:02.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 10 librados: watch_flush exit 2024-02-23T00:42:02.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 20 client.38216.objecter shutdown clearing up homeless session... 2024-02-23T00:42:02.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 10 client.38216.objecter successfully canceled tick 2024-02-23T00:42:02.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 >> v2:172.21.15.87:6800/1231597463 conn(0x7fed80062c90 msgr2=0x7fed80065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:02.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 --2- 172.21.15.87:0/335085347 >> v2:172.21.15.87:6800/1231597463 conn(0x7fed80062c90 0x7fed80065150 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 rx=0x7fed981097d0 tx=0x7fed8c009040).stop 2024-02-23T00:42:02.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 msgr2=0x7fed9810de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 --2- 172.21.15.87:0/335085347 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 0x7fed9810de90 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 rx=0x7fed880048f0 tx=0x7fed88009cb0).stop 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 shutdown_connections 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 --2- 172.21.15.87:0/335085347 >> v2:172.21.15.87:6800/1231597463 conn(0x7fed80062c90 0x7fed80065150 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 --2- 172.21.15.87:0/335085347 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fed980fe510 0x7fed98108030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 --2- 172.21.15.87:0/335085347 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fed980f8420 0x7fed9810de90 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 --2- 172.21.15.87:0/335085347 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fed980fdbb0 0x7fed9810e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 >> 172.21.15.87:0/335085347 conn(0x7fed980f3800 msgr2=0x7fed980f5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 shutdown_connections 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 -- 172.21.15.87:0/335085347 wait complete. 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:02.246+0000 7fed9deee700 1 librados: shutdown 2024-02-23T00:42:02.250 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:42:02.350 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:02.377 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:02.397 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:02.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:02.501 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:02.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:01.202806+0000 mgr.y (mgr.24437) 601 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:02.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:01.211785+0000 mon.a (mon.0) 3573 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:02.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:01.219024+0000 mon.a (mon.0) 3574 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120"}]': finished 2024-02-23T00:42:02.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:01.219091+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-02-23T00:42:02.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:01.880157+0000 mon.c (mon.2) 491 : audit [INF] from='client.? 172.21.15.87:0/335085347' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:02.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:01.880445+0000 mon.a (mon.0) 3576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:02.526 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:02.206891+0000 mon.a (mon.0) 3577 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:42:02.528 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:02.529 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:02] "GET /metrics HTTP/1.1" 200 187633 "" "Prometheus/2.33.4" 2024-02-23T00:42:02.529 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:01.202806+0000 mgr.y (mgr.24437) 601 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:01.211785+0000 mon.a (mon.0) 3573 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:01.219024+0000 mon.a (mon.0) 3574 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120"}]': finished 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:01.219091+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:01.880157+0000 mon.c (mon.2) 491 : audit [INF] from='client.? 172.21.15.87:0/335085347' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:01.880445+0000 mon.a (mon.0) 3576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:02.206891+0000 mon.a (mon.0) 3577 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:01.202806+0000 mgr.y (mgr.24437) 601 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:01.211785+0000 mon.a (mon.0) 3573 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:01.219024+0000 mon.a (mon.0) 3574 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-120"}]': finished 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:01.219091+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-02-23T00:42:02.530 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:01.880157+0000 mon.c (mon.2) 491 : audit [INF] from='client.? 172.21.15.87:0/335085347' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:02.531 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:01.880445+0000 mon.a (mon.0) 3576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:02.531 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:02.206891+0000 mon.a (mon.0) 3577 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:42:03.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:01.776003+0000 mgr.y (mgr.24437) 602 : cluster [DBG] pgmap v1067: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:03.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:02.222632+0000 mon.a (mon.0) 3578 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:03.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:02.235505+0000 mon.a (mon.0) 3579 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:42:03.325 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:02.235569+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-02-23T00:42:03.377 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:03.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:03.481 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:03.510 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:01.776003+0000 mgr.y (mgr.24437) 602 : cluster [DBG] pgmap v1067: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:02.222632+0000 mon.a (mon.0) 3578 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:02.235505+0000 mon.a (mon.0) 3579 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:42:03.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:02.235569+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-02-23T00:42:03.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:01.776003+0000 mgr.y (mgr.24437) 602 : cluster [DBG] pgmap v1067: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:02.222632+0000 mon.a (mon.0) 3578 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:02.235505+0000 mon.a (mon.0) 3579 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:42:03.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:02.235569+0000 mon.a (mon.0) 3580 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-02-23T00:42:03.609 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:03.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:03.713 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:03.743 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:03.841 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:03.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:03.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:03.862 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:42:03.862 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:42:03.887 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:42:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:42:03.945 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:03.973 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:04.120 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:04.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:04.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:04.265 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:04.295 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:04.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:03.234623+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-02-23T00:42:04.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:03.236915+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:04.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:03.234623+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-02-23T00:42:04.297 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:03.236915+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:04.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:04.399 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:04.427 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:04.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:03.234623+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-02-23T00:42:04.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:03.236915+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:04.550 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:04.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:04.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:04.567 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-23T00:42:04.654 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:04.680 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:04.699 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:04.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:04.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:04.803 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:04.831 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:04.850 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:04.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:04.880 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:42:04.956 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:04.982 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:05.169 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:42:05.169 INFO:teuthology.orchestra.run.smithi087.stdout: "success": "" 2024-02-23T00:42:05.169 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:42:05.181 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:05.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:05.285 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:05.313 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:03.776608+0000 mgr.y (mgr.24437) 603 : cluster [DBG] pgmap v1070: 194 pgs: 32 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:05.313 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:04.245544+0000 mon.a (mon.0) 3583 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:05.313 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:04.245734+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-02-23T00:42:05.313 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:04.255968+0000 mon.a (mon.0) 3585 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:05.314 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:03.776608+0000 mgr.y (mgr.24437) 603 : cluster [DBG] pgmap v1070: 194 pgs: 32 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:05.314 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:04.245544+0000 mon.a (mon.0) 3583 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:05.314 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:04.245734+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-02-23T00:42:05.314 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:04.255968+0000 mon.a (mon.0) 3585 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:05.316 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:05.354 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:05.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:03.776608+0000 mgr.y (mgr.24437) 603 : cluster [DBG] pgmap v1070: 194 pgs: 32 unknown, 2 active+remapped+backfill_toofull, 160 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:04.245544+0000 mon.a (mon.0) 3583 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:04.245734+0000 mon.a (mon.0) 3584 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-02-23T00:42:05.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:04.255968+0000 mon.a (mon.0) 3585 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:05.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:05.486 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:05.835 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:05.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:05.939 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:05.966 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:06.064 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:06.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:06.167 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:06.197 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:06.217 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:06.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:06.321 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:06.349 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:06.451 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:05.248533+0000 mon.a (mon.0) 3586 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:06.452 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:05.248609+0000 mon.a (mon.0) 3587 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-02-23T00:42:06.452 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:05.250361+0000 mon.a (mon.0) 3588 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]: dispatch 2024-02-23T00:42:06.497 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:06.499 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:05.248533+0000 mon.a (mon.0) 3586 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:06.499 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:05.248609+0000 mon.a (mon.0) 3587 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-02-23T00:42:06.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:05.250361+0000 mon.a (mon.0) 3588 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]: dispatch 2024-02-23T00:42:06.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:05.248533+0000 mon.a (mon.0) 3586 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:06.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:05.248609+0000 mon.a (mon.0) 3587 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-02-23T00:42:06.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:05.250361+0000 mon.a (mon.0) 3588 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]: dispatch 2024-02-23T00:42:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:06.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:06.605 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:06.635 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:06.709 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:06.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:06.715 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:42:06.715 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:42:06.813 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:06.839 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:06.859 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:06.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:06.963 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:06.991 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:07.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.111+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/414846404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc0f7400 msgr2=0x7f8edc103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.111+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/414846404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc0f7400 0x7f8edc103a20 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 rx=0x7f8ed4009be0 tx=0x7f8ed4004d10).stop 2024-02-23T00:42:07.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.111+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/414846404 shutdown_connections 2024-02-23T00:42:07.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.111+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/414846404 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc0f7400 0x7f8edc103a20 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.111+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/414846404 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8edc0fcbb0 0x7f8edc0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.111+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/414846404 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8edc0fd510 0x7f8edc0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.111+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/414846404 >> 172.21.15.87:0/414846404 conn(0x7f8edc0f2820 msgr2=0x7f8edc0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:07.116 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/414846404 shutdown_connections 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/414846404 wait complete. 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 librados: starting msgr at 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 librados: starting objecter 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 Processor -- start 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 -- start start 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 librados: setting wanted keys 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 librados: calling monclient init 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc0f7400 0x7f8edc10ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8edc0fcbb0 0x7f8edc10d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8edc0fd510 0x7f8edc107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.117 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc107540 con 0x7f8edc0f7400 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc1076b0 con 0x7f8edc0fd510 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee3d6d700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc107870 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee1b09700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc0f7400 0x7f8edc10ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee1b09700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc0f7400 0x7f8edc10ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:59416/0 (socket says 172.21.15.87:59416) 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee1b09700 1 -- 172.21.15.87:0/2148054030 learned_addr learned my addr 172.21.15.87:0/2148054030 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.112+0000 7f8ee1308700 1 --2- 172.21.15.87:0/2148054030 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8edc0fcbb0 0x7f8edc10d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee230a700 1 --2- 172.21.15.87:0/2148054030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8edc0fd510 0x7f8edc107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee1308700 1 -- 172.21.15.87:0/2148054030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8edc0fd510 msgr2=0x7f8edc107050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee1308700 1 --2- 172.21.15.87:0/2148054030 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f8edc0fd510 0x7f8edc107050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee1308700 1 -- 172.21.15.87:0/2148054030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc0f7400 msgr2=0x7f8edc10ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee1308700 1 --2- 172.21.15.87:0/2148054030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc0f7400 0x7f8edc10ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee1308700 1 -- 172.21.15.87:0/2148054030 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed8005040 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.118 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee1b09700 1 --2- 172.21.15.87:0/2148054030 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f8edc0f7400 0x7f8edc10ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee1308700 1 --2- 172.21.15.87:0/2148054030 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8edc0fcbb0 0x7f8edc10d350 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7f8ed8004920 tx=0x7f8ed8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ed2ffd700 10 client.?.objecter ms_handle_connect 0x7f8edc0fcbb0 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ed2ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed800e030 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ed800f040 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed40097c0 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8014850 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee3d6d700 10 client.38222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8edc107e50 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.119 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.113+0000 7f8ee3d6d700 1 librados: init done 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.114+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8edc05fa60 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.114+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f8ed8010500 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ed2ffd700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8ec8062c90 0x7f8ec8065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(704..704 src has 173..704) v4 ==== 8543+0+0 (secure 0 0 0) 0x7f8ed807d4d0 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 osd_map(704..704 src has 173..704) v4 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ed2ffd700 3 client.38222.objecter handle_osd_map got epochs [704,704] > 0 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ed2ffd700 3 client.38222.objecter handle_osd_map decoding full epoch 704 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee1b09700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8ec8062c90 0x7f8ec8065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ed2ffd700 20 client.38222.objecter dump_active .. 0 homeless 2024-02-23T00:42:07.121 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v704) v1 ==== 54+0+16 (secure 0 0 0) 0x7f8ed8084070 con 0x7f8edc0fcbb0 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ed2ffd700 10 client.38222.objecter ms_handle_connect 0x7f8ec8062c90 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee1b09700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8ec8062c90 0x7f8ec8065150 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7f8ecc004fd0 tx=0x7f8ecc002040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 1 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6801/484081831 conn(0x7f8edc061320 0x7f8edc061700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc05ee80 osd=0 3 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc05ee80 osd=0 3 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc05ee80 osd=0 3 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 1 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc05ee80 osd=0 3 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 0 1 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc05ee80 osd=0 4 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 1 2024-02-23T00:42:07.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8edc106120 con 0x7f8edc061320 2024-02-23T00:42:07.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.115+0000 7f8ee230a700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6801/484081831 conn(0x7f8edc061320 0x7f8edc061700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.116+0000 7f8ed2ffd700 10 client.38222.objecter ms_handle_connect 0x7f8edc061320 2024-02-23T00:42:07.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.116+0000 7f8ee230a700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6801/484081831 conn(0x7f8edc061320 0x7f8edc061700 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.116+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8edc106120 con 0x7f8edc061320 2024-02-23T00:42:07.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.116+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:42:07.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.116+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 1 = system:0 2024-02-23T00:42:07.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.116+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc05ee80 osd=0 4 2024-02-23T00:42:07.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.116+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 0 1 2024-02-23T00:42:07.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:07.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 2 2024-02-23T00:42:07.133 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc05ee80 osd=0 3 2024-02-23T00:42:07.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc05ee80 osd=0 3 2024-02-23T00:42:07.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc05ee80 osd=0 3 2024-02-23T00:42:07.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 2 2024-02-23T00:42:07.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc05ee80 osd=0 3 2024-02-23T00:42:07.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 0 2 2024-02-23T00:42:07.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc05ee80 osd=0 4 2024-02-23T00:42:07.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 2 2024-02-23T00:42:07.134 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.131+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8edc0fd990 con 0x7f8edc061320 2024-02-23T00:42:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.132+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8edc0fd990 con 0x7f8edc061320 2024-02-23T00:42:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.132+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.132+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.132+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc05ee80 osd=0 4 2024-02-23T00:42:07.135 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.132+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 0 2 2024-02-23T00:42:07.135 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:42:07.136 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 3 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6805/685354171 conn(0x7f8edc0f4b70 0x7f8edc0273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0f4a00 osd=1 3 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0f4a00 osd=1 3 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0f4a00 osd=1 3 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 3 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc0f4a00 osd=1 3 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 1 3 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0f4a00 osd=1 4 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 3 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8edc0fe840 con 0x7f8edc0f4b70 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee1b09700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6805/685354171 conn(0x7f8edc0f4b70 0x7f8edc0273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ee1b09700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6805/685354171 conn(0x7f8edc0f4b70 0x7f8edc0273c0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.137 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ed2ffd700 10 client.38222.objecter ms_handle_connect 0x7f8edc0f4b70 2024-02-23T00:42:07.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8edc0fe840 con 0x7f8edc0f4b70 2024-02-23T00:42:07.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:42:07.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 3 = system:0 2024-02-23T00:42:07.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc0f4a00 osd=1 4 2024-02-23T00:42:07.138 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.133+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 1 3 2024-02-23T00:42:07.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:07.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 4 2024-02-23T00:42:07.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0f4a00 osd=1 3 2024-02-23T00:42:07.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0f4a00 osd=1 3 2024-02-23T00:42:07.150 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0f4a00 osd=1 3 2024-02-23T00:42:07.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 4 2024-02-23T00:42:07.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc0f4a00 osd=1 3 2024-02-23T00:42:07.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 1 4 2024-02-23T00:42:07.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0f4a00 osd=1 4 2024-02-23T00:42:07.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 4 2024-02-23T00:42:07.151 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.148+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8edc0fdda0 con 0x7f8edc0f4b70 2024-02-23T00:42:07.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.149+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8edc0fdda0 con 0x7f8edc0f4b70 2024-02-23T00:42:07.152 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.149+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:07.152 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:42:07.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.149+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.149+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc0f4a00 osd=1 4 2024-02-23T00:42:07.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.149+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 1 4 2024-02-23T00:42:07.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:07.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 5 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8edc028590 0x7f8edc119cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0fe1b0 osd=2 3 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0fe1b0 osd=2 3 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0fe1b0 osd=2 3 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 5 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc0fe1b0 osd=2 3 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 2 5 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0fe1b0 osd=2 4 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 5 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8edc11ca60 con 0x7f8edc028590 2024-02-23T00:42:07.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee1308700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8edc028590 0x7f8edc119cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ed2ffd700 10 client.38222.objecter ms_handle_connect 0x7f8edc028590 2024-02-23T00:42:07.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ee1308700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8edc028590 0x7f8edc119cc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8edc11ca60 con 0x7f8edc028590 2024-02-23T00:42:07.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:42:07.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 5 = system:0 2024-02-23T00:42:07.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc0fe1b0 osd=2 4 2024-02-23T00:42:07.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.150+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 2 5 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 6 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0fe1b0 osd=2 3 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0fe1b0 osd=2 3 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0fe1b0 osd=2 3 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 6 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc0fe1b0 osd=2 3 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 2 6 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0fe1b0 osd=2 4 2024-02-23T00:42:07.167 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 6 2024-02-23T00:42:07.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.165+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8edc027bb0 con 0x7f8edc028590 2024-02-23T00:42:07.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8edc027bb0 con 0x7f8edc028590 2024-02-23T00:42:07.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:07.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc0fe1b0 osd=2 4 2024-02-23T00:42:07.168 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 2 6 2024-02-23T00:42:07.169 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:42:07.169 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 7 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6813/128059790 conn(0x7f8edc11a1b0 0x7f8edc11a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0598f0 osd=3 3 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0598f0 osd=3 3 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0598f0 osd=3 3 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 7 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc0598f0 osd=3 3 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 3 7 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0598f0 osd=3 4 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 7 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8edc05a0e0 con 0x7f8edc11a1b0 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.166+0000 7f8ee230a700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6813/128059790 conn(0x7f8edc11a1b0 0x7f8edc11a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.167+0000 7f8ed2ffd700 10 client.38222.objecter ms_handle_connect 0x7f8edc11a1b0 2024-02-23T00:42:07.170 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.167+0000 7f8ee230a700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6813/128059790 conn(0x7f8edc11a1b0 0x7f8edc11a5b0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.167+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8edc05a0e0 con 0x7f8edc11a1b0 2024-02-23T00:42:07.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.167+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:42:07.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.167+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 7 = system:0 2024-02-23T00:42:07.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.167+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc0598f0 osd=3 4 2024-02-23T00:42:07.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.167+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 3 7 2024-02-23T00:42:07.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:07.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 8 2024-02-23T00:42:07.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0598f0 osd=3 3 2024-02-23T00:42:07.183 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0598f0 osd=3 3 2024-02-23T00:42:07.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc0598f0 osd=3 3 2024-02-23T00:42:07.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 8 2024-02-23T00:42:07.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc0598f0 osd=3 3 2024-02-23T00:42:07.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 3 8 2024-02-23T00:42:07.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0598f0 osd=3 4 2024-02-23T00:42:07.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 8 2024-02-23T00:42:07.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.181+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8edc11c420 con 0x7f8edc11a1b0 2024-02-23T00:42:07.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.182+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8edc11c420 con 0x7f8edc11a1b0 2024-02-23T00:42:07.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:07.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc0598f0 osd=3 4 2024-02-23T00:42:07.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 3 8 2024-02-23T00:42:07.186 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:42:07.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:07.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 9 2024-02-23T00:42:07.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8edc11ec40 0x7f8edc121030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c420 osd=4 3 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c420 osd=4 3 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c420 osd=4 3 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 9 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc11c420 osd=4 3 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 4 9 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c420 osd=4 4 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 9 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8edc11c6e0 con 0x7f8edc11ec40 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.183+0000 7f8ee1b09700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8edc11ec40 0x7f8edc121030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.184+0000 7f8ed2ffd700 10 client.38222.objecter ms_handle_connect 0x7f8edc11ec40 2024-02-23T00:42:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.184+0000 7f8ee1b09700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8edc11ec40 0x7f8edc121030 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.184+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8edc11c6e0 con 0x7f8edc11ec40 2024-02-23T00:42:07.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.184+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:42:07.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.184+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 9 = system:0 2024-02-23T00:42:07.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.184+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc11c420 osd=4 4 2024-02-23T00:42:07.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.184+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 4 9 2024-02-23T00:42:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 10 2024-02-23T00:42:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c420 osd=4 3 2024-02-23T00:42:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c420 osd=4 3 2024-02-23T00:42:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c420 osd=4 3 2024-02-23T00:42:07.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 10 2024-02-23T00:42:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc11c420 osd=4 3 2024-02-23T00:42:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 4 10 2024-02-23T00:42:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c420 osd=4 4 2024-02-23T00:42:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 10 2024-02-23T00:42:07.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.199+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8edc11c6e0 con 0x7f8edc11ec40 2024-02-23T00:42:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8edc11c6e0 con 0x7f8edc11ec40 2024-02-23T00:42:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc11c420 osd=4 4 2024-02-23T00:42:07.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 4 10 2024-02-23T00:42:07.203 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:42:07.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:07.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 11 2024-02-23T00:42:07.204 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8edc121a30 0x7f8edc121e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c6e0 osd=5 3 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c6e0 osd=5 3 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c6e0 osd=5 3 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 11 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc11c6e0 osd=5 3 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 5 11 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c6e0 osd=5 4 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 11 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8edc123ab0 con 0x7f8edc121a30 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee1308700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8edc121a30 0x7f8edc121e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ed2ffd700 10 client.38222.objecter ms_handle_connect 0x7f8edc121a30 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.201+0000 7f8ee1308700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8edc121a30 0x7f8edc121e70 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.202+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8edc123ab0 con 0x7f8edc121a30 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.202+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:42:07.205 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.202+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 11 = system:0 2024-02-23T00:42:07.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.202+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc11c6e0 osd=5 4 2024-02-23T00:42:07.206 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.202+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 5 11 2024-02-23T00:42:07.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:07.222 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 12 2024-02-23T00:42:07.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c6e0 osd=5 3 2024-02-23T00:42:07.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c6e0 osd=5 3 2024-02-23T00:42:07.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c6e0 osd=5 3 2024-02-23T00:42:07.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 12 2024-02-23T00:42:07.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc11c6e0 osd=5 3 2024-02-23T00:42:07.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 5 12 2024-02-23T00:42:07.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c6e0 osd=5 4 2024-02-23T00:42:07.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 12 2024-02-23T00:42:07.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.220+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8edc123ab0 con 0x7f8edc121a30 2024-02-23T00:42:07.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8edc123ab0 con 0x7f8edc121a30 2024-02-23T00:42:07.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:07.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc11c6e0 osd=5 4 2024-02-23T00:42:07.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 5 12 2024-02-23T00:42:07.224 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:42:07.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:07.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 13 2024-02-23T00:42:07.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8edc11bc50 0x7f8edc122450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc123ab0 osd=6 3 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc123ab0 osd=6 3 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc123ab0 osd=6 3 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 13 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc123ab0 osd=6 3 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 6 13 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc123ab0 osd=6 4 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 13 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8edc11c210 con 0x7f8edc11bc50 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee230a700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8edc11bc50 0x7f8edc122450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ed2ffd700 10 client.38222.objecter ms_handle_connect 0x7f8edc11bc50 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.222+0000 7f8ee230a700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8edc11bc50 0x7f8edc122450 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.223+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8edc11c210 con 0x7f8edc11bc50 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.223+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.223+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 13 = system:0 2024-02-23T00:42:07.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.223+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc123ab0 osd=6 4 2024-02-23T00:42:07.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.223+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 6 13 2024-02-23T00:42:07.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:07.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 14 2024-02-23T00:42:07.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc123ab0 osd=6 3 2024-02-23T00:42:07.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc123ab0 osd=6 3 2024-02-23T00:42:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc123ab0 osd=6 3 2024-02-23T00:42:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 14 2024-02-23T00:42:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc123ab0 osd=6 3 2024-02-23T00:42:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 6 14 2024-02-23T00:42:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc123ab0 osd=6 4 2024-02-23T00:42:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 14 2024-02-23T00:42:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.254+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8edc11c210 con 0x7f8edc11bc50 2024-02-23T00:42:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.256+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8edc11c210 con 0x7f8edc11bc50 2024-02-23T00:42:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.256+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.256+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.256+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc123ab0 osd=6 4 2024-02-23T00:42:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.256+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 6 14 2024-02-23T00:42:07.259 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:42:07.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:07.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 15 2024-02-23T00:42:07.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6812/373787000 conn(0x7f8edc126ab0 0x7f8edc122eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:07.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c210 osd=7 3 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c210 osd=7 3 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c210 osd=7 3 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 15 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc11c210 osd=7 3 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 7 15 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c210 osd=7 4 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 15 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8edc123550 con 0x7f8edc126ab0 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee1b09700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6812/373787000 conn(0x7f8edc126ab0 0x7f8edc122eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ed2ffd700 10 client.38222.objecter ms_handle_connect 0x7f8edc126ab0 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.257+0000 7f8ee1b09700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6812/373787000 conn(0x7f8edc126ab0 0x7f8edc122eb0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.258+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8edc123550 con 0x7f8edc126ab0 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.258+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.258+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 15 = system:0 2024-02-23T00:42:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.258+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc11c210 osd=7 4 2024-02-23T00:42:07.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.258+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 7 15 2024-02-23T00:42:07.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 10 client.38222.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign -1 16 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c210 osd=7 3 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c210 osd=7 3 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 20 client.38222.objecter _get_session s=0x7f8edc11c210 osd=7 3 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_remove -1 16 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 20 client.38222.objecter get_session s=0x7f8edc11c210 osd=7 3 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 15 client.38222.objecter _session_command_op_assign 7 16 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c210 osd=7 4 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 10 client.38222.objecter _send_command 16 2024-02-23T00:42:07.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.272+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8edc123550 con 0x7f8edc126ab0 2024-02-23T00:42:07.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ed2ffd700 1 -- 172.21.15.87:0/2148054030 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8edc123550 con 0x7f8edc126ab0 2024-02-23T00:42:07.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ed2ffd700 10 client.38222.objecter ms_dispatch 0x7f8edc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:07.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ed2ffd700 10 client.38222.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ed2ffd700 20 client.38222.objecter put_session s=0x7f8edc11c210 osd=7 4 2024-02-23T00:42:07.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ed2ffd700 15 client.38222.objecter _session_command_op_remove 7 16 2024-02-23T00:42:07.277 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:42:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 librados: watch_flush enter 2024-02-23T00:42:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 librados: watch_flush exit 2024-02-23T00:42:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 client.38222.objecter close_session for osd.0 2024-02-23T00:42:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6801/484081831 conn(0x7f8edc061320 msgr2=0x7f8edc061700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6801/484081831 conn(0x7f8edc061320 0x7f8edc061700 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc05ee80 osd=0 1 2024-02-23T00:42:07.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 client.38222.objecter close_session for osd.1 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6805/685354171 conn(0x7f8edc0f4b70 msgr2=0x7f8edc0273c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6805/685354171 conn(0x7f8edc0f4b70 0x7f8edc0273c0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0f4a00 osd=1 1 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 client.38222.objecter close_session for osd.2 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8edc028590 msgr2=0x7f8edc119cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8edc028590 0x7f8edc119cc0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0fe1b0 osd=2 1 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 client.38222.objecter close_session for osd.3 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6813/128059790 conn(0x7f8edc11a1b0 msgr2=0x7f8edc11a5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6813/128059790 conn(0x7f8edc11a1b0 0x7f8edc11a5b0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc0598f0 osd=3 1 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 client.38222.objecter close_session for osd.4 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8edc11ec40 msgr2=0x7f8edc121030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8edc11ec40 0x7f8edc121030 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c420 osd=4 1 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 client.38222.objecter close_session for osd.5 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee230a700 1 -- 172.21.15.87:0/2148054030 reap_dead start 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8edc121a30 msgr2=0x7f8edc121e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8edc121a30 0x7f8edc121e70 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee230a700 1 -- 172.21.15.87:0/2148054030 reap_dead start 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c6e0 osd=5 1 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 client.38222.objecter close_session for osd.6 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8edc11bc50 msgr2=0x7f8edc122450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8edc11bc50 0x7f8edc122450 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee230a700 1 -- 172.21.15.87:0/2148054030 reap_dead start 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc123ab0 osd=6 1 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 client.38222.objecter close_session for osd.7 2024-02-23T00:42:07.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6812/373787000 conn(0x7f8edc126ab0 msgr2=0x7f8edc122eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6812/373787000 conn(0x7f8edc126ab0 0x7f8edc122eb0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 20 client.38222.objecter put_session s=0x7f8edc11c210 osd=7 1 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 20 client.38222.objecter shutdown clearing up homeless session... 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 10 client.38222.objecter successfully canceled tick 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8ec8062c90 msgr2=0x7f8ec8065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8ec8062c90 0x7f8ec8065150 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 rx=0x7f8ecc004fd0 tx=0x7f8ecc002040).stop 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8edc0fcbb0 msgr2=0x7f8edc10d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.274+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8edc0fcbb0 0x7f8edc10d350 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 rx=0x7f8ed8004920 tx=0x7f8ed8009cb0).stop 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.275+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 shutdown_connections 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.275+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8edc11bc50 0x7f8edc122450 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.275+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.171:6812/373787000 conn(0x7f8edc126ab0 0x7f8edc122eb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.275+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8ec8062c90 0x7f8ec8065150 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.275+0000 7f8ee3d6d700 1 --2- 172.21.15.87:0/2148054030 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f8edc0fcbb0 0x7f8edc10d350 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:07.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.275+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 >> 172.21.15.87:0/2148054030 conn(0x7f8edc0f2820 msgr2=0x7f8edc0f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.275+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 shutdown_connections 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.275+0000 7f8ee3d6d700 1 -- 172.21.15.87:0/2148054030 wait complete. 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:07.275+0000 7f8ee3d6d700 1 librados: shutdown 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.282 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:42:07.399 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-23T00:42:07.440 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:07.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:07.543 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:07.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:05.777507+0000 mgr.y (mgr.24437) 604 : cluster [DBG] pgmap v1073: 194 pgs: 2 unknown, 2 active+remapped+backfill_toofull, 190 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:07.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:06.259555+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]': finished 2024-02-23T00:42:07.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:06.259622+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-02-23T00:42:07.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:06.260665+0000 mon.a (mon.0) 3591 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-122", "mode": "writeback"}]: dispatch 2024-02-23T00:42:07.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:07.116423+0000 mon.c (mon.2) 492 : audit [DBG] from='client.? 172.21.15.87:0/2148054030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:07.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:07.208649+0000 mon.a (mon.0) 3592 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:42:07.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:07.212773+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-122", "mode": "writeback"}]': finished 2024-02-23T00:42:07.571 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:07.212827+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-02-23T00:42:07.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:05.777507+0000 mgr.y (mgr.24437) 604 : cluster [DBG] pgmap v1073: 194 pgs: 2 unknown, 2 active+remapped+backfill_toofull, 190 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:07.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:06.259555+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]': finished 2024-02-23T00:42:07.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:06.259622+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-02-23T00:42:07.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:06.260665+0000 mon.a (mon.0) 3591 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-122", "mode": "writeback"}]: dispatch 2024-02-23T00:42:07.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:07.116423+0000 mon.c (mon.2) 492 : audit [DBG] from='client.? 172.21.15.87:0/2148054030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:07.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:07.208649+0000 mon.a (mon.0) 3592 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:42:07.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:07.212773+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-122", "mode": "writeback"}]': finished 2024-02-23T00:42:07.572 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:07.212827+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-02-23T00:42:07.573 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:07.671 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:07.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:05.777507+0000 mgr.y (mgr.24437) 604 : cluster [DBG] pgmap v1073: 194 pgs: 2 unknown, 2 active+remapped+backfill_toofull, 190 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:06.259555+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]': finished 2024-02-23T00:42:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:06.259622+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-02-23T00:42:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:06.260665+0000 mon.a (mon.0) 3591 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-122", "mode": "writeback"}]: dispatch 2024-02-23T00:42:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:07.116423+0000 mon.c (mon.2) 492 : audit [DBG] from='client.? 172.21.15.87:0/2148054030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:07.208649+0000 mon.a (mon.0) 3592 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:42:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:07.212773+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-122", "mode": "writeback"}]': finished 2024-02-23T00:42:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:07.212827+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-02-23T00:42:07.701 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:42:07.781 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:07.807 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:08.005 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:08.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:08.110 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:08.179 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:08.198 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:08.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:08.302 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:08.329 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:08.348 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:08.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:08.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:08.451 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:08.477 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:08.495 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:08.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:08.519 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:42:08.599 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:08.627 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:08.810 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:09.176 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:09.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:09.279 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:09.308 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:07.777932+0000 mgr.y (mgr.24437) 605 : cluster [DBG] pgmap v1076: 194 pgs: 2 unknown, 2 active+remapped+backfill_toofull, 190 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:09.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:08.224703+0000 mon.a (mon.0) 3595 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-02-23T00:42:09.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:07.777932+0000 mgr.y (mgr.24437) 605 : cluster [DBG] pgmap v1076: 194 pgs: 2 unknown, 2 active+remapped+backfill_toofull, 190 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:09.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:08.224703+0000 mon.a (mon.0) 3595 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-02-23T00:42:09.383 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:09.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:09.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:09.438 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:42:09.438 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:42:09.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:42:09.438 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:42:09.438 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.438 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:2079 63.3 63:c0773ebc:test-rados-api-smithi087-82091-26::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e234)", 2024-02-23T00:42:09.438 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:05.027503+0000", 2024-02-23T00:42:09.438 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 544.40667998100002, 2024-02-23T00:42:09.438 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019673782000000001, 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2079 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027503+0000", 2024-02-23T00:42:09.439 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.440 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.440 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.440 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027503+0000", 2024-02-23T00:42:09.440 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:09.440 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.441 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.441 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027501+0000", 2024-02-23T00:42:09.441 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0348e-05 2024-02-23T00:42:09.441 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.441 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027511+0000", 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3602e-05 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027525+0000", 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.7789999999999996e-06 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.442 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.027533+0000", 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0073932110000000002 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.034926+0000", 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014632299999999999 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.035072+0000", 2024-02-23T00:42:09.443 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000197312 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 3,4", 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.035269+0000", 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0040784150000000002 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.039348+0000", 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035992540000000001 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.444 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.042947+0000", 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041140780000000002 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.047061+0000", 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1672999999999998e-05 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:09.445 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.047113+0000", 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3707999999999994e-05 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:05.047177+0000", 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019673782000000001 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:5418 63.1b 63:d8ef31d3:test-rados-api-smithi087-82091-50::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e372)", 2024-02-23T00:42:09.446 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:35:23.068992+0000", 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 406.36519029800002, 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033546511000000001, 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 5418 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.447 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.068992+0000", 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.068992+0000", 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.068990+0000", 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0651e-05 2024-02-23T00:42:09.448 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069001+0000", 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4694e-05 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069015+0000", 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.1549999999999996e-06 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069025+0000", 2024-02-23T00:42:09.449 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4701999999999999e-05 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069079+0000", 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0494e-05 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.069090+0000", 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0035708100000000002 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.450 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.072661+0000", 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014544499999999999 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.072806+0000", 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018308800000000001 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,2", 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.072989+0000", 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0036468820000000002 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.451 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.076636+0000", 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024542562 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.101179+0000", 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00126581 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.452 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102444+0000", 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5225000000000001e-05 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102500+0000", 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9344999999999997e-05 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:35:23.102539+0000", 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.033546511000000001 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.453 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6302 63.17 63:e8b72b4a:test-rados-api-smithi087-82091-64::foo:head [write 0~8] snapc 1d=[1d,1c] ondisk+write+known_if_redirected e432)", 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:36:44.294999+0000", 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 325.13918359000002, 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023072202, 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:09.454 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6302 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.294999+0000", 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.455 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.294999+0000", 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999971 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.294996+0000", 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4647000000000001e-05 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.456 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295011+0000", 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1302e-05 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295032+0000", 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0713999999999999e-05 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.457 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295043+0000", 2024-02-23T00:42:09.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7822999999999998e-05 2024-02-23T00:42:09.458 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.458 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.458 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.458 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295101+0000", 2024-02-23T00:42:09.458 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4919999999999993e-06 2024-02-23T00:42:09.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:09.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.295109+0000", 2024-02-23T00:42:09.459 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0058836230000000002 2024-02-23T00:42:09.459 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.459 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.459 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.459 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.300993+0000", 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000110759 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.301104+0000", 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00020068000000000001 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.301305+0000", 2024-02-23T00:42:09.460 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00069117699999999996 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.301996+0000", 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00045108700000000003 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.302447+0000", 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.015506539999999999 2024-02-23T00:42:09.461 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.317953+0000", 2024-02-23T00:42:09.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7464000000000003e-05 2024-02-23T00:42:09.462 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.462 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.462 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:09.462 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.318021+0000", 2024-02-23T00:42:09.462 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0603000000000003e-05 2024-02-23T00:42:09.463 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.463 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.463 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.463 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.318071+0000", 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.023072202 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6304 63.17 63:e8b72b4a:test-rados-api-smithi087-82091-64::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e432)", 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:36:44.326711+0000", 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 325.107471653, 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024147567000000002, 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.464 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6304 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326711+0000", 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326711+0000", 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.465 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326710+0000", 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2559999999999996e-06 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326717+0000", 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3875e-05 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326731+0000", 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9949999999999999e-06 2024-02-23T00:42:09.466 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326738+0000", 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7929000000000001e-05 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326786+0000", 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2782999999999996e-05 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.326879+0000", 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0076338630000000003 2024-02-23T00:42:09.467 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.334513+0000", 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011122320000000001 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.335625+0000", 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0062604449999999999 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.341885+0000", 2024-02-23T00:42:09.468 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0088814819999999996 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.350767+0000", 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0455999999999999e-05 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.350817+0000", 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1442000000000001e-05 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.469 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.350859+0000", 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024147567000000002 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6305 63.17 63:e8b72b4a:test-rados-api-smithi087-82091-64::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e432)", 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:36:44.361680+0000", 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 325.07250317099999, 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026227607999999999, 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:42:09.470 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6305 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361680+0000", 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361680+0000", 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:42:09.471 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361678+0000", 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2040000000000003e-06 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361687+0000", 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4222999999999999e-05 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.472 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361701+0000", 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2899999999999997e-06 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361709+0000", 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7270000000000001e-05 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361746+0000", 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9739000000000002e-05 2024-02-23T00:42:09.473 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.361846+0000", 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.02429605 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.386142+0000", 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00122338 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387365+0000", 2024-02-23T00:42:09.474 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00033618999999999999 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387701+0000", 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000123817 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387825+0000", 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4799000000000003e-05 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:09.475 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387870+0000", 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7274000000000003e-05 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:36:44.387907+0000", 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026227607999999999 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.24413.0:676 3.c 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e509)", 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:48.490504+0000", 2024-02-23T00:42:09.476 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 200.943678361, 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.69719402699999999, 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.24413", 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.171:0/2453883515", 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 676 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490504+0000", 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.477 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490504+0000", 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490502+0000", 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0909999999999999e-06 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490511+0000", 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.396e-05 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.478 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490525+0000", 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3749999999999992e-06 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490535+0000", 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.6785000000000004e-05 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490611+0000", 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8079999999999999e-06 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.479 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:48.490621+0000", 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.69679959300000005 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:49.187421+0000", 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00013327599999999999 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:49.187554+0000", 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000144265 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.480 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:49.187698+0000", 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.69719402699999999 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37979.0:1 229.2s0 229:48558768:test-rados-api-smithi087-82091-95::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected e561)", 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:39:40.184380+0000", 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 149.24980270899999, 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0235315789999999, 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.481 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37979", 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/403995334", 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184380+0000", 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184380+0000", 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.482 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184379+0000", 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.8969999999999999e-06 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184384+0000", 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2439999999999993e-06 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184392+0000", 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8159999999999999e-06 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.483 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184398+0000", 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4269e-05 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184432+0000", 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.5000000000000002e-06 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:40.184435+0000", 2024-02-23T00:42:09.484 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.004191152 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:41.188626+0000", 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012520899999999999 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:41.188751+0000", 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.019160613999999999 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:39:41.207912+0000", 2024-02-23T00:42:09.485 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0235315789999999 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDScrubReserve(22.7 REQUEST e644)", 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:03.314375+0000", 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 66.119808118999998, 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044733610000000004, 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.486 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.314375+0000", 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.314375+0000", 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.314374+0000", 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0419999999999998e-06 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.487 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.314378+0000", 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.4841000000000001e-05 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.314412+0000", 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0074999999999999e-05 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.314422+0000", 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6569000000000006e-05 2024-02-23T00:42:09.488 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.314489+0000", 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5439999999999998e-06 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.314495+0000", 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0043251649999999997 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.318820+0000", 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1321000000000001e-05 2024-02-23T00:42:09.489 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.318831+0000", 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6952e-05 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:03.318848+0000", 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0044733610000000004 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.490 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:372 238.a 238:55fe7071:test-rados-api-smithi087-82091-109::foo:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e645)", 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:04.200332+0000", 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 65.233850884999995, 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078265230000000002, 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 372 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.200332+0000", 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.491 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.200332+0000", 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.200330+0000", 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9229999999999996e-06 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.200337+0000", 2024-02-23T00:42:09.492 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1379999999999999e-05 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.200348+0000", 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5369999999999997e-06 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.200357+0000", 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9106999999999997e-05 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.493 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.200396+0000", 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6099999999999997e-06 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.200402+0000", 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003604426 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.204006+0000", 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000147689 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:42:09.494 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.204154+0000", 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0031193900000000001 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.207273+0000", 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015003799999999999 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.207423+0000", 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00066905100000000004 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.495 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.208092+0000", 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.6141000000000003e-05 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:04.208158+0000", 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0078265230000000002 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDECSubOpWrite(234.7s1 650/597 ECSubWrite(tid=565, reqid=client.27567.0:474, at_version=650'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:09.221534+0000", 2024-02-23T00:42:09.496 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 60.212649255999999, 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023313320000000002, 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.221534+0000", 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.221534+0000", 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.497 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.221533+0000", 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0145000000000003e-05 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.221573+0000", 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1940999999999999e-05 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.221595+0000", 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2700000000000004e-06 2024-02-23T00:42:09.498 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.221604+0000", 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00069644300000000004 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.222300+0000", 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1192000000000001e-05 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.222311+0000", 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015396979999999999 2024-02-23T00:42:09.499 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.223851+0000", 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3954999999999999e-05 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.223865+0000", 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023313320000000002 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.500 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:476 239.11 239:8b8d2756:test-rados-api-smithi087-82091-111::bar:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:12.211493+0000", 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 57.222689899999999, 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0021674419999999999, 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 476 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.211493+0000", 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.501 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.211493+0000", 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.211491+0000", 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9390000000000003e-06 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.211500+0000", 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5758000000000001e-05 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.502 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.211516+0000", 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.815e-06 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.211523+0000", 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0139000000000002e-05 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.211573+0000", 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000197109 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.503 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.211770+0000", 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0010609409999999999 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.212831+0000", 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5499000000000004e-05 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.212896+0000", 2024-02-23T00:42:09.504 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000763936 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.213660+0000", 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0021674419999999999 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:477 239.11 239:8b8d2756:test-rados-api-smithi087-82091-111::bar:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected e653)", 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:12.213952+0000", 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 57.220230440999998, 2024-02-23T00:42:09.505 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015572520000000001, 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 477 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.213952+0000", 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.506 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.213952+0000", 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.213951+0000", 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2259999999999996e-06 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.213957+0000", 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9600000000000006e-06 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.507 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.213965+0000", 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.262e-06 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.213970+0000", 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.5651e-05 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.214025+0000", 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3759999999999999e-05 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.508 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.214089+0000", 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0014204140000000001 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:12.215510+0000", 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0015572520000000001 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.509 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:597 239.11 239:88000000::::head [pgnls start_epoch 655] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e655)", 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:14.214696+0000", 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 55.21948647, 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020692649999999998, 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 597 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.214696+0000", 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.510 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.214696+0000", 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999971 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.214693+0000", 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7499999999999998e-05 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.214711+0000", 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0547999999999999e-05 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.511 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.214751+0000", 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3870000000000001e-06 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.214761+0000", 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0723999999999997e-05 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.214821+0000", 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7889999999999997e-06 2024-02-23T00:42:09.512 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.214828+0000", 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0017628209999999999 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.216591+0000", 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0164000000000002e-05 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.216651+0000", 2024-02-23T00:42:09.513 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000114419 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:14.216766+0000", 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0020692649999999998 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDECSubOpWrite(234.7s1 667/597 ECSubWrite(tid=575, reqid=client.27567.0:796, at_version=0'0, trim_to=0'0, roll_forward_to=667'6))", 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:26.231195+0000", 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 43.202987604999997, 2024-02-23T00:42:09.514 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025470100000000002, 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "reached pg", 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.231195+0000", 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.231195+0000", 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.515 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.231194+0000", 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7382000000000001e-05 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.231212+0000", 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5468000000000002e-05 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.231227+0000", 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.5800000000000003e-06 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.516 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.231235+0000", 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0081999999999998e-05 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.231295+0000", 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9019999999999994e-06 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.231305+0000", 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0024150920000000002 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.517 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.233720+0000", 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2540000000000001e-05 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:26.233742+0000", 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025470100000000002 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:800 241.7 241:e276024f:test-rados-api-smithi087-82091-115::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e670)", 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:29.246242+0000", 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 40.187940544, 2024-02-23T00:42:09.518 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010421501999999999, 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 800 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.246242+0000", 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.519 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.246242+0000", 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.246240+0000", 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1409e-05 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.246252+0000", 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.6986999999999998e-05 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.520 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.246269+0000", 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0860000000000008e-06 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.246278+0000", 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7636999999999997e-05 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.246335+0000", 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.3089999999999996e-06 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:09.521 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.246344+0000", 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028382030000000001 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.249182+0000", 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014351000000000001 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.249325+0000", 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.004383484 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.253709+0000", 2024-02-23T00:42:09.522 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.8403999999999999e-05 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.253767+0000", 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00048479799999999999 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.254252+0000", 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6152999999999997e-05 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.254298+0000", 2024-02-23T00:42:09.523 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002365603 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.256664+0000", 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.010421501999999999 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:804 241.7 241:e276024f:test-rados-api-smithi087-82091-115::foo:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e670)", 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:29.264736+0000", 2024-02-23T00:42:09.524 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 40.169446579000002, 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0049520509999999998, 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 804 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.264736+0000", 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.525 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.264736+0000", 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.264735+0000", 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2199999999999992e-06 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.264743+0000", 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2996999999999999e-05 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.526 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.264756+0000", 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1430000000000003e-06 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.264762+0000", 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9437999999999997e-05 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.264801+0000", 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014180599999999999 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.527 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for blocked object", 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.264943+0000", 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0012906840000000001 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.266234+0000", 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5104000000000006e-05 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for rw locks", 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.266299+0000", 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038936299999999999 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.528 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.266688+0000", 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3608999999999994e-05 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.266752+0000", 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002936311 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:29.269688+0000", 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0049520509999999998 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.529 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:864 241.1e 241:78000000::::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:32.219683+0000", 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 37.214499693, 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0029910169999999999, 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 864 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.530 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.219683+0000", 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.219683+0000", 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999967 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.219680+0000", 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5156e-05 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.531 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.219695+0000", 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.2498999999999999e-05 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.219718+0000", 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0818e-05 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.219728+0000", 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.1917999999999998e-05 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.532 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.219800+0000", 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.9780000000000005e-06 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.219808+0000", 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002745763 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.222554+0000", 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3762e-05 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.533 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.222588+0000", 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6335999999999995e-05 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:32.222674+0000", 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0029910169999999999 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.14397.0:2106 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94532396648704 gen 3] snapc 0=[] ondisk+write+known_if_redirected e681)", 2024-02-23T00:42:09.534 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:40.499803+0000", 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 28.934379336999999, 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013328330000000001, 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.14397", 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 2106 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:40.499803+0000", 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.535 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:40.499803+0000", 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:40.499801+0000", 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3380000000000001e-05 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:40.499814+0000", 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.72e-05 2024-02-23T00:42:09.536 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:40.499842+0000", 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7436e-05 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:40.499859+0000", 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000118885 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:40.499978+0000", 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000114738 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.537 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:40.500093+0000", 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001043581 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:40.501136+0000", 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013328330000000001 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:1232 243.4 243:20000000::::head [pgnls start_epoch 691] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e691)", 2024-02-23T00:42:09.538 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:53.316208+0000", 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.117974422, 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025525439999999999, 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1232 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.316208+0000", 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.539 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.316208+0000", 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.316206+0000", 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2367e-05 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.316218+0000", 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7829000000000002e-05 2024-02-23T00:42:09.540 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.316236+0000", 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.6120999999999999e-05 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.316262+0000", 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0569999999999999e-05 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.316313+0000", 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4510000000000002e-06 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.541 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.316317+0000", 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0023201960000000001 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.318638+0000", 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2499000000000001e-05 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.318680+0000", 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.0735999999999994e-05 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.542 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:53.318761+0000", 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0025525439999999999 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:1534 245.14 245:28000000:test-rados-api-smithi087-82091-123:::head [pgnls start_epoch 707] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e707)", 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:42:09.240671+0000", 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 0.19351196900000001, 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002175288, 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:09.543 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1534 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.240671+0000", 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.240671+0000", 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.544 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.240669+0000", 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3080000000000004e-06 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.240676+0000", 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.9539999999999999e-06 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.240686+0000", 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0879999999999997e-06 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.545 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.240693+0000", 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.1338999999999998e-05 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.240734+0000", 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.6529999999999999e-06 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.240740+0000", 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0019903579999999998 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.546 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.242730+0000", 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3390999999999999e-05 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.242763+0000", 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2589000000000005e-05 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.242846+0000", 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002175288 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.547 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.548 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.548 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:09.548 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:09.548 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:42:09.548 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:09.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:07.777932+0000 mgr.y (mgr.24437) 605 : cluster [DBG] pgmap v1076: 194 pgs: 2 unknown, 2 active+remapped+backfill_toofull, 190 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:09.549 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:08.224703+0000 mon.a (mon.0) 3595 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-02-23T00:42:09.577 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:09.700 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:09.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:09.803 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:09.831 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:10.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:42:10.158 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-23T00:42:10.180 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:10.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:10.283 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:10.314 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:10.336 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:10.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:10.441 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:09.231027+0000 mon.a (mon.0) 3596 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-02-23T00:42:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:09.264296+0000 mon.a (mon.0) 3597 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:09.656846+0000 mon.a (mon.0) 3598 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:10.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:10.068443+0000 mon.a (mon.0) 3599 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:42:10.469 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:10.471 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:42:10.489 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:10.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:09.231027+0000 mon.a (mon.0) 3596 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-02-23T00:42:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:09.264296+0000 mon.a (mon.0) 3597 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:09.656846+0000 mon.a (mon.0) 3598 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:10.068443+0000 mon.a (mon.0) 3599 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:42:10.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:09.231027+0000 mon.a (mon.0) 3596 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-02-23T00:42:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:09.264296+0000 mon.a (mon.0) 3597 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:09.656846+0000 mon.a (mon.0) 3598 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:10.068443+0000 mon.a (mon.0) 3599 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:42:10.592 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:10.619 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:10.817 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:10.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:10.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:10.856 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-23T00:42:10.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:42:10.856 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-23T00:42:10.857 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-23T00:42:10.922 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:10.949 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:11.097 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:11.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:11.201 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:11.231 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:11.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.332+0000 7f698423e700 1 -- 172.21.15.87:0/2510385264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f697c0fd510 msgr2=0x7f697c0f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:11.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.332+0000 7f698423e700 1 --2- 172.21.15.87:0/2510385264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f697c0fd510 0x7f697c0f6d40 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 rx=0x7f696c004870 tx=0x7f696c009cb0).stop 2024-02-23T00:42:11.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.332+0000 7f698423e700 1 -- 172.21.15.87:0/2510385264 shutdown_connections 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.332+0000 7f698423e700 1 --2- 172.21.15.87:0/2510385264 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f697c0f7400 0x7f697c103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.332+0000 7f698423e700 1 --2- 172.21.15.87:0/2510385264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0fcbb0 0x7f697c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.332+0000 7f698423e700 1 --2- 172.21.15.87:0/2510385264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f697c0fd510 0x7f697c0f6d40 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.332+0000 7f698423e700 1 -- 172.21.15.87:0/2510385264 >> 172.21.15.87:0/2510385264 conn(0x7f697c0f2820 msgr2=0x7f697c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.332+0000 7f698423e700 1 -- 172.21.15.87:0/2510385264 shutdown_connections 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.332+0000 7f698423e700 1 -- 172.21.15.87:0/2510385264 wait complete. 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 librados: starting msgr at 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 librados: starting objecter 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 Processor -- start 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 -- start start 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 librados: setting wanted keys 2024-02-23T00:42:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 librados: calling monclient init 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0f7400 0x7f697c10d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f697c0fcbb0 0x7f697c10d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f697c0fd510 0x7f697c107220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c107710 con 0x7f697c0fd510 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697c107880 con 0x7f697c0f7400 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f698423e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697c107a40 con 0x7f697c0fcbb0 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f69817d9700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f697c0fcbb0 0x7f697c10d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f69827db700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f697c0fd510 0x7f697c107220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f69817d9700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f697c0fcbb0 0x7f697c10d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60850/0 (socket says 172.21.15.87:60850) 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f69817d9700 1 -- 172.21.15.87:0/3564627515 learned_addr learned my addr 172.21.15.87:0/3564627515 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.333+0000 7f6981fda700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0f7400 0x7f697c10d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f69817d9700 1 -- 172.21.15.87:0/3564627515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0f7400 msgr2=0x7f697c10d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f69817d9700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0f7400 0x7f697c10d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:11.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f69817d9700 1 -- 172.21.15.87:0/3564627515 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f697c0fd510 msgr2=0x7f697c107220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f69817d9700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f697c0fd510 0x7f697c107220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f69817d9700 1 -- 172.21.15.87:0/3564627515 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69740097c0 con 0x7f697c0fcbb0 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f6981fda700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0f7400 0x7f697c10d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f69827db700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f697c0fd510 0x7f697c107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f69817d9700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f697c0fcbb0 0x7f697c10d570 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f696c009eb0 tx=0x7f696c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f6972ffd700 10 client.?.objecter ms_handle_connect 0x7f697c0fcbb0 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f6972ffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f6972ffd700 1 -- 172.21.15.87:0/3564627515 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c00e070 con 0x7f697c0fcbb0 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f6972ffd700 1 -- 172.21.15.87:0/3564627515 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f696c013da0 con 0x7f697c0fcbb0 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f6972ffd700 1 -- 172.21.15.87:0/3564627515 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c019070 con 0x7f697c0fcbb0 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c005040 con 0x7f697c0fcbb0 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f698423e700 10 client.38228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c107ee0 con 0x7f697c0fcbb0 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.334+0000 7f698423e700 1 librados: init done 2024-02-23T00:42:11.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.335+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697c1080d0 con 0x7f697c0fcbb0 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.335+0000 7f6972ffd700 1 -- 172.21.15.87:0/3564627515 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f696c0090d0 con 0x7f697c0fcbb0 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.335+0000 7f6972ffd700 1 --2- 172.21.15.87:0/3564627515 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6968062c90 0x7f6968065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.336+0000 7f6972ffd700 1 -- 172.21.15.87:0/3564627515 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(709..709 src has 173..709) v4 ==== 8510+0+0 (secure 0 0 0) 0x7f696c07dfd0 con 0x7f697c0fcbb0 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.336+0000 7f6972ffd700 10 client.38228.objecter ms_dispatch 0x7f697c001550 osd_map(709..709 src has 173..709) v4 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.336+0000 7f6972ffd700 3 client.38228.objecter handle_osd_map got epochs [709,709] > 0 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.336+0000 7f6972ffd700 3 client.38228.objecter handle_osd_map decoding full epoch 709 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.336+0000 7f6981fda700 1 --2- 172.21.15.87:0/3564627515 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6968062c90 0x7f6968065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.336+0000 7f6972ffd700 20 client.38228.objecter dump_active .. 0 homeless 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.336+0000 7f6981fda700 1 --2- 172.21.15.87:0/3564627515 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6968062c90 0x7f6968065150 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7f6978007fd0 tx=0x7f6978009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.336+0000 7f6972ffd700 10 client.38228.objecter ms_handle_connect 0x7f6968062c90 2024-02-23T00:42:11.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.340+0000 7f6972ffd700 1 -- 172.21.15.87:0/3564627515 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f696c0acf60 con 0x7f697c0fcbb0 2024-02-23T00:42:11.384 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:11.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:11.423 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:11.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:42:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:09.778370+0000 mgr.y (mgr.24437) 606 : cluster [DBG] pgmap v1079: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:11.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:10.232383+0000 mon.a (mon.0) 3600 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:10.232448+0000 mon.a (mon.0) 3601 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-02-23T00:42:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:10.246991+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]: dispatch 2024-02-23T00:42:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:10.861797+0000 osd.5 (osd.5) 267 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:10.862103+0000 osd.5 (osd.5) 268 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:10.905581+0000 mon.a (mon.0) 3603 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:42:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:10.906264+0000 mon.a (mon.0) 3604 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:42:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:10.910710+0000 mon.a (mon.0) 3605 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:11.487 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:11.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:09.778370+0000 mgr.y (mgr.24437) 606 : cluster [DBG] pgmap v1079: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:11.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:10.232383+0000 mon.a (mon.0) 3600 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:11.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:10.232448+0000 mon.a (mon.0) 3601 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-02-23T00:42:11.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:10.246991+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]: dispatch 2024-02-23T00:42:11.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:10.861797+0000 osd.5 (osd.5) 267 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:11.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:10.862103+0000 osd.5 (osd.5) 268 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:11.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:10.905581+0000 mon.a (mon.0) 3603 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:42:11.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:10.906264+0000 mon.a (mon.0) 3604 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:42:11.514 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:10.910710+0000 mon.a (mon.0) 3605 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:11.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:09.778370+0000 mgr.y (mgr.24437) 606 : cluster [DBG] pgmap v1079: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7/732 objects misplaced (0.956%) 2024-02-23T00:42:11.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:10.232383+0000 mon.a (mon.0) 3600 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:11.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:10.232448+0000 mon.a (mon.0) 3601 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-02-23T00:42:11.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:10.246991+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]: dispatch 2024-02-23T00:42:11.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:10.861797+0000 osd.5 (osd.5) 267 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:11.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:10.862103+0000 osd.5 (osd.5) 268 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:11.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:10.905581+0000 mon.a (mon.0) 3603 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:42:11.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:10.906264+0000 mon.a (mon.0) 3604 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:42:11.515 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:10.910710+0000 mon.a (mon.0) 3605 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:11.516 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:11.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:11.555+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f697c106120 con 0x7f697c0fcbb0 2024-02-23T00:42:11.667 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:11.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:11.771 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:11.806 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:11.829 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:11.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:11.933 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:11.962 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:12.060 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:12.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:12.164 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:12.193 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:12.213 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:12.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:12.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:12.253 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.248+0000 7f6972ffd700 1 -- 172.21.15.87:0/3564627515 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v710) v1 ==== 91+0+0 (secure 0 0 0) 0x7f696c023030 con 0x7f697c0fcbb0 2024-02-23T00:42:12.256 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.253+0000 7f698423e700 10 librados: watch_flush enter 2024-02-23T00:42:12.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.254+0000 7f698423e700 10 librados: watch_flush exit 2024-02-23T00:42:12.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.254+0000 7f698423e700 20 client.38228.objecter shutdown clearing up homeless session... 2024-02-23T00:42:12.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.254+0000 7f698423e700 10 client.38228.objecter successfully canceled tick 2024-02-23T00:42:12.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.254+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6968062c90 msgr2=0x7f6968065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:12.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.254+0000 7f698423e700 1 --2- 172.21.15.87:0/3564627515 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6968062c90 0x7f6968065150 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 rx=0x7f6978007fd0 tx=0x7f6978009040).stop 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.254+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f697c0fcbb0 msgr2=0x7f697c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.254+0000 7f698423e700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f697c0fcbb0 0x7f697c10d570 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f696c009eb0 tx=0x7f696c00b040).stop 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.255+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 shutdown_connections 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.255+0000 7f698423e700 1 --2- 172.21.15.87:0/3564627515 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6968062c90 0x7f6968065150 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.255+0000 7f698423e700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f697c0fd510 0x7f697c107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.255+0000 7f698423e700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f697c0f7400 0x7f697c10d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.255+0000 7f698423e700 1 --2- 172.21.15.87:0/3564627515 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f697c0fcbb0 0x7f697c10d570 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.255+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 >> 172.21.15.87:0/3564627515 conn(0x7f697c0f2820 msgr2=0x7f697c0f43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.255+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 shutdown_connections 2024-02-23T00:42:12.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.255+0000 7f698423e700 1 -- 172.21.15.87:0/3564627515 wait complete. 2024-02-23T00:42:12.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:12.256+0000 7f698423e700 1 librados: shutdown 2024-02-23T00:42:12.262 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:42:12.293 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:12.317 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:12.342 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:42:12.342 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:42:12.342 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:42:12.342 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:42:12.346 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:12.372 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:11.214001+0000 mgr.y (mgr.24437) 607 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:12.373 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:11.243179+0000 mon.a (mon.0) 3606 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]': finished 2024-02-23T00:42:12.373 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:11.243234+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-02-23T00:42:12.373 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:11.557348+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.87:0/3564627515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:42:12.373 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:11.557648+0000 mon.a (mon.0) 3608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:42:12.547 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:12.549 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:12] "GET /metrics HTTP/1.1" 200 190219 "" "Prometheus/2.33.4" 2024-02-23T00:42:12.550 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:11.214001+0000 mgr.y (mgr.24437) 607 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:12.550 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:11.243179+0000 mon.a (mon.0) 3606 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]': finished 2024-02-23T00:42:12.550 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:11.243234+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-02-23T00:42:12.550 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:11.557348+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.87:0/3564627515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:42:12.550 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:11.557648+0000 mon.a (mon.0) 3608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:42:12.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:11.214001+0000 mgr.y (mgr.24437) 607 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:12.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:11.243179+0000 mon.a (mon.0) 3606 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-122"}]': finished 2024-02-23T00:42:12.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:11.243234+0000 mon.a (mon.0) 3607 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-02-23T00:42:12.550 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:11.557348+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.87:0/3564627515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:42:12.551 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:11.557648+0000 mon.a (mon.0) 3608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:42:12.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:12.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:12.652 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:12.680 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:13.007 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-23T00:42:13.129 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:13.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:13.233 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:13.266 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:13.275 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:42:13.338 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:13.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:11.778760+0000 mgr.y (mgr.24437) 608 : cluster [DBG] pgmap v1082: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 7/750 objects misplaced (0.933%) 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:12.241141+0000 mon.a (mon.0) 3609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:12.249484+0000 mon.a (mon.0) 3610 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:12.249548+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:12.275210+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:12.431990+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:12.433154+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:12.889926+0000 osd.4 (osd.4) 317 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:12.891048+0000 osd.4 (osd.4) 318 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.005121+0000 osd.5 (osd.5) 269 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:42:13.389 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.006828+0000 osd.5 (osd.5) 270 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:11.778760+0000 mgr.y (mgr.24437) 608 : cluster [DBG] pgmap v1082: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 7/750 objects misplaced (0.933%) 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:12.241141+0000 mon.a (mon.0) 3609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:12.249484+0000 mon.a (mon.0) 3610 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:12.249548+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:12.275210+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:12.431990+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:12.433154+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:12.889926+0000 osd.4 (osd.4) 317 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:12.891048+0000 osd.4 (osd.4) 318 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.005121+0000 osd.5 (osd.5) 269 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:42:13.390 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.006828+0000 osd.5 (osd.5) 270 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:42:13.442 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:13.469 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:11.778760+0000 mgr.y (mgr.24437) 608 : cluster [DBG] pgmap v1082: 194 pgs: 2 active+remapped+backfill_toofull, 192 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 7/750 objects misplaced (0.933%) 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:12.241141+0000 mon.a (mon.0) 3609 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:12.249484+0000 mon.a (mon.0) 3610 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:12.249548+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:12.275210+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:12.431990+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:12.433154+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:12.889926+0000 osd.4 (osd.4) 317 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:12.891048+0000 osd.4 (osd.4) 318 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.005121+0000 osd.5 (osd.5) 269 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:42:13.520 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.006828+0000 osd.5 (osd.5) 270 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:42:13.522 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:13.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:13.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:13.626 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:13.653 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:13.784 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:42:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:42:14.102 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:14.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:14.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:14.206 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:14.235 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:14.289 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:14.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:14.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:14.399 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:14.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:12.786609+0000 osd.0 (osd.0) 13 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:42:14.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:12.789801+0000 osd.0 (osd.0) 14 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:42:14.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:13.258037+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:14.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.258104+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-02-23T00:42:14.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:13.258654+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122"}]: dispatch 2024-02-23T00:42:14.424 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.430543+0000 osd.2 (osd.2) 253 : cluster [DBG] 234.5 scrub starts 2024-02-23T00:42:14.425 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.431787+0000 osd.2 (osd.2) 254 : cluster [DBG] 234.5 scrub ok 2024-02-23T00:42:14.425 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.608294+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:42:14.425 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.611190+0000 osd.6 (osd.6) 285 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:42:14.425 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.973193+0000 osd.1 (osd.1) 322 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:14.425 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.973474+0000 osd.1 (osd.1) 323 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:14.426 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:12.786609+0000 osd.0 (osd.0) 13 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:12.789801+0000 osd.0 (osd.0) 14 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:13.258037+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.258104+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:13.258654+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122"}]: dispatch 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.430543+0000 osd.2 (osd.2) 253 : cluster [DBG] 234.5 scrub starts 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.431787+0000 osd.2 (osd.2) 254 : cluster [DBG] 234.5 scrub ok 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.608294+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.611190+0000 osd.6 (osd.6) 285 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:42:14.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.973193+0000 osd.1 (osd.1) 322 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:14.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.973474+0000 osd.1 (osd.1) 323 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:14.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:12.786609+0000 osd.0 (osd.0) 13 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:42:14.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:12.789801+0000 osd.0 (osd.0) 14 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:42:14.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:13.258037+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:14.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.258104+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-02-23T00:42:14.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:13.258654+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122"}]: dispatch 2024-02-23T00:42:14.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.430543+0000 osd.2 (osd.2) 253 : cluster [DBG] 234.5 scrub starts 2024-02-23T00:42:14.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.431787+0000 osd.2 (osd.2) 254 : cluster [DBG] 234.5 scrub ok 2024-02-23T00:42:14.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.608294+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:42:14.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.611190+0000 osd.6 (osd.6) 285 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:42:14.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.973193+0000 osd.1 (osd.1) 322 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:14.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.973474+0000 osd.1 (osd.1) 323 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:14.624 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:14.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:14.727 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:14.755 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:14.853 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:14.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:14.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.921+0000 7f988a746700 1 -- 172.21.15.87:0/3700883974 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f98840fdbb0 msgr2=0x7f98840fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.921+0000 7f988a746700 1 --2- 172.21.15.87:0/3700883974 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f98840fdbb0 0x7f98840fdf90 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 rx=0x7f9878004870 tx=0x7f9878009cb0).stop 2024-02-23T00:42:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.921+0000 7f988a746700 1 -- 172.21.15.87:0/3700883974 shutdown_connections 2024-02-23T00:42:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.921+0000 7f988a746700 1 --2- 172.21.15.87:0/3700883974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98840f8420 0x7f9884104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.921+0000 7f988a746700 1 --2- 172.21.15.87:0/3700883974 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f98840fdbb0 0x7f98840fdf90 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:14.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.921+0000 7f988a746700 1 --2- 172.21.15.87:0/3700883974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98840fe510 0x7f98840f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.921+0000 7f988a746700 1 -- 172.21.15.87:0/3700883974 >> 172.21.15.87:0/3700883974 conn(0x7f98840f3800 msgr2=0x7f98840f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 -- 172.21.15.87:0/3700883974 shutdown_connections 2024-02-23T00:42:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 -- 172.21.15.87:0/3700883974 wait complete. 2024-02-23T00:42:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 librados: starting msgr at 2024-02-23T00:42:14.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 librados: starting objecter 2024-02-23T00:42:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 Processor -- start 2024-02-23T00:42:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 -- start start 2024-02-23T00:42:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 librados: setting wanted keys 2024-02-23T00:42:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 librados: calling monclient init 2024-02-23T00:42:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98840f8420 0x7f988406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f98840fdbb0 0x7f988406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:14.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98840fe510 0x7f988406bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9884116580 con 0x7f98840f8420 2024-02-23T00:42:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98841166f0 con 0x7f98840fe510 2024-02-23T00:42:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.922+0000 7f988a746700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98841168b0 con 0x7f98840fdbb0 2024-02-23T00:42:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9883fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98840f8420 0x7f988406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f98837fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f98840fdbb0 0x7f988406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:14.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9883fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98840f8420 0x7f988406b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53562/0 (socket says 172.21.15.87:53562) 2024-02-23T00:42:14.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9883fff700 1 -- 172.21.15.87:0/1150755745 learned_addr learned my addr 172.21.15.87:0/1150755745 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:14.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9888ce3700 1 --2- 172.21.15.87:0/1150755745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98840fe510 0x7f988406bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:14.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9883fff700 1 -- 172.21.15.87:0/1150755745 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f98840fdbb0 msgr2=0x7f988406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:14.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9883fff700 1 --2- 172.21.15.87:0/1150755745 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f98840fdbb0 0x7f988406b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:14.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9883fff700 1 -- 172.21.15.87:0/1150755745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98840fe510 msgr2=0x7f988406bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:14.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9883fff700 1 --2- 172.21.15.87:0/1150755745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98840fe510 0x7f988406bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:14.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9883fff700 1 -- 172.21.15.87:0/1150755745 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c0097c0 con 0x7f98840f8420 2024-02-23T00:42:14.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f9883fff700 1 --2- 172.21.15.87:0/1150755745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98840f8420 0x7f988406b0d0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7f9878009eb0 tx=0x7f987800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:14.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f98817fa700 10 client.?.objecter ms_handle_connect 0x7f98840f8420 2024-02-23T00:42:14.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f98817fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:14.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f98817fa700 1 -- 172.21.15.87:0/1150755745 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987800e070 con 0x7f98840f8420 2024-02-23T00:42:14.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f98817fa700 1 -- 172.21.15.87:0/1150755745 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9878013da0 con 0x7f98840f8420 2024-02-23T00:42:14.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9878005040 con 0x7f98840f8420 2024-02-23T00:42:14.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f98817fa700 1 -- 172.21.15.87:0/1150755745 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878019070 con 0x7f98840f8420 2024-02-23T00:42:14.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f988a746700 10 client.27759.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:14.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9884116e00 con 0x7f98840f8420 2024-02-23T00:42:14.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.923+0000 7f988a746700 1 librados: init done 2024-02-23T00:42:14.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.924+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988405ee80 con 0x7f98840f8420 2024-02-23T00:42:14.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98817fa700 1 -- 172.21.15.87:0/1150755745 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f987801e420 con 0x7f98840f8420 2024-02-23T00:42:14.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98817fa700 1 --2- 172.21.15.87:0/1150755745 >> v2:172.21.15.87:6800/1231597463 conn(0x7f986c062c90 0x7f986c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:14.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98817fa700 1 -- 172.21.15.87:0/1150755745 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(712..712 src has 173..712) v4 ==== 8558+0+0 (secure 0 0 0) 0x7f987807dec0 con 0x7f98840f8420 2024-02-23T00:42:14.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98817fa700 10 client.27759.objecter ms_dispatch 0x7f9884001550 osd_map(712..712 src has 173..712) v4 2024-02-23T00:42:14.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98817fa700 3 client.27759.objecter handle_osd_map got epochs [712,712] > 0 2024-02-23T00:42:14.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98817fa700 3 client.27759.objecter handle_osd_map decoding full epoch 712 2024-02-23T00:42:14.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98837fe700 1 --2- 172.21.15.87:0/1150755745 >> v2:172.21.15.87:6800/1231597463 conn(0x7f986c062c90 0x7f986c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:14.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98817fa700 20 client.27759.objecter dump_active .. 0 homeless 2024-02-23T00:42:14.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98817fa700 10 client.27759.objecter ms_handle_connect 0x7f986c062c90 2024-02-23T00:42:14.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.925+0000 7f98837fe700 1 --2- 172.21.15.87:0/1150755745 >> v2:172.21.15.87:6800/1231597463 conn(0x7f986c062c90 0x7f986c065150 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f988406c870 tx=0x7f987400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:14.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:14.930+0000 7f98817fa700 1 -- 172.21.15.87:0/1150755745 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9878051280 con 0x7f98840f8420 2024-02-23T00:42:14.957 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:14.988 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:15.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.100+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f988405fa60 con 0x7f98840f8420 2024-02-23T00:42:15.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.102+0000 7f98817fa700 1 -- 172.21.15.87:0/1150755745 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v712) v1 ==== 81+0+49150 (secure 0 0 0) 0x7f9878023030 con 0x7f98840f8420 2024-02-23T00:42:15.105 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:42:15.105 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:42:15.105 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 712, 2024-02-23T00:42:15.105 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:42:15.105 INFO:teuthology.orchestra.run.smithi087.stdout: "created": "2024-02-23T00:21:28.861318+0000", 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "modified": "2024-02-23T00:42:14.252699+0000", 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "last_up_change": "2024-02-23T00:41:18.209057+0000", 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "last_in_change": "2024-02-23T00:41:36.208573+0000", 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_num": 5804032, 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_set": [ 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "nodeep-scrub", 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "pglog_hardlimit", 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "purged_snapdirs", 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "recovery_deletes", 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "sortbitwise" 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_version": 114, 2024-02-23T00:42:15.106 INFO:teuthology.orchestra.run.smithi087.stdout: "full_ratio": 0.94999998807907104, 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_snapshot": "", 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_max": 245, 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd": 8, 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "require_min_compat_client": "luminous", 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "min_compat_client": "luminous", 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "require_osd_release": "pacific", 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "pools": [ 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "device_health_metrics", 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:25:09.208309+0000", 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:42:15.107 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 1, 2024-02-23T00:42:15.108 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 1, 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 1, 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 1, 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 1, 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "68", 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:42:15.109 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:42:15.110 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:42:15.111 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_max": 32, 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_min": 1 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_devicehealth": {} 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.112 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 2, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": ".rgw.root", 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.562186+0000", 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:42:15.113 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:42:15.114 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "174", 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:42:15.115 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.116 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.117 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 3, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "iscsi", 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.989206+0000", 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:42:15.118 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 13, 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 13, 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 13, 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 13, 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 13, 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "137", 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:42:15.119 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "135", 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:42:15.120 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:42:15.121 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd": {} 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 4, 2024-02-23T00:42:15.122 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.log", 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:17.282440+0000", 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:42:15.123 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "4.16", 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 371, 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 370, 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 370, 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "63'21", 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "62'12" 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "532", 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:42:15.124 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "371", 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "481", 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:42:15.125 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:42:15.126 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 5, 2024-02-23T00:42:15.127 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.control", 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:19.262164+0000", 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:42:15.128 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "5.20", 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 641, 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 640, 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 640, 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "562'12", 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "330'9" 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "641", 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "641", 2024-02-23T00:42:15.129 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "639", 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:42:15.130 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:42:15.131 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 6, 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.meta", 2024-02-23T00:42:15.132 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:21.319951+0000", 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 12, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 12, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 12, 2024-02-23T00:42:15.133 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 12, 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 12, 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "6.c", 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 665, 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 664, 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 664, 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "78'2", 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "665", 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "665", 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "663", 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:42:15.134 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:42:15.135 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:42:15.136 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_bias": 4 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 22, 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:46.071806+0000", 2024-02-23T00:42:15.137 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:42:15.138 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:42:15.139 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:42:15.140 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:42:15.141 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 234, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-101", 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:40:15.435168+0000", 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 3, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:42:15.142 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 1, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 8, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 8, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 8, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 8, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 8, 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:42:15.143 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "712", 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "711", 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "711", 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "711", 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 10, 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 710, 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:42:15.144 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:42:15.145 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi087-82091-101", 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 8192, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:42:15.146 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 245, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-122", 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:42:02.493920+0000", 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8201, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:42:15.147 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:42:15.148 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "712", 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "711", 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "711", 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "711", 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 10, 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 710, 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:42:15.149 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 0, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 0, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 0, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.150 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:42:15.151 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7", 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 8, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 424, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 659, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 707, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 425, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:42:15.152 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6802", 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.153 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6804", 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6803", 2024-02-23T00:42:15.154 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6801/484081831", 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6802/484081831", 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6804/484081831", 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6803/484081831", 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:42:15.155 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "85525ce8-38d5-4840-8057-547cb176c859", 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 13, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 706, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.156 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6805", 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6806", 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.157 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6808", 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6807", 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.158 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6805/685354171", 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6806/685354171", 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6808/685354171", 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6807/685354171", 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa", 2024-02-23T00:42:15.159 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 18, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 701, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.160 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6809", 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6810", 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:42:15.161 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6812", 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6811", 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.162 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6809/3605394228", 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6810/3605394228", 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6812/3605394228", 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6811/3605394228", 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a", 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:42:15.163 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 26, 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 701, 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6813", 2024-02-23T00:42:15.164 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6814", 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.165 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6816", 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6815", 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.166 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6813/128059790", 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6814/128059790", 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6816/128059790", 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6815/128059790", 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c", 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.5775604248046875, 2024-02-23T00:42:15.167 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 31, 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 701, 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6800", 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.168 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6801", 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6803", 2024-02-23T00:42:15.169 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6802", 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.170 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6800/4175844757", 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6801/4175844757", 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4175844757", 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4175844757", 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d", 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:42:15.171 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.8517608642578125, 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 38, 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 701, 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6804", 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:42:15.172 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6805", 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:42:15.173 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6807", 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6806", 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.174 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6804/2550013795", 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6805/2550013795", 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2550013795", 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2550013795", 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01", 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:42:15.175 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 44, 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 701, 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6808", 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:42:15.176 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6809", 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.177 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6811", 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6810", 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.178 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6808/2509408498", 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6809/2509408498", 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6811/2509408498", 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6810/2509408498", 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939", 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:42:15.179 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 51, 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 701, 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6812", 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.180 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6813", 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.181 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6815", 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6814", 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.182 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6812/373787000", 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6813/373787000", 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6815/373787000", 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6814/373787000", 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_xinfo": [ 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:42:15.183 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:41:17.571824+0000", 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 424 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:13.973488+0000", 2024-02-23T00:42:15.184 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:41:43.491272+0000", 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:42:15.185 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:42:15.186 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:42:15.186 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:42:15.186 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:41:04.154188+0000", 2024-02-23T00:42:15.186 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:42:15.186 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.186 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.186 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:41:09.190052+0000", 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:42:15.187 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:10.862105+0000", 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:41:01.921983+0000", 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:42:15.188 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:41:10.945471+0000", 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap": [], 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap_items": [ 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.189 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.18", 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 1 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.1c", 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 4 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.190 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.13", 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "5.19", 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:42:15.191 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 6 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "22.1a", 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 5 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.192 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_temp": [ 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "5.3", 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: 3, 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: 6, 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: 5 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_temp": [], 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: "blocklist": { 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1121018082": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/2138166229": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:42:15.193 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/27794119": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/1675126043": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/3767016525": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/969918521": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/795095363": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2273151327": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1685406414": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2093733106": "2024-02-24T00:21:58.144574+0000" 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "range_blocklist": {}, 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profiles": { 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "default": { 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:42:15.194 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van" 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "testprofile-test-rados-api-smithi087-82091-101": { 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-device-class": "", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-root": "default", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van", 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: "w": "8" 2024-02-23T00:42:15.195 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps_queue": [ 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 7, 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.196 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 21, 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.197 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 61, 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:42:15.198 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 29, 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.199 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 78, 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.200 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 86, 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.201 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 95, 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 4, 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 4 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.202 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 107, 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 112, 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:42:15.203 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 125, 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.204 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 10, 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 133, 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.205 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 154, 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 14, 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 164, 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.206 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 17, 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 177, 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 20, 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.207 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 234, 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 237, 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.208 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 241, 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 245, 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:15.209 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "new_removed_snaps": [], 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "new_purged_snaps": [], 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_node_flags": {}, 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "device_class_flags": {}, 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode": { 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_enabled": false, 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_bucket_count": 0, 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "degraded_stretch_mode": 0, 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "recovering_stretch_mode": 0, 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_bucket": 0 2024-02-23T00:42:15.210 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:15.211 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:42:15.211 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:42:15.211 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:42:15.211 INFO:teuthology.orchestra.run.smithi171.stdout: "complaint_time": 30, 2024-02-23T00:42:15.212 INFO:teuthology.orchestra.run.smithi171.stdout: "num_blocked_ops": 0 2024-02-23T00:42:15.212 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:42:15.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 10 librados: watch_flush enter 2024-02-23T00:42:15.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 10 librados: watch_flush exit 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 20 client.27759.objecter shutdown clearing up homeless session... 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 10 client.27759.objecter successfully canceled tick 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 >> v2:172.21.15.87:6800/1231597463 conn(0x7f986c062c90 msgr2=0x7f986c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 --2- 172.21.15.87:0/1150755745 >> v2:172.21.15.87:6800/1231597463 conn(0x7f986c062c90 0x7f986c065150 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f988406c870 tx=0x7f987400b040).stop 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98840f8420 msgr2=0x7f988406b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 --2- 172.21.15.87:0/1150755745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98840f8420 0x7f988406b0d0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 rx=0x7f9878009eb0 tx=0x7f987800b040).stop 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 shutdown_connections 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 --2- 172.21.15.87:0/1150755745 >> v2:172.21.15.87:6800/1231597463 conn(0x7f986c062c90 0x7f986c065150 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 --2- 172.21.15.87:0/1150755745 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f98840fe510 0x7f988406bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 --2- 172.21.15.87:0/1150755745 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f98840f8420 0x7f988406b0d0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 --2- 172.21.15.87:0/1150755745 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f98840fdbb0 0x7f988406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 >> 172.21.15.87:0/1150755745 conn(0x7f98840f3800 msgr2=0x7f98840f53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 shutdown_connections 2024-02-23T00:42:15.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 -- 172.21.15.87:0/1150755745 wait complete. 2024-02-23T00:42:15.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:15.104+0000 7f988a746700 1 librados: shutdown 2024-02-23T00:42:15.637 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:15.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:15.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:15.741 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:13.779153+0000 mgr.y (mgr.24437) 609 : cluster [DBG] pgmap v1085: 194 pgs: 1 active+remapped+backfill_toofull, 193 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 1/750 objects misplaced (0.133%); 0 B/s, 1 objects/s recovering 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.252221+0000 mon.a (mon.0) 3616 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.252394+0000 osd.1 (osd.1) 324 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.252957+0000 mon.a (mon.0) 3617 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.253311+0000 osd.1 (osd.1) 325 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:14.261493+0000 mon.a (mon.0) 3618 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122"}]': finished 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.261534+0000 mon.a (mon.0) 3619 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.275881+0000 osd.3 (osd.3) 222 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.278776+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.426834+0000 osd.7 (osd.7) 314 : cluster [DBG] 22.c scrub starts 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.427840+0000 osd.7 (osd.7) 315 : cluster [DBG] 22.c scrub ok 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.877917+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:42:15.769 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.878766+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:42:15.770 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.038110+0000 osd.5 (osd.5) 271 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:42:15.770 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.038894+0000 osd.5 (osd.5) 272 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:42:15.770 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:15.102497+0000 mon.a (mon.0) 3620 : audit [DBG] from='client.? 172.21.15.87:0/1150755745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:42:15.771 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:15.795 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:15.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:15.802 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-23T00:42:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:13.779153+0000 mgr.y (mgr.24437) 609 : cluster [DBG] pgmap v1085: 194 pgs: 1 active+remapped+backfill_toofull, 193 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 1/750 objects misplaced (0.133%); 0 B/s, 1 objects/s recovering 2024-02-23T00:42:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.252221+0000 mon.a (mon.0) 3616 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:42:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.252394+0000 osd.1 (osd.1) 324 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:42:15.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.252957+0000 mon.a (mon.0) 3617 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.253311+0000 osd.1 (osd.1) 325 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:14.261493+0000 mon.a (mon.0) 3618 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122"}]': finished 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.261534+0000 mon.a (mon.0) 3619 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.275881+0000 osd.3 (osd.3) 222 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.278776+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.426834+0000 osd.7 (osd.7) 314 : cluster [DBG] 22.c scrub starts 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.427840+0000 osd.7 (osd.7) 315 : cluster [DBG] 22.c scrub ok 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.877917+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.878766+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.038110+0000 osd.5 (osd.5) 271 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.038894+0000 osd.5 (osd.5) 272 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:42:15.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:15.102497+0000 mon.a (mon.0) 3620 : audit [DBG] from='client.? 172.21.15.87:0/1150755745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:13.779153+0000 mgr.y (mgr.24437) 609 : cluster [DBG] pgmap v1085: 194 pgs: 1 active+remapped+backfill_toofull, 193 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s; 1/750 objects misplaced (0.133%); 0 B/s, 1 objects/s recovering 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.252221+0000 mon.a (mon.0) 3616 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.252394+0000 osd.1 (osd.1) 324 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.252957+0000 mon.a (mon.0) 3617 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.253311+0000 osd.1 (osd.1) 325 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:14.261493+0000 mon.a (mon.0) 3618 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-122"}]': finished 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.261534+0000 mon.a (mon.0) 3619 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.275881+0000 osd.3 (osd.3) 222 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.278776+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.426834+0000 osd.7 (osd.7) 314 : cluster [DBG] 22.c scrub starts 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.427840+0000 osd.7 (osd.7) 315 : cluster [DBG] 22.c scrub ok 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.877917+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.878766+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:42:15.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.038110+0000 osd.5 (osd.5) 271 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:42:15.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.038894+0000 osd.5 (osd.5) 272 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:42:15.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:15.102497+0000 mon.a (mon.0) 3620 : audit [DBG] from='client.? 172.21.15.87:0/1150755745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:42:15.899 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:15.926 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:15.945 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:15.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:16.049 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:16.079 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:16.116 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:42:16.503 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 -- 172.21.15.87:0/190023015 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24b00fcbb0 msgr2=0x7f24b00fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 --2- 172.21.15.87:0/190023015 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24b00fcbb0 0x7f24b00fcf90 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 rx=0x7f24ac004870 tx=0x7f24ac009cb0).stop 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 -- 172.21.15.87:0/190023015 shutdown_connections 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 --2- 172.21.15.87:0/190023015 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24b00f7400 0x7f24b0103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 --2- 172.21.15.87:0/190023015 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24b00fcbb0 0x7f24b00fcf90 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 --2- 172.21.15.87:0/190023015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24b00fd510 0x7f24b00f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 -- 172.21.15.87:0/190023015 >> 172.21.15.87:0/190023015 conn(0x7f24b00f2820 msgr2=0x7f24b00f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 -- 172.21.15.87:0/190023015 shutdown_connections 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 -- 172.21.15.87:0/190023015 wait complete. 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 librados: starting msgr at 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.499+0000 7f24b7360700 1 librados: starting objecter 2024-02-23T00:42:16.504 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 Processor -- start 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 -- start start 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 librados: setting wanted keys 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 librados: calling monclient init 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24b00f7400 0x7f24b0109210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24b00fcbb0 0x7f24b0109700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24b00fd510 0x7f24b0109d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0114440 con 0x7f24b00fd510 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b01145b0 con 0x7f24b00f7400 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b7360700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0114770 con 0x7f24b00fcbb0 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b48fb700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24b00fcbb0 0x7f24b0109700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b48fb700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24b00fcbb0 0x7f24b0109700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60866/0 (socket says 172.21.15.87:60866) 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b48fb700 1 -- 172.21.15.87:0/1397319183 learned_addr learned my addr 172.21.15.87:0/1397319183 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b58fd700 1 --2- 172.21.15.87:0/1397319183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24b00fd510 0x7f24b0109d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.505 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24b00f7400 0x7f24b0109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b48fb700 1 -- 172.21.15.87:0/1397319183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24b00f7400 msgr2=0x7f24b0109210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b48fb700 1 --2- 172.21.15.87:0/1397319183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24b00f7400 0x7f24b0109210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b48fb700 1 -- 172.21.15.87:0/1397319183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24b00fd510 msgr2=0x7f24b0109d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b48fb700 1 --2- 172.21.15.87:0/1397319183 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f24b00fd510 0x7f24b0109d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b48fb700 1 -- 172.21.15.87:0/1397319183 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ac005040 con 0x7f24b00fcbb0 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f24b00f7400 0x7f24b0109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24b48fb700 1 --2- 172.21.15.87:0/1397319183 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24b00fcbb0 0x7f24b0109700 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7f24a000c810 tx=0x7f24a0004330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24a67fc700 10 client.?.objecter ms_handle_connect 0x7f24b00fcbb0 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24a67fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.500+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a00043a0 con 0x7f24b00fcbb0 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.501+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a00109e0 con 0x7f24b00fcbb0 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.501+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b0114a50 con 0x7f24b00fcbb0 2024-02-23T00:42:16.506 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.501+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0012ab0 con 0x7f24b00fcbb0 2024-02-23T00:42:16.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.501+0000 7f24b7360700 10 client.38240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:16.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.501+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b0114f90 con 0x7f24b00fcbb0 2024-02-23T00:42:16.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.501+0000 7f24b7360700 1 librados: init done 2024-02-23T00:42:16.507 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.501+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f24b005fa60 con 0x7f24b00fcbb0 2024-02-23T00:42:16.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.502+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f24a0010b50 con 0x7f24b00fcbb0 2024-02-23T00:42:16.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.502+0000 7f24a67fc700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6800/1231597463 conn(0x7f249c062c90 0x7f249c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.502+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(714..714 src has 173..714) v4 ==== 8558+0+0 (secure 0 0 0) 0x7f24a007d2a0 con 0x7f24b00fcbb0 2024-02-23T00:42:16.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.502+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6800/1231597463 conn(0x7f249c062c90 0x7f249c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.502+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 osd_map(714..714 src has 173..714) v4 2024-02-23T00:42:16.508 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.502+0000 7f24a67fc700 3 client.38240.objecter handle_osd_map got epochs [714,714] > 0 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.502+0000 7f24a67fc700 3 client.38240.objecter handle_osd_map decoding full epoch 714 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24a67fc700 20 client.38240.objecter dump_active .. 0 homeless 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v714) v1 ==== 54+0+16 (secure 0 0 0) 0x7f24a0084070 con 0x7f24b00fcbb0 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24a67fc700 10 client.38240.objecter ms_handle_connect 0x7f249c062c90 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6800/1231597463 conn(0x7f249c062c90 0x7f249c065150 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f24ac0043d0 tx=0x7f24ac00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 10 client.38240.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 1 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6801/484081831 conn(0x7f24b00fdea0 0x7f24b00fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b01060e0 osd=0 3 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b01060e0 osd=0 3 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b01060e0 osd=0 3 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 1 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b01060e0 osd=0 3 2024-02-23T00:42:16.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 0 1 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b01060e0 osd=0 4 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 10 client.38240.objecter _send_command 1 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b005ee80 con 0x7f24b00fdea0 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b58fd700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6801/484081831 conn(0x7f24b00fdea0 0x7f24b00fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24a67fc700 10 client.38240.objecter ms_handle_connect 0x7f24b00fdea0 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.503+0000 7f24b58fd700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6801/484081831 conn(0x7f24b00fdea0 0x7f24b00fe2c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.504+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b005ee80 con 0x7f24b00fdea0 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.504+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.504+0000 7f24a67fc700 10 client.38240.objecter _finish_command 1 = system:0 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.504+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b01060e0 osd=0 4 2024-02-23T00:42:16.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.504+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 0 1 2024-02-23T00:42:16.524 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 10 client.38240.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 2 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b01060e0 osd=0 3 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b01060e0 osd=0 3 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b01060e0 osd=0 3 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 2 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b01060e0 osd=0 3 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 0 2 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b01060e0 osd=0 4 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 10 client.38240.objecter _send_command 2 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.520+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b00273c0 con 0x7f24b00fdea0 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b00273c0 con 0x7f24b00fdea0 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24a67fc700 10 client.38240.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b01060e0 osd=0 4 2024-02-23T00:42:16.525 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 0 2 2024-02-23T00:42:16.526 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:42:16.526 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 10 client.38240.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 3 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6805/685354171 conn(0x7f24b00598f0 0x7f24b0059d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0028710 osd=1 3 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0028710 osd=1 3 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0028710 osd=1 3 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 3 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0028710 osd=1 3 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 1 3 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0028710 osd=1 4 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 10 client.38240.objecter _send_command 3 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b011b900 con 0x7f24b00598f0 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6805/685354171 conn(0x7f24b00598f0 0x7f24b0059d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24a67fc700 10 client.38240.objecter ms_handle_connect 0x7f24b00598f0 2024-02-23T00:42:16.527 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.522+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6805/685354171 conn(0x7f24b00598f0 0x7f24b0059d10 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.523+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b011b900 con 0x7f24b00598f0 2024-02-23T00:42:16.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.523+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:42:16.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.523+0000 7f24a67fc700 10 client.38240.objecter _finish_command 3 = system:0 2024-02-23T00:42:16.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.523+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0028710 osd=1 4 2024-02-23T00:42:16.528 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.523+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 1 3 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 10 client.38240.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 4 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0028710 osd=1 3 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0028710 osd=1 3 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0028710 osd=1 3 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 4 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0028710 osd=1 3 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 1 4 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0028710 osd=1 4 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 10 client.38240.objecter _send_command 4 2024-02-23T00:42:16.540 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.537+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b0059360 con 0x7f24b00598f0 2024-02-23T00:42:16.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b0059360 con 0x7f24b00598f0 2024-02-23T00:42:16.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:16.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24a67fc700 10 client.38240.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0028710 osd=1 4 2024-02-23T00:42:16.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 1 4 2024-02-23T00:42:16.542 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:42:16.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 10 client.38240.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 5 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6809/3605394228 conn(0x7f24b0028270 0x7f24b011b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0059770 osd=2 3 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0059770 osd=2 3 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0059770 osd=2 3 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 5 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0059770 osd=2 3 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 2 5 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0059770 osd=2 4 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 10 client.38240.objecter _send_command 5 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b011b6a0 con 0x7f24b0028270 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b48fb700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6809/3605394228 conn(0x7f24b0028270 0x7f24b011b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24a67fc700 10 client.38240.objecter ms_handle_connect 0x7f24b0028270 2024-02-23T00:42:16.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.539+0000 7f24b48fb700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6809/3605394228 conn(0x7f24b0028270 0x7f24b011b1b0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.540+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b011b6a0 con 0x7f24b0028270 2024-02-23T00:42:16.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.540+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:42:16.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.540+0000 7f24a67fc700 10 client.38240.objecter _finish_command 5 = system:0 2024-02-23T00:42:16.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.540+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0059770 osd=2 4 2024-02-23T00:42:16.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.540+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 2 5 2024-02-23T00:42:16.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 10 client.38240.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:16.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 6 2024-02-23T00:42:16.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0059770 osd=2 3 2024-02-23T00:42:16.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0059770 osd=2 3 2024-02-23T00:42:16.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0059770 osd=2 3 2024-02-23T00:42:16.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 6 2024-02-23T00:42:16.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0059770 osd=2 3 2024-02-23T00:42:16.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 2 6 2024-02-23T00:42:16.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0059770 osd=2 4 2024-02-23T00:42:16.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 10 client.38240.objecter _send_command 6 2024-02-23T00:42:16.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.555+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b00277d0 con 0x7f24b0028270 2024-02-23T00:42:16.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b00277d0 con 0x7f24b0028270 2024-02-23T00:42:16.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:16.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24a67fc700 10 client.38240.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0059770 osd=2 4 2024-02-23T00:42:16.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 2 6 2024-02-23T00:42:16.559 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:42:16.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 10 client.38240.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:16.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 7 2024-02-23T00:42:16.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6813/128059790 conn(0x7f24b011fe80 0x7f24b0122270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0027be0 osd=3 3 2024-02-23T00:42:16.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0027be0 osd=3 3 2024-02-23T00:42:16.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0027be0 osd=3 3 2024-02-23T00:42:16.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 7 2024-02-23T00:42:16.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0027be0 osd=3 3 2024-02-23T00:42:16.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 3 7 2024-02-23T00:42:16.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0027be0 osd=3 4 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 10 client.38240.objecter _send_command 7 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b0108e90 con 0x7f24b011fe80 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.556+0000 7f24b58fd700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6813/128059790 conn(0x7f24b011fe80 0x7f24b0122270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.557+0000 7f24a67fc700 10 client.38240.objecter ms_handle_connect 0x7f24b011fe80 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.557+0000 7f24b58fd700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6813/128059790 conn(0x7f24b011fe80 0x7f24b0122270 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.557+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b0108e90 con 0x7f24b011fe80 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.557+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.557+0000 7f24a67fc700 10 client.38240.objecter _finish_command 7 = system:0 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.557+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0027be0 osd=3 4 2024-02-23T00:42:16.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.557+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 3 7 2024-02-23T00:42:16.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 10 client.38240.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:16.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 8 2024-02-23T00:42:16.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0027be0 osd=3 3 2024-02-23T00:42:16.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0027be0 osd=3 3 2024-02-23T00:42:16.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0027be0 osd=3 3 2024-02-23T00:42:16.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 8 2024-02-23T00:42:16.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0027be0 osd=3 3 2024-02-23T00:42:16.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 3 8 2024-02-23T00:42:16.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0027be0 osd=3 4 2024-02-23T00:42:16.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 10 client.38240.objecter _send_command 8 2024-02-23T00:42:16.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.572+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b0107a10 con 0x7f24b011fe80 2024-02-23T00:42:16.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b0107a10 con 0x7f24b011fe80 2024-02-23T00:42:16.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:16.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24a67fc700 10 client.38240.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0027be0 osd=3 4 2024-02-23T00:42:16.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 3 8 2024-02-23T00:42:16.576 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:42:16.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 10 client.38240.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:16.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 9 2024-02-23T00:42:16.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6800/4175844757 conn(0x7f24b01080c0 0x7f24b01258f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0107a10 osd=4 3 2024-02-23T00:42:16.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0107a10 osd=4 3 2024-02-23T00:42:16.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0107a10 osd=4 3 2024-02-23T00:42:16.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 9 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0107a10 osd=4 3 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 4 9 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0107a10 osd=4 4 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 10 client.38240.objecter _send_command 9 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.573+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b0108550 con 0x7f24b01080c0 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.574+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6800/4175844757 conn(0x7f24b01080c0 0x7f24b01258f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.574+0000 7f24a67fc700 10 client.38240.objecter ms_handle_connect 0x7f24b01080c0 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.574+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6800/4175844757 conn(0x7f24b01080c0 0x7f24b01258f0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.575+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b0108550 con 0x7f24b01080c0 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.575+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:42:16.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.575+0000 7f24a67fc700 10 client.38240.objecter _finish_command 9 = system:0 2024-02-23T00:42:16.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.575+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0107a10 osd=4 4 2024-02-23T00:42:16.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.575+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 4 9 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 10 client.38240.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 10 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0107a10 osd=4 3 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0107a10 osd=4 3 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0107a10 osd=4 3 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 10 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0107a10 osd=4 3 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 4 10 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0107a10 osd=4 4 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 10 client.38240.objecter _send_command 10 2024-02-23T00:42:16.592 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.589+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b0108550 con 0x7f24b01080c0 2024-02-23T00:42:16.593 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b0108550 con 0x7f24b01080c0 2024-02-23T00:42:16.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:16.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24a67fc700 10 client.38240.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0107a10 osd=4 4 2024-02-23T00:42:16.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 4 10 2024-02-23T00:42:16.594 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:42:16.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 10 client.38240.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:16.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 11 2024-02-23T00:42:16.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6804/2550013795 conn(0x7f24b01262f0 0x7f24b01286e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0108550 osd=5 3 2024-02-23T00:42:16.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0108550 osd=5 3 2024-02-23T00:42:16.595 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0108550 osd=5 3 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 11 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0108550 osd=5 3 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 5 11 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0108550 osd=5 4 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 10 client.38240.objecter _send_command 11 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.591+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b012b3c0 con 0x7f24b01262f0 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.592+0000 7f24b48fb700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6804/2550013795 conn(0x7f24b01262f0 0x7f24b01286e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.592+0000 7f24a67fc700 10 client.38240.objecter ms_handle_connect 0x7f24b01262f0 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.592+0000 7f24b48fb700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6804/2550013795 conn(0x7f24b01262f0 0x7f24b01286e0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.592+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b012b3c0 con 0x7f24b01262f0 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.592+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.592+0000 7f24a67fc700 10 client.38240.objecter _finish_command 11 = system:0 2024-02-23T00:42:16.596 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.592+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0108550 osd=5 4 2024-02-23T00:42:16.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.592+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 5 11 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 10 client.38240.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 12 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0108550 osd=5 3 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0108550 osd=5 3 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b0108550 osd=5 3 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 12 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b0108550 osd=5 3 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 5 12 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0108550 osd=5 4 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 10 client.38240.objecter _send_command 12 2024-02-23T00:42:16.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.607+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b012b3c0 con 0x7f24b01262f0 2024-02-23T00:42:16.611 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:42:16.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.608+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b012b3c0 con 0x7f24b01262f0 2024-02-23T00:42:16.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.608+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:16.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.608+0000 7f24a67fc700 10 client.38240.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.608+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b0108550 osd=5 4 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.608+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 5 12 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 10 client.38240.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 13 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6808/2509408498 conn(0x7f24b01290e0 0x7f24b0129560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b012b3c0 osd=6 3 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b3c0 osd=6 3 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b012b3c0 osd=6 3 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 13 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b012b3c0 osd=6 3 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 6 13 2024-02-23T00:42:16.612 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b3c0 osd=6 4 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 10 client.38240.objecter _send_command 13 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b012b1b0 con 0x7f24b01290e0 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b58fd700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6808/2509408498 conn(0x7f24b01290e0 0x7f24b0129560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24a67fc700 10 client.38240.objecter ms_handle_connect 0x7f24b01290e0 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.609+0000 7f24b58fd700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6808/2509408498 conn(0x7f24b01290e0 0x7f24b0129560 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.610+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b012b1b0 con 0x7f24b01290e0 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.610+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.610+0000 7f24a67fc700 10 client.38240.objecter _finish_command 13 = system:0 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.610+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b012b3c0 osd=6 4 2024-02-23T00:42:16.613 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.610+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 6 13 2024-02-23T00:42:16.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 10 client.38240.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:16.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 14 2024-02-23T00:42:16.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b012b3c0 osd=6 3 2024-02-23T00:42:16.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b3c0 osd=6 3 2024-02-23T00:42:16.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b012b3c0 osd=6 3 2024-02-23T00:42:16.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 14 2024-02-23T00:42:16.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b012b3c0 osd=6 3 2024-02-23T00:42:16.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 6 14 2024-02-23T00:42:16.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b3c0 osd=6 4 2024-02-23T00:42:16.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 10 client.38240.objecter _send_command 14 2024-02-23T00:42:16.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.625+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b012b1b0 con 0x7f24b01290e0 2024-02-23T00:42:16.628 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:16.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b012b1b0 con 0x7f24b01290e0 2024-02-23T00:42:16.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:16.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24a67fc700 10 client.38240.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b012b3c0 osd=6 4 2024-02-23T00:42:16.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 6 14 2024-02-23T00:42:16.629 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:42:16.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 10 client.38240.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:16.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 15 2024-02-23T00:42:16.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6812/373787000 conn(0x7f24b012db30 0x7f24b012df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:16.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b012b1b0 osd=7 3 2024-02-23T00:42:16.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b1b0 osd=7 3 2024-02-23T00:42:16.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b012b1b0 osd=7 3 2024-02-23T00:42:16.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 15 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b012b1b0 osd=7 3 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 7 15 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b1b0 osd=7 4 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 10 client.38240.objecter _send_command 15 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f24b012e630 con 0x7f24b012db30 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.627+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6812/373787000 conn(0x7f24b012db30 0x7f24b012df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.628+0000 7f24a67fc700 10 client.38240.objecter ms_handle_connect 0x7f24b012db30 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.628+0000 7f24b50fc700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6812/373787000 conn(0x7f24b012db30 0x7f24b012df90 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.628+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f24b012e630 con 0x7f24b012db30 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.628+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:42:16.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.628+0000 7f24a67fc700 10 client.38240.objecter _finish_command 15 = system:0 2024-02-23T00:42:16.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.628+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b012b1b0 osd=7 4 2024-02-23T00:42:16.632 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.628+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 7 15 2024-02-23T00:42:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:16.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 10 client.38240.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign -1 16 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b012b1b0 osd=7 3 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b1b0 osd=7 3 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 20 client.38240.objecter _get_session s=0x7f24b012b1b0 osd=7 3 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 15 client.38240.objecter _session_command_op_remove -1 16 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 20 client.38240.objecter get_session s=0x7f24b012b1b0 osd=7 3 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 15 client.38240.objecter _session_command_op_assign 7 16 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b1b0 osd=7 4 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 10 client.38240.objecter _send_command 16 2024-02-23T00:42:16.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.643+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f24b012e630 con 0x7f24b012db30 2024-02-23T00:42:16.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24a67fc700 1 -- 172.21.15.87:0/1397319183 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f24b012e630 con 0x7f24b012db30 2024-02-23T00:42:16.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24a67fc700 10 client.38240.objecter ms_dispatch 0x7f24b0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:16.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24a67fc700 10 client.38240.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24a67fc700 20 client.38240.objecter put_session s=0x7f24b012b1b0 osd=7 4 2024-02-23T00:42:16.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24a67fc700 15 client.38240.objecter _session_command_op_remove 7 16 2024-02-23T00:42:16.648 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:42:16.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 10 librados: watch_flush enter 2024-02-23T00:42:16.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 10 librados: watch_flush exit 2024-02-23T00:42:16.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 10 client.38240.objecter close_session for osd.0 2024-02-23T00:42:16.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6801/484081831 conn(0x7f24b00fdea0 msgr2=0x7f24b00fe2c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6801/484081831 conn(0x7f24b00fdea0 0x7f24b00fe2c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b01060e0 osd=0 1 2024-02-23T00:42:16.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 10 client.38240.objecter close_session for osd.1 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6805/685354171 conn(0x7f24b00598f0 msgr2=0x7f24b0059d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6805/685354171 conn(0x7f24b00598f0 0x7f24b0059d10 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.645+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0028710 osd=1 1 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 10 client.38240.objecter close_session for osd.2 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6809/3605394228 conn(0x7f24b0028270 msgr2=0x7f24b011b1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6809/3605394228 conn(0x7f24b0028270 0x7f24b011b1b0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0059770 osd=2 1 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 10 client.38240.objecter close_session for osd.3 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6813/128059790 conn(0x7f24b011fe80 msgr2=0x7f24b0122270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6813/128059790 conn(0x7f24b011fe80 0x7f24b0122270 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0027be0 osd=3 1 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 10 client.38240.objecter close_session for osd.4 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6800/4175844757 conn(0x7f24b01080c0 msgr2=0x7f24b01258f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6800/4175844757 conn(0x7f24b01080c0 0x7f24b01258f0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b58fd700 1 -- 172.21.15.87:0/1397319183 reap_dead start 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b58fd700 1 -- 172.21.15.87:0/1397319183 reap_dead start 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0107a10 osd=4 1 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 10 client.38240.objecter close_session for osd.5 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6804/2550013795 conn(0x7f24b01262f0 msgr2=0x7f24b01286e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6804/2550013795 conn(0x7f24b01262f0 0x7f24b01286e0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b0108550 osd=5 1 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 10 client.38240.objecter close_session for osd.6 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6808/2509408498 conn(0x7f24b01290e0 msgr2=0x7f24b0129560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6808/2509408498 conn(0x7f24b01290e0 0x7f24b0129560 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b3c0 osd=6 1 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 10 client.38240.objecter close_session for osd.7 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6812/373787000 conn(0x7f24b012db30 msgr2=0x7f24b012df90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.171:6812/373787000 conn(0x7f24b012db30 0x7f24b012df90 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:16.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 20 client.38240.objecter put_session s=0x7f24b012b1b0 osd=7 1 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 20 client.38240.objecter shutdown clearing up homeless session... 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 10 client.38240.objecter successfully canceled tick 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6800/1231597463 conn(0x7f249c062c90 msgr2=0x7f249c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> v2:172.21.15.87:6800/1231597463 conn(0x7f249c062c90 0x7f249c065150 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 rx=0x7f24ac0043d0 tx=0x7f24ac00b040).stop 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24b00fcbb0 msgr2=0x7f24b0109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 --2- 172.21.15.87:0/1397319183 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f24b00fcbb0 0x7f24b0109700 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 rx=0x7f24a000c810 tx=0x7f24a0004330).stop 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b58fd700 1 -- 172.21.15.87:0/1397319183 reap_dead start 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 shutdown_connections 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 >> 172.21.15.87:0/1397319183 conn(0x7f24b00f2820 msgr2=0x7f24b00f43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 shutdown_connections 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 -- 172.21.15.87:0/1397319183 wait complete. 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:16.646+0000 7f24b7360700 1 librados: shutdown 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.652 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.653 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.653 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.653 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.653 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.653 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.653 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:42:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.756663+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:14.760302+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.220561+0000 osd.1 (osd.1) 326 : cluster [DBG] 2.a scrub starts 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.223451+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.a scrub ok 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.263307+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.387718+0000 osd.7 (osd.7) 316 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.388505+0000 osd.7 (osd.7) 317 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.779848+0000 mgr.y (mgr.24437) 610 : cluster [DBG] pgmap v1088: 162 pgs: 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1/738 objects misplaced (0.136%); 0 B/s, 1 objects/s recovering 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:16.065468+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:16.067582+0000 osd.5 (osd.5) 274 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:16.265866+0000 mon.a (mon.0) 3622 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:16.267107+0000 mon.a (mon.0) 3623 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:16.503924+0000 mon.c (mon.2) 494 : audit [DBG] from='client.? 172.21.15.87:0/1397319183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:16.720 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:42:16.736 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:16.764 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:16.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.756663+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:42:16.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:14.760302+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:42:16.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.220561+0000 osd.1 (osd.1) 326 : cluster [DBG] 2.a scrub starts 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.223451+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.a scrub ok 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.263307+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.387718+0000 osd.7 (osd.7) 316 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.388505+0000 osd.7 (osd.7) 317 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.779848+0000 mgr.y (mgr.24437) 610 : cluster [DBG] pgmap v1088: 162 pgs: 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1/738 objects misplaced (0.136%); 0 B/s, 1 objects/s recovering 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:16.065468+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:16.067582+0000 osd.5 (osd.5) 274 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:16.265866+0000 mon.a (mon.0) 3622 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:16.267107+0000 mon.a (mon.0) 3623 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:16.503924+0000 mon.c (mon.2) 494 : audit [DBG] from='client.? 172.21.15.87:0/1397319183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.756663+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:42:16.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:14.760302+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.220561+0000 osd.1 (osd.1) 326 : cluster [DBG] 2.a scrub starts 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.223451+0000 osd.1 (osd.1) 327 : cluster [DBG] 2.a scrub ok 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.263307+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.387718+0000 osd.7 (osd.7) 316 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.388505+0000 osd.7 (osd.7) 317 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.779848+0000 mgr.y (mgr.24437) 610 : cluster [DBG] pgmap v1088: 162 pgs: 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1/738 objects misplaced (0.136%); 0 B/s, 1 objects/s recovering 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:16.065468+0000 osd.5 (osd.5) 273 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:16.067582+0000 osd.5 (osd.5) 274 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:16.265866+0000 mon.a (mon.0) 3622 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:16.267107+0000 mon.a (mon.0) 3623 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:16.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:16.503924+0000 mon.c (mon.2) 494 : audit [DBG] from='client.? 172.21.15.87:0/1397319183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:16.862 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:16.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:16.965 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:16.993 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:17.036 INFO:tasks.thrashosds.thrasher:j is {'epoch': 712, 'fsid': '66db207c-d1e1-11ee-95c0-87774f69a715', 'created': '2024-02-23T00:21:28.861318+0000', 'modified': '2024-02-23T00:42:14.252699+0000', 'last_up_change': '2024-02-23T00:41:18.209057+0000', 'last_in_change': '2024-02-23T00:41:36.208573+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 114, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 245, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 13, 'pg_placement_num': 13, 'pg_placement_num_target': 13, 'pg_num_target': 13, 'pg_num_pending': 13, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '135', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '4.16', 'ready_epoch': 371, 'last_epoch_started': 370, 'last_epoch_clean': 370, 'source_version': "63'21", 'target_version': "62'12"}, 'last_change': '532', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '371', 'last_force_op_resend_preluminous': '481', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '5.20', 'ready_epoch': 641, 'last_epoch_started': 640, 'last_epoch_clean': 640, 'source_version': "562'12", 'target_version': "330'9"}, 'last_change': '641', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '641', 'last_force_op_resend_preluminous': '639', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 12, 'pg_placement_num': 12, 'pg_placement_num_target': 12, 'pg_num_target': 12, 'pg_num_pending': 12, 'last_pg_merge_meta': {'source_pgid': '6.c', 'ready_epoch': 665, 'last_epoch_started': 664, 'last_epoch_clean': 664, 'source_version': "78'2", 'target_version': "0'0"}, 'last_change': '665', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '665', 'last_force_op_resend_preluminous': '663', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 22, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-23T00:30:46.071806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi087-82091-101', 'create_time': '2024-02-23T00:40:15.435168+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '712', 'last_force_op_resend': '711', 'last_force_op_resend_prenautilus': '711', 'last_force_op_resend_preluminous': '711', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 710, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi087-82091-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 245, 'pool_name': 'test-rados-api-smithi087-82091-122', 'create_time': '2024-02-23T00:42:02.493920+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '712', 'last_force_op_resend': '711', 'last_force_op_resend_prenautilus': '711', 'last_force_op_resend_preluminous': '711', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 710, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 0, 'cache_target_dirty_high_ratio_micro': 0, 'cache_target_full_ratio_micro': 0, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': 'e3549bad-2891-47b9-bf77-132cadc20bf7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 424, 'up_from': 659, 'up_thru': 707, 'down_at': 425, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6801', 'nonce': 484081831}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6802', 'nonce': 484081831}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6804', 'nonce': 484081831}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6803', 'nonce': 484081831}]}, 'public_addr': '172.21.15.87:6801/484081831', 'cluster_addr': '172.21.15.87:6802/484081831', 'heartbeat_back_addr': '172.21.15.87:6804/484081831', 'heartbeat_front_addr': '172.21.15.87:6803/484081831', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '85525ce8-38d5-4840-8057-547cb176c859', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 706, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6805', 'nonce': 685354171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6806', 'nonce': 685354171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6808', 'nonce': 685354171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6807', 'nonce': 685354171}]}, 'public_addr': '172.21.15.87:6805/685354171', 'cluster_addr': '172.21.15.87:6806/685354171', 'heartbeat_back_addr': '172.21.15.87:6808/685354171', 'heartbeat_front_addr': '172.21.15.87:6807/685354171', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 701, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6809', 'nonce': 3605394228}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6810', 'nonce': 3605394228}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6812', 'nonce': 3605394228}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6811', 'nonce': 3605394228}]}, 'public_addr': '172.21.15.87:6809/3605394228', 'cluster_addr': '172.21.15.87:6810/3605394228', 'heartbeat_back_addr': '172.21.15.87:6812/3605394228', 'heartbeat_front_addr': '172.21.15.87:6811/3605394228', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 701, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6813', 'nonce': 128059790}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6814', 'nonce': 128059790}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6816', 'nonce': 128059790}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6815', 'nonce': 128059790}]}, 'public_addr': '172.21.15.87:6813/128059790', 'cluster_addr': '172.21.15.87:6814/128059790', 'heartbeat_back_addr': '172.21.15.87:6816/128059790', 'heartbeat_front_addr': '172.21.15.87:6815/128059790', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c3734a04-7343-4251-9b5b-bedb9e8b882c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5775604248046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 701, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6800', 'nonce': 4175844757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6801', 'nonce': 4175844757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6803', 'nonce': 4175844757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6802', 'nonce': 4175844757}]}, 'public_addr': '172.21.15.171:6800/4175844757', 'cluster_addr': '172.21.15.171:6801/4175844757', 'heartbeat_back_addr': '172.21.15.171:6803/4175844757', 'heartbeat_front_addr': '172.21.15.171:6802/4175844757', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fe86232d-96b9-4cbf-9cac-373286fa103d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8517608642578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 701, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6804', 'nonce': 2550013795}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6805', 'nonce': 2550013795}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6807', 'nonce': 2550013795}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6806', 'nonce': 2550013795}]}, 'public_addr': '172.21.15.171:6804/2550013795', 'cluster_addr': '172.21.15.171:6805/2550013795', 'heartbeat_back_addr': '172.21.15.171:6807/2550013795', 'heartbeat_front_addr': '172.21.15.171:6806/2550013795', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 701, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6808', 'nonce': 2509408498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6809', 'nonce': 2509408498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6811', 'nonce': 2509408498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6810', 'nonce': 2509408498}]}, 'public_addr': '172.21.15.171:6808/2509408498', 'cluster_addr': '172.21.15.171:6809/2509408498', 'heartbeat_back_addr': '172.21.15.171:6811/2509408498', 'heartbeat_front_addr': '172.21.15.171:6810/2509408498', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '29ddcf97-369a-4a2c-b3e2-142e84ccf939', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 701, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6812', 'nonce': 373787000}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6813', 'nonce': 373787000}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6815', 'nonce': 373787000}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6814', 'nonce': 373787000}]}, 'public_addr': '172.21.15.171:6812/373787000', 'cluster_addr': '172.21.15.171:6813/373787000', 'heartbeat_back_addr': '172.21.15.171:6815/373787000', 'heartbeat_front_addr': '172.21.15.171:6814/373787000', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:41:17.571824+0000', 'dead_epoch': 424}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:13.973488+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:41:43.491272+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:41:04.154188+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:41:09.190052+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:10.862105+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:41:01.921983+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:41:10.945471+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '2.1c', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '22.1a', 'mappings': [{'from': 7, 'to': 5}]}], 'pg_temp': [{'pgid': '5.3', 'osds': [3, 6, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.87:6800/1121018082': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/2138166229': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/27794119': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/1675126043': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/3767016525': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/969918521': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/795095363': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2273151327': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/1685406414': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2093733106': '2024-02-24T00:21:58.144574+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi087-82091-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 21, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 61, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 63, 'snaps': [{'begin': 29, 'length': 1}]}, {'pool': 78, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 86, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 177, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 234, 'snaps': [{'begin': 9, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 241, 'snaps': [{'begin': 6, 'length': 1}]}, {'pool': 245, 'snaps': [{'begin': 9, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-23T00:42:17.036 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-02-23T00:42:17.328 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:17.538 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.732106+0000 osd.0 (osd.0) 17 : cluster [DBG] 4.b scrub starts 2024-02-23T00:42:17.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:15.735601+0000 osd.0 (osd.0) 18 : cluster [DBG] 4.b scrub ok 2024-02-23T00:42:17.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:16.222160+0000 osd.1 (osd.1) 328 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:42:17.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:16.223715+0000 osd.1 (osd.1) 329 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:42:17.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:16.495443+0000 osd.2 (osd.2) 255 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:42:17.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:16.497194+0000 osd.2 (osd.2) 256 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:42:17.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:17.055174+0000 osd.5 (osd.5) 275 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:42:17.624 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:17.056145+0000 osd.5 (osd.5) 276 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:42:17.625 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:17.216053+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:17.625 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:17.216228+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-02-23T00:42:17.625 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:17.223777+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:17.626 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:17.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:17.727 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:17.758 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:17.784 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:17.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:17.788 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:42:17.788 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:42:17.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.732106+0000 osd.0 (osd.0) 17 : cluster [DBG] 4.b scrub starts 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:15.735601+0000 osd.0 (osd.0) 18 : cluster [DBG] 4.b scrub ok 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:16.222160+0000 osd.1 (osd.1) 328 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:16.223715+0000 osd.1 (osd.1) 329 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:16.495443+0000 osd.2 (osd.2) 255 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:16.497194+0000 osd.2 (osd.2) 256 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:17.055174+0000 osd.5 (osd.5) 275 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:17.056145+0000 osd.5 (osd.5) 276 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:17.216053+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:17.216228+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:17.223777+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:17.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.732106+0000 osd.0 (osd.0) 17 : cluster [DBG] 4.b scrub starts 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:15.735601+0000 osd.0 (osd.0) 18 : cluster [DBG] 4.b scrub ok 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:16.222160+0000 osd.1 (osd.1) 328 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:16.223715+0000 osd.1 (osd.1) 329 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:16.495443+0000 osd.2 (osd.2) 255 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:16.497194+0000 osd.2 (osd.2) 256 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:17.055174+0000 osd.5 (osd.5) 275 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:17.056145+0000 osd.5 (osd.5) 276 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:17.216053+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:17.216228+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-02-23T00:42:17.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:17.223777+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:17.889 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:17.916 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:18.014 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:18.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:18.118 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:18.146 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:18.468 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-23T00:42:18.733 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:42:18.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:17.472190+0000 osd.2 (osd.2) 257 : cluster [DBG] 5.b scrub starts 2024-02-23T00:42:18.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:17.473469+0000 osd.2 (osd.2) 258 : cluster [DBG] 5.b scrub ok 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:17.780278+0000 mgr.y (mgr.24437) 611 : cluster [DBG] pgmap v1091: 194 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1/738 objects misplaced (0.136%) 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:18.041105+0000 osd.5 (osd.5) 277 : cluster [DBG] 234.3 scrub starts 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:18.042151+0000 osd.5 (osd.5) 278 : cluster [DBG] 234.3 scrub ok 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:18.224734+0000 mon.a (mon.0) 3627 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:18.224801+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:18.225229+0000 mon.a (mon.0) 3629 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-124"}]: dispatch 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:17.472190+0000 osd.2 (osd.2) 257 : cluster [DBG] 5.b scrub starts 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:17.473469+0000 osd.2 (osd.2) 258 : cluster [DBG] 5.b scrub ok 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:17.780278+0000 mgr.y (mgr.24437) 611 : cluster [DBG] pgmap v1091: 194 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1/738 objects misplaced (0.136%) 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:18.041105+0000 osd.5 (osd.5) 277 : cluster [DBG] 234.3 scrub starts 2024-02-23T00:42:18.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:18.042151+0000 osd.5 (osd.5) 278 : cluster [DBG] 234.3 scrub ok 2024-02-23T00:42:18.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:18.224734+0000 mon.a (mon.0) 3627 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:18.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:18.224801+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-02-23T00:42:18.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:18.225229+0000 mon.a (mon.0) 3629 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-124"}]: dispatch 2024-02-23T00:42:18.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:17.472190+0000 osd.2 (osd.2) 257 : cluster [DBG] 5.b scrub starts 2024-02-23T00:42:18.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:17.473469+0000 osd.2 (osd.2) 258 : cluster [DBG] 5.b scrub ok 2024-02-23T00:42:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:17.780278+0000 mgr.y (mgr.24437) 611 : cluster [DBG] pgmap v1091: 194 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1/738 objects misplaced (0.136%) 2024-02-23T00:42:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:18.041105+0000 osd.5 (osd.5) 277 : cluster [DBG] 234.3 scrub starts 2024-02-23T00:42:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:18.042151+0000 osd.5 (osd.5) 278 : cluster [DBG] 234.3 scrub ok 2024-02-23T00:42:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:18.224734+0000 mon.a (mon.0) 3627 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:18.224801+0000 mon.a (mon.0) 3628 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-02-23T00:42:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:18.225229+0000 mon.a (mon.0) 3629 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-124"}]: dispatch 2024-02-23T00:42:19.497 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:19.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:19.601 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:19.630 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 -- 172.21.15.87:0/3553209959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70940f9610 msgr2=0x7f7094101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 --2- 172.21.15.87:0/3553209959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70940f9610 0x7f7094101d20 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 rx=0x7f7084004870 tx=0x7f7084009cb0).stop 2024-02-23T00:42:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 -- 172.21.15.87:0/3553209959 shutdown_connections 2024-02-23T00:42:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 --2- 172.21.15.87:0/3553209959 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7094102210 0x7f7094104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:19.734 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 --2- 172.21.15.87:0/3553209959 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70940f8cb0 0x7f70940f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 --2- 172.21.15.87:0/3553209959 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70940f9610 0x7f7094101d20 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 -- 172.21.15.87:0/3553209959 >> 172.21.15.87:0/3553209959 conn(0x7f70940f2820 msgr2=0x7f70940f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 -- 172.21.15.87:0/3553209959 shutdown_connections 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 -- 172.21.15.87:0/3553209959 wait complete. 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 librados: starting msgr at 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 librados: starting objecter 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 Processor -- start 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 -- start start 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 librados: setting wanted keys 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 librados: calling monclient init 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.709+0000 7f709b562700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70940f8cb0 0x7f709410e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f709b562700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70940f9610 0x7f709410e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f709b562700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7094102210 0x7f7094108410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:19.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f709b562700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7094108900 con 0x7f70940f8cb0 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f709b562700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7094108a70 con 0x7f7094102210 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f709b562700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7094108c30 con 0x7f70940f9610 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f70992fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70940f8cb0 0x7f709410e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f70992fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70940f8cb0 0x7f709410e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53594/0 (socket says 172.21.15.87:53594) 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f7098afd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70940f9610 0x7f709410e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f70992fe700 1 -- 172.21.15.87:0/3500355048 learned_addr learned my addr 172.21.15.87:0/3500355048 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f7099aff700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7094102210 0x7f7094108410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f70992fe700 1 -- 172.21.15.87:0/3500355048 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70940f9610 msgr2=0x7f709410e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f70992fe700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70940f9610 0x7f709410e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f70992fe700 1 -- 172.21.15.87:0/3500355048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7094102210 msgr2=0x7f7094108410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f70992fe700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7094102210 0x7f7094108410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f70992fe700 1 -- 172.21.15.87:0/3500355048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7090005040 con 0x7f70940f8cb0 2024-02-23T00:42:19.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f7098afd700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70940f9610 0x7f709410e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f7099aff700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7094102210 0x7f7094108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f70992fe700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70940f8cb0 0x7f709410e220 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7f70900048f0 tx=0x7f7090009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f708a7fc700 10 client.?.objecter ms_handle_connect 0x7f70940f8cb0 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f708a7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f708a7fc700 1 -- 172.21.15.87:0/3500355048 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7090007b00 con 0x7f70940f8cb0 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7084005040 con 0x7f70940f8cb0 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f708a7fc700 1 -- 172.21.15.87:0/3500355048 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f709000f040 con 0x7f70940f8cb0 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f708a7fc700 1 -- 172.21.15.87:0/3500355048 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7090014a00 con 0x7f70940f8cb0 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f709b562700 10 client.27771.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7094109260 con 0x7f70940f8cb0 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.710+0000 7f709b562700 1 librados: init done 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.711+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f709405fa60 con 0x7f70940f8cb0 2024-02-23T00:42:19.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f708a7fc700 1 -- 172.21.15.87:0/3500355048 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f709000e580 con 0x7f70940f8cb0 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f708a7fc700 1 --2- 172.21.15.87:0/3500355048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7080062c90 0x7f7080065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f7098afd700 1 --2- 172.21.15.87:0/3500355048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7080062c90 0x7f7080065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f708a7fc700 1 -- 172.21.15.87:0/3500355048 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(717..717 src has 173..717) v4 ==== 8538+0+0 (secure 0 0 0) 0x7f709007cfa0 con 0x7f70940f8cb0 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f708a7fc700 10 client.27771.objecter ms_dispatch 0x7f7094001550 osd_map(717..717 src has 173..717) v4 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f708a7fc700 3 client.27771.objecter handle_osd_map got epochs [717,717] > 0 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f708a7fc700 3 client.27771.objecter handle_osd_map decoding full epoch 717 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f708a7fc700 20 client.27771.objecter dump_active .. 0 homeless 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f708a7fc700 10 client.27771.objecter ms_handle_connect 0x7f7080062c90 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.712+0000 7f7098afd700 1 --2- 172.21.15.87:0/3500355048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7080062c90 0x7f7080065150 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 rx=0x7f70840043d0 tx=0x7f708401a040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:19.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.716+0000 7f708a7fc700 1 -- 172.21.15.87:0/3500355048 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7090049f60 con 0x7f70940f8cb0 2024-02-23T00:42:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:17.717997+0000 osd.0 (osd.0) 19 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:42:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:17.721464+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:42:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:18.358310+0000 osd.7 (osd.7) 318 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:42:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:18.360536+0000 osd.7 (osd.7) 319 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:42:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:18.655923+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.e scrub starts 2024-02-23T00:42:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:18.659530+0000 osd.6 (osd.6) 287 : cluster [DBG] 4.e scrub ok 2024-02-23T00:42:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.047351+0000 osd.5 (osd.5) 279 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:42:19.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.049186+0000 osd.5 (osd.5) 280 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:19.226673+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-124"}]': finished 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.226738+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:19.228916+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-124", "mode": "writeback"}]: dispatch 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.482350+0000 osd.2 (osd.2) 259 : cluster [DBG] 4.d scrub starts 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.485042+0000 osd.2 (osd.2) 260 : cluster [DBG] 4.d scrub ok 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:17.717997+0000 osd.0 (osd.0) 19 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:17.721464+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:18.358310+0000 osd.7 (osd.7) 318 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:18.360536+0000 osd.7 (osd.7) 319 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:18.655923+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.e scrub starts 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:18.659530+0000 osd.6 (osd.6) 287 : cluster [DBG] 4.e scrub ok 2024-02-23T00:42:19.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.047351+0000 osd.5 (osd.5) 279 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:42:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.049186+0000 osd.5 (osd.5) 280 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:42:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:19.226673+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-124"}]': finished 2024-02-23T00:42:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.226738+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-02-23T00:42:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:19.228916+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-124", "mode": "writeback"}]: dispatch 2024-02-23T00:42:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.482350+0000 osd.2 (osd.2) 259 : cluster [DBG] 4.d scrub starts 2024-02-23T00:42:19.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.485042+0000 osd.2 (osd.2) 260 : cluster [DBG] 4.d scrub ok 2024-02-23T00:42:19.893 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:19.890+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f70941098a0 con 0x7f70940f8cb0 2024-02-23T00:42:19.906 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:17.717997+0000 osd.0 (osd.0) 19 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:42:19.906 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:17.721464+0000 osd.0 (osd.0) 20 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:42:19.906 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:18.358310+0000 osd.7 (osd.7) 318 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:18.360536+0000 osd.7 (osd.7) 319 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:18.655923+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.e scrub starts 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:18.659530+0000 osd.6 (osd.6) 287 : cluster [DBG] 4.e scrub ok 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.047351+0000 osd.5 (osd.5) 279 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.049186+0000 osd.5 (osd.5) 280 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:19.226673+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-124"}]': finished 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.226738+0000 mon.a (mon.0) 3631 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:19.228916+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-124", "mode": "writeback"}]: dispatch 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.482350+0000 osd.2 (osd.2) 259 : cluster [DBG] 4.d scrub starts 2024-02-23T00:42:19.907 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.485042+0000 osd.2 (osd.2) 260 : cluster [DBG] 4.d scrub ok 2024-02-23T00:42:20.179 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:20.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:20.229 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.226+0000 7f708a7fc700 1 -- 172.21.15.87:0/3500355048 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v718) v1 ==== 101+0+0 (secure 0 0 0) 0x7f7090050290 con 0x7f70940f8cb0 2024-02-23T00:42:20.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 10 librados: watch_flush enter 2024-02-23T00:42:20.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 10 librados: watch_flush exit 2024-02-23T00:42:20.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 20 client.27771.objecter shutdown clearing up homeless session... 2024-02-23T00:42:20.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 10 client.27771.objecter successfully canceled tick 2024-02-23T00:42:20.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7080062c90 msgr2=0x7f7080065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:20.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 --2- 172.21.15.87:0/3500355048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7080062c90 0x7f7080065150 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 rx=0x7f70840043d0 tx=0x7f708401a040).stop 2024-02-23T00:42:20.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70940f8cb0 msgr2=0x7f709410e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:20.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70940f8cb0 0x7f709410e220 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 rx=0x7f70900048f0 tx=0x7f7090009cb0).stop 2024-02-23T00:42:20.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 shutdown_connections 2024-02-23T00:42:20.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 --2- 172.21.15.87:0/3500355048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7080062c90 0x7f7080065150 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:20.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7094102210 0x7f7094108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:20.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f70940f8cb0 0x7f709410e220 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:20.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 --2- 172.21.15.87:0/3500355048 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f70940f9610 0x7f709410e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:20.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 >> 172.21.15.87:0/3500355048 conn(0x7f70940f2820 msgr2=0x7f70940f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:20.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 shutdown_connections 2024-02-23T00:42:20.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.229+0000 7f709b562700 1 -- 172.21.15.87:0/3500355048 wait complete. 2024-02-23T00:42:20.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:20.230+0000 7f709b562700 1 librados: shutdown 2024-02-23T00:42:20.234 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:42:20.273 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:42:20.283 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:20.344 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:20.441 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:42:20.441 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:42:20.468 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:20.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:20.572 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:20.600 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:20.653 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:20.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:20.756 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:20.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:18.669396+0000 osd.0 (osd.0) 21 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:42:20.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:18.672925+0000 osd.0 (osd.0) 22 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:42:20.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.319447+0000 osd.7 (osd.7) 320 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:42:20.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.323640+0000 osd.7 (osd.7) 321 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:42:20.758 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.652291+0000 osd.6 (osd.6) 288 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:42:20.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.655183+0000 osd.6 (osd.6) 289 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:42:20.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.780679+0000 mgr.y (mgr.24437) 612 : cluster [DBG] pgmap v1094: 194 pgs: 30 unknown, 1 active+remapped+backfill_toofull, 163 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:20.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:19.892519+0000 mon.a (mon.0) 3633 : audit [INF] from='client.? 172.21.15.87:0/3500355048' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:20.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:20.223353+0000 mon.a (mon.0) 3634 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:42:20.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:20.223379+0000 mon.a (mon.0) 3635 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:42:20.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:20.227693+0000 mon.a (mon.0) 3636 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-124", "mode": "writeback"}]': finished 2024-02-23T00:42:20.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:20.227775+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? 172.21.15.87:0/3500355048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:42:20.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:20.227826+0000 mon.a (mon.0) 3638 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-02-23T00:42:20.759 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:20.269794+0000 mon.a (mon.0) 3639 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:20.783 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:18.669396+0000 osd.0 (osd.0) 21 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:18.672925+0000 osd.0 (osd.0) 22 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.319447+0000 osd.7 (osd.7) 320 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.323640+0000 osd.7 (osd.7) 321 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.652291+0000 osd.6 (osd.6) 288 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.655183+0000 osd.6 (osd.6) 289 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.780679+0000 mgr.y (mgr.24437) 612 : cluster [DBG] pgmap v1094: 194 pgs: 30 unknown, 1 active+remapped+backfill_toofull, 163 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:19.892519+0000 mon.a (mon.0) 3633 : audit [INF] from='client.? 172.21.15.87:0/3500355048' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:20.223353+0000 mon.a (mon.0) 3634 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:42:20.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:20.223379+0000 mon.a (mon.0) 3635 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:20.227693+0000 mon.a (mon.0) 3636 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-124", "mode": "writeback"}]': finished 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:20.227775+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? 172.21.15.87:0/3500355048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:20.227826+0000 mon.a (mon.0) 3638 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:20.269794+0000 mon.a (mon.0) 3639 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:18.669396+0000 osd.0 (osd.0) 21 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:18.672925+0000 osd.0 (osd.0) 22 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.319447+0000 osd.7 (osd.7) 320 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.323640+0000 osd.7 (osd.7) 321 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.652291+0000 osd.6 (osd.6) 288 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.655183+0000 osd.6 (osd.6) 289 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.780679+0000 mgr.y (mgr.24437) 612 : cluster [DBG] pgmap v1094: 194 pgs: 30 unknown, 1 active+remapped+backfill_toofull, 163 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:20.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:19.892519+0000 mon.a (mon.0) 3633 : audit [INF] from='client.? 172.21.15.87:0/3500355048' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:20.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:20.223353+0000 mon.a (mon.0) 3634 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:42:20.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:20.223379+0000 mon.a (mon.0) 3635 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:42:20.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:20.227693+0000 mon.a (mon.0) 3636 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-124", "mode": "writeback"}]': finished 2024-02-23T00:42:20.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:20.227775+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? 172.21.15.87:0/3500355048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:42:20.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:20.227826+0000 mon.a (mon.0) 3638 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-02-23T00:42:20.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:20.269794+0000 mon.a (mon.0) 3639 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:21.050 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-23T00:42:21.120 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:21.131 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:21.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:21.235 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:21.262 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:21.325 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:42:21.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:42:21.460 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:21.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:21.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:21.563 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:21.591 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:21.611 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:21.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:21.715 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:21.742 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:21.779 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:21.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:21.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:21.800 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.679357+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.e scrub starts 2024-02-23T00:42:21.800 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:19.682894+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.e scrub ok 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:20.216730+0000 osd.1 (osd.1) 330 : cluster [DBG] 22.b scrub starts 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:20.220090+0000 osd.1 (osd.1) 331 : cluster [DBG] 22.b scrub ok 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:20.333002+0000 osd.3 (osd.3) 224 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:20.334005+0000 osd.3 (osd.3) 225 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:20.632222+0000 osd.6 (osd.6) 290 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:20.635890+0000 osd.6 (osd.6) 291 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:21.224869+0000 mgr.y (mgr.24437) 613 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:21.233619+0000 mon.a (mon.0) 3640 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:21.233680+0000 mon.a (mon.0) 3641 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:21.236359+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124"}]: dispatch 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:21.441453+0000 osd.2 (osd.2) 261 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:42:21.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:21.442353+0000 osd.2 (osd.2) 262 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:42:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.679357+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.e scrub starts 2024-02-23T00:42:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:19.682894+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.e scrub ok 2024-02-23T00:42:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:20.216730+0000 osd.1 (osd.1) 330 : cluster [DBG] 22.b scrub starts 2024-02-23T00:42:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:20.220090+0000 osd.1 (osd.1) 331 : cluster [DBG] 22.b scrub ok 2024-02-23T00:42:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:20.333002+0000 osd.3 (osd.3) 224 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:42:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:20.334005+0000 osd.3 (osd.3) 225 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:42:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:20.632222+0000 osd.6 (osd.6) 290 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:20.635890+0000 osd.6 (osd.6) 291 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:21.224869+0000 mgr.y (mgr.24437) 613 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:21.233619+0000 mon.a (mon.0) 3640 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:21.233680+0000 mon.a (mon.0) 3641 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:21.236359+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124"}]: dispatch 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:21.441453+0000 osd.2 (osd.2) 261 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:21.442353+0000 osd.2 (osd.2) 262 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.679357+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.e scrub starts 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:19.682894+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.e scrub ok 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:20.216730+0000 osd.1 (osd.1) 330 : cluster [DBG] 22.b scrub starts 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:20.220090+0000 osd.1 (osd.1) 331 : cluster [DBG] 22.b scrub ok 2024-02-23T00:42:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:20.333002+0000 osd.3 (osd.3) 224 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:42:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:20.334005+0000 osd.3 (osd.3) 225 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:42:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:20.632222+0000 osd.6 (osd.6) 290 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:42:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:20.635890+0000 osd.6 (osd.6) 291 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:42:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:21.224869+0000 mgr.y (mgr.24437) 613 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:21.233619+0000 mon.a (mon.0) 3640 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:21.233680+0000 mon.a (mon.0) 3641 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-02-23T00:42:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:21.236359+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124"}]: dispatch 2024-02-23T00:42:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:21.441453+0000 osd.2 (osd.2) 261 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:42:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:21.442353+0000 osd.2 (osd.2) 262 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:42:21.882 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:21.911 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:22.009 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:22.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:22.037 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-23T00:42:22.038 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:42:22.038 INFO:tasks.thrashosds.thrasher:Growing pool iscsi 2024-02-23T00:42:22.038 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:42:22.113 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:22.140 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:22.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:42:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:22] "GET /metrics HTTP/1.1" 200 190219 "" "Prometheus/2.33.4" 2024-02-23T00:42:22.595 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:22.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:22.699 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:22.726 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:22.799 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:22.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:22.906 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:22.935 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:23.060 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:23.060 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:42:23.060 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:42:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:23.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:23.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:42:23.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.148+0000 7f9d77746700 1 -- 172.21.15.87:0/1190794147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d700f7400 msgr2=0x7f9d70103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.148+0000 7f9d77746700 1 --2- 172.21.15.87:0/1190794147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d700f7400 0x7f9d70103a20 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 rx=0x7f9d68009be0 tx=0x7f9d68004d10).stop 2024-02-23T00:42:23.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.148+0000 7f9d77746700 1 -- 172.21.15.87:0/1190794147 shutdown_connections 2024-02-23T00:42:23.153 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.148+0000 7f9d77746700 1 --2- 172.21.15.87:0/1190794147 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d700f7400 0x7f9d70103a20 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.148+0000 7f9d77746700 1 --2- 172.21.15.87:0/1190794147 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d700fcbb0 0x7f9d700fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.148+0000 7f9d77746700 1 --2- 172.21.15.87:0/1190794147 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d700fd510 0x7f9d700f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.148+0000 7f9d77746700 1 -- 172.21.15.87:0/1190794147 >> 172.21.15.87:0/1190794147 conn(0x7f9d700f2820 msgr2=0x7f9d700f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 -- 172.21.15.87:0/1190794147 shutdown_connections 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 -- 172.21.15.87:0/1190794147 wait complete. 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 librados: starting msgr at 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 librados: starting objecter 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 Processor -- start 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 -- start start 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 librados: setting wanted keys 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 librados: calling monclient init 2024-02-23T00:42:23.154 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d700f7400 0x7f9d700f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d700fcbb0 0x7f9d700f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d700fd510 0x7f9d700f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d701163a0 con 0x7f9d700fcbb0 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d70116510 con 0x7f9d700fd510 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.149+0000 7f9d77746700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d701166d0 con 0x7f9d700f7400 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d74ce1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d700fcbb0 0x7f9d700f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d754e2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d700f7400 0x7f9d700f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d754e2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d700f7400 0x7f9d700f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50300/0 (socket says 172.21.15.87:50300) 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d74ce1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d700fcbb0 0x7f9d700f8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42308/0 (socket says 172.21.15.87:42308) 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d754e2700 1 -- 172.21.15.87:0/620677692 learned_addr learned my addr 172.21.15.87:0/620677692 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d75ce3700 1 --2- 172.21.15.87:0/620677692 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d700fd510 0x7f9d700f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.155 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d754e2700 1 -- 172.21.15.87:0/620677692 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d700fd510 msgr2=0x7f9d700f9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d754e2700 1 --2- 172.21.15.87:0/620677692 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9d700fd510 0x7f9d700f9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d754e2700 1 -- 172.21.15.87:0/620677692 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d700fcbb0 msgr2=0x7f9d700f8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d754e2700 1 --2- 172.21.15.87:0/620677692 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d700fcbb0 0x7f9d700f8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d754e2700 1 -- 172.21.15.87:0/620677692 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d60005040 con 0x7f9d700f7400 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d74ce1700 1 --2- 172.21.15.87:0/620677692 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9d700fcbb0 0x7f9d700f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d754e2700 1 --2- 172.21.15.87:0/620677692 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d700f7400 0x7f9d700f89b0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7f9d6c00c7c0 tx=0x7f9d6c007ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d667fc700 10 client.?.objecter ms_handle_connect 0x7f9d700f7400 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d667fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c004430 con 0x7f9d700f7400 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d680097c0 con 0x7f9d700f7400 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d6c00e970 con 0x7f9d700f7400 2024-02-23T00:42:23.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c013a50 con 0x7f9d700f7400 2024-02-23T00:42:23.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d77746700 10 client.38252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:23.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d70116d10 con 0x7f9d700f7400 2024-02-23T00:42:23.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.150+0000 7f9d77746700 1 librados: init done 2024-02-23T00:42:23.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.151+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9d7005fa60 con 0x7f9d700f7400 2024-02-23T00:42:23.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.151+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f9d6c00e4a0 con 0x7f9d700f7400 2024-02-23T00:42:23.158 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d667fc700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d5c062c90 0x7f9d5c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(720..720 src has 173..720) v4 ==== 8530+0+0 (secure 0 0 0) 0x7f9d6c07d640 con 0x7f9d700f7400 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 osd_map(720..720 src has 173..720) v4 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d667fc700 3 client.38252.objecter handle_osd_map got epochs [720,720] > 0 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d667fc700 3 client.38252.objecter handle_osd_map decoding full epoch 720 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d74ce1700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d5c062c90 0x7f9d5c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d667fc700 20 client.38252.objecter dump_active .. 0 homeless 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v720) v1 ==== 54+0+16 (secure 0 0 0) 0x7f9d6c084070 con 0x7f9d700f7400 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 10 client.38252.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 1 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d667fc700 10 client.38252.objecter ms_handle_connect 0x7f9d5c062c90 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d74ce1700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d5c062c90 0x7f9d5c065150 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7f9d700fa150 tx=0x7f9d6000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6801/484081831 conn(0x7f9d700ffed0 0x7f9d701002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7005ee80 osd=0 3 2024-02-23T00:42:23.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7005ee80 osd=0 3 2024-02-23T00:42:23.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7005ee80 osd=0 3 2024-02-23T00:42:23.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 1 2024-02-23T00:42:23.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d7005ee80 osd=0 3 2024-02-23T00:42:23.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 0 1 2024-02-23T00:42:23.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7005ee80 osd=0 4 2024-02-23T00:42:23.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 10 client.38252.objecter _send_command 1 2024-02-23T00:42:23.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.152+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d70106120 con 0x7f9d700ffed0 2024-02-23T00:42:23.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.153+0000 7f9d75ce3700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6801/484081831 conn(0x7f9d700ffed0 0x7f9d701002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.153+0000 7f9d667fc700 10 client.38252.objecter ms_handle_connect 0x7f9d700ffed0 2024-02-23T00:42:23.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.153+0000 7f9d75ce3700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6801/484081831 conn(0x7f9d700ffed0 0x7f9d701002b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.153+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d70106120 con 0x7f9d700ffed0 2024-02-23T00:42:23.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.153+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:42:23.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.153+0000 7f9d667fc700 10 client.38252.objecter _finish_command 1 = system:0 2024-02-23T00:42:23.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.153+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d7005ee80 osd=0 4 2024-02-23T00:42:23.161 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.153+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 0 1 2024-02-23T00:42:23.163 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 10 client.38252.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 2 2024-02-23T00:42:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7005ee80 osd=0 3 2024-02-23T00:42:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7005ee80 osd=0 3 2024-02-23T00:42:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7005ee80 osd=0 3 2024-02-23T00:42:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 2 2024-02-23T00:42:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d7005ee80 osd=0 3 2024-02-23T00:42:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 0 2 2024-02-23T00:42:23.171 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7005ee80 osd=0 4 2024-02-23T00:42:23.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 10 client.38252.objecter _send_command 2 2024-02-23T00:42:23.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.169+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d700273c0 con 0x7f9d700ffed0 2024-02-23T00:42:23.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d700273c0 con 0x7f9d700ffed0 2024-02-23T00:42:23.172 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:23.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d667fc700 10 client.38252.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d7005ee80 osd=0 4 2024-02-23T00:42:23.173 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 0 2 2024-02-23T00:42:23.173 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:42:23.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 10 client.38252.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:23.174 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 3 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d700598f0 0x7f9d70059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70028710 osd=1 3 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70028710 osd=1 3 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70028710 osd=1 3 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 3 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d70028710 osd=1 3 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 1 3 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70028710 osd=1 4 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 10 client.38252.objecter _send_command 3 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.170+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7011d710 con 0x7f9d700598f0 2024-02-23T00:42:23.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.171+0000 7f9d754e2700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d700598f0 0x7f9d70059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.171+0000 7f9d667fc700 10 client.38252.objecter ms_handle_connect 0x7f9d700598f0 2024-02-23T00:42:23.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.171+0000 7f9d754e2700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d700598f0 0x7f9d70059cd0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.171+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7011d710 con 0x7f9d700598f0 2024-02-23T00:42:23.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.171+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:42:23.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.171+0000 7f9d667fc700 10 client.38252.objecter _finish_command 3 = system:0 2024-02-23T00:42:23.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.171+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d70028710 osd=1 4 2024-02-23T00:42:23.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.171+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 1 3 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 10 client.38252.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 4 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70028710 osd=1 3 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70028710 osd=1 3 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70028710 osd=1 3 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 4 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d70028710 osd=1 3 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 1 4 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70028710 osd=1 4 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 10 client.38252.objecter _send_command 4 2024-02-23T00:42:23.191 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.188+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d70059360 con 0x7f9d700598f0 2024-02-23T00:42:23.192 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:23.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d70059360 con 0x7f9d700598f0 2024-02-23T00:42:23.192 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d667fc700 10 client.38252.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d70028710 osd=1 4 2024-02-23T00:42:23.193 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 1 4 2024-02-23T00:42:23.193 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 10 client.38252.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 5 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d70028270 0x7f9d7011c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70059770 osd=2 3 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70059770 osd=2 3 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70059770 osd=2 3 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 5 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d70059770 osd=2 3 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 2 5 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70059770 osd=2 4 2024-02-23T00:42:23.194 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 10 client.38252.objecter _send_command 5 2024-02-23T00:42:23.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.190+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7011c620 con 0x7f9d70028270 2024-02-23T00:42:23.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.191+0000 7f9d74ce1700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d70028270 0x7f9d7011c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.191+0000 7f9d667fc700 10 client.38252.objecter ms_handle_connect 0x7f9d70028270 2024-02-23T00:42:23.195 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.191+0000 7f9d74ce1700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d70028270 0x7f9d7011c130 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.192+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7011c620 con 0x7f9d70028270 2024-02-23T00:42:23.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.192+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:42:23.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.192+0000 7f9d667fc700 10 client.38252.objecter _finish_command 5 = system:0 2024-02-23T00:42:23.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.192+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d70059770 osd=2 4 2024-02-23T00:42:23.196 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.192+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 2 5 2024-02-23T00:42:23.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 10 client.38252.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:23.211 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 6 2024-02-23T00:42:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70059770 osd=2 3 2024-02-23T00:42:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70059770 osd=2 3 2024-02-23T00:42:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70059770 osd=2 3 2024-02-23T00:42:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 6 2024-02-23T00:42:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d70059770 osd=2 3 2024-02-23T00:42:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 2 6 2024-02-23T00:42:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70059770 osd=2 4 2024-02-23T00:42:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 10 client.38252.objecter _send_command 6 2024-02-23T00:42:23.212 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.208+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d700277d0 con 0x7f9d70028270 2024-02-23T00:42:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.210+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d700277d0 con 0x7f9d70028270 2024-02-23T00:42:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.210+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.210+0000 7f9d667fc700 10 client.38252.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.210+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d70059770 osd=2 4 2024-02-23T00:42:23.213 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.210+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 2 6 2024-02-23T00:42:23.213 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.210+0000 7f9d77746700 10 client.38252.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.210+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 7 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.210+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7011f890 0x7f9d70121c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70027be0 osd=3 3 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70027be0 osd=3 3 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70027be0 osd=3 3 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 7 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d70027be0 osd=3 3 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 3 7 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70027be0 osd=3 4 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d77746700 10 client.38252.objecter _send_command 7 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d70124940 con 0x7f9d7011f890 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d75ce3700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7011f890 0x7f9d70121c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.214 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d75ce3700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7011f890 0x7f9d70121c80 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.215 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:23.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.211+0000 7f9d667fc700 10 client.38252.objecter ms_handle_connect 0x7f9d7011f890 2024-02-23T00:42:23.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.212+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d70124940 con 0x7f9d7011f890 2024-02-23T00:42:23.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.212+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:42:23.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.212+0000 7f9d667fc700 10 client.38252.objecter _finish_command 7 = system:0 2024-02-23T00:42:23.215 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.212+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d70027be0 osd=3 4 2024-02-23T00:42:23.216 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.212+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 3 7 2024-02-23T00:42:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:23.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 10 client.38252.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 8 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70027be0 osd=3 3 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70027be0 osd=3 3 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d70027be0 osd=3 3 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 8 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d70027be0 osd=3 3 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 3 8 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70027be0 osd=3 4 2024-02-23T00:42:23.231 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 10 client.38252.objecter _send_command 8 2024-02-23T00:42:23.232 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.229+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7011ca80 con 0x7f9d7011f890 2024-02-23T00:42:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.230+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7011ca80 con 0x7f9d7011f890 2024-02-23T00:42:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.230+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.230+0000 7f9d667fc700 10 client.38252.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.230+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d70027be0 osd=3 4 2024-02-23T00:42:23.233 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.230+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 3 8 2024-02-23T00:42:23.233 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 10 client.38252.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 9 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7011cbf0 0x7f9d701232f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7011ca80 osd=4 3 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011ca80 osd=4 3 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7011ca80 osd=4 3 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 9 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d7011ca80 osd=4 3 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 4 9 2024-02-23T00:42:23.234 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011ca80 osd=4 4 2024-02-23T00:42:23.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 10 client.38252.objecter _send_command 9 2024-02-23T00:42:23.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7011bde0 con 0x7f9d7011cbf0 2024-02-23T00:42:23.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d754e2700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7011cbf0 0x7f9d701232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d667fc700 10 client.38252.objecter ms_handle_connect 0x7f9d7011cbf0 2024-02-23T00:42:23.235 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.231+0000 7f9d754e2700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7011cbf0 0x7f9d701232f0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.232+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7011bde0 con 0x7f9d7011cbf0 2024-02-23T00:42:23.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.232+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:42:23.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.232+0000 7f9d667fc700 10 client.38252.objecter _finish_command 9 = system:0 2024-02-23T00:42:23.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.232+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d7011ca80 osd=4 4 2024-02-23T00:42:23.236 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.232+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 4 9 2024-02-23T00:42:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 10 client.38252.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 10 2024-02-23T00:42:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7011ca80 osd=4 3 2024-02-23T00:42:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011ca80 osd=4 3 2024-02-23T00:42:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7011ca80 osd=4 3 2024-02-23T00:42:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 10 2024-02-23T00:42:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d7011ca80 osd=4 3 2024-02-23T00:42:23.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 4 10 2024-02-23T00:42:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011ca80 osd=4 4 2024-02-23T00:42:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 10 client.38252.objecter _send_command 10 2024-02-23T00:42:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.254+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7011bde0 con 0x7f9d7011cbf0 2024-02-23T00:42:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7011bde0 con 0x7f9d7011cbf0 2024-02-23T00:42:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d667fc700 10 client.38252.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d7011ca80 osd=4 4 2024-02-23T00:42:23.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 4 10 2024-02-23T00:42:23.259 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:42:23.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 10 client.38252.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:23.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 11 2024-02-23T00:42:23.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d70123cf0 0x7f9d70124130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7011bde0 osd=5 3 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011bde0 osd=5 3 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7011bde0 osd=5 3 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 11 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d7011bde0 osd=5 3 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 5 11 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011bde0 osd=5 4 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 10 client.38252.objecter _send_command 11 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.256+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d701246b0 con 0x7f9d70123cf0 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.257+0000 7f9d74ce1700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d70123cf0 0x7f9d70124130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.257+0000 7f9d667fc700 10 client.38252.objecter ms_handle_connect 0x7f9d70123cf0 2024-02-23T00:42:23.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.257+0000 7f9d74ce1700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d70123cf0 0x7f9d70124130 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.258+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d701246b0 con 0x7f9d70123cf0 2024-02-23T00:42:23.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.258+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:42:23.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.258+0000 7f9d667fc700 10 client.38252.objecter _finish_command 11 = system:0 2024-02-23T00:42:23.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.258+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d7011bde0 osd=5 4 2024-02-23T00:42:23.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.258+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 5 11 2024-02-23T00:42:23.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 10 client.38252.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:23.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 12 2024-02-23T00:42:23.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7011bde0 osd=5 3 2024-02-23T00:42:23.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011bde0 osd=5 3 2024-02-23T00:42:23.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7011bde0 osd=5 3 2024-02-23T00:42:23.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 12 2024-02-23T00:42:23.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d7011bde0 osd=5 3 2024-02-23T00:42:23.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 5 12 2024-02-23T00:42:23.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011bde0 osd=5 4 2024-02-23T00:42:23.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 10 client.38252.objecter _send_command 12 2024-02-23T00:42:23.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.274+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d701246b0 con 0x7f9d70123cf0 2024-02-23T00:42:23.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.275+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d701246b0 con 0x7f9d70123cf0 2024-02-23T00:42:23.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.275+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:23.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.275+0000 7f9d667fc700 10 client.38252.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.275+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d7011bde0 osd=5 4 2024-02-23T00:42:23.278 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.275+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 5 12 2024-02-23T00:42:23.278 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 10 client.38252.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 13 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d70129090 0x7f9d7012b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d701246b0 osd=6 3 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d701246b0 osd=6 3 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d701246b0 osd=6 3 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 13 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d701246b0 osd=6 3 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 6 13 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d701246b0 osd=6 4 2024-02-23T00:42:23.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 10 client.38252.objecter _send_command 13 2024-02-23T00:42:23.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d7012e2a0 con 0x7f9d70129090 2024-02-23T00:42:23.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.276+0000 7f9d75ce3700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d70129090 0x7f9d7012b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.278+0000 7f9d667fc700 10 client.38252.objecter ms_handle_connect 0x7f9d70129090 2024-02-23T00:42:23.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.278+0000 7f9d75ce3700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d70129090 0x7f9d7012b4a0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.279+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d7012e2a0 con 0x7f9d70129090 2024-02-23T00:42:23.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.279+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:42:23.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.279+0000 7f9d667fc700 10 client.38252.objecter _finish_command 13 = system:0 2024-02-23T00:42:23.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.279+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d701246b0 osd=6 4 2024-02-23T00:42:23.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.279+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 6 13 2024-02-23T00:42:23.286 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 10 client.38252.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 14 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d701246b0 osd=6 3 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d701246b0 osd=6 3 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d701246b0 osd=6 3 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 14 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d701246b0 osd=6 3 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 6 14 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d701246b0 osd=6 4 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 10 client.38252.objecter _send_command 14 2024-02-23T00:42:23.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.302+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7012e2a0 con 0x7f9d70129090 2024-02-23T00:42:23.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.304+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7012e2a0 con 0x7f9d70129090 2024-02-23T00:42:23.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.304+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:23.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.304+0000 7f9d667fc700 10 client.38252.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.304+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d701246b0 osd=6 4 2024-02-23T00:42:23.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.304+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 6 14 2024-02-23T00:42:23.307 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.304+0000 7f9d77746700 10 client.38252.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 15 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7012d670 0x7f9d7012dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7012e2a0 osd=7 3 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7012e2a0 osd=7 3 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7012e2a0 osd=7 3 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 15 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d7012e2a0 osd=7 3 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 7 15 2024-02-23T00:42:23.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7012e2a0 osd=7 4 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 10 client.38252.objecter _send_command 15 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d70131650 con 0x7f9d7012d670 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d754e2700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7012d670 0x7f9d7012dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d667fc700 10 client.38252.objecter ms_handle_connect 0x7f9d7012d670 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.305+0000 7f9d754e2700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7012d670 0x7f9d7012dad0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.306+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f9d70131650 con 0x7f9d7012d670 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.306+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.306+0000 7f9d667fc700 10 client.38252.objecter _finish_command 15 = system:0 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.306+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d7012e2a0 osd=7 4 2024-02-23T00:42:23.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.306+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 7 15 2024-02-23T00:42:23.317 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 10 client.38252.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign -1 16 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7012e2a0 osd=7 3 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7012e2a0 osd=7 3 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 20 client.38252.objecter _get_session s=0x7f9d7012e2a0 osd=7 3 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 15 client.38252.objecter _session_command_op_remove -1 16 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 20 client.38252.objecter get_session s=0x7f9d7012e2a0 osd=7 3 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 15 client.38252.objecter _session_command_op_assign 7 16 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7012e2a0 osd=7 4 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 10 client.38252.objecter _send_command 16 2024-02-23T00:42:23.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.327+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f9d7012ba80 con 0x7f9d7012d670 2024-02-23T00:42:23.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.329+0000 7f9d667fc700 1 -- 172.21.15.87:0/620677692 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f9d7012ba80 con 0x7f9d7012d670 2024-02-23T00:42:23.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.329+0000 7f9d667fc700 10 client.38252.objecter ms_dispatch 0x7f9d70001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:23.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.329+0000 7f9d667fc700 10 client.38252.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.329+0000 7f9d667fc700 20 client.38252.objecter put_session s=0x7f9d7012e2a0 osd=7 4 2024-02-23T00:42:23.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.329+0000 7f9d667fc700 15 client.38252.objecter _session_command_op_remove 7 16 2024-02-23T00:42:23.332 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:42:23.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.329+0000 7f9d77746700 10 librados: watch_flush enter 2024-02-23T00:42:23.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 librados: watch_flush exit 2024-02-23T00:42:23.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 client.38252.objecter close_session for osd.0 2024-02-23T00:42:23.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6801/484081831 conn(0x7f9d700ffed0 msgr2=0x7f9d701002b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6801/484081831 conn(0x7f9d700ffed0 0x7f9d701002b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7005ee80 osd=0 1 2024-02-23T00:42:23.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 client.38252.objecter close_session for osd.1 2024-02-23T00:42:23.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d700598f0 msgr2=0x7f9d70059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6805/685354171 conn(0x7f9d700598f0 0x7f9d70059cd0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70028710 osd=1 1 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 client.38252.objecter close_session for osd.2 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d70028270 msgr2=0x7f9d7011c130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6809/3605394228 conn(0x7f9d70028270 0x7f9d7011c130 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70059770 osd=2 1 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 client.38252.objecter close_session for osd.3 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7011f890 msgr2=0x7f9d70121c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6813/128059790 conn(0x7f9d7011f890 0x7f9d70121c80 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d70027be0 osd=3 1 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 client.38252.objecter close_session for osd.4 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7011cbf0 msgr2=0x7f9d701232f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6800/4175844757 conn(0x7f9d7011cbf0 0x7f9d701232f0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011ca80 osd=4 1 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 client.38252.objecter close_session for osd.5 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d70123cf0 msgr2=0x7f9d70124130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6804/2550013795 conn(0x7f9d70123cf0 0x7f9d70124130 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7011bde0 osd=5 1 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 client.38252.objecter close_session for osd.6 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d70129090 msgr2=0x7f9d7012b4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d75ce3700 1 -- 172.21.15.87:0/620677692 reap_dead start 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d70129090 0x7f9d7012b4a0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d75ce3700 1 -- 172.21.15.87:0/620677692 reap_dead start 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d75ce3700 1 -- 172.21.15.87:0/620677692 reap_dead start 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d701246b0 osd=6 1 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 client.38252.objecter close_session for osd.7 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7012d670 msgr2=0x7f9d7012dad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7012d670 0x7f9d7012dad0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 20 client.38252.objecter put_session s=0x7f9d7012e2a0 osd=7 1 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 20 client.38252.objecter shutdown clearing up homeless session... 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 10 client.38252.objecter successfully canceled tick 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d5c062c90 msgr2=0x7f9d5c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d5c062c90 0x7f9d5c065150 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 rx=0x7f9d700fa150 tx=0x7f9d6000b040).stop 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d700f7400 msgr2=0x7f9d700f89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d700f7400 0x7f9d700f89b0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 rx=0x7f9d6c00c7c0 tx=0x7f9d6c007ef0).stop 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 shutdown_connections 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6808/2509408498 conn(0x7f9d70129090 0x7f9d7012b4a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.171:6812/373787000 conn(0x7f9d7012d670 0x7f9d7012dad0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9d5c062c90 0x7f9d5c065150 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 --2- 172.21.15.87:0/620677692 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9d700f7400 0x7f9d700f89b0 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 >> 172.21.15.87:0/620677692 conn(0x7f9d700f2820 msgr2=0x7f9d700f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 shutdown_connections 2024-02-23T00:42:23.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 -- 172.21.15.87:0/620677692 wait complete. 2024-02-23T00:42:23.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:23.330+0000 7f9d77746700 1 librados: shutdown 2024-02-23T00:42:23.337 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.337 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.337 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.337 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.337 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.337 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.337 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.337 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:42:23.343 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:21.262223+0000 osd.7 (osd.7) 322 : cluster [DBG] 234.7 scrub starts 2024-02-23T00:42:23.343 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:21.267344+0000 osd.7 (osd.7) 323 : cluster [DBG] 234.7 scrub ok 2024-02-23T00:42:23.343 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:21.781125+0000 mgr.y (mgr.24437) 614 : cluster [DBG] pgmap v1097: 194 pgs: 1 active+remapped+backfill_toofull, 193 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:23.343 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:22.035897+0000 osd.5 (osd.5) 281 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:42:23.343 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:22.037003+0000 osd.5 (osd.5) 282 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:42:23.344 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:22.228160+0000 mon.a (mon.0) 3643 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:23.344 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:22.237077+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124"}]': finished 2024-02-23T00:42:23.344 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:22.237146+0000 mon.a (mon.0) 3645 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-02-23T00:42:23.344 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:22.665634+0000 osd.6 (osd.6) 292 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:42:23.344 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:22.669086+0000 osd.6 (osd.6) 293 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:42:23.344 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:23.153463+0000 mon.c (mon.2) 495 : audit [DBG] from='client.? 172.21.15.87:0/620677692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:23.345 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:21.262223+0000 osd.7 (osd.7) 322 : cluster [DBG] 234.7 scrub starts 2024-02-23T00:42:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:21.267344+0000 osd.7 (osd.7) 323 : cluster [DBG] 234.7 scrub ok 2024-02-23T00:42:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:21.781125+0000 mgr.y (mgr.24437) 614 : cluster [DBG] pgmap v1097: 194 pgs: 1 active+remapped+backfill_toofull, 193 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:22.035897+0000 osd.5 (osd.5) 281 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:42:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:22.037003+0000 osd.5 (osd.5) 282 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:42:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:22.228160+0000 mon.a (mon.0) 3643 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:23.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:22.237077+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124"}]': finished 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:22.237146+0000 mon.a (mon.0) 3645 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:22.665634+0000 osd.6 (osd.6) 292 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:22.669086+0000 osd.6 (osd.6) 293 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:23.153463+0000 mon.c (mon.2) 495 : audit [DBG] from='client.? 172.21.15.87:0/620677692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:21.262223+0000 osd.7 (osd.7) 322 : cluster [DBG] 234.7 scrub starts 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:21.267344+0000 osd.7 (osd.7) 323 : cluster [DBG] 234.7 scrub ok 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:21.781125+0000 mgr.y (mgr.24437) 614 : cluster [DBG] pgmap v1097: 194 pgs: 1 active+remapped+backfill_toofull, 193 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:22.035897+0000 osd.5 (osd.5) 281 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:22.037003+0000 osd.5 (osd.5) 282 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:42:23.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:22.228160+0000 mon.a (mon.0) 3643 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:23.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:22.237077+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-124"}]': finished 2024-02-23T00:42:23.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:22.237146+0000 mon.a (mon.0) 3645 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-02-23T00:42:23.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:22.665634+0000 osd.6 (osd.6) 292 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:42:23.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:22.669086+0000 osd.6 (osd.6) 293 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:42:23.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:23.153463+0000 mon.c (mon.2) 495 : audit [DBG] from='client.? 172.21.15.87:0/620677692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:23.795 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:23.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:23.814 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-23T00:42:23.899 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:23.927 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:24.122 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:42:24.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:42:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:42:24.407 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:24.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:24.509 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:24.535 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:22.298277+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:42:24.535 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:22.301720+0000 osd.7 (osd.7) 325 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:42:24.535 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:23.235250+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-02-23T00:42:24.535 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:22.298277+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:42:24.535 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:22.301720+0000 osd.7 (osd.7) 325 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:42:24.536 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:23.235250+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-02-23T00:42:24.537 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:24.564 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:24.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:24.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:22.298277+0000 osd.7 (osd.7) 324 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:42:24.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:22.301720+0000 osd.7 (osd.7) 325 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:42:24.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:23.235250+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-02-23T00:42:24.668 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:24.696 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:25.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:23.781684+0000 mgr.y (mgr.24437) 615 : cluster [DBG] pgmap v1100: 162 pgs: 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:25.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:24.244864+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-02-23T00:42:25.256 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:24.250118+0000 mon.a (mon.0) 3648 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:25.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:24.446967+0000 osd.2 (osd.2) 263 : cluster [DBG] 2.c scrub starts 2024-02-23T00:42:25.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:24.447728+0000 osd.2 (osd.2) 264 : cluster [DBG] 2.c scrub ok 2024-02-23T00:42:25.257 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:24.656394+0000 mon.a (mon.0) 3649 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:25.393 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:23.781684+0000 mgr.y (mgr.24437) 615 : cluster [DBG] pgmap v1100: 162 pgs: 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:25.394 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:24.244864+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-02-23T00:42:25.394 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:24.250118+0000 mon.a (mon.0) 3648 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:25.394 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:24.446967+0000 osd.2 (osd.2) 263 : cluster [DBG] 2.c scrub starts 2024-02-23T00:42:25.394 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:24.447728+0000 osd.2 (osd.2) 264 : cluster [DBG] 2.c scrub ok 2024-02-23T00:42:25.394 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:24.656394+0000 mon.a (mon.0) 3649 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:23.781684+0000 mgr.y (mgr.24437) 615 : cluster [DBG] pgmap v1100: 162 pgs: 1 active+remapped+backfill_toofull, 161 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:24.244864+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-02-23T00:42:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:24.250118+0000 mon.a (mon.0) 3648 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:24.446967+0000 osd.2 (osd.2) 263 : cluster [DBG] 2.c scrub starts 2024-02-23T00:42:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:24.447728+0000 osd.2 (osd.2) 264 : cluster [DBG] 2.c scrub ok 2024-02-23T00:42:25.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:24.656394+0000 mon.a (mon.0) 3649 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.891+0000 7f3eaeab3700 1 -- 172.21.15.87:0/3337475948 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80fdbb0 msgr2=0x7f3ea80fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.891+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/3337475948 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80fdbb0 0x7f3ea80fdf90 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 rx=0x7f3e9c0047b0 tx=0x7f3e9c009cb0).stop 2024-02-23T00:42:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.891+0000 7f3eaeab3700 1 -- 172.21.15.87:0/3337475948 shutdown_connections 2024-02-23T00:42:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.891+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/3337475948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80f8420 0x7f3ea8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.891+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/3337475948 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80fdbb0 0x7f3ea80fdf90 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.891+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/3337475948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fe510 0x7f3ea80f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:25.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.891+0000 7f3eaeab3700 1 -- 172.21.15.87:0/3337475948 >> 172.21.15.87:0/3337475948 conn(0x7f3ea80f3800 msgr2=0x7f3ea80f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.891+0000 7f3eaeab3700 1 -- 172.21.15.87:0/3337475948 shutdown_connections 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.891+0000 7f3eaeab3700 1 -- 172.21.15.87:0/3337475948 wait complete. 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 librados: starting msgr at 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 librados: starting objecter 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 Processor -- start 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 -- start start 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 librados: setting wanted keys 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 librados: calling monclient init 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80f8420 0x7f3ea810e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fdbb0 0x7f3ea810e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80fe510 0x7f3ea8108280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8108770 con 0x7f3ea80fdbb0 2024-02-23T00:42:25.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea81088e0 con 0x7f3ea80fe510 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eaeab3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8108aa0 con 0x7f3ea80f8420 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eac84f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80f8420 0x7f3ea810e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3ea7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fdbb0 0x7f3ea810e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eac84f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80f8420 0x7f3ea810e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50310/0 (socket says 172.21.15.87:50310) 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3eac84f700 1 -- 172.21.15.87:0/897322109 learned_addr learned my addr 172.21.15.87:0/897322109 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.892+0000 7f3ead050700 1 --2- 172.21.15.87:0/897322109 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80fe510 0x7f3ea8108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eac84f700 1 -- 172.21.15.87:0/897322109 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80fe510 msgr2=0x7f3ea8108280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eac84f700 1 --2- 172.21.15.87:0/897322109 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80fe510 0x7f3ea8108280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eac84f700 1 -- 172.21.15.87:0/897322109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fdbb0 msgr2=0x7f3ea810e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eac84f700 1 --2- 172.21.15.87:0/897322109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fdbb0 0x7f3ea810e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eac84f700 1 -- 172.21.15.87:0/897322109 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e94005040 con 0x7f3ea80f8420 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/897322109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fdbb0 0x7f3ea810e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eac84f700 1 --2- 172.21.15.87:0/897322109 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80f8420 0x7f3ea810e0e0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f3e9c0044e0 tx=0x7f3e9c007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3ea5f76700 10 client.?.objecter ms_handle_connect 0x7f3ea80f8420 2024-02-23T00:42:25.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3ea5f76700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3ea5f76700 1 -- 172.21.15.87:0/897322109 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c018070 con 0x7f3ea80f8420 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3ea5f76700 1 -- 172.21.15.87:0/897322109 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e9c0093b0 con 0x7f3ea80f8420 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c005040 con 0x7f3ea80f8420 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3ea5f76700 1 -- 172.21.15.87:0/897322109 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c01dd90 con 0x7f3ea80f8420 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eaeab3700 10 client.38258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea8108f40 con 0x7f3ea80f8420 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.893+0000 7f3eaeab3700 1 librados: init done 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.894+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea805ee80 con 0x7f3ea80f8420 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.894+0000 7f3ea5f76700 1 -- 172.21.15.87:0/897322109 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3e9c01d420 con 0x7f3ea80f8420 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.895+0000 7f3ea5f76700 1 --2- 172.21.15.87:0/897322109 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 0x7f3e98065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.895+0000 7f3ea5f76700 1 -- 172.21.15.87:0/897322109 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(723..723 src has 173..723) v4 ==== 8543+0+0 (secure 0 0 0) 0x7f3e9c07e7f0 con 0x7f3ea80f8420 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.895+0000 7f3ea5f76700 10 client.38258.objecter ms_dispatch 0x7f3ea8001550 osd_map(723..723 src has 173..723) v4 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.895+0000 7f3ea5f76700 3 client.38258.objecter handle_osd_map got epochs [723,723] > 0 2024-02-23T00:42:25.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.895+0000 7f3ea5f76700 3 client.38258.objecter handle_osd_map decoding full epoch 723 2024-02-23T00:42:25.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.895+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/897322109 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 0x7f3e98065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:25.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.895+0000 7f3ea5f76700 20 client.38258.objecter dump_active .. 0 homeless 2024-02-23T00:42:25.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.895+0000 7f3ea5f76700 10 client.38258.objecter ms_handle_connect 0x7f3e98062c90 2024-02-23T00:42:25.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.895+0000 7f3ea7fff700 1 --2- 172.21.15.87:0/897322109 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 0x7f3e98065150 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7f3ea8109a20 tx=0x7f3e9400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:25.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:25.899+0000 7f3ea5f76700 1 -- 172.21.15.87:0/897322109 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3e9c051ae0 con 0x7f3ea80f8420 2024-02-23T00:42:25.929 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:42:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "size": 20, 2024-02-23T00:42:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 600, 2024-02-23T00:42:25.929 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [ 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.44587.0:1 151.6s0 151:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e219)", 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 575.887870991, 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763, 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.44587", 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/843545378", 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.930 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038319+0000", 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038316+0000", 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.9619999999999999e-06 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.931 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038325+0000", 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7385e-05 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038343+0000", 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.0739999999999999e-06 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.038352+0000", 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077681699999999996 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.932 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039128+0000", 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0860000000000002e-06 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:50.039132+0000", 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0036840789999999 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042816+0000", 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000158937 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.933 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.042975+0000", 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000568418 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.043543+0000", 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00061003399999999997 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.044153+0000", 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011287789999999999 2024-02-23T00:42:25.934 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045282+0000", 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9601e-05 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:32:51.045351+0000", 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.007032763 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.935 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26886.0:1 181.5s0 181:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e278)", 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 516.87986874700005, 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133, 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26886", 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/134760323", 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:42:25.936 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046321+0000", 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999976 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.937 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046318+0000", 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9741999999999998e-05 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046338+0000", 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.2594e-05 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046351+0000", 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5399999999999996e-06 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.938 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.046359+0000", 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00098343299999999992 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047343+0000", 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9160000000000003e-06 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:49.047347+0000", 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0029936850000001 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.939 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050340+0000", 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014100300000000001 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050481+0000", 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00038932099999999998 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.050871+0000", 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00045121199999999999 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.940 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.051322+0000", 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00089874699999999998 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052221+0000", 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1542999999999998e-05 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:33:50.052272+0000", 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.00595133 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.941 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27003.0:1 194.5s0 194:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e323)", 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 471.86571504900002, 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568, 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27003", 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3963489775", 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.942 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060475+0000", 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060473+0000", 2024-02-23T00:42:25.943 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.417e-06 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060479+0000", 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8080000000000005e-06 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060486+0000", 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.4359999999999999e-06 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.944 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060493+0000", 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6943999999999997e-05 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060530+0000", 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.4229999999999999e-06 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:34.060532+0000", 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.99955465700000001 2024-02-23T00:42:25.945 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060087+0000", 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2388999999999999e-05 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060179+0000", 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00037435399999999997 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:42:25.946 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.060554+0000", 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00057048899999999998 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061124+0000", 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00075843999999999998 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061883+0000", 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8547999999999998e-05 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.947 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:35.061921+0000", 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.001446568 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.37526.0:1 199.2s0 199:482bd08b:::oid_SimpleStatPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e347)", 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 447.84796720000003, 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.011307347, 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:25.948 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.37526", 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/4086862305", 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078222+0000", 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.949 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078221+0000", 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.1790000000000008e-06 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078229+0000", 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1657e-05 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078240+0000", 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.5909999999999996e-06 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.950 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.078249+0000", 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0011604219999999999 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.079409+0000", 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0110000000000001e-06 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for peered", 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:58.079412+0000", 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0080054119999999 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.951 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087418+0000", 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00014009999999999999 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087558+0000", 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00028455700000000001 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.087842+0000", 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00026429900000000001 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.952 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.088107+0000", 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0013807730000000001 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.089488+0000", 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2237000000000002e-05 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:34:59.089530+0000", 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.011307347 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.953 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:6796 63.0 63:04a528c6:test-rados-api-smithi087-82091-72::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e458)", 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:37:32.149684+0000", 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 293.776505364, 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030257736, 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 6796 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.954 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149684+0000", 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149684+0000", 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149683+0000", 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5489999999999995e-06 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.955 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149692+0000", 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3900000000000001e-05 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149706+0000", 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.9720000000000003e-06 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.956 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149713+0000", 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.2049000000000003e-05 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149765+0000", 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5885999999999996e-05 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.149861+0000", 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.025043958000000002 2024-02-23T00:42:25.957 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.174905+0000", 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00077345500000000004 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.175678+0000", 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0041288080000000003 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.179807+0000", 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6297000000000001e-05 2024-02-23T00:42:25.958 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.179853+0000", 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8549999999999998e-05 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:37:32.179942+0000", 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030257736 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.959 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:92 63.18 63:1d0125d1:test-rados-api-smithi087-82091-78::732be9d7c0f985174f3dbff4d0300cd89f82a6ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309667+0000", 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 245.61652303, 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030556142000000001, 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309667+0000", 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.960 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309667+0000", 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309665+0000", 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.038e-06 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309673+0000", 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1586999999999999e-05 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.961 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309684+0000", 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0350000000000004e-06 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309692+0000", 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000105009 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309797+0000", 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012772900000000001 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.962 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309924+0000", 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0039668359999999996 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.313891+0000", 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.014977572 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.328869+0000", 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00024878600000000003 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.963 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.329117+0000", 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011034924999999999 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340152+0000", 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7619e-05 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340190+0000", 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2716000000000003e-05 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.964 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340223+0000", 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030556142000000001 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:95 63.0 63:03bea9d3:test-rados-api-smithi087-82091-78::e98f1f7dcd528309122e8d5522249bd2aff3fd83:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 245.61647957599999, 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030136591000000001, 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.965 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:42:25.966 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309710+0000", 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0749999999999999e-06 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309714+0000", 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0990000000000004e-06 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309720+0000", 2024-02-23T00:42:25.967 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6360000000000002e-06 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309724+0000", 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00097421499999999996 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.310699+0000", 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00012222900000000001 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.968 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.310821+0000", 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00020901700000000001 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311030+0000", 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0075810119999999998 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.318611+0000", 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021088788000000001 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339700+0000", 2024-02-23T00:42:25.969 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.3208000000000003e-05 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339753+0000", 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.3796999999999997e-05 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339847+0000", 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030136591000000001 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.970 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:96 63.13 63:cc844047:test-rados-api-smithi087-82091-78::4b02657dec822853c72fefcaa266fff97efe7078:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309730+0000", 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 245.616459969, 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031035795000000001, 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309730+0000", 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.971 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309730+0000", 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309729+0000", 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.2880000000000001e-06 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309732+0000", 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.0950000000000003e-06 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.972 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309738+0000", 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.2869999999999998e-06 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309742+0000", 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.4709000000000001e-05 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309797+0000", 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000127417 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.973 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309924+0000", 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000217624 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,5", 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.310142+0000", 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0096070679999999999 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.319749+0000", 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0099226969999999994 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.974 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.329671+0000", 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.011011036 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340682+0000", 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5581000000000003e-05 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340728+0000", 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.7443e-05 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.975 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340765+0000", 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.031035795000000001 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(osd.6.44:100 63.0 63:0389d7c9:test-rados-api-smithi087-82091-78::b277a617178f08ca242dc7700d86bf23f0f3ea67:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.309754+0000", 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 245.61643600900001, 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030390868000000001, 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.976 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309754+0000", 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309754+0000", 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.977 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309753+0000", 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.8859999999999998e-06 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309760+0000", 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1314000000000003e-05 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309821+0000", 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.899e-06 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.309826+0000", 2024-02-23T00:42:25.978 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.001460601 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311287+0000", 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00011481600000000001 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311402+0000", 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00021134800000000001 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 6", 2024-02-23T00:42:25.979 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311613+0000", 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0070281809999999997 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.318641+0000", 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.021369968 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340011+0000", 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6655e-05 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.980 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340058+0000", 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6738999999999997e-05 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.340144+0000", 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.030390868000000001 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.6.44:97 63.1d e483/330 63:bb7c65c5:test-rados-api-smithi087-82091-78::c5740e5a8abdfdd24173d1f0746ed7b8f004fd8c:head v 483'14, mlcod=340'12)", 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:20.311169+0000", 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 245.61502048599999, 2024-02-23T00:42:25.981 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028548314000000002, 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311169+0000", 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311169+0000", 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.982 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311168+0000", 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.4179999999999997e-06 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311177+0000", 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.8430000000000005e-06 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311186+0000", 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2770000000000002e-06 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.983 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311193+0000", 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.3401000000000001e-05 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311226+0000", 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.1882000000000002e-05 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.311248+0000", 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028438485999999999 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.984 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339687+0000", 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.0916e-05 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:20.339717+0000", 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.028548314000000002 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:7287 221.f 221:f2233b45:test-rados-api-smithi087-82091-80::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e494)", 2024-02-23T00:42:25.985 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.218861+0000", 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 234.70732876, 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036222770000000001, 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 7287 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218861+0000", 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.986 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218861+0000", 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218860+0000", 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.1700000000000002e-06 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218866+0000", 2024-02-23T00:42:25.987 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0623e-05 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218877+0000", 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5566e-05 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218892+0000", 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5850999999999999e-05 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.988 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.218938+0000", 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.4792999999999999e-05 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.219033+0000", 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036050749999999999 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255084+0000", 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.036222770000000001 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.989 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop(osd.7.51:63 63.10 e494/425 63:0bcb77f5:test-rados-api-smithi087-82091-80::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head v 494'38, mlcod=0'0)", 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 234.70360223099999, 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026196710000000002, 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.990 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222587+0000", 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3339999999999996e-06 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222594+0000", 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.0679999999999999e-06 2024-02-23T00:42:25.991 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222602+0000", 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6450000000000004e-06 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222606+0000", 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.1100999999999997e-05 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222657+0000", 2024-02-23T00:42:25.992 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9842000000000001e-05 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.222677+0000", 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026066840000000001 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.248744+0000", 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0111000000000003e-05 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.248784+0000", 2024-02-23T00:42:25.993 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.026196710000000002 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.26224.0:7288 221.f 221:f2233b45:test-rados-api-smithi087-82091-80::foo:3 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e494)", 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.255388+0000", 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 234.67080136300001, 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041802895, 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.26224", 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:42:25.994 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 7288 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255388+0000", 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255388+0000", 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.995 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255387+0000", 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.5440000000000004e-06 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255393+0000", 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0249e-05 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255404+0000", 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7889999999999997e-06 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.996 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255411+0000", 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.2548999999999999e-05 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255483+0000", 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7108000000000006e-05 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.255580+0000", 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041610951 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.997 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297191+0000", 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.041802895 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_repop_reply(osd.7.0:543 221.f e494/487 ondisk, result = 0)", 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:31.267731+0000", 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 234.65845892600001, 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029799866000000001, 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:25.998 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267731+0000", 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267731+0000", 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.999999 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267730+0000", 2024-02-23T00:42:25.999 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7200000000000003e-06 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267736+0000", 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1474000000000001e-05 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267747+0000", 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.7590000000000003e-06 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.267753+0000", 2024-02-23T00:42:26.000 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029659342000000002 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297412+0000", 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.951e-06 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297419+0000", 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.000111462 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:31.297531+0000", 2024-02-23T00:42:26.001 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.029799866000000001 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "pg_backfill(finish_ack 4.1e e 515/515 lb MIN)", 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:52.183084+0000", 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.74310552099999, 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024878091000000001, 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:26.002 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183084+0000", 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183084+0000", 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183084+0000", 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.081e-06 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183093+0000", 2024-02-23T00:42:26.003 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.5926999999999998e-05 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183109+0000", 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.8450000000000003e-06 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183117+0000", 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.4298000000000002e-05 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.004 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183161+0000", 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.1062e-05 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183172+0000", 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024756987000000001 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207929+0000", 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.3018e-05 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.005 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207942+0000", 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 2.0004999999999999e-05 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207962+0000", 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024878091000000001 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "pg_backfill(progress 4.16 e 515/515 lb 4:6b0d4d95:lc::lc.20:head)", 2024-02-23T00:42:26.006 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:52.183360+0000", 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.74282995300001, 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024472534000000001, 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183360+0000", 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183360+0000", 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:42:26.007 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183359+0000", 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6789999999999994e-06 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183368+0000", 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0594000000000001e-05 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183379+0000", 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.2260000000000002e-06 2024-02-23T00:42:26.008 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183385+0000", 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.9660000000000003e-05 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183425+0000", 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.695e-06 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183432+0000", 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024336048999999998 2024-02-23T00:42:26.009 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207768+0000", 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4369000000000001e-05 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207783+0000", 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9452e-05 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207832+0000", 2024-02-23T00:42:26.010 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024472534000000001 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDPGPush(4.16 515/511 [PushOp(4:6d4431ed:lc::lc.7:head, version: 63'21, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d4431ed:lc::lc.7:head@63'21, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:38:52.183392+0000", 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 213.742797489, 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024578755000000001, 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:26.011 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183392+0000", 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183392+0000", 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967296 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183392+0000", 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.0757999999999999e-05 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.012 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183403+0000", 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.7390000000000005e-06 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183411+0000", 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9799999999999998e-06 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183416+0000", 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.6529999999999997e-05 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.013 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183463+0000", 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.5279999999999997e-06 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting_for_map not empty", 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.183467+0000", 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024400908999999998 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207868+0000", 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.7630000000000003e-06 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.014 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207875+0000", 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.5723000000000005e-05 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:38:52.207971+0000", 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.024578755000000001 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:474 234.7s0 234:fe98049f:test-rados-api-smithi087-82091-111::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e650)", 2024-02-23T00:42:26.015 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:09.210823+0000", 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 76.715367040999993, 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013635741, 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 474 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.210823+0000", 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:26.016 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.210823+0000", 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.210820+0000", 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.6170000000000007e-06 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.210829+0000", 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4452e-05 2024-02-23T00:42:26.017 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.210844+0000", 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.7890000000000006e-06 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.210853+0000", 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0038640419999999998 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.214717+0000", 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.2279999999999998e-06 2024-02-23T00:42:26.018 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.214726+0000", 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0028115369999999998 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.217537+0000", 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00015758999999999999 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.217695+0000", 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00050842699999999999 2024-02-23T00:42:26.019 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_started", 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.218203+0000", 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0029497870000000002 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_committed", 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.221153+0000", 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0032320320000000001 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.224385+0000", 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 7.3348000000000003e-05 2024-02-23T00:42:26.020 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:09.224458+0000", 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.013635741 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "MOSDScrubReserve(5.6 REQUEST e689)", 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:41:50.260024+0000", 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 35.666165538000001, 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0031849130000000002, 2024-02-23T00:42:26.021 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "started", 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.260024+0000", 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.260024+0000", 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.260024+0000", 2024-02-23T00:42:26.022 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6179999999999999e-06 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.260027+0000", 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7167999999999999e-05 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.260045+0000", 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.7309999999999999e-05 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.260062+0000", 2024-02-23T00:42:26.023 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.0541999999999997e-05 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.260122+0000", 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 6.3339999999999998e-06 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.260129+0000", 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.003023778 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.263153+0000", 2024-02-23T00:42:26.024 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.9562999999999999e-05 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.263172+0000", 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.6899e-05 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:41:50.263209+0000", 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0031849130000000002 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.025 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "description": "osd_op(client.27567.0:1528 245.6 245:6586c251:test-rados-api-smithi087-82091-123::foo:head [writefull 0~1] snapc 9=[9,8] ondisk+write+known_if_redirected e707)", 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "initiated_at": "2024-02-23T00:42:09.233473+0000", 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "age": 16.692716355000002, 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048959720000000002, 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "type_data": { 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "client_info": { 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "client": "client.27567", 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "tid": 1528 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "events": [ 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "initiated", 2024-02-23T00:42:26.026 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.233473+0000", 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "throttled", 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.233473+0000", 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4294967295.9999986 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "header_read", 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.233472+0000", 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 5.9220000000000002e-06 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.027 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "all_read", 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.233478+0000", 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 9.2390000000000008e-06 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "dispatched", 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.233487+0000", 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 1.4549000000000001e-05 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "queued_for_pg", 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.233502+0000", 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 3.8056999999999998e-05 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.028 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.233540+0000", 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.0740000000000003e-06 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op must wait for map", 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.233544+0000", 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.002751674 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "reached_pg", 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.236295+0000", 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 8.8032e-05 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.029 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "started", 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.236383+0000", 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00018780500000000001 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "waiting for subops from 2,4", 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.236571+0000", 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00066619800000000001 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "op_commit", 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.237237+0000", 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00056474300000000004 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.030 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.237802+0000", 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.00047010999999999998 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.238272+0000", 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.7406e-05 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "commit_sent", 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.238320+0000", 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 4.9545000000000001e-05 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: }, 2024-02-23T00:42:26.031 INFO:teuthology.orchestra.run.smithi171.stdout: { 2024-02-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi171.stdout: "event": "done", 2024-02-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi171.stdout: "time": "2024-02-23T00:42:09.238369+0000", 2024-02-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi171.stdout: "duration": 0.0048959720000000002 2024-02-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi171.stdout: } 2024-02-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi171.stdout: ] 2024-02-23T00:42:26.032 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:42:26.033 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:26.034 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:26.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:26.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.069+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ea805fa60 con 0x7f3e98062c90 2024-02-23T00:42:26.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.085+0000 7f3ea5f76700 1 -- 172.21.15.87:0/897322109 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+504436 (secure 0 0 0) 0x7f3ea805fa60 con 0x7f3e98062c90 2024-02-23T00:42:26.095 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 10 librados: watch_flush enter 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 10 librados: watch_flush exit 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 20 client.38258.objecter shutdown clearing up homeless session... 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 10 client.38258.objecter successfully canceled tick 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 msgr2=0x7f3e98065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/897322109 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 0x7f3e98065150 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 rx=0x7f3ea8109a20 tx=0x7f3e9400b040).stop 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80f8420 msgr2=0x7f3ea810e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/897322109 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80f8420 0x7f3ea810e0e0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 rx=0x7f3e9c0044e0 tx=0x7f3e9c007f90).stop 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 shutdown_connections 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/897322109 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3e98062c90 0x7f3e98065150 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/897322109 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3ea80fe510 0x7f3ea8108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/897322109 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3ea80f8420 0x7f3ea810e0e0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:26.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 --2- 172.21.15.87:0/897322109 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3ea80fdbb0 0x7f3ea810e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:26.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 >> 172.21.15.87:0/897322109 conn(0x7f3ea80f3800 msgr2=0x7f3ea80f53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:26.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 shutdown_connections 2024-02-23T00:42:26.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 -- 172.21.15.87:0/897322109 wait complete. 2024-02-23T00:42:26.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:26.089+0000 7f3eaeab3700 1 librados: shutdown 2024-02-23T00:42:26.101 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:42:26.135 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:26.167 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:26.206 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:26.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:26.310 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:26.340 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:26.408 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:42:26.539 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:26.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:25.244026+0000 mon.a (mon.0) 3650 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:26.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:25.244133+0000 mon.a (mon.0) 3651 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-02-23T00:42:26.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:25.260820+0000 mon.a (mon.0) 3652 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:26.542 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:25.244026+0000 mon.a (mon.0) 3650 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:26.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:25.244133+0000 mon.a (mon.0) 3651 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-02-23T00:42:26.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:25.260820+0000 mon.a (mon.0) 3652 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:26.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:26.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:25.244026+0000 mon.a (mon.0) 3650 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:26.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:25.244133+0000 mon.a (mon.0) 3651 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-02-23T00:42:26.634 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:25.260820+0000 mon.a (mon.0) 3652 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:26.645 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:26.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-23T00:42:26.672 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:26.921 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:26.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:27.024 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:27.051 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:27.500 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:27.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:25.721493+0000 osd.6 (osd.6) 294 : cluster [DBG] 2.17 deep-scrub starts 2024-02-23T00:42:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:25.724371+0000 osd.6 (osd.6) 295 : cluster [DBG] 2.17 deep-scrub ok 2024-02-23T00:42:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:25.782441+0000 mgr.y (mgr.24437) 616 : cluster [DBG] pgmap v1103: 194 pgs: 1 active+remapped+backfill_toofull, 5 unknown, 188 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:26.071117+0000 mgr.y (mgr.24437) 617 : audit [DBG] from='client.38258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:26.237662+0000 osd.3 (osd.3) 226 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:26.240643+0000 osd.3 (osd.3) 227 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:26.259547+0000 mon.a (mon.0) 3653 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:26.259590+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:26.260123+0000 mon.a (mon.0) 3655 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-126"}]: dispatch 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:26.306771+0000 osd.7 (osd.7) 326 : cluster [DBG] 5.1a deep-scrub starts 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:26.307627+0000 osd.7 (osd.7) 327 : cluster [DBG] 5.1a deep-scrub ok 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:26.379690+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.8 deep-scrub starts 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:26.380663+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.8 deep-scrub ok 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:26.420678+0000 osd.7 (osd.7) 328 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:26.420809+0000 osd.7 (osd.7) 329 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:27.219571+0000 mon.a (mon.0) 3656 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-126"}]': finished 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:27.219639+0000 mon.a (mon.0) 3657 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-02-23T00:42:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:27.222847+0000 mon.a (mon.0) 3658 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-126", "mode": "writeback"}]: dispatch 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:25.721493+0000 osd.6 (osd.6) 294 : cluster [DBG] 2.17 deep-scrub starts 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:25.724371+0000 osd.6 (osd.6) 295 : cluster [DBG] 2.17 deep-scrub ok 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:25.782441+0000 mgr.y (mgr.24437) 616 : cluster [DBG] pgmap v1103: 194 pgs: 1 active+remapped+backfill_toofull, 5 unknown, 188 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:26.071117+0000 mgr.y (mgr.24437) 617 : audit [DBG] from='client.38258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:26.237662+0000 osd.3 (osd.3) 226 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:26.240643+0000 osd.3 (osd.3) 227 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:26.259547+0000 mon.a (mon.0) 3653 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:26.259590+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:26.260123+0000 mon.a (mon.0) 3655 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-126"}]: dispatch 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:26.306771+0000 osd.7 (osd.7) 326 : cluster [DBG] 5.1a deep-scrub starts 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:26.307627+0000 osd.7 (osd.7) 327 : cluster [DBG] 5.1a deep-scrub ok 2024-02-23T00:42:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:26.379690+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.8 deep-scrub starts 2024-02-23T00:42:27.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:26.380663+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.8 deep-scrub ok 2024-02-23T00:42:27.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:26.420678+0000 osd.7 (osd.7) 328 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:27.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:26.420809+0000 osd.7 (osd.7) 329 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:27.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:27.219571+0000 mon.a (mon.0) 3656 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-126"}]': finished 2024-02-23T00:42:27.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:27.219639+0000 mon.a (mon.0) 3657 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-02-23T00:42:27.586 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:27.222847+0000 mon.a (mon.0) 3658 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-126", "mode": "writeback"}]: dispatch 2024-02-23T00:42:27.604 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:27.638 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:25.721493+0000 osd.6 (osd.6) 294 : cluster [DBG] 2.17 deep-scrub starts 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:25.724371+0000 osd.6 (osd.6) 295 : cluster [DBG] 2.17 deep-scrub ok 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:25.782441+0000 mgr.y (mgr.24437) 616 : cluster [DBG] pgmap v1103: 194 pgs: 1 active+remapped+backfill_toofull, 5 unknown, 188 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:26.071117+0000 mgr.y (mgr.24437) 617 : audit [DBG] from='client.38258 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:26.237662+0000 osd.3 (osd.3) 226 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:26.240643+0000 osd.3 (osd.3) 227 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:26.259547+0000 mon.a (mon.0) 3653 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:26.259590+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:26.260123+0000 mon.a (mon.0) 3655 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-126"}]: dispatch 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:26.306771+0000 osd.7 (osd.7) 326 : cluster [DBG] 5.1a deep-scrub starts 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:26.307627+0000 osd.7 (osd.7) 327 : cluster [DBG] 5.1a deep-scrub ok 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:26.379690+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.8 deep-scrub starts 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:26.380663+0000 osd.2 (osd.2) 266 : cluster [DBG] 5.8 deep-scrub ok 2024-02-23T00:42:27.639 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:26.420678+0000 osd.7 (osd.7) 328 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:27.640 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:26.420809+0000 osd.7 (osd.7) 329 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:27.640 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:27.219571+0000 mon.a (mon.0) 3656 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-126"}]': finished 2024-02-23T00:42:27.640 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:27.219639+0000 mon.a (mon.0) 3657 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-02-23T00:42:27.640 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:27.222847+0000 mon.a (mon.0) 3658 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-126", "mode": "writeback"}]: dispatch 2024-02-23T00:42:27.642 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:27.663 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:27.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:27.767 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:27.798 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:27.819 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:27.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:27.922 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:27.950 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:27.970 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:27.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:28.074 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:28.104 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:28.184 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 -- 172.21.15.87:0/4155593486 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 msgr2=0x7f30200fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 --2- 172.21.15.87:0/4155593486 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 0x7f30200fdf90 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 rx=0x7f30140047b0 tx=0x7f3014009cb0).stop 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 -- 172.21.15.87:0/4155593486 shutdown_connections 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 --2- 172.21.15.87:0/4155593486 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30200f8420 0x7f3020104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 --2- 172.21.15.87:0/4155593486 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 0x7f30200fdf90 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 --2- 172.21.15.87:0/4155593486 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30200fe510 0x7f30200f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 -- 172.21.15.87:0/4155593486 >> 172.21.15.87:0/4155593486 conn(0x7f30200f3800 msgr2=0x7f30200f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 -- 172.21.15.87:0/4155593486 shutdown_connections 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 -- 172.21.15.87:0/4155593486 wait complete. 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 librados: starting msgr at 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.180+0000 7f3024b00700 1 librados: starting objecter 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 Processor -- start 2024-02-23T00:42:28.185 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 -- start start 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 librados: setting wanted keys 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 librados: calling monclient init 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30200f8420 0x7f302006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 0x7f302006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30200fe510 0x7f302006bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30201165a0 con 0x7f30200fe510 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3020116710 con 0x7f30200f8420 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f3024b00700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30201168d0 con 0x7f30200fdbb0 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301dd9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 0x7f302006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301dd9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 0x7f302006b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50318/0 (socket says 172.21.15.87:50318) 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301dd9b700 1 -- 172.21.15.87:0/1264766427 learned_addr learned my addr 172.21.15.87:0/1264766427 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301ed9d700 1 --2- 172.21.15.87:0/1264766427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30200fe510 0x7f302006bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301e59c700 1 --2- 172.21.15.87:0/1264766427 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30200f8420 0x7f302006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301dd9b700 1 -- 172.21.15.87:0/1264766427 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30200f8420 msgr2=0x7f302006b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:28.186 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301dd9b700 1 --2- 172.21.15.87:0/1264766427 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30200f8420 0x7f302006b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301dd9b700 1 -- 172.21.15.87:0/1264766427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30200fe510 msgr2=0x7f302006bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301dd9b700 1 --2- 172.21.15.87:0/1264766427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30200fe510 0x7f302006bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301dd9b700 1 -- 172.21.15.87:0/1264766427 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3010005040 con 0x7f30200fdbb0 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f301dd9b700 1 --2- 172.21.15.87:0/1264766427 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 0x7f302006b5c0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f3010009400 tx=0x7f3010009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f300f7fe700 10 client.?.objecter ms_handle_connect 0x7f30200fdbb0 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.181+0000 7f300f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.182+0000 7f300f7fe700 1 -- 172.21.15.87:0/1264766427 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010007a40 con 0x7f30200fdbb0 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.182+0000 7f300f7fe700 1 -- 172.21.15.87:0/1264766427 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f301000f040 con 0x7f30200fdbb0 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.182+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3014005040 con 0x7f30200fdbb0 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.182+0000 7f300f7fe700 1 -- 172.21.15.87:0/1264766427 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010014a70 con 0x7f30200fdbb0 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.182+0000 7f3024b00700 10 client.38264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.182+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3020116f90 con 0x7f30200fdbb0 2024-02-23T00:42:28.187 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.182+0000 7f3024b00700 1 librados: init done 2024-02-23T00:42:28.188 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.183+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302005ee80 con 0x7f30200fdbb0 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.183+0000 7f300f7fe700 1 -- 172.21.15.87:0/1264766427 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3010010530 con 0x7f30200fdbb0 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.183+0000 7f300f7fe700 1 --2- 172.21.15.87:0/1264766427 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3008062c90 0x7f3008065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.183+0000 7f300f7fe700 1 -- 172.21.15.87:0/1264766427 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(725..725 src has 173..725) v4 ==== 8538+0+0 (secure 0 0 0) 0x7f301007cfa0 con 0x7f30200fdbb0 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.183+0000 7f300f7fe700 10 client.38264.objecter ms_dispatch 0x7f3020001550 osd_map(725..725 src has 173..725) v4 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.183+0000 7f300f7fe700 3 client.38264.objecter handle_osd_map got epochs [725,725] > 0 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.183+0000 7f300f7fe700 3 client.38264.objecter handle_osd_map decoding full epoch 725 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.183+0000 7f301e59c700 1 --2- 172.21.15.87:0/1264766427 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3008062c90 0x7f3008065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.184+0000 7f300f7fe700 20 client.38264.objecter dump_active .. 0 homeless 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.184+0000 7f300f7fe700 10 client.38264.objecter ms_handle_connect 0x7f3008062c90 2024-02-23T00:42:28.189 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.184+0000 7f301e59c700 1 --2- 172.21.15.87:0/1264766427 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3008062c90 0x7f3008065150 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7f3014005010 tx=0x7f3014007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:28.190 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.188+0000 7f300f7fe700 1 -- 172.21.15.87:0/1264766427 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3010049f60 con 0x7f30200fdbb0 2024-02-23T00:42:28.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:28.380+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f302005fa60 con 0x7f30200fdbb0 2024-02-23T00:42:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:27.194394+0000 osd.3 (osd.3) 228 : cluster [DBG] 5.13 deep-scrub starts 2024-02-23T00:42:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:27.197286+0000 osd.3 (osd.3) 229 : cluster [DBG] 5.13 deep-scrub ok 2024-02-23T00:42:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:27.909843+0000 osd.4 (osd.4) 321 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:27.911593+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:28.217218+0000 mon.a (mon.0) 3659 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:28.223760+0000 mon.a (mon.0) 3660 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-126", "mode": "writeback"}]': finished 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:28.223817+0000 mon.a (mon.0) 3661 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:27.194394+0000 osd.3 (osd.3) 228 : cluster [DBG] 5.13 deep-scrub starts 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:27.197286+0000 osd.3 (osd.3) 229 : cluster [DBG] 5.13 deep-scrub ok 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:27.909843+0000 osd.4 (osd.4) 321 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:27.911593+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:28.217218+0000 mon.a (mon.0) 3659 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:28.223760+0000 mon.a (mon.0) 3660 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-126", "mode": "writeback"}]': finished 2024-02-23T00:42:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:28.223817+0000 mon.a (mon.0) 3661 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-02-23T00:42:28.597 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:28.598 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:28.618 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1103,"stamp":"2024-02-23T00:42:25.781854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5937,"num_objects":247,"num_object_clones":1,"num_object_copies":741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":0,"num_read":1498,"num_read_kb":1322,"num_write":574,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":7883,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":897,"ondisk_log_size":897,"up":567,"acting":567,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":569,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2502476,"kb_used_data":73860,"kb_used_omap":85,"kb_used_meta":2428394,"kb_avail":747458740,"statfs":{"total":767960285184,"available":765397749760,"internally_reserved":0,"allocated":75632640,"data_stored":43420243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":2486675584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-38,"num_read_kb":-30,"num_write":-32,"num_write_kb":-5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-16,"num_bytes_recovered":-275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":-5,"num_flush_kb":-5,"num_evict":0,"num_evict_kb":0,"num_promote":-1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000960"},"pg_stats":[{"pgid":"247.17","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.258555+0000","last_change":"2024-02-23T00:42:25.258555+0000","last_active":"2024-02-23T00:42:25.258555+0000","last_peered":"2024-02-23T00:42:25.258555+0000","last_clean":"2024-02-23T00:42:25.258555+0000","last_became_active":"2024-02-23T00:42:25.258281+0000","last_became_peered":"2024-02-23T00:42:25.258281+0000","last_unstale":"2024-02-23T00:42:25.258555+0000","last_undegraded":"2024-02-23T00:42:25.258555+0000","last_fullsized":"2024-02-23T00:42:25.258555+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"247.16","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.257773+0000","last_change":"2024-02-23T00:42:25.257773+0000","last_active":"2024-02-23T00:42:25.257773+0000","last_peered":"2024-02-23T00:42:25.257773+0000","last_clean":"2024-02-23T00:42:25.257773+0000","last_became_active":"2024-02-23T00:42:25.257491+0000","last_became_peered":"2024-02-23T00:42:25.257491+0000","last_unstale":"2024-02-23T00:42:25.257773+0000","last_undegraded":"2024-02-23T00:42:25.257773+0000","last_fullsized":"2024-02-23T00:42:25.257773+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.15","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.259300+0000","last_change":"2024-02-23T00:42:25.259300+0000","last_active":"2024-02-23T00:42:25.259300+0000","last_peered":"2024-02-23T00:42:25.259300+0000","last_clean":"2024-02-23T00:42:25.259300+0000","last_became_active":"2024-02-23T00:42:25.258998+0000","last_became_peered":"2024-02-23T00:42:25.258998+0000","last_unstale":"2024-02-23T00:42:25.259300+0000","last_undegraded":"2024-02-23T00:42:25.259300+0000","last_fullsized":"2024-02-23T00:42:25.259300+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"247.14","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.251584+0000","last_change":"2024-02-23T00:42:25.251584+0000","last_active":"2024-02-23T00:42:25.251584+0000","last_peered":"2024-02-23T00:42:25.251584+0000","last_clean":"2024-02-23T00:42:25.251584+0000","last_became_active":"2024-02-23T00:42:25.251273+0000","last_became_peered":"2024-02-23T00:42:25.251273+0000","last_unstale":"2024-02-23T00:42:25.251584+0000","last_undegraded":"2024-02-23T00:42:25.251584+0000","last_fullsized":"2024-02-23T00:42:25.251584+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:42:24.232704+0000","last_change":"2024-02-23T00:42:24.232704+0000","last_active":"2024-02-23T00:42:24.232704+0000","last_peered":"2024-02-23T00:42:24.232704+0000","last_clean":"2024-02-23T00:42:24.232704+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:42:24.232704+0000","last_undegraded":"2024-02-23T00:42:24.232704+0000","last_fullsized":"2024-02-23T00:42:24.232704+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"247.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:42:24.232704+0000","last_change":"2024-02-23T00:42:24.232704+0000","last_active":"2024-02-23T00:42:24.232704+0000","last_peered":"2024-02-23T00:42:24.232704+0000","last_clean":"2024-02-23T00:42:24.232704+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:42:24.232704+0000","last_undegraded":"2024-02-23T00:42:24.232704+0000","last_fullsized":"2024-02-23T00:42:24.232704+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"247.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:42:24.232704+0000","last_change":"2024-02-23T00:42:24.232704+0000","last_active":"2024-02-23T00:42:24.232704+0000","last_peered":"2024-02-23T00:42:24.232704+0000","last_clean":"2024-02-23T00:42:24.232704+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:42:24.232704+0000","last_undegraded":"2024-02-23T00:42:24.232704+0000","last_fullsized":"2024-02-23T00:42:24.232704+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"247.10","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.261162+0000","last_change":"2024-02-23T00:42:25.261162+0000","last_active":"2024-02-23T00:42:25.261162+0000","last_peered":"2024-02-23T00:42:25.261162+0000","last_clean":"2024-02-23T00:42:25.261162+0000","last_became_active":"2024-02-23T00:42:25.260841+0000","last_became_peered":"2024-02-23T00:42:25.260841+0000","last_unstale":"2024-02-23T00:42:25.261162+0000","last_undegraded":"2024-02-23T00:42:25.261162+0000","last_fullsized":"2024-02-23T00:42:25.261162+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"247.f","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.261467+0000","last_change":"2024-02-23T00:42:25.261467+0000","last_active":"2024-02-23T00:42:25.261467+0000","last_peered":"2024-02-23T00:42:25.261467+0000","last_clean":"2024-02-23T00:42:25.261467+0000","last_became_active":"2024-02-23T00:42:25.261164+0000","last_became_peered":"2024-02-23T00:42:25.261164+0000","last_unstale":"2024-02-23T00:42:25.261467+0000","last_undegraded":"2024-02-23T00:42:25.261467+0000","last_fullsized":"2024-02-23T00:42:25.261467+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"247.e","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.264234+0000","last_change":"2024-02-23T00:42:25.264234+0000","last_active":"2024-02-23T00:42:25.264234+0000","last_peered":"2024-02-23T00:42:25.264234+0000","last_clean":"2024-02-23T00:42:25.264234+0000","last_became_active":"2024-02-23T00:42:25.263973+0000","last_became_peered":"2024-02-23T00:42:25.263973+0000","last_unstale":"2024-02-23T00:42:25.264234+0000","last_undegraded":"2024-02-23T00:42:25.264234+0000","last_fullsized":"2024-02-23T00:42:25.264234+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"247.d","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.261199+0000","last_change":"2024-02-23T00:42:25.261199+0000","last_active":"2024-02-23T00:42:25.261199+0000","last_peered":"2024-02-23T00:42:25.261199+0000","last_clean":"2024-02-23T00:42:25.261199+0000","last_became_active":"2024-02-23T00:42:25.260962+0000","last_became_peered":"2024-02-23T00:42:25.260962+0000","last_unstale":"2024-02-23T00:42:25.261199+0000","last_undegraded":"2024-02-23T00:42:25.261199+0000","last_fullsized":"2024-02-23T00:42:25.261199+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"247.c","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.263047+0000","last_change":"2024-02-23T00:42:25.263047+0000","last_active":"2024-02-23T00:42:25.263047+0000","last_peered":"2024-02-23T00:42:25.263047+0000","last_clean":"2024-02-23T00:42:25.263047+0000","last_became_active":"2024-02-23T00:42:25.262743+0000","last_became_peered":"2024-02-23T00:42:25.262743+0000","last_unstale":"2024-02-23T00:42:25.263047+0000","last_undegraded":"2024-02-23T00:42:25.263047+0000","last_fullsized":"2024-02-23T00:42:25.263047+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"247.b","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.257381+0000","last_change":"2024-02-23T00:42:25.257381+0000","last_active":"2024-02-23T00:42:25.257381+0000","last_peered":"2024-02-23T00:42:25.257381+0000","last_clean":"2024-02-23T00:42:25.257381+0000","last_became_active":"2024-02-23T00:42:25.257090+0000","last_became_peered":"2024-02-23T00:42:25.257090+0000","last_unstale":"2024-02-23T00:42:25.257381+0000","last_undegraded":"2024-02-23T00:42:25.257381+0000","last_fullsized":"2024-02-23T00:42:25.257381+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"247.a","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.254432+0000","last_change":"2024-02-23T00:42:25.254432+0000","last_active":"2024-02-23T00:42:25.254432+0000","last_peered":"2024-02-23T00:42:25.254432+0000","last_clean":"2024-02-23T00:42:25.254432+0000","last_became_active":"2024-02-23T00:42:25.253964+0000","last_became_peered":"2024-02-23T00:42:25.253964+0000","last_unstale":"2024-02-23T00:42:25.254432+0000","last_undegraded":"2024-02-23T00:42:25.254432+0000","last_fullsized":"2024-02-23T00:42:25.254432+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.9","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.262099+0000","last_change":"2024-02-23T00:42:25.262099+0000","last_active":"2024-02-23T00:42:25.262099+0000","last_peered":"2024-02-23T00:42:25.262099+0000","last_clean":"2024-02-23T00:42:25.262099+0000","last_became_active":"2024-02-23T00:42:25.261814+0000","last_became_peered":"2024-02-23T00:42:25.261814+0000","last_unstale":"2024-02-23T00:42:25.262099+0000","last_undegraded":"2024-02-23T00:42:25.262099+0000","last_fullsized":"2024-02-23T00:42:25.262099+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"247.8","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.257514+0000","last_change":"2024-02-23T00:42:25.257514+0000","last_active":"2024-02-23T00:42:25.257514+0000","last_peered":"2024-02-23T00:42:25.257514+0000","last_clean":"2024-02-23T00:42:25.257514+0000","last_became_active":"2024-02-23T00:42:25.257188+0000","last_became_peered":"2024-02-23T00:42:25.257188+0000","last_unstale":"2024-02-23T00:42:25.257514+0000","last_undegraded":"2024-02-23T00:42:25.257514+0000","last_fullsized":"2024-02-23T00:42:25.257514+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"247.7","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.258606+0000","last_change":"2024-02-23T00:42:25.258606+0000","last_active":"2024-02-23T00:42:25.258606+0000","last_peered":"2024-02-23T00:42:25.258606+0000","last_clean":"2024-02-23T00:42:25.258606+0000","last_became_active":"2024-02-23T00:42:25.258273+0000","last_became_peered":"2024-02-23T00:42:25.258273+0000","last_unstale":"2024-02-23T00:42:25.258606+0000","last_undegraded":"2024-02-23T00:42:25.258606+0000","last_fullsized":"2024-02-23T00:42:25.258606+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:42:24.232704+0000","last_change":"2024-02-23T00:42:24.232704+0000","last_active":"2024-02-23T00:42:24.232704+0000","last_peered":"2024-02-23T00:42:24.232704+0000","last_clean":"2024-02-23T00:42:24.232704+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:42:24.232704+0000","last_undegraded":"2024-02-23T00:42:24.232704+0000","last_fullsized":"2024-02-23T00:42:24.232704+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"247.5","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.255965+0000","last_change":"2024-02-23T00:42:25.255965+0000","last_active":"2024-02-23T00:42:25.255965+0000","last_peered":"2024-02-23T00:42:25.255965+0000","last_clean":"2024-02-23T00:42:25.255965+0000","last_became_active":"2024-02-23T00:42:25.255671+0000","last_became_peered":"2024-02-23T00:42:25.255671+0000","last_unstale":"2024-02-23T00:42:25.255965+0000","last_undegraded":"2024-02-23T00:42:25.255965+0000","last_fullsized":"2024-02-23T00:42:25.255965+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"247.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-23T00:42:24.232704+0000","last_change":"2024-02-23T00:42:24.232704+0000","last_active":"2024-02-23T00:42:24.232704+0000","last_peered":"2024-02-23T00:42:24.232704+0000","last_clean":"2024-02-23T00:42:24.232704+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-23T00:42:24.232704+0000","last_undegraded":"2024-02-23T00:42:24.232704+0000","last_fullsized":"2024-02-23T00:42:24.232704+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"247.3","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.258489+0000","last_change":"2024-02-23T00:42:25.258489+0000","last_active":"2024-02-23T00:42:25.258489+0000","last_peered":"2024-02-23T00:42:25.258489+0000","last_clean":"2024-02-23T00:42:25.258489+0000","last_became_active":"2024-02-23T00:42:25.258230+0000","last_became_peered":"2024-02-23T00:42:25.258230+0000","last_unstale":"2024-02-23T00:42:25.258489+0000","last_undegraded":"2024-02-23T00:42:25.258489+0000","last_fullsized":"2024-02-23T00:42:25.258489+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"247.2","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.250106+0000","last_change":"2024-02-23T00:42:25.250106+0000","last_active":"2024-02-23T00:42:25.250106+0000","last_peered":"2024-02-23T00:42:25.250106+0000","last_clean":"2024-02-23T00:42:25.250106+0000","last_became_active":"2024-02-23T00:42:25.249861+0000","last_became_peered":"2024-02-23T00:42:25.249861+0000","last_unstale":"2024-02-23T00:42:25.250106+0000","last_undegraded":"2024-02-23T00:42:25.250106+0000","last_fullsized":"2024-02-23T00:42:25.250106+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"247.1","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.262078+0000","last_change":"2024-02-23T00:42:25.262078+0000","last_active":"2024-02-23T00:42:25.262078+0000","last_peered":"2024-02-23T00:42:25.262078+0000","last_clean":"2024-02-23T00:42:25.262078+0000","last_became_active":"2024-02-23T00:42:25.261807+0000","last_became_peered":"2024-02-23T00:42:25.261807+0000","last_unstale":"2024-02-23T00:42:25.262078+0000","last_undegraded":"2024-02-23T00:42:25.262078+0000","last_fullsized":"2024-02-23T00:42:25.262078+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"247.0","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.256984+0000","last_change":"2024-02-23T00:42:25.256984+0000","last_active":"2024-02-23T00:42:25.256984+0000","last_peered":"2024-02-23T00:42:25.256984+0000","last_clean":"2024-02-23T00:42:25.256984+0000","last_became_active":"2024-02-23T00:42:25.256699+0000","last_became_peered":"2024-02-23T00:42:25.256699+0000","last_unstale":"2024-02-23T00:42:25.256984+0000","last_undegraded":"2024-02-23T00:42:25.256984+0000","last_fullsized":"2024-02-23T00:42:25.256984+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.1a","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.255864+0000","last_change":"2024-02-23T00:42:25.255864+0000","last_active":"2024-02-23T00:42:25.255864+0000","last_peered":"2024-02-23T00:42:25.255864+0000","last_clean":"2024-02-23T00:42:25.255864+0000","last_became_active":"2024-02-23T00:42:25.255565+0000","last_became_peered":"2024-02-23T00:42:25.255565+0000","last_unstale":"2024-02-23T00:42:25.255864+0000","last_undegraded":"2024-02-23T00:42:25.255864+0000","last_fullsized":"2024-02-23T00:42:25.255864+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.7","version":"689'9","reported_seq":172,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249919+0000","last_change":"2024-02-23T00:42:21.267470+0000","last_active":"2024-02-23T00:42:24.249919+0000","last_peered":"2024-02-23T00:42:24.249919+0000","last_clean":"2024-02-23T00:42:24.249919+0000","last_became_active":"2024-02-23T00:40:17.211405+0000","last_became_peered":"2024-02-23T00:40:17.211405+0000","last_unstale":"2024-02-23T00:42:24.249919+0000","last_undegraded":"2024-02-23T00:42:24.249919+0000","last_fullsized":"2024-02-23T00:42:24.249919+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'9","last_scrub_stamp":"2024-02-23T00:42:21.267354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:21.267354+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"247.1b","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.258185+0000","last_change":"2024-02-23T00:42:25.258185+0000","last_active":"2024-02-23T00:42:25.258185+0000","last_peered":"2024-02-23T00:42:25.258185+0000","last_clean":"2024-02-23T00:42:25.258185+0000","last_became_active":"2024-02-23T00:42:25.257896+0000","last_became_peered":"2024-02-23T00:42:25.257896+0000","last_unstale":"2024-02-23T00:42:25.258185+0000","last_undegraded":"2024-02-23T00:42:25.258185+0000","last_fullsized":"2024-02-23T00:42:25.258185+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.6","version":"710'16","reported_seq":181,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245027+0000","last_change":"2024-02-23T00:42:14.256795+0000","last_active":"2024-02-23T00:42:24.245027+0000","last_peered":"2024-02-23T00:42:24.245027+0000","last_clean":"2024-02-23T00:42:24.245027+0000","last_became_active":"2024-02-23T00:40:17.215263+0000","last_became_peered":"2024-02-23T00:40:17.215263+0000","last_unstale":"2024-02-23T00:42:24.245027+0000","last_undegraded":"2024-02-23T00:42:24.245027+0000","last_fullsized":"2024-02-23T00:42:24.245027+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'8","last_scrub_stamp":"2024-02-23T00:41:36.605800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:41:36.605800+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"247.18","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.259686+0000","last_change":"2024-02-23T00:42:25.259686+0000","last_active":"2024-02-23T00:42:25.259686+0000","last_peered":"2024-02-23T00:42:25.259686+0000","last_clean":"2024-02-23T00:42:25.259686+0000","last_became_active":"2024-02-23T00:42:25.259402+0000","last_became_peered":"2024-02-23T00:42:25.259402+0000","last_unstale":"2024-02-23T00:42:25.259686+0000","last_undegraded":"2024-02-23T00:42:25.259686+0000","last_fullsized":"2024-02-23T00:42:25.259686+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.5","version":"635'5","reported_seq":185,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246693+0000","last_change":"2024-02-23T00:42:13.431865+0000","last_active":"2024-02-23T00:42:24.246693+0000","last_peered":"2024-02-23T00:42:24.246693+0000","last_clean":"2024-02-23T00:42:24.246693+0000","last_became_active":"2024-02-23T00:41:37.238894+0000","last_became_peered":"2024-02-23T00:41:37.238894+0000","last_unstale":"2024-02-23T00:42:24.246693+0000","last_undegraded":"2024-02-23T00:42:24.246693+0000","last_fullsized":"2024-02-23T00:42:24.246693+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"635'5","last_scrub_stamp":"2024-02-23T00:42:13.431791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.431791+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.19","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.259763+0000","last_change":"2024-02-23T00:42:25.259763+0000","last_active":"2024-02-23T00:42:25.259763+0000","last_peered":"2024-02-23T00:42:25.259763+0000","last_clean":"2024-02-23T00:42:25.259763+0000","last_became_active":"2024-02-23T00:42:25.259463+0000","last_became_peered":"2024-02-23T00:42:25.259463+0000","last_unstale":"2024-02-23T00:42:25.259763+0000","last_undegraded":"2024-02-23T00:42:25.259763+0000","last_fullsized":"2024-02-23T00:42:25.259763+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.4","version":"718'20","reported_seq":178,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250068+0000","last_change":"2024-02-23T00:42:12.255652+0000","last_active":"2024-02-23T00:42:24.250068+0000","last_peered":"2024-02-23T00:42:24.250068+0000","last_clean":"2024-02-23T00:42:24.250068+0000","last_became_active":"2024-02-23T00:40:17.215235+0000","last_became_peered":"2024-02-23T00:40:17.215235+0000","last_unstale":"2024-02-23T00:42:24.250068+0000","last_undegraded":"2024-02-23T00:42:24.250068+0000","last_fullsized":"2024-02-23T00:42:24.250068+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"247.1e","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.262109+0000","last_change":"2024-02-23T00:42:25.262109+0000","last_active":"2024-02-23T00:42:25.262109+0000","last_peered":"2024-02-23T00:42:25.262109+0000","last_clean":"2024-02-23T00:42:25.262109+0000","last_became_active":"2024-02-23T00:42:25.261842+0000","last_became_peered":"2024-02-23T00:42:25.261842+0000","last_unstale":"2024-02-23T00:42:25.262109+0000","last_undegraded":"2024-02-23T00:42:25.262109+0000","last_fullsized":"2024-02-23T00:42:25.262109+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.3","version":"697'2","reported_seq":161,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246864+0000","last_change":"2024-02-23T00:42:18.042228+0000","last_active":"2024-02-23T00:42:24.246864+0000","last_peered":"2024-02-23T00:42:24.246864+0000","last_clean":"2024-02-23T00:42:24.246864+0000","last_became_active":"2024-02-23T00:40:17.202244+0000","last_became_peered":"2024-02-23T00:40:17.202244+0000","last_unstale":"2024-02-23T00:42:24.246864+0000","last_undegraded":"2024-02-23T00:42:24.246864+0000","last_fullsized":"2024-02-23T00:42:24.246864+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'2","last_scrub_stamp":"2024-02-23T00:42:18.042156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.042156+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"247.1f","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.256745+0000","last_change":"2024-02-23T00:42:25.256745+0000","last_active":"2024-02-23T00:42:25.256745+0000","last_peered":"2024-02-23T00:42:25.256745+0000","last_clean":"2024-02-23T00:42:25.256745+0000","last_became_active":"2024-02-23T00:42:25.256437+0000","last_became_peered":"2024-02-23T00:42:25.256437+0000","last_unstale":"2024-02-23T00:42:25.256745+0000","last_undegraded":"2024-02-23T00:42:25.256745+0000","last_fullsized":"2024-02-23T00:42:25.256745+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.2","version":"679'6","reported_seq":187,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244290+0000","last_change":"2024-02-23T00:42:12.249043+0000","last_active":"2024-02-23T00:42:24.244290+0000","last_peered":"2024-02-23T00:42:24.244290+0000","last_clean":"2024-02-23T00:42:24.244290+0000","last_became_active":"2024-02-23T00:41:37.219553+0000","last_became_peered":"2024-02-23T00:41:37.219553+0000","last_unstale":"2024-02-23T00:42:24.244290+0000","last_undegraded":"2024-02-23T00:42:24.244290+0000","last_fullsized":"2024-02-23T00:42:24.244290+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"679'6","last_scrub_stamp":"2024-02-23T00:41:43.619119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:41:43.619119+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"247.1c","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.261183+0000","last_change":"2024-02-23T00:42:25.261183+0000","last_active":"2024-02-23T00:42:25.261183+0000","last_peered":"2024-02-23T00:42:25.261183+0000","last_clean":"2024-02-23T00:42:25.261183+0000","last_became_active":"2024-02-23T00:42:25.260948+0000","last_became_peered":"2024-02-23T00:42:25.260948+0000","last_unstale":"2024-02-23T00:42:25.261183+0000","last_undegraded":"2024-02-23T00:42:25.261183+0000","last_fullsized":"2024-02-23T00:42:25.261183+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.1","version":"607'1","reported_seq":175,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249187+0000","last_change":"2024-02-23T00:42:12.251737+0000","last_active":"2024-02-23T00:42:24.249187+0000","last_peered":"2024-02-23T00:42:24.249187+0000","last_clean":"2024-02-23T00:42:24.249187+0000","last_became_active":"2024-02-23T00:41:37.231330+0000","last_became_peered":"2024-02-23T00:41:37.231330+0000","last_unstale":"2024-02-23T00:42:24.249187+0000","last_undegraded":"2024-02-23T00:42:24.249187+0000","last_fullsized":"2024-02-23T00:42:24.249187+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"247.1d","version":"0'0","reported_seq":9,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.258149+0000","last_change":"2024-02-23T00:42:25.258149+0000","last_active":"2024-02-23T00:42:25.258149+0000","last_peered":"2024-02-23T00:42:25.258149+0000","last_clean":"2024-02-23T00:42:25.258149+0000","last_became_active":"2024-02-23T00:42:25.257858+0000","last_became_peered":"2024-02-23T00:42:25.257858+0000","last_unstale":"2024-02-23T00:42:25.258149+0000","last_undegraded":"2024-02-23T00:42:25.258149+0000","last_fullsized":"2024-02-23T00:42:25.258149+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":722,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:24.232704+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.232704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.0","version":"671'10","reported_seq":158,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244168+0000","last_change":"2024-02-23T00:42:12.248572+0000","last_active":"2024-02-23T00:42:24.244168+0000","last_peered":"2024-02-23T00:42:24.244168+0000","last_clean":"2024-02-23T00:42:24.244168+0000","last_became_active":"2024-02-23T00:41:50.253705+0000","last_became_peered":"2024-02-23T00:41:50.253705+0000","last_unstale":"2024-02-23T00:42:24.244168+0000","last_undegraded":"2024-02-23T00:42:24.244168+0000","last_fullsized":"2024-02-23T00:42:24.244168+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"671'10","last_scrub_stamp":"2024-02-23T00:41:50.664981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:41:50.664981+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"571'11","reported_seq":921,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.249694+0000","last_change":"2024-02-23T00:42:18.360637+0000","last_active":"2024-02-23T00:42:25.249694+0000","last_peered":"2024-02-23T00:42:25.249694+0000","last_clean":"2024-02-23T00:42:25.249694+0000","last_became_active":"2024-02-23T00:39:49.195036+0000","last_became_peered":"2024-02-23T00:39:49.195036+0000","last_unstale":"2024-02-23T00:42:25.249694+0000","last_undegraded":"2024-02-23T00:42:25.249694+0000","last_fullsized":"2024-02-23T00:42:25.249694+0000","mapping_epoch":58,"log_start":"379'9","ondisk_log_start":"379'9","created":375,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'11","last_scrub_stamp":"2024-02-23T00:42:18.360544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.360544+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":814,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246669+0000","last_change":"2024-02-23T00:42:16.067711+0000","last_active":"2024-02-23T00:42:24.246669+0000","last_peered":"2024-02-23T00:42:24.246669+0000","last_clean":"2024-02-23T00:42:24.246669+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:42:24.246669+0000","last_undegraded":"2024-02-23T00:42:24.246669+0000","last_fullsized":"2024-02-23T00:42:24.246669+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:42:16.067590+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.067590+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":740,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246865+0000","last_change":"2024-02-23T00:41:52.314002+0000","last_active":"2024-02-23T00:42:24.246865+0000","last_peered":"2024-02-23T00:42:24.246865+0000","last_clean":"2024-02-23T00:42:24.246865+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:42:24.246865+0000","last_undegraded":"2024-02-23T00:42:24.246865+0000","last_fullsized":"2024-02-23T00:42:24.246865+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:52.313932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:41:52.313932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":723,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246876+0000","last_change":"2024-02-23T00:41:37.222406+0000","last_active":"2024-02-23T00:42:24.246876+0000","last_peered":"2024-02-23T00:42:24.246876+0000","last_clean":"2024-02-23T00:42:24.246876+0000","last_became_active":"2024-02-23T00:41:37.222092+0000","last_became_peered":"2024-02-23T00:41:37.222092+0000","last_unstale":"2024-02-23T00:42:24.246876+0000","last_undegraded":"2024-02-23T00:42:24.246876+0000","last_fullsized":"2024-02-23T00:42:24.246876+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:49.854635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:49.854635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":671,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247453+0000","last_change":"2024-02-23T00:41:37.214599+0000","last_active":"2024-02-23T00:42:24.247453+0000","last_peered":"2024-02-23T00:42:24.247453+0000","last_clean":"2024-02-23T00:42:24.247453+0000","last_became_active":"2024-02-23T00:41:37.214214+0000","last_became_peered":"2024-02-23T00:41:37.214214+0000","last_unstale":"2024-02-23T00:42:24.247453+0000","last_undegraded":"2024-02-23T00:42:24.247453+0000","last_fullsized":"2024-02-23T00:42:24.247453+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:45.172542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.172542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":1295,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.256886+0000","last_change":"2024-02-23T00:41:39.898258+0000","last_active":"2024-02-23T00:42:25.256886+0000","last_peered":"2024-02-23T00:42:25.256886+0000","last_clean":"2024-02-23T00:42:25.256886+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:42:25.256886+0000","last_undegraded":"2024-02-23T00:42:25.256886+0000","last_fullsized":"2024-02-23T00:42:25.256886+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:41:39.898173+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:41:39.898173+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":586,"num_read_kb":586,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":429,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420031+0000","last_change":"2024-02-23T00:42:15.735663+0000","last_active":"2024-02-23T00:42:21.420031+0000","last_peered":"2024-02-23T00:42:21.420031+0000","last_clean":"2024-02-23T00:42:21.420031+0000","last_became_active":"2024-02-23T00:41:41.226195+0000","last_became_peered":"2024-02-23T00:41:41.226195+0000","last_unstale":"2024-02-23T00:42:21.420031+0000","last_undegraded":"2024-02-23T00:42:21.420031+0000","last_fullsized":"2024-02-23T00:42:21.420031+0000","mapping_epoch":681,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:42:15.735606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.735606+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":714,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244933+0000","last_change":"2024-02-23T00:40:08.076879+0000","last_active":"2024-02-23T00:42:24.244933+0000","last_peered":"2024-02-23T00:42:24.244933+0000","last_clean":"2024-02-23T00:42:24.244933+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:42:24.244933+0000","last_undegraded":"2024-02-23T00:42:24.244933+0000","last_fullsized":"2024-02-23T00:42:24.244933+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:08.076820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:08.076820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":727,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245606+0000","last_change":"2024-02-23T00:41:36.045647+0000","last_active":"2024-02-23T00:42:24.245606+0000","last_peered":"2024-02-23T00:42:24.245606+0000","last_clean":"2024-02-23T00:42:24.245606+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:42:24.245606+0000","last_undegraded":"2024-02-23T00:42:24.245606+0000","last_fullsized":"2024-02-23T00:42:24.245606+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:36.045582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:41:36.045582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":57,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.419675+0000","last_change":"2024-02-23T00:41:50.712474+0000","last_active":"2024-02-23T00:42:21.419675+0000","last_peered":"2024-02-23T00:42:21.419675+0000","last_clean":"2024-02-23T00:42:21.419675+0000","last_became_active":"2024-02-23T00:41:37.228937+0000","last_became_peered":"2024-02-23T00:41:37.228937+0000","last_unstale":"2024-02-23T00:42:21.419675+0000","last_undegraded":"2024-02-23T00:42:21.419675+0000","last_fullsized":"2024-02-23T00:42:21.419675+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:50.712410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:41:50.712410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":653,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244831+0000","last_change":"2024-02-23T00:40:57.491379+0000","last_active":"2024-02-23T00:42:24.244831+0000","last_peered":"2024-02-23T00:42:24.244831+0000","last_clean":"2024-02-23T00:42:24.244831+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:42:24.244831+0000","last_undegraded":"2024-02-23T00:42:24.244831+0000","last_fullsized":"2024-02-23T00:42:24.244831+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:57.491323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:40:57.491323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":832,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244503+0000","last_change":"2024-02-23T00:40:47.429480+0000","last_active":"2024-02-23T00:42:24.244503+0000","last_peered":"2024-02-23T00:42:24.244503+0000","last_clean":"2024-02-23T00:42:24.244503+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:42:24.244503+0000","last_undegraded":"2024-02-23T00:42:24.244503+0000","last_fullsized":"2024-02-23T00:42:24.244503+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:40:47.429409+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:40:47.429409+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":738,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.447824+0000","last_change":"2024-02-23T00:42:24.447799+0000","last_active":"2024-02-23T00:42:24.447824+0000","last_peered":"2024-02-23T00:42:24.447824+0000","last_clean":"2024-02-23T00:42:24.447824+0000","last_became_active":"2024-02-23T00:41:37.238728+0000","last_became_peered":"2024-02-23T00:41:37.238728+0000","last_unstale":"2024-02-23T00:42:24.447824+0000","last_undegraded":"2024-02-23T00:42:24.447824+0000","last_fullsized":"2024-02-23T00:42:24.447824+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.447731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.447731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":727,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245914+0000","last_change":"2024-02-23T00:42:17.473540+0000","last_active":"2024-02-23T00:42:24.245914+0000","last_peered":"2024-02-23T00:42:24.245914+0000","last_clean":"2024-02-23T00:42:24.245914+0000","last_became_active":"2024-02-23T00:41:37.239529+0000","last_became_peered":"2024-02-23T00:41:37.239529+0000","last_unstale":"2024-02-23T00:42:24.245914+0000","last_undegraded":"2024-02-23T00:42:24.245914+0000","last_fullsized":"2024-02-23T00:42:24.245914+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:17.473474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.473474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":734,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248661+0000","last_change":"2024-02-23T00:41:49.588387+0000","last_active":"2024-02-23T00:42:24.248661+0000","last_peered":"2024-02-23T00:42:24.248661+0000","last_clean":"2024-02-23T00:42:24.248661+0000","last_became_active":"2024-02-23T00:40:37.197010+0000","last_became_peered":"2024-02-23T00:40:37.197010+0000","last_unstale":"2024-02-23T00:42:24.248661+0000","last_undegraded":"2024-02-23T00:42:24.248661+0000","last_fullsized":"2024-02-23T00:42:24.248661+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:49.588290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.588290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":52,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.419485+0000","last_change":"2024-02-23T00:41:37.248755+0000","last_active":"2024-02-23T00:42:21.419485+0000","last_peered":"2024-02-23T00:42:21.419485+0000","last_clean":"2024-02-23T00:42:21.419485+0000","last_became_active":"2024-02-23T00:41:37.248272+0000","last_became_peered":"2024-02-23T00:41:37.248272+0000","last_unstale":"2024-02-23T00:42:21.419485+0000","last_undegraded":"2024-02-23T00:42:21.419485+0000","last_fullsized":"2024-02-23T00:42:21.419485+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:59.125249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:59.125249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":717,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244676+0000","last_change":"2024-02-23T00:40:50.421297+0000","last_active":"2024-02-23T00:42:24.244676+0000","last_peered":"2024-02-23T00:42:24.244676+0000","last_clean":"2024-02-23T00:42:24.244676+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:42:24.244676+0000","last_undegraded":"2024-02-23T00:42:24.244676+0000","last_fullsized":"2024-02-23T00:42:24.244676+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:50.421224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.421224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":549,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246967+0000","last_change":"2024-02-23T00:42:19.485117+0000","last_active":"2024-02-23T00:42:24.246967+0000","last_peered":"2024-02-23T00:42:24.246967+0000","last_clean":"2024-02-23T00:42:24.246967+0000","last_became_active":"2024-02-23T00:41:50.256494+0000","last_became_peered":"2024-02-23T00:41:50.256494+0000","last_unstale":"2024-02-23T00:42:24.246967+0000","last_undegraded":"2024-02-23T00:42:24.246967+0000","last_fullsized":"2024-02-23T00:42:24.246967+0000","mapping_epoch":688,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:42:19.485046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.485046+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":704,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249362+0000","last_change":"2024-02-23T00:41:47.545036+0000","last_active":"2024-02-23T00:42:24.249362+0000","last_peered":"2024-02-23T00:42:24.249362+0000","last_clean":"2024-02-23T00:42:24.249362+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:42:24.249362+0000","last_undegraded":"2024-02-23T00:42:24.249362+0000","last_fullsized":"2024-02-23T00:42:24.249362+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.544952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.544952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":711,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245424+0000","last_change":"2024-02-23T00:41:37.229878+0000","last_active":"2024-02-23T00:42:24.245424+0000","last_peered":"2024-02-23T00:42:24.245424+0000","last_clean":"2024-02-23T00:42:24.245424+0000","last_became_active":"2024-02-23T00:41:37.229416+0000","last_became_peered":"2024-02-23T00:41:37.229416+0000","last_unstale":"2024-02-23T00:42:24.245424+0000","last_undegraded":"2024-02-23T00:42:24.245424+0000","last_fullsized":"2024-02-23T00:42:24.245424+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:00.151031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:00.151031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":678,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250188+0000","last_change":"2024-02-23T00:41:55.514102+0000","last_active":"2024-02-23T00:42:24.250188+0000","last_peered":"2024-02-23T00:42:24.250188+0000","last_clean":"2024-02-23T00:42:24.250188+0000","last_became_active":"2024-02-23T00:38:49.194450+0000","last_became_peered":"2024-02-23T00:38:49.194450+0000","last_unstale":"2024-02-23T00:42:24.250188+0000","last_undegraded":"2024-02-23T00:42:24.250188+0000","last_fullsized":"2024-02-23T00:42:24.250188+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.514031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.514031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":240,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250408+0000","last_change":"2024-02-23T00:41:37.253004+0000","last_active":"2024-02-23T00:42:24.250408+0000","last_peered":"2024-02-23T00:42:24.250408+0000","last_clean":"2024-02-23T00:42:24.250408+0000","last_became_active":"2024-02-23T00:41:37.252083+0000","last_became_peered":"2024-02-23T00:41:37.252083+0000","last_unstale":"2024-02-23T00:42:24.250408+0000","last_undegraded":"2024-02-23T00:42:24.250408+0000","last_fullsized":"2024-02-23T00:42:24.250408+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":649,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250159+0000","last_change":"2024-02-23T00:41:50.256382+0000","last_active":"2024-02-23T00:42:24.250159+0000","last_peered":"2024-02-23T00:42:24.250159+0000","last_clean":"2024-02-23T00:42:24.250159+0000","last_became_active":"2024-02-23T00:41:50.256004+0000","last_became_peered":"2024-02-23T00:41:50.256004+0000","last_unstale":"2024-02-23T00:42:24.250159+0000","last_undegraded":"2024-02-23T00:42:24.250159+0000","last_fullsized":"2024-02-23T00:42:24.250159+0000","mapping_epoch":688,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:40:51.632948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:51.632948+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":732,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247265+0000","last_change":"2024-02-23T00:42:15.223520+0000","last_active":"2024-02-23T00:42:24.247265+0000","last_peered":"2024-02-23T00:42:24.247265+0000","last_clean":"2024-02-23T00:42:24.247265+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:42:24.247265+0000","last_undegraded":"2024-02-23T00:42:24.247265+0000","last_fullsized":"2024-02-23T00:42:24.247265+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.223455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.223455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":898,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.246814+0000","last_change":"2024-02-23T00:41:33.047241+0000","last_active":"2024-02-23T00:42:25.246814+0000","last_peered":"2024-02-23T00:42:25.246814+0000","last_clean":"2024-02-23T00:42:25.246814+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:42:25.246814+0000","last_undegraded":"2024-02-23T00:42:25.246814+0000","last_fullsized":"2024-02-23T00:42:25.246814+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:41:33.047155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.047155+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":652,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249391+0000","last_change":"2024-02-23T00:41:34.719462+0000","last_active":"2024-02-23T00:42:24.249391+0000","last_peered":"2024-02-23T00:42:24.249391+0000","last_clean":"2024-02-23T00:42:24.249391+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:42:24.249391+0000","last_undegraded":"2024-02-23T00:42:24.249391+0000","last_fullsized":"2024-02-23T00:42:24.249391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:34.719388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.719388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":226,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249035+0000","last_change":"2024-02-23T00:40:15.208236+0000","last_active":"2024-02-23T00:42:24.249035+0000","last_peered":"2024-02-23T00:42:24.249035+0000","last_clean":"2024-02-23T00:42:24.249035+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:42:24.249035+0000","last_undegraded":"2024-02-23T00:42:24.249035+0000","last_fullsized":"2024-02-23T00:42:24.249035+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:15.208124+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.208124+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":886,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245124+0000","last_change":"2024-02-23T00:41:08.116872+0000","last_active":"2024-02-23T00:42:24.245124+0000","last_peered":"2024-02-23T00:42:24.245124+0000","last_clean":"2024-02-23T00:42:24.245124+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:42:24.245124+0000","last_undegraded":"2024-02-23T00:42:24.245124+0000","last_fullsized":"2024-02-23T00:42:24.245124+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:41:08.116765+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:41:08.116765+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":716,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244384+0000","last_change":"2024-02-23T00:42:14.253417+0000","last_active":"2024-02-23T00:42:24.244384+0000","last_peered":"2024-02-23T00:42:24.244384+0000","last_clean":"2024-02-23T00:42:24.244384+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:42:24.244384+0000","last_undegraded":"2024-02-23T00:42:24.244384+0000","last_fullsized":"2024-02-23T00:42:24.244384+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.253319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.253319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"689'12","reported_seq":906,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420114+0000","last_change":"2024-02-23T00:42:19.683007+0000","last_active":"2024-02-23T00:42:21.420114+0000","last_peered":"2024-02-23T00:42:21.420114+0000","last_clean":"2024-02-23T00:42:21.420114+0000","last_became_active":"2024-02-23T00:41:50.256773+0000","last_became_peered":"2024-02-23T00:41:50.256773+0000","last_unstale":"2024-02-23T00:42:21.420114+0000","last_undegraded":"2024-02-23T00:42:21.420114+0000","last_fullsized":"2024-02-23T00:42:21.420114+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'12","last_scrub_stamp":"2024-02-23T00:42:19.682902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.682902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":643,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244532+0000","last_change":"2024-02-23T00:41:40.171696+0000","last_active":"2024-02-23T00:42:24.244532+0000","last_peered":"2024-02-23T00:42:24.244532+0000","last_clean":"2024-02-23T00:42:24.244532+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:42:24.244532+0000","last_undegraded":"2024-02-23T00:42:24.244532+0000","last_fullsized":"2024-02-23T00:42:24.244532+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:40.171630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:41:40.171630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":714,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248026+0000","last_change":"2024-02-23T00:40:59.669093+0000","last_active":"2024-02-23T00:42:24.248026+0000","last_peered":"2024-02-23T00:42:24.248026+0000","last_clean":"2024-02-23T00:42:24.248026+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:42:24.248026+0000","last_undegraded":"2024-02-23T00:42:24.248026+0000","last_fullsized":"2024-02-23T00:42:24.248026+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:59.669025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:59.669025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":558,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245164+0000","last_change":"2024-02-23T00:42:18.659610+0000","last_active":"2024-02-23T00:42:24.245164+0000","last_peered":"2024-02-23T00:42:24.245164+0000","last_clean":"2024-02-23T00:42:24.245164+0000","last_became_active":"2024-02-23T00:41:50.253028+0000","last_became_peered":"2024-02-23T00:41:50.253028+0000","last_unstale":"2024-02-23T00:42:24.245164+0000","last_undegraded":"2024-02-23T00:42:24.245164+0000","last_fullsized":"2024-02-23T00:42:24.245164+0000","mapping_epoch":688,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:42:18.659534+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.659534+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":740,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250626+0000","last_change":"2024-02-23T00:41:37.375803+0000","last_active":"2024-02-23T00:42:24.250626+0000","last_peered":"2024-02-23T00:42:24.250626+0000","last_clean":"2024-02-23T00:42:24.250626+0000","last_became_active":"2024-02-23T00:41:37.229862+0000","last_became_peered":"2024-02-23T00:41:37.229862+0000","last_unstale":"2024-02-23T00:42:24.250626+0000","last_undegraded":"2024-02-23T00:42:24.250626+0000","last_fullsized":"2024-02-23T00:42:24.250626+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:40:52.825480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.825480+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":688,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248229+0000","last_change":"2024-02-23T00:40:54.788960+0000","last_active":"2024-02-23T00:42:24.248229+0000","last_peered":"2024-02-23T00:42:24.248229+0000","last_clean":"2024-02-23T00:42:24.248229+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:42:24.248229+0000","last_undegraded":"2024-02-23T00:42:24.248229+0000","last_fullsized":"2024-02-23T00:42:24.248229+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_clean_scrub_stamp":"2024-02-23T00:40:54.788865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":661,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245598+0000","last_change":"2024-02-23T00:40:19.126448+0000","last_active":"2024-02-23T00:42:24.245598+0000","last_peered":"2024-02-23T00:42:24.245598+0000","last_clean":"2024-02-23T00:42:24.245598+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:42:24.245598+0000","last_undegraded":"2024-02-23T00:42:24.245598+0000","last_fullsized":"2024-02-23T00:42:24.245598+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.126375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":57,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420466+0000","last_change":"2024-02-23T00:42:12.789891+0000","last_active":"2024-02-23T00:42:21.420466+0000","last_peered":"2024-02-23T00:42:21.420466+0000","last_clean":"2024-02-23T00:42:21.420466+0000","last_became_active":"2024-02-23T00:41:37.233017+0000","last_became_peered":"2024-02-23T00:41:37.233017+0000","last_unstale":"2024-02-23T00:42:21.420466+0000","last_undegraded":"2024-02-23T00:42:21.420466+0000","last_fullsized":"2024-02-23T00:42:21.420466+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.789819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.789819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"642'14","reported_seq":258,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.243009+0000","last_change":"2024-02-23T00:41:01.213974+0000","last_active":"2024-02-23T00:42:25.243009+0000","last_peered":"2024-02-23T00:42:25.243009+0000","last_clean":"2024-02-23T00:42:25.243009+0000","last_became_active":"2024-02-23T00:41:01.212780+0000","last_became_peered":"2024-02-23T00:41:01.212780+0000","last_unstale":"2024-02-23T00:42:25.243009+0000","last_undegraded":"2024-02-23T00:42:25.243009+0000","last_fullsized":"2024-02-23T00:42:25.243009+0000","mapping_epoch":511,"log_start":"562'12","ondisk_log_start":"562'12","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":529,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246788+0000","last_change":"2024-02-23T00:42:19.049262+0000","last_active":"2024-02-23T00:42:24.246788+0000","last_peered":"2024-02-23T00:42:24.246788+0000","last_clean":"2024-02-23T00:42:24.246788+0000","last_became_active":"2024-02-23T00:41:50.255315+0000","last_became_peered":"2024-02-23T00:41:50.255315+0000","last_unstale":"2024-02-23T00:42:24.246788+0000","last_undegraded":"2024-02-23T00:42:24.246788+0000","last_fullsized":"2024-02-23T00:42:24.246788+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'34","last_scrub_stamp":"2024-02-23T00:42:19.049190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.049190+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":705,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245368+0000","last_change":"2024-02-23T00:40:46.427758+0000","last_active":"2024-02-23T00:42:24.245368+0000","last_peered":"2024-02-23T00:42:24.245368+0000","last_clean":"2024-02-23T00:42:24.245368+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:42:24.245368+0000","last_undegraded":"2024-02-23T00:42:24.245368+0000","last_fullsized":"2024-02-23T00:42:24.245368+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:46.427689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:46.427689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":721,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249790+0000","last_change":"2024-02-23T00:42:12.433222+0000","last_active":"2024-02-23T00:42:24.249790+0000","last_peered":"2024-02-23T00:42:24.249790+0000","last_clean":"2024-02-23T00:42:24.249790+0000","last_became_active":"2024-02-23T00:40:57.488797+0000","last_became_peered":"2024-02-23T00:40:57.488797+0000","last_unstale":"2024-02-23T00:42:24.249790+0000","last_undegraded":"2024-02-23T00:42:24.249790+0000","last_fullsized":"2024-02-23T00:42:24.249790+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.433157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.433157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":686,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244781+0000","last_change":"2024-02-23T00:41:51.307510+0000","last_active":"2024-02-23T00:42:24.244781+0000","last_peered":"2024-02-23T00:42:24.244781+0000","last_clean":"2024-02-23T00:42:24.244781+0000","last_became_active":"2024-02-23T00:41:37.229061+0000","last_became_peered":"2024-02-23T00:41:37.229061+0000","last_unstale":"2024-02-23T00:42:24.244781+0000","last_undegraded":"2024-02-23T00:42:24.244781+0000","last_fullsized":"2024-02-23T00:42:24.244781+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:51.307443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:51.307443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":252,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.251002+0000","last_change":"2024-02-23T00:41:55.355660+0000","last_active":"2024-02-23T00:42:24.251002+0000","last_peered":"2024-02-23T00:42:24.251002+0000","last_clean":"2024-02-23T00:42:24.251002+0000","last_became_active":"2024-02-23T00:41:37.240968+0000","last_became_peered":"2024-02-23T00:41:37.240968+0000","last_unstale":"2024-02-23T00:42:24.251002+0000","last_undegraded":"2024-02-23T00:42:24.251002+0000","last_fullsized":"2024-02-23T00:42:24.251002+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.355553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.355553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":752,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248271+0000","last_change":"2024-02-23T00:41:53.922666+0000","last_active":"2024-02-23T00:42:24.248271+0000","last_peered":"2024-02-23T00:42:24.248271+0000","last_clean":"2024-02-23T00:42:24.248271+0000","last_became_active":"2024-02-23T00:40:57.218869+0000","last_became_peered":"2024-02-23T00:40:57.218869+0000","last_unstale":"2024-02-23T00:42:24.248271+0000","last_undegraded":"2024-02-23T00:42:24.248271+0000","last_fullsized":"2024-02-23T00:42:24.248271+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.922592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.922592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":289,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248949+0000","last_change":"2024-02-23T00:41:38.280528+0000","last_active":"2024-02-23T00:42:24.248949+0000","last_peered":"2024-02-23T00:42:24.248949+0000","last_clean":"2024-02-23T00:42:24.248949+0000","last_became_active":"2024-02-23T00:41:37.228326+0000","last_became_peered":"2024-02-23T00:41:37.228326+0000","last_unstale":"2024-02-23T00:42:24.248949+0000","last_undegraded":"2024-02-23T00:42:24.248949+0000","last_fullsized":"2024-02-23T00:42:24.248949+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:39:28.324185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.324185+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":696,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244408+0000","last_change":"2024-02-23T00:41:42.160301+0000","last_active":"2024-02-23T00:42:24.244408+0000","last_peered":"2024-02-23T00:42:24.244408+0000","last_clean":"2024-02-23T00:42:24.244408+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:42:24.244408+0000","last_undegraded":"2024-02-23T00:42:24.244408+0000","last_fullsized":"2024-02-23T00:42:24.244408+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.160190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.160190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":687,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250022+0000","last_change":"2024-02-23T00:41:50.256151+0000","last_active":"2024-02-23T00:42:24.250022+0000","last_peered":"2024-02-23T00:42:24.250022+0000","last_clean":"2024-02-23T00:42:24.250022+0000","last_became_active":"2024-02-23T00:41:50.255850+0000","last_became_peered":"2024-02-23T00:41:50.255850+0000","last_unstale":"2024-02-23T00:42:24.250022+0000","last_undegraded":"2024-02-23T00:42:24.250022+0000","last_fullsized":"2024-02-23T00:42:24.250022+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:01.638560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:01.638560+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":226,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250042+0000","last_change":"2024-02-23T00:41:02.306933+0000","last_active":"2024-02-23T00:42:24.250042+0000","last_peered":"2024-02-23T00:42:24.250042+0000","last_clean":"2024-02-23T00:42:24.250042+0000","last_became_active":"2024-02-23T00:38:49.175801+0000","last_became_peered":"2024-02-23T00:38:49.175801+0000","last_unstale":"2024-02-23T00:42:24.250042+0000","last_undegraded":"2024-02-23T00:42:24.250042+0000","last_fullsized":"2024-02-23T00:42:24.250042+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:02.306855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.306855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":959,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245195+0000","last_change":"2024-02-23T00:40:17.043400+0000","last_active":"2024-02-23T00:42:24.245195+0000","last_peered":"2024-02-23T00:42:24.245195+0000","last_clean":"2024-02-23T00:42:24.245195+0000","last_became_active":"2024-02-23T00:31:29.095286+0000","last_became_peered":"2024-02-23T00:31:29.095286+0000","last_unstale":"2024-02-23T00:42:24.245195+0000","last_undegraded":"2024-02-23T00:42:24.245195+0000","last_fullsized":"2024-02-23T00:42:24.245195+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:17.043322+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.043322+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":753,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244525+0000","last_change":"2024-02-23T00:41:37.221015+0000","last_active":"2024-02-23T00:42:24.244525+0000","last_peered":"2024-02-23T00:42:24.244525+0000","last_clean":"2024-02-23T00:42:24.244525+0000","last_became_active":"2024-02-23T00:41:37.220531+0000","last_became_peered":"2024-02-23T00:41:37.220531+0000","last_unstale":"2024-02-23T00:42:24.244525+0000","last_undegraded":"2024-02-23T00:42:24.244525+0000","last_fullsized":"2024-02-23T00:42:24.244525+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.603558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.603558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":72,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420119+0000","last_change":"2024-02-23T00:41:38.797092+0000","last_active":"2024-02-23T00:42:21.420119+0000","last_peered":"2024-02-23T00:42:21.420119+0000","last_clean":"2024-02-23T00:42:21.420119+0000","last_became_active":"2024-02-23T00:41:37.235230+0000","last_became_peered":"2024-02-23T00:41:37.235230+0000","last_unstale":"2024-02-23T00:42:21.420119+0000","last_undegraded":"2024-02-23T00:42:21.420119+0000","last_fullsized":"2024-02-23T00:42:21.420119+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'35","last_scrub_stamp":"2024-02-23T00:40:50.813709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.813709+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":723,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.251483+0000","last_change":"2024-02-23T00:41:37.230256+0000","last_active":"2024-02-23T00:42:24.251483+0000","last_peered":"2024-02-23T00:42:24.251483+0000","last_clean":"2024-02-23T00:42:24.251483+0000","last_became_active":"2024-02-23T00:41:37.229922+0000","last_became_peered":"2024-02-23T00:41:37.229922+0000","last_unstale":"2024-02-23T00:42:24.251483+0000","last_undegraded":"2024-02-23T00:42:24.251483+0000","last_fullsized":"2024-02-23T00:42:24.251483+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:56.829791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:56.829791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":708,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245363+0000","last_change":"2024-02-23T00:41:05.523604+0000","last_active":"2024-02-23T00:42:24.245363+0000","last_peered":"2024-02-23T00:42:24.245363+0000","last_clean":"2024-02-23T00:42:24.245363+0000","last_became_active":"2024-02-23T00:41:05.523365+0000","last_became_peered":"2024-02-23T00:41:05.523365+0000","last_unstale":"2024-02-23T00:42:24.245363+0000","last_undegraded":"2024-02-23T00:42:24.245363+0000","last_fullsized":"2024-02-23T00:42:24.245363+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:01.179668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:41:01.179668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":643,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244961+0000","last_change":"2024-02-23T00:42:16.223783+0000","last_active":"2024-02-23T00:42:24.244961+0000","last_peered":"2024-02-23T00:42:24.244961+0000","last_clean":"2024-02-23T00:42:24.244961+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:42:24.244961+0000","last_undegraded":"2024-02-23T00:42:24.244961+0000","last_fullsized":"2024-02-23T00:42:24.244961+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:16.223720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.223720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":734,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248303+0000","last_change":"2024-02-23T00:41:37.228614+0000","last_active":"2024-02-23T00:42:24.248303+0000","last_peered":"2024-02-23T00:42:24.248303+0000","last_clean":"2024-02-23T00:42:24.248303+0000","last_became_active":"2024-02-23T00:41:37.228303+0000","last_became_peered":"2024-02-23T00:41:37.228303+0000","last_unstale":"2024-02-23T00:42:24.248303+0000","last_undegraded":"2024-02-23T00:42:24.248303+0000","last_fullsized":"2024-02-23T00:42:24.248303+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:07.803509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:41:07.803509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":438,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420194+0000","last_change":"2024-02-23T00:42:17.721535+0000","last_active":"2024-02-23T00:42:21.420194+0000","last_peered":"2024-02-23T00:42:21.420194+0000","last_clean":"2024-02-23T00:42:21.420194+0000","last_became_active":"2024-02-23T00:41:41.224814+0000","last_became_peered":"2024-02-23T00:41:41.224814+0000","last_unstale":"2024-02-23T00:42:21.420194+0000","last_undegraded":"2024-02-23T00:42:21.420194+0000","last_fullsized":"2024-02-23T00:42:21.420194+0000","mapping_epoch":681,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:42:17.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.721471+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":723,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248736+0000","last_change":"2024-02-23T00:41:42.606731+0000","last_active":"2024-02-23T00:42:24.248736+0000","last_peered":"2024-02-23T00:42:24.248736+0000","last_clean":"2024-02-23T00:42:24.248736+0000","last_became_active":"2024-02-23T00:41:37.229722+0000","last_became_peered":"2024-02-23T00:41:37.229722+0000","last_unstale":"2024-02-23T00:42:24.248736+0000","last_undegraded":"2024-02-23T00:42:24.248736+0000","last_fullsized":"2024-02-23T00:42:24.248736+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.606635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.606635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":57,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.419528+0000","last_change":"2024-02-23T00:41:44.733090+0000","last_active":"2024-02-23T00:42:21.419528+0000","last_peered":"2024-02-23T00:42:21.419528+0000","last_clean":"2024-02-23T00:42:21.419528+0000","last_became_active":"2024-02-23T00:41:37.234720+0000","last_became_peered":"2024-02-23T00:41:37.234720+0000","last_unstale":"2024-02-23T00:42:21.419528+0000","last_undegraded":"2024-02-23T00:42:21.419528+0000","last_fullsized":"2024-02-23T00:42:21.419528+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.733024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.733024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"78'2","reported_seq":744,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245515+0000","last_change":"2024-02-23T00:41:53.371931+0000","last_active":"2024-02-23T00:42:24.245515+0000","last_peered":"2024-02-23T00:42:24.245515+0000","last_clean":"2024-02-23T00:42:24.245515+0000","last_became_active":"2024-02-23T00:41:25.236069+0000","last_became_peered":"2024-02-23T00:41:25.236069+0000","last_unstale":"2024-02-23T00:42:24.245515+0000","last_undegraded":"2024-02-23T00:42:24.245515+0000","last_fullsized":"2024-02-23T00:42:24.245515+0000","mapping_epoch":511,"log_start":"78'2","ondisk_log_start":"78'2","created":60,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:41:53.371852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.371852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":652,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249952+0000","last_change":"2024-02-23T00:41:36.715086+0000","last_active":"2024-02-23T00:42:24.249952+0000","last_peered":"2024-02-23T00:42:24.249952+0000","last_clean":"2024-02-23T00:42:24.249952+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:42:24.249952+0000","last_undegraded":"2024-02-23T00:42:24.249952+0000","last_fullsized":"2024-02-23T00:42:24.249952+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:36.715011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:36.715011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":732,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249745+0000","last_change":"2024-02-23T00:42:15.388568+0000","last_active":"2024-02-23T00:42:24.249745+0000","last_peered":"2024-02-23T00:42:24.249745+0000","last_clean":"2024-02-23T00:42:24.249745+0000","last_became_active":"2024-02-23T00:40:45.191453+0000","last_became_peered":"2024-02-23T00:40:45.191453+0000","last_unstale":"2024-02-23T00:42:24.249745+0000","last_undegraded":"2024-02-23T00:42:24.249745+0000","last_fullsized":"2024-02-23T00:42:24.249745+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.388510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.388510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":749,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244971+0000","last_change":"2024-02-23T00:41:33.595415+0000","last_active":"2024-02-23T00:42:24.244971+0000","last_peered":"2024-02-23T00:42:24.244971+0000","last_clean":"2024-02-23T00:42:24.244971+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:42:24.244971+0000","last_undegraded":"2024-02-23T00:42:24.244971+0000","last_fullsized":"2024-02-23T00:42:24.244971+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:41:33.595268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.595268+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":726,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250551+0000","last_change":"2024-02-23T00:41:37.230168+0000","last_active":"2024-02-23T00:42:24.250551+0000","last_peered":"2024-02-23T00:42:24.250551+0000","last_clean":"2024-02-23T00:42:24.250551+0000","last_became_active":"2024-02-23T00:41:37.229335+0000","last_became_peered":"2024-02-23T00:41:37.229335+0000","last_unstale":"2024-02-23T00:42:24.250551+0000","last_undegraded":"2024-02-23T00:42:24.250551+0000","last_fullsized":"2024-02-23T00:42:24.250551+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:46.771465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:40:46.771465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":697,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247267+0000","last_change":"2024-02-23T00:41:43.984556+0000","last_active":"2024-02-23T00:42:24.247267+0000","last_peered":"2024-02-23T00:42:24.247267+0000","last_clean":"2024-02-23T00:42:24.247267+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:42:24.247267+0000","last_undegraded":"2024-02-23T00:42:24.247267+0000","last_fullsized":"2024-02-23T00:42:24.247267+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:43.984490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:43.984490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":226,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250122+0000","last_change":"2024-02-23T00:40:52.474608+0000","last_active":"2024-02-23T00:42:24.250122+0000","last_peered":"2024-02-23T00:42:24.250122+0000","last_clean":"2024-02-23T00:42:24.250122+0000","last_became_active":"2024-02-23T00:38:49.226188+0000","last_became_peered":"2024-02-23T00:38:49.226188+0000","last_unstale":"2024-02-23T00:42:24.250122+0000","last_undegraded":"2024-02-23T00:42:24.250122+0000","last_fullsized":"2024-02-23T00:42:24.250122+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.474495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.474495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":745,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248356+0000","last_change":"2024-02-23T00:42:22.037093+0000","last_active":"2024-02-23T00:42:24.248356+0000","last_peered":"2024-02-23T00:42:24.248356+0000","last_clean":"2024-02-23T00:42:24.248356+0000","last_became_active":"2024-02-23T00:41:13.469953+0000","last_became_peered":"2024-02-23T00:41:13.469953+0000","last_unstale":"2024-02-23T00:42:24.248356+0000","last_undegraded":"2024-02-23T00:42:24.248356+0000","last_fullsized":"2024-02-23T00:42:24.248356+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:22.037008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.037008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":654,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249212+0000","last_change":"2024-02-23T00:41:35.764933+0000","last_active":"2024-02-23T00:42:24.249212+0000","last_peered":"2024-02-23T00:42:24.249212+0000","last_clean":"2024-02-23T00:42:24.249212+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:42:24.249212+0000","last_undegraded":"2024-02-23T00:42:24.249212+0000","last_fullsized":"2024-02-23T00:42:24.249212+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.764859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.764859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":740,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247011+0000","last_change":"2024-02-23T00:41:50.264305+0000","last_active":"2024-02-23T00:42:24.247011+0000","last_peered":"2024-02-23T00:42:24.247011+0000","last_clean":"2024-02-23T00:42:24.247011+0000","last_became_active":"2024-02-23T00:40:38.061696+0000","last_became_peered":"2024-02-23T00:40:38.061696+0000","last_unstale":"2024-02-23T00:42:24.247011+0000","last_undegraded":"2024-02-23T00:42:24.247011+0000","last_fullsized":"2024-02-23T00:42:24.247011+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:50.264240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:41:50.264240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":845,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245160+0000","last_change":"2024-02-23T00:41:40.219753+0000","last_active":"2024-02-23T00:42:24.245160+0000","last_peered":"2024-02-23T00:42:24.245160+0000","last_clean":"2024-02-23T00:42:24.245160+0000","last_became_active":"2024-02-23T00:41:40.219384+0000","last_became_peered":"2024-02-23T00:41:40.219384+0000","last_unstale":"2024-02-23T00:42:24.245160+0000","last_undegraded":"2024-02-23T00:42:24.245160+0000","last_fullsized":"2024-02-23T00:42:24.245160+0000","mapping_epoch":680,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":755,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245327+0000","last_change":"2024-02-23T00:41:44.257867+0000","last_active":"2024-02-23T00:42:24.245327+0000","last_peered":"2024-02-23T00:42:24.245327+0000","last_clean":"2024-02-23T00:42:24.245327+0000","last_became_active":"2024-02-23T00:41:37.219000+0000","last_became_peered":"2024-02-23T00:41:37.219000+0000","last_unstale":"2024-02-23T00:42:24.245327+0000","last_undegraded":"2024-02-23T00:42:24.245327+0000","last_fullsized":"2024-02-23T00:42:24.245327+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.257805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.257805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":952,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245913+0000","last_change":"2024-02-23T00:40:12.984115+0000","last_active":"2024-02-23T00:42:24.245913+0000","last_peered":"2024-02-23T00:42:24.245913+0000","last_clean":"2024-02-23T00:42:24.245913+0000","last_became_active":"2024-02-23T00:31:27.040299+0000","last_became_peered":"2024-02-23T00:31:27.040299+0000","last_unstale":"2024-02-23T00:42:24.245913+0000","last_undegraded":"2024-02-23T00:42:24.245913+0000","last_fullsized":"2024-02-23T00:42:24.245913+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:12.984040+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.984040+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"62'1","reported_seq":757,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.251407+0000","last_change":"2024-02-23T00:42:22.301796+0000","last_active":"2024-02-23T00:42:24.251407+0000","last_peered":"2024-02-23T00:42:24.251407+0000","last_clean":"2024-02-23T00:42:24.251407+0000","last_became_active":"2024-02-23T00:41:21.226819+0000","last_became_peered":"2024-02-23T00:41:21.226819+0000","last_unstale":"2024-02-23T00:42:24.251407+0000","last_undegraded":"2024-02-23T00:42:24.251407+0000","last_fullsized":"2024-02-23T00:42:24.251407+0000","mapping_epoch":511,"log_start":"62'1","ondisk_log_start":"62'1","created":60,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:42:22.301726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.301726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":659,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246787+0000","last_change":"2024-02-23T00:41:04.069352+0000","last_active":"2024-02-23T00:42:24.246787+0000","last_peered":"2024-02-23T00:42:24.246787+0000","last_clean":"2024-02-23T00:42:24.246787+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:42:24.246787+0000","last_undegraded":"2024-02-23T00:42:24.246787+0000","last_fullsized":"2024-02-23T00:42:24.246787+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:04.069272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:04.069272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":727,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.419722+0000","last_change":"2024-02-23T00:42:18.673003+0000","last_active":"2024-02-23T00:42:21.419722+0000","last_peered":"2024-02-23T00:42:21.419722+0000","last_clean":"2024-02-23T00:42:21.419722+0000","last_became_active":"2024-02-23T00:41:50.256743+0000","last_became_peered":"2024-02-23T00:41:50.256743+0000","last_unstale":"2024-02-23T00:42:21.419722+0000","last_undegraded":"2024-02-23T00:42:21.419722+0000","last_fullsized":"2024-02-23T00:42:21.419722+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:18.672928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.672928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420091+0000","last_change":"2024-02-23T00:41:54.690476+0000","last_active":"2024-02-23T00:42:21.420091+0000","last_peered":"2024-02-23T00:42:21.420091+0000","last_clean":"2024-02-23T00:42:21.420091+0000","last_became_active":"2024-02-23T00:41:37.239608+0000","last_became_peered":"2024-02-23T00:41:37.239608+0000","last_unstale":"2024-02-23T00:42:21.420091+0000","last_undegraded":"2024-02-23T00:42:21.420091+0000","last_fullsized":"2024-02-23T00:42:21.420091+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:54.690407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:54.690407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":1023,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.254642+0000","last_change":"2024-02-23T00:40:15.043521+0000","last_active":"2024-02-23T00:42:25.254642+0000","last_peered":"2024-02-23T00:42:25.254642+0000","last_clean":"2024-02-23T00:42:25.254642+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:42:25.254642+0000","last_undegraded":"2024-02-23T00:42:25.254642+0000","last_fullsized":"2024-02-23T00:42:25.254642+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'48","last_scrub_stamp":"2024-02-23T00:40:15.043419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.043419+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"601'2","reported_seq":335,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247120+0000","last_change":"2024-02-23T00:40:21.186280+0000","last_active":"2024-02-23T00:42:24.247120+0000","last_peered":"2024-02-23T00:42:24.247120+0000","last_clean":"2024-02-23T00:42:24.247120+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:42:24.247120+0000","last_undegraded":"2024-02-23T00:42:24.247120+0000","last_fullsized":"2024-02-23T00:42:24.247120+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'2","last_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_deep_scrub":"601'2","last_deep_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_clean_scrub_stamp":"2024-02-23T00:40:21.186215+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":262,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250673+0000","last_change":"2024-02-23T00:41:17.220628+0000","last_active":"2024-02-23T00:42:24.250673+0000","last_peered":"2024-02-23T00:42:24.250673+0000","last_clean":"2024-02-23T00:42:24.250673+0000","last_became_active":"2024-02-23T00:41:17.220294+0000","last_became_peered":"2024-02-23T00:41:17.220294+0000","last_unstale":"2024-02-23T00:42:24.250673+0000","last_undegraded":"2024-02-23T00:42:24.250673+0000","last_fullsized":"2024-02-23T00:42:24.250673+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:40.370090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:39:40.370090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":664,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247191+0000","last_change":"2024-02-23T00:41:55.275163+0000","last_active":"2024-02-23T00:42:24.247191+0000","last_peered":"2024-02-23T00:42:24.247191+0000","last_clean":"2024-02-23T00:42:24.247191+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:42:24.247191+0000","last_undegraded":"2024-02-23T00:42:24.247191+0000","last_fullsized":"2024-02-23T00:42:24.247191+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.275099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.275099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":625,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247894+0000","last_change":"2024-02-23T00:41:55.024771+0000","last_active":"2024-02-23T00:42:24.247894+0000","last_peered":"2024-02-23T00:42:24.247894+0000","last_clean":"2024-02-23T00:42:24.247894+0000","last_became_active":"2024-02-23T00:38:49.183777+0000","last_became_peered":"2024-02-23T00:38:49.183777+0000","last_unstale":"2024-02-23T00:42:24.247894+0000","last_undegraded":"2024-02-23T00:42:24.247894+0000","last_fullsized":"2024-02-23T00:42:24.247894+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.024705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"681'21","reported_seq":279,"reported_epoch":723,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:42:25.244247+0000","last_change":"2024-02-23T00:42:08.416966+0000","last_active":"2024-02-23T00:42:25.244247+0000","last_peered":"2024-02-23T00:42:25.244247+0000","last_clean":"2024-02-23T00:27:24.172824+0000","last_became_active":"2024-02-23T00:41:38.214049+0000","last_became_peered":"2024-02-23T00:41:38.214049+0000","last_unstale":"2024-02-23T00:42:25.244247+0000","last_undegraded":"2024-02-23T00:42:25.244247+0000","last_fullsized":"2024-02-23T00:42:25.244247+0000","mapping_epoch":678,"log_start":"606'17","ondisk_log_start":"606'17","created":58,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'19","last_scrub_stamp":"2024-02-23T00:40:58.498752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.498752+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":147,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247031+0000","last_change":"2024-02-23T00:40:53.236854+0000","last_active":"2024-02-23T00:42:24.247031+0000","last_peered":"2024-02-23T00:42:24.247031+0000","last_clean":"2024-02-23T00:42:24.247031+0000","last_became_active":"2024-02-23T00:40:23.194277+0000","last_became_peered":"2024-02-23T00:40:23.194277+0000","last_unstale":"2024-02-23T00:42:24.247031+0000","last_undegraded":"2024-02-23T00:42:24.247031+0000","last_fullsized":"2024-02-23T00:42:24.247031+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:40:53.236791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.236791+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":718,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245310+0000","last_change":"2024-02-23T00:41:37.229713+0000","last_active":"2024-02-23T00:42:24.245310+0000","last_peered":"2024-02-23T00:42:24.245310+0000","last_clean":"2024-02-23T00:42:24.245310+0000","last_became_active":"2024-02-23T00:41:37.229003+0000","last_became_peered":"2024-02-23T00:41:37.229003+0000","last_unstale":"2024-02-23T00:42:24.245310+0000","last_undegraded":"2024-02-23T00:42:24.245310+0000","last_fullsized":"2024-02-23T00:42:24.245310+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.208784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.208784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":57,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.419563+0000","last_change":"2024-02-23T00:41:49.737992+0000","last_active":"2024-02-23T00:42:21.419563+0000","last_peered":"2024-02-23T00:42:21.419563+0000","last_clean":"2024-02-23T00:42:21.419563+0000","last_became_active":"2024-02-23T00:41:37.248459+0000","last_became_peered":"2024-02-23T00:41:37.248459+0000","last_unstale":"2024-02-23T00:42:21.419563+0000","last_undegraded":"2024-02-23T00:42:21.419563+0000","last_fullsized":"2024-02-23T00:42:21.419563+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:49.737927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.737927+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":652,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247767+0000","last_change":"2024-02-23T00:41:40.761546+0000","last_active":"2024-02-23T00:42:24.247767+0000","last_peered":"2024-02-23T00:42:24.247767+0000","last_clean":"2024-02-23T00:42:24.247767+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:42:24.247767+0000","last_undegraded":"2024-02-23T00:42:24.247767+0000","last_fullsized":"2024-02-23T00:42:24.247767+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:40.761466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:41:40.761466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":749,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246393+0000","last_change":"2024-02-23T00:41:37.240320+0000","last_active":"2024-02-23T00:42:24.246393+0000","last_peered":"2024-02-23T00:42:24.246393+0000","last_clean":"2024-02-23T00:42:24.246393+0000","last_became_active":"2024-02-23T00:41:37.239598+0000","last_became_peered":"2024-02-23T00:41:37.239598+0000","last_unstale":"2024-02-23T00:42:24.246393+0000","last_undegraded":"2024-02-23T00:42:24.246393+0000","last_fullsized":"2024-02-23T00:42:24.246393+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:01.091909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:01.091909+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":636,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244228+0000","last_change":"2024-02-23T00:41:50.258032+0000","last_active":"2024-02-23T00:42:24.244228+0000","last_peered":"2024-02-23T00:42:24.244228+0000","last_clean":"2024-02-23T00:42:24.244228+0000","last_became_active":"2024-02-23T00:41:50.257656+0000","last_became_peered":"2024-02-23T00:41:50.257656+0000","last_unstale":"2024-02-23T00:42:24.244228+0000","last_undegraded":"2024-02-23T00:42:24.244228+0000","last_fullsized":"2024-02-23T00:42:24.244228+0000","mapping_epoch":688,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:40:50.676380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.676380+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":754,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249348+0000","last_change":"2024-02-23T00:41:54.923278+0000","last_active":"2024-02-23T00:42:24.249348+0000","last_peered":"2024-02-23T00:42:24.249348+0000","last_clean":"2024-02-23T00:42:24.249348+0000","last_became_active":"2024-02-23T00:41:37.238708+0000","last_became_peered":"2024-02-23T00:41:37.238708+0000","last_unstale":"2024-02-23T00:42:24.249348+0000","last_undegraded":"2024-02-23T00:42:24.249348+0000","last_fullsized":"2024-02-23T00:42:24.249348+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:41:54.923214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:54.923214+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":244,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250475+0000","last_change":"2024-02-23T00:41:33.042594+0000","last_active":"2024-02-23T00:42:24.250475+0000","last_peered":"2024-02-23T00:42:24.250475+0000","last_clean":"2024-02-23T00:42:24.250475+0000","last_became_active":"2024-02-23T00:40:17.214288+0000","last_became_peered":"2024-02-23T00:40:17.214288+0000","last_unstale":"2024-02-23T00:42:24.250475+0000","last_undegraded":"2024-02-23T00:42:24.250475+0000","last_fullsized":"2024-02-23T00:42:24.250475+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:33.042520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.042520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":646,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246947+0000","last_change":"2024-02-23T00:42:13.006928+0000","last_active":"2024-02-23T00:42:24.246947+0000","last_peered":"2024-02-23T00:42:24.246947+0000","last_clean":"2024-02-23T00:42:24.246947+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:42:24.246947+0000","last_undegraded":"2024-02-23T00:42:24.246947+0000","last_fullsized":"2024-02-23T00:42:24.246947+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:13.006833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.006833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":491,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420538+0000","last_change":"2024-02-23T00:42:14.760378+0000","last_active":"2024-02-23T00:42:21.420538+0000","last_peered":"2024-02-23T00:42:21.420538+0000","last_clean":"2024-02-23T00:42:21.420538+0000","last_became_active":"2024-02-23T00:41:38.217979+0000","last_became_peered":"2024-02-23T00:41:38.217979+0000","last_unstale":"2024-02-23T00:42:21.420538+0000","last_undegraded":"2024-02-23T00:42:21.420538+0000","last_fullsized":"2024-02-23T00:42:21.420538+0000","mapping_epoch":678,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:42:14.760308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.760308+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":731,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246310+0000","last_change":"2024-02-23T00:42:21.442459+0000","last_active":"2024-02-23T00:42:24.246310+0000","last_peered":"2024-02-23T00:42:24.246310+0000","last_clean":"2024-02-23T00:42:24.246310+0000","last_became_active":"2024-02-23T00:41:37.238750+0000","last_became_peered":"2024-02-23T00:41:37.238750+0000","last_unstale":"2024-02-23T00:42:24.246310+0000","last_undegraded":"2024-02-23T00:42:24.246310+0000","last_fullsized":"2024-02-23T00:42:24.246310+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:21.442362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:21.442362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":231,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.251045+0000","last_change":"2024-02-23T00:42:20.334112+0000","last_active":"2024-02-23T00:42:24.251045+0000","last_peered":"2024-02-23T00:42:24.251045+0000","last_clean":"2024-02-23T00:42:24.251045+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:42:24.251045+0000","last_undegraded":"2024-02-23T00:42:24.251045+0000","last_fullsized":"2024-02-23T00:42:24.251045+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.334013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.334013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":226,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250773+0000","last_change":"2024-02-23T00:41:48.297291+0000","last_active":"2024-02-23T00:42:24.250773+0000","last_peered":"2024-02-23T00:42:24.250773+0000","last_clean":"2024-02-23T00:42:24.250773+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:42:24.250773+0000","last_undegraded":"2024-02-23T00:42:24.250773+0000","last_fullsized":"2024-02-23T00:42:24.250773+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:48.297220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:48.297220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":834,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250779+0000","last_change":"2024-02-23T00:40:51.316847+0000","last_active":"2024-02-23T00:42:24.250779+0000","last_peered":"2024-02-23T00:42:24.250779+0000","last_clean":"2024-02-23T00:42:24.250779+0000","last_became_active":"2024-02-23T00:40:51.316452+0000","last_became_peered":"2024-02-23T00:40:51.316452+0000","last_unstale":"2024-02-23T00:42:24.250779+0000","last_undegraded":"2024-02-23T00:42:24.250779+0000","last_fullsized":"2024-02-23T00:42:24.250779+0000","mapping_epoch":631,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":711,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244486+0000","last_change":"2024-02-23T00:42:19.655247+0000","last_active":"2024-02-23T00:42:24.244486+0000","last_peered":"2024-02-23T00:42:24.244486+0000","last_clean":"2024-02-23T00:42:24.244486+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:42:24.244486+0000","last_undegraded":"2024-02-23T00:42:24.244486+0000","last_fullsized":"2024-02-23T00:42:24.244486+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.655188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.655188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":706,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247858+0000","last_change":"2024-02-23T00:41:53.984748+0000","last_active":"2024-02-23T00:42:24.247858+0000","last_peered":"2024-02-23T00:42:24.247858+0000","last_clean":"2024-02-23T00:42:24.247858+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:42:24.247858+0000","last_undegraded":"2024-02-23T00:42:24.247858+0000","last_fullsized":"2024-02-23T00:42:24.247858+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.984679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.984679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":671,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245073+0000","last_change":"2024-02-23T00:41:02.451992+0000","last_active":"2024-02-23T00:42:24.245073+0000","last_peered":"2024-02-23T00:42:24.245073+0000","last_clean":"2024-02-23T00:42:24.245073+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:42:24.245073+0000","last_undegraded":"2024-02-23T00:42:24.245073+0000","last_fullsized":"2024-02-23T00:42:24.245073+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:02.451884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.451884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"688'39","reported_seq":249,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250745+0000","last_change":"2024-02-23T00:41:49.338035+0000","last_active":"2024-02-23T00:42:24.250745+0000","last_peered":"2024-02-23T00:42:24.250745+0000","last_clean":"2024-02-23T00:42:24.250745+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:42:24.250745+0000","last_undegraded":"2024-02-23T00:42:24.250745+0000","last_fullsized":"2024-02-23T00:42:24.250745+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'39","last_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_deep_scrub":"688'39","last_deep_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.337969+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":723,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246964+0000","last_change":"2024-02-23T00:41:41.955717+0000","last_active":"2024-02-23T00:42:24.246964+0000","last_peered":"2024-02-23T00:42:24.246964+0000","last_clean":"2024-02-23T00:42:24.246964+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:42:24.246964+0000","last_undegraded":"2024-02-23T00:42:24.246964+0000","last_fullsized":"2024-02-23T00:42:24.246964+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:41.955589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:41:41.955589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"681'12","reported_seq":907,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.255762+0000","last_change":"2024-02-23T00:41:37.247785+0000","last_active":"2024-02-23T00:42:25.255762+0000","last_peered":"2024-02-23T00:42:25.255762+0000","last_clean":"2024-02-23T00:42:25.255762+0000","last_became_active":"2024-02-23T00:41:37.228441+0000","last_became_peered":"2024-02-23T00:41:37.228441+0000","last_unstale":"2024-02-23T00:42:25.255762+0000","last_undegraded":"2024-02-23T00:42:25.255762+0000","last_fullsized":"2024-02-23T00:42:25.255762+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"606'11","last_scrub_stamp":"2024-02-23T00:40:45.004428+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.004428+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":646,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248248+0000","last_change":"2024-02-23T00:42:17.056253+0000","last_active":"2024-02-23T00:42:24.248248+0000","last_peered":"2024-02-23T00:42:24.248248+0000","last_clean":"2024-02-23T00:42:24.248248+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:42:24.248248+0000","last_undegraded":"2024-02-23T00:42:24.248248+0000","last_fullsized":"2024-02-23T00:42:24.248248+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:17.056153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.056153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":777,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247981+0000","last_change":"2024-02-23T00:41:02.799494+0000","last_active":"2024-02-23T00:42:24.247981+0000","last_peered":"2024-02-23T00:42:24.247981+0000","last_clean":"2024-02-23T00:42:24.247981+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:42:24.247981+0000","last_undegraded":"2024-02-23T00:42:24.247981+0000","last_fullsized":"2024-02-23T00:42:24.247981+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:41:02.799406+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.799406+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":57,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.419792+0000","last_change":"2024-02-23T00:41:47.719694+0000","last_active":"2024-02-23T00:42:21.419792+0000","last_peered":"2024-02-23T00:42:21.419792+0000","last_clean":"2024-02-23T00:42:21.419792+0000","last_became_active":"2024-02-23T00:41:37.235746+0000","last_became_peered":"2024-02-23T00:41:37.235746+0000","last_unstale":"2024-02-23T00:42:21.419792+0000","last_undegraded":"2024-02-23T00:42:21.419792+0000","last_fullsized":"2024-02-23T00:42:21.419792+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.719590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.719590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":274,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.245898+0000","last_change":"2024-02-23T00:40:50.453642+0000","last_active":"2024-02-23T00:42:25.245898+0000","last_peered":"2024-02-23T00:42:25.245898+0000","last_clean":"2024-02-23T00:42:25.245898+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:42:25.245898+0000","last_undegraded":"2024-02-23T00:42:25.245898+0000","last_fullsized":"2024-02-23T00:42:25.245898+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'11","last_scrub_stamp":"2024-02-23T00:40:50.453579+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.453579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":226,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250946+0000","last_change":"2024-02-23T00:41:03.320478+0000","last_active":"2024-02-23T00:42:24.250946+0000","last_peered":"2024-02-23T00:42:24.250946+0000","last_clean":"2024-02-23T00:42:24.250946+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:42:24.250946+0000","last_undegraded":"2024-02-23T00:42:24.250946+0000","last_fullsized":"2024-02-23T00:42:24.250946+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:03.320387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:03.320387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":719,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247272+0000","last_change":"2024-02-23T00:41:02.134977+0000","last_active":"2024-02-23T00:42:24.247272+0000","last_peered":"2024-02-23T00:42:24.247272+0000","last_clean":"2024-02-23T00:42:24.247272+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:42:24.247272+0000","last_undegraded":"2024-02-23T00:42:24.247272+0000","last_fullsized":"2024-02-23T00:42:24.247272+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'15","last_scrub_stamp":"2024-02-23T00:41:02.134898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.134898+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":733,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244968+0000","last_change":"2024-02-23T00:41:42.607771+0000","last_active":"2024-02-23T00:42:24.244968+0000","last_peered":"2024-02-23T00:42:24.244968+0000","last_clean":"2024-02-23T00:42:24.244968+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:42:24.244968+0000","last_undegraded":"2024-02-23T00:42:24.244968+0000","last_fullsized":"2024-02-23T00:42:24.244968+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.607642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.607642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":245,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250318+0000","last_change":"2024-02-23T00:41:37.253216+0000","last_active":"2024-02-23T00:42:24.250318+0000","last_peered":"2024-02-23T00:42:24.250318+0000","last_clean":"2024-02-23T00:42:24.250318+0000","last_became_active":"2024-02-23T00:41:37.252612+0000","last_became_peered":"2024-02-23T00:41:37.252612+0000","last_unstale":"2024-02-23T00:42:24.250318+0000","last_undegraded":"2024-02-23T00:42:24.250318+0000","last_fullsized":"2024-02-23T00:42:24.250318+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:30.370620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:39:30.370620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":676,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247691+0000","last_change":"2024-02-23T00:42:12.891119+0000","last_active":"2024-02-23T00:42:24.247691+0000","last_peered":"2024-02-23T00:42:24.247691+0000","last_clean":"2024-02-23T00:42:24.247691+0000","last_became_active":"2024-02-23T00:41:37.238185+0000","last_became_peered":"2024-02-23T00:41:37.238185+0000","last_unstale":"2024-02-23T00:42:24.247691+0000","last_undegraded":"2024-02-23T00:42:24.247691+0000","last_fullsized":"2024-02-23T00:42:24.247691+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.891052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.891052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":755,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249051+0000","last_change":"2024-02-23T00:41:33.715504+0000","last_active":"2024-02-23T00:42:24.249051+0000","last_peered":"2024-02-23T00:42:24.249051+0000","last_clean":"2024-02-23T00:42:24.249051+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:42:24.249051+0000","last_undegraded":"2024-02-23T00:42:24.249051+0000","last_fullsized":"2024-02-23T00:42:24.249051+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:41:33.715382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.715382+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":715,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244782+0000","last_change":"2024-02-23T00:41:37.230185+0000","last_active":"2024-02-23T00:42:24.244782+0000","last_peered":"2024-02-23T00:42:24.244782+0000","last_clean":"2024-02-23T00:42:24.244782+0000","last_became_active":"2024-02-23T00:41:37.229726+0000","last_became_peered":"2024-02-23T00:41:37.229726+0000","last_unstale":"2024-02-23T00:42:24.244782+0000","last_undegraded":"2024-02-23T00:42:24.244782+0000","last_fullsized":"2024-02-23T00:42:24.244782+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:56.203202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:40:56.203202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":710,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247374+0000","last_change":"2024-02-23T00:41:35.138434+0000","last_active":"2024-02-23T00:42:24.247374+0000","last_peered":"2024-02-23T00:42:24.247374+0000","last_clean":"2024-02-23T00:42:24.247374+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:42:24.247374+0000","last_undegraded":"2024-02-23T00:42:24.247374+0000","last_fullsized":"2024-02-23T00:42:24.247374+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.138319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.138319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":702,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244758+0000","last_change":"2024-02-23T00:42:22.669179+0000","last_active":"2024-02-23T00:42:24.244758+0000","last_peered":"2024-02-23T00:42:24.244758+0000","last_clean":"2024-02-23T00:42:24.244758+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:42:24.244758+0000","last_undegraded":"2024-02-23T00:42:24.244758+0000","last_fullsized":"2024-02-23T00:42:24.244758+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:42:22.669090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.669090+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":232,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249083+0000","last_change":"2024-02-23T00:40:12.128771+0000","last_active":"2024-02-23T00:42:24.249083+0000","last_peered":"2024-02-23T00:42:24.249083+0000","last_clean":"2024-02-23T00:42:24.249083+0000","last_became_active":"2024-02-23T00:38:49.217612+0000","last_became_peered":"2024-02-23T00:38:49.217612+0000","last_unstale":"2024-02-23T00:42:24.249083+0000","last_undegraded":"2024-02-23T00:42:24.249083+0000","last_fullsized":"2024-02-23T00:42:24.249083+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'10","last_scrub_stamp":"2024-02-23T00:40:12.128684+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.128684+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":697,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248064+0000","last_change":"2024-02-23T00:42:15.038961+0000","last_active":"2024-02-23T00:42:24.248064+0000","last_peered":"2024-02-23T00:42:24.248064+0000","last_clean":"2024-02-23T00:42:24.248064+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:42:24.248064+0000","last_undegraded":"2024-02-23T00:42:24.248064+0000","last_fullsized":"2024-02-23T00:42:24.248064+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.038897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.038897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":702,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244408+0000","last_change":"2024-02-23T00:41:34.564079+0000","last_active":"2024-02-23T00:42:24.244408+0000","last_peered":"2024-02-23T00:42:24.244408+0000","last_clean":"2024-02-23T00:42:24.244408+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:42:24.244408+0000","last_undegraded":"2024-02-23T00:42:24.244408+0000","last_fullsized":"2024-02-23T00:42:24.244408+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:34.563969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.563969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":643,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247372+0000","last_change":"2024-02-23T00:41:44.196162+0000","last_active":"2024-02-23T00:42:24.247372+0000","last_peered":"2024-02-23T00:42:24.247372+0000","last_clean":"2024-02-23T00:42:24.247372+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:42:24.247372+0000","last_undegraded":"2024-02-23T00:42:24.247372+0000","last_fullsized":"2024-02-23T00:42:24.247372+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.196088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.196088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":243,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249767+0000","last_change":"2024-02-23T00:40:43.233713+0000","last_active":"2024-02-23T00:42:24.249767+0000","last_peered":"2024-02-23T00:42:24.249767+0000","last_clean":"2024-02-23T00:42:24.249767+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:42:24.249767+0000","last_undegraded":"2024-02-23T00:42:24.249767+0000","last_fullsized":"2024-02-23T00:42:24.249767+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:40:43.233609+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:40:43.233609+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":705,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245325+0000","last_change":"2024-02-23T00:40:53.462082+0000","last_active":"2024-02-23T00:42:24.245325+0000","last_peered":"2024-02-23T00:42:24.245325+0000","last_clean":"2024-02-23T00:42:24.245325+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:42:24.245325+0000","last_undegraded":"2024-02-23T00:42:24.245325+0000","last_fullsized":"2024-02-23T00:42:24.245325+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:53.462018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.462018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":700,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.251275+0000","last_change":"2024-02-23T00:41:44.520521+0000","last_active":"2024-02-23T00:42:24.251275+0000","last_peered":"2024-02-23T00:42:24.251275+0000","last_clean":"2024-02-23T00:42:24.251275+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:42:24.251275+0000","last_undegraded":"2024-02-23T00:42:24.251275+0000","last_fullsized":"2024-02-23T00:42:24.251275+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.520415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.520415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":226,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250237+0000","last_change":"2024-02-23T00:41:37.096978+0000","last_active":"2024-02-23T00:42:24.250237+0000","last_peered":"2024-02-23T00:42:24.250237+0000","last_clean":"2024-02-23T00:42:24.250237+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:42:24.250237+0000","last_undegraded":"2024-02-23T00:42:24.250237+0000","last_fullsized":"2024-02-23T00:42:24.250237+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:37.096906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:37.096906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":612,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420266+0000","last_change":"2024-02-23T00:42:10.241262+0000","last_active":"2024-02-23T00:42:21.420266+0000","last_peered":"2024-02-23T00:42:21.420266+0000","last_clean":"2024-02-23T00:42:21.420266+0000","last_became_active":"2024-02-23T00:42:10.240911+0000","last_became_peered":"2024-02-23T00:42:10.240911+0000","last_unstale":"2024-02-23T00:42:21.420266+0000","last_undegraded":"2024-02-23T00:42:21.420266+0000","last_fullsized":"2024-02-23T00:42:21.420266+0000","mapping_epoch":707,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":701,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.246764+0000","last_change":"2024-02-23T00:40:55.816232+0000","last_active":"2024-02-23T00:42:24.246764+0000","last_peered":"2024-02-23T00:42:24.246764+0000","last_clean":"2024-02-23T00:42:24.246764+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:42:24.246764+0000","last_undegraded":"2024-02-23T00:42:24.246764+0000","last_fullsized":"2024-02-23T00:42:24.246764+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:55.816155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:55.816155+0000","last_clean_scrub_stamp":"2024-02-23T00:40:55.816155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":895,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-02-23T00:42:25.250593+0000","last_change":"2024-02-23T00:40:45.431895+0000","last_active":"2024-02-23T00:42:25.250593+0000","last_peered":"2024-02-23T00:42:25.250593+0000","last_clean":"2024-02-23T00:42:25.250593+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:42:25.250593+0000","last_undegraded":"2024-02-23T00:42:25.250593+0000","last_fullsized":"2024-02-23T00:42:25.250593+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:40:45.431814+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.431814+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":695,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249433+0000","last_change":"2024-02-23T00:42:14.427990+0000","last_active":"2024-02-23T00:42:24.249433+0000","last_peered":"2024-02-23T00:42:24.249433+0000","last_clean":"2024-02-23T00:42:24.249433+0000","last_became_active":"2024-02-23T00:41:37.229426+0000","last_became_peered":"2024-02-23T00:41:37.229426+0000","last_unstale":"2024-02-23T00:42:24.249433+0000","last_undegraded":"2024-02-23T00:42:24.249433+0000","last_fullsized":"2024-02-23T00:42:24.249433+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.427861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.427861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":914,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245245+0000","last_change":"2024-02-23T00:42:20.635986+0000","last_active":"2024-02-23T00:42:24.245245+0000","last_peered":"2024-02-23T00:42:24.245245+0000","last_clean":"2024-02-23T00:42:24.245245+0000","last_became_active":"2024-02-23T00:41:55.327964+0000","last_became_peered":"2024-02-23T00:41:55.327964+0000","last_unstale":"2024-02-23T00:42:24.245245+0000","last_undegraded":"2024-02-23T00:42:24.245245+0000","last_fullsized":"2024-02-23T00:42:24.245245+0000","mapping_epoch":692,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:42:20.635897+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.635897+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":250,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249854+0000","last_change":"2024-02-23T00:42:14.278845+0000","last_active":"2024-02-23T00:42:24.249854+0000","last_peered":"2024-02-23T00:42:24.249854+0000","last_clean":"2024-02-23T00:42:24.249854+0000","last_became_active":"2024-02-23T00:41:37.228694+0000","last_became_peered":"2024-02-23T00:41:37.228694+0000","last_unstale":"2024-02-23T00:42:24.249854+0000","last_undegraded":"2024-02-23T00:42:24.249854+0000","last_fullsized":"2024-02-23T00:42:24.249854+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.278782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.278782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":52,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420427+0000","last_change":"2024-02-23T00:41:37.248384+0000","last_active":"2024-02-23T00:42:21.420427+0000","last_peered":"2024-02-23T00:42:21.420427+0000","last_clean":"2024-02-23T00:42:21.420427+0000","last_became_active":"2024-02-23T00:41:37.247946+0000","last_became_peered":"2024-02-23T00:41:37.247946+0000","last_unstale":"2024-02-23T00:42:21.420427+0000","last_undegraded":"2024-02-23T00:42:21.420427+0000","last_fullsized":"2024-02-23T00:42:21.420427+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:59.777391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:59.777391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":652,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.251698+0000","last_change":"2024-02-23T00:41:53.497861+0000","last_active":"2024-02-23T00:42:24.251698+0000","last_peered":"2024-02-23T00:42:24.251698+0000","last_clean":"2024-02-23T00:42:24.251698+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:42:24.251698+0000","last_undegraded":"2024-02-23T00:42:24.251698+0000","last_fullsized":"2024-02-23T00:42:24.251698+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.497792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.497792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":842,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247201+0000","last_change":"2024-02-23T00:42:16.497262+0000","last_active":"2024-02-23T00:42:24.247201+0000","last_peered":"2024-02-23T00:42:24.247201+0000","last_clean":"2024-02-23T00:42:24.247201+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:42:24.247201+0000","last_undegraded":"2024-02-23T00:42:24.247201+0000","last_fullsized":"2024-02-23T00:42:24.247201+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:42:16.497198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.497198+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":714,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244579+0000","last_change":"2024-02-23T00:41:48.632172+0000","last_active":"2024-02-23T00:42:24.244579+0000","last_peered":"2024-02-23T00:42:24.244579+0000","last_clean":"2024-02-23T00:42:24.244579+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:42:24.244579+0000","last_undegraded":"2024-02-23T00:42:24.244579+0000","last_fullsized":"2024-02-23T00:42:24.244579+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:48.632098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:41:48.632098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":713,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245018+0000","last_change":"2024-02-23T00:41:37.230024+0000","last_active":"2024-02-23T00:42:24.245018+0000","last_peered":"2024-02-23T00:42:24.245018+0000","last_clean":"2024-02-23T00:42:24.245018+0000","last_became_active":"2024-02-23T00:41:37.229323+0000","last_became_peered":"2024-02-23T00:41:37.229323+0000","last_unstale":"2024-02-23T00:42:24.245018+0000","last_undegraded":"2024-02-23T00:42:24.245018+0000","last_fullsized":"2024-02-23T00:42:24.245018+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:57.174945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:40:57.174945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":228,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250546+0000","last_change":"2024-02-23T00:40:11.172134+0000","last_active":"2024-02-23T00:42:24.250546+0000","last_peered":"2024-02-23T00:42:24.250546+0000","last_clean":"2024-02-23T00:42:24.250546+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:42:24.250546+0000","last_undegraded":"2024-02-23T00:42:24.250546+0000","last_fullsized":"2024-02-23T00:42:24.250546+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:11.172065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.172065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":810,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250580+0000","last_change":"2024-02-23T00:40:58.399481+0000","last_active":"2024-02-23T00:42:24.250580+0000","last_peered":"2024-02-23T00:42:24.250580+0000","last_clean":"2024-02-23T00:42:24.250580+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:42:24.250580+0000","last_undegraded":"2024-02-23T00:42:24.250580+0000","last_fullsized":"2024-02-23T00:42:24.250580+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_deep_scrub":"453'44","last_deep_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.399368+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":226,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.251068+0000","last_change":"2024-02-23T00:40:07.255442+0000","last_active":"2024-02-23T00:42:24.251068+0000","last_peered":"2024-02-23T00:42:24.251068+0000","last_clean":"2024-02-23T00:42:24.251068+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:42:24.251068+0000","last_undegraded":"2024-02-23T00:42:24.251068+0000","last_fullsized":"2024-02-23T00:42:24.251068+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:07.255349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:07.255349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":536,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250432+0000","last_change":"2024-02-23T00:41:50.257038+0000","last_active":"2024-02-23T00:42:24.250432+0000","last_peered":"2024-02-23T00:42:24.250432+0000","last_clean":"2024-02-23T00:42:24.250432+0000","last_became_active":"2024-02-23T00:41:50.256751+0000","last_became_peered":"2024-02-23T00:41:50.256751+0000","last_unstale":"2024-02-23T00:42:24.250432+0000","last_undegraded":"2024-02-23T00:42:24.250432+0000","last_fullsized":"2024-02-23T00:42:24.250432+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:07.610329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:07.610329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":641,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247919+0000","last_change":"2024-02-23T00:40:53.782744+0000","last_active":"2024-02-23T00:42:24.247919+0000","last_peered":"2024-02-23T00:42:24.247919+0000","last_clean":"2024-02-23T00:42:24.247919+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:42:24.247919+0000","last_undegraded":"2024-02-23T00:42:24.247919+0000","last_fullsized":"2024-02-23T00:42:24.247919+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:53.782669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.782669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":910,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248855+0000","last_change":"2024-02-23T00:41:40.216294+0000","last_active":"2024-02-23T00:42:24.248855+0000","last_peered":"2024-02-23T00:42:24.248855+0000","last_clean":"2024-02-23T00:42:24.248855+0000","last_became_active":"2024-02-23T00:41:40.215919+0000","last_became_peered":"2024-02-23T00:41:40.215919+0000","last_unstale":"2024-02-23T00:42:24.248855+0000","last_undegraded":"2024-02-23T00:42:24.248855+0000","last_fullsized":"2024-02-23T00:42:24.248855+0000","mapping_epoch":680,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":415,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249672+0000","last_change":"2024-02-23T00:42:14.878836+0000","last_active":"2024-02-23T00:42:24.249672+0000","last_peered":"2024-02-23T00:42:24.249672+0000","last_clean":"2024-02-23T00:42:24.249672+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:42:24.249672+0000","last_undegraded":"2024-02-23T00:42:24.249672+0000","last_fullsized":"2024-02-23T00:42:24.249672+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.878770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.878770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":707,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247343+0000","last_change":"2024-02-23T00:41:37.228970+0000","last_active":"2024-02-23T00:42:24.247343+0000","last_peered":"2024-02-23T00:42:24.247343+0000","last_clean":"2024-02-23T00:42:24.247343+0000","last_became_active":"2024-02-23T00:41:37.228310+0000","last_became_peered":"2024-02-23T00:41:37.228310+0000","last_unstale":"2024-02-23T00:42:24.247343+0000","last_undegraded":"2024-02-23T00:42:24.247343+0000","last_fullsized":"2024-02-23T00:42:24.247343+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:46.004307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:46.004307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":657,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.248600+0000","last_change":"2024-02-23T00:42:19.323714+0000","last_active":"2024-02-23T00:42:24.248600+0000","last_peered":"2024-02-23T00:42:24.248600+0000","last_clean":"2024-02-23T00:42:24.248600+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:42:24.248600+0000","last_undegraded":"2024-02-23T00:42:24.248600+0000","last_fullsized":"2024-02-23T00:42:24.248600+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.323643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.323643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":241,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250122+0000","last_change":"2024-02-23T00:41:50.256190+0000","last_active":"2024-02-23T00:42:24.250122+0000","last_peered":"2024-02-23T00:42:24.250122+0000","last_clean":"2024-02-23T00:42:24.250122+0000","last_became_active":"2024-02-23T00:41:50.255862+0000","last_became_peered":"2024-02-23T00:41:50.255862+0000","last_unstale":"2024-02-23T00:42:24.250122+0000","last_undegraded":"2024-02-23T00:42:24.250122+0000","last_fullsized":"2024-02-23T00:42:24.250122+0000","mapping_epoch":688,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:41:00.643135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:00.643135+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":723,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.251208+0000","last_change":"2024-02-23T00:41:37.232071+0000","last_active":"2024-02-23T00:42:24.251208+0000","last_peered":"2024-02-23T00:42:24.251208+0000","last_clean":"2024-02-23T00:42:24.251208+0000","last_became_active":"2024-02-23T00:41:37.230054+0000","last_became_peered":"2024-02-23T00:41:37.230054+0000","last_unstale":"2024-02-23T00:42:24.251208+0000","last_undegraded":"2024-02-23T00:42:24.251208+0000","last_fullsized":"2024-02-23T00:42:24.251208+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:07.753676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:41:07.753676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":700,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249614+0000","last_change":"2024-02-23T00:40:19.660039+0000","last_active":"2024-02-23T00:42:24.249614+0000","last_peered":"2024-02-23T00:42:24.249614+0000","last_clean":"2024-02-23T00:42:24.249614+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:42:24.249614+0000","last_undegraded":"2024-02-23T00:42:24.249614+0000","last_fullsized":"2024-02-23T00:42:24.249614+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.659970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:38.549228+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.659970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":667,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.245282+0000","last_change":"2024-02-23T00:41:47.254406+0000","last_active":"2024-02-23T00:42:24.245282+0000","last_peered":"2024-02-23T00:42:24.245282+0000","last_clean":"2024-02-23T00:42:24.245282+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:42:24.245282+0000","last_undegraded":"2024-02-23T00:42:24.245282+0000","last_fullsized":"2024-02-23T00:42:24.245282+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.254328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.254328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":852,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.247395+0000","last_change":"2024-02-23T00:41:03.631984+0000","last_active":"2024-02-23T00:42:24.247395+0000","last_peered":"2024-02-23T00:42:24.247395+0000","last_clean":"2024-02-23T00:42:24.247395+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:42:24.247395+0000","last_undegraded":"2024-02-23T00:42:24.247395+0000","last_fullsized":"2024-02-23T00:42:24.247395+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:41:03.631913+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:41:03.631913+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":245,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.250611+0000","last_change":"2024-02-23T00:41:37.241505+0000","last_active":"2024-02-23T00:42:24.250611+0000","last_peered":"2024-02-23T00:42:24.250611+0000","last_clean":"2024-02-23T00:42:24.250611+0000","last_became_active":"2024-02-23T00:41:37.241142+0000","last_became_peered":"2024-02-23T00:41:37.241142+0000","last_unstale":"2024-02-23T00:42:24.250611+0000","last_undegraded":"2024-02-23T00:42:24.250611+0000","last_fullsized":"2024-02-23T00:42:24.250611+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.346259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.346259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":357,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244213+0000","last_change":"2024-02-23T00:40:52.433576+0000","last_active":"2024-02-23T00:42:24.244213+0000","last_peered":"2024-02-23T00:42:24.244213+0000","last_clean":"2024-02-23T00:42:24.244213+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:42:24.244213+0000","last_undegraded":"2024-02-23T00:42:24.244213+0000","last_fullsized":"2024-02-23T00:42:24.244213+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.433499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.433499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":134,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249735+0000","last_change":"2024-02-23T00:41:34.730482+0000","last_active":"2024-02-23T00:42:24.249735+0000","last_peered":"2024-02-23T00:42:24.249735+0000","last_clean":"2024-02-23T00:42:24.249735+0000","last_became_active":"2024-02-23T00:40:23.199068+0000","last_became_peered":"2024-02-23T00:40:23.199068+0000","last_unstale":"2024-02-23T00:42:24.249735+0000","last_undegraded":"2024-02-23T00:42:24.249735+0000","last_fullsized":"2024-02-23T00:42:24.249735+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:34.730403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.730403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":1001,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.249626+0000","last_change":"2024-02-23T00:41:41.223239+0000","last_active":"2024-02-23T00:42:24.249626+0000","last_peered":"2024-02-23T00:42:24.249626+0000","last_clean":"2024-02-23T00:42:24.249626+0000","last_became_active":"2024-02-23T00:41:41.222892+0000","last_became_peered":"2024-02-23T00:41:41.222892+0000","last_unstale":"2024-02-23T00:42:24.249626+0000","last_undegraded":"2024-02-23T00:42:24.249626+0000","last_fullsized":"2024-02-23T00:42:24.249626+0000","mapping_epoch":681,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:41:34.076323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.076323+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":52,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-02-23T00:42:21.420558+0000","last_change":"2024-02-23T00:41:37.235163+0000","last_active":"2024-02-23T00:42:21.420558+0000","last_peered":"2024-02-23T00:42:21.420558+0000","last_clean":"2024-02-23T00:42:21.420558+0000","last_became_active":"2024-02-23T00:41:37.234830+0000","last_became_peered":"2024-02-23T00:41:37.234830+0000","last_unstale":"2024-02-23T00:42:21.420558+0000","last_undegraded":"2024-02-23T00:42:21.420558+0000","last_fullsized":"2024-02-23T00:42:21.420558+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:42.783207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:42.783207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":53,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244497+0000","last_change":"2024-02-23T00:42:13.611259+0000","last_active":"2024-02-23T00:42:24.244497+0000","last_peered":"2024-02-23T00:42:24.244497+0000","last_clean":"2024-02-23T00:42:24.244497+0000","last_became_active":"2024-02-23T00:41:50.253284+0000","last_became_peered":"2024-02-23T00:41:50.253284+0000","last_unstale":"2024-02-23T00:42:24.244497+0000","last_undegraded":"2024-02-23T00:42:24.244497+0000","last_fullsized":"2024-02-23T00:42:24.244497+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:13.611193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.611193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":666,"reported_epoch":722,"state":"active+clean","last_fresh":"2024-02-23T00:42:24.244854+0000","last_change":"2024-02-23T00:42:20.220171+0000","last_active":"2024-02-23T00:42:24.244854+0000","last_peered":"2024-02-23T00:42:24.244854+0000","last_clean":"2024-02-23T00:42:24.244854+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:42:24.244854+0000","last_undegraded":"2024-02-23T00:42:24.244854+0000","last_fullsized":"2024-02-23T00:42:24.244854+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.220094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.220094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":247,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":81,"acting":81,"num_store_stats":7},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":81,"num_objects":16,"num_object_clones":1,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":36,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":24,"acting":24,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":12,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3757,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":873,"num_read_kb":709,"num_write":524,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":4401,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":732,"ondisk_log_size":732,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":13,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":586,"num_read_kb":586,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":39,"acting":39,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332281,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306472,"kb_used_data":9236,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93438680,"statfs":{"total":95995035648,"available":95681208320,"internally_reserved":0,"allocated":9457664,"data_stored":5427355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.34200000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.45600000000000002,"15min":0.67800000000000005},"min":{"1min":0.35399999999999998,"5min":0.23999999999999999,"15min":0.217},"max":{"1min":0.63700000000000001,"5min":1.23,"15min":31.175000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.64700000000000002,"15min":0.73899999999999999},"min":{"1min":0.34300000000000003,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":0.71899999999999997,"5min":11.066000000000001,"15min":29.423999999999999},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.68799999999999994,"15min":0.72199999999999998},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.28100000000000003},"max":{"1min":0.67600000000000005,"5min":8.0099999999999998,"15min":16.603999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.58999999999999997,"15min":0.64000000000000001},"min":{"1min":0.32100000000000001,"5min":0.30599999999999999,"15min":0.26600000000000001},"max":{"1min":0.68899999999999995,"5min":5.7229999999999999,"15min":7.5339999999999998},"last":0.42799999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.60599999999999998,"15min":1.0109999999999999},"min":{"1min":0.29499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.89000000000000001,"5min":8.0449999999999999,"15min":57.729999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54100000000000004,"15min":0.91400000000000003},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72499999999999998,"5min":3.8479999999999999,"15min":27.573},"last":0.46300000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.39300000000000002,"15min":0.41599999999999998},"min":{"1min":0.28699999999999998,"5min":0.214,"15min":0.214},"max":{"1min":0.72999999999999998,"5min":1.2589999999999999,"15min":1.706},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46999999999999997,"15min":0.45600000000000002},"min":{"1min":0.24299999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.74299999999999999,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.49399999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.40699999999999997,"15min":0.42599999999999999},"min":{"1min":0.28599999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68899999999999995,"5min":0.746,"15min":1.0209999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.44900000000000001,"15min":0.46300000000000002},"min":{"1min":0.32900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67800000000000005,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.48599999999999999},"min":{"1min":0.35099999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.72799999999999998,"5min":1.3129999999999999,"15min":2.1339999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.502,"15min":0.49199999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.752,"5min":5.194,"15min":5.194},"last":0.53800000000000003}]}]},{"osd":6,"up_from":44,"seq":188978561214,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306464,"kb_used_data":9232,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93438688,"statfs":{"total":95995035648,"available":95681216512,"internally_reserved":0,"allocated":9453568,"data_stored":5442354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.441},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53400000000000003,"15min":1.024},"min":{"1min":0.40100000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.66100000000000003,"5min":6.056,"15min":59.347999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.75900000000000001,"15min":1.3799999999999999},"min":{"1min":0.41299999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.1619999999999999,"5min":18.423999999999999,"15min":87.221000000000004},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.77300000000000002,"15min":0.90500000000000003},"min":{"1min":0.35699999999999998,"5min":0.214,"15min":0.214},"max":{"1min":1.1120000000000001,"5min":18.381,"15min":25.550999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66100000000000003,"15min":0.91700000000000004},"min":{"1min":0.375,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.9460000000000002,"5min":6.1749999999999998,"15min":40.914000000000001},"last":0.39800000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63800000000000001,"15min":0.85799999999999998},"min":{"1min":0.40999999999999998,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":2.859,"5min":7.1630000000000003,"15min":31.309999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.754,"15min":1.4059999999999999},"min":{"1min":0.44600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72299999999999998,"5min":18.448,"15min":86.311999999999998},"last":0.47599999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.53500000000000003,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60499999999999998,"5min":7.8769999999999998,"15min":7.8769999999999998},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48199999999999998,"15min":0.53000000000000003},"min":{"1min":0.307,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":6.0300000000000002,"15min":17.718},"last":0.34300000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.54900000000000004,"15min":0.52200000000000002},"min":{"1min":0.30599999999999999,"5min":0.27800000000000002,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":6.7999999999999998,"15min":8.0990000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.505},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82599999999999996,"5min":6.0780000000000003,"15min":8.1370000000000005},"last":0.311}]},{"osd":7,"last update":"Fri Feb 23 00:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.61499999999999999},"min":{"1min":0.34200000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69599999999999995,"5min":6.4279999999999999,"15min":17.881},"last":0.372},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.623,"15min":0.64500000000000002},"min":{"1min":0.33000000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76000000000000001,"5min":18.475000000000001,"15min":18.475000000000001},"last":0.42299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575061,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310608,"kb_used_data":9280,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93434544,"statfs":{"total":95995035648,"available":95676973056,"internally_reserved":0,"allocated":9502720,"data_stored":5442825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.252},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.34699999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.35999999999999999,"15min":0.67100000000000004},"min":{"1min":0.185,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.677,"5min":3.9980000000000002,"15min":35.313000000000002},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.38200000000000001,"15min":0.58299999999999996},"min":{"1min":0.20599999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.123,"5min":2.7999999999999998,"15min":27.908999999999999},"last":0.28100000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.435,"15min":0.64900000000000002},"min":{"1min":0.26200000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.6279999999999999,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.308},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40600000000000003,"15min":0.63900000000000001},"min":{"1min":0.26000000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.71,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.32800000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58399999999999996,"15min":0.70699999999999996},"min":{"1min":0.39000000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.877,"5min":3.4900000000000002,"15min":16.231000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.52700000000000002,"15min":0.71599999999999997},"min":{"1min":0.32200000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":29.212},"last":0.36799999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.71999999999999997},"min":{"1min":0.37,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76400000000000001,"5min":2.8410000000000002,"15min":16.294},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58599999999999997,"15min":0.68500000000000005},"min":{"1min":0.39700000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.238,"5min":4.3380000000000001,"15min":11.105},"last":0.42499999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.53600000000000003,"15min":0.754},"min":{"1min":0.39500000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.76,"5min":2.4049999999999998,"15min":29.187000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60999999999999999,"15min":0.749},"min":{"1min":0.44,"5min":0.312,"15min":0.312},"max":{"1min":0.90500000000000003,"5min":2.927,"15min":16.369},"last":0.436}]},{"osd":7,"last update":"Fri Feb 23 00:41:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.624,"15min":0.78800000000000003},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.92000000000000004,"5min":5.2119999999999997,"15min":16.395},"last":0.501},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60599999999999998,"15min":0.66800000000000004},"min":{"1min":0.35999999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.27,"5min":3.7669999999999999,"15min":11.135999999999999},"last":0.49099999999999999}]}]},{"osd":0,"up_from":659,"seq":2830383448078,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312120,"kb_used_data":9188,"kb_used_omap":85,"kb_used_meta":302826,"kb_avail":93433032,"statfs":{"total":95995035648,"available":95675424768,"internally_reserved":0,"allocated":9408512,"data_stored":5398893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":310093952},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.0880000000000001,"5min":5.0880000000000001,"15min":5.0880000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.0540000000000003,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.309}]},{"osd":3,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":11.891,"5min":11.891,"15min":11.891},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":15.864000000000001,"5min":15.864000000000001,"15min":15.864000000000001},"last":0.34100000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.56599999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.70199999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.68000000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":0.747},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.72399999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411537,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318740,"kb_used_data":9220,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426412,"statfs":{"total":95995035648,"available":95668645888,"internally_reserved":0,"allocated":9441280,"data_stored":5432660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.29799999999999999,"5min":0.43099999999999999,"15min":0.59699999999999998},"min":{"1min":0.17999999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.436,"5min":7.7110000000000003,"15min":19.324999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.30199999999999999,"5min":0.495,"15min":0.755},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.45000000000000001,"5min":7.9779999999999998,"15min":35.052},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.501,"15min":0.73199999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.46999999999999997,"5min":10.032,"15min":43.5},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.53000000000000003,"15min":0.68400000000000005},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.5,"5min":11.815,"15min":27.414000000000001},"last":0.51400000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:41:24 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.67600000000000005,"15min":0.66700000000000004},"min":{"1min":0.29499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.83399999999999996,"5min":10.106,"15min":14.614000000000001},"last":0.62},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.33000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.94399999999999995,"5min":7.766,"15min":7.766},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:42:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.68400000000000005,"15min":0.71399999999999997},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":1.248,"5min":7.915,"15min":17.765000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.64800000000000002,"15min":0.67200000000000004},"min":{"1min":0.371,"5min":0.312,"15min":0.312},"max":{"1min":0.83899999999999997,"5min":10.138,"15min":14.589},"last":0.79600000000000004}]},{"osd":6,"last update":"Fri Feb 23 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.628},"min":{"1min":0.42399999999999999,"5min":0.316,"15min":0.29699999999999999},"max":{"1min":0.81100000000000005,"5min":2.2250000000000001,"15min":7.7869999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71999999999999997,"15min":0.79300000000000004},"min":{"1min":0.41799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.218,"5min":7.9589999999999996,"15min":20.574999999999999},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.72599999999999998,"15min":1.073},"min":{"1min":0.38400000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.85799999999999998,"5min":7.9480000000000004,"15min":66.632000000000005},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.72999999999999998,"15min":0.84699999999999998},"min":{"1min":0.41599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0149999999999999,"5min":10.163,"15min":19.603999999999999},"last":0.75600000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149901,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314696,"kb_used_data":9272,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93430456,"statfs":{"total":95995035648,"available":95672786944,"internally_reserved":0,"allocated":9494528,"data_stored":5436986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.30499999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:42:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.60899999999999999},"min":{"1min":0.21299999999999999,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":3.2970000000000002,"5min":3.2970000000000002,"15min":54.351999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.78100000000000003},"min":{"1min":0.252,"5min":0.219,"15min":0.219},"max":{"1min":2.1680000000000001,"5min":3.7120000000000002,"15min":55.337000000000003},"last":0.29399999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:42:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.442,"15min":0.75800000000000001},"min":{"1min":0.27000000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.70199999999999996,"5min":3.7570000000000001,"15min":55.368000000000002},"last":0.434},{"interface":"front","average":{"1min":0.375,"5min":0.442,"15min":0.73199999999999998},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.185},"max":{"1min":0.52800000000000002,"5min":4.181,"15min":54.383000000000003},"last":0.33600000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.63300000000000001,"15min":0.93799999999999994},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.27900000000000003},"max":{"1min":0.80900000000000005,"5min":6.8799999999999999,"15min":54.530000000000001},"last":0.38},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.59199999999999997,"15min":0.625},"min":{"1min":0.40899999999999997,"5min":0.30199999999999999,"15min":0.24099999999999999},"max":{"1min":0.79200000000000004,"5min":4.9829999999999997,"15min":8.0359999999999996},"last":0.56799999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54600000000000004,"15min":0.876},"min":{"1min":0.372,"5min":0.34699999999999998,"15min":0.28899999999999998},"max":{"1min":0.80300000000000005,"5min":3.8180000000000001,"15min":55.401000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.65700000000000003,"15min":0.86499999999999999},"min":{"1min":0.40000000000000002,"5min":0.32700000000000001,"15min":0.30299999999999999},"max":{"1min":0.75700000000000001,"5min":7.0179999999999998,"15min":54.454999999999998},"last":0.49299999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.76800000000000002,"15min":0.85399999999999998},"min":{"1min":0.42999999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.7069999999999999,"5min":9.7490000000000006,"15min":54.305},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.86599999999999999,"15min":1.0389999999999999},"min":{"1min":0.374,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":8.5860000000000003,"5min":9.6820000000000004,"15min":55.424999999999997},"last":0.47999999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.627,"15min":0.78800000000000003},"min":{"1min":0.437,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81699999999999995,"5min":4.9790000000000001,"15min":54.222000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.92800000000000005,"15min":1.028},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76500000000000001,"5min":23.928000000000001,"15min":55.460000000000001},"last":0.61399999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986376,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310540,"kb_used_data":9212,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93434612,"statfs":{"total":95995035648,"available":95677042688,"internally_reserved":0,"allocated":9433088,"data_stored":5410355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.503},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.495}]},{"osd":1,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.69299999999999995,"15min":1.1020000000000001},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.64000000000000001,"5min":8.1199999999999992,"15min":41.137999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.65200000000000002,"15min":0.93000000000000005},"min":{"1min":0.40600000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":10.952999999999999,"15min":26.210000000000001},"last":0.47399999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.72199999999999998,"15min":1.1910000000000001},"min":{"1min":0.36699999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64100000000000001,"5min":9.3490000000000002,"15min":67.253},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.83099999999999996,"15min":1.1240000000000001},"min":{"1min":0.374,"5min":0.251,"15min":0.251},"max":{"1min":0.67100000000000004,"5min":14.442,"15min":41.927999999999997},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.68999999999999995,"15min":1.1240000000000001},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.627,"5min":9.5579999999999998,"15min":36.301000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.63100000000000001,"15min":1.1970000000000001},"min":{"1min":0.38200000000000001,"5min":0.26700000000000002,"15min":0.24299999999999999},"max":{"1min":0.69499999999999995,"5min":5.1879999999999997,"15min":42.064},"last":0.44600000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38900000000000001,"15min":0.56599999999999995},"min":{"1min":0.26900000000000002,"5min":0.219,"15min":0.217},"max":{"1min":0.58499999999999996,"5min":1.4279999999999999,"15min":14.242000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.435,"15min":0.60899999999999999},"min":{"1min":0.30499999999999999,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.63700000000000001,"5min":1.4079999999999999,"15min":14.124000000000001},"last":0.46500000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46700000000000003,"15min":0.57599999999999996},"min":{"1min":0.315,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":0.67000000000000004,"5min":1.734,"15min":8.0730000000000004},"last":0.378},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.54000000000000004,"15min":0.67300000000000004},"min":{"1min":0.30299999999999999,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.61599999999999999,"5min":13.339,"15min":45.814999999999998},"last":0.35699999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:41:36 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.55900000000000005,"15min":0.63700000000000001},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":7.9210000000000003,"15min":17.178999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.71599999999999997,"15min":0.71699999999999997},"min":{"1min":0.34200000000000003,"5min":0.307,"15min":0.28299999999999997},"max":{"1min":1.385,"5min":15.85,"15min":15.85},"last":0.40799999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757443,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322836,"kb_used_data":9220,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93422316,"statfs":{"total":95995035648,"available":95664451584,"internally_reserved":0,"allocated":9441280,"data_stored":5428815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.498}]},{"osd":1,"last update":"Fri Feb 23 00:41:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47099999999999997,"15min":0.751},"min":{"1min":0.31900000000000001,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":1.1180000000000001,"5min":3.7629999999999999,"15min":38.838000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.50800000000000001,"15min":1.1759999999999999},"min":{"1min":0.26600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60399999999999998,"5min":3.7250000000000001,"15min":127.794},"last":0.44400000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:41:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.70799999999999996,"15min":0.95599999999999996},"min":{"1min":0.32400000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.2969999999999999,"5min":11.435,"15min":37.442999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.48799999999999999,"15min":0.69599999999999995},"min":{"1min":0.28999999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.67500000000000004,"5min":3.734,"15min":23.297999999999998},"last":0.32600000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:41:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.74299999999999999,"15min":0.82499999999999996},"min":{"1min":0.375,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.3300000000000001,"5min":12.507,"15min":24.414999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.90700000000000003},"min":{"1min":0.36699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.075,"5min":3.8119999999999998,"15min":40.865000000000002},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:41:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.376,"15min":0.78100000000000003},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.219},"max":{"1min":0.69099999999999995,"5min":2.8940000000000001,"15min":103.13},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.27200000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.66200000000000003,"5min":11.381,"15min":12.795999999999999},"last":0.28999999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.56899999999999995,"15min":0.54800000000000004},"min":{"1min":0.32200000000000001,"5min":0.26400000000000001,"15min":0.26200000000000001},"max":{"1min":0.70799999999999996,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.373},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.82299999999999995},"min":{"1min":0.307,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.75,"5min":1.5229999999999999,"15min":103.61499999999999},"last":0.47199999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:41:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.59699999999999998,"15min":0.56000000000000005},"min":{"1min":0.308,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73499999999999999,"5min":11.478,"15min":11.478},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.52400000000000002,"15min":0.52000000000000002},"min":{"1min":0.34399999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72499999999999998,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:42:28.627 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-23T00:42:28.627 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set iscsi pg_num 23 2024-02-23T00:42:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:27.194394+0000 osd.3 (osd.3) 228 : cluster [DBG] 5.13 deep-scrub starts 2024-02-23T00:42:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:27.197286+0000 osd.3 (osd.3) 229 : cluster [DBG] 5.13 deep-scrub ok 2024-02-23T00:42:28.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:27.909843+0000 osd.4 (osd.4) 321 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:42:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:27.911593+0000 osd.4 (osd.4) 322 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:42:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:28.217218+0000 mon.a (mon.0) 3659 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:42:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:28.223760+0000 mon.a (mon.0) 3660 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-126", "mode": "writeback"}]': finished 2024-02-23T00:42:28.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:28.223817+0000 mon.a (mon.0) 3661 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-02-23T00:42:28.956 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:28.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:29.061 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:29.093 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:29.114 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:29.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:29.217 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:29.245 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:29.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.274+0000 7f300f7fe700 1 -- 172.21.15.87:0/1264766427 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v727) v1 ==== 87+0+0 (secure 0 0 0) 0x7f3010050290 con 0x7f30200fdbb0 2024-02-23T00:42:29.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 10 librados: watch_flush enter 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 10 librados: watch_flush exit 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 20 client.38264.objecter shutdown clearing up homeless session... 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 10 client.38264.objecter successfully canceled tick 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3008062c90 msgr2=0x7f3008065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 --2- 172.21.15.87:0/1264766427 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3008062c90 0x7f3008065150 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 rx=0x7f3014005010 tx=0x7f3014007f90).stop 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 msgr2=0x7f302006b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 --2- 172.21.15.87:0/1264766427 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 0x7f302006b5c0 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 rx=0x7f3010009400 tx=0x7f3010009cb0).stop 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 shutdown_connections 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 --2- 172.21.15.87:0/1264766427 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3008062c90 0x7f3008065150 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 --2- 172.21.15.87:0/1264766427 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30200fe510 0x7f302006bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 --2- 172.21.15.87:0/1264766427 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30200f8420 0x7f302006b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:29.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 --2- 172.21.15.87:0/1264766427 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30200fdbb0 0x7f302006b5c0 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:29.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 >> 172.21.15.87:0/1264766427 conn(0x7f30200f3800 msgr2=0x7f30200f5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:29.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 shutdown_connections 2024-02-23T00:42:29.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 -- 172.21.15.87:0/1264766427 wait complete. 2024-02-23T00:42:29.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:29.277+0000 7f3024b00700 1 librados: shutdown 2024-02-23T00:42:29.281 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:27.629318+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:27.632765+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:27.782847+0000 mgr.y (mgr.24437) 618 : cluster [DBG] pgmap v1106: 194 pgs: 1 active+remapped+backfill_toofull, 5 unknown, 188 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:28.290274+0000 mon.a (mon.0) 3662 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:28.366593+0000 osd.7 (osd.7) 330 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:28.369475+0000 osd.7 (osd.7) 331 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:28.382083+0000 mon.c (mon.2) 496 : audit [INF] from='client.? 172.21.15.87:0/1264766427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:28.382373+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:28.608936+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.1f deep-scrub starts 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:28.612103+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.1f deep-scrub ok 2024-02-23T00:42:29.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:28.965331+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:42:29.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:28.967203+0000 osd.5 (osd.5) 284 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:42:29.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:29.116274+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.d scrub starts 2024-02-23T00:42:29.444 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:29.119163+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.d scrub ok 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:27.629318+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:27.632765+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:27.782847+0000 mgr.y (mgr.24437) 618 : cluster [DBG] pgmap v1106: 194 pgs: 1 active+remapped+backfill_toofull, 5 unknown, 188 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:28.290274+0000 mon.a (mon.0) 3662 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:28.366593+0000 osd.7 (osd.7) 330 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:28.369475+0000 osd.7 (osd.7) 331 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:28.382083+0000 mon.c (mon.2) 496 : audit [INF] from='client.? 172.21.15.87:0/1264766427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:28.382373+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:28.608936+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.1f deep-scrub starts 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:28.612103+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.1f deep-scrub ok 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:28.965331+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:42:29.465 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:28.967203+0000 osd.5 (osd.5) 284 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:29.116274+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.d scrub starts 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:29.119163+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.d scrub ok 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:27.629318+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:27.632765+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:27.782847+0000 mgr.y (mgr.24437) 618 : cluster [DBG] pgmap v1106: 194 pgs: 1 active+remapped+backfill_toofull, 5 unknown, 188 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:28.290274+0000 mon.a (mon.0) 3662 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:28.366593+0000 osd.7 (osd.7) 330 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:28.369475+0000 osd.7 (osd.7) 331 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:28.382083+0000 mon.c (mon.2) 496 : audit [INF] from='client.? 172.21.15.87:0/1264766427' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:28.382373+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:28.608936+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.1f deep-scrub starts 2024-02-23T00:42:29.466 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:28.612103+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.1f deep-scrub ok 2024-02-23T00:42:29.467 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:28.965331+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:42:29.467 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:28.967203+0000 osd.5 (osd.5) 284 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:42:29.467 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:29.116274+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.d scrub starts 2024-02-23T00:42:29.467 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:29.119163+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.d scrub ok 2024-02-23T00:42:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:29.268157+0000 mon.a (mon.0) 3664 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:29.275733+0000 mon.a (mon.0) 3665 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:29.275798+0000 mon.a (mon.0) 3666 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:42:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:29.275857+0000 mon.a (mon.0) 3667 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-23T00:42:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:29.278779+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126"}]: dispatch 2024-02-23T00:42:30.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:29.904257+0000 osd.4 (osd.4) 323 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:42:30.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:29.905704+0000 osd.4 (osd.4) 324 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:42:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:29.268157+0000 mon.a (mon.0) 3664 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:29.275733+0000 mon.a (mon.0) 3665 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:29.275798+0000 mon.a (mon.0) 3666 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:42:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:29.275857+0000 mon.a (mon.0) 3667 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-23T00:42:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:29.278779+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126"}]: dispatch 2024-02-23T00:42:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:29.904257+0000 osd.4 (osd.4) 323 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:42:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:29.905704+0000 osd.4 (osd.4) 324 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:42:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:29.268157+0000 mon.a (mon.0) 3664 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:29.275733+0000 mon.a (mon.0) 3665 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:42:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:29.275798+0000 mon.a (mon.0) 3666 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:42:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:29.275857+0000 mon.a (mon.0) 3667 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-23T00:42:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:29.278779+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126"}]: dispatch 2024-02-23T00:42:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:29.904257+0000 osd.4 (osd.4) 323 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:42:30.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:29.905704+0000 osd.4 (osd.4) 324 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:42:31.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (18998 ms) 2024-02-23T00:42:31.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-02-23T00:42:31.287 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:42:31.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (5974 ms) 2024-02-23T00:42:31.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-02-23T00:42:31.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12094 ms) 2024-02-23T00:42:31.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-02-23T00:42:31.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-02-23T00:42:31.288 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 00000010 69 73 |is| 2024-02-23T00:42:31.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 00000012 2024-02-23T00:42:31.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (11946 ms) 2024-02-23T00:42:31.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-02-23T00:42:31.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: require_osd_release = pacific 2024-02-23T00:42:31.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5961 ms) 2024-02-23T00:42:31.289 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 43 tests from LibRadosTwoPoolsPP (509979 ms total) 2024-02-23T00:42:31.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 2024-02-23T00:42:31.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-02-23T00:42:31.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-02-23T00:42:31.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1040 ms) 2024-02-23T00:42:31.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-02-23T00:42:31.290 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10958 ms) 2024-02-23T00:42:31.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-02-23T00:42:31.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18031 ms) 2024-02-23T00:42:31.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-02-23T00:42:31.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-02-23T00:42:31.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30031 ms total) 2024-02-23T00:42:31.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 2024-02-23T00:42:31.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-02-23T00:42:31.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-02-23T00:42:31.291 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7998 ms) 2024-02-23T00:42:31.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-02-23T00:42:31.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8005 ms) 2024-02-23T00:42:31.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-02-23T00:42:31.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: waiting for scrub... 2024-02-23T00:42:31.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: done waiting 2024-02-23T00:42:31.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (25024 ms) 2024-02-23T00:42:31.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-02-23T00:42:31.292 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9990 ms) 2024-02-23T00:42:31.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-02-23T00:42:31.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8007 ms) 2024-02-23T00:42:31.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-02-23T00:42:31.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8003 ms) 2024-02-23T00:42:31.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-02-23T00:42:31.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10006 ms) 2024-02-23T00:42:31.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-02-23T00:42:31.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7999 ms) 2024-02-23T00:42:31.293 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-02-23T00:42:31.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13106 ms) 2024-02-23T00:42:31.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-02-23T00:42:31.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7916 ms) 2024-02-23T00:42:31.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-02-23T00:42:31.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13030 ms) 2024-02-23T00:42:31.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-02-23T00:42:31.294 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7979 ms) 2024-02-23T00:42:31.295 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-02-23T00:42:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:29.783432+0000 mgr.y (mgr.24437) 619 : cluster [DBG] pgmap v1109: 194 pgs: 1 active+remapped+backfill_toofull, 193 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:29.973596+0000 osd.6 (osd.6) 296 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:29.973809+0000 osd.6 (osd.6) 297 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:31.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:30.273964+0000 mon.a (mon.0) 3669 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:31.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:30.288077+0000 mon.a (mon.0) 3670 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126"}]': finished 2024-02-23T00:42:31.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:30.288153+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-23T00:42:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:29.783432+0000 mgr.y (mgr.24437) 619 : cluster [DBG] pgmap v1109: 194 pgs: 1 active+remapped+backfill_toofull, 193 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:29.973596+0000 osd.6 (osd.6) 296 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:29.973809+0000 osd.6 (osd.6) 297 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:30.273964+0000 mon.a (mon.0) 3669 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:30.288077+0000 mon.a (mon.0) 3670 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126"}]': finished 2024-02-23T00:42:31.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:30.288153+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-23T00:42:31.594 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:31.595 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:31.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:31.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:42:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:42:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:29.783432+0000 mgr.y (mgr.24437) 619 : cluster [DBG] pgmap v1109: 194 pgs: 1 active+remapped+backfill_toofull, 193 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/741 objects misplaced (0.135%) 2024-02-23T00:42:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:29.973596+0000 osd.6 (osd.6) 296 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:29.973809+0000 osd.6 (osd.6) 297 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:30.273964+0000 mon.a (mon.0) 3669 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:42:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:30.288077+0000 mon.a (mon.0) 3670 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-126"}]': finished 2024-02-23T00:42:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:30.288153+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-23T00:42:31.702 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:31.735 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:31.757 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:31.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:31.861 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:31.891 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:31.911 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:31.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:32.015 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:32.048 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:32.072 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:32.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:32.173 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:32.201 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:32.220 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:32.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:32.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:32.281 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:42:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:32] "GET /metrics HTTP/1.1" 200 190344 "" "Prometheus/2.33.4" 2024-02-23T00:42:32.327 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:32.359 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:32.377 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:32.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:32.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:32.481 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:32.510 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:32.564 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:32.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:32.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:31.235599+0000 mgr.y (mgr.24437) 620 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:32.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:31.285903+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-23T00:42:32.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:31.854582+0000 osd.4 (osd.4) 325 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:32.580 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:31.854912+0000 osd.4 (osd.4) 326 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:32.581 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:31.235599+0000 mgr.y (mgr.24437) 620 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:32.581 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:31.285903+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-23T00:42:32.581 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:31.854582+0000 osd.4 (osd.4) 325 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:32.581 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:31.854912+0000 osd.4 (osd.4) 326 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:32.668 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:31.235599+0000 mgr.y (mgr.24437) 620 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:31.285903+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-23T00:42:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:31.854582+0000 osd.4 (osd.4) 325 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:31.854912+0000 osd.4 (osd.4) 326 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:32.696 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:32.794 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:32.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:32.897 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:32.925 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:32.944 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:32.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:32.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:33.048 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:33.075 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:33.130 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:42:33.423 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:33.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:33.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:33.527 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:33.548 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:42:33.548 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:42:33.548 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:42:33.548 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:42:33.554 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:33.572 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:33.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:33.676 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:31.783897+0000 mgr.y (mgr.24437) 621 : cluster [DBG] pgmap v1112: 162 pgs: 1 active+remapped+backfill_toofull, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:32.316425+0000 osd.3 (osd.3) 230 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:32.316581+0000 osd.3 (osd.3) 231 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:32.330535+0000 mon.a (mon.0) 3673 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-23T00:42:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:32.334470+0000 mon.a (mon.0) 3674 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:33.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:31.783897+0000 mgr.y (mgr.24437) 621 : cluster [DBG] pgmap v1112: 162 pgs: 1 active+remapped+backfill_toofull, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:33.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:32.316425+0000 osd.3 (osd.3) 230 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:33.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:32.316581+0000 osd.3 (osd.3) 231 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:33.703 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:32.330535+0000 mon.a (mon.0) 3673 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-23T00:42:33.704 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:32.334470+0000 mon.a (mon.0) 3674 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:33.707 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:31.783897+0000 mgr.y (mgr.24437) 621 : cluster [DBG] pgmap v1112: 162 pgs: 1 active+remapped+backfill_toofull, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:33.707 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:32.316425+0000 osd.3 (osd.3) 230 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:33.707 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:32.316581+0000 osd.3 (osd.3) 231 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:33.708 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:32.330535+0000 mon.a (mon.0) 3673 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-23T00:42:33.708 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:32.334470+0000 mon.a (mon.0) 3674 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:33.708 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:33.743 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:33.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:33.853 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:33.880 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:33.899 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:33.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:34.002 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:34.030 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:34.179 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:34.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:34.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:42:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:42:34.283 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:34.310 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:34.329 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:34.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:34.432 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:34.458 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:34.476 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:34.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:34.580 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:34.604 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:33.334024+0000 mon.a (mon.0) 3675 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:34.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:33.334180+0000 mon.a (mon.0) 3676 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-23T00:42:34.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:33.347929+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:34.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:34.336084+0000 mon.a (mon.0) 3678 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:34.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:34.336147+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-23T00:42:34.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:34.336707+0000 mon.a (mon.0) 3680 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:42:34.606 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:34.607 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:33.334024+0000 mon.a (mon.0) 3675 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:34.607 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:33.334180+0000 mon.a (mon.0) 3676 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:33.347929+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:34.336084+0000 mon.a (mon.0) 3678 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:34.336147+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:34.336707+0000 mon.a (mon.0) 3680 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:33.334024+0000 mon.a (mon.0) 3675 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:33.334180+0000 mon.a (mon.0) 3676 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:33.347929+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:34.336084+0000 mon.a (mon.0) 3678 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:34.336147+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-23T00:42:34.608 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:34.336707+0000 mon.a (mon.0) 3680 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:42:34.624 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:34.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:34.728 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:34.755 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:34.773 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:34.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:34.877 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:34.906 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:35.030 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:35.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:35.157 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:35.184 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:35.435 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:35.437 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:33.784408+0000 mgr.y (mgr.24437) 622 : cluster [DBG] pgmap v1115: 194 pgs: 3 creating+peering, 29 unknown, 1 active+remapped+backfill_toofull, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:35.437 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:33.795516+0000 osd.0 (osd.0) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:35.437 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:33.795685+0000 osd.0 (osd.0) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:35.437 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:34.428891+0000 mon.c (mon.2) 497 : audit [DBG] from='client.? 172.21.15.87:0/516925737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:35.437 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:35.335766+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:42:35.437 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:35.335828+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-23T00:42:35.438 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:35.339550+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:42:35.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:33.784408+0000 mgr.y (mgr.24437) 622 : cluster [DBG] pgmap v1115: 194 pgs: 3 creating+peering, 29 unknown, 1 active+remapped+backfill_toofull, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:35.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:33.795516+0000 osd.0 (osd.0) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:35.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:33.795685+0000 osd.0 (osd.0) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:35.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:34.428891+0000 mon.c (mon.2) 497 : audit [DBG] from='client.? 172.21.15.87:0/516925737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:35.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:35.335766+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:42:35.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:35.335828+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-23T00:42:35.438 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:35.339550+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:42:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:35.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:35.445 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-23T00:42:35.540 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:35.556 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 -- 172.21.15.87:0/3502436761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d180fdbb0 msgr2=0x7f2d180fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 --2- 172.21.15.87:0/3502436761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d180fdbb0 0x7f2d180fdf90 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 rx=0x7f2d140047b0 tx=0x7f2d14009cb0).stop 2024-02-23T00:42:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 -- 172.21.15.87:0/3502436761 shutdown_connections 2024-02-23T00:42:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 --2- 172.21.15.87:0/3502436761 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d180f8420 0x7f2d18104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 --2- 172.21.15.87:0/3502436761 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d180fdbb0 0x7f2d180fdf90 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 --2- 172.21.15.87:0/3502436761 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d180fe510 0x7f2d180f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 -- 172.21.15.87:0/3502436761 >> 172.21.15.87:0/3502436761 conn(0x7f2d180f3800 msgr2=0x7f2d180f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 -- 172.21.15.87:0/3502436761 shutdown_connections 2024-02-23T00:42:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 -- 172.21.15.87:0/3502436761 wait complete. 2024-02-23T00:42:35.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 librados: starting msgr at 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.424+0000 7f2d1f402700 1 librados: starting objecter 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 Processor -- start 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 -- start start 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 librados: setting wanted keys 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 librados: calling monclient init 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d180f8420 0x7f2d1810e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d180fdbb0 0x7f2d1810e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d180fe510 0x7f2d181081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d181086b0 con 0x7f2d180fdbb0 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18108820 con 0x7f2d180fe510 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1f402700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d181089e0 con 0x7f2d180f8420 2024-02-23T00:42:35.558 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d19e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d180f8420 0x7f2d1810e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d19e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d180f8420 0x7f2d1810e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60382/0 (socket says 172.21.15.87:60382) 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d19e700 1 -- 172.21.15.87:0/516925737 learned_addr learned my addr 172.21.15.87:0/516925737 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1c99d700 1 --2- 172.21.15.87:0/516925737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d180fdbb0 0x7f2d1810e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d99f700 1 --2- 172.21.15.87:0/516925737 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d180fe510 0x7f2d181081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d19e700 1 -- 172.21.15.87:0/516925737 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d180fe510 msgr2=0x7f2d181081c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d19e700 1 --2- 172.21.15.87:0/516925737 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f2d180fe510 0x7f2d181081c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d19e700 1 -- 172.21.15.87:0/516925737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d180fdbb0 msgr2=0x7f2d1810e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d19e700 1 --2- 172.21.15.87:0/516925737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d180fdbb0 0x7f2d1810e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d19e700 1 -- 172.21.15.87:0/516925737 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d100097c0 con 0x7f2d180f8420 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1c99d700 1 --2- 172.21.15.87:0/516925737 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f2d180fdbb0 0x7f2d1810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d1d19e700 1 --2- 172.21.15.87:0/516925737 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d180f8420 0x7f2d1810e020 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7f2d140044e0 tx=0x7f2d14007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d0e7fc700 10 client.?.objecter ms_handle_connect 0x7f2d180f8420 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d0e7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:35.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.425+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14018070 con 0x7f2d180f8420 2024-02-23T00:42:35.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.426+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d140093b0 con 0x7f2d180f8420 2024-02-23T00:42:35.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.426+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d14005040 con 0x7f2d180f8420 2024-02-23T00:42:35.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.426+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1401dd90 con 0x7f2d180f8420 2024-02-23T00:42:35.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.426+0000 7f2d1f402700 10 client.38270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:35.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.426+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d18108f00 con 0x7f2d180f8420 2024-02-23T00:42:35.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.426+0000 7f2d1f402700 1 librados: init done 2024-02-23T00:42:35.560 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.426+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2d1805ee80 con 0x7f2d180f8420 2024-02-23T00:42:35.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.427+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f2d1401d420 con 0x7f2d180f8420 2024-02-23T00:42:35.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.427+0000 7f2d0e7fc700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d04062c90 0x7f2d04065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.427+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(732..732 src has 400..732) v4 ==== 8538+0+0 (secure 0 0 0) 0x7f2d1407e880 con 0x7f2d180f8420 2024-02-23T00:42:35.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.427+0000 7f2d1c99d700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d04062c90 0x7f2d04065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.427+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 osd_map(732..732 src has 400..732) v4 2024-02-23T00:42:35.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.427+0000 7f2d0e7fc700 3 client.38270.objecter handle_osd_map got epochs [732,732] > 0 2024-02-23T00:42:35.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.427+0000 7f2d0e7fc700 3 client.38270.objecter handle_osd_map decoding full epoch 732 2024-02-23T00:42:35.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d0e7fc700 20 client.38270.objecter dump_active .. 0 homeless 2024-02-23T00:42:35.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v732) v1 ==== 54+0+16 (secure 0 0 0) 0x7f2d14085070 con 0x7f2d180f8420 2024-02-23T00:42:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1c99d700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d04062c90 0x7f2d04065150 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7f2d18109960 tx=0x7f2d0800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d0e7fc700 10 client.38270.objecter ms_handle_connect 0x7f2d04062c90 2024-02-23T00:42:35.562 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:42:35.562 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:42:35.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 10 client.38270.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 1 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6801/484081831 conn(0x7f2d180fee80 0x7f2d180ff260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18107110 osd=0 3 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18107110 osd=0 3 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18107110 osd=0 3 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 1 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d18107110 osd=0 3 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 0 1 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18107110 osd=0 4 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 10 client.38270.objecter _send_command 1 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d181096a0 con 0x7f2d180fee80 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1d99f700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6801/484081831 conn(0x7f2d180fee80 0x7f2d180ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d0e7fc700 10 client.38270.objecter ms_handle_connect 0x7f2d180fee80 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.428+0000 7f2d1d99f700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6801/484081831 conn(0x7f2d180fee80 0x7f2d180ff260 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.429+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2d181096a0 con 0x7f2d180fee80 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.429+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.429+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 1 = system:0 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.429+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d18107110 osd=0 4 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.429+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 0 1 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 10 client.38270.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 2 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18107110 osd=0 3 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18107110 osd=0 3 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18107110 osd=0 3 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 2 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d18107110 osd=0 3 2024-02-23T00:42:35.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 0 2 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18107110 osd=0 4 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 10 client.38270.objecter _send_command 2 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.443+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2d180273c0 con 0x7f2d180fee80 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2d180273c0 con 0x7f2d180fee80 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d18107110 osd=0 4 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 0 2 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 10 client.38270.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 3 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6805/685354171 conn(0x7f2d180598f0 0x7f2d18059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18028710 osd=1 3 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18028710 osd=1 3 2024-02-23T00:42:35.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18028710 osd=1 3 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 3 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d18028710 osd=1 3 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 1 3 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18028710 osd=1 4 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 10 client.38270.objecter _send_command 3 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d1811b5d0 con 0x7f2d180598f0 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1d19e700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6805/685354171 conn(0x7f2d180598f0 0x7f2d18059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 10 client.38270.objecter ms_handle_connect 0x7f2d180598f0 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d1d19e700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6805/685354171 conn(0x7f2d180598f0 0x7f2d18059cd0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2d1811b5d0 con 0x7f2d180598f0 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 3 = system:0 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d18028710 osd=1 4 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.446+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 1 3 2024-02-23T00:42:35.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 10 client.38270.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 4 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18028710 osd=1 3 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18028710 osd=1 3 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18028710 osd=1 3 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 4 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d18028710 osd=1 3 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 1 4 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18028710 osd=1 4 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 10 client.38270.objecter _send_command 4 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.461+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2d18059360 con 0x7f2d180598f0 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2d18059360 con 0x7f2d180598f0 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d18028710 osd=1 4 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 1 4 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 10 client.38270.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 5 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6809/3605394228 conn(0x7f2d18119ff0 0x7f2d1811a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18027d60 osd=2 3 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18027d60 osd=2 3 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18027d60 osd=2 3 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 5 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d18027d60 osd=2 3 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 2 5 2024-02-23T00:42:35.568 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18027d60 osd=2 4 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 10 client.38270.objecter _send_command 5 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d180283e0 con 0x7f2d18119ff0 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1c99d700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6809/3605394228 conn(0x7f2d18119ff0 0x7f2d1811a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 10 client.38270.objecter ms_handle_connect 0x7f2d18119ff0 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d1c99d700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6809/3605394228 conn(0x7f2d18119ff0 0x7f2d1811a3f0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2d180283e0 con 0x7f2d18119ff0 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:42:35.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 5 = system:0 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d18027d60 osd=2 4 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.463+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 2 5 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 10 client.38270.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 6 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18027d60 osd=2 3 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18027d60 osd=2 3 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18027d60 osd=2 3 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 6 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d18027d60 osd=2 3 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 2 6 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18027d60 osd=2 4 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 10 client.38270.objecter _send_command 6 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.477+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2d1811b0b0 con 0x7f2d18119ff0 2024-02-23T00:42:35.570 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2d1811b0b0 con 0x7f2d18119ff0 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d18027d60 osd=2 4 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 2 6 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 10 client.38270.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 7 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6813/128059790 conn(0x7f2d18027940 0x7f2d1811fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d180277d0 osd=3 3 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d180277d0 osd=3 3 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d180277d0 osd=3 3 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 7 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d180277d0 osd=3 3 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 3 7 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d180277d0 osd=3 4 2024-02-23T00:42:35.571 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 10 client.38270.objecter _send_command 7 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d181228c0 con 0x7f2d18027940 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1d99f700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6813/128059790 conn(0x7f2d18027940 0x7f2d1811fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d0e7fc700 10 client.38270.objecter ms_handle_connect 0x7f2d18027940 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.479+0000 7f2d1d99f700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6813/128059790 conn(0x7f2d18027940 0x7f2d1811fc70 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.480+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2d181228c0 con 0x7f2d18027940 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.480+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.480+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 7 = system:0 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.480+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d180277d0 osd=3 4 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.480+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 3 7 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 10 client.38270.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 8 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d180277d0 osd=3 3 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d180277d0 osd=3 3 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d180277d0 osd=3 3 2024-02-23T00:42:35.572 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 8 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d180277d0 osd=3 3 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 3 8 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d180277d0 osd=3 4 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 10 client.38270.objecter _send_command 8 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.494+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2d1811ab20 con 0x7f2d18027940 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2d1811ab20 con 0x7f2d18027940 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d180277d0 osd=3 4 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 3 8 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 10 client.38270.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 9 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6800/4175844757 conn(0x7f2d1811ac90 0x7f2d18122170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1811ab20 osd=4 3 2024-02-23T00:42:35.573 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1811ab20 osd=4 3 2024-02-23T00:42:35.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1811ab20 osd=4 3 2024-02-23T00:42:35.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 9 2024-02-23T00:42:35.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d1811ab20 osd=4 3 2024-02-23T00:42:35.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 4 9 2024-02-23T00:42:35.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1811ab20 osd=4 4 2024-02-23T00:42:35.574 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:42:35.574 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 10 client.38270.objecter _send_command 9 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d18119b50 con 0x7f2d1811ac90 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1d19e700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6800/4175844757 conn(0x7f2d1811ac90 0x7f2d18122170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d1d19e700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6800/4175844757 conn(0x7f2d1811ac90 0x7f2d18122170 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.496+0000 7f2d0e7fc700 10 client.38270.objecter ms_handle_connect 0x7f2d1811ac90 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.498+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2d18119b50 con 0x7f2d1811ac90 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.498+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.498+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 9 = system:0 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.498+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d1811ab20 osd=4 4 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.498+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 4 9 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 10 client.38270.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 10 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1811ab20 osd=4 3 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1811ab20 osd=4 3 2024-02-23T00:42:35.575 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1811ab20 osd=4 3 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 10 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d1811ab20 osd=4 3 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 4 10 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1811ab20 osd=4 4 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 10 client.38270.objecter _send_command 10 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.512+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2d18119b50 con 0x7f2d1811ac90 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2d18119b50 con 0x7f2d1811ac90 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d1811ab20 osd=4 4 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 4 10 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 10 client.38270.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 11 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2d1806c5a0 0x7f2d1806ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18119b50 osd=5 3 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18119b50 osd=5 3 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18119b50 osd=5 3 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 11 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d18119b50 osd=5 3 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 5 11 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18119b50 osd=5 4 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 10 client.38270.objecter _send_command 11 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d1806cf60 con 0x7f2d1806c5a0 2024-02-23T00:42:35.577 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1c99d700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2d1806c5a0 0x7f2d1806ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d0e7fc700 10 client.38270.objecter ms_handle_connect 0x7f2d1806c5a0 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.514+0000 7f2d1c99d700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2d1806c5a0 0x7f2d1806ca00 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.515+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2d1806cf60 con 0x7f2d1806c5a0 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.515+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.515+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 11 = system:0 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.515+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d18119b50 osd=5 4 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.515+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 5 11 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 10 client.38270.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 12 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18119b50 osd=5 3 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18119b50 osd=5 3 2024-02-23T00:42:35.578 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d18119b50 osd=5 3 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 12 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d18119b50 osd=5 3 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 5 12 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18119b50 osd=5 4 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 10 client.38270.objecter _send_command 12 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.529+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2d1806cf60 con 0x7f2d1806c5a0 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2d1806cf60 con 0x7f2d1806c5a0 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d18119b50 osd=5 4 2024-02-23T00:42:35.579 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 5 12 2024-02-23T00:42:35.580 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:42:35.580 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:42:35.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 10 client.38270.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:35.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 13 2024-02-23T00:42:35.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2d18129360 0x7f2d1812b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1806cf60 osd=6 3 2024-02-23T00:42:35.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1806cf60 osd=6 3 2024-02-23T00:42:35.580 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1806cf60 osd=6 3 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 13 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d1806cf60 osd=6 3 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 6 13 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1806cf60 osd=6 4 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 10 client.38270.objecter _send_command 13 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d1812e430 con 0x7f2d18129360 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1d99f700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2d18129360 0x7f2d1812b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d1d99f700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2d18129360 0x7f2d1812b750 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 10 client.38270.objecter ms_handle_connect 0x7f2d18129360 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2d1812e430 con 0x7f2d18129360 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 13 = system:0 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d1806cf60 osd=6 4 2024-02-23T00:42:35.581 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.532+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 6 13 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 10 client.38270.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 14 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1806cf60 osd=6 3 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1806cf60 osd=6 3 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1806cf60 osd=6 3 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 14 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d1806cf60 osd=6 3 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 6 14 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1806cf60 osd=6 4 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 10 client.38270.objecter _send_command 14 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.547+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2d1812e430 con 0x7f2d18129360 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.548+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2d1812e430 con 0x7f2d18129360 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.548+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:35.582 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.548+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.548+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d1806cf60 osd=6 4 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.548+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 6 14 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 10 client.38270.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 15 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6812/373787000 conn(0x7f2d1812c150 0x7f2d1812c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1812e430 osd=7 3 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1812e430 osd=7 3 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1812e430 osd=7 3 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 15 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d1812e430 osd=7 3 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 7 15 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1812e430 osd=7 4 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 10 client.38270.objecter _send_command 15 2024-02-23T00:42:35.583 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d1812e1f0 con 0x7f2d1812c150 2024-02-23T00:42:35.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1d19e700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6812/373787000 conn(0x7f2d1812c150 0x7f2d1812c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:35.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d0e7fc700 10 client.38270.objecter ms_handle_connect 0x7f2d1812c150 2024-02-23T00:42:35.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.549+0000 7f2d1d19e700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6812/373787000 conn(0x7f2d1812c150 0x7f2d1812c5d0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:35.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.550+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f2d1812e1f0 con 0x7f2d1812c150 2024-02-23T00:42:35.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.550+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:42:35.584 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.550+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 15 = system:0 2024-02-23T00:42:35.585 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:42:35.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.550+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d1812e430 osd=7 4 2024-02-23T00:42:35.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.550+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 7 15 2024-02-23T00:42:35.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 10 client.38270.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:35.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign -1 16 2024-02-23T00:42:35.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1812e430 osd=7 3 2024-02-23T00:42:35.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1812e430 osd=7 3 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 20 client.38270.objecter _get_session s=0x7f2d1812e430 osd=7 3 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_remove -1 16 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 20 client.38270.objecter get_session s=0x7f2d1812e430 osd=7 3 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 15 client.38270.objecter _session_command_op_assign 7 16 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1812e430 osd=7 4 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 10 client.38270.objecter _send_command 16 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.564+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2d1812e1f0 con 0x7f2d1812c150 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d0e7fc700 1 -- 172.21.15.87:0/516925737 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f2d1812e1f0 con 0x7f2d1812c150 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d0e7fc700 10 client.38270.objecter ms_dispatch 0x7f2d18001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d0e7fc700 10 client.38270.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d0e7fc700 20 client.38270.objecter put_session s=0x7f2d1812e430 osd=7 4 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d0e7fc700 15 client.38270.objecter _session_command_op_remove 7 16 2024-02-23T00:42:35.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 librados: watch_flush enter 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 librados: watch_flush exit 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 client.38270.objecter close_session for osd.0 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6801/484081831 conn(0x7f2d180fee80 msgr2=0x7f2d180ff260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6801/484081831 conn(0x7f2d180fee80 0x7f2d180ff260 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18107110 osd=0 1 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 client.38270.objecter close_session for osd.1 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6805/685354171 conn(0x7f2d180598f0 msgr2=0x7f2d18059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6805/685354171 conn(0x7f2d180598f0 0x7f2d18059cd0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18028710 osd=1 1 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 client.38270.objecter close_session for osd.2 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6809/3605394228 conn(0x7f2d18119ff0 msgr2=0x7f2d1811a3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6809/3605394228 conn(0x7f2d18119ff0 0x7f2d1811a3f0 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18027d60 osd=2 1 2024-02-23T00:42:35.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 client.38270.objecter close_session for osd.3 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6813/128059790 conn(0x7f2d18027940 msgr2=0x7f2d1811fc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1d99f700 1 -- 172.21.15.87:0/516925737 reap_dead start 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6813/128059790 conn(0x7f2d18027940 0x7f2d1811fc70 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d180277d0 osd=3 1 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 client.38270.objecter close_session for osd.4 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6800/4175844757 conn(0x7f2d1811ac90 msgr2=0x7f2d18122170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6800/4175844757 conn(0x7f2d1811ac90 0x7f2d18122170 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1811ab20 osd=4 1 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 client.38270.objecter close_session for osd.5 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2d1806c5a0 msgr2=0x7f2d1806ca00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6804/2550013795 conn(0x7f2d1806c5a0 0x7f2d1806ca00 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d18119b50 osd=5 1 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 client.38270.objecter close_session for osd.6 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2d18129360 msgr2=0x7f2d1812b750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6808/2509408498 conn(0x7f2d18129360 0x7f2d1812b750 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1806cf60 osd=6 1 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 client.38270.objecter close_session for osd.7 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6812/373787000 conn(0x7f2d1812c150 msgr2=0x7f2d1812c5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.171:6812/373787000 conn(0x7f2d1812c150 0x7f2d1812c5d0 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 20 client.38270.objecter put_session s=0x7f2d1812e430 osd=7 1 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 20 client.38270.objecter shutdown clearing up homeless session... 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 10 client.38270.objecter successfully canceled tick 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1d99f700 1 -- 172.21.15.87:0/516925737 reap_dead start 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d04062c90 msgr2=0x7f2d04065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d04062c90 0x7f2d04065150 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 rx=0x7f2d18109960 tx=0x7f2d0800b040).stop 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d180f8420 msgr2=0x7f2d1810e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d180f8420 0x7f2d1810e020 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 rx=0x7f2d140044e0 tx=0x7f2d14007f90).stop 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 shutdown_connections 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> v2:172.21.15.87:6800/1231597463 conn(0x7f2d04062c90 0x7f2d04065150 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 --2- 172.21.15.87:0/516925737 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f2d180f8420 0x7f2d1810e020 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.566+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 >> 172.21.15.87:0/516925737 conn(0x7f2d180f3800 msgr2=0x7f2d180f5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.567+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 shutdown_connections 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.567+0000 7f2d1f402700 1 -- 172.21.15.87:0/516925737 wait complete. 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:34.567+0000 7f2d1f402700 1 librados: shutdown 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.590 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:42:35.591 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:33.784408+0000 mgr.y (mgr.24437) 622 : cluster [DBG] pgmap v1115: 194 pgs: 3 creating+peering, 29 unknown, 1 active+remapped+backfill_toofull, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:33.795516+0000 osd.0 (osd.0) 29 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:42:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:33.795685+0000 osd.0 (osd.0) 30 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:42:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:34.428891+0000 mon.c (mon.2) 497 : audit [DBG] from='client.? 172.21.15.87:0/516925737' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:35.335766+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:42:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:35.335828+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-23T00:42:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:35.339550+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:42:35.815 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:35.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:35.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:35.945 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:35.964 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:35.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:36.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:36.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.077+0000 7fb2a481a700 1 -- 172.21.15.87:0/3429975788 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00fd4a0 msgr2=0x7fb2a00f74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:36.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.077+0000 7fb2a481a700 1 --2- 172.21.15.87:0/3429975788 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00fd4a0 0x7fb2a00f74c0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 rx=0x7fb288004890 tx=0x7fb288009cb0).stop 2024-02-23T00:42:36.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.078+0000 7fb2a481a700 1 -- 172.21.15.87:0/3429975788 shutdown_connections 2024-02-23T00:42:36.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.078+0000 7fb2a481a700 1 --2- 172.21.15.87:0/3429975788 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2a00f7ba0 0x7fb2a0103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:36.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.078+0000 7fb2a481a700 1 --2- 172.21.15.87:0/3429975788 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a00fcb40 0x7fb2a00fcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:36.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.078+0000 7fb2a481a700 1 --2- 172.21.15.87:0/3429975788 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00fd4a0 0x7fb2a00f74c0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:36.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.078+0000 7fb2a481a700 1 -- 172.21.15.87:0/3429975788 >> 172.21.15.87:0/3429975788 conn(0x7fb2a006c370 msgr2=0x7fb2a006c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:36.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.078+0000 7fb2a481a700 1 -- 172.21.15.87:0/3429975788 shutdown_connections 2024-02-23T00:42:36.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.078+0000 7fb2a481a700 1 -- 172.21.15.87:0/3429975788 wait complete. 2024-02-23T00:42:36.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.079+0000 7fb2a481a700 1 librados: starting msgr at 2024-02-23T00:42:36.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.079+0000 7fb2a481a700 1 librados: starting objecter 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.079+0000 7fb2a481a700 1 Processor -- start 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.079+0000 7fb2a481a700 1 -- start start 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.079+0000 7fb2a481a700 1 librados: setting wanted keys 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.079+0000 7fb2a481a700 1 librados: calling monclient init 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb2a481a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00f7ba0 0x7fb2a006aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb2a481a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a00fcb40 0x7fb2a0063b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb2a481a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2a00fd4a0 0x7fb2a0064220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb2a481a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00648b0 con 0x7fb2a00fcb40 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb2a481a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00685c0 con 0x7fb2a00fd4a0 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb2a481a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a0068780 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29e59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00f7ba0 0x7fb2a006aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29e59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00f7ba0 0x7fb2a006aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60402/0 (socket says 172.21.15.87:60402) 2024-02-23T00:42:36.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29e59c700 1 -- 172.21.15.87:0/1791921510 learned_addr learned my addr 172.21.15.87:0/1791921510 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:36.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29ed9d700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2a00fd4a0 0x7fb2a0064220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:36.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29dd9b700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a00fcb40 0x7fb2a0063b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:36.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29e59c700 1 -- 172.21.15.87:0/1791921510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2a00fd4a0 msgr2=0x7fb2a0064220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:36.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29e59c700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2a00fd4a0 0x7fb2a0064220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:36.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29e59c700 1 -- 172.21.15.87:0/1791921510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a00fcb40 msgr2=0x7fb2a0063b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:36.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29e59c700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a00fcb40 0x7fb2a0063b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:36.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29e59c700 1 -- 172.21.15.87:0/1791921510 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288005040 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.080+0000 7fb29dd9b700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a00fcb40 0x7fb2a0063b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb29e59c700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00f7ba0 0x7fb2a006aa10 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fb29000e970 tx=0x7fb29000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb2977fe700 10 client.?.objecter ms_handle_connect 0x7fb2a00f7ba0 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb2977fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb2977fe700 1 -- 172.21.15.87:0/1791921510 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290004660 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb2977fe700 1 -- 172.21.15.87:0/1791921510 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb29000f040 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb2977fe700 1 -- 172.21.15.87:0/1791921510 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29000c840 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a0068a60 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb2a481a700 10 client.38276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a0068f00 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.081+0000 7fb2a481a700 1 librados: init done 2024-02-23T00:42:36.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.083+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb280004f50 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.084+0000 7fb2977fe700 1 -- 172.21.15.87:0/1791921510 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb290009630 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.085+0000 7fb2977fe700 1 --2- 172.21.15.87:0/1791921510 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb28c062ce0 0x7fb28c0651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:36.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.085+0000 7fb2977fe700 1 -- 172.21.15.87:0/1791921510 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(733..733 src has 400..733) v4 ==== 8538+0+0 (secure 0 0 0) 0x7fb29007cf60 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.085+0000 7fb2977fe700 10 client.38276.objecter ms_dispatch 0x7fb2a0001550 osd_map(733..733 src has 400..733) v4 2024-02-23T00:42:36.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.085+0000 7fb2977fe700 3 client.38276.objecter handle_osd_map got epochs [733,733] > 0 2024-02-23T00:42:36.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.085+0000 7fb2977fe700 3 client.38276.objecter handle_osd_map decoding full epoch 733 2024-02-23T00:42:36.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.085+0000 7fb29dd9b700 1 --2- 172.21.15.87:0/1791921510 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb28c062ce0 0x7fb28c0651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:36.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.085+0000 7fb2977fe700 20 client.38276.objecter dump_active .. 0 homeless 2024-02-23T00:42:36.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.085+0000 7fb29dd9b700 1 --2- 172.21.15.87:0/1791921510 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb28c062ce0 0x7fb28c0651a0 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7fb288004730 tx=0x7fb28800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:36.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.085+0000 7fb2977fe700 10 client.38276.objecter ms_handle_connect 0x7fb28c062ce0 2024-02-23T00:42:36.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.090+0000 7fb2977fe700 1 -- 172.21.15.87:0/1791921510 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb29004af60 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.101 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:36.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.260+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"} v 0) v1 -- 0x7fb280005780 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.337+0000 7fb2977fe700 1 -- 172.21.15.87:0/1791921510 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]=0 set pool 3 pg_num to 23 v734) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb2900190b0 con 0x7fb2a00f7ba0 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 10 librados: watch_flush enter 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 10 librados: watch_flush exit 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 20 client.38276.objecter shutdown clearing up homeless session... 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 10 client.38276.objecter successfully canceled tick 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb28c062ce0 msgr2=0x7fb28c0651a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 --2- 172.21.15.87:0/1791921510 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb28c062ce0 0x7fb28c0651a0 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 rx=0x7fb288004730 tx=0x7fb28800b040).stop 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00f7ba0 msgr2=0x7fb2a006aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00f7ba0 0x7fb2a006aa10 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 rx=0x7fb29000e970 tx=0x7fb29000b040).stop 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 shutdown_connections 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 --2- 172.21.15.87:0/1791921510 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb28c062ce0 0x7fb28c0651a0 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb2a00fd4a0 0x7fb2a0064220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:36.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb2a00f7ba0 0x7fb2a006aa10 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:36.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 --2- 172.21.15.87:0/1791921510 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb2a00fcb40 0x7fb2a0063b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:36.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 >> 172.21.15.87:0/1791921510 conn(0x7fb2a006c370 msgr2=0x7fb2a00fa900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:36.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 shutdown_connections 2024-02-23T00:42:36.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 -- 172.21.15.87:0/1791921510 wait complete. 2024-02-23T00:42:36.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:36.340+0000 7fb2a481a700 1 librados: shutdown 2024-02-23T00:42:36.343 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 3 pg_num to 23 2024-02-23T00:42:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:36.260849+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.87:0/1791921510' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]: dispatch 2024-02-23T00:42:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:36.261127+0000 mon.a (mon.0) 3684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]: dispatch 2024-02-23T00:42:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:36.337253+0000 mon.a (mon.0) 3685 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:42:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:36.337329+0000 mon.a (mon.0) 3686 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]': finished 2024-02-23T00:42:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:36.337410+0000 mon.a (mon.0) 3687 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-23T00:42:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:36.338542+0000 mon.a (mon.0) 3688 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:42:36.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:36.260849+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.87:0/1791921510' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]: dispatch 2024-02-23T00:42:36.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:36.261127+0000 mon.a (mon.0) 3684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]: dispatch 2024-02-23T00:42:36.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:36.337253+0000 mon.a (mon.0) 3685 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:42:36.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:36.337329+0000 mon.a (mon.0) 3686 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]': finished 2024-02-23T00:42:36.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:36.337410+0000 mon.a (mon.0) 3687 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-23T00:42:36.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:36.338542+0000 mon.a (mon.0) 3688 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:42:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:36.260849+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.87:0/1791921510' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]: dispatch 2024-02-23T00:42:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:36.261127+0000 mon.a (mon.0) 3684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]: dispatch 2024-02-23T00:42:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:36.337253+0000 mon.a (mon.0) 3685 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:42:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:36.337329+0000 mon.a (mon.0) 3686 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num", "val": "23"}]': finished 2024-02-23T00:42:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:36.337410+0000 mon.a (mon.0) 3687 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-23T00:42:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:36.338542+0000 mon.a (mon.0) 3688 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-23T00:42:37.164 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:37.164 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:37.640 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:35.785285+0000 mgr.y (mgr.24437) 623 : cluster [DBG] pgmap v1118: 194 pgs: 1 active+remapped+backfill_toofull, 3 creating+peering, 190 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:37.640 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:37.227763+0000 mon.a (mon.0) 3689 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:42:37.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:37.227832+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-23T00:42:37.641 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:35.785285+0000 mgr.y (mgr.24437) 623 : cluster [DBG] pgmap v1118: 194 pgs: 1 active+remapped+backfill_toofull, 3 creating+peering, 190 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:37.641 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:37.227763+0000 mon.a (mon.0) 3689 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:42:37.641 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:37.227832+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-23T00:42:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:35.785285+0000 mgr.y (mgr.24437) 623 : cluster [DBG] pgmap v1118: 194 pgs: 1 active+remapped+backfill_toofull, 3 creating+peering, 190 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:37.227763+0000 mon.a (mon.0) 3689 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-23T00:42:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:37.227832+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-23T00:42:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:37.786520+0000 mon.a (mon.0) 3691 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:42:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:38.251009+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:38.251397+0000 mon.a (mon.0) 3693 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128"}]: dispatch 2024-02-23T00:42:38.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:37.786520+0000 mon.a (mon.0) 3691 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:42:38.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:38.251009+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:38.766 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:38.251397+0000 mon.a (mon.0) 3693 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128"}]: dispatch 2024-02-23T00:42:38.766 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:37.786520+0000 mon.a (mon.0) 3691 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-23T00:42:38.766 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:38.251009+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:38.766 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:38.251397+0000 mon.a (mon.0) 3693 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128"}]: dispatch 2024-02-23T00:42:38.923 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:42:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:37.785627+0000 mgr.y (mgr.24437) 624 : cluster [DBG] pgmap v1121: 194 pgs: 1 active+remapped+backfill_toofull, 3 creating+peering, 190 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:38.366205+0000 mon.a (mon.0) 3694 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:42:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:38.366285+0000 mon.a (mon.0) 3695 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128"}]': finished 2024-02-23T00:42:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:38.366344+0000 mon.a (mon.0) 3696 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-23T00:42:39.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:37.785627+0000 mgr.y (mgr.24437) 624 : cluster [DBG] pgmap v1121: 194 pgs: 1 active+remapped+backfill_toofull, 3 creating+peering, 190 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:38.366205+0000 mon.a (mon.0) 3694 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:42:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:38.366285+0000 mon.a (mon.0) 3695 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128"}]': finished 2024-02-23T00:42:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:38.366344+0000 mon.a (mon.0) 3696 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-23T00:42:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:37.785627+0000 mgr.y (mgr.24437) 624 : cluster [DBG] pgmap v1121: 194 pgs: 1 active+remapped+backfill_toofull, 3 creating+peering, 190 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:38.366205+0000 mon.a (mon.0) 3694 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-23T00:42:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:38.366285+0000 mon.a (mon.0) 3695 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-128"}]': finished 2024-02-23T00:42:39.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:38.366344+0000 mon.a (mon.0) 3696 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-23T00:42:40.645 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:42:40.645 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:42:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:39.380215+0000 mon.a (mon.0) 3697 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-23T00:42:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:39.658122+0000 mon.a (mon.0) 3698 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:40.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:39.787040+0000 mon.a (mon.0) 3699 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-23T00:42:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:39.380215+0000 mon.a (mon.0) 3697 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-23T00:42:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:39.658122+0000 mon.a (mon.0) 3698 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:39.787040+0000 mon.a (mon.0) 3699 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-23T00:42:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:39.380215+0000 mon.a (mon.0) 3697 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-23T00:42:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:39.658122+0000 mon.a (mon.0) 3698 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:42:40.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:39.787040+0000 mon.a (mon.0) 3699 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "23"}]: dispatch 2024-02-23T00:42:40.882 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:40.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:40.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:40.986 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:41.018 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:41.039 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:41.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:41.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.646+0000 7fc772f8b700 1 -- 172.21.15.87:0/3215151838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fcb40 msgr2=0x7fc76c0fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.646+0000 7fc772f8b700 1 --2- 172.21.15.87:0/3215151838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fcb40 0x7fc76c0fcf20 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 rx=0x7fc760004870 tx=0x7fc760009cb0).stop 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.646+0000 7fc772f8b700 1 -- 172.21.15.87:0/3215151838 shutdown_connections 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.646+0000 7fc772f8b700 1 --2- 172.21.15.87:0/3215151838 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7ba0 0x7fc76c103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.646+0000 7fc772f8b700 1 --2- 172.21.15.87:0/3215151838 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fcb40 0x7fc76c0fcf20 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.646+0000 7fc772f8b700 1 --2- 172.21.15.87:0/3215151838 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fd4a0 0x7fc76c0f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.646+0000 7fc772f8b700 1 -- 172.21.15.87:0/3215151838 >> 172.21.15.87:0/3215151838 conn(0x7fc76c06c370 msgr2=0x7fc76c06c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.646+0000 7fc772f8b700 1 -- 172.21.15.87:0/3215151838 shutdown_connections 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.646+0000 7fc772f8b700 1 -- 172.21.15.87:0/3215151838 wait complete. 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 librados: starting msgr at 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 librados: starting objecter 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 Processor -- start 2024-02-23T00:42:41.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 -- start start 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 librados: setting wanted keys 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 librados: calling monclient init 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7ba0 0x7fc76c10e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fcb40 0x7fc76c10e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fd4a0 0x7fc76c108280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c108770 con 0x7fc76c0fd4a0 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c1088e0 con 0x7fc76c0fcb40 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc772f8b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c108aa0 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc770d27700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7ba0 0x7fc76c10e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc770d27700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7ba0 0x7fc76c10e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60420/0 (socket says 172.21.15.87:60420) 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc771528700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fd4a0 0x7fc76c108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc770d27700 1 -- 172.21.15.87:0/2461627724 learned_addr learned my addr 172.21.15.87:0/2461627724 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:41.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.647+0000 7fc76bfff700 1 --2- 172.21.15.87:0/2461627724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fcb40 0x7fc76c10e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc770d27700 1 -- 172.21.15.87:0/2461627724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fcb40 msgr2=0x7fc76c10e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc770d27700 1 --2- 172.21.15.87:0/2461627724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fcb40 0x7fc76c10e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc770d27700 1 -- 172.21.15.87:0/2461627724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fd4a0 msgr2=0x7fc76c108280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc770d27700 1 --2- 172.21.15.87:0/2461627724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fd4a0 0x7fc76c108280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc770d27700 1 -- 172.21.15.87:0/2461627724 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc760005040 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc76bfff700 1 --2- 172.21.15.87:0/2461627724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fcb40 0x7fc76c10e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc770d27700 1 --2- 172.21.15.87:0/2461627724 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7ba0 0x7fc76c10e0e0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7fc760009eb0 tx=0x7fc76000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc769ffb700 10 client.?.objecter ms_handle_connect 0x7fc76c0f7ba0 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc769ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc769ffb700 1 -- 172.21.15.87:0/2461627724 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76000e070 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76c108d20 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc769ffb700 1 -- 172.21.15.87:0/2461627724 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc760013da0 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc769ffb700 1 -- 172.21.15.87:0/2461627724 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760019070 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc772f8b700 10 client.38282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76c114f10 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.648+0000 7fc772f8b700 1 librados: init done 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.649+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76c05fa40 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.649+0000 7fc769ffb700 1 -- 172.21.15.87:0/2461627724 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc76001e420 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.650+0000 7fc769ffb700 1 --2- 172.21.15.87:0/2461627724 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc75c062c90 0x7fc75c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.650+0000 7fc769ffb700 1 -- 172.21.15.87:0/2461627724 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(738..738 src has 400..738) v4 ==== 8530+0+0 (secure 0 0 0) 0x7fc76007df60 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.650+0000 7fc769ffb700 10 client.38282.objecter ms_dispatch 0x7fc76c001550 osd_map(738..738 src has 400..738) v4 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.650+0000 7fc769ffb700 3 client.38282.objecter handle_osd_map got epochs [738,738] > 0 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.650+0000 7fc769ffb700 3 client.38282.objecter handle_osd_map decoding full epoch 738 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.650+0000 7fc76bfff700 1 --2- 172.21.15.87:0/2461627724 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc75c062c90 0x7fc75c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.650+0000 7fc769ffb700 20 client.38282.objecter dump_active .. 0 homeless 2024-02-23T00:42:41.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.650+0000 7fc769ffb700 10 client.38282.objecter ms_handle_connect 0x7fc75c062c90 2024-02-23T00:42:41.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.650+0000 7fc76bfff700 1 --2- 172.21.15.87:0/2461627724 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc75c062c90 0x7fc75c065150 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7fc758009a00 tx=0x7fc75800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:41.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.654+0000 7fc769ffb700 1 -- 172.21.15.87:0/2461627724 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc760051250 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:40.830+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc76c107240 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.143 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:41.173 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:41.193 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:41.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:41.297 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:41.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:42:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:42:41.324 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:41.343 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:41.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:41.394 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.390+0000 7fc769ffb700 1 -- 172.21.15.87:0/2461627724 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v739) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc760050e40 con 0x7fc76c0f7ba0 2024-02-23T00:42:41.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.395+0000 7fc772f8b700 10 librados: watch_flush enter 2024-02-23T00:42:41.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 10 librados: watch_flush exit 2024-02-23T00:42:41.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 20 client.38282.objecter shutdown clearing up homeless session... 2024-02-23T00:42:41.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 10 client.38282.objecter successfully canceled tick 2024-02-23T00:42:41.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc75c062c90 msgr2=0x7fc75c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:41.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 --2- 172.21.15.87:0/2461627724 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc75c062c90 0x7fc75c065150 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 rx=0x7fc758009a00 tx=0x7fc75800b040).stop 2024-02-23T00:42:41.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7ba0 msgr2=0x7fc76c10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:41.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 --2- 172.21.15.87:0/2461627724 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7ba0 0x7fc76c10e0e0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7fc760009eb0 tx=0x7fc76000b040).stop 2024-02-23T00:42:41.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 shutdown_connections 2024-02-23T00:42:41.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 --2- 172.21.15.87:0/2461627724 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc75c062c90 0x7fc75c065150 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:41.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 --2- 172.21.15.87:0/2461627724 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc76c0fd4a0 0x7fc76c108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:41.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 --2- 172.21.15.87:0/2461627724 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc76c0f7ba0 0x7fc76c10e0e0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:41.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 --2- 172.21.15.87:0/2461627724 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc76c0fcb40 0x7fc76c10e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:41.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 >> 172.21.15.87:0/2461627724 conn(0x7fc76c06c370 msgr2=0x7fc76c0f5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:41.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.400+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 shutdown_connections 2024-02-23T00:42:41.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.403+0000 7fc772f8b700 1 -- 172.21.15.87:0/2461627724 wait complete. 2024-02-23T00:42:41.404 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:41.403+0000 7fc772f8b700 1 librados: shutdown 2024-02-23T00:42:41.407 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:42:41.447 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:41.477 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:39.786087+0000 mgr.y (mgr.24437) 625 : cluster [DBG] pgmap v1124: 172 pgs: 1 active+remapped+backfill_toofull, 10 unknown, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:40.380362+0000 mon.a (mon.0) 3700 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-23T00:42:41.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:40.380417+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-23T00:42:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:40.389610+0000 mon.a (mon.0) 3702 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:40.830652+0000 mon.c (mon.2) 499 : audit [INF] from='client.? 172.21.15.87:0/2461627724' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:41.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:40.830933+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:41.826 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:41.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:39.786087+0000 mgr.y (mgr.24437) 625 : cluster [DBG] pgmap v1124: 172 pgs: 1 active+remapped+backfill_toofull, 10 unknown, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:41.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:40.380362+0000 mon.a (mon.0) 3700 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-23T00:42:41.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:40.380417+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-23T00:42:41.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:40.389610+0000 mon.a (mon.0) 3702 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:41.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:40.830652+0000 mon.c (mon.2) 499 : audit [INF] from='client.? 172.21.15.87:0/2461627724' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:41.828 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:40.830933+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:41.829 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:39.786087+0000 mgr.y (mgr.24437) 625 : cluster [DBG] pgmap v1124: 172 pgs: 1 active+remapped+backfill_toofull, 10 unknown, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:41.829 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:40.380362+0000 mon.a (mon.0) 3700 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num_actual", "val": "23"}]': finished 2024-02-23T00:42:41.829 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:40.380417+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-23T00:42:41.829 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:40.389610+0000 mon.a (mon.0) 3702 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:42:41.829 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:40.830652+0000 mon.c (mon.2) 499 : audit [INF] from='client.? 172.21.15.87:0/2461627724' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:41.829 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:40.830933+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:42:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:41.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:41.931 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:41.960 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:41.981 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:41.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:42.085 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:42.114 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:42.134 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:42.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:42.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:42.237 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:42.268 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:42.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:42] "GET /metrics HTTP/1.1" 200 187923 "" "Prometheus/2.33.4" 2024-02-23T00:42:42.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:41.245795+0000 mgr.y (mgr.24437) 626 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:41.378259+0000 mon.a (mon.0) 3704 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:41.384625+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:41.384748+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:41.384801+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:41.386140+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:42.384298+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:42.384391+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:42.386612+0000 mon.a (mon.0) 3711 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:41.245795+0000 mgr.y (mgr.24437) 626 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:41.378259+0000 mon.a (mon.0) 3704 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:41.384625+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:42.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:41.384748+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:42:42.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:41.384801+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-23T00:42:42.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:41.386140+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:42.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:42.384298+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:42.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:42.384391+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-02-23T00:42:42.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:42.386612+0000 mon.a (mon.0) 3711 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-23T00:42:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:41.245795+0000 mgr.y (mgr.24437) 626 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:42.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:41.378259+0000 mon.a (mon.0) 3704 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:41.384625+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:42:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:41.384748+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:42:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:41.384801+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-23T00:42:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:41.386140+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:42:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:42.384298+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:42:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:42.384391+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e740: 8 total, 8 up, 8 in 2024-02-23T00:42:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:42.386612+0000 mon.a (mon.0) 3711 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-23T00:42:43.318 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:43.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:43.338 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-23T00:42:43.349 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:43.422 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:43.448 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:41.786602+0000 mgr.y (mgr.24437) 627 : cluster [DBG] pgmap v1127: 204 pgs: 1 active+remapped+backfill_toofull, 9 peering, 27 creating+peering, 6 unknown, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:42.392231+0000 mon.a (mon.0) 3712 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:42:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:43.387550+0000 mon.a (mon.0) 3713 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_count","val": "3"}]': finished 2024-02-23T00:42:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:43.387598+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-02-23T00:42:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:43.387974+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-23T00:42:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:41.786602+0000 mgr.y (mgr.24437) 627 : cluster [DBG] pgmap v1127: 204 pgs: 1 active+remapped+backfill_toofull, 9 peering, 27 creating+peering, 6 unknown, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:42.392231+0000 mon.a (mon.0) 3712 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:42:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:43.387550+0000 mon.a (mon.0) 3713 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_count","val": "3"}]': finished 2024-02-23T00:42:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:43.387598+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-02-23T00:42:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:43.387974+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-23T00:42:43.646 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:43.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:41.786602+0000 mgr.y (mgr.24437) 627 : cluster [DBG] pgmap v1127: 204 pgs: 1 active+remapped+backfill_toofull, 9 peering, 27 creating+peering, 6 unknown, 161 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:43.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:42.392231+0000 mon.a (mon.0) 3712 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-23T00:42:43.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:43.387550+0000 mon.a (mon.0) 3713 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_count","val": "3"}]': finished 2024-02-23T00:42:43.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:43.387598+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e741: 8 total, 8 up, 8 in 2024-02-23T00:42:43.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:43.387974+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-23T00:42:43.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:43.777 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:43.899 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:43.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:44.003 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:44.029 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:44.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:42:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:42:44.228 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:44.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:44.331 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:44.359 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:44.783 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:44.808 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:44.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:44.848 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:42:44.913 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:44.944 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:45.048 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:45.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:45.153 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:45.186 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:45.208 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:45.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:45.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:45.311 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:45.342 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:43.786983+0000 mgr.y (mgr.24437) 628 : cluster [DBG] pgmap v1130: 204 pgs: 1 active+remapped+backfill_toofull, 9 peering, 27 creating+peering, 167 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:44.388590+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_period","val": "3"}]': finished 2024-02-23T00:42:45.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:44.388655+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-02-23T00:42:45.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:44.390395+0000 mon.a (mon.0) 3718 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:42:45.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:43.786983+0000 mgr.y (mgr.24437) 628 : cluster [DBG] pgmap v1130: 204 pgs: 1 active+remapped+backfill_toofull, 9 peering, 27 creating+peering, 167 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:45.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:44.388590+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_period","val": "3"}]': finished 2024-02-23T00:42:45.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:44.388655+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-02-23T00:42:45.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:44.390395+0000 mon.a (mon.0) 3718 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:42:45.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:43.786983+0000 mgr.y (mgr.24437) 628 : cluster [DBG] pgmap v1130: 204 pgs: 1 active+remapped+backfill_toofull, 9 peering, 27 creating+peering, 167 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:45.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:44.388590+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_period","val": "3"}]': finished 2024-02-23T00:42:45.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:44.388655+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e742: 8 total, 8 up, 8 in 2024-02-23T00:42:45.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:44.390395+0000 mon.a (mon.0) 3718 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 -- 172.21.15.87:0/1950967364 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f81480fcbb0 msgr2=0x7f81480fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 --2- 172.21.15.87:0/1950967364 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f81480fcbb0 0x7f81480fcf90 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 rx=0x7f81300047b0 tx=0x7f8130009cb0).stop 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 -- 172.21.15.87:0/1950967364 shutdown_connections 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 --2- 172.21.15.87:0/1950967364 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81480f7400 0x7f8148103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 --2- 172.21.15.87:0/1950967364 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f81480fcbb0 0x7f81480fcf90 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 --2- 172.21.15.87:0/1950967364 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81480fd510 0x7f81480f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 -- 172.21.15.87:0/1950967364 >> 172.21.15.87:0/1950967364 conn(0x7f81480f2820 msgr2=0x7f81480f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 -- 172.21.15.87:0/1950967364 shutdown_connections 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 -- 172.21.15.87:0/1950967364 wait complete. 2024-02-23T00:42:45.692 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 librados: starting msgr at 2024-02-23T00:42:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.669+0000 7f814d9a5700 1 librados: starting objecter 2024-02-23T00:42:45.695 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 Processor -- start 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 -- start start 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 librados: setting wanted keys 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 librados: calling monclient init 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81480f7400 0x7f8148109250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81480fcbb0 0x7f8148109740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f81480fd510 0x7f8148109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81481143e0 con 0x7f81480fcbb0 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8148114550 con 0x7f81480f7400 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f814d9a5700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8148114710 con 0x7f81480fd510 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f81467fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81480fcbb0 0x7f8148109740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f81477fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f81480fd510 0x7f8148109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f81467fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81480fcbb0 0x7f8148109740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47964/0 (socket says 172.21.15.87:47964) 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f81477fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f81480fd510 0x7f8148109c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:43020/0 (socket says 172.21.15.87:43020) 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f81467fc700 1 -- 172.21.15.87:0/2386201048 learned_addr learned my addr 172.21.15.87:0/2386201048 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.670+0000 7f8146ffd700 1 --2- 172.21.15.87:0/2386201048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81480f7400 0x7f8148109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f81467fc700 1 -- 172.21.15.87:0/2386201048 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f81480fd510 msgr2=0x7f8148109c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f81467fc700 1 --2- 172.21.15.87:0/2386201048 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f81480fd510 0x7f8148109c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f81467fc700 1 -- 172.21.15.87:0/2386201048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81480f7400 msgr2=0x7f8148109250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f81467fc700 1 --2- 172.21.15.87:0/2386201048 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f81480f7400 0x7f8148109250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f81467fc700 1 -- 172.21.15.87:0/2386201048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81400097c0 con 0x7f81480fcbb0 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f81477fe700 1 --2- 172.21.15.87:0/2386201048 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f81480fd510 0x7f8148109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f81467fc700 1 --2- 172.21.15.87:0/2386201048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81480fcbb0 0x7f8148109740 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7f813800c810 tx=0x7f813800cb20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f813ffff700 10 client.?.objecter ms_handle_connect 0x7f81480fcbb0 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f813ffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813800e4a0 con 0x7f81480fcbb0 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8130005040 con 0x7f81480fcbb0 2024-02-23T00:42:45.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f813800ead0 con 0x7f81480fcbb0 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8138012a70 con 0x7f81480fcbb0 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f814d9a5700 10 client.27810.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8148114d50 con 0x7f81480fcbb0 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.671+0000 7f814d9a5700 1 librados: init done 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.672+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f814805fa60 con 0x7f81480fcbb0 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.672+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f813800ec40 con 0x7f81480fcbb0 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f813ffff700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8134062c90 0x7f8134065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(743..743 src has 400..743) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f813807d760 con 0x7f81480fcbb0 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 osd_map(743..743 src has 400..743) v4 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f813ffff700 3 client.27810.objecter handle_osd_map got epochs [743,743] > 0 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f813ffff700 3 client.27810.objecter handle_osd_map decoding full epoch 743 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f8146ffd700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8134062c90 0x7f8134065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.698 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f813ffff700 20 client.27810.objecter dump_active .. 0 homeless 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v743) v1 ==== 54+0+16 (secure 0 0 0) 0x7f813807dc00 con 0x7f81480fcbb0 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f813ffff700 10 client.27810.objecter ms_handle_connect 0x7f8134062c90 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f8146ffd700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8134062c90 0x7f8134065150 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7f8130005010 tx=0x7f8130009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 10 client.27810.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 1 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6801/484081831 conn(0x7f8148028540 0x7f81480fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f814805ee80 osd=0 3 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814805ee80 osd=0 3 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f814805ee80 osd=0 3 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 1 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f814805ee80 osd=0 3 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 0 1 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814805ee80 osd=0 4 2024-02-23T00:42:45.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 10 client.27810.objecter _send_command 1 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8148061460 con 0x7f8148028540 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.673+0000 7f81477fe700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6801/484081831 conn(0x7f8148028540 0x7f81480fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.674+0000 7f813ffff700 10 client.27810.objecter ms_handle_connect 0x7f8148028540 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.674+0000 7f81477fe700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6801/484081831 conn(0x7f8148028540 0x7f81480fd9b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.674+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8148061460 con 0x7f8148028540 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.674+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.674+0000 7f813ffff700 10 client.27810.objecter _finish_command 1 = system:0 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.674+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f814805ee80 osd=0 4 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.674+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 0 1 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 10 client.27810.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:45.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 2 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f814805ee80 osd=0 3 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814805ee80 osd=0 3 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f814805ee80 osd=0 3 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 2 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f814805ee80 osd=0 3 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 0 2 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814805ee80 osd=0 4 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 10 client.27810.objecter _send_command 2 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.689+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8148106120 con 0x7f8148028540 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8148106120 con 0x7f8148028540 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f813ffff700 10 client.27810.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f814805ee80 osd=0 4 2024-02-23T00:42:45.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 0 2 2024-02-23T00:42:45.702 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:42:45.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 10 client.27810.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 3 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6805/685354171 conn(0x7f8148027b90 0x7f8148027fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81480ff020 osd=1 3 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81480ff020 osd=1 3 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81480ff020 osd=1 3 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 3 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f81480ff020 osd=1 3 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 1 3 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81480ff020 osd=1 4 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 10 client.27810.objecter _send_command 3 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81480ff190 con 0x7f8148027b90 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.690+0000 7f8146ffd700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6805/685354171 conn(0x7f8148027b90 0x7f8148027fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.691+0000 7f813ffff700 10 client.27810.objecter ms_handle_connect 0x7f8148027b90 2024-02-23T00:42:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.691+0000 7f8146ffd700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6805/685354171 conn(0x7f8148027b90 0x7f8148027fa0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.691+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f81480ff190 con 0x7f8148027b90 2024-02-23T00:42:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.691+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:42:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.691+0000 7f813ffff700 10 client.27810.objecter _finish_command 3 = system:0 2024-02-23T00:42:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.691+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f81480ff020 osd=1 4 2024-02-23T00:42:45.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.691+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 1 3 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.706+0000 7f814d9a5700 10 client.27810.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.706+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 4 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.707+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81480ff020 osd=1 3 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.707+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81480ff020 osd=1 3 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.707+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81480ff020 osd=1 3 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.707+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 4 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.707+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f81480ff020 osd=1 3 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.707+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 1 4 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.707+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81480ff020 osd=1 4 2024-02-23T00:42:45.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.707+0000 7f814d9a5700 10 client.27810.objecter _send_command 4 2024-02-23T00:42:45.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.707+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8148027600 con 0x7f8148027b90 2024-02-23T00:42:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8148027600 con 0x7f8148027b90 2024-02-23T00:42:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f813ffff700 10 client.27810.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f81480ff020 osd=1 4 2024-02-23T00:42:45.710 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 1 4 2024-02-23T00:42:45.710 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:42:45.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 10 client.27810.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:45.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 5 2024-02-23T00:42:45.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6809/3605394228 conn(0x7f81480feb80 0x7f814811c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148027a10 osd=2 3 2024-02-23T00:42:45.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148027a10 osd=2 3 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148027a10 osd=2 3 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 5 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f8148027a10 osd=2 3 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 2 5 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148027a10 osd=2 4 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 10 client.27810.objecter _send_command 5 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.708+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f814811edd0 con 0x7f81480feb80 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.709+0000 7f81467fc700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6809/3605394228 conn(0x7f81480feb80 0x7f814811c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.709+0000 7f813ffff700 10 client.27810.objecter ms_handle_connect 0x7f81480feb80 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.709+0000 7f81467fc700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6809/3605394228 conn(0x7f81480feb80 0x7f814811c180 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.709+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f814811edd0 con 0x7f81480feb80 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.709+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.709+0000 7f813ffff700 10 client.27810.objecter _finish_command 5 = system:0 2024-02-23T00:42:45.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.709+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f8148027a10 osd=2 4 2024-02-23T00:42:45.713 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.709+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 2 5 2024-02-23T00:42:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 10 client.27810.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 6 2024-02-23T00:42:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148027a10 osd=2 3 2024-02-23T00:42:45.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148027a10 osd=2 3 2024-02-23T00:42:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148027a10 osd=2 3 2024-02-23T00:42:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 6 2024-02-23T00:42:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f8148027a10 osd=2 3 2024-02-23T00:42:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 2 6 2024-02-23T00:42:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148027a10 osd=2 4 2024-02-23T00:42:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 10 client.27810.objecter _send_command 6 2024-02-23T00:42:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.726+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f81480fe0e0 con 0x7f81480feb80 2024-02-23T00:42:45.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f81480fe0e0 con 0x7f81480feb80 2024-02-23T00:42:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f813ffff700 10 client.27810.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f8148027a10 osd=2 4 2024-02-23T00:42:45.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 2 6 2024-02-23T00:42:45.730 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 10 client.27810.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 7 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6813/128059790 conn(0x7f81481081c0 0x7f814811e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81481197c0 osd=3 3 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481197c0 osd=3 3 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81481197c0 osd=3 3 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 7 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f81481197c0 osd=3 3 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 3 7 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481197c0 osd=3 4 2024-02-23T00:42:45.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 10 client.27810.objecter _send_command 7 2024-02-23T00:42:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.728+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f814811ec00 con 0x7f81481081c0 2024-02-23T00:42:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.729+0000 7f81477fe700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6813/128059790 conn(0x7f81481081c0 0x7f814811e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.729+0000 7f813ffff700 10 client.27810.objecter ms_handle_connect 0x7f81481081c0 2024-02-23T00:42:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.729+0000 7f81477fe700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6813/128059790 conn(0x7f81481081c0 0x7f814811e680 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.729+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f814811ec00 con 0x7f81481081c0 2024-02-23T00:42:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.729+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:42:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.729+0000 7f813ffff700 10 client.27810.objecter _finish_command 7 = system:0 2024-02-23T00:42:45.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.729+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f81481197c0 osd=3 4 2024-02-23T00:42:45.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.729+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 3 7 2024-02-23T00:42:45.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 10 client.27810.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:45.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 8 2024-02-23T00:42:45.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81481197c0 osd=3 3 2024-02-23T00:42:45.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481197c0 osd=3 3 2024-02-23T00:42:45.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81481197c0 osd=3 3 2024-02-23T00:42:45.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 8 2024-02-23T00:42:45.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f81481197c0 osd=3 3 2024-02-23T00:42:45.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 3 8 2024-02-23T00:42:45.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481197c0 osd=3 4 2024-02-23T00:42:45.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 10 client.27810.objecter _send_command 8 2024-02-23T00:42:45.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.744+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f81481085c0 con 0x7f81481081c0 2024-02-23T00:42:45.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f81481085c0 con 0x7f81481081c0 2024-02-23T00:42:45.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:45.746 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f813ffff700 10 client.27810.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f81481197c0 osd=3 4 2024-02-23T00:42:45.747 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 3 8 2024-02-23T00:42:45.747 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:42:45.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 10 client.27810.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:45.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 9 2024-02-23T00:42:45.748 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8148108c40 0x7f8148125360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81481085c0 osd=4 3 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481085c0 osd=4 3 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81481085c0 osd=4 3 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 9 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f81481085c0 osd=4 3 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 4 9 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481085c0 osd=4 4 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 10 client.27810.objecter _send_command 9 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.745+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8148127fb0 con 0x7f8148108c40 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.746+0000 7f8146ffd700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8148108c40 0x7f8148125360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.746+0000 7f813ffff700 10 client.27810.objecter ms_handle_connect 0x7f8148108c40 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.746+0000 7f8146ffd700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8148108c40 0x7f8148125360 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.747+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8148127fb0 con 0x7f8148108c40 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.747+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:42:45.749 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.747+0000 7f813ffff700 10 client.27810.objecter _finish_command 9 = system:0 2024-02-23T00:42:45.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.747+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f81481085c0 osd=4 4 2024-02-23T00:42:45.750 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.747+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 4 9 2024-02-23T00:42:45.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 10 client.27810.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 10 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81481085c0 osd=4 3 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481085c0 osd=4 3 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f81481085c0 osd=4 3 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 10 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f81481085c0 osd=4 3 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 4 10 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481085c0 osd=4 4 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 10 client.27810.objecter _send_command 10 2024-02-23T00:42:45.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.761+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8148125850 con 0x7f8148108c40 2024-02-23T00:42:45.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8148125850 con 0x7f8148108c40 2024-02-23T00:42:45.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:45.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f813ffff700 10 client.27810.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f81481085c0 osd=4 4 2024-02-23T00:42:45.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 4 10 2024-02-23T00:42:45.765 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 10 client.27810.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 11 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8148125f30 0x7f8148128360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148125850 osd=5 3 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148125850 osd=5 3 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148125850 osd=5 3 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 11 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f8148125850 osd=5 3 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 5 11 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148125850 osd=5 4 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 10 client.27810.objecter _send_command 11 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.763+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f814812b130 con 0x7f8148125f30 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.764+0000 7f81467fc700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8148125f30 0x7f8148128360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.764+0000 7f813ffff700 10 client.27810.objecter ms_handle_connect 0x7f8148125f30 2024-02-23T00:42:45.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.764+0000 7f81467fc700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8148125f30 0x7f8148128360 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.765+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f814812b130 con 0x7f8148125f30 2024-02-23T00:42:45.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.765+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:42:45.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.765+0000 7f813ffff700 10 client.27810.objecter _finish_command 11 = system:0 2024-02-23T00:42:45.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.765+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f8148125850 osd=5 4 2024-02-23T00:42:45.767 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.765+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 5 11 2024-02-23T00:42:45.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 10 client.27810.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:45.780 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 12 2024-02-23T00:42:45.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148125850 osd=5 3 2024-02-23T00:42:45.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148125850 osd=5 3 2024-02-23T00:42:45.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148125850 osd=5 3 2024-02-23T00:42:45.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 12 2024-02-23T00:42:45.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f8148125850 osd=5 3 2024-02-23T00:42:45.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 5 12 2024-02-23T00:42:45.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148125850 osd=5 4 2024-02-23T00:42:45.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 10 client.27810.objecter _send_command 12 2024-02-23T00:42:45.781 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.779+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f814812b130 con 0x7f8148125f30 2024-02-23T00:42:45.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f814812b130 con 0x7f8148125f30 2024-02-23T00:42:45.782 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:45.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f813ffff700 10 client.27810.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f8148125850 osd=5 4 2024-02-23T00:42:45.783 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 5 12 2024-02-23T00:42:45.783 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 10 client.27810.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 13 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6808/2509408498 conn(0x7f814812a4e0 0x7f814812a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f814812b130 osd=6 3 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814812b130 osd=6 3 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f814812b130 osd=6 3 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 13 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f814812b130 osd=6 3 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 6 13 2024-02-23T00:42:45.784 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814812b130 osd=6 4 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 10 client.27810.objecter _send_command 13 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.781+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f814812e4e0 con 0x7f814812a4e0 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.782+0000 7f81477fe700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6808/2509408498 conn(0x7f814812a4e0 0x7f814812a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.782+0000 7f813ffff700 10 client.27810.objecter ms_handle_connect 0x7f814812a4e0 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.782+0000 7f81477fe700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6808/2509408498 conn(0x7f814812a4e0 0x7f814812a940 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.783+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f814812e4e0 con 0x7f814812a4e0 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.783+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.783+0000 7f813ffff700 10 client.27810.objecter _finish_command 13 = system:0 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.783+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f814812b130 osd=6 4 2024-02-23T00:42:45.785 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.783+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 6 13 2024-02-23T00:42:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 10 client.27810.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 14 2024-02-23T00:42:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f814812b130 osd=6 3 2024-02-23T00:42:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814812b130 osd=6 3 2024-02-23T00:42:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f814812b130 osd=6 3 2024-02-23T00:42:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 14 2024-02-23T00:42:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f814812b130 osd=6 3 2024-02-23T00:42:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 6 14 2024-02-23T00:42:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814812b130 osd=6 4 2024-02-23T00:42:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 10 client.27810.objecter _send_command 14 2024-02-23T00:42:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.797+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8148128910 con 0x7f814812a4e0 2024-02-23T00:42:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8148128910 con 0x7f814812a4e0 2024-02-23T00:42:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f813ffff700 10 client.27810.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f814812b130 osd=6 4 2024-02-23T00:42:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 6 14 2024-02-23T00:42:45.801 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:42:45.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 10 client.27810.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:45.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 15 2024-02-23T00:42:45.801 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6812/373787000 conn(0x7f8148128fc0 0x7f8148129420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148128910 osd=7 3 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148128910 osd=7 3 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148128910 osd=7 3 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 15 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f8148128910 osd=7 3 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 7 15 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148128910 osd=7 4 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 10 client.27810.objecter _send_command 15 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8148129d90 con 0x7f8148128fc0 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.799+0000 7f8146ffd700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6812/373787000 conn(0x7f8148128fc0 0x7f8148129420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.800+0000 7f813ffff700 10 client.27810.objecter ms_handle_connect 0x7f8148128fc0 2024-02-23T00:42:45.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.800+0000 7f8146ffd700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6812/373787000 conn(0x7f8148128fc0 0x7f8148129420 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:45.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.801+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8148129d90 con 0x7f8148128fc0 2024-02-23T00:42:45.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.801+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:42:45.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.801+0000 7f813ffff700 10 client.27810.objecter _finish_command 15 = system:0 2024-02-23T00:42:45.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.801+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f8148128910 osd=7 4 2024-02-23T00:42:45.803 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.801+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 7 15 2024-02-23T00:42:45.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 10 client.27810.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:42:45.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign -1 16 2024-02-23T00:42:45.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148128910 osd=7 3 2024-02-23T00:42:45.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148128910 osd=7 3 2024-02-23T00:42:45.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 20 client.27810.objecter _get_session s=0x7f8148128910 osd=7 3 2024-02-23T00:42:45.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_remove -1 16 2024-02-23T00:42:45.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 20 client.27810.objecter get_session s=0x7f8148128910 osd=7 3 2024-02-23T00:42:45.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 15 client.27810.objecter _session_command_op_assign 7 16 2024-02-23T00:42:45.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148128910 osd=7 4 2024-02-23T00:42:45.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 10 client.27810.objecter _send_command 16 2024-02-23T00:42:45.817 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.815+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f8148129d90 con 0x7f8148128fc0 2024-02-23T00:42:45.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f813ffff700 1 -- 172.21.15.87:0/2386201048 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f8148129d90 con 0x7f8148128fc0 2024-02-23T00:42:45.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f813ffff700 10 client.27810.objecter ms_dispatch 0x7f8148001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:42:45.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f813ffff700 10 client.27810.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f813ffff700 20 client.27810.objecter put_session s=0x7f8148128910 osd=7 4 2024-02-23T00:42:45.818 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f813ffff700 15 client.27810.objecter _session_command_op_remove 7 16 2024-02-23T00:42:45.819 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:42:45.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 librados: watch_flush enter 2024-02-23T00:42:45.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 librados: watch_flush exit 2024-02-23T00:42:45.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 client.27810.objecter close_session for osd.0 2024-02-23T00:42:45.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6801/484081831 conn(0x7f8148028540 msgr2=0x7f81480fd9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6801/484081831 conn(0x7f8148028540 0x7f81480fd9b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814805ee80 osd=0 1 2024-02-23T00:42:45.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 client.27810.objecter close_session for osd.1 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6805/685354171 conn(0x7f8148027b90 msgr2=0x7f8148027fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6805/685354171 conn(0x7f8148027b90 0x7f8148027fa0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81480ff020 osd=1 1 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 client.27810.objecter close_session for osd.2 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6809/3605394228 conn(0x7f81480feb80 msgr2=0x7f814811c180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6809/3605394228 conn(0x7f81480feb80 0x7f814811c180 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148027a10 osd=2 1 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 client.27810.objecter close_session for osd.3 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6813/128059790 conn(0x7f81481081c0 msgr2=0x7f814811e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6813/128059790 conn(0x7f81481081c0 0x7f814811e680 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481197c0 osd=3 1 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 client.27810.objecter close_session for osd.4 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8148108c40 msgr2=0x7f8148125360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.821 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6800/4175844757 conn(0x7f8148108c40 0x7f8148125360 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f81481085c0 osd=4 1 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 client.27810.objecter close_session for osd.5 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8148125f30 msgr2=0x7f8148128360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8148125f30 0x7f8148128360 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148125850 osd=5 1 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 client.27810.objecter close_session for osd.6 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6808/2509408498 conn(0x7f814812a4e0 msgr2=0x7f814812a940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6808/2509408498 conn(0x7f814812a4e0 0x7f814812a940 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f814812b130 osd=6 1 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 client.27810.objecter close_session for osd.7 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6812/373787000 conn(0x7f8148128fc0 msgr2=0x7f8148129420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.171:6812/373787000 conn(0x7f8148128fc0 0x7f8148129420 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 20 client.27810.objecter put_session s=0x7f8148128910 osd=7 1 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 20 client.27810.objecter shutdown clearing up homeless session... 2024-02-23T00:42:45.822 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 10 client.27810.objecter successfully canceled tick 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f81477fe700 1 -- 172.21.15.87:0/2386201048 reap_dead start 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f81477fe700 1 -- 172.21.15.87:0/2386201048 reap_dead start 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8134062c90 msgr2=0x7f8134065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> v2:172.21.15.87:6800/1231597463 conn(0x7f8134062c90 0x7f8134065150 secure :-1 s=READY pgs=2964 cs=0 l=1 rev1=1 rx=0x7f8130005010 tx=0x7f8130009cb0).stop 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81480fcbb0 msgr2=0x7f8148109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f81477fe700 1 -- 172.21.15.87:0/2386201048 reap_dead start 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f814d9a5700 1 --2- 172.21.15.87:0/2386201048 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f81480fcbb0 0x7f8148109740 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 rx=0x7f813800c810 tx=0x7f813800cb20).stop 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f81477fe700 1 -- 172.21.15.87:0/2386201048 reap_dead start 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.817+0000 7f81477fe700 1 -- 172.21.15.87:0/2386201048 reap_dead start 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.818+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 shutdown_connections 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.818+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 >> 172.21.15.87:0/2386201048 conn(0x7f81480f2820 msgr2=0x7f81480fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.818+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 shutdown_connections 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.818+0000 7f814d9a5700 1 -- 172.21.15.87:0/2386201048 wait complete. 2024-02-23T00:42:45.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:45.818+0000 7f814d9a5700 1 librados: shutdown 2024-02-23T00:42:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:42:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:42:46.615 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-23T00:42:46.615 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:42:46.616 INFO:tasks.thrashosds.thrasher:Shrinking pool device_health_metrics 2024-02-23T00:42:46.616 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool device_health_metrics by 10 min 8 2024-02-23T00:42:46.616 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:42:46.645 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:45.389787+0000 mon.a (mon.0) 3719 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:42:46.645 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:45.389850+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-02-23T00:42:46.645 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:45.391640+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-23T00:42:46.645 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:45.672801+0000 mon.a (mon.0) 3722 : audit [DBG] from='client.? 172.21.15.87:0/2386201048' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:46.645 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:46.390196+0000 mon.a (mon.0) 3723 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-23T00:42:46.646 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:46.390270+0000 mon.a (mon.0) 3724 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-02-23T00:42:46.646 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:45.389787+0000 mon.a (mon.0) 3719 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:42:46.646 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:45.389850+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-02-23T00:42:46.646 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:45.391640+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-23T00:42:46.646 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:45.672801+0000 mon.a (mon.0) 3722 : audit [DBG] from='client.? 172.21.15.87:0/2386201048' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:46.646 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:46.390196+0000 mon.a (mon.0) 3723 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-23T00:42:46.646 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:46.390270+0000 mon.a (mon.0) 3724 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-02-23T00:42:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:45.389787+0000 mon.a (mon.0) 3719 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:42:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:45.389850+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e743: 8 total, 8 up, 8 in 2024-02-23T00:42:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:45.391640+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-23T00:42:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:45.672801+0000 mon.a (mon.0) 3722 : audit [DBG] from='client.? 172.21.15.87:0/2386201048' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:46.390196+0000 mon.a (mon.0) 3723 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-23T00:42:46.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:46.390270+0000 mon.a (mon.0) 3724 : cluster [DBG] osdmap e744: 8 total, 8 up, 8 in 2024-02-23T00:42:47.391 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:47.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:47.496 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:47.525 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:45.787789+0000 mgr.y (mgr.24437) 629 : cluster [DBG] pgmap v1133: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:46.393097+0000 mon.a (mon.0) 3725 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-23T00:42:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:47.228877+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-02-23T00:42:47.771 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:45.787789+0000 mgr.y (mgr.24437) 629 : cluster [DBG] pgmap v1133: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:47.771 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:46.393097+0000 mon.a (mon.0) 3725 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-23T00:42:47.771 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:47.228877+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-02-23T00:42:47.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:45.787789+0000 mgr.y (mgr.24437) 629 : cluster [DBG] pgmap v1133: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:47.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:46.393097+0000 mon.a (mon.0) 3725 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-23T00:42:47.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:47.228877+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e745: 8 total, 8 up, 8 in 2024-02-23T00:42:47.773 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:47.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:47.877 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:47.910 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:47.931 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:47.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:48.034 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPush(4.3 678/677 [PushOp(4:c623b173:gc::gc.16:head, version: 62'13, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:c623b173:gc::gc.16:head@62'13, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:37.422176+0000", 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 70.608878527000002, 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00063715400000000002, 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.035 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.422176+0000", 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.422176+0000", 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.036 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.037 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.422175+0000", 2024-02-23T00:42:48.037 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3825e-05 2024-02-23T00:42:48.039 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.039 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.039 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.040 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.422198+0000", 2024-02-23T00:42:48.040 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9913999999999999e-05 2024-02-23T00:42:48.040 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.040 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.040 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.040 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.422218+0000", 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6964000000000001e-05 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.422245+0000", 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0518999999999993e-05 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.422316+0000", 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1459e-05 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.422327+0000", 2024-02-23T00:42:48.041 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00048627700000000002 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.422813+0000", 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00063715400000000002 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPush(4.3 678/677 [PushOp(4:c60da066:gc::gc.4:head, version: 62'15, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:c60da066:gc::gc.4:head@62'15, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:37.594316+0000", 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 70.436738564999999, 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00055990300000000005, 2024-02-23T00:42:48.042 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.594316+0000", 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.594316+0000", 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.043 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.594316+0000", 2024-02-23T00:42:48.044 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.045e-05 2024-02-23T00:42:48.044 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.044 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.044 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.044 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.594326+0000", 2024-02-23T00:42:48.044 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9119000000000003e-05 2024-02-23T00:42:48.044 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.044 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.044 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.045 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.594365+0000", 2024-02-23T00:42:48.045 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.973e-06 2024-02-23T00:42:48.045 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.045 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.045 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.045 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.594372+0000", 2024-02-23T00:42:48.045 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9417000000000003e-05 2024-02-23T00:42:48.045 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.046 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.046 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.046 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.594442+0000", 2024-02-23T00:42:48.046 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0122000000000001e-05 2024-02-23T00:42:48.046 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.046 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.046 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.046 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.594452+0000", 2024-02-23T00:42:48.047 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00042444200000000003 2024-02-23T00:42:48.047 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.047 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.047 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.047 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.594876+0000", 2024-02-23T00:42:48.047 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00055990300000000005 2024-02-23T00:42:48.047 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.047 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.047 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.048 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.048 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.048 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPush(4.6 679/678 [PushOp(4:6192870f:::obj_delete_at_hint.0000000053:head, version: 453'33, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6192870f:::obj_delete_at_hint.0000000053:head@453'33, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-23T00:42:48.048 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.941414+0000", 2024-02-23T00:42:48.048 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 69.089640974000005, 2024-02-23T00:42:48.048 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0021045769999999998, 2024-02-23T00:42:48.048 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.048 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.048 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.049 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.049 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.049 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.941414+0000", 2024-02-23T00:42:48.049 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.049 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.049 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.049 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.049 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.941414+0000", 2024-02-23T00:42:48.050 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:48.050 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.050 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.050 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.050 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.941413+0000", 2024-02-23T00:42:48.050 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4579999999999993e-06 2024-02-23T00:42:48.050 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.050 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.050 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.051 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.941422+0000", 2024-02-23T00:42:48.051 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4269e-05 2024-02-23T00:42:48.051 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.051 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.051 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.051 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.941436+0000", 2024-02-23T00:42:48.051 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9629999999999998e-06 2024-02-23T00:42:48.051 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.051 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.941444+0000", 2024-02-23T00:42:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0019893340000000002 2024-02-23T00:42:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.943433+0000", 2024-02-23T00:42:48.052 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5010000000000001e-06 2024-02-23T00:42:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.943440+0000", 2024-02-23T00:42:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8590000000000005e-05 2024-02-23T00:42:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.053 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.943518+0000", 2024-02-23T00:42:48.054 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0021045769999999998 2024-02-23T00:42:48.054 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.054 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.054 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.054 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.054 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.054 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_backfill(progress 4.19 e 679/679 lb 4:9f626c48:lc::lc.21:head)", 2024-02-23T00:42:48.054 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.209553+0000", 2024-02-23T00:42:48.054 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 68.821501806000001, 2024-02-23T00:42:48.055 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00061034200000000005, 2024-02-23T00:42:48.055 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.055 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.055 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.055 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.055 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.055 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209553+0000", 2024-02-23T00:42:48.055 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.056 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.056 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.056 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.056 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209553+0000", 2024-02-23T00:42:48.056 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:48.056 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.056 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.056 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209553+0000", 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.0219999999999993e-06 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209562+0000", 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8909000000000001e-05 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.057 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209581+0000", 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0009e-05 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209591+0000", 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00049498999999999995 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.210086+0000", 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1418999999999999e-05 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.058 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.210107+0000", 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6350000000000001e-05 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.210163+0000", 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00061034200000000005 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPush(4.19 679/678 [PushOp(4:9fecf831:::obj_delete_at_hint.0000000057:head, version: 453'60, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:9fecf831:::obj_delete_at_hint.0000000057:head@453'60, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.209605+0000", 2024-02-23T00:42:48.059 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 68.821449955000006, 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00076825299999999999, 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209605+0000", 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209605+0000", 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:48.060 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209605+0000", 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5917999999999999e-05 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209620+0000", 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.385e-05 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209634+0000", 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0214999999999999e-05 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.061 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.209654+0000", 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00055555100000000005 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.210210+0000", 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4509999999999998e-06 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.210219+0000", 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000153664 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.062 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.210373+0000", 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00076825299999999999 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_backfill(progress 4.b e 680/680 lb 4:d4f18cce:::obj_delete_at_hint.0000000107:head)", 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.211064+0000", 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 68.819991282000004, 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.004541855, 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.063 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.211064+0000", 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.211064+0000", 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.211063+0000", 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2600000000000005e-06 2024-02-23T00:42:48.064 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.211071+0000", 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1210000000000001e-06 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.211078+0000", 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0160000000000003e-06 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.211085+0000", 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000150576 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.065 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.211236+0000", 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8720000000000002e-06 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.211244+0000", 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0043081300000000003 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.215552+0000", 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3502e-05 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.066 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.215566+0000", 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9891000000000001e-05 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.215605+0000", 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.004541855 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.067 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_backfill(progress 4.6 e 680/680 lb 4:63d917ed:::obj_delete_at_hint.0000000065:head)", 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.216536+0000", 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 68.814518437999993, 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0010003970000000001, 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216536+0000", 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.068 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216536+0000", 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216535+0000", 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0648e-05 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216546+0000", 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0200999999999999e-05 2024-02-23T00:42:48.069 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216556+0000", 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5019999999999998e-06 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216563+0000", 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00092620799999999998 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.070 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217489+0000", 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3916e-05 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217503+0000", 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3850999999999998e-05 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217537+0000", 2024-02-23T00:42:48.071 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0010003970000000001 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPush(4.6 680/678 [PushOp(4:64735f45:::obj_delete_at_hint.0000000083:head, version: 453'37, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:64735f45:::obj_delete_at_hint.0000000083:head@453'37, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.216837+0000", 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 68.814218335000007, 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000817515, 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.072 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216837+0000", 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216837+0000", 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216836+0000", 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1858999999999999e-05 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.073 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216848+0000", 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2532e-05 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216861+0000", 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8969999999999997e-06 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216869+0000", 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00070608499999999996 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.074 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217575+0000", 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4859999999999999e-06 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217580+0000", 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4054999999999995e-05 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217654+0000", 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000817515 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.075 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_update_log_missing(5.3 epoch 681/678 rep_tid 724 entries 681'20 (0'0) error 5:c609908c:::notify.5:head by client.14397.0:2107 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:40.500133+0000", 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 67.530921868999997, 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00055763600000000003, 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:40.500133+0000", 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.076 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:40.500133+0000", 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:40.500132+0000", 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2695e-05 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:40.500145+0000", 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2427e-05 2024-02-23T00:42:48.077 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:40.500157+0000", 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3280000000000002e-06 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:40.500163+0000", 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1608999999999999e-05 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.078 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:40.500225+0000", 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046571399999999999 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:40.500691+0000", 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00055763600000000003 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2127 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94532396649856 gen 4] snapc 0=[] ondisk+write+known_if_redirected e689)", 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:50.501198+0000", 2024-02-23T00:42:48.079 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 57.529857292999999, 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001069551, 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2127 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.080 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.501198+0000", 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.501198+0000", 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.501196+0000", 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6519999999999995e-06 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.501205+0000", 2024-02-23T00:42:48.081 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4819e-05 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.501219+0000", 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9259999999999997e-06 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.501225+0000", 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3570999999999999e-05 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.082 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.501279+0000", 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000112086 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.501391+0000", 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00087609600000000004 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.502267+0000", 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001069551 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.083 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:1231 243.18 243:18000000::::head [pgnls start_epoch 691] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e691)", 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:53.314129+0000", 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 54.71692616, 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016560349999999999, 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1231 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.084 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.314129+0000", 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.314129+0000", 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.314127+0000", 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8300000000000008e-06 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.085 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.314137+0000", 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6070999999999999e-05 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.314153+0000", 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4860000000000002e-06 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.314159+0000", 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9313000000000001e-05 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.086 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.314199+0000", 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2780000000000004e-06 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.314203+0000", 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001445841 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.315649+0000", 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7138000000000003e-05 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.087 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.315686+0000", 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8879e-05 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:53.315785+0000", 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016560349999999999 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDScrubReserve(3.7 REQUEST e692)", 2024-02-23T00:42:48.088 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:54.359101+0000", 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 53.671953887999997, 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0030414399999999999, 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.359101+0000", 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.359101+0000", 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.089 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.359101+0000", 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5130000000000001e-06 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.359104+0000", 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2739999999999999e-05 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.359127+0000", 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2579999999999995e-06 2024-02-23T00:42:48.090 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.359135+0000", 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6050000000000001e-05 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.359181+0000", 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.9080000000000001e-06 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.359190+0000", 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0029170300000000001 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.091 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.362107+0000", 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2767e-05 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.362120+0000", 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2402e-05 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:54.362142+0000", 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0030414399999999999 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.092 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2128 5.e 5:7759931f:::notify.3:head [watch ping cookie 94532396649856 gen 4] snapc 0=[] ondisk+write+known_if_redirected e689)", 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:55.501526+0000", 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 52.529528919999997, 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00068506600000000002, 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2128 2024-02-23T00:42:48.093 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:55.501526+0000", 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:55.501526+0000", 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:55.501523+0000", 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3701e-05 2024-02-23T00:42:48.094 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:55.501537+0000", 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2282000000000001e-05 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:55.501559+0000", 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1355999999999999e-05 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:55.501571+0000", 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046418000000000002 2024-02-23T00:42:48.095 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:55.502035+0000", 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8527000000000002e-05 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:55.502114+0000", 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.7462999999999998e-05 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:55.502211+0000", 2024-02-23T00:42:48.096 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00068506600000000002 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2155 5.e 5:7759931f:::notify.3:head [watch ping cookie 94532396649856 gen 4] snapc 0=[] ondisk+write+known_if_redirected e703)", 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:10.504574+0000", 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 37.526480644000003, 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00068604400000000002, 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2155 2024-02-23T00:42:48.097 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:10.504574+0000", 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:10.504574+0000", 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999967 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:10.504571+0000", 2024-02-23T00:42:48.098 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6724000000000001e-05 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:10.504587+0000", 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3400999999999998e-05 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:10.504611+0000", 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.057e-05 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:10.504621+0000", 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046162800000000001 2024-02-23T00:42:48.099 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:10.505083+0000", 2024-02-23T00:42:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8095000000000004e-05 2024-02-23T00:42:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.100 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:10.505161+0000", 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9168999999999999e-05 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:10.505260+0000", 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00068604400000000002 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:1584 245.18 245:18000000:%01:::head [pgnls start_epoch 710] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e710)", 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:12.256010+0000", 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 35.775045200999998, 2024-02-23T00:42:48.101 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0028375969999999999, 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1584 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256010+0000", 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.102 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256010+0000", 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256008+0000", 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1370000000000008e-06 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.103 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256017+0000", 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5344000000000001e-05 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256032+0000", 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9439999999999999e-06 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256039+0000", 2024-02-23T00:42:48.104 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3953000000000003e-05 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256093+0000", 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0999999999999998e-06 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256100+0000", 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0026205669999999999 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.258721+0000", 2024-02-23T00:42:48.105 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7098000000000003e-05 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.258758+0000", 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.9449000000000004e-05 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.258847+0000", 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0028375969999999999 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.106 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:1618 245.18 245:18000000::::head [pgnls start_epoch 712] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e712)", 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:14.267308+0000", 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 33.763746394000002, 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0019384960000000001, 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1618 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.107 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.267308+0000", 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.267308+0000", 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.267307+0000", 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4420000000000004e-06 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.108 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.267316+0000", 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3127e-05 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.267329+0000", 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6930000000000001e-05 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.267356+0000", 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9809999999999999e-05 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.109 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.267406+0000", 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.587e-06 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.267412+0000", 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001710561 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.269122+0000", 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2127999999999999e-05 2024-02-23T00:42:48.110 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.269144+0000", 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000102496 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:14.269247+0000", 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0019384960000000001 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.111 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDScrubReserve(5.8 REQUEST e724)", 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:26.378161+0000", 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 21.652893546000001, 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0013114120000000001, 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.378161+0000", 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.112 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.378161+0000", 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.378161+0000", 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.954e-06 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.378164+0000", 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1702e-05 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.378186+0000", 2024-02-23T00:42:48.113 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3950000000000001e-06 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.378194+0000", 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3356999999999998e-05 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.378238+0000", 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3509999999999996e-06 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.378245+0000", 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011841379999999999 2024-02-23T00:42:48.114 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.379429+0000", 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2408999999999999e-05 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.379441+0000", 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1254000000000003e-05 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:26.379473+0000", 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0013114120000000001 2024-02-23T00:42:48.115 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:1800 247.4 247:20000000:%01:::head [pgnls start_epoch 726] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e726)", 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.277437+0000", 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 19.753618045, 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001949337, 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1800 2024-02-23T00:42:48.116 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.277437+0000", 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.277437+0000", 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.277435+0000", 2024-02-23T00:42:48.117 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0727000000000001e-05 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.277446+0000", 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5868000000000001e-05 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.277461+0000", 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0630000000000002e-06 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.277467+0000", 2024-02-23T00:42:48.118 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7608999999999998e-05 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.277515+0000", 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8300000000000003e-06 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.277520+0000", 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0017180019999999999 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.279238+0000", 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0148000000000003e-05 2024-02-23T00:42:48.119 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.279278+0000", 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000108121 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.279386+0000", 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001949337 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.120 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:1834 247.4 247:20000000::::head [pgnls start_epoch 728] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e728)", 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:30.297412+0000", 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 17.733643119, 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0022685560000000001, 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1834 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.121 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.297412+0000", 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.297412+0000", 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.297409+0000", 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3305999999999999e-05 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.122 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.297422+0000", 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.4372999999999999e-05 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.297447+0000", 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3759999999999997e-06 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.297456+0000", 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1589999999999998e-05 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.123 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.297518+0000", 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1399999999999997e-06 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.297524+0000", 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002010282 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.299534+0000", 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1485000000000001e-05 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.124 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.299566+0000", 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000114571 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:30.299680+0000", 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0022685560000000001 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27567.0:1998 249.6 e744/738 249:602f83fe:::foo:head v 744'1, mlcod=0'0)", 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:46.395275+0000", 2024-02-23T00:42:48.125 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 1.6357803049999999, 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00098940599999999997, 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.395275+0000", 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.395275+0000", 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.126 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.395274+0000", 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4094000000000003e-05 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.395308+0000", 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1644000000000001e-05 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.395320+0000", 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7690000000000002e-06 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.127 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:42:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.395329+0000", 2024-02-23T00:42:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5731999999999998e-05 2024-02-23T00:42:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:42:48.128 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.395414+0000", 2024-02-23T00:42:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2755000000000002e-05 2024-02-23T00:42:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:42:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.395437+0000", 2024-02-23T00:42:48.129 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00079670399999999997 2024-02-23T00:42:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:42:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.396234+0000", 2024-02-23T00:42:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0073000000000002e-05 2024-02-23T00:42:48.130 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:42:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:42:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:42:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:46.396264+0000", 2024-02-23T00:42:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00098940599999999997 2024-02-23T00:42:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:42:48.131 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:42:48.132 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:42:48.132 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:48.133 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:48.133 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:48.161 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:48.284 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:48.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:48.388 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:48.415 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:48.562 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:48.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:48.668 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:48.696 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:48.821 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:48.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:48.925 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:48.952 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:49.402 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:49.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:49.414 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:42:49.507 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:49.540 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:49.560 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:49.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:49.665 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:47.788257+0000 mgr.y (mgr.24437) 630 : cluster [DBG] pgmap v1136: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:49.695 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:49.716 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:49.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:49.820 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:49.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:47.788257+0000 mgr.y (mgr.24437) 630 : cluster [DBG] pgmap v1136: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:49.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:47.788257+0000 mgr.y (mgr.24437) 630 : cluster [DBG] pgmap v1136: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:49.848 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:49.868 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:49.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:49.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:50.000 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:50.019 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:50.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:50.123 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:50.150 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:50.168 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:50.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:50.272 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:50.300 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:50.949 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:50.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:51.053 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:51.083 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:51.103 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:51.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:51.206 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:51.239 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:51.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:42:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:42:51.423 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:49.788821+0000 mgr.y (mgr.24437) 631 : cluster [DBG] pgmap v1137: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:51.423 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:49.788821+0000 mgr.y (mgr.24437) 631 : cluster [DBG] pgmap v1137: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:51.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 -- 172.21.15.87:0/931416196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380fe510 msgr2=0x7fd1380f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:51.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 --2- 172.21.15.87:0/931416196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380fe510 0x7fd1380f7d40 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 rx=0x7fd134004870 tx=0x7fd134009cb0).stop 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 -- 172.21.15.87:0/931416196 shutdown_connections 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 --2- 172.21.15.87:0/931416196 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1380f8420 0x7fd138104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 --2- 172.21.15.87:0/931416196 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1380fdbb0 0x7fd1380fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 --2- 172.21.15.87:0/931416196 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380fe510 0x7fd1380f7d40 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 -- 172.21.15.87:0/931416196 >> 172.21.15.87:0/931416196 conn(0x7fd1380f3800 msgr2=0x7fd1380f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 -- 172.21.15.87:0/931416196 shutdown_connections 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 -- 172.21.15.87:0/931416196 wait complete. 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 librados: starting msgr at 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.655+0000 7fd13fc33700 1 librados: starting objecter 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 Processor -- start 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 -- start start 2024-02-23T00:42:51.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 librados: setting wanted keys 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 librados: calling monclient init 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380f8420 0x7fd13810dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1380fdbb0 0x7fd13810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1380fe510 0x7fd138108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd138108680 con 0x7fd1380f8420 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1381087f0 con 0x7fd1380fe510 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13fc33700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1381089b0 con 0x7fd1380fdbb0 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d9cf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380f8420 0x7fd13810dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d1ce700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1380fdbb0 0x7fd13810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d9cf700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380f8420 0x7fd13810dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52964/0 (socket says 172.21.15.87:52964) 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d9cf700 1 -- 172.21.15.87:0/735771470 learned_addr learned my addr 172.21.15.87:0/735771470 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13e1d0700 1 --2- 172.21.15.87:0/735771470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1380fe510 0x7fd138108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d1ce700 1 -- 172.21.15.87:0/735771470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1380fe510 msgr2=0x7fd138108190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:51.661 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d1ce700 1 --2- 172.21.15.87:0/735771470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1380fe510 0x7fd138108190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d1ce700 1 -- 172.21.15.87:0/735771470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380f8420 msgr2=0x7fd13810dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d1ce700 1 --2- 172.21.15.87:0/735771470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380f8420 0x7fd13810dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d1ce700 1 -- 172.21.15.87:0/735771470 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd128005040 con 0x7fd1380fdbb0 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d9cf700 1 --2- 172.21.15.87:0/735771470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380f8420 0x7fd13810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd13d1ce700 1 --2- 172.21.15.87:0/735771470 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1380fdbb0 0x7fd13810e4e0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7fd134004870 tx=0x7fd13400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd12effd700 10 client.?.objecter ms_handle_connect 0x7fd1380fdbb0 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd12effd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd12effd700 1 -- 172.21.15.87:0/735771470 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13400e070 con 0x7fd1380fdbb0 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.656+0000 7fd12effd700 1 -- 172.21.15.87:0/735771470 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd134013da0 con 0x7fd1380fdbb0 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.657+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd134005040 con 0x7fd1380fdbb0 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.657+0000 7fd12effd700 1 -- 172.21.15.87:0/735771470 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd134019070 con 0x7fd1380fdbb0 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.657+0000 7fd13fc33700 10 client.38294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.657+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd138108f20 con 0x7fd1380fdbb0 2024-02-23T00:42:51.662 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.657+0000 7fd13fc33700 1 librados: init done 2024-02-23T00:42:51.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.658+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13805ee80 con 0x7fd1380fdbb0 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.658+0000 7fd12effd700 1 -- 172.21.15.87:0/735771470 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd13401e420 con 0x7fd1380fdbb0 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.658+0000 7fd12effd700 1 --2- 172.21.15.87:0/735771470 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd124062c90 0x7fd124065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.658+0000 7fd12effd700 1 -- 172.21.15.87:0/735771470 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(745..745 src has 400..745) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fd13407dd40 con 0x7fd1380fdbb0 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.658+0000 7fd12effd700 10 client.38294.objecter ms_dispatch 0x7fd138001550 osd_map(745..745 src has 400..745) v4 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.658+0000 7fd12effd700 3 client.38294.objecter handle_osd_map got epochs [745,745] > 0 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.658+0000 7fd12effd700 3 client.38294.objecter handle_osd_map decoding full epoch 745 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.658+0000 7fd13d9cf700 1 --2- 172.21.15.87:0/735771470 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd124062c90 0x7fd124065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.658+0000 7fd12effd700 20 client.38294.objecter dump_active .. 0 homeless 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.659+0000 7fd12effd700 10 client.38294.objecter ms_handle_connect 0x7fd124062c90 2024-02-23T00:42:51.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.659+0000 7fd13d9cf700 1 --2- 172.21.15.87:0/735771470 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd124062c90 0x7fd124065150 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7fd128009810 tx=0x7fd12800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:51.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.662+0000 7fd12effd700 1 -- 172.21.15.87:0/735771470 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd1340510f0 con 0x7fd1380fdbb0 2024-02-23T00:42:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:49.788821+0000 mgr.y (mgr.24437) 631 : cluster [DBG] pgmap v1137: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 947 B/s rd, 0 op/s; 1/744 objects misplaced (0.134%) 2024-02-23T00:42:51.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.831+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd138109780 con 0x7fd124062c90 2024-02-23T00:42:51.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.859+0000 7fd12effd700 1 -- 172.21.15.87:0/735771470 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+527437 (secure 0 0 0) 0x7fd138109780 con 0x7fd124062c90 2024-02-23T00:42:51.870 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:42:51.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 10 librados: watch_flush enter 2024-02-23T00:42:51.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 10 librados: watch_flush exit 2024-02-23T00:42:51.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 20 client.38294.objecter shutdown clearing up homeless session... 2024-02-23T00:42:51.875 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 10 client.38294.objecter successfully canceled tick 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd124062c90 msgr2=0x7fd124065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 --2- 172.21.15.87:0/735771470 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd124062c90 0x7fd124065150 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 rx=0x7fd128009810 tx=0x7fd12800b040).stop 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1380fdbb0 msgr2=0x7fd13810e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 --2- 172.21.15.87:0/735771470 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1380fdbb0 0x7fd13810e4e0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 rx=0x7fd134004870 tx=0x7fd13400b040).stop 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 shutdown_connections 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 --2- 172.21.15.87:0/735771470 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd124062c90 0x7fd124065150 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 --2- 172.21.15.87:0/735771470 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd1380fe510 0x7fd138108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 --2- 172.21.15.87:0/735771470 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd1380f8420 0x7fd13810dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 --2- 172.21.15.87:0/735771470 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd1380fdbb0 0x7fd13810e4e0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.876 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.863+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 >> 172.21.15.87:0/735771470 conn(0x7fd1380f3800 msgr2=0x7fd1380fb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:51.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.864+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 shutdown_connections 2024-02-23T00:42:51.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.864+0000 7fd13fc33700 1 -- 172.21.15.87:0/735771470 wait complete. 2024-02-23T00:42:51.877 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.864+0000 7fd13fc33700 1 librados: shutdown 2024-02-23T00:42:51.877 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:42:51.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.888+0000 7fc2ee4af700 1 -- 172.21.15.87:0/136260266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80f8420 msgr2=0x7fc2e8104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:51.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.888+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/136260266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80f8420 0x7fc2e8104a20 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 rx=0x7fc2e0009be0 tx=0x7fc2e0004d10).stop 2024-02-23T00:42:51.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 -- 172.21.15.87:0/136260266 shutdown_connections 2024-02-23T00:42:51.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/136260266 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80f8420 0x7fc2e8104a20 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/136260266 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2e80fdbb0 0x7fc2e80fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.891 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/136260266 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2e80fe510 0x7fc2e80f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 -- 172.21.15.87:0/136260266 >> 172.21.15.87:0/136260266 conn(0x7fc2e80f3800 msgr2=0x7fc2e80f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:51.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 -- 172.21.15.87:0/136260266 shutdown_connections 2024-02-23T00:42:51.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 -- 172.21.15.87:0/136260266 wait complete. 2024-02-23T00:42:51.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 librados: starting msgr at 2024-02-23T00:42:51.892 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 librados: starting objecter 2024-02-23T00:42:51.893 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:51.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 Processor -- start 2024-02-23T00:42:51.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 -- start start 2024-02-23T00:42:51.894 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 librados: setting wanted keys 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.889+0000 7fc2ee4af700 1 librados: calling monclient init 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2ee4af700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2e80f8420 0x7fc2e810df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2ee4af700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80fdbb0 0x7fc2e810e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2ee4af700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2e80fe510 0x7fc2e81080f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2ee4af700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e81085e0 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2ee4af700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8108750 con 0x7fc2e80fe510 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2ee4af700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8108910 con 0x7fc2e80f8420 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80fdbb0 0x7fc2e810e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80fdbb0 0x7fc2e810e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:52984/0 (socket says 172.21.15.87:52984) 2024-02-23T00:42:51.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e77fe700 1 -- 172.21.15.87:0/902657941 learned_addr learned my addr 172.21.15.87:0/902657941 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2eca4c700 1 --2- 172.21.15.87:0/902657941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2e80fe510 0x7fc2e81080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e77fe700 1 -- 172.21.15.87:0/902657941 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2e80f8420 msgr2=0x7fc2e810df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e77fe700 1 --2- 172.21.15.87:0/902657941 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2e80f8420 0x7fc2e810df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e77fe700 1 -- 172.21.15.87:0/902657941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2e80fe510 msgr2=0x7fc2e81080f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e77fe700 1 --2- 172.21.15.87:0/902657941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2e80fe510 0x7fc2e81080f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e77fe700 1 -- 172.21.15.87:0/902657941 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8005040 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e77fe700 1 --2- 172.21.15.87:0/902657941 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80fdbb0 0x7fc2e810e3f0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7fc2dc00c810 tx=0x7fc2dc004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.890+0000 7fc2e57fa700 10 client.?.objecter ms_handle_connect 0x7fc2e80fdbb0 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.891+0000 7fc2e57fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.891+0000 7fc2e57fa700 1 -- 172.21.15.87:0/902657941 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0043a0 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.891+0000 7fc2e57fa700 1 -- 172.21.15.87:0/902657941 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2dc0109e0 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.891+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e00097c0 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.891+0000 7fc2e57fa700 1 -- 172.21.15.87:0/902657941 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc012ab0 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.891+0000 7fc2ee4af700 10 client.27822.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:51.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.891+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e8108ef0 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.891+0000 7fc2ee4af700 1 librados: init done 2024-02-23T00:42:51.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.892+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e805ee80 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.893+0000 7fc2e57fa700 1 -- 172.21.15.87:0/902657941 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc2dc010510 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.893+0000 7fc2e57fa700 1 --2- 172.21.15.87:0/902657941 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc2d0062c90 0x7fc2d0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:51.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.893+0000 7fc2e57fa700 1 -- 172.21.15.87:0/902657941 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(745..745 src has 400..745) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fc2dc07e020 con 0x7fc2e80fdbb0 2024-02-23T00:42:51.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.893+0000 7fc2e57fa700 10 client.27822.objecter ms_dispatch 0x7fc2e8001550 osd_map(745..745 src has 400..745) v4 2024-02-23T00:42:51.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.893+0000 7fc2e57fa700 3 client.27822.objecter handle_osd_map got epochs [745,745] > 0 2024-02-23T00:42:51.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.893+0000 7fc2e57fa700 3 client.27822.objecter handle_osd_map decoding full epoch 745 2024-02-23T00:42:51.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.893+0000 7fc2e7fff700 1 --2- 172.21.15.87:0/902657941 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc2d0062c90 0x7fc2d0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:51.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.893+0000 7fc2e57fa700 20 client.27822.objecter dump_active .. 0 homeless 2024-02-23T00:42:51.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.894+0000 7fc2e57fa700 10 client.27822.objecter ms_handle_connect 0x7fc2d0062c90 2024-02-23T00:42:51.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.894+0000 7fc2e7fff700 1 --2- 172.21.15.87:0/902657941 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc2d0062c90 0x7fc2d0065150 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7fc2d8009170 tx=0x7fc2d8009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:51.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:51.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:51.897+0000 7fc2e57fa700 1 -- 172.21.15.87:0/902657941 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc2dc051370 con 0x7fc2e80fdbb0 2024-02-23T00:42:52.001 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:52.034 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:52.055 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:52.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:52.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.081+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fc2e8107110 con 0x7fc2e80fdbb0 2024-02-23T00:42:52.159 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:52.188 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:52.389 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:52.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:52.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.411+0000 7fc2e57fa700 1 -- 172.21.15.87:0/902657941 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v746) v1 ==== 91+0+0 (secure 0 0 0) 0x7fc2dc050f60 con 0x7fc2e80fdbb0 2024-02-23T00:42:52.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.414+0000 7fc2ee4af700 10 librados: watch_flush enter 2024-02-23T00:42:52.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.414+0000 7fc2ee4af700 10 librados: watch_flush exit 2024-02-23T00:42:52.416 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.414+0000 7fc2ee4af700 20 client.27822.objecter shutdown clearing up homeless session... 2024-02-23T00:42:52.417 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.414+0000 7fc2ee4af700 10 client.27822.objecter successfully canceled tick 2024-02-23T00:42:52.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.415+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc2d0062c90 msgr2=0x7fc2d0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:52.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.415+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/902657941 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc2d0062c90 0x7fc2d0065150 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 rx=0x7fc2d8009170 tx=0x7fc2d8009cb0).stop 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80fdbb0 msgr2=0x7fc2e810e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/902657941 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80fdbb0 0x7fc2e810e3f0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 rx=0x7fc2dc00c810 tx=0x7fc2dc004330).stop 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 shutdown_connections 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/902657941 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc2d0062c90 0x7fc2d0065150 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/902657941 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc2e80fe510 0x7fc2e81080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/902657941 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc2e80f8420 0x7fc2e810df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 --2- 172.21.15.87:0/902657941 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc2e80fdbb0 0x7fc2e810e3f0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 >> 172.21.15.87:0/902657941 conn(0x7fc2e80f3800 msgr2=0x7fc2e80fbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 shutdown_connections 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 -- 172.21.15.87:0/902657941 wait complete. 2024-02-23T00:42:52.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:52.416+0000 7fc2ee4af700 1 librados: shutdown 2024-02-23T00:42:52.420 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:42:52.442 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:51.256467+0000 mgr.y (mgr.24437) 632 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:52.442 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:52.082733+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? 172.21.15.87:0/902657941' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:42:52.442 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:51.256467+0000 mgr.y (mgr.24437) 632 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:52.443 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:52.082733+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? 172.21.15.87:0/902657941' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:42:52.443 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:42:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:52] "GET /metrics HTTP/1.1" 200 187923 "" "Prometheus/2.33.4" 2024-02-23T00:42:52.496 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:52.528 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:52.550 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:52.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:52.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:52.654 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:52.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:51.256467+0000 mgr.y (mgr.24437) 632 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:42:52.680 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:52.082733+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? 172.21.15.87:0/902657941' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:42:52.683 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:52.702 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:52.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:52.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:52.806 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:42:52.834 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:52.851 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1138,"stamp":"2024-02-23T00:42:51.788974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4047038,"num_objects":250,"num_object_clones":1,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":1532,"num_read_kb":1356,"num_write":582,"num_write_kb":4014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":7883,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":84,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":612,"acting":612,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":613,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2475736,"kb_used_data":55384,"kb_used_omap":85,"kb_used_meta":2420202,"kb_avail":747485480,"statfs":{"total":767960285184,"available":765425131520,"internally_reserved":0,"allocated":56713216,"data_stored":32899559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":2478286976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41044,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":6,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":84,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.401958"},"pg_stats":[{"pgid":"249.1f","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022699+0000","last_change":"2024-02-23T00:42:41.403243+0000","last_active":"2024-02-23T00:42:47.022699+0000","last_peered":"2024-02-23T00:42:47.022699+0000","last_clean":"2024-02-23T00:42:47.022699+0000","last_became_active":"2024-02-23T00:42:41.402967+0000","last_became_peered":"2024-02-23T00:42:41.402967+0000","last_unstale":"2024-02-23T00:42:47.022699+0000","last_undegraded":"2024-02-23T00:42:47.022699+0000","last_fullsized":"2024-02-23T00:42:47.022699+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"249.1e","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391382+0000","last_change":"2024-02-23T00:42:41.402824+0000","last_active":"2024-02-23T00:42:46.391382+0000","last_peered":"2024-02-23T00:42:46.391382+0000","last_clean":"2024-02-23T00:42:46.391382+0000","last_became_active":"2024-02-23T00:42:41.402545+0000","last_became_peered":"2024-02-23T00:42:41.402545+0000","last_unstale":"2024-02-23T00:42:46.391382+0000","last_undegraded":"2024-02-23T00:42:46.391382+0000","last_fullsized":"2024-02-23T00:42:46.391382+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"249.1d","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390375+0000","last_change":"2024-02-23T00:42:41.395444+0000","last_active":"2024-02-23T00:42:46.390375+0000","last_peered":"2024-02-23T00:42:46.390375+0000","last_clean":"2024-02-23T00:42:46.390375+0000","last_became_active":"2024-02-23T00:42:41.395141+0000","last_became_peered":"2024-02-23T00:42:41.395141+0000","last_unstale":"2024-02-23T00:42:46.390375+0000","last_undegraded":"2024-02-23T00:42:46.390375+0000","last_fullsized":"2024-02-23T00:42:46.390375+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.1c","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022579+0000","last_change":"2024-02-23T00:42:41.396631+0000","last_active":"2024-02-23T00:42:47.022579+0000","last_peered":"2024-02-23T00:42:47.022579+0000","last_clean":"2024-02-23T00:42:47.022579+0000","last_became_active":"2024-02-23T00:42:41.396352+0000","last_became_peered":"2024-02-23T00:42:41.396352+0000","last_unstale":"2024-02-23T00:42:47.022579+0000","last_undegraded":"2024-02-23T00:42:47.022579+0000","last_fullsized":"2024-02-23T00:42:47.022579+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"249.1b","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389572+0000","last_change":"2024-02-23T00:42:41.407048+0000","last_active":"2024-02-23T00:42:46.389572+0000","last_peered":"2024-02-23T00:42:46.389572+0000","last_clean":"2024-02-23T00:42:46.389572+0000","last_became_active":"2024-02-23T00:42:41.406699+0000","last_became_peered":"2024-02-23T00:42:41.406699+0000","last_unstale":"2024-02-23T00:42:46.389572+0000","last_undegraded":"2024-02-23T00:42:46.389572+0000","last_fullsized":"2024-02-23T00:42:46.389572+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.1a","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389538+0000","last_change":"2024-02-23T00:42:41.405601+0000","last_active":"2024-02-23T00:42:46.389538+0000","last_peered":"2024-02-23T00:42:46.389538+0000","last_clean":"2024-02-23T00:42:46.389538+0000","last_became_active":"2024-02-23T00:42:41.405274+0000","last_became_peered":"2024-02-23T00:42:41.405274+0000","last_unstale":"2024-02-23T00:42:46.389538+0000","last_undegraded":"2024-02-23T00:42:46.389538+0000","last_fullsized":"2024-02-23T00:42:46.389538+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.19","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.396910+0000","last_change":"2024-02-23T00:42:41.399045+0000","last_active":"2024-02-23T00:42:46.396910+0000","last_peered":"2024-02-23T00:42:46.396910+0000","last_clean":"2024-02-23T00:42:46.396910+0000","last_became_active":"2024-02-23T00:42:41.398750+0000","last_became_peered":"2024-02-23T00:42:41.398750+0000","last_unstale":"2024-02-23T00:42:46.396910+0000","last_undegraded":"2024-02-23T00:42:46.396910+0000","last_fullsized":"2024-02-23T00:42:46.396910+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"249.18","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391334+0000","last_change":"2024-02-23T00:42:41.404729+0000","last_active":"2024-02-23T00:42:46.391334+0000","last_peered":"2024-02-23T00:42:46.391334+0000","last_clean":"2024-02-23T00:42:46.391334+0000","last_became_active":"2024-02-23T00:42:41.404281+0000","last_became_peered":"2024-02-23T00:42:41.404281+0000","last_unstale":"2024-02-23T00:42:46.391334+0000","last_undegraded":"2024-02-23T00:42:46.391334+0000","last_fullsized":"2024-02-23T00:42:46.391334+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"249.f","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643785+0000","last_change":"2024-02-23T00:42:41.401042+0000","last_active":"2024-02-23T00:42:46.643785+0000","last_peered":"2024-02-23T00:42:46.643785+0000","last_clean":"2024-02-23T00:42:46.643785+0000","last_became_active":"2024-02-23T00:42:41.400593+0000","last_became_peered":"2024-02-23T00:42:41.400593+0000","last_unstale":"2024-02-23T00:42:46.643785+0000","last_undegraded":"2024-02-23T00:42:46.643785+0000","last_fullsized":"2024-02-23T00:42:46.643785+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"249.e","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643257+0000","last_change":"2024-02-23T00:42:41.399898+0000","last_active":"2024-02-23T00:42:46.643257+0000","last_peered":"2024-02-23T00:42:46.643257+0000","last_clean":"2024-02-23T00:42:46.643257+0000","last_became_active":"2024-02-23T00:42:41.399485+0000","last_became_peered":"2024-02-23T00:42:41.399485+0000","last_unstale":"2024-02-23T00:42:46.643257+0000","last_undegraded":"2024-02-23T00:42:46.643257+0000","last_fullsized":"2024-02-23T00:42:46.643257+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"249.d","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390627+0000","last_change":"2024-02-23T00:42:41.409116+0000","last_active":"2024-02-23T00:42:46.390627+0000","last_peered":"2024-02-23T00:42:46.390627+0000","last_clean":"2024-02-23T00:42:46.390627+0000","last_became_active":"2024-02-23T00:42:41.408839+0000","last_became_peered":"2024-02-23T00:42:41.408839+0000","last_unstale":"2024-02-23T00:42:46.390627+0000","last_undegraded":"2024-02-23T00:42:46.390627+0000","last_fullsized":"2024-02-23T00:42:46.390627+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"249.c","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642857+0000","last_change":"2024-02-23T00:42:41.393284+0000","last_active":"2024-02-23T00:42:46.642857+0000","last_peered":"2024-02-23T00:42:46.642857+0000","last_clean":"2024-02-23T00:42:46.642857+0000","last_became_active":"2024-02-23T00:42:41.393001+0000","last_became_peered":"2024-02-23T00:42:41.393001+0000","last_unstale":"2024-02-23T00:42:46.642857+0000","last_undegraded":"2024-02-23T00:42:46.642857+0000","last_fullsized":"2024-02-23T00:42:46.642857+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"249.b","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391867+0000","last_change":"2024-02-23T00:42:41.405980+0000","last_active":"2024-02-23T00:42:46.391867+0000","last_peered":"2024-02-23T00:42:46.391867+0000","last_clean":"2024-02-23T00:42:46.391867+0000","last_became_active":"2024-02-23T00:42:41.405585+0000","last_became_peered":"2024-02-23T00:42:41.405585+0000","last_unstale":"2024-02-23T00:42:46.391867+0000","last_undegraded":"2024-02-23T00:42:46.391867+0000","last_fullsized":"2024-02-23T00:42:46.391867+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.a","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390470+0000","last_change":"2024-02-23T00:42:41.405664+0000","last_active":"2024-02-23T00:42:46.390470+0000","last_peered":"2024-02-23T00:42:46.390470+0000","last_clean":"2024-02-23T00:42:46.390470+0000","last_became_active":"2024-02-23T00:42:41.405393+0000","last_became_peered":"2024-02-23T00:42:41.405393+0000","last_unstale":"2024-02-23T00:42:46.390470+0000","last_undegraded":"2024-02-23T00:42:46.390470+0000","last_fullsized":"2024-02-23T00:42:46.390470+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.9","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642118+0000","last_change":"2024-02-23T00:42:41.397923+0000","last_active":"2024-02-23T00:42:46.642118+0000","last_peered":"2024-02-23T00:42:46.642118+0000","last_clean":"2024-02-23T00:42:46.642118+0000","last_became_active":"2024-02-23T00:42:41.397572+0000","last_became_peered":"2024-02-23T00:42:41.397572+0000","last_unstale":"2024-02-23T00:42:46.642118+0000","last_undegraded":"2024-02-23T00:42:46.642118+0000","last_fullsized":"2024-02-23T00:42:46.642118+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"249.8","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389101+0000","last_change":"2024-02-23T00:42:41.401956+0000","last_active":"2024-02-23T00:42:46.389101+0000","last_peered":"2024-02-23T00:42:46.389101+0000","last_clean":"2024-02-23T00:42:46.389101+0000","last_became_active":"2024-02-23T00:42:41.401660+0000","last_became_peered":"2024-02-23T00:42:41.401660+0000","last_unstale":"2024-02-23T00:42:46.389101+0000","last_undegraded":"2024-02-23T00:42:46.389101+0000","last_fullsized":"2024-02-23T00:42:46.389101+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.7","version":"0'0","reported_seq":13,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.394877+0000","last_change":"2024-02-23T00:42:41.401271+0000","last_active":"2024-02-23T00:42:46.394877+0000","last_peered":"2024-02-23T00:42:46.394877+0000","last_clean":"2024-02-23T00:42:46.394877+0000","last_became_active":"2024-02-23T00:42:41.400986+0000","last_became_peered":"2024-02-23T00:42:41.400986+0000","last_unstale":"2024-02-23T00:42:46.394877+0000","last_undegraded":"2024-02-23T00:42:46.394877+0000","last_fullsized":"2024-02-23T00:42:46.394877+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"249.6","version":"745'6","reported_seq":19,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-02-23T00:42:49.406507+0000","last_change":"2024-02-23T00:42:41.401797+0000","last_active":"2024-02-23T00:42:49.406507+0000","last_peered":"2024-02-23T00:42:49.406507+0000","last_clean":"2024-02-23T00:42:49.406507+0000","last_became_active":"2024-02-23T00:42:41.400969+0000","last_became_peered":"2024-02-23T00:42:41.400969+0000","last_unstale":"2024-02-23T00:42:49.406507+0000","last_undegraded":"2024-02-23T00:42:49.406507+0000","last_fullsized":"2024-02-23T00:42:49.406507+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41044,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":84,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"249.5","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390845+0000","last_change":"2024-02-23T00:42:41.405005+0000","last_active":"2024-02-23T00:42:46.390845+0000","last_peered":"2024-02-23T00:42:46.390845+0000","last_clean":"2024-02-23T00:42:46.390845+0000","last_became_active":"2024-02-23T00:42:41.404722+0000","last_became_peered":"2024-02-23T00:42:41.404722+0000","last_unstale":"2024-02-23T00:42:46.390845+0000","last_undegraded":"2024-02-23T00:42:46.390845+0000","last_fullsized":"2024-02-23T00:42:46.390845+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"249.4","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643060+0000","last_change":"2024-02-23T00:42:41.403374+0000","last_active":"2024-02-23T00:42:46.643060+0000","last_peered":"2024-02-23T00:42:46.643060+0000","last_clean":"2024-02-23T00:42:46.643060+0000","last_became_active":"2024-02-23T00:42:41.403100+0000","last_became_peered":"2024-02-23T00:42:41.403100+0000","last_unstale":"2024-02-23T00:42:46.643060+0000","last_undegraded":"2024-02-23T00:42:46.643060+0000","last_fullsized":"2024-02-23T00:42:46.643060+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"249.3","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391556+0000","last_change":"2024-02-23T00:42:41.405798+0000","last_active":"2024-02-23T00:42:46.391556+0000","last_peered":"2024-02-23T00:42:46.391556+0000","last_clean":"2024-02-23T00:42:46.391556+0000","last_became_active":"2024-02-23T00:42:41.405308+0000","last_became_peered":"2024-02-23T00:42:41.405308+0000","last_unstale":"2024-02-23T00:42:46.391556+0000","last_undegraded":"2024-02-23T00:42:46.391556+0000","last_fullsized":"2024-02-23T00:42:46.391556+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.2","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390911+0000","last_change":"2024-02-23T00:42:41.405955+0000","last_active":"2024-02-23T00:42:46.390911+0000","last_peered":"2024-02-23T00:42:46.390911+0000","last_clean":"2024-02-23T00:42:46.390911+0000","last_became_active":"2024-02-23T00:42:41.405652+0000","last_became_peered":"2024-02-23T00:42:41.405652+0000","last_unstale":"2024-02-23T00:42:46.390911+0000","last_undegraded":"2024-02-23T00:42:46.390911+0000","last_fullsized":"2024-02-23T00:42:46.390911+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"249.1","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389160+0000","last_change":"2024-02-23T00:42:41.402704+0000","last_active":"2024-02-23T00:42:46.389160+0000","last_peered":"2024-02-23T00:42:46.389160+0000","last_clean":"2024-02-23T00:42:46.389160+0000","last_became_active":"2024-02-23T00:42:41.402432+0000","last_became_peered":"2024-02-23T00:42:41.402432+0000","last_unstale":"2024-02-23T00:42:46.389160+0000","last_undegraded":"2024-02-23T00:42:46.389160+0000","last_fullsized":"2024-02-23T00:42:46.389160+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"249.0","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391674+0000","last_change":"2024-02-23T00:42:41.402936+0000","last_active":"2024-02-23T00:42:46.391674+0000","last_peered":"2024-02-23T00:42:46.391674+0000","last_clean":"2024-02-23T00:42:46.391674+0000","last_became_active":"2024-02-23T00:42:41.402623+0000","last_became_peered":"2024-02-23T00:42:41.402623+0000","last_unstale":"2024-02-23T00:42:46.391674+0000","last_undegraded":"2024-02-23T00:42:46.391674+0000","last_fullsized":"2024-02-23T00:42:46.391674+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"249.14","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391089+0000","last_change":"2024-02-23T00:42:41.405022+0000","last_active":"2024-02-23T00:42:46.391089+0000","last_peered":"2024-02-23T00:42:46.391089+0000","last_clean":"2024-02-23T00:42:46.391089+0000","last_became_active":"2024-02-23T00:42:41.404700+0000","last_became_peered":"2024-02-23T00:42:41.404700+0000","last_unstale":"2024-02-23T00:42:46.391089+0000","last_undegraded":"2024-02-23T00:42:46.391089+0000","last_fullsized":"2024-02-23T00:42:46.391089+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.7","version":"689'9","reported_seq":194,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643390+0000","last_change":"2024-02-23T00:42:21.267470+0000","last_active":"2024-02-23T00:42:46.643390+0000","last_peered":"2024-02-23T00:42:46.643390+0000","last_clean":"2024-02-23T00:42:46.643390+0000","last_became_active":"2024-02-23T00:40:17.211405+0000","last_became_peered":"2024-02-23T00:40:17.211405+0000","last_unstale":"2024-02-23T00:42:46.643390+0000","last_undegraded":"2024-02-23T00:42:46.643390+0000","last_fullsized":"2024-02-23T00:42:46.643390+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'9","last_scrub_stamp":"2024-02-23T00:42:21.267354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:21.267354+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"249.15","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390971+0000","last_change":"2024-02-23T00:42:41.398842+0000","last_active":"2024-02-23T00:42:46.390971+0000","last_peered":"2024-02-23T00:42:46.390971+0000","last_clean":"2024-02-23T00:42:46.390971+0000","last_became_active":"2024-02-23T00:42:41.398329+0000","last_became_peered":"2024-02-23T00:42:41.398329+0000","last_unstale":"2024-02-23T00:42:46.390971+0000","last_undegraded":"2024-02-23T00:42:46.390971+0000","last_fullsized":"2024-02-23T00:42:46.390971+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.6","version":"710'16","reported_seq":202,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391102+0000","last_change":"2024-02-23T00:42:14.256795+0000","last_active":"2024-02-23T00:42:46.391102+0000","last_peered":"2024-02-23T00:42:46.391102+0000","last_clean":"2024-02-23T00:42:46.391102+0000","last_became_active":"2024-02-23T00:40:17.215263+0000","last_became_peered":"2024-02-23T00:40:17.215263+0000","last_unstale":"2024-02-23T00:42:46.391102+0000","last_undegraded":"2024-02-23T00:42:46.391102+0000","last_fullsized":"2024-02-23T00:42:46.391102+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'8","last_scrub_stamp":"2024-02-23T00:41:36.605800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:41:36.605800+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.16","version":"0'0","reported_seq":13,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.394727+0000","last_change":"2024-02-23T00:42:41.401575+0000","last_active":"2024-02-23T00:42:46.394727+0000","last_peered":"2024-02-23T00:42:46.394727+0000","last_clean":"2024-02-23T00:42:46.394727+0000","last_became_active":"2024-02-23T00:42:41.401205+0000","last_became_peered":"2024-02-23T00:42:41.401205+0000","last_unstale":"2024-02-23T00:42:46.394727+0000","last_undegraded":"2024-02-23T00:42:46.394727+0000","last_fullsized":"2024-02-23T00:42:46.394727+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.5","version":"635'5","reported_seq":207,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391368+0000","last_change":"2024-02-23T00:42:13.431865+0000","last_active":"2024-02-23T00:42:46.391368+0000","last_peered":"2024-02-23T00:42:46.391368+0000","last_clean":"2024-02-23T00:42:46.391368+0000","last_became_active":"2024-02-23T00:41:37.238894+0000","last_became_peered":"2024-02-23T00:41:37.238894+0000","last_unstale":"2024-02-23T00:42:46.391368+0000","last_undegraded":"2024-02-23T00:42:46.391368+0000","last_fullsized":"2024-02-23T00:42:46.391368+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"635'5","last_scrub_stamp":"2024-02-23T00:42:13.431791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.431791+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"249.17","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022886+0000","last_change":"2024-02-23T00:42:41.402663+0000","last_active":"2024-02-23T00:42:47.022886+0000","last_peered":"2024-02-23T00:42:47.022886+0000","last_clean":"2024-02-23T00:42:47.022886+0000","last_became_active":"2024-02-23T00:42:41.401914+0000","last_became_peered":"2024-02-23T00:42:41.401914+0000","last_unstale":"2024-02-23T00:42:47.022886+0000","last_undegraded":"2024-02-23T00:42:47.022886+0000","last_fullsized":"2024-02-23T00:42:47.022886+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.4","version":"726'21","reported_seq":201,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022042+0000","last_change":"2024-02-23T00:42:12.255652+0000","last_active":"2024-02-23T00:42:47.022042+0000","last_peered":"2024-02-23T00:42:47.022042+0000","last_clean":"2024-02-23T00:42:47.022042+0000","last_became_active":"2024-02-23T00:40:17.215235+0000","last_became_peered":"2024-02-23T00:40:17.215235+0000","last_unstale":"2024-02-23T00:42:47.022042+0000","last_undegraded":"2024-02-23T00:42:47.022042+0000","last_fullsized":"2024-02-23T00:42:47.022042+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000031,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":3918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"249.10","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.392513+0000","last_change":"2024-02-23T00:42:41.404556+0000","last_active":"2024-02-23T00:42:46.392513+0000","last_peered":"2024-02-23T00:42:46.392513+0000","last_clean":"2024-02-23T00:42:46.392513+0000","last_became_active":"2024-02-23T00:42:41.403979+0000","last_became_peered":"2024-02-23T00:42:41.403979+0000","last_unstale":"2024-02-23T00:42:46.392513+0000","last_undegraded":"2024-02-23T00:42:46.392513+0000","last_fullsized":"2024-02-23T00:42:46.392513+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.3","version":"697'2","reported_seq":183,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022309+0000","last_change":"2024-02-23T00:42:18.042228+0000","last_active":"2024-02-23T00:42:47.022309+0000","last_peered":"2024-02-23T00:42:47.022309+0000","last_clean":"2024-02-23T00:42:47.022309+0000","last_became_active":"2024-02-23T00:40:17.202244+0000","last_became_peered":"2024-02-23T00:40:17.202244+0000","last_unstale":"2024-02-23T00:42:47.022309+0000","last_undegraded":"2024-02-23T00:42:47.022309+0000","last_fullsized":"2024-02-23T00:42:47.022309+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'2","last_scrub_stamp":"2024-02-23T00:42:18.042156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.042156+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"249.11","version":"0'0","reported_seq":14,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390434+0000","last_change":"2024-02-23T00:42:41.404883+0000","last_active":"2024-02-23T00:42:46.390434+0000","last_peered":"2024-02-23T00:42:46.390434+0000","last_clean":"2024-02-23T00:42:46.390434+0000","last_became_active":"2024-02-23T00:42:41.404611+0000","last_became_peered":"2024-02-23T00:42:41.404611+0000","last_unstale":"2024-02-23T00:42:46.390434+0000","last_undegraded":"2024-02-23T00:42:46.390434+0000","last_fullsized":"2024-02-23T00:42:46.390434+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.2","version":"679'6","reported_seq":208,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390449+0000","last_change":"2024-02-23T00:42:12.249043+0000","last_active":"2024-02-23T00:42:46.390449+0000","last_peered":"2024-02-23T00:42:46.390449+0000","last_clean":"2024-02-23T00:42:46.390449+0000","last_became_active":"2024-02-23T00:41:37.219553+0000","last_became_peered":"2024-02-23T00:41:37.219553+0000","last_unstale":"2024-02-23T00:42:46.390449+0000","last_undegraded":"2024-02-23T00:42:46.390449+0000","last_fullsized":"2024-02-23T00:42:46.390449+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"679'6","last_scrub_stamp":"2024-02-23T00:41:43.619119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:41:43.619119+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"249.12","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390711+0000","last_change":"2024-02-23T00:42:41.406075+0000","last_active":"2024-02-23T00:42:46.390711+0000","last_peered":"2024-02-23T00:42:46.390711+0000","last_clean":"2024-02-23T00:42:46.390711+0000","last_became_active":"2024-02-23T00:42:41.405764+0000","last_became_peered":"2024-02-23T00:42:41.405764+0000","last_unstale":"2024-02-23T00:42:46.390711+0000","last_undegraded":"2024-02-23T00:42:46.390711+0000","last_fullsized":"2024-02-23T00:42:46.390711+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.1","version":"607'1","reported_seq":197,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.021796+0000","last_change":"2024-02-23T00:42:12.251737+0000","last_active":"2024-02-23T00:42:47.021796+0000","last_peered":"2024-02-23T00:42:47.021796+0000","last_clean":"2024-02-23T00:42:47.021796+0000","last_became_active":"2024-02-23T00:41:37.231330+0000","last_became_peered":"2024-02-23T00:41:37.231330+0000","last_unstale":"2024-02-23T00:42:47.021796+0000","last_undegraded":"2024-02-23T00:42:47.021796+0000","last_fullsized":"2024-02-23T00:42:47.021796+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"249.13","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642508+0000","last_change":"2024-02-23T00:42:41.402245+0000","last_active":"2024-02-23T00:42:46.642508+0000","last_peered":"2024-02-23T00:42:46.642508+0000","last_clean":"2024-02-23T00:42:46.642508+0000","last_became_active":"2024-02-23T00:42:41.401523+0000","last_became_peered":"2024-02-23T00:42:41.401523+0000","last_unstale":"2024-02-23T00:42:46.642508+0000","last_undegraded":"2024-02-23T00:42:46.642508+0000","last_fullsized":"2024-02-23T00:42:46.642508+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":738,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:40.372155+0000","last_clean_scrub_stamp":"2024-02-23T00:42:40.372155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.0","version":"671'10","reported_seq":179,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390213+0000","last_change":"2024-02-23T00:42:12.248572+0000","last_active":"2024-02-23T00:42:46.390213+0000","last_peered":"2024-02-23T00:42:46.390213+0000","last_clean":"2024-02-23T00:42:46.390213+0000","last_became_active":"2024-02-23T00:41:50.253705+0000","last_became_peered":"2024-02-23T00:41:50.253705+0000","last_unstale":"2024-02-23T00:42:46.390213+0000","last_undegraded":"2024-02-23T00:42:46.390213+0000","last_fullsized":"2024-02-23T00:42:46.390213+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"671'10","last_scrub_stamp":"2024-02-23T00:41:50.664981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:41:50.664981+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":321,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643377+0000","last_change":"2024-02-23T00:42:41.400879+0000","last_active":"2024-02-23T00:42:46.643377+0000","last_peered":"2024-02-23T00:42:46.643377+0000","last_clean":"2024-02-23T00:42:46.643377+0000","last_became_active":"2024-02-23T00:42:41.400209+0000","last_became_peered":"2024-02-23T00:42:41.400209+0000","last_unstale":"2024-02-23T00:42:46.643377+0000","last_undegraded":"2024-02-23T00:42:46.643377+0000","last_fullsized":"2024-02-23T00:42:46.643377+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.355553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.355553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"571'11","reported_seq":948,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.381662+0000","last_change":"2024-02-23T00:42:18.360637+0000","last_active":"2024-02-23T00:42:47.381662+0000","last_peered":"2024-02-23T00:42:47.381662+0000","last_clean":"2024-02-23T00:42:47.381662+0000","last_became_active":"2024-02-23T00:39:49.195036+0000","last_became_peered":"2024-02-23T00:39:49.195036+0000","last_unstale":"2024-02-23T00:42:47.381662+0000","last_undegraded":"2024-02-23T00:42:47.381662+0000","last_fullsized":"2024-02-23T00:42:47.381662+0000","mapping_epoch":58,"log_start":"379'9","ondisk_log_start":"379'9","created":375,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'11","last_scrub_stamp":"2024-02-23T00:42:18.360544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.360544+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":836,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.021814+0000","last_change":"2024-02-23T00:42:16.067711+0000","last_active":"2024-02-23T00:42:47.021814+0000","last_peered":"2024-02-23T00:42:47.021814+0000","last_clean":"2024-02-23T00:42:47.021814+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:42:47.021814+0000","last_undegraded":"2024-02-23T00:42:47.021814+0000","last_fullsized":"2024-02-23T00:42:47.021814+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:42:16.067590+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.067590+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":762,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391757+0000","last_change":"2024-02-23T00:41:52.314002+0000","last_active":"2024-02-23T00:42:46.391757+0000","last_peered":"2024-02-23T00:42:46.391757+0000","last_clean":"2024-02-23T00:42:46.391757+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:42:46.391757+0000","last_undegraded":"2024-02-23T00:42:46.391757+0000","last_fullsized":"2024-02-23T00:42:46.391757+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:52.313932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:41:52.313932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":745,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022022+0000","last_change":"2024-02-23T00:41:37.222406+0000","last_active":"2024-02-23T00:42:47.022022+0000","last_peered":"2024-02-23T00:42:47.022022+0000","last_clean":"2024-02-23T00:42:47.022022+0000","last_became_active":"2024-02-23T00:41:37.222092+0000","last_became_peered":"2024-02-23T00:41:37.222092+0000","last_unstale":"2024-02-23T00:42:47.022022+0000","last_undegraded":"2024-02-23T00:42:47.022022+0000","last_fullsized":"2024-02-23T00:42:47.022022+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:49.854635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:40:49.854635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":692,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397631+0000","last_change":"2024-02-23T00:41:37.214599+0000","last_active":"2024-02-23T00:42:46.397631+0000","last_peered":"2024-02-23T00:42:46.397631+0000","last_clean":"2024-02-23T00:42:46.397631+0000","last_became_active":"2024-02-23T00:41:37.214214+0000","last_became_peered":"2024-02-23T00:41:37.214214+0000","last_unstale":"2024-02-23T00:42:46.397631+0000","last_undegraded":"2024-02-23T00:42:46.397631+0000","last_fullsized":"2024-02-23T00:42:46.397631+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:45.172542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.172542+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":1341,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-02-23T00:42:48.544914+0000","last_change":"2024-02-23T00:41:39.898258+0000","last_active":"2024-02-23T00:42:48.544914+0000","last_peered":"2024-02-23T00:42:48.544914+0000","last_clean":"2024-02-23T00:42:48.544914+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:42:48.544914+0000","last_undegraded":"2024-02-23T00:42:48.544914+0000","last_fullsized":"2024-02-23T00:42:48.544914+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:41:39.898173+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:41:39.898173+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":610,"num_read_kb":610,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":453,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390124+0000","last_change":"2024-02-23T00:42:15.735663+0000","last_active":"2024-02-23T00:42:46.390124+0000","last_peered":"2024-02-23T00:42:46.390124+0000","last_clean":"2024-02-23T00:42:46.390124+0000","last_became_active":"2024-02-23T00:41:41.226195+0000","last_became_peered":"2024-02-23T00:41:41.226195+0000","last_unstale":"2024-02-23T00:42:46.390124+0000","last_undegraded":"2024-02-23T00:42:46.390124+0000","last_fullsized":"2024-02-23T00:42:46.390124+0000","mapping_epoch":681,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:42:15.735606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.735606+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":740,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.398242+0000","last_change":"2024-02-23T00:42:29.119245+0000","last_active":"2024-02-23T00:42:46.398242+0000","last_peered":"2024-02-23T00:42:46.398242+0000","last_clean":"2024-02-23T00:42:46.398242+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:42:46.398242+0000","last_undegraded":"2024-02-23T00:42:46.398242+0000","last_fullsized":"2024-02-23T00:42:46.398242+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:29.119169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:29.119169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":749,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391053+0000","last_change":"2024-02-23T00:41:36.045647+0000","last_active":"2024-02-23T00:42:46.391053+0000","last_peered":"2024-02-23T00:42:46.391053+0000","last_clean":"2024-02-23T00:42:46.391053+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:42:46.391053+0000","last_undegraded":"2024-02-23T00:42:46.391053+0000","last_fullsized":"2024-02-23T00:42:46.391053+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:36.045582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:41:36.045582+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":81,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389352+0000","last_change":"2024-02-23T00:41:50.712474+0000","last_active":"2024-02-23T00:42:46.389352+0000","last_peered":"2024-02-23T00:42:46.389352+0000","last_clean":"2024-02-23T00:42:46.389352+0000","last_became_active":"2024-02-23T00:41:37.228937+0000","last_became_peered":"2024-02-23T00:41:37.228937+0000","last_unstale":"2024-02-23T00:42:46.389352+0000","last_undegraded":"2024-02-23T00:42:46.389352+0000","last_fullsized":"2024-02-23T00:42:46.389352+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:50.712410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:41:50.712410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":674,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391257+0000","last_change":"2024-02-23T00:40:57.491379+0000","last_active":"2024-02-23T00:42:46.391257+0000","last_peered":"2024-02-23T00:42:46.391257+0000","last_clean":"2024-02-23T00:42:46.391257+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:42:46.391257+0000","last_undegraded":"2024-02-23T00:42:46.391257+0000","last_fullsized":"2024-02-23T00:42:46.391257+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:57.491323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:40:57.491323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":16,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642960+0000","last_change":"2024-02-23T00:42:41.393838+0000","last_active":"2024-02-23T00:42:46.642960+0000","last_peered":"2024-02-23T00:42:46.642960+0000","last_clean":"2024-02-23T00:42:46.642960+0000","last_became_active":"2024-02-23T00:42:41.393491+0000","last_became_peered":"2024-02-23T00:42:41.393491+0000","last_unstale":"2024-02-23T00:42:46.642960+0000","last_undegraded":"2024-02-23T00:42:46.642960+0000","last_fullsized":"2024-02-23T00:42:46.642960+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.024705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":853,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391316+0000","last_change":"2024-02-23T00:40:47.429480+0000","last_active":"2024-02-23T00:42:46.391316+0000","last_peered":"2024-02-23T00:42:46.391316+0000","last_clean":"2024-02-23T00:42:46.391316+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:42:46.391316+0000","last_undegraded":"2024-02-23T00:42:46.391316+0000","last_fullsized":"2024-02-23T00:42:46.391316+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:40:47.429409+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:40:47.429409+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":760,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390860+0000","last_change":"2024-02-23T00:42:24.447799+0000","last_active":"2024-02-23T00:42:46.390860+0000","last_peered":"2024-02-23T00:42:46.390860+0000","last_clean":"2024-02-23T00:42:46.390860+0000","last_became_active":"2024-02-23T00:41:37.238728+0000","last_became_peered":"2024-02-23T00:41:37.238728+0000","last_unstale":"2024-02-23T00:42:46.390860+0000","last_undegraded":"2024-02-23T00:42:46.390860+0000","last_fullsized":"2024-02-23T00:42:46.390860+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.447731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.447731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":749,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391126+0000","last_change":"2024-02-23T00:42:17.473540+0000","last_active":"2024-02-23T00:42:46.391126+0000","last_peered":"2024-02-23T00:42:46.391126+0000","last_clean":"2024-02-23T00:42:46.391126+0000","last_became_active":"2024-02-23T00:41:37.239529+0000","last_became_peered":"2024-02-23T00:41:37.239529+0000","last_unstale":"2024-02-23T00:42:46.391126+0000","last_undegraded":"2024-02-23T00:42:46.391126+0000","last_fullsized":"2024-02-23T00:42:46.391126+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:17.473474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.473474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":756,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.641994+0000","last_change":"2024-02-23T00:41:49.588387+0000","last_active":"2024-02-23T00:42:46.641994+0000","last_peered":"2024-02-23T00:42:46.641994+0000","last_clean":"2024-02-23T00:42:46.641994+0000","last_became_active":"2024-02-23T00:40:37.197010+0000","last_became_peered":"2024-02-23T00:40:37.197010+0000","last_unstale":"2024-02-23T00:42:46.641994+0000","last_undegraded":"2024-02-23T00:42:46.641994+0000","last_fullsized":"2024-02-23T00:42:46.641994+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:49.588290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.588290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":76,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389319+0000","last_change":"2024-02-23T00:41:37.248755+0000","last_active":"2024-02-23T00:42:46.389319+0000","last_peered":"2024-02-23T00:42:46.389319+0000","last_clean":"2024-02-23T00:42:46.389319+0000","last_became_active":"2024-02-23T00:41:37.248272+0000","last_became_peered":"2024-02-23T00:41:37.248272+0000","last_unstale":"2024-02-23T00:42:46.389319+0000","last_undegraded":"2024-02-23T00:42:46.389319+0000","last_fullsized":"2024-02-23T00:42:46.389319+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:59.125249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:59.125249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":738,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391464+0000","last_change":"2024-02-23T00:40:50.421297+0000","last_active":"2024-02-23T00:42:46.391464+0000","last_peered":"2024-02-23T00:42:46.391464+0000","last_clean":"2024-02-23T00:42:46.391464+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:42:46.391464+0000","last_undegraded":"2024-02-23T00:42:46.391464+0000","last_fullsized":"2024-02-23T00:42:46.391464+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:50.421224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.421224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":571,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.392467+0000","last_change":"2024-02-23T00:42:19.485117+0000","last_active":"2024-02-23T00:42:46.392467+0000","last_peered":"2024-02-23T00:42:46.392467+0000","last_clean":"2024-02-23T00:42:46.392467+0000","last_became_active":"2024-02-23T00:41:50.256494+0000","last_became_peered":"2024-02-23T00:41:50.256494+0000","last_unstale":"2024-02-23T00:42:46.392467+0000","last_undegraded":"2024-02-23T00:42:46.392467+0000","last_fullsized":"2024-02-23T00:42:46.392467+0000","mapping_epoch":688,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:42:19.485046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.485046+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":726,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642488+0000","last_change":"2024-02-23T00:41:47.545036+0000","last_active":"2024-02-23T00:42:46.642488+0000","last_peered":"2024-02-23T00:42:46.642488+0000","last_clean":"2024-02-23T00:42:46.642488+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:42:46.642488+0000","last_undegraded":"2024-02-23T00:42:46.642488+0000","last_fullsized":"2024-02-23T00:42:46.642488+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.544952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.544952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":732,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397553+0000","last_change":"2024-02-23T00:41:37.229878+0000","last_active":"2024-02-23T00:42:46.397553+0000","last_peered":"2024-02-23T00:42:46.397553+0000","last_clean":"2024-02-23T00:42:46.397553+0000","last_became_active":"2024-02-23T00:41:37.229416+0000","last_became_peered":"2024-02-23T00:41:37.229416+0000","last_unstale":"2024-02-23T00:42:46.397553+0000","last_undegraded":"2024-02-23T00:42:46.397553+0000","last_fullsized":"2024-02-23T00:42:46.397553+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:00.151031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:00.151031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":700,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643699+0000","last_change":"2024-02-23T00:41:55.514102+0000","last_active":"2024-02-23T00:42:46.643699+0000","last_peered":"2024-02-23T00:42:46.643699+0000","last_clean":"2024-02-23T00:42:46.643699+0000","last_became_active":"2024-02-23T00:38:49.194450+0000","last_became_peered":"2024-02-23T00:38:49.194450+0000","last_unstale":"2024-02-23T00:42:46.643699+0000","last_undegraded":"2024-02-23T00:42:46.643699+0000","last_fullsized":"2024-02-23T00:42:46.643699+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.514031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.514031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":262,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022611+0000","last_change":"2024-02-23T00:41:37.253004+0000","last_active":"2024-02-23T00:42:47.022611+0000","last_peered":"2024-02-23T00:42:47.022611+0000","last_clean":"2024-02-23T00:42:47.022611+0000","last_became_active":"2024-02-23T00:41:37.252083+0000","last_became_peered":"2024-02-23T00:41:37.252083+0000","last_unstale":"2024-02-23T00:42:47.022611+0000","last_undegraded":"2024-02-23T00:42:47.022611+0000","last_fullsized":"2024-02-23T00:42:47.022611+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":671,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022139+0000","last_change":"2024-02-23T00:41:50.256382+0000","last_active":"2024-02-23T00:42:47.022139+0000","last_peered":"2024-02-23T00:42:47.022139+0000","last_clean":"2024-02-23T00:42:47.022139+0000","last_became_active":"2024-02-23T00:41:50.256004+0000","last_became_peered":"2024-02-23T00:41:50.256004+0000","last_unstale":"2024-02-23T00:42:47.022139+0000","last_undegraded":"2024-02-23T00:42:47.022139+0000","last_fullsized":"2024-02-23T00:42:47.022139+0000","mapping_epoch":688,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:40:51.632948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:51.632948+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":753,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397272+0000","last_change":"2024-02-23T00:42:15.223520+0000","last_active":"2024-02-23T00:42:46.397272+0000","last_peered":"2024-02-23T00:42:46.397272+0000","last_clean":"2024-02-23T00:42:46.397272+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:42:46.397272+0000","last_undegraded":"2024-02-23T00:42:46.397272+0000","last_fullsized":"2024-02-23T00:42:46.397272+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.223455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.223455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":925,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.230610+0000","last_change":"2024-02-23T00:41:33.047241+0000","last_active":"2024-02-23T00:42:47.230610+0000","last_peered":"2024-02-23T00:42:47.230610+0000","last_clean":"2024-02-23T00:42:47.230610+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:42:47.230610+0000","last_undegraded":"2024-02-23T00:42:47.230610+0000","last_fullsized":"2024-02-23T00:42:47.230610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:41:33.047155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.047155+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":672,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.395222+0000","last_change":"2024-02-23T00:41:34.719462+0000","last_active":"2024-02-23T00:42:46.395222+0000","last_peered":"2024-02-23T00:42:46.395222+0000","last_clean":"2024-02-23T00:42:46.395222+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:42:46.395222+0000","last_undegraded":"2024-02-23T00:42:46.395222+0000","last_fullsized":"2024-02-23T00:42:46.395222+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:34.719388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.719388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":248,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022020+0000","last_change":"2024-02-23T00:40:15.208236+0000","last_active":"2024-02-23T00:42:47.022020+0000","last_peered":"2024-02-23T00:42:47.022020+0000","last_clean":"2024-02-23T00:42:47.022020+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:42:47.022020+0000","last_undegraded":"2024-02-23T00:42:47.022020+0000","last_fullsized":"2024-02-23T00:42:47.022020+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:15.208124+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.208124+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":907,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.398645+0000","last_change":"2024-02-23T00:41:08.116872+0000","last_active":"2024-02-23T00:42:46.398645+0000","last_peered":"2024-02-23T00:42:46.398645+0000","last_clean":"2024-02-23T00:42:46.398645+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:42:46.398645+0000","last_undegraded":"2024-02-23T00:42:46.398645+0000","last_fullsized":"2024-02-23T00:42:46.398645+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:41:08.116765+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:41:08.116765+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":737,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397137+0000","last_change":"2024-02-23T00:42:14.253417+0000","last_active":"2024-02-23T00:42:46.397137+0000","last_peered":"2024-02-23T00:42:46.397137+0000","last_clean":"2024-02-23T00:42:46.397137+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:42:46.397137+0000","last_undegraded":"2024-02-23T00:42:46.397137+0000","last_fullsized":"2024-02-23T00:42:46.397137+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.253319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.253319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"689'12","reported_seq":935,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390432+0000","last_change":"2024-02-23T00:42:19.683007+0000","last_active":"2024-02-23T00:42:46.390432+0000","last_peered":"2024-02-23T00:42:46.390432+0000","last_clean":"2024-02-23T00:42:46.390432+0000","last_became_active":"2024-02-23T00:41:50.256773+0000","last_became_peered":"2024-02-23T00:41:50.256773+0000","last_unstale":"2024-02-23T00:42:46.390432+0000","last_undegraded":"2024-02-23T00:42:46.390432+0000","last_fullsized":"2024-02-23T00:42:46.390432+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'12","last_scrub_stamp":"2024-02-23T00:42:19.682902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.682902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":664,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.398073+0000","last_change":"2024-02-23T00:41:40.171696+0000","last_active":"2024-02-23T00:42:46.398073+0000","last_peered":"2024-02-23T00:42:46.398073+0000","last_clean":"2024-02-23T00:42:46.398073+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:42:46.398073+0000","last_undegraded":"2024-02-23T00:42:46.398073+0000","last_fullsized":"2024-02-23T00:42:46.398073+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:40.171630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:41:40.171630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":739,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.393698+0000","last_change":"2024-02-23T00:42:29.905840+0000","last_active":"2024-02-23T00:42:46.393698+0000","last_peered":"2024-02-23T00:42:46.393698+0000","last_clean":"2024-02-23T00:42:46.393698+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:42:46.393698+0000","last_undegraded":"2024-02-23T00:42:46.393698+0000","last_fullsized":"2024-02-23T00:42:46.393698+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:29.905710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:29.905710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":579,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391393+0000","last_change":"2024-02-23T00:42:18.659610+0000","last_active":"2024-02-23T00:42:46.391393+0000","last_peered":"2024-02-23T00:42:46.391393+0000","last_clean":"2024-02-23T00:42:46.391393+0000","last_became_active":"2024-02-23T00:41:50.253028+0000","last_became_peered":"2024-02-23T00:41:50.253028+0000","last_unstale":"2024-02-23T00:42:46.391393+0000","last_undegraded":"2024-02-23T00:42:46.391393+0000","last_fullsized":"2024-02-23T00:42:46.391393+0000","mapping_epoch":688,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:42:18.659534+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.659534+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":762,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642382+0000","last_change":"2024-02-23T00:41:37.375803+0000","last_active":"2024-02-23T00:42:46.642382+0000","last_peered":"2024-02-23T00:42:46.642382+0000","last_clean":"2024-02-23T00:42:46.642382+0000","last_became_active":"2024-02-23T00:41:37.229862+0000","last_became_peered":"2024-02-23T00:41:37.229862+0000","last_unstale":"2024-02-23T00:42:46.642382+0000","last_undegraded":"2024-02-23T00:42:46.642382+0000","last_fullsized":"2024-02-23T00:42:46.642382+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:40:52.825480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.825480+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":710,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.023028+0000","last_change":"2024-02-23T00:40:54.788960+0000","last_active":"2024-02-23T00:42:47.023028+0000","last_peered":"2024-02-23T00:42:47.023028+0000","last_clean":"2024-02-23T00:42:47.023028+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:42:47.023028+0000","last_undegraded":"2024-02-23T00:42:47.023028+0000","last_fullsized":"2024-02-23T00:42:47.023028+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_clean_scrub_stamp":"2024-02-23T00:40:54.788865+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":682,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397712+0000","last_change":"2024-02-23T00:40:19.126448+0000","last_active":"2024-02-23T00:42:46.397712+0000","last_peered":"2024-02-23T00:42:46.397712+0000","last_clean":"2024-02-23T00:42:46.397712+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:42:46.397712+0000","last_undegraded":"2024-02-23T00:42:46.397712+0000","last_fullsized":"2024-02-23T00:42:46.397712+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_clean_scrub_stamp":"2024-02-23T00:40:19.126375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":90,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390815+0000","last_change":"2024-02-23T00:42:39.394403+0000","last_active":"2024-02-23T00:42:46.390815+0000","last_peered":"2024-02-23T00:42:46.390815+0000","last_clean":"2024-02-23T00:42:46.390815+0000","last_became_active":"2024-02-23T00:42:39.394104+0000","last_became_peered":"2024-02-23T00:42:39.394104+0000","last_unstale":"2024-02-23T00:42:46.390815+0000","last_undegraded":"2024-02-23T00:42:46.390815+0000","last_fullsized":"2024-02-23T00:42:46.390815+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.789819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.789819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"642'14","reported_seq":285,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.228013+0000","last_change":"2024-02-23T00:41:01.213974+0000","last_active":"2024-02-23T00:42:47.228013+0000","last_peered":"2024-02-23T00:42:47.228013+0000","last_clean":"2024-02-23T00:42:47.228013+0000","last_became_active":"2024-02-23T00:41:01.212780+0000","last_became_peered":"2024-02-23T00:41:01.212780+0000","last_unstale":"2024-02-23T00:42:47.228013+0000","last_undegraded":"2024-02-23T00:42:47.228013+0000","last_fullsized":"2024-02-23T00:42:47.228013+0000","mapping_epoch":511,"log_start":"562'12","ondisk_log_start":"562'12","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:31:41.945907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:31:41.945907+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":551,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022014+0000","last_change":"2024-02-23T00:42:19.049262+0000","last_active":"2024-02-23T00:42:47.022014+0000","last_peered":"2024-02-23T00:42:47.022014+0000","last_clean":"2024-02-23T00:42:47.022014+0000","last_became_active":"2024-02-23T00:41:50.255315+0000","last_became_peered":"2024-02-23T00:41:50.255315+0000","last_unstale":"2024-02-23T00:42:47.022014+0000","last_undegraded":"2024-02-23T00:42:47.022014+0000","last_fullsized":"2024-02-23T00:42:47.022014+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'34","last_scrub_stamp":"2024-02-23T00:42:19.049190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.049190+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":726,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391531+0000","last_change":"2024-02-23T00:40:46.427758+0000","last_active":"2024-02-23T00:42:46.391531+0000","last_peered":"2024-02-23T00:42:46.391531+0000","last_clean":"2024-02-23T00:42:46.391531+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:42:46.391531+0000","last_undegraded":"2024-02-23T00:42:46.391531+0000","last_fullsized":"2024-02-23T00:42:46.391531+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:46.427689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:46.427689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":743,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642864+0000","last_change":"2024-02-23T00:42:12.433222+0000","last_active":"2024-02-23T00:42:46.642864+0000","last_peered":"2024-02-23T00:42:46.642864+0000","last_clean":"2024-02-23T00:42:46.642864+0000","last_became_active":"2024-02-23T00:40:57.488797+0000","last_became_peered":"2024-02-23T00:40:57.488797+0000","last_unstale":"2024-02-23T00:42:46.642864+0000","last_undegraded":"2024-02-23T00:42:46.642864+0000","last_fullsized":"2024-02-23T00:42:46.642864+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.433157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.433157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":707,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397907+0000","last_change":"2024-02-23T00:41:51.307510+0000","last_active":"2024-02-23T00:42:46.397907+0000","last_peered":"2024-02-23T00:42:46.397907+0000","last_clean":"2024-02-23T00:42:46.397907+0000","last_became_active":"2024-02-23T00:41:37.229061+0000","last_became_peered":"2024-02-23T00:41:37.229061+0000","last_unstale":"2024-02-23T00:42:46.397907+0000","last_undegraded":"2024-02-23T00:42:46.397907+0000","last_fullsized":"2024-02-23T00:42:46.397907+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:51.307443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:51.307443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":283,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022941+0000","last_change":"2024-02-23T00:42:39.396325+0000","last_active":"2024-02-23T00:42:47.022941+0000","last_peered":"2024-02-23T00:42:47.022941+0000","last_clean":"2024-02-23T00:42:47.022941+0000","last_became_active":"2024-02-23T00:42:39.396063+0000","last_became_peered":"2024-02-23T00:42:39.396063+0000","last_unstale":"2024-02-23T00:42:47.022941+0000","last_undegraded":"2024-02-23T00:42:47.022941+0000","last_fullsized":"2024-02-23T00:42:47.022941+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.355553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.355553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":772,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.394291+0000","last_change":"2024-02-23T00:41:53.922666+0000","last_active":"2024-02-23T00:42:46.394291+0000","last_peered":"2024-02-23T00:42:46.394291+0000","last_clean":"2024-02-23T00:42:46.394291+0000","last_became_active":"2024-02-23T00:40:57.218869+0000","last_became_peered":"2024-02-23T00:40:57.218869+0000","last_unstale":"2024-02-23T00:42:46.394291+0000","last_undegraded":"2024-02-23T00:42:46.394291+0000","last_fullsized":"2024-02-23T00:42:46.394291+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.922592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.922592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":311,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.021912+0000","last_change":"2024-02-23T00:41:38.280528+0000","last_active":"2024-02-23T00:42:47.021912+0000","last_peered":"2024-02-23T00:42:47.021912+0000","last_clean":"2024-02-23T00:42:47.021912+0000","last_became_active":"2024-02-23T00:41:37.228326+0000","last_became_peered":"2024-02-23T00:41:37.228326+0000","last_unstale":"2024-02-23T00:42:47.021912+0000","last_undegraded":"2024-02-23T00:42:47.021912+0000","last_fullsized":"2024-02-23T00:42:47.021912+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:39:28.324185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:39:28.324185+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":717,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397595+0000","last_change":"2024-02-23T00:41:42.160301+0000","last_active":"2024-02-23T00:42:46.397595+0000","last_peered":"2024-02-23T00:42:46.397595+0000","last_clean":"2024-02-23T00:42:46.397595+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:42:46.397595+0000","last_undegraded":"2024-02-23T00:42:46.397595+0000","last_fullsized":"2024-02-23T00:42:46.397595+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.160190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.160190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":709,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022176+0000","last_change":"2024-02-23T00:41:50.256151+0000","last_active":"2024-02-23T00:42:47.022176+0000","last_peered":"2024-02-23T00:42:47.022176+0000","last_clean":"2024-02-23T00:42:47.022176+0000","last_became_active":"2024-02-23T00:41:50.255850+0000","last_became_peered":"2024-02-23T00:41:50.255850+0000","last_unstale":"2024-02-23T00:42:47.022176+0000","last_undegraded":"2024-02-23T00:42:47.022176+0000","last_fullsized":"2024-02-23T00:42:47.022176+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:01.638560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:01.638560+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":248,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022940+0000","last_change":"2024-02-23T00:41:02.306933+0000","last_active":"2024-02-23T00:42:47.022940+0000","last_peered":"2024-02-23T00:42:47.022940+0000","last_clean":"2024-02-23T00:42:47.022940+0000","last_became_active":"2024-02-23T00:38:49.175801+0000","last_became_peered":"2024-02-23T00:38:49.175801+0000","last_unstale":"2024-02-23T00:42:47.022940+0000","last_undegraded":"2024-02-23T00:42:47.022940+0000","last_fullsized":"2024-02-23T00:42:47.022940+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:02.306855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.306855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":989,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397244+0000","last_change":"2024-02-23T00:42:39.399220+0000","last_active":"2024-02-23T00:42:46.397244+0000","last_peered":"2024-02-23T00:42:46.397244+0000","last_clean":"2024-02-23T00:42:46.397244+0000","last_became_active":"2024-02-23T00:42:39.398790+0000","last_became_peered":"2024-02-23T00:42:39.398790+0000","last_unstale":"2024-02-23T00:42:46.397244+0000","last_undegraded":"2024-02-23T00:42:46.397244+0000","last_fullsized":"2024-02-23T00:42:46.397244+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:17.043322+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.043322+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":774,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391146+0000","last_change":"2024-02-23T00:41:37.221015+0000","last_active":"2024-02-23T00:42:46.391146+0000","last_peered":"2024-02-23T00:42:46.391146+0000","last_clean":"2024-02-23T00:42:46.391146+0000","last_became_active":"2024-02-23T00:41:37.220531+0000","last_became_peered":"2024-02-23T00:41:37.220531+0000","last_unstale":"2024-02-23T00:42:46.391146+0000","last_undegraded":"2024-02-23T00:42:46.391146+0000","last_fullsized":"2024-02-23T00:42:46.391146+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.603558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.603558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":103,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389970+0000","last_change":"2024-02-23T00:42:27.632835+0000","last_active":"2024-02-23T00:42:46.389970+0000","last_peered":"2024-02-23T00:42:46.389970+0000","last_clean":"2024-02-23T00:42:46.389970+0000","last_became_active":"2024-02-23T00:41:37.235230+0000","last_became_peered":"2024-02-23T00:41:37.235230+0000","last_unstale":"2024-02-23T00:42:46.389970+0000","last_undegraded":"2024-02-23T00:42:46.389970+0000","last_fullsized":"2024-02-23T00:42:46.389970+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'35","last_scrub_stamp":"2024-02-23T00:42:27.632769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.632769+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":745,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643375+0000","last_change":"2024-02-23T00:41:37.230256+0000","last_active":"2024-02-23T00:42:46.643375+0000","last_peered":"2024-02-23T00:42:46.643375+0000","last_clean":"2024-02-23T00:42:46.643375+0000","last_became_active":"2024-02-23T00:41:37.229922+0000","last_became_peered":"2024-02-23T00:41:37.229922+0000","last_unstale":"2024-02-23T00:42:46.643375+0000","last_undegraded":"2024-02-23T00:42:46.643375+0000","last_fullsized":"2024-02-23T00:42:46.643375+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:56.829791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:56.829791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":729,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397212+0000","last_change":"2024-02-23T00:41:05.523604+0000","last_active":"2024-02-23T00:42:46.397212+0000","last_peered":"2024-02-23T00:42:46.397212+0000","last_clean":"2024-02-23T00:42:46.397212+0000","last_became_active":"2024-02-23T00:41:05.523365+0000","last_became_peered":"2024-02-23T00:41:05.523365+0000","last_unstale":"2024-02-23T00:42:46.397212+0000","last_undegraded":"2024-02-23T00:42:46.397212+0000","last_fullsized":"2024-02-23T00:42:46.397212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:01.179668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:41:01.179668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":664,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397259+0000","last_change":"2024-02-23T00:42:16.223783+0000","last_active":"2024-02-23T00:42:46.397259+0000","last_peered":"2024-02-23T00:42:46.397259+0000","last_clean":"2024-02-23T00:42:46.397259+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:42:46.397259+0000","last_undegraded":"2024-02-23T00:42:46.397259+0000","last_fullsized":"2024-02-23T00:42:46.397259+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:16.223720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.223720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":756,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.023066+0000","last_change":"2024-02-23T00:41:37.228614+0000","last_active":"2024-02-23T00:42:47.023066+0000","last_peered":"2024-02-23T00:42:47.023066+0000","last_clean":"2024-02-23T00:42:47.023066+0000","last_became_active":"2024-02-23T00:41:37.228303+0000","last_became_peered":"2024-02-23T00:41:37.228303+0000","last_unstale":"2024-02-23T00:42:47.023066+0000","last_undegraded":"2024-02-23T00:42:47.023066+0000","last_fullsized":"2024-02-23T00:42:47.023066+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:07.803509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:41:07.803509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":462,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390519+0000","last_change":"2024-02-23T00:42:17.721535+0000","last_active":"2024-02-23T00:42:46.390519+0000","last_peered":"2024-02-23T00:42:46.390519+0000","last_clean":"2024-02-23T00:42:46.390519+0000","last_became_active":"2024-02-23T00:41:41.224814+0000","last_became_peered":"2024-02-23T00:41:41.224814+0000","last_unstale":"2024-02-23T00:42:46.390519+0000","last_undegraded":"2024-02-23T00:42:46.390519+0000","last_fullsized":"2024-02-23T00:42:46.390519+0000","mapping_epoch":681,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:42:17.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.721471+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":745,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642141+0000","last_change":"2024-02-23T00:41:42.606731+0000","last_active":"2024-02-23T00:42:46.642141+0000","last_peered":"2024-02-23T00:42:46.642141+0000","last_clean":"2024-02-23T00:42:46.642141+0000","last_became_active":"2024-02-23T00:41:37.229722+0000","last_became_peered":"2024-02-23T00:41:37.229722+0000","last_unstale":"2024-02-23T00:42:46.642141+0000","last_undegraded":"2024-02-23T00:42:46.642141+0000","last_fullsized":"2024-02-23T00:42:46.642141+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.606635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.606635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":90,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389333+0000","last_change":"2024-02-23T00:42:39.396108+0000","last_active":"2024-02-23T00:42:46.389333+0000","last_peered":"2024-02-23T00:42:46.389333+0000","last_clean":"2024-02-23T00:42:46.389333+0000","last_became_active":"2024-02-23T00:42:39.395793+0000","last_became_peered":"2024-02-23T00:42:39.395793+0000","last_unstale":"2024-02-23T00:42:46.389333+0000","last_undegraded":"2024-02-23T00:42:46.389333+0000","last_fullsized":"2024-02-23T00:42:46.389333+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.733024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.733024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"78'2","reported_seq":765,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397637+0000","last_change":"2024-02-23T00:41:53.371931+0000","last_active":"2024-02-23T00:42:46.397637+0000","last_peered":"2024-02-23T00:42:46.397637+0000","last_clean":"2024-02-23T00:42:46.397637+0000","last_became_active":"2024-02-23T00:41:25.236069+0000","last_became_peered":"2024-02-23T00:41:25.236069+0000","last_unstale":"2024-02-23T00:42:46.397637+0000","last_undegraded":"2024-02-23T00:42:46.397637+0000","last_fullsized":"2024-02-23T00:42:46.397637+0000","mapping_epoch":511,"log_start":"78'2","ondisk_log_start":"78'2","created":60,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:41:53.371852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.371852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":672,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.394473+0000","last_change":"2024-02-23T00:41:36.715086+0000","last_active":"2024-02-23T00:42:46.394473+0000","last_peered":"2024-02-23T00:42:46.394473+0000","last_clean":"2024-02-23T00:42:46.394473+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:42:46.394473+0000","last_undegraded":"2024-02-23T00:42:46.394473+0000","last_fullsized":"2024-02-23T00:42:46.394473+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:36.715011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:36.715011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":754,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643049+0000","last_change":"2024-02-23T00:42:15.388568+0000","last_active":"2024-02-23T00:42:46.643049+0000","last_peered":"2024-02-23T00:42:46.643049+0000","last_clean":"2024-02-23T00:42:46.643049+0000","last_became_active":"2024-02-23T00:40:45.191453+0000","last_became_peered":"2024-02-23T00:40:45.191453+0000","last_unstale":"2024-02-23T00:42:46.643049+0000","last_undegraded":"2024-02-23T00:42:46.643049+0000","last_fullsized":"2024-02-23T00:42:46.643049+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.388510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.388510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":770,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390902+0000","last_change":"2024-02-23T00:41:33.595415+0000","last_active":"2024-02-23T00:42:46.390902+0000","last_peered":"2024-02-23T00:42:46.390902+0000","last_clean":"2024-02-23T00:42:46.390902+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:42:46.390902+0000","last_undegraded":"2024-02-23T00:42:46.390902+0000","last_fullsized":"2024-02-23T00:42:46.390902+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:41:33.595268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.595268+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":753,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642189+0000","last_change":"2024-02-23T00:42:28.369569+0000","last_active":"2024-02-23T00:42:46.642189+0000","last_peered":"2024-02-23T00:42:46.642189+0000","last_clean":"2024-02-23T00:42:46.642189+0000","last_became_active":"2024-02-23T00:41:37.229335+0000","last_became_peered":"2024-02-23T00:41:37.229335+0000","last_unstale":"2024-02-23T00:42:46.642189+0000","last_undegraded":"2024-02-23T00:42:46.642189+0000","last_fullsized":"2024-02-23T00:42:46.642189+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:28.369481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.369481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":719,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022533+0000","last_change":"2024-02-23T00:41:43.984556+0000","last_active":"2024-02-23T00:42:47.022533+0000","last_peered":"2024-02-23T00:42:47.022533+0000","last_clean":"2024-02-23T00:42:47.022533+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:42:47.022533+0000","last_undegraded":"2024-02-23T00:42:47.022533+0000","last_fullsized":"2024-02-23T00:42:47.022533+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:43.984490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:43.984490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":257,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022297+0000","last_change":"2024-02-23T00:42:39.396374+0000","last_active":"2024-02-23T00:42:47.022297+0000","last_peered":"2024-02-23T00:42:47.022297+0000","last_clean":"2024-02-23T00:42:47.022297+0000","last_became_active":"2024-02-23T00:42:39.396071+0000","last_became_peered":"2024-02-23T00:42:39.396071+0000","last_unstale":"2024-02-23T00:42:47.022297+0000","last_undegraded":"2024-02-23T00:42:47.022297+0000","last_fullsized":"2024-02-23T00:42:47.022297+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.474495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.474495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":767,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.023130+0000","last_change":"2024-02-23T00:42:22.037093+0000","last_active":"2024-02-23T00:42:47.023130+0000","last_peered":"2024-02-23T00:42:47.023130+0000","last_clean":"2024-02-23T00:42:47.023130+0000","last_became_active":"2024-02-23T00:41:13.469953+0000","last_became_peered":"2024-02-23T00:41:13.469953+0000","last_unstale":"2024-02-23T00:42:47.023130+0000","last_undegraded":"2024-02-23T00:42:47.023130+0000","last_fullsized":"2024-02-23T00:42:47.023130+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:22.037008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.037008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":674,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.394828+0000","last_change":"2024-02-23T00:41:35.764933+0000","last_active":"2024-02-23T00:42:46.394828+0000","last_peered":"2024-02-23T00:42:46.394828+0000","last_clean":"2024-02-23T00:42:46.394828+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:42:46.394828+0000","last_undegraded":"2024-02-23T00:42:46.394828+0000","last_fullsized":"2024-02-23T00:42:46.394828+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.764859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.764859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":762,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391616+0000","last_change":"2024-02-23T00:41:50.264305+0000","last_active":"2024-02-23T00:42:46.391616+0000","last_peered":"2024-02-23T00:42:46.391616+0000","last_clean":"2024-02-23T00:42:46.391616+0000","last_became_active":"2024-02-23T00:40:38.061696+0000","last_became_peered":"2024-02-23T00:40:38.061696+0000","last_unstale":"2024-02-23T00:42:46.391616+0000","last_undegraded":"2024-02-23T00:42:46.391616+0000","last_fullsized":"2024-02-23T00:42:46.391616+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:50.264240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:41:50.264240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":866,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.398556+0000","last_change":"2024-02-23T00:41:40.219753+0000","last_active":"2024-02-23T00:42:46.398556+0000","last_peered":"2024-02-23T00:42:46.398556+0000","last_clean":"2024-02-23T00:42:46.398556+0000","last_became_active":"2024-02-23T00:41:40.219384+0000","last_became_peered":"2024-02-23T00:41:40.219384+0000","last_unstale":"2024-02-23T00:42:46.398556+0000","last_undegraded":"2024-02-23T00:42:46.398556+0000","last_fullsized":"2024-02-23T00:42:46.398556+0000","mapping_epoch":680,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":777,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391678+0000","last_change":"2024-02-23T00:41:44.257867+0000","last_active":"2024-02-23T00:42:46.391678+0000","last_peered":"2024-02-23T00:42:46.391678+0000","last_clean":"2024-02-23T00:42:46.391678+0000","last_became_active":"2024-02-23T00:41:37.219000+0000","last_became_peered":"2024-02-23T00:41:37.219000+0000","last_unstale":"2024-02-23T00:42:46.391678+0000","last_undegraded":"2024-02-23T00:42:46.391678+0000","last_fullsized":"2024-02-23T00:42:46.391678+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.257805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.257805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":982,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397044+0000","last_change":"2024-02-23T00:42:39.401952+0000","last_active":"2024-02-23T00:42:46.397044+0000","last_peered":"2024-02-23T00:42:46.397044+0000","last_clean":"2024-02-23T00:42:46.397044+0000","last_became_active":"2024-02-23T00:42:39.401599+0000","last_became_peered":"2024-02-23T00:42:39.401599+0000","last_unstale":"2024-02-23T00:42:46.397044+0000","last_undegraded":"2024-02-23T00:42:46.397044+0000","last_fullsized":"2024-02-23T00:42:46.397044+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:12.984040+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.984040+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"62'1","reported_seq":779,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643316+0000","last_change":"2024-02-23T00:42:22.301796+0000","last_active":"2024-02-23T00:42:46.643316+0000","last_peered":"2024-02-23T00:42:46.643316+0000","last_clean":"2024-02-23T00:42:46.643316+0000","last_became_active":"2024-02-23T00:41:21.226819+0000","last_became_peered":"2024-02-23T00:41:21.226819+0000","last_unstale":"2024-02-23T00:42:46.643316+0000","last_undegraded":"2024-02-23T00:42:46.643316+0000","last_fullsized":"2024-02-23T00:42:46.643316+0000","mapping_epoch":511,"log_start":"62'1","ondisk_log_start":"62'1","created":60,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:42:22.301726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.301726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":681,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.392419+0000","last_change":"2024-02-23T00:41:04.069352+0000","last_active":"2024-02-23T00:42:46.392419+0000","last_peered":"2024-02-23T00:42:46.392419+0000","last_clean":"2024-02-23T00:42:46.392419+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:42:46.392419+0000","last_undegraded":"2024-02-23T00:42:46.392419+0000","last_fullsized":"2024-02-23T00:42:46.392419+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:04.069272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:04.069272+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":760,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389794+0000","last_change":"2024-02-23T00:42:39.393321+0000","last_active":"2024-02-23T00:42:46.389794+0000","last_peered":"2024-02-23T00:42:46.389794+0000","last_clean":"2024-02-23T00:42:46.389794+0000","last_became_active":"2024-02-23T00:42:39.393007+0000","last_became_peered":"2024-02-23T00:42:39.393007+0000","last_unstale":"2024-02-23T00:42:46.389794+0000","last_undegraded":"2024-02-23T00:42:46.389794+0000","last_fullsized":"2024-02-23T00:42:46.389794+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:18.672928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.672928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":81,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390394+0000","last_change":"2024-02-23T00:41:54.690476+0000","last_active":"2024-02-23T00:42:46.390394+0000","last_peered":"2024-02-23T00:42:46.390394+0000","last_clean":"2024-02-23T00:42:46.390394+0000","last_became_active":"2024-02-23T00:41:37.239608+0000","last_became_peered":"2024-02-23T00:41:37.239608+0000","last_unstale":"2024-02-23T00:42:46.390394+0000","last_undegraded":"2024-02-23T00:42:46.390394+0000","last_fullsized":"2024-02-23T00:42:46.390394+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:54.690407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:54.690407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":1049,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.378606+0000","last_change":"2024-02-23T00:40:15.043521+0000","last_active":"2024-02-23T00:42:47.378606+0000","last_peered":"2024-02-23T00:42:47.378606+0000","last_clean":"2024-02-23T00:42:47.378606+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:42:47.378606+0000","last_undegraded":"2024-02-23T00:42:47.378606+0000","last_fullsized":"2024-02-23T00:42:47.378606+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'48","last_scrub_stamp":"2024-02-23T00:40:15.043419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.043419+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"601'2","reported_seq":356,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397060+0000","last_change":"2024-02-23T00:40:21.186280+0000","last_active":"2024-02-23T00:42:46.397060+0000","last_peered":"2024-02-23T00:42:46.397060+0000","last_clean":"2024-02-23T00:42:46.397060+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:42:46.397060+0000","last_undegraded":"2024-02-23T00:42:46.397060+0000","last_fullsized":"2024-02-23T00:42:46.397060+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'2","last_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_deep_scrub":"601'2","last_deep_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_clean_scrub_stamp":"2024-02-23T00:40:21.186215+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":289,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022616+0000","last_change":"2024-02-23T00:42:26.240735+0000","last_active":"2024-02-23T00:42:47.022616+0000","last_peered":"2024-02-23T00:42:47.022616+0000","last_clean":"2024-02-23T00:42:47.022616+0000","last_became_active":"2024-02-23T00:41:17.220294+0000","last_became_peered":"2024-02-23T00:41:17.220294+0000","last_unstale":"2024-02-23T00:42:47.022616+0000","last_undegraded":"2024-02-23T00:42:47.022616+0000","last_fullsized":"2024-02-23T00:42:47.022616+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.240647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.240647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":686,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391807+0000","last_change":"2024-02-23T00:41:55.275163+0000","last_active":"2024-02-23T00:42:46.391807+0000","last_peered":"2024-02-23T00:42:46.391807+0000","last_clean":"2024-02-23T00:42:46.391807+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:42:46.391807+0000","last_undegraded":"2024-02-23T00:42:46.391807+0000","last_fullsized":"2024-02-23T00:42:46.391807+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.275099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.275099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":656,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022829+0000","last_change":"2024-02-23T00:42:39.399345+0000","last_active":"2024-02-23T00:42:47.022829+0000","last_peered":"2024-02-23T00:42:47.022829+0000","last_clean":"2024-02-23T00:42:47.022829+0000","last_became_active":"2024-02-23T00:42:39.399113+0000","last_became_peered":"2024-02-23T00:42:39.399113+0000","last_unstale":"2024-02-23T00:42:47.022829+0000","last_undegraded":"2024-02-23T00:42:47.022829+0000","last_fullsized":"2024-02-23T00:42:47.022829+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.024705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"681'21","reported_seq":310,"reported_epoch":745,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:42:47.228207+0000","last_change":"2024-02-23T00:42:38.417964+0000","last_active":"2024-02-23T00:42:47.228207+0000","last_peered":"2024-02-23T00:42:47.228207+0000","last_clean":"2024-02-23T00:27:24.172824+0000","last_became_active":"2024-02-23T00:41:38.214049+0000","last_became_peered":"2024-02-23T00:41:38.214049+0000","last_unstale":"2024-02-23T00:42:47.228207+0000","last_undegraded":"2024-02-23T00:42:47.228207+0000","last_fullsized":"2024-02-23T00:42:47.228207+0000","mapping_epoch":678,"log_start":"606'17","ondisk_log_start":"606'17","created":58,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'19","last_scrub_stamp":"2024-02-23T00:40:58.498752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.498752+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":168,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.396927+0000","last_change":"2024-02-23T00:40:53.236854+0000","last_active":"2024-02-23T00:42:46.396927+0000","last_peered":"2024-02-23T00:42:46.396927+0000","last_clean":"2024-02-23T00:42:46.396927+0000","last_became_active":"2024-02-23T00:40:23.194277+0000","last_became_peered":"2024-02-23T00:40:23.194277+0000","last_unstale":"2024-02-23T00:42:46.396927+0000","last_undegraded":"2024-02-23T00:42:46.396927+0000","last_fullsized":"2024-02-23T00:42:46.396927+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:40:53.236791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.236791+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":739,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397332+0000","last_change":"2024-02-23T00:41:37.229713+0000","last_active":"2024-02-23T00:42:46.397332+0000","last_peered":"2024-02-23T00:42:46.397332+0000","last_clean":"2024-02-23T00:42:46.397332+0000","last_became_active":"2024-02-23T00:41:37.229003+0000","last_became_peered":"2024-02-23T00:41:37.229003+0000","last_unstale":"2024-02-23T00:42:46.397332+0000","last_undegraded":"2024-02-23T00:42:46.397332+0000","last_fullsized":"2024-02-23T00:42:46.397332+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.208784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.208784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":81,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389425+0000","last_change":"2024-02-23T00:41:49.737992+0000","last_active":"2024-02-23T00:42:46.389425+0000","last_peered":"2024-02-23T00:42:46.389425+0000","last_clean":"2024-02-23T00:42:46.389425+0000","last_became_active":"2024-02-23T00:41:37.248459+0000","last_became_peered":"2024-02-23T00:41:37.248459+0000","last_unstale":"2024-02-23T00:42:46.389425+0000","last_undegraded":"2024-02-23T00:42:46.389425+0000","last_fullsized":"2024-02-23T00:42:46.389425+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:49.737927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.737927+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":672,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.393817+0000","last_change":"2024-02-23T00:41:40.761546+0000","last_active":"2024-02-23T00:42:46.393817+0000","last_peered":"2024-02-23T00:42:46.393817+0000","last_clean":"2024-02-23T00:42:46.393817+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:42:46.393817+0000","last_undegraded":"2024-02-23T00:42:46.393817+0000","last_fullsized":"2024-02-23T00:42:46.393817+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:40.761466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:41:40.761466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":16,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643070+0000","last_change":"2024-02-23T00:42:41.402454+0000","last_active":"2024-02-23T00:42:46.643070+0000","last_peered":"2024-02-23T00:42:46.643070+0000","last_clean":"2024-02-23T00:42:46.643070+0000","last_became_active":"2024-02-23T00:42:41.402111+0000","last_became_peered":"2024-02-23T00:42:41.402111+0000","last_unstale":"2024-02-23T00:42:46.643070+0000","last_undegraded":"2024-02-23T00:42:46.643070+0000","last_fullsized":"2024-02-23T00:42:46.643070+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.789819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.789819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":776,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391895+0000","last_change":"2024-02-23T00:42:26.380761+0000","last_active":"2024-02-23T00:42:46.391895+0000","last_peered":"2024-02-23T00:42:46.391895+0000","last_clean":"2024-02-23T00:42:46.391895+0000","last_became_active":"2024-02-23T00:41:37.239598+0000","last_became_peered":"2024-02-23T00:41:37.239598+0000","last_unstale":"2024-02-23T00:42:46.391895+0000","last_undegraded":"2024-02-23T00:42:46.391895+0000","last_fullsized":"2024-02-23T00:42:46.391895+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.380671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":657,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397000+0000","last_change":"2024-02-23T00:41:50.258032+0000","last_active":"2024-02-23T00:42:46.397000+0000","last_peered":"2024-02-23T00:42:46.397000+0000","last_clean":"2024-02-23T00:42:46.397000+0000","last_became_active":"2024-02-23T00:41:50.257656+0000","last_became_peered":"2024-02-23T00:41:50.257656+0000","last_unstale":"2024-02-23T00:42:46.397000+0000","last_undegraded":"2024-02-23T00:42:46.397000+0000","last_fullsized":"2024-02-23T00:42:46.397000+0000","mapping_epoch":688,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:40:50.676380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.676380+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":774,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.394995+0000","last_change":"2024-02-23T00:41:54.923278+0000","last_active":"2024-02-23T00:42:46.394995+0000","last_peered":"2024-02-23T00:42:46.394995+0000","last_clean":"2024-02-23T00:42:46.394995+0000","last_became_active":"2024-02-23T00:41:37.238708+0000","last_became_peered":"2024-02-23T00:41:37.238708+0000","last_unstale":"2024-02-23T00:42:46.394995+0000","last_undegraded":"2024-02-23T00:42:46.394995+0000","last_fullsized":"2024-02-23T00:42:46.394995+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:41:54.923214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:54.923214+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":266,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022640+0000","last_change":"2024-02-23T00:41:33.042594+0000","last_active":"2024-02-23T00:42:47.022640+0000","last_peered":"2024-02-23T00:42:47.022640+0000","last_clean":"2024-02-23T00:42:47.022640+0000","last_became_active":"2024-02-23T00:40:17.214288+0000","last_became_peered":"2024-02-23T00:40:17.214288+0000","last_unstale":"2024-02-23T00:42:47.022640+0000","last_undegraded":"2024-02-23T00:42:47.022640+0000","last_fullsized":"2024-02-23T00:42:47.022640+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:33.042520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.042520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":668,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022296+0000","last_change":"2024-02-23T00:42:13.006928+0000","last_active":"2024-02-23T00:42:47.022296+0000","last_peered":"2024-02-23T00:42:47.022296+0000","last_clean":"2024-02-23T00:42:47.022296+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:42:47.022296+0000","last_undegraded":"2024-02-23T00:42:47.022296+0000","last_fullsized":"2024-02-23T00:42:47.022296+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:13.006833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.006833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.641956+0000","last_change":"2024-02-23T00:42:41.400606+0000","last_active":"2024-02-23T00:42:46.641956+0000","last_peered":"2024-02-23T00:42:46.641956+0000","last_clean":"2024-02-23T00:42:46.641956+0000","last_became_active":"2024-02-23T00:42:41.399155+0000","last_became_peered":"2024-02-23T00:42:41.399155+0000","last_unstale":"2024-02-23T00:42:46.641956+0000","last_undegraded":"2024-02-23T00:42:46.641956+0000","last_fullsized":"2024-02-23T00:42:46.641956+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.733024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.733024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":515,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390897+0000","last_change":"2024-02-23T00:42:14.760378+0000","last_active":"2024-02-23T00:42:46.390897+0000","last_peered":"2024-02-23T00:42:46.390897+0000","last_clean":"2024-02-23T00:42:46.390897+0000","last_became_active":"2024-02-23T00:41:38.217979+0000","last_became_peered":"2024-02-23T00:41:38.217979+0000","last_unstale":"2024-02-23T00:42:46.390897+0000","last_undegraded":"2024-02-23T00:42:46.390897+0000","last_fullsized":"2024-02-23T00:42:46.390897+0000","mapping_epoch":678,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:42:14.760308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.760308+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":753,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391827+0000","last_change":"2024-02-23T00:42:21.442459+0000","last_active":"2024-02-23T00:42:46.391827+0000","last_peered":"2024-02-23T00:42:46.391827+0000","last_clean":"2024-02-23T00:42:46.391827+0000","last_became_active":"2024-02-23T00:41:37.238750+0000","last_became_peered":"2024-02-23T00:41:37.238750+0000","last_unstale":"2024-02-23T00:42:46.391827+0000","last_undegraded":"2024-02-23T00:42:46.391827+0000","last_fullsized":"2024-02-23T00:42:46.391827+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:21.442362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:21.442362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":253,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022944+0000","last_change":"2024-02-23T00:42:20.334112+0000","last_active":"2024-02-23T00:42:47.022944+0000","last_peered":"2024-02-23T00:42:47.022944+0000","last_clean":"2024-02-23T00:42:47.022944+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:42:47.022944+0000","last_undegraded":"2024-02-23T00:42:47.022944+0000","last_fullsized":"2024-02-23T00:42:47.022944+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.334013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.334013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":248,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022599+0000","last_change":"2024-02-23T00:41:48.297291+0000","last_active":"2024-02-23T00:42:47.022599+0000","last_peered":"2024-02-23T00:42:47.022599+0000","last_clean":"2024-02-23T00:42:47.022599+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:42:47.022599+0000","last_undegraded":"2024-02-23T00:42:47.022599+0000","last_fullsized":"2024-02-23T00:42:47.022599+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:48.297220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:48.297220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":142,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390081+0000","last_change":"2024-02-23T00:42:41.406692+0000","last_active":"2024-02-23T00:42:46.390081+0000","last_peered":"2024-02-23T00:42:46.390081+0000","last_clean":"2024-02-23T00:42:46.390081+0000","last_became_active":"2024-02-23T00:42:41.406233+0000","last_became_peered":"2024-02-23T00:42:41.406233+0000","last_unstale":"2024-02-23T00:42:46.390081+0000","last_undegraded":"2024-02-23T00:42:46.390081+0000","last_fullsized":"2024-02-23T00:42:46.390081+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:12.984040+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.984040+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":856,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022759+0000","last_change":"2024-02-23T00:40:51.316847+0000","last_active":"2024-02-23T00:42:47.022759+0000","last_peered":"2024-02-23T00:42:47.022759+0000","last_clean":"2024-02-23T00:42:47.022759+0000","last_became_active":"2024-02-23T00:40:51.316452+0000","last_became_peered":"2024-02-23T00:40:51.316452+0000","last_unstale":"2024-02-23T00:42:47.022759+0000","last_undegraded":"2024-02-23T00:42:47.022759+0000","last_fullsized":"2024-02-23T00:42:47.022759+0000","mapping_epoch":631,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"63'19","last_scrub_stamp":"2024-02-23T00:36:23.773847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:36:23.773847+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":732,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390717+0000","last_change":"2024-02-23T00:42:19.655247+0000","last_active":"2024-02-23T00:42:46.390717+0000","last_peered":"2024-02-23T00:42:46.390717+0000","last_clean":"2024-02-23T00:42:46.390717+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:42:46.390717+0000","last_undegraded":"2024-02-23T00:42:46.390717+0000","last_fullsized":"2024-02-23T00:42:46.390717+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.655188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.655188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":728,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022754+0000","last_change":"2024-02-23T00:41:53.984748+0000","last_active":"2024-02-23T00:42:47.022754+0000","last_peered":"2024-02-23T00:42:47.022754+0000","last_clean":"2024-02-23T00:42:47.022754+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:42:47.022754+0000","last_undegraded":"2024-02-23T00:42:47.022754+0000","last_fullsized":"2024-02-23T00:42:47.022754+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.984679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.984679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":692,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391129+0000","last_change":"2024-02-23T00:41:02.451992+0000","last_active":"2024-02-23T00:42:46.391129+0000","last_peered":"2024-02-23T00:42:46.391129+0000","last_clean":"2024-02-23T00:42:46.391129+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:42:46.391129+0000","last_undegraded":"2024-02-23T00:42:46.391129+0000","last_fullsized":"2024-02-23T00:42:46.391129+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:02.451884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.451884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":16,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642315+0000","last_change":"2024-02-23T00:42:41.402889+0000","last_active":"2024-02-23T00:42:46.642315+0000","last_peered":"2024-02-23T00:42:46.642315+0000","last_clean":"2024-02-23T00:42:46.642315+0000","last_became_active":"2024-02-23T00:42:41.402543+0000","last_became_peered":"2024-02-23T00:42:41.402543+0000","last_unstale":"2024-02-23T00:42:46.642315+0000","last_undegraded":"2024-02-23T00:42:46.642315+0000","last_fullsized":"2024-02-23T00:42:46.642315+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:18.672928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.672928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"688'39","reported_seq":271,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022535+0000","last_change":"2024-02-23T00:41:49.338035+0000","last_active":"2024-02-23T00:42:47.022535+0000","last_peered":"2024-02-23T00:42:47.022535+0000","last_clean":"2024-02-23T00:42:47.022535+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:42:47.022535+0000","last_undegraded":"2024-02-23T00:42:47.022535+0000","last_fullsized":"2024-02-23T00:42:47.022535+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'39","last_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_deep_scrub":"688'39","last_deep_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.337969+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":745,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022098+0000","last_change":"2024-02-23T00:41:41.955717+0000","last_active":"2024-02-23T00:42:47.022098+0000","last_peered":"2024-02-23T00:42:47.022098+0000","last_clean":"2024-02-23T00:42:47.022098+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:42:47.022098+0000","last_undegraded":"2024-02-23T00:42:47.022098+0000","last_fullsized":"2024-02-23T00:42:47.022098+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:41.955589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:41:41.955589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"681'12","reported_seq":934,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.379287+0000","last_change":"2024-02-23T00:41:37.247785+0000","last_active":"2024-02-23T00:42:47.379287+0000","last_peered":"2024-02-23T00:42:47.379287+0000","last_clean":"2024-02-23T00:42:47.379287+0000","last_became_active":"2024-02-23T00:41:37.228441+0000","last_became_peered":"2024-02-23T00:41:37.228441+0000","last_unstale":"2024-02-23T00:42:47.379287+0000","last_undegraded":"2024-02-23T00:42:47.379287+0000","last_fullsized":"2024-02-23T00:42:47.379287+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"606'11","last_scrub_stamp":"2024-02-23T00:40:45.004428+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.004428+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":668,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.023042+0000","last_change":"2024-02-23T00:42:17.056253+0000","last_active":"2024-02-23T00:42:47.023042+0000","last_peered":"2024-02-23T00:42:47.023042+0000","last_clean":"2024-02-23T00:42:47.023042+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:42:47.023042+0000","last_undegraded":"2024-02-23T00:42:47.023042+0000","last_fullsized":"2024-02-23T00:42:47.023042+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:17.056153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.056153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389267+0000","last_change":"2024-02-23T00:42:41.403218+0000","last_active":"2024-02-23T00:42:46.389267+0000","last_peered":"2024-02-23T00:42:46.389267+0000","last_clean":"2024-02-23T00:42:46.389267+0000","last_became_active":"2024-02-23T00:42:41.402683+0000","last_became_peered":"2024-02-23T00:42:41.402683+0000","last_unstale":"2024-02-23T00:42:46.389267+0000","last_undegraded":"2024-02-23T00:42:46.389267+0000","last_fullsized":"2024-02-23T00:42:46.389267+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.474495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.474495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":804,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022854+0000","last_change":"2024-02-23T00:42:28.967292+0000","last_active":"2024-02-23T00:42:47.022854+0000","last_peered":"2024-02-23T00:42:47.022854+0000","last_clean":"2024-02-23T00:42:47.022854+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:42:47.022854+0000","last_undegraded":"2024-02-23T00:42:47.022854+0000","last_fullsized":"2024-02-23T00:42:47.022854+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:42:28.967208+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.967208+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":81,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.389862+0000","last_change":"2024-02-23T00:41:47.719694+0000","last_active":"2024-02-23T00:42:46.389862+0000","last_peered":"2024-02-23T00:42:46.389862+0000","last_clean":"2024-02-23T00:42:46.389862+0000","last_became_active":"2024-02-23T00:41:37.235746+0000","last_became_peered":"2024-02-23T00:41:37.235746+0000","last_unstale":"2024-02-23T00:42:46.389862+0000","last_undegraded":"2024-02-23T00:42:46.389862+0000","last_fullsized":"2024-02-23T00:42:46.389862+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.719590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.719590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":301,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.229643+0000","last_change":"2024-02-23T00:40:50.453642+0000","last_active":"2024-02-23T00:42:47.229643+0000","last_peered":"2024-02-23T00:42:47.229643+0000","last_clean":"2024-02-23T00:42:47.229643+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:42:47.229643+0000","last_undegraded":"2024-02-23T00:42:47.229643+0000","last_fullsized":"2024-02-23T00:42:47.229643+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'11","last_scrub_stamp":"2024-02-23T00:40:50.453579+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.453579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":248,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022881+0000","last_change":"2024-02-23T00:41:03.320478+0000","last_active":"2024-02-23T00:42:47.022881+0000","last_peered":"2024-02-23T00:42:47.022881+0000","last_clean":"2024-02-23T00:42:47.022881+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:42:47.022881+0000","last_undegraded":"2024-02-23T00:42:47.022881+0000","last_fullsized":"2024-02-23T00:42:47.022881+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:03.320387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:03.320387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643086+0000","last_change":"2024-02-23T00:42:41.399293+0000","last_active":"2024-02-23T00:42:46.643086+0000","last_peered":"2024-02-23T00:42:46.643086+0000","last_clean":"2024-02-23T00:42:46.643086+0000","last_became_active":"2024-02-23T00:42:41.398941+0000","last_became_peered":"2024-02-23T00:42:41.398941+0000","last_unstale":"2024-02-23T00:42:46.643086+0000","last_undegraded":"2024-02-23T00:42:46.643086+0000","last_fullsized":"2024-02-23T00:42:46.643086+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.024705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":740,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397375+0000","last_change":"2024-02-23T00:41:02.134977+0000","last_active":"2024-02-23T00:42:46.397375+0000","last_peered":"2024-02-23T00:42:46.397375+0000","last_clean":"2024-02-23T00:42:46.397375+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:42:46.397375+0000","last_undegraded":"2024-02-23T00:42:46.397375+0000","last_fullsized":"2024-02-23T00:42:46.397375+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'15","last_scrub_stamp":"2024-02-23T00:41:02.134898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.134898+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":754,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391032+0000","last_change":"2024-02-23T00:41:42.607771+0000","last_active":"2024-02-23T00:42:46.391032+0000","last_peered":"2024-02-23T00:42:46.391032+0000","last_clean":"2024-02-23T00:42:46.391032+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:42:46.391032+0000","last_undegraded":"2024-02-23T00:42:46.391032+0000","last_fullsized":"2024-02-23T00:42:46.391032+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.607642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.607642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":272,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022551+0000","last_change":"2024-02-23T00:42:27.197397+0000","last_active":"2024-02-23T00:42:47.022551+0000","last_peered":"2024-02-23T00:42:47.022551+0000","last_clean":"2024-02-23T00:42:47.022551+0000","last_became_active":"2024-02-23T00:41:37.252612+0000","last_became_peered":"2024-02-23T00:41:37.252612+0000","last_unstale":"2024-02-23T00:42:47.022551+0000","last_undegraded":"2024-02-23T00:42:47.022551+0000","last_fullsized":"2024-02-23T00:42:47.022551+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.197290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":696,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.393790+0000","last_change":"2024-02-23T00:42:12.891119+0000","last_active":"2024-02-23T00:42:46.393790+0000","last_peered":"2024-02-23T00:42:46.393790+0000","last_clean":"2024-02-23T00:42:46.393790+0000","last_became_active":"2024-02-23T00:41:37.238185+0000","last_became_peered":"2024-02-23T00:41:37.238185+0000","last_unstale":"2024-02-23T00:42:46.393790+0000","last_undegraded":"2024-02-23T00:42:46.393790+0000","last_fullsized":"2024-02-23T00:42:46.393790+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.891052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.891052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"78'6","reported_seq":16,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642733+0000","last_change":"2024-02-23T00:42:41.398157+0000","last_active":"2024-02-23T00:42:46.642733+0000","last_peered":"2024-02-23T00:42:46.642733+0000","last_clean":"2024-02-23T00:42:46.642733+0000","last_became_active":"2024-02-23T00:42:41.397727+0000","last_became_peered":"2024-02-23T00:42:41.397727+0000","last_unstale":"2024-02-23T00:42:46.642733+0000","last_undegraded":"2024-02-23T00:42:46.642733+0000","last_fullsized":"2024-02-23T00:42:46.642733+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:17.043322+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.043322+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":775,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.394189+0000","last_change":"2024-02-23T00:41:33.715504+0000","last_active":"2024-02-23T00:42:46.394189+0000","last_peered":"2024-02-23T00:42:46.394189+0000","last_clean":"2024-02-23T00:42:46.394189+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:42:46.394189+0000","last_undegraded":"2024-02-23T00:42:46.394189+0000","last_fullsized":"2024-02-23T00:42:46.394189+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:41:33.715382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.715382+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":736,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.398169+0000","last_change":"2024-02-23T00:41:37.230185+0000","last_active":"2024-02-23T00:42:46.398169+0000","last_peered":"2024-02-23T00:42:46.398169+0000","last_clean":"2024-02-23T00:42:46.398169+0000","last_became_active":"2024-02-23T00:41:37.229726+0000","last_became_peered":"2024-02-23T00:41:37.229726+0000","last_unstale":"2024-02-23T00:42:46.398169+0000","last_undegraded":"2024-02-23T00:42:46.398169+0000","last_fullsized":"2024-02-23T00:42:46.398169+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:56.203202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:40:56.203202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":731,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397459+0000","last_change":"2024-02-23T00:41:35.138434+0000","last_active":"2024-02-23T00:42:46.397459+0000","last_peered":"2024-02-23T00:42:46.397459+0000","last_clean":"2024-02-23T00:42:46.397459+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:42:46.397459+0000","last_undegraded":"2024-02-23T00:42:46.397459+0000","last_fullsized":"2024-02-23T00:42:46.397459+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.138319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.138319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":723,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391146+0000","last_change":"2024-02-23T00:42:22.669179+0000","last_active":"2024-02-23T00:42:46.391146+0000","last_peered":"2024-02-23T00:42:46.391146+0000","last_clean":"2024-02-23T00:42:46.391146+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:42:46.391146+0000","last_undegraded":"2024-02-23T00:42:46.391146+0000","last_fullsized":"2024-02-23T00:42:46.391146+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:42:22.669090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.669090+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":254,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022047+0000","last_change":"2024-02-23T00:40:12.128771+0000","last_active":"2024-02-23T00:42:47.022047+0000","last_peered":"2024-02-23T00:42:47.022047+0000","last_clean":"2024-02-23T00:42:47.022047+0000","last_became_active":"2024-02-23T00:38:49.217612+0000","last_became_peered":"2024-02-23T00:38:49.217612+0000","last_unstale":"2024-02-23T00:42:47.022047+0000","last_undegraded":"2024-02-23T00:42:47.022047+0000","last_fullsized":"2024-02-23T00:42:47.022047+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'10","last_scrub_stamp":"2024-02-23T00:40:12.128684+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.128684+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":719,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022881+0000","last_change":"2024-02-23T00:42:15.038961+0000","last_active":"2024-02-23T00:42:47.022881+0000","last_peered":"2024-02-23T00:42:47.022881+0000","last_clean":"2024-02-23T00:42:47.022881+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:42:47.022881+0000","last_undegraded":"2024-02-23T00:42:47.022881+0000","last_fullsized":"2024-02-23T00:42:47.022881+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.038897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.038897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":723,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390686+0000","last_change":"2024-02-23T00:41:34.564079+0000","last_active":"2024-02-23T00:42:46.390686+0000","last_peered":"2024-02-23T00:42:46.390686+0000","last_clean":"2024-02-23T00:42:46.390686+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:42:46.390686+0000","last_undegraded":"2024-02-23T00:42:46.390686+0000","last_fullsized":"2024-02-23T00:42:46.390686+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:34.563969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.563969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":664,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397549+0000","last_change":"2024-02-23T00:41:44.196162+0000","last_active":"2024-02-23T00:42:46.397549+0000","last_peered":"2024-02-23T00:42:46.397549+0000","last_clean":"2024-02-23T00:42:46.397549+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:42:46.397549+0000","last_undegraded":"2024-02-23T00:42:46.397549+0000","last_fullsized":"2024-02-23T00:42:46.397549+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.196088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.196088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":16,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022619+0000","last_change":"2024-02-23T00:42:41.391815+0000","last_active":"2024-02-23T00:42:47.022619+0000","last_peered":"2024-02-23T00:42:47.022619+0000","last_clean":"2024-02-23T00:42:47.022619+0000","last_became_active":"2024-02-23T00:42:41.391484+0000","last_became_peered":"2024-02-23T00:42:41.391484+0000","last_unstale":"2024-02-23T00:42:47.022619+0000","last_undegraded":"2024-02-23T00:42:47.022619+0000","last_fullsized":"2024-02-23T00:42:47.022619+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.789819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.789819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":265,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022365+0000","last_change":"2024-02-23T00:40:43.233713+0000","last_active":"2024-02-23T00:42:47.022365+0000","last_peered":"2024-02-23T00:42:47.022365+0000","last_clean":"2024-02-23T00:42:47.022365+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:42:47.022365+0000","last_undegraded":"2024-02-23T00:42:47.022365+0000","last_fullsized":"2024-02-23T00:42:47.022365+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:40:43.233609+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:40:43.233609+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":731,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391380+0000","last_change":"2024-02-23T00:42:25.724438+0000","last_active":"2024-02-23T00:42:46.391380+0000","last_peered":"2024-02-23T00:42:46.391380+0000","last_clean":"2024-02-23T00:42:46.391380+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:42:46.391380+0000","last_undegraded":"2024-02-23T00:42:46.391380+0000","last_fullsized":"2024-02-23T00:42:46.391380+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:25.724375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:25.724375+0000","last_clean_scrub_stamp":"2024-02-23T00:42:25.724375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":722,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642597+0000","last_change":"2024-02-23T00:41:44.520521+0000","last_active":"2024-02-23T00:42:46.642597+0000","last_peered":"2024-02-23T00:42:46.642597+0000","last_clean":"2024-02-23T00:42:46.642597+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:42:46.642597+0000","last_undegraded":"2024-02-23T00:42:46.642597+0000","last_fullsized":"2024-02-23T00:42:46.642597+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.520415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.520415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":248,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022529+0000","last_change":"2024-02-23T00:41:37.096978+0000","last_active":"2024-02-23T00:42:47.022529+0000","last_peered":"2024-02-23T00:42:47.022529+0000","last_clean":"2024-02-23T00:42:47.022529+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:42:47.022529+0000","last_undegraded":"2024-02-23T00:42:47.022529+0000","last_fullsized":"2024-02-23T00:42:47.022529+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:37.096906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:37.096906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":636,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390600+0000","last_change":"2024-02-23T00:42:10.241262+0000","last_active":"2024-02-23T00:42:46.390600+0000","last_peered":"2024-02-23T00:42:46.390600+0000","last_clean":"2024-02-23T00:42:46.390600+0000","last_became_active":"2024-02-23T00:42:10.240911+0000","last_became_peered":"2024-02-23T00:42:10.240911+0000","last_unstale":"2024-02-23T00:42:46.390600+0000","last_undegraded":"2024-02-23T00:42:46.390600+0000","last_fullsized":"2024-02-23T00:42:46.390600+0000","mapping_epoch":707,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:38:13.072082+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.072082+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":723,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.021850+0000","last_change":"2024-02-23T00:40:55.816232+0000","last_active":"2024-02-23T00:42:47.021850+0000","last_peered":"2024-02-23T00:42:47.021850+0000","last_clean":"2024-02-23T00:42:47.021850+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:42:47.021850+0000","last_undegraded":"2024-02-23T00:42:47.021850+0000","last_fullsized":"2024-02-23T00:42:47.021850+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:55.816155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:55.816155+0000","last_clean_scrub_stamp":"2024-02-23T00:40:55.816155+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":921,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.380228+0000","last_change":"2024-02-23T00:40:45.431895+0000","last_active":"2024-02-23T00:42:47.380228+0000","last_peered":"2024-02-23T00:42:47.380228+0000","last_clean":"2024-02-23T00:42:47.380228+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:42:47.380228+0000","last_undegraded":"2024-02-23T00:42:47.380228+0000","last_fullsized":"2024-02-23T00:42:47.380228+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:40:45.431814+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:40:45.431814+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":717,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643187+0000","last_change":"2024-02-23T00:42:14.427990+0000","last_active":"2024-02-23T00:42:46.643187+0000","last_peered":"2024-02-23T00:42:46.643187+0000","last_clean":"2024-02-23T00:42:46.643187+0000","last_became_active":"2024-02-23T00:41:37.229426+0000","last_became_peered":"2024-02-23T00:41:37.229426+0000","last_unstale":"2024-02-23T00:42:46.643187+0000","last_undegraded":"2024-02-23T00:42:46.643187+0000","last_fullsized":"2024-02-23T00:42:46.643187+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.427861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.427861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":935,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391464+0000","last_change":"2024-02-23T00:42:20.635986+0000","last_active":"2024-02-23T00:42:46.391464+0000","last_peered":"2024-02-23T00:42:46.391464+0000","last_clean":"2024-02-23T00:42:46.391464+0000","last_became_active":"2024-02-23T00:41:55.327964+0000","last_became_peered":"2024-02-23T00:41:55.327964+0000","last_unstale":"2024-02-23T00:42:46.391464+0000","last_undegraded":"2024-02-23T00:42:46.391464+0000","last_fullsized":"2024-02-23T00:42:46.391464+0000","mapping_epoch":692,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:42:20.635897+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.635897+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":272,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022208+0000","last_change":"2024-02-23T00:42:14.278845+0000","last_active":"2024-02-23T00:42:47.022208+0000","last_peered":"2024-02-23T00:42:47.022208+0000","last_clean":"2024-02-23T00:42:47.022208+0000","last_became_active":"2024-02-23T00:41:37.228694+0000","last_became_peered":"2024-02-23T00:41:37.228694+0000","last_unstale":"2024-02-23T00:42:47.022208+0000","last_undegraded":"2024-02-23T00:42:47.022208+0000","last_fullsized":"2024-02-23T00:42:47.022208+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.278782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.278782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":76,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390741+0000","last_change":"2024-02-23T00:41:37.248384+0000","last_active":"2024-02-23T00:42:46.390741+0000","last_peered":"2024-02-23T00:42:46.390741+0000","last_clean":"2024-02-23T00:42:46.390741+0000","last_became_active":"2024-02-23T00:41:37.247946+0000","last_became_peered":"2024-02-23T00:41:37.247946+0000","last_unstale":"2024-02-23T00:42:46.390741+0000","last_undegraded":"2024-02-23T00:42:46.390741+0000","last_fullsized":"2024-02-23T00:42:46.390741+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:59.777391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:59.777391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":674,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643514+0000","last_change":"2024-02-23T00:41:53.497861+0000","last_active":"2024-02-23T00:42:46.643514+0000","last_peered":"2024-02-23T00:42:46.643514+0000","last_clean":"2024-02-23T00:42:46.643514+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:42:46.643514+0000","last_undegraded":"2024-02-23T00:42:46.643514+0000","last_fullsized":"2024-02-23T00:42:46.643514+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.497792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.497792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":864,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390538+0000","last_change":"2024-02-23T00:42:16.497262+0000","last_active":"2024-02-23T00:42:46.390538+0000","last_peered":"2024-02-23T00:42:46.390538+0000","last_clean":"2024-02-23T00:42:46.390538+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:42:46.390538+0000","last_undegraded":"2024-02-23T00:42:46.390538+0000","last_fullsized":"2024-02-23T00:42:46.390538+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:42:16.497198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.497198+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":735,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391265+0000","last_change":"2024-02-23T00:41:48.632172+0000","last_active":"2024-02-23T00:42:46.391265+0000","last_peered":"2024-02-23T00:42:46.391265+0000","last_clean":"2024-02-23T00:42:46.391265+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:42:46.391265+0000","last_undegraded":"2024-02-23T00:42:46.391265+0000","last_fullsized":"2024-02-23T00:42:46.391265+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:48.632098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:41:48.632098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":734,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.398303+0000","last_change":"2024-02-23T00:41:37.230024+0000","last_active":"2024-02-23T00:42:46.398303+0000","last_peered":"2024-02-23T00:42:46.398303+0000","last_clean":"2024-02-23T00:42:46.398303+0000","last_became_active":"2024-02-23T00:41:37.229323+0000","last_became_peered":"2024-02-23T00:41:37.229323+0000","last_unstale":"2024-02-23T00:42:46.398303+0000","last_undegraded":"2024-02-23T00:42:46.398303+0000","last_fullsized":"2024-02-23T00:42:46.398303+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:57.174945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:40:57.174945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":250,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022522+0000","last_change":"2024-02-23T00:40:11.172134+0000","last_active":"2024-02-23T00:42:47.022522+0000","last_peered":"2024-02-23T00:42:47.022522+0000","last_clean":"2024-02-23T00:42:47.022522+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:42:47.022522+0000","last_undegraded":"2024-02-23T00:42:47.022522+0000","last_fullsized":"2024-02-23T00:42:47.022522+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:11.172065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:11.172065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":832,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022534+0000","last_change":"2024-02-23T00:40:58.399481+0000","last_active":"2024-02-23T00:42:47.022534+0000","last_peered":"2024-02-23T00:42:47.022534+0000","last_clean":"2024-02-23T00:42:47.022534+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:42:47.022534+0000","last_undegraded":"2024-02-23T00:42:47.022534+0000","last_fullsized":"2024-02-23T00:42:47.022534+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_deep_scrub":"453'44","last_deep_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.399368+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":248,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022719+0000","last_change":"2024-02-23T00:40:07.255442+0000","last_active":"2024-02-23T00:42:47.022719+0000","last_peered":"2024-02-23T00:42:47.022719+0000","last_clean":"2024-02-23T00:42:47.022719+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:42:47.022719+0000","last_undegraded":"2024-02-23T00:42:47.022719+0000","last_fullsized":"2024-02-23T00:42:47.022719+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:07.255349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:07.255349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":558,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022430+0000","last_change":"2024-02-23T00:41:50.257038+0000","last_active":"2024-02-23T00:42:47.022430+0000","last_peered":"2024-02-23T00:42:47.022430+0000","last_clean":"2024-02-23T00:42:47.022430+0000","last_became_active":"2024-02-23T00:41:50.256751+0000","last_became_peered":"2024-02-23T00:41:50.256751+0000","last_unstale":"2024-02-23T00:42:47.022430+0000","last_undegraded":"2024-02-23T00:42:47.022430+0000","last_fullsized":"2024-02-23T00:42:47.022430+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:07.610329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:07.610329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":663,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022775+0000","last_change":"2024-02-23T00:40:53.782744+0000","last_active":"2024-02-23T00:42:47.022775+0000","last_peered":"2024-02-23T00:42:47.022775+0000","last_clean":"2024-02-23T00:42:47.022775+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:42:47.022775+0000","last_undegraded":"2024-02-23T00:42:47.022775+0000","last_fullsized":"2024-02-23T00:42:47.022775+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:53.782669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.782669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":932,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.021990+0000","last_change":"2024-02-23T00:41:40.216294+0000","last_active":"2024-02-23T00:42:47.021990+0000","last_peered":"2024-02-23T00:42:47.021990+0000","last_clean":"2024-02-23T00:42:47.021990+0000","last_became_active":"2024-02-23T00:41:40.215919+0000","last_became_peered":"2024-02-23T00:41:40.215919+0000","last_unstale":"2024-02-23T00:42:47.021990+0000","last_undegraded":"2024-02-23T00:42:47.021990+0000","last_fullsized":"2024-02-23T00:42:47.021990+0000","mapping_epoch":680,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":435,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.394257+0000","last_change":"2024-02-23T00:42:14.878836+0000","last_active":"2024-02-23T00:42:46.394257+0000","last_peered":"2024-02-23T00:42:46.394257+0000","last_clean":"2024-02-23T00:42:46.394257+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:42:46.394257+0000","last_undegraded":"2024-02-23T00:42:46.394257+0000","last_fullsized":"2024-02-23T00:42:46.394257+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.878770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.878770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":729,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022619+0000","last_change":"2024-02-23T00:41:37.228970+0000","last_active":"2024-02-23T00:42:47.022619+0000","last_peered":"2024-02-23T00:42:47.022619+0000","last_clean":"2024-02-23T00:42:47.022619+0000","last_became_active":"2024-02-23T00:41:37.228310+0000","last_became_peered":"2024-02-23T00:41:37.228310+0000","last_unstale":"2024-02-23T00:42:47.022619+0000","last_undegraded":"2024-02-23T00:42:47.022619+0000","last_fullsized":"2024-02-23T00:42:47.022619+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:46.004307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:46.004307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":679,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.641974+0000","last_change":"2024-02-23T00:42:19.323714+0000","last_active":"2024-02-23T00:42:46.641974+0000","last_peered":"2024-02-23T00:42:46.641974+0000","last_clean":"2024-02-23T00:42:46.641974+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:42:46.641974+0000","last_undegraded":"2024-02-23T00:42:46.641974+0000","last_fullsized":"2024-02-23T00:42:46.641974+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.323643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.323643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":263,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022454+0000","last_change":"2024-02-23T00:41:50.256190+0000","last_active":"2024-02-23T00:42:47.022454+0000","last_peered":"2024-02-23T00:42:47.022454+0000","last_clean":"2024-02-23T00:42:47.022454+0000","last_became_active":"2024-02-23T00:41:50.255862+0000","last_became_peered":"2024-02-23T00:41:50.255862+0000","last_unstale":"2024-02-23T00:42:47.022454+0000","last_undegraded":"2024-02-23T00:42:47.022454+0000","last_fullsized":"2024-02-23T00:42:47.022454+0000","mapping_epoch":688,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:41:00.643135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:00.643135+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":745,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.643179+0000","last_change":"2024-02-23T00:41:37.232071+0000","last_active":"2024-02-23T00:42:46.643179+0000","last_peered":"2024-02-23T00:42:46.643179+0000","last_clean":"2024-02-23T00:42:46.643179+0000","last_became_active":"2024-02-23T00:41:37.230054+0000","last_became_peered":"2024-02-23T00:41:37.230054+0000","last_unstale":"2024-02-23T00:42:46.643179+0000","last_undegraded":"2024-02-23T00:42:46.643179+0000","last_fullsized":"2024-02-23T00:42:46.643179+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:07.753676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:41:07.753676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":727,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642575+0000","last_change":"2024-02-23T00:42:26.307705+0000","last_active":"2024-02-23T00:42:46.642575+0000","last_peered":"2024-02-23T00:42:46.642575+0000","last_clean":"2024-02-23T00:42:46.642575+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:42:46.642575+0000","last_undegraded":"2024-02-23T00:42:46.642575+0000","last_fullsized":"2024-02-23T00:42:46.642575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.307631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.307631+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.307631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":688,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397317+0000","last_change":"2024-02-23T00:41:47.254406+0000","last_active":"2024-02-23T00:42:46.397317+0000","last_peered":"2024-02-23T00:42:46.397317+0000","last_clean":"2024-02-23T00:42:46.397317+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:42:46.397317+0000","last_undegraded":"2024-02-23T00:42:46.397317+0000","last_fullsized":"2024-02-23T00:42:46.397317+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.254328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.254328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":877,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.393583+0000","last_change":"2024-02-23T00:42:27.911692+0000","last_active":"2024-02-23T00:42:46.393583+0000","last_peered":"2024-02-23T00:42:46.393583+0000","last_clean":"2024-02-23T00:42:46.393583+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:42:46.393583+0000","last_undegraded":"2024-02-23T00:42:46.393583+0000","last_fullsized":"2024-02-23T00:42:46.393583+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:42:27.911611+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.911611+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":267,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022917+0000","last_change":"2024-02-23T00:41:37.241505+0000","last_active":"2024-02-23T00:42:47.022917+0000","last_peered":"2024-02-23T00:42:47.022917+0000","last_clean":"2024-02-23T00:42:47.022917+0000","last_became_active":"2024-02-23T00:41:37.241142+0000","last_became_peered":"2024-02-23T00:41:37.241142+0000","last_unstale":"2024-02-23T00:42:47.022917+0000","last_undegraded":"2024-02-23T00:42:47.022917+0000","last_fullsized":"2024-02-23T00:42:47.022917+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:39:31.346259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:39:31.346259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":378,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390395+0000","last_change":"2024-02-23T00:40:52.433576+0000","last_active":"2024-02-23T00:42:46.390395+0000","last_peered":"2024-02-23T00:42:46.390395+0000","last_clean":"2024-02-23T00:42:46.390395+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:42:46.390395+0000","last_undegraded":"2024-02-23T00:42:46.390395+0000","last_fullsized":"2024-02-23T00:42:46.390395+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.433499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.433499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":156,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.642242+0000","last_change":"2024-02-23T00:41:34.730482+0000","last_active":"2024-02-23T00:42:46.642242+0000","last_peered":"2024-02-23T00:42:46.642242+0000","last_clean":"2024-02-23T00:42:46.642242+0000","last_became_active":"2024-02-23T00:40:23.199068+0000","last_became_peered":"2024-02-23T00:40:23.199068+0000","last_unstale":"2024-02-23T00:42:46.642242+0000","last_undegraded":"2024-02-23T00:42:46.642242+0000","last_fullsized":"2024-02-23T00:42:46.642242+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:34.730403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.730403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":1023,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:47.022485+0000","last_change":"2024-02-23T00:41:41.223239+0000","last_active":"2024-02-23T00:42:47.022485+0000","last_peered":"2024-02-23T00:42:47.022485+0000","last_clean":"2024-02-23T00:42:47.022485+0000","last_became_active":"2024-02-23T00:41:41.222892+0000","last_became_peered":"2024-02-23T00:41:41.222892+0000","last_unstale":"2024-02-23T00:42:47.022485+0000","last_undegraded":"2024-02-23T00:42:47.022485+0000","last_fullsized":"2024-02-23T00:42:47.022485+0000","mapping_epoch":681,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:41:34.076323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.076323+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":81,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.391124+0000","last_change":"2024-02-23T00:42:28.612255+0000","last_active":"2024-02-23T00:42:46.391124+0000","last_peered":"2024-02-23T00:42:46.391124+0000","last_clean":"2024-02-23T00:42:46.391124+0000","last_became_active":"2024-02-23T00:41:37.234830+0000","last_became_peered":"2024-02-23T00:41:37.234830+0000","last_unstale":"2024-02-23T00:42:46.391124+0000","last_undegraded":"2024-02-23T00:42:46.391124+0000","last_fullsized":"2024-02-23T00:42:46.391124+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:28.612113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:28.612113+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.612113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":74,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.390568+0000","last_change":"2024-02-23T00:42:13.611259+0000","last_active":"2024-02-23T00:42:46.390568+0000","last_peered":"2024-02-23T00:42:46.390568+0000","last_clean":"2024-02-23T00:42:46.390568+0000","last_became_active":"2024-02-23T00:41:50.253284+0000","last_became_peered":"2024-02-23T00:41:50.253284+0000","last_unstale":"2024-02-23T00:42:46.390568+0000","last_undegraded":"2024-02-23T00:42:46.390568+0000","last_fullsized":"2024-02-23T00:42:46.390568+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:13.611193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.611193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":687,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-02-23T00:42:46.397969+0000","last_change":"2024-02-23T00:42:20.220171+0000","last_active":"2024-02-23T00:42:46.397969+0000","last_peered":"2024-02-23T00:42:46.397969+0000","last_clean":"2024-02-23T00:42:46.397969+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:42:46.397969+0000","last_undegraded":"2024-02-23T00:42:46.397969+0000","last_fullsized":"2024-02-23T00:42:46.397969+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.220094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.220094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":249,"num_pg":32,"stat_sum":{"num_bytes":41044,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":84,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":98472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4000089,"num_objects":17,"num_object_clones":1,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":37,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6193152,"data_stored":6193152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":24,"acting":24,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":12,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3757,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":873,"num_read_kb":709,"num_write":524,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":4401,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":732,"ondisk_log_size":732,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":23,"stat_sum":{"num_bytes":438,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":620,"num_read_kb":620,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":69,"acting":69,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332286,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303392,"kb_used_data":6168,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441760,"statfs":{"total":95995035648,"available":95684362240,"internally_reserved":0,"allocated":6316032,"data_stored":3342966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.38200000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46100000000000002,"15min":0.67300000000000004},"min":{"1min":0.34300000000000003,"5min":0.23999999999999999,"15min":0.217},"max":{"1min":0.625,"5min":1.23,"15min":31.175000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.65600000000000003,"15min":0.73699999999999999},"min":{"1min":0.35299999999999998,"5min":0.28599999999999998,"15min":0.249},"max":{"1min":0.63500000000000001,"5min":11.066000000000001,"15min":29.423999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.68999999999999995,"15min":0.71699999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28100000000000003},"max":{"1min":0.64000000000000001,"5min":8.0099999999999998,"15min":16.603999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.59499999999999997,"15min":0.63800000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.26600000000000001},"max":{"1min":0.70599999999999996,"5min":5.7229999999999999,"15min":7.5339999999999998},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.61099999999999999,"15min":1.0069999999999999},"min":{"1min":0.36499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61699999999999999,"5min":8.0449999999999999,"15min":57.729999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.91300000000000003},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67900000000000005,"5min":3.8479999999999999,"15min":27.573},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.41699999999999998},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.69599999999999995,"5min":1.2589999999999999,"15min":1.706},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.67300000000000004,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.33400000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":1.0209999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.45000000000000001,"15min":0.46400000000000002},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59699999999999998,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.45600000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46200000000000002,"15min":0.48499999999999999},"min":{"1min":0.30399999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.65100000000000002,"5min":1.3129999999999999,"15min":2.1339999999999999},"last":0.497},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51100000000000001,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80500000000000005,"5min":5.194,"15min":5.194},"last":0.51900000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561219,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307500,"kb_used_data":6180,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437652,"statfs":{"total":95995035648,"available":95680155648,"internally_reserved":0,"allocated":6328320,"data_stored":3367104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.59099999999999997}]},{"osd":1,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53400000000000003,"15min":1.024},"min":{"1min":0.40100000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.66100000000000003,"5min":6.056,"15min":59.347999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.75900000000000001,"15min":1.3799999999999999},"min":{"1min":0.41299999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.1619999999999999,"5min":18.423999999999999,"15min":87.221000000000004},"last":0.51800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.77300000000000002,"15min":0.90500000000000003},"min":{"1min":0.35699999999999998,"5min":0.214,"15min":0.214},"max":{"1min":1.1120000000000001,"5min":18.381,"15min":25.550999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66100000000000003,"15min":0.91700000000000004},"min":{"1min":0.375,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.9460000000000002,"5min":6.1749999999999998,"15min":40.914000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63800000000000001,"15min":0.85799999999999998},"min":{"1min":0.40999999999999998,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":2.859,"5min":7.1630000000000003,"15min":31.309999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.754,"15min":1.4059999999999999},"min":{"1min":0.44600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72299999999999998,"5min":18.448,"15min":86.311999999999998},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.53500000000000003,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60499999999999998,"5min":7.8769999999999998,"15min":7.8769999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48199999999999998,"15min":0.53000000000000003},"min":{"1min":0.307,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":6.0300000000000002,"15min":17.718},"last":0.41399999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.54900000000000004,"15min":0.52200000000000002},"min":{"1min":0.30599999999999999,"5min":0.27800000000000002,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":6.7999999999999998,"15min":8.0990000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.505},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82599999999999996,"5min":6.0780000000000003,"15min":8.1370000000000005},"last":0.46200000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54000000000000004,"15min":0.61899999999999999},"min":{"1min":0.32100000000000001,"5min":0.28999999999999998,"15min":0.27300000000000002},"max":{"1min":0.70799999999999996,"5min":6.4279999999999999,"15min":17.881},"last":0.443},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.26500000000000001},"max":{"1min":0.74399999999999999,"5min":18.475000000000001,"15min":18.475000000000001},"last":0.42599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575066,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309540,"kb_used_data":8220,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93435612,"statfs":{"total":95995035648,"available":95678066688,"internally_reserved":0,"allocated":8417280,"data_stored":5403371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.38}]},{"osd":2,"last update":"Fri Feb 23 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36699999999999999,"15min":0.66700000000000004},"min":{"1min":0.20799999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.54000000000000004,"5min":3.9980000000000002,"15min":35.313000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.39700000000000002,"15min":0.57899999999999996},"min":{"1min":0.27300000000000002,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.623,"5min":2.7999999999999998,"15min":27.908999999999999},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:42:05 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.435,"15min":0.64900000000000002},"min":{"1min":0.26200000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.6279999999999999,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40600000000000003,"15min":0.63900000000000001},"min":{"1min":0.26000000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.71,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.311}]},{"osd":4,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58399999999999996,"15min":0.70699999999999996},"min":{"1min":0.39000000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.877,"5min":3.4900000000000002,"15min":16.231000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.52700000000000002,"15min":0.71599999999999997},"min":{"1min":0.32200000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":29.212},"last":0.60999999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.71999999999999997},"min":{"1min":0.37,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76400000000000001,"5min":2.8410000000000002,"15min":16.294},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.58599999999999997,"15min":0.68500000000000005},"min":{"1min":0.39700000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.238,"5min":4.3380000000000001,"15min":11.105},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.53600000000000003,"15min":0.754},"min":{"1min":0.39500000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.76,"5min":2.4049999999999998,"15min":29.187000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60999999999999999,"15min":0.749},"min":{"1min":0.44,"5min":0.312,"15min":0.312},"max":{"1min":0.90500000000000003,"5min":2.927,"15min":16.369},"last":0.72099999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.63500000000000001,"15min":0.78800000000000003},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.32000000000000001},"max":{"1min":0.82099999999999995,"5min":5.2119999999999997,"15min":16.395},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.622,"15min":0.66600000000000004},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81899999999999995,"5min":3.7669999999999999,"15min":11.135999999999999},"last":0.66300000000000003}]}]},{"osd":0,"up_from":659,"seq":2830383448083,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309104,"kb_used_data":6184,"kb_used_omap":85,"kb_used_meta":302826,"kb_avail":93436048,"statfs":{"total":95995035648,"available":95678513152,"internally_reserved":0,"allocated":6332416,"data_stored":3357840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":310093952},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.0880000000000001,"5min":5.0880000000000001,"15min":5.0880000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.53200000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.0540000000000003,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":11.891,"5min":11.891,"15min":11.891},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":15.864000000000001,"5min":15.864000000000001,"15min":15.864000000000001},"last":0.48699999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.65600000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.83699999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.628}]}]},{"osd":2,"up_from":18,"seq":77309411542,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315664,"kb_used_data":6152,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93429488,"statfs":{"total":95995035648,"available":95671795712,"internally_reserved":0,"allocated":6299648,"data_stored":3348271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.34599999999999997}]},{"osd":1,"last update":"Fri Feb 23 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34300000000000003,"15min":0.59399999999999997},"min":{"1min":0.23300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.84399999999999997,"5min":1.8919999999999999,"15min":19.324999999999999},"last":0.379},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.47399999999999998,"15min":0.752},"min":{"1min":0.251,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68500000000000005,"5min":7.9779999999999998,"15min":35.052},"last":0.38700000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.501,"15min":0.73199999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.46999999999999997,"5min":10.032,"15min":43.5},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.53000000000000003,"15min":0.68400000000000005},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.5,"5min":11.815,"15min":27.414000000000001},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.52100000000000002,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.81499999999999995,"5min":2.0899999999999999,"15min":14.614000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53500000000000003,"15min":0.621},"min":{"1min":0.39400000000000002,"5min":0.30199999999999999,"15min":0.27600000000000002},"max":{"1min":0.89500000000000002,"5min":1.9750000000000001,"15min":7.766},"last":0.55000000000000004}]},{"osd":5,"last update":"Fri Feb 23 00:42:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.68400000000000005,"15min":0.71399999999999997},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.316},"max":{"1min":1.248,"5min":7.915,"15min":17.765000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.64800000000000002,"15min":0.67200000000000004},"min":{"1min":0.371,"5min":0.312,"15min":0.312},"max":{"1min":0.83899999999999997,"5min":10.138,"15min":14.589},"last":0.57399999999999995}]},{"osd":6,"last update":"Fri Feb 23 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.628},"min":{"1min":0.42399999999999999,"5min":0.316,"15min":0.29699999999999999},"max":{"1min":0.81100000000000005,"5min":2.2250000000000001,"15min":7.7869999999999999},"last":0.62},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71999999999999997,"15min":0.79300000000000004},"min":{"1min":0.41799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.218,"5min":7.9589999999999996,"15min":20.574999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.70499999999999996,"15min":1.073},"min":{"1min":0.38700000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.81599999999999995,"5min":7.9480000000000004,"15min":66.632000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58199999999999996,"15min":0.84799999999999998},"min":{"1min":0.44800000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78800000000000003,"5min":2.1789999999999998,"15min":19.603999999999999},"last":0.68600000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149906,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313576,"kb_used_data":8160,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431576,"statfs":{"total":95995035648,"available":95673933824,"internally_reserved":0,"allocated":8355840,"data_stored":5355541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.26100000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:42:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.60899999999999999},"min":{"1min":0.21299999999999999,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":3.2970000000000002,"5min":3.2970000000000002,"15min":54.351999999999997},"last":0.248},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.45000000000000001,"15min":0.78100000000000003},"min":{"1min":0.252,"5min":0.219,"15min":0.219},"max":{"1min":2.1680000000000001,"5min":3.7120000000000002,"15min":55.337000000000003},"last":0.376}]},{"osd":2,"last update":"Fri Feb 23 00:42:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.442,"15min":0.75800000000000001},"min":{"1min":0.27000000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.70199999999999996,"5min":3.7570000000000001,"15min":55.368000000000002},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.442,"15min":0.73199999999999998},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.185},"max":{"1min":0.52800000000000002,"5min":4.181,"15min":54.383000000000003},"last":0.29499999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.627,"15min":0.93300000000000005},"min":{"1min":0.36499999999999999,"5min":0.34000000000000002,"15min":0.27900000000000003},"max":{"1min":0.72899999999999998,"5min":6.8799999999999999,"15min":54.530000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.314,"5min":0.30199999999999999,"15min":0.24099999999999999},"max":{"1min":5.8109999999999999,"5min":5.8109999999999999,"15min":8.0359999999999996},"last":0.502}]},{"osd":5,"last update":"Fri Feb 23 00:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54600000000000004,"15min":0.876},"min":{"1min":0.372,"5min":0.34699999999999998,"15min":0.28899999999999998},"max":{"1min":0.80300000000000005,"5min":3.8180000000000001,"15min":55.401000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.65700000000000003,"15min":0.86499999999999999},"min":{"1min":0.40000000000000002,"5min":0.32700000000000001,"15min":0.30299999999999999},"max":{"1min":0.75700000000000001,"5min":7.0179999999999998,"15min":54.454999999999998},"last":0.54900000000000004}]},{"osd":6,"last update":"Fri Feb 23 00:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.76800000000000002,"15min":0.85399999999999998},"min":{"1min":0.42999999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.7069999999999999,"5min":9.7490000000000006,"15min":54.305},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.86599999999999999,"15min":1.0389999999999999},"min":{"1min":0.374,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":8.5860000000000003,"5min":9.6820000000000004,"15min":55.424999999999997},"last":0.56200000000000006}]},{"osd":7,"last update":"Fri Feb 23 00:41:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.627,"15min":0.78800000000000003},"min":{"1min":0.437,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81699999999999995,"5min":4.9790000000000001,"15min":54.222000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.92800000000000005,"15min":1.028},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76500000000000001,"5min":23.928000000000001,"15min":55.460000000000001},"last":0.59799999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986381,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311612,"kb_used_data":6196,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433540,"statfs":{"total":95995035648,"available":95675944960,"internally_reserved":0,"allocated":6344704,"data_stored":3367957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.68300000000000005}]},{"osd":1,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.69299999999999995,"15min":1.1020000000000001},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.64000000000000001,"5min":8.1199999999999992,"15min":41.137999999999998},"last":0.748},{"interface":"front","average":{"1min":0.502,"5min":0.65200000000000002,"15min":0.93000000000000005},"min":{"1min":0.40600000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":10.952999999999999,"15min":26.210000000000001},"last":0.58699999999999997}]},{"osd":2,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.72199999999999998,"15min":1.1910000000000001},"min":{"1min":0.36699999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64100000000000001,"5min":9.3490000000000002,"15min":67.253},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.83099999999999996,"15min":1.1240000000000001},"min":{"1min":0.374,"5min":0.251,"15min":0.251},"max":{"1min":0.67100000000000004,"5min":14.442,"15min":41.927999999999997},"last":0.76700000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.68999999999999995,"15min":1.1240000000000001},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.627,"5min":9.5579999999999998,"15min":36.301000000000002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.63100000000000001,"15min":1.1970000000000001},"min":{"1min":0.38200000000000001,"5min":0.26700000000000002,"15min":0.24299999999999999},"max":{"1min":0.69499999999999995,"5min":5.1879999999999997,"15min":42.064},"last":0.65000000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:41:52 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38900000000000001,"15min":0.56599999999999995},"min":{"1min":0.26900000000000002,"5min":0.219,"15min":0.217},"max":{"1min":0.58499999999999996,"5min":1.4279999999999999,"15min":14.242000000000001},"last":0.441},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.435,"15min":0.60899999999999999},"min":{"1min":0.30499999999999999,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":0.63700000000000001,"5min":1.4079999999999999,"15min":14.124000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46700000000000003,"15min":0.57599999999999996},"min":{"1min":0.315,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":0.67000000000000004,"5min":1.734,"15min":8.0730000000000004},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.54000000000000004,"15min":0.67300000000000004},"min":{"1min":0.30299999999999999,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.61599999999999999,"5min":13.339,"15min":45.814999999999998},"last":0.54800000000000004}]},{"osd":7,"last update":"Fri Feb 23 00:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.64300000000000002},"min":{"1min":0.32900000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5129999999999999,"5min":7.9210000000000003,"15min":17.178999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56599999999999995,"15min":0.71899999999999997},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.28299999999999997},"max":{"1min":0.63800000000000001,"5min":6.3220000000000001,"15min":15.85},"last":0.72799999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757448,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305348,"kb_used_data":8124,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439804,"statfs":{"total":95995035648,"available":95682359296,"internally_reserved":0,"allocated":8318976,"data_stored":5356509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.439}]},{"osd":1,"last update":"Fri Feb 23 00:41:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47099999999999997,"15min":0.751},"min":{"1min":0.31900000000000001,"5min":0.26200000000000001,"15min":0.24099999999999999},"max":{"1min":1.1180000000000001,"5min":3.7629999999999999,"15min":38.838000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.50800000000000001,"15min":1.1759999999999999},"min":{"1min":0.26600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60399999999999998,"5min":3.7250000000000001,"15min":127.794},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Feb 23 00:41:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.70799999999999996,"15min":0.95599999999999996},"min":{"1min":0.32400000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.2969999999999999,"5min":11.435,"15min":37.442999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.48799999999999999,"15min":0.69599999999999995},"min":{"1min":0.28999999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.67500000000000004,"5min":3.734,"15min":23.297999999999998},"last":0.53400000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:41:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.74299999999999999,"15min":0.82499999999999996},"min":{"1min":0.375,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.3300000000000001,"5min":12.507,"15min":24.414999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.90700000000000003},"min":{"1min":0.36699999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.075,"5min":3.8119999999999998,"15min":40.865000000000002},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:41:53 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.376,"15min":0.78100000000000003},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.219},"max":{"1min":0.69099999999999995,"5min":2.8940000000000001,"15min":103.13},"last":0.499},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.27200000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.66200000000000003,"5min":11.381,"15min":12.795999999999999},"last":0.55500000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.56899999999999995,"15min":0.54800000000000004},"min":{"1min":0.32200000000000001,"5min":0.26400000000000001,"15min":0.26200000000000001},"max":{"1min":0.70799999999999996,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.82299999999999995},"min":{"1min":0.307,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.75,"5min":1.5229999999999999,"15min":103.61499999999999},"last":0.64100000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70299999999999996,"5min":11.478,"15min":11.478},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.501,"5min":0.46899999999999997,"15min":0.52000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.25800000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":5.4050000000000002},"last":0.65100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2039808,"data_stored":2039808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":2027520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":2027520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":41044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":41044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":249,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:42:52.859 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-02-23T00:42:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:52.861 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:42:52.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:42:52.962 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:52.989 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:53.007 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:53.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:53.111 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:53.140 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:53.641 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:53.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:51.789889+0000 mgr.y (mgr.24437) 633 : cluster [DBG] pgmap v1138: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.4 KiB/s wr, 2 op/s; 1/750 objects misplaced (0.133%) 2024-02-23T00:42:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:51.832990+0000 mgr.y (mgr.24437) 634 : audit [DBG] from='client.38294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:42:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:52.401596+0000 mon.a (mon.0) 3728 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:52.412979+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? 172.21.15.87:0/902657941' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:42:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:52.413016+0000 mon.a (mon.0) 3730 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-02-23T00:42:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:52.535358+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:42:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:52.538250+0000 osd.6 (osd.6) 299 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:42:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:52.935344+0000 osd.4 (osd.4) 327 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:42:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:52.938880+0000 osd.4 (osd.4) 328 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:42:53.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:53.770 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:51.789889+0000 mgr.y (mgr.24437) 633 : cluster [DBG] pgmap v1138: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.4 KiB/s wr, 2 op/s; 1/750 objects misplaced (0.133%) 2024-02-23T00:42:53.770 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:51.832990+0000 mgr.y (mgr.24437) 634 : audit [DBG] from='client.38294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:42:53.770 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:52.401596+0000 mon.a (mon.0) 3728 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:53.770 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:52.412979+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? 172.21.15.87:0/902657941' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:42:53.770 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:52.413016+0000 mon.a (mon.0) 3730 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:52.535358+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:52.538250+0000 osd.6 (osd.6) 299 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:52.935344+0000 osd.4 (osd.4) 327 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:52.938880+0000 osd.4 (osd.4) 328 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:51.789889+0000 mgr.y (mgr.24437) 633 : cluster [DBG] pgmap v1138: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.4 KiB/s wr, 2 op/s; 1/750 objects misplaced (0.133%) 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:51.832990+0000 mgr.y (mgr.24437) 634 : audit [DBG] from='client.38294 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:52.401596+0000 mon.a (mon.0) 3728 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:52.412979+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? 172.21.15.87:0/902657941' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:52.413016+0000 mon.a (mon.0) 3730 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:52.535358+0000 osd.6 (osd.6) 298 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:42:53.771 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:52.538250+0000 osd.6 (osd.6) 299 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:42:53.772 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:52.935344+0000 osd.4 (osd.4) 327 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:42:53.772 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:52.938880+0000 osd.4 (osd.4) 328 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:42:53.772 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:53.971 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:53.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:53.982 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-23T00:42:53.990 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:54.077 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:54.105 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:54.193 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:42:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:42:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:42:54.304 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:54.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:54.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:54.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:54.421 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.260533+0000 osd.3 (osd.3) 232 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:42:54.421 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.263422+0000 osd.3 (osd.3) 233 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:42:54.421 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.369449+0000 osd.1 (osd.1) 334 : cluster [DBG] 22.1c scrub starts 2024-02-23T00:42:54.421 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.372419+0000 osd.1 (osd.1) 335 : cluster [DBG] 22.1c scrub ok 2024-02-23T00:42:54.421 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.385870+0000 osd.2 (osd.2) 267 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:42:54.421 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.387244+0000 osd.2 (osd.2) 268 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:42:54.422 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.557182+0000 osd.0 (osd.0) 31 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:42:54.422 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.560865+0000 osd.0 (osd.0) 32 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:42:54.422 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.641193+0000 osd.7 (osd.7) 332 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:42:54.422 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.644083+0000 osd.7 (osd.7) 333 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:42:54.422 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.790401+0000 mgr.y (mgr.24437) 635 : cluster [DBG] pgmap v1140: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 553 B/s rd, 5.5 KiB/s wr, 1 op/s; 1/750 objects misplaced (0.133%) 2024-02-23T00:42:54.422 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.958820+0000 osd.4 (osd.4) 329 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:42:54.422 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:53.959780+0000 osd.4 (osd.4) 330 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:42:54.422 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:54.045816+0000 osd.5 (osd.5) 285 : cluster [DBG] 6.a scrub starts 2024-02-23T00:42:54.422 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:54.046899+0000 osd.5 (osd.5) 286 : cluster [DBG] 6.a scrub ok 2024-02-23T00:42:54.439 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:54.537 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:54.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:54.641 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:54.670 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.260533+0000 osd.3 (osd.3) 232 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:42:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.263422+0000 osd.3 (osd.3) 233 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:42:54.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.369449+0000 osd.1 (osd.1) 334 : cluster [DBG] 22.1c scrub starts 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.372419+0000 osd.1 (osd.1) 335 : cluster [DBG] 22.1c scrub ok 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.385870+0000 osd.2 (osd.2) 267 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.387244+0000 osd.2 (osd.2) 268 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.557182+0000 osd.0 (osd.0) 31 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.560865+0000 osd.0 (osd.0) 32 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.641193+0000 osd.7 (osd.7) 332 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.644083+0000 osd.7 (osd.7) 333 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.790401+0000 mgr.y (mgr.24437) 635 : cluster [DBG] pgmap v1140: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 553 B/s rd, 5.5 KiB/s wr, 1 op/s; 1/750 objects misplaced (0.133%) 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.958820+0000 osd.4 (osd.4) 329 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:53.959780+0000 osd.4 (osd.4) 330 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:54.045816+0000 osd.5 (osd.5) 285 : cluster [DBG] 6.a scrub starts 2024-02-23T00:42:54.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:54.046899+0000 osd.5 (osd.5) 286 : cluster [DBG] 6.a scrub ok 2024-02-23T00:42:54.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.260533+0000 osd.3 (osd.3) 232 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:42:54.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.263422+0000 osd.3 (osd.3) 233 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:42:54.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.369449+0000 osd.1 (osd.1) 334 : cluster [DBG] 22.1c scrub starts 2024-02-23T00:42:54.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.372419+0000 osd.1 (osd.1) 335 : cluster [DBG] 22.1c scrub ok 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.385870+0000 osd.2 (osd.2) 267 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.387244+0000 osd.2 (osd.2) 268 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.557182+0000 osd.0 (osd.0) 31 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.560865+0000 osd.0 (osd.0) 32 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.641193+0000 osd.7 (osd.7) 332 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.644083+0000 osd.7 (osd.7) 333 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.790401+0000 mgr.y (mgr.24437) 635 : cluster [DBG] pgmap v1140: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 553 B/s rd, 5.5 KiB/s wr, 1 op/s; 1/750 objects misplaced (0.133%) 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.958820+0000 osd.4 (osd.4) 329 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:53.959780+0000 osd.4 (osd.4) 330 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:54.045816+0000 osd.5 (osd.5) 285 : cluster [DBG] 6.a scrub starts 2024-02-23T00:42:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:54.046899+0000 osd.5 (osd.5) 286 : cluster [DBG] 6.a scrub ok 2024-02-23T00:42:54.868 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:54.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:54.972 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:55.005 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:55.026 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:55.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:55.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:55.130 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:55.161 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:55.181 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:55.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:55.285 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:55.313 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:55.390 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:55.411 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:55.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:55.516 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:55.543 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:54.351371+0000 osd.1 (osd.1) 336 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:42:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:54.354841+0000 osd.1 (osd.1) 337 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:42:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:54.595951+0000 osd.6 (osd.6) 300 : cluster [DBG] 3.a scrub starts 2024-02-23T00:42:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:54.599066+0000 osd.6 (osd.6) 301 : cluster [DBG] 3.a scrub ok 2024-02-23T00:42:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:54.630351+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:42:55.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:54.633809+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:42:55.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:54.351371+0000 osd.1 (osd.1) 336 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:42:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:54.354841+0000 osd.1 (osd.1) 337 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:42:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:54.595951+0000 osd.6 (osd.6) 300 : cluster [DBG] 3.a scrub starts 2024-02-23T00:42:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:54.599066+0000 osd.6 (osd.6) 301 : cluster [DBG] 3.a scrub ok 2024-02-23T00:42:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:54.630351+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:42:55.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:54.633809+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:42:55.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:42:55.685 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:55.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:54.351371+0000 osd.1 (osd.1) 336 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:42:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:54.354841+0000 osd.1 (osd.1) 337 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:42:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:54.595951+0000 osd.6 (osd.6) 300 : cluster [DBG] 3.a scrub starts 2024-02-23T00:42:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:54.599066+0000 osd.6 (osd.6) 301 : cluster [DBG] 3.a scrub ok 2024-02-23T00:42:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:54.630351+0000 osd.7 (osd.7) 334 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:42:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:54.633809+0000 osd.7 (osd.7) 335 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:42:55.789 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:55.821 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:55.842 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:55.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:55.946 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:55.975 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:56.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.259+0000 7fb9d1044700 1 -- 172.21.15.87:0/676223264 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 msgr2=0x7fb9cc104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.259+0000 7fb9d1044700 1 --2- 172.21.15.87:0/676223264 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 0x7fb9cc104570 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 rx=0x7fb9c4009c30 tx=0x7fb9c4004c50).stop 2024-02-23T00:42:56.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.259+0000 7fb9d1044700 1 -- 172.21.15.87:0/676223264 shutdown_connections 2024-02-23T00:42:56.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.259+0000 7fb9d1044700 1 --2- 172.21.15.87:0/676223264 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 0x7fb9cc104570 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.259+0000 7fb9d1044700 1 --2- 172.21.15.87:0/676223264 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9cc0f8c20 0x7fb9cc0f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.259+0000 7fb9d1044700 1 --2- 172.21.15.87:0/676223264 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9cc0f9580 0x7fb9cc101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.259+0000 7fb9d1044700 1 -- 172.21.15.87:0/676223264 >> 172.21.15.87:0/676223264 conn(0x7fb9cc0f26f0 msgr2=0x7fb9cc0f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:56.326 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.259+0000 7fb9d1044700 1 -- 172.21.15.87:0/676223264 shutdown_connections 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.259+0000 7fb9d1044700 1 -- 172.21.15.87:0/676223264 wait complete. 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 librados: starting msgr at 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 librados: starting objecter 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 Processor -- start 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 -- start start 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 librados: setting wanted keys 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 librados: calling monclient init 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9cc0f8c20 0x7fb9cc10df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9cc0f9580 0x7fb9cc10e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 0x7fb9cc108170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.260+0000 7fb9d1044700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc108660 con 0x7fb9cc0f9580 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9d1044700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc1087d0 con 0x7fb9cc0f8c20 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9d1044700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc108990 con 0x7fb9cc102180 2024-02-23T00:42:56.327 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9ca59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9cc0f9580 0x7fb9cc10e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9cb59e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 0x7fb9cc108170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9ca59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9cc0f9580 0x7fb9cc10e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:53006/0 (socket says 172.21.15.87:53006) 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9cb59e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 0x7fb9cc108170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:47176/0 (socket says 172.21.15.87:47176) 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9ca59c700 1 -- 172.21.15.87:0/945354192 learned_addr learned my addr 172.21.15.87:0/945354192 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9cad9d700 1 --2- 172.21.15.87:0/945354192 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9cc0f8c20 0x7fb9cc10df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9cb59e700 1 -- 172.21.15.87:0/945354192 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9cc0f8c20 msgr2=0x7fb9cc10df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9cb59e700 1 --2- 172.21.15.87:0/945354192 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb9cc0f8c20 0x7fb9cc10df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9cb59e700 1 -- 172.21.15.87:0/945354192 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9cc0f9580 msgr2=0x7fb9cc10e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9cb59e700 1 --2- 172.21.15.87:0/945354192 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9cc0f9580 0x7fb9cc10e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9cb59e700 1 -- 172.21.15.87:0/945354192 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c0005040 con 0x7fb9cc102180 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9ca59c700 1 --2- 172.21.15.87:0/945354192 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb9cc0f9580 0x7fb9cc10e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9cb59e700 1 --2- 172.21.15.87:0/945354192 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 0x7fb9cc108170 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7fb9c400b600 tx=0x7fb9c4004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9b3fff700 10 client.?.objecter ms_handle_connect 0x7fb9cc102180 2024-02-23T00:42:56.328 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9b3fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c400e040 con 0x7fb9cc102180 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c40097c0 con 0x7fb9cc102180 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9c4018040 con 0x7fb9cc102180 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c401c3f0 con 0x7fb9cc102180 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9d1044700 10 client.38303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc108e20 con 0x7fb9cc102180 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.261+0000 7fb9d1044700 1 librados: init done 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.262+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb9cc05f990 con 0x7fb9cc102180 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.262+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb9c4007bf0 con 0x7fb9cc102180 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9b3fff700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb9b4062c90 0x7fb9b4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(746..746 src has 400..746) v4 ==== 8564+0+0 (secure 0 0 0) 0x7fb9c407e530 con 0x7fb9cc102180 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 osd_map(746..746 src has 400..746) v4 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9b3fff700 3 client.38303.objecter handle_osd_map got epochs [746,746] > 0 2024-02-23T00:42:56.329 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9b3fff700 3 client.38303.objecter handle_osd_map decoding full epoch 746 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9cad9d700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb9b4062c90 0x7fb9b4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9b3fff700 20 client.38303.objecter dump_active .. 0 homeless 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v746) v1 ==== 54+0+16 (secure 0 0 0) 0x7fb9c4085070 con 0x7fb9cc102180 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9b3fff700 10 client.38303.objecter ms_handle_connect 0x7fb9b4062c90 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9cad9d700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb9b4062c90 0x7fb9b4065150 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 rx=0x7fb9bc0041d0 tx=0x7fb9bc009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 10 client.38303.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 1 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6801/484081831 conn(0x7fb9cc0fc110 0x7fb9cc0fc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc109010 osd=0 3 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc109010 osd=0 3 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc109010 osd=0 3 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 1 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc109010 osd=0 3 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 0 1 2024-02-23T00:42:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc109010 osd=0 4 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 10 client.38303.objecter _send_command 1 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9cc05ee80 con 0x7fb9cc0fc110 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.263+0000 7fb9ca59c700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6801/484081831 conn(0x7fb9cc0fc110 0x7fb9cc0fc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.264+0000 7fb9b3fff700 10 client.38303.objecter ms_handle_connect 0x7fb9cc0fc110 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.264+0000 7fb9ca59c700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6801/484081831 conn(0x7fb9cc0fc110 0x7fb9cc0fc4f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.264+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb9cc05ee80 con 0x7fb9cc0fc110 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.264+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.264+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 1 = system:0 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.264+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc109010 osd=0 4 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.264+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 0 1 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 10 client.38303.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 2 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc109010 osd=0 3 2024-02-23T00:42:56.331 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc109010 osd=0 3 2024-02-23T00:42:56.332 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:42:56.332 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:42:56.332 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:42:56.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc109010 osd=0 3 2024-02-23T00:42:56.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 2 2024-02-23T00:42:56.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc109010 osd=0 3 2024-02-23T00:42:56.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 0 2 2024-02-23T00:42:56.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc109010 osd=0 4 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 10 client.38303.objecter _send_command 2 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.279+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb9cc107160 con 0x7fb9cc0fc110 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.280+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb9cc107160 con 0x7fb9cc0fc110 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.280+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.280+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.280+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc109010 osd=0 4 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.280+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 0 2 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 10 client.38303.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 3 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6805/685354171 conn(0x7fb9cc0598f0 0x7fb9cc059cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc028540 osd=1 3 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc028540 osd=1 3 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc028540 osd=1 3 2024-02-23T00:42:56.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 3 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc028540 osd=1 3 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 1 3 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc028540 osd=1 4 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 10 client.38303.objecter _send_command 3 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9cc0286b0 con 0x7fb9cc0598f0 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9cad9d700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6805/685354171 conn(0x7fb9cc0598f0 0x7fb9cc059cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9b3fff700 10 client.38303.objecter ms_handle_connect 0x7fb9cc0598f0 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.281+0000 7fb9cad9d700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6805/685354171 conn(0x7fb9cc0598f0 0x7fb9cc059cf0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.282+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb9cc0286b0 con 0x7fb9cc0598f0 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.282+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.282+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 3 = system:0 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.282+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc028540 osd=1 4 2024-02-23T00:42:56.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.282+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 1 3 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 10 client.38303.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 4 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc028540 osd=1 3 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc028540 osd=1 3 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc028540 osd=1 3 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 4 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc028540 osd=1 3 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 1 4 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc028540 osd=1 4 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 10 client.38303.objecter _send_command 4 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.296+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb9cc059360 con 0x7fb9cc0598f0 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb9cc059360 con 0x7fb9cc0598f0 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.335 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc028540 osd=1 4 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 1 4 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 10 client.38303.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 5 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb9cc119f50 0x7fb9cc11c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc027b90 osd=2 3 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027b90 osd=2 3 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc027b90 osd=2 3 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 5 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc027b90 osd=2 3 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 2 5 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027b90 osd=2 4 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 10 client.38303.objecter _send_command 5 2024-02-23T00:42:56.336 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9cc028210 con 0x7fb9cc119f50 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9ca59c700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb9cc119f50 0x7fb9cc11c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9b3fff700 10 client.38303.objecter ms_handle_connect 0x7fb9cc119f50 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.298+0000 7fb9ca59c700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb9cc119f50 0x7fb9cc11c340 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.299+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb9cc028210 con 0x7fb9cc119f50 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.299+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.299+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 5 = system:0 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.299+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc027b90 osd=2 4 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.299+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 2 5 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 10 client.38303.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 6 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc027b90 osd=2 3 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027b90 osd=2 3 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc027b90 osd=2 3 2024-02-23T00:42:56.337 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 6 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc027b90 osd=2 3 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 2 6 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027b90 osd=2 4 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 10 client.38303.objecter _send_command 6 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.313+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb9cc027600 con 0x7fb9cc119f50 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb9cc027600 con 0x7fb9cc119f50 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc027b90 osd=2 4 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 2 6 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 10 client.38303.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 7 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6813/128059790 conn(0x7fb9cc119ab0 0x7fb9cc11f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc027a10 osd=3 3 2024-02-23T00:42:56.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027a10 osd=3 3 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc027a10 osd=3 3 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 7 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc027a10 osd=3 3 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 3 7 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027a10 osd=3 4 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 10 client.38303.objecter _send_command 7 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9cc122610 con 0x7fb9cc119ab0 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9cb59e700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6813/128059790 conn(0x7fb9cc119ab0 0x7fb9cc11f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9b3fff700 10 client.38303.objecter ms_handle_connect 0x7fb9cc119ab0 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.315+0000 7fb9cb59e700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6813/128059790 conn(0x7fb9cc119ab0 0x7fb9cc11f9c0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.316+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb9cc122610 con 0x7fb9cc119ab0 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.316+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.316+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 7 = system:0 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.316+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc027a10 osd=3 4 2024-02-23T00:42:56.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.316+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 3 7 2024-02-23T00:42:56.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:56.341 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:56.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:56.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 10 client.38303.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:56.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 8 2024-02-23T00:42:56.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc027a10 osd=3 3 2024-02-23T00:42:56.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027a10 osd=3 3 2024-02-23T00:42:56.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc027a10 osd=3 3 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 8 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc027a10 osd=3 3 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 3 8 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027a10 osd=3 4 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 10 client.38303.objecter _send_command 8 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.331+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb9cc119010 con 0x7fb9cc119ab0 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.332+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb9cc119010 con 0x7fb9cc119ab0 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.332+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.332+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.332+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc027a10 osd=3 4 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.332+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 3 8 2024-02-23T00:42:56.342 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:42:56.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 10 client.38303.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:56.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 9 2024-02-23T00:42:56.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb9cc11d510 0x7fb9cc121ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc119010 osd=4 3 2024-02-23T00:42:56.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119010 osd=4 3 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc119010 osd=4 3 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 9 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc119010 osd=4 3 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 4 9 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119010 osd=4 4 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 10 client.38303.objecter _send_command 9 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9cc122470 con 0x7fb9cc11d510 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.333+0000 7fb9cad9d700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb9cc11d510 0x7fb9cc121ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.334+0000 7fb9b3fff700 10 client.38303.objecter ms_handle_connect 0x7fb9cc11d510 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.334+0000 7fb9cad9d700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb9cc11d510 0x7fb9cc121ec0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.334+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb9cc122470 con 0x7fb9cc11d510 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.334+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:42:56.344 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.334+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 9 = system:0 2024-02-23T00:42:56.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.334+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc119010 osd=4 4 2024-02-23T00:42:56.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.334+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 4 9 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 10 client.38303.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 10 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc119010 osd=4 3 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119010 osd=4 3 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc119010 osd=4 3 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 10 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc119010 osd=4 3 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 4 10 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119010 osd=4 4 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 10 client.38303.objecter _send_command 10 2024-02-23T00:42:56.352 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.349+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb9cc122470 con 0x7fb9cc11d510 2024-02-23T00:42:56.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.351+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb9cc122470 con 0x7fb9cc11d510 2024-02-23T00:42:56.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.351+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:56.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc119010 osd=4 4 2024-02-23T00:42:56.354 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 4 10 2024-02-23T00:42:56.354 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:42:56.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 10 client.38303.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:56.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 11 2024-02-23T00:42:56.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb9cc0f80a0 0x7fb9cc0f84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc122470 osd=5 3 2024-02-23T00:42:56.355 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc122470 osd=5 3 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc122470 osd=5 3 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 11 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc122470 osd=5 3 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 5 11 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc122470 osd=5 4 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 10 client.38303.objecter _send_command 11 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9cc119310 con 0x7fb9cc0f80a0 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.352+0000 7fb9ca59c700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb9cc0f80a0 0x7fb9cc0f84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.353+0000 7fb9b3fff700 10 client.38303.objecter ms_handle_connect 0x7fb9cc0f80a0 2024-02-23T00:42:56.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.353+0000 7fb9ca59c700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb9cc0f80a0 0x7fb9cc0f84e0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.353+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb9cc119310 con 0x7fb9cc0f80a0 2024-02-23T00:42:56.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.353+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:42:56.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.353+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 11 = system:0 2024-02-23T00:42:56.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.353+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc122470 osd=5 4 2024-02-23T00:42:56.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.353+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 5 11 2024-02-23T00:42:56.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 10 client.38303.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:56.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 12 2024-02-23T00:42:56.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc122470 osd=5 3 2024-02-23T00:42:56.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc122470 osd=5 3 2024-02-23T00:42:56.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc122470 osd=5 3 2024-02-23T00:42:56.370 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 12 2024-02-23T00:42:56.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc122470 osd=5 3 2024-02-23T00:42:56.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 5 12 2024-02-23T00:42:56.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc122470 osd=5 4 2024-02-23T00:42:56.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 10 client.38303.objecter _send_command 12 2024-02-23T00:42:56.371 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.368+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb9cc119310 con 0x7fb9cc0f80a0 2024-02-23T00:42:56.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb9cc119310 con 0x7fb9cc0f80a0 2024-02-23T00:42:56.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:56.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.372 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc122470 osd=5 4 2024-02-23T00:42:56.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 5 12 2024-02-23T00:42:56.373 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 10 client.38303.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 13 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb9cc1290b0 0x7fb9cc12b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc119310 osd=6 3 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119310 osd=6 3 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc119310 osd=6 3 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 13 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc119310 osd=6 3 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 6 13 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119310 osd=6 4 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 10 client.38303.objecter _send_command 13 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9cc12e1b0 con 0x7fb9cc1290b0 2024-02-23T00:42:56.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.370+0000 7fb9cb59e700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb9cc1290b0 0x7fb9cc12b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.371+0000 7fb9b3fff700 10 client.38303.objecter ms_handle_connect 0x7fb9cc1290b0 2024-02-23T00:42:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.371+0000 7fb9cb59e700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb9cc1290b0 0x7fb9cc12b4a0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.371+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb9cc12e1b0 con 0x7fb9cc1290b0 2024-02-23T00:42:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.371+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:42:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.371+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 13 = system:0 2024-02-23T00:42:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.371+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc119310 osd=6 4 2024-02-23T00:42:56.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.371+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 6 13 2024-02-23T00:42:56.387 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 10 client.38303.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 14 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc119310 osd=6 3 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119310 osd=6 3 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc119310 osd=6 3 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 14 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc119310 osd=6 3 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 6 14 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119310 osd=6 4 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 10 client.38303.objecter _send_command 14 2024-02-23T00:42:56.388 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.385+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb9cc12e1b0 con 0x7fb9cc1290b0 2024-02-23T00:42:56.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.387+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb9cc12e1b0 con 0x7fb9cc1290b0 2024-02-23T00:42:56.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.387+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:56.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.387+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.387+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc119310 osd=6 4 2024-02-23T00:42:56.390 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.387+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 6 14 2024-02-23T00:42:56.390 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 10 client.38303.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 15 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6812/373787000 conn(0x7fb9cc12d540 0x7fb9cc12d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc12e1b0 osd=7 3 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc12e1b0 osd=7 3 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc12e1b0 osd=7 3 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 15 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc12e1b0 osd=7 3 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 7 15 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc12e1b0 osd=7 4 2024-02-23T00:42:56.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 10 client.38303.objecter _send_command 15 2024-02-23T00:42:56.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9cc131560 con 0x7fb9cc12d540 2024-02-23T00:42:56.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9cad9d700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6812/373787000 conn(0x7fb9cc12d540 0x7fb9cc12d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:42:56.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9b3fff700 10 client.38303.objecter ms_handle_connect 0x7fb9cc12d540 2024-02-23T00:42:56.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.388+0000 7fb9cad9d700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6812/373787000 conn(0x7fb9cc12d540 0x7fb9cc12d9c0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:42:56.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.389+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb9cc131560 con 0x7fb9cc12d540 2024-02-23T00:42:56.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.389+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:42:56.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.389+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 15 = system:0 2024-02-23T00:42:56.392 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.389+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc12e1b0 osd=7 4 2024-02-23T00:42:56.393 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.389+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 7 15 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 10 client.38303.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign -1 16 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc12e1b0 osd=7 3 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc12e1b0 osd=7 3 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 20 client.38303.objecter _get_session s=0x7fb9cc12e1b0 osd=7 3 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_remove -1 16 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 20 client.38303.objecter get_session s=0x7fb9cc12e1b0 osd=7 3 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 15 client.38303.objecter _session_command_op_assign 7 16 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc12e1b0 osd=7 4 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 10 client.38303.objecter _send_command 16 2024-02-23T00:42:56.406 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.403+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb9cc12b990 con 0x7fb9cc12d540 2024-02-23T00:42:56.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.405+0000 7fb9b3fff700 1 -- 172.21.15.87:0/945354192 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fb9cc12b990 con 0x7fb9cc12d540 2024-02-23T00:42:56.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.405+0000 7fb9b3fff700 10 client.38303.objecter ms_dispatch 0x7fb9cc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:42:56.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.405+0000 7fb9b3fff700 10 client.38303.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.405+0000 7fb9b3fff700 20 client.38303.objecter put_session s=0x7fb9cc12e1b0 osd=7 4 2024-02-23T00:42:56.408 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.405+0000 7fb9b3fff700 15 client.38303.objecter _session_command_op_remove 7 16 2024-02-23T00:42:56.408 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:42:56.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 librados: watch_flush enter 2024-02-23T00:42:56.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 librados: watch_flush exit 2024-02-23T00:42:56.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 client.38303.objecter close_session for osd.0 2024-02-23T00:42:56.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6801/484081831 conn(0x7fb9cc0fc110 msgr2=0x7fb9cc0fc4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.410 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6801/484081831 conn(0x7fb9cc0fc110 0x7fb9cc0fc4f0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc109010 osd=0 1 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 client.38303.objecter close_session for osd.1 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6805/685354171 conn(0x7fb9cc0598f0 msgr2=0x7fb9cc059cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6805/685354171 conn(0x7fb9cc0598f0 0x7fb9cc059cf0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc028540 osd=1 1 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 client.38303.objecter close_session for osd.2 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb9cc119f50 msgr2=0x7fb9cc11c340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb9cc119f50 0x7fb9cc11c340 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027b90 osd=2 1 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 client.38303.objecter close_session for osd.3 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6813/128059790 conn(0x7fb9cc119ab0 msgr2=0x7fb9cc11f9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6813/128059790 conn(0x7fb9cc119ab0 0x7fb9cc11f9c0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc027a10 osd=3 1 2024-02-23T00:42:56.411 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 client.38303.objecter close_session for osd.4 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb9cc11d510 msgr2=0x7fb9cc121ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6800/4175844757 conn(0x7fb9cc11d510 0x7fb9cc121ec0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119010 osd=4 1 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 client.38303.objecter close_session for osd.5 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb9cc0f80a0 msgr2=0x7fb9cc0f84e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6804/2550013795 conn(0x7fb9cc0f80a0 0x7fb9cc0f84e0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc122470 osd=5 1 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 client.38303.objecter close_session for osd.6 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb9cc1290b0 msgr2=0x7fb9cc12b4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6808/2509408498 conn(0x7fb9cc1290b0 0x7fb9cc12b4a0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc119310 osd=6 1 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 client.38303.objecter close_session for osd.7 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6812/373787000 conn(0x7fb9cc12d540 msgr2=0x7fb9cc12d9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.412 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.171:6812/373787000 conn(0x7fb9cc12d540 0x7fb9cc12d9c0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 20 client.38303.objecter put_session s=0x7fb9cc12e1b0 osd=7 1 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 20 client.38303.objecter shutdown clearing up homeless session... 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 10 client.38303.objecter successfully canceled tick 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9cb59e700 1 -- 172.21.15.87:0/945354192 reap_dead start 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9cb59e700 1 -- 172.21.15.87:0/945354192 reap_dead start 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9cb59e700 1 -- 172.21.15.87:0/945354192 reap_dead start 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb9b4062c90 msgr2=0x7fb9b4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9cb59e700 1 -- 172.21.15.87:0/945354192 reap_dead start 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb9b4062c90 0x7fb9b4065150 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 rx=0x7fb9bc0041d0 tx=0x7fb9bc009040).stop 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9cb59e700 1 -- 172.21.15.87:0/945354192 reap_dead start 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 msgr2=0x7fb9cc108170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 0x7fb9cc108170 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 rx=0x7fb9c400b600 tx=0x7fb9c4004c50).stop 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 shutdown_connections 2024-02-23T00:42:56.413 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 --2- 172.21.15.87:0/945354192 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb9cc102180 0x7fb9cc108170 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 >> 172.21.15.87:0/945354192 conn(0x7fb9cc0f26f0 msgr2=0x7fb9cc0f32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 shutdown_connections 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 -- 172.21.15.87:0/945354192 wait complete. 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:42:56.406+0000 7fb9d1044700 1 librados: shutdown 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.414 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:42:56.442 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:42:56.473 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:56.493 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:42:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:56.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:42:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:55.226347+0000 osd.3 (osd.3) 234 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:42:56.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:55.229960+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:55.470081+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.d scrub starts 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:55.471649+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.d scrub ok 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:55.604433+0000 osd.0 (osd.0) 33 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:55.607347+0000 osd.0 (osd.0) 34 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:55.791493+0000 mgr.y (mgr.24437) 636 : cluster [DBG] pgmap v1141: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 9.7 KiB/s wr, 2 op/s; 1/756 objects misplaced (0.132%) 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:56.264449+0000 mon.c (mon.2) 500 : audit [DBG] from='client.? 172.21.15.87:0/945354192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:55.226347+0000 osd.3 (osd.3) 234 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:55.229960+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:55.470081+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.d scrub starts 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:55.471649+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.d scrub ok 2024-02-23T00:42:56.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:55.604433+0000 osd.0 (osd.0) 33 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:42:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:55.607347+0000 osd.0 (osd.0) 34 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:42:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:55.791493+0000 mgr.y (mgr.24437) 636 : cluster [DBG] pgmap v1141: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 9.7 KiB/s wr, 2 op/s; 1/756 objects misplaced (0.132%) 2024-02-23T00:42:56.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:56.264449+0000 mon.c (mon.2) 500 : audit [DBG] from='client.? 172.21.15.87:0/945354192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:56.597 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:56.626 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:56.646 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:56.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:55.226347+0000 osd.3 (osd.3) 234 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:42:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:55.229960+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:42:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:55.470081+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.d scrub starts 2024-02-23T00:42:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:55.471649+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.d scrub ok 2024-02-23T00:42:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:55.604433+0000 osd.0 (osd.0) 33 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:42:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:55.607347+0000 osd.0 (osd.0) 34 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:42:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:55.791493+0000 mgr.y (mgr.24437) 636 : cluster [DBG] pgmap v1141: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 9.7 KiB/s wr, 2 op/s; 1/756 objects misplaced (0.132%) 2024-02-23T00:42:56.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:56.264449+0000 mon.c (mon.2) 500 : audit [DBG] from='client.? 172.21.15.87:0/945354192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:42:56.751 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:42:56.781 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:56.854 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:42:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:56.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:42:56.963 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:56.990 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:57.438 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:57.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:57.543 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:42:57.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:56.443194+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.a scrub starts 2024-02-23T00:42:57.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:56.444083+0000 osd.2 (osd.2) 272 : cluster [DBG] 5.a scrub ok 2024-02-23T00:42:57.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:56.634985+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:42:57.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:56.638005+0000 osd.7 (osd.7) 337 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:42:57.568 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:57.232764+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-02-23T00:42:57.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:56.443194+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.a scrub starts 2024-02-23T00:42:57.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:56.444083+0000 osd.2 (osd.2) 272 : cluster [DBG] 5.a scrub ok 2024-02-23T00:42:57.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:56.634985+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:42:57.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:56.638005+0000 osd.7 (osd.7) 337 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:42:57.569 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:57.232764+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-02-23T00:42:57.570 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:56.443194+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.a scrub starts 2024-02-23T00:42:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:56.444083+0000 osd.2 (osd.2) 272 : cluster [DBG] 5.a scrub ok 2024-02-23T00:42:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:56.634985+0000 osd.7 (osd.7) 336 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:42:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:56.638005+0000 osd.7 (osd.7) 337 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:42:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:57.232764+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-02-23T00:42:57.718 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:42:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:57.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:42:57.821 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:57.849 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:57.860 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-23T00:42:57.860 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:42:57.860 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-23T00:42:57.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-23T00:42:57.902 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:57.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:58.005 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:58.035 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:42:58.035 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:42:58.037 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:58.059 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:58.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:58.164 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:42:58.194 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:58.216 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:42:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:58.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:42:58.320 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:58.348 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:57.586317+0000 osd.0 (osd.0) 35 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:42:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:57.587561+0000 osd.0 (osd.0) 36 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:42:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:57.791869+0000 mgr.y (mgr.24437) 637 : cluster [DBG] pgmap v1143: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 10 KiB/s wr, 2 op/s; 1/756 objects misplaced (0.132%) 2024-02-23T00:42:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:58.208860+0000 osd.5 (osd.5) 287 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:42:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:58.209951+0000 osd.5 (osd.5) 288 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:42:58.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:57.586317+0000 osd.0 (osd.0) 35 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:42:58.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:57.587561+0000 osd.0 (osd.0) 36 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:42:58.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:57.791869+0000 mgr.y (mgr.24437) 637 : cluster [DBG] pgmap v1143: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 10 KiB/s wr, 2 op/s; 1/756 objects misplaced (0.132%) 2024-02-23T00:42:58.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:58.208860+0000 osd.5 (osd.5) 287 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:42:58.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:58.209951+0000 osd.5 (osd.5) 288 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:42:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:57.586317+0000 osd.0 (osd.0) 35 : cluster [DBG] 22.18 scrub starts 2024-02-23T00:42:58.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:57.587561+0000 osd.0 (osd.0) 36 : cluster [DBG] 22.18 scrub ok 2024-02-23T00:42:58.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:57.791869+0000 mgr.y (mgr.24437) 637 : cluster [DBG] pgmap v1143: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 10 KiB/s wr, 2 op/s; 1/756 objects misplaced (0.132%) 2024-02-23T00:42:58.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:58.208860+0000 osd.5 (osd.5) 287 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:42:58.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:58.209951+0000 osd.5 (osd.5) 288 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:42:58.905 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-23T00:42:58.996 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:42:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:59.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:42:59.101 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:42:59.130 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:59.381 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:42:59.386 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:59.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:42:59.489 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:59.519 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:59.540 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:59.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:59.644 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:42:59.672 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:59.679 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:58.446030+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:59.679 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:58.446393+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130"}]: dispatch 2024-02-23T00:42:59.679 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:58.642509+0000 osd.6 (osd.6) 302 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:42:59.679 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:58.645912+0000 osd.6 (osd.6) 303 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:42:59.679 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:58.673496+0000 osd.7 (osd.7) 338 : cluster [DBG] 3.f scrub starts 2024-02-23T00:42:59.679 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:58.676427+0000 osd.7 (osd.7) 339 : cluster [DBG] 3.f scrub ok 2024-02-23T00:42:59.679 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.228443+0000 osd.5 (osd.5) 289 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:42:59.679 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.230216+0000 osd.5 (osd.5) 290 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:42:59.680 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:58.446030+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:59.680 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:58.446393+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130"}]: dispatch 2024-02-23T00:42:59.680 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:58.642509+0000 osd.6 (osd.6) 302 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:42:59.680 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:58.645912+0000 osd.6 (osd.6) 303 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:42:59.680 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:58.673496+0000 osd.7 (osd.7) 338 : cluster [DBG] 3.f scrub starts 2024-02-23T00:42:59.680 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:58.676427+0000 osd.7 (osd.7) 339 : cluster [DBG] 3.f scrub ok 2024-02-23T00:42:59.680 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.228443+0000 osd.5 (osd.5) 289 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:42:59.680 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:42:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.230216+0000 osd.5 (osd.5) 290 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:42:59.692 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:42:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:42:59.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:42:59.766 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:42:59.795 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:42:59.824 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:42:59.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:58.446030+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:42:59.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:58.446393+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130"}]: dispatch 2024-02-23T00:42:59.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:58.642509+0000 osd.6 (osd.6) 302 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:42:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:58.645912+0000 osd.6 (osd.6) 303 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:42:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:58.673496+0000 osd.7 (osd.7) 338 : cluster [DBG] 3.f scrub starts 2024-02-23T00:42:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:58.676427+0000 osd.7 (osd.7) 339 : cluster [DBG] 3.f scrub ok 2024-02-23T00:42:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.228443+0000 osd.5 (osd.5) 289 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:42:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:42:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.230216+0000 osd.5 (osd.5) 290 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:43:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.172704+0000 osd.3 (osd.3) 236 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:43:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.176238+0000 osd.3 (osd.3) 237 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:43:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.281096+0000 osd.1 (osd.1) 338 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:43:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.284592+0000 osd.1 (osd.1) 339 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:43:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:42:59.437726+0000 mon.a (mon.0) 3734 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130"}]': finished 2024-02-23T00:43:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.437767+0000 mon.a (mon.0) 3735 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-02-23T00:43:00.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.623169+0000 osd.6 (osd.6) 304 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.626632+0000 osd.6 (osd.6) 305 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.676870+0000 osd.7 (osd.7) 340 : cluster [DBG] 3.13 scrub starts 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.677891+0000 osd.7 (osd.7) 341 : cluster [DBG] 3.13 scrub ok 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:42:59.792253+0000 mgr.y (mgr.24437) 638 : cluster [DBG] pgmap v1145: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/756 objects misplaced (0.132%) 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.003686+0000 osd.4 (osd.4) 331 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.007432+0000 osd.4 (osd.4) 332 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.172704+0000 osd.3 (osd.3) 236 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.176238+0000 osd.3 (osd.3) 237 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.281096+0000 osd.1 (osd.1) 338 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.284592+0000 osd.1 (osd.1) 339 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:43:00.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:42:59.437726+0000 mon.a (mon.0) 3734 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130"}]': finished 2024-02-23T00:43:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.437767+0000 mon.a (mon.0) 3735 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-02-23T00:43:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.623169+0000 osd.6 (osd.6) 304 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:43:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.626632+0000 osd.6 (osd.6) 305 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:43:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.676870+0000 osd.7 (osd.7) 340 : cluster [DBG] 3.13 scrub starts 2024-02-23T00:43:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.677891+0000 osd.7 (osd.7) 341 : cluster [DBG] 3.13 scrub ok 2024-02-23T00:43:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:42:59.792253+0000 mgr.y (mgr.24437) 638 : cluster [DBG] pgmap v1145: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/756 objects misplaced (0.132%) 2024-02-23T00:43:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.003686+0000 osd.4 (osd.4) 331 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:43:00.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.007432+0000 osd.4 (osd.4) 332 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:43:00.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.172704+0000 osd.3 (osd.3) 236 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.176238+0000 osd.3 (osd.3) 237 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.281096+0000 osd.1 (osd.1) 338 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.284592+0000 osd.1 (osd.1) 339 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:42:59.437726+0000 mon.a (mon.0) 3734 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-130"}]': finished 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.437767+0000 mon.a (mon.0) 3735 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.623169+0000 osd.6 (osd.6) 304 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.626632+0000 osd.6 (osd.6) 305 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.676870+0000 osd.7 (osd.7) 340 : cluster [DBG] 3.13 scrub starts 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.677891+0000 osd.7 (osd.7) 341 : cluster [DBG] 3.13 scrub ok 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:42:59.792253+0000 mgr.y (mgr.24437) 638 : cluster [DBG] pgmap v1145: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/756 objects misplaced (0.132%) 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.003686+0000 osd.4 (osd.4) 331 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:43:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.007432+0000 osd.4 (osd.4) 332 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:43:01.150 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:01.173 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:01.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:01.278 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:01.305 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:01.415 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:43:01.453 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:01.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:01.559 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:01.591 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:01.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:43:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.206251+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:43:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.208014+0000 osd.3 (osd.3) 239 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:43:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.259271+0000 osd.5 (osd.5) 291 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:43:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.260238+0000 osd.5 (osd.5) 292 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:43:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.292979+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:43:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.296445+0000 osd.1 (osd.1) 341 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:43:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.441641+0000 mon.a (mon.0) 3736 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:43:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.456126+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-02-23T00:43:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.580731+0000 osd.6 (osd.6) 306 : cluster [DBG] 4.a scrub starts 2024-02-23T00:43:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.586911+0000 osd.6 (osd.6) 307 : cluster [DBG] 4.a scrub ok 2024-02-23T00:43:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.725364+0000 osd.7 (osd.7) 342 : cluster [DBG] 3.14 scrub starts 2024-02-23T00:43:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.726428+0000 osd.7 (osd.7) 343 : cluster [DBG] 3.14 scrub ok 2024-02-23T00:43:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:01.265083+0000 mgr.y (mgr.24437) 639 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.206251+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:43:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.208014+0000 osd.3 (osd.3) 239 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:43:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.259271+0000 osd.5 (osd.5) 291 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:43:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.260238+0000 osd.5 (osd.5) 292 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:43:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.292979+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:43:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.296445+0000 osd.1 (osd.1) 341 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:43:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.441641+0000 mon.a (mon.0) 3736 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:43:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.456126+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-02-23T00:43:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.580731+0000 osd.6 (osd.6) 306 : cluster [DBG] 4.a scrub starts 2024-02-23T00:43:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.586911+0000 osd.6 (osd.6) 307 : cluster [DBG] 4.a scrub ok 2024-02-23T00:43:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.725364+0000 osd.7 (osd.7) 342 : cluster [DBG] 3.14 scrub starts 2024-02-23T00:43:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.726428+0000 osd.7 (osd.7) 343 : cluster [DBG] 3.14 scrub ok 2024-02-23T00:43:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:01.265083+0000 mgr.y (mgr.24437) 639 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.206251+0000 osd.3 (osd.3) 238 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:43:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.208014+0000 osd.3 (osd.3) 239 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:43:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.259271+0000 osd.5 (osd.5) 291 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:43:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.260238+0000 osd.5 (osd.5) 292 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:43:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.292979+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:43:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.296445+0000 osd.1 (osd.1) 341 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:43:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.441641+0000 mon.a (mon.0) 3736 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:43:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.456126+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-02-23T00:43:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.580731+0000 osd.6 (osd.6) 306 : cluster [DBG] 4.a scrub starts 2024-02-23T00:43:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.586911+0000 osd.6 (osd.6) 307 : cluster [DBG] 4.a scrub ok 2024-02-23T00:43:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.725364+0000 osd.7 (osd.7) 342 : cluster [DBG] 3.14 scrub starts 2024-02-23T00:43:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.726428+0000 osd.7 (osd.7) 343 : cluster [DBG] 3.14 scrub ok 2024-02-23T00:43:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:01.265083+0000 mgr.y (mgr.24437) 639 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:02.152 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:02.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:02.253 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:02.292 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:02.313 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:02.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:02.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:02.417 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:02.443 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:02] "GET /metrics HTTP/1.1" 200 190515 "" "Prometheus/2.33.4" 2024-02-23T00:43:02.445 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:02.517 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:02.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:02.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:02.622 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:02.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.627991+0000 osd.0 (osd.0) 37 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:43:02.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:00.633182+0000 osd.0 (osd.0) 38 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:43:02.649 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:01.464777+0000 mon.a (mon.0) 3738 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-02-23T00:43:02.650 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:01.468171+0000 mon.a (mon.0) 3739 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:02.650 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:01.792653+0000 mgr.y (mgr.24437) 640 : cluster [DBG] pgmap v1148: 204 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:43:02.650 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:01.986496+0000 osd.4 (osd.4) 333 : cluster [DBG] 2.f scrub starts 2024-02-23T00:43:02.650 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:01.989973+0000 osd.4 (osd.4) 334 : cluster [DBG] 2.f scrub ok 2024-02-23T00:43:02.650 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:02.236983+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:02.650 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:02.237176+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-02-23T00:43:02.650 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:02.354629+0000 mon.a (mon.0) 3742 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:02.652 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:02.671 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:02.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:02.742 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.627991+0000 osd.0 (osd.0) 37 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:43:02.742 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:00.633182+0000 osd.0 (osd.0) 38 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:43:02.742 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:01.464777+0000 mon.a (mon.0) 3738 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-02-23T00:43:02.742 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:01.468171+0000 mon.a (mon.0) 3739 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:02.742 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:01.792653+0000 mgr.y (mgr.24437) 640 : cluster [DBG] pgmap v1148: 204 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:01.986496+0000 osd.4 (osd.4) 333 : cluster [DBG] 2.f scrub starts 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:01.989973+0000 osd.4 (osd.4) 334 : cluster [DBG] 2.f scrub ok 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:02.236983+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:02.237176+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:02.354629+0000 mon.a (mon.0) 3742 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.627991+0000 osd.0 (osd.0) 37 : cluster [DBG] 3.3 scrub starts 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:00.633182+0000 osd.0 (osd.0) 38 : cluster [DBG] 3.3 scrub ok 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:01.464777+0000 mon.a (mon.0) 3738 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:01.468171+0000 mon.a (mon.0) 3739 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:01.792653+0000 mgr.y (mgr.24437) 640 : cluster [DBG] pgmap v1148: 204 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:01.986496+0000 osd.4 (osd.4) 333 : cluster [DBG] 2.f scrub starts 2024-02-23T00:43:02.743 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:01.989973+0000 osd.4 (osd.4) 334 : cluster [DBG] 2.f scrub ok 2024-02-23T00:43:02.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:02.236983+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:02.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:02.237176+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-02-23T00:43:02.744 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:02.354629+0000 mon.a (mon.0) 3742 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:02.776 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:02.804 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:02.823 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:02.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:02.926 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:02.953 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:02.973 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:02.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:03.077 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:03.106 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:04.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 -- 172.21.15.87:0/217446675 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f978c0f8cb0 msgr2=0x7f978c0f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:04.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 --2- 172.21.15.87:0/217446675 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f978c0f8cb0 0x7f978c0f9090 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 rx=0x7f977c004870 tx=0x7f977c009cb0).stop 2024-02-23T00:43:04.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 -- 172.21.15.87:0/217446675 shutdown_connections 2024-02-23T00:43:04.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 --2- 172.21.15.87:0/217446675 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f978c102210 0x7f978c104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:04.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 --2- 172.21.15.87:0/217446675 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f978c0f8cb0 0x7f978c0f9090 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 --2- 172.21.15.87:0/217446675 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f978c0f9610 0x7f978c101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 -- 172.21.15.87:0/217446675 >> 172.21.15.87:0/217446675 conn(0x7f978c0f2820 msgr2=0x7f978c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 -- 172.21.15.87:0/217446675 shutdown_connections 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 -- 172.21.15.87:0/217446675 wait complete. 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 librados: starting msgr at 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 librados: starting objecter 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.037+0000 7f9791d9c700 1 Processor -- start 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f9791d9c700 1 -- start start 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f9791d9c700 1 librados: setting wanted keys 2024-02-23T00:43:04.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f9791d9c700 1 librados: calling monclient init 2024-02-23T00:43:04.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f9791d9c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f978c0f8cb0 0x7f978c0f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:04.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f9791d9c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f978c0f9610 0x7f978c0f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:04.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f9791d9c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f978c102210 0x7f978c0f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:04.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f9791d9c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c116650 con 0x7f978c0f9610 2024-02-23T00:43:04.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f9791d9c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1167c0 con 0x7f978c0f8cb0 2024-02-23T00:43:04.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f9791d9c700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c116980 con 0x7f978c102210 2024-02-23T00:43:04.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f978c0f9610 0x7f978c0f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:04.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f978c0f9610 0x7f978c0f7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47216/0 (socket says 172.21.15.87:47216) 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978affd700 1 -- 172.21.15.87:0/216347142 learned_addr learned my addr 172.21.15.87:0/216347142 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978b7fe700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f978c0f8cb0 0x7f978c0f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978bfff700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f978c102210 0x7f978c0f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978bfff700 1 -- 172.21.15.87:0/216347142 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f978c0f8cb0 msgr2=0x7f978c0f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978bfff700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f978c0f8cb0 0x7f978c0f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978bfff700 1 -- 172.21.15.87:0/216347142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f978c0f9610 msgr2=0x7f978c0f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978bfff700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f978c0f9610 0x7f978c0f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978bfff700 1 -- 172.21.15.87:0/216347142 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97840097c0 con 0x7f978c102210 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.038+0000 7f978affd700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f978c0f9610 0x7f978c0f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f978bfff700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f978c102210 0x7f978c0f7640 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f978c060200 tx=0x7f9784004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f9788ff9700 10 client.?.objecter ms_handle_connect 0x7f978c102210 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f9788ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f9788ff9700 1 -- 172.21.15.87:0/216347142 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840079b0 con 0x7f978c102210 2024-02-23T00:43:04.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f9788ff9700 1 -- 172.21.15.87:0/216347142 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97840090d0 con 0x7f978c102210 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c005040 con 0x7f978c102210 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f9788ff9700 1 -- 172.21.15.87:0/216347142 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978400fc90 con 0x7f978c102210 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f9791d9c700 10 client.38309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c116fc0 con 0x7f978c102210 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.039+0000 7f9791d9c700 1 librados: init done 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.040+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c05fa60 con 0x7f978c102210 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.040+0000 7f9788ff9700 1 -- 172.21.15.87:0/216347142 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f9784007b10 con 0x7f978c102210 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.041+0000 7f9788ff9700 1 --2- 172.21.15.87:0/216347142 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9774062c90 0x7f9774065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.041+0000 7f9788ff9700 1 -- 172.21.15.87:0/216347142 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(752..752 src has 400..752) v4 ==== 8538+0+0 (secure 0 0 0) 0x7f978407db30 con 0x7f978c102210 2024-02-23T00:43:04.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.041+0000 7f9788ff9700 10 client.38309.objecter ms_dispatch 0x7f978c001550 osd_map(752..752 src has 400..752) v4 2024-02-23T00:43:04.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.041+0000 7f978b7fe700 1 --2- 172.21.15.87:0/216347142 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9774062c90 0x7f9774065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:04.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.041+0000 7f9788ff9700 3 client.38309.objecter handle_osd_map got epochs [752,752] > 0 2024-02-23T00:43:04.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.041+0000 7f9788ff9700 3 client.38309.objecter handle_osd_map decoding full epoch 752 2024-02-23T00:43:04.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.041+0000 7f9788ff9700 20 client.38309.objecter dump_active .. 0 homeless 2024-02-23T00:43:04.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.041+0000 7f9788ff9700 10 client.38309.objecter ms_handle_connect 0x7f9774062c90 2024-02-23T00:43:04.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.041+0000 7f978b7fe700 1 --2- 172.21.15.87:0/216347142 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9774062c90 0x7f9774065150 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7f977c0043d0 tx=0x7f977c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:04.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.045+0000 7f9788ff9700 1 -- 172.21.15.87:0/216347142 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9784050ea0 con 0x7f978c102210 2024-02-23T00:43:04.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:43:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:43:04.217 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.215+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f978c0f8140 con 0x7f978c102210 2024-02-23T00:43:04.218 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.216+0000 7f9788ff9700 1 -- 172.21.15.87:0/216347142 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v752) v1 ==== 81+0+48992 (secure 0 0 0) 0x7f97840191b0 con 0x7f978c102210 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 752, 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: "created": "2024-02-23T00:21:28.861318+0000", 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: "modified": "2024-02-23T00:43:03.231665+0000", 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: "last_up_change": "2024-02-23T00:41:18.209057+0000", 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: "last_in_change": "2024-02-23T00:41:36.208573+0000", 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_num": 5804032, 2024-02-23T00:43:04.220 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_set": [ 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "nodeep-scrub", 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "pglog_hardlimit", 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "purged_snapdirs", 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "recovery_deletes", 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "sortbitwise" 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_version": 114, 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "full_ratio": 0.94999998807907104, 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_snapshot": "", 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_max": 250, 2024-02-23T00:43:04.221 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd": 8, 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "require_min_compat_client": "luminous", 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "min_compat_client": "luminous", 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "require_osd_release": "pacific", 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "pools": [ 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "device_health_metrics", 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:25:09.208309+0000", 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:04.222 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:04.223 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:04.223 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:04.223 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:04.223 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:04.223 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:04.223 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:04.223 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 1, 2024-02-23T00:43:04.223 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 1, 2024-02-23T00:43:04.223 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 1, 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 1, 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 1, 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "68", 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:04.224 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:04.225 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:04.226 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_max": 32, 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_min": 1 2024-02-23T00:43:04.227 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.228 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:04.228 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_devicehealth": {} 2024-02-23T00:43:04.228 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.228 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.228 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.228 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 2, 2024-02-23T00:43:04.228 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": ".rgw.root", 2024-02-23T00:43:04.228 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.562186+0000", 2024-02-23T00:43:04.228 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:04.229 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "174", 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:43:04.230 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:04.231 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:04.232 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 3, 2024-02-23T00:43:04.233 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "iscsi", 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.989206+0000", 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 23, 2024-02-23T00:43:04.234 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 23, 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 23, 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 23, 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 23, 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "738", 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:04.235 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "736", 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:04.236 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:04.237 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd": {} 2024-02-23T00:43:04.238 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 4, 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.log", 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:17.282440+0000", 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:04.239 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "4.16", 2024-02-23T00:43:04.240 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 371, 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 370, 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 370, 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "63'21", 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "62'12" 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "532", 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "371", 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "481", 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:04.241 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:04.242 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:04.243 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.244 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 5, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.control", 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:19.262164+0000", 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:04.245 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "5.20", 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 641, 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 640, 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 640, 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "562'12", 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "330'9" 2024-02-23T00:43:04.246 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "641", 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "641", 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "639", 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:04.247 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:04.248 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:04.249 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 6, 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.meta", 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:21.319951+0000", 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:04.250 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 12, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 12, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 12, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 12, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 12, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "6.c", 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 665, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 664, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 664, 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "78'2", 2024-02-23T00:43:04.251 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "665", 2024-02-23T00:43:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "665", 2024-02-23T00:43:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "663", 2024-02-23T00:43:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:04.252 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:04.253 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:04.254 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_bias": 4 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 22, 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:46.071806+0000", 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:04.255 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:04.256 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:43:04.257 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:04.258 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:04.259 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 234, 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-101", 2024-02-23T00:43:04.260 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:40:15.435168+0000", 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 3, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 1, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 8, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 8, 2024-02-23T00:43:04.261 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 8, 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 8, 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 8, 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "752", 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "727", 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "727", 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "727", 2024-02-23T00:43:04.262 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 10, 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 752, 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [ 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: 250 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:04.263 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi087-82091-101", 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:04.264 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 8192, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 250, 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-132", 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:43:00.700240+0000", 2024-02-23T00:43:04.265 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:04.266 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "752", 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:04.267 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 10, 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 752, 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": 234, 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:04.268 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:04.269 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7", 2024-02-23T00:43:04.270 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 8, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 424, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 659, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 750, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 425, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.271 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6802", 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.272 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6804", 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6803", 2024-02-23T00:43:04.273 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6801/484081831", 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6802/484081831", 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6804/484081831", 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6803/484081831", 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.274 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "85525ce8-38d5-4840-8057-547cb176c859", 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 13, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 750, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.275 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6805", 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6806", 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:43:04.276 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6808", 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.277 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6807", 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6805/685354171", 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6806/685354171", 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6808/685354171", 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6807/685354171", 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:04.278 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa", 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 18, 2024-02-23T00:43:04.279 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 750, 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6809", 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:04.280 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6810", 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6812", 2024-02-23T00:43:04.281 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6811", 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6809/3605394228", 2024-02-23T00:43:04.282 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6810/3605394228", 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6812/3605394228", 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6811/3605394228", 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a", 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:04.283 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 26, 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 750, 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6813", 2024-02-23T00:43:04.284 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6814", 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:04.285 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6816", 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6815", 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.286 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6813/128059790", 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6814/128059790", 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6816/128059790", 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6815/128059790", 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c", 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:04.287 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.5775604248046875, 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 31, 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 750, 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6800", 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:43:04.288 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6801", 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:04.289 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6803", 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6802", 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.290 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6800/4175844757", 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6801/4175844757", 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4175844757", 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4175844757", 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d", 2024-02-23T00:43:04.291 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.8517608642578125, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 38, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 750, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6804", 2024-02-23T00:43:04.292 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6805", 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.293 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6807", 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6806", 2024-02-23T00:43:04.294 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6804/2550013795", 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6805/2550013795", 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2550013795", 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2550013795", 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.295 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01", 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 44, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 750, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:04.296 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6808", 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6809", 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:43:04.297 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6811", 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.298 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6810", 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6808/2509408498", 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6809/2509408498", 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6811/2509408498", 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6810/2509408498", 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:04.299 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939", 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 51, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 750, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:04.300 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6812", 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6813", 2024-02-23T00:43:04.301 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6815", 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:04.302 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6814", 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6812/373787000", 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6813/373787000", 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6815/373787000", 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6814/373787000", 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.303 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_xinfo": [ 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:33.795687+0000", 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 424 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.304 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:13.973488+0000", 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:04.305 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:41:43.491272+0000", 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:04.306 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:32.316583+0000", 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:31.854915+0000", 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.307 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:10.862105+0000", 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:04.308 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:29.973812+0000", 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:26.420812+0000", 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:04.309 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap": [], 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap_items": [ 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.18", 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 1 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.1c", 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 4 2024-02-23T00:43:04.310 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.13", 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "5.19", 2024-02-23T00:43:04.311 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 6 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "22.1a", 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 5 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.312 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_temp": [ 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "5.3", 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: 3, 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: 6, 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: 5 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_temp": [], 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: "blocklist": { 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1121018082": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:43:04.313 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/2138166229": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/27794119": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/1675126043": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/3767016525": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/969918521": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/795095363": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2273151327": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1685406414": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2093733106": "2024-02-24T00:21:58.144574+0000" 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "range_blocklist": {}, 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profiles": { 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "default": { 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:43:04.314 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van" 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "testprofile-test-rados-api-smithi087-82091-101": { 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-device-class": "", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-root": "default", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:43:04.315 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van", 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: "w": "8" 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps_queue": [ 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 7, 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.316 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.317 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 21, 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.318 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 61, 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.319 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 29, 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 78, 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.320 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 86, 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.321 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:43:04.322 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 95, 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 4, 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 4 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 107, 2024-02-23T00:43:04.323 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 112, 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.324 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 125, 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:04.325 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 10, 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 133, 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.326 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 154, 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 14, 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 164, 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 17, 2024-02-23T00:43:04.327 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 177, 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 20, 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 237, 2024-02-23T00:43:04.328 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 241, 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.329 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 245, 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: "new_removed_snaps": [], 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: "new_purged_snaps": [], 2024-02-23T00:43:04.330 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_node_flags": {}, 2024-02-23T00:43:04.331 INFO:teuthology.orchestra.run.smithi087.stdout: "device_class_flags": {}, 2024-02-23T00:43:04.331 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode": { 2024-02-23T00:43:04.331 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_enabled": false, 2024-02-23T00:43:04.331 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_bucket_count": 0, 2024-02-23T00:43:04.331 INFO:teuthology.orchestra.run.smithi087.stdout: "degraded_stretch_mode": 0, 2024-02-23T00:43:04.331 INFO:teuthology.orchestra.run.smithi087.stdout: "recovering_stretch_mode": 0, 2024-02-23T00:43:04.331 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_bucket": 0 2024-02-23T00:43:04.331 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:04.331 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:43:04.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 10 librados: watch_flush enter 2024-02-23T00:43:04.332 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 10 librados: watch_flush exit 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 20 client.38309.objecter shutdown clearing up homeless session... 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 10 client.38309.objecter successfully canceled tick 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9774062c90 msgr2=0x7f9774065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 --2- 172.21.15.87:0/216347142 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9774062c90 0x7f9774065150 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 rx=0x7f977c0043d0 tx=0x7f977c00b040).stop 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f978c102210 msgr2=0x7f978c0f7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f978c102210 0x7f978c0f7640 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 rx=0x7f978c060200 tx=0x7f9784004c50).stop 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 shutdown_connections 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 --2- 172.21.15.87:0/216347142 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9774062c90 0x7f9774065150 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f978c102210 0x7f978c0f7640 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f978c0f8cb0 0x7f978c0f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 --2- 172.21.15.87:0/216347142 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f978c0f9610 0x7f978c0f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 >> 172.21.15.87:0/216347142 conn(0x7f978c0f2820 msgr2=0x7f978c0f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:04.333 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 shutdown_connections 2024-02-23T00:43:04.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 -- 172.21.15.87:0/216347142 wait complete. 2024-02-23T00:43:04.334 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:04.218+0000 7f9791d9c700 1 librados: shutdown 2024-02-23T00:43:04.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:02.720105+0000 osd.7 (osd.7) 344 : cluster [DBG] 3.e scrub starts 2024-02-23T00:43:04.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:02.723202+0000 osd.7 (osd.7) 345 : cluster [DBG] 3.e scrub ok 2024-02-23T00:43:04.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:03.236209+0000 mon.a (mon.0) 3743 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:04.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:03.236286+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-02-23T00:43:04.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:03.237932+0000 mon.a (mon.0) 3745 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-132"}]: dispatch 2024-02-23T00:43:04.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:03.433075+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:43:04.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:03.434384+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:43:04.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:04.216808+0000 mon.c (mon.2) 501 : audit [DBG] from='client.? 172.21.15.87:0/216347142' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:43:04.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:02.720105+0000 osd.7 (osd.7) 344 : cluster [DBG] 3.e scrub starts 2024-02-23T00:43:04.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:02.723202+0000 osd.7 (osd.7) 345 : cluster [DBG] 3.e scrub ok 2024-02-23T00:43:04.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:03.236209+0000 mon.a (mon.0) 3743 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:04.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:03.236286+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-02-23T00:43:04.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:03.237932+0000 mon.a (mon.0) 3745 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-132"}]: dispatch 2024-02-23T00:43:04.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:03.433075+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:43:04.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:03.434384+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:43:04.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:04.216808+0000 mon.c (mon.2) 501 : audit [DBG] from='client.? 172.21.15.87:0/216347142' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:43:04.655 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:04.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:04.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:02.720105+0000 osd.7 (osd.7) 344 : cluster [DBG] 3.e scrub starts 2024-02-23T00:43:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:02.723202+0000 osd.7 (osd.7) 345 : cluster [DBG] 3.e scrub ok 2024-02-23T00:43:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:03.236209+0000 mon.a (mon.0) 3743 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:03.236286+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-02-23T00:43:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:03.237932+0000 mon.a (mon.0) 3745 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-132"}]: dispatch 2024-02-23T00:43:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:03.433075+0000 osd.2 (osd.2) 273 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:43:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:03.434384+0000 osd.2 (osd.2) 274 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:43:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:04.216808+0000 mon.c (mon.2) 501 : audit [DBG] from='client.? 172.21.15.87:0/216347142' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:43:04.760 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:04.793 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:04.815 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:04.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:04.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:04.917 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:04.947 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:04.967 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:04.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:05.019 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:43:05.019 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:43:05.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:05.101 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:05.225 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:05.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:05.329 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:05.356 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:03.631305+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:03.634218+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:03.793036+0000 mgr.y (mgr.24437) 641 : cluster [DBG] pgmap v1151: 204 pgs: 25 unknown, 1 active+remapped+backfill_toofull, 178 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:04.138608+0000 osd.3 (osd.3) 240 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:04.142188+0000 osd.3 (osd.3) 241 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:04.241955+0000 mon.a (mon.0) 3746 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-132"}]': finished 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:04.242028+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:04.242776+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-132", "mode": "writeback"}]: dispatch 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:04.618325+0000 osd.6 (osd.6) 308 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:04.619500+0000 osd.6 (osd.6) 309 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:05.148074+0000 osd.5 (osd.5) 293 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:43:05.499 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:05.148927+0000 osd.5 (osd.5) 294 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:03.631305+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:03.634218+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:03.793036+0000 mgr.y (mgr.24437) 641 : cluster [DBG] pgmap v1151: 204 pgs: 25 unknown, 1 active+remapped+backfill_toofull, 178 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:04.138608+0000 osd.3 (osd.3) 240 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:04.142188+0000 osd.3 (osd.3) 241 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:04.241955+0000 mon.a (mon.0) 3746 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-132"}]': finished 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:04.242028+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:04.242776+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-132", "mode": "writeback"}]: dispatch 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:04.618325+0000 osd.6 (osd.6) 308 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:43:05.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:04.619500+0000 osd.6 (osd.6) 309 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:43:05.501 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:05.148074+0000 osd.5 (osd.5) 293 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:43:05.501 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:05.148927+0000 osd.5 (osd.5) 294 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:43:05.505 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:05.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:05.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:05.624 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:05.650 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:03.631305+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:43:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:03.634218+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:43:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:03.793036+0000 mgr.y (mgr.24437) 641 : cluster [DBG] pgmap v1151: 204 pgs: 25 unknown, 1 active+remapped+backfill_toofull, 178 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/744 objects misplaced (0.134%) 2024-02-23T00:43:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:04.138608+0000 osd.3 (osd.3) 240 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:43:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:04.142188+0000 osd.3 (osd.3) 241 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:43:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:04.241955+0000 mon.a (mon.0) 3746 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-132"}]': finished 2024-02-23T00:43:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:04.242028+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-02-23T00:43:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:04.242776+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-132", "mode": "writeback"}]: dispatch 2024-02-23T00:43:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:04.618325+0000 osd.6 (osd.6) 308 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:43:05.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:04.619500+0000 osd.6 (osd.6) 309 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:43:05.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:05.148074+0000 osd.5 (osd.5) 293 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:43:05.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:05.148927+0000 osd.5 (osd.5) 294 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:43:06.007 INFO:tasks.thrashosds.thrasher:j is {'epoch': 752, 'fsid': '66db207c-d1e1-11ee-95c0-87774f69a715', 'created': '2024-02-23T00:21:28.861318+0000', 'modified': '2024-02-23T00:43:03.231665+0000', 'last_up_change': '2024-02-23T00:41:18.209057+0000', 'last_in_change': '2024-02-23T00:41:36.208573+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 114, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 250, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 23, 'pg_placement_num': 23, 'pg_placement_num_target': 23, 'pg_num_target': 23, 'pg_num_pending': 23, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '738', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '736', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '4.16', 'ready_epoch': 371, 'last_epoch_started': 370, 'last_epoch_clean': 370, 'source_version': "63'21", 'target_version': "62'12"}, 'last_change': '532', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '371', 'last_force_op_resend_preluminous': '481', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '5.20', 'ready_epoch': 641, 'last_epoch_started': 640, 'last_epoch_clean': 640, 'source_version': "562'12", 'target_version': "330'9"}, 'last_change': '641', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '641', 'last_force_op_resend_preluminous': '639', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 12, 'pg_placement_num': 12, 'pg_placement_num_target': 12, 'pg_num_target': 12, 'pg_num_pending': 12, 'last_pg_merge_meta': {'source_pgid': '6.c', 'ready_epoch': 665, 'last_epoch_started': 664, 'last_epoch_clean': 664, 'source_version': "78'2", 'target_version': "0'0"}, 'last_change': '665', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '665', 'last_force_op_resend_preluminous': '663', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 22, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-23T00:30:46.071806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi087-82091-101', 'create_time': '2024-02-23T00:40:15.435168+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '752', 'last_force_op_resend': '727', 'last_force_op_resend_prenautilus': '727', 'last_force_op_resend_preluminous': '727', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 752, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [250], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi087-82091-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 250, 'pool_name': 'test-rados-api-smithi087-82091-132', 'create_time': '2024-02-23T00:43:00.700240+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '752', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 752, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 234, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': 'e3549bad-2891-47b9-bf77-132cadc20bf7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 424, 'up_from': 659, 'up_thru': 750, 'down_at': 425, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6801', 'nonce': 484081831}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6802', 'nonce': 484081831}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6804', 'nonce': 484081831}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6803', 'nonce': 484081831}]}, 'public_addr': '172.21.15.87:6801/484081831', 'cluster_addr': '172.21.15.87:6802/484081831', 'heartbeat_back_addr': '172.21.15.87:6804/484081831', 'heartbeat_front_addr': '172.21.15.87:6803/484081831', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '85525ce8-38d5-4840-8057-547cb176c859', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 750, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6805', 'nonce': 685354171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6806', 'nonce': 685354171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6808', 'nonce': 685354171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6807', 'nonce': 685354171}]}, 'public_addr': '172.21.15.87:6805/685354171', 'cluster_addr': '172.21.15.87:6806/685354171', 'heartbeat_back_addr': '172.21.15.87:6808/685354171', 'heartbeat_front_addr': '172.21.15.87:6807/685354171', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 750, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6809', 'nonce': 3605394228}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6810', 'nonce': 3605394228}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6812', 'nonce': 3605394228}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6811', 'nonce': 3605394228}]}, 'public_addr': '172.21.15.87:6809/3605394228', 'cluster_addr': '172.21.15.87:6810/3605394228', 'heartbeat_back_addr': '172.21.15.87:6812/3605394228', 'heartbeat_front_addr': '172.21.15.87:6811/3605394228', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 750, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6813', 'nonce': 128059790}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6814', 'nonce': 128059790}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6816', 'nonce': 128059790}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6815', 'nonce': 128059790}]}, 'public_addr': '172.21.15.87:6813/128059790', 'cluster_addr': '172.21.15.87:6814/128059790', 'heartbeat_back_addr': '172.21.15.87:6816/128059790', 'heartbeat_front_addr': '172.21.15.87:6815/128059790', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c3734a04-7343-4251-9b5b-bedb9e8b882c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5775604248046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 750, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6800', 'nonce': 4175844757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6801', 'nonce': 4175844757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6803', 'nonce': 4175844757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6802', 'nonce': 4175844757}]}, 'public_addr': '172.21.15.171:6800/4175844757', 'cluster_addr': '172.21.15.171:6801/4175844757', 'heartbeat_back_addr': '172.21.15.171:6803/4175844757', 'heartbeat_front_addr': '172.21.15.171:6802/4175844757', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fe86232d-96b9-4cbf-9cac-373286fa103d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8517608642578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 750, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6804', 'nonce': 2550013795}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6805', 'nonce': 2550013795}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6807', 'nonce': 2550013795}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6806', 'nonce': 2550013795}]}, 'public_addr': '172.21.15.171:6804/2550013795', 'cluster_addr': '172.21.15.171:6805/2550013795', 'heartbeat_back_addr': '172.21.15.171:6807/2550013795', 'heartbeat_front_addr': '172.21.15.171:6806/2550013795', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 750, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6808', 'nonce': 2509408498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6809', 'nonce': 2509408498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6811', 'nonce': 2509408498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6810', 'nonce': 2509408498}]}, 'public_addr': '172.21.15.171:6808/2509408498', 'cluster_addr': '172.21.15.171:6809/2509408498', 'heartbeat_back_addr': '172.21.15.171:6811/2509408498', 'heartbeat_front_addr': '172.21.15.171:6810/2509408498', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '29ddcf97-369a-4a2c-b3e2-142e84ccf939', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 750, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6812', 'nonce': 373787000}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6813', 'nonce': 373787000}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6815', 'nonce': 373787000}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6814', 'nonce': 373787000}]}, 'public_addr': '172.21.15.171:6812/373787000', 'cluster_addr': '172.21.15.171:6813/373787000', 'heartbeat_back_addr': '172.21.15.171:6815/373787000', 'heartbeat_front_addr': '172.21.15.171:6814/373787000', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:33.795687+0000', 'dead_epoch': 424}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:13.973488+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:41:43.491272+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:32.316583+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:31.854915+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:10.862105+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:29.973812+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:26.420812+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '2.1c', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '22.1a', 'mappings': [{'from': 7, 'to': 5}]}], 'pg_temp': [{'pgid': '5.3', 'osds': [3, 6, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.87:6800/1121018082': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/2138166229': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/27794119': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/1675126043': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/3767016525': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/969918521': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/795095363': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2273151327': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/1685406414': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2093733106': '2024-02-24T00:21:58.144574+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi087-82091-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 21, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 61, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 63, 'snaps': [{'begin': 29, 'length': 1}]}, {'pool': 78, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 86, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 177, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 241, 'snaps': [{'begin': 6, 'length': 1}]}, {'pool': 245, 'snaps': [{'begin': 9, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-23T00:43:06.007 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:43:06.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-23T00:43:06.578 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:04.776161+0000 osd.7 (osd.7) 346 : cluster [DBG] 3.11 scrub starts 2024-02-23T00:43:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:04.777264+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.11 scrub ok 2024-02-23T00:43:06.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:05.238494+0000 mon.a (mon.0) 3749 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:05.247447+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-132", "mode": "writeback"}]': finished 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:05.247507+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:05.250677+0000 mon.a (mon.0) 3752 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:05.361517+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:05.364141+0000 osd.1 (osd.1) 343 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:05.737149+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.15 scrub starts 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:05.740108+0000 osd.7 (osd.7) 349 : cluster [DBG] 3.15 scrub ok 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.142357+0000 osd.5 (osd.5) 295 : cluster [DBG] 5.f scrub starts 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.143451+0000 osd.5 (osd.5) 296 : cluster [DBG] 5.f scrub ok 2024-02-23T00:43:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:04.776161+0000 osd.7 (osd.7) 346 : cluster [DBG] 3.11 scrub starts 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:04.777264+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.11 scrub ok 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:05.238494+0000 mon.a (mon.0) 3749 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:05.247447+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-132", "mode": "writeback"}]': finished 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:05.247507+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:05.250677+0000 mon.a (mon.0) 3752 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:05.361517+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:05.364141+0000 osd.1 (osd.1) 343 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:05.737149+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.15 scrub starts 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:05.740108+0000 osd.7 (osd.7) 349 : cluster [DBG] 3.15 scrub ok 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.142357+0000 osd.5 (osd.5) 295 : cluster [DBG] 5.f scrub starts 2024-02-23T00:43:06.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.143451+0000 osd.5 (osd.5) 296 : cluster [DBG] 5.f scrub ok 2024-02-23T00:43:06.600 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:06.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.598+0000 7f68fff55700 1 -- 172.21.15.87:0/2686753240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 msgr2=0x7f68f8108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:06.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.598+0000 7f68fff55700 1 --2- 172.21.15.87:0/2686753240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 0x7f68f8108300 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f68e8004870 tx=0x7f68e8009cb0).stop 2024-02-23T00:43:06.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.598+0000 7f68fff55700 1 -- 172.21.15.87:0/2686753240 shutdown_connections 2024-02-23T00:43:06.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.598+0000 7f68fff55700 1 --2- 172.21.15.87:0/2686753240 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f68f81087f0 0x7f68f810abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:06.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.598+0000 7f68fff55700 1 --2- 172.21.15.87:0/2686753240 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68f80f63b0 0x7f68f80f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:43:06.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.598+0000 7f68fff55700 1 --2- 172.21.15.87:0/2686753240 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 0x7f68f8108300 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:06.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.598+0000 7f68fff55700 1 -- 172.21.15.87:0/2686753240 >> 172.21.15.87:0/2686753240 conn(0x7f68f806c370 msgr2=0x7f68f806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:06.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.599+0000 7f68fff55700 1 -- 172.21.15.87:0/2686753240 shutdown_connections 2024-02-23T00:43:06.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.599+0000 7f68fff55700 1 -- 172.21.15.87:0/2686753240 wait complete. 2024-02-23T00:43:06.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 librados: starting msgr at 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 librados: starting objecter 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 Processor -- start 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 -- start start 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 librados: setting wanted keys 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 librados: calling monclient init 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 0x7f68f806a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f68f81087f0 0x7f68f806a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68f8063490 0x7f68f8063910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f8063fa0 con 0x7f68f80f6c80 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f8064110 con 0x7f68f8063490 2024-02-23T00:43:06.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fff55700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f80642d0 con 0x7f68f81087f0 2024-02-23T00:43:06.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fdcf1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 0x7f68f806a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:06.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fdcf1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 0x7f68f806a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47238/0 (socket says 172.21.15.87:47238) 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fdcf1700 1 -- 172.21.15.87:0/2028809208 learned_addr learned my addr 172.21.15.87:0/2028809208 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fd4f0700 1 --2- 172.21.15.87:0/2028809208 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f68f81087f0 0x7f68f806a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fe4f2700 1 --2- 172.21.15.87:0/2028809208 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68f8063490 0x7f68f8063910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fdcf1700 1 -- 172.21.15.87:0/2028809208 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f68f81087f0 msgr2=0x7f68f806a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fdcf1700 1 --2- 172.21.15.87:0/2028809208 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f68f81087f0 0x7f68f806a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fdcf1700 1 -- 172.21.15.87:0/2028809208 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68f8063490 msgr2=0x7f68f8063910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fdcf1700 1 --2- 172.21.15.87:0/2028809208 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68f8063490 0x7f68f8063910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.600+0000 7f68fdcf1700 1 -- 172.21.15.87:0/2028809208 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f00097c0 con 0x7f68f80f6c80 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68fdcf1700 1 --2- 172.21.15.87:0/2028809208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 0x7f68f806a370 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f68f400b420 tx=0x7f68f4006f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68eeffd700 10 client.?.objecter ms_handle_connect 0x7f68f80f6c80 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68eeffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68eeffd700 1 -- 172.21.15.87:0/2028809208 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4012410 con 0x7f68f80f6c80 2024-02-23T00:43:06.605 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e8005040 con 0x7f68f80f6c80 2024-02-23T00:43:06.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68fff55700 10 client.27840.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:06.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f8068340 con 0x7f68f80f6c80 2024-02-23T00:43:06.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68fff55700 1 librados: init done 2024-02-23T00:43:06.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68eeffd700 1 -- 172.21.15.87:0/2028809208 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68f4012a40 con 0x7f68f80f6c80 2024-02-23T00:43:06.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.601+0000 7f68eeffd700 1 -- 172.21.15.87:0/2028809208 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f400d980 con 0x7f68f80f6c80 2024-02-23T00:43:06.606 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.602+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68dc004f50 con 0x7f68f80f6c80 2024-02-23T00:43:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:06.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:06.607 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.605+0000 7f68eeffd700 1 -- 172.21.15.87:0/2028809208 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f68f400ddb0 con 0x7f68f80f6c80 2024-02-23T00:43:06.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.605+0000 7f68eeffd700 1 --2- 172.21.15.87:0/2028809208 >> v2:172.21.15.87:6800/1231597463 conn(0x7f68e4062c90 0x7f68e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:06.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.605+0000 7f68eeffd700 1 -- 172.21.15.87:0/2028809208 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(755..755 src has 400..755) v4 ==== 8538+0+0 (secure 0 0 0) 0x7f68f407d600 con 0x7f68f80f6c80 2024-02-23T00:43:06.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.605+0000 7f68eeffd700 10 client.27840.objecter ms_dispatch 0x7f68f8001550 osd_map(755..755 src has 400..755) v4 2024-02-23T00:43:06.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.605+0000 7f68eeffd700 3 client.27840.objecter handle_osd_map got epochs [755,755] > 0 2024-02-23T00:43:06.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.605+0000 7f68eeffd700 3 client.27840.objecter handle_osd_map decoding full epoch 755 2024-02-23T00:43:06.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.605+0000 7f68fd4f0700 1 --2- 172.21.15.87:0/2028809208 >> v2:172.21.15.87:6800/1231597463 conn(0x7f68e4062c90 0x7f68e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:06.608 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.605+0000 7f68eeffd700 20 client.27840.objecter dump_active .. 0 homeless 2024-02-23T00:43:06.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.606+0000 7f68eeffd700 10 client.27840.objecter ms_handle_connect 0x7f68e4062c90 2024-02-23T00:43:06.609 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.606+0000 7f68fd4f0700 1 --2- 172.21.15.87:0/2028809208 >> v2:172.21.15.87:6800/1231597463 conn(0x7f68e4062c90 0x7f68e4065150 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7f68e8004ee0 tx=0x7f68e8009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:06.611 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.609+0000 7f68eeffd700 1 -- 172.21.15.87:0/2028809208 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f68f40508d0 con 0x7f68f80f6c80 2024-02-23T00:43:06.625 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:04.776161+0000 osd.7 (osd.7) 346 : cluster [DBG] 3.11 scrub starts 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:04.777264+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.11 scrub ok 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:05.238494+0000 mon.a (mon.0) 3749 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:05.247447+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-132", "mode": "writeback"}]': finished 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:05.247507+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:05.250677+0000 mon.a (mon.0) 3752 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:05.361517+0000 osd.1 (osd.1) 342 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:05.364141+0000 osd.1 (osd.1) 343 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:05.737149+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.15 scrub starts 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:05.740108+0000 osd.7 (osd.7) 349 : cluster [DBG] 3.15 scrub ok 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.142357+0000 osd.5 (osd.5) 295 : cluster [DBG] 5.f scrub starts 2024-02-23T00:43:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.143451+0000 osd.5 (osd.5) 296 : cluster [DBG] 5.f scrub ok 2024-02-23T00:43:06.708 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:06.739 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:06.760 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:06.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:06.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:06.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:06.789+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f68dc0054d0 con 0x7f68f80f6c80 2024-02-23T00:43:06.864 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:06.897 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:07.145 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:07.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:07.249 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:07.255 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.252+0000 7f68eeffd700 1 -- 172.21.15.87:0/2028809208 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v756) v1 ==== 101+0+0 (secure 0 0 0) 0x7f68f40504c0 con 0x7f68f80f6c80 2024-02-23T00:43:07.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.255+0000 7f68fff55700 10 librados: watch_flush enter 2024-02-23T00:43:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 10 librados: watch_flush exit 2024-02-23T00:43:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 20 client.27840.objecter shutdown clearing up homeless session... 2024-02-23T00:43:07.258 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 10 client.27840.objecter successfully canceled tick 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 >> v2:172.21.15.87:6800/1231597463 conn(0x7f68e4062c90 msgr2=0x7f68e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 --2- 172.21.15.87:0/2028809208 >> v2:172.21.15.87:6800/1231597463 conn(0x7f68e4062c90 0x7f68e4065150 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 rx=0x7f68e8004ee0 tx=0x7f68e8009cb0).stop 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 msgr2=0x7f68f806a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 --2- 172.21.15.87:0/2028809208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 0x7f68f806a370 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 rx=0x7f68f400b420 tx=0x7f68f4006f60).stop 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 shutdown_connections 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 --2- 172.21.15.87:0/2028809208 >> v2:172.21.15.87:6800/1231597463 conn(0x7f68e4062c90 0x7f68e4065150 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 --2- 172.21.15.87:0/2028809208 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f68f8063490 0x7f68f8063910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 --2- 172.21.15.87:0/2028809208 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f68f80f6c80 0x7f68f806a370 secure :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 rx=0x7f68f400b420 tx=0x7f68f4006f60).stop 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 --2- 172.21.15.87:0/2028809208 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f68f81087f0 0x7f68f806a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:07.259 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.256+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 >> 172.21.15.87:0/2028809208 conn(0x7f68f806c370 msgr2=0x7f68f8106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.258+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 shutdown_connections 2024-02-23T00:43:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.259+0000 7f68fff55700 1 -- 172.21.15.87:0/2028809208 wait complete. 2024-02-23T00:43:07.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:07.259+0000 7f68fff55700 1 librados: shutdown 2024-02-23T00:43:07.262 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:43:07.279 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:07.302 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:07.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:07.409 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:07.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:05.793827+0000 mgr.y (mgr.24437) 642 : cluster [DBG] pgmap v1154: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:07.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:06.252029+0000 mon.a (mon.0) 3753 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:43:07.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.252093+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-02-23T00:43:07.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:06.252718+0000 mon.a (mon.0) 3755 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:43:07.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.394856+0000 osd.1 (osd.1) 344 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:43:07.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.397975+0000 osd.1 (osd.1) 345 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:43:07.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.613557+0000 osd.0 (osd.0) 41 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:43:07.434 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.616176+0000 osd.0 (osd.0) 42 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.702354+0000 osd.6 (osd.6) 310 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.705279+0000 osd.6 (osd.6) 311 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:06.791217+0000 mon.a (mon.0) 3756 : audit [INF] from='client.? 172.21.15.87:0/2028809208' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.973542+0000 osd.4 (osd.4) 335 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:06.975082+0000 osd.4 (osd.4) 336 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.130495+0000 osd.5 (osd.5) 297 : cluster [DBG] 22.f scrub starts 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.133003+0000 osd.5 (osd.5) 298 : cluster [DBG] 22.f scrub ok 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.231416+0000 mon.a (mon.0) 3757 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:05.793827+0000 mgr.y (mgr.24437) 642 : cluster [DBG] pgmap v1154: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:06.252029+0000 mon.a (mon.0) 3753 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.252093+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:06.252718+0000 mon.a (mon.0) 3755 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:43:07.435 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.394856+0000 osd.1 (osd.1) 344 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.397975+0000 osd.1 (osd.1) 345 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.613557+0000 osd.0 (osd.0) 41 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.616176+0000 osd.0 (osd.0) 42 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.702354+0000 osd.6 (osd.6) 310 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.705279+0000 osd.6 (osd.6) 311 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:06.791217+0000 mon.a (mon.0) 3756 : audit [INF] from='client.? 172.21.15.87:0/2028809208' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.973542+0000 osd.4 (osd.4) 335 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:06.975082+0000 osd.4 (osd.4) 336 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.130495+0000 osd.5 (osd.5) 297 : cluster [DBG] 22.f scrub starts 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.133003+0000 osd.5 (osd.5) 298 : cluster [DBG] 22.f scrub ok 2024-02-23T00:43:07.436 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.231416+0000 mon.a (mon.0) 3757 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:43:07.437 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:05.793827+0000 mgr.y (mgr.24437) 642 : cluster [DBG] pgmap v1154: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:06.252029+0000 mon.a (mon.0) 3753 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.252093+0000 mon.a (mon.0) 3754 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:06.252718+0000 mon.a (mon.0) 3755 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.394856+0000 osd.1 (osd.1) 344 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.397975+0000 osd.1 (osd.1) 345 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.613557+0000 osd.0 (osd.0) 41 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.616176+0000 osd.0 (osd.0) 42 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.702354+0000 osd.6 (osd.6) 310 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:43:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.705279+0000 osd.6 (osd.6) 311 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:43:07.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:06.791217+0000 mon.a (mon.0) 3756 : audit [INF] from='client.? 172.21.15.87:0/2028809208' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:07.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.973542+0000 osd.4 (osd.4) 335 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:43:07.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:06.975082+0000 osd.4 (osd.4) 336 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:43:07.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.130495+0000 osd.5 (osd.5) 297 : cluster [DBG] 22.f scrub starts 2024-02-23T00:43:07.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.133003+0000 osd.5 (osd.5) 298 : cluster [DBG] 22.f scrub ok 2024-02-23T00:43:07.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.231416+0000 mon.a (mon.0) 3757 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:43:08.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.159007+0000 osd.3 (osd.3) 242 : cluster [DBG] 22.e scrub starts 2024-02-23T00:43:08.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.162079+0000 osd.3 (osd.3) 243 : cluster [DBG] 22.e scrub ok 2024-02-23T00:43:08.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.248850+0000 mon.a (mon.0) 3758 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:43:08.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:07.252798+0000 mon.a (mon.0) 3759 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:07.252864+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? 172.21.15.87:0/2028809208' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.252904+0000 mon.a (mon.0) 3761 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:07.254617+0000 mon.a (mon.0) 3762 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.409423+0000 osd.1 (osd.1) 346 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.413021+0000 osd.1 (osd.1) 347 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.567828+0000 osd.0 (osd.0) 43 : cluster [DBG] 3.12 scrub starts 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.568778+0000 osd.0 (osd.0) 44 : cluster [DBG] 3.12 scrub ok 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.687933+0000 osd.6 (osd.6) 312 : cluster [DBG] 3.10 deep-scrub starts 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.690934+0000 osd.6 (osd.6) 313 : cluster [DBG] 3.10 deep-scrub ok 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:08.250378+0000 mon.a (mon.0) 3763 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:43:08.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.159007+0000 osd.3 (osd.3) 242 : cluster [DBG] 22.e scrub starts 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.162079+0000 osd.3 (osd.3) 243 : cluster [DBG] 22.e scrub ok 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.248850+0000 mon.a (mon.0) 3758 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:07.252798+0000 mon.a (mon.0) 3759 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:07.252864+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? 172.21.15.87:0/2028809208' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.252904+0000 mon.a (mon.0) 3761 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:07.254617+0000 mon.a (mon.0) 3762 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.409423+0000 osd.1 (osd.1) 346 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.413021+0000 osd.1 (osd.1) 347 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.567828+0000 osd.0 (osd.0) 43 : cluster [DBG] 3.12 scrub starts 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.568778+0000 osd.0 (osd.0) 44 : cluster [DBG] 3.12 scrub ok 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.687933+0000 osd.6 (osd.6) 312 : cluster [DBG] 3.10 deep-scrub starts 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.690934+0000 osd.6 (osd.6) 313 : cluster [DBG] 3.10 deep-scrub ok 2024-02-23T00:43:08.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:08.250378+0000 mon.a (mon.0) 3763 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:43:08.612 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:08.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.159007+0000 osd.3 (osd.3) 242 : cluster [DBG] 22.e scrub starts 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.162079+0000 osd.3 (osd.3) 243 : cluster [DBG] 22.e scrub ok 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.248850+0000 mon.a (mon.0) 3758 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:07.252798+0000 mon.a (mon.0) 3759 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:07.252864+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? 172.21.15.87:0/2028809208' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.252904+0000 mon.a (mon.0) 3761 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:07.254617+0000 mon.a (mon.0) 3762 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.409423+0000 osd.1 (osd.1) 346 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.413021+0000 osd.1 (osd.1) 347 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.567828+0000 osd.0 (osd.0) 43 : cluster [DBG] 3.12 scrub starts 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.568778+0000 osd.0 (osd.0) 44 : cluster [DBG] 3.12 scrub ok 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.687933+0000 osd.6 (osd.6) 312 : cluster [DBG] 3.10 deep-scrub starts 2024-02-23T00:43:08.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.690934+0000 osd.6 (osd.6) 313 : cluster [DBG] 3.10 deep-scrub ok 2024-02-23T00:43:08.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:08.250378+0000 mon.a (mon.0) 3763 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:43:09.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:07.794246+0000 mgr.y (mgr.24437) 643 : cluster [DBG] pgmap v1157: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:09.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:08.255236+0000 mon.a (mon.0) 3764 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:43:09.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:08.255296+0000 mon.a (mon.0) 3765 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-02-23T00:43:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:08.256999+0000 mon.a (mon.0) 3766 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:43:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:09.259356+0000 mon.a (mon.0) 3767 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:43:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:09.259441+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-02-23T00:43:09.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:09.261062+0000 mon.a (mon.0) 3769 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-23T00:43:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:07.794246+0000 mgr.y (mgr.24437) 643 : cluster [DBG] pgmap v1157: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:08.255236+0000 mon.a (mon.0) 3764 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:43:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:08.255296+0000 mon.a (mon.0) 3765 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-02-23T00:43:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:08.256999+0000 mon.a (mon.0) 3766 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:43:09.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:09.259356+0000 mon.a (mon.0) 3767 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:43:09.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:09.259441+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-02-23T00:43:09.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:09.261062+0000 mon.a (mon.0) 3769 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-23T00:43:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:07.794246+0000 mgr.y (mgr.24437) 643 : cluster [DBG] pgmap v1157: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:08.255236+0000 mon.a (mon.0) 3764 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:43:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:08.255296+0000 mon.a (mon.0) 3765 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-02-23T00:43:09.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:08.256999+0000 mon.a (mon.0) 3766 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:43:09.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:09.259356+0000 mon.a (mon.0) 3767 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:43:09.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:09.259441+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-02-23T00:43:09.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:09.261062+0000 mon.a (mon.0) 3769 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-23T00:43:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:09.380383+0000 osd.1 (osd.1) 348 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:43:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:09.384746+0000 osd.1 (osd.1) 349 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:43:10.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:09.607899+0000 osd.0 (osd.0) 45 : cluster [DBG] 5.5 deep-scrub starts 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:09.610956+0000 osd.0 (osd.0) 46 : cluster [DBG] 5.5 deep-scrub ok 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:09.657225+0000 mon.a (mon.0) 3770 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:10.257497+0000 mon.a (mon.0) 3771 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:10.257570+0000 mon.a (mon.0) 3772 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:10.259667+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:09.380383+0000 osd.1 (osd.1) 348 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:09.384746+0000 osd.1 (osd.1) 349 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:09.607899+0000 osd.0 (osd.0) 45 : cluster [DBG] 5.5 deep-scrub starts 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:09.610956+0000 osd.0 (osd.0) 46 : cluster [DBG] 5.5 deep-scrub ok 2024-02-23T00:43:10.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:09.657225+0000 mon.a (mon.0) 3770 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:10.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:10.257497+0000 mon.a (mon.0) 3771 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-23T00:43:10.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:10.257570+0000 mon.a (mon.0) 3772 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-02-23T00:43:10.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:10.259667+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-23T00:43:10.687 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:10.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:10.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:09.380383+0000 osd.1 (osd.1) 348 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:43:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:09.384746+0000 osd.1 (osd.1) 349 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:43:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:09.607899+0000 osd.0 (osd.0) 45 : cluster [DBG] 5.5 deep-scrub starts 2024-02-23T00:43:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:09.610956+0000 osd.0 (osd.0) 46 : cluster [DBG] 5.5 deep-scrub ok 2024-02-23T00:43:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:09.657225+0000 mon.a (mon.0) 3770 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:10.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:10.257497+0000 mon.a (mon.0) 3771 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-23T00:43:10.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:10.257570+0000 mon.a (mon.0) 3772 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-02-23T00:43:10.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:10.259667+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-23T00:43:10.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:10.820 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:11.020 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:11.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:11.136 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:11.164 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:11.185 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:11.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:11.275 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:09.794650+0000 mgr.y (mgr.24437) 644 : cluster [DBG] pgmap v1160: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:11.275 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:10.165660+0000 osd.3 (osd.3) 244 : cluster [DBG] 234.1 scrub starts 2024-02-23T00:43:11.276 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:10.169113+0000 osd.3 (osd.3) 245 : cluster [DBG] 234.1 scrub ok 2024-02-23T00:43:11.276 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:10.610003+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:43:11.276 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:10.611494+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:43:11.276 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:10.917108+0000 mon.a (mon.0) 3774 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:43:11.276 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:11.198398+0000 mon.a (mon.0) 3775 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:11.276 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:11.261032+0000 mon.a (mon.0) 3776 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-23T00:43:11.276 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:11.261074+0000 mon.a (mon.0) 3777 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-02-23T00:43:11.276 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:09.794650+0000 mgr.y (mgr.24437) 644 : cluster [DBG] pgmap v1160: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:11.276 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:10.165660+0000 osd.3 (osd.3) 244 : cluster [DBG] 234.1 scrub starts 2024-02-23T00:43:11.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:10.169113+0000 osd.3 (osd.3) 245 : cluster [DBG] 234.1 scrub ok 2024-02-23T00:43:11.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:10.610003+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:43:11.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:10.611494+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:43:11.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:10.917108+0000 mon.a (mon.0) 3774 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:43:11.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:11.198398+0000 mon.a (mon.0) 3775 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:11.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:11.261032+0000 mon.a (mon.0) 3776 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-23T00:43:11.277 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:11.261074+0000 mon.a (mon.0) 3777 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-02-23T00:43:11.288 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 -- 172.21.15.87:0/1598076379 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c0f8cb0 msgr2=0x7f337c0f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 --2- 172.21.15.87:0/1598076379 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c0f8cb0 0x7f337c0f9090 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 rx=0x7f336c004870 tx=0x7f336c009cb0).stop 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 -- 172.21.15.87:0/1598076379 shutdown_connections 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 --2- 172.21.15.87:0/1598076379 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f337c102210 0x7f337c104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 --2- 172.21.15.87:0/1598076379 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c0f8cb0 0x7f337c0f9090 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 --2- 172.21.15.87:0/1598076379 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f337c0f9610 0x7f337c101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 -- 172.21.15.87:0/1598076379 >> 172.21.15.87:0/1598076379 conn(0x7f337c0f2820 msgr2=0x7f337c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 -- 172.21.15.87:0/1598076379 shutdown_connections 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 -- 172.21.15.87:0/1598076379 wait complete. 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 librados: starting msgr at 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.240+0000 7f33825f3700 1 librados: starting objecter 2024-02-23T00:43:11.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 Processor -- start 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 -- start start 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 librados: setting wanted keys 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 librados: calling monclient init 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f337c0f8cb0 0x7f337c0f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f337c0f9610 0x7f337c0f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c102210 0x7f337c0f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c1166e0 con 0x7f337c102210 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337c116850 con 0x7f337c0f8cb0 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f33825f3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337c116a10 con 0x7f337c0f9610 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f337b7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f337c0f9610 0x7f337c0f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f3380b90700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c102210 0x7f337c0f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f3380b90700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c102210 0x7f337c0f7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:47250/0 (socket says 172.21.15.87:47250) 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f337b7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f337c0f9610 0x7f337c0f7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:38164/0 (socket says 172.21.15.87:38164) 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f3380b90700 1 -- 172.21.15.87:0/1531016467 learned_addr learned my addr 172.21.15.87:0/1531016467 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:11.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f337bfff700 1 --2- 172.21.15.87:0/1531016467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f337c0f8cb0 0x7f337c0f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f337b7fe700 1 -- 172.21.15.87:0/1531016467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f337c0f8cb0 msgr2=0x7f337c0f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f337b7fe700 1 --2- 172.21.15.87:0/1531016467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f337c0f8cb0 0x7f337c0f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f337b7fe700 1 -- 172.21.15.87:0/1531016467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c102210 msgr2=0x7f337c0f7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f337b7fe700 1 --2- 172.21.15.87:0/1531016467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c102210 0x7f337c0f7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f337b7fe700 1 -- 172.21.15.87:0/1531016467 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370005040 con 0x7f337c0f9610 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.241+0000 7f3380b90700 1 --2- 172.21.15.87:0/1531016467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c102210 0x7f337c0f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f337b7fe700 1 --2- 172.21.15.87:0/1531016467 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f337c0f9610 0x7f337c0f7150 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7f3370004920 tx=0x7f3370009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f33797fa700 10 client.?.objecter ms_handle_connect 0x7f337c0f9610 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f33797fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f33797fa700 1 -- 172.21.15.87:0/1531016467 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337000e030 con 0x7f337c0f9610 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c005040 con 0x7f337c0f9610 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f33797fa700 1 -- 172.21.15.87:0/1531016467 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f337000f040 con 0x7f337c0f9610 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f33797fa700 1 -- 172.21.15.87:0/1531016467 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370014850 con 0x7f337c0f9610 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f33825f3700 10 client.38321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337c1170a0 con 0x7f337c0f9610 2024-02-23T00:43:11.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.242+0000 7f33825f3700 1 librados: init done 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.243+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337c05fa60 con 0x7f337c0f9610 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.243+0000 7f33797fa700 1 -- 172.21.15.87:0/1531016467 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3370014c80 con 0x7f337c0f9610 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.243+0000 7f33797fa700 1 --2- 172.21.15.87:0/1531016467 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3364062c90 0x7f3364065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.244+0000 7f33797fa700 1 -- 172.21.15.87:0/1531016467 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(759..759 src has 400..759) v4 ==== 8564+0+0 (secure 0 0 0) 0x7f337007d600 con 0x7f337c0f9610 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.244+0000 7f33797fa700 10 client.38321.objecter ms_dispatch 0x7f337c001550 osd_map(759..759 src has 400..759) v4 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.244+0000 7f33797fa700 3 client.38321.objecter handle_osd_map got epochs [759,759] > 0 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.244+0000 7f33797fa700 3 client.38321.objecter handle_osd_map decoding full epoch 759 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.244+0000 7f337bfff700 1 --2- 172.21.15.87:0/1531016467 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3364062c90 0x7f3364065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.244+0000 7f33797fa700 20 client.38321.objecter dump_active .. 0 homeless 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.244+0000 7f33797fa700 10 client.38321.objecter ms_handle_connect 0x7f3364062c90 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.244+0000 7f337bfff700 1 --2- 172.21.15.87:0/1531016467 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3364062c90 0x7f3364065150 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7f336c0043d0 tx=0x7f336c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:11.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.248+0000 7f33797fa700 1 -- 172.21.15.87:0/1531016467 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3370050a20 con 0x7f337c0f9610 2024-02-23T00:43:11.318 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:11.339 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:11.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:11.442 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:43:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:09.794650+0000 mgr.y (mgr.24437) 644 : cluster [DBG] pgmap v1160: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:10.165660+0000 osd.3 (osd.3) 244 : cluster [DBG] 234.1 scrub starts 2024-02-23T00:43:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:10.169113+0000 osd.3 (osd.3) 245 : cluster [DBG] 234.1 scrub ok 2024-02-23T00:43:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:10.610003+0000 osd.0 (osd.0) 47 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:43:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:10.611494+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:43:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:10.917108+0000 mon.a (mon.0) 3774 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:43:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:11.198398+0000 mon.a (mon.0) 3775 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:11.261032+0000 mon.a (mon.0) 3776 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-23T00:43:11.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:11.261074+0000 mon.a (mon.0) 3777 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-02-23T00:43:11.444 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:11.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.440+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f337c0f8140 con 0x7f3364062c90 2024-02-23T00:43:11.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.455+0000 7f33797fa700 1 -- 172.21.15.87:0/1531016467 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+527440 (secure 0 0 0) 0x7f337c0f8140 con 0x7f3364062c90 2024-02-23T00:43:11.464 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:43:11.468 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 10 librados: watch_flush enter 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 10 librados: watch_flush exit 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 20 client.38321.objecter shutdown clearing up homeless session... 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 10 client.38321.objecter successfully canceled tick 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3364062c90 msgr2=0x7f3364065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 --2- 172.21.15.87:0/1531016467 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3364062c90 0x7f3364065150 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7f336c0043d0 tx=0x7f336c00b040).stop 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f337c0f9610 msgr2=0x7f337c0f7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 --2- 172.21.15.87:0/1531016467 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f337c0f9610 0x7f337c0f7150 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 rx=0x7f3370004920 tx=0x7f3370009cb0).stop 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 shutdown_connections 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 --2- 172.21.15.87:0/1531016467 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3364062c90 0x7f3364065150 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 --2- 172.21.15.87:0/1531016467 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f337c102210 0x7f337c0f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 --2- 172.21.15.87:0/1531016467 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f337c0f8cb0 0x7f337c0f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 --2- 172.21.15.87:0/1531016467 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f337c0f9610 0x7f337c0f7150 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 >> 172.21.15.87:0/1531016467 conn(0x7f337c0f2820 msgr2=0x7f337c0f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 shutdown_connections 2024-02-23T00:43:11.469 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 -- 172.21.15.87:0/1531016467 wait complete. 2024-02-23T00:43:11.470 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:11.459+0000 7f33825f3700 1 librados: shutdown 2024-02-23T00:43:11.470 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:43:11.470 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:11.488 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:11.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:11.592 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:11.619 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:12.169 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:12.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:12.272 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:12.300 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:12.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:12] "GET /metrics HTTP/1.1" 200 190597 "" "Prometheus/2.33.4" 2024-02-23T00:43:12.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:10.746821+0000 osd.6 (osd.6) 314 : cluster [DBG] 234.6 deep-scrub starts 2024-02-23T00:43:12.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:10.751613+0000 osd.6 (osd.6) 315 : cluster [DBG] 234.6 deep-scrub ok 2024-02-23T00:43:12.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:10.808292+0000 osd.2 (osd.2) 275 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:10.808449+0000 osd.2 (osd.2) 276 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:11.274846+0000 mgr.y (mgr.24437) 645 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:11.302014+0000 mon.a (mon.0) 3778 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:11.399412+0000 osd.1 (osd.1) 350 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:11.402328+0000 osd.1 (osd.1) 351 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:11.568539+0000 osd.0 (osd.0) 49 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:11.571653+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:11.572102+0000 mon.a (mon.0) 3779 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:11.572805+0000 mon.a (mon.0) 3780 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:11.576923+0000 mon.a (mon.0) 3781 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:12.059724+0000 osd.5 (osd.5) 299 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:12.060580+0000 osd.5 (osd.5) 300 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:12.239015+0000 mon.a (mon.0) 3782 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:43:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:12.239080+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:12.242897+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132"}]: dispatch 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:10.746821+0000 osd.6 (osd.6) 314 : cluster [DBG] 234.6 deep-scrub starts 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:10.751613+0000 osd.6 (osd.6) 315 : cluster [DBG] 234.6 deep-scrub ok 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:10.808292+0000 osd.2 (osd.2) 275 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:10.808449+0000 osd.2 (osd.2) 276 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:11.274846+0000 mgr.y (mgr.24437) 645 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:11.302014+0000 mon.a (mon.0) 3778 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:11.399412+0000 osd.1 (osd.1) 350 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:11.402328+0000 osd.1 (osd.1) 351 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:11.568539+0000 osd.0 (osd.0) 49 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:11.571653+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:43:12.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:11.572102+0000 mon.a (mon.0) 3779 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:43:12.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:11.572805+0000 mon.a (mon.0) 3780 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:43:12.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:11.576923+0000 mon.a (mon.0) 3781 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:12.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:12.059724+0000 osd.5 (osd.5) 299 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:43:12.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:12.060580+0000 osd.5 (osd.5) 300 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:43:12.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:12.239015+0000 mon.a (mon.0) 3782 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:43:12.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:12.239080+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-02-23T00:43:12.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:12.242897+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132"}]: dispatch 2024-02-23T00:43:12.661 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:43:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:10.746821+0000 osd.6 (osd.6) 314 : cluster [DBG] 234.6 deep-scrub starts 2024-02-23T00:43:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:10.751613+0000 osd.6 (osd.6) 315 : cluster [DBG] 234.6 deep-scrub ok 2024-02-23T00:43:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:10.808292+0000 osd.2 (osd.2) 275 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:10.808449+0000 osd.2 (osd.2) 276 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:11.274846+0000 mgr.y (mgr.24437) 645 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:11.302014+0000 mon.a (mon.0) 3778 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:11.399412+0000 osd.1 (osd.1) 350 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:11.402328+0000 osd.1 (osd.1) 351 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:11.568539+0000 osd.0 (osd.0) 49 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:11.571653+0000 osd.0 (osd.0) 50 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:11.572102+0000 mon.a (mon.0) 3779 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:11.572805+0000 mon.a (mon.0) 3780 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:11.576923+0000 mon.a (mon.0) 3781 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:12.059724+0000 osd.5 (osd.5) 299 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:12.060580+0000 osd.5 (osd.5) 300 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:12.239015+0000 mon.a (mon.0) 3782 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:43:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:12.239080+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-02-23T00:43:12.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:12.242897+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132"}]: dispatch 2024-02-23T00:43:12.698 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1162,"stamp":"2024-02-23T00:43:11.258986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4006105,"num_objects":266,"num_object_clones":1,"num_object_copies":798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":266,"num_whiteouts":0,"num_read":1347,"num_read_kb":1206,"num_write":483,"num_write_kb":3988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":6783,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":930,"ondisk_log_size":930,"up":612,"acting":612,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":613,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2481632,"kb_used_data":57184,"kb_used_omap":85,"kb_used_meta":2424298,"kb_avail":747479584,"statfs":{"total":767960285184,"available":765419094016,"internally_reserved":0,"allocated":58556416,"data_stored":33999330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":2482481280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-49,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-5,"num_read_kb":-5,"num_write":-1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008541"},"pg_stats":[{"pgid":"250.1f","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.660530+0000","last_change":"2024-02-23T00:43:02.253586+0000","last_active":"2024-02-23T00:43:08.660530+0000","last_peered":"2024-02-23T00:43:08.660530+0000","last_clean":"2024-02-23T00:43:08.660530+0000","last_became_active":"2024-02-23T00:43:02.252698+0000","last_became_peered":"2024-02-23T00:43:02.252698+0000","last_unstale":"2024-02-23T00:43:08.660530+0000","last_undegraded":"2024-02-23T00:43:08.660530+0000","last_fullsized":"2024-02-23T00:43:08.660530+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"250.1e","version":"0'0","reported_seq":17,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.384470+0000","last_change":"2024-02-23T00:43:02.259289+0000","last_active":"2024-02-23T00:43:09.384470+0000","last_peered":"2024-02-23T00:43:09.384470+0000","last_clean":"2024-02-23T00:43:09.384470+0000","last_became_active":"2024-02-23T00:43:02.258933+0000","last_became_peered":"2024-02-23T00:43:02.258933+0000","last_unstale":"2024-02-23T00:43:09.384470+0000","last_undegraded":"2024-02-23T00:43:09.384470+0000","last_fullsized":"2024-02-23T00:43:09.384470+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"250.1d","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627926+0000","last_change":"2024-02-23T00:43:02.254941+0000","last_active":"2024-02-23T00:43:08.627926+0000","last_peered":"2024-02-23T00:43:08.627926+0000","last_clean":"2024-02-23T00:43:08.627926+0000","last_became_active":"2024-02-23T00:43:02.254568+0000","last_became_peered":"2024-02-23T00:43:02.254568+0000","last_unstale":"2024-02-23T00:43:08.627926+0000","last_undegraded":"2024-02-23T00:43:08.627926+0000","last_fullsized":"2024-02-23T00:43:08.627926+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"250.1c","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662397+0000","last_change":"2024-02-23T00:43:02.252880+0000","last_active":"2024-02-23T00:43:08.662397+0000","last_peered":"2024-02-23T00:43:08.662397+0000","last_clean":"2024-02-23T00:43:08.662397+0000","last_became_active":"2024-02-23T00:43:02.252636+0000","last_became_peered":"2024-02-23T00:43:02.252636+0000","last_unstale":"2024-02-23T00:43:08.662397+0000","last_undegraded":"2024-02-23T00:43:08.662397+0000","last_fullsized":"2024-02-23T00:43:08.662397+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"250.1b","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627728+0000","last_change":"2024-02-23T00:43:02.254348+0000","last_active":"2024-02-23T00:43:08.627728+0000","last_peered":"2024-02-23T00:43:08.627728+0000","last_clean":"2024-02-23T00:43:08.627728+0000","last_became_active":"2024-02-23T00:43:02.254004+0000","last_became_peered":"2024-02-23T00:43:02.254004+0000","last_unstale":"2024-02-23T00:43:08.627728+0000","last_undegraded":"2024-02-23T00:43:08.627728+0000","last_fullsized":"2024-02-23T00:43:08.627728+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"250.1a","version":"0'0","reported_seq":14,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.359634+0000","last_change":"2024-02-23T00:43:02.243415+0000","last_active":"2024-02-23T00:43:06.359634+0000","last_peered":"2024-02-23T00:43:06.359634+0000","last_clean":"2024-02-23T00:43:06.359634+0000","last_became_active":"2024-02-23T00:43:02.243114+0000","last_became_peered":"2024-02-23T00:43:02.243114+0000","last_unstale":"2024-02-23T00:43:06.359634+0000","last_undegraded":"2024-02-23T00:43:06.359634+0000","last_fullsized":"2024-02-23T00:43:06.359634+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"250.19","version":"0'0","reported_seq":17,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258881+0000","last_change":"2024-02-23T00:43:02.254754+0000","last_active":"2024-02-23T00:43:09.258881+0000","last_peered":"2024-02-23T00:43:09.258881+0000","last_clean":"2024-02-23T00:43:09.258881+0000","last_became_active":"2024-02-23T00:43:02.254548+0000","last_became_peered":"2024-02-23T00:43:02.254548+0000","last_unstale":"2024-02-23T00:43:09.258881+0000","last_undegraded":"2024-02-23T00:43:09.258881+0000","last_fullsized":"2024-02-23T00:43:09.258881+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"250.18","version":"0'0","reported_seq":14,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.359896+0000","last_change":"2024-02-23T00:43:02.251783+0000","last_active":"2024-02-23T00:43:06.359896+0000","last_peered":"2024-02-23T00:43:06.359896+0000","last_clean":"2024-02-23T00:43:06.359896+0000","last_became_active":"2024-02-23T00:43:02.251499+0000","last_became_peered":"2024-02-23T00:43:02.251499+0000","last_unstale":"2024-02-23T00:43:06.359896+0000","last_undegraded":"2024-02-23T00:43:06.359896+0000","last_fullsized":"2024-02-23T00:43:06.359896+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"250.f","version":"0'0","reported_seq":17,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260447+0000","last_change":"2024-02-23T00:43:02.256125+0000","last_active":"2024-02-23T00:43:09.260447+0000","last_peered":"2024-02-23T00:43:09.260447+0000","last_clean":"2024-02-23T00:43:09.260447+0000","last_became_active":"2024-02-23T00:43:02.255764+0000","last_became_peered":"2024-02-23T00:43:02.255764+0000","last_unstale":"2024-02-23T00:43:09.260447+0000","last_undegraded":"2024-02-23T00:43:09.260447+0000","last_fullsized":"2024-02-23T00:43:09.260447+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"250.e","version":"0'0","reported_seq":14,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.361680+0000","last_change":"2024-02-23T00:43:02.251139+0000","last_active":"2024-02-23T00:43:06.361680+0000","last_peered":"2024-02-23T00:43:06.361680+0000","last_clean":"2024-02-23T00:43:06.361680+0000","last_became_active":"2024-02-23T00:43:02.250869+0000","last_became_peered":"2024-02-23T00:43:02.250869+0000","last_unstale":"2024-02-23T00:43:06.361680+0000","last_undegraded":"2024-02-23T00:43:06.361680+0000","last_fullsized":"2024-02-23T00:43:06.361680+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"250.d","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.660669+0000","last_change":"2024-02-23T00:43:02.249364+0000","last_active":"2024-02-23T00:43:08.660669+0000","last_peered":"2024-02-23T00:43:08.660669+0000","last_clean":"2024-02-23T00:43:08.660669+0000","last_became_active":"2024-02-23T00:43:02.249209+0000","last_became_peered":"2024-02-23T00:43:02.249209+0000","last_unstale":"2024-02-23T00:43:08.660669+0000","last_undegraded":"2024-02-23T00:43:08.660669+0000","last_fullsized":"2024-02-23T00:43:08.660669+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"250.c","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256778+0000","last_change":"2024-02-23T00:43:02.254815+0000","last_active":"2024-02-23T00:43:08.256778+0000","last_peered":"2024-02-23T00:43:08.256778+0000","last_clean":"2024-02-23T00:43:08.256778+0000","last_became_active":"2024-02-23T00:43:02.254470+0000","last_became_peered":"2024-02-23T00:43:02.254470+0000","last_unstale":"2024-02-23T00:43:08.256778+0000","last_undegraded":"2024-02-23T00:43:08.256778+0000","last_fullsized":"2024-02-23T00:43:08.256778+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"250.b","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256791+0000","last_change":"2024-02-23T00:43:02.254475+0000","last_active":"2024-02-23T00:43:08.256791+0000","last_peered":"2024-02-23T00:43:08.256791+0000","last_clean":"2024-02-23T00:43:08.256791+0000","last_became_active":"2024-02-23T00:43:02.254185+0000","last_became_peered":"2024-02-23T00:43:02.254185+0000","last_unstale":"2024-02-23T00:43:08.256791+0000","last_undegraded":"2024-02-23T00:43:08.256791+0000","last_fullsized":"2024-02-23T00:43:08.256791+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"250.a","version":"0'0","reported_seq":14,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.359835+0000","last_change":"2024-02-23T00:43:02.259745+0000","last_active":"2024-02-23T00:43:06.359835+0000","last_peered":"2024-02-23T00:43:06.359835+0000","last_clean":"2024-02-23T00:43:06.359835+0000","last_became_active":"2024-02-23T00:43:02.259447+0000","last_became_peered":"2024-02-23T00:43:02.259447+0000","last_unstale":"2024-02-23T00:43:06.359835+0000","last_undegraded":"2024-02-23T00:43:06.359835+0000","last_fullsized":"2024-02-23T00:43:06.359835+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"250.9","version":"0'0","reported_seq":17,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259334+0000","last_change":"2024-02-23T00:43:02.253989+0000","last_active":"2024-02-23T00:43:09.259334+0000","last_peered":"2024-02-23T00:43:09.259334+0000","last_clean":"2024-02-23T00:43:09.259334+0000","last_became_active":"2024-02-23T00:43:02.253729+0000","last_became_peered":"2024-02-23T00:43:02.253729+0000","last_unstale":"2024-02-23T00:43:09.259334+0000","last_undegraded":"2024-02-23T00:43:09.259334+0000","last_fullsized":"2024-02-23T00:43:09.259334+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"250.8","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661828+0000","last_change":"2024-02-23T00:43:02.257652+0000","last_active":"2024-02-23T00:43:08.661828+0000","last_peered":"2024-02-23T00:43:08.661828+0000","last_clean":"2024-02-23T00:43:08.661828+0000","last_became_active":"2024-02-23T00:43:02.257324+0000","last_became_peered":"2024-02-23T00:43:02.257324+0000","last_unstale":"2024-02-23T00:43:08.661828+0000","last_undegraded":"2024-02-23T00:43:08.661828+0000","last_fullsized":"2024-02-23T00:43:08.661828+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"250.7","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.663919+0000","last_change":"2024-02-23T00:43:02.262022+0000","last_active":"2024-02-23T00:43:08.663919+0000","last_peered":"2024-02-23T00:43:08.663919+0000","last_clean":"2024-02-23T00:43:08.663919+0000","last_became_active":"2024-02-23T00:43:02.261716+0000","last_became_peered":"2024-02-23T00:43:02.261716+0000","last_unstale":"2024-02-23T00:43:08.663919+0000","last_undegraded":"2024-02-23T00:43:08.663919+0000","last_fullsized":"2024-02-23T00:43:08.663919+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"250.6","version":"0'0","reported_seq":17,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258712+0000","last_change":"2024-02-23T00:43:02.253543+0000","last_active":"2024-02-23T00:43:09.258712+0000","last_peered":"2024-02-23T00:43:09.258712+0000","last_clean":"2024-02-23T00:43:09.258712+0000","last_became_active":"2024-02-23T00:43:02.253204+0000","last_became_peered":"2024-02-23T00:43:02.253204+0000","last_unstale":"2024-02-23T00:43:09.258712+0000","last_undegraded":"2024-02-23T00:43:09.258712+0000","last_fullsized":"2024-02-23T00:43:09.258712+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"250.5","version":"0'0","reported_seq":17,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259326+0000","last_change":"2024-02-23T00:43:02.248832+0000","last_active":"2024-02-23T00:43:09.259326+0000","last_peered":"2024-02-23T00:43:09.259326+0000","last_clean":"2024-02-23T00:43:09.259326+0000","last_became_active":"2024-02-23T00:43:02.248541+0000","last_became_peered":"2024-02-23T00:43:02.248541+0000","last_unstale":"2024-02-23T00:43:09.259326+0000","last_undegraded":"2024-02-23T00:43:09.259326+0000","last_fullsized":"2024-02-23T00:43:09.259326+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"250.4","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627709+0000","last_change":"2024-02-23T00:43:02.253994+0000","last_active":"2024-02-23T00:43:08.627709+0000","last_peered":"2024-02-23T00:43:08.627709+0000","last_clean":"2024-02-23T00:43:08.627709+0000","last_became_active":"2024-02-23T00:43:02.253606+0000","last_became_peered":"2024-02-23T00:43:02.253606+0000","last_unstale":"2024-02-23T00:43:08.627709+0000","last_undegraded":"2024-02-23T00:43:08.627709+0000","last_fullsized":"2024-02-23T00:43:08.627709+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"250.3","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661868+0000","last_change":"2024-02-23T00:43:02.253306+0000","last_active":"2024-02-23T00:43:08.661868+0000","last_peered":"2024-02-23T00:43:08.661868+0000","last_clean":"2024-02-23T00:43:08.661868+0000","last_became_active":"2024-02-23T00:43:02.253008+0000","last_became_peered":"2024-02-23T00:43:02.253008+0000","last_unstale":"2024-02-23T00:43:08.661868+0000","last_undegraded":"2024-02-23T00:43:08.661868+0000","last_fullsized":"2024-02-23T00:43:08.661868+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"250.2","version":"0'0","reported_seq":17,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258735+0000","last_change":"2024-02-23T00:43:02.247788+0000","last_active":"2024-02-23T00:43:09.258735+0000","last_peered":"2024-02-23T00:43:09.258735+0000","last_clean":"2024-02-23T00:43:09.258735+0000","last_became_active":"2024-02-23T00:43:02.247503+0000","last_became_peered":"2024-02-23T00:43:02.247503+0000","last_unstale":"2024-02-23T00:43:09.258735+0000","last_undegraded":"2024-02-23T00:43:09.258735+0000","last_fullsized":"2024-02-23T00:43:09.258735+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"250.1","version":"0'0","reported_seq":17,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.384926+0000","last_change":"2024-02-23T00:43:02.247406+0000","last_active":"2024-02-23T00:43:09.384926+0000","last_peered":"2024-02-23T00:43:09.384926+0000","last_clean":"2024-02-23T00:43:09.384926+0000","last_became_active":"2024-02-23T00:43:02.247095+0000","last_became_peered":"2024-02-23T00:43:02.247095+0000","last_unstale":"2024-02-23T00:43:09.384926+0000","last_undegraded":"2024-02-23T00:43:09.384926+0000","last_fullsized":"2024-02-23T00:43:09.384926+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"250.0","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627166+0000","last_change":"2024-02-23T00:43:02.255383+0000","last_active":"2024-02-23T00:43:08.627166+0000","last_peered":"2024-02-23T00:43:08.627166+0000","last_clean":"2024-02-23T00:43:08.627166+0000","last_became_active":"2024-02-23T00:43:02.255047+0000","last_became_peered":"2024-02-23T00:43:02.255047+0000","last_unstale":"2024-02-23T00:43:08.627166+0000","last_undegraded":"2024-02-23T00:43:08.627166+0000","last_fullsized":"2024-02-23T00:43:08.627166+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"250.17","version":"0'0","reported_seq":14,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.361398+0000","last_change":"2024-02-23T00:43:02.250392+0000","last_active":"2024-02-23T00:43:06.361398+0000","last_peered":"2024-02-23T00:43:06.361398+0000","last_clean":"2024-02-23T00:43:06.361398+0000","last_became_active":"2024-02-23T00:43:02.250071+0000","last_became_peered":"2024-02-23T00:43:02.250071+0000","last_unstale":"2024-02-23T00:43:06.361398+0000","last_undegraded":"2024-02-23T00:43:06.361398+0000","last_fullsized":"2024-02-23T00:43:06.361398+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.7","version":"689'9","reported_seq":207,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.664158+0000","last_change":"2024-02-23T00:42:21.267470+0000","last_active":"2024-02-23T00:43:08.664158+0000","last_peered":"2024-02-23T00:43:08.664158+0000","last_clean":"2024-02-23T00:43:08.664158+0000","last_became_active":"2024-02-23T00:40:17.211405+0000","last_became_peered":"2024-02-23T00:40:17.211405+0000","last_unstale":"2024-02-23T00:43:08.664158+0000","last_undegraded":"2024-02-23T00:43:08.664158+0000","last_fullsized":"2024-02-23T00:43:08.664158+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'9","last_scrub_stamp":"2024-02-23T00:42:21.267354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:21.267354+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"250.16","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257390+0000","last_change":"2024-02-23T00:43:02.250662+0000","last_active":"2024-02-23T00:43:08.257390+0000","last_peered":"2024-02-23T00:43:08.257390+0000","last_clean":"2024-02-23T00:43:08.257390+0000","last_became_active":"2024-02-23T00:43:02.250356+0000","last_became_peered":"2024-02-23T00:43:02.250356+0000","last_unstale":"2024-02-23T00:43:08.257390+0000","last_undegraded":"2024-02-23T00:43:08.257390+0000","last_fullsized":"2024-02-23T00:43:08.257390+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.6","version":"751'19","reported_seq":218,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628023+0000","last_change":"2024-02-23T00:42:14.256795+0000","last_active":"2024-02-23T00:43:08.628023+0000","last_peered":"2024-02-23T00:43:08.628023+0000","last_clean":"2024-02-23T00:43:08.628023+0000","last_became_active":"2024-02-23T00:40:17.215263+0000","last_became_peered":"2024-02-23T00:40:17.215263+0000","last_unstale":"2024-02-23T00:43:08.628023+0000","last_undegraded":"2024-02-23T00:43:08.628023+0000","last_fullsized":"2024-02-23T00:43:08.628023+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'8","last_scrub_stamp":"2024-02-23T00:41:36.605800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:41:36.605800+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"250.15","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628044+0000","last_change":"2024-02-23T00:43:02.258641+0000","last_active":"2024-02-23T00:43:08.628044+0000","last_peered":"2024-02-23T00:43:08.628044+0000","last_clean":"2024-02-23T00:43:08.628044+0000","last_became_active":"2024-02-23T00:43:02.258333+0000","last_became_peered":"2024-02-23T00:43:02.258333+0000","last_unstale":"2024-02-23T00:43:08.628044+0000","last_undegraded":"2024-02-23T00:43:08.628044+0000","last_fullsized":"2024-02-23T00:43:08.628044+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.5","version":"751'7","reported_seq":223,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259555+0000","last_change":"2024-02-23T00:42:13.431865+0000","last_active":"2024-02-23T00:43:09.259555+0000","last_peered":"2024-02-23T00:43:09.259555+0000","last_clean":"2024-02-23T00:43:09.259555+0000","last_became_active":"2024-02-23T00:41:37.238894+0000","last_became_peered":"2024-02-23T00:41:37.238894+0000","last_unstale":"2024-02-23T00:43:09.259555+0000","last_undegraded":"2024-02-23T00:43:09.259555+0000","last_fullsized":"2024-02-23T00:43:09.259555+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"635'5","last_scrub_stamp":"2024-02-23T00:42:13.431791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.431791+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"250.14","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662416+0000","last_change":"2024-02-23T00:43:02.254701+0000","last_active":"2024-02-23T00:43:08.662416+0000","last_peered":"2024-02-23T00:43:08.662416+0000","last_clean":"2024-02-23T00:43:08.662416+0000","last_became_active":"2024-02-23T00:43:02.254427+0000","last_became_peered":"2024-02-23T00:43:02.254427+0000","last_unstale":"2024-02-23T00:43:08.662416+0000","last_undegraded":"2024-02-23T00:43:08.662416+0000","last_fullsized":"2024-02-23T00:43:08.662416+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.4","version":"751'23","reported_seq":216,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662500+0000","last_change":"2024-02-23T00:42:12.255652+0000","last_active":"2024-02-23T00:43:08.662500+0000","last_peered":"2024-02-23T00:43:08.662500+0000","last_clean":"2024-02-23T00:43:08.662500+0000","last_became_active":"2024-02-23T00:40:17.215235+0000","last_became_peered":"2024-02-23T00:40:17.215235+0000","last_unstale":"2024-02-23T00:43:08.662500+0000","last_undegraded":"2024-02-23T00:43:08.662500+0000","last_fullsized":"2024-02-23T00:43:08.662500+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000047,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":3920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"250.13","version":"0'0","reported_seq":16,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661942+0000","last_change":"2024-02-23T00:43:02.250314+0000","last_active":"2024-02-23T00:43:08.661942+0000","last_peered":"2024-02-23T00:43:08.661942+0000","last_clean":"2024-02-23T00:43:08.661942+0000","last_became_active":"2024-02-23T00:43:02.250005+0000","last_became_peered":"2024-02-23T00:43:02.250005+0000","last_unstale":"2024-02-23T00:43:08.661942+0000","last_undegraded":"2024-02-23T00:43:08.661942+0000","last_fullsized":"2024-02-23T00:43:08.661942+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.3","version":"751'7","reported_seq":201,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256581+0000","last_change":"2024-02-23T00:42:18.042228+0000","last_active":"2024-02-23T00:43:08.256581+0000","last_peered":"2024-02-23T00:43:08.256581+0000","last_clean":"2024-02-23T00:43:08.256581+0000","last_became_active":"2024-02-23T00:40:17.202244+0000","last_became_peered":"2024-02-23T00:40:17.202244+0000","last_unstale":"2024-02-23T00:43:08.256581+0000","last_undegraded":"2024-02-23T00:43:08.256581+0000","last_fullsized":"2024-02-23T00:43:08.256581+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'2","last_scrub_stamp":"2024-02-23T00:42:18.042156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.042156+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"250.12","version":"0'0","reported_seq":14,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.359739+0000","last_change":"2024-02-23T00:43:02.243333+0000","last_active":"2024-02-23T00:43:06.359739+0000","last_peered":"2024-02-23T00:43:06.359739+0000","last_clean":"2024-02-23T00:43:06.359739+0000","last_became_active":"2024-02-23T00:43:02.243052+0000","last_became_peered":"2024-02-23T00:43:02.243052+0000","last_unstale":"2024-02-23T00:43:06.359739+0000","last_undegraded":"2024-02-23T00:43:06.359739+0000","last_fullsized":"2024-02-23T00:43:06.359739+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.2","version":"751'10","reported_seq":225,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627262+0000","last_change":"2024-02-23T00:42:12.249043+0000","last_active":"2024-02-23T00:43:08.627262+0000","last_peered":"2024-02-23T00:43:08.627262+0000","last_clean":"2024-02-23T00:43:08.627262+0000","last_became_active":"2024-02-23T00:41:37.219553+0000","last_became_peered":"2024-02-23T00:41:37.219553+0000","last_unstale":"2024-02-23T00:43:08.627262+0000","last_undegraded":"2024-02-23T00:43:08.627262+0000","last_fullsized":"2024-02-23T00:43:08.627262+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"679'6","last_scrub_stamp":"2024-02-23T00:41:43.619119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:41:43.619119+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"250.11","version":"0'0","reported_seq":17,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259115+0000","last_change":"2024-02-23T00:43:02.250159+0000","last_active":"2024-02-23T00:43:09.259115+0000","last_peered":"2024-02-23T00:43:09.259115+0000","last_clean":"2024-02-23T00:43:09.259115+0000","last_became_active":"2024-02-23T00:43:02.249784+0000","last_became_peered":"2024-02-23T00:43:02.249784+0000","last_unstale":"2024-02-23T00:43:09.259115+0000","last_undegraded":"2024-02-23T00:43:09.259115+0000","last_fullsized":"2024-02-23T00:43:09.259115+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.1","version":"751'4","reported_seq":218,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:10.169193+0000","last_change":"2024-02-23T00:43:10.169193+0000","last_active":"2024-02-23T00:43:10.169193+0000","last_peered":"2024-02-23T00:43:10.169193+0000","last_clean":"2024-02-23T00:43:10.169193+0000","last_became_active":"2024-02-23T00:41:37.231330+0000","last_became_peered":"2024-02-23T00:41:37.231330+0000","last_unstale":"2024-02-23T00:43:10.169193+0000","last_undegraded":"2024-02-23T00:43:10.169193+0000","last_fullsized":"2024-02-23T00:43:10.169193+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"751'4","last_scrub_stamp":"2024-02-23T00:43:10.169118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:43:10.169118+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"250.10","version":"0'0","reported_seq":14,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.359404+0000","last_change":"2024-02-23T00:43:02.240397+0000","last_active":"2024-02-23T00:43:06.359404+0000","last_peered":"2024-02-23T00:43:06.359404+0000","last_clean":"2024-02-23T00:43:06.359404+0000","last_became_active":"2024-02-23T00:43:02.240086+0000","last_became_peered":"2024-02-23T00:43:02.240086+0000","last_unstale":"2024-02-23T00:43:06.359404+0000","last_undegraded":"2024-02-23T00:43:06.359404+0000","last_fullsized":"2024-02-23T00:43:06.359404+0000","mapping_epoch":750,"log_start":"0'0","ondisk_log_start":"0'0","created":750,"last_epoch_clean":751,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:01.453177+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.453177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.0","version":"751'11","reported_seq":193,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627128+0000","last_change":"2024-02-23T00:42:12.248572+0000","last_active":"2024-02-23T00:43:08.627128+0000","last_peered":"2024-02-23T00:43:08.627128+0000","last_clean":"2024-02-23T00:43:08.627128+0000","last_became_active":"2024-02-23T00:41:50.253705+0000","last_became_peered":"2024-02-23T00:41:50.253705+0000","last_unstale":"2024-02-23T00:43:08.627128+0000","last_undegraded":"2024-02-23T00:43:08.627128+0000","last_fullsized":"2024-02-23T00:43:08.627128+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"671'10","last_scrub_stamp":"2024-02-23T00:41:50.664981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:41:50.664981+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":339,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.664017+0000","last_change":"2024-02-23T00:42:58.676506+0000","last_active":"2024-02-23T00:43:08.664017+0000","last_peered":"2024-02-23T00:43:08.664017+0000","last_clean":"2024-02-23T00:43:08.664017+0000","last_became_active":"2024-02-23T00:42:41.400209+0000","last_became_peered":"2024-02-23T00:42:41.400209+0000","last_unstale":"2024-02-23T00:43:08.664017+0000","last_undegraded":"2024-02-23T00:43:08.664017+0000","last_fullsized":"2024-02-23T00:43:08.664017+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:58.676432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:58.676432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"571'11","reported_seq":965,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.384167+0000","last_change":"2024-02-23T00:42:18.360637+0000","last_active":"2024-02-23T00:43:09.384167+0000","last_peered":"2024-02-23T00:43:09.384167+0000","last_clean":"2024-02-23T00:43:09.384167+0000","last_became_active":"2024-02-23T00:39:49.195036+0000","last_became_peered":"2024-02-23T00:39:49.195036+0000","last_unstale":"2024-02-23T00:43:09.384167+0000","last_undegraded":"2024-02-23T00:43:09.384167+0000","last_fullsized":"2024-02-23T00:43:09.384167+0000","mapping_epoch":58,"log_start":"379'9","ondisk_log_start":"379'9","created":375,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'11","last_scrub_stamp":"2024-02-23T00:42:18.360544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.360544+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":849,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256160+0000","last_change":"2024-02-23T00:42:16.067711+0000","last_active":"2024-02-23T00:43:08.256160+0000","last_peered":"2024-02-23T00:43:08.256160+0000","last_clean":"2024-02-23T00:43:08.256160+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:43:08.256160+0000","last_undegraded":"2024-02-23T00:43:08.256160+0000","last_fullsized":"2024-02-23T00:43:08.256160+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:42:16.067590+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.067590+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":776,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258198+0000","last_change":"2024-02-23T00:41:52.314002+0000","last_active":"2024-02-23T00:43:09.258198+0000","last_peered":"2024-02-23T00:43:09.258198+0000","last_clean":"2024-02-23T00:43:09.258198+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:43:09.258198+0000","last_undegraded":"2024-02-23T00:43:09.258198+0000","last_fullsized":"2024-02-23T00:43:09.258198+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:52.313932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:41:52.313932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":763,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256538+0000","last_change":"2024-02-23T00:42:54.046985+0000","last_active":"2024-02-23T00:43:08.256538+0000","last_peered":"2024-02-23T00:43:08.256538+0000","last_clean":"2024-02-23T00:43:08.256538+0000","last_became_active":"2024-02-23T00:41:37.222092+0000","last_became_peered":"2024-02-23T00:41:37.222092+0000","last_unstale":"2024-02-23T00:43:08.256538+0000","last_undegraded":"2024-02-23T00:43:08.256538+0000","last_fullsized":"2024-02-23T00:43:08.256538+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:54.046903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.046903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":711,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.261104+0000","last_change":"2024-02-23T00:43:06.398049+0000","last_active":"2024-02-23T00:43:09.261104+0000","last_peered":"2024-02-23T00:43:09.261104+0000","last_clean":"2024-02-23T00:43:09.261104+0000","last_became_active":"2024-02-23T00:41:37.214214+0000","last_became_peered":"2024-02-23T00:41:37.214214+0000","last_unstale":"2024-02-23T00:43:09.261104+0000","last_undegraded":"2024-02-23T00:43:09.261104+0000","last_fullsized":"2024-02-23T00:43:09.261104+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.397979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.397979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":1374,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.256003+0000","last_change":"2024-02-23T00:41:39.898258+0000","last_active":"2024-02-23T00:43:09.256003+0000","last_peered":"2024-02-23T00:43:09.256003+0000","last_clean":"2024-02-23T00:43:09.256003+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:43:09.256003+0000","last_undegraded":"2024-02-23T00:43:09.256003+0000","last_fullsized":"2024-02-23T00:43:09.256003+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:41:39.898173+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:41:39.898173+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":630,"num_read_kb":630,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":464,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.360229+0000","last_change":"2024-02-23T00:42:15.735663+0000","last_active":"2024-02-23T00:43:06.360229+0000","last_peered":"2024-02-23T00:43:06.360229+0000","last_clean":"2024-02-23T00:43:06.360229+0000","last_became_active":"2024-02-23T00:41:41.226195+0000","last_became_peered":"2024-02-23T00:41:41.226195+0000","last_unstale":"2024-02-23T00:43:06.360229+0000","last_undegraded":"2024-02-23T00:43:06.360229+0000","last_fullsized":"2024-02-23T00:43:06.360229+0000","mapping_epoch":681,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:42:15.735606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.735606+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":754,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259893+0000","last_change":"2024-02-23T00:42:29.119245+0000","last_active":"2024-02-23T00:43:09.259893+0000","last_peered":"2024-02-23T00:43:09.259893+0000","last_clean":"2024-02-23T00:43:09.259893+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:43:09.259893+0000","last_undegraded":"2024-02-23T00:43:09.259893+0000","last_fullsized":"2024-02-23T00:43:09.259893+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:29.119169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:29.119169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":768,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258870+0000","last_change":"2024-02-23T00:42:56.444185+0000","last_active":"2024-02-23T00:43:09.258870+0000","last_peered":"2024-02-23T00:43:09.258870+0000","last_clean":"2024-02-23T00:43:09.258870+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:43:09.258870+0000","last_undegraded":"2024-02-23T00:43:09.258870+0000","last_fullsized":"2024-02-23T00:43:09.258870+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:56.444091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:42:56.444091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":92,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.359624+0000","last_change":"2024-02-23T00:41:50.712474+0000","last_active":"2024-02-23T00:43:06.359624+0000","last_peered":"2024-02-23T00:43:06.359624+0000","last_clean":"2024-02-23T00:43:06.359624+0000","last_became_active":"2024-02-23T00:41:37.228937+0000","last_became_peered":"2024-02-23T00:41:37.228937+0000","last_unstale":"2024-02-23T00:43:06.359624+0000","last_undegraded":"2024-02-23T00:43:06.359624+0000","last_fullsized":"2024-02-23T00:43:06.359624+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:50.712410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:41:50.712410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":692,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628113+0000","last_change":"2024-02-23T00:42:58.645999+0000","last_active":"2024-02-23T00:43:08.628113+0000","last_peered":"2024-02-23T00:43:08.628113+0000","last_clean":"2024-02-23T00:43:08.628113+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:43:08.628113+0000","last_undegraded":"2024-02-23T00:43:08.628113+0000","last_fullsized":"2024-02-23T00:43:08.628113+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:58.645916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:42:58.645916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.663823+0000","last_change":"2024-02-23T00:42:41.393838+0000","last_active":"2024-02-23T00:43:08.663823+0000","last_peered":"2024-02-23T00:43:08.663823+0000","last_clean":"2024-02-23T00:43:08.663823+0000","last_became_active":"2024-02-23T00:42:41.393491+0000","last_became_peered":"2024-02-23T00:42:41.393491+0000","last_unstale":"2024-02-23T00:43:08.663823+0000","last_undegraded":"2024-02-23T00:43:08.663823+0000","last_fullsized":"2024-02-23T00:43:08.663823+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.024705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":871,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627535+0000","last_change":"2024-02-23T00:43:00.587024+0000","last_active":"2024-02-23T00:43:08.627535+0000","last_peered":"2024-02-23T00:43:08.627535+0000","last_clean":"2024-02-23T00:43:08.627535+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:43:08.627535+0000","last_undegraded":"2024-02-23T00:43:08.627535+0000","last_fullsized":"2024-02-23T00:43:08.627535+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:43:00.586920+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.586920+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":774,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259444+0000","last_change":"2024-02-23T00:42:24.447799+0000","last_active":"2024-02-23T00:43:09.259444+0000","last_peered":"2024-02-23T00:43:09.259444+0000","last_clean":"2024-02-23T00:43:09.259444+0000","last_became_active":"2024-02-23T00:41:37.238728+0000","last_became_peered":"2024-02-23T00:41:37.238728+0000","last_unstale":"2024-02-23T00:43:09.259444+0000","last_undegraded":"2024-02-23T00:43:09.259444+0000","last_fullsized":"2024-02-23T00:43:09.259444+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.447731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.447731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":763,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258210+0000","last_change":"2024-02-23T00:42:17.473540+0000","last_active":"2024-02-23T00:43:09.258210+0000","last_peered":"2024-02-23T00:43:09.258210+0000","last_clean":"2024-02-23T00:43:09.258210+0000","last_became_active":"2024-02-23T00:41:37.239529+0000","last_became_peered":"2024-02-23T00:41:37.239529+0000","last_unstale":"2024-02-23T00:43:09.258210+0000","last_undegraded":"2024-02-23T00:43:09.258210+0000","last_fullsized":"2024-02-23T00:43:09.258210+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:17.473474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.473474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":769,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.660868+0000","last_change":"2024-02-23T00:41:49.588387+0000","last_active":"2024-02-23T00:43:08.660868+0000","last_peered":"2024-02-23T00:43:08.660868+0000","last_clean":"2024-02-23T00:43:08.660868+0000","last_became_active":"2024-02-23T00:40:37.197010+0000","last_became_peered":"2024-02-23T00:40:37.197010+0000","last_unstale":"2024-02-23T00:43:08.660868+0000","last_undegraded":"2024-02-23T00:43:08.660868+0000","last_fullsized":"2024-02-23T00:43:08.660868+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:49.588290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.588290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":92,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.359838+0000","last_change":"2024-02-23T00:42:57.587635+0000","last_active":"2024-02-23T00:43:06.359838+0000","last_peered":"2024-02-23T00:43:06.359838+0000","last_clean":"2024-02-23T00:43:06.359838+0000","last_became_active":"2024-02-23T00:41:37.248272+0000","last_became_peered":"2024-02-23T00:41:37.248272+0000","last_unstale":"2024-02-23T00:43:06.359838+0000","last_undegraded":"2024-02-23T00:43:06.359838+0000","last_fullsized":"2024-02-23T00:43:06.359838+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:57.587566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:57.587566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":756,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628349+0000","last_change":"2024-02-23T00:42:54.599143+0000","last_active":"2024-02-23T00:43:08.628349+0000","last_peered":"2024-02-23T00:43:08.628349+0000","last_clean":"2024-02-23T00:43:08.628349+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:43:08.628349+0000","last_undegraded":"2024-02-23T00:43:08.628349+0000","last_fullsized":"2024-02-23T00:43:08.628349+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:54.599070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.599070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":585,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259847+0000","last_change":"2024-02-23T00:42:19.485117+0000","last_active":"2024-02-23T00:43:09.259847+0000","last_peered":"2024-02-23T00:43:09.259847+0000","last_clean":"2024-02-23T00:43:09.259847+0000","last_became_active":"2024-02-23T00:41:50.256494+0000","last_became_peered":"2024-02-23T00:41:50.256494+0000","last_unstale":"2024-02-23T00:43:09.259847+0000","last_undegraded":"2024-02-23T00:43:09.259847+0000","last_fullsized":"2024-02-23T00:43:09.259847+0000","mapping_epoch":688,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:42:19.485046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.485046+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":739,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662138+0000","last_change":"2024-02-23T00:41:47.545036+0000","last_active":"2024-02-23T00:43:08.662138+0000","last_peered":"2024-02-23T00:43:08.662138+0000","last_clean":"2024-02-23T00:43:08.662138+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:43:08.662138+0000","last_undegraded":"2024-02-23T00:43:08.662138+0000","last_fullsized":"2024-02-23T00:43:08.662138+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.544952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.544952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":746,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260263+0000","last_change":"2024-02-23T00:41:37.229878+0000","last_active":"2024-02-23T00:43:09.260263+0000","last_peered":"2024-02-23T00:43:09.260263+0000","last_clean":"2024-02-23T00:43:09.260263+0000","last_became_active":"2024-02-23T00:41:37.229416+0000","last_became_peered":"2024-02-23T00:41:37.229416+0000","last_unstale":"2024-02-23T00:43:09.260263+0000","last_undegraded":"2024-02-23T00:43:09.260263+0000","last_fullsized":"2024-02-23T00:43:09.260263+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:00.151031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:00.151031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":713,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.664555+0000","last_change":"2024-02-23T00:41:55.514102+0000","last_active":"2024-02-23T00:43:08.664555+0000","last_peered":"2024-02-23T00:43:08.664555+0000","last_clean":"2024-02-23T00:43:08.664555+0000","last_became_active":"2024-02-23T00:38:49.194450+0000","last_became_peered":"2024-02-23T00:38:49.194450+0000","last_unstale":"2024-02-23T00:43:08.664555+0000","last_undegraded":"2024-02-23T00:43:08.664555+0000","last_fullsized":"2024-02-23T00:43:08.664555+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.514031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.514031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":275,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662403+0000","last_change":"2024-02-23T00:41:37.253004+0000","last_active":"2024-02-23T00:43:08.662403+0000","last_peered":"2024-02-23T00:43:08.662403+0000","last_clean":"2024-02-23T00:43:08.662403+0000","last_became_active":"2024-02-23T00:41:37.252083+0000","last_became_peered":"2024-02-23T00:41:37.252083+0000","last_unstale":"2024-02-23T00:43:08.662403+0000","last_undegraded":"2024-02-23T00:43:08.662403+0000","last_fullsized":"2024-02-23T00:43:08.662403+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":684,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662507+0000","last_change":"2024-02-23T00:41:50.256382+0000","last_active":"2024-02-23T00:43:08.662507+0000","last_peered":"2024-02-23T00:43:08.662507+0000","last_clean":"2024-02-23T00:43:08.662507+0000","last_became_active":"2024-02-23T00:41:50.256004+0000","last_became_peered":"2024-02-23T00:41:50.256004+0000","last_unstale":"2024-02-23T00:43:08.662507+0000","last_undegraded":"2024-02-23T00:43:08.662507+0000","last_fullsized":"2024-02-23T00:43:08.662507+0000","mapping_epoch":688,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:40:51.632948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:51.632948+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":767,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260897+0000","last_change":"2024-02-23T00:42:15.223520+0000","last_active":"2024-02-23T00:43:09.260897+0000","last_peered":"2024-02-23T00:43:09.260897+0000","last_clean":"2024-02-23T00:43:09.260897+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:43:09.260897+0000","last_undegraded":"2024-02-23T00:43:09.260897+0000","last_fullsized":"2024-02-23T00:43:09.260897+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.223455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.223455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":948,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-02-23T00:43:10.261550+0000","last_change":"2024-02-23T00:42:55.471736+0000","last_active":"2024-02-23T00:43:10.261550+0000","last_peered":"2024-02-23T00:43:10.261550+0000","last_clean":"2024-02-23T00:43:10.261550+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:43:10.261550+0000","last_undegraded":"2024-02-23T00:43:10.261550+0000","last_fullsized":"2024-02-23T00:43:10.261550+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:42:55.471653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:55.471653+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":691,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.384830+0000","last_change":"2024-02-23T00:42:52.938985+0000","last_active":"2024-02-23T00:43:09.384830+0000","last_peered":"2024-02-23T00:43:09.384830+0000","last_clean":"2024-02-23T00:43:09.384830+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:43:09.384830+0000","last_undegraded":"2024-02-23T00:43:09.384830+0000","last_fullsized":"2024-02-23T00:43:09.384830+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:52.938885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:52.938885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":261,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662004+0000","last_change":"2024-02-23T00:40:15.208236+0000","last_active":"2024-02-23T00:43:08.662004+0000","last_peered":"2024-02-23T00:43:08.662004+0000","last_clean":"2024-02-23T00:43:08.662004+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:43:08.662004+0000","last_undegraded":"2024-02-23T00:43:08.662004+0000","last_fullsized":"2024-02-23T00:43:08.662004+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:15.208124+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.208124+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":921,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259004+0000","last_change":"2024-02-23T00:41:08.116872+0000","last_active":"2024-02-23T00:43:09.259004+0000","last_peered":"2024-02-23T00:43:09.259004+0000","last_clean":"2024-02-23T00:43:09.259004+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:43:09.259004+0000","last_undegraded":"2024-02-23T00:43:09.259004+0000","last_fullsized":"2024-02-23T00:43:09.259004+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:41:08.116765+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:41:08.116765+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":751,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259633+0000","last_change":"2024-02-23T00:42:14.253417+0000","last_active":"2024-02-23T00:43:09.259633+0000","last_peered":"2024-02-23T00:43:09.259633+0000","last_clean":"2024-02-23T00:43:09.259633+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:43:09.259633+0000","last_undegraded":"2024-02-23T00:43:09.259633+0000","last_fullsized":"2024-02-23T00:43:09.259633+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.253319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.253319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"689'12","reported_seq":950,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.361815+0000","last_change":"2024-02-23T00:42:19.683007+0000","last_active":"2024-02-23T00:43:06.361815+0000","last_peered":"2024-02-23T00:43:06.361815+0000","last_clean":"2024-02-23T00:43:06.361815+0000","last_became_active":"2024-02-23T00:41:50.256773+0000","last_became_peered":"2024-02-23T00:41:50.256773+0000","last_unstale":"2024-02-23T00:43:06.361815+0000","last_undegraded":"2024-02-23T00:43:06.361815+0000","last_fullsized":"2024-02-23T00:43:06.361815+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'12","last_scrub_stamp":"2024-02-23T00:42:19.682902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.682902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":678,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259521+0000","last_change":"2024-02-23T00:41:40.171696+0000","last_active":"2024-02-23T00:43:09.259521+0000","last_peered":"2024-02-23T00:43:09.259521+0000","last_clean":"2024-02-23T00:43:09.259521+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:43:09.259521+0000","last_undegraded":"2024-02-23T00:43:09.259521+0000","last_fullsized":"2024-02-23T00:43:09.259521+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:40.171630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:41:40.171630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":753,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.385057+0000","last_change":"2024-02-23T00:42:29.905840+0000","last_active":"2024-02-23T00:43:09.385057+0000","last_peered":"2024-02-23T00:43:09.385057+0000","last_clean":"2024-02-23T00:43:09.385057+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:43:09.385057+0000","last_undegraded":"2024-02-23T00:43:09.385057+0000","last_fullsized":"2024-02-23T00:43:09.385057+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:29.905710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:29.905710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":592,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628125+0000","last_change":"2024-02-23T00:42:18.659610+0000","last_active":"2024-02-23T00:43:08.628125+0000","last_peered":"2024-02-23T00:43:08.628125+0000","last_clean":"2024-02-23T00:43:08.628125+0000","last_became_active":"2024-02-23T00:41:50.253028+0000","last_became_peered":"2024-02-23T00:41:50.253028+0000","last_unstale":"2024-02-23T00:43:08.628125+0000","last_undegraded":"2024-02-23T00:43:08.628125+0000","last_fullsized":"2024-02-23T00:43:08.628125+0000","mapping_epoch":688,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:42:18.659534+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.659534+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":780,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661199+0000","last_change":"2024-02-23T00:42:54.633894+0000","last_active":"2024-02-23T00:43:08.661199+0000","last_peered":"2024-02-23T00:43:08.661199+0000","last_clean":"2024-02-23T00:43:08.661199+0000","last_became_active":"2024-02-23T00:41:37.229862+0000","last_became_peered":"2024-02-23T00:41:37.229862+0000","last_unstale":"2024-02-23T00:43:08.661199+0000","last_undegraded":"2024-02-23T00:43:08.661199+0000","last_fullsized":"2024-02-23T00:43:08.661199+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:42:54.633815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.633815+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":728,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257730+0000","last_change":"2024-02-23T00:43:06.143533+0000","last_active":"2024-02-23T00:43:08.257730+0000","last_peered":"2024-02-23T00:43:08.257730+0000","last_clean":"2024-02-23T00:43:08.257730+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:43:08.257730+0000","last_undegraded":"2024-02-23T00:43:08.257730+0000","last_fullsized":"2024-02-23T00:43:08.257730+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.143455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.143455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":701,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260417+0000","last_change":"2024-02-23T00:42:53.372480+0000","last_active":"2024-02-23T00:43:09.260417+0000","last_peered":"2024-02-23T00:43:09.260417+0000","last_clean":"2024-02-23T00:43:09.260417+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:43:09.260417+0000","last_undegraded":"2024-02-23T00:43:09.260417+0000","last_fullsized":"2024-02-23T00:43:09.260417+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.372422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.372422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":106,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.362133+0000","last_change":"2024-02-23T00:43:03.634296+0000","last_active":"2024-02-23T00:43:06.362133+0000","last_peered":"2024-02-23T00:43:06.362133+0000","last_clean":"2024-02-23T00:43:06.362133+0000","last_became_active":"2024-02-23T00:42:39.394104+0000","last_became_peered":"2024-02-23T00:42:39.394104+0000","last_unstale":"2024-02-23T00:43:06.362133+0000","last_undegraded":"2024-02-23T00:43:06.362133+0000","last_fullsized":"2024-02-23T00:43:06.362133+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:03.634240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:43:03.634240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"642'14","reported_seq":307,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.382675+0000","last_change":"2024-02-23T00:42:59.176336+0000","last_active":"2024-02-23T00:43:09.382675+0000","last_peered":"2024-02-23T00:43:09.382675+0000","last_clean":"2024-02-23T00:43:09.382675+0000","last_became_active":"2024-02-23T00:41:01.212780+0000","last_became_peered":"2024-02-23T00:41:01.212780+0000","last_unstale":"2024-02-23T00:43:09.382675+0000","last_undegraded":"2024-02-23T00:43:09.382675+0000","last_fullsized":"2024-02-23T00:43:09.382675+0000","mapping_epoch":511,"log_start":"562'12","ondisk_log_start":"562'12","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'14","last_scrub_stamp":"2024-02-23T00:42:59.176246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.176246+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":564,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256516+0000","last_change":"2024-02-23T00:42:19.049262+0000","last_active":"2024-02-23T00:43:08.256516+0000","last_peered":"2024-02-23T00:43:08.256516+0000","last_clean":"2024-02-23T00:43:08.256516+0000","last_became_active":"2024-02-23T00:41:50.255315+0000","last_became_peered":"2024-02-23T00:41:50.255315+0000","last_unstale":"2024-02-23T00:43:08.256516+0000","last_undegraded":"2024-02-23T00:43:08.256516+0000","last_fullsized":"2024-02-23T00:43:08.256516+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'34","last_scrub_stamp":"2024-02-23T00:42:19.049190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.049190+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":744,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628519+0000","last_change":"2024-02-23T00:42:52.538341+0000","last_active":"2024-02-23T00:43:08.628519+0000","last_peered":"2024-02-23T00:43:08.628519+0000","last_clean":"2024-02-23T00:43:08.628519+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:43:08.628519+0000","last_undegraded":"2024-02-23T00:43:08.628519+0000","last_fullsized":"2024-02-23T00:43:08.628519+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:52.538255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:52.538255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":756,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662530+0000","last_change":"2024-02-23T00:42:12.433222+0000","last_active":"2024-02-23T00:43:08.662530+0000","last_peered":"2024-02-23T00:43:08.662530+0000","last_clean":"2024-02-23T00:43:08.662530+0000","last_became_active":"2024-02-23T00:40:57.488797+0000","last_became_peered":"2024-02-23T00:40:57.488797+0000","last_unstale":"2024-02-23T00:43:08.662530+0000","last_undegraded":"2024-02-23T00:43:08.662530+0000","last_fullsized":"2024-02-23T00:43:08.662530+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.433157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.433157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":721,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258319+0000","last_change":"2024-02-23T00:41:51.307510+0000","last_active":"2024-02-23T00:43:09.258319+0000","last_peered":"2024-02-23T00:43:09.258319+0000","last_clean":"2024-02-23T00:43:09.258319+0000","last_became_active":"2024-02-23T00:41:37.229061+0000","last_became_peered":"2024-02-23T00:41:37.229061+0000","last_unstale":"2024-02-23T00:43:09.258319+0000","last_undegraded":"2024-02-23T00:43:09.258319+0000","last_fullsized":"2024-02-23T00:43:09.258319+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:51.307443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:51.307443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":296,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661894+0000","last_change":"2024-02-23T00:42:39.396325+0000","last_active":"2024-02-23T00:43:08.661894+0000","last_peered":"2024-02-23T00:43:08.661894+0000","last_clean":"2024-02-23T00:43:08.661894+0000","last_became_active":"2024-02-23T00:42:39.396063+0000","last_became_peered":"2024-02-23T00:42:39.396063+0000","last_unstale":"2024-02-23T00:43:08.661894+0000","last_undegraded":"2024-02-23T00:43:08.661894+0000","last_fullsized":"2024-02-23T00:43:08.661894+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.355553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.355553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":786,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.385283+0000","last_change":"2024-02-23T00:41:53.922666+0000","last_active":"2024-02-23T00:43:09.385283+0000","last_peered":"2024-02-23T00:43:09.385283+0000","last_clean":"2024-02-23T00:43:09.385283+0000","last_became_active":"2024-02-23T00:40:57.218869+0000","last_became_peered":"2024-02-23T00:40:57.218869+0000","last_unstale":"2024-02-23T00:43:09.385283+0000","last_undegraded":"2024-02-23T00:43:09.385283+0000","last_fullsized":"2024-02-23T00:43:09.385283+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.922592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.922592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":329,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661630+0000","last_change":"2024-02-23T00:42:55.230049+0000","last_active":"2024-02-23T00:43:08.661630+0000","last_peered":"2024-02-23T00:43:08.661630+0000","last_clean":"2024-02-23T00:43:08.661630+0000","last_became_active":"2024-02-23T00:41:37.228326+0000","last_became_peered":"2024-02-23T00:41:37.228326+0000","last_unstale":"2024-02-23T00:43:08.661630+0000","last_undegraded":"2024-02-23T00:43:08.661630+0000","last_fullsized":"2024-02-23T00:43:08.661630+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:42:55.229966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:55.229966+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":731,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259091+0000","last_change":"2024-02-23T00:41:42.160301+0000","last_active":"2024-02-23T00:43:09.259091+0000","last_peered":"2024-02-23T00:43:09.259091+0000","last_clean":"2024-02-23T00:43:09.259091+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:43:09.259091+0000","last_undegraded":"2024-02-23T00:43:09.259091+0000","last_fullsized":"2024-02-23T00:43:09.259091+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.160190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.160190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":722,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661158+0000","last_change":"2024-02-23T00:41:50.256151+0000","last_active":"2024-02-23T00:43:08.661158+0000","last_peered":"2024-02-23T00:43:08.661158+0000","last_clean":"2024-02-23T00:43:08.661158+0000","last_became_active":"2024-02-23T00:41:50.255850+0000","last_became_peered":"2024-02-23T00:41:50.255850+0000","last_unstale":"2024-02-23T00:43:08.661158+0000","last_undegraded":"2024-02-23T00:43:08.661158+0000","last_fullsized":"2024-02-23T00:43:08.661158+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:01.638560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:01.638560+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":261,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661230+0000","last_change":"2024-02-23T00:41:02.306933+0000","last_active":"2024-02-23T00:43:08.661230+0000","last_peered":"2024-02-23T00:43:08.661230+0000","last_clean":"2024-02-23T00:43:08.661230+0000","last_became_active":"2024-02-23T00:38:49.175801+0000","last_became_peered":"2024-02-23T00:38:49.175801+0000","last_unstale":"2024-02-23T00:43:08.661230+0000","last_undegraded":"2024-02-23T00:43:08.661230+0000","last_fullsized":"2024-02-23T00:43:08.661230+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:02.306855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.306855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":1003,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259725+0000","last_change":"2024-02-23T00:42:39.399220+0000","last_active":"2024-02-23T00:43:09.259725+0000","last_peered":"2024-02-23T00:43:09.259725+0000","last_clean":"2024-02-23T00:43:09.259725+0000","last_became_active":"2024-02-23T00:42:39.398790+0000","last_became_peered":"2024-02-23T00:42:39.398790+0000","last_unstale":"2024-02-23T00:43:09.259725+0000","last_undegraded":"2024-02-23T00:43:09.259725+0000","last_fullsized":"2024-02-23T00:43:09.259725+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:17.043322+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.043322+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":787,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628050+0000","last_change":"2024-02-23T00:41:37.221015+0000","last_active":"2024-02-23T00:43:08.628050+0000","last_peered":"2024-02-23T00:43:08.628050+0000","last_clean":"2024-02-23T00:43:08.628050+0000","last_became_active":"2024-02-23T00:41:37.220531+0000","last_became_peered":"2024-02-23T00:41:37.220531+0000","last_unstale":"2024-02-23T00:43:08.628050+0000","last_undegraded":"2024-02-23T00:43:08.628050+0000","last_fullsized":"2024-02-23T00:43:08.628050+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.603558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.603558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":114,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.360275+0000","last_change":"2024-02-23T00:42:27.632835+0000","last_active":"2024-02-23T00:43:06.360275+0000","last_peered":"2024-02-23T00:43:06.360275+0000","last_clean":"2024-02-23T00:43:06.360275+0000","last_became_active":"2024-02-23T00:41:37.235230+0000","last_became_peered":"2024-02-23T00:41:37.235230+0000","last_unstale":"2024-02-23T00:43:06.360275+0000","last_undegraded":"2024-02-23T00:43:06.360275+0000","last_fullsized":"2024-02-23T00:43:06.360275+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'35","last_scrub_stamp":"2024-02-23T00:42:27.632769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.632769+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":763,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.664200+0000","last_change":"2024-02-23T00:42:53.644165+0000","last_active":"2024-02-23T00:43:08.664200+0000","last_peered":"2024-02-23T00:43:08.664200+0000","last_clean":"2024-02-23T00:43:08.664200+0000","last_became_active":"2024-02-23T00:41:37.229922+0000","last_became_peered":"2024-02-23T00:41:37.229922+0000","last_unstale":"2024-02-23T00:43:08.664200+0000","last_undegraded":"2024-02-23T00:43:08.664200+0000","last_fullsized":"2024-02-23T00:43:08.664200+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.644087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.644087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":748,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.384855+0000","last_change":"2024-02-23T00:43:09.384826+0000","last_active":"2024-02-23T00:43:09.384855+0000","last_peered":"2024-02-23T00:43:09.384855+0000","last_clean":"2024-02-23T00:43:09.384855+0000","last_became_active":"2024-02-23T00:41:05.523365+0000","last_became_peered":"2024-02-23T00:41:05.523365+0000","last_unstale":"2024-02-23T00:43:09.384855+0000","last_undegraded":"2024-02-23T00:43:09.384855+0000","last_fullsized":"2024-02-23T00:43:09.384855+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:09.384750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:43:09.384750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":678,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259840+0000","last_change":"2024-02-23T00:42:16.223783+0000","last_active":"2024-02-23T00:43:09.259840+0000","last_peered":"2024-02-23T00:43:09.259840+0000","last_clean":"2024-02-23T00:43:09.259840+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:43:09.259840+0000","last_undegraded":"2024-02-23T00:43:09.259840+0000","last_fullsized":"2024-02-23T00:43:09.259840+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:16.223720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.223720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":774,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257729+0000","last_change":"2024-02-23T00:42:58.210034+0000","last_active":"2024-02-23T00:43:08.257729+0000","last_peered":"2024-02-23T00:43:08.257729+0000","last_clean":"2024-02-23T00:43:08.257729+0000","last_became_active":"2024-02-23T00:41:37.228303+0000","last_became_peered":"2024-02-23T00:41:37.228303+0000","last_unstale":"2024-02-23T00:43:08.257729+0000","last_undegraded":"2024-02-23T00:43:08.257729+0000","last_fullsized":"2024-02-23T00:43:08.257729+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:58.209955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:42:58.209955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":473,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.361904+0000","last_change":"2024-02-23T00:42:17.721535+0000","last_active":"2024-02-23T00:43:06.361904+0000","last_peered":"2024-02-23T00:43:06.361904+0000","last_clean":"2024-02-23T00:43:06.361904+0000","last_became_active":"2024-02-23T00:41:41.224814+0000","last_became_peered":"2024-02-23T00:41:41.224814+0000","last_unstale":"2024-02-23T00:43:06.361904+0000","last_undegraded":"2024-02-23T00:43:06.361904+0000","last_fullsized":"2024-02-23T00:43:06.361904+0000","mapping_epoch":681,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:42:17.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.721471+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":758,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661020+0000","last_change":"2024-02-23T00:41:42.606731+0000","last_active":"2024-02-23T00:43:08.661020+0000","last_peered":"2024-02-23T00:43:08.661020+0000","last_clean":"2024-02-23T00:43:08.661020+0000","last_became_active":"2024-02-23T00:41:37.229722+0000","last_became_peered":"2024-02-23T00:41:37.229722+0000","last_unstale":"2024-02-23T00:43:08.661020+0000","last_undegraded":"2024-02-23T00:43:08.661020+0000","last_fullsized":"2024-02-23T00:43:08.661020+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.606635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.606635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":106,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.616277+0000","last_change":"2024-02-23T00:43:06.616251+0000","last_active":"2024-02-23T00:43:06.616277+0000","last_peered":"2024-02-23T00:43:06.616277+0000","last_clean":"2024-02-23T00:43:06.616277+0000","last_became_active":"2024-02-23T00:42:39.395793+0000","last_became_peered":"2024-02-23T00:42:39.395793+0000","last_unstale":"2024-02-23T00:43:06.616277+0000","last_undegraded":"2024-02-23T00:43:06.616277+0000","last_fullsized":"2024-02-23T00:43:06.616277+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.616180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.616180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"78'2","reported_seq":779,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260353+0000","last_change":"2024-02-23T00:41:53.371931+0000","last_active":"2024-02-23T00:43:09.260353+0000","last_peered":"2024-02-23T00:43:09.260353+0000","last_clean":"2024-02-23T00:43:09.260353+0000","last_became_active":"2024-02-23T00:41:25.236069+0000","last_became_peered":"2024-02-23T00:41:25.236069+0000","last_unstale":"2024-02-23T00:43:09.260353+0000","last_undegraded":"2024-02-23T00:43:09.260353+0000","last_fullsized":"2024-02-23T00:43:09.260353+0000","mapping_epoch":511,"log_start":"78'2","ondisk_log_start":"78'2","created":60,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:41:53.371852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.371852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":691,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.384004+0000","last_change":"2024-02-23T00:42:53.959862+0000","last_active":"2024-02-23T00:43:09.384004+0000","last_peered":"2024-02-23T00:43:09.384004+0000","last_clean":"2024-02-23T00:43:09.384004+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:43:09.384004+0000","last_undegraded":"2024-02-23T00:43:09.384004+0000","last_fullsized":"2024-02-23T00:43:09.384004+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.959787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.959787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":767,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662430+0000","last_change":"2024-02-23T00:42:15.388568+0000","last_active":"2024-02-23T00:43:08.662430+0000","last_peered":"2024-02-23T00:43:08.662430+0000","last_clean":"2024-02-23T00:43:08.662430+0000","last_became_active":"2024-02-23T00:40:45.191453+0000","last_became_peered":"2024-02-23T00:40:45.191453+0000","last_unstale":"2024-02-23T00:43:08.662430+0000","last_undegraded":"2024-02-23T00:43:08.662430+0000","last_fullsized":"2024-02-23T00:43:08.662430+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.388510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.388510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":783,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628090+0000","last_change":"2024-02-23T00:41:33.595415+0000","last_active":"2024-02-23T00:43:08.628090+0000","last_peered":"2024-02-23T00:43:08.628090+0000","last_clean":"2024-02-23T00:43:08.628090+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:43:08.628090+0000","last_undegraded":"2024-02-23T00:43:08.628090+0000","last_fullsized":"2024-02-23T00:43:08.628090+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:41:33.595268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.595268+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":766,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661106+0000","last_change":"2024-02-23T00:42:28.369569+0000","last_active":"2024-02-23T00:43:08.661106+0000","last_peered":"2024-02-23T00:43:08.661106+0000","last_clean":"2024-02-23T00:43:08.661106+0000","last_became_active":"2024-02-23T00:41:37.229335+0000","last_became_peered":"2024-02-23T00:41:37.229335+0000","last_unstale":"2024-02-23T00:43:08.661106+0000","last_undegraded":"2024-02-23T00:43:08.661106+0000","last_fullsized":"2024-02-23T00:43:08.661106+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:28.369481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.369481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":732,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257078+0000","last_change":"2024-02-23T00:41:43.984556+0000","last_active":"2024-02-23T00:43:08.257078+0000","last_peered":"2024-02-23T00:43:08.257078+0000","last_clean":"2024-02-23T00:43:08.257078+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:43:08.257078+0000","last_undegraded":"2024-02-23T00:43:08.257078+0000","last_fullsized":"2024-02-23T00:43:08.257078+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:43.984490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:43.984490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":270,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661298+0000","last_change":"2024-02-23T00:42:39.396374+0000","last_active":"2024-02-23T00:43:08.661298+0000","last_peered":"2024-02-23T00:43:08.661298+0000","last_clean":"2024-02-23T00:43:08.661298+0000","last_became_active":"2024-02-23T00:42:39.396071+0000","last_became_peered":"2024-02-23T00:42:39.396071+0000","last_unstale":"2024-02-23T00:43:08.661298+0000","last_undegraded":"2024-02-23T00:43:08.661298+0000","last_fullsized":"2024-02-23T00:43:08.661298+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.474495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.474495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":780,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257808+0000","last_change":"2024-02-23T00:42:22.037093+0000","last_active":"2024-02-23T00:43:08.257808+0000","last_peered":"2024-02-23T00:43:08.257808+0000","last_clean":"2024-02-23T00:43:08.257808+0000","last_became_active":"2024-02-23T00:41:13.469953+0000","last_became_peered":"2024-02-23T00:41:13.469953+0000","last_unstale":"2024-02-23T00:43:08.257808+0000","last_undegraded":"2024-02-23T00:43:08.257808+0000","last_fullsized":"2024-02-23T00:43:08.257808+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:22.037008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.037008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":693,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.383826+0000","last_change":"2024-02-23T00:43:00.007515+0000","last_active":"2024-02-23T00:43:09.383826+0000","last_peered":"2024-02-23T00:43:09.383826+0000","last_clean":"2024-02-23T00:43:09.383826+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:43:09.383826+0000","last_undegraded":"2024-02-23T00:43:09.383826+0000","last_fullsized":"2024-02-23T00:43:09.383826+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:00.007437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.007437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":778,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258418+0000","last_change":"2024-02-23T00:43:07.415143+0000","last_active":"2024-02-23T00:43:09.258418+0000","last_peered":"2024-02-23T00:43:09.258418+0000","last_clean":"2024-02-23T00:43:09.258418+0000","last_became_active":"2024-02-23T00:40:38.061696+0000","last_became_peered":"2024-02-23T00:40:38.061696+0000","last_unstale":"2024-02-23T00:43:09.258418+0000","last_undegraded":"2024-02-23T00:43:09.258418+0000","last_fullsized":"2024-02-23T00:43:09.258418+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:50.264240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:41:50.264240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":885,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258859+0000","last_change":"2024-02-23T00:42:59.284657+0000","last_active":"2024-02-23T00:43:09.258859+0000","last_peered":"2024-02-23T00:43:09.258859+0000","last_clean":"2024-02-23T00:43:09.258859+0000","last_became_active":"2024-02-23T00:41:40.219384+0000","last_became_peered":"2024-02-23T00:41:40.219384+0000","last_unstale":"2024-02-23T00:43:09.258859+0000","last_undegraded":"2024-02-23T00:43:09.258859+0000","last_fullsized":"2024-02-23T00:43:09.258859+0000","mapping_epoch":680,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:42:59.284597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.284597+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":796,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258199+0000","last_change":"2024-02-23T00:43:03.434468+0000","last_active":"2024-02-23T00:43:09.258199+0000","last_peered":"2024-02-23T00:43:09.258199+0000","last_clean":"2024-02-23T00:43:09.258199+0000","last_became_active":"2024-02-23T00:41:37.219000+0000","last_became_peered":"2024-02-23T00:41:37.219000+0000","last_unstale":"2024-02-23T00:43:09.258199+0000","last_undegraded":"2024-02-23T00:43:09.258199+0000","last_fullsized":"2024-02-23T00:43:09.258199+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:03.434391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:03.434391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":996,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260490+0000","last_change":"2024-02-23T00:42:39.401952+0000","last_active":"2024-02-23T00:43:09.260490+0000","last_peered":"2024-02-23T00:43:09.260490+0000","last_clean":"2024-02-23T00:43:09.260490+0000","last_became_active":"2024-02-23T00:42:39.401599+0000","last_became_peered":"2024-02-23T00:42:39.401599+0000","last_unstale":"2024-02-23T00:43:09.260490+0000","last_undegraded":"2024-02-23T00:43:09.260490+0000","last_fullsized":"2024-02-23T00:43:09.260490+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:12.984040+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.984040+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"62'1","reported_seq":792,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.664148+0000","last_change":"2024-02-23T00:42:22.301796+0000","last_active":"2024-02-23T00:43:08.664148+0000","last_peered":"2024-02-23T00:43:08.664148+0000","last_clean":"2024-02-23T00:43:08.664148+0000","last_became_active":"2024-02-23T00:41:21.226819+0000","last_became_peered":"2024-02-23T00:41:21.226819+0000","last_unstale":"2024-02-23T00:43:08.664148+0000","last_undegraded":"2024-02-23T00:43:08.664148+0000","last_fullsized":"2024-02-23T00:43:08.664148+0000","mapping_epoch":511,"log_start":"62'1","ondisk_log_start":"62'1","created":60,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:42:22.301726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.301726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":700,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259647+0000","last_change":"2024-02-23T00:42:53.387347+0000","last_active":"2024-02-23T00:43:09.259647+0000","last_peered":"2024-02-23T00:43:09.259647+0000","last_clean":"2024-02-23T00:43:09.259647+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:43:09.259647+0000","last_undegraded":"2024-02-23T00:43:09.259647+0000","last_fullsized":"2024-02-23T00:43:09.259647+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.387251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.387251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":776,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.360180+0000","last_change":"2024-02-23T00:43:00.633273+0000","last_active":"2024-02-23T00:43:06.360180+0000","last_peered":"2024-02-23T00:43:06.360180+0000","last_clean":"2024-02-23T00:43:06.360180+0000","last_became_active":"2024-02-23T00:42:39.393007+0000","last_became_peered":"2024-02-23T00:42:39.393007+0000","last_unstale":"2024-02-23T00:43:06.360180+0000","last_undegraded":"2024-02-23T00:43:06.360180+0000","last_fullsized":"2024-02-23T00:43:06.360180+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:00.633189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.633189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":92,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.361253+0000","last_change":"2024-02-23T00:41:54.690476+0000","last_active":"2024-02-23T00:43:06.361253+0000","last_peered":"2024-02-23T00:43:06.361253+0000","last_clean":"2024-02-23T00:43:06.361253+0000","last_became_active":"2024-02-23T00:41:37.239608+0000","last_became_peered":"2024-02-23T00:41:37.239608+0000","last_unstale":"2024-02-23T00:43:06.361253+0000","last_undegraded":"2024-02-23T00:43:06.361253+0000","last_fullsized":"2024-02-23T00:43:06.361253+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:54.690407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:54.690407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":1072,"reported_epoch":759,"state":"active+clean","last_fresh":"2024-02-23T00:43:10.416570+0000","last_change":"2024-02-23T00:42:54.354917+0000","last_active":"2024-02-23T00:43:10.416570+0000","last_peered":"2024-02-23T00:43:10.416570+0000","last_clean":"2024-02-23T00:43:10.416570+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:43:10.416570+0000","last_undegraded":"2024-02-23T00:43:10.416570+0000","last_fullsized":"2024-02-23T00:43:10.416570+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'48","last_scrub_stamp":"2024-02-23T00:42:54.354848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.354848+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"601'2","reported_seq":375,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260735+0000","last_change":"2024-02-23T00:43:00.296517+0000","last_active":"2024-02-23T00:43:09.260735+0000","last_peered":"2024-02-23T00:43:09.260735+0000","last_clean":"2024-02-23T00:43:09.260735+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:43:09.260735+0000","last_undegraded":"2024-02-23T00:43:09.260735+0000","last_fullsized":"2024-02-23T00:43:09.260735+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'2","last_scrub_stamp":"2024-02-23T00:43:00.296450+0000","last_deep_scrub":"601'2","last_deep_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.296450+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":302,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661629+0000","last_change":"2024-02-23T00:42:26.240735+0000","last_active":"2024-02-23T00:43:08.661629+0000","last_peered":"2024-02-23T00:43:08.661629+0000","last_clean":"2024-02-23T00:43:08.661629+0000","last_became_active":"2024-02-23T00:41:17.220294+0000","last_became_peered":"2024-02-23T00:41:17.220294+0000","last_unstale":"2024-02-23T00:43:08.661629+0000","last_undegraded":"2024-02-23T00:43:08.661629+0000","last_fullsized":"2024-02-23T00:43:08.661629+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.240647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.240647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":700,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258609+0000","last_change":"2024-02-23T00:41:55.275163+0000","last_active":"2024-02-23T00:43:09.258609+0000","last_peered":"2024-02-23T00:43:09.258609+0000","last_clean":"2024-02-23T00:43:09.258609+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:43:09.258609+0000","last_undegraded":"2024-02-23T00:43:09.258609+0000","last_fullsized":"2024-02-23T00:43:09.258609+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.275099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.275099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":669,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256879+0000","last_change":"2024-02-23T00:42:39.399345+0000","last_active":"2024-02-23T00:43:08.256879+0000","last_peered":"2024-02-23T00:43:08.256879+0000","last_clean":"2024-02-23T00:43:08.256879+0000","last_became_active":"2024-02-23T00:42:39.399113+0000","last_became_peered":"2024-02-23T00:42:39.399113+0000","last_unstale":"2024-02-23T00:43:08.256879+0000","last_undegraded":"2024-02-23T00:43:08.256879+0000","last_fullsized":"2024-02-23T00:43:08.256879+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.024705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"681'21","reported_seq":331,"reported_epoch":758,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:43:09.382072+0000","last_change":"2024-02-23T00:43:08.418715+0000","last_active":"2024-02-23T00:43:09.382072+0000","last_peered":"2024-02-23T00:43:09.382072+0000","last_clean":"2024-02-23T00:27:24.172824+0000","last_became_active":"2024-02-23T00:41:38.214049+0000","last_became_peered":"2024-02-23T00:41:38.214049+0000","last_unstale":"2024-02-23T00:43:09.382072+0000","last_undegraded":"2024-02-23T00:43:09.382072+0000","last_fullsized":"2024-02-23T00:43:09.382072+0000","mapping_epoch":678,"log_start":"606'17","ondisk_log_start":"606'17","created":58,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'19","last_scrub_stamp":"2024-02-23T00:40:58.498752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.498752+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[3,6,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":182,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260651+0000","last_change":"2024-02-23T00:40:53.236854+0000","last_active":"2024-02-23T00:43:09.260651+0000","last_peered":"2024-02-23T00:43:09.260651+0000","last_clean":"2024-02-23T00:43:09.260651+0000","last_became_active":"2024-02-23T00:40:23.194277+0000","last_became_peered":"2024-02-23T00:40:23.194277+0000","last_unstale":"2024-02-23T00:43:09.260651+0000","last_undegraded":"2024-02-23T00:43:09.260651+0000","last_fullsized":"2024-02-23T00:43:09.260651+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:40:53.236791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.236791+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":758,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259809+0000","last_change":"2024-02-23T00:43:07.413095+0000","last_active":"2024-02-23T00:43:09.259809+0000","last_peered":"2024-02-23T00:43:09.259809+0000","last_clean":"2024-02-23T00:43:09.259809+0000","last_became_active":"2024-02-23T00:41:37.229003+0000","last_became_peered":"2024-02-23T00:41:37.229003+0000","last_unstale":"2024-02-23T00:43:09.259809+0000","last_undegraded":"2024-02-23T00:43:09.259809+0000","last_fullsized":"2024-02-23T00:43:09.259809+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.413026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.413026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":92,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.359915+0000","last_change":"2024-02-23T00:41:49.737992+0000","last_active":"2024-02-23T00:43:06.359915+0000","last_peered":"2024-02-23T00:43:06.359915+0000","last_clean":"2024-02-23T00:43:06.359915+0000","last_became_active":"2024-02-23T00:41:37.248459+0000","last_became_peered":"2024-02-23T00:41:37.248459+0000","last_unstale":"2024-02-23T00:43:06.359915+0000","last_undegraded":"2024-02-23T00:43:06.359915+0000","last_fullsized":"2024-02-23T00:43:06.359915+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:49.737927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.737927+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":686,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.383106+0000","last_change":"2024-02-23T00:41:40.761546+0000","last_active":"2024-02-23T00:43:09.383106+0000","last_peered":"2024-02-23T00:43:09.383106+0000","last_clean":"2024-02-23T00:43:09.383106+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:43:09.383106+0000","last_undegraded":"2024-02-23T00:43:09.383106+0000","last_fullsized":"2024-02-23T00:43:09.383106+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:40.761466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:41:40.761466+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.663109+0000","last_change":"2024-02-23T00:43:02.723267+0000","last_active":"2024-02-23T00:43:08.663109+0000","last_peered":"2024-02-23T00:43:08.663109+0000","last_clean":"2024-02-23T00:43:08.663109+0000","last_became_active":"2024-02-23T00:42:41.402111+0000","last_became_peered":"2024-02-23T00:42:41.402111+0000","last_unstale":"2024-02-23T00:43:08.663109+0000","last_undegraded":"2024-02-23T00:43:08.663109+0000","last_fullsized":"2024-02-23T00:43:08.663109+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:02.723205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:43:02.723205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":790,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.257757+0000","last_change":"2024-02-23T00:42:26.380761+0000","last_active":"2024-02-23T00:43:09.257757+0000","last_peered":"2024-02-23T00:43:09.257757+0000","last_clean":"2024-02-23T00:43:09.257757+0000","last_became_active":"2024-02-23T00:41:37.239598+0000","last_became_peered":"2024-02-23T00:41:37.239598+0000","last_unstale":"2024-02-23T00:43:09.257757+0000","last_undegraded":"2024-02-23T00:43:09.257757+0000","last_fullsized":"2024-02-23T00:43:09.257757+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.380671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":676,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259439+0000","last_change":"2024-02-23T00:43:05.364229+0000","last_active":"2024-02-23T00:43:09.259439+0000","last_peered":"2024-02-23T00:43:09.259439+0000","last_clean":"2024-02-23T00:43:09.259439+0000","last_became_active":"2024-02-23T00:41:50.257656+0000","last_became_peered":"2024-02-23T00:41:50.257656+0000","last_unstale":"2024-02-23T00:43:09.259439+0000","last_undegraded":"2024-02-23T00:43:09.259439+0000","last_fullsized":"2024-02-23T00:43:09.259439+0000","mapping_epoch":688,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:43:05.364147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:05.364147+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":793,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.384002+0000","last_change":"2024-02-23T00:43:01.990077+0000","last_active":"2024-02-23T00:43:09.384002+0000","last_peered":"2024-02-23T00:43:09.384002+0000","last_clean":"2024-02-23T00:43:09.384002+0000","last_became_active":"2024-02-23T00:41:37.238708+0000","last_became_peered":"2024-02-23T00:41:37.238708+0000","last_unstale":"2024-02-23T00:43:09.384002+0000","last_undegraded":"2024-02-23T00:43:09.384002+0000","last_fullsized":"2024-02-23T00:43:09.384002+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:43:01.989978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.989978+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":279,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662414+0000","last_change":"2024-02-23T00:41:33.042594+0000","last_active":"2024-02-23T00:43:08.662414+0000","last_peered":"2024-02-23T00:43:08.662414+0000","last_clean":"2024-02-23T00:43:08.662414+0000","last_became_active":"2024-02-23T00:40:17.214288+0000","last_became_peered":"2024-02-23T00:40:17.214288+0000","last_unstale":"2024-02-23T00:43:08.662414+0000","last_undegraded":"2024-02-23T00:43:08.662414+0000","last_fullsized":"2024-02-23T00:43:08.662414+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:33.042520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.042520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":681,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256709+0000","last_change":"2024-02-23T00:42:13.006928+0000","last_active":"2024-02-23T00:43:08.256709+0000","last_peered":"2024-02-23T00:43:08.256709+0000","last_clean":"2024-02-23T00:43:08.256709+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:43:08.256709+0000","last_undegraded":"2024-02-23T00:43:08.256709+0000","last_fullsized":"2024-02-23T00:43:08.256709+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:13.006833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.006833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661659+0000","last_change":"2024-02-23T00:43:04.777343+0000","last_active":"2024-02-23T00:43:08.661659+0000","last_peered":"2024-02-23T00:43:08.661659+0000","last_clean":"2024-02-23T00:43:08.661659+0000","last_became_active":"2024-02-23T00:42:41.399155+0000","last_became_peered":"2024-02-23T00:42:41.399155+0000","last_unstale":"2024-02-23T00:43:08.661659+0000","last_undegraded":"2024-02-23T00:43:08.661659+0000","last_fullsized":"2024-02-23T00:43:08.661659+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:04.777269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:04.777269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":526,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.362213+0000","last_change":"2024-02-23T00:42:14.760378+0000","last_active":"2024-02-23T00:43:06.362213+0000","last_peered":"2024-02-23T00:43:06.362213+0000","last_clean":"2024-02-23T00:43:06.362213+0000","last_became_active":"2024-02-23T00:41:38.217979+0000","last_became_peered":"2024-02-23T00:41:38.217979+0000","last_unstale":"2024-02-23T00:43:06.362213+0000","last_undegraded":"2024-02-23T00:43:06.362213+0000","last_fullsized":"2024-02-23T00:43:06.362213+0000","mapping_epoch":678,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:42:14.760308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.760308+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":767,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.257713+0000","last_change":"2024-02-23T00:42:21.442459+0000","last_active":"2024-02-23T00:43:09.257713+0000","last_peered":"2024-02-23T00:43:09.257713+0000","last_clean":"2024-02-23T00:43:09.257713+0000","last_became_active":"2024-02-23T00:41:37.238750+0000","last_became_peered":"2024-02-23T00:41:37.238750+0000","last_unstale":"2024-02-23T00:43:09.257713+0000","last_undegraded":"2024-02-23T00:43:09.257713+0000","last_fullsized":"2024-02-23T00:43:09.257713+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:21.442362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:21.442362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":266,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661931+0000","last_change":"2024-02-23T00:42:20.334112+0000","last_active":"2024-02-23T00:43:08.661931+0000","last_peered":"2024-02-23T00:43:08.661931+0000","last_clean":"2024-02-23T00:43:08.661931+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:43:08.661931+0000","last_undegraded":"2024-02-23T00:43:08.661931+0000","last_fullsized":"2024-02-23T00:43:08.661931+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.334013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.334013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":261,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662769+0000","last_change":"2024-02-23T00:41:48.297291+0000","last_active":"2024-02-23T00:43:08.662769+0000","last_peered":"2024-02-23T00:43:08.662769+0000","last_clean":"2024-02-23T00:43:08.662769+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:43:08.662769+0000","last_undegraded":"2024-02-23T00:43:08.662769+0000","last_fullsized":"2024-02-23T00:43:08.662769+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:48.297220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:48.297220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":160,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627021+0000","last_change":"2024-02-23T00:43:07.691013+0000","last_active":"2024-02-23T00:43:08.627021+0000","last_peered":"2024-02-23T00:43:08.627021+0000","last_clean":"2024-02-23T00:43:08.627021+0000","last_became_active":"2024-02-23T00:42:41.406233+0000","last_became_peered":"2024-02-23T00:42:41.406233+0000","last_unstale":"2024-02-23T00:43:08.627021+0000","last_undegraded":"2024-02-23T00:43:08.627021+0000","last_fullsized":"2024-02-23T00:43:08.627021+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:43:07.690939+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:43:07.690939+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.690939+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":874,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661770+0000","last_change":"2024-02-23T00:43:00.208080+0000","last_active":"2024-02-23T00:43:08.661770+0000","last_peered":"2024-02-23T00:43:08.661770+0000","last_clean":"2024-02-23T00:43:08.661770+0000","last_became_active":"2024-02-23T00:40:51.316452+0000","last_became_peered":"2024-02-23T00:40:51.316452+0000","last_unstale":"2024-02-23T00:43:08.661770+0000","last_undegraded":"2024-02-23T00:43:08.661770+0000","last_fullsized":"2024-02-23T00:43:08.661770+0000","mapping_epoch":631,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:43:00.208019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.208019+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":745,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627443+0000","last_change":"2024-02-23T00:42:19.655247+0000","last_active":"2024-02-23T00:43:08.627443+0000","last_peered":"2024-02-23T00:43:08.627443+0000","last_clean":"2024-02-23T00:43:08.627443+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:43:08.627443+0000","last_undegraded":"2024-02-23T00:43:08.627443+0000","last_fullsized":"2024-02-23T00:43:08.627443+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.655188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.655188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":741,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257504+0000","last_change":"2024-02-23T00:41:53.984748+0000","last_active":"2024-02-23T00:43:08.257504+0000","last_peered":"2024-02-23T00:43:08.257504+0000","last_clean":"2024-02-23T00:43:08.257504+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:43:08.257504+0000","last_undegraded":"2024-02-23T00:43:08.257504+0000","last_fullsized":"2024-02-23T00:43:08.257504+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.984679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.984679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":710,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628323+0000","last_change":"2024-02-23T00:43:04.619611+0000","last_active":"2024-02-23T00:43:08.628323+0000","last_peered":"2024-02-23T00:43:08.628323+0000","last_clean":"2024-02-23T00:43:08.628323+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:43:08.628323+0000","last_undegraded":"2024-02-23T00:43:08.628323+0000","last_fullsized":"2024-02-23T00:43:08.628323+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:04.619508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:04.619508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662047+0000","last_change":"2024-02-23T00:42:59.678005+0000","last_active":"2024-02-23T00:43:08.662047+0000","last_peered":"2024-02-23T00:43:08.662047+0000","last_clean":"2024-02-23T00:43:08.662047+0000","last_became_active":"2024-02-23T00:42:41.402543+0000","last_became_peered":"2024-02-23T00:42:41.402543+0000","last_unstale":"2024-02-23T00:43:08.662047+0000","last_undegraded":"2024-02-23T00:43:08.662047+0000","last_fullsized":"2024-02-23T00:43:08.662047+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:59.677899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.677899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"688'39","reported_seq":284,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662712+0000","last_change":"2024-02-23T00:41:49.338035+0000","last_active":"2024-02-23T00:43:08.662712+0000","last_peered":"2024-02-23T00:43:08.662712+0000","last_clean":"2024-02-23T00:43:08.662712+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:43:08.662712+0000","last_undegraded":"2024-02-23T00:43:08.662712+0000","last_fullsized":"2024-02-23T00:43:08.662712+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'39","last_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_deep_scrub":"688'39","last_deep_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.337969+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":758,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256684+0000","last_change":"2024-02-23T00:41:41.955717+0000","last_active":"2024-02-23T00:43:08.256684+0000","last_peered":"2024-02-23T00:43:08.256684+0000","last_clean":"2024-02-23T00:43:08.256684+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:43:08.256684+0000","last_undegraded":"2024-02-23T00:43:08.256684+0000","last_fullsized":"2024-02-23T00:43:08.256684+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:41.955589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:41:41.955589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"681'12","reported_seq":956,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.256234+0000","last_change":"2024-02-23T00:42:59.230304+0000","last_active":"2024-02-23T00:43:09.256234+0000","last_peered":"2024-02-23T00:43:09.256234+0000","last_clean":"2024-02-23T00:43:09.256234+0000","last_became_active":"2024-02-23T00:41:37.228441+0000","last_became_peered":"2024-02-23T00:41:37.228441+0000","last_unstale":"2024-02-23T00:43:09.256234+0000","last_undegraded":"2024-02-23T00:43:09.256234+0000","last_fullsized":"2024-02-23T00:43:09.256234+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'12","last_scrub_stamp":"2024-02-23T00:42:59.230220+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.230220+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":681,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257765+0000","last_change":"2024-02-23T00:42:17.056253+0000","last_active":"2024-02-23T00:43:08.257765+0000","last_peered":"2024-02-23T00:43:08.257765+0000","last_clean":"2024-02-23T00:43:08.257765+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:43:08.257765+0000","last_undegraded":"2024-02-23T00:43:08.257765+0000","last_fullsized":"2024-02-23T00:43:08.257765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:17.056153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.056153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":31,"reported_epoch":756,"state":"active+clean","last_fresh":"2024-02-23T00:43:07.568838+0000","last_change":"2024-02-23T00:43:07.568838+0000","last_active":"2024-02-23T00:43:07.568838+0000","last_peered":"2024-02-23T00:43:07.568838+0000","last_clean":"2024-02-23T00:43:07.568838+0000","last_became_active":"2024-02-23T00:42:41.402683+0000","last_became_peered":"2024-02-23T00:42:41.402683+0000","last_unstale":"2024-02-23T00:43:07.568838+0000","last_undegraded":"2024-02-23T00:43:07.568838+0000","last_fullsized":"2024-02-23T00:43:07.568838+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.568782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.568782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":817,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256947+0000","last_change":"2024-02-23T00:42:28.967292+0000","last_active":"2024-02-23T00:43:08.256947+0000","last_peered":"2024-02-23T00:43:08.256947+0000","last_clean":"2024-02-23T00:43:08.256947+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:43:08.256947+0000","last_undegraded":"2024-02-23T00:43:08.256947+0000","last_fullsized":"2024-02-23T00:43:08.256947+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:42:28.967208+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.967208+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":92,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.360230+0000","last_change":"2024-02-23T00:41:47.719694+0000","last_active":"2024-02-23T00:43:06.360230+0000","last_peered":"2024-02-23T00:43:06.360230+0000","last_clean":"2024-02-23T00:43:06.360230+0000","last_became_active":"2024-02-23T00:41:37.235746+0000","last_became_peered":"2024-02-23T00:41:37.235746+0000","last_unstale":"2024-02-23T00:43:06.360230+0000","last_undegraded":"2024-02-23T00:43:06.360230+0000","last_fullsized":"2024-02-23T00:43:06.360230+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.719590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.719590+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":318,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.384842+0000","last_change":"2024-02-23T00:40:50.453642+0000","last_active":"2024-02-23T00:43:09.384842+0000","last_peered":"2024-02-23T00:43:09.384842+0000","last_clean":"2024-02-23T00:43:09.384842+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:43:09.384842+0000","last_undegraded":"2024-02-23T00:43:09.384842+0000","last_fullsized":"2024-02-23T00:43:09.384842+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'11","last_scrub_stamp":"2024-02-23T00:40:50.453579+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.453579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":261,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661855+0000","last_change":"2024-02-23T00:41:03.320478+0000","last_active":"2024-02-23T00:43:08.661855+0000","last_peered":"2024-02-23T00:43:08.661855+0000","last_clean":"2024-02-23T00:43:08.661855+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:43:08.661855+0000","last_undegraded":"2024-02-23T00:43:08.661855+0000","last_fullsized":"2024-02-23T00:43:08.661855+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:03.320387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:03.320387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.663890+0000","last_change":"2024-02-23T00:43:05.740185+0000","last_active":"2024-02-23T00:43:08.663890+0000","last_peered":"2024-02-23T00:43:08.663890+0000","last_clean":"2024-02-23T00:43:08.663890+0000","last_became_active":"2024-02-23T00:42:41.398941+0000","last_became_peered":"2024-02-23T00:42:41.398941+0000","last_unstale":"2024-02-23T00:43:08.663890+0000","last_undegraded":"2024-02-23T00:43:08.663890+0000","last_fullsized":"2024-02-23T00:43:08.663890+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:05.740112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:43:05.740112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":754,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260999+0000","last_change":"2024-02-23T00:41:02.134977+0000","last_active":"2024-02-23T00:43:09.260999+0000","last_peered":"2024-02-23T00:43:09.260999+0000","last_clean":"2024-02-23T00:43:09.260999+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:43:09.260999+0000","last_undegraded":"2024-02-23T00:43:09.260999+0000","last_fullsized":"2024-02-23T00:43:09.260999+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'15","last_scrub_stamp":"2024-02-23T00:41:02.134898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.134898+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":767,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628223+0000","last_change":"2024-02-23T00:41:42.607771+0000","last_active":"2024-02-23T00:43:08.628223+0000","last_peered":"2024-02-23T00:43:08.628223+0000","last_clean":"2024-02-23T00:43:08.628223+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:43:08.628223+0000","last_undegraded":"2024-02-23T00:43:08.628223+0000","last_fullsized":"2024-02-23T00:43:08.628223+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.607642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.607642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":285,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662359+0000","last_change":"2024-02-23T00:42:27.197397+0000","last_active":"2024-02-23T00:43:08.662359+0000","last_peered":"2024-02-23T00:43:08.662359+0000","last_clean":"2024-02-23T00:43:08.662359+0000","last_became_active":"2024-02-23T00:41:37.252612+0000","last_became_peered":"2024-02-23T00:41:37.252612+0000","last_unstale":"2024-02-23T00:43:08.662359+0000","last_undegraded":"2024-02-23T00:43:08.662359+0000","last_fullsized":"2024-02-23T00:43:08.662359+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.197290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":710,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.383009+0000","last_change":"2024-02-23T00:42:12.891119+0000","last_active":"2024-02-23T00:43:09.383009+0000","last_peered":"2024-02-23T00:43:09.383009+0000","last_clean":"2024-02-23T00:43:09.383009+0000","last_became_active":"2024-02-23T00:41:37.238185+0000","last_became_peered":"2024-02-23T00:41:37.238185+0000","last_unstale":"2024-02-23T00:43:09.383009+0000","last_undegraded":"2024-02-23T00:43:09.383009+0000","last_fullsized":"2024-02-23T00:43:09.383009+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.891052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.891052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"78'6","reported_seq":34,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662208+0000","last_change":"2024-02-23T00:43:00.726511+0000","last_active":"2024-02-23T00:43:08.662208+0000","last_peered":"2024-02-23T00:43:08.662208+0000","last_clean":"2024-02-23T00:43:08.662208+0000","last_became_active":"2024-02-23T00:42:41.397727+0000","last_became_peered":"2024-02-23T00:42:41.397727+0000","last_unstale":"2024-02-23T00:43:08.662208+0000","last_undegraded":"2024-02-23T00:43:08.662208+0000","last_fullsized":"2024-02-23T00:43:08.662208+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:43:00.726434+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.726434+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":794,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.385127+0000","last_change":"2024-02-23T00:43:06.975177+0000","last_active":"2024-02-23T00:43:09.385127+0000","last_peered":"2024-02-23T00:43:09.385127+0000","last_clean":"2024-02-23T00:43:09.385127+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:43:09.385127+0000","last_undegraded":"2024-02-23T00:43:09.385127+0000","last_fullsized":"2024-02-23T00:43:09.385127+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:43:06.975089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.975089+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":750,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259719+0000","last_change":"2024-02-23T00:41:37.230185+0000","last_active":"2024-02-23T00:43:09.259719+0000","last_peered":"2024-02-23T00:43:09.259719+0000","last_clean":"2024-02-23T00:43:09.259719+0000","last_became_active":"2024-02-23T00:41:37.229726+0000","last_became_peered":"2024-02-23T00:41:37.229726+0000","last_unstale":"2024-02-23T00:43:09.259719+0000","last_undegraded":"2024-02-23T00:43:09.259719+0000","last_fullsized":"2024-02-23T00:43:09.259719+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:56.203202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:40:56.203202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":745,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.260989+0000","last_change":"2024-02-23T00:41:35.138434+0000","last_active":"2024-02-23T00:43:09.260989+0000","last_peered":"2024-02-23T00:43:09.260989+0000","last_clean":"2024-02-23T00:43:09.260989+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:43:09.260989+0000","last_undegraded":"2024-02-23T00:43:09.260989+0000","last_fullsized":"2024-02-23T00:43:09.260989+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.138319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.138319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":736,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628198+0000","last_change":"2024-02-23T00:42:22.669179+0000","last_active":"2024-02-23T00:43:08.628198+0000","last_peered":"2024-02-23T00:43:08.628198+0000","last_clean":"2024-02-23T00:43:08.628198+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:43:08.628198+0000","last_undegraded":"2024-02-23T00:43:08.628198+0000","last_fullsized":"2024-02-23T00:43:08.628198+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:42:22.669090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.669090+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":272,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662091+0000","last_change":"2024-02-23T00:43:04.142262+0000","last_active":"2024-02-23T00:43:08.662091+0000","last_peered":"2024-02-23T00:43:08.662091+0000","last_clean":"2024-02-23T00:43:08.662091+0000","last_became_active":"2024-02-23T00:38:49.217612+0000","last_became_peered":"2024-02-23T00:38:49.217612+0000","last_unstale":"2024-02-23T00:43:08.662091+0000","last_undegraded":"2024-02-23T00:43:08.662091+0000","last_fullsized":"2024-02-23T00:43:08.662091+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'10","last_scrub_stamp":"2024-02-23T00:43:04.142193+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:43:04.142193+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":732,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257614+0000","last_change":"2024-02-23T00:42:15.038961+0000","last_active":"2024-02-23T00:43:08.257614+0000","last_peered":"2024-02-23T00:43:08.257614+0000","last_clean":"2024-02-23T00:43:08.257614+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:43:08.257614+0000","last_undegraded":"2024-02-23T00:43:08.257614+0000","last_fullsized":"2024-02-23T00:43:08.257614+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.038897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.038897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":736,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627350+0000","last_change":"2024-02-23T00:41:34.564079+0000","last_active":"2024-02-23T00:43:08.627350+0000","last_peered":"2024-02-23T00:43:08.627350+0000","last_clean":"2024-02-23T00:43:08.627350+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:43:08.627350+0000","last_undegraded":"2024-02-23T00:43:08.627350+0000","last_fullsized":"2024-02-23T00:43:08.627350+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:34.563969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.563969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":678,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.261069+0000","last_change":"2024-02-23T00:41:44.196162+0000","last_active":"2024-02-23T00:43:09.261069+0000","last_peered":"2024-02-23T00:43:09.261069+0000","last_clean":"2024-02-23T00:43:09.261069+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:43:09.261069+0000","last_undegraded":"2024-02-23T00:43:09.261069+0000","last_fullsized":"2024-02-23T00:43:09.261069+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.196088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.196088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257325+0000","last_change":"2024-02-23T00:42:41.391815+0000","last_active":"2024-02-23T00:43:08.257325+0000","last_peered":"2024-02-23T00:43:08.257325+0000","last_clean":"2024-02-23T00:43:08.257325+0000","last_became_active":"2024-02-23T00:42:41.391484+0000","last_became_peered":"2024-02-23T00:42:41.391484+0000","last_unstale":"2024-02-23T00:43:08.257325+0000","last_undegraded":"2024-02-23T00:43:08.257325+0000","last_fullsized":"2024-02-23T00:43:08.257325+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.789819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.789819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":278,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662327+0000","last_change":"2024-02-23T00:40:43.233713+0000","last_active":"2024-02-23T00:43:08.662327+0000","last_peered":"2024-02-23T00:43:08.662327+0000","last_clean":"2024-02-23T00:43:08.662327+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:43:08.662327+0000","last_undegraded":"2024-02-23T00:43:08.662327+0000","last_fullsized":"2024-02-23T00:43:08.662327+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:40:43.233609+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:40:43.233609+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":744,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628472+0000","last_change":"2024-02-23T00:42:25.724438+0000","last_active":"2024-02-23T00:43:08.628472+0000","last_peered":"2024-02-23T00:43:08.628472+0000","last_clean":"2024-02-23T00:43:08.628472+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:43:08.628472+0000","last_undegraded":"2024-02-23T00:43:08.628472+0000","last_fullsized":"2024-02-23T00:43:08.628472+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:25.724375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:25.724375+0000","last_clean_scrub_stamp":"2024-02-23T00:42:25.724375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":735,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661359+0000","last_change":"2024-02-23T00:41:44.520521+0000","last_active":"2024-02-23T00:43:08.661359+0000","last_peered":"2024-02-23T00:43:08.661359+0000","last_clean":"2024-02-23T00:43:08.661359+0000","last_became_active":"2024-02-23T00:27:21.027832+0000","last_became_peered":"2024-02-23T00:27:21.027832+0000","last_unstale":"2024-02-23T00:43:08.661359+0000","last_undegraded":"2024-02-23T00:43:08.661359+0000","last_fullsized":"2024-02-23T00:43:08.661359+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.520415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.520415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":261,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662300+0000","last_change":"2024-02-23T00:41:37.096978+0000","last_active":"2024-02-23T00:43:08.662300+0000","last_peered":"2024-02-23T00:43:08.662300+0000","last_clean":"2024-02-23T00:43:08.662300+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:43:08.662300+0000","last_undegraded":"2024-02-23T00:43:08.662300+0000","last_fullsized":"2024-02-23T00:43:08.662300+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:37.096906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:37.096906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":652,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.361983+0000","last_change":"2024-02-23T00:42:53.560946+0000","last_active":"2024-02-23T00:43:06.361983+0000","last_peered":"2024-02-23T00:43:06.361983+0000","last_clean":"2024-02-23T00:43:06.361983+0000","last_became_active":"2024-02-23T00:42:10.240911+0000","last_became_peered":"2024-02-23T00:42:10.240911+0000","last_unstale":"2024-02-23T00:43:06.361983+0000","last_undegraded":"2024-02-23T00:43:06.361983+0000","last_fullsized":"2024-02-23T00:43:06.361983+0000","mapping_epoch":707,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:42:53.560884+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.560884+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":741,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.256256+0000","last_change":"2024-02-23T00:43:05.149002+0000","last_active":"2024-02-23T00:43:08.256256+0000","last_peered":"2024-02-23T00:43:08.256256+0000","last_clean":"2024-02-23T00:43:08.256256+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:43:08.256256+0000","last_undegraded":"2024-02-23T00:43:08.256256+0000","last_fullsized":"2024-02-23T00:43:08.256256+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:05.148931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:55.816155+0000","last_clean_scrub_stamp":"2024-02-23T00:43:05.148931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":943,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.381890+0000","last_change":"2024-02-23T00:42:59.626715+0000","last_active":"2024-02-23T00:43:09.381890+0000","last_peered":"2024-02-23T00:43:09.381890+0000","last_clean":"2024-02-23T00:43:09.381890+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:43:09.381890+0000","last_undegraded":"2024-02-23T00:43:09.381890+0000","last_fullsized":"2024-02-23T00:43:09.381890+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:42:59.626636+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.626636+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":730,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662585+0000","last_change":"2024-02-23T00:42:14.427990+0000","last_active":"2024-02-23T00:43:08.662585+0000","last_peered":"2024-02-23T00:43:08.662585+0000","last_clean":"2024-02-23T00:43:08.662585+0000","last_became_active":"2024-02-23T00:41:37.229426+0000","last_became_peered":"2024-02-23T00:41:37.229426+0000","last_unstale":"2024-02-23T00:43:08.662585+0000","last_undegraded":"2024-02-23T00:43:08.662585+0000","last_fullsized":"2024-02-23T00:43:08.662585+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.427861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.427861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":948,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628420+0000","last_change":"2024-02-23T00:42:20.635986+0000","last_active":"2024-02-23T00:43:08.628420+0000","last_peered":"2024-02-23T00:43:08.628420+0000","last_clean":"2024-02-23T00:43:08.628420+0000","last_became_active":"2024-02-23T00:41:55.327964+0000","last_became_peered":"2024-02-23T00:41:55.327964+0000","last_unstale":"2024-02-23T00:43:08.628420+0000","last_undegraded":"2024-02-23T00:43:08.628420+0000","last_fullsized":"2024-02-23T00:43:08.628420+0000","mapping_epoch":692,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:42:20.635897+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.635897+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":285,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662216+0000","last_change":"2024-02-23T00:42:14.278845+0000","last_active":"2024-02-23T00:43:08.662216+0000","last_peered":"2024-02-23T00:43:08.662216+0000","last_clean":"2024-02-23T00:43:08.662216+0000","last_became_active":"2024-02-23T00:41:37.228694+0000","last_became_peered":"2024-02-23T00:41:37.228694+0000","last_unstale":"2024-02-23T00:43:08.662216+0000","last_undegraded":"2024-02-23T00:43:08.662216+0000","last_fullsized":"2024-02-23T00:43:08.662216+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.278782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.278782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":92,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.362059+0000","last_change":"2024-02-23T00:42:55.607417+0000","last_active":"2024-02-23T00:43:06.362059+0000","last_peered":"2024-02-23T00:43:06.362059+0000","last_clean":"2024-02-23T00:43:06.362059+0000","last_became_active":"2024-02-23T00:41:37.247946+0000","last_became_peered":"2024-02-23T00:41:37.247946+0000","last_unstale":"2024-02-23T00:43:06.362059+0000","last_undegraded":"2024-02-23T00:43:06.362059+0000","last_fullsized":"2024-02-23T00:43:06.362059+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:55.607352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:55.607352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":687,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.664361+0000","last_change":"2024-02-23T00:41:53.497861+0000","last_active":"2024-02-23T00:43:08.664361+0000","last_peered":"2024-02-23T00:43:08.664361+0000","last_clean":"2024-02-23T00:43:08.664361+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:43:08.664361+0000","last_undegraded":"2024-02-23T00:43:08.664361+0000","last_fullsized":"2024-02-23T00:43:08.664361+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.497792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.497792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":878,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259139+0000","last_change":"2024-02-23T00:42:16.497262+0000","last_active":"2024-02-23T00:43:09.259139+0000","last_peered":"2024-02-23T00:43:09.259139+0000","last_clean":"2024-02-23T00:43:09.259139+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:43:09.259139+0000","last_undegraded":"2024-02-23T00:43:09.259139+0000","last_fullsized":"2024-02-23T00:43:09.259139+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:42:16.497198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.497198+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":748,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.628342+0000","last_change":"2024-02-23T00:41:48.632172+0000","last_active":"2024-02-23T00:43:08.628342+0000","last_peered":"2024-02-23T00:43:08.628342+0000","last_clean":"2024-02-23T00:43:08.628342+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:43:08.628342+0000","last_undegraded":"2024-02-23T00:43:08.628342+0000","last_fullsized":"2024-02-23T00:43:08.628342+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:48.632098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:41:48.632098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":748,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259964+0000","last_change":"2024-02-23T00:41:37.230024+0000","last_active":"2024-02-23T00:43:09.259964+0000","last_peered":"2024-02-23T00:43:09.259964+0000","last_clean":"2024-02-23T00:43:09.259964+0000","last_became_active":"2024-02-23T00:41:37.229323+0000","last_became_peered":"2024-02-23T00:41:37.229323+0000","last_unstale":"2024-02-23T00:43:09.259964+0000","last_undegraded":"2024-02-23T00:43:09.259964+0000","last_fullsized":"2024-02-23T00:43:09.259964+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:57.174945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:40:57.174945+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":268,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661485+0000","last_change":"2024-02-23T00:43:07.162145+0000","last_active":"2024-02-23T00:43:08.661485+0000","last_peered":"2024-02-23T00:43:08.661485+0000","last_clean":"2024-02-23T00:43:08.661485+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:43:08.661485+0000","last_undegraded":"2024-02-23T00:43:08.661485+0000","last_fullsized":"2024-02-23T00:43:08.661485+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.162083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.162083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":845,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661064+0000","last_change":"2024-02-23T00:40:58.399481+0000","last_active":"2024-02-23T00:43:08.661064+0000","last_peered":"2024-02-23T00:43:08.661064+0000","last_clean":"2024-02-23T00:43:08.661064+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:43:08.661064+0000","last_undegraded":"2024-02-23T00:43:08.661064+0000","last_fullsized":"2024-02-23T00:43:08.661064+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_deep_scrub":"453'44","last_deep_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.399368+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":261,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662493+0000","last_change":"2024-02-23T00:40:07.255442+0000","last_active":"2024-02-23T00:43:08.662493+0000","last_peered":"2024-02-23T00:43:08.662493+0000","last_clean":"2024-02-23T00:43:08.662493+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:43:08.662493+0000","last_undegraded":"2024-02-23T00:43:08.662493+0000","last_fullsized":"2024-02-23T00:43:08.662493+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:07.255349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:07.255349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":571,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662656+0000","last_change":"2024-02-23T00:41:50.257038+0000","last_active":"2024-02-23T00:43:08.662656+0000","last_peered":"2024-02-23T00:43:08.662656+0000","last_clean":"2024-02-23T00:43:08.662656+0000","last_became_active":"2024-02-23T00:41:50.256751+0000","last_became_peered":"2024-02-23T00:41:50.256751+0000","last_unstale":"2024-02-23T00:43:08.662656+0000","last_undegraded":"2024-02-23T00:43:08.662656+0000","last_fullsized":"2024-02-23T00:43:08.662656+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:07.610329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:07.610329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":681,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257556+0000","last_change":"2024-02-23T00:43:07.133088+0000","last_active":"2024-02-23T00:43:08.257556+0000","last_peered":"2024-02-23T00:43:08.257556+0000","last_clean":"2024-02-23T00:43:08.257556+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:43:08.257556+0000","last_undegraded":"2024-02-23T00:43:08.257556+0000","last_fullsized":"2024-02-23T00:43:08.257556+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.133007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.133007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":945,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.660840+0000","last_change":"2024-02-23T00:41:40.216294+0000","last_active":"2024-02-23T00:43:08.660840+0000","last_peered":"2024-02-23T00:43:08.660840+0000","last_clean":"2024-02-23T00:43:08.660840+0000","last_became_active":"2024-02-23T00:41:40.215919+0000","last_became_peered":"2024-02-23T00:41:40.215919+0000","last_unstale":"2024-02-23T00:43:08.660840+0000","last_undegraded":"2024-02-23T00:43:08.660840+0000","last_fullsized":"2024-02-23T00:43:08.660840+0000","mapping_epoch":680,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":449,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.383333+0000","last_change":"2024-02-23T00:42:14.878836+0000","last_active":"2024-02-23T00:43:09.383333+0000","last_peered":"2024-02-23T00:43:09.383333+0000","last_clean":"2024-02-23T00:43:09.383333+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:43:09.383333+0000","last_undegraded":"2024-02-23T00:43:09.383333+0000","last_fullsized":"2024-02-23T00:43:09.383333+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.878770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.878770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":747,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.257147+0000","last_change":"2024-02-23T00:43:00.260324+0000","last_active":"2024-02-23T00:43:08.257147+0000","last_peered":"2024-02-23T00:43:08.257147+0000","last_clean":"2024-02-23T00:43:08.257147+0000","last_became_active":"2024-02-23T00:41:37.228310+0000","last_became_peered":"2024-02-23T00:41:37.228310+0000","last_unstale":"2024-02-23T00:43:08.257147+0000","last_undegraded":"2024-02-23T00:43:08.257147+0000","last_fullsized":"2024-02-23T00:43:08.257147+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:00.260242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.260242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":692,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.660769+0000","last_change":"2024-02-23T00:42:19.323714+0000","last_active":"2024-02-23T00:43:08.660769+0000","last_peered":"2024-02-23T00:43:08.660769+0000","last_clean":"2024-02-23T00:43:08.660769+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:43:08.660769+0000","last_undegraded":"2024-02-23T00:43:08.660769+0000","last_fullsized":"2024-02-23T00:43:08.660769+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.323643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.323643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":276,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662242+0000","last_change":"2024-02-23T00:41:50.256190+0000","last_active":"2024-02-23T00:43:08.662242+0000","last_peered":"2024-02-23T00:43:08.662242+0000","last_clean":"2024-02-23T00:43:08.662242+0000","last_became_active":"2024-02-23T00:41:50.255862+0000","last_became_peered":"2024-02-23T00:41:50.255862+0000","last_unstale":"2024-02-23T00:43:08.662242+0000","last_undegraded":"2024-02-23T00:43:08.662242+0000","last_fullsized":"2024-02-23T00:43:08.662242+0000","mapping_epoch":688,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:41:00.643135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:00.643135+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":763,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.663979+0000","last_change":"2024-02-23T00:42:56.638098+0000","last_active":"2024-02-23T00:43:08.663979+0000","last_peered":"2024-02-23T00:43:08.663979+0000","last_clean":"2024-02-23T00:43:08.663979+0000","last_became_active":"2024-02-23T00:41:37.230054+0000","last_became_peered":"2024-02-23T00:41:37.230054+0000","last_unstale":"2024-02-23T00:43:08.663979+0000","last_undegraded":"2024-02-23T00:43:08.663979+0000","last_fullsized":"2024-02-23T00:43:08.663979+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:56.638011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:42:56.638011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":740,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662865+0000","last_change":"2024-02-23T00:42:26.307705+0000","last_active":"2024-02-23T00:43:08.662865+0000","last_peered":"2024-02-23T00:43:08.662865+0000","last_clean":"2024-02-23T00:43:08.662865+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:43:08.662865+0000","last_undegraded":"2024-02-23T00:43:08.662865+0000","last_fullsized":"2024-02-23T00:43:08.662865+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.307631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.307631+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.307631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":702,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.259953+0000","last_change":"2024-02-23T00:41:47.254406+0000","last_active":"2024-02-23T00:43:09.259953+0000","last_peered":"2024-02-23T00:43:09.259953+0000","last_clean":"2024-02-23T00:43:09.259953+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:43:09.259953+0000","last_undegraded":"2024-02-23T00:43:09.259953+0000","last_fullsized":"2024-02-23T00:43:09.259953+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.254328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.254328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":891,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.382417+0000","last_change":"2024-02-23T00:42:27.911692+0000","last_active":"2024-02-23T00:43:09.382417+0000","last_peered":"2024-02-23T00:43:09.382417+0000","last_clean":"2024-02-23T00:43:09.382417+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:43:09.382417+0000","last_undegraded":"2024-02-23T00:43:09.382417+0000","last_fullsized":"2024-02-23T00:43:09.382417+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:42:27.911611+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.911611+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":285,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661559+0000","last_change":"2024-02-23T00:42:53.263490+0000","last_active":"2024-02-23T00:43:08.661559+0000","last_peered":"2024-02-23T00:43:08.661559+0000","last_clean":"2024-02-23T00:43:08.661559+0000","last_became_active":"2024-02-23T00:41:37.241142+0000","last_became_peered":"2024-02-23T00:41:37.241142+0000","last_unstale":"2024-02-23T00:43:08.661559+0000","last_undegraded":"2024-02-23T00:43:08.661559+0000","last_fullsized":"2024-02-23T00:43:08.661559+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.263425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.263425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":396,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627122+0000","last_change":"2024-02-23T00:43:06.705370+0000","last_active":"2024-02-23T00:43:08.627122+0000","last_peered":"2024-02-23T00:43:08.627122+0000","last_clean":"2024-02-23T00:43:08.627122+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:43:08.627122+0000","last_undegraded":"2024-02-23T00:43:08.627122+0000","last_fullsized":"2024-02-23T00:43:08.627122+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.705283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.705283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":169,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.661819+0000","last_change":"2024-02-23T00:41:34.730482+0000","last_active":"2024-02-23T00:43:08.661819+0000","last_peered":"2024-02-23T00:43:08.661819+0000","last_clean":"2024-02-23T00:43:08.661819+0000","last_became_active":"2024-02-23T00:40:23.199068+0000","last_became_peered":"2024-02-23T00:40:23.199068+0000","last_unstale":"2024-02-23T00:43:08.661819+0000","last_undegraded":"2024-02-23T00:43:08.661819+0000","last_fullsized":"2024-02-23T00:43:08.661819+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:34.730403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.730403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":1036,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.662215+0000","last_change":"2024-02-23T00:41:41.223239+0000","last_active":"2024-02-23T00:43:08.662215+0000","last_peered":"2024-02-23T00:43:08.662215+0000","last_clean":"2024-02-23T00:43:08.662215+0000","last_became_active":"2024-02-23T00:41:41.222892+0000","last_became_peered":"2024-02-23T00:41:41.222892+0000","last_unstale":"2024-02-23T00:43:08.662215+0000","last_undegraded":"2024-02-23T00:43:08.662215+0000","last_fullsized":"2024-02-23T00:43:08.662215+0000","mapping_epoch":681,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:41:34.076323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.076323+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":92,"reported_epoch":755,"state":"active+clean","last_fresh":"2024-02-23T00:43:06.361589+0000","last_change":"2024-02-23T00:42:28.612255+0000","last_active":"2024-02-23T00:43:06.361589+0000","last_peered":"2024-02-23T00:43:06.361589+0000","last_clean":"2024-02-23T00:43:06.361589+0000","last_became_active":"2024-02-23T00:41:37.234830+0000","last_became_peered":"2024-02-23T00:41:37.234830+0000","last_unstale":"2024-02-23T00:43:06.361589+0000","last_undegraded":"2024-02-23T00:43:06.361589+0000","last_fullsized":"2024-02-23T00:43:06.361589+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:28.612113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:28.612113+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.612113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":87,"reported_epoch":757,"state":"active+clean","last_fresh":"2024-02-23T00:43:08.627637+0000","last_change":"2024-02-23T00:42:13.611259+0000","last_active":"2024-02-23T00:43:08.627637+0000","last_peered":"2024-02-23T00:43:08.627637+0000","last_clean":"2024-02-23T00:43:08.627637+0000","last_became_active":"2024-02-23T00:41:50.253284+0000","last_became_peered":"2024-02-23T00:41:50.253284+0000","last_unstale":"2024-02-23T00:43:08.627637+0000","last_undegraded":"2024-02-23T00:43:08.627637+0000","last_fullsized":"2024-02-23T00:43:08.627637+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:13.611193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.611193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":701,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-02-23T00:43:09.258400+0000","last_change":"2024-02-23T00:42:20.220171+0000","last_active":"2024-02-23T00:43:09.258400+0000","last_peered":"2024-02-23T00:43:09.258400+0000","last_clean":"2024-02-23T00:43:09.258400+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:43:09.258400+0000","last_undegraded":"2024-02-23T00:43:09.258400+0000","last_fullsized":"2024-02-23T00:43:09.258400+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.220094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.220094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":250,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4000249,"num_objects":37,"num_object_clones":1,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":57,"num_write_kb":3959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6438912,"data_stored":6438912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":24,"acting":24,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":12,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60,"ondisk_log_size":60,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3757,"num_objects":213,"num_object_clones":0,"num_object_copies":639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":678,"num_read_kb":549,"num_write":412,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":3301,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":732,"ondisk_log_size":732,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":23,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":630,"num_read_kb":630,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":69,"acting":69,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332290,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303616,"kb_used_data":6392,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441536,"statfs":{"total":95995035648,"available":95684132864,"internally_reserved":0,"allocated":6545408,"data_stored":3477501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.61899999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46100000000000002,"15min":0.67300000000000004},"min":{"1min":0.34300000000000003,"5min":0.23999999999999999,"15min":0.217},"max":{"1min":0.625,"5min":1.23,"15min":31.175000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.65600000000000003,"15min":0.73699999999999999},"min":{"1min":0.35299999999999998,"5min":0.28599999999999998,"15min":0.249},"max":{"1min":0.63500000000000001,"5min":11.066000000000001,"15min":29.423999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.68999999999999995,"15min":0.71699999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28100000000000003},"max":{"1min":0.64000000000000001,"5min":8.0099999999999998,"15min":16.603999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.59499999999999997,"15min":0.63800000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.26600000000000001},"max":{"1min":0.70599999999999996,"5min":5.7229999999999999,"15min":7.5339999999999998},"last":0.55500000000000005}]},{"osd":3,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.61099999999999999,"15min":1.0069999999999999},"min":{"1min":0.36499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61699999999999999,"5min":8.0449999999999999,"15min":57.729999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54900000000000004,"15min":0.91300000000000003},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.67900000000000005,"5min":3.8479999999999999,"15min":27.573},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.40200000000000002,"15min":0.41699999999999998},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.69599999999999995,"5min":1.2589999999999999,"15min":1.706},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.67300000000000004,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.38100000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41499999999999998,"15min":0.42699999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":1.0209999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.438,"5min":0.45000000000000001,"15min":0.46400000000000002},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.59699999999999998,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.53000000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:42:42 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.46200000000000002,"15min":0.48499999999999999},"min":{"1min":0.30399999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.65100000000000002,"5min":1.3129999999999999,"15min":2.1339999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51100000000000001,"15min":0.49299999999999999},"min":{"1min":0.33800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80500000000000005,"5min":5.194,"15min":5.194},"last":0.433}]}]},{"osd":6,"up_from":44,"seq":188978561223,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307760,"kb_used_data":6440,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437392,"statfs":{"total":95995035648,"available":95679889408,"internally_reserved":0,"allocated":6594560,"data_stored":3538503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.627},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.48099999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53400000000000003,"15min":1.024},"min":{"1min":0.40100000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.66100000000000003,"5min":6.056,"15min":59.347999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.75900000000000001,"15min":1.3799999999999999},"min":{"1min":0.41299999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.1619999999999999,"5min":18.423999999999999,"15min":87.221000000000004},"last":0.67100000000000004}]},{"osd":2,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.77300000000000002,"15min":0.90500000000000003},"min":{"1min":0.35699999999999998,"5min":0.214,"15min":0.214},"max":{"1min":1.1120000000000001,"5min":18.381,"15min":25.550999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66100000000000003,"15min":0.91700000000000004},"min":{"1min":0.375,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.9460000000000002,"5min":6.1749999999999998,"15min":40.914000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63800000000000001,"15min":0.85799999999999998},"min":{"1min":0.40999999999999998,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":2.859,"5min":7.1630000000000003,"15min":31.309999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.754,"15min":1.4059999999999999},"min":{"1min":0.44600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72299999999999998,"5min":18.448,"15min":86.311999999999998},"last":0.73399999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.53500000000000003,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60499999999999998,"5min":7.8769999999999998,"15min":7.8769999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48199999999999998,"15min":0.53000000000000003},"min":{"1min":0.307,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":6.0300000000000002,"15min":17.718},"last":0.377}]},{"osd":5,"last update":"Fri Feb 23 00:42:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.54900000000000004,"15min":0.52200000000000002},"min":{"1min":0.30599999999999999,"5min":0.27800000000000002,"15min":0.254},"max":{"1min":0.60399999999999998,"5min":6.7999999999999998,"15min":8.0990000000000002},"last":0.504},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.505},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.82599999999999996,"5min":6.0780000000000003,"15min":8.1370000000000005},"last":0.45400000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54000000000000004,"15min":0.61899999999999999},"min":{"1min":0.32100000000000001,"5min":0.28999999999999998,"15min":0.27300000000000002},"max":{"1min":0.70799999999999996,"5min":6.4279999999999999,"15min":17.881},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.26500000000000001},"max":{"1min":0.74399999999999999,"5min":18.475000000000001,"15min":18.475000000000001},"last":0.55000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575070,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309732,"kb_used_data":8412,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93435420,"statfs":{"total":95995035648,"available":95677870080,"internally_reserved":0,"allocated":8613888,"data_stored":5509150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.28999999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36699999999999999,"15min":0.66700000000000004},"min":{"1min":0.20799999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.54000000000000004,"5min":3.9980000000000002,"15min":35.313000000000002},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.39700000000000002,"15min":0.57899999999999996},"min":{"1min":0.27300000000000002,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.623,"5min":2.7999999999999998,"15min":27.908999999999999},"last":0.37}]},{"osd":3,"last update":"Fri Feb 23 00:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.44900000000000001,"15min":0.64600000000000002},"min":{"1min":0.245,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.52400000000000002,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41399999999999998,"15min":0.63300000000000001},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47799999999999998,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58399999999999996,"15min":0.70699999999999996},"min":{"1min":0.39000000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.877,"5min":3.4900000000000002,"15min":16.231000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.52700000000000002,"15min":0.71599999999999997},"min":{"1min":0.32200000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":29.212},"last":0.52700000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.57299999999999995,"15min":0.71699999999999997},"min":{"1min":0.371,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72099999999999997,"5min":2.8410000000000002,"15min":16.294},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59899999999999998,"15min":0.68200000000000005},"min":{"1min":0.35399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72899999999999998,"5min":4.3380000000000001,"15min":11.105},"last":0.51500000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.53600000000000003,"15min":0.754},"min":{"1min":0.39500000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.76,"5min":2.4049999999999998,"15min":29.187000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60999999999999999,"15min":0.749},"min":{"1min":0.44,"5min":0.312,"15min":0.312},"max":{"1min":0.90500000000000003,"5min":2.927,"15min":16.369},"last":0.63600000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.63500000000000001,"15min":0.78800000000000003},"min":{"1min":0.39500000000000002,"5min":0.33300000000000002,"15min":0.32000000000000001},"max":{"1min":0.82099999999999995,"5min":5.2119999999999997,"15min":16.395},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.622,"15min":0.66600000000000004},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81899999999999995,"5min":3.7669999999999999,"15min":11.135999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":659,"seq":2830383448087,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309300,"kb_used_data":6380,"kb_used_omap":85,"kb_used_meta":302826,"kb_avail":93435852,"statfs":{"total":95995035648,"available":95678312448,"internally_reserved":0,"allocated":6533120,"data_stored":3478253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":310093952},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.0880000000000001,"5min":5.0880000000000001,"15min":5.0880000000000001},"last":0.315},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.27200000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.0540000000000003,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.35999999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":11.891,"5min":11.891,"15min":11.891},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":15.864000000000001,"5min":15.864000000000001,"15min":15.864000000000001},"last":0.35399999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.68999999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.65800000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71399999999999997}]},{"osd":7,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.1280000000000001,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.85699999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411546,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320024,"kb_used_data":6416,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93425128,"statfs":{"total":95995035648,"available":95667331072,"internally_reserved":0,"allocated":6569984,"data_stored":3516491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:20 2024","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.35899999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34300000000000003,"15min":0.59399999999999997},"min":{"1min":0.23300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.84399999999999997,"5min":1.8919999999999999,"15min":19.324999999999999},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.47399999999999998,"15min":0.752},"min":{"1min":0.251,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68500000000000005,"5min":7.9779999999999998,"15min":35.052},"last":0.218}]},{"osd":3,"last update":"Fri Feb 23 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.501,"15min":0.73199999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.46999999999999997,"5min":10.032,"15min":43.5},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.53000000000000003,"15min":0.68400000000000005},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.5,"5min":11.815,"15min":27.414000000000001},"last":0.34100000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:42:30 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.52100000000000002,"15min":0.67000000000000004},"min":{"1min":0.432,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.81499999999999995,"5min":2.0899999999999999,"15min":14.614000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53500000000000003,"15min":0.621},"min":{"1min":0.39400000000000002,"5min":0.30199999999999999,"15min":0.27600000000000002},"max":{"1min":0.89500000000000002,"5min":1.9750000000000001,"15min":7.766},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.316},"max":{"1min":0.77200000000000002,"5min":7.915,"15min":17.765000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.55500000000000005,"15min":0.67400000000000004},"min":{"1min":0.39600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.79600000000000004,"5min":2.129,"15min":14.589},"last":0.63600000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.628},"min":{"1min":0.42399999999999999,"5min":0.316,"15min":0.29699999999999999},"max":{"1min":0.81100000000000005,"5min":2.2250000000000001,"15min":7.7869999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71999999999999997,"15min":0.79300000000000004},"min":{"1min":0.41799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.218,"5min":7.9589999999999996,"15min":20.574999999999999},"last":0.52200000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.70499999999999996,"15min":1.073},"min":{"1min":0.38700000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.81599999999999995,"5min":7.9480000000000004,"15min":66.632000000000005},"last":1.004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58199999999999996,"15min":0.84799999999999998},"min":{"1min":0.44800000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78800000000000003,"5min":2.1789999999999998,"15min":19.603999999999999},"last":0.47499999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149910,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313808,"kb_used_data":8392,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431344,"statfs":{"total":95995035648,"available":95673696256,"internally_reserved":0,"allocated":8593408,"data_stored":5498268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.40300000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40600000000000003,"15min":0.60299999999999998},"min":{"1min":0.19600000000000001,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":1.1870000000000001,"5min":3.2970000000000002,"15min":54.351999999999997},"last":0.311},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.45700000000000002,"15min":0.77200000000000002},"min":{"1min":0.246,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":0.53400000000000003,"5min":3.7120000000000002,"15min":55.337000000000003},"last":0.38}]},{"osd":2,"last update":"Fri Feb 23 00:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.44900000000000001,"15min":0.752},"min":{"1min":0.27100000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.44700000000000001,"5min":3.7570000000000001,"15min":55.368000000000002},"last":0.435},{"interface":"front","average":{"1min":0.438,"5min":0.46400000000000002,"15min":0.73099999999999998},"min":{"1min":0.28399999999999997,"5min":0.214,"15min":0.185},"max":{"1min":2.0059999999999998,"5min":4.181,"15min":54.383000000000003},"last":0.35999999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:42:28 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.627,"15min":0.93300000000000005},"min":{"1min":0.36499999999999999,"5min":0.34000000000000002,"15min":0.27900000000000003},"max":{"1min":0.72899999999999998,"5min":6.8799999999999999,"15min":54.530000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.63300000000000001,"15min":0.63700000000000001},"min":{"1min":0.314,"5min":0.30199999999999999,"15min":0.24099999999999999},"max":{"1min":5.8109999999999999,"5min":5.8109999999999999,"15min":8.0359999999999996},"last":0.629}]},{"osd":5,"last update":"Fri Feb 23 00:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56000000000000005,"15min":0.877},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.28899999999999998},"max":{"1min":0.90900000000000003,"5min":3.8180000000000001,"15min":55.401000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.66000000000000003,"15min":0.86299999999999999},"min":{"1min":0.35899999999999999,"5min":0.32700000000000001,"15min":0.30299999999999999},"max":{"1min":0.71499999999999997,"5min":7.0179999999999998,"15min":54.454999999999998},"last":0.60399999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:42:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.76800000000000002,"15min":0.85399999999999998},"min":{"1min":0.42999999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.7069999999999999,"5min":9.7490000000000006,"15min":54.305},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.86599999999999999,"15min":1.0389999999999999},"min":{"1min":0.374,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":8.5860000000000003,"5min":9.6820000000000004,"15min":55.424999999999997},"last":0.505}]},{"osd":7,"last update":"Fri Feb 23 00:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.66600000000000004,"15min":0.80200000000000005},"min":{"1min":0.45500000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.6710000000000003,"5min":4.9790000000000001,"15min":54.222000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.95299999999999996,"15min":1.034},"min":{"1min":0.41799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.0459999999999998,"5min":23.928000000000001,"15min":55.460000000000001},"last":0.61499999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986385,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311808,"kb_used_data":6392,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433344,"statfs":{"total":95995035648,"available":95675744256,"internally_reserved":0,"allocated":6545408,"data_stored":3477832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.68000000000000005}]},{"osd":1,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.69299999999999995,"15min":1.1020000000000001},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.64000000000000001,"5min":8.1199999999999992,"15min":41.137999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.65200000000000002,"15min":0.93000000000000005},"min":{"1min":0.40600000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.59299999999999997,"5min":10.952999999999999,"15min":26.210000000000001},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.72199999999999998,"15min":1.1910000000000001},"min":{"1min":0.36699999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64100000000000001,"5min":9.3490000000000002,"15min":67.253},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.83099999999999996,"15min":1.1240000000000001},"min":{"1min":0.374,"5min":0.251,"15min":0.251},"max":{"1min":0.67100000000000004,"5min":14.442,"15min":41.927999999999997},"last":0.65500000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.68999999999999995,"15min":1.1240000000000001},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.627,"5min":9.5579999999999998,"15min":36.301000000000002},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.5,"5min":0.63100000000000001,"15min":1.1970000000000001},"min":{"1min":0.38200000000000001,"5min":0.26700000000000002,"15min":0.24299999999999999},"max":{"1min":0.69499999999999995,"5min":5.1879999999999997,"15min":42.064},"last":0.47899999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.56899999999999995},"min":{"1min":0.28699999999999998,"5min":0.22500000000000001,"15min":0.217},"max":{"1min":0.58099999999999996,"5min":1.4279999999999999,"15min":14.242000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.45400000000000001,"15min":0.61599999999999999},"min":{"1min":0.33500000000000002,"5min":0.255,"15min":0.24299999999999999},"max":{"1min":0.92300000000000004,"5min":1.4079999999999999,"15min":14.124000000000001},"last":0.628}]},{"osd":6,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.46700000000000003,"15min":0.57599999999999996},"min":{"1min":0.315,"5min":0.27300000000000002,"15min":0.254},"max":{"1min":0.67000000000000004,"5min":1.734,"15min":8.0730000000000004},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.54000000000000004,"15min":0.67300000000000004},"min":{"1min":0.30299999999999999,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.61599999999999999,"5min":13.339,"15min":45.814999999999998},"last":0.55100000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:42:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53000000000000003,"15min":0.64300000000000002},"min":{"1min":0.32900000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.5129999999999999,"5min":7.9210000000000003,"15min":17.178999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56599999999999995,"15min":0.71899999999999997},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.28299999999999997},"max":{"1min":0.63800000000000001,"5min":6.3220000000000001,"15min":15.85},"last":0.73999999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757452,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305584,"kb_used_data":8360,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439568,"statfs":{"total":95995035648,"available":95682117632,"internally_reserved":0,"allocated":8560640,"data_stored":5503332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:42:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Feb 23 00:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45300000000000001,"15min":0.753},"min":{"1min":0.36499999999999999,"5min":0.27500000000000002,"15min":0.24099999999999999},"max":{"1min":1.04,"5min":1.1180000000000001,"15min":38.838000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.44,"15min":1.1779999999999999},"min":{"1min":0.312,"5min":0.252,"15min":0.23699999999999999},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":127.794},"last":0.65100000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.64300000000000002,"15min":0.95399999999999996},"min":{"1min":0.32900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.71699999999999997,"5min":11.435,"15min":37.442999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.47599999999999998,"15min":0.69899999999999995},"min":{"1min":0.32600000000000001,"5min":0.27600000000000002,"15min":0.249},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":23.297999999999998},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Feb 23 00:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.61099999999999999,"15min":0.82799999999999996},"min":{"1min":0.371,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.0049999999999999,"5min":11.457000000000001,"15min":24.414999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.49299999999999999,"15min":0.90800000000000003},"min":{"1min":0.317,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":40.865000000000002},"last":0.629}]},{"osd":4,"last update":"Fri Feb 23 00:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.372,"15min":0.78600000000000003},"min":{"1min":0.253,"5min":0.22500000000000001,"15min":0.219},"max":{"1min":0.753,"5min":0.753,"15min":103.13},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.28499999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.745,"5min":11.381,"15min":12.795999999999999},"last":0.45900000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:42:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.56899999999999995,"15min":0.54800000000000004},"min":{"1min":0.32200000000000001,"5min":0.26400000000000001,"15min":0.26200000000000001},"max":{"1min":0.70799999999999996,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.439,"15min":0.82299999999999995},"min":{"1min":0.307,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.75,"5min":1.5229999999999999,"15min":103.61499999999999},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:42:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70299999999999996,"5min":11.478,"15min":11.478},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.46899999999999997,"15min":0.52000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.25800000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":5.4050000000000002},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2060288,"data_stored":2060288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2052096,"data_stored":2052096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2056192,"data_stored":2056192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":250,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":250,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":250,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":250,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":250,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":250,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":250,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":250,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:43:12.706 INFO:tasks.thrashosds.thrasher:Setting 5.10 to [1, 2, 7, 3, 5, 0] 2024-02-23T00:43:12.707 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.10', '1', '2', '7', '3', '5', '0'] 2024-02-23T00:43:12.707 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pg-upmap-items 5.10 1 2 7 3 5 0 2024-02-23T00:43:13.349 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:13.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:13.453 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:13.486 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:13.508 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:13.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:11.441912+0000 mgr.y (mgr.24437) 646 : audit [DBG] from='client.38321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:43:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:11.795299+0000 mgr.y (mgr.24437) 647 : cluster [DBG] pgmap v1163: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:12.688860+0000 osd.6 (osd.6) 316 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:43:13.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:12.690921+0000 osd.6 (osd.6) 317 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:12.775725+0000 osd.7 (osd.7) 350 : cluster [DBG] 3.d scrub starts 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:12.778691+0000 osd.7 (osd.7) 351 : cluster [DBG] 3.d scrub ok 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:12.977811+0000 osd.4 (osd.4) 337 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:12.980531+0000 osd.4 (osd.4) 338 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:13.241483+0000 mon.a (mon.0) 3785 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132"}]': finished 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:13.241545+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:13.265361+0000 mon.a (mon.0) 3787 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:13.265643+0000 mon.a (mon.0) 3788 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132"}]: dispatch 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:11.441912+0000 mgr.y (mgr.24437) 646 : audit [DBG] from='client.38321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:43:13.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:11.795299+0000 mgr.y (mgr.24437) 647 : cluster [DBG] pgmap v1163: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:12.688860+0000 osd.6 (osd.6) 316 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:12.690921+0000 osd.6 (osd.6) 317 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:12.775725+0000 osd.7 (osd.7) 350 : cluster [DBG] 3.d scrub starts 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:12.778691+0000 osd.7 (osd.7) 351 : cluster [DBG] 3.d scrub ok 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:12.977811+0000 osd.4 (osd.4) 337 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:12.980531+0000 osd.4 (osd.4) 338 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:13.241483+0000 mon.a (mon.0) 3785 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132"}]': finished 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:13.241545+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:13.265361+0000 mon.a (mon.0) 3787 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:13.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:13.265643+0000 mon.a (mon.0) 3788 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132"}]: dispatch 2024-02-23T00:43:13.612 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:13.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.551+0000 7fbba0103700 1 -- 172.21.15.87:0/2991903015 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb98104510 msgr2=0x7fbb98106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.551+0000 7fbba0103700 1 --2- 172.21.15.87:0/2991903015 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb98104510 0x7fbb98106900 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 rx=0x7fbb90009c30 tx=0x7fbb90004c50).stop 2024-02-23T00:43:13.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 -- 172.21.15.87:0/2991903015 shutdown_connections 2024-02-23T00:43:13.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 --2- 172.21.15.87:0/2991903015 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb98104510 0x7fbb98106900 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 --2- 172.21.15.87:0/2991903015 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb980f69b0 0x7fbb980f6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 --2- 172.21.15.87:0/2991903015 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb980f7280 0x7fbb98104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 -- 172.21.15.87:0/2991903015 >> 172.21.15.87:0/2991903015 conn(0x7fbb980f26f0 msgr2=0x7fbb980f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 -- 172.21.15.87:0/2991903015 shutdown_connections 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 -- 172.21.15.87:0/2991903015 wait complete. 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 librados: starting msgr at 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 librados: starting objecter 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 Processor -- start 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 -- start start 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 librados: setting wanted keys 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 librados: calling monclient init 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb980f69b0 0x7fbb98110060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb980f7280 0x7fbb98110550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb98104510 0x7fbb9810a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9810a6f0 con 0x7fbb980f7280 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9810a860 con 0x7fbb98104510 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.552+0000 7fbba0103700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9810aa20 con 0x7fbb980f69b0 2024-02-23T00:43:13.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9de9f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb980f69b0 0x7fbb98110060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9d69e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb980f7280 0x7fbb98110550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9de9f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb980f69b0 0x7fbb98110060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33658/0 (socket says 172.21.15.87:33658) 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9de9f700 1 -- 172.21.15.87:0/245147479 learned_addr learned my addr 172.21.15.87:0/245147479 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9e6a0700 1 --2- 172.21.15.87:0/245147479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb98104510 0x7fbb9810a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9d69e700 1 -- 172.21.15.87:0/245147479 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb980f69b0 msgr2=0x7fbb98110060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9d69e700 1 --2- 172.21.15.87:0/245147479 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fbb980f69b0 0x7fbb98110060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9d69e700 1 -- 172.21.15.87:0/245147479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb98104510 msgr2=0x7fbb9810a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9d69e700 1 --2- 172.21.15.87:0/245147479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb98104510 0x7fbb9810a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9d69e700 1 -- 172.21.15.87:0/245147479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb94005040 con 0x7fbb980f7280 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9e6a0700 1 --2- 172.21.15.87:0/245147479 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fbb98104510 0x7fbb9810a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb9d69e700 1 --2- 172.21.15.87:0/245147479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb980f7280 0x7fbb98110550 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7fbb8800c810 tx=0x7fbb88007e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb8effd700 10 client.?.objecter ms_handle_connect 0x7fbb980f7280 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb8effd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8800e4a0 con 0x7fbb980f7280 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb8800ead0 con 0x7fbb980f7280 2024-02-23T00:43:13.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb900097c0 con 0x7fbb980f7280 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb88012a40 con 0x7fbb980f7280 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbba0103700 10 client.27852.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb98116e90 con 0x7fbb980f7280 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.553+0000 7fbba0103700 1 librados: init done 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.554+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbb980606d0 con 0x7fbb980f7280 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.554+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fbb8800ec40 con 0x7fbb980f7280 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb8effd700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbb84062c90 0x7fbb84065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(762..762 src has 400..762) v4 ==== 8530+0+0 (secure 0 0 0) 0x7fbb8807cf80 con 0x7fbb980f7280 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 osd_map(762..762 src has 400..762) v4 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb8effd700 3 client.27852.objecter handle_osd_map got epochs [762,762] > 0 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb8effd700 3 client.27852.objecter handle_osd_map decoding full epoch 762 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb9de9f700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbb84062c90 0x7fbb84065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb8effd700 20 client.27852.objecter dump_active .. 0 homeless 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v762) v1 ==== 54+0+16 (secure 0 0 0) 0x7fbb8807f610 con 0x7fbb980f7280 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 10 client.27852.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 1 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb8effd700 10 client.27852.objecter ms_handle_connect 0x7fbb84062c90 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb9de9f700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbb84062c90 0x7fbb84065150 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7fbb94004930 tx=0x7fbb9400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6801/484081831 conn(0x7fbb980fe4f0 0x7fbb980fe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9810ace0 osd=0 3 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9810ace0 osd=0 3 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9810ace0 osd=0 3 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 1 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9810ace0 osd=0 3 2024-02-23T00:43:13.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 0 1 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9810ace0 osd=0 4 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 10 client.27852.objecter _send_command 1 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb981092d0 con 0x7fbb980fe4f0 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.555+0000 7fbb9e6a0700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6801/484081831 conn(0x7fbb980fe4f0 0x7fbb980fe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.556+0000 7fbb8effd700 10 client.27852.objecter ms_handle_connect 0x7fbb980fe4f0 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.556+0000 7fbb9e6a0700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6801/484081831 conn(0x7fbb980fe4f0 0x7fbb980fe8d0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.556+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb981092d0 con 0x7fbb980fe4f0 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.556+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.556+0000 7fbb8effd700 10 client.27852.objecter _finish_command 1 = system:0 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.556+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9810ace0 osd=0 4 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.556+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 0 1 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 10 client.27852.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:13.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 2 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9810ace0 osd=0 3 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9810ace0 osd=0 3 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9810ace0 osd=0 3 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 2 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9810ace0 osd=0 3 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 0 2 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9810ace0 osd=0 4 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 10 client.27852.objecter _send_command 2 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.571+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb980273c0 con 0x7fbb980fe4f0 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.572+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb980273c0 con 0x7fbb980fe4f0 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.572+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:13.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.572+0000 7fbb8effd700 10 client.27852.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.643 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:43:13.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.572+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9810ace0 osd=0 4 2024-02-23T00:43:13.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.572+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 0 2 2024-02-23T00:43:13.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 10 client.27852.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:13.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 3 2024-02-23T00:43:13.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6805/685354171 conn(0x7fbb980598f0 0x7fbb98059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98028710 osd=1 3 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98028710 osd=1 3 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98028710 osd=1 3 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 3 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb98028710 osd=1 3 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 1 3 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98028710 osd=1 4 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 10 client.27852.objecter _send_command 3 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb9811d7a0 con 0x7fbb980598f0 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbb9de9f700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6805/685354171 conn(0x7fbb980598f0 0x7fbb98059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbb8effd700 10 client.27852.objecter ms_handle_connect 0x7fbb980598f0 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.573+0000 7fbb9de9f700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6805/685354171 conn(0x7fbb980598f0 0x7fbb98059cd0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.574+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb9811d7a0 con 0x7fbb980598f0 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.574+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:43:13.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.574+0000 7fbb8effd700 10 client.27852.objecter _finish_command 3 = system:0 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.574+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb98028710 osd=1 4 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.574+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 1 3 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 10 client.27852.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 4 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98028710 osd=1 3 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98028710 osd=1 3 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98028710 osd=1 3 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 4 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb98028710 osd=1 3 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 1 4 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98028710 osd=1 4 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 10 client.27852.objecter _send_command 4 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.588+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb98059360 con 0x7fbb980598f0 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb98059360 con 0x7fbb980598f0 2024-02-23T00:43:13.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:13.646 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:43:13.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbb8effd700 10 client.27852.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb98028710 osd=1 4 2024-02-23T00:43:13.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 1 4 2024-02-23T00:43:13.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 10 client.27852.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:13.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 5 2024-02-23T00:43:13.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6809/3605394228 conn(0x7fbb98028270 0x7fbb9811c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98059770 osd=2 3 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98059770 osd=2 3 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98059770 osd=2 3 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 5 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb98059770 osd=2 3 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 2 5 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98059770 osd=2 4 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 10 client.27852.objecter _send_command 5 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb9811c6b0 con 0x7fbb98028270 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbb9d69e700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6809/3605394228 conn(0x7fbb98028270 0x7fbb9811c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbb8effd700 10 client.27852.objecter ms_handle_connect 0x7fbb98028270 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.590+0000 7fbb9d69e700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6809/3605394228 conn(0x7fbb98028270 0x7fbb9811c1c0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.591+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb9811c6b0 con 0x7fbb98028270 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.591+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:43:13.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.591+0000 7fbb8effd700 10 client.27852.objecter _finish_command 5 = system:0 2024-02-23T00:43:13.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.591+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb98059770 osd=2 4 2024-02-23T00:43:13.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.591+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 2 5 2024-02-23T00:43:13.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 10 client.27852.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:13.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 6 2024-02-23T00:43:13.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98059770 osd=2 3 2024-02-23T00:43:13.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98059770 osd=2 3 2024-02-23T00:43:13.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98059770 osd=2 3 2024-02-23T00:43:13.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 6 2024-02-23T00:43:13.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb98059770 osd=2 3 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 2 6 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98059770 osd=2 4 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 10 client.27852.objecter _send_command 6 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.605+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb98027830 con 0x7fbb98028270 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.606+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb98027830 con 0x7fbb98028270 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.606+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.606+0000 7fbb8effd700 10 client.27852.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.606+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb98059770 osd=2 4 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.606+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 2 6 2024-02-23T00:43:13.649 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 10 client.27852.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 7 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6813/128059790 conn(0x7fbb9811f920 0x7fbb98121d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811d0a0 osd=3 3 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811d0a0 osd=3 3 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811d0a0 osd=3 3 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 7 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9811d0a0 osd=3 3 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 3 7 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811d0a0 osd=3 4 2024-02-23T00:43:13.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 10 client.27852.objecter _send_command 7 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb98124960 con 0x7fbb9811f920 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbb9e6a0700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6813/128059790 conn(0x7fbb9811f920 0x7fbb98121d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbb9e6a0700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6813/128059790 conn(0x7fbb9811f920 0x7fbb98121d10 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.607+0000 7fbb8effd700 10 client.27852.objecter ms_handle_connect 0x7fbb9811f920 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.608+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb98124960 con 0x7fbb9811f920 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.608+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.608+0000 7fbb8effd700 10 client.27852.objecter _finish_command 7 = system:0 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.608+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9811d0a0 osd=3 4 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.608+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 3 7 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 10 client.27852.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 8 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811d0a0 osd=3 3 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811d0a0 osd=3 3 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811d0a0 osd=3 3 2024-02-23T00:43:13.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 8 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9811d0a0 osd=3 3 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 3 8 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811d0a0 osd=3 4 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 10 client.27852.objecter _send_command 8 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.625+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb9811cb10 con 0x7fbb9811f920 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb9811cb10 con 0x7fbb9811f920 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbb8effd700 10 client.27852.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9811d0a0 osd=3 4 2024-02-23T00:43:13.652 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:43:13.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 3 8 2024-02-23T00:43:13.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 10 client.27852.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:13.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 9 2024-02-23T00:43:13.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6800/4175844757 conn(0x7fbb9811ccb0 0x7fbb98123380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811cb10 osd=4 3 2024-02-23T00:43:13.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811cb10 osd=4 3 2024-02-23T00:43:13.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811cb10 osd=4 3 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 9 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9811cb10 osd=4 3 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 4 9 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811cb10 osd=4 4 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 10 client.27852.objecter _send_command 9 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.626+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb9811bf10 con 0x7fbb9811ccb0 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.627+0000 7fbb9de9f700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6800/4175844757 conn(0x7fbb9811ccb0 0x7fbb98123380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.627+0000 7fbb8effd700 10 client.27852.objecter ms_handle_connect 0x7fbb9811ccb0 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.627+0000 7fbb9de9f700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6800/4175844757 conn(0x7fbb9811ccb0 0x7fbb98123380 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.628+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb9811bf10 con 0x7fbb9811ccb0 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.628+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.628+0000 7fbb8effd700 10 client.27852.objecter _finish_command 9 = system:0 2024-02-23T00:43:13.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.628+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9811cb10 osd=4 4 2024-02-23T00:43:13.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.628+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 4 9 2024-02-23T00:43:13.655 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:13.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 10 client.27852.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:13.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 10 2024-02-23T00:43:13.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811cb10 osd=4 3 2024-02-23T00:43:13.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811cb10 osd=4 3 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811cb10 osd=4 3 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 10 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9811cb10 osd=4 3 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 4 10 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811cb10 osd=4 4 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 10 client.27852.objecter _send_command 10 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.648+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb9811bf10 con 0x7fbb9811ccb0 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.650+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb9811bf10 con 0x7fbb9811ccb0 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.650+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.650+0000 7fbb8effd700 10 client.27852.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.650+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9811cb10 osd=4 4 2024-02-23T00:43:13.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.650+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 4 10 2024-02-23T00:43:13.657 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:43:13.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 10 client.27852.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:13.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 11 2024-02-23T00:43:13.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6804/2550013795 conn(0x7fbb98123d80 0x7fbb981241a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811bf10 osd=5 3 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811bf10 osd=5 3 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811bf10 osd=5 3 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 11 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9811bf10 osd=5 3 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 5 11 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811bf10 osd=5 4 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 10 client.27852.objecter _send_command 11 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb9812b300 con 0x7fbb98123d80 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbb9d69e700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6804/2550013795 conn(0x7fbb98123d80 0x7fbb981241a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbb8effd700 10 client.27852.objecter ms_handle_connect 0x7fbb98123d80 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.651+0000 7fbb9d69e700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6804/2550013795 conn(0x7fbb98123d80 0x7fbb981241a0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.652+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb9812b300 con 0x7fbb98123d80 2024-02-23T00:43:13.658 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.652+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:43:13.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.652+0000 7fbb8effd700 10 client.27852.objecter _finish_command 11 = system:0 2024-02-23T00:43:13.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.652+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9811bf10 osd=5 4 2024-02-23T00:43:13.659 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.652+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 5 11 2024-02-23T00:43:13.673 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:13.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 10 client.27852.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:13.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 12 2024-02-23T00:43:13.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811bf10 osd=5 3 2024-02-23T00:43:13.678 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811bf10 osd=5 3 2024-02-23T00:43:13.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9811bf10 osd=5 3 2024-02-23T00:43:13.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 12 2024-02-23T00:43:13.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9811bf10 osd=5 3 2024-02-23T00:43:13.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 5 12 2024-02-23T00:43:13.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811bf10 osd=5 4 2024-02-23T00:43:13.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 10 client.27852.objecter _send_command 12 2024-02-23T00:43:13.679 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.676+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb98128ba0 con 0x7fbb98123d80 2024-02-23T00:43:13.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb98128ba0 con 0x7fbb98123d80 2024-02-23T00:43:13.680 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:13.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbb8effd700 10 client.27852.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9811bf10 osd=5 4 2024-02-23T00:43:13.681 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 5 12 2024-02-23T00:43:13.681 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 10 client.27852.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 13 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6808/2509408498 conn(0x7fbb98129220 0x7fbb9812b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98128ba0 osd=6 3 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98128ba0 osd=6 3 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98128ba0 osd=6 3 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 13 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb98128ba0 osd=6 3 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 6 13 2024-02-23T00:43:13.682 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98128ba0 osd=6 4 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 10 client.27852.objecter _send_command 13 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.678+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb9812e390 con 0x7fbb98129220 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.679+0000 7fbb9e6a0700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6808/2509408498 conn(0x7fbb98129220 0x7fbb9812b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.679+0000 7fbb8effd700 10 client.27852.objecter ms_handle_connect 0x7fbb98129220 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.679+0000 7fbb9e6a0700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6808/2509408498 conn(0x7fbb98129220 0x7fbb9812b610 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.680+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb9812e390 con 0x7fbb98129220 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.680+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.680+0000 7fbb8effd700 10 client.27852.objecter _finish_command 13 = system:0 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.680+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb98128ba0 osd=6 4 2024-02-23T00:43:13.683 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.680+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 6 13 2024-02-23T00:43:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:11.441912+0000 mgr.y (mgr.24437) 646 : audit [DBG] from='client.38321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:43:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:11.795299+0000 mgr.y (mgr.24437) 647 : cluster [DBG] pgmap v1163: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:12.688860+0000 osd.6 (osd.6) 316 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:43:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:12.690921+0000 osd.6 (osd.6) 317 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:43:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:12.775725+0000 osd.7 (osd.7) 350 : cluster [DBG] 3.d scrub starts 2024-02-23T00:43:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:12.778691+0000 osd.7 (osd.7) 351 : cluster [DBG] 3.d scrub ok 2024-02-23T00:43:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:12.977811+0000 osd.4 (osd.4) 337 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:43:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:12.980531+0000 osd.4 (osd.4) 338 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:43:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:13.241483+0000 mon.a (mon.0) 3785 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132"}]': finished 2024-02-23T00:43:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:13.241545+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-02-23T00:43:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:13.265361+0000 mon.a (mon.0) 3787 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:13.265643+0000 mon.a (mon.0) 3788 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-132"}]: dispatch 2024-02-23T00:43:13.696 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 10 client.27852.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 14 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98128ba0 osd=6 3 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98128ba0 osd=6 3 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb98128ba0 osd=6 3 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 14 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb98128ba0 osd=6 3 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 6 14 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98128ba0 osd=6 4 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 10 client.27852.objecter _send_command 14 2024-02-23T00:43:13.697 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.694+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb9812e390 con 0x7fbb98129220 2024-02-23T00:43:13.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.696+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb9812e390 con 0x7fbb98129220 2024-02-23T00:43:13.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.696+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:13.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.696+0000 7fbb8effd700 10 client.27852.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.696+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb98128ba0 osd=6 4 2024-02-23T00:43:13.699 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.696+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 6 14 2024-02-23T00:43:13.699 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:43:13.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 10 client.27852.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:13.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 15 2024-02-23T00:43:13.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6812/373787000 conn(0x7fbb9812c0b0 0x7fbb9812c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:13.700 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9812e390 osd=7 3 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9812e390 osd=7 3 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9812e390 osd=7 3 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 15 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9812e390 osd=7 3 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 7 15 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9812e390 osd=7 4 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 10 client.27852.objecter _send_command 15 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb9812e180 con 0x7fbb9812c0b0 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbb9de9f700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6812/373787000 conn(0x7fbb9812c0b0 0x7fbb9812c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbb8effd700 10 client.27852.objecter ms_handle_connect 0x7fbb9812c0b0 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.697+0000 7fbb9de9f700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6812/373787000 conn(0x7fbb9812c0b0 0x7fbb9812c530 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.698+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fbb9812e180 con 0x7fbb9812c0b0 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.698+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:43:13.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.698+0000 7fbb8effd700 10 client.27852.objecter _finish_command 15 = system:0 2024-02-23T00:43:13.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.698+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9812e390 osd=7 4 2024-02-23T00:43:13.702 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.698+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 7 15 2024-02-23T00:43:13.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 10 client.27852.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:13.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign -1 16 2024-02-23T00:43:13.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9812e390 osd=7 3 2024-02-23T00:43:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9812e390 osd=7 3 2024-02-23T00:43:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 20 client.27852.objecter _get_session s=0x7fbb9812e390 osd=7 3 2024-02-23T00:43:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 15 client.27852.objecter _session_command_op_remove -1 16 2024-02-23T00:43:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 20 client.27852.objecter get_session s=0x7fbb9812e390 osd=7 3 2024-02-23T00:43:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 15 client.27852.objecter _session_command_op_assign 7 16 2024-02-23T00:43:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9812e390 osd=7 4 2024-02-23T00:43:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 10 client.27852.objecter _send_command 16 2024-02-23T00:43:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.712+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbb9812e180 con 0x7fbb9812c0b0 2024-02-23T00:43:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.714+0000 7fbb8effd700 1 -- 172.21.15.87:0/245147479 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7fbb9812e180 con 0x7fbb9812c0b0 2024-02-23T00:43:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.714+0000 7fbb8effd700 10 client.27852.objecter ms_dispatch 0x7fbb98001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.714+0000 7fbb8effd700 10 client.27852.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.714+0000 7fbb8effd700 20 client.27852.objecter put_session s=0x7fbb9812e390 osd=7 4 2024-02-23T00:43:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.714+0000 7fbb8effd700 15 client.27852.objecter _session_command_op_remove 7 16 2024-02-23T00:43:13.717 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:43:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.714+0000 7fbba0103700 10 librados: watch_flush enter 2024-02-23T00:43:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.714+0000 7fbba0103700 10 librados: watch_flush exit 2024-02-23T00:43:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 10 client.27852.objecter close_session for osd.0 2024-02-23T00:43:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6801/484081831 conn(0x7fbb980fe4f0 msgr2=0x7fbb980fe8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6801/484081831 conn(0x7fbb980fe4f0 0x7fbb980fe8d0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9810ace0 osd=0 1 2024-02-23T00:43:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 10 client.27852.objecter close_session for osd.1 2024-02-23T00:43:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6805/685354171 conn(0x7fbb980598f0 msgr2=0x7fbb98059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6805/685354171 conn(0x7fbb980598f0 0x7fbb98059cd0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98028710 osd=1 1 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 10 client.27852.objecter close_session for osd.2 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6809/3605394228 conn(0x7fbb98028270 msgr2=0x7fbb9811c1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6809/3605394228 conn(0x7fbb98028270 0x7fbb9811c1c0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98059770 osd=2 1 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 10 client.27852.objecter close_session for osd.3 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6813/128059790 conn(0x7fbb9811f920 msgr2=0x7fbb98121d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbb9e6a0700 1 -- 172.21.15.87:0/245147479 reap_dead start 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6813/128059790 conn(0x7fbb9811f920 0x7fbb98121d10 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811d0a0 osd=3 1 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 10 client.27852.objecter close_session for osd.4 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6800/4175844757 conn(0x7fbb9811ccb0 msgr2=0x7fbb98123380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6800/4175844757 conn(0x7fbb9811ccb0 0x7fbb98123380 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.719 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811cb10 osd=4 1 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 10 client.27852.objecter close_session for osd.5 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6804/2550013795 conn(0x7fbb98123d80 msgr2=0x7fbb981241a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6804/2550013795 conn(0x7fbb98123d80 0x7fbb981241a0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9811bf10 osd=5 1 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 10 client.27852.objecter close_session for osd.6 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6808/2509408498 conn(0x7fbb98129220 msgr2=0x7fbb9812b610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6808/2509408498 conn(0x7fbb98129220 0x7fbb9812b610 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb98128ba0 osd=6 1 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 10 client.27852.objecter close_session for osd.7 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6812/373787000 conn(0x7fbb9812c0b0 msgr2=0x7fbb9812c530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.171:6812/373787000 conn(0x7fbb9812c0b0 0x7fbb9812c530 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 20 client.27852.objecter put_session s=0x7fbb9812e390 osd=7 1 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 20 client.27852.objecter shutdown clearing up homeless session... 2024-02-23T00:43:13.720 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 10 client.27852.objecter successfully canceled tick 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbb84062c90 msgr2=0x7fbb84065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> v2:172.21.15.87:6800/1231597463 conn(0x7fbb84062c90 0x7fbb84065150 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 rx=0x7fbb94004930 tx=0x7fbb9400b040).stop 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb980f7280 msgr2=0x7fbb98110550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 --2- 172.21.15.87:0/245147479 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fbb980f7280 0x7fbb98110550 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 rx=0x7fbb8800c810 tx=0x7fbb88007e30).stop 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbb9e6a0700 1 -- 172.21.15.87:0/245147479 reap_dead start 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 shutdown_connections 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 >> 172.21.15.87:0/245147479 conn(0x7fbb980f26f0 msgr2=0x7fbb980f4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 shutdown_connections 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 -- 172.21.15.87:0/245147479 wait complete. 2024-02-23T00:43:13.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:13.715+0000 7fbba0103700 1 librados: shutdown 2024-02-23T00:43:13.722 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.722 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.722 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.722 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.722 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.722 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.722 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.722 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:43:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:13.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:13.723 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:13.824 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:13.851 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:14.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:43:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:43:14.557 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:13.556468+0000 mon.a (mon.0) 3789 : audit [DBG] from='client.? 172.21.15.87:0/245147479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:14.557 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:13.795236+0000 osd.7 (osd.7) 352 : cluster [DBG] 22.a scrub starts 2024-02-23T00:43:14.557 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:13.796384+0000 osd.7 (osd.7) 353 : cluster [DBG] 22.a scrub ok 2024-02-23T00:43:14.557 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:14.040266+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:14.557 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:14.040503+0000 osd.5 (osd.5) 302 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:14.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:13.556468+0000 mon.a (mon.0) 3789 : audit [DBG] from='client.? 172.21.15.87:0/245147479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:14.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:13.795236+0000 osd.7 (osd.7) 352 : cluster [DBG] 22.a scrub starts 2024-02-23T00:43:14.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:13.796384+0000 osd.7 (osd.7) 353 : cluster [DBG] 22.a scrub ok 2024-02-23T00:43:14.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:14.040266+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:14.558 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:14.040503+0000 osd.5 (osd.5) 302 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:14.600 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:14.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:14.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:13.556468+0000 mon.a (mon.0) 3789 : audit [DBG] from='client.? 172.21.15.87:0/245147479' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:13.795236+0000 osd.7 (osd.7) 352 : cluster [DBG] 22.a scrub starts 2024-02-23T00:43:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:13.796384+0000 osd.7 (osd.7) 353 : cluster [DBG] 22.a scrub ok 2024-02-23T00:43:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:14.040266+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:14.040503+0000 osd.5 (osd.5) 302 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:14.703 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:14.732 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:14.976 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:14.980 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:14.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:15.084 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:15.111 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:15.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:13.629326+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:43:15.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:13.630671+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:43:15.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:13.795729+0000 mgr.y (mgr.24437) 648 : cluster [DBG] pgmap v1166: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:15.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:14.284003+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-02-23T00:43:15.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:13.629326+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:43:15.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:13.630671+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:43:15.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:13.795729+0000 mgr.y (mgr.24437) 648 : cluster [DBG] pgmap v1166: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:15.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:14.284003+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-02-23T00:43:15.657 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:43:15.657 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:43:15.657 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:43:15.657 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:43:15.657 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.657 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.26224.0:2298 169.17 169:ed571587:test-rados-api-smithi087-82091-30::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e256)", 2024-02-23T00:43:15.657 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:43:15.657 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 587.81404380799995, 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869, 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.26224", 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/2563268269", 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2298 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.658 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025407+0000", 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025406+0000", 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1740000000000006e-06 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025415+0000", 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1041e-05 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.659 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025426+0000", 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0450000000000003e-06 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025433+0000", 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7723999999999998e-05 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025491+0000", 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3479999999999993e-06 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.660 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.025500+0000", 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.21538338000000001 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.240883+0000", 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00021381300000000001 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.241097+0000", 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00573626 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246834+0000", 2024-02-23T00:43:15.661 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6838999999999999e-05 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.246930+0000", 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013423800000000001 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 0,2", 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247065+0000", 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037269000000000001 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247437+0000", 2024-02-23T00:43:15.662 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026355299999999998 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.247701+0000", 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0011556330000000001 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248856+0000", 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.7636999999999998e-05 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248904+0000", 2024-02-23T00:43:15.663 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0179000000000003e-05 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:33:27.248944+0000", 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.223536869 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 322/321 [PushReplyOp(4:e9187ab8:reshard::reshard.0000000000:head)])", 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:43:15.664 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 521.51608175199999, 2024-02-23T00:43:15.665 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089931579999999997, 2024-02-23T00:43:15.667 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.667 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.667 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.667 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.667 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.667 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323369+0000", 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9330000000000004e-06 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.668 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323376+0000", 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1073000000000001e-05 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323387+0000", 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.489e-06 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.323393+0000", 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089729134000000002 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.413122+0000", 2024-02-23T00:43:15.669 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00017875200000000001 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.413301+0000", 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089931579999999997 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 322/321 [PushReplyOp(4:ec529b31:::obj_delete_at_hint.0000000050:head)])", 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 521.08182702099998, 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089810938000000007, 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.670 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757624+0000", 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757623+0000", 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9920000000000007e-06 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.671 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757631+0000", 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1764e-05 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757643+0000", 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4240000000000004e-06 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.757650+0000", 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089629418000000002 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.672 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.847280+0000", 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000155267 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:33.847435+0000", 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089810938000000007 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 323/321 [PushReplyOp(4:ece66ccb:::obj_delete_at_hint.0000000102:head)])", 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:43:15.673 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 520.73370279300002, 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089865278000000007, 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.674 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105748+0000", 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.0180000000000001e-06 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105753+0000", 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3409999999999997e-06 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105760+0000", 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2069999999999997e-06 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.675 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.105764+0000", 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089713752999999993 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.195478+0000", 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013554999999999999 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.195614+0000", 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089865278000000007 2024-02-23T00:43:15.676 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.17 323/321 [PushReplyOp(4:ee62d5d2:lc::lc.1:head)])", 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 520.55745591499999, 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.090010914999999997, 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:43:15.677 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281995+0000", 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7539999999999998e-06 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.281998+0000", 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8160000000000004e-06 2024-02-23T00:43:15.678 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.282002+0000", 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.794e-06 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.282005+0000", 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.089811657000000003 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.371817+0000", 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000189092 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.679 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:34:34.372006+0000", 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.090010914999999997 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_update_log_missing(205.3s1 epoch 376/375 rep_tid 700 entries 376'1 (0'0) error 205:cbb9903e:::test_obj:head by client.37589.0:1 0.000000 -95 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 467.74117551299997, 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065042928, 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.680 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098276+0000", 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098275+0000", 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2939999999999994e-06 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.681 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098285+0000", 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2446000000000001e-05 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098297+0000", 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2310000000000002e-06 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098306+0000", 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012984100000000001 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.682 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.098436+0000", 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.064882361999999999 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:35:27.163319+0000", 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.065042928 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.683 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.4.31:55 216.9 e451/444 216:939aa6ec:test-rados-api-smithi087-82091-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head v 451'3, mlcod=446'1)", 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:37:17.130726+0000", 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 357.70872564699999, 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056389038000000002, 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130726+0000", 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.684 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130726+0000", 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130725+0000", 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.5980000000000003e-06 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130734+0000", 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0025e-05 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.685 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130744+0000", 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6889999999999998e-06 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130751+0000", 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5022999999999999e-05 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130796+0000", 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.4711000000000001e-05 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:15.686 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130821+0000", 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056043386000000001 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.186864+0000", 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025056500000000001 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187115+0000", 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056389038000000002 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.687 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.4.31:56 216.17 e451/444 216:ec5c3d73:test-rados-api-smithi087-82091-70::b1a341c59dc4c60bbe00244ac8bba87528936bc7:head v 451'2, mlcod=0'0)", 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:37:17.143192+0000", 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 357.69625897499998, 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044005469999999998, 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143192+0000", 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.688 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143192+0000", 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143190+0000", 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5917999999999999e-05 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143206+0000", 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5682999999999999e-05 2024-02-23T00:43:15.689 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143222+0000", 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2430000000000005e-06 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143230+0000", 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.672e-05 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143277+0000", 2024-02-23T00:43:15.690 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2272e-05 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.143299+0000", 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.043740635999999999 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187040+0000", 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00015778499999999999 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187198+0000", 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.044005469999999998 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.691 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.7.51:63 63.10 63:0bcb77f5:test-rados-api-smithi087-82091-80::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e494)", 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:38:31.219712+0000", 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 283.61973931300003, 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030054408000000001, 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.692 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219712+0000", 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219712+0000", 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219710+0000", 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1394e-05 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.693 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219721+0000", 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.4367e-05 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219736+0000", 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9110000000000004e-06 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219743+0000", 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1088999999999999e-05 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.694 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219784+0000", 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3820000000000003e-06 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.219789+0000", 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020061689999999999 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.221795+0000", 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012612300000000001 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.695 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.696 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:15.696 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.221921+0000", 2024-02-23T00:43:15.696 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000363552 2024-02-23T00:43:15.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.696 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.696 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 7", 2024-02-23T00:43:15.696 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.222285+0000", 2024-02-23T00:43:15.696 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00093142099999999999 2024-02-23T00:43:15.696 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.223216+0000", 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.026412662999999999 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.249629+0000", 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.9842999999999997e-05 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.249679+0000", 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.7428000000000004e-05 2024-02-23T00:43:15.697 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:31.249766+0000", 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.030054408000000001 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.7 679/678 [PushReplyOp(4:e07a6548:gc::gc.15:head)])", 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.392959+0000", 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 96.446491933000004, 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037954400000000002, 2024-02-23T00:43:15.698 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.392959+0000", 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.392959+0000", 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.699 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.392959+0000", 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7359999999999999e-06 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.392963+0000", 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.993e-06 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.700 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.392969+0000", 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3480000000000003e-06 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.392974+0000", 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0035071159999999998 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.396481+0000", 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000273505 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.701 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.396755+0000", 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037954400000000002 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.7 679/678 [PushReplyOp(4:e622f76a:gc::gc.20:head)])", 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.850683+0000", 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 95.988767831999994, 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039013920000000001, 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.702 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.850683+0000", 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.850683+0000", 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.850683+0000", 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8430000000000003e-06 2024-02-23T00:43:15.703 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.850687+0000", 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9449999999999998e-06 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.850693+0000", 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.7960000000000001e-06 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.850699+0000", 2024-02-23T00:43:15.704 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0036126520000000001 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.854311+0000", 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00027337599999999998 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.854585+0000", 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039013920000000001 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.705 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.6 679/678 [PushReplyOp(4:624175ac:::obj_delete_at_hint.0000000091:head)])", 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.031983+0000", 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 95.807468717000006, 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038970519999999998, 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.031983+0000", 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.706 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.031983+0000", 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.031982+0000", 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.2210000000000002e-06 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.031985+0000", 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.2289999999999997e-06 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.707 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.031993+0000", 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3689999999999998e-06 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.031998+0000", 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037258170000000002 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.035724+0000", 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000155647 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.708 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.035880+0000", 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038970519999999998 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.6 679/678 [PushReplyOp(4:63d917ed:::obj_delete_at_hint.0000000065:head)])", 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.122361+0000", 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 95.717090357000004, 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040540970000000004, 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.709 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.122361+0000", 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.122361+0000", 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.122361+0000", 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.1179999999999992e-06 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.710 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.122369+0000", 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0779999999999998e-06 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.122376+0000", 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9089999999999997e-06 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.122382+0000", 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037520729999999999 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.711 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.126134+0000", 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000281272 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.126415+0000", 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040540970000000004 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.712 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.6 680/678 [PushReplyOp(4:64735f45:::obj_delete_at_hint.0000000083:head)])", 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.217890+0000", 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 95.621561087999993, 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.084857887000000007, 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217890+0000", 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217890+0000", 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217890+0000", 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.788e-06 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217894+0000", 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4200000000000004e-06 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217900+0000", 2024-02-23T00:43:15.714 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8459999999999996e-06 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.217906+0000", 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.084570939999999997 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.302477+0000", 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00027114100000000003 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.302748+0000", 2024-02-23T00:43:15.715 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.084857887000000007 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.b 680/678 [PushReplyOp(4:d5ecafc5:::obj_delete_at_hint.0000000046:head)])", 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.301813+0000", 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 95.537638584000007, 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039730879999999996, 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.716 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.301813+0000", 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.301813+0000", 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.301812+0000", 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6059999999999996e-06 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.717 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.301817+0000", 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9630000000000001e-06 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.301824+0000", 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6819999999999997e-06 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.301830+0000", 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0036788939999999998 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.718 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.305509+0000", 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00027634900000000001 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.305786+0000", 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039730879999999996 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.6 680/678 [PushReplyOp(4:64760185:::obj_delete_at_hint.0000000010:head)])", 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.389165+0000", 2024-02-23T00:43:15.719 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 95.450286423999998, 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038471669999999999, 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.389165+0000", 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.389165+0000", 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.720 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.389164+0000", 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.6290000000000001e-06 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.389169+0000", 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0809999999999996e-06 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.389176+0000", 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9190000000000004e-06 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.721 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.389182+0000", 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0036858540000000001 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.392868+0000", 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00014402600000000001 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.393012+0000", 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038471669999999999 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.722 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.6 680/678 [PushReplyOp(4:66924383:reshard::reshard.0000000005:head)])", 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.479468+0000", 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 95.359983298000003, 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003987361, 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.479468+0000", 2024-02-23T00:43:15.723 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.479468+0000", 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.479467+0000", 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.057e-06 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.479473+0000", 2024-02-23T00:43:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9310000000000002e-06 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.479479+0000", 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5980000000000002e-06 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.479486+0000", 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038029959999999999 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.725 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.483289+0000", 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016620200000000001 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.483455+0000", 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003987361 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.1e 706/678 [PushReplyOp(4:7e82ab18:lc::lc.13:head)])", 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:08.956087+0000", 2024-02-23T00:43:15.726 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 65.883364041999997, 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.004022126, 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:08.956087+0000", 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:08.956087+0000", 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.727 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:08.956087+0000", 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4000000000000001e-06 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:08.956090+0000", 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2920000000000001e-06 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:08.956097+0000", 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.75e-06 2024-02-23T00:43:15.728 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:08.956102+0000", 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037224720000000001 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:08.959825+0000", 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00028441000000000002 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:08.960109+0000", 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.004022126 2024-02-23T00:43:15.729 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(234.4s1 726/597 ECSubWrite(tid=743, reqid=osd.2.18:62, at_version=726'21, trim_to=0'0, roll_forward_to=718'20))", 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.267280+0000", 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 46.572170956999997, 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0052824719999999999, 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.267280+0000", 2024-02-23T00:43:15.730 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.267280+0000", 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.267279+0000", 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001114372 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.268394+0000", 2024-02-23T00:43:15.731 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.7787999999999999e-05 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.268422+0000", 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2537999999999999e-05 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.268434+0000", 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3161000000000001e-05 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.732 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.268497+0000", 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2337999999999999e-05 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.268510+0000", 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.004029637 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.272539+0000", 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3411999999999999e-05 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.733 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.272563+0000", 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0052824719999999999 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27567.0:2014 249.6 e746/738 249:602f83fe:::foo:head v 746'11, mlcod=745'9)", 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:54.419496+0000", 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 20.419955381000001, 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020896679999999998, 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.734 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.419496+0000", 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.419496+0000", 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999952 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.419491+0000", 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2796999999999999e-05 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:15.735 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.419514+0000", 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2768e-05 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.419537+0000", 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8920000000000006e-06 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.419545+0000", 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00090719800000000001 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:15.736 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.420453+0000", 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5052e-05 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.420478+0000", 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0010728039999999999 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.421550+0000", 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5045000000000001e-05 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:54.421586+0000", 2024-02-23T00:43:15.737 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020896679999999998 2024-02-23T00:43:15.738 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.738 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.738 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.738 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:15.738 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:15.738 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:43:15.738 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:13.629326+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:43:15.738 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:13.630671+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:43:15.738 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:13.795729+0000 mgr.y (mgr.24437) 648 : cluster [DBG] pgmap v1166: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:15.738 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:14.284003+0000 mon.a (mon.0) 3790 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-02-23T00:43:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:14.751178+0000 osd.6 (osd.6) 318 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:43:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:14.754198+0000 osd.6 (osd.6) 319 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:43:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:15.288616+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-02-23T00:43:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:15.295245+0000 mon.a (mon.0) 3792 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:14.751178+0000 osd.6 (osd.6) 318 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:43:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:14.754198+0000 osd.6 (osd.6) 319 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:43:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:15.288616+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-02-23T00:43:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:15.295245+0000 mon.a (mon.0) 3792 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:14.751178+0000 osd.6 (osd.6) 318 : cluster [DBG] 5.11 scrub starts 2024-02-23T00:43:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:14.754198+0000 osd.6 (osd.6) 319 : cluster [DBG] 5.11 scrub ok 2024-02-23T00:43:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:15.288616+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-02-23T00:43:16.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:15.295245+0000 mon.a (mon.0) 3792 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:17.042 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-23T00:43:17.061 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:17.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:17.167 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:17.194 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:17.546 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:43:17.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:15.725110+0000 osd.6 (osd.6) 320 : cluster [DBG] 234.0 scrub starts 2024-02-23T00:43:17.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:15.728616+0000 osd.6 (osd.6) 321 : cluster [DBG] 234.0 scrub ok 2024-02-23T00:43:17.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:15.796593+0000 mgr.y (mgr.24437) 649 : cluster [DBG] pgmap v1169: 204 pgs: 1 active+remapped+backfill_toofull, 31 creating+peering, 1 unknown, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:17.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:16.294625+0000 mon.a (mon.0) 3793 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:17.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:16.294776+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-02-23T00:43:17.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:16.307375+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:17.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:16.697784+0000 osd.6 (osd.6) 322 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:43:17.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:16.698721+0000 osd.6 (osd.6) 323 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:43:17.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:16.741954+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:16.745216+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:16.989802+0000 osd.4 (osd.4) 339 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:16.990727+0000 osd.4 (osd.4) 340 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:17.076685+0000 osd.5 (osd.5) 303 : cluster [DBG] 3.16 deep-scrub starts 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:17.077577+0000 osd.5 (osd.5) 304 : cluster [DBG] 3.16 deep-scrub ok 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:15.725110+0000 osd.6 (osd.6) 320 : cluster [DBG] 234.0 scrub starts 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:15.728616+0000 osd.6 (osd.6) 321 : cluster [DBG] 234.0 scrub ok 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:15.796593+0000 mgr.y (mgr.24437) 649 : cluster [DBG] pgmap v1169: 204 pgs: 1 active+remapped+backfill_toofull, 31 creating+peering, 1 unknown, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:16.294625+0000 mon.a (mon.0) 3793 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:16.294776+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:16.307375+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:16.697784+0000 osd.6 (osd.6) 322 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:43:17.666 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:16.698721+0000 osd.6 (osd.6) 323 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:43:17.667 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:16.741954+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:43:17.667 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:16.745216+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:43:17.667 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:16.989802+0000 osd.4 (osd.4) 339 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:43:17.667 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:16.990727+0000 osd.4 (osd.4) 340 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:43:17.667 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:17.076685+0000 osd.5 (osd.5) 303 : cluster [DBG] 3.16 deep-scrub starts 2024-02-23T00:43:17.667 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:17.077577+0000 osd.5 (osd.5) 304 : cluster [DBG] 3.16 deep-scrub ok 2024-02-23T00:43:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:15.725110+0000 osd.6 (osd.6) 320 : cluster [DBG] 234.0 scrub starts 2024-02-23T00:43:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:15.728616+0000 osd.6 (osd.6) 321 : cluster [DBG] 234.0 scrub ok 2024-02-23T00:43:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:15.796593+0000 mgr.y (mgr.24437) 649 : cluster [DBG] pgmap v1169: 204 pgs: 1 active+remapped+backfill_toofull, 31 creating+peering, 1 unknown, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:16.294625+0000 mon.a (mon.0) 3793 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:16.294776+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-02-23T00:43:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:16.307375+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:16.697784+0000 osd.6 (osd.6) 322 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:43:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:16.698721+0000 osd.6 (osd.6) 323 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:43:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:16.741954+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:43:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:16.745216+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:43:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:16.989802+0000 osd.4 (osd.4) 339 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:43:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:16.990727+0000 osd.4 (osd.4) 340 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:43:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:17.076685+0000 osd.5 (osd.5) 303 : cluster [DBG] 3.16 deep-scrub starts 2024-02-23T00:43:17.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:17.077577+0000 osd.5 (osd.5) 304 : cluster [DBG] 3.16 deep-scrub ok 2024-02-23T00:43:17.844 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:17.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:17.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:17.948 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:17.975 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:17.385940+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:43:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:17.386827+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:43:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:17.434081+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:17.434128+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-02-23T00:43:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:17.434507+0000 mon.a (mon.0) 3798 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-134"}]: dispatch 2024-02-23T00:43:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:17.665328+0000 osd.6 (osd.6) 324 : cluster [DBG] 234.2 scrub starts 2024-02-23T00:43:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:17.669023+0000 osd.6 (osd.6) 325 : cluster [DBG] 234.2 scrub ok 2024-02-23T00:43:18.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:17.796993+0000 mgr.y (mgr.24437) 650 : cluster [DBG] pgmap v1172: 204 pgs: 1 active+remapped+backfill_toofull, 31 creating+peering, 1 unknown, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:18.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:17.385940+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:43:18.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:17.386827+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:43:18.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:17.434081+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:18.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:17.434128+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-02-23T00:43:18.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:17.434507+0000 mon.a (mon.0) 3798 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-134"}]: dispatch 2024-02-23T00:43:18.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:17.665328+0000 osd.6 (osd.6) 324 : cluster [DBG] 234.2 scrub starts 2024-02-23T00:43:18.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:17.669023+0000 osd.6 (osd.6) 325 : cluster [DBG] 234.2 scrub ok 2024-02-23T00:43:18.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:17.796993+0000 mgr.y (mgr.24437) 650 : cluster [DBG] pgmap v1172: 204 pgs: 1 active+remapped+backfill_toofull, 31 creating+peering, 1 unknown, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:17.385940+0000 osd.2 (osd.2) 277 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:43:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:17.386827+0000 osd.2 (osd.2) 278 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:43:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:17.434081+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:17.434128+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-02-23T00:43:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:17.434507+0000 mon.a (mon.0) 3798 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-134"}]: dispatch 2024-02-23T00:43:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:17.665328+0000 osd.6 (osd.6) 324 : cluster [DBG] 234.2 scrub starts 2024-02-23T00:43:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:17.669023+0000 osd.6 (osd.6) 325 : cluster [DBG] 234.2 scrub ok 2024-02-23T00:43:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:17.796993+0000 mgr.y (mgr.24437) 650 : cluster [DBG] pgmap v1172: 204 pgs: 1 active+remapped+backfill_toofull, 31 creating+peering, 1 unknown, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:19.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.695+0000 7fb3f06ec700 1 -- 172.21.15.87:0/2880466256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f8cb0 msgr2=0x7fb3e80f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:19.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.695+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/2880466256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f8cb0 0x7fb3e80f9090 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 rx=0x7fb3e4004870 tx=0x7fb3e4009cb0).stop 2024-02-23T00:43:19.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.695+0000 7fb3f06ec700 1 -- 172.21.15.87:0/2880466256 shutdown_connections 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.695+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/2880466256 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3e8102210 0x7fb3e8104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.695+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/2880466256 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f8cb0 0x7fb3e80f9090 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.695+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/2880466256 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3e80f9610 0x7fb3e8101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.695+0000 7fb3f06ec700 1 -- 172.21.15.87:0/2880466256 >> 172.21.15.87:0/2880466256 conn(0x7fb3e80f2820 msgr2=0x7fb3e80f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.695+0000 7fb3f06ec700 1 -- 172.21.15.87:0/2880466256 shutdown_connections 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.695+0000 7fb3f06ec700 1 -- 172.21.15.87:0/2880466256 wait complete. 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 librados: starting msgr at 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 librados: starting objecter 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 Processor -- start 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 -- start start 2024-02-23T00:43:19.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 librados: setting wanted keys 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 librados: calling monclient init 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3e80f8cb0 0x7fb3e810e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f9610 0x7fb3e810e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3e8102210 0x7fb3e81083d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e81088c0 con 0x7fb3e80f9610 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e8108a30 con 0x7fb3e80f8cb0 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3f06ec700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e8108bf0 con 0x7fb3e8102210 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3edc87700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f9610 0x7fb3e810e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3eec89700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3e8102210 0x7fb3e81083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3edc87700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f9610 0x7fb3e810e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35316/0 (socket says 172.21.15.87:35316) 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3eec89700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3e8102210 0x7fb3e81083d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33694/0 (socket says 172.21.15.87:33694) 2024-02-23T00:43:19.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.696+0000 7fb3edc87700 1 -- 172.21.15.87:0/871784530 learned_addr learned my addr 172.21.15.87:0/871784530 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3ee488700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3e80f8cb0 0x7fb3e810e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3edc87700 1 -- 172.21.15.87:0/871784530 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3e8102210 msgr2=0x7fb3e81083d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3edc87700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3e8102210 0x7fb3e81083d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3edc87700 1 -- 172.21.15.87:0/871784530 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3e80f8cb0 msgr2=0x7fb3e810e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3edc87700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3e80f8cb0 0x7fb3e810e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3edc87700 1 -- 172.21.15.87:0/871784530 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8005040 con 0x7fb3e80f9610 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3eec89700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3e8102210 0x7fb3e81083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3ee488700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3e80f8cb0 0x7fb3e810e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3edc87700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f9610 0x7fb3e810e720 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7fb3d80048f0 tx=0x7fb3d8009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3df7fe700 10 client.?.objecter ms_handle_connect 0x7fb3e80f9610 2024-02-23T00:43:19.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3df7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3df7fe700 1 -- 172.21.15.87:0/871784530 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8007b00 con 0x7fb3e80f9610 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e4005040 con 0x7fb3e80f9610 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3df7fe700 1 -- 172.21.15.87:0/871784530 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3d800f040 con 0x7fb3e80f9610 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3df7fe700 1 -- 172.21.15.87:0/871784530 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8014a00 con 0x7fb3e80f9610 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3f06ec700 10 client.27864.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e8115060 con 0x7fb3e80f9610 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.697+0000 7fb3f06ec700 1 librados: init done 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.698+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e805fa60 con 0x7fb3e80f9610 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.698+0000 7fb3df7fe700 1 -- 172.21.15.87:0/871784530 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb3d800e580 con 0x7fb3e80f9610 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.699+0000 7fb3df7fe700 1 --2- 172.21.15.87:0/871784530 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3d4062c90 0x7fb3d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.699+0000 7fb3ee488700 1 --2- 172.21.15.87:0/871784530 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3d4062c90 0x7fb3d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:19.708 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.699+0000 7fb3df7fe700 1 -- 172.21.15.87:0/871784530 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(768..768 src has 400..768) v4 ==== 8538+0+0 (secure 0 0 0) 0x7fb3d807cfa0 con 0x7fb3e80f9610 2024-02-23T00:43:19.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.699+0000 7fb3df7fe700 10 client.27864.objecter ms_dispatch 0x7fb3e8001550 osd_map(768..768 src has 400..768) v4 2024-02-23T00:43:19.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.699+0000 7fb3df7fe700 3 client.27864.objecter handle_osd_map got epochs [768,768] > 0 2024-02-23T00:43:19.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.699+0000 7fb3df7fe700 3 client.27864.objecter handle_osd_map decoding full epoch 768 2024-02-23T00:43:19.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.699+0000 7fb3df7fe700 20 client.27864.objecter dump_active .. 0 homeless 2024-02-23T00:43:19.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.699+0000 7fb3df7fe700 10 client.27864.objecter ms_handle_connect 0x7fb3d4062c90 2024-02-23T00:43:19.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.699+0000 7fb3ee488700 1 --2- 172.21.15.87:0/871784530 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3d4062c90 0x7fb3d4065150 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7fb3e4004ee0 tx=0x7fb3e400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:19.709 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.703+0000 7fb3df7fe700 1 -- 172.21.15.87:0/871784530 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb3d8049f60 con 0x7fb3e80f9610 2024-02-23T00:43:19.725 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:19.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:18.436755+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-134"}]': finished 2024-02-23T00:43:19.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:18.436823+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-02-23T00:43:19.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:18.437979+0000 mon.a (mon.0) 3801 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-134", "mode": "readproxy"}]: dispatch 2024-02-23T00:43:19.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:18.438363+0000 osd.2 (osd.2) 279 : cluster [DBG] 2.e scrub starts 2024-02-23T00:43:19.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:18.439343+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.e scrub ok 2024-02-23T00:43:19.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:18.459309+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:43:19.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:18.462468+0000 osd.1 (osd.1) 353 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:43:19.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:18.436755+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-134"}]': finished 2024-02-23T00:43:19.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:18.436823+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-02-23T00:43:19.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:18.437979+0000 mon.a (mon.0) 3801 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-134", "mode": "readproxy"}]: dispatch 2024-02-23T00:43:19.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:18.438363+0000 osd.2 (osd.2) 279 : cluster [DBG] 2.e scrub starts 2024-02-23T00:43:19.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:18.439343+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.e scrub ok 2024-02-23T00:43:19.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:18.459309+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:43:19.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:18.462468+0000 osd.1 (osd.1) 353 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:43:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:19.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:19.830 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:19.860 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:18.436755+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-134"}]': finished 2024-02-23T00:43:19.860 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:18.436823+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-02-23T00:43:19.860 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:18.437979+0000 mon.a (mon.0) 3801 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-134", "mode": "readproxy"}]: dispatch 2024-02-23T00:43:19.860 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:18.438363+0000 osd.2 (osd.2) 279 : cluster [DBG] 2.e scrub starts 2024-02-23T00:43:19.861 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:18.439343+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.e scrub ok 2024-02-23T00:43:19.861 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:18.459309+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:43:19.861 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:18.462468+0000 osd.1 (osd.1) 353 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:43:19.862 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:19.883 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:19.885 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.883+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fb3e8107290 con 0x7fb3e80f9610 2024-02-23T00:43:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:19.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:19.987 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:19.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.546+0000 7f550e27b700 1 -- 172.21.15.87:0/3576021167 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f8cb0 msgr2=0x7f55080f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:19.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.546+0000 7f550e27b700 1 --2- 172.21.15.87:0/3576021167 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f8cb0 0x7f55080f9090 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 rx=0x7f54f8004870 tx=0x7f54f8009cb0).stop 2024-02-23T00:43:19.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.546+0000 7f550e27b700 1 -- 172.21.15.87:0/3576021167 shutdown_connections 2024-02-23T00:43:19.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.546+0000 7f550e27b700 1 --2- 172.21.15.87:0/3576021167 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f5508102210 0x7f5508104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.546+0000 7f550e27b700 1 --2- 172.21.15.87:0/3576021167 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f8cb0 0x7f55080f9090 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.546+0000 7f550e27b700 1 --2- 172.21.15.87:0/3576021167 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f55080f9610 0x7f5508101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.546+0000 7f550e27b700 1 -- 172.21.15.87:0/3576021167 >> 172.21.15.87:0/3576021167 conn(0x7f55080f2820 msgr2=0x7f55080f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:19.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.546+0000 7f550e27b700 1 -- 172.21.15.87:0/3576021167 shutdown_connections 2024-02-23T00:43:19.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.546+0000 7f550e27b700 1 -- 172.21.15.87:0/3576021167 wait complete. 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 librados: starting msgr at 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 librados: starting objecter 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 Processor -- start 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 -- start start 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 librados: setting wanted keys 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 librados: calling monclient init 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f55080f8cb0 0x7f550810e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f9610 0x7f550810e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5508102210 0x7f55081083a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5508108890 con 0x7f5508102210 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5508108a00 con 0x7f55080f8cb0 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550e27b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5508108bc0 con 0x7f55080f9610 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550c818700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5508102210 0x7f55081083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f55077fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f9610 0x7f550810e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:19.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550c818700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5508102210 0x7f55081083a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:35302/0 (socket says 172.21.15.87:35302) 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f55077fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f9610 0x7f550810e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33682/0 (socket says 172.21.15.87:33682) 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f550c818700 1 -- 172.21.15.87:0/1574564853 learned_addr learned my addr 172.21.15.87:0/1574564853 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.547+0000 7f5507fff700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f55080f8cb0 0x7f550810e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550c818700 1 -- 172.21.15.87:0/1574564853 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f9610 msgr2=0x7f550810e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550c818700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f9610 0x7f550810e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550c818700 1 -- 172.21.15.87:0/1574564853 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f55080f8cb0 msgr2=0x7f550810e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550c818700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f55080f8cb0 0x7f550810e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550c818700 1 -- 172.21.15.87:0/1574564853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc005040 con 0x7f5508102210 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f55077fe700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f9610 0x7f550810e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f5507fff700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f55080f8cb0 0x7f550810e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550c818700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5508102210 0x7f55081083a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 rx=0x7f550000d940 tx=0x7f5500004e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:19.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f5505775700 10 client.?.objecter ms_handle_connect 0x7f5508102210 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f5505775700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f5505775700 1 -- 172.21.15.87:0/1574564853 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500004020 con 0x7f5508102210 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f8005040 con 0x7f5508102210 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f5505775700 1 -- 172.21.15.87:0/1574564853 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f550000f040 con 0x7f5508102210 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f5505775700 1 -- 172.21.15.87:0/1574564853 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55000036c0 con 0x7f5508102210 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550e27b700 10 client.27858.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55081091b0 con 0x7f5508102210 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.548+0000 7f550e27b700 1 librados: init done 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.549+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550805fa60 con 0x7f5508102210 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.549+0000 7f5505775700 1 -- 172.21.15.87:0/1574564853 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f5500004180 con 0x7f5508102210 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.550+0000 7f5505775700 1 --2- 172.21.15.87:0/1574564853 >> v2:172.21.15.87:6800/1231597463 conn(0x7f54f0062c90 0x7f54f0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.550+0000 7f5505775700 1 -- 172.21.15.87:0/1574564853 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(768..768 src has 400..768) v4 ==== 8538+0+0 (secure 0 0 0) 0x7f550007ceb0 con 0x7f5508102210 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.550+0000 7f5505775700 10 client.27858.objecter ms_dispatch 0x7f5508001550 osd_map(768..768 src has 400..768) v4 2024-02-23T00:43:19.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.550+0000 7f5505775700 3 client.27858.objecter handle_osd_map got epochs [768,768] > 0 2024-02-23T00:43:19.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.550+0000 7f5505775700 3 client.27858.objecter handle_osd_map decoding full epoch 768 2024-02-23T00:43:19.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.550+0000 7f5507fff700 1 --2- 172.21.15.87:0/1574564853 >> v2:172.21.15.87:6800/1231597463 conn(0x7f54f0062c90 0x7f54f0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:19.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.550+0000 7f5505775700 20 client.27858.objecter dump_active .. 0 homeless 2024-02-23T00:43:19.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.550+0000 7f5505775700 10 client.27858.objecter ms_handle_connect 0x7f54f0062c90 2024-02-23T00:43:19.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.550+0000 7f5507fff700 1 --2- 172.21.15.87:0/1574564853 >> v2:172.21.15.87:6800/1231597463 conn(0x7f54f0062c90 0x7f54f0065150 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f54f8004ee0 tx=0x7f54f800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:19.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.554+0000 7f5505775700 1 -- 172.21.15.87:0/1574564853 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f55000501a0 con 0x7f5508102210 2024-02-23T00:43:19.993 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:19.740+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [1, 2, 7, 3, 5, 0]} v 0) v1 -- 0x7f5508109880 con 0x7f5508102210 2024-02-23T00:43:20.016 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:20.022 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:20.042 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:20.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:20.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:20.052 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:20.146 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:20.175 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:20.195 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:20.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:20.298 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:20.330 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:20.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.456+0000 7f5505775700 1 -- 172.21.15.87:0/1574564853 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [1, 2, 7, 3, 5, 0]}]=0 set 5.10 pg_upmap_items mapping to [1->2,7->3,5->0] v769) v1 ==== 159+0+0 (secure 0 0 0) 0x7f550004fd90 con 0x7f5508102210 2024-02-23T00:43:20.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.457+0000 7fb3df7fe700 1 -- 172.21.15.87:0/871784530 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v769) v1 ==== 87+0+0 (secure 0 0 0) 0x7fb3d8050290 con 0x7fb3e80f9610 2024-02-23T00:43:20.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 10 librados: watch_flush enter 2024-02-23T00:43:20.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 10 librados: watch_flush exit 2024-02-23T00:43:20.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 20 client.27858.objecter shutdown clearing up homeless session... 2024-02-23T00:43:20.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 10 client.27858.objecter successfully canceled tick 2024-02-23T00:43:20.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 >> v2:172.21.15.87:6800/1231597463 conn(0x7f54f0062c90 msgr2=0x7f54f0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:20.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 --2- 172.21.15.87:0/1574564853 >> v2:172.21.15.87:6800/1231597463 conn(0x7f54f0062c90 0x7f54f0065150 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 rx=0x7f54f8004ee0 tx=0x7f54f800b040).stop 2024-02-23T00:43:20.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5508102210 msgr2=0x7f55081083a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:20.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5508102210 0x7f55081083a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 rx=0x7f550000d940 tx=0x7f5500004e40).stop 2024-02-23T00:43:20.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 shutdown_connections 2024-02-23T00:43:20.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 --2- 172.21.15.87:0/1574564853 >> v2:172.21.15.87:6800/1231597463 conn(0x7f54f0062c90 0x7f54f0065150 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:20.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f5508102210 0x7f55081083a0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:20.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f55080f8cb0 0x7f550810e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:20.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 --2- 172.21.15.87:0/1574564853 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f55080f9610 0x7f550810e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:20.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 >> 172.21.15.87:0/1574564853 conn(0x7f55080f2820 msgr2=0x7f55080f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:20.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.459+0000 7fb3f06ec700 10 librados: watch_flush enter 2024-02-23T00:43:20.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.460+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 shutdown_connections 2024-02-23T00:43:20.462 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.460+0000 7f550e27b700 1 -- 172.21.15.87:0/1574564853 wait complete. 2024-02-23T00:43:20.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.460+0000 7f550e27b700 1 librados: shutdown 2024-02-23T00:43:20.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.460+0000 7fb3f06ec700 10 librados: watch_flush exit 2024-02-23T00:43:20.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.460+0000 7fb3f06ec700 20 client.27864.objecter shutdown clearing up homeless session... 2024-02-23T00:43:20.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.460+0000 7fb3f06ec700 10 client.27864.objecter successfully canceled tick 2024-02-23T00:43:20.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3d4062c90 msgr2=0x7fb3d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:20.463 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/871784530 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3d4062c90 0x7fb3d4065150 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 rx=0x7fb3e4004ee0 tx=0x7fb3e400b040).stop 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f9610 msgr2=0x7fb3e810e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f9610 0x7fb3e810e720 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 rx=0x7fb3d80048f0 tx=0x7fb3d8009cb0).stop 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 shutdown_connections 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/871784530 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb3d4062c90 0x7fb3d4065150 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb3e8102210 0x7fb3e81083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb3e80f8cb0 0x7fb3e810e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 --2- 172.21.15.87:0/871784530 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb3e80f9610 0x7fb3e810e720 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 >> 172.21.15.87:0/871784530 conn(0x7fb3e80f2820 msgr2=0x7fb3e80f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 shutdown_connections 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 -- 172.21.15.87:0/871784530 wait complete. 2024-02-23T00:43:20.464 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:20.461+0000 7fb3f06ec700 1 librados: shutdown 2024-02-23T00:43:20.465 INFO:teuthology.orchestra.run.smithi087.stderr:set 5.10 pg_upmap_items mapping to [1->2,7->3,5->0] 2024-02-23T00:43:20.465 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:43:20.569 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:19.432467+0000 mon.a (mon.0) 3802 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:43:20.569 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:19.439948+0000 mon.a (mon.0) 3803 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-134", "mode": "readproxy"}]': finished 2024-02-23T00:43:20.569 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:19.440002+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-02-23T00:43:20.569 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:19.742508+0000 mon.a (mon.0) 3805 : audit [INF] from='client.? 172.21.15.87:0/1574564853' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [1, 2, 7, 3, 5, 0]}]: dispatch 2024-02-23T00:43:20.569 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:19.797390+0000 mgr.y (mgr.24437) 651 : cluster [DBG] pgmap v1175: 204 pgs: 1 active+remapped+backfill_toofull, 31 creating+peering, 172 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:20.569 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:19.884855+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? 172.21.15.87:0/871784530' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:43:20.570 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:19.432467+0000 mon.a (mon.0) 3802 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:43:20.570 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:19.439948+0000 mon.a (mon.0) 3803 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-134", "mode": "readproxy"}]': finished 2024-02-23T00:43:20.570 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:19.440002+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-02-23T00:43:20.570 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:19.742508+0000 mon.a (mon.0) 3805 : audit [INF] from='client.? 172.21.15.87:0/1574564853' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [1, 2, 7, 3, 5, 0]}]: dispatch 2024-02-23T00:43:20.570 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:19.797390+0000 mgr.y (mgr.24437) 651 : cluster [DBG] pgmap v1175: 204 pgs: 1 active+remapped+backfill_toofull, 31 creating+peering, 172 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:20.570 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:19.884855+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? 172.21.15.87:0/871784530' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:43:20.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:19.432467+0000 mon.a (mon.0) 3802 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:43:20.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:19.439948+0000 mon.a (mon.0) 3803 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-134", "mode": "readproxy"}]': finished 2024-02-23T00:43:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:19.440002+0000 mon.a (mon.0) 3804 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-02-23T00:43:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:19.742508+0000 mon.a (mon.0) 3805 : audit [INF] from='client.? 172.21.15.87:0/1574564853' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [1, 2, 7, 3, 5, 0]}]: dispatch 2024-02-23T00:43:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:19.797390+0000 mgr.y (mgr.24437) 651 : cluster [DBG] pgmap v1175: 204 pgs: 1 active+remapped+backfill_toofull, 31 creating+peering, 172 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1/798 objects misplaced (0.125%) 2024-02-23T00:43:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:19.884855+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? 172.21.15.87:0/871784530' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:43:21.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:43:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:43:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:20.444087+0000 mon.a (mon.0) 3807 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:43:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:20.456733+0000 mon.a (mon.0) 3808 : audit [INF] from='client.? 172.21.15.87:0/1574564853' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [1, 2, 7, 3, 5, 0]}]': finished 2024-02-23T00:43:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:20.456774+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? 172.21.15.87:0/871784530' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:43:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:20.456801+0000 mon.a (mon.0) 3810 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-02-23T00:43:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:21.285056+0000 mgr.y (mgr.24437) 652 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:20.444087+0000 mon.a (mon.0) 3807 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:43:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:20.456733+0000 mon.a (mon.0) 3808 : audit [INF] from='client.? 172.21.15.87:0/1574564853' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [1, 2, 7, 3, 5, 0]}]': finished 2024-02-23T00:43:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:20.456774+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? 172.21.15.87:0/871784530' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:43:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:20.456801+0000 mon.a (mon.0) 3810 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-02-23T00:43:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:21.285056+0000 mgr.y (mgr.24437) 652 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:20.444087+0000 mon.a (mon.0) 3807 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:43:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:20.456733+0000 mon.a (mon.0) 3808 : audit [INF] from='client.? 172.21.15.87:0/1574564853' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.10", "id": [1, 2, 7, 3, 5, 0]}]': finished 2024-02-23T00:43:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:20.456774+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? 172.21.15.87:0/871784530' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:43:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:20.456801+0000 mon.a (mon.0) 3810 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-02-23T00:43:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:21.285056+0000 mgr.y (mgr.24437) 652 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:22.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:43:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:22] "GET /metrics HTTP/1.1" 200 190597 "" "Prometheus/2.33.4" 2024-02-23T00:43:22.580 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:22.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:22.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:22.713 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:22.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:21.468553+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-02-23T00:43:22.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:21.797816+0000 mgr.y (mgr.24437) 653 : cluster [DBG] pgmap v1178: 204 pgs: 1 peering, 1 active+remapped+backfill_toofull, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:22.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:22.241320+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-02-23T00:43:22.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:21.468553+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-02-23T00:43:22.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:21.797816+0000 mgr.y (mgr.24437) 653 : cluster [DBG] pgmap v1178: 204 pgs: 1 peering, 1 active+remapped+backfill_toofull, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:22.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:22.241320+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-02-23T00:43:22.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:21.468553+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-02-23T00:43:22.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:21.797816+0000 mgr.y (mgr.24437) 653 : cluster [DBG] pgmap v1178: 204 pgs: 1 peering, 1 active+remapped+backfill_toofull, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:22.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:22.241320+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-02-23T00:43:24.163 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:24.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:24.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:43:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:43:24.285 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:24.315 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:24.714 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:43:24.714 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:43:24.714 INFO:teuthology.orchestra.run.smithi087.stdout: "num_ops": 0 2024-02-23T00:43:24.715 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:43:25.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:23.798261+0000 mgr.y (mgr.24437) 654 : cluster [DBG] pgmap v1180: 204 pgs: 1 peering, 1 active+remapped+backfill_toofull, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 234 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:25.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:24.657571+0000 mon.a (mon.0) 3813 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:23.798261+0000 mgr.y (mgr.24437) 654 : cluster [DBG] pgmap v1180: 204 pgs: 1 peering, 1 active+remapped+backfill_toofull, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 234 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:25.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:24.657571+0000 mon.a (mon.0) 3813 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:23.798261+0000 mgr.y (mgr.24437) 654 : cluster [DBG] pgmap v1180: 204 pgs: 1 peering, 1 active+remapped+backfill_toofull, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 234 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:25.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:24.657571+0000 mon.a (mon.0) 3813 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:25.505 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 -- 172.21.15.87:0/916665310 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43540fdbb0 msgr2=0x7f43540fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 --2- 172.21.15.87:0/916665310 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43540fdbb0 0x7f43540fdf90 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 rx=0x7f4348004870 tx=0x7f4348009cb0).stop 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 -- 172.21.15.87:0/916665310 shutdown_connections 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 --2- 172.21.15.87:0/916665310 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43540f8420 0x7f4354104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 --2- 172.21.15.87:0/916665310 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43540fdbb0 0x7f43540fdf90 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 --2- 172.21.15.87:0/916665310 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43540fe510 0x7f43540f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 -- 172.21.15.87:0/916665310 >> 172.21.15.87:0/916665310 conn(0x7f43540f3800 msgr2=0x7f43540f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 -- 172.21.15.87:0/916665310 shutdown_connections 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 -- 172.21.15.87:0/916665310 wait complete. 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 librados: starting msgr at 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 librados: starting objecter 2024-02-23T00:43:25.737 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 Processor -- start 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 -- start start 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 librados: setting wanted keys 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.699+0000 7f435abb2700 1 librados: calling monclient init 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435abb2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43540f8420 0x7f435410e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435abb2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43540fdbb0 0x7f435410e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435abb2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43540fe510 0x7f43541082e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435abb2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43541087d0 con 0x7f43540fe510 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435abb2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4354108940 con 0x7f43540fdbb0 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435abb2700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4354108b00 con 0x7f43540f8420 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435914f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43540fe510 0x7f43541082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435914f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43540fe510 0x7f43541082e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:46410/0 (socket says 172.21.15.87:46410) 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435914f700 1 -- 172.21.15.87:0/2425654517 learned_addr learned my addr 172.21.15.87:0/2425654517 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43540f8420 0x7f435410e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.738 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f4353fff700 1 --2- 172.21.15.87:0/2425654517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43540fdbb0 0x7f435410e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435894e700 1 -- 172.21.15.87:0/2425654517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43540fdbb0 msgr2=0x7f435410e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43540fdbb0 0x7f435410e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435894e700 1 -- 172.21.15.87:0/2425654517 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43540fe510 msgr2=0x7f43541082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f43540fe510 0x7f43541082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435894e700 1 -- 172.21.15.87:0/2425654517 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4340005040 con 0x7f43540f8420 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f4353fff700 1 --2- 172.21.15.87:0/2425654517 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f43540fdbb0 0x7f435410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43540f8420 0x7f435410e0f0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 rx=0x7f4348004870 tx=0x7f434800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f4351ffb700 10 client.?.objecter ms_handle_connect 0x7f43540f8420 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f4351ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434800e070 con 0x7f43540f8420 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4348013da0 con 0x7f43540f8420 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348019070 con 0x7f43540f8420 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.700+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4348005040 con 0x7f43540f8420 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.701+0000 7f435abb2700 10 client.38345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:25.739 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.701+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4354109020 con 0x7f43540f8420 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.701+0000 7f435abb2700 1 librados: init done 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.702+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f435405ee80 con 0x7f43540f8420 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.702+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f434801e420 con 0x7f43540f8420 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.702+0000 7f4351ffb700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4344062c90 0x7f4344065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.702+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(771..771 src has 400..771) v4 ==== 8567+0+0 (secure 0 0 0) 0x7f434807dd40 con 0x7f43540f8420 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.702+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 osd_map(771..771 src has 400..771) v4 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.702+0000 7f4351ffb700 3 client.38345.objecter handle_osd_map got epochs [771,771] > 0 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.702+0000 7f4351ffb700 3 client.38345.objecter handle_osd_map decoding full epoch 771 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.702+0000 7f4353fff700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4344062c90 0x7f4344065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f4351ffb700 20 client.38345.objecter dump_active .. 0 homeless 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v771) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4348082030 con 0x7f43540f8420 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 10 client.38345.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:25.740 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 1 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6801/484081831 conn(0x7f4354028540 0x7f43540fe950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f4354107060 osd=0 3 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354107060 osd=0 3 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f4354107060 osd=0 3 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 1 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f4354107060 osd=0 3 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 0 1 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354107060 osd=0 4 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 10 client.38345.objecter _send_command 1 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4354109770 con 0x7f4354028540 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f4351ffb700 10 client.38345.objecter ms_handle_connect 0x7f4344062c90 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f4353fff700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4344062c90 0x7f4344065150 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 rx=0x7f4354109a30 tx=0x7f4340009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435914f700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6801/484081831 conn(0x7f4354028540 0x7f43540fe950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f4351ffb700 10 client.38345.objecter ms_handle_connect 0x7f4354028540 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.703+0000 7f435914f700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6801/484081831 conn(0x7f4354028540 0x7f43540fe950 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.741 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.704+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4354109770 con 0x7f4354028540 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.704+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.704+0000 7f4351ffb700 10 client.38345.objecter _finish_command 1 = system:0 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.704+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f4354107060 osd=0 4 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.704+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 0 1 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 10 client.38345.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 2 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f4354107060 osd=0 3 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354107060 osd=0 3 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f4354107060 osd=0 3 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 2 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f4354107060 osd=0 3 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 0 2 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354107060 osd=0 4 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 10 client.38345.objecter _send_command 2 2024-02-23T00:43:25.742 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.734+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f435405a040 con 0x7f4354028540 2024-02-23T00:43:25.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f435405a040 con 0x7f4354028540 2024-02-23T00:43:25.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:25.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f4351ffb700 10 client.38345.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f4354107060 osd=0 4 2024-02-23T00:43:25.743 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 0 2 2024-02-23T00:43:25.743 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:43:25.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 10 client.38345.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:25.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 3 2024-02-23T00:43:25.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6805/685354171 conn(0x7f4354027b90 0x7f4354027f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.744 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43540fffc0 osd=1 3 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540fffc0 osd=1 3 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43540fffc0 osd=1 3 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 3 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f43540fffc0 osd=1 3 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 1 3 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540fffc0 osd=1 4 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 10 client.38345.objecter _send_command 3 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4354100130 con 0x7f4354027b90 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.736+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6805/685354171 conn(0x7f4354027b90 0x7f4354027f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.737+0000 7f4351ffb700 10 client.38345.objecter ms_handle_connect 0x7f4354027b90 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.737+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6805/685354171 conn(0x7f4354027b90 0x7f4354027f70 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.737+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4354100130 con 0x7f4354027b90 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.737+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.737+0000 7f4351ffb700 10 client.38345.objecter _finish_command 3 = system:0 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.737+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f43540fffc0 osd=1 4 2024-02-23T00:43:25.745 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.737+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 1 3 2024-02-23T00:43:25.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 10 client.38345.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:25.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 4 2024-02-23T00:43:25.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43540fffc0 osd=1 3 2024-02-23T00:43:25.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540fffc0 osd=1 3 2024-02-23T00:43:25.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43540fffc0 osd=1 3 2024-02-23T00:43:25.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 4 2024-02-23T00:43:25.753 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f43540fffc0 osd=1 3 2024-02-23T00:43:25.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 1 4 2024-02-23T00:43:25.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540fffc0 osd=1 4 2024-02-23T00:43:25.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 10 client.38345.objecter _send_command 4 2024-02-23T00:43:25.754 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.752+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4354027600 con 0x7f4354027b90 2024-02-23T00:43:25.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.753+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f4354027600 con 0x7f4354027b90 2024-02-23T00:43:25.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.753+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:25.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.753+0000 7f4351ffb700 10 client.38345.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.753+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f43540fffc0 osd=1 4 2024-02-23T00:43:25.755 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.753+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 1 4 2024-02-23T00:43:25.755 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:43:25.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 10 client.38345.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:25.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 5 2024-02-23T00:43:25.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43540ffb20 0x7f435411c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f4354027a10 osd=2 3 2024-02-23T00:43:25.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354027a10 osd=2 3 2024-02-23T00:43:25.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f4354027a10 osd=2 3 2024-02-23T00:43:25.756 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 5 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f4354027a10 osd=2 3 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 2 5 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354027a10 osd=2 4 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 10 client.38345.objecter _send_command 5 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f435411ede0 con 0x7f43540ffb20 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f4353fff700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43540ffb20 0x7f435411c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f4351ffb700 10 client.38345.objecter ms_handle_connect 0x7f43540ffb20 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.754+0000 7f4353fff700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43540ffb20 0x7f435411c0d0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.755+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f435411ede0 con 0x7f43540ffb20 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.755+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.755+0000 7f4351ffb700 10 client.38345.objecter _finish_command 5 = system:0 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.755+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f4354027a10 osd=2 4 2024-02-23T00:43:25.757 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.755+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 2 5 2024-02-23T00:43:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:25.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:25.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 10 client.38345.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 6 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f4354027a10 osd=2 3 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354027a10 osd=2 3 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f4354027a10 osd=2 3 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 6 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f4354027a10 osd=2 3 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 2 6 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354027a10 osd=2 4 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 10 client.38345.objecter _send_command 6 2024-02-23T00:43:25.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.771+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f43540ff140 con 0x7f43540ffb20 2024-02-23T00:43:25.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f43540ff140 con 0x7f43540ffb20 2024-02-23T00:43:25.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:25.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f4351ffb700 10 client.38345.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f4354027a10 osd=2 4 2024-02-23T00:43:25.775 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 2 6 2024-02-23T00:43:25.775 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:43:25.776 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 10 client.38345.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 7 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6813/128059790 conn(0x7f435406af10 0x7f435411e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43540ff3c0 osd=3 3 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540ff3c0 osd=3 3 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43540ff3c0 osd=3 3 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 7 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f43540ff3c0 osd=3 3 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 3 7 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540ff3c0 osd=3 4 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 10 client.38345.objecter _send_command 7 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.773+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f435411eac0 con 0x7f435406af10 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.774+0000 7f435914f700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6813/128059790 conn(0x7f435406af10 0x7f435411e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.774+0000 7f4351ffb700 10 client.38345.objecter ms_handle_connect 0x7f435406af10 2024-02-23T00:43:25.777 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.774+0000 7f435914f700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6813/128059790 conn(0x7f435406af10 0x7f435411e5d0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.775+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f435411eac0 con 0x7f435406af10 2024-02-23T00:43:25.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.775+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:43:25.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.775+0000 7f4351ffb700 10 client.38345.objecter _finish_command 7 = system:0 2024-02-23T00:43:25.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.775+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f43540ff3c0 osd=3 4 2024-02-23T00:43:25.778 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.775+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 3 7 2024-02-23T00:43:25.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 10 client.38345.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:25.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 8 2024-02-23T00:43:25.791 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43540ff3c0 osd=3 3 2024-02-23T00:43:25.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540ff3c0 osd=3 3 2024-02-23T00:43:25.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43540ff3c0 osd=3 3 2024-02-23T00:43:25.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 8 2024-02-23T00:43:25.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f43540ff3c0 osd=3 3 2024-02-23T00:43:25.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 3 8 2024-02-23T00:43:25.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540ff3c0 osd=3 4 2024-02-23T00:43:25.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 10 client.38345.objecter _send_command 8 2024-02-23T00:43:25.792 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.790+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f43541193c0 con 0x7f435406af10 2024-02-23T00:43:25.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f43541193c0 con 0x7f435406af10 2024-02-23T00:43:25.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:25.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f4351ffb700 10 client.38345.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f43540ff3c0 osd=3 4 2024-02-23T00:43:25.793 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 3 8 2024-02-23T00:43:25.794 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:43:25.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 10 client.38345.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:25.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 9 2024-02-23T00:43:25.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4354119530 0x7f435406c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43541193c0 osd=4 3 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43541193c0 osd=4 3 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43541193c0 osd=4 3 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 9 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f43541193c0 osd=4 3 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 4 9 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43541193c0 osd=4 4 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 10 client.38345.objecter _send_command 9 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f435411fd50 con 0x7f4354119530 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.792+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4354119530 0x7f435406c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.793+0000 7f4351ffb700 10 client.38345.objecter ms_handle_connect 0x7f4354119530 2024-02-23T00:43:25.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.793+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4354119530 0x7f435406c4f0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.794+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f435411fd50 con 0x7f4354119530 2024-02-23T00:43:25.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.794+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:43:25.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.794+0000 7f4351ffb700 10 client.38345.objecter _finish_command 9 = system:0 2024-02-23T00:43:25.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.794+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f43541193c0 osd=4 4 2024-02-23T00:43:25.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.794+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 4 9 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 10 client.38345.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 10 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43541193c0 osd=4 3 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43541193c0 osd=4 3 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f43541193c0 osd=4 3 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 10 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f43541193c0 osd=4 3 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 4 10 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43541193c0 osd=4 4 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 10 client.38345.objecter _send_command 10 2024-02-23T00:43:25.811 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.809+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f435411fd50 con 0x7f4354119530 2024-02-23T00:43:25.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.811+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f435411fd50 con 0x7f4354119530 2024-02-23T00:43:25.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.811+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:25.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.811+0000 7f4351ffb700 10 client.38345.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.811+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f43541193c0 osd=4 4 2024-02-23T00:43:25.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.811+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 4 10 2024-02-23T00:43:25.813 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:43:25.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 10 client.38345.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:25.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 11 2024-02-23T00:43:25.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6804/2550013795 conn(0x7f43541265a0 0x7f4354128990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.814 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435411fd50 osd=5 3 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435411fd50 osd=5 3 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435411fd50 osd=5 3 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 11 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f435411fd50 osd=5 3 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 5 11 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435411fd50 osd=5 4 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 10 client.38345.objecter _send_command 11 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f435406cf20 con 0x7f43541265a0 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f4353fff700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6804/2550013795 conn(0x7f43541265a0 0x7f4354128990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f4351ffb700 10 client.38345.objecter ms_handle_connect 0x7f43541265a0 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.812+0000 7f4353fff700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6804/2550013795 conn(0x7f43541265a0 0x7f4354128990 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.813+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f435406cf20 con 0x7f43541265a0 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.813+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.813+0000 7f4351ffb700 10 client.38345.objecter _finish_command 11 = system:0 2024-02-23T00:43:25.815 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.813+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f435411fd50 osd=5 4 2024-02-23T00:43:25.816 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.813+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 5 11 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 10 client.38345.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 12 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435411fd50 osd=5 3 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435411fd50 osd=5 3 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435411fd50 osd=5 3 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 12 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f435411fd50 osd=5 3 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 5 12 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435411fd50 osd=5 4 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 10 client.38345.objecter _send_command 12 2024-02-23T00:43:25.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.827+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f435406cf20 con 0x7f43541265a0 2024-02-23T00:43:25.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.829+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f435406cf20 con 0x7f43541265a0 2024-02-23T00:43:25.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.829+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:25.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.829+0000 7f4351ffb700 10 client.38345.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.829+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f435411fd50 osd=5 4 2024-02-23T00:43:25.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.829+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 5 12 2024-02-23T00:43:25.831 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:43:25.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.829+0000 7f435abb2700 10 client.38345.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:25.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 13 2024-02-23T00:43:25.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4354129390 0x7f43541297f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435406cf20 osd=6 3 2024-02-23T00:43:25.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435406cf20 osd=6 3 2024-02-23T00:43:25.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435406cf20 osd=6 3 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 13 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f435406cf20 osd=6 3 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 6 13 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435406cf20 osd=6 4 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 10 client.38345.objecter _send_command 13 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f435412b440 con 0x7f4354129390 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435914f700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4354129390 0x7f43541297f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f4351ffb700 10 client.38345.objecter ms_handle_connect 0x7f4354129390 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.830+0000 7f435914f700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4354129390 0x7f43541297f0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.831+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f435412b440 con 0x7f4354129390 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.831+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.831+0000 7f4351ffb700 10 client.38345.objecter _finish_command 13 = system:0 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.831+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f435406cf20 osd=6 4 2024-02-23T00:43:25.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.831+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 6 13 2024-02-23T00:43:25.846 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 10 client.38345.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 14 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435406cf20 osd=6 3 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435406cf20 osd=6 3 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435406cf20 osd=6 3 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 14 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f435406cf20 osd=6 3 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 6 14 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435406cf20 osd=6 4 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 10 client.38345.objecter _send_command 14 2024-02-23T00:43:25.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.845+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f435412b440 con 0x7f4354129390 2024-02-23T00:43:25.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f435412b440 con 0x7f4354129390 2024-02-23T00:43:25.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:25.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f4351ffb700 10 client.38345.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f435406cf20 osd=6 4 2024-02-23T00:43:25.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 6 14 2024-02-23T00:43:25.849 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 10 client.38345.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 15 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6812/373787000 conn(0x7f435412db60 0x7f435412dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435412b440 osd=7 3 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435412b440 osd=7 3 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435412b440 osd=7 3 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 15 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f435412b440 osd=7 3 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 7 15 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435412b440 osd=7 4 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 10 client.38345.objecter _send_command 15 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.847+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f435412e610 con 0x7f435412db60 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.848+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6812/373787000 conn(0x7f435412db60 0x7f435412dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:25.850 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.848+0000 7f4351ffb700 10 client.38345.objecter ms_handle_connect 0x7f435412db60 2024-02-23T00:43:25.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.848+0000 7f435894e700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6812/373787000 conn(0x7f435412db60 0x7f435412dfa0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:25.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.849+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f435412e610 con 0x7f435412db60 2024-02-23T00:43:25.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.849+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:43:25.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.849+0000 7f4351ffb700 10 client.38345.objecter _finish_command 15 = system:0 2024-02-23T00:43:25.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.849+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f435412b440 osd=7 4 2024-02-23T00:43:25.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.849+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 7 15 2024-02-23T00:43:25.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 10 client.38345.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:25.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign -1 16 2024-02-23T00:43:25.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435412b440 osd=7 3 2024-02-23T00:43:25.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435412b440 osd=7 3 2024-02-23T00:43:25.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 20 client.38345.objecter _get_session s=0x7f435412b440 osd=7 3 2024-02-23T00:43:25.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 15 client.38345.objecter _session_command_op_remove -1 16 2024-02-23T00:43:25.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 20 client.38345.objecter get_session s=0x7f435412b440 osd=7 3 2024-02-23T00:43:25.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 15 client.38345.objecter _session_command_op_assign 7 16 2024-02-23T00:43:25.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435412b440 osd=7 4 2024-02-23T00:43:25.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 10 client.38345.objecter _send_command 16 2024-02-23T00:43:25.866 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.864+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f435412e610 con 0x7f435412db60 2024-02-23T00:43:25.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f4351ffb700 1 -- 172.21.15.87:0/2425654517 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f435412e610 con 0x7f435412db60 2024-02-23T00:43:25.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f4351ffb700 10 client.38345.objecter ms_dispatch 0x7f4354001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:25.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f4351ffb700 10 client.38345.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f4351ffb700 20 client.38345.objecter put_session s=0x7f435412b440 osd=7 4 2024-02-23T00:43:25.867 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f4351ffb700 15 client.38345.objecter _session_command_op_remove 7 16 2024-02-23T00:43:25.868 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:43:25.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 librados: watch_flush enter 2024-02-23T00:43:25.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 librados: watch_flush exit 2024-02-23T00:43:25.869 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 client.38345.objecter close_session for osd.0 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6801/484081831 conn(0x7f4354028540 msgr2=0x7f43540fe950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6801/484081831 conn(0x7f4354028540 0x7f43540fe950 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354107060 osd=0 1 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 client.38345.objecter close_session for osd.1 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6805/685354171 conn(0x7f4354027b90 msgr2=0x7f4354027f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6805/685354171 conn(0x7f4354027b90 0x7f4354027f70 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540fffc0 osd=1 1 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 client.38345.objecter close_session for osd.2 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43540ffb20 msgr2=0x7f435411c0d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6809/3605394228 conn(0x7f43540ffb20 0x7f435411c0d0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f4354027a10 osd=2 1 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 client.38345.objecter close_session for osd.3 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6813/128059790 conn(0x7f435406af10 msgr2=0x7f435411e5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6813/128059790 conn(0x7f435406af10 0x7f435411e5d0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.870 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435914f700 1 -- 172.21.15.87:0/2425654517 reap_dead start 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43540ff3c0 osd=3 1 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 client.38345.objecter close_session for osd.4 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4354119530 msgr2=0x7f435406c4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4354119530 0x7f435406c4f0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f43541193c0 osd=4 1 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 client.38345.objecter close_session for osd.5 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6804/2550013795 conn(0x7f43541265a0 msgr2=0x7f4354128990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6804/2550013795 conn(0x7f43541265a0 0x7f4354128990 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435411fd50 osd=5 1 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 client.38345.objecter close_session for osd.6 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4354129390 msgr2=0x7f43541297f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.871 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4354129390 0x7f43541297f0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435406cf20 osd=6 1 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 client.38345.objecter close_session for osd.7 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6812/373787000 conn(0x7f435412db60 msgr2=0x7f435412dfa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.171:6812/373787000 conn(0x7f435412db60 0x7f435412dfa0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 20 client.38345.objecter put_session s=0x7f435412b440 osd=7 1 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 20 client.38345.objecter shutdown clearing up homeless session... 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 10 client.38345.objecter successfully canceled tick 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435914f700 1 -- 172.21.15.87:0/2425654517 reap_dead start 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4344062c90 msgr2=0x7f4344065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.872 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4344062c90 0x7f4344065150 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 rx=0x7f4354109a30 tx=0x7f4340009cb0).stop 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43540f8420 msgr2=0x7f435410e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43540f8420 0x7f435410e0f0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 rx=0x7f4348004870 tx=0x7f434800b040).stop 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 shutdown_connections 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4344062c90 0x7f4344065150 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 --2- 172.21.15.87:0/2425654517 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f43540f8420 0x7f435410e0f0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 >> 172.21.15.87:0/2425654517 conn(0x7f43540f3800 msgr2=0x7f43540fb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 shutdown_connections 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 -- 172.21.15.87:0/2425654517 wait complete. 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:25.866+0000 7f435abb2700 1 librados: shutdown 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.873 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.874 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.874 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.874 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:43:25.874 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:25.903 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:26.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:25.702912+0000 mon.c (mon.2) 502 : audit [DBG] from='client.? 172.21.15.87:0/2425654517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:26.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:25.702912+0000 mon.c (mon.2) 502 : audit [DBG] from='client.? 172.21.15.87:0/2425654517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:25.702912+0000 mon.c (mon.2) 502 : audit [DBG] from='client.? 172.21.15.87:0/2425654517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:26.204 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:26.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:26.311 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:26.343 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:26.364 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:26.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:26.468 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:26.476 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:26.498 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:26.518 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:26.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:26.622 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:26.650 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:26.668 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:26.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:26.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:26.800 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:26.818 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:26.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:26.922 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:26.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:25.799046+0000 mgr.y (mgr.24437) 655 : cluster [DBG] pgmap v1181: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:26.949 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:26.968 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:26.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:27.072 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:27.101 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:27.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:25.799046+0000 mgr.y (mgr.24437) 655 : cluster [DBG] pgmap v1181: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:27.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:25.799046+0000 mgr.y (mgr.24437) 655 : cluster [DBG] pgmap v1181: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:27.350 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:27.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:27.455 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:27.482 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:27.500 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:27.502 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:43:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:43:27+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-02-23T00:43:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:27.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:27.575 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-23T00:43:27.575 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:43:27.575 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-23T00:43:27.575 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-23T00:43:27.604 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:27.631 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:27.650 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:27.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:27.753 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:27.781 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:27.878 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:27.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:27.982 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:28.009 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:28.060 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:28.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:28.167 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:28.195 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:28.215 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:28.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:28.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-23T00:43:28.319 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:28.348 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:28.367 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:28.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:28.471 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:28.498 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:28.516 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:28.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:28.533 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:28.620 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:28.646 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:28.666 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:28.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:28.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:28.796 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:28.833 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:28.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:28.937 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:28.963 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:29.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:27.799443+0000 mgr.y (mgr.24437) 656 : cluster [DBG] pgmap v1182: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 139 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:29.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:27.799443+0000 mgr.y (mgr.24437) 656 : cluster [DBG] pgmap v1182: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 139 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:29.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:27.799443+0000 mgr.y (mgr.24437) 656 : cluster [DBG] pgmap v1182: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 139 B/s wr, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:29.240 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:29.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:29.347 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:29.370 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:43:29.375 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:29.448 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:29.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:29.551 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:29.581 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:29.585 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:43:29.585 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:43:30.036 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:30.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:30.140 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:30.166 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:29.636648+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:30.167 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:29.636648+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:30.168 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:29.636648+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:30.326 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:30.358 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:30.616 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:30.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:30.725 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:30.760 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:30.780 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:30.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:30.874 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:29.799879+0000 mgr.y (mgr.24437) 657 : cluster [DBG] pgmap v1183: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 0 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:30.884 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:30.915 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:30.935 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:30.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:31.038 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:31.066 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:29.799879+0000 mgr.y (mgr.24437) 657 : cluster [DBG] pgmap v1183: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 0 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:29.878734+0000 mon.a (mon.0) 3815 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:43:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:29.878789+0000 mon.a (mon.0) 3816 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-02-23T00:43:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:29.879235+0000 mon.a (mon.0) 3817 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134"}]: dispatch 2024-02-23T00:43:31.206 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:29.799879+0000 mgr.y (mgr.24437) 657 : cluster [DBG] pgmap v1183: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 0 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:31.206 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:29.878734+0000 mon.a (mon.0) 3815 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:43:31.206 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:29.878789+0000 mon.a (mon.0) 3816 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-02-23T00:43:31.206 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:29.879235+0000 mon.a (mon.0) 3817 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134"}]: dispatch 2024-02-23T00:43:31.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:29.878734+0000 mon.a (mon.0) 3815 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:43:31.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:29.878789+0000 mon.a (mon.0) 3816 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-02-23T00:43:31.207 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:29.879235+0000 mon.a (mon.0) 3817 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134"}]: dispatch 2024-02-23T00:43:31.215 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:31.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:31.228 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-23T00:43:31.319 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:31.349 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:31.369 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:31.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:31.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:31.503 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:31.523 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:31.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:31.626 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:31.653 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:31.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:43:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:43:31.876 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:30.871414+0000 mon.a (mon.0) 3818 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:43:31.876 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:30.880420+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134"}]': finished 2024-02-23T00:43:31.876 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:30.880482+0000 mon.a (mon.0) 3820 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-02-23T00:43:31.877 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:30.902389+0000 mon.a (mon.0) 3821 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:31.877 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:30.902662+0000 mon.a (mon.0) 3822 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134"}]: dispatch 2024-02-23T00:43:31.877 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:31.293205+0000 mgr.y (mgr.24437) 658 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:31.877 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:30.871414+0000 mon.a (mon.0) 3818 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:43:31.877 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:30.880420+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134"}]': finished 2024-02-23T00:43:31.877 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:30.880482+0000 mon.a (mon.0) 3820 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-02-23T00:43:31.877 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:30.902389+0000 mon.a (mon.0) 3821 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:31.877 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:30.902662+0000 mon.a (mon.0) 3822 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134"}]: dispatch 2024-02-23T00:43:31.877 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:31.293205+0000 mgr.y (mgr.24437) 658 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:32.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:30.871414+0000 mon.a (mon.0) 3818 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:43:32.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:30.880420+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134"}]': finished 2024-02-23T00:43:32.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:30.880482+0000 mon.a (mon.0) 3820 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-02-23T00:43:32.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:30.902389+0000 mon.a (mon.0) 3821 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:32.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:30.902662+0000 mon.a (mon.0) 3822 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-134"}]: dispatch 2024-02-23T00:43:32.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:31.293205+0000 mgr.y (mgr.24437) 658 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:32.482 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:43:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:32] "GET /metrics HTTP/1.1" 200 190715 "" "Prometheus/2.33.4" 2024-02-23T00:43:32.703 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:32.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:32.808 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:32.837 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:31.800539+0000 mgr.y (mgr.24437) 659 : cluster [DBG] pgmap v1186: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:31.883222+0000 mon.a (mon.0) 3823 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-02-23T00:43:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:32.247084+0000 mon.a (mon.0) 3824 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-02-23T00:43:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:32.249254+0000 mon.a (mon.0) 3825 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:33.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:31.800539+0000 mgr.y (mgr.24437) 659 : cluster [DBG] pgmap v1186: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:33.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:31.883222+0000 mon.a (mon.0) 3823 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-02-23T00:43:33.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:32.247084+0000 mon.a (mon.0) 3824 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-02-23T00:43:33.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:32.249254+0000 mon.a (mon.0) 3825 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:31.800539+0000 mgr.y (mgr.24437) 659 : cluster [DBG] pgmap v1186: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:31.883222+0000 mon.a (mon.0) 3823 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-02-23T00:43:33.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:32.247084+0000 mon.a (mon.0) 3824 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-02-23T00:43:33.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:32.249254+0000 mon.a (mon.0) 3825 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:33.385 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:33.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:33.489 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:33.522 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:33.543 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:33.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:33.647 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:33.676 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:33.696 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:33.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:33.800 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:33.828 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:33.885 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:43:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:43:33.952 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:33.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:33.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:34.056 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:34.081 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:33.232881+0000 mon.a (mon.0) 3826 : audit [DBG] from='client.? 172.21.15.87:0/3645691748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:43:34.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:33.248310+0000 mon.a (mon.0) 3827 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:34.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:33.248433+0000 mon.a (mon.0) 3828 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-02-23T00:43:34.082 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:33.273638+0000 mon.a (mon.0) 3829 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:34.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:33.232881+0000 mon.a (mon.0) 3826 : audit [DBG] from='client.? 172.21.15.87:0/3645691748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:43:34.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:33.248310+0000 mon.a (mon.0) 3827 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:34.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:33.248433+0000 mon.a (mon.0) 3828 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-02-23T00:43:34.082 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:33.273638+0000 mon.a (mon.0) 3829 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:34.084 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:34.181 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:34.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:33.232881+0000 mon.a (mon.0) 3826 : audit [DBG] from='client.? 172.21.15.87:0/3645691748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-23T00:43:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:33.248310+0000 mon.a (mon.0) 3827 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:33.248433+0000 mon.a (mon.0) 3828 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-02-23T00:43:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:33.273638+0000 mon.a (mon.0) 3829 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:34.285 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:34.317 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:34.338 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:34.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 -- 172.21.15.87:0/1074431174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fdbb0 msgr2=0x7fd8c00fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/1074431174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fdbb0 0x7fd8c00fdf90 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 rx=0x7fd8bc004870 tx=0x7fd8bc009cb0).stop 2024-02-23T00:43:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 -- 172.21.15.87:0/1074431174 shutdown_connections 2024-02-23T00:43:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/1074431174 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8c00f8420 0x7fd8c0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/1074431174 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fdbb0 0x7fd8c00fdf90 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/1074431174 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8c00fe510 0x7fd8c00f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:34.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 -- 172.21.15.87:0/1074431174 >> 172.21.15.87:0/1074431174 conn(0x7fd8c00f3800 msgr2=0x7fd8c00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 -- 172.21.15.87:0/1074431174 shutdown_connections 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 -- 172.21.15.87:0/1074431174 wait complete. 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 librados: starting msgr at 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 librados: starting objecter 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 Processor -- start 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 -- start start 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 librados: setting wanted keys 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.059+0000 7fd8c86fb700 1 librados: calling monclient init 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8c00f8420 0x7fd8c010e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8c00fdbb0 0x7fd8c010e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fe510 0x7fd8c01081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c01086b0 con 0x7fd8c00fe510 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c0108820 con 0x7fd8c00fdbb0 2024-02-23T00:43:34.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c01089e0 con 0x7fd8c00f8420 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6c98700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fe510 0x7fd8c01081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6497700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8c00f8420 0x7fd8c010e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6c98700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fe510 0x7fd8c01081c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38956/0 (socket says 172.21.15.87:38956) 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6c98700 1 -- 172.21.15.87:0/3645691748 learned_addr learned my addr 172.21.15.87:0/3645691748 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c5c96700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8c00fdbb0 0x7fd8c010e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6c98700 1 -- 172.21.15.87:0/3645691748 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8c00f8420 msgr2=0x7fd8c010e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6c98700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8c00f8420 0x7fd8c010e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6c98700 1 -- 172.21.15.87:0/3645691748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8c00fdbb0 msgr2=0x7fd8c010e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6c98700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8c00fdbb0 0x7fd8c010e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6c98700 1 -- 172.21.15.87:0/3645691748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b80097c0 con 0x7fd8c00fe510 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c5c96700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8c00fdbb0 0x7fd8c010e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6497700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8c00f8420 0x7fd8c010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c6c98700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fe510 0x7fd8c01081c0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 rx=0x7fd8b8001df0 tx=0x7fd8b8004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:34.440 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8b77fe700 10 client.?.objecter ms_handle_connect 0x7fd8c00fe510 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8b77fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8b77fe700 1 -- 172.21.15.87:0/3645691748 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8009a10 con 0x7fd8c00fe510 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc005040 con 0x7fd8c00fe510 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8b77fe700 1 -- 172.21.15.87:0/3645691748 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8b80095a0 con 0x7fd8c00fe510 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8b77fe700 1 -- 172.21.15.87:0/3645691748 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8007da0 con 0x7fd8c00fe510 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 10 client.27876.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c0108fd0 con 0x7fd8c00fe510 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.060+0000 7fd8c86fb700 1 librados: init done 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.061+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c005ee80 con 0x7fd8c00fe510 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8b77fe700 1 -- 172.21.15.87:0/3645691748 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd8b80090d0 con 0x7fd8c00fe510 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8b77fe700 1 --2- 172.21.15.87:0/3645691748 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd8ac062c90 0x7fd8ac065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8b77fe700 1 -- 172.21.15.87:0/3645691748 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(775..775 src has 400..775) v4 ==== 8559+0+0 (secure 0 0 0) 0x7fd8b807dda0 con 0x7fd8c00fe510 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8b77fe700 10 client.27876.objecter ms_dispatch 0x7fd8c0001550 osd_map(775..775 src has 400..775) v4 2024-02-23T00:43:34.441 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8b77fe700 3 client.27876.objecter handle_osd_map got epochs [775,775] > 0 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8b77fe700 3 client.27876.objecter handle_osd_map decoding full epoch 775 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8c6497700 1 --2- 172.21.15.87:0/3645691748 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd8ac062c90 0x7fd8ac065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8b77fe700 20 client.27876.objecter dump_active .. 0 homeless 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8b77fe700 10 client.27876.objecter ms_handle_connect 0x7fd8ac062c90 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.062+0000 7fd8c6497700 1 --2- 172.21.15.87:0/3645691748 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd8ac062c90 0x7fd8ac065150 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7fd8bc0043d0 tx=0x7fd8bc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.066+0000 7fd8b77fe700 1 -- 172.21.15.87:0/3645691748 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd8b8051080 con 0x7fd8c00fe510 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.232+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd8c01097b0 con 0x7fd8c00fe510 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.233+0000 7fd8b77fe700 1 -- 172.21.15.87:0/3645691748 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v775) v1 ==== 81+0+49121 (secure 0 0 0) 0x7fd8b8050c70 con 0x7fd8c00fe510 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:43:34.442 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "epoch": 775, 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "fsid": "66db207c-d1e1-11ee-95c0-87774f69a715", 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "created": "2024-02-23T00:21:28.861318+0000", 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "modified": "2024-02-23T00:43:32.240009+0000", 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "last_up_change": "2024-02-23T00:41:18.209057+0000", 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "last_in_change": "2024-02-23T00:41:36.208573+0000", 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_num": 5801984, 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_set": [ 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "noscrub", 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "pglog_hardlimit", 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "purged_snapdirs", 2024-02-23T00:43:34.443 INFO:teuthology.orchestra.run.smithi087.stdout: "recovery_deletes", 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "sortbitwise" 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_version": 114, 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "full_ratio": 0.94999998807907104, 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_snapshot": "", 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_max": 252, 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "max_osd": 8, 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "require_min_compat_client": "luminous", 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "min_compat_client": "luminous", 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "require_osd_release": "pacific", 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: "pools": [ 2024-02-23T00:43:34.444 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 1, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "device_health_metrics", 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:25:09.208309+0000", 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:34.445 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 1, 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 1, 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 1, 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 1, 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 1, 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "68", 2024-02-23T00:43:34.446 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:34.447 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:34.448 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_max": 32, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_min": 1 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.449 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "mgr_devicehealth": {} 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 2, 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": ".rgw.root", 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.562186+0000", 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:34.450 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:34.451 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "174", 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:34.452 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:34.453 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:34.454 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.455 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 3, 2024-02-23T00:43:34.456 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "iscsi", 2024-02-23T00:43:34.456 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:15.989206+0000", 2024-02-23T00:43:34.456 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:34.456 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:34.456 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:34.456 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:34.456 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:34.456 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:34.456 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 23, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 23, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 23, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 23, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 23, 2024-02-23T00:43:34.457 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "738", 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "736", 2024-02-23T00:43:34.458 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:34.459 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:34.460 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:34.460 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:34.460 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:34.460 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:34.460 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:34.460 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:34.460 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:34.460 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:34.460 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:34.461 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: "rbd": {} 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 4, 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.log", 2024-02-23T00:43:34.462 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:17.282440+0000", 2024-02-23T00:43:34.463 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:34.463 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:34.463 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:34.463 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:34.463 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:34.463 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:34.463 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:34.463 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:34.463 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "4.16", 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 371, 2024-02-23T00:43:34.464 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 370, 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 370, 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "63'21", 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "62'12" 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "532", 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "371", 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "481", 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:34.465 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:34.466 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:34.466 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:34.466 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:34.466 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:34.466 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:34.466 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:34.466 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:34.466 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:34.466 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.467 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:34.468 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 5, 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.control", 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:19.262164+0000", 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:34.469 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:34.470 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "5.20", 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 641, 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 640, 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 640, 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "562'12", 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "330'9" 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "641", 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "641", 2024-02-23T00:43:34.471 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "639", 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:34.472 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:34.473 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:34.474 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 6, 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "default.rgw.meta", 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:27:21.319951+0000", 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:34.475 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 12, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 12, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 12, 2024-02-23T00:43:34.476 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 12, 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 12, 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "6.c", 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 665, 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 664, 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 664, 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "78'2", 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "665", 2024-02-23T00:43:34.477 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "665", 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "663", 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:34.478 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:34.479 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:34.480 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:34.480 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.480 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:34.480 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:34.480 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:34.480 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:34.480 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:34.480 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:34.480 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:34.481 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:34.481 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:34.481 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:34.481 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:34.481 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 10 librados: watch_flush enter 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 10 librados: watch_flush exit 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 20 client.27876.objecter shutdown clearing up homeless session... 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 10 client.27876.objecter successfully canceled tick 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd8ac062c90 msgr2=0x7fd8ac065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/3645691748 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd8ac062c90 0x7fd8ac065150 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 rx=0x7fd8bc0043d0 tx=0x7fd8bc00b040).stop 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fe510 msgr2=0x7fd8c01081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fe510 0x7fd8c01081c0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 rx=0x7fd8b8001df0 tx=0x7fd8b8004c50).stop 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 shutdown_connections 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/3645691748 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd8ac062c90 0x7fd8ac065150 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd8c00fe510 0x7fd8c01081c0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:34.482 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd8c00f8420 0x7fd8c010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 --2- 172.21.15.87:0/3645691748 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd8c00fdbb0 0x7fd8c010e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.235+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 >> 172.21.15.87:0/3645691748 conn(0x7fd8c00f3800 msgr2=0x7fd8c00f5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.236+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 shutdown_connections 2024-02-23T00:43:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.236+0000 7fd8c86fb700 1 -- 172.21.15.87:0/3645691748 wait complete. 2024-02-23T00:43:34.483 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:33.236+0000 7fd8c86fb700 1 librados: shutdown 2024-02-23T00:43:34.483 INFO:teuthology.orchestra.run.smithi087.stdout: "options": { 2024-02-23T00:43:34.483 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_bias": 4 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": { 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: "rgw": {} 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 22, 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:30:46.071806+0000", 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 1, 2024-02-23T00:43:34.484 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool", 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:34.485 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "99", 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:34.486 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:34.487 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:34.488 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:34.489 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 234, 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-101", 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:40:15.435168+0000", 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 8193, 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 3, 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 1, 2024-02-23T00:43:34.490 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 8, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 8, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 8, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 8, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 8, 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:34.491 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "773", 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "772", 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "772", 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "772", 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:34.492 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 10, 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 766, 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:34.493 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "testprofile-test-rados-api-smithi087-82091-101", 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:34.494 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 8192, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.495 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 252, 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_name": "test-rados-api-smithi087-82091-136", 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "create_time": "2024-02-23T00:43:32.130424+0000", 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "flags": 32769, 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "flags_names": "hashpspool,creating", 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "type": 1, 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 3, 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "min_size": 2, 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_rule": 0, 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_count": 0, 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_target": 0, 2024-02-23T00:43:34.496 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_barrier": 0, 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "object_hash": 2, 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_autoscale_mode": "off", 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num": 32, 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num": 32, 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_placement_num_target": 32, 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_target": 32, 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_num_pending": 32, 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "last_pg_merge_meta": { 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "source_pgid": "0.0", 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "ready_epoch": 0, 2024-02-23T00:43:34.497 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_started": 0, 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "last_epoch_clean": 0, 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "source_version": "0'0", 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "target_version": "0'0" 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "last_change": "775", 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend": "0", 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "last_force_op_resend_preluminous": "0", 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "auid": 0, 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_mode": "selfmanaged", 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_seq": 0, 2024-02-23T00:43:34.498 INFO:teuthology.orchestra.run.smithi087.stdout: "snap_epoch": 0, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "pool_snaps": [], 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps": "[]", 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_bytes": 0, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "quota_max_objects": 0, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "tiers": [], 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "tier_of": -1, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "read_tier": -1, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "write_tier": -1, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_mode": "none", 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_bytes": 0, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "target_max_objects": 0, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-23T00:43:34.499 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_flush_age": 0, 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "cache_min_evict_age": 0, 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profile": "", 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_params": { 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "none" 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_period": 0, 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_count": 0, 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "use_gmt_hitset": true, 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "min_read_recency_for_promote": 0, 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "min_write_recency_for_promote": 0, 2024-02-23T00:43:34.500 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_grade_decay_rate": 0, 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "hit_set_search_last_n": 0, 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "grade_table": [], 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "stripe_width": 0, 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "expected_num_objects": 0, 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "fast_read": false, 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "options": {}, 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "application_metadata": {} 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "e3549bad-2891-47b9-bf77-132cadc20bf7", 2024-02-23T00:43:34.501 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 8, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 424, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 659, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 764, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 425, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.502 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6801", 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6802", 2024-02-23T00:43:34.503 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6804", 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.504 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6803", 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 484081831 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6801/484081831", 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6802/484081831", 2024-02-23T00:43:34.505 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6804/484081831", 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6803/484081831", 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "85525ce8-38d5-4840-8057-547cb176c859", 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:34.506 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 13, 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 764, 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.507 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6805", 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6806", 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:43:34.508 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6808", 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.509 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6807", 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 685354171 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6805/685354171", 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6806/685354171", 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6808/685354171", 2024-02-23T00:43:34.510 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6807/685354171", 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "48ea64b6-ed15-48d2-b0ed-aa4053938aaa", 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:34.511 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 18, 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 764, 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6809", 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:43:34.512 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6810", 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.513 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6812", 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.514 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6811", 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 3605394228 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6809/3605394228", 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6810/3605394228", 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6812/3605394228", 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6811/3605394228", 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:34.515 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "cb8705aa-e114-4dfd-ac9d-ec12f85f877a", 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:34.516 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 26, 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 769, 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6813", 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6814", 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:34.518 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6816", 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.87:6815", 2024-02-23T00:43:34.519 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 128059790 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.87:6813/128059790", 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.87:6814/128059790", 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.87:6816/128059790", 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.87:6815/128059790", 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "c3734a04-7343-4251-9b5b-bedb9e8b882c", 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.5775604248046875, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 31, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 764, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:34.521 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6800", 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6801", 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6803", 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.523 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6802", 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 4175844757 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.524 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6800/4175844757", 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6801/4175844757", 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6803/4175844757", 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6802/4175844757", 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "fe86232d-96b9-4cbf-9cac-373286fa103d", 2024-02-23T00:43:34.525 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 0.8517608642578125, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 38, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 764, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.526 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6804", 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6805", 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:43:34.527 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6807", 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:34.528 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6806", 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2550013795 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6804/2550013795", 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6805/2550013795", 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6807/2550013795", 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6806/2550013795", 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:34.529 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "bfc5dd69-5cfb-409f-985f-83ee0dd4dc01", 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:34.530 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 44, 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 764, 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6808", 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.531 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6809", 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.532 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6811", 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6810", 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 2509408498 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6808/2509408498", 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6809/2509408498", 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6811/2509408498", 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6810/2509408498", 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:43:34.534 INFO:teuthology.orchestra.run.smithi087.stdout: "uuid": "29ddcf97-369a-4a2c-b3e2-142e84ccf939", 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "up": 1, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "in": 1, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "weight": 1, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_affinity": 1, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_begin": 0, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "last_clean_end": 0, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "up_from": 51, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "up_thru": 764, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "down_at": 0, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "lost_at": 0, 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addrs": { 2024-02-23T00:43:34.535 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6812", 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addrs": { 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.536 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6813", 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addrs": { 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6815", 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addrs": { 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "addrvec": [ 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "type": "v2", 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "addr": "172.21.15.171:6814", 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "nonce": 373787000 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "public_addr": "172.21.15.171:6812/373787000", 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "cluster_addr": "172.21.15.171:6813/373787000", 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_back_addr": "172.21.15.171:6815/373787000", 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "heartbeat_front_addr": "172.21.15.171:6814/373787000", 2024-02-23T00:43:34.538 INFO:teuthology.orchestra.run.smithi087.stdout: "state": [ 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: "exists", 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: "up" 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: "osd_xinfo": [ 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 0, 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:34.539 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:33.795687+0000", 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 424 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 1, 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:13.973488+0000", 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:34.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 2, 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:43:10.808451+0000", 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 3, 2024-02-23T00:43:34.541 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:32.316583+0000", 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 4, 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:34.542 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:31.854915+0000", 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 5, 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:43:14.040508+0000", 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:34.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 6, 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:29.973812+0000", 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.544 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "osd": 7, 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "down_stamp": "0.000000", 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_probability": 0, 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "laggy_interval": 0, 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "features": 4540138314316775423, 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "old_weight": 0, 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "last_purged_snaps_scrub": "2024-02-23T00:42:26.420812+0000", 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "dead_epoch": 0 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap": [], 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_upmap_items": [ 2024-02-23T00:43:34.545 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.18", 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 1 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "2.1c", 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:34.546 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 4 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "4.13", 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 2 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.547 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "5.10", 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 3 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.548 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "5.19", 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 1, 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 6 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "22.1a", 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: "mappings": [ 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.549 INFO:teuthology.orchestra.run.smithi087.stdout: "from": 7, 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: "to": 5 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: "pg_temp": [ 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: "pgid": "5.3", 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: "osds": [ 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: 3, 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: 6, 2024-02-23T00:43:34.550 INFO:teuthology.orchestra.run.smithi087.stdout: 5 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: "primary_temp": [], 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: "blocklist": { 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1121018082": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/2138166229": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/27794119": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/1675126043": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/3767016525": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:43:34.551 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/969918521": "2024-02-24T00:22:22.047081+0000", 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/795095363": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2273151327": "2024-02-24T00:27:39.443000+0000", 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:6800/1685406414": "2024-02-24T00:21:58.144574+0000", 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "172.21.15.87:0/2093733106": "2024-02-24T00:21:58.144574+0000" 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "range_blocklist": {}, 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "erasure_code_profiles": { 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "default": { 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:43:34.552 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van" 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "testprofile-test-rados-api-smithi087-82091-101": { 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-device-class": "", 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-failure-domain": "osd", 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "crush-root": "default", 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "k": "2", 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "m": "1", 2024-02-23T00:43:34.553 INFO:teuthology.orchestra.run.smithi087.stdout: "plugin": "jerasure", 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: "ruleset-failure-domain": "osd", 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: "technique": "reed_sol_van", 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: "w": "8" 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: "removed_snaps_queue": [ 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 7, 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:43:34.554 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:43:34.555 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 21, 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 3, 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.556 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 61, 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.557 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 63, 2024-02-23T00:43:34.558 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 29, 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 78, 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.559 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 86, 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.560 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 93, 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:34.561 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 95, 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 4, 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 4 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.562 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 107, 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 112, 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.563 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 5, 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 7, 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 125, 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.564 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 10, 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 2 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.565 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 133, 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 12, 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.566 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 154, 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 14, 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 164, 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 17, 2024-02-23T00:43:34.567 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 177, 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 20, 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.568 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 237, 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 2, 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.569 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 241, 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 6, 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: "pool": 245, 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: "snaps": [ 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: "begin": 9, 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: "length": 1 2024-02-23T00:43:34.570 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: ], 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "new_removed_snaps": [], 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "new_purged_snaps": [], 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "crush_node_flags": {}, 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "device_class_flags": {}, 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode": { 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_enabled": false, 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_bucket_count": 0, 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "degraded_stretch_mode": 0, 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "recovering_stretch_mode": 0, 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: "stretch_mode_bucket": 0 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:34.571 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:43:34.572 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:34.603 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:34.623 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:34.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:34.727 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:34.756 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:34.776 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:34.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:34.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:34.880 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:34.911 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:34.932 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:34.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:35.036 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:35.065 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:35.086 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:35.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:35.190 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:35.218 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:35.237 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:35.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:35.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:35.371 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:35.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:33.800942+0000 mgr.y (mgr.24437) 660 : cluster [DBG] pgmap v1190: 204 pgs: 7 creating+peering, 25 unknown, 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:35.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:34.251446+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:35.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:34.251512+0000 mon.a (mon.0) 3831 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-02-23T00:43:35.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:34.253676+0000 mon.a (mon.0) 3832 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-136"}]: dispatch 2024-02-23T00:43:35.504 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:33.800942+0000 mgr.y (mgr.24437) 660 : cluster [DBG] pgmap v1190: 204 pgs: 7 creating+peering, 25 unknown, 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:34.251446+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:34.251512+0000 mon.a (mon.0) 3831 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-02-23T00:43:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:34.253676+0000 mon.a (mon.0) 3832 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-136"}]: dispatch 2024-02-23T00:43:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:33.800942+0000 mgr.y (mgr.24437) 660 : cluster [DBG] pgmap v1190: 204 pgs: 7 creating+peering, 25 unknown, 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 1/801 objects misplaced (0.125%) 2024-02-23T00:43:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:34.251446+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:34.251512+0000 mon.a (mon.0) 3831 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-02-23T00:43:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:34.253676+0000 mon.a (mon.0) 3832 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-136"}]: dispatch 2024-02-23T00:43:35.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.597+0000 7fc971eb4700 1 -- 172.21.15.87:0/2761431997 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fd420 msgr2=0x7fc96c0f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.597+0000 7fc971eb4700 1 --2- 172.21.15.87:0/2761431997 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fd420 0x7fc96c0f6c50 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 rx=0x7fc960004870 tx=0x7fc960009cb0).stop 2024-02-23T00:43:35.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.597+0000 7fc971eb4700 1 -- 172.21.15.87:0/2761431997 shutdown_connections 2024-02-23T00:43:35.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.597+0000 7fc971eb4700 1 --2- 172.21.15.87:0/2761431997 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c0f7330 0x7fc96c103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.653 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.597+0000 7fc971eb4700 1 --2- 172.21.15.87:0/2761431997 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc96c0fcac0 0x7fc96c0fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.597+0000 7fc971eb4700 1 --2- 172.21.15.87:0/2761431997 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fd420 0x7fc96c0f6c50 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.597+0000 7fc971eb4700 1 -- 172.21.15.87:0/2761431997 >> 172.21.15.87:0/2761431997 conn(0x7fc96c0f26f0 msgr2=0x7fc96c0f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.597+0000 7fc971eb4700 1 -- 172.21.15.87:0/2761431997 shutdown_connections 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.597+0000 7fc971eb4700 1 -- 172.21.15.87:0/2761431997 wait complete. 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 librados: starting msgr at 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 librados: starting objecter 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 Processor -- start 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 -- start start 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 librados: setting wanted keys 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 librados: calling monclient init 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc96c0f7330 0x7fc96c10e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fcac0 0x7fc96c10e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c0fd420 0x7fc96c108220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c108710 con 0x7fc96c0f7330 2024-02-23T00:43:35.654 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c108880 con 0x7fc96c0fd420 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc971eb4700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c108a40 con 0x7fc96c0fcac0 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc96affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fcac0 0x7fc96c10e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc96b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc96c0f7330 0x7fc96c10e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc96affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fcac0 0x7fc96c10e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:43998/0 (socket says 172.21.15.87:43998) 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc96b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc96c0f7330 0x7fc96c10e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38978/0 (socket says 172.21.15.87:38978) 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.598+0000 7fc96affd700 1 -- 172.21.15.87:0/1649302779 learned_addr learned my addr 172.21.15.87:0/1649302779 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc96bfff700 1 --2- 172.21.15.87:0/1649302779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c0fd420 0x7fc96c108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc96b7fe700 1 -- 172.21.15.87:0/1649302779 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fcac0 msgr2=0x7fc96c10e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc96b7fe700 1 --2- 172.21.15.87:0/1649302779 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fcac0 0x7fc96c10e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc96b7fe700 1 -- 172.21.15.87:0/1649302779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c0fd420 msgr2=0x7fc96c108220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc96b7fe700 1 --2- 172.21.15.87:0/1649302779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c0fd420 0x7fc96c108220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc96b7fe700 1 -- 172.21.15.87:0/1649302779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c005040 con 0x7fc96c0f7330 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc96affd700 1 --2- 172.21.15.87:0/1649302779 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fcac0 0x7fc96c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc96b7fe700 1 --2- 172.21.15.87:0/1649302779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc96c0f7330 0x7fc96c10e080 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7fc95c004920 tx=0x7fc95c009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.655 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc968ff9700 10 client.?.objecter ms_handle_connect 0x7fc96c0f7330 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc968ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc968ff9700 1 -- 172.21.15.87:0/1649302779 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c007a40 con 0x7fc96c0f7330 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc960005040 con 0x7fc96c0f7330 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc968ff9700 1 -- 172.21.15.87:0/1649302779 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc95c00f040 con 0x7fc96c0f7330 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc968ff9700 1 -- 172.21.15.87:0/1649302779 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c014a70 con 0x7fc96c0f7330 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc971eb4700 10 client.27882.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96c114eb0 con 0x7fc96c0f7330 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.599+0000 7fc971eb4700 1 librados: init done 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.600+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96c0606d0 con 0x7fc96c0f7330 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc968ff9700 1 -- 172.21.15.87:0/1649302779 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc95c010530 con 0x7fc96c0f7330 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc968ff9700 1 --2- 172.21.15.87:0/1649302779 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc954062c90 0x7fc954065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc968ff9700 1 -- 172.21.15.87:0/1649302779 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(778..778 src has 400..778) v4 ==== 8567+0+0 (secure 0 0 0) 0x7fc95c07d220 con 0x7fc96c0f7330 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc968ff9700 10 client.27882.objecter ms_dispatch 0x7fc96c001550 osd_map(778..778 src has 400..778) v4 2024-02-23T00:43:35.656 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc968ff9700 3 client.27882.objecter handle_osd_map got epochs [778,778] > 0 2024-02-23T00:43:35.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc968ff9700 3 client.27882.objecter handle_osd_map decoding full epoch 778 2024-02-23T00:43:35.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc96affd700 1 --2- 172.21.15.87:0/1649302779 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc954062c90 0x7fc954065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc968ff9700 20 client.27882.objecter dump_active .. 0 homeless 2024-02-23T00:43:35.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc96affd700 1 --2- 172.21.15.87:0/1649302779 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc954062c90 0x7fc954065150 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 rx=0x7fc960009a00 tx=0x7fc96000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.601+0000 7fc968ff9700 10 client.27882.objecter ms_handle_connect 0x7fc954062c90 2024-02-23T00:43:35.657 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.605+0000 7fc968ff9700 1 -- 172.21.15.87:0/1649302779 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc95c050510 con 0x7fc96c0f7330 2024-02-23T00:43:35.786 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.783+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc96c05ee80 con 0x7fc96c0f7330 2024-02-23T00:43:35.842 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.838+0000 7f4ec061b700 1 -- 172.21.15.87:0/2621069660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 msgr2=0x7f4eb8104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.838+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2621069660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 0x7f4eb8104a20 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 rx=0x7f4eb0009c30 tx=0x7f4eb0004c50).stop 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.838+0000 7f4ec061b700 1 -- 172.21.15.87:0/2621069660 shutdown_connections 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.838+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2621069660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 0x7f4eb8104a20 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.838+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2621069660 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4eb80fdbb0 0x7f4eb80fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.838+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2621069660 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4eb80fe510 0x7f4eb80f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.838+0000 7f4ec061b700 1 -- 172.21.15.87:0/2621069660 >> 172.21.15.87:0/2621069660 conn(0x7f4eb80f3800 msgr2=0x7f4eb80f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.838+0000 7f4ec061b700 1 -- 172.21.15.87:0/2621069660 shutdown_connections 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.838+0000 7f4ec061b700 1 -- 172.21.15.87:0/2621069660 wait complete. 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 librados: starting msgr at 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 librados: starting objecter 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 Processor -- start 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 -- start start 2024-02-23T00:43:35.843 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 librados: setting wanted keys 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 librados: calling monclient init 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 0x7f4eb810de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4eb80fdbb0 0x7f4eb810e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4eb80fe510 0x7f4eb8108020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8108510 con 0x7f4eb80f8420 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8108680 con 0x7f4eb80fdbb0 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ec061b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8108840 con 0x7f4eb80fe510 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebe3b7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 0x7f4eb810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebe3b7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 0x7f4eb810de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:38990/0 (socket says 172.21.15.87:38990) 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebebb8700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4eb80fe510 0x7f4eb8108020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebe3b7700 1 -- 172.21.15.87:0/2026936685 learned_addr learned my addr 172.21.15.87:0/2026936685 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebdbb6700 1 --2- 172.21.15.87:0/2026936685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4eb80fdbb0 0x7f4eb810e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebe3b7700 1 -- 172.21.15.87:0/2026936685 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4eb80fe510 msgr2=0x7f4eb8108020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebe3b7700 1 --2- 172.21.15.87:0/2026936685 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4eb80fe510 0x7f4eb8108020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebe3b7700 1 -- 172.21.15.87:0/2026936685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4eb80fdbb0 msgr2=0x7f4eb810e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebe3b7700 1 --2- 172.21.15.87:0/2026936685 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4eb80fdbb0 0x7f4eb810e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebe3b7700 1 -- 172.21.15.87:0/2026936685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb4005040 con 0x7f4eb80f8420 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.839+0000 7f4ebebb8700 1 --2- 172.21.15.87:0/2026936685 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4eb80fe510 0x7f4eb8108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4ebe3b7700 1 --2- 172.21.15.87:0/2026936685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 0x7f4eb810de80 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7f4ea80041d0 tx=0x7f4ea8009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4eaf7fe700 10 client.?.objecter ms_handle_connect 0x7f4eb80f8420 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4eaf7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8007b40 con 0x7f4eb80f8420 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb00097c0 con 0x7f4eb80f8420 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ea800f040 con 0x7f4eb80f8420 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8014730 con 0x7f4eb80f8420 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4ec061b700 10 client.27888.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb8108e80 con 0x7f4eb80f8420 2024-02-23T00:43:35.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4ec061b700 1 librados: init done 2024-02-23T00:43:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.840+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4eb8107110 con 0x7f4eb80f8420 2024-02-23T00:43:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.841+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4ea8009110 con 0x7f4eb80f8420 2024-02-23T00:43:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.841+0000 7f4eaf7fe700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ea4062c90 0x7f4ea4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.847 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(778..778 src has 400..778) v4 ==== 8567+0+0 (secure 0 0 0) 0x7f4ea807d430 con 0x7f4eb80f8420 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ebdbb6700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ea4062c90 0x7f4ea4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 osd_map(778..778 src has 400..778) v4 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4eaf7fe700 3 client.27888.objecter handle_osd_map got epochs [778,778] > 0 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4eaf7fe700 3 client.27888.objecter handle_osd_map decoding full epoch 778 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4eaf7fe700 20 client.27888.objecter dump_active .. 0 homeless 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v778) v1 ==== 54+0+16 (secure 0 0 0) 0x7f4ea8084070 con 0x7f4eb80f8420 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4eaf7fe700 10 client.27888.objecter ms_handle_connect 0x7f4ea4062c90 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ebdbb6700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ea4062c90 0x7f4ea4065150 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7f4eb81097c0 tx=0x7f4eb400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 10 client.27888.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 1 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6801/484081831 conn(0x7f4eb80598f0 0x7f4eb8059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8109500 osd=0 3 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8109500 osd=0 3 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8109500 osd=0 3 2024-02-23T00:43:35.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 1 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8109500 osd=0 3 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 0 1 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8109500 osd=0 4 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 10 client.27888.objecter _send_command 1 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4eb805fa60 con 0x7f4eb80598f0 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ebebb8700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6801/484081831 conn(0x7f4eb80598f0 0x7f4eb8059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4eaf7fe700 10 client.27888.objecter ms_handle_connect 0x7f4eb80598f0 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.842+0000 7f4ebebb8700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6801/484081831 conn(0x7f4eb80598f0 0x7f4eb8059cd0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.843+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4eb805fa60 con 0x7f4eb80598f0 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.843+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.843+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 1 = system:0 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.843+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8109500 osd=0 4 2024-02-23T00:43:35.849 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.843+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 0 1 2024-02-23T00:43:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 10 client.27888.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 2 2024-02-23T00:43:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8109500 osd=0 3 2024-02-23T00:43:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8109500 osd=0 3 2024-02-23T00:43:35.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8109500 osd=0 3 2024-02-23T00:43:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 2 2024-02-23T00:43:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8109500 osd=0 3 2024-02-23T00:43:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 0 2 2024-02-23T00:43:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8109500 osd=0 4 2024-02-23T00:43:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 10 client.27888.objecter _send_command 2 2024-02-23T00:43:35.861 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.858+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4eb805fbd0 con 0x7f4eb80598f0 2024-02-23T00:43:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4eb805fbd0 con 0x7f4eb80598f0 2024-02-23T00:43:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.862 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8109500 osd=0 4 2024-02-23T00:43:35.863 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 0 2 2024-02-23T00:43:35.863 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 10 client.27888.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 3 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6805/685354171 conn(0x7f4eb80ff630 0x7f4eb80273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb80595a0 osd=1 3 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80595a0 osd=1 3 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb80595a0 osd=1 3 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 3 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb80595a0 osd=1 3 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 1 3 2024-02-23T00:43:35.864 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80595a0 osd=1 4 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 10 client.27888.objecter _send_command 3 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4eb8059710 con 0x7f4eb80ff630 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ebe3b7700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6805/685354171 conn(0x7f4eb80ff630 0x7f4eb80273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4eaf7fe700 10 client.27888.objecter ms_handle_connect 0x7f4eb80ff630 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.860+0000 7f4ebe3b7700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6805/685354171 conn(0x7f4eb80ff630 0x7f4eb80273c0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.861+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4eb8059710 con 0x7f4eb80ff630 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.861+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.861+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 3 = system:0 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.861+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb80595a0 osd=1 4 2024-02-23T00:43:35.865 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.861+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 1 3 2024-02-23T00:43:35.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 10 client.27888.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:35.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 4 2024-02-23T00:43:35.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb80595a0 osd=1 3 2024-02-23T00:43:35.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80595a0 osd=1 3 2024-02-23T00:43:35.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb80595a0 osd=1 3 2024-02-23T00:43:35.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 4 2024-02-23T00:43:35.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb80595a0 osd=1 3 2024-02-23T00:43:35.878 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 1 4 2024-02-23T00:43:35.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80595a0 osd=1 4 2024-02-23T00:43:35.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 10 client.27888.objecter _send_command 4 2024-02-23T00:43:35.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.876+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4eb806cf20 con 0x7f4eb80ff630 2024-02-23T00:43:35.879 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.877+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4eb806cf20 con 0x7f4eb80ff630 2024-02-23T00:43:35.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.877+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:35.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.877+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.877+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb80595a0 osd=1 4 2024-02-23T00:43:35.880 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.877+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 1 4 2024-02-23T00:43:35.880 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:43:35.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.877+0000 7f4ec061b700 10 client.27888.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:35.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.877+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 5 2024-02-23T00:43:35.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4eb8028590 0x7f4eb811e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb80fedd0 osd=2 3 2024-02-23T00:43:35.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80fedd0 osd=2 3 2024-02-23T00:43:35.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb80fedd0 osd=2 3 2024-02-23T00:43:35.881 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 5 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb80fedd0 osd=2 3 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 2 5 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80fedd0 osd=2 4 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 10 client.27888.objecter _send_command 5 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4eb80fef40 con 0x7f4eb8028590 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ebdbb6700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4eb8028590 0x7f4eb811e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4eaf7fe700 10 client.27888.objecter ms_handle_connect 0x7f4eb8028590 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4ebdbb6700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4eb8028590 0x7f4eb811e350 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4eb80fef40 con 0x7f4eb8028590 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 5 = system:0 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb80fedd0 osd=2 4 2024-02-23T00:43:35.882 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.878+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 2 5 2024-02-23T00:43:35.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 10 client.27888.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:35.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 6 2024-02-23T00:43:35.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb80fedd0 osd=2 3 2024-02-23T00:43:35.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80fedd0 osd=2 3 2024-02-23T00:43:35.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb80fedd0 osd=2 3 2024-02-23T00:43:35.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 6 2024-02-23T00:43:35.895 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb80fedd0 osd=2 3 2024-02-23T00:43:35.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 2 6 2024-02-23T00:43:35.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80fedd0 osd=2 4 2024-02-23T00:43:35.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 10 client.27888.objecter _send_command 6 2024-02-23T00:43:35.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.893+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4eb8027bb0 con 0x7f4eb8028590 2024-02-23T00:43:35.896 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4eb8027bb0 con 0x7f4eb8028590 2024-02-23T00:43:35.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:35.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb80fedd0 osd=2 4 2024-02-23T00:43:35.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 2 6 2024-02-23T00:43:35.897 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:43:35.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4ec061b700 10 client.27888.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:35.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 7 2024-02-23T00:43:35.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6813/128059790 conn(0x7f4eb811f9c0 0x7f4eb811fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8117580 osd=3 3 2024-02-23T00:43:35.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117580 osd=3 3 2024-02-23T00:43:35.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.894+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8117580 osd=3 3 2024-02-23T00:43:35.898 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 7 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8117580 osd=3 3 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 3 7 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117580 osd=3 4 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4ec061b700 10 client.27888.objecter _send_command 7 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4eb8117da0 con 0x7f4eb811f9c0 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4ebebb8700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6813/128059790 conn(0x7f4eb811f9c0 0x7f4eb811fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4eaf7fe700 10 client.27888.objecter ms_handle_connect 0x7f4eb811f9c0 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4ebebb8700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6813/128059790 conn(0x7f4eb811f9c0 0x7f4eb811fe30 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4eb8117da0 con 0x7f4eb811f9c0 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 7 = system:0 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8117580 osd=3 4 2024-02-23T00:43:35.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.895+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 3 7 2024-02-23T00:43:35.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 10 client.27888.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:35.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 8 2024-02-23T00:43:35.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8117580 osd=3 3 2024-02-23T00:43:35.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117580 osd=3 3 2024-02-23T00:43:35.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8117580 osd=3 3 2024-02-23T00:43:35.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 8 2024-02-23T00:43:35.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8117580 osd=3 3 2024-02-23T00:43:35.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 3 8 2024-02-23T00:43:35.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117580 osd=3 4 2024-02-23T00:43:35.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 10 client.27888.objecter _send_command 8 2024-02-23T00:43:35.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.910+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4eb8120ba0 con 0x7f4eb811f9c0 2024-02-23T00:43:35.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.911+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4eb8120ba0 con 0x7f4eb811f9c0 2024-02-23T00:43:35.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.911+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:35.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.911+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.911+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8117580 osd=3 4 2024-02-23T00:43:35.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.911+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 3 8 2024-02-23T00:43:35.914 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:43:35.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.911+0000 7f4ec061b700 10 client.27888.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:35.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.911+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 9 2024-02-23T00:43:35.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.911+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4eb8123510 0x7f4eb8125900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8120ba0 osd=4 3 2024-02-23T00:43:35.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8120ba0 osd=4 3 2024-02-23T00:43:35.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8120ba0 osd=4 3 2024-02-23T00:43:35.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 9 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8120ba0 osd=4 3 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 4 9 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8120ba0 osd=4 4 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ec061b700 10 client.27888.objecter _send_command 9 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4eb8117230 con 0x7f4eb8123510 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ebe3b7700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4eb8123510 0x7f4eb8125900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4eaf7fe700 10 client.27888.objecter ms_handle_connect 0x7f4eb8123510 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.912+0000 7f4ebe3b7700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4eb8123510 0x7f4eb8125900 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.913+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4eb8117230 con 0x7f4eb8123510 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.913+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.913+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 9 = system:0 2024-02-23T00:43:35.916 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.913+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8120ba0 osd=4 4 2024-02-23T00:43:35.917 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.913+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 4 9 2024-02-23T00:43:35.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 10 client.27888.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 10 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8120ba0 osd=4 3 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8120ba0 osd=4 3 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8120ba0 osd=4 3 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 10 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8120ba0 osd=4 3 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 4 10 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8120ba0 osd=4 4 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 10 client.27888.objecter _send_command 10 2024-02-23T00:43:35.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.927+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4eb8117230 con 0x7f4eb8123510 2024-02-23T00:43:35.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.929+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4eb8117230 con 0x7f4eb8123510 2024-02-23T00:43:35.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.929+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:35.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.929+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.929+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8120ba0 osd=4 4 2024-02-23T00:43:35.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.929+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 4 10 2024-02-23T00:43:35.932 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:43:35.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.929+0000 7f4ec061b700 10 client.27888.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:35.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.929+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 11 2024-02-23T00:43:35.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4eb8126300 0x7f4eb81286f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8117230 osd=5 3 2024-02-23T00:43:35.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117230 osd=5 3 2024-02-23T00:43:35.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8117230 osd=5 3 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 11 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8117230 osd=5 3 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 5 11 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117230 osd=5 4 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 10 client.27888.objecter _send_command 11 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4eb812b340 con 0x7f4eb8126300 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ebdbb6700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4eb8126300 0x7f4eb81286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4ebdbb6700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4eb8126300 0x7f4eb81286f0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.930+0000 7f4eaf7fe700 10 client.27888.objecter ms_handle_connect 0x7f4eb8126300 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.931+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4eb812b340 con 0x7f4eb8126300 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.931+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.931+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 11 = system:0 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.931+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8117230 osd=5 4 2024-02-23T00:43:35.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.931+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 5 11 2024-02-23T00:43:35.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 10 client.27888.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:35.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 12 2024-02-23T00:43:35.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8117230 osd=5 3 2024-02-23T00:43:35.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117230 osd=5 3 2024-02-23T00:43:35.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8117230 osd=5 3 2024-02-23T00:43:35.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 12 2024-02-23T00:43:35.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8117230 osd=5 3 2024-02-23T00:43:35.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 5 12 2024-02-23T00:43:35.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117230 osd=5 4 2024-02-23T00:43:35.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 10 client.27888.objecter _send_command 12 2024-02-23T00:43:35.948 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.945+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4eb8128be0 con 0x7f4eb8126300 2024-02-23T00:43:35.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.947+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4eb8128be0 con 0x7f4eb8126300 2024-02-23T00:43:35.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.947+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:35.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.947+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.947+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8117230 osd=5 4 2024-02-23T00:43:35.950 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.947+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 5 12 2024-02-23T00:43:35.950 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:43:35.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 10 client.27888.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:35.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 13 2024-02-23T00:43:35.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4eb812a8a0 0x7f4eb812ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8128be0 osd=6 3 2024-02-23T00:43:35.951 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128be0 osd=6 3 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8128be0 osd=6 3 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 13 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8128be0 osd=6 3 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 6 13 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128be0 osd=6 4 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 10 client.27888.objecter _send_command 13 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4eb812e6c0 con 0x7f4eb812a8a0 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ebebb8700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4eb812a8a0 0x7f4eb812ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4ebebb8700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4eb812a8a0 0x7f4eb812ad00 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.948+0000 7f4eaf7fe700 10 client.27888.objecter ms_handle_connect 0x7f4eb812a8a0 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.949+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4eb812e6c0 con 0x7f4eb812a8a0 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.949+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:43:35.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.949+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 13 = system:0 2024-02-23T00:43:35.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.949+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8128be0 osd=6 4 2024-02-23T00:43:35.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.949+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 6 13 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 10 client.27888.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 14 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8128be0 osd=6 3 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128be0 osd=6 3 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8128be0 osd=6 3 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 14 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8128be0 osd=6 3 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 6 14 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128be0 osd=6 4 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 10 client.27888.objecter _send_command 14 2024-02-23T00:43:35.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.963+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4eb8128d80 con 0x7f4eb812a8a0 2024-02-23T00:43:35.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.965+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4eb8128d80 con 0x7f4eb812a8a0 2024-02-23T00:43:35.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.965+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:35.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.965+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.965+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8128be0 osd=6 4 2024-02-23T00:43:35.968 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.965+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 6 14 2024-02-23T00:43:35.968 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:43:35.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 10 client.27888.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:35.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 15 2024-02-23T00:43:35.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6812/373787000 conn(0x7f4eb8129400 0x7f4eb8129840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:35.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8128d80 osd=7 3 2024-02-23T00:43:35.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128d80 osd=7 3 2024-02-23T00:43:35.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8128d80 osd=7 3 2024-02-23T00:43:35.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 15 2024-02-23T00:43:35.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8128d80 osd=7 3 2024-02-23T00:43:35.969 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 7 15 2024-02-23T00:43:35.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128d80 osd=7 4 2024-02-23T00:43:35.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 10 client.27888.objecter _send_command 15 2024-02-23T00:43:35.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4eb8129eb0 con 0x7f4eb8129400 2024-02-23T00:43:35.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ebe3b7700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6812/373787000 conn(0x7f4eb8129400 0x7f4eb8129840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:35.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4ebe3b7700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6812/373787000 conn(0x7f4eb8129400 0x7f4eb8129840 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:35.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.966+0000 7f4eaf7fe700 10 client.27888.objecter ms_handle_connect 0x7f4eb8129400 2024-02-23T00:43:35.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.967+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f4eb8129eb0 con 0x7f4eb8129400 2024-02-23T00:43:35.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.967+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:43:35.970 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.967+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 15 = system:0 2024-02-23T00:43:35.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.967+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8128d80 osd=7 4 2024-02-23T00:43:35.971 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.967+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 7 15 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 10 client.27888.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign -1 16 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8128d80 osd=7 3 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128d80 osd=7 3 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 20 client.27888.objecter _get_session s=0x7f4eb8128d80 osd=7 3 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_remove -1 16 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 20 client.27888.objecter get_session s=0x7f4eb8128d80 osd=7 3 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 15 client.27888.objecter _session_command_op_assign 7 16 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128d80 osd=7 4 2024-02-23T00:43:35.984 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 10 client.27888.objecter _send_command 16 2024-02-23T00:43:35.985 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.981+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4eb8129eb0 con 0x7f4eb8129400 2024-02-23T00:43:35.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4eaf7fe700 1 -- 172.21.15.87:0/2026936685 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f4eb8129eb0 con 0x7f4eb8129400 2024-02-23T00:43:35.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4eaf7fe700 10 client.27888.objecter ms_dispatch 0x7f4eb8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:43:35.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4eaf7fe700 10 client.27888.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4eaf7fe700 20 client.27888.objecter put_session s=0x7f4eb8128d80 osd=7 4 2024-02-23T00:43:35.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4eaf7fe700 15 client.27888.objecter _session_command_op_remove 7 16 2024-02-23T00:43:35.987 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:43:35.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 librados: watch_flush enter 2024-02-23T00:43:35.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 librados: watch_flush exit 2024-02-23T00:43:35.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 client.27888.objecter close_session for osd.0 2024-02-23T00:43:35.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6801/484081831 conn(0x7f4eb80598f0 msgr2=0x7f4eb8059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6801/484081831 conn(0x7f4eb80598f0 0x7f4eb8059cd0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8109500 osd=0 1 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 client.27888.objecter close_session for osd.1 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6805/685354171 conn(0x7f4eb80ff630 msgr2=0x7f4eb80273c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6805/685354171 conn(0x7f4eb80ff630 0x7f4eb80273c0 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80595a0 osd=1 1 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 client.27888.objecter close_session for osd.2 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4eb8028590 msgr2=0x7f4eb811e350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6809/3605394228 conn(0x7f4eb8028590 0x7f4eb811e350 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb80fedd0 osd=2 1 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 client.27888.objecter close_session for osd.3 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6813/128059790 conn(0x7f4eb811f9c0 msgr2=0x7f4eb811fe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6813/128059790 conn(0x7f4eb811f9c0 0x7f4eb811fe30 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ebebb8700 1 -- 172.21.15.87:0/2026936685 reap_dead start 2024-02-23T00:43:35.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117580 osd=3 1 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 client.27888.objecter close_session for osd.4 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4eb8123510 msgr2=0x7f4eb8125900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6800/4175844757 conn(0x7f4eb8123510 0x7f4eb8125900 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8120ba0 osd=4 1 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 client.27888.objecter close_session for osd.5 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4eb8126300 msgr2=0x7f4eb81286f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6804/2550013795 conn(0x7f4eb8126300 0x7f4eb81286f0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8117230 osd=5 1 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 client.27888.objecter close_session for osd.6 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4eb812a8a0 msgr2=0x7f4eb812ad00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6808/2509408498 conn(0x7f4eb812a8a0 0x7f4eb812ad00 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128be0 osd=6 1 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 client.27888.objecter close_session for osd.7 2024-02-23T00:43:35.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6812/373787000 conn(0x7f4eb8129400 msgr2=0x7f4eb8129840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.171:6812/373787000 conn(0x7f4eb8129400 0x7f4eb8129840 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 20 client.27888.objecter put_session s=0x7f4eb8128d80 osd=7 1 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 20 client.27888.objecter shutdown clearing up homeless session... 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 10 client.27888.objecter successfully canceled tick 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ebebb8700 1 -- 172.21.15.87:0/2026936685 reap_dead start 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ea4062c90 msgr2=0x7f4ea4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ea4062c90 0x7f4ea4065150 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7f4eb81097c0 tx=0x7f4eb400b040).stop 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 msgr2=0x7f4eb810de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 0x7f4eb810de80 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 rx=0x7f4ea80041d0 tx=0x7f4ea8009f60).stop 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 shutdown_connections 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4eb80f8420 0x7f4eb810de80 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 --2- 172.21.15.87:0/2026936685 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4ea4062c90 0x7f4ea4065150 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 >> 172.21.15.87:0/2026936685 conn(0x7f4eb80f3800 msgr2=0x7f4eb80f5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:35.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 shutdown_connections 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.984+0000 7f4ec061b700 1 -- 172.21.15.87:0/2026936685 wait complete. 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:35.985+0000 7f4ec061b700 1 librados: shutdown 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:43:35.992 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:43:36.257 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.255+0000 7fc968ff9700 1 -- 172.21.15.87:0/1649302779 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v779) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc95c022090 con 0x7fc96c0f7330 2024-02-23T00:43:36.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.259+0000 7fc971eb4700 10 librados: watch_flush enter 2024-02-23T00:43:36.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.259+0000 7fc971eb4700 10 librados: watch_flush exit 2024-02-23T00:43:36.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.259+0000 7fc971eb4700 20 client.27882.objecter shutdown clearing up homeless session... 2024-02-23T00:43:36.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.259+0000 7fc971eb4700 10 client.27882.objecter successfully canceled tick 2024-02-23T00:43:36.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.259+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc954062c90 msgr2=0x7fc954065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:36.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.259+0000 7fc971eb4700 1 --2- 172.21.15.87:0/1649302779 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc954062c90 0x7fc954065150 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 rx=0x7fc960009a00 tx=0x7fc96000b040).stop 2024-02-23T00:43:36.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.259+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc96c0f7330 msgr2=0x7fc96c10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:36.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.259+0000 7fc971eb4700 1 --2- 172.21.15.87:0/1649302779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc96c0f7330 0x7fc96c10e080 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 rx=0x7fc95c004920 tx=0x7fc95c009cb0).stop 2024-02-23T00:43:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.261+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 shutdown_connections 2024-02-23T00:43:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.261+0000 7fc971eb4700 1 --2- 172.21.15.87:0/1649302779 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc954062c90 0x7fc954065150 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.261+0000 7fc971eb4700 1 --2- 172.21.15.87:0/1649302779 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc96c0fd420 0x7fc96c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.261+0000 7fc971eb4700 1 --2- 172.21.15.87:0/1649302779 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc96c0f7330 0x7fc96c10e080 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.261+0000 7fc971eb4700 1 --2- 172.21.15.87:0/1649302779 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc96c0fcac0 0x7fc96c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.261+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 >> 172.21.15.87:0/1649302779 conn(0x7fc96c0f26f0 msgr2=0x7fc96c0f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.261+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 shutdown_connections 2024-02-23T00:43:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.261+0000 7fc971eb4700 1 -- 172.21.15.87:0/1649302779 wait complete. 2024-02-23T00:43:36.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:36.261+0000 7fc971eb4700 1 librados: shutdown 2024-02-23T00:43:36.265 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is set 2024-02-23T00:43:36.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:35.254153+0000 mon.a (mon.0) 3833 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-136"}]': finished 2024-02-23T00:43:36.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:35.254227+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-02-23T00:43:36.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:35.255839+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-136", "mode": "writeback"}]: dispatch 2024-02-23T00:43:36.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:35.785198+0000 mon.a (mon.0) 3836 : audit [INF] from='client.? 172.21.15.87:0/1649302779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:36.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:35.842996+0000 mon.a (mon.0) 3837 : audit [DBG] from='client.? 172.21.15.87:0/2026936685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:36.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:36.250829+0000 mon.a (mon.0) 3838 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:43:36.519 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:36.250868+0000 mon.a (mon.0) 3839 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:43:36.519 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:35.254153+0000 mon.a (mon.0) 3833 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-136"}]': finished 2024-02-23T00:43:36.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:35.254227+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-02-23T00:43:36.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:35.255839+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-136", "mode": "writeback"}]: dispatch 2024-02-23T00:43:36.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:35.785198+0000 mon.a (mon.0) 3836 : audit [INF] from='client.? 172.21.15.87:0/1649302779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:36.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:35.842996+0000 mon.a (mon.0) 3837 : audit [DBG] from='client.? 172.21.15.87:0/2026936685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:36.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:36.250829+0000 mon.a (mon.0) 3838 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:43:36.520 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:36.250868+0000 mon.a (mon.0) 3839 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:43:36.520 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:36.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:36.636 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:36.663 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:35.254153+0000 mon.a (mon.0) 3833 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi087-82091-101", "overlaypool": "test-rados-api-smithi087-82091-136"}]': finished 2024-02-23T00:43:36.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:35.254227+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-02-23T00:43:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:35.255839+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-136", "mode": "writeback"}]: dispatch 2024-02-23T00:43:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:35.785198+0000 mon.a (mon.0) 3836 : audit [INF] from='client.? 172.21.15.87:0/1649302779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:35.842996+0000 mon.a (mon.0) 3837 : audit [DBG] from='client.? 172.21.15.87:0/2026936685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:36.250829+0000 mon.a (mon.0) 3838 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-23T00:43:36.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:36.250868+0000 mon.a (mon.0) 3839 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:43:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:35.801770+0000 mgr.y (mgr.24437) 661 : cluster [DBG] pgmap v1193: 204 pgs: 1 active+remapped+backfill_toofull, 7 creating+peering, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s; 1/813 objects misplaced (0.123%) 2024-02-23T00:43:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:36.256102+0000 mon.a (mon.0) 3840 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-136", "mode": "writeback"}]': finished 2024-02-23T00:43:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:36.256174+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? 172.21.15.87:0/1649302779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:43:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:36.256211+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-02-23T00:43:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:36.302312+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:43:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:35.801770+0000 mgr.y (mgr.24437) 661 : cluster [DBG] pgmap v1193: 204 pgs: 1 active+remapped+backfill_toofull, 7 creating+peering, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s; 1/813 objects misplaced (0.123%) 2024-02-23T00:43:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:36.256102+0000 mon.a (mon.0) 3840 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-136", "mode": "writeback"}]': finished 2024-02-23T00:43:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:36.256174+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? 172.21.15.87:0/1649302779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:43:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:36.256211+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-02-23T00:43:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:36.302312+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:43:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:35.801770+0000 mgr.y (mgr.24437) 661 : cluster [DBG] pgmap v1193: 204 pgs: 1 active+remapped+backfill_toofull, 7 creating+peering, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 2 op/s; 1/813 objects misplaced (0.123%) 2024-02-23T00:43:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:36.256102+0000 mon.a (mon.0) 3840 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi087-82091-136", "mode": "writeback"}]': finished 2024-02-23T00:43:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:36.256174+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? 172.21.15.87:0/1649302779' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-23T00:43:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:36.256211+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-02-23T00:43:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:36.302312+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-23T00:43:37.969 INFO:tasks.thrashosds.thrasher:j is {'epoch': 775, 'fsid': '66db207c-d1e1-11ee-95c0-87774f69a715', 'created': '2024-02-23T00:21:28.861318+0000', 'modified': '2024-02-23T00:43:32.240009+0000', 'last_up_change': '2024-02-23T00:41:18.209057+0000', 'last_in_change': '2024-02-23T00:41:36.208573+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 114, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 252, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-23T00:25:09.208309+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-23T00:27:15.562186+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-23T00:27:15.989206+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 23, 'pg_placement_num': 23, 'pg_placement_num_target': 23, 'pg_num_target': 23, 'pg_num_pending': 23, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '738', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '736', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-23T00:27:17.282440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '4.16', 'ready_epoch': 371, 'last_epoch_started': 370, 'last_epoch_clean': 370, 'source_version': "63'21", 'target_version': "62'12"}, 'last_change': '532', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '371', 'last_force_op_resend_preluminous': '481', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-23T00:27:19.262164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '5.20', 'ready_epoch': 641, 'last_epoch_started': 640, 'last_epoch_clean': 640, 'source_version': "562'12", 'target_version': "330'9"}, 'last_change': '641', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '641', 'last_force_op_resend_preluminous': '639', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-23T00:27:21.319951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 12, 'pg_placement_num': 12, 'pg_placement_num_target': 12, 'pg_num_target': 12, 'pg_num_pending': 12, 'last_pg_merge_meta': {'source_pgid': '6.c', 'ready_epoch': 665, 'last_epoch_started': 664, 'last_epoch_clean': 664, 'source_version': "78'2", 'target_version': "0'0"}, 'last_change': '665', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '665', 'last_force_op_resend_preluminous': '663', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 22, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-23T00:30:46.071806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 234, 'pool_name': 'test-rados-api-smithi087-82091-101', 'create_time': '2024-02-23T00:40:15.435168+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '773', 'last_force_op_resend': '772', 'last_force_op_resend_prenautilus': '772', 'last_force_op_resend_preluminous': '772', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 10, 'snap_epoch': 766, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-test-rados-api-smithi087-82091-101', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 252, 'pool_name': 'test-rados-api-smithi087-82091-136', 'create_time': '2024-02-23T00:43:32.130424+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '775', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': 'e3549bad-2891-47b9-bf77-132cadc20bf7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 424, 'up_from': 659, 'up_thru': 764, 'down_at': 425, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6801', 'nonce': 484081831}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6802', 'nonce': 484081831}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6804', 'nonce': 484081831}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6803', 'nonce': 484081831}]}, 'public_addr': '172.21.15.87:6801/484081831', 'cluster_addr': '172.21.15.87:6802/484081831', 'heartbeat_back_addr': '172.21.15.87:6804/484081831', 'heartbeat_front_addr': '172.21.15.87:6803/484081831', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '85525ce8-38d5-4840-8057-547cb176c859', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6805', 'nonce': 685354171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6806', 'nonce': 685354171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6808', 'nonce': 685354171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6807', 'nonce': 685354171}]}, 'public_addr': '172.21.15.87:6805/685354171', 'cluster_addr': '172.21.15.87:6806/685354171', 'heartbeat_back_addr': '172.21.15.87:6808/685354171', 'heartbeat_front_addr': '172.21.15.87:6807/685354171', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '48ea64b6-ed15-48d2-b0ed-aa4053938aaa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6809', 'nonce': 3605394228}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6810', 'nonce': 3605394228}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6812', 'nonce': 3605394228}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6811', 'nonce': 3605394228}]}, 'public_addr': '172.21.15.87:6809/3605394228', 'cluster_addr': '172.21.15.87:6810/3605394228', 'heartbeat_back_addr': '172.21.15.87:6812/3605394228', 'heartbeat_front_addr': '172.21.15.87:6811/3605394228', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'cb8705aa-e114-4dfd-ac9d-ec12f85f877a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 769, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6813', 'nonce': 128059790}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6814', 'nonce': 128059790}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6816', 'nonce': 128059790}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.87:6815', 'nonce': 128059790}]}, 'public_addr': '172.21.15.87:6813/128059790', 'cluster_addr': '172.21.15.87:6814/128059790', 'heartbeat_back_addr': '172.21.15.87:6816/128059790', 'heartbeat_front_addr': '172.21.15.87:6815/128059790', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'c3734a04-7343-4251-9b5b-bedb9e8b882c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5775604248046875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6800', 'nonce': 4175844757}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6801', 'nonce': 4175844757}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6803', 'nonce': 4175844757}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6802', 'nonce': 4175844757}]}, 'public_addr': '172.21.15.171:6800/4175844757', 'cluster_addr': '172.21.15.171:6801/4175844757', 'heartbeat_back_addr': '172.21.15.171:6803/4175844757', 'heartbeat_front_addr': '172.21.15.171:6802/4175844757', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fe86232d-96b9-4cbf-9cac-373286fa103d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.8517608642578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6804', 'nonce': 2550013795}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6805', 'nonce': 2550013795}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6807', 'nonce': 2550013795}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6806', 'nonce': 2550013795}]}, 'public_addr': '172.21.15.171:6804/2550013795', 'cluster_addr': '172.21.15.171:6805/2550013795', 'heartbeat_back_addr': '172.21.15.171:6807/2550013795', 'heartbeat_front_addr': '172.21.15.171:6806/2550013795', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'bfc5dd69-5cfb-409f-985f-83ee0dd4dc01', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 44, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6808', 'nonce': 2509408498}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6809', 'nonce': 2509408498}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6811', 'nonce': 2509408498}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6810', 'nonce': 2509408498}]}, 'public_addr': '172.21.15.171:6808/2509408498', 'cluster_addr': '172.21.15.171:6809/2509408498', 'heartbeat_back_addr': '172.21.15.171:6811/2509408498', 'heartbeat_front_addr': '172.21.15.171:6810/2509408498', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '29ddcf97-369a-4a2c-b3e2-142e84ccf939', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 764, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6812', 'nonce': 373787000}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6813', 'nonce': 373787000}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6815', 'nonce': 373787000}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.171:6814', 'nonce': 373787000}]}, 'public_addr': '172.21.15.171:6812/373787000', 'cluster_addr': '172.21.15.171:6813/373787000', 'heartbeat_back_addr': '172.21.15.171:6815/373787000', 'heartbeat_front_addr': '172.21.15.171:6814/373787000', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:33.795687+0000', 'dead_epoch': 424}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:13.973488+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:43:10.808451+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:32.316583+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:31.854915+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:43:14.040508+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:29.973812+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-23T00:42:26.420812+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '2.1c', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '22.1a', 'mappings': [{'from': 7, 'to': 5}]}], 'pg_temp': [{'pgid': '5.3', 'osds': [3, 6, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.87:6800/1121018082': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/2138166229': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/27794119': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/1675126043': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:0/3767016525': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/969918521': '2024-02-24T00:22:22.047081+0000', '172.21.15.87:0/795095363': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2273151327': '2024-02-24T00:27:39.443000+0000', '172.21.15.87:6800/1685406414': '2024-02-24T00:21:58.144574+0000', '172.21.15.87:0/2093733106': '2024-02-24T00:21:58.144574+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-test-rados-api-smithi087-82091-101': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 21, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 61, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 63, 'snaps': [{'begin': 29, 'length': 1}]}, {'pool': 78, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 86, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 93, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 95, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 125, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 133, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 177, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 241, 'snaps': [{'begin': 6, 'length': 1}]}, {'pool': 245, 'snaps': [{'begin': 9, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-23T00:43:37.969 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:43:38.212 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:38.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:38.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:38.317 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:38.349 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:38.370 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:38.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:38.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:38.499 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:37.269188+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:43:38.499 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:37.269270+0000 mon.a (mon.0) 3845 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-02-23T00:43:38.499 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:37.272250+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:43:38.499 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:38.272971+0000 mon.a (mon.0) 3847 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:43:38.499 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:38.273031+0000 mon.a (mon.0) 3848 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-02-23T00:43:38.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:38.273953+0000 mon.a (mon.0) 3849 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:43:38.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:37.269188+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:43:38.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:37.269270+0000 mon.a (mon.0) 3845 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-02-23T00:43:38.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:37.272250+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:43:38.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:38.272971+0000 mon.a (mon.0) 3847 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:43:38.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:38.273031+0000 mon.a (mon.0) 3848 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-02-23T00:43:38.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:38.273953+0000 mon.a (mon.0) 3849 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:43:38.501 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:37.269188+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_count","val": "2"}]': finished 2024-02-23T00:43:38.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:37.269270+0000 mon.a (mon.0) 3845 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-02-23T00:43:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:37.272250+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-23T00:43:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:38.272971+0000 mon.a (mon.0) 3847 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_period","val": "600"}]': finished 2024-02-23T00:43:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:38.273031+0000 mon.a (mon.0) 3848 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-02-23T00:43:38.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:38.273953+0000 mon.a (mon.0) 3849 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-23T00:43:38.849 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:38.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:38.952 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:38.979 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:39.105 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:43:39.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:37.802379+0000 mgr.y (mgr.24437) 662 : cluster [DBG] pgmap v1196: 204 pgs: 1 active+remapped+backfill_toofull, 7 creating+peering, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 1/813 objects misplaced (0.123%) 2024-02-23T00:43:39.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:39.267471+0000 mon.a (mon.0) 3850 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:43:39.405 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:39.271803+0000 mon.a (mon.0) 3851 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:43:39.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:39.271893+0000 mon.a (mon.0) 3852 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-02-23T00:43:39.406 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:39.272864+0000 mon.a (mon.0) 3853 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:43:39.406 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:37.802379+0000 mgr.y (mgr.24437) 662 : cluster [DBG] pgmap v1196: 204 pgs: 1 active+remapped+backfill_toofull, 7 creating+peering, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 1/813 objects misplaced (0.123%) 2024-02-23T00:43:39.406 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:39.267471+0000 mon.a (mon.0) 3850 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:43:39.406 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:39.271803+0000 mon.a (mon.0) 3851 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:43:39.406 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:39.271893+0000 mon.a (mon.0) 3852 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-02-23T00:43:39.406 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:39.272864+0000 mon.a (mon.0) 3853 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:43:39.445 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:39.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:39.553 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:39.583 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:37.802379+0000 mgr.y (mgr.24437) 662 : cluster [DBG] pgmap v1196: 204 pgs: 1 active+remapped+backfill_toofull, 7 creating+peering, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 1/813 objects misplaced (0.123%) 2024-02-23T00:43:39.583 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:39.267471+0000 mon.a (mon.0) 3850 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-23T00:43:39.583 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:39.271803+0000 mon.a (mon.0) 3851 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-23T00:43:39.583 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:39.271893+0000 mon.a (mon.0) 3852 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-02-23T00:43:39.583 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:39.272864+0000 mon.a (mon.0) 3853 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-23T00:43:39.586 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:39.606 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:39.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:39.710 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:39.740 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:39.760 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:39.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:39.864 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:39.892 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:39.911 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:39.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:40.015 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:40.043 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:40.062 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:40.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:40.152 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:43:40.152 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:43:40.152 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:43:40.152 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:43:40.152 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.152 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.4.31:55 216.9 216:939aa6ec:test-rados-api-smithi087-82091-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e451)", 2024-02-23T00:43:40.152 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:37:17.129723+0000", 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 383.01892600999997, 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.057627593999999997, 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129723+0000", 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129723+0000", 2024-02-23T00:43:40.153 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129720+0000", 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2629999999999999e-05 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129733+0000", 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9596e-05 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.154 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129752+0000", 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8230000000000006e-06 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129762+0000", 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4041000000000005e-05 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.155 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129836+0000", 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000126854 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.129963+0000", 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00019699 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 1,4", 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130160+0000", 2024-02-23T00:43:40.156 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00052119500000000001 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130681+0000", 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00031204600000000001 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.130993+0000", 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.056278791000000002 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.157 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187272+0000", 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3816000000000002e-05 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187316+0000", 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4229e-05 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:37:17.187350+0000", 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.057627593999999997 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.158 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.6.44:88 63.6 63:6735b1f6:test-rados-api-smithi087-82091-78::0b0692abe73bf7a995bf44f5e9951ddb037acf28:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e483)", 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:38:20.305065+0000", 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 319.843583604, 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024404019999999998, 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305065+0000", 2024-02-23T00:43:40.159 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305065+0000", 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305063+0000", 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6510000000000001e-06 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.160 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305071+0000", 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6728e-05 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305088+0000", 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5950000000000001e-06 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.161 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305095+0000", 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1520000000000002e-05 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305137+0000", 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4927999999999996e-05 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305232+0000", 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00017876200000000001 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.162 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 6", 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305410+0000", 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000352796 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.305763+0000", 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.023542632000000001 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.163 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.329306+0000", 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4443000000000006e-05 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.329370+0000", 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.8653999999999994e-05 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:38:20.329469+0000", 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.024404019999999998 2024-02-23T00:43:40.164 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27567.0:544 239.1f e653/649 239:fe98049f:test-rados-api-smithi087-82091-111::foo:head v 653'4, mlcod=653'2)", 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:12.243408+0000", 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 147.90524068600001, 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.063953162999999993, 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.165 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243408+0000", 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243408+0000", 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243407+0000", 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9549999999999992e-06 2024-02-23T00:43:40.166 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243415+0000", 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5590000000000006e-06 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243424+0000", 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.929e-06 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.167 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243428+0000", 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3080999999999997e-05 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243461+0000", 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6864999999999999e-05 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.243478+0000", 2024-02-23T00:43:40.168 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.063840279999999999 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.307318+0000", 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3244999999999997e-05 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:12.307361+0000", 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.063953162999999993 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.169 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2079 5.d 5:b4812045:::notify.4:head [watch ping cookie 94532396651008] snapc 0=[] ondisk+write+known_if_redirected e666)", 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:30.496638+0000", 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 129.65201051, 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001277294, 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:43:40.170 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2079 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:30.496638+0000", 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:30.496638+0000", 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999962 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.171 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:30.496634+0000", 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9272000000000001e-05 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:30.496654+0000", 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.9067000000000001e-05 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.172 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:30.496683+0000", 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2483000000000001e-05 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:30.496695+0000", 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0010820789999999999 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:30.497777+0000", 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0210999999999996e-05 2024-02-23T00:43:40.173 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:30.497847+0000", 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7998999999999997e-05 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:30.497915+0000", 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001277294 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.174 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2119 5.d 5:b4812045:::notify.4:head [watch ping cookie 94532396651008] snapc 0=[] ondisk+write+known_if_redirected e686)", 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:50.500243+0000", 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 109.6484056, 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00065678799999999995, 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2119 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.175 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.500243+0000", 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.500243+0000", 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999962 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.176 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.500239+0000", 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9162999999999999e-05 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.500258+0000", 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.1009e-05 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.500299+0000", 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9847000000000002e-05 2024-02-23T00:43:40.177 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.500319+0000", 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000368833 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.500688+0000", 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4055999999999996e-05 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.178 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.500772+0000", 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012768999999999999 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:50.500900+0000", 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00065678799999999995 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.179 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27567.0:1528 245.6 e707/701 245:6586c251:test-rados-api-smithi087-82091-123::foo:head v 707'6, mlcod=705'2)", 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:09.236568+0000", 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 90.912080535000001, 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00098692700000000007, 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.236568+0000", 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.180 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.236568+0000", 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.236568+0000", 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5920000000000001e-06 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.181 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.236576+0000", 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6342e-05 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.236593+0000", 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3919999999999997e-06 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.236600+0000", 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9893999999999999e-05 2024-02-23T00:43:40.182 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.236660+0000", 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3703999999999999e-05 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.236684+0000", 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00084605800000000001 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.183 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.237530+0000", 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.535e-05 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:09.237555+0000", 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00098692700000000007 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.184 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(234.6s1 710/597 ECSubWrite(tid=1660, reqid=osd.6.0:1660, at_version=710'16, trim_to=0'0, roll_forward_to=707'14))", 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:12.251438+0000", 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 87.897210903000001, 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0044985260000000001, 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.251438+0000", 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.185 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.251438+0000", 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967296 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.251437+0000", 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5511e-05 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.251463+0000", 2024-02-23T00:43:40.186 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2399999999999996e-06 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.251472+0000", 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5070000000000003e-06 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.251480+0000", 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0012512669999999999 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.187 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.252731+0000", 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0261000000000001e-05 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.252741+0000", 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0031691039999999998 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:43:40.188 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255910+0000", 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.5854999999999999e-05 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255936+0000", 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0044985260000000001 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:1583 245.8 245:10000000:%01:::head [pgnls start_epoch 710] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e710)", 2024-02-23T00:43:40.189 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:12.255202+0000", 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 87.893446913000005, 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00053852299999999995, 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1583 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255202+0000", 2024-02-23T00:43:40.190 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255202+0000", 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255200+0000", 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.3090000000000001e-06 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.191 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255209+0000", 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3042e-05 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255222+0000", 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5779999999999999e-06 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.192 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255231+0000", 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037660900000000003 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255607+0000", 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.6186000000000002e-05 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255643+0000", 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6835000000000004e-05 2024-02-23T00:43:40.193 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.255740+0000", 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00053852299999999995 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(234.6s1 710/597 ECSubWrite(tid=1661, reqid=osd.6.0:1660, at_version=0'0, trim_to=0'0, roll_forward_to=710'16))", 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:12.256424+0000", 2024-02-23T00:43:40.194 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 87.892224185000003, 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020049870000000002, 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256424+0000", 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256424+0000", 2024-02-23T00:43:40.195 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256424+0000", 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5877999999999999e-05 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256439+0000", 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.092e-05 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.196 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256450+0000", 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8210000000000004e-06 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256457+0000", 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000122867 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.197 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256580+0000", 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9199999999999999e-06 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.256590+0000", 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0018170020000000001 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.258407+0000", 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.2446000000000002e-05 2024-02-23T00:43:40.198 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.258429+0000", 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0020049870000000002 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.7.0:594 245.6 e710/701 245:6586c251:test-rados-api-smithi087-82091-123::foo:9 v 710'11, mlcod=707'8)", 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:12.257863+0000", 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 87.890785957000006, 2024-02-23T00:43:40.199 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00060131599999999996, 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.257863+0000", 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.257863+0000", 2024-02-23T00:43:40.200 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.257862+0000", 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0450000000000002e-06 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.257866+0000", 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.9120000000000004e-06 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.201 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.257875+0000", 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5270000000000004e-06 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.257882+0000", 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8769e-05 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.257931+0000", 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6505000000000001e-05 2024-02-23T00:43:40.202 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.257957+0000", 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00047583599999999999 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.258433+0000", 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.0994000000000002e-05 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.203 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:12.258464+0000", 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00060131599999999996 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2166 5.d 5:b4812045:::notify.4:head [watch ping cookie 94532396651008] snapc 0=[] ondisk+write+known_if_redirected e708)", 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:15.505557+0000", 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 84.643091712, 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001094544, 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.204 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2166 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:15.505557+0000", 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.205 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:15.505557+0000", 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:15.505555+0000", 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4429999999999992e-06 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.206 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:15.505564+0000", 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9000000000000001e-05 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:15.505583+0000", 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3319999999999992e-06 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:15.505591+0000", 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00087461400000000001 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.207 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:15.506466+0000", 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.9736999999999999e-05 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:15.506546+0000", 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000105615 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.208 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:15.506651+0000", 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001094544 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:1794 247.14 247:2b095bf0:test-rados-api-smithi087-82091-127::foo:head [writefull 0~4000008] snapc 0=[] ondisk+write+known_if_redirected e726)", 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.228628+0000", 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 71.920021082000005, 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027448127999999999, 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:43:40.209 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1794 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.228628+0000", 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.228628+0000", 2024-02-23T00:43:40.210 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.228625+0000", 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001546696 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.230172+0000", 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3445e-05 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.211 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.230195+0000", 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6225e-05 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.230222+0000", 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0470999999999997e-05 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.230292+0000", 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00019688299999999999 2024-02-23T00:43:40.212 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for blocked object", 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.230489+0000", 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0032835099999999999 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.233772+0000", 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2772999999999998e-05 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.213 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.233825+0000", 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2207999999999995e-05 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.233917+0000", 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0046284020000000002 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.238546+0000", 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0087450219999999999 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.214 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.247291+0000", 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0087151959999999997 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256006+0000", 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0809999999999997e-05 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256047+0000", 2024-02-23T00:43:40.215 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.8806000000000002e-05 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256076+0000", 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.027448127999999999 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop_reply(osd.2.0:529 247.14 e726/722 ondisk, result = 0)", 2024-02-23T00:43:40.216 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.234567+0000", 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 71.914081722000006, 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002533623, 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234567+0000", 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.217 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234567+0000", 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234566+0000", 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.0539999999999995e-06 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234574+0000", 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6320999999999998e-05 2024-02-23T00:43:40.218 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234590+0000", 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0726e-05 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234601+0000", 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0024698300000000001 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.219 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.237071+0000", 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2140000000000001e-05 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.237083+0000", 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.7748999999999999e-05 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.237100+0000", 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002533623 2024-02-23T00:43:40.220 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop_reply(osd.2.0:529 247.14 e725/722 ondisk, result = 0)", 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.234817+0000", 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 71.913832033000006, 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002410382, 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.221 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234817+0000", 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234817+0000", 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234816+0000", 2024-02-23T00:43:40.222 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4160000000000003e-06 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234821+0000", 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.9340000000000001e-06 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234831+0000", 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2829999999999998e-06 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.223 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.234837+0000", 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0023145930000000002 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.237152+0000", 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.5990000000000001e-06 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.224 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.237158+0000", 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9292e-05 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.237227+0000", 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002410382 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.225 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:1795 247.14 247:2b095bf0:test-rados-api-smithi087-82091-127::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected e726)", 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.256296+0000", 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 71.892353094000001, 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018098091, 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1795 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.226 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256296+0000", 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256296+0000", 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.227 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256294+0000", 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.9160000000000007e-06 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256303+0000", 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5699e-05 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256318+0000", 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3629999999999996e-06 2024-02-23T00:43:40.228 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256326+0000", 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.5537999999999998e-05 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256361+0000", 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000115096 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.229 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256476+0000", 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017751728000000001 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.274228+0000", 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4832000000000003e-05 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.274323+0000", 2024-02-23T00:43:40.230 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0745000000000001e-05 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.274394+0000", 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.018098091 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.3.26:59 247.14 247:2b095bf0:test-rados-api-smithi087-82091-127::foo:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+flush+map_snap_clone+known_if_redirected e726)", 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.256869+0000", 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 71.891780085999997, 2024-02-23T00:43:40.231 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0047809419999999998, 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256869+0000", 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.232 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256869+0000", 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256867+0000", 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.2859999999999999e-06 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256875+0000", 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3658e-05 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.233 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256889+0000", 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3139999999999998e-06 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256896+0000", 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2880999999999998e-05 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256939+0000", 2024-02-23T00:43:40.234 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3746999999999998e-05 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.257003+0000", 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0046463820000000001 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.261649+0000", 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0047809419999999998 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:1829 247.14 247:2b095bf0:test-rados-api-smithi087-82091-127::foo:head [cache-evict] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e726)", 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.288867+0000", 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 71.859781240000004, 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001125287, 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1829 2024-02-23T00:43:40.236 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.288867+0000", 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.288867+0000", 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.237 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.288867+0000", 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.6119999999999999e-06 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.288870+0000", 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3180000000000003e-06 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.288877+0000", 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.4290000000000001e-06 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.238 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.288880+0000", 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.9909000000000001e-05 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.288910+0000", 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4803999999999999e-05 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.239 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.288965+0000", 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.3875000000000007e-05 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for subops from 3,7", 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.289039+0000", 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037420500000000001 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op_commit", 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.289413+0000", 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00036253099999999998 2024-02-23T00:43:40.240 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.289775+0000", 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016598700000000001 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_commit_rec", 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.289941+0000", 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.9694999999999999e-05 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.241 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.289971+0000", 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.1610000000000001e-05 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.289993+0000", 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001125287 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.242 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2274 5.d 5:b4812045:::notify.4:head [watch ping cookie 94532396651008] snapc 0=[] ondisk+write+known_if_redirected e759)", 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:43:15.516599+0000", 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 24.632049417000001, 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00075765800000000003, 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2274 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.243 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:15.516599+0000", 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:15.516599+0000", 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:15.516597+0000", 2024-02-23T00:43:40.244 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0268000000000001e-05 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:15.516607+0000", 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9446e-05 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:15.516627+0000", 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.8259999999999993e-06 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.245 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:15.516634+0000", 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00049540200000000004 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:15.517130+0000", 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6397999999999994e-05 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.246 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:15.517226+0000", 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013068300000000001 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:15.517357+0000", 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00075765800000000003 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDScrubReserve(2.e GRANT e766)", 2024-02-23T00:43:40.247 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:43:18.436679+0000", 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 21.711969133, 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016502820000000001, 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.436679+0000", 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.248 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.436679+0000", 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.436678+0000", 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.899e-06 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.436683+0000", 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0278000000000001e-05 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.249 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.436694+0000", 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.8989999999999998e-06 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.436700+0000", 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001611024 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.438311+0000", 2024-02-23T00:43:40.250 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.046e-05 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.438321+0000", 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2860000000000001e-06 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.438322+0000", 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4059999999999999e-06 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.251 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:18.438330+0000", 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0016502820000000001 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(osd.6.0:1686 252.0 e779/775 252:0662e81e:test-rados-api-smithi087-82091-137::baz:head v 779'1, mlcod=0'0)", 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:43:36.283212+0000", 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 3.8654364750000001, 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00053698500000000002, 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:43:40.252 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283212+0000", 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283212+0000", 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.253 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283211+0000", 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2310000000000001e-05 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283223+0000", 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.6159000000000001e-05 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283250+0000", 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6736e-05 2024-02-23T00:43:40.254 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283266+0000", 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000108193 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283375+0000", 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0999e-05 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:43:40.255 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283396+0000", 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00032939500000000001 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283725+0000", 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.4199e-05 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:36.283749+0000", 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00053698500000000002 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.256 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.257 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.257 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:43:40.257 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:43:40.257 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:43:40.257 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:40.258 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:40.290 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:40.439 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:40.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:40.544 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:40.571 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:40.714 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:40.787 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:43:40.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:39.658201+0000 mon.a (mon.0) 3854 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:40.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:39.802850+0000 mgr.y (mgr.24437) 663 : cluster [DBG] pgmap v1199: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1/819 objects misplaced (0.122%) 2024-02-23T00:43:40.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:40.273304+0000 mon.a (mon.0) 3855 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:43:40.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:40.273393+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-02-23T00:43:40.813 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:40.276437+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-23T00:43:40.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:39.658201+0000 mon.a (mon.0) 3854 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:40.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:39.802850+0000 mgr.y (mgr.24437) 663 : cluster [DBG] pgmap v1199: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1/819 objects misplaced (0.122%) 2024-02-23T00:43:40.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:40.273304+0000 mon.a (mon.0) 3855 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:43:40.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:40.273393+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-02-23T00:43:40.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:40.276437+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-23T00:43:40.823 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:40.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:40.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:40.927 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:39.658201+0000 mon.a (mon.0) 3854 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:39.802850+0000 mgr.y (mgr.24437) 663 : cluster [DBG] pgmap v1199: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 1/819 objects misplaced (0.122%) 2024-02-23T00:43:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:40.273304+0000 mon.a (mon.0) 3855 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-23T00:43:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:40.273393+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-02-23T00:43:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:40.276437+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-23T00:43:40.957 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:40.977 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:40.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:40.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:41.081 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:41.109 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:41.129 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:41.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:41.233 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:41.260 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:41.283 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:41.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:41.389 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:41.415 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:41.433 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:41.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:41.537 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:41.561 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:43:41.563 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:41.581 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:41.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:41.647 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-23T00:43:41.685 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:41.717 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.122474+0000 mon.a (mon.0) 3858 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-23T00:43:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.122656+0000 mon.a (mon.0) 3859 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-23T00:43:41.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.122797+0000 mon.a (mon.0) 3860 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.12", "id": [7, 0]}]: dispatch 2024-02-23T00:43:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.122937+0000 mon.a (mon.0) 3861 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1f", "id": [7, 0]}]: dispatch 2024-02-23T00:43:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.274126+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "target_max_objects","val": "1"}]': finished 2024-02-23T00:43:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.274209+0000 mon.a (mon.0) 3863 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-02-23T00:43:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.274274+0000 mon.a (mon.0) 3864 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-23T00:43:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.274329+0000 mon.a (mon.0) 3865 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.12", "id": [7, 0]}]': finished 2024-02-23T00:43:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.274386+0000 mon.a (mon.0) 3866 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1f", "id": [7, 0]}]': finished 2024-02-23T00:43:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:41.274432+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-02-23T00:43:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:41.304454+0000 mgr.y (mgr.24437) 664 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:41.964 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.122474+0000 mon.a (mon.0) 3858 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-23T00:43:41.964 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.122656+0000 mon.a (mon.0) 3859 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-23T00:43:41.964 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.122797+0000 mon.a (mon.0) 3860 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.12", "id": [7, 0]}]: dispatch 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.122937+0000 mon.a (mon.0) 3861 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1f", "id": [7, 0]}]: dispatch 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.274126+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "target_max_objects","val": "1"}]': finished 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.274209+0000 mon.a (mon.0) 3863 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.274274+0000 mon.a (mon.0) 3864 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.274329+0000 mon.a (mon.0) 3865 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.12", "id": [7, 0]}]': finished 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.274386+0000 mon.a (mon.0) 3866 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1f", "id": [7, 0]}]': finished 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:41.274432+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:41.304454+0000 mgr.y (mgr.24437) 664 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.122474+0000 mon.a (mon.0) 3858 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]: dispatch 2024-02-23T00:43:41.965 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.122656+0000 mon.a (mon.0) 3859 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-02-23T00:43:41.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.122797+0000 mon.a (mon.0) 3860 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.12", "id": [7, 0]}]: dispatch 2024-02-23T00:43:41.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.122937+0000 mon.a (mon.0) 3861 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1f", "id": [7, 0]}]: dispatch 2024-02-23T00:43:41.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.274126+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-136","var": "target_max_objects","val": "1"}]': finished 2024-02-23T00:43:41.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.274209+0000 mon.a (mon.0) 3863 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 5]}]': finished 2024-02-23T00:43:41.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.274274+0000 mon.a (mon.0) 3864 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-02-23T00:43:41.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.274329+0000 mon.a (mon.0) 3865 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.12", "id": [7, 0]}]': finished 2024-02-23T00:43:41.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.274386+0000 mon.a (mon.0) 3866 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1f", "id": [7, 0]}]': finished 2024-02-23T00:43:41.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:41.274432+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-02-23T00:43:41.966 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:41.304454+0000 mgr.y (mgr.24437) 664 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:41.966 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:41.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:42.071 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:42.098 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:42.117 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:42.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:42.221 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:42.248 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:42.270 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:42.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:42.373 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:42.401 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:42.532 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:43:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:42] "GET /metrics HTTP/1.1" 200 190861 "" "Prometheus/2.33.4" 2024-02-23T00:43:42.537 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:42.548 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:42.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:42.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:42.680 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:41.803278+0000 mgr.y (mgr.24437) 665 : cluster [DBG] pgmap v1202: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 1/825 objects misplaced (0.121%) 2024-02-23T00:43:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:42.250683+0000 mon.a (mon.0) 3868 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-02-23T00:43:43.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:42.661712+0000 mon.a (mon.0) 3869 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-23T00:43:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:41.803278+0000 mgr.y (mgr.24437) 665 : cluster [DBG] pgmap v1202: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 1/825 objects misplaced (0.121%) 2024-02-23T00:43:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:42.250683+0000 mon.a (mon.0) 3868 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-02-23T00:43:43.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:42.661712+0000 mon.a (mon.0) 3869 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-23T00:43:43.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:41.803278+0000 mgr.y (mgr.24437) 665 : cluster [DBG] pgmap v1202: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 1/825 objects misplaced (0.121%) 2024-02-23T00:43:43.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:42.250683+0000 mon.a (mon.0) 3868 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-02-23T00:43:43.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:42.661712+0000 mon.a (mon.0) 3869 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-23T00:43:44.128 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:44.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:44.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:43:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:43:44.234 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:44.267 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:44.289 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:44.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:44.392 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:44.422 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:44.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:43.254163+0000 osd.3 (osd.3) 246 : cluster [DBG] 4.9 starting backfill to osd.5 from (0'0,0'0] MAX to 453'68 2024-02-23T00:43:44.632 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:43.255746+0000 mon.a (mon.0) 3870 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-02-23T00:43:44.632 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:43.291431+0000 osd.1 (osd.1) 354 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:44.632 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:43.291590+0000 osd.1 (osd.1) 355 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:44.633 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:43.254163+0000 osd.3 (osd.3) 246 : cluster [DBG] 4.9 starting backfill to osd.5 from (0'0,0'0] MAX to 453'68 2024-02-23T00:43:44.633 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:43.255746+0000 mon.a (mon.0) 3870 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-02-23T00:43:44.633 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:43.291431+0000 osd.1 (osd.1) 354 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:44.633 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:43.291590+0000 osd.1 (osd.1) 355 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:44.634 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:44.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:43.254163+0000 osd.3 (osd.3) 246 : cluster [DBG] 4.9 starting backfill to osd.5 from (0'0,0'0] MAX to 453'68 2024-02-23T00:43:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:43.255746+0000 mon.a (mon.0) 3870 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-02-23T00:43:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:43.291431+0000 osd.1 (osd.1) 354 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:44.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:43.291590+0000 osd.1 (osd.1) 355 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:44.735 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:44.768 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:44.789 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:44.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:44.893 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:44.922 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:44.942 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:44.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:45.046 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:45.074 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:45.093 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:45.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:45.198 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:45.226 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:45.245 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:45.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:45.348 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:45.373 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:43.803702+0000 mgr.y (mgr.24437) 666 : cluster [DBG] pgmap v1205: 204 pgs: 204 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-23T00:43:45.373 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:44.257525+0000 mon.a (mon.0) 3871 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-23T00:43:45.374 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:44.269180+0000 mon.a (mon.0) 3872 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-02-23T00:43:45.374 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:44.676105+0000 osd.6 (osd.6) 326 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:45.374 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:44.676353+0000 osd.6 (osd.6) 327 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:45.374 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:45.063049+0000 mon.c (mon.2) 503 : audit [INF] from='client.? 172.21.15.87:0/3982446584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:43:45.374 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:45.063331+0000 mon.a (mon.0) 3873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:43:45.375 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:45.395 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:45.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:45.499 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:45.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:43.803702+0000 mgr.y (mgr.24437) 666 : cluster [DBG] pgmap v1205: 204 pgs: 204 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-23T00:43:45.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:44.257525+0000 mon.a (mon.0) 3871 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-23T00:43:45.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:44.269180+0000 mon.a (mon.0) 3872 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-02-23T00:43:45.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:44.676105+0000 osd.6 (osd.6) 326 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:45.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:44.676353+0000 osd.6 (osd.6) 327 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:45.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:45.063049+0000 mon.c (mon.2) 503 : audit [INF] from='client.? 172.21.15.87:0/3982446584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:43:45.527 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:45.063331+0000 mon.a (mon.0) 3873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:43:45.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:43.803702+0000 mgr.y (mgr.24437) 666 : cluster [DBG] pgmap v1205: 204 pgs: 204 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-23T00:43:45.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:44.257525+0000 mon.a (mon.0) 3871 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-23T00:43:45.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:44.269180+0000 mon.a (mon.0) 3872 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-02-23T00:43:45.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:44.676105+0000 osd.6 (osd.6) 326 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:45.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:44.676353+0000 osd.6 (osd.6) 327 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:45.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:45.063049+0000 mon.c (mon.2) 503 : audit [INF] from='client.? 172.21.15.87:0/3982446584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:43:45.528 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:45.063331+0000 mon.a (mon.0) 3873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:43:45.529 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:45.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.789+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/1052812646 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40f7400 msgr2=0x7fb4a4103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.789+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/1052812646 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40f7400 0x7fb4a4103a20 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 rx=0x7fb49c009c30 tx=0x7fb49c004c50).stop 2024-02-23T00:43:45.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/1052812646 shutdown_connections 2024-02-23T00:43:45.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/1052812646 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40f7400 0x7fb4a4103a20 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.794 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/1052812646 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4a40fcbb0 0x7fb4a40fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/1052812646 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4a40fd510 0x7fb4a40f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/1052812646 >> 172.21.15.87:0/1052812646 conn(0x7fb4a40f2820 msgr2=0x7fb4a40f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/1052812646 shutdown_connections 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/1052812646 wait complete. 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 librados: starting msgr at 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 librados: starting objecter 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 Processor -- start 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 -- start start 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 librados: setting wanted keys 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 librados: calling monclient init 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4a40f7400 0x7fb4a410df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:45.795 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40fcbb0 0x7fb4a410e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4a40fd510 0x7fb4a4108110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4108600 con 0x7fb4a40fd510 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4108770 con 0x7fb4a40f7400 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.790+0000 7fb4ac7f7700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4108930 con 0x7fb4a40fcbb0 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4a9d92700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40fcbb0 0x7fb4a410e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4aad94700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4a40fd510 0x7fb4a4108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4a9d92700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40fcbb0 0x7fb4a410e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35430/0 (socket says 172.21.15.87:35430) 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4aad94700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4a40fd510 0x7fb4a4108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51768/0 (socket says 172.21.15.87:51768) 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4a9d92700 1 -- 172.21.15.87:0/3895124845 learned_addr learned my addr 172.21.15.87:0/3895124845 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4aa593700 1 --2- 172.21.15.87:0/3895124845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4a40f7400 0x7fb4a410df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:45.796 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4aad94700 1 -- 172.21.15.87:0/3895124845 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40fcbb0 msgr2=0x7fb4a410e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4aad94700 1 --2- 172.21.15.87:0/3895124845 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40fcbb0 0x7fb4a410e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4aad94700 1 -- 172.21.15.87:0/3895124845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4a40f7400 msgr2=0x7fb4a410df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4aad94700 1 --2- 172.21.15.87:0/3895124845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4a40f7400 0x7fb4a410df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4aad94700 1 -- 172.21.15.87:0/3895124845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb494005040 con 0x7fb4a40fd510 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4a9d92700 1 --2- 172.21.15.87:0/3895124845 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40fcbb0 0x7fb4a410e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4aad94700 1 --2- 172.21.15.87:0/3895124845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4a40fd510 0x7fb4a4108110 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7fb49c009ad0 tx=0x7fb49c004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb49b7fe700 10 client.?.objecter ms_handle_connect 0x7fb4a40fd510 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb49b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb49b7fe700 1 -- 172.21.15.87:0/3895124845 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c00e040 con 0x7fb4a40fd510 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb49b7fe700 1 -- 172.21.15.87:0/3895124845 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb49c018040 con 0x7fb4a40fd510 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c0097c0 con 0x7fb4a40fd510 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb49b7fe700 1 -- 172.21.15.87:0/3895124845 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c013e20 con 0x7fb4a40fd510 2024-02-23T00:43:45.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4ac7f7700 10 client.27900.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a4108dd0 con 0x7fb4a40fd510 2024-02-23T00:43:45.798 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.791+0000 7fb4ac7f7700 1 librados: init done 2024-02-23T00:43:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.792+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a4108fc0 con 0x7fb4a40fd510 2024-02-23T00:43:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb49b7fe700 1 -- 172.21.15.87:0/3895124845 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb49c007b20 con 0x7fb4a40fd510 2024-02-23T00:43:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb49b7fe700 1 --2- 172.21.15.87:0/3895124845 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb490062c90 0x7fb490065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb49b7fe700 1 -- 172.21.15.87:0/3895124845 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(788..788 src has 400..788) v4 ==== 8709+0+0 (secure 0 0 0) 0x7fb49c07e650 con 0x7fb4a40fd510 2024-02-23T00:43:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb49b7fe700 10 client.27900.objecter ms_dispatch 0x7fb4a4001550 osd_map(788..788 src has 400..788) v4 2024-02-23T00:43:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb49b7fe700 3 client.27900.objecter handle_osd_map got epochs [788,788] > 0 2024-02-23T00:43:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb49b7fe700 3 client.27900.objecter handle_osd_map decoding full epoch 788 2024-02-23T00:43:45.799 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb4aa593700 1 --2- 172.21.15.87:0/3895124845 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb490062c90 0x7fb490065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb49b7fe700 20 client.27900.objecter dump_active .. 0 homeless 2024-02-23T00:43:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb49b7fe700 10 client.27900.objecter ms_handle_connect 0x7fb490062c90 2024-02-23T00:43:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.793+0000 7fb4aa593700 1 --2- 172.21.15.87:0/3895124845 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb490062c90 0x7fb490065150 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7fb4a0004dc0 tx=0x7fb4a000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:45.800 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.797+0000 7fb49b7fe700 1 -- 172.21.15.87:0/3895124845 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb49c051890 con 0x7fb4a40fd510 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 -- 172.21.15.87:0/1652518186 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840fcbb0 msgr2=0x7f59840fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 --2- 172.21.15.87:0/1652518186 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840fcbb0 0x7f59840fcf90 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 rx=0x7f59780047b0 tx=0x7f5978009cb0).stop 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 -- 172.21.15.87:0/1652518186 shutdown_connections 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 --2- 172.21.15.87:0/1652518186 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59840f7400 0x7f5984103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 --2- 172.21.15.87:0/1652518186 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840fcbb0 0x7f59840fcf90 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 --2- 172.21.15.87:0/1652518186 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59840fd510 0x7f59840f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 -- 172.21.15.87:0/1652518186 >> 172.21.15.87:0/1652518186 conn(0x7f59840f2820 msgr2=0x7f59840f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 -- 172.21.15.87:0/1652518186 shutdown_connections 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 -- 172.21.15.87:0/1652518186 wait complete. 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 librados: starting msgr at 2024-02-23T00:43:45.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 librados: starting objecter 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 Processor -- start 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 -- start start 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 librados: setting wanted keys 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.880+0000 7f598ab7a700 1 librados: calling monclient init 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840f7400 0x7f59840f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59840fcbb0 0x7f59840f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59840fd510 0x7f59840f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59841166f0 con 0x7f59840fd510 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5984116860 con 0x7f59840fcbb0 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5984116a20 con 0x7f59840f7400 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5988916700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840f7400 0x7f59840f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5988916700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840f7400 0x7f59840f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35410/0 (socket says 172.21.15.87:35410) 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5988916700 1 -- 172.21.15.87:0/3982446584 learned_addr learned my addr 172.21.15.87:0/3982446584 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:45.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5989117700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59840fd510 0x7f59840f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5983fff700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59840fcbb0 0x7f59840f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5988916700 1 -- 172.21.15.87:0/3982446584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59840fcbb0 msgr2=0x7f59840f8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5988916700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59840fcbb0 0x7f59840f8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5988916700 1 -- 172.21.15.87:0/3982446584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59840fd510 msgr2=0x7f59840f9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5988916700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59840fd510 0x7f59840f9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5988916700 1 -- 172.21.15.87:0/3982446584 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5970005040 con 0x7f59840f7400 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5989117700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59840fd510 0x7f59840f9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5988916700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840f7400 0x7f59840f89b0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f5978009b10 tx=0x7f5978009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5981ffb700 10 client.?.objecter ms_handle_connect 0x7f59840f7400 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5981ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5981ffb700 1 -- 172.21.15.87:0/3982446584 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5978018070 con 0x7f59840f7400 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5981ffb700 1 -- 172.21.15.87:0/3982446584 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5978013e70 con 0x7f59840f7400 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f5981ffb700 1 -- 172.21.15.87:0/3982446584 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597801ddb0 con 0x7f59840f7400 2024-02-23T00:43:45.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5978005040 con 0x7f59840f7400 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 10 client.38369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5984116ec0 con 0x7f59840f7400 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.881+0000 7f598ab7a700 1 librados: init done 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.882+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598405fa60 con 0x7f59840f7400 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.883+0000 7f5981ffb700 1 -- 172.21.15.87:0/3982446584 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f597801d420 con 0x7f59840f7400 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.883+0000 7f5981ffb700 1 --2- 172.21.15.87:0/3982446584 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5974062c90 0x7f5974065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.883+0000 7f5981ffb700 1 -- 172.21.15.87:0/3982446584 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(787..787 src has 400..787) v4 ==== 8709+0+0 (secure 0 0 0) 0x7f597807e8a0 con 0x7f59840f7400 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.883+0000 7f5981ffb700 10 client.38369.objecter ms_dispatch 0x7f5984001550 osd_map(787..787 src has 400..787) v4 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.883+0000 7f5981ffb700 3 client.38369.objecter handle_osd_map got epochs [787,787] > 0 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.883+0000 7f5981ffb700 3 client.38369.objecter handle_osd_map decoding full epoch 787 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.883+0000 7f5983fff700 1 --2- 172.21.15.87:0/3982446584 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5974062c90 0x7f5974065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.883+0000 7f5981ffb700 20 client.38369.objecter dump_active .. 0 homeless 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.884+0000 7f5981ffb700 10 client.38369.objecter ms_handle_connect 0x7f5974062c90 2024-02-23T00:43:45.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.884+0000 7f5983fff700 1 --2- 172.21.15.87:0/3982446584 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5974062c90 0x7f5974065150 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f59840fa150 tx=0x7f5970009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:44.887+0000 7f5981ffb700 1 -- 172.21.15.87:0/3982446584 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f5978051b60 con 0x7f59840f7400 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.061+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f59840f9e90 con 0x7f59840f7400 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.270+0000 7f5981ffb700 1 -- 172.21.15.87:0/3982446584 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v788) v1 ==== 91+0+0 (secure 0 0 0) 0x7f5978051750 con 0x7f59840f7400 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.272+0000 7f598ab7a700 10 librados: watch_flush enter 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.272+0000 7f598ab7a700 10 librados: watch_flush exit 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.272+0000 7f598ab7a700 20 client.38369.objecter shutdown clearing up homeless session... 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.272+0000 7f598ab7a700 10 client.38369.objecter successfully canceled tick 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.272+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5974062c90 msgr2=0x7f5974065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.272+0000 7f598ab7a700 1 --2- 172.21.15.87:0/3982446584 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5974062c90 0x7f5974065150 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 rx=0x7f59840fa150 tx=0x7f5970009cb0).stop 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.272+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840f7400 msgr2=0x7f59840f89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.272+0000 7f598ab7a700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840f7400 0x7f59840f89b0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f5978009b10 tx=0x7f5978009cb0).stop 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.273+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 shutdown_connections 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.273+0000 7f598ab7a700 1 --2- 172.21.15.87:0/3982446584 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5974062c90 0x7f5974065150 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.273+0000 7f598ab7a700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f59840fd510 0x7f59840f9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.273+0000 7f598ab7a700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f59840f7400 0x7f59840f89b0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.273+0000 7f598ab7a700 1 --2- 172.21.15.87:0/3982446584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f59840fcbb0 0x7f59840f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.273+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 >> 172.21.15.87:0/3982446584 conn(0x7f59840f2820 msgr2=0x7f59840f4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:45.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.273+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 shutdown_connections 2024-02-23T00:43:45.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.273+0000 7f598ab7a700 1 -- 172.21.15.87:0/3982446584 wait complete. 2024-02-23T00:43:45.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.273+0000 7f598ab7a700 1 librados: shutdown 2024-02-23T00:43:45.934 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:43:45.966 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.963+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4a4106120 con 0x7fb490062c90 2024-02-23T00:43:45.979 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.977+0000 7fb49b7fe700 1 -- 172.21.15.87:0/3895124845 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+527759 (secure 0 0 0) 0x7fb4a4106120 con 0x7fb490062c90 2024-02-23T00:43:45.986 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:43:45.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 10 librados: watch_flush enter 2024-02-23T00:43:45.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 10 librados: watch_flush exit 2024-02-23T00:43:45.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 20 client.27900.objecter shutdown clearing up homeless session... 2024-02-23T00:43:45.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 10 client.27900.objecter successfully canceled tick 2024-02-23T00:43:45.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb490062c90 msgr2=0x7fb490065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/3895124845 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb490062c90 0x7fb490065150 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 rx=0x7fb4a0004dc0 tx=0x7fb4a000b040).stop 2024-02-23T00:43:45.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4a40fd510 msgr2=0x7fb4a4108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/3895124845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4a40fd510 0x7fb4a4108110 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 rx=0x7fb49c009ad0 tx=0x7fb49c004c50).stop 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 shutdown_connections 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/3895124845 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb490062c90 0x7fb490065150 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/3895124845 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb4a40fd510 0x7fb4a4108110 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/3895124845 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb4a40f7400 0x7fb4a410df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 --2- 172.21.15.87:0/3895124845 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb4a40fcbb0 0x7fb4a410e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 >> 172.21.15.87:0/3895124845 conn(0x7fb4a40f2820 msgr2=0x7fb4a4060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 shutdown_connections 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 -- 172.21.15.87:0/3895124845 wait complete. 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:45.981+0000 7fb4ac7f7700 1 librados: shutdown 2024-02-23T00:43:45.992 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:43:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:45.263841+0000 mon.a (mon.0) 3874 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:43:46.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:45.270839+0000 mon.a (mon.0) 3875 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:43:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:45.270907+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-02-23T00:43:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:45.957119+0000 osd.7 (osd.7) 356 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:43:46.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:45.960015+0000 osd.7 (osd.7) 357 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:43:46.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:45.263841+0000 mon.a (mon.0) 3874 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:43:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:45.270839+0000 mon.a (mon.0) 3875 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:43:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:45.270907+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-02-23T00:43:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:45.957119+0000 osd.7 (osd.7) 356 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:43:46.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:45.960015+0000 osd.7 (osd.7) 357 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:43:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:45.263841+0000 mon.a (mon.0) 3874 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:43:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:45.270839+0000 mon.a (mon.0) 3875 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:43:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:45.270907+0000 mon.a (mon.0) 3876 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-02-23T00:43:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:45.957119+0000 osd.7 (osd.7) 356 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:43:46.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:45.960015+0000 osd.7 (osd.7) 357 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:45.804544+0000 mgr.y (mgr.24437) 667 : cluster [DBG] pgmap v1208: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 255 B/s wr, 30 op/s; 9/825 objects misplaced (1.091%); 164 B/s, 7 objects/s recovering 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:45.965442+0000 mgr.y (mgr.24437) 668 : audit [DBG] from='client.27900 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:46.268598+0000 mon.a (mon.0) 3877 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:46.456392+0000 osd.7 (osd.7) 358 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:46.456553+0000 osd.7 (osd.7) 359 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:46.604127+0000 osd.1 (osd.1) 356 : cluster [DBG] 5.c scrub starts 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:46.607002+0000 osd.1 (osd.1) 357 : cluster [DBG] 5.c scrub ok 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:46.986330+0000 osd.4 (osd.4) 341 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:46.986582+0000 osd.4 (osd.4) 342 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.009301+0000 osd.4 (osd.4) 343 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.010229+0000 osd.4 (osd.4) 344 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:43:47.273 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:47.242844+0000 mon.c (mon.2) 504 : audit [DBG] from='client.? 172.21.15.87:0/1906222321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:47.274 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.250614+0000 mon.a (mon.0) 3878 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-02-23T00:43:47.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 -- 172.21.15.87:0/1662601471 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0f6a90 msgr2=0x7fd71c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1662601471 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0f6a90 0x7fd71c104a20 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 rx=0x7fd70c009be0 tx=0x7fd70c004d10).stop 2024-02-23T00:43:47.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 -- 172.21.15.87:0/1662601471 shutdown_connections 2024-02-23T00:43:47.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1662601471 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0f6a90 0x7fd71c104a20 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1662601471 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71c0fdbd0 0x7fd71c0fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1662601471 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd71c0fe530 0x7fd71c0f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.298 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 -- 172.21.15.87:0/1662601471 >> 172.21.15.87:0/1662601471 conn(0x7fd71c06c370 msgr2=0x7fd71c06c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 -- 172.21.15.87:0/1662601471 shutdown_connections 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 -- 172.21.15.87:0/1662601471 wait complete. 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 librados: starting msgr at 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.238+0000 7fd723b6b700 1 librados: starting objecter 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 Processor -- start 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 -- start start 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 librados: setting wanted keys 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 librados: calling monclient init 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd71c0f6a90 0x7fd71c1101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71c0fdbd0 0x7fd71c1106c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0fe530 0x7fd71c10a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c10a860 con 0x7fd71c0fdbd0 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c10a9d0 con 0x7fd71c0f6a90 2024-02-23T00:43:47.299 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd723b6b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71c10ab90 con 0x7fd71c0fe530 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd721106700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71c0fdbd0 0x7fd71c1106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd722108700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0fe530 0x7fd71c10a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd721106700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71c0fdbd0 0x7fd71c1106c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51780/0 (socket says 172.21.15.87:51780) 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd721106700 1 -- 172.21.15.87:0/1906222321 learned_addr learned my addr 172.21.15.87:0/1906222321 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd722108700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0fe530 0x7fd71c10a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:35440/0 (socket says 172.21.15.87:35440) 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd721907700 1 --2- 172.21.15.87:0/1906222321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd71c0f6a90 0x7fd71c1101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd722108700 1 -- 172.21.15.87:0/1906222321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd71c0f6a90 msgr2=0x7fd71c1101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd722108700 1 --2- 172.21.15.87:0/1906222321 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd71c0f6a90 0x7fd71c1101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd722108700 1 -- 172.21.15.87:0/1906222321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71c0fdbd0 msgr2=0x7fd71c1106c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd722108700 1 --2- 172.21.15.87:0/1906222321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71c0fdbd0 0x7fd71c1106c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd722108700 1 -- 172.21.15.87:0/1906222321 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd714005040 con 0x7fd71c0fe530 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd721106700 1 --2- 172.21.15.87:0/1906222321 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd71c0fdbd0 0x7fd71c1106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:47.300 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd722108700 1 --2- 172.21.15.87:0/1906222321 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0fe530 0x7fd71c10a370 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7fd70c00b6d0 tx=0x7fd70c004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd71affd700 10 client.?.objecter ms_handle_connect 0x7fd71c0fe530 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.239+0000 7fd71affd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.240+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00e030 con 0x7fd71c0fe530 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.240+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70c0097c0 con 0x7fd71c0fe530 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.240+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd70c007bf0 con 0x7fd71c0fe530 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.240+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c018070 con 0x7fd71c0fe530 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.240+0000 7fd723b6b700 10 client.38381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.240+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71c10b020 con 0x7fd71c0fe530 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.240+0000 7fd723b6b700 1 librados: init done 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.240+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fd71c05fa60 con 0x7fd71c0fe530 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.241+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd70c01d420 con 0x7fd71c0fe530 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.241+0000 7fd71affd700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd704062c90 0x7fd704065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.241+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(788..788 src has 400..788) v4 ==== 8709+0+0 (secure 0 0 0) 0x7fd70c07ec50 con 0x7fd71c0fe530 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.241+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 osd_map(788..788 src has 400..788) v4 2024-02-23T00:43:47.301 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.241+0000 7fd71affd700 3 client.38381.objecter handle_osd_map got epochs [788,788] > 0 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.241+0000 7fd71affd700 3 client.38381.objecter handle_osd_map decoding full epoch 788 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.241+0000 7fd721907700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd704062c90 0x7fd704065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd71affd700 20 client.38381.objecter dump_active .. 0 homeless 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v788) v1 ==== 54+0+16 (secure 0 0 0) 0x7fd70c084220 con 0x7fd71c0fe530 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd71affd700 10 client.38381.objecter ms_handle_connect 0x7fd704062c90 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd721907700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd704062c90 0x7fd704065150 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 rx=0x7fd7140092f0 tx=0x7fd714009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 10 client.38381.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 1 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6801/484081831 conn(0x7fd71c028540 0x7fd71c0f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c10b210 osd=0 3 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c10b210 osd=0 3 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c10b210 osd=0 3 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 1 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c10b210 osd=0 3 2024-02-23T00:43:47.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 0 1 2024-02-23T00:43:47.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c10b210 osd=0 4 2024-02-23T00:43:47.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 10 client.38381.objecter _send_command 1 2024-02-23T00:43:47.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd71c05ee80 con 0x7fd71c028540 2024-02-23T00:43:47.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd721106700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6801/484081831 conn(0x7fd71c028540 0x7fd71c0f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd71affd700 10 client.38381.objecter ms_handle_connect 0x7fd71c028540 2024-02-23T00:43:47.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.242+0000 7fd721106700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6801/484081831 conn(0x7fd71c028540 0x7fd71c0f9820 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.243+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd71c05ee80 con 0x7fd71c028540 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.243+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.243+0000 7fd71affd700 10 client.38381.objecter _finish_command 1 = system:0 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.243+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c10b210 osd=0 4 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.243+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 0 1 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 10 client.38381.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 2 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c10b210 osd=0 3 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c10b210 osd=0 3 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c10b210 osd=0 3 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 2 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c10b210 osd=0 3 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 0 2 2024-02-23T00:43:47.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c10b210 osd=0 4 2024-02-23T00:43:47.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 10 client.38381.objecter _send_command 2 2024-02-23T00:43:47.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.260+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd71c107170 con 0x7fd71c028540 2024-02-23T00:43:47.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.262+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd71c107170 con 0x7fd71c028540 2024-02-23T00:43:47.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.262+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:47.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.262+0000 7fd71affd700 10 client.38381.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.262+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c10b210 osd=0 4 2024-02-23T00:43:47.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.262+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 0 2 2024-02-23T00:43:47.305 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:43:47.305 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:43:47.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.262+0000 7fd723b6b700 10 client.38381.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:47.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 3 2024-02-23T00:43:47.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6805/685354171 conn(0x7fd71c0fa390 0x7fd71c0fa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c0f9d10 osd=1 3 2024-02-23T00:43:47.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c0f9d10 osd=1 3 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c0f9d10 osd=1 3 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 3 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c0f9d10 osd=1 3 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 1 3 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c0f9d10 osd=1 4 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 10 client.38381.objecter _send_command 3 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd71c0fac60 con 0x7fd71c0fa390 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd721907700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6805/685354171 conn(0x7fd71c0fa390 0x7fd71c0fa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd71affd700 10 client.38381.objecter ms_handle_connect 0x7fd71c0fa390 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.263+0000 7fd721907700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6805/685354171 conn(0x7fd71c0fa390 0x7fd71c0fa770 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.264+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd71c0fac60 con 0x7fd71c0fa390 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.264+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.264+0000 7fd71affd700 10 client.38381.objecter _finish_command 3 = system:0 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.264+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c0f9d10 osd=1 4 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.264+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 1 3 2024-02-23T00:43:47.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 10 client.38381.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 4 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c0f9d10 osd=1 3 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c0f9d10 osd=1 3 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c0f9d10 osd=1 3 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 4 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c0f9d10 osd=1 3 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 1 4 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c0f9d10 osd=1 4 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 10 client.38381.objecter _send_command 4 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.281+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd71c11b020 con 0x7fd71c0fa390 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd71c11b020 con 0x7fd71c0fa390 2024-02-23T00:43:47.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd71affd700 10 client.38381.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c0f9d10 osd=1 4 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 1 4 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 10 client.38381.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 5 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd71c0594d0 0x7fd71c11e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c059360 osd=2 3 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c059360 osd=2 3 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c059360 osd=2 3 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 5 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c059360 osd=2 3 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 2 5 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c059360 osd=2 4 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 10 client.38381.objecter _send_command 5 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.283+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd71c0282f0 con 0x7fd71c0594d0 2024-02-23T00:43:47.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.284+0000 7fd721106700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd71c0594d0 0x7fd71c11e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.284+0000 7fd71affd700 10 client.38381.objecter ms_handle_connect 0x7fd71c0594d0 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.284+0000 7fd721106700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd71c0594d0 0x7fd71c11e3b0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.285+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd71c0282f0 con 0x7fd71c0594d0 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.285+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.285+0000 7fd71affd700 10 client.38381.objecter _finish_command 5 = system:0 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.285+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c059360 osd=2 4 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.285+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 2 5 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 10 client.38381.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 6 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c059360 osd=2 3 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c059360 osd=2 3 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c059360 osd=2 3 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 6 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c059360 osd=2 3 2024-02-23T00:43:47.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 2 6 2024-02-23T00:43:47.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c059360 osd=2 4 2024-02-23T00:43:47.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 10 client.38381.objecter _send_command 6 2024-02-23T00:43:47.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.300+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd71c0276c0 con 0x7fd71c0594d0 2024-02-23T00:43:47.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.302+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd71c0276c0 con 0x7fd71c0594d0 2024-02-23T00:43:47.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.302+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:47.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.302+0000 7fd71affd700 10 client.38381.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.302+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c059360 osd=2 4 2024-02-23T00:43:47.311 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.302+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 2 6 2024-02-23T00:43:47.312 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:43:47.312 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 10 client.38381.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 7 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6813/128059790 conn(0x7fd71c11fa20 0x7fd71c121e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b9f0 osd=3 3 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b9f0 osd=3 3 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b9f0 osd=3 3 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 7 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c11b9f0 osd=3 3 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 3 7 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b9f0 osd=3 4 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 10 client.38381.objecter _send_command 7 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd71c11c1c0 con 0x7fd71c11fa20 2024-02-23T00:43:47.313 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd722108700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6813/128059790 conn(0x7fd71c11fa20 0x7fd71c121e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd71affd700 10 client.38381.objecter ms_handle_connect 0x7fd71c11fa20 2024-02-23T00:43:47.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.303+0000 7fd722108700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6813/128059790 conn(0x7fd71c11fa20 0x7fd71c121e10 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.304+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd71c11c1c0 con 0x7fd71c11fa20 2024-02-23T00:43:47.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.304+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:43:47.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.304+0000 7fd71affd700 10 client.38381.objecter _finish_command 7 = system:0 2024-02-23T00:43:47.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.304+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c11b9f0 osd=3 4 2024-02-23T00:43:47.314 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.304+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 3 7 2024-02-23T00:43:47.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 10 client.38381.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 8 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b9f0 osd=3 3 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b9f0 osd=3 3 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b9f0 osd=3 3 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 8 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c11b9f0 osd=3 3 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 3 8 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b9f0 osd=3 4 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 10 client.38381.objecter _send_command 8 2024-02-23T00:43:47.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.318+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd71c11b460 con 0x7fd71c11fa20 2024-02-23T00:43:47.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd71c11b460 con 0x7fd71c11fa20 2024-02-23T00:43:47.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:47.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd71affd700 10 client.38381.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c11b9f0 osd=3 4 2024-02-23T00:43:47.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 3 8 2024-02-23T00:43:47.323 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:43:47.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 10 client.38381.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 9 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd71c11f580 0x7fd71c123480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b600 osd=4 3 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b600 osd=4 3 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b600 osd=4 3 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 9 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c11b600 osd=4 3 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 4 9 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b600 osd=4 4 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 10 client.38381.objecter _send_command 9 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd71c11b770 con 0x7fd71c11f580 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.320+0000 7fd721907700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd71c11f580 0x7fd71c123480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.321+0000 7fd71affd700 10 client.38381.objecter ms_handle_connect 0x7fd71c11f580 2024-02-23T00:43:47.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.321+0000 7fd721907700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd71c11f580 0x7fd71c123480 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.321+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd71c11b770 con 0x7fd71c11f580 2024-02-23T00:43:47.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.321+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:43:47.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.321+0000 7fd71affd700 10 client.38381.objecter _finish_command 9 = system:0 2024-02-23T00:43:47.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.321+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c11b600 osd=4 4 2024-02-23T00:43:47.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.321+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 4 9 2024-02-23T00:43:47.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 10 client.38381.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:47.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 10 2024-02-23T00:43:47.338 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b600 osd=4 3 2024-02-23T00:43:47.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b600 osd=4 3 2024-02-23T00:43:47.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b600 osd=4 3 2024-02-23T00:43:47.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 10 2024-02-23T00:43:47.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c11b600 osd=4 3 2024-02-23T00:43:47.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 4 10 2024-02-23T00:43:47.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b600 osd=4 4 2024-02-23T00:43:47.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 10 client.38381.objecter _send_command 10 2024-02-23T00:43:47.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.336+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd71c11b770 con 0x7fd71c11f580 2024-02-23T00:43:47.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd71c11b770 con 0x7fd71c11f580 2024-02-23T00:43:47.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:47.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd71affd700 10 client.38381.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c11b600 osd=4 4 2024-02-23T00:43:47.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 4 10 2024-02-23T00:43:47.341 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:43:47.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 10 client.38381.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:47.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 11 2024-02-23T00:43:47.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd71c123e80 0x7fd71c124280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b770 osd=5 3 2024-02-23T00:43:47.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b770 osd=5 3 2024-02-23T00:43:47.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b770 osd=5 3 2024-02-23T00:43:47.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 11 2024-02-23T00:43:47.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c11b770 osd=5 3 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 5 11 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b770 osd=5 4 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 10 client.38381.objecter _send_command 11 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd71c1248f0 con 0x7fd71c123e80 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.338+0000 7fd721106700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd71c123e80 0x7fd71c124280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.339+0000 7fd71affd700 10 client.38381.objecter ms_handle_connect 0x7fd71c123e80 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.339+0000 7fd721106700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd71c123e80 0x7fd71c124280 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.339+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd71c1248f0 con 0x7fd71c123e80 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.339+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.339+0000 7fd71affd700 10 client.38381.objecter _finish_command 11 = system:0 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.340+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c11b770 osd=5 4 2024-02-23T00:43:47.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.340+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 5 11 2024-02-23T00:43:47.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 10 client.38381.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:47.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 12 2024-02-23T00:43:47.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b770 osd=5 3 2024-02-23T00:43:47.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b770 osd=5 3 2024-02-23T00:43:47.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c11b770 osd=5 3 2024-02-23T00:43:47.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 12 2024-02-23T00:43:47.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c11b770 osd=5 3 2024-02-23T00:43:47.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 5 12 2024-02-23T00:43:47.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b770 osd=5 4 2024-02-23T00:43:47.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 10 client.38381.objecter _send_command 12 2024-02-23T00:43:47.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.355+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd71c1248f0 con 0x7fd71c123e80 2024-02-23T00:43:47.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd71c1248f0 con 0x7fd71c123e80 2024-02-23T00:43:47.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:47.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd71affd700 10 client.38381.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c11b770 osd=5 4 2024-02-23T00:43:47.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 5 12 2024-02-23T00:43:47.360 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 10 client.38381.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 13 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd71c128fe0 0x7fd71c129440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c1248f0 osd=6 3 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c1248f0 osd=6 3 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c1248f0 osd=6 3 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 13 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c1248f0 osd=6 3 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 6 13 2024-02-23T00:43:47.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c1248f0 osd=6 4 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 10 client.38381.objecter _send_command 13 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd71c12b010 con 0x7fd71c128fe0 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.357+0000 7fd722108700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd71c128fe0 0x7fd71c129440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.358+0000 7fd71affd700 10 client.38381.objecter ms_handle_connect 0x7fd71c128fe0 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.358+0000 7fd722108700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd71c128fe0 0x7fd71c129440 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.358+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd71c12b010 con 0x7fd71c128fe0 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.358+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.358+0000 7fd71affd700 10 client.38381.objecter _finish_command 13 = system:0 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.358+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c1248f0 osd=6 4 2024-02-23T00:43:47.362 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.358+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 6 13 2024-02-23T00:43:47.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 10 client.38381.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:47.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 14 2024-02-23T00:43:47.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c1248f0 osd=6 3 2024-02-23T00:43:47.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c1248f0 osd=6 3 2024-02-23T00:43:47.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c1248f0 osd=6 3 2024-02-23T00:43:47.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 14 2024-02-23T00:43:47.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c1248f0 osd=6 3 2024-02-23T00:43:47.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 6 14 2024-02-23T00:43:47.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c1248f0 osd=6 4 2024-02-23T00:43:47.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 10 client.38381.objecter _send_command 14 2024-02-23T00:43:47.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.373+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd71c12b010 con 0x7fd71c128fe0 2024-02-23T00:43:47.377 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.374+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd71c12b010 con 0x7fd71c128fe0 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.374+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.374+0000 7fd71affd700 10 client.38381.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.374+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c1248f0 osd=6 4 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.374+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 6 14 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 10 client.38381.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 15 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6812/373787000 conn(0x7fd71c129f60 0x7fd71c12a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c12b010 osd=7 3 2024-02-23T00:43:47.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c12b010 osd=7 3 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c12b010 osd=7 3 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 15 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c12b010 osd=7 3 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 7 15 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c12b010 osd=7 4 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 10 client.38381.objecter _send_command 15 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd71c12aa30 con 0x7fd71c129f60 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd721907700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6812/373787000 conn(0x7fd71c129f60 0x7fd71c12a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd71affd700 10 client.38381.objecter ms_handle_connect 0x7fd71c129f60 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.375+0000 7fd721907700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6812/373787000 conn(0x7fd71c129f60 0x7fd71c12a3c0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:47.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.376+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fd71c12aa30 con 0x7fd71c129f60 2024-02-23T00:43:47.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.376+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:43:47.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.376+0000 7fd71affd700 10 client.38381.objecter _finish_command 15 = system:0 2024-02-23T00:43:47.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.376+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c12b010 osd=7 4 2024-02-23T00:43:47.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.376+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 7 15 2024-02-23T00:43:47.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 10 client.38381.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:43:47.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign -1 16 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c12b010 osd=7 3 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c12b010 osd=7 3 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 20 client.38381.objecter _get_session s=0x7fd71c12b010 osd=7 3 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_remove -1 16 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 20 client.38381.objecter get_session s=0x7fd71c12b010 osd=7 3 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 15 client.38381.objecter _session_command_op_assign 7 16 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c12b010 osd=7 4 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 10 client.38381.objecter _send_command 16 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.390+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fd71c12aa30 con 0x7fd71c129f60 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd71affd700 1 -- 172.21.15.87:0/1906222321 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7fd71c12aa30 con 0x7fd71c129f60 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd71affd700 10 client.38381.objecter ms_dispatch 0x7fd71c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd71affd700 10 client.38381.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd71affd700 20 client.38381.objecter put_session s=0x7fd71c12b010 osd=7 4 2024-02-23T00:43:47.397 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd71affd700 15 client.38381.objecter _session_command_op_remove 7 16 2024-02-23T00:43:47.397 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:43:47.398 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 librados: watch_flush enter 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 librados: watch_flush exit 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 client.38381.objecter close_session for osd.0 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6801/484081831 conn(0x7fd71c028540 msgr2=0x7fd71c0f9820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6801/484081831 conn(0x7fd71c028540 0x7fd71c0f9820 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c10b210 osd=0 1 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 client.38381.objecter close_session for osd.1 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6805/685354171 conn(0x7fd71c0fa390 msgr2=0x7fd71c0fa770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6805/685354171 conn(0x7fd71c0fa390 0x7fd71c0fa770 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c0f9d10 osd=1 1 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 client.38381.objecter close_session for osd.2 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd71c0594d0 msgr2=0x7fd71c11e3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.399 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6809/3605394228 conn(0x7fd71c0594d0 0x7fd71c11e3b0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c059360 osd=2 1 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 client.38381.objecter close_session for osd.3 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd722108700 1 -- 172.21.15.87:0/1906222321 reap_dead start 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6813/128059790 conn(0x7fd71c11fa20 msgr2=0x7fd71c121e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6813/128059790 conn(0x7fd71c11fa20 0x7fd71c121e10 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b9f0 osd=3 1 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 client.38381.objecter close_session for osd.4 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd71c11f580 msgr2=0x7fd71c123480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6800/4175844757 conn(0x7fd71c11f580 0x7fd71c123480 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b600 osd=4 1 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 client.38381.objecter close_session for osd.5 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd71c123e80 msgr2=0x7fd71c124280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6804/2550013795 conn(0x7fd71c123e80 0x7fd71c124280 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.400 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c11b770 osd=5 1 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 client.38381.objecter close_session for osd.6 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd71c128fe0 msgr2=0x7fd71c129440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6808/2509408498 conn(0x7fd71c128fe0 0x7fd71c129440 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c1248f0 osd=6 1 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 client.38381.objecter close_session for osd.7 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6812/373787000 conn(0x7fd71c129f60 msgr2=0x7fd71c12a3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.171:6812/373787000 conn(0x7fd71c129f60 0x7fd71c12a3c0 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 20 client.38381.objecter put_session s=0x7fd71c12b010 osd=7 1 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 20 client.38381.objecter shutdown clearing up homeless session... 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 10 client.38381.objecter successfully canceled tick 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd722108700 1 -- 172.21.15.87:0/1906222321 reap_dead start 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd704062c90 msgr2=0x7fd704065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd704062c90 0x7fd704065150 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 rx=0x7fd7140092f0 tx=0x7fd714009cb0).stop 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0fe530 msgr2=0x7fd71c10a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:47.401 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0fe530 0x7fd71c10a370 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 rx=0x7fd70c00b6d0 tx=0x7fd70c004d10).stop 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 shutdown_connections 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd704062c90 0x7fd704065150 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 --2- 172.21.15.87:0/1906222321 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd71c0fe530 0x7fd71c10a370 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 >> 172.21.15.87:0/1906222321 conn(0x7fd71c06c370 msgr2=0x7fd71c103160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 shutdown_connections 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 -- 172.21.15.87:0/1906222321 wait complete. 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:47.393+0000 7fd723b6b700 1 librados: shutdown 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.402 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.403 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:45.804544+0000 mgr.y (mgr.24437) 667 : cluster [DBG] pgmap v1208: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 255 B/s wr, 30 op/s; 9/825 objects misplaced (1.091%); 164 B/s, 7 objects/s recovering 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:45.965442+0000 mgr.y (mgr.24437) 668 : audit [DBG] from='client.27900 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:46.268598+0000 mon.a (mon.0) 3877 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:46.456392+0000 osd.7 (osd.7) 358 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:46.456553+0000 osd.7 (osd.7) 359 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:46.604127+0000 osd.1 (osd.1) 356 : cluster [DBG] 5.c scrub starts 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:46.607002+0000 osd.1 (osd.1) 357 : cluster [DBG] 5.c scrub ok 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:46.986330+0000 osd.4 (osd.4) 341 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:46.986582+0000 osd.4 (osd.4) 342 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:47.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.009301+0000 osd.4 (osd.4) 343 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:43:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.010229+0000 osd.4 (osd.4) 344 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:43:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:47.242844+0000 mon.c (mon.2) 504 : audit [DBG] from='client.? 172.21.15.87:0/1906222321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.250614+0000 mon.a (mon.0) 3878 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-02-23T00:43:47.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:45.804544+0000 mgr.y (mgr.24437) 667 : cluster [DBG] pgmap v1208: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 255 B/s wr, 30 op/s; 9/825 objects misplaced (1.091%); 164 B/s, 7 objects/s recovering 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:45.965442+0000 mgr.y (mgr.24437) 668 : audit [DBG] from='client.27900 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:46.268598+0000 mon.a (mon.0) 3877 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:46.456392+0000 osd.7 (osd.7) 358 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:46.456553+0000 osd.7 (osd.7) 359 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:46.604127+0000 osd.1 (osd.1) 356 : cluster [DBG] 5.c scrub starts 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:46.607002+0000 osd.1 (osd.1) 357 : cluster [DBG] 5.c scrub ok 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:46.986330+0000 osd.4 (osd.4) 341 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:46.986582+0000 osd.4 (osd.4) 342 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.009301+0000 osd.4 (osd.4) 343 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.010229+0000 osd.4 (osd.4) 344 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:47.242844+0000 mon.c (mon.2) 504 : audit [DBG] from='client.? 172.21.15.87:0/1906222321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:43:47.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.250614+0000 mon.a (mon.0) 3878 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-02-23T00:43:48.224 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.033478+0000 osd.3 (osd.3) 247 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:43:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.036482+0000 osd.3 (osd.3) 248 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:43:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.038152+0000 osd.3 (osd.3) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.038264+0000 osd.3 (osd.3) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.430429+0000 osd.2 (osd.2) 281 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:43:48.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.431636+0000 osd.2 (osd.2) 282 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.449187+0000 osd.0 (osd.0) 53 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.452760+0000 osd.0 (osd.0) 54 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.958617+0000 osd.5 (osd.5) 305 : cluster [DBG] 3.c scrub starts 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.960394+0000 osd.5 (osd.5) 306 : cluster [DBG] 3.c scrub ok 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.992558+0000 osd.7 (osd.7) 360 : cluster [DBG] 2.b scrub starts 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.993575+0000 osd.7 (osd.7) 361 : cluster [DBG] 2.b scrub ok 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.033478+0000 osd.3 (osd.3) 247 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.036482+0000 osd.3 (osd.3) 248 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.038152+0000 osd.3 (osd.3) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.038264+0000 osd.3 (osd.3) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.430429+0000 osd.2 (osd.2) 281 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:43:48.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.431636+0000 osd.2 (osd.2) 282 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:43:48.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.449187+0000 osd.0 (osd.0) 53 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:43:48.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.452760+0000 osd.0 (osd.0) 54 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:43:48.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.958617+0000 osd.5 (osd.5) 305 : cluster [DBG] 3.c scrub starts 2024-02-23T00:43:48.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.960394+0000 osd.5 (osd.5) 306 : cluster [DBG] 3.c scrub ok 2024-02-23T00:43:48.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.992558+0000 osd.7 (osd.7) 360 : cluster [DBG] 2.b scrub starts 2024-02-23T00:43:48.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.993575+0000 osd.7 (osd.7) 361 : cluster [DBG] 2.b scrub ok 2024-02-23T00:43:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.033478+0000 osd.3 (osd.3) 247 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:43:48.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.036482+0000 osd.3 (osd.3) 248 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.038152+0000 osd.3 (osd.3) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.038264+0000 osd.3 (osd.3) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.430429+0000 osd.2 (osd.2) 281 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.431636+0000 osd.2 (osd.2) 282 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.449187+0000 osd.0 (osd.0) 53 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.452760+0000 osd.0 (osd.0) 54 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.958617+0000 osd.5 (osd.5) 305 : cluster [DBG] 3.c scrub starts 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.960394+0000 osd.5 (osd.5) 306 : cluster [DBG] 3.c scrub ok 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.992558+0000 osd.7 (osd.7) 360 : cluster [DBG] 2.b scrub starts 2024-02-23T00:43:48.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.993575+0000 osd.7 (osd.7) 361 : cluster [DBG] 2.b scrub ok 2024-02-23T00:43:48.777 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1208,"stamp":"2024-02-23T00:43:45.803886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4006106,"num_objects":275,"num_object_clones":1,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":1469,"num_read_kb":1311,"num_write":534,"num_write_kb":3994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":7443,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":2,"num_evict_kb":2,"num_promote":4,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":944,"ondisk_log_size":944,"up":612,"acting":612,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":611,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2489508,"kb_used_data":60964,"kb_used_omap":85,"kb_used_meta":2428394,"kb_avail":747471708,"statfs":{"total":767960285184,"available":765411028992,"internally_reserved":0,"allocated":62427136,"data_stored":36171843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":2486675584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-71,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":68,"num_write":44,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":2,"num_evict_kb":2,"num_promote":0,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000859"},"pg_stats":[{"pgid":"252.1f","version":"0'0","reported_seq":18,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281468+0000","last_change":"2024-02-23T00:43:33.262808+0000","last_active":"2024-02-23T00:43:41.281468+0000","last_peered":"2024-02-23T00:43:41.281468+0000","last_clean":"2024-02-23T00:43:41.281468+0000","last_became_active":"2024-02-23T00:43:33.262522+0000","last_became_peered":"2024-02-23T00:43:33.262522+0000","last_unstale":"2024-02-23T00:43:41.281468+0000","last_undegraded":"2024-02-23T00:43:41.281468+0000","last_fullsized":"2024-02-23T00:43:41.281468+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.1e","version":"0'0","reported_seq":20,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433415+0000","last_change":"2024-02-23T00:43:33.264545+0000","last_active":"2024-02-23T00:43:43.433415+0000","last_peered":"2024-02-23T00:43:43.433415+0000","last_clean":"2024-02-23T00:43:43.433415+0000","last_became_active":"2024-02-23T00:43:33.264155+0000","last_became_peered":"2024-02-23T00:43:33.264155+0000","last_unstale":"2024-02-23T00:43:43.433415+0000","last_undegraded":"2024-02-23T00:43:43.433415+0000","last_fullsized":"2024-02-23T00:43:43.433415+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"252.1d","version":"0'0","reported_seq":20,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259175+0000","last_change":"2024-02-23T00:43:33.261388+0000","last_active":"2024-02-23T00:43:43.259175+0000","last_peered":"2024-02-23T00:43:43.259175+0000","last_clean":"2024-02-23T00:43:43.259175+0000","last_became_active":"2024-02-23T00:43:33.261122+0000","last_became_peered":"2024-02-23T00:43:33.261122+0000","last_unstale":"2024-02-23T00:43:43.259175+0000","last_undegraded":"2024-02-23T00:43:43.259175+0000","last_fullsized":"2024-02-23T00:43:43.259175+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"252.1c","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.272484+0000","last_change":"2024-02-23T00:43:33.261561+0000","last_active":"2024-02-23T00:43:44.272484+0000","last_peered":"2024-02-23T00:43:44.272484+0000","last_clean":"2024-02-23T00:43:44.272484+0000","last_became_active":"2024-02-23T00:43:33.261225+0000","last_became_peered":"2024-02-23T00:43:33.261225+0000","last_unstale":"2024-02-23T00:43:44.272484+0000","last_undegraded":"2024-02-23T00:43:44.272484+0000","last_fullsized":"2024-02-23T00:43:44.272484+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"252.1b","version":"0'0","reported_seq":20,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893903+0000","last_change":"2024-02-23T00:43:33.263319+0000","last_active":"2024-02-23T00:43:43.893903+0000","last_peered":"2024-02-23T00:43:43.893903+0000","last_clean":"2024-02-23T00:43:43.893903+0000","last_became_active":"2024-02-23T00:43:33.263003+0000","last_became_peered":"2024-02-23T00:43:33.263003+0000","last_unstale":"2024-02-23T00:43:43.893903+0000","last_undegraded":"2024-02-23T00:43:43.893903+0000","last_fullsized":"2024-02-23T00:43:43.893903+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"252.1a","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.265644+0000","last_change":"2024-02-23T00:43:33.263281+0000","last_active":"2024-02-23T00:43:44.265644+0000","last_peered":"2024-02-23T00:43:44.265644+0000","last_clean":"2024-02-23T00:43:44.265644+0000","last_became_active":"2024-02-23T00:43:33.262931+0000","last_became_peered":"2024-02-23T00:43:33.262931+0000","last_unstale":"2024-02-23T00:43:44.265644+0000","last_undegraded":"2024-02-23T00:43:44.265644+0000","last_fullsized":"2024-02-23T00:43:44.265644+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"252.19","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748663+0000","last_change":"2024-02-23T00:43:33.269759+0000","last_active":"2024-02-23T00:43:44.748663+0000","last_peered":"2024-02-23T00:43:44.748663+0000","last_clean":"2024-02-23T00:43:44.748663+0000","last_became_active":"2024-02-23T00:43:33.269435+0000","last_became_peered":"2024-02-23T00:43:33.269435+0000","last_unstale":"2024-02-23T00:43:44.748663+0000","last_undegraded":"2024-02-23T00:43:44.748663+0000","last_fullsized":"2024-02-23T00:43:44.748663+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.18","version":"784'2","reported_seq":21,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.283604+0000","last_change":"2024-02-23T00:43:33.259145+0000","last_active":"2024-02-23T00:43:41.283604+0000","last_peered":"2024-02-23T00:43:41.283604+0000","last_clean":"2024-02-23T00:43:41.283604+0000","last_became_active":"2024-02-23T00:43:33.258856+0000","last_became_peered":"2024-02-23T00:43:33.258856+0000","last_unstale":"2024-02-23T00:43:41.283604+0000","last_undegraded":"2024-02-23T00:43:41.283604+0000","last_fullsized":"2024-02-23T00:43:41.283604+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.f","version":"784'2","reported_seq":21,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.284398+0000","last_change":"2024-02-23T00:43:33.264145+0000","last_active":"2024-02-23T00:43:41.284398+0000","last_peered":"2024-02-23T00:43:41.284398+0000","last_clean":"2024-02-23T00:43:41.284398+0000","last_became_active":"2024-02-23T00:43:33.263864+0000","last_became_peered":"2024-02-23T00:43:33.263864+0000","last_unstale":"2024-02-23T00:43:41.284398+0000","last_undegraded":"2024-02-23T00:43:41.284398+0000","last_fullsized":"2024-02-23T00:43:41.284398+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":1,"num_evict_kb":1,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.e","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.272678+0000","last_change":"2024-02-23T00:43:33.261424+0000","last_active":"2024-02-23T00:43:44.272678+0000","last_peered":"2024-02-23T00:43:44.272678+0000","last_clean":"2024-02-23T00:43:44.272678+0000","last_became_active":"2024-02-23T00:43:33.261165+0000","last_became_peered":"2024-02-23T00:43:33.261165+0000","last_unstale":"2024-02-23T00:43:44.272678+0000","last_undegraded":"2024-02-23T00:43:44.272678+0000","last_fullsized":"2024-02-23T00:43:44.272678+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.d","version":"0'0","reported_seq":18,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281369+0000","last_change":"2024-02-23T00:43:33.264802+0000","last_active":"2024-02-23T00:43:41.281369+0000","last_peered":"2024-02-23T00:43:41.281369+0000","last_clean":"2024-02-23T00:43:41.281369+0000","last_became_active":"2024-02-23T00:43:33.264522+0000","last_became_peered":"2024-02-23T00:43:33.264522+0000","last_unstale":"2024-02-23T00:43:41.281369+0000","last_undegraded":"2024-02-23T00:43:41.281369+0000","last_fullsized":"2024-02-23T00:43:41.281369+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.c","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273402+0000","last_change":"2024-02-23T00:43:33.257393+0000","last_active":"2024-02-23T00:43:44.273402+0000","last_peered":"2024-02-23T00:43:44.273402+0000","last_clean":"2024-02-23T00:43:44.273402+0000","last_became_active":"2024-02-23T00:43:33.257088+0000","last_became_peered":"2024-02-23T00:43:33.257088+0000","last_unstale":"2024-02-23T00:43:44.273402+0000","last_undegraded":"2024-02-23T00:43:44.273402+0000","last_fullsized":"2024-02-23T00:43:44.273402+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.b","version":"0'0","reported_seq":20,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258576+0000","last_change":"2024-02-23T00:43:33.261388+0000","last_active":"2024-02-23T00:43:43.258576+0000","last_peered":"2024-02-23T00:43:43.258576+0000","last_clean":"2024-02-23T00:43:43.258576+0000","last_became_active":"2024-02-23T00:43:33.261104+0000","last_became_peered":"2024-02-23T00:43:33.261104+0000","last_unstale":"2024-02-23T00:43:43.258576+0000","last_undegraded":"2024-02-23T00:43:43.258576+0000","last_fullsized":"2024-02-23T00:43:43.258576+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"252.a","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.270955+0000","last_change":"2024-02-23T00:43:33.257254+0000","last_active":"2024-02-23T00:43:44.270955+0000","last_peered":"2024-02-23T00:43:44.270955+0000","last_clean":"2024-02-23T00:43:44.270955+0000","last_became_active":"2024-02-23T00:43:33.256962+0000","last_became_peered":"2024-02-23T00:43:33.256962+0000","last_unstale":"2024-02-23T00:43:44.270955+0000","last_undegraded":"2024-02-23T00:43:44.270955+0000","last_fullsized":"2024-02-23T00:43:44.270955+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.9","version":"0'0","reported_seq":18,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.278683+0000","last_change":"2024-02-23T00:43:33.264849+0000","last_active":"2024-02-23T00:43:41.278683+0000","last_peered":"2024-02-23T00:43:41.278683+0000","last_clean":"2024-02-23T00:43:41.278683+0000","last_became_active":"2024-02-23T00:43:33.264543+0000","last_became_peered":"2024-02-23T00:43:33.264543+0000","last_unstale":"2024-02-23T00:43:41.278683+0000","last_undegraded":"2024-02-23T00:43:41.278683+0000","last_fullsized":"2024-02-23T00:43:41.278683+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.8","version":"0'0","reported_seq":18,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281586+0000","last_change":"2024-02-23T00:43:33.265106+0000","last_active":"2024-02-23T00:43:41.281586+0000","last_peered":"2024-02-23T00:43:41.281586+0000","last_clean":"2024-02-23T00:43:41.281586+0000","last_became_active":"2024-02-23T00:43:33.264554+0000","last_became_peered":"2024-02-23T00:43:33.264554+0000","last_unstale":"2024-02-23T00:43:41.281586+0000","last_undegraded":"2024-02-23T00:43:41.281586+0000","last_fullsized":"2024-02-23T00:43:41.281586+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.7","version":"0'0","reported_seq":20,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893983+0000","last_change":"2024-02-23T00:43:33.265445+0000","last_active":"2024-02-23T00:43:43.893983+0000","last_peered":"2024-02-23T00:43:43.893983+0000","last_clean":"2024-02-23T00:43:43.893983+0000","last_became_active":"2024-02-23T00:43:33.265159+0000","last_became_peered":"2024-02-23T00:43:33.265159+0000","last_unstale":"2024-02-23T00:43:43.893983+0000","last_undegraded":"2024-02-23T00:43:43.893983+0000","last_fullsized":"2024-02-23T00:43:43.893983+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"252.6","version":"0'0","reported_seq":18,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281583+0000","last_change":"2024-02-23T00:43:33.262853+0000","last_active":"2024-02-23T00:43:41.281583+0000","last_peered":"2024-02-23T00:43:41.281583+0000","last_clean":"2024-02-23T00:43:41.281583+0000","last_became_active":"2024-02-23T00:43:33.262581+0000","last_became_peered":"2024-02-23T00:43:33.262581+0000","last_unstale":"2024-02-23T00:43:41.281583+0000","last_undegraded":"2024-02-23T00:43:41.281583+0000","last_fullsized":"2024-02-23T00:43:41.281583+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"252.5","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748001+0000","last_change":"2024-02-23T00:43:33.265870+0000","last_active":"2024-02-23T00:43:44.748001+0000","last_peered":"2024-02-23T00:43:44.748001+0000","last_clean":"2024-02-23T00:43:44.748001+0000","last_became_active":"2024-02-23T00:43:33.265558+0000","last_became_peered":"2024-02-23T00:43:33.265558+0000","last_unstale":"2024-02-23T00:43:44.748001+0000","last_undegraded":"2024-02-23T00:43:44.748001+0000","last_fullsized":"2024-02-23T00:43:44.748001+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.4","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.272560+0000","last_change":"2024-02-23T00:43:33.260365+0000","last_active":"2024-02-23T00:43:44.272560+0000","last_peered":"2024-02-23T00:43:44.272560+0000","last_clean":"2024-02-23T00:43:44.272560+0000","last_became_active":"2024-02-23T00:43:33.260152+0000","last_became_peered":"2024-02-23T00:43:33.260152+0000","last_unstale":"2024-02-23T00:43:44.272560+0000","last_undegraded":"2024-02-23T00:43:44.272560+0000","last_fullsized":"2024-02-23T00:43:44.272560+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"252.3","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.266388+0000","last_change":"2024-02-23T00:43:33.265195+0000","last_active":"2024-02-23T00:43:44.266388+0000","last_peered":"2024-02-23T00:43:44.266388+0000","last_clean":"2024-02-23T00:43:44.266388+0000","last_became_active":"2024-02-23T00:43:33.264758+0000","last_became_peered":"2024-02-23T00:43:33.264758+0000","last_unstale":"2024-02-23T00:43:44.266388+0000","last_undegraded":"2024-02-23T00:43:44.266388+0000","last_fullsized":"2024-02-23T00:43:44.266388+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"252.2","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.747693+0000","last_change":"2024-02-23T00:43:33.265817+0000","last_active":"2024-02-23T00:43:44.747693+0000","last_peered":"2024-02-23T00:43:44.747693+0000","last_clean":"2024-02-23T00:43:44.747693+0000","last_became_active":"2024-02-23T00:43:33.265503+0000","last_became_peered":"2024-02-23T00:43:33.265503+0000","last_unstale":"2024-02-23T00:43:44.747693+0000","last_undegraded":"2024-02-23T00:43:44.747693+0000","last_fullsized":"2024-02-23T00:43:44.747693+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.1","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748746+0000","last_change":"2024-02-23T00:43:33.269482+0000","last_active":"2024-02-23T00:43:44.748746+0000","last_peered":"2024-02-23T00:43:44.748746+0000","last_clean":"2024-02-23T00:43:44.748746+0000","last_became_active":"2024-02-23T00:43:33.269189+0000","last_became_peered":"2024-02-23T00:43:33.269189+0000","last_unstale":"2024-02-23T00:43:44.748746+0000","last_undegraded":"2024-02-23T00:43:44.748746+0000","last_fullsized":"2024-02-23T00:43:44.748746+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.0","version":"779'2","reported_seq":23,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.432443+0000","last_change":"2024-02-23T00:43:33.269447+0000","last_active":"2024-02-23T00:43:43.432443+0000","last_peered":"2024-02-23T00:43:43.432443+0000","last_clean":"2024-02-23T00:43:43.432443+0000","last_became_active":"2024-02-23T00:43:33.268825+0000","last_became_peered":"2024-02-23T00:43:33.268825+0000","last_unstale":"2024-02-23T00:43:43.432443+0000","last_undegraded":"2024-02-23T00:43:43.432443+0000","last_fullsized":"2024-02-23T00:43:43.432443+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"252.11","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.269835+0000","last_change":"2024-02-23T00:43:33.261826+0000","last_active":"2024-02-23T00:43:44.269835+0000","last_peered":"2024-02-23T00:43:44.269835+0000","last_clean":"2024-02-23T00:43:44.269835+0000","last_became_active":"2024-02-23T00:43:33.261546+0000","last_became_peered":"2024-02-23T00:43:33.261546+0000","last_unstale":"2024-02-23T00:43:44.269835+0000","last_undegraded":"2024-02-23T00:43:44.269835+0000","last_fullsized":"2024-02-23T00:43:44.269835+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.7","version":"776'11","reported_seq":242,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745590+0000","last_change":"2024-02-23T00:42:21.267470+0000","last_active":"2024-02-23T00:43:44.745590+0000","last_peered":"2024-02-23T00:43:44.745590+0000","last_clean":"2024-02-23T00:43:44.745590+0000","last_became_active":"2024-02-23T00:40:17.211405+0000","last_became_peered":"2024-02-23T00:40:17.211405+0000","last_unstale":"2024-02-23T00:43:44.745590+0000","last_undegraded":"2024-02-23T00:43:44.745590+0000","last_fullsized":"2024-02-23T00:43:44.745590+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'9","last_scrub_stamp":"2024-02-23T00:42:21.267354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:21.267354+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"252.10","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273140+0000","last_change":"2024-02-23T00:43:33.262132+0000","last_active":"2024-02-23T00:43:44.273140+0000","last_peered":"2024-02-23T00:43:44.273140+0000","last_clean":"2024-02-23T00:43:44.273140+0000","last_became_active":"2024-02-23T00:43:33.261845+0000","last_became_peered":"2024-02-23T00:43:33.261845+0000","last_unstale":"2024-02-23T00:43:44.273140+0000","last_undegraded":"2024-02-23T00:43:44.273140+0000","last_fullsized":"2024-02-23T00:43:44.273140+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.6","version":"751'19","reported_seq":251,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433269+0000","last_change":"2024-02-23T00:43:10.751724+0000","last_active":"2024-02-23T00:43:43.433269+0000","last_peered":"2024-02-23T00:43:43.433269+0000","last_clean":"2024-02-23T00:43:43.433269+0000","last_became_active":"2024-02-23T00:40:17.215263+0000","last_became_peered":"2024-02-23T00:40:17.215263+0000","last_unstale":"2024-02-23T00:43:43.433269+0000","last_undegraded":"2024-02-23T00:43:43.433269+0000","last_fullsized":"2024-02-23T00:43:43.433269+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"751'19","last_scrub_stamp":"2024-02-23T00:43:10.751620+0000","last_deep_scrub":"751'19","last_deep_scrub_stamp":"2024-02-23T00:43:10.751620+0000","last_clean_scrub_stamp":"2024-02-23T00:43:10.751620+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"252.13","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271066+0000","last_change":"2024-02-23T00:43:33.264755+0000","last_active":"2024-02-23T00:43:44.271066+0000","last_peered":"2024-02-23T00:43:44.271066+0000","last_clean":"2024-02-23T00:43:44.271066+0000","last_became_active":"2024-02-23T00:43:33.264455+0000","last_became_peered":"2024-02-23T00:43:33.264455+0000","last_unstale":"2024-02-23T00:43:44.271066+0000","last_undegraded":"2024-02-23T00:43:44.271066+0000","last_fullsized":"2024-02-23T00:43:44.271066+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.5","version":"751'7","reported_seq":252,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.270974+0000","last_change":"2024-02-23T00:42:13.431865+0000","last_active":"2024-02-23T00:43:44.270974+0000","last_peered":"2024-02-23T00:43:44.270974+0000","last_clean":"2024-02-23T00:43:44.270974+0000","last_became_active":"2024-02-23T00:41:37.238894+0000","last_became_peered":"2024-02-23T00:41:37.238894+0000","last_unstale":"2024-02-23T00:43:44.270974+0000","last_undegraded":"2024-02-23T00:43:44.270974+0000","last_fullsized":"2024-02-23T00:43:44.270974+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"635'5","last_scrub_stamp":"2024-02-23T00:42:13.431791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.431791+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"252.12","version":"0'0","reported_seq":20,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258416+0000","last_change":"2024-02-23T00:43:33.258720+0000","last_active":"2024-02-23T00:43:43.258416+0000","last_peered":"2024-02-23T00:43:43.258416+0000","last_clean":"2024-02-23T00:43:43.258416+0000","last_became_active":"2024-02-23T00:43:33.258435+0000","last_became_peered":"2024-02-23T00:43:33.258435+0000","last_unstale":"2024-02-23T00:43:43.258416+0000","last_undegraded":"2024-02-23T00:43:43.258416+0000","last_fullsized":"2024-02-23T00:43:43.258416+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.4","version":"765'24","reported_seq":248,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273495+0000","last_change":"2024-02-23T00:42:12.255652+0000","last_active":"2024-02-23T00:43:44.273495+0000","last_peered":"2024-02-23T00:43:44.273495+0000","last_clean":"2024-02-23T00:43:44.273495+0000","last_became_active":"2024-02-23T00:40:17.215235+0000","last_became_peered":"2024-02-23T00:40:17.215235+0000","last_unstale":"2024-02-23T00:43:44.273495+0000","last_undegraded":"2024-02-23T00:43:44.273495+0000","last_fullsized":"2024-02-23T00:43:44.273495+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:40:16.171649+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000055,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":15,"num_write_kb":3921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.15","version":"0'0","reported_seq":20,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433258+0000","last_change":"2024-02-23T00:43:33.267884+0000","last_active":"2024-02-23T00:43:43.433258+0000","last_peered":"2024-02-23T00:43:43.433258+0000","last_clean":"2024-02-23T00:43:43.433258+0000","last_became_active":"2024-02-23T00:43:33.267574+0000","last_became_peered":"2024-02-23T00:43:33.267574+0000","last_unstale":"2024-02-23T00:43:43.433258+0000","last_undegraded":"2024-02-23T00:43:43.433258+0000","last_fullsized":"2024-02-23T00:43:43.433258+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.3","version":"751'7","reported_seq":229,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893278+0000","last_change":"2024-02-23T00:42:18.042228+0000","last_active":"2024-02-23T00:43:43.893278+0000","last_peered":"2024-02-23T00:43:43.893278+0000","last_clean":"2024-02-23T00:43:43.893278+0000","last_became_active":"2024-02-23T00:40:17.202244+0000","last_became_peered":"2024-02-23T00:40:17.202244+0000","last_unstale":"2024-02-23T00:43:43.893278+0000","last_undegraded":"2024-02-23T00:43:43.893278+0000","last_fullsized":"2024-02-23T00:43:43.893278+0000","mapping_epoch":597,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'2","last_scrub_stamp":"2024-02-23T00:42:18.042156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.042156+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"252.14","version":"0'0","reported_seq":20,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433195+0000","last_change":"2024-02-23T00:43:33.269297+0000","last_active":"2024-02-23T00:43:43.433195+0000","last_peered":"2024-02-23T00:43:43.433195+0000","last_clean":"2024-02-23T00:43:43.433195+0000","last_became_active":"2024-02-23T00:43:33.268960+0000","last_became_peered":"2024-02-23T00:43:33.268960+0000","last_unstale":"2024-02-23T00:43:43.433195+0000","last_undegraded":"2024-02-23T00:43:43.433195+0000","last_fullsized":"2024-02-23T00:43:43.433195+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.2","version":"751'10","reported_seq":258,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.432767+0000","last_change":"2024-02-23T00:43:17.669150+0000","last_active":"2024-02-23T00:43:43.432767+0000","last_peered":"2024-02-23T00:43:43.432767+0000","last_clean":"2024-02-23T00:43:43.432767+0000","last_became_active":"2024-02-23T00:41:37.219553+0000","last_became_peered":"2024-02-23T00:41:37.219553+0000","last_unstale":"2024-02-23T00:43:43.432767+0000","last_undegraded":"2024-02-23T00:43:43.432767+0000","last_fullsized":"2024-02-23T00:43:43.432767+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"751'10","last_scrub_stamp":"2024-02-23T00:43:17.669031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:43:17.669031+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"252.17","version":"779'2","reported_seq":24,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.266367+0000","last_change":"2024-02-23T00:43:33.264807+0000","last_active":"2024-02-23T00:43:44.266367+0000","last_peered":"2024-02-23T00:43:44.266367+0000","last_clean":"2024-02-23T00:43:44.266367+0000","last_became_active":"2024-02-23T00:43:33.264479+0000","last_became_peered":"2024-02-23T00:43:33.264479+0000","last_unstale":"2024-02-23T00:43:44.266367+0000","last_undegraded":"2024-02-23T00:43:44.266367+0000","last_fullsized":"2024-02-23T00:43:44.266367+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":1,"num_flush_mode_low":0,"num_evict_mode_some":1,"num_evict_mode_full":0,"num_objects_pinned":1,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.1","version":"751'4","reported_seq":248,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.270086+0000","last_change":"2024-02-23T00:43:10.169193+0000","last_active":"2024-02-23T00:43:44.270086+0000","last_peered":"2024-02-23T00:43:44.270086+0000","last_clean":"2024-02-23T00:43:44.270086+0000","last_became_active":"2024-02-23T00:41:37.231330+0000","last_became_peered":"2024-02-23T00:41:37.231330+0000","last_unstale":"2024-02-23T00:43:44.270086+0000","last_undegraded":"2024-02-23T00:43:44.270086+0000","last_fullsized":"2024-02-23T00:43:44.270086+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"751'4","last_scrub_stamp":"2024-02-23T00:43:10.169118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:43:10.169118+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"252.16","version":"0'0","reported_seq":21,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.746686+0000","last_change":"2024-02-23T00:43:33.266828+0000","last_active":"2024-02-23T00:43:44.746686+0000","last_peered":"2024-02-23T00:43:44.746686+0000","last_clean":"2024-02-23T00:43:44.746686+0000","last_became_active":"2024-02-23T00:43:33.266515+0000","last_became_peered":"2024-02-23T00:43:33.266515+0000","last_unstale":"2024-02-23T00:43:44.746686+0000","last_undegraded":"2024-02-23T00:43:44.746686+0000","last_fullsized":"2024-02-23T00:43:44.746686+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":775,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:32.240009+0000","last_clean_scrub_stamp":"2024-02-23T00:43:32.240009+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.0","version":"776'13","reported_seq":236,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.432551+0000","last_change":"2024-02-23T00:43:15.728719+0000","last_active":"2024-02-23T00:43:43.432551+0000","last_peered":"2024-02-23T00:43:43.432551+0000","last_clean":"2024-02-23T00:43:43.432551+0000","last_became_active":"2024-02-23T00:41:50.253705+0000","last_became_peered":"2024-02-23T00:41:50.253705+0000","last_unstale":"2024-02-23T00:43:43.432551+0000","last_undegraded":"2024-02-23T00:43:43.432551+0000","last_fullsized":"2024-02-23T00:43:43.432551+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":597,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"751'11","last_scrub_stamp":"2024-02-23T00:43:15.728622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:16.171649+0000","last_clean_scrub_stamp":"2024-02-23T00:43:15.728622+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":368,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745483+0000","last_change":"2024-02-23T00:42:58.676506+0000","last_active":"2024-02-23T00:43:44.745483+0000","last_peered":"2024-02-23T00:43:44.745483+0000","last_clean":"2024-02-23T00:43:44.745483+0000","last_became_active":"2024-02-23T00:42:41.400209+0000","last_became_peered":"2024-02-23T00:42:41.400209+0000","last_unstale":"2024-02-23T00:43:44.745483+0000","last_undegraded":"2024-02-23T00:43:44.745483+0000","last_fullsized":"2024-02-23T00:43:44.745483+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:58.676432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:58.676432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"571'11","reported_seq":1001,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-23T00:43:45.330283+0000","last_change":"2024-02-23T00:42:18.360637+0000","last_active":"2024-02-23T00:43:45.330283+0000","last_peered":"2024-02-23T00:43:45.330283+0000","last_clean":"2024-02-23T00:43:45.330283+0000","last_became_active":"2024-02-23T00:39:49.195036+0000","last_became_peered":"2024-02-23T00:39:49.195036+0000","last_unstale":"2024-02-23T00:43:45.330283+0000","last_undegraded":"2024-02-23T00:43:45.330283+0000","last_fullsized":"2024-02-23T00:43:45.330283+0000","mapping_epoch":58,"log_start":"379'9","ondisk_log_start":"379'9","created":375,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'11","last_scrub_stamp":"2024-02-23T00:42:18.360544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.360544+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":877,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893045+0000","last_change":"2024-02-23T00:42:16.067711+0000","last_active":"2024-02-23T00:43:43.893045+0000","last_peered":"2024-02-23T00:43:43.893045+0000","last_clean":"2024-02-23T00:43:43.893045+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:43:43.893045+0000","last_undegraded":"2024-02-23T00:43:43.893045+0000","last_fullsized":"2024-02-23T00:43:43.893045+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:42:16.067590+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.067590+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":810,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271952+0000","last_change":"2024-02-23T00:43:18.439448+0000","last_active":"2024-02-23T00:43:44.271952+0000","last_peered":"2024-02-23T00:43:44.271952+0000","last_clean":"2024-02-23T00:43:44.271952+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:43:44.271952+0000","last_undegraded":"2024-02-23T00:43:44.271952+0000","last_fullsized":"2024-02-23T00:43:44.271952+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:18.439349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:43:18.439349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":791,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893181+0000","last_change":"2024-02-23T00:42:54.046985+0000","last_active":"2024-02-23T00:43:43.893181+0000","last_peered":"2024-02-23T00:43:43.893181+0000","last_clean":"2024-02-23T00:43:43.893181+0000","last_became_active":"2024-02-23T00:41:37.222092+0000","last_became_peered":"2024-02-23T00:41:37.222092+0000","last_unstale":"2024-02-23T00:43:43.893181+0000","last_undegraded":"2024-02-23T00:43:43.893181+0000","last_fullsized":"2024-02-23T00:43:43.893181+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:54.046903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.046903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":739,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258770+0000","last_change":"2024-02-23T00:43:06.398049+0000","last_active":"2024-02-23T00:43:43.258770+0000","last_peered":"2024-02-23T00:43:43.258770+0000","last_clean":"2024-02-23T00:43:43.258770+0000","last_became_active":"2024-02-23T00:41:37.214214+0000","last_became_peered":"2024-02-23T00:41:37.214214+0000","last_unstale":"2024-02-23T00:43:43.258770+0000","last_undegraded":"2024-02-23T00:43:43.258770+0000","last_fullsized":"2024-02-23T00:43:43.258770+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.397979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.397979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":1437,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.634489+0000","last_change":"2024-02-23T00:41:39.898258+0000","last_active":"2024-02-23T00:43:44.634489+0000","last_peered":"2024-02-23T00:43:44.634489+0000","last_clean":"2024-02-23T00:43:44.634489+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:43:44.634489+0000","last_undegraded":"2024-02-23T00:43:44.634489+0000","last_fullsized":"2024-02-23T00:43:44.634489+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:41:39.898173+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:41:39.898173+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":665,"num_read_kb":665,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":493,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.280829+0000","last_change":"2024-02-23T00:42:15.735663+0000","last_active":"2024-02-23T00:43:41.280829+0000","last_peered":"2024-02-23T00:43:41.280829+0000","last_clean":"2024-02-23T00:43:41.280829+0000","last_became_active":"2024-02-23T00:41:41.226195+0000","last_became_peered":"2024-02-23T00:41:41.226195+0000","last_unstale":"2024-02-23T00:43:41.280829+0000","last_undegraded":"2024-02-23T00:43:41.280829+0000","last_fullsized":"2024-02-23T00:43:41.280829+0000","mapping_epoch":681,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:42:15.735606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.735606+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":782,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259519+0000","last_change":"2024-02-23T00:42:29.119245+0000","last_active":"2024-02-23T00:43:43.259519+0000","last_peered":"2024-02-23T00:43:43.259519+0000","last_clean":"2024-02-23T00:43:43.259519+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:43:43.259519+0000","last_undegraded":"2024-02-23T00:43:43.259519+0000","last_fullsized":"2024-02-23T00:43:43.259519+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:29.119169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:29.119169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":797,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.270682+0000","last_change":"2024-02-23T00:42:56.444185+0000","last_active":"2024-02-23T00:43:44.270682+0000","last_peered":"2024-02-23T00:43:44.270682+0000","last_clean":"2024-02-23T00:43:44.270682+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:43:44.270682+0000","last_undegraded":"2024-02-23T00:43:44.270682+0000","last_fullsized":"2024-02-23T00:43:44.270682+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:56.444091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:42:56.444091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":126,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.280357+0000","last_change":"2024-02-23T00:43:13.630793+0000","last_active":"2024-02-23T00:43:41.280357+0000","last_peered":"2024-02-23T00:43:41.280357+0000","last_clean":"2024-02-23T00:43:41.280357+0000","last_became_active":"2024-02-23T00:41:37.228937+0000","last_became_peered":"2024-02-23T00:41:37.228937+0000","last_unstale":"2024-02-23T00:43:41.280357+0000","last_undegraded":"2024-02-23T00:43:41.280357+0000","last_fullsized":"2024-02-23T00:43:41.280357+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:13.630679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:43:13.630679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":720,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433091+0000","last_change":"2024-02-23T00:42:58.645999+0000","last_active":"2024-02-23T00:43:43.433091+0000","last_peered":"2024-02-23T00:43:43.433091+0000","last_clean":"2024-02-23T00:43:43.433091+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:43:43.433091+0000","last_undegraded":"2024-02-23T00:43:43.433091+0000","last_fullsized":"2024-02-23T00:43:43.433091+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:58.645916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:42:58.645916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":63,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748085+0000","last_change":"2024-02-23T00:43:12.778764+0000","last_active":"2024-02-23T00:43:44.748085+0000","last_peered":"2024-02-23T00:43:44.748085+0000","last_clean":"2024-02-23T00:43:44.748085+0000","last_became_active":"2024-02-23T00:42:41.393491+0000","last_became_peered":"2024-02-23T00:42:41.393491+0000","last_unstale":"2024-02-23T00:43:44.748085+0000","last_undegraded":"2024-02-23T00:43:44.748085+0000","last_fullsized":"2024-02-23T00:43:44.748085+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:12.778695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:43:12.778695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":899,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433032+0000","last_change":"2024-02-23T00:43:00.587024+0000","last_active":"2024-02-23T00:43:43.433032+0000","last_peered":"2024-02-23T00:43:43.433032+0000","last_clean":"2024-02-23T00:43:43.433032+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:43:43.433032+0000","last_undegraded":"2024-02-23T00:43:43.433032+0000","last_fullsized":"2024-02-23T00:43:43.433032+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:43:00.586920+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.586920+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":803,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.272908+0000","last_change":"2024-02-23T00:42:24.447799+0000","last_active":"2024-02-23T00:43:44.272908+0000","last_peered":"2024-02-23T00:43:44.272908+0000","last_clean":"2024-02-23T00:43:44.272908+0000","last_became_active":"2024-02-23T00:41:37.238728+0000","last_became_peered":"2024-02-23T00:41:37.238728+0000","last_unstale":"2024-02-23T00:43:44.272908+0000","last_undegraded":"2024-02-23T00:43:44.272908+0000","last_fullsized":"2024-02-23T00:43:44.272908+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:24.447731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:24.447731+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":792,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.270774+0000","last_change":"2024-02-23T00:42:17.473540+0000","last_active":"2024-02-23T00:43:44.270774+0000","last_peered":"2024-02-23T00:43:44.270774+0000","last_clean":"2024-02-23T00:43:44.270774+0000","last_became_active":"2024-02-23T00:41:37.239529+0000","last_became_peered":"2024-02-23T00:41:37.239529+0000","last_unstale":"2024-02-23T00:43:44.270774+0000","last_undegraded":"2024-02-23T00:43:44.270774+0000","last_fullsized":"2024-02-23T00:43:44.270774+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:17.473474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.473474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":798,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745624+0000","last_change":"2024-02-23T00:41:49.588387+0000","last_active":"2024-02-23T00:43:44.745624+0000","last_peered":"2024-02-23T00:43:44.745624+0000","last_clean":"2024-02-23T00:43:44.745624+0000","last_became_active":"2024-02-23T00:40:37.197010+0000","last_became_peered":"2024-02-23T00:40:37.197010+0000","last_unstale":"2024-02-23T00:43:44.745624+0000","last_undegraded":"2024-02-23T00:43:44.745624+0000","last_fullsized":"2024-02-23T00:43:44.745624+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:49.588290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.588290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":121,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.279015+0000","last_change":"2024-02-23T00:42:57.587635+0000","last_active":"2024-02-23T00:43:41.279015+0000","last_peered":"2024-02-23T00:43:41.279015+0000","last_clean":"2024-02-23T00:43:41.279015+0000","last_became_active":"2024-02-23T00:41:37.248272+0000","last_became_peered":"2024-02-23T00:41:37.248272+0000","last_unstale":"2024-02-23T00:43:41.279015+0000","last_undegraded":"2024-02-23T00:43:41.279015+0000","last_fullsized":"2024-02-23T00:43:41.279015+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:57.587566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:57.587566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":784,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433333+0000","last_change":"2024-02-23T00:42:54.599143+0000","last_active":"2024-02-23T00:43:43.433333+0000","last_peered":"2024-02-23T00:43:43.433333+0000","last_clean":"2024-02-23T00:43:43.433333+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:43:43.433333+0000","last_undegraded":"2024-02-23T00:43:43.433333+0000","last_fullsized":"2024-02-23T00:43:43.433333+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:54.599070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.599070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":614,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271297+0000","last_change":"2024-02-23T00:42:19.485117+0000","last_active":"2024-02-23T00:43:44.271297+0000","last_peered":"2024-02-23T00:43:44.271297+0000","last_clean":"2024-02-23T00:43:44.271297+0000","last_became_active":"2024-02-23T00:41:50.256494+0000","last_became_peered":"2024-02-23T00:41:50.256494+0000","last_unstale":"2024-02-23T00:43:44.271297+0000","last_undegraded":"2024-02-23T00:43:44.271297+0000","last_fullsized":"2024-02-23T00:43:44.271297+0000","mapping_epoch":688,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:42:19.485046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.485046+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":768,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745040+0000","last_change":"2024-02-23T00:41:47.545036+0000","last_active":"2024-02-23T00:43:44.745040+0000","last_peered":"2024-02-23T00:43:44.745040+0000","last_clean":"2024-02-23T00:43:44.745040+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:43:44.745040+0000","last_undegraded":"2024-02-23T00:43:44.745040+0000","last_fullsized":"2024-02-23T00:43:44.745040+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.544952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.544952+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":774,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259430+0000","last_change":"2024-02-23T00:41:37.229878+0000","last_active":"2024-02-23T00:43:43.259430+0000","last_peered":"2024-02-23T00:43:43.259430+0000","last_clean":"2024-02-23T00:43:43.259430+0000","last_became_active":"2024-02-23T00:41:37.229416+0000","last_became_peered":"2024-02-23T00:41:37.229416+0000","last_unstale":"2024-02-23T00:43:43.259430+0000","last_undegraded":"2024-02-23T00:43:43.259430+0000","last_fullsized":"2024-02-23T00:43:43.259430+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:00.151031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:00.151031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":10,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-02-23T00:43:42.260695+0000","last_change":"2024-02-23T00:43:42.260695+0000","last_active":"2024-02-23T00:43:42.260695+0000","last_peered":"2024-02-23T00:43:42.260695+0000","last_clean":"2024-02-23T00:43:42.260695+0000","last_became_active":"2024-02-23T00:43:42.260383+0000","last_became_peered":"2024-02-23T00:43:42.260383+0000","last_unstale":"2024-02-23T00:43:42.260695+0000","last_undegraded":"2024-02-23T00:43:42.260695+0000","last_fullsized":"2024-02-23T00:43:42.260695+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.514031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.514031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":305,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.274311+0000","last_change":"2024-02-23T00:41:37.253004+0000","last_active":"2024-02-23T00:43:44.274311+0000","last_peered":"2024-02-23T00:43:44.274311+0000","last_clean":"2024-02-23T00:43:44.274311+0000","last_became_active":"2024-02-23T00:41:37.252083+0000","last_became_peered":"2024-02-23T00:41:37.252083+0000","last_unstale":"2024-02-23T00:43:44.274311+0000","last_undegraded":"2024-02-23T00:43:44.274311+0000","last_fullsized":"2024-02-23T00:43:44.274311+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:35:47.196196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:35:47.196196+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":714,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273585+0000","last_change":"2024-02-23T00:41:50.256382+0000","last_active":"2024-02-23T00:43:44.273585+0000","last_peered":"2024-02-23T00:43:44.273585+0000","last_clean":"2024-02-23T00:43:44.273585+0000","last_became_active":"2024-02-23T00:41:50.256004+0000","last_became_peered":"2024-02-23T00:41:50.256004+0000","last_unstale":"2024-02-23T00:43:44.273585+0000","last_undegraded":"2024-02-23T00:43:44.273585+0000","last_fullsized":"2024-02-23T00:43:44.273585+0000","mapping_epoch":688,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:40:51.632948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:51.632948+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":795,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258595+0000","last_change":"2024-02-23T00:42:15.223520+0000","last_active":"2024-02-23T00:43:43.258595+0000","last_peered":"2024-02-23T00:43:43.258595+0000","last_clean":"2024-02-23T00:43:43.258595+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:43:43.258595+0000","last_undegraded":"2024-02-23T00:43:43.258595+0000","last_fullsized":"2024-02-23T00:43:43.258595+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.223455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.223455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":984,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-23T00:43:45.272163+0000","last_change":"2024-02-23T00:42:55.471736+0000","last_active":"2024-02-23T00:43:45.272163+0000","last_peered":"2024-02-23T00:43:45.272163+0000","last_clean":"2024-02-23T00:43:45.272163+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:43:45.272163+0000","last_undegraded":"2024-02-23T00:43:45.272163+0000","last_fullsized":"2024-02-23T00:43:45.272163+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:42:55.471653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:55.471653+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":720,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.267295+0000","last_change":"2024-02-23T00:42:52.938985+0000","last_active":"2024-02-23T00:43:44.267295+0000","last_peered":"2024-02-23T00:43:44.267295+0000","last_clean":"2024-02-23T00:43:44.267295+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:43:44.267295+0000","last_undegraded":"2024-02-23T00:43:44.267295+0000","last_fullsized":"2024-02-23T00:43:44.267295+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:52.938885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:52.938885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":291,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.269949+0000","last_change":"2024-02-23T00:40:15.208236+0000","last_active":"2024-02-23T00:43:44.269949+0000","last_peered":"2024-02-23T00:43:44.269949+0000","last_clean":"2024-02-23T00:43:44.269949+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:43:44.269949+0000","last_undegraded":"2024-02-23T00:43:44.269949+0000","last_fullsized":"2024-02-23T00:43:44.269949+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:15.208124+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:15.208124+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":949,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259886+0000","last_change":"2024-02-23T00:41:08.116872+0000","last_active":"2024-02-23T00:43:43.259886+0000","last_peered":"2024-02-23T00:43:43.259886+0000","last_clean":"2024-02-23T00:43:43.259886+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:43:43.259886+0000","last_undegraded":"2024-02-23T00:43:43.259886+0000","last_fullsized":"2024-02-23T00:43:43.259886+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:41:08.116765+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:41:08.116765+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":779,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258246+0000","last_change":"2024-02-23T00:42:14.253417+0000","last_active":"2024-02-23T00:43:43.258246+0000","last_peered":"2024-02-23T00:43:43.258246+0000","last_clean":"2024-02-23T00:43:43.258246+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:43:43.258246+0000","last_undegraded":"2024-02-23T00:43:43.258246+0000","last_fullsized":"2024-02-23T00:43:43.258246+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.253319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.253319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"689'12","reported_seq":986,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281703+0000","last_change":"2024-02-23T00:42:19.683007+0000","last_active":"2024-02-23T00:43:41.281703+0000","last_peered":"2024-02-23T00:43:41.281703+0000","last_clean":"2024-02-23T00:43:41.281703+0000","last_became_active":"2024-02-23T00:41:50.256773+0000","last_became_peered":"2024-02-23T00:41:50.256773+0000","last_unstale":"2024-02-23T00:43:41.281703+0000","last_undegraded":"2024-02-23T00:43:41.281703+0000","last_fullsized":"2024-02-23T00:43:41.281703+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'12","last_scrub_stamp":"2024-02-23T00:42:19.682902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.682902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":706,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259307+0000","last_change":"2024-02-23T00:41:40.171696+0000","last_active":"2024-02-23T00:43:43.259307+0000","last_peered":"2024-02-23T00:43:43.259307+0000","last_clean":"2024-02-23T00:43:43.259307+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:43:43.259307+0000","last_undegraded":"2024-02-23T00:43:43.259307+0000","last_fullsized":"2024-02-23T00:43:43.259307+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:40.171630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:41:40.171630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":782,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.267319+0000","last_change":"2024-02-23T00:42:29.905840+0000","last_active":"2024-02-23T00:43:44.267319+0000","last_peered":"2024-02-23T00:43:44.267319+0000","last_clean":"2024-02-23T00:43:44.267319+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:43:44.267319+0000","last_undegraded":"2024-02-23T00:43:44.267319+0000","last_fullsized":"2024-02-23T00:43:44.267319+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:29.905710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:29.905710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":620,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433417+0000","last_change":"2024-02-23T00:42:18.659610+0000","last_active":"2024-02-23T00:43:43.433417+0000","last_peered":"2024-02-23T00:43:43.433417+0000","last_clean":"2024-02-23T00:43:43.433417+0000","last_became_active":"2024-02-23T00:41:50.253028+0000","last_became_peered":"2024-02-23T00:41:50.253028+0000","last_unstale":"2024-02-23T00:43:43.433417+0000","last_undegraded":"2024-02-23T00:43:43.433417+0000","last_fullsized":"2024-02-23T00:43:43.433417+0000","mapping_epoch":688,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:42:18.659534+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:42:18.659534+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":809,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745956+0000","last_change":"2024-02-23T00:42:54.633894+0000","last_active":"2024-02-23T00:43:44.745956+0000","last_peered":"2024-02-23T00:43:44.745956+0000","last_clean":"2024-02-23T00:43:44.745956+0000","last_became_active":"2024-02-23T00:41:37.229862+0000","last_became_peered":"2024-02-23T00:41:37.229862+0000","last_unstale":"2024-02-23T00:43:44.745956+0000","last_undegraded":"2024-02-23T00:43:44.745956+0000","last_fullsized":"2024-02-23T00:43:44.745956+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:42:54.633815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.633815+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":756,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.894200+0000","last_change":"2024-02-23T00:43:06.143533+0000","last_active":"2024-02-23T00:43:43.894200+0000","last_peered":"2024-02-23T00:43:43.894200+0000","last_clean":"2024-02-23T00:43:43.894200+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:43:43.894200+0000","last_undegraded":"2024-02-23T00:43:43.894200+0000","last_fullsized":"2024-02-23T00:43:43.894200+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.143455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.143455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":729,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259502+0000","last_change":"2024-02-23T00:42:53.372480+0000","last_active":"2024-02-23T00:43:43.259502+0000","last_peered":"2024-02-23T00:43:43.259502+0000","last_clean":"2024-02-23T00:43:43.259502+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:43:43.259502+0000","last_undegraded":"2024-02-23T00:43:43.259502+0000","last_fullsized":"2024-02-23T00:43:43.259502+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.372422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.372422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":135,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281973+0000","last_change":"2024-02-23T00:43:03.634296+0000","last_active":"2024-02-23T00:43:41.281973+0000","last_peered":"2024-02-23T00:43:41.281973+0000","last_clean":"2024-02-23T00:43:41.281973+0000","last_became_active":"2024-02-23T00:42:39.394104+0000","last_became_peered":"2024-02-23T00:42:39.394104+0000","last_unstale":"2024-02-23T00:43:41.281973+0000","last_undegraded":"2024-02-23T00:43:41.281973+0000","last_fullsized":"2024-02-23T00:43:41.281973+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:03.634240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:43:03.634240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"642'14","reported_seq":344,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-23T00:43:45.269875+0000","last_change":"2024-02-23T00:42:59.176336+0000","last_active":"2024-02-23T00:43:45.269875+0000","last_peered":"2024-02-23T00:43:45.269875+0000","last_clean":"2024-02-23T00:43:45.269875+0000","last_became_active":"2024-02-23T00:41:01.212780+0000","last_became_peered":"2024-02-23T00:41:01.212780+0000","last_unstale":"2024-02-23T00:43:45.269875+0000","last_undegraded":"2024-02-23T00:43:45.269875+0000","last_fullsized":"2024-02-23T00:43:45.269875+0000","mapping_epoch":511,"log_start":"562'12","ondisk_log_start":"562'12","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'14","last_scrub_stamp":"2024-02-23T00:42:59.176246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.176246+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":592,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893628+0000","last_change":"2024-02-23T00:42:19.049262+0000","last_active":"2024-02-23T00:43:43.893628+0000","last_peered":"2024-02-23T00:43:43.893628+0000","last_clean":"2024-02-23T00:43:43.893628+0000","last_became_active":"2024-02-23T00:41:50.255315+0000","last_became_peered":"2024-02-23T00:41:50.255315+0000","last_unstale":"2024-02-23T00:43:43.893628+0000","last_undegraded":"2024-02-23T00:43:43.893628+0000","last_fullsized":"2024-02-23T00:43:43.893628+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'34","last_scrub_stamp":"2024-02-23T00:42:19.049190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.049190+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":772,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433669+0000","last_change":"2024-02-23T00:42:52.538341+0000","last_active":"2024-02-23T00:43:43.433669+0000","last_peered":"2024-02-23T00:43:43.433669+0000","last_clean":"2024-02-23T00:43:43.433669+0000","last_became_active":"2024-02-23T00:27:17.007552+0000","last_became_peered":"2024-02-23T00:27:17.007552+0000","last_unstale":"2024-02-23T00:43:43.433669+0000","last_undegraded":"2024-02-23T00:43:43.433669+0000","last_fullsized":"2024-02-23T00:43:43.433669+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:52.538255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:52.538255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":785,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745482+0000","last_change":"2024-02-23T00:42:12.433222+0000","last_active":"2024-02-23T00:43:44.745482+0000","last_peered":"2024-02-23T00:43:44.745482+0000","last_clean":"2024-02-23T00:43:44.745482+0000","last_became_active":"2024-02-23T00:40:57.488797+0000","last_became_peered":"2024-02-23T00:40:57.488797+0000","last_unstale":"2024-02-23T00:43:44.745482+0000","last_undegraded":"2024-02-23T00:43:44.745482+0000","last_fullsized":"2024-02-23T00:43:44.745482+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.433157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.433157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":749,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259155+0000","last_change":"2024-02-23T00:41:51.307510+0000","last_active":"2024-02-23T00:43:43.259155+0000","last_peered":"2024-02-23T00:43:43.259155+0000","last_clean":"2024-02-23T00:43:43.259155+0000","last_became_active":"2024-02-23T00:41:37.229061+0000","last_became_peered":"2024-02-23T00:41:37.229061+0000","last_unstale":"2024-02-23T00:43:43.259155+0000","last_undegraded":"2024-02-23T00:43:43.259155+0000","last_fullsized":"2024-02-23T00:43:43.259155+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:51.307443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:51.307443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":326,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271430+0000","last_change":"2024-02-23T00:42:39.396325+0000","last_active":"2024-02-23T00:43:44.271430+0000","last_peered":"2024-02-23T00:43:44.271430+0000","last_clean":"2024-02-23T00:43:44.271430+0000","last_became_active":"2024-02-23T00:42:39.396063+0000","last_became_peered":"2024-02-23T00:42:39.396063+0000","last_unstale":"2024-02-23T00:43:44.271430+0000","last_undegraded":"2024-02-23T00:43:44.271430+0000","last_fullsized":"2024-02-23T00:43:44.271430+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.355553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.355553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":820,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.267546+0000","last_change":"2024-02-23T00:43:12.980617+0000","last_active":"2024-02-23T00:43:44.267546+0000","last_peered":"2024-02-23T00:43:44.267546+0000","last_clean":"2024-02-23T00:43:44.267546+0000","last_became_active":"2024-02-23T00:40:57.218869+0000","last_became_peered":"2024-02-23T00:40:57.218869+0000","last_unstale":"2024-02-23T00:43:44.267546+0000","last_undegraded":"2024-02-23T00:43:44.267546+0000","last_fullsized":"2024-02-23T00:43:44.267546+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:12.980537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:12.980537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":359,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.269794+0000","last_change":"2024-02-23T00:42:55.230049+0000","last_active":"2024-02-23T00:43:44.269794+0000","last_peered":"2024-02-23T00:43:44.269794+0000","last_clean":"2024-02-23T00:43:44.269794+0000","last_became_active":"2024-02-23T00:41:37.228326+0000","last_became_peered":"2024-02-23T00:41:37.228326+0000","last_unstale":"2024-02-23T00:43:44.269794+0000","last_undegraded":"2024-02-23T00:43:44.269794+0000","last_fullsized":"2024-02-23T00:43:44.269794+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:42:55.229966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:55.229966+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":759,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259848+0000","last_change":"2024-02-23T00:41:42.160301+0000","last_active":"2024-02-23T00:43:43.259848+0000","last_peered":"2024-02-23T00:43:43.259848+0000","last_clean":"2024-02-23T00:43:43.259848+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:43:43.259848+0000","last_undegraded":"2024-02-23T00:43:43.259848+0000","last_fullsized":"2024-02-23T00:43:43.259848+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.160190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.160190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":752,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271578+0000","last_change":"2024-02-23T00:41:50.256151+0000","last_active":"2024-02-23T00:43:44.271578+0000","last_peered":"2024-02-23T00:43:44.271578+0000","last_clean":"2024-02-23T00:43:44.271578+0000","last_became_active":"2024-02-23T00:41:50.255850+0000","last_became_peered":"2024-02-23T00:41:50.255850+0000","last_unstale":"2024-02-23T00:43:44.271578+0000","last_undegraded":"2024-02-23T00:43:44.271578+0000","last_fullsized":"2024-02-23T00:43:44.271578+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:01.638560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:01.638560+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":300,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271579+0000","last_change":"2024-02-23T00:43:42.257077+0000","last_active":"2024-02-23T00:43:44.271579+0000","last_peered":"2024-02-23T00:43:44.271579+0000","last_clean":"2024-02-23T00:43:44.271579+0000","last_became_active":"2024-02-23T00:43:42.256711+0000","last_became_peered":"2024-02-23T00:43:42.256711+0000","last_unstale":"2024-02-23T00:43:44.271579+0000","last_undegraded":"2024-02-23T00:43:44.271579+0000","last_fullsized":"2024-02-23T00:43:44.271579+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:02.306855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.306855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":1031,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259224+0000","last_change":"2024-02-23T00:42:39.399220+0000","last_active":"2024-02-23T00:43:43.259224+0000","last_peered":"2024-02-23T00:43:43.259224+0000","last_clean":"2024-02-23T00:43:43.259224+0000","last_became_active":"2024-02-23T00:42:39.398790+0000","last_became_peered":"2024-02-23T00:42:39.398790+0000","last_unstale":"2024-02-23T00:43:43.259224+0000","last_undegraded":"2024-02-23T00:43:43.259224+0000","last_fullsized":"2024-02-23T00:43:43.259224+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:17.043322+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:17.043322+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":815,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433099+0000","last_change":"2024-02-23T00:41:37.221015+0000","last_active":"2024-02-23T00:43:43.433099+0000","last_peered":"2024-02-23T00:43:43.433099+0000","last_clean":"2024-02-23T00:43:43.433099+0000","last_became_active":"2024-02-23T00:41:37.220531+0000","last_became_peered":"2024-02-23T00:41:37.220531+0000","last_unstale":"2024-02-23T00:43:43.433099+0000","last_undegraded":"2024-02-23T00:43:43.433099+0000","last_fullsized":"2024-02-23T00:43:43.433099+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.603558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.603558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"453'35","reported_seq":143,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.280914+0000","last_change":"2024-02-23T00:42:27.632835+0000","last_active":"2024-02-23T00:43:41.280914+0000","last_peered":"2024-02-23T00:43:41.280914+0000","last_clean":"2024-02-23T00:43:41.280914+0000","last_became_active":"2024-02-23T00:41:37.235230+0000","last_became_peered":"2024-02-23T00:41:37.235230+0000","last_unstale":"2024-02-23T00:43:41.280914+0000","last_undegraded":"2024-02-23T00:43:41.280914+0000","last_fullsized":"2024-02-23T00:43:41.280914+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'35","last_scrub_stamp":"2024-02-23T00:42:27.632769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.632769+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":792,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748668+0000","last_change":"2024-02-23T00:42:53.644165+0000","last_active":"2024-02-23T00:43:44.748668+0000","last_peered":"2024-02-23T00:43:44.748668+0000","last_clean":"2024-02-23T00:43:44.748668+0000","last_became_active":"2024-02-23T00:41:37.229922+0000","last_became_peered":"2024-02-23T00:41:37.229922+0000","last_unstale":"2024-02-23T00:43:44.748668+0000","last_undegraded":"2024-02-23T00:43:44.748668+0000","last_fullsized":"2024-02-23T00:43:44.748668+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.644087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.644087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":776,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258351+0000","last_change":"2024-02-23T00:43:09.384826+0000","last_active":"2024-02-23T00:43:43.258351+0000","last_peered":"2024-02-23T00:43:43.258351+0000","last_clean":"2024-02-23T00:43:43.258351+0000","last_became_active":"2024-02-23T00:41:05.523365+0000","last_became_peered":"2024-02-23T00:41:05.523365+0000","last_unstale":"2024-02-23T00:43:43.258351+0000","last_undegraded":"2024-02-23T00:43:43.258351+0000","last_fullsized":"2024-02-23T00:43:43.258351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:09.384750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:43:09.384750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":706,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258408+0000","last_change":"2024-02-23T00:42:16.223783+0000","last_active":"2024-02-23T00:43:43.258408+0000","last_peered":"2024-02-23T00:43:43.258408+0000","last_clean":"2024-02-23T00:43:43.258408+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:43:43.258408+0000","last_undegraded":"2024-02-23T00:43:43.258408+0000","last_fullsized":"2024-02-23T00:43:43.258408+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:16.223720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.223720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":802,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.894243+0000","last_change":"2024-02-23T00:42:58.210034+0000","last_active":"2024-02-23T00:43:43.894243+0000","last_peered":"2024-02-23T00:43:43.894243+0000","last_clean":"2024-02-23T00:43:43.894243+0000","last_became_active":"2024-02-23T00:41:37.228303+0000","last_became_peered":"2024-02-23T00:41:37.228303+0000","last_unstale":"2024-02-23T00:43:43.894243+0000","last_undegraded":"2024-02-23T00:43:43.894243+0000","last_fullsized":"2024-02-23T00:43:43.894243+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:58.209955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:42:58.209955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":502,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281720+0000","last_change":"2024-02-23T00:42:17.721535+0000","last_active":"2024-02-23T00:43:41.281720+0000","last_peered":"2024-02-23T00:43:41.281720+0000","last_clean":"2024-02-23T00:43:41.281720+0000","last_became_active":"2024-02-23T00:41:41.224814+0000","last_became_peered":"2024-02-23T00:41:41.224814+0000","last_unstale":"2024-02-23T00:43:41.281720+0000","last_undegraded":"2024-02-23T00:43:41.281720+0000","last_fullsized":"2024-02-23T00:43:41.281720+0000","mapping_epoch":681,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:42:17.721471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.721471+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":792,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745773+0000","last_change":"2024-02-23T00:43:16.745314+0000","last_active":"2024-02-23T00:43:44.745773+0000","last_peered":"2024-02-23T00:43:44.745773+0000","last_clean":"2024-02-23T00:43:44.745773+0000","last_became_active":"2024-02-23T00:41:37.229722+0000","last_became_peered":"2024-02-23T00:41:37.229722+0000","last_unstale":"2024-02-23T00:43:44.745773+0000","last_undegraded":"2024-02-23T00:43:44.745773+0000","last_fullsized":"2024-02-23T00:43:44.745773+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:16.745223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:16.745223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":135,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.280336+0000","last_change":"2024-02-23T00:43:06.616251+0000","last_active":"2024-02-23T00:43:41.280336+0000","last_peered":"2024-02-23T00:43:41.280336+0000","last_clean":"2024-02-23T00:43:41.280336+0000","last_became_active":"2024-02-23T00:42:39.395793+0000","last_became_peered":"2024-02-23T00:42:39.395793+0000","last_unstale":"2024-02-23T00:43:41.280336+0000","last_undegraded":"2024-02-23T00:43:41.280336+0000","last_fullsized":"2024-02-23T00:43:41.280336+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.616180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.616180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"78'2","reported_seq":807,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259480+0000","last_change":"2024-02-23T00:41:53.371931+0000","last_active":"2024-02-23T00:43:43.259480+0000","last_peered":"2024-02-23T00:43:43.259480+0000","last_clean":"2024-02-23T00:43:43.259480+0000","last_became_active":"2024-02-23T00:41:25.236069+0000","last_became_peered":"2024-02-23T00:41:25.236069+0000","last_unstale":"2024-02-23T00:43:43.259480+0000","last_undegraded":"2024-02-23T00:43:43.259480+0000","last_fullsized":"2024-02-23T00:43:43.259480+0000","mapping_epoch":511,"log_start":"78'2","ondisk_log_start":"78'2","created":60,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:41:53.371852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.371852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":720,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.266244+0000","last_change":"2024-02-23T00:42:53.959862+0000","last_active":"2024-02-23T00:43:44.266244+0000","last_peered":"2024-02-23T00:43:44.266244+0000","last_clean":"2024-02-23T00:43:44.266244+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:43:44.266244+0000","last_undegraded":"2024-02-23T00:43:44.266244+0000","last_fullsized":"2024-02-23T00:43:44.266244+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.959787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.959787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":796,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745248+0000","last_change":"2024-02-23T00:42:15.388568+0000","last_active":"2024-02-23T00:43:44.745248+0000","last_peered":"2024-02-23T00:43:44.745248+0000","last_clean":"2024-02-23T00:43:44.745248+0000","last_became_active":"2024-02-23T00:40:45.191453+0000","last_became_peered":"2024-02-23T00:40:45.191453+0000","last_unstale":"2024-02-23T00:43:44.745248+0000","last_undegraded":"2024-02-23T00:43:44.745248+0000","last_fullsized":"2024-02-23T00:43:44.745248+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.388510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.388510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":816,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433380+0000","last_change":"2024-02-23T00:43:12.691014+0000","last_active":"2024-02-23T00:43:43.433380+0000","last_peered":"2024-02-23T00:43:43.433380+0000","last_clean":"2024-02-23T00:43:43.433380+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:43:43.433380+0000","last_undegraded":"2024-02-23T00:43:43.433380+0000","last_fullsized":"2024-02-23T00:43:43.433380+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:43:12.690926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:12.690926+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":795,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745864+0000","last_change":"2024-02-23T00:42:28.369569+0000","last_active":"2024-02-23T00:43:44.745864+0000","last_peered":"2024-02-23T00:43:44.745864+0000","last_clean":"2024-02-23T00:43:44.745864+0000","last_became_active":"2024-02-23T00:41:37.229335+0000","last_became_peered":"2024-02-23T00:41:37.229335+0000","last_unstale":"2024-02-23T00:43:44.745864+0000","last_undegraded":"2024-02-23T00:43:44.745864+0000","last_fullsized":"2024-02-23T00:43:44.745864+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:28.369481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.369481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":760,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893666+0000","last_change":"2024-02-23T00:41:43.984556+0000","last_active":"2024-02-23T00:43:43.893666+0000","last_peered":"2024-02-23T00:43:43.893666+0000","last_clean":"2024-02-23T00:43:43.893666+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:43:43.893666+0000","last_undegraded":"2024-02-23T00:43:43.893666+0000","last_fullsized":"2024-02-23T00:43:43.893666+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:43.984490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:43.984490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":300,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271655+0000","last_change":"2024-02-23T00:42:39.396374+0000","last_active":"2024-02-23T00:43:44.271655+0000","last_peered":"2024-02-23T00:43:44.271655+0000","last_clean":"2024-02-23T00:43:44.271655+0000","last_became_active":"2024-02-23T00:42:39.396071+0000","last_became_peered":"2024-02-23T00:42:39.396071+0000","last_unstale":"2024-02-23T00:43:44.271655+0000","last_undegraded":"2024-02-23T00:43:44.271655+0000","last_fullsized":"2024-02-23T00:43:44.271655+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.474495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.474495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":808,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.894305+0000","last_change":"2024-02-23T00:42:22.037093+0000","last_active":"2024-02-23T00:43:43.894305+0000","last_peered":"2024-02-23T00:43:43.894305+0000","last_clean":"2024-02-23T00:43:43.894305+0000","last_became_active":"2024-02-23T00:41:13.469953+0000","last_became_peered":"2024-02-23T00:41:13.469953+0000","last_unstale":"2024-02-23T00:43:43.894305+0000","last_undegraded":"2024-02-23T00:43:43.894305+0000","last_fullsized":"2024-02-23T00:43:43.894305+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:22.037008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.037008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":722,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.266588+0000","last_change":"2024-02-23T00:43:00.007515+0000","last_active":"2024-02-23T00:43:44.266588+0000","last_peered":"2024-02-23T00:43:44.266588+0000","last_clean":"2024-02-23T00:43:44.266588+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:43:44.266588+0000","last_undegraded":"2024-02-23T00:43:44.266588+0000","last_fullsized":"2024-02-23T00:43:44.266588+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:00.007437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.007437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":812,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.272090+0000","last_change":"2024-02-23T00:43:17.386897+0000","last_active":"2024-02-23T00:43:44.272090+0000","last_peered":"2024-02-23T00:43:44.272090+0000","last_clean":"2024-02-23T00:43:44.272090+0000","last_became_active":"2024-02-23T00:40:38.061696+0000","last_became_peered":"2024-02-23T00:40:38.061696+0000","last_unstale":"2024-02-23T00:43:44.272090+0000","last_undegraded":"2024-02-23T00:43:44.272090+0000","last_fullsized":"2024-02-23T00:43:44.272090+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:17.386832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:43:17.386832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":913,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259807+0000","last_change":"2024-02-23T00:42:59.284657+0000","last_active":"2024-02-23T00:43:43.259807+0000","last_peered":"2024-02-23T00:43:43.259807+0000","last_clean":"2024-02-23T00:43:43.259807+0000","last_became_active":"2024-02-23T00:41:40.219384+0000","last_became_peered":"2024-02-23T00:41:40.219384+0000","last_unstale":"2024-02-23T00:43:43.259807+0000","last_undegraded":"2024-02-23T00:43:43.259807+0000","last_fullsized":"2024-02-23T00:43:43.259807+0000","mapping_epoch":680,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:42:59.284597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.284597+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":825,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.270552+0000","last_change":"2024-02-23T00:43:03.434468+0000","last_active":"2024-02-23T00:43:44.270552+0000","last_peered":"2024-02-23T00:43:44.270552+0000","last_clean":"2024-02-23T00:43:44.270552+0000","last_became_active":"2024-02-23T00:41:37.219000+0000","last_became_peered":"2024-02-23T00:41:37.219000+0000","last_unstale":"2024-02-23T00:43:44.270552+0000","last_undegraded":"2024-02-23T00:43:44.270552+0000","last_fullsized":"2024-02-23T00:43:44.270552+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:03.434391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:03.434391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":1024,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258012+0000","last_change":"2024-02-23T00:42:39.401952+0000","last_active":"2024-02-23T00:43:43.258012+0000","last_peered":"2024-02-23T00:43:43.258012+0000","last_clean":"2024-02-23T00:43:43.258012+0000","last_became_active":"2024-02-23T00:42:39.401599+0000","last_became_peered":"2024-02-23T00:42:39.401599+0000","last_unstale":"2024-02-23T00:43:43.258012+0000","last_undegraded":"2024-02-23T00:43:43.258012+0000","last_fullsized":"2024-02-23T00:43:43.258012+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:40:12.984040+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:40:12.984040+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"62'1","reported_seq":821,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748563+0000","last_change":"2024-02-23T00:42:22.301796+0000","last_active":"2024-02-23T00:43:44.748563+0000","last_peered":"2024-02-23T00:43:44.748563+0000","last_clean":"2024-02-23T00:43:44.748563+0000","last_became_active":"2024-02-23T00:41:21.226819+0000","last_became_peered":"2024-02-23T00:41:21.226819+0000","last_unstale":"2024-02-23T00:43:44.748563+0000","last_undegraded":"2024-02-23T00:43:44.748563+0000","last_fullsized":"2024-02-23T00:43:44.748563+0000","mapping_epoch":511,"log_start":"62'1","ondisk_log_start":"62'1","created":60,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:42:22.301726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.301726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":729,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271056+0000","last_change":"2024-02-23T00:42:53.387347+0000","last_active":"2024-02-23T00:43:44.271056+0000","last_peered":"2024-02-23T00:43:44.271056+0000","last_clean":"2024-02-23T00:43:44.271056+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:43:44.271056+0000","last_undegraded":"2024-02-23T00:43:44.271056+0000","last_fullsized":"2024-02-23T00:43:44.271056+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.387251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.387251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":805,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.280658+0000","last_change":"2024-02-23T00:43:00.633273+0000","last_active":"2024-02-23T00:43:41.280658+0000","last_peered":"2024-02-23T00:43:41.280658+0000","last_clean":"2024-02-23T00:43:41.280658+0000","last_became_active":"2024-02-23T00:42:39.393007+0000","last_became_peered":"2024-02-23T00:42:39.393007+0000","last_unstale":"2024-02-23T00:43:41.280658+0000","last_undegraded":"2024-02-23T00:43:41.280658+0000","last_fullsized":"2024-02-23T00:43:41.280658+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:00.633189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.633189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":126,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281801+0000","last_change":"2024-02-23T00:43:09.611028+0000","last_active":"2024-02-23T00:43:41.281801+0000","last_peered":"2024-02-23T00:43:41.281801+0000","last_clean":"2024-02-23T00:43:41.281801+0000","last_became_active":"2024-02-23T00:41:37.239608+0000","last_became_peered":"2024-02-23T00:41:37.239608+0000","last_unstale":"2024-02-23T00:43:41.281801+0000","last_undegraded":"2024-02-23T00:43:41.281801+0000","last_fullsized":"2024-02-23T00:43:41.281801+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:09.610960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:09.610960+0000","last_clean_scrub_stamp":"2024-02-23T00:43:09.610960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":1107,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.695073+0000","last_change":"2024-02-23T00:42:54.354917+0000","last_active":"2024-02-23T00:43:44.695073+0000","last_peered":"2024-02-23T00:43:44.695073+0000","last_clean":"2024-02-23T00:43:44.695073+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:43:44.695073+0000","last_undegraded":"2024-02-23T00:43:44.695073+0000","last_fullsized":"2024-02-23T00:43:44.695073+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'48","last_scrub_stamp":"2024-02-23T00:42:54.354848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.354848+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"601'2","reported_seq":403,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258532+0000","last_change":"2024-02-23T00:43:00.296517+0000","last_active":"2024-02-23T00:43:43.258532+0000","last_peered":"2024-02-23T00:43:43.258532+0000","last_clean":"2024-02-23T00:43:43.258532+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:43:43.258532+0000","last_undegraded":"2024-02-23T00:43:43.258532+0000","last_fullsized":"2024-02-23T00:43:43.258532+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'2","last_scrub_stamp":"2024-02-23T00:43:00.296450+0000","last_deep_scrub":"601'2","last_deep_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.296450+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":332,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273222+0000","last_change":"2024-02-23T00:42:26.240735+0000","last_active":"2024-02-23T00:43:44.273222+0000","last_peered":"2024-02-23T00:43:44.273222+0000","last_clean":"2024-02-23T00:43:44.273222+0000","last_became_active":"2024-02-23T00:41:17.220294+0000","last_became_peered":"2024-02-23T00:41:17.220294+0000","last_unstale":"2024-02-23T00:43:44.273222+0000","last_undegraded":"2024-02-23T00:43:44.273222+0000","last_fullsized":"2024-02-23T00:43:44.273222+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.240647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.240647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":729,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.272304+0000","last_change":"2024-02-23T00:41:55.275163+0000","last_active":"2024-02-23T00:43:44.272304+0000","last_peered":"2024-02-23T00:43:44.272304+0000","last_clean":"2024-02-23T00:43:44.272304+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:43:44.272304+0000","last_undegraded":"2024-02-23T00:43:44.272304+0000","last_fullsized":"2024-02-23T00:43:44.272304+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.275099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.275099+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":702,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.892615+0000","last_change":"2024-02-23T00:43:12.060701+0000","last_active":"2024-02-23T00:43:43.892615+0000","last_peered":"2024-02-23T00:43:43.892615+0000","last_clean":"2024-02-23T00:43:43.892615+0000","last_became_active":"2024-02-23T00:42:39.399113+0000","last_became_peered":"2024-02-23T00:42:39.399113+0000","last_unstale":"2024-02-23T00:43:43.892615+0000","last_undegraded":"2024-02-23T00:43:43.892615+0000","last_fullsized":"2024-02-23T00:43:43.892615+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:12.060587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:43:12.060587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"783'22","reported_seq":378,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.280999+0000","last_change":"2024-02-23T00:43:40.280239+0000","last_active":"2024-02-23T00:43:41.280999+0000","last_peered":"2024-02-23T00:43:41.280999+0000","last_clean":"2024-02-23T00:43:41.280999+0000","last_became_active":"2024-02-23T00:43:40.280015+0000","last_became_peered":"2024-02-23T00:43:40.280015+0000","last_unstale":"2024-02-23T00:43:41.280999+0000","last_undegraded":"2024-02-23T00:43:41.280999+0000","last_fullsized":"2024-02-23T00:43:41.280999+0000","mapping_epoch":782,"log_start":"606'17","ondisk_log_start":"606'17","created":58,"last_epoch_clean":783,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'19","last_scrub_stamp":"2024-02-23T00:40:58.498752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.498752+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":210,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258422+0000","last_change":"2024-02-23T00:40:53.236854+0000","last_active":"2024-02-23T00:43:43.258422+0000","last_peered":"2024-02-23T00:43:43.258422+0000","last_clean":"2024-02-23T00:43:43.258422+0000","last_became_active":"2024-02-23T00:40:23.194277+0000","last_became_peered":"2024-02-23T00:40:23.194277+0000","last_unstale":"2024-02-23T00:43:43.258422+0000","last_undegraded":"2024-02-23T00:43:43.258422+0000","last_fullsized":"2024-02-23T00:43:43.258422+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:40:53.236791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:40:53.236791+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":786,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259265+0000","last_change":"2024-02-23T00:43:07.413095+0000","last_active":"2024-02-23T00:43:43.259265+0000","last_peered":"2024-02-23T00:43:43.259265+0000","last_clean":"2024-02-23T00:43:43.259265+0000","last_became_active":"2024-02-23T00:41:37.229003+0000","last_became_peered":"2024-02-23T00:41:37.229003+0000","last_unstale":"2024-02-23T00:43:43.259265+0000","last_undegraded":"2024-02-23T00:43:43.259265+0000","last_fullsized":"2024-02-23T00:43:43.259265+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.413026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.413026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":126,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.279402+0000","last_change":"2024-02-23T00:43:10.611567+0000","last_active":"2024-02-23T00:43:41.279402+0000","last_peered":"2024-02-23T00:43:41.279402+0000","last_clean":"2024-02-23T00:43:41.279402+0000","last_became_active":"2024-02-23T00:41:37.248459+0000","last_became_peered":"2024-02-23T00:41:37.248459+0000","last_unstale":"2024-02-23T00:43:41.279402+0000","last_undegraded":"2024-02-23T00:43:41.279402+0000","last_fullsized":"2024-02-23T00:43:41.279402+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:10.611499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:43:10.611499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":720,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.265781+0000","last_change":"2024-02-23T00:43:16.990810+0000","last_active":"2024-02-23T00:43:44.265781+0000","last_peered":"2024-02-23T00:43:44.265781+0000","last_clean":"2024-02-23T00:43:44.265781+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:43:44.265781+0000","last_undegraded":"2024-02-23T00:43:44.265781+0000","last_fullsized":"2024-02-23T00:43:44.265781+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:16.990733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:43:16.990733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":63,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.746885+0000","last_change":"2024-02-23T00:43:02.723267+0000","last_active":"2024-02-23T00:43:44.746885+0000","last_peered":"2024-02-23T00:43:44.746885+0000","last_clean":"2024-02-23T00:43:44.746885+0000","last_became_active":"2024-02-23T00:42:41.402111+0000","last_became_peered":"2024-02-23T00:42:41.402111+0000","last_unstale":"2024-02-23T00:43:44.746885+0000","last_undegraded":"2024-02-23T00:43:44.746885+0000","last_fullsized":"2024-02-23T00:43:44.746885+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:02.723205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:43:02.723205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":819,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273221+0000","last_change":"2024-02-23T00:42:26.380761+0000","last_active":"2024-02-23T00:43:44.273221+0000","last_peered":"2024-02-23T00:43:44.273221+0000","last_clean":"2024-02-23T00:43:44.273221+0000","last_became_active":"2024-02-23T00:41:37.239598+0000","last_became_peered":"2024-02-23T00:41:37.239598+0000","last_unstale":"2024-02-23T00:43:44.273221+0000","last_undegraded":"2024-02-23T00:43:44.273221+0000","last_fullsized":"2024-02-23T00:43:44.273221+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.380671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":712,"reported_epoch":787,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:43:44.270136+0000","last_change":"2024-02-23T00:43:43.895272+0000","last_active":"2024-02-23T00:43:44.270136+0000","last_peered":"2024-02-23T00:43:44.270136+0000","last_clean":"2024-02-23T00:38:47.154201+0000","last_became_active":"2024-02-23T00:43:43.894446+0000","last_became_peered":"2024-02-23T00:43:43.894446+0000","last_unstale":"2024-02-23T00:43:44.270136+0000","last_undegraded":"2024-02-23T00:43:44.270136+0000","last_fullsized":"2024-02-23T00:43:44.270136+0000","mapping_epoch":785,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:43:05.364147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:05.364147+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,1],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":822,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.266806+0000","last_change":"2024-02-23T00:43:01.990077+0000","last_active":"2024-02-23T00:43:44.266806+0000","last_peered":"2024-02-23T00:43:44.266806+0000","last_clean":"2024-02-23T00:43:44.266806+0000","last_became_active":"2024-02-23T00:41:37.238708+0000","last_became_peered":"2024-02-23T00:41:37.238708+0000","last_unstale":"2024-02-23T00:43:44.266806+0000","last_undegraded":"2024-02-23T00:43:44.266806+0000","last_fullsized":"2024-02-23T00:43:44.266806+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:43:01.989978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.989978+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":309,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273537+0000","last_change":"2024-02-23T00:41:33.042594+0000","last_active":"2024-02-23T00:43:44.273537+0000","last_peered":"2024-02-23T00:43:44.273537+0000","last_clean":"2024-02-23T00:43:44.273537+0000","last_became_active":"2024-02-23T00:40:17.214288+0000","last_became_peered":"2024-02-23T00:40:17.214288+0000","last_unstale":"2024-02-23T00:43:44.273537+0000","last_undegraded":"2024-02-23T00:43:44.273537+0000","last_fullsized":"2024-02-23T00:43:44.273537+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:33.042520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.042520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":709,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893311+0000","last_change":"2024-02-23T00:42:13.006928+0000","last_active":"2024-02-23T00:43:43.893311+0000","last_peered":"2024-02-23T00:43:43.893311+0000","last_clean":"2024-02-23T00:43:43.893311+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:43:43.893311+0000","last_undegraded":"2024-02-23T00:43:43.893311+0000","last_fullsized":"2024-02-23T00:43:43.893311+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:13.006833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.006833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":63,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748800+0000","last_change":"2024-02-23T00:43:04.777343+0000","last_active":"2024-02-23T00:43:44.748800+0000","last_peered":"2024-02-23T00:43:44.748800+0000","last_clean":"2024-02-23T00:43:44.748800+0000","last_became_active":"2024-02-23T00:42:41.399155+0000","last_became_peered":"2024-02-23T00:42:41.399155+0000","last_unstale":"2024-02-23T00:43:44.748800+0000","last_undegraded":"2024-02-23T00:43:44.748800+0000","last_fullsized":"2024-02-23T00:43:44.748800+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:04.777269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:04.777269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":555,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281902+0000","last_change":"2024-02-23T00:42:14.760378+0000","last_active":"2024-02-23T00:43:41.281902+0000","last_peered":"2024-02-23T00:43:41.281902+0000","last_clean":"2024-02-23T00:43:41.281902+0000","last_became_active":"2024-02-23T00:41:38.217979+0000","last_became_peered":"2024-02-23T00:41:38.217979+0000","last_unstale":"2024-02-23T00:43:41.281902+0000","last_undegraded":"2024-02-23T00:43:41.281902+0000","last_fullsized":"2024-02-23T00:43:41.281902+0000","mapping_epoch":678,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:42:14.760308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.760308+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":796,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273216+0000","last_change":"2024-02-23T00:42:21.442459+0000","last_active":"2024-02-23T00:43:44.273216+0000","last_peered":"2024-02-23T00:43:44.273216+0000","last_clean":"2024-02-23T00:43:44.273216+0000","last_became_active":"2024-02-23T00:41:37.238750+0000","last_became_peered":"2024-02-23T00:41:37.238750+0000","last_unstale":"2024-02-23T00:43:44.273216+0000","last_undegraded":"2024-02-23T00:43:44.273216+0000","last_fullsized":"2024-02-23T00:43:44.273216+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:21.442362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:21.442362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":296,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271529+0000","last_change":"2024-02-23T00:42:20.334112+0000","last_active":"2024-02-23T00:43:44.271529+0000","last_peered":"2024-02-23T00:43:44.271529+0000","last_clean":"2024-02-23T00:43:44.271529+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:43:44.271529+0000","last_undegraded":"2024-02-23T00:43:44.271529+0000","last_fullsized":"2024-02-23T00:43:44.271529+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.334013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.334013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":291,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.274243+0000","last_change":"2024-02-23T00:41:48.297291+0000","last_active":"2024-02-23T00:43:44.274243+0000","last_peered":"2024-02-23T00:43:44.274243+0000","last_clean":"2024-02-23T00:43:44.274243+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:43:44.274243+0000","last_undegraded":"2024-02-23T00:43:44.274243+0000","last_fullsized":"2024-02-23T00:43:44.274243+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:48.297220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:48.297220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":188,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.432518+0000","last_change":"2024-02-23T00:43:07.691013+0000","last_active":"2024-02-23T00:43:43.432518+0000","last_peered":"2024-02-23T00:43:43.432518+0000","last_clean":"2024-02-23T00:43:43.432518+0000","last_became_active":"2024-02-23T00:42:41.406233+0000","last_became_peered":"2024-02-23T00:42:41.406233+0000","last_unstale":"2024-02-23T00:43:43.432518+0000","last_undegraded":"2024-02-23T00:43:43.432518+0000","last_fullsized":"2024-02-23T00:43:43.432518+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:43:07.690939+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:43:07.690939+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.690939+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":904,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271088+0000","last_change":"2024-02-23T00:43:00.208080+0000","last_active":"2024-02-23T00:43:44.271088+0000","last_peered":"2024-02-23T00:43:44.271088+0000","last_clean":"2024-02-23T00:43:44.271088+0000","last_became_active":"2024-02-23T00:40:51.316452+0000","last_became_peered":"2024-02-23T00:40:51.316452+0000","last_unstale":"2024-02-23T00:43:44.271088+0000","last_undegraded":"2024-02-23T00:43:44.271088+0000","last_fullsized":"2024-02-23T00:43:44.271088+0000","mapping_epoch":631,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:43:00.208019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.208019+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":773,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.432766+0000","last_change":"2024-02-23T00:42:19.655247+0000","last_active":"2024-02-23T00:43:43.432766+0000","last_peered":"2024-02-23T00:43:43.432766+0000","last_clean":"2024-02-23T00:43:43.432766+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:43:43.432766+0000","last_undegraded":"2024-02-23T00:43:43.432766+0000","last_fullsized":"2024-02-23T00:43:43.432766+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.655188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.655188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":769,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.894097+0000","last_change":"2024-02-23T00:41:53.984748+0000","last_active":"2024-02-23T00:43:43.894097+0000","last_peered":"2024-02-23T00:43:43.894097+0000","last_clean":"2024-02-23T00:43:43.894097+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:43:43.894097+0000","last_undegraded":"2024-02-23T00:43:43.894097+0000","last_fullsized":"2024-02-23T00:43:43.894097+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.984679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.984679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":738,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433594+0000","last_change":"2024-02-23T00:43:04.619611+0000","last_active":"2024-02-23T00:43:43.433594+0000","last_peered":"2024-02-23T00:43:43.433594+0000","last_clean":"2024-02-23T00:43:43.433594+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:43:43.433594+0000","last_undegraded":"2024-02-23T00:43:43.433594+0000","last_fullsized":"2024-02-23T00:43:43.433594+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:04.619508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:04.619508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.744940+0000","last_change":"2024-02-23T00:42:59.678005+0000","last_active":"2024-02-23T00:43:44.744940+0000","last_peered":"2024-02-23T00:43:44.744940+0000","last_clean":"2024-02-23T00:43:44.744940+0000","last_became_active":"2024-02-23T00:42:41.402543+0000","last_became_peered":"2024-02-23T00:42:41.402543+0000","last_unstale":"2024-02-23T00:43:44.744940+0000","last_undegraded":"2024-02-23T00:43:44.744940+0000","last_fullsized":"2024-02-23T00:43:44.744940+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:59.677899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.677899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"688'39","reported_seq":314,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.274069+0000","last_change":"2024-02-23T00:41:49.338035+0000","last_active":"2024-02-23T00:43:44.274069+0000","last_peered":"2024-02-23T00:43:44.274069+0000","last_clean":"2024-02-23T00:43:44.274069+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:43:44.274069+0000","last_undegraded":"2024-02-23T00:43:44.274069+0000","last_fullsized":"2024-02-23T00:43:44.274069+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'39","last_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_deep_scrub":"688'39","last_deep_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.337969+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":786,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893217+0000","last_change":"2024-02-23T00:41:41.955717+0000","last_active":"2024-02-23T00:43:43.893217+0000","last_peered":"2024-02-23T00:43:43.893217+0000","last_clean":"2024-02-23T00:43:43.893217+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:43:43.893217+0000","last_undegraded":"2024-02-23T00:43:43.893217+0000","last_fullsized":"2024-02-23T00:43:43.893217+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:41.955589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:41:41.955589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"681'12","reported_seq":991,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745571+0000","last_change":"2024-02-23T00:42:59.230304+0000","last_active":"2024-02-23T00:43:44.745571+0000","last_peered":"2024-02-23T00:43:44.745571+0000","last_clean":"2024-02-23T00:43:44.745571+0000","last_became_active":"2024-02-23T00:41:37.228441+0000","last_became_peered":"2024-02-23T00:41:37.228441+0000","last_unstale":"2024-02-23T00:43:44.745571+0000","last_undegraded":"2024-02-23T00:43:44.745571+0000","last_fullsized":"2024-02-23T00:43:44.745571+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'12","last_scrub_stamp":"2024-02-23T00:42:59.230220+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.230220+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":709,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.894241+0000","last_change":"2024-02-23T00:42:17.056253+0000","last_active":"2024-02-23T00:43:43.894241+0000","last_peered":"2024-02-23T00:43:43.894241+0000","last_clean":"2024-02-23T00:43:43.894241+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:43:43.894241+0000","last_undegraded":"2024-02-23T00:43:43.894241+0000","last_fullsized":"2024-02-23T00:43:43.894241+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:17.056153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.056153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":60,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.280224+0000","last_change":"2024-02-23T00:43:07.568838+0000","last_active":"2024-02-23T00:43:41.280224+0000","last_peered":"2024-02-23T00:43:41.280224+0000","last_clean":"2024-02-23T00:43:41.280224+0000","last_became_active":"2024-02-23T00:42:41.402683+0000","last_became_peered":"2024-02-23T00:42:41.402683+0000","last_unstale":"2024-02-23T00:43:41.280224+0000","last_undegraded":"2024-02-23T00:43:41.280224+0000","last_fullsized":"2024-02-23T00:43:41.280224+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.568782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.568782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":845,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893764+0000","last_change":"2024-02-23T00:42:28.967292+0000","last_active":"2024-02-23T00:43:43.893764+0000","last_peered":"2024-02-23T00:43:43.893764+0000","last_clean":"2024-02-23T00:43:43.893764+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:43:43.893764+0000","last_undegraded":"2024-02-23T00:43:43.893764+0000","last_fullsized":"2024-02-23T00:43:43.893764+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:42:28.967208+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.967208+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":126,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.280749+0000","last_change":"2024-02-23T00:43:11.571763+0000","last_active":"2024-02-23T00:43:41.280749+0000","last_peered":"2024-02-23T00:43:41.280749+0000","last_clean":"2024-02-23T00:43:41.280749+0000","last_became_active":"2024-02-23T00:41:37.235746+0000","last_became_peered":"2024-02-23T00:41:37.235746+0000","last_unstale":"2024-02-23T00:43:41.280749+0000","last_undegraded":"2024-02-23T00:43:41.280749+0000","last_fullsized":"2024-02-23T00:43:41.280749+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:11.571661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:11.571661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":355,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-02-23T00:43:45.271505+0000","last_change":"2024-02-23T00:40:50.453642+0000","last_active":"2024-02-23T00:43:45.271505+0000","last_peered":"2024-02-23T00:43:45.271505+0000","last_clean":"2024-02-23T00:43:45.271505+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:43:45.271505+0000","last_undegraded":"2024-02-23T00:43:45.271505+0000","last_fullsized":"2024-02-23T00:43:45.271505+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'11","last_scrub_stamp":"2024-02-23T00:40:50.453579+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.453579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":291,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271300+0000","last_change":"2024-02-23T00:41:03.320478+0000","last_active":"2024-02-23T00:43:44.271300+0000","last_peered":"2024-02-23T00:43:44.271300+0000","last_clean":"2024-02-23T00:43:44.271300+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:43:44.271300+0000","last_undegraded":"2024-02-23T00:43:44.271300+0000","last_fullsized":"2024-02-23T00:43:44.271300+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:03.320387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:03.320387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":63,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748180+0000","last_change":"2024-02-23T00:43:05.740185+0000","last_active":"2024-02-23T00:43:44.748180+0000","last_peered":"2024-02-23T00:43:44.748180+0000","last_clean":"2024-02-23T00:43:44.748180+0000","last_became_active":"2024-02-23T00:42:41.398941+0000","last_became_peered":"2024-02-23T00:42:41.398941+0000","last_unstale":"2024-02-23T00:43:44.748180+0000","last_undegraded":"2024-02-23T00:43:44.748180+0000","last_fullsized":"2024-02-23T00:43:44.748180+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:05.740112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:43:05.740112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":782,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258685+0000","last_change":"2024-02-23T00:41:02.134977+0000","last_active":"2024-02-23T00:43:43.258685+0000","last_peered":"2024-02-23T00:43:43.258685+0000","last_clean":"2024-02-23T00:43:43.258685+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:43:43.258685+0000","last_undegraded":"2024-02-23T00:43:43.258685+0000","last_fullsized":"2024-02-23T00:43:43.258685+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'15","last_scrub_stamp":"2024-02-23T00:41:02.134898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.134898+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":800,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433474+0000","last_change":"2024-02-23T00:43:16.698802+0000","last_active":"2024-02-23T00:43:43.433474+0000","last_peered":"2024-02-23T00:43:43.433474+0000","last_clean":"2024-02-23T00:43:43.433474+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:43:43.433474+0000","last_undegraded":"2024-02-23T00:43:43.433474+0000","last_fullsized":"2024-02-23T00:43:43.433474+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:16.698726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:16.698726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":315,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271468+0000","last_change":"2024-02-23T00:42:27.197397+0000","last_active":"2024-02-23T00:43:44.271468+0000","last_peered":"2024-02-23T00:43:44.271468+0000","last_clean":"2024-02-23T00:43:44.271468+0000","last_became_active":"2024-02-23T00:41:37.252612+0000","last_became_peered":"2024-02-23T00:41:37.252612+0000","last_unstale":"2024-02-23T00:43:44.271468+0000","last_undegraded":"2024-02-23T00:43:44.271468+0000","last_fullsized":"2024-02-23T00:43:44.271468+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.197290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":739,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.265727+0000","last_change":"2024-02-23T00:42:12.891119+0000","last_active":"2024-02-23T00:43:44.265727+0000","last_peered":"2024-02-23T00:43:44.265727+0000","last_clean":"2024-02-23T00:43:44.265727+0000","last_became_active":"2024-02-23T00:41:37.238185+0000","last_became_peered":"2024-02-23T00:41:37.238185+0000","last_unstale":"2024-02-23T00:43:44.265727+0000","last_undegraded":"2024-02-23T00:43:44.265727+0000","last_fullsized":"2024-02-23T00:43:44.265727+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:12.891052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:12.891052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"78'6","reported_seq":63,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745106+0000","last_change":"2024-02-23T00:43:00.726511+0000","last_active":"2024-02-23T00:43:44.745106+0000","last_peered":"2024-02-23T00:43:44.745106+0000","last_clean":"2024-02-23T00:43:44.745106+0000","last_became_active":"2024-02-23T00:42:41.397727+0000","last_became_peered":"2024-02-23T00:42:41.397727+0000","last_unstale":"2024-02-23T00:43:44.745106+0000","last_undegraded":"2024-02-23T00:43:44.745106+0000","last_fullsized":"2024-02-23T00:43:44.745106+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:43:00.726434+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.726434+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":823,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.266699+0000","last_change":"2024-02-23T00:43:06.975177+0000","last_active":"2024-02-23T00:43:44.266699+0000","last_peered":"2024-02-23T00:43:44.266699+0000","last_clean":"2024-02-23T00:43:44.266699+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:43:44.266699+0000","last_undegraded":"2024-02-23T00:43:44.266699+0000","last_fullsized":"2024-02-23T00:43:44.266699+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:43:06.975089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.975089+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":783,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259483+0000","last_change":"2024-02-23T00:43:18.462532+0000","last_active":"2024-02-23T00:43:43.259483+0000","last_peered":"2024-02-23T00:43:43.259483+0000","last_clean":"2024-02-23T00:43:43.259483+0000","last_became_active":"2024-02-23T00:41:37.229726+0000","last_became_peered":"2024-02-23T00:41:37.229726+0000","last_unstale":"2024-02-23T00:43:43.259483+0000","last_undegraded":"2024-02-23T00:43:43.259483+0000","last_fullsized":"2024-02-23T00:43:43.259483+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:18.462472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:43:18.462472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":773,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258678+0000","last_change":"2024-02-23T00:41:35.138434+0000","last_active":"2024-02-23T00:43:43.258678+0000","last_peered":"2024-02-23T00:43:43.258678+0000","last_clean":"2024-02-23T00:43:43.258678+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:43:43.258678+0000","last_undegraded":"2024-02-23T00:43:43.258678+0000","last_fullsized":"2024-02-23T00:43:43.258678+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.138319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.138319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":764,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433068+0000","last_change":"2024-02-23T00:42:22.669179+0000","last_active":"2024-02-23T00:43:43.433068+0000","last_peered":"2024-02-23T00:43:43.433068+0000","last_clean":"2024-02-23T00:43:43.433068+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:43:43.433068+0000","last_undegraded":"2024-02-23T00:43:43.433068+0000","last_fullsized":"2024-02-23T00:43:43.433068+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:42:22.669090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.669090+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":317,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.269954+0000","last_change":"2024-02-23T00:43:42.433192+0000","last_active":"2024-02-23T00:43:44.269954+0000","last_peered":"2024-02-23T00:43:44.269954+0000","last_clean":"2024-02-23T00:43:44.269954+0000","last_became_active":"2024-02-23T00:43:42.256788+0000","last_became_peered":"2024-02-23T00:43:42.256788+0000","last_unstale":"2024-02-23T00:43:44.269954+0000","last_undegraded":"2024-02-23T00:43:44.269954+0000","last_fullsized":"2024-02-23T00:43:44.269954+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'10","last_scrub_stamp":"2024-02-23T00:43:04.142193+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:43:04.142193+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":760,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.894160+0000","last_change":"2024-02-23T00:42:15.038961+0000","last_active":"2024-02-23T00:43:43.894160+0000","last_peered":"2024-02-23T00:43:43.894160+0000","last_clean":"2024-02-23T00:43:43.894160+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:43:43.894160+0000","last_undegraded":"2024-02-23T00:43:43.894160+0000","last_fullsized":"2024-02-23T00:43:43.894160+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.038897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.038897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":769,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.432712+0000","last_change":"2024-02-23T00:43:14.754285+0000","last_active":"2024-02-23T00:43:43.432712+0000","last_peered":"2024-02-23T00:43:43.432712+0000","last_clean":"2024-02-23T00:43:43.432712+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:43:43.432712+0000","last_undegraded":"2024-02-23T00:43:43.432712+0000","last_fullsized":"2024-02-23T00:43:43.432712+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:14.754203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:14.754203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":706,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258727+0000","last_change":"2024-02-23T00:41:44.196162+0000","last_active":"2024-02-23T00:43:43.258727+0000","last_peered":"2024-02-23T00:43:43.258727+0000","last_clean":"2024-02-23T00:43:43.258727+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:43:43.258727+0000","last_undegraded":"2024-02-23T00:43:43.258727+0000","last_fullsized":"2024-02-23T00:43:43.258727+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.196088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.196088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":62,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893980+0000","last_change":"2024-02-23T00:43:17.077666+0000","last_active":"2024-02-23T00:43:43.893980+0000","last_peered":"2024-02-23T00:43:43.893980+0000","last_clean":"2024-02-23T00:43:43.893980+0000","last_became_active":"2024-02-23T00:42:41.391484+0000","last_became_peered":"2024-02-23T00:42:41.391484+0000","last_unstale":"2024-02-23T00:43:43.893980+0000","last_undegraded":"2024-02-23T00:43:43.893980+0000","last_fullsized":"2024-02-23T00:43:43.893980+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:17.077581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:17.077581+0000","last_clean_scrub_stamp":"2024-02-23T00:43:17.077581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":308,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271713+0000","last_change":"2024-02-23T00:40:43.233713+0000","last_active":"2024-02-23T00:43:44.271713+0000","last_peered":"2024-02-23T00:43:44.271713+0000","last_clean":"2024-02-23T00:43:44.271713+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:43:44.271713+0000","last_undegraded":"2024-02-23T00:43:44.271713+0000","last_fullsized":"2024-02-23T00:43:44.271713+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:40:43.233609+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:40:43.233609+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":772,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433599+0000","last_change":"2024-02-23T00:42:25.724438+0000","last_active":"2024-02-23T00:43:43.433599+0000","last_peered":"2024-02-23T00:43:43.433599+0000","last_clean":"2024-02-23T00:43:43.433599+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:43:43.433599+0000","last_undegraded":"2024-02-23T00:43:43.433599+0000","last_fullsized":"2024-02-23T00:43:43.433599+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:25.724375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:25.724375+0000","last_clean_scrub_stamp":"2024-02-23T00:42:25.724375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":28,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.268908+0000","last_change":"2024-02-23T00:43:21.499211+0000","last_active":"2024-02-23T00:43:44.268908+0000","last_peered":"2024-02-23T00:43:44.268908+0000","last_clean":"2024-02-23T00:43:44.268908+0000","last_became_active":"2024-02-23T00:43:21.498988+0000","last_became_peered":"2024-02-23T00:43:21.498988+0000","last_unstale":"2024-02-23T00:43:44.268908+0000","last_undegraded":"2024-02-23T00:43:44.268908+0000","last_fullsized":"2024-02-23T00:43:44.268908+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.520415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.520415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":291,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271395+0000","last_change":"2024-02-23T00:41:37.096978+0000","last_active":"2024-02-23T00:43:44.271395+0000","last_peered":"2024-02-23T00:43:44.271395+0000","last_clean":"2024-02-23T00:43:44.271395+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:43:44.271395+0000","last_undegraded":"2024-02-23T00:43:44.271395+0000","last_fullsized":"2024-02-23T00:43:44.271395+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:37.096906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:37.096906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":681,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281802+0000","last_change":"2024-02-23T00:42:53.560946+0000","last_active":"2024-02-23T00:43:41.281802+0000","last_peered":"2024-02-23T00:43:41.281802+0000","last_clean":"2024-02-23T00:43:41.281802+0000","last_became_active":"2024-02-23T00:42:10.240911+0000","last_became_peered":"2024-02-23T00:42:10.240911+0000","last_unstale":"2024-02-23T00:43:41.281802+0000","last_undegraded":"2024-02-23T00:43:41.281802+0000","last_fullsized":"2024-02-23T00:43:41.281802+0000","mapping_epoch":707,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:42:53.560884+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.560884+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":769,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893062+0000","last_change":"2024-02-23T00:43:05.149002+0000","last_active":"2024-02-23T00:43:43.893062+0000","last_peered":"2024-02-23T00:43:43.893062+0000","last_clean":"2024-02-23T00:43:43.893062+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:43:43.893062+0000","last_undegraded":"2024-02-23T00:43:43.893062+0000","last_fullsized":"2024-02-23T00:43:43.893062+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:05.148931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:55.816155+0000","last_clean_scrub_stamp":"2024-02-23T00:43:05.148931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":978,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748634+0000","last_change":"2024-02-23T00:42:59.626715+0000","last_active":"2024-02-23T00:43:44.748634+0000","last_peered":"2024-02-23T00:43:44.748634+0000","last_clean":"2024-02-23T00:43:44.748634+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:43:44.748634+0000","last_undegraded":"2024-02-23T00:43:44.748634+0000","last_fullsized":"2024-02-23T00:43:44.748634+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:42:59.626636+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.626636+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":759,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.746343+0000","last_change":"2024-02-23T00:42:14.427990+0000","last_active":"2024-02-23T00:43:44.746343+0000","last_peered":"2024-02-23T00:43:44.746343+0000","last_clean":"2024-02-23T00:43:44.746343+0000","last_became_active":"2024-02-23T00:41:37.229426+0000","last_became_peered":"2024-02-23T00:41:37.229426+0000","last_unstale":"2024-02-23T00:43:44.746343+0000","last_undegraded":"2024-02-23T00:43:44.746343+0000","last_fullsized":"2024-02-23T00:43:44.746343+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.427861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.427861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":976,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433568+0000","last_change":"2024-02-23T00:42:20.635986+0000","last_active":"2024-02-23T00:43:43.433568+0000","last_peered":"2024-02-23T00:43:43.433568+0000","last_clean":"2024-02-23T00:43:43.433568+0000","last_became_active":"2024-02-23T00:41:55.327964+0000","last_became_peered":"2024-02-23T00:41:55.327964+0000","last_unstale":"2024-02-23T00:43:43.433568+0000","last_undegraded":"2024-02-23T00:43:43.433568+0000","last_fullsized":"2024-02-23T00:43:43.433568+0000","mapping_epoch":692,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:42:20.635897+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.635897+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":315,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271512+0000","last_change":"2024-02-23T00:42:14.278845+0000","last_active":"2024-02-23T00:43:44.271512+0000","last_peered":"2024-02-23T00:43:44.271512+0000","last_clean":"2024-02-23T00:43:44.271512+0000","last_became_active":"2024-02-23T00:41:37.228694+0000","last_became_peered":"2024-02-23T00:41:37.228694+0000","last_unstale":"2024-02-23T00:43:44.271512+0000","last_undegraded":"2024-02-23T00:43:44.271512+0000","last_fullsized":"2024-02-23T00:43:44.271512+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.278782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.278782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":121,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.281801+0000","last_change":"2024-02-23T00:42:55.607417+0000","last_active":"2024-02-23T00:43:41.281801+0000","last_peered":"2024-02-23T00:43:41.281801+0000","last_clean":"2024-02-23T00:43:41.281801+0000","last_became_active":"2024-02-23T00:41:37.247946+0000","last_became_peered":"2024-02-23T00:41:37.247946+0000","last_unstale":"2024-02-23T00:43:41.281801+0000","last_undegraded":"2024-02-23T00:43:41.281801+0000","last_fullsized":"2024-02-23T00:43:41.281801+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:55.607352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:55.607352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":716,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748910+0000","last_change":"2024-02-23T00:41:53.497861+0000","last_active":"2024-02-23T00:43:44.748910+0000","last_peered":"2024-02-23T00:43:44.748910+0000","last_clean":"2024-02-23T00:43:44.748910+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:43:44.748910+0000","last_undegraded":"2024-02-23T00:43:44.748910+0000","last_fullsized":"2024-02-23T00:43:44.748910+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:53.497792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.497792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":907,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.272641+0000","last_change":"2024-02-23T00:42:16.497262+0000","last_active":"2024-02-23T00:43:44.272641+0000","last_peered":"2024-02-23T00:43:44.272641+0000","last_clean":"2024-02-23T00:43:44.272641+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:43:44.272641+0000","last_undegraded":"2024-02-23T00:43:44.272641+0000","last_fullsized":"2024-02-23T00:43:44.272641+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:42:16.497198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.497198+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":776,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.433284+0000","last_change":"2024-02-23T00:41:48.632172+0000","last_active":"2024-02-23T00:43:43.433284+0000","last_peered":"2024-02-23T00:43:43.433284+0000","last_clean":"2024-02-23T00:43:43.433284+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:43:43.433284+0000","last_undegraded":"2024-02-23T00:43:43.433284+0000","last_fullsized":"2024-02-23T00:43:43.433284+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:48.632098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:41:48.632098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":781,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259568+0000","last_change":"2024-02-23T00:43:11.402457+0000","last_active":"2024-02-23T00:43:43.259568+0000","last_peered":"2024-02-23T00:43:43.259568+0000","last_clean":"2024-02-23T00:43:43.259568+0000","last_became_active":"2024-02-23T00:41:37.229323+0000","last_became_peered":"2024-02-23T00:41:37.229323+0000","last_unstale":"2024-02-23T00:43:43.259568+0000","last_undegraded":"2024-02-23T00:43:43.259568+0000","last_fullsized":"2024-02-23T00:43:43.259568+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:11.402342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:43:11.402342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":298,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273030+0000","last_change":"2024-02-23T00:43:07.162145+0000","last_active":"2024-02-23T00:43:44.273030+0000","last_peered":"2024-02-23T00:43:44.273030+0000","last_clean":"2024-02-23T00:43:44.273030+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:43:44.273030+0000","last_undegraded":"2024-02-23T00:43:44.273030+0000","last_fullsized":"2024-02-23T00:43:44.273030+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.162083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.162083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":875,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271566+0000","last_change":"2024-02-23T00:40:58.399481+0000","last_active":"2024-02-23T00:43:44.271566+0000","last_peered":"2024-02-23T00:43:44.271566+0000","last_clean":"2024-02-23T00:43:44.271566+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:43:44.271566+0000","last_undegraded":"2024-02-23T00:43:44.271566+0000","last_fullsized":"2024-02-23T00:43:44.271566+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_deep_scrub":"453'44","last_deep_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.399368+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":291,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273621+0000","last_change":"2024-02-23T00:40:07.255442+0000","last_active":"2024-02-23T00:43:44.273621+0000","last_peered":"2024-02-23T00:43:44.273621+0000","last_clean":"2024-02-23T00:43:44.273621+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:43:44.273621+0000","last_undegraded":"2024-02-23T00:43:44.273621+0000","last_fullsized":"2024-02-23T00:43:44.273621+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:07.255349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:07.255349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":601,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273877+0000","last_change":"2024-02-23T00:41:50.257038+0000","last_active":"2024-02-23T00:43:44.273877+0000","last_peered":"2024-02-23T00:43:44.273877+0000","last_clean":"2024-02-23T00:43:44.273877+0000","last_became_active":"2024-02-23T00:41:50.256751+0000","last_became_peered":"2024-02-23T00:41:50.256751+0000","last_unstale":"2024-02-23T00:43:44.273877+0000","last_undegraded":"2024-02-23T00:43:44.273877+0000","last_fullsized":"2024-02-23T00:43:44.273877+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:07.610329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:07.610329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":709,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893996+0000","last_change":"2024-02-23T00:43:07.133088+0000","last_active":"2024-02-23T00:43:43.893996+0000","last_peered":"2024-02-23T00:43:43.893996+0000","last_clean":"2024-02-23T00:43:43.893996+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:43:43.893996+0000","last_undegraded":"2024-02-23T00:43:43.893996+0000","last_fullsized":"2024-02-23T00:43:43.893996+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.133007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.133007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":975,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271058+0000","last_change":"2024-02-23T00:41:40.216294+0000","last_active":"2024-02-23T00:43:44.271058+0000","last_peered":"2024-02-23T00:43:44.271058+0000","last_clean":"2024-02-23T00:43:44.271058+0000","last_became_active":"2024-02-23T00:41:40.215919+0000","last_became_peered":"2024-02-23T00:41:40.215919+0000","last_unstale":"2024-02-23T00:43:44.271058+0000","last_undegraded":"2024-02-23T00:43:44.271058+0000","last_fullsized":"2024-02-23T00:43:44.271058+0000","mapping_epoch":680,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:38:13.528949+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":35,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":478,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.266065+0000","last_change":"2024-02-23T00:42:14.878836+0000","last_active":"2024-02-23T00:43:44.266065+0000","last_peered":"2024-02-23T00:43:44.266065+0000","last_clean":"2024-02-23T00:43:44.266065+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:43:44.266065+0000","last_undegraded":"2024-02-23T00:43:44.266065+0000","last_fullsized":"2024-02-23T00:43:44.266065+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.878770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.878770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":775,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.893788+0000","last_change":"2024-02-23T00:43:00.260324+0000","last_active":"2024-02-23T00:43:43.893788+0000","last_peered":"2024-02-23T00:43:43.893788+0000","last_clean":"2024-02-23T00:43:43.893788+0000","last_became_active":"2024-02-23T00:41:37.228310+0000","last_became_peered":"2024-02-23T00:41:37.228310+0000","last_unstale":"2024-02-23T00:43:43.893788+0000","last_undegraded":"2024-02-23T00:43:43.893788+0000","last_fullsized":"2024-02-23T00:43:43.893788+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:00.260242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.260242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":721,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.745521+0000","last_change":"2024-02-23T00:42:19.323714+0000","last_active":"2024-02-23T00:43:44.745521+0000","last_peered":"2024-02-23T00:43:44.745521+0000","last_clean":"2024-02-23T00:43:44.745521+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:43:44.745521+0000","last_undegraded":"2024-02-23T00:43:44.745521+0000","last_fullsized":"2024-02-23T00:43:44.745521+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.323643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.323643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":306,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271257+0000","last_change":"2024-02-23T00:41:50.256190+0000","last_active":"2024-02-23T00:43:44.271257+0000","last_peered":"2024-02-23T00:43:44.271257+0000","last_clean":"2024-02-23T00:43:44.271257+0000","last_became_active":"2024-02-23T00:41:50.255862+0000","last_became_peered":"2024-02-23T00:41:50.255862+0000","last_unstale":"2024-02-23T00:43:44.271257+0000","last_undegraded":"2024-02-23T00:43:44.271257+0000","last_fullsized":"2024-02-23T00:43:44.271257+0000","mapping_epoch":688,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:41:00.643135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:00.643135+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":792,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748333+0000","last_change":"2024-02-23T00:42:56.638098+0000","last_active":"2024-02-23T00:43:44.748333+0000","last_peered":"2024-02-23T00:43:44.748333+0000","last_clean":"2024-02-23T00:43:44.748333+0000","last_became_active":"2024-02-23T00:41:37.230054+0000","last_became_peered":"2024-02-23T00:41:37.230054+0000","last_unstale":"2024-02-23T00:43:44.748333+0000","last_undegraded":"2024-02-23T00:43:44.748333+0000","last_fullsized":"2024-02-23T00:43:44.748333+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:56.638011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:42:56.638011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":769,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.748257+0000","last_change":"2024-02-23T00:42:26.307705+0000","last_active":"2024-02-23T00:43:44.748257+0000","last_peered":"2024-02-23T00:43:44.748257+0000","last_clean":"2024-02-23T00:43:44.748257+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:43:44.748257+0000","last_undegraded":"2024-02-23T00:43:44.748257+0000","last_fullsized":"2024-02-23T00:43:44.748257+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.307631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.307631+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.307631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":730,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.258440+0000","last_change":"2024-02-23T00:41:47.254406+0000","last_active":"2024-02-23T00:43:43.258440+0000","last_peered":"2024-02-23T00:43:43.258440+0000","last_clean":"2024-02-23T00:43:43.258440+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:43:43.258440+0000","last_undegraded":"2024-02-23T00:43:43.258440+0000","last_fullsized":"2024-02-23T00:43:43.258440+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.254328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.254328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":920,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.265520+0000","last_change":"2024-02-23T00:42:27.911692+0000","last_active":"2024-02-23T00:43:44.265520+0000","last_peered":"2024-02-23T00:43:44.265520+0000","last_clean":"2024-02-23T00:43:44.265520+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:43:44.265520+0000","last_undegraded":"2024-02-23T00:43:44.265520+0000","last_fullsized":"2024-02-23T00:43:44.265520+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:42:27.911611+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.911611+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":315,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.273107+0000","last_change":"2024-02-23T00:42:53.263490+0000","last_active":"2024-02-23T00:43:44.273107+0000","last_peered":"2024-02-23T00:43:44.273107+0000","last_clean":"2024-02-23T00:43:44.273107+0000","last_became_active":"2024-02-23T00:41:37.241142+0000","last_became_peered":"2024-02-23T00:41:37.241142+0000","last_unstale":"2024-02-23T00:43:44.273107+0000","last_undegraded":"2024-02-23T00:43:44.273107+0000","last_fullsized":"2024-02-23T00:43:44.273107+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.263425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.263425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":424,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.432495+0000","last_change":"2024-02-23T00:43:06.705370+0000","last_active":"2024-02-23T00:43:43.432495+0000","last_peered":"2024-02-23T00:43:43.432495+0000","last_clean":"2024-02-23T00:43:43.432495+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:43:43.432495+0000","last_undegraded":"2024-02-23T00:43:43.432495+0000","last_fullsized":"2024-02-23T00:43:43.432495+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.705283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.705283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":203,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.747839+0000","last_change":"2024-02-23T00:43:13.796473+0000","last_active":"2024-02-23T00:43:44.747839+0000","last_peered":"2024-02-23T00:43:44.747839+0000","last_clean":"2024-02-23T00:43:44.747839+0000","last_became_active":"2024-02-23T00:40:23.199068+0000","last_became_peered":"2024-02-23T00:40:23.199068+0000","last_unstale":"2024-02-23T00:43:44.747839+0000","last_undegraded":"2024-02-23T00:43:44.747839+0000","last_fullsized":"2024-02-23T00:43:44.747839+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:13.796391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:13.796391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":1066,"reported_epoch":787,"state":"active+clean","last_fresh":"2024-02-23T00:43:44.271214+0000","last_change":"2024-02-23T00:41:41.223239+0000","last_active":"2024-02-23T00:43:44.271214+0000","last_peered":"2024-02-23T00:43:44.271214+0000","last_clean":"2024-02-23T00:43:44.271214+0000","last_became_active":"2024-02-23T00:41:41.222892+0000","last_became_peered":"2024-02-23T00:41:41.222892+0000","last_unstale":"2024-02-23T00:43:44.271214+0000","last_undegraded":"2024-02-23T00:43:44.271214+0000","last_fullsized":"2024-02-23T00:43:44.271214+0000","mapping_epoch":681,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:41:34.076323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.076323+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":121,"reported_epoch":784,"state":"active+clean","last_fresh":"2024-02-23T00:43:41.282028+0000","last_change":"2024-02-23T00:42:28.612255+0000","last_active":"2024-02-23T00:43:41.282028+0000","last_peered":"2024-02-23T00:43:41.282028+0000","last_clean":"2024-02-23T00:43:41.282028+0000","last_became_active":"2024-02-23T00:41:37.234830+0000","last_became_peered":"2024-02-23T00:41:37.234830+0000","last_unstale":"2024-02-23T00:43:41.282028+0000","last_undegraded":"2024-02-23T00:43:41.282028+0000","last_fullsized":"2024-02-23T00:43:41.282028+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:28.612113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:28.612113+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.612113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":115,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.432832+0000","last_change":"2024-02-23T00:42:13.611259+0000","last_active":"2024-02-23T00:43:43.432832+0000","last_peered":"2024-02-23T00:43:43.432832+0000","last_clean":"2024-02-23T00:43:43.432832+0000","last_became_active":"2024-02-23T00:41:50.253284+0000","last_became_peered":"2024-02-23T00:41:50.253284+0000","last_unstale":"2024-02-23T00:43:43.432832+0000","last_undegraded":"2024-02-23T00:43:43.432832+0000","last_fullsized":"2024-02-23T00:43:43.432832+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:13.611193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:13.611193+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":729,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-02-23T00:43:43.259165+0000","last_change":"2024-02-23T00:42:20.220171+0000","last_active":"2024-02-23T00:43:43.259165+0000","last_peered":"2024-02-23T00:43:43.259165+0000","last_clean":"2024-02-23T00:43:43.259165+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:43:43.259165+0000","last_undegraded":"2024-02-23T00:43:43.259165+0000","last_fullsized":"2024-02-23T00:43:43.259165+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.220094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.220094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":252,"num_pg":32,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":2,"num_evict_kb":2,"num_promote":4,"num_flush_mode_high":2,"num_flush_mode_low":0,"num_evict_mode_some":2,"num_evict_mode_full":0,"num_objects_pinned":2,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":48,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4000289,"num_objects":42,"num_object_clones":1,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":24,"num_read_kb":7,"num_write":62,"num_write_kb":3964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6500352,"data_stored":6500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":24,"acting":24,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":12,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":753,"num_read_kb":612,"num_write":454,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":3961,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":732,"ondisk_log_size":732,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":23,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":665,"num_read_kb":665,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":69,"acting":69,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332297,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304092,"kb_used_data":6868,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93441060,"statfs":{"total":95995035648,"available":95683645440,"internally_reserved":0,"allocated":7032832,"data_stored":3750673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.625,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.432},{"interface":"front","average":{"1min":0.501,"5min":0.49199999999999999,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63600000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.45400000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.47999999999999998,"15min":0.67300000000000004},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.217},"max":{"1min":0.65800000000000003,"5min":1.23,"15min":31.175000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.67500000000000004,"15min":0.73999999999999999},"min":{"1min":0.40999999999999998,"5min":0.29899999999999999,"15min":0.249},"max":{"1min":0.73099999999999998,"5min":11.066000000000001,"15min":29.423999999999999},"last":0.497}]},{"osd":2,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.64100000000000001,"15min":0.71799999999999997},"min":{"1min":0.40400000000000003,"5min":0.28699999999999998,"15min":0.28100000000000003},"max":{"1min":0.72199999999999998,"5min":6.9729999999999999,"15min":16.603999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61699999999999999,"15min":0.64100000000000001},"min":{"1min":0.47999999999999998,"5min":0.29099999999999998,"15min":0.26600000000000001},"max":{"1min":0.71099999999999997,"5min":5.7229999999999999,"15min":7.5339999999999998},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":1.0069999999999999},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":0.66500000000000004,"5min":3.9900000000000002,"15min":57.729999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.91400000000000003},"min":{"1min":0.435,"5min":0.318,"15min":0.29999999999999999},"max":{"1min":0.67900000000000005,"5min":3.8479999999999999,"15min":27.573},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41499999999999998,"15min":0.41799999999999998},"min":{"1min":0.255,"5min":0.214,"15min":0.214},"max":{"1min":0.61699999999999999,"5min":1.2589999999999999,"15min":1.706},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.43099999999999999,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.69399999999999995,"5min":0.74299999999999999,"15min":8.1359999999999992},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.68300000000000005,"5min":0.77000000000000002,"15min":1.0209999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46100000000000002,"15min":0.46800000000000003},"min":{"1min":0.29199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67100000000000004,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.51800000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.47299999999999998,"15min":0.48799999999999999},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.70299999999999996,"5min":1.3129999999999999,"15min":2.1339999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.48399999999999999,"15min":0.499},"min":{"1min":0.33100000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73099999999999998,"5min":0.80500000000000005,"15min":5.194},"last":0.58999999999999997}]}]},{"osd":6,"up_from":44,"seq":188978561230,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308244,"kb_used_data":6924,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436908,"statfs":{"total":95995035648,"available":95679393792,"internally_reserved":0,"allocated":7090176,"data_stored":3819867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.73299999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.502,"15min":0.5},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55000000000000004,"15min":1.024},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.79500000000000004,"5min":6.056,"15min":59.347999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.77800000000000002,"15min":1.3819999999999999},"min":{"1min":0.36899999999999999,"5min":0.24299999999999999,"15min":0.20999999999999999},"max":{"1min":0.67100000000000004,"5min":18.423999999999999,"15min":87.221000000000004},"last":0.63900000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.78900000000000003,"15min":0.90600000000000003},"min":{"1min":0.39600000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.66100000000000003,"5min":18.381,"15min":25.550999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.67700000000000005,"15min":0.91300000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":6.1749999999999998,"15min":40.914000000000001},"last":0.627}]},{"osd":3,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.65200000000000002,"15min":0.85999999999999999},"min":{"1min":0.32500000000000001,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":0.751,"5min":7.1630000000000003,"15min":31.309999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.77300000000000002,"15min":1.4079999999999999},"min":{"1min":0.35799999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.89900000000000002,"5min":18.448,"15min":86.311999999999998},"last":0.67000000000000004}]},{"osd":4,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.55200000000000005,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.251,"15min":0.23699999999999999},"max":{"1min":0.66900000000000004,"5min":7.8769999999999998,"15min":7.8769999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.502,"15min":0.53500000000000003},"min":{"1min":0.311,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.69099999999999995,"5min":6.0300000000000002,"15min":17.718},"last":0.68400000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.57099999999999995,"15min":0.52500000000000002},"min":{"1min":0.32300000000000001,"5min":0.28499999999999998,"15min":0.254},"max":{"1min":0.86899999999999999,"5min":6.7999999999999998,"15min":8.0990000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.97999999999999998,"5min":6.0780000000000003,"15min":8.1370000000000005},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:42:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54000000000000004,"15min":0.61899999999999999},"min":{"1min":0.32100000000000001,"5min":0.28999999999999998,"15min":0.27300000000000002},"max":{"1min":0.70799999999999996,"5min":6.4279999999999999,"15min":17.881},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.26500000000000001},"max":{"1min":0.74399999999999999,"5min":18.475000000000001,"15min":18.475000000000001},"last":0.70599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314284,"kb_used_data":8868,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93430868,"statfs":{"total":95995035648,"available":95673208832,"internally_reserved":0,"allocated":9080832,"data_stored":5769275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.40600000000000003,"15min":0.42299999999999999},"min":{"1min":0.20899999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.47399999999999998,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.249,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.53900000000000003,"5min":1.161,"15min":1.161},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Feb 23 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36699999999999999,"15min":0.66700000000000004},"min":{"1min":0.20799999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.54000000000000004,"5min":3.9980000000000002,"15min":35.313000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.379,"5min":0.39700000000000002,"15min":0.57899999999999996},"min":{"1min":0.27300000000000002,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.623,"5min":2.7999999999999998,"15min":27.908999999999999},"last":1.5549999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.44900000000000001,"15min":0.64600000000000002},"min":{"1min":0.245,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.52400000000000002,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41399999999999998,"15min":0.63300000000000001},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47799999999999998,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.86299999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58899999999999997,"15min":0.70699999999999996},"min":{"1min":0.42499999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.68300000000000005,"5min":3.4900000000000002,"15min":16.231000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52300000000000002,"15min":0.71399999999999997},"min":{"1min":0.36799999999999999,"5min":0.29599999999999999,"15min":0.28999999999999998},"max":{"1min":0.66600000000000004,"5min":1.7350000000000001,"15min":29.212},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.57299999999999995,"15min":0.71699999999999997},"min":{"1min":0.371,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72099999999999997,"5min":2.8410000000000002,"15min":16.294},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.59899999999999998,"15min":0.68200000000000005},"min":{"1min":0.35399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72899999999999998,"5min":4.3380000000000001,"15min":11.105},"last":0.755}]},{"osd":6,"last update":"Fri Feb 23 00:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53100000000000003,"15min":0.749},"min":{"1min":0.33700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67700000000000005,"5min":2.4049999999999998,"15min":29.187000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.625,"15min":0.747},"min":{"1min":0.40400000000000003,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.81599999999999995,"5min":2.927,"15min":16.369},"last":0.624}]},{"osd":7,"last update":"Fri Feb 23 00:43:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64800000000000002,"15min":0.79000000000000004},"min":{"1min":0.46600000000000003,"5min":0.33300000000000002,"15min":0.32000000000000001},"max":{"1min":0.75800000000000001,"5min":5.2119999999999997,"15min":16.395},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.622,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.35599999999999998,"15min":0.34300000000000003},"max":{"1min":0.82699999999999996,"5min":3.7669999999999999,"15min":11.135999999999999},"last":0.77800000000000002}]}]},{"osd":0,"up_from":659,"seq":2830383448094,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309764,"kb_used_data":6844,"kb_used_omap":85,"kb_used_meta":302826,"kb_avail":93435388,"statfs":{"total":95995035648,"available":95677837312,"internally_reserved":0,"allocated":7008256,"data_stored":3743583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":310093952},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.52500000000000002,"15min":0.55500000000000005},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.60599999999999998,"5min":5.0880000000000001,"15min":5.0880000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.40899999999999997,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.53200000000000003,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.34100000000000003,"15min":0.33900000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55200000000000005,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.55800000000000005,"15min":0.59199999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56999999999999995,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.90800000000000003,"15min":0.996},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":11.891,"15min":11.891},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":1.095,"15min":1.216},"min":{"1min":0.29099999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.54100000000000004,"5min":15.864000000000001,"15min":15.864000000000001},"last":0.53600000000000003}]},{"osd":4,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.53600000000000003,"15min":0.52700000000000002},"min":{"1min":0.42599999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.53600000000000003,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.82099999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.41599999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.62,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.83699999999999997,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.95799999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.746,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.997}]},{"osd":7,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64400000000000002,"15min":0.64500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80700000000000005,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61799999999999999,"15min":0.61399999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.93700000000000006}]}]},{"osd":2,"up_from":18,"seq":77309411553,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320492,"kb_used_data":6884,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93424660,"statfs":{"total":95995035648,"available":95666851840,"internally_reserved":0,"allocated":7049216,"data_stored":3781313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.30599999999999999,"15min":0.30299999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":0.43099999999999999,"5min":0.503,"15min":0.503},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.254}]},{"osd":1,"last update":"Fri Feb 23 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34300000000000003,"15min":0.59399999999999997},"min":{"1min":0.23300000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.84399999999999997,"5min":1.8919999999999999,"15min":19.324999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.47399999999999998,"15min":0.752},"min":{"1min":0.251,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.68500000000000005,"5min":7.9779999999999998,"15min":35.052},"last":0.24099999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38100000000000001,"15min":0.72899999999999998},"min":{"1min":0.24099999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.46999999999999997,"5min":3.944,"15min":43.5},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40699999999999997,"15min":0.68000000000000005},"min":{"1min":0.28199999999999997,"5min":0.22,"15min":0.22},"max":{"1min":0.54300000000000004,"5min":3.0230000000000001,"15min":27.414000000000001},"last":0.29899999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:43:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52600000000000002,"15min":0.67200000000000004},"min":{"1min":0.372,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.81999999999999995,"5min":0.93999999999999995,"15min":14.614000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53900000000000003,"15min":0.624},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.27600000000000002},"max":{"1min":0.79300000000000004,"5min":0.94399999999999995,"15min":7.766},"last":0.41899999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.316},"max":{"1min":0.77200000000000002,"5min":7.915,"15min":17.765000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.55500000000000005,"15min":0.67400000000000004},"min":{"1min":0.39600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.79600000000000004,"5min":2.129,"15min":14.589},"last":0.316}]},{"osd":6,"last update":"Fri Feb 23 00:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55500000000000005,"15min":0.63100000000000001},"min":{"1min":0.379,"5min":0.316,"15min":0.29699999999999999},"max":{"1min":0.78900000000000003,"5min":1.944,"15min":7.7869999999999999},"last":0.374},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.72099999999999997,"15min":0.79300000000000004},"min":{"1min":0.38300000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.82199999999999995,"5min":7.9589999999999996,"15min":20.574999999999999},"last":0.36399999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:42:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.70499999999999996,"15min":1.073},"min":{"1min":0.38700000000000001,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.81599999999999995,"5min":7.9480000000000004,"15min":66.632000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58199999999999996,"15min":0.84799999999999998},"min":{"1min":0.44800000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78800000000000003,"5min":2.1789999999999998,"15min":19.603999999999999},"last":0.38100000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149917,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314284,"kb_used_data":8868,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93430868,"statfs":{"total":95995035648,"available":95673208832,"internally_reserved":0,"allocated":9080832,"data_stored":5775528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.38400000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":0.245},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33400000000000002,"15min":0.32900000000000001},"min":{"1min":0.245,"5min":0.215,"15min":0.215},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.35099999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40600000000000003,"15min":0.60299999999999998},"min":{"1min":0.19600000000000001,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":1.1870000000000001,"5min":3.2970000000000002,"15min":54.351999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.45700000000000002,"15min":0.77200000000000002},"min":{"1min":0.246,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":0.53400000000000003,"5min":3.7120000000000002,"15min":55.337000000000003},"last":0.42099999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.44900000000000001,"15min":0.752},"min":{"1min":0.27100000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.44700000000000001,"5min":3.7570000000000001,"15min":55.368000000000002},"last":0.31},{"interface":"front","average":{"1min":0.438,"5min":0.46400000000000002,"15min":0.73099999999999998},"min":{"1min":0.28399999999999997,"5min":0.214,"15min":0.185},"max":{"1min":2.0059999999999998,"5min":4.181,"15min":54.383000000000003},"last":0.32000000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65200000000000002,"15min":0.93799999999999994},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.27900000000000003},"max":{"1min":2.5720000000000001,"5min":6.8799999999999999,"15min":54.530000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.68000000000000005,"15min":0.65000000000000002},"min":{"1min":0.35599999999999998,"5min":0.30199999999999999,"15min":0.24099999999999999},"max":{"1min":3.8969999999999998,"5min":5.8109999999999999,"15min":8.0359999999999996},"last":0.40999999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56000000000000005,"15min":0.877},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.28899999999999998},"max":{"1min":0.90900000000000003,"5min":3.8180000000000001,"15min":55.401000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.66000000000000003,"15min":0.86299999999999999},"min":{"1min":0.35899999999999999,"5min":0.32700000000000001,"15min":0.30299999999999999},"max":{"1min":0.71499999999999997,"5min":7.0179999999999998,"15min":54.454999999999998},"last":0.38300000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.82599999999999996,"15min":0.86899999999999999},"min":{"1min":0.39800000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.7879999999999998,"5min":9.7490000000000006,"15min":54.305},"last":0.437},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.92800000000000005,"15min":1.0529999999999999},"min":{"1min":0.42399999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":5.1509999999999998,"5min":9.6820000000000004,"15min":55.424999999999997},"last":0.54600000000000004}]},{"osd":7,"last update":"Fri Feb 23 00:42:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.66600000000000004,"15min":0.80200000000000005},"min":{"1min":0.45500000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.6710000000000003,"5min":4.9790000000000001,"15min":54.222000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.95299999999999996,"15min":1.034},"min":{"1min":0.41799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.0459999999999998,"5min":23.928000000000001,"15min":55.460000000000001},"last":0.60099999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986392,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312288,"kb_used_data":6872,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432864,"statfs":{"total":95995035648,"available":95675252736,"internally_reserved":0,"allocated":7036928,"data_stored":3751012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.51500000000000001,"15min":0.505},"min":{"1min":0.38100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51100000000000001,"15min":0.50600000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.36799999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.626,"15min":1.107},"min":{"1min":0.39700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.748,"5min":8.0540000000000003,"15min":41.137999999999998},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.67400000000000004,"15min":0.93600000000000005},"min":{"1min":0.40500000000000003,"5min":0.255,"15min":0.255},"max":{"1min":1.4510000000000001,"5min":10.952999999999999,"15min":26.210000000000001},"last":0.48199999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.66800000000000004,"15min":1.198},"min":{"1min":0.39200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.472,"5min":9.3490000000000002,"15min":67.253},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.85599999999999998,"15min":1.129},"min":{"1min":0.49299999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.76700000000000002,"5min":14.442,"15min":41.927999999999997},"last":0.499}]},{"osd":3,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.70999999999999996,"15min":1.1299999999999999},"min":{"1min":0.443,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84499999999999997,"5min":9.5579999999999998,"15min":36.301000000000002},"last":0.441},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.57899999999999996,"15min":1.2010000000000001},"min":{"1min":0.39200000000000002,"5min":0.26700000000000002,"15min":0.24299999999999999},"max":{"1min":0.67000000000000004,"5min":5.1879999999999997,"15min":42.064},"last":0.52100000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.56899999999999995},"min":{"1min":0.28699999999999998,"5min":0.22500000000000001,"15min":0.217},"max":{"1min":0.58099999999999996,"5min":1.4279999999999999,"15min":14.242000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.45400000000000001,"15min":0.61599999999999999},"min":{"1min":0.33500000000000002,"5min":0.255,"15min":0.24299999999999999},"max":{"1min":0.92300000000000004,"5min":1.4079999999999999,"15min":14.124000000000001},"last":0.55600000000000005}]},{"osd":6,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.48899999999999999,"15min":0.58199999999999996},"min":{"1min":0.35499999999999998,"5min":0.28999999999999998,"15min":0.254},"max":{"1min":0.753,"5min":1.734,"15min":8.0730000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57399999999999995,"15min":0.68400000000000005},"min":{"1min":0.32500000000000001,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":1.4039999999999999,"5min":13.339,"15min":45.814999999999998},"last":0.58199999999999996}]},{"osd":7,"last update":"Fri Feb 23 00:43:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.64800000000000002},"min":{"1min":0.35099999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.95399999999999996,"5min":7.9210000000000003,"15min":17.178999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59299999999999997,"15min":0.72499999999999998},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.28299999999999997},"max":{"1min":0.875,"5min":6.3220000000000001,"15min":15.85},"last":0.61799999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757459,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306060,"kb_used_data":8836,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93439092,"statfs":{"total":95995035648,"available":95681630208,"internally_reserved":0,"allocated":9048064,"data_stored":5780592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68799999999999994,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.33200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72399999999999998,"5min":1.087,"15min":1.087},"last":0.55900000000000005}]},{"osd":1,"last update":"Fri Feb 23 00:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45300000000000001,"15min":0.753},"min":{"1min":0.36499999999999999,"5min":0.27500000000000002,"15min":0.24099999999999999},"max":{"1min":1.04,"5min":1.1180000000000001,"15min":38.838000000000001},"last":0.628},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.44,"15min":1.1779999999999999},"min":{"1min":0.312,"5min":0.252,"15min":0.23699999999999999},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":127.794},"last":0.48799999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.64300000000000002,"15min":0.95399999999999996},"min":{"1min":0.32900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.71699999999999997,"5min":11.435,"15min":37.442999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.47599999999999998,"15min":0.69899999999999995},"min":{"1min":0.32600000000000001,"5min":0.27600000000000002,"15min":0.249},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":23.297999999999998},"last":2.875}]},{"osd":3,"last update":"Fri Feb 23 00:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.61099999999999999,"15min":0.82799999999999996},"min":{"1min":0.371,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.0049999999999999,"5min":11.457000000000001,"15min":24.414999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.49299999999999999,"15min":0.90800000000000003},"min":{"1min":0.317,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":40.865000000000002},"last":0.60399999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.372,"15min":0.78600000000000003},"min":{"1min":0.253,"5min":0.22500000000000001,"15min":0.219},"max":{"1min":0.753,"5min":0.753,"15min":103.13},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.51600000000000001,"15min":0.52200000000000002},"min":{"1min":0.28499999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.745,"5min":11.381,"15min":12.795999999999999},"last":0.91100000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.33900000000000002,"5min":0.26400000000000001,"15min":0.26200000000000001},"max":{"1min":0.67800000000000005,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.45100000000000001,"15min":0.83199999999999996},"min":{"1min":0.30599999999999999,"5min":0.27000000000000002,"15min":0.26200000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":103.61499999999999},"last":0.65500000000000003}]},{"osd":7,"last update":"Fri Feb 23 00:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59499999999999997,"15min":0.56899999999999995},"min":{"1min":0.45300000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.298,"5min":11.478,"15min":11.478},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.50700000000000001,"15min":0.53100000000000003},"min":{"1min":0.41699999999999998,"5min":0.27900000000000003,"15min":0.25800000000000001},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":5.4050000000000002},"last":1.0740000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2064384,"data_stored":2064384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2064384,"data_stored":2064384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2068480,"data_stored":2068480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":252,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:43:48.786 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [7, 1, 6, 5, 0, 4] 2024-02-23T00:43:48.786 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.7', '7', '1', '6', '5', '0', '4'] 2024-02-23T00:43:48.786 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pg-upmap-items 2.7 7 1 6 5 0 4 2024-02-23T00:43:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:48.788 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:48.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:48.889 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:48.917 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:49.040 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:49.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:49.146 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:49.178 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:49.200 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:49.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:49.304 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:47.805038+0000 mgr.y (mgr.24437) 669 : cluster [DBG] pgmap v1210: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 224 B/s wr, 26 op/s; 9/825 objects misplaced (1.091%); 145 B/s, 6 objects/s recovering 2024-02-23T00:43:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:48.071928+0000 osd.3 (osd.3) 251 : cluster [DBG] 3.b scrub starts 2024-02-23T00:43:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:48.075023+0000 osd.3 (osd.3) 252 : cluster [DBG] 3.b scrub ok 2024-02-23T00:43:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:48.961003+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:43:49.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:48.961901+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:43:49.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:47.805038+0000 mgr.y (mgr.24437) 669 : cluster [DBG] pgmap v1210: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 224 B/s wr, 26 op/s; 9/825 objects misplaced (1.091%); 145 B/s, 6 objects/s recovering 2024-02-23T00:43:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:48.071928+0000 osd.3 (osd.3) 251 : cluster [DBG] 3.b scrub starts 2024-02-23T00:43:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:48.075023+0000 osd.3 (osd.3) 252 : cluster [DBG] 3.b scrub ok 2024-02-23T00:43:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:48.961003+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:43:49.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:48.961901+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:43:49.334 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:49.353 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:49.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:49.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:47.805038+0000 mgr.y (mgr.24437) 669 : cluster [DBG] pgmap v1210: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 224 B/s wr, 26 op/s; 9/825 objects misplaced (1.091%); 145 B/s, 6 objects/s recovering 2024-02-23T00:43:49.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:48.071928+0000 osd.3 (osd.3) 251 : cluster [DBG] 3.b scrub starts 2024-02-23T00:43:49.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:48.075023+0000 osd.3 (osd.3) 252 : cluster [DBG] 3.b scrub ok 2024-02-23T00:43:49.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:48.961003+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:43:49.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:48.961901+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:43:49.457 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:49.484 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:49.504 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:49.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:49.607 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:49.638 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:49.788 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:43:49.988 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:49.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:49.995 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:50.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:50.118 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:50.317 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:50.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:50.421 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:50.454 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:50.475 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:50.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:50.579 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:48.821432+0000 osd.6 (osd.6) 328 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:43:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:48.824472+0000 osd.6 (osd.6) 329 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:43:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:49.535397+0000 osd.1 (osd.1) 358 : cluster [DBG] 4.f scrub starts 2024-02-23T00:43:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:49.538974+0000 osd.1 (osd.1) 359 : cluster [DBG] 4.f scrub ok 2024-02-23T00:43:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:49.991374+0000 osd.5 (osd.5) 309 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:43:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:49.992268+0000 osd.5 (osd.5) 310 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:43:50.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:50.175204+0000 mon.a (mon.0) 3879 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:48.821432+0000 osd.6 (osd.6) 328 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:43:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:48.824472+0000 osd.6 (osd.6) 329 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:43:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:49.535397+0000 osd.1 (osd.1) 358 : cluster [DBG] 4.f scrub starts 2024-02-23T00:43:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:49.538974+0000 osd.1 (osd.1) 359 : cluster [DBG] 4.f scrub ok 2024-02-23T00:43:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:49.991374+0000 osd.5 (osd.5) 309 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:43:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:49.992268+0000 osd.5 (osd.5) 310 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:43:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:50.175204+0000 mon.a (mon.0) 3879 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:50.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:48.821432+0000 osd.6 (osd.6) 328 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:43:50.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:48.824472+0000 osd.6 (osd.6) 329 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:43:50.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:49.535397+0000 osd.1 (osd.1) 358 : cluster [DBG] 4.f scrub starts 2024-02-23T00:43:50.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:49.538974+0000 osd.1 (osd.1) 359 : cluster [DBG] 4.f scrub ok 2024-02-23T00:43:50.606 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:49.991374+0000 osd.5 (osd.5) 309 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:43:50.607 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:49.992268+0000 osd.5 (osd.5) 310 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:43:50.607 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:50.175204+0000 mon.a (mon.0) 3879 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:43:50.608 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:50.628 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:50.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:50.732 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:50.760 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:51.208 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:51.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:51.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:51.312 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:51.344 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:51.365 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:51.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:51.469 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:49.805529+0000 mgr.y (mgr.24437) 670 : cluster [DBG] pgmap v1211: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 170 B/s wr, 20 op/s; 9/825 objects misplaced (1.091%); 109 B/s, 4 objects/s recovering 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:50.460696+0000 osd.2 (osd.2) 283 : cluster [DBG] 4.d scrub starts 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:50.462508+0000 osd.2 (osd.2) 284 : cluster [DBG] 4.d scrub ok 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:50.958011+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:50.958979+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:50.975578+0000 osd.7 (osd.7) 362 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:50.976621+0000 osd.7 (osd.7) 363 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:49.805529+0000 mgr.y (mgr.24437) 670 : cluster [DBG] pgmap v1211: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 170 B/s wr, 20 op/s; 9/825 objects misplaced (1.091%); 109 B/s, 4 objects/s recovering 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:50.460696+0000 osd.2 (osd.2) 283 : cluster [DBG] 4.d scrub starts 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:50.462508+0000 osd.2 (osd.2) 284 : cluster [DBG] 4.d scrub ok 2024-02-23T00:43:51.495 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:50.958011+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:43:51.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:50.958979+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:43:51.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:50.975578+0000 osd.7 (osd.7) 362 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:43:51.496 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:50.976621+0000 osd.7 (osd.7) 363 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:43:51.497 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:51.535 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:51.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:51.639 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:51.668 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:51.688 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:51.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:51.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:43:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:43:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:49.805529+0000 mgr.y (mgr.24437) 670 : cluster [DBG] pgmap v1211: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 170 B/s wr, 20 op/s; 9/825 objects misplaced (1.091%); 109 B/s, 4 objects/s recovering 2024-02-23T00:43:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:50.460696+0000 osd.2 (osd.2) 283 : cluster [DBG] 4.d scrub starts 2024-02-23T00:43:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:50.462508+0000 osd.2 (osd.2) 284 : cluster [DBG] 4.d scrub ok 2024-02-23T00:43:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:50.958011+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:43:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:50.958979+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:43:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:50.975578+0000 osd.7 (osd.7) 362 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:43:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:50.976621+0000 osd.7 (osd.7) 363 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:43:51.791 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:51.807 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:43:51.807 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:43:51.823 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:51.846 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:51.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:51.950 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:51.980 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:52.002 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:52.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:52.013 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-23T00:43:52.082 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:52.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:52.132 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:52.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:52] "GET /metrics HTTP/1.1" 200 190861 "" "Prometheus/2.33.4" 2024-02-23T00:43:52.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:51.287598+0000 mon.a (mon.0) 3880 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:51.315038+0000 mgr.y (mgr.24437) 671 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:51.344788+0000 mon.a (mon.0) 3881 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:51.344852+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:51.346625+0000 mon.a (mon.0) 3883 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136"}]: dispatch 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:51.991575+0000 osd.7 (osd.7) 364 : cluster [DBG] 22.d scrub starts 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:51.994767+0000 osd.7 (osd.7) 365 : cluster [DBG] 22.d scrub ok 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:52.039934+0000 osd.4 (osd.4) 345 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:52.040801+0000 osd.4 (osd.4) 346 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:52.254239+0000 mon.a (mon.0) 3884 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136"}]': finished 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:52.254296+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:52.280597+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:52.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:52.280936+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136"}]: dispatch 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:51.287598+0000 mon.a (mon.0) 3880 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:51.315038+0000 mgr.y (mgr.24437) 671 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:51.344788+0000 mon.a (mon.0) 3881 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:51.344852+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:51.346625+0000 mon.a (mon.0) 3883 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136"}]: dispatch 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:51.991575+0000 osd.7 (osd.7) 364 : cluster [DBG] 22.d scrub starts 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:51.994767+0000 osd.7 (osd.7) 365 : cluster [DBG] 22.d scrub ok 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:52.039934+0000 osd.4 (osd.4) 345 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:52.040801+0000 osd.4 (osd.4) 346 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:52.254239+0000 mon.a (mon.0) 3884 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136"}]': finished 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:52.254296+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:52.280597+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:52.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:52.280936+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136"}]: dispatch 2024-02-23T00:43:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:51.287598+0000 mon.a (mon.0) 3880 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:51.315038+0000 mgr.y (mgr.24437) 671 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:43:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:51.344788+0000 mon.a (mon.0) 3881 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:51.344852+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:51.346625+0000 mon.a (mon.0) 3883 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136"}]: dispatch 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:51.991575+0000 osd.7 (osd.7) 364 : cluster [DBG] 22.d scrub starts 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:51.994767+0000 osd.7 (osd.7) 365 : cluster [DBG] 22.d scrub ok 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:52.039934+0000 osd.4 (osd.4) 345 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:52.040801+0000 osd.4 (osd.4) 346 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:52.254239+0000 mon.a (mon.0) 3884 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136"}]': finished 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:52.254296+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:52.280597+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:52.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:52.280936+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-136"}]: dispatch 2024-02-23T00:43:52.981 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:43:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:52.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:43:53.092 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:53.124 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:53.145 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:53.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:53.250 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:53.278 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:53.301 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:53.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:53.350 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-23T00:43:53.404 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:53.432 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:51.805895+0000 mgr.y (mgr.24437) 672 : cluster [DBG] pgmap v1213: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 156 B/s wr, 19 op/s; 9/825 objects misplaced (1.091%); 100 B/s, 4 objects/s recovering 2024-02-23T00:43:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:51.831643+0000 osd.6 (osd.6) 330 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:43:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:51.834510+0000 osd.6 (osd.6) 331 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:43:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.035524+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:43:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.037096+0000 osd.7 (osd.7) 367 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:43:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.067430+0000 osd.4 (osd.4) 347 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:43:53.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.069242+0000 osd.4 (osd.4) 348 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:43:53.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.293345+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-02-23T00:43:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:51.805895+0000 mgr.y (mgr.24437) 672 : cluster [DBG] pgmap v1213: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 156 B/s wr, 19 op/s; 9/825 objects misplaced (1.091%); 100 B/s, 4 objects/s recovering 2024-02-23T00:43:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:51.831643+0000 osd.6 (osd.6) 330 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:43:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:51.834510+0000 osd.6 (osd.6) 331 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:43:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.035524+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:43:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.037096+0000 osd.7 (osd.7) 367 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:43:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.067430+0000 osd.4 (osd.4) 347 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:43:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.069242+0000 osd.4 (osd.4) 348 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:43:53.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.293345+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-02-23T00:43:53.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:43:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:43:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:43:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:51.805895+0000 mgr.y (mgr.24437) 672 : cluster [DBG] pgmap v1213: 204 pgs: 1 active+remapped+backfill_toofull, 203 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 156 B/s wr, 19 op/s; 9/825 objects misplaced (1.091%); 100 B/s, 4 objects/s recovering 2024-02-23T00:43:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:51.831643+0000 osd.6 (osd.6) 330 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:43:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:51.834510+0000 osd.6 (osd.6) 331 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:43:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.035524+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:43:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.037096+0000 osd.7 (osd.7) 367 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:43:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.067430+0000 osd.4 (osd.4) 347 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:43:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.069242+0000 osd.4 (osd.4) 348 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:43:53.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.293345+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-02-23T00:43:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 -- 172.21.15.87:0/1599726660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf140fcb40 msgr2=0x7fcf140fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/1599726660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf140fcb40 0x7fcf140fcf20 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 rx=0x7fcf04004870 tx=0x7fcf04009cb0).stop 2024-02-23T00:43:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 -- 172.21.15.87:0/1599726660 shutdown_connections 2024-02-23T00:43:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/1599726660 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf140f7ba0 0x7fcf14103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/1599726660 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf140fcb40 0x7fcf140fcf20 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/1599726660 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf140fd4a0 0x7fcf140f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:54.345 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 -- 172.21.15.87:0/1599726660 >> 172.21.15.87:0/1599726660 conn(0x7fcf1406c370 msgr2=0x7fcf1406c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 -- 172.21.15.87:0/1599726660 shutdown_connections 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 -- 172.21.15.87:0/1599726660 wait complete. 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 librados: starting msgr at 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 librados: starting objecter 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.235+0000 7fcf18ff6700 1 Processor -- start 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 -- start start 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 librados: setting wanted keys 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 librados: calling monclient init 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf140f7ba0 0x7fcf1410cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf140fcb40 0x7fcf1410d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:54.346 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf140fd4a0 0x7fcf141071b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf141076a0 con 0x7fcf140fcb40 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf14107810 con 0x7fcf140f7ba0 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf141079d0 con 0x7fcf140fd4a0 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf12d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf140fd4a0 0x7fcf141071b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf11d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf140fcb40 0x7fcf1410d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf12d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf140fd4a0 0x7fcf141071b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:49644/0 (socket says 172.21.15.87:49644) 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf12d9d700 1 -- 172.21.15.87:0/2008163805 learned_addr learned my addr 172.21.15.87:0/2008163805 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf1259c700 1 --2- 172.21.15.87:0/2008163805 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf140f7ba0 0x7fcf1410cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf12d9d700 1 -- 172.21.15.87:0/2008163805 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf140f7ba0 msgr2=0x7fcf1410cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf12d9d700 1 --2- 172.21.15.87:0/2008163805 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf140f7ba0 0x7fcf1410cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf12d9d700 1 -- 172.21.15.87:0/2008163805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf140fcb40 msgr2=0x7fcf1410d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf12d9d700 1 --2- 172.21.15.87:0/2008163805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf140fcb40 0x7fcf1410d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:54.347 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf12d9d700 1 -- 172.21.15.87:0/2008163805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c0097c0 con 0x7fcf140fd4a0 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf11d9b700 1 --2- 172.21.15.87:0/2008163805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf140fcb40 0x7fcf1410d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf12d9d700 1 --2- 172.21.15.87:0/2008163805 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf140fd4a0 0x7fcf141071b0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7fcf0c001df0 tx=0x7fcf0c004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf037fe700 10 client.?.objecter ms_handle_connect 0x7fcf140fd4a0 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf037fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf037fe700 1 -- 172.21.15.87:0/2008163805 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c009a10 con 0x7fcf140fd4a0 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf037fe700 1 -- 172.21.15.87:0/2008163805 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf0c0090d0 con 0x7fcf140fd4a0 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf04005040 con 0x7fcf140fd4a0 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf037fe700 1 -- 172.21.15.87:0/2008163805 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c007d30 con 0x7fcf140fd4a0 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 10 client.38387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf14107fc0 con 0x7fcf140fd4a0 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.236+0000 7fcf18ff6700 1 librados: init done 2024-02-23T00:43:54.348 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.237+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1405f910 con 0x7fcf140fd4a0 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf037fe700 1 -- 172.21.15.87:0/2008163805 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fcf0c009240 con 0x7fcf140fd4a0 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf037fe700 1 --2- 172.21.15.87:0/2008163805 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcefc062c90 0x7fcefc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf037fe700 1 -- 172.21.15.87:0/2008163805 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(792..792 src has 400..792) v4 ==== 8302+0+0 (secure 0 0 0) 0x7fcf0c07db90 con 0x7fcf140fd4a0 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf037fe700 10 client.38387.objecter ms_dispatch 0x7fcf14001550 osd_map(792..792 src has 400..792) v4 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf037fe700 3 client.38387.objecter handle_osd_map got epochs [792,792] > 0 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf037fe700 3 client.38387.objecter handle_osd_map decoding full epoch 792 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf1259c700 1 --2- 172.21.15.87:0/2008163805 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcefc062c90 0x7fcefc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf037fe700 20 client.38387.objecter dump_active .. 0 homeless 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf037fe700 10 client.38387.objecter ms_handle_connect 0x7fcefc062c90 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.238+0000 7fcf1259c700 1 --2- 172.21.15.87:0/2008163805 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcefc062c90 0x7fcefc065150 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7fcf040043d0 tx=0x7fcf0400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:54.349 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.243+0000 7fcf037fe700 1 -- 172.21.15.87:0/2008163805 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fcf0c050f10 con 0x7fcf140fd4a0 2024-02-23T00:43:54.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:54.434+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]} v 0) v1 -- 0x7fcf1405ee80 con 0x7fcf140fd4a0 2024-02-23T00:43:54.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:52.875326+0000 osd.6 (osd.6) 332 : cluster [DBG] 4.e scrub starts 2024-02-23T00:43:54.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:52.878906+0000 osd.6 (osd.6) 333 : cluster [DBG] 4.e scrub ok 2024-02-23T00:43:54.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:52.978314+0000 osd.3 (osd.3) 253 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:43:54.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:52.981789+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:43:54.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.540397+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:54.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.540533+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:54.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.806401+0000 mgr.y (mgr.24437) 673 : cluster [DBG] pgmap v1216: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 9/819 objects misplaced (1.099%) 2024-02-23T00:43:54.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:54.298392+0000 mon.a (mon.0) 3889 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-02-23T00:43:54.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:54.300767+0000 mon.a (mon.0) 3890 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:54.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:54.435918+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.87:0/2008163805' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]: dispatch 2024-02-23T00:43:54.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:54.436267+0000 mon.a (mon.0) 3891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]: dispatch 2024-02-23T00:43:54.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:54.439473+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.c scrub starts 2024-02-23T00:43:54.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:54.440318+0000 osd.2 (osd.2) 286 : cluster [DBG] 2.c scrub ok 2024-02-23T00:43:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:52.875326+0000 osd.6 (osd.6) 332 : cluster [DBG] 4.e scrub starts 2024-02-23T00:43:54.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:52.878906+0000 osd.6 (osd.6) 333 : cluster [DBG] 4.e scrub ok 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:52.978314+0000 osd.3 (osd.3) 253 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:52.981789+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.540397+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.540533+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.806401+0000 mgr.y (mgr.24437) 673 : cluster [DBG] pgmap v1216: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 9/819 objects misplaced (1.099%) 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:54.298392+0000 mon.a (mon.0) 3889 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:54.300767+0000 mon.a (mon.0) 3890 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:54.435918+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.87:0/2008163805' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]: dispatch 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:54.436267+0000 mon.a (mon.0) 3891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]: dispatch 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:54.439473+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.c scrub starts 2024-02-23T00:43:54.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:54.440318+0000 osd.2 (osd.2) 286 : cluster [DBG] 2.c scrub ok 2024-02-23T00:43:54.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:52.875326+0000 osd.6 (osd.6) 332 : cluster [DBG] 4.e scrub starts 2024-02-23T00:43:54.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:52.878906+0000 osd.6 (osd.6) 333 : cluster [DBG] 4.e scrub ok 2024-02-23T00:43:54.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:52.978314+0000 osd.3 (osd.3) 253 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:52.981789+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.540397+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.540533+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.806401+0000 mgr.y (mgr.24437) 673 : cluster [DBG] pgmap v1216: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 9/819 objects misplaced (1.099%) 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:54.298392+0000 mon.a (mon.0) 3889 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:54.300767+0000 mon.a (mon.0) 3890 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:54.435918+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.87:0/2008163805' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]: dispatch 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:54.436267+0000 mon.a (mon.0) 3891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]: dispatch 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:54.439473+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.c scrub starts 2024-02-23T00:43:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:54.440318+0000 osd.2 (osd.2) 286 : cluster [DBG] 2.c scrub ok 2024-02-23T00:43:55.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.301+0000 7fcf037fe700 1 -- 172.21.15.87:0/2008163805 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]=0 set 2.7 pg_upmap_items mapping to [7->1,6->5,0->4] v794) v1 ==== 157+0+0 (secure 0 0 0) 0x7fcf0c01c300 con 0x7fcf140fd4a0 2024-02-23T00:43:55.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.304+0000 7fcf18ff6700 10 librados: watch_flush enter 2024-02-23T00:43:55.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.304+0000 7fcf18ff6700 10 librados: watch_flush exit 2024-02-23T00:43:55.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.304+0000 7fcf18ff6700 20 client.38387.objecter shutdown clearing up homeless session... 2024-02-23T00:43:55.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.304+0000 7fcf18ff6700 10 client.38387.objecter successfully canceled tick 2024-02-23T00:43:55.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.305+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcefc062c90 msgr2=0x7fcefc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:55.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.305+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/2008163805 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcefc062c90 0x7fcefc065150 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 rx=0x7fcf040043d0 tx=0x7fcf0400b040).stop 2024-02-23T00:43:55.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.305+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf140fd4a0 msgr2=0x7fcf141071b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:55.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.305+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/2008163805 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf140fd4a0 0x7fcf141071b0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 rx=0x7fcf0c001df0 tx=0x7fcf0c004c50).stop 2024-02-23T00:43:55.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.306+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 shutdown_connections 2024-02-23T00:43:55.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.306+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/2008163805 >> v2:172.21.15.87:6800/1231597463 conn(0x7fcefc062c90 0x7fcefc065150 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:55.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.306+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/2008163805 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcf140fd4a0 0x7fcf141071b0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:55.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.306+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/2008163805 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcf140f7ba0 0x7fcf1410cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:55.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.306+0000 7fcf18ff6700 1 --2- 172.21.15.87:0/2008163805 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcf140fcb40 0x7fcf1410d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:55.309 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.307+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 >> 172.21.15.87:0/2008163805 conn(0x7fcf1406c370 msgr2=0x7fcf140f54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:55.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.308+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 shutdown_connections 2024-02-23T00:43:55.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.308+0000 7fcf18ff6700 1 -- 172.21.15.87:0/2008163805 wait complete. 2024-02-23T00:43:55.310 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:55.308+0000 7fcf18ff6700 1 librados: shutdown 2024-02-23T00:43:55.311 INFO:teuthology.orchestra.run.smithi087.stderr:set 2.7 pg_upmap_items mapping to [7->1,6->5,0->4] 2024-02-23T00:43:55.812 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:55.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.857237+0000 osd.6 (osd.6) 334 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:43:55.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:53.860826+0000 osd.6 (osd.6) 335 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:43:55.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:54.470315+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:43:55.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:54.473951+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:43:55.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:54.552703+0000 mon.a (mon.0) 3892 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-23T00:43:55.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:54.974591+0000 osd.7 (osd.7) 368 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:43:55.814 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:54.978035+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:55.302118+0000 mon.a (mon.0) 3893 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:55.302304+0000 mon.a (mon.0) 3894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]': finished 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:55.302384+0000 mon.a (mon.0) 3895 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:55.424649+0000 mon.a (mon.0) 3896 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:55.426326+0000 osd.2 (osd.2) 287 : cluster [DBG] 234.5 scrub starts 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:55.427935+0000 osd.2 (osd.2) 288 : cluster [DBG] 234.5 scrub ok 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.857237+0000 osd.6 (osd.6) 334 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:53.860826+0000 osd.6 (osd.6) 335 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:54.470315+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:54.473951+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:43:55.815 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:54.552703+0000 mon.a (mon.0) 3892 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-23T00:43:55.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:54.974591+0000 osd.7 (osd.7) 368 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:43:55.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:54.978035+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:43:55.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:55.302118+0000 mon.a (mon.0) 3893 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:55.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:55.302304+0000 mon.a (mon.0) 3894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]': finished 2024-02-23T00:43:55.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:55.302384+0000 mon.a (mon.0) 3895 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-02-23T00:43:55.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:55.424649+0000 mon.a (mon.0) 3896 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:55.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:55.426326+0000 osd.2 (osd.2) 287 : cluster [DBG] 234.5 scrub starts 2024-02-23T00:43:55.816 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:55.427935+0000 osd.2 (osd.2) 288 : cluster [DBG] 234.5 scrub ok 2024-02-23T00:43:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:55.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:55.918 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.857237+0000 osd.6 (osd.6) 334 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:43:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:53.860826+0000 osd.6 (osd.6) 335 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:43:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:54.470315+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:43:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:54.473951+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:43:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:54.552703+0000 mon.a (mon.0) 3892 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-23T00:43:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:54.974591+0000 osd.7 (osd.7) 368 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:43:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:54.978035+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:43:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:55.302118+0000 mon.a (mon.0) 3893 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:43:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:55.302304+0000 mon.a (mon.0) 3894 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.7", "id": [7, 1, 6, 5, 0, 4]}]': finished 2024-02-23T00:43:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:55.302384+0000 mon.a (mon.0) 3895 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-02-23T00:43:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:55.424649+0000 mon.a (mon.0) 3896 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-23T00:43:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:55.426326+0000 osd.2 (osd.2) 287 : cluster [DBG] 234.5 scrub starts 2024-02-23T00:43:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:55.427935+0000 osd.2 (osd.2) 288 : cluster [DBG] 234.5 scrub ok 2024-02-23T00:43:55.948 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:55.968 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:55.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:55.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:56.072 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:43:56.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.071+0000 7fdfb237f700 1 -- 172.21.15.87:0/2564544938 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fd510 msgr2=0x7fdfac0f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:56.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.071+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2564544938 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fd510 0x7fdfac0f6d40 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 rx=0x7fdfa00047b0 tx=0x7fdfa0009cb0).stop 2024-02-23T00:43:56.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.071+0000 7fdfb237f700 1 -- 172.21.15.87:0/2564544938 shutdown_connections 2024-02-23T00:43:56.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.071+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2564544938 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac0f7400 0x7fdfac103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.071+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2564544938 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdfac0fcbb0 0x7fdfac0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.071+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2564544938 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fd510 0x7fdfac0f6d40 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.071+0000 7fdfb237f700 1 -- 172.21.15.87:0/2564544938 >> 172.21.15.87:0/2564544938 conn(0x7fdfac0f2820 msgr2=0x7fdfac0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.071+0000 7fdfb237f700 1 -- 172.21.15.87:0/2564544938 shutdown_connections 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.071+0000 7fdfb237f700 1 -- 172.21.15.87:0/2564544938 wait complete. 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 librados: starting msgr at 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 librados: starting objecter 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 Processor -- start 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 -- start start 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 librados: setting wanted keys 2024-02-23T00:43:56.075 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 librados: calling monclient init 2024-02-23T00:43:56.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac0f7400 0x7fdfac110430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:56.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fcbb0 0x7fdfac110920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:56.078 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdfac0fd510 0x7fdfac10a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac10ab60 con 0x7fdfac0f7400 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac10aca0 con 0x7fdfac0fd510 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.072+0000 7fdfb237f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac10ae60 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfab7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fcbb0 0x7fdfac110920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfab7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fcbb0 0x7fdfac110920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:49674/0 (socket says 172.21.15.87:49674) 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfabfff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac0f7400 0x7fdfac110430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfab7fe700 1 -- 172.21.15.87:0/2379953338 learned_addr learned my addr 172.21.15.87:0/2379953338 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfb091c700 1 --2- 172.21.15.87:0/2379953338 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdfac0fd510 0x7fdfac10a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfab7fe700 1 -- 172.21.15.87:0/2379953338 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdfac0fd510 msgr2=0x7fdfac10a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfab7fe700 1 --2- 172.21.15.87:0/2379953338 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdfac0fd510 0x7fdfac10a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfab7fe700 1 -- 172.21.15.87:0/2379953338 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac0f7400 msgr2=0x7fdfac110430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfab7fe700 1 --2- 172.21.15.87:0/2379953338 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac0f7400 0x7fdfac110430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:56.079 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfab7fe700 1 -- 172.21.15.87:0/2379953338 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa40097c0 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfabfff700 1 --2- 172.21.15.87:0/2379953338 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac0f7400 0x7fdfac110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfab7fe700 1 --2- 172.21.15.87:0/2379953338 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fcbb0 0x7fdfac110920 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7fdfa0009b10 tx=0x7fdfa0009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfa97fa700 10 client.?.objecter ms_handle_connect 0x7fdfac0fcbb0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfa97fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfa97fa700 1 -- 172.21.15.87:0/2379953338 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0018070 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa0005040 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfa97fa700 1 -- 172.21.15.87:0/2379953338 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfa0013e70 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfa97fa700 1 -- 172.21.15.87:0/2379953338 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa001ddb0 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfb237f700 10 client.38393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfac10b300 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.073+0000 7fdfb237f700 1 librados: init done 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.074+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfac10b520 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfa97fa700 1 -- 172.21.15.87:0/2379953338 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fdfa001d420 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.080 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfa97fa700 1 --2- 172.21.15.87:0/2379953338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdf94062c90 0x7fdf94065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:43:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfa97fa700 1 -- 172.21.15.87:0/2379953338 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(794..794 src has 400..794) v4 ==== 8725+0+0 (secure 0 0 0) 0x7fdfa007e960 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfa97fa700 10 client.38393.objecter ms_dispatch 0x7fdfac001550 osd_map(794..794 src has 400..794) v4 2024-02-23T00:43:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfa97fa700 3 client.38393.objecter handle_osd_map got epochs [794,794] > 0 2024-02-23T00:43:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfa97fa700 3 client.38393.objecter handle_osd_map decoding full epoch 794 2024-02-23T00:43:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfabfff700 1 --2- 172.21.15.87:0/2379953338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdf94062c90 0x7fdf94065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:43:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfa97fa700 20 client.38393.objecter dump_active .. 0 homeless 2024-02-23T00:43:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfa97fa700 10 client.38393.objecter ms_handle_connect 0x7fdf94062c90 2024-02-23T00:43:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.075+0000 7fdfabfff700 1 --2- 172.21.15.87:0/2379953338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdf94062c90 0x7fdf94065150 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 rx=0x7fdf9c006fd0 tx=0x7fdf9c009060).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:43:56.081 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.079+0000 7fdfa97fa700 1 -- 172.21.15.87:0/2379953338 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fdfa0051b30 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.104 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:56.125 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:43:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:56.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:43:56.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:56.256 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:56.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.262+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fdfac05eee0 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.302+0000 7fdfa97fa700 1 -- 172.21.15.87:0/2379953338 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v795) v1 ==== 101+0+0 (secure 0 0 0) 0x7fdfa0051720 con 0x7fdfac0fcbb0 2024-02-23T00:43:56.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 10 librados: watch_flush enter 2024-02-23T00:43:56.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 10 librados: watch_flush exit 2024-02-23T00:43:56.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 20 client.38393.objecter shutdown clearing up homeless session... 2024-02-23T00:43:56.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 10 client.38393.objecter successfully canceled tick 2024-02-23T00:43:56.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdf94062c90 msgr2=0x7fdf94065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:56.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2379953338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdf94062c90 0x7fdf94065150 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 rx=0x7fdf9c006fd0 tx=0x7fdf9c009060).stop 2024-02-23T00:43:56.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fcbb0 msgr2=0x7fdfac110920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:43:56.307 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2379953338 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fcbb0 0x7fdfac110920 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 rx=0x7fdfa0009b10 tx=0x7fdfa0009cb0).stop 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 shutdown_connections 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2379953338 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdf94062c90 0x7fdf94065150 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2379953338 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdfac0fd510 0x7fdfac10a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2379953338 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdfac0f7400 0x7fdfac110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 --2- 172.21.15.87:0/2379953338 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdfac0fcbb0 0x7fdfac110920 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 >> 172.21.15.87:0/2379953338 conn(0x7fdfac0f2820 msgr2=0x7fdfac060e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.304+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 shutdown_connections 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.305+0000 7fdfb237f700 1 -- 172.21.15.87:0/2379953338 wait complete. 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:43:56.305+0000 7fdfb237f700 1 librados: shutdown 2024-02-23T00:43:56.308 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:43:56.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:55.455405+0000 osd.0 (osd.0) 59 : cluster [DBG] 4.b scrub starts 2024-02-23T00:43:56.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:55.458876+0000 osd.0 (osd.0) 60 : cluster [DBG] 4.b scrub ok 2024-02-23T00:43:56.572 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:55.807291+0000 mgr.y (mgr.24437) 674 : cluster [DBG] pgmap v1219: 204 pgs: 1 active+remapped+backfill_toofull, 1 peering, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/819 objects misplaced (1.099%) 2024-02-23T00:43:56.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:56.264029+0000 mon.c (mon.2) 506 : audit [INF] from='client.? 172.21.15.87:0/2379953338' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:56.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:56.264350+0000 mon.a (mon.0) 3897 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:56.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:56.294013+0000 mon.a (mon.0) 3898 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:43:56.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:56.301102+0000 mon.a (mon.0) 3899 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:56.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:56.301181+0000 mon.a (mon.0) 3900 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:43:56.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:56.301230+0000 mon.a (mon.0) 3901 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-02-23T00:43:56.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:56.311366+0000 mon.a (mon.0) 3902 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138"}]: dispatch 2024-02-23T00:43:56.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:56.413862+0000 osd.2 (osd.2) 289 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:43:56.573 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:56.415907+0000 osd.2 (osd.2) 290 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:43:56.704 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:56.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:56.811 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:55.455405+0000 osd.0 (osd.0) 59 : cluster [DBG] 4.b scrub starts 2024-02-23T00:43:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:55.458876+0000 osd.0 (osd.0) 60 : cluster [DBG] 4.b scrub ok 2024-02-23T00:43:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:55.807291+0000 mgr.y (mgr.24437) 674 : cluster [DBG] pgmap v1219: 204 pgs: 1 active+remapped+backfill_toofull, 1 peering, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/819 objects misplaced (1.099%) 2024-02-23T00:43:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:56.264029+0000 mon.c (mon.2) 506 : audit [INF] from='client.? 172.21.15.87:0/2379953338' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:56.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:56.264350+0000 mon.a (mon.0) 3897 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:56.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:56.294013+0000 mon.a (mon.0) 3898 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:43:56.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:56.301102+0000 mon.a (mon.0) 3899 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:56.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:56.301181+0000 mon.a (mon.0) 3900 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:43:56.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:56.301230+0000 mon.a (mon.0) 3901 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-02-23T00:43:56.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:56.311366+0000 mon.a (mon.0) 3902 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138"}]: dispatch 2024-02-23T00:43:56.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:56.413862+0000 osd.2 (osd.2) 289 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:43:56.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:56.415907+0000 osd.2 (osd.2) 290 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:55.455405+0000 osd.0 (osd.0) 59 : cluster [DBG] 4.b scrub starts 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:55.458876+0000 osd.0 (osd.0) 60 : cluster [DBG] 4.b scrub ok 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:55.807291+0000 mgr.y (mgr.24437) 674 : cluster [DBG] pgmap v1219: 204 pgs: 1 active+remapped+backfill_toofull, 1 peering, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/819 objects misplaced (1.099%) 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:56.264029+0000 mon.c (mon.2) 506 : audit [INF] from='client.? 172.21.15.87:0/2379953338' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:56.264350+0000 mon.a (mon.0) 3897 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:56.294013+0000 mon.a (mon.0) 3898 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:56.301102+0000 mon.a (mon.0) 3899 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:56.301181+0000 mon.a (mon.0) 3900 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:56.301230+0000 mon.a (mon.0) 3901 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-02-23T00:43:56.841 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:56.311366+0000 mon.a (mon.0) 3902 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138"}]: dispatch 2024-02-23T00:43:56.842 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:56.413862+0000 osd.2 (osd.2) 289 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:43:56.842 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:56.415907+0000 osd.2 (osd.2) 290 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:43:56.843 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:56.864 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:56.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:56.949 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:56.967 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:56.997 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:57.017 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:57.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:57.121 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:57.150 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:57.170 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:57.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:57.274 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:57.302 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:57.322 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:57.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:57.427 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:43:57.454 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:57.472 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:43:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:57.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:43:57.513 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:43:57.575 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:43:57.602 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:57.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:57.027099+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:43:57.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:57.028356+0000 osd.7 (osd.7) 371 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:43:57.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:57.342516+0000 mon.a (mon.0) 3903 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138"}]': finished 2024-02-23T00:43:57.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:57.342589+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-02-23T00:43:57.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:57.381208+0000 mon.a (mon.0) 3905 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:57.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:57.381590+0000 mon.a (mon.0) 3906 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138"}]: dispatch 2024-02-23T00:43:57.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:57.419054+0000 osd.2 (osd.2) 291 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:43:57.822 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:57.419995+0000 osd.2 (osd.2) 292 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:43:57.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:57.027099+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:43:57.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:57.028356+0000 osd.7 (osd.7) 371 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:43:57.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:57.342516+0000 mon.a (mon.0) 3903 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138"}]': finished 2024-02-23T00:43:57.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:57.342589+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-02-23T00:43:57.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:57.381208+0000 mon.a (mon.0) 3905 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:57.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:57.381590+0000 mon.a (mon.0) 3906 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138"}]: dispatch 2024-02-23T00:43:57.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:57.419054+0000 osd.2 (osd.2) 291 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:43:57.823 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:57.419995+0000 osd.2 (osd.2) 292 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:43:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:57.027099+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:43:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:57.028356+0000 osd.7 (osd.7) 371 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:43:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:57.342516+0000 mon.a (mon.0) 3903 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138"}]': finished 2024-02-23T00:43:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:57.342589+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-02-23T00:43:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:57.381208+0000 mon.a (mon.0) 3905 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:43:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:57.381590+0000 mon.a (mon.0) 3906 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-138"}]: dispatch 2024-02-23T00:43:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:57.419054+0000 osd.2 (osd.2) 291 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:43:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:57.419995+0000 osd.2 (osd.2) 292 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:43:58.352 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:43:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:58.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:43:58.458 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:43:58.491 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:58.511 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:43:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:58.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:43:58.616 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:43:58.645 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:58.698 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:43:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:58.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:43:58.801 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:43:58.827 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:59.176 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:43:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:59.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:43:59.280 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:43:59.312 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:59.333 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:43:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:43:59.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:43:59.437 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:43:59.462 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:57.807722+0000 mgr.y (mgr.24437) 675 : cluster [DBG] pgmap v1222: 204 pgs: 1 active+remapped+backfill_toofull, 1 peering, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/819 objects misplaced (1.099%) 2024-02-23T00:43:59.462 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:58.341491+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-02-23T00:43:59.462 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:58.522454+0000 osd.1 (osd.1) 360 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:58.528025+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:58.592288+0000 mon.a (mon.0) 3908 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:58.931904+0000 osd.5 (osd.5) 313 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:58.934063+0000 osd.5 (osd.5) 314 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:57.807722+0000 mgr.y (mgr.24437) 675 : cluster [DBG] pgmap v1222: 204 pgs: 1 active+remapped+backfill_toofull, 1 peering, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/819 objects misplaced (1.099%) 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:58.341491+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:58.522454+0000 osd.1 (osd.1) 360 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:58.528025+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:58.592288+0000 mon.a (mon.0) 3908 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:58.931904+0000 osd.5 (osd.5) 313 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:43:59.463 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:43:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:58.934063+0000 osd.5 (osd.5) 314 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:43:59.464 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:43:59.536 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd set noscrub 2024-02-23T00:43:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:57.807722+0000 mgr.y (mgr.24437) 675 : cluster [DBG] pgmap v1222: 204 pgs: 1 active+remapped+backfill_toofull, 1 peering, 202 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/819 objects misplaced (1.099%) 2024-02-23T00:43:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:58.341491+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-02-23T00:43:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:58.522454+0000 osd.1 (osd.1) 360 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:43:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:58.528025+0000 osd.1 (osd.1) 361 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:43:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:58.592288+0000 mon.a (mon.0) 3908 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:43:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:58.931904+0000 osd.5 (osd.5) 313 : cluster [DBG] 22.1b scrub starts 2024-02-23T00:43:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:43:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:58.934063+0000 osd.5 (osd.5) 314 : cluster [DBG] 22.1b scrub ok 2024-02-23T00:44:00.213 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:44:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:00.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:44:00.319 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:44:00.353 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:00.386 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:44:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:00.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:44:00.482 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:58.941093+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:44:00.482 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:58.944252+0000 osd.3 (osd.3) 256 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:59.347752+0000 mon.a (mon.0) 3909 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:43:59.350751+0000 mon.a (mon.0) 3910 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:59.376841+0000 osd.2 (osd.2) 293 : cluster [DBG] 5.b scrub starts 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:59.378081+0000 osd.2 (osd.2) 294 : cluster [DBG] 5.b scrub ok 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:58.941093+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:58.944252+0000 osd.3 (osd.3) 256 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:59.347752+0000 mon.a (mon.0) 3909 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:43:59.350751+0000 mon.a (mon.0) 3910 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:59.376841+0000 osd.2 (osd.2) 293 : cluster [DBG] 5.b scrub starts 2024-02-23T00:44:00.483 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:59.378081+0000 osd.2 (osd.2) 294 : cluster [DBG] 5.b scrub ok 2024-02-23T00:44:00.491 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:44:00.518 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:00.537 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:44:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:00.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:44:00.641 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:44:00.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:58.941093+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:44:00.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:58.944252+0000 osd.3 (osd.3) 256 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:44:00.666 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:59.347752+0000 mon.a (mon.0) 3909 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-02-23T00:44:00.666 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:43:59.350751+0000 mon.a (mon.0) 3910 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:44:00.666 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:59.376841+0000 osd.2 (osd.2) 293 : cluster [DBG] 5.b scrub starts 2024-02-23T00:44:00.666 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:59.378081+0000 osd.2 (osd.2) 294 : cluster [DBG] 5.b scrub ok 2024-02-23T00:44:00.667 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:00.686 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:44:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:00.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:44:00.791 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:44:00.819 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.977+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/2779380244 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7e880fdbb0 msgr2=0x7f7e880fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.977+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/2779380244 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7e880fdbb0 0x7f7e880fdf90 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 rx=0x7f7e7c004870 tx=0x7f7e7c009cb0).stop 2024-02-23T00:44:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.977+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/2779380244 shutdown_connections 2024-02-23T00:44:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.977+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/2779380244 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7e880f8420 0x7f7e88104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.977+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/2779380244 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7e880fdbb0 0x7f7e880fdf90 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.977+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/2779380244 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e880fe510 0x7f7e880f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.977+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/2779380244 >> 172.21.15.87:0/2779380244 conn(0x7f7e880f3800 msgr2=0x7f7e880f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/2779380244 shutdown_connections 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/2779380244 wait complete. 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 librados: starting msgr at 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 librados: starting objecter 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 Processor -- start 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 -- start start 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 librados: setting wanted keys 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 librados: calling monclient init 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e880f8420 0x7f7e8810e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7e880fdbb0 0x7f7e8810e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7e880fe510 0x7f7e88108240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88108730 con 0x7f7e880f8420 2024-02-23T00:44:01.002 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e881088a0 con 0x7f7e880fdbb0 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8ebe7700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e88108a60 con 0x7f7e880fe510 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8c983700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e880f8420 0x7f7e8810e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8d184700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7e880fe510 0x7f7e88108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.978+0000 7f7e8c983700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e880f8420 0x7f7e8810e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:41776/0 (socket says 172.21.15.87:41776) 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8c983700 1 -- 172.21.15.87:0/459726525 learned_addr learned my addr 172.21.15.87:0/459726525 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e87fff700 1 --2- 172.21.15.87:0/459726525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7e880fdbb0 0x7f7e8810e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8c983700 1 -- 172.21.15.87:0/459726525 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7e880fe510 msgr2=0x7f7e88108240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8c983700 1 --2- 172.21.15.87:0/459726525 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7e880fe510 0x7f7e88108240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8c983700 1 -- 172.21.15.87:0/459726525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7e880fdbb0 msgr2=0x7f7e8810e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8c983700 1 --2- 172.21.15.87:0/459726525 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7e880fdbb0 0x7f7e8810e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8c983700 1 -- 172.21.15.87:0/459726525 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74005040 con 0x7f7e880f8420 2024-02-23T00:44:01.003 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8d184700 1 --2- 172.21.15.87:0/459726525 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7e880fe510 0x7f7e88108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8c983700 1 --2- 172.21.15.87:0/459726525 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e880f8420 0x7f7e8810e0a0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f7e7c009eb0 tx=0x7f7e7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e85ffb700 10 client.?.objecter ms_handle_connect 0x7f7e880f8420 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e85ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c00e070 con 0x7f7e880f8420 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c005040 con 0x7f7e880f8420 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e7c013da0 con 0x7f7e880f8420 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c019070 con 0x7f7e880f8420 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8ebe7700 10 client.27924.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e88108fd0 con 0x7f7e880f8420 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.979+0000 7f7e8ebe7700 1 librados: init done 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.980+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7e880606d0 con 0x7f7e880f8420 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7e7c01e420 con 0x7f7e880f8420 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e85ffb700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7e78062c90 0x7f7e78065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.004 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(799..799 src has 400..799) v4 ==== 8725+0+0 (secure 0 0 0) 0x7f7e7c07df20 con 0x7f7e880f8420 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 osd_map(799..799 src has 400..799) v4 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e85ffb700 3 client.27924.objecter handle_osd_map got epochs [799,799] > 0 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e85ffb700 3 client.27924.objecter handle_osd_map decoding full epoch 799 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e87fff700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7e78062c90 0x7f7e78065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e85ffb700 20 client.27924.objecter dump_active .. 0 homeless 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v799) v1 ==== 54+0+16 (secure 0 0 0) 0x7f7e7c0805b0 con 0x7f7e880f8420 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 1 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e85ffb700 10 client.27924.objecter ms_handle_connect 0x7f7e78062c90 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e87fff700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7e78062c90 0x7f7e78065150 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 rx=0x7f7e881099e0 tx=0x7f7e7400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6801/484081831 conn(0x7f7e880fee60 0x7f7e880ff240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8805ee80 osd=0 3 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8805ee80 osd=0 3 2024-02-23T00:44:01.005 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8805ee80 osd=0 3 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 1 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e8805ee80 osd=0 3 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 0 1 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8805ee80 osd=0 4 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 1 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e881070a0 con 0x7f7e880fee60 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.981+0000 7f7e8d184700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6801/484081831 conn(0x7f7e880fee60 0x7f7e880ff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.982+0000 7f7e85ffb700 10 client.27924.objecter ms_handle_connect 0x7f7e880fee60 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.982+0000 7f7e8d184700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6801/484081831 conn(0x7f7e880fee60 0x7f7e880ff240 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.982+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7e881070a0 con 0x7f7e880fee60 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.982+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.982+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 1 = system:0 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.982+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e8805ee80 osd=0 4 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.982+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 0 1 2024-02-23T00:44:01.006 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 2 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8805ee80 osd=0 3 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8805ee80 osd=0 3 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8805ee80 osd=0 3 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 2 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e8805ee80 osd=0 3 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 0 2 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8805ee80 osd=0 4 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 2 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.997+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7e880273c0 con 0x7f7e880fee60 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.998+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7e880273c0 con 0x7f7e880fee60 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.998+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.998+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.998+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e8805ee80 osd=0 4 2024-02-23T00:44:01.007 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.998+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 0 2 2024-02-23T00:44:01.008 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:44:01.008 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 3 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6805/685354171 conn(0x7f7e880598f0 0x7f7e88059cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88028710 osd=1 3 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88028710 osd=1 3 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88028710 osd=1 3 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 3 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e88028710 osd=1 3 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 1 3 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88028710 osd=1 4 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 3 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e8811b650 con 0x7f7e880598f0 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8c983700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6805/685354171 conn(0x7f7e880598f0 0x7f7e88059cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.009 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e85ffb700 10 client.27924.objecter ms_handle_connect 0x7f7e880598f0 2024-02-23T00:44:01.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e8c983700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6805/685354171 conn(0x7f7e880598f0 0x7f7e88059cd0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7e8811b650 con 0x7f7e880598f0 2024-02-23T00:44:01.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:44:01.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 3 = system:0 2024-02-23T00:44:01.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e88028710 osd=1 4 2024-02-23T00:44:01.010 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:00.999+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 1 3 2024-02-23T00:44:01.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:44:01.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 4 2024-02-23T00:44:01.016 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88028710 osd=1 3 2024-02-23T00:44:01.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88028710 osd=1 3 2024-02-23T00:44:01.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88028710 osd=1 3 2024-02-23T00:44:01.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 4 2024-02-23T00:44:01.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e88028710 osd=1 3 2024-02-23T00:44:01.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 1 4 2024-02-23T00:44:01.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88028710 osd=1 4 2024-02-23T00:44:01.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 4 2024-02-23T00:44:01.017 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.014+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7e88059360 con 0x7f7e880598f0 2024-02-23T00:44:01.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7e88059360 con 0x7f7e880598f0 2024-02-23T00:44:01.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:44:01.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e88028710 osd=1 4 2024-02-23T00:44:01.018 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 1 4 2024-02-23T00:44:01.018 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:44:01.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:01.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 5 2024-02-23T00:44:01.019 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7e88028270 0x7f7e8811a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88059770 osd=2 3 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88059770 osd=2 3 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88059770 osd=2 3 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 5 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e88059770 osd=2 3 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 2 5 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88059770 osd=2 4 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 5 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e8811a560 con 0x7f7e88028270 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e87fff700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7e88028270 0x7f7e8811a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e85ffb700 10 client.27924.objecter ms_handle_connect 0x7f7e88028270 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.016+0000 7f7e87fff700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7e88028270 0x7f7e8811a070 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.020 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.017+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7e8811a560 con 0x7f7e88028270 2024-02-23T00:44:01.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.017+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:44:01.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.017+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 5 = system:0 2024-02-23T00:44:01.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.017+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e88059770 osd=2 4 2024-02-23T00:44:01.021 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.017+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 2 5 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 6 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88059770 osd=2 3 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88059770 osd=2 3 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88059770 osd=2 3 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 6 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e88059770 osd=2 3 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 2 6 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88059770 osd=2 4 2024-02-23T00:44:01.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 6 2024-02-23T00:44:01.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.031+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7e880277d0 con 0x7f7e88028270 2024-02-23T00:44:01.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7e880277d0 con 0x7f7e88028270 2024-02-23T00:44:01.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:44:01.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e88059770 osd=2 4 2024-02-23T00:44:01.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 2 6 2024-02-23T00:44:01.036 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:44:01.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:01.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 7 2024-02-23T00:44:01.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6813/128059790 conn(0x7f7e8811d7d0 0x7f7e8811fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8811af00 osd=3 3 2024-02-23T00:44:01.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811af00 osd=3 3 2024-02-23T00:44:01.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8811af00 osd=3 3 2024-02-23T00:44:01.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 7 2024-02-23T00:44:01.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e8811af00 osd=3 3 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 3 7 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811af00 osd=3 4 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 7 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e88122840 con 0x7f7e8811d7d0 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8d184700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6813/128059790 conn(0x7f7e8811d7d0 0x7f7e8811fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e85ffb700 10 client.27924.objecter ms_handle_connect 0x7f7e8811d7d0 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.033+0000 7f7e8d184700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6813/128059790 conn(0x7f7e8811d7d0 0x7f7e8811fbc0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.034+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7e88122840 con 0x7f7e8811d7d0 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.034+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.034+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 7 = system:0 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.034+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e8811af00 osd=3 4 2024-02-23T00:44:01.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.034+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 3 7 2024-02-23T00:44:01.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:44:01.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 8 2024-02-23T00:44:01.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8811af00 osd=3 3 2024-02-23T00:44:01.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811af00 osd=3 3 2024-02-23T00:44:01.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8811af00 osd=3 3 2024-02-23T00:44:01.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 8 2024-02-23T00:44:01.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e8811af00 osd=3 3 2024-02-23T00:44:01.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 3 8 2024-02-23T00:44:01.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811af00 osd=3 4 2024-02-23T00:44:01.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 8 2024-02-23T00:44:01.052 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.049+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7e8811a9d0 con 0x7f7e8811d7d0 2024-02-23T00:44:01.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7e8811a9d0 con 0x7f7e8811d7d0 2024-02-23T00:44:01.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:44:01.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e8811af00 osd=3 4 2024-02-23T00:44:01.053 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 3 8 2024-02-23T00:44:01.054 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 9 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7e88119bd0 0x7f7e88121230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8811a790 osd=4 3 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811a790 osd=4 3 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8811a790 osd=4 3 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 9 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e8811a790 osd=4 3 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 4 9 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811a790 osd=4 4 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 9 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.050+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e88121720 con 0x7f7e88119bd0 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.051+0000 7f7e8c983700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7e88119bd0 0x7f7e88121230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.055 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.051+0000 7f7e85ffb700 10 client.27924.objecter ms_handle_connect 0x7f7e88119bd0 2024-02-23T00:44:01.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.051+0000 7f7e8c983700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7e88119bd0 0x7f7e88121230 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.052+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7e88121720 con 0x7f7e88119bd0 2024-02-23T00:44:01.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.052+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:44:01.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.052+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 9 = system:0 2024-02-23T00:44:01.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.052+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e8811a790 osd=4 4 2024-02-23T00:44:01.056 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.052+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 4 9 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 10 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8811a790 osd=4 3 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811a790 osd=4 3 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e8811a790 osd=4 3 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 10 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e8811a790 osd=4 3 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 4 10 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811a790 osd=4 4 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 10 2024-02-23T00:44:01.069 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.067+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7e88121720 con 0x7f7e88119bd0 2024-02-23T00:44:01.070 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7e88121720 con 0x7f7e88119bd0 2024-02-23T00:44:01.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:44:01.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e8811a790 osd=4 4 2024-02-23T00:44:01.071 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 4 10 2024-02-23T00:44:01.071 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:44:01.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:01.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 11 2024-02-23T00:44:01.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7e88121e50 0x7f7e88122290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.072 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88121720 osd=5 3 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88121720 osd=5 3 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88121720 osd=5 3 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 11 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e88121720 osd=5 3 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 5 11 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88121720 osd=5 4 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 11 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.068+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e881291e0 con 0x7f7e88121e50 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.069+0000 7f7e87fff700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7e88121e50 0x7f7e88122290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.069+0000 7f7e85ffb700 10 client.27924.objecter ms_handle_connect 0x7f7e88121e50 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.069+0000 7f7e87fff700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7e88121e50 0x7f7e88122290 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.070+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7e881291e0 con 0x7f7e88121e50 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.070+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.070+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 11 = system:0 2024-02-23T00:44:01.073 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.070+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e88121720 osd=5 4 2024-02-23T00:44:01.074 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.070+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 5 11 2024-02-23T00:44:01.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:44:01.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 12 2024-02-23T00:44:01.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88121720 osd=5 3 2024-02-23T00:44:01.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88121720 osd=5 3 2024-02-23T00:44:01.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88121720 osd=5 3 2024-02-23T00:44:01.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 12 2024-02-23T00:44:01.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e88121720 osd=5 3 2024-02-23T00:44:01.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 5 12 2024-02-23T00:44:01.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88121720 osd=5 4 2024-02-23T00:44:01.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 12 2024-02-23T00:44:01.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.084+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7e88126a80 con 0x7f7e88121e50 2024-02-23T00:44:01.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7e88126a80 con 0x7f7e88121e50 2024-02-23T00:44:01.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:44:01.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e88121720 osd=5 4 2024-02-23T00:44:01.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 5 12 2024-02-23T00:44:01.089 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 13 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7e88127140 0x7f7e88127560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88126a80 osd=6 3 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88126a80 osd=6 3 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88126a80 osd=6 3 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 13 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e88126a80 osd=6 3 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 6 13 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88126a80 osd=6 4 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 13 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e881291e0 con 0x7f7e88127140 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.086+0000 7f7e8d184700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7e88127140 0x7f7e88127560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.087+0000 7f7e85ffb700 10 client.27924.objecter ms_handle_connect 0x7f7e88127140 2024-02-23T00:44:01.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.087+0000 7f7e8d184700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7e88127140 0x7f7e88127560 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.087+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7e881291e0 con 0x7f7e88127140 2024-02-23T00:44:01.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.087+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:44:01.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.087+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 13 = system:0 2024-02-23T00:44:01.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.087+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e88126a80 osd=6 4 2024-02-23T00:44:01.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.087+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 6 13 2024-02-23T00:44:01.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:44:01.104 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 14 2024-02-23T00:44:01.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88126a80 osd=6 3 2024-02-23T00:44:01.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88126a80 osd=6 3 2024-02-23T00:44:01.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e88126a80 osd=6 3 2024-02-23T00:44:01.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 14 2024-02-23T00:44:01.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e88126a80 osd=6 3 2024-02-23T00:44:01.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 6 14 2024-02-23T00:44:01.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88126a80 osd=6 4 2024-02-23T00:44:01.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 14 2024-02-23T00:44:01.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.102+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7e881291e0 con 0x7f7e88127140 2024-02-23T00:44:01.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7e881291e0 con 0x7f7e88127140 2024-02-23T00:44:01.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:44:01.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e88126a80 osd=6 4 2024-02-23T00:44:01.107 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 6 14 2024-02-23T00:44:01.107 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 15 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6812/373787000 conn(0x7f7e8812b9a0 0x7f7e8812be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e881291e0 osd=7 3 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e881291e0 osd=7 3 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e881291e0 osd=7 3 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 15 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e881291e0 osd=7 3 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 7 15 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e881291e0 osd=7 4 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 15 2024-02-23T00:44:01.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e8812c470 con 0x7f7e8812b9a0 2024-02-23T00:44:01.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.104+0000 7f7e8c983700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6812/373787000 conn(0x7f7e8812b9a0 0x7f7e8812be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:01.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.105+0000 7f7e85ffb700 10 client.27924.objecter ms_handle_connect 0x7f7e8812b9a0 2024-02-23T00:44:01.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.105+0000 7f7e8c983700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6812/373787000 conn(0x7f7e8812b9a0 0x7f7e8812be00 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:01.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.105+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f7e8812c470 con 0x7f7e8812b9a0 2024-02-23T00:44:01.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.105+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:44:01.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.105+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 15 = system:0 2024-02-23T00:44:01.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.105+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e881291e0 osd=7 4 2024-02-23T00:44:01.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.105+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 7 15 2024-02-23T00:44:01.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 10 client.27924.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-23T00:44:01.122 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign -1 16 2024-02-23T00:44:01.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e881291e0 osd=7 3 2024-02-23T00:44:01.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e881291e0 osd=7 3 2024-02-23T00:44:01.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 20 client.27924.objecter _get_session s=0x7f7e881291e0 osd=7 3 2024-02-23T00:44:01.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_remove -1 16 2024-02-23T00:44:01.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 20 client.27924.objecter get_session s=0x7f7e881291e0 osd=7 3 2024-02-23T00:44:01.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 15 client.27924.objecter _session_command_op_assign 7 16 2024-02-23T00:44:01.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e881291e0 osd=7 4 2024-02-23T00:44:01.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 10 client.27924.objecter _send_command 16 2024-02-23T00:44:01.123 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.120+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7e8812c470 con 0x7f7e8812b9a0 2024-02-23T00:44:01.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e85ffb700 1 -- 172.21.15.87:0/459726525 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (crc 0 0 0) 0x7f7e8812c470 con 0x7f7e8812b9a0 2024-02-23T00:44:01.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e85ffb700 10 client.27924.objecter ms_dispatch 0x7f7e88001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-23T00:44:01.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e85ffb700 10 client.27924.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e85ffb700 20 client.27924.objecter put_session s=0x7f7e881291e0 osd=7 4 2024-02-23T00:44:01.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e85ffb700 15 client.27924.objecter _session_command_op_remove 7 16 2024-02-23T00:44:01.125 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:44:01.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 librados: watch_flush enter 2024-02-23T00:44:01.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 librados: watch_flush exit 2024-02-23T00:44:01.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 client.27924.objecter close_session for osd.0 2024-02-23T00:44:01.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6801/484081831 conn(0x7f7e880fee60 msgr2=0x7f7e880ff240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6801/484081831 conn(0x7f7e880fee60 0x7f7e880ff240 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8805ee80 osd=0 1 2024-02-23T00:44:01.127 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 client.27924.objecter close_session for osd.1 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6805/685354171 conn(0x7f7e880598f0 msgr2=0x7f7e88059cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6805/685354171 conn(0x7f7e880598f0 0x7f7e88059cd0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88028710 osd=1 1 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 client.27924.objecter close_session for osd.2 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7e88028270 msgr2=0x7f7e8811a070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6809/3605394228 conn(0x7f7e88028270 0x7f7e8811a070 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88059770 osd=2 1 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 client.27924.objecter close_session for osd.3 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6813/128059790 conn(0x7f7e8811d7d0 msgr2=0x7f7e8811fbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8d184700 1 -- 172.21.15.87:0/459726525 reap_dead start 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6813/128059790 conn(0x7f7e8811d7d0 0x7f7e8811fbc0 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811af00 osd=3 1 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 client.27924.objecter close_session for osd.4 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7e88119bd0 msgr2=0x7f7e88121230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.128 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6800/4175844757 conn(0x7f7e88119bd0 0x7f7e88121230 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e8811a790 osd=4 1 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 client.27924.objecter close_session for osd.5 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7e88121e50 msgr2=0x7f7e88122290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6804/2550013795 conn(0x7f7e88121e50 0x7f7e88122290 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88121720 osd=5 1 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 client.27924.objecter close_session for osd.6 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7e88127140 msgr2=0x7f7e88127560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6808/2509408498 conn(0x7f7e88127140 0x7f7e88127560 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e88126a80 osd=6 1 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 client.27924.objecter close_session for osd.7 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6812/373787000 conn(0x7f7e8812b9a0 msgr2=0x7f7e8812be00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.171:6812/373787000 conn(0x7f7e8812b9a0 0x7f7e8812be00 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 20 client.27924.objecter put_session s=0x7f7e881291e0 osd=7 1 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 20 client.27924.objecter shutdown clearing up homeless session... 2024-02-23T00:44:01.129 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 10 client.27924.objecter successfully canceled tick 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8d184700 1 -- 172.21.15.87:0/459726525 reap_dead start 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7e78062c90 msgr2=0x7f7e78065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7e78062c90 0x7f7e78065150 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 rx=0x7f7e881099e0 tx=0x7f7e7400b040).stop 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e880f8420 msgr2=0x7f7e8810e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.122+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e880f8420 0x7f7e8810e0a0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 rx=0x7f7e7c009eb0 tx=0x7f7e7c00b040).stop 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.123+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 shutdown_connections 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.123+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7e880f8420 0x7f7e8810e0a0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.123+0000 7f7e8ebe7700 1 --2- 172.21.15.87:0/459726525 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7e78062c90 0x7f7e78065150 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.123+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 >> 172.21.15.87:0/459726525 conn(0x7f7e880f3800 msgr2=0x7f7e880f5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.123+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 shutdown_connections 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.123+0000 7f7e8ebe7700 1 -- 172.21.15.87:0/459726525 wait complete. 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:01.123+0000 7f7e8ebe7700 1 librados: shutdown 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.130 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.131 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.131 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.131 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.131 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-23T00:44:01.218 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:44:01.218 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [], 2024-02-23T00:44:01.218 INFO:teuthology.orchestra.run.smithi087.stdout: "complaint_time": 30, 2024-02-23T00:44:01.219 INFO:teuthology.orchestra.run.smithi087.stdout: "num_blocked_ops": 0 2024-02-23T00:44:01.219 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:44:01.615 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:59.808267+0000 mgr.y (mgr.24437) 676 : cluster [DBG] pgmap v1225: 204 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 peering, 170 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:59.932949+0000 osd.3 (osd.3) 257 : cluster [DBG] 234.4 scrub starts 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:43:59.936419+0000 osd.3 (osd.3) 258 : cluster [DBG] 234.4 scrub ok 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:00.356226+0000 mon.a (mon.0) 3911 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.356433+0000 mon.a (mon.0) 3912 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:00.398806+0000 mon.a (mon.0) 3913 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:00.399202+0000 mon.a (mon.0) 3914 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-140"}]: dispatch 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.418407+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.421873+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.523378+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.526900+0000 osd.1 (osd.1) 363 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:44:01.617 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.939980+0000 osd.5 (osd.5) 315 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:44:01.618 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.940723+0000 osd.5 (osd.5) 316 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:44:01.618 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:00.982618+0000 mon.a (mon.0) 3915 : audit [DBG] from='client.? 172.21.15.87:0/459726525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:44:01.618 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:01.135097+0000 osd.4 (osd.4) 349 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:44:01.618 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:01.138393+0000 osd.4 (osd.4) 350 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:44:01.640 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:01.716 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-02-23T00:44:01.716 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-23T00:44:01.716 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.control 2024-02-23T00:44:01.717 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:44:02.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:59.808267+0000 mgr.y (mgr.24437) 676 : cluster [DBG] pgmap v1225: 204 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 peering, 170 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:02.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:59.932949+0000 osd.3 (osd.3) 257 : cluster [DBG] 234.4 scrub starts 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:43:59.936419+0000 osd.3 (osd.3) 258 : cluster [DBG] 234.4 scrub ok 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:00.356226+0000 mon.a (mon.0) 3911 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.356433+0000 mon.a (mon.0) 3912 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:00.398806+0000 mon.a (mon.0) 3913 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:00.399202+0000 mon.a (mon.0) 3914 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-140"}]: dispatch 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.418407+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.421873+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.523378+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.526900+0000 osd.1 (osd.1) 363 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.939980+0000 osd.5 (osd.5) 315 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.940723+0000 osd.5 (osd.5) 316 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:00.982618+0000 mon.a (mon.0) 3915 : audit [DBG] from='client.? 172.21.15.87:0/459726525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:01.135097+0000 osd.4 (osd.4) 349 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:44:02.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:01.138393+0000 osd.4 (osd.4) 350 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:59.808267+0000 mgr.y (mgr.24437) 676 : cluster [DBG] pgmap v1225: 204 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 1 peering, 170 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:59.932949+0000 osd.3 (osd.3) 257 : cluster [DBG] 234.4 scrub starts 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:43:59.936419+0000 osd.3 (osd.3) 258 : cluster [DBG] 234.4 scrub ok 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:00.356226+0000 mon.a (mon.0) 3911 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.356433+0000 mon.a (mon.0) 3912 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:00.398806+0000 mon.a (mon.0) 3913 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:00.399202+0000 mon.a (mon.0) 3914 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-140"}]: dispatch 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.418407+0000 osd.0 (osd.0) 61 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.421873+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:44:02.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.523378+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:44:02.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.526900+0000 osd.1 (osd.1) 363 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:44:02.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.939980+0000 osd.5 (osd.5) 315 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:44:02.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.940723+0000 osd.5 (osd.5) 316 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:44:02.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:00.982618+0000 mon.a (mon.0) 3915 : audit [DBG] from='client.? 172.21.15.87:0/459726525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:44:02.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:01.135097+0000 osd.4 (osd.4) 349 : cluster [DBG] 22.1e scrub starts 2024-02-23T00:44:02.086 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:01.138393+0000 osd.4 (osd.4) 350 : cluster [DBG] 22.1e scrub ok 2024-02-23T00:44:02.271 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:44:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:02.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:44:02.378 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:44:02.402 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:02] "GET /metrics HTTP/1.1" 200 190948 "" "Prometheus/2.33.4" 2024-02-23T00:44:02.405 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:02.724 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.798527+0000 osd.6 (osd.6) 336 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.802059+0000 osd.6 (osd.6) 337 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.891265+0000 osd.3 (osd.3) 259 : cluster [DBG] 4.c scrub starts 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:00.894772+0000 osd.3 (osd.3) 260 : cluster [DBG] 4.c scrub ok 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:01.325991+0000 mgr.y (mgr.24437) 677 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:01.357709+0000 mon.a (mon.0) 3916 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:01.808756+0000 mgr.y (mgr.24437) 678 : cluster [DBG] pgmap v1228: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:02.257017+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:02.259287+0000 mon.a (mon.0) 3918 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.798527+0000 osd.6 (osd.6) 336 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.802059+0000 osd.6 (osd.6) 337 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:44:02.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.891265+0000 osd.3 (osd.3) 259 : cluster [DBG] 4.c scrub starts 2024-02-23T00:44:02.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:00.894772+0000 osd.3 (osd.3) 260 : cluster [DBG] 4.c scrub ok 2024-02-23T00:44:02.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:01.325991+0000 mgr.y (mgr.24437) 677 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:02.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:01.357709+0000 mon.a (mon.0) 3916 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-02-23T00:44:02.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:01.808756+0000 mgr.y (mgr.24437) 678 : cluster [DBG] pgmap v1228: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:02.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:02.257017+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-02-23T00:44:02.726 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:02.259287+0000 mon.a (mon.0) 3918 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:44:02.744 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-23T00:44:02.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.798527+0000 osd.6 (osd.6) 336 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:44:02.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.802059+0000 osd.6 (osd.6) 337 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:44:02.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.891265+0000 osd.3 (osd.3) 259 : cluster [DBG] 4.c scrub starts 2024-02-23T00:44:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:00.894772+0000 osd.3 (osd.3) 260 : cluster [DBG] 4.c scrub ok 2024-02-23T00:44:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:01.325991+0000 mgr.y (mgr.24437) 677 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:01.357709+0000 mon.a (mon.0) 3916 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-02-23T00:44:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:01.808756+0000 mgr.y (mgr.24437) 678 : cluster [DBG] pgmap v1228: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:02.257017+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-02-23T00:44:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:02.259287+0000 mon.a (mon.0) 3918 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:44:02.955 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:44:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:02.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:44:03.059 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:44:03.092 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:03.113 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:44:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:03.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:44:03.217 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:44:03.247 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:03.277 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:44:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:03.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:44:03.382 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:44:03.410 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:03.533 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:44:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:03.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:44:03.567 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:03.569 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:03.638 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:44:03.664 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:03.683 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:44:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:03.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:44:03.787 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:44:03.813 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:03.832 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:44:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:02.469410+0000 osd.0 (osd.0) 63 : cluster [DBG] 4.3 deep-scrub starts 2024-02-23T00:44:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:02.476348+0000 osd.0 (osd.0) 64 : cluster [DBG] 4.3 deep-scrub ok 2024-02-23T00:44:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.135804+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.9 deep-scrub starts 2024-02-23T00:44:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.136675+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.9 deep-scrub ok 2024-02-23T00:44:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:03.257677+0000 mon.a (mon.0) 3919 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:44:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.257834+0000 mon.a (mon.0) 3920 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-02-23T00:44:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:03.305226+0000 mon.a (mon.0) 3921 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:03.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:03.305516+0000 mon.a (mon.0) 3922 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-142"}]: dispatch 2024-02-23T00:44:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:02.469410+0000 osd.0 (osd.0) 63 : cluster [DBG] 4.3 deep-scrub starts 2024-02-23T00:44:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:02.476348+0000 osd.0 (osd.0) 64 : cluster [DBG] 4.3 deep-scrub ok 2024-02-23T00:44:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.135804+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.9 deep-scrub starts 2024-02-23T00:44:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.136675+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.9 deep-scrub ok 2024-02-23T00:44:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:03.257677+0000 mon.a (mon.0) 3919 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:44:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.257834+0000 mon.a (mon.0) 3920 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-02-23T00:44:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:03.305226+0000 mon.a (mon.0) 3921 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:03.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:03.305516+0000 mon.a (mon.0) 3922 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-142"}]: dispatch 2024-02-23T00:44:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:03.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:44:03.936 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:44:03.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:44:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:44:03.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:02.469410+0000 osd.0 (osd.0) 63 : cluster [DBG] 4.3 deep-scrub starts 2024-02-23T00:44:03.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:02.476348+0000 osd.0 (osd.0) 64 : cluster [DBG] 4.3 deep-scrub ok 2024-02-23T00:44:03.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.135804+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.9 deep-scrub starts 2024-02-23T00:44:03.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.136675+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.9 deep-scrub ok 2024-02-23T00:44:03.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:03.257677+0000 mon.a (mon.0) 3919 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:44:03.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.257834+0000 mon.a (mon.0) 3920 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-02-23T00:44:03.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:03.305226+0000 mon.a (mon.0) 3921 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:03.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:03.305516+0000 mon.a (mon.0) 3922 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-142"}]: dispatch 2024-02-23T00:44:03.965 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:03.986 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:44:03.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:03.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:44:04.091 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:44:04.118 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:04.136 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:44:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:04.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:44:04.141 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-23T00:44:04.240 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:44:04.268 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:04.286 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:44:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:04.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:44:04.390 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:44:04.416 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:04.434 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:44:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:04.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:44:04.537 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:44:04.565 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:04.583 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:44:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:04.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:44:04.687 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:44:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:02.819739+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:44:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:02.822616+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:44:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.532041+0000 osd.1 (osd.1) 364 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:44:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.533452+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:44:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.809197+0000 mgr.y (mgr.24437) 679 : cluster [DBG] pgmap v1231: 204 pgs: 4 creating+peering, 28 unknown, 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.966797+0000 osd.5 (osd.5) 317 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:44:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.968679+0000 osd.5 (osd.5) 318 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:44:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:04.085284+0000 osd.7 (osd.7) 372 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:44:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:04.086016+0000 osd.7 (osd.7) 373 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:44:04.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:04.262916+0000 mon.a (mon.0) 3923 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-02-23T00:44:04.713 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:04.731 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:44:04.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:04.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:44:04.835 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:44:04.861 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:04.879 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:44:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:04.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:44:04.983 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:44:05.009 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:05.028 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:44:05.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:05.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:44:05.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:02.819739+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:44:05.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:02.822616+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.532041+0000 osd.1 (osd.1) 364 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.533452+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.809197+0000 mgr.y (mgr.24437) 679 : cluster [DBG] pgmap v1231: 204 pgs: 4 creating+peering, 28 unknown, 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.966797+0000 osd.5 (osd.5) 317 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.968679+0000 osd.5 (osd.5) 318 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:04.085284+0000 osd.7 (osd.7) 372 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:04.086016+0000 osd.7 (osd.7) 373 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:04.262916+0000 mon.a (mon.0) 3923 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:02.819739+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:44:05.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:02.822616+0000 osd.6 (osd.6) 339 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:44:05.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.532041+0000 osd.1 (osd.1) 364 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:44:05.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.533452+0000 osd.1 (osd.1) 365 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:44:05.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.809197+0000 mgr.y (mgr.24437) 679 : cluster [DBG] pgmap v1231: 204 pgs: 4 creating+peering, 28 unknown, 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:05.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.966797+0000 osd.5 (osd.5) 317 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:44:05.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.968679+0000 osd.5 (osd.5) 318 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:44:05.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:04.085284+0000 osd.7 (osd.7) 372 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:44:05.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:04.086016+0000 osd.7 (osd.7) 373 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:44:05.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:04.262916+0000 mon.a (mon.0) 3923 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-02-23T00:44:05.131 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:44:05.158 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:05.177 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:44:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:05.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:44:05.280 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:44:05.310 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:05.381 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:44:05.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:05.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:44:05.486 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:44:05.513 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:05.551 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:44:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:05.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:44:05.656 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:44:05.683 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:05.700 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:44:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:05.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:44:05.804 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0 2024-02-23T00:44:05.829 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.837982+0000 osd.6 (osd.6) 340 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:44:05.829 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:03.840875+0000 osd.6 (osd.6) 341 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:44:05.829 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:04.540997+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:44:05.829 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:04.547229+0000 osd.1 (osd.1) 367 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:44:05.829 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:05.049776+0000 osd.7 (osd.7) 374 : cluster [DBG] 22.c scrub starts 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:05.053572+0000 osd.7 (osd.7) 375 : cluster [DBG] 22.c scrub ok 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:05.261359+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:05.262767+0000 mon.a (mon.0) 3925 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.837982+0000 osd.6 (osd.6) 340 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:03.840875+0000 osd.6 (osd.6) 341 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:04.540997+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:04.547229+0000 osd.1 (osd.1) 367 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:05.049776+0000 osd.7 (osd.7) 374 : cluster [DBG] 22.c scrub starts 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:05.053572+0000 osd.7 (osd.7) 375 : cluster [DBG] 22.c scrub ok 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:05.261359+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-02-23T00:44:05.830 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:05.262767+0000 mon.a (mon.0) 3925 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:44:05.831 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.837982+0000 osd.6 (osd.6) 340 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:44:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:03.840875+0000 osd.6 (osd.6) 341 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:44:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:04.540997+0000 osd.1 (osd.1) 366 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:44:05.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:04.547229+0000 osd.1 (osd.1) 367 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:44:05.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:05.049776+0000 osd.7 (osd.7) 374 : cluster [DBG] 22.c scrub starts 2024-02-23T00:44:05.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:05.053572+0000 osd.7 (osd.7) 375 : cluster [DBG] 22.c scrub ok 2024-02-23T00:44:05.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:05.261359+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-02-23T00:44:05.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:05.262767+0000 mon.a (mon.0) 3925 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-23T00:44:05.954 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:44:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:05.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0' 2024-02-23T00:44:06.058 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1 2024-02-23T00:44:06.086 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:06.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.171+0000 7f0fdfe81700 1 -- 172.21.15.87:0/3024802370 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fd80fcb40 msgr2=0x7f0fd80fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:06.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.171+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/3024802370 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fd80fcb40 0x7f0fd80fcf20 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 rx=0x7f0fc8004870 tx=0x7f0fc8009cb0).stop 2024-02-23T00:44:06.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.171+0000 7f0fdfe81700 1 -- 172.21.15.87:0/3024802370 shutdown_connections 2024-02-23T00:44:06.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.171+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/3024802370 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fd80f7ba0 0x7f0fd8103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:06.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.171+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/3024802370 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fd80fcb40 0x7f0fd80fcf20 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:06.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.171+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/3024802370 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fd80fd4a0 0x7f0fd80f74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:06.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.171+0000 7f0fdfe81700 1 -- 172.21.15.87:0/3024802370 >> 172.21.15.87:0/3024802370 conn(0x7f0fd806c370 msgr2=0x7f0fd806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:06.175 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.171+0000 7f0fdfe81700 1 -- 172.21.15.87:0/3024802370 shutdown_connections 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.171+0000 7f0fdfe81700 1 -- 172.21.15.87:0/3024802370 wait complete. 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 librados: starting msgr at 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 librados: starting objecter 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 Processor -- start 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 -- start start 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 librados: setting wanted keys 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 librados: calling monclient init 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fd80f7ba0 0x7f0fd810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fd80fcb40 0x7f0fd810d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fd80fd4a0 0x7f0fd8107190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd8107680 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd81077f0 con 0x7f0fd80f7ba0 2024-02-23T00:44:06.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdfe81700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd81079b0 con 0x7f0fd80fcb40 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdd41c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fd80fcb40 0x7f0fd810d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fde41e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fd80fd4a0 0x7f0fd8107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdd41c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fd80fcb40 0x7f0fd810d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:42596/0 (socket says 172.21.15.87:42596) 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fdd41c700 1 -- 172.21.15.87:0/200433555 learned_addr learned my addr 172.21.15.87:0/200433555 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fddc1d700 1 --2- 172.21.15.87:0/200433555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fd80f7ba0 0x7f0fd810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fde41e700 1 -- 172.21.15.87:0/200433555 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fd80fcb40 msgr2=0x7f0fd810d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fde41e700 1 --2- 172.21.15.87:0/200433555 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fd80fcb40 0x7f0fd810d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fde41e700 1 -- 172.21.15.87:0/200433555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fd80f7ba0 msgr2=0x7f0fd810cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fde41e700 1 --2- 172.21.15.87:0/200433555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fd80f7ba0 0x7f0fd810cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.172+0000 7f0fde41e700 1 -- 172.21.15.87:0/200433555 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd00097c0 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fde41e700 1 --2- 172.21.15.87:0/200433555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fd80fd4a0 0x7f0fd8107190 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f0fd0001cb0 tx=0x7f0fd0004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fceffd700 10 client.?.objecter ms_handle_connect 0x7f0fd80fd4a0 2024-02-23T00:44:06.177 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fceffd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:06.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fceffd700 1 -- 172.21.15.87:0/200433555 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd00079b0 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc8005040 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fceffd700 1 -- 172.21.15.87:0/200433555 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fd00090d0 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fdfe81700 10 client.27930.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:06.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fceffd700 1 -- 172.21.15.87:0/200433555 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd000fc20 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd8107ff0 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.178 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.173+0000 7f0fdfe81700 1 librados: init done 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.174+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fbc004f50 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.174+0000 7f0fceffd700 1 -- 172.21.15.87:0/200433555 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0fd000fd80 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.175+0000 7f0fceffd700 1 --2- 172.21.15.87:0/200433555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0fc4062c90 0x7f0fc4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.175+0000 7f0fceffd700 1 -- 172.21.15.87:0/200433555 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(804..804 src has 400..804) v4 ==== 8712+0+0 (secure 0 0 0) 0x7f0fd007db70 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.175+0000 7f0fceffd700 10 client.27930.objecter ms_dispatch 0x7f0fd8001550 osd_map(804..804 src has 400..804) v4 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.175+0000 7f0fceffd700 3 client.27930.objecter handle_osd_map got epochs [804,804] > 0 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.175+0000 7f0fceffd700 3 client.27930.objecter handle_osd_map decoding full epoch 804 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.175+0000 7f0fddc1d700 1 --2- 172.21.15.87:0/200433555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0fc4062c90 0x7f0fc4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.175+0000 7f0fceffd700 20 client.27930.objecter dump_active .. 0 homeless 2024-02-23T00:44:06.180 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.175+0000 7f0fceffd700 10 client.27930.objecter ms_handle_connect 0x7f0fc4062c90 2024-02-23T00:44:06.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.175+0000 7f0fddc1d700 1 --2- 172.21.15.87:0/200433555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0fc4062c90 0x7f0fc4065150 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 rx=0x7f0fc80043d0 tx=0x7f0fc800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:06.181 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.179+0000 7f0fceffd700 1 -- 172.21.15.87:0/200433555 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0fd0050d50 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.189 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:44:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:06.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1' 2024-02-23T00:44:06.293 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:44:06.322 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:06.358 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:44:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:06.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:44:06.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:06.378+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f0fbc005780 con 0x7f0fd80fd4a0 2024-02-23T00:44:06.474 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:44:06.503 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:06.522 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:44:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:06.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:44:06.625 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:44:06.653 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:06.671 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:44:06.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:06.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:44:06.775 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:44:06.800 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:05.810010+0000 mgr.y (mgr.24437) 680 : cluster [DBG] pgmap v1234: 204 pgs: 1 active+remapped+backfill_toofull, 29 creating+peering, 3 unknown, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:06.800 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:06.097013+0000 osd.7 (osd.7) 376 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:44:06.800 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:06.099890+0000 osd.7 (osd.7) 377 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:44:06.800 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:06.263080+0000 mon.a (mon.0) 3926 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:44:06.800 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:06.263246+0000 mon.a (mon.0) 3927 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-02-23T00:44:06.800 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:06.266390+0000 mon.a (mon.0) 3928 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-101","var": "dedup_tier","val": "test-rados-api-smithi087-82091-144"}]: dispatch 2024-02-23T00:44:06.800 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:06.292981+0000 mon.a (mon.0) 3929 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:06.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:06.293335+0000 mon.a (mon.0) 3930 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-144"}]: dispatch 2024-02-23T00:44:06.801 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:06.379209+0000 mon.a (mon.0) 3931 : audit [INF] from='client.? 172.21.15.87:0/200433555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:44:06.802 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:06.820 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:44:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:06.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:44:06.868 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:06.924 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:44:06.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:05.810010+0000 mgr.y (mgr.24437) 680 : cluster [DBG] pgmap v1234: 204 pgs: 1 active+remapped+backfill_toofull, 29 creating+peering, 3 unknown, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:06.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:06.097013+0000 osd.7 (osd.7) 376 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:44:06.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:06.099890+0000 osd.7 (osd.7) 377 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:44:06.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:06.263080+0000 mon.a (mon.0) 3926 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:44:06.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:06.263246+0000 mon.a (mon.0) 3927 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-02-23T00:44:06.950 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:06.266390+0000 mon.a (mon.0) 3928 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-101","var": "dedup_tier","val": "test-rados-api-smithi087-82091-144"}]: dispatch 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:06.292981+0000 mon.a (mon.0) 3929 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:06.293335+0000 mon.a (mon.0) 3930 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-144"}]: dispatch 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:06.379209+0000 mon.a (mon.0) 3931 : audit [INF] from='client.? 172.21.15.87:0/200433555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:05.810010+0000 mgr.y (mgr.24437) 680 : cluster [DBG] pgmap v1234: 204 pgs: 1 active+remapped+backfill_toofull, 29 creating+peering, 3 unknown, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:06.097013+0000 osd.7 (osd.7) 376 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:06.099890+0000 osd.7 (osd.7) 377 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:06.263080+0000 mon.a (mon.0) 3926 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi087-82091-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:06.263246+0000 mon.a (mon.0) 3927 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:06.266390+0000 mon.a (mon.0) 3928 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi087-82091-101","var": "dedup_tier","val": "test-rados-api-smithi087-82091-144"}]: dispatch 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:06.292981+0000 mon.a (mon.0) 3929 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:06.951 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:06.293335+0000 mon.a (mon.0) 3930 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi087-82091-101", "tierpool": "test-rados-api-smithi087-82091-144"}]: dispatch 2024-02-23T00:44:06.952 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:06.379209+0000 mon.a (mon.0) 3931 : audit [INF] from='client.? 172.21.15.87:0/200433555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-23T00:44:06.952 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:07.101 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:44:07.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:07.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:44:07.205 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:44:07.232 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:07.251 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:44:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:07.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:44:07.268 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.266+0000 7f0fceffd700 1 -- 172.21.15.87:0/200433555 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v806) v1 ==== 87+0+0 (secure 0 0 0) 0x7f0fd0050940 con 0x7f0fd80fd4a0 2024-02-23T00:44:07.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.269+0000 7f0fdfe81700 10 librados: watch_flush enter 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 10 librados: watch_flush exit 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 20 client.27930.objecter shutdown clearing up homeless session... 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 10 client.27930.objecter successfully canceled tick 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0fc4062c90 msgr2=0x7f0fc4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/200433555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0fc4062c90 0x7f0fc4065150 secure :-1 s=READY pgs=2984 cs=0 l=1 rev1=1 rx=0x7f0fc80043d0 tx=0x7f0fc800b040).stop 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fd80fd4a0 msgr2=0x7f0fd8107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/200433555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fd80fd4a0 0x7f0fd8107190 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 rx=0x7f0fd0001cb0 tx=0x7f0fd0004c50).stop 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 shutdown_connections 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/200433555 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0fc4062c90 0x7f0fc4065150 secure :-1 s=CLOSED pgs=2984 cs=0 l=1 rev1=1 rx=0x7f0fc80043d0 tx=0x7f0fc800b040).stop 2024-02-23T00:44:07.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/200433555 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0fd80fd4a0 0x7f0fd8107190 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/200433555 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0fd80f7ba0 0x7f0fd810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 --2- 172.21.15.87:0/200433555 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0fd80fcb40 0x7f0fd810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 >> 172.21.15.87:0/200433555 conn(0x7f0fd806c370 msgr2=0x7f0fd80f5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 shutdown_connections 2024-02-23T00:44:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.270+0000 7f0fdfe81700 1 -- 172.21.15.87:0/200433555 wait complete. 2024-02-23T00:44:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:07.271+0000 7f0fdfe81700 1 librados: shutdown 2024-02-23T00:44:07.273 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is set 2024-02-23T00:44:07.355 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6 2024-02-23T00:44:07.382 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:07.400 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:44:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:07.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6' 2024-02-23T00:44:07.503 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4 2024-02-23T00:44:07.509 INFO:teuthology.orchestra.run.smithi087.stdout:{ 2024-02-23T00:44:07.509 INFO:teuthology.orchestra.run.smithi087.stdout: "size": 20, 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 600, 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "ops": [ 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:1877 5.20 5:05bf5b68:::notify.1:head [watch reconnect cookie 94532396646400 gen 4] snapc 0=[] ondisk+write+known_if_redirected e561)", 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:39:40.477301+0000", 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 267.02958885700002, 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.73050895699999996, 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 1877 2024-02-23T00:44:07.510 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477301+0000", 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477301+0000", 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477300+0000", 2024-02-23T00:44:07.511 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9739999999999999e-06 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477306+0000", 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1486e-05 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477317+0000", 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5060000000000001e-06 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477324+0000", 2024-02-23T00:44:07.512 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2507000000000003e-05 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477386+0000", 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.4099999999999997e-06 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:40.477396+0000", 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.70815977299999999 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.185555+0000", 2024-02-23T00:44:07.513 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013567700000000001 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for missing object", 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.185691+0000", 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003571552 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.189263+0000", 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000101983 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for rw locks", 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.189365+0000", 2024-02-23T00:44:07.514 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0061286350000000003 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.195493+0000", 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.5148000000000005e-05 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.195578+0000", 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.012231357999999999 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:39:41.207810+0000", 2024-02-23T00:44:07.515 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.73050895699999996 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:41 235.0 235:00000000::::head [pgnls start_epoch 603] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e603)", 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:40:22.179696+0000", 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 225.32719340400001, 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0204403200000001, 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:44:07.516 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 41 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179696+0000", 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179696+0000", 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.517 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179695+0000", 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9789999999999998e-06 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179701+0000", 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.9279999999999999e-06 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179710+0000", 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.6839999999999996e-06 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.518 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.179717+0000", 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0094283470000000001 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.189146+0000", 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.8720000000000008e-06 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:22.189155+0000", 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.010297435 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.519 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:23.199452+0000", 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.7379000000000002e-05 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:23.199489+0000", 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00064718399999999998 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:40:23.200137+0000", 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0204403200000001 2024-02-23T00:44:07.520 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2031 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94532396646400 gen 5] snapc 0=[] ondisk+write+known_if_redirected e641)", 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:00.492645+0000", 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 187.014244851, 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72160946599999998, 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2031 2024-02-23T00:44:07.521 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492645+0000", 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492645+0000", 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.522 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492644+0000", 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.9739999999999999e-06 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492650+0000", 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.3244000000000001e-05 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492663+0000", 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7290000000000003e-06 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.523 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492670+0000", 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.3585999999999997e-05 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492733+0000", 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0555e-05 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492744+0000", 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.720318397 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.524 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.213062+0000", 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00016163399999999999 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.213224+0000", 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0010303999999999999 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.214254+0000", 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72160946599999998 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.525 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.14397.0:2032 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94532396646400 gen 6] snapc 0=[] ondisk+write+known_if_redirected e641)", 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:00.492681+0000", 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 187.01420884500001, 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72175932899999995, 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.14397", 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3823246443", 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2032 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.526 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492681+0000", 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492681+0000", 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492680+0000", 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.988e-06 2024-02-23T00:44:07.527 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492683+0000", 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5169999999999996e-06 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492690+0000", 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.51e-06 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492694+0000", 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4622000000000005e-05 2024-02-23T00:44:07.528 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492769+0000", 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.4109999999999996e-06 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "waiting for peered", 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:00.492777+0000", 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72074307900000001 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.213520+0000", 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4667999999999997e-05 2024-02-23T00:44:07.529 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.213595+0000", 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00084507899999999999 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:01.214440+0000", 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.72175932899999995 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.530 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:05588ce8:lc::lc.6:head)])", 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:37.648001+0000", 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 149.85888900800001, 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040277480000000003, 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648001+0000", 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.531 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648001+0000", 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648000+0000", 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3129999999999999e-06 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648008+0000", 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.3129999999999999e-06 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.532 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648017+0000", 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.296e-06 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.648021+0000", 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038703909999999999 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.651891+0000", 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00013709599999999999 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.533 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.652028+0000", 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040277480000000003 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:02683654:reshard::reshard.0000000010:head)])", 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:37.738432+0000", 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 149.76845769299999, 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039951580000000004, 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.534 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738432+0000", 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738432+0000", 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738431+0000", 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5740000000000001e-06 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.535 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738438+0000", 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1375999999999999e-05 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738449+0000", 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.4710000000000001e-06 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.738454+0000", 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038411320000000001 2024-02-23T00:44:07.536 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.742295+0000", 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000132378 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.742427+0000", 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039951580000000004 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.537 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:0705c68b:::obj_delete_at_hint.0000000028:head)])", 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:37.919259+0000", 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 149.58763039300001, 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003982579, 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919259+0000", 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919259+0000", 2024-02-23T00:44:07.538 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919258+0000", 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.7479999999999996e-06 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919265+0000", 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2201999999999999e-05 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919277+0000", 2024-02-23T00:44:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4779999999999996e-06 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.919284+0000", 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038323099999999998 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.923116+0000", 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000125709 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.540 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:37.923242+0000", 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003982579 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:0216b5b6:::obj_delete_at_hint.0000000070:head)])", 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.009599+0000", 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 149.49729085800001, 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040234959999999997, 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.541 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009599+0000", 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009599+0000", 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.999999 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009598+0000", 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.6440000000000003e-06 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.542 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009605+0000", 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2456000000000001e-05 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009617+0000", 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5989999999999998e-06 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.009624+0000", 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038776380000000001 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.543 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.013501+0000", 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012103 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.013622+0000", 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0040234959999999997 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.0 678/677 [PushReplyOp(4:0255b7ba:::obj_delete_at_hint.0000000104:head)])", 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.190491+0000", 2024-02-23T00:44:07.544 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 149.316398357, 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039407130000000002, 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190491+0000", 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190491+0000", 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.545 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190491+0000", 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.698e-06 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190495+0000", 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.2920000000000001e-06 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190501+0000", 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.908e-06 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.190507+0000", 2024-02-23T00:44:07.546 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037958549999999999 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.194303+0000", 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012923700000000001 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.194432+0000", 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039407130000000002 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.547 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "pg_scan(digest 4.1a MIN-MAX e 679/678)", 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.216534+0000", 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 149.290356156, 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039114390000000001, 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216534+0000", 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.548 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216534+0000", 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216533+0000", 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8890000000000004e-06 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216537+0000", 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.6380000000000001e-06 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.549 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216543+0000", 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8800000000000002e-06 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.216550+0000", 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038478010000000001 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.220397+0000", 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.5129e-05 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.550 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.220413+0000", 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.2404999999999999e-05 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.220445+0000", 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039114390000000001 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.551 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.1a 679/678 [PushReplyOp(4:5909e671:gc::gc.27:head)])", 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.302205+0000", 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 149.204684786, 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039522969999999996, 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302205+0000", 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302205+0000", 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302205+0000", 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8999999999999999e-06 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302208+0000", 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0290000000000002e-06 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302215+0000", 2024-02-23T00:44:07.553 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.1099999999999999e-06 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.302221+0000", 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0036789309999999999 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.305900+0000", 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00025763700000000001 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.306157+0000", 2024-02-23T00:44:07.554 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039522969999999996 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.19 679/678 [PushReplyOp(4:994d36d1:lc::lc.9:head)])", 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.393809+0000", 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 149.11308086299999, 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039211339999999997, 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.555 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393809+0000", 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393809+0000", 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393808+0000", 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.9199999999999997e-06 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.556 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393812+0000", 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4679999999999997e-06 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393819+0000", 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.3820000000000003e-06 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.393824+0000", 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.003639775 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.557 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.397464+0000", 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00026597399999999998 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.397730+0000", 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039211339999999997 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.1a 679/678 [PushReplyOp(4:5f2e3d26:lc::lc.18:head)])", 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:38.763294+0000", 2024-02-23T00:44:07.558 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 148.74359588199999, 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038946779999999999, 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763294+0000", 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763294+0000", 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.559 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763293+0000", 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.1159999999999999e-06 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763297+0000", 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4370000000000004e-06 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763303+0000", 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.4269999999999999e-06 2024-02-23T00:44:07.560 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.763308+0000", 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037399030000000002 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.767048+0000", 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00014011199999999999 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:38.767188+0000", 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0038946779999999999 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.561 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(4.19 679/678 [PushReplyOp(4:9fecf831:::obj_delete_at_hint.0000000057:head)])", 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:41:39.216473+0000", 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 148.29041646100001, 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.082714981000000007, 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216473+0000", 2024-02-23T00:44:07.562 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216473+0000", 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216473+0000", 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.0299999999999999e-06 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.563 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216479+0000", 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0036e-05 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216489+0000", 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.8669999999999996e-06 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.216496+0000", 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.082519156999999996 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.564 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.299015+0000", 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000173456 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:41:39.299188+0000", 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.082714981000000007 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_repop(client.27567.0:1794 247.14 e726/722 247:2b095bf0:test-rados-api-smithi087-82091-127::foo:head v 726'2, mlcod=0'0)", 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.233990+0000", 2024-02-23T00:44:07.565 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 99.272900034000003, 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013165815000000001, 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.233990+0000", 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.233990+0000", 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.566 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.233989+0000", 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0023927250000000001 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.236382+0000", 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.3785e-05 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.236406+0000", 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.0248000000000001e-05 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.567 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.236416+0000", 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.8786999999999999e-05 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.236465+0000", 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.9758e-05 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.236484+0000", 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010630100999999999 2024-02-23T00:44:07.568 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.247114+0000", 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0970999999999998e-05 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.247155+0000", 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.013165815000000001 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.569 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(osd.2.18:62 234.4s0 234:2b095bf0:test-rados-api-smithi087-82091-127::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected e726)", 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:42:28.256578+0000", 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 99.250311761000006, 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017201311, 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256578+0000", 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.570 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256578+0000", 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999976 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256576+0000", 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.2082000000000001e-05 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256588+0000", 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.8547999999999999e-05 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.571 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256606+0000", 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 8.6589999999999996e-06 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256615+0000", 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.2403e-05 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256667+0000", 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.6006000000000002e-05 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.572 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.256763+0000", 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.010406540000000001 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.267170+0000", 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0055316580000000001 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.272702+0000", 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.001032065 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.573 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.273734+0000", 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.5541999999999999e-05 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:42:28.273779+0000", 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.017201311 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:2312 234.4s0 234:3736061f:test-rados-api-smithi087-82091-135::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e765)", 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:43:16.300376+0000", 2024-02-23T00:44:07.574 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 51.206514091999999, 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0058262770000000004, 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2312 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.300376+0000", 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.575 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.300376+0000", 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999986 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.300374+0000", 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.0500000000000003e-06 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.300381+0000", 2024-02-23T00:44:07.576 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.1889999999999996e-06 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.300390+0000", 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.9870000000000002e-06 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.300397+0000", 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0021571530000000002 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.577 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.302554+0000", 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.000135098 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.302690+0000", 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00046569200000000003 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.303155+0000", 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.002632718 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:44:07.578 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.305788+0000", 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00037006700000000002 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "commit_sent", 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.306158+0000", 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.3862000000000001e-05 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:16.306202+0000", 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0058262770000000004 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.579 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "osd_op(client.27567.0:2411 251.10 251:08000000:test-rados-api-smithi087-82091-135:::head [pgnls start_epoch 771] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e771)", 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:43:22.503365+0000", 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 45.003524777999999, 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.043661493000000003, 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "started", 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "client_info": { 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "client": "client.27567", 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "client_addr": "172.21.15.87:0/3013340640", 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "tid": 2411 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.580 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.503365+0000", 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.503365+0000", 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999981 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.503363+0000", 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.1090999999999999e-05 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.581 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.503374+0000", 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.6382e-05 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.503390+0000", 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.4359999999999996e-06 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.503398+0000", 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.4094e-05 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.582 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.503462+0000", 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 5.1229999999999999e-06 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "op must wait for map", 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.503467+0000", 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.043395990000000002 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.546863+0000", 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.0061999999999997e-05 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "started", 2024-02-23T00:44:07.583 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.546903+0000", 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00012345499999999999 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:22.547026+0000", 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.043661493000000003 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDPGPushReply(5.3 781/678 [PushReplyOp(5:c609908c:::notify.5:head)])", 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:43:38.635162+0000", 2024-02-23T00:44:07.584 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 28.871727255, 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039604779999999999, 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:38.635162+0000", 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:38.635162+0000", 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.585 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:38.635162+0000", 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.3790000000000002e-06 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:38.635165+0000", 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4.2490000000000003e-06 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:38.635170+0000", 2024-02-23T00:44:07.586 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 9.2769999999999994e-06 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:38.635179+0000", 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0037680729999999998 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:38.638947+0000", 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00017579499999999999 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:43:38.639123+0000", 2024-02-23T00:44:07.587 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.0039604779999999999 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: "description": "MOSDECSubOpWrite(234.0s2 802/688 ECSubWrite(tid=1698, reqid=client.27567.0:3125, at_version=802'17, trim_to=0'0, roll_forward_to=799'16))", 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: "initiated_at": "2024-02-23T00:44:03.285172+0000", 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: "age": 4.2217175679999999, 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00038966800000000002, 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: "type_data": { 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: "flag_point": "reached pg", 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: "events": [ 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "initiated", 2024-02-23T00:44:07.588 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285172+0000", 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "throttled", 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285172+0000", 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 4294967295.9999995 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "header_read", 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285172+0000", 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 2.0718999999999999e-05 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "all_read", 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285193+0000", 2024-02-23T00:44:07.589 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.5909999999999998e-06 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "dispatched", 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285200+0000", 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 6.5930000000000002e-06 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "queued_for_pg", 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285207+0000", 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 3.8776000000000003e-05 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "reached_pg", 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285245+0000", 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 7.1720000000000001e-06 2024-02-23T00:44:07.590 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_started", 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285253+0000", 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00029175499999999999 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "sub_op_committed", 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285544+0000", 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 1.7357e-05 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: }, 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: { 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: "event": "done", 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: "time": "2024-02-23T00:44:03.285562+0000", 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: "duration": 0.00038966800000000002 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.591 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.592 INFO:teuthology.orchestra.run.smithi087.stdout: } 2024-02-23T00:44:07.592 INFO:teuthology.orchestra.run.smithi087.stdout: ] 2024-02-23T00:44:07.592 INFO:teuthology.orchestra.run.smithi087.stdout:} 2024-02-23T00:44:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:07.593 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:07.593 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:44:07.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4' 2024-02-23T00:44:07.694 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:44:07.720 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:07.739 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:44:07.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:07.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:44:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:06.565321+0000 osd.1 (osd.1) 368 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:44:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:06.568334+0000 osd.1 (osd.1) 369 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:44:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:07.253753+0000 mon.a (mon.0) 3932 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:44:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:07.260133+0000 mon.a (mon.0) 3933 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:44:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:07.264294+0000 mon.a (mon.0) 3934 : audit [INF] from='client.? 172.21.15.87:0/200433555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:44:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:07.264357+0000 mon.a (mon.0) 3935 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-02-23T00:44:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:06.565321+0000 osd.1 (osd.1) 368 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:44:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:06.568334+0000 osd.1 (osd.1) 369 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:44:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:07.253753+0000 mon.a (mon.0) 3932 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:44:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:07.260133+0000 mon.a (mon.0) 3933 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:44:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:07.264294+0000 mon.a (mon.0) 3934 : audit [INF] from='client.? 172.21.15.87:0/200433555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:44:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:07.264357+0000 mon.a (mon.0) 3935 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-02-23T00:44:07.842 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7 2024-02-23T00:44:07.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:06.565321+0000 osd.1 (osd.1) 368 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:44:07.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:06.568334+0000 osd.1 (osd.1) 369 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:44:07.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:07.253753+0000 mon.a (mon.0) 3932 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:44:07.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:07.260133+0000 mon.a (mon.0) 3933 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-23T00:44:07.867 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:07.264294+0000 mon.a (mon.0) 3934 : audit [INF] from='client.? 172.21.15.87:0/200433555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-23T00:44:07.868 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:07.264357+0000 mon.a (mon.0) 3935 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-02-23T00:44:07.868 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:07.887 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:44:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:07.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7' 2024-02-23T00:44:07.990 DEBUG:teuthology.orchestra.run.smithi171:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5 2024-02-23T00:44:08.016 INFO:teuthology.orchestra.run.smithi171.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:08.035 INFO:teuthology.orchestra.run.smithi171.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:44:08.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:08.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi171 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5' 2024-02-23T00:44:08.139 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2 2024-02-23T00:44:08.167 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:08.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:06.885465+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:44:08.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:06.888367+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:44:08.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:07.810441+0000 mgr.y (mgr.24437) 681 : cluster [DBG] pgmap v1237: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:08.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:08.268172+0000 mon.a (mon.0) 3936 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-02-23T00:44:08.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:08.270393+0000 mon.a (mon.0) 3937 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:08.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:06.885465+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:44:08.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:06.888367+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:44:08.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:07.810441+0000 mgr.y (mgr.24437) 681 : cluster [DBG] pgmap v1237: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:08.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:08.268172+0000 mon.a (mon.0) 3936 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-02-23T00:44:08.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:08.270393+0000 mon.a (mon.0) 3937 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:08.895 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:44:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:08.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2' 2024-02-23T00:44:08.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:06.885465+0000 osd.6 (osd.6) 342 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:44:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:06.888367+0000 osd.6 (osd.6) 343 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:44:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:07.810441+0000 mgr.y (mgr.24437) 681 : cluster [DBG] pgmap v1237: 172 pgs: 1 active+remapped+backfill_toofull, 171 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/816 objects misplaced (1.103%) 2024-02-23T00:44:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:08.268172+0000 mon.a (mon.0) 3936 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-02-23T00:44:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:08.270393+0000 mon.a (mon.0) 3937 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:08.998 DEBUG:teuthology.orchestra.run.smithi087:> sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3 2024-02-23T00:44:09.028 INFO:teuthology.orchestra.run.smithi087.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-23T00:44:09.543 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.223+0000 7f3069727700 1 -- 172.21.15.87:0/2177940713 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30640f8d10 msgr2=0x7f30640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.223+0000 7f3069727700 1 --2- 172.21.15.87:0/2177940713 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30640f8d10 0x7f30640f90f0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 rx=0x7f3054004870 tx=0x7f3054009cb0).stop 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.223+0000 7f3069727700 1 -- 172.21.15.87:0/2177940713 shutdown_connections 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.223+0000 7f3069727700 1 --2- 172.21.15.87:0/2177940713 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3064102270 0x7f3064104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.223+0000 7f3069727700 1 --2- 172.21.15.87:0/2177940713 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30640f8d10 0x7f30640f90f0 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.223+0000 7f3069727700 1 --2- 172.21.15.87:0/2177940713 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f30640f9670 0x7f3064101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.223+0000 7f3069727700 1 -- 172.21.15.87:0/2177940713 >> 172.21.15.87:0/2177940713 conn(0x7f30640f2820 msgr2=0x7f30640f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.223+0000 7f3069727700 1 -- 172.21.15.87:0/2177940713 shutdown_connections 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.223+0000 7f3069727700 1 -- 172.21.15.87:0/2177940713 wait complete. 2024-02-23T00:44:09.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 librados: starting msgr at 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 librados: starting objecter 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 Processor -- start 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 -- start start 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 librados: setting wanted keys 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 librados: calling monclient init 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30640f8d10 0x7f306410e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30640f9670 0x7f306410e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3064102270 0x7f30641083c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30641088b0 con 0x7f3064102270 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3064108a20 con 0x7f30640f9670 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3069727700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3064108be0 con 0x7f30640f8d10 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f30637fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3064102270 0x7f30641083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f3062ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30640f8d10 0x7f306410e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:09.705 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f30637fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3064102270 0x7f30641083c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55306/0 (socket says 172.21.15.87:55306) 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f30637fe700 1 -- 172.21.15.87:0/3558025530 learned_addr learned my addr 172.21.15.87:0/3558025530 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.224+0000 7f30627fc700 1 --2- 172.21.15.87:0/3558025530 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30640f9670 0x7f306410e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3062ffd700 1 -- 172.21.15.87:0/3558025530 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30640f9670 msgr2=0x7f306410e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3062ffd700 1 --2- 172.21.15.87:0/3558025530 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30640f9670 0x7f306410e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3062ffd700 1 -- 172.21.15.87:0/3558025530 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3064102270 msgr2=0x7f30641083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3062ffd700 1 --2- 172.21.15.87:0/3558025530 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3064102270 0x7f30641083c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3062ffd700 1 -- 172.21.15.87:0/3558025530 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3058005040 con 0x7f30640f8d10 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f30637fe700 1 --2- 172.21.15.87:0/3558025530 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3064102270 0x7f30641083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3062ffd700 1 --2- 172.21.15.87:0/3558025530 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30640f8d10 0x7f306410e220 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7f3054004870 tx=0x7f305400b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f304bfff700 10 client.?.objecter ms_handle_connect 0x7f30640f8d10 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f304bfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f304bfff700 1 -- 172.21.15.87:0/3558025530 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305400e070 con 0x7f30640f8d10 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3054005040 con 0x7f30640f8d10 2024-02-23T00:44:09.706 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f304bfff700 1 -- 172.21.15.87:0/3558025530 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3054013da0 con 0x7f30640f8d10 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f304bfff700 1 -- 172.21.15.87:0/3558025530 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054019070 con 0x7f30640f8d10 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3069727700 10 client.38411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3064109080 con 0x7f30640f8d10 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.225+0000 7f3069727700 1 librados: init done 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.226+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3064109270 con 0x7f30640f8d10 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.226+0000 7f304bfff700 1 -- 172.21.15.87:0/3558025530 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f305401e420 con 0x7f30640f8d10 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.227+0000 7f304bfff700 1 --2- 172.21.15.87:0/3558025530 >> v2:172.21.15.87:6800/1231597463 conn(0x7f304c062c90 0x7f304c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.227+0000 7f304bfff700 1 -- 172.21.15.87:0/3558025530 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(807..807 src has 400..807) v4 ==== 7920+0+0 (secure 0 0 0) 0x7f305407d9b0 con 0x7f30640f8d10 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.227+0000 7f304bfff700 10 client.38411.objecter ms_dispatch 0x7f3064001550 osd_map(807..807 src has 400..807) v4 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.227+0000 7f304bfff700 3 client.38411.objecter handle_osd_map got epochs [807,807] > 0 2024-02-23T00:44:09.707 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.227+0000 7f304bfff700 3 client.38411.objecter handle_osd_map decoding full epoch 807 2024-02-23T00:44:09.708 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:44:09.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.227+0000 7f30627fc700 1 --2- 172.21.15.87:0/3558025530 >> v2:172.21.15.87:6800/1231597463 conn(0x7f304c062c90 0x7f304c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:09.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.227+0000 7f304bfff700 20 client.38411.objecter dump_active .. 0 homeless 2024-02-23T00:44:09.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.227+0000 7f304bfff700 10 client.38411.objecter ms_handle_connect 0x7f304c062c90 2024-02-23T00:44:09.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.227+0000 7f30627fc700 1 --2- 172.21.15.87:0/3558025530 >> v2:172.21.15.87:6800/1231597463 conn(0x7f304c062c90 0x7f304c065150 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 rx=0x7f3064109b60 tx=0x7f305800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:09.711 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.231+0000 7f304bfff700 1 -- 172.21.15.87:0/3558025530 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3054051030 con 0x7f30640f8d10 2024-02-23T00:44:09.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.397+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3064107310 con 0x7f304c062c90 2024-02-23T00:44:09.712 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.409+0000 7f304bfff700 1 -- 172.21.15.87:0/3558025530 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+432371 (secure 0 0 0) 0x7f3064107310 con 0x7f304c062c90 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 10 librados: watch_flush enter 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 10 librados: watch_flush exit 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 20 client.38411.objecter shutdown clearing up homeless session... 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 10 client.38411.objecter successfully canceled tick 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 >> v2:172.21.15.87:6800/1231597463 conn(0x7f304c062c90 msgr2=0x7f304c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 --2- 172.21.15.87:0/3558025530 >> v2:172.21.15.87:6800/1231597463 conn(0x7f304c062c90 0x7f304c065150 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 rx=0x7f3064109b60 tx=0x7f305800b040).stop 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30640f8d10 msgr2=0x7f306410e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 --2- 172.21.15.87:0/3558025530 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30640f8d10 0x7f306410e220 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 rx=0x7f3054004870 tx=0x7f305400b040).stop 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 shutdown_connections 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 --2- 172.21.15.87:0/3558025530 >> v2:172.21.15.87:6800/1231597463 conn(0x7f304c062c90 0x7f304c065150 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:09.721 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 --2- 172.21.15.87:0/3558025530 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3064102270 0x7f30641083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:09.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 --2- 172.21.15.87:0/3558025530 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f30640f8d10 0x7f306410e220 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:09.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 --2- 172.21.15.87:0/3558025530 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f30640f9670 0x7f306410e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:09.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.704+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 >> 172.21.15.87:0/3558025530 conn(0x7f30640f2820 msgr2=0x7f30640f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:09.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.705+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 shutdown_connections 2024-02-23T00:44:09.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.705+0000 7f3069727700 1 -- 172.21.15.87:0/3558025530 wait complete. 2024-02-23T00:44:09.722 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:09.705+0000 7f3069727700 1 librados: shutdown 2024-02-23T00:44:09.722 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:44:09.816 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:10.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8041 ms) 2024-02-23T00:44:10.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-02-23T00:44:10.278 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: ok, hit_set contains 248:602f83fe:::foo:head 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8099 ms) 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: first is 1708648967 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,1708648969,1708648970,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,1708648969,1708648970,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,1708648969,1708648970,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,1708648969,1708648970,1708648972,1708648973,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,1708648969,1708648970,1708648972,1708648973,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,1708648969,1708648970,1708648972,1708648973,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,1708648969,1708648970,1708648972,1708648973,1708648975,1708648976,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,1708648969,1708648970,1708648972,1708648973,1708648975,1708648976,0 2024-02-23T00:44:10.279 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648967,1708648969,1708648970,1708648972,1708648973,1708648975,1708648976,0 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: got ls 1708648970,1708648972,1708648973,1708648975,1708648976,1708648978,1708648979,0 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: first now 1708648970, trimmed 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21071 ms) 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: foo0 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13829 ms) 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17604 ms) 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (21407 ms) 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5046 ms) 2024-02-23T00:44:10.280 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3018 ms) 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (2904 ms) 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3003 ms) 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (231085 ms total) 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [==========] 72 tests from 4 test suites ran. (808162 ms total) 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stdout: api_tier_pp: [ PASSED ] 72 tests. 2024-02-23T00:44:10.281 INFO:tasks.workunit.client.0.smithi087.stderr:+ exit 0 2024-02-23T00:44:10.282 INFO:tasks.workunit.client.0.smithi087.stderr:+ cleanup 2024-02-23T00:44:10.282 INFO:tasks.workunit.client.0.smithi087.stderr:+ pkill -P 82001 2024-02-23T00:44:10.288 INFO:tasks.workunit.client.0.smithi087.stderr:+ true 2024-02-23T00:44:10.289 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-23T00:44:10.289 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-23T00:44:10.317 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-02-23T00:44:10.317 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-23T00:44:10.338 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:09.268135+0000 mon.a (mon.0) 3938 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:44:10.339 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:09.268193+0000 mon.a (mon.0) 3939 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-02-23T00:44:10.339 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:09.268735+0000 mon.a (mon.0) 3940 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:10.339 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:09.397705+0000 mgr.y (mgr.24437) 682 : audit [DBG] from='client.38411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:44:10.381 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:09.268135+0000 mon.a (mon.0) 3938 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:44:10.381 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:09.268193+0000 mon.a (mon.0) 3939 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-02-23T00:44:10.381 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:09.268735+0000 mon.a (mon.0) 3940 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:10.381 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:09.397705+0000 mgr.y (mgr.24437) 682 : audit [DBG] from='client.38411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:44:10.382 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:09.268135+0000 mon.a (mon.0) 3938 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:44:10.382 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:09.268193+0000 mon.a (mon.0) 3939 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-02-23T00:44:10.382 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:09.268735+0000 mon.a (mon.0) 3940 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-101"}]: dispatch 2024-02-23T00:44:10.382 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:09.397705+0000 mgr.y (mgr.24437) 682 : audit [DBG] from='client.38411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:44:10.647 DEBUG:teuthology.parallel:result is None 2024-02-23T00:44:10.647 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-02-23T00:44:10.714 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-02-23T00:44:10.714 DEBUG:teuthology.orchestra.run.smithi087:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-02-23T00:44:10.806 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-02-23T00:44:10.806 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-23T00:44:10.814 INFO:tasks.thrashosds:joining thrashosds 2024-02-23T00:44:11.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:09.810792+0000 mgr.y (mgr.24437) 683 : cluster [DBG] pgmap v1240: 164 pgs: 1 active+remapped+backfill_toofull, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/684 objects misplaced (1.316%) 2024-02-23T00:44:11.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:10.272449+0000 mon.a (mon.0) 3941 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:44:11.430 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:10.272511+0000 mon.a (mon.0) 3942 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-02-23T00:44:11.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:44:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:44:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:09.810792+0000 mgr.y (mgr.24437) 683 : cluster [DBG] pgmap v1240: 164 pgs: 1 active+remapped+backfill_toofull, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/684 objects misplaced (1.316%) 2024-02-23T00:44:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:10.272449+0000 mon.a (mon.0) 3941 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:44:11.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:10.272511+0000 mon.a (mon.0) 3942 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-02-23T00:44:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:09.810792+0000 mgr.y (mgr.24437) 683 : cluster [DBG] pgmap v1240: 164 pgs: 1 active+remapped+backfill_toofull, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/684 objects misplaced (1.316%) 2024-02-23T00:44:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:10.272449+0000 mon.a (mon.0) 3941 : audit [INF] from='client.? 172.21.15.87:0/3013340640' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi087-82091-101"}]': finished 2024-02-23T00:44:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:10.272511+0000 mon.a (mon.0) 3942 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-02-23T00:44:11.635 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1239,"stamp":"2024-02-23T00:44:09.265503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5746,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1367,"num_read_kb":1243,"num_write":409,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":6935,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":850,"ondisk_log_size":850,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":559,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2487644,"kb_used_data":63184,"kb_used_omap":85,"kb_used_meta":2424298,"kb_avail":747473572,"statfs":{"total":767960285184,"available":765412937728,"internally_reserved":0,"allocated":64700416,"data_stored":37404149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":2482481280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000344,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-58,"num_read_kb":-33,"num_write":-91,"num_write_kb":-3967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-19,"num_bytes_recovered":-123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006574"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":384,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.052232+0000","last_change":"2024-02-23T00:42:58.676506+0000","last_active":"2024-02-23T00:44:05.052232+0000","last_peered":"2024-02-23T00:44:05.052232+0000","last_clean":"2024-02-23T00:44:05.052232+0000","last_became_active":"2024-02-23T00:42:41.400209+0000","last_became_peered":"2024-02-23T00:42:41.400209+0000","last_unstale":"2024-02-23T00:44:05.052232+0000","last_undegraded":"2024-02-23T00:44:05.052232+0000","last_fullsized":"2024-02-23T00:44:05.052232+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:58.676432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:58.676432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"571'11","reported_seq":1026,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.266605+0000","last_change":"2024-02-23T00:43:53.037184+0000","last_active":"2024-02-23T00:44:05.266605+0000","last_peered":"2024-02-23T00:44:05.266605+0000","last_clean":"2024-02-23T00:44:05.266605+0000","last_became_active":"2024-02-23T00:39:49.195036+0000","last_became_peered":"2024-02-23T00:39:49.195036+0000","last_unstale":"2024-02-23T00:44:05.266605+0000","last_undegraded":"2024-02-23T00:44:05.266605+0000","last_fullsized":"2024-02-23T00:44:05.266605+0000","mapping_epoch":58,"log_start":"379'9","ondisk_log_start":"379'9","created":375,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'11","last_scrub_stamp":"2024-02-23T00:43:53.037102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:53.037102+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":894,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.545596+0000","last_change":"2024-02-23T00:42:16.067711+0000","last_active":"2024-02-23T00:44:04.545596+0000","last_peered":"2024-02-23T00:44:04.545596+0000","last_clean":"2024-02-23T00:44:04.545596+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:44:04.545596+0000","last_undegraded":"2024-02-23T00:44:04.545596+0000","last_fullsized":"2024-02-23T00:44:04.545596+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:42:16.067590+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.067590+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":826,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.267101+0000","last_change":"2024-02-23T00:43:18.439448+0000","last_active":"2024-02-23T00:44:04.267101+0000","last_peered":"2024-02-23T00:44:04.267101+0000","last_clean":"2024-02-23T00:44:04.267101+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:44:04.267101+0000","last_undegraded":"2024-02-23T00:44:04.267101+0000","last_fullsized":"2024-02-23T00:44:04.267101+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:18.439349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:43:18.439349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":808,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.545717+0000","last_change":"2024-02-23T00:42:54.046985+0000","last_active":"2024-02-23T00:44:04.545717+0000","last_peered":"2024-02-23T00:44:04.545717+0000","last_clean":"2024-02-23T00:44:04.545717+0000","last_became_active":"2024-02-23T00:41:37.222092+0000","last_became_peered":"2024-02-23T00:41:37.222092+0000","last_unstale":"2024-02-23T00:44:04.545717+0000","last_undegraded":"2024-02-23T00:44:04.545717+0000","last_fullsized":"2024-02-23T00:44:04.545717+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:54.046903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.046903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":756,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543714+0000","last_change":"2024-02-23T00:43:06.398049+0000","last_active":"2024-02-23T00:44:04.543714+0000","last_peered":"2024-02-23T00:44:04.543714+0000","last_clean":"2024-02-23T00:44:04.543714+0000","last_became_active":"2024-02-23T00:41:37.214214+0000","last_became_peered":"2024-02-23T00:41:37.214214+0000","last_unstale":"2024-02-23T00:44:04.543714+0000","last_undegraded":"2024-02-23T00:44:04.543714+0000","last_fullsized":"2024-02-23T00:44:04.543714+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.397979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.397979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":1479,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.666107+0000","last_change":"2024-02-23T00:43:47.960478+0000","last_active":"2024-02-23T00:44:04.666107+0000","last_peered":"2024-02-23T00:44:04.666107+0000","last_clean":"2024-02-23T00:44:04.666107+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:44:04.666107+0000","last_undegraded":"2024-02-23T00:44:04.666107+0000","last_fullsized":"2024-02-23T00:44:04.666107+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:43:47.960399+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:43:47.960399+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":685,"num_read_kb":685,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"453'22","reported_seq":518,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262703+0000","last_change":"2024-02-23T00:43:55.458947+0000","last_active":"2024-02-23T00:44:06.262703+0000","last_peered":"2024-02-23T00:44:06.262703+0000","last_clean":"2024-02-23T00:44:06.262703+0000","last_became_active":"2024-02-23T00:41:41.226195+0000","last_became_peered":"2024-02-23T00:41:41.226195+0000","last_unstale":"2024-02-23T00:44:06.262703+0000","last_undegraded":"2024-02-23T00:44:06.262703+0000","last_fullsized":"2024-02-23T00:44:06.262703+0000","mapping_epoch":681,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:43:55.458881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:55.458881+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":799,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542626+0000","last_change":"2024-02-23T00:42:29.119245+0000","last_active":"2024-02-23T00:44:04.542626+0000","last_peered":"2024-02-23T00:44:04.542626+0000","last_clean":"2024-02-23T00:44:04.542626+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:44:04.542626+0000","last_undegraded":"2024-02-23T00:44:04.542626+0000","last_fullsized":"2024-02-23T00:44:04.542626+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:29.119169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:29.119169+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":813,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.265056+0000","last_change":"2024-02-23T00:42:56.444185+0000","last_active":"2024-02-23T00:44:04.265056+0000","last_peered":"2024-02-23T00:44:04.265056+0000","last_clean":"2024-02-23T00:44:04.265056+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:44:04.265056+0000","last_undegraded":"2024-02-23T00:44:04.265056+0000","last_fullsized":"2024-02-23T00:44:04.265056+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:56.444091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:34:25.679076+0000","last_clean_scrub_stamp":"2024-02-23T00:42:56.444091+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":146,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.261731+0000","last_change":"2024-02-23T00:43:13.630793+0000","last_active":"2024-02-23T00:44:06.261731+0000","last_peered":"2024-02-23T00:44:06.261731+0000","last_clean":"2024-02-23T00:44:06.261731+0000","last_became_active":"2024-02-23T00:41:37.228937+0000","last_became_peered":"2024-02-23T00:41:37.228937+0000","last_unstale":"2024-02-23T00:44:06.261731+0000","last_undegraded":"2024-02-23T00:44:06.261731+0000","last_fullsized":"2024-02-23T00:44:06.261731+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:13.630679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:43:13.630679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":737,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.259371+0000","last_change":"2024-02-23T00:42:58.645999+0000","last_active":"2024-02-23T00:44:04.259371+0000","last_peered":"2024-02-23T00:44:04.259371+0000","last_clean":"2024-02-23T00:44:04.259371+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:44:04.259371+0000","last_undegraded":"2024-02-23T00:44:04.259371+0000","last_fullsized":"2024-02-23T00:44:04.259371+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:58.645916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:42:58.645916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":79,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.052059+0000","last_change":"2024-02-23T00:43:12.778764+0000","last_active":"2024-02-23T00:44:05.052059+0000","last_peered":"2024-02-23T00:44:05.052059+0000","last_clean":"2024-02-23T00:44:05.052059+0000","last_became_active":"2024-02-23T00:42:41.393491+0000","last_became_peered":"2024-02-23T00:42:41.393491+0000","last_unstale":"2024-02-23T00:44:05.052059+0000","last_undegraded":"2024-02-23T00:44:05.052059+0000","last_fullsized":"2024-02-23T00:44:05.052059+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:12.778695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:43:12.778695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":916,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.260930+0000","last_change":"2024-02-23T00:43:00.587024+0000","last_active":"2024-02-23T00:44:04.260930+0000","last_peered":"2024-02-23T00:44:04.260930+0000","last_clean":"2024-02-23T00:44:04.260930+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:44:04.260930+0000","last_undegraded":"2024-02-23T00:44:04.260930+0000","last_fullsized":"2024-02-23T00:44:04.260930+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:43:00.586920+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.586920+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":824,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.266871+0000","last_change":"2024-02-23T00:43:54.440398+0000","last_active":"2024-02-23T00:44:04.266871+0000","last_peered":"2024-02-23T00:44:04.266871+0000","last_clean":"2024-02-23T00:44:04.266871+0000","last_became_active":"2024-02-23T00:41:37.238728+0000","last_became_peered":"2024-02-23T00:41:37.238728+0000","last_unstale":"2024-02-23T00:44:04.266871+0000","last_undegraded":"2024-02-23T00:44:04.266871+0000","last_fullsized":"2024-02-23T00:44:04.266871+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:54.440323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:54.440323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":813,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.268067+0000","last_change":"2024-02-23T00:43:59.378149+0000","last_active":"2024-02-23T00:44:04.268067+0000","last_peered":"2024-02-23T00:44:04.268067+0000","last_clean":"2024-02-23T00:44:04.268067+0000","last_became_active":"2024-02-23T00:41:37.239529+0000","last_became_peered":"2024-02-23T00:41:37.239529+0000","last_unstale":"2024-02-23T00:44:04.268067+0000","last_undegraded":"2024-02-23T00:44:04.268067+0000","last_fullsized":"2024-02-23T00:44:04.268067+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:59.378086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.787191+0000","last_clean_scrub_stamp":"2024-02-23T00:43:59.378086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":819,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.050550+0000","last_change":"2024-02-23T00:43:45.960090+0000","last_active":"2024-02-23T00:44:05.050550+0000","last_peered":"2024-02-23T00:44:05.050550+0000","last_clean":"2024-02-23T00:44:05.050550+0000","last_became_active":"2024-02-23T00:40:37.197010+0000","last_became_peered":"2024-02-23T00:40:37.197010+0000","last_unstale":"2024-02-23T00:44:05.050550+0000","last_undegraded":"2024-02-23T00:44:05.050550+0000","last_fullsized":"2024-02-23T00:44:05.050550+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:45.960020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:50.176208+0000","last_clean_scrub_stamp":"2024-02-23T00:43:45.960020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":141,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.261706+0000","last_change":"2024-02-23T00:42:57.587635+0000","last_active":"2024-02-23T00:44:06.261706+0000","last_peered":"2024-02-23T00:44:06.261706+0000","last_clean":"2024-02-23T00:44:06.261706+0000","last_became_active":"2024-02-23T00:41:37.248272+0000","last_became_peered":"2024-02-23T00:41:37.248272+0000","last_unstale":"2024-02-23T00:44:06.261706+0000","last_undegraded":"2024-02-23T00:44:06.261706+0000","last_fullsized":"2024-02-23T00:44:06.261706+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:57.587566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:57.587566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":801,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.261127+0000","last_change":"2024-02-23T00:42:54.599143+0000","last_active":"2024-02-23T00:44:04.261127+0000","last_peered":"2024-02-23T00:44:04.261127+0000","last_clean":"2024-02-23T00:44:04.261127+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:44:04.261127+0000","last_undegraded":"2024-02-23T00:44:04.261127+0000","last_fullsized":"2024-02-23T00:44:04.261127+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:54.599070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.599070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":635,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.263220+0000","last_change":"2024-02-23T00:43:50.462572+0000","last_active":"2024-02-23T00:44:04.263220+0000","last_peered":"2024-02-23T00:44:04.263220+0000","last_clean":"2024-02-23T00:44:04.263220+0000","last_became_active":"2024-02-23T00:41:50.256494+0000","last_became_peered":"2024-02-23T00:41:50.256494+0000","last_unstale":"2024-02-23T00:44:04.263220+0000","last_undegraded":"2024-02-23T00:44:04.263220+0000","last_fullsized":"2024-02-23T00:44:04.263220+0000","mapping_epoch":688,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:43:50.462511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:50.462511+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":789,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.050404+0000","last_change":"2024-02-23T00:43:47.993670+0000","last_active":"2024-02-23T00:44:05.050404+0000","last_peered":"2024-02-23T00:44:05.050404+0000","last_clean":"2024-02-23T00:44:05.050404+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:44:05.050404+0000","last_undegraded":"2024-02-23T00:44:05.050404+0000","last_fullsized":"2024-02-23T00:44:05.050404+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:47.993579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:47.993579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":796,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543958+0000","last_change":"2024-02-23T00:43:46.607069+0000","last_active":"2024-02-23T00:44:04.543958+0000","last_peered":"2024-02-23T00:44:04.543958+0000","last_clean":"2024-02-23T00:44:04.543958+0000","last_became_active":"2024-02-23T00:41:37.229416+0000","last_became_peered":"2024-02-23T00:41:37.229416+0000","last_unstale":"2024-02-23T00:44:04.543958+0000","last_undegraded":"2024-02-23T00:44:04.543958+0000","last_fullsized":"2024-02-23T00:44:04.543958+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:46.607006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:46.607006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":35,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.264928+0000","last_change":"2024-02-23T00:43:47.452889+0000","last_active":"2024-02-23T00:44:06.264928+0000","last_peered":"2024-02-23T00:44:06.264928+0000","last_clean":"2024-02-23T00:44:06.264928+0000","last_became_active":"2024-02-23T00:43:42.260383+0000","last_became_peered":"2024-02-23T00:43:42.260383+0000","last_unstale":"2024-02-23T00:44:06.264928+0000","last_undegraded":"2024-02-23T00:44:06.264928+0000","last_fullsized":"2024-02-23T00:44:06.264928+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:47.452768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:47.452768+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":326,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264684+0000","last_change":"2024-02-23T00:43:48.075085+0000","last_active":"2024-02-23T00:44:04.264684+0000","last_peered":"2024-02-23T00:44:04.264684+0000","last_clean":"2024-02-23T00:44:04.264684+0000","last_became_active":"2024-02-23T00:41:37.252083+0000","last_became_peered":"2024-02-23T00:41:37.252083+0000","last_unstale":"2024-02-23T00:44:04.264684+0000","last_undegraded":"2024-02-23T00:44:04.264684+0000","last_fullsized":"2024-02-23T00:44:04.264684+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:48.075026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:48.075026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":735,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.265793+0000","last_change":"2024-02-23T00:44:00.894837+0000","last_active":"2024-02-23T00:44:04.265793+0000","last_peered":"2024-02-23T00:44:04.265793+0000","last_clean":"2024-02-23T00:44:04.265793+0000","last_became_active":"2024-02-23T00:41:50.256004+0000","last_became_peered":"2024-02-23T00:41:50.256004+0000","last_unstale":"2024-02-23T00:44:04.265793+0000","last_undegraded":"2024-02-23T00:44:04.265793+0000","last_fullsized":"2024-02-23T00:44:04.265793+0000","mapping_epoch":688,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:44:00.894776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:44:00.894776+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":812,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543100+0000","last_change":"2024-02-23T00:42:15.223520+0000","last_active":"2024-02-23T00:44:04.543100+0000","last_peered":"2024-02-23T00:44:04.543100+0000","last_clean":"2024-02-23T00:44:04.543100+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:44:04.543100+0000","last_undegraded":"2024-02-23T00:44:04.543100+0000","last_fullsized":"2024-02-23T00:44:04.543100+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:15.223455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:42:15.223455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":1004,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.262299+0000","last_change":"2024-02-23T00:42:55.471736+0000","last_active":"2024-02-23T00:44:05.262299+0000","last_peered":"2024-02-23T00:44:05.262299+0000","last_clean":"2024-02-23T00:44:05.262299+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:44:05.262299+0000","last_undegraded":"2024-02-23T00:44:05.262299+0000","last_fullsized":"2024-02-23T00:44:05.262299+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:42:55.471653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:55.471653+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":741,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.199562+0000","last_change":"2024-02-23T00:44:01.138487+0000","last_active":"2024-02-23T00:44:05.199562+0000","last_peered":"2024-02-23T00:44:05.199562+0000","last_clean":"2024-02-23T00:44:05.199562+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:44:05.199562+0000","last_undegraded":"2024-02-23T00:44:05.199562+0000","last_fullsized":"2024-02-23T00:44:05.199562+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:01.138396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:44:01.138396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":312,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.263575+0000","last_change":"2024-02-23T00:43:58.944323+0000","last_active":"2024-02-23T00:44:04.263575+0000","last_peered":"2024-02-23T00:44:04.263575+0000","last_clean":"2024-02-23T00:44:04.263575+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:44:04.263575+0000","last_undegraded":"2024-02-23T00:44:04.263575+0000","last_fullsized":"2024-02-23T00:44:04.263575+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:43:58.944256+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:43:58.944256+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":971,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543850+0000","last_change":"2024-02-23T00:43:49.539057+0000","last_active":"2024-02-23T00:44:04.543850+0000","last_peered":"2024-02-23T00:44:04.543850+0000","last_clean":"2024-02-23T00:44:04.543850+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:44:04.543850+0000","last_undegraded":"2024-02-23T00:44:04.543850+0000","last_fullsized":"2024-02-23T00:44:04.543850+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:43:49.538978+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:43:49.538978+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":796,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542267+0000","last_change":"2024-02-23T00:42:14.253417+0000","last_active":"2024-02-23T00:44:04.542267+0000","last_peered":"2024-02-23T00:44:04.542267+0000","last_clean":"2024-02-23T00:44:04.542267+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:44:04.542267+0000","last_undegraded":"2024-02-23T00:44:04.542267+0000","last_fullsized":"2024-02-23T00:44:04.542267+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.253319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:06.657998+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.253319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"689'12","reported_seq":1011,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262469+0000","last_change":"2024-02-23T00:42:19.683007+0000","last_active":"2024-02-23T00:44:06.262469+0000","last_peered":"2024-02-23T00:44:06.262469+0000","last_clean":"2024-02-23T00:44:06.262469+0000","last_became_active":"2024-02-23T00:41:50.256773+0000","last_became_peered":"2024-02-23T00:41:50.256773+0000","last_unstale":"2024-02-23T00:44:06.262469+0000","last_undegraded":"2024-02-23T00:44:06.262469+0000","last_fullsized":"2024-02-23T00:44:06.262469+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"689'12","last_scrub_stamp":"2024-02-23T00:42:19.682902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.682902+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":723,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542274+0000","last_change":"2024-02-23T00:41:40.171696+0000","last_active":"2024-02-23T00:44:04.542274+0000","last_peered":"2024-02-23T00:44:04.542274+0000","last_clean":"2024-02-23T00:44:04.542274+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:44:04.542274+0000","last_undegraded":"2024-02-23T00:44:04.542274+0000","last_fullsized":"2024-02-23T00:44:04.542274+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:40.171630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:41:40.171630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":803,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.197024+0000","last_change":"2024-02-23T00:44:03.136753+0000","last_active":"2024-02-23T00:44:05.197024+0000","last_peered":"2024-02-23T00:44:05.197024+0000","last_clean":"2024-02-23T00:44:05.197024+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:44:05.197024+0000","last_undegraded":"2024-02-23T00:44:05.197024+0000","last_fullsized":"2024-02-23T00:44:05.197024+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:03.136682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:03.136682+0000","last_clean_scrub_stamp":"2024-02-23T00:44:03.136682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":642,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.260856+0000","last_change":"2024-02-23T00:43:52.879005+0000","last_active":"2024-02-23T00:44:04.260856+0000","last_peered":"2024-02-23T00:44:04.260856+0000","last_clean":"2024-02-23T00:44:04.260856+0000","last_became_active":"2024-02-23T00:41:50.253028+0000","last_became_peered":"2024-02-23T00:41:50.253028+0000","last_unstale":"2024-02-23T00:44:04.260856+0000","last_undegraded":"2024-02-23T00:44:04.260856+0000","last_fullsized":"2024-02-23T00:44:04.260856+0000","mapping_epoch":688,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:43:52.878911+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:43:52.878911+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":825,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.050989+0000","last_change":"2024-02-23T00:42:54.633894+0000","last_active":"2024-02-23T00:44:05.050989+0000","last_peered":"2024-02-23T00:44:05.050989+0000","last_clean":"2024-02-23T00:44:05.050989+0000","last_became_active":"2024-02-23T00:41:37.229862+0000","last_became_peered":"2024-02-23T00:41:37.229862+0000","last_unstale":"2024-02-23T00:44:05.050989+0000","last_undegraded":"2024-02-23T00:44:05.050989+0000","last_fullsized":"2024-02-23T00:44:05.050989+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:42:54.633815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.633815+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":773,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.548200+0000","last_change":"2024-02-23T00:43:06.143533+0000","last_active":"2024-02-23T00:44:04.548200+0000","last_peered":"2024-02-23T00:44:04.548200+0000","last_clean":"2024-02-23T00:44:04.548200+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:44:04.548200+0000","last_undegraded":"2024-02-23T00:44:04.548200+0000","last_fullsized":"2024-02-23T00:44:04.548200+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.143455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.143455+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":746,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.544119+0000","last_change":"2024-02-23T00:42:53.372480+0000","last_active":"2024-02-23T00:44:04.544119+0000","last_peered":"2024-02-23T00:44:04.544119+0000","last_clean":"2024-02-23T00:44:04.544119+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:44:04.544119+0000","last_undegraded":"2024-02-23T00:44:04.544119+0000","last_fullsized":"2024-02-23T00:44:04.544119+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.372422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.372422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":155,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262632+0000","last_change":"2024-02-23T00:43:03.634296+0000","last_active":"2024-02-23T00:44:06.262632+0000","last_peered":"2024-02-23T00:44:06.262632+0000","last_clean":"2024-02-23T00:44:06.262632+0000","last_became_active":"2024-02-23T00:42:39.394104+0000","last_became_peered":"2024-02-23T00:42:39.394104+0000","last_unstale":"2024-02-23T00:44:06.262632+0000","last_undegraded":"2024-02-23T00:44:06.262632+0000","last_fullsized":"2024-02-23T00:44:06.262632+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:03.634240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:43:03.634240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"642'14","reported_seq":364,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.265145+0000","last_change":"2024-02-23T00:42:59.176336+0000","last_active":"2024-02-23T00:44:05.265145+0000","last_peered":"2024-02-23T00:44:05.265145+0000","last_clean":"2024-02-23T00:44:05.265145+0000","last_became_active":"2024-02-23T00:41:01.212780+0000","last_became_peered":"2024-02-23T00:41:01.212780+0000","last_unstale":"2024-02-23T00:44:05.265145+0000","last_undegraded":"2024-02-23T00:44:05.265145+0000","last_fullsized":"2024-02-23T00:44:05.265145+0000","mapping_epoch":511,"log_start":"562'12","ondisk_log_start":"562'12","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'14","last_scrub_stamp":"2024-02-23T00:42:59.176246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.176246+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":614,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.546349+0000","last_change":"2024-02-23T00:44:03.968753+0000","last_active":"2024-02-23T00:44:04.546349+0000","last_peered":"2024-02-23T00:44:04.546349+0000","last_clean":"2024-02-23T00:44:04.546349+0000","last_became_active":"2024-02-23T00:41:50.255315+0000","last_became_peered":"2024-02-23T00:41:50.255315+0000","last_unstale":"2024-02-23T00:44:04.546349+0000","last_undegraded":"2024-02-23T00:44:04.546349+0000","last_fullsized":"2024-02-23T00:44:04.546349+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'34","last_scrub_stamp":"2024-02-23T00:44:03.968683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:44:03.968683+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.541498+0000","last_change":"2024-02-23T00:43:56.309156+0000","last_active":"2024-02-23T00:44:04.541498+0000","last_peered":"2024-02-23T00:44:04.541498+0000","last_clean":"2024-02-23T00:44:04.541498+0000","last_became_active":"2024-02-23T00:43:56.308886+0000","last_became_peered":"2024-02-23T00:43:56.308886+0000","last_unstale":"2024-02-23T00:44:04.541498+0000","last_undegraded":"2024-02-23T00:44:04.541498+0000","last_fullsized":"2024-02-23T00:44:04.541498+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:52.538255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:52.538255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":806,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.052973+0000","last_change":"2024-02-23T00:43:57.028427+0000","last_active":"2024-02-23T00:44:05.052973+0000","last_peered":"2024-02-23T00:44:05.052973+0000","last_clean":"2024-02-23T00:44:05.052973+0000","last_became_active":"2024-02-23T00:40:57.488797+0000","last_became_peered":"2024-02-23T00:40:57.488797+0000","last_unstale":"2024-02-23T00:44:05.052973+0000","last_undegraded":"2024-02-23T00:44:05.052973+0000","last_fullsized":"2024-02-23T00:44:05.052973+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:57.028360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:43:57.028360+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":766,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542969+0000","last_change":"2024-02-23T00:41:51.307510+0000","last_active":"2024-02-23T00:44:04.542969+0000","last_peered":"2024-02-23T00:44:04.542969+0000","last_clean":"2024-02-23T00:44:04.542969+0000","last_became_active":"2024-02-23T00:41:37.229061+0000","last_became_peered":"2024-02-23T00:41:37.229061+0000","last_unstale":"2024-02-23T00:44:04.542969+0000","last_undegraded":"2024-02-23T00:44:04.542969+0000","last_fullsized":"2024-02-23T00:44:04.542969+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:51.307443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:51.307443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":342,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264297+0000","last_change":"2024-02-23T00:42:39.396325+0000","last_active":"2024-02-23T00:44:04.264297+0000","last_peered":"2024-02-23T00:44:04.264297+0000","last_clean":"2024-02-23T00:44:04.264297+0000","last_became_active":"2024-02-23T00:42:39.396063+0000","last_became_peered":"2024-02-23T00:42:39.396063+0000","last_unstale":"2024-02-23T00:44:04.264297+0000","last_undegraded":"2024-02-23T00:44:04.264297+0000","last_fullsized":"2024-02-23T00:44:04.264297+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:55.355553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:55.355553+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":836,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.197250+0000","last_change":"2024-02-23T00:43:12.980617+0000","last_active":"2024-02-23T00:44:05.197250+0000","last_peered":"2024-02-23T00:44:05.197250+0000","last_clean":"2024-02-23T00:44:05.197250+0000","last_became_active":"2024-02-23T00:40:57.218869+0000","last_became_peered":"2024-02-23T00:40:57.218869+0000","last_unstale":"2024-02-23T00:44:05.197250+0000","last_undegraded":"2024-02-23T00:44:05.197250+0000","last_fullsized":"2024-02-23T00:44:05.197250+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:12.980537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:12.980537+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":375,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.263403+0000","last_change":"2024-02-23T00:42:55.230049+0000","last_active":"2024-02-23T00:44:04.263403+0000","last_peered":"2024-02-23T00:44:04.263403+0000","last_clean":"2024-02-23T00:44:04.263403+0000","last_became_active":"2024-02-23T00:41:37.228326+0000","last_became_peered":"2024-02-23T00:41:37.228326+0000","last_unstale":"2024-02-23T00:44:04.263403+0000","last_undegraded":"2024-02-23T00:44:04.263403+0000","last_fullsized":"2024-02-23T00:44:04.263403+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:42:55.229966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:55.229966+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":776,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543619+0000","last_change":"2024-02-23T00:41:42.160301+0000","last_active":"2024-02-23T00:44:04.543619+0000","last_peered":"2024-02-23T00:44:04.543619+0000","last_clean":"2024-02-23T00:44:04.543619+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:44:04.543619+0000","last_undegraded":"2024-02-23T00:44:04.543619+0000","last_fullsized":"2024-02-23T00:44:04.543619+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:42.160190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:41:42.160190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":768,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.263223+0000","last_change":"2024-02-23T00:41:50.256151+0000","last_active":"2024-02-23T00:44:04.263223+0000","last_peered":"2024-02-23T00:44:04.263223+0000","last_clean":"2024-02-23T00:44:04.263223+0000","last_became_active":"2024-02-23T00:41:50.255850+0000","last_became_peered":"2024-02-23T00:41:50.255850+0000","last_unstale":"2024-02-23T00:44:04.263223+0000","last_undegraded":"2024-02-23T00:44:04.263223+0000","last_fullsized":"2024-02-23T00:44:04.263223+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:01.638560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:01.638560+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":316,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.263269+0000","last_change":"2024-02-23T00:43:42.257077+0000","last_active":"2024-02-23T00:44:04.263269+0000","last_peered":"2024-02-23T00:44:04.263269+0000","last_clean":"2024-02-23T00:44:04.263269+0000","last_became_active":"2024-02-23T00:43:42.256711+0000","last_became_peered":"2024-02-23T00:43:42.256711+0000","last_unstale":"2024-02-23T00:44:04.263269+0000","last_undegraded":"2024-02-23T00:44:04.263269+0000","last_fullsized":"2024-02-23T00:44:04.263269+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:02.306855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:02.306855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":1053,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.547313+0000","last_change":"2024-02-23T00:44:04.547292+0000","last_active":"2024-02-23T00:44:04.547313+0000","last_peered":"2024-02-23T00:44:04.547313+0000","last_clean":"2024-02-23T00:44:04.547313+0000","last_became_active":"2024-02-23T00:42:39.398790+0000","last_became_peered":"2024-02-23T00:42:39.398790+0000","last_unstale":"2024-02-23T00:44:04.547313+0000","last_undegraded":"2024-02-23T00:44:04.547313+0000","last_fullsized":"2024-02-23T00:44:04.547313+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:44:04.547236+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:44:04.547236+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":837,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.260994+0000","last_change":"2024-02-23T00:43:48.824592+0000","last_active":"2024-02-23T00:44:04.260994+0000","last_peered":"2024-02-23T00:44:04.260994+0000","last_clean":"2024-02-23T00:44:04.260994+0000","last_became_active":"2024-02-23T00:41:37.220531+0000","last_became_peered":"2024-02-23T00:41:37.220531+0000","last_unstale":"2024-02-23T00:44:04.260994+0000","last_undegraded":"2024-02-23T00:44:04.260994+0000","last_fullsized":"2024-02-23T00:44:04.260994+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:48.824482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_clean_scrub_stamp":"2024-02-23T00:43:48.824482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"801'44","reported_seq":179,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262752+0000","last_change":"2024-02-23T00:44:02.476413+0000","last_active":"2024-02-23T00:44:06.262752+0000","last_peered":"2024-02-23T00:44:06.262752+0000","last_clean":"2024-02-23T00:44:06.262752+0000","last_became_active":"2024-02-23T00:41:37.235230+0000","last_became_peered":"2024-02-23T00:41:37.235230+0000","last_unstale":"2024-02-23T00:44:06.262752+0000","last_undegraded":"2024-02-23T00:44:06.262752+0000","last_fullsized":"2024-02-23T00:44:06.262752+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'44","last_scrub_stamp":"2024-02-23T00:44:02.476354+0000","last_deep_scrub":"801'44","last_deep_scrub_stamp":"2024-02-23T00:44:02.476354+0000","last_clean_scrub_stamp":"2024-02-23T00:44:02.476354+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":808,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.053049+0000","last_change":"2024-02-23T00:42:53.644165+0000","last_active":"2024-02-23T00:44:05.053049+0000","last_peered":"2024-02-23T00:44:05.053049+0000","last_clean":"2024-02-23T00:44:05.053049+0000","last_became_active":"2024-02-23T00:41:37.229922+0000","last_became_peered":"2024-02-23T00:41:37.229922+0000","last_unstale":"2024-02-23T00:44:05.053049+0000","last_undegraded":"2024-02-23T00:44:05.053049+0000","last_fullsized":"2024-02-23T00:44:05.053049+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.644087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.644087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":793,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542284+0000","last_change":"2024-02-23T00:43:09.384826+0000","last_active":"2024-02-23T00:44:04.542284+0000","last_peered":"2024-02-23T00:44:04.542284+0000","last_clean":"2024-02-23T00:44:04.542284+0000","last_became_active":"2024-02-23T00:41:05.523365+0000","last_became_peered":"2024-02-23T00:41:05.523365+0000","last_unstale":"2024-02-23T00:44:04.542284+0000","last_undegraded":"2024-02-23T00:44:04.542284+0000","last_fullsized":"2024-02-23T00:44:04.542284+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:09.384750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:43:09.384750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":723,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542373+0000","last_change":"2024-02-23T00:42:16.223783+0000","last_active":"2024-02-23T00:44:04.542373+0000","last_peered":"2024-02-23T00:44:04.542373+0000","last_clean":"2024-02-23T00:44:04.542373+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:44:04.542373+0000","last_undegraded":"2024-02-23T00:44:04.542373+0000","last_fullsized":"2024-02-23T00:44:04.542373+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:16.223720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:16.223720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":819,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.548269+0000","last_change":"2024-02-23T00:42:58.210034+0000","last_active":"2024-02-23T00:44:04.548269+0000","last_peered":"2024-02-23T00:44:04.548269+0000","last_clean":"2024-02-23T00:44:04.548269+0000","last_became_active":"2024-02-23T00:41:37.228303+0000","last_became_peered":"2024-02-23T00:41:37.228303+0000","last_unstale":"2024-02-23T00:44:04.548269+0000","last_undegraded":"2024-02-23T00:44:04.548269+0000","last_fullsized":"2024-02-23T00:44:04.548269+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:58.209955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:42:58.209955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":529,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262528+0000","last_change":"2024-02-23T00:44:00.421944+0000","last_active":"2024-02-23T00:44:06.262528+0000","last_peered":"2024-02-23T00:44:06.262528+0000","last_clean":"2024-02-23T00:44:06.262528+0000","last_became_active":"2024-02-23T00:41:41.224814+0000","last_became_peered":"2024-02-23T00:41:41.224814+0000","last_unstale":"2024-02-23T00:44:06.262528+0000","last_undegraded":"2024-02-23T00:44:06.262528+0000","last_fullsized":"2024-02-23T00:44:06.262528+0000","mapping_epoch":681,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:44:00.421878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:44:00.421878+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":808,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.050810+0000","last_change":"2024-02-23T00:43:16.745314+0000","last_active":"2024-02-23T00:44:05.050810+0000","last_peered":"2024-02-23T00:44:05.050810+0000","last_clean":"2024-02-23T00:44:05.050810+0000","last_became_active":"2024-02-23T00:41:37.229722+0000","last_became_peered":"2024-02-23T00:41:37.229722+0000","last_unstale":"2024-02-23T00:44:05.050810+0000","last_undegraded":"2024-02-23T00:44:05.050810+0000","last_fullsized":"2024-02-23T00:44:05.050810+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:16.745223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:16.745223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":155,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.261726+0000","last_change":"2024-02-23T00:43:06.616251+0000","last_active":"2024-02-23T00:44:06.261726+0000","last_peered":"2024-02-23T00:44:06.261726+0000","last_clean":"2024-02-23T00:44:06.261726+0000","last_became_active":"2024-02-23T00:42:39.395793+0000","last_became_peered":"2024-02-23T00:42:39.395793+0000","last_unstale":"2024-02-23T00:44:06.261726+0000","last_undegraded":"2024-02-23T00:44:06.261726+0000","last_fullsized":"2024-02-23T00:44:06.261726+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.616180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.616180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"78'2","reported_seq":824,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.544037+0000","last_change":"2024-02-23T00:41:53.371931+0000","last_active":"2024-02-23T00:44:04.544037+0000","last_peered":"2024-02-23T00:44:04.544037+0000","last_clean":"2024-02-23T00:44:04.544037+0000","last_became_active":"2024-02-23T00:41:25.236069+0000","last_became_peered":"2024-02-23T00:41:25.236069+0000","last_unstale":"2024-02-23T00:44:04.544037+0000","last_undegraded":"2024-02-23T00:44:04.544037+0000","last_fullsized":"2024-02-23T00:44:04.544037+0000","mapping_epoch":511,"log_start":"78'2","ondisk_log_start":"78'2","created":60,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:41:53.371852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:53.371852+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":736,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.198567+0000","last_change":"2024-02-23T00:42:53.959862+0000","last_active":"2024-02-23T00:44:05.198567+0000","last_peered":"2024-02-23T00:44:05.198567+0000","last_clean":"2024-02-23T00:44:05.198567+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:44:05.198567+0000","last_undegraded":"2024-02-23T00:44:05.198567+0000","last_fullsized":"2024-02-23T00:44:05.198567+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.959787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.959787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":817,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.051660+0000","last_change":"2024-02-23T00:43:50.976740+0000","last_active":"2024-02-23T00:44:05.051660+0000","last_peered":"2024-02-23T00:44:05.051660+0000","last_clean":"2024-02-23T00:44:05.051660+0000","last_became_active":"2024-02-23T00:40:45.191453+0000","last_became_peered":"2024-02-23T00:40:45.191453+0000","last_unstale":"2024-02-23T00:44:05.051660+0000","last_undegraded":"2024-02-23T00:44:05.051660+0000","last_fullsized":"2024-02-23T00:44:05.051660+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:50.976629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:43:50.976629+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":833,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.260038+0000","last_change":"2024-02-23T00:43:12.691014+0000","last_active":"2024-02-23T00:44:04.260038+0000","last_peered":"2024-02-23T00:44:04.260038+0000","last_clean":"2024-02-23T00:44:04.260038+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:44:04.260038+0000","last_undegraded":"2024-02-23T00:44:04.260038+0000","last_fullsized":"2024-02-23T00:44:04.260038+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:43:12.690926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:12.690926+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":811,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.050898+0000","last_change":"2024-02-23T00:42:28.369569+0000","last_active":"2024-02-23T00:44:05.050898+0000","last_peered":"2024-02-23T00:44:05.050898+0000","last_clean":"2024-02-23T00:44:05.050898+0000","last_became_active":"2024-02-23T00:41:37.229335+0000","last_became_peered":"2024-02-23T00:41:37.229335+0000","last_unstale":"2024-02-23T00:44:05.050898+0000","last_undegraded":"2024-02-23T00:44:05.050898+0000","last_fullsized":"2024-02-23T00:44:05.050898+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:28.369481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.369481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":782,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.546752+0000","last_change":"2024-02-23T00:43:49.992357+0000","last_active":"2024-02-23T00:44:04.546752+0000","last_peered":"2024-02-23T00:44:04.546752+0000","last_clean":"2024-02-23T00:44:04.546752+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:44:04.546752+0000","last_undegraded":"2024-02-23T00:44:04.546752+0000","last_fullsized":"2024-02-23T00:44:04.546752+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:49.992274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:49.992274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":316,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.263310+0000","last_change":"2024-02-23T00:42:39.396374+0000","last_active":"2024-02-23T00:44:04.263310+0000","last_peered":"2024-02-23T00:44:04.263310+0000","last_clean":"2024-02-23T00:44:04.263310+0000","last_became_active":"2024-02-23T00:42:39.396071+0000","last_became_peered":"2024-02-23T00:42:39.396071+0000","last_unstale":"2024-02-23T00:44:04.263310+0000","last_undegraded":"2024-02-23T00:44:04.263310+0000","last_fullsized":"2024-02-23T00:44:04.263310+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:40:52.474495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:40:52.474495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":825,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.548350+0000","last_change":"2024-02-23T00:42:22.037093+0000","last_active":"2024-02-23T00:44:04.548350+0000","last_peered":"2024-02-23T00:44:04.548350+0000","last_clean":"2024-02-23T00:44:04.548350+0000","last_became_active":"2024-02-23T00:41:13.469953+0000","last_became_peered":"2024-02-23T00:41:13.469953+0000","last_unstale":"2024-02-23T00:44:04.548350+0000","last_undegraded":"2024-02-23T00:44:04.548350+0000","last_fullsized":"2024-02-23T00:44:04.548350+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:22.037008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:42:22.037008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":738,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.198260+0000","last_change":"2024-02-23T00:43:00.007515+0000","last_active":"2024-02-23T00:44:05.198260+0000","last_peered":"2024-02-23T00:44:05.198260+0000","last_clean":"2024-02-23T00:44:05.198260+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:44:05.198260+0000","last_undegraded":"2024-02-23T00:44:05.198260+0000","last_fullsized":"2024-02-23T00:44:05.198260+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:00.007437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.007437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":828,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.267238+0000","last_change":"2024-02-23T00:43:17.386897+0000","last_active":"2024-02-23T00:44:04.267238+0000","last_peered":"2024-02-23T00:44:04.267238+0000","last_clean":"2024-02-23T00:44:04.267238+0000","last_became_active":"2024-02-23T00:40:38.061696+0000","last_became_peered":"2024-02-23T00:40:38.061696+0000","last_unstale":"2024-02-23T00:44:04.267238+0000","last_undegraded":"2024-02-23T00:44:04.267238+0000","last_fullsized":"2024-02-23T00:44:04.267238+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:17.386832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:43:17.386832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":930,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543183+0000","last_change":"2024-02-23T00:42:59.284657+0000","last_active":"2024-02-23T00:44:04.543183+0000","last_peered":"2024-02-23T00:44:04.543183+0000","last_clean":"2024-02-23T00:44:04.543183+0000","last_became_active":"2024-02-23T00:41:40.219384+0000","last_became_peered":"2024-02-23T00:41:40.219384+0000","last_unstale":"2024-02-23T00:44:04.543183+0000","last_undegraded":"2024-02-23T00:44:04.543183+0000","last_fullsized":"2024-02-23T00:44:04.543183+0000","mapping_epoch":680,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:42:59.284597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.284597+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":841,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.262798+0000","last_change":"2024-02-23T00:43:03.434468+0000","last_active":"2024-02-23T00:44:04.262798+0000","last_peered":"2024-02-23T00:44:04.262798+0000","last_clean":"2024-02-23T00:44:04.262798+0000","last_became_active":"2024-02-23T00:41:37.219000+0000","last_became_peered":"2024-02-23T00:41:37.219000+0000","last_unstale":"2024-02-23T00:44:04.262798+0000","last_undegraded":"2024-02-23T00:44:04.262798+0000","last_fullsized":"2024-02-23T00:44:04.262798+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:03.434391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:03.434391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":1046,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542327+0000","last_change":"2024-02-23T00:44:03.533574+0000","last_active":"2024-02-23T00:44:04.542327+0000","last_peered":"2024-02-23T00:44:04.542327+0000","last_clean":"2024-02-23T00:44:04.542327+0000","last_became_active":"2024-02-23T00:42:39.401599+0000","last_became_peered":"2024-02-23T00:42:39.401599+0000","last_unstale":"2024-02-23T00:44:04.542327+0000","last_undegraded":"2024-02-23T00:44:04.542327+0000","last_fullsized":"2024-02-23T00:44:04.542327+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:44:03.533460+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:44:03.533460+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"62'1","reported_seq":842,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.053092+0000","last_change":"2024-02-23T00:43:54.978131+0000","last_active":"2024-02-23T00:44:05.053092+0000","last_peered":"2024-02-23T00:44:05.053092+0000","last_clean":"2024-02-23T00:44:05.053092+0000","last_became_active":"2024-02-23T00:41:21.226819+0000","last_became_peered":"2024-02-23T00:41:21.226819+0000","last_unstale":"2024-02-23T00:44:05.053092+0000","last_undegraded":"2024-02-23T00:44:05.053092+0000","last_fullsized":"2024-02-23T00:44:05.053092+0000","mapping_epoch":511,"log_start":"62'1","ondisk_log_start":"62'1","created":60,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:43:54.978040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:43:54.978040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":745,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.263046+0000","last_change":"2024-02-23T00:42:53.387347+0000","last_active":"2024-02-23T00:44:04.263046+0000","last_peered":"2024-02-23T00:44:04.263046+0000","last_clean":"2024-02-23T00:44:04.263046+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:44:04.263046+0000","last_undegraded":"2024-02-23T00:44:04.263046+0000","last_fullsized":"2024-02-23T00:44:04.263046+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.387251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.387251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":825,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262050+0000","last_change":"2024-02-23T00:43:00.633273+0000","last_active":"2024-02-23T00:44:06.262050+0000","last_peered":"2024-02-23T00:44:06.262050+0000","last_clean":"2024-02-23T00:44:06.262050+0000","last_became_active":"2024-02-23T00:42:39.393007+0000","last_became_peered":"2024-02-23T00:42:39.393007+0000","last_unstale":"2024-02-23T00:44:06.262050+0000","last_undegraded":"2024-02-23T00:44:06.262050+0000","last_fullsized":"2024-02-23T00:44:06.262050+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:00.633189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.633189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":146,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262994+0000","last_change":"2024-02-23T00:43:09.611028+0000","last_active":"2024-02-23T00:44:06.262994+0000","last_peered":"2024-02-23T00:44:06.262994+0000","last_clean":"2024-02-23T00:44:06.262994+0000","last_became_active":"2024-02-23T00:41:37.239608+0000","last_became_peered":"2024-02-23T00:41:37.239608+0000","last_unstale":"2024-02-23T00:44:06.262994+0000","last_undegraded":"2024-02-23T00:44:06.262994+0000","last_fullsized":"2024-02-23T00:44:06.262994+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:09.610960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:09.610960+0000","last_clean_scrub_stamp":"2024-02-23T00:43:09.610960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":1128,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.261988+0000","last_change":"2024-02-23T00:42:54.354917+0000","last_active":"2024-02-23T00:44:05.261988+0000","last_peered":"2024-02-23T00:44:05.261988+0000","last_clean":"2024-02-23T00:44:05.261988+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:44:05.261988+0000","last_undegraded":"2024-02-23T00:44:05.261988+0000","last_fullsized":"2024-02-23T00:44:05.261988+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'48","last_scrub_stamp":"2024-02-23T00:42:54.354848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:42:54.354848+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"601'2","reported_seq":420,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542869+0000","last_change":"2024-02-23T00:43:00.296517+0000","last_active":"2024-02-23T00:44:04.542869+0000","last_peered":"2024-02-23T00:44:04.542869+0000","last_clean":"2024-02-23T00:44:04.542869+0000","last_became_active":"2024-02-23T00:36:17.101805+0000","last_became_peered":"2024-02-23T00:36:17.101805+0000","last_unstale":"2024-02-23T00:44:04.542869+0000","last_undegraded":"2024-02-23T00:44:04.542869+0000","last_fullsized":"2024-02-23T00:44:04.542869+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'2","last_scrub_stamp":"2024-02-23T00:43:00.296450+0000","last_deep_scrub":"601'2","last_deep_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.296450+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":348,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.262987+0000","last_change":"2024-02-23T00:42:26.240735+0000","last_active":"2024-02-23T00:44:04.262987+0000","last_peered":"2024-02-23T00:44:04.262987+0000","last_clean":"2024-02-23T00:44:04.262987+0000","last_became_active":"2024-02-23T00:41:17.220294+0000","last_became_peered":"2024-02-23T00:41:17.220294+0000","last_unstale":"2024-02-23T00:44:04.262987+0000","last_undegraded":"2024-02-23T00:44:04.262987+0000","last_fullsized":"2024-02-23T00:44:04.262987+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.240647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.240647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":750,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.267421+0000","last_change":"2024-02-23T00:43:47.431707+0000","last_active":"2024-02-23T00:44:04.267421+0000","last_peered":"2024-02-23T00:44:04.267421+0000","last_clean":"2024-02-23T00:44:04.267421+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:44:04.267421+0000","last_undegraded":"2024-02-23T00:44:04.267421+0000","last_fullsized":"2024-02-23T00:44:04.267421+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:47.431641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:47.431641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":719,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.546781+0000","last_change":"2024-02-23T00:43:12.060701+0000","last_active":"2024-02-23T00:44:04.546781+0000","last_peered":"2024-02-23T00:44:04.546781+0000","last_clean":"2024-02-23T00:44:04.546781+0000","last_became_active":"2024-02-23T00:42:39.399113+0000","last_became_peered":"2024-02-23T00:42:39.399113+0000","last_unstale":"2024-02-23T00:44:04.546781+0000","last_undegraded":"2024-02-23T00:44:04.546781+0000","last_fullsized":"2024-02-23T00:44:04.546781+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:12.060587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:43:12.060587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"783'22","reported_seq":403,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262809+0000","last_change":"2024-02-23T00:43:40.280239+0000","last_active":"2024-02-23T00:44:06.262809+0000","last_peered":"2024-02-23T00:44:06.262809+0000","last_clean":"2024-02-23T00:44:06.262809+0000","last_became_active":"2024-02-23T00:43:40.280015+0000","last_became_peered":"2024-02-23T00:43:40.280015+0000","last_unstale":"2024-02-23T00:44:06.262809+0000","last_undegraded":"2024-02-23T00:44:06.262809+0000","last_fullsized":"2024-02-23T00:44:06.262809+0000","mapping_epoch":782,"log_start":"606'17","ondisk_log_start":"606'17","created":58,"last_epoch_clean":783,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'19","last_scrub_stamp":"2024-02-23T00:40:58.498752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.498752+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"453'26","reported_seq":232,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542194+0000","last_change":"2024-02-23T00:43:58.528120+0000","last_active":"2024-02-23T00:44:04.542194+0000","last_peered":"2024-02-23T00:44:04.542194+0000","last_clean":"2024-02-23T00:44:04.542194+0000","last_became_active":"2024-02-23T00:40:23.194277+0000","last_became_peered":"2024-02-23T00:40:23.194277+0000","last_unstale":"2024-02-23T00:44:04.542194+0000","last_undegraded":"2024-02-23T00:44:04.542194+0000","last_fullsized":"2024-02-23T00:44:04.542194+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:43:58.528032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:58.528032+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":803,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543723+0000","last_change":"2024-02-23T00:43:07.413095+0000","last_active":"2024-02-23T00:44:04.543723+0000","last_peered":"2024-02-23T00:44:04.543723+0000","last_clean":"2024-02-23T00:44:04.543723+0000","last_became_active":"2024-02-23T00:41:37.229003+0000","last_became_peered":"2024-02-23T00:41:37.229003+0000","last_unstale":"2024-02-23T00:44:04.543723+0000","last_undegraded":"2024-02-23T00:44:04.543723+0000","last_fullsized":"2024-02-23T00:44:04.543723+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.413026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.413026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":146,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.261792+0000","last_change":"2024-02-23T00:43:10.611567+0000","last_active":"2024-02-23T00:44:06.261792+0000","last_peered":"2024-02-23T00:44:06.261792+0000","last_clean":"2024-02-23T00:44:06.261792+0000","last_became_active":"2024-02-23T00:41:37.248459+0000","last_became_peered":"2024-02-23T00:41:37.248459+0000","last_unstale":"2024-02-23T00:44:06.261792+0000","last_undegraded":"2024-02-23T00:44:06.261792+0000","last_fullsized":"2024-02-23T00:44:06.261792+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:10.611499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:43:10.611499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":736,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.197081+0000","last_change":"2024-02-23T00:43:16.990810+0000","last_active":"2024-02-23T00:44:05.197081+0000","last_peered":"2024-02-23T00:44:05.197081+0000","last_clean":"2024-02-23T00:44:05.197081+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:44:05.197081+0000","last_undegraded":"2024-02-23T00:44:05.197081+0000","last_fullsized":"2024-02-23T00:44:05.197081+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:16.990733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:43:16.990733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":79,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.051341+0000","last_change":"2024-02-23T00:43:02.723267+0000","last_active":"2024-02-23T00:44:05.051341+0000","last_peered":"2024-02-23T00:44:05.051341+0000","last_clean":"2024-02-23T00:44:05.051341+0000","last_became_active":"2024-02-23T00:42:41.402111+0000","last_became_peered":"2024-02-23T00:42:41.402111+0000","last_unstale":"2024-02-23T00:44:05.051341+0000","last_undegraded":"2024-02-23T00:44:05.051341+0000","last_fullsized":"2024-02-23T00:44:05.051341+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:02.723205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:43:02.723205+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":837,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.270157+0000","last_change":"2024-02-23T00:44:01.381012+0000","last_active":"2024-02-23T00:44:04.270157+0000","last_peered":"2024-02-23T00:44:04.270157+0000","last_clean":"2024-02-23T00:44:04.270157+0000","last_became_active":"2024-02-23T00:41:37.239598+0000","last_became_peered":"2024-02-23T00:41:37.239598+0000","last_unstale":"2024-02-23T00:44:04.270157+0000","last_undegraded":"2024-02-23T00:44:04.270157+0000","last_fullsized":"2024-02-23T00:44:04.270157+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_clean_scrub_stamp":"2024-02-23T00:42:26.380671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":728,"reported_epoch":803,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-23T00:44:04.264368+0000","last_change":"2024-02-23T00:43:43.895272+0000","last_active":"2024-02-23T00:44:04.264368+0000","last_peered":"2024-02-23T00:44:04.264368+0000","last_clean":"2024-02-23T00:38:47.154201+0000","last_became_active":"2024-02-23T00:43:43.894446+0000","last_became_peered":"2024-02-23T00:43:43.894446+0000","last_unstale":"2024-02-23T00:44:04.264368+0000","last_undegraded":"2024-02-23T00:44:04.264368+0000","last_fullsized":"2024-02-23T00:44:04.264368+0000","mapping_epoch":785,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:43:05.364147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:05.364147+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":165,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,1],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":838,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.198380+0000","last_change":"2024-02-23T00:43:01.990077+0000","last_active":"2024-02-23T00:44:05.198380+0000","last_peered":"2024-02-23T00:44:05.198380+0000","last_clean":"2024-02-23T00:44:05.198380+0000","last_became_active":"2024-02-23T00:41:37.238708+0000","last_became_peered":"2024-02-23T00:41:37.238708+0000","last_unstale":"2024-02-23T00:44:05.198380+0000","last_undegraded":"2024-02-23T00:44:05.198380+0000","last_fullsized":"2024-02-23T00:44:05.198380+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:43:01.989978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:01.989978+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":325,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264719+0000","last_change":"2024-02-23T00:41:33.042594+0000","last_active":"2024-02-23T00:44:04.264719+0000","last_peered":"2024-02-23T00:44:04.264719+0000","last_clean":"2024-02-23T00:44:04.264719+0000","last_became_active":"2024-02-23T00:40:17.214288+0000","last_became_peered":"2024-02-23T00:40:17.214288+0000","last_unstale":"2024-02-23T00:44:04.264719+0000","last_undegraded":"2024-02-23T00:44:04.264719+0000","last_fullsized":"2024-02-23T00:44:04.264719+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:33.042520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:41:33.042520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":731,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.546472+0000","last_change":"2024-02-23T00:43:58.934152+0000","last_active":"2024-02-23T00:44:04.546472+0000","last_peered":"2024-02-23T00:44:04.546472+0000","last_clean":"2024-02-23T00:44:04.546472+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:44:04.546472+0000","last_undegraded":"2024-02-23T00:44:04.546472+0000","last_fullsized":"2024-02-23T00:44:04.546472+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:58.934069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:58.934069+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":79,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.051003+0000","last_change":"2024-02-23T00:43:04.777343+0000","last_active":"2024-02-23T00:44:05.051003+0000","last_peered":"2024-02-23T00:44:05.051003+0000","last_clean":"2024-02-23T00:44:05.051003+0000","last_became_active":"2024-02-23T00:42:41.399155+0000","last_became_peered":"2024-02-23T00:42:41.399155+0000","last_unstale":"2024-02-23T00:44:05.051003+0000","last_undegraded":"2024-02-23T00:44:05.051003+0000","last_fullsized":"2024-02-23T00:44:05.051003+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:04.777269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:04.777269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":582,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262680+0000","last_change":"2024-02-23T00:43:54.474016+0000","last_active":"2024-02-23T00:44:06.262680+0000","last_peered":"2024-02-23T00:44:06.262680+0000","last_clean":"2024-02-23T00:44:06.262680+0000","last_became_active":"2024-02-23T00:41:38.217979+0000","last_became_peered":"2024-02-23T00:41:38.217979+0000","last_unstale":"2024-02-23T00:44:06.262680+0000","last_undegraded":"2024-02-23T00:44:06.262680+0000","last_fullsized":"2024-02-23T00:44:06.262680+0000","mapping_epoch":678,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:43:54.473954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:54.473954+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":817,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.270099+0000","last_change":"2024-02-23T00:43:57.420063+0000","last_active":"2024-02-23T00:44:04.270099+0000","last_peered":"2024-02-23T00:44:04.270099+0000","last_clean":"2024-02-23T00:44:04.270099+0000","last_became_active":"2024-02-23T00:41:37.238750+0000","last_became_peered":"2024-02-23T00:41:37.238750+0000","last_unstale":"2024-02-23T00:44:04.270099+0000","last_undegraded":"2024-02-23T00:44:04.270099+0000","last_fullsized":"2024-02-23T00:44:04.270099+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:57.419998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:57.419998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":312,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264379+0000","last_change":"2024-02-23T00:42:20.334112+0000","last_active":"2024-02-23T00:44:04.264379+0000","last_peered":"2024-02-23T00:44:04.264379+0000","last_clean":"2024-02-23T00:44:04.264379+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:44:04.264379+0000","last_undegraded":"2024-02-23T00:44:04.264379+0000","last_fullsized":"2024-02-23T00:44:04.264379+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.334013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.334013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":307,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.265369+0000","last_change":"2024-02-23T00:41:48.297291+0000","last_active":"2024-02-23T00:44:04.265369+0000","last_peered":"2024-02-23T00:44:04.265369+0000","last_clean":"2024-02-23T00:44:04.265369+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:44:04.265369+0000","last_undegraded":"2024-02-23T00:44:04.265369+0000","last_fullsized":"2024-02-23T00:44:04.265369+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:48.297220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:48.297220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":205,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.258614+0000","last_change":"2024-02-23T00:43:07.691013+0000","last_active":"2024-02-23T00:44:04.258614+0000","last_peered":"2024-02-23T00:44:04.258614+0000","last_clean":"2024-02-23T00:44:04.258614+0000","last_became_active":"2024-02-23T00:42:41.406233+0000","last_became_peered":"2024-02-23T00:42:41.406233+0000","last_unstale":"2024-02-23T00:44:04.258614+0000","last_undegraded":"2024-02-23T00:44:04.258614+0000","last_fullsized":"2024-02-23T00:44:04.258614+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:43:07.690939+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:43:07.690939+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.690939+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":920,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264023+0000","last_change":"2024-02-23T00:43:00.208080+0000","last_active":"2024-02-23T00:44:04.264023+0000","last_peered":"2024-02-23T00:44:04.264023+0000","last_clean":"2024-02-23T00:44:04.264023+0000","last_became_active":"2024-02-23T00:40:51.316452+0000","last_became_peered":"2024-02-23T00:40:51.316452+0000","last_unstale":"2024-02-23T00:44:04.264023+0000","last_undegraded":"2024-02-23T00:44:04.264023+0000","last_fullsized":"2024-02-23T00:44:04.264023+0000","mapping_epoch":631,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:43:00.208019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.208019+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":790,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.259009+0000","last_change":"2024-02-23T00:42:19.655247+0000","last_active":"2024-02-23T00:44:04.259009+0000","last_peered":"2024-02-23T00:44:04.259009+0000","last_clean":"2024-02-23T00:44:04.259009+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:44:04.259009+0000","last_undegraded":"2024-02-23T00:44:04.259009+0000","last_fullsized":"2024-02-23T00:44:04.259009+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.655188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.655188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":791,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.547117+0000","last_change":"2024-02-23T00:43:48.961981+0000","last_active":"2024-02-23T00:44:04.547117+0000","last_peered":"2024-02-23T00:44:04.547117+0000","last_clean":"2024-02-23T00:44:04.547117+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:44:04.547117+0000","last_undegraded":"2024-02-23T00:44:04.547117+0000","last_fullsized":"2024-02-23T00:44:04.547117+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:48.961905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:48.961905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":755,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.260170+0000","last_change":"2024-02-23T00:43:04.619611+0000","last_active":"2024-02-23T00:44:04.260170+0000","last_peered":"2024-02-23T00:44:04.260170+0000","last_clean":"2024-02-23T00:44:04.260170+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:44:04.260170+0000","last_undegraded":"2024-02-23T00:44:04.260170+0000","last_fullsized":"2024-02-23T00:44:04.260170+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:04.619508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:04.619508+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":79,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.051549+0000","last_change":"2024-02-23T00:42:59.678005+0000","last_active":"2024-02-23T00:44:05.051549+0000","last_peered":"2024-02-23T00:44:05.051549+0000","last_clean":"2024-02-23T00:44:05.051549+0000","last_became_active":"2024-02-23T00:42:41.402543+0000","last_became_peered":"2024-02-23T00:42:41.402543+0000","last_unstale":"2024-02-23T00:44:05.051549+0000","last_undegraded":"2024-02-23T00:44:05.051549+0000","last_fullsized":"2024-02-23T00:44:05.051549+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:59.677899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.677899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"688'39","reported_seq":330,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.265236+0000","last_change":"2024-02-23T00:41:49.338035+0000","last_active":"2024-02-23T00:44:04.265236+0000","last_peered":"2024-02-23T00:44:04.265236+0000","last_clean":"2024-02-23T00:44:04.265236+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:44:04.265236+0000","last_undegraded":"2024-02-23T00:44:04.265236+0000","last_fullsized":"2024-02-23T00:44:04.265236+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'39","last_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_deep_scrub":"688'39","last_deep_scrub_stamp":"2024-02-23T00:41:49.337969+0000","last_clean_scrub_stamp":"2024-02-23T00:41:49.337969+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":808,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.545782+0000","last_change":"2024-02-23T00:43:50.959065+0000","last_active":"2024-02-23T00:44:04.545782+0000","last_peered":"2024-02-23T00:44:04.545782+0000","last_clean":"2024-02-23T00:44:04.545782+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:44:04.545782+0000","last_undegraded":"2024-02-23T00:44:04.545782+0000","last_fullsized":"2024-02-23T00:44:04.545782+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:50.958985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:43:50.958985+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"681'12","reported_seq":1012,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.262508+0000","last_change":"2024-02-23T00:42:59.230304+0000","last_active":"2024-02-23T00:44:05.262508+0000","last_peered":"2024-02-23T00:44:05.262508+0000","last_clean":"2024-02-23T00:44:05.262508+0000","last_became_active":"2024-02-23T00:41:37.228441+0000","last_became_peered":"2024-02-23T00:41:37.228441+0000","last_unstale":"2024-02-23T00:44:05.262508+0000","last_undegraded":"2024-02-23T00:44:05.262508+0000","last_fullsized":"2024-02-23T00:44:05.262508+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'12","last_scrub_stamp":"2024-02-23T00:42:59.230220+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.230220+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":726,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.547326+0000","last_change":"2024-02-23T00:42:17.056253+0000","last_active":"2024-02-23T00:44:04.547326+0000","last_peered":"2024-02-23T00:44:04.547326+0000","last_clean":"2024-02-23T00:44:04.547326+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:44:04.547326+0000","last_undegraded":"2024-02-23T00:44:04.547326+0000","last_fullsized":"2024-02-23T00:44:04.547326+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:17.056153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:17.056153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":80,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.261956+0000","last_change":"2024-02-23T00:43:07.568838+0000","last_active":"2024-02-23T00:44:06.261956+0000","last_peered":"2024-02-23T00:44:06.261956+0000","last_clean":"2024-02-23T00:44:06.261956+0000","last_became_active":"2024-02-23T00:42:41.402683+0000","last_became_peered":"2024-02-23T00:42:41.402683+0000","last_unstale":"2024-02-23T00:44:06.261956+0000","last_undegraded":"2024-02-23T00:44:06.261956+0000","last_fullsized":"2024-02-23T00:44:06.261956+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.568782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.568782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":862,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.546856+0000","last_change":"2024-02-23T00:42:28.967292+0000","last_active":"2024-02-23T00:44:04.546856+0000","last_peered":"2024-02-23T00:44:04.546856+0000","last_clean":"2024-02-23T00:44:04.546856+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:44:04.546856+0000","last_undegraded":"2024-02-23T00:44:04.546856+0000","last_fullsized":"2024-02-23T00:44:04.546856+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:42:28.967208+0000","last_deep_scrub":"144'20","last_deep_scrub_stamp":"2024-02-23T00:31:35.423227+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.967208+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":146,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262143+0000","last_change":"2024-02-23T00:43:11.571763+0000","last_active":"2024-02-23T00:44:06.262143+0000","last_peered":"2024-02-23T00:44:06.262143+0000","last_clean":"2024-02-23T00:44:06.262143+0000","last_became_active":"2024-02-23T00:41:37.235746+0000","last_became_peered":"2024-02-23T00:41:37.235746+0000","last_unstale":"2024-02-23T00:44:06.262143+0000","last_undegraded":"2024-02-23T00:44:06.262143+0000","last_fullsized":"2024-02-23T00:44:06.262143+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:11.571661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:11.571661+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":375,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.264579+0000","last_change":"2024-02-23T00:40:50.453642+0000","last_active":"2024-02-23T00:44:05.264579+0000","last_peered":"2024-02-23T00:44:05.264579+0000","last_clean":"2024-02-23T00:44:05.264579+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:44:05.264579+0000","last_undegraded":"2024-02-23T00:44:05.264579+0000","last_fullsized":"2024-02-23T00:44:05.264579+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'11","last_scrub_stamp":"2024-02-23T00:40:50.453579+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:40:50.453579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":307,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264218+0000","last_change":"2024-02-23T00:41:03.320478+0000","last_active":"2024-02-23T00:44:04.264218+0000","last_peered":"2024-02-23T00:44:04.264218+0000","last_clean":"2024-02-23T00:44:04.264218+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:44:04.264218+0000","last_undegraded":"2024-02-23T00:44:04.264218+0000","last_fullsized":"2024-02-23T00:44:04.264218+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:03.320387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:03.320387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":79,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.052140+0000","last_change":"2024-02-23T00:43:05.740185+0000","last_active":"2024-02-23T00:44:05.052140+0000","last_peered":"2024-02-23T00:44:05.052140+0000","last_clean":"2024-02-23T00:44:05.052140+0000","last_became_active":"2024-02-23T00:42:41.398941+0000","last_became_peered":"2024-02-23T00:42:41.398941+0000","last_unstale":"2024-02-23T00:44:05.052140+0000","last_undegraded":"2024-02-23T00:44:05.052140+0000","last_fullsized":"2024-02-23T00:44:05.052140+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:05.740112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:43:05.740112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":804,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543493+0000","last_change":"2024-02-23T00:44:00.526974+0000","last_active":"2024-02-23T00:44:04.543493+0000","last_peered":"2024-02-23T00:44:04.543493+0000","last_clean":"2024-02-23T00:44:04.543493+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:44:04.543493+0000","last_undegraded":"2024-02-23T00:44:04.543493+0000","last_fullsized":"2024-02-23T00:44:04.543493+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'15","last_scrub_stamp":"2024-02-23T00:44:00.526904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:44:00.526904+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":817,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.259750+0000","last_change":"2024-02-23T00:43:16.698802+0000","last_active":"2024-02-23T00:44:04.259750+0000","last_peered":"2024-02-23T00:44:04.259750+0000","last_clean":"2024-02-23T00:44:04.259750+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:44:04.259750+0000","last_undegraded":"2024-02-23T00:44:04.259750+0000","last_fullsized":"2024-02-23T00:44:04.259750+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:16.698726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:16.698726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":331,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264642+0000","last_change":"2024-02-23T00:42:27.197397+0000","last_active":"2024-02-23T00:44:04.264642+0000","last_peered":"2024-02-23T00:44:04.264642+0000","last_clean":"2024-02-23T00:44:04.264642+0000","last_became_active":"2024-02-23T00:41:37.252612+0000","last_became_peered":"2024-02-23T00:41:37.252612+0000","last_unstale":"2024-02-23T00:44:04.264642+0000","last_undegraded":"2024-02-23T00:44:04.264642+0000","last_fullsized":"2024-02-23T00:44:04.264642+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_clean_scrub_stamp":"2024-02-23T00:42:27.197290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":760,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.197005+0000","last_change":"2024-02-23T00:43:52.040876+0000","last_active":"2024-02-23T00:44:05.197005+0000","last_peered":"2024-02-23T00:44:05.197005+0000","last_clean":"2024-02-23T00:44:05.197005+0000","last_became_active":"2024-02-23T00:41:37.238185+0000","last_became_peered":"2024-02-23T00:41:37.238185+0000","last_unstale":"2024-02-23T00:44:05.197005+0000","last_undegraded":"2024-02-23T00:44:05.197005+0000","last_fullsized":"2024-02-23T00:44:05.197005+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:52.040805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:52.040805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"78'6","reported_seq":79,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.051244+0000","last_change":"2024-02-23T00:43:00.726511+0000","last_active":"2024-02-23T00:44:05.051244+0000","last_peered":"2024-02-23T00:44:05.051244+0000","last_clean":"2024-02-23T00:44:05.051244+0000","last_became_active":"2024-02-23T00:42:41.397727+0000","last_became_peered":"2024-02-23T00:42:41.397727+0000","last_unstale":"2024-02-23T00:44:05.051244+0000","last_undegraded":"2024-02-23T00:44:05.051244+0000","last_fullsized":"2024-02-23T00:44:05.051244+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:43:00.726434+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.726434+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":839,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.197560+0000","last_change":"2024-02-23T00:43:06.975177+0000","last_active":"2024-02-23T00:44:05.197560+0000","last_peered":"2024-02-23T00:44:05.197560+0000","last_clean":"2024-02-23T00:44:05.197560+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:44:05.197560+0000","last_undegraded":"2024-02-23T00:44:05.197560+0000","last_fullsized":"2024-02-23T00:44:05.197560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:43:06.975089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.975089+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":800,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542544+0000","last_change":"2024-02-23T00:43:18.462532+0000","last_active":"2024-02-23T00:44:04.542544+0000","last_peered":"2024-02-23T00:44:04.542544+0000","last_clean":"2024-02-23T00:44:04.542544+0000","last_became_active":"2024-02-23T00:41:37.229726+0000","last_became_peered":"2024-02-23T00:41:37.229726+0000","last_unstale":"2024-02-23T00:44:04.542544+0000","last_undegraded":"2024-02-23T00:44:04.542544+0000","last_fullsized":"2024-02-23T00:44:04.542544+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:18.462472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:43:18.462472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":790,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543574+0000","last_change":"2024-02-23T00:41:35.138434+0000","last_active":"2024-02-23T00:44:04.543574+0000","last_peered":"2024-02-23T00:44:04.543574+0000","last_clean":"2024-02-23T00:44:04.543574+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:44:04.543574+0000","last_undegraded":"2024-02-23T00:44:04.543574+0000","last_fullsized":"2024-02-23T00:44:04.543574+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:35.138319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:41:35.138319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"98'7","reported_seq":786,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.259289+0000","last_change":"2024-02-23T00:43:53.860943+0000","last_active":"2024-02-23T00:44:04.259289+0000","last_peered":"2024-02-23T00:44:04.259289+0000","last_clean":"2024-02-23T00:44:04.259289+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:44:04.259289+0000","last_undegraded":"2024-02-23T00:44:04.259289+0000","last_fullsized":"2024-02-23T00:44:04.259289+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'7","last_scrub_stamp":"2024-02-23T00:43:53.860835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:53.860835+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":333,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.263658+0000","last_change":"2024-02-23T00:43:42.433192+0000","last_active":"2024-02-23T00:44:04.263658+0000","last_peered":"2024-02-23T00:44:04.263658+0000","last_clean":"2024-02-23T00:44:04.263658+0000","last_became_active":"2024-02-23T00:43:42.256788+0000","last_became_peered":"2024-02-23T00:43:42.256788+0000","last_unstale":"2024-02-23T00:44:04.263658+0000","last_undegraded":"2024-02-23T00:44:04.263658+0000","last_fullsized":"2024-02-23T00:44:04.263658+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'10","last_scrub_stamp":"2024-02-23T00:43:04.142193+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:43:04.142193+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":782,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.547216+0000","last_change":"2024-02-23T00:44:00.940789+0000","last_active":"2024-02-23T00:44:04.547216+0000","last_peered":"2024-02-23T00:44:04.547216+0000","last_clean":"2024-02-23T00:44:04.547216+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:44:04.547216+0000","last_undegraded":"2024-02-23T00:44:04.547216+0000","last_fullsized":"2024-02-23T00:44:04.547216+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:00.940728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:44:00.940728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":786,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.258925+0000","last_change":"2024-02-23T00:43:14.754285+0000","last_active":"2024-02-23T00:44:04.258925+0000","last_peered":"2024-02-23T00:44:04.258925+0000","last_clean":"2024-02-23T00:44:04.258925+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:44:04.258925+0000","last_undegraded":"2024-02-23T00:44:04.258925+0000","last_fullsized":"2024-02-23T00:44:04.258925+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:14.754203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:14.754203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":723,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543651+0000","last_change":"2024-02-23T00:41:44.196162+0000","last_active":"2024-02-23T00:44:04.543651+0000","last_peered":"2024-02-23T00:44:04.543651+0000","last_clean":"2024-02-23T00:44:04.543651+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:44:04.543651+0000","last_undegraded":"2024-02-23T00:44:04.543651+0000","last_fullsized":"2024-02-23T00:44:04.543651+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.196088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.196088+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":79,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.547008+0000","last_change":"2024-02-23T00:43:17.077666+0000","last_active":"2024-02-23T00:44:04.547008+0000","last_peered":"2024-02-23T00:44:04.547008+0000","last_clean":"2024-02-23T00:44:04.547008+0000","last_became_active":"2024-02-23T00:42:41.391484+0000","last_became_peered":"2024-02-23T00:42:41.391484+0000","last_unstale":"2024-02-23T00:44:04.547008+0000","last_undegraded":"2024-02-23T00:44:04.547008+0000","last_fullsized":"2024-02-23T00:44:04.547008+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:17.077581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:17.077581+0000","last_clean_scrub_stamp":"2024-02-23T00:43:17.077581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":324,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264825+0000","last_change":"2024-02-23T00:40:43.233713+0000","last_active":"2024-02-23T00:44:04.264825+0000","last_peered":"2024-02-23T00:44:04.264825+0000","last_clean":"2024-02-23T00:44:04.264825+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:44:04.264825+0000","last_undegraded":"2024-02-23T00:44:04.264825+0000","last_fullsized":"2024-02-23T00:44:04.264825+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:40:43.233609+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:40:43.233609+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":794,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.260512+0000","last_change":"2024-02-23T00:44:03.840950+0000","last_active":"2024-02-23T00:44:04.260512+0000","last_peered":"2024-02-23T00:44:04.260512+0000","last_clean":"2024-02-23T00:44:04.260512+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:44:04.260512+0000","last_undegraded":"2024-02-23T00:44:04.260512+0000","last_fullsized":"2024-02-23T00:44:04.260512+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:03.840880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:25.724375+0000","last_clean_scrub_stamp":"2024-02-23T00:44:03.840880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":44,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.263111+0000","last_change":"2024-02-23T00:43:21.499211+0000","last_active":"2024-02-23T00:44:04.263111+0000","last_peered":"2024-02-23T00:44:04.263111+0000","last_clean":"2024-02-23T00:44:04.263111+0000","last_became_active":"2024-02-23T00:43:21.498988+0000","last_became_peered":"2024-02-23T00:43:21.498988+0000","last_unstale":"2024-02-23T00:44:04.263111+0000","last_undegraded":"2024-02-23T00:44:04.263111+0000","last_fullsized":"2024-02-23T00:44:04.263111+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:44.520415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:41:44.520415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":307,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264604+0000","last_change":"2024-02-23T00:41:37.096978+0000","last_active":"2024-02-23T00:44:04.264604+0000","last_peered":"2024-02-23T00:44:04.264604+0000","last_clean":"2024-02-23T00:44:04.264604+0000","last_became_active":"2024-02-23T00:38:49.180188+0000","last_became_peered":"2024-02-23T00:38:49.180188+0000","last_unstale":"2024-02-23T00:44:04.264604+0000","last_undegraded":"2024-02-23T00:44:04.264604+0000","last_fullsized":"2024-02-23T00:44:04.264604+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:37.096906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:37.096906+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":701,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262588+0000","last_change":"2024-02-23T00:42:53.560946+0000","last_active":"2024-02-23T00:44:06.262588+0000","last_peered":"2024-02-23T00:44:06.262588+0000","last_clean":"2024-02-23T00:44:06.262588+0000","last_became_active":"2024-02-23T00:42:10.240911+0000","last_became_peered":"2024-02-23T00:42:10.240911+0000","last_unstale":"2024-02-23T00:44:06.262588+0000","last_undegraded":"2024-02-23T00:44:06.262588+0000","last_fullsized":"2024-02-23T00:44:06.262588+0000","mapping_epoch":707,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:42:53.560884+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.560884+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":786,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.545628+0000","last_change":"2024-02-23T00:43:05.149002+0000","last_active":"2024-02-23T00:44:04.545628+0000","last_peered":"2024-02-23T00:44:04.545628+0000","last_clean":"2024-02-23T00:44:04.545628+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:44:04.545628+0000","last_undegraded":"2024-02-23T00:44:04.545628+0000","last_fullsized":"2024-02-23T00:44:04.545628+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:05.148931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:55.816155+0000","last_clean_scrub_stamp":"2024-02-23T00:43:05.148931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":999,"reported_epoch":804,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.270461+0000","last_change":"2024-02-23T00:42:59.626715+0000","last_active":"2024-02-23T00:44:05.270461+0000","last_peered":"2024-02-23T00:44:05.270461+0000","last_clean":"2024-02-23T00:44:05.270461+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:44:05.270461+0000","last_undegraded":"2024-02-23T00:44:05.270461+0000","last_fullsized":"2024-02-23T00:44:05.270461+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:42:59.626636+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:42:59.626636+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":780,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.053738+0000","last_change":"2024-02-23T00:44:05.053687+0000","last_active":"2024-02-23T00:44:05.053738+0000","last_peered":"2024-02-23T00:44:05.053738+0000","last_clean":"2024-02-23T00:44:05.053738+0000","last_became_active":"2024-02-23T00:41:37.229426+0000","last_became_peered":"2024-02-23T00:41:37.229426+0000","last_unstale":"2024-02-23T00:44:05.053738+0000","last_undegraded":"2024-02-23T00:44:05.053738+0000","last_fullsized":"2024-02-23T00:44:05.053738+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:05.053579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:44:05.053579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":998,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.260376+0000","last_change":"2024-02-23T00:44:00.802127+0000","last_active":"2024-02-23T00:44:04.260376+0000","last_peered":"2024-02-23T00:44:04.260376+0000","last_clean":"2024-02-23T00:44:04.260376+0000","last_became_active":"2024-02-23T00:41:55.327964+0000","last_became_peered":"2024-02-23T00:41:55.327964+0000","last_unstale":"2024-02-23T00:44:04.260376+0000","last_undegraded":"2024-02-23T00:44:04.260376+0000","last_fullsized":"2024-02-23T00:44:04.260376+0000","mapping_epoch":692,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:44:00.802063+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:44:00.802063+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":331,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264686+0000","last_change":"2024-02-23T00:42:14.278845+0000","last_active":"2024-02-23T00:44:04.264686+0000","last_peered":"2024-02-23T00:44:04.264686+0000","last_clean":"2024-02-23T00:44:04.264686+0000","last_became_active":"2024-02-23T00:41:37.228694+0000","last_became_peered":"2024-02-23T00:41:37.228694+0000","last_unstale":"2024-02-23T00:44:04.264686+0000","last_undegraded":"2024-02-23T00:44:04.264686+0000","last_fullsized":"2024-02-23T00:44:04.264686+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:14.278782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:42:14.278782+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":141,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.262636+0000","last_change":"2024-02-23T00:42:55.607417+0000","last_active":"2024-02-23T00:44:06.262636+0000","last_peered":"2024-02-23T00:44:06.262636+0000","last_clean":"2024-02-23T00:44:06.262636+0000","last_became_active":"2024-02-23T00:41:37.247946+0000","last_became_peered":"2024-02-23T00:41:37.247946+0000","last_unstale":"2024-02-23T00:44:06.262636+0000","last_undegraded":"2024-02-23T00:44:06.262636+0000","last_fullsized":"2024-02-23T00:44:06.262636+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:55.607352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:42:55.607352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":737,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.053193+0000","last_change":"2024-02-23T00:43:51.994856+0000","last_active":"2024-02-23T00:44:05.053193+0000","last_peered":"2024-02-23T00:44:05.053193+0000","last_clean":"2024-02-23T00:44:05.053193+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:44:05.053193+0000","last_undegraded":"2024-02-23T00:44:05.053193+0000","last_fullsized":"2024-02-23T00:44:05.053193+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:51.994771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:51.994771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":928,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.266580+0000","last_change":"2024-02-23T00:43:56.415990+0000","last_active":"2024-02-23T00:44:04.266580+0000","last_peered":"2024-02-23T00:44:04.266580+0000","last_clean":"2024-02-23T00:44:04.266580+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:44:04.266580+0000","last_undegraded":"2024-02-23T00:44:04.266580+0000","last_fullsized":"2024-02-23T00:44:04.266580+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:43:56.415913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:43:56.415913+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":798,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.261033+0000","last_change":"2024-02-23T00:43:51.834595+0000","last_active":"2024-02-23T00:44:04.261033+0000","last_peered":"2024-02-23T00:44:04.261033+0000","last_clean":"2024-02-23T00:44:04.261033+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:44:04.261033+0000","last_undegraded":"2024-02-23T00:44:04.261033+0000","last_fullsized":"2024-02-23T00:44:04.261033+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:51.834514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:43:51.834514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":798,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542690+0000","last_change":"2024-02-23T00:43:11.402457+0000","last_active":"2024-02-23T00:44:04.542690+0000","last_peered":"2024-02-23T00:44:04.542690+0000","last_clean":"2024-02-23T00:44:04.542690+0000","last_became_active":"2024-02-23T00:41:37.229323+0000","last_became_peered":"2024-02-23T00:41:37.229323+0000","last_unstale":"2024-02-23T00:44:04.542690+0000","last_undegraded":"2024-02-23T00:44:04.542690+0000","last_fullsized":"2024-02-23T00:44:04.542690+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:11.402342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:43:11.402342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":314,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.262829+0000","last_change":"2024-02-23T00:43:07.162145+0000","last_active":"2024-02-23T00:44:04.262829+0000","last_peered":"2024-02-23T00:44:04.262829+0000","last_clean":"2024-02-23T00:44:04.262829+0000","last_became_active":"2024-02-23T00:38:49.226210+0000","last_became_peered":"2024-02-23T00:38:49.226210+0000","last_unstale":"2024-02-23T00:44:04.262829+0000","last_undegraded":"2024-02-23T00:44:04.262829+0000","last_fullsized":"2024-02-23T00:44:04.262829+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.162083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.162083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":891,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.262458+0000","last_change":"2024-02-23T00:40:58.399481+0000","last_active":"2024-02-23T00:44:04.262458+0000","last_peered":"2024-02-23T00:44:04.262458+0000","last_clean":"2024-02-23T00:44:04.262458+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:44:04.262458+0000","last_undegraded":"2024-02-23T00:44:04.262458+0000","last_fullsized":"2024-02-23T00:44:04.262458+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_deep_scrub":"453'44","last_deep_scrub_stamp":"2024-02-23T00:40:58.399368+0000","last_clean_scrub_stamp":"2024-02-23T00:40:58.399368+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":312,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264764+0000","last_change":"2024-02-23T00:43:47.036569+0000","last_active":"2024-02-23T00:44:04.264764+0000","last_peered":"2024-02-23T00:44:04.264764+0000","last_clean":"2024-02-23T00:44:04.264764+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:44:04.264764+0000","last_undegraded":"2024-02-23T00:44:04.264764+0000","last_fullsized":"2024-02-23T00:44:04.264764+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:47.036488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:47.036488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":617,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.265047+0000","last_change":"2024-02-23T00:41:50.257038+0000","last_active":"2024-02-23T00:44:04.265047+0000","last_peered":"2024-02-23T00:44:04.265047+0000","last_clean":"2024-02-23T00:44:04.265047+0000","last_became_active":"2024-02-23T00:41:50.256751+0000","last_became_peered":"2024-02-23T00:41:50.256751+0000","last_unstale":"2024-02-23T00:44:04.265047+0000","last_undegraded":"2024-02-23T00:44:04.265047+0000","last_fullsized":"2024-02-23T00:44:04.265047+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:07.610329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:41:07.610329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":726,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.547951+0000","last_change":"2024-02-23T00:43:07.133088+0000","last_active":"2024-02-23T00:44:04.547951+0000","last_peered":"2024-02-23T00:44:04.547951+0000","last_clean":"2024-02-23T00:44:04.547951+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:44:04.547951+0000","last_undegraded":"2024-02-23T00:44:04.547951+0000","last_fullsized":"2024-02-23T00:44:04.547951+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:07.133007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:07.133007+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":996,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.262649+0000","last_change":"2024-02-23T00:43:52.981866+0000","last_active":"2024-02-23T00:44:04.262649+0000","last_peered":"2024-02-23T00:44:04.262649+0000","last_clean":"2024-02-23T00:44:04.262649+0000","last_became_active":"2024-02-23T00:41:40.215919+0000","last_became_peered":"2024-02-23T00:41:40.215919+0000","last_unstale":"2024-02-23T00:44:04.262649+0000","last_undegraded":"2024-02-23T00:44:04.262649+0000","last_fullsized":"2024-02-23T00:44:04.262649+0000","mapping_epoch":680,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:43:52.981794+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:43:52.981794+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":499,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.198199+0000","last_change":"2024-02-23T00:43:47.010319+0000","last_active":"2024-02-23T00:44:05.198199+0000","last_peered":"2024-02-23T00:44:05.198199+0000","last_clean":"2024-02-23T00:44:05.198199+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:44:05.198199+0000","last_undegraded":"2024-02-23T00:44:05.198199+0000","last_fullsized":"2024-02-23T00:44:05.198199+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:47.010235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:43:47.010235+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":792,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.546839+0000","last_change":"2024-02-23T00:43:00.260324+0000","last_active":"2024-02-23T00:44:04.546839+0000","last_peered":"2024-02-23T00:44:04.546839+0000","last_clean":"2024-02-23T00:44:04.546839+0000","last_became_active":"2024-02-23T00:41:37.228310+0000","last_became_peered":"2024-02-23T00:41:37.228310+0000","last_unstale":"2024-02-23T00:44:04.546839+0000","last_undegraded":"2024-02-23T00:44:04.546839+0000","last_fullsized":"2024-02-23T00:44:04.546839+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:00.260242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:00.260242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":737,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.050529+0000","last_change":"2024-02-23T00:42:19.323714+0000","last_active":"2024-02-23T00:44:05.050529+0000","last_peered":"2024-02-23T00:44:05.050529+0000","last_clean":"2024-02-23T00:44:05.050529+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:44:05.050529+0000","last_undegraded":"2024-02-23T00:44:05.050529+0000","last_fullsized":"2024-02-23T00:44:05.050529+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:19.323643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:19.323643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":322,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264499+0000","last_change":"2024-02-23T00:41:50.256190+0000","last_active":"2024-02-23T00:44:04.264499+0000","last_peered":"2024-02-23T00:44:04.264499+0000","last_clean":"2024-02-23T00:44:04.264499+0000","last_became_active":"2024-02-23T00:41:50.255862+0000","last_became_peered":"2024-02-23T00:41:50.255862+0000","last_unstale":"2024-02-23T00:44:04.264499+0000","last_undegraded":"2024-02-23T00:44:04.264499+0000","last_fullsized":"2024-02-23T00:44:04.264499+0000","mapping_epoch":688,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:41:00.643135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:00.643135+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":808,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.052275+0000","last_change":"2024-02-23T00:42:56.638098+0000","last_active":"2024-02-23T00:44:05.052275+0000","last_peered":"2024-02-23T00:44:05.052275+0000","last_clean":"2024-02-23T00:44:05.052275+0000","last_became_active":"2024-02-23T00:41:37.230054+0000","last_became_peered":"2024-02-23T00:41:37.230054+0000","last_unstale":"2024-02-23T00:44:05.052275+0000","last_undegraded":"2024-02-23T00:44:05.052275+0000","last_fullsized":"2024-02-23T00:44:05.052275+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:56.638011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:42:56.638011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":790,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.051926+0000","last_change":"2024-02-23T00:44:04.086087+0000","last_active":"2024-02-23T00:44:05.051926+0000","last_peered":"2024-02-23T00:44:05.051926+0000","last_clean":"2024-02-23T00:44:05.051926+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:44:05.051926+0000","last_undegraded":"2024-02-23T00:44:05.051926+0000","last_fullsized":"2024-02-23T00:44:05.051926+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:04.086021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.307631+0000","last_clean_scrub_stamp":"2024-02-23T00:44:04.086021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":747,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.542386+0000","last_change":"2024-02-23T00:41:47.254406+0000","last_active":"2024-02-23T00:44:04.542386+0000","last_peered":"2024-02-23T00:44:04.542386+0000","last_clean":"2024-02-23T00:44:04.542386+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:44:04.542386+0000","last_undegraded":"2024-02-23T00:44:04.542386+0000","last_fullsized":"2024-02-23T00:44:04.542386+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:41:47.254328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:41:47.254328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":941,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.196660+0000","last_change":"2024-02-23T00:43:53.069327+0000","last_active":"2024-02-23T00:44:05.196660+0000","last_peered":"2024-02-23T00:44:05.196660+0000","last_clean":"2024-02-23T00:44:05.196660+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:44:05.196660+0000","last_undegraded":"2024-02-23T00:44:05.196660+0000","last_fullsized":"2024-02-23T00:44:05.196660+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:43:53.069248+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:43:53.069248+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":331,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.262907+0000","last_change":"2024-02-23T00:42:53.263490+0000","last_active":"2024-02-23T00:44:04.262907+0000","last_peered":"2024-02-23T00:44:04.262907+0000","last_clean":"2024-02-23T00:44:04.262907+0000","last_became_active":"2024-02-23T00:41:37.241142+0000","last_became_peered":"2024-02-23T00:41:37.241142+0000","last_unstale":"2024-02-23T00:44:04.262907+0000","last_undegraded":"2024-02-23T00:44:04.262907+0000","last_fullsized":"2024-02-23T00:44:04.262907+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:53.263425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:42:53.263425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":441,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.258700+0000","last_change":"2024-02-23T00:43:06.705370+0000","last_active":"2024-02-23T00:44:04.258700+0000","last_peered":"2024-02-23T00:44:04.258700+0000","last_clean":"2024-02-23T00:44:04.258700+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:44:04.258700+0000","last_undegraded":"2024-02-23T00:44:04.258700+0000","last_fullsized":"2024-02-23T00:44:04.258700+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:06.705283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:43:06.705283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":219,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:05.050945+0000","last_change":"2024-02-23T00:43:13.796473+0000","last_active":"2024-02-23T00:44:05.050945+0000","last_peered":"2024-02-23T00:44:05.050945+0000","last_clean":"2024-02-23T00:44:05.050945+0000","last_became_active":"2024-02-23T00:40:23.199068+0000","last_became_peered":"2024-02-23T00:40:23.199068+0000","last_unstale":"2024-02-23T00:44:05.050945+0000","last_undegraded":"2024-02-23T00:44:05.050945+0000","last_fullsized":"2024-02-23T00:44:05.050945+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:43:13.796391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:43:13.796391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":1082,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.264799+0000","last_change":"2024-02-23T00:41:41.223239+0000","last_active":"2024-02-23T00:44:04.264799+0000","last_peered":"2024-02-23T00:44:04.264799+0000","last_clean":"2024-02-23T00:44:04.264799+0000","last_became_active":"2024-02-23T00:41:41.222892+0000","last_became_peered":"2024-02-23T00:41:41.222892+0000","last_unstale":"2024-02-23T00:44:04.264799+0000","last_undegraded":"2024-02-23T00:44:04.264799+0000","last_fullsized":"2024-02-23T00:44:04.264799+0000","mapping_epoch":681,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:41:34.076323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:41:34.076323+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":141,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-02-23T00:44:06.265393+0000","last_change":"2024-02-23T00:42:28.612255+0000","last_active":"2024-02-23T00:44:06.265393+0000","last_peered":"2024-02-23T00:44:06.265393+0000","last_clean":"2024-02-23T00:44:06.265393+0000","last_became_active":"2024-02-23T00:41:37.234830+0000","last_became_peered":"2024-02-23T00:41:37.234830+0000","last_unstale":"2024-02-23T00:44:06.265393+0000","last_undegraded":"2024-02-23T00:44:06.265393+0000","last_fullsized":"2024-02-23T00:44:06.265393+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:28.612113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:28.612113+0000","last_clean_scrub_stamp":"2024-02-23T00:42:28.612113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":137,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.259370+0000","last_change":"2024-02-23T00:44:02.822726+0000","last_active":"2024-02-23T00:44:04.259370+0000","last_peered":"2024-02-23T00:44:04.259370+0000","last_clean":"2024-02-23T00:44:04.259370+0000","last_became_active":"2024-02-23T00:41:50.253284+0000","last_became_peered":"2024-02-23T00:41:50.253284+0000","last_unstale":"2024-02-23T00:44:04.259370+0000","last_undegraded":"2024-02-23T00:44:04.259370+0000","last_fullsized":"2024-02-23T00:44:04.259370+0000","mapping_epoch":688,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":689,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:02.822620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:44:02.822620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":746,"reported_epoch":803,"state":"active+clean","last_fresh":"2024-02-23T00:44:04.543055+0000","last_change":"2024-02-23T00:42:20.220171+0000","last_active":"2024-02-23T00:44:04.543055+0000","last_peered":"2024-02-23T00:44:04.543055+0000","last_clean":"2024-02-23T00:44:04.543055+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:44:04.543055+0000","last_undegraded":"2024-02-23T00:44:04.543055+0000","last_fullsized":"2024-02-23T00:44:04.543055+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:42:20.220094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:42:20.220094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":12,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3647,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":212,"num_whiteouts":0,"num_read":659,"num_read_kb":535,"num_write":395,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":3466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":23,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":685,"num_read_kb":685,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":69,"acting":69,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332301,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308444,"kb_used_data":7120,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436708,"statfs":{"total":95995035648,"available":95679188992,"internally_reserved":0,"allocated":7290880,"data_stored":3893709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.625,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.49199999999999999,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.63600000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.47999999999999998,"15min":0.67300000000000004},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.217},"max":{"1min":0.65800000000000003,"5min":1.23,"15min":31.175000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.67500000000000004,"15min":0.73999999999999999},"min":{"1min":0.40999999999999998,"5min":0.29899999999999999,"15min":0.249},"max":{"1min":0.73099999999999998,"5min":11.066000000000001,"15min":29.423999999999999},"last":0.432}]},{"osd":2,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.64100000000000001,"15min":0.71799999999999997},"min":{"1min":0.40400000000000003,"5min":0.28699999999999998,"15min":0.28100000000000003},"max":{"1min":0.72199999999999998,"5min":6.9729999999999999,"15min":16.603999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.61699999999999999,"15min":0.64100000000000001},"min":{"1min":0.47999999999999998,"5min":0.29099999999999998,"15min":0.26600000000000001},"max":{"1min":0.71099999999999997,"5min":5.7229999999999999,"15min":7.5339999999999998},"last":0.53800000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55500000000000005,"15min":1.0069999999999999},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":0.66500000000000004,"5min":3.9900000000000002,"15min":57.729999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.91400000000000003},"min":{"1min":0.435,"5min":0.318,"15min":0.29999999999999999},"max":{"1min":0.67900000000000005,"5min":3.8479999999999999,"15min":27.573},"last":0.46999999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.41499999999999998,"15min":0.41799999999999998},"min":{"1min":0.255,"5min":0.214,"15min":0.214},"max":{"1min":0.61699999999999999,"5min":1.2589999999999999,"15min":1.706},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.43099999999999999,"15min":0.46100000000000002},"min":{"1min":0.32000000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.69399999999999995,"5min":0.74299999999999999,"15min":8.1359999999999992},"last":0.41499999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.68300000000000005,"5min":0.77000000000000002,"15min":1.0209999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46100000000000002,"15min":0.46800000000000003},"min":{"1min":0.29199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67100000000000004,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.48299999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.47299999999999998,"15min":0.48799999999999999},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.70299999999999996,"5min":1.3129999999999999,"15min":2.1339999999999999},"last":0.441},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.48399999999999999,"15min":0.499},"min":{"1min":0.33100000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.73099999999999998,"5min":0.80500000000000005,"15min":5.194},"last":0.57199999999999995}]}]},{"osd":6,"up_from":44,"seq":188978561234,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308516,"kb_used_data":7196,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436636,"statfs":{"total":95995035648,"available":95679115264,"internally_reserved":0,"allocated":7368704,"data_stored":3971816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.73299999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.496},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.502,"15min":0.5},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55000000000000004,"15min":1.024},"min":{"1min":0.28599999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.79500000000000004,"5min":6.056,"15min":59.347999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.77800000000000002,"15min":1.3819999999999999},"min":{"1min":0.36899999999999999,"5min":0.24299999999999999,"15min":0.20999999999999999},"max":{"1min":0.67100000000000004,"5min":18.423999999999999,"15min":87.221000000000004},"last":0.33300000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.78900000000000003,"15min":0.90600000000000003},"min":{"1min":0.39600000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.66100000000000003,"5min":18.381,"15min":25.550999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.67700000000000005,"15min":0.91300000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.70399999999999996,"5min":6.1749999999999998,"15min":40.914000000000001},"last":0.378}]},{"osd":3,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.65200000000000002,"15min":0.85999999999999999},"min":{"1min":0.32500000000000001,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":0.751,"5min":7.1630000000000003,"15min":31.309999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.77300000000000002,"15min":1.4079999999999999},"min":{"1min":0.35799999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.89900000000000002,"5min":18.448,"15min":86.311999999999998},"last":0.39600000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.55200000000000005,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.251,"15min":0.23699999999999999},"max":{"1min":0.66900000000000004,"5min":7.8769999999999998,"15min":7.8769999999999998},"last":0.443},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.502,"15min":0.53500000000000003},"min":{"1min":0.311,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":0.69099999999999995,"5min":6.0300000000000002,"15min":17.718},"last":0.40699999999999997}]},{"osd":5,"last update":"Fri Feb 23 00:43:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.57099999999999995,"15min":0.52500000000000002},"min":{"1min":0.32300000000000001,"5min":0.28499999999999998,"15min":0.254},"max":{"1min":0.86899999999999999,"5min":6.7999999999999998,"15min":8.0990000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52200000000000002,"15min":0.51300000000000001},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.97999999999999998,"5min":6.0780000000000003,"15min":8.1370000000000005},"last":0.55400000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.495,"15min":0.623},"min":{"1min":0.35699999999999998,"5min":0.29199999999999998,"15min":0.27300000000000002},"max":{"1min":1.014,"5min":1.5629999999999999,"15min":17.881},"last":0.435},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.47499999999999998,"15min":0.65200000000000002},"min":{"1min":0.40699999999999997,"5min":0.27700000000000002,"15min":0.26500000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":18.475000000000001},"last":0.48199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575081,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314584,"kb_used_data":9168,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93430568,"statfs":{"total":95995035648,"available":95672901632,"internally_reserved":0,"allocated":9388032,"data_stored":5935356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.40600000000000003,"15min":0.42299999999999999},"min":{"1min":0.20899999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.47399999999999998,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.40699999999999997,"15min":0.41599999999999998},"min":{"1min":0.249,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.53900000000000003,"5min":1.161,"15min":1.161},"last":0.24399999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:43:46 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.373,"15min":0.66700000000000004},"min":{"1min":0.251,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.70799999999999996,"5min":3.9980000000000002,"15min":35.313000000000002},"last":0.22},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.41699999999999998,"15min":0.58199999999999996},"min":{"1min":0.21099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.5549999999999999,"5min":2.7999999999999998,"15min":27.908999999999999},"last":0.23100000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:43:06 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.44900000000000001,"15min":0.64600000000000002},"min":{"1min":0.245,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.52400000000000002,"5min":3.8119999999999998,"15min":21.013999999999999},"last":0.308},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.41399999999999998,"15min":0.63300000000000001},"min":{"1min":0.252,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.47799999999999998,"5min":4.0259999999999998,"15min":29.234999999999999},"last":0.26300000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:43:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58899999999999997,"15min":0.70699999999999996},"min":{"1min":0.42499999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.68300000000000005,"5min":3.4900000000000002,"15min":16.231000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.52300000000000002,"15min":0.71399999999999997},"min":{"1min":0.36799999999999999,"5min":0.29599999999999999,"15min":0.28999999999999998},"max":{"1min":0.66600000000000004,"5min":1.7350000000000001,"15min":29.212},"last":0.39700000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:43:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58799999999999997,"15min":0.72099999999999997},"min":{"1min":0.376,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.93700000000000006,"5min":2.8410000000000002,"15min":16.294},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60999999999999999,"15min":0.68500000000000005},"min":{"1min":0.379,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78900000000000003,"5min":4.3380000000000001,"15min":11.105},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:43:13 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53100000000000003,"15min":0.749},"min":{"1min":0.33700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67700000000000005,"5min":2.4049999999999998,"15min":29.187000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.625,"15min":0.747},"min":{"1min":0.40400000000000003,"5min":0.33500000000000002,"15min":0.312},"max":{"1min":0.81599999999999995,"5min":2.927,"15min":16.369},"last":0.51100000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:43:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64800000000000002,"15min":0.79000000000000004},"min":{"1min":0.46600000000000003,"5min":0.33300000000000002,"15min":0.32000000000000001},"max":{"1min":0.75800000000000001,"5min":5.2119999999999997,"15min":16.395},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.622,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.35599999999999998,"15min":0.34300000000000003},"max":{"1min":0.82699999999999996,"5min":3.7669999999999999,"15min":11.135999999999999},"last":0.498}]}]},{"osd":0,"up_from":659,"seq":2830383448099,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314196,"kb_used_data":7180,"kb_used_omap":85,"kb_used_meta":306922,"kb_avail":93430956,"statfs":{"total":95995035648,"available":95673298944,"internally_reserved":0,"allocated":7352320,"data_stored":3927894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":314288256},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.52500000000000002,"15min":0.55500000000000005},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.60599999999999998,"5min":5.0880000000000001,"15min":5.0880000000000001},"last":0.253},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.40899999999999997,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.53200000000000003,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.311}]},{"osd":2,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.34100000000000003,"15min":0.33900000000000002},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55200000000000005,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.55800000000000005,"15min":0.59199999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56999999999999995,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.36599999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.90800000000000003,"15min":0.996},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55700000000000005,"5min":11.891,"15min":11.891},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":1.095,"15min":1.216},"min":{"1min":0.29099999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.54100000000000004,"5min":15.864000000000001,"15min":15.864000000000001},"last":0.35099999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.53600000000000003,"15min":0.52700000000000002},"min":{"1min":0.42599999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.53600000000000003,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.41599999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.62,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.83699999999999997,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.41699999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":0.505},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.746,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56100000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:43:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64400000000000002,"15min":0.64500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80700000000000005,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61799999999999999,"15min":0.61399999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.61799999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411557,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304380,"kb_used_data":7152,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440772,"statfs":{"total":95995035648,"available":95683350528,"internally_reserved":0,"allocated":7323648,"data_stored":3929492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.30599999999999999,"15min":0.30299999999999999},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":0.43099999999999999,"5min":0.503,"15min":0.503},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.27900000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:43:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.59199999999999997},"min":{"1min":0.245,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.74299999999999999,"5min":0.84399999999999997,"15min":19.324999999999999},"last":0.317},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.752},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.51000000000000001,"5min":0.68500000000000005,"15min":35.052},"last":0.377}]},{"osd":3,"last update":"Fri Feb 23 00:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38100000000000001,"15min":0.72899999999999998},"min":{"1min":0.24099999999999999,"5min":0.193,"15min":0.193},"max":{"1min":0.46999999999999997,"5min":3.944,"15min":43.5},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40699999999999997,"15min":0.68000000000000005},"min":{"1min":0.28199999999999997,"5min":0.22,"15min":0.22},"max":{"1min":0.54300000000000004,"5min":3.0230000000000001,"15min":27.414000000000001},"last":0.36499999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:43:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.52600000000000002,"15min":0.67200000000000004},"min":{"1min":0.372,"5min":0.29499999999999998,"15min":0.26400000000000001},"max":{"1min":0.81999999999999995,"5min":0.93999999999999995,"15min":14.614000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53900000000000003,"15min":0.624},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.27600000000000002},"max":{"1min":0.79300000000000004,"5min":0.94399999999999995,"15min":7.766},"last":0.48199999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:43:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.68100000000000005,"15min":0.71499999999999997},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.316},"max":{"1min":0.77200000000000002,"5min":7.915,"15min":17.765000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.55500000000000005,"15min":0.67400000000000004},"min":{"1min":0.39600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.79600000000000004,"5min":2.129,"15min":14.589},"last":0.61399999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:43:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55500000000000005,"15min":0.63100000000000001},"min":{"1min":0.379,"5min":0.316,"15min":0.29699999999999999},"max":{"1min":0.78900000000000003,"5min":1.944,"15min":7.7869999999999999},"last":0.501},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.72099999999999997,"15min":0.79300000000000004},"min":{"1min":0.38300000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.82199999999999995,"5min":7.9589999999999996,"15min":20.574999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Feb 23 00:43:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.58599999999999997,"15min":1.075},"min":{"1min":0.40899999999999997,"5min":0.373,"15min":0.34499999999999997},"max":{"1min":1.004,"5min":1.1359999999999999,"15min":66.632000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58199999999999996,"15min":0.84799999999999998},"min":{"1min":0.38100000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76400000000000001,"5min":1.3660000000000001,"15min":19.603999999999999},"last":0.52500000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149921,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318652,"kb_used_data":9140,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93426500,"statfs":{"total":95995035648,"available":95668736000,"internally_reserved":0,"allocated":9359360,"data_stored":5923389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.39300000000000002,"15min":0.38400000000000001},"min":{"1min":0.22700000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.33400000000000002,"15min":0.32900000000000001},"min":{"1min":0.245,"5min":0.215,"15min":0.215},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.29599999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.40600000000000003,"15min":0.60299999999999998},"min":{"1min":0.19600000000000001,"5min":0.193,"15min":0.17899999999999999},"max":{"1min":1.1870000000000001,"5min":3.2970000000000002,"15min":54.351999999999997},"last":0.251},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.45700000000000002,"15min":0.77200000000000002},"min":{"1min":0.246,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":0.53400000000000003,"5min":3.7120000000000002,"15min":55.337000000000003},"last":0.27400000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.44900000000000001,"15min":0.752},"min":{"1min":0.27100000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.44700000000000001,"5min":3.7570000000000001,"15min":55.368000000000002},"last":0.318},{"interface":"front","average":{"1min":0.438,"5min":0.46400000000000002,"15min":0.73099999999999998},"min":{"1min":0.28399999999999997,"5min":0.214,"15min":0.185},"max":{"1min":2.0059999999999998,"5min":4.181,"15min":54.383000000000003},"last":0.33200000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:43:30 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.65200000000000002,"15min":0.93799999999999994},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.27900000000000003},"max":{"1min":2.5720000000000001,"5min":6.8799999999999999,"15min":54.530000000000001},"last":0.628},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.68000000000000005,"15min":0.65000000000000002},"min":{"1min":0.35599999999999998,"5min":0.30199999999999999,"15min":0.24099999999999999},"max":{"1min":3.8969999999999998,"5min":5.8109999999999999,"15min":8.0359999999999996},"last":0.48899999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:43:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.628,"15min":0.89700000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.28899999999999998},"max":{"1min":5.2160000000000002,"5min":5.2160000000000002,"15min":55.401000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.69699999999999995,"15min":0.873},"min":{"1min":0.38300000000000001,"5min":0.32700000000000001,"15min":0.30299999999999999},"max":{"1min":2.5219999999999998,"5min":7.0179999999999998,"15min":54.454999999999998},"last":0.496}]},{"osd":6,"last update":"Fri Feb 23 00:43:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.82599999999999996,"15min":0.86899999999999999},"min":{"1min":0.39800000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.7879999999999998,"5min":9.7490000000000006,"15min":54.305},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.92800000000000005,"15min":1.0529999999999999},"min":{"1min":0.42399999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":5.1509999999999998,"5min":9.6820000000000004,"15min":55.424999999999997},"last":0.55300000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:43:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.66400000000000003,"15min":0.81599999999999995},"min":{"1min":0.41099999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.8570000000000002,"5min":4.6710000000000003,"15min":54.222000000000001},"last":0.505},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.94799999999999995,"15min":1.05},"min":{"1min":0.46999999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.1840000000000002,"5min":23.928000000000001,"15min":55.460000000000001},"last":0.63800000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986396,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312536,"kb_used_data":7116,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432616,"statfs":{"total":95995035648,"available":95674998784,"internally_reserved":0,"allocated":7286784,"data_stored":3889944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.51500000000000001,"15min":0.505},"min":{"1min":0.38100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51100000000000001,"15min":0.50600000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48099999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.626,"15min":1.107},"min":{"1min":0.39700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.748,"5min":8.0540000000000003,"15min":41.137999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.67400000000000004,"15min":0.93600000000000005},"min":{"1min":0.40500000000000003,"5min":0.255,"15min":0.255},"max":{"1min":1.4510000000000001,"5min":10.952999999999999,"15min":26.210000000000001},"last":0.44600000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.66800000000000004,"15min":1.198},"min":{"1min":0.39200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.472,"5min":9.3490000000000002,"15min":67.253},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.85599999999999998,"15min":1.129},"min":{"1min":0.49299999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.76700000000000002,"5min":14.442,"15min":41.927999999999997},"last":0.52200000000000002}]},{"osd":3,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.70999999999999996,"15min":1.1299999999999999},"min":{"1min":0.443,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84499999999999997,"5min":9.5579999999999998,"15min":36.301000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.57899999999999996,"15min":1.2010000000000001},"min":{"1min":0.39200000000000002,"5min":0.26700000000000002,"15min":0.24299999999999999},"max":{"1min":0.67000000000000004,"5min":5.1879999999999997,"15min":42.064},"last":0.42699999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:43:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.40899999999999997,"15min":0.57399999999999995},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.217},"max":{"1min":0.63900000000000001,"5min":1.4279999999999999,"15min":14.242000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.47699999999999998,"15min":0.624},"min":{"1min":0.316,"5min":0.255,"15min":0.24299999999999999},"max":{"1min":0.80600000000000005,"5min":1.4079999999999999,"15min":14.124000000000001},"last":0.438}]},{"osd":6,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.48899999999999999,"15min":0.58199999999999996},"min":{"1min":0.35499999999999998,"5min":0.28999999999999998,"15min":0.254},"max":{"1min":0.753,"5min":1.734,"15min":8.0730000000000004},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57399999999999995,"15min":0.68400000000000005},"min":{"1min":0.32500000000000001,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":1.4039999999999999,"5min":13.339,"15min":45.814999999999998},"last":0.47199999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:43:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55200000000000005,"15min":0.64800000000000002},"min":{"1min":0.35099999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.95399999999999996,"5min":7.9210000000000003,"15min":17.178999999999998},"last":0.379},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59299999999999997,"15min":0.72499999999999998},"min":{"1min":0.35599999999999998,"5min":0.34200000000000003,"15min":0.28299999999999997},"max":{"1min":0.875,"5min":6.3220000000000001,"15min":15.85},"last":0.5}]}]},{"osd":5,"up_from":38,"seq":163208757463,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306336,"kb_used_data":9112,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93438816,"statfs":{"total":95995035648,"available":95681347584,"internally_reserved":0,"allocated":9330688,"data_stored":5932549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:43:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.68799999999999994,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.33200000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72399999999999998,"5min":1.087,"15min":1.087},"last":0.45300000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:43:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.47199999999999998,"15min":0.755},"min":{"1min":0.34499999999999997,"5min":0.27700000000000002,"15min":0.24099999999999999},"max":{"1min":0.82599999999999996,"5min":1.1180000000000001,"15min":38.838000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.46400000000000002,"15min":1.1819999999999999},"min":{"1min":0.42699999999999999,"5min":0.252,"15min":0.23699999999999999},"max":{"1min":0.68899999999999995,"5min":1.1200000000000001,"15min":127.794},"last":0.60899999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:43:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54900000000000004,"15min":0.95799999999999996},"min":{"1min":0.35699999999999998,"5min":0.26800000000000002,"15min":0.22},"max":{"1min":0.64000000000000001,"5min":6.7869999999999999,"15min":37.442999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.52900000000000003,"15min":0.71199999999999997},"min":{"1min":0.41199999999999998,"5min":0.27600000000000002,"15min":0.249},"max":{"1min":2.875,"5min":2.875,"15min":23.297999999999998},"last":0.48499999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:43:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.51500000000000001,"15min":0.83299999999999996},"min":{"1min":0.39800000000000002,"5min":0.33300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67700000000000005,"5min":1.3300000000000001,"15min":24.414999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.51800000000000002,"15min":0.91300000000000003},"min":{"1min":0.42399999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.78700000000000003,"5min":1.1679999999999999,"15min":40.865000000000002},"last":0.42399999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:43:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.40000000000000002,"15min":0.79100000000000004},"min":{"1min":0.32000000000000001,"5min":0.22500000000000001,"15min":0.219},"max":{"1min":0.69799999999999995,"5min":0.753,"15min":103.13},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.436,"15min":0.53100000000000003},"min":{"1min":0.34399999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":12.795999999999999},"last":0.60299999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56200000000000006,"15min":0.55400000000000005},"min":{"1min":0.33900000000000002,"5min":0.26400000000000001,"15min":0.26200000000000001},"max":{"1min":0.67800000000000005,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.63},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.45100000000000001,"15min":0.83199999999999996},"min":{"1min":0.30599999999999999,"5min":0.27000000000000002,"15min":0.26200000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":103.61499999999999},"last":0.63900000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.59499999999999997,"15min":0.56899999999999995},"min":{"1min":0.45300000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.298,"5min":11.478,"15min":11.478},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.50700000000000001,"15min":0.53100000000000003},"min":{"1min":0.41699999999999998,"5min":0.27900000000000003,"15min":0.25800000000000001},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":5.4050000000000002},"last":0.54900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:44:11.642 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.control pgp_num 32 2024-02-23T00:44:11.680 INFO:teuthology.orchestra.run.smithi087.stderr:Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:44:11.682 INFO:teuthology.orchestra.run.smithi171.stdout:{ 2024-02-23T00:44:11.682 INFO:teuthology.orchestra.run.smithi171.stdout: "ops": [], 2024-02-23T00:44:11.682 INFO:teuthology.orchestra.run.smithi171.stdout: "num_ops": 0 2024-02-23T00:44:11.682 INFO:teuthology.orchestra.run.smithi171.stdout:} 2024-02-23T00:44:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-23T00:44:11.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi087 with status 125: 'sudo docker kill -s 1 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3' 2024-02-23T00:44:12.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:11.336815+0000 mgr.y (mgr.24437) 684 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:12.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:11.581173+0000 mon.a (mon.0) 3943 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:44:12.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:12.007468+0000 mon.a (mon.0) 3944 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:44:12.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:12.255134+0000 mon.a (mon.0) 3945 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:44:12.464 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-23T00:44:12.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:44:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:12] "GET /metrics HTTP/1.1" 200 186061 "" "Prometheus/2.33.4" 2024-02-23T00:44:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:11.336815+0000 mgr.y (mgr.24437) 684 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:11.581173+0000 mon.a (mon.0) 3943 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:44:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:12.007468+0000 mon.a (mon.0) 3944 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:44:12.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:12.255134+0000 mon.a (mon.0) 3945 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:44:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:11.336815+0000 mgr.y (mgr.24437) 684 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:11.581173+0000 mon.a (mon.0) 3943 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:44:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:12.007468+0000 mon.a (mon.0) 3944 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:44:12.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:12.255134+0000 mon.a (mon.0) 3945 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-23T00:44:12.740 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 -- 172.21.15.87:0/2355204150 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100fe510 msgr2=0x7f85100f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 --2- 172.21.15.87:0/2355204150 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100fe510 0x7f85100f7d40 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 rx=0x7f85000047b0 tx=0x7f8500009cb0).stop 2024-02-23T00:44:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 -- 172.21.15.87:0/2355204150 shutdown_connections 2024-02-23T00:44:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 --2- 172.21.15.87:0/2355204150 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85100f8420 0x7f8510104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 --2- 172.21.15.87:0/2355204150 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85100fdbb0 0x7f85100fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 --2- 172.21.15.87:0/2355204150 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100fe510 0x7f85100f7d40 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.219 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 -- 172.21.15.87:0/2355204150 >> 172.21.15.87:0/2355204150 conn(0x7f85100f3800 msgr2=0x7f85100f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 -- 172.21.15.87:0/2355204150 shutdown_connections 2024-02-23T00:44:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 -- 172.21.15.87:0/2355204150 wait complete. 2024-02-23T00:44:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 librados: starting msgr at 2024-02-23T00:44:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.217+0000 7f85153aa700 1 librados: starting objecter 2024-02-23T00:44:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 Processor -- start 2024-02-23T00:44:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 -- start start 2024-02-23T00:44:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 librados: setting wanted keys 2024-02-23T00:44:13.220 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 librados: calling monclient init 2024-02-23T00:44:13.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100f8420 0x7f851006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.223 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85100fdbb0 0x7f851006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85100fe510 0x7f851006bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8510116580 con 0x7f85100fe510 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85101166f0 con 0x7f85100fdbb0 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f85153aa700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85101168b0 con 0x7f85100f8420 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850effd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100f8420 0x7f851006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850effd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100f8420 0x7f851006b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:33108/0 (socket says 172.21.15.87:33108) 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850f7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85100fe510 0x7f851006bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850effd700 1 -- 172.21.15.87:0/562102849 learned_addr learned my addr 172.21.15.87:0/562102849 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850e7fc700 1 --2- 172.21.15.87:0/562102849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85100fdbb0 0x7f851006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850effd700 1 -- 172.21.15.87:0/562102849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85100fdbb0 msgr2=0x7f851006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.224 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850effd700 1 --2- 172.21.15.87:0/562102849 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f85100fdbb0 0x7f851006b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850effd700 1 -- 172.21.15.87:0/562102849 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85100fe510 msgr2=0x7f851006bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850effd700 1 --2- 172.21.15.87:0/562102849 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85100fe510 0x7f851006bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850effd700 1 -- 172.21.15.87:0/562102849 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f8005040 con 0x7f85100f8420 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.218+0000 7f850f7fe700 1 --2- 172.21.15.87:0/562102849 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f85100fe510 0x7f851006bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f850effd700 1 --2- 172.21.15.87:0/562102849 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100f8420 0x7f851006b0d0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7f84f8009400 tx=0x7f84f8009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f8507fff700 10 client.?.objecter ms_handle_connect 0x7f85100f8420 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f8507fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f8007a40 con 0x7f85100f8420 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8500005040 con 0x7f85100f8420 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f85153aa700 10 client.38417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8510116fc0 con 0x7f85100f8420 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84f800f040 con 0x7f85100f8420 2024-02-23T00:44:13.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f85153aa700 1 librados: init done 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.219+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f8014a70 con 0x7f85100f8420 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.220+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f84f8010530 con 0x7f85100f8420 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.220+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f851005ee80 con 0x7f85100f8420 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f8507fff700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6800/1231597463 conn(0x7f84fc062c90 0x7f84fc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(809..809 src has 400..809) v4 ==== 7519+0+0 (secure 0 0 0) 0x7f84f807cac0 con 0x7f85100f8420 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 osd_map(809..809 src has 400..809) v4 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f8507fff700 3 client.38417.objecter handle_osd_map got epochs [809,809] > 0 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f8507fff700 3 client.38417.objecter handle_osd_map decoding full epoch 809 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f850e7fc700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6800/1231597463 conn(0x7f84fc062c90 0x7f84fc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f8507fff700 20 client.38417.objecter dump_active .. 0 homeless 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v809) v1 ==== 54+0+16 (secure 0 0 0) 0x7f84f8050290 con 0x7f85100f8420 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f8507fff700 10 client.38417.objecter ms_handle_connect 0x7f84fc062c90 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f850e7fc700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6800/1231597463 conn(0x7f84fc062c90 0x7f84fc065150 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f851006c870 tx=0x7f8500007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.226 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 10 client.38417.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 1 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6801/484081831 conn(0x7f85100fee00 0x7f85100ff220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510107110 osd=0 3 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510107110 osd=0 3 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510107110 osd=0 3 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 1 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510107110 osd=0 3 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 0 1 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510107110 osd=0 4 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 10 client.38417.objecter _send_command 1 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f851006c5b0 con 0x7f85100fee00 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f850f7fe700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6801/484081831 conn(0x7f85100fee00 0x7f85100ff220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f8507fff700 10 client.38417.objecter ms_handle_connect 0x7f85100fee00 2024-02-23T00:44:13.227 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.221+0000 7f850f7fe700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6801/484081831 conn(0x7f85100fee00 0x7f85100ff220 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.222+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f851006c5b0 con 0x7f85100fee00 2024-02-23T00:44:13.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.222+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:44:13.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.222+0000 7f8507fff700 10 client.38417.objecter _finish_command 1 = system:0 2024-02-23T00:44:13.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.222+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510107110 osd=0 4 2024-02-23T00:44:13.228 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.222+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 0 1 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 10 client.38417.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 2 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510107110 osd=0 3 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510107110 osd=0 3 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510107110 osd=0 3 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 2 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510107110 osd=0 3 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 0 2 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510107110 osd=0 4 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 10 client.38417.objecter _send_command 2 2024-02-23T00:44:13.242 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.240+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f85100273c0 con 0x7f85100fee00 2024-02-23T00:44:13.243 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f85100273c0 con 0x7f85100fee00 2024-02-23T00:44:13.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:44:13.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f8507fff700 10 client.38417.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510107110 osd=0 4 2024-02-23T00:44:13.244 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 0 2 2024-02-23T00:44:13.244 INFO:teuthology.orchestra.run.smithi087.stdout:osd.0: {} 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 10 client.38417.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 3 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6805/685354171 conn(0x7f85100598f0 0x7f8510059d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510028710 osd=1 3 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510028710 osd=1 3 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510028710 osd=1 3 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 3 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510028710 osd=1 3 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 1 3 2024-02-23T00:44:13.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510028710 osd=1 4 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 10 client.38417.objecter _send_command 3 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.242+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.87:6805/685354171 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f851011d930 con 0x7f85100598f0 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.243+0000 7f850effd700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6805/685354171 conn(0x7f85100598f0 0x7f8510059d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.243+0000 7f8507fff700 10 client.38417.objecter ms_handle_connect 0x7f85100598f0 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.243+0000 7f850effd700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6805/685354171 conn(0x7f85100598f0 0x7f8510059d30 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.243+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f851011d930 con 0x7f85100598f0 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.243+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 3: 0 ) v1 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.243+0000 7f8507fff700 10 client.38417.objecter _finish_command 3 = system:0 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.243+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510028710 osd=1 4 2024-02-23T00:44:13.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.243+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 1 3 2024-02-23T00:44:13.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 10 client.38417.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:44:13.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 4 2024-02-23T00:44:13.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510028710 osd=1 3 2024-02-23T00:44:13.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510028710 osd=1 3 2024-02-23T00:44:13.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510028710 osd=1 3 2024-02-23T00:44:13.260 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 4 2024-02-23T00:44:13.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510028710 osd=1 3 2024-02-23T00:44:13.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 1 4 2024-02-23T00:44:13.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510028710 osd=1 4 2024-02-23T00:44:13.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 10 client.38417.objecter _send_command 4 2024-02-23T00:44:13.261 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.259+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.87:6805/685354171 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8510059360 con 0x7f85100598f0 2024-02-23T00:44:13.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.260+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8510059360 con 0x7f85100598f0 2024-02-23T00:44:13.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.260+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:44:13.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.260+0000 7f8507fff700 10 client.38417.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.260+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510028710 osd=1 4 2024-02-23T00:44:13.262 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.260+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 1 4 2024-02-23T00:44:13.263 INFO:teuthology.orchestra.run.smithi087.stdout:osd.1: {} 2024-02-23T00:44:13.263 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 10 client.38417.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 5 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8510028270 0x7f851011c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510059770 osd=2 3 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510059770 osd=2 3 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510059770 osd=2 3 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 5 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510059770 osd=2 3 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 2 5 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510059770 osd=2 4 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 10 client.38417.objecter _send_command 5 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.87:6809/3605394228 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f851011c840 con 0x7f8510028270 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f850e7fc700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8510028270 0x7f851011c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.264 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f8507fff700 10 client.38417.objecter ms_handle_connect 0x7f8510028270 2024-02-23T00:44:13.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.261+0000 7f850e7fc700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8510028270 0x7f851011c350 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.262+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f851011c840 con 0x7f8510028270 2024-02-23T00:44:13.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.262+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 5: 0 ) v1 2024-02-23T00:44:13.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.262+0000 7f8507fff700 10 client.38417.objecter _finish_command 5 = system:0 2024-02-23T00:44:13.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.262+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510059770 osd=2 4 2024-02-23T00:44:13.265 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.262+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 2 5 2024-02-23T00:44:13.280 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:11.811487+0000 mgr.y (mgr.24437) 685 : cluster [DBG] pgmap v1242: 164 pgs: 1 active+remapped+backfill_toofull, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/684 objects misplaced (1.316%) 2024-02-23T00:44:13.280 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:12.628027+0000 mon.a (mon.0) 3946 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:44:13.280 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:12.628734+0000 mon.a (mon.0) 3947 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:44:13.280 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:12.633211+0000 mon.a (mon.0) 3948 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:44:13.280 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:13.221289+0000 mon.c (mon.2) 507 : audit [DBG] from='client.? 172.21.15.87:0/562102849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:44:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.280+0000 7f85153aa700 10 client.38417.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:44:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 6 2024-02-23T00:44:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510059770 osd=2 3 2024-02-23T00:44:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510059770 osd=2 3 2024-02-23T00:44:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510059770 osd=2 3 2024-02-23T00:44:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 6 2024-02-23T00:44:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510059770 osd=2 3 2024-02-23T00:44:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 2 6 2024-02-23T00:44:13.282 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510059770 osd=2 4 2024-02-23T00:44:13.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 10 client.38417.objecter _send_command 6 2024-02-23T00:44:13.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.281+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.87:6809/3605394228 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8510027830 con 0x7f8510028270 2024-02-23T00:44:13.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8510027830 con 0x7f8510028270 2024-02-23T00:44:13.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:44:13.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f8507fff700 10 client.38417.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510059770 osd=2 4 2024-02-23T00:44:13.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 2 6 2024-02-23T00:44:13.285 INFO:teuthology.orchestra.run.smithi087.stdout:osd.2: {} 2024-02-23T00:44:13.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 10 client.38417.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:13.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 7 2024-02-23T00:44:13.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6813/128059790 conn(0x7f851011fab0 0x7f8510121ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851011d230 osd=3 3 2024-02-23T00:44:13.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011d230 osd=3 3 2024-02-23T00:44:13.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851011d230 osd=3 3 2024-02-23T00:44:13.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 7 2024-02-23T00:44:13.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f851011d230 osd=3 3 2024-02-23T00:44:13.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 3 7 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011d230 osd=3 4 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 10 client.38417.objecter _send_command 7 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.87:6813/128059790 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8510124b70 con 0x7f851011fab0 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.283+0000 7f850f7fe700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6813/128059790 conn(0x7f851011fab0 0x7f8510121ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.284+0000 7f8507fff700 10 client.38417.objecter ms_handle_connect 0x7f851011fab0 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.284+0000 7f850f7fe700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6813/128059790 conn(0x7f851011fab0 0x7f8510121ea0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.284+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8510124b70 con 0x7f851011fab0 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.285+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 7: 0 ) v1 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.285+0000 7f8507fff700 10 client.38417.objecter _finish_command 7 = system:0 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.285+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f851011d230 osd=3 4 2024-02-23T00:44:13.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.285+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 3 7 2024-02-23T00:44:13.288 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 10 client.38417.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 8 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851011d230 osd=3 3 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011d230 osd=3 3 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851011d230 osd=3 3 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 8 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f851011d230 osd=3 3 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 3 8 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011d230 osd=3 4 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 10 client.38417.objecter _send_command 8 2024-02-23T00:44:13.302 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.300+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.87:6813/128059790 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f851011cd60 con 0x7f851011fab0 2024-02-23T00:44:13.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f851011cd60 con 0x7f851011fab0 2024-02-23T00:44:13.303 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:44:13.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f8507fff700 10 client.38417.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f851011d230 osd=3 4 2024-02-23T00:44:13.304 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 3 8 2024-02-23T00:44:13.304 INFO:teuthology.orchestra.run.smithi087.stdout:osd.3: {} 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 10 client.38417.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 9 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6800/4175844757 conn(0x7f851011beb0 0x7f8510122390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851011cb20 osd=4 3 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011cb20 osd=4 3 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851011cb20 osd=4 3 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 9 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f851011cb20 osd=4 3 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 4 9 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011cb20 osd=4 4 2024-02-23T00:44:13.305 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 10 client.38417.objecter _send_command 9 2024-02-23T00:44:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.302+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.171:6800/4175844757 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8510122880 con 0x7f851011beb0 2024-02-23T00:44:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.303+0000 7f850effd700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6800/4175844757 conn(0x7f851011beb0 0x7f8510122390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.303+0000 7f8507fff700 10 client.38417.objecter ms_handle_connect 0x7f851011beb0 2024-02-23T00:44:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.303+0000 7f850effd700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6800/4175844757 conn(0x7f851011beb0 0x7f8510122390 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.304+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8510122880 con 0x7f851011beb0 2024-02-23T00:44:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.304+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 9: 0 ) v1 2024-02-23T00:44:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.304+0000 7f8507fff700 10 client.38417.objecter _finish_command 9 = system:0 2024-02-23T00:44:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.304+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f851011cb20 osd=4 4 2024-02-23T00:44:13.306 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.304+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 4 9 2024-02-23T00:44:13.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 10 client.38417.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 10 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851011cb20 osd=4 3 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011cb20 osd=4 3 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851011cb20 osd=4 3 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 10 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f851011cb20 osd=4 3 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 4 10 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011cb20 osd=4 4 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 10 client.38417.objecter _send_command 10 2024-02-23T00:44:13.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.319+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.171:6800/4175844757 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8510122880 con 0x7f851011beb0 2024-02-23T00:44:13.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8510122880 con 0x7f851011beb0 2024-02-23T00:44:13.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:44:13.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f8507fff700 10 client.38417.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f851011cb20 osd=4 4 2024-02-23T00:44:13.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 4 10 2024-02-23T00:44:13.323 INFO:teuthology.orchestra.run.smithi087.stdout:osd.4: {} 2024-02-23T00:44:13.323 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 10 client.38417.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 11 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8510124130 0x7f8510124590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510122880 osd=5 3 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510122880 osd=5 3 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510122880 osd=5 3 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 11 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510122880 osd=5 3 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 5 11 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510122880 osd=5 4 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 10 client.38417.objecter _send_command 11 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.171:6804/2550013795 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f851012b510 con 0x7f8510124130 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.321+0000 7f850e7fc700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8510124130 0x7f8510124590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.322+0000 7f8507fff700 10 client.38417.objecter ms_handle_connect 0x7f8510124130 2024-02-23T00:44:13.324 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.322+0000 7f850e7fc700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8510124130 0x7f8510124590 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.322+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f851012b510 con 0x7f8510124130 2024-02-23T00:44:13.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.322+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 11: 0 ) v1 2024-02-23T00:44:13.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.322+0000 7f8507fff700 10 client.38417.objecter _finish_command 11 = system:0 2024-02-23T00:44:13.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.322+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510122880 osd=5 4 2024-02-23T00:44:13.325 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.322+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 5 11 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 10 client.38417.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 12 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510122880 osd=5 3 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510122880 osd=5 3 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510122880 osd=5 3 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 12 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510122880 osd=5 3 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 5 12 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510122880 osd=5 4 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 10 client.38417.objecter _send_command 12 2024-02-23T00:44:13.339 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.337+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.171:6804/2550013795 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8510128db0 con 0x7f8510124130 2024-02-23T00:44:13.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8510128db0 con 0x7f8510124130 2024-02-23T00:44:13.340 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:44:13.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f8507fff700 10 client.38417.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510122880 osd=5 4 2024-02-23T00:44:13.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 5 12 2024-02-23T00:44:13.341 INFO:teuthology.orchestra.run.smithi087.stdout:osd.5: {} 2024-02-23T00:44:13.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 10 client.38417.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:13.341 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 13 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8510129490 0x7f85101298d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510128db0 osd=6 3 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510128db0 osd=6 3 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510128db0 osd=6 3 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 13 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510128db0 osd=6 3 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 6 13 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510128db0 osd=6 4 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 10 client.38417.objecter _send_command 13 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.339+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.171:6808/2509408498 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f851012b490 con 0x7f8510129490 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.340+0000 7f850f7fe700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8510129490 0x7f85101298d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.340+0000 7f850f7fe700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8510129490 0x7f85101298d0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.342 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.340+0000 7f8507fff700 10 client.38417.objecter ms_handle_connect 0x7f8510129490 2024-02-23T00:44:13.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.341+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f851012b490 con 0x7f8510129490 2024-02-23T00:44:13.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.341+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 13: 0 ) v1 2024-02-23T00:44:13.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.341+0000 7f8507fff700 10 client.38417.objecter _finish_command 13 = system:0 2024-02-23T00:44:13.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.341+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510128db0 osd=6 4 2024-02-23T00:44:13.343 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.341+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 6 13 2024-02-23T00:44:13.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 10 client.38417.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:44:13.356 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 14 2024-02-23T00:44:13.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510128db0 osd=6 3 2024-02-23T00:44:13.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510128db0 osd=6 3 2024-02-23T00:44:13.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f8510128db0 osd=6 3 2024-02-23T00:44:13.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 14 2024-02-23T00:44:13.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f8510128db0 osd=6 3 2024-02-23T00:44:13.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 6 14 2024-02-23T00:44:13.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510128db0 osd=6 4 2024-02-23T00:44:13.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 10 client.38417.objecter _send_command 14 2024-02-23T00:44:13.357 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.355+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.171:6808/2509408498 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f851012b490 con 0x7f8510129490 2024-02-23T00:44:13.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f851012b490 con 0x7f8510129490 2024-02-23T00:44:13.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:44:13.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f8507fff700 10 client.38417.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f8510128db0 osd=6 4 2024-02-23T00:44:13.358 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 6 14 2024-02-23T00:44:13.359 INFO:teuthology.orchestra.run.smithi087.stdout:osd.6: {} 2024-02-23T00:44:13.359 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 10 client.38417.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 15 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6812/373787000 conn(0x7f851012dcb0 0x7f851012e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851012b490 osd=7 3 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851012b490 osd=7 3 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851012b490 osd=7 3 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 15 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f851012b490 osd=7 3 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 7 15 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851012b490 osd=7 4 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 10 client.38417.objecter _send_command 15 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.171:6812/373787000 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8510131a50 con 0x7f851012dcb0 2024-02-23T00:44:13.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.357+0000 7f850effd700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6812/373787000 conn(0x7f851012dcb0 0x7f851012e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:13.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.358+0000 7f850effd700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6812/373787000 conn(0x7f851012dcb0 0x7f851012e110 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:13.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.358+0000 7f8507fff700 10 client.38417.objecter ms_handle_connect 0x7f851012dcb0 2024-02-23T00:44:13.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.358+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f8510131a50 con 0x7f851012dcb0 2024-02-23T00:44:13.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.358+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 15: 0 ) v1 2024-02-23T00:44:13.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.358+0000 7f8507fff700 10 client.38417.objecter _finish_command 15 = system:0 2024-02-23T00:44:13.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.358+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f851012b490 osd=7 4 2024-02-23T00:44:13.361 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.358+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 7 15 2024-02-23T00:44:13.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 10 client.38417.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-23T00:44:13.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign -1 16 2024-02-23T00:44:13.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851012b490 osd=7 3 2024-02-23T00:44:13.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851012b490 osd=7 3 2024-02-23T00:44:13.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 20 client.38417.objecter _get_session s=0x7f851012b490 osd=7 3 2024-02-23T00:44:13.374 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 15 client.38417.objecter _session_command_op_remove -1 16 2024-02-23T00:44:13.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 20 client.38417.objecter get_session s=0x7f851012b490 osd=7 3 2024-02-23T00:44:13.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 15 client.38417.objecter _session_command_op_assign 7 16 2024-02-23T00:44:13.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851012b490 osd=7 4 2024-02-23T00:44:13.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 10 client.38417.objecter _send_command 16 2024-02-23T00:44:13.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.373+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 --> v2:172.21.15.171:6812/373787000 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8510129e80 con 0x7f851012dcb0 2024-02-23T00:44:13.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f8507fff700 1 -- 172.21.15.87:0/562102849 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (crc 0 0 0) 0x7f8510129e80 con 0x7f851012dcb0 2024-02-23T00:44:13.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f8507fff700 10 client.38417.objecter ms_dispatch 0x7f8510001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-23T00:44:13.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f8507fff700 10 client.38417.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f8507fff700 20 client.38417.objecter put_session s=0x7f851012b490 osd=7 4 2024-02-23T00:44:13.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f8507fff700 15 client.38417.objecter _session_command_op_remove 7 16 2024-02-23T00:44:13.376 INFO:teuthology.orchestra.run.smithi087.stdout:osd.7: {} 2024-02-23T00:44:13.377 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 librados: watch_flush enter 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 librados: watch_flush exit 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 client.38417.objecter close_session for osd.0 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6801/484081831 conn(0x7f85100fee00 msgr2=0x7f85100ff220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6801/484081831 conn(0x7f85100fee00 0x7f85100ff220 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510107110 osd=0 1 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 client.38417.objecter close_session for osd.1 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6805/685354171 conn(0x7f85100598f0 msgr2=0x7f8510059d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6805/685354171 conn(0x7f85100598f0 0x7f8510059d30 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510028710 osd=1 1 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 client.38417.objecter close_session for osd.2 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8510028270 msgr2=0x7f851011c350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6809/3605394228 conn(0x7f8510028270 0x7f851011c350 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.378 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510059770 osd=2 1 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 client.38417.objecter close_session for osd.3 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f850f7fe700 1 -- 172.21.15.87:0/562102849 reap_dead start 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6813/128059790 conn(0x7f851011fab0 msgr2=0x7f8510121ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6813/128059790 conn(0x7f851011fab0 0x7f8510121ea0 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011d230 osd=3 1 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 client.38417.objecter close_session for osd.4 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6800/4175844757 conn(0x7f851011beb0 msgr2=0x7f8510122390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6800/4175844757 conn(0x7f851011beb0 0x7f8510122390 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851011cb20 osd=4 1 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 client.38417.objecter close_session for osd.5 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8510124130 msgr2=0x7f8510124590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6804/2550013795 conn(0x7f8510124130 0x7f8510124590 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510122880 osd=5 1 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 client.38417.objecter close_session for osd.6 2024-02-23T00:44:13.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8510129490 msgr2=0x7f85101298d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6808/2509408498 conn(0x7f8510129490 0x7f85101298d0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f8510128db0 osd=6 1 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 client.38417.objecter close_session for osd.7 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6812/373787000 conn(0x7f851012dcb0 msgr2=0x7f851012e110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.171:6812/373787000 conn(0x7f851012dcb0 0x7f851012e110 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 20 client.38417.objecter put_session s=0x7f851012b490 osd=7 1 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 20 client.38417.objecter shutdown clearing up homeless session... 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 10 client.38417.objecter successfully canceled tick 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f850f7fe700 1 -- 172.21.15.87:0/562102849 reap_dead start 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6800/1231597463 conn(0x7f84fc062c90 msgr2=0x7f84fc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6800/1231597463 conn(0x7f84fc062c90 0x7f84fc065150 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 rx=0x7f851006c870 tx=0x7f8500007f90).stop 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100f8420 msgr2=0x7f851006b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100f8420 0x7f851006b0d0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 rx=0x7f84f8009400 tx=0x7f84f8009cb0).stop 2024-02-23T00:44:13.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 shutdown_connections 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> v2:172.21.15.87:6800/1231597463 conn(0x7f84fc062c90 0x7f84fc065150 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 --2- 172.21.15.87:0/562102849 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f85100f8420 0x7f851006b0d0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 >> 172.21.15.87:0/562102849 conn(0x7f85100f3800 msgr2=0x7f85100f5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 shutdown_connections 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 -- 172.21.15.87:0/562102849 wait complete. 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:13.375+0000 7f85153aa700 1 librados: shutdown 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.381 INFO:teuthology.orchestra.run.smithi087.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-23T00:44:13.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:11.811487+0000 mgr.y (mgr.24437) 685 : cluster [DBG] pgmap v1242: 164 pgs: 1 active+remapped+backfill_toofull, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/684 objects misplaced (1.316%) 2024-02-23T00:44:13.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:12.628027+0000 mon.a (mon.0) 3946 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:44:13.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:12.628734+0000 mon.a (mon.0) 3947 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:44:13.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:12.633211+0000 mon.a (mon.0) 3948 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:44:13.544 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:13.221289+0000 mon.c (mon.2) 507 : audit [DBG] from='client.? 172.21.15.87:0/562102849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:44:13.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:11.811487+0000 mgr.y (mgr.24437) 685 : cluster [DBG] pgmap v1242: 164 pgs: 1 active+remapped+backfill_toofull, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/684 objects misplaced (1.316%) 2024-02-23T00:44:13.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:12.628027+0000 mon.a (mon.0) 3946 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:44:13.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:12.628734+0000 mon.a (mon.0) 3947 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:44:13.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:12.633211+0000 mon.a (mon.0) 3948 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:44:13.605 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:13.221289+0000 mon.c (mon.2) 507 : audit [DBG] from='client.? 172.21.15.87:0/562102849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-23T00:44:13.910 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:44:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:44:14.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset noscrub 2024-02-23T00:44:14.434 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:14.434 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:15.265 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-23T00:44:15.537 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:15.538 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:15.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:13.811931+0000 mgr.y (mgr.24437) 686 : cluster [DBG] pgmap v1243: 164 pgs: 1 active+remapped+backfill_toofull, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/684 objects misplaced (1.316%) 2024-02-23T00:44:15.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:13.811931+0000 mgr.y (mgr.24437) 686 : cluster [DBG] pgmap v1243: 164 pgs: 1 active+remapped+backfill_toofull, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/684 objects misplaced (1.316%) 2024-02-23T00:44:15.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:13.811931+0000 mgr.y (mgr.24437) 686 : cluster [DBG] pgmap v1243: 164 pgs: 1 active+remapped+backfill_toofull, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9/684 objects misplaced (1.316%) 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.253+0000 7f3a073a1700 1 -- 172.21.15.87:0/3591005148 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a000fdbd0 msgr2=0x7f3a000fdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.253+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3591005148 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a000fdbd0 0x7f3a000fdfb0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 rx=0x7f39f00047b0 tx=0x7f39f0009cb0).stop 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.253+0000 7f3a073a1700 1 -- 172.21.15.87:0/3591005148 shutdown_connections 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.253+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3591005148 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a000f6a90 0x7f3a00104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.253+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3591005148 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a000fdbd0 0x7f3a000fdfb0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.253+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3591005148 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a000fe530 0x7f3a000f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.253+0000 7f3a073a1700 1 -- 172.21.15.87:0/3591005148 >> 172.21.15.87:0/3591005148 conn(0x7f3a0006c370 msgr2=0x7f3a0006c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 -- 172.21.15.87:0/3591005148 shutdown_connections 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 -- 172.21.15.87:0/3591005148 wait complete. 2024-02-23T00:44:16.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 librados: starting msgr at 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 librados: starting objecter 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 Processor -- start 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 -- start start 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 librados: setting wanted keys 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 librados: calling monclient init 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a000f6a90 0x7f3a0010e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a000fdbd0 0x7f3a0010e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a000fe530 0x7f3a00108320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a00108810 con 0x7f3a000f6a90 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a00108980 con 0x7f3a000fdbd0 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a073a1700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a00108b40 con 0x7f3a000fe530 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a0513d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a000f6a90 0x7f3a0010e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a0513d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a000f6a90 0x7f3a0010e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42256/0 (socket says 172.21.15.87:42256) 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a0593e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a000fe530 0x7f3a00108320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:16.270 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.254+0000 7f3a0513d700 1 -- 172.21.15.87:0/3321951989 learned_addr learned my addr 172.21.15.87:0/3321951989 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a0493c700 1 --2- 172.21.15.87:0/3321951989 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a000fdbd0 0x7f3a0010e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a0513d700 1 -- 172.21.15.87:0/3321951989 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a000fe530 msgr2=0x7f3a00108320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a0513d700 1 --2- 172.21.15.87:0/3321951989 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a000fe530 0x7f3a00108320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a0513d700 1 -- 172.21.15.87:0/3321951989 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a000fdbd0 msgr2=0x7f3a0010e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a0513d700 1 --2- 172.21.15.87:0/3321951989 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a000fdbd0 0x7f3a0010e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a0513d700 1 -- 172.21.15.87:0/3321951989 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc005040 con 0x7f3a000f6a90 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a0593e700 1 --2- 172.21.15.87:0/3321951989 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a000fe530 0x7f3a00108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a0513d700 1 --2- 172.21.15.87:0/3321951989 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a000f6a90 0x7f3a0010e180 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f39f0009b10 tx=0x7f39f0009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f39f67fc700 10 client.?.objecter ms_handle_connect 0x7f3a000f6a90 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f39f67fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f39f67fc700 1 -- 172.21.15.87:0/3321951989 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f0018070 con 0x7f3a000f6a90 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f39f67fc700 1 -- 172.21.15.87:0/3321951989 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39f0013da0 con 0x7f3a000f6a90 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f0005040 con 0x7f3a000f6a90 2024-02-23T00:44:16.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f39f67fc700 1 -- 172.21.15.87:0/3321951989 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f001dcc0 con 0x7f3a000f6a90 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a073a1700 10 client.27948.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a00108fe0 con 0x7f3a000f6a90 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.255+0000 7f3a073a1700 1 librados: init done 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.256+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a001091d0 con 0x7f3a000f6a90 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.257+0000 7f39f67fc700 1 -- 172.21.15.87:0/3321951989 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f39f001d420 con 0x7f3a000f6a90 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.257+0000 7f39f67fc700 1 --2- 172.21.15.87:0/3321951989 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39ec062c90 0x7f39ec065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.257+0000 7f39f67fc700 1 -- 172.21.15.87:0/3321951989 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(810..810 src has 400..810) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f39f007e990 con 0x7f3a000f6a90 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.257+0000 7f39f67fc700 10 client.27948.objecter ms_dispatch 0x7f3a00001550 osd_map(810..810 src has 400..810) v4 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.257+0000 7f3a0493c700 1 --2- 172.21.15.87:0/3321951989 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39ec062c90 0x7f39ec065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.257+0000 7f39f67fc700 3 client.27948.objecter handle_osd_map got epochs [810,810] > 0 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.257+0000 7f39f67fc700 3 client.27948.objecter handle_osd_map decoding full epoch 810 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.258+0000 7f39f67fc700 20 client.27948.objecter dump_active .. 0 homeless 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.258+0000 7f39f67fc700 10 client.27948.objecter ms_handle_connect 0x7f39ec062c90 2024-02-23T00:44:16.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.258+0000 7f3a0493c700 1 --2- 172.21.15.87:0/3321951989 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39ec062c90 0x7f39ec065150 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f3a00109ac0 tx=0x7f39fc00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:16.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.261+0000 7f39f67fc700 1 -- 172.21.15.87:0/3321951989 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f39f0052180 con 0x7f3a000f6a90 2024-02-23T00:44:16.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:15.284124+0000 mon.a (mon.0) 3949 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-02-23T00:44:16.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:15.284124+0000 mon.a (mon.0) 3949 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-02-23T00:44:16.437 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:16.435+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"} v 0) v1 -- 0x7f3a00109800 con 0x7f3a000f6a90 2024-02-23T00:44:16.682 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:15.284124+0000 mon.a (mon.0) 3949 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-02-23T00:44:17.251 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:17.251 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:17.360 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.358+0000 7f39f67fc700 1 -- 172.21.15.87:0/3321951989 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]=0 set pool 5 pgp_num to 32 v812) v1 ==== 146+0+0 (secure 0 0 0) 0x7f39f0051d70 con 0x7f3a000f6a90 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 10 librados: watch_flush enter 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 10 librados: watch_flush exit 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 20 client.27948.objecter shutdown clearing up homeless session... 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 10 client.27948.objecter successfully canceled tick 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39ec062c90 msgr2=0x7f39ec065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3321951989 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39ec062c90 0x7f39ec065150 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 rx=0x7f3a00109ac0 tx=0x7f39fc00b040).stop 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a000f6a90 msgr2=0x7f3a0010e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3321951989 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a000f6a90 0x7f3a0010e180 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f39f0009b10 tx=0x7f39f0009cb0).stop 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 shutdown_connections 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3321951989 >> v2:172.21.15.87:6800/1231597463 conn(0x7f39ec062c90 0x7f39ec065150 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3321951989 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a000fe530 0x7f3a00108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:17.363 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3321951989 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a000f6a90 0x7f3a0010e180 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:17.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 --2- 172.21.15.87:0/3321951989 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a000fdbd0 0x7f3a0010e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:17.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 >> 172.21.15.87:0/3321951989 conn(0x7f3a0006c370 msgr2=0x7f3a000f58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:17.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 shutdown_connections 2024-02-23T00:44:17.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 -- 172.21.15.87:0/3321951989 wait complete. 2024-02-23T00:44:17.364 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:17.361+0000 7f3a073a1700 1 librados: shutdown 2024-02-23T00:44:17.364 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 5 pgp_num to 32 2024-02-23T00:44:17.667 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:15.812666+0000 mgr.y (mgr.24437) 687 : cluster [DBG] pgmap v1245: 164 pgs: 1 peering, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 33 B/s, 1 objects/s recovering 2024-02-23T00:44:17.668 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:16.287101+0000 mon.a (mon.0) 3950 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-02-23T00:44:17.668 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:16.288125+0000 mon.a (mon.0) 3951 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-23T00:44:17.668 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:16.436428+0000 mon.a (mon.0) 3952 : audit [INF] from='client.? 172.21.15.87:0/3321951989' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:44:17.733 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:15.812666+0000 mgr.y (mgr.24437) 687 : cluster [DBG] pgmap v1245: 164 pgs: 1 peering, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 33 B/s, 1 objects/s recovering 2024-02-23T00:44:17.734 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:16.287101+0000 mon.a (mon.0) 3950 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-02-23T00:44:17.734 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:16.288125+0000 mon.a (mon.0) 3951 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-23T00:44:17.734 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:16.436428+0000 mon.a (mon.0) 3952 : audit [INF] from='client.? 172.21.15.87:0/3321951989' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:44:17.734 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:15.812666+0000 mgr.y (mgr.24437) 687 : cluster [DBG] pgmap v1245: 164 pgs: 1 peering, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 33 B/s, 1 objects/s recovering 2024-02-23T00:44:17.734 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:16.287101+0000 mon.a (mon.0) 3950 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-02-23T00:44:17.734 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:16.288125+0000 mon.a (mon.0) 3951 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-23T00:44:17.734 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:16.436428+0000 mon.a (mon.0) 3952 : audit [INF] from='client.? 172.21.15.87:0/3321951989' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-23T00:44:17.983 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-23T00:44:18.251 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:18.492 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:17.358789+0000 mon.a (mon.0) 3953 : audit [INF] from='client.? 172.21.15.87:0/3321951989' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:44:18.492 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:17.358828+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-02-23T00:44:18.492 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:17.358789+0000 mon.a (mon.0) 3953 : audit [INF] from='client.? 172.21.15.87:0/3321951989' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:44:18.493 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:17.358828+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-02-23T00:44:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:17.358789+0000 mon.a (mon.0) 3953 : audit [INF] from='client.? 172.21.15.87:0/3321951989' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]': finished 2024-02-23T00:44:18.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:17.358828+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-02-23T00:44:19.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.780+0000 7f0dfd701700 1 -- 172.21.15.87:0/4176411286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80fdbd0 msgr2=0x7f0df80fdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:19.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.780+0000 7f0dfd701700 1 --2- 172.21.15.87:0/4176411286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80fdbd0 0x7f0df80fdfb0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 rx=0x7f0dec004870 tx=0x7f0dec009cb0).stop 2024-02-23T00:44:19.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.780+0000 7f0dfd701700 1 -- 172.21.15.87:0/4176411286 shutdown_connections 2024-02-23T00:44:19.199 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.780+0000 7f0dfd701700 1 --2- 172.21.15.87:0/4176411286 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0df80f6a90 0x7f0df8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.780+0000 7f0dfd701700 1 --2- 172.21.15.87:0/4176411286 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80fdbd0 0x7f0df80fdfb0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.780+0000 7f0dfd701700 1 --2- 172.21.15.87:0/4176411286 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0df80fe530 0x7f0df80f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.780+0000 7f0dfd701700 1 -- 172.21.15.87:0/4176411286 >> 172.21.15.87:0/4176411286 conn(0x7f0df806c370 msgr2=0x7f0df806c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.780+0000 7f0dfd701700 1 -- 172.21.15.87:0/4176411286 shutdown_connections 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.781+0000 7f0dfd701700 1 -- 172.21.15.87:0/4176411286 wait complete. 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.781+0000 7f0dfd701700 1 librados: starting msgr at 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.781+0000 7f0dfd701700 1 librados: starting objecter 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.781+0000 7f0dfd701700 1 Processor -- start 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.781+0000 7f0dfd701700 1 -- start start 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.781+0000 7f0dfd701700 1 librados: setting wanted keys 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.781+0000 7f0dfd701700 1 librados: calling monclient init 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0dfd701700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80f6a90 0x7f0df80f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0dfd701700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0df80fdbd0 0x7f0df80f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:19.200 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0dfd701700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0df80fe530 0x7f0df80f8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0dfd701700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df8116560 con 0x7f0df80f6a90 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0dfd701700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df81166d0 con 0x7f0df80fdbd0 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0dfd701700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df8116890 con 0x7f0df80fe530 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80f6a90 0x7f0df80f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df6ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80f6a90 0x7f0df80f7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:42276/0 (socket says 172.21.15.87:42276) 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df6ffd700 1 -- 172.21.15.87:0/1320593284 learned_addr learned my addr 172.21.15.87:0/1320593284 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df77fe700 1 --2- 172.21.15.87:0/1320593284 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0df80fe530 0x7f0df80f8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df67fc700 1 --2- 172.21.15.87:0/1320593284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0df80fdbd0 0x7f0df80f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df6ffd700 1 -- 172.21.15.87:0/1320593284 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0df80fe530 msgr2=0x7f0df80f8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df6ffd700 1 --2- 172.21.15.87:0/1320593284 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0df80fe530 0x7f0df80f8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df6ffd700 1 -- 172.21.15.87:0/1320593284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0df80fdbd0 msgr2=0x7f0df80f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df6ffd700 1 --2- 172.21.15.87:0/1320593284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0df80fdbd0 0x7f0df80f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:19.201 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df6ffd700 1 -- 172.21.15.87:0/1320593284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec005040 con 0x7f0df80f6a90 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0df6ffd700 1 --2- 172.21.15.87:0/1320593284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80f6a90 0x7f0df80f7680 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 rx=0x7f0dec009eb0 tx=0x7f0dec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0ddffff700 10 client.?.objecter ms_handle_connect 0x7f0df80f6a90 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0ddffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0ddffff700 1 -- 172.21.15.87:0/1320593284 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec00e070 con 0x7f0df80f6a90 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df8116b10 con 0x7f0df80f6a90 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0ddffff700 1 -- 172.21.15.87:0/1320593284 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0dec013da0 con 0x7f0df80f6a90 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.782+0000 7f0ddffff700 1 -- 172.21.15.87:0/1320593284 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dec019070 con 0x7f0df80f6a90 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.783+0000 7f0dfd701700 10 client.27954.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.783+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df8116f10 con 0x7f0df80f6a90 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.783+0000 7f0dfd701700 1 librados: init done 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.783+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df805f930 con 0x7f0df80f6a90 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.784+0000 7f0ddffff700 1 -- 172.21.15.87:0/1320593284 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0dec0090d0 con 0x7f0df80f6a90 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.784+0000 7f0ddffff700 1 --2- 172.21.15.87:0/1320593284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0de0062c90 0x7f0de0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:19.202 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.784+0000 7f0df67fc700 1 --2- 172.21.15.87:0/1320593284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0de0062c90 0x7f0de0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.784+0000 7f0ddffff700 1 -- 172.21.15.87:0/1320593284 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(812..812 src has 400..812) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f0dec07db30 con 0x7f0df80f6a90 2024-02-23T00:44:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.784+0000 7f0ddffff700 10 client.27954.objecter ms_dispatch 0x7f0df8001550 osd_map(812..812 src has 400..812) v4 2024-02-23T00:44:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.784+0000 7f0ddffff700 3 client.27954.objecter handle_osd_map got epochs [812,812] > 0 2024-02-23T00:44:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.784+0000 7f0ddffff700 3 client.27954.objecter handle_osd_map decoding full epoch 812 2024-02-23T00:44:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.784+0000 7f0ddffff700 20 client.27954.objecter dump_active .. 0 homeless 2024-02-23T00:44:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.785+0000 7f0df67fc700 1 --2- 172.21.15.87:0/1320593284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0de0062c90 0x7f0de0065150 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7f0df80f8e20 tx=0x7f0de800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.785+0000 7f0ddffff700 10 client.27954.objecter ms_handle_connect 0x7f0de0062c90 2024-02-23T00:44:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.789+0000 7f0ddffff700 1 -- 172.21.15.87:0/1320593284 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0dec0513d0 con 0x7f0df80f6a90 2024-02-23T00:44:19.203 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:18.962+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f0df80f8b60 con 0x7f0df80f6a90 2024-02-23T00:44:19.373 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.371+0000 7f0ddffff700 1 -- 172.21.15.87:0/1320593284 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v813) v1 ==== 91+0+0 (secure 0 0 0) 0x7f0dec023020 con 0x7f0df80f6a90 2024-02-23T00:44:19.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.373+0000 7f0dfd701700 10 librados: watch_flush enter 2024-02-23T00:44:19.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.373+0000 7f0dfd701700 10 librados: watch_flush exit 2024-02-23T00:44:19.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.373+0000 7f0dfd701700 20 client.27954.objecter shutdown clearing up homeless session... 2024-02-23T00:44:19.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.373+0000 7f0dfd701700 10 client.27954.objecter successfully canceled tick 2024-02-23T00:44:19.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.373+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0de0062c90 msgr2=0x7f0de0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:19.375 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.373+0000 7f0dfd701700 1 --2- 172.21.15.87:0/1320593284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0de0062c90 0x7f0de0065150 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 rx=0x7f0df80f8e20 tx=0x7f0de800b040).stop 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.373+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80f6a90 msgr2=0x7f0df80f7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.373+0000 7f0dfd701700 1 --2- 172.21.15.87:0/1320593284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80f6a90 0x7f0df80f7680 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 rx=0x7f0dec009eb0 tx=0x7f0dec00b040).stop 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.374+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 shutdown_connections 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.374+0000 7f0dfd701700 1 --2- 172.21.15.87:0/1320593284 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0de0062c90 0x7f0de0065150 unknown :-1 s=CLOSED pgs=2988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.374+0000 7f0dfd701700 1 --2- 172.21.15.87:0/1320593284 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0df80fe530 0x7f0df80f8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.374+0000 7f0dfd701700 1 --2- 172.21.15.87:0/1320593284 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0df80f6a90 0x7f0df80f7680 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.374+0000 7f0dfd701700 1 --2- 172.21.15.87:0/1320593284 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0df80fdbd0 0x7f0df80f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.374+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 >> 172.21.15.87:0/1320593284 conn(0x7f0df806c370 msgr2=0x7f0df80fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.374+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 shutdown_connections 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.374+0000 7f0dfd701700 1 -- 172.21.15.87:0/1320593284 wait complete. 2024-02-23T00:44:19.376 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:19.374+0000 7f0dfd701700 1 librados: shutdown 2024-02-23T00:44:19.377 INFO:teuthology.orchestra.run.smithi087.stderr:noscrub is unset 2024-02-23T00:44:19.637 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:17.813211+0000 mgr.y (mgr.24437) 688 : cluster [DBG] pgmap v1248: 164 pgs: 1 peering, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-02-23T00:44:19.637 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:18.963174+0000 mon.a (mon.0) 3955 : audit [INF] from='client.? 172.21.15.87:0/1320593284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:44:19.724 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:17.813211+0000 mgr.y (mgr.24437) 688 : cluster [DBG] pgmap v1248: 164 pgs: 1 peering, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-02-23T00:44:19.724 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:18.963174+0000 mon.a (mon.0) 3955 : audit [INF] from='client.? 172.21.15.87:0/1320593284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:44:19.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:17.813211+0000 mgr.y (mgr.24437) 688 : cluster [DBG] pgmap v1248: 164 pgs: 1 peering, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-02-23T00:44:19.725 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:18.963174+0000 mon.a (mon.0) 3955 : audit [INF] from='client.? 172.21.15.87:0/1320593284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-23T00:44:20.003 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:20.003 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:20.212 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-23T00:44:20.484 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:19.362121+0000 mon.a (mon.0) 3956 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-23T00:44:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:19.371669+0000 mon.a (mon.0) 3957 : audit [INF] from='client.? 172.21.15.87:0/1320593284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:44:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:19.371730+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-02-23T00:44:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:20.037411+0000 osd.7 (osd.7) 378 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:44:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:20.040303+0000 osd.7 (osd.7) 379 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:44:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:19.362121+0000 mon.a (mon.0) 3956 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-23T00:44:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:19.371669+0000 mon.a (mon.0) 3957 : audit [INF] from='client.? 172.21.15.87:0/1320593284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:44:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:19.371730+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-02-23T00:44:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:20.037411+0000 osd.7 (osd.7) 378 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:44:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:20.040303+0000 osd.7 (osd.7) 379 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:44:20.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:19.362121+0000 mon.a (mon.0) 3956 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-23T00:44:20.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:19.371669+0000 mon.a (mon.0) 3957 : audit [INF] from='client.? 172.21.15.87:0/1320593284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-23T00:44:20.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:19.371730+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-02-23T00:44:20.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:20.037411+0000 osd.7 (osd.7) 378 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:44:20.670 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:20.040303+0000 osd.7 (osd.7) 379 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:44:20.688 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-23T00:44:20.957 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:21.472 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:19.813729+0000 mgr.y (mgr.24437) 689 : cluster [DBG] pgmap v1250: 164 pgs: 1 peering, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 48 B/s, 1 objects/s recovering 2024-02-23T00:44:21.473 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:20.297055+0000 osd.4 (osd.4) 353 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:44:21.473 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:20.300926+0000 osd.4 (osd.4) 354 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:44:21.473 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:20.978963+0000 osd.5 (osd.5) 319 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:44:21.473 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:20.980007+0000 osd.5 (osd.5) 320 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:44:21.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:44:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:44:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:19.813729+0000 mgr.y (mgr.24437) 689 : cluster [DBG] pgmap v1250: 164 pgs: 1 peering, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 48 B/s, 1 objects/s recovering 2024-02-23T00:44:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:20.297055+0000 osd.4 (osd.4) 353 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:44:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:20.300926+0000 osd.4 (osd.4) 354 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:44:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:20.978963+0000 osd.5 (osd.5) 319 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:44:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:20.980007+0000 osd.5 (osd.5) 320 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:44:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:19.813729+0000 mgr.y (mgr.24437) 689 : cluster [DBG] pgmap v1250: 164 pgs: 1 peering, 163 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 48 B/s, 1 objects/s recovering 2024-02-23T00:44:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:20.297055+0000 osd.4 (osd.4) 353 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:44:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:20.300926+0000 osd.4 (osd.4) 354 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:44:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:20.978963+0000 osd.5 (osd.5) 319 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:44:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:20.980007+0000 osd.5 (osd.5) 320 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:44:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:21.075679+0000 osd.3 (osd.3) 261 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:44:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:21.079170+0000 osd.3 (osd.3) 262 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:44:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:21.290168+0000 osd.4 (osd.4) 355 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:44:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:21.291245+0000 osd.4 (osd.4) 356 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:44:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:21.346330+0000 mgr.y (mgr.24437) 690 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:21.607482+0000 osd.0 (osd.0) 65 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:44:22.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:21.610570+0000 osd.0 (osd.0) 66 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:44:22.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:22] "GET /metrics HTTP/1.1" 200 186061 "" "Prometheus/2.33.4" 2024-02-23T00:44:22.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:21.075679+0000 osd.3 (osd.3) 261 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:44:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:21.079170+0000 osd.3 (osd.3) 262 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:44:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:21.290168+0000 osd.4 (osd.4) 355 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:44:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:21.291245+0000 osd.4 (osd.4) 356 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:44:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:21.346330+0000 mgr.y (mgr.24437) 690 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:21.607482+0000 osd.0 (osd.0) 65 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:44:22.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:21.610570+0000 osd.0 (osd.0) 66 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:44:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:21.075679+0000 osd.3 (osd.3) 261 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:44:22.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:21.079170+0000 osd.3 (osd.3) 262 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:44:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:21.290168+0000 osd.4 (osd.4) 355 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:44:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:21.291245+0000 osd.4 (osd.4) 356 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:44:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:21.346330+0000 mgr.y (mgr.24437) 690 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:21.607482+0000 osd.0 (osd.0) 65 : cluster [DBG] 2.1f scrub starts 2024-02-23T00:44:22.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:21.610570+0000 osd.0 (osd.0) 66 : cluster [DBG] 2.1f scrub ok 2024-02-23T00:44:22.661 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:22.661 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.611+0000 7fc13b837700 1 -- 172.21.15.87:0/1087758636 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc1340fd370 msgr2=0x7fc1340f6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.611+0000 7fc13b837700 1 --2- 172.21.15.87:0/1087758636 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc1340fd370 0x7fc1340f6ba0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 rx=0x7fc124004870 tx=0x7fc124009cb0).stop 2024-02-23T00:44:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 -- 172.21.15.87:0/1087758636 shutdown_connections 2024-02-23T00:44:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 --2- 172.21.15.87:0/1087758636 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1340f7280 0x7fc134103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 --2- 172.21.15.87:0/1087758636 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1340fca10 0x7fc1340fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 --2- 172.21.15.87:0/1087758636 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc1340fd370 0x7fc1340f6ba0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 -- 172.21.15.87:0/1087758636 >> 172.21.15.87:0/1087758636 conn(0x7fc1340f2660 msgr2=0x7fc1340f4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 -- 172.21.15.87:0/1087758636 shutdown_connections 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 -- 172.21.15.87:0/1087758636 wait complete. 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 librados: starting msgr at 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 librados: starting objecter 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 Processor -- start 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 -- start start 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 librados: setting wanted keys 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 librados: calling monclient init 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1340f7280 0x7fc13410e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1340fca10 0x7fc13410e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:22.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc1340fd370 0x7fc134108280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134108770 con 0x7fc1340f7280 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1341088e0 con 0x7fc1340fca10 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.612+0000 7fc13b837700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc134108aa0 con 0x7fc1340fd370 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc1395d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1340f7280 0x7fc13410e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc1395d3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1340f7280 0x7fc13410e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39416/0 (socket says 172.21.15.87:39416) 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc139dd4700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc1340fd370 0x7fc134108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc1395d3700 1 -- 172.21.15.87:0/625196998 learned_addr learned my addr 172.21.15.87:0/625196998 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc138dd2700 1 --2- 172.21.15.87:0/625196998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1340fca10 0x7fc13410e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc1395d3700 1 -- 172.21.15.87:0/625196998 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc1340fd370 msgr2=0x7fc134108280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc1395d3700 1 --2- 172.21.15.87:0/625196998 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc1340fd370 0x7fc134108280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc1395d3700 1 -- 172.21.15.87:0/625196998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1340fca10 msgr2=0x7fc13410e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc1395d3700 1 --2- 172.21.15.87:0/625196998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1340fca10 0x7fc13410e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc1395d3700 1 -- 172.21.15.87:0/625196998 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130005040 con 0x7fc1340f7280 2024-02-23T00:44:22.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc139dd4700 1 --2- 172.21.15.87:0/625196998 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc1340fd370 0x7fc134108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc1395d3700 1 --2- 172.21.15.87:0/625196998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1340f7280 0x7fc13410e0e0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7fc1300048f0 tx=0x7fc130009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc12a7fc700 10 client.?.objecter ms_handle_connect 0x7fc1340f7280 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc12a7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc12a7fc700 1 -- 172.21.15.87:0/625196998 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130007b00 con 0x7fc1340f7280 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc124005040 con 0x7fc1340f7280 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc12a7fc700 1 -- 172.21.15.87:0/625196998 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc13000f040 con 0x7fc1340f7280 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc12a7fc700 1 -- 172.21.15.87:0/625196998 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130014990 con 0x7fc1340f7280 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc13b837700 10 client.27960.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc134109080 con 0x7fc1340f7280 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.613+0000 7fc13b837700 1 librados: init done 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.614+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc134047540 con 0x7fc1340f7280 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.614+0000 7fc12a7fc700 1 -- 172.21.15.87:0/625196998 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc130010500 con 0x7fc1340f7280 2024-02-23T00:44:22.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.615+0000 7fc12a7fc700 1 --2- 172.21.15.87:0/625196998 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc120062c90 0x7fc120065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:22.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.615+0000 7fc12a7fc700 1 -- 172.21.15.87:0/625196998 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(813..813 src has 400..813) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fc13007caa0 con 0x7fc1340f7280 2024-02-23T00:44:22.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.615+0000 7fc12a7fc700 10 client.27960.objecter ms_dispatch 0x7fc134001550 osd_map(813..813 src has 400..813) v4 2024-02-23T00:44:22.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.615+0000 7fc12a7fc700 3 client.27960.objecter handle_osd_map got epochs [813,813] > 0 2024-02-23T00:44:22.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.615+0000 7fc12a7fc700 3 client.27960.objecter handle_osd_map decoding full epoch 813 2024-02-23T00:44:22.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.615+0000 7fc138dd2700 1 --2- 172.21.15.87:0/625196998 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc120062c90 0x7fc120065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:22.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.615+0000 7fc12a7fc700 20 client.27960.objecter dump_active .. 0 homeless 2024-02-23T00:44:22.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.615+0000 7fc12a7fc700 10 client.27960.objecter ms_handle_connect 0x7fc120062c90 2024-02-23T00:44:22.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.615+0000 7fc138dd2700 1 --2- 172.21.15.87:0/625196998 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc120062c90 0x7fc120065150 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7fc124004ee0 tx=0x7fc12400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:22.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.619+0000 7fc12a7fc700 1 -- 172.21.15.87:0/625196998 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc130049f60 con 0x7fc1340f7280 2024-02-23T00:44:22.797 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:22.794+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fc13405fa60 con 0x7fc1340f7280 2024-02-23T00:44:23.329 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-23T00:44:23.391 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.389+0000 7fc12a7fc700 1 -- 172.21.15.87:0/625196998 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v814) v1 ==== 101+0+0 (secure 0 0 0) 0x7fc1300501f0 con 0x7fc1340f7280 2024-02-23T00:44:23.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 10 librados: watch_flush enter 2024-02-23T00:44:23.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 10 librados: watch_flush exit 2024-02-23T00:44:23.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 20 client.27960.objecter shutdown clearing up homeless session... 2024-02-23T00:44:23.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 10 client.27960.objecter successfully canceled tick 2024-02-23T00:44:23.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc120062c90 msgr2=0x7fc120065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:23.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 --2- 172.21.15.87:0/625196998 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc120062c90 0x7fc120065150 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 rx=0x7fc124004ee0 tx=0x7fc12400b040).stop 2024-02-23T00:44:23.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1340f7280 msgr2=0x7fc13410e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:23.395 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 --2- 172.21.15.87:0/625196998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1340f7280 0x7fc13410e0e0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 rx=0x7fc1300048f0 tx=0x7fc130009cb0).stop 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 shutdown_connections 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 --2- 172.21.15.87:0/625196998 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc120062c90 0x7fc120065150 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 --2- 172.21.15.87:0/625196998 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc1340fd370 0x7fc134108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 --2- 172.21.15.87:0/625196998 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc1340f7280 0x7fc13410e0e0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 --2- 172.21.15.87:0/625196998 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc1340fca10 0x7fc13410e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.392+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 >> 172.21.15.87:0/625196998 conn(0x7fc1340f2660 msgr2=0x7fc134060e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.393+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 shutdown_connections 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.393+0000 7fc13b837700 1 -- 172.21.15.87:0/625196998 wait complete. 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:23.393+0000 7fc13b837700 1 librados: shutdown 2024-02-23T00:44:23.396 INFO:teuthology.orchestra.run.smithi087.stderr:nodeep-scrub is unset 2024-02-23T00:44:23.599 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:21.814597+0000 mgr.y (mgr.24437) 691 : cluster [DBG] pgmap v1251: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:22.055753+0000 osd.6 (osd.6) 344 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:44:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:22.059202+0000 osd.6 (osd.6) 345 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:44:23.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:22.332083+0000 osd.2 (osd.2) 295 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:44:23.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:22.332902+0000 osd.2 (osd.2) 296 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:44:23.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:22.661862+0000 osd.1 (osd.1) 370 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:44:23.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:22.664877+0000 osd.1 (osd.1) 371 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:44:23.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:22.796086+0000 mon.a (mon.0) 3959 : audit [INF] from='client.? 172.21.15.87:0/625196998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:44:23.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:21.814597+0000 mgr.y (mgr.24437) 691 : cluster [DBG] pgmap v1251: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:23.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:22.055753+0000 osd.6 (osd.6) 344 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:44:23.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:22.059202+0000 osd.6 (osd.6) 345 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:44:23.794 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:22.332083+0000 osd.2 (osd.2) 295 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:22.332902+0000 osd.2 (osd.2) 296 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:22.661862+0000 osd.1 (osd.1) 370 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:22.664877+0000 osd.1 (osd.1) 371 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:22.796086+0000 mon.a (mon.0) 3959 : audit [INF] from='client.? 172.21.15.87:0/625196998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:21.814597+0000 mgr.y (mgr.24437) 691 : cluster [DBG] pgmap v1251: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:22.055753+0000 osd.6 (osd.6) 344 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:22.059202+0000 osd.6 (osd.6) 345 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:22.332083+0000 osd.2 (osd.2) 295 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:22.332902+0000 osd.2 (osd.2) 296 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:22.661862+0000 osd.1 (osd.1) 370 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:22.664877+0000 osd.1 (osd.1) 371 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:44:23.795 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:22.796086+0000 mon.a (mon.0) 3959 : audit [INF] from='client.? 172.21.15.87:0/625196998' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-23T00:44:24.105 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.meta 2024-02-23T00:44:24.106 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set default.rgw.meta pgp_num 12 2024-02-23T00:44:24.127 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:44:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:44:24.398 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:24.664 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:23.338360+0000 osd.2 (osd.2) 297 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:44:24.664 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:23.340371+0000 osd.2 (osd.2) 298 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:44:24.664 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:23.351059+0000 osd.4 (osd.4) 357 : cluster [DBG] 2.f scrub starts 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:23.354710+0000 osd.4 (osd.4) 358 : cluster [DBG] 2.f scrub ok 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:23.387141+0000 mon.a (mon.0) 3960 : audit [INF] from='client.? 172.21.15.87:0/625196998' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:23.387204+0000 mon.a (mon.0) 3961 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:23.647895+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:23.650784+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:23.338360+0000 osd.2 (osd.2) 297 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:23.340371+0000 osd.2 (osd.2) 298 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:23.351059+0000 osd.4 (osd.4) 357 : cluster [DBG] 2.f scrub starts 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:23.354710+0000 osd.4 (osd.4) 358 : cluster [DBG] 2.f scrub ok 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:23.387141+0000 mon.a (mon.0) 3960 : audit [INF] from='client.? 172.21.15.87:0/625196998' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:23.387204+0000 mon.a (mon.0) 3961 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-02-23T00:44:24.665 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:23.647895+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:44:24.666 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:23.650784+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:44:24.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:23.338360+0000 osd.2 (osd.2) 297 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:44:24.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:23.340371+0000 osd.2 (osd.2) 298 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:44:24.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:23.351059+0000 osd.4 (osd.4) 357 : cluster [DBG] 2.f scrub starts 2024-02-23T00:44:24.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:23.354710+0000 osd.4 (osd.4) 358 : cluster [DBG] 2.f scrub ok 2024-02-23T00:44:24.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:23.387141+0000 mon.a (mon.0) 3960 : audit [INF] from='client.? 172.21.15.87:0/625196998' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-23T00:44:24.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:23.387204+0000 mon.a (mon.0) 3961 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-02-23T00:44:24.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:23.647895+0000 osd.1 (osd.1) 372 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:44:24.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:23.650784+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:44:25.540 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:25.540 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:25.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:23.815056+0000 mgr.y (mgr.24437) 692 : cluster [DBG] pgmap v1253: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-02-23T00:44:25.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:24.053900+0000 osd.6 (osd.6) 346 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:44:25.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:24.057783+0000 osd.6 (osd.6) 347 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:44:25.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:24.081526+0000 osd.3 (osd.3) 263 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:44:25.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:24.084063+0000 osd.3 (osd.3) 264 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:44:25.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:24.659293+0000 mon.a (mon.0) 3962 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:44:25.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:25.134379+0000 osd.7 (osd.7) 380 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:44:25.559 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:25.137270+0000 osd.7 (osd.7) 381 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:44:25.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:23.815056+0000 mgr.y (mgr.24437) 692 : cluster [DBG] pgmap v1253: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-02-23T00:44:25.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:24.053900+0000 osd.6 (osd.6) 346 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:44:25.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:24.057783+0000 osd.6 (osd.6) 347 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:44:25.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:24.081526+0000 osd.3 (osd.3) 263 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:44:25.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:24.084063+0000 osd.3 (osd.3) 264 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:44:25.825 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:24.659293+0000 mon.a (mon.0) 3962 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:25.134379+0000 osd.7 (osd.7) 380 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:25.137270+0000 osd.7 (osd.7) 381 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:23.815056+0000 mgr.y (mgr.24437) 692 : cluster [DBG] pgmap v1253: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:24.053900+0000 osd.6 (osd.6) 346 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:24.057783+0000 osd.6 (osd.6) 347 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:24.081526+0000 osd.3 (osd.3) 263 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:24.084063+0000 osd.3 (osd.3) 264 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:24.659293+0000 mon.a (mon.0) 3962 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:25.134379+0000 osd.7 (osd.7) 380 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:44:25.826 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:25.137270+0000 osd.7 (osd.7) 381 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:44:26.159 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-23T00:44:26.425 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 -- 172.21.15.87:0/4163167394 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28fc0f7400 msgr2=0x7f28fc103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 --2- 172.21.15.87:0/4163167394 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28fc0f7400 0x7f28fc103a20 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 rx=0x7f28f4009be0 tx=0x7f28f4004d10).stop 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 -- 172.21.15.87:0/4163167394 shutdown_connections 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 --2- 172.21.15.87:0/4163167394 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28fc0f7400 0x7f28fc103a20 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 --2- 172.21.15.87:0/4163167394 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28fc0fcbb0 0x7f28fc0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 --2- 172.21.15.87:0/4163167394 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28fc0fd510 0x7f28fc0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 -- 172.21.15.87:0/4163167394 >> 172.21.15.87:0/4163167394 conn(0x7f28fc0f2820 msgr2=0x7f28fc0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 -- 172.21.15.87:0/4163167394 shutdown_connections 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 -- 172.21.15.87:0/4163167394 wait complete. 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 librados: starting msgr at 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.376+0000 7f29022f3700 1 librados: starting objecter 2024-02-23T00:44:26.432 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 Processor -- start 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 -- start start 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 librados: setting wanted keys 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 librados: calling monclient init 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28fc0f7400 0x7f28fc10dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28fc0fcbb0 0x7f28fc10e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28fc0fd510 0x7f28fc108150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc108640 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc1087b0 con 0x7f28fc0fd510 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f29022f3700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc108970 con 0x7f28fc0f7400 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fbfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28fc0f7400 0x7f28fc10dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fbfff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28fc0f7400 0x7f28fc10dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:54694/0 (socket says 172.21.15.87:54694) 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fb7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28fc0fcbb0 0x7f28fc10e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fbfff700 1 -- 172.21.15.87:0/156348744 learned_addr learned my addr 172.21.15.87:0/156348744 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f2900890700 1 --2- 172.21.15.87:0/156348744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28fc0fd510 0x7f28fc108150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:26.433 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fb7fe700 1 -- 172.21.15.87:0/156348744 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28fc0f7400 msgr2=0x7f28fc10dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fb7fe700 1 --2- 172.21.15.87:0/156348744 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28fc0f7400 0x7f28fc10dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fb7fe700 1 -- 172.21.15.87:0/156348744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28fc0fd510 msgr2=0x7f28fc108150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fb7fe700 1 --2- 172.21.15.87:0/156348744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28fc0fd510 0x7f28fc108150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fb7fe700 1 -- 172.21.15.87:0/156348744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f40097c0 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fbfff700 1 --2- 172.21.15.87:0/156348744 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28fc0f7400 0x7f28fc10dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28fb7fe700 1 --2- 172.21.15.87:0/156348744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28fc0fcbb0 0x7f28fc10e4a0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 rx=0x7f28f000e8f0 tx=0x7f28f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28f97fa700 10 client.?.objecter ms_handle_connect 0x7f28fc0fcbb0 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.377+0000 7f28f97fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.378+0000 7f28f97fa700 1 -- 172.21.15.87:0/156348744 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0007b40 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.378+0000 7f28f97fa700 1 -- 172.21.15.87:0/156348744 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28f000f040 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.378+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc108c50 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.378+0000 7f28f97fa700 1 -- 172.21.15.87:0/156348744 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0014710 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.378+0000 7f29022f3700 10 client.27966.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.378+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc114de0 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.434 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.378+0000 7f29022f3700 1 librados: init done 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.379+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc05fa60 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.379+0000 7f28f97fa700 1 -- 172.21.15.87:0/156348744 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f28f0014b40 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.379+0000 7f28f97fa700 1 --2- 172.21.15.87:0/156348744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28e4062c90 0x7f28e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.380+0000 7f28fbfff700 1 --2- 172.21.15.87:0/156348744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28e4062c90 0x7f28e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.380+0000 7f28f97fa700 1 -- 172.21.15.87:0/156348744 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(814..814 src has 400..814) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f28f007ca30 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.380+0000 7f28f97fa700 10 client.27966.objecter ms_dispatch 0x7f28fc001550 osd_map(814..814 src has 400..814) v4 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.380+0000 7f28f97fa700 3 client.27966.objecter handle_osd_map got epochs [814,814] > 0 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.380+0000 7f28f97fa700 3 client.27966.objecter handle_osd_map decoding full epoch 814 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.380+0000 7f28f97fa700 20 client.27966.objecter dump_active .. 0 homeless 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.380+0000 7f28f97fa700 10 client.27966.objecter ms_handle_connect 0x7f28e4062c90 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.380+0000 7f28fbfff700 1 --2- 172.21.15.87:0/156348744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28e4062c90 0x7f28e4065150 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f28ec004fd0 tx=0x7f28ec00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:26.435 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.384+0000 7f28f97fa700 1 -- 172.21.15.87:0/156348744 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f28f0049f60 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.553 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:26.551+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "12"} v 0) v1 -- 0x7f28fc106120 con 0x7f28fc0fcbb0 2024-02-23T00:44:26.678 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:25.076527+0000 osd.3 (osd.3) 265 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:44:26.678 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:25.079399+0000 osd.3 (osd.3) 266 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:44:26.679 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:25.625882+0000 osd.1 (osd.1) 374 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:44:26.679 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:25.629362+0000 osd.1 (osd.1) 375 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:44:26.679 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.032570+0000 osd.5 (osd.5) 321 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:44:26.679 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.033573+0000 osd.5 (osd.5) 322 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:44:26.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:25.076527+0000 osd.3 (osd.3) 265 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:44:26.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:25.079399+0000 osd.3 (osd.3) 266 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:44:26.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:25.625882+0000 osd.1 (osd.1) 374 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:44:26.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:25.629362+0000 osd.1 (osd.1) 375 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:44:26.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.032570+0000 osd.5 (osd.5) 321 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:44:26.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.033573+0000 osd.5 (osd.5) 322 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:44:26.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:25.076527+0000 osd.3 (osd.3) 265 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:44:26.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:25.079399+0000 osd.3 (osd.3) 266 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:44:26.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:25.625882+0000 osd.1 (osd.1) 374 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:44:26.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:25.629362+0000 osd.1 (osd.1) 375 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:44:26.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.032570+0000 osd.5 (osd.5) 321 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:44:26.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.033573+0000 osd.5 (osd.5) 322 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:44:27.269 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.265+0000 7f28f97fa700 1 -- 172.21.15.87:0/156348744 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "12"}]=0 set pool 6 pgp_num to 12 v815) v1 ==== 143+0+0 (secure 0 0 0) 0x7f28f0050220 con 0x7f28fc0fcbb0 2024-02-23T00:44:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 10 librados: watch_flush enter 2024-02-23T00:44:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 10 librados: watch_flush exit 2024-02-23T00:44:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 20 client.27966.objecter shutdown clearing up homeless session... 2024-02-23T00:44:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 10 client.27966.objecter successfully canceled tick 2024-02-23T00:44:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28e4062c90 msgr2=0x7f28e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 --2- 172.21.15.87:0/156348744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28e4062c90 0x7f28e4065150 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 rx=0x7f28ec004fd0 tx=0x7f28ec00b040).stop 2024-02-23T00:44:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28fc0fcbb0 msgr2=0x7f28fc10e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:27.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 --2- 172.21.15.87:0/156348744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28fc0fcbb0 0x7f28fc10e4a0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 rx=0x7f28f000e8f0 tx=0x7f28f000b040).stop 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 shutdown_connections 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 --2- 172.21.15.87:0/156348744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f28e4062c90 0x7f28e4065150 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 --2- 172.21.15.87:0/156348744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f28fc0fd510 0x7f28fc108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 --2- 172.21.15.87:0/156348744 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f28fc0f7400 0x7f28fc10dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 --2- 172.21.15.87:0/156348744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f28fc0fcbb0 0x7f28fc10e4a0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.268+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 >> 172.21.15.87:0/156348744 conn(0x7f28fc0f2820 msgr2=0x7f28fc060d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.269+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 shutdown_connections 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.269+0000 7f29022f3700 1 -- 172.21.15.87:0/156348744 wait complete. 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:27.269+0000 7f29022f3700 1 librados: shutdown 2024-02-23T00:44:27.272 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 6 pgp_num to 12 2024-02-23T00:44:27.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:25.673497+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:44:27.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:25.676973+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:44:27.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:25.815973+0000 mgr.y (mgr.24437) 693 : cluster [DBG] pgmap v1254: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:27.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.046416+0000 osd.6 (osd.6) 348 : cluster [DBG] 4.a deep-scrub starts 2024-02-23T00:44:27.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.050322+0000 osd.6 (osd.6) 349 : cluster [DBG] 4.a deep-scrub ok 2024-02-23T00:44:27.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.373167+0000 osd.2 (osd.2) 299 : cluster [DBG] 5.a deep-scrub starts 2024-02-23T00:44:27.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.374077+0000 osd.2 (osd.2) 300 : cluster [DBG] 5.a deep-scrub ok 2024-02-23T00:44:27.555 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:26.552857+0000 mon.a (mon.0) 3963 : audit [INF] from='client.? 172.21.15.87:0/156348744' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "12"}]: dispatch 2024-02-23T00:44:27.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.607877+0000 osd.1 (osd.1) 376 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:44:27.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.609294+0000 osd.1 (osd.1) 377 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:44:27.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:27.266872+0000 mon.a (mon.0) 3964 : audit [INF] from='client.? 172.21.15.87:0/156348744' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "12"}]': finished 2024-02-23T00:44:27.556 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.266982+0000 mon.a (mon.0) 3965 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-02-23T00:44:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:25.673497+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:44:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:25.676973+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:44:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:25.815973+0000 mgr.y (mgr.24437) 693 : cluster [DBG] pgmap v1254: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.046416+0000 osd.6 (osd.6) 348 : cluster [DBG] 4.a deep-scrub starts 2024-02-23T00:44:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.050322+0000 osd.6 (osd.6) 349 : cluster [DBG] 4.a deep-scrub ok 2024-02-23T00:44:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.373167+0000 osd.2 (osd.2) 299 : cluster [DBG] 5.a deep-scrub starts 2024-02-23T00:44:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.374077+0000 osd.2 (osd.2) 300 : cluster [DBG] 5.a deep-scrub ok 2024-02-23T00:44:27.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:26.552857+0000 mon.a (mon.0) 3963 : audit [INF] from='client.? 172.21.15.87:0/156348744' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "12"}]: dispatch 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.607877+0000 osd.1 (osd.1) 376 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.609294+0000 osd.1 (osd.1) 377 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:27.266872+0000 mon.a (mon.0) 3964 : audit [INF] from='client.? 172.21.15.87:0/156348744' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "12"}]': finished 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.266982+0000 mon.a (mon.0) 3965 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:25.673497+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:25.676973+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:25.815973+0000 mgr.y (mgr.24437) 693 : cluster [DBG] pgmap v1254: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.046416+0000 osd.6 (osd.6) 348 : cluster [DBG] 4.a deep-scrub starts 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.050322+0000 osd.6 (osd.6) 349 : cluster [DBG] 4.a deep-scrub ok 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.373167+0000 osd.2 (osd.2) 299 : cluster [DBG] 5.a deep-scrub starts 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.374077+0000 osd.2 (osd.2) 300 : cluster [DBG] 5.a deep-scrub ok 2024-02-23T00:44:27.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:26.552857+0000 mon.a (mon.0) 3963 : audit [INF] from='client.? 172.21.15.87:0/156348744' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "12"}]: dispatch 2024-02-23T00:44:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.607877+0000 osd.1 (osd.1) 376 : cluster [DBG] 22.9 scrub starts 2024-02-23T00:44:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.609294+0000 osd.1 (osd.1) 377 : cluster [DBG] 22.9 scrub ok 2024-02-23T00:44:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:27.266872+0000 mon.a (mon.0) 3964 : audit [INF] from='client.? 172.21.15.87:0/156348744' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "12"}]': finished 2024-02-23T00:44:27.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.266982+0000 mon.a (mon.0) 3965 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-02-23T00:44:28.147 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:28.147 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:28.191 INFO:tasks.thrashosds.thrasher:fixing pg num pool iscsi 2024-02-23T00:44:28.192 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd pool set iscsi pgp_num 23 2024-02-23T00:44:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.716041+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.e scrub starts 2024-02-23T00:44:28.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:26.719516+0000 osd.0 (osd.0) 70 : cluster [DBG] 5.e scrub ok 2024-02-23T00:44:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.060971+0000 osd.6 (osd.6) 350 : cluster [DBG] 3.10 scrub starts 2024-02-23T00:44:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.063901+0000 osd.6 (osd.6) 351 : cluster [DBG] 3.10 scrub ok 2024-02-23T00:44:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.280974+0000 osd.4 (osd.4) 359 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:44:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.282309+0000 osd.4 (osd.4) 360 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:44:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.328947+0000 osd.2 (osd.2) 301 : cluster [DBG] 5.d scrub starts 2024-02-23T00:44:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.330421+0000 osd.2 (osd.2) 302 : cluster [DBG] 5.d scrub ok 2024-02-23T00:44:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.590091+0000 osd.1 (osd.1) 378 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:44:28.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.593307+0000 osd.1 (osd.1) 379 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:44:28.466 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.716041+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.e scrub starts 2024-02-23T00:44:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:26.719516+0000 osd.0 (osd.0) 70 : cluster [DBG] 5.e scrub ok 2024-02-23T00:44:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.060971+0000 osd.6 (osd.6) 350 : cluster [DBG] 3.10 scrub starts 2024-02-23T00:44:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.063901+0000 osd.6 (osd.6) 351 : cluster [DBG] 3.10 scrub ok 2024-02-23T00:44:28.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.280974+0000 osd.4 (osd.4) 359 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.282309+0000 osd.4 (osd.4) 360 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.328947+0000 osd.2 (osd.2) 301 : cluster [DBG] 5.d scrub starts 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.330421+0000 osd.2 (osd.2) 302 : cluster [DBG] 5.d scrub ok 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.590091+0000 osd.1 (osd.1) 378 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.593307+0000 osd.1 (osd.1) 379 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.716041+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.e scrub starts 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:26.719516+0000 osd.0 (osd.0) 70 : cluster [DBG] 5.e scrub ok 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.060971+0000 osd.6 (osd.6) 350 : cluster [DBG] 3.10 scrub starts 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.063901+0000 osd.6 (osd.6) 351 : cluster [DBG] 3.10 scrub ok 2024-02-23T00:44:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.280974+0000 osd.4 (osd.4) 359 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:44:28.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.282309+0000 osd.4 (osd.4) 360 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:44:28.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.328947+0000 osd.2 (osd.2) 301 : cluster [DBG] 5.d scrub starts 2024-02-23T00:44:28.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.330421+0000 osd.2 (osd.2) 302 : cluster [DBG] 5.d scrub ok 2024-02-23T00:44:28.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.590091+0000 osd.1 (osd.1) 378 : cluster [DBG] 22.2 scrub starts 2024-02-23T00:44:28.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.593307+0000 osd.1 (osd.1) 379 : cluster [DBG] 22.2 scrub ok 2024-02-23T00:44:28.815 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-23T00:44:29.081 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:27.816399+0000 mgr.y (mgr.24437) 694 : cluster [DBG] pgmap v1256: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:28.106613+0000 osd.6 (osd.6) 352 : cluster [DBG] 3.a scrub starts 2024-02-23T00:44:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:28.109677+0000 osd.6 (osd.6) 353 : cluster [DBG] 3.a scrub ok 2024-02-23T00:44:29.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:28.356919+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.e scrub starts 2024-02-23T00:44:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:28.357940+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.e scrub ok 2024-02-23T00:44:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.036682+0000 osd.5 (osd.5) 323 : cluster [DBG] 4.8 deep-scrub starts 2024-02-23T00:44:29.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.039140+0000 osd.5 (osd.5) 324 : cluster [DBG] 4.8 deep-scrub ok 2024-02-23T00:44:29.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:27.816399+0000 mgr.y (mgr.24437) 694 : cluster [DBG] pgmap v1256: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:29.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:28.106613+0000 osd.6 (osd.6) 352 : cluster [DBG] 3.a scrub starts 2024-02-23T00:44:29.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:28.109677+0000 osd.6 (osd.6) 353 : cluster [DBG] 3.a scrub ok 2024-02-23T00:44:29.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:28.356919+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.e scrub starts 2024-02-23T00:44:29.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:28.357940+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.e scrub ok 2024-02-23T00:44:29.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.036682+0000 osd.5 (osd.5) 323 : cluster [DBG] 4.8 deep-scrub starts 2024-02-23T00:44:29.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.039140+0000 osd.5 (osd.5) 324 : cluster [DBG] 4.8 deep-scrub ok 2024-02-23T00:44:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:27.816399+0000 mgr.y (mgr.24437) 694 : cluster [DBG] pgmap v1256: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:28.106613+0000 osd.6 (osd.6) 352 : cluster [DBG] 3.a scrub starts 2024-02-23T00:44:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:28.109677+0000 osd.6 (osd.6) 353 : cluster [DBG] 3.a scrub ok 2024-02-23T00:44:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:28.356919+0000 osd.2 (osd.2) 303 : cluster [DBG] 2.e scrub starts 2024-02-23T00:44:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:28.357940+0000 osd.2 (osd.2) 304 : cluster [DBG] 2.e scrub ok 2024-02-23T00:44:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.036682+0000 osd.5 (osd.5) 323 : cluster [DBG] 4.8 deep-scrub starts 2024-02-23T00:44:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.039140+0000 osd.5 (osd.5) 324 : cluster [DBG] 4.8 deep-scrub ok 2024-02-23T00:44:30.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:28.718374+0000 osd.0 (osd.0) 71 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:44:30.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:28.722005+0000 osd.0 (osd.0) 72 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:44:30.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:28.950662+0000 osd.3 (osd.3) 267 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:44:30.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:28.954127+0000 osd.3 (osd.3) 268 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:44:30.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.343883+0000 osd.4 (osd.4) 361 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:44:30.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.345372+0000 osd.4 (osd.4) 362 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:44:30.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.593721+0000 osd.1 (osd.1) 380 : cluster [DBG] 22.13 deep-scrub starts 2024-02-23T00:44:30.486 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.596979+0000 osd.1 (osd.1) 381 : cluster [DBG] 22.13 deep-scrub ok 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.453+0000 7f7a37179700 1 -- 172.21.15.87:0/2324329976 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a30102180 msgr2=0x7f7a30104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.453+0000 7f7a37179700 1 --2- 172.21.15.87:0/2324329976 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a30102180 0x7f7a30104570 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 rx=0x7f7a28009be0 tx=0x7f7a28004d10).stop 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.453+0000 7f7a37179700 1 -- 172.21.15.87:0/2324329976 shutdown_connections 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.453+0000 7f7a37179700 1 --2- 172.21.15.87:0/2324329976 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a30102180 0x7f7a30104570 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.453+0000 7f7a37179700 1 --2- 172.21.15.87:0/2324329976 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a300f8c20 0x7f7a300f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.453+0000 7f7a37179700 1 --2- 172.21.15.87:0/2324329976 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a300f9580 0x7f7a30101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.453+0000 7f7a37179700 1 -- 172.21.15.87:0/2324329976 >> 172.21.15.87:0/2324329976 conn(0x7f7a300f26f0 msgr2=0x7f7a300f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.453+0000 7f7a37179700 1 -- 172.21.15.87:0/2324329976 shutdown_connections 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 -- 172.21.15.87:0/2324329976 wait complete. 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 librados: starting msgr at 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 librados: starting objecter 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 Processor -- start 2024-02-23T00:44:30.498 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 -- start start 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 librados: setting wanted keys 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 librados: calling monclient init 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a300f8c20 0x7f7a3010dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a300f9580 0x7f7a3010e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a30102180 0x7f7a30108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30108680 con 0x7f7a300f9580 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301087f0 con 0x7f7a300f8c20 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a37179700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30108960 con 0x7f7a30102180 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a2ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a300f9580 0x7f7a3010e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a2ffff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a300f9580 0x7f7a3010e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39444/0 (socket says 172.21.15.87:39444) 2024-02-23T00:44:30.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a2ffff700 1 -- 172.21.15.87:0/3918254298 learned_addr learned my addr 172.21.15.87:0/3918254298 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.454+0000 7f7a35716700 1 --2- 172.21.15.87:0/3918254298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a30102180 0x7f7a30108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a34f15700 1 --2- 172.21.15.87:0/3918254298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a300f8c20 0x7f7a3010dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2ffff700 1 -- 172.21.15.87:0/3918254298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a30102180 msgr2=0x7f7a30108190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2ffff700 1 --2- 172.21.15.87:0/3918254298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a30102180 0x7f7a30108190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2ffff700 1 -- 172.21.15.87:0/3918254298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a300f8c20 msgr2=0x7f7a3010dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2ffff700 1 --2- 172.21.15.87:0/3918254298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a300f8c20 0x7f7a3010dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2ffff700 1 -- 172.21.15.87:0/3918254298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a24005040 con 0x7f7a300f9580 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2ffff700 1 --2- 172.21.15.87:0/3918254298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a300f9580 0x7f7a3010e4a0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f7a1c0041d0 tx=0x7f7a1c009f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2dffb700 10 client.?.objecter ms_handle_connect 0x7f7a300f9580 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2dffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:30.500 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2dffb700 1 -- 172.21.15.87:0/3918254298 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c007b40 con 0x7f7a300f9580 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2dffb700 1 -- 172.21.15.87:0/3918254298 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a1c00f040 con 0x7f7a300f9580 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a280097c0 con 0x7f7a300f9580 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a2dffb700 1 -- 172.21.15.87:0/3918254298 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c014860 con 0x7f7a300f9580 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a37179700 10 client.27972.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a30108fa0 con 0x7f7a300f9580 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.455+0000 7f7a37179700 1 librados: init done 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.456+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a300606d0 con 0x7f7a300f9580 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a2dffb700 1 -- 172.21.15.87:0/3918254298 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f7a1c014c50 con 0x7f7a300f9580 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a2dffb700 1 --2- 172.21.15.87:0/3918254298 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7a20062c90 0x7f7a20065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a2dffb700 1 -- 172.21.15.87:0/3918254298 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(815..815 src has 400..815) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f7a1c07ca70 con 0x7f7a300f9580 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a2dffb700 10 client.27972.objecter ms_dispatch 0x7f7a30001550 osd_map(815..815 src has 400..815) v4 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a2dffb700 3 client.27972.objecter handle_osd_map got epochs [815,815] > 0 2024-02-23T00:44:30.501 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a2dffb700 3 client.27972.objecter handle_osd_map decoding full epoch 815 2024-02-23T00:44:30.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a34f15700 1 --2- 172.21.15.87:0/3918254298 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7a20062c90 0x7f7a20065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:30.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a2dffb700 20 client.27972.objecter dump_active .. 0 homeless 2024-02-23T00:44:30.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a2dffb700 10 client.27972.objecter ms_handle_connect 0x7f7a20062c90 2024-02-23T00:44:30.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.457+0000 7f7a34f15700 1 --2- 172.21.15.87:0/3918254298 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7a20062c90 0x7f7a20065150 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 rx=0x7f7a24009350 tx=0x7f7a2400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:30.502 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.461+0000 7f7a2dffb700 1 -- 172.21.15.87:0/3918254298 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f7a1c0501c0 con 0x7f7a300f9580 2024-02-23T00:44:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:28.718374+0000 osd.0 (osd.0) 71 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:44:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:28.722005+0000 osd.0 (osd.0) 72 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:44:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:28.950662+0000 osd.3 (osd.3) 267 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:44:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:28.954127+0000 osd.3 (osd.3) 268 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:44:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.343883+0000 osd.4 (osd.4) 361 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:44:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.345372+0000 osd.4 (osd.4) 362 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:44:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.593721+0000 osd.1 (osd.1) 380 : cluster [DBG] 22.13 deep-scrub starts 2024-02-23T00:44:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.596979+0000 osd.1 (osd.1) 381 : cluster [DBG] 22.13 deep-scrub ok 2024-02-23T00:44:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:28.718374+0000 osd.0 (osd.0) 71 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:44:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:28.722005+0000 osd.0 (osd.0) 72 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:44:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:28.950662+0000 osd.3 (osd.3) 267 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:44:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:28.954127+0000 osd.3 (osd.3) 268 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:44:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.343883+0000 osd.4 (osd.4) 361 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:44:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.345372+0000 osd.4 (osd.4) 362 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:44:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.593721+0000 osd.1 (osd.1) 380 : cluster [DBG] 22.13 deep-scrub starts 2024-02-23T00:44:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.596979+0000 osd.1 (osd.1) 381 : cluster [DBG] 22.13 deep-scrub ok 2024-02-23T00:44:30.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:30.629+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "23"} v 0) v1 -- 0x7f7a30107160 con 0x7f7a300f9580 2024-02-23T00:44:30.752 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:30.752 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:31.354 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-23T00:44:31.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.415+0000 7f7a2dffb700 1 -- 172.21.15.87:0/3918254298 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "23"}]=0 set pool 3 pgp_num to 23 v816) v1 ==== 132+0+0 (secure 0 0 0) 0x7f7a1c04fdb0 con 0x7f7a300f9580 2024-02-23T00:44:31.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.418+0000 7f7a37179700 10 librados: watch_flush enter 2024-02-23T00:44:31.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.418+0000 7f7a37179700 10 librados: watch_flush exit 2024-02-23T00:44:31.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.418+0000 7f7a37179700 20 client.27972.objecter shutdown clearing up homeless session... 2024-02-23T00:44:31.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.418+0000 7f7a37179700 10 client.27972.objecter successfully canceled tick 2024-02-23T00:44:31.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.419+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7a20062c90 msgr2=0x7f7a20065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:31.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.419+0000 7f7a37179700 1 --2- 172.21.15.87:0/3918254298 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7a20062c90 0x7f7a20065150 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 rx=0x7f7a24009350 tx=0x7f7a2400b040).stop 2024-02-23T00:44:31.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.419+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a300f9580 msgr2=0x7f7a3010e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:31.421 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.419+0000 7f7a37179700 1 --2- 172.21.15.87:0/3918254298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a300f9580 0x7f7a3010e4a0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 rx=0x7f7a1c0041d0 tx=0x7f7a1c009f60).stop 2024-02-23T00:44:31.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.421+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 shutdown_connections 2024-02-23T00:44:31.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.421+0000 7f7a37179700 1 --2- 172.21.15.87:0/3918254298 >> v2:172.21.15.87:6800/1231597463 conn(0x7f7a20062c90 0x7f7a20065150 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:31.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.421+0000 7f7a37179700 1 --2- 172.21.15.87:0/3918254298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7a30102180 0x7f7a30108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:31.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.421+0000 7f7a37179700 1 --2- 172.21.15.87:0/3918254298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f7a300f8c20 0x7f7a3010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:31.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.421+0000 7f7a37179700 1 --2- 172.21.15.87:0/3918254298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f7a300f9580 0x7f7a3010e4a0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:31.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.421+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 >> 172.21.15.87:0/3918254298 conn(0x7f7a300f26f0 msgr2=0x7f7a300f32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:31.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.421+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 shutdown_connections 2024-02-23T00:44:31.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.421+0000 7f7a37179700 1 -- 172.21.15.87:0/3918254298 wait complete. 2024-02-23T00:44:31.424 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:31.421+0000 7f7a37179700 1 librados: shutdown 2024-02-23T00:44:31.425 INFO:teuthology.orchestra.run.smithi087.stderr:set pool 3 pgp_num to 23 2024-02-23T00:44:31.624 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:31.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:44:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:44:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.693075+0000 osd.0 (osd.0) 73 : cluster [DBG] 22.18 deep-scrub starts 2024-02-23T00:44:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.693959+0000 osd.0 (osd.0) 74 : cluster [DBG] 22.18 deep-scrub ok 2024-02-23T00:44:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.816887+0000 mgr.y (mgr.24437) 695 : cluster [DBG] pgmap v1257: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.940606+0000 osd.3 (osd.3) 269 : cluster [DBG] 4.1d deep-scrub starts 2024-02-23T00:44:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:29.944798+0000 osd.3 (osd.3) 270 : cluster [DBG] 4.1d deep-scrub ok 2024-02-23T00:44:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:30.630801+0000 mon.a (mon.0) 3966 : audit [INF] from='client.? 172.21.15.87:0/3918254298' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "23"}]: dispatch 2024-02-23T00:44:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:31.055114+0000 osd.5 (osd.5) 325 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:44:31.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:31.056874+0000 osd.5 (osd.5) 326 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:44:31.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.693075+0000 osd.0 (osd.0) 73 : cluster [DBG] 22.18 deep-scrub starts 2024-02-23T00:44:31.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.693959+0000 osd.0 (osd.0) 74 : cluster [DBG] 22.18 deep-scrub ok 2024-02-23T00:44:31.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.816887+0000 mgr.y (mgr.24437) 695 : cluster [DBG] pgmap v1257: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:31.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.940606+0000 osd.3 (osd.3) 269 : cluster [DBG] 4.1d deep-scrub starts 2024-02-23T00:44:31.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:29.944798+0000 osd.3 (osd.3) 270 : cluster [DBG] 4.1d deep-scrub ok 2024-02-23T00:44:31.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:30.630801+0000 mon.a (mon.0) 3966 : audit [INF] from='client.? 172.21.15.87:0/3918254298' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "23"}]: dispatch 2024-02-23T00:44:31.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:31.055114+0000 osd.5 (osd.5) 325 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:44:31.827 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:31.056874+0000 osd.5 (osd.5) 326 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:44:31.828 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.693075+0000 osd.0 (osd.0) 73 : cluster [DBG] 22.18 deep-scrub starts 2024-02-23T00:44:31.828 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.693959+0000 osd.0 (osd.0) 74 : cluster [DBG] 22.18 deep-scrub ok 2024-02-23T00:44:31.828 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.816887+0000 mgr.y (mgr.24437) 695 : cluster [DBG] pgmap v1257: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:31.828 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.940606+0000 osd.3 (osd.3) 269 : cluster [DBG] 4.1d deep-scrub starts 2024-02-23T00:44:31.828 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:29.944798+0000 osd.3 (osd.3) 270 : cluster [DBG] 4.1d deep-scrub ok 2024-02-23T00:44:31.828 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:30.630801+0000 mon.a (mon.0) 3966 : audit [INF] from='client.? 172.21.15.87:0/3918254298' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "23"}]: dispatch 2024-02-23T00:44:31.828 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:31.055114+0000 osd.5 (osd.5) 325 : cluster [DBG] 4.15 scrub starts 2024-02-23T00:44:31.828 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:31.056874+0000 osd.5 (osd.5) 326 : cluster [DBG] 4.15 scrub ok 2024-02-23T00:44:32.241 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-23T00:44:32.516 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:32.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:32] "GET /metrics HTTP/1.1" 200 186063 "" "Prometheus/2.33.4" 2024-02-23T00:44:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:30.949259+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:44:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:30.952156+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:44:32.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:31.357182+0000 mgr.y (mgr.24437) 696 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:31.416780+0000 mon.a (mon.0) 3967 : audit [INF] from='client.? 172.21.15.87:0/3918254298' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "23"}]': finished 2024-02-23T00:44:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:31.416857+0000 mon.a (mon.0) 3968 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-02-23T00:44:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:31.561914+0000 osd.1 (osd.1) 382 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:44:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:31.563525+0000 osd.1 (osd.1) 383 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:44:32.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:32.268220+0000 mon.a (mon.0) 3969 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-02-23T00:44:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:30.949259+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:44:32.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:30.952156+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:44:32.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:31.357182+0000 mgr.y (mgr.24437) 696 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:32.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:31.416780+0000 mon.a (mon.0) 3967 : audit [INF] from='client.? 172.21.15.87:0/3918254298' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "23"}]': finished 2024-02-23T00:44:32.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:31.416857+0000 mon.a (mon.0) 3968 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-02-23T00:44:32.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:31.561914+0000 osd.1 (osd.1) 382 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:44:32.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:31.563525+0000 osd.1 (osd.1) 383 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:44:32.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:32.268220+0000 mon.a (mon.0) 3969 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-02-23T00:44:32.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:30.949259+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:44:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:30.952156+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:44:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:31.357182+0000 mgr.y (mgr.24437) 696 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:31.416780+0000 mon.a (mon.0) 3967 : audit [INF] from='client.? 172.21.15.87:0/3918254298' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "23"}]': finished 2024-02-23T00:44:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:31.416857+0000 mon.a (mon.0) 3968 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-02-23T00:44:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:31.561914+0000 osd.1 (osd.1) 382 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:44:32.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:31.563525+0000 osd.1 (osd.1) 383 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:44:32.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:32.268220+0000 mon.a (mon.0) 3969 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-02-23T00:44:33.390 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:33.390 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:31.817431+0000 mgr.y (mgr.24437) 697 : cluster [DBG] pgmap v1259: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:31.976119+0000 osd.3 (osd.3) 273 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:44:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:31.979316+0000 osd.3 (osd.3) 274 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:44:33.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:32.100740+0000 osd.6 (osd.6) 354 : cluster [DBG] 22.5 deep-scrub starts 2024-02-23T00:44:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:32.101905+0000 osd.6 (osd.6) 355 : cluster [DBG] 22.5 deep-scrub ok 2024-02-23T00:44:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:32.543687+0000 osd.1 (osd.1) 384 : cluster [DBG] 2.a scrub starts 2024-02-23T00:44:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:32.546603+0000 osd.1 (osd.1) 385 : cluster [DBG] 2.a scrub ok 2024-02-23T00:44:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:33.152832+0000 osd.7 (osd.7) 382 : cluster [DBG] 3.f scrub starts 2024-02-23T00:44:33.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:33.155825+0000 osd.7 (osd.7) 383 : cluster [DBG] 3.f scrub ok 2024-02-23T00:44:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:31.817431+0000 mgr.y (mgr.24437) 697 : cluster [DBG] pgmap v1259: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:31.976119+0000 osd.3 (osd.3) 273 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:44:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:31.979316+0000 osd.3 (osd.3) 274 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:44:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:32.100740+0000 osd.6 (osd.6) 354 : cluster [DBG] 22.5 deep-scrub starts 2024-02-23T00:44:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:32.101905+0000 osd.6 (osd.6) 355 : cluster [DBG] 22.5 deep-scrub ok 2024-02-23T00:44:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:32.543687+0000 osd.1 (osd.1) 384 : cluster [DBG] 2.a scrub starts 2024-02-23T00:44:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:32.546603+0000 osd.1 (osd.1) 385 : cluster [DBG] 2.a scrub ok 2024-02-23T00:44:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:33.152832+0000 osd.7 (osd.7) 382 : cluster [DBG] 3.f scrub starts 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:33.155825+0000 osd.7 (osd.7) 383 : cluster [DBG] 3.f scrub ok 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:31.817431+0000 mgr.y (mgr.24437) 697 : cluster [DBG] pgmap v1259: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:31.976119+0000 osd.3 (osd.3) 273 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:31.979316+0000 osd.3 (osd.3) 274 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:32.100740+0000 osd.6 (osd.6) 354 : cluster [DBG] 22.5 deep-scrub starts 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:32.101905+0000 osd.6 (osd.6) 355 : cluster [DBG] 22.5 deep-scrub ok 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:32.543687+0000 osd.1 (osd.1) 384 : cluster [DBG] 2.a scrub starts 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:32.546603+0000 osd.1 (osd.1) 385 : cluster [DBG] 2.a scrub ok 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:33.152832+0000 osd.7 (osd.7) 382 : cluster [DBG] 3.f scrub starts 2024-02-23T00:44:33.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:33.155825+0000 osd.7 (osd.7) 383 : cluster [DBG] 3.f scrub ok 2024-02-23T00:44:34.083 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:44:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:44:34.102 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-23T00:44:34.372 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:34.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.598+0000 7f7789a64700 1 -- 172.21.15.87:0/233968345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe510 msgr2=0x7f77840f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.598+0000 7f7789a64700 1 --2- 172.21.15.87:0/233968345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe510 0x7f77840f7d40 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 rx=0x7f7778004870 tx=0x7f7778009cb0).stop 2024-02-23T00:44:34.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.598+0000 7f7789a64700 1 -- 172.21.15.87:0/233968345 shutdown_connections 2024-02-23T00:44:34.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.598+0000 7f7789a64700 1 --2- 172.21.15.87:0/233968345 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77840f8420 0x7f7784104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.598+0000 7f7789a64700 1 --2- 172.21.15.87:0/233968345 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840fdbb0 0x7f77840fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.640 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.598+0000 7f7789a64700 1 --2- 172.21.15.87:0/233968345 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe510 0x7f77840f7d40 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.598+0000 7f7789a64700 1 -- 172.21.15.87:0/233968345 >> 172.21.15.87:0/233968345 conn(0x7f77840f3800 msgr2=0x7f77840f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.598+0000 7f7789a64700 1 -- 172.21.15.87:0/233968345 shutdown_connections 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.598+0000 7f7789a64700 1 -- 172.21.15.87:0/233968345 wait complete. 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 librados: starting msgr at 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 librados: starting objecter 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 Processor -- start 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 -- start start 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 librados: setting wanted keys 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 librados: calling monclient init 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840f8420 0x7f778406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77840fdbb0 0x7f778406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe510 0x7f778406bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7784116460 con 0x7f77840fe510 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77841165d0 con 0x7f77840f8420 2024-02-23T00:44:34.641 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7789a64700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7784116790 con 0x7f77840fdbb0 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7783fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe510 0x7f778406bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7782ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77840fdbb0 0x7f778406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7782ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77840fdbb0 0x7f778406b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:38178/0 (socket says 172.21.15.87:38178) 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7783fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe510 0x7f778406bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60184/0 (socket says 172.21.15.87:60184) 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.599+0000 7f7782ffd700 1 -- 172.21.15.87:0/324507271 learned_addr learned my addr 172.21.15.87:0/324507271 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f77837fe700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840f8420 0x7f778406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7782ffd700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840f8420 msgr2=0x7f778406b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7782ffd700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840f8420 0x7f778406b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7782ffd700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe510 msgr2=0x7f778406bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7782ffd700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe510 0x7f778406bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7782ffd700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0097c0 con 0x7f77840fdbb0 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe510 0x7f778406bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f77837fe700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840f8420 0x7f778406b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:44:34.642 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7782ffd700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77840fdbb0 0x7f778406b5c0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f7778009eb0 tx=0x7f777800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7780f74700 10 client.?.objecter ms_handle_connect 0x7f77840fdbb0 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7780f74700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777800e070 con 0x7f77840fdbb0 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7778005040 con 0x7f77840fdbb0 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7778013da0 con 0x7f77840fdbb0 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778019070 con 0x7f77840fdbb0 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7789a64700 10 client.38453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7784116ce0 con 0x7f77840fdbb0 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.600+0000 7f7789a64700 1 librados: init done 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.601+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f778405eec0 con 0x7f77840fdbb0 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.601+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f777801e420 con 0x7f77840fdbb0 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> v2:172.21.15.87:6800/1231597463 conn(0x7f776c062c90 0x7f776c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.643 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(817..817 src has 400..817) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f777807dba0 con 0x7f77840fdbb0 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 10 client.38453.objecter ms_dispatch 0x7f7784001550 osd_map(817..817 src has 400..817) v4 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 3 client.38453.objecter handle_osd_map got epochs [817,817] > 0 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f77837fe700 1 --2- 172.21.15.87:0/324507271 >> v2:172.21.15.87:6800/1231597463 conn(0x7f776c062c90 0x7f776c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 3 client.38453.objecter handle_osd_map decoding full epoch 817 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 20 client.38453.objecter dump_active .. 0 homeless 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1259 (secure 0 0 0) 0x7f7778082220 con 0x7f77840fdbb0 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 10 client.38453.objecter ms_handle_connect 0x7f776c062c90 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f77837fe700 1 --2- 172.21.15.87:0/324507271 >> v2:172.21.15.87:6800/1231597463 conn(0x7f776c062c90 0x7f776c065150 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7f7774009250 tx=0x7f7774009de0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7789a64700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe8f0 0x7f77840fecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778406c5b0 con 0x7f77840fe8f0 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe8f0 0x7f77840fecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.644 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7783fff700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7f778405fa60 con 0x7f77840fe8f0 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 10 client.38453.objecter ms_handle_connect 0x7f77840fe8f0 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7780f74700 10 client.38453.objecter resend_mon_ops 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.602+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe8f0 0x7f77840fecd0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f777c004fd0 tx=0x7f777c004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.603+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c004020 con 0x7f77840fe8f0 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.603+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.0 v2:172.21.15.87:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f777c00f030 con 0x7f77840fe8f0 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.603+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe8f0 msgr2=0x7f77840fecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.603+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe8f0 0x7f77840fecd0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 rx=0x7f777c004fd0 tx=0x7f777c004c50).stop 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.603+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe8f0 msgr2=0x7f77840fecd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.603+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840fe8f0 0x7f77840fecd0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.609+0000 7f7789a64700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840ff1c0 0x7f77840ff5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.609+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778406c6f0 con 0x7f77840ff1c0 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.609+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840ff1c0 0x7f77840ff5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.610+0000 7f7783fff700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f778411af20 con 0x7f77840ff1c0 2024-02-23T00:44:34.645 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.610+0000 7f7780f74700 10 client.38453.objecter ms_handle_connect 0x7f77840ff1c0 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.610+0000 7f7780f74700 10 client.38453.objecter resend_mon_ops 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.610+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840ff1c0 0x7f77840ff5a0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7f777c004fd0 tx=0x7f777c019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.610+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c015d80 con 0x7f77840ff1c0 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.611+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.0 v2:172.21.15.87:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (secure 0 0 0) 0x7f777c0077a0 con 0x7f77840ff1c0 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.611+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840ff1c0 msgr2=0x7f77840ff5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.611+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840ff1c0 0x7f77840ff5a0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 rx=0x7f777c004fd0 tx=0x7f777c019000).stop 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.611+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840ff1c0 msgr2=0x7f77840ff5a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.611+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f77840ff1c0 0x7f77840ff5a0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.611+0000 7f7789a64700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840273c0 0x7f77840277d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.611+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7784027d50 con 0x7f77840273c0 2024-02-23T00:44:34.646 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840273c0 0x7f77840277d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.647 INFO:teuthology.orchestra.run.smithi087.stdout:mon.a: {} 2024-02-23T00:44:34.647 INFO:teuthology.orchestra.run.smithi087.stdout:mon.b: {} 2024-02-23T00:44:34.647 INFO:teuthology.orchestra.run.smithi087.stdout:mon.c: {} 2024-02-23T00:44:34.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7783fff700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7784027f10 con 0x7f77840273c0 2024-02-23T00:44:34.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840273c0 0x7f77840277d0 secure :-1 s=READY pgs=4994 cs=0 l=1 rev1=1 rx=0x7f777c0044b0 tx=0x7f777c024000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:34.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7780f74700 10 client.38453.objecter ms_handle_connect 0x7f77840273c0 2024-02-23T00:44:34.647 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7780f74700 10 client.38453.objecter resend_mon_ops 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c015520 con 0x7f77840273c0 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.1 v2:172.21.15.171:3300/0 2 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f777c015ab0 con 0x7f77840273c0 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840273c0 msgr2=0x7f77840277d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840273c0 0x7f77840277d0 secure :-1 s=READY pgs=4994 cs=0 l=1 rev1=1 rx=0x7f777c0044b0 tx=0x7f777c024000).stop 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840273c0 msgr2=0x7f77840277d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7783fff700 1 -- 172.21.15.87:0/324507271 reap_dead start 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.612+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f77840273c0 0x7f77840277d0 unknown :-1 s=CLOSED pgs=4994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.619+0000 7f7789a64700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f778406bab0 0x7f77840280e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.619+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778406b280 con 0x7f778406bab0 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.619+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f778406bab0 0x7f77840280e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.619+0000 7f7783fff700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f778406b3c0 con 0x7f778406bab0 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.620+0000 7f7780f74700 10 client.38453.objecter ms_handle_connect 0x7f778406bab0 2024-02-23T00:44:34.648 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.620+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f778406bab0 0x7f77840280e0 secure :-1 s=READY pgs=4995 cs=0 l=1 rev1=1 rx=0x7f777c004c90 tx=0x7f777c032000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.620+0000 7f7780f74700 10 client.38453.objecter resend_mon_ops 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.620+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.1 v2:172.21.15.171:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c031920 con 0x7f778406bab0 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.621+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.1 v2:172.21.15.171:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (secure 0 0 0) 0x7f777c0093b0 con 0x7f778406bab0 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.621+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f778406bab0 msgr2=0x7f77840280e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.621+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f778406bab0 0x7f77840280e0 secure :-1 s=READY pgs=4995 cs=0 l=1 rev1=1 rx=0x7f777c004c90 tx=0x7f777c032000).stop 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.621+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f778406bab0 msgr2=0x7f77840280e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.621+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f778406bab0 0x7f77840280e0 unknown :-1 s=CLOSED pgs=4995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7789a64700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114400 0x7f77841147e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77841162b0 con 0x7f7784114400 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114400 0x7f77841147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7783fff700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77840286c0 con 0x7f7784114400 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7780f74700 10 client.38453.objecter ms_handle_connect 0x7f7784114400 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7780f74700 10 client.38453.objecter resend_mon_ops 2024-02-23T00:44:34.649 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114400 0x7f77841147e0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7f777c004dc0 tx=0x7f777c03f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c04a430 con 0x7f7784114400 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 2 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8167 (secure 0 0 0) 0x7f777c031650 con 0x7f7784114400 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114400 msgr2=0x7f77841147e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114400 0x7f77841147e0 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 rx=0x7f777c004dc0 tx=0x7f777c03f000).stop 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114400 msgr2=0x7f77841147e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.622+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114400 0x7f77841147e0 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.629+0000 7f7789a64700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114cd0 0x7f7784121150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.629+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77841216e0 con 0x7f7784114cd0 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.629+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114cd0 0x7f7784121150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.629+0000 7f7783fff700 1 -- 172.21.15.87:0/324507271 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7f7784121850 con 0x7f7784114cd0 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.629+0000 7f7783fff700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114cd0 0x7f7784121150 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7f777c007f80 tx=0x7f777c04b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.629+0000 7f7780f74700 10 client.38453.objecter ms_handle_connect 0x7f7784114cd0 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.629+0000 7f7780f74700 10 client.38453.objecter resend_mon_ops 2024-02-23T00:44:34.650 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.629+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c03e6d0 con 0x7f7784114cd0 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.630+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 <== mon.2 v2:172.21.15.87:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (secure 0 0 0) 0x7f777c032d20 con 0x7f7784114cd0 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.630+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114cd0 msgr2=0x7f7784121150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.630+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114cd0 0x7f7784121150 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 rx=0x7f777c007f80 tx=0x7f777c04b000).stop 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.630+0000 7f7780f74700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114cd0 msgr2=0x7f7784121150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.630+0000 7f7780f74700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f7784114cd0 0x7f7784121150 secure :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 rx=0x7f777c007f80 tx=0x7f777c04b000).stop 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 10 librados: watch_flush enter 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 10 librados: watch_flush exit 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 20 client.38453.objecter shutdown clearing up homeless session... 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 10 client.38453.objecter successfully canceled tick 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 >> v2:172.21.15.87:6800/1231597463 conn(0x7f776c062c90 msgr2=0x7f776c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 1 --2- 172.21.15.87:0/324507271 >> v2:172.21.15.87:6800/1231597463 conn(0x7f776c062c90 0x7f776c065150 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 rx=0x7f7774009250 tx=0x7f7774009de0).stop 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77840fdbb0 msgr2=0x7f778406b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 1 --2- 172.21.15.87:0/324507271 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f77840fdbb0 0x7f778406b5c0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f7778009eb0 tx=0x7f777800b040).stop 2024-02-23T00:44:34.651 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7783fff700 1 -- 172.21.15.87:0/324507271 reap_dead start 2024-02-23T00:44:34.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 shutdown_connections 2024-02-23T00:44:34.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 >> 172.21.15.87:0/324507271 conn(0x7f77840f3800 msgr2=0x7f77840fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:34.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 shutdown_connections 2024-02-23T00:44:34.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 1 -- 172.21.15.87:0/324507271 wait complete. 2024-02-23T00:44:34.652 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:34.631+0000 7f7789a64700 1 librados: shutdown 2024-02-23T00:44:34.652 INFO:teuthology.orchestra.run.smithi087.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-23T00:44:34.652 INFO:teuthology.orchestra.run.smithi087.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-23T00:44:34.652 INFO:teuthology.orchestra.run.smithi087.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:33.288927+0000 osd.2 (osd.2) 305 : cluster [DBG] 2.1 deep-scrub starts 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:33.289748+0000 osd.2 (osd.2) 306 : cluster [DBG] 2.1 deep-scrub ok 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:33.817913+0000 mgr.y (mgr.24437) 698 : cluster [DBG] pgmap v1261: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:34.156879+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.e scrub starts 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:34.159770+0000 osd.7 (osd.7) 385 : cluster [DBG] 3.e scrub ok 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:33.288927+0000 osd.2 (osd.2) 305 : cluster [DBG] 2.1 deep-scrub starts 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:33.289748+0000 osd.2 (osd.2) 306 : cluster [DBG] 2.1 deep-scrub ok 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:33.817913+0000 mgr.y (mgr.24437) 698 : cluster [DBG] pgmap v1261: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:34.156879+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.e scrub starts 2024-02-23T00:44:34.654 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:34.159770+0000 osd.7 (osd.7) 385 : cluster [DBG] 3.e scrub ok 2024-02-23T00:44:34.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:33.288927+0000 osd.2 (osd.2) 305 : cluster [DBG] 2.1 deep-scrub starts 2024-02-23T00:44:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:33.289748+0000 osd.2 (osd.2) 306 : cluster [DBG] 2.1 deep-scrub ok 2024-02-23T00:44:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:33.817913+0000 mgr.y (mgr.24437) 698 : cluster [DBG] pgmap v1261: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-02-23T00:44:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:34.156879+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.e scrub starts 2024-02-23T00:44:34.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:34.159770+0000 osd.7 (osd.7) 385 : cluster [DBG] 3.e scrub ok 2024-02-23T00:44:35.305 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight 1 1 2024-02-23T00:44:35.576 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:33.921718+0000 osd.3 (osd.3) 275 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:44:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:33.924942+0000 osd.3 (osd.3) 276 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:44:35.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:34.434317+0000 osd.4 (osd.4) 363 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:44:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:34.435074+0000 osd.4 (osd.4) 364 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:44:35.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:34.603213+0000 mon.c (mon.2) 508 : audit [DBG] from='client.? 172.21.15.87:0/324507271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:44:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:33.921718+0000 osd.3 (osd.3) 275 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:44:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:33.924942+0000 osd.3 (osd.3) 276 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:44:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:34.434317+0000 osd.4 (osd.4) 363 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:44:35.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:34.435074+0000 osd.4 (osd.4) 364 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:44:35.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:34.603213+0000 mon.c (mon.2) 508 : audit [DBG] from='client.? 172.21.15.87:0/324507271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:44:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:33.921718+0000 osd.3 (osd.3) 275 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:44:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:33.924942+0000 osd.3 (osd.3) 276 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:44:35.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:34.434317+0000 osd.4 (osd.4) 363 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:44:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:34.435074+0000 osd.4 (osd.4) 364 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:44:35.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:34.603213+0000 mon.c (mon.2) 508 : audit [DBG] from='client.? 172.21.15.87:0/324507271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-23T00:44:36.257 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:36.257 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:36.598 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:35.818645+0000 mgr.y (mgr.24437) 699 : cluster [DBG] pgmap v1262: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:44:36.599 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:35.885387+0000 osd.5 (osd.5) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:44:36.599 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:35.885729+0000 osd.5 (osd.5) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:44:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:35.818645+0000 mgr.y (mgr.24437) 699 : cluster [DBG] pgmap v1262: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:44:36.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:35.885387+0000 osd.5 (osd.5) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:44:36.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:35.885729+0000 osd.5 (osd.5) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:44:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:35.818645+0000 mgr.y (mgr.24437) 699 : cluster [DBG] pgmap v1262: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:44:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:35.885387+0000 osd.5 (osd.5) 327 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:44:36.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:35.885729+0000 osd.5 (osd.5) 328 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:44:36.941 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-23T00:44:37.211 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:37.541 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.536+0000 7f1082e9f700 1 -- 172.21.15.87:0/3190521679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 msgr2=0x7f107c103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.536+0000 7f1082e9f700 1 --2- 172.21.15.87:0/3190521679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 0x7f107c103a20 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 rx=0x7f1074009be0 tx=0x7f1074004d10).stop 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.536+0000 7f1082e9f700 1 -- 172.21.15.87:0/3190521679 shutdown_connections 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.536+0000 7f1082e9f700 1 --2- 172.21.15.87:0/3190521679 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 0x7f107c103a20 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.536+0000 7f1082e9f700 1 --2- 172.21.15.87:0/3190521679 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f107c0fcbb0 0x7f107c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.536+0000 7f1082e9f700 1 --2- 172.21.15.87:0/3190521679 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f107c0fd510 0x7f107c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.536+0000 7f1082e9f700 1 -- 172.21.15.87:0/3190521679 >> 172.21.15.87:0/3190521679 conn(0x7f107c0f2820 msgr2=0x7f107c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.536+0000 7f1082e9f700 1 -- 172.21.15.87:0/3190521679 shutdown_connections 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.536+0000 7f1082e9f700 1 -- 172.21.15.87:0/3190521679 wait complete. 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 librados: starting msgr at 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 librados: starting objecter 2024-02-23T00:44:37.542 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 Processor -- start 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 -- start start 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 librados: setting wanted keys 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 librados: calling monclient init 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 0x7f107c0f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f107c0fcbb0 0x7f107c0f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f107c0fd510 0x7f107c0f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c116430 con 0x7f107c0f7400 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c1165a0 con 0x7f107c0fcbb0 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1082e9f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c116760 con 0x7f107c0fd510 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1080c3b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 0x7f107c0f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f108143c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f107c0fd510 0x7f107c0f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1080c3b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 0x7f107c0f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60232/0 (socket says 172.21.15.87:60232) 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.537+0000 7f1080c3b700 1 -- 172.21.15.87:0/2547325356 learned_addr learned my addr 172.21.15.87:0/2547325356 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f107bfff700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f107c0fcbb0 0x7f107c0f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:37.543 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1080c3b700 1 -- 172.21.15.87:0/2547325356 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f107c0fd510 msgr2=0x7f107c0f9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1080c3b700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f107c0fd510 0x7f107c0f9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1080c3b700 1 -- 172.21.15.87:0/2547325356 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f107c0fcbb0 msgr2=0x7f107c0f8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1080c3b700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f107c0fcbb0 0x7f107c0f8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1080c3b700 1 -- 172.21.15.87:0/2547325356 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10740097c0 con 0x7f107c0f7400 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f108143c700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f107c0fd510 0x7f107c0f9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f107bfff700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f107c0fcbb0 0x7f107c0f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1080c3b700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 0x7f107c0f89b0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 rx=0x7f107000c810 tx=0x7f1070004330).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1079ffb700 10 client.?.objecter ms_handle_connect 0x7f107c0f7400 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1079ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1079ffb700 1 -- 172.21.15.87:0/2547325356 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10700043a0 con 0x7f107c0f7400 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1079ffb700 1 -- 172.21.15.87:0/2547325356 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10700109e0 con 0x7f107c0f7400 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c116a40 con 0x7f107c0f7400 2024-02-23T00:44:37.544 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1079ffb700 1 -- 172.21.15.87:0/2547325356 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070012ab0 con 0x7f107c0f7400 2024-02-23T00:44:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1082e9f700 10 client.27990.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c116f80 con 0x7f107c0f7400 2024-02-23T00:44:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.538+0000 7f1082e9f700 1 librados: init done 2024-02-23T00:44:37.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.539+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c05fa60 con 0x7f107c0f7400 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f1079ffb700 1 -- 172.21.15.87:0/2547325356 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1070010b50 con 0x7f107c0f7400 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f1079ffb700 1 --2- 172.21.15.87:0/2547325356 >> v2:172.21.15.87:6800/1231597463 conn(0x7f106c062c90 0x7f106c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f1079ffb700 1 -- 172.21.15.87:0/2547325356 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(819..819 src has 400..819) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f107007ce10 con 0x7f107c0f7400 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f1079ffb700 10 client.27990.objecter ms_dispatch 0x7f107c001550 osd_map(819..819 src has 400..819) v4 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f1079ffb700 3 client.27990.objecter handle_osd_map got epochs [819,819] > 0 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f1079ffb700 3 client.27990.objecter handle_osd_map decoding full epoch 819 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f107bfff700 1 --2- 172.21.15.87:0/2547325356 >> v2:172.21.15.87:6800/1231597463 conn(0x7f106c062c90 0x7f106c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f1079ffb700 20 client.27990.objecter dump_active .. 0 homeless 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f1079ffb700 10 client.27990.objecter ms_handle_connect 0x7f106c062c90 2024-02-23T00:44:37.546 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.540+0000 7f107bfff700 1 --2- 172.21.15.87:0/2547325356 >> v2:172.21.15.87:6800/1231597463 conn(0x7f106c062c90 0x7f106c065150 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7f1068007fd0 tx=0x7f106800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:37.547 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.544+0000 7f1079ffb700 1 -- 172.21.15.87:0/2547325356 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1070050600 con 0x7f107c0f7400 2024-02-23T00:44:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:35.904555+0000 osd.2 (osd.2) 307 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:44:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:35.904753+0000 osd.2 (osd.2) 308 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:44:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:36.045367+0000 osd.6 (osd.6) 356 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:44:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:36.048272+0000 osd.6 (osd.6) 357 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:44:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:36.441995+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-02-23T00:44:37.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.195000+0000 osd.7 (osd.7) 386 : cluster [DBG] 3.13 scrub starts 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.195911+0000 osd.7 (osd.7) 387 : cluster [DBG] 3.13 scrub ok 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.271917+0000 mon.a (mon.0) 3971 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:35.904555+0000 osd.2 (osd.2) 307 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:35.904753+0000 osd.2 (osd.2) 308 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:36.045367+0000 osd.6 (osd.6) 356 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:36.048272+0000 osd.6 (osd.6) 357 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:36.441995+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.195000+0000 osd.7 (osd.7) 386 : cluster [DBG] 3.13 scrub starts 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.195911+0000 osd.7 (osd.7) 387 : cluster [DBG] 3.13 scrub ok 2024-02-23T00:44:37.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.271917+0000 mon.a (mon.0) 3971 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-02-23T00:44:37.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:35.904555+0000 osd.2 (osd.2) 307 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:44:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:35.904753+0000 osd.2 (osd.2) 308 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:44:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:36.045367+0000 osd.6 (osd.6) 356 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:44:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:36.048272+0000 osd.6 (osd.6) 357 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:44:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:36.441995+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-02-23T00:44:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.195000+0000 osd.7 (osd.7) 386 : cluster [DBG] 3.13 scrub starts 2024-02-23T00:44:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.195911+0000 osd.7 (osd.7) 387 : cluster [DBG] 3.13 scrub ok 2024-02-23T00:44:37.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.271917+0000 mon.a (mon.0) 3971 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-02-23T00:44:37.723 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:37.721+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f107c05ee80 con 0x7f107c0f7400 2024-02-23T00:44:38.444 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.442+0000 7f1079ffb700 1 -- 172.21.15.87:0/2547325356 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v820) v1 ==== 113+0+0 (secure 0 0 0) 0x7f10700501f0 con 0x7f107c0f7400 2024-02-23T00:44:38.447 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 10 librados: watch_flush enter 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 10 librados: watch_flush exit 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 20 client.27990.objecter shutdown clearing up homeless session... 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 10 client.27990.objecter successfully canceled tick 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 >> v2:172.21.15.87:6800/1231597463 conn(0x7f106c062c90 msgr2=0x7f106c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 --2- 172.21.15.87:0/2547325356 >> v2:172.21.15.87:6800/1231597463 conn(0x7f106c062c90 0x7f106c065150 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 rx=0x7f1068007fd0 tx=0x7f106800b040).stop 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 msgr2=0x7f107c0f89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 0x7f107c0f89b0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 rx=0x7f107000c810 tx=0x7f1070004330).stop 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 shutdown_connections 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 --2- 172.21.15.87:0/2547325356 >> v2:172.21.15.87:6800/1231597463 conn(0x7f106c062c90 0x7f106c065150 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f107c0fd510 0x7f107c0f9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f107c0f7400 0x7f107c0f89b0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:38.448 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 --2- 172.21.15.87:0/2547325356 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f107c0fcbb0 0x7f107c0f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:38.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 >> 172.21.15.87:0/2547325356 conn(0x7f107c0f2820 msgr2=0x7f107c060db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:38.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 shutdown_connections 2024-02-23T00:44:38.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 -- 172.21.15.87:0/2547325356 wait complete. 2024-02-23T00:44:38.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:38.445+0000 7f1082e9f700 1 librados: shutdown 2024-02-23T00:44:38.449 INFO:teuthology.orchestra.run.smithi087.stderr:reweighted osd.1 to 1 (10000) 2024-02-23T00:44:38.664 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:36.765636+0000 osd.0 (osd.0) 75 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:44:38.664 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:36.768715+0000 osd.0 (osd.0) 76 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:44:38.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.051447+0000 osd.6 (osd.6) 358 : cluster [DBG] 5.11 deep-scrub starts 2024-02-23T00:44:38.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.054338+0000 osd.6 (osd.6) 359 : cluster [DBG] 5.11 deep-scrub ok 2024-02-23T00:44:38.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.638142+0000 osd.1 (osd.1) 386 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:44:38.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.639934+0000 osd.1 (osd.1) 387 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:44:38.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:37.722779+0000 mon.a (mon.0) 3972 : audit [INF] from='client.? 172.21.15.87:0/2547325356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:44:38.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.819032+0000 mgr.y (mgr.24437) 700 : cluster [DBG] pgmap v1265: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:38.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:38.146448+0000 osd.7 (osd.7) 388 : cluster [DBG] 3.14 scrub starts 2024-02-23T00:44:38.665 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:38.147287+0000 osd.7 (osd.7) 389 : cluster [DBG] 3.14 scrub ok 2024-02-23T00:44:38.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:36.765636+0000 osd.0 (osd.0) 75 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:44:38.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:36.768715+0000 osd.0 (osd.0) 76 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:44:38.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.051447+0000 osd.6 (osd.6) 358 : cluster [DBG] 5.11 deep-scrub starts 2024-02-23T00:44:38.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.054338+0000 osd.6 (osd.6) 359 : cluster [DBG] 5.11 deep-scrub ok 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.638142+0000 osd.1 (osd.1) 386 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.639934+0000 osd.1 (osd.1) 387 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:37.722779+0000 mon.a (mon.0) 3972 : audit [INF] from='client.? 172.21.15.87:0/2547325356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.819032+0000 mgr.y (mgr.24437) 700 : cluster [DBG] pgmap v1265: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:38.146448+0000 osd.7 (osd.7) 388 : cluster [DBG] 3.14 scrub starts 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:38.147287+0000 osd.7 (osd.7) 389 : cluster [DBG] 3.14 scrub ok 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:36.765636+0000 osd.0 (osd.0) 75 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:36.768715+0000 osd.0 (osd.0) 76 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.051447+0000 osd.6 (osd.6) 358 : cluster [DBG] 5.11 deep-scrub starts 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.054338+0000 osd.6 (osd.6) 359 : cluster [DBG] 5.11 deep-scrub ok 2024-02-23T00:44:38.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.638142+0000 osd.1 (osd.1) 386 : cluster [DBG] 2.9 scrub starts 2024-02-23T00:44:38.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.639934+0000 osd.1 (osd.1) 387 : cluster [DBG] 2.9 scrub ok 2024-02-23T00:44:38.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:37.722779+0000 mon.a (mon.0) 3972 : audit [INF] from='client.? 172.21.15.87:0/2547325356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:44:38.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.819032+0000 mgr.y (mgr.24437) 700 : cluster [DBG] pgmap v1265: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:38.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:38.146448+0000 osd.7 (osd.7) 388 : cluster [DBG] 3.14 scrub starts 2024-02-23T00:44:38.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:38.147287+0000 osd.7 (osd.7) 389 : cluster [DBG] 3.14 scrub ok 2024-02-23T00:44:38.940 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:38.940 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:39.331 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-23T00:44:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.761286+0000 osd.0 (osd.0) 77 : cluster [DBG] 3.12 scrub starts 2024-02-23T00:44:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.762129+0000 osd.0 (osd.0) 78 : cluster [DBG] 3.12 scrub ok 2024-02-23T00:44:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.953271+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.b scrub starts 2024-02-23T00:44:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:37.956385+0000 osd.3 (osd.3) 278 : cluster [DBG] 6.b scrub ok 2024-02-23T00:44:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:38.034388+0000 osd.6 (osd.6) 360 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:44:39.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:38.035195+0000 osd.6 (osd.6) 361 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:38.443375+0000 mon.a (mon.0) 3973 : audit [INF] from='client.? 172.21.15.87:0/2547325356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:38.443451+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.097101+0000 osd.5 (osd.5) 329 : cluster [DBG] 22.f scrub starts 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.100125+0000 osd.5 (osd.5) 330 : cluster [DBG] 22.f scrub ok 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.117894+0000 osd.7 (osd.7) 390 : cluster [DBG] 3.15 scrub starts 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.120772+0000 osd.7 (osd.7) 391 : cluster [DBG] 3.15 scrub ok 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.761286+0000 osd.0 (osd.0) 77 : cluster [DBG] 3.12 scrub starts 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.762129+0000 osd.0 (osd.0) 78 : cluster [DBG] 3.12 scrub ok 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.953271+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.b scrub starts 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:37.956385+0000 osd.3 (osd.3) 278 : cluster [DBG] 6.b scrub ok 2024-02-23T00:44:39.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:38.034388+0000 osd.6 (osd.6) 360 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:44:39.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:38.035195+0000 osd.6 (osd.6) 361 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:44:39.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:38.443375+0000 mon.a (mon.0) 3973 : audit [INF] from='client.? 172.21.15.87:0/2547325356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-23T00:44:39.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:38.443451+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-23T00:44:39.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.097101+0000 osd.5 (osd.5) 329 : cluster [DBG] 22.f scrub starts 2024-02-23T00:44:39.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.100125+0000 osd.5 (osd.5) 330 : cluster [DBG] 22.f scrub ok 2024-02-23T00:44:39.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.117894+0000 osd.7 (osd.7) 390 : cluster [DBG] 3.15 scrub starts 2024-02-23T00:44:39.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.120772+0000 osd.7 (osd.7) 391 : cluster [DBG] 3.15 scrub ok 2024-02-23T00:44:39.606 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:39.656 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.761286+0000 osd.0 (osd.0) 77 : cluster [DBG] 3.12 scrub starts 2024-02-23T00:44:39.656 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.762129+0000 osd.0 (osd.0) 78 : cluster [DBG] 3.12 scrub ok 2024-02-23T00:44:39.656 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.953271+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.b scrub starts 2024-02-23T00:44:39.656 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:37.956385+0000 osd.3 (osd.3) 278 : cluster [DBG] 6.b scrub ok 2024-02-23T00:44:39.656 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:38.034388+0000 osd.6 (osd.6) 360 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:44:39.656 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:38.035195+0000 osd.6 (osd.6) 361 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:44:39.657 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:38.443375+0000 mon.a (mon.0) 3973 : audit [INF] from='client.? 172.21.15.87:0/2547325356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-23T00:44:39.657 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:38.443451+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-23T00:44:39.657 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.097101+0000 osd.5 (osd.5) 329 : cluster [DBG] 22.f scrub starts 2024-02-23T00:44:39.657 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.100125+0000 osd.5 (osd.5) 330 : cluster [DBG] 22.f scrub ok 2024-02-23T00:44:39.657 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.117894+0000 osd.7 (osd.7) 390 : cluster [DBG] 3.15 scrub starts 2024-02-23T00:44:39.657 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.120772+0000 osd.7 (osd.7) 391 : cluster [DBG] 3.15 scrub ok 2024-02-23T00:44:39.674 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-23T00:44:39.938 INFO:teuthology.orchestra.run.smithi171.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.b/config 2024-02-23T00:44:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:38.943269+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:44:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:38.946186+0000 osd.3 (osd.3) 280 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:44:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.311147+0000 osd.2 (osd.2) 309 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:44:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.312177+0000 osd.2 (osd.2) 310 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:44:40.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.639014+0000 osd.1 (osd.1) 388 : cluster [DBG] 22.b scrub starts 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.642270+0000 osd.1 (osd.1) 389 : cluster [DBG] 22.b scrub ok 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.819414+0000 mgr.y (mgr.24437) 701 : cluster [DBG] pgmap v1267: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:40.107469+0000 osd.7 (osd.7) 392 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:40.110937+0000 osd.7 (osd.7) 393 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:38.943269+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:38.946186+0000 osd.3 (osd.3) 280 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.311147+0000 osd.2 (osd.2) 309 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.312177+0000 osd.2 (osd.2) 310 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.639014+0000 osd.1 (osd.1) 388 : cluster [DBG] 22.b scrub starts 2024-02-23T00:44:40.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.642270+0000 osd.1 (osd.1) 389 : cluster [DBG] 22.b scrub ok 2024-02-23T00:44:40.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.819414+0000 mgr.y (mgr.24437) 701 : cluster [DBG] pgmap v1267: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:40.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:40.107469+0000 osd.7 (osd.7) 392 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:44:40.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:40.110937+0000 osd.7 (osd.7) 393 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:44:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:38.943269+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:44:40.992 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:38.946186+0000 osd.3 (osd.3) 280 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:44:40.992 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.311147+0000 osd.2 (osd.2) 309 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:44:40.992 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.312177+0000 osd.2 (osd.2) 310 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:44:40.992 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.639014+0000 osd.1 (osd.1) 388 : cluster [DBG] 22.b scrub starts 2024-02-23T00:44:40.992 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.642270+0000 osd.1 (osd.1) 389 : cluster [DBG] 22.b scrub ok 2024-02-23T00:44:40.993 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.819414+0000 mgr.y (mgr.24437) 701 : cluster [DBG] pgmap v1267: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:40.993 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:40.107469+0000 osd.7 (osd.7) 392 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:44:40.993 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:40.110937+0000 osd.7 (osd.7) 393 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:44:41.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:44:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:44:41.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.902446+0000 osd.3 (osd.3) 281 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:44:41.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:39.903262+0000 osd.3 (osd.3) 282 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:44:41.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:41.086915+0000 osd.5 (osd.5) 331 : cluster [DBG] 6.a scrub starts 2024-02-23T00:44:41.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:41.087945+0000 osd.5 (osd.5) 332 : cluster [DBG] 6.a scrub ok 2024-02-23T00:44:41.504 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:41.368005+0000 mgr.y (mgr.24437) 702 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.662+0000 7f673c8c2700 1 -- 172.21.15.87:0/2496063311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6738102210 msgr2=0x7f6738104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.662+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2496063311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6738102210 0x7f6738104600 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 rx=0x7f6730009be0 tx=0x7f6730004d10).stop 2024-02-23T00:44:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.662+0000 7f673c8c2700 1 -- 172.21.15.87:0/2496063311 shutdown_connections 2024-02-23T00:44:41.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.662+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2496063311 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f6738102210 0x7f6738104600 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.662+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2496063311 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67380f8cb0 0x7f67380f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.662+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2496063311 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f67380f9610 0x7f6738101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.662+0000 7f673c8c2700 1 -- 172.21.15.87:0/2496063311 >> 172.21.15.87:0/2496063311 conn(0x7f67380f2820 msgr2=0x7f67380f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.662+0000 7f673c8c2700 1 -- 172.21.15.87:0/2496063311 shutdown_connections 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.662+0000 7f673c8c2700 1 -- 172.21.15.87:0/2496063311 wait complete. 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 librados: starting msgr at 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 librados: starting objecter 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 Processor -- start 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 -- start start 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 librados: setting wanted keys 2024-02-23T00:44:41.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 librados: calling monclient init 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67380f8cb0 0x7f673810dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67380f9610 0x7f673810e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6738102210 0x7f6738108170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6738108660 con 0x7f67380f8cb0 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67381087d0 con 0x7f67380f9610 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673c8c2700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6738108990 con 0x7f6738102210 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f6736d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6738102210 0x7f6738108170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67380f8cb0 0x7f673810dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f6736d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6738102210 0x7f6738108170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:48550/0 (socket says 172.21.15.87:48550) 2024-02-23T00:44:41.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f6736d9d700 1 -- 172.21.15.87:0/2133045252 learned_addr learned my addr 172.21.15.87:0/2133045252 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f673659c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67380f8cb0 0x7f673810dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:55168/0 (socket says 172.21.15.87:55168) 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.663+0000 7f6735d9b700 1 --2- 172.21.15.87:0/2133045252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67380f9610 0x7f673810e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f6736d9d700 1 -- 172.21.15.87:0/2133045252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67380f9610 msgr2=0x7f673810e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f6736d9d700 1 --2- 172.21.15.87:0/2133045252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67380f9610 0x7f673810e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f6736d9d700 1 -- 172.21.15.87:0/2133045252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67380f8cb0 msgr2=0x7f673810dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f6736d9d700 1 --2- 172.21.15.87:0/2133045252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67380f8cb0 0x7f673810dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f6736d9d700 1 -- 172.21.15.87:0/2133045252 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6728005040 con 0x7f6738102210 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f673659c700 1 --2- 172.21.15.87:0/2133045252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67380f8cb0 0x7f673810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f6736d9d700 1 --2- 172.21.15.87:0/2133045252 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6738102210 0x7f6738108170 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7f6730004ed0 tx=0x7f6730004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f67277fe700 10 client.?.objecter ms_handle_connect 0x7f6738102210 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f67277fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f67277fe700 1 -- 172.21.15.87:0/2133045252 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673000e040 con 0x7f6738102210 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67300097c0 con 0x7f6738102210 2024-02-23T00:44:41.669 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f67277fe700 1 -- 172.21.15.87:0/2133045252 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6730018040 con 0x7f6738102210 2024-02-23T00:44:41.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f67277fe700 1 -- 172.21.15.87:0/2133045252 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673001c3f0 con 0x7f6738102210 2024-02-23T00:44:41.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f673c8c2700 10 client.38471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:41.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6738108e30 con 0x7f6738102210 2024-02-23T00:44:41.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.664+0000 7f673c8c2700 1 librados: init done 2024-02-23T00:44:41.670 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.665+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6738109020 con 0x7f6738102210 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.666+0000 7f67277fe700 1 -- 172.21.15.87:0/2133045252 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f6730013590 con 0x7f6738102210 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.666+0000 7f67277fe700 1 --2- 172.21.15.87:0/2133045252 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6720062c90 0x7f6720065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.666+0000 7f67277fe700 1 -- 172.21.15.87:0/2133045252 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(820..820 src has 400..820) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f673007e6a0 con 0x7f6738102210 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.666+0000 7f67277fe700 10 client.38471.objecter ms_dispatch 0x7f6738001550 osd_map(820..820 src has 400..820) v4 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.666+0000 7f67277fe700 3 client.38471.objecter handle_osd_map got epochs [820,820] > 0 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.666+0000 7f67277fe700 3 client.38471.objecter handle_osd_map decoding full epoch 820 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.666+0000 7f673659c700 1 --2- 172.21.15.87:0/2133045252 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6720062c90 0x7f6720065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.666+0000 7f67277fe700 20 client.38471.objecter dump_active .. 0 homeless 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.667+0000 7f67277fe700 10 client.38471.objecter ms_handle_connect 0x7f6720062c90 2024-02-23T00:44:41.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.667+0000 7f673659c700 1 --2- 172.21.15.87:0/2133045252 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6720062c90 0x7f6720065150 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7f672c0103c0 tx=0x7f672c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:41.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.670+0000 7f67277fe700 1 -- 172.21.15.87:0/2133045252 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f6730051f40 con 0x7f6738102210 2024-02-23T00:44:41.742 INFO:teuthology.orchestra.run.smithi171.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:41.743 INFO:teuthology.orchestra.run.smithi171.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.902446+0000 osd.3 (osd.3) 281 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:44:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:39.903262+0000 osd.3 (osd.3) 282 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:44:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:41.086915+0000 osd.5 (osd.5) 331 : cluster [DBG] 6.a scrub starts 2024-02-23T00:44:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:41.087945+0000 osd.5 (osd.5) 332 : cluster [DBG] 6.a scrub ok 2024-02-23T00:44:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:41.368005+0000 mgr.y (mgr.24437) 702 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:41.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.902446+0000 osd.3 (osd.3) 281 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:44:41.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:39.903262+0000 osd.3 (osd.3) 282 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:44:41.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:41.086915+0000 osd.5 (osd.5) 331 : cluster [DBG] 6.a scrub starts 2024-02-23T00:44:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:41.087945+0000 osd.5 (osd.5) 332 : cluster [DBG] 6.a scrub ok 2024-02-23T00:44:41.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:41.368005+0000 mgr.y (mgr.24437) 702 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:41.851 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:41.849+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f6738107290 con 0x7f6738102210 2024-02-23T00:44:42.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.271+0000 7f67277fe700 1 -- 172.21.15.87:0/2133045252 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v821) v1 ==== 133+0+0 (secure 0 0 0) 0x7f67300210a0 con 0x7f6738102210 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 10 librados: watch_flush enter 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 10 librados: watch_flush exit 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 20 client.38471.objecter shutdown clearing up homeless session... 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 10 client.38471.objecter successfully canceled tick 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6720062c90 msgr2=0x7f6720065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2133045252 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6720062c90 0x7f6720065150 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 rx=0x7f672c0103c0 tx=0x7f672c00b040).stop 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6738102210 msgr2=0x7f6738108170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2133045252 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6738102210 0x7f6738108170 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 rx=0x7f6730004ed0 tx=0x7f6730004d10).stop 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 shutdown_connections 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2133045252 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6720062c90 0x7f6720065150 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2133045252 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f6738102210 0x7f6738108170 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:42.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2133045252 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f67380f8cb0 0x7f673810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:42.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 --2- 172.21.15.87:0/2133045252 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f67380f9610 0x7f673810e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:42.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 >> 172.21.15.87:0/2133045252 conn(0x7f67380f2820 msgr2=0x7f67380f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:42.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 shutdown_connections 2024-02-23T00:44:42.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 -- 172.21.15.87:0/2133045252 wait complete. 2024-02-23T00:44:42.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:42.273+0000 7f673c8c2700 1 librados: shutdown 2024-02-23T00:44:42.277 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-23T00:44:42.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-23T00:44:42.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:42] "GET /metrics HTTP/1.1" 200 186063 "" "Prometheus/2.33.4" 2024-02-23T00:44:42.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:41.682684+0000 osd.1 (osd.1) 390 : cluster [DBG] 2.d scrub starts 2024-02-23T00:44:42.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:41.685576+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.d scrub ok 2024-02-23T00:44:42.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:41.820148+0000 mgr.y (mgr.24437) 703 : cluster [DBG] pgmap v1268: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:41.851235+0000 mon.c (mon.2) 509 : audit [INF] from='client.? 172.21.15.87:0/2133045252' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:41.851512+0000 mon.a (mon.0) 3975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:42.122948+0000 osd.5 (osd.5) 333 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:42.124455+0000 osd.5 (osd.5) 334 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:42.197536+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:42.200433+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:42.272121+0000 mon.a (mon.0) 3976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:42.272210+0000 mon.a (mon.0) 3977 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:41.682684+0000 osd.1 (osd.1) 390 : cluster [DBG] 2.d scrub starts 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:41.685576+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.d scrub ok 2024-02-23T00:44:42.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:41.820148+0000 mgr.y (mgr.24437) 703 : cluster [DBG] pgmap v1268: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:42.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:41.851235+0000 mon.c (mon.2) 509 : audit [INF] from='client.? 172.21.15.87:0/2133045252' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:44:42.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:41.851512+0000 mon.a (mon.0) 3975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:44:42.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:42.122948+0000 osd.5 (osd.5) 333 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:44:42.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:42.124455+0000 osd.5 (osd.5) 334 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:44:42.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:42.197536+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:44:42.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:42.200433+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:44:42.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:42.272121+0000 mon.a (mon.0) 3976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-23T00:44:42.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:42.272210+0000 mon.a (mon.0) 3977 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:41.682684+0000 osd.1 (osd.1) 390 : cluster [DBG] 2.d scrub starts 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:41.685576+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.d scrub ok 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:41.820148+0000 mgr.y (mgr.24437) 703 : cluster [DBG] pgmap v1268: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:41.851235+0000 mon.c (mon.2) 509 : audit [INF] from='client.? 172.21.15.87:0/2133045252' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:41.851512+0000 mon.a (mon.0) 3975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:42.122948+0000 osd.5 (osd.5) 333 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:42.124455+0000 osd.5 (osd.5) 334 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:42.197536+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:42.200433+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:42.272121+0000 mon.a (mon.0) 3976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-23T00:44:42.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:42.272210+0000 mon.a (mon.0) 3977 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-23T00:44:43.117 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight 2 1 2024-02-23T00:44:43.184 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:42.676399+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.7 deep-scrub starts 2024-02-23T00:44:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:42.679539+0000 osd.1 (osd.1) 393 : cluster [DBG] 2.7 deep-scrub ok 2024-02-23T00:44:43.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:42.676399+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.7 deep-scrub starts 2024-02-23T00:44:43.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:42.679539+0000 osd.1 (osd.1) 393 : cluster [DBG] 2.7 deep-scrub ok 2024-02-23T00:44:43.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:44:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:44:43.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:42.676399+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.7 deep-scrub starts 2024-02-23T00:44:43.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:42.679539+0000 osd.1 (osd.1) 393 : cluster [DBG] 2.7 deep-scrub ok 2024-02-23T00:44:44.251 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:43.820589+0000 mgr.y (mgr.24437) 704 : cluster [DBG] pgmap v1270: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-02-23T00:44:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:44.110554+0000 osd.5 (osd.5) 335 : cluster [DBG] 5.f scrub starts 2024-02-23T00:44:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:44.111672+0000 osd.5 (osd.5) 336 : cluster [DBG] 5.f scrub ok 2024-02-23T00:44:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:44.242823+0000 osd.7 (osd.7) 396 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:44:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:44.245757+0000 osd.7 (osd.7) 397 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:44:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:43.820589+0000 mgr.y (mgr.24437) 704 : cluster [DBG] pgmap v1270: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-02-23T00:44:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:44.110554+0000 osd.5 (osd.5) 335 : cluster [DBG] 5.f scrub starts 2024-02-23T00:44:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:44.111672+0000 osd.5 (osd.5) 336 : cluster [DBG] 5.f scrub ok 2024-02-23T00:44:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:44.242823+0000 osd.7 (osd.7) 396 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:44:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:44.245757+0000 osd.7 (osd.7) 397 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:44:44.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:43.820589+0000 mgr.y (mgr.24437) 704 : cluster [DBG] pgmap v1270: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-02-23T00:44:44.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:44.110554+0000 osd.5 (osd.5) 335 : cluster [DBG] 5.f scrub starts 2024-02-23T00:44:44.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:44.111672+0000 osd.5 (osd.5) 336 : cluster [DBG] 5.f scrub ok 2024-02-23T00:44:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:44.242823+0000 osd.7 (osd.7) 396 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:44:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:44.245757+0000 osd.7 (osd.7) 397 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:44:45.706 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:45.706 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:45.707 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:43.759495+0000 osd.0 (osd.0) 79 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:44:45.707 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:43.762449+0000 osd.0 (osd.0) 80 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:44:45.707 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:44.680302+0000 osd.1 (osd.1) 394 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:44:45.707 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:44.683760+0000 osd.1 (osd.1) 395 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:44:45.707 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:43.759495+0000 osd.0 (osd.0) 79 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:44:45.707 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:43.762449+0000 osd.0 (osd.0) 80 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:44:45.707 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:44.680302+0000 osd.1 (osd.1) 394 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:44:45.707 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:44.683760+0000 osd.1 (osd.1) 395 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:44:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:43.759495+0000 osd.0 (osd.0) 79 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:44:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:43.762449+0000 osd.0 (osd.0) 80 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:44:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:44.680302+0000 osd.1 (osd.1) 394 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:44:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:44.683760+0000 osd.1 (osd.1) 395 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:44:46.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:45.641732+0000 osd.1 (osd.1) 396 : cluster [DBG] 22.1c scrub starts 2024-02-23T00:44:46.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:45.644643+0000 osd.1 (osd.1) 397 : cluster [DBG] 22.1c scrub ok 2024-02-23T00:44:46.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:45.821272+0000 mgr.y (mgr.24437) 705 : cluster [DBG] pgmap v1271: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:44:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:45.641732+0000 osd.1 (osd.1) 396 : cluster [DBG] 22.1c scrub starts 2024-02-23T00:44:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:45.644643+0000 osd.1 (osd.1) 397 : cluster [DBG] 22.1c scrub ok 2024-02-23T00:44:46.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:45.821272+0000 mgr.y (mgr.24437) 705 : cluster [DBG] pgmap v1271: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:44:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:45.641732+0000 osd.1 (osd.1) 396 : cluster [DBG] 22.1c scrub starts 2024-02-23T00:44:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:45.644643+0000 osd.1 (osd.1) 397 : cluster [DBG] 22.1c scrub ok 2024-02-23T00:44:46.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:45.821272+0000 mgr.y (mgr.24437) 705 : cluster [DBG] pgmap v1271: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:44:47.629 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-23T00:44:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:45.813285+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.3 deep-scrub starts 2024-02-23T00:44:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:45.816635+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.3 deep-scrub ok 2024-02-23T00:44:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:46.141859+0000 osd.6 (osd.6) 362 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:44:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:46.143979+0000 osd.6 (osd.6) 363 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:44:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:46.261976+0000 osd.7 (osd.7) 398 : cluster [DBG] 3.d scrub starts 2024-02-23T00:44:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:46.264980+0000 osd.7 (osd.7) 399 : cluster [DBG] 3.d scrub ok 2024-02-23T00:44:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:46.615420+0000 osd.1 (osd.1) 398 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:44:47.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:46.619051+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:47.136951+0000 osd.5 (osd.5) 337 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:47.138055+0000 osd.5 (osd.5) 338 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:45.813285+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.3 deep-scrub starts 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:45.816635+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.3 deep-scrub ok 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:46.141859+0000 osd.6 (osd.6) 362 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:46.143979+0000 osd.6 (osd.6) 363 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:46.261976+0000 osd.7 (osd.7) 398 : cluster [DBG] 3.d scrub starts 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:46.264980+0000 osd.7 (osd.7) 399 : cluster [DBG] 3.d scrub ok 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:46.615420+0000 osd.1 (osd.1) 398 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:46.619051+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:47.136951+0000 osd.5 (osd.5) 337 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:44:47.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:47.138055+0000 osd.5 (osd.5) 338 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:44:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:45.813285+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.3 deep-scrub starts 2024-02-23T00:44:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:45.816635+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.3 deep-scrub ok 2024-02-23T00:44:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:46.141859+0000 osd.6 (osd.6) 362 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:44:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:46.143979+0000 osd.6 (osd.6) 363 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:44:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:46.261976+0000 osd.7 (osd.7) 398 : cluster [DBG] 3.d scrub starts 2024-02-23T00:44:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:46.264980+0000 osd.7 (osd.7) 399 : cluster [DBG] 3.d scrub ok 2024-02-23T00:44:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:46.615420+0000 osd.1 (osd.1) 398 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:44:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:46.619051+0000 osd.1 (osd.1) 399 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:44:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:47.136951+0000 osd.5 (osd.5) 337 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:44:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:47.138055+0000 osd.5 (osd.5) 338 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:44:47.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.921+0000 7f4e691a6700 1 -- 172.21.15.87:0/2790592104 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e64102210 msgr2=0x7f4e64104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:47.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.921+0000 7f4e691a6700 1 --2- 172.21.15.87:0/2790592104 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e64102210 0x7f4e64104600 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 rx=0x7f4e5c009be0 tx=0x7f4e5c004d10).stop 2024-02-23T00:44:47.952 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.921+0000 7f4e691a6700 1 -- 172.21.15.87:0/2790592104 shutdown_connections 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.921+0000 7f4e691a6700 1 --2- 172.21.15.87:0/2790592104 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e64102210 0x7f4e64104600 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.921+0000 7f4e691a6700 1 --2- 172.21.15.87:0/2790592104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e640f8cb0 0x7f4e640f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.921+0000 7f4e691a6700 1 --2- 172.21.15.87:0/2790592104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e640f9610 0x7f4e64101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.921+0000 7f4e691a6700 1 -- 172.21.15.87:0/2790592104 >> 172.21.15.87:0/2790592104 conn(0x7f4e640f2820 msgr2=0x7f4e640f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.921+0000 7f4e691a6700 1 -- 172.21.15.87:0/2790592104 shutdown_connections 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.921+0000 7f4e691a6700 1 -- 172.21.15.87:0/2790592104 wait complete. 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 librados: starting msgr at 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 librados: starting objecter 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 Processor -- start 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 -- start start 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 librados: setting wanted keys 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 librados: calling monclient init 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e640f8cb0 0x7f4e6410df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:47.953 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e640f9610 0x7f4e6410e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e64102210 0x7f4e641080f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e641085e0 con 0x7f4e64102210 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e64108750 con 0x7f4e640f9610 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e691a6700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e64108910 con 0x7f4e640f8cb0 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e62d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e640f8cb0 0x7f4e6410df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e6359e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e64102210 0x7f4e641080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e62d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e640f8cb0 0x7f4e6410df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:48574/0 (socket says 172.21.15.87:48574) 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.922+0000 7f4e62d9d700 1 -- 172.21.15.87:0/696222229 learned_addr learned my addr 172.21.15.87:0/696222229 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e6259c700 1 --2- 172.21.15.87:0/696222229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e640f9610 0x7f4e6410e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e62d9d700 1 -- 172.21.15.87:0/696222229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e640f9610 msgr2=0x7f4e6410e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e62d9d700 1 --2- 172.21.15.87:0/696222229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e640f9610 0x7f4e6410e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e62d9d700 1 -- 172.21.15.87:0/696222229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e64102210 msgr2=0x7f4e641080f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e62d9d700 1 --2- 172.21.15.87:0/696222229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e64102210 0x7f4e641080f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e62d9d700 1 -- 172.21.15.87:0/696222229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e58005040 con 0x7f4e640f8cb0 2024-02-23T00:44:47.954 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e6359e700 1 --2- 172.21.15.87:0/696222229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e64102210 0x7f4e641080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e62d9d700 1 --2- 172.21.15.87:0/696222229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e640f8cb0 0x7f4e6410df50 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7f4e5400c7c0 tx=0x7f4e54007ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e4bfff700 10 client.?.objecter ms_handle_connect 0x7f4e640f8cb0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e4bfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e4bfff700 1 -- 172.21.15.87:0/696222229 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e54004430 con 0x7f4e640f8cb0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c0097c0 con 0x7f4e640f8cb0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e4bfff700 1 -- 172.21.15.87:0/696222229 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e5400e970 con 0x7f4e640f8cb0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e4bfff700 1 -- 172.21.15.87:0/696222229 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e54013a50 con 0x7f4e640f8cb0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e691a6700 10 client.38477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e64108f50 con 0x7f4e640f8cb0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.923+0000 7f4e691a6700 1 librados: init done 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.924+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6405fa60 con 0x7f4e640f8cb0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.924+0000 7f4e4bfff700 1 -- 172.21.15.87:0/696222229 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f4e5400e4a0 con 0x7f4e640f8cb0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.925+0000 7f4e4bfff700 1 --2- 172.21.15.87:0/696222229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e4c062c90 0x7f4e4c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.925+0000 7f4e4bfff700 1 -- 172.21.15.87:0/696222229 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(821..821 src has 400..821) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f4e5407d2a0 con 0x7f4e640f8cb0 2024-02-23T00:44:47.955 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.925+0000 7f4e4bfff700 10 client.38477.objecter ms_dispatch 0x7f4e64001550 osd_map(821..821 src has 400..821) v4 2024-02-23T00:44:47.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.925+0000 7f4e4bfff700 3 client.38477.objecter handle_osd_map got epochs [821,821] > 0 2024-02-23T00:44:47.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.925+0000 7f4e4bfff700 3 client.38477.objecter handle_osd_map decoding full epoch 821 2024-02-23T00:44:47.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.925+0000 7f4e6259c700 1 --2- 172.21.15.87:0/696222229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e4c062c90 0x7f4e4c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:47.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.925+0000 7f4e4bfff700 20 client.38477.objecter dump_active .. 0 homeless 2024-02-23T00:44:47.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.925+0000 7f4e4bfff700 10 client.38477.objecter ms_handle_connect 0x7f4e4c062c90 2024-02-23T00:44:47.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.925+0000 7f4e6259c700 1 --2- 172.21.15.87:0/696222229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e4c062c90 0x7f4e4c065150 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f4e64109890 tx=0x7f4e58009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:47.956 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:47.929+0000 7f4e4bfff700 1 -- 172.21.15.87:0/696222229 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f4e54050930 con 0x7f4e640f8cb0 2024-02-23T00:44:48.027 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:48.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.122+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f4e641095d0 con 0x7f4e640f8cb0 2024-02-23T00:44:48.485 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.482+0000 7f4e4bfff700 1 -- 172.21.15.87:0/696222229 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v822) v1 ==== 113+0+0 (secure 0 0 0) 0x7f4e54050520 con 0x7f4e640f8cb0 2024-02-23T00:44:48.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.485+0000 7f4e691a6700 10 librados: watch_flush enter 2024-02-23T00:44:48.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.485+0000 7f4e691a6700 10 librados: watch_flush exit 2024-02-23T00:44:48.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.485+0000 7f4e691a6700 20 client.38477.objecter shutdown clearing up homeless session... 2024-02-23T00:44:48.487 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.485+0000 7f4e691a6700 10 client.38477.objecter successfully canceled tick 2024-02-23T00:44:48.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e4c062c90 msgr2=0x7f4e4c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:48.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 --2- 172.21.15.87:0/696222229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e4c062c90 0x7f4e4c065150 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 rx=0x7f4e64109890 tx=0x7f4e58009cb0).stop 2024-02-23T00:44:48.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e640f8cb0 msgr2=0x7f4e6410df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:48.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 --2- 172.21.15.87:0/696222229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e640f8cb0 0x7f4e6410df50 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 rx=0x7f4e5400c7c0 tx=0x7f4e54007ef0).stop 2024-02-23T00:44:48.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 shutdown_connections 2024-02-23T00:44:48.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 --2- 172.21.15.87:0/696222229 >> v2:172.21.15.87:6800/1231597463 conn(0x7f4e4c062c90 0x7f4e4c065150 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:48.490 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 --2- 172.21.15.87:0/696222229 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f4e64102210 0x7f4e641080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:48.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 --2- 172.21.15.87:0/696222229 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f4e640f8cb0 0x7f4e6410df50 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:48.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 --2- 172.21.15.87:0/696222229 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f4e640f9610 0x7f4e6410e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:48.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 >> 172.21.15.87:0/696222229 conn(0x7f4e640f2820 msgr2=0x7f4e640f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:48.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 shutdown_connections 2024-02-23T00:44:48.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 -- 172.21.15.87:0/696222229 wait complete. 2024-02-23T00:44:48.491 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:48.487+0000 7f4e691a6700 1 librados: shutdown 2024-02-23T00:44:48.491 INFO:teuthology.orchestra.run.smithi087.stderr:reweighted osd.2 to 1 (10000) 2024-02-23T00:44:48.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:47.821604+0000 mgr.y (mgr.24437) 706 : cluster [DBG] pgmap v1272: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:44:48.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:48.123669+0000 mon.c (mon.2) 510 : audit [INF] from='client.? 172.21.15.87:0/696222229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:48.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:48.123970+0000 mon.a (mon.0) 3978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:47.821604+0000 mgr.y (mgr.24437) 706 : cluster [DBG] pgmap v1272: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:44:48.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:48.123669+0000 mon.c (mon.2) 510 : audit [INF] from='client.? 172.21.15.87:0/696222229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:48.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:48.123970+0000 mon.a (mon.0) 3978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:47.821604+0000 mgr.y (mgr.24437) 706 : cluster [DBG] pgmap v1272: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:44:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:48.123669+0000 mon.c (mon.2) 510 : audit [INF] from='client.? 172.21.15.87:0/696222229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:48.123970+0000 mon.a (mon.0) 3978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:48.483747+0000 mon.a (mon.0) 3979 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-23T00:44:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:48.483826+0000 mon.a (mon.0) 3980 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-23T00:44:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:49.138854+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.16 scrub starts 2024-02-23T00:44:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:49.139933+0000 osd.5 (osd.5) 340 : cluster [DBG] 3.16 scrub ok 2024-02-23T00:44:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:48.483747+0000 mon.a (mon.0) 3979 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-23T00:44:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:48.483826+0000 mon.a (mon.0) 3980 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-23T00:44:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:49.138854+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.16 scrub starts 2024-02-23T00:44:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:49.139933+0000 osd.5 (osd.5) 340 : cluster [DBG] 3.16 scrub ok 2024-02-23T00:44:49.719 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-23T00:44:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:48.483747+0000 mon.a (mon.0) 3979 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-23T00:44:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:48.483826+0000 mon.a (mon.0) 3980 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-23T00:44:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:49.138854+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.16 scrub starts 2024-02-23T00:44:49.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:49.139933+0000 osd.5 (osd.5) 340 : cluster [DBG] 3.16 scrub ok 2024-02-23T00:44:50.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:49.654784+0000 osd.1 (osd.1) 400 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:44:50.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:49.657771+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:44:50.500 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:49.821989+0000 mgr.y (mgr.24437) 707 : cluster [DBG] pgmap v1274: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:44:50.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:49.654784+0000 osd.1 (osd.1) 400 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:44:50.500 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:49.657771+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:44:50.501 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:49.821989+0000 mgr.y (mgr.24437) 707 : cluster [DBG] pgmap v1274: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:44:50.759 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:50.787 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:50.787 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:50.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:49.654784+0000 osd.1 (osd.1) 400 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:44:50.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:49.657771+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:44:50.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:49.821989+0000 mgr.y (mgr.24437) 707 : cluster [DBG] pgmap v1274: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:44:51.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:50.653078+0000 osd.1 (osd.1) 402 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:44:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:50.656546+0000 osd.1 (osd.1) 403 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:44:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:51.379002+0000 mgr.y (mgr.24437) 708 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:51.693 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:44:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:44:51.785 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:50.653078+0000 osd.1 (osd.1) 402 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:44:51.785 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:50.656546+0000 osd.1 (osd.1) 403 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:44:51.785 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:51.379002+0000 mgr.y (mgr.24437) 708 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:51.785 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:50.653078+0000 osd.1 (osd.1) 402 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:44:51.785 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:50.656546+0000 osd.1 (osd.1) 403 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:44:51.785 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:51.379002+0000 mgr.y (mgr.24437) 708 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:44:52.515 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:44:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:52] "GET /metrics HTTP/1.1" 200 186063 "" "Prometheus/2.33.4" 2024-02-23T00:44:52.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 -- 172.21.15.87:0/2673929373 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140fcbb0 msgr2=0x7f35140fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:52.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 --2- 172.21.15.87:0/2673929373 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140fcbb0 0x7f35140fcf90 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 rx=0x7f3508004870 tx=0x7f3508009cb0).stop 2024-02-23T00:44:52.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 -- 172.21.15.87:0/2673929373 shutdown_connections 2024-02-23T00:44:52.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 --2- 172.21.15.87:0/2673929373 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35140f7400 0x7f3514103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 --2- 172.21.15.87:0/2673929373 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140fcbb0 0x7f35140fcf90 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 --2- 172.21.15.87:0/2673929373 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35140fd510 0x7f35140f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 -- 172.21.15.87:0/2673929373 >> 172.21.15.87:0/2673929373 conn(0x7f35140f2820 msgr2=0x7f35140f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 -- 172.21.15.87:0/2673929373 shutdown_connections 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 -- 172.21.15.87:0/2673929373 wait complete. 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 librados: starting msgr at 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 librados: starting objecter 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.906+0000 7f3519f8a700 1 Processor -- start 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 -- start start 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 librados: setting wanted keys 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 librados: calling monclient init 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140f7400 0x7f351410e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35140fcbb0 0x7f351410e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35140fd510 0x7f35141084a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:52.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3514108990 con 0x7f35140fcbb0 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3514108b00 con 0x7f35140fd510 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3514108cc0 con 0x7f35140f7400 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f35137fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140f7400 0x7f351410e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3512ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35140fcbb0 0x7f351410e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3512ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35140fcbb0 0x7f351410e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39810/0 (socket says 172.21.15.87:39810) 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f35137fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140f7400 0x7f351410e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:42418/0 (socket says 172.21.15.87:42418) 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3512ffd700 1 -- 172.21.15.87:0/1839051234 learned_addr learned my addr 172.21.15.87:0/1839051234 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3513fff700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35140fd510 0x7f35141084a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f35137fe700 1 -- 172.21.15.87:0/1839051234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35140fd510 msgr2=0x7f35141084a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f35137fe700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35140fd510 0x7f35141084a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f35137fe700 1 -- 172.21.15.87:0/1839051234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35140fcbb0 msgr2=0x7f351410e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f35137fe700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35140fcbb0 0x7f351410e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f35137fe700 1 -- 172.21.15.87:0/1839051234 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504005040 con 0x7f35140f7400 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3512ffd700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35140fcbb0 0x7f351410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3513fff700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35140fd510 0x7f35141084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f35137fe700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140f7400 0x7f351410e300 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 rx=0x7f3508004870 tx=0x7f350800b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3510ff9700 10 client.?.objecter ms_handle_connect 0x7f35140f7400 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3510ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3510ff9700 1 -- 172.21.15.87:0/1839051234 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350800e070 con 0x7f35140f7400 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3510ff9700 1 -- 172.21.15.87:0/1839051234 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3508013da0 con 0x7f35140f7400 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3510ff9700 1 -- 172.21.15.87:0/1839051234 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508019070 con 0x7f35140f7400 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.907+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3508005040 con 0x7f35140f7400 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.908+0000 7f3519f8a700 10 client.38483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.908+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3514109160 con 0x7f35140f7400 2024-02-23T00:44:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.908+0000 7f3519f8a700 1 librados: init done 2024-02-23T00:44:52.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.909+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3514109350 con 0x7f35140f7400 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.909+0000 7f3510ff9700 1 -- 172.21.15.87:0/1839051234 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f350801e420 con 0x7f35140f7400 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.909+0000 7f3510ff9700 1 --2- 172.21.15.87:0/1839051234 >> v2:172.21.15.87:6800/1231597463 conn(0x7f34fc062c90 0x7f34fc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.909+0000 7f3510ff9700 1 -- 172.21.15.87:0/1839051234 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(823..823 src has 400..823) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f350807d820 con 0x7f35140f7400 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.909+0000 7f3510ff9700 10 client.38483.objecter ms_dispatch 0x7f3514001550 osd_map(823..823 src has 400..823) v4 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.909+0000 7f3510ff9700 3 client.38483.objecter handle_osd_map got epochs [823,823] > 0 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.909+0000 7f3512ffd700 1 --2- 172.21.15.87:0/1839051234 >> v2:172.21.15.87:6800/1231597463 conn(0x7f34fc062c90 0x7f34fc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.909+0000 7f3510ff9700 3 client.38483.objecter handle_osd_map decoding full epoch 823 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.909+0000 7f3510ff9700 20 client.38483.objecter dump_active .. 0 homeless 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.910+0000 7f3512ffd700 1 --2- 172.21.15.87:0/1839051234 >> v2:172.21.15.87:6800/1231597463 conn(0x7f34fc062c90 0x7f34fc065150 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7f3514109c40 tx=0x7f350400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.910+0000 7f3510ff9700 10 client.38483.objecter ms_handle_connect 0x7f34fc062c90 2024-02-23T00:44:52.915 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:52.914+0000 7f3510ff9700 1 -- 172.21.15.87:0/1839051234 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3508050fe0 con 0x7f35140f7400 2024-02-23T00:44:53.093 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.091+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f351405fa60 con 0x7f35140f7400 2024-02-23T00:44:53.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-23T00:44:53.509 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.506+0000 7f3510ff9700 1 -- 172.21.15.87:0/1839051234 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v824) v1 ==== 133+0+0 (secure 0 0 0) 0x7f351405fa60 con 0x7f35140f7400 2024-02-23T00:44:53.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 10 librados: watch_flush enter 2024-02-23T00:44:53.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 10 librados: watch_flush exit 2024-02-23T00:44:53.510 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 20 client.38483.objecter shutdown clearing up homeless session... 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 10 client.38483.objecter successfully canceled tick 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 >> v2:172.21.15.87:6800/1231597463 conn(0x7f34fc062c90 msgr2=0x7f34fc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 --2- 172.21.15.87:0/1839051234 >> v2:172.21.15.87:6800/1231597463 conn(0x7f34fc062c90 0x7f34fc065150 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7f3514109c40 tx=0x7f350400b040).stop 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140f7400 msgr2=0x7f351410e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140f7400 0x7f351410e300 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 rx=0x7f3508004870 tx=0x7f350800b040).stop 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 shutdown_connections 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 --2- 172.21.15.87:0/1839051234 >> v2:172.21.15.87:6800/1231597463 conn(0x7f34fc062c90 0x7f34fc065150 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f35140fd510 0x7f35141084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f35140f7400 0x7f351410e300 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 --2- 172.21.15.87:0/1839051234 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f35140fcbb0 0x7f351410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:53.511 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.508+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 >> 172.21.15.87:0/1839051234 conn(0x7f35140f2820 msgr2=0x7f3514060e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.509+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 shutdown_connections 2024-02-23T00:44:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.509+0000 7f3519f8a700 1 -- 172.21.15.87:0/1839051234 wait complete. 2024-02-23T00:44:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:53.509+0000 7f3519f8a700 1 librados: shutdown 2024-02-23T00:44:53.512 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-23T00:44:53.536 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:51.822739+0000 mgr.y (mgr.24437) 709 : cluster [DBG] pgmap v1275: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:51.847734+0000 osd.0 (osd.0) 83 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:51.850785+0000 osd.0 (osd.0) 84 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:52.036666+0000 osd.6 (osd.6) 364 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:52.039573+0000 osd.6 (osd.6) 365 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:52.274694+0000 mon.a (mon.0) 3981 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:52.282813+0000 osd.4 (osd.4) 365 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:52.283928+0000 osd.4 (osd.4) 366 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:53.092596+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.87:0/1839051234' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:53.092955+0000 mon.a (mon.0) 3982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:53.537 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:51.822739+0000 mgr.y (mgr.24437) 709 : cluster [DBG] pgmap v1275: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:44:53.538 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:51.847734+0000 osd.0 (osd.0) 83 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:44:53.538 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:51.850785+0000 osd.0 (osd.0) 84 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:44:53.538 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:52.036666+0000 osd.6 (osd.6) 364 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:44:53.538 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:52.039573+0000 osd.6 (osd.6) 365 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:44:53.538 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:52.274694+0000 mon.a (mon.0) 3981 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-23T00:44:53.538 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:52.282813+0000 osd.4 (osd.4) 365 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:44:53.538 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:52.283928+0000 osd.4 (osd.4) 366 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:44:53.538 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:53.092596+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.87:0/1839051234' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:53.538 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:53.092955+0000 mon.a (mon.0) 3982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:51.822739+0000 mgr.y (mgr.24437) 709 : cluster [DBG] pgmap v1275: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:44:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:51.847734+0000 osd.0 (osd.0) 83 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:44:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:51.850785+0000 osd.0 (osd.0) 84 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:44:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:52.036666+0000 osd.6 (osd.6) 364 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:44:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:52.039573+0000 osd.6 (osd.6) 365 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:44:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:52.274694+0000 mon.a (mon.0) 3981 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-23T00:44:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:52.282813+0000 osd.4 (osd.4) 365 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:44:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:52.283928+0000 osd.4 (osd.4) 366 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:44:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:53.092596+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.87:0/1839051234' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:53.092955+0000 mon.a (mon.0) 3982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-23T00:44:53.792 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:54.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:44:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:44:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:44:54.425 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight 3 1 2024-02-23T00:44:54.674 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:53.506316+0000 mon.a (mon.0) 3983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-23T00:44:54.674 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:53.506389+0000 mon.a (mon.0) 3984 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-23T00:44:54.674 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:53.590897+0000 osd.1 (osd.1) 404 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:44:54.675 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:53.594082+0000 osd.1 (osd.1) 405 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:44:54.675 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:53.823138+0000 mgr.y (mgr.24437) 710 : cluster [DBG] pgmap v1278: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:54.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:53.506316+0000 mon.a (mon.0) 3983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-23T00:44:54.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:53.506389+0000 mon.a (mon.0) 3984 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-23T00:44:54.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:53.590897+0000 osd.1 (osd.1) 404 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:44:54.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:53.594082+0000 osd.1 (osd.1) 405 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:44:54.675 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:53.823138+0000 mgr.y (mgr.24437) 710 : cluster [DBG] pgmap v1278: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:54.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:53.506316+0000 mon.a (mon.0) 3983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-23T00:44:54.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:53.506389+0000 mon.a (mon.0) 3984 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-23T00:44:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:53.590897+0000 osd.1 (osd.1) 404 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:44:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:53.594082+0000 osd.1 (osd.1) 405 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:44:54.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:53.823138+0000 mgr.y (mgr.24437) 710 : cluster [DBG] pgmap v1278: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:55.263 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:44:55.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:53.899865+0000 osd.0 (osd.0) 85 : cluster [DBG] 2.13 deep-scrub starts 2024-02-23T00:44:55.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:53.903002+0000 osd.0 (osd.0) 86 : cluster [DBG] 2.13 deep-scrub ok 2024-02-23T00:44:55.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:54.638699+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:44:55.694 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:54.642085+0000 osd.1 (osd.1) 407 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:44:55.694 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:53.899865+0000 osd.0 (osd.0) 85 : cluster [DBG] 2.13 deep-scrub starts 2024-02-23T00:44:55.694 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:53.903002+0000 osd.0 (osd.0) 86 : cluster [DBG] 2.13 deep-scrub ok 2024-02-23T00:44:55.694 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:54.638699+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:44:55.694 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:54.642085+0000 osd.1 (osd.1) 407 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:44:55.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:53.899865+0000 osd.0 (osd.0) 85 : cluster [DBG] 2.13 deep-scrub starts 2024-02-23T00:44:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:53.903002+0000 osd.0 (osd.0) 86 : cluster [DBG] 2.13 deep-scrub ok 2024-02-23T00:44:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:54.638699+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:44:55.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:54.642085+0000 osd.1 (osd.1) 407 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:44:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:54.884264+0000 osd.3 (osd.3) 283 : cluster [DBG] 4.19 deep-scrub starts 2024-02-23T00:44:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:54.888342+0000 osd.3 (osd.3) 284 : cluster [DBG] 4.19 deep-scrub ok 2024-02-23T00:44:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:55.823901+0000 mgr.y (mgr.24437) 711 : cluster [DBG] pgmap v1279: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:44:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:56.100510+0000 osd.5 (osd.5) 341 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:44:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:56.101766+0000 osd.5 (osd.5) 342 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:44:56.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:56.231169+0000 osd.4 (osd.4) 367 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:44:56.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:56.232105+0000 osd.4 (osd.4) 368 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:44:56.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:54.884264+0000 osd.3 (osd.3) 283 : cluster [DBG] 4.19 deep-scrub starts 2024-02-23T00:44:56.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:54.888342+0000 osd.3 (osd.3) 284 : cluster [DBG] 4.19 deep-scrub ok 2024-02-23T00:44:56.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:55.823901+0000 mgr.y (mgr.24437) 711 : cluster [DBG] pgmap v1279: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:44:56.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:56.100510+0000 osd.5 (osd.5) 341 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:44:56.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:56.101766+0000 osd.5 (osd.5) 342 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:44:56.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:56.231169+0000 osd.4 (osd.4) 367 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:44:56.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:56.232105+0000 osd.4 (osd.4) 368 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:44:56.911 INFO:teuthology.orchestra.run.smithi087.stdout:ERROR: (22) Invalid argument 2024-02-23T00:44:56.911 INFO:teuthology.orchestra.run.smithi087.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-23T00:44:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:54.884264+0000 osd.3 (osd.3) 283 : cluster [DBG] 4.19 deep-scrub starts 2024-02-23T00:44:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:54.888342+0000 osd.3 (osd.3) 284 : cluster [DBG] 4.19 deep-scrub ok 2024-02-23T00:44:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:55.823901+0000 mgr.y (mgr.24437) 711 : cluster [DBG] pgmap v1279: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:44:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:56.100510+0000 osd.5 (osd.5) 341 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:44:56.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:56.101766+0000 osd.5 (osd.5) 342 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:44:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:56.231169+0000 osd.4 (osd.4) 367 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:44:56.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:56.232105+0000 osd.4 (osd.4) 368 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:44:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:55.912227+0000 osd.0 (osd.0) 87 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:44:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:55.913149+0000 osd.0 (osd.0) 88 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:44:57.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:57.275738+0000 mon.a (mon.0) 3985 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-23T00:44:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:55.912227+0000 osd.0 (osd.0) 87 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:44:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:55.913149+0000 osd.0 (osd.0) 88 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:44:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:57.275738+0000 mon.a (mon.0) 3985 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-23T00:44:57.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:55.912227+0000 osd.0 (osd.0) 87 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:44:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:55.913149+0000 osd.0 (osd.0) 88 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:44:57.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:57.275738+0000 mon.a (mon.0) 3985 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-23T00:44:58.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.723+0000 7f57dd64c700 1 -- 172.21.15.87:0/4095896026 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 msgr2=0x7f57d80f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:58.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.723+0000 7f57dd64c700 1 --2- 172.21.15.87:0/4095896026 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 0x7f57d80f6c50 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 rx=0x7f57cc0047b0 tx=0x7f57cc009cb0).stop 2024-02-23T00:44:58.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.723+0000 7f57dd64c700 1 -- 172.21.15.87:0/4095896026 shutdown_connections 2024-02-23T00:44:58.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.723+0000 7f57dd64c700 1 --2- 172.21.15.87:0/4095896026 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57d80f7330 0x7f57d8103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:58.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.723+0000 7f57dd64c700 1 --2- 172.21.15.87:0/4095896026 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57d80fcac0 0x7f57d80fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:58.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.723+0000 7f57dd64c700 1 --2- 172.21.15.87:0/4095896026 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 0x7f57d80f6c50 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:58.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.723+0000 7f57dd64c700 1 -- 172.21.15.87:0/4095896026 >> 172.21.15.87:0/4095896026 conn(0x7f57d80f26f0 msgr2=0x7f57d80f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:58.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.723+0000 7f57dd64c700 1 -- 172.21.15.87:0/4095896026 shutdown_connections 2024-02-23T00:44:58.727 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.723+0000 7f57dd64c700 1 -- 172.21.15.87:0/4095896026 wait complete. 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 librados: starting msgr at 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 librados: starting objecter 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 Processor -- start 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 -- start start 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 librados: setting wanted keys 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 librados: calling monclient init 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57d80f7330 0x7f57d8110300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57d80fcac0 0x7f57d81107f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 0x7f57d810a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d810a990 con 0x7f57d80fd420 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d810ab00 con 0x7f57d80fcac0 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57dd64c700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d810acc0 con 0x7f57d80f7330 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57d77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 0x7f57d810a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:58.728 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.724+0000 7f57d6ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57d80f7330 0x7f57d8110300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d77fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 0x7f57d810a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39822/0 (socket says 172.21.15.87:39822) 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d6ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57d80f7330 0x7f57d8110300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:42432/0 (socket says 172.21.15.87:42432) 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d77fe700 1 -- 172.21.15.87:0/1653845398 learned_addr learned my addr 172.21.15.87:0/1653845398 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d67fc700 1 --2- 172.21.15.87:0/1653845398 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57d80fcac0 0x7f57d81107f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d77fe700 1 -- 172.21.15.87:0/1653845398 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57d80f7330 msgr2=0x7f57d8110300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d77fe700 1 --2- 172.21.15.87:0/1653845398 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57d80f7330 0x7f57d8110300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d77fe700 1 -- 172.21.15.87:0/1653845398 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57d80fcac0 msgr2=0x7f57d81107f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d77fe700 1 --2- 172.21.15.87:0/1653845398 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57d80fcac0 0x7f57d81107f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d77fe700 1 -- 172.21.15.87:0/1653845398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc005040 con 0x7f57d80fd420 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d6ffd700 1 --2- 172.21.15.87:0/1653845398 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57d80f7330 0x7f57d8110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57d77fe700 1 --2- 172.21.15.87:0/1653845398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 0x7f57d810a4a0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7f57d000d970 tx=0x7f57d0004e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57bffff700 10 client.?.objecter ms_handle_connect 0x7f57d80fd420 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57bffff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57bffff700 1 -- 172.21.15.87:0/1653845398 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0004020 con 0x7f57d80fd420 2024-02-23T00:44:58.729 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d810afa0 con 0x7f57d80fd420 2024-02-23T00:44:58.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57bffff700 1 -- 172.21.15.87:0/1653845398 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57d000f040 con 0x7f57d80fd420 2024-02-23T00:44:58.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57bffff700 1 -- 172.21.15.87:0/1653845398 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d00036c0 con 0x7f57d80fd420 2024-02-23T00:44:58.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57dd64c700 10 client.28014.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:44:58.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d810b320 con 0x7f57d80fd420 2024-02-23T00:44:58.730 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.725+0000 7f57dd64c700 1 librados: init done 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.726+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d805f990 con 0x7f57d80fd420 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.726+0000 7f57bffff700 1 -- 172.21.15.87:0/1653845398 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f57d0003af0 con 0x7f57d80fd420 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.727+0000 7f57bffff700 1 --2- 172.21.15.87:0/1653845398 >> v2:172.21.15.87:6800/1231597463 conn(0x7f57c0062c90 0x7f57c0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.727+0000 7f57bffff700 1 -- 172.21.15.87:0/1653845398 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(825..825 src has 400..825) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f57d007ca80 con 0x7f57d80fd420 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.727+0000 7f57bffff700 10 client.28014.objecter ms_dispatch 0x7f57d8001550 osd_map(825..825 src has 400..825) v4 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.727+0000 7f57bffff700 3 client.28014.objecter handle_osd_map got epochs [825,825] > 0 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.727+0000 7f57bffff700 3 client.28014.objecter handle_osd_map decoding full epoch 825 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.727+0000 7f57d6ffd700 1 --2- 172.21.15.87:0/1653845398 >> v2:172.21.15.87:6800/1231597463 conn(0x7f57c0062c90 0x7f57c0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.727+0000 7f57bffff700 20 client.28014.objecter dump_active .. 0 homeless 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.727+0000 7f57bffff700 10 client.28014.objecter ms_handle_connect 0x7f57c0062c90 2024-02-23T00:44:58.731 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.727+0000 7f57d6ffd700 1 --2- 172.21.15.87:0/1653845398 >> v2:172.21.15.87:6800/1231597463 conn(0x7f57c0062c90 0x7f57c0065150 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7f57c8004fd0 tx=0x7f57c800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:44:58.732 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.731+0000 7f57bffff700 1 -- 172.21.15.87:0/1653845398 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f57d0048f60 con 0x7f57d80fd420 2024-02-23T00:44:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:56.819614+0000 osd.3 (osd.3) 285 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:44:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:56.822685+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:44:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:57.824250+0000 mgr.y (mgr.24437) 712 : cluster [DBG] pgmap v1281: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:58.023502+0000 osd.5 (osd.5) 343 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:44:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:58.024943+0000 osd.5 (osd.5) 344 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:44:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:58.243140+0000 osd.4 (osd.4) 369 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:44:58.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:58.244949+0000 osd.4 (osd.4) 370 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:44:58.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:56.819614+0000 osd.3 (osd.3) 285 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:44:58.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:56.822685+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:44:58.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:57.824250+0000 mgr.y (mgr.24437) 712 : cluster [DBG] pgmap v1281: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:58.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:58.023502+0000 osd.5 (osd.5) 343 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:44:58.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:58.024943+0000 osd.5 (osd.5) 344 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:44:58.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:58.243140+0000 osd.4 (osd.4) 369 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:44:58.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:58.244949+0000 osd.4 (osd.4) 370 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:44:58.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:58.909+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f57d805ee80 con 0x7f57d80fd420 2024-02-23T00:44:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:56.819614+0000 osd.3 (osd.3) 285 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:44:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:56.822685+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:44:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:57.824250+0000 mgr.y (mgr.24437) 712 : cluster [DBG] pgmap v1281: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:44:58.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:58.023502+0000 osd.5 (osd.5) 343 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:44:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:58.024943+0000 osd.5 (osd.5) 344 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:44:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:58.243140+0000 osd.4 (osd.4) 369 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:44:58.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:58.244949+0000 osd.4 (osd.4) 370 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:44:59.529 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.527+0000 7f57bffff700 1 -- 172.21.15.87:0/1653845398 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v826) v1 ==== 113+0+0 (secure 0 0 0) 0x7f57d0050320 con 0x7f57d80fd420 2024-02-23T00:44:59.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 10 librados: watch_flush enter 2024-02-23T00:44:59.531 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 10 librados: watch_flush exit 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 20 client.28014.objecter shutdown clearing up homeless session... 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 10 client.28014.objecter successfully canceled tick 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 >> v2:172.21.15.87:6800/1231597463 conn(0x7f57c0062c90 msgr2=0x7f57c0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 --2- 172.21.15.87:0/1653845398 >> v2:172.21.15.87:6800/1231597463 conn(0x7f57c0062c90 0x7f57c0065150 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 rx=0x7f57c8004fd0 tx=0x7f57c800b040).stop 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 msgr2=0x7f57d810a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 --2- 172.21.15.87:0/1653845398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 0x7f57d810a4a0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 rx=0x7f57d000d970 tx=0x7f57d0004e60).stop 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 shutdown_connections 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 --2- 172.21.15.87:0/1653845398 >> v2:172.21.15.87:6800/1231597463 conn(0x7f57c0062c90 0x7f57c0065150 unknown :-1 s=CLOSED pgs=2997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 --2- 172.21.15.87:0/1653845398 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f57d80fd420 0x7f57d810a4a0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 --2- 172.21.15.87:0/1653845398 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f57d80f7330 0x7f57d8110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 --2- 172.21.15.87:0/1653845398 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f57d80fcac0 0x7f57d81107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 >> 172.21.15.87:0/1653845398 conn(0x7f57d80f26f0 msgr2=0x7f57d8060e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 shutdown_connections 2024-02-23T00:44:59.532 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 -- 172.21.15.87:0/1653845398 wait complete. 2024-02-23T00:44:59.533 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:44:59.530+0000 7f57dd64c700 1 librados: shutdown 2024-02-23T00:44:59.533 INFO:teuthology.orchestra.run.smithi087.stderr:reweighted osd.3 to 1 (10000) 2024-02-23T00:44:59.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:57.840925+0000 osd.3 (osd.3) 287 : cluster [DBG] 4.14 deep-scrub starts 2024-02-23T00:44:59.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:57.844463+0000 osd.3 (osd.3) 288 : cluster [DBG] 4.14 deep-scrub ok 2024-02-23T00:44:59.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:57.879290+0000 osd.0 (osd.0) 89 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:44:59.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:57.880308+0000 osd.0 (osd.0) 90 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:44:59.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:58.166699+0000 osd.6 (osd.6) 366 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:44:59.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:58.170339+0000 osd.6 (osd.6) 367 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:44:59.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:58.909727+0000 mon.a (mon.0) 3986 : audit [INF] from='client.? 172.21.15.87:0/1653845398' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-23T00:44:59.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:57.840925+0000 osd.3 (osd.3) 287 : cluster [DBG] 4.14 deep-scrub starts 2024-02-23T00:44:59.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:57.844463+0000 osd.3 (osd.3) 288 : cluster [DBG] 4.14 deep-scrub ok 2024-02-23T00:44:59.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:57.879290+0000 osd.0 (osd.0) 89 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:44:59.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:57.880308+0000 osd.0 (osd.0) 90 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:44:59.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:58.166699+0000 osd.6 (osd.6) 366 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:44:59.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:58.170339+0000 osd.6 (osd.6) 367 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:44:59.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:44:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:58.909727+0000 mon.a (mon.0) 3986 : audit [INF] from='client.? 172.21.15.87:0/1653845398' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-23T00:44:59.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:57.840925+0000 osd.3 (osd.3) 287 : cluster [DBG] 4.14 deep-scrub starts 2024-02-23T00:44:59.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:57.844463+0000 osd.3 (osd.3) 288 : cluster [DBG] 4.14 deep-scrub ok 2024-02-23T00:44:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:57.879290+0000 osd.0 (osd.0) 89 : cluster [DBG] 6.9 scrub starts 2024-02-23T00:44:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:57.880308+0000 osd.0 (osd.0) 90 : cluster [DBG] 6.9 scrub ok 2024-02-23T00:44:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:58.166699+0000 osd.6 (osd.6) 366 : cluster [DBG] 22.1 scrub starts 2024-02-23T00:44:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:58.170339+0000 osd.6 (osd.6) 367 : cluster [DBG] 22.1 scrub ok 2024-02-23T00:44:59.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:44:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:58.909727+0000 mon.a (mon.0) 3986 : audit [INF] from='client.? 172.21.15.87:0/1653845398' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-23T00:45:00.341 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-23T00:45:00.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:59.437304+0000 osd.7 (osd.7) 400 : cluster [DBG] 3.11 scrub starts 2024-02-23T00:45:00.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:59.438338+0000 osd.7 (osd.7) 401 : cluster [DBG] 3.11 scrub ok 2024-02-23T00:45:00.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:44:59.526983+0000 mon.a (mon.0) 3987 : audit [INF] from='client.? 172.21.15.87:0/1653845398' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:59.527068+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:59.824737+0000 mgr.y (mgr.24437) 713 : cluster [DBG] pgmap v1283: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:00.048393+0000 osd.5 (osd.5) 345 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:00.050013+0000 osd.5 (osd.5) 346 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:59.437304+0000 osd.7 (osd.7) 400 : cluster [DBG] 3.11 scrub starts 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:59.438338+0000 osd.7 (osd.7) 401 : cluster [DBG] 3.11 scrub ok 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:44:59.526983+0000 mon.a (mon.0) 3987 : audit [INF] from='client.? 172.21.15.87:0/1653845398' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:59.527068+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:59.824737+0000 mgr.y (mgr.24437) 713 : cluster [DBG] pgmap v1283: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:00.048393+0000 osd.5 (osd.5) 345 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:45:00.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:00.050013+0000 osd.5 (osd.5) 346 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:45:00.616 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:00.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:59.437304+0000 osd.7 (osd.7) 400 : cluster [DBG] 3.11 scrub starts 2024-02-23T00:45:00.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:59.438338+0000 osd.7 (osd.7) 401 : cluster [DBG] 3.11 scrub ok 2024-02-23T00:45:00.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:44:59.526983+0000 mon.a (mon.0) 3987 : audit [INF] from='client.? 172.21.15.87:0/1653845398' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-23T00:45:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:59.527068+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-23T00:45:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:59.824737+0000 mgr.y (mgr.24437) 713 : cluster [DBG] pgmap v1283: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-02-23T00:45:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:00.048393+0000 osd.5 (osd.5) 345 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:45:00.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:00.050013+0000 osd.5 (osd.5) 346 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:45:01.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:59.788924+0000 osd.3 (osd.3) 289 : cluster [DBG] 22.3 deep-scrub starts 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:59.793156+0000 osd.3 (osd.3) 290 : cluster [DBG] 22.3 deep-scrub ok 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:59.924757+0000 osd.0 (osd.0) 91 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:44:59.925822+0000 osd.0 (osd.0) 92 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:00.354470+0000 osd.2 (osd.2) 311 : cluster [DBG] 4.d scrub starts 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:00.356612+0000 osd.2 (osd.2) 312 : cluster [DBG] 4.d scrub ok 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:00.389836+0000 osd.1 (osd.1) 408 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:00.390104+0000 osd.1 (osd.1) 409 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:00.452804+0000 osd.7 (osd.7) 402 : cluster [DBG] 22.a deep-scrub starts 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:00.454071+0000 osd.7 (osd.7) 403 : cluster [DBG] 22.a deep-scrub ok 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:00.541064+0000 mon.a (mon.0) 3989 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-23T00:45:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:01.056078+0000 osd.5 (osd.5) 347 : cluster [DBG] 3.c scrub starts 2024-02-23T00:45:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:01.058386+0000 osd.5 (osd.5) 348 : cluster [DBG] 3.c scrub ok 2024-02-23T00:45:01.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:01.388786+0000 mgr.y (mgr.24437) 714 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:59.788924+0000 osd.3 (osd.3) 289 : cluster [DBG] 22.3 deep-scrub starts 2024-02-23T00:45:01.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:59.793156+0000 osd.3 (osd.3) 290 : cluster [DBG] 22.3 deep-scrub ok 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:59.924757+0000 osd.0 (osd.0) 91 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:44:59.925822+0000 osd.0 (osd.0) 92 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:00.354470+0000 osd.2 (osd.2) 311 : cluster [DBG] 4.d scrub starts 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:00.356612+0000 osd.2 (osd.2) 312 : cluster [DBG] 4.d scrub ok 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:00.389836+0000 osd.1 (osd.1) 408 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:00.390104+0000 osd.1 (osd.1) 409 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:00.452804+0000 osd.7 (osd.7) 402 : cluster [DBG] 22.a deep-scrub starts 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:00.454071+0000 osd.7 (osd.7) 403 : cluster [DBG] 22.a deep-scrub ok 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:00.541064+0000 mon.a (mon.0) 3989 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:01.056078+0000 osd.5 (osd.5) 347 : cluster [DBG] 3.c scrub starts 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:01.058386+0000 osd.5 (osd.5) 348 : cluster [DBG] 3.c scrub ok 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:01.388786+0000 mgr.y (mgr.24437) 714 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:01.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:59.788924+0000 osd.3 (osd.3) 289 : cluster [DBG] 22.3 deep-scrub starts 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:59.793156+0000 osd.3 (osd.3) 290 : cluster [DBG] 22.3 deep-scrub ok 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:59.924757+0000 osd.0 (osd.0) 91 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:44:59.925822+0000 osd.0 (osd.0) 92 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:00.354470+0000 osd.2 (osd.2) 311 : cluster [DBG] 4.d scrub starts 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:00.356612+0000 osd.2 (osd.2) 312 : cluster [DBG] 4.d scrub ok 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:00.389836+0000 osd.1 (osd.1) 408 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:00.390104+0000 osd.1 (osd.1) 409 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:00.452804+0000 osd.7 (osd.7) 402 : cluster [DBG] 22.a deep-scrub starts 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:00.454071+0000 osd.7 (osd.7) 403 : cluster [DBG] 22.a deep-scrub ok 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:00.541064+0000 mon.a (mon.0) 3989 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:01.056078+0000 osd.5 (osd.5) 347 : cluster [DBG] 3.c scrub starts 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:01.058386+0000 osd.5 (osd.5) 348 : cluster [DBG] 3.c scrub ok 2024-02-23T00:45:01.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:01.388786+0000 mgr.y (mgr.24437) 714 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:02.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.312+0000 7fb8ace18700 1 -- 172.21.15.87:0/4192925301 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8a80fe510 msgr2=0x7fb8a80f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:02.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.312+0000 7fb8ace18700 1 --2- 172.21.15.87:0/4192925301 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8a80fe510 0x7fb8a80f7d40 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 rx=0x7fb8900047b0 tx=0x7fb890009cb0).stop 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.312+0000 7fb8ace18700 1 -- 172.21.15.87:0/4192925301 shutdown_connections 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.312+0000 7fb8ace18700 1 --2- 172.21.15.87:0/4192925301 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8a80f8420 0x7fb8a8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.312+0000 7fb8ace18700 1 --2- 172.21.15.87:0/4192925301 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8a80fdbb0 0x7fb8a80fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.312+0000 7fb8ace18700 1 --2- 172.21.15.87:0/4192925301 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8a80fe510 0x7fb8a80f7d40 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.312+0000 7fb8ace18700 1 -- 172.21.15.87:0/4192925301 >> 172.21.15.87:0/4192925301 conn(0x7fb8a80f3800 msgr2=0x7fb8a80f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.312+0000 7fb8ace18700 1 -- 172.21.15.87:0/4192925301 shutdown_connections 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.312+0000 7fb8ace18700 1 -- 172.21.15.87:0/4192925301 wait complete. 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 librados: starting msgr at 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 librados: starting objecter 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 Processor -- start 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 -- start start 2024-02-23T00:45:02.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 librados: setting wanted keys 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 librados: calling monclient init 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8a80f8420 0x7fb8a810e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8a80fdbb0 0x7fb8a810e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8a80fe510 0x7fb8a81081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a81086b0 con 0x7fb8a80fdbb0 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a8108820 con 0x7fb8a80fe510 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8ace18700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a81089e0 con 0x7fb8a80f8420 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8a5d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8a80fdbb0 0x7fb8a810e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8a5d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8a80fdbb0 0x7fb8a810e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58536/0 (socket says 172.21.15.87:58536) 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8a5d9b700 1 -- 172.21.15.87:0/2787110643 learned_addr learned my addr 172.21.15.87:0/2787110643 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8a659c700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8a80f8420 0x7fb8a810e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.313+0000 7fb8a6d9d700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8a80fe510 0x7fb8a81081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:02.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8a659c700 1 -- 172.21.15.87:0/2787110643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8a80fe510 msgr2=0x7fb8a81081c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8a659c700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8a80fe510 0x7fb8a81081c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8a659c700 1 -- 172.21.15.87:0/2787110643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8a80fdbb0 msgr2=0x7fb8a810e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8a659c700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8a80fdbb0 0x7fb8a810e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8a659c700 1 -- 172.21.15.87:0/2787110643 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980097c0 con 0x7fb8a80f8420 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8a5d9b700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8a80fdbb0 0x7fb8a810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8a659c700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8a80f8420 0x7fb8a810e020 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7fb898001e40 tx=0x7fb898004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb89f7fe700 10 client.?.objecter ms_handle_connect 0x7fb8a80f8420 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb89f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb89f7fe700 1 -- 172.21.15.87:0/2787110643 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898009aa0 con 0x7fb8a80f8420 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb89f7fe700 1 -- 172.21.15.87:0/2787110643 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8980090d0 con 0x7fb8a80f8420 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb890005040 con 0x7fb8a80f8420 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb89f7fe700 1 -- 172.21.15.87:0/2787110643 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898007d50 con 0x7fb8a80f8420 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8ace18700 10 client.38495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a8114e50 con 0x7fb8a80f8420 2024-02-23T00:45:02.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.314+0000 7fb8ace18700 1 librados: init done 2024-02-23T00:45:02.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.315+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a8107110 con 0x7fb8a80f8420 2024-02-23T00:45:02.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.315+0000 7fb89f7fe700 1 -- 172.21.15.87:0/2787110643 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb89800fb30 con 0x7fb8a80f8420 2024-02-23T00:45:02.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.316+0000 7fb89f7fe700 1 --2- 172.21.15.87:0/2787110643 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb894062c90 0x7fb894065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:02.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.316+0000 7fb89f7fe700 1 -- 172.21.15.87:0/2787110643 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(827..827 src has 400..827) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fb89807d7b0 con 0x7fb8a80f8420 2024-02-23T00:45:02.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.316+0000 7fb89f7fe700 10 client.38495.objecter ms_dispatch 0x7fb8a8001550 osd_map(827..827 src has 400..827) v4 2024-02-23T00:45:02.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.316+0000 7fb89f7fe700 3 client.38495.objecter handle_osd_map got epochs [827,827] > 0 2024-02-23T00:45:02.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.316+0000 7fb8a5d9b700 1 --2- 172.21.15.87:0/2787110643 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb894062c90 0x7fb894065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:02.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.316+0000 7fb89f7fe700 3 client.38495.objecter handle_osd_map decoding full epoch 827 2024-02-23T00:45:02.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.316+0000 7fb89f7fe700 20 client.38495.objecter dump_active .. 0 homeless 2024-02-23T00:45:02.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.316+0000 7fb89f7fe700 10 client.38495.objecter ms_handle_connect 0x7fb894062c90 2024-02-23T00:45:02.321 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.316+0000 7fb8a5d9b700 1 --2- 172.21.15.87:0/2787110643 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb894062c90 0x7fb894065150 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 rx=0x7fb890009f20 tx=0x7fb890007f90).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:02.322 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.320+0000 7fb89f7fe700 1 -- 172.21.15.87:0/2787110643 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb898050fa0 con 0x7fb8a80f8420 2024-02-23T00:45:02.499 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:02.498+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) v1 -- 0x7fb8a81096a0 con 0x7fb8a80f8420 2024-02-23T00:45:02.546 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:02] "GET /metrics HTTP/1.1" 200 186063 "" "Prometheus/2.33.4" 2024-02-23T00:45:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:01.566881+0000 osd.1 (osd.1) 410 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:45:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:01.569942+0000 osd.1 (osd.1) 411 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:45:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:01.825665+0000 mgr.y (mgr.24437) 715 : cluster [DBG] pgmap v1285: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:02.498645+0000 mon.c (mon.2) 512 : audit [INF] from='client.? 172.21.15.87:0/2787110643' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-23T00:45:02.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:02.498932+0000 mon.a (mon.0) 3990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-23T00:45:02.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:01.566881+0000 osd.1 (osd.1) 410 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:45:02.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:01.569942+0000 osd.1 (osd.1) 411 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:45:02.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:01.825665+0000 mgr.y (mgr.24437) 715 : cluster [DBG] pgmap v1285: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:02.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:02.498645+0000 mon.c (mon.2) 512 : audit [INF] from='client.? 172.21.15.87:0/2787110643' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-23T00:45:02.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:02.498932+0000 mon.a (mon.0) 3990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-23T00:45:02.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:01.566881+0000 osd.1 (osd.1) 410 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:45:02.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:01.569942+0000 osd.1 (osd.1) 411 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:45:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:01.825665+0000 mgr.y (mgr.24437) 715 : cluster [DBG] pgmap v1285: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:02.498645+0000 mon.c (mon.2) 512 : audit [INF] from='client.? 172.21.15.87:0/2787110643' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-23T00:45:02.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:02.498932+0000 mon.a (mon.0) 3990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-23T00:45:03.281 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.279+0000 7fb89f7fe700 1 -- 172.21.15.87:0/2787110643 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v828) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb898050b90 con 0x7fb8a80f8420 2024-02-23T00:45:03.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.281+0000 7fb8ace18700 10 librados: watch_flush enter 2024-02-23T00:45:03.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.281+0000 7fb8ace18700 10 librados: watch_flush exit 2024-02-23T00:45:03.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.281+0000 7fb8ace18700 20 client.38495.objecter shutdown clearing up homeless session... 2024-02-23T00:45:03.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.281+0000 7fb8ace18700 10 client.38495.objecter successfully canceled tick 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.281+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb894062c90 msgr2=0x7fb894065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.281+0000 7fb8ace18700 1 --2- 172.21.15.87:0/2787110643 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb894062c90 0x7fb894065150 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 rx=0x7fb890009f20 tx=0x7fb890007f90).stop 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.281+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8a80f8420 msgr2=0x7fb8a810e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.281+0000 7fb8ace18700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8a80f8420 0x7fb8a810e020 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 rx=0x7fb898001e40 tx=0x7fb898004d10).stop 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.282+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 shutdown_connections 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.282+0000 7fb8ace18700 1 --2- 172.21.15.87:0/2787110643 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb894062c90 0x7fb894065150 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.282+0000 7fb8ace18700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb8a80fe510 0x7fb8a81081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.282+0000 7fb8ace18700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb8a80f8420 0x7fb8a810e020 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.282+0000 7fb8ace18700 1 --2- 172.21.15.87:0/2787110643 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb8a80fdbb0 0x7fb8a810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.282+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 >> 172.21.15.87:0/2787110643 conn(0x7fb8a80f3800 msgr2=0x7fb8a80f5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.282+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 shutdown_connections 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.282+0000 7fb8ace18700 1 -- 172.21.15.87:0/2787110643 wait complete. 2024-02-23T00:45:03.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:03.282+0000 7fb8ace18700 1 librados: shutdown 2024-02-23T00:45:03.285 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-23T00:45:03.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:02.355164+0000 osd.7 (osd.7) 404 : cluster [DBG] 6.8 deep-scrub starts 2024-02-23T00:45:03.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:02.358052+0000 osd.7 (osd.7) 405 : cluster [DBG] 6.8 deep-scrub ok 2024-02-23T00:45:03.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:02.750318+0000 osd.3 (osd.3) 291 : cluster [DBG] 22.4 deep-scrub starts 2024-02-23T00:45:03.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:02.753325+0000 osd.3 (osd.3) 292 : cluster [DBG] 22.4 deep-scrub ok 2024-02-23T00:45:03.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:03.279445+0000 mon.a (mon.0) 3991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-23T00:45:03.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:03.279528+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-23T00:45:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:02.355164+0000 osd.7 (osd.7) 404 : cluster [DBG] 6.8 deep-scrub starts 2024-02-23T00:45:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:02.358052+0000 osd.7 (osd.7) 405 : cluster [DBG] 6.8 deep-scrub ok 2024-02-23T00:45:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:02.750318+0000 osd.3 (osd.3) 291 : cluster [DBG] 22.4 deep-scrub starts 2024-02-23T00:45:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:02.753325+0000 osd.3 (osd.3) 292 : cluster [DBG] 22.4 deep-scrub ok 2024-02-23T00:45:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:03.279445+0000 mon.a (mon.0) 3991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-23T00:45:03.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:03.279528+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-23T00:45:03.943 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:45:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:45:03.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:02.355164+0000 osd.7 (osd.7) 404 : cluster [DBG] 6.8 deep-scrub starts 2024-02-23T00:45:03.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:02.358052+0000 osd.7 (osd.7) 405 : cluster [DBG] 6.8 deep-scrub ok 2024-02-23T00:45:03.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:02.750318+0000 osd.3 (osd.3) 291 : cluster [DBG] 22.4 deep-scrub starts 2024-02-23T00:45:03.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:02.753325+0000 osd.3 (osd.3) 292 : cluster [DBG] 22.4 deep-scrub ok 2024-02-23T00:45:03.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:03.279445+0000 mon.a (mon.0) 3991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-23T00:45:03.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:03.279528+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-23T00:45:04.071 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight 4 1 2024-02-23T00:45:04.343 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:03.739517+0000 osd.3 (osd.3) 293 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:45:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:03.742400+0000 osd.3 (osd.3) 294 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:45:04.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:03.826045+0000 mgr.y (mgr.24437) 716 : cluster [DBG] pgmap v1287: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:04.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:03.739517+0000 osd.3 (osd.3) 293 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:45:04.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:03.742400+0000 osd.3 (osd.3) 294 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:45:04.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:03.826045+0000 mgr.y (mgr.24437) 716 : cluster [DBG] pgmap v1287: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:03.739517+0000 osd.3 (osd.3) 293 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:45:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:03.742400+0000 osd.3 (osd.3) 294 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:45:04.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:03.826045+0000 mgr.y (mgr.24437) 716 : cluster [DBG] pgmap v1287: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:05.819 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:03.806160+0000 osd.0 (osd.0) 93 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:45:05.819 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:03.809631+0000 osd.0 (osd.0) 94 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:45:05.819 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:04.712460+0000 osd.3 (osd.3) 295 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:45:05.820 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:04.715459+0000 osd.3 (osd.3) 296 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:45:05.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:03.806160+0000 osd.0 (osd.0) 93 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:45:05.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:03.809631+0000 osd.0 (osd.0) 94 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:45:05.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:04.712460+0000 osd.3 (osd.3) 295 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:45:05.820 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:04.715459+0000 osd.3 (osd.3) 296 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:45:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:03.806160+0000 osd.0 (osd.0) 93 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:45:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:03.809631+0000 osd.0 (osd.0) 94 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:45:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:04.712460+0000 osd.3 (osd.3) 295 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:45:05.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:04.715459+0000 osd.3 (osd.3) 296 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:45:06.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.279+0000 7f1246554700 1 -- 172.21.15.87:0/1783306865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12400f8cb0 msgr2=0x7f12400f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:06.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.279+0000 7f1246554700 1 --2- 172.21.15.87:0/1783306865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12400f8cb0 0x7f12400f9090 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f1230004870 tx=0x7f1230009cb0).stop 2024-02-23T00:45:06.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.279+0000 7f1246554700 1 -- 172.21.15.87:0/1783306865 shutdown_connections 2024-02-23T00:45:06.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.279+0000 7f1246554700 1 --2- 172.21.15.87:0/1783306865 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1240102210 0x7f1240104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:45:06.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.279+0000 7f1246554700 1 --2- 172.21.15.87:0/1783306865 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f12400f8cb0 0x7f12400f9090 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:06.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.279+0000 7f1246554700 1 --2- 172.21.15.87:0/1783306865 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f12400f9610 0x7f1240101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:06.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.279+0000 7f1246554700 1 -- 172.21.15.87:0/1783306865 >> 172.21.15.87:0/1783306865 conn(0x7f12400f2820 msgr2=0x7f12400f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:06.316 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.279+0000 7f1246554700 1 -- 172.21.15.87:0/1783306865 shutdown_connections 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.279+0000 7f1246554700 1 -- 172.21.15.87:0/1783306865 wait complete. 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 librados: starting msgr at 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 librados: starting objecter 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 Processor -- start 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 -- start start 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 librados: setting wanted keys 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 librados: calling monclient init 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12400f8cb0 0x7f12400f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f12400f9610 0x7f12400f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1240102210 0x7f12400f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1240116810 con 0x7f1240102210 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1240116980 con 0x7f12400f8cb0 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1246554700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1240116b40 con 0x7f12400f9610 2024-02-23T00:45:06.317 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f123f7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f12400f9610 0x7f12400f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f1244af1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1240102210 0x7f12400f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f123f7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f12400f9610 0x7f12400f7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:37364/0 (socket says 172.21.15.87:37364) 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.280+0000 7f123f7fe700 1 -- 172.21.15.87:0/2124464942 learned_addr learned my addr 172.21.15.87:0/2124464942 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123ffff700 1 --2- 172.21.15.87:0/2124464942 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12400f8cb0 0x7f12400f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123f7fe700 1 -- 172.21.15.87:0/2124464942 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12400f8cb0 msgr2=0x7f12400f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123f7fe700 1 --2- 172.21.15.87:0/2124464942 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12400f8cb0 0x7f12400f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123f7fe700 1 -- 172.21.15.87:0/2124464942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1240102210 msgr2=0x7f12400f7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123f7fe700 1 --2- 172.21.15.87:0/2124464942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1240102210 0x7f12400f7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123f7fe700 1 -- 172.21.15.87:0/2124464942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1234005040 con 0x7f12400f9610 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f1244af1700 1 --2- 172.21.15.87:0/2124464942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1240102210 0x7f12400f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123f7fe700 1 --2- 172.21.15.87:0/2124464942 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f12400f9610 0x7f12400f7150 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7f12340048f0 tx=0x7f1234009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123d7fa700 10 client.?.objecter ms_handle_connect 0x7f12400f9610 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123d7fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:06.318 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123d7fa700 1 -- 172.21.15.87:0/2124464942 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234007b00 con 0x7f12400f9610 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1230005040 con 0x7f12400f9610 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123d7fa700 1 -- 172.21.15.87:0/2124464942 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f123400f040 con 0x7f12400f9610 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f123d7fa700 1 -- 172.21.15.87:0/2124464942 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234014990 con 0x7f12400f9610 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f1246554700 10 client.38501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1240117180 con 0x7f12400f9610 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.281+0000 7f1246554700 1 librados: init done 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.282+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124005fa60 con 0x7f12400f9610 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.282+0000 7f123d7fa700 1 -- 172.21.15.87:0/2124464942 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1234010500 con 0x7f12400f9610 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.283+0000 7f123d7fa700 1 --2- 172.21.15.87:0/2124464942 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1228062c90 0x7f1228065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.283+0000 7f123d7fa700 1 -- 172.21.15.87:0/2124464942 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(828..828 src has 400..828) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f123407cac0 con 0x7f12400f9610 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.283+0000 7f123d7fa700 10 client.38501.objecter ms_dispatch 0x7f1240001550 osd_map(828..828 src has 400..828) v4 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.283+0000 7f123d7fa700 3 client.38501.objecter handle_osd_map got epochs [828,828] > 0 2024-02-23T00:45:06.319 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.283+0000 7f123d7fa700 3 client.38501.objecter handle_osd_map decoding full epoch 828 2024-02-23T00:45:06.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.283+0000 7f123ffff700 1 --2- 172.21.15.87:0/2124464942 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1228062c90 0x7f1228065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:06.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.283+0000 7f123d7fa700 20 client.38501.objecter dump_active .. 0 homeless 2024-02-23T00:45:06.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.283+0000 7f123d7fa700 10 client.38501.objecter ms_handle_connect 0x7f1228062c90 2024-02-23T00:45:06.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.283+0000 7f123ffff700 1 --2- 172.21.15.87:0/2124464942 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1228062c90 0x7f1228065150 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f1230009a00 tx=0x7f1230009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:06.320 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.287+0000 7f123d7fa700 1 -- 172.21.15.87:0/2124464942 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1234049f60 con 0x7f12400f9610 2024-02-23T00:45:06.465 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.462+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f12401072e0 con 0x7f12400f9610 2024-02-23T00:45:06.559 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.557+0000 7f123d7fa700 1 -- 172.21.15.87:0/2124464942 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v829) v1 ==== 113+0+0 (secure 0 0 0) 0x7f1234050210 con 0x7f12400f9610 2024-02-23T00:45:06.561 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 10 librados: watch_flush enter 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 10 librados: watch_flush exit 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 20 client.38501.objecter shutdown clearing up homeless session... 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 10 client.38501.objecter successfully canceled tick 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1228062c90 msgr2=0x7f1228065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 --2- 172.21.15.87:0/2124464942 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1228062c90 0x7f1228065150 secure :-1 s=READY pgs=2999 cs=0 l=1 rev1=1 rx=0x7f1230009a00 tx=0x7f1230009cb0).stop 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f12400f9610 msgr2=0x7f12400f7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 --2- 172.21.15.87:0/2124464942 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f12400f9610 0x7f12400f7150 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 rx=0x7f12340048f0 tx=0x7f1234009cb0).stop 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 shutdown_connections 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 --2- 172.21.15.87:0/2124464942 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1228062c90 0x7f1228065150 unknown :-1 s=CLOSED pgs=2999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 --2- 172.21.15.87:0/2124464942 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1240102210 0x7f12400f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 --2- 172.21.15.87:0/2124464942 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f12400f8cb0 0x7f12400f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 --2- 172.21.15.87:0/2124464942 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f12400f9610 0x7f12400f7150 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 >> 172.21.15.87:0/2124464942 conn(0x7f12400f2820 msgr2=0x7f12400f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:06.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 shutdown_connections 2024-02-23T00:45:06.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 -- 172.21.15.87:0/2124464942 wait complete. 2024-02-23T00:45:06.563 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:06.559+0000 7f1246554700 1 librados: shutdown 2024-02-23T00:45:06.566 INFO:teuthology.orchestra.run.smithi087.stderr:reweighted osd.4 to 1 (10000) 2024-02-23T00:45:06.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:05.672532+0000 osd.3 (osd.3) 297 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:45:06.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:05.675507+0000 osd.3 (osd.3) 298 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:45:06.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:05.826966+0000 mgr.y (mgr.24437) 717 : cluster [DBG] pgmap v1288: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:06.372372+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.c scrub starts 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:06.373478+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.c scrub ok 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:06.464267+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.87:0/2124464942' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:06.464549+0000 mon.a (mon.0) 3993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:05.672532+0000 osd.3 (osd.3) 297 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:05.675507+0000 osd.3 (osd.3) 298 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:05.826966+0000 mgr.y (mgr.24437) 717 : cluster [DBG] pgmap v1288: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:06.372372+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.c scrub starts 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:06.373478+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.c scrub ok 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:06.464267+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.87:0/2124464942' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-23T00:45:06.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:06.464549+0000 mon.a (mon.0) 3993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-23T00:45:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:05.672532+0000 osd.3 (osd.3) 297 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:45:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:05.675507+0000 osd.3 (osd.3) 298 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:45:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:05.826966+0000 mgr.y (mgr.24437) 717 : cluster [DBG] pgmap v1288: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-23T00:45:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:06.372372+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.c scrub starts 2024-02-23T00:45:06.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:06.373478+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.c scrub ok 2024-02-23T00:45:06.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:06.464267+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.87:0/2124464942' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-23T00:45:06.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:06.464549+0000 mon.a (mon.0) 3993 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-23T00:45:07.508 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-23T00:45:07.784 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:06.557346+0000 mon.a (mon.0) 3994 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-23T00:45:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:06.557409+0000 mon.a (mon.0) 3995 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-23T00:45:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:06.790811+0000 osd.7 (osd.7) 406 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:06.791014+0000 osd.7 (osd.7) 407 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.414414+0000 osd.2 (osd.2) 315 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:45:07.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.416413+0000 osd.2 (osd.2) 316 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:45:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:06.557346+0000 mon.a (mon.0) 3994 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-23T00:45:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:06.557409+0000 mon.a (mon.0) 3995 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-23T00:45:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:06.790811+0000 osd.7 (osd.7) 406 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:06.791014+0000 osd.7 (osd.7) 407 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.414414+0000 osd.2 (osd.2) 315 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:45:07.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.416413+0000 osd.2 (osd.2) 316 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:45:07.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:06.557346+0000 mon.a (mon.0) 3994 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-23T00:45:07.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:06.557409+0000 mon.a (mon.0) 3995 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-23T00:45:07.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:06.790811+0000 osd.7 (osd.7) 406 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:07.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:06.791014+0000 osd.7 (osd.7) 407 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:07.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.414414+0000 osd.2 (osd.2) 315 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:45:07.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.416413+0000 osd.2 (osd.2) 316 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:45:08.811 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.343862+0000 osd.7 (osd.7) 408 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:45:08.811 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.347486+0000 osd.7 (osd.7) 409 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:45:08.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.574706+0000 mon.a (mon.0) 3996 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-23T00:45:08.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.827323+0000 mgr.y (mgr.24437) 718 : cluster [DBG] pgmap v1291: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:08.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:08.094099+0000 osd.5 (osd.5) 349 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:45:08.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:08.095153+0000 osd.5 (osd.5) 350 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:45:08.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:08.415139+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:45:08.812 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:08.416075+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:45:08.812 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.343862+0000 osd.7 (osd.7) 408 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:45:08.812 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.347486+0000 osd.7 (osd.7) 409 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:45:08.812 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.574706+0000 mon.a (mon.0) 3996 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-23T00:45:08.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.827323+0000 mgr.y (mgr.24437) 718 : cluster [DBG] pgmap v1291: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:08.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:08.094099+0000 osd.5 (osd.5) 349 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:45:08.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:08.095153+0000 osd.5 (osd.5) 350 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:45:08.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:08.415139+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:45:08.813 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:08.416075+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:45:08.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.343862+0000 osd.7 (osd.7) 408 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:45:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.347486+0000 osd.7 (osd.7) 409 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:45:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.574706+0000 mon.a (mon.0) 3996 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-23T00:45:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.827323+0000 mgr.y (mgr.24437) 718 : cluster [DBG] pgmap v1291: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:08.094099+0000 osd.5 (osd.5) 349 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:45:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:08.095153+0000 osd.5 (osd.5) 350 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:45:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:08.415139+0000 osd.2 (osd.2) 317 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:45:08.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:08.416075+0000 osd.2 (osd.2) 318 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:45:09.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 -- 172.21.15.87:0/906733144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fe510 msgr2=0x7f9b980f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 --2- 172.21.15.87:0/906733144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fe510 0x7f9b980f7d40 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 rx=0x7f9b84004870 tx=0x7f9b84009cb0).stop 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 -- 172.21.15.87:0/906733144 shutdown_connections 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 --2- 172.21.15.87:0/906733144 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b980f8420 0x7f9b98104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 --2- 172.21.15.87:0/906733144 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9b980fdbb0 0x7f9b980fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 --2- 172.21.15.87:0/906733144 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fe510 0x7f9b980f7d40 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 -- 172.21.15.87:0/906733144 >> 172.21.15.87:0/906733144 conn(0x7f9b980f3800 msgr2=0x7f9b980f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 -- 172.21.15.87:0/906733144 shutdown_connections 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 -- 172.21.15.87:0/906733144 wait complete. 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 librados: starting msgr at 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 librados: starting objecter 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.658+0000 7f9b9edbb700 1 Processor -- start 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9edbb700 1 -- start start 2024-02-23T00:45:09.666 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9edbb700 1 librados: setting wanted keys 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9edbb700 1 librados: calling monclient init 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9edbb700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b980f8420 0x7f9b9810e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9edbb700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fdbb0 0x7f9b9810e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9edbb700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9b980fe510 0x7f9b98108240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9edbb700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b98108730 con 0x7f9b980fdbb0 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9edbb700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b981088a0 con 0x7f9b980f8420 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9edbb700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b98108a60 con 0x7f9b980fe510 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b97fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fdbb0 0x7f9b9810e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b97fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fdbb0 0x7f9b9810e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:58586/0 (socket says 172.21.15.87:58586) 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b97fff700 1 -- 172.21.15.87:0/134787121 learned_addr learned my addr 172.21.15.87:0/134787121 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9d358700 1 --2- 172.21.15.87:0/134787121 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9b980fe510 0x7f9b98108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b9cb57700 1 --2- 172.21.15.87:0/134787121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b980f8420 0x7f9b9810e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:09.667 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b97fff700 1 -- 172.21.15.87:0/134787121 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9b980fe510 msgr2=0x7f9b98108240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:09.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b97fff700 1 --2- 172.21.15.87:0/134787121 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9b980fe510 0x7f9b98108240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:09.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b97fff700 1 -- 172.21.15.87:0/134787121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b980f8420 msgr2=0x7f9b9810e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:09.668 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b97fff700 1 --2- 172.21.15.87:0/134787121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b980f8420 0x7f9b9810e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:09.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b97fff700 1 -- 172.21.15.87:0/134787121 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c005040 con 0x7f9b980fdbb0 2024-02-23T00:45:09.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b97fff700 1 --2- 172.21.15.87:0/134787121 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fdbb0 0x7f9b9810e590 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7f9b84009eb0 tx=0x7f9b8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:09.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b95ffb700 10 client.?.objecter ms_handle_connect 0x7f9b980fdbb0 2024-02-23T00:45:09.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b95ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:09.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.659+0000 7f9b95ffb700 1 -- 172.21.15.87:0/134787121 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8400e070 con 0x7f9b980fdbb0 2024-02-23T00:45:09.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.660+0000 7f9b9edbb700 1 -- 172.21.15.87:0/134787121 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b84005040 con 0x7f9b980fdbb0 2024-02-23T00:45:09.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.660+0000 7f9b95ffb700 1 -- 172.21.15.87:0/134787121 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b84013da0 con 0x7f9b980fdbb0 2024-02-23T00:45:09.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.660+0000 7f9b95ffb700 1 -- 172.21.15.87:0/134787121 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84019070 con 0x7f9b980fdbb0 2024-02-23T00:45:09.671 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.660+0000 7f9b9edbb700 10 client.28032.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.660+0000 7f9b9edbb700 1 -- 172.21.15.87:0/134787121 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b98108fd0 con 0x7f9b980fdbb0 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.660+0000 7f9b9edbb700 1 librados: init done 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.661+0000 7f9b837fe700 1 -- 172.21.15.87:0/134787121 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b78004f20 con 0x7f9b980fdbb0 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.661+0000 7f9b95ffb700 1 -- 172.21.15.87:0/134787121 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f9b8401e420 con 0x7f9b980fdbb0 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.661+0000 7f9b95ffb700 1 --2- 172.21.15.87:0/134787121 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9b88062c90 0x7f9b88065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.662+0000 7f9b95ffb700 1 -- 172.21.15.87:0/134787121 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(832..832 src has 400..832) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f9b8407dab0 con 0x7f9b980fdbb0 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.662+0000 7f9b95ffb700 10 client.28032.objecter ms_dispatch 0x7f9b98001550 osd_map(832..832 src has 400..832) v4 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.662+0000 7f9b95ffb700 3 client.28032.objecter handle_osd_map got epochs [832,832] > 0 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.662+0000 7f9b95ffb700 3 client.28032.objecter handle_osd_map decoding full epoch 832 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.662+0000 7f9b9cb57700 1 --2- 172.21.15.87:0/134787121 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9b88062c90 0x7f9b88065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:09.672 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.662+0000 7f9b95ffb700 20 client.28032.objecter dump_active .. 0 homeless 2024-02-23T00:45:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.662+0000 7f9b95ffb700 10 client.28032.objecter ms_handle_connect 0x7f9b88062c90 2024-02-23T00:45:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.662+0000 7f9b9cb57700 1 --2- 172.21.15.87:0/134787121 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9b88062c90 0x7f9b88065150 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f9b8c0092f0 tx=0x7f9b8c009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:09.673 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.666+0000 7f9b95ffb700 1 -- 172.21.15.87:0/134787121 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f9b84051200 con 0x7f9b980fdbb0 2024-02-23T00:45:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.823954+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:45:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.827407+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:45:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.837994+0000 osd.6 (osd.6) 368 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:07.838215+0000 osd.6 (osd.6) 369 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:08.529754+0000 osd.1 (osd.1) 412 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:45:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:08.532772+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:45:09.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:08.581263+0000 mon.a (mon.0) 3997 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-23T00:45:09.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.823954+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:45:09.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.827407+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:45:09.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.837994+0000 osd.6 (osd.6) 368 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:09.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:07.838215+0000 osd.6 (osd.6) 369 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:09.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:08.529754+0000 osd.1 (osd.1) 412 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:45:09.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:08.532772+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:45:09.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:08.581263+0000 mon.a (mon.0) 3997 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-23T00:45:09.848 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:09.845+0000 7f9b837fe700 1 -- 172.21.15.87:0/134787121 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f9b78005750 con 0x7f9b980fdbb0 2024-02-23T00:45:09.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.823954+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.16 scrub starts 2024-02-23T00:45:09.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.827407+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.16 scrub ok 2024-02-23T00:45:09.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.837994+0000 osd.6 (osd.6) 368 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:09.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:07.838215+0000 osd.6 (osd.6) 369 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:09.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:08.529754+0000 osd.1 (osd.1) 412 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:45:09.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:08.532772+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:45:09.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:08.581263+0000 mon.a (mon.0) 3997 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-23T00:45:10.594 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.591+0000 7f9b95ffb700 1 -- 172.21.15.87:0/134787121 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v833) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9b84050df0 con 0x7f9b980fdbb0 2024-02-23T00:45:10.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.594+0000 7f9b837fe700 10 librados: watch_flush enter 2024-02-23T00:45:10.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.594+0000 7f9b837fe700 10 librados: watch_flush exit 2024-02-23T00:45:10.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.594+0000 7f9b837fe700 20 client.28032.objecter shutdown clearing up homeless session... 2024-02-23T00:45:10.597 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.594+0000 7f9b837fe700 10 client.28032.objecter successfully canceled tick 2024-02-23T00:45:10.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.595+0000 7f9b837fe700 1 -- 172.21.15.87:0/134787121 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9b88062c90 msgr2=0x7f9b88065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:10.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.595+0000 7f9b837fe700 1 --2- 172.21.15.87:0/134787121 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9b88062c90 0x7f9b88065150 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 rx=0x7f9b8c0092f0 tx=0x7f9b8c009cb0).stop 2024-02-23T00:45:10.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.595+0000 7f9b837fe700 1 -- 172.21.15.87:0/134787121 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fdbb0 msgr2=0x7f9b9810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:10.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.595+0000 7f9b837fe700 1 --2- 172.21.15.87:0/134787121 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fdbb0 0x7f9b9810e590 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 rx=0x7f9b84009eb0 tx=0x7f9b8400b040).stop 2024-02-23T00:45:10.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.597+0000 7f9b837fe700 1 -- 172.21.15.87:0/134787121 shutdown_connections 2024-02-23T00:45:10.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.597+0000 7f9b837fe700 1 --2- 172.21.15.87:0/134787121 >> v2:172.21.15.87:6800/1231597463 conn(0x7f9b88062c90 0x7f9b88065150 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:10.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.597+0000 7f9b837fe700 1 --2- 172.21.15.87:0/134787121 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f9b980fe510 0x7f9b98108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:10.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.597+0000 7f9b837fe700 1 --2- 172.21.15.87:0/134787121 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f9b980f8420 0x7f9b9810e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:10.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.597+0000 7f9b837fe700 1 --2- 172.21.15.87:0/134787121 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f9b980fdbb0 0x7f9b9810e590 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:10.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.597+0000 7f9b837fe700 1 -- 172.21.15.87:0/134787121 >> 172.21.15.87:0/134787121 conn(0x7f9b980f3800 msgr2=0x7f9b980fb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:10.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.597+0000 7f9b837fe700 1 -- 172.21.15.87:0/134787121 shutdown_connections 2024-02-23T00:45:10.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.597+0000 7f9b837fe700 1 -- 172.21.15.87:0/134787121 wait complete. 2024-02-23T00:45:10.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:10.597+0000 7f9b837fe700 1 librados: shutdown 2024-02-23T00:45:10.600 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-23T00:45:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.191187+0000 osd.6 (osd.6) 370 : cluster [DBG] 4.e scrub starts 2024-02-23T00:45:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.194794+0000 osd.6 (osd.6) 371 : cluster [DBG] 4.e scrub ok 2024-02-23T00:45:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.278519+0000 osd.3 (osd.3) 299 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.278753+0000 osd.3 (osd.3) 300 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.280869+0000 osd.4 (osd.4) 371 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.281227+0000 osd.4 (osd.4) 372 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:10.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.486466+0000 osd.1 (osd.1) 414 : cluster [DBG] 4.f scrub starts 2024-02-23T00:45:10.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.490215+0000 osd.1 (osd.1) 415 : cluster [DBG] 4.f scrub ok 2024-02-23T00:45:10.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.586331+0000 mon.a (mon.0) 3998 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-23T00:45:10.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:09.827686+0000 mgr.y (mgr.24437) 719 : cluster [DBG] pgmap v1294: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:10.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:10 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:09.847381+0000 mon.a (mon.0) 3999 : audit [INF] from='client.? 172.21.15.87:0/134787121' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-23T00:45:11.036 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.191187+0000 osd.6 (osd.6) 370 : cluster [DBG] 4.e scrub starts 2024-02-23T00:45:11.036 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.194794+0000 osd.6 (osd.6) 371 : cluster [DBG] 4.e scrub ok 2024-02-23T00:45:11.036 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.278519+0000 osd.3 (osd.3) 299 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:11.036 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.278753+0000 osd.3 (osd.3) 300 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:11.036 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.280869+0000 osd.4 (osd.4) 371 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:11.036 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.281227+0000 osd.4 (osd.4) 372 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.486466+0000 osd.1 (osd.1) 414 : cluster [DBG] 4.f scrub starts 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.490215+0000 osd.1 (osd.1) 415 : cluster [DBG] 4.f scrub ok 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.586331+0000 mon.a (mon.0) 3998 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:09.827686+0000 mgr.y (mgr.24437) 719 : cluster [DBG] pgmap v1294: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:09.847381+0000 mon.a (mon.0) 3999 : audit [INF] from='client.? 172.21.15.87:0/134787121' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.191187+0000 osd.6 (osd.6) 370 : cluster [DBG] 4.e scrub starts 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.194794+0000 osd.6 (osd.6) 371 : cluster [DBG] 4.e scrub ok 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.278519+0000 osd.3 (osd.3) 299 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.278753+0000 osd.3 (osd.3) 300 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.280869+0000 osd.4 (osd.4) 371 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.281227+0000 osd.4 (osd.4) 372 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:11.037 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.486466+0000 osd.1 (osd.1) 414 : cluster [DBG] 4.f scrub starts 2024-02-23T00:45:11.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.490215+0000 osd.1 (osd.1) 415 : cluster [DBG] 4.f scrub ok 2024-02-23T00:45:11.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.586331+0000 mon.a (mon.0) 3998 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-23T00:45:11.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:09.827686+0000 mgr.y (mgr.24437) 719 : cluster [DBG] pgmap v1294: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:11.038 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:09.847381+0000 mon.a (mon.0) 3999 : audit [INF] from='client.? 172.21.15.87:0/134787121' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-23T00:45:11.474 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight 5 1 2024-02-23T00:45:11.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:45:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:45:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:10.229615+0000 osd.6 (osd.6) 372 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:45:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:10.232525+0000 osd.6 (osd.6) 373 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:45:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:10.523052+0000 osd.1 (osd.1) 416 : cluster [DBG] 4.2 deep-scrub starts 2024-02-23T00:45:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:10.528453+0000 osd.1 (osd.1) 417 : cluster [DBG] 4.2 deep-scrub ok 2024-02-23T00:45:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:10.592327+0000 mon.a (mon.0) 4000 : audit [INF] from='client.? 172.21.15.87:0/134787121' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-23T00:45:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:10.592428+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-23T00:45:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:11.391210+0000 mgr.y (mgr.24437) 720 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:11.589603+0000 mon.a (mon.0) 4002 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-23T00:45:11.755 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:11.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:10.229615+0000 osd.6 (osd.6) 372 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:45:11.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:10.232525+0000 osd.6 (osd.6) 373 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:45:11.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:10.523052+0000 osd.1 (osd.1) 416 : cluster [DBG] 4.2 deep-scrub starts 2024-02-23T00:45:11.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:10.528453+0000 osd.1 (osd.1) 417 : cluster [DBG] 4.2 deep-scrub ok 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:10.592327+0000 mon.a (mon.0) 4000 : audit [INF] from='client.? 172.21.15.87:0/134787121' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:10.592428+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:11.391210+0000 mgr.y (mgr.24437) 720 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:11.589603+0000 mon.a (mon.0) 4002 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:10.229615+0000 osd.6 (osd.6) 372 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:10.232525+0000 osd.6 (osd.6) 373 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:10.523052+0000 osd.1 (osd.1) 416 : cluster [DBG] 4.2 deep-scrub starts 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:10.528453+0000 osd.1 (osd.1) 417 : cluster [DBG] 4.2 deep-scrub ok 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:10.592327+0000 mon.a (mon.0) 4000 : audit [INF] from='client.? 172.21.15.87:0/134787121' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:10.592428+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-23T00:45:11.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:11.391210+0000 mgr.y (mgr.24437) 720 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:11.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:11.589603+0000 mon.a (mon.0) 4002 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-23T00:45:12.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:45:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:12] "GET /metrics HTTP/1.1" 200 186063 "" "Prometheus/2.33.4" 2024-02-23T00:45:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:11.558972+0000 osd.1 (osd.1) 418 : cluster [DBG] 5.c scrub starts 2024-02-23T00:45:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:11.562049+0000 osd.1 (osd.1) 419 : cluster [DBG] 5.c scrub ok 2024-02-23T00:45:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:11.828030+0000 mgr.y (mgr.24437) 721 : cluster [DBG] pgmap v1297: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:12.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:12.281080+0000 mon.a (mon.0) 4003 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-23T00:45:13.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:11.558972+0000 osd.1 (osd.1) 418 : cluster [DBG] 5.c scrub starts 2024-02-23T00:45:13.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:11.562049+0000 osd.1 (osd.1) 419 : cluster [DBG] 5.c scrub ok 2024-02-23T00:45:13.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:11.828030+0000 mgr.y (mgr.24437) 721 : cluster [DBG] pgmap v1297: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:13.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:12.281080+0000 mon.a (mon.0) 4003 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-23T00:45:13.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:11.558972+0000 osd.1 (osd.1) 418 : cluster [DBG] 5.c scrub starts 2024-02-23T00:45:13.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:11.562049+0000 osd.1 (osd.1) 419 : cluster [DBG] 5.c scrub ok 2024-02-23T00:45:13.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:11.828030+0000 mgr.y (mgr.24437) 721 : cluster [DBG] pgmap v1297: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:13.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:12.281080+0000 mon.a (mon.0) 4003 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-23T00:45:13.714 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.671+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/1362191274 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6b40fe510 msgr2=0x7fd6b40f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.671+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/1362191274 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6b40fe510 0x7fd6b40f7d40 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 rx=0x7fd6b0004870 tx=0x7fd6b0009cb0).stop 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.671+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/1362191274 shutdown_connections 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.671+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/1362191274 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6b40f8420 0x7fd6b4104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.671+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/1362191274 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd6b40fdbb0 0x7fd6b40fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.671+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/1362191274 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6b40fe510 0x7fd6b40f7d40 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.671+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/1362191274 >> 172.21.15.87:0/1362191274 conn(0x7fd6b40f3800 msgr2=0x7fd6b40f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.671+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/1362191274 shutdown_connections 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.671+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/1362191274 wait complete. 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 librados: starting msgr at 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 librados: starting objecter 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 Processor -- start 2024-02-23T00:45:13.715 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 -- start start 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 librados: setting wanted keys 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 librados: calling monclient init 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6b40f8420 0x7fd6b406b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6b40fdbb0 0x7fd6b406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd6b40fe510 0x7fd6b406bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4116530 con 0x7fd6b40f8420 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b41166a0 con 0x7fd6b40fdbb0 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6bb9ed700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4116860 con 0x7fd6b40fe510 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9f8a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd6b40fe510 0x7fd6b406bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9f8a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd6b40fe510 0x7fd6b406bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:60072/0 (socket says 172.21.15.87:60072) 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9f8a700 1 -- 172.21.15.87:0/2321511684 learned_addr learned my addr 172.21.15.87:0/2321511684 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:13.716 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b8f88700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6b40fdbb0 0x7fd6b406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9789700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6b40f8420 0x7fd6b406b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9f8a700 1 -- 172.21.15.87:0/2321511684 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6b40fdbb0 msgr2=0x7fd6b406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9f8a700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6b40fdbb0 0x7fd6b406b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9f8a700 1 -- 172.21.15.87:0/2321511684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6b40f8420 msgr2=0x7fd6b406b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9f8a700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6b40f8420 0x7fd6b406b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9f8a700 1 -- 172.21.15.87:0/2321511684 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a4005040 con 0x7fd6b40fe510 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9789700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6b40f8420 0x7fd6b406b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6aa7fc700 10 client.?.objecter ms_handle_connect 0x7fd6b40fe510 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6aa7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6b9f8a700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd6b40fe510 0x7fd6b406bab0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7fd6ac00d970 tx=0x7fd6ac004e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.672+0000 7fd6aa7fc700 1 -- 172.21.15.87:0/2321511684 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac00de60 con 0x7fd6b40fe510 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.673+0000 7fd6aa7fc700 1 -- 172.21.15.87:0/2321511684 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6ac00f040 con 0x7fd6b40fe510 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.673+0000 7fd6aa7fc700 1 -- 172.21.15.87:0/2321511684 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0036c0 con 0x7fd6b40fe510 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.673+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b0005040 con 0x7fd6b40fe510 2024-02-23T00:45:13.717 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.673+0000 7fd6bb9ed700 10 client.38513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.673+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b4116ed0 con 0x7fd6b40fe510 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.673+0000 7fd6bb9ed700 1 librados: init done 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.673+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b4107060 con 0x7fd6b40fe510 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.674+0000 7fd6aa7fc700 1 -- 172.21.15.87:0/2321511684 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd6ac003ab0 con 0x7fd6b40fe510 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.674+0000 7fd6aa7fc700 1 --2- 172.21.15.87:0/2321511684 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd6a0062c90 0x7fd6a0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.674+0000 7fd6aa7fc700 1 -- 172.21.15.87:0/2321511684 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(835..835 src has 400..835) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fd6ac07caf0 con 0x7fd6b40fe510 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.674+0000 7fd6aa7fc700 10 client.38513.objecter ms_dispatch 0x7fd6b4001550 osd_map(835..835 src has 400..835) v4 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.674+0000 7fd6aa7fc700 3 client.38513.objecter handle_osd_map got epochs [835,835] > 0 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.674+0000 7fd6aa7fc700 3 client.38513.objecter handle_osd_map decoding full epoch 835 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.674+0000 7fd6b9789700 1 --2- 172.21.15.87:0/2321511684 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd6a0062c90 0x7fd6a0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.674+0000 7fd6aa7fc700 20 client.38513.objecter dump_active .. 0 homeless 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.675+0000 7fd6aa7fc700 10 client.38513.objecter ms_handle_connect 0x7fd6a0062c90 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.675+0000 7fd6b9789700 1 --2- 172.21.15.87:0/2321511684 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd6a0062c90 0x7fd6a0065150 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 rx=0x7fd6a4007fd0 tx=0x7fd6a4009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:13.718 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.679+0000 7fd6aa7fc700 1 -- 172.21.15.87:0/2321511684 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd6ac048f60 con 0x7fd6b40fe510 2024-02-23T00:45:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:12.427666+0000 osd.7 (osd.7) 410 : cluster [DBG] 2.b scrub starts 2024-02-23T00:45:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:12.428780+0000 osd.7 (osd.7) 411 : cluster [DBG] 2.b scrub ok 2024-02-23T00:45:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:12.637591+0000 mon.a (mon.0) 4004 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:45:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:12.775924+0000 osd.3 (osd.3) 301 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:45:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:12.776738+0000 osd.3 (osd.3) 302 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:45:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:13.261389+0000 mon.a (mon.0) 4005 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:45:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:13.262026+0000 mon.a (mon.0) 4006 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:45:13.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:13.266113+0000 mon.a (mon.0) 4007 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:45:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:12.427666+0000 osd.7 (osd.7) 410 : cluster [DBG] 2.b scrub starts 2024-02-23T00:45:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:12.428780+0000 osd.7 (osd.7) 411 : cluster [DBG] 2.b scrub ok 2024-02-23T00:45:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:12.637591+0000 mon.a (mon.0) 4004 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:45:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:12.775924+0000 osd.3 (osd.3) 301 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:45:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:12.776738+0000 osd.3 (osd.3) 302 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:45:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:13.261389+0000 mon.a (mon.0) 4005 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:45:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:13.262026+0000 mon.a (mon.0) 4006 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:45:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:13.266113+0000 mon.a (mon.0) 4007 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:45:13.860 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:13.858+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7fd6b405fa60 con 0x7fd6b40fe510 2024-02-23T00:45:13.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:45:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:45:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:12.427666+0000 osd.7 (osd.7) 410 : cluster [DBG] 2.b scrub starts 2024-02-23T00:45:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:12.428780+0000 osd.7 (osd.7) 411 : cluster [DBG] 2.b scrub ok 2024-02-23T00:45:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:12.637591+0000 mon.a (mon.0) 4004 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:45:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:12.775924+0000 osd.3 (osd.3) 301 : cluster [DBG] 5.17 scrub starts 2024-02-23T00:45:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:12.776738+0000 osd.3 (osd.3) 302 : cluster [DBG] 5.17 scrub ok 2024-02-23T00:45:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:13.261389+0000 mon.a (mon.0) 4005 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:45:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:13.262026+0000 mon.a (mon.0) 4006 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:45:13.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:13.266113+0000 mon.a (mon.0) 4007 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:45:14.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.270+0000 7fd6aa7fc700 1 -- 172.21.15.87:0/2321511684 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v836) v1 ==== 113+0+0 (secure 0 0 0) 0x7fd6ac050240 con 0x7fd6b40fe510 2024-02-23T00:45:14.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 10 librados: watch_flush enter 2024-02-23T00:45:14.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 10 librados: watch_flush exit 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 20 client.38513.objecter shutdown clearing up homeless session... 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 10 client.38513.objecter successfully canceled tick 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd6a0062c90 msgr2=0x7fd6a0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/2321511684 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd6a0062c90 0x7fd6a0065150 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 rx=0x7fd6a4007fd0 tx=0x7fd6a4009cb0).stop 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd6b40fe510 msgr2=0x7fd6b406bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd6b40fe510 0x7fd6b406bab0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 rx=0x7fd6ac00d970 tx=0x7fd6ac004e60).stop 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 shutdown_connections 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/2321511684 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd6a0062c90 0x7fd6a0065150 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd6b40fe510 0x7fd6b406bab0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd6b40f8420 0x7fd6b406b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 --2- 172.21.15.87:0/2321511684 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd6b40fdbb0 0x7fd6b406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:14.276 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 >> 172.21.15.87:0/2321511684 conn(0x7fd6b40f3800 msgr2=0x7fd6b40fb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:14.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 shutdown_connections 2024-02-23T00:45:14.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 -- 172.21.15.87:0/2321511684 wait complete. 2024-02-23T00:45:14.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:14.273+0000 7fd6bb9ed700 1 librados: shutdown 2024-02-23T00:45:14.277 INFO:teuthology.orchestra.run.smithi087.stderr:reweighted osd.5 to 1 (10000) 2024-02-23T00:45:14.906 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:13.390082+0000 osd.7 (osd.7) 412 : cluster [DBG] 22.d deep-scrub starts 2024-02-23T00:45:14.906 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:13.393083+0000 osd.7 (osd.7) 413 : cluster [DBG] 22.d deep-scrub ok 2024-02-23T00:45:14.906 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:13.589696+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:14.906 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:13.589842+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:14.906 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:13.828464+0000 mgr.y (mgr.24437) 722 : cluster [DBG] pgmap v1299: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:14.906 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:13.860106+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.87:0/2321511684' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:14.906 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:13.860411+0000 mon.a (mon.0) 4008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:14.906 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:14.271413+0000 mon.a (mon.0) 4009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-23T00:45:14.906 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:14.271498+0000 mon.a (mon.0) 4010 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-23T00:45:14.907 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:13.390082+0000 osd.7 (osd.7) 412 : cluster [DBG] 22.d deep-scrub starts 2024-02-23T00:45:14.907 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:13.393083+0000 osd.7 (osd.7) 413 : cluster [DBG] 22.d deep-scrub ok 2024-02-23T00:45:14.907 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:13.589696+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:14.907 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:13.589842+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:14.907 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:13.828464+0000 mgr.y (mgr.24437) 722 : cluster [DBG] pgmap v1299: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:14.907 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:13.860106+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.87:0/2321511684' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:14.907 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:13.860411+0000 mon.a (mon.0) 4008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:14.907 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:14.271413+0000 mon.a (mon.0) 4009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-23T00:45:14.907 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:14.271498+0000 mon.a (mon.0) 4010 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-23T00:45:14.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:13.390082+0000 osd.7 (osd.7) 412 : cluster [DBG] 22.d deep-scrub starts 2024-02-23T00:45:14.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:13.393083+0000 osd.7 (osd.7) 413 : cluster [DBG] 22.d deep-scrub ok 2024-02-23T00:45:14.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:13.589696+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:14.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:13.589842+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:14.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:13.828464+0000 mgr.y (mgr.24437) 722 : cluster [DBG] pgmap v1299: 164 pgs: 164 active+clean; 5.6 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:14.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:13.860106+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.87:0/2321511684' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:14.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:13.860411+0000 mon.a (mon.0) 4008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:14.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:14.271413+0000 mon.a (mon.0) 4009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-23T00:45:14.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:14.271498+0000 mon.a (mon.0) 4010 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-23T00:45:15.372 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-23T00:45:15.647 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:14.393267+0000 osd.7 (osd.7) 414 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:45:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:14.395603+0000 osd.7 (osd.7) 415 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:45:15.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:14.544345+0000 osd.1 (osd.1) 420 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:45:15.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:14.546392+0000 osd.1 (osd.1) 421 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:45:15.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:14.736325+0000 osd.3 (osd.3) 303 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:45:15.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:14.740132+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:45:15.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:15.113614+0000 osd.5 (osd.5) 351 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:45:15.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:15.114552+0000 osd.5 (osd.5) 352 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:45:15.954 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:14.393267+0000 osd.7 (osd.7) 414 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:45:15.954 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:14.395603+0000 osd.7 (osd.7) 415 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:45:15.954 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:14.544345+0000 osd.1 (osd.1) 420 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:45:15.954 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:14.546392+0000 osd.1 (osd.1) 421 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:45:15.954 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:14.736325+0000 osd.3 (osd.3) 303 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:45:15.954 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:14.740132+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:45:15.954 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:15.113614+0000 osd.5 (osd.5) 351 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:45:15.954 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:15.114552+0000 osd.5 (osd.5) 352 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:45:15.955 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:14.393267+0000 osd.7 (osd.7) 414 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:45:15.955 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:14.395603+0000 osd.7 (osd.7) 415 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:45:15.955 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:14.544345+0000 osd.1 (osd.1) 420 : cluster [DBG] 3.0 scrub starts 2024-02-23T00:45:15.955 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:14.546392+0000 osd.1 (osd.1) 421 : cluster [DBG] 3.0 scrub ok 2024-02-23T00:45:15.955 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:14.736325+0000 osd.3 (osd.3) 303 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:45:15.955 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:14.740132+0000 osd.3 (osd.3) 304 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:45:15.955 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:15.113614+0000 osd.5 (osd.5) 351 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:45:15.955 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:15.114552+0000 osd.5 (osd.5) 352 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:45:16.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:15.829120+0000 mgr.y (mgr.24437) 723 : cluster [DBG] pgmap v1301: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:16.956 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:15.829120+0000 mgr.y (mgr.24437) 723 : cluster [DBG] pgmap v1301: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:16.957 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:15.829120+0000 mgr.y (mgr.24437) 723 : cluster [DBG] pgmap v1301: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:17.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/661350583 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0c24104600 msgr2=0x7f0c241069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:17.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/661350583 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0c24104600 0x7f0c241069f0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 rx=0x7f0c1c009c30 tx=0x7f0c1c004c50).stop 2024-02-23T00:45:17.513 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/661350583 shutdown_connections 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/661350583 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0c24104600 0x7f0c241069f0 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/661350583 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0c240f6aa0 0x7f0c240f6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/661350583 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c240f7370 0x7f0c24104110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/661350583 >> 172.21.15.87:0/661350583 conn(0x7f0c240f2820 msgr2=0x7f0c240f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/661350583 shutdown_connections 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/661350583 wait complete. 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 librados: starting msgr at 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.503+0000 7f0c2a1d9700 1 librados: starting objecter 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 Processor -- start 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 -- start start 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 librados: setting wanted keys 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 librados: calling monclient init 2024-02-23T00:45:17.514 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0c240f6aa0 0x7f0c241101f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0c240f7370 0x7f0c241106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c24104600 0x7f0c2410a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2410a8d0 con 0x7f0c24104600 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2410aa40 con 0x7f0c240f6aa0 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c2a1d9700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2410ac00 con 0x7f0c240f7370 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c23fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c24104600 0x7f0c2410a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c22ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0c240f7370 0x7f0c241106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c23fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c24104600 0x7f0c2410a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51224/0 (socket says 172.21.15.87:51224) 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c23fff700 1 -- 172.21.15.87:0/182046703 learned_addr learned my addr 172.21.15.87:0/182046703 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c237fe700 1 --2- 172.21.15.87:0/182046703 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0c240f6aa0 0x7f0c241101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c22ffd700 1 -- 172.21.15.87:0/182046703 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0c240f6aa0 msgr2=0x7f0c241101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c22ffd700 1 --2- 172.21.15.87:0/182046703 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0c240f6aa0 0x7f0c241101f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c22ffd700 1 -- 172.21.15.87:0/182046703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c24104600 msgr2=0x7f0c2410a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c22ffd700 1 --2- 172.21.15.87:0/182046703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c24104600 0x7f0c2410a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c22ffd700 1 -- 172.21.15.87:0/182046703 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c14005040 con 0x7f0c240f7370 2024-02-23T00:45:17.515 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.504+0000 7f0c23fff700 1 --2- 172.21.15.87:0/182046703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c24104600 0x7f0c2410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c22ffd700 1 --2- 172.21.15.87:0/182046703 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0c240f7370 0x7f0c241106e0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7f0c140091f0 tx=0x7f0c14009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c20ff9700 10 client.?.objecter ms_handle_connect 0x7f0c240f7370 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c20ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c20ff9700 1 -- 172.21.15.87:0/182046703 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1400e030 con 0x7f0c240f7370 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1c0097c0 con 0x7f0c240f7370 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c20ff9700 1 -- 172.21.15.87:0/182046703 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c1400f040 con 0x7f0c240f7370 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c20ff9700 1 -- 172.21.15.87:0/182046703 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c140147e0 con 0x7f0c240f7370 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c2a1d9700 10 client.38519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2410b240 con 0x7f0c240f7370 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.505+0000 7f0c2a1d9700 1 librados: init done 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.506+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2405fa60 con 0x7f0c240f7370 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.506+0000 7f0c20ff9700 1 -- 172.21.15.87:0/182046703 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0c14014b90 con 0x7f0c240f7370 2024-02-23T00:45:17.516 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.507+0000 7f0c20ff9700 1 --2- 172.21.15.87:0/182046703 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0c0c062c90 0x7f0c0c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:17.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.507+0000 7f0c20ff9700 1 -- 172.21.15.87:0/182046703 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(837..837 src has 400..837) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f0c1407d2a0 con 0x7f0c240f7370 2024-02-23T00:45:17.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.507+0000 7f0c237fe700 1 --2- 172.21.15.87:0/182046703 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0c0c062c90 0x7f0c0c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:17.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.507+0000 7f0c20ff9700 10 client.38519.objecter ms_dispatch 0x7f0c24001550 osd_map(837..837 src has 400..837) v4 2024-02-23T00:45:17.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.507+0000 7f0c20ff9700 3 client.38519.objecter handle_osd_map got epochs [837,837] > 0 2024-02-23T00:45:17.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.507+0000 7f0c20ff9700 3 client.38519.objecter handle_osd_map decoding full epoch 837 2024-02-23T00:45:17.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.507+0000 7f0c20ff9700 20 client.38519.objecter dump_active .. 0 homeless 2024-02-23T00:45:17.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.507+0000 7f0c237fe700 1 --2- 172.21.15.87:0/182046703 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0c0c062c90 0x7f0c0c065150 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7f0c18004de0 tx=0x7f0c1800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:17.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.507+0000 7f0c20ff9700 10 client.38519.objecter ms_handle_connect 0x7f0c0c062c90 2024-02-23T00:45:17.517 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.510+0000 7f0c20ff9700 1 -- 172.21.15.87:0/182046703 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0c14050a90 con 0x7f0c240f7370 2024-02-23T00:45:17.689 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:17.687+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f0c2410b870 con 0x7f0c240f7370 2024-02-23T00:45:17.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:16.490022+0000 osd.1 (osd.1) 422 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:45:17.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:16.493738+0000 osd.1 (osd.1) 423 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:45:17.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:16.693660+0000 osd.3 (osd.3) 305 : cluster [DBG] 22.e scrub starts 2024-02-23T00:45:17.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:16.696795+0000 osd.3 (osd.3) 306 : cluster [DBG] 22.e scrub ok 2024-02-23T00:45:17.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:17.287011+0000 mon.a (mon.0) 4011 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-23T00:45:18.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:16.490022+0000 osd.1 (osd.1) 422 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:45:18.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:16.493738+0000 osd.1 (osd.1) 423 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:45:18.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:16.693660+0000 osd.3 (osd.3) 305 : cluster [DBG] 22.e scrub starts 2024-02-23T00:45:18.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:16.696795+0000 osd.3 (osd.3) 306 : cluster [DBG] 22.e scrub ok 2024-02-23T00:45:18.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:17.287011+0000 mon.a (mon.0) 4011 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-23T00:45:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:16.490022+0000 osd.1 (osd.1) 422 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:45:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:16.493738+0000 osd.1 (osd.1) 423 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:45:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:16.693660+0000 osd.3 (osd.3) 305 : cluster [DBG] 22.e scrub starts 2024-02-23T00:45:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:16.696795+0000 osd.3 (osd.3) 306 : cluster [DBG] 22.e scrub ok 2024-02-23T00:45:18.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:17.287011+0000 mon.a (mon.0) 4011 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-23T00:45:18.633 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.630+0000 7f0c20ff9700 1 -- 172.21.15.87:0/182046703 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v838) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0c14050680 con 0x7f0c240f7370 2024-02-23T00:45:18.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.634+0000 7f0c2a1d9700 10 librados: watch_flush enter 2024-02-23T00:45:18.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.634+0000 7f0c2a1d9700 10 librados: watch_flush exit 2024-02-23T00:45:18.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.634+0000 7f0c2a1d9700 20 client.38519.objecter shutdown clearing up homeless session... 2024-02-23T00:45:18.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.634+0000 7f0c2a1d9700 10 client.38519.objecter successfully canceled tick 2024-02-23T00:45:18.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0c0c062c90 msgr2=0x7f0c0c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:18.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/182046703 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0c0c062c90 0x7f0c0c065150 secure :-1 s=READY pgs=3002 cs=0 l=1 rev1=1 rx=0x7f0c18004de0 tx=0x7f0c1800b040).stop 2024-02-23T00:45:18.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0c240f7370 msgr2=0x7f0c241106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:18.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/182046703 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0c240f7370 0x7f0c241106e0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7f0c140091f0 tx=0x7f0c14009cb0).stop 2024-02-23T00:45:18.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 shutdown_connections 2024-02-23T00:45:18.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/182046703 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0c0c062c90 0x7f0c0c065150 unknown :-1 s=CLOSED pgs=3002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/182046703 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0c24104600 0x7f0c2410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/182046703 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0c240f6aa0 0x7f0c241101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 --2- 172.21.15.87:0/182046703 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0c240f7370 0x7f0c241106e0 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 >> 172.21.15.87:0/182046703 conn(0x7f0c240f2820 msgr2=0x7f0c240f4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 shutdown_connections 2024-02-23T00:45:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 -- 172.21.15.87:0/182046703 wait complete. 2024-02-23T00:45:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:18.635+0000 7f0c2a1d9700 1 librados: shutdown 2024-02-23T00:45:18.639 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-23T00:45:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:16.949672+0000 osd.0 (osd.0) 99 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:45:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:16.953393+0000 osd.0 (osd.0) 100 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:45:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:17.688798+0000 mon.c (mon.2) 515 : audit [INF] from='client.? 172.21.15.87:0/182046703' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:17.689084+0000 mon.a (mon.0) 4012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:17.829547+0000 mgr.y (mgr.24437) 724 : cluster [DBG] pgmap v1303: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:18.238293+0000 osd.2 (osd.2) 319 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:45:18.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:18.239711+0000 osd.2 (osd.2) 320 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:45:18.983 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:16.949672+0000 osd.0 (osd.0) 99 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:45:18.984 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:16.953393+0000 osd.0 (osd.0) 100 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:45:18.984 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:17.688798+0000 mon.c (mon.2) 515 : audit [INF] from='client.? 172.21.15.87:0/182046703' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:18.984 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:17.689084+0000 mon.a (mon.0) 4012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:18.984 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:17.829547+0000 mgr.y (mgr.24437) 724 : cluster [DBG] pgmap v1303: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:18.984 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:18.238293+0000 osd.2 (osd.2) 319 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:45:18.984 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:18.239711+0000 osd.2 (osd.2) 320 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:45:18.984 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:16.949672+0000 osd.0 (osd.0) 99 : cluster [DBG] 22.1f scrub starts 2024-02-23T00:45:18.985 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:16.953393+0000 osd.0 (osd.0) 100 : cluster [DBG] 22.1f scrub ok 2024-02-23T00:45:18.985 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:17.688798+0000 mon.c (mon.2) 515 : audit [INF] from='client.? 172.21.15.87:0/182046703' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:18.985 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:17.689084+0000 mon.a (mon.0) 4012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-23T00:45:18.985 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:17.829547+0000 mgr.y (mgr.24437) 724 : cluster [DBG] pgmap v1303: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:18.985 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:18.238293+0000 osd.2 (osd.2) 319 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:45:18.985 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:18.239711+0000 osd.2 (osd.2) 320 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:45:19.397 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight 6 1 2024-02-23T00:45:19.673 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:19.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:18.631653+0000 mon.a (mon.0) 4013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-23T00:45:19.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:18.631746+0000 mon.a (mon.0) 4014 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-23T00:45:19.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:19.042506+0000 osd.5 (osd.5) 353 : cluster [DBG] 2.3 deep-scrub starts 2024-02-23T00:45:19.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:19.043447+0000 osd.5 (osd.5) 354 : cluster [DBG] 2.3 deep-scrub ok 2024-02-23T00:45:20.003 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:18.631653+0000 mon.a (mon.0) 4013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-23T00:45:20.004 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:18.631746+0000 mon.a (mon.0) 4014 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-23T00:45:20.004 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:19.042506+0000 osd.5 (osd.5) 353 : cluster [DBG] 2.3 deep-scrub starts 2024-02-23T00:45:20.004 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:19.043447+0000 osd.5 (osd.5) 354 : cluster [DBG] 2.3 deep-scrub ok 2024-02-23T00:45:20.004 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:18.631653+0000 mon.a (mon.0) 4013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-23T00:45:20.004 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:18.631746+0000 mon.a (mon.0) 4014 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-23T00:45:20.004 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:19.042506+0000 osd.5 (osd.5) 353 : cluster [DBG] 2.3 deep-scrub starts 2024-02-23T00:45:20.004 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:19.043447+0000 osd.5 (osd.5) 354 : cluster [DBG] 2.3 deep-scrub ok 2024-02-23T00:45:20.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:19.274151+0000 osd.6 (osd.6) 374 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:45:20.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:19.277057+0000 osd.6 (osd.6) 375 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:45:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:19.582133+0000 osd.3 (osd.3) 307 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:45:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:19.585063+0000 osd.3 (osd.3) 308 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:45:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:19.637816+0000 mon.a (mon.0) 4015 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-23T00:45:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:19.829887+0000 mgr.y (mgr.24437) 725 : cluster [DBG] pgmap v1306: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:20.259788+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.b deep-scrub starts 2024-02-23T00:45:20.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:20.260855+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.b deep-scrub ok 2024-02-23T00:45:21.065 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:19.274151+0000 osd.6 (osd.6) 374 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:45:21.065 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:19.277057+0000 osd.6 (osd.6) 375 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:45:21.065 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:19.582133+0000 osd.3 (osd.3) 307 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:45:21.065 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:19.585063+0000 osd.3 (osd.3) 308 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:45:21.065 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:19.637816+0000 mon.a (mon.0) 4015 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:19.829887+0000 mgr.y (mgr.24437) 725 : cluster [DBG] pgmap v1306: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:20.259788+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.b deep-scrub starts 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:20.260855+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.b deep-scrub ok 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:19.274151+0000 osd.6 (osd.6) 374 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:19.277057+0000 osd.6 (osd.6) 375 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:19.582133+0000 osd.3 (osd.3) 307 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:19.585063+0000 osd.3 (osd.3) 308 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:19.637816+0000 mon.a (mon.0) 4015 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:19.829887+0000 mgr.y (mgr.24437) 725 : cluster [DBG] pgmap v1306: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:20.259788+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.b deep-scrub starts 2024-02-23T00:45:21.066 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:20.260855+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.b deep-scrub ok 2024-02-23T00:45:21.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 -- 172.21.15.87:0/3003474670 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f057c0f9610 msgr2=0x7f057c101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:21.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 --2- 172.21.15.87:0/3003474670 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f057c0f9610 0x7f057c101d20 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 rx=0x7f05700047b0 tx=0x7f0570009cb0).stop 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 -- 172.21.15.87:0/3003474670 shutdown_connections 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 --2- 172.21.15.87:0/3003474670 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f057c102210 0x7f057c104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 --2- 172.21.15.87:0/3003474670 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c0f8cb0 0x7f057c0f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 --2- 172.21.15.87:0/3003474670 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f057c0f9610 0x7f057c101d20 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 -- 172.21.15.87:0/3003474670 >> 172.21.15.87:0/3003474670 conn(0x7f057c0f2820 msgr2=0x7f057c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 -- 172.21.15.87:0/3003474670 shutdown_connections 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 -- 172.21.15.87:0/3003474670 wait complete. 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 librados: starting msgr at 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 librados: starting objecter 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.621+0000 7f0581505700 1 Processor -- start 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0581505700 1 -- start start 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0581505700 1 librados: setting wanted keys 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0581505700 1 librados: calling monclient init 2024-02-23T00:45:21.626 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0581505700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f057c0f8cb0 0x7f057c10e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0581505700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c0f9610 0x7f057c10e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0581505700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f057c102210 0x7f057c108470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0581505700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c108960 con 0x7f057c0f8cb0 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0581505700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c108ad0 con 0x7f057c0f9610 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0581505700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057c108c40 con 0x7f057c102210 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f057c0f8cb0 0x7f057c10e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057affd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f057c0f8cb0 0x7f057c10e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49616/0 (socket says 172.21.15.87:49616) 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057b7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f057c102210 0x7f057c108470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057affd700 1 -- 172.21.15.87:0/3097224729 learned_addr learned my addr 172.21.15.87:0/3097224729 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057a7fc700 1 --2- 172.21.15.87:0/3097224729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c0f9610 0x7f057c10e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057b7fe700 1 -- 172.21.15.87:0/3097224729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c0f9610 msgr2=0x7f057c10e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057b7fe700 1 --2- 172.21.15.87:0/3097224729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c0f9610 0x7f057c10e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057b7fe700 1 -- 172.21.15.87:0/3097224729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f057c0f8cb0 msgr2=0x7f057c10e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:21.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057b7fe700 1 --2- 172.21.15.87:0/3097224729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f057c0f8cb0 0x7f057c10e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057b7fe700 1 -- 172.21.15.87:0/3097224729 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05740097c0 con 0x7f057c102210 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057affd700 1 --2- 172.21.15.87:0/3097224729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f057c0f8cb0 0x7f057c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f057b7fe700 1 --2- 172.21.15.87:0/3097224729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f057c102210 0x7f057c108470 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 rx=0x7f0574001dc0 tx=0x7f0574004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0563fff700 10 client.?.objecter ms_handle_connect 0x7f057c102210 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.622+0000 7f0563fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.623+0000 7f0563fff700 1 -- 172.21.15.87:0/3097224729 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05740099f0 con 0x7f057c102210 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.623+0000 7f0563fff700 1 -- 172.21.15.87:0/3097224729 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05740090d0 con 0x7f057c102210 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.623+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0570005040 con 0x7f057c102210 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.623+0000 7f0563fff700 1 -- 172.21.15.87:0/3097224729 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574007c30 con 0x7f057c102210 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.623+0000 7f0581505700 10 client.38525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.623+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c109230 con 0x7f057c102210 2024-02-23T00:45:21.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.623+0000 7f0581505700 1 librados: init done 2024-02-23T00:45:21.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.623+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c05fa60 con 0x7f057c102210 2024-02-23T00:45:21.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.624+0000 7f0563fff700 1 -- 172.21.15.87:0/3097224729 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f057400fad0 con 0x7f057c102210 2024-02-23T00:45:21.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.624+0000 7f0563fff700 1 --2- 172.21.15.87:0/3097224729 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0564062c90 0x7f0564065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:21.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.624+0000 7f0563fff700 1 -- 172.21.15.87:0/3097224729 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(839..839 src has 400..839) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f057407d8c0 con 0x7f057c102210 2024-02-23T00:45:21.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.624+0000 7f0563fff700 10 client.38525.objecter ms_dispatch 0x7f057c001550 osd_map(839..839 src has 400..839) v4 2024-02-23T00:45:21.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.624+0000 7f0563fff700 3 client.38525.objecter handle_osd_map got epochs [839,839] > 0 2024-02-23T00:45:21.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.624+0000 7f0563fff700 3 client.38525.objecter handle_osd_map decoding full epoch 839 2024-02-23T00:45:21.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.624+0000 7f057affd700 1 --2- 172.21.15.87:0/3097224729 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0564062c90 0x7f0564065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:21.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.625+0000 7f0563fff700 20 client.38525.objecter dump_active .. 0 homeless 2024-02-23T00:45:21.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.625+0000 7f0563fff700 10 client.38525.objecter ms_handle_connect 0x7f0564062c90 2024-02-23T00:45:21.630 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.625+0000 7f057affd700 1 --2- 172.21.15.87:0/3097224729 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0564062c90 0x7f0564065150 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 rx=0x7f056c007d30 tx=0x7f056c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:21.631 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.629+0000 7f0563fff700 1 -- 172.21.15.87:0/3097224729 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0574050f50 con 0x7f057c102210 2024-02-23T00:45:21.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:45:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:45:21.813 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:21.810+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f057c107290 con 0x7f057c102210 2024-02-23T00:45:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:20.362105+0000 osd.4 (osd.4) 373 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:45:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:20.364009+0000 osd.4 (osd.4) 374 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:45:21.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:20.571352+0000 osd.3 (osd.3) 309 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:45:21.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:20.572993+0000 osd.3 (osd.3) 310 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:45:21.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:21.402331+0000 mgr.y (mgr.24437) 726 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:20.362105+0000 osd.4 (osd.4) 373 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:45:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:20.364009+0000 osd.4 (osd.4) 374 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:45:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:20.571352+0000 osd.3 (osd.3) 309 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:45:22.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:20.572993+0000 osd.3 (osd.3) 310 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:45:22.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:21.402331+0000 mgr.y (mgr.24437) 726 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:22.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:20.362105+0000 osd.4 (osd.4) 373 : cluster [DBG] 4.9 scrub starts 2024-02-23T00:45:22.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:20.364009+0000 osd.4 (osd.4) 374 : cluster [DBG] 4.9 scrub ok 2024-02-23T00:45:22.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:20.571352+0000 osd.3 (osd.3) 309 : cluster [DBG] 4.17 scrub starts 2024-02-23T00:45:22.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:20.572993+0000 osd.3 (osd.3) 310 : cluster [DBG] 4.17 scrub ok 2024-02-23T00:45:22.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:21.402331+0000 mgr.y (mgr.24437) 726 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:22.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:22] "GET /metrics HTTP/1.1" 200 186063 "" "Prometheus/2.33.4" 2024-02-23T00:45:22.660 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.657+0000 7f0563fff700 1 -- 172.21.15.87:0/3097224729 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v840) v1 ==== 113+0+0 (secure 0 0 0) 0x7f057401c300 con 0x7f057c102210 2024-02-23T00:45:22.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.660+0000 7f0581505700 10 librados: watch_flush enter 2024-02-23T00:45:22.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.660+0000 7f0581505700 10 librados: watch_flush exit 2024-02-23T00:45:22.663 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.660+0000 7f0581505700 20 client.38525.objecter shutdown clearing up homeless session... 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.660+0000 7f0581505700 10 client.38525.objecter successfully canceled tick 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0564062c90 msgr2=0x7f0564065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 --2- 172.21.15.87:0/3097224729 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0564062c90 0x7f0564065150 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 rx=0x7f056c007d30 tx=0x7f056c00b040).stop 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f057c102210 msgr2=0x7f057c108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 --2- 172.21.15.87:0/3097224729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f057c102210 0x7f057c108470 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 rx=0x7f0574001dc0 tx=0x7f0574004d10).stop 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 shutdown_connections 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 --2- 172.21.15.87:0/3097224729 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0564062c90 0x7f0564065150 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 --2- 172.21.15.87:0/3097224729 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f057c102210 0x7f057c108470 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 --2- 172.21.15.87:0/3097224729 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f057c0f8cb0 0x7f057c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 --2- 172.21.15.87:0/3097224729 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f057c0f9610 0x7f057c10e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 >> 172.21.15.87:0/3097224729 conn(0x7f057c0f2820 msgr2=0x7f057c0f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 shutdown_connections 2024-02-23T00:45:22.664 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 -- 172.21.15.87:0/3097224729 wait complete. 2024-02-23T00:45:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:22.661+0000 7f0581505700 1 librados: shutdown 2024-02-23T00:45:22.665 INFO:teuthology.orchestra.run.smithi087.stderr:reweighted osd.6 to 1 (10000) 2024-02-23T00:45:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:21.459792+0000 osd.7 (osd.7) 416 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:45:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:21.461433+0000 osd.7 (osd.7) 417 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:45:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:21.618993+0000 osd.3 (osd.3) 311 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:45:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:21.622477+0000 osd.3 (osd.3) 312 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:45:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:21.812438+0000 mon.c (mon.2) 516 : audit [INF] from='client.? 172.21.15.87:0/3097224729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-23T00:45:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:21.812713+0000 mon.a (mon.0) 4016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-23T00:45:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:21.830584+0000 mgr.y (mgr.24437) 727 : cluster [DBG] pgmap v1307: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:22.051531+0000 osd.5 (osd.5) 355 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:45:22.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:22.052493+0000 osd.5 (osd.5) 356 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:45:23.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:21.459792+0000 osd.7 (osd.7) 416 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:45:23.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:21.461433+0000 osd.7 (osd.7) 417 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:45:23.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:21.618993+0000 osd.3 (osd.3) 311 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:45:23.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:21.622477+0000 osd.3 (osd.3) 312 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:21.812438+0000 mon.c (mon.2) 516 : audit [INF] from='client.? 172.21.15.87:0/3097224729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:21.812713+0000 mon.a (mon.0) 4016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:21.830584+0000 mgr.y (mgr.24437) 727 : cluster [DBG] pgmap v1307: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:22.051531+0000 osd.5 (osd.5) 355 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:22.052493+0000 osd.5 (osd.5) 356 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:21.459792+0000 osd.7 (osd.7) 416 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:21.461433+0000 osd.7 (osd.7) 417 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:21.618993+0000 osd.3 (osd.3) 311 : cluster [DBG] 5.0 scrub starts 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:21.622477+0000 osd.3 (osd.3) 312 : cluster [DBG] 5.0 scrub ok 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:21.812438+0000 mon.c (mon.2) 516 : audit [INF] from='client.? 172.21.15.87:0/3097224729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-23T00:45:23.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:21.812713+0000 mon.a (mon.0) 4016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-23T00:45:23.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:21.830584+0000 mgr.y (mgr.24437) 727 : cluster [DBG] pgmap v1307: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:23.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:22.051531+0000 osd.5 (osd.5) 355 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:45:23.085 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:22.052493+0000 osd.5 (osd.5) 356 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:45:23.555 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-23T00:45:23.831 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:22.424685+0000 osd.1 (osd.1) 424 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:22.427987+0000 osd.1 (osd.1) 425 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:22.440707+0000 osd.7 (osd.7) 418 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:22.441620+0000 osd.7 (osd.7) 419 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:22.611820+0000 osd.3 (osd.3) 313 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:22.615407+0000 osd.3 (osd.3) 314 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:22.658552+0000 mon.a (mon.0) 4017 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:22.658629+0000 mon.a (mon.0) 4018 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:23.043042+0000 osd.5 (osd.5) 357 : cluster [DBG] 22.1b deep-scrub starts 2024-02-23T00:45:23.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:23.045086+0000 osd.5 (osd.5) 358 : cluster [DBG] 22.1b deep-scrub ok 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:22.424685+0000 osd.1 (osd.1) 424 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:22.427987+0000 osd.1 (osd.1) 425 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:22.440707+0000 osd.7 (osd.7) 418 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:22.441620+0000 osd.7 (osd.7) 419 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:22.611820+0000 osd.3 (osd.3) 313 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:22.615407+0000 osd.3 (osd.3) 314 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:22.658552+0000 mon.a (mon.0) 4017 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:22.658629+0000 mon.a (mon.0) 4018 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:23.043042+0000 osd.5 (osd.5) 357 : cluster [DBG] 22.1b deep-scrub starts 2024-02-23T00:45:23.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:23.045086+0000 osd.5 (osd.5) 358 : cluster [DBG] 22.1b deep-scrub ok 2024-02-23T00:45:23.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:22.424685+0000 osd.1 (osd.1) 424 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:22.427987+0000 osd.1 (osd.1) 425 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:22.440707+0000 osd.7 (osd.7) 418 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:22.441620+0000 osd.7 (osd.7) 419 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:22.611820+0000 osd.3 (osd.3) 313 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:22.615407+0000 osd.3 (osd.3) 314 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:22.658552+0000 mon.a (mon.0) 4017 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:22.658629+0000 mon.a (mon.0) 4018 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:23.043042+0000 osd.5 (osd.5) 357 : cluster [DBG] 22.1b deep-scrub starts 2024-02-23T00:45:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:23.045086+0000 osd.5 (osd.5) 358 : cluster [DBG] 22.1b deep-scrub ok 2024-02-23T00:45:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:23.568418+0000 osd.3 (osd.3) 315 : cluster [DBG] 3.b scrub starts 2024-02-23T00:45:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:23.571420+0000 osd.3 (osd.3) 316 : cluster [DBG] 3.b scrub ok 2024-02-23T00:45:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:23.831114+0000 mgr.y (mgr.24437) 728 : cluster [DBG] pgmap v1309: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:25.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:23.568418+0000 osd.3 (osd.3) 315 : cluster [DBG] 3.b scrub starts 2024-02-23T00:45:25.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:23.571420+0000 osd.3 (osd.3) 316 : cluster [DBG] 3.b scrub ok 2024-02-23T00:45:25.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:23.831114+0000 mgr.y (mgr.24437) 728 : cluster [DBG] pgmap v1309: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:25.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:23.568418+0000 osd.3 (osd.3) 315 : cluster [DBG] 3.b scrub starts 2024-02-23T00:45:25.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:23.571420+0000 osd.3 (osd.3) 316 : cluster [DBG] 3.b scrub ok 2024-02-23T00:45:25.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:23.831114+0000 mgr.y (mgr.24437) 728 : cluster [DBG] pgmap v1309: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:25.761 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 -- 172.21.15.87:0/4290695304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 msgr2=0x7f95f8101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 --2- 172.21.15.87:0/4290695304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 0x7f95f8101dc0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 rx=0x7f95e8004870 tx=0x7f95e8009cb0).stop 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 -- 172.21.15.87:0/4290695304 shutdown_connections 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 --2- 172.21.15.87:0/4290695304 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95f81022b0 0x7f95f81046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 --2- 172.21.15.87:0/4290695304 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95f80f8d50 0x7f95f80f9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 --2- 172.21.15.87:0/4290695304 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 0x7f95f8101dc0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 -- 172.21.15.87:0/4290695304 >> 172.21.15.87:0/4290695304 conn(0x7f95f80f2820 msgr2=0x7f95f80f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 -- 172.21.15.87:0/4290695304 shutdown_connections 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 -- 172.21.15.87:0/4290695304 wait complete. 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 librados: starting msgr at 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.720+0000 7f95fce55700 1 librados: starting objecter 2024-02-23T00:45:25.762 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 Processor -- start 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 -- start start 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 librados: setting wanted keys 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 librados: calling monclient init 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95f80f8d50 0x7f95f80f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 0x7f95f80f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95f81022b0 0x7f95f80f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f81167b0 con 0x7f95f80f9620 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f81168f0 con 0x7f95f80f8d50 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f8116ab0 con 0x7f95f81022b0 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f5d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 0x7f95f80f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f5d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 0x7f95f80f71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49628/0 (socket says 172.21.15.87:49628) 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f5d9b700 1 -- 172.21.15.87:0/3381843215 learned_addr learned my addr 172.21.15.87:0/3381843215 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f6d9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95f81022b0 0x7f95f80f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f659c700 1 --2- 172.21.15.87:0/3381843215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95f80f8d50 0x7f95f80f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:25.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f5d9b700 1 -- 172.21.15.87:0/3381843215 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95f81022b0 msgr2=0x7f95f80f76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f5d9b700 1 --2- 172.21.15.87:0/3381843215 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95f81022b0 0x7f95f80f76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f5d9b700 1 -- 172.21.15.87:0/3381843215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95f80f8d50 msgr2=0x7f95f80f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f5d9b700 1 --2- 172.21.15.87:0/3381843215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95f80f8d50 0x7f95f80f6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f5d9b700 1 -- 172.21.15.87:0/3381843215 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e8005040 con 0x7f95f80f9620 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f659c700 1 --2- 172.21.15.87:0/3381843215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95f80f8d50 0x7f95f80f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95f5d9b700 1 --2- 172.21.15.87:0/3381843215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 0x7f95f80f71f0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f95e8009b10 tx=0x7f95e8007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95e77fe700 10 client.?.objecter ms_handle_connect 0x7f95f80f9620 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95e77fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95e77fe700 1 -- 172.21.15.87:0/3381843215 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e800e030 con 0x7f95f80f9620 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95e77fe700 1 -- 172.21.15.87:0/3381843215 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95e80092e0 con 0x7f95f80f9620 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f8116d30 con 0x7f95f80f9620 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.721+0000 7f95e77fe700 1 -- 172.21.15.87:0/3381843215 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8018070 con 0x7f95f80f9620 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.722+0000 7f95fce55700 10 client.28053.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:25.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.722+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f81170e0 con 0x7f95f80f9620 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.722+0000 7f95fce55700 1 librados: init done 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.723+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f805fa60 con 0x7f95f80f9620 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.723+0000 7f95e77fe700 1 -- 172.21.15.87:0/3381843215 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f95e801d420 con 0x7f95f80f9620 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.723+0000 7f95e77fe700 1 --2- 172.21.15.87:0/3381843215 >> v2:172.21.15.87:6800/1231597463 conn(0x7f95e0062c90 0x7f95e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.723+0000 7f95e77fe700 1 -- 172.21.15.87:0/3381843215 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(840..840 src has 400..840) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f95e807e270 con 0x7f95f80f9620 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.723+0000 7f95e77fe700 10 client.28053.objecter ms_dispatch 0x7f95f8001550 osd_map(840..840 src has 400..840) v4 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.723+0000 7f95e77fe700 3 client.28053.objecter handle_osd_map got epochs [840,840] > 0 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.723+0000 7f95e77fe700 3 client.28053.objecter handle_osd_map decoding full epoch 840 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.723+0000 7f95f659c700 1 --2- 172.21.15.87:0/3381843215 >> v2:172.21.15.87:6800/1231597463 conn(0x7f95e0062c90 0x7f95e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:25.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.723+0000 7f95e77fe700 20 client.28053.objecter dump_active .. 0 homeless 2024-02-23T00:45:25.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.724+0000 7f95e77fe700 10 client.28053.objecter ms_handle_connect 0x7f95e0062c90 2024-02-23T00:45:25.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.724+0000 7f95f659c700 1 --2- 172.21.15.87:0/3381843215 >> v2:172.21.15.87:6800/1231597463 conn(0x7f95e0062c90 0x7f95e0065150 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7f95ec009990 tx=0x7f95ec008040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:25.766 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.727+0000 7f95e77fe700 1 -- 172.21.15.87:0/3381843215 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f95e80519c0 con 0x7f95f80f9620 2024-02-23T00:45:25.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:25.905+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f95f80f8140 con 0x7f95f80f9620 2024-02-23T00:45:26.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:25.340580+0000 osd.6 (osd.6) 376 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:45:26.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:25.344053+0000 osd.6 (osd.6) 377 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:45:26.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:25.340580+0000 osd.6 (osd.6) 376 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:45:26.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:25.344053+0000 osd.6 (osd.6) 377 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:45:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:25.340580+0000 osd.6 (osd.6) 376 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:45:26.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:25.344053+0000 osd.6 (osd.6) 377 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:45:26.802 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.799+0000 7f95e77fe700 1 -- 172.21.15.87:0/3381843215 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v841) v1 ==== 133+0+0 (secure 0 0 0) 0x7f95e8004490 con 0x7f95f80f9620 2024-02-23T00:45:26.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.802+0000 7f95fce55700 10 librados: watch_flush enter 2024-02-23T00:45:26.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 10 librados: watch_flush exit 2024-02-23T00:45:26.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 20 client.28053.objecter shutdown clearing up homeless session... 2024-02-23T00:45:26.805 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 10 client.28053.objecter successfully canceled tick 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 >> v2:172.21.15.87:6800/1231597463 conn(0x7f95e0062c90 msgr2=0x7f95e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 --2- 172.21.15.87:0/3381843215 >> v2:172.21.15.87:6800/1231597463 conn(0x7f95e0062c90 0x7f95e0065150 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 rx=0x7f95ec009990 tx=0x7f95ec008040).stop 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 msgr2=0x7f95f80f71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 --2- 172.21.15.87:0/3381843215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 0x7f95f80f71f0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 rx=0x7f95e8009b10 tx=0x7f95e8007f90).stop 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 shutdown_connections 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 --2- 172.21.15.87:0/3381843215 >> v2:172.21.15.87:6800/1231597463 conn(0x7f95e0062c90 0x7f95e0065150 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 --2- 172.21.15.87:0/3381843215 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f95f81022b0 0x7f95f80f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 --2- 172.21.15.87:0/3381843215 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f95f80f8d50 0x7f95f80f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 --2- 172.21.15.87:0/3381843215 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f95f80f9620 0x7f95f80f71f0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 >> 172.21.15.87:0/3381843215 conn(0x7f95f80f2820 msgr2=0x7f95f80f3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 shutdown_connections 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 -- 172.21.15.87:0/3381843215 wait complete. 2024-02-23T00:45:26.806 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:26.803+0000 7f95fce55700 1 librados: shutdown 2024-02-23T00:45:26.807 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-23T00:45:27.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:25.831813+0000 mgr.y (mgr.24437) 729 : cluster [DBG] pgmap v1310: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:45:27.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:25.907289+0000 mon.a (mon.0) 4019 : audit [INF] from='client.? 172.21.15.87:0/3381843215' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-23T00:45:27.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:26.115695+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.b deep-scrub starts 2024-02-23T00:45:27.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:26.120886+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.b deep-scrub ok 2024-02-23T00:45:27.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:26.555524+0000 osd.3 (osd.3) 317 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:45:27.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:26.558808+0000 osd.3 (osd.3) 318 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:45:27.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:25.831813+0000 mgr.y (mgr.24437) 729 : cluster [DBG] pgmap v1310: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:45:27.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:25.907289+0000 mon.a (mon.0) 4019 : audit [INF] from='client.? 172.21.15.87:0/3381843215' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-23T00:45:27.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:26.115695+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.b deep-scrub starts 2024-02-23T00:45:27.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:26.120886+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.b deep-scrub ok 2024-02-23T00:45:27.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:26.555524+0000 osd.3 (osd.3) 317 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:45:27.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:26.558808+0000 osd.3 (osd.3) 318 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:45:27.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:25.831813+0000 mgr.y (mgr.24437) 729 : cluster [DBG] pgmap v1310: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:45:27.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:25.907289+0000 mon.a (mon.0) 4019 : audit [INF] from='client.? 172.21.15.87:0/3381843215' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-23T00:45:27.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:26.115695+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.b deep-scrub starts 2024-02-23T00:45:27.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:26.120886+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.b deep-scrub ok 2024-02-23T00:45:27.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:26.555524+0000 osd.3 (osd.3) 317 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:45:27.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:26.558808+0000 osd.3 (osd.3) 318 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:45:27.616 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight 7 1 2024-02-23T00:45:27.888 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:28.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:26.800602+0000 mon.a (mon.0) 4020 : audit [INF] from='client.? 172.21.15.87:0/3381843215' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-23T00:45:28.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:26.800688+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-02-23T00:45:28.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:27.335880+0000 osd.6 (osd.6) 378 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:45:28.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:27.339165+0000 osd.6 (osd.6) 379 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:45:28.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:26.800602+0000 mon.a (mon.0) 4020 : audit [INF] from='client.? 172.21.15.87:0/3381843215' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-23T00:45:28.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:26.800688+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-02-23T00:45:28.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:27.335880+0000 osd.6 (osd.6) 378 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:45:28.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:27.339165+0000 osd.6 (osd.6) 379 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:45:28.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:26.800602+0000 mon.a (mon.0) 4020 : audit [INF] from='client.? 172.21.15.87:0/3381843215' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-23T00:45:28.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:26.800688+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-02-23T00:45:28.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:27.335880+0000 osd.6 (osd.6) 378 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:45:28.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:27.339165+0000 osd.6 (osd.6) 379 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:45:29.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:27.423469+0000 osd.4 (osd.4) 375 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:45:29.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:27.426396+0000 osd.4 (osd.4) 376 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:45:29.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:27.832212+0000 mgr.y (mgr.24437) 730 : cluster [DBG] pgmap v1312: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:29.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:28.133910+0000 osd.5 (osd.5) 359 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:45:29.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:28.135227+0000 osd.5 (osd.5) 360 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:45:29.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:28.549620+0000 osd.3 (osd.3) 319 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:45:29.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:28.552523+0000 osd.3 (osd.3) 320 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:45:29.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:27.423469+0000 osd.4 (osd.4) 375 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:45:29.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:27.426396+0000 osd.4 (osd.4) 376 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:45:29.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:27.832212+0000 mgr.y (mgr.24437) 730 : cluster [DBG] pgmap v1312: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:29.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:28.133910+0000 osd.5 (osd.5) 359 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:45:29.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:28.135227+0000 osd.5 (osd.5) 360 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:45:29.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:28.549620+0000 osd.3 (osd.3) 319 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:45:29.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:28.552523+0000 osd.3 (osd.3) 320 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:45:29.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:27.423469+0000 osd.4 (osd.4) 375 : cluster [DBG] 4.1 scrub starts 2024-02-23T00:45:29.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:27.426396+0000 osd.4 (osd.4) 376 : cluster [DBG] 4.1 scrub ok 2024-02-23T00:45:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:27.832212+0000 mgr.y (mgr.24437) 730 : cluster [DBG] pgmap v1312: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:28.133910+0000 osd.5 (osd.5) 359 : cluster [DBG] 22.6 scrub starts 2024-02-23T00:45:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:28.135227+0000 osd.5 (osd.5) 360 : cluster [DBG] 22.6 scrub ok 2024-02-23T00:45:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:28.549620+0000 osd.3 (osd.3) 319 : cluster [DBG] 3.8 scrub starts 2024-02-23T00:45:29.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:28.552523+0000 osd.3 (osd.3) 320 : cluster [DBG] 3.8 scrub ok 2024-02-23T00:45:29.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 -- 172.21.15.87:0/296312862 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 msgr2=0x7f3760103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:29.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 --2- 172.21.15.87:0/296312862 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 0x7f3760103a20 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 rx=0x7f3750004870 tx=0x7f3750009cb0).stop 2024-02-23T00:45:29.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 -- 172.21.15.87:0/296312862 shutdown_connections 2024-02-23T00:45:29.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 --2- 172.21.15.87:0/296312862 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 0x7f3760103a20 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:29.994 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 --2- 172.21.15.87:0/296312862 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37600fcbb0 0x7f37600fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:29.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 --2- 172.21.15.87:0/296312862 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37600fd510 0x7f37600f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:29.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 -- 172.21.15.87:0/296312862 >> 172.21.15.87:0/296312862 conn(0x7f37600f2820 msgr2=0x7f37600f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:29.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 -- 172.21.15.87:0/296312862 shutdown_connections 2024-02-23T00:45:29.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 -- 172.21.15.87:0/296312862 wait complete. 2024-02-23T00:45:29.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 librados: starting msgr at 2024-02-23T00:45:29.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.991+0000 7f376510e700 1 librados: starting objecter 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.992+0000 7f376510e700 1 Processor -- start 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.992+0000 7f376510e700 1 -- start start 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.992+0000 7f376510e700 1 librados: setting wanted keys 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.992+0000 7f376510e700 1 librados: calling monclient init 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 0x7f3760108f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37600fcbb0 0x7f3760109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37600fd510 0x7f3760109a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3760114110 con 0x7f37600fd510 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3760114280 con 0x7f37600fcbb0 2024-02-23T00:45:29.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3760114440 con 0x7f37600f7400 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375ed9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 0x7f3760108f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375f59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37600fd510 0x7f3760109a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375ed9d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 0x7f3760108f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:40968/0 (socket says 172.21.15.87:40968) 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375f59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37600fd510 0x7f3760109a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49650/0 (socket says 172.21.15.87:49650) 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375ed9d700 1 -- 172.21.15.87:0/2930598136 learned_addr learned my addr 172.21.15.87:0/2930598136 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375e59c700 1 --2- 172.21.15.87:0/2930598136 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37600fcbb0 0x7f3760109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375ed9d700 1 -- 172.21.15.87:0/2930598136 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37600fcbb0 msgr2=0x7f3760109420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375ed9d700 1 --2- 172.21.15.87:0/2930598136 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37600fcbb0 0x7f3760109420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375ed9d700 1 -- 172.21.15.87:0/2930598136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37600fd510 msgr2=0x7f3760109a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375ed9d700 1 --2- 172.21.15.87:0/2930598136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37600fd510 0x7f3760109a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375ed9d700 1 -- 172.21.15.87:0/2930598136 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580097c0 con 0x7f37600f7400 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375f59e700 1 --2- 172.21.15.87:0/2930598136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37600fd510 0x7f3760109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f375ed9d700 1 --2- 172.21.15.87:0/2930598136 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 0x7f3760108f30 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f3758001e70 tx=0x7f3758004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:29.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f3757fff700 10 client.?.objecter ms_handle_connect 0x7f37600f7400 2024-02-23T00:45:29.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f3757fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:29.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f3757fff700 1 -- 172.21.15.87:0/2930598136 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37580099f0 con 0x7f37600f7400 2024-02-23T00:45:29.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3750005040 con 0x7f37600f7400 2024-02-23T00:45:29.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f3757fff700 1 -- 172.21.15.87:0/2930598136 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37580090d0 con 0x7f37600f7400 2024-02-23T00:45:29.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f3757fff700 1 -- 172.21.15.87:0/2930598136 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758007c70 con 0x7f37600f7400 2024-02-23T00:45:29.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 10 client.38537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:29.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3760107130 con 0x7f37600f7400 2024-02-23T00:45:29.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.993+0000 7f376510e700 1 librados: init done 2024-02-23T00:45:29.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.994+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376005fa60 con 0x7f37600f7400 2024-02-23T00:45:30.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.995+0000 7f3757fff700 1 -- 172.21.15.87:0/2930598136 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f375800fad0 con 0x7f37600f7400 2024-02-23T00:45:30.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.995+0000 7f3757fff700 1 --2- 172.21.15.87:0/2930598136 >> v2:172.21.15.87:6800/1231597463 conn(0x7f374c062c90 0x7f374c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:30.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.996+0000 7f3757fff700 1 -- 172.21.15.87:0/2930598136 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(841..841 src has 400..841) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f375807dbe0 con 0x7f37600f7400 2024-02-23T00:45:30.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.996+0000 7f3757fff700 10 client.38537.objecter ms_dispatch 0x7f3760001550 osd_map(841..841 src has 400..841) v4 2024-02-23T00:45:30.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.996+0000 7f375e59c700 1 --2- 172.21.15.87:0/2930598136 >> v2:172.21.15.87:6800/1231597463 conn(0x7f374c062c90 0x7f374c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:30.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.996+0000 7f3757fff700 3 client.38537.objecter handle_osd_map got epochs [841,841] > 0 2024-02-23T00:45:30.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.996+0000 7f3757fff700 3 client.38537.objecter handle_osd_map decoding full epoch 841 2024-02-23T00:45:30.000 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.996+0000 7f3757fff700 20 client.38537.objecter dump_active .. 0 homeless 2024-02-23T00:45:30.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.996+0000 7f3757fff700 10 client.38537.objecter ms_handle_connect 0x7f374c062c90 2024-02-23T00:45:30.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.996+0000 7f375e59c700 1 --2- 172.21.15.87:0/2930598136 >> v2:172.21.15.87:6800/1231597463 conn(0x7f374c062c90 0x7f374c065150 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7f3760061700 tx=0x7f3748009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:30.001 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:29.999+0000 7f3757fff700 1 -- 172.21.15.87:0/2930598136 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f37580511a0 con 0x7f37600f7400 2024-02-23T00:45:30.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:28.349112+0000 osd.1 (osd.1) 426 : cluster [DBG] 3.4 deep-scrub starts 2024-02-23T00:45:30.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:28.352029+0000 osd.1 (osd.1) 427 : cluster [DBG] 3.4 deep-scrub ok 2024-02-23T00:45:30.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:29.337807+0000 osd.6 (osd.6) 380 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:45:30.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:29.341338+0000 osd.6 (osd.6) 381 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:45:30.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:28.349112+0000 osd.1 (osd.1) 426 : cluster [DBG] 3.4 deep-scrub starts 2024-02-23T00:45:30.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:28.352029+0000 osd.1 (osd.1) 427 : cluster [DBG] 3.4 deep-scrub ok 2024-02-23T00:45:30.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:29.337807+0000 osd.6 (osd.6) 380 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:45:30.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:29.341338+0000 osd.6 (osd.6) 381 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:45:30.176 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.174+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f376005ee80 con 0x7f37600f7400 2024-02-23T00:45:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:28.349112+0000 osd.1 (osd.1) 426 : cluster [DBG] 3.4 deep-scrub starts 2024-02-23T00:45:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:28.352029+0000 osd.1 (osd.1) 427 : cluster [DBG] 3.4 deep-scrub ok 2024-02-23T00:45:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:29.337807+0000 osd.6 (osd.6) 380 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:45:30.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:29.341338+0000 osd.6 (osd.6) 381 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:45:30.820 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.817+0000 7f3757fff700 1 -- 172.21.15.87:0/2930598136 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v842) v1 ==== 113+0+0 (secure 0 0 0) 0x7f3758050fc0 con 0x7f37600f7400 2024-02-23T00:45:30.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 10 librados: watch_flush enter 2024-02-23T00:45:30.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 10 librados: watch_flush exit 2024-02-23T00:45:30.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 20 client.38537.objecter shutdown clearing up homeless session... 2024-02-23T00:45:30.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 10 client.38537.objecter successfully canceled tick 2024-02-23T00:45:30.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 >> v2:172.21.15.87:6800/1231597463 conn(0x7f374c062c90 msgr2=0x7f374c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:30.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 --2- 172.21.15.87:0/2930598136 >> v2:172.21.15.87:6800/1231597463 conn(0x7f374c062c90 0x7f374c065150 secure :-1 s=READY pgs=3005 cs=0 l=1 rev1=1 rx=0x7f3760061700 tx=0x7f3748009040).stop 2024-02-23T00:45:30.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 msgr2=0x7f3760108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:30.823 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 --2- 172.21.15.87:0/2930598136 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 0x7f3760108f30 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 rx=0x7f3758001e70 tx=0x7f3758004c50).stop 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 shutdown_connections 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 --2- 172.21.15.87:0/2930598136 >> v2:172.21.15.87:6800/1231597463 conn(0x7f374c062c90 0x7f374c065150 unknown :-1 s=CLOSED pgs=3005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 --2- 172.21.15.87:0/2930598136 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f37600fd510 0x7f3760109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 --2- 172.21.15.87:0/2930598136 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f37600f7400 0x7f3760108f30 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 --2- 172.21.15.87:0/2930598136 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f37600fcbb0 0x7f3760109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 >> 172.21.15.87:0/2930598136 conn(0x7f37600f2820 msgr2=0x7f37600faa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 shutdown_connections 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.820+0000 7f376510e700 1 -- 172.21.15.87:0/2930598136 wait complete. 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:30.821+0000 7f376510e700 1 librados: shutdown 2024-02-23T00:45:30.824 INFO:teuthology.orchestra.run.smithi087.stderr:reweighted osd.7 to 1 (10000) 2024-02-23T00:45:31.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:29.387834+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.6 deep-scrub starts 2024-02-23T00:45:31.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:29.390754+0000 osd.1 (osd.1) 429 : cluster [DBG] 2.6 deep-scrub ok 2024-02-23T00:45:31.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:29.457109+0000 osd.4 (osd.4) 377 : cluster [DBG] 3.14 scrub starts 2024-02-23T00:45:31.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:29.458177+0000 osd.4 (osd.4) 378 : cluster [DBG] 3.14 scrub ok 2024-02-23T00:45:31.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:29.832854+0000 mgr.y (mgr.24437) 731 : cluster [DBG] pgmap v1313: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:45:31.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:30.158897+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:45:31.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:30.162368+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:45:31.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:30.175852+0000 mon.c (mon.2) 517 : audit [INF] from='client.? 172.21.15.87:0/2930598136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:31.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:30.176128+0000 mon.a (mon.0) 4022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:31.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:29.387834+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.6 deep-scrub starts 2024-02-23T00:45:31.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:29.390754+0000 osd.1 (osd.1) 429 : cluster [DBG] 2.6 deep-scrub ok 2024-02-23T00:45:31.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:29.457109+0000 osd.4 (osd.4) 377 : cluster [DBG] 3.14 scrub starts 2024-02-23T00:45:31.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:29.458177+0000 osd.4 (osd.4) 378 : cluster [DBG] 3.14 scrub ok 2024-02-23T00:45:31.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:29.832854+0000 mgr.y (mgr.24437) 731 : cluster [DBG] pgmap v1313: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:45:31.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:30.158897+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:45:31.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:30.162368+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:45:31.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:30.175852+0000 mon.c (mon.2) 517 : audit [INF] from='client.? 172.21.15.87:0/2930598136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:31.085 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:30.176128+0000 mon.a (mon.0) 4022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:29.387834+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.6 deep-scrub starts 2024-02-23T00:45:31.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:29.390754+0000 osd.1 (osd.1) 429 : cluster [DBG] 2.6 deep-scrub ok 2024-02-23T00:45:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:29.457109+0000 osd.4 (osd.4) 377 : cluster [DBG] 3.14 scrub starts 2024-02-23T00:45:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:29.458177+0000 osd.4 (osd.4) 378 : cluster [DBG] 3.14 scrub ok 2024-02-23T00:45:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:29.832854+0000 mgr.y (mgr.24437) 731 : cluster [DBG] pgmap v1313: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:45:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:30.158897+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.3 scrub starts 2024-02-23T00:45:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:30.162368+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.3 scrub ok 2024-02-23T00:45:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:30.175852+0000 mon.c (mon.2) 517 : audit [INF] from='client.? 172.21.15.87:0/2930598136' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:31.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:30.176128+0000 mon.a (mon.0) 4022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:31.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:45:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:45:31.712 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-23T00:45:31.986 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:32.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:30.818464+0000 mon.a (mon.0) 4023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-23T00:45:32.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:30.818547+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-02-23T00:45:32.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:31.413074+0000 mgr.y (mgr.24437) 732 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:32.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:30.818464+0000 mon.a (mon.0) 4023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-23T00:45:32.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:30.818547+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-02-23T00:45:32.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:31.413074+0000 mgr.y (mgr.24437) 732 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:32.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:30.818464+0000 mon.a (mon.0) 4023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-23T00:45:32.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:30.818547+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-02-23T00:45:32.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:31.413074+0000 mgr.y (mgr.24437) 732 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:32.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:32] "GET /metrics HTTP/1.1" 200 186066 "" "Prometheus/2.33.4" 2024-02-23T00:45:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:31.449685+0000 osd.4 (osd.4) 379 : cluster [DBG] 3.3 deep-scrub starts 2024-02-23T00:45:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:31.450744+0000 osd.4 (osd.4) 380 : cluster [DBG] 3.3 deep-scrub ok 2024-02-23T00:45:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:31.833202+0000 mgr.y (mgr.24437) 733 : cluster [DBG] pgmap v1315: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:32.287636+0000 mon.a (mon.0) 4025 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-02-23T00:45:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:32.488921+0000 osd.3 (osd.3) 321 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:45:33.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:32.492418+0000 osd.3 (osd.3) 322 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:45:33.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:31.449685+0000 osd.4 (osd.4) 379 : cluster [DBG] 3.3 deep-scrub starts 2024-02-23T00:45:33.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:31.450744+0000 osd.4 (osd.4) 380 : cluster [DBG] 3.3 deep-scrub ok 2024-02-23T00:45:33.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:31.833202+0000 mgr.y (mgr.24437) 733 : cluster [DBG] pgmap v1315: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:33.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:32.287636+0000 mon.a (mon.0) 4025 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-02-23T00:45:33.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:32.488921+0000 osd.3 (osd.3) 321 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:45:33.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:32.492418+0000 osd.3 (osd.3) 322 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:45:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:31.449685+0000 osd.4 (osd.4) 379 : cluster [DBG] 3.3 deep-scrub starts 2024-02-23T00:45:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:31.450744+0000 osd.4 (osd.4) 380 : cluster [DBG] 3.3 deep-scrub ok 2024-02-23T00:45:33.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:31.833202+0000 mgr.y (mgr.24437) 733 : cluster [DBG] pgmap v1315: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:33.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:32.287636+0000 mon.a (mon.0) 4025 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-02-23T00:45:33.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:32.488921+0000 osd.3 (osd.3) 321 : cluster [DBG] 4.11 scrub starts 2024-02-23T00:45:33.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:32.492418+0000 osd.3 (osd.3) 322 : cluster [DBG] 4.11 scrub ok 2024-02-23T00:45:33.927 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/1498205078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fcab0 msgr2=0x7f0a740fce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/1498205078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fcab0 0x7f0a740fce90 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 rx=0x7f0a68004870 tx=0x7f0a68009cb0).stop 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/1498205078 shutdown_connections 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/1498205078 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0a740f7320 0x7f0a74103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/1498205078 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fcab0 0x7f0a740fce90 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/1498205078 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a740fd410 0x7f0a740f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/1498205078 >> 172.21.15.87:0/1498205078 conn(0x7f0a740f26e0 msgr2=0x7f0a740f4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/1498205078 shutdown_connections 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/1498205078 wait complete. 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 librados: starting msgr at 2024-02-23T00:45:33.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 librados: starting objecter 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.897+0000 7f0a7a9d2700 1 Processor -- start 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a7a9d2700 1 -- start start 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a7a9d2700 1 librados: setting wanted keys 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a7a9d2700 1 librados: calling monclient init 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a7a9d2700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a740f7320 0x7f0a741091f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a7a9d2700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0a740fcab0 0x7f0a741096e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a7a9d2700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fd410 0x7f0a74109ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a7a9d2700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a741143e0 con 0x7f0a740fcab0 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a7a9d2700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a74114550 con 0x7f0a740f7320 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a7a9d2700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a74114710 con 0x7f0a740fd410 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a78f6f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fd410 0x7f0a74109ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a737fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0a740fcab0 0x7f0a741096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a78f6f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fd410 0x7f0a74109ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:38712/0 (socket says 172.21.15.87:38712) 2024-02-23T00:45:33.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a737fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0a740fcab0 0x7f0a741096e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34956/0 (socket says 172.21.15.87:34956) 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a78f6f700 1 -- 172.21.15.87:0/2978962766 learned_addr learned my addr 172.21.15.87:0/2978962766 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a73fff700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a740f7320 0x7f0a741091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a78f6f700 1 -- 172.21.15.87:0/2978962766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a740f7320 msgr2=0x7f0a741091f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a78f6f700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a740f7320 0x7f0a741091f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a78f6f700 1 -- 172.21.15.87:0/2978962766 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0a740fcab0 msgr2=0x7f0a741096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a78f6f700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0a740fcab0 0x7f0a741096e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a78f6f700 1 -- 172.21.15.87:0/2978962766 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0097c0 con 0x7f0a740fd410 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a73fff700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a740f7320 0x7f0a741091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a737fe700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0a740fcab0 0x7f0a741096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a78f6f700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fd410 0x7f0a74109ca0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 rx=0x7f0a6c001df0 tx=0x7f0a6c004c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a717fa700 10 client.?.objecter ms_handle_connect 0x7f0a740fd410 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.898+0000 7f0a717fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.899+0000 7f0a717fa700 1 -- 172.21.15.87:0/2978962766 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c009a10 con 0x7f0a740fd410 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.899+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a68005040 con 0x7f0a740fd410 2024-02-23T00:45:33.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.899+0000 7f0a717fa700 1 -- 172.21.15.87:0/2978962766 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a6c0095a0 con 0x7f0a740fd410 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.899+0000 7f0a717fa700 1 -- 172.21.15.87:0/2978962766 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c007da0 con 0x7f0a740fd410 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.899+0000 7f0a7a9d2700 10 client.38543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.899+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a74114d50 con 0x7f0a740fd410 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.899+0000 7f0a7a9d2700 1 librados: init done 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7405f950 con 0x7f0a740fd410 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a717fa700 1 -- 172.21.15.87:0/2978962766 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f0a6c0090d0 con 0x7f0a740fd410 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a717fa700 1 --2- 172.21.15.87:0/2978962766 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0a5c062c90 0x7f0a5c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a717fa700 1 -- 172.21.15.87:0/2978962766 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(843..843 src has 400..843) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f0a6c07d8f0 con 0x7f0a740fd410 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a717fa700 10 client.38543.objecter ms_dispatch 0x7f0a74001550 osd_map(843..843 src has 400..843) v4 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a717fa700 3 client.38543.objecter handle_osd_map got epochs [843,843] > 0 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a717fa700 3 client.38543.objecter handle_osd_map decoding full epoch 843 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a73fff700 1 --2- 172.21.15.87:0/2978962766 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0a5c062c90 0x7f0a5c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a717fa700 20 client.38543.objecter dump_active .. 0 homeless 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a717fa700 10 client.38543.objecter ms_handle_connect 0x7f0a5c062c90 2024-02-23T00:45:33.931 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.900+0000 7f0a73fff700 1 --2- 172.21.15.87:0/2978962766 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0a5c062c90 0x7f0a5c065150 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7f0a680043d0 tx=0x7f0a6800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:33.932 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:33.904+0000 7f0a717fa700 1 -- 172.21.15.87:0/2978962766 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f0a6c051040 con 0x7f0a740fd410 2024-02-23T00:45:34.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:32.642961+0000 osd.7 (osd.7) 420 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:45:34.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:32.645869+0000 osd.7 (osd.7) 421 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:45:34.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:33.437057+0000 osd.3 (osd.3) 323 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:45:34.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:33.440633+0000 osd.3 (osd.3) 324 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:45:34.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:32.642961+0000 osd.7 (osd.7) 420 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:45:34.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:32.645869+0000 osd.7 (osd.7) 421 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:45:34.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:33.437057+0000 osd.3 (osd.3) 323 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:45:34.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:33.440633+0000 osd.3 (osd.3) 324 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:45:34.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.082+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f0a74061570 con 0x7f0a740fd410 2024-02-23T00:45:34.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:45:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:45:34.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:32.642961+0000 osd.7 (osd.7) 420 : cluster [DBG] 1.0 scrub starts 2024-02-23T00:45:34.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:32.645869+0000 osd.7 (osd.7) 421 : cluster [DBG] 1.0 scrub ok 2024-02-23T00:45:34.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:33.437057+0000 osd.3 (osd.3) 323 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:45:34.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:33.440633+0000 osd.3 (osd.3) 324 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:45:34.841 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.840+0000 7f0a717fa700 1 -- 172.21.15.87:0/2978962766 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v844) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0a6c050c30 con 0x7f0a740fd410 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 10 librados: watch_flush enter 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 10 librados: watch_flush exit 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 20 client.38543.objecter shutdown clearing up homeless session... 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 10 client.38543.objecter successfully canceled tick 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0a5c062c90 msgr2=0x7f0a5c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/2978962766 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0a5c062c90 0x7f0a5c065150 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 rx=0x7f0a680043d0 tx=0x7f0a6800b040).stop 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fd410 msgr2=0x7f0a74109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fd410 0x7f0a74109ca0 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 rx=0x7f0a6c001df0 tx=0x7f0a6c004c50).stop 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 shutdown_connections 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/2978962766 >> v2:172.21.15.87:6800/1231597463 conn(0x7f0a5c062c90 0x7f0a5c065150 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f0a740fd410 0x7f0a74109ca0 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:34.844 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f0a740f7320 0x7f0a741091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:34.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 --2- 172.21.15.87:0/2978962766 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f0a740fcab0 0x7f0a741096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:34.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 >> 172.21.15.87:0/2978962766 conn(0x7f0a740f26e0 msgr2=0x7f0a740fa8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:34.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 shutdown_connections 2024-02-23T00:45:34.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 -- 172.21.15.87:0/2978962766 wait complete. 2024-02-23T00:45:34.845 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:34.842+0000 7f0a7a9d2700 1 librados: shutdown 2024-02-23T00:45:34.845 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-23T00:45:35.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:33.407691+0000 osd.1 (osd.1) 430 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:45:35.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:33.410631+0000 osd.1 (osd.1) 431 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:45:35.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:33.462794+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:45:35.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:33.464721+0000 osd.4 (osd.4) 382 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:45:35.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:33.833683+0000 mgr.y (mgr.24437) 734 : cluster [DBG] pgmap v1317: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-02-23T00:45:35.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:34.083058+0000 mon.c (mon.2) 518 : audit [INF] from='client.? 172.21.15.87:0/2978962766' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:35.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:34.083396+0000 mon.a (mon.0) 4026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:35.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:33.407691+0000 osd.1 (osd.1) 430 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:45:35.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:33.410631+0000 osd.1 (osd.1) 431 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:45:35.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:33.462794+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:45:35.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:33.464721+0000 osd.4 (osd.4) 382 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:45:35.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:33.833683+0000 mgr.y (mgr.24437) 734 : cluster [DBG] pgmap v1317: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-02-23T00:45:35.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:34.083058+0000 mon.c (mon.2) 518 : audit [INF] from='client.? 172.21.15.87:0/2978962766' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:35.206 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:34.083396+0000 mon.a (mon.0) 4026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:35.206 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:33.407691+0000 osd.1 (osd.1) 430 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:45:35.206 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:33.410631+0000 osd.1 (osd.1) 431 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:45:35.207 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:33.462794+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.1a scrub starts 2024-02-23T00:45:35.207 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:33.464721+0000 osd.4 (osd.4) 382 : cluster [DBG] 4.1a scrub ok 2024-02-23T00:45:35.207 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:33.833683+0000 mgr.y (mgr.24437) 734 : cluster [DBG] pgmap v1317: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-02-23T00:45:35.207 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:34.083058+0000 mon.c (mon.2) 518 : audit [INF] from='client.? 172.21.15.87:0/2978962766' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:35.207 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:34.083396+0000 mon.a (mon.0) 4026 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-23T00:45:35.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd reweight 0 1 2024-02-23T00:45:35.927 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:36.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:34.836481+0000 mon.a (mon.0) 4027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-23T00:45:36.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:34.836564+0000 mon.a (mon.0) 4028 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-02-23T00:45:36.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:35.171203+0000 osd.2 (osd.2) 323 : cluster [DBG] 2.e scrub starts 2024-02-23T00:45:36.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:35.172241+0000 osd.2 (osd.2) 324 : cluster [DBG] 2.e scrub ok 2024-02-23T00:45:36.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:34.836481+0000 mon.a (mon.0) 4027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-23T00:45:36.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:34.836564+0000 mon.a (mon.0) 4028 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-02-23T00:45:36.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:35.171203+0000 osd.2 (osd.2) 323 : cluster [DBG] 2.e scrub starts 2024-02-23T00:45:36.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:35.172241+0000 osd.2 (osd.2) 324 : cluster [DBG] 2.e scrub ok 2024-02-23T00:45:36.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:34.836481+0000 mon.a (mon.0) 4027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-23T00:45:36.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:34.836564+0000 mon.a (mon.0) 4028 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-02-23T00:45:36.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:35.171203+0000 osd.2 (osd.2) 323 : cluster [DBG] 2.e scrub starts 2024-02-23T00:45:36.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:35.172241+0000 osd.2 (osd.2) 324 : cluster [DBG] 2.e scrub ok 2024-02-23T00:45:37.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:35.757314+0000 osd.7 (osd.7) 422 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:45:37.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:35.760229+0000 osd.7 (osd.7) 423 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:45:37.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:35.834390+0000 mgr.y (mgr.24437) 735 : cluster [DBG] pgmap v1319: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:37.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:35.757314+0000 osd.7 (osd.7) 422 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:45:37.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:35.760229+0000 osd.7 (osd.7) 423 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:45:37.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:35.834390+0000 mgr.y (mgr.24437) 735 : cluster [DBG] pgmap v1319: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:37.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:35.757314+0000 osd.7 (osd.7) 422 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:45:37.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:35.760229+0000 osd.7 (osd.7) 423 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:45:37.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:35.834390+0000 mgr.y (mgr.24437) 735 : cluster [DBG] pgmap v1319: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:37.941 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.938+0000 7f1850719700 1 -- 172.21.15.87:0/3234804781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fe510 msgr2=0x7f18480f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.938+0000 7f1850719700 1 --2- 172.21.15.87:0/3234804781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fe510 0x7f18480f7d40 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 rx=0x7f18440047b0 tx=0x7f1844009cb0).stop 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.938+0000 7f1850719700 1 -- 172.21.15.87:0/3234804781 shutdown_connections 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.938+0000 7f1850719700 1 --2- 172.21.15.87:0/3234804781 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18480f8420 0x7f1848104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.938+0000 7f1850719700 1 --2- 172.21.15.87:0/3234804781 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18480fdbb0 0x7f18480fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.938+0000 7f1850719700 1 --2- 172.21.15.87:0/3234804781 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fe510 0x7f18480f7d40 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.938+0000 7f1850719700 1 -- 172.21.15.87:0/3234804781 >> 172.21.15.87:0/3234804781 conn(0x7f18480f3800 msgr2=0x7f18480f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.938+0000 7f1850719700 1 -- 172.21.15.87:0/3234804781 shutdown_connections 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.938+0000 7f1850719700 1 -- 172.21.15.87:0/3234804781 wait complete. 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 librados: starting msgr at 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 librados: starting objecter 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 Processor -- start 2024-02-23T00:45:37.942 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 -- start start 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 librados: setting wanted keys 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 librados: calling monclient init 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18480f8420 0x7f184810e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fdbb0 0x7f184810e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18480fe510 0x7f1848108230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1848108720 con 0x7f18480fdbb0 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1848108890 con 0x7f18480f8420 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f1850719700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1848108a00 con 0x7f18480fe510 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184dcb4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fdbb0 0x7f184810e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184ecb6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18480fe510 0x7f1848108230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184dcb4700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fdbb0 0x7f184810e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:34974/0 (socket says 172.21.15.87:34974) 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184ecb6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18480fe510 0x7f1848108230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:38720/0 (socket says 172.21.15.87:38720) 2024-02-23T00:45:37.943 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184dcb4700 1 -- 172.21.15.87:0/2736599096 learned_addr learned my addr 172.21.15.87:0/2736599096 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184e4b5700 1 --2- 172.21.15.87:0/2736599096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18480f8420 0x7f184810e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184ecb6700 1 -- 172.21.15.87:0/2736599096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18480f8420 msgr2=0x7f184810e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184ecb6700 1 --2- 172.21.15.87:0/2736599096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18480f8420 0x7f184810e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184ecb6700 1 -- 172.21.15.87:0/2736599096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fdbb0 msgr2=0x7f184810e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184ecb6700 1 --2- 172.21.15.87:0/2736599096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fdbb0 0x7f184810e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184ecb6700 1 -- 172.21.15.87:0/2736599096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18400097c0 con 0x7f18480fe510 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184dcb4700 1 --2- 172.21.15.87:0/2736599096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fdbb0 0x7f184810e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f184ecb6700 1 --2- 172.21.15.87:0/2736599096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18480fe510 0x7f1848108230 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f1840001df0 tx=0x7f1840004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f183f7fe700 10 client.?.objecter ms_handle_connect 0x7f18480fe510 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.939+0000 7f183f7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.940+0000 7f183f7fe700 1 -- 172.21.15.87:0/2736599096 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18400099f0 con 0x7f18480fe510 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.940+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1844005040 con 0x7f18480fe510 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.940+0000 7f183f7fe700 1 -- 172.21.15.87:0/2736599096 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18400090d0 con 0x7f18480fe510 2024-02-23T00:45:37.944 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.940+0000 7f183f7fe700 1 -- 172.21.15.87:0/2736599096 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1840007c60 con 0x7f18480fe510 2024-02-23T00:45:37.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.940+0000 7f1850719700 10 client.38549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:37.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.940+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1848108ff0 con 0x7f18480fe510 2024-02-23T00:45:37.945 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.940+0000 7f1850719700 1 librados: init done 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.941+0000 7f183f7fe700 1 -- 172.21.15.87:0/2736599096 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f184000fad0 con 0x7f18480fe510 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.941+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1848107110 con 0x7f18480fe510 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.941+0000 7f183f7fe700 1 --2- 172.21.15.87:0/2736599096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1834062c90 0x7f1834065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.941+0000 7f183f7fe700 1 -- 172.21.15.87:0/2736599096 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(845..845 src has 400..845) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f184007d7d0 con 0x7f18480fe510 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.941+0000 7f183f7fe700 10 client.38549.objecter ms_dispatch 0x7f1848001550 osd_map(845..845 src has 400..845) v4 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.941+0000 7f183f7fe700 3 client.38549.objecter handle_osd_map got epochs [845,845] > 0 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.941+0000 7f183f7fe700 3 client.38549.objecter handle_osd_map decoding full epoch 845 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.941+0000 7f184e4b5700 1 --2- 172.21.15.87:0/2736599096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1834062c90 0x7f1834065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.941+0000 7f183f7fe700 20 client.38549.objecter dump_active .. 0 homeless 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.942+0000 7f183f7fe700 10 client.38549.objecter ms_handle_connect 0x7f1834062c90 2024-02-23T00:45:37.946 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.942+0000 7f184e4b5700 1 --2- 172.21.15.87:0/2736599096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1834062c90 0x7f1834065150 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7f18380041d0 tx=0x7f1838009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:37.947 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:37.946+0000 7f183f7fe700 1 -- 172.21.15.87:0/2736599096 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1840050f20 con 0x7f18480fe510 2024-02-23T00:45:38.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:36.389601+0000 osd.1 (osd.1) 432 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:45:38.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:36.392936+0000 osd.1 (osd.1) 433 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:45:38.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:37.267086+0000 osd.6 (osd.6) 382 : cluster [DBG] 4.a deep-scrub starts 2024-02-23T00:45:38.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:37.271117+0000 osd.6 (osd.6) 383 : cluster [DBG] 4.a deep-scrub ok 2024-02-23T00:45:38.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:37.289991+0000 mon.a (mon.0) 4029 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-02-23T00:45:38.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:36.389601+0000 osd.1 (osd.1) 432 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:45:38.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:36.392936+0000 osd.1 (osd.1) 433 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:45:38.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:37.267086+0000 osd.6 (osd.6) 382 : cluster [DBG] 4.a deep-scrub starts 2024-02-23T00:45:38.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:37.271117+0000 osd.6 (osd.6) 383 : cluster [DBG] 4.a deep-scrub ok 2024-02-23T00:45:38.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:37.289991+0000 mon.a (mon.0) 4029 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-02-23T00:45:38.124 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.122+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7f184805fa60 con 0x7f18480fe510 2024-02-23T00:45:38.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:36.389601+0000 osd.1 (osd.1) 432 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:45:38.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:36.392936+0000 osd.1 (osd.1) 433 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:45:38.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:37.267086+0000 osd.6 (osd.6) 382 : cluster [DBG] 4.a deep-scrub starts 2024-02-23T00:45:38.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:37.271117+0000 osd.6 (osd.6) 383 : cluster [DBG] 4.a deep-scrub ok 2024-02-23T00:45:38.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:37.289991+0000 mon.a (mon.0) 4029 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-02-23T00:45:38.897 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.895+0000 7f183f7fe700 1 -- 172.21.15.87:0/2736599096 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v846) v1 ==== 113+0+0 (secure 0 0 0) 0x7f1840050b10 con 0x7f18480fe510 2024-02-23T00:45:38.899 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.897+0000 7f1850719700 10 librados: watch_flush enter 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.897+0000 7f1850719700 10 librados: watch_flush exit 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.897+0000 7f1850719700 20 client.38549.objecter shutdown clearing up homeless session... 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.897+0000 7f1850719700 10 client.38549.objecter successfully canceled tick 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1834062c90 msgr2=0x7f1834065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 --2- 172.21.15.87:0/2736599096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1834062c90 0x7f1834065150 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 rx=0x7f18380041d0 tx=0x7f1838009040).stop 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18480fe510 msgr2=0x7f1848108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 --2- 172.21.15.87:0/2736599096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18480fe510 0x7f1848108230 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 rx=0x7f1840001df0 tx=0x7f1840004d10).stop 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 shutdown_connections 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 --2- 172.21.15.87:0/2736599096 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1834062c90 0x7f1834065150 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 --2- 172.21.15.87:0/2736599096 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f18480fe510 0x7f1848108230 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 --2- 172.21.15.87:0/2736599096 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f18480f8420 0x7f184810e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 --2- 172.21.15.87:0/2736599096 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f18480fdbb0 0x7f184810e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:38.900 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 >> 172.21.15.87:0/2736599096 conn(0x7f18480f3800 msgr2=0x7f18480f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:38.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 shutdown_connections 2024-02-23T00:45:38.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 -- 172.21.15.87:0/2736599096 wait complete. 2024-02-23T00:45:38.901 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:38.898+0000 7f1850719700 1 librados: shutdown 2024-02-23T00:45:38.901 INFO:teuthology.orchestra.run.smithi087.stderr:reweighted osd.0 to 1 (10000) 2024-02-23T00:45:39.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:37.834745+0000 mgr.y (mgr.24437) 736 : cluster [DBG] pgmap v1321: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:39.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:38.123334+0000 mon.c (mon.2) 519 : audit [INF] from='client.? 172.21.15.87:0/2736599096' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:39.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:38.123691+0000 mon.a (mon.0) 4030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:39.306 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:37.834745+0000 mgr.y (mgr.24437) 736 : cluster [DBG] pgmap v1321: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:39.306 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:38.123334+0000 mon.c (mon.2) 519 : audit [INF] from='client.? 172.21.15.87:0/2736599096' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:39.306 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:38.123691+0000 mon.a (mon.0) 4030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:39.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:37.834745+0000 mgr.y (mgr.24437) 736 : cluster [DBG] pgmap v1321: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:39.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:38.123334+0000 mon.c (mon.2) 519 : audit [INF] from='client.? 172.21.15.87:0/2736599096' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:39.306 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:38.123691+0000 mon.a (mon.0) 4030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:39.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-23T00:45:39.995 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:40.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:38.894551+0000 mon.a (mon.0) 4031 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-23T00:45:40.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:38.894644+0000 mon.a (mon.0) 4032 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-02-23T00:45:40.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:38.894551+0000 mon.a (mon.0) 4031 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-23T00:45:40.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:38.894644+0000 mon.a (mon.0) 4032 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-02-23T00:45:40.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:38.894551+0000 mon.a (mon.0) 4031 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-23T00:45:40.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:38.894644+0000 mon.a (mon.0) 4032 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-02-23T00:45:41.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:39.835099+0000 mgr.y (mgr.24437) 737 : cluster [DBG] pgmap v1323: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:41.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:39.835099+0000 mgr.y (mgr.24437) 737 : cluster [DBG] pgmap v1323: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:41.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:39.835099+0000 mgr.y (mgr.24437) 737 : cluster [DBG] pgmap v1323: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-23T00:45:41.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:45:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:45:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 -- 172.21.15.87:0/1180884580 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 msgr2=0x7fc678104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/1180884580 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 0x7fc678104660 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 rx=0x7fc670009c30 tx=0x7fc670004c50).stop 2024-02-23T00:45:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 -- 172.21.15.87:0/1180884580 shutdown_connections 2024-02-23T00:45:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/1180884580 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 0x7fc678104660 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:42.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/1180884580 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc6780f8d10 0x7fc6780f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/1180884580 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6780f9670 0x7fc678101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 -- 172.21.15.87:0/1180884580 >> 172.21.15.87:0/1180884580 conn(0x7fc6780f2820 msgr2=0x7fc6780f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 -- 172.21.15.87:0/1180884580 shutdown_connections 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 -- 172.21.15.87:0/1180884580 wait complete. 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 librados: starting msgr at 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.040+0000 7fc67f4fe700 1 librados: starting objecter 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 Processor -- start 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 -- start start 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 librados: setting wanted keys 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 librados: calling monclient init 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc6780f8d10 0x7fc6780f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6780f9670 0x7fc6780f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:42.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 0x7fc6780f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6781165d0 con 0x7fc678102270 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc678116740 con 0x7fc6780f9670 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67f4fe700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc678116900 con 0x7fc6780f8d10 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67d29a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc6780f8d10 0x7fc6780f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67da9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 0x7fc6780f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67d29a700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc6780f8d10 0x7fc6780f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:47892/0 (socket says 172.21.15.87:47892) 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67da9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 0x7fc6780f76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40304/0 (socket says 172.21.15.87:40304) 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67d29a700 1 -- 172.21.15.87:0/2077845010 learned_addr learned my addr 172.21.15.87:0/2077845010 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67ca99700 1 --2- 172.21.15.87:0/2077845010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6780f9670 0x7fc6780f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67d29a700 1 -- 172.21.15.87:0/2077845010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6780f9670 msgr2=0x7fc6780f71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67d29a700 1 --2- 172.21.15.87:0/2077845010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6780f9670 0x7fc6780f71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67d29a700 1 -- 172.21.15.87:0/2077845010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 msgr2=0x7fc6780f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67d29a700 1 --2- 172.21.15.87:0/2077845010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 0x7fc6780f76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:42.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67d29a700 1 -- 172.21.15.87:0/2077845010 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668005040 con 0x7fc6780f8d10 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67da9b700 1 --2- 172.21.15.87:0/2077845010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 0x7fc6780f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc67d29a700 1 --2- 172.21.15.87:0/2077845010 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc6780f8d10 0x7fc6780f6cc0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7fc668009400 tx=0x7fc668009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc6767fc700 10 client.?.objecter ms_handle_connect 0x7fc6780f8d10 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.041+0000 7fc6767fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.042+0000 7fc6767fc700 1 -- 172.21.15.87:0/2077845010 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668007a40 con 0x7fc6780f8d10 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.042+0000 7fc6767fc700 1 -- 172.21.15.87:0/2077845010 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc66800f040 con 0x7fc6780f8d10 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.042+0000 7fc6767fc700 1 -- 172.21.15.87:0/2077845010 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668014aa0 con 0x7fc6780f8d10 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.042+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6700097c0 con 0x7fc6780f8d10 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.042+0000 7fc67f4fe700 10 client.38555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.042+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc678116ec0 con 0x7fc6780f8d10 2024-02-23T00:45:42.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.042+0000 7fc67f4fe700 1 librados: init done 2024-02-23T00:45:42.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.043+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc67805fa60 con 0x7fc6780f8d10 2024-02-23T00:45:42.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.043+0000 7fc6767fc700 1 -- 172.21.15.87:0/2077845010 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc66800e630 con 0x7fc6780f8d10 2024-02-23T00:45:42.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.043+0000 7fc6767fc700 1 --2- 172.21.15.87:0/2077845010 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc660062c90 0x7fc660065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:42.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.043+0000 7fc6767fc700 1 -- 172.21.15.87:0/2077845010 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(847..847 src has 400..847) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fc66807cbb0 con 0x7fc6780f8d10 2024-02-23T00:45:42.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.043+0000 7fc6767fc700 10 client.38555.objecter ms_dispatch 0x7fc678001550 osd_map(847..847 src has 400..847) v4 2024-02-23T00:45:42.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.043+0000 7fc67ca99700 1 --2- 172.21.15.87:0/2077845010 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc660062c90 0x7fc660065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:42.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.043+0000 7fc6767fc700 3 client.38555.objecter handle_osd_map got epochs [847,847] > 0 2024-02-23T00:45:42.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.043+0000 7fc6767fc700 3 client.38555.objecter handle_osd_map decoding full epoch 847 2024-02-23T00:45:42.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.044+0000 7fc6767fc700 20 client.38555.objecter dump_active .. 0 homeless 2024-02-23T00:45:42.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.044+0000 7fc67ca99700 1 --2- 172.21.15.87:0/2077845010 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc660062c90 0x7fc660065150 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7fc6780f8460 tx=0x7fc66c008040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:42.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.044+0000 7fc6767fc700 10 client.38555.objecter ms_handle_connect 0x7fc660062c90 2024-02-23T00:45:42.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.048+0000 7fc6767fc700 1 -- 172.21.15.87:0/2077845010 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc668049f60 con 0x7fc6780f8d10 2024-02-23T00:45:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:40.435724+0000 osd.5 (osd.5) 361 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:40.436148+0000 osd.5 (osd.5) 362 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:40.912559+0000 mon.a (mon.0) 4033 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-02-23T00:45:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:41.229500+0000 osd.4 (osd.4) 383 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:45:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:41.232080+0000 osd.4 (osd.4) 384 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:45:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:41.309254+0000 osd.3 (osd.3) 325 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:45:42.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:41.312413+0000 osd.3 (osd.3) 326 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:45:42.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:41.416727+0000 mgr.y (mgr.24437) 738 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:40.435724+0000 osd.5 (osd.5) 361 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:40.436148+0000 osd.5 (osd.5) 362 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:40.912559+0000 mon.a (mon.0) 4033 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-02-23T00:45:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:41.229500+0000 osd.4 (osd.4) 383 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:45:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:41.232080+0000 osd.4 (osd.4) 384 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:45:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:41.309254+0000 osd.3 (osd.3) 325 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:45:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:41.312413+0000 osd.3 (osd.3) 326 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:45:42.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:41.416727+0000 mgr.y (mgr.24437) 738 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:42.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:40.435724+0000 osd.5 (osd.5) 361 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:42.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:40.436148+0000 osd.5 (osd.5) 362 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:42.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:40.912559+0000 mon.a (mon.0) 4033 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-02-23T00:45:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:41.229500+0000 osd.4 (osd.4) 383 : cluster [DBG] 5.18 scrub starts 2024-02-23T00:45:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:41.232080+0000 osd.4 (osd.4) 384 : cluster [DBG] 5.18 scrub ok 2024-02-23T00:45:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:41.309254+0000 osd.3 (osd.3) 325 : cluster [DBG] 3.2 scrub starts 2024-02-23T00:45:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:41.312413+0000 osd.3 (osd.3) 326 : cluster [DBG] 3.2 scrub ok 2024-02-23T00:45:42.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:41.416727+0000 mgr.y (mgr.24437) 738 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:42.225 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.223+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fc6780f81a0 con 0x7fc6780f8d10 2024-02-23T00:45:42.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:42] "GET /metrics HTTP/1.1" 200 186061 "" "Prometheus/2.33.4" 2024-02-23T00:45:42.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.924+0000 7fc6767fc700 1 -- 172.21.15.87:0/2077845010 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v848) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc668050300 con 0x7fc6780f8d10 2024-02-23T00:45:42.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.926+0000 7fc67f4fe700 10 librados: watch_flush enter 2024-02-23T00:45:42.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 10 librados: watch_flush exit 2024-02-23T00:45:42.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 20 client.38555.objecter shutdown clearing up homeless session... 2024-02-23T00:45:42.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 10 client.38555.objecter successfully canceled tick 2024-02-23T00:45:42.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc660062c90 msgr2=0x7fc660065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:42.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/2077845010 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc660062c90 0x7fc660065150 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 rx=0x7fc6780f8460 tx=0x7fc66c008040).stop 2024-02-23T00:45:42.928 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc6780f8d10 msgr2=0x7fc6780f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:42.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/2077845010 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc6780f8d10 0x7fc6780f6cc0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 rx=0x7fc668009400 tx=0x7fc668009cb0).stop 2024-02-23T00:45:42.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 shutdown_connections 2024-02-23T00:45:42.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/2077845010 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc660062c90 0x7fc660065150 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:42.929 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/2077845010 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc678102270 0x7fc6780f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:42.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/2077845010 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc6780f8d10 0x7fc6780f6cc0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:42.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 --2- 172.21.15.87:0/2077845010 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc6780f9670 0x7fc6780f71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:42.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.927+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 >> 172.21.15.87:0/2077845010 conn(0x7fc6780f2820 msgr2=0x7fc6780f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:42.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.928+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 shutdown_connections 2024-02-23T00:45:42.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.928+0000 7fc67f4fe700 1 -- 172.21.15.87:0/2077845010 wait complete. 2024-02-23T00:45:42.930 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:42.928+0000 7fc67f4fe700 1 librados: shutdown 2024-02-23T00:45:42.930 INFO:teuthology.orchestra.run.smithi087.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-23T00:45:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:41.835575+0000 mgr.y (mgr.24437) 739 : cluster [DBG] pgmap v1325: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:43.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:42.220492+0000 osd.4 (osd.4) 385 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:45:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:42.221472+0000 osd.4 (osd.4) 386 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:45:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:42.224229+0000 mon.c (mon.2) 520 : audit [INF] from='client.? 172.21.15.87:0/2077845010' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:42.224528+0000 mon.a (mon.0) 4034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:42.325975+0000 osd.3 (osd.3) 327 : cluster [DBG] 6.b scrub starts 2024-02-23T00:45:43.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:42.329287+0000 osd.3 (osd.3) 328 : cluster [DBG] 6.b scrub ok 2024-02-23T00:45:43.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:41.835575+0000 mgr.y (mgr.24437) 739 : cluster [DBG] pgmap v1325: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:43.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:42.220492+0000 osd.4 (osd.4) 385 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:45:43.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:42.221472+0000 osd.4 (osd.4) 386 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:45:43.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:42.224229+0000 mon.c (mon.2) 520 : audit [INF] from='client.? 172.21.15.87:0/2077845010' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:43.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:42.224528+0000 mon.a (mon.0) 4034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:43.297 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:42.325975+0000 osd.3 (osd.3) 327 : cluster [DBG] 6.b scrub starts 2024-02-23T00:45:43.298 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:42.329287+0000 osd.3 (osd.3) 328 : cluster [DBG] 6.b scrub ok 2024-02-23T00:45:43.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:41.835575+0000 mgr.y (mgr.24437) 739 : cluster [DBG] pgmap v1325: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:43.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:42.220492+0000 osd.4 (osd.4) 385 : cluster [DBG] 3.9 scrub starts 2024-02-23T00:45:43.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:42.221472+0000 osd.4 (osd.4) 386 : cluster [DBG] 3.9 scrub ok 2024-02-23T00:45:43.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:42.224229+0000 mon.c (mon.2) 520 : audit [INF] from='client.? 172.21.15.87:0/2077845010' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:43.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:42.224528+0000 mon.a (mon.0) 4034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-23T00:45:43.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:42.325975+0000 osd.3 (osd.3) 327 : cluster [DBG] 6.b scrub starts 2024-02-23T00:45:43.298 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:42.329287+0000 osd.3 (osd.3) 328 : cluster [DBG] 6.b scrub ok 2024-02-23T00:45:43.751 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-23T00:45:43.751 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-23T00:45:43.751 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-23T00:45:43.751 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-23T00:45:43.751 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-23T00:45:43.752 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump --format=json 2024-02-23T00:45:44.026 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:44.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:42.367266+0000 osd.1 (osd.1) 434 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:45:44.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:42.370756+0000 osd.1 (osd.1) 435 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:45:44.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:42.752328+0000 osd.7 (osd.7) 424 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:45:44.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:42.753237+0000 osd.7 (osd.7) 425 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:45:44.083 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:42.924102+0000 mon.a (mon.0) 4035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:42.924159+0000 mon.a (mon.0) 4036 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:43.262173+0000 osd.2 (osd.2) 325 : cluster [DBG] 5.a deep-scrub starts 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:43.263304+0000 osd.2 (osd.2) 326 : cluster [DBG] 5.a deep-scrub ok 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:42.367266+0000 osd.1 (osd.1) 434 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:42.370756+0000 osd.1 (osd.1) 435 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:42.752328+0000 osd.7 (osd.7) 424 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:42.753237+0000 osd.7 (osd.7) 425 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:42.924102+0000 mon.a (mon.0) 4035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:42.924159+0000 mon.a (mon.0) 4036 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:43.262173+0000 osd.2 (osd.2) 325 : cluster [DBG] 5.a deep-scrub starts 2024-02-23T00:45:44.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:43.263304+0000 osd.2 (osd.2) 326 : cluster [DBG] 5.a deep-scrub ok 2024-02-23T00:45:44.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:45:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:45:44.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:42.367266+0000 osd.1 (osd.1) 434 : cluster [DBG] 6.4 scrub starts 2024-02-23T00:45:44.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:42.370756+0000 osd.1 (osd.1) 435 : cluster [DBG] 6.4 scrub ok 2024-02-23T00:45:44.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:42.752328+0000 osd.7 (osd.7) 424 : cluster [DBG] 5.1a scrub starts 2024-02-23T00:45:44.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:42.753237+0000 osd.7 (osd.7) 425 : cluster [DBG] 5.1a scrub ok 2024-02-23T00:45:44.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:42.924102+0000 mon.a (mon.0) 4035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-23T00:45:44.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:42.924159+0000 mon.a (mon.0) 4036 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-02-23T00:45:44.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:43.262173+0000 osd.2 (osd.2) 325 : cluster [DBG] 5.a deep-scrub starts 2024-02-23T00:45:44.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:43.263304+0000 osd.2 (osd.2) 326 : cluster [DBG] 5.a deep-scrub ok 2024-02-23T00:45:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:43.836031+0000 mgr.y (mgr.24437) 740 : cluster [DBG] pgmap v1327: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:45.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:44.198381+0000 osd.4 (osd.4) 387 : cluster [DBG] 22.1e deep-scrub starts 2024-02-23T00:45:45.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:44.201805+0000 osd.4 (osd.4) 388 : cluster [DBG] 22.1e deep-scrub ok 2024-02-23T00:45:45.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:44.208676+0000 osd.6 (osd.6) 384 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:45:45.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:44.212051+0000 osd.6 (osd.6) 385 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:45:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:43.836031+0000 mgr.y (mgr.24437) 740 : cluster [DBG] pgmap v1327: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:45.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:44.198381+0000 osd.4 (osd.4) 387 : cluster [DBG] 22.1e deep-scrub starts 2024-02-23T00:45:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:44.201805+0000 osd.4 (osd.4) 388 : cluster [DBG] 22.1e deep-scrub ok 2024-02-23T00:45:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:44.208676+0000 osd.6 (osd.6) 384 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:45:45.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:44.212051+0000 osd.6 (osd.6) 385 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:45:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:43.836031+0000 mgr.y (mgr.24437) 740 : cluster [DBG] pgmap v1327: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:45:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:44.198381+0000 osd.4 (osd.4) 387 : cluster [DBG] 22.1e deep-scrub starts 2024-02-23T00:45:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:44.201805+0000 osd.4 (osd.4) 388 : cluster [DBG] 22.1e deep-scrub ok 2024-02-23T00:45:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:44.208676+0000 osd.6 (osd.6) 384 : cluster [DBG] 22.19 scrub starts 2024-02-23T00:45:45.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:44.212051+0000 osd.6 (osd.6) 385 : cluster [DBG] 22.19 scrub ok 2024-02-23T00:45:46.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:44.730783+0000 osd.7 (osd.7) 426 : cluster [DBG] 22.c scrub starts 2024-02-23T00:45:46.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:44.731866+0000 osd.7 (osd.7) 427 : cluster [DBG] 22.c scrub ok 2024-02-23T00:45:46.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 -- 172.21.15.87:0/4059520095 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280fcbb0 msgr2=0x7faa280fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:46.245 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 --2- 172.21.15.87:0/4059520095 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280fcbb0 0x7faa280fcf90 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 rx=0x7faa1c0047b0 tx=0x7faa1c009cb0).stop 2024-02-23T00:45:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 -- 172.21.15.87:0/4059520095 shutdown_connections 2024-02-23T00:45:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 --2- 172.21.15.87:0/4059520095 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa280f7400 0x7faa28103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:45:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 --2- 172.21.15.87:0/4059520095 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280fcbb0 0x7faa280fcf90 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 --2- 172.21.15.87:0/4059520095 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa280fd510 0x7faa280f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 -- 172.21.15.87:0/4059520095 >> 172.21.15.87:0/4059520095 conn(0x7faa280f2820 msgr2=0x7faa280f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 -- 172.21.15.87:0/4059520095 shutdown_connections 2024-02-23T00:45:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 -- 172.21.15.87:0/4059520095 wait complete. 2024-02-23T00:45:46.246 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 librados: starting msgr at 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.242+0000 7faa2f044700 1 librados: starting objecter 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 Processor -- start 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 -- start start 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 librados: setting wanted keys 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 librados: calling monclient init 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280f7400 0x7faa2810d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa280fcbb0 0x7faa2810d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa280fd510 0x7faa28107390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28107880 con 0x7faa280fd510 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa281079f0 con 0x7faa280fcbb0 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2f044700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa28107bb0 con 0x7faa280f7400 2024-02-23T00:45:46.247 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2cde0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280f7400 0x7faa2810d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2d5e1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa280fd510 0x7faa28107390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2cde0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280f7400 0x7faa2810d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:47916/0 (socket says 172.21.15.87:47916) 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2d5e1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa280fd510 0x7faa28107390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:40328/0 (socket says 172.21.15.87:40328) 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2cde0700 1 -- 172.21.15.87:0/2684358298 learned_addr learned my addr 172.21.15.87:0/2684358298 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa27fff700 1 --2- 172.21.15.87:0/2684358298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa280fcbb0 0x7faa2810d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2cde0700 1 -- 172.21.15.87:0/2684358298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa280fcbb0 msgr2=0x7faa2810d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2cde0700 1 --2- 172.21.15.87:0/2684358298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa280fcbb0 0x7faa2810d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2cde0700 1 -- 172.21.15.87:0/2684358298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa280fd510 msgr2=0x7faa28107390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2cde0700 1 --2- 172.21.15.87:0/2684358298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa280fd510 0x7faa28107390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2cde0700 1 -- 172.21.15.87:0/2684358298 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14005040 con 0x7faa280f7400 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2d5e1700 1 --2- 172.21.15.87:0/2684358298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa280fd510 0x7faa28107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:46.248 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa2cde0700 1 --2- 172.21.15.87:0/2684358298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280f7400 0x7faa2810d1f0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 rx=0x7faa1c009b10 tx=0x7faa1c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:46.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa25ffb700 10 client.?.objecter ms_handle_connect 0x7faa280f7400 2024-02-23T00:45:46.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa25ffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:46.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.243+0000 7faa25ffb700 1 -- 172.21.15.87:0/2684358298 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c018070 con 0x7faa280f7400 2024-02-23T00:45:46.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.244+0000 7faa25ffb700 1 -- 172.21.15.87:0/2684358298 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa1c013e70 con 0x7faa280f7400 2024-02-23T00:45:46.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.244+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c005040 con 0x7faa280f7400 2024-02-23T00:45:46.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.244+0000 7faa25ffb700 1 -- 172.21.15.87:0/2684358298 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1c01ddb0 con 0x7faa280f7400 2024-02-23T00:45:46.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.244+0000 7faa2f044700 10 client.38561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:46.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.244+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa28108050 con 0x7faa280f7400 2024-02-23T00:45:46.249 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.244+0000 7faa2f044700 1 librados: init done 2024-02-23T00:45:46.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.245+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2805fa60 con 0x7faa280f7400 2024-02-23T00:45:46.250 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.245+0000 7faa25ffb700 1 -- 172.21.15.87:0/2684358298 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7faa1c01d420 con 0x7faa280f7400 2024-02-23T00:45:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.245+0000 7faa25ffb700 1 --2- 172.21.15.87:0/2684358298 >> v2:172.21.15.87:6800/1231597463 conn(0x7faa18062c90 0x7faa18065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.245+0000 7faa25ffb700 1 -- 172.21.15.87:0/2684358298 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(848..848 src has 400..848) v4 ==== 7486+0+0 (secure 0 0 0) 0x7faa1c07e300 con 0x7faa280f7400 2024-02-23T00:45:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.245+0000 7faa25ffb700 10 client.38561.objecter ms_dispatch 0x7faa28001550 osd_map(848..848 src has 400..848) v4 2024-02-23T00:45:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.245+0000 7faa25ffb700 3 client.38561.objecter handle_osd_map got epochs [848,848] > 0 2024-02-23T00:45:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.245+0000 7faa25ffb700 3 client.38561.objecter handle_osd_map decoding full epoch 848 2024-02-23T00:45:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.245+0000 7faa27fff700 1 --2- 172.21.15.87:0/2684358298 >> v2:172.21.15.87:6800/1231597463 conn(0x7faa18062c90 0x7faa18065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.245+0000 7faa25ffb700 20 client.38561.objecter dump_active .. 0 homeless 2024-02-23T00:45:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.246+0000 7faa25ffb700 10 client.38561.objecter ms_handle_connect 0x7faa18062c90 2024-02-23T00:45:46.251 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.246+0000 7faa27fff700 1 --2- 172.21.15.87:0/2684358298 >> v2:172.21.15.87:6800/1231597463 conn(0x7faa18062c90 0x7faa18065150 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 rx=0x7faa28108b30 tx=0x7faa14009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:46.252 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.250+0000 7faa25ffb700 1 -- 172.21.15.87:0/2684358298 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7faa1c051af0 con 0x7faa280f7400 2024-02-23T00:45:46.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:44.730783+0000 osd.7 (osd.7) 426 : cluster [DBG] 22.c scrub starts 2024-02-23T00:45:46.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:44.731866+0000 osd.7 (osd.7) 427 : cluster [DBG] 22.c scrub ok 2024-02-23T00:45:46.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:44.730783+0000 osd.7 (osd.7) 426 : cluster [DBG] 22.c scrub starts 2024-02-23T00:45:46.333 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:44.731866+0000 osd.7 (osd.7) 427 : cluster [DBG] 22.c scrub ok 2024-02-23T00:45:46.418 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.417+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa28106120 con 0x7faa280f7400 2024-02-23T00:45:46.419 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.418+0000 7faa25ffb700 1 -- 172.21.15.87:0/2684358298 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v848) v1 ==== 74+0+22226 (secure 0 0 0) 0x7faa1c0516e0 con 0x7faa280f7400 2024-02-23T00:45:46.420 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:45:46.420 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":848,"fsid":"66db207c-d1e1-11ee-95c0-87774f69a715","created":"2024-02-23T00:21:28.861318+0000","modified":"2024-02-23T00:45:42.919335+0000","last_up_change":"2024-02-23T00:41:18.209057+0000","last_in_change":"2024-02-23T00:41:36.208573+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":115,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":256,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-23T00:25:09.208309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-23T00:27:15.562186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-23T00:27:15.989206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":23,"pg_placement_num":23,"pg_placement_num_target":23,"pg_num_target":23,"pg_num_pending":23,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"816","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"736","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-23T00:27:17.282440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"4.16","ready_epoch":371,"last_epoch_started":370,"last_epoch_clean":370,"source_version":"63'21","target_version":"62'12"},"last_change":"532","last_force_op_resend":"0","last_force_op_resend_prenautilus":"371","last_force_op_resend_preluminous":"481","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-23T00:27:19.262164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"5.20","ready_epoch":641,"last_epoch_started":640,"last_epoch_clean":640,"source_version":"562'12","target_version":"330'9"},"last_change":"812","last_force_op_resend":"0","last_force_op_resend_prenautilus":"641","last_force_op_resend_preluminous":"639","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-23T00:27:21.319951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":12,"pg_placement_num":12,"pg_placement_num_target":12,"pg_num_target":12,"pg_num_pending":12,"last_pg_merge_meta":{"source_pgid":"6.c","ready_epoch":665,"last_epoch_started":664,"last_epoch_clean":664,"source_version":"78'2","target_version":"0'0"},"last_change":"815","last_force_op_resend":"0","last_force_op_resend_prenautilus":"665","last_force_op_resend_preluminous":"663","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":22,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-02-23T00:30:46.071806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"e3549bad-2891-47b9-bf77-132cadc20bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":424,"up_from":659,"up_thru":804,"down_at":425,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":484081831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":484081831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":484081831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":484081831}]},"public_addr":"172.21.15.87:6801/484081831","cluster_addr":"172.21.15.87:6802/484081831","heartbeat_back_addr":"172.21.15.87:6804/484081831","heartbeat_front_addr":"172.21.15.87:6803/484081831","state":["exists","up"]},{"osd":1,"uuid":"85525ce8-38d5-4840-8057-547cb176c859","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":804,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":685354171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":685354171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":685354171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":685354171}]},"public_addr":"172.21.15.87:6805/685354171","cluster_addr":"172.21.15.87:6806/685354171","heartbeat_back_addr":"172.21.15.87:6808/685354171","heartbeat_front_addr":"172.21.15.87:6807/685354171","state":["exists","up"]},{"osd":2,"uuid":"48ea64b6-ed15-48d2-b0ed-aa4053938aaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":804,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3605394228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3605394228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3605394228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3605394228}]},"public_addr":"172.21.15.87:6809/3605394228","cluster_addr":"172.21.15.87:6810/3605394228","heartbeat_back_addr":"172.21.15.87:6812/3605394228","heartbeat_front_addr":"172.21.15.87:6811/3605394228","state":["exists","up"]},{"osd":3,"uuid":"cb8705aa-e114-4dfd-ac9d-ec12f85f877a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":810,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":128059790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":128059790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":128059790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":128059790}]},"public_addr":"172.21.15.87:6813/128059790","cluster_addr":"172.21.15.87:6814/128059790","heartbeat_back_addr":"172.21.15.87:6816/128059790","heartbeat_front_addr":"172.21.15.87:6815/128059790","state":["exists","up"]},{"osd":4,"uuid":"c3734a04-7343-4251-9b5b-bedb9e8b882c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4175844757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6801","nonce":4175844757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6803","nonce":4175844757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4175844757}]},"public_addr":"172.21.15.171:6800/4175844757","cluster_addr":"172.21.15.171:6801/4175844757","heartbeat_back_addr":"172.21.15.171:6803/4175844757","heartbeat_front_addr":"172.21.15.171:6802/4175844757","state":["exists","up"]},{"osd":5,"uuid":"fe86232d-96b9-4cbf-9cac-373286fa103d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":838,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2550013795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6805","nonce":2550013795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6807","nonce":2550013795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2550013795}]},"public_addr":"172.21.15.171:6804/2550013795","cluster_addr":"172.21.15.171:6805/2550013795","heartbeat_back_addr":"172.21.15.171:6807/2550013795","heartbeat_front_addr":"172.21.15.171:6806/2550013795","state":["exists","up"]},{"osd":6,"uuid":"bfc5dd69-5cfb-409f-985f-83ee0dd4dc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":804,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2509408498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2509408498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6811","nonce":2509408498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":2509408498}]},"public_addr":"172.21.15.171:6808/2509408498","cluster_addr":"172.21.15.171:6809/2509408498","heartbeat_back_addr":"172.21.15.171:6811/2509408498","heartbeat_front_addr":"172.21.15.171:6810/2509408498","state":["exists","up"]},{"osd":7,"uuid":"29ddcf97-369a-4a2c-b3e2-142e84ccf939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":804,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":373787000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":373787000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6815","nonce":373787000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":373787000}]},"public_addr":"172.21.15.171:6812/373787000","cluster_addr":"172.21.15.171:6813/373787000","heartbeat_back_addr":"172.21.15.171:6815/373787000","heartbeat_front_addr":"172.21.15.171:6814/373787000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:13.589844+0000","dead_epoch":424},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:00.390106+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:44:35.904756+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:09.278758+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:09.281229+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:40.436151+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:07.838219+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:06.791018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.7","mappings":[{"from":7,"to":1},{"from":6,"to":5}]},{"pgid":"2.18","mappings":[{"from":7,"to":1}]},{"pgid":"2.1c","mappings":[{"from":7,"to":4}]},{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"5.10","mappings":[{"from":7,"to":3}]},{"pgid":"5.19","mappings":[{"from":1,"to":6}]},{"pgid":"22.12","mappings":[{"from":7,"to":0}]},{"pgid":"22.1a","mappings":[{"from":7,"to":5}]},{"pgid":"22.1f","mappings":[{"from":7,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:6800/1121018082":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/2138166229":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/27794119":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/1675126043":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/3767016525":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/969918521":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/795095363":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2273151327":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/1685406414":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2093733106":"2024-02-24T00:21:58.144574+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":21,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":61,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":63,"snaps":[{"begin":29,"length":1}]},{"pool":78,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":86,"snaps":[{"begin":2,"length":1}]},{"pool":93,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":95,"snaps":[{"begin":4,"length":4}]},{"pool":107,"snaps":[{"begin":2,"length":1}]},{"pool":112,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":125,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":133,"snaps":[{"begin":12,"length":1}]},{"pool":154,"snaps":[{"begin":14,"length":1}]},{"pool":164,"snaps":[{"begin":17,"length":1}]},{"pool":177,"snaps":[{"begin":20,"length":1}]},{"pool":237,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":6,"length":1}]},{"pool":245,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 10 librados: watch_flush enter 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 10 librados: watch_flush exit 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 20 client.38561.objecter shutdown clearing up homeless session... 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 10 client.38561.objecter successfully canceled tick 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 >> v2:172.21.15.87:6800/1231597463 conn(0x7faa18062c90 msgr2=0x7faa18065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 --2- 172.21.15.87:0/2684358298 >> v2:172.21.15.87:6800/1231597463 conn(0x7faa18062c90 0x7faa18065150 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 rx=0x7faa28108b30 tx=0x7faa14009cb0).stop 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280f7400 msgr2=0x7faa2810d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 --2- 172.21.15.87:0/2684358298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280f7400 0x7faa2810d1f0 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 rx=0x7faa1c009b10 tx=0x7faa1c009cb0).stop 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 shutdown_connections 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 --2- 172.21.15.87:0/2684358298 >> v2:172.21.15.87:6800/1231597463 conn(0x7faa18062c90 0x7faa18065150 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 --2- 172.21.15.87:0/2684358298 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7faa280fd510 0x7faa28107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:46.422 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 --2- 172.21.15.87:0/2684358298 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7faa280f7400 0x7faa2810d1f0 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:46.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 --2- 172.21.15.87:0/2684358298 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7faa280fcbb0 0x7faa2810d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:46.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 >> 172.21.15.87:0/2684358298 conn(0x7faa280f2820 msgr2=0x7faa280fb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:46.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 shutdown_connections 2024-02-23T00:45:46.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 -- 172.21.15.87:0/2684358298 wait complete. 2024-02-23T00:45:46.423 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:46.420+0000 7faa2f044700 1 librados: shutdown 2024-02-23T00:45:47.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:45.748086+0000 osd.7 (osd.7) 428 : cluster [DBG] 3.15 scrub starts 2024-02-23T00:45:47.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:45.751008+0000 osd.7 (osd.7) 429 : cluster [DBG] 3.15 scrub ok 2024-02-23T00:45:47.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:45.836771+0000 mgr.y (mgr.24437) 741 : cluster [DBG] pgmap v1328: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:45:47.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:45.985455+0000 osd.5 (osd.5) 363 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:45:47.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:45.986793+0000 osd.5 (osd.5) 364 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:45:47.084 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:46.417979+0000 mon.c (mon.2) 521 : audit [DBG] from='client.? 172.21.15.87:0/2684358298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:45:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:45.748086+0000 osd.7 (osd.7) 428 : cluster [DBG] 3.15 scrub starts 2024-02-23T00:45:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:45.751008+0000 osd.7 (osd.7) 429 : cluster [DBG] 3.15 scrub ok 2024-02-23T00:45:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:45.836771+0000 mgr.y (mgr.24437) 741 : cluster [DBG] pgmap v1328: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:45:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:45.985455+0000 osd.5 (osd.5) 363 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:45:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:45.986793+0000 osd.5 (osd.5) 364 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:45:47.084 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:46.417979+0000 mon.c (mon.2) 521 : audit [DBG] from='client.? 172.21.15.87:0/2684358298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:45:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:45.748086+0000 osd.7 (osd.7) 428 : cluster [DBG] 3.15 scrub starts 2024-02-23T00:45:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:45.751008+0000 osd.7 (osd.7) 429 : cluster [DBG] 3.15 scrub ok 2024-02-23T00:45:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:45.836771+0000 mgr.y (mgr.24437) 741 : cluster [DBG] pgmap v1328: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-23T00:45:47.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:45.985455+0000 osd.5 (osd.5) 363 : cluster [DBG] 6.7 scrub starts 2024-02-23T00:45:47.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:45.986793+0000 osd.5 (osd.5) 364 : cluster [DBG] 6.7 scrub ok 2024-02-23T00:45:47.193 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:46.417979+0000 mon.c (mon.2) 521 : audit [DBG] from='client.? 172.21.15.87:0/2684358298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:45:47.247 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-23T00:45:47.247 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd dump --format=json 2024-02-23T00:45:47.519 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:48.063 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:46.390990+0000 osd.1 (osd.1) 436 : cluster [DBG] 22.9 deep-scrub starts 2024-02-23T00:45:48.063 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:46.392912+0000 osd.1 (osd.1) 437 : cluster [DBG] 22.9 deep-scrub ok 2024-02-23T00:45:48.063 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:47.031208+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.a scrub starts 2024-02-23T00:45:48.064 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:47.032664+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.a scrub ok 2024-02-23T00:45:48.064 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:47.292733+0000 mon.a (mon.0) 4037 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-02-23T00:45:48.064 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:47.338504+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.d scrub starts 2024-02-23T00:45:48.064 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:47.340014+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.d scrub ok 2024-02-23T00:45:48.064 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:46.390990+0000 osd.1 (osd.1) 436 : cluster [DBG] 22.9 deep-scrub starts 2024-02-23T00:45:48.064 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:46.392912+0000 osd.1 (osd.1) 437 : cluster [DBG] 22.9 deep-scrub ok 2024-02-23T00:45:48.064 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:47.031208+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.a scrub starts 2024-02-23T00:45:48.064 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:47.032664+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.a scrub ok 2024-02-23T00:45:48.064 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:47.292733+0000 mon.a (mon.0) 4037 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-02-23T00:45:48.065 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:47.338504+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.d scrub starts 2024-02-23T00:45:48.065 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:47.340014+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.d scrub ok 2024-02-23T00:45:48.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:46.390990+0000 osd.1 (osd.1) 436 : cluster [DBG] 22.9 deep-scrub starts 2024-02-23T00:45:48.442 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:46.392912+0000 osd.1 (osd.1) 437 : cluster [DBG] 22.9 deep-scrub ok 2024-02-23T00:45:48.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:47.031208+0000 osd.5 (osd.5) 365 : cluster [DBG] 6.a scrub starts 2024-02-23T00:45:48.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:47.032664+0000 osd.5 (osd.5) 366 : cluster [DBG] 6.a scrub ok 2024-02-23T00:45:48.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:47.292733+0000 mon.a (mon.0) 4037 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-02-23T00:45:48.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:47.338504+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.d scrub starts 2024-02-23T00:45:48.443 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:47.340014+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.d scrub ok 2024-02-23T00:45:49.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.450+0000 7fd640f47700 1 -- 172.21.15.87:0/1421675885 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 msgr2=0x7fd63c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:49.453 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.450+0000 7fd640f47700 1 --2- 172.21.15.87:0/1421675885 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 0x7fd63c104a20 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 rx=0x7fd634009be0 tx=0x7fd634004d10).stop 2024-02-23T00:45:49.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.450+0000 7fd640f47700 1 -- 172.21.15.87:0/1421675885 shutdown_connections 2024-02-23T00:45:49.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.450+0000 7fd640f47700 1 --2- 172.21.15.87:0/1421675885 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 0x7fd63c104a20 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:49.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.450+0000 7fd640f47700 1 --2- 172.21.15.87:0/1421675885 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd63c0fdbb0 0x7fd63c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:49.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.450+0000 7fd640f47700 1 --2- 172.21.15.87:0/1421675885 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd63c0fe510 0x7fd63c0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:49.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.450+0000 7fd640f47700 1 -- 172.21.15.87:0/1421675885 >> 172.21.15.87:0/1421675885 conn(0x7fd63c0f3800 msgr2=0x7fd63c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:49.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.450+0000 7fd640f47700 1 -- 172.21.15.87:0/1421675885 shutdown_connections 2024-02-23T00:45:49.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.450+0000 7fd640f47700 1 -- 172.21.15.87:0/1421675885 wait complete. 2024-02-23T00:45:49.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.451+0000 7fd640f47700 1 librados: starting msgr at 2024-02-23T00:45:49.454 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.451+0000 7fd640f47700 1 librados: starting objecter 2024-02-23T00:45:49.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.451+0000 7fd640f47700 1 Processor -- start 2024-02-23T00:45:49.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.451+0000 7fd640f47700 1 -- start start 2024-02-23T00:45:49.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.451+0000 7fd640f47700 1 librados: setting wanted keys 2024-02-23T00:45:49.456 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.451+0000 7fd640f47700 1 librados: calling monclient init 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd640f47700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 0x7fd63c10ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd640f47700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd63c0fdbb0 0x7fd63c10e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd640f47700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd63c0fe510 0x7fd63c107f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd640f47700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c108480 con 0x7fd63c0fdbb0 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd640f47700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c1085f0 con 0x7fd63c0fe510 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd640f47700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c1087b0 con 0x7fd63c0f8420 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63a59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 0x7fd63c10ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd639d9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd63c0fdbb0 0x7fd63c10e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63a59c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 0x7fd63c10ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:47930/0 (socket says 172.21.15.87:47930) 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63a59c700 1 -- 172.21.15.87:0/1355327104 learned_addr learned my addr 172.21.15.87:0/1355327104 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63a59c700 1 -- 172.21.15.87:0/1355327104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd63c0fe510 msgr2=0x7fd63c107f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63ad9d700 1 --2- 172.21.15.87:0/1355327104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd63c0fe510 0x7fd63c107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63a59c700 1 --2- 172.21.15.87:0/1355327104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd63c0fe510 0x7fd63c107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:49.457 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63a59c700 1 -- 172.21.15.87:0/1355327104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd63c0fdbb0 msgr2=0x7fd63c10e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63a59c700 1 --2- 172.21.15.87:0/1355327104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd63c0fdbb0 0x7fd63c10e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63a59c700 1 -- 172.21.15.87:0/1355327104 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd630005040 con 0x7fd63c0f8420 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd639d9b700 1 --2- 172.21.15.87:0/1355327104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd63c0fdbb0 0x7fd63c10e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd63a59c700 1 --2- 172.21.15.87:0/1355327104 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 0x7fd63c10ddf0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7fd6300048f0 tx=0x7fd630009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd62b7fe700 10 client.?.objecter ms_handle_connect 0x7fd63c0f8420 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd62b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd62b7fe700 1 -- 172.21.15.87:0/1355327104 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630007b00 con 0x7fd63c0f8420 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd62b7fe700 1 -- 172.21.15.87:0/1355327104 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd63000f040 con 0x7fd63c0f8420 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6340097c0 con 0x7fd63c0f8420 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.452+0000 7fd62b7fe700 1 -- 172.21.15.87:0/1355327104 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630014a00 con 0x7fd63c0f8420 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.453+0000 7fd640f47700 10 client.38567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.453+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c108e10 con 0x7fd63c0f8420 2024-02-23T00:45:49.458 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.453+0000 7fd640f47700 1 librados: init done 2024-02-23T00:45:49.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.453+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63c107110 con 0x7fd63c0f8420 2024-02-23T00:45:49.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.454+0000 7fd62b7fe700 1 -- 172.21.15.87:0/1355327104 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fd63000e580 con 0x7fd63c0f8420 2024-02-23T00:45:49.459 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.454+0000 7fd62b7fe700 1 --2- 172.21.15.87:0/1355327104 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd624062c90 0x7fd624065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:49.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.454+0000 7fd62b7fe700 1 -- 172.21.15.87:0/1355327104 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(849..849 src has 400..849) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fd63007cb70 con 0x7fd63c0f8420 2024-02-23T00:45:49.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.454+0000 7fd62b7fe700 10 client.38567.objecter ms_dispatch 0x7fd63c001550 osd_map(849..849 src has 400..849) v4 2024-02-23T00:45:49.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.454+0000 7fd62b7fe700 3 client.38567.objecter handle_osd_map got epochs [849,849] > 0 2024-02-23T00:45:49.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.454+0000 7fd62b7fe700 3 client.38567.objecter handle_osd_map decoding full epoch 849 2024-02-23T00:45:49.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.454+0000 7fd639d9b700 1 --2- 172.21.15.87:0/1355327104 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd624062c90 0x7fd624065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:49.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.454+0000 7fd62b7fe700 20 client.38567.objecter dump_active .. 0 homeless 2024-02-23T00:45:49.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.455+0000 7fd62b7fe700 10 client.38567.objecter ms_handle_connect 0x7fd624062c90 2024-02-23T00:45:49.460 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.455+0000 7fd639d9b700 1 --2- 172.21.15.87:0/1355327104 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd624062c90 0x7fd624065150 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7fd63c109730 tx=0x7fd62c007ec0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:49.461 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.459+0000 7fd62b7fe700 1 -- 172.21.15.87:0/1355327104 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fd630049f60 con 0x7fd63c0f8420 2024-02-23T00:45:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:47.837193+0000 mgr.y (mgr.24437) 742 : cluster [DBG] pgmap v1330: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-02-23T00:45:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:48.070486+0000 osd.0 (osd.0) 105 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:45:49.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:48.073588+0000 osd.0 (osd.0) 106 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:45:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:47.837193+0000 mgr.y (mgr.24437) 742 : cluster [DBG] pgmap v1330: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-02-23T00:45:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:48.070486+0000 osd.0 (osd.0) 105 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:45:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:48.073588+0000 osd.0 (osd.0) 106 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:45:49.624 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.622+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd63c05faf0 con 0x7fd63c0f8420 2024-02-23T00:45:49.625 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.623+0000 7fd62b7fe700 1 -- 172.21.15.87:0/1355327104 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v849) v1 ==== 74+0+22226 (secure 0 0 0) 0x7fd6300502c0 con 0x7fd63c0f8420 2024-02-23T00:45:49.626 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:45:49.626 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":849,"fsid":"66db207c-d1e1-11ee-95c0-87774f69a715","created":"2024-02-23T00:21:28.861318+0000","modified":"2024-02-23T00:45:47.288104+0000","last_up_change":"2024-02-23T00:41:18.209057+0000","last_in_change":"2024-02-23T00:41:36.208573+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":115,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":256,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-23T00:25:09.208309+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-23T00:27:15.562186+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-23T00:27:15.989206+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":23,"pg_placement_num":23,"pg_placement_num_target":23,"pg_num_target":23,"pg_num_pending":23,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"816","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"736","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-23T00:27:17.282440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"4.16","ready_epoch":371,"last_epoch_started":370,"last_epoch_clean":370,"source_version":"63'21","target_version":"62'12"},"last_change":"532","last_force_op_resend":"0","last_force_op_resend_prenautilus":"371","last_force_op_resend_preluminous":"481","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-23T00:27:19.262164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"5.20","ready_epoch":641,"last_epoch_started":640,"last_epoch_clean":640,"source_version":"562'12","target_version":"330'9"},"last_change":"812","last_force_op_resend":"0","last_force_op_resend_prenautilus":"641","last_force_op_resend_preluminous":"639","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-23T00:27:21.319951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":12,"pg_placement_num":12,"pg_placement_num_target":12,"pg_num_target":12,"pg_num_pending":12,"last_pg_merge_meta":{"source_pgid":"6.c","ready_epoch":665,"last_epoch_started":664,"last_epoch_clean":664,"source_version":"78'2","target_version":"0'0"},"last_change":"815","last_force_op_resend":"0","last_force_op_resend_prenautilus":"665","last_force_op_resend_preluminous":"663","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":22,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-02-23T00:30:46.071806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"e3549bad-2891-47b9-bf77-132cadc20bf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":424,"up_from":659,"up_thru":804,"down_at":425,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6801","nonce":484081831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":484081831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":484081831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6803","nonce":484081831}]},"public_addr":"172.21.15.87:6801/484081831","cluster_addr":"172.21.15.87:6802/484081831","heartbeat_back_addr":"172.21.15.87:6804/484081831","heartbeat_front_addr":"172.21.15.87:6803/484081831","state":["exists","up"]},{"osd":1,"uuid":"85525ce8-38d5-4840-8057-547cb176c859","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":804,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6805","nonce":685354171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":685354171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":685354171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6807","nonce":685354171}]},"public_addr":"172.21.15.87:6805/685354171","cluster_addr":"172.21.15.87:6806/685354171","heartbeat_back_addr":"172.21.15.87:6808/685354171","heartbeat_front_addr":"172.21.15.87:6807/685354171","state":["exists","up"]},{"osd":2,"uuid":"48ea64b6-ed15-48d2-b0ed-aa4053938aaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":804,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6809","nonce":3605394228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":3605394228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":3605394228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6811","nonce":3605394228}]},"public_addr":"172.21.15.87:6809/3605394228","cluster_addr":"172.21.15.87:6810/3605394228","heartbeat_back_addr":"172.21.15.87:6812/3605394228","heartbeat_front_addr":"172.21.15.87:6811/3605394228","state":["exists","up"]},{"osd":3,"uuid":"cb8705aa-e114-4dfd-ac9d-ec12f85f877a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":810,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6813","nonce":128059790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":128059790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":128059790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6815","nonce":128059790}]},"public_addr":"172.21.15.87:6813/128059790","cluster_addr":"172.21.15.87:6814/128059790","heartbeat_back_addr":"172.21.15.87:6816/128059790","heartbeat_front_addr":"172.21.15.87:6815/128059790","state":["exists","up"]},{"osd":4,"uuid":"c3734a04-7343-4251-9b5b-bedb9e8b882c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6800","nonce":4175844757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6801","nonce":4175844757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6803","nonce":4175844757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6802","nonce":4175844757}]},"public_addr":"172.21.15.171:6800/4175844757","cluster_addr":"172.21.15.171:6801/4175844757","heartbeat_back_addr":"172.21.15.171:6803/4175844757","heartbeat_front_addr":"172.21.15.171:6802/4175844757","state":["exists","up"]},{"osd":5,"uuid":"fe86232d-96b9-4cbf-9cac-373286fa103d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":838,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6804","nonce":2550013795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6805","nonce":2550013795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6807","nonce":2550013795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6806","nonce":2550013795}]},"public_addr":"172.21.15.171:6804/2550013795","cluster_addr":"172.21.15.171:6805/2550013795","heartbeat_back_addr":"172.21.15.171:6807/2550013795","heartbeat_front_addr":"172.21.15.171:6806/2550013795","state":["exists","up"]},{"osd":6,"uuid":"bfc5dd69-5cfb-409f-985f-83ee0dd4dc01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":804,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6808","nonce":2509408498}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6809","nonce":2509408498}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6811","nonce":2509408498}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6810","nonce":2509408498}]},"public_addr":"172.21.15.171:6808/2509408498","cluster_addr":"172.21.15.171:6809/2509408498","heartbeat_back_addr":"172.21.15.171:6811/2509408498","heartbeat_front_addr":"172.21.15.171:6810/2509408498","state":["exists","up"]},{"osd":7,"uuid":"29ddcf97-369a-4a2c-b3e2-142e84ccf939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":804,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6812","nonce":373787000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6813","nonce":373787000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6815","nonce":373787000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.171:6814","nonce":373787000}]},"public_addr":"172.21.15.171:6812/373787000","cluster_addr":"172.21.15.171:6813/373787000","heartbeat_back_addr":"172.21.15.171:6815/373787000","heartbeat_front_addr":"172.21.15.171:6814/373787000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:13.589844+0000","dead_epoch":424},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:00.390106+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:44:35.904756+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:09.278758+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:09.281229+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:40.436151+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:07.838219+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-23T00:45:06.791018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.7","mappings":[{"from":7,"to":1},{"from":6,"to":5}]},{"pgid":"2.18","mappings":[{"from":7,"to":1}]},{"pgid":"2.1c","mappings":[{"from":7,"to":4}]},{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"5.10","mappings":[{"from":7,"to":3}]},{"pgid":"5.19","mappings":[{"from":1,"to":6}]},{"pgid":"22.12","mappings":[{"from":7,"to":0}]},{"pgid":"22.1a","mappings":[{"from":7,"to":5}]},{"pgid":"22.1f","mappings":[{"from":7,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:6800/1121018082":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/2138166229":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/27794119":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/1675126043":"2024-02-24T00:27:39.443000+0000","172.21.15.87:0/3767016525":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/969918521":"2024-02-24T00:22:22.047081+0000","172.21.15.87:0/795095363":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2273151327":"2024-02-24T00:27:39.443000+0000","172.21.15.87:6800/1685406414":"2024-02-24T00:21:58.144574+0000","172.21.15.87:0/2093733106":"2024-02-24T00:21:58.144574+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":21,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":61,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":63,"snaps":[{"begin":29,"length":1}]},{"pool":78,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":86,"snaps":[{"begin":2,"length":1}]},{"pool":93,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":95,"snaps":[{"begin":4,"length":4}]},{"pool":107,"snaps":[{"begin":2,"length":1}]},{"pool":112,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":125,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":133,"snaps":[{"begin":12,"length":1}]},{"pool":154,"snaps":[{"begin":14,"length":1}]},{"pool":164,"snaps":[{"begin":17,"length":1}]},{"pool":177,"snaps":[{"begin":20,"length":1}]},{"pool":237,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":6,"length":1}]},{"pool":245,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-23T00:45:49.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.624+0000 7fd640f47700 10 librados: watch_flush enter 2024-02-23T00:45:49.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.624+0000 7fd640f47700 10 librados: watch_flush exit 2024-02-23T00:45:49.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.624+0000 7fd640f47700 20 client.38567.objecter shutdown clearing up homeless session... 2024-02-23T00:45:49.627 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.624+0000 7fd640f47700 10 client.38567.objecter successfully canceled tick 2024-02-23T00:45:49.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd624062c90 msgr2=0x7fd624065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:49.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 --2- 172.21.15.87:0/1355327104 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd624062c90 0x7fd624065150 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 rx=0x7fd63c109730 tx=0x7fd62c007ec0).stop 2024-02-23T00:45:49.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 msgr2=0x7fd63c10ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:49.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 --2- 172.21.15.87:0/1355327104 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 0x7fd63c10ddf0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 rx=0x7fd6300048f0 tx=0x7fd630009cb0).stop 2024-02-23T00:45:49.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 shutdown_connections 2024-02-23T00:45:49.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 --2- 172.21.15.87:0/1355327104 >> v2:172.21.15.87:6800/1231597463 conn(0x7fd624062c90 0x7fd624065150 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:49.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 --2- 172.21.15.87:0/1355327104 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fd63c0fe510 0x7fd63c107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:49.628 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 --2- 172.21.15.87:0/1355327104 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fd63c0f8420 0x7fd63c10ddf0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:49.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 --2- 172.21.15.87:0/1355327104 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fd63c0fdbb0 0x7fd63c10e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:49.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 >> 172.21.15.87:0/1355327104 conn(0x7fd63c0f3800 msgr2=0x7fd63c0f5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:49.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 shutdown_connections 2024-02-23T00:45:49.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 -- 172.21.15.87:0/1355327104 wait complete. 2024-02-23T00:45:49.629 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:49.625+0000 7fd640f47700 1 librados: shutdown 2024-02-23T00:45:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:47.837193+0000 mgr.y (mgr.24437) 742 : cluster [DBG] pgmap v1330: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-02-23T00:45:49.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:48.070486+0000 osd.0 (osd.0) 105 : cluster [DBG] 5.5 scrub starts 2024-02-23T00:45:49.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:48.073588+0000 osd.0 (osd.0) 106 : cluster [DBG] 5.5 scrub ok 2024-02-23T00:45:50.380 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-23T00:45:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:49.036958+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.12 scrub starts 2024-02-23T00:45:50.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:49.037938+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.12 scrub ok 2024-02-23T00:45:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:49.310483+0000 osd.6 (osd.6) 386 : cluster [DBG] 5.11 deep-scrub starts 2024-02-23T00:45:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:49.313387+0000 osd.6 (osd.6) 387 : cluster [DBG] 5.11 deep-scrub ok 2024-02-23T00:45:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:49.623814+0000 mon.c (mon.2) 522 : audit [DBG] from='client.? 172.21.15.87:0/1355327104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:45:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:50.074530+0000 osd.5 (osd.5) 367 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:45:50.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:50.076706+0000 osd.5 (osd.5) 368 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:45:50.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:49.036958+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.12 scrub starts 2024-02-23T00:45:50.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:49.037938+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.12 scrub ok 2024-02-23T00:45:50.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:49.310483+0000 osd.6 (osd.6) 386 : cluster [DBG] 5.11 deep-scrub starts 2024-02-23T00:45:50.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:49.313387+0000 osd.6 (osd.6) 387 : cluster [DBG] 5.11 deep-scrub ok 2024-02-23T00:45:50.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:49.623814+0000 mon.c (mon.2) 522 : audit [DBG] from='client.? 172.21.15.87:0/1355327104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:45:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:50.074530+0000 osd.5 (osd.5) 367 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:45:50.586 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:50.076706+0000 osd.5 (osd.5) 368 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:45:50.649 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:50.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:49.036958+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.12 scrub starts 2024-02-23T00:45:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:49.037938+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.12 scrub ok 2024-02-23T00:45:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:49.310483+0000 osd.6 (osd.6) 386 : cluster [DBG] 5.11 deep-scrub starts 2024-02-23T00:45:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:49.313387+0000 osd.6 (osd.6) 387 : cluster [DBG] 5.11 deep-scrub ok 2024-02-23T00:45:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:49.623814+0000 mon.c (mon.2) 522 : audit [DBG] from='client.? 172.21.15.87:0/1355327104' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-23T00:45:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:50.074530+0000 osd.5 (osd.5) 367 : cluster [DBG] 4.8 scrub starts 2024-02-23T00:45:50.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:50.076706+0000 osd.5 (osd.5) 368 : cluster [DBG] 4.8 scrub ok 2024-02-23T00:45:51.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:49.837818+0000 mgr.y (mgr.24437) 743 : cluster [DBG] pgmap v1331: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:45:51.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:51.093556+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:45:51.486 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:51.094835+0000 osd.5 (osd.5) 370 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:45:51.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:49.837818+0000 mgr.y (mgr.24437) 743 : cluster [DBG] pgmap v1331: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:45:51.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:51.093556+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:45:51.486 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:51.094835+0000 osd.5 (osd.5) 370 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:45:51.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:45:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:45:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:49.837818+0000 mgr.y (mgr.24437) 743 : cluster [DBG] pgmap v1331: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:45:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:51.093556+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.7 scrub starts 2024-02-23T00:45:51.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:51.094835+0000 osd.5 (osd.5) 370 : cluster [DBG] 2.7 scrub ok 2024-02-23T00:45:52.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:45:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:52] "GET /metrics HTTP/1.1" 200 186061 "" "Prometheus/2.33.4" 2024-02-23T00:45:52.641 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:45:51.427666+0000 mgr.y (mgr.24437) 744 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:52.641 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:51.750479+0000 osd.7 (osd.7) 430 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:45:52.641 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:51.753963+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:45:52.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:45:51.427666+0000 mgr.y (mgr.24437) 744 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:52.641 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:51.750479+0000 osd.7 (osd.7) 430 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:45:52.642 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:51.753963+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:45:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:45:51.427666+0000 mgr.y (mgr.24437) 744 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:45:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:51.750479+0000 osd.7 (osd.7) 430 : cluster [DBG] 2.8 scrub starts 2024-02-23T00:45:52.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:51.753963+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.8 scrub ok 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.010+0000 7fc55436e700 1 -- 172.21.15.87:0/3599868288 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc54c0fe510 msgr2=0x7fc54c0f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.010+0000 7fc55436e700 1 --2- 172.21.15.87:0/3599868288 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc54c0fe510 0x7fc54c0f7d40 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 rx=0x7fc548004870 tx=0x7fc548009cb0).stop 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.010+0000 7fc55436e700 1 -- 172.21.15.87:0/3599868288 shutdown_connections 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.010+0000 7fc55436e700 1 --2- 172.21.15.87:0/3599868288 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc54c0f8420 0x7fc54c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.010+0000 7fc55436e700 1 --2- 172.21.15.87:0/3599868288 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc54c0fdbb0 0x7fc54c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.010+0000 7fc55436e700 1 --2- 172.21.15.87:0/3599868288 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc54c0fe510 0x7fc54c0f7d40 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.010+0000 7fc55436e700 1 -- 172.21.15.87:0/3599868288 >> 172.21.15.87:0/3599868288 conn(0x7fc54c0f3800 msgr2=0x7fc54c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.010+0000 7fc55436e700 1 -- 172.21.15.87:0/3599868288 shutdown_connections 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.010+0000 7fc55436e700 1 -- 172.21.15.87:0/3599868288 wait complete. 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 librados: starting msgr at 2024-02-23T00:45:53.033 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 librados: starting objecter 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 Processor -- start 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 -- start start 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 librados: setting wanted keys 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 librados: calling monclient init 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc54c0f8420 0x7fc54c10e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc54c0fdbb0 0x7fc54c10e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc54c0fe510 0x7fc54c1082b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c1087a0 con 0x7fc54c0fdbb0 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c108910 con 0x7fc54c0f8420 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55436e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c108ad0 con 0x7fc54c0fe510 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55290b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc54c0fe510 0x7fc54c1082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc551909700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc54c0fdbb0 0x7fc54c10e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55290b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc54c0fe510 0x7fc54c1082b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:42596/0 (socket says 172.21.15.87:42596) 2024-02-23T00:45:53.034 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55290b700 1 -- 172.21.15.87:0/600957921 learned_addr learned my addr 172.21.15.87:0/600957921 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55210a700 1 --2- 172.21.15.87:0/600957921 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc54c0f8420 0x7fc54c10e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55290b700 1 -- 172.21.15.87:0/600957921 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc54c0f8420 msgr2=0x7fc54c10e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55290b700 1 --2- 172.21.15.87:0/600957921 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc54c0f8420 0x7fc54c10e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55290b700 1 -- 172.21.15.87:0/600957921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc54c0fdbb0 msgr2=0x7fc54c10e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55290b700 1 --2- 172.21.15.87:0/600957921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc54c0fdbb0 0x7fc54c10e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc55290b700 1 -- 172.21.15.87:0/600957921 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c005040 con 0x7fc54c0fe510 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.011+0000 7fc551909700 1 --2- 172.21.15.87:0/600957921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc54c0fdbb0 0x7fc54c10e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc55290b700 1 --2- 172.21.15.87:0/600957921 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc54c0fe510 0x7fc54c1082b0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7fc54400c960 tx=0x7fc544004d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc5437fe700 10 client.?.objecter ms_handle_connect 0x7fc54c0fe510 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc5437fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc5437fe700 1 -- 172.21.15.87:0/600957921 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54400a5c0 con 0x7fc54c0fe510 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc5437fe700 1 -- 172.21.15.87:0/600957921 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc54400abf0 con 0x7fc54c0fe510 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc548005040 con 0x7fc54c0fe510 2024-02-23T00:45:53.035 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc5437fe700 1 -- 172.21.15.87:0/600957921 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544012c00 con 0x7fc54c0fe510 2024-02-23T00:45:53.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc55436e700 10 client.38573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:53.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c114f40 con 0x7fc54c0fe510 2024-02-23T00:45:53.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.012+0000 7fc55436e700 1 librados: init done 2024-02-23T00:45:53.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.013+0000 7fc55436e700 10 client.38573.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:45:53.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.013+0000 7fc55436e700 15 client.38573.objecter _session_command_op_assign -1 1 2024-02-23T00:45:53.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.013+0000 7fc55436e700 20 client.38573.objecter _get_session osd=-1 returning homeless 2024-02-23T00:45:53.036 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.013+0000 7fc55436e700 10 client.38573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.013+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc54c05ee80 con 0x7fc54c0fe510 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.013+0000 7fc5437fe700 1 -- 172.21.15.87:0/600957921 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc54400a720 con 0x7fc54c0fe510 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.013+0000 7fc5437fe700 1 --2- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc538062c90 0x7fc538065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 1 -- 172.21.15.87:0/600957921 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(849..849 src has 400..849) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fc54407cb90 con 0x7fc54c0fe510 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 10 client.38573.objecter ms_dispatch 0x7fc54c001550 osd_map(849..849 src has 400..849) v4 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 3 client.38573.objecter handle_osd_map got epochs [849,849] > 0 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 3 client.38573.objecter handle_osd_map decoding full epoch 849 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc55210a700 1 --2- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc538062c90 0x7fc538065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 10 client.38573.objecter checking command 1 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 1 --2- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6801/484081831 conn(0x7fc53806e0f0 0x7fc538070590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 20 client.38573.objecter _get_session s=0x7fc53806da40 osd=0 3 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 20 client.38573.objecter put_session s=0x7fc53806da40 osd=0 3 2024-02-23T00:45:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 15 client.38573.objecter _session_command_op_remove -1 1 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 20 client.38573.objecter _get_session s=0x7fc53806da40 osd=0 3 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 20 client.38573.objecter get_session s=0x7fc53806da40 osd=0 3 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 15 client.38573.objecter _session_command_op_assign 0 1 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 20 client.38573.objecter put_session s=0x7fc53806da40 osd=0 4 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 10 client.38573.objecter _send_command 1 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 1 -- 172.21.15.87:0/600957921 --> v2:172.21.15.87:6801/484081831 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc538070bd0 con 0x7fc53806e0f0 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 20 client.38573.objecter dump_active .. 0 homeless 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 1 -- 172.21.15.87:0/600957921 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=849) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc544081220 con 0x7fc54c0fe510 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc551909700 1 --2- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6801/484081831 conn(0x7fc53806e0f0 0x7fc538070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 10 client.38573.objecter ms_handle_connect 0x7fc538062c90 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc55210a700 1 --2- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc538062c90 0x7fc538065150 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 rx=0x7fc53c0092f0 tx=0x7fc53c009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc5437fe700 10 client.38573.objecter ms_handle_connect 0x7fc53806e0f0 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.014+0000 7fc551909700 1 --2- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6801/484081831 conn(0x7fc53806e0f0 0x7fc538070590 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:53.038 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.015+0000 7fc5437fe700 1 -- 172.21.15.87:0/600957921 <== osd.0 v2:172.21.15.87:6801/484081831 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fc538070bd0 con 0x7fc53806e0f0 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.015+0000 7fc5437fe700 10 client.38573.objecter ms_dispatch 0x7fc54c001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.015+0000 7fc5437fe700 10 client.38573.objecter _finish_command 1 = system:0 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.015+0000 7fc5437fe700 20 client.38573.objecter put_session s=0x7fc53806da40 osd=0 4 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.015+0000 7fc5437fe700 15 client.38573.objecter _session_command_op_remove 0 1 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 10 client.38573.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 15 client.38573.objecter _session_command_op_assign -1 2 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 20 client.38573.objecter _get_session s=0x7fc53806da40 osd=0 3 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 20 client.38573.objecter put_session s=0x7fc53806da40 osd=0 3 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 20 client.38573.objecter _get_session s=0x7fc53806da40 osd=0 3 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 15 client.38573.objecter _session_command_op_remove -1 2 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 20 client.38573.objecter get_session s=0x7fc53806da40 osd=0 3 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 15 client.38573.objecter _session_command_op_assign 0 2 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 20 client.38573.objecter put_session s=0x7fc53806da40 osd=0 4 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 10 client.38573.objecter _send_command 2 2024-02-23T00:45:53.039 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 --> v2:172.21.15.87:6801/484081831 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc54c107060 con 0x7fc53806e0f0 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc5437fe700 1 -- 172.21.15.87:0/600957921 <== osd.0 v2:172.21.15.87:6801/484081831 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+13 (crc 0 0 0) 0x7fc54c107060 con 0x7fc53806e0f0 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc5437fe700 10 client.38573.objecter ms_dispatch 0x7fc54c001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc5437fe700 10 client.38573.objecter _finish_command 2 = system:0 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc5437fe700 20 client.38573.objecter put_session s=0x7fc53806da40 osd=0 4 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc5437fe700 15 client.38573.objecter _session_command_op_remove 0 2 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 10 librados: watch_flush enter 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 10 librados: watch_flush exit 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 10 client.38573.objecter close_session for osd.0 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6801/484081831 conn(0x7fc53806e0f0 msgr2=0x7fc538070590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 --2- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6801/484081831 conn(0x7fc53806e0f0 0x7fc538070590 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 20 client.38573.objecter put_session s=0x7fc53806da40 osd=0 1 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 20 client.38573.objecter shutdown clearing up homeless session... 2024-02-23T00:45:53.040 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 10 client.38573.objecter successfully canceled tick 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc538062c90 msgr2=0x7fc538065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 --2- 172.21.15.87:0/600957921 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc538062c90 0x7fc538065150 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 rx=0x7fc53c0092f0 tx=0x7fc53c009cb0).stop 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc54c0fe510 msgr2=0x7fc54c1082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 --2- 172.21.15.87:0/600957921 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc54c0fe510 0x7fc54c1082b0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 rx=0x7fc54400c960 tx=0x7fc544004d60).stop 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55290b700 1 -- 172.21.15.87:0/600957921 reap_dead start 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 shutdown_connections 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 >> 172.21.15.87:0/600957921 conn(0x7fc54c0f3800 msgr2=0x7fc54c0f5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 shutdown_connections 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 -- 172.21.15.87:0/600957921 wait complete. 2024-02-23T00:45:53.041 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:53.029+0000 7fc55436e700 1 librados: shutdown 2024-02-23T00:45:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:51.838665+0000 mgr.y (mgr.24437) 745 : cluster [DBG] pgmap v1332: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:45:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:52.303117+0000 osd.1 (osd.1) 438 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:45:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:52.304827+0000 osd.1 (osd.1) 439 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:45:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:52.410710+0000 osd.2 (osd.2) 329 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:45:53.333 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:52.411827+0000 osd.2 (osd.2) 330 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:45:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:52.709550+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:45:53.334 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:52.712634+0000 osd.7 (osd.7) 433 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:45:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:51.838665+0000 mgr.y (mgr.24437) 745 : cluster [DBG] pgmap v1332: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:45:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:52.303117+0000 osd.1 (osd.1) 438 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:45:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:52.304827+0000 osd.1 (osd.1) 439 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:45:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:52.410710+0000 osd.2 (osd.2) 329 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:45:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:52.411827+0000 osd.2 (osd.2) 330 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:45:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:52.709550+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:45:53.334 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:52.712634+0000 osd.7 (osd.7) 433 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:45:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:51.838665+0000 mgr.y (mgr.24437) 745 : cluster [DBG] pgmap v1332: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:45:53.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:52.303117+0000 osd.1 (osd.1) 438 : cluster [DBG] 22.11 scrub starts 2024-02-23T00:45:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:52.304827+0000 osd.1 (osd.1) 439 : cluster [DBG] 22.11 scrub ok 2024-02-23T00:45:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:52.410710+0000 osd.2 (osd.2) 329 : cluster [DBG] 2.1 scrub starts 2024-02-23T00:45:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:52.411827+0000 osd.2 (osd.2) 330 : cluster [DBG] 2.1 scrub ok 2024-02-23T00:45:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:52.709550+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.1d scrub starts 2024-02-23T00:45:53.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:52.712634+0000 osd.7 (osd.7) 433 : cluster [DBG] 2.1d scrub ok 2024-02-23T00:45:53.801 INFO:teuthology.orchestra.run.smithi087.stdout:2830383448120 2024-02-23T00:45:53.802 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-23T00:45:54.070 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:54.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:45:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:45:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:45:55.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:53.839114+0000 mgr.y (mgr.24437) 746 : cluster [DBG] pgmap v1333: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:45:55.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:53.898750+0000 osd.2 (osd.2) 331 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:55.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:53.898924+0000 osd.2 (osd.2) 332 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:55.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:54.236929+0000 osd.1 (osd.1) 440 : cluster [DBG] 22.b scrub starts 2024-02-23T00:45:55.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:54.240848+0000 osd.1 (osd.1) 441 : cluster [DBG] 22.b scrub ok 2024-02-23T00:45:55.561 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:54.317839+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-02-23T00:45:55.561 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:54.736987+0000 osd.7 (osd.7) 434 : cluster [DBG] 3.f scrub starts 2024-02-23T00:45:55.561 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:54.740121+0000 osd.7 (osd.7) 435 : cluster [DBG] 3.f scrub ok 2024-02-23T00:45:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:53.839114+0000 mgr.y (mgr.24437) 746 : cluster [DBG] pgmap v1333: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:45:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:53.898750+0000 osd.2 (osd.2) 331 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:53.898924+0000 osd.2 (osd.2) 332 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:54.236929+0000 osd.1 (osd.1) 440 : cluster [DBG] 22.b scrub starts 2024-02-23T00:45:55.561 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:54.240848+0000 osd.1 (osd.1) 441 : cluster [DBG] 22.b scrub ok 2024-02-23T00:45:55.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:54.317839+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-02-23T00:45:55.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:54.736987+0000 osd.7 (osd.7) 434 : cluster [DBG] 3.f scrub starts 2024-02-23T00:45:55.562 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:55 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:54.740121+0000 osd.7 (osd.7) 435 : cluster [DBG] 3.f scrub ok 2024-02-23T00:45:55.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:53.839114+0000 mgr.y (mgr.24437) 746 : cluster [DBG] pgmap v1333: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:45:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:53.898750+0000 osd.2 (osd.2) 331 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:45:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:53.898924+0000 osd.2 (osd.2) 332 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:45:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:54.236929+0000 osd.1 (osd.1) 440 : cluster [DBG] 22.b scrub starts 2024-02-23T00:45:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:54.240848+0000 osd.1 (osd.1) 441 : cluster [DBG] 22.b scrub ok 2024-02-23T00:45:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:54.317839+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-02-23T00:45:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:54.736987+0000 osd.7 (osd.7) 434 : cluster [DBG] 3.f scrub starts 2024-02-23T00:45:55.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:54.740121+0000 osd.7 (osd.7) 435 : cluster [DBG] 3.f scrub ok 2024-02-23T00:45:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 -- 172.21.15.87:0/2528896005 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29f00fcab0 msgr2=0x7f29f00fce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 --2- 172.21.15.87:0/2528896005 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29f00fcab0 0x7f29f00fce90 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 rx=0x7f29e4004870 tx=0x7f29e4009cb0).stop 2024-02-23T00:45:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 -- 172.21.15.87:0/2528896005 shutdown_connections 2024-02-23T00:45:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 --2- 172.21.15.87:0/2528896005 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29f00f7320 0x7f29f0103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 --2- 172.21.15.87:0/2528896005 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29f00fcab0 0x7f29f00fce90 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 --2- 172.21.15.87:0/2528896005 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f00fd410 0x7f29f00f6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:45:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 -- 172.21.15.87:0/2528896005 >> 172.21.15.87:0/2528896005 conn(0x7f29f00f26e0 msgr2=0x7f29f00f4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:56.082 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 -- 172.21.15.87:0/2528896005 shutdown_connections 2024-02-23T00:45:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 -- 172.21.15.87:0/2528896005 wait complete. 2024-02-23T00:45:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 librados: starting msgr at 2024-02-23T00:45:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 librados: starting objecter 2024-02-23T00:45:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 Processor -- start 2024-02-23T00:45:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 -- start start 2024-02-23T00:45:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 librados: setting wanted keys 2024-02-23T00:45:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 librados: calling monclient init 2024-02-23T00:45:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f00f7320 0x7f29f010e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:56.083 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29f00fcab0 0x7f29f010e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:56.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29f00fd410 0x7f29f01082b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:56.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f01087a0 con 0x7f29f00fd410 2024-02-23T00:45:56.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f0108910 con 0x7f29f00f7320 2024-02-23T00:45:56.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.074+0000 7f29f6bf1700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f0108ad0 con 0x7f29f00fcab0 2024-02-23T00:45:56.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f518e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29f00fd410 0x7f29f01082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:56.084 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29effff700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29f00fcab0 0x7f29f010e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:56.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f518e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29f00fd410 0x7f29f01082b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54912/0 (socket says 172.21.15.87:54912) 2024-02-23T00:45:56.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f518e700 1 -- 172.21.15.87:0/3591150744 learned_addr learned my addr 172.21.15.87:0/3591150744 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:56.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f498d700 1 --2- 172.21.15.87:0/3591150744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f00f7320 0x7f29f010e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:56.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f518e700 1 -- 172.21.15.87:0/3591150744 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29f00fcab0 msgr2=0x7f29f010e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:56.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f518e700 1 --2- 172.21.15.87:0/3591150744 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29f00fcab0 0x7f29f010e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:56.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f518e700 1 -- 172.21.15.87:0/3591150744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f00f7320 msgr2=0x7f29f010e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:56.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f518e700 1 --2- 172.21.15.87:0/3591150744 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f29f00f7320 0x7f29f010e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:56.085 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f518e700 1 -- 172.21.15.87:0/3591150744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e80097c0 con 0x7f29f00fd410 2024-02-23T00:45:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29effff700 1 --2- 172.21.15.87:0/3591150744 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f29f00fcab0 0x7f29f010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f518e700 1 --2- 172.21.15.87:0/3591150744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29f00fd410 0x7f29f01082b0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7f29e8001df0 tx=0x7f29e8004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29edffb700 10 client.?.objecter ms_handle_connect 0x7f29f00fd410 2024-02-23T00:45:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29edffb700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29edffb700 1 -- 172.21.15.87:0/3591150744 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8009a10 con 0x7f29f00fd410 2024-02-23T00:45:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29edffb700 1 -- 172.21.15.87:0/3591150744 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29e80090d0 con 0x7f29f00fd410 2024-02-23T00:45:56.086 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e4005040 con 0x7f29f00fd410 2024-02-23T00:45:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.075+0000 7f29edffb700 1 -- 172.21.15.87:0/3591150744 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e8007d30 con 0x7f29f00fd410 2024-02-23T00:45:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.076+0000 7f29f6bf1700 10 client.28101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.076+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f0114f40 con 0x7f29f00fd410 2024-02-23T00:45:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.076+0000 7f29f6bf1700 1 librados: init done 2024-02-23T00:45:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.076+0000 7f29f6bf1700 10 client.28101.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:45:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.076+0000 7f29f6bf1700 15 client.28101.objecter _session_command_op_assign -1 1 2024-02-23T00:45:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.076+0000 7f29f6bf1700 20 client.28101.objecter _get_session osd=-1 returning homeless 2024-02-23T00:45:56.087 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.076+0000 7f29f6bf1700 10 client.28101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:56.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.076+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f29f005f9d0 con 0x7f29f00fd410 2024-02-23T00:45:56.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 1 -- 172.21.15.87:0/3591150744 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f29e8009240 con 0x7f29f00fd410 2024-02-23T00:45:56.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 1 --2- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f29e0062c90 0x7f29e0065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:56.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 1 -- 172.21.15.87:0/3591150744 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(850..850 src has 400..850) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f29e807d750 con 0x7f29f00fd410 2024-02-23T00:45:56.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 10 client.28101.objecter ms_dispatch 0x7f29f0001550 osd_map(850..850 src has 400..850) v4 2024-02-23T00:45:56.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 3 client.28101.objecter handle_osd_map got epochs [850,850] > 0 2024-02-23T00:45:56.088 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 3 client.28101.objecter handle_osd_map decoding full epoch 850 2024-02-23T00:45:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29f498d700 1 --2- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f29e0062c90 0x7f29e0065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 10 client.28101.objecter checking command 1 2024-02-23T00:45:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 1 --2- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6805/685354171 conn(0x7f29e006e170 0x7f29e0070610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 20 client.28101.objecter _get_session s=0x7f29e006dac0 osd=1 3 2024-02-23T00:45:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 20 client.28101.objecter put_session s=0x7f29e006dac0 osd=1 3 2024-02-23T00:45:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 15 client.28101.objecter _session_command_op_remove -1 1 2024-02-23T00:45:56.089 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 20 client.28101.objecter _get_session s=0x7f29e006dac0 osd=1 3 2024-02-23T00:45:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 20 client.28101.objecter get_session s=0x7f29e006dac0 osd=1 3 2024-02-23T00:45:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 15 client.28101.objecter _session_command_op_assign 1 1 2024-02-23T00:45:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 20 client.28101.objecter put_session s=0x7f29e006dac0 osd=1 4 2024-02-23T00:45:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 10 client.28101.objecter _send_command 1 2024-02-23T00:45:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 1 -- 172.21.15.87:0/3591150744 --> v2:172.21.15.87:6805/685354171 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29e0070c50 con 0x7f29e006e170 2024-02-23T00:45:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 20 client.28101.objecter dump_active .. 0 homeless 2024-02-23T00:45:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 1 -- 172.21.15.87:0/3591150744 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=850) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29e807fdc0 con 0x7f29f00fd410 2024-02-23T00:45:56.090 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29effff700 1 --2- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6805/685354171 conn(0x7f29e006e170 0x7f29e0070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29edffb700 10 client.28101.objecter ms_handle_connect 0x7f29e0062c90 2024-02-23T00:45:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.077+0000 7f29f498d700 1 --2- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f29e0062c90 0x7f29e0065150 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7f29e40043d0 tx=0x7f29e400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.078+0000 7f29edffb700 10 client.28101.objecter ms_handle_connect 0x7f29e006e170 2024-02-23T00:45:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.078+0000 7f29effff700 1 --2- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6805/685354171 conn(0x7f29e006e170 0x7f29e0070610 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.078+0000 7f29edffb700 1 -- 172.21.15.87:0/3591150744 <== osd.1 v2:172.21.15.87:6805/685354171 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f29e0070c50 con 0x7f29e006e170 2024-02-23T00:45:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.078+0000 7f29edffb700 10 client.28101.objecter ms_dispatch 0x7f29f0001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:45:56.091 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.078+0000 7f29edffb700 10 client.28101.objecter _finish_command 1 = system:0 2024-02-23T00:45:56.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.078+0000 7f29edffb700 20 client.28101.objecter put_session s=0x7f29e006dac0 osd=1 4 2024-02-23T00:45:56.092 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.078+0000 7f29edffb700 15 client.28101.objecter _session_command_op_remove 1 1 2024-02-23T00:45:56.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 10 client.28101.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:45:56.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 15 client.28101.objecter _session_command_op_assign -1 2 2024-02-23T00:45:56.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 20 client.28101.objecter _get_session s=0x7f29e006dac0 osd=1 3 2024-02-23T00:45:56.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 20 client.28101.objecter put_session s=0x7f29e006dac0 osd=1 3 2024-02-23T00:45:56.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 20 client.28101.objecter _get_session s=0x7f29e006dac0 osd=1 3 2024-02-23T00:45:56.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 15 client.28101.objecter _session_command_op_remove -1 2 2024-02-23T00:45:56.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 20 client.28101.objecter get_session s=0x7f29e006dac0 osd=1 3 2024-02-23T00:45:56.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 15 client.28101.objecter _session_command_op_assign 1 2 2024-02-23T00:45:56.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 20 client.28101.objecter put_session s=0x7f29e006dac0 osd=1 4 2024-02-23T00:45:56.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 10 client.28101.objecter _send_command 2 2024-02-23T00:45:56.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 --> v2:172.21.15.87:6805/685354171 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f29f005ee80 con 0x7f29e006e170 2024-02-23T00:45:56.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29edffb700 1 -- 172.21.15.87:0/3591150744 <== osd.1 v2:172.21.15.87:6805/685354171 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f29f005ee80 con 0x7f29e006e170 2024-02-23T00:45:56.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29edffb700 10 client.28101.objecter ms_dispatch 0x7f29f0001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:45:56.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29edffb700 10 client.28101.objecter _finish_command 2 = system:0 2024-02-23T00:45:56.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29edffb700 20 client.28101.objecter put_session s=0x7f29e006dac0 osd=1 4 2024-02-23T00:45:56.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.092+0000 7f29edffb700 15 client.28101.objecter _session_command_op_remove 1 2 2024-02-23T00:45:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 10 librados: watch_flush enter 2024-02-23T00:45:56.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 10 librados: watch_flush exit 2024-02-23T00:45:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 10 client.28101.objecter close_session for osd.1 2024-02-23T00:45:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6805/685354171 conn(0x7f29e006e170 msgr2=0x7f29e0070610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 --2- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6805/685354171 conn(0x7f29e006e170 0x7f29e0070610 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 20 client.28101.objecter put_session s=0x7f29e006dac0 osd=1 1 2024-02-23T00:45:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 20 client.28101.objecter shutdown clearing up homeless session... 2024-02-23T00:45:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 10 client.28101.objecter successfully canceled tick 2024-02-23T00:45:56.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f29e0062c90 msgr2=0x7f29e0065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 --2- 172.21.15.87:0/3591150744 >> v2:172.21.15.87:6800/1231597463 conn(0x7f29e0062c90 0x7f29e0065150 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 rx=0x7f29e40043d0 tx=0x7f29e400b040).stop 2024-02-23T00:45:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29f00fd410 msgr2=0x7f29f01082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 --2- 172.21.15.87:0/3591150744 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f29f00fd410 0x7f29f01082b0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 rx=0x7f29e8001df0 tx=0x7f29e8004c50).stop 2024-02-23T00:45:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f518e700 1 -- 172.21.15.87:0/3591150744 reap_dead start 2024-02-23T00:45:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 shutdown_connections 2024-02-23T00:45:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 >> 172.21.15.87:0/3591150744 conn(0x7f29f00f26e0 msgr2=0x7f29f0060f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:56.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 shutdown_connections 2024-02-23T00:45:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 -- 172.21.15.87:0/3591150744 wait complete. 2024-02-23T00:45:56.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:56.093+0000 7f29f6bf1700 1 librados: shutdown 2024-02-23T00:45:56.510 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:55.442994+0000 osd.6 (osd.6) 388 : cluster [DBG] 3.10 scrub starts 2024-02-23T00:45:56.510 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:55.445992+0000 osd.6 (osd.6) 389 : cluster [DBG] 3.10 scrub ok 2024-02-23T00:45:56.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:55.442994+0000 osd.6 (osd.6) 388 : cluster [DBG] 3.10 scrub starts 2024-02-23T00:45:56.510 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:56 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:55.445992+0000 osd.6 (osd.6) 389 : cluster [DBG] 3.10 scrub ok 2024-02-23T00:45:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:55.442994+0000 osd.6 (osd.6) 388 : cluster [DBG] 3.10 scrub starts 2024-02-23T00:45:56.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:56 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:55.445992+0000 osd.6 (osd.6) 389 : cluster [DBG] 3.10 scrub ok 2024-02-23T00:45:56.820 INFO:teuthology.orchestra.run.smithi087.stdout:55834575104 2024-02-23T00:45:56.821 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-23T00:45:57.092 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:45:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:55.839862+0000 mgr.y (mgr.24437) 747 : cluster [DBG] pgmap v1335: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:56.267428+0000 osd.2 (osd.2) 333 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:45:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:56.268245+0000 osd.2 (osd.2) 334 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:45:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:56.711612+0000 osd.7 (osd.7) 436 : cluster [DBG] 3.e deep-scrub starts 2024-02-23T00:45:57.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:56.714501+0000 osd.7 (osd.7) 437 : cluster [DBG] 3.e deep-scrub ok 2024-02-23T00:45:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.115512+0000 osd.5 (osd.5) 371 : cluster [DBG] 4.15 deep-scrub starts 2024-02-23T00:45:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.117672+0000 osd.5 (osd.5) 372 : cluster [DBG] 4.15 deep-scrub ok 2024-02-23T00:45:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.149642+0000 osd.4 (osd.4) 389 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:45:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.151355+0000 osd.4 (osd.4) 390 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:45:57.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.298812+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-02-23T00:45:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:55.839862+0000 mgr.y (mgr.24437) 747 : cluster [DBG] pgmap v1335: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:56.267428+0000 osd.2 (osd.2) 333 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:45:57.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:56.268245+0000 osd.2 (osd.2) 334 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:45:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:56.711612+0000 osd.7 (osd.7) 436 : cluster [DBG] 3.e deep-scrub starts 2024-02-23T00:45:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:56.714501+0000 osd.7 (osd.7) 437 : cluster [DBG] 3.e deep-scrub ok 2024-02-23T00:45:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.115512+0000 osd.5 (osd.5) 371 : cluster [DBG] 4.15 deep-scrub starts 2024-02-23T00:45:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.117672+0000 osd.5 (osd.5) 372 : cluster [DBG] 4.15 deep-scrub ok 2024-02-23T00:45:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.149642+0000 osd.4 (osd.4) 389 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:45:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.151355+0000 osd.4 (osd.4) 390 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:45:57.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.298812+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-02-23T00:45:57.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:55.839862+0000 mgr.y (mgr.24437) 747 : cluster [DBG] pgmap v1335: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:56.267428+0000 osd.2 (osd.2) 333 : cluster [DBG] 5.6 scrub starts 2024-02-23T00:45:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:56.268245+0000 osd.2 (osd.2) 334 : cluster [DBG] 5.6 scrub ok 2024-02-23T00:45:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:56.711612+0000 osd.7 (osd.7) 436 : cluster [DBG] 3.e deep-scrub starts 2024-02-23T00:45:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:56.714501+0000 osd.7 (osd.7) 437 : cluster [DBG] 3.e deep-scrub ok 2024-02-23T00:45:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.115512+0000 osd.5 (osd.5) 371 : cluster [DBG] 4.15 deep-scrub starts 2024-02-23T00:45:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.117672+0000 osd.5 (osd.5) 372 : cluster [DBG] 4.15 deep-scrub ok 2024-02-23T00:45:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.149642+0000 osd.4 (osd.4) 389 : cluster [DBG] 4.1b scrub starts 2024-02-23T00:45:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.151355+0000 osd.4 (osd.4) 390 : cluster [DBG] 4.1b scrub ok 2024-02-23T00:45:57.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:57 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.298812+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-02-23T00:45:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:56.987408+0000 osd.0 (osd.0) 109 : cluster [DBG] 2.1f deep-scrub starts 2024-02-23T00:45:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:56.990381+0000 osd.0 (osd.0) 110 : cluster [DBG] 2.1f deep-scrub ok 2024-02-23T00:45:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.295981+0000 osd.2 (osd.2) 335 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:45:58.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.297852+0000 osd.2 (osd.2) 336 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.370394+0000 osd.6 (osd.6) 390 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.371520+0000 osd.6 (osd.6) 391 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:58.128241+0000 osd.4 (osd.4) 391 : cluster [DBG] 4.c scrub starts 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:58.130369+0000 osd.4 (osd.4) 392 : cluster [DBG] 4.c scrub ok 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:56.987408+0000 osd.0 (osd.0) 109 : cluster [DBG] 2.1f deep-scrub starts 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:56.990381+0000 osd.0 (osd.0) 110 : cluster [DBG] 2.1f deep-scrub ok 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.295981+0000 osd.2 (osd.2) 335 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.297852+0000 osd.2 (osd.2) 336 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.370394+0000 osd.6 (osd.6) 390 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:45:58.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.371520+0000 osd.6 (osd.6) 391 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:45:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:58.128241+0000 osd.4 (osd.4) 391 : cluster [DBG] 4.c scrub starts 2024-02-23T00:45:58.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:58.130369+0000 osd.4 (osd.4) 392 : cluster [DBG] 4.c scrub ok 2024-02-23T00:45:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:56.987408+0000 osd.0 (osd.0) 109 : cluster [DBG] 2.1f deep-scrub starts 2024-02-23T00:45:58.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:56.990381+0000 osd.0 (osd.0) 110 : cluster [DBG] 2.1f deep-scrub ok 2024-02-23T00:45:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.295981+0000 osd.2 (osd.2) 335 : cluster [DBG] 5.8 scrub starts 2024-02-23T00:45:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.297852+0000 osd.2 (osd.2) 336 : cluster [DBG] 5.8 scrub ok 2024-02-23T00:45:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.370394+0000 osd.6 (osd.6) 390 : cluster [DBG] 22.5 scrub starts 2024-02-23T00:45:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.371520+0000 osd.6 (osd.6) 391 : cluster [DBG] 22.5 scrub ok 2024-02-23T00:45:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:58.128241+0000 osd.4 (osd.4) 391 : cluster [DBG] 4.c scrub starts 2024-02-23T00:45:58.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:58 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:58.130369+0000 osd.4 (osd.4) 392 : cluster [DBG] 4.c scrub ok 2024-02-23T00:45:58.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.914+0000 7fb0f746e700 1 -- 172.21.15.87:0/591174886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0f01044a0 msgr2=0x7fb0f0106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:58.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.914+0000 7fb0f746e700 1 --2- 172.21.15.87:0/591174886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0f01044a0 0x7fb0f0106890 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 rx=0x7fb0ec004870 tx=0x7fb0ec009cb0).stop 2024-02-23T00:45:58.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.914+0000 7fb0f746e700 1 -- 172.21.15.87:0/591174886 shutdown_connections 2024-02-23T00:45:58.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.914+0000 7fb0f746e700 1 --2- 172.21.15.87:0/591174886 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0f01044a0 0x7fb0f0106890 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:58.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.914+0000 7fb0f746e700 1 --2- 172.21.15.87:0/591174886 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb0f006b110 0x7fb0f006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:45:58.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.914+0000 7fb0f746e700 1 --2- 172.21.15.87:0/591174886 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb0f006b9e0 0x7fb0f0103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:58.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.914+0000 7fb0f746e700 1 -- 172.21.15.87:0/591174886 >> 172.21.15.87:0/591174886 conn(0x7fb0f00f3800 msgr2=0x7fb0f00f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:58.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.914+0000 7fb0f746e700 1 -- 172.21.15.87:0/591174886 shutdown_connections 2024-02-23T00:45:58.919 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.914+0000 7fb0f746e700 1 -- 172.21.15.87:0/591174886 wait complete. 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 librados: starting msgr at 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 librados: starting objecter 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 Processor -- start 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 -- start start 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 librados: setting wanted keys 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 librados: calling monclient init 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb0f006b110 0x7fb0f01101b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0f006b9e0 0x7fb0f01106a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb0f01044a0 0x7fb0f010a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f010a880 con 0x7fb0f006b9e0 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f010a9f0 con 0x7fb0f006b110 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f746e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f010ab60 con 0x7fb0f01044a0 2024-02-23T00:45:58.920 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f5a0b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb0f01044a0 0x7fb0f010a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f4a09700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0f006b9e0 0x7fb0f01106a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f5a0b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb0f01044a0 0x7fb0f010a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:42638/0 (socket says 172.21.15.87:42638) 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f4a09700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0f006b9e0 0x7fb0f01106a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:54932/0 (socket says 172.21.15.87:54932) 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f5a0b700 1 -- 172.21.15.87:0/4254668613 learned_addr learned my addr 172.21.15.87:0/4254668613 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f520a700 1 --2- 172.21.15.87:0/4254668613 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb0f006b110 0x7fb0f01101b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f5a0b700 1 -- 172.21.15.87:0/4254668613 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb0f006b110 msgr2=0x7fb0f01101b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f5a0b700 1 --2- 172.21.15.87:0/4254668613 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb0f006b110 0x7fb0f01101b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f5a0b700 1 -- 172.21.15.87:0/4254668613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0f006b9e0 msgr2=0x7fb0f01106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f5a0b700 1 --2- 172.21.15.87:0/4254668613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0f006b9e0 0x7fb0f01106a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f5a0b700 1 -- 172.21.15.87:0/4254668613 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e0005040 con 0x7fb0f01044a0 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f4a09700 1 --2- 172.21.15.87:0/4254668613 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb0f006b9e0 0x7fb0f01106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f520a700 1 --2- 172.21.15.87:0/4254668613 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb0f006b110 0x7fb0f01101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0f5a0b700 1 --2- 172.21.15.87:0/4254668613 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb0f01044a0 0x7fb0f010a390 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 rx=0x7fb0ec014ec0 tx=0x7fb0ec00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0e67fc700 10 client.?.objecter ms_handle_connect 0x7fb0f01044a0 2024-02-23T00:45:58.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.915+0000 7fb0e67fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.916+0000 7fb0e67fc700 1 -- 172.21.15.87:0/4254668613 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec00e030 con 0x7fb0f01044a0 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.916+0000 7fb0e67fc700 1 -- 172.21.15.87:0/4254668613 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0ec009710 con 0x7fb0f01044a0 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.916+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec005040 con 0x7fb0f01044a0 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.916+0000 7fb0e67fc700 1 -- 172.21.15.87:0/4254668613 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec019070 con 0x7fb0f01044a0 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.916+0000 7fb0f746e700 10 client.38585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.916+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f010b000 con 0x7fb0f01044a0 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.916+0000 7fb0f746e700 1 librados: init done 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0f746e700 10 client.38585.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0f746e700 15 client.38585.objecter _session_command_op_assign -1 1 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0f746e700 20 client.38585.objecter _get_session osd=-1 returning homeless 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0f746e700 10 client.38585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:45:58.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb0f010b220 con 0x7fb0f01044a0 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0e67fc700 1 -- 172.21.15.87:0/4254668613 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb0ec01e420 con 0x7fb0f01044a0 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0e67fc700 1 --2- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb0dc062c90 0x7fb0dc065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0f520a700 1 --2- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb0dc062c90 0x7fb0dc065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0e67fc700 1 -- 172.21.15.87:0/4254668613 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(851..851 src has 400..851) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fb0ec07e250 con 0x7fb0f01044a0 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0e67fc700 10 client.38585.objecter ms_dispatch 0x7fb0f0001550 osd_map(851..851 src has 400..851) v4 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0e67fc700 3 client.38585.objecter handle_osd_map got epochs [851,851] > 0 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.917+0000 7fb0e67fc700 3 client.38585.objecter handle_osd_map decoding full epoch 851 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 10 client.38585.objecter checking command 1 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 1 --2- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb0dc06e0f0 0x7fb0dc070590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:45:58.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 20 client.38585.objecter _get_session s=0x7fb0dc06da40 osd=2 3 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 20 client.38585.objecter put_session s=0x7fb0dc06da40 osd=2 3 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 15 client.38585.objecter _session_command_op_remove -1 1 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 20 client.38585.objecter _get_session s=0x7fb0dc06da40 osd=2 3 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 20 client.38585.objecter get_session s=0x7fb0dc06da40 osd=2 3 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 15 client.38585.objecter _session_command_op_assign 2 1 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 20 client.38585.objecter put_session s=0x7fb0dc06da40 osd=2 4 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 10 client.38585.objecter _send_command 1 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 1 -- 172.21.15.87:0/4254668613 --> v2:172.21.15.87:6809/3605394228 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0dc070bd0 con 0x7fb0dc06e0f0 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 20 client.38585.objecter dump_active .. 0 homeless 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 1 -- 172.21.15.87:0/4254668613 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0ec080870 con 0x7fb0f01044a0 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0f4a09700 1 --2- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb0dc06e0f0 0x7fb0dc070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 10 client.38585.objecter ms_handle_connect 0x7fb0dc062c90 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0f520a700 1 --2- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb0dc062c90 0x7fb0dc065150 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7fb0e8007fd0 tx=0x7fb0e80074f0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:58.925 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0e67fc700 10 client.38585.objecter ms_handle_connect 0x7fb0dc06e0f0 2024-02-23T00:45:58.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.918+0000 7fb0f4a09700 1 --2- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb0dc06e0f0 0x7fb0dc070590 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:45:58.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.919+0000 7fb0e67fc700 1 -- 172.21.15.87:0/4254668613 <== osd.2 v2:172.21.15.87:6809/3605394228 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7fb0dc070bd0 con 0x7fb0dc06e0f0 2024-02-23T00:45:58.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.919+0000 7fb0e67fc700 10 client.38585.objecter ms_dispatch 0x7fb0f0001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:45:58.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.919+0000 7fb0e67fc700 10 client.38585.objecter _finish_command 1 = system:0 2024-02-23T00:45:58.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.919+0000 7fb0e67fc700 20 client.38585.objecter put_session s=0x7fb0dc06da40 osd=2 4 2024-02-23T00:45:58.926 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.919+0000 7fb0e67fc700 15 client.38585.objecter _session_command_op_remove 2 1 2024-02-23T00:45:58.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 10 client.38585.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:45:58.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 15 client.38585.objecter _session_command_op_assign -1 2 2024-02-23T00:45:58.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 20 client.38585.objecter _get_session s=0x7fb0dc06da40 osd=2 3 2024-02-23T00:45:58.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 20 client.38585.objecter put_session s=0x7fb0dc06da40 osd=2 3 2024-02-23T00:45:58.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 20 client.38585.objecter _get_session s=0x7fb0dc06da40 osd=2 3 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 15 client.38585.objecter _session_command_op_remove -1 2 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 20 client.38585.objecter get_session s=0x7fb0dc06da40 osd=2 3 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 15 client.38585.objecter _session_command_op_assign 2 2 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 20 client.38585.objecter put_session s=0x7fb0dc06da40 osd=2 4 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 10 client.38585.objecter _send_command 2 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.932+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 --> v2:172.21.15.87:6809/3605394228 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb0f005ee80 con 0x7fb0dc06e0f0 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0e67fc700 1 -- 172.21.15.87:0/4254668613 <== osd.2 v2:172.21.15.87:6809/3605394228 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb0f005ee80 con 0x7fb0dc06e0f0 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0e67fc700 10 client.38585.objecter ms_dispatch 0x7fb0f0001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0e67fc700 10 client.38585.objecter _finish_command 2 = system:0 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0e67fc700 20 client.38585.objecter put_session s=0x7fb0dc06da40 osd=2 4 2024-02-23T00:45:58.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0e67fc700 15 client.38585.objecter _session_command_op_remove 2 2 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 10 librados: watch_flush enter 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 10 librados: watch_flush exit 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 10 client.38585.objecter close_session for osd.2 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb0dc06e0f0 msgr2=0x7fb0dc070590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 --2- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6809/3605394228 conn(0x7fb0dc06e0f0 0x7fb0dc070590 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 20 client.38585.objecter put_session s=0x7fb0dc06da40 osd=2 1 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 20 client.38585.objecter shutdown clearing up homeless session... 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 10 client.38585.objecter successfully canceled tick 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb0dc062c90 msgr2=0x7fb0dc065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 --2- 172.21.15.87:0/4254668613 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb0dc062c90 0x7fb0dc065150 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 rx=0x7fb0e8007fd0 tx=0x7fb0e80074f0).stop 2024-02-23T00:45:58.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb0f01044a0 msgr2=0x7fb0f010a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:45:58.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 --2- 172.21.15.87:0/4254668613 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb0f01044a0 0x7fb0f010a390 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 rx=0x7fb0ec014ec0 tx=0x7fb0ec00b040).stop 2024-02-23T00:45:58.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f5a0b700 1 -- 172.21.15.87:0/4254668613 reap_dead start 2024-02-23T00:45:58.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 shutdown_connections 2024-02-23T00:45:58.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 >> 172.21.15.87:0/4254668613 conn(0x7fb0f00f3800 msgr2=0x7fb0f00f4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:45:58.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 shutdown_connections 2024-02-23T00:45:58.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 -- 172.21.15.87:0/4254668613 wait complete. 2024-02-23T00:45:58.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:45:58.933+0000 7fb0f746e700 1 librados: shutdown 2024-02-23T00:45:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:57.840216+0000 mgr.y (mgr.24437) 748 : cluster [DBG] pgmap v1337: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:58.312270+0000 osd.2 (osd.2) 337 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:45:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:58.314362+0000 osd.2 (osd.2) 338 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:45:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:59.108442+0000 osd.5 (osd.5) 373 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:45:59.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:59.110132+0000 osd.5 (osd.5) 374 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:45:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:57.840216+0000 mgr.y (mgr.24437) 748 : cluster [DBG] pgmap v1337: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:58.312270+0000 osd.2 (osd.2) 337 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:45:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:58.314362+0000 osd.2 (osd.2) 338 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:45:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:59.108442+0000 osd.5 (osd.5) 373 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:45:59.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:45:59 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:59.110132+0000 osd.5 (osd.5) 374 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:45:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:57.840216+0000 mgr.y (mgr.24437) 748 : cluster [DBG] pgmap v1337: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:45:59.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:58.312270+0000 osd.2 (osd.2) 337 : cluster [DBG] 22.15 scrub starts 2024-02-23T00:45:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:58.314362+0000 osd.2 (osd.2) 338 : cluster [DBG] 22.15 scrub ok 2024-02-23T00:45:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:59.108442+0000 osd.5 (osd.5) 373 : cluster [DBG] 2.2 scrub starts 2024-02-23T00:45:59.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:45:59 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:59.110132+0000 osd.5 (osd.5) 374 : cluster [DBG] 2.2 scrub ok 2024-02-23T00:45:59.749 INFO:teuthology.orchestra.run.smithi087.stdout:77309411580 2024-02-23T00:45:59.749 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-23T00:46:00.018 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:00.545 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.061173+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:46:00.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.062780+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:46:00.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.172877+0000 osd.4 (osd.4) 393 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:46:00.546 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.176866+0000 osd.4 (osd.4) 394 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:46:00.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.061173+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:46:00.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.062780+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:46:00.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.172877+0000 osd.4 (osd.4) 393 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:46:00.546 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.176866+0000 osd.4 (osd.4) 394 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:46:00.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.061173+0000 osd.5 (osd.5) 375 : cluster [DBG] 5.15 scrub starts 2024-02-23T00:46:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.062780+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.15 scrub ok 2024-02-23T00:46:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.172877+0000 osd.4 (osd.4) 393 : cluster [DBG] 22.17 scrub starts 2024-02-23T00:46:00.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:00 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.176866+0000 osd.4 (osd.4) 394 : cluster [DBG] 22.17 scrub ok 2024-02-23T00:46:01.662 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:45:59.840696+0000 mgr.y (mgr.24437) 749 : cluster [DBG] pgmap v1338: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:46:01.662 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.030942+0000 osd.0 (osd.0) 111 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:46:01.662 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.034667+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.290837+0000 osd.1 (osd.1) 442 : cluster [DBG] 2.d deep-scrub starts 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.293761+0000 osd.1 (osd.1) 443 : cluster [DBG] 2.d deep-scrub ok 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.314458+0000 osd.6 (osd.6) 392 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.317335+0000 osd.6 (osd.6) 393 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.680640+0000 osd.7 (osd.7) 438 : cluster [DBG] 3.13 scrub starts 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:00.681821+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.13 scrub ok 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:45:59.840696+0000 mgr.y (mgr.24437) 749 : cluster [DBG] pgmap v1338: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.030942+0000 osd.0 (osd.0) 111 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.034667+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:46:01.663 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.290837+0000 osd.1 (osd.1) 442 : cluster [DBG] 2.d deep-scrub starts 2024-02-23T00:46:01.664 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.293761+0000 osd.1 (osd.1) 443 : cluster [DBG] 2.d deep-scrub ok 2024-02-23T00:46:01.664 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.314458+0000 osd.6 (osd.6) 392 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:46:01.664 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.317335+0000 osd.6 (osd.6) 393 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:46:01.664 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.680640+0000 osd.7 (osd.7) 438 : cluster [DBG] 3.13 scrub starts 2024-02-23T00:46:01.664 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:01 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:00.681821+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.13 scrub ok 2024-02-23T00:46:01.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:46:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:45:59.840696+0000 mgr.y (mgr.24437) 749 : cluster [DBG] pgmap v1338: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-23T00:46:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.030942+0000 osd.0 (osd.0) 111 : cluster [DBG] 4.1e scrub starts 2024-02-23T00:46:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.034667+0000 osd.0 (osd.0) 112 : cluster [DBG] 4.1e scrub ok 2024-02-23T00:46:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.290837+0000 osd.1 (osd.1) 442 : cluster [DBG] 2.d deep-scrub starts 2024-02-23T00:46:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.293761+0000 osd.1 (osd.1) 443 : cluster [DBG] 2.d deep-scrub ok 2024-02-23T00:46:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.314458+0000 osd.6 (osd.6) 392 : cluster [DBG] 5.19 scrub starts 2024-02-23T00:46:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.317335+0000 osd.6 (osd.6) 393 : cluster [DBG] 5.19 scrub ok 2024-02-23T00:46:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.680640+0000 osd.7 (osd.7) 438 : cluster [DBG] 3.13 scrub starts 2024-02-23T00:46:01.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:01 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:00.681821+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.13 scrub ok 2024-02-23T00:46:01.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.884+0000 7f612cad6700 1 -- 172.21.15.87:0/2699461514 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61280fe510 msgr2=0x7f61280f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:01.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.884+0000 7f612cad6700 1 --2- 172.21.15.87:0/2699461514 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61280fe510 0x7f61280f7d40 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 rx=0x7f6118004870 tx=0x7f6118009cb0).stop 2024-02-23T00:46:01.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.884+0000 7f612cad6700 1 -- 172.21.15.87:0/2699461514 shutdown_connections 2024-02-23T00:46:01.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.884+0000 7f612cad6700 1 --2- 172.21.15.87:0/2699461514 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61280f8420 0x7f6128104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:01.903 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.884+0000 7f612cad6700 1 --2- 172.21.15.87:0/2699461514 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61280fdbb0 0x7f61280fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.884+0000 7f612cad6700 1 --2- 172.21.15.87:0/2699461514 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61280fe510 0x7f61280f7d40 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.884+0000 7f612cad6700 1 -- 172.21.15.87:0/2699461514 >> 172.21.15.87:0/2699461514 conn(0x7f61280f3800 msgr2=0x7f61280f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.884+0000 7f612cad6700 1 -- 172.21.15.87:0/2699461514 shutdown_connections 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.884+0000 7f612cad6700 1 -- 172.21.15.87:0/2699461514 wait complete. 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 librados: starting msgr at 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 librados: starting objecter 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 Processor -- start 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 -- start start 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 librados: setting wanted keys 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 librados: calling monclient init 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61280f8420 0x7f612810e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:01.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61280fdbb0 0x7f612810e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61280fe510 0x7f61281081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61281086b0 con 0x7f61280fe510 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6128108820 con 0x7f61280fdbb0 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612cad6700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61281089e0 con 0x7f61280f8420 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.885+0000 7f612659c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61280f8420 0x7f612810e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f6126d9d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61280fe510 0x7f61281081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f612659c700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61280f8420 0x7f612810e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50904/0 (socket says 172.21.15.87:50904) 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f612659c700 1 -- 172.21.15.87:0/1251109699 learned_addr learned my addr 172.21.15.87:0/1251109699 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f6125d9b700 1 --2- 172.21.15.87:0/1251109699 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61280fdbb0 0x7f612810e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f6126d9d700 1 -- 172.21.15.87:0/1251109699 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61280f8420 msgr2=0x7f612810e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f6126d9d700 1 --2- 172.21.15.87:0/1251109699 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61280f8420 0x7f612810e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f6126d9d700 1 -- 172.21.15.87:0/1251109699 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61280fdbb0 msgr2=0x7f612810e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f6126d9d700 1 --2- 172.21.15.87:0/1251109699 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f61280fdbb0 0x7f612810e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:01.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f6126d9d700 1 -- 172.21.15.87:0/1251109699 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61200097c0 con 0x7f61280fe510 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f612659c700 1 --2- 172.21.15.87:0/1251109699 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f61280f8420 0x7f612810e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f6126d9d700 1 --2- 172.21.15.87:0/1251109699 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61280fe510 0x7f61281081c0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f6120001cb0 tx=0x7f6120004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f61177fe700 10 client.?.objecter ms_handle_connect 0x7f61280fe510 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f61177fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f61177fe700 1 -- 172.21.15.87:0/1251109699 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61200079b0 con 0x7f61280fe510 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6118005040 con 0x7f61280fe510 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f61177fe700 1 -- 172.21.15.87:0/1251109699 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61200090d0 con 0x7f61280fe510 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f61177fe700 1 -- 172.21.15.87:0/1251109699 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612000fc20 con 0x7f61280fe510 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f612cad6700 10 client.28113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6128108fd0 con 0x7f61280fe510 2024-02-23T00:46:01.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.886+0000 7f612cad6700 1 librados: init done 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.887+0000 7f612cad6700 10 client.28113.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.887+0000 7f612cad6700 15 client.28113.objecter _session_command_op_assign -1 1 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.887+0000 7f612cad6700 20 client.28113.objecter _get_session osd=-1 returning homeless 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.887+0000 7f612cad6700 10 client.28113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.887+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f612805ee80 con 0x7f61280fe510 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.887+0000 7f61177fe700 1 -- 172.21.15.87:0/1251109699 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f612000fd80 con 0x7f61280fe510 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 1 --2- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6110062c90 0x7f6110065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 1 -- 172.21.15.87:0/1251109699 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(851..851 src has 400..851) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f612007d800 con 0x7f61280fe510 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 10 client.28113.objecter ms_dispatch 0x7f6128001550 osd_map(851..851 src has 400..851) v4 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f612659c700 1 --2- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6110062c90 0x7f6110065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 3 client.28113.objecter handle_osd_map got epochs [851,851] > 0 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 3 client.28113.objecter handle_osd_map decoding full epoch 851 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 10 client.28113.objecter checking command 1 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 1 --2- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6813/128059790 conn(0x7f611006e0f0 0x7f6110070590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:01.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 20 client.28113.objecter _get_session s=0x7f611006da40 osd=3 3 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 20 client.28113.objecter put_session s=0x7f611006da40 osd=3 3 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 15 client.28113.objecter _session_command_op_remove -1 1 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 20 client.28113.objecter _get_session s=0x7f611006da40 osd=3 3 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 20 client.28113.objecter get_session s=0x7f611006da40 osd=3 3 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 15 client.28113.objecter _session_command_op_assign 3 1 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 20 client.28113.objecter put_session s=0x7f611006da40 osd=3 4 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 10 client.28113.objecter _send_command 1 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 1 -- 172.21.15.87:0/1251109699 --> v2:172.21.15.87:6813/128059790 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6110070bd0 con 0x7f611006e0f0 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 20 client.28113.objecter dump_active .. 0 homeless 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 1 -- 172.21.15.87:0/1251109699 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6120082220 con 0x7f61280fe510 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f6125d9b700 1 --2- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6813/128059790 conn(0x7f611006e0f0 0x7f6110070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 10 client.28113.objecter ms_handle_connect 0x7f6110062c90 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f612659c700 1 --2- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6110062c90 0x7f6110065150 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 rx=0x7f611c004fd0 tx=0x7f611c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:01.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f61177fe700 10 client.28113.objecter ms_handle_connect 0x7f611006e0f0 2024-02-23T00:46:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.888+0000 7f6125d9b700 1 --2- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6813/128059790 conn(0x7f611006e0f0 0x7f6110070590 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.889+0000 7f61177fe700 1 -- 172.21.15.87:0/1251109699 <== osd.3 v2:172.21.15.87:6813/128059790 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f6110070bd0 con 0x7f611006e0f0 2024-02-23T00:46:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.889+0000 7f61177fe700 10 client.28113.objecter ms_dispatch 0x7f6128001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:46:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.889+0000 7f61177fe700 10 client.28113.objecter _finish_command 1 = system:0 2024-02-23T00:46:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.889+0000 7f61177fe700 20 client.28113.objecter put_session s=0x7f611006da40 osd=3 4 2024-02-23T00:46:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.889+0000 7f61177fe700 15 client.28113.objecter _session_command_op_remove 3 1 2024-02-23T00:46:01.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 10 client.28113.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 15 client.28113.objecter _session_command_op_assign -1 2 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 20 client.28113.objecter _get_session s=0x7f611006da40 osd=3 3 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 20 client.28113.objecter put_session s=0x7f611006da40 osd=3 3 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 20 client.28113.objecter _get_session s=0x7f611006da40 osd=3 3 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 15 client.28113.objecter _session_command_op_remove -1 2 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 20 client.28113.objecter get_session s=0x7f611006da40 osd=3 3 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 15 client.28113.objecter _session_command_op_assign 3 2 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 20 client.28113.objecter put_session s=0x7f611006da40 osd=3 4 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 10 client.28113.objecter _send_command 2 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.903+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 --> v2:172.21.15.87:6813/128059790 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f61281070a0 con 0x7f611006e0f0 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f61177fe700 1 -- 172.21.15.87:0/1251109699 <== osd.3 v2:172.21.15.87:6813/128059790 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f61281070a0 con 0x7f611006e0f0 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f61177fe700 10 client.28113.objecter ms_dispatch 0x7f6128001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f61177fe700 10 client.28113.objecter _finish_command 2 = system:0 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f61177fe700 20 client.28113.objecter put_session s=0x7f611006da40 osd=3 4 2024-02-23T00:46:01.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f61177fe700 15 client.28113.objecter _session_command_op_remove 3 2 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 10 librados: watch_flush enter 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 10 librados: watch_flush exit 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 10 client.28113.objecter close_session for osd.3 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6813/128059790 conn(0x7f611006e0f0 msgr2=0x7f6110070590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 --2- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6813/128059790 conn(0x7f611006e0f0 0x7f6110070590 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 20 client.28113.objecter put_session s=0x7f611006da40 osd=3 1 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 20 client.28113.objecter shutdown clearing up homeless session... 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 10 client.28113.objecter successfully canceled tick 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6110062c90 msgr2=0x7f6110065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 --2- 172.21.15.87:0/1251109699 >> v2:172.21.15.87:6800/1231597463 conn(0x7f6110062c90 0x7f6110065150 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 rx=0x7f611c004fd0 tx=0x7f611c00b040).stop 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61280fe510 msgr2=0x7f61281081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 --2- 172.21.15.87:0/1251109699 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f61280fe510 0x7f61281081c0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 rx=0x7f6120001cb0 tx=0x7f6120004c50).stop 2024-02-23T00:46:01.912 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f6126d9d700 1 -- 172.21.15.87:0/1251109699 reap_dead start 2024-02-23T00:46:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 shutdown_connections 2024-02-23T00:46:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 >> 172.21.15.87:0/1251109699 conn(0x7f61280f3800 msgr2=0x7f61280f53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 shutdown_connections 2024-02-23T00:46:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 -- 172.21.15.87:0/1251109699 wait complete. 2024-02-23T00:46:01.913 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:01.904+0000 7f612cad6700 1 librados: shutdown 2024-02-23T00:46:02.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:02] "GET /metrics HTTP/1.1" 200 186060 "" "Prometheus/2.33.4" 2024-02-23T00:46:02.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:01.290377+0000 osd.1 (osd.1) 444 : cluster [DBG] 22.2 deep-scrub starts 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:01.293598+0000 osd.1 (osd.1) 445 : cluster [DBG] 22.2 deep-scrub ok 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:01.346103+0000 osd.6 (osd.6) 394 : cluster [DBG] 3.a scrub starts 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:01.349000+0000 osd.6 (osd.6) 395 : cluster [DBG] 3.a scrub ok 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:02.173497+0000 osd.4 (osd.4) 395 : cluster [DBG] 5.e deep-scrub starts 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:02.177540+0000 osd.4 (osd.4) 396 : cluster [DBG] 5.e deep-scrub ok 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:01.290377+0000 osd.1 (osd.1) 444 : cluster [DBG] 22.2 deep-scrub starts 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:01.293598+0000 osd.1 (osd.1) 445 : cluster [DBG] 22.2 deep-scrub ok 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:01.346103+0000 osd.6 (osd.6) 394 : cluster [DBG] 3.a scrub starts 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:01.349000+0000 osd.6 (osd.6) 395 : cluster [DBG] 3.a scrub ok 2024-02-23T00:46:02.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:02.173497+0000 osd.4 (osd.4) 395 : cluster [DBG] 5.e deep-scrub starts 2024-02-23T00:46:02.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:02 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:02.177540+0000 osd.4 (osd.4) 396 : cluster [DBG] 5.e deep-scrub ok 2024-02-23T00:46:02.691 INFO:teuthology.orchestra.run.smithi087.stdout:111669149945 2024-02-23T00:46:02.692 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-23T00:46:02.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:01.290377+0000 osd.1 (osd.1) 444 : cluster [DBG] 22.2 deep-scrub starts 2024-02-23T00:46:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:01.293598+0000 osd.1 (osd.1) 445 : cluster [DBG] 22.2 deep-scrub ok 2024-02-23T00:46:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:01.346103+0000 osd.6 (osd.6) 394 : cluster [DBG] 3.a scrub starts 2024-02-23T00:46:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:01.349000+0000 osd.6 (osd.6) 395 : cluster [DBG] 3.a scrub ok 2024-02-23T00:46:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:02.173497+0000 osd.4 (osd.4) 395 : cluster [DBG] 5.e deep-scrub starts 2024-02-23T00:46:02.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:02 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:02.177540+0000 osd.4 (osd.4) 396 : cluster [DBG] 5.e deep-scrub ok 2024-02-23T00:46:02.961 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:03.493 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:01.438608+0000 mgr.y (mgr.24437) 750 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:03.493 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:01.841657+0000 mgr.y (mgr.24437) 751 : cluster [DBG] pgmap v1339: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:03.494 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:03.076324+0000 osd.5 (osd.5) 377 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:46:03.494 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:03.077250+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:46:03.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:01.438608+0000 mgr.y (mgr.24437) 750 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:03.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:01.841657+0000 mgr.y (mgr.24437) 751 : cluster [DBG] pgmap v1339: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:03.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:03.076324+0000 osd.5 (osd.5) 377 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:46:03.494 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:03.077250+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:46:03.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:01.438608+0000 mgr.y (mgr.24437) 750 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:01.841657+0000 mgr.y (mgr.24437) 751 : cluster [DBG] pgmap v1339: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:03.076324+0000 osd.5 (osd.5) 377 : cluster [DBG] 3.5 scrub starts 2024-02-23T00:46:03.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:03.077250+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.5 scrub ok 2024-02-23T00:46:04.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:46:03 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:46:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:03.251046+0000 osd.1 (osd.1) 446 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:46:04.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:04 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:03.253935+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:46:04.753 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:03.251046+0000 osd.1 (osd.1) 446 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:46:04.753 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:03.253935+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:46:04.754 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:03.251046+0000 osd.1 (osd.1) 446 : cluster [DBG] 2.4 scrub starts 2024-02-23T00:46:04.754 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:04 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:03.253935+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.4 scrub ok 2024-02-23T00:46:05.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.037+0000 7f522012d700 1 -- 172.21.15.87:0/2330148039 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f52180fe510 msgr2=0x7f52180f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:05.042 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.037+0000 7f522012d700 1 --2- 172.21.15.87:0/2330148039 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f52180fe510 0x7f52180f7d40 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 rx=0x7f5214004870 tx=0x7f5214009cb0).stop 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.037+0000 7f522012d700 1 -- 172.21.15.87:0/2330148039 shutdown_connections 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.037+0000 7f522012d700 1 --2- 172.21.15.87:0/2330148039 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52180f8420 0x7f5218104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.037+0000 7f522012d700 1 --2- 172.21.15.87:0/2330148039 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52180fdbb0 0x7f52180fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.037+0000 7f522012d700 1 --2- 172.21.15.87:0/2330148039 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f52180fe510 0x7f52180f7d40 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.037+0000 7f522012d700 1 -- 172.21.15.87:0/2330148039 >> 172.21.15.87:0/2330148039 conn(0x7f52180f3800 msgr2=0x7f52180f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.037+0000 7f522012d700 1 -- 172.21.15.87:0/2330148039 shutdown_connections 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.037+0000 7f522012d700 1 -- 172.21.15.87:0/2330148039 wait complete. 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 librados: starting msgr at 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 librados: starting objecter 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 Processor -- start 2024-02-23T00:46:05.043 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 -- start start 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 librados: setting wanted keys 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 librados: calling monclient init 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52180f8420 0x7f521810e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52180fdbb0 0x7f521810e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f52180fe510 0x7f52181081e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52181086d0 con 0x7f52180f8420 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5218108840 con 0x7f52180fdbb0 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f522012d700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5218108a00 con 0x7f52180fe510 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f521dec9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52180f8420 0x7f521810e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:05.044 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f521e6ca700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f52180fe510 0x7f52181081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f521dec9700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52180f8420 0x7f521810e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56876/0 (socket says 172.21.15.87:56876) 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f521dec9700 1 -- 172.21.15.87:0/2656626081 learned_addr learned my addr 172.21.15.87:0/2656626081 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f521d6c8700 1 --2- 172.21.15.87:0/2656626081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52180fdbb0 0x7f521810e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f521dec9700 1 -- 172.21.15.87:0/2656626081 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f52180fe510 msgr2=0x7f52181081e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.038+0000 7f521dec9700 1 --2- 172.21.15.87:0/2656626081 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f52180fe510 0x7f52181081e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f521dec9700 1 -- 172.21.15.87:0/2656626081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52180fdbb0 msgr2=0x7f521810e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f521dec9700 1 --2- 172.21.15.87:0/2656626081 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f52180fdbb0 0x7f521810e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f521dec9700 1 -- 172.21.15.87:0/2656626081 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5208005040 con 0x7f52180f8420 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f521e6ca700 1 --2- 172.21.15.87:0/2656626081 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f52180fe510 0x7f52181081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f521dec9700 1 --2- 172.21.15.87:0/2656626081 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52180f8420 0x7f521810e040 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 rx=0x7f52080048f0 tx=0x7f5208009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:05.045 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f520effd700 10 client.?.objecter ms_handle_connect 0x7f52180f8420 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f520effd700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f520effd700 1 -- 172.21.15.87:0/2656626081 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208007b00 con 0x7f52180f8420 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f520effd700 1 -- 172.21.15.87:0/2656626081 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f520800f040 con 0x7f52180f8420 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5214005040 con 0x7f52180f8420 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f520effd700 1 -- 172.21.15.87:0/2656626081 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5208014990 con 0x7f52180f8420 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f522012d700 10 client.28119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5218109040 con 0x7f52180f8420 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.039+0000 7f522012d700 1 librados: init done 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.040+0000 7f522012d700 10 client.28119.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.040+0000 7f522012d700 15 client.28119.objecter _session_command_op_assign -1 1 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.040+0000 7f522012d700 20 client.28119.objecter _get_session osd=-1 returning homeless 2024-02-23T00:46:05.046 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.040+0000 7f522012d700 10 client.28119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:05.047 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.040+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f521805ee80 con 0x7f52180f8420 2024-02-23T00:46:05.048 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.040+0000 7f520effd700 1 -- 172.21.15.87:0/2656626081 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f5208010500 con 0x7f52180f8420 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 1 --2- 172.21.15.87:0/2656626081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5204062c90 0x7f5204065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 1 -- 172.21.15.87:0/2656626081 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(851..851 src has 400..851) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f520807ce00 con 0x7f52180f8420 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f521d6c8700 1 --2- 172.21.15.87:0/2656626081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5204062c90 0x7f5204065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 10 client.28119.objecter ms_dispatch 0x7f5218001550 osd_map(851..851 src has 400..851) v4 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 3 client.28119.objecter handle_osd_map got epochs [851,851] > 0 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 3 client.28119.objecter handle_osd_map decoding full epoch 851 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 10 client.28119.objecter checking command 1 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 1 --2- 172.21.15.87:0/2656626081 >> v2:172.21.15.171:6800/4175844757 conn(0x7f520406e0f0 0x7f5204070590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 20 client.28119.objecter _get_session s=0x7f520406da40 osd=4 3 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 20 client.28119.objecter put_session s=0x7f520406da40 osd=4 3 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 15 client.28119.objecter _session_command_op_remove -1 1 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 20 client.28119.objecter _get_session s=0x7f520406da40 osd=4 3 2024-02-23T00:46:05.049 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 20 client.28119.objecter get_session s=0x7f520406da40 osd=4 3 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 15 client.28119.objecter _session_command_op_assign 4 1 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 20 client.28119.objecter put_session s=0x7f520406da40 osd=4 4 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 10 client.28119.objecter _send_command 1 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 1 -- 172.21.15.87:0/2656626081 --> v2:172.21.15.171:6800/4175844757 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5204070bd0 con 0x7f520406e0f0 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 20 client.28119.objecter dump_active .. 0 homeless 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 1 -- 172.21.15.87:0/2656626081 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (secure 0 0 0) 0x7f520807d1b0 con 0x7f52180f8420 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f520effd700 10 client.28119.objecter ms_handle_connect 0x7f5204062c90 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f521d6c8700 1 --2- 172.21.15.87:0/2656626081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5204062c90 0x7f5204065150 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 rx=0x7f5214004ee0 tx=0x7f521400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.041+0000 7f521e6ca700 1 --2- 172.21.15.87:0/2656626081 >> v2:172.21.15.171:6800/4175844757 conn(0x7f520406e0f0 0x7f5204070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.042+0000 7f520effd700 10 client.28119.objecter ms_handle_connect 0x7f520406e0f0 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.042+0000 7f521e6ca700 1 --2- 172.21.15.87:0/2656626081 >> v2:172.21.15.171:6800/4175844757 conn(0x7f520406e0f0 0x7f5204070590 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:05.050 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.042+0000 7f520effd700 1 -- 172.21.15.87:0/2656626081 <== osd.4 v2:172.21.15.171:6800/4175844757 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f5204070bd0 con 0x7f520406e0f0 2024-02-23T00:46:05.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.042+0000 7f520effd700 10 client.28119.objecter ms_dispatch 0x7f5218001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:46:05.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.042+0000 7f520effd700 10 client.28119.objecter _finish_command 1 = system:0 2024-02-23T00:46:05.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.042+0000 7f520effd700 20 client.28119.objecter put_session s=0x7f520406da40 osd=4 4 2024-02-23T00:46:05.051 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.042+0000 7f520effd700 15 client.28119.objecter _session_command_op_remove 4 1 2024-02-23T00:46:05.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 10 client.28119.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:46:05.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 15 client.28119.objecter _session_command_op_assign -1 2 2024-02-23T00:46:05.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 20 client.28119.objecter _get_session s=0x7f520406da40 osd=4 3 2024-02-23T00:46:05.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 20 client.28119.objecter put_session s=0x7f520406da40 osd=4 3 2024-02-23T00:46:05.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 20 client.28119.objecter _get_session s=0x7f520406da40 osd=4 3 2024-02-23T00:46:05.059 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 15 client.28119.objecter _session_command_op_remove -1 2 2024-02-23T00:46:05.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 20 client.28119.objecter get_session s=0x7f520406da40 osd=4 3 2024-02-23T00:46:05.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 15 client.28119.objecter _session_command_op_assign 4 2 2024-02-23T00:46:05.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 20 client.28119.objecter put_session s=0x7f520406da40 osd=4 4 2024-02-23T00:46:05.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 10 client.28119.objecter _send_command 2 2024-02-23T00:46:05.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 --> v2:172.21.15.171:6800/4175844757 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f52181070a0 con 0x7f520406e0f0 2024-02-23T00:46:05.060 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f520effd700 1 -- 172.21.15.87:0/2656626081 <== osd.4 v2:172.21.15.171:6800/4175844757 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f52181070a0 con 0x7f520406e0f0 2024-02-23T00:46:05.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f520effd700 10 client.28119.objecter ms_dispatch 0x7f5218001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:46:05.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f520effd700 10 client.28119.objecter _finish_command 2 = system:0 2024-02-23T00:46:05.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f520effd700 20 client.28119.objecter put_session s=0x7f520406da40 osd=4 4 2024-02-23T00:46:05.061 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f520effd700 15 client.28119.objecter _session_command_op_remove 4 2 2024-02-23T00:46:05.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 10 librados: watch_flush enter 2024-02-23T00:46:05.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 10 librados: watch_flush exit 2024-02-23T00:46:05.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 10 client.28119.objecter close_session for osd.4 2024-02-23T00:46:05.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.057+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 >> v2:172.21.15.171:6800/4175844757 conn(0x7f520406e0f0 msgr2=0x7f5204070590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:05.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 --2- 172.21.15.87:0/2656626081 >> v2:172.21.15.171:6800/4175844757 conn(0x7f520406e0f0 0x7f5204070590 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:05.062 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 20 client.28119.objecter put_session s=0x7f520406da40 osd=4 1 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 20 client.28119.objecter shutdown clearing up homeless session... 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 10 client.28119.objecter successfully canceled tick 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5204062c90 msgr2=0x7f5204065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 --2- 172.21.15.87:0/2656626081 >> v2:172.21.15.87:6800/1231597463 conn(0x7f5204062c90 0x7f5204065150 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 rx=0x7f5214004ee0 tx=0x7f521400b040).stop 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52180f8420 msgr2=0x7f521810e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 --2- 172.21.15.87:0/2656626081 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f52180f8420 0x7f521810e040 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 rx=0x7f52080048f0 tx=0x7f5208009cb0).stop 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f521e6ca700 1 -- 172.21.15.87:0/2656626081 reap_dead start 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 shutdown_connections 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 >> 172.21.15.87:0/2656626081 conn(0x7f52180f3800 msgr2=0x7f52180f5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 shutdown_connections 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 -- 172.21.15.87:0/2656626081 wait complete. 2024-02-23T00:46:05.063 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:05.058+0000 7f522012d700 1 librados: shutdown 2024-02-23T00:46:05.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:03.842166+0000 mgr.y (mgr.24437) 752 : cluster [DBG] pgmap v1340: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:46:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:04.134442+0000 osd.3 (osd.3) 329 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:46:05.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:05 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:04.136486+0000 osd.3 (osd.3) 330 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:46:05.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:03.842166+0000 mgr.y (mgr.24437) 752 : cluster [DBG] pgmap v1340: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:46:05.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:04.134442+0000 osd.3 (osd.3) 329 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:46:05.778 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:04.136486+0000 osd.3 (osd.3) 330 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:46:05.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:03.842166+0000 mgr.y (mgr.24437) 752 : cluster [DBG] pgmap v1340: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-23T00:46:05.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:04.134442+0000 osd.3 (osd.3) 329 : cluster [DBG] 22.7 scrub starts 2024-02-23T00:46:05.779 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:04.136486+0000 osd.3 (osd.3) 330 : cluster [DBG] 22.7 scrub ok 2024-02-23T00:46:05.796 INFO:teuthology.orchestra.run.smithi087.stdout:133143986420 2024-02-23T00:46:05.796 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-23T00:46:06.063 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:06.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:05.126298+0000 osd.3 (osd.3) 331 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:46:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:05.129756+0000 osd.3 (osd.3) 332 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:46:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:05.280148+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.a scrub starts 2024-02-23T00:46:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:05.283345+0000 osd.1 (osd.1) 449 : cluster [DBG] 2.a scrub ok 2024-02-23T00:46:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:05.659058+0000 osd.7 (osd.7) 440 : cluster [DBG] 3.d scrub starts 2024-02-23T00:46:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:05.662032+0000 osd.7 (osd.7) 441 : cluster [DBG] 3.d scrub ok 2024-02-23T00:46:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:06.108912+0000 osd.4 (osd.4) 397 : cluster [DBG] 2.f scrub starts 2024-02-23T00:46:06.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:06.112484+0000 osd.4 (osd.4) 398 : cluster [DBG] 2.f scrub ok 2024-02-23T00:46:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:05.126298+0000 osd.3 (osd.3) 331 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:46:06.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:05.129756+0000 osd.3 (osd.3) 332 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:46:06.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:05.280148+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.a scrub starts 2024-02-23T00:46:06.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:05.283345+0000 osd.1 (osd.1) 449 : cluster [DBG] 2.a scrub ok 2024-02-23T00:46:06.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:05.659058+0000 osd.7 (osd.7) 440 : cluster [DBG] 3.d scrub starts 2024-02-23T00:46:06.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:05.662032+0000 osd.7 (osd.7) 441 : cluster [DBG] 3.d scrub ok 2024-02-23T00:46:06.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:06.108912+0000 osd.4 (osd.4) 397 : cluster [DBG] 2.f scrub starts 2024-02-23T00:46:06.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:06 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:06.112484+0000 osd.4 (osd.4) 398 : cluster [DBG] 2.f scrub ok 2024-02-23T00:46:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:05.126298+0000 osd.3 (osd.3) 331 : cluster [DBG] 5.14 scrub starts 2024-02-23T00:46:06.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:05.129756+0000 osd.3 (osd.3) 332 : cluster [DBG] 5.14 scrub ok 2024-02-23T00:46:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:05.280148+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.a scrub starts 2024-02-23T00:46:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:05.283345+0000 osd.1 (osd.1) 449 : cluster [DBG] 2.a scrub ok 2024-02-23T00:46:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:05.659058+0000 osd.7 (osd.7) 440 : cluster [DBG] 3.d scrub starts 2024-02-23T00:46:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:05.662032+0000 osd.7 (osd.7) 441 : cluster [DBG] 3.d scrub ok 2024-02-23T00:46:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:06.108912+0000 osd.4 (osd.4) 397 : cluster [DBG] 2.f scrub starts 2024-02-23T00:46:06.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:06 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:06.112484+0000 osd.4 (osd.4) 398 : cluster [DBG] 2.f scrub ok 2024-02-23T00:46:07.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:05.842879+0000 mgr.y (mgr.24437) 753 : cluster [DBG] pgmap v1341: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:07.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:06.299889+0000 osd.1 (osd.1) 450 : cluster [DBG] 2.9 deep-scrub starts 2024-02-23T00:46:07.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:06.300876+0000 osd.1 (osd.1) 451 : cluster [DBG] 2.9 deep-scrub ok 2024-02-23T00:46:07.542 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:07.101805+0000 osd.4 (osd.4) 399 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:46:07.543 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:07.103027+0000 osd.4 (osd.4) 400 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:46:07.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:05.842879+0000 mgr.y (mgr.24437) 753 : cluster [DBG] pgmap v1341: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:07.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:06.299889+0000 osd.1 (osd.1) 450 : cluster [DBG] 2.9 deep-scrub starts 2024-02-23T00:46:07.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:06.300876+0000 osd.1 (osd.1) 451 : cluster [DBG] 2.9 deep-scrub ok 2024-02-23T00:46:07.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:07.101805+0000 osd.4 (osd.4) 399 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:46:07.543 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:07 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:07.103027+0000 osd.4 (osd.4) 400 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:46:07.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:05.842879+0000 mgr.y (mgr.24437) 753 : cluster [DBG] pgmap v1341: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:06.299889+0000 osd.1 (osd.1) 450 : cluster [DBG] 2.9 deep-scrub starts 2024-02-23T00:46:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:06.300876+0000 osd.1 (osd.1) 451 : cluster [DBG] 2.9 deep-scrub ok 2024-02-23T00:46:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:07.101805+0000 osd.4 (osd.4) 399 : cluster [DBG] 5.1 scrub starts 2024-02-23T00:46:07.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:07 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:07.103027+0000 osd.4 (osd.4) 400 : cluster [DBG] 5.1 scrub ok 2024-02-23T00:46:07.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.784+0000 7f314c397700 1 -- 172.21.15.87:0/2035215676 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31440fe510 msgr2=0x7f31440f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:07.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.784+0000 7f314c397700 1 --2- 172.21.15.87:0/2035215676 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31440fe510 0x7f31440f7d40 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 rx=0x7f3140004870 tx=0x7f3140009cb0).stop 2024-02-23T00:46:07.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.784+0000 7f314c397700 1 -- 172.21.15.87:0/2035215676 shutdown_connections 2024-02-23T00:46:07.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.784+0000 7f314c397700 1 --2- 172.21.15.87:0/2035215676 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f31440f8420 0x7f3144104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:07.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.784+0000 7f314c397700 1 --2- 172.21.15.87:0/2035215676 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31440fdbb0 0x7f31440fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:46:07.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.784+0000 7f314c397700 1 --2- 172.21.15.87:0/2035215676 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31440fe510 0x7f31440f7d40 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:07.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.784+0000 7f314c397700 1 -- 172.21.15.87:0/2035215676 >> 172.21.15.87:0/2035215676 conn(0x7f31440f3800 msgr2=0x7f31440f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:07.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.784+0000 7f314c397700 1 -- 172.21.15.87:0/2035215676 shutdown_connections 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.784+0000 7f314c397700 1 -- 172.21.15.87:0/2035215676 wait complete. 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 librados: starting msgr at 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 librados: starting objecter 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 Processor -- start 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 -- start start 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 librados: setting wanted keys 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 librados: calling monclient init 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31440f8420 0x7f314410e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f31440fdbb0 0x7f314410e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31440fe510 0x7f3144108280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3144108770 con 0x7f31440fe510 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31441088e0 con 0x7f31440f8420 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314c397700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3144108aa0 con 0x7f31440fdbb0 2024-02-23T00:46:07.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f3149932700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f31440fdbb0 0x7f314410e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f314a934700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31440fe510 0x7f3144108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f3149932700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f31440fdbb0 0x7f314410e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50930/0 (socket says 172.21.15.87:50930) 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.785+0000 7f3149932700 1 -- 172.21.15.87:0/571837371 learned_addr learned my addr 172.21.15.87:0/571837371 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f314a133700 1 --2- 172.21.15.87:0/571837371 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31440f8420 0x7f314410e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f3149932700 1 -- 172.21.15.87:0/571837371 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31440f8420 msgr2=0x7f314410e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f3149932700 1 --2- 172.21.15.87:0/571837371 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31440f8420 0x7f314410e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f3149932700 1 -- 172.21.15.87:0/571837371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31440fe510 msgr2=0x7f3144108280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f3149932700 1 --2- 172.21.15.87:0/571837371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31440fe510 0x7f3144108280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f3149932700 1 -- 172.21.15.87:0/571837371 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3140005040 con 0x7f31440fdbb0 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f314a934700 1 --2- 172.21.15.87:0/571837371 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f31440fe510 0x7f3144108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f314a133700 1 --2- 172.21.15.87:0/571837371 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f31440f8420 0x7f314410e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f3149932700 1 --2- 172.21.15.87:0/571837371 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f31440fdbb0 0x7f314410e5d0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7f3140009eb0 tx=0x7f314000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f313b7fe700 10 client.?.objecter ms_handle_connect 0x7f31440fdbb0 2024-02-23T00:46:07.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f313b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f313b7fe700 1 -- 172.21.15.87:0/571837371 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314000e070 con 0x7f31440fdbb0 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f313b7fe700 1 -- 172.21.15.87:0/571837371 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3140013da0 con 0x7f31440fdbb0 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3144108d20 con 0x7f31440fdbb0 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f313b7fe700 1 -- 172.21.15.87:0/571837371 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140019070 con 0x7f31440fdbb0 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f314c397700 10 client.38603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3144114f10 con 0x7f31440fdbb0 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.786+0000 7f314c397700 1 librados: init done 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.787+0000 7f314c397700 10 client.38603.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.787+0000 7f314c397700 15 client.38603.objecter _session_command_op_assign -1 1 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.787+0000 7f314c397700 20 client.38603.objecter _get_session osd=-1 returning homeless 2024-02-23T00:46:07.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.787+0000 7f314c397700 10 client.38603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.787+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f314405ee80 con 0x7f31440fdbb0 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.787+0000 7f313b7fe700 1 -- 172.21.15.87:0/571837371 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f314001e420 con 0x7f31440fdbb0 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 1 --2- 172.21.15.87:0/571837371 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3130062c90 0x7f3130065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 1 -- 172.21.15.87:0/571837371 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(851..851 src has 400..851) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f314007dab0 con 0x7f31440fdbb0 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 10 client.38603.objecter ms_dispatch 0x7f3144001550 osd_map(851..851 src has 400..851) v4 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 3 client.38603.objecter handle_osd_map got epochs [851,851] > 0 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 3 client.38603.objecter handle_osd_map decoding full epoch 851 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f314a133700 1 --2- 172.21.15.87:0/571837371 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3130062c90 0x7f3130065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 10 client.38603.objecter checking command 1 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 1 --2- 172.21.15.87:0/571837371 >> v2:172.21.15.171:6804/2550013795 conn(0x7f313006e170 0x7f3130070610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 20 client.38603.objecter _get_session s=0x7f313006dac0 osd=5 3 2024-02-23T00:46:07.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 20 client.38603.objecter put_session s=0x7f313006dac0 osd=5 3 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 15 client.38603.objecter _session_command_op_remove -1 1 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 20 client.38603.objecter _get_session s=0x7f313006dac0 osd=5 3 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 20 client.38603.objecter get_session s=0x7f313006dac0 osd=5 3 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 15 client.38603.objecter _session_command_op_assign 5 1 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 20 client.38603.objecter put_session s=0x7f313006dac0 osd=5 4 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 10 client.38603.objecter _send_command 1 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 1 -- 172.21.15.87:0/571837371 --> v2:172.21.15.171:6804/2550013795 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3130070c50 con 0x7f313006e170 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 20 client.38603.objecter dump_active .. 0 homeless 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 1 -- 172.21.15.87:0/571837371 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3140082220 con 0x7f31440fdbb0 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 10 client.38603.objecter ms_handle_connect 0x7f3130062c90 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f314a133700 1 --2- 172.21.15.87:0/571837371 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3130062c90 0x7f3130065150 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7f313400a900 tx=0x7f3134008040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f314a934700 1 --2- 172.21.15.87:0/571837371 >> v2:172.21.15.171:6804/2550013795 conn(0x7f313006e170 0x7f3130070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f313b7fe700 10 client.38603.objecter ms_handle_connect 0x7f313006e170 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.788+0000 7f314a934700 1 --2- 172.21.15.87:0/571837371 >> v2:172.21.15.171:6804/2550013795 conn(0x7f313006e170 0x7f3130070610 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.789+0000 7f313b7fe700 1 -- 172.21.15.87:0/571837371 <== osd.5 v2:172.21.15.171:6804/2550013795 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f3130070c50 con 0x7f313006e170 2024-02-23T00:46:07.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.789+0000 7f313b7fe700 10 client.38603.objecter ms_dispatch 0x7f3144001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.789+0000 7f313b7fe700 10 client.38603.objecter _finish_command 1 = system:0 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.789+0000 7f313b7fe700 20 client.38603.objecter put_session s=0x7f313006dac0 osd=5 4 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.789+0000 7f313b7fe700 15 client.38603.objecter _session_command_op_remove 5 1 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 10 client.38603.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 15 client.38603.objecter _session_command_op_assign -1 2 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 20 client.38603.objecter _get_session s=0x7f313006dac0 osd=5 3 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 20 client.38603.objecter put_session s=0x7f313006dac0 osd=5 3 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 20 client.38603.objecter _get_session s=0x7f313006dac0 osd=5 3 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 15 client.38603.objecter _session_command_op_remove -1 2 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 20 client.38603.objecter get_session s=0x7f313006dac0 osd=5 3 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 15 client.38603.objecter _session_command_op_assign 5 2 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 20 client.38603.objecter put_session s=0x7f313006dac0 osd=5 4 2024-02-23T00:46:07.832 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 10 client.38603.objecter _send_command 2 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.803+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 --> v2:172.21.15.171:6804/2550013795 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f31441070a0 con 0x7f313006e170 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f313b7fe700 1 -- 172.21.15.87:0/571837371 <== osd.5 v2:172.21.15.171:6804/2550013795 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f31441070a0 con 0x7f313006e170 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f313b7fe700 10 client.38603.objecter ms_dispatch 0x7f3144001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f313b7fe700 10 client.38603.objecter _finish_command 2 = system:0 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f313b7fe700 20 client.38603.objecter put_session s=0x7f313006dac0 osd=5 4 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f313b7fe700 15 client.38603.objecter _session_command_op_remove 5 2 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 10 librados: watch_flush enter 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 10 librados: watch_flush exit 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 10 client.38603.objecter close_session for osd.5 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 >> v2:172.21.15.171:6804/2550013795 conn(0x7f313006e170 msgr2=0x7f3130070610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 --2- 172.21.15.87:0/571837371 >> v2:172.21.15.171:6804/2550013795 conn(0x7f313006e170 0x7f3130070610 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 20 client.38603.objecter put_session s=0x7f313006dac0 osd=5 1 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 20 client.38603.objecter shutdown clearing up homeless session... 2024-02-23T00:46:07.833 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 10 client.38603.objecter successfully canceled tick 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3130062c90 msgr2=0x7f3130065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 --2- 172.21.15.87:0/571837371 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3130062c90 0x7f3130065150 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 rx=0x7f313400a900 tx=0x7f3134008040).stop 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f31440fdbb0 msgr2=0x7f314410e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 --2- 172.21.15.87:0/571837371 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f31440fdbb0 0x7f314410e5d0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7f3140009eb0 tx=0x7f314000b040).stop 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314a934700 1 -- 172.21.15.87:0/571837371 reap_dead start 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 shutdown_connections 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 >> 172.21.15.87:0/571837371 conn(0x7f31440f3800 msgr2=0x7f31440fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 shutdown_connections 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 -- 172.21.15.87:0/571837371 wait complete. 2024-02-23T00:46:07.834 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:07.804+0000 7f314c397700 1 librados: shutdown 2024-02-23T00:46:08.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:06.941731+0000 osd.0 (osd.0) 113 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:46:08.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:06.945045+0000 osd.0 (osd.0) 114 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:46:08.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:08.138224+0000 osd.4 (osd.4) 401 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:46:08.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:08.139329+0000 osd.4 (osd.4) 402 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:46:08.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:06.941731+0000 osd.0 (osd.0) 113 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:46:08.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:06.945045+0000 osd.0 (osd.0) 114 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:46:08.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:08.138224+0000 osd.4 (osd.4) 401 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:46:08.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:08 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:08.139329+0000 osd.4 (osd.4) 402 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:46:08.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:06.941731+0000 osd.0 (osd.0) 113 : cluster [DBG] 5.1e scrub starts 2024-02-23T00:46:08.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:06.945045+0000 osd.0 (osd.0) 114 : cluster [DBG] 5.1e scrub ok 2024-02-23T00:46:08.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:08.138224+0000 osd.4 (osd.4) 401 : cluster [DBG] 22.10 scrub starts 2024-02-23T00:46:08.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:08 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:08.139329+0000 osd.4 (osd.4) 402 : cluster [DBG] 22.10 scrub ok 2024-02-23T00:46:08.709 INFO:teuthology.orchestra.run.smithi087.stdout:163208757488 2024-02-23T00:46:08.710 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-23T00:46:08.982 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:09.497 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:07.843376+0000 mgr.y (mgr.24437) 754 : cluster [DBG] pgmap v1342: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:46:09.497 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:08.113777+0000 osd.3 (osd.3) 333 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:46:09.497 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:08.116759+0000 osd.3 (osd.3) 334 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:46:09.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:07.843376+0000 mgr.y (mgr.24437) 754 : cluster [DBG] pgmap v1342: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:46:09.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:08.113777+0000 osd.3 (osd.3) 333 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:46:09.497 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:09 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:08.116759+0000 osd.3 (osd.3) 334 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:46:09.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:07.843376+0000 mgr.y (mgr.24437) 754 : cluster [DBG] pgmap v1342: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:46:09.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:08.113777+0000 osd.3 (osd.3) 333 : cluster [DBG] 22.12 scrub starts 2024-02-23T00:46:09.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:08.116759+0000 osd.3 (osd.3) 334 : cluster [DBG] 22.12 scrub ok 2024-02-23T00:46:10.904 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.899+0000 7f13ef397700 1 -- 172.21.15.87:0/2736642952 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f13e80f7400 msgr2=0x7f13e8103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.899+0000 7f13ef397700 1 --2- 172.21.15.87:0/2736642952 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f13e80f7400 0x7f13e8103a20 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 rx=0x7f13e0009c30 tx=0x7f13e0004c50).stop 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.899+0000 7f13ef397700 1 -- 172.21.15.87:0/2736642952 shutdown_connections 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.899+0000 7f13ef397700 1 --2- 172.21.15.87:0/2736642952 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f13e80f7400 0x7f13e8103a20 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.899+0000 7f13ef397700 1 --2- 172.21.15.87:0/2736642952 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13e80fcbb0 0x7f13e80fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.899+0000 7f13ef397700 1 --2- 172.21.15.87:0/2736642952 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13e80fd510 0x7f13e80f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.899+0000 7f13ef397700 1 -- 172.21.15.87:0/2736642952 >> 172.21.15.87:0/2736642952 conn(0x7f13e80f2820 msgr2=0x7f13e80f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.899+0000 7f13ef397700 1 -- 172.21.15.87:0/2736642952 shutdown_connections 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.899+0000 7f13ef397700 1 -- 172.21.15.87:0/2736642952 wait complete. 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 librados: starting msgr at 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 librados: starting objecter 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 Processor -- start 2024-02-23T00:46:10.905 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 -- start start 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 librados: setting wanted keys 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 librados: calling monclient init 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13e80f7400 0x7f13e8108fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f13e80fcbb0 0x7f13e81094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13e80fd510 0x7f13e8109af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e8114200 con 0x7f13e80fd510 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e8114370 con 0x7f13e80f7400 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ef397700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e8114530 con 0x7f13e80fcbb0 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ed934700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13e80fd510 0x7f13e8109af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ec932700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f13e80fcbb0 0x7f13e81094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ec932700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f13e80fcbb0 0x7f13e81094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:50948/0 (socket says 172.21.15.87:50948) 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.900+0000 7f13ed934700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13e80fd510 0x7f13e8109af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:56918/0 (socket says 172.21.15.87:56918) 2024-02-23T00:46:10.906 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ec932700 1 -- 172.21.15.87:0/2506260114 learned_addr learned my addr 172.21.15.87:0/2506260114 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ed133700 1 --2- 172.21.15.87:0/2506260114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13e80f7400 0x7f13e8108fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ec932700 1 -- 172.21.15.87:0/2506260114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13e80f7400 msgr2=0x7f13e8108fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ec932700 1 --2- 172.21.15.87:0/2506260114 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f13e80f7400 0x7f13e8108fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ec932700 1 -- 172.21.15.87:0/2506260114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13e80fd510 msgr2=0x7f13e8109af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ec932700 1 --2- 172.21.15.87:0/2506260114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13e80fd510 0x7f13e8109af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ec932700 1 -- 172.21.15.87:0/2506260114 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d8005040 con 0x7f13e80fcbb0 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ed934700 1 --2- 172.21.15.87:0/2506260114 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f13e80fd510 0x7f13e8109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ec932700 1 --2- 172.21.15.87:0/2506260114 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f13e80fcbb0 0x7f13e81094c0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7f13e400c810 tx=0x7f13e4004330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13de7fc700 10 client.?.objecter ms_handle_connect 0x7f13e80fcbb0 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13de7fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13de7fc700 1 -- 172.21.15.87:0/2506260114 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e40043a0 con 0x7f13e80fcbb0 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e00097c0 con 0x7f13e80fcbb0 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13de7fc700 1 -- 172.21.15.87:0/2506260114 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13e40109e0 con 0x7f13e80fcbb0 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13de7fc700 1 -- 172.21.15.87:0/2506260114 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4012ab0 con 0x7f13e80fcbb0 2024-02-23T00:46:10.907 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ef397700 10 client.38609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:10.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e8114bc0 con 0x7f13e80fcbb0 2024-02-23T00:46:10.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.901+0000 7f13ef397700 1 librados: init done 2024-02-23T00:46:10.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.902+0000 7f13ef397700 10 client.38609.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:46:10.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.902+0000 7f13ef397700 15 client.38609.objecter _session_command_op_assign -1 1 2024-02-23T00:46:10.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.902+0000 7f13ef397700 20 client.38609.objecter _get_session osd=-1 returning homeless 2024-02-23T00:46:10.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.902+0000 7f13ef397700 10 client.38609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:10.908 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.902+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f13e805fa60 con 0x7f13e80fcbb0 2024-02-23T00:46:10.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 1 -- 172.21.15.87:0/2506260114 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f13e4010b50 con 0x7f13e80fcbb0 2024-02-23T00:46:10.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 1 --2- 172.21.15.87:0/2506260114 >> v2:172.21.15.87:6800/1231597463 conn(0x7f13d4062c90 0x7f13d4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:10.909 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 1 -- 172.21.15.87:0/2506260114 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(851..851 src has 400..851) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f13e407ce50 con 0x7f13e80fcbb0 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 10 client.38609.objecter ms_dispatch 0x7f13e8001550 osd_map(851..851 src has 400..851) v4 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 3 client.38609.objecter handle_osd_map got epochs [851,851] > 0 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13ed133700 1 --2- 172.21.15.87:0/2506260114 >> v2:172.21.15.87:6800/1231597463 conn(0x7f13d4062c90 0x7f13d4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 3 client.38609.objecter handle_osd_map decoding full epoch 851 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 10 client.38609.objecter checking command 1 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 1 --2- 172.21.15.87:0/2506260114 >> v2:172.21.15.171:6808/2509408498 conn(0x7f13d406e0f0 0x7f13d4070590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 20 client.38609.objecter _get_session s=0x7f13d406da40 osd=6 3 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 20 client.38609.objecter put_session s=0x7f13d406da40 osd=6 3 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 15 client.38609.objecter _session_command_op_remove -1 1 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 20 client.38609.objecter _get_session s=0x7f13d406da40 osd=6 3 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 20 client.38609.objecter get_session s=0x7f13d406da40 osd=6 3 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 15 client.38609.objecter _session_command_op_assign 6 1 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 20 client.38609.objecter put_session s=0x7f13d406da40 osd=6 4 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 10 client.38609.objecter _send_command 1 2024-02-23T00:46:10.910 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 1 -- 172.21.15.87:0/2506260114 --> v2:172.21.15.171:6808/2509408498 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13d4070bd0 con 0x7f13d406e0f0 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 20 client.38609.objecter dump_active .. 0 homeless 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 1 -- 172.21.15.87:0/2506260114 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13e407f470 con 0x7f13e80fcbb0 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13de7fc700 10 client.38609.objecter ms_handle_connect 0x7f13d4062c90 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13ed133700 1 --2- 172.21.15.87:0/2506260114 >> v2:172.21.15.87:6800/1231597463 conn(0x7f13d4062c90 0x7f13d4065150 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7f13d8009f20 tx=0x7f13d8009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.903+0000 7f13ed934700 1 --2- 172.21.15.87:0/2506260114 >> v2:172.21.15.171:6808/2509408498 conn(0x7f13d406e0f0 0x7f13d4070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.904+0000 7f13de7fc700 10 client.38609.objecter ms_handle_connect 0x7f13d406e0f0 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.904+0000 7f13ed934700 1 --2- 172.21.15.87:0/2506260114 >> v2:172.21.15.171:6808/2509408498 conn(0x7f13d406e0f0 0x7f13d4070590 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.904+0000 7f13de7fc700 1 -- 172.21.15.87:0/2506260114 <== osd.6 v2:172.21.15.171:6808/2509408498 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f13d4070bd0 con 0x7f13d406e0f0 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.904+0000 7f13de7fc700 10 client.38609.objecter ms_dispatch 0x7f13e8001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.904+0000 7f13de7fc700 10 client.38609.objecter _finish_command 1 = system:0 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.904+0000 7f13de7fc700 20 client.38609.objecter put_session s=0x7f13d406da40 osd=6 4 2024-02-23T00:46:10.911 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.904+0000 7f13de7fc700 15 client.38609.objecter _session_command_op_remove 6 1 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 10 client.38609.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 15 client.38609.objecter _session_command_op_assign -1 2 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 20 client.38609.objecter _get_session s=0x7f13d406da40 osd=6 3 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 20 client.38609.objecter put_session s=0x7f13d406da40 osd=6 3 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 20 client.38609.objecter _get_session s=0x7f13d406da40 osd=6 3 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 15 client.38609.objecter _session_command_op_remove -1 2 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 20 client.38609.objecter get_session s=0x7f13d406da40 osd=6 3 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 15 client.38609.objecter _session_command_op_assign 6 2 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 20 client.38609.objecter put_session s=0x7f13d406da40 osd=6 4 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 10 client.38609.objecter _send_command 2 2024-02-23T00:46:10.921 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.918+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 --> v2:172.21.15.171:6808/2509408498 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f13e805ee80 con 0x7f13d406e0f0 2024-02-23T00:46:10.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13de7fc700 1 -- 172.21.15.87:0/2506260114 <== osd.6 v2:172.21.15.171:6808/2509408498 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f13e805ee80 con 0x7f13d406e0f0 2024-02-23T00:46:10.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13de7fc700 10 client.38609.objecter ms_dispatch 0x7f13e8001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:46:10.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13de7fc700 10 client.38609.objecter _finish_command 2 = system:0 2024-02-23T00:46:10.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13de7fc700 20 client.38609.objecter put_session s=0x7f13d406da40 osd=6 4 2024-02-23T00:46:10.922 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13de7fc700 15 client.38609.objecter _session_command_op_remove 6 2 2024-02-23T00:46:10.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 10 librados: watch_flush enter 2024-02-23T00:46:10.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 10 librados: watch_flush exit 2024-02-23T00:46:10.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 10 client.38609.objecter close_session for osd.6 2024-02-23T00:46:10.923 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 >> v2:172.21.15.171:6808/2509408498 conn(0x7f13d406e0f0 msgr2=0x7f13d4070590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 1 --2- 172.21.15.87:0/2506260114 >> v2:172.21.15.171:6808/2509408498 conn(0x7f13d406e0f0 0x7f13d4070590 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 20 client.38609.objecter put_session s=0x7f13d406da40 osd=6 1 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 20 client.38609.objecter shutdown clearing up homeless session... 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 10 client.38609.objecter successfully canceled tick 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 >> v2:172.21.15.87:6800/1231597463 conn(0x7f13d4062c90 msgr2=0x7f13d4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 1 --2- 172.21.15.87:0/2506260114 >> v2:172.21.15.87:6800/1231597463 conn(0x7f13d4062c90 0x7f13d4065150 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 rx=0x7f13d8009f20 tx=0x7f13d8009cb0).stop 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f13e80fcbb0 msgr2=0x7f13e81094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 1 --2- 172.21.15.87:0/2506260114 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f13e80fcbb0 0x7f13e81094c0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 rx=0x7f13e400c810 tx=0x7f13e4004330).stop 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ed934700 1 -- 172.21.15.87:0/2506260114 reap_dead start 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 shutdown_connections 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.919+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 >> 172.21.15.87:0/2506260114 conn(0x7f13e80f2820 msgr2=0x7f13e80f40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.920+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 shutdown_connections 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.920+0000 7f13ef397700 1 -- 172.21.15.87:0/2506260114 wait complete. 2024-02-23T00:46:10.924 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:10.920+0000 7f13ef397700 1 librados: shutdown 2024-02-23T00:46:11.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:09.844119+0000 mgr.y (mgr.24437) 755 : cluster [DBG] pgmap v1343: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:09.955791+0000 osd.0 (osd.0) 115 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:46:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:09.959243+0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:46:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:10.361621+0000 osd.6 (osd.6) 396 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:46:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:10.362538+0000 osd.6 (osd.6) 397 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:46:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:10.531209+0000 osd.7 (osd.7) 442 : cluster [DBG] 22.a scrub starts 2024-02-23T00:46:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:10.532443+0000 osd.7 (osd.7) 443 : cluster [DBG] 22.a scrub ok 2024-02-23T00:46:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:10.869044+0000 osd.5 (osd.5) 379 : cluster [DBG] 22.f deep-scrub starts 2024-02-23T00:46:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:10.871916+0000 osd.5 (osd.5) 380 : cluster [DBG] 22.f deep-scrub ok 2024-02-23T00:46:11.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.171978+0000 osd.4 (osd.4) 403 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.173236+0000 osd.4 (osd.4) 404 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:09.844119+0000 mgr.y (mgr.24437) 755 : cluster [DBG] pgmap v1343: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:09.955791+0000 osd.0 (osd.0) 115 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:09.959243+0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:10.361621+0000 osd.6 (osd.6) 396 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:10.362538+0000 osd.6 (osd.6) 397 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:10.531209+0000 osd.7 (osd.7) 442 : cluster [DBG] 22.a scrub starts 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:10.532443+0000 osd.7 (osd.7) 443 : cluster [DBG] 22.a scrub ok 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:10.869044+0000 osd.5 (osd.5) 379 : cluster [DBG] 22.f deep-scrub starts 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:10.871916+0000 osd.5 (osd.5) 380 : cluster [DBG] 22.f deep-scrub ok 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.171978+0000 osd.4 (osd.4) 403 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:46:11.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:11 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.173236+0000 osd.4 (osd.4) 404 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:46:11.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:09.844119+0000 mgr.y (mgr.24437) 755 : cluster [DBG] pgmap v1343: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:09.955791+0000 osd.0 (osd.0) 115 : cluster [DBG] 5.3 scrub starts 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:09.959243+0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 scrub ok 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:10.361621+0000 osd.6 (osd.6) 396 : cluster [DBG] 2.14 scrub starts 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:10.362538+0000 osd.6 (osd.6) 397 : cluster [DBG] 2.14 scrub ok 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:10.531209+0000 osd.7 (osd.7) 442 : cluster [DBG] 22.a scrub starts 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:10.532443+0000 osd.7 (osd.7) 443 : cluster [DBG] 22.a scrub ok 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:10.869044+0000 osd.5 (osd.5) 379 : cluster [DBG] 22.f deep-scrub starts 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:10.871916+0000 osd.5 (osd.5) 380 : cluster [DBG] 22.f deep-scrub ok 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.171978+0000 osd.4 (osd.4) 403 : cluster [DBG] 22.14 scrub starts 2024-02-23T00:46:11.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:11 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.173236+0000 osd.4 (osd.4) 404 : cluster [DBG] 22.14 scrub ok 2024-02-23T00:46:11.762 INFO:teuthology.orchestra.run.smithi087.stdout:188978561260 2024-02-23T00:46:11.762 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-23T00:46:12.031 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:12.558 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.321790+0000 osd.1 (osd.1) 452 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:46:12.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.323132+0000 osd.1 (osd.1) 453 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:46:12.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.380794+0000 osd.2 (osd.2) 339 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:46:12.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.382532+0000 osd.2 (osd.2) 340 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:46:12.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.501077+0000 osd.7 (osd.7) 444 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:46:12.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.503965+0000 osd.7 (osd.7) 445 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:46:12.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.820843+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.f scrub starts 2024-02-23T00:46:12.559 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.821932+0000 osd.5 (osd.5) 382 : cluster [DBG] 5.f scrub ok 2024-02-23T00:46:12.559 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:12] "GET /metrics HTTP/1.1" 200 186068 "" "Prometheus/2.33.4" 2024-02-23T00:46:12.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.321790+0000 osd.1 (osd.1) 452 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:46:12.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.323132+0000 osd.1 (osd.1) 453 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:46:12.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.380794+0000 osd.2 (osd.2) 339 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:46:12.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.382532+0000 osd.2 (osd.2) 340 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:46:12.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.501077+0000 osd.7 (osd.7) 444 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:46:12.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.503965+0000 osd.7 (osd.7) 445 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:46:12.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.820843+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.f scrub starts 2024-02-23T00:46:12.560 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.821932+0000 osd.5 (osd.5) 382 : cluster [DBG] 5.f scrub ok 2024-02-23T00:46:12.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.321790+0000 osd.1 (osd.1) 452 : cluster [DBG] 6.1 scrub starts 2024-02-23T00:46:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.323132+0000 osd.1 (osd.1) 453 : cluster [DBG] 6.1 scrub ok 2024-02-23T00:46:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.380794+0000 osd.2 (osd.2) 339 : cluster [DBG] 4.1c scrub starts 2024-02-23T00:46:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.382532+0000 osd.2 (osd.2) 340 : cluster [DBG] 4.1c scrub ok 2024-02-23T00:46:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.501077+0000 osd.7 (osd.7) 444 : cluster [DBG] 2.0 scrub starts 2024-02-23T00:46:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.503965+0000 osd.7 (osd.7) 445 : cluster [DBG] 2.0 scrub ok 2024-02-23T00:46:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.820843+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.f scrub starts 2024-02-23T00:46:12.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:12 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.821932+0000 osd.5 (osd.5) 382 : cluster [DBG] 5.f scrub ok 2024-02-23T00:46:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:11.445496+0000 mgr.y (mgr.24437) 756 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:11.844876+0000 mgr.y (mgr.24437) 757 : cluster [DBG] pgmap v1344: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:13.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:12.095631+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:46:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:12.096660+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:46:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:12.337738+0000 osd.1 (osd.1) 454 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:46:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:12.341832+0000 osd.1 (osd.1) 455 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:46:13.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:13.270504+0000 mon.a (mon.0) 4040 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:46:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:11.445496+0000 mgr.y (mgr.24437) 756 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:11.844876+0000 mgr.y (mgr.24437) 757 : cluster [DBG] pgmap v1344: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:12.095631+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:46:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:12.096660+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:46:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:12.337738+0000 osd.1 (osd.1) 454 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:46:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:12.341832+0000 osd.1 (osd.1) 455 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:46:13.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:13.270504+0000 mon.a (mon.0) 4040 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:46:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:11.445496+0000 mgr.y (mgr.24437) 756 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:11.844876+0000 mgr.y (mgr.24437) 757 : cluster [DBG] pgmap v1344: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:12.095631+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.7 scrub starts 2024-02-23T00:46:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:12.096660+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.7 scrub ok 2024-02-23T00:46:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:12.337738+0000 osd.1 (osd.1) 454 : cluster [DBG] 4.4 scrub starts 2024-02-23T00:46:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:12.341832+0000 osd.1 (osd.1) 455 : cluster [DBG] 4.4 scrub ok 2024-02-23T00:46:13.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:13 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:13.270504+0000 mon.a (mon.0) 4040 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.089+0000 7f5409ee0700 1 -- 172.21.15.87:0/2330601683 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54040fdbb0 msgr2=0x7f54040fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.089+0000 7f5409ee0700 1 --2- 172.21.15.87:0/2330601683 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54040fdbb0 0x7f54040fdf90 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 rx=0x7f53f8004870 tx=0x7f53f8009cb0).stop 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.089+0000 7f5409ee0700 1 -- 172.21.15.87:0/2330601683 shutdown_connections 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.089+0000 7f5409ee0700 1 --2- 172.21.15.87:0/2330601683 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54040f8420 0x7f5404104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.089+0000 7f5409ee0700 1 --2- 172.21.15.87:0/2330601683 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54040fdbb0 0x7f54040fdf90 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.089+0000 7f5409ee0700 1 --2- 172.21.15.87:0/2330601683 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54040fe510 0x7f54040f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.089+0000 7f5409ee0700 1 -- 172.21.15.87:0/2330601683 >> 172.21.15.87:0/2330601683 conn(0x7f54040f3800 msgr2=0x7f54040f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.089+0000 7f5409ee0700 1 -- 172.21.15.87:0/2330601683 shutdown_connections 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.089+0000 7f5409ee0700 1 -- 172.21.15.87:0/2330601683 wait complete. 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 librados: starting msgr at 2024-02-23T00:46:14.094 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 librados: starting objecter 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 Processor -- start 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 -- start start 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 librados: setting wanted keys 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 librados: calling monclient init 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54040f8420 0x7f540410e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54040fdbb0 0x7f540410e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54040fe510 0x7f5404108280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5404108770 con 0x7f54040f8420 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54041088e0 con 0x7f54040fdbb0 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5409ee0700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5404108aa0 con 0x7f54040fe510 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f54037fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54040f8420 0x7f540410e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f54037fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54040f8420 0x7f540410e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:51936/0 (socket says 172.21.15.87:51936) 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f54037fe700 1 -- 172.21.15.87:0/649107948 learned_addr learned my addr 172.21.15.87:0/649107948 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5403fff700 1 --2- 172.21.15.87:0/649107948 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54040fe510 0x7f5404108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:14.095 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.090+0000 7f5402ffd700 1 --2- 172.21.15.87:0/649107948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54040fdbb0 0x7f540410e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f54037fe700 1 -- 172.21.15.87:0/649107948 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54040fe510 msgr2=0x7f5404108280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f54037fe700 1 --2- 172.21.15.87:0/649107948 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54040fe510 0x7f5404108280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f54037fe700 1 -- 172.21.15.87:0/649107948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54040fdbb0 msgr2=0x7f540410e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f54037fe700 1 --2- 172.21.15.87:0/649107948 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f54040fdbb0 0x7f540410e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f54037fe700 1 -- 172.21.15.87:0/649107948 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc0097c0 con 0x7f54040f8420 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5403fff700 1 --2- 172.21.15.87:0/649107948 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f54040fe510 0x7f5404108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f54037fe700 1 --2- 172.21.15.87:0/649107948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54040f8420 0x7f540410e0e0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f53f80043a0 tx=0x7f53f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5400ff9700 10 client.?.objecter ms_handle_connect 0x7f54040f8420 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5400ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5400ff9700 1 -- 172.21.15.87:0/649107948 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f800e070 con 0x7f54040f8420 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5400ff9700 1 -- 172.21.15.87:0/649107948 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53f8009710 con 0x7f54040f8420 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f8005040 con 0x7f54040f8420 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5400ff9700 1 -- 172.21.15.87:0/649107948 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8019070 con 0x7f54040f8420 2024-02-23T00:46:14.096 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5409ee0700 10 client.28137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:14.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5404109010 con 0x7f54040f8420 2024-02-23T00:46:14.097 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.091+0000 7f5409ee0700 1 librados: init done 2024-02-23T00:46:14.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.092+0000 7f5409ee0700 10 client.28137.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-23T00:46:14.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.092+0000 7f5409ee0700 15 client.28137.objecter _session_command_op_assign -1 1 2024-02-23T00:46:14.098 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.092+0000 7f5409ee0700 20 client.28137.objecter _get_session osd=-1 returning homeless 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.092+0000 7f5409ee0700 10 client.28137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.092+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f540405ee80 con 0x7f54040f8420 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.092+0000 7f5400ff9700 1 -- 172.21.15.87:0/649107948 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f53f80090d0 con 0x7f54040f8420 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 1 --2- 172.21.15.87:0/649107948 >> v2:172.21.15.87:6800/1231597463 conn(0x7f53ec062c90 0x7f53ec065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 1 -- 172.21.15.87:0/649107948 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(851..851 src has 400..851) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f53f807d7e0 con 0x7f54040f8420 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 10 client.28137.objecter ms_dispatch 0x7f5404001550 osd_map(851..851 src has 400..851) v4 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 3 client.28137.objecter handle_osd_map got epochs [851,851] > 0 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 3 client.28137.objecter handle_osd_map decoding full epoch 851 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5402ffd700 1 --2- 172.21.15.87:0/649107948 >> v2:172.21.15.87:6800/1231597463 conn(0x7f53ec062c90 0x7f53ec065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 10 client.28137.objecter checking command 1 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 1 --2- 172.21.15.87:0/649107948 >> v2:172.21.15.171:6812/373787000 conn(0x7f53ec06e170 0x7f53ec070610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 20 client.28137.objecter _get_session s=0x7f53ec06dac0 osd=7 3 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 20 client.28137.objecter put_session s=0x7f53ec06dac0 osd=7 3 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 15 client.28137.objecter _session_command_op_remove -1 1 2024-02-23T00:46:14.099 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 20 client.28137.objecter _get_session s=0x7f53ec06dac0 osd=7 3 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 20 client.28137.objecter get_session s=0x7f53ec06dac0 osd=7 3 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 15 client.28137.objecter _session_command_op_assign 7 1 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 20 client.28137.objecter put_session s=0x7f53ec06dac0 osd=7 4 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 10 client.28137.objecter _send_command 1 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 1 -- 172.21.15.87:0/649107948 --> v2:172.21.15.171:6812/373787000 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53ec070c50 con 0x7f53ec06e170 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 20 client.28137.objecter dump_active .. 0 homeless 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 1 -- 172.21.15.87:0/649107948 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53f8082220 con 0x7f54040f8420 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 10 client.28137.objecter ms_handle_connect 0x7f53ec062c90 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5402ffd700 1 --2- 172.21.15.87:0/649107948 >> v2:172.21.15.87:6800/1231597463 conn(0x7f53ec062c90 0x7f53ec065150 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 rx=0x7f5404109a20 tx=0x7f53f400b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5403fff700 1 --2- 172.21.15.87:0/649107948 >> v2:172.21.15.171:6812/373787000 conn(0x7f53ec06e170 0x7f53ec070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5400ff9700 10 client.28137.objecter ms_handle_connect 0x7f53ec06e170 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.093+0000 7f5403fff700 1 --2- 172.21.15.87:0/649107948 >> v2:172.21.15.171:6812/373787000 conn(0x7f53ec06e170 0x7f53ec070610 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.094+0000 7f5400ff9700 1 -- 172.21.15.87:0/649107948 <== osd.7 v2:172.21.15.171:6812/373787000 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (crc 0 0 0) 0x7f53ec070c50 con 0x7f53ec06e170 2024-02-23T00:46:14.100 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.094+0000 7f5400ff9700 10 client.28137.objecter ms_dispatch 0x7f5404001550 command_reply(tid 1: 0 ) v1 2024-02-23T00:46:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.094+0000 7f5400ff9700 10 client.28137.objecter _finish_command 1 = system:0 2024-02-23T00:46:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.094+0000 7f5400ff9700 20 client.28137.objecter put_session s=0x7f53ec06dac0 osd=7 4 2024-02-23T00:46:14.101 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.094+0000 7f5400ff9700 15 client.28137.objecter _session_command_op_remove 7 1 2024-02-23T00:46:14.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 10 client.28137.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 15 client.28137.objecter _session_command_op_assign -1 2 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 20 client.28137.objecter _get_session s=0x7f53ec06dac0 osd=7 3 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 20 client.28137.objecter put_session s=0x7f53ec06dac0 osd=7 3 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 20 client.28137.objecter _get_session s=0x7f53ec06dac0 osd=7 3 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 15 client.28137.objecter _session_command_op_remove -1 2 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 20 client.28137.objecter get_session s=0x7f53ec06dac0 osd=7 3 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 15 client.28137.objecter _session_command_op_assign 7 2 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 20 client.28137.objecter put_session s=0x7f53ec06dac0 osd=7 4 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 10 client.28137.objecter _send_command 2 2024-02-23T00:46:14.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.108+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 --> v2:172.21.15.171:6812/373787000 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f54041070a0 con 0x7f53ec06e170 2024-02-23T00:46:14.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5400ff9700 1 -- 172.21.15.87:0/649107948 <== osd.7 v2:172.21.15.171:6812/373787000 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f54041070a0 con 0x7f53ec06e170 2024-02-23T00:46:14.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5400ff9700 10 client.28137.objecter ms_dispatch 0x7f5404001550 command_reply(tid 2: 0 ) v1 2024-02-23T00:46:14.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5400ff9700 10 client.28137.objecter _finish_command 2 = system:0 2024-02-23T00:46:14.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5400ff9700 20 client.28137.objecter put_session s=0x7f53ec06dac0 osd=7 4 2024-02-23T00:46:14.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5400ff9700 15 client.28137.objecter _session_command_op_remove 7 2 2024-02-23T00:46:14.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 10 librados: watch_flush enter 2024-02-23T00:46:14.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 10 librados: watch_flush exit 2024-02-23T00:46:14.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 10 client.28137.objecter close_session for osd.7 2024-02-23T00:46:14.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 >> v2:172.21.15.171:6812/373787000 conn(0x7f53ec06e170 msgr2=0x7f53ec070610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:14.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 --2- 172.21.15.87:0/649107948 >> v2:172.21.15.171:6812/373787000 conn(0x7f53ec06e170 0x7f53ec070610 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:14.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 20 client.28137.objecter put_session s=0x7f53ec06dac0 osd=7 1 2024-02-23T00:46:14.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 20 client.28137.objecter shutdown clearing up homeless session... 2024-02-23T00:46:14.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 10 client.28137.objecter successfully canceled tick 2024-02-23T00:46:14.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 >> v2:172.21.15.87:6800/1231597463 conn(0x7f53ec062c90 msgr2=0x7f53ec065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 --2- 172.21.15.87:0/649107948 >> v2:172.21.15.87:6800/1231597463 conn(0x7f53ec062c90 0x7f53ec065150 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 rx=0x7f5404109a20 tx=0x7f53f400b040).stop 2024-02-23T00:46:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54040f8420 msgr2=0x7f540410e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 --2- 172.21.15.87:0/649107948 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f54040f8420 0x7f540410e0e0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f53f80043a0 tx=0x7f53f800b040).stop 2024-02-23T00:46:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5403fff700 1 -- 172.21.15.87:0/649107948 reap_dead start 2024-02-23T00:46:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 shutdown_connections 2024-02-23T00:46:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 >> 172.21.15.87:0/649107948 conn(0x7f54040f3800 msgr2=0x7f54040f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 shutdown_connections 2024-02-23T00:46:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 -- 172.21.15.87:0/649107948 wait complete. 2024-02-23T00:46:14.114 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:14.109+0000 7f5409ee0700 1 librados: shutdown 2024-02-23T00:46:14.192 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:46:13 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:46:14.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:12.992434+0000 osd.0 (osd.0) 117 : cluster [DBG] 22.18 deep-scrub starts 2024-02-23T00:46:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:12.993704+0000 osd.0 (osd.0) 118 : cluster [DBG] 22.18 deep-scrub ok 2024-02-23T00:46:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:13.374583+0000 osd.1 (osd.1) 456 : cluster [DBG] 22.1c scrub starts 2024-02-23T00:46:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:13.377471+0000 osd.1 (osd.1) 457 : cluster [DBG] 22.1c scrub ok 2024-02-23T00:46:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:13.680162+0000 mon.a (mon.0) 4041 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:46:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:13.680858+0000 mon.a (mon.0) 4042 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:46:14.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:14 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:13.687527+0000 mon.a (mon.0) 4043 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:46:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:12.992434+0000 osd.0 (osd.0) 117 : cluster [DBG] 22.18 deep-scrub starts 2024-02-23T00:46:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:12.993704+0000 osd.0 (osd.0) 118 : cluster [DBG] 22.18 deep-scrub ok 2024-02-23T00:46:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:13.374583+0000 osd.1 (osd.1) 456 : cluster [DBG] 22.1c scrub starts 2024-02-23T00:46:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:13.377471+0000 osd.1 (osd.1) 457 : cluster [DBG] 22.1c scrub ok 2024-02-23T00:46:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:13.680162+0000 mon.a (mon.0) 4041 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:46:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:13.680858+0000 mon.a (mon.0) 4042 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:46:14.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:13.687527+0000 mon.a (mon.0) 4043 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:46:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:12.992434+0000 osd.0 (osd.0) 117 : cluster [DBG] 22.18 deep-scrub starts 2024-02-23T00:46:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:12.993704+0000 osd.0 (osd.0) 118 : cluster [DBG] 22.18 deep-scrub ok 2024-02-23T00:46:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:13.374583+0000 osd.1 (osd.1) 456 : cluster [DBG] 22.1c scrub starts 2024-02-23T00:46:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:13.377471+0000 osd.1 (osd.1) 457 : cluster [DBG] 22.1c scrub ok 2024-02-23T00:46:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:13.680162+0000 mon.a (mon.0) 4041 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:46:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:13.680858+0000 mon.a (mon.0) 4042 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:46:14.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:14 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:13.687527+0000 mon.a (mon.0) 4043 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:46:14.989 INFO:teuthology.orchestra.run.smithi087.stdout:219043332327 2024-02-23T00:46:14.990 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-23T00:46:15.260 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:15.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:13.845346+0000 mgr.y (mgr.24437) 758 : cluster [DBG] pgmap v1345: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:14.563924+0000 osd.7 (osd.7) 446 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:46:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:14.566983+0000 osd.7 (osd.7) 447 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:46:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:14.765812+0000 osd.5 (osd.5) 383 : cluster [DBG] 3.c deep-scrub starts 2024-02-23T00:46:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:14.767635+0000 osd.5 (osd.5) 384 : cluster [DBG] 3.c deep-scrub ok 2024-02-23T00:46:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:15.148124+0000 osd.4 (osd.4) 405 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:46:15.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:15.149027+0000 osd.4 (osd.4) 406 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:46:15.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:13.845346+0000 mgr.y (mgr.24437) 758 : cluster [DBG] pgmap v1345: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:15.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:14.563924+0000 osd.7 (osd.7) 446 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:46:15.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:14.566983+0000 osd.7 (osd.7) 447 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:46:15.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:14.765812+0000 osd.5 (osd.5) 383 : cluster [DBG] 3.c deep-scrub starts 2024-02-23T00:46:15.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:14.767635+0000 osd.5 (osd.5) 384 : cluster [DBG] 3.c deep-scrub ok 2024-02-23T00:46:15.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:15.148124+0000 osd.4 (osd.4) 405 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:46:15.790 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:15.149027+0000 osd.4 (osd.4) 406 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:46:15.791 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:13.845346+0000 mgr.y (mgr.24437) 758 : cluster [DBG] pgmap v1345: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:15.791 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:14.563924+0000 osd.7 (osd.7) 446 : cluster [DBG] 2.5 scrub starts 2024-02-23T00:46:15.791 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:14.566983+0000 osd.7 (osd.7) 447 : cluster [DBG] 2.5 scrub ok 2024-02-23T00:46:15.791 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:14.765812+0000 osd.5 (osd.5) 383 : cluster [DBG] 3.c deep-scrub starts 2024-02-23T00:46:15.791 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:14.767635+0000 osd.5 (osd.5) 384 : cluster [DBG] 3.c deep-scrub ok 2024-02-23T00:46:15.791 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:15.148124+0000 osd.4 (osd.4) 405 : cluster [DBG] 5.1c scrub starts 2024-02-23T00:46:15.791 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:15 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:15.149027+0000 osd.4 (osd.4) 406 : cluster [DBG] 5.1c scrub ok 2024-02-23T00:46:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:15.252308+0000 osd.6 (osd.6) 398 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:46:16.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:15.254276+0000 osd.6 (osd.6) 399 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:46:16.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:15.252308+0000 osd.6 (osd.6) 398 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:46:16.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:16 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:15.254276+0000 osd.6 (osd.6) 399 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:46:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:15.252308+0000 osd.6 (osd.6) 398 : cluster [DBG] 4.5 scrub starts 2024-02-23T00:46:16.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:16 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:15.254276+0000 osd.6 (osd.6) 399 : cluster [DBG] 4.5 scrub ok 2024-02-23T00:46:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:15.846047+0000 mgr.y (mgr.24437) 759 : cluster [DBG] pgmap v1346: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:16.125469+0000 osd.3 (osd.3) 337 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:46:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:16.129075+0000 osd.3 (osd.3) 338 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:46:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:16.737222+0000 osd.5 (osd.5) 385 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:46:17.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:16.738305+0000 osd.5 (osd.5) 386 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:46:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:15.846047+0000 mgr.y (mgr.24437) 759 : cluster [DBG] pgmap v1346: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:16.125469+0000 osd.3 (osd.3) 337 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:46:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:16.129075+0000 osd.3 (osd.3) 338 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:46:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:16.737222+0000 osd.5 (osd.5) 385 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:46:17.585 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:17 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:16.738305+0000 osd.5 (osd.5) 386 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:46:17.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 -- 172.21.15.87:0/1927337911 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0f7400 msgr2=0x7fdc0c103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:17.635 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 --2- 172.21.15.87:0/1927337911 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0f7400 0x7fdc0c103a20 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 rx=0x7fdc04009be0 tx=0x7fdc04004d10).stop 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 -- 172.21.15.87:0/1927337911 shutdown_connections 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 --2- 172.21.15.87:0/1927337911 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0f7400 0x7fdc0c103a20 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 --2- 172.21.15.87:0/1927337911 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc0c0fcbb0 0x7fdc0c0fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 --2- 172.21.15.87:0/1927337911 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc0c0fd510 0x7fdc0c0f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 -- 172.21.15.87:0/1927337911 >> 172.21.15.87:0/1927337911 conn(0x7fdc0c0f2820 msgr2=0x7fdc0c0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 -- 172.21.15.87:0/1927337911 shutdown_connections 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 -- 172.21.15.87:0/1927337911 wait complete. 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 librados: starting msgr at 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 librados: starting objecter 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 Processor -- start 2024-02-23T00:46:17.636 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 -- start start 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.585+0000 7fdc1176f700 1 librados: setting wanted keys 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 librados: calling monclient init 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc0c0f7400 0x7fdc0c106ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc0c0fcbb0 0x7fdc0c1074e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0fd510 0x7fdc0c1079d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c116670 con 0x7fdc0c0fcbb0 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c1167b0 con 0x7fdc0c0f7400 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c116920 con 0x7fdc0c0fd510 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0b7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0fd510 0x7fdc0c1079d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0a7fc700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc0c0fcbb0 0x7fdc0c1074e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0b7fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0fd510 0x7fdc0c1079d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44630/0 (socket says 172.21.15.87:44630) 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0b7fe700 1 -- 172.21.15.87:0/725155401 learned_addr learned my addr 172.21.15.87:0/725155401 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:17.637 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0affd700 1 --2- 172.21.15.87:0/725155401 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc0c0f7400 0x7fdc0c106ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0b7fe700 1 -- 172.21.15.87:0/725155401 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc0c0f7400 msgr2=0x7fdc0c106ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0b7fe700 1 --2- 172.21.15.87:0/725155401 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc0c0f7400 0x7fdc0c106ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0b7fe700 1 -- 172.21.15.87:0/725155401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc0c0fcbb0 msgr2=0x7fdc0c1074e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0b7fe700 1 --2- 172.21.15.87:0/725155401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc0c0fcbb0 0x7fdc0c1074e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0b7fe700 1 -- 172.21.15.87:0/725155401 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc00005040 con 0x7fdc0c0fd510 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc0b7fe700 1 --2- 172.21.15.87:0/725155401 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0fd510 0x7fdc0c1079d0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7fdc04004ed0 tx=0x7fdc04004d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdbf3fff700 10 client.?.objecter ms_handle_connect 0x7fdc0c0fd510 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdbf3fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdbf3fff700 1 -- 172.21.15.87:0/725155401 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400e040 con 0x7fdc0c0fd510 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdbf3fff700 1 -- 172.21.15.87:0/725155401 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc04018040 con 0x7fdc0c0fd510 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc040097c0 con 0x7fdc0c0fd510 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdbf3fff700 1 -- 172.21.15.87:0/725155401 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0401c3f0 con 0x7fdc0c0fd510 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 10 client.38621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c116e10 con 0x7fdc0c0fd510 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.586+0000 7fdc1176f700 1 librados: init done 2024-02-23T00:46:17.638 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.587+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c05fa60 con 0x7fdc0c0fd510 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.588+0000 7fdbf3fff700 1 -- 172.21.15.87:0/725155401 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fdc04013420 con 0x7fdc0c0fd510 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.588+0000 7fdbf3fff700 1 --2- 172.21.15.87:0/725155401 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdbf4062c90 0x7fdbf4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.588+0000 7fdc0affd700 1 --2- 172.21.15.87:0/725155401 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdbf4062c90 0x7fdbf4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.588+0000 7fdbf3fff700 1 -- 172.21.15.87:0/725155401 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(851..851 src has 400..851) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fdc0407e040 con 0x7fdc0c0fd510 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.588+0000 7fdbf3fff700 10 client.38621.objecter ms_dispatch 0x7fdc0c001550 osd_map(851..851 src has 400..851) v4 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.588+0000 7fdbf3fff700 3 client.38621.objecter handle_osd_map got epochs [851,851] > 0 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.588+0000 7fdbf3fff700 3 client.38621.objecter handle_osd_map decoding full epoch 851 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.588+0000 7fdbf3fff700 20 client.38621.objecter dump_active .. 0 homeless 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.589+0000 7fdbf3fff700 10 client.38621.objecter ms_handle_connect 0x7fdbf4062c90 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.589+0000 7fdc0affd700 1 --2- 172.21.15.87:0/725155401 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdbf4062c90 0x7fdbf4065150 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7fdc000092f0 tx=0x7fdc00009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:17.639 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.592+0000 7fdbf3fff700 1 -- 172.21.15.87:0/725155401 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fdc040acf60 con 0x7fdc0c0fd510 2024-02-23T00:46:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:15.846047+0000 mgr.y (mgr.24437) 759 : cluster [DBG] pgmap v1346: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:16.125469+0000 osd.3 (osd.3) 337 : cluster [DBG] 4.14 scrub starts 2024-02-23T00:46:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:16.129075+0000 osd.3 (osd.3) 338 : cluster [DBG] 4.14 scrub ok 2024-02-23T00:46:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:16.737222+0000 osd.5 (osd.5) 385 : cluster [DBG] 5.1b scrub starts 2024-02-23T00:46:17.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:17 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:16.738305+0000 osd.5 (osd.5) 386 : cluster [DBG] 5.1b scrub ok 2024-02-23T00:46:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.757+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdc0c106120 con 0x7fdc0c0fd510 2024-02-23T00:46:17.760 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.758+0000 7fdbf3fff700 1 -- 172.21.15.87:0/725155401 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+14 (secure 0 0 0) 0x7fdc04021030 con 0x7fdc0c0fd510 2024-02-23T00:46:17.760 INFO:teuthology.orchestra.run.smithi087.stdout:2830383448125 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 10 librados: watch_flush enter 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 10 librados: watch_flush exit 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 20 client.38621.objecter shutdown clearing up homeless session... 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 10 client.38621.objecter successfully canceled tick 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdbf4062c90 msgr2=0x7fdbf4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 --2- 172.21.15.87:0/725155401 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdbf4062c90 0x7fdbf4065150 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 rx=0x7fdc000092f0 tx=0x7fdc00009cb0).stop 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0fd510 msgr2=0x7fdc0c1079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 --2- 172.21.15.87:0/725155401 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0fd510 0x7fdc0c1079d0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 rx=0x7fdc04004ed0 tx=0x7fdc04004d10).stop 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 shutdown_connections 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 --2- 172.21.15.87:0/725155401 >> v2:172.21.15.87:6800/1231597463 conn(0x7fdbf4062c90 0x7fdbf4065150 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:17.763 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 --2- 172.21.15.87:0/725155401 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fdc0c0fd510 0x7fdc0c1079d0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:17.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 --2- 172.21.15.87:0/725155401 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fdc0c0f7400 0x7fdc0c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:17.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 --2- 172.21.15.87:0/725155401 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fdc0c0fcbb0 0x7fdc0c1074e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:17.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 >> 172.21.15.87:0/725155401 conn(0x7fdc0c0f2820 msgr2=0x7fdc0c0f4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:17.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 shutdown_connections 2024-02-23T00:46:17.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 -- 172.21.15.87:0/725155401 wait complete. 2024-02-23T00:46:17.764 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:17.760+0000 7fdc1176f700 1 librados: shutdown 2024-02-23T00:46:18.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:17.759578+0000 mon.c (mon.2) 523 : audit [DBG] from='client.? 172.21.15.87:0/725155401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-23T00:46:18.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:18 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:17.759578+0000 mon.c (mon.2) 523 : audit [DBG] from='client.? 172.21.15.87:0/725155401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-23T00:46:18.634 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2830383448120 got 2830383448125 for osd.0 2024-02-23T00:46:18.634 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-23T00:46:18.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:18 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:17.759578+0000 mon.c (mon.2) 523 : audit [DBG] from='client.? 172.21.15.87:0/725155401' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-23T00:46:18.906 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:19.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:17.846409+0000 mgr.y (mgr.24437) 760 : cluster [DBG] pgmap v1347: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:18.128402+0000 osd.3 (osd.3) 339 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:46:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:18.131282+0000 osd.3 (osd.3) 340 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:46:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:19.022182+0000 osd.4 (osd.4) 407 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:46:19.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:19 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:19.023600+0000 osd.4 (osd.4) 408 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:46:19.729 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:17.846409+0000 mgr.y (mgr.24437) 760 : cluster [DBG] pgmap v1347: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:19.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:18.128402+0000 osd.3 (osd.3) 339 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:46:19.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:18.131282+0000 osd.3 (osd.3) 340 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:46:19.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:19.022182+0000 osd.4 (osd.4) 407 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:46:19.730 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:19.023600+0000 osd.4 (osd.4) 408 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:46:19.730 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:17.846409+0000 mgr.y (mgr.24437) 760 : cluster [DBG] pgmap v1347: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:19.730 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:18.128402+0000 osd.3 (osd.3) 339 : cluster [DBG] 5.10 scrub starts 2024-02-23T00:46:19.730 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:18.131282+0000 osd.3 (osd.3) 340 : cluster [DBG] 5.10 scrub ok 2024-02-23T00:46:19.730 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:19.022182+0000 osd.4 (osd.4) 407 : cluster [DBG] 4.13 scrub starts 2024-02-23T00:46:19.730 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:19 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:19.023600+0000 osd.4 (osd.4) 408 : cluster [DBG] 4.13 scrub ok 2024-02-23T00:46:20.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:19.809909+0000 osd.7 (osd.7) 448 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:20.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:19.810063+0000 osd.7 (osd.7) 449 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:19.809909+0000 osd.7 (osd.7) 448 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:20.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:19.810063+0000 osd.7 (osd.7) 449 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:20.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.526+0000 7fcab4407700 1 -- 172.21.15.87:0/2338211066 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcaac0fd420 msgr2=0x7fcaac0f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:20.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.526+0000 7fcab4407700 1 --2- 172.21.15.87:0/2338211066 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcaac0fd420 0x7fcaac0f6c50 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 rx=0x7fcaa0004870 tx=0x7fcaa0009cb0).stop 2024-02-23T00:46:20.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.526+0000 7fcab4407700 1 -- 172.21.15.87:0/2338211066 shutdown_connections 2024-02-23T00:46:20.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.526+0000 7fcab4407700 1 --2- 172.21.15.87:0/2338211066 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcaac0f7330 0x7fcaac103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:20.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.526+0000 7fcab4407700 1 --2- 172.21.15.87:0/2338211066 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcaac0fcac0 0x7fcaac0fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:46:20.585 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.526+0000 7fcab4407700 1 --2- 172.21.15.87:0/2338211066 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcaac0fd420 0x7fcaac0f6c50 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.526+0000 7fcab4407700 1 -- 172.21.15.87:0/2338211066 >> 172.21.15.87:0/2338211066 conn(0x7fcaac0f26f0 msgr2=0x7fcaac0f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.526+0000 7fcab4407700 1 -- 172.21.15.87:0/2338211066 shutdown_connections 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.526+0000 7fcab4407700 1 -- 172.21.15.87:0/2338211066 wait complete. 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 librados: starting msgr at 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 librados: starting objecter 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 Processor -- start 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 -- start start 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 librados: setting wanted keys 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 librados: calling monclient init 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcaac0f7330 0x7fcaac0f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcaac0fcac0 0x7fcaac0f8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcaac0fd420 0x7fcaac0f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac116640 con 0x7fcaac0fcac0 2024-02-23T00:46:20.586 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac1167b0 con 0x7fcaac0fd420 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab4407700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaac116970 con 0x7fcaac0f7330 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab21a3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcaac0f7330 0x7fcaac0f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab21a3700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcaac0f7330 0x7fcaac0f88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:44652/0 (socket says 172.21.15.87:44652) 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab21a3700 1 -- 172.21.15.87:0/706240544 learned_addr learned my addr 172.21.15.87:0/706240544 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab19a2700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcaac0fcac0 0x7fcaac0f8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.527+0000 7fcab29a4700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcaac0fd420 0x7fcaac0f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab21a3700 1 -- 172.21.15.87:0/706240544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcaac0fd420 msgr2=0x7fcaac0f92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab21a3700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcaac0fd420 0x7fcaac0f92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab21a3700 1 -- 172.21.15.87:0/706240544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcaac0fcac0 msgr2=0x7fcaac0f8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:20.587 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab21a3700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcaac0fcac0 0x7fcaac0f8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab21a3700 1 -- 172.21.15.87:0/706240544 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4005040 con 0x7fcaac0f7330 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab19a2700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcaac0fcac0 0x7fcaac0f8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab21a3700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcaac0f7330 0x7fcaac0f88c0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 rx=0x7fca9c00bc30 tx=0x7fca9c009840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcaab7fe700 10 client.?.objecter ms_handle_connect 0x7fcaac0f7330 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcaab7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcaab7fe700 1 -- 172.21.15.87:0/706240544 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c00c630 con 0x7fcaac0f7330 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcaab7fe700 1 -- 172.21.15.87:0/706240544 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca9c010070 con 0x7fcaac0f7330 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa0005040 con 0x7fcaac0f7330 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcaab7fe700 1 -- 172.21.15.87:0/706240544 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c007810 con 0x7fcaac0f7330 2024-02-23T00:46:20.588 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab4407700 10 client.38627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaac116fb0 con 0x7fcaac0f7330 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.528+0000 7fcab4407700 1 librados: init done 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.529+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaac05f990 con 0x7fcaac0f7330 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.529+0000 7fcaab7fe700 1 -- 172.21.15.87:0/706240544 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fca9c007bc0 con 0x7fcaac0f7330 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.530+0000 7fcaab7fe700 1 --2- 172.21.15.87:0/706240544 >> v2:172.21.15.87:6800/1231597463 conn(0x7fca94062c90 0x7fca94065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.530+0000 7fcaab7fe700 1 -- 172.21.15.87:0/706240544 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(852..852 src has 400..852) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fca9c07c850 con 0x7fcaac0f7330 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.530+0000 7fcaab7fe700 10 client.38627.objecter ms_dispatch 0x7fcaac001550 osd_map(852..852 src has 400..852) v4 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.530+0000 7fcaab7fe700 3 client.38627.objecter handle_osd_map got epochs [852,852] > 0 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.530+0000 7fcaab7fe700 3 client.38627.objecter handle_osd_map decoding full epoch 852 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.530+0000 7fcab19a2700 1 --2- 172.21.15.87:0/706240544 >> v2:172.21.15.87:6800/1231597463 conn(0x7fca94062c90 0x7fca94065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.530+0000 7fcaab7fe700 20 client.38627.objecter dump_active .. 0 homeless 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.530+0000 7fcaab7fe700 10 client.38627.objecter ms_handle_connect 0x7fca94062c90 2024-02-23T00:46:20.589 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.530+0000 7fcab19a2700 1 --2- 172.21.15.87:0/706240544 >> v2:172.21.15.87:6800/1231597463 conn(0x7fca94062c90 0x7fca94065150 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 rx=0x7fcaa0004ee0 tx=0x7fcaa000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:20.590 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.534+0000 7fcaab7fe700 1 -- 172.21.15.87:0/706240544 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fca9c04ffa0 con 0x7fcaac0f7330 2024-02-23T00:46:20.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:19.809909+0000 osd.7 (osd.7) 448 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:20.692 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:20 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:19.810063+0000 osd.7 (osd.7) 449 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:20.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.699+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fcaac0f9da0 con 0x7fcaac0f7330 2024-02-23T00:46:20.701 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.700+0000 7fcaab7fe700 1 -- 172.21.15.87:0/706240544 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fca9c04fb90 con 0x7fcaac0f7330 2024-02-23T00:46:20.701 INFO:teuthology.orchestra.run.smithi087.stdout:55834575108 2024-02-23T00:46:20.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 10 librados: watch_flush enter 2024-02-23T00:46:20.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 10 librados: watch_flush exit 2024-02-23T00:46:20.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 20 client.38627.objecter shutdown clearing up homeless session... 2024-02-23T00:46:20.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 10 client.38627.objecter successfully canceled tick 2024-02-23T00:46:20.703 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 >> v2:172.21.15.87:6800/1231597463 conn(0x7fca94062c90 msgr2=0x7fca94065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 --2- 172.21.15.87:0/706240544 >> v2:172.21.15.87:6800/1231597463 conn(0x7fca94062c90 0x7fca94065150 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 rx=0x7fcaa0004ee0 tx=0x7fcaa000b040).stop 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcaac0f7330 msgr2=0x7fcaac0f88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcaac0f7330 0x7fcaac0f88c0 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 rx=0x7fca9c00bc30 tx=0x7fca9c009840).stop 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 shutdown_connections 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 --2- 172.21.15.87:0/706240544 >> v2:172.21.15.87:6800/1231597463 conn(0x7fca94062c90 0x7fca94065150 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fcaac0fd420 0x7fcaac0f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fcaac0f7330 0x7fcaac0f88c0 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 --2- 172.21.15.87:0/706240544 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fcaac0fcac0 0x7fcaac0f8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.701+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 >> 172.21.15.87:0/706240544 conn(0x7fcaac0f26f0 msgr2=0x7fcaac060eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.702+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 shutdown_connections 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.702+0000 7fcab4407700 1 -- 172.21.15.87:0/706240544 wait complete. 2024-02-23T00:46:20.704 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:20.702+0000 7fcab4407700 1 librados: shutdown 2024-02-23T00:46:21.572 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575104 got 55834575108 for osd.1 2024-02-23T00:46:21.572 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-23T00:46:21.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:19.846920+0000 mgr.y (mgr.24437) 761 : cluster [DBG] pgmap v1348: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:19.984900+0000 osd.0 (osd.0) 119 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:19.987882+0000 osd.0 (osd.0) 120 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:20.062513+0000 osd.6 (osd.6) 400 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:20.062866+0000 osd.6 (osd.6) 401 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:20.194716+0000 osd.6 (osd.6) 402 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:20.197703+0000 osd.6 (osd.6) 403 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:20.204531+0000 osd.3 (osd.3) 341 : cluster [DBG] 4.19 deep-scrub starts 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:20.208796+0000 osd.3 (osd.3) 342 : cluster [DBG] 4.19 deep-scrub ok 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:20.442438+0000 mon.a (mon.0) 4044 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-02-23T00:46:21.693 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:20.700217+0000 osd.5 (osd.5) 387 : cluster [DBG] 3.16 scrub starts 2024-02-23T00:46:21.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:20.700300+0000 mon.c (mon.2) 524 : audit [DBG] from='client.? 172.21.15.87:0/706240544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-23T00:46:21.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:20.701129+0000 osd.5 (osd.5) 388 : cluster [DBG] 3.16 scrub ok 2024-02-23T00:46:21.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:21.025585+0000 osd.4 (osd.4) 409 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:46:21.694 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:21 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:21.026573+0000 osd.4 (osd.4) 410 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:46:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:19.846920+0000 mgr.y (mgr.24437) 761 : cluster [DBG] pgmap v1348: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-23T00:46:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:19.984900+0000 osd.0 (osd.0) 119 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:46:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:19.987882+0000 osd.0 (osd.0) 120 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:46:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:20.062513+0000 osd.6 (osd.6) 400 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:20.062866+0000 osd.6 (osd.6) 401 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:21.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:20.194716+0000 osd.6 (osd.6) 402 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:20.197703+0000 osd.6 (osd.6) 403 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:20.204531+0000 osd.3 (osd.3) 341 : cluster [DBG] 4.19 deep-scrub starts 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:20.208796+0000 osd.3 (osd.3) 342 : cluster [DBG] 4.19 deep-scrub ok 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:20.442438+0000 mon.a (mon.0) 4044 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:20.700217+0000 osd.5 (osd.5) 387 : cluster [DBG] 3.16 scrub starts 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:20.700300+0000 mon.c (mon.2) 524 : audit [DBG] from='client.? 172.21.15.87:0/706240544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:20.701129+0000 osd.5 (osd.5) 388 : cluster [DBG] 3.16 scrub ok 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:21.025585+0000 osd.4 (osd.4) 409 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:21.026573+0000 osd.4 (osd.4) 410 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:19.846920+0000 mgr.y (mgr.24437) 761 : cluster [DBG] pgmap v1348: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-23T00:46:21.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:19.984900+0000 osd.0 (osd.0) 119 : cluster [DBG] 2.13 scrub starts 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:19.987882+0000 osd.0 (osd.0) 120 : cluster [DBG] 2.13 scrub ok 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:20.062513+0000 osd.6 (osd.6) 400 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:20.062866+0000 osd.6 (osd.6) 401 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:20.194716+0000 osd.6 (osd.6) 402 : cluster [DBG] 5.2 scrub starts 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:20.197703+0000 osd.6 (osd.6) 403 : cluster [DBG] 5.2 scrub ok 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:20.204531+0000 osd.3 (osd.3) 341 : cluster [DBG] 4.19 deep-scrub starts 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:20.208796+0000 osd.3 (osd.3) 342 : cluster [DBG] 4.19 deep-scrub ok 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:20.442438+0000 mon.a (mon.0) 4044 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:20.700217+0000 osd.5 (osd.5) 387 : cluster [DBG] 3.16 scrub starts 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:20.700300+0000 mon.c (mon.2) 524 : audit [DBG] from='client.? 172.21.15.87:0/706240544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:20.701129+0000 osd.5 (osd.5) 388 : cluster [DBG] 3.16 scrub ok 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:21.025585+0000 osd.4 (osd.4) 409 : cluster [DBG] 2.1c scrub starts 2024-02-23T00:46:21.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:21 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:21.026573+0000 osd.4 (osd.4) 410 : cluster [DBG] 2.1c scrub ok 2024-02-23T00:46:21.845 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:22.333 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:22] "GET /metrics HTTP/1.1" 200 186068 "" "Prometheus/2.33.4" 2024-02-23T00:46:22.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:21.252541+0000 osd.3 (osd.3) 343 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:46:22.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:21.255538+0000 osd.3 (osd.3) 344 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:46:22.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:21.318753+0000 osd.1 (osd.1) 458 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:46:22.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:21.322224+0000 osd.1 (osd.1) 459 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:21.616468+0000 osd.7 (osd.7) 450 : cluster [DBG] 3.11 scrub starts 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:21.617460+0000 osd.7 (osd.7) 451 : cluster [DBG] 3.11 scrub ok 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:22.303073+0000 mon.a (mon.0) 4045 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:21.252541+0000 osd.3 (osd.3) 343 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:21.255538+0000 osd.3 (osd.3) 344 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:21.318753+0000 osd.1 (osd.1) 458 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:21.322224+0000 osd.1 (osd.1) 459 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:21.616468+0000 osd.7 (osd.7) 450 : cluster [DBG] 3.11 scrub starts 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:21.617460+0000 osd.7 (osd.7) 451 : cluster [DBG] 3.11 scrub ok 2024-02-23T00:46:22.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:22 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:22.303073+0000 mon.a (mon.0) 4045 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-02-23T00:46:22.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:21.252541+0000 osd.3 (osd.3) 343 : cluster [DBG] 22.4 scrub starts 2024-02-23T00:46:22.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:21.255538+0000 osd.3 (osd.3) 344 : cluster [DBG] 22.4 scrub ok 2024-02-23T00:46:22.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:21.318753+0000 osd.1 (osd.1) 458 : cluster [DBG] 4.7 scrub starts 2024-02-23T00:46:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:21.322224+0000 osd.1 (osd.1) 459 : cluster [DBG] 4.7 scrub ok 2024-02-23T00:46:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:21.616468+0000 osd.7 (osd.7) 450 : cluster [DBG] 3.11 scrub starts 2024-02-23T00:46:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:21.617460+0000 osd.7 (osd.7) 451 : cluster [DBG] 3.11 scrub ok 2024-02-23T00:46:22.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:22 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:22.303073+0000 mon.a (mon.0) 4045 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-02-23T00:46:23.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:21.456470+0000 mgr.y (mgr.24437) 762 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:23.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:21.847626+0000 mgr.y (mgr.24437) 763 : cluster [DBG] pgmap v1350: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:23.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:22.037712+0000 osd.0 (osd.0) 121 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:46:23.727 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:22.040676+0000 osd.0 (osd.0) 122 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:22.345973+0000 osd.4 (osd.4) 411 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:22.346200+0000 osd.4 (osd.4) 412 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:22.690430+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:22.691453+0000 osd.5 (osd.5) 390 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:22.988712+0000 osd.4 (osd.4) 413 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:22.989874+0000 osd.4 (osd.4) 414 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:23.309771+0000 mon.a (mon.0) 4046 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:21.456470+0000 mgr.y (mgr.24437) 762 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:21.847626+0000 mgr.y (mgr.24437) 763 : cluster [DBG] pgmap v1350: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:22.037712+0000 osd.0 (osd.0) 121 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:46:23.728 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:22.040676+0000 osd.0 (osd.0) 122 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:46:23.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:22.345973+0000 osd.4 (osd.4) 411 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:23.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:22.346200+0000 osd.4 (osd.4) 412 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:23.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:22.690430+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:46:23.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:22.691453+0000 osd.5 (osd.5) 390 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:46:23.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:22.988712+0000 osd.4 (osd.4) 413 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:46:23.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:22.989874+0000 osd.4 (osd.4) 414 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:46:23.729 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:23.309771+0000 mon.a (mon.0) 4046 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-02-23T00:46:23.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:46:23.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:21.456470+0000 mgr.y (mgr.24437) 762 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:21.847626+0000 mgr.y (mgr.24437) 763 : cluster [DBG] pgmap v1350: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:22.037712+0000 osd.0 (osd.0) 121 : cluster [DBG] 3.6 scrub starts 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:22.040676+0000 osd.0 (osd.0) 122 : cluster [DBG] 3.6 scrub ok 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:22.345973+0000 osd.4 (osd.4) 411 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:22.346200+0000 osd.4 (osd.4) 412 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:22.690430+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.3 scrub starts 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:22.691453+0000 osd.5 (osd.5) 390 : cluster [DBG] 2.3 scrub ok 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:22.988712+0000 osd.4 (osd.4) 413 : cluster [DBG] 6.2 scrub starts 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:22.989874+0000 osd.4 (osd.4) 414 : cluster [DBG] 6.2 scrub ok 2024-02-23T00:46:23.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:23.309771+0000 mon.a (mon.0) 4046 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.982+0000 7fc54c4fe700 1 -- 172.21.15.87:0/743051584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fcbb0 msgr2=0x7fc5440fcf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.982+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/743051584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fcbb0 0x7fc5440fcf90 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 rx=0x7fc540004870 tx=0x7fc540009cb0).stop 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 -- 172.21.15.87:0/743051584 shutdown_connections 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/743051584 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5440f7400 0x7fc544103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/743051584 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fcbb0 0x7fc5440fcf90 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/743051584 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5440fd510 0x7fc5440f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 -- 172.21.15.87:0/743051584 >> 172.21.15.87:0/743051584 conn(0x7fc5440f2820 msgr2=0x7fc5440f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 -- 172.21.15.87:0/743051584 shutdown_connections 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 -- 172.21.15.87:0/743051584 wait complete. 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 librados: starting msgr at 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 librados: starting objecter 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 Processor -- start 2024-02-23T00:46:23.986 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 -- start start 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 librados: setting wanted keys 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 librados: calling monclient init 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5440f7400 0x7fc54410e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5440fcbb0 0x7fc54410e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fd510 0x7fc544108360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc544108850 con 0x7fc5440fd510 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5441089c0 con 0x7fc5440f7400 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.983+0000 7fc54c4fe700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc544108b80 con 0x7fc5440fcbb0 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54aa9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fd510 0x7fc544108360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc549a99700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5440fcbb0 0x7fc54410e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54aa9b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fd510 0x7fc544108360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49690/0 (socket says 172.21.15.87:49690) 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54aa9b700 1 -- 172.21.15.87:0/4762826 learned_addr learned my addr 172.21.15.87:0/4762826 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54a29a700 1 --2- 172.21.15.87:0/4762826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5440f7400 0x7fc54410e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54aa9b700 1 -- 172.21.15.87:0/4762826 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5440fcbb0 msgr2=0x7fc54410e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54aa9b700 1 --2- 172.21.15.87:0/4762826 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5440fcbb0 0x7fc54410e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:23.987 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54aa9b700 1 -- 172.21.15.87:0/4762826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5440f7400 msgr2=0x7fc54410e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54aa9b700 1 --2- 172.21.15.87:0/4762826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5440f7400 0x7fc54410e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54aa9b700 1 -- 172.21.15.87:0/4762826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c0097c0 con 0x7fc5440fd510 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc549a99700 1 --2- 172.21.15.87:0/4762826 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5440fcbb0 0x7fc54410e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54aa9b700 1 --2- 172.21.15.87:0/4762826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fd510 0x7fc544108360 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7fc53c001df0 tx=0x7fc53c004c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc53b7fe700 10 client.?.objecter ms_handle_connect 0x7fc5440fd510 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc53b7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc53b7fe700 1 -- 172.21.15.87:0/4762826 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c009a10 con 0x7fc5440fd510 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc53b7fe700 1 -- 172.21.15.87:0/4762826 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc53c0090d0 con 0x7fc5440fd510 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc540005040 con 0x7fc5440fd510 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc53b7fe700 1 -- 172.21.15.87:0/4762826 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c007d30 con 0x7fc5440fd510 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54c4fe700 10 client.28155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc544109170 con 0x7fc5440fd510 2024-02-23T00:46:23.988 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.984+0000 7fc54c4fe700 1 librados: init done 2024-02-23T00:46:23.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.985+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54405fa60 con 0x7fc5440fd510 2024-02-23T00:46:23.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc53b7fe700 1 -- 172.21.15.87:0/4762826 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fc53c009240 con 0x7fc5440fd510 2024-02-23T00:46:23.989 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc53b7fe700 1 --2- 172.21.15.87:0/4762826 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc530062c90 0x7fc530065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:23.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc53b7fe700 1 -- 172.21.15.87:0/4762826 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(854..854 src has 400..854) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fc53c07d820 con 0x7fc5440fd510 2024-02-23T00:46:23.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc53b7fe700 10 client.28155.objecter ms_dispatch 0x7fc544001550 osd_map(854..854 src has 400..854) v4 2024-02-23T00:46:23.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc53b7fe700 3 client.28155.objecter handle_osd_map got epochs [854,854] > 0 2024-02-23T00:46:23.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc53b7fe700 3 client.28155.objecter handle_osd_map decoding full epoch 854 2024-02-23T00:46:23.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc54a29a700 1 --2- 172.21.15.87:0/4762826 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc530062c90 0x7fc530065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:23.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc53b7fe700 20 client.28155.objecter dump_active .. 0 homeless 2024-02-23T00:46:23.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc54a29a700 1 --2- 172.21.15.87:0/4762826 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc530062c90 0x7fc530065150 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7fc5400043d0 tx=0x7fc54000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:23.990 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.986+0000 7fc53b7fe700 10 client.28155.objecter ms_handle_connect 0x7fc530062c90 2024-02-23T00:46:23.991 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:23.990+0000 7fc53b7fe700 1 -- 172.21.15.87:0/4762826 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fc53c050f10 con 0x7fc5440fd510 2024-02-23T00:46:24.156 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.155+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc5441060e0 con 0x7fc5440fd510 2024-02-23T00:46:24.157 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.155+0000 7fc53b7fe700 1 -- 172.21.15.87:0/4762826 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc53c01c300 con 0x7fc5440fd510 2024-02-23T00:46:24.157 INFO:teuthology.orchestra.run.smithi087.stdout:77309411585 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 10 librados: watch_flush enter 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 10 librados: watch_flush exit 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 20 client.28155.objecter shutdown clearing up homeless session... 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 10 client.28155.objecter successfully canceled tick 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc530062c90 msgr2=0x7fc530065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/4762826 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc530062c90 0x7fc530065150 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 rx=0x7fc5400043d0 tx=0x7fc54000b040).stop 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fd510 msgr2=0x7fc544108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/4762826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fd510 0x7fc544108360 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 rx=0x7fc53c001df0 tx=0x7fc53c004c50).stop 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 shutdown_connections 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/4762826 >> v2:172.21.15.87:6800/1231597463 conn(0x7fc530062c90 0x7fc530065150 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/4762826 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fc5440fd510 0x7fc544108360 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:24.159 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/4762826 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fc5440f7400 0x7fc54410e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:24.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 --2- 172.21.15.87:0/4762826 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fc5440fcbb0 0x7fc54410e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:24.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 >> 172.21.15.87:0/4762826 conn(0x7fc5440f2820 msgr2=0x7fc5440f4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:24.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 shutdown_connections 2024-02-23T00:46:24.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 -- 172.21.15.87:0/4762826 wait complete. 2024-02-23T00:46:24.160 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:24.157+0000 7fc54c4fe700 1 librados: shutdown 2024-02-23T00:46:24.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:23.324364+0000 osd.1 (osd.1) 460 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:46:24.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:23.327409+0000 osd.1 (osd.1) 461 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:46:24.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:23.848227+0000 mgr.y (mgr.24437) 764 : cluster [DBG] pgmap v1353: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:24.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:24.155805+0000 mon.a (mon.0) 4047 : audit [DBG] from='client.? 172.21.15.87:0/4762826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-23T00:46:24.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:23.324364+0000 osd.1 (osd.1) 460 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:46:24.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:23.327409+0000 osd.1 (osd.1) 461 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:46:24.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:23.848227+0000 mgr.y (mgr.24437) 764 : cluster [DBG] pgmap v1353: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:24.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:24 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:24.155805+0000 mon.a (mon.0) 4047 : audit [DBG] from='client.? 172.21.15.87:0/4762826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-23T00:46:24.896 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411580 got 77309411585 for osd.2 2024-02-23T00:46:24.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-23T00:46:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:23.324364+0000 osd.1 (osd.1) 460 : cluster [DBG] 22.1a scrub starts 2024-02-23T00:46:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:23.327409+0000 osd.1 (osd.1) 461 : cluster [DBG] 22.1a scrub ok 2024-02-23T00:46:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:23.848227+0000 mgr.y (mgr.24437) 764 : cluster [DBG] pgmap v1353: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:24.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:24.155805+0000 mon.a (mon.0) 4047 : audit [DBG] from='client.? 172.21.15.87:0/4762826' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-23T00:46:25.163 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:24.269037+0000 osd.3 (osd.3) 345 : cluster [DBG] 5.17 deep-scrub starts 2024-02-23T00:46:25.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:24.269953+0000 osd.3 (osd.3) 346 : cluster [DBG] 5.17 deep-scrub ok 2024-02-23T00:46:25.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:24.269037+0000 osd.3 (osd.3) 345 : cluster [DBG] 5.17 deep-scrub starts 2024-02-23T00:46:25.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:25 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:24.269953+0000 osd.3 (osd.3) 346 : cluster [DBG] 5.17 deep-scrub ok 2024-02-23T00:46:25.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:24.269037+0000 osd.3 (osd.3) 345 : cluster [DBG] 5.17 deep-scrub starts 2024-02-23T00:46:25.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:24.269953+0000 osd.3 (osd.3) 346 : cluster [DBG] 5.17 deep-scrub ok 2024-02-23T00:46:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:25.640762+0000 osd.5 (osd.5) 391 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:46:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:25.642133+0000 osd.5 (osd.5) 392 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:46:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:25.652126+0000 osd.7 (osd.7) 452 : cluster [DBG] 2.b scrub starts 2024-02-23T00:46:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:25.653118+0000 osd.7 (osd.7) 453 : cluster [DBG] 2.b scrub ok 2024-02-23T00:46:26.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:25.848981+0000 mgr.y (mgr.24437) 765 : cluster [DBG] pgmap v1354: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-23T00:46:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:25.640762+0000 osd.5 (osd.5) 391 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:46:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:25.642133+0000 osd.5 (osd.5) 392 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:46:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:25.652126+0000 osd.7 (osd.7) 452 : cluster [DBG] 2.b scrub starts 2024-02-23T00:46:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:25.653118+0000 osd.7 (osd.7) 453 : cluster [DBG] 2.b scrub ok 2024-02-23T00:46:26.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:26 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:25.848981+0000 mgr.y (mgr.24437) 765 : cluster [DBG] pgmap v1354: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-23T00:46:26.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:25.640762+0000 osd.5 (osd.5) 391 : cluster [DBG] 5.7 scrub starts 2024-02-23T00:46:26.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:25.642133+0000 osd.5 (osd.5) 392 : cluster [DBG] 5.7 scrub ok 2024-02-23T00:46:26.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:25.652126+0000 osd.7 (osd.7) 452 : cluster [DBG] 2.b scrub starts 2024-02-23T00:46:26.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:25.653118+0000 osd.7 (osd.7) 453 : cluster [DBG] 2.b scrub ok 2024-02-23T00:46:26.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:25.848981+0000 mgr.y (mgr.24437) 765 : cluster [DBG] pgmap v1354: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-23T00:46:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.260+0000 7f1bae635700 1 -- 172.21.15.87:0/2729988921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba81006b0 msgr2=0x7f1ba80f9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.260+0000 7f1bae635700 1 --2- 172.21.15.87:0/2729988921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba81006b0 0x7f1ba80f9f70 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 rx=0x7f1b9c004870 tx=0x7f1b9c009cb0).stop 2024-02-23T00:46:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 -- 172.21.15.87:0/2729988921 shutdown_connections 2024-02-23T00:46:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 --2- 172.21.15.87:0/2729988921 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ba80fa540 0x7f1ba80fa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-02-23T00:46:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 --2- 172.21.15.87:0/2729988921 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1ba80ffde0 0x7f1ba81001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 --2- 172.21.15.87:0/2729988921 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba81006b0 0x7f1ba80f9f70 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:27.283 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 -- 172.21.15.87:0/2729988921 >> 172.21.15.87:0/2729988921 conn(0x7f1ba80f59f0 msgr2=0x7f1ba80f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 -- 172.21.15.87:0/2729988921 shutdown_connections 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 -- 172.21.15.87:0/2729988921 wait complete. 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 librados: starting msgr at 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 librados: starting objecter 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 Processor -- start 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 -- start start 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 librados: setting wanted keys 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.261+0000 7f1bae635700 1 librados: calling monclient init 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba80fa540 0x7f1ba81104c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ba80ffde0 0x7f1ba81109b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1ba81006b0 0x7f1ba810a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba810ab50 con 0x7f1ba80fa540 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba810ac90 con 0x7f1ba80ffde0 2024-02-23T00:46:27.284 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba810ae50 con 0x7f1ba81006b0 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba80fa540 0x7f1ba81104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba7fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba80fa540 0x7f1ba81104c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49706/0 (socket says 172.21.15.87:49706) 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba7fff700 1 -- 172.21.15.87:0/4095243327 learned_addr learned my addr 172.21.15.87:0/4095243327 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bacbd2700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1ba81006b0 0x7f1ba810a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba7fff700 1 -- 172.21.15.87:0/4095243327 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1ba81006b0 msgr2=0x7f1ba810a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba7fff700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1ba81006b0 0x7f1ba810a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba77fe700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ba80ffde0 0x7f1ba81109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba7fff700 1 -- 172.21.15.87:0/4095243327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ba80ffde0 msgr2=0x7f1ba81109b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba7fff700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ba80ffde0 0x7f1ba81109b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba7fff700 1 -- 172.21.15.87:0/4095243327 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b98005040 con 0x7f1ba80fa540 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba77fe700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ba80ffde0 0x7f1ba81109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba7fff700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba80fa540 0x7f1ba81104c0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f1b980048f0 tx=0x7f1b98009cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba57fa700 10 client.?.objecter ms_handle_connect 0x7f1ba80fa540 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba57fa700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba57fa700 1 -- 172.21.15.87:0/4095243327 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98007b00 con 0x7f1ba80fa540 2024-02-23T00:46:27.285 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba57fa700 1 -- 172.21.15.87:0/4095243327 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b9800f040 con 0x7f1ba80fa540 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9c005040 con 0x7f1ba80fa540 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1ba57fa700 1 -- 172.21.15.87:0/4095243327 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98014a00 con 0x7f1ba80fa540 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 10 client.28161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba810b490 con 0x7f1ba80fa540 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.262+0000 7f1bae635700 1 librados: init done 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.263+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba805ee80 con 0x7f1ba80fa540 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba57fa700 1 -- 172.21.15.87:0/4095243327 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f1b9800e580 con 0x7f1ba80fa540 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba57fa700 1 --2- 172.21.15.87:0/4095243327 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b90062c90 0x7f1b90065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba57fa700 1 -- 172.21.15.87:0/4095243327 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(855..855 src has 400..855) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f1b9807cb70 con 0x7f1ba80fa540 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba57fa700 10 client.28161.objecter ms_dispatch 0x7f1ba8001550 osd_map(855..855 src has 400..855) v4 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba57fa700 3 client.28161.objecter handle_osd_map got epochs [855,855] > 0 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba57fa700 3 client.28161.objecter handle_osd_map decoding full epoch 855 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba77fe700 1 --2- 172.21.15.87:0/4095243327 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b90062c90 0x7f1b90065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba57fa700 20 client.28161.objecter dump_active .. 0 homeless 2024-02-23T00:46:27.286 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba77fe700 1 --2- 172.21.15.87:0/4095243327 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b90062c90 0x7f1b90065150 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f1ba810be00 tx=0x7f1b9c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:27.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.264+0000 7f1ba57fa700 10 client.28161.objecter ms_handle_connect 0x7f1b90062c90 2024-02-23T00:46:27.287 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.268+0000 7f1ba57fa700 1 -- 172.21.15.87:0/4095243327 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f1b98049f60 con 0x7f1ba80fa540 2024-02-23T00:46:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.434+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1ba810bc40 con 0x7f1ba80fa540 2024-02-23T00:46:27.436 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.434+0000 7f1ba57fa700 1 -- 172.21.15.87:0/4095243327 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1b980502c0 con 0x7f1ba80fa540 2024-02-23T00:46:27.437 INFO:teuthology.orchestra.run.smithi087.stdout:111669149950 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 10 librados: watch_flush enter 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 10 librados: watch_flush exit 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 20 client.28161.objecter shutdown clearing up homeless session... 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 10 client.28161.objecter successfully canceled tick 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b90062c90 msgr2=0x7f1b90065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 --2- 172.21.15.87:0/4095243327 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b90062c90 0x7f1b90065150 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f1ba810be00 tx=0x7f1b9c00b040).stop 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba80fa540 msgr2=0x7f1ba81104c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba80fa540 0x7f1ba81104c0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 rx=0x7f1b980048f0 tx=0x7f1b98009cb0).stop 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 shutdown_connections 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 --2- 172.21.15.87:0/4095243327 >> v2:172.21.15.87:6800/1231597463 conn(0x7f1b90062c90 0x7f1b90065150 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f1ba81006b0 0x7f1ba810a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:27.438 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f1ba80fa540 0x7f1ba81104c0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:27.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 --2- 172.21.15.87:0/4095243327 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f1ba80ffde0 0x7f1ba81109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:27.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 >> 172.21.15.87:0/4095243327 conn(0x7f1ba80f59f0 msgr2=0x7f1ba80fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:27.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 shutdown_connections 2024-02-23T00:46:27.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 -- 172.21.15.87:0/4095243327 wait complete. 2024-02-23T00:46:27.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:27.436+0000 7f1bae635700 1 librados: shutdown 2024-02-23T00:46:27.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:25.675464+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:27.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:25.675620+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:27.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:26.466418+0000 mon.a (mon.0) 4048 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-02-23T00:46:27.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:27.435090+0000 mon.a (mon.0) 4049 : audit [DBG] from='client.? 172.21.15.87:0/4095243327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-23T00:46:27.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:25.675464+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:27.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:25.675620+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:27.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:26.466418+0000 mon.a (mon.0) 4048 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-02-23T00:46:27.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:27 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:27.435090+0000 mon.a (mon.0) 4049 : audit [DBG] from='client.? 172.21.15.87:0/4095243327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-23T00:46:27.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:25.675464+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:27.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:25.675620+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:27.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:26.466418+0000 mon.a (mon.0) 4048 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-02-23T00:46:27.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:27.435090+0000 mon.a (mon.0) 4049 : audit [DBG] from='client.? 172.21.15.87:0/4095243327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-23T00:46:28.337 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149945 got 111669149950 for osd.3 2024-02-23T00:46:28.338 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-23T00:46:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:27.301312+0000 osd.6 (osd.6) 404 : cluster [DBG] 22.1 deep-scrub starts 2024-02-23T00:46:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:27.306797+0000 osd.6 (osd.6) 405 : cluster [DBG] 22.1 deep-scrub ok 2024-02-23T00:46:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:27.413252+0000 osd.1 (osd.1) 462 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:46:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:27.416773+0000 osd.1 (osd.1) 463 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:46:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:27.647184+0000 osd.5 (osd.5) 393 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:46:28.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:27.648141+0000 osd.5 (osd.5) 394 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:27.732364+0000 osd.7 (osd.7) 454 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:27.735607+0000 osd.7 (osd.7) 455 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:27.849552+0000 mgr.y (mgr.24437) 766 : cluster [DBG] pgmap v1356: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:27.301312+0000 osd.6 (osd.6) 404 : cluster [DBG] 22.1 deep-scrub starts 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:27.306797+0000 osd.6 (osd.6) 405 : cluster [DBG] 22.1 deep-scrub ok 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:27.413252+0000 osd.1 (osd.1) 462 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:27.416773+0000 osd.1 (osd.1) 463 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:27.647184+0000 osd.5 (osd.5) 393 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:27.648141+0000 osd.5 (osd.5) 394 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:27.732364+0000 osd.7 (osd.7) 454 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:46:28.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:27.735607+0000 osd.7 (osd.7) 455 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:46:28.585 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:28 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:27.849552+0000 mgr.y (mgr.24437) 766 : cluster [DBG] pgmap v1356: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:28.616 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:27.301312+0000 osd.6 (osd.6) 404 : cluster [DBG] 22.1 deep-scrub starts 2024-02-23T00:46:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:27.306797+0000 osd.6 (osd.6) 405 : cluster [DBG] 22.1 deep-scrub ok 2024-02-23T00:46:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:27.413252+0000 osd.1 (osd.1) 462 : cluster [DBG] 4.12 scrub starts 2024-02-23T00:46:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:27.416773+0000 osd.1 (osd.1) 463 : cluster [DBG] 4.12 scrub ok 2024-02-23T00:46:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:27.647184+0000 osd.5 (osd.5) 393 : cluster [DBG] 2.18 scrub starts 2024-02-23T00:46:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:27.648141+0000 osd.5 (osd.5) 394 : cluster [DBG] 2.18 scrub ok 2024-02-23T00:46:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:27.732364+0000 osd.7 (osd.7) 454 : cluster [DBG] 6.8 scrub starts 2024-02-23T00:46:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:27.735607+0000 osd.7 (osd.7) 455 : cluster [DBG] 6.8 scrub ok 2024-02-23T00:46:28.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:27.849552+0000 mgr.y (mgr.24437) 766 : cluster [DBG] pgmap v1356: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.197082+0000 osd.1 (osd.1) 464 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.197471+0000 osd.1 (osd.1) 465 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.234100+0000 osd.3 (osd.3) 347 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.234282+0000 osd.3 (osd.3) 348 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.288349+0000 osd.3 (osd.3) 349 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:46:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.291519+0000 osd.3 (osd.3) 350 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:46:29.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.429890+0000 osd.1 (osd.1) 466 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.433166+0000 osd.1 (osd.1) 467 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.473443+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.612644+0000 osd.5 (osd.5) 395 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:28.613760+0000 osd.5 (osd.5) 396 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.197082+0000 osd.1 (osd.1) 464 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.197471+0000 osd.1 (osd.1) 465 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.234100+0000 osd.3 (osd.3) 347 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.234282+0000 osd.3 (osd.3) 348 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.288349+0000 osd.3 (osd.3) 349 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.291519+0000 osd.3 (osd.3) 350 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.429890+0000 osd.1 (osd.1) 466 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:46:29.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.433166+0000 osd.1 (osd.1) 467 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:46:29.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.473443+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-02-23T00:46:29.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.612644+0000 osd.5 (osd.5) 395 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:46:29.835 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:29 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:28.613760+0000 osd.5 (osd.5) 396 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:46:29.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.197082+0000 osd.1 (osd.1) 464 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.197471+0000 osd.1 (osd.1) 465 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.234100+0000 osd.3 (osd.3) 347 : cluster [DBG] purged_snaps scrub starts 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.234282+0000 osd.3 (osd.3) 348 : cluster [DBG] purged_snaps scrub ok 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.288349+0000 osd.3 (osd.3) 349 : cluster [DBG] 2.1e scrub starts 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.291519+0000 osd.3 (osd.3) 350 : cluster [DBG] 2.1e scrub ok 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.429890+0000 osd.1 (osd.1) 466 : cluster [DBG] 2.15 scrub starts 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.433166+0000 osd.1 (osd.1) 467 : cluster [DBG] 2.15 scrub ok 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.473443+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.612644+0000 osd.5 (osd.5) 395 : cluster [DBG] 2.12 scrub starts 2024-02-23T00:46:29.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:28.613760+0000 osd.5 (osd.5) 396 : cluster [DBG] 2.12 scrub ok 2024-02-23T00:46:30.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.560+0000 7fb5610a1700 1 -- 172.21.15.87:0/3446323032 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c0f9580 msgr2=0x7fb55c101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:30.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.560+0000 7fb5610a1700 1 --2- 172.21.15.87:0/3446323032 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c0f9580 0x7fb55c101c90 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 rx=0x7fb5500047b0 tx=0x7fb550009cb0).stop 2024-02-23T00:46:30.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.560+0000 7fb5610a1700 1 -- 172.21.15.87:0/3446323032 shutdown_connections 2024-02-23T00:46:30.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.560+0000 7fb5610a1700 1 --2- 172.21.15.87:0/3446323032 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb55c102180 0x7fb55c104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:30.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.560+0000 7fb5610a1700 1 --2- 172.21.15.87:0/3446323032 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb55c0f8c20 0x7fb55c0f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:30.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.560+0000 7fb5610a1700 1 --2- 172.21.15.87:0/3446323032 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c0f9580 0x7fb55c101c90 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:30.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.560+0000 7fb5610a1700 1 -- 172.21.15.87:0/3446323032 >> 172.21.15.87:0/3446323032 conn(0x7fb55c0f26f0 msgr2=0x7fb55c0f4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:30.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.560+0000 7fb5610a1700 1 -- 172.21.15.87:0/3446323032 shutdown_connections 2024-02-23T00:46:30.562 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.560+0000 7fb5610a1700 1 -- 172.21.15.87:0/3446323032 wait complete. 2024-02-23T00:46:30.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 librados: starting msgr at 2024-02-23T00:46:30.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 librados: starting objecter 2024-02-23T00:46:30.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 Processor -- start 2024-02-23T00:46:30.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 -- start start 2024-02-23T00:46:30.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 librados: setting wanted keys 2024-02-23T00:46:30.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 librados: calling monclient init 2024-02-23T00:46:30.564 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb55c0f8c20 0x7fb55c0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb55c0f9580 0x7fb55c0f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c102180 0x7fb55c0f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c116610 con 0x7fb55c102180 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c116780 con 0x7fb55c0f8c20 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.561+0000 7fb5610a1700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c116940 con 0x7fb55c0f9580 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55b59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c102180 0x7fb55c0f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55ad9d700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb55c0f8c20 0x7fb55c0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55b59e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c102180 0x7fb55c0f75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:49732/0 (socket says 172.21.15.87:49732) 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55b59e700 1 -- 172.21.15.87:0/4209288112 learned_addr learned my addr 172.21.15.87:0/4209288112 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55b59e700 1 -- 172.21.15.87:0/4209288112 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb55c0f9580 msgr2=0x7fb55c0f70c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55b59e700 1 --2- 172.21.15.87:0/4209288112 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb55c0f9580 0x7fb55c0f70c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55b59e700 1 -- 172.21.15.87:0/4209288112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb55c0f8c20 msgr2=0x7fb55c0f6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55b59e700 1 --2- 172.21.15.87:0/4209288112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb55c0f8c20 0x7fb55c0f6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:30.565 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55b59e700 1 -- 172.21.15.87:0/4209288112 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5540097c0 con 0x7fb55c102180 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb55b59e700 1 --2- 172.21.15.87:0/4209288112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c102180 0x7fb55c0f75b0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 rx=0x7fb554001e80 tx=0x7fb554004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb543fff700 10 client.?.objecter ms_handle_connect 0x7fb55c102180 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb543fff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb543fff700 1 -- 172.21.15.87:0/4209288112 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55400fbe0 con 0x7fb55c102180 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb550005040 con 0x7fb55c102180 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb543fff700 1 -- 172.21.15.87:0/4209288112 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5540095a0 con 0x7fb55c102180 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb543fff700 1 -- 172.21.15.87:0/4209288112 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb554007ea0 con 0x7fb55c102180 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb5610a1700 10 client.28167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c116fd0 con 0x7fb55c102180 2024-02-23T00:46:30.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.562+0000 7fb5610a1700 1 librados: init done 2024-02-23T00:46:30.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.563+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c05f990 con 0x7fb55c102180 2024-02-23T00:46:30.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.564+0000 7fb543fff700 1 -- 172.21.15.87:0/4209288112 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb5540090d0 con 0x7fb55c102180 2024-02-23T00:46:30.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.564+0000 7fb543fff700 1 --2- 172.21.15.87:0/4209288112 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb544062c90 0x7fb544065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:30.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.564+0000 7fb543fff700 1 -- 172.21.15.87:0/4209288112 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(856..856 src has 400..856) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fb55407dbe0 con 0x7fb55c102180 2024-02-23T00:46:30.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.564+0000 7fb543fff700 10 client.28167.objecter ms_dispatch 0x7fb55c001550 osd_map(856..856 src has 400..856) v4 2024-02-23T00:46:30.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.564+0000 7fb543fff700 3 client.28167.objecter handle_osd_map got epochs [856,856] > 0 2024-02-23T00:46:30.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.564+0000 7fb543fff700 3 client.28167.objecter handle_osd_map decoding full epoch 856 2024-02-23T00:46:30.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.564+0000 7fb55ad9d700 1 --2- 172.21.15.87:0/4209288112 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb544062c90 0x7fb544065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:30.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.565+0000 7fb543fff700 20 client.28167.objecter dump_active .. 0 homeless 2024-02-23T00:46:30.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.565+0000 7fb543fff700 10 client.28167.objecter ms_handle_connect 0x7fb544062c90 2024-02-23T00:46:30.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.565+0000 7fb55ad9d700 1 --2- 172.21.15.87:0/4209288112 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb544062c90 0x7fb544065150 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 rx=0x7fb54c004dc0 tx=0x7fb54c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:30.569 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.568+0000 7fb543fff700 1 -- 172.21.15.87:0/4209288112 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb554051370 con 0x7fb55c102180 2024-02-23T00:46:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:29.324883+0000 osd.3 (osd.3) 351 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:46:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:29.328273+0000 osd.3 (osd.3) 352 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:46:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:29.633836+0000 osd.5 (osd.5) 397 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:46:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:29.634808+0000 osd.5 (osd.5) 398 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:46:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:29.724825+0000 osd.7 (osd.7) 456 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:46:30.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:29.726443+0000 osd.7 (osd.7) 457 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:46:30.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:29.850073+0000 mgr.y (mgr.24437) 767 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-02-23T00:46:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:29.324883+0000 osd.3 (osd.3) 351 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:46:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:29.328273+0000 osd.3 (osd.3) 352 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:46:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:29.633836+0000 osd.5 (osd.5) 397 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:46:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:29.634808+0000 osd.5 (osd.5) 398 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:46:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:29.724825+0000 osd.7 (osd.7) 456 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:46:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:29.726443+0000 osd.7 (osd.7) 457 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:46:30.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:29.850073+0000 mgr.y (mgr.24437) 767 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-02-23T00:46:30.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.731+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb55c05ee80 con 0x7fb55c102180 2024-02-23T00:46:30.733 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.732+0000 7fb543fff700 1 -- 172.21.15.87:0/4209288112 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb554050f60 con 0x7fb55c102180 2024-02-23T00:46:30.733 INFO:teuthology.orchestra.run.smithi087.stdout:133143986425 2024-02-23T00:46:30.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 10 librados: watch_flush enter 2024-02-23T00:46:30.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 10 librados: watch_flush exit 2024-02-23T00:46:30.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 20 client.28167.objecter shutdown clearing up homeless session... 2024-02-23T00:46:30.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 10 client.28167.objecter successfully canceled tick 2024-02-23T00:46:30.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb544062c90 msgr2=0x7fb544065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:30.735 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 --2- 172.21.15.87:0/4209288112 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb544062c90 0x7fb544065150 secure :-1 s=READY pgs=3023 cs=0 l=1 rev1=1 rx=0x7fb54c004dc0 tx=0x7fb54c00b040).stop 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c102180 msgr2=0x7fb55c0f75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 --2- 172.21.15.87:0/4209288112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c102180 0x7fb55c0f75b0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 rx=0x7fb554001e80 tx=0x7fb554004d10).stop 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 shutdown_connections 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 --2- 172.21.15.87:0/4209288112 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb544062c90 0x7fb544065150 unknown :-1 s=CLOSED pgs=3023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 --2- 172.21.15.87:0/4209288112 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb55c102180 0x7fb55c0f75b0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 --2- 172.21.15.87:0/4209288112 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb55c0f8c20 0x7fb55c0f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 --2- 172.21.15.87:0/4209288112 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb55c0f9580 0x7fb55c0f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.733+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 >> 172.21.15.87:0/4209288112 conn(0x7fb55c0f26f0 msgr2=0x7fb55c0f32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.734+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 shutdown_connections 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.734+0000 7fb5610a1700 1 -- 172.21.15.87:0/4209288112 wait complete. 2024-02-23T00:46:30.736 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:30.734+0000 7fb5610a1700 1 librados: shutdown 2024-02-23T00:46:30.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:29.324883+0000 osd.3 (osd.3) 351 : cluster [DBG] 6.6 scrub starts 2024-02-23T00:46:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:29.328273+0000 osd.3 (osd.3) 352 : cluster [DBG] 6.6 scrub ok 2024-02-23T00:46:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:29.633836+0000 osd.5 (osd.5) 397 : cluster [DBG] 5.16 scrub starts 2024-02-23T00:46:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:29.634808+0000 osd.5 (osd.5) 398 : cluster [DBG] 5.16 scrub ok 2024-02-23T00:46:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:29.724825+0000 osd.7 (osd.7) 456 : cluster [DBG] 5.9 scrub starts 2024-02-23T00:46:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:29.726443+0000 osd.7 (osd.7) 457 : cluster [DBG] 5.9 scrub ok 2024-02-23T00:46:30.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:29.850073+0000 mgr.y (mgr.24437) 767 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-02-23T00:46:31.512 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986420 got 133143986425 for osd.4 2024-02-23T00:46:31.513 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-23T00:46:31.788 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:31.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:30.310466+0000 osd.6 (osd.6) 406 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:46:31.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:30.313393+0000 osd.6 (osd.6) 407 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:46:31.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:30.343080+0000 osd.3 (osd.3) 353 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:46:31.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:30.345971+0000 osd.3 (osd.3) 354 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:30.484561+0000 osd.1 (osd.1) 468 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:30.488036+0000 osd.1 (osd.1) 469 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:30.732335+0000 mon.a (mon.0) 4051 : audit [DBG] from='client.? 172.21.15.87:0/4209288112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:30.310466+0000 osd.6 (osd.6) 406 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:30.313393+0000 osd.6 (osd.6) 407 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:30.343080+0000 osd.3 (osd.3) 353 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:30.345971+0000 osd.3 (osd.3) 354 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:30.484561+0000 osd.1 (osd.1) 468 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:30.488036+0000 osd.1 (osd.1) 469 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:46:31.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:31 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:30.732335+0000 mon.a (mon.0) 4051 : audit [DBG] from='client.? 172.21.15.87:0/4209288112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-23T00:46:31.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:46:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:46:31.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:30.310466+0000 osd.6 (osd.6) 406 : cluster [DBG] 2.17 scrub starts 2024-02-23T00:46:31.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:30.313393+0000 osd.6 (osd.6) 407 : cluster [DBG] 2.17 scrub ok 2024-02-23T00:46:31.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:30.343080+0000 osd.3 (osd.3) 353 : cluster [DBG] 5.13 scrub starts 2024-02-23T00:46:31.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:30.345971+0000 osd.3 (osd.3) 354 : cluster [DBG] 5.13 scrub ok 2024-02-23T00:46:31.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:30.484561+0000 osd.1 (osd.1) 468 : cluster [DBG] 4.2 scrub starts 2024-02-23T00:46:31.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:30.488036+0000 osd.1 (osd.1) 469 : cluster [DBG] 4.2 scrub ok 2024-02-23T00:46:31.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:30.732335+0000 mon.a (mon.0) 4051 : audit [DBG] from='client.? 172.21.15.87:0/4209288112' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-23T00:46:32.346 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:32] "GET /metrics HTTP/1.1" 200 186071 "" "Prometheus/2.33.4" 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:31.463106+0000 osd.1 (osd.1) 470 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:31.466210+0000 osd.1 (osd.1) 471 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:31.466898+0000 mgr.y (mgr.24437) 768 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:31.681253+0000 osd.7 (osd.7) 458 : cluster [DBG] 22.d deep-scrub starts 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:31.684454+0000 osd.7 (osd.7) 459 : cluster [DBG] 22.d deep-scrub ok 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:31.850837+0000 mgr.y (mgr.24437) 769 : cluster [DBG] pgmap v1359: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:32.305808+0000 mon.a (mon.0) 4052 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:31.463106+0000 osd.1 (osd.1) 470 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:31.466210+0000 osd.1 (osd.1) 471 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:31.466898+0000 mgr.y (mgr.24437) 768 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:32.737 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:31.681253+0000 osd.7 (osd.7) 458 : cluster [DBG] 22.d deep-scrub starts 2024-02-23T00:46:32.738 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:31.684454+0000 osd.7 (osd.7) 459 : cluster [DBG] 22.d deep-scrub ok 2024-02-23T00:46:32.738 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:31.850837+0000 mgr.y (mgr.24437) 769 : cluster [DBG] pgmap v1359: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:32.738 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:32 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:32.305808+0000 mon.a (mon.0) 4052 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-02-23T00:46:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:31.463106+0000 osd.1 (osd.1) 470 : cluster [DBG] 22.1d scrub starts 2024-02-23T00:46:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:31.466210+0000 osd.1 (osd.1) 471 : cluster [DBG] 22.1d scrub ok 2024-02-23T00:46:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:31.466898+0000 mgr.y (mgr.24437) 768 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:31.681253+0000 osd.7 (osd.7) 458 : cluster [DBG] 22.d deep-scrub starts 2024-02-23T00:46:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:31.684454+0000 osd.7 (osd.7) 459 : cluster [DBG] 22.d deep-scrub ok 2024-02-23T00:46:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:31.850837+0000 mgr.y (mgr.24437) 769 : cluster [DBG] pgmap v1359: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:32.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:32.305808+0000 mon.a (mon.0) 4052 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-02-23T00:46:33.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 -- 172.21.15.87:0/144870213 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880fd510 msgr2=0x7fb5880f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:33.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 --2- 172.21.15.87:0/144870213 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880fd510 0x7fb5880f6d40 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 rx=0x7fb57c0047b0 tx=0x7fb57c009cb0).stop 2024-02-23T00:46:33.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 -- 172.21.15.87:0/144870213 shutdown_connections 2024-02-23T00:46:33.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 --2- 172.21.15.87:0/144870213 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5880f7400 0x7fb588103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:33.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 --2- 172.21.15.87:0/144870213 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5880fcbb0 0x7fb5880fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:33.825 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 --2- 172.21.15.87:0/144870213 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880fd510 0x7fb5880f6d40 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 -- 172.21.15.87:0/144870213 >> 172.21.15.87:0/144870213 conn(0x7fb5880f2820 msgr2=0x7fb5880f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 -- 172.21.15.87:0/144870213 shutdown_connections 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 -- 172.21.15.87:0/144870213 wait complete. 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 librados: starting msgr at 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 librados: starting objecter 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 Processor -- start 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 -- start start 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 librados: setting wanted keys 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.821+0000 7fb58dc44700 1 librados: calling monclient init 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880f7400 0x7fb588110420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5880fcbb0 0x7fb588110910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5880fd510 0x7fb58810a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58810aab0 con 0x7fb5880fcbb0 2024-02-23T00:46:33.826 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58810abf0 con 0x7fb5880fd510 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58810adb0 con 0x7fb5880f7400 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb586ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5880fcbb0 0x7fb588110910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb5877fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880f7400 0x7fb588110420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb5877fe700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880f7400 0x7fb588110420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:32864/0 (socket says 172.21.15.87:32864) 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb5877fe700 1 -- 172.21.15.87:0/2560447945 learned_addr learned my addr 172.21.15.87:0/2560447945 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb586ffd700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5880fcbb0 0x7fb588110910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60558/0 (socket says 172.21.15.87:60558) 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb587fff700 1 --2- 172.21.15.87:0/2560447945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5880fd510 0x7fb58810a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb5877fe700 1 -- 172.21.15.87:0/2560447945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5880fd510 msgr2=0x7fb58810a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb5877fe700 1 --2- 172.21.15.87:0/2560447945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5880fd510 0x7fb58810a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb5877fe700 1 -- 172.21.15.87:0/2560447945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5880fcbb0 msgr2=0x7fb588110910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb5877fe700 1 --2- 172.21.15.87:0/2560447945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5880fcbb0 0x7fb588110910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb5877fe700 1 -- 172.21.15.87:0/2560447945 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb578005040 con 0x7fb5880f7400 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb586ffd700 1 --2- 172.21.15.87:0/2560447945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5880fcbb0 0x7fb588110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb5877fe700 1 --2- 172.21.15.87:0/2560447945 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880f7400 0x7fb588110420 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 rx=0x7fb578009400 tx=0x7fb578009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:33.827 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb584ff9700 10 client.?.objecter ms_handle_connect 0x7fb5880f7400 2024-02-23T00:46:33.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb584ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:33.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb584ff9700 1 -- 172.21.15.87:0/2560447945 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578007a40 con 0x7fb5880f7400 2024-02-23T00:46:33.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c005040 con 0x7fb5880f7400 2024-02-23T00:46:33.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb584ff9700 1 -- 172.21.15.87:0/2560447945 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb57800f040 con 0x7fb5880f7400 2024-02-23T00:46:33.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb584ff9700 1 -- 172.21.15.87:0/2560447945 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578014a70 con 0x7fb5880f7400 2024-02-23T00:46:33.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 10 client.38648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:33.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58810b3f0 con 0x7fb5880f7400 2024-02-23T00:46:33.828 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.822+0000 7fb58dc44700 1 librados: init done 2024-02-23T00:46:33.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.823+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58805fa60 con 0x7fb5880f7400 2024-02-23T00:46:33.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.824+0000 7fb584ff9700 1 -- 172.21.15.87:0/2560447945 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fb578010530 con 0x7fb5880f7400 2024-02-23T00:46:33.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.824+0000 7fb584ff9700 1 --2- 172.21.15.87:0/2560447945 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb570062c90 0x7fb570065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:33.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.824+0000 7fb584ff9700 1 -- 172.21.15.87:0/2560447945 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(857..857 src has 400..857) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fb57807cb70 con 0x7fb5880f7400 2024-02-23T00:46:33.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.824+0000 7fb584ff9700 10 client.38648.objecter ms_dispatch 0x7fb588001550 osd_map(857..857 src has 400..857) v4 2024-02-23T00:46:33.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.824+0000 7fb584ff9700 3 client.38648.objecter handle_osd_map got epochs [857,857] > 0 2024-02-23T00:46:33.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.824+0000 7fb584ff9700 3 client.38648.objecter handle_osd_map decoding full epoch 857 2024-02-23T00:46:33.829 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.824+0000 7fb586ffd700 1 --2- 172.21.15.87:0/2560447945 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb570062c90 0x7fb570065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:33.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.824+0000 7fb584ff9700 20 client.38648.objecter dump_active .. 0 homeless 2024-02-23T00:46:33.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.825+0000 7fb584ff9700 10 client.38648.objecter ms_handle_connect 0x7fb570062c90 2024-02-23T00:46:33.830 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.825+0000 7fb586ffd700 1 --2- 172.21.15.87:0/2560447945 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb570062c90 0x7fb570065150 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 rx=0x7fb58810bd60 tx=0x7fb57c009cb0).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:33.831 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.829+0000 7fb584ff9700 1 -- 172.21.15.87:0/2560447945 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fb578049f60 con 0x7fb5880f7400 2024-02-23T00:46:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:32.399660+0000 osd.3 (osd.3) 355 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:46:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:32.403630+0000 osd.3 (osd.3) 356 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:46:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:32.443028+0000 osd.1 (osd.1) 472 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:46:33.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:32.445912+0000 osd.1 (osd.1) 473 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:46:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:32.399660+0000 osd.3 (osd.3) 355 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:46:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:32.403630+0000 osd.3 (osd.3) 356 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:46:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:32.443028+0000 osd.1 (osd.1) 472 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:46:33.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:32.445912+0000 osd.1 (osd.1) 473 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:46:33.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:46:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:46:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:32.399660+0000 osd.3 (osd.3) 355 : cluster [DBG] 4.0 scrub starts 2024-02-23T00:46:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:32.403630+0000 osd.3 (osd.3) 356 : cluster [DBG] 4.0 scrub ok 2024-02-23T00:46:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:32.443028+0000 osd.1 (osd.1) 472 : cluster [DBG] 5.1d scrub starts 2024-02-23T00:46:33.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:32.445912+0000 osd.1 (osd.1) 473 : cluster [DBG] 5.1d scrub ok 2024-02-23T00:46:33.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.993+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb588107250 con 0x7fb5880f7400 2024-02-23T00:46:33.995 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.993+0000 7fb584ff9700 1 -- 172.21.15.87:0/2560447945 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb5780502c0 con 0x7fb5880f7400 2024-02-23T00:46:33.996 INFO:teuthology.orchestra.run.smithi087.stdout:163208757493 2024-02-23T00:46:33.997 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 10 librados: watch_flush enter 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 10 librados: watch_flush exit 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 20 client.38648.objecter shutdown clearing up homeless session... 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 10 client.38648.objecter successfully canceled tick 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb570062c90 msgr2=0x7fb570065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 --2- 172.21.15.87:0/2560447945 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb570062c90 0x7fb570065150 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 rx=0x7fb58810bd60 tx=0x7fb57c009cb0).stop 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880f7400 msgr2=0x7fb588110420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 --2- 172.21.15.87:0/2560447945 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880f7400 0x7fb588110420 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 rx=0x7fb578009400 tx=0x7fb578009cb0).stop 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 shutdown_connections 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 --2- 172.21.15.87:0/2560447945 >> v2:172.21.15.87:6800/1231597463 conn(0x7fb570062c90 0x7fb570065150 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 --2- 172.21.15.87:0/2560447945 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fb5880fd510 0x7fb58810a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 --2- 172.21.15.87:0/2560447945 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fb5880f7400 0x7fb588110420 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 --2- 172.21.15.87:0/2560447945 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fb5880fcbb0 0x7fb588110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:33.998 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 >> 172.21.15.87:0/2560447945 conn(0x7fb5880f2820 msgr2=0x7fb588102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:33.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 shutdown_connections 2024-02-23T00:46:33.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 -- 172.21.15.87:0/2560447945 wait complete. 2024-02-23T00:46:33.999 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:33.995+0000 7fb58dc44700 1 librados: shutdown 2024-02-23T00:46:34.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:33.139875+0000 osd.0 (osd.0) 125 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:33.140931+0000 osd.0 (osd.0) 126 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:33.608331+0000 osd.7 (osd.7) 460 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:33.611865+0000 osd.7 (osd.7) 461 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:33.826468+0000 osd.4 (osd.4) 415 : cluster [DBG] 4.d scrub starts 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:33.830092+0000 osd.4 (osd.4) 416 : cluster [DBG] 4.d scrub ok 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:33.851243+0000 mgr.y (mgr.24437) 770 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 693 B/s rd, 0 op/s 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:33.994635+0000 mon.c (mon.2) 525 : audit [DBG] from='client.? 172.21.15.87:0/2560447945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:33.139875+0000 osd.0 (osd.0) 125 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:33.140931+0000 osd.0 (osd.0) 126 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:46:34.758 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:33.608331+0000 osd.7 (osd.7) 460 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:46:34.759 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:33.611865+0000 osd.7 (osd.7) 461 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:46:34.759 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:33.826468+0000 osd.4 (osd.4) 415 : cluster [DBG] 4.d scrub starts 2024-02-23T00:46:34.759 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:33.830092+0000 osd.4 (osd.4) 416 : cluster [DBG] 4.d scrub ok 2024-02-23T00:46:34.759 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:33.851243+0000 mgr.y (mgr.24437) 770 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 693 B/s rd, 0 op/s 2024-02-23T00:46:34.759 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:34 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:33.994635+0000 mon.c (mon.2) 525 : audit [DBG] from='client.? 172.21.15.87:0/2560447945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-23T00:46:34.778 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757488 got 163208757493 for osd.5 2024-02-23T00:46:34.779 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-23T00:46:34.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:33.139875+0000 osd.0 (osd.0) 125 : cluster [DBG] 3.1 scrub starts 2024-02-23T00:46:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:33.140931+0000 osd.0 (osd.0) 126 : cluster [DBG] 3.1 scrub ok 2024-02-23T00:46:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:33.608331+0000 osd.7 (osd.7) 460 : cluster [DBG] 6.5 scrub starts 2024-02-23T00:46:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:33.611865+0000 osd.7 (osd.7) 461 : cluster [DBG] 6.5 scrub ok 2024-02-23T00:46:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:33.826468+0000 osd.4 (osd.4) 415 : cluster [DBG] 4.d scrub starts 2024-02-23T00:46:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:33.830092+0000 osd.4 (osd.4) 416 : cluster [DBG] 4.d scrub ok 2024-02-23T00:46:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:33.851243+0000 mgr.y (mgr.24437) 770 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 693 B/s rd, 0 op/s 2024-02-23T00:46:34.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:33.994635+0000 mon.c (mon.2) 525 : audit [DBG] from='client.? 172.21.15.87:0/2560447945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-23T00:46:35.046 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:35.608 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:34.488795+0000 osd.1 (osd.1) 474 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:46:35.608 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:34.491733+0000 osd.1 (osd.1) 475 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:46:35.609 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:34.488795+0000 osd.1 (osd.1) 474 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:46:35.609 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:35 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:34.491733+0000 osd.1 (osd.1) 475 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:46:35.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:34.488795+0000 osd.1 (osd.1) 474 : cluster [DBG] 3.4 scrub starts 2024-02-23T00:46:35.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:34.491733+0000 osd.1 (osd.1) 475 : cluster [DBG] 3.4 scrub ok 2024-02-23T00:46:36.698 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:35.159230+0000 osd.2 (osd.2) 341 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:35.160013+0000 osd.2 (osd.2) 342 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:35.346077+0000 osd.6 (osd.6) 408 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:35.349570+0000 osd.6 (osd.6) 409 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:35.450364+0000 osd.1 (osd.1) 476 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:35.453264+0000 osd.1 (osd.1) 477 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:35.852077+0000 mgr.y (mgr.24437) 771 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:35.159230+0000 osd.2 (osd.2) 341 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:35.160013+0000 osd.2 (osd.2) 342 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:35.346077+0000 osd.6 (osd.6) 408 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:35.349570+0000 osd.6 (osd.6) 409 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:35.450364+0000 osd.1 (osd.1) 476 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:46:36.699 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:35.453264+0000 osd.1 (osd.1) 477 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:46:36.700 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:36 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:35.852077+0000 mgr.y (mgr.24437) 771 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:36.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.929+0000 7f3a1dbac700 1 -- 172.21.15.87:0/4002359553 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fd510 msgr2=0x7f3a180f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:36.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.929+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/4002359553 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fd510 0x7f3a180f6d40 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 rx=0x7f3a080047b0 tx=0x7f3a08009cb0).stop 2024-02-23T00:46:36.933 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.929+0000 7f3a1dbac700 1 -- 172.21.15.87:0/4002359553 shutdown_connections 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.929+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/4002359553 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a180f7400 0x7f3a18103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.929+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/4002359553 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a180fcbb0 0x7f3a180fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.929+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/4002359553 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fd510 0x7f3a180f6d40 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.929+0000 7f3a1dbac700 1 -- 172.21.15.87:0/4002359553 >> 172.21.15.87:0/4002359553 conn(0x7f3a180f2820 msgr2=0x7f3a180f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.929+0000 7f3a1dbac700 1 -- 172.21.15.87:0/4002359553 shutdown_connections 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.929+0000 7f3a1dbac700 1 -- 172.21.15.87:0/4002359553 wait complete. 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 librados: starting msgr at 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 librados: starting objecter 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 Processor -- start 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 -- start start 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 librados: setting wanted keys 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 librados: calling monclient init 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a180f7400 0x7f3a1810d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:36.934 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fcbb0 0x7f3a1810d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a180fd510 0x7f3a18107260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a18107750 con 0x7f3a180fd510 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a181078c0 con 0x7f3a180f7400 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a1dbac700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a18107a80 con 0x7f3a180fcbb0 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a16ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fcbb0 0x7f3a1810d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a17fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a180fd510 0x7f3a18107260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a16ffd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fcbb0 0x7f3a1810d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3301/0 says I am v2:172.21.15.87:32884/0 (socket says 172.21.15.87:32884) 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a17fff700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a180fd510 0x7f3a18107260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60578/0 (socket says 172.21.15.87:60578) 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a16ffd700 1 -- 172.21.15.87:0/2734338579 learned_addr learned my addr 172.21.15.87:0/2734338579 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a177fe700 1 --2- 172.21.15.87:0/2734338579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a180f7400 0x7f3a1810d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a16ffd700 1 -- 172.21.15.87:0/2734338579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a180f7400 msgr2=0x7f3a1810d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a16ffd700 1 --2- 172.21.15.87:0/2734338579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a180f7400 0x7f3a1810d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a16ffd700 1 -- 172.21.15.87:0/2734338579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a180fd510 msgr2=0x7f3a18107260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:36.935 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a16ffd700 1 --2- 172.21.15.87:0/2734338579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a180fd510 0x7f3a18107260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.930+0000 7f3a16ffd700 1 -- 172.21.15.87:0/2734338579 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a100097c0 con 0x7f3a180fcbb0 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a17fff700 1 --2- 172.21.15.87:0/2734338579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a180fd510 0x7f3a18107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a16ffd700 1 --2- 172.21.15.87:0/2734338579 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fcbb0 0x7f3a1810d5b0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f3a08009b10 tx=0x7f3a08009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a14ff9700 10 client.?.objecter ms_handle_connect 0x7f3a180fcbb0 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a14ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a14ff9700 1 -- 172.21.15.87:0/2734338579 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a08018070 con 0x7f3a180fcbb0 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a08005040 con 0x7f3a180fcbb0 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a14ff9700 1 -- 172.21.15.87:0/2734338579 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a08013da0 con 0x7f3a180fcbb0 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a14ff9700 1 -- 172.21.15.87:0/2734338579 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0801dcc0 con 0x7f3a180fcbb0 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a1dbac700 10 client.38654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a18107f20 con 0x7f3a180fcbb0 2024-02-23T00:46:36.936 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.931+0000 7f3a1dbac700 1 librados: init done 2024-02-23T00:46:36.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.932+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a18108110 con 0x7f3a180fcbb0 2024-02-23T00:46:36.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.932+0000 7f3a14ff9700 1 -- 172.21.15.87:0/2734338579 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3a0801d420 con 0x7f3a180fcbb0 2024-02-23T00:46:36.937 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.933+0000 7f3a14ff9700 1 --2- 172.21.15.87:0/2734338579 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3a00062c90 0x7f3a00065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:36.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.933+0000 7f3a14ff9700 1 -- 172.21.15.87:0/2734338579 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(857..857 src has 400..857) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f3a0807eb10 con 0x7f3a180fcbb0 2024-02-23T00:46:36.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.933+0000 7f3a177fe700 1 --2- 172.21.15.87:0/2734338579 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3a00062c90 0x7f3a00065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:36.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.933+0000 7f3a14ff9700 10 client.38654.objecter ms_dispatch 0x7f3a18001550 osd_map(857..857 src has 400..857) v4 2024-02-23T00:46:36.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.933+0000 7f3a14ff9700 3 client.38654.objecter handle_osd_map got epochs [857,857] > 0 2024-02-23T00:46:36.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.933+0000 7f3a14ff9700 3 client.38654.objecter handle_osd_map decoding full epoch 857 2024-02-23T00:46:36.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.933+0000 7f3a14ff9700 20 client.38654.objecter dump_active .. 0 homeless 2024-02-23T00:46:36.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.933+0000 7f3a14ff9700 10 client.38654.objecter ms_handle_connect 0x7f3a00062c90 2024-02-23T00:46:36.938 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.933+0000 7f3a177fe700 1 --2- 172.21.15.87:0/2734338579 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3a00062c90 0x7f3a00065150 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 rx=0x7f3a0c004d50 tx=0x7f3a0c00b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:36.939 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:36.937+0000 7f3a14ff9700 1 -- 172.21.15.87:0/2734338579 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3a080521c0 con 0x7f3a180fcbb0 2024-02-23T00:46:36.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:35.159230+0000 osd.2 (osd.2) 341 : cluster [DBG] 2.10 scrub starts 2024-02-23T00:46:36.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:35.160013+0000 osd.2 (osd.2) 342 : cluster [DBG] 2.10 scrub ok 2024-02-23T00:46:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:35.346077+0000 osd.6 (osd.6) 408 : cluster [DBG] 4.1f scrub starts 2024-02-23T00:46:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:35.349570+0000 osd.6 (osd.6) 409 : cluster [DBG] 4.1f scrub ok 2024-02-23T00:46:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:35.450364+0000 osd.1 (osd.1) 476 : cluster [DBG] 2.6 scrub starts 2024-02-23T00:46:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:35.453264+0000 osd.1 (osd.1) 477 : cluster [DBG] 2.6 scrub ok 2024-02-23T00:46:36.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:35.852077+0000 mgr.y (mgr.24437) 771 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:37.102 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.100+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3a18106120 con 0x7f3a180fcbb0 2024-02-23T00:46:37.103 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.100+0000 7f3a14ff9700 1 -- 172.21.15.87:0/2734338579 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3a08051db0 con 0x7f3a180fcbb0 2024-02-23T00:46:37.103 INFO:teuthology.orchestra.run.smithi087.stdout:188978561265 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 10 librados: watch_flush enter 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 10 librados: watch_flush exit 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 20 client.38654.objecter shutdown clearing up homeless session... 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 10 client.38654.objecter successfully canceled tick 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3a00062c90 msgr2=0x7f3a00065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/2734338579 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3a00062c90 0x7f3a00065150 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 rx=0x7f3a0c004d50 tx=0x7f3a0c00b040).stop 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fcbb0 msgr2=0x7f3a1810d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/2734338579 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fcbb0 0x7f3a1810d5b0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f3a08009b10 tx=0x7f3a08009cb0).stop 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 shutdown_connections 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/2734338579 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3a00062c90 0x7f3a00065150 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/2734338579 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3a180fd510 0x7f3a18107260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:37.105 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/2734338579 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3a180f7400 0x7f3a1810d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:37.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 --2- 172.21.15.87:0/2734338579 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3a180fcbb0 0x7f3a1810d5b0 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:37.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 >> 172.21.15.87:0/2734338579 conn(0x7f3a180f2820 msgr2=0x7f3a180fa940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:37.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 shutdown_connections 2024-02-23T00:46:37.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 -- 172.21.15.87:0/2734338579 wait complete. 2024-02-23T00:46:37.106 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:37.102+0000 7f3a1dbac700 1 librados: shutdown 2024-02-23T00:46:37.817 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:37.101594+0000 mon.c (mon.2) 526 : audit [DBG] from='client.? 172.21.15.87:0/2734338579' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-23T00:46:37.818 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:37 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:37.101594+0000 mon.c (mon.2) 526 : audit [DBG] from='client.? 172.21.15.87:0/2734338579' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-23T00:46:37.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561260 got 188978561265 for osd.6 2024-02-23T00:46:37.837 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-23T00:46:37.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:37.101594+0000 mon.c (mon.2) 526 : audit [DBG] from='client.? 172.21.15.87:0/2734338579' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-23T00:46:38.105 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:38.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:37.323240+0000 osd.3 (osd.3) 357 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:46:38.715 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:37.326271+0000 osd.3 (osd.3) 358 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:46:38.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:37.506210+0000 osd.1 (osd.1) 478 : cluster [DBG] 4.f scrub starts 2024-02-23T00:46:38.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:37.509716+0000 osd.1 (osd.1) 479 : cluster [DBG] 4.f scrub ok 2024-02-23T00:46:38.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:37.533931+0000 osd.7 (osd.7) 462 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:46:38.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:37.535337+0000 osd.7 (osd.7) 463 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:46:38.716 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:37.852489+0000 mgr.y (mgr.24437) 772 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:46:38.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:37.323240+0000 osd.3 (osd.3) 357 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:46:38.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:37.326271+0000 osd.3 (osd.3) 358 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:46:38.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:37.506210+0000 osd.1 (osd.1) 478 : cluster [DBG] 4.f scrub starts 2024-02-23T00:46:38.716 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:37.509716+0000 osd.1 (osd.1) 479 : cluster [DBG] 4.f scrub ok 2024-02-23T00:46:38.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:37.533931+0000 osd.7 (osd.7) 462 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:46:38.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:37.535337+0000 osd.7 (osd.7) 463 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:46:38.717 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:38 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:37.852489+0000 mgr.y (mgr.24437) 772 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:46:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:37.323240+0000 osd.3 (osd.3) 357 : cluster [DBG] 2.19 scrub starts 2024-02-23T00:46:38.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:37.326271+0000 osd.3 (osd.3) 358 : cluster [DBG] 2.19 scrub ok 2024-02-23T00:46:38.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:37.506210+0000 osd.1 (osd.1) 478 : cluster [DBG] 4.f scrub starts 2024-02-23T00:46:38.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:37.509716+0000 osd.1 (osd.1) 479 : cluster [DBG] 4.f scrub ok 2024-02-23T00:46:38.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:37.533931+0000 osd.7 (osd.7) 462 : cluster [DBG] 6.3 scrub starts 2024-02-23T00:46:38.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:37.535337+0000 osd.7 (osd.7) 463 : cluster [DBG] 6.3 scrub ok 2024-02-23T00:46:38.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:37.852489+0000 mgr.y (mgr.24437) 772 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-23T00:46:39.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:38.205758+0000 osd.2 (osd.2) 343 : cluster [DBG] 2.c scrub starts 2024-02-23T00:46:39.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:38.206909+0000 osd.2 (osd.2) 344 : cluster [DBG] 2.c scrub ok 2024-02-23T00:46:39.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:38.370332+0000 osd.6 (osd.6) 410 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:46:39.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:38.373265+0000 osd.6 (osd.6) 411 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:46:39.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:38.481600+0000 osd.1 (osd.1) 480 : cluster [DBG] 5.c scrub starts 2024-02-23T00:46:39.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:38.484518+0000 osd.1 (osd.1) 481 : cluster [DBG] 5.c scrub ok 2024-02-23T00:46:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:38.205758+0000 osd.2 (osd.2) 343 : cluster [DBG] 2.c scrub starts 2024-02-23T00:46:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:38.206909+0000 osd.2 (osd.2) 344 : cluster [DBG] 2.c scrub ok 2024-02-23T00:46:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:38.370332+0000 osd.6 (osd.6) 410 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:46:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:38.373265+0000 osd.6 (osd.6) 411 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:46:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:38.481600+0000 osd.1 (osd.1) 480 : cluster [DBG] 5.c scrub starts 2024-02-23T00:46:39.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:39 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:38.484518+0000 osd.1 (osd.1) 481 : cluster [DBG] 5.c scrub ok 2024-02-23T00:46:39.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:38.205758+0000 osd.2 (osd.2) 343 : cluster [DBG] 2.c scrub starts 2024-02-23T00:46:39.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:38.206909+0000 osd.2 (osd.2) 344 : cluster [DBG] 2.c scrub ok 2024-02-23T00:46:39.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:38.370332+0000 osd.6 (osd.6) 410 : cluster [DBG] 2.11 scrub starts 2024-02-23T00:46:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:38.373265+0000 osd.6 (osd.6) 411 : cluster [DBG] 2.11 scrub ok 2024-02-23T00:46:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:38.481600+0000 osd.1 (osd.1) 480 : cluster [DBG] 5.c scrub starts 2024-02-23T00:46:39.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:38.484518+0000 osd.1 (osd.1) 481 : cluster [DBG] 5.c scrub ok 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.103+0000 7fa15982b700 1 -- 172.21.15.87:0/437274439 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fdbb0 msgr2=0x7fa1540fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.103+0000 7fa15982b700 1 --2- 172.21.15.87:0/437274439 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fdbb0 0x7fa1540fdf90 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 rx=0x7fa148004870 tx=0x7fa148009cb0).stop 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.103+0000 7fa15982b700 1 -- 172.21.15.87:0/437274439 shutdown_connections 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.103+0000 7fa15982b700 1 --2- 172.21.15.87:0/437274439 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1540f8420 0x7fa154104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.103+0000 7fa15982b700 1 --2- 172.21.15.87:0/437274439 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fdbb0 0x7fa1540fdf90 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.103+0000 7fa15982b700 1 --2- 172.21.15.87:0/437274439 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1540fe510 0x7fa1540f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.103+0000 7fa15982b700 1 -- 172.21.15.87:0/437274439 >> 172.21.15.87:0/437274439 conn(0x7fa1540f3800 msgr2=0x7fa1540f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.103+0000 7fa15982b700 1 -- 172.21.15.87:0/437274439 shutdown_connections 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.103+0000 7fa15982b700 1 -- 172.21.15.87:0/437274439 wait complete. 2024-02-23T00:46:40.108 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 librados: starting msgr at 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 librados: starting objecter 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 Processor -- start 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 -- start start 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 librados: setting wanted keys 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 librados: calling monclient init 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1540f8420 0x7fa15410e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1540fdbb0 0x7fa15410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fe510 0x7fa154108220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa154108710 con 0x7fa1540fe510 2024-02-23T00:46:40.109 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa154108880 con 0x7fa1540fdbb0 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa15982b700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa154108a40 con 0x7fa1540f8420 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa1537fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fe510 0x7fa154108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa1537fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fe510 0x7fa154108220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:60586/0 (socket says 172.21.15.87:60586) 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa1537fe700 1 -- 172.21.15.87:0/696254723 learned_addr learned my addr 172.21.15.87:0/696254723 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa152ffd700 1 --2- 172.21.15.87:0/696254723 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1540f8420 0x7fa15410e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.104+0000 7fa1527fc700 1 --2- 172.21.15.87:0/696254723 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1540fdbb0 0x7fa15410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa1537fe700 1 -- 172.21.15.87:0/696254723 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1540f8420 msgr2=0x7fa15410e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa1537fe700 1 --2- 172.21.15.87:0/696254723 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1540f8420 0x7fa15410e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa1537fe700 1 -- 172.21.15.87:0/696254723 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1540fdbb0 msgr2=0x7fa15410e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa1537fe700 1 --2- 172.21.15.87:0/696254723 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1540fdbb0 0x7fa15410e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa1537fe700 1 -- 172.21.15.87:0/696254723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa144005040 con 0x7fa1540fe510 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa1537fe700 1 --2- 172.21.15.87:0/696254723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fe510 0x7fa154108220 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7fa14c00c870 tx=0x7fa14c007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa13bfff700 10 client.?.objecter ms_handle_connect 0x7fa1540fe510 2024-02-23T00:46:40.110 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa13bfff700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:40.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa13bfff700 1 -- 172.21.15.87:0/696254723 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c010510 con 0x7fa1540fe510 2024-02-23T00:46:40.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa13bfff700 1 -- 172.21.15.87:0/696254723 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa14c010b40 con 0x7fa1540fe510 2024-02-23T00:46:40.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148005040 con 0x7fa1540fe510 2024-02-23T00:46:40.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa13bfff700 1 -- 172.21.15.87:0/696254723 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c012ae0 con 0x7fa1540fe510 2024-02-23T00:46:40.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa15982b700 10 client.28185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:40.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa154109030 con 0x7fa1540fe510 2024-02-23T00:46:40.111 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.105+0000 7fa15982b700 1 librados: init done 2024-02-23T00:46:40.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.106+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1540606d0 con 0x7fa1540fe510 2024-02-23T00:46:40.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.106+0000 7fa13bfff700 1 -- 172.21.15.87:0/696254723 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fa14c010670 con 0x7fa1540fe510 2024-02-23T00:46:40.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.107+0000 7fa13bfff700 1 --2- 172.21.15.87:0/696254723 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa13c062c90 0x7fa13c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:40.112 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.107+0000 7fa152ffd700 1 --2- 172.21.15.87:0/696254723 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa13c062c90 0x7fa13c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:40.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.107+0000 7fa13bfff700 1 -- 172.21.15.87:0/696254723 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(857..857 src has 400..857) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fa14c07d060 con 0x7fa1540fe510 2024-02-23T00:46:40.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.107+0000 7fa13bfff700 10 client.28185.objecter ms_dispatch 0x7fa154001550 osd_map(857..857 src has 400..857) v4 2024-02-23T00:46:40.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.107+0000 7fa13bfff700 3 client.28185.objecter handle_osd_map got epochs [857,857] > 0 2024-02-23T00:46:40.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.107+0000 7fa13bfff700 3 client.28185.objecter handle_osd_map decoding full epoch 857 2024-02-23T00:46:40.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.107+0000 7fa13bfff700 20 client.28185.objecter dump_active .. 0 homeless 2024-02-23T00:46:40.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.107+0000 7fa13bfff700 10 client.28185.objecter ms_handle_connect 0x7fa13c062c90 2024-02-23T00:46:40.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.107+0000 7fa152ffd700 1 --2- 172.21.15.87:0/696254723 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa13c062c90 0x7fa13c065150 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 rx=0x7fa1480043d0 tx=0x7fa14800b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:40.113 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.111+0000 7fa13bfff700 1 -- 172.21.15.87:0/696254723 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fa14c0507b0 con 0x7fa1540fe510 2024-02-23T00:46:40.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.274+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa154109700 con 0x7fa1540fe510 2024-02-23T00:46:40.277 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.275+0000 7fa13bfff700 1 -- 172.21.15.87:0/696254723 <== mon.0 v2:172.21.15.87:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa14c0503a0 con 0x7fa1540fe510 2024-02-23T00:46:40.278 INFO:teuthology.orchestra.run.smithi087.stdout:219043332332 2024-02-23T00:46:40.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.276+0000 7fa15982b700 10 librados: watch_flush enter 2024-02-23T00:46:40.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.276+0000 7fa15982b700 10 librados: watch_flush exit 2024-02-23T00:46:40.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.276+0000 7fa15982b700 20 client.28185.objecter shutdown clearing up homeless session... 2024-02-23T00:46:40.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.276+0000 7fa15982b700 10 client.28185.objecter successfully canceled tick 2024-02-23T00:46:40.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.276+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa13c062c90 msgr2=0x7fa13c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:40.279 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.276+0000 7fa15982b700 1 --2- 172.21.15.87:0/696254723 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa13c062c90 0x7fa13c065150 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 rx=0x7fa1480043d0 tx=0x7fa14800b040).stop 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.276+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fe510 msgr2=0x7fa154108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.276+0000 7fa15982b700 1 --2- 172.21.15.87:0/696254723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fe510 0x7fa154108220 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 rx=0x7fa14c00c870 tx=0x7fa14c007dd0).stop 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.277+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 shutdown_connections 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.277+0000 7fa15982b700 1 --2- 172.21.15.87:0/696254723 >> v2:172.21.15.87:6800/1231597463 conn(0x7fa13c062c90 0x7fa13c065150 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.277+0000 7fa15982b700 1 --2- 172.21.15.87:0/696254723 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fa1540fe510 0x7fa154108220 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.277+0000 7fa15982b700 1 --2- 172.21.15.87:0/696254723 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fa1540f8420 0x7fa15410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.277+0000 7fa15982b700 1 --2- 172.21.15.87:0/696254723 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fa1540fdbb0 0x7fa15410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.277+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 >> 172.21.15.87:0/696254723 conn(0x7fa1540f3800 msgr2=0x7fa1540f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.277+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 shutdown_connections 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.277+0000 7fa15982b700 1 -- 172.21.15.87:0/696254723 wait complete. 2024-02-23T00:46:40.280 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:40.277+0000 7fa15982b700 1 librados: shutdown 2024-02-23T00:46:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:39.334540+0000 osd.3 (osd.3) 359 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:46:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:39.338199+0000 osd.3 (osd.3) 360 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:46:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:39.495796+0000 osd.1 (osd.1) 482 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:46:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:39.499332+0000 osd.1 (osd.1) 483 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:46:40.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:39.852925+0000 mgr.y (mgr.24437) 773 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:40.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:40.276441+0000 mon.a (mon.0) 4053 : audit [DBG] from='client.? 172.21.15.87:0/696254723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-23T00:46:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:39.334540+0000 osd.3 (osd.3) 359 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:46:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:39.338199+0000 osd.3 (osd.3) 360 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:46:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:39.495796+0000 osd.1 (osd.1) 482 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:46:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:39.499332+0000 osd.1 (osd.1) 483 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:46:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:39.852925+0000 mgr.y (mgr.24437) 773 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:40.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:40.276441+0000 mon.a (mon.0) 4053 : audit [DBG] from='client.? 172.21.15.87:0/696254723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-23T00:46:40.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:39.334540+0000 osd.3 (osd.3) 359 : cluster [DBG] 4.10 scrub starts 2024-02-23T00:46:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:39.338199+0000 osd.3 (osd.3) 360 : cluster [DBG] 4.10 scrub ok 2024-02-23T00:46:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:39.495796+0000 osd.1 (osd.1) 482 : cluster [DBG] 22.13 scrub starts 2024-02-23T00:46:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:39.499332+0000 osd.1 (osd.1) 483 : cluster [DBG] 22.13 scrub ok 2024-02-23T00:46:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:39.852925+0000 mgr.y (mgr.24437) 773 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:40.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:40.276441+0000 mon.a (mon.0) 4053 : audit [DBG] from='client.? 172.21.15.87:0/696254723' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-23T00:46:41.053 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332327 got 219043332332 for osd.7 2024-02-23T00:46:41.053 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-23T00:46:41.053 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:46:41.325 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:41.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:40.302014+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.b scrub starts 2024-02-23T00:46:41.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:40.305079+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.b scrub ok 2024-02-23T00:46:41.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:40.407561+0000 osd.6 (osd.6) 412 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:46:41.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:40.410470+0000 osd.6 (osd.6) 413 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:46:41.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:40.302014+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.b scrub starts 2024-02-23T00:46:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:40.305079+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.b scrub ok 2024-02-23T00:46:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:40.407561+0000 osd.6 (osd.6) 412 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:46:41.834 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:41 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:40.410470+0000 osd.6 (osd.6) 413 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:46:41.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:46:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:46:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:40.302014+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.b scrub starts 2024-02-23T00:46:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:40.305079+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.b scrub ok 2024-02-23T00:46:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:40.407561+0000 osd.6 (osd.6) 412 : cluster [DBG] 2.1a scrub starts 2024-02-23T00:46:41.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:40.410470+0000 osd.6 (osd.6) 413 : cluster [DBG] 2.1a scrub ok 2024-02-23T00:46:42.527 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:42] "GET /metrics HTTP/1.1" 200 186064 "" "Prometheus/2.33.4" 2024-02-23T00:46:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:41.089715+0000 osd.0 (osd.0) 127 : cluster [DBG] 6.9 deep-scrub starts 2024-02-23T00:46:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:41.091255+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.9 deep-scrub ok 2024-02-23T00:46:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:41.476457+0000 mgr.y (mgr.24437) 774 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:42.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:41.853701+0000 mgr.y (mgr.24437) 775 : cluster [DBG] pgmap v1365: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:41.089715+0000 osd.0 (osd.0) 127 : cluster [DBG] 6.9 deep-scrub starts 2024-02-23T00:46:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:41.091255+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.9 deep-scrub ok 2024-02-23T00:46:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:41.476457+0000 mgr.y (mgr.24437) 774 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:42.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:42 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:41.853701+0000 mgr.y (mgr.24437) 775 : cluster [DBG] pgmap v1365: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:41.089715+0000 osd.0 (osd.0) 127 : cluster [DBG] 6.9 deep-scrub starts 2024-02-23T00:46:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:41.091255+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.9 deep-scrub ok 2024-02-23T00:46:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:41.476457+0000 mgr.y (mgr.24437) 774 : audit [DBG] from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-23T00:46:42.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:41.853701+0000 mgr.y (mgr.24437) 775 : cluster [DBG] pgmap v1365: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-23T00:46:43.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 -- 172.21.15.87:0/4097147457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc102270 msgr2=0x7fe0fc104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:43.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 --2- 172.21.15.87:0/4097147457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc102270 0x7fe0fc104660 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 rx=0x7fe0f4009be0 tx=0x7fe0f4004d10).stop 2024-02-23T00:46:43.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 -- 172.21.15.87:0/4097147457 shutdown_connections 2024-02-23T00:46:43.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 --2- 172.21.15.87:0/4097147457 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc102270 0x7fe0fc104660 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:43.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 --2- 172.21.15.87:0/4097147457 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0fc0f8d10 0x7fe0fc0f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:43.271 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 --2- 172.21.15.87:0/4097147457 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0fc0f9670 0x7fe0fc101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 -- 172.21.15.87:0/4097147457 >> 172.21.15.87:0/4097147457 conn(0x7fe0fc0f2820 msgr2=0x7fe0fc0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 -- 172.21.15.87:0/4097147457 shutdown_connections 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 -- 172.21.15.87:0/4097147457 wait complete. 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 librados: starting msgr at 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.256+0000 7fe100a6e700 1 librados: starting objecter 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 Processor -- start 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 -- start start 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 librados: setting wanted keys 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 librados: calling monclient init 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc0f8d10 0x7fe0fc10df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0fc0f9670 0x7fe0fc10e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0fc102270 0x7fe0fc1080b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc1085a0 con 0x7fe0fc0f8d10 2024-02-23T00:46:43.272 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc108710 con 0x7fe0fc102270 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc1088d0 con 0x7fe0fc0f9670 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0fa59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc0f8d10 0x7fe0fc10df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0f9d9b700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0fc0f9670 0x7fe0fc10e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0fa59c700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc0f8d10 0x7fe0fc10df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39180/0 (socket says 172.21.15.87:39180) 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0fa59c700 1 -- 172.21.15.87:0/2801660336 learned_addr learned my addr 172.21.15.87:0/2801660336 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0fad9d700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0fc102270 0x7fe0fc1080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0f9d9b700 1 -- 172.21.15.87:0/2801660336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0fc102270 msgr2=0x7fe0fc1080b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0f9d9b700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0fc102270 0x7fe0fc1080b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0f9d9b700 1 -- 172.21.15.87:0/2801660336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc0f8d10 msgr2=0x7fe0fc10df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0f9d9b700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc0f8d10 0x7fe0fc10df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0f9d9b700 1 -- 172.21.15.87:0/2801660336 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f40097c0 con 0x7fe0fc0f9670 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0fa59c700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc0f8d10 0x7fe0fc10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0fad9d700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0fc102270 0x7fe0fc1080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0f9d9b700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0fc0f9670 0x7fe0fc10e400 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7fe0ec00e8f0 tx=0x7fe0ec00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:43.273 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0eb7fe700 10 client.?.objecter ms_handle_connect 0x7fe0fc0f9670 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0eb7fe700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0eb7fe700 1 -- 172.21.15.87:0/2801660336 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec004620 con 0x7fe0fc0f9670 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0eb7fe700 1 -- 172.21.15.87:0/2801660336 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe0ec00f040 con 0x7fe0fc0f9670 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0fc108bb0 con 0x7fe0fc0f9670 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.257+0000 7fe0eb7fe700 1 -- 172.21.15.87:0/2801660336 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec0146f0 con 0x7fe0fc0f9670 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.258+0000 7fe100a6e700 10 client.38666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.258+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0fc108f30 con 0x7fe0fc0f9670 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.258+0000 7fe100a6e700 1 librados: init done 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.259+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0fc05fa60 con 0x7fe0fc0f9670 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.259+0000 7fe0eb7fe700 1 -- 172.21.15.87:0/2801660336 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7fe0ec00c3d0 con 0x7fe0fc0f9670 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.259+0000 7fe0eb7fe700 1 --2- 172.21.15.87:0/2801660336 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe0e4062c90 0x7fe0e4065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.259+0000 7fe0eb7fe700 1 -- 172.21.15.87:0/2801660336 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(857..857 src has 400..857) v4 ==== 7486+0+0 (secure 0 0 0) 0x7fe0ec07cae0 con 0x7fe0fc0f9670 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.259+0000 7fe0eb7fe700 10 client.38666.objecter ms_dispatch 0x7fe0fc001550 osd_map(857..857 src has 400..857) v4 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.259+0000 7fe0eb7fe700 3 client.38666.objecter handle_osd_map got epochs [857,857] > 0 2024-02-23T00:46:43.274 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.259+0000 7fe0eb7fe700 3 client.38666.objecter handle_osd_map decoding full epoch 857 2024-02-23T00:46:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.259+0000 7fe0fa59c700 1 --2- 172.21.15.87:0/2801660336 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe0e4062c90 0x7fe0e4065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.259+0000 7fe0eb7fe700 20 client.38666.objecter dump_active .. 0 homeless 2024-02-23T00:46:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.260+0000 7fe0fa59c700 1 --2- 172.21.15.87:0/2801660336 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe0e4062c90 0x7fe0e4065150 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 rx=0x7fe0f000f3d0 tx=0x7fe0f0009040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.260+0000 7fe0eb7fe700 10 client.38666.objecter ms_handle_connect 0x7fe0e4062c90 2024-02-23T00:46:43.275 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.263+0000 7fe0eb7fe700 1 -- 172.21.15.87:0/2801660336 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7fe0ec050280 con 0x7fe0fc0f9670 2024-02-23T00:46:43.427 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.425+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe0fc1072c0 con 0x7fe0e4062c90 2024-02-23T00:46:43.439 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.437+0000 7fe0eb7fe700 1 -- 172.21.15.87:0/2801660336 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+432221 (secure 0 0 0) 0x7fe0fc1072c0 con 0x7fe0e4062c90 2024-02-23T00:46:43.445 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 10 librados: watch_flush enter 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 10 librados: watch_flush exit 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 20 client.38666.objecter shutdown clearing up homeless session... 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 10 client.38666.objecter successfully canceled tick 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe0e4062c90 msgr2=0x7fe0e4065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 --2- 172.21.15.87:0/2801660336 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe0e4062c90 0x7fe0e4065150 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 rx=0x7fe0f000f3d0 tx=0x7fe0f0009040).stop 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0fc0f9670 msgr2=0x7fe0fc10e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0fc0f9670 0x7fe0fc10e400 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 rx=0x7fe0ec00e8f0 tx=0x7fe0ec00b040).stop 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 shutdown_connections 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 --2- 172.21.15.87:0/2801660336 >> v2:172.21.15.87:6800/1231597463 conn(0x7fe0e4062c90 0x7fe0e4065150 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7fe0fc102270 0x7fe0fc1080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:43.449 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7fe0fc0f8d10 0x7fe0fc10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:43.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 --2- 172.21.15.87:0/2801660336 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7fe0fc0f9670 0x7fe0fc10e400 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:43.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 >> 172.21.15.87:0/2801660336 conn(0x7fe0fc0f2820 msgr2=0x7fe0fc0f3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:43.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 shutdown_connections 2024-02-23T00:46:43.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 -- 172.21.15.87:0/2801660336 wait complete. 2024-02-23T00:46:43.450 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:43.440+0000 7fe100a6e700 1 librados: shutdown 2024-02-23T00:46:43.450 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:46:43.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:42.754555+0000 osd.4 (osd.4) 417 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:46:43.832 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:42.756515+0000 osd.4 (osd.4) 418 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:46:43.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:43.426842+0000 mgr.y (mgr.24437) 776 : audit [DBG] from='client.38666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:46:43.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:42.754555+0000 osd.4 (osd.4) 417 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:46:43.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:42.756515+0000 osd.4 (osd.4) 418 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:46:43.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:43.426842+0000 mgr.y (mgr.24437) 776 : audit [DBG] from='client.38666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:46:43.942 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:46:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:46:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:42.754555+0000 osd.4 (osd.4) 417 : cluster [DBG] 22.3 scrub starts 2024-02-23T00:46:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:42.756515+0000 osd.4 (osd.4) 418 : cluster [DBG] 22.3 scrub ok 2024-02-23T00:46:43.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:43.426842+0000 mgr.y (mgr.24437) 776 : audit [DBG] from='client.38666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:46:44.176 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1365,"stamp":"2024-02-23T00:46:41.853160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":1417,"num_read_kb":1311,"num_write":349,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":6055,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":864,"ondisk_log_size":864,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2494784,"kb_used_data":62144,"kb_used_omap":85,"kb_used_meta":2432490,"kb_avail":747466432,"statfs":{"total":767960285184,"available":765405626368,"internally_reserved":0,"allocated":63635456,"data_stored":34259151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":2490869888},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002871"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":447,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616376+0000","last_change":"2024-02-23T00:45:54.740202+0000","last_active":"2024-02-23T00:46:28.616376+0000","last_peered":"2024-02-23T00:46:28.616376+0000","last_clean":"2024-02-23T00:46:28.616376+0000","last_became_active":"2024-02-23T00:42:41.400209+0000","last_became_peered":"2024-02-23T00:42:41.400209+0000","last_unstale":"2024-02-23T00:46:28.616376+0000","last_undegraded":"2024-02-23T00:46:28.616376+0000","last_fullsized":"2024-02-23T00:46:28.616376+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:54.740125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:54.740125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"571'11","reported_seq":1120,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.552234+0000","last_change":"2024-02-23T00:46:29.726533+0000","last_active":"2024-02-23T00:46:35.552234+0000","last_peered":"2024-02-23T00:46:35.552234+0000","last_clean":"2024-02-23T00:46:35.552234+0000","last_became_active":"2024-02-23T00:39:49.195036+0000","last_became_peered":"2024-02-23T00:39:49.195036+0000","last_unstale":"2024-02-23T00:46:35.552234+0000","last_undegraded":"2024-02-23T00:46:35.552234+0000","last_fullsized":"2024-02-23T00:46:35.552234+0000","mapping_epoch":58,"log_start":"379'9","ondisk_log_start":"379'9","created":375,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'11","last_scrub_stamp":"2024-02-23T00:46:29.726449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:29.726449+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":957,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.613713+0000","last_change":"2024-02-23T00:45:50.076790+0000","last_active":"2024-02-23T00:46:28.613713+0000","last_peered":"2024-02-23T00:46:28.613713+0000","last_clean":"2024-02-23T00:46:28.613713+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:46:28.613713+0000","last_undegraded":"2024-02-23T00:46:28.613713+0000","last_fullsized":"2024-02-23T00:46:28.613713+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:45:50.076710+0000","last_deep_scrub":"453'38","last_deep_scrub_stamp":"2024-02-23T00:44:29.039144+0000","last_clean_scrub_stamp":"2024-02-23T00:45:50.076710+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":889,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473662+0000","last_change":"2024-02-23T00:45:35.172325+0000","last_active":"2024-02-23T00:46:28.473662+0000","last_peered":"2024-02-23T00:46:28.473662+0000","last_clean":"2024-02-23T00:46:28.473662+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:46:28.473662+0000","last_undegraded":"2024-02-23T00:46:28.473662+0000","last_fullsized":"2024-02-23T00:46:28.473662+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:35.172247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:45:35.172247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.613904+0000","last_change":"2024-02-23T00:45:47.032756+0000","last_active":"2024-02-23T00:46:28.613904+0000","last_peered":"2024-02-23T00:46:28.613904+0000","last_clean":"2024-02-23T00:46:28.613904+0000","last_became_active":"2024-02-23T00:41:37.222092+0000","last_became_peered":"2024-02-23T00:41:37.222092+0000","last_unstale":"2024-02-23T00:46:28.613904+0000","last_undegraded":"2024-02-23T00:46:28.613904+0000","last_fullsized":"2024-02-23T00:46:28.613904+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:47.032669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:45:47.032669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":819,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711718+0000","last_change":"2024-02-23T00:46:23.327485+0000","last_active":"2024-02-23T00:46:28.711718+0000","last_peered":"2024-02-23T00:46:28.711718+0000","last_clean":"2024-02-23T00:46:28.711718+0000","last_became_active":"2024-02-23T00:41:37.214214+0000","last_became_peered":"2024-02-23T00:41:37.214214+0000","last_unstale":"2024-02-23T00:46:28.711718+0000","last_undegraded":"2024-02-23T00:46:28.711718+0000","last_fullsized":"2024-02-23T00:46:28.711718+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:23.327413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:23.327413+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":1697,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:38.902507+0000","last_change":"2024-02-23T00:46:14.767726+0000","last_active":"2024-02-23T00:46:38.902507+0000","last_peered":"2024-02-23T00:46:38.902507+0000","last_clean":"2024-02-23T00:46:38.902507+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:46:38.902507+0000","last_undegraded":"2024-02-23T00:46:38.902507+0000","last_fullsized":"2024-02-23T00:46:38.902507+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:46:14.767639+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:46:14.767639+0000","last_clean_scrub_stamp":"2024-02-23T00:46:14.767639+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":840,"num_read_kb":840,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"840'27","reported_seq":579,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711448+0000","last_change":"2024-02-23T00:45:26.120956+0000","last_active":"2024-02-23T00:46:28.711448+0000","last_peered":"2024-02-23T00:46:28.711448+0000","last_clean":"2024-02-23T00:46:28.711448+0000","last_became_active":"2024-02-23T00:41:41.226195+0000","last_became_peered":"2024-02-23T00:41:41.226195+0000","last_unstale":"2024-02-23T00:46:28.711448+0000","last_undegraded":"2024-02-23T00:46:28.711448+0000","last_fullsized":"2024-02-23T00:46:28.711448+0000","mapping_epoch":681,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"840'27","last_scrub_stamp":"2024-02-23T00:45:26.120891+0000","last_deep_scrub":"840'27","last_deep_scrub_stamp":"2024-02-23T00:45:26.120891+0000","last_clean_scrub_stamp":"2024-02-23T00:45:26.120891+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":862,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710774+0000","last_change":"2024-02-23T00:46:00.293842+0000","last_active":"2024-02-23T00:46:28.710774+0000","last_peered":"2024-02-23T00:46:28.710774+0000","last_clean":"2024-02-23T00:46:28.710774+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:46:28.710774+0000","last_undegraded":"2024-02-23T00:46:28.710774+0000","last_fullsized":"2024-02-23T00:46:28.710774+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:00.293765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:00.293765+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.293765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":876,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472000+0000","last_change":"2024-02-23T00:45:43.263443+0000","last_active":"2024-02-23T00:46:28.472000+0000","last_peered":"2024-02-23T00:46:28.472000+0000","last_clean":"2024-02-23T00:46:28.472000+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:46:28.472000+0000","last_undegraded":"2024-02-23T00:46:28.472000+0000","last_fullsized":"2024-02-23T00:46:28.472000+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:43.263312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:43.263312+0000","last_clean_scrub_stamp":"2024-02-23T00:45:43.263312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":202,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710001+0000","last_change":"2024-02-23T00:44:57.880383+0000","last_active":"2024-02-23T00:46:28.710001+0000","last_peered":"2024-02-23T00:46:28.710001+0000","last_clean":"2024-02-23T00:46:28.710001+0000","last_became_active":"2024-02-23T00:41:37.228937+0000","last_became_peered":"2024-02-23T00:41:37.228937+0000","last_unstale":"2024-02-23T00:46:28.710001+0000","last_undegraded":"2024-02-23T00:46:28.710001+0000","last_fullsized":"2024-02-23T00:46:28.710001+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:57.880312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:47.610212+0000","last_clean_scrub_stamp":"2024-02-23T00:44:57.880312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710953+0000","last_change":"2024-02-23T00:45:44.212146+0000","last_active":"2024-02-23T00:46:28.710953+0000","last_peered":"2024-02-23T00:46:28.710953+0000","last_clean":"2024-02-23T00:46:28.710953+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:46:28.710953+0000","last_undegraded":"2024-02-23T00:46:28.710953+0000","last_fullsized":"2024-02-23T00:46:28.710953+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:44.212054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:45:44.212054+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616101+0000","last_change":"2024-02-23T00:46:05.662122+0000","last_active":"2024-02-23T00:46:28.616101+0000","last_peered":"2024-02-23T00:46:28.616101+0000","last_clean":"2024-02-23T00:46:28.616101+0000","last_became_active":"2024-02-23T00:42:41.393491+0000","last_became_peered":"2024-02-23T00:42:41.393491+0000","last_unstale":"2024-02-23T00:46:28.616101+0000","last_undegraded":"2024-02-23T00:46:28.616101+0000","last_fullsized":"2024-02-23T00:46:28.616101+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:05.662037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:46:05.662037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":979,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710832+0000","last_change":"2024-02-23T00:45:37.271204+0000","last_active":"2024-02-23T00:46:28.710832+0000","last_peered":"2024-02-23T00:46:28.710832+0000","last_clean":"2024-02-23T00:46:28.710832+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:46:28.710832+0000","last_undegraded":"2024-02-23T00:46:28.710832+0000","last_fullsized":"2024-02-23T00:46:28.710832+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:45:37.271124+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:45:37.271124+0000","last_clean_scrub_stamp":"2024-02-23T00:45:37.271124+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":887,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:38.206978+0000","last_change":"2024-02-23T00:46:38.206978+0000","last_active":"2024-02-23T00:46:38.206978+0000","last_peered":"2024-02-23T00:46:38.206978+0000","last_clean":"2024-02-23T00:46:38.206978+0000","last_became_active":"2024-02-23T00:41:37.238728+0000","last_became_peered":"2024-02-23T00:41:37.238728+0000","last_unstale":"2024-02-23T00:46:38.206978+0000","last_undegraded":"2024-02-23T00:46:38.206978+0000","last_fullsized":"2024-02-23T00:46:38.206978+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:38.206914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:38.206914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473375+0000","last_change":"2024-02-23T00:45:20.260925+0000","last_active":"2024-02-23T00:46:28.473375+0000","last_peered":"2024-02-23T00:46:28.473375+0000","last_clean":"2024-02-23T00:46:28.473375+0000","last_became_active":"2024-02-23T00:41:37.239529+0000","last_became_peered":"2024-02-23T00:41:37.239529+0000","last_unstale":"2024-02-23T00:46:28.473375+0000","last_undegraded":"2024-02-23T00:46:28.473375+0000","last_fullsized":"2024-02-23T00:46:28.473375+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:20.260860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:20.260860+0000","last_clean_scrub_stamp":"2024-02-23T00:45:20.260860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":882,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615360+0000","last_change":"2024-02-23T00:46:27.735713+0000","last_active":"2024-02-23T00:46:28.615360+0000","last_peered":"2024-02-23T00:46:28.615360+0000","last_clean":"2024-02-23T00:46:28.615360+0000","last_became_active":"2024-02-23T00:40:37.197010+0000","last_became_peered":"2024-02-23T00:40:37.197010+0000","last_unstale":"2024-02-23T00:46:28.615360+0000","last_undegraded":"2024-02-23T00:46:28.615360+0000","last_fullsized":"2024-02-23T00:46:28.615360+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:27.735611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:02.358056+0000","last_clean_scrub_stamp":"2024-02-23T00:46:27.735611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":202,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710084+0000","last_change":"2024-02-23T00:46:12.993779+0000","last_active":"2024-02-23T00:46:28.710084+0000","last_peered":"2024-02-23T00:46:28.710084+0000","last_clean":"2024-02-23T00:46:28.710084+0000","last_became_active":"2024-02-23T00:41:37.248272+0000","last_became_peered":"2024-02-23T00:41:37.248272+0000","last_unstale":"2024-02-23T00:46:28.710084+0000","last_undegraded":"2024-02-23T00:46:28.710084+0000","last_fullsized":"2024-02-23T00:46:28.710084+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:12.993709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:12.993709+0000","last_clean_scrub_stamp":"2024-02-23T00:46:12.993709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":864,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711069+0000","last_change":"2024-02-23T00:46:01.349093+0000","last_active":"2024-02-23T00:46:28.711069+0000","last_peered":"2024-02-23T00:46:28.711069+0000","last_clean":"2024-02-23T00:46:28.711069+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:46:28.711069+0000","last_undegraded":"2024-02-23T00:46:28.711069+0000","last_fullsized":"2024-02-23T00:46:28.711069+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:01.349005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:01.349005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":792,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:33.830195+0000","last_change":"2024-02-23T00:46:33.830195+0000","last_active":"2024-02-23T00:46:33.830195+0000","last_peered":"2024-02-23T00:46:33.830195+0000","last_clean":"2024-02-23T00:46:33.830195+0000","last_became_active":"2024-02-23T00:45:11.598917+0000","last_became_peered":"2024-02-23T00:45:11.598917+0000","last_unstale":"2024-02-23T00:46:33.830195+0000","last_undegraded":"2024-02-23T00:46:33.830195+0000","last_fullsized":"2024-02-23T00:46:33.830195+0000","mapping_epoch":833,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:46:33.830097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:33.830097+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":852,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615644+0000","last_change":"2024-02-23T00:46:25.653202+0000","last_active":"2024-02-23T00:46:28.615644+0000","last_peered":"2024-02-23T00:46:28.615644+0000","last_clean":"2024-02-23T00:46:28.615644+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:46:28.615644+0000","last_undegraded":"2024-02-23T00:46:28.615644+0000","last_fullsized":"2024-02-23T00:46:28.615644+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:25.653122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:25.653122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":859,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:38.484586+0000","last_change":"2024-02-23T00:46:38.484586+0000","last_active":"2024-02-23T00:46:38.484586+0000","last_peered":"2024-02-23T00:46:38.484586+0000","last_clean":"2024-02-23T00:46:38.484586+0000","last_became_active":"2024-02-23T00:41:37.229416+0000","last_became_peered":"2024-02-23T00:41:37.229416+0000","last_unstale":"2024-02-23T00:46:38.484586+0000","last_undegraded":"2024-02-23T00:46:38.484586+0000","last_fullsized":"2024-02-23T00:46:38.484586+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:38.484522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:38.484522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":91,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710776+0000","last_change":"2024-02-23T00:45:16.953513+0000","last_active":"2024-02-23T00:46:28.710776+0000","last_peered":"2024-02-23T00:46:28.710776+0000","last_clean":"2024-02-23T00:46:28.710776+0000","last_became_active":"2024-02-23T00:43:42.260383+0000","last_became_peered":"2024-02-23T00:43:42.260383+0000","last_unstale":"2024-02-23T00:46:28.710776+0000","last_undegraded":"2024-02-23T00:46:28.710776+0000","last_fullsized":"2024-02-23T00:46:28.710776+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:16.953402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:16.953402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":388,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.305191+0000","last_change":"2024-02-23T00:46:40.305191+0000","last_active":"2024-02-23T00:46:40.305191+0000","last_peered":"2024-02-23T00:46:40.305191+0000","last_clean":"2024-02-23T00:46:40.305191+0000","last_became_active":"2024-02-23T00:41:37.252083+0000","last_became_peered":"2024-02-23T00:41:37.252083+0000","last_unstale":"2024-02-23T00:46:40.305191+0000","last_undegraded":"2024-02-23T00:46:40.305191+0000","last_fullsized":"2024-02-23T00:46:40.305191+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:40.305086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:40.305086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":831,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471762+0000","last_change":"2024-02-23T00:45:58.130467+0000","last_active":"2024-02-23T00:46:28.471762+0000","last_peered":"2024-02-23T00:46:28.471762+0000","last_clean":"2024-02-23T00:46:28.471762+0000","last_became_active":"2024-02-23T00:45:11.600573+0000","last_became_peered":"2024-02-23T00:45:11.600573+0000","last_unstale":"2024-02-23T00:46:28.471762+0000","last_undegraded":"2024-02-23T00:46:28.471762+0000","last_fullsized":"2024-02-23T00:46:28.471762+0000","mapping_epoch":833,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:45:58.130374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:58.130374+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":875,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711572+0000","last_change":"2024-02-23T00:46:05.283413+0000","last_active":"2024-02-23T00:46:28.711572+0000","last_peered":"2024-02-23T00:46:28.711572+0000","last_clean":"2024-02-23T00:46:28.711572+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:46:28.711572+0000","last_undegraded":"2024-02-23T00:46:28.711572+0000","last_fullsized":"2024-02-23T00:46:28.711572+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:05.283350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:46:05.283350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":1098,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.552221+0000","last_change":"2024-02-23T00:45:47.340086+0000","last_active":"2024-02-23T00:46:35.552221+0000","last_peered":"2024-02-23T00:46:35.552221+0000","last_clean":"2024-02-23T00:46:35.552221+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:46:35.552221+0000","last_undegraded":"2024-02-23T00:46:35.552221+0000","last_fullsized":"2024-02-23T00:46:35.552221+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:45:47.340019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:45:47.340019+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472155+0000","last_change":"2024-02-23T00:45:44.201895+0000","last_active":"2024-02-23T00:46:28.472155+0000","last_peered":"2024-02-23T00:46:28.472155+0000","last_clean":"2024-02-23T00:46:28.472155+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:46:28.472155+0000","last_undegraded":"2024-02-23T00:46:28.472155+0000","last_fullsized":"2024-02-23T00:46:28.472155+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:44.201810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:44.201810+0000","last_clean_scrub_stamp":"2024-02-23T00:45:44.201810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":369,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473263+0000","last_change":"2024-02-23T00:45:28.552593+0000","last_active":"2024-02-23T00:46:28.473263+0000","last_peered":"2024-02-23T00:46:28.473263+0000","last_clean":"2024-02-23T00:46:28.473263+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:46:28.473263+0000","last_undegraded":"2024-02-23T00:46:28.473263+0000","last_fullsized":"2024-02-23T00:46:28.473263+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:45:28.552529+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:45:28.552529+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":1034,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:37.509803+0000","last_change":"2024-02-23T00:46:37.509803+0000","last_active":"2024-02-23T00:46:37.509803+0000","last_peered":"2024-02-23T00:46:37.509803+0000","last_clean":"2024-02-23T00:46:37.509803+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:46:37.509803+0000","last_undegraded":"2024-02-23T00:46:37.509803+0000","last_fullsized":"2024-02-23T00:46:37.509803+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:46:37.509720+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:46:37.509720+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":859,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710160+0000","last_change":"2024-02-23T00:46:06.300950+0000","last_active":"2024-02-23T00:46:28.710160+0000","last_peered":"2024-02-23T00:46:28.710160+0000","last_clean":"2024-02-23T00:46:28.710160+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:46:28.710160+0000","last_undegraded":"2024-02-23T00:46:28.710160+0000","last_fullsized":"2024-02-23T00:46:28.710160+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:06.300882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:06.300882+0000","last_clean_scrub_stamp":"2024-02-23T00:46:06.300882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"851'14","reported_seq":933,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.552209+0000","last_change":"2024-02-23T00:46:02.177625+0000","last_active":"2024-02-23T00:46:35.552209+0000","last_peered":"2024-02-23T00:46:35.552209+0000","last_clean":"2024-02-23T00:46:35.552209+0000","last_became_active":"2024-02-23T00:45:11.600072+0000","last_became_peered":"2024-02-23T00:45:11.600072+0000","last_unstale":"2024-02-23T00:46:35.552209+0000","last_undegraded":"2024-02-23T00:46:35.552209+0000","last_fullsized":"2024-02-23T00:46:35.552209+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'14","last_scrub_stamp":"2024-02-23T00:46:02.177545+0000","last_deep_scrub":"851'14","last_deep_scrub_stamp":"2024-02-23T00:46:02.177545+0000","last_clean_scrub_stamp":"2024-02-23T00:46:02.177545+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":791,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:31.466305+0000","last_change":"2024-02-23T00:46:31.466281+0000","last_active":"2024-02-23T00:46:31.466305+0000","last_peered":"2024-02-23T00:46:31.466305+0000","last_clean":"2024-02-23T00:46:31.466305+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:46:31.466305+0000","last_undegraded":"2024-02-23T00:46:31.466305+0000","last_fullsized":"2024-02-23T00:46:31.466305+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:31.466214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:46:31.466214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":861,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471061+0000","last_change":"2024-02-23T00:45:42.221560+0000","last_active":"2024-02-23T00:46:28.471061+0000","last_peered":"2024-02-23T00:46:28.471061+0000","last_clean":"2024-02-23T00:46:28.471061+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:46:28.471061+0000","last_undegraded":"2024-02-23T00:46:28.471061+0000","last_fullsized":"2024-02-23T00:46:28.471061+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:42.221479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:03.136682+0000","last_clean_scrub_stamp":"2024-02-23T00:45:42.221479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":801,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472043+0000","last_change":"2024-02-23T00:45:11.601330+0000","last_active":"2024-02-23T00:46:28.472043+0000","last_peered":"2024-02-23T00:46:28.472043+0000","last_clean":"2024-02-23T00:46:28.472043+0000","last_became_active":"2024-02-23T00:45:11.600967+0000","last_became_peered":"2024-02-23T00:45:11.600967+0000","last_unstale":"2024-02-23T00:46:28.472043+0000","last_undegraded":"2024-02-23T00:46:28.472043+0000","last_fullsized":"2024-02-23T00:46:28.472043+0000","mapping_epoch":833,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:45:09.194799+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:45:09.194799+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":888,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615604+0000","last_change":"2024-02-23T00:45:51.754064+0000","last_active":"2024-02-23T00:46:28.615604+0000","last_peered":"2024-02-23T00:46:28.615604+0000","last_clean":"2024-02-23T00:46:28.615604+0000","last_became_active":"2024-02-23T00:41:37.229862+0000","last_became_peered":"2024-02-23T00:41:37.229862+0000","last_unstale":"2024-02-23T00:46:28.615604+0000","last_undegraded":"2024-02-23T00:46:28.615604+0000","last_fullsized":"2024-02-23T00:46:28.615604+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:45:51.753983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:51.753983+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":836,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615011+0000","last_change":"2024-02-23T00:46:11.822018+0000","last_active":"2024-02-23T00:46:28.615011+0000","last_peered":"2024-02-23T00:46:28.615011+0000","last_clean":"2024-02-23T00:46:28.615011+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:46:28.615011+0000","last_undegraded":"2024-02-23T00:46:28.615011+0000","last_fullsized":"2024-02-23T00:46:28.615011+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:11.821938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.821938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":809,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711007+0000","last_change":"2024-02-23T00:46:13.377537+0000","last_active":"2024-02-23T00:46:28.711007+0000","last_peered":"2024-02-23T00:46:28.711007+0000","last_clean":"2024-02-23T00:46:28.711007+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:46:28.711007+0000","last_undegraded":"2024-02-23T00:46:28.711007+0000","last_fullsized":"2024-02-23T00:46:28.711007+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:13.377475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_clean_scrub_stamp":"2024-02-23T00:46:13.377475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":216,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711139+0000","last_change":"2024-02-23T00:46:22.040756+0000","last_active":"2024-02-23T00:46:28.711139+0000","last_peered":"2024-02-23T00:46:28.711139+0000","last_clean":"2024-02-23T00:46:28.711139+0000","last_became_active":"2024-02-23T00:42:39.394104+0000","last_became_peered":"2024-02-23T00:42:39.394104+0000","last_unstale":"2024-02-23T00:46:28.711139+0000","last_undegraded":"2024-02-23T00:46:28.711139+0000","last_fullsized":"2024-02-23T00:46:28.711139+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:22.040679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:46:22.040679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"642'14","reported_seq":452,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.552072+0000","last_change":"2024-02-23T00:45:21.622607+0000","last_active":"2024-02-23T00:46:35.552072+0000","last_peered":"2024-02-23T00:46:35.552072+0000","last_clean":"2024-02-23T00:46:35.552072+0000","last_became_active":"2024-02-23T00:41:01.212780+0000","last_became_peered":"2024-02-23T00:41:01.212780+0000","last_unstale":"2024-02-23T00:46:35.552072+0000","last_undegraded":"2024-02-23T00:46:35.552072+0000","last_fullsized":"2024-02-23T00:46:35.552072+0000","mapping_epoch":511,"log_start":"562'12","ondisk_log_start":"562'12","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'14","last_scrub_stamp":"2024-02-23T00:45:21.622489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:45:21.622489+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":758,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471349+0000","last_change":"2024-02-23T00:45:27.426484+0000","last_active":"2024-02-23T00:46:28.471349+0000","last_peered":"2024-02-23T00:46:28.471349+0000","last_clean":"2024-02-23T00:46:28.471349+0000","last_became_active":"2024-02-23T00:45:11.601293+0000","last_became_peered":"2024-02-23T00:45:11.601293+0000","last_unstale":"2024-02-23T00:46:28.471349+0000","last_undegraded":"2024-02-23T00:46:28.471349+0000","last_fullsized":"2024-02-23T00:46:28.471349+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'34","last_scrub_stamp":"2024-02-23T00:45:27.426401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:27.426401+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614720+0000","last_change":"2024-02-23T00:45:51.094921+0000","last_active":"2024-02-23T00:46:28.614720+0000","last_peered":"2024-02-23T00:46:28.614720+0000","last_clean":"2024-02-23T00:46:28.614720+0000","last_became_active":"2024-02-23T00:45:19.644963+0000","last_became_peered":"2024-02-23T00:45:19.644963+0000","last_unstale":"2024-02-23T00:46:28.614720+0000","last_undegraded":"2024-02-23T00:46:28.614720+0000","last_fullsized":"2024-02-23T00:46:28.614720+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:51.094839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:42.679545+0000","last_clean_scrub_stamp":"2024-02-23T00:45:51.094839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":869,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:37.535423+0000","last_change":"2024-02-23T00:46:37.535423+0000","last_active":"2024-02-23T00:46:37.535423+0000","last_peered":"2024-02-23T00:46:37.535423+0000","last_clean":"2024-02-23T00:46:37.535423+0000","last_became_active":"2024-02-23T00:40:57.488797+0000","last_became_peered":"2024-02-23T00:40:57.488797+0000","last_unstale":"2024-02-23T00:46:37.535423+0000","last_undegraded":"2024-02-23T00:46:37.535423+0000","last_fullsized":"2024-02-23T00:46:37.535423+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:37.535342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:46:37.535342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":834,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:39.499430+0000","last_change":"2024-02-23T00:46:39.499430+0000","last_active":"2024-02-23T00:46:39.499430+0000","last_peered":"2024-02-23T00:46:39.499430+0000","last_clean":"2024-02-23T00:46:39.499430+0000","last_became_active":"2024-02-23T00:41:37.229061+0000","last_became_peered":"2024-02-23T00:41:37.229061+0000","last_unstale":"2024-02-23T00:46:39.499430+0000","last_undegraded":"2024-02-23T00:46:39.499430+0000","last_fullsized":"2024-02-23T00:46:39.499430+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:39.499336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:29.596983+0000","last_clean_scrub_stamp":"2024-02-23T00:46:39.499336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":404,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472786+0000","last_change":"2024-02-23T00:46:12.096727+0000","last_active":"2024-02-23T00:46:28.472786+0000","last_peered":"2024-02-23T00:46:28.472786+0000","last_clean":"2024-02-23T00:46:28.472786+0000","last_became_active":"2024-02-23T00:42:39.396063+0000","last_became_peered":"2024-02-23T00:42:39.396063+0000","last_unstale":"2024-02-23T00:46:28.472786+0000","last_undegraded":"2024-02-23T00:46:28.472786+0000","last_fullsized":"2024-02-23T00:46:28.472786+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:12.096663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:12.096663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":899,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471598+0000","last_change":"2024-02-23T00:46:07.103115+0000","last_active":"2024-02-23T00:46:28.471598+0000","last_peered":"2024-02-23T00:46:28.471598+0000","last_clean":"2024-02-23T00:46:28.471598+0000","last_became_active":"2024-02-23T00:40:57.218869+0000","last_became_peered":"2024-02-23T00:40:57.218869+0000","last_unstale":"2024-02-23T00:46:28.471598+0000","last_undegraded":"2024-02-23T00:46:28.471598+0000","last_fullsized":"2024-02-23T00:46:28.471598+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:07.103031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:07.103031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":437,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:32.403751+0000","last_change":"2024-02-23T00:46:32.403751+0000","last_active":"2024-02-23T00:46:32.403751+0000","last_peered":"2024-02-23T00:46:32.403751+0000","last_clean":"2024-02-23T00:46:32.403751+0000","last_became_active":"2024-02-23T00:41:37.228326+0000","last_became_peered":"2024-02-23T00:41:37.228326+0000","last_unstale":"2024-02-23T00:46:32.403751+0000","last_undegraded":"2024-02-23T00:46:32.403751+0000","last_fullsized":"2024-02-23T00:46:32.403751+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:46:32.403638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:32.403638+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":844,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.453333+0000","last_change":"2024-02-23T00:46:35.453333+0000","last_active":"2024-02-23T00:46:35.453333+0000","last_peered":"2024-02-23T00:46:35.453333+0000","last_clean":"2024-02-23T00:46:35.453333+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:46:35.453333+0000","last_undegraded":"2024-02-23T00:46:35.453333+0000","last_fullsized":"2024-02-23T00:46:35.453333+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:35.453269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:29.390757+0000","last_clean_scrub_stamp":"2024-02-23T00:46:35.453269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":738,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471668+0000","last_change":"2024-02-23T00:46:22.989999+0000","last_active":"2024-02-23T00:46:28.471668+0000","last_peered":"2024-02-23T00:46:28.471668+0000","last_clean":"2024-02-23T00:46:28.471668+0000","last_became_active":"2024-02-23T00:45:11.594841+0000","last_became_peered":"2024-02-23T00:45:11.594841+0000","last_unstale":"2024-02-23T00:46:28.471668+0000","last_undegraded":"2024-02-23T00:46:28.471668+0000","last_fullsized":"2024-02-23T00:46:28.471668+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:22.989879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:46:22.989879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":378,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472638+0000","last_change":"2024-02-23T00:46:08.116839+0000","last_active":"2024-02-23T00:46:28.472638+0000","last_peered":"2024-02-23T00:46:28.472638+0000","last_clean":"2024-02-23T00:46:28.472638+0000","last_became_active":"2024-02-23T00:43:42.256711+0000","last_became_peered":"2024-02-23T00:43:42.256711+0000","last_unstale":"2024-02-23T00:46:28.472638+0000","last_undegraded":"2024-02-23T00:46:28.472638+0000","last_fullsized":"2024-02-23T00:46:28.472638+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:08.116763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:08.116763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":1116,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:34.491825+0000","last_change":"2024-02-23T00:46:34.491825+0000","last_active":"2024-02-23T00:46:34.491825+0000","last_peered":"2024-02-23T00:46:34.491825+0000","last_clean":"2024-02-23T00:46:34.491825+0000","last_became_active":"2024-02-23T00:42:39.398790+0000","last_became_peered":"2024-02-23T00:42:39.398790+0000","last_unstale":"2024-02-23T00:46:34.491825+0000","last_undegraded":"2024-02-23T00:46:34.491825+0000","last_fullsized":"2024-02-23T00:46:34.491825+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:46:34.491739+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:45:28.352035+0000","last_clean_scrub_stamp":"2024-02-23T00:46:34.491739+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":900,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710948+0000","last_change":"2024-02-23T00:46:20.197787+0000","last_active":"2024-02-23T00:46:28.710948+0000","last_peered":"2024-02-23T00:46:28.710948+0000","last_clean":"2024-02-23T00:46:28.710948+0000","last_became_active":"2024-02-23T00:41:37.220531+0000","last_became_peered":"2024-02-23T00:41:37.220531+0000","last_unstale":"2024-02-23T00:46:28.710948+0000","last_undegraded":"2024-02-23T00:46:28.710948+0000","last_fullsized":"2024-02-23T00:46:28.710948+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:20.197708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_clean_scrub_stamp":"2024-02-23T00:46:20.197708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"801'44","reported_seq":235,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711490+0000","last_change":"2024-02-23T00:45:30.162438+0000","last_active":"2024-02-23T00:46:28.711490+0000","last_peered":"2024-02-23T00:46:28.711490+0000","last_clean":"2024-02-23T00:46:28.711490+0000","last_became_active":"2024-02-23T00:41:37.235230+0000","last_became_peered":"2024-02-23T00:41:37.235230+0000","last_unstale":"2024-02-23T00:46:28.711490+0000","last_undegraded":"2024-02-23T00:46:28.711490+0000","last_fullsized":"2024-02-23T00:46:28.711490+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'44","last_scrub_stamp":"2024-02-23T00:45:30.162372+0000","last_deep_scrub":"801'44","last_deep_scrub_stamp":"2024-02-23T00:44:02.476354+0000","last_clean_scrub_stamp":"2024-02-23T00:45:30.162372+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616412+0000","last_change":"2024-02-23T00:46:14.567073+0000","last_active":"2024-02-23T00:46:28.616412+0000","last_peered":"2024-02-23T00:46:28.616412+0000","last_clean":"2024-02-23T00:46:28.616412+0000","last_became_active":"2024-02-23T00:41:37.229922+0000","last_became_peered":"2024-02-23T00:41:37.229922+0000","last_unstale":"2024-02-23T00:46:28.616412+0000","last_undegraded":"2024-02-23T00:46:28.616412+0000","last_fullsized":"2024-02-23T00:46:28.616412+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:14.566990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:14.566990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":856,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710246+0000","last_change":"2024-02-23T00:46:11.323233+0000","last_active":"2024-02-23T00:46:28.710246+0000","last_peered":"2024-02-23T00:46:28.710246+0000","last_clean":"2024-02-23T00:46:28.710246+0000","last_became_active":"2024-02-23T00:41:05.523365+0000","last_became_peered":"2024-02-23T00:41:05.523365+0000","last_unstale":"2024-02-23T00:46:28.710246+0000","last_undegraded":"2024-02-23T00:46:28.710246+0000","last_fullsized":"2024-02-23T00:46:28.710246+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:11.323140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.323140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":786,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710323+0000","last_change":"2024-02-23T00:45:52.304920+0000","last_active":"2024-02-23T00:46:28.710323+0000","last_peered":"2024-02-23T00:46:28.710323+0000","last_clean":"2024-02-23T00:46:28.710323+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:46:28.710323+0000","last_undegraded":"2024-02-23T00:46:28.710323+0000","last_fullsized":"2024-02-23T00:46:28.710323+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:52.304832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:52.304832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":882,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615073+0000","last_change":"2024-02-23T00:46:25.642212+0000","last_active":"2024-02-23T00:46:28.615073+0000","last_peered":"2024-02-23T00:46:28.615073+0000","last_clean":"2024-02-23T00:46:28.615073+0000","last_became_active":"2024-02-23T00:41:37.228303+0000","last_became_peered":"2024-02-23T00:41:37.228303+0000","last_unstale":"2024-02-23T00:46:28.615073+0000","last_undegraded":"2024-02-23T00:46:28.615073+0000","last_fullsized":"2024-02-23T00:46:28.615073+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:25.642137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:46:25.642137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":585,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710983+0000","last_change":"2024-02-23T00:45:03.809706+0000","last_active":"2024-02-23T00:46:28.710983+0000","last_peered":"2024-02-23T00:46:28.710983+0000","last_clean":"2024-02-23T00:46:28.710983+0000","last_became_active":"2024-02-23T00:41:41.224814+0000","last_became_peered":"2024-02-23T00:41:41.224814+0000","last_unstale":"2024-02-23T00:46:28.710983+0000","last_undegraded":"2024-02-23T00:46:28.710983+0000","last_fullsized":"2024-02-23T00:46:28.710983+0000","mapping_epoch":681,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:45:03.809635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:03.809635+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615473+0000","last_change":"2024-02-23T00:46:11.504052+0000","last_active":"2024-02-23T00:46:28.615473+0000","last_peered":"2024-02-23T00:46:28.615473+0000","last_clean":"2024-02-23T00:46:28.615473+0000","last_became_active":"2024-02-23T00:41:37.229722+0000","last_became_peered":"2024-02-23T00:41:37.229722+0000","last_unstale":"2024-02-23T00:46:28.615473+0000","last_undegraded":"2024-02-23T00:46:28.615473+0000","last_fullsized":"2024-02-23T00:46:28.615473+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:11.503969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.503969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":216,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:33.141005+0000","last_change":"2024-02-23T00:46:33.141005+0000","last_active":"2024-02-23T00:46:33.141005+0000","last_peered":"2024-02-23T00:46:33.141005+0000","last_clean":"2024-02-23T00:46:33.141005+0000","last_became_active":"2024-02-23T00:42:39.395793+0000","last_became_peered":"2024-02-23T00:42:39.395793+0000","last_unstale":"2024-02-23T00:46:33.141005+0000","last_undegraded":"2024-02-23T00:46:33.141005+0000","last_fullsized":"2024-02-23T00:46:33.141005+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:33.140935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:33.140935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"78'2","reported_seq":887,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710921+0000","last_change":"2024-02-23T00:45:42.370861+0000","last_active":"2024-02-23T00:46:28.710921+0000","last_peered":"2024-02-23T00:46:28.710921+0000","last_clean":"2024-02-23T00:46:28.710921+0000","last_became_active":"2024-02-23T00:41:25.236069+0000","last_became_peered":"2024-02-23T00:41:25.236069+0000","last_unstale":"2024-02-23T00:46:28.710921+0000","last_undegraded":"2024-02-23T00:46:28.710921+0000","last_fullsized":"2024-02-23T00:46:28.710921+0000","mapping_epoch":511,"log_start":"78'2","ondisk_log_start":"78'2","created":60,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:45:42.370765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:45:42.370765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471844+0000","last_change":"2024-02-23T00:46:11.173320+0000","last_active":"2024-02-23T00:46:28.471844+0000","last_peered":"2024-02-23T00:46:28.471844+0000","last_clean":"2024-02-23T00:46:28.471844+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:46:28.471844+0000","last_undegraded":"2024-02-23T00:46:28.471844+0000","last_fullsized":"2024-02-23T00:46:28.471844+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:11.173242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.173242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":875,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616064+0000","last_change":"2024-02-23T00:45:22.441742+0000","last_active":"2024-02-23T00:46:28.616064+0000","last_peered":"2024-02-23T00:46:28.616064+0000","last_clean":"2024-02-23T00:46:28.616064+0000","last_became_active":"2024-02-23T00:40:45.191453+0000","last_became_peered":"2024-02-23T00:40:45.191453+0000","last_unstale":"2024-02-23T00:46:28.616064+0000","last_undegraded":"2024-02-23T00:46:28.616064+0000","last_fullsized":"2024-02-23T00:46:28.616064+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:22.441631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:45:22.441631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":896,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710914+0000","last_change":"2024-02-23T00:46:15.254363+0000","last_active":"2024-02-23T00:46:28.710914+0000","last_peered":"2024-02-23T00:46:28.710914+0000","last_clean":"2024-02-23T00:46:28.710914+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:46:28.710914+0000","last_undegraded":"2024-02-23T00:46:28.710914+0000","last_fullsized":"2024-02-23T00:46:28.710914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:46:15.254281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:15.254281+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":874,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615467+0000","last_change":"2024-02-23T00:45:32.645969+0000","last_active":"2024-02-23T00:46:28.615467+0000","last_peered":"2024-02-23T00:46:28.615467+0000","last_clean":"2024-02-23T00:46:28.615467+0000","last_became_active":"2024-02-23T00:41:37.229335+0000","last_became_peered":"2024-02-23T00:41:37.229335+0000","last_unstale":"2024-02-23T00:46:28.615467+0000","last_undegraded":"2024-02-23T00:46:28.615467+0000","last_fullsized":"2024-02-23T00:46:28.615467+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:32.645873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:45:32.645873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":845,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614038+0000","last_change":"2024-02-23T00:46:22.691597+0000","last_active":"2024-02-23T00:46:28.614038+0000","last_peered":"2024-02-23T00:46:28.614038+0000","last_clean":"2024-02-23T00:46:28.614038+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:46:28.614038+0000","last_undegraded":"2024-02-23T00:46:28.614038+0000","last_fullsized":"2024-02-23T00:46:28.614038+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:22.691462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:19.043451+0000","last_clean_scrub_stamp":"2024-02-23T00:46:22.691462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":378,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472402+0000","last_change":"2024-02-23T00:45:41.312482+0000","last_active":"2024-02-23T00:46:28.472402+0000","last_peered":"2024-02-23T00:46:28.472402+0000","last_clean":"2024-02-23T00:46:28.472402+0000","last_became_active":"2024-02-23T00:42:39.396071+0000","last_became_peered":"2024-02-23T00:42:39.396071+0000","last_unstale":"2024-02-23T00:46:28.472402+0000","last_undegraded":"2024-02-23T00:46:28.472402+0000","last_fullsized":"2024-02-23T00:46:28.472402+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:41.312417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:41.312417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":888,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615108+0000","last_change":"2024-02-23T00:45:45.986880+0000","last_active":"2024-02-23T00:46:28.615108+0000","last_peered":"2024-02-23T00:46:28.615108+0000","last_clean":"2024-02-23T00:46:28.615108+0000","last_became_active":"2024-02-23T00:41:13.469953+0000","last_became_peered":"2024-02-23T00:41:13.469953+0000","last_unstale":"2024-02-23T00:46:28.615108+0000","last_undegraded":"2024-02-23T00:46:28.615108+0000","last_fullsized":"2024-02-23T00:46:28.615108+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:45.986798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:45:45.986798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":801,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472183+0000","last_change":"2024-02-23T00:46:00.176954+0000","last_active":"2024-02-23T00:46:28.472183+0000","last_peered":"2024-02-23T00:46:28.472183+0000","last_clean":"2024-02-23T00:46:28.472183+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:46:28.472183+0000","last_undegraded":"2024-02-23T00:46:28.472183+0000","last_fullsized":"2024-02-23T00:46:28.472183+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:00.176870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.176870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":891,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473799+0000","last_change":"2024-02-23T00:45:56.268314+0000","last_active":"2024-02-23T00:46:28.473799+0000","last_peered":"2024-02-23T00:46:28.473799+0000","last_clean":"2024-02-23T00:46:28.473799+0000","last_became_active":"2024-02-23T00:40:38.061696+0000","last_became_peered":"2024-02-23T00:40:38.061696+0000","last_unstale":"2024-02-23T00:46:28.473799+0000","last_undegraded":"2024-02-23T00:46:28.473799+0000","last_fullsized":"2024-02-23T00:46:28.473799+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:56.268250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:45:56.268250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":993,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711074+0000","last_change":"2024-02-23T00:46:21.322307+0000","last_active":"2024-02-23T00:46:28.711074+0000","last_peered":"2024-02-23T00:46:28.711074+0000","last_clean":"2024-02-23T00:46:28.711074+0000","last_became_active":"2024-02-23T00:41:40.219384+0000","last_became_peered":"2024-02-23T00:41:40.219384+0000","last_unstale":"2024-02-23T00:46:28.711074+0000","last_undegraded":"2024-02-23T00:46:28.711074+0000","last_fullsized":"2024-02-23T00:46:28.711074+0000","mapping_epoch":680,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:46:21.322229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:21.322229+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":904,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472603+0000","last_change":"2024-02-23T00:45:52.411968+0000","last_active":"2024-02-23T00:46:28.472603+0000","last_peered":"2024-02-23T00:46:28.472603+0000","last_clean":"2024-02-23T00:46:28.472603+0000","last_became_active":"2024-02-23T00:41:37.219000+0000","last_became_peered":"2024-02-23T00:41:37.219000+0000","last_unstale":"2024-02-23T00:46:28.472603+0000","last_undegraded":"2024-02-23T00:46:28.472603+0000","last_fullsized":"2024-02-23T00:46:28.472603+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:52.411836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:33.289752+0000","last_clean_scrub_stamp":"2024-02-23T00:45:52.411836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":1104,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710943+0000","last_change":"2024-02-23T00:45:14.546523+0000","last_active":"2024-02-23T00:46:28.710943+0000","last_peered":"2024-02-23T00:46:28.710943+0000","last_clean":"2024-02-23T00:46:28.710943+0000","last_became_active":"2024-02-23T00:42:39.401599+0000","last_became_peered":"2024-02-23T00:42:39.401599+0000","last_unstale":"2024-02-23T00:46:28.710943+0000","last_undegraded":"2024-02-23T00:46:28.710943+0000","last_fullsized":"2024-02-23T00:46:28.710943+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:45:14.546400+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:45:14.546400+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"62'1","reported_seq":905,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:33.612024+0000","last_change":"2024-02-23T00:46:33.612024+0000","last_active":"2024-02-23T00:46:33.612024+0000","last_peered":"2024-02-23T00:46:33.612024+0000","last_clean":"2024-02-23T00:46:33.612024+0000","last_became_active":"2024-02-23T00:41:21.226819+0000","last_became_peered":"2024-02-23T00:41:21.226819+0000","last_unstale":"2024-02-23T00:46:33.612024+0000","last_undegraded":"2024-02-23T00:46:33.612024+0000","last_fullsized":"2024-02-23T00:46:33.612024+0000","mapping_epoch":511,"log_start":"62'1","ondisk_log_start":"62'1","created":60,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:46:33.611875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:46:33.611875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":808,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473553+0000","last_change":"2024-02-23T00:45:58.314436+0000","last_active":"2024-02-23T00:46:28.473553+0000","last_peered":"2024-02-23T00:46:28.473553+0000","last_clean":"2024-02-23T00:46:28.473553+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:46:28.473553+0000","last_undegraded":"2024-02-23T00:46:28.473553+0000","last_fullsized":"2024-02-23T00:46:28.473553+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:58.314366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:58.314366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":784,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471409+0000","last_change":"2024-02-23T00:45:31.450822+0000","last_active":"2024-02-23T00:46:28.471409+0000","last_peered":"2024-02-23T00:46:28.471409+0000","last_clean":"2024-02-23T00:46:28.471409+0000","last_became_active":"2024-02-23T00:45:11.601050+0000","last_became_peered":"2024-02-23T00:45:11.601050+0000","last_unstale":"2024-02-23T00:46:28.471409+0000","last_undegraded":"2024-02-23T00:46:28.471409+0000","last_fullsized":"2024-02-23T00:46:28.471409+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:31.450749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:31.450749+0000","last_clean_scrub_stamp":"2024-02-23T00:45:31.450749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":207,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710719+0000","last_change":"2024-02-23T00:45:48.073661+0000","last_active":"2024-02-23T00:46:28.710719+0000","last_peered":"2024-02-23T00:46:28.710719+0000","last_clean":"2024-02-23T00:46:28.710719+0000","last_became_active":"2024-02-23T00:41:37.239608+0000","last_became_peered":"2024-02-23T00:41:37.239608+0000","last_unstale":"2024-02-23T00:46:28.710719+0000","last_undegraded":"2024-02-23T00:46:28.710719+0000","last_fullsized":"2024-02-23T00:46:28.710719+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:48.073592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:09.610960+0000","last_clean_scrub_stamp":"2024-02-23T00:45:48.073592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":1222,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.551992+0000","last_change":"2024-02-23T00:46:12.341957+0000","last_active":"2024-02-23T00:46:35.551992+0000","last_peered":"2024-02-23T00:46:35.551992+0000","last_clean":"2024-02-23T00:46:35.551992+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:46:35.551992+0000","last_undegraded":"2024-02-23T00:46:35.551992+0000","last_fullsized":"2024-02-23T00:46:35.551992+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'48","last_scrub_stamp":"2024-02-23T00:46:12.341836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:12.341836+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"601'2","reported_seq":244,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614155+0000","last_change":"2024-02-23T00:45:59.110227+0000","last_active":"2024-02-23T00:46:28.614155+0000","last_peered":"2024-02-23T00:46:28.614155+0000","last_clean":"2024-02-23T00:46:28.614155+0000","last_became_active":"2024-02-23T00:45:19.644956+0000","last_became_peered":"2024-02-23T00:45:19.644956+0000","last_unstale":"2024-02-23T00:46:28.614155+0000","last_undegraded":"2024-02-23T00:46:28.614155+0000","last_fullsized":"2024-02-23T00:46:28.614155+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'2","last_scrub_stamp":"2024-02-23T00:45:59.110136+0000","last_deep_scrub":"601'2","last_deep_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_clean_scrub_stamp":"2024-02-23T00:45:59.110136+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":410,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:29.328381+0000","last_change":"2024-02-23T00:46:29.328359+0000","last_active":"2024-02-23T00:46:29.328381+0000","last_peered":"2024-02-23T00:46:29.328381+0000","last_clean":"2024-02-23T00:46:29.328381+0000","last_became_active":"2024-02-23T00:41:17.220294+0000","last_became_peered":"2024-02-23T00:41:17.220294+0000","last_unstale":"2024-02-23T00:46:29.328381+0000","last_undegraded":"2024-02-23T00:46:29.328381+0000","last_fullsized":"2024-02-23T00:46:29.328381+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:29.328276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:46:29.328276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":808,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473874+0000","last_change":"2024-02-23T00:45:18.239785+0000","last_active":"2024-02-23T00:46:28.473874+0000","last_peered":"2024-02-23T00:46:28.473874+0000","last_clean":"2024-02-23T00:46:28.473874+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:46:28.473874+0000","last_undegraded":"2024-02-23T00:46:28.473874+0000","last_fullsized":"2024-02-23T00:46:28.473874+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:18.239715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:18.239715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":782,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614561+0000","last_change":"2024-02-23T00:46:03.077343+0000","last_active":"2024-02-23T00:46:28.614561+0000","last_peered":"2024-02-23T00:46:28.614561+0000","last_clean":"2024-02-23T00:46:28.614561+0000","last_became_active":"2024-02-23T00:42:39.399113+0000","last_became_peered":"2024-02-23T00:42:39.399113+0000","last_unstale":"2024-02-23T00:46:28.614561+0000","last_undegraded":"2024-02-23T00:46:28.614561+0000","last_fullsized":"2024-02-23T00:46:28.614561+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:03.077257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:46:03.077257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"783'22","reported_seq":494,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:32.399409+0000","last_change":"2024-02-23T00:46:09.959333+0000","last_active":"2024-02-23T00:46:32.399409+0000","last_peered":"2024-02-23T00:46:32.399409+0000","last_clean":"2024-02-23T00:46:32.399409+0000","last_became_active":"2024-02-23T00:43:40.280015+0000","last_became_peered":"2024-02-23T00:43:40.280015+0000","last_unstale":"2024-02-23T00:46:32.399409+0000","last_undegraded":"2024-02-23T00:46:32.399409+0000","last_fullsized":"2024-02-23T00:46:32.399409+0000","mapping_epoch":782,"log_start":"606'17","ondisk_log_start":"606'17","created":58,"last_epoch_clean":783,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'22","last_scrub_stamp":"2024-02-23T00:46:09.959246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:09.959246+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"832'32","reported_seq":301,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:30.488143+0000","last_change":"2024-02-23T00:46:30.488114+0000","last_active":"2024-02-23T00:46:30.488143+0000","last_peered":"2024-02-23T00:46:30.488143+0000","last_clean":"2024-02-23T00:46:30.488143+0000","last_became_active":"2024-02-23T00:40:23.194277+0000","last_became_peered":"2024-02-23T00:40:23.194277+0000","last_unstale":"2024-02-23T00:46:30.488143+0000","last_undegraded":"2024-02-23T00:46:30.488143+0000","last_fullsized":"2024-02-23T00:46:30.488143+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'32","last_scrub_stamp":"2024-02-23T00:46:30.488040+0000","last_deep_scrub":"832'32","last_deep_scrub_stamp":"2024-02-23T00:45:10.528461+0000","last_clean_scrub_stamp":"2024-02-23T00:46:30.488040+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":866,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710574+0000","last_change":"2024-02-23T00:46:03.254011+0000","last_active":"2024-02-23T00:46:28.710574+0000","last_peered":"2024-02-23T00:46:28.710574+0000","last_clean":"2024-02-23T00:46:28.710574+0000","last_became_active":"2024-02-23T00:41:37.229003+0000","last_became_peered":"2024-02-23T00:41:37.229003+0000","last_unstale":"2024-02-23T00:46:28.710574+0000","last_undegraded":"2024-02-23T00:46:28.710574+0000","last_fullsized":"2024-02-23T00:46:28.710574+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:03.253940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:46:03.253940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":202,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710158+0000","last_change":"2024-02-23T00:44:59.925931+0000","last_active":"2024-02-23T00:46:28.710158+0000","last_peered":"2024-02-23T00:46:28.710158+0000","last_clean":"2024-02-23T00:46:28.710158+0000","last_became_active":"2024-02-23T00:41:37.248459+0000","last_became_peered":"2024-02-23T00:41:37.248459+0000","last_unstale":"2024-02-23T00:46:28.710158+0000","last_undegraded":"2024-02-23T00:46:28.710158+0000","last_fullsized":"2024-02-23T00:46:28.710158+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:59.925826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:44:59.925826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471261+0000","last_change":"2024-02-23T00:46:08.139421+0000","last_active":"2024-02-23T00:46:28.471261+0000","last_peered":"2024-02-23T00:46:28.471261+0000","last_clean":"2024-02-23T00:46:28.471261+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:46:28.471261+0000","last_undegraded":"2024-02-23T00:46:28.471261+0000","last_fullsized":"2024-02-23T00:46:28.471261+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:08.139334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:46:08.139334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616320+0000","last_change":"2024-02-23T00:45:56.714581+0000","last_active":"2024-02-23T00:46:28.616320+0000","last_peered":"2024-02-23T00:46:28.616320+0000","last_clean":"2024-02-23T00:46:28.616320+0000","last_became_active":"2024-02-23T00:42:41.402111+0000","last_became_peered":"2024-02-23T00:42:41.402111+0000","last_unstale":"2024-02-23T00:46:28.616320+0000","last_undegraded":"2024-02-23T00:46:28.616320+0000","last_fullsized":"2024-02-23T00:46:28.616320+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:56.714506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:56.714506+0000","last_clean_scrub_stamp":"2024-02-23T00:45:56.714506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":900,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472111+0000","last_change":"2024-02-23T00:45:57.297956+0000","last_active":"2024-02-23T00:46:28.472111+0000","last_peered":"2024-02-23T00:46:28.472111+0000","last_clean":"2024-02-23T00:46:28.472111+0000","last_became_active":"2024-02-23T00:41:37.239598+0000","last_became_peered":"2024-02-23T00:41:37.239598+0000","last_unstale":"2024-02-23T00:46:28.472111+0000","last_undegraded":"2024-02-23T00:46:28.472111+0000","last_fullsized":"2024-02-23T00:46:28.472111+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:57.297859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_clean_scrub_stamp":"2024-02-23T00:45:57.297859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":924,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471825+0000","last_change":"2024-02-23T00:45:20.364101+0000","last_active":"2024-02-23T00:46:28.471825+0000","last_peered":"2024-02-23T00:46:28.471825+0000","last_clean":"2024-02-23T00:46:28.471825+0000","last_became_active":"2024-02-23T00:45:11.594930+0000","last_became_peered":"2024-02-23T00:45:11.594930+0000","last_unstale":"2024-02-23T00:46:28.471825+0000","last_undegraded":"2024-02-23T00:46:28.471825+0000","last_fullsized":"2024-02-23T00:46:28.471825+0000","mapping_epoch":833,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:45:20.364016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:20.364016+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":903,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472294+0000","last_change":"2024-02-23T00:46:06.112588+0000","last_active":"2024-02-23T00:46:28.472294+0000","last_peered":"2024-02-23T00:46:28.472294+0000","last_clean":"2024-02-23T00:46:28.472294+0000","last_became_active":"2024-02-23T00:41:37.238708+0000","last_became_peered":"2024-02-23T00:41:37.238708+0000","last_unstale":"2024-02-23T00:46:28.472294+0000","last_undegraded":"2024-02-23T00:46:28.472294+0000","last_fullsized":"2024-02-23T00:46:28.472294+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:46:06.112489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:06.112489+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":387,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472792+0000","last_change":"2024-02-23T00:45:42.329411+0000","last_active":"2024-02-23T00:46:28.472792+0000","last_peered":"2024-02-23T00:46:28.472792+0000","last_clean":"2024-02-23T00:46:28.472792+0000","last_became_active":"2024-02-23T00:40:17.214288+0000","last_became_peered":"2024-02-23T00:40:17.214288+0000","last_unstale":"2024-02-23T00:46:28.472792+0000","last_undegraded":"2024-02-23T00:46:28.472792+0000","last_fullsized":"2024-02-23T00:46:28.472792+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:42.329312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:45:42.329312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":789,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.613866+0000","last_change":"2024-02-23T00:45:23.045215+0000","last_active":"2024-02-23T00:46:28.613866+0000","last_peered":"2024-02-23T00:46:28.613866+0000","last_clean":"2024-02-23T00:46:28.613866+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:46:28.613866+0000","last_undegraded":"2024-02-23T00:46:28.613866+0000","last_fullsized":"2024-02-23T00:46:28.613866+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:23.045097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:23.045097+0000","last_clean_scrub_stamp":"2024-02-23T00:45:23.045097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615385+0000","last_change":"2024-02-23T00:46:21.617550+0000","last_active":"2024-02-23T00:46:28.615385+0000","last_peered":"2024-02-23T00:46:28.615385+0000","last_clean":"2024-02-23T00:46:28.615385+0000","last_became_active":"2024-02-23T00:42:41.399155+0000","last_became_peered":"2024-02-23T00:42:41.399155+0000","last_unstale":"2024-02-23T00:46:28.615385+0000","last_undegraded":"2024-02-23T00:46:28.615385+0000","last_fullsized":"2024-02-23T00:46:28.615385+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:21.617465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:21.617465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":638,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711182+0000","last_change":"2024-02-23T00:45:07.827481+0000","last_active":"2024-02-23T00:46:28.711182+0000","last_peered":"2024-02-23T00:46:28.711182+0000","last_clean":"2024-02-23T00:46:28.711182+0000","last_became_active":"2024-02-23T00:41:38.217979+0000","last_became_peered":"2024-02-23T00:41:38.217979+0000","last_unstale":"2024-02-23T00:46:28.711182+0000","last_undegraded":"2024-02-23T00:46:28.711182+0000","last_fullsized":"2024-02-23T00:46:28.711182+0000","mapping_epoch":678,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:45:07.827410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:07.827410+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":880,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.160114+0000","last_change":"2024-02-23T00:46:35.160114+0000","last_active":"2024-02-23T00:46:35.160114+0000","last_peered":"2024-02-23T00:46:35.160114+0000","last_clean":"2024-02-23T00:46:35.160114+0000","last_became_active":"2024-02-23T00:41:37.238750+0000","last_became_peered":"2024-02-23T00:41:37.238750+0000","last_unstale":"2024-02-23T00:46:35.160114+0000","last_undegraded":"2024-02-23T00:46:35.160114+0000","last_fullsized":"2024-02-23T00:46:35.160114+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:35.160020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:35.160020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":374,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472865+0000","last_change":"2024-02-23T00:46:24.270022+0000","last_active":"2024-02-23T00:46:28.472865+0000","last_peered":"2024-02-23T00:46:28.472865+0000","last_clean":"2024-02-23T00:46:28.472865+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:46:28.472865+0000","last_undegraded":"2024-02-23T00:46:28.472865+0000","last_fullsized":"2024-02-23T00:46:28.472865+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:24.269957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:24.269957+0000","last_clean_scrub_stamp":"2024-02-23T00:46:24.269957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":369,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472835+0000","last_change":"2024-02-23T00:46:21.255605+0000","last_active":"2024-02-23T00:46:28.472835+0000","last_peered":"2024-02-23T00:46:28.472835+0000","last_clean":"2024-02-23T00:46:28.472835+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:46:28.472835+0000","last_undegraded":"2024-02-23T00:46:28.472835+0000","last_fullsized":"2024-02-23T00:46:28.472835+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:21.255543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:02.753328+0000","last_clean_scrub_stamp":"2024-02-23T00:46:21.255543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":268,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710268+0000","last_change":"2024-02-23T00:45:55.446110+0000","last_active":"2024-02-23T00:46:28.710268+0000","last_peered":"2024-02-23T00:46:28.710268+0000","last_clean":"2024-02-23T00:46:28.710268+0000","last_became_active":"2024-02-23T00:42:41.406233+0000","last_became_peered":"2024-02-23T00:42:41.406233+0000","last_unstale":"2024-02-23T00:46:28.710268+0000","last_undegraded":"2024-02-23T00:46:28.710268+0000","last_fullsized":"2024-02-23T00:46:28.710268+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:45:55.446002+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:43:07.690939+0000","last_clean_scrub_stamp":"2024-02-23T00:45:55.446002+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":977,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472512+0000","last_change":"2024-02-23T00:45:20.573104+0000","last_active":"2024-02-23T00:46:28.472512+0000","last_peered":"2024-02-23T00:46:28.472512+0000","last_clean":"2024-02-23T00:46:28.472512+0000","last_became_active":"2024-02-23T00:40:51.316452+0000","last_became_peered":"2024-02-23T00:40:51.316452+0000","last_unstale":"2024-02-23T00:46:28.472512+0000","last_undegraded":"2024-02-23T00:46:28.472512+0000","last_fullsized":"2024-02-23T00:46:28.472512+0000","mapping_epoch":631,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:45:20.573002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:20.573002+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":858,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:38.373376+0000","last_change":"2024-02-23T00:46:38.373376+0000","last_active":"2024-02-23T00:46:38.373376+0000","last_peered":"2024-02-23T00:46:38.373376+0000","last_clean":"2024-02-23T00:46:38.373376+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:46:38.373376+0000","last_undegraded":"2024-02-23T00:46:38.373376+0000","last_fullsized":"2024-02-23T00:46:38.373376+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:38.373274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:46:38.373274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":854,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:29.634920+0000","last_change":"2024-02-23T00:46:29.634886+0000","last_active":"2024-02-23T00:46:29.634920+0000","last_peered":"2024-02-23T00:46:29.634920+0000","last_clean":"2024-02-23T00:46:29.634920+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:46:29.634920+0000","last_undegraded":"2024-02-23T00:46:29.634920+0000","last_fullsized":"2024-02-23T00:46:29.634920+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:29.634812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:29.634812+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":818,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711019+0000","last_change":"2024-02-23T00:45:57.371599+0000","last_active":"2024-02-23T00:46:28.711019+0000","last_peered":"2024-02-23T00:46:28.711019+0000","last_clean":"2024-02-23T00:46:28.711019+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:46:28.711019+0000","last_undegraded":"2024-02-23T00:46:28.711019+0000","last_fullsized":"2024-02-23T00:46:28.711019+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:57.371524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:32.101913+0000","last_clean_scrub_stamp":"2024-02-23T00:45:57.371524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615909+0000","last_change":"2024-02-23T00:46:00.681933+0000","last_active":"2024-02-23T00:46:28.615909+0000","last_peered":"2024-02-23T00:46:28.615909+0000","last_clean":"2024-02-23T00:46:28.615909+0000","last_became_active":"2024-02-23T00:42:41.402543+0000","last_became_peered":"2024-02-23T00:42:41.402543+0000","last_unstale":"2024-02-23T00:46:28.615909+0000","last_undegraded":"2024-02-23T00:46:28.615909+0000","last_fullsized":"2024-02-23T00:46:28.615909+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:00.681827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.681827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"688'39","reported_seq":392,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472693+0000","last_change":"2024-02-23T00:46:16.129153+0000","last_active":"2024-02-23T00:46:28.472693+0000","last_peered":"2024-02-23T00:46:28.472693+0000","last_clean":"2024-02-23T00:46:28.472693+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:46:28.472693+0000","last_undegraded":"2024-02-23T00:46:28.472693+0000","last_fullsized":"2024-02-23T00:46:28.472693+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'39","last_scrub_stamp":"2024-02-23T00:46:16.129079+0000","last_deep_scrub":"688'39","last_deep_scrub_stamp":"2024-02-23T00:44:57.844469+0000","last_clean_scrub_stamp":"2024-02-23T00:46:16.129079+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614404+0000","last_change":"2024-02-23T00:46:28.613906+0000","last_active":"2024-02-23T00:46:28.614404+0000","last_peered":"2024-02-23T00:46:28.614404+0000","last_clean":"2024-02-23T00:46:28.614404+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:46:28.614404+0000","last_undegraded":"2024-02-23T00:46:28.614404+0000","last_fullsized":"2024-02-23T00:46:28.614404+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:28.613768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:46:28.613768+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"681'12","reported_seq":1106,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.552227+0000","last_change":"2024-02-23T00:46:00.062864+0000","last_active":"2024-02-23T00:46:35.552227+0000","last_peered":"2024-02-23T00:46:35.552227+0000","last_clean":"2024-02-23T00:46:35.552227+0000","last_became_active":"2024-02-23T00:41:37.228441+0000","last_became_peered":"2024-02-23T00:41:37.228441+0000","last_unstale":"2024-02-23T00:46:35.552227+0000","last_undegraded":"2024-02-23T00:46:35.552227+0000","last_fullsized":"2024-02-23T00:46:35.552227+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'12","last_scrub_stamp":"2024-02-23T00:46:00.062784+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.062784+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":789,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614966+0000","last_change":"2024-02-23T00:45:28.135319+0000","last_active":"2024-02-23T00:46:28.614966+0000","last_peered":"2024-02-23T00:46:28.614966+0000","last_clean":"2024-02-23T00:46:28.614966+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:46:28.614966+0000","last_undegraded":"2024-02-23T00:46:28.614966+0000","last_fullsized":"2024-02-23T00:46:28.614966+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:28.135231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:28.135231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":141,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710240+0000","last_change":"2024-02-23T00:45:49.038011+0000","last_active":"2024-02-23T00:46:28.710240+0000","last_peered":"2024-02-23T00:46:28.710240+0000","last_clean":"2024-02-23T00:46:28.710240+0000","last_became_active":"2024-02-23T00:42:41.402683+0000","last_became_peered":"2024-02-23T00:42:41.402683+0000","last_unstale":"2024-02-23T00:46:28.710240+0000","last_undegraded":"2024-02-23T00:46:28.710240+0000","last_fullsized":"2024-02-23T00:46:28.710240+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:49.037943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:49.037943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":925,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614685+0000","last_change":"2024-02-23T00:45:57.117759+0000","last_active":"2024-02-23T00:46:28.614685+0000","last_peered":"2024-02-23T00:46:28.614685+0000","last_clean":"2024-02-23T00:46:28.614685+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:46:28.614685+0000","last_undegraded":"2024-02-23T00:46:28.614685+0000","last_fullsized":"2024-02-23T00:46:28.614685+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:45:57.117677+0000","last_deep_scrub":"453'26","last_deep_scrub_stamp":"2024-02-23T00:45:57.117677+0000","last_clean_scrub_stamp":"2024-02-23T00:45:57.117677+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":207,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711406+0000","last_change":"2024-02-23T00:46:19.987965+0000","last_active":"2024-02-23T00:46:28.711406+0000","last_peered":"2024-02-23T00:46:28.711406+0000","last_clean":"2024-02-23T00:46:28.711406+0000","last_became_active":"2024-02-23T00:41:37.235746+0000","last_became_peered":"2024-02-23T00:41:37.235746+0000","last_unstale":"2024-02-23T00:46:28.711406+0000","last_undegraded":"2024-02-23T00:46:28.711406+0000","last_fullsized":"2024-02-23T00:46:28.711406+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:19.987886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:53.903007+0000","last_clean_scrub_stamp":"2024-02-23T00:46:19.987886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":468,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.552144+0000","last_change":"2024-02-23T00:46:05.129838+0000","last_active":"2024-02-23T00:46:35.552144+0000","last_peered":"2024-02-23T00:46:35.552144+0000","last_clean":"2024-02-23T00:46:35.552144+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:46:35.552144+0000","last_undegraded":"2024-02-23T00:46:35.552144+0000","last_fullsized":"2024-02-23T00:46:35.552144+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'11","last_scrub_stamp":"2024-02-23T00:46:05.129760+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:46:05.129760+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":371,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472920+0000","last_change":"2024-02-23T00:46:04.136581+0000","last_active":"2024-02-23T00:46:28.472920+0000","last_peered":"2024-02-23T00:46:28.472920+0000","last_clean":"2024-02-23T00:46:28.472920+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:46:28.472920+0000","last_undegraded":"2024-02-23T00:46:28.472920+0000","last_fullsized":"2024-02-23T00:46:28.472920+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:04.136496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:04.136496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616138+0000","last_change":"2024-02-23T00:45:45.751087+0000","last_active":"2024-02-23T00:46:28.616138+0000","last_peered":"2024-02-23T00:46:28.616138+0000","last_clean":"2024-02-23T00:46:28.616138+0000","last_became_active":"2024-02-23T00:42:41.398941+0000","last_became_peered":"2024-02-23T00:42:41.398941+0000","last_unstale":"2024-02-23T00:46:28.616138+0000","last_undegraded":"2024-02-23T00:46:28.616138+0000","last_fullsized":"2024-02-23T00:46:28.616138+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:45.751013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:45:45.751013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":867,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711632+0000","last_change":"2024-02-23T00:46:27.416846+0000","last_active":"2024-02-23T00:46:28.711632+0000","last_peered":"2024-02-23T00:46:28.711632+0000","last_clean":"2024-02-23T00:46:28.711632+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:46:28.711632+0000","last_undegraded":"2024-02-23T00:46:28.711632+0000","last_fullsized":"2024-02-23T00:46:28.711632+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'15","last_scrub_stamp":"2024-02-23T00:46:27.416778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:27.416778+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":880,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710828+0000","last_change":"2024-02-23T00:46:10.362620+0000","last_active":"2024-02-23T00:46:28.710828+0000","last_peered":"2024-02-23T00:46:28.710828+0000","last_clean":"2024-02-23T00:46:28.710828+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:46:28.710828+0000","last_undegraded":"2024-02-23T00:46:28.710828+0000","last_fullsized":"2024-02-23T00:46:28.710828+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:10.362544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:10.362544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":393,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:30.346073+0000","last_change":"2024-02-23T00:46:30.346046+0000","last_active":"2024-02-23T00:46:30.346073+0000","last_peered":"2024-02-23T00:46:30.346073+0000","last_clean":"2024-02-23T00:46:30.346073+0000","last_became_active":"2024-02-23T00:41:37.252612+0000","last_became_peered":"2024-02-23T00:41:37.252612+0000","last_unstale":"2024-02-23T00:46:30.346073+0000","last_undegraded":"2024-02-23T00:46:30.346073+0000","last_fullsized":"2024-02-23T00:46:30.346073+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:30.345975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_clean_scrub_stamp":"2024-02-23T00:46:30.345975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":818,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471138+0000","last_change":"2024-02-23T00:44:56.232184+0000","last_active":"2024-02-23T00:46:28.471138+0000","last_peered":"2024-02-23T00:46:28.471138+0000","last_clean":"2024-02-23T00:46:28.471138+0000","last_became_active":"2024-02-23T00:41:37.238185+0000","last_became_peered":"2024-02-23T00:41:37.238185+0000","last_unstale":"2024-02-23T00:46:28.471138+0000","last_undegraded":"2024-02-23T00:46:28.471138+0000","last_fullsized":"2024-02-23T00:46:28.471138+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:56.232109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:44:56.232109+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"78'6","reported_seq":41,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471481+0000","last_change":"2024-02-23T00:45:29.458259+0000","last_active":"2024-02-23T00:46:28.471481+0000","last_peered":"2024-02-23T00:46:28.471481+0000","last_clean":"2024-02-23T00:46:28.471481+0000","last_became_active":"2024-02-23T00:45:11.600802+0000","last_became_peered":"2024-02-23T00:45:11.600802+0000","last_unstale":"2024-02-23T00:46:28.471481+0000","last_undegraded":"2024-02-23T00:46:28.471481+0000","last_fullsized":"2024-02-23T00:46:28.471481+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:45:29.458182+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:45:29.458182+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":902,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471560+0000","last_change":"2024-02-23T00:46:19.023703+0000","last_active":"2024-02-23T00:46:28.471560+0000","last_peered":"2024-02-23T00:46:28.471560+0000","last_clean":"2024-02-23T00:46:28.471560+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:46:28.471560+0000","last_undegraded":"2024-02-23T00:46:28.471560+0000","last_fullsized":"2024-02-23T00:46:28.471560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:46:19.023606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:19.023606+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":863,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710646+0000","last_change":"2024-02-23T00:46:28.433283+0000","last_active":"2024-02-23T00:46:28.710646+0000","last_peered":"2024-02-23T00:46:28.710646+0000","last_clean":"2024-02-23T00:46:28.710646+0000","last_became_active":"2024-02-23T00:41:37.229726+0000","last_became_peered":"2024-02-23T00:41:37.229726+0000","last_unstale":"2024-02-23T00:46:28.710646+0000","last_undegraded":"2024-02-23T00:46:28.710646+0000","last_fullsized":"2024-02-23T00:46:28.710646+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:28.433175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:46:28.433175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":853,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711644+0000","last_change":"2024-02-23T00:45:33.410727+0000","last_active":"2024-02-23T00:46:28.711644+0000","last_peered":"2024-02-23T00:46:28.711644+0000","last_clean":"2024-02-23T00:46:28.711644+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:46:28.711644+0000","last_undegraded":"2024-02-23T00:46:28.711644+0000","last_fullsized":"2024-02-23T00:46:28.711644+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:33.410635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:45:33.410635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"855'8","reported_seq":850,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710837+0000","last_change":"2024-02-23T00:46:27.306881+0000","last_active":"2024-02-23T00:46:28.710837+0000","last_peered":"2024-02-23T00:46:28.710837+0000","last_clean":"2024-02-23T00:46:28.710837+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:46:28.710837+0000","last_undegraded":"2024-02-23T00:46:28.710837+0000","last_fullsized":"2024-02-23T00:46:28.710837+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"855'8","last_scrub_stamp":"2024-02-23T00:46:27.306802+0000","last_deep_scrub":"855'8","last_deep_scrub_stamp":"2024-02-23T00:46:27.306802+0000","last_clean_scrub_stamp":"2024-02-23T00:46:27.306802+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":395,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:39.338322+0000","last_change":"2024-02-23T00:46:39.338322+0000","last_active":"2024-02-23T00:46:39.338322+0000","last_peered":"2024-02-23T00:46:39.338322+0000","last_clean":"2024-02-23T00:46:39.338322+0000","last_became_active":"2024-02-23T00:43:42.256788+0000","last_became_peered":"2024-02-23T00:43:42.256788+0000","last_unstale":"2024-02-23T00:46:39.338322+0000","last_undegraded":"2024-02-23T00:46:39.338322+0000","last_fullsized":"2024-02-23T00:46:39.338322+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'10","last_scrub_stamp":"2024-02-23T00:46:39.338207+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:46:39.338207+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":840,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614830+0000","last_change":"2024-02-23T00:45:22.052575+0000","last_active":"2024-02-23T00:46:28.614830+0000","last_peered":"2024-02-23T00:46:28.614830+0000","last_clean":"2024-02-23T00:46:28.614830+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:46:28.614830+0000","last_undegraded":"2024-02-23T00:46:28.614830+0000","last_fullsized":"2024-02-23T00:46:28.614830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:22.052497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:22.052497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":849,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710576+0000","last_change":"2024-02-23T00:45:49.313466+0000","last_active":"2024-02-23T00:46:28.710576+0000","last_peered":"2024-02-23T00:46:28.710576+0000","last_clean":"2024-02-23T00:46:28.710576+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:46:28.710576+0000","last_undegraded":"2024-02-23T00:46:28.710576+0000","last_fullsized":"2024-02-23T00:46:28.710576+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:49.313391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:49.313391+0000","last_clean_scrub_stamp":"2024-02-23T00:45:49.313391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":786,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711712+0000","last_change":"2024-02-23T00:46:01.293672+0000","last_active":"2024-02-23T00:46:28.711712+0000","last_peered":"2024-02-23T00:46:28.711712+0000","last_clean":"2024-02-23T00:46:28.711712+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:46:28.711712+0000","last_undegraded":"2024-02-23T00:46:28.711712+0000","last_fullsized":"2024-02-23T00:46:28.711712+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:01.293605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:01.293605+0000","last_clean_scrub_stamp":"2024-02-23T00:46:01.293605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614569+0000","last_change":"2024-02-23T00:46:20.701211+0000","last_active":"2024-02-23T00:46:28.614569+0000","last_peered":"2024-02-23T00:46:28.614569+0000","last_clean":"2024-02-23T00:46:28.614569+0000","last_became_active":"2024-02-23T00:42:41.391484+0000","last_became_peered":"2024-02-23T00:42:41.391484+0000","last_unstale":"2024-02-23T00:46:28.614569+0000","last_undegraded":"2024-02-23T00:46:28.614569+0000","last_fullsized":"2024-02-23T00:46:28.614569+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:20.701134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:17.077581+0000","last_clean_scrub_stamp":"2024-02-23T00:46:20.701134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":386,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473332+0000","last_change":"2024-02-23T00:45:32.492526+0000","last_active":"2024-02-23T00:46:28.473332+0000","last_peered":"2024-02-23T00:46:28.473332+0000","last_clean":"2024-02-23T00:46:28.473332+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:46:28.473332+0000","last_undegraded":"2024-02-23T00:46:28.473332+0000","last_fullsized":"2024-02-23T00:46:28.473332+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:45:32.492427+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:45:32.492427+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":857,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:30.313531+0000","last_change":"2024-02-23T00:46:30.313487+0000","last_active":"2024-02-23T00:46:30.313531+0000","last_peered":"2024-02-23T00:46:30.313531+0000","last_clean":"2024-02-23T00:46:30.313531+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:46:30.313531+0000","last_undegraded":"2024-02-23T00:46:30.313531+0000","last_fullsized":"2024-02-23T00:46:30.313531+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:30.313398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:25.724375+0000","last_clean_scrub_stamp":"2024-02-23T00:46:30.313398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":106,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473020+0000","last_change":"2024-02-23T00:46:18.131373+0000","last_active":"2024-02-23T00:46:28.473020+0000","last_peered":"2024-02-23T00:46:28.473020+0000","last_clean":"2024-02-23T00:46:28.473020+0000","last_became_active":"2024-02-23T00:43:21.498988+0000","last_became_peered":"2024-02-23T00:43:21.498988+0000","last_unstale":"2024-02-23T00:46:28.473020+0000","last_undegraded":"2024-02-23T00:46:28.473020+0000","last_fullsized":"2024-02-23T00:46:28.473020+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:18.131285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:46:18.131285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":372,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471818+0000","last_change":"2024-02-23T00:45:11.600687+0000","last_active":"2024-02-23T00:46:28.471818+0000","last_peered":"2024-02-23T00:46:28.471818+0000","last_clean":"2024-02-23T00:46:28.471818+0000","last_became_active":"2024-02-23T00:45:11.600302+0000","last_became_peered":"2024-02-23T00:45:11.600302+0000","last_unstale":"2024-02-23T00:46:28.471818+0000","last_undegraded":"2024-02-23T00:46:28.471818+0000","last_fullsized":"2024-02-23T00:46:28.471818+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:59.793161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:59.793161+0000","last_clean_scrub_stamp":"2024-02-23T00:44:59.793161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":762,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711051+0000","last_change":"2024-02-23T00:46:00.034799+0000","last_active":"2024-02-23T00:46:28.711051+0000","last_peered":"2024-02-23T00:46:28.711051+0000","last_clean":"2024-02-23T00:46:28.711051+0000","last_became_active":"2024-02-23T00:42:10.240911+0000","last_became_peered":"2024-02-23T00:42:10.240911+0000","last_unstale":"2024-02-23T00:46:28.711051+0000","last_undegraded":"2024-02-23T00:46:28.711051+0000","last_fullsized":"2024-02-23T00:46:28.711051+0000","mapping_epoch":707,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:46:00.034676+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.034676+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":849,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.613741+0000","last_change":"2024-02-23T00:46:27.648224+0000","last_active":"2024-02-23T00:46:28.613741+0000","last_peered":"2024-02-23T00:46:28.613741+0000","last_clean":"2024-02-23T00:46:28.613741+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:46:28.613741+0000","last_undegraded":"2024-02-23T00:46:28.613741+0000","last_fullsized":"2024-02-23T00:46:28.613741+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:27.648147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:55.816155+0000","last_clean_scrub_stamp":"2024-02-23T00:46:27.648147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":1093,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.552403+0000","last_change":"2024-02-23T00:45:25.344165+0000","last_active":"2024-02-23T00:46:35.552403+0000","last_peered":"2024-02-23T00:46:35.552403+0000","last_clean":"2024-02-23T00:46:35.552403+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:46:35.552403+0000","last_undegraded":"2024-02-23T00:46:35.552403+0000","last_fullsized":"2024-02-23T00:46:35.552403+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:45:25.344058+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:45:25.344058+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":838,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616193+0000","last_change":"2024-02-23T00:45:44.731947+0000","last_active":"2024-02-23T00:46:28.616193+0000","last_peered":"2024-02-23T00:46:28.616193+0000","last_clean":"2024-02-23T00:46:28.616193+0000","last_became_active":"2024-02-23T00:41:37.229426+0000","last_became_peered":"2024-02-23T00:41:37.229426+0000","last_unstale":"2024-02-23T00:46:28.616193+0000","last_undegraded":"2024-02-23T00:46:28.616193+0000","last_fullsized":"2024-02-23T00:46:28.616193+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:44.731870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:44.731870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":1061,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.349673+0000","last_change":"2024-02-23T00:46:35.349673+0000","last_active":"2024-02-23T00:46:35.349673+0000","last_peered":"2024-02-23T00:46:35.349673+0000","last_clean":"2024-02-23T00:46:35.349673+0000","last_became_active":"2024-02-23T00:41:55.327964+0000","last_became_peered":"2024-02-23T00:41:55.327964+0000","last_unstale":"2024-02-23T00:46:35.349673+0000","last_undegraded":"2024-02-23T00:46:35.349673+0000","last_fullsized":"2024-02-23T00:46:35.349673+0000","mapping_epoch":692,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:46:35.349575+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:46:35.349575+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":393,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:37.326399+0000","last_change":"2024-02-23T00:46:37.326399+0000","last_active":"2024-02-23T00:46:37.326399+0000","last_peered":"2024-02-23T00:46:37.326399+0000","last_clean":"2024-02-23T00:46:37.326399+0000","last_became_active":"2024-02-23T00:41:37.228694+0000","last_became_peered":"2024-02-23T00:41:37.228694+0000","last_unstale":"2024-02-23T00:46:37.326399+0000","last_undegraded":"2024-02-23T00:46:37.326399+0000","last_fullsized":"2024-02-23T00:46:37.326399+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:37.326279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:46:37.326279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":202,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711096+0000","last_change":"2024-02-23T00:46:06.945114+0000","last_active":"2024-02-23T00:46:28.711096+0000","last_peered":"2024-02-23T00:46:28.711096+0000","last_clean":"2024-02-23T00:46:28.711096+0000","last_became_active":"2024-02-23T00:41:37.247946+0000","last_became_peered":"2024-02-23T00:41:37.247946+0000","last_unstale":"2024-02-23T00:46:28.711096+0000","last_undegraded":"2024-02-23T00:46:28.711096+0000","last_fullsized":"2024-02-23T00:46:28.711096+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:06.945049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:06.945049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:31.684586+0000","last_change":"2024-02-23T00:46:31.684540+0000","last_active":"2024-02-23T00:46:31.684586+0000","last_peered":"2024-02-23T00:46:31.684586+0000","last_clean":"2024-02-23T00:46:31.684586+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:46:31.684586+0000","last_undegraded":"2024-02-23T00:46:31.684586+0000","last_fullsized":"2024-02-23T00:46:31.684586+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:31.684458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:31.684458+0000","last_clean_scrub_stamp":"2024-02-23T00:46:31.684458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":991,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472990+0000","last_change":"2024-02-23T00:46:11.382604+0000","last_active":"2024-02-23T00:46:28.472990+0000","last_peered":"2024-02-23T00:46:28.472990+0000","last_clean":"2024-02-23T00:46:28.472990+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:46:28.472990+0000","last_undegraded":"2024-02-23T00:46:28.472990+0000","last_fullsized":"2024-02-23T00:46:28.472990+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:46:11.382537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.382537+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":861,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.410548+0000","last_change":"2024-02-23T00:46:40.410548+0000","last_active":"2024-02-23T00:46:40.410548+0000","last_peered":"2024-02-23T00:46:40.410548+0000","last_clean":"2024-02-23T00:46:40.410548+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:46:40.410548+0000","last_undegraded":"2024-02-23T00:46:40.410548+0000","last_fullsized":"2024-02-23T00:46:40.410548+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:40.410474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:46:40.410474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":861,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:32.445988+0000","last_change":"2024-02-23T00:46:32.445988+0000","last_active":"2024-02-23T00:46:32.445988+0000","last_peered":"2024-02-23T00:46:32.445988+0000","last_clean":"2024-02-23T00:46:32.445988+0000","last_became_active":"2024-02-23T00:41:37.229323+0000","last_became_peered":"2024-02-23T00:41:37.229323+0000","last_unstale":"2024-02-23T00:46:32.445988+0000","last_undegraded":"2024-02-23T00:46:32.445988+0000","last_fullsized":"2024-02-23T00:46:32.445988+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:32.445916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:46:32.445916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":365,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614910+0000","last_change":"2024-02-23T00:45:19.684362+0000","last_active":"2024-02-23T00:46:28.614910+0000","last_peered":"2024-02-23T00:46:28.614910+0000","last_clean":"2024-02-23T00:46:28.614910+0000","last_became_active":"2024-02-23T00:45:19.684020+0000","last_became_peered":"2024-02-23T00:45:19.684020+0000","last_unstale":"2024-02-23T00:46:28.614910+0000","last_undegraded":"2024-02-23T00:46:28.614910+0000","last_fullsized":"2024-02-23T00:46:28.614910+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:16.696798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:16.696798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":953,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472435+0000","last_change":"2024-02-23T00:45:33.440720+0000","last_active":"2024-02-23T00:46:28.472435+0000","last_peered":"2024-02-23T00:46:28.472435+0000","last_clean":"2024-02-23T00:46:28.472435+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:46:28.472435+0000","last_undegraded":"2024-02-23T00:46:28.472435+0000","last_fullsized":"2024-02-23T00:46:28.472435+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:45:33.440637+0000","last_deep_scrub":"453'44","last_deep_scrub_stamp":"2024-02-23T00:44:29.944804+0000","last_clean_scrub_stamp":"2024-02-23T00:45:33.440637+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":369,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472876+0000","last_change":"2024-02-23T00:45:26.558887+0000","last_active":"2024-02-23T00:46:28.472876+0000","last_peered":"2024-02-23T00:46:28.472876+0000","last_clean":"2024-02-23T00:46:28.472876+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:46:28.472876+0000","last_undegraded":"2024-02-23T00:46:28.472876+0000","last_fullsized":"2024-02-23T00:46:28.472876+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:26.558811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:26.558811+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":717,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471639+0000","last_change":"2024-02-23T00:46:15.149118+0000","last_active":"2024-02-23T00:46:28.471639+0000","last_peered":"2024-02-23T00:46:28.471639+0000","last_clean":"2024-02-23T00:46:28.471639+0000","last_became_active":"2024-02-23T00:45:11.598288+0000","last_became_peered":"2024-02-23T00:45:11.598288+0000","last_unstale":"2024-02-23T00:46:28.471639+0000","last_undegraded":"2024-02-23T00:46:28.471639+0000","last_fullsized":"2024-02-23T00:46:28.471639+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:15.149033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:15.149033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":789,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614850+0000","last_change":"2024-02-23T00:46:10.872007+0000","last_active":"2024-02-23T00:46:28.614850+0000","last_peered":"2024-02-23T00:46:28.614850+0000","last_clean":"2024-02-23T00:46:28.614850+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:46:28.614850+0000","last_undegraded":"2024-02-23T00:46:28.614850+0000","last_fullsized":"2024-02-23T00:46:28.614850+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:10.871921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:10.871921+0000","last_clean_scrub_stamp":"2024-02-23T00:46:10.871921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":688,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471153+0000","last_change":"2024-02-23T00:45:33.464804+0000","last_active":"2024-02-23T00:46:28.471153+0000","last_peered":"2024-02-23T00:46:28.471153+0000","last_clean":"2024-02-23T00:46:28.471153+0000","last_became_active":"2024-02-23T00:45:11.596036+0000","last_became_peered":"2024-02-23T00:45:11.596036+0000","last_unstale":"2024-02-23T00:46:28.471153+0000","last_undegraded":"2024-02-23T00:46:28.471153+0000","last_fullsized":"2024-02-23T00:46:28.471153+0000","mapping_epoch":833,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:45:33.464727+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:45:33.464727+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":562,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471533+0000","last_change":"2024-02-23T00:46:21.026675+0000","last_active":"2024-02-23T00:46:28.471533+0000","last_peered":"2024-02-23T00:46:28.471533+0000","last_clean":"2024-02-23T00:46:28.471533+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:46:28.471533+0000","last_undegraded":"2024-02-23T00:46:28.471533+0000","last_fullsized":"2024-02-23T00:46:28.471533+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:21.026579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:21.026579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":855,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614104+0000","last_change":"2024-02-23T00:46:16.738414+0000","last_active":"2024-02-23T00:46:28.614104+0000","last_peered":"2024-02-23T00:46:28.614104+0000","last_clean":"2024-02-23T00:46:28.614104+0000","last_became_active":"2024-02-23T00:41:37.228310+0000","last_became_peered":"2024-02-23T00:41:37.228310+0000","last_unstale":"2024-02-23T00:46:28.614104+0000","last_undegraded":"2024-02-23T00:46:28.614104+0000","last_fullsized":"2024-02-23T00:46:28.614104+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:16.738313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:16.738313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615300+0000","last_change":"2024-02-23T00:45:35.760307+0000","last_active":"2024-02-23T00:46:28.615300+0000","last_peered":"2024-02-23T00:46:28.615300+0000","last_clean":"2024-02-23T00:46:28.615300+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:46:28.615300+0000","last_undegraded":"2024-02-23T00:46:28.615300+0000","last_fullsized":"2024-02-23T00:46:28.615300+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:35.760234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:35.760234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":398,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471332+0000","last_change":"2024-02-23T00:45:57.151443+0000","last_active":"2024-02-23T00:46:28.471332+0000","last_peered":"2024-02-23T00:46:28.471332+0000","last_clean":"2024-02-23T00:46:28.471332+0000","last_became_active":"2024-02-23T00:45:11.598687+0000","last_became_peered":"2024-02-23T00:45:11.598687+0000","last_unstale":"2024-02-23T00:46:28.471332+0000","last_undegraded":"2024-02-23T00:46:28.471332+0000","last_fullsized":"2024-02-23T00:46:28.471332+0000","mapping_epoch":833,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:45:57.151362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:57.151362+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616245+0000","last_change":"2024-02-23T00:45:52.712758+0000","last_active":"2024-02-23T00:46:28.616245+0000","last_peered":"2024-02-23T00:46:28.616245+0000","last_clean":"2024-02-23T00:46:28.616245+0000","last_became_active":"2024-02-23T00:41:37.230054+0000","last_became_peered":"2024-02-23T00:41:37.230054+0000","last_unstale":"2024-02-23T00:46:28.616245+0000","last_undegraded":"2024-02-23T00:46:28.616245+0000","last_fullsized":"2024-02-23T00:46:28.616245+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:52.712651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:45:52.712651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":848,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615806+0000","last_change":"2024-02-23T00:45:42.753321+0000","last_active":"2024-02-23T00:46:28.615806+0000","last_peered":"2024-02-23T00:46:28.615806+0000","last_clean":"2024-02-23T00:46:28.615806+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:46:28.615806+0000","last_undegraded":"2024-02-23T00:46:28.615806+0000","last_fullsized":"2024-02-23T00:46:28.615806+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:42.753243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.307631+0000","last_clean_scrub_stamp":"2024-02-23T00:45:42.753243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":810,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710437+0000","last_change":"2024-02-23T00:45:46.392980+0000","last_active":"2024-02-23T00:46:28.710437+0000","last_peered":"2024-02-23T00:46:28.710437+0000","last_clean":"2024-02-23T00:46:28.710437+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:46:28.710437+0000","last_undegraded":"2024-02-23T00:46:28.710437+0000","last_fullsized":"2024-02-23T00:46:28.710437+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:46.392916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:46.392916+0000","last_clean_scrub_stamp":"2024-02-23T00:45:46.392916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":999,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.470942+0000","last_change":"2024-02-23T00:44:58.245037+0000","last_active":"2024-02-23T00:46:28.470942+0000","last_peered":"2024-02-23T00:46:28.470942+0000","last_clean":"2024-02-23T00:46:28.470942+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:46:28.470942+0000","last_undegraded":"2024-02-23T00:46:28.470942+0000","last_fullsized":"2024-02-23T00:46:28.470942+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:44:58.244955+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:44:58.244955+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":393,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472992+0000","last_change":"2024-02-23T00:46:28.291609+0000","last_active":"2024-02-23T00:46:28.472992+0000","last_peered":"2024-02-23T00:46:28.472992+0000","last_clean":"2024-02-23T00:46:28.472992+0000","last_became_active":"2024-02-23T00:41:37.241142+0000","last_became_peered":"2024-02-23T00:41:37.241142+0000","last_unstale":"2024-02-23T00:46:28.472992+0000","last_undegraded":"2024-02-23T00:46:28.472992+0000","last_fullsized":"2024-02-23T00:46:28.472992+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:28.291525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:28.291525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":504,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710364+0000","last_change":"2024-02-23T00:46:00.317428+0000","last_active":"2024-02-23T00:46:28.710364+0000","last_peered":"2024-02-23T00:46:28.710364+0000","last_clean":"2024-02-23T00:46:28.710364+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:46:28.710364+0000","last_undegraded":"2024-02-23T00:46:28.710364+0000","last_fullsized":"2024-02-23T00:46:28.710364+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:00.317340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.317340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":282,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615561+0000","last_change":"2024-02-23T00:46:10.532532+0000","last_active":"2024-02-23T00:46:28.615561+0000","last_peered":"2024-02-23T00:46:28.615561+0000","last_clean":"2024-02-23T00:46:28.615561+0000","last_became_active":"2024-02-23T00:40:23.199068+0000","last_became_peered":"2024-02-23T00:40:23.199068+0000","last_unstale":"2024-02-23T00:46:28.615561+0000","last_undegraded":"2024-02-23T00:46:28.615561+0000","last_fullsized":"2024-02-23T00:46:28.615561+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:10.532448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:00.454077+0000","last_clean_scrub_stamp":"2024-02-23T00:46:10.532448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":1144,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473224+0000","last_change":"2024-02-23T00:46:20.208922+0000","last_active":"2024-02-23T00:46:28.473224+0000","last_peered":"2024-02-23T00:46:28.473224+0000","last_clean":"2024-02-23T00:46:28.473224+0000","last_became_active":"2024-02-23T00:41:41.222892+0000","last_became_peered":"2024-02-23T00:41:41.222892+0000","last_unstale":"2024-02-23T00:46:28.473224+0000","last_undegraded":"2024-02-23T00:46:28.473224+0000","last_fullsized":"2024-02-23T00:46:28.473224+0000","mapping_epoch":681,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:46:20.208806+0000","last_deep_scrub":"453'68","last_deep_scrub_stamp":"2024-02-23T00:46:20.208806+0000","last_clean_scrub_stamp":"2024-02-23T00:46:20.208806+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":202,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711122+0000","last_change":"2024-02-23T00:45:56.990448+0000","last_active":"2024-02-23T00:46:28.711122+0000","last_peered":"2024-02-23T00:46:28.711122+0000","last_clean":"2024-02-23T00:46:28.711122+0000","last_became_active":"2024-02-23T00:41:37.234830+0000","last_became_peered":"2024-02-23T00:41:37.234830+0000","last_unstale":"2024-02-23T00:46:28.711122+0000","last_undegraded":"2024-02-23T00:46:28.711122+0000","last_fullsized":"2024-02-23T00:46:28.711122+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:56.990386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:56.990386+0000","last_clean_scrub_stamp":"2024-02-23T00:45:56.990386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":699,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471155+0000","last_change":"2024-02-23T00:45:41.232189+0000","last_active":"2024-02-23T00:46:28.471155+0000","last_peered":"2024-02-23T00:46:28.471155+0000","last_clean":"2024-02-23T00:46:28.471155+0000","last_became_active":"2024-02-23T00:45:11.601270+0000","last_became_peered":"2024-02-23T00:45:11.601270+0000","last_unstale":"2024-02-23T00:46:28.471155+0000","last_undegraded":"2024-02-23T00:46:28.471155+0000","last_fullsized":"2024-02-23T00:46:28.471155+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:41.232090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:45:41.232090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":809,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710799+0000","last_change":"2024-02-23T00:45:54.240974+0000","last_active":"2024-02-23T00:46:28.710799+0000","last_peered":"2024-02-23T00:46:28.710799+0000","last_clean":"2024-02-23T00:46:28.710799+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:46:28.710799+0000","last_undegraded":"2024-02-23T00:46:28.710799+0000","last_fullsized":"2024-02-23T00:46:28.710799+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:54.240855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:54.240855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":12,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63,"ondisk_log_size":63,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":554,"num_read_kb":448,"num_write":335,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":2586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":752,"ondisk_log_size":752,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":23,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":840,"num_read_kb":840,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":69,"acting":69,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332333,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309084,"kb_used_data":7764,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436068,"statfs":{"total":95995035648,"available":95678533632,"internally_reserved":0,"allocated":7950336,"data_stored":4280936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.499,"15min":0.48499999999999999},"min":{"1min":0.40100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68300000000000005,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.52100000000000002,"15min":0.5},"min":{"1min":0.433,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.499,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.217},"max":{"1min":0.64500000000000002,"5min":1.304,"15min":31.175000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.51900000000000002,"15min":0.74099999999999999},"min":{"1min":0.38500000000000001,"5min":0.32600000000000001,"15min":0.249},"max":{"1min":0.748,"5min":0.748,"15min":29.423999999999999},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.5,"15min":0.71599999999999997},"min":{"1min":0.40699999999999997,"5min":0.28699999999999998,"15min":0.28100000000000003},"max":{"1min":0.70099999999999996,"5min":0.72199999999999998,"15min":16.603999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.64600000000000002},"min":{"1min":0.40200000000000002,"5min":0.29099999999999998,"15min":0.26600000000000001},"max":{"1min":0.72899999999999998,"5min":1.05,"15min":7.5339999999999998},"last":0.56799999999999995}]},{"osd":3,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51600000000000001,"15min":1.008},"min":{"1min":0.40400000000000003,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":0.73699999999999999,"5min":0.89000000000000001,"15min":57.729999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.92000000000000004},"min":{"1min":0.47399999999999998,"5min":0.318,"15min":0.29999999999999999},"max":{"1min":0.72399999999999998,"5min":1.111,"15min":27.573},"last":0.59199999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.32400000000000001,"5min":0.255,"15min":0.214},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":1.706},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46200000000000002,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.24299999999999999,"15min":0.23300000000000001},"max":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":8.1359999999999992},"last":0.41599999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.432},"min":{"1min":0.28499999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.48099999999999998,"15min":0.46500000000000002},"min":{"1min":0.42099999999999999,"5min":0.29199999999999998,"15min":0.26500000000000001},"max":{"1min":0.66700000000000004,"5min":1.0660000000000001,"15min":1.2889999999999999},"last":0.51000000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.30599999999999999,"5min":0.30399999999999999,"15min":0.246},"max":{"1min":0.67500000000000004,"5min":0.72799999999999998,"15min":2.1339999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53600000000000003,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69199999999999995,"5min":1.127,"15min":5.194},"last":0.49399999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561266,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309072,"kb_used_data":7752,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436080,"statfs":{"total":95995035648,"available":95678545920,"internally_reserved":0,"allocated":7938048,"data_stored":4280498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52000000000000002,"15min":0.54000000000000004},"min":{"1min":0.29699999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.504,"15min":0.501},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78100000000000003,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.42299999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52400000000000002,"15min":0.96899999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.222},"max":{"1min":0.70899999999999996,"5min":0.79500000000000004,"15min":59.347999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51600000000000001,"15min":1.347},"min":{"1min":0.34499999999999997,"5min":0.31900000000000001,"15min":0.20999999999999999},"max":{"1min":0.68300000000000005,"5min":1.1619999999999999,"15min":87.221000000000004},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53300000000000003,"15min":0.81899999999999995},"min":{"1min":0.33000000000000002,"5min":0.32900000000000001,"15min":0.214},"max":{"1min":0.76200000000000001,"5min":1.163,"15min":25.550999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.81999999999999995},"min":{"1min":0.30099999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71299999999999997,"5min":2.9460000000000002,"15min":40.914000000000001},"last":0.59399999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56699999999999995,"15min":0.79100000000000004},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.23400000000000001},"max":{"1min":0.77400000000000002,"5min":2.859,"15min":31.309999999999999},"last":0.441},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54700000000000004,"15min":1.2470000000000001},"min":{"1min":0.36899999999999999,"5min":0.35799999999999998,"15min":0.27500000000000002},"max":{"1min":0.73099999999999998,"5min":0.89900000000000002,"15min":86.311999999999998},"last":0.55200000000000005}]},{"osd":4,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45000000000000001,"15min":0.47799999999999998},"min":{"1min":0.32500000000000001,"5min":0.25800000000000001,"15min":0.23699999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":7.8769999999999998},"last":0.505},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.48799999999999999,"15min":0.47599999999999998},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.23100000000000001},"max":{"1min":0.754,"5min":0.754,"15min":6.0300000000000002},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.65400000000000003,"5min":0.86899999999999999,"15min":6.7999999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.49199999999999999},"min":{"1min":0.35199999999999998,"5min":0.29299999999999998,"15min":0.26600000000000001},"max":{"1min":0.78100000000000003,"5min":0.97999999999999998,"15min":6.0780000000000003},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Feb 23 00:45:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53000000000000003,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":1.014,"15min":17.881},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.53000000000000003,"15min":0.65500000000000003},"min":{"1min":0.39800000000000002,"5min":0.32900000000000001,"15min":0.26500000000000001},"max":{"1min":0.73499999999999999,"5min":0.85699999999999998,"15min":18.475000000000001},"last":0.58499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575113,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313212,"kb_used_data":7796,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431940,"statfs":{"total":95995035648,"available":95674306560,"internally_reserved":0,"allocated":7983104,"data_stored":4284845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.36099999999999999,"15min":0.40799999999999997},"min":{"1min":0.22900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.61699999999999999,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.40899999999999997},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.66900000000000004,"5min":1.161,"15min":1.161},"last":0.21299999999999999}]},{"osd":2,"last update":"Fri Feb 23 00:45:50 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35999999999999999,"15min":0.66900000000000004},"min":{"1min":0.23999999999999999,"5min":0.185,"15min":0.17999999999999999},"max":{"1min":0.50700000000000001,"5min":1.677,"15min":35.313000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39700000000000002,"15min":0.58699999999999997},"min":{"1min":0.25900000000000001,"5min":0.20599999999999999,"15min":0.18099999999999999},"max":{"1min":0.52700000000000002,"5min":1.5549999999999999,"15min":27.908999999999999},"last":0.23699999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:46:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39700000000000002,"15min":0.64900000000000002},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.21199999999999999},"max":{"1min":0.69499999999999995,"5min":1.6279999999999999,"15min":21.013999999999999},"last":0.253},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40600000000000003,"15min":0.63400000000000001},"min":{"1min":0.313,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.66000000000000003,"5min":1.71,"15min":29.234999999999999},"last":0.32300000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56499999999999995,"15min":0.70699999999999996},"min":{"1min":0.39400000000000002,"5min":0.376,"15min":0.27400000000000002},"max":{"1min":0.82799999999999996,"5min":0.90300000000000002,"15min":16.231000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.71299999999999997},"min":{"1min":0.379,"5min":0.32200000000000001,"15min":0.28999999999999998},"max":{"1min":0.86699999999999999,"5min":1.7350000000000001,"15min":29.212},"last":0.48099999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56999999999999995,"15min":0.72399999999999998},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.29199999999999998},"max":{"1min":0.90700000000000003,"5min":0.93700000000000006,"15min":16.294},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.68600000000000005},"min":{"1min":0.442,"5min":0.35399999999999998,"15min":0.28699999999999998},"max":{"1min":0.84299999999999997,"5min":1.238,"15min":11.105},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57199999999999995,"15min":0.749},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.32200000000000001},"max":{"1min":0.88800000000000001,"5min":1.76,"15min":29.187000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.75},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.312},"max":{"1min":0.86899999999999999,"5min":0.90500000000000003,"15min":16.369},"last":0.68200000000000005}]},{"osd":7,"last update":"Fri Feb 23 00:46:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.75},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.32000000000000001},"max":{"1min":0.86899999999999999,"5min":0.92000000000000004,"15min":16.395},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60599999999999998,"15min":0.628},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.82099999999999995,"5min":0.95599999999999996,"15min":4.4379999999999997},"last":0.57199999999999995}]}]},{"osd":0,"up_from":659,"seq":2830383448130,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314784,"kb_used_data":7768,"kb_used_omap":85,"kb_used_meta":306922,"kb_avail":93430368,"statfs":{"total":95995035648,"available":95672696832,"internally_reserved":0,"allocated":7954432,"data_stored":4283522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":314288256},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38600000000000001,"15min":0.50900000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.496,"5min":5.0880000000000001,"15min":5.0880000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.36499999999999999,"15min":0.40500000000000003},"min":{"1min":0.23999999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45400000000000001,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35099999999999998,"15min":0.34200000000000003},"min":{"1min":0.222,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.57499999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.41599999999999998,"15min":0.54400000000000004},"min":{"1min":0.25900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56299999999999994,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.34100000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.51500000000000001,"15min":0.86499999999999999},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52700000000000002,"5min":11.891,"15min":11.891},"last":0.37},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.54600000000000004,"15min":1.0329999999999999},"min":{"1min":0.255,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54600000000000004,"5min":15.864000000000001,"15min":15.864000000000001},"last":0.29699999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.41199999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.92100000000000004,"5min":1.004,"15min":1.004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.57799999999999996,"15min":0.53800000000000003},"min":{"1min":0.47999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78800000000000003,"5min":0.996,"15min":0.996},"last":0.48499999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.56599999999999995,"15min":0.53800000000000003},"min":{"1min":0.45200000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.82299999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.44400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.82699999999999996,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.47699999999999998}]},{"osd":6,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61299999999999999,"15min":0.61599999999999999},"min":{"1min":0.443,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.60199999999999998},"min":{"1min":0.51100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.88400000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.46400000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.48599999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90900000000000003,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.432,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77500000000000002,"5min":1.04,"15min":1.04},"last":0.498}]}]},{"osd":2,"up_from":18,"seq":77309411589,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304968,"kb_used_data":7744,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440184,"statfs":{"total":95995035648,"available":95682748416,"internally_reserved":0,"allocated":7929856,"data_stored":4282145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32900000000000001,"15min":0.311},"min":{"1min":0.215,"5min":0.182,"15min":0.182},"max":{"1min":0.47199999999999998,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.318},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.34100000000000003,"15min":0.33100000000000002},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.50900000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.36099999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:45:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33700000000000002,"15min":0.59099999999999997},"min":{"1min":0.23200000000000001,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":0.47799999999999998,"5min":0.84399999999999997,"15min":19.324999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.752},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.45600000000000002,"5min":0.72899999999999998,"15min":35.052},"last":0.38300000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.35099999999999998,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.193,"15min":0.193},"max":{"1min":0.48499999999999999,"5min":0.69699999999999995,"15min":43.5},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39100000000000001,"15min":0.63800000000000001},"min":{"1min":0.29999999999999999,"5min":0.23599999999999999,"15min":0.22},"max":{"1min":0.54700000000000004,"5min":0.82399999999999995,"15min":27.414000000000001},"last":0.374}]},{"osd":4,"last update":"Fri Feb 23 00:46:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55900000000000005,"15min":0.65300000000000002},"min":{"1min":0.36899999999999999,"5min":0.34300000000000003,"15min":0.26400000000000001},"max":{"1min":0.83599999999999997,"5min":0.94299999999999995,"15min":14.614000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.84699999999999998,"5min":0.90600000000000003,"15min":7.766},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Feb 23 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56799999999999995,"15min":0.71399999999999997},"min":{"1min":0.40400000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81799999999999995,"5min":1.248,"15min":17.765000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.66400000000000003},"min":{"1min":0.379,"5min":0.316,"15min":0.312},"max":{"1min":0.85399999999999998,"5min":0.98099999999999998,"15min":14.589},"last":0.63400000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58099999999999996,"15min":0.61899999999999999},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.29699999999999999},"max":{"1min":0.82499999999999996,"5min":0.95599999999999996,"15min":7.7869999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.79400000000000004},"min":{"1min":0.40799999999999997,"5min":0.36399999999999999,"15min":0.32000000000000001},"max":{"1min":0.86399999999999999,"5min":1.218,"15min":20.574999999999999},"last":0.625}]},{"osd":7,"last update":"Fri Feb 23 00:45:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57799999999999996,"15min":1.0740000000000001},"min":{"1min":0.44800000000000001,"5min":0.38400000000000001,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":1.004,"15min":66.632000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59499999999999997,"15min":0.85099999999999998},"min":{"1min":0.46400000000000002,"5min":0.38100000000000001,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":1.0149999999999999,"15min":19.603999999999999},"last":0.55800000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149953,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317304,"kb_used_data":7792,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93427848,"statfs":{"total":95995035648,"available":95670116352,"internally_reserved":0,"allocated":7979008,"data_stored":4282375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:30 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.35599999999999998,"15min":0.372},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.443,"5min":2.468,"15min":2.468},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34699999999999998,"15min":0.33300000000000002},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.45400000000000001,"5min":1.012,"15min":1.012},"last":0.34399999999999997}]},{"osd":1,"last update":"Fri Feb 23 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.40400000000000003,"15min":0.61499999999999999},"min":{"1min":0.26800000000000002,"5min":0.19600000000000001,"15min":0.17899999999999999},"max":{"1min":0.436,"5min":5.0030000000000001,"15min":54.351999999999997},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.77200000000000002},"min":{"1min":0.28799999999999998,"5min":0.23000000000000001,"15min":0.219},"max":{"1min":0.55400000000000005,"5min":2.1680000000000001,"15min":55.337000000000003},"last":0.35799999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38100000000000001,"15min":0.754},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":0.501,"5min":0.89200000000000002,"15min":55.368000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.438,"15min":0.75},"min":{"1min":0.30299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47399999999999998,"5min":6.3150000000000004,"15min":54.383000000000003},"last":0.30399999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:46:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55500000000000005,"15min":0.93100000000000005},"min":{"1min":0.42199999999999999,"5min":0.33000000000000002,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":2.5720000000000001,"15min":54.530000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.63600000000000001},"min":{"1min":0.39300000000000002,"5min":0.314,"15min":0.24099999999999999},"max":{"1min":0.745,"5min":5.8109999999999999,"15min":5.8109999999999999},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Feb 23 00:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.623,"15min":0.89500000000000002},"min":{"1min":0.47399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85999999999999999,"5min":5.2160000000000002,"15min":55.401000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.871},"min":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":2.5219999999999998,"15min":54.454999999999998},"last":0.70399999999999996}]},{"osd":6,"last update":"Fri Feb 23 00:46:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.84899999999999998},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.28699999999999998},"max":{"1min":0.875,"5min":4.7069999999999999,"15min":54.305},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.72099999999999997,"15min":0.91500000000000004},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.32800000000000001},"max":{"1min":0.81799999999999995,"5min":8.5860000000000003,"15min":55.424999999999997},"last":0.54400000000000004}]},{"osd":7,"last update":"Fri Feb 23 00:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64900000000000002,"15min":0.81599999999999995},"min":{"1min":0.439,"5min":0.374,"15min":0.32200000000000001},"max":{"1min":0.78600000000000003,"5min":4.6710000000000003,"15min":54.222000000000001},"last":0.623},{"interface":"front","average":{"1min":0.624,"5min":0.66200000000000003,"15min":1.0489999999999999},"min":{"1min":0.47999999999999998,"5min":0.39400000000000002,"15min":0.32200000000000001},"max":{"1min":0.83499999999999996,"5min":5.1840000000000002,"15min":55.460000000000001},"last":0.69699999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986428,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317276,"kb_used_data":7764,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93427876,"statfs":{"total":95995035648,"available":95670145024,"internally_reserved":0,"allocated":7950336,"data_stored":4281267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52300000000000002,"15min":0.50700000000000001},"min":{"1min":0.374,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78000000000000003,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.34200000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76000000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.496,"15min":0.88900000000000001},"min":{"1min":0.34399999999999997,"5min":0.31,"15min":0.24299999999999999},"max":{"1min":0.72199999999999998,"5min":0.748,"15min":16.696000000000002},"last":0.373},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.78500000000000003},"min":{"1min":0.434,"5min":0.28000000000000003,"15min":0.255},"max":{"1min":0.748,"5min":1.4510000000000001,"15min":13.455},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.90000000000000002},"min":{"1min":0.436,"5min":0.32700000000000001,"15min":0.27700000000000002},"max":{"1min":0.68200000000000005,"5min":1.472,"15min":41.886000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.56799999999999995,"15min":1.0309999999999999},"min":{"1min":0.42099999999999999,"5min":0.33900000000000002,"15min":0.251},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":41.927999999999997},"last":0.58699999999999997}]},{"osd":3,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.55900000000000005,"15min":0.93400000000000005},"min":{"1min":0.40600000000000003,"5min":0.314,"15min":0.314},"max":{"1min":1.381,"5min":1.381,"15min":36.301000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51300000000000001,"15min":1.056},"min":{"1min":0.40600000000000003,"5min":0.35199999999999998,"15min":0.24299999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":42.064},"last":0.42199999999999999}]},{"osd":5,"last update":"Fri Feb 23 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.44900000000000001,"15min":0.57699999999999996},"min":{"1min":0.35999999999999999,"5min":0.26900000000000002,"15min":0.217},"max":{"1min":1.542,"5min":1.542,"15min":14.242000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.52800000000000002,"15min":0.629},"min":{"1min":0.40600000000000003,"5min":0.29099999999999998,"15min":0.24299999999999999},"max":{"1min":1.597,"5min":1.597,"15min":14.124000000000001},"last":0.45100000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.53600000000000003,"15min":0.57499999999999996},"min":{"1min":0.38500000000000001,"5min":0.315,"15min":0.254},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":8.0730000000000004},"last":0.498},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.52300000000000002,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.28299999999999997,"15min":0.23100000000000001},"max":{"1min":1.454,"5min":1.454,"15min":13.339},"last":0.47199999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52800000000000002,"15min":0.65100000000000002},"min":{"1min":0.42299999999999999,"5min":0.311,"15min":0.27300000000000002},"max":{"1min":1.0329999999999999,"5min":1.5129999999999999,"15min":17.178999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.58199999999999996,"15min":0.72899999999999998},"min":{"1min":0.49399999999999999,"5min":0.34200000000000003,"15min":0.28299999999999997},"max":{"1min":1.48,"5min":1.48,"15min":15.85},"last":0.61799999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757495,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309084,"kb_used_data":7764,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436068,"statfs":{"total":95995035648,"available":95678533632,"internally_reserved":0,"allocated":7950336,"data_stored":4283563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.36399999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.80900000000000005,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.312},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.39100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98199999999999998,"5min":1.087,"15min":1.087},"last":0.38800000000000001}]},{"osd":1,"last update":"Fri Feb 23 00:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.50900000000000001,"15min":0.75800000000000001},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.24099999999999999},"max":{"1min":0.72599999999999998,"5min":1.1180000000000001,"15min":38.838000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51800000000000002,"15min":1.1859999999999999},"min":{"1min":0.36799999999999999,"5min":0.26600000000000001,"15min":0.23699999999999999},"max":{"1min":0.92900000000000005,"5min":1.1200000000000001,"15min":127.794},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52100000000000002,"15min":0.95699999999999996},"min":{"1min":0.39900000000000002,"5min":0.32400000000000001,"15min":0.22},"max":{"1min":0.90600000000000003,"5min":1.2969999999999999,"15min":37.442999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55700000000000005,"15min":0.71199999999999997},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.249},"max":{"1min":0.76900000000000002,"5min":2.875,"15min":23.297999999999998},"last":0.41199999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53600000000000003,"15min":0.82199999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27300000000000002},"max":{"1min":0.83099999999999996,"5min":1.3300000000000001,"15min":24.414999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53800000000000003,"15min":0.90300000000000002},"min":{"1min":0.39800000000000002,"5min":0.317,"15min":0.27400000000000002},"max":{"1min":0.94899999999999995,"5min":1.1679999999999999,"15min":40.865000000000002},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Feb 23 00:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.78900000000000003},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":0.61099999999999999,"5min":0.753,"15min":103.13},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.45400000000000001,"15min":0.52500000000000002},"min":{"1min":0.27900000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.63200000000000001,"5min":0.91100000000000003,"15min":12.795999999999999},"last":0.48499999999999999}]},{"osd":6,"last update":"Fri Feb 23 00:46:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51200000000000001,"15min":0.53000000000000003},"min":{"1min":0.33800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.87,"5min":1.214,"15min":11.412000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51500000000000001,"15min":0.47599999999999998},"min":{"1min":0.35799999999999998,"5min":0.28299999999999997,"15min":0.26200000000000001},"max":{"1min":0.68600000000000005,"5min":1.3149999999999999,"15min":1.5229999999999999},"last":0.496}]},{"osd":7,"last update":"Fri Feb 23 00:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51400000000000001,"15min":0.56699999999999995},"min":{"1min":0.314,"5min":0.28299999999999997,"15min":0.27700000000000002},"max":{"1min":0.72699999999999998,"5min":1.298,"15min":11.478},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.35399999999999998,"5min":0.27900000000000003,"15min":0.25800000000000001},"max":{"1min":0.84499999999999997,"5min":1.2689999999999999,"15min":5.4050000000000002},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:46:44.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph pg dump --format=json 2024-02-23T00:46:44.457 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:43.177487+0000 osd.3 (osd.3) 363 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:46:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:43.180587+0000 osd.3 (osd.3) 364 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:46:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:43.721686+0000 osd.4 (osd.4) 419 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:46:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:43.722918+0000 osd.4 (osd.4) 420 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:46:44.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:43.854198+0000 mgr.y (mgr.24437) 777 : cluster [DBG] pgmap v1366: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-02-23T00:46:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:43.177487+0000 osd.3 (osd.3) 363 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:46:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:43.180587+0000 osd.3 (osd.3) 364 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:46:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:43.721686+0000 osd.4 (osd.4) 419 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:46:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:43.722918+0000 osd.4 (osd.4) 420 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:46:44.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:44 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:43.854198+0000 mgr.y (mgr.24437) 777 : cluster [DBG] pgmap v1366: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-02-23T00:46:44.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:43.177487+0000 osd.3 (osd.3) 363 : cluster [DBG] 2.1b scrub starts 2024-02-23T00:46:44.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:43.180587+0000 osd.3 (osd.3) 364 : cluster [DBG] 2.1b scrub ok 2024-02-23T00:46:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:43.721686+0000 osd.4 (osd.4) 419 : cluster [DBG] 22.0 scrub starts 2024-02-23T00:46:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:43.722918+0000 osd.4 (osd.4) 420 : cluster [DBG] 22.0 scrub ok 2024-02-23T00:46:44.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:43.854198+0000 mgr.y (mgr.24437) 777 : cluster [DBG] pgmap v1366: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-02-23T00:46:45.804 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:44.175551+0000 osd.3 (osd.3) 365 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:44.179184+0000 osd.3 (osd.3) 366 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:44.526717+0000 osd.1 (osd.1) 484 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:44.529753+0000 osd.1 (osd.1) 485 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:44.580800+0000 osd.5 (osd.5) 399 : cluster [DBG] 22.e scrub starts 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:44.582022+0000 osd.5 (osd.5) 400 : cluster [DBG] 22.e scrub ok 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:44.175551+0000 osd.3 (osd.3) 365 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:44.179184+0000 osd.3 (osd.3) 366 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:44.526717+0000 osd.1 (osd.1) 484 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:44.529753+0000 osd.1 (osd.1) 485 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:46:45.805 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:44.580800+0000 osd.5 (osd.5) 399 : cluster [DBG] 22.e scrub starts 2024-02-23T00:46:45.806 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:45 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:44.582022+0000 osd.5 (osd.5) 400 : cluster [DBG] 22.e scrub ok 2024-02-23T00:46:45.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:44.175551+0000 osd.3 (osd.3) 365 : cluster [DBG] 4.1d scrub starts 2024-02-23T00:46:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:44.179184+0000 osd.3 (osd.3) 366 : cluster [DBG] 4.1d scrub ok 2024-02-23T00:46:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:44.526717+0000 osd.1 (osd.1) 484 : cluster [DBG] 5.12 scrub starts 2024-02-23T00:46:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:44.529753+0000 osd.1 (osd.1) 485 : cluster [DBG] 5.12 scrub ok 2024-02-23T00:46:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:44.580800+0000 osd.5 (osd.5) 399 : cluster [DBG] 22.e scrub starts 2024-02-23T00:46:45.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:44.582022+0000 osd.5 (osd.5) 400 : cluster [DBG] 22.e scrub ok 2024-02-23T00:46:46.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.374+0000 7f3d479e8700 1 -- 172.21.15.87:0/607063974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 msgr2=0x7f3d400fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:46.379 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.374+0000 7f3d479e8700 1 --2- 172.21.15.87:0/607063974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 0x7f3d400fdf90 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 rx=0x7f3d3c004870 tx=0x7f3d3c009cb0).stop 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 -- 172.21.15.87:0/607063974 shutdown_connections 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 --2- 172.21.15.87:0/607063974 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400f8420 0x7f3d40104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 --2- 172.21.15.87:0/607063974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400fdbb0 0x7f3d400fdf90 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 --2- 172.21.15.87:0/607063974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400fe510 0x7f3d400f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 -- 172.21.15.87:0/607063974 >> 172.21.15.87:0/607063974 conn(0x7f3d400f3800 msgr2=0x7f3d400f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 -- 172.21.15.87:0/607063974 shutdown_connections 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 -- 172.21.15.87:0/607063974 wait complete. 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 librados: starting msgr at 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 librados: starting objecter 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 Processor -- start 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 -- start start 2024-02-23T00:46:46.380 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 librados: setting wanted keys 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 librados: calling monclient init 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400f8420 0x7f3d4010e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40108220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d40108710 con 0x7f3d400f8420 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d40108880 con 0x7f3d400fdbb0 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.375+0000 7f3d479e8700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d401089f0 con 0x7f3d400fe510 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45784700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400f8420 0x7f3d4010e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45784700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400f8420 0x7f3d4010e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39202/0 (socket says 172.21.15.87:39202) 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45784700 1 -- 172.21.15.87:0/1936367974 learned_addr learned my addr 172.21.15.87:0/1936367974 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45f85700 1 --2- 172.21.15.87:0/1936367974 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d44f83700 1 --2- 172.21.15.87:0/1936367974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:46.381 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45784700 1 -- 172.21.15.87:0/1936367974 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 msgr2=0x7f3d40108220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45784700 1 --2- 172.21.15.87:0/1936367974 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40108220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45784700 1 -- 172.21.15.87:0/1936367974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400fdbb0 msgr2=0x7f3d4010e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45784700 1 --2- 172.21.15.87:0/1936367974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45784700 1 -- 172.21.15.87:0/1936367974 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380097c0 con 0x7f3d400f8420 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d45784700 1 --2- 172.21.15.87:0/1936367974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400f8420 0x7f3d4010e040 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7f3d3c009eb0 tx=0x7f3d3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d367fc700 10 client.?.objecter ms_handle_connect 0x7f3d400f8420 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d367fc700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d367fc700 1 -- 172.21.15.87:0/1936367974 <== mon.0 v2:172.21.15.87:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c00e070 con 0x7f3d400f8420 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c005040 con 0x7f3d400f8420 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d367fc700 1 -- 172.21.15.87:0/1936367974 <== mon.0 v2:172.21.15.87:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d3c013da0 con 0x7f3d400f8420 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d367fc700 1 -- 172.21.15.87:0/1936367974 <== mon.0 v2:172.21.15.87:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c019070 con 0x7f3d400f8420 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d479e8700 10 client.28197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d40108f40 con 0x7f3d400f8420 2024-02-23T00:46:46.382 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.376+0000 7f3d479e8700 1 librados: init done 2024-02-23T00:46:46.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.377+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4005ee80 con 0x7f3d400f8420 2024-02-23T00:46:46.383 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d367fc700 1 -- 172.21.15.87:0/1936367974 <== mon.0 v2:172.21.15.87:3300/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7f3d3c01e420 con 0x7f3d400f8420 2024-02-23T00:46:46.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d367fc700 1 --2- 172.21.15.87:0/1936367974 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:46.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d367fc700 1 -- 172.21.15.87:0/1936367974 <== mon.0 v2:172.21.15.87:3300/0 5 ==== osd_map(857..857 src has 400..857) v4 ==== 7486+0+0 (secure 0 0 0) 0x7f3d3c07da80 con 0x7f3d400f8420 2024-02-23T00:46:46.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d367fc700 10 client.28197.objecter ms_dispatch 0x7f3d40001550 osd_map(857..857 src has 400..857) v4 2024-02-23T00:46:46.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d44f83700 1 --2- 172.21.15.87:0/1936367974 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:46.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d367fc700 3 client.28197.objecter handle_osd_map got epochs [857,857] > 0 2024-02-23T00:46:46.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d367fc700 3 client.28197.objecter handle_osd_map decoding full epoch 857 2024-02-23T00:46:46.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d367fc700 20 client.28197.objecter dump_active .. 0 homeless 2024-02-23T00:46:46.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d367fc700 10 client.28197.objecter ms_handle_connect 0x7f3d2c062c90 2024-02-23T00:46:46.384 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.378+0000 7f3d44f83700 1 --2- 172.21.15.87:0/1936367974 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 rx=0x7f3d401099c0 tx=0x7f3d3000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:46.385 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.382+0000 7f3d367fc700 1 -- 172.21.15.87:0/1936367974 <== mon.0 v2:172.21.15.87:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7f3d3c0512b0 con 0x7f3d400f8420 2024-02-23T00:46:46.540 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:45.102730+0000 osd.2 (osd.2) 345 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:46:46.540 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:45.103975+0000 osd.2 (osd.2) 346 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:46:46.540 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:45.161636+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:46:46.540 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:45.165104+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:46:46.541 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:45.431096+0000 osd.6 (osd.6) 414 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:46:46.545 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.543+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 --> v2:172.21.15.87:6800/1231597463 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d4005fa60 con 0x7f3d2c062c90 2024-02-23T00:46:46.557 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.555+0000 7f3d367fc700 1 -- 172.21.15.87:0/1936367974 <== mgr.24437 v2:172.21.15.87:6800/1231597463 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+432278 (secure 0 0 0) 0x7f3d4005fa60 con 0x7f3d2c062c90 2024-02-23T00:46:46.563 INFO:teuthology.orchestra.run.smithi087.stdout: 2024-02-23T00:46:46.566 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 10 librados: watch_flush enter 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 10 librados: watch_flush exit 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 20 client.28197.objecter shutdown clearing up homeless session... 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 10 client.28197.objecter successfully canceled tick 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 msgr2=0x7f3d2c065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 --2- 172.21.15.87:0/1936367974 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 rx=0x7f3d401099c0 tx=0x7f3d3000b040).stop 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400f8420 msgr2=0x7f3d4010e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 --2- 172.21.15.87:0/1936367974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400f8420 0x7f3d4010e040 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 rx=0x7f3d3c009eb0 tx=0x7f3d3c00b040).stop 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 shutdown_connections 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 --2- 172.21.15.87:0/1936367974 >> v2:172.21.15.87:6800/1231597463 conn(0x7f3d2c062c90 0x7f3d2c065150 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 --2- 172.21.15.87:0/1936367974 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7f3d400fe510 0x7f3d40108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 --2- 172.21.15.87:0/1936367974 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7f3d400f8420 0x7f3d4010e040 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 --2- 172.21.15.87:0/1936367974 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7f3d400fdbb0 0x7f3d4010e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 >> 172.21.15.87:0/1936367974 conn(0x7f3d400f3800 msgr2=0x7f3d400f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 shutdown_connections 2024-02-23T00:46:46.567 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 -- 172.21.15.87:0/1936367974 wait complete. 2024-02-23T00:46:46.568 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:46.558+0000 7f3d479e8700 1 librados: shutdown 2024-02-23T00:46:46.568 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2024-02-23T00:46:46.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:45.434594+0000 osd.6 (osd.6) 415 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:46:46.833 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:45.854966+0000 mgr.y (mgr.24437) 778 : cluster [DBG] pgmap v1367: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:46.833 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:45.102730+0000 osd.2 (osd.2) 345 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:46:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:45.103975+0000 osd.2 (osd.2) 346 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:46:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:45.161636+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:46:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:45.165104+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:46:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:45.431096+0000 osd.6 (osd.6) 414 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:46:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:45.434594+0000 osd.6 (osd.6) 415 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:46:46.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:46 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:45.854966+0000 mgr.y (mgr.24437) 778 : cluster [DBG] pgmap v1367: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:45.102730+0000 osd.2 (osd.2) 345 : cluster [DBG] 22.16 scrub starts 2024-02-23T00:46:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:45.103975+0000 osd.2 (osd.2) 346 : cluster [DBG] 22.16 scrub ok 2024-02-23T00:46:46.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:45.161636+0000 osd.0 (osd.0) 129 : cluster [DBG] 4.6 scrub starts 2024-02-23T00:46:46.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:45.165104+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.6 scrub ok 2024-02-23T00:46:46.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:45.431096+0000 osd.6 (osd.6) 414 : cluster [DBG] 5.1f scrub starts 2024-02-23T00:46:46.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:45.434594+0000 osd.6 (osd.6) 415 : cluster [DBG] 5.1f scrub ok 2024-02-23T00:46:46.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:45.854966+0000 mgr.y (mgr.24437) 778 : cluster [DBG] pgmap v1367: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-23T00:46:47.333 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1367,"stamp":"2024-02-23T00:46:45.854406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":225,"num_whiteouts":0,"num_read":1422,"num_read_kb":1316,"num_write":349,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":6055,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":864,"ondisk_log_size":864,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2494784,"kb_used_data":62144,"kb_used_omap":85,"kb_used_meta":2432490,"kb_avail":747466432,"statfs":{"total":767960285184,"available":765405626368,"internally_reserved":0,"allocated":63635456,"data_stored":34259151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":2490869888},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003411"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":447,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616376+0000","last_change":"2024-02-23T00:45:54.740202+0000","last_active":"2024-02-23T00:46:28.616376+0000","last_peered":"2024-02-23T00:46:28.616376+0000","last_clean":"2024-02-23T00:46:28.616376+0000","last_became_active":"2024-02-23T00:42:41.400209+0000","last_became_peered":"2024-02-23T00:42:41.400209+0000","last_unstale":"2024-02-23T00:46:28.616376+0000","last_undegraded":"2024-02-23T00:46:28.616376+0000","last_fullsized":"2024-02-23T00:46:28.616376+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:54.740125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:54.740125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"571'11","reported_seq":1121,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.553611+0000","last_change":"2024-02-23T00:46:29.726533+0000","last_active":"2024-02-23T00:46:40.553611+0000","last_peered":"2024-02-23T00:46:40.553611+0000","last_clean":"2024-02-23T00:46:40.553611+0000","last_became_active":"2024-02-23T00:39:49.195036+0000","last_became_peered":"2024-02-23T00:39:49.195036+0000","last_unstale":"2024-02-23T00:46:40.553611+0000","last_undegraded":"2024-02-23T00:46:40.553611+0000","last_fullsized":"2024-02-23T00:46:40.553611+0000","mapping_epoch":58,"log_start":"379'9","ondisk_log_start":"379'9","created":375,"last_epoch_clean":570,"parent":"0.0","parent_split_bits":0,"last_scrub":"571'11","last_scrub_stamp":"2024-02-23T00:46:29.726449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:29.726449+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"453'38","reported_seq":957,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.613713+0000","last_change":"2024-02-23T00:45:50.076790+0000","last_active":"2024-02-23T00:46:28.613713+0000","last_peered":"2024-02-23T00:46:28.613713+0000","last_clean":"2024-02-23T00:46:28.613713+0000","last_became_active":"2024-02-23T00:38:19.164624+0000","last_became_peered":"2024-02-23T00:38:19.164624+0000","last_unstale":"2024-02-23T00:46:28.613713+0000","last_undegraded":"2024-02-23T00:46:28.613713+0000","last_fullsized":"2024-02-23T00:46:28.613713+0000","mapping_epoch":56,"log_start":"81'20","ondisk_log_start":"81'20","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:45:50.076710+0000","last_deep_scrub":"453'38","last_deep_scrub_stamp":"2024-02-23T00:44:29.039144+0000","last_clean_scrub_stamp":"2024-02-23T00:45:50.076710+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":889,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473662+0000","last_change":"2024-02-23T00:45:35.172325+0000","last_active":"2024-02-23T00:46:28.473662+0000","last_peered":"2024-02-23T00:46:28.473662+0000","last_clean":"2024-02-23T00:46:28.473662+0000","last_became_active":"2024-02-23T00:38:49.198881+0000","last_became_peered":"2024-02-23T00:38:49.198881+0000","last_unstale":"2024-02-23T00:46:28.473662+0000","last_undegraded":"2024-02-23T00:46:28.473662+0000","last_fullsized":"2024-02-23T00:46:28.473662+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:35.172247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:05.018499+0000","last_clean_scrub_stamp":"2024-02-23T00:45:35.172247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.613904+0000","last_change":"2024-02-23T00:45:47.032756+0000","last_active":"2024-02-23T00:46:28.613904+0000","last_peered":"2024-02-23T00:46:28.613904+0000","last_clean":"2024-02-23T00:46:28.613904+0000","last_became_active":"2024-02-23T00:41:37.222092+0000","last_became_peered":"2024-02-23T00:41:37.222092+0000","last_unstale":"2024-02-23T00:46:28.613904+0000","last_undegraded":"2024-02-23T00:46:28.613904+0000","last_fullsized":"2024-02-23T00:46:28.613904+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:47.032669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:45:47.032669+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":819,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711718+0000","last_change":"2024-02-23T00:46:23.327485+0000","last_active":"2024-02-23T00:46:28.711718+0000","last_peered":"2024-02-23T00:46:28.711718+0000","last_clean":"2024-02-23T00:46:28.711718+0000","last_became_active":"2024-02-23T00:41:37.214214+0000","last_became_peered":"2024-02-23T00:41:37.214214+0000","last_unstale":"2024-02-23T00:46:28.711718+0000","last_undegraded":"2024-02-23T00:46:28.711718+0000","last_fullsized":"2024-02-23T00:46:28.711718+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:23.327413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:23.327413+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"78'6","reported_seq":1702,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:43.910894+0000","last_change":"2024-02-23T00:46:14.767726+0000","last_active":"2024-02-23T00:46:43.910894+0000","last_peered":"2024-02-23T00:46:43.910894+0000","last_clean":"2024-02-23T00:46:43.910894+0000","last_became_active":"2024-02-23T00:38:49.186830+0000","last_became_peered":"2024-02-23T00:38:49.186830+0000","last_unstale":"2024-02-23T00:46:43.910894+0000","last_undegraded":"2024-02-23T00:46:43.910894+0000","last_fullsized":"2024-02-23T00:46:43.910894+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:46:14.767639+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:46:14.767639+0000","last_clean_scrub_stamp":"2024-02-23T00:46:14.767639+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":845,"num_read_kb":845,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"840'27","reported_seq":579,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711448+0000","last_change":"2024-02-23T00:45:26.120956+0000","last_active":"2024-02-23T00:46:28.711448+0000","last_peered":"2024-02-23T00:46:28.711448+0000","last_clean":"2024-02-23T00:46:28.711448+0000","last_became_active":"2024-02-23T00:41:41.226195+0000","last_became_peered":"2024-02-23T00:41:41.226195+0000","last_unstale":"2024-02-23T00:46:28.711448+0000","last_undegraded":"2024-02-23T00:46:28.711448+0000","last_fullsized":"2024-02-23T00:46:28.711448+0000","mapping_epoch":681,"log_start":"78'14","ondisk_log_start":"78'14","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"840'27","last_scrub_stamp":"2024-02-23T00:45:26.120891+0000","last_deep_scrub":"840'27","last_deep_scrub_stamp":"2024-02-23T00:45:26.120891+0000","last_clean_scrub_stamp":"2024-02-23T00:45:26.120891+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":862,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710774+0000","last_change":"2024-02-23T00:46:00.293842+0000","last_active":"2024-02-23T00:46:28.710774+0000","last_peered":"2024-02-23T00:46:28.710774+0000","last_clean":"2024-02-23T00:46:28.710774+0000","last_became_active":"2024-02-23T00:38:49.227825+0000","last_became_peered":"2024-02-23T00:38:49.227825+0000","last_unstale":"2024-02-23T00:46:28.710774+0000","last_undegraded":"2024-02-23T00:46:28.710774+0000","last_fullsized":"2024-02-23T00:46:28.710774+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:00.293765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:00.293765+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.293765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":876,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472000+0000","last_change":"2024-02-23T00:45:43.263443+0000","last_active":"2024-02-23T00:46:28.472000+0000","last_peered":"2024-02-23T00:46:28.472000+0000","last_clean":"2024-02-23T00:46:28.472000+0000","last_became_active":"2024-02-23T00:38:49.175472+0000","last_became_peered":"2024-02-23T00:38:49.175472+0000","last_unstale":"2024-02-23T00:46:28.472000+0000","last_undegraded":"2024-02-23T00:46:28.472000+0000","last_fullsized":"2024-02-23T00:46:28.472000+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:43.263312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:43.263312+0000","last_clean_scrub_stamp":"2024-02-23T00:45:43.263312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":207,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:41.091382+0000","last_change":"2024-02-23T00:46:41.091382+0000","last_active":"2024-02-23T00:46:41.091382+0000","last_peered":"2024-02-23T00:46:41.091382+0000","last_clean":"2024-02-23T00:46:41.091382+0000","last_became_active":"2024-02-23T00:41:37.228937+0000","last_became_peered":"2024-02-23T00:41:37.228937+0000","last_unstale":"2024-02-23T00:46:41.091382+0000","last_undegraded":"2024-02-23T00:46:41.091382+0000","last_fullsized":"2024-02-23T00:46:41.091382+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:41.091264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:41.091264+0000","last_clean_scrub_stamp":"2024-02-23T00:46:41.091264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710953+0000","last_change":"2024-02-23T00:45:44.212146+0000","last_active":"2024-02-23T00:46:28.710953+0000","last_peered":"2024-02-23T00:46:28.710953+0000","last_clean":"2024-02-23T00:46:28.710953+0000","last_became_active":"2024-02-23T00:30:48.712562+0000","last_became_peered":"2024-02-23T00:30:48.712562+0000","last_unstale":"2024-02-23T00:46:28.710953+0000","last_undegraded":"2024-02-23T00:46:28.710953+0000","last_fullsized":"2024-02-23T00:46:28.710953+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:44.212054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:35.342427+0000","last_clean_scrub_stamp":"2024-02-23T00:45:44.212054+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616101+0000","last_change":"2024-02-23T00:46:05.662122+0000","last_active":"2024-02-23T00:46:28.616101+0000","last_peered":"2024-02-23T00:46:28.616101+0000","last_clean":"2024-02-23T00:46:28.616101+0000","last_became_active":"2024-02-23T00:42:41.393491+0000","last_became_peered":"2024-02-23T00:42:41.393491+0000","last_unstale":"2024-02-23T00:46:28.616101+0000","last_undegraded":"2024-02-23T00:46:28.616101+0000","last_fullsized":"2024-02-23T00:46:28.616101+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:05.662037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:46:05.662037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"476'63","reported_seq":979,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710832+0000","last_change":"2024-02-23T00:45:37.271204+0000","last_active":"2024-02-23T00:46:28.710832+0000","last_peered":"2024-02-23T00:46:28.710832+0000","last_clean":"2024-02-23T00:46:28.710832+0000","last_became_active":"2024-02-23T00:38:19.166026+0000","last_became_peered":"2024-02-23T00:38:19.166026+0000","last_unstale":"2024-02-23T00:46:28.710832+0000","last_undegraded":"2024-02-23T00:46:28.710832+0000","last_fullsized":"2024-02-23T00:46:28.710832+0000","mapping_epoch":56,"log_start":"63'25","ondisk_log_start":"63'25","created":56,"last_epoch_clean":482,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:45:37.271124+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:45:37.271124+0000","last_clean_scrub_stamp":"2024-02-23T00:45:37.271124+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":887,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:38.206978+0000","last_change":"2024-02-23T00:46:38.206978+0000","last_active":"2024-02-23T00:46:38.206978+0000","last_peered":"2024-02-23T00:46:38.206978+0000","last_clean":"2024-02-23T00:46:38.206978+0000","last_became_active":"2024-02-23T00:41:37.238728+0000","last_became_peered":"2024-02-23T00:41:37.238728+0000","last_unstale":"2024-02-23T00:46:38.206978+0000","last_undegraded":"2024-02-23T00:46:38.206978+0000","last_fullsized":"2024-02-23T00:46:38.206978+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:38.206914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:38.206914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473375+0000","last_change":"2024-02-23T00:45:20.260925+0000","last_active":"2024-02-23T00:46:28.473375+0000","last_peered":"2024-02-23T00:46:28.473375+0000","last_clean":"2024-02-23T00:46:28.473375+0000","last_became_active":"2024-02-23T00:41:37.239529+0000","last_became_peered":"2024-02-23T00:41:37.239529+0000","last_unstale":"2024-02-23T00:46:28.473375+0000","last_undegraded":"2024-02-23T00:46:28.473375+0000","last_fullsized":"2024-02-23T00:46:28.473375+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:20.260860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:20.260860+0000","last_clean_scrub_stamp":"2024-02-23T00:45:20.260860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":882,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615360+0000","last_change":"2024-02-23T00:46:27.735713+0000","last_active":"2024-02-23T00:46:28.615360+0000","last_peered":"2024-02-23T00:46:28.615360+0000","last_clean":"2024-02-23T00:46:28.615360+0000","last_became_active":"2024-02-23T00:40:37.197010+0000","last_became_peered":"2024-02-23T00:40:37.197010+0000","last_unstale":"2024-02-23T00:46:28.615360+0000","last_undegraded":"2024-02-23T00:46:28.615360+0000","last_fullsized":"2024-02-23T00:46:28.615360+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:27.735611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:02.358056+0000","last_clean_scrub_stamp":"2024-02-23T00:46:27.735611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":202,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710084+0000","last_change":"2024-02-23T00:46:12.993779+0000","last_active":"2024-02-23T00:46:28.710084+0000","last_peered":"2024-02-23T00:46:28.710084+0000","last_clean":"2024-02-23T00:46:28.710084+0000","last_became_active":"2024-02-23T00:41:37.248272+0000","last_became_peered":"2024-02-23T00:41:37.248272+0000","last_unstale":"2024-02-23T00:46:28.710084+0000","last_undegraded":"2024-02-23T00:46:28.710084+0000","last_fullsized":"2024-02-23T00:46:28.710084+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:12.993709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:12.993709+0000","last_clean_scrub_stamp":"2024-02-23T00:46:12.993709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":864,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711069+0000","last_change":"2024-02-23T00:46:01.349093+0000","last_active":"2024-02-23T00:46:28.711069+0000","last_peered":"2024-02-23T00:46:28.711069+0000","last_clean":"2024-02-23T00:46:28.711069+0000","last_became_active":"2024-02-23T00:31:29.088359+0000","last_became_peered":"2024-02-23T00:31:29.088359+0000","last_unstale":"2024-02-23T00:46:28.711069+0000","last_undegraded":"2024-02-23T00:46:28.711069+0000","last_fullsized":"2024-02-23T00:46:28.711069+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:01.349005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:01.349005+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"453'44","reported_seq":792,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:33.830195+0000","last_change":"2024-02-23T00:46:33.830195+0000","last_active":"2024-02-23T00:46:33.830195+0000","last_peered":"2024-02-23T00:46:33.830195+0000","last_clean":"2024-02-23T00:46:33.830195+0000","last_became_active":"2024-02-23T00:45:11.598917+0000","last_became_peered":"2024-02-23T00:45:11.598917+0000","last_unstale":"2024-02-23T00:46:33.830195+0000","last_undegraded":"2024-02-23T00:46:33.830195+0000","last_fullsized":"2024-02-23T00:46:33.830195+0000","mapping_epoch":833,"log_start":"76'24","ondisk_log_start":"76'24","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:46:33.830097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:33.830097+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":852,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615644+0000","last_change":"2024-02-23T00:46:25.653202+0000","last_active":"2024-02-23T00:46:28.615644+0000","last_peered":"2024-02-23T00:46:28.615644+0000","last_clean":"2024-02-23T00:46:28.615644+0000","last_became_active":"2024-02-23T00:27:17.013957+0000","last_became_peered":"2024-02-23T00:27:17.013957+0000","last_unstale":"2024-02-23T00:46:28.615644+0000","last_undegraded":"2024-02-23T00:46:28.615644+0000","last_fullsized":"2024-02-23T00:46:28.615644+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:25.653122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:25.653122+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":859,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:38.484586+0000","last_change":"2024-02-23T00:46:38.484586+0000","last_active":"2024-02-23T00:46:38.484586+0000","last_peered":"2024-02-23T00:46:38.484586+0000","last_clean":"2024-02-23T00:46:38.484586+0000","last_became_active":"2024-02-23T00:41:37.229416+0000","last_became_peered":"2024-02-23T00:41:37.229416+0000","last_unstale":"2024-02-23T00:46:38.484586+0000","last_undegraded":"2024-02-23T00:46:38.484586+0000","last_fullsized":"2024-02-23T00:46:38.484586+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:38.484522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:38.484522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":91,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710776+0000","last_change":"2024-02-23T00:45:16.953513+0000","last_active":"2024-02-23T00:46:28.710776+0000","last_peered":"2024-02-23T00:46:28.710776+0000","last_clean":"2024-02-23T00:46:28.710776+0000","last_became_active":"2024-02-23T00:43:42.260383+0000","last_became_peered":"2024-02-23T00:43:42.260383+0000","last_unstale":"2024-02-23T00:46:28.710776+0000","last_undegraded":"2024-02-23T00:46:28.710776+0000","last_fullsized":"2024-02-23T00:46:28.710776+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:16.953402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:16.953402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":388,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.305191+0000","last_change":"2024-02-23T00:46:40.305191+0000","last_active":"2024-02-23T00:46:40.305191+0000","last_peered":"2024-02-23T00:46:40.305191+0000","last_clean":"2024-02-23T00:46:40.305191+0000","last_became_active":"2024-02-23T00:41:37.252083+0000","last_became_peered":"2024-02-23T00:41:37.252083+0000","last_unstale":"2024-02-23T00:46:40.305191+0000","last_undegraded":"2024-02-23T00:46:40.305191+0000","last_fullsized":"2024-02-23T00:46:40.305191+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:40.305086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:40.305086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"453'37","reported_seq":831,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471762+0000","last_change":"2024-02-23T00:45:58.130467+0000","last_active":"2024-02-23T00:46:28.471762+0000","last_peered":"2024-02-23T00:46:28.471762+0000","last_clean":"2024-02-23T00:46:28.471762+0000","last_became_active":"2024-02-23T00:45:11.600573+0000","last_became_peered":"2024-02-23T00:45:11.600573+0000","last_unstale":"2024-02-23T00:46:28.471762+0000","last_undegraded":"2024-02-23T00:46:28.471762+0000","last_fullsized":"2024-02-23T00:46:28.471762+0000","mapping_epoch":833,"log_start":"62'15","ondisk_log_start":"62'15","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:45:58.130374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:58.130374+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":875,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711572+0000","last_change":"2024-02-23T00:46:05.283413+0000","last_active":"2024-02-23T00:46:28.711572+0000","last_peered":"2024-02-23T00:46:28.711572+0000","last_clean":"2024-02-23T00:46:28.711572+0000","last_became_active":"2024-02-23T00:38:49.224945+0000","last_became_peered":"2024-02-23T00:38:49.224945+0000","last_unstale":"2024-02-23T00:46:28.711572+0000","last_undegraded":"2024-02-23T00:46:28.711572+0000","last_fullsized":"2024-02-23T00:46:28.711572+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:05.283350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:35.937969+0000","last_clean_scrub_stamp":"2024-02-23T00:46:05.283350+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"62'8","reported_seq":1099,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.553157+0000","last_change":"2024-02-23T00:45:47.340086+0000","last_active":"2024-02-23T00:46:40.553157+0000","last_peered":"2024-02-23T00:46:40.553157+0000","last_clean":"2024-02-23T00:46:40.553157+0000","last_became_active":"2024-02-23T00:27:21.024945+0000","last_became_peered":"2024-02-23T00:27:21.024945+0000","last_unstale":"2024-02-23T00:46:40.553157+0000","last_undegraded":"2024-02-23T00:46:40.553157+0000","last_fullsized":"2024-02-23T00:46:40.553157+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'8","last_scrub_stamp":"2024-02-23T00:45:47.340019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:45:47.340019+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472155+0000","last_change":"2024-02-23T00:45:44.201895+0000","last_active":"2024-02-23T00:46:28.472155+0000","last_peered":"2024-02-23T00:46:28.472155+0000","last_clean":"2024-02-23T00:46:28.472155+0000","last_became_active":"2024-02-23T00:30:48.655686+0000","last_became_peered":"2024-02-23T00:30:48.655686+0000","last_unstale":"2024-02-23T00:46:28.472155+0000","last_undegraded":"2024-02-23T00:46:28.472155+0000","last_fullsized":"2024-02-23T00:46:28.472155+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:44.201810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:44.201810+0000","last_clean_scrub_stamp":"2024-02-23T00:45:44.201810+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"78'6","reported_seq":369,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473263+0000","last_change":"2024-02-23T00:45:28.552593+0000","last_active":"2024-02-23T00:46:28.473263+0000","last_peered":"2024-02-23T00:46:28.473263+0000","last_clean":"2024-02-23T00:46:28.473263+0000","last_became_active":"2024-02-23T00:38:49.217111+0000","last_became_peered":"2024-02-23T00:38:49.217111+0000","last_unstale":"2024-02-23T00:46:28.473263+0000","last_undegraded":"2024-02-23T00:46:28.473263+0000","last_fullsized":"2024-02-23T00:46:28.473263+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:45:28.552529+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:45:28.552529+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"453'42","reported_seq":1034,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:37.509803+0000","last_change":"2024-02-23T00:46:37.509803+0000","last_active":"2024-02-23T00:46:37.509803+0000","last_peered":"2024-02-23T00:46:37.509803+0000","last_clean":"2024-02-23T00:46:37.509803+0000","last_became_active":"2024-02-23T00:39:25.199903+0000","last_became_peered":"2024-02-23T00:39:25.199903+0000","last_unstale":"2024-02-23T00:46:37.509803+0000","last_undegraded":"2024-02-23T00:46:37.509803+0000","last_fullsized":"2024-02-23T00:46:37.509803+0000","mapping_epoch":545,"log_start":"87'22","ondisk_log_start":"87'22","created":56,"last_epoch_clean":546,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:46:37.509720+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:46:37.509720+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":859,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710160+0000","last_change":"2024-02-23T00:46:06.300950+0000","last_active":"2024-02-23T00:46:28.710160+0000","last_peered":"2024-02-23T00:46:28.710160+0000","last_clean":"2024-02-23T00:46:28.710160+0000","last_became_active":"2024-02-23T00:38:49.228515+0000","last_became_peered":"2024-02-23T00:38:49.228515+0000","last_unstale":"2024-02-23T00:46:28.710160+0000","last_undegraded":"2024-02-23T00:46:28.710160+0000","last_fullsized":"2024-02-23T00:46:28.710160+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:06.300882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:06.300882+0000","last_clean_scrub_stamp":"2024-02-23T00:46:06.300882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"851'14","reported_seq":934,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.553709+0000","last_change":"2024-02-23T00:46:02.177625+0000","last_active":"2024-02-23T00:46:40.553709+0000","last_peered":"2024-02-23T00:46:40.553709+0000","last_clean":"2024-02-23T00:46:40.553709+0000","last_became_active":"2024-02-23T00:45:11.600072+0000","last_became_peered":"2024-02-23T00:45:11.600072+0000","last_unstale":"2024-02-23T00:46:40.553709+0000","last_undegraded":"2024-02-23T00:46:40.553709+0000","last_fullsized":"2024-02-23T00:46:40.553709+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'14","last_scrub_stamp":"2024-02-23T00:46:02.177545+0000","last_deep_scrub":"851'14","last_deep_scrub_stamp":"2024-02-23T00:46:02.177545+0000","last_clean_scrub_stamp":"2024-02-23T00:46:02.177545+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":791,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:31.466305+0000","last_change":"2024-02-23T00:46:31.466281+0000","last_active":"2024-02-23T00:46:31.466305+0000","last_peered":"2024-02-23T00:46:31.466305+0000","last_clean":"2024-02-23T00:46:31.466305+0000","last_became_active":"2024-02-23T00:30:48.712653+0000","last_became_peered":"2024-02-23T00:30:48.712653+0000","last_unstale":"2024-02-23T00:46:31.466305+0000","last_undegraded":"2024-02-23T00:46:31.466305+0000","last_fullsized":"2024-02-23T00:46:31.466305+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:31.466214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.396824+0000","last_clean_scrub_stamp":"2024-02-23T00:46:31.466214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":861,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471061+0000","last_change":"2024-02-23T00:45:42.221560+0000","last_active":"2024-02-23T00:46:28.471061+0000","last_peered":"2024-02-23T00:46:28.471061+0000","last_clean":"2024-02-23T00:46:28.471061+0000","last_became_active":"2024-02-23T00:31:29.073986+0000","last_became_peered":"2024-02-23T00:31:29.073986+0000","last_unstale":"2024-02-23T00:46:28.471061+0000","last_undegraded":"2024-02-23T00:46:28.471061+0000","last_fullsized":"2024-02-23T00:46:28.471061+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:42.221479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:03.136682+0000","last_clean_scrub_stamp":"2024-02-23T00:45:42.221479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"453'40","reported_seq":801,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472043+0000","last_change":"2024-02-23T00:45:11.601330+0000","last_active":"2024-02-23T00:46:28.472043+0000","last_peered":"2024-02-23T00:46:28.472043+0000","last_clean":"2024-02-23T00:46:28.472043+0000","last_became_active":"2024-02-23T00:45:11.600967+0000","last_became_peered":"2024-02-23T00:45:11.600967+0000","last_unstale":"2024-02-23T00:46:28.472043+0000","last_undegraded":"2024-02-23T00:46:28.472043+0000","last_fullsized":"2024-02-23T00:46:28.472043+0000","mapping_epoch":833,"log_start":"92'22","ondisk_log_start":"92'22","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:45:09.194799+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:45:09.194799+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"55'1","reported_seq":888,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615604+0000","last_change":"2024-02-23T00:45:51.754064+0000","last_active":"2024-02-23T00:46:28.615604+0000","last_peered":"2024-02-23T00:46:28.615604+0000","last_clean":"2024-02-23T00:46:28.615604+0000","last_became_active":"2024-02-23T00:41:37.229862+0000","last_became_peered":"2024-02-23T00:41:37.229862+0000","last_unstale":"2024-02-23T00:46:28.615604+0000","last_undegraded":"2024-02-23T00:46:28.615604+0000","last_fullsized":"2024-02-23T00:46:28.615604+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'1","last_scrub_stamp":"2024-02-23T00:45:51.753983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:51.753983+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":836,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615011+0000","last_change":"2024-02-23T00:46:11.822018+0000","last_active":"2024-02-23T00:46:28.615011+0000","last_peered":"2024-02-23T00:46:28.615011+0000","last_clean":"2024-02-23T00:46:28.615011+0000","last_became_active":"2024-02-23T00:27:21.023032+0000","last_became_peered":"2024-02-23T00:27:21.023032+0000","last_unstale":"2024-02-23T00:46:28.615011+0000","last_undegraded":"2024-02-23T00:46:28.615011+0000","last_fullsized":"2024-02-23T00:46:28.615011+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:11.821938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:54.788865+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.821938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":809,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711007+0000","last_change":"2024-02-23T00:46:13.377537+0000","last_active":"2024-02-23T00:46:28.711007+0000","last_peered":"2024-02-23T00:46:28.711007+0000","last_clean":"2024-02-23T00:46:28.711007+0000","last_became_active":"2024-02-23T00:38:49.227384+0000","last_became_peered":"2024-02-23T00:38:49.227384+0000","last_unstale":"2024-02-23T00:46:28.711007+0000","last_undegraded":"2024-02-23T00:46:28.711007+0000","last_fullsized":"2024-02-23T00:46:28.711007+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:13.377475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:19.126375+0000","last_clean_scrub_stamp":"2024-02-23T00:46:13.377475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":216,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711139+0000","last_change":"2024-02-23T00:46:22.040756+0000","last_active":"2024-02-23T00:46:28.711139+0000","last_peered":"2024-02-23T00:46:28.711139+0000","last_clean":"2024-02-23T00:46:28.711139+0000","last_became_active":"2024-02-23T00:42:39.394104+0000","last_became_peered":"2024-02-23T00:42:39.394104+0000","last_unstale":"2024-02-23T00:46:28.711139+0000","last_undegraded":"2024-02-23T00:46:28.711139+0000","last_fullsized":"2024-02-23T00:46:28.711139+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:22.040679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:39.641617+0000","last_clean_scrub_stamp":"2024-02-23T00:46:22.040679+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"642'14","reported_seq":453,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.553183+0000","last_change":"2024-02-23T00:45:21.622607+0000","last_active":"2024-02-23T00:46:40.553183+0000","last_peered":"2024-02-23T00:46:40.553183+0000","last_clean":"2024-02-23T00:46:40.553183+0000","last_became_active":"2024-02-23T00:41:01.212780+0000","last_became_peered":"2024-02-23T00:41:01.212780+0000","last_unstale":"2024-02-23T00:46:40.553183+0000","last_undegraded":"2024-02-23T00:46:40.553183+0000","last_fullsized":"2024-02-23T00:46:40.553183+0000","mapping_epoch":511,"log_start":"562'12","ondisk_log_start":"562'12","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'14","last_scrub_stamp":"2024-02-23T00:45:21.622489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:45:21.622489+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"453'34","reported_seq":758,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471349+0000","last_change":"2024-02-23T00:45:27.426484+0000","last_active":"2024-02-23T00:46:28.471349+0000","last_peered":"2024-02-23T00:46:28.471349+0000","last_clean":"2024-02-23T00:46:28.471349+0000","last_became_active":"2024-02-23T00:45:11.601293+0000","last_became_peered":"2024-02-23T00:45:11.601293+0000","last_unstale":"2024-02-23T00:46:28.471349+0000","last_undegraded":"2024-02-23T00:46:28.471349+0000","last_fullsized":"2024-02-23T00:46:28.471349+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'34","last_scrub_stamp":"2024-02-23T00:45:27.426401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:27.426401+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":46,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614720+0000","last_change":"2024-02-23T00:45:51.094921+0000","last_active":"2024-02-23T00:46:28.614720+0000","last_peered":"2024-02-23T00:46:28.614720+0000","last_clean":"2024-02-23T00:46:28.614720+0000","last_became_active":"2024-02-23T00:45:19.644963+0000","last_became_peered":"2024-02-23T00:45:19.644963+0000","last_unstale":"2024-02-23T00:46:28.614720+0000","last_undegraded":"2024-02-23T00:46:28.614720+0000","last_fullsized":"2024-02-23T00:46:28.614720+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:51.094839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:42.679545+0000","last_clean_scrub_stamp":"2024-02-23T00:45:51.094839+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":869,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:37.535423+0000","last_change":"2024-02-23T00:46:37.535423+0000","last_active":"2024-02-23T00:46:37.535423+0000","last_peered":"2024-02-23T00:46:37.535423+0000","last_clean":"2024-02-23T00:46:37.535423+0000","last_became_active":"2024-02-23T00:40:57.488797+0000","last_became_peered":"2024-02-23T00:40:57.488797+0000","last_unstale":"2024-02-23T00:46:37.535423+0000","last_undegraded":"2024-02-23T00:46:37.535423+0000","last_fullsized":"2024-02-23T00:46:37.535423+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:37.535342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:46:37.535342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":834,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:39.499430+0000","last_change":"2024-02-23T00:46:39.499430+0000","last_active":"2024-02-23T00:46:39.499430+0000","last_peered":"2024-02-23T00:46:39.499430+0000","last_clean":"2024-02-23T00:46:39.499430+0000","last_became_active":"2024-02-23T00:41:37.229061+0000","last_became_peered":"2024-02-23T00:41:37.229061+0000","last_unstale":"2024-02-23T00:46:39.499430+0000","last_undegraded":"2024-02-23T00:46:39.499430+0000","last_fullsized":"2024-02-23T00:46:39.499430+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:39.499336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:29.596983+0000","last_clean_scrub_stamp":"2024-02-23T00:46:39.499336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":404,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472786+0000","last_change":"2024-02-23T00:46:12.096727+0000","last_active":"2024-02-23T00:46:28.472786+0000","last_peered":"2024-02-23T00:46:28.472786+0000","last_clean":"2024-02-23T00:46:28.472786+0000","last_became_active":"2024-02-23T00:42:39.396063+0000","last_became_peered":"2024-02-23T00:42:39.396063+0000","last_unstale":"2024-02-23T00:46:28.472786+0000","last_undegraded":"2024-02-23T00:46:28.472786+0000","last_fullsized":"2024-02-23T00:46:28.472786+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:12.096663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:12.096663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":899,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471598+0000","last_change":"2024-02-23T00:46:07.103115+0000","last_active":"2024-02-23T00:46:28.471598+0000","last_peered":"2024-02-23T00:46:28.471598+0000","last_clean":"2024-02-23T00:46:28.471598+0000","last_became_active":"2024-02-23T00:40:57.218869+0000","last_became_peered":"2024-02-23T00:40:57.218869+0000","last_unstale":"2024-02-23T00:46:28.471598+0000","last_undegraded":"2024-02-23T00:46:28.471598+0000","last_fullsized":"2024-02-23T00:46:28.471598+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":638,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:07.103031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:07.103031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"453'42","reported_seq":437,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:32.403751+0000","last_change":"2024-02-23T00:46:32.403751+0000","last_active":"2024-02-23T00:46:32.403751+0000","last_peered":"2024-02-23T00:46:32.403751+0000","last_clean":"2024-02-23T00:46:32.403751+0000","last_became_active":"2024-02-23T00:41:37.228326+0000","last_became_peered":"2024-02-23T00:41:37.228326+0000","last_unstale":"2024-02-23T00:46:32.403751+0000","last_undegraded":"2024-02-23T00:46:32.403751+0000","last_fullsized":"2024-02-23T00:46:32.403751+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:46:32.403638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:32.403638+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":73,"num_read_kb":59,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":844,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.453333+0000","last_change":"2024-02-23T00:46:35.453333+0000","last_active":"2024-02-23T00:46:35.453333+0000","last_peered":"2024-02-23T00:46:35.453333+0000","last_clean":"2024-02-23T00:46:35.453333+0000","last_became_active":"2024-02-23T00:27:17.013903+0000","last_became_peered":"2024-02-23T00:27:17.013903+0000","last_unstale":"2024-02-23T00:46:35.453333+0000","last_undegraded":"2024-02-23T00:46:35.453333+0000","last_fullsized":"2024-02-23T00:46:35.453333+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:35.453269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:29.390757+0000","last_clean_scrub_stamp":"2024-02-23T00:46:35.453269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":738,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471668+0000","last_change":"2024-02-23T00:46:22.989999+0000","last_active":"2024-02-23T00:46:28.471668+0000","last_peered":"2024-02-23T00:46:28.471668+0000","last_clean":"2024-02-23T00:46:28.471668+0000","last_became_active":"2024-02-23T00:45:11.594841+0000","last_became_peered":"2024-02-23T00:45:11.594841+0000","last_unstale":"2024-02-23T00:46:28.471668+0000","last_undegraded":"2024-02-23T00:46:28.471668+0000","last_fullsized":"2024-02-23T00:46:28.471668+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:22.989879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:46:22.989879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":378,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472638+0000","last_change":"2024-02-23T00:46:08.116839+0000","last_active":"2024-02-23T00:46:28.472638+0000","last_peered":"2024-02-23T00:46:28.472638+0000","last_clean":"2024-02-23T00:46:28.472638+0000","last_became_active":"2024-02-23T00:43:42.256711+0000","last_became_peered":"2024-02-23T00:43:42.256711+0000","last_unstale":"2024-02-23T00:46:28.472638+0000","last_undegraded":"2024-02-23T00:46:28.472638+0000","last_fullsized":"2024-02-23T00:46:28.472638+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:08.116763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:08.116763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"78'6","reported_seq":1116,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:34.491825+0000","last_change":"2024-02-23T00:46:34.491825+0000","last_active":"2024-02-23T00:46:34.491825+0000","last_peered":"2024-02-23T00:46:34.491825+0000","last_clean":"2024-02-23T00:46:34.491825+0000","last_became_active":"2024-02-23T00:42:39.398790+0000","last_became_peered":"2024-02-23T00:42:39.398790+0000","last_unstale":"2024-02-23T00:46:34.491825+0000","last_undegraded":"2024-02-23T00:46:34.491825+0000","last_fullsized":"2024-02-23T00:46:34.491825+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":4,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:46:34.491739+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:45:28.352035+0000","last_clean_scrub_stamp":"2024-02-23T00:46:34.491739+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":900,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710948+0000","last_change":"2024-02-23T00:46:20.197787+0000","last_active":"2024-02-23T00:46:28.710948+0000","last_peered":"2024-02-23T00:46:28.710948+0000","last_clean":"2024-02-23T00:46:28.710948+0000","last_became_active":"2024-02-23T00:41:37.220531+0000","last_became_peered":"2024-02-23T00:41:37.220531+0000","last_unstale":"2024-02-23T00:46:28.710948+0000","last_undegraded":"2024-02-23T00:46:28.710948+0000","last_fullsized":"2024-02-23T00:46:28.710948+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:20.197708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:25.452086+0000","last_clean_scrub_stamp":"2024-02-23T00:46:20.197708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"801'44","reported_seq":235,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711490+0000","last_change":"2024-02-23T00:45:30.162438+0000","last_active":"2024-02-23T00:46:28.711490+0000","last_peered":"2024-02-23T00:46:28.711490+0000","last_clean":"2024-02-23T00:46:28.711490+0000","last_became_active":"2024-02-23T00:41:37.235230+0000","last_became_peered":"2024-02-23T00:41:37.235230+0000","last_unstale":"2024-02-23T00:46:28.711490+0000","last_undegraded":"2024-02-23T00:46:28.711490+0000","last_fullsized":"2024-02-23T00:46:28.711490+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'44","last_scrub_stamp":"2024-02-23T00:45:30.162372+0000","last_deep_scrub":"801'44","last_deep_scrub_stamp":"2024-02-23T00:44:02.476354+0000","last_clean_scrub_stamp":"2024-02-23T00:45:30.162372+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":64,"num_read_kb":49,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616412+0000","last_change":"2024-02-23T00:46:14.567073+0000","last_active":"2024-02-23T00:46:28.616412+0000","last_peered":"2024-02-23T00:46:28.616412+0000","last_clean":"2024-02-23T00:46:28.616412+0000","last_became_active":"2024-02-23T00:41:37.229922+0000","last_became_peered":"2024-02-23T00:41:37.229922+0000","last_unstale":"2024-02-23T00:46:28.616412+0000","last_undegraded":"2024-02-23T00:46:28.616412+0000","last_fullsized":"2024-02-23T00:46:28.616412+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:14.566990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:14.566990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":856,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710246+0000","last_change":"2024-02-23T00:46:11.323233+0000","last_active":"2024-02-23T00:46:28.710246+0000","last_peered":"2024-02-23T00:46:28.710246+0000","last_clean":"2024-02-23T00:46:28.710246+0000","last_became_active":"2024-02-23T00:41:05.523365+0000","last_became_peered":"2024-02-23T00:41:05.523365+0000","last_unstale":"2024-02-23T00:46:28.710246+0000","last_undegraded":"2024-02-23T00:46:28.710246+0000","last_fullsized":"2024-02-23T00:46:28.710246+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":646,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:11.323140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.842931+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.323140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":786,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710323+0000","last_change":"2024-02-23T00:45:52.304920+0000","last_active":"2024-02-23T00:46:28.710323+0000","last_peered":"2024-02-23T00:46:28.710323+0000","last_clean":"2024-02-23T00:46:28.710323+0000","last_became_active":"2024-02-23T00:30:48.707213+0000","last_became_peered":"2024-02-23T00:30:48.707213+0000","last_unstale":"2024-02-23T00:46:28.710323+0000","last_undegraded":"2024-02-23T00:46:28.710323+0000","last_fullsized":"2024-02-23T00:46:28.710323+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:52.304832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:52.304832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":882,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615073+0000","last_change":"2024-02-23T00:46:25.642212+0000","last_active":"2024-02-23T00:46:28.615073+0000","last_peered":"2024-02-23T00:46:28.615073+0000","last_clean":"2024-02-23T00:46:28.615073+0000","last_became_active":"2024-02-23T00:41:37.228303+0000","last_became_peered":"2024-02-23T00:41:37.228303+0000","last_unstale":"2024-02-23T00:46:28.615073+0000","last_undegraded":"2024-02-23T00:46:28.615073+0000","last_fullsized":"2024-02-23T00:46:28.615073+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:25.642137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:24.953349+0000","last_clean_scrub_stamp":"2024-02-23T00:46:25.642137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"453'39","reported_seq":585,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710983+0000","last_change":"2024-02-23T00:45:03.809706+0000","last_active":"2024-02-23T00:46:28.710983+0000","last_peered":"2024-02-23T00:46:28.710983+0000","last_clean":"2024-02-23T00:46:28.710983+0000","last_became_active":"2024-02-23T00:41:41.224814+0000","last_became_peered":"2024-02-23T00:41:41.224814+0000","last_unstale":"2024-02-23T00:46:28.710983+0000","last_undegraded":"2024-02-23T00:46:28.710983+0000","last_fullsized":"2024-02-23T00:46:28.710983+0000","mapping_epoch":681,"log_start":"63'21","ondisk_log_start":"63'21","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:45:03.809635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:03.809635+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615473+0000","last_change":"2024-02-23T00:46:11.504052+0000","last_active":"2024-02-23T00:46:28.615473+0000","last_peered":"2024-02-23T00:46:28.615473+0000","last_clean":"2024-02-23T00:46:28.615473+0000","last_became_active":"2024-02-23T00:41:37.229722+0000","last_became_peered":"2024-02-23T00:41:37.229722+0000","last_unstale":"2024-02-23T00:46:28.615473+0000","last_undegraded":"2024-02-23T00:46:28.615473+0000","last_fullsized":"2024-02-23T00:46:28.615473+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:11.503969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.503969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":216,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:33.141005+0000","last_change":"2024-02-23T00:46:33.141005+0000","last_active":"2024-02-23T00:46:33.141005+0000","last_peered":"2024-02-23T00:46:33.141005+0000","last_clean":"2024-02-23T00:46:33.141005+0000","last_became_active":"2024-02-23T00:42:39.395793+0000","last_became_peered":"2024-02-23T00:42:39.395793+0000","last_unstale":"2024-02-23T00:46:33.141005+0000","last_undegraded":"2024-02-23T00:46:33.141005+0000","last_fullsized":"2024-02-23T00:46:33.141005+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:33.140935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:33.140935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"78'2","reported_seq":887,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710921+0000","last_change":"2024-02-23T00:45:42.370861+0000","last_active":"2024-02-23T00:46:28.710921+0000","last_peered":"2024-02-23T00:46:28.710921+0000","last_clean":"2024-02-23T00:46:28.710921+0000","last_became_active":"2024-02-23T00:41:25.236069+0000","last_became_peered":"2024-02-23T00:41:25.236069+0000","last_unstale":"2024-02-23T00:46:28.710921+0000","last_undegraded":"2024-02-23T00:46:28.710921+0000","last_fullsized":"2024-02-23T00:46:28.710921+0000","mapping_epoch":511,"log_start":"78'2","ondisk_log_start":"78'2","created":60,"last_epoch_clean":666,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'2","last_scrub_stamp":"2024-02-23T00:45:42.370765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:45:42.370765+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471844+0000","last_change":"2024-02-23T00:46:11.173320+0000","last_active":"2024-02-23T00:46:28.471844+0000","last_peered":"2024-02-23T00:46:28.471844+0000","last_clean":"2024-02-23T00:46:28.471844+0000","last_became_active":"2024-02-23T00:30:48.708272+0000","last_became_peered":"2024-02-23T00:30:48.708272+0000","last_unstale":"2024-02-23T00:46:28.471844+0000","last_undegraded":"2024-02-23T00:46:28.471844+0000","last_fullsized":"2024-02-23T00:46:28.471844+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:11.173242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.173242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":875,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616064+0000","last_change":"2024-02-23T00:45:22.441742+0000","last_active":"2024-02-23T00:46:28.616064+0000","last_peered":"2024-02-23T00:46:28.616064+0000","last_clean":"2024-02-23T00:46:28.616064+0000","last_became_active":"2024-02-23T00:40:45.191453+0000","last_became_peered":"2024-02-23T00:40:45.191453+0000","last_unstale":"2024-02-23T00:46:28.616064+0000","last_undegraded":"2024-02-23T00:46:28.616064+0000","last_fullsized":"2024-02-23T00:46:28.616064+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":626,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:22.441631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:39:39.963439+0000","last_clean_scrub_stamp":"2024-02-23T00:45:22.441631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"453'26","reported_seq":896,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710914+0000","last_change":"2024-02-23T00:46:15.254363+0000","last_active":"2024-02-23T00:46:28.710914+0000","last_peered":"2024-02-23T00:46:28.710914+0000","last_clean":"2024-02-23T00:46:28.710914+0000","last_became_active":"2024-02-23T00:27:19.016370+0000","last_became_peered":"2024-02-23T00:27:19.016370+0000","last_unstale":"2024-02-23T00:46:28.710914+0000","last_undegraded":"2024-02-23T00:46:28.710914+0000","last_fullsized":"2024-02-23T00:46:28.710914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:46:15.254281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:15.254281+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":874,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615467+0000","last_change":"2024-02-23T00:45:32.645969+0000","last_active":"2024-02-23T00:46:28.615467+0000","last_peered":"2024-02-23T00:46:28.615467+0000","last_clean":"2024-02-23T00:46:28.615467+0000","last_became_active":"2024-02-23T00:41:37.229335+0000","last_became_peered":"2024-02-23T00:41:37.229335+0000","last_unstale":"2024-02-23T00:46:28.615467+0000","last_undegraded":"2024-02-23T00:46:28.615467+0000","last_fullsized":"2024-02-23T00:46:28.615467+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:32.645873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:25.267773+0000","last_clean_scrub_stamp":"2024-02-23T00:45:32.645873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":845,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614038+0000","last_change":"2024-02-23T00:46:22.691597+0000","last_active":"2024-02-23T00:46:28.614038+0000","last_peered":"2024-02-23T00:46:28.614038+0000","last_clean":"2024-02-23T00:46:28.614038+0000","last_became_active":"2024-02-23T00:27:17.015320+0000","last_became_peered":"2024-02-23T00:27:17.015320+0000","last_unstale":"2024-02-23T00:46:28.614038+0000","last_undegraded":"2024-02-23T00:46:28.614038+0000","last_fullsized":"2024-02-23T00:46:28.614038+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:22.691462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:19.043451+0000","last_clean_scrub_stamp":"2024-02-23T00:46:22.691462+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":378,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472402+0000","last_change":"2024-02-23T00:45:41.312482+0000","last_active":"2024-02-23T00:46:28.472402+0000","last_peered":"2024-02-23T00:46:28.472402+0000","last_clean":"2024-02-23T00:46:28.472402+0000","last_became_active":"2024-02-23T00:42:39.396071+0000","last_became_peered":"2024-02-23T00:42:39.396071+0000","last_unstale":"2024-02-23T00:46:28.472402+0000","last_undegraded":"2024-02-23T00:46:28.472402+0000","last_fullsized":"2024-02-23T00:46:28.472402+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:41.312417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:41.312417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":888,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615108+0000","last_change":"2024-02-23T00:45:45.986880+0000","last_active":"2024-02-23T00:46:28.615108+0000","last_peered":"2024-02-23T00:46:28.615108+0000","last_clean":"2024-02-23T00:46:28.615108+0000","last_became_active":"2024-02-23T00:41:13.469953+0000","last_became_peered":"2024-02-23T00:41:13.469953+0000","last_unstale":"2024-02-23T00:46:28.615108+0000","last_undegraded":"2024-02-23T00:46:28.615108+0000","last_fullsized":"2024-02-23T00:46:28.615108+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:45.986798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:37:20.008726+0000","last_clean_scrub_stamp":"2024-02-23T00:45:45.986798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":801,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472183+0000","last_change":"2024-02-23T00:46:00.176954+0000","last_active":"2024-02-23T00:46:28.472183+0000","last_peered":"2024-02-23T00:46:28.472183+0000","last_clean":"2024-02-23T00:46:28.472183+0000","last_became_active":"2024-02-23T00:30:48.663482+0000","last_became_peered":"2024-02-23T00:30:48.663482+0000","last_unstale":"2024-02-23T00:46:28.472183+0000","last_undegraded":"2024-02-23T00:46:28.472183+0000","last_fullsized":"2024-02-23T00:46:28.472183+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:00.176870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.176870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":891,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473799+0000","last_change":"2024-02-23T00:45:56.268314+0000","last_active":"2024-02-23T00:46:28.473799+0000","last_peered":"2024-02-23T00:46:28.473799+0000","last_clean":"2024-02-23T00:46:28.473799+0000","last_became_active":"2024-02-23T00:40:38.061696+0000","last_became_peered":"2024-02-23T00:40:38.061696+0000","last_unstale":"2024-02-23T00:46:28.473799+0000","last_undegraded":"2024-02-23T00:46:28.473799+0000","last_fullsized":"2024-02-23T00:46:28.473799+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":375,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:56.268250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.742210+0000","last_clean_scrub_stamp":"2024-02-23T00:45:56.268250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"453'29","reported_seq":993,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711074+0000","last_change":"2024-02-23T00:46:21.322307+0000","last_active":"2024-02-23T00:46:28.711074+0000","last_peered":"2024-02-23T00:46:28.711074+0000","last_clean":"2024-02-23T00:46:28.711074+0000","last_became_active":"2024-02-23T00:41:40.219384+0000","last_became_peered":"2024-02-23T00:41:40.219384+0000","last_unstale":"2024-02-23T00:46:28.711074+0000","last_undegraded":"2024-02-23T00:46:28.711074+0000","last_fullsized":"2024-02-23T00:46:28.711074+0000","mapping_epoch":680,"log_start":"63'19","ondisk_log_start":"63'19","created":56,"last_epoch_clean":681,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:46:21.322229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:21.322229+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":904,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472603+0000","last_change":"2024-02-23T00:45:52.411968+0000","last_active":"2024-02-23T00:46:28.472603+0000","last_peered":"2024-02-23T00:46:28.472603+0000","last_clean":"2024-02-23T00:46:28.472603+0000","last_became_active":"2024-02-23T00:41:37.219000+0000","last_became_peered":"2024-02-23T00:41:37.219000+0000","last_unstale":"2024-02-23T00:46:28.472603+0000","last_undegraded":"2024-02-23T00:46:28.472603+0000","last_fullsized":"2024-02-23T00:46:28.472603+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:52.411836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:33.289752+0000","last_clean_scrub_stamp":"2024-02-23T00:45:52.411836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"78'6","reported_seq":1104,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710943+0000","last_change":"2024-02-23T00:45:14.546523+0000","last_active":"2024-02-23T00:46:28.710943+0000","last_peered":"2024-02-23T00:46:28.710943+0000","last_clean":"2024-02-23T00:46:28.710943+0000","last_became_active":"2024-02-23T00:42:39.401599+0000","last_became_peered":"2024-02-23T00:42:39.401599+0000","last_unstale":"2024-02-23T00:46:28.710943+0000","last_undegraded":"2024-02-23T00:46:28.710943+0000","last_fullsized":"2024-02-23T00:46:28.710943+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:45:14.546400+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:45:14.546400+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"62'1","reported_seq":905,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:33.612024+0000","last_change":"2024-02-23T00:46:33.612024+0000","last_active":"2024-02-23T00:46:33.612024+0000","last_peered":"2024-02-23T00:46:33.612024+0000","last_clean":"2024-02-23T00:46:33.612024+0000","last_became_active":"2024-02-23T00:41:21.226819+0000","last_became_peered":"2024-02-23T00:41:21.226819+0000","last_unstale":"2024-02-23T00:46:33.612024+0000","last_undegraded":"2024-02-23T00:46:33.612024+0000","last_fullsized":"2024-02-23T00:46:33.612024+0000","mapping_epoch":511,"log_start":"62'1","ondisk_log_start":"62'1","created":60,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'1","last_scrub_stamp":"2024-02-23T00:46:33.611875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:46:33.611875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":808,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473553+0000","last_change":"2024-02-23T00:45:58.314436+0000","last_active":"2024-02-23T00:46:28.473553+0000","last_peered":"2024-02-23T00:46:28.473553+0000","last_clean":"2024-02-23T00:46:28.473553+0000","last_became_active":"2024-02-23T00:30:48.685802+0000","last_became_peered":"2024-02-23T00:30:48.685802+0000","last_unstale":"2024-02-23T00:46:28.473553+0000","last_undegraded":"2024-02-23T00:46:28.473553+0000","last_fullsized":"2024-02-23T00:46:28.473553+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:58.314366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:58.314366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":784,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471409+0000","last_change":"2024-02-23T00:45:31.450822+0000","last_active":"2024-02-23T00:46:28.471409+0000","last_peered":"2024-02-23T00:46:28.471409+0000","last_clean":"2024-02-23T00:46:28.471409+0000","last_became_active":"2024-02-23T00:45:11.601050+0000","last_became_peered":"2024-02-23T00:45:11.601050+0000","last_unstale":"2024-02-23T00:46:28.471409+0000","last_undegraded":"2024-02-23T00:46:28.471409+0000","last_fullsized":"2024-02-23T00:46:28.471409+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:31.450749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:31.450749+0000","last_clean_scrub_stamp":"2024-02-23T00:45:31.450749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":207,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710719+0000","last_change":"2024-02-23T00:45:48.073661+0000","last_active":"2024-02-23T00:46:28.710719+0000","last_peered":"2024-02-23T00:46:28.710719+0000","last_clean":"2024-02-23T00:46:28.710719+0000","last_became_active":"2024-02-23T00:41:37.239608+0000","last_became_peered":"2024-02-23T00:41:37.239608+0000","last_unstale":"2024-02-23T00:46:28.710719+0000","last_undegraded":"2024-02-23T00:46:28.710719+0000","last_fullsized":"2024-02-23T00:46:28.710719+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:48.073592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:09.610960+0000","last_clean_scrub_stamp":"2024-02-23T00:45:48.073592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"513'48","reported_seq":1223,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.553172+0000","last_change":"2024-02-23T00:46:12.341957+0000","last_active":"2024-02-23T00:46:40.553172+0000","last_peered":"2024-02-23T00:46:40.553172+0000","last_clean":"2024-02-23T00:46:40.553172+0000","last_became_active":"2024-02-23T00:38:49.226898+0000","last_became_peered":"2024-02-23T00:38:49.226898+0000","last_unstale":"2024-02-23T00:46:40.553172+0000","last_undegraded":"2024-02-23T00:46:40.553172+0000","last_fullsized":"2024-02-23T00:46:40.553172+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'48","last_scrub_stamp":"2024-02-23T00:46:12.341836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:12.341836+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":62,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"601'2","reported_seq":244,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614155+0000","last_change":"2024-02-23T00:45:59.110227+0000","last_active":"2024-02-23T00:46:28.614155+0000","last_peered":"2024-02-23T00:46:28.614155+0000","last_clean":"2024-02-23T00:46:28.614155+0000","last_became_active":"2024-02-23T00:45:19.644956+0000","last_became_peered":"2024-02-23T00:45:19.644956+0000","last_unstale":"2024-02-23T00:46:28.614155+0000","last_undegraded":"2024-02-23T00:46:28.614155+0000","last_fullsized":"2024-02-23T00:46:28.614155+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"601'2","last_scrub_stamp":"2024-02-23T00:45:59.110136+0000","last_deep_scrub":"601'2","last_deep_scrub_stamp":"2024-02-23T00:40:21.186215+0000","last_clean_scrub_stamp":"2024-02-23T00:45:59.110136+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":410,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:29.328381+0000","last_change":"2024-02-23T00:46:29.328359+0000","last_active":"2024-02-23T00:46:29.328381+0000","last_peered":"2024-02-23T00:46:29.328381+0000","last_clean":"2024-02-23T00:46:29.328381+0000","last_became_active":"2024-02-23T00:41:17.220294+0000","last_became_peered":"2024-02-23T00:41:17.220294+0000","last_unstale":"2024-02-23T00:46:29.328381+0000","last_undegraded":"2024-02-23T00:46:29.328381+0000","last_fullsized":"2024-02-23T00:46:29.328381+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":658,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:29.328276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:38.521760+0000","last_clean_scrub_stamp":"2024-02-23T00:46:29.328276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":813,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:45.104049+0000","last_change":"2024-02-23T00:46:45.104049+0000","last_active":"2024-02-23T00:46:45.104049+0000","last_peered":"2024-02-23T00:46:45.104049+0000","last_clean":"2024-02-23T00:46:45.104049+0000","last_became_active":"2024-02-23T00:30:48.694555+0000","last_became_peered":"2024-02-23T00:30:48.694555+0000","last_unstale":"2024-02-23T00:46:45.104049+0000","last_undegraded":"2024-02-23T00:46:45.104049+0000","last_fullsized":"2024-02-23T00:46:45.104049+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:45.103980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:45.103980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":782,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614561+0000","last_change":"2024-02-23T00:46:03.077343+0000","last_active":"2024-02-23T00:46:28.614561+0000","last_peered":"2024-02-23T00:46:28.614561+0000","last_clean":"2024-02-23T00:46:28.614561+0000","last_became_active":"2024-02-23T00:42:39.399113+0000","last_became_peered":"2024-02-23T00:42:39.399113+0000","last_unstale":"2024-02-23T00:46:28.614561+0000","last_undegraded":"2024-02-23T00:46:28.614561+0000","last_fullsized":"2024-02-23T00:46:28.614561+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":737,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:03.077257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:46:03.077257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"783'22","reported_seq":495,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.551990+0000","last_change":"2024-02-23T00:46:09.959333+0000","last_active":"2024-02-23T00:46:35.551990+0000","last_peered":"2024-02-23T00:46:35.551990+0000","last_clean":"2024-02-23T00:46:35.551990+0000","last_became_active":"2024-02-23T00:43:40.280015+0000","last_became_peered":"2024-02-23T00:43:40.280015+0000","last_unstale":"2024-02-23T00:46:35.551990+0000","last_undegraded":"2024-02-23T00:46:35.551990+0000","last_fullsized":"2024-02-23T00:46:35.551990+0000","mapping_epoch":782,"log_start":"606'17","ondisk_log_start":"606'17","created":58,"last_epoch_clean":783,"parent":"0.0","parent_split_bits":0,"last_scrub":"783'22","last_scrub_stamp":"2024-02-23T00:46:09.959246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:09.959246+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"832'32","reported_seq":301,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:30.488143+0000","last_change":"2024-02-23T00:46:30.488114+0000","last_active":"2024-02-23T00:46:30.488143+0000","last_peered":"2024-02-23T00:46:30.488143+0000","last_clean":"2024-02-23T00:46:30.488143+0000","last_became_active":"2024-02-23T00:40:23.194277+0000","last_became_peered":"2024-02-23T00:40:23.194277+0000","last_unstale":"2024-02-23T00:46:30.488143+0000","last_undegraded":"2024-02-23T00:46:30.488143+0000","last_fullsized":"2024-02-23T00:46:30.488143+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":605,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'32","last_scrub_stamp":"2024-02-23T00:46:30.488040+0000","last_deep_scrub":"832'32","last_deep_scrub_stamp":"2024-02-23T00:45:10.528461+0000","last_clean_scrub_stamp":"2024-02-23T00:46:30.488040+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":866,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710574+0000","last_change":"2024-02-23T00:46:03.254011+0000","last_active":"2024-02-23T00:46:28.710574+0000","last_peered":"2024-02-23T00:46:28.710574+0000","last_clean":"2024-02-23T00:46:28.710574+0000","last_became_active":"2024-02-23T00:41:37.229003+0000","last_became_peered":"2024-02-23T00:41:37.229003+0000","last_unstale":"2024-02-23T00:46:28.710574+0000","last_undegraded":"2024-02-23T00:46:28.710574+0000","last_fullsized":"2024-02-23T00:46:28.710574+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:03.253940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:51.664367+0000","last_clean_scrub_stamp":"2024-02-23T00:46:03.253940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":202,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710158+0000","last_change":"2024-02-23T00:44:59.925931+0000","last_active":"2024-02-23T00:46:28.710158+0000","last_peered":"2024-02-23T00:46:28.710158+0000","last_clean":"2024-02-23T00:46:28.710158+0000","last_became_active":"2024-02-23T00:41:37.248459+0000","last_became_peered":"2024-02-23T00:41:37.248459+0000","last_unstale":"2024-02-23T00:46:28.710158+0000","last_undegraded":"2024-02-23T00:46:28.710158+0000","last_fullsized":"2024-02-23T00:46:28.710158+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:44:59.925826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:44:59.925826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":799,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471261+0000","last_change":"2024-02-23T00:46:08.139421+0000","last_active":"2024-02-23T00:46:28.471261+0000","last_peered":"2024-02-23T00:46:28.471261+0000","last_clean":"2024-02-23T00:46:28.471261+0000","last_became_active":"2024-02-23T00:30:48.616287+0000","last_became_peered":"2024-02-23T00:30:48.616287+0000","last_unstale":"2024-02-23T00:46:28.471261+0000","last_undegraded":"2024-02-23T00:46:28.471261+0000","last_fullsized":"2024-02-23T00:46:28.471261+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:08.139334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:36:42.093181+0000","last_clean_scrub_stamp":"2024-02-23T00:46:08.139334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616320+0000","last_change":"2024-02-23T00:45:56.714581+0000","last_active":"2024-02-23T00:46:28.616320+0000","last_peered":"2024-02-23T00:46:28.616320+0000","last_clean":"2024-02-23T00:46:28.616320+0000","last_became_active":"2024-02-23T00:42:41.402111+0000","last_became_peered":"2024-02-23T00:42:41.402111+0000","last_unstale":"2024-02-23T00:46:28.616320+0000","last_undegraded":"2024-02-23T00:46:28.616320+0000","last_fullsized":"2024-02-23T00:46:28.616320+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:56.714506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:56.714506+0000","last_clean_scrub_stamp":"2024-02-23T00:45:56.714506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":900,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472111+0000","last_change":"2024-02-23T00:45:57.297956+0000","last_active":"2024-02-23T00:46:28.472111+0000","last_peered":"2024-02-23T00:46:28.472111+0000","last_clean":"2024-02-23T00:46:28.472111+0000","last_became_active":"2024-02-23T00:41:37.239598+0000","last_became_peered":"2024-02-23T00:41:37.239598+0000","last_unstale":"2024-02-23T00:46:28.472111+0000","last_undegraded":"2024-02-23T00:46:28.472111+0000","last_fullsized":"2024-02-23T00:46:28.472111+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:57.297859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.380671+0000","last_clean_scrub_stamp":"2024-02-23T00:45:57.297859+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"453'68","reported_seq":924,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471825+0000","last_change":"2024-02-23T00:45:20.364101+0000","last_active":"2024-02-23T00:46:28.471825+0000","last_peered":"2024-02-23T00:46:28.471825+0000","last_clean":"2024-02-23T00:46:28.471825+0000","last_became_active":"2024-02-23T00:45:11.594930+0000","last_became_peered":"2024-02-23T00:45:11.594930+0000","last_unstale":"2024-02-23T00:46:28.471825+0000","last_undegraded":"2024-02-23T00:46:28.471825+0000","last_fullsized":"2024-02-23T00:46:28.471825+0000","mapping_epoch":833,"log_start":"278'50","ondisk_log_start":"278'50","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:45:20.364016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:20.364016+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"55'2","reported_seq":903,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472294+0000","last_change":"2024-02-23T00:46:06.112588+0000","last_active":"2024-02-23T00:46:28.472294+0000","last_peered":"2024-02-23T00:46:28.472294+0000","last_clean":"2024-02-23T00:46:28.472294+0000","last_became_active":"2024-02-23T00:41:37.238708+0000","last_became_peered":"2024-02-23T00:41:37.238708+0000","last_unstale":"2024-02-23T00:46:28.472294+0000","last_undegraded":"2024-02-23T00:46:28.472294+0000","last_fullsized":"2024-02-23T00:46:28.472294+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'2","last_scrub_stamp":"2024-02-23T00:46:06.112489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:06.112489+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":387,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472792+0000","last_change":"2024-02-23T00:45:42.329411+0000","last_active":"2024-02-23T00:46:28.472792+0000","last_peered":"2024-02-23T00:46:28.472792+0000","last_clean":"2024-02-23T00:46:28.472792+0000","last_became_active":"2024-02-23T00:40:17.214288+0000","last_became_peered":"2024-02-23T00:40:17.214288+0000","last_unstale":"2024-02-23T00:46:28.472792+0000","last_undegraded":"2024-02-23T00:46:28.472792+0000","last_fullsized":"2024-02-23T00:46:28.472792+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":598,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:42.329312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:22.009070+0000","last_clean_scrub_stamp":"2024-02-23T00:45:42.329312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":789,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.613866+0000","last_change":"2024-02-23T00:45:23.045215+0000","last_active":"2024-02-23T00:46:28.613866+0000","last_peered":"2024-02-23T00:46:28.613866+0000","last_clean":"2024-02-23T00:46:28.613866+0000","last_became_active":"2024-02-23T00:30:48.673855+0000","last_became_peered":"2024-02-23T00:30:48.673855+0000","last_unstale":"2024-02-23T00:46:28.613866+0000","last_undegraded":"2024-02-23T00:46:28.613866+0000","last_fullsized":"2024-02-23T00:46:28.613866+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:23.045097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:23.045097+0000","last_clean_scrub_stamp":"2024-02-23T00:45:23.045097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615385+0000","last_change":"2024-02-23T00:46:21.617550+0000","last_active":"2024-02-23T00:46:28.615385+0000","last_peered":"2024-02-23T00:46:28.615385+0000","last_clean":"2024-02-23T00:46:28.615385+0000","last_became_active":"2024-02-23T00:42:41.399155+0000","last_became_peered":"2024-02-23T00:42:41.399155+0000","last_unstale":"2024-02-23T00:46:28.615385+0000","last_undegraded":"2024-02-23T00:46:28.615385+0000","last_fullsized":"2024-02-23T00:46:28.615385+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:21.617465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:21.617465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"453'39","reported_seq":638,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711182+0000","last_change":"2024-02-23T00:45:07.827481+0000","last_active":"2024-02-23T00:46:28.711182+0000","last_peered":"2024-02-23T00:46:28.711182+0000","last_clean":"2024-02-23T00:46:28.711182+0000","last_became_active":"2024-02-23T00:41:38.217979+0000","last_became_peered":"2024-02-23T00:41:38.217979+0000","last_unstale":"2024-02-23T00:46:28.711182+0000","last_undegraded":"2024-02-23T00:46:28.711182+0000","last_fullsized":"2024-02-23T00:46:28.711182+0000","mapping_epoch":678,"log_start":"63'21","ondisk_log_start":"63'21","created":481,"last_epoch_clean":679,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'39","last_scrub_stamp":"2024-02-23T00:45:07.827410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:07.827410+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":880,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.160114+0000","last_change":"2024-02-23T00:46:35.160114+0000","last_active":"2024-02-23T00:46:35.160114+0000","last_peered":"2024-02-23T00:46:35.160114+0000","last_clean":"2024-02-23T00:46:35.160114+0000","last_became_active":"2024-02-23T00:41:37.238750+0000","last_became_peered":"2024-02-23T00:41:37.238750+0000","last_unstale":"2024-02-23T00:46:35.160114+0000","last_undegraded":"2024-02-23T00:46:35.160114+0000","last_fullsized":"2024-02-23T00:46:35.160114+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:35.160020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:35.160020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":374,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472865+0000","last_change":"2024-02-23T00:46:24.270022+0000","last_active":"2024-02-23T00:46:28.472865+0000","last_peered":"2024-02-23T00:46:28.472865+0000","last_clean":"2024-02-23T00:46:28.472865+0000","last_became_active":"2024-02-23T00:38:49.221526+0000","last_became_peered":"2024-02-23T00:38:49.221526+0000","last_unstale":"2024-02-23T00:46:28.472865+0000","last_undegraded":"2024-02-23T00:46:28.472865+0000","last_fullsized":"2024-02-23T00:46:28.472865+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:24.269957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:24.269957+0000","last_clean_scrub_stamp":"2024-02-23T00:46:24.269957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":369,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472835+0000","last_change":"2024-02-23T00:46:21.255605+0000","last_active":"2024-02-23T00:46:28.472835+0000","last_peered":"2024-02-23T00:46:28.472835+0000","last_clean":"2024-02-23T00:46:28.472835+0000","last_became_active":"2024-02-23T00:38:49.229591+0000","last_became_peered":"2024-02-23T00:38:49.229591+0000","last_unstale":"2024-02-23T00:46:28.472835+0000","last_undegraded":"2024-02-23T00:46:28.472835+0000","last_fullsized":"2024-02-23T00:46:28.472835+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:21.255543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:02.753328+0000","last_clean_scrub_stamp":"2024-02-23T00:46:21.255543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"78'6","reported_seq":268,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710268+0000","last_change":"2024-02-23T00:45:55.446110+0000","last_active":"2024-02-23T00:46:28.710268+0000","last_peered":"2024-02-23T00:46:28.710268+0000","last_clean":"2024-02-23T00:46:28.710268+0000","last_became_active":"2024-02-23T00:42:41.406233+0000","last_became_peered":"2024-02-23T00:42:41.406233+0000","last_unstale":"2024-02-23T00:46:28.710268+0000","last_undegraded":"2024-02-23T00:46:28.710268+0000","last_fullsized":"2024-02-23T00:46:28.710268+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:45:55.446002+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:43:07.690939+0000","last_clean_scrub_stamp":"2024-02-23T00:45:55.446002+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"453'29","reported_seq":977,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472512+0000","last_change":"2024-02-23T00:45:20.573104+0000","last_active":"2024-02-23T00:46:28.472512+0000","last_peered":"2024-02-23T00:46:28.472512+0000","last_clean":"2024-02-23T00:46:28.472512+0000","last_became_active":"2024-02-23T00:40:51.316452+0000","last_became_peered":"2024-02-23T00:40:51.316452+0000","last_unstale":"2024-02-23T00:46:28.472512+0000","last_undegraded":"2024-02-23T00:46:28.472512+0000","last_fullsized":"2024-02-23T00:46:28.472512+0000","mapping_epoch":631,"log_start":"63'19","ondisk_log_start":"63'19","created":481,"last_epoch_clean":632,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'29","last_scrub_stamp":"2024-02-23T00:45:20.573002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:20.573002+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":858,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:38.373376+0000","last_change":"2024-02-23T00:46:38.373376+0000","last_active":"2024-02-23T00:46:38.373376+0000","last_peered":"2024-02-23T00:46:38.373376+0000","last_clean":"2024-02-23T00:46:38.373376+0000","last_became_active":"2024-02-23T00:27:17.008934+0000","last_became_peered":"2024-02-23T00:27:17.008934+0000","last_unstale":"2024-02-23T00:46:38.373376+0000","last_undegraded":"2024-02-23T00:46:38.373376+0000","last_fullsized":"2024-02-23T00:46:38.373376+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:38.373274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:42.431641+0000","last_clean_scrub_stamp":"2024-02-23T00:46:38.373274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":854,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:29.634920+0000","last_change":"2024-02-23T00:46:29.634886+0000","last_active":"2024-02-23T00:46:29.634920+0000","last_peered":"2024-02-23T00:46:29.634920+0000","last_clean":"2024-02-23T00:46:29.634920+0000","last_became_active":"2024-02-23T00:38:49.189960+0000","last_became_peered":"2024-02-23T00:38:49.189960+0000","last_unstale":"2024-02-23T00:46:29.634920+0000","last_undegraded":"2024-02-23T00:46:29.634920+0000","last_fullsized":"2024-02-23T00:46:29.634920+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:29.634812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:29.634812+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":818,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711019+0000","last_change":"2024-02-23T00:45:57.371599+0000","last_active":"2024-02-23T00:46:28.711019+0000","last_peered":"2024-02-23T00:46:28.711019+0000","last_clean":"2024-02-23T00:46:28.711019+0000","last_became_active":"2024-02-23T00:38:49.195243+0000","last_became_peered":"2024-02-23T00:38:49.195243+0000","last_unstale":"2024-02-23T00:46:28.711019+0000","last_undegraded":"2024-02-23T00:46:28.711019+0000","last_fullsized":"2024-02-23T00:46:28.711019+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:57.371524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:32.101913+0000","last_clean_scrub_stamp":"2024-02-23T00:45:57.371524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615909+0000","last_change":"2024-02-23T00:46:00.681933+0000","last_active":"2024-02-23T00:46:28.615909+0000","last_peered":"2024-02-23T00:46:28.615909+0000","last_clean":"2024-02-23T00:46:28.615909+0000","last_became_active":"2024-02-23T00:42:41.402543+0000","last_became_peered":"2024-02-23T00:42:41.402543+0000","last_unstale":"2024-02-23T00:46:28.615909+0000","last_undegraded":"2024-02-23T00:46:28.615909+0000","last_fullsized":"2024-02-23T00:46:28.615909+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:00.681827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.681827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"688'39","reported_seq":392,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472693+0000","last_change":"2024-02-23T00:46:16.129153+0000","last_active":"2024-02-23T00:46:28.472693+0000","last_peered":"2024-02-23T00:46:28.472693+0000","last_clean":"2024-02-23T00:46:28.472693+0000","last_became_active":"2024-02-23T00:38:49.224617+0000","last_became_peered":"2024-02-23T00:38:49.224617+0000","last_unstale":"2024-02-23T00:46:28.472693+0000","last_undegraded":"2024-02-23T00:46:28.472693+0000","last_fullsized":"2024-02-23T00:46:28.472693+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":514,"parent":"0.0","parent_split_bits":0,"last_scrub":"688'39","last_scrub_stamp":"2024-02-23T00:46:16.129079+0000","last_deep_scrub":"688'39","last_deep_scrub_stamp":"2024-02-23T00:44:57.844469+0000","last_clean_scrub_stamp":"2024-02-23T00:46:16.129079+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":45,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614404+0000","last_change":"2024-02-23T00:46:28.613906+0000","last_active":"2024-02-23T00:46:28.614404+0000","last_peered":"2024-02-23T00:46:28.614404+0000","last_clean":"2024-02-23T00:46:28.614404+0000","last_became_active":"2024-02-23T00:38:49.185972+0000","last_became_peered":"2024-02-23T00:38:49.185972+0000","last_unstale":"2024-02-23T00:46:28.614404+0000","last_undegraded":"2024-02-23T00:46:28.614404+0000","last_fullsized":"2024-02-23T00:46:28.614404+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:28.613768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:49.012698+0000","last_clean_scrub_stamp":"2024-02-23T00:46:28.613768+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"681'12","reported_seq":1107,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.553605+0000","last_change":"2024-02-23T00:46:00.062864+0000","last_active":"2024-02-23T00:46:40.553605+0000","last_peered":"2024-02-23T00:46:40.553605+0000","last_clean":"2024-02-23T00:46:40.553605+0000","last_became_active":"2024-02-23T00:41:37.228441+0000","last_became_peered":"2024-02-23T00:41:37.228441+0000","last_unstale":"2024-02-23T00:46:40.553605+0000","last_undegraded":"2024-02-23T00:46:40.553605+0000","last_fullsized":"2024-02-23T00:46:40.553605+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'12","last_scrub_stamp":"2024-02-23T00:46:00.062784+0000","last_deep_scrub":"76'9","last_deep_scrub_stamp":"2024-02-23T00:32:56.376370+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.062784+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":789,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614966+0000","last_change":"2024-02-23T00:45:28.135319+0000","last_active":"2024-02-23T00:46:28.614966+0000","last_peered":"2024-02-23T00:46:28.614966+0000","last_clean":"2024-02-23T00:46:28.614966+0000","last_became_active":"2024-02-23T00:30:48.685635+0000","last_became_peered":"2024-02-23T00:30:48.685635+0000","last_unstale":"2024-02-23T00:46:28.614966+0000","last_undegraded":"2024-02-23T00:46:28.614966+0000","last_fullsized":"2024-02-23T00:46:28.614966+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:28.135231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:28.135231+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":141,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710240+0000","last_change":"2024-02-23T00:45:49.038011+0000","last_active":"2024-02-23T00:46:28.710240+0000","last_peered":"2024-02-23T00:46:28.710240+0000","last_clean":"2024-02-23T00:46:28.710240+0000","last_became_active":"2024-02-23T00:42:41.402683+0000","last_became_peered":"2024-02-23T00:42:41.402683+0000","last_unstale":"2024-02-23T00:46:28.710240+0000","last_undegraded":"2024-02-23T00:46:28.710240+0000","last_fullsized":"2024-02-23T00:46:28.710240+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:49.037943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:49.037943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"453'26","reported_seq":925,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614685+0000","last_change":"2024-02-23T00:45:57.117759+0000","last_active":"2024-02-23T00:46:28.614685+0000","last_peered":"2024-02-23T00:46:28.614685+0000","last_clean":"2024-02-23T00:46:28.614685+0000","last_became_active":"2024-02-23T00:38:49.227038+0000","last_became_peered":"2024-02-23T00:38:49.227038+0000","last_unstale":"2024-02-23T00:46:28.614685+0000","last_undegraded":"2024-02-23T00:46:28.614685+0000","last_fullsized":"2024-02-23T00:46:28.614685+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'26","last_scrub_stamp":"2024-02-23T00:45:57.117677+0000","last_deep_scrub":"453'26","last_deep_scrub_stamp":"2024-02-23T00:45:57.117677+0000","last_clean_scrub_stamp":"2024-02-23T00:45:57.117677+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":207,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711406+0000","last_change":"2024-02-23T00:46:19.987965+0000","last_active":"2024-02-23T00:46:28.711406+0000","last_peered":"2024-02-23T00:46:28.711406+0000","last_clean":"2024-02-23T00:46:28.711406+0000","last_became_active":"2024-02-23T00:41:37.235746+0000","last_became_peered":"2024-02-23T00:41:37.235746+0000","last_unstale":"2024-02-23T00:46:28.711406+0000","last_undegraded":"2024-02-23T00:46:28.711406+0000","last_fullsized":"2024-02-23T00:46:28.711406+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:19.987886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:53.903007+0000","last_clean_scrub_stamp":"2024-02-23T00:46:19.987886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"513'11","reported_seq":469,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.553240+0000","last_change":"2024-02-23T00:46:05.129838+0000","last_active":"2024-02-23T00:46:40.553240+0000","last_peered":"2024-02-23T00:46:40.553240+0000","last_clean":"2024-02-23T00:46:40.553240+0000","last_became_active":"2024-02-23T00:38:49.180532+0000","last_became_peered":"2024-02-23T00:38:49.180532+0000","last_unstale":"2024-02-23T00:46:40.553240+0000","last_undegraded":"2024-02-23T00:46:40.553240+0000","last_fullsized":"2024-02-23T00:46:40.553240+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"513'11","last_scrub_stamp":"2024-02-23T00:46:05.129760+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.098442+0000","last_clean_scrub_stamp":"2024-02-23T00:46:05.129760+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":371,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472920+0000","last_change":"2024-02-23T00:46:04.136581+0000","last_active":"2024-02-23T00:46:28.472920+0000","last_peered":"2024-02-23T00:46:28.472920+0000","last_clean":"2024-02-23T00:46:28.472920+0000","last_became_active":"2024-02-23T00:38:49.227170+0000","last_became_peered":"2024-02-23T00:38:49.227170+0000","last_unstale":"2024-02-23T00:46:28.472920+0000","last_undegraded":"2024-02-23T00:46:28.472920+0000","last_fullsized":"2024-02-23T00:46:28.472920+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:04.136496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:04.136496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616138+0000","last_change":"2024-02-23T00:45:45.751087+0000","last_active":"2024-02-23T00:46:28.616138+0000","last_peered":"2024-02-23T00:46:28.616138+0000","last_clean":"2024-02-23T00:46:28.616138+0000","last_became_active":"2024-02-23T00:42:41.398941+0000","last_became_peered":"2024-02-23T00:42:41.398941+0000","last_unstale":"2024-02-23T00:46:28.616138+0000","last_undegraded":"2024-02-23T00:46:28.616138+0000","last_fullsized":"2024-02-23T00:46:28.616138+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:45.751013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:41:55.024705+0000","last_clean_scrub_stamp":"2024-02-23T00:45:45.751013+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"453'15","reported_seq":867,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711632+0000","last_change":"2024-02-23T00:46:27.416846+0000","last_active":"2024-02-23T00:46:28.711632+0000","last_peered":"2024-02-23T00:46:28.711632+0000","last_clean":"2024-02-23T00:46:28.711632+0000","last_became_active":"2024-02-23T00:27:19.023015+0000","last_became_peered":"2024-02-23T00:27:19.023015+0000","last_unstale":"2024-02-23T00:46:28.711632+0000","last_undegraded":"2024-02-23T00:46:28.711632+0000","last_fullsized":"2024-02-23T00:46:28.711632+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'15","last_scrub_stamp":"2024-02-23T00:46:27.416778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:27.416778+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":880,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710828+0000","last_change":"2024-02-23T00:46:10.362620+0000","last_active":"2024-02-23T00:46:28.710828+0000","last_peered":"2024-02-23T00:46:28.710828+0000","last_clean":"2024-02-23T00:46:28.710828+0000","last_became_active":"2024-02-23T00:38:49.194721+0000","last_became_peered":"2024-02-23T00:38:49.194721+0000","last_unstale":"2024-02-23T00:46:28.710828+0000","last_undegraded":"2024-02-23T00:46:28.710828+0000","last_fullsized":"2024-02-23T00:46:28.710828+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:10.362544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:10.362544+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":393,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:30.346073+0000","last_change":"2024-02-23T00:46:30.346046+0000","last_active":"2024-02-23T00:46:30.346073+0000","last_peered":"2024-02-23T00:46:30.346073+0000","last_clean":"2024-02-23T00:46:30.346073+0000","last_became_active":"2024-02-23T00:41:37.252612+0000","last_became_peered":"2024-02-23T00:41:37.252612+0000","last_unstale":"2024-02-23T00:46:30.346073+0000","last_undegraded":"2024-02-23T00:46:30.346073+0000","last_fullsized":"2024-02-23T00:46:30.346073+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:30.345975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:27.197290+0000","last_clean_scrub_stamp":"2024-02-23T00:46:30.345975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":823,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:43.723007+0000","last_change":"2024-02-23T00:46:43.723007+0000","last_active":"2024-02-23T00:46:43.723007+0000","last_peered":"2024-02-23T00:46:43.723007+0000","last_clean":"2024-02-23T00:46:43.723007+0000","last_became_active":"2024-02-23T00:41:37.238185+0000","last_became_peered":"2024-02-23T00:41:37.238185+0000","last_unstale":"2024-02-23T00:46:43.723007+0000","last_undegraded":"2024-02-23T00:46:43.723007+0000","last_fullsized":"2024-02-23T00:46:43.723007+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:43.722924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:43.722924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"78'6","reported_seq":41,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471481+0000","last_change":"2024-02-23T00:45:29.458259+0000","last_active":"2024-02-23T00:46:28.471481+0000","last_peered":"2024-02-23T00:46:28.471481+0000","last_clean":"2024-02-23T00:46:28.471481+0000","last_became_active":"2024-02-23T00:45:11.600802+0000","last_became_peered":"2024-02-23T00:45:11.600802+0000","last_unstale":"2024-02-23T00:46:28.471481+0000","last_undegraded":"2024-02-23T00:46:28.471481+0000","last_fullsized":"2024-02-23T00:46:28.471481+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'6","last_scrub_stamp":"2024-02-23T00:45:29.458182+0000","last_deep_scrub":"78'6","last_deep_scrub_stamp":"2024-02-23T00:28:52.653823+0000","last_clean_scrub_stamp":"2024-02-23T00:45:29.458182+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"453'19","reported_seq":902,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471560+0000","last_change":"2024-02-23T00:46:19.023703+0000","last_active":"2024-02-23T00:46:28.471560+0000","last_peered":"2024-02-23T00:46:28.471560+0000","last_clean":"2024-02-23T00:46:28.471560+0000","last_became_active":"2024-02-23T00:28:41.204460+0000","last_became_peered":"2024-02-23T00:28:41.204460+0000","last_unstale":"2024-02-23T00:46:28.471560+0000","last_undegraded":"2024-02-23T00:46:28.471560+0000","last_fullsized":"2024-02-23T00:46:28.471560+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'19","last_scrub_stamp":"2024-02-23T00:46:19.023606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:19.023606+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":863,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710646+0000","last_change":"2024-02-23T00:46:28.433283+0000","last_active":"2024-02-23T00:46:28.710646+0000","last_peered":"2024-02-23T00:46:28.710646+0000","last_clean":"2024-02-23T00:46:28.710646+0000","last_became_active":"2024-02-23T00:41:37.229726+0000","last_became_peered":"2024-02-23T00:41:37.229726+0000","last_unstale":"2024-02-23T00:46:28.710646+0000","last_undegraded":"2024-02-23T00:46:28.710646+0000","last_fullsized":"2024-02-23T00:46:28.710646+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:28.433175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:29:02.711284+0000","last_clean_scrub_stamp":"2024-02-23T00:46:28.433175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":858,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:44.529861+0000","last_change":"2024-02-23T00:46:44.529861+0000","last_active":"2024-02-23T00:46:44.529861+0000","last_peered":"2024-02-23T00:46:44.529861+0000","last_clean":"2024-02-23T00:46:44.529861+0000","last_became_active":"2024-02-23T00:38:49.224571+0000","last_became_peered":"2024-02-23T00:38:49.224571+0000","last_unstale":"2024-02-23T00:46:44.529861+0000","last_undegraded":"2024-02-23T00:46:44.529861+0000","last_fullsized":"2024-02-23T00:46:44.529861+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:44.529760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:33:46.151986+0000","last_clean_scrub_stamp":"2024-02-23T00:46:44.529760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1","version":"855'8","reported_seq":850,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710837+0000","last_change":"2024-02-23T00:46:27.306881+0000","last_active":"2024-02-23T00:46:28.710837+0000","last_peered":"2024-02-23T00:46:28.710837+0000","last_clean":"2024-02-23T00:46:28.710837+0000","last_became_active":"2024-02-23T00:38:49.207189+0000","last_became_peered":"2024-02-23T00:38:49.207189+0000","last_unstale":"2024-02-23T00:46:28.710837+0000","last_undegraded":"2024-02-23T00:46:28.710837+0000","last_fullsized":"2024-02-23T00:46:28.710837+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"855'8","last_scrub_stamp":"2024-02-23T00:46:27.306802+0000","last_deep_scrub":"855'8","last_deep_scrub_stamp":"2024-02-23T00:46:27.306802+0000","last_clean_scrub_stamp":"2024-02-23T00:46:27.306802+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"453'10","reported_seq":395,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:39.338322+0000","last_change":"2024-02-23T00:46:39.338322+0000","last_active":"2024-02-23T00:46:39.338322+0000","last_peered":"2024-02-23T00:46:39.338322+0000","last_clean":"2024-02-23T00:46:39.338322+0000","last_became_active":"2024-02-23T00:43:42.256788+0000","last_became_peered":"2024-02-23T00:43:42.256788+0000","last_unstale":"2024-02-23T00:46:39.338322+0000","last_undegraded":"2024-02-23T00:46:39.338322+0000","last_fullsized":"2024-02-23T00:46:39.338322+0000","mapping_epoch":784,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":785,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'10","last_scrub_stamp":"2024-02-23T00:46:39.338207+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-02-23T00:28:20.075487+0000","last_clean_scrub_stamp":"2024-02-23T00:46:39.338207+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":840,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614830+0000","last_change":"2024-02-23T00:45:22.052575+0000","last_active":"2024-02-23T00:46:28.614830+0000","last_peered":"2024-02-23T00:46:28.614830+0000","last_clean":"2024-02-23T00:46:28.614830+0000","last_became_active":"2024-02-23T00:27:17.013196+0000","last_became_peered":"2024-02-23T00:27:17.013196+0000","last_unstale":"2024-02-23T00:46:28.614830+0000","last_undegraded":"2024-02-23T00:46:28.614830+0000","last_fullsized":"2024-02-23T00:46:28.614830+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:22.052497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:45:22.052497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":849,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710576+0000","last_change":"2024-02-23T00:45:49.313466+0000","last_active":"2024-02-23T00:46:28.710576+0000","last_peered":"2024-02-23T00:46:28.710576+0000","last_clean":"2024-02-23T00:46:28.710576+0000","last_became_active":"2024-02-23T00:27:21.027549+0000","last_became_peered":"2024-02-23T00:27:21.027549+0000","last_unstale":"2024-02-23T00:46:28.710576+0000","last_undegraded":"2024-02-23T00:46:28.710576+0000","last_fullsized":"2024-02-23T00:46:28.710576+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:49.313391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:49.313391+0000","last_clean_scrub_stamp":"2024-02-23T00:45:49.313391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":786,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711712+0000","last_change":"2024-02-23T00:46:01.293672+0000","last_active":"2024-02-23T00:46:28.711712+0000","last_peered":"2024-02-23T00:46:28.711712+0000","last_clean":"2024-02-23T00:46:28.711712+0000","last_became_active":"2024-02-23T00:30:48.695955+0000","last_became_peered":"2024-02-23T00:30:48.695955+0000","last_unstale":"2024-02-23T00:46:28.711712+0000","last_undegraded":"2024-02-23T00:46:28.711712+0000","last_fullsized":"2024-02-23T00:46:28.711712+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:01.293605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:01.293605+0000","last_clean_scrub_stamp":"2024-02-23T00:46:01.293605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":142,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614569+0000","last_change":"2024-02-23T00:46:20.701211+0000","last_active":"2024-02-23T00:46:28.614569+0000","last_peered":"2024-02-23T00:46:28.614569+0000","last_clean":"2024-02-23T00:46:28.614569+0000","last_became_active":"2024-02-23T00:42:41.391484+0000","last_became_peered":"2024-02-23T00:42:41.391484+0000","last_unstale":"2024-02-23T00:46:28.614569+0000","last_undegraded":"2024-02-23T00:46:28.614569+0000","last_fullsized":"2024-02-23T00:46:28.614569+0000","mapping_epoch":738,"log_start":"0'0","ondisk_log_start":"0'0","created":736,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:20.701134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:43:17.077581+0000","last_clean_scrub_stamp":"2024-02-23T00:46:20.701134+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"453'32","reported_seq":386,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473332+0000","last_change":"2024-02-23T00:45:32.492526+0000","last_active":"2024-02-23T00:46:28.473332+0000","last_peered":"2024-02-23T00:46:28.473332+0000","last_clean":"2024-02-23T00:46:28.473332+0000","last_became_active":"2024-02-23T00:38:49.194324+0000","last_became_peered":"2024-02-23T00:38:49.194324+0000","last_unstale":"2024-02-23T00:46:28.473332+0000","last_undegraded":"2024-02-23T00:46:28.473332+0000","last_fullsized":"2024-02-23T00:46:28.473332+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'32","last_scrub_stamp":"2024-02-23T00:45:32.492427+0000","last_deep_scrub":"79'24","last_deep_scrub_stamp":"2024-02-23T00:28:57.039660+0000","last_clean_scrub_stamp":"2024-02-23T00:45:32.492427+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":43,"num_read_kb":35,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":857,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:30.313531+0000","last_change":"2024-02-23T00:46:30.313487+0000","last_active":"2024-02-23T00:46:30.313531+0000","last_peered":"2024-02-23T00:46:30.313531+0000","last_clean":"2024-02-23T00:46:30.313531+0000","last_became_active":"2024-02-23T00:27:17.010432+0000","last_became_peered":"2024-02-23T00:27:17.010432+0000","last_unstale":"2024-02-23T00:46:30.313531+0000","last_undegraded":"2024-02-23T00:46:30.313531+0000","last_fullsized":"2024-02-23T00:46:30.313531+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:30.313398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:25.724375+0000","last_clean_scrub_stamp":"2024-02-23T00:46:30.313398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":106,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473020+0000","last_change":"2024-02-23T00:46:18.131373+0000","last_active":"2024-02-23T00:46:28.473020+0000","last_peered":"2024-02-23T00:46:28.473020+0000","last_clean":"2024-02-23T00:46:28.473020+0000","last_became_active":"2024-02-23T00:43:21.498988+0000","last_became_peered":"2024-02-23T00:43:21.498988+0000","last_unstale":"2024-02-23T00:46:28.473020+0000","last_undegraded":"2024-02-23T00:46:28.473020+0000","last_fullsized":"2024-02-23T00:46:28.473020+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:18.131285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:38:16.055159+0000","last_clean_scrub_stamp":"2024-02-23T00:46:18.131285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":377,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:42.756625+0000","last_change":"2024-02-23T00:46:42.756625+0000","last_active":"2024-02-23T00:46:42.756625+0000","last_peered":"2024-02-23T00:46:42.756625+0000","last_clean":"2024-02-23T00:46:42.756625+0000","last_became_active":"2024-02-23T00:45:11.600302+0000","last_became_peered":"2024-02-23T00:45:11.600302+0000","last_unstale":"2024-02-23T00:46:42.756625+0000","last_undegraded":"2024-02-23T00:46:42.756625+0000","last_fullsized":"2024-02-23T00:46:42.756625+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:42.756521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:44:59.793161+0000","last_clean_scrub_stamp":"2024-02-23T00:46:42.756521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"453'40","reported_seq":762,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711051+0000","last_change":"2024-02-23T00:46:00.034799+0000","last_active":"2024-02-23T00:46:28.711051+0000","last_peered":"2024-02-23T00:46:28.711051+0000","last_clean":"2024-02-23T00:46:28.711051+0000","last_became_active":"2024-02-23T00:42:10.240911+0000","last_became_peered":"2024-02-23T00:42:10.240911+0000","last_unstale":"2024-02-23T00:46:28.711051+0000","last_undegraded":"2024-02-23T00:46:28.711051+0000","last_fullsized":"2024-02-23T00:46:28.711051+0000","mapping_epoch":707,"log_start":"92'22","ondisk_log_start":"92'22","created":481,"last_epoch_clean":708,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'40","last_scrub_stamp":"2024-02-23T00:46:00.034676+0000","last_deep_scrub":"86'23","last_deep_scrub_stamp":"2024-02-23T00:29:56.044331+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.034676+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":849,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.613741+0000","last_change":"2024-02-23T00:46:27.648224+0000","last_active":"2024-02-23T00:46:28.613741+0000","last_peered":"2024-02-23T00:46:28.613741+0000","last_clean":"2024-02-23T00:46:28.613741+0000","last_became_active":"2024-02-23T00:34:42.063819+0000","last_became_peered":"2024-02-23T00:34:42.063819+0000","last_unstale":"2024-02-23T00:46:28.613741+0000","last_undegraded":"2024-02-23T00:46:28.613741+0000","last_fullsized":"2024-02-23T00:46:28.613741+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:27.648147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:40:55.816155+0000","last_clean_scrub_stamp":"2024-02-23T00:46:27.648147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"82'9","reported_seq":1099,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:45.434790+0000","last_change":"2024-02-23T00:46:45.434745+0000","last_active":"2024-02-23T00:46:45.434790+0000","last_peered":"2024-02-23T00:46:45.434790+0000","last_clean":"2024-02-23T00:46:45.434790+0000","last_became_active":"2024-02-23T00:27:21.027472+0000","last_became_peered":"2024-02-23T00:27:21.027472+0000","last_unstale":"2024-02-23T00:46:45.434790+0000","last_undegraded":"2024-02-23T00:46:45.434790+0000","last_fullsized":"2024-02-23T00:46:45.434790+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'9","last_scrub_stamp":"2024-02-23T00:46:45.434603+0000","last_deep_scrub":"82'9","last_deep_scrub_stamp":"2024-02-23T00:29:38.884131+0000","last_clean_scrub_stamp":"2024-02-23T00:46:45.434603+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":838,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616193+0000","last_change":"2024-02-23T00:45:44.731947+0000","last_active":"2024-02-23T00:46:28.616193+0000","last_peered":"2024-02-23T00:46:28.616193+0000","last_clean":"2024-02-23T00:46:28.616193+0000","last_became_active":"2024-02-23T00:41:37.229426+0000","last_became_peered":"2024-02-23T00:41:37.229426+0000","last_unstale":"2024-02-23T00:46:28.616193+0000","last_undegraded":"2024-02-23T00:46:28.616193+0000","last_fullsized":"2024-02-23T00:46:28.616193+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:44.731870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:44.731870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"453'42","reported_seq":1061,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:35.349673+0000","last_change":"2024-02-23T00:46:35.349673+0000","last_active":"2024-02-23T00:46:35.349673+0000","last_peered":"2024-02-23T00:46:35.349673+0000","last_clean":"2024-02-23T00:46:35.349673+0000","last_became_active":"2024-02-23T00:41:55.327964+0000","last_became_peered":"2024-02-23T00:41:55.327964+0000","last_unstale":"2024-02-23T00:46:35.349673+0000","last_undegraded":"2024-02-23T00:46:35.349673+0000","last_fullsized":"2024-02-23T00:46:35.349673+0000","mapping_epoch":692,"log_start":"87'22","ondisk_log_start":"87'22","created":481,"last_epoch_clean":693,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'42","last_scrub_stamp":"2024-02-23T00:46:35.349575+0000","last_deep_scrub":"87'22","last_deep_scrub_stamp":"2024-02-23T00:29:57.567427+0000","last_clean_scrub_stamp":"2024-02-23T00:46:35.349575+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":393,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:37.326399+0000","last_change":"2024-02-23T00:46:37.326399+0000","last_active":"2024-02-23T00:46:37.326399+0000","last_peered":"2024-02-23T00:46:37.326399+0000","last_clean":"2024-02-23T00:46:37.326399+0000","last_became_active":"2024-02-23T00:41:37.228694+0000","last_became_peered":"2024-02-23T00:41:37.228694+0000","last_unstale":"2024-02-23T00:46:37.326399+0000","last_undegraded":"2024-02-23T00:46:37.326399+0000","last_fullsized":"2024-02-23T00:46:37.326399+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:37.326279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:35:02.151731+0000","last_clean_scrub_stamp":"2024-02-23T00:46:37.326279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":202,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711096+0000","last_change":"2024-02-23T00:46:06.945114+0000","last_active":"2024-02-23T00:46:28.711096+0000","last_peered":"2024-02-23T00:46:28.711096+0000","last_clean":"2024-02-23T00:46:28.711096+0000","last_became_active":"2024-02-23T00:41:37.247946+0000","last_became_peered":"2024-02-23T00:41:37.247946+0000","last_unstale":"2024-02-23T00:46:28.711096+0000","last_undegraded":"2024-02-23T00:46:28.711096+0000","last_fullsized":"2024-02-23T00:46:28.711096+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:06.945049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:06.945049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:31.684586+0000","last_change":"2024-02-23T00:46:31.684540+0000","last_active":"2024-02-23T00:46:31.684586+0000","last_peered":"2024-02-23T00:46:31.684586+0000","last_clean":"2024-02-23T00:46:31.684586+0000","last_became_active":"2024-02-23T00:30:48.736943+0000","last_became_peered":"2024-02-23T00:30:48.736943+0000","last_unstale":"2024-02-23T00:46:31.684586+0000","last_undegraded":"2024-02-23T00:46:31.684586+0000","last_fullsized":"2024-02-23T00:46:31.684586+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:31.684458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:31.684458+0000","last_clean_scrub_stamp":"2024-02-23T00:46:31.684458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"453'37","reported_seq":991,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472990+0000","last_change":"2024-02-23T00:46:11.382604+0000","last_active":"2024-02-23T00:46:28.472990+0000","last_peered":"2024-02-23T00:46:28.472990+0000","last_clean":"2024-02-23T00:46:28.472990+0000","last_became_active":"2024-02-23T00:38:52.159130+0000","last_became_peered":"2024-02-23T00:38:52.159130+0000","last_unstale":"2024-02-23T00:46:28.472990+0000","last_undegraded":"2024-02-23T00:46:28.472990+0000","last_fullsized":"2024-02-23T00:46:28.472990+0000","mapping_epoch":514,"log_start":"62'15","ondisk_log_start":"62'15","created":481,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'37","last_scrub_stamp":"2024-02-23T00:46:11.382537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:46:11.382537+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":861,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:40.410548+0000","last_change":"2024-02-23T00:46:40.410548+0000","last_active":"2024-02-23T00:46:40.410548+0000","last_peered":"2024-02-23T00:46:40.410548+0000","last_clean":"2024-02-23T00:46:40.410548+0000","last_became_active":"2024-02-23T00:27:17.012154+0000","last_became_peered":"2024-02-23T00:27:17.012154+0000","last_unstale":"2024-02-23T00:46:40.410548+0000","last_undegraded":"2024-02-23T00:46:40.410548+0000","last_fullsized":"2024-02-23T00:46:40.410548+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:40.410474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:40.183453+0000","last_clean_scrub_stamp":"2024-02-23T00:46:40.410474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":861,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:32.445988+0000","last_change":"2024-02-23T00:46:32.445988+0000","last_active":"2024-02-23T00:46:32.445988+0000","last_peered":"2024-02-23T00:46:32.445988+0000","last_clean":"2024-02-23T00:46:32.445988+0000","last_became_active":"2024-02-23T00:41:37.229323+0000","last_became_peered":"2024-02-23T00:41:37.229323+0000","last_unstale":"2024-02-23T00:46:32.445988+0000","last_undegraded":"2024-02-23T00:46:32.445988+0000","last_fullsized":"2024-02-23T00:46:32.445988+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:32.445916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:31:38.887597+0000","last_clean_scrub_stamp":"2024-02-23T00:46:32.445916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":370,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:44.582103+0000","last_change":"2024-02-23T00:46:44.582103+0000","last_active":"2024-02-23T00:46:44.582103+0000","last_peered":"2024-02-23T00:46:44.582103+0000","last_clean":"2024-02-23T00:46:44.582103+0000","last_became_active":"2024-02-23T00:45:19.684020+0000","last_became_peered":"2024-02-23T00:45:19.684020+0000","last_unstale":"2024-02-23T00:46:44.582103+0000","last_undegraded":"2024-02-23T00:46:44.582103+0000","last_fullsized":"2024-02-23T00:46:44.582103+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:44.582028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:46:44.582028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"453'44","reported_seq":958,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:44.179259+0000","last_change":"2024-02-23T00:46:44.179259+0000","last_active":"2024-02-23T00:46:44.179259+0000","last_peered":"2024-02-23T00:46:44.179259+0000","last_clean":"2024-02-23T00:46:44.179259+0000","last_became_active":"2024-02-23T00:38:55.177674+0000","last_became_peered":"2024-02-23T00:38:55.177674+0000","last_unstale":"2024-02-23T00:46:44.179259+0000","last_undegraded":"2024-02-23T00:46:44.179259+0000","last_fullsized":"2024-02-23T00:46:44.179259+0000","mapping_epoch":517,"log_start":"76'24","ondisk_log_start":"76'24","created":481,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'44","last_scrub_stamp":"2024-02-23T00:46:44.179188+0000","last_deep_scrub":"453'44","last_deep_scrub_stamp":"2024-02-23T00:44:29.944804+0000","last_clean_scrub_stamp":"2024-02-23T00:46:44.179188+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":374,"reported_epoch":857,"state":"active+clean","last_fresh":"2024-02-23T00:46:43.180706+0000","last_change":"2024-02-23T00:46:43.180706+0000","last_active":"2024-02-23T00:46:43.180706+0000","last_peered":"2024-02-23T00:46:43.180706+0000","last_clean":"2024-02-23T00:46:43.180706+0000","last_became_active":"2024-02-23T00:38:49.181028+0000","last_became_peered":"2024-02-23T00:38:49.181028+0000","last_unstale":"2024-02-23T00:46:43.180706+0000","last_undegraded":"2024-02-23T00:46:43.180706+0000","last_fullsized":"2024-02-23T00:46:43.180706+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:43.180595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:43.180595+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":717,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471639+0000","last_change":"2024-02-23T00:46:15.149118+0000","last_active":"2024-02-23T00:46:28.471639+0000","last_peered":"2024-02-23T00:46:28.471639+0000","last_clean":"2024-02-23T00:46:28.471639+0000","last_became_active":"2024-02-23T00:45:11.598288+0000","last_became_peered":"2024-02-23T00:45:11.598288+0000","last_unstale":"2024-02-23T00:46:28.471639+0000","last_undegraded":"2024-02-23T00:46:28.471639+0000","last_fullsized":"2024-02-23T00:46:28.471639+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:15.149033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:15.149033+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":789,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614850+0000","last_change":"2024-02-23T00:46:10.872007+0000","last_active":"2024-02-23T00:46:28.614850+0000","last_peered":"2024-02-23T00:46:28.614850+0000","last_clean":"2024-02-23T00:46:28.614850+0000","last_became_active":"2024-02-23T00:30:48.678444+0000","last_became_peered":"2024-02-23T00:30:48.678444+0000","last_unstale":"2024-02-23T00:46:28.614850+0000","last_undegraded":"2024-02-23T00:46:28.614850+0000","last_fullsized":"2024-02-23T00:46:28.614850+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:10.871921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:46:10.871921+0000","last_clean_scrub_stamp":"2024-02-23T00:46:10.871921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"476'63","reported_seq":688,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471153+0000","last_change":"2024-02-23T00:45:33.464804+0000","last_active":"2024-02-23T00:46:28.471153+0000","last_peered":"2024-02-23T00:46:28.471153+0000","last_clean":"2024-02-23T00:46:28.471153+0000","last_became_active":"2024-02-23T00:45:11.596036+0000","last_became_peered":"2024-02-23T00:45:11.596036+0000","last_unstale":"2024-02-23T00:46:28.471153+0000","last_undegraded":"2024-02-23T00:46:28.471153+0000","last_fullsized":"2024-02-23T00:46:28.471153+0000","mapping_epoch":833,"log_start":"63'25","ondisk_log_start":"63'25","created":481,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"476'63","last_scrub_stamp":"2024-02-23T00:45:33.464727+0000","last_deep_scrub":"476'63","last_deep_scrub_stamp":"2024-02-23T00:38:13.528949+0000","last_clean_scrub_stamp":"2024-02-23T00:45:33.464727+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":562,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471533+0000","last_change":"2024-02-23T00:46:21.026675+0000","last_active":"2024-02-23T00:46:28.471533+0000","last_peered":"2024-02-23T00:46:28.471533+0000","last_clean":"2024-02-23T00:46:28.471533+0000","last_became_active":"2024-02-23T00:34:42.058527+0000","last_became_peered":"2024-02-23T00:34:42.058527+0000","last_unstale":"2024-02-23T00:46:28.471533+0000","last_undegraded":"2024-02-23T00:46:28.471533+0000","last_fullsized":"2024-02-23T00:46:28.471533+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:21.026579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:21.026579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":855,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.614104+0000","last_change":"2024-02-23T00:46:16.738414+0000","last_active":"2024-02-23T00:46:28.614104+0000","last_peered":"2024-02-23T00:46:28.614104+0000","last_clean":"2024-02-23T00:46:28.614104+0000","last_became_active":"2024-02-23T00:41:37.228310+0000","last_became_peered":"2024-02-23T00:41:37.228310+0000","last_unstale":"2024-02-23T00:46:28.614104+0000","last_undegraded":"2024-02-23T00:46:28.614104+0000","last_fullsized":"2024-02-23T00:46:28.614104+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:16.738313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:16.738313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":800,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615300+0000","last_change":"2024-02-23T00:45:35.760307+0000","last_active":"2024-02-23T00:46:28.615300+0000","last_peered":"2024-02-23T00:46:28.615300+0000","last_clean":"2024-02-23T00:46:28.615300+0000","last_became_active":"2024-02-23T00:30:48.732215+0000","last_became_peered":"2024-02-23T00:30:48.732215+0000","last_unstale":"2024-02-23T00:46:28.615300+0000","last_undegraded":"2024-02-23T00:46:28.615300+0000","last_fullsized":"2024-02-23T00:46:28.615300+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:35.760234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:35.760234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"453'22","reported_seq":398,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471332+0000","last_change":"2024-02-23T00:45:57.151443+0000","last_active":"2024-02-23T00:46:28.471332+0000","last_peered":"2024-02-23T00:46:28.471332+0000","last_clean":"2024-02-23T00:46:28.471332+0000","last_became_active":"2024-02-23T00:45:11.598687+0000","last_became_peered":"2024-02-23T00:45:11.598687+0000","last_unstale":"2024-02-23T00:46:28.471332+0000","last_undegraded":"2024-02-23T00:46:28.471332+0000","last_fullsized":"2024-02-23T00:46:28.471332+0000","mapping_epoch":833,"log_start":"78'14","ondisk_log_start":"78'14","created":481,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'22","last_scrub_stamp":"2024-02-23T00:45:57.151362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:17.999562+0000","last_clean_scrub_stamp":"2024-02-23T00:45:57.151362+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":871,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.616245+0000","last_change":"2024-02-23T00:45:52.712758+0000","last_active":"2024-02-23T00:46:28.616245+0000","last_peered":"2024-02-23T00:46:28.616245+0000","last_clean":"2024-02-23T00:46:28.616245+0000","last_became_active":"2024-02-23T00:41:37.230054+0000","last_became_peered":"2024-02-23T00:41:37.230054+0000","last_unstale":"2024-02-23T00:46:28.616245+0000","last_undegraded":"2024-02-23T00:46:28.616245+0000","last_fullsized":"2024-02-23T00:46:28.616245+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:52.712651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:28:46.396471+0000","last_clean_scrub_stamp":"2024-02-23T00:45:52.712651+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":848,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615806+0000","last_change":"2024-02-23T00:45:42.753321+0000","last_active":"2024-02-23T00:46:28.615806+0000","last_peered":"2024-02-23T00:46:28.615806+0000","last_clean":"2024-02-23T00:46:28.615806+0000","last_became_active":"2024-02-23T00:27:21.027327+0000","last_became_peered":"2024-02-23T00:27:21.027327+0000","last_unstale":"2024-02-23T00:46:28.615806+0000","last_undegraded":"2024-02-23T00:46:28.615806+0000","last_fullsized":"2024-02-23T00:46:28.615806+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:42.753243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:42:26.307631+0000","last_clean_scrub_stamp":"2024-02-23T00:45:42.753243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":810,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710437+0000","last_change":"2024-02-23T00:45:46.392980+0000","last_active":"2024-02-23T00:46:28.710437+0000","last_peered":"2024-02-23T00:46:28.710437+0000","last_clean":"2024-02-23T00:46:28.710437+0000","last_became_active":"2024-02-23T00:38:49.224748+0000","last_became_peered":"2024-02-23T00:38:49.224748+0000","last_unstale":"2024-02-23T00:46:28.710437+0000","last_undegraded":"2024-02-23T00:46:28.710437+0000","last_fullsized":"2024-02-23T00:46:28.710437+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:46.392916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:46.392916+0000","last_clean_scrub_stamp":"2024-02-23T00:45:46.392916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"453'38","reported_seq":999,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.470942+0000","last_change":"2024-02-23T00:44:58.245037+0000","last_active":"2024-02-23T00:46:28.470942+0000","last_peered":"2024-02-23T00:46:28.470942+0000","last_clean":"2024-02-23T00:46:28.470942+0000","last_became_active":"2024-02-23T00:39:24.164661+0000","last_became_peered":"2024-02-23T00:39:24.164661+0000","last_unstale":"2024-02-23T00:46:28.470942+0000","last_undegraded":"2024-02-23T00:46:28.470942+0000","last_fullsized":"2024-02-23T00:46:28.470942+0000","mapping_epoch":544,"log_start":"81'20","ondisk_log_start":"81'20","created":481,"last_epoch_clean":545,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'38","last_scrub_stamp":"2024-02-23T00:44:58.244955+0000","last_deep_scrub":"78'36","last_deep_scrub_stamp":"2024-02-23T00:28:46.118918+0000","last_clean_scrub_stamp":"2024-02-23T00:44:58.244955+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":393,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.472992+0000","last_change":"2024-02-23T00:46:28.291609+0000","last_active":"2024-02-23T00:46:28.472992+0000","last_peered":"2024-02-23T00:46:28.472992+0000","last_clean":"2024-02-23T00:46:28.472992+0000","last_became_active":"2024-02-23T00:41:37.241142+0000","last_became_peered":"2024-02-23T00:41:37.241142+0000","last_unstale":"2024-02-23T00:46:28.472992+0000","last_undegraded":"2024-02-23T00:46:28.472992+0000","last_fullsized":"2024-02-23T00:46:28.472992+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:28.291525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:15.989319+0000","last_clean_scrub_stamp":"2024-02-23T00:46:28.291525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":504,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710364+0000","last_change":"2024-02-23T00:46:00.317428+0000","last_active":"2024-02-23T00:46:28.710364+0000","last_peered":"2024-02-23T00:46:28.710364+0000","last_clean":"2024-02-23T00:46:28.710364+0000","last_became_active":"2024-02-23T00:35:42.472840+0000","last_became_peered":"2024-02-23T00:35:42.472840+0000","last_unstale":"2024-02-23T00:46:28.710364+0000","last_undegraded":"2024-02-23T00:46:28.710364+0000","last_fullsized":"2024-02-23T00:46:28.710364+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:00.317340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:46:00.317340+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":282,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.615561+0000","last_change":"2024-02-23T00:46:10.532532+0000","last_active":"2024-02-23T00:46:28.615561+0000","last_peered":"2024-02-23T00:46:28.615561+0000","last_clean":"2024-02-23T00:46:28.615561+0000","last_became_active":"2024-02-23T00:40:23.199068+0000","last_became_peered":"2024-02-23T00:40:23.199068+0000","last_unstale":"2024-02-23T00:46:28.615561+0000","last_undegraded":"2024-02-23T00:46:28.615561+0000","last_fullsized":"2024-02-23T00:46:28.615561+0000","mapping_epoch":603,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":604,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:46:10.532448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:00.454077+0000","last_clean_scrub_stamp":"2024-02-23T00:46:10.532448+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"453'68","reported_seq":1144,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.473224+0000","last_change":"2024-02-23T00:46:20.208922+0000","last_active":"2024-02-23T00:46:28.473224+0000","last_peered":"2024-02-23T00:46:28.473224+0000","last_clean":"2024-02-23T00:46:28.473224+0000","last_became_active":"2024-02-23T00:41:41.222892+0000","last_became_peered":"2024-02-23T00:41:41.222892+0000","last_unstale":"2024-02-23T00:46:28.473224+0000","last_undegraded":"2024-02-23T00:46:28.473224+0000","last_fullsized":"2024-02-23T00:46:28.473224+0000","mapping_epoch":681,"log_start":"278'50","ondisk_log_start":"278'50","created":481,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":5,"last_scrub":"453'68","last_scrub_stamp":"2024-02-23T00:46:20.208806+0000","last_deep_scrub":"453'68","last_deep_scrub_stamp":"2024-02-23T00:46:20.208806+0000","last_clean_scrub_stamp":"2024-02-23T00:46:20.208806+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":202,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.711122+0000","last_change":"2024-02-23T00:45:56.990448+0000","last_active":"2024-02-23T00:46:28.711122+0000","last_peered":"2024-02-23T00:46:28.711122+0000","last_clean":"2024-02-23T00:46:28.711122+0000","last_became_active":"2024-02-23T00:41:37.234830+0000","last_became_peered":"2024-02-23T00:41:37.234830+0000","last_unstale":"2024-02-23T00:46:28.711122+0000","last_undegraded":"2024-02-23T00:46:28.711122+0000","last_fullsized":"2024-02-23T00:46:28.711122+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:56.990386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:45:56.990386+0000","last_clean_scrub_stamp":"2024-02-23T00:45:56.990386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":699,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.471155+0000","last_change":"2024-02-23T00:45:41.232189+0000","last_active":"2024-02-23T00:46:28.471155+0000","last_peered":"2024-02-23T00:46:28.471155+0000","last_clean":"2024-02-23T00:46:28.471155+0000","last_became_active":"2024-02-23T00:45:11.601270+0000","last_became_peered":"2024-02-23T00:45:11.601270+0000","last_unstale":"2024-02-23T00:46:28.471155+0000","last_undegraded":"2024-02-23T00:46:28.471155+0000","last_fullsized":"2024-02-23T00:46:28.471155+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:41.232090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:27:20.002687+0000","last_clean_scrub_stamp":"2024-02-23T00:45:41.232090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":809,"reported_epoch":856,"state":"active+clean","last_fresh":"2024-02-23T00:46:28.710799+0000","last_change":"2024-02-23T00:45:54.240974+0000","last_active":"2024-02-23T00:46:28.710799+0000","last_peered":"2024-02-23T00:46:28.710799+0000","last_clean":"2024-02-23T00:46:28.710799+0000","last_became_active":"2024-02-23T00:38:49.227176+0000","last_became_peered":"2024-02-23T00:38:49.227176+0000","last_unstale":"2024-02-23T00:46:28.710799+0000","last_undegraded":"2024-02-23T00:46:28.710799+0000","last_fullsized":"2024-02-23T00:46:28.710799+0000","mapping_epoch":511,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":512,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-23T00:45:54.240855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-23T00:30:47.537238+0000","last_clean_scrub_stamp":"2024-02-23T00:45:54.240855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":12,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63,"ondisk_log_size":63,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":209,"num_object_clones":0,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":554,"num_read_kb":448,"num_write":335,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":2586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":752,"ondisk_log_size":752,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":23,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":845,"num_read_kb":845,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36,"ondisk_log_size":36,"up":69,"acting":69,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":51,"seq":219043332334,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309084,"kb_used_data":7764,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436068,"statfs":{"total":95995035648,"available":95678533632,"internally_reserved":0,"allocated":7950336,"data_stored":4280936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.499,"15min":0.48499999999999999},"min":{"1min":0.40100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68300000000000005,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.746},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.52100000000000002,"15min":0.5},"min":{"1min":0.433,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.77000000000000002}]},{"osd":1,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.499,"15min":0.67100000000000004},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.217},"max":{"1min":0.64500000000000002,"5min":1.304,"15min":31.175000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.51900000000000002,"15min":0.74099999999999999},"min":{"1min":0.38500000000000001,"5min":0.32600000000000001,"15min":0.249},"max":{"1min":0.748,"5min":0.748,"15min":29.423999999999999},"last":0.69099999999999995}]},{"osd":2,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.5,"15min":0.71599999999999997},"min":{"1min":0.40699999999999997,"5min":0.28699999999999998,"15min":0.28100000000000003},"max":{"1min":0.70099999999999996,"5min":0.72199999999999998,"15min":16.603999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54600000000000004,"15min":0.64600000000000002},"min":{"1min":0.40200000000000002,"5min":0.29099999999999998,"15min":0.26600000000000001},"max":{"1min":0.72899999999999998,"5min":1.05,"15min":7.5339999999999998},"last":0.65900000000000003}]},{"osd":3,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51600000000000001,"15min":1.008},"min":{"1min":0.40400000000000003,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":0.73699999999999999,"5min":0.89000000000000001,"15min":57.729999999999997},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.92000000000000004},"min":{"1min":0.47399999999999998,"5min":0.318,"15min":0.29999999999999999},"max":{"1min":0.72399999999999998,"5min":1.111,"15min":27.573},"last":0.73399999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.32400000000000001,"5min":0.255,"15min":0.214},"max":{"1min":0.65200000000000002,"5min":0.72999999999999998,"15min":1.706},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46200000000000002,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.24299999999999999,"15min":0.23300000000000001},"max":{"1min":0.67200000000000004,"5min":0.74299999999999999,"15min":8.1359999999999992},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.432},"min":{"1min":0.28499999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.48099999999999998,"15min":0.46500000000000002},"min":{"1min":0.42099999999999999,"5min":0.29199999999999998,"15min":0.26500000000000001},"max":{"1min":0.66700000000000004,"5min":1.0660000000000001,"15min":1.2889999999999999},"last":0.63100000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:45:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.30599999999999999,"5min":0.30399999999999999,"15min":0.246},"max":{"1min":0.67500000000000004,"5min":0.72799999999999998,"15min":2.1339999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.53600000000000003,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.69199999999999995,"5min":1.127,"15min":5.194},"last":0.61599999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561267,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309072,"kb_used_data":7752,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436080,"statfs":{"total":95995035648,"available":95678545920,"internally_reserved":0,"allocated":7938048,"data_stored":4280498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52000000000000002,"15min":0.54000000000000004},"min":{"1min":0.29699999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.504,"15min":0.501},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78100000000000003,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52400000000000002,"15min":0.96899999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.222},"max":{"1min":0.70899999999999996,"5min":0.79500000000000004,"15min":59.347999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51600000000000001,"15min":1.347},"min":{"1min":0.34499999999999997,"5min":0.31900000000000001,"15min":0.20999999999999999},"max":{"1min":0.68300000000000005,"5min":1.1619999999999999,"15min":87.221000000000004},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.53300000000000003,"15min":0.81899999999999995},"min":{"1min":0.33000000000000002,"5min":0.32900000000000001,"15min":0.214},"max":{"1min":0.76200000000000001,"5min":1.163,"15min":25.550999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.81999999999999995},"min":{"1min":0.30099999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71299999999999997,"5min":2.9460000000000002,"15min":40.914000000000001},"last":0.56799999999999995}]},{"osd":3,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56699999999999995,"15min":0.79100000000000004},"min":{"1min":0.36499999999999999,"5min":0.315,"15min":0.23400000000000001},"max":{"1min":0.77400000000000002,"5min":2.859,"15min":31.309999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54700000000000004,"15min":1.2470000000000001},"min":{"1min":0.36899999999999999,"5min":0.35799999999999998,"15min":0.27500000000000002},"max":{"1min":0.73099999999999998,"5min":0.89900000000000002,"15min":86.311999999999998},"last":0.61799999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45000000000000001,"15min":0.47799999999999998},"min":{"1min":0.32500000000000001,"5min":0.25800000000000001,"15min":0.23699999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":7.8769999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.48799999999999999,"15min":0.47599999999999998},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.23100000000000001},"max":{"1min":0.754,"5min":0.754,"15min":6.0300000000000002},"last":0.41099999999999998}]},{"osd":5,"last update":"Fri Feb 23 00:46:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46899999999999997,"15min":0.498},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.65400000000000003,"5min":0.86899999999999999,"15min":6.7999999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.49199999999999999},"min":{"1min":0.35199999999999998,"5min":0.29299999999999998,"15min":0.26600000000000001},"max":{"1min":0.78100000000000003,"5min":0.97999999999999998,"15min":6.0780000000000003},"last":0.46500000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:45:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.53000000000000003,"15min":0.625},"min":{"1min":0.40699999999999997,"5min":0.28499999999999998,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":1.014,"15min":17.881},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.53000000000000003,"15min":0.65500000000000003},"min":{"1min":0.39800000000000002,"5min":0.32900000000000001,"15min":0.26500000000000001},"max":{"1min":0.73499999999999999,"5min":0.85699999999999998,"15min":18.475000000000001},"last":0.45200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575114,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313212,"kb_used_data":7796,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431940,"statfs":{"total":95995035648,"available":95674306560,"internally_reserved":0,"allocated":7983104,"data_stored":4284845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.36099999999999999,"15min":0.40799999999999997},"min":{"1min":0.22900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.61699999999999999,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.40899999999999997},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.66900000000000004,"5min":1.161,"15min":1.161},"last":0.38800000000000001}]},{"osd":2,"last update":"Fri Feb 23 00:45:50 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35999999999999999,"15min":0.66900000000000004},"min":{"1min":0.23999999999999999,"5min":0.185,"15min":0.17999999999999999},"max":{"1min":0.50700000000000001,"5min":1.677,"15min":35.313000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.39700000000000002,"15min":0.58699999999999997},"min":{"1min":0.25900000000000001,"5min":0.20599999999999999,"15min":0.18099999999999999},"max":{"1min":0.52700000000000002,"5min":1.5549999999999999,"15min":27.908999999999999},"last":0.25900000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:46:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.39700000000000002,"15min":0.64900000000000002},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.21199999999999999},"max":{"1min":0.69499999999999995,"5min":1.6279999999999999,"15min":21.013999999999999},"last":0.312},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40600000000000003,"15min":0.63400000000000001},"min":{"1min":0.313,"5min":0.252,"15min":0.22900000000000001},"max":{"1min":0.66000000000000003,"5min":1.71,"15min":29.234999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56499999999999995,"15min":0.70699999999999996},"min":{"1min":0.39400000000000002,"5min":0.376,"15min":0.27400000000000002},"max":{"1min":0.82799999999999996,"5min":0.90300000000000002,"15min":16.231000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55300000000000005,"15min":0.71299999999999997},"min":{"1min":0.379,"5min":0.32200000000000001,"15min":0.28999999999999998},"max":{"1min":0.86699999999999999,"5min":1.7350000000000001,"15min":29.212},"last":0.55800000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56999999999999995,"15min":0.72399999999999998},"min":{"1min":0.41499999999999998,"5min":0.37,"15min":0.29199999999999998},"max":{"1min":0.90700000000000003,"5min":0.93700000000000006,"15min":16.294},"last":0.501},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59199999999999997,"15min":0.68600000000000005},"min":{"1min":0.442,"5min":0.35399999999999998,"15min":0.28699999999999998},"max":{"1min":0.84299999999999997,"5min":1.238,"15min":11.105},"last":0.53500000000000003}]},{"osd":6,"last update":"Fri Feb 23 00:46:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57199999999999995,"15min":0.749},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.32200000000000001},"max":{"1min":0.88800000000000001,"5min":1.76,"15min":29.187000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.75},"min":{"1min":0.40000000000000002,"5min":0.35799999999999998,"15min":0.312},"max":{"1min":0.86899999999999999,"5min":0.90500000000000003,"15min":16.369},"last":0.63800000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:46:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.75},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.32000000000000001},"max":{"1min":0.86899999999999999,"5min":0.92000000000000004,"15min":16.395},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60599999999999998,"15min":0.628},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.34300000000000003},"max":{"1min":0.82099999999999995,"5min":0.95599999999999996,"15min":4.4379999999999997},"last":0.56899999999999995}]}]},{"osd":0,"up_from":659,"seq":2830383448131,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314784,"kb_used_data":7768,"kb_used_omap":85,"kb_used_meta":306922,"kb_avail":93430368,"statfs":{"total":95995035648,"available":95672696832,"internally_reserved":0,"allocated":7954432,"data_stored":4283522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87936,"internal_metadata":314288256},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.38600000000000001,"15min":0.50900000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.496,"5min":5.0880000000000001,"15min":5.0880000000000001},"last":0.439},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.36499999999999999,"15min":0.40500000000000003},"min":{"1min":0.23999999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.45400000000000001,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.41099999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35099999999999998,"15min":0.34200000000000003},"min":{"1min":0.222,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.57499999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.41599999999999998,"15min":0.54400000000000004},"min":{"1min":0.25900000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56299999999999994,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.48299999999999998}]},{"osd":3,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.51500000000000001,"15min":0.86499999999999999},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.52700000000000002,"5min":11.891,"15min":11.891},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.54600000000000004,"15min":1.0329999999999999},"min":{"1min":0.255,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54600000000000004,"5min":15.864000000000001,"15min":15.864000000000001},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56399999999999995,"15min":0.53600000000000003},"min":{"1min":0.41199999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.92100000000000004,"5min":1.004,"15min":1.004},"last":0.621},{"interface":"front","average":{"1min":0.623,"5min":0.57799999999999996,"15min":0.53800000000000003},"min":{"1min":0.47999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78800000000000003,"5min":0.996,"15min":0.996},"last":0.76100000000000001}]},{"osd":5,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.56599999999999995,"15min":0.53800000000000003},"min":{"1min":0.45200000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.82299999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.44400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.82699999999999996,"5min":2.0720000000000001,"15min":2.0720000000000001},"last":0.58899999999999997}]},{"osd":6,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61299999999999999,"15min":0.61599999999999999},"min":{"1min":0.443,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93899999999999995,"5min":2.1059999999999999,"15min":2.1059999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.60699999999999998,"15min":0.60199999999999998},"min":{"1min":0.51100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.88400000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.65000000000000002}]},{"osd":7,"last update":"Fri Feb 23 00:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.48599999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90900000000000003,"5min":2.1280000000000001,"15min":2.1280000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.62,"15min":0.61499999999999999},"min":{"1min":0.432,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77500000000000002,"5min":1.04,"15min":1.04},"last":0.73899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411590,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304968,"kb_used_data":7744,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440184,"statfs":{"total":95995035648,"available":95682748416,"internally_reserved":0,"allocated":7929856,"data_stored":4282145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32900000000000001,"15min":0.311},"min":{"1min":0.215,"5min":0.182,"15min":0.182},"max":{"1min":0.47199999999999998,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.34100000000000003,"15min":0.33100000000000002},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.50900000000000001,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.30499999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:45:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33700000000000002,"15min":0.59099999999999997},"min":{"1min":0.23200000000000001,"5min":0.17999999999999999,"15min":0.17000000000000001},"max":{"1min":0.47799999999999998,"5min":0.84399999999999997,"15min":19.324999999999999},"last":0.318},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33600000000000002,"15min":0.752},"min":{"1min":0.22,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.45600000000000002,"5min":0.72899999999999998,"15min":35.052},"last":0.32500000000000001}]},{"osd":3,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.35099999999999998,"15min":0.69699999999999995},"min":{"1min":0.252,"5min":0.193,"15min":0.193},"max":{"1min":0.48499999999999999,"5min":0.69699999999999995,"15min":43.5},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.39100000000000001,"15min":0.63800000000000001},"min":{"1min":0.29999999999999999,"5min":0.23599999999999999,"15min":0.22},"max":{"1min":0.54700000000000004,"5min":0.82399999999999995,"15min":27.414000000000001},"last":0.36399999999999999}]},{"osd":4,"last update":"Fri Feb 23 00:46:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55900000000000005,"15min":0.65300000000000002},"min":{"1min":0.36899999999999999,"5min":0.34300000000000003,"15min":0.26400000000000001},"max":{"1min":0.83599999999999997,"5min":0.94299999999999995,"15min":14.614000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57299999999999995,"15min":0.60799999999999998},"min":{"1min":0.371,"5min":0.30499999999999999,"15min":0.27600000000000002},"max":{"1min":0.84699999999999998,"5min":0.90600000000000003,"15min":7.766},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:46:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56799999999999995,"15min":0.71399999999999997},"min":{"1min":0.40400000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81799999999999995,"5min":1.248,"15min":17.765000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.66400000000000003},"min":{"1min":0.379,"5min":0.316,"15min":0.312},"max":{"1min":0.85399999999999998,"5min":0.98099999999999998,"15min":14.589},"last":0.54000000000000004}]},{"osd":6,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58099999999999996,"15min":0.61899999999999999},"min":{"1min":0.40799999999999997,"5min":0.35999999999999999,"15min":0.29699999999999999},"max":{"1min":0.82499999999999996,"5min":0.95599999999999996,"15min":7.7869999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.57999999999999996,"15min":0.79400000000000004},"min":{"1min":0.40799999999999997,"5min":0.36399999999999999,"15min":0.32000000000000001},"max":{"1min":0.86399999999999999,"5min":1.218,"15min":20.574999999999999},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Feb 23 00:45:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57799999999999996,"15min":1.0740000000000001},"min":{"1min":0.44800000000000001,"5min":0.38400000000000001,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":1.004,"15min":66.632000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59499999999999997,"15min":0.85099999999999998},"min":{"1min":0.46400000000000002,"5min":0.38100000000000001,"15min":0.34000000000000002},"max":{"1min":0.82899999999999996,"5min":1.0149999999999999,"15min":19.603999999999999},"last":0.65900000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149954,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317304,"kb_used_data":7792,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93427848,"statfs":{"total":95995035648,"available":95670116352,"internally_reserved":0,"allocated":7979008,"data_stored":4282375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:30 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.35599999999999998,"15min":0.372},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.443,"5min":2.468,"15min":2.468},"last":0.217},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34699999999999998,"15min":0.33300000000000002},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.45400000000000001,"5min":1.012,"15min":1.012},"last":0.29299999999999998}]},{"osd":1,"last update":"Fri Feb 23 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.40400000000000003,"15min":0.61499999999999999},"min":{"1min":0.26800000000000002,"5min":0.19600000000000001,"15min":0.17899999999999999},"max":{"1min":0.436,"5min":5.0030000000000001,"15min":54.351999999999997},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38900000000000001,"15min":0.77200000000000002},"min":{"1min":0.28799999999999998,"5min":0.23000000000000001,"15min":0.219},"max":{"1min":0.55400000000000005,"5min":2.1680000000000001,"15min":55.337000000000003},"last":0.26800000000000002}]},{"osd":2,"last update":"Fri Feb 23 00:46:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38100000000000001,"15min":0.754},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.223},"max":{"1min":0.501,"5min":0.89200000000000002,"15min":55.368000000000002},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.438,"15min":0.75},"min":{"1min":0.30299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.47399999999999998,"5min":6.3150000000000004,"15min":54.383000000000003},"last":0.31900000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:46:34 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55500000000000005,"15min":0.93100000000000005},"min":{"1min":0.42199999999999999,"5min":0.33000000000000002,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":2.5720000000000001,"15min":54.530000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.63600000000000001},"min":{"1min":0.39300000000000002,"5min":0.314,"15min":0.24099999999999999},"max":{"1min":0.745,"5min":5.8109999999999999,"15min":5.8109999999999999},"last":0.55500000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.623,"15min":0.89500000000000002},"min":{"1min":0.47399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.85999999999999999,"5min":5.2160000000000002,"15min":55.401000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.871},"min":{"1min":0.38600000000000001,"5min":0.35599999999999998,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":2.5219999999999998,"15min":54.454999999999998},"last":0.65200000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:46:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.84899999999999998},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.28699999999999998},"max":{"1min":0.875,"5min":4.7069999999999999,"15min":54.305},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.72099999999999997,"15min":0.91500000000000004},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.32800000000000001},"max":{"1min":0.81799999999999995,"5min":8.5860000000000003,"15min":55.424999999999997},"last":0.58199999999999996}]},{"osd":7,"last update":"Fri Feb 23 00:45:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64900000000000002,"15min":0.81599999999999995},"min":{"1min":0.439,"5min":0.374,"15min":0.32200000000000001},"max":{"1min":0.78600000000000003,"5min":4.6710000000000003,"15min":54.222000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.66200000000000003,"15min":1.0489999999999999},"min":{"1min":0.47999999999999998,"5min":0.39400000000000002,"15min":0.32200000000000001},"max":{"1min":0.83499999999999996,"5min":5.1840000000000002,"15min":55.460000000000001},"last":0.63600000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986429,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317276,"kb_used_data":7764,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93427876,"statfs":{"total":95995035648,"available":95670145024,"internally_reserved":0,"allocated":7950336,"data_stored":4281267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52300000000000002,"15min":0.50700000000000001},"min":{"1min":0.374,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78000000000000003,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.34200000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.76000000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.61399999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.496,"15min":0.88900000000000001},"min":{"1min":0.34399999999999997,"5min":0.31,"15min":0.24299999999999999},"max":{"1min":0.72199999999999998,"5min":0.748,"15min":16.696000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.78500000000000003},"min":{"1min":0.434,"5min":0.28000000000000003,"15min":0.255},"max":{"1min":0.748,"5min":1.4510000000000001,"15min":13.455},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.53300000000000003,"15min":0.90000000000000002},"min":{"1min":0.436,"5min":0.32700000000000001,"15min":0.27700000000000002},"max":{"1min":0.68200000000000005,"5min":1.472,"15min":41.886000000000003},"last":0.504},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.56799999999999995,"15min":1.0309999999999999},"min":{"1min":0.42099999999999999,"5min":0.33900000000000002,"15min":0.251},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":41.927999999999997},"last":0.57799999999999996}]},{"osd":3,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.55900000000000005,"15min":0.93400000000000005},"min":{"1min":0.40600000000000003,"5min":0.314,"15min":0.314},"max":{"1min":1.381,"5min":1.381,"15min":36.301000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51300000000000001,"15min":1.056},"min":{"1min":0.40600000000000003,"5min":0.35199999999999998,"15min":0.24299999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":42.064},"last":0.68100000000000005}]},{"osd":5,"last update":"Fri Feb 23 00:46:07 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.44900000000000001,"15min":0.57699999999999996},"min":{"1min":0.35999999999999999,"5min":0.26900000000000002,"15min":0.217},"max":{"1min":1.542,"5min":1.542,"15min":14.242000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.52800000000000002,"15min":0.629},"min":{"1min":0.40600000000000003,"5min":0.29099999999999998,"15min":0.24299999999999999},"max":{"1min":1.597,"5min":1.597,"15min":14.124000000000001},"last":0.44400000000000001}]},{"osd":6,"last update":"Fri Feb 23 00:46:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.53600000000000003,"15min":0.57499999999999996},"min":{"1min":0.38500000000000001,"5min":0.315,"15min":0.254},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":8.0730000000000004},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.52300000000000002,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.28299999999999997,"15min":0.23100000000000001},"max":{"1min":1.454,"5min":1.454,"15min":13.339},"last":0.41599999999999998}]},{"osd":7,"last update":"Fri Feb 23 00:45:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52800000000000002,"15min":0.65100000000000002},"min":{"1min":0.42299999999999999,"5min":0.311,"15min":0.27300000000000002},"max":{"1min":1.0329999999999999,"5min":1.5129999999999999,"15min":17.178999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.58199999999999996,"15min":0.72899999999999998},"min":{"1min":0.49399999999999999,"5min":0.34200000000000003,"15min":0.28299999999999997},"max":{"1min":1.48,"5min":1.48,"15min":15.85},"last":0.48399999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757496,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309084,"kb_used_data":7764,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436068,"statfs":{"total":95995035648,"available":95678533632,"internally_reserved":0,"allocated":7950336,"data_stored":4283563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 23 00:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.49299999999999999,"15min":0.48399999999999999},"min":{"1min":0.36399999999999999,"5min":0.254,"15min":0.254},"max":{"1min":0.80900000000000005,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52800000000000002,"15min":0.52000000000000002},"min":{"1min":0.39100000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.98199999999999998,"5min":1.087,"15min":1.087},"last":0.36899999999999999}]},{"osd":1,"last update":"Fri Feb 23 00:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.50900000000000001,"15min":0.75800000000000001},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.24099999999999999},"max":{"1min":0.72599999999999998,"5min":1.1180000000000001,"15min":38.838000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51800000000000002,"15min":1.1859999999999999},"min":{"1min":0.36799999999999999,"5min":0.26600000000000001,"15min":0.23699999999999999},"max":{"1min":0.92900000000000005,"5min":1.1200000000000001,"15min":127.794},"last":0.41299999999999998}]},{"osd":2,"last update":"Fri Feb 23 00:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52100000000000002,"15min":0.95699999999999996},"min":{"1min":0.39900000000000002,"5min":0.32400000000000001,"15min":0.22},"max":{"1min":0.90600000000000003,"5min":1.2969999999999999,"15min":37.442999999999998},"last":0.503},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55700000000000005,"15min":0.71199999999999997},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.249},"max":{"1min":0.76900000000000002,"5min":2.875,"15min":23.297999999999998},"last":0.42399999999999999}]},{"osd":3,"last update":"Fri Feb 23 00:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.53600000000000003,"15min":0.82199999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.27300000000000002},"max":{"1min":0.83099999999999996,"5min":1.3300000000000001,"15min":24.414999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53800000000000003,"15min":0.90300000000000002},"min":{"1min":0.39800000000000002,"5min":0.317,"15min":0.27400000000000002},"max":{"1min":0.94899999999999995,"5min":1.1679999999999999,"15min":40.865000000000002},"last":0.38200000000000001}]},{"osd":4,"last update":"Fri Feb 23 00:46:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.42999999999999999,"15min":0.78900000000000003},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":0.61099999999999999,"5min":0.753,"15min":103.13},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.45400000000000001,"15min":0.52500000000000002},"min":{"1min":0.27900000000000003,"5min":0.223,"15min":0.223},"max":{"1min":0.63200000000000001,"5min":0.91100000000000003,"15min":12.795999999999999},"last":0.39600000000000002}]},{"osd":6,"last update":"Fri Feb 23 00:46:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.51200000000000001,"15min":0.53000000000000003},"min":{"1min":0.33800000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.87,"5min":1.214,"15min":11.412000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51500000000000001,"15min":0.47599999999999998},"min":{"1min":0.35799999999999998,"5min":0.28299999999999997,"15min":0.26200000000000001},"max":{"1min":0.68600000000000005,"5min":1.3149999999999999,"15min":1.5229999999999999},"last":0.51200000000000001}]},{"osd":7,"last update":"Fri Feb 23 00:45:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51400000000000001,"15min":0.56699999999999995},"min":{"1min":0.314,"5min":0.28299999999999997,"15min":0.27700000000000002},"max":{"1min":0.72699999999999998,"5min":1.298,"15min":11.478},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52700000000000002,"15min":0.52500000000000002},"min":{"1min":0.35399999999999998,"5min":0.27900000000000003,"15min":0.25800000000000001},"max":{"1min":0.84499999999999997,"5min":1.2689999999999999,"15min":5.4050000000000002},"last":0.52700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-23T00:46:47.340 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-23T00:46:47.340 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-23T00:46:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:46.534899+0000 osd.1 (osd.1) 486 : cluster [DBG] 3.0 deep-scrub starts 2024-02-23T00:46:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:46.535962+0000 osd.1 (osd.1) 487 : cluster [DBG] 3.0 deep-scrub ok 2024-02-23T00:46:47.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:46.544980+0000 mgr.y (mgr.24437) 779 : audit [DBG] from='client.28197 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:46:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:46.534899+0000 osd.1 (osd.1) 486 : cluster [DBG] 3.0 deep-scrub starts 2024-02-23T00:46:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:46.535962+0000 osd.1 (osd.1) 487 : cluster [DBG] 3.0 deep-scrub ok 2024-02-23T00:46:47.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:47 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:46.544980+0000 mgr.y (mgr.24437) 779 : audit [DBG] from='client.28197 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:46:47.613 INFO:teuthology.orchestra.run.smithi087.stderr:Inferring config /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/mon.a/config 2024-02-23T00:46:47.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:46.534899+0000 osd.1 (osd.1) 486 : cluster [DBG] 3.0 deep-scrub starts 2024-02-23T00:46:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:46.535962+0000 osd.1 (osd.1) 487 : cluster [DBG] 3.0 deep-scrub ok 2024-02-23T00:46:47.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:46.544980+0000 mgr.y (mgr.24437) 779 : audit [DBG] from='client.28197 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-23T00:46:48.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:47.158218+0000 osd.0 (osd.0) 131 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:46:48.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:47.159209+0000 osd.0 (osd.0) 132 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:46:48.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:47.552637+0000 osd.7 (osd.7) 464 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:46:48.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:47.555531+0000 osd.7 (osd.7) 465 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:46:48.756 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:47.855367+0000 mgr.y (mgr.24437) 780 : cluster [DBG] pgmap v1368: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:48.756 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:47.158218+0000 osd.0 (osd.0) 131 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:46:48.756 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:47.159209+0000 osd.0 (osd.0) 132 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:46:48.756 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:47.552637+0000 osd.7 (osd.7) 464 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:46:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:47.555531+0000 osd.7 (osd.7) 465 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:46:48.757 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:48 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:47.855367+0000 mgr.y (mgr.24437) 780 : cluster [DBG] pgmap v1368: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:48.942 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:47.158218+0000 osd.0 (osd.0) 131 : cluster [DBG] 6.0 scrub starts 2024-02-23T00:46:48.955 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:47.159209+0000 osd.0 (osd.0) 132 : cluster [DBG] 6.0 scrub ok 2024-02-23T00:46:48.955 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:47.552637+0000 osd.7 (osd.7) 464 : cluster [DBG] 22.8 scrub starts 2024-02-23T00:46:48.956 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:47.555531+0000 osd.7 (osd.7) 465 : cluster [DBG] 22.8 scrub ok 2024-02-23T00:46:48.956 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:47.855367+0000 mgr.y (mgr.24437) 780 : cluster [DBG] pgmap v1368: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:49.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:48.050182+0000 osd.2 (osd.2) 347 : cluster [DBG] 5.b scrub starts 2024-02-23T00:46:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:48.051509+0000 osd.2 (osd.2) 348 : cluster [DBG] 5.b scrub ok 2024-02-23T00:46:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:48.774512+0000 osd.4 (osd.4) 421 : cluster [DBG] 4.e scrub starts 2024-02-23T00:46:49.584 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:48.778109+0000 osd.4 (osd.4) 422 : cluster [DBG] 4.e scrub ok 2024-02-23T00:46:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:48.050182+0000 osd.2 (osd.2) 347 : cluster [DBG] 5.b scrub starts 2024-02-23T00:46:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:48.051509+0000 osd.2 (osd.2) 348 : cluster [DBG] 5.b scrub ok 2024-02-23T00:46:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:48.774512+0000 osd.4 (osd.4) 421 : cluster [DBG] 4.e scrub starts 2024-02-23T00:46:49.584 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:48.778109+0000 osd.4 (osd.4) 422 : cluster [DBG] 4.e scrub ok 2024-02-23T00:46:49.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.593+0000 7efc31f17700 1 -- 172.21.15.87:0/1985720350 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc2c0f8420 msgr2=0x7efc2c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:49.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.593+0000 7efc31f17700 1 --2- 172.21.15.87:0/1985720350 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc2c0f8420 0x7efc2c104a20 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 rx=0x7efc24009c30 tx=0x7efc24004c50).stop 2024-02-23T00:46:49.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.593+0000 7efc31f17700 1 -- 172.21.15.87:0/1985720350 shutdown_connections 2024-02-23T00:46:49.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.593+0000 7efc31f17700 1 --2- 172.21.15.87:0/1985720350 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc2c0f8420 0x7efc2c104a20 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:49.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.593+0000 7efc31f17700 1 --2- 172.21.15.87:0/1985720350 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc2c0fdbb0 0x7efc2c0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:49.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.593+0000 7efc31f17700 1 --2- 172.21.15.87:0/1985720350 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc2c0fe510 0x7efc2c0f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:49.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.593+0000 7efc31f17700 1 -- 172.21.15.87:0/1985720350 >> 172.21.15.87:0/1985720350 conn(0x7efc2c0f3800 msgr2=0x7efc2c0f5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:49.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.593+0000 7efc31f17700 1 -- 172.21.15.87:0/1985720350 shutdown_connections 2024-02-23T00:46:49.598 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.593+0000 7efc31f17700 1 -- 172.21.15.87:0/1985720350 wait complete. 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 librados: starting msgr at 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 librados: starting objecter 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 Processor -- start 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 -- start start 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 librados: setting wanted keys 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 librados: calling monclient init 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc2c0f8420 0x7efc2c06b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc2c0fdbb0 0x7efc2c06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 --2- >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc2c0fe510 0x7efc2c06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 -- --> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c116310 con 0x7efc2c0f8420 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 -- --> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c116480 con 0x7efc2c0fe510 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc31f17700 1 -- --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c116640 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.599 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc2b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc2c0f8420 0x7efc2c06b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc2b7fe700 1 --2- >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc2c0f8420 0x7efc2c06b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.87:3300/0 says I am v2:172.21.15.87:39224/0 (socket says 172.21.15.87:39224) 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc2affd700 1 --2- >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc2c0fdbb0 0x7efc2c06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc2b7fe700 1 -- 172.21.15.87:0/633359355 learned_addr learned my addr 172.21.15.87:0/633359355 (peer_addr_for_me v2:172.21.15.87:0/0) 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.594+0000 7efc2bfff700 1 --2- 172.21.15.87:0/633359355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc2c0fe510 0x7efc2c06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc2affd700 1 -- 172.21.15.87:0/633359355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc2c0fe510 msgr2=0x7efc2c06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc2affd700 1 --2- 172.21.15.87:0/633359355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc2c0fe510 0x7efc2c06bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc2affd700 1 -- 172.21.15.87:0/633359355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc2c0f8420 msgr2=0x7efc2c06b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc2affd700 1 --2- 172.21.15.87:0/633359355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc2c0f8420 0x7efc2c06b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc2affd700 1 -- 172.21.15.87:0/633359355 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c005040 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc2b7fe700 1 --2- 172.21.15.87:0/633359355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc2c0f8420 0x7efc2c06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-02-23T00:46:49.600 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc2affd700 1 --2- 172.21.15.87:0/633359355 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc2c0fdbb0 0x7efc2c06b5c0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7efc1c0091f0 tx=0x7efc1c009cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:49.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc28ff9700 10 client.?.objecter ms_handle_connect 0x7efc2c0fdbb0 2024-02-23T00:46:49.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc28ff9700 10 client.?.objecter resend_mon_ops 2024-02-23T00:46:49.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc28ff9700 1 -- 172.21.15.87:0/633359355 <== mon.2 v2:172.21.15.87:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c00e030 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc28ff9700 1 -- 172.21.15.87:0/633359355 <== mon.2 v2:172.21.15.87:3301/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc1c00f040 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc28ff9700 1 -- 172.21.15.87:0/633359355 <== mon.2 v2:172.21.15.87:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c0147e0 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc240097c0 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc31f17700 10 client.38678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-23T00:46:49.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2c116cb0 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.601 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.595+0000 7efc31f17700 1 librados: init done 2024-02-23T00:46:49.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.596+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2c05ee80 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.596+0000 7efc28ff9700 1 -- 172.21.15.87:0/633359355 <== mon.2 v2:172.21.15.87:3301/0 4 ==== mgrmap(e 19) v1 ==== 77086+0+0 (secure 0 0 0) 0x7efc1c0104f0 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.597+0000 7efc28ff9700 1 --2- 172.21.15.87:0/633359355 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc14062c90 0x7efc14065150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-23T00:46:49.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.597+0000 7efc2b7fe700 1 --2- 172.21.15.87:0/633359355 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc14062c90 0x7efc14065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-23T00:46:49.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.597+0000 7efc28ff9700 1 -- 172.21.15.87:0/633359355 <== mon.2 v2:172.21.15.87:3301/0 5 ==== osd_map(857..857 src has 400..857) v4 ==== 7486+0+0 (secure 0 0 0) 0x7efc1c07cff0 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.597+0000 7efc28ff9700 10 client.38678.objecter ms_dispatch 0x7efc2c001550 osd_map(857..857 src has 400..857) v4 2024-02-23T00:46:49.602 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.597+0000 7efc28ff9700 3 client.38678.objecter handle_osd_map got epochs [857,857] > 0 2024-02-23T00:46:49.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.597+0000 7efc28ff9700 3 client.38678.objecter handle_osd_map decoding full epoch 857 2024-02-23T00:46:49.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.597+0000 7efc28ff9700 20 client.38678.objecter dump_active .. 0 homeless 2024-02-23T00:46:49.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.597+0000 7efc28ff9700 10 client.38678.objecter ms_handle_connect 0x7efc14062c90 2024-02-23T00:46:49.603 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.597+0000 7efc2b7fe700 1 --2- 172.21.15.87:0/633359355 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc14062c90 0x7efc14065150 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 rx=0x7efc20004fd0 tx=0x7efc2000b040).ready entity=mgr.24437 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-23T00:46:49.604 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.601+0000 7efc28ff9700 1 -- 172.21.15.87:0/633359355 <== mon.2 v2:172.21.15.87:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (secure 0 0 0) 0x7efc1c050740 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.765 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.763+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 --> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) v1 -- 0x7efc2c107110 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.770 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.768+0000 7efc28ff9700 1 -- 172.21.15.87:0/633359355 <== mon.2 v2:172.21.15.87:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v30) v1 ==== 118+0+0 (secure 0 0 0) 0x7efc2c107110 con 0x7efc2c0fdbb0 2024-02-23T00:46:49.772 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 10 librados: watch_flush enter 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 10 librados: watch_flush exit 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 20 client.38678.objecter shutdown clearing up homeless session... 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 10 client.38678.objecter successfully canceled tick 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc14062c90 msgr2=0x7efc14065150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 --2- 172.21.15.87:0/633359355 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc14062c90 0x7efc14065150 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 rx=0x7efc20004fd0 tx=0x7efc2000b040).stop 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc2c0fdbb0 msgr2=0x7efc2c06b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 --2- 172.21.15.87:0/633359355 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc2c0fdbb0 0x7efc2c06b5c0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 rx=0x7efc1c0091f0 tx=0x7efc1c009cb0).stop 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 shutdown_connections 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 --2- 172.21.15.87:0/633359355 >> v2:172.21.15.87:6800/1231597463 conn(0x7efc14062c90 0x7efc14065150 unknown :-1 s=CLOSED pgs=3029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 --2- 172.21.15.87:0/633359355 >> [v2:172.21.15.171:3300/0,v1:172.21.15.171:6789/0] conn(0x7efc2c0fe510 0x7efc2c06bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 --2- 172.21.15.87:0/633359355 >> [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] conn(0x7efc2c0f8420 0x7efc2c06b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 --2- 172.21.15.87:0/633359355 >> [v2:172.21.15.87:3301/0,v1:172.21.15.87:6790/0] conn(0x7efc2c0fdbb0 0x7efc2c06b5c0 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-23T00:46:49.773 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 >> 172.21.15.87:0/633359355 conn(0x7efc2c0f3800 msgr2=0x7efc2c0fbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-23T00:46:49.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 shutdown_connections 2024-02-23T00:46:49.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 -- 172.21.15.87:0/633359355 wait complete. 2024-02-23T00:46:49.774 INFO:teuthology.orchestra.run.smithi087.stderr:2024-02-23T00:46:49.770+0000 7efc31f17700 1 librados: shutdown 2024-02-23T00:46:49.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:48.050182+0000 osd.2 (osd.2) 347 : cluster [DBG] 5.b scrub starts 2024-02-23T00:46:49.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:48.051509+0000 osd.2 (osd.2) 348 : cluster [DBG] 5.b scrub ok 2024-02-23T00:46:49.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:48.774512+0000 osd.4 (osd.4) 421 : cluster [DBG] 4.e scrub starts 2024-02-23T00:46:49.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:48.778109+0000 osd.4 (osd.4) 422 : cluster [DBG] 4.e scrub ok 2024-02-23T00:46:50.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:49] ENGINE Bus STOPPING 2024-02-23T00:46:50.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-23T00:46:50.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:49] ENGINE Bus STOPPED 2024-02-23T00:46:50.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:49] ENGINE Bus STARTING 2024-02-23T00:46:50.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:49] ENGINE Serving on http://:::9283 2024-02-23T00:46:50.083 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:49 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:49] ENGINE Bus STARTED 2024-02-23T00:46:50.447 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-23T00:46:50.457 INFO:tasks.cephadm:Teardown begin 2024-02-23T00:46:50.458 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:46:50.489 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:46:50.526 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-23T00:46:50.526 DEBUG:teuthology.orchestra.run.smithi087:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-23T00:46:50.548 DEBUG:teuthology.orchestra.run.smithi171:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-23T00:46:50.586 INFO:tasks.cephadm:Stopping all daemons... 2024-02-23T00:46:50.586 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-23T00:46:50.586 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.a 2024-02-23T00:46:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.041452+0000 osd.2 (osd.2) 349 : cluster [DBG] 2.e scrub starts 2024-02-23T00:46:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.042408+0000 osd.2 (osd.2) 350 : cluster [DBG] 2.e scrub ok 2024-02-23T00:46:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.173606+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.b scrub starts 2024-02-23T00:46:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.177071+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.b scrub ok 2024-02-23T00:46:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.518878+0000 osd.5 (osd.5) 401 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:46:50.834 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.519779+0000 osd.5 (osd.5) 402 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:46:50.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.620537+0000 osd.7 (osd.7) 466 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:46:50.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.621446+0000 osd.7 (osd.7) 467 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:46:50.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:49.765037+0000 mon.c (mon.2) 527 : audit [INF] from='client.? 172.21.15.87:0/633359355' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-23T00:46:50.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:49.765359+0000 mon.a (mon.0) 4054 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-23T00:46:50.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:49.769141+0000 mon.a (mon.0) 4055 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-23T00:46:50.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:49.777673+0000 mon.a (mon.0) 4056 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:46:50.835 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:49.779049+0000 mon.a (mon.0) 4057 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:46:50.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:49.779660+0000 mon.a (mon.0) 4058 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:46:50.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: audit 2024-02-23T00:46:49.786236+0000 mon.a (mon.0) 4059 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:46:50.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.810236+0000 osd.4 (osd.4) 423 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:46:50.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.812319+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:46:50.836 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: cluster 2024-02-23T00:46:49.855806+0000 mgr.y (mgr.24437) 781 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:50.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.041452+0000 osd.2 (osd.2) 349 : cluster [DBG] 2.e scrub starts 2024-02-23T00:46:50.836 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.042408+0000 osd.2 (osd.2) 350 : cluster [DBG] 2.e scrub ok 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.173606+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.b scrub starts 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.177071+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.b scrub ok 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.518878+0000 osd.5 (osd.5) 401 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.519779+0000 osd.5 (osd.5) 402 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.620537+0000 osd.7 (osd.7) 466 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.621446+0000 osd.7 (osd.7) 467 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:49.765037+0000 mon.c (mon.2) 527 : audit [INF] from='client.? 172.21.15.87:0/633359355' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:49.765359+0000 mon.a (mon.0) 4054 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:49.769141+0000 mon.a (mon.0) 4055 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:49.777673+0000 mon.a (mon.0) 4056 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:49.779049+0000 mon.a (mon.0) 4057 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:49.779660+0000 mon.a (mon.0) 4058 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: audit 2024-02-23T00:46:49.786236+0000 mon.a (mon.0) 4059 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.810236+0000 osd.4 (osd.4) 423 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:46:50.837 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.812319+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:46:50.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: cluster 2024-02-23T00:46:49.855806+0000 mgr.y (mgr.24437) 781 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:50.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 systemd[1]: Stopping Ceph mon.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:46:50.838 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 bash[256779]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon.a" found: no such container 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.041452+0000 osd.2 (osd.2) 349 : cluster [DBG] 2.e scrub starts 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.042408+0000 osd.2 (osd.2) 350 : cluster [DBG] 2.e scrub ok 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.173606+0000 osd.0 (osd.0) 133 : cluster [DBG] 4.b scrub starts 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.177071+0000 osd.0 (osd.0) 134 : cluster [DBG] 4.b scrub ok 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.518878+0000 osd.5 (osd.5) 401 : cluster [DBG] 2.16 scrub starts 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.519779+0000 osd.5 (osd.5) 402 : cluster [DBG] 2.16 scrub ok 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.620537+0000 osd.7 (osd.7) 466 : cluster [DBG] 5.4 scrub starts 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.621446+0000 osd.7 (osd.7) 467 : cluster [DBG] 5.4 scrub ok 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:49.765037+0000 mon.c (mon.2) 527 : audit [INF] from='client.? 172.21.15.87:0/633359355' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:49.765359+0000 mon.a (mon.0) 4054 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:49.769141+0000 mon.a (mon.0) 4055 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-23T00:46:50.943 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:49.777673+0000 mon.a (mon.0) 4056 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-23T00:46:50.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:49.779049+0000 mon.a (mon.0) 4057 : audit [DBG] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-23T00:46:50.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:49.779660+0000 mon.a (mon.0) 4058 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-23T00:46:50.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: audit 2024-02-23T00:46:49.786236+0000 mon.a (mon.0) 4059 : audit [INF] from='mgr.24437 172.21.15.87:0/2291582883' entity='mgr.y' 2024-02-23T00:46:50.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.810236+0000 osd.4 (osd.4) 423 : cluster [DBG] 4.18 scrub starts 2024-02-23T00:46:50.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.812319+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.18 scrub ok 2024-02-23T00:46:50.944 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: cluster 2024-02-23T00:46:49.855806+0000 mgr.y (mgr.24437) 781 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-23T00:46:51.261 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:46:50.897+0000 7fe8e88a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-23T00:46:51.262 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a[38299]: debug 2024-02-23T00:46:50.897+0000 7fe8e88a4700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-23T00:46:51.262 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:50] ENGINE Bus STOPPING 2024-02-23T00:46:51.262 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-23T00:46:51.262 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:50] ENGINE Bus STOPPED 2024-02-23T00:46:51.262 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:50] ENGINE Bus STARTING 2024-02-23T00:46:51.263 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:51] ENGINE Serving on http://:::9283 2024-02-23T00:46:51.263 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:51 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:51] ENGINE Bus STARTED 2024-02-23T00:46:51.583 INFO:journalctl@ceph.mon.a.smithi087.stdout:Feb 23 00:46:51 smithi087 bash[256819]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-a 2024-02-23T00:46:51.812 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.a.service' 2024-02-23T00:46:51.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:46:51.852 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-23T00:46:51.852 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-23T00:46:51.852 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.c 2024-02-23T00:46:51.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:46:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug there is no tcmu-runner data available 2024-02-23T00:46:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:51 smithi087 systemd[1]: Stopping Ceph mon.c for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:46:52.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:51 smithi087 bash[256948]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon.c" found: no such container 2024-02-23T00:46:52.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:46:52.231+0000 7f8a3ba95700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-23T00:46:52.583 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c[44823]: debug 2024-02-23T00:46:52.231+0000 7f8a3ba95700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-23T00:46:52.584 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:52] "GET /metrics HTTP/1.1" 200 186064 "" "Prometheus/2.33.4" 2024-02-23T00:46:52.584 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:52] ENGINE Bus STOPPING 2024-02-23T00:46:52.584 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-23T00:46:52.584 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:52] ENGINE Bus STOPPED 2024-02-23T00:46:52.584 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:52] ENGINE Bus STARTING 2024-02-23T00:46:52.584 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:52] ENGINE Serving on http://:::9283 2024-02-23T00:46:52.584 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:52 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: [23/Feb/2024:00:46:52] ENGINE Bus STARTED 2024-02-23T00:46:53.083 INFO:journalctl@ceph.mon.c.smithi087.stdout:Feb 23 00:46:52 smithi087 bash[256987]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-c 2024-02-23T00:46:53.225 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.c.service' 2024-02-23T00:46:53.262 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:46:53.263 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-23T00:46:53.263 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-23T00:46:53.263 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.b 2024-02-23T00:46:53.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:53 smithi171 systemd[1]: Stopping Ceph mon.b for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:46:53.550 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:53 smithi171 bash[120254]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon.b" found: no such container 2024-02-23T00:46:53.915 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:46:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x[40913]: ::ffff:172.21.15.171 - - [23/Feb/2024:00:46:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-23T00:46:53.916 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:46:53.548+0000 7f0bbe227700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-23T00:46:53.916 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b[39584]: debug 2024-02-23T00:46:53.548+0000 7f0bbe227700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-23T00:46:54.192 INFO:journalctl@ceph.mon.b.smithi171.stdout:Feb 23 00:46:54 smithi171 bash[120293]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mon-b 2024-02-23T00:46:54.522 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mon.b.service' 2024-02-23T00:46:54.568 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:46:54.568 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-23T00:46:54.568 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-23T00:46:54.568 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mgr.y 2024-02-23T00:46:54.875 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:54 smithi087 systemd[1]: Stopping Ceph mgr.y for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:46:54.875 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:54 smithi087 bash[257116]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr.y" found: no such container 2024-02-23T00:46:55.245 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:46:54.871+0000 7f6efb8c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:46:55.245 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:54 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y[38574]: debug 2024-02-23T00:46:54.871+0000 7f6efb8c3700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-23T00:46:55.583 INFO:journalctl@ceph.mgr.y.smithi087.stdout:Feb 23 00:46:55 smithi087 bash[257155]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-y 2024-02-23T00:46:55.813 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mgr.y.service' 2024-02-23T00:46:55.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:46:55.849 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-23T00:46:55.849 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-23T00:46:55.849 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mgr.x 2024-02-23T00:46:56.193 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:46:55 smithi171 systemd[1]: Stopping Ceph mgr.x for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:46:56.193 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:46:55 smithi171 bash[120414]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr.x" found: no such container 2024-02-23T00:46:56.908 INFO:journalctl@ceph.mgr.x.smithi171.stdout:Feb 23 00:46:56 smithi171 bash[120455]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-mgr-x 2024-02-23T00:46:57.012 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@mgr.x.service' 2024-02-23T00:46:57.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:46:57.047 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-23T00:46:57.047 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-23T00:46:57.047 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.0 2024-02-23T00:46:57.333 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:46:57 smithi087 systemd[1]: Stopping Ceph osd.0 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:46:57.333 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:46:57 smithi087 bash[257275]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.0" found: no such container 2024-02-23T00:46:57.833 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:46:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[224705]: debug 2024-02-23T00:46:57.352+0000 7fc02e097700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:46:57.833 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:46:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[224705]: debug 2024-02-23T00:46:57.352+0000 7fc02e097700 -1 osd.0 857 *** Got signal Terminated *** 2024-02-23T00:46:57.833 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:46:57 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0[224705]: debug 2024-02-23T00:46:57.352+0000 7fc02e097700 -1 osd.0 857 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-23T00:47:03.083 INFO:journalctl@ceph.osd.0.smithi087.stdout:Feb 23 00:47:02 smithi087 bash[257315]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-0 2024-02-23T00:47:04.717 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.0.service' 2024-02-23T00:47:04.758 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:47:04.758 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-23T00:47:04.759 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-23T00:47:04.759 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.1 2024-02-23T00:47:05.082 INFO:journalctl@ceph.osd.1.smithi087.stdout:Feb 23 00:47:04 smithi087 systemd[1]: Stopping Ceph osd.1 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:47:05.082 INFO:journalctl@ceph.osd.1.smithi087.stdout:Feb 23 00:47:04 smithi087 bash[257559]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.1" found: no such container 2024-02-23T00:47:05.333 INFO:journalctl@ceph.osd.1.smithi087.stdout:Feb 23 00:47:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-1[52702]: debug 2024-02-23T00:47:05.080+0000 7f108f77f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:47:05.333 INFO:journalctl@ceph.osd.1.smithi087.stdout:Feb 23 00:47:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-1[52702]: debug 2024-02-23T00:47:05.080+0000 7f108f77f700 -1 osd.1 857 *** Got signal Terminated *** 2024-02-23T00:47:05.333 INFO:journalctl@ceph.osd.1.smithi087.stdout:Feb 23 00:47:05 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-1[52702]: debug 2024-02-23T00:47:05.080+0000 7f108f77f700 -1 osd.1 857 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-23T00:47:10.468 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:10.145Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:10.469 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:10.145Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:11.083 INFO:journalctl@ceph.osd.1.smithi087.stdout:Feb 23 00:47:10 smithi087 bash[257600]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-1 2024-02-23T00:47:12.619 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.1.service' 2024-02-23T00:47:12.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:47:12.662 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-23T00:47:12.663 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-23T00:47:12.663 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.2 2024-02-23T00:47:12.969 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:47:12 smithi087 systemd[1]: Stopping Ceph osd.2 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:47:12.969 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:47:12 smithi087 bash[257837]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.2" found: no such container 2024-02-23T00:47:13.333 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:47:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-2[57400]: debug 2024-02-23T00:47:12.967+0000 7f684d9db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:47:13.333 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:47:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-2[57400]: debug 2024-02-23T00:47:12.967+0000 7f684d9db700 -1 osd.2 857 *** Got signal Terminated *** 2024-02-23T00:47:13.333 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:47:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-2[57400]: debug 2024-02-23T00:47:12.967+0000 7f684d9db700 -1 osd.2 857 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-23T00:47:18.583 INFO:journalctl@ceph.osd.2.smithi087.stdout:Feb 23 00:47:18 smithi087 bash[257875]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-2 2024-02-23T00:47:20.194 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:47:20.144Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:20.195 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:20.145Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:20.195 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:20.146Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:20.580 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.2.service' 2024-02-23T00:47:20.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:47:20.625 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-23T00:47:20.626 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-23T00:47:20.626 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.3 2024-02-23T00:47:20.833 INFO:journalctl@ceph.osd.3.smithi087.stdout:Feb 23 00:47:20 smithi087 systemd[1]: Stopping Ceph osd.3 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:47:20.833 INFO:journalctl@ceph.osd.3.smithi087.stdout:Feb 23 00:47:20 smithi087 bash[258110]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.3" found: no such container 2024-02-23T00:47:21.333 INFO:journalctl@ceph.osd.3.smithi087.stdout:Feb 23 00:47:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-3[61865]: debug 2024-02-23T00:47:20.985+0000 7fda8248d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:47:21.333 INFO:journalctl@ceph.osd.3.smithi087.stdout:Feb 23 00:47:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-3[61865]: debug 2024-02-23T00:47:20.985+0000 7fda8248d700 -1 osd.3 857 *** Got signal Terminated *** 2024-02-23T00:47:21.333 INFO:journalctl@ceph.osd.3.smithi087.stdout:Feb 23 00:47:20 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-3[61865]: debug 2024-02-23T00:47:20.985+0000 7fda8248d700 -1 osd.3 857 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-23T00:47:23.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:23.578Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:23.834 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:23 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:23.578Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:24.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:23 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:23.887+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:24.922 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:24.661+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:25.192 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:24 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:24.919+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:25.672 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:25.200+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:25.942 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:25.896+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:25.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:25 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:25.669+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:26.674 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:26.241+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:26.833 INFO:journalctl@ceph.osd.3.smithi087.stdout:Feb 23 00:47:26 smithi087 bash[258150]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-3 2024-02-23T00:47:26.942 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:26.859+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:26.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:26.688+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:26.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:26 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:26.672+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:27.653 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:27.208+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:27.942 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:27.733+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:27.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:27.651+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:27.943 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:27 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:27.839+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:28.424 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.3.service' 2024-02-23T00:47:28.465 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:47:28.466 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-23T00:47:28.466 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-23T00:47:28.466 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.4 2024-02-23T00:47:28.500 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:28.242+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:28.764 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:28 smithi171 systemd[1]: Stopping Ceph osd.4 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:47:28.765 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:28 smithi171 bash[120590]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.4" found: no such container 2024-02-23T00:47:28.765 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:28.614+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:29.192 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:28.762+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:29.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:28.794+0000 7f0bcffaa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:47:29.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:28.794+0000 7f0bcffaa700 -1 osd.4 857 *** Got signal Terminated *** 2024-02-23T00:47:29.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:28.794+0000 7f0bcffaa700 -1 osd.4 857 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-23T00:47:29.193 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:28.831+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:29.660 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:29.208+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:29.942 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:29.785+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:29.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:29.759+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:29.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:29 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:29.658+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:30.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:30.178+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:30.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:47:30.145Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:30.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:30.145Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:30.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:30 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:30.145Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:30.942 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:30.795+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:30.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:30.718+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:30.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:30.619+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:30.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:30 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:30.619+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:31.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:31.162+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:31.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:31.162+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:31.942 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:31.763+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:31.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:31.732+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:31.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:31.732+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:31.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:31.640+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:31.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:31 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:31.640+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:32.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:32.181+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:32.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:32.181+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:32.943 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:32.715+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:32.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:32.705+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:32.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:32.705+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:32.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:32.646+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:32.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:32 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:32.647+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:33.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:33.135+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:33.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:33.135+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:33.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:47:33.578Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:33.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:33.578Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:33.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:33 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:33.579Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:33.942 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:33.736+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.443047+0000 front 2024-02-23T00:47:00.443168+0000 (oldest deadline 2024-02-23T00:47:23.343052+0000) 2024-02-23T00:47:33.943 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4[43598]: debug 2024-02-23T00:47:33.736+0000 7f0bcb724700 -1 osd.4 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:09.744750+0000 front 2024-02-23T00:47:09.744816+0000 (oldest deadline 2024-02-23T00:47:33.244550+0000) 2024-02-23T00:47:33.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:33.728+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:33.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:33.728+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:33.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:33.689+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:33.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:33 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:33.689+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:34.442 INFO:journalctl@ceph.osd.4.smithi171.stdout:Feb 23 00:47:34 smithi171 bash[120631]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-4 2024-02-23T00:47:34.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:34.112+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:34.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:34.112+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:34.909 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:34.647+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:34.909 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:34.647+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:34.909 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:34.718+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:34.909 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:34 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:34.718+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:35.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:35.067+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:35.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:35.067+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:35.877 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:35.671+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:35.877 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:35.671+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:35.878 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:35.650+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:35.878 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:35 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:35.650+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:35.885 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.4.service' 2024-02-23T00:47:35.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:47:35.967 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-23T00:47:35.967 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-23T00:47:35.967 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.5 2024-02-23T00:47:36.192 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:36 smithi171 systemd[1]: Stopping Ceph osd.5 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:47:36.193 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:36 smithi171 bash[120866]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.5" found: no such container 2024-02-23T00:47:36.193 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:36.116+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:36.193 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:36.116+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:36.650 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:36.280+0000 7f5ebb461700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:47:36.650 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:36.280+0000 7f5ebb461700 -1 osd.5 857 *** Got signal Terminated *** 2024-02-23T00:47:36.650 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:36.280+0000 7f5ebb461700 -1 osd.5 857 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-23T00:47:36.942 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:36.648+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:36.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:36.648+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:36.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:36.648+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:36.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:36 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:36.648+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:37.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:37.099+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:37.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:37.099+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:37.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:37.637+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:37.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:37.637+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:37.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:37.663+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:37.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:37 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:37.663+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:38.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:38.139+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:38.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:38.139+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:38.942 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:38.693+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:38.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:38.693+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:38.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:38.632+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:38.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:38.632+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:38.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:38 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:38.632+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:39.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:39.166+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:39.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:39.166+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:39.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:39.717+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:39.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:39.717+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:39.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:39.659+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:39.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:39.659+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:39.944 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:39 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:39.659+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:40.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:40.126+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:40.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:40.126+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:40.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:47:40.146Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:40.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:40.146Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:40.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:40 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:40.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:40.942 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:40.715+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:46:58.417667+0000 front 2024-02-23T00:46:58.417735+0000 (oldest deadline 2024-02-23T00:47:24.317759+0000) 2024-02-23T00:47:40.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:40.715+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.618825+0000 front 2024-02-23T00:47:06.618891+0000 (oldest deadline 2024-02-23T00:47:31.318858+0000) 2024-02-23T00:47:40.943 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5[48189]: debug 2024-02-23T00:47:40.715+0000 7f5eb83de700 -1 osd.5 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:15.920341+0000 front 2024-02-23T00:47:15.920280+0000 (oldest deadline 2024-02-23T00:47:40.620334+0000) 2024-02-23T00:47:40.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:40.666+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:40.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:40.666+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:40.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:40 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:40.666+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:41.444 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:41.112+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:41.444 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:41.112+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:41.942 INFO:journalctl@ceph.osd.5.smithi171.stdout:Feb 23 00:47:41 smithi171 bash[120906]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-5 2024-02-23T00:47:41.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:41.689+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:41.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:41.689+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:41.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:41 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:41.689+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:42.351 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:42.093+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:42.351 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:42.093+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:42.351 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:42.093+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:17.862928+0000 front 2024-02-23T00:47:17.862957+0000 (oldest deadline 2024-02-23T00:47:41.962967+0000) 2024-02-23T00:47:42.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:42.669+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:42.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:42.669+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:42.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:42 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:42.669+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:43.341 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:43.085+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:43.342 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:43.085+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:43.342 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:43.085+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:17.862928+0000 front 2024-02-23T00:47:17.862957+0000 (oldest deadline 2024-02-23T00:47:41.962967+0000) 2024-02-23T00:47:43.349 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.5.service' 2024-02-23T00:47:43.389 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:47:43.389 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-23T00:47:43.389 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-23T00:47:43.389 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.6 2024-02-23T00:47:43.692 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:43 smithi171 systemd[1]: Stopping Ceph osd.6 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:47:43.692 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:43 smithi171 bash[121142]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.6" found: no such container 2024-02-23T00:47:43.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:47:43.578Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:43.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:43.579Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:43.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:43 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:43.579Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:44.105 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:43.729+0000 7fef570e2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:47:44.105 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:43.729+0000 7fef570e2700 -1 osd.6 857 *** Got signal Terminated *** 2024-02-23T00:47:44.105 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:43.729+0000 7fef570e2700 -1 osd.6 857 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-23T00:47:44.105 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:43.713+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:44.105 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:43.713+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:44.105 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:43 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:43.713+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:44.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:44.103+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:44.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:44.103+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:44.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:44.103+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:17.862928+0000 front 2024-02-23T00:47:17.862957+0000 (oldest deadline 2024-02-23T00:47:41.962967+0000) 2024-02-23T00:47:44.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:44.673+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:44.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:44.673+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:44.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:44 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:44.673+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:45.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:45.149+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:45.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:45.149+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:45.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:45.149+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:17.862928+0000 front 2024-02-23T00:47:17.862957+0000 (oldest deadline 2024-02-23T00:47:41.962967+0000) 2024-02-23T00:47:46.126 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:45.713+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:46.126 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:45.713+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:46.127 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:45 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:45.713+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:46.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:46.124+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:46.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:46.124+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:46.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:46.124+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:17.862928+0000 front 2024-02-23T00:47:17.862957+0000 (oldest deadline 2024-02-23T00:47:41.962967+0000) 2024-02-23T00:47:46.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:46.678+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:46.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:46.678+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:46.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:46 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:46.678+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:47.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:47.159+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:47.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:47.160+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:47.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:47.160+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:17.862928+0000 front 2024-02-23T00:47:17.862957+0000 (oldest deadline 2024-02-23T00:47:41.962967+0000) 2024-02-23T00:47:47.443 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:47.160+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6815 osd.3 since back 2024-02-23T00:47:21.963473+0000 front 2024-02-23T00:47:21.963380+0000 (oldest deadline 2024-02-23T00:47:46.663427+0000) 2024-02-23T00:47:48.192 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:47.702+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:48.192 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:47.702+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:48.193 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:47 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:47.702+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:48.678 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:48.209+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.360900+0000 front 2024-02-23T00:47:00.361020+0000 (oldest deadline 2024-02-23T00:47:24.460838+0000) 2024-02-23T00:47:48.678 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:48.209+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:04.461318+0000 front 2024-02-23T00:47:04.461290+0000 (oldest deadline 2024-02-23T00:47:30.361231+0000) 2024-02-23T00:47:48.678 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:48.209+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:17.862928+0000 front 2024-02-23T00:47:17.862957+0000 (oldest deadline 2024-02-23T00:47:41.962967+0000) 2024-02-23T00:47:48.678 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6[52606]: debug 2024-02-23T00:47:48.209+0000 7fef5185a700 -1 osd.6 857 heartbeat_check: no reply from 172.21.15.87:6815 osd.3 since back 2024-02-23T00:47:21.963473+0000 front 2024-02-23T00:47:21.963380+0000 (oldest deadline 2024-02-23T00:47:46.663427+0000) 2024-02-23T00:47:48.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:48.676+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:48.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:48.676+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:48.944 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:48 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:48.676+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:49.442 INFO:journalctl@ceph.osd.6.smithi171.stdout:Feb 23 00:47:49 smithi171 bash[121183]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-6 2024-02-23T00:47:49.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:49.636+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:49.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:49.636+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:49.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:49.636+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:49.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:49 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:49.636+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6815 osd.3 since back 2024-02-23T00:47:24.960904+0000 front 2024-02-23T00:47:24.960915+0000 (oldest deadline 2024-02-23T00:47:49.060770+0000) 2024-02-23T00:47:50.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:47:50.146Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:50.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:50.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:50.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:50 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:50.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:50.670 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:50.588+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:50.670 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:50.589+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:50.670 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:50.589+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:50.670 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:50 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:50.589+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6815 osd.3 since back 2024-02-23T00:47:24.960904+0000 front 2024-02-23T00:47:24.960915+0000 (oldest deadline 2024-02-23T00:47:49.060770+0000) 2024-02-23T00:47:50.677 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.6.service' 2024-02-23T00:47:50.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:47:50.718 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-23T00:47:50.718 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-23T00:47:50.718 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.7 2024-02-23T00:47:50.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:50 smithi171 systemd[1]: Stopping Ceph osd.7 for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:47:50.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:50 smithi171 bash[121418]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd.7" found: no such container 2024-02-23T00:47:51.443 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:51.021+0000 7fbed9faf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:47:51.443 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:51.021+0000 7fbed9faf700 -1 osd.7 857 *** Got signal Terminated *** 2024-02-23T00:47:51.443 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:51.021+0000 7fbed9faf700 -1 osd.7 857 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-23T00:47:51.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:51.589+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:51.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:51.589+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:51.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:51.589+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:51.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:51 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:51.589+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6815 osd.3 since back 2024-02-23T00:47:24.960904+0000 front 2024-02-23T00:47:24.960915+0000 (oldest deadline 2024-02-23T00:47:49.060770+0000) 2024-02-23T00:47:52.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:52.539+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:52.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:52.539+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:52.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:52.539+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:52.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:52 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:52.539+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6815 osd.3 since back 2024-02-23T00:47:24.960904+0000 front 2024-02-23T00:47:24.960915+0000 (oldest deadline 2024-02-23T00:47:49.060770+0000) 2024-02-23T00:47:53.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:47:53.578Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:53.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:53.579Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:47:53.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:47:53 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:47:53.579Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:47:53.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:53.491+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:53.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:53.491+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:53.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:53.491+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:53.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:53 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:53.491+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6815 osd.3 since back 2024-02-23T00:47:24.960904+0000 front 2024-02-23T00:47:24.960915+0000 (oldest deadline 2024-02-23T00:47:49.060770+0000) 2024-02-23T00:47:54.942 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:54.445+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:54.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:54.445+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:54.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:54.445+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:54.943 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:54 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:54.445+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6815 osd.3 since back 2024-02-23T00:47:24.960904+0000 front 2024-02-23T00:47:24.960915+0000 (oldest deadline 2024-02-23T00:47:49.060770+0000) 2024-02-23T00:47:55.692 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:55.430+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6803 osd.0 since back 2024-02-23T00:47:00.556785+0000 front 2024-02-23T00:47:00.556918+0000 (oldest deadline 2024-02-23T00:47:26.456876+0000) 2024-02-23T00:47:55.692 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:55.430+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6807 osd.1 since back 2024-02-23T00:47:06.457601+0000 front 2024-02-23T00:47:06.457615+0000 (oldest deadline 2024-02-23T00:47:30.557608+0000) 2024-02-23T00:47:55.692 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:55.430+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6811 osd.2 since back 2024-02-23T00:47:14.058686+0000 front 2024-02-23T00:47:14.058671+0000 (oldest deadline 2024-02-23T00:47:38.158586+0000) 2024-02-23T00:47:55.693 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:55 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7[57208]: debug 2024-02-23T00:47:55.430+0000 7fbed4f28700 -1 osd.7 857 heartbeat_check: no reply from 172.21.15.87:6815 osd.3 since back 2024-02-23T00:47:24.960904+0000 front 2024-02-23T00:47:24.960915+0000 (oldest deadline 2024-02-23T00:47:49.060770+0000) 2024-02-23T00:47:56.693 INFO:journalctl@ceph.osd.7.smithi171.stdout:Feb 23 00:47:56 smithi171 bash[121456]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-osd-7 2024-02-23T00:47:58.215 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@osd.7.service' 2024-02-23T00:47:58.258 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:47:58.258 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-23T00:47:58.259 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-23T00:47:58.259 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@rgw.foo.a 2024-02-23T00:47:58.555 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:47:58 smithi087 systemd[1]: Stopping Ceph rgw.foo.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:47:58.556 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:47:58 smithi087 bash[258383]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-rgw.foo.a" found: no such container 2024-02-23T00:47:58.833 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:47:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-rgw-foo-a[64469]: debug 2024-02-23T00:47:58.553+0000 7fb184b77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-23T00:47:58.833 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:47:58 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-rgw-foo-a[64469]: debug 2024-02-23T00:47:58.553+0000 7fb19bc41880 -1 shutting down 2024-02-23T00:48:00.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:48:00.147Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:48:00.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:48:00.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:48:00.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:00 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:48:00.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:48:03.834 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:48:03.579Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:48:03.834 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:48:03.580Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:48:03.834 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:03 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:48:03.580Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:48:08.833 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:48:08 smithi087 bash[258425]: time="2024-02-23T00:48:08Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-66db207c-d1e1-11ee-95c0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-23T00:48:09.333 INFO:journalctl@ceph.rgw.foo.a.smithi087.stdout:Feb 23 00:48:09 smithi087 bash[258425]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-rgw-foo-a 2024-02-23T00:48:09.533 DEBUG:teuthology.orchestra.run.smithi087:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@rgw.foo.a.service' 2024-02-23T00:48:09.571 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:48:09.571 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-23T00:48:09.571 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-23T00:48:09.572 DEBUG:teuthology.orchestra.run.smithi171:> sudo systemctl stop ceph-66db207c-d1e1-11ee-95c0-87774f69a715@prometheus.a 2024-02-23T00:48:09.880 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 systemd[1]: Stopping Ceph prometheus.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:48:09.880 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 bash[121701]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus.a" found: no such container 2024-02-23T00:48:10.165 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.877Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-23T00:48:10.165 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-23T00:48:10.166 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:09 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a[62947]: ts=2024-02-23T00:48:09.878Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-23T00:48:10.442 INFO:journalctl@ceph.prometheus.a.smithi171.stdout:Feb 23 00:48:10 smithi171 bash[121741]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-prometheus-a 2024-02-23T00:48:10.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=error ts=2024-02-23T00:48:10.147Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:48:10.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:48:10.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi087.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.87:8443: connect: connection refused" 2024-02-23T00:48:10.583 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:10 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=warn ts=2024-02-23T00:48:10.149Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi171.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.171:8443: connect: connection refused" 2024-02-23T00:48:10.652 DEBUG:teuthology.orchestra.run.smithi171:> sudo pkill -f 'journalctl -f -n 0 -u ceph-66db207c-d1e1-11ee-95c0-87774f69a715@prometheus.a.service' 2024-02-23T00:48:10.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-23T00:48:10.688 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-23T00:48:10.689 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 --force --keep-logs 2024-02-23T00:48:12.084 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:11 smithi087 systemd[1]: Stopping Ceph alertmanager.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:48:12.084 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:11 smithi087 bash[258857]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager.a" found: no such container 2024-02-23T00:48:12.420 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:12 smithi087 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a[71119]: level=info ts=2024-02-23T00:48:12.113Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-23T00:48:12.833 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:12 smithi087 bash[258895]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-alertmanager-a 2024-02-23T00:48:13.333 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:13 smithi087 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-23T00:48:13.333 INFO:journalctl@ceph.alertmanager.a.smithi087.stdout:Feb 23 00:48:13 smithi087 systemd[1]: Stopped Ceph alertmanager.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:48:13.334 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:48:13 smithi087 systemd[1]: Stopping Ceph node-exporter.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:48:13.334 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:48:13 smithi087 bash[258998]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter.a" found: no such container 2024-02-23T00:48:13.958 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:48:13 smithi087 bash[259037]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-a 2024-02-23T00:48:14.258 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:48:13 smithi087 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-23T00:48:14.258 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:48:14 smithi087 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-23T00:48:14.258 INFO:journalctl@ceph.node-exporter.a.smithi087.stdout:Feb 23 00:48:14 smithi087 systemd[1]: Stopped Ceph node-exporter.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:48:14.504 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 --force --keep-logs 2024-02-23T00:48:15.693 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:48:15 smithi171 systemd[1]: Stopping Ceph iscsi.iscsi.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:48:15.693 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:48:15 smithi171 bash[122098]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi.iscsi.a" found: no such container 2024-02-23T00:48:16.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:48:15 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a[60754]: debug Shutdown received 2024-02-23T00:48:26.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:48:25 smithi171 bash[122136]: time="2024-02-23T00:48:25Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-23T00:48:26.443 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:48:26 smithi171 bash[122136]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a 2024-02-23T00:48:26.923 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:48:26 smithi171 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-23T00:48:27.193 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:48:27 smithi171 systemd[1]: Stopping Ceph node-exporter.b for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:48:27.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:48:27 smithi171 bash[122210]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-iscsi-iscsi-a-tcmu 2024-02-23T00:48:27.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:48:27 smithi171 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-23T00:48:27.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi171.stdout:Feb 23 00:48:27 smithi171 systemd[1]: Stopped Ceph iscsi.iscsi.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:48:27.692 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:48:27 smithi171 bash[122287]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter.b" found: no such container 2024-02-23T00:48:27.956 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:48:27 smithi171 bash[122327]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-node-exporter-b 2024-02-23T00:48:28.230 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:48:27 smithi171 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-23T00:48:28.230 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:48:28 smithi171 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-23T00:48:28.230 INFO:journalctl@ceph.node-exporter.b.smithi171.stdout:Feb 23 00:48:28 smithi171 systemd[1]: Stopped Ceph node-exporter.b for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:48:28.614 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:48:28 smithi171 systemd[1]: Stopping Ceph grafana.a for 66db207c-d1e1-11ee-95c0-87774f69a715... 2024-02-23T00:48:28.614 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:48:28 smithi171 bash[122461]: Error: no container with name or ID "ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana.a" found: no such container 2024-02-23T00:48:28.931 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:48:28 smithi171 ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a[63313]: t=2024-02-23T00:48:28+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-23T00:48:29.192 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:48:28 smithi171 bash[122501]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715-grafana-a 2024-02-23T00:48:29.519 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:48:29 smithi171 systemd[1]: ceph-66db207c-d1e1-11ee-95c0-87774f69a715@grafana.a.service: Succeeded. 2024-02-23T00:48:29.519 INFO:journalctl@ceph.grafana.a.smithi171.stdout:Feb 23 00:48:29 smithi171 systemd[1]: Stopped Ceph grafana.a for 66db207c-d1e1-11ee-95c0-87774f69a715. 2024-02-23T00:48:29.630 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:48:29.669 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-23T00:48:29.699 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-23T00:48:29.701 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394/remote/ubuntu@smithi087.front.sepia.ceph.com/crash 2024-02-23T00:48:29.702 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/crash -- . 2024-02-23T00:48:29.779 INFO:teuthology.orchestra.run.smithi087.stderr:tar: /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/crash: Cannot open: No such file or directory 2024-02-23T00:48:29.780 INFO:teuthology.orchestra.run.smithi087.stderr:tar: Error is not recoverable: exiting now 2024-02-23T00:48:29.781 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394/remote/ubuntu@smithi171.front.sepia.ceph.com/crash 2024-02-23T00:48:29.782 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/crash -- . 2024-02-23T00:48:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:tar: /var/lib/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/crash: Cannot open: No such file or directory 2024-02-23T00:48:29.831 INFO:teuthology.orchestra.run.smithi171.stderr:tar: Error is not recoverable: exiting now 2024-02-23T00:48:29.832 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-23T00:48:29.832 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v 'flag\(s\) set' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(POOL_' | egrep -v '\(CEPHADM_STRAY_DAEMON\)' | egrep -v PG_ | egrep -v degraded | egrep -v backfill | egrep -v 'mons down' | egrep -v OSD_ | egrep -v 'is down' | egrep -v acting | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-23T00:48:29.868 INFO:teuthology.orchestra.run.smithi087.stdout:2024-02-23T00:35:32.304538+0000 mon.a (mon.0) 2466 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-23T00:48:29.869 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-23T00:48:29.869 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[SEC\]' /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v 'flag\(s\) set' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(POOL_' | egrep -v '\(CEPHADM_STRAY_DAEMON\)' | egrep -v PG_ | egrep -v degraded | egrep -v backfill | egrep -v 'mons down' | egrep -v OSD_ | egrep -v 'is down' | egrep -v acting | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-23T00:48:29.943 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]' /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v 'flag\(s\) set' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(POOL_' | egrep -v '\(CEPHADM_STRAY_DAEMON\)' | egrep -v PG_ | egrep -v degraded | egrep -v backfill | egrep -v 'mons down' | egrep -v OSD_ | egrep -v 'is down' | egrep -v acting | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-23T00:48:30.018 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[WRN\]' /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v 'flag\(s\) set' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(POOL_' | egrep -v '\(CEPHADM_STRAY_DAEMON\)' | egrep -v PG_ | egrep -v degraded | egrep -v backfill | egrep -v 'mons down' | egrep -v OSD_ | egrep -v 'is down' | egrep -v acting | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-02-23T00:48:30.091 INFO:teuthology.orchestra.run.smithi087.stdout:2024-02-23T00:35:32.304538+0000 mon.a (mon.0) 2466 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-23T00:48:30.092 INFO:tasks.cephadm:Compressing logs... 2024-02-23T00:48:30.092 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-23T00:48:30.134 DEBUG:teuthology.orchestra.run.smithi171:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-23T00:48:30.159 INFO:teuthology.orchestra.run.smithi087.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-23T00:48:30.160 INFO:teuthology.orchestra.run.smithi087.stderr:: No such file or directory 2024-02-23T00:48:30.160 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-volume.log 2024-02-23T00:48:30.160 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.2.log 2024-02-23T00:48:30.160 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mgr.y.log 2024-02-23T00:48:30.160 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-volume.log: /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.audit.log 2024-02-23T00:48:30.161 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.cephadm.log 2024-02-23T00:48:30.161 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.3.log 2024-02-23T00:48:30.161 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.cephadm.log: 88.9% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.cephadm.log.gz 2024-02-23T00:48:30.163 INFO:teuthology.orchestra.run.smithi171.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-23T00:48:30.163 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-23T00:48:30.163 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.6.log 2024-02-23T00:48:30.163 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-volume.log 2024-02-23T00:48:30.163 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.audit.log 2024-02-23T00:48:30.163 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.cephadm.log 2024-02-23T00:48:30.163 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-volume.log: /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.7.log 2024-02-23T00:48:30.163 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.cephadm.log: 81.5% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.cephadm.log.gz 2024-02-23T00:48:30.164 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.0.log 2024-02-23T00:48:30.165 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.3.log: /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-23T00:48:30.168 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.7.log: 91.5% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-volume.log.gz 2024-02-23T00:48:30.171 INFO:teuthology.orchestra.run.smithi171.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-23T00:48:30.173 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mgr.x.log 2024-02-23T00:48:30.174 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mon.b.log 2024-02-23T00:48:30.174 INFO:teuthology.orchestra.run.smithi171.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log 2024-02-23T00:48:30.174 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mgr.x.log: 92.0% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.audit.log.gz 2024-02-23T00:48:30.176 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-client.rgw.foo.a.log: 91.4% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-volume.log.gz 2024-02-23T00:48:30.176 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/objectstore_tool.7.log 2024-02-23T00:48:30.178 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.4.log 2024-02-23T00:48:30.178 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.5.log 2024-02-23T00:48:30.178 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.4.log: /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/tcmu-runner.log 2024-02-23T00:48:30.180 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/objectstore_tool.7.log: 95.0% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.audit.log.gz 2024-02-23T00:48:30.180 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mon.a.log 2024-02-23T00:48:30.180 INFO:teuthology.orchestra.run.smithi171.stderr: 92.0% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mgr.x.log.gz 2024-02-23T00:48:30.182 INFO:teuthology.orchestra.run.smithi087.stderr: 95.2% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/objectstore_tool.7.log.gz 2024-02-23T00:48:30.185 INFO:teuthology.orchestra.run.smithi171.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/tcmu-runner.log: 68.2% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/tcmu-runner.log.gz 2024-02-23T00:48:30.185 INFO:teuthology.orchestra.run.smithi171.stderr: 86.1% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log.gz 2024-02-23T00:48:30.193 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mon.c.log 2024-02-23T00:48:30.204 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log 2024-02-23T00:48:30.204 INFO:teuthology.orchestra.run.smithi087.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-23T00:48:30.217 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.1.log 2024-02-23T00:48:30.224 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.1.log: 92.2% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph.log.gz 2024-02-23T00:48:30.390 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-23T00:48:30.599 INFO:teuthology.orchestra.run.smithi087.stderr: 90.6% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mgr.y.log.gz 2024-02-23T00:48:31.161 INFO:teuthology.orchestra.run.smithi171.stderr: 92.4% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mon.b.log.gz 2024-02-23T00:48:31.234 INFO:teuthology.orchestra.run.smithi087.stderr: 92.0% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mon.c.log.gz 2024-02-23T00:48:32.130 INFO:teuthology.orchestra.run.smithi087.stderr: 91.4% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-mon.a.log.gz 2024-02-23T00:48:34.344 INFO:teuthology.orchestra.run.smithi087.stderr: 94.2% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.0.log.gz 2024-02-23T00:48:35.760 INFO:teuthology.orchestra.run.smithi087.stderr: 94.2% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.3.log.gz 2024-02-23T00:48:36.118 INFO:teuthology.orchestra.run.smithi171.stderr: 94.0% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.6.log.gz 2024-02-23T00:48:36.189 INFO:teuthology.orchestra.run.smithi087.stderr: 94.1% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.2.log.gz 2024-02-23T00:48:36.210 INFO:teuthology.orchestra.run.smithi171.stderr: 94.1% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.7.log.gz 2024-02-23T00:48:36.248 INFO:teuthology.orchestra.run.smithi171.stderr: 94.1% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.4.log.gz 2024-02-23T00:48:36.269 INFO:teuthology.orchestra.run.smithi171.stderr: 94.0% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.5.log.gz 2024-02-23T00:48:36.271 INFO:teuthology.orchestra.run.smithi171.stderr: 2024-02-23T00:48:36.271 INFO:teuthology.orchestra.run.smithi171.stderr:real 0m6.120s 2024-02-23T00:48:36.271 INFO:teuthology.orchestra.run.smithi171.stderr:user 0m24.234s 2024-02-23T00:48:36.271 INFO:teuthology.orchestra.run.smithi171.stderr:sys 0m0.960s 2024-02-23T00:48:37.061 INFO:teuthology.orchestra.run.smithi087.stderr: 94.2% -- replaced with /var/log/ceph/66db207c-d1e1-11ee-95c0-87774f69a715/ceph-osd.1.log.gz 2024-02-23T00:48:37.063 INFO:teuthology.orchestra.run.smithi087.stderr: 2024-02-23T00:48:37.063 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m6.914s 2024-02-23T00:48:37.063 INFO:teuthology.orchestra.run.smithi087.stderr:user 0m25.428s 2024-02-23T00:48:37.063 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m0.952s 2024-02-23T00:48:37.063 INFO:tasks.cephadm:Archiving logs... 2024-02-23T00:48:37.063 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394/remote/ubuntu@smithi087.front.sepia.ceph.com/log 2024-02-23T00:48:37.064 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-23T00:48:40.547 DEBUG:teuthology.misc:Transferring archived files from smithi171:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394/remote/ubuntu@smithi171.front.sepia.ceph.com/log 2024-02-23T00:48:40.547 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-23T00:48:44.189 INFO:tasks.cephadm:Removing cluster... 2024-02-23T00:48:44.190 DEBUG:teuthology.orchestra.run.smithi087:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 --force 2024-02-23T00:48:44.673 DEBUG:teuthology.orchestra.run.smithi171:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 66db207c-d1e1-11ee-95c0-87774f69a715 --force 2024-02-23T00:48:45.167 INFO:tasks.cephadm:Removing cephadm ... 2024-02-23T00:48:45.167 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-23T00:48:45.185 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-23T00:48:45.203 INFO:tasks.cephadm:Teardown complete 2024-02-23T00:48:45.204 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-23T00:48:45.214 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-23T00:48:45.215 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-23T00:48:45.227 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-23T00:48:45.327 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-23T00:48:45.338 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-23T00:48:45.339 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-23T00:48:45.341 DEBUG:teuthology.orchestra.run.smithi171:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-23T00:48:45.355 INFO:teuthology.orchestra.run.smithi087.stderr:bash: ntpq: command not found 2024-02-23T00:48:45.358 INFO:teuthology.orchestra.run.smithi171.stderr:bash: ntpq: command not found 2024-02-23T00:48:45.359 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-23T00:48:45.360 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2024-02-23T00:48:45.360 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 +2218us[+2224us] +/- 81ms 2024-02-23T00:48:45.360 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -829us[ -844us] +/- 56ms 2024-02-23T00:48:45.360 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -830us[ -830us] +/- 86ms 2024-02-23T00:48:45.360 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:48:45.361 INFO:teuthology.orchestra.run.smithi171.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-23T00:48:45.362 INFO:teuthology.orchestra.run.smithi171.stdout:=============================================================================== 2024-02-23T00:48:45.362 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 64 +2244us[+2244us] +/- 81ms 2024-02-23T00:48:45.362 INFO:teuthology.orchestra.run.smithi171.stdout:^* hv02.front.sepia.ceph.com 3 6 377 64 -789us[ -748us] +/- 55ms 2024-02-23T00:48:45.362 INFO:teuthology.orchestra.run.smithi171.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -764us[ -764us] +/- 86ms 2024-02-23T00:48:45.362 INFO:teuthology.orchestra.run.smithi171.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-23T00:48:45.363 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-23T00:48:45.404 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-23T00:48:45.404 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-23T00:48:45.437 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-23T00:48:45.527 DEBUG:teuthology.orchestra.run.smithi171:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-23T00:48:45.612 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-23T00:48:45.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:48:45.645 DEBUG:teuthology.orchestra.run.smithi171:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-23T00:48:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:48:45.677 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-23T00:48:45.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-23T00:48:45.747 INFO:teuthology.task.internal:Duration was 2754.259871 seconds 2024-02-23T00:48:45.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-23T00:48:45.769 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-23T00:48:45.769 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-23T00:48:45.772 DEBUG:teuthology.orchestra.run.smithi171:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-23T00:48:45.816 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-23T00:48:45.818 INFO:teuthology.orchestra.run.smithi171.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-23T00:48:46.202 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-23T00:48:46.202 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2024-02-23T00:48:46.202 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-23T00:48:46.225 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi171.front.sepia.ceph.com 2024-02-23T00:48:46.225 DEBUG:teuthology.orchestra.run.smithi171:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-23T00:48:46.247 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-23T00:48:46.247 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-23T00:48:46.268 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-23T00:48:46.383 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-23T00:48:46.383 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-23T00:48:46.385 DEBUG:teuthology.orchestra.run.smithi171:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-23T00:48:46.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-23T00:48:46.690 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-23T00:48:46.690 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-23T00:48:46.717 DEBUG:teuthology.orchestra.run.smithi171:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-23T00:48:46.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-23T00:48:46.755 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-23T00:48:46.760 DEBUG:teuthology.orchestra.run.smithi171:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-23T00:48:46.783 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2024-02-23T00:48:46.810 INFO:teuthology.orchestra.run.smithi171.stdout:kernel.core_pattern = core 2024-02-23T00:48:46.837 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-23T00:48:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:48:46.853 DEBUG:teuthology.orchestra.run.smithi171:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-23T00:48:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-23T00:48:46.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-23T00:48:46.904 INFO:teuthology.task.internal:Transferring archived files... 2024-02-23T00:48:46.905 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394/remote/smithi087 2024-02-23T00:48:46.906 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-23T00:48:46.966 DEBUG:teuthology.misc:Transferring archived files from smithi171:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394/remote/smithi171 2024-02-23T00:48:46.967 DEBUG:teuthology.orchestra.run.smithi171:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-23T00:48:47.017 INFO:teuthology.task.internal:Removing archive directory... 2024-02-23T00:48:47.017 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-23T00:48:47.019 DEBUG:teuthology.orchestra.run.smithi171:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-23T00:48:47.065 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-23T00:48:47.077 INFO:teuthology.task.internal:Not uploading archives. 2024-02-23T00:48:47.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-23T00:48:47.104 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-23T00:48:47.104 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-23T00:48:47.106 DEBUG:teuthology.orchestra.run.smithi171:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-23T00:48:47.121 INFO:teuthology.orchestra.run.smithi087.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 23 00:48 /home/ubuntu/cephtest 2024-02-23T00:48:47.124 INFO:teuthology.orchestra.run.smithi171.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 23 00:48 /home/ubuntu/cephtest 2024-02-23T00:48:47.125 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-23T00:48:47.136 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-23T00:48:47.204 INFO:teuthology.nuke:Checking targets against current locks 2024-02-23T00:48:47.228 DEBUG:teuthology.nuke:shortname: smithi087 2024-02-23T00:48:47.228 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-23T00:48:47.255 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 23:53:49.978859', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG18gcYhtOJmt3HHvenammzLLFirbbVECGK7ZCuWfAeEmu+6oX0B/nLyGnYIE1yKHtRGRYHHNYKNZ0yaR+SYmrw='} 2024-02-23T00:48:47.256 DEBUG:teuthology.nuke:shortname: smithi171 2024-02-23T00:48:47.256 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-23T00:48:47.276 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi171.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-22_21:39:39-rados-pacific-release-distro-default-smithi/7571394', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-22 23:53:49.980546', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzOyZei/1wpgCXnjKU9xo9p1MMg970O+tjaEwG0Bi5A16K5vGrdD/3ohhWIj8h2INcrdRW5/wN/mxE9yVeEBYk='} 2024-02-23T00:48:47.302 INFO:teuthology.orchestra.console.smithi087:Power off 2024-02-23T00:48:47.302 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-23T00:48:47.327 DEBUG:teuthology.orchestra.console.smithi087:power off output: Chassis Power Control: Down/Off 2024-02-23T00:48:47.327 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:48:47.343 INFO:teuthology.orchestra.console.smithi171:Power off 2024-02-23T00:48:47.343 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-23T00:48:47.353 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-02-23T00:48:47.366 DEBUG:teuthology.orchestra.console.smithi171:power off output: Chassis Power Control: Down/Off 2024-02-23T00:48:47.366 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:48:47.390 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-02-23T00:48:51.355 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:48:51.392 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:48:51.465 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-02-23T00:48:51.501 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is on 2024-02-23T00:48:55.468 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:48:55.502 DEBUG:teuthology.orchestra.console.smithi171:pexpect command: ipmitool -H smithi171.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-23T00:48:55.579 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2024-02-23T00:48:55.612 DEBUG:teuthology.orchestra.console.smithi171:check power output: Chassis Power is off 2024-02-23T00:48:55.680 INFO:teuthology.orchestra.console.smithi087:Power off completed 2024-02-23T00:48:55.713 INFO:teuthology.orchestra.console.smithi171:Power off completed 2024-02-23T00:48:55.817 INFO:teuthology.run:Summary data: description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} duration: 2754.259870529175 failure_reason: '"2024-02-23T00:35:32.304538+0000 mon.a (mon.0) 2466 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-02-23T00:48:55.817 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-23T00:48:55.907 INFO:teuthology.run:FAIL